Repository: burningmyself/burningmyself.github.io Branch: master Commit: 0ee36298a35f Files: 177 Total size: 3.6 MB Directory structure: gitextract_h3qke_bo/ ├── .gitignore ├── CNAME ├── README.md ├── docs/ │ ├── CNAME │ ├── cloud/ │ │ ├── apprelease.md │ │ ├── compute.md │ │ ├── containerlinux.md │ │ ├── docker/ │ │ │ ├── docker_compose.md │ │ │ ├── docker_container.md │ │ │ ├── docker_container_enterprice.md │ │ │ ├── docker_date.md │ │ │ ├── docker_devops.md │ │ │ ├── docker_file.md │ │ │ ├── docker_image.md │ │ │ ├── docker_image_fast.md │ │ │ ├── docker_native.md │ │ │ ├── docker_network.md │ │ │ ├── docker_nginx.md │ │ │ └── docker_swarm.md │ │ ├── kubernetes/ │ │ │ ├── kubernetes_ack.md │ │ │ ├── kubernetes_calico.md │ │ │ ├── kubernetes_cluster.md │ │ │ ├── kubernetes_cluster_serve.md │ │ │ ├── kubernetes_configMap_secret.md │ │ │ ├── kubernetes_core.md │ │ │ ├── kubernetes_deploy_golang.md │ │ │ ├── kubernetes_deploy_java.md │ │ │ ├── kubernetes_deploy_python.md │ │ │ ├── kubernetes_devops.md │ │ │ ├── kubernetes_flannel.md │ │ │ ├── kubernetes_flink.md │ │ │ ├── kubernetes_gitops.md │ │ │ ├── kubernetes_harbor.md │ │ │ ├── kubernetes_hdfs.md │ │ │ ├── kubernetes_helm.md │ │ │ ├── kubernetes_helm_prometheus.md │ │ │ ├── kubernetes_hight.md │ │ │ ├── kubernetes_hight_bin1.md │ │ │ ├── kubernetes_hight_bin2.md │ │ │ ├── kubernetes_hybridnet.md │ │ │ ├── kubernetes_introduce.md │ │ │ ├── kubernetes_ipv4_and_ipv6.md │ │ │ ├── kubernetes_kafka.md │ │ │ ├── kubernetes_karmada.md │ │ │ ├── kubernetes_kubeconfig.md │ │ │ ├── kubernetes_kubesphere.md │ │ │ ├── kubernetes_kubesphere_devops.md │ │ │ ├── kubernetes_kurator.md │ │ │ ├── kubernetes_kustomize.md │ │ │ ├── kubernetes_logs_collect.md │ │ │ ├── kubernetes_master.md │ │ │ ├── kubernetes_nginx_ingress_controller.md │ │ │ ├── kubernetes_openfass.md │ │ │ ├── kubernetes_rancher.md │ │ │ ├── kubernetes_rke.md │ │ │ ├── kubernetes_rokectmq.md │ │ │ ├── kubernetes_safety.md │ │ │ ├── kubernetes_sealos.md │ │ │ ├── kubernetes_spark.md │ │ │ ├── kubernetes_storage_ceph.md │ │ │ ├── kubernetes_storage_volume.md │ │ │ ├── kubernetes_traefik.md │ │ │ ├── kubernetes_ui.md │ │ │ ├── kubernetes_velero.md │ │ │ ├── kubernetes_way.md │ │ │ └── kubernetes_zookeeper.md │ │ ├── kubernetes.md │ │ ├── native.md │ │ └── virtual.md │ ├── dart/ │ │ └── syntax.md │ ├── docker/ │ │ ├── docker-compose.md │ │ ├── docker-jenkins.md │ │ ├── docker-phabricator.md │ │ ├── docker.md │ │ └── docker_core.md │ ├── emotion/ │ │ ├── emotion.md │ │ ├── eq.md │ │ ├── lifetime.md │ │ ├── livefail.md │ │ ├── lookbook.md │ │ ├── losecome.md │ │ ├── onepath.md │ │ ├── selfdiscipline.md │ │ ├── threeheart.md │ │ ├── twopath.md │ │ └── workheard.md │ ├── exp/ │ │ ├── ai.md │ │ ├── cl.md │ │ ├── code-principle.md │ │ ├── cto.md │ │ ├── devops.md │ │ ├── four_deep_learning.md │ │ ├── learnweetout.md │ │ ├── micro-service.md │ │ ├── pt.md │ │ ├── raft-gossip.md │ │ ├── techbig.md │ │ └── tl.md │ ├── framework/ │ │ ├── agility.md │ │ ├── algorithm-ten.md │ │ ├── data_middle.md │ │ ├── fgb.md │ │ ├── fwork.md │ │ └── split.md │ ├── go/ │ │ └── go_base.md │ ├── index.md │ ├── java/ │ │ ├── bio-nio.md │ │ ├── feature.md │ │ ├── java-simple.md │ │ ├── java-string.md │ │ ├── java-utils.md │ │ ├── java-validator.md │ │ ├── javavm.md │ │ ├── load-class.md │ │ ├── orm.md │ │ ├── rocketmq/ │ │ │ └── rmq-1.md │ │ ├── spring-cloud.md │ │ ├── springAnnotation.md │ │ └── springdesign.md │ ├── js/ │ │ ├── baidu-tongji.js │ │ ├── extra.js │ │ └── google.js │ ├── kubernetes/ │ │ ├── k8s_controller_manager.md │ │ ├── k8s_etcd.md │ │ ├── k8s_fw_rule_and_object_design.md │ │ └── k8s_kube_APIServer.md │ ├── linux/ │ │ ├── linux.md │ │ ├── often.md │ │ └── ope.md │ ├── micro/ │ │ ├── ddd.md │ │ ├── design.md │ │ ├── distrimsg.md │ │ ├── fbs-lock.md │ │ ├── kafka.md │ │ ├── redis_cluster.md │ │ └── spring-cloud-micro.md │ ├── net/ │ │ ├── c_core_safety.md │ │ ├── c_core_study_route.md │ │ ├── c_docker.md │ │ ├── c_sharp.md │ │ └── c_sqlserver_nginx.md │ ├── php/ │ │ └── kj.md │ ├── python/ │ │ ├── fabric.md │ │ ├── feature.md │ │ ├── str_joint.md │ │ └── syntax_rule.md │ ├── sql/ │ │ ├── data_split.md │ │ ├── mybatis.md │ │ ├── mysql_backups.md │ │ ├── mysql_index.md │ │ ├── mysql_log.md │ │ ├── mysql_pxc.md │ │ ├── mysql_use.md │ │ ├── mysql_yh.md │ │ ├── mysql_yh17.md │ │ └── sql_server_master.md │ ├── tool/ │ │ ├── cat-monitoring.md │ │ ├── cicd.md │ │ ├── git.md │ │ ├── gitbook.md │ │ ├── gitcmr.md │ │ ├── gitflow.md │ │ ├── gitquestion.md │ │ ├── gitstudy.md │ │ ├── gitusual.md │ │ ├── markdown.md │ │ ├── minio.md │ │ └── mkdocs.md │ └── web/ │ ├── ali_js_style.md │ ├── es6.md │ ├── javascript.md │ ├── js_tool_method.md │ ├── node.js.md │ ├── react.md │ ├── react_interview.md │ └── vue_cp_react.md └── mkdocs.yml ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ /site /.obsidian /docs/csdn.md ================================================ FILE: CNAME ================================================ burningmyself.cn ================================================ FILE: README.md ================================================ # 个人博客 --- 本仓库下存放个人博客的源文件。持续更新，欢迎 star。 如果大家觉得那里写的不合适的可以给我提 Issue [Github](https://github.com/burningmyself) [Gitee](https://gitee.com/burningmyself) ![GitHub issues](https://img.shields.io/github/issues/burningmyself/burningmyself.github.io) ![GitHub forks](https://img.shields.io/github/forks/burningmyself/burningmyself.github.io) ![GitHub stars](https://img.shields.io/github/stars/burningmyself/burningmyself.github.io) ![GitHub license](https://img.shields.io/github/license/burningmyself/burningmyself.github.io) ![Twitter](https://img.shields.io/twitter/url?url=https%3A%2F%2Fgithub.com%2Fburningmyself%2Fburningmyself.github.io) ## 捐赠 如果你觉得这写文章能帮助到了你，你可以帮作者买一杯果汁表示鼓励 ![pay](docs/img/pay.png) [Paypal Me](https://paypal.me/yangfubing) ## 目录 * [介绍](docs/index.md) * [README.md](README.md) * [Docker] * [Docker介绍使用](docs/docker/docker.md) * [Docker 核心技术](docs/docker/docker_core.md) * [docker和docker-compose配置常用环境](docs/docker/docker-compose.md) * [Docker部署Jenkins](docs/docker/docker-jenkins.md) * [Docker部署Phabricator](docs/docker/docker-phabricator.md) * [K8s] * [Kubernetes架构原则和对象设计](docs/kubernetes/k8s_fw_rule_and_object_design.md) * [Kubernetes控制平面组件etcd](docs/kubernetes/k8s_etcd.md) * [深入理解Kube-APIServer](docs/kubernetes/k8s_kube_APIServer.md) * [kubernetes控制平面组件](docs/kubernetes/k8s_controller_manager.md) * [经验] * [集群和负载均衡](docs/exp/cl.md) * [DevOps是什么](docs/exp/devops.md) * [微服务架构技术栈选型手册](docs/exp/micro-service.md) * [Raft算法和Gossip协议](docs/exp/raft-gossip.md) * [人工智能](docs/exp/ai.md) * [代码原则](docs/exp/code-principle.md) * [如何成为技术大牛](docs/exp/techbig.md) * [程序员如何技术成长](docs/exp/learnweetout.md) * [产品与技术](docs/exp/pt.md) * [深度学习框架](docs/exp/four_deep_learning.md) * [在阿里做了5年技术Leader，我总结出这些套路！](docs/exp/tl.md) * [CTO 技能图谱](docs/exp/cto.md) * [Java] * [Java 新特性](docs/java/feature.md) * [Java 类加载机制](docs/java/load-class.md) * [Orm的优缺点](docs/java/orm.md) * [BIO与NIO](docs/java/bio-nio.md) * [RocketMq下载与安装](docs/java/rocketmq/rmq-1.md) * [Spring常用注解](docs/java/springAnnotation.md) * [Java 虚拟机](docs/java/javavm.md) * [Spring 常用设计](docs/java/springdesign.md) * [Spring Cloud](docs/java/spring-cloud.md) * [Java Simple](docs/java/java-simple.md) * [Java Validator](docs/java/java-validator.md) * [Java String](docs/java/java-string.md) * [Java Utils](docs/java/java-utils.md) * [Linux] * [Linux 学习笔记](docs/linux/linux.md) * [Linux常用命令](docs/linux/often.md) * [实用的Linux 命令](docs/linux/ope.md) * [微服务] * [分布式锁](docs/micro/fbs-lock.md) * [微服务设计](docs/micro/design.md) * [分布式系统与消息的投递](docs/micro/distrimsg.md) * [基于DDD的微服务设计和开发实战](docs/micro/ddd.md) * [Kafka架构原理](docs/micro/kafka.md) * [Redis Cluster原理](docs/micro/redis_cluster.md) * [分布式架构](docs/micro/spring-cloud-micro.md) * [.NET] * [C#新特性语法](docs/net/c_sharp.md) * [.Net Core Docker 部署](docs/net/c_docker.md) * [Docker容器化部署ASP.NET Core](docs/net/c_sqlserver_nginx.md) * [让你的ASP.NET Core应用程序更安全](docs/net/c_core_safety.md) * [ASP.NET Core开发者指南](docs/net/c_core_study_route.md) * [PHP] * [框架](docs/php/kj.md) * [Python] * [Python新特性](docs/python/feature.md) * [Python的字符操作](docs/python/str_joint.md) * [Python语法技巧](docs/python/syntax_rule.md) * [远程部署神器 Fabric](docs/python/fabric.md) * [Go] * [Go基础语法](docs/go/go_base.md) * [SQL] * [MySQL 优化指南](docs/sql/mysql_yh.md) * [MySQL优化](docs/sql/mysql_yh17.md) * [MySQL 日志](docs/sql/mysql_log.md) * [MySQL 索引](docs/sql/mysql_index.md) * [MySQL PXC集群](docs/sql/mysql_pxc.md) * [MySQL MySQL数据库应用](docs/sql/mysql_use.md) * [MySql 备份](docs/sql/mysql_backups.md) * [Sql Server 主从备份](docs/sql/sql_server_master.md) * [数据库之互联网常用分库分表方案](docs/sql/data_split.md) * [Mybatis使用心德](docs/sql/mybatis.md) * [TOOL] * [Git命令](docs/tool/git.md) * [Git命令动画展示](docs/tool/gitusual.md) * [GitBook](docs/tool/gitbook.md) * [GitFlow](docs/tool/gitflow.md) * [Git问题处理](docs/tool/gitquestion.md) * [Git提交日志规范](docs/tool/gitcmr.md) * [MarkDown](docs/tool/markdown.md) * [CI/CD](docs/tool/cicd.md) * [mkdocs简单使用](docs/tool/mkdocs.md) * [Git的黑魔法](docs/tool/gitstudy.md) * [MinIO 搭建使用](docs/tool/minio.md) * [Cat分布式监控](docs/tool.cat-monitoring.md) * [前端] * [阿里js样式](docs/web/ali_js_style.md) * [ES6语法](docs/web/es6.md) * [Node.js](docs/web/node.js.md) * [React 开发者指南](docs/web/react.md) * [Dart语法学习](docs/dart/syntax.md) * [React 面试](docs/web/react_interview.md) * [Js 工具函数](docs/web/js_tool_method.md) * [vue与react比较](docs/web/vue_cp_react.md) * [JavaScript 基础](docs/web/javascript.md) * [构架] * [构架拆分](docs/framework/split.md) * [分布式、高并发、多线程](docs/framework/fgb.md) * [如何理解敏捷开发](docs/framework/agility.md) * [走向架构师必备的技能](docs/framework/fwork.md) * [数据中台的思考与总结](docs/framework/data_middle.md) * [必学的 10 大算法](docs/framework/algorithm-ten.md) * [云架构] * [云原生](docs/cloud/native.md) * [虚拟化技术](docs/cloud/virtual.md) * [云计算](docs/cloud/compute.md) * [应用部署容器化演进](docs/cloud/apprelease.md) * [容器技术所涉及Linux内核关键技术](docs/cloud/containerlinux.md) * [容器管理工具Docker生态架构及部署](docs/cloud/docker/docker_native.md) * [使用容器运行Nginx应用及Docker命令](docs/cloud/docker/docker_nginx.md) * [Docker容器镜像](docs/cloud/docker/docker_image.md) * [Docker容器镜像加速器及本地容器镜像仓库](docs/cloud/docker/docker_image_fast.md) * [Docker容器化部署企业级应用集群](docs/cloud/docker/docker_container_enterprice.md) * [Dockerfile精讲及新型容器镜像构建技术](docs/cloud/docker/docker_file.md) * [Docker容器网络与通信原理深度解析](docs/cloud/docker/docker_network.md) * [Docker容器数据持久化存储机制](docs/cloud/docker/docker_date.md) * [Docker容器服务编排利器Docker Compose应用实战](docs/cloud/docker/docker_compose.md) * [Docker主机集群化方案 Docker Swarm](docs/cloud/docker/docker_swarm.md) * [基于Docker容器DevOps应用方案 企业业务代码发布系统](docs/cloud/docker/docker_devops.md) * [轻量级或工业级容器管理工具](docs/cloud/docker/docker_container.md) * [kubeadm极速部署Kubernetes](docs/cloud/kubernetes.md) * [kubernetes介绍与集群架构](docs/cloud/kubernetes/kubernetes_introduce.md) * [Kubernetes集群部署方式说明](docs/cloud/kubernetes/kubernetes_way.md) * [kubeadm部署单Master节点kubernetes集群](docs/cloud/kubernetes/kubernetes_master.md) * [kubeadm部署高可用kubernetes集群](docs/cloud/kubernetes/kubernetes_hight.md) * [使用RKE构建企业生产级Kubernetes集群](docs/cloud/kubernetes/kubernetes_rke.md) * [Kubernetes高可用集群二进制部署（Runtime Docker）](docs/cloud/kubernetes/kubernetes_hight_bin1.md) * [Kubernetes高可用集群二进制部署（Runtime Containerd）](docs/cloud/kubernetes/kubernetes_hight_bin2.md) * [Kubernetes集群UI及主机资源监控](docs/cloud/kubernetes/kubernetes_ui.md) * [使用sealos部署kubernetes集群并实现集群管理](docs/cloud/kubernetes/kubernetes_sealos.md) * [kubernetes集群命令语法](docs/cloud/kubernetes/kubernetes_cluster.md) * [Kubernetes核心概念](docs/cloud/kubernetes/kubernetes_core.md) * [Kubernetes集群 服务暴露 Nginx Ingress Controller](docs/cloud/kubernetes/kubernetes_nginx_ingress_controller.md) * [Kubernetes集群 服务暴露 Traefik](docs/cloud/kubernetes/kubernetes_traefik.md) * [Kubernetes配置与密钥管理 ConfigMap&Secret](docs/cloud/kubernetes/kubernetes_configMap_secret.md) * [Kubernetes集群使用容器镜像仓库Harbor](docs/cloud/kubernetes/kubernetes_harbor.md) * [Kubernetes集群安全管理](docs/cloud/kubernetes/kubernetes_safety.md) * [kubernetes持久化存储卷](docs/cloud/kubernetes/kubernetes_storage_volume.md) * [kubernetes存储解决方案Ceph](docs/cloud/kubernetes/kubernetes_storage_ceph.md) * [Kubernetes集群公共服务](docs/cloud/kubernetes/kubernetes_cluster_serve.md) * [kubernetes集群java项目上云部署](docs/cloud/kubernetes/kubernetes_deploy_java.md) * [kubernetes集群Python项目上云部署](docs/cloud/kubernetes/kubernetes_deploy_python.md) * [Kubernetes集群golang项目上云部署](docs/cloud/kubernetes/kubernetes_deploy_golang.md) * [helm部署prometheus监控系统及应用](docs/cloud/kubernetes/kubernetes_helm_prometheus.md) * [kubernetes日志收集方案ELK](docs/cloud/kubernetes/kubernetes_logs_collect.md) * [企业级中间件上云部署zookeeper](docs/cloud/kubernetes/kubernetes_zookeeper.md) * [kubernetes云原生中间件上云部署kafka](docs/cloud/kubernetes/kubernetes_kafka.md) * [rocketmq部署控](docs/cloud/kubernetes/kubernetes_rokectmq.md) * [Kubernetes集群包管理解决方案 Helm](docs/cloud/kubernetes/kubernetes_helm.md) * [Kubernetes原生配置管理利器 kustomize](docs/cloud/kubernetes/kubernetes_kustomize.md) * [kubernetes集群网络解决方案 flannel](docs/cloud/kubernetes/kubernetes_flannel.md) * [kubernetes集群网络解决方案 calico](docs/cloud/kubernetes/kubernetes_calico.md) * [kubernetes集群 underlay 网络方案 hybridnet](docs/cloud/kubernetes/kubernetes_hybridnet.md) * [kubernetes版本双栈协议（IPv4&IPv6）集群部署](docs/cloud/kubernetes/kubernetes_ipv4_and_ipv6.md) * [Rancher容器云管理平台监控](docs/cloud/kubernetes/kubernetes_rancher.md) * [使用kubeconfig管理多集群方法控](docs/cloud/kubernetes/kubernetes_kubeconfig.md) * [karmada实现k8s集群联邦](docs/cloud/kubernetes/kubernetes_karamada.md) * [安装kubesphere使用](docs/cloud/kubernetes/kubernetes_kubesphere.md) * [阿里云容器服务ACK](docs/cloud/kubernetes/kubernetes_ack.md) * [基于kubernetes集群构建大中型企业CICD应用平台](docs/cloud/kubernetes/kubernetes_devops.md) * [基于KubeSphere实现DevOps](docs/cloud/kubernetes/kubernetes_kubesphere_devops.md.md) * [云原生多云持续交付GitOps](docs/cloud/kubernetes/kubernetes_gitops.md) * [kubernetes集群备份与恢复管理利器Velero](docs/cloud/kubernetes/kubernetes_velero.md) * [kubernetes集群舰队管理Kurator](docs/cloud/kubernetes/kubernetes_kurator.md) * [Serverless之OpenFaaS函数即服务](docs/cloud/kubernetes/kubernetes_openfass.md) * [Flink基于Kubernetes部署](docs/cloud/kubernetes/kubernetes_flink.md) * [大数据HDFS分布式文件系统搭建](docs/cloud/kubernetes/kubernetes_hdfs.md) * [Spark与Kubernetes整合](docs/cloud/kubernetes/kubernetes_spark.md) * [源监控](docs/cloud/kubernetes/kubernetes_ui.md) * [情感] * [情商](docs/emotion/eq.md) * [工作心得](docs/emotion/workheard.md) * [看书](docs/emotion/lookbook.md) * [自律](docs/emotion/selfdiscipline.md) * [为什么活着失败](docs/emotion/livefail.md) * [情绪管理](docs/emotion/emotion.md) * [所有的失去，都会以另一种方式归来](docs/emotion/losecome.md) * [人生有这三种好心态](docs/emotion/threeheart.md) * [余生，学会一个人走，不管有没有人陪](docs/emotion/onepath.md) * [往后余生还很精彩，别被熬夜拖垮了](docs/emotion/twopath.md) * [心态好的人，一辈子都好](docs/emotion/lifetime.md) ================================================ FILE: docs/CNAME ================================================ burningmyself.cn ================================================ FILE: docs/cloud/apprelease.md ================================================ # 应用（Application）部署容器化演进之路 # 一、应用程序部署痛点 ## 1.1 应用程序部署流程 **举例：部署一个JAVA编程语言开发的Web应用，以War包放入Tomcat方式部署。** - 部署过程如下： - 服务器配置运行环境:JAVA代码运行环境，例如JDK或JRE - 服务器上安装Tomcat web中间件，用于运行War包 - 把JAVA Web对应的War包放置于Tomcat对应目录 - 在服务器上启动Tomcat应用 - 可选：涉及数据库（MySQL）或缓存系统(Redis)等都需要逐一部署。 ## 1.2 应用程序扩缩容 - 涉及多台服务器部署相同的上述环境 - 痛点：上述环境部署要重新实施一遍，较浪费人力与物力成本 ## 1.3 应用程序多环境部署 - 环境：本地测试环境、预发布环境、生产环境 - 在本地测试环境运行没有问题，但在预发布环境中出现了问题，甚至上面2种环境都没有问题，到了生产环境就有问题了。 - 需求：一次成功，可到处运行。 # 二、 计算资源应用演进过程 ## 2.1 使用物理服务器痛点 ![image-20220118161531257](../img/apprelease/image-20220118161531257.png) - 从物理服务器自身管理角度 - 物理服务器环境部署人力成本大，特别是在自动化手段不足的情况下，依靠人肉运维的方式解决。 - 当物理服务器出现宕机后，服务器重启时间过长，短则1-2分钟，长则3-5分钟，有背于服务器在线时长达到99.999999999%标准的要求 - 物理服务器在应用程序运行期间硬件出现故障，解决较麻烦 - 物理服务器计算资源不能有效调度使用，无法发挥其充足资源的优势 - 从物理服务器部署应用程序角度 - 物理服务器环境部署浪费时间，没有自动化运维手段，时间是成倍增加的 - 在物理服务器上进行应用程序配置变更，需要重新实施前述步骤 ## 2.2 使用虚拟机优点与缺点 ![image-20220118161210084](../img/apprelease/image-20220118161210084.png) ### 2.2.1 使用虚拟机优秀点 - 从虚拟机本身管理角度 - 虚拟机较物理服务器轻量，可借助虚拟机模板实现虚拟机快捷生成及应用 - 虚拟机中部署应用与物理服务器一样可控性强，且当虚拟机出现故障时，可直接使用新的虚拟机代替 - 在物理服务器中使用虚拟机可高效使用物理服务器的资源 - 虚拟机与物理服务器一样可达到良好的应用程序运行环境的隔离 - 从在虚拟机中部署应用程序角度 - 在虚拟机中部署应用，容易扩容及缩容实现 - 与物理服务器相比较，当部署应用程序的虚拟机出现宕机时，可以快速启动，时间通常可达秒级，10秒或20秒即可启动，应用程序可以继续提供服务 - 应用程序迁移方便 ### 2.2.2 使用虚拟机缺点 - 虚拟机管理软件本身占用物理服务器计算资源较多，例如:VMware Workstation Pro就会占用物理服务器大量资源，所以一般在企业应用中使用KVM虚拟机较多。 - 虚拟机底层硬件消耗物理服务器资源较大，例如：虚拟机操作系统硬盘，会直接占用大量物理服务器硬盘空间 - 相较于容器技术，虚拟机启动时间过长，容器启动可按毫秒级计算 - 虚拟机对物理服务器硬件资源调用添加了调链条，存在浪费时间的现象，所以虚拟机性能弱于物理服务器 - 由于应用程序是直接部署在虚拟机硬盘上，应用程序迁移时，需要连同虚拟机硬盘中的操作系统一同迁移，会导致迁移文件过大，浪费更多的存储空间及时间消耗过长 ## 2.3 使用容器的优点与缺点 ![image-20220118161301903](../img/apprelease/image-20220118161301903.png) ### 2.3.1 使用容器的优点 - 不需要为容器安装操作系统，可以节约大量时间 - 不需要通过手动的方式在容器中部署应用程序的运行环境，直接部署应用就可以了 - 不需要管理容器网络，以自动调用的方式访问容器中应用提供的服务 - 方便分享与构建应用容器，一次构建，到处运行 - 毫秒级启动 - 容器可直接使用物理服务器硬件资源，物理服务器硬件资源利用率高，性能较好。 ### 2.3.2 使用容器的缺点 对于对使用物理服务器、虚拟机已成为习惯的小伙伴来说，容器化可控性不强，最直观的就是对容器管理访问，总想按物理服务器或虚拟机的方式去管理它，其实容器与物理服务器、虚拟机管理方式上有着本质的区别的，最好不要管理。 # 三、 What is a Container? ## 3.1 容器定义 ![image-20220118161407959](../img/apprelease/image-20220118161407959.png) - 虚拟机 - 采用虚拟化技术手段实现物理服务器计算资源打包的方式，为应用程序提供类物理服务器运行环境 - 能够实现应用程序与应用程序之间的隔离 - 使用自动化技术部署应用程序及应用程序迁移较方便 - 可横向扩展 - 容器 - 容器是轻量级物理服务器计算资源的打包方式，即轻量级虚拟机，为应用程序提供类虚拟机运行环境。 - 可在物理服务器中实现高密度部署 - 容器与虚拟机对比 | 对比属性 | 容器（Container） | 虚拟机（VM） | | ---------------------- | ----------------- | ------------------ | | 隔离性 | 基于进程隔离 | 提供资源的完全隔离 | | 启动时间 | 毫秒级或秒级 | 秒级或分钟级 | | 内核 | 共用宿主机内核 | 使用独立内核 | | 占用资源 | MB级 | GB级 | | 系统支持容量（同级别） | 支持上千个容器 | 几十台虚拟机 | ## 3.2 容器功能 - 安装容器管理工具，例如Docker,Containerd等，把应用以容器化的方式运行 - 应用在自己的容器中运行，实现应用程序间隔离 - 应用程序运行的容器可以生成应用程序模板文件，即容器镜像（Image），其不可变，即为云原生代表技术基础设施不可变,且可以在其它的物理服务器中运行。 ## 3.3 容器解决了什么问题 - 快速交付和部署应用 (镜像与容器) - 资源的高效利用和隔离 (在物理机上实现高密度部署) - 便捷迁移和扩缩容(一次构建，多处运行) # 四、使用容器步骤 - 安装容器管理工具 - Docker (Docker公司) - Containerd (2017年docker捐给CNCF云原生计算基金会) - Pouch (阿里云) - 搜索/下载容器镜像(Image) - 使用容器镜像生成容器(容器镜像中的应用程序启动) - 终端用户(互联网用户或其它应用程序)访问 - 迁移部署(可直接把正在运行的容器打包成新的容器镜像，在其它主机上运行即可。) ================================================ FILE: docs/cloud/compute.md ================================================ # 云计算 # 一、计算资源使用方式 ## 1.1 主机资源使用方式 在云计算出现之前，常用的主机资源使用方式有： - 自己购买物理机 - IDC托管物理机 - IDC租用物理机 - 虚拟机 - 虚拟主机 ## 1.2 传统资源管理方式 ### 1.2.1 资源方面 - 初期投入/后期维护成本高 - 后期资源闲置浪费 ### 1.2.2 人力方面 - 纯手工操作，自动化能力差 - 技术水平限制，资源分配不合理 ### 1.2.3 最终效果 - 人力物力成本大 - 资源利用率低 # 二、为什么要用云计算 ## 2.1 对提供商而言 - 海量资源动态管理 - 资源灵活调配 - 资源高效率使用 - 技术团队高效使用 ## 2.2 对客户而言 - 使用方式多：通过网络访问，服务无处不在 - 投入成本低：按需使用的自助服务，资源可以弹性伸缩 # 三、云计算历史 2006年 Google提出"云计算"概念 2009年 美日韩将其纳入政府议程 2010年 中国将其纳入战略性产业，云计算开始在中国进入迅速发展期 2013年 政府工信部发布基于云计算的政务平台设计指南 2015年 云计算脱离争论不休和宣扬阶段，开始进入落地实施阶段 # 四、云计算定义 ## 4.1 从表现形式定义 - 底层由物理硬件构建出一个环境，在这个环境上运行一个操作系统，对终端用户而言，当我们需要用到一个操作系统或应用实现特殊功能时，它只需要向CloudOS提出申请而就能够立即申请获取一个对应的请求环境，这个环境我们可以随时终止，开启等功能。 - 对于用户而言，无需关心它所需要的计算能力从哪里来，有别于传统使用计算机操作系统的状况(看得见、摸得着) ## 4.2 从具体应用定义 - 云计算是一种资源交付的模式，即打包资源给客户使用。 - 它的特点是：基于网络、按需付费、弹性拓展。 - 云服务提供商基于有效的网络通信对所有资源进行统一管理，客户对使用的计算资源按需付费，计算资源使用过程中支持弹性拓展，客户只需投入很少的管理工作就可以高效率的使用计算资源。 # 五、云计算实现方式 ![image-20220110212726445](../img/compute/image-20220110212726445.png) ## 5.1 传统/私有方式 优点：所有事情都亲自做，可控 缺点：用户成本比较高，要求自身技术水平高 典型软件：传统物理主机 ## 5.2 Iaas IaaS（基础设施即服务） - 优点：底层硬件到操作系统，都不需要用户操心，省事，可以集中精力做业务项目。 - 缺点：服务商提供的东西，非自己自由定制，所以不可控 - 典型软件：OpenStack，CloudStack ## 5.3 PaaS PaaS（平台即服务） Management 是 云计算实现的一种方式，因为包含众多组件，所以也有人称之为Cloud OS - 优点：我不会运维，我只会开发，底层到运行环境，都不需要用户操心，省事，可以集中精力做应用项目 - 缺点：服务商提供的东西，定制太强，不灵活，只适用于特殊的应用项目， - 典型软件：Docker、Rocket、Openshift... ## 5.4 SaaS SaaS（软件即服务） - 优点：所有东西都由服务商提供，自己只需要花钱使用就行了，对于广大(大中小)企业来说，SaaS是采用先进技术实施信息化的最好途径。例如：企业邮箱服务，财务软件云服务 - 缺点：对客户来说，所有东西都不可控，安全不安全，看情况 # 六、云计算应用分类 ![image-20220110213717770](../img/compute/image-20220110213717770.png) ## 6.1 公有云 - 普遍性 - 用户按需使用，成本低廉，管理方面。 - 用户的数据保存在公有云的提供商那里，从技术上来讲，数据安全是没有办法保证的，这能从业务层面上来看待。 - 比如：银行不用公有云 - 举例：亚马逊、阿里云、openstack ## 6.2 私有云 - 专用性 - 所有资源都自己提供，安全有保障 - 技术/人力/业务成本高昂，资源利用效率低。 - 举例：vmware、企业云。。。 ## 6.3 混合云 - 协调性 - 核心业务用私有云，临时需求/轻量级业务需求使用公有云 - 成本的最优使用效率 # 七、虚拟化与云计算 ![image-20220110214040474](../img/compute/image-20220110214040474.png) ![image-20220110214100170](../img/compute/image-20220110214100170.png) ## 7.1 虚拟化 - 虚拟化是一种技术，它的目的在于提高资源的使用率，并将底层硬件和上层的应用软件进行隔离，使得上层软件及应用计算变得更加弹性可控。最终达到有限成本的高价值。 - 默认情况下，虚拟化技术默认并不对外提供使用抽象的上层应用软件服务组件，一个没有被服务化的虚拟化环境只能被称为"资源池"，只有内部管理人员才可以操作。 ## 7.2 云计算 云计算是以虚拟化技术为核心技术和基础，面向服务架构(SOA)的一种实现,将虚拟化环境"资源池"隐藏起来，将其上层 应用软件形成丰富的云管理接口，达到所有人自由使用所有资源的一种现象，它是一种资源使用模式的变革。 > 虚拟化是一种技术，云计算是一种计算资源交付模式。 ================================================ FILE: docs/cloud/containerlinux.md ================================================ # 容器技术所涉及Linux内核关键技术 # 一、容器技术前世今生 ## 1.1 1979年 — chroot - 容器技术的概念可以追溯到1979年的UNIX chroot。 - 它是一套“UNIX操作系统”系统，旨在将其root目录及其它子目录变更至文件系统内的新位置，且只接受特定进程的访问。 - 这项功能的设计目的在于为每个进程提供一套隔离化磁盘空间。 - 1982年其被添加至BSD当中。 ## 1.2 2000年 — FreeBSD Jails - FreeBSD Jails是由Derrick T. Woolworth于2000年在FreeBSD研发协会中构建而成的早期容器技术之一。 - 这是一套“操作系统”系统，与chroot的定位类似，不过其中包含有其它进程沙箱机制以对文件系统、用户及网络等资源进行隔离。 - 通过这种方式，它能够为每个Jail、定制化软件安装包乃至配置方案等提供一个对应的IP地址。 ## 1.3 2001年 — Linux VServer - Linux VServer属于另一种jail机制，其能够被用于保护计算机系统之上各分区资源的安全(包括文件系统、CPU时间、网络地址以及内存等)。 - 每个分区被称为一套安全背景(security context)，而其中的虚拟化系统则被称为一套虚拟私有服务器。 ## 1.4 2004年 — Solaris容器 - Solaris容器诞生之时面向x86与SPARC系统架构，其最初亮相于2004年2月的Solaris 10 Build 51 beta当中，随后于2005年正式登陆Solaris 10的完整版本。 - Solaris容器相当于将系统资源控制与由分区提供的边界加以结合。各分区立足于单一操作系统实例之内以完全隔离的虚拟服务器形式运行。 ## 1.5 2005年 — OpenVZ - OpenVZ与Solaris容器非常相似，且使用安装有补丁的Linux内核以实现虚拟化、隔离能力、资源管理以及检查点交付。 - 每套OpenVZ容器拥有一套隔离化文件系统、用户与用户群组、一套进程树、网络、设备以及IPC对象。 ## 1.6 2006年 — Process容器 - Process容器于2006年由谷歌公司推出，旨在对一整套进程集合中的资源使用量(包括CPU、内存、磁盘I/O以及网络等等)加以限制、分配与隔离。 - 此后其被更名为Control Groups(即控制组)，从而避免其中的“容器”字眼与Linux内核2.6.24中的另一术语出现冲突。这表明了谷歌公司率先重视容器技术的敏锐眼光以及为其做出的突出贡献。 ## 1.7 2007年 — Control Groups Control Groups也就是谷歌实现的cgroups，其于2007年被添加至Linux内核当中。 ## 1.8 2008年 — LXC - LXC指代的是Linux Containers - 是第一套完整的Linux容器管理实现方案。 - 其功能通过cgroups以及Linux namespaces实现。 - LXC通过liblxc库进行交付，并提供可与Python3、Python2、Lua、Go、Ruby以及Haskell等语言相对接的API。 - 相较于其它容器技术，LXC能够在无需任何额外补丁的前提下运行在原版Linux内核之上。 ## 1.9 2011年 — Warden - Warden由CloudFoundry公司于2011年所建立，其利用LXC作为初始阶段，随后又将其替换为自家实现方案。 - 与LXC不同，Warden并不会与Linux紧密耦合。相反，其能够运行在任意能够提供多种隔离环境方式的操作系统之上。Warden以后台进程方式运行并提供API以实现容器管理。 ## 1.10 2013年 — LMCTFY - Lmctfy代表的是“Let Me Contain That For You(帮你实现容器化)”。它其实属于谷歌容器技术堆栈的开源版本，负责提供Linux应用程序容器。谷歌公司在该项目的起步阶段宣称其能够提供值得信赖的性能表现、高资源利用率、共享资源机制、充裕的发展空间以及趋近于零的额外资源消耗。 - 2013年10月lmctfy的首个版本正式推出，谷歌公司在2015年决定将lmctfy的核心概念与抽象机制转化为libcontainer。在失去了主干之后，如今lmctfy已经失去一切积极的发展势头。 　　Libcontainer项目最初由Docker公司建立，如今已经被归入开放容器基金会的管辖范畴。 ## 1.11 2013年-Docker - 在2013年Docker刚发布的时候，它是一款基于LXC的开源容器管理引擎。 - 把LXC复杂的容器创建与使用方式简化为Docker自己的一套命令体系。 - 随着Docker的不断发展，它开始有了更为远大的目标，那就是反向定义容器的实现标准，将底层实现都抽象化到Libcontainer的接口。这就意味着，底层容器的实现方式变成了一种可变的方案，无论是使用namespace、cgroups技术抑或是使用systemd等其他方案，只要实现了Libcontainer定义的一组接口，Docker都可以运行。这也为Docker实现全面的跨平台带来了可能。 # 二、NameSpace ## 2.1 NameSpace介绍 - 很多编程语言都包含了命名空间的概念，我们可以认为命名空间是一种封装，封装本身实际上实现了代码的隔离 - 在操作系统中命名空间命名空间提供的是系统资源的隔离，其中系统资源包括了：进程、网络、文件系统...... - 实际上linux系统实现命名空间主要目的之一就是为了实现轻量级虚拟化服务，也就是我们说的容器，在同一个命名空间下的进程可以感知彼此的变化，而对其他命名空间的进程一无所知，这样就可以让容器中的进程产生一个错觉，仿佛它自己置身于一个独立的系统环境当中，以此达到独立和隔离的目的。 ## 2.2 Linux系统中NameSpace分类 | 命名空间 | 描述 | 作用 | 备注 | | :----------: | :--------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | 进程命名空间 | 隔离进程ID | Linux通过命名空间管理进程号，同一个进程，在不同的命名空间进程号不同 | 进程命名空间是一个父子结构，子空间对于父空间可见 | | 网络命名空间 | 隔离网络设备、协议栈、端口等 | 通过网络命名空间，实现网络隔离 | docker采用虚拟网络设备，将不同命名空间的网络设备连接到一起 | | IPC命名空间 | 隔离进程间通信 | 进程间交互方法 | PID命名空间和IPC命名空间可以组合起来用，同一个IPC名字空间内的进程可以彼此看见，允许进行交互，不同空间进程无法交互 | | 挂载命名空间 | 隔离挂载点 | 隔离文件目录 | 进程运行时可以将挂载点与系统分离，使用这个功能时，我们可以达到 chroot 的功能，而在安全性方面比 chroot 更高 | | UTS命名空间 | 隔离Hostname和NIS域名 | 让容器拥有独立的主机名和域名，从而让容器看起来像个独立的主机 | 目的是独立出主机名和网络信息服务（NIS） | | 用户命名空间 | 隔离用户和group ID | 每个容器内上的用户跟宿主主机上不在一个命名空间 | 同进程 ID 一样，用户 ID 和组 ID 在命名空间内外是不一样的，并且在不同命名空间内可以存在相同的 ID | ## 2.3 NameSpace应用案例 > 以net namespace为例 - 在 Linux 中，网络命名空间可以被认为是隔离的拥有单独网络栈（网卡、路由转发表、iptables）的环境。网络命名空间经常用来隔离网络设备和服务，只有拥有同样网络命名空间的设备，才能看到彼此。 - 从逻辑上说，网络命名空间是网络栈的副本，拥有自己的网络设备、路由选择表、邻接表、Netfilter表、网络套接字、网络procfs条目、网络sysfs条目和其他网络资源。 - 从系统的角度来看，当通过clone()系统调用创建新进程时，传递标志CLONE_NEWNET将在新进程中创建一个全新的网络命名空间。 - 从用户的角度来看，我们只需使用工具ip（package is iproute2）来创建一个新的持久网络命名空间。 ![image-20220111132707215](../img/containerlinux/image-20220111132707215.png) ### 2.3.1 创建net命名空间 ~~~powershell 创建名称为msb的网络命名空间 # ip netns add msb ~~~ ~~~powershell 查看已创建的网络命名空间 # ip netns ls msb ~~~ ### 2.3.2 删除net命名空间 ~~~powershell 删除已创建的网络命名空间 # ip netns delete msb ~~~ ### 2.3.3 在net命名空间中执行命令 ~~~powershell 在网络命名空间中执行bash命令，如果想退出，需要使用exit # ip netns exec msb bash ~~~ ### 2.3.4 在net命令空间中执行查看网络连接(网卡)命令 ~~~powershell 在网络命名空间中查看网络命名空间中的网卡信息 # ip link 1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 ~~~ ~~~powershell 在Linux主机系统中查看 # ip netns exec msb ip link list 1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 ~~~ ### 2.3.5 退出当前的net命名空间 ~~~powershell 退出已进入的网络命名空间 # exit exit ~~~ ### 2.3.6 在net命名空间中执行多条命令 ~~~powershell 在网络命名空间中查看路由表 # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface ~~~ ~~~powershell 在网络命名空间中查看防火墙规则 # iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination ~~~ ### 2.3.7 创建虚拟网卡 > 同时创建一对虚拟网卡 ~~~powershell 创建虚拟网卡对 # ip link add veth0 type veth peer name veth1 ~~~ ~~~powershell 在物理机上查看 # ip a s ...... 10: veth1@veth0: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether de:44:f8:b7:12:65 brd ff:ff:ff:ff:ff:ff 11: veth0@veth1: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 46:5e:89:8c:cb:b3 brd ff:ff:ff:ff:ff:ff ~~~ ### 2.3.8 迁移虚拟网卡到命名空间中 >这两个网卡还都属于“default”或“global”命名空间，和物理网卡一样。把其中一个网卡转移到命名空间msb中。 ~~~powershell 把创建的veth1网卡添加到msb网络命名空间中 # ip link set veth1 netns msb ~~~ ~~~powershell 在Linux系统命令行查看网络命名空间中的网络 # ip netns exec msb ip link 1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 10: veth1@if11: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether de:44:f8:b7:12:65 brd ff:ff:ff:ff:ff:ff link-netnsid 0 ~~~ ### 2.3.9 命名空间中迁出虚拟网卡 ~~~powershell 在Linux系统命令行把虚拟网卡veth1从网络命名空间删除 # ip netns exec msb ip link delete veth1 ~~~ ~~~powershell 在Linux系统命令行查看结果 # ip netns exec msb ip link 1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 ~~~ ### 2.3.10 配置虚拟网卡IP地址 ~~~powershell 再次创建虚拟网卡，添加到msb网络命名空间，并设置IP地址 # ip link add veth0 type veth peer name veth1 # ip link set veth1 netns msb # ip netns exec msb ip addr add 192.168.50.2/24 dev veth1 ~~~ ~~~powershell 在Linux系统命令行查看网络状态 # ip netns exec msb ip addr 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 12: veth1@if13: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether fe:20:ac:a8:13:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.50.2/24 scope global veth1 valid_lft forever preferred_lft forever ~~~ ~~~powershell 启动虚拟网卡,veth1与lo全部要启动 # ip netns exec msb ip link set veth1 up # ip netns exec msb ip link set lo up ~~~ ~~~powershell 为物理机veth0添加IP地址 # ip a s ...... 15: veth0@if14: mtu 1500 qdisc noop state DOWN group defau lt qlen 1000 link/ether 2e:b4:40:c8:73:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 ~~~ ~~~powershell # ip addr add 192.168.50.3/24 dev veth0 # ip a s veth0 15: veth0@if14: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 2e:b4:40:c8:73:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.50.3/24 scope global veth0 valid_lft forever preferred_lft forever ~~~ ~~~powershell # ip link set veth0 up ~~~ ~~~powershell 在宿主机上ping msb中的veth1 # ping 192.168.50.2 PING 192.168.50.2 (192.168.50.2) 56(84) bytes of data. 64 bytes from 192.168.50.2: icmp_seq=1 ttl=64 time=0.102 ms 64 bytes from 192.168.50.2: icmp_seq=2 ttl=64 time=0.068 ms 64 bytes from 192.168.50.2: icmp_seq=3 ttl=64 time=0.068 ms ~~~ ~~~powershell 在msb中的veth1 ping 宿主机上veth0 # ip netns exec msb ping 192.168.50.3 PING 192.168.50.3 (192.168.50.3) 56(84) bytes of data. 64 bytes from 192.168.50.3: icmp_seq=1 ttl=64 time=0.053 ms 64 bytes from 192.168.50.3: icmp_seq=2 ttl=64 time=0.031 ms 64 bytes from 192.168.50.3: icmp_seq=3 ttl=64 time=0.029 ms ~~~ ~~~powershell 如果需要访问本机的其它网段，可手动添加如下默认路由条目。 # ip netns exec msb ip route add default via 192.168.50.3 ~~~ > 关于如何ping通外网主机，可设置路由转发完成。 # 三、CGroups ## 3.1 CGroups介绍 - Control groups(cgroups) 控制组 - linux内核提供的可以限制、记录、隔离进程组所使用的物理资源的机制。为容器而生，没有cgroups就没有今天的容器技术。 ![image-20220112182824405](../img/containerlinux/image-20220112182824405.png) ## 3.2 CGroups功能 - 资源限制（Resource Limitation）：cgroups 可以对进程组使用的资源总额进行限制。如设定应用运行时使用内存的上限，一旦超过这个配额就发出 OOM（Out of Memory）。 - 优先级分配（Prioritization）：通过分配的 CPU 时间片数量及硬盘 IO 带宽大小，实际上就相当于控制了进程运行的优先级。 - 资源统计（Accounting）： cgroups 可以统计系统的资源使用量，如 CPU 使用时长、内存用量等等，这个功能非常适用于计费。 - 进程控制（Control）：cgroups 可以对进程组执行挂起、恢复等操作。 ## 3.3 CGroups应用案例 ### 3.3.1 安装及开启服务 ~~~shell [root@localhost ~]# yum -y install libcgroup [root@localhost ~]# systemctl start cgconfig.service [root@localhost ~]# systemctl enable cgconfig.service ~~~ ### 3.3.2 限制进程使用CPU #### 3.3.2.1 查看cpu shares ~~~powershell 查看资源限制子系统 [root@localhost ~]# lssubsys cpuset cpu,cpuacct memory devices freezer net_cls,net_prio blkio perf_event hugetlb pids 查看子系统配置文件所在位置 [root@localhost ~]# ls /sys/fs/cgroup/ blkio cpuacct cpuset freezer memory net_cls,net_prio perf_event systemd cpu cpu,cpuacct devices hugetlb net_cls net_prio pids [root@localhost ~]# ls /sys/fs/cgroup/cpu cgroup.clone_children cpuacct.stat cpu.cfs_quota_us cpu.stat cgroup.event_control cpuacct.usage cpu.rt_period_us notify_on_release cgroup.procs cpuacct.usage_percpu cpu.rt_runtime_us release_agent cgroup.sane_behavior cpu.cfs_period_us cpu.shares tasks 查看CPU时间分片，用于保证分组所得到的CPU分片总量。 [root@localhost ~]# cat /sys/fs/cgroup/cpu/cpu.shares 1024 ~~~ #### 3.3.2.2 使用CPU子系统创建2个group分组 ~~~shell [root@localhost ~]# vim /etc/cgconfig.conf group lesscpu { cpu{ cpu.shares=200; } } group morecpu { cpu{ cpu.shares=800; } } [root@localhost ~]# systemctl restart cgconfig ~~~ 准备一个脚本 ~~~powershell #!/bin/bash a=1 while true do a=$[$a+1] done ~~~ 将将要运行的应用程序分配到指定分组(**请使用单CPU机器,三个终端验证**) ~~~shell 终端1# cgexec -g cpu:lesscpu sh /tmp/1.sh 终端2# cgexec -g cpu:morecpu sh /tmp/1.sh 终端3# top ~~~ **PS: 如果主机有多CPU，为了验证效果，可以进行如下操作** ~~~shell # lscpu # echo 0 > /sys/devices/system/cpu/cpu0/online # echo 1 > /sys/devices/system/cpu/cpu1/online ~~~ ================================================ FILE: docs/cloud/docker/docker_compose.md ================================================ # Docker容器服务编排利器 Docker Compose应用实战 # 一、使用Docker Compose必要性及定义 用容器运行一个服务,需要使用`docker run`命令。但如果我要运行多个服务呢? 假设我要运行一个web服务,还要运行一个db服务,那么是用一个容器运行,还是用多个容器运行呢? 一个容器运行多个服务会造成镜像的复杂度提高,**docker倾向于一个容器运行一个应用**。 那么复杂的架构就会需要很多的容器,并且需要它们之间有关联(容器之间的依赖和连接)就更复杂了。 这个复杂的问题需要解决,这就涉及到了**==容器编排==**的问题了。 - Compose - 编排 - 是对多个容器进行启动和管理的方法 - 例如：LNMT，先启动MySQL,再启动Tomcat，最后启动Nginx - 服务架构的演进 - 单体服务架构 - 分布式服务架构 - 微服务架构 - 超微服务架构 - 容器编排工具 - docker machine - 在虚拟机中部署docker容器引擎的工具 - docker compose - 是一个用于定义和运行多容器Docker的应用程序工具 - docker swarm - 是Docker Host主机批量管理及资源调度管理工具 - mesos+marathon - mesos 对计算机计算资源进行管理和调度 - marathon 服务发现及负载均衡的功能 - kubernetes - google开源的容器编排工具 # 二、Docker Compose应用参考资料 - 网址 - https://docs.docker.com/compose/ ![image-20220215214445753](../../img/docker_compose/image-20220215214445753.png) - yaml格式 - https://yaml.org/ # 三、Docker Compose应用最佳实践步骤 ## 3.1 概念 - 工程（project） - 服务 (Service) - 容器 (Container) ## 3.2 步骤 1.定义应用的Dockerfile文件，为了anywhere进行构建。 2.使用docker-compose.yaml定义一套服务，这套服务可以一起在一个隔离环境中运行。 3.使用docker-compose up就可以启动整套服务。 # 四、Docker Compose安装 ![image-20220215215146472](../../img/docker_compose/image-20220215215146472.png) ![image-20220215215216047](../../img/docker_compose/image-20220215215216047.png) ![image-20220215215239151](../../img/docker_compose/image-20220215215239151.png) ![image-20220215215359944](../../img/docker_compose/image-20220215215359944.png) ~~~powershell # wget https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 ~~~ ~~~powershell # mv docker-compose-linux-x86_64 /usr/bin/docker-compose ~~~ ~~~powershell # chmod +x /usr/bin/docker-compose ~~~ ~~~powershell # docker-compose version Docker Compose version v2.2.3 ~~~ # 五、Docker Compose应用案例 > 运行Python语言开发的网站 ## 5.1 网站文件准备 ~~~powershell # mkdir flaskproject [root@localhost ~]# cd flaskproject/ [root@localhost flaskproject]# ~~~ ~~~powershell [root@localhost flaskproject]# vim app.py [root@localhost flaskproject]# cat app.py import time import redis from flask import Flask app = Flask(__name__) cache = redis.Redis(host='redis', port=6379) def get_hit_count(): retries = 5 while True: try: return cache.incr('hits') except redis.exceptions.ConnectionError as exc: if retries == 0: raise exc retries -= 1 time.sleep(0.5) @app.route('/') def hello(): count = get_hit_count() return 'Hello World! I have been seen {} times.\n'.format(count) ~~~ ~~~powershell [root@localhost flaskproject]# vim requirements.txt [root@localhost flaskproject]# cat requirements.txt flask redis ~~~ ## 5.2 Dockerfile文件准备 ~~~powershell [root@localhost flaskproject]# vim Dockerfile [root@localhost flaskproject]# cat Dockerfile FROM python:3.7-alpine WORKDIR /code ENV FLASK_APP app.py ENV FLASK_RUN_HOST 0.0.0.0 RUN apk add --no-cache gcc musl-dev linux-headers COPY requirements.txt requirements.txt RUN pip install -r requirements.txt COPY . . CMD ["flask", "run"] ~~~ ## 5.3 Compose文件准备 ~~~powershell [root@localhost flaskproject]# vim docker-compose.yaml [root@localhost flaskproject]# cat docker-compose.yaml version: '3' services: web: build: . ports: - "5000:5000" redis: image: "redis:alpine" ~~~ ## 5.4 使用docker-compose up启动容器 ~~~powershell [root@localhost flaskproject]# ls app.py docker-compose.yaml Dockerfile requirements.txt ~~~ ~~~powershell [root@localhost flaskproject]# docker-compose up ~~~ ~~~powershell 输出： [+] Running 7/7 ⠿ redis Pulled 15.8s ⠿ 59bf1c3509f3 Pull complete 2.9s ⠿ 719adce26c52 Pull complete 3.0s ⠿ b8f35e378c31 Pull complete 5.8s ⠿ d034517f789c Pull complete 6.5s ⠿ 3772d4d76753 Pull complete 6.6s ⠿ 211a7f52febb Pull complete 6.8s Sending build context to Docker daemon 714B Step 1/9 : FROM python:3.7-alpine 3.7-alpine: Pulling from library/python 59bf1c3509f3: Already exists 07a400e93df3: Already exists bdabb07397e1: Already exists cd0af01c7b70: Already exists d0f18e022200: Already exists Digest: sha256:5a776e3b5336827faf7a1c3a191b73b5b2eef4cdcfe8b94f59b79cb749a2b5d8 Status: Downloaded newer image for python:3.7-alpine ---> e72b511ad78e Step 2/9 : WORKDIR /code ---> Running in 2b9d07bef719 Removing intermediate container 2b9d07bef719 ---> 7d39e96fadf1 Step 3/9 : ENV FLASK_APP app.py ---> Running in 9bcb28bd632a Removing intermediate container 9bcb28bd632a ---> 79f656a616d5 Step 4/9 : ENV FLASK_RUN_HOST 0.0.0.0 ---> Running in 8470c2dbd6c2 Removing intermediate container 8470c2dbd6c2 ---> e212ba688fcd Step 5/9 : RUN apk add --no-cache gcc musl-dev linux-headers ---> Running in 6e9ca0766bc8 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/13) Installing libgcc (10.3.1_git20211027-r0) (2/13) Installing libstdc++ (10.3.1_git20211027-r0) (3/13) Installing binutils (2.37-r3) (4/13) Installing libgomp (10.3.1_git20211027-r0) (5/13) Installing libatomic (10.3.1_git20211027-r0) (6/13) Installing libgphobos (10.3.1_git20211027-r0) (7/13) Installing gmp (6.2.1-r1) (8/13) Installing isl22 (0.22-r0) (9/13) Installing mpfr4 (4.1.0-r0) (10/13) Installing mpc1 (1.2.1-r0) (11/13) Installing gcc (10.3.1_git20211027-r0) (12/13) Installing linux-headers (5.10.41-r0) (13/13) Installing musl-dev (1.2.2-r7) Executing busybox-1.34.1-r3.trigger OK: 143 MiB in 49 packages Removing intermediate container 6e9ca0766bc8 ---> 273d4f04dfbc Step 6/9 : COPY requirements.txt requirements.txt ---> daf51c54e8ba Step 7/9 : RUN pip install -r requirements.txt ---> Running in 2aa2d30c5311 Collecting flask Downloading Flask-2.0.3-py3-none-any.whl (95 kB) Collecting redis Downloading redis-4.1.3-py3-none-any.whl (173 kB) Collecting Jinja2>=3.0 Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB) Collecting itsdangerous>=2.0 Downloading itsdangerous-2.0.1-py3-none-any.whl (18 kB) Collecting click>=7.1.2 Downloading click-8.0.3-py3-none-any.whl (97 kB) Collecting Werkzeug>=2.0 Downloading Werkzeug-2.0.3-py3-none-any.whl (289 kB) Collecting deprecated>=1.2.3 Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB) Collecting packaging>=20.4 Downloading packaging-21.3-py3-none-any.whl (40 kB) Collecting importlib-metadata>=1.0 Downloading importlib_metadata-4.11.1-py3-none-any.whl (17 kB) Collecting wrapt<2,>=1.10 Downloading wrapt-1.13.3-cp37-cp37m-musllinux_1_1_x86_64.whl (78 kB) Collecting typing-extensions>=3.6.4 Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) Collecting zipp>=0.5 Downloading zipp-3.7.0-py3-none-any.whl (5.3 kB) Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.0.1-cp37-cp37m-musllinux_1_1_x86_64.whl (30 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB) Installing collected packages: zipp, typing-extensions, wrapt, pyparsing, MarkupSafe, importlib-metadata, Werkzeug, packaging, Jinja2, itsdangerous, deprecated, click, redis, flask Successfully installed Jinja2-3.0.3 MarkupSafe-2.0.1 Werkzeug-2.0.3 click-8.0.3 deprecated-1.2.13 flask-2.0.3 importlib-metadata-4.11.1 itsdangerous-2.0.1 packaging-21.3 pyparsing-3.0.7 redis-4.1.3 typing-extensions-4.1.1 wrapt-1.13.3 zipp-3.7.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: You are using pip version 21.2.4; however, version 22.0.3 is available. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command. Removing intermediate container 2aa2d30c5311 ---> dd8f52b132f8 Step 8/9 : COPY . . ---> b36938a26cf5 Step 9/9 : CMD ["flask", "run"] ---> Running in 260cbfa02959 Removing intermediate container 260cbfa02959 ---> fa04dfec6ff2 Successfully built fa04dfec6ff2 Successfully tagged flaskproject_web:latest Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them [+] Running 3/3 ⠿ Network flaskproject_default Created 0.1s ⠿ Container flaskproject-redis-1 Created 0.1s ⠿ Container flaskproject-web-1 Created 0.1s Attaching to flaskproject-redis-1, flaskproject-web-1 flaskproject-redis-1 | 1:C 15 Feb 2022 14:14:21.696 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo flaskproject-redis-1 | 1:C 15 Feb 2022 14:14:21.696 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started flaskproject-redis-1 | 1:C 15 Feb 2022 14:14:21.696 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf flaskproject-redis-1 | 1:M 15 Feb 2022 14:14:21.697 * monotonic clock: POSIX clock_gettime flaskproject-redis-1 | 1:M 15 Feb 2022 14:14:21.698 * Running mode=standalone, port=6379. flaskproject-redis-1 | 1:M 15 Feb 2022 14:14:21.698 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. flaskproject-redis-1 | 1:M 15 Feb 2022 14:14:21.698 # Server initialized flaskproject-redis-1 | 1:M 15 Feb 2022 14:14:21.698 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. flaskproject-redis-1 | 1:M 15 Feb 2022 14:14:21.698 * Ready to accept connections flaskproject-web-1 | * Serving Flask app 'app.py' (lazy loading) flaskproject-web-1 | * Environment: production flaskproject-web-1 | WARNING: This is a development server. Do not use it in a production deployment. flaskproject-web-1 | Use a production WSGI server instead. flaskproject-web-1 | * Debug mode: off flaskproject-web-1 | * Running on all addresses. flaskproject-web-1 | WARNING: This is a development server. Do not use it in a production deployment. flaskproject-web-1 | * Running on http://172.18.0.2:5000/ (Press CTRL+C to quit) ~~~ ## 5.5 访问 ![image-20220215221648195](../../img/docker_compose/image-20220215221648195.png) ================================================ FILE: docs/cloud/docker/docker_container.md ================================================ # 轻量级或工业级容器管理工具 Containerd # 一、Containerd介绍 ## 1.0 前言 - 早在2016年3月，Docker 1.11的Docker Engine里就包含了containerd，而现在则是把containerd从Docker Engine里彻底剥离出来，作为一个独立的开源项目独立发展，目标是提供一个更加开放、稳定的容器运行基础设施。和原先包含在Docker Engine里containerd相比，独立的containerd将具有更多的功能，可以涵盖整个容器运行时管理的所有需求。 - containerd并不是直接面向最终用户的，而是主要用于集成到更上层的系统里，比如Swarm, Kubernetes, Mesos等容器编排系统。 - containerd以Daemon的形式运行在系统上，通过暴露底层的gRPC API，上层系统可以通过这些API管理机器上的容器。 - 每个containerd只负责一台机器，Pull镜像，对容器的操作（启动、停止等），网络，存储都是由containerd完成。具体运行容器由runC负责，实际上只要是符合OCI规范的容器都可以支持。 - 对于容器编排服务来说，运行时只需要使用containerd+runC，更加轻量，容易管理。 - 独立之后containerd的特性演进可以和Docker Engine分开，专注容器运行时管理，可以更稳定。 ![image-20220218112209583](../../img/docker_container/image-20220218112209583.png) ![image-20220218221135407](../../img/docker_container/image-20220218221135407.png) ## 1.1 Containerd前世今生 2013年docker公司在推出docker产品后,由于其对全球技术产生了一定的影响力，Google公司明显感觉到自己公司内部所使用的Brog系统江湖地位受到的威胁，希望Docker公司能够与自己联合打造一款开源的容器运行时作为Docker核心依赖，但Docker公司拒绝了；接着Google公司联合RedHat、IBM等公司说服Docker公司把其容器核心技术libcontainer捐给中立社区(OCI,Open Container Intiative)，并更名为runC。 为了进一步遏制Docker在未来技术市场影响力，避免在容器市场上Docker一家独大，Google公司带领导RedHat、IBM等成立了CNCF(Cloud Native Computing Fundation)基金会，即云原生计算基金会。CNCF的目标很明确，既然在容器应用领域无法与Docker相抗衡，那就做Google更有经验的技术市场------大规模容器编排应用场景，Google公司把自己内部使用的Brog系统开源------Kubernetes，也就是我们今天所说的云原生技术生态。 2016年Docker公司推出了Docker Swarm，意在一统Docker生态，让Docker既可以实现容器应用管理，也可以实现大规模容器编排，经过近1年左右时间的市场验证后，发现在容器编排方面无法独立抗衡kubernetes,所以Docker公司于2017年正式宣布原生支持Kubernetes,至此，Docker在大规模容器编排应用市场败下阵来，但是Docker依然不甘心失败，把Docker核心依赖Containerd捐给了CNCF，依此说明Docker依旧是一个PaaS平台。 2020年CNCF基金会宣布Kubernetes 1.20版本将不再仅支持Docker容器管理工具，此事的起因主要也与Docker捐给CNCF基金会的Containerd有关，早期为了实现Kubernetes能够使用Docker实现容器管理，专门在Kubernetes组件中集成一个shim（垫片）技术，用来将Kubernetes容器运行时接口（CRI，Container Runntime Interface）调用翻译成Docker的API,这样就可以很好地使用Docker了，但是随着Kubernetes在全球技术市场的广泛应用，有更多的容器管理工具的出现，它们都想能够借助于Kubernetes被用户所使用，所以就提出标准化容器运行时接口，只要适配了这个接口就可以集成到Kubernetes生态当中，所以Kubernetes取消了对shim的维护，并且由于Containerd技术的成功，可以实现无缝对接Kubernetes，所以接下来Kubernetes容器运行时的主角是Containerd。 ## 1.2 Containerd架构 ### 1.2.1 架构图 Containerd设计的目的是为了嵌入到Kubernetes中使用，它是一个工业级的容器运行时，不提供给开发人员和终端用户直接使用，这样就避免了与Docker产生竞争，但事实上，Containerd已经实现大多数容器管理功能，例如：容器生命周期管理、容器镜像传输和管理、容器存储与网络管理等。 ![image-20220220093315501](../../img/docker_container/image-20220220093315501.png) - Containerd 采用标准的 C/S 架构 - 服务端通过 GRPC 协议提供稳定的 API - 客户端通过调用服务端的 API 进行高级的操作 - 为了实现解耦，Containerd 将不同的职责划分给不同的组件，每个组件就相当于一个子系统（subsystem）。连接不同子系统的组件被称为模块。 - Containerd 两大子系统为： - Bundle : 在 Containerd 中，Bundle 包含了配置、元数据和根文件系统数据，你可以理解为容器的文件系统。而 Bundle 子系统允许用户从镜像中提取和打包 Bundles。 - Runtime : Runtime 子系统用来执行 Bundles，比如创建容器。 其中，每一个子系统的行为都由一个或多个模块协作完成（架构图中的 Core 部分）。每一种类型的模块都以插件的形式集成到 Containerd 中，而且插件之间是相互依赖的。 例如，上图中的每一个长虚线的方框都表示一种类型的插件，包括 Service Plugin、Metadata Plugin、GC Plugin、Runtime Plugin 等，其中 Service Plugin 又会依赖 Metadata Plugin、GC Plugin 和 Runtime Plugin。每一个小方框都表示一个细分的插件，例如 Metadata Plugin 依赖 Containers Plugin、Content Plugin 等。 ### 1.2.2 常用插件 - **Content Plugin** : 提供对镜像中可寻址内容的访问，所有不可变的内容都被存储在这里。 - **Snapshot Plugin** : 用来管理容器镜像的文件系统快照。镜像中的每一个 layer 都会被解压成文件系统快照，类似于 Docker 中的 `graphdriver`。 - **Metrics** : 暴露各个组件的监控指标。 ![image-20220220093649732](../../img/docker_container/image-20220220093649732.png) ### 1.2.3 架构缩略图 Containerd 被分为三个大块：`Storage`、`Metadata` 和 `Runtime` ![image-20220220093958799](../../img/docker_container/image-20220220093958799.png) ### 1.2.4 与其它容器运行时工具性能对比 这是使用 bucketbench 对 Docker、crio 和 Containerd 的性能测试结果，包括启动、停止和删除容器，以比较它们所耗的时间： ![image-20220220095224783](../../img/docker_container/image-20220220095224783.png) 结论： Containerd 在各个方面都表现良好，总体性能优于 `Docker` 和 `crio` 。 # 二、Containerd安装 > 课程操作系统环境为centos7u6 ## 2.1 YUM方式安装 ### 2.1.1 获取YUM源 ~~~powershell 获取阿里云YUM源 # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell 查看YUM源中Containerd软件 # yum list | grep containerd containerd.io.x86_64 1.4.12-3.1.el7 docker-ce-stable ~~~ ### 2.1.2 使用yum命令安装 ~~~powershell 安装Containerd.io软件，即可安装Containerd # yum -y install containerd.io ~~~ ### 2.1.3 验证安装及启动服务 ~~~powershell 使用rpm -qa命令查看是否安装 # rpm -qa | grep containerd containerd.io-1.4.12-3.1.el7.x86_64 ~~~ ~~~powershell 设置containerd服务启动及开机自启动 # systemctl enable containerd # systemctl start containerd ~~~ ~~~powershell 查看containerd服务启动状态 # systemctl status containerd ● containerd.service - containerd container runtime Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled) Active: active (running) since 五 2022-02-18 11:38:30 CST; 9s ago 此行第二列及第三列表示其正在运行状态 Docs: https://containerd.io Process: 59437 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 59439 (containerd) Tasks: 7 Memory: 19.5M CGroup: /system.slice/containerd.service └─59439 /usr/bin/containerd ...... ~~~ ### 2.1.4 验证可用性 ~~~powershell 安装Containerd时ctr命令亦可使用，ctr命令主要用于管理容器及容器镜像等。 使用ctr命令查看Containerd客户端及服务端相关信息。 # ctr version Client: Version: 1.4.12 Revision: 7b11cfaabd73bb80907dd23182b9347b4245eb5d Go version: go1.16.10 Server: Version: 1.4.12 Revision: 7b11cfaabd73bb80907dd23182b9347b4245eb5d UUID: 3c4b142d-d91d-44a5-aae2-9673785d4b2c ~~~ ## 2.2 二进制方式安装 Containerd有两种安装包： * 第一种是`containerd-xxx`,这种包用于单机测试没问题，不包含runC，需要提前安装。 * 第二种是`cri-containerd-cni-xxxx`，包含runc和k8s里的所需要的相关文件。k8s集群里需要用到此包。虽然包含runC，但是依赖系统中的seccomp（安全计算模式，是一种限制容器调用系统资源的模式。） ### 2.2.1 获取安装包 ![image-20220218114231583](../../img/docker_container/image-20220218114231583.png) ![image-20220218114320745](../../img/docker_container/image-20220218114320745.png) ![image-20220218114353357](../../img/docker_container/image-20220218114353357.png) ![image-20220218123330897](../../img/docker_container/image-20220218123330897.png) ~~~powershell 下载Containerd安装包 # wget https://github.com/containerd/containerd/releases/download/v1.6.0/cri-containerd-cni-1.6.0-linux-amd64.tar.gz ~~~ ### 2.2.2 安装并测试可用性 #### 2.2.2.1 安装containerd ~~~powershell 查看已获取的安装包 # ls cri-containerd-cni-1.6.0-linux-amd64.tar.gz ~~~ ~~~powershell 解压已下载的软件包 # tar xf cri-containerd-cni-1.6.0-linux-amd64.tar.gz ~~~ ~~~powershell 查看解压后目录 # ls etc opt usr ~~~ ~~~powershell 查看etc目录，主要为containerd服务管理配置文件及cni虚拟网卡配置文件 # ls etc cni crictl.yaml systemd # ls etc/systemd/ system # ls etc/systemd/system/ containerd.service 查看opt目录，主要为gce环境中使用containerd配置文件及cni插件 # ls opt cni containerd # ls opt/containerd/ cluster # ls opt/containerd/cluster/ gce version # ls opt/containerd/cluster/gce cloud-init cni.template configure.sh env 查看usr目录，主要为containerd运行时文件，包含runc # ls usr local # ls usr/local/ bin sbin # ls usr/local/bin containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress crictl critest ctd-decoder ctr # ls usr/local/sbin runc ~~~ #### 2.2.2.2 查看containerd安装位置 ~~~powershell 查看containerd.service文件，了解containerd文件安装位置 # cat etc/systemd/system/containerd.service # Copyright The containerd Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd 查看此位置,把containerd二进制文件放置于此处即可完成安装。 Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=infinity # Comment TasksMax if your systemd version does not supports it. # Only systemd 226 and above support this version. TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target ~~~ #### 2.2.2.3 复制containerd运行时文件至系统 ~~~powershell 查看宿主机/usr/local/bin目录，里面没有任何内容。 # ls /usr/local/bin/ 查看解压后usr/local/bin目录，里面包含containerd运行时文件 # ls usr/ local # ls usr/local/ bin sbin # ls usr/local/bin/ containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress crictl critest ctd-decoder ctr 复制containerd文件至/usr/local/bin目录中，本次可仅复制containerd一个文件也可复制全部文件。 # cp usr/local/bin/containerd /usr/local/bin/ # ls /usr/local/bin/ containerd ~~~ #### 2.2.2.4 添加containerd.service文件至系统 ~~~powershell 查看解压后的etc/system目录 # ls etc cni crictl.yaml systemd # ls etc/systemd/ system # ls etc/systemd/system/ containerd.service 复制containerd服务管理配置文件至/usr/lib/systemd/system/目录中 # cp etc/systemd/system/containerd.service /usr/lib/systemd/system/containerd.service 查看复制后结果 # ls /usr/lib/systemd/system/containerd.service /usr/lib/systemd/system/containerd.service ~~~ #### 2.2.2.5 查看containerd使用帮助 ~~~powershell # containerd --help NAME: containerd - __ _ __ _________ ____ / /_____ _(_)___ ___ _________/ / / ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __ / / /__/ /_/ / / / / /_/ /_/ / / / / / __/ / / /_/ / \___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/ \__,_/ high performance container runtime USAGE: containerd [global options] command [command options] [arguments...] VERSION: v1.6.0 DESCRIPTION: containerd is a high performance container runtime whose daemon can be started by using this command. If none of the *config*, *publish*, or *help* commands are specified, the default action of the **containerd** command is to start the containerd daemon in the foreground. A default configuration is used if no TOML configuration is specified or located at the default file location. The *containerd config* command can be used to generate the default configuration for containerd. The output of that command can be used and modified as necessary as a custom configuration. COMMANDS: config information on the containerd config publish binary to publish events to containerd oci-hook provides a base for OCI runtime hooks to allow arguments to be injected. help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --config value, -c value path to the configuration file (default: "/etc/containerd/config.toml") --log-level value, -l value set the logging level [trace, debug, info, warn, error, fatal, panic] --address value, -a value address for containerd's GRPC server --root value containerd root directory --state value containerd state directory --help, -h show help --version, -v print the version ~~~ #### 2.2.2.6 生成containerd模块配置文件 ##### 2.2.2.6.1 生成默认模块配置文件 Containerd 的默认配置文件为 `/etc/containerd/config.toml`，可以使用`containerd config default > /etc/containerd/config.toml`命令创建一份模块配置文件 ~~~powershell 创建配置文件目录 # mkdir /etc/containerd ~~~ ~~~powershell 生成配置文件 # containerd config default > /etc/containerd/config.toml ~~~ ~~~powershell 查看配置文件 # cat /etc/containerd/config.toml disabled_plugins = [] imports = [] oom_score = 0 plugin_dir = "" required_plugins = [] root = "/var/lib/containerd" state = "/run/containerd" temp = "" version = 2 [cgroup] path = "" [debug] address = "" format = "" gid = 0 level = "" uid = 0 [grpc] address = "/run/containerd/containerd.sock" gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 tcp_address = "" tcp_tls_ca = "" tcp_tls_cert = "" tcp_tls_key = "" uid = 0 [metrics] address = "" grpc_histogram = false [plugins] [plugins."io.containerd.gc.v1.scheduler"] deletion_threshold = 0 mutation_threshold = 100 pause_threshold = 0.02 schedule_delay = "0s" startup_delay = "100ms" [plugins."io.containerd.grpc.v1.cri"] device_ownership_from_security_context = false disable_apparmor = false disable_cgroup = false disable_hugetlb_controller = true disable_proc_mount = false disable_tcp_service = true enable_selinux = false enable_tls_streaming = false enable_unprivileged_icmp = false enable_unprivileged_ports = false ignore_image_defined_volumes = false max_concurrent_downloads = 3 max_container_log_line_size = 16384 netns_mounts_under_state_dir = false restrict_oom_score_adj = false sandbox_image = "k8s.gcr.io/pause:3.6" 由于网络原因，此处被替换 selinux_category_range = 1024 stats_collect_period = 10 stream_idle_timeout = "4h0m0s" stream_server_address = "127.0.0.1" stream_server_port = "0" systemd_cgroup = false tolerate_missing_hugetlb_controller = true unset_seccomp_profile = "" [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" conf_template = "" ip_pref = "" max_conf_num = 1 [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" disable_snapshot_annotations = true discard_unpacked_layers = false ignore_rdt_not_enabled_errors = false no_pivot = false snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime] base_runtime_spec = "" cni_conf_dir = "" cni_max_conf_num = 0 container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_path = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] base_runtime_spec = "" cni_conf_dir = "" cni_max_conf_num = 0 container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_path = "" runtime_root = "" runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] BinaryName = "" CriuImagePath = "" CriuPath = "" CriuWorkPath = "" IoGid = 0 IoUid = 0 NoNewKeyring = false NoPivotRoot = false Root = "" ShimCgroup = "" SystemdCgroup = false [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime] base_runtime_spec = "" cni_conf_dir = "" cni_max_conf_num = 0 container_annotations = [] pod_annotations = [] privileged_without_host_devices = false runtime_engine = "" runtime_path = "" runtime_root = "" runtime_type = "" [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options] [plugins."io.containerd.grpc.v1.cri".image_decryption] key_model = "node" [plugins."io.containerd.grpc.v1.cri".registry] config_path = "" [plugins."io.containerd.grpc.v1.cri".registry.auths] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.headers] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins."io.containerd.internal.v1.opt"] path = "/opt/containerd" [plugins."io.containerd.internal.v1.restart"] interval = "10s" [plugins."io.containerd.internal.v1.tracing"] sampling_ratio = 1.0 service_name = "containerd" [plugins."io.containerd.metadata.v1.bolt"] content_sharing_policy = "shared" [plugins."io.containerd.monitor.v1.cgroups"] no_prometheus = false [plugins."io.containerd.runtime.v1.linux"] no_shim = false runtime = "runc" runtime_root = "" shim = "containerd-shim" shim_debug = false [plugins."io.containerd.runtime.v2.task"] platforms = ["linux/amd64"] sched_core = false [plugins."io.containerd.service.v1.diff-service"] default = ["walking"] [plugins."io.containerd.service.v1.tasks-service"] rdt_config_file = "" [plugins."io.containerd.snapshotter.v1.aufs"] root_path = "" [plugins."io.containerd.snapshotter.v1.btrfs"] root_path = "" [plugins."io.containerd.snapshotter.v1.devmapper"] async_remove = false base_image_size = "" discard_blocks = false fs_options = "" fs_type = "" pool_name = "" root_path = "" [plugins."io.containerd.snapshotter.v1.native"] root_path = "" [plugins."io.containerd.snapshotter.v1.overlayfs"] root_path = "" upperdir_label = false [plugins."io.containerd.snapshotter.v1.zfs"] root_path = "" [plugins."io.containerd.tracing.processor.v1.otlp"] endpoint = "" insecure = false protocol = "" [proxy_plugins] [stream_processors] [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"] accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar" [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"] accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"] args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"] env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"] path = "ctd-decoder" returns = "application/vnd.oci.image.layer.v1.tar+gzip" [timeouts] "io.containerd.timeout.bolt.open" = "0s" "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s" [ttrpc] address = "" gid = 0 uid = 0 ~~~ ##### 2.2.2.6.2 替换默认配置文件 但上述配置文件后期改动的地方较多，这里直接换成可单机使用也可k8s环境使用的配置文件并配置好镜像加速器。 ~~~powershell # vim /etc/containerd/config.toml # cat /etc/containerd/config.toml root = "/var/lib/containerd" state = "/run/containerd" oom_score = -999 [grpc] address = "/run/containerd/containerd.sock" uid = 0 gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 [debug] address = "" uid = 0 gid = 0 level = "" [metrics] address = "" grpc_histogram = false [cgroup] path = "" [plugins] [plugins.cgroups] no_prometheus = false [plugins.cri] stream_server_address = "127.0.0.1" stream_server_port = "0" enable_selinux = false sandbox_image = "easzlab/pause-amd64:3.2" stats_collect_period = 10 systemd_cgroup = false enable_tls_streaming = false max_container_log_line_size = 16384 [plugins.cri.containerd] snapshotter = "overlayfs" no_pivot = false [plugins.cri.containerd.default_runtime] runtime_type = "io.containerd.runtime.v1.linux" runtime_engine = "" runtime_root = "" [plugins.cri.containerd.untrusted_workload_runtime] runtime_type = "" runtime_engine = "" runtime_root = "" [plugins.cri.cni] bin_dir = "/opt/kube/bin" conf_dir = "/etc/cni/net.d" conf_template = "/etc/cni/net.d/10-default.conf" [plugins.cri.registry] [plugins.cri.registry.mirrors] [plugins.cri.registry.mirrors."docker.io"] endpoint = [ "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com" ] [plugins.cri.registry.mirrors."gcr.io"] endpoint = [ "https://gcr.mirrors.ustc.edu.cn" ] [plugins.cri.registry.mirrors."k8s.gcr.io"] endpoint = [ "https://gcr.mirrors.ustc.edu.cn/google-containers/" ] [plugins.cri.registry.mirrors."quay.io"] endpoint = [ "https://quay.mirrors.ustc.edu.cn" ] [plugins.cri.registry.mirrors."harbor.kubemsb.com"] 此处添加了本地容器镜像仓库 Harbor,做为本地容器镜像仓库。 endpoint = [ "http://harbor.kubemsb.com" ] [plugins.cri.x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins.diff-service] default = ["walking"] [plugins.linux] shim = "containerd-shim" runtime = "runc" runtime_root = "" no_shim = false shim_debug = false [plugins.opt] path = "/opt/containerd" [plugins.restart] interval = "10s" [plugins.scheduler] pause_threshold = 0.02 deletion_threshold = 0 mutation_threshold = 100 schedule_delay = "0s" startup_delay = "100ms" ~~~ #### 2.2.2.7 启动containerd服务并设置开机自启动 ~~~powershell # systemctl enable containerd Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /usr/lib/systemd/system/containerd.service. # systemctl start containerd ~~~ ~~~powershell # systemctl status containerd ● containerd.service - containerd container runtime Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled) Active: active (running) since 五 2022-02-18 13:02:37 CST; 7s ago Docs: https://containerd.io Process: 60383 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 60384 (containerd) Tasks: 8 Memory: 20.0M CGroup: /system.slice/containerd.service └─60384 /usr/local/bin/containerd ...... ~~~ #### 2.2.2.8 复制ctr命令至系统 ~~~powershell # ls usr/local/bin/ containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress crictl critest ctd-decoder ctr # cp usr/local/bin/ctr /usr/bin/ ~~~ #### 2.2.2.9 查看已安装containerd服务版本 ~~~powershell # ctr version Client: Version: v1.6.0 Revision: 39259a8f35919a0d02c9ecc2871ddd6ccf6a7c6e Go version: go1.17.2 Server: Version: v1.6.0 Revision: 39259a8f35919a0d02c9ecc2871ddd6ccf6a7c6e UUID: c1972cbe-884a-41b0-867f-f8a58c168e6d ~~~ #### 2.2.2.10 安装runC > 由于二进制包中提供的runC默认需要系统中安装seccomp支持，需要单独安装，且不同版本runC对seccomp版本要求一致，所以建议单独下载runC 二进制包进行安装，里面包含了seccomp模块支持。 ##### 2.2.2.10.1 获取runC ![image-20220218215925093](../../img/docker_container/image-20220218215925093.png) ![image-20220218215958943](../../img/docker_container/image-20220218215958943.png) ![image-20220218220023181](../../img/docker_container/image-20220218220023181.png) ![image-20220218220124773](../../img/docker_container/image-20220218220124773.png) ~~~powershell 使用wget下载 # wget https://github.com/opencontainers/runc/releases/download/v1.1.0/runc.amd64 ~~~ ##### 2.2.2.10.2 安装runC并验证安装结果 ~~~powershell 查看已下载文件 # ls runc.amd64 ~~~ ~~~powershell 安装runC # mv runc.amd64 /usr/sbin/runc ~~~ ~~~powershell 为runC添加可执行权限 # chmod +x /usr/sbin/runc ~~~ ~~~powershell 使用runc命令验证是否安装成功 # runc -v runc version 1.1.0 commit: v1.1.0-0-g067aaf85 spec: 1.0.2-dev go: go1.17.6 libseccomp: 2.5.3 ~~~ # 三、Containerd容器镜像管理 ## 3.1 Containerd容器镜像管理命令 * docker使用docker images命令管理镜像 * 单机containerd使用ctr images命令管理镜像,containerd本身的CLI * k8s中containerd使用crictl images命令管理镜像,Kubernetes社区的专用CLI工具 ~~~powershell 获取命令帮助 # ctr --help NAME: ctr - __ _____/ /______ / ___/ __/ ___/ / /__/ /_/ / \___/\__/_/ containerd CLI USAGE: ctr [global options] command [command options] [arguments...] VERSION: v1.6.0 DESCRIPTION: ctr is an unsupported debug and administrative client for interacting with the containerd daemon. Because it is unsupported, the commands, options, and operations are not guaranteed to be backward compatible or stable from release to release of the containerd project. COMMANDS: plugins, plugin provides information about containerd plugins version print the client and server versions containers, c, container manage containers content manage content events, event display containerd events images, image, i manage images leases manage leases namespaces, namespace, ns manage namespaces pprof provide golang pprof outputs for containerd run run a container snapshots, snapshot manage snapshots tasks, t, task manage tasks install install a new package oci OCI tools shim interact with a shim directly help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --debug enable debug output in logs --address value, -a value address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS] --timeout value total timeout for ctr commands (default: 0s) --connect-timeout value timeout for connecting to containerd (default: 0s) --namespace value, -n value namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE] --help, -h show help --version, -v print the version ~~~ ~~~powershell 获取命令帮助 # ctr images NAME: ctr images - manage images USAGE: ctr images command [command options] [arguments...] COMMANDS: check check existing images to ensure all content is available locally export export images import import images list, ls list images known to containerd mount mount an image to a target path unmount unmount the image from the target pull pull an image from a remote push push an image to a remote delete, del, remove, rm remove one or more images by reference tag tag an image label set and clear labels for an image convert convert an image OPTIONS: --help, -h show help ~~~ ## 3.2 查看镜像 ~~~powershell # ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS ~~~ ## 3.3 下载镜像 containerd支持oci标准的镜像，所以可以直接使用docker官方或dockerfile构建的镜像 ~~~powershell # ctr images pull --all-platforms docker.io/library/nginx:alpine docker.io/library/nginx:alpine: resolved |++++++++++++++++++++++++++++++++++++++| docker.io/library/nginx:alpine: resolved |++++++++++++++++++++++++++++++++++++++| index-sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3: done |++++++++++++++++++++++++++++++++++++++| manifest-sha256:050385609d832fae11b007fbbfba77d0bba12bf72bc0dca0ac03e09b1998580f: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:f2303c6c88653b9a6739d50f611c170b9d97d161c6432409c680f6b46a5f112f: done |++++++++++++++++++++++++++++++++++++++| config-sha256:bef258acf10dc257d641c47c3a600c92f87be4b4ce4a5e4752b3eade7533dcd9: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:8d6ba530f6489d12676d7f61628427d067243ba4a3a512c3e28813b977cb3b0e: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:5288d7ad7a7f84bdd19c1e8f0abb8684b5338f3da86fe9ae1d7f0e9bc2de6595: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:39e51c61c033442d00c40a30b2a9ed01f40205875fbd8664c50b4dc3e99ad5cf: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:ee6f71c6f4a82b2afd01f92bdf6be0079364d03020e8a2c569062e1c06d3822b: done |++++++++++++++++++++++++++++++++++++++| elapsed: 11.0s total: 8.7 Mi (809.5 KiB/s) unpacking linux/amd64 sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3... done: 1.860946163s ~~~ ~~~powershell 说明： 这里ctr命令pull镜像时，不能直接把镜像名字写成`nginx:alpine` ~~~ ~~~powershell 查看已下载容器镜像 # ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS docker.io/library/nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 9.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x - ~~~ | REF | TYPE | DIGEST | | ------------------------------ | --------------------------------------------------------- | ------------------------------------------------------------ | | docker.io/library/nginx:alpine | application/vnd.docker.distribution.manifest.list.v2+json | sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 | | SIZE | PLATFORMS | LABELS | | ------- | ------------------------------------------------------------ | ------ | | 9.7 MiB | linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x | - | ~~~powershell 指定平台下载容器镜像 # ctr images pull --platform linux/amd64 docker.io/library/nginx:alpine ~~~ ## 3.4 镜像挂载 > 方便查看镜像中包含的内容。 ~~~powershell 把已下载的容器镜像挂载至当前文件系统 # ctr images mount docker.io/library/nginx:alpine /mnt sha256:af2fcce448e2e4451a5f4796a9bf9cb5c9b5f88e0d6d10029cada42fb9d268ac /mnt [root@localhost ~]# ls /mnt bin dev docker-entrypoint.d docker-entrypoint.sh etc home lib media mnt opt proc root run sbin srv sys tmp usr var ~~~ ~~~powershell 卸载 # umount /mnt ~~~ ## 3.5 镜像导出 ~~~powershell 把容器镜像导出 # ctr i export --all-platforms nginx.img docker.io/library/nginx:alpine ~~~ ~~~powershell 说明 --all-platforms,导出所有平台镜像，本版本为1.6版本，1.4版本不需要添加此选项。 ~~~ ~~~powershell 查看已导出容器镜像 # ls nginx.img # ls -lh 总用量 196M -rw-r--r-- 1 root root 73M 2月 18 14:48 nginx.img ~~~ ## 3.6 镜像删除 ~~~powershell 删除指定容器镜像 # ctr image rm docker.io/library/nginx:alpine docker.io/library/nginx:alpine 再次查看容器镜像 [root@192 ~]# ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS ~~~ ## 3.7 镜像导入 ~~~powershell 导入容器镜像 # ctr images import nginx.img unpacking docker.io/library/nginx:alpine (sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3)...done ~~~ ## 3.8 修改镜像tag ~~~powershell # ctr images tag docker.io/library/nginx:alpine nginx:alpine nginx:alpine ~~~ ~~~powershell 说明： 把docker.io/library/nginx:alpine 修改为 nginx:alpine ~~~ ~~~powershell 查看修改后的容器镜像 # ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS docker.io/library/nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 9.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x - nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 9.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x - ~~~ ~~~powershell 修改后对容器镜像做检查比对 # ctr images check REF TYPE DIGEST STATUS SIZE UNPACKED docker.io/library/nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 complete (7/7) 9.7 MiB/9.7 MiB true nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:da9c94bec1da829ebd52431a84502ec471c8e548ffb2cedbf36260fd9bd1d4d3 complete (7/7) 9.7 MiB/9.7 MiB true ~~~ # 四、Containerd容器管理 ## 4.1 获取命令帮助 ### 4.1.1 获取ctr命令帮助 ~~~powershell [root@localhost ~]# ctr --help NAME: ctr - __ _____/ /______ / ___/ __/ ___/ / /__/ /_/ / \___/\__/_/ containerd CLI USAGE: ctr [global options] command [command options] [arguments...] VERSION: v1.6.0 DESCRIPTION: ctr is an unsupported debug and administrative client for interacting with the containerd daemon. Because it is unsupported, the commands, options, and operations are not guaranteed to be backward compatible or stable from release to release of the containerd project. COMMANDS: plugins, plugin provides information about containerd plugins version print the client and server versions containers, c, container manage containers content manage content events, event display containerd events images, image, i manage images leases manage leases namespaces, namespace, ns manage namespaces pprof provide golang pprof outputs for containerd run run a container snapshots, snapshot manage snapshots tasks, t, task manage tasks install install a new package oci OCI tools shim interact with a shim directly help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --debug enable debug output in logs --address value, -a value address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS] --timeout value total timeout for ctr commands (default: 0s) --connect-timeout value timeout for connecting to containerd (default: 0s) --namespace value, -n value namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE] --help, -h show help --version, -v print the version ~~~ ### 4.1.2 获取创建静态容器命令帮助 ~~~powershell # ctr container --help NAME: ctr containers - manage containers USAGE: ctr containers command [command options] [arguments...] COMMANDS: create create container delete, del, remove, rm delete one or more existing containers info get info about a container list, ls list containers label set and clear labels for a container checkpoint checkpoint a container restore restore a container from checkpoint OPTIONS: --help, -h show help ~~~ ~~~powershell 说明： 使用`ctr container create `命令创建容器后，容器并没有处于运行状态，其只是一个静态的容器。这个 container 对象只是包含了运行一个容器所需的资源及配置的数据结构，例如： namespaces、rootfs 和容器的配置都已经初始化成功了，只是用户进程(本案例为nginx)还没有启动。需要使用`ctr tasks`命令才能获取一个动态容器。 ~~~ ### 4.1.3 获取动态容器命令帮助 ~~~powershell # ctr run --help NAME: ctr run - run a container USAGE: ctr run [command options] [flags] Image|RootFS ID [COMMAND] [ARG...] OPTIONS: --rm remove the container after running --null-io send all IO to /dev/null --log-uri value log uri --detach, -d detach from the task after it has started execution --fifo-dir value directory used for storing IO FIFOs --cgroup value cgroup path (To disable use of cgroup, set to "" explicitly) --platform value run image for specific platform --cni enable cni networking for the container --runc-binary value specify runc-compatible binary --runc-root value specify runc-compatible root --runc-systemd-cgroup start runc with systemd cgroup manager --uidmap container-uid:host-uid:length run inside a user namespace with the specified UID mapping range; specified with the format container-uid:host-uid:length --gidmap container-gid:host-gid:length run inside a user namespace with the specified GID mapping range; specified with the format container-gid:host-gid:length --remap-labels provide the user namespace ID remapping to the snapshotter via label options; requires snapshotter support --cpus value set the CFS cpu quota (default: 0) --cpu-shares value set the cpu shares (default: 1024) --snapshotter value snapshotter name. Empty value stands for the default value. [$CONTAINERD_SNAPSHOTTER] --snapshotter-label value labels added to the new snapshot for this container. --config value, -c value path to the runtime-specific spec config file --cwd value specify the working directory of the process --env value specify additional container environment variables (e.g. FOO=bar) --env-file value specify additional container environment variables in a file(e.g. FOO=bar, one per line) --label value specify additional labels (e.g. foo=bar) --mount value specify additional container mount (e.g. type=bind,src=/tmp,dst=/host,options=rbind:ro) --net-host enable host networking for the container --privileged run privileged container --read-only set the containers filesystem as readonly --runtime value runtime name (default: "io.containerd.runc.v2") --runtime-config-path value optional runtime config path --tty, -t allocate a TTY for the container --with-ns value specify existing Linux namespaces to join at container runtime (format ':') --pid-file value file path to write the task's pid --gpus value add gpus to the container --allow-new-privs turn off OCI spec's NoNewPrivileges feature flag --memory-limit value memory limit (in bytes) for the container (default: 0) --device value file path to a device to add to the container; or a path to a directory tree of devices to add to the container --cap-add value add Linux capabilities (Set capabilities with 'CAP_' prefix) --cap-drop value drop Linux capabilities (Set capabilities with 'CAP_' prefix) --seccomp enable the default seccomp profile --seccomp-profile value file path to custom seccomp profile. seccomp must be set to true, before using seccomp-profile --apparmor-default-profile value enable AppArmor with the default profile with the specified name, e.g. "cri-containerd.apparmor.d" --apparmor-profile value enable AppArmor with an existing custom profile --rdt-class value name of the RDT class to associate the container with. Specifies a Class of Service (CLOS) for cache and memory bandwidth management. --rootfs use custom rootfs that is not managed by containerd snapshotter --no-pivot disable use of pivot-root (linux only) --cpu-quota value Limit CPU CFS quota (default: -1) --cpu-period value Limit CPU CFS period (default: 0) --rootfs-propagation value set the propagation of the container rootfs ~~~ ~~~powershell 说明： 使用`ctr run`命令可以创建一个静态容器并使其运行。一步到位运行容器。 ~~~ ## 4.2 查看容器 container表示静态容器，可用c缩写代表container ~~~powershell # ctr container ls CONTAINER IMAGE RUNTIME ~~~ 或 ~~~powershell # ctr c ls CONTAINER IMAGE RUNTIME ~~~ ## 4.3 查看任务 task表示容器里跑的进程, 可用t缩写代表task ~~~powershell # ctr task ls TASK PID STATUS ~~~ 或 ~~~powershell # ctr t ls TASK PID STATUS ~~~ ## 4.4 创建静态容器 ~~~powershell # ctr c create docker.io/library/nginx:alpine nginx1 ~~~ ~~~powershell # ctr container ls CONTAINER IMAGE RUNTIME nginx1 docker.io/library/nginx:alpine io.containerd.runc.v2 ~~~ ~~~powershell 查看容器详细信息 # ctr container info nginx1 ~~~ ## 4.5 静态容器启动为动态容器 ~~~powershell 复制containerd连接runC垫片工具至系统 # ls usr/local/bin/ containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress crictl critest ctd-decoder ctr [root@localhost ~]# cp usr/local/bin/containerd-shim-runc-v2 /usr/bin/ ~~~ ~~~powershell 启动task，即表时在容器中运行了进程，即为动态容器。 # ctr task start -d nginx1 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ ~~~ ~~~powershell 说明： -d表示daemon或者后台的意思，否则会卡住终端 ~~~ ~~~powershell 查看容器所在宿主机进程，是以宿主机进程的方式存在的。 # ctr task ls TASK PID STATUS nginx1 3395 RUNNING ~~~ ~~~powershell 查看容器的进程(都是物理机的进程) # ctr task ps nginx1 PID INFO 3395 - 3434 - ~~~ ~~~powershell 物理机查看到相应的进程 # ps -ef | grep 3395 root 3395 3375 0 19:16 ? 00:00:00 nginx: master process nginx -g daemon off; 101 3434 3395 0 19:16 ? 00:00:00 nginx: worker process ~~~ ## 4.6 进入容器操作 ~~~powershell # ctr task exec --exec-id 1 nginx1 /bin/sh ifconfig 查看网卡信息 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) curl 127.0.0.1 访问本地提供的web服务 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

100 615 100 615 0 0 429k 0 --:--:-- --:--:-- --:--:-- 600k ~~~ ~~~powershell 说明： 为exec进程设定一个id，可以随意输入，只要保证唯一即可，也可使用$RANDOM变量。 ~~~ ## 4.7 直接运行一个动态容器 ~~~powershell # ctr run -d --net-host docker.io/library/nginx:alpine nginx2 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ ~~~ ~~~powershell 说明： * -d 代表dameon，后台运行 * --net-host 代表容器的IP就是宿主机的IP(相当于docker里的host类型网络) ~~~ ~~~powershell 查看已运行容器 # ctr container ls CONTAINER IMAGE RUNTIME nginx2 docker.io/library/nginx:alpine io.containerd.runc.v2 ~~~ ~~~powershell 查看已运行容器中运行的进程,既tasks # ctr tasks ls TASK PID STATUS nginx2 4061 RUNNING ~~~ ~~~powershell 进入容器 # ctr task exec --exec-id 1 -t nginx2 /bin/sh ~~~ ~~~powershell / # ifconfig ens33 Link encap:Ethernet HWaddr 00:0C:29:B1:B6:1D inet addr:192.168.10.164 Bcast:192.168.10.255 Mask:255.255.255.0 inet6 addr: fe80::2b33:40ed:9311:8812/64 Scope:Link inet6 addr: fe80::adf4:a8bc:a1c:a9f7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:55360 errors:0 dropped:0 overruns:0 frame:0 TX packets:30526 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:53511295 (51.0 MiB) TX bytes:2735050 (2.6 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:68 errors:0 dropped:0 overruns:0 frame:0 TX packets:68 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5916 (5.7 KiB) TX bytes:5916 (5.7 KiB) virbr0 Link encap:Ethernet HWaddr 52:54:00:E9:51:82 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) ~~~ ~~~powershell 为容器中运行的网站添加网站文件 / # echo "nginx2" > /usr/share/nginx/html/index.html / # exit ~~~ ~~~powershell 在宿主机上访问网站 [root@localhost ~]# curl 192.168.10.164 nginx2 ~~~ ## 4.8 暂停容器 ~~~powershell 查看容器状态 # ctr tasks ls TASK PID STATUS nginx2 4061 RUNNING ~~~ ~~~powershell 暂停容器 # ctr tasks pause nginx2 ~~~ ~~~powershell 再次查看容器状态，看到其状态为PAUSED，表示停止。 # ctr tasks ls TASK PID STATUS nginx2 4061 PAUSED ~~~ ~~~powershell [root@localhost ~]# curl http://192.168.10.164 在宿主机访问，发现不可以访问到网站 ~~~ ## 4.9 恢复容器 ~~~powershell 使用resume命令恢复容器 # ctr tasks resume nginx2 ~~~ ~~~powershell 查看恢复后状态 # ctr tasks ls TASK PID STATUS nginx2 4061 RUNNING ~~~ ~~~powershell 在宿主机上访问容器中提供的服务 # curl http://192.168.10.164 nginx2 ~~~ ## 4.10 停止容器 ~~~powershell # ctr tasks --help NAME: ctr tasks - manage tasks USAGE: ctr tasks command [command options] [arguments...] COMMANDS: attach attach to the IO of a running container checkpoint checkpoint a container delete, del, remove, rm delete one or more tasks exec execute additional processes in an existing container list, ls list tasks kill signal a container (default: SIGTERM) pause pause an existing container ps list processes for container resume resume a paused container start start a container that has been created metrics, metric get a single data point of metrics for a task with the built-in Linux runtime OPTIONS: --help, -h show help ~~~ ~~~powershell 使用kill命令停止容器中运行的进程，既为停止容器 # ctr tasks kill nginx2 ~~~ ~~~powershell 查看容器停止后状态，STATUS为STOPPED # ctr tasks ls TASK PID STATUS nginx1 3395 RUNNING nginx2 4061 STOPPED ~~~ ## 4.11 删除容器 ~~~powershell # ctr tasks delete nginx2 必须先停止tasks或先删除task，再删除容器 ~~~ ~~~powershell 查看静态容器，确认其还存在于系统中 # ctr container ls CONTAINER IMAGE RUNTIME nginx2 docker.io/library/nginx:alpine io.containerd.runc.v2 ~~~ ~~~powershell 删除容器 # ctr container delete nginx2 ~~~ # 五、Containerd使用私有容器镜像仓库 Harbor ## 5.1 Harbor准备 ![image-20220218211611942](../../img/docker_container/image-20220218211611942.png) ![image-20220218211755113](../../img/docker_container/image-20220218211755113.png) ## 5.2 配置Containerd使用Harbor仓库 ### 5.2.1 Harbor主机名解析 > 在所有安装containerd宿主机上添加此配置信息。 ~~~powershell # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.165 harbor.kubemsb.com ~~~ ~~~powershell 说明 * 192.168.10.165是harbor的IP * harbor.kubemsb.com建议用FQDN形式，如果用类似harbor这种短名，后面下载镜像会出问题 ~~~ ### 5.2.2 修改Containerd配置文件 ~~~powershell 此配置文件已提前替换过，仅修改本地容器镜像仓库地址即可。 # vim /etc/containerd/config.toml # cat /etc/containerd/config.toml root = "/var/lib/containerd" state = "/run/containerd" oom_score = -999 [grpc] address = "/run/containerd/containerd.sock" uid = 0 gid = 0 max_recv_message_size = 16777216 max_send_message_size = 16777216 [debug] address = "" uid = 0 gid = 0 level = "" [metrics] address = "" grpc_histogram = false [cgroup] path = "" [plugins] [plugins.cgroups] no_prometheus = false [plugins.cri] stream_server_address = "127.0.0.1" stream_server_port = "0" enable_selinux = false sandbox_image = "easzlab/pause-amd64:3.2" stats_collect_period = 10 systemd_cgroup = false enable_tls_streaming = false max_container_log_line_size = 16384 [plugins.cri.containerd] snapshotter = "overlayfs" no_pivot = false [plugins.cri.containerd.default_runtime] runtime_type = "io.containerd.runtime.v1.linux" runtime_engine = "" runtime_root = "" [plugins.cri.containerd.untrusted_workload_runtime] runtime_type = "" runtime_engine = "" runtime_root = "" [plugins.cri.cni] bin_dir = "/opt/kube/bin" conf_dir = "/etc/cni/net.d" conf_template = "/etc/cni/net.d/10-default.conf" [plugins.cri.registry] [plugins.cri.registry.mirrors] [plugins.cri.registry.mirrors."docker.io"] endpoint = [ "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com" ] [plugins.cri.registry.mirrors."gcr.io"] endpoint = [ "https://gcr.mirrors.ustc.edu.cn" ] [plugins.cri.registry.mirrors."k8s.gcr.io"] endpoint = [ "https://gcr.mirrors.ustc.edu.cn/google-containers/" ] [plugins.cri.registry.mirrors."quay.io"] endpoint = [ "https://quay.mirrors.ustc.edu.cn" ] [plugins.cri.registry.mirrors."harbor.kubemsb.com"] 在此处添加,在镜像加速器下面添加这一段 endpoint = [ "http://harbor.kubemsb.com" ] [plugins.cri.x509_key_pair_streaming] tls_cert_file = "" tls_key_file = "" [plugins.diff-service] default = ["walking"] [plugins.linux] shim = "containerd-shim" runtime = "runc" runtime_root = "" no_shim = false shim_debug = false [plugins.opt] path = "/opt/containerd" [plugins.restart] interval = "10s" [plugins.scheduler] pause_threshold = 0.02 deletion_threshold = 0 mutation_threshold = 100 schedule_delay = "0s" startup_delay = "100ms" ~~~ ~~~powershell 重启containerd，以便于重新加载配置文件。 # systemctl restart containerd ~~~ ### 5.2.3 ctr下载镜像 ~~~powershell 下载容器镜像 # ctr images pull --platform linux/amd64 docker.io/library/nginx:latest ~~~ ~~~powershell 说明: * --platform linux/amd64 指定系统平台，也可以使用--all-platforms指定所有平台镜像。 ~~~ ~~~powershell 输出： docker.io/library/nginx:latest: resolved |++++++++++++++++++++++++++++++++++++++| index-sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767: done |++++++++++++++++++++++++++++++++++++++| manifest-sha256:bb129a712c2431ecce4af8dde831e980373b26368233ef0f3b2bae9e9ec515ee: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a: done |++++++++++++++++++++++++++++++++++++++| config-sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b: done |++++++++++++++++++++++++++++++++++++++| elapsed: 20.0s total: 53.2 M (2.7 MiB/s) unpacking linux/amd64 sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767... done: 3.028652226s ~~~ ~~~powershell 查看已下载容器镜像 # ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS docker.io/library/nginx:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 54.1 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x - ~~~ ### 5.2.4 ctr上传镜像 >上传到Harbor library公有项目 ~~~powershell 重新生成新的tag # ctr images tag docker.io/library/nginx:latest harbor.kubemsb.com/library/nginx:latest harbor.kubemsb.com/library/nginx:latest ~~~ ~~~powershell 查看已生成容器镜像 # ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS docker.io/library/nginx:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 54.1 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x - harbor.kubemsb.com/library/nginx:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767 54.1 MiB linux/386,linux/amd64,linux/arm/v5,linux/arm/v7,linux/arm64/v8,linux/mips64le,linux/ppc64le,linux/s390x - ~~~ ~~~powershell 推送容器镜像至Harbor # ctr images push --platform linux/amd64 --plain-http -u admin:Harbor12345 harbor.kubemsb.com/library/nginx:latest ~~~ ~~~powershell 说明: * 先tag再push * 因为我们harbor是http协议，不是https协议，所以需要加上`--plain-http` * `--user admin:Harbor12345`指定harbor的用户名与密码 ~~~ ~~~powershell 输出： manifest-sha256:0fd68ec4b64b8dbb2bef1f1a5de9d47b658afd3635dc9c45bf0cbeac46e72101: done |++++++++++++++++++++++++++++++++++++++| config-sha256:dd025cdfe837e1c6395365870a491cf16bae668218edb07d85c626928a60e478: done |++++++++++++++++++++++++++++++++++++++| elapsed: 0.5 s total: 9.3 Ki (18.1 KiB/s) ~~~ ![image-20220220111100489](../../img/docker_container/image-20220220111100489.png) ~~~powershell 下载已上传容器镜像 # ctr images pull --plain-http harbor.kubemsb.com/library/nginx:latest ~~~ # 六、Containerd NameSpace管理 > containerd中namespace的作用为:隔离运行的容器，可以实现运行多个容器。 ~~~powershell 查看命令帮助 # ctr namespace --help NAME: ctr namespaces - manage namespaces USAGE: ctr namespaces command [command options] [arguments...] COMMANDS: create, c create a new namespace list, ls list namespaces remove, rm remove one or more namespaces label set and clear labels for a namespace OPTIONS: --help, -h show help ~~~ ~~~powershell 列出已有namespace # ctr namespace ls NAME LABELS default k8s.io ~~~ ~~~powershell 创建namespace # ctr namespace create kubemsb [root@localhost ~]# ctr namespace ls NAME LABELS default k8s.io kubemsb 此命名空间为新添加的 ~~~ ~~~powershell 删除namespace # ctr namespace rm kubemsb kubemsb 再次查看是否删除 [root@localhost ~]# ctr namespace ls NAME LABELS default k8s.io ~~~ ~~~powershell 查看指定namespace中是否有用户进程在运行 # ctr -n kubemsb tasks ls TASK PID STATUS ~~~ ~~~powershell 在指定namespace中下载容器镜像 # ctr -n kubemsb images pull docker.io/library/nginx:latest ~~~ ~~~powershell 在指定namespace中创建静态容器 # ctr -n kubemsb container create docker.io/library/nginx:latest nginxapp ~~~ ~~~powershell 查看在指定namespace中创建的容器 # ctr -n kubemsb container ls CONTAINER IMAGE RUNTIME nginxapp docker.io/library/nginx:latest io.containerd.runc.v2 ~~~ # 七、Containerd Network管理 > 默认Containerd管理的容器仅有lo网络，无法访问容器之外的网络，可以为其添加网络插件，使用容器可以连接外网。CNI（Container Network Interface） ## 7.1 创建CNI网络 | [*containernetworking*/*cni*](https://github.com/containernetworking/cni) | [ CNI v1.0.1](https://github.com/containernetworking/cni/releases/tag/v1.0.1) | | ------------------------------------------------------------ | ------------------------------------------------------------ | | [*containernetworking*/*plugins*](https://github.com/containernetworking/plugins) | [ CNI Plugins v1.0.1](https://github.com/containernetworking/plugins/releases/tag/v1.0.1) | ### 7.1.1 获取CNI工具源码 ![image-20220219095355845](../../img/docker_container/image-20220219095355845.png) ![image-20220219095427153](../../img/docker_container/image-20220219095427153.png) ![image-20220219095515772](../../img/docker_container/image-20220219095515772.png) ![image-20220219095615236](../../img/docker_container/image-20220219095615236.png) ~~~powershell 使用wget下载cni工具源码包 # wget https://github.com/containernetworking/cni/archive/refs/tags/v1.0.1.tar.gz ~~~ ~~~powershell 查看已下载cni工具源码包 # ls v1.0.1.tar.gz 解压已下载cni工具源码包 # tar xf v1.0.1.tar.gz 查看解压后已下载cni工具源码包 # ls cni-1.0.1 重命名已下载cni工具源码包目录 # mv cni-1.0.1 cni 查看重新命名后目录 # ls cni 查看cni工具目录中包含的文件 # ls cni cnitool CONTRIBUTING.md DCO go.mod GOVERNANCE.md LICENSE MAINTAINERS plugins RELEASING.md scripts test.sh CODE-OF-CONDUCT.md CONVENTIONS.md Documentation go.sum libcni logo.png pkg README.md ROADMAP.md SPEC.md ~~~ ### 7.1.2 获取CNI Plugins（CNI插件） ![image-20220219095946940](../../img/docker_container/image-20220219095946940.png) ![image-20220219100008810](../../img/docker_container/image-20220219100008810.png) ![image-20220219100056059](../../img/docker_container/image-20220219100056059.png) ![image-20220219100303944](../../img/docker_container/image-20220219100303944.png) ~~~powershell 使用wget下载cni插件工具源码包 # wget https://github.com/containernetworking/plugins/releases/download/v1.0.1/cni-plugins-linux-amd64-v1.0.1.tgz ~~~ ~~~powershell 查看已下载cni插件工具源码包 # ls cni-plugins-linux-amd64-v1.0.1.tgz cni 创建cni插件工具解压目录 # mkdir /home/cni-plugins 解压cni插件工具至上述创建的目录中 # tar xf cni-plugins-linux-amd64-v1.0.1.tgz -C /home/cni-plugins 查看解压后目录 # ls cni-plugins bandwidth bridge dhcp firewall host-device host-local ipvlan loopback macvlan portmap ptp sbr static tuning vlan vrf ~~~ ### 7.1.3 准备CNI网络配置文件 > 准备容器网络配置文件，用于为容器提供网关、IP地址等。 ~~~powershell 创建名为mynet的网络，其中包含名为cni0的网桥 # vim /etc/cni/net.d/10-mynet.conf # cat /etc/cni/net.d/10-mynet.conf { "cniVersion": "1.0.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.66.0.0/16", "routes": [ { "dst": "0.0.0.0/0" } ] } } ~~~ ~~~powershell # vim /etc/cni/net.d/99-loopback.conf # cat /etc/cni/net.d/99-loopback.conf { "cniVerion": "1.0.0", "name": "lo", "type": "loopback" } ~~~ ### 7.1.4 生成CNI网络 ~~~powershell 获取epel源 # wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo 安装jq # yum -y install jq ~~~ ~~~powershell 进入cni工具目录 # cd cni [root@localhost cni]# ls cnitool CONTRIBUTING.md DCO go.mod GOVERNANCE.md LICENSE MAINTAINERS plugins RELEASING.md scripts test.sh CODE-OF-CONDUCT.md CONVENTIONS.md Documentation go.sum libcni logo.png pkg README.md ROADMAP.md SPEC.md 必须在scripts目录中执行，需要依赖exec-plugins.sh文件，再次进入scripts目录 [root@localhost cni]# cd scripts/ 查看执行脚本文件 [root@localhost scripts]# ls docker-run.sh exec-plugins.sh priv-net-run.sh release.sh 执行脚本文件，基于/etc/cni/net.d/目录中的*.conf配置文件生成容器网络 [root@localhost scripts]# CNI_PATH=/home/cni-plugins ./priv-net-run.sh echo "Hello World" Hello World ~~~ ~~~powershell 在宿主机上查看是否生成容器网络名为cni0的网桥 # ip a s ...... 5: cni0: mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 36:af:7a:4a:d6:12 brd ff:ff:ff:ff:ff:ff inet 10.66.0.1/16 brd 10.66.255.255 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::34af:7aff:fe4a:d612/64 scope link valid_lft forever preferred_lft forever ~~~ ~~~powershell 在宿主机上查看其路由表情况 # ip route default via 192.168.10.2 dev ens33 proto dhcp metric 100 10.66.0.0/16 dev cni0 proto kernel scope link src 10.66.0.1 192.168.10.0/24 dev ens33 proto kernel scope link src 192.168.10.164 metric 100 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 ~~~ ## 7.2 为Containerd容器配置网络功能 ### 7.2.1 创建一个容器 ~~~powershell # ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS # ctr images pull docker.io/library/busybox:latest # ctr run -d docker.io/library/busybox:latest busybox # ctr container ls CONTAINER IMAGE RUNTIME busybox docker.io/library/busybox:latest io.containerd.runc.v2 # ctr tasks ls TASK PID STATUS busybox 8377 RUNNING ~~~ ### 7.2.2 进入容器查看其网络情况 ~~~powershell # ctr tasks exec --exec-id $RANDOM -t busybox sh / # ip a s 1: lo: mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever ~~~ ### 7.2.3 获取容器进程ID及其网络命名空间 ~~~powershell 在宿主机中完成指定容器进程ID获取 # pid=$(ctr tasks ls | grep busybox | awk '{print $2}') # echo $pid 8377 ~~~ ~~~powershell 在宿主机中完成指定容器网络命名空间路径获取 # netnspath=/proc/$pid/ns/net # echo $netnspath /proc/8377/ns/net ~~~ ### 7.2.4 为指定容器添加网络配置 ~~~powershell 确认执行脚本文件时所在的目录 [root@localhost scripts]# pwd /home/cni/scripts ~~~ ~~~powershell 执行脚本文件为容器添加网络配置 [root@localhost scripts]# CNI_PATH=/home/cni-plugins ./exec-plugins.sh add $pid $netnspath ~~~ ~~~powershell 进入容器确认是否添加网卡信息 # ctr tasks exec --exec-id $RANDOM -t busybox sh / # ip a s 1: lo: mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0@if7: mtu 1500 qdisc noqueue link/ether a2:35:b7:e0:60:0a brd ff:ff:ff:ff:ff:ff inet 10.66.0.3/16 brd 10.66.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::a035:b7ff:fee0:600a/64 scope link valid_lft forever preferred_lft forever 在容器中ping容器宿主机IP地址 / # ping -c 2 192.168.10.164 PING 192.168.10.164 (192.168.10.164): 56 data bytes 64 bytes from 192.168.10.164: seq=0 ttl=64 time=0.132 ms 64 bytes from 192.168.10.164: seq=1 ttl=64 time=0.044 ms --- 192.168.10.164 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.044/0.088/0.132 ms 在容器中ping宿主机所在网络的网关IP地址 / # ping -c 2 192.168.10.2 PING 192.168.10.2 (192.168.10.2): 56 data bytes 64 bytes from 192.168.10.2: seq=0 ttl=127 time=0.338 ms 64 bytes from 192.168.10.2: seq=1 ttl=127 time=0.280 ms --- 192.168.10.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.280/0.309/0.338 ms 在容器中ping宿主机所在网络中的其它主机IP地址 / # ping -c 2 192.168.10.165 PING 192.168.10.165 (192.168.10.165): 56 data bytes 64 bytes from 192.168.10.165: seq=0 ttl=63 time=0.422 ms 64 bytes from 192.168.10.165: seq=1 ttl=63 time=0.908 ms --- 192.168.10.165 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.422/0.665/0.908 ms ~~~ ~~~powershell 在容器中开启httpd服务 / # echo "containerd net web test" > /tmp/index.html / # httpd -h /tmp / # wget -O - -q 127.0.0.1 containerd net web test / # exit ~~~ ~~~powershell 在宿主机访问容器提供的httpd服务 [root@localhost scripts]# curl http://10.66.0.3 containerd net web test ~~~ # 八、Containerd容器数据持久化存储 > 实现把宿主机目录挂载至Containerd容器中，实现容器数据持久化存储 ~~~powershell # ctr container create docker.io/library/busybox:latest busybox3 --mount type=bind,src=/tmp,dst=/hostdir,options=rbind:rw ~~~ ~~~powershell 说明： 创建一个静态容器，实现宿主机目录与容器挂载 src=/tmp 为宿主机目录 dst=/hostdir 为容器中目录 ~~~ ~~~powershell 运行用户进程 # ctr tasks start -d busybox3 bash ~~~ ~~~powershell 进入容器，查看是否挂载成功 # ctr tasks exec --exec-id $RANDOM -t busybox3 sh / # ls /hostdir VMwareDnD systemd-private-cf1fe70805214c80867e7eb62dff5be7-bolt.service-MWV1Ju systemd-private-cf1fe70805214c80867e7eb62dff5be7-chronyd.service-6B6j8p systemd-private-cf1fe70805214c80867e7eb62dff5be7-colord.service-6fI31A systemd-private-cf1fe70805214c80867e7eb62dff5be7-cups.service-tuK4zI systemd-private-cf1fe70805214c80867e7eb62dff5be7-rtkit-daemon.service-vhP67o tracker-extract-files.0 vmware-root_703-3988031936 vmware-root_704-2990744159 vmware-root_713-4290166671 向容器中挂载目录中添加文件 / # echo "hello world" > /hostdir/test.txt 退出容器 / # exit 在宿主机上查看被容器挂载的目录中是否添加了新的文件，已添加表明被容器挂载成功，并可以读写此目录中内容。 [root@localhost ~]# cat /tmp/test.txt hello world ~~~ # 九、与其它Containerd容器共享命名空间 > 当需要与其它Containerd管理的容器共享命名空间时，可使用如下方法。 ~~~powershell # ctr tasks ls TASK PID STATUS busybox3 13778 RUNNING busybox 8377 RUNNING busybox1 12469 RUNNING ~~~ ~~~powershell # ctr container create --with-ns "pid:/proc/13778/ns/pid" docker.io/library/busybox:latest busybox4 [root@localhost ~]# ctr tasks start -d busybox4 bash [root@localhost ~]# ctr tasks exec --exec-id $RANDOM -t busybox3 sh / # ps aux PID USER TIME COMMAND 1 root 0:00 sh 20 root 0:00 sh 26 root 0:00 sh 32 root 0:00 ps aux ~~~ # 十、Docker集成Containerd实现容器管理 目前Containerd主要任务还在于解决容器运行时的问题，对于其周边生态还不完善，所以可以借助Docker结合Containerd来实现Docker完整的功能应用。 ~~~powershell 准备Docker安装YUM源 # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell 安装Docker-ce # yum -y install docker-ce ~~~ ~~~powershell 修改Docker服务文件，以便使用已安装的containerd。 # vim /etc/systemd/system/multi-user.target.wants/docker.service 修改前： [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 此处 ExecReload=/bin/kill -s HUP $MAINPID 修改后： [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd --containerd /run/containerd/containerd.sock --debug 此处 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always ~~~ ~~~powershell 设置docker daemon启动并设置其开机自启动 # systemctl daemon-reload # systemctl enable docker Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. # systemctl start docker ~~~ ~~~powershell 查看其启动后进程 # ps aux | grep docker root 16270 0.0 3.1 1155116 63320 ? Ssl 12:09 0:00 /usr/bin/dockerd --containerd /run/containerd/containerd.sock --debug ~~~ ~~~powershell 使用docker运行容器 # docker run -d nginx:latest ...... 219a9c6727bcd162d0a4868746c513a277276a110f47e15368b4229988003c13 ~~~ ~~~powershell 使用docker ps命令查看正在运行的容器 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 219a9c6727bc nginx:latest "/docker-entrypoint.…" 14 seconds ago Up 13 seconds 80/tcp happy_tu ~~~ ~~~powershell 使用ctr查看是否添加一个新的namespace，本案例中发现添加一个moby命名空间，即为docker使用的命名空间。 # ctr namespace ls NAME LABELS default k8s.io kubemsb moby ~~~ ~~~powershell 查看moby命名空间，发现使用docker run运行的容器包含在其中。 # ctr -n moby container ls CONTAINER IMAGE RUNTIME 219a9c6727bcd162d0a4868746c513a277276a110f47e15368b4229988003c13 - io.containerd.runc.v2 ~~~ ~~~powershell 使用ctr能够查看到一个正在运行的容器，既表示docker run运行的容器是被containerd管理的。 # ctr -n moby tasks ls TASK PID STATUS 219a9c6727bcd162d0a4868746c513a277276a110f47e15368b4229988003c13 16719 RUNNING ~~~ ~~~powershell 使用docker stop停止且使用docker rm删除容器后再观察，发现容器被删除。 # docker stop 219;docker rm 219 219 219 # ctr -n moby container ls CONTAINER IMAGE RUNTIME # ctr -n moby tasks ls TASK PID STATUS ~~~ ================================================ FILE: docs/cloud/docker/docker_container_enterprice.md ================================================ # Docker容器化部署企业级应用集群 # 一、Docker容器化部署企业级应用 ## 1.1 使用Docker容器化部署企业级应用必要性 - 有利于快速实现企业级应用部署 - 有利于快速实现企业级应用恢复 ## 1.2 使用Docker容器化部署企业级应用参考资料 ![image-20220211145757283](../../img/docker_container_enterprice/image-20220211145757283.png) # 二、使用Docker容器实现Nginx部署 ## 2.1 获取参考资料 ![image-20220211145839441](../../img/docker_container_enterprice/image-20220211145839441.png) ![image-20220211145905117](../../img/docker_container_enterprice/image-20220211145905117.png) ![image-20220211145956450](../../img/docker_container_enterprice/image-20220211145956450.png) ## 2.2 运行Nginx应用容器 > 不在docker host暴露端口 ~~~powershell # docker run -d --name nginx-server -v /opt/nginx-server:/usr/share/nginx/html:ro nginx 664cd1bbda4ad2a71cbd09f0c6baa9b34db80db2d69496670a960be07b9521cb ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 664cd1bbda4a nginx "/docker-entrypoint.…" 4 seconds ago Up 3 seconds 80/tcp nginx-server ~~~ ~~~powershell # docker inspect 664 | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "172.17.0.3", "IPAddress": "172.17.0.3", ~~~ ~~~powershell # curl http://172.17.0.3 403 Forbidden

403 Forbidden

nginx/1.21.6
~~~ ~~~powershell # ls /opt nginx-server # echo "nginx is working" > /opt/nginx-server/index.html ~~~ ~~~powershell # curl http://172.17.0.3 nginx is working ~~~ ## 2.3 运行Nginx应用容器 >在docker host暴露80端口 ~~~powershell # docker run -d -p 80:80 --name nginx-server-port -v /opt/nginx-server-port:/usr/share/nginx/html:ro nginx ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 74dddf51983d nginx "/docker-entrypoint.…" 3 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp nginx-server-port ~~~ ~~~powershell # ls /opt nginx-server nginx-server-port ~~~ ~~~powershell # echo "nginx is running" > /opt/nginx-server-port/index.html ~~~ **在宿主机上访问** ![image-20220211151131609](../../img/docker_container_enterprice/image-20220211151131609.png) ~~~powershell # docker top nginx-server-port UID PID PPID C STIME TTY TIME CMD root 22195 22163 0 15:08 ? 00:00:00 nginx: master process nginx -g daemon off; 101 22387 22195 0 15:08 ? 00:00:00 nginx: worker process ~~~ ## 2.4 运行Nginx应用容器 > 挂载配置文件,需要创建一个nginx容器，把配置文件复制出来修改后使用。 ~~~powershell # docker cp nginxwebcontainername:/etc/nginx/nginx.conf /opt/nginxcon/ 修改后即可使用 ~~~ ~~~powershell # ls /opt/nginxcon/nginx.conf /opt/nginxcon/nginx.conf ~~~ ~~~powershell # docker run -d \ -p 82:80 --name nginx-server-conf \ -v /opt/nginx-server-conf:/usr/share/nginx/html:ro \ -v /opt/nginxcon/nginx.conf:/etc/nginx/nginx.conf:ro \ nginx 76251ec44e5049445399303944fc96eb8161ccb49e27b673b99cb2492009523c ~~~ ~~~powershell # docker top nginx-server-conf UID PID PPID C STIME TTY TIME CMD root 25005 24972 0 15:38 ? 00:00:00 nginx: master process nginx -g daemon off; 101 25178 25005 0 15:38 ? 00:00:00 nginx: worker process 101 25179 25005 0 15:38 ? 00:00:00 nginx: worker process ~~~ # 三、使用Docker容器实现Tomcat部署 ## 3.1 获取参考资料 ![image-20220211154602595](../../img/docker_container_enterprice/image-20220211154602595.png) ![image-20220211154639682](../../img/docker_container_enterprice/image-20220211154639682.png) ![image-20220211154747062](../../img/docker_container_enterprice/image-20220211154747062.png) ## 3.2 运行tomcat应用容器 ### 3.2.1 不暴露端口运行 ~~~powershell # docker run -d --rm tomcat:9.0 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c20a0e781246 tomcat:9.0 "catalina.sh run" 27 seconds ago Up 25 seconds 8080/tcp heuristic_cori ~~~ ### 3.2.2 暴露端口运行 ~~~powershell # docker run -d -p 8080:8080 --rm tomcat:9.0 2fcf5762314373c824928490b871138a01a94abedd7e6814ad5f361d09fbe1de ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2fcf57623143 tomcat:9.0 "catalina.sh run" 3 seconds ago Up 1 second 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp eloquent_chatelet ~~~ **在宿主机访问** ![image-20220211160925125](../../img/docker_container_enterprice/image-20220211160925125.png) ~~~powershell # docker exec 2fc ls /usr/local/tomcat/webapps 里面为空，所以可以添加网站文件。 ~~~ ### 3.2.3 暴露端口及添加网站文件 ~~~powershell # docker run -d -p 8081:8080 -v /opt/tomcat-server:/usr/local/tomcat/webapps/ROOT tomcat:9.0 f456e705d48fc603b7243a435f0edd6284558c194e105d87befff2dccddc0b63 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f456e705d48f tomcat:9.0 "catalina.sh run" 3 seconds ago Up 2 seconds 0.0.0.0:8081->8080/tcp, :::8081->8080/tcp cool_germain ~~~ ~~~powershell # echo "tomcat running" > /opt/tomcat-server/index.html ~~~ **在宿主机访问** ![image-20220211162127222](../../img/docker_container_enterprice/image-20220211162127222.png) # 四、使用Docker容器实现MySQL部署 ## 4.1 单节点MySQL部署 ![image-20220211162728055](../../img/docker_container_enterprice/image-20220211162728055.png) ![image-20220211162817731](../../img/docker_container_enterprice/image-20220211162817731.png) ![image-20220211162911952](../../img/docker_container_enterprice/image-20220211162911952.png) ~~~powershell # docker run -p 3306:3306 \ --name mysql \ -v /opt/mysql/log:/var/log/mysql \ -v /opt/mysql/data:/var/lib/mysql \ -v /opt/mysql/conf:/etc/mysql \ -e MYSQL_ROOT_PASSWORD=root \ -d \ mysql:5.7 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6d16ca21cf31 mysql:5.7 "docker-entrypoint.s…" 32 seconds ago Up 30 seconds 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql ~~~ ~~~powershell 通过容器中客户端访问 # docker exec -it mysql mysql -uroot -proot mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.7.37 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> ~~~ ~~~powershell 在docker host上访问 # yum -y install mariadb # mysql -h 192.168.255.157 -uroot -proot -P 3306 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 7 Server version: 5.7.37 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec) ~~~ ## 4.2 MySQL主从复制集群部署 ### 4.2.1 MySQL主节点部署 ~~~powershell # docker run -p 3306:3306 \ --name mysql-master \ -v /opt/mysql-master/log:/var/log/mysql \ -v /opt/mysql-master/data:/var/lib/mysql \ -v /opt/mysql-master/conf:/etc/mysql \ -e MYSQL_ROOT_PASSWORD=root \ -d mysql:5.7 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2dbbed8e35c7 mysql:5.7 "docker-entrypoint.s…" 58 seconds ago Up 57 seconds 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mysql-master ~~~ ### 4.2.2 MySQL主节点配置 ~~~powershell # vim /opt/mysql-master/conf/my.cnf # cat /opt/mysql-master/conf/my.cnf [client] default-character-set=utf8 [mysql] default-character-set=utf8 [mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake skip-name-resolve server_id=1 log-bin=mysql-bin read-only=0 binlog-do-db=kubemsb_test replicate-ignore-db=mysql replicate-ignore-db=sys replicate-ignore-db=information_schema replicate-ignore-db=performance_schema ~~~ ### 4.2.3 MySQL从节点部署 ~~~powershell # docker run -p 3307:3306 \ --name mysql-slave \ -v /opt/mysql-slave/log:/var/log/mysql \ -v /opt/mysql-slave/data:/var/lib/mysql \ -v /opt/mysql-slave/conf:/etc/mysql \ -e MYSQL_ROOT_PASSWORD=root \ -d --link mysql-master:mysql-master mysql:5.7 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES caf7bf3fc68f mysql:5.7 "docker-entrypoint.s…" 8 seconds ago Up 6 seconds 33060/tcp, 0.0.0.0:3307->3306/tcp, :::3307->3306/tcp mysql-slave ~~~ ### 4.2.4 MySQL从节点配置 ~~~powershell # vim /opt/mysql-slave/conf/my.cnf # cat /opt/mysql-slave/conf/my.cnf [client] default-character-set=utf8 [mysql] default-character-set=utf8 [mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake skip-name-resolve server_id=2 log-bin=mysql-bin read-only=1 binlog-do-db=kubemsb_test replicate-ignore-db=mysql replicate-ignore-db=sys replicate-ignore-db=information_schema replicate-ignore-db=performance_schema ~~~ ### 4.2.5 master节点配置 ~~~powershell # mysql -h 192.168.255.157 -uroot -proot -P 3306 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.37 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> ~~~ ~~~powershell 授权 MySQL [(none)]> grant replication slave on *.* to 'backup'@'%' identified by '123456'; ~~~ ~~~powershell 重启容器，使用配置生效 # docker restart mysql-master ~~~ ~~~powershell 查看状态 MySQL [(none)]> show master status\G *************************** 1. row *************************** File: mysql-bin.000001 Position: 154 Binlog_Do_DB: kubemsb_test Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) ~~~ ### 4.2.6 slave节点配置 ~~~powershell # docker restart mysql-slave ~~~ ~~~powershell # mysql -h 192.168.255.157 -uroot -proot -P 3307 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.37 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]> ~~~ ~~~powershell MySQL [(none)]> change master to master_host='mysql-master', master_user='backup', master_password='123456', master_log_file='mysql-bin.000001', master_log_pos=154, master_port=3306; ~~~ ~~~powershell MySQL [(none)]> start slave; ~~~ ~~~powershell MySQL [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: mysql-master Master_User: backup Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 154 Relay_Log_File: e0872f94c377-relay-bin.000002 Relay_Log_Pos: 320 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: mysql,sys,information_schema,performance_schema Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 154 Relay_Log_Space: 534 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 0130b415-8b21-11ec-8982-0242ac110002 Master_Info_File: /var/lib/mysql/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec) ~~~ ### 4.2.7 验证MySQL集群可用性 ~~~powershell 在MySQL Master节点添加kubemsb_test数据库 # mysql -h 192.168.255.157 -uroot -proot -P3306 MySQL [(none)]> create database kubemsb_test; Query OK, 1 row affected (0.00 sec) MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | kubemsb_test | | | mysql | | performance_schema | | sys | +--------------------+ 6 rows in set (0.00 sec) ~~~ ~~~powershell 在MySQL Slave节点查看同步情况 # mysql -h 192.168.255.157 -uroot -proot -P3307 MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | kubemsb_test | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.00 sec) ~~~ # 五、使用Docker容器实现Oracle部署 ## 5.1 获取参考资料 ![image-20220212111348491](../../img/docker_container_enterprice/image-20220212111348491.png) ![image-20220212111511130](../../img/docker_container_enterprice/image-20220212111511130.png) ## 5.2 运行oracle容器 ~~~powershell # docker pull oracleinanutshell/oracle-xe-11g ~~~ ~~~powershell # docker run -h oracle --name oracle -d -p 49160:22 -p 49161:1521 -p 49162:8080 oracleinanutshell/oracle-xe-11g 237db949020abf2cee12e3193fa8a34d9dfadaafd9d5604564668d4472abe0b2 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 237db949020a oracleinanutshell/oracle-xe-11g "/bin/sh -c '/usr/sb…" 7 seconds ago Up 4 seconds 0.0.0.0:49160->22/tcp, :::49160->22/tcp, 0.0.0.0:49161->1521/tcp, :::49161->1521/tcp, 0.0.0.0:49162->8080/tcp, :::49162->8080/tcp oracle ~~~ ~~~powershell 说明： 49160 为ssh端口 49161 为sqlplus端口 49162 为oem端口 ~~~ ~~~powershell oracle数据库连接信息 port:49161 sid:xe username:system password:oracle SYS用户密码为:oracle ~~~ ## 5.3 下载客户端连接工具 下载链接地址：https://www.oracle.com/tools/downloads/sqldev-downloads.html ![image-20220212115409934](../../img/docker_container_enterprice/image-20220212115409934.png) ![image-20220212115556498](../../img/docker_container_enterprice/image-20220212115556498.png) ![image-20220212115858640](../../img/docker_container_enterprice/image-20220212115858640.png) ![image-20220212115929175](../../img/docker_container_enterprice/image-20220212115929175.png) ![image-20220212120017387](../../img/docker_container_enterprice/image-20220212120017387.png) ![image-20220212120525692](../../img/docker_container_enterprice/image-20220212120525692.png) ![image-20220212120808803](../../img/docker_container_enterprice/image-20220212120808803.png) ![image-20220212120839513](../../img/docker_container_enterprice/image-20220212120839513.png) ![image-20220212120910215](../../img/docker_container_enterprice/image-20220212120910215.png) ![image-20220212120952721](../../img/docker_container_enterprice/image-20220212120952721.png) # 六、使用Docker容器实现ElasticSearch+Kibana部署 ## 6.1 获取参考资料 ### 6.1.1 ES部署参考资料 ![image-20220212211742909](../../img/docker_container_enterprice/image-20220212211742909.png) ![image-20220212211840530](../../img/docker_container_enterprice/image-20220212211840530.png) ![image-20220212211857732](../../img/docker_container_enterprice/image-20220212211857732.png) ![image-20220212211950202](../../img/docker_container_enterprice/image-20220212211950202.png) ### 6.1.2 Kibana部署参考资料 ![image-20220212212223735](../../img/docker_container_enterprice/image-20220212212223735.png) ![image-20220212212245791](../../img/docker_container_enterprice/image-20220212212245791.png) ![image-20220212212305429](../../img/docker_container_enterprice/image-20220212212305429.png) ![image-20220212212341841](../../img/docker_container_enterprice/image-20220212212341841.png) ## 6.2 ES部署 ~~~powershell # docker pull elasticsearch:7.17.0 ~~~ ~~~powershell # mkdir -p /opt/es/config # mkdir -p /opt/es/data ~~~ ~~~powershell # echo "http.host: 0.0.0.0" >> /opt/es/config/elasticsearch.yml ~~~ ~~~powershell # chmod -R 777 /opt/es/ ~~~ ~~~powershell # docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \ -e "discovery.type=single-node" \ -e ES_JAVA_OPTS="-Xms64m -Xmx512m" \ -v /opt/es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /opt/es/data:/usr/share/elasticsearch/data \ -v /opt/es/plugins:/usr/share/elasticsearch/plugins \ -d elasticsearch:7.17.0 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e1c306e6e5a3 elasticsearch:7.17.0 "/bin/tini -- /usr/l…" 22 seconds ago Up 20 seconds 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp elasticsearch ~~~ ![image-20220212224446838](../../img/docker_container_enterprice/image-20220212224446838.png) ## 6.3 Kibana部署 ~~~powershell # docker pull kibana:7.17.0 ~~~ ~~~powershell # docker run --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.255.157:9200 -p 5601:5601 \ -d kibana:7.17.0 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fb60e73f9cd5 kibana:7.17.0 "/bin/tini -- /usr/l…" 2 minutes ago Up 2 minutes 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp kibana ~~~ ![image-20220212224524598](../../img/docker_container_enterprice/image-20220212224524598.png) # 七、使用Docker容器实现Redis部署 ## 7.1 获取参考资料 ![image-20220212225251173](../../img/docker_container_enterprice/image-20220212225251173.png) ![image-20220212225313006](../../img/docker_container_enterprice/image-20220212225313006.png) ![image-20220212225336437](../../img/docker_container_enterprice/image-20220212225336437.png) ![image-20220212225412367](../../img/docker_container_enterprice/image-20220212225412367.png) ## 7.2 运行Redis容器 ~~~powershell # mkdir -p /opt/redis/conf ~~~ ~~~powershell # touch /opt/redis/conf/redis.conf ~~~ ~~~powershell # docker run -p 6379:6379 --name redis -v /opt/redis/data:/data \ -v /opt/redis/conf:/etc/redis \ -d redis redis-server /etc/redis/redis.conf ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9bd2b39cd92a redis "docker-entrypoint.s…" 44 seconds ago Up 42 seconds 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp redis ~~~ ## 7.3 验证 ~~~powershell # wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo ~~~ ~~~powershell # yum -y install redis ~~~ ~~~powershell # redis-cli -h 192.168.255.157 -p 6379 192.168.255.157:6379> set test1 a OK 192.168.255.157:6379> get test1 "a" ~~~ ## 7.4 Redis集群 安装redis-cluster；3主3从方式，从为了同步备份，主进行slot数据分片 ~~~powershell 编辑运行多个redis容器脚本文件 # vim redis-cluster.sh # cat redis-cluster.sh for port in $(seq 8001 8006); \ do \ mkdir -p /mydata/redis/node-${port}/conf touch /mydata/redis/node-${port}/conf/redis.conf cat << EOF >/mydata/redis/node-${port}/conf/redis.conf port ${port} cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000 cluster-announce-ip 192.168.255.157 cluster-announce-port ${port} cluster-announce-bus-port 1${port} appendonly yes EOF docker run -p ${port}:${port} -p 1${port}:1${port} --name redis-${port} \ -v /mydata/redis/node-${port}/data:/data \ -v /mydata/redis/node-${port}/conf/redis.conf:/etc/redis/redis.conf \ -d redis:5.0.7 redis-server /etc/redis/redis.conf; \ done ~~~ ~~~powershell 执行脚本 # sh redis-cluster.sh ~~~ ~~~powershell 查看已运行容器 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8d53864a98ce redis:5.0.7 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:8006->8006/tcp, :::8006->8006/tcp, 6379/tcp, 0.0.0.0:18006->18006/tcp, :::18006->18006/tcp redis-8006 e2b5da0f0605 redis:5.0.7 "docker-entrypoint.s…" 2 minutes ago Up About a minute 0.0.0.0:8005->8005/tcp, :::8005->8005/tcp, 6379/tcp, 0.0.0.0:18005->18005/tcp, :::18005->18005/tcp redis-8005 70e8e8f15aea redis:5.0.7 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:8004->8004/tcp, :::8004->8004/tcp, 6379/tcp, 0.0.0.0:18004->18004/tcp, :::18004->18004/tcp redis-8004 dff8e4bf02b4 redis:5.0.7 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:8003->8003/tcp, :::8003->8003/tcp, 6379/tcp, 0.0.0.0:18003->18003/tcp, :::18003->18003/tcp redis-8003 c34dc4c423ef redis:5.0.7 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:8002->8002/tcp, :::8002->8002/tcp, 6379/tcp, 0.0.0.0:18002->18002/tcp, :::18002->18002/tcp redis-8002 b8cb5feffb43 redis:5.0.7 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp, 6379/tcp, 0.0.0.0:18001->18001/tcp, :::18001->18001/tcp redis-8001 ~~~ ~~~powershell 登录redis容器 # docker exec -it redis-8001 bash root@b8cb5feffb43:/data# ~~~ ~~~powershell 创建redis-cluster root@b8cb5feffb43:/data# redis-cli --cluster create 192.168.255.157:8001 192.168.255.157:8002 192.168.255.157:8003 192.168.255.157:8004 192.168.255.157:8005 192.168.255.157:8006 --cluster-replicas 1 ~~~ ~~~powershell 输出： >>> Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 192.168.255.157:8005 to 192.168.255.157:8001 Adding replica 192.168.255.157:8006 to 192.168.255.157:8002 Adding replica 192.168.255.157:8004 to 192.168.255.157:8003 >>> Trying to optimize slaves allocation for anti-affinity [WARNING] Some slaves are in the same host as their master M: abd07f1a2679fe77558bad3ff4b7ab70ec41efa5 192.168.255.157:8001 slots:[0-5460] (5461 slots) master M: 40e69202bb3eab13a8157c33da6240bb31f2fd6f 192.168.255.157:8002 slots:[5461-10922] (5462 slots) master M: 9a927abf3c2982ba9ffdb29176fc8ffa77a2cf03 192.168.255.157:8003 slots:[10923-16383] (5461 slots) master S: 81d0a4056328830a555fcd75cf523d4c9d52205c 192.168.255.157:8004 replicates 9a927abf3c2982ba9ffdb29176fc8ffa77a2cf03 S: 8121a28519e5b52e4817913aa3969d9431bb68af 192.168.255.157:8005 replicates abd07f1a2679fe77558bad3ff4b7ab70ec41efa5 S: 3a8dd5343c0b8f5580bc44f6b3bb5b4371d4dde5 192.168.255.157:8006 replicates 40e69202bb3eab13a8157c33da6240bb31f2fd6f Can I set the above configuration? (type 'yes' to accept): yes 输入yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join ..... >>> Performing Cluster Check (using node 192.168.255.157:8001) M: abd07f1a2679fe77558bad3ff4b7ab70ec41efa5 192.168.255.157:8001 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 81d0a4056328830a555fcd75cf523d4c9d52205c 192.168.255.157:8004 slots: (0 slots) slave replicates 9a927abf3c2982ba9ffdb29176fc8ffa77a2cf03 M: 40e69202bb3eab13a8157c33da6240bb31f2fd6f 192.168.255.157:8002 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 8121a28519e5b52e4817913aa3969d9431bb68af 192.168.255.157:8005 slots: (0 slots) slave replicates abd07f1a2679fe77558bad3ff4b7ab70ec41efa5 M: 9a927abf3c2982ba9ffdb29176fc8ffa77a2cf03 192.168.255.157:8003 slots:[10923-16383] (5461 slots) master 1 additional replica(s) S: 3a8dd5343c0b8f5580bc44f6b3bb5b4371d4dde5 192.168.255.157:8006 slots: (0 slots) slave replicates 40e69202bb3eab13a8157c33da6240bb31f2fd6f [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. ~~~ # 八、使用Docker容器实现RabbitMQ部署 ## 8.1 获取参考资料 ![image-20220213123228483](../../img/docker_container_enterprice/image-20220213123228483.png) ![image-20220213123307806](../../img/docker_container_enterprice/image-20220213123307806.png) ![image-20220213123355531](../../img/docker_container_enterprice/image-20220213123355531.png) ![image-20220213123503083](../../img/docker_container_enterprice/image-20220213123503083.png) ## 8.2 部署RabbitMQ > 部署带管理控制台的RabbitMQ ~~~powershell # docker run -d --name rabbitmq -p 5671:5671 -p 5672:5672 -p 4369:4369 -p 25672:25672 -p 15671:15671 -p 15672:15672 -v /opt/rabbitmq:/var/lib/rabbitmq rabbitmq:management ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 97d28093faa4 rabbitmq:management "docker-entrypoint.s…" 11 seconds ago Up 6 seconds 0.0.0.0:4369->4369/tcp, :::4369->4369/tcp, 0.0.0.0:5671-5672->5671-5672/tcp, :::5671-5672->5671-5672/tcp, 0.0.0.0:15671-15672->15671-15672/tcp, :::15671-15672->15671-15672/tcp, 0.0.0.0:25672->25672/tcp, :::25672->25672/tcp, 15691-15692/tcp rabbitmq ~~~ ~~~powershell 端口说明： 4369, 25672 (Erlang发现&集群端口) 5672, 5671 (AMQP端口) 15672 (web管理后台端口) 61613, 61614 (STOMP协议端口) 1883, 8883 (MQTT协议端口) ~~~ ![image-20220213124157710](../../img/docker_container_enterprice/image-20220213124157710.png) ![image-20220213124232819](../../img/docker_container_enterprice/image-20220213124232819.png) ![image-20220213124302137](../../img/docker_container_enterprice/image-20220213124302137.png) ================================================ FILE: docs/cloud/docker/docker_date.md ================================================ # Docker容器数据持久化存储机制 # 一、Docker容器数据持久化存储介绍 - 物理机或虚拟机数据持久化存储 - 由于物理机或虚拟机本身就拥有大容量的磁盘，所以可以直接把数据存储在物理机或虚拟机本地文件系统中，亦或者也可以通过使用额外的存储系统（NFS、GlusterFS、Ceph等）来完成数据持久化存储。 - Docker容器数据持久化存储 - 由于Docker容器是由容器镜像生成的，所以一般容器镜像中包含什么文件或目录，在容器启动后，我们依旧可以看到相同的文件或目录。 - 由于Docker容器属于“用后即焚”型计算资源，因此Docker容器不适合做数据持久化存储 # 二、Docker容器数据持久化存储方式 Docker提供三种方式将数据从宿主机挂载到容器中： - docker run -v - 运行容器时，直接挂载本地目录至容器中 - volumes - Docker管理宿主机文件系统的一部分(/var/lib/docker/volumes) - 是Docker默认存储数据方式 - bind mounts - 将宿主机上的任意位置文件或目录挂载到容器中 # 三、Docker容器数据持久化存储方式应用案例演示 ## 3.1 docker run -v ### 3.1.1 未挂载本地目录 ~~~powershell 运行一个容器，未挂载本地目录 # docker run -d --name web1 nginx:latest ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c4ad9f2c15fa nginx:latest "/docker-entrypoint.…" 46 seconds ago Up 44 seconds 80/tcp web1 ~~~ ~~~powershell 使用curl命令访问容器 # curl http://172.17.0.2 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ ~~~powershell 查看容器中/usr/share/nginx/html目录中目录或子目录 # docker exec web ls /usr/share/nginx/html 50x.html index.html ~~~ ### 3.1.2 挂载本地目录 ~~~powershell 创建本地目录 # mkdir /opt/wwwroot ~~~ ~~~powershell 向本地目录中添加index.html文件 # echo 'kubemsb' > /opt/wwwroot/index.html ~~~ ~~~powershell 运行web2容器，把/opt/wwwroot目录挂载到/usr/share/nginx/html目录中 # docker run -d --name web2 -v /opt/wwwroot/:/usr/share/nginx/html/ nginx:latest ~~~ ~~~powershell 查看容器IP地址 # docker inspect web2 ...... "IPAddress": "172.17.0.3", ...... ~~~ ~~~powershell 使用curl命令访问容器 # curl http://172.17.0.3 kubemsb ~~~ ### 3.1.3 未创建本地目录 ~~~powershell 运行web3容器，挂载未创建的本地目录，启动容器时将自动创建本地目录 # docker run -d --name web3 -v /opt/web3root/:/usr/share/nginx/html/ nginx:latest ~~~ ~~~powershell 往自动创建的目录中添加一个index.html文件 # echo "kubemsb web3" > /opt/web3root/index.html ~~~ ~~~powershell 在容器中执行查看文件命令 # docker exec web3 cat /usr/share/nginx/html/index.html kubemsb web3 ~~~ ## 3.2 volumes ### 3.2.1 创建数据卷 ~~~powershell 创建一个名称为nginx-vol的数据卷 # docker volume create nginx-vol nginx-vol ~~~ ~~~powershell 确认数据卷创建后的位置 # ls /var/lib/docker/volumes/ backingFsBlockDev metadata.db nginx-vol ~~~ ~~~powershell 查看已经创建数据卷 # docker volume ls DRIVER VOLUME NAME local nginx-vol ~~~ ~~~powershell 查看数据卷详细信息 # docker volume inspect nginx-vol [ { "CreatedAt": "2022-02-08T14:36:16+08:00", "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/nginx-vol/_data", "Name": "nginx-vol", "Options": {}, "Scope": "local" } ] ~~~ ### 3.2.2 使用数据卷 ~~~powershell 运行web4容器，使用--mount选项，实现数据卷挂载 # docker run -d --name web4 --mount src=nginx-vol,dst=/usr/share/nginx/html nginx:latest ~~~ 或 ~~~powershell 运行web4容器，使用-v选项，实现数据卷挂载 # docker run -d --name web4 -v nginx-vol:/usr/share/nginx/html/ nginx:latest ~~~ ~~~powershell 查看容器运行后数据卷中文件或子目录 # ls /var/lib/docker/volumes/nginx-vol/_data/ 50x.html index.html ~~~ ~~~powershell 使用curl命令访问容器 # curl http://172.17.0.2 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ ~~~powershell 修改index.html文件内容 # echo "web4" > /var/lib/docker/volumes/nginx-vol/_data/index.html ~~~ ~~~powershell 再次使用curl命令访问容器 # curl http://172.17.0.2 web4 ~~~ ## 3.3 bind mounts ~~~powershell 创建用于容器挂载的目录web5root # mkdir /opt/web5root ~~~ ~~~powershell 运行web5容器并使用bind mount方法实现本地任意目录挂载 # docker run -d --name web5 --mount type=bind,src=/opt/web5root,dst=/usr/share/nginx/html nginx:latest ~~~ ~~~powershell 查看已挂载目录，里面没有任何数据 # ls /opt/web5root/ ~~~ ~~~powershell 添加内容至/opt/web5root/index.html中 # echo "web5" > /opt/web5root/index.html ~~~ ~~~powershell 使用curl命令访问容器 # curl http://172.17.0.3 web5 ~~~ ================================================ FILE: docs/cloud/docker/docker_devops.md ================================================ # 基于Docker容器DevOps应用方案 企业业务代码发布系统 # 一、企业业务代码发布方式 ## 1.1 传统方式 - 以物理机或虚拟机为颗粒度部署 - 部署环境比较复杂，需要有先进的自动化运维手段 - 出现问题后重新部署成本大，一般采用集群方式部署 - 部署后以静态方式展现 ## 1.2 容器化方式 - 以容器为颗粒度部署 - 部署方式简单，启动速度快 - 一次构建可到处运行 - 出现故障后，可随时恢复 - 可同时部署多套环境（测试、预发布、生产环境等） # 二、企业业务代码发布逻辑图 ![image-20220223152003734](../../img/docker_devops/image-20220223152003734.png) # 三、企业业务代码发布工具及流程图 ## 3.1 工具 | 序号 | 工具 | 工具用途 | | ---- | ------- | ------------------------------------------------------------ | | 1 | git | 用于提交业务代码或克隆业务代码仓库 | | 2 | gitlab | 用于存储业务代码 | | 3 | jenkins | 用于利用插件完成业务代码编译、构建、推送至Harbor容器镜像仓库及项目部署 | | 4 | tomcat | 用于运行JAVA业务代码 | | 5 | maven | 用于编译业务代码 | | 6 | harbor | 用于存储业务代码构建的容器镜像存储 | | 7 | docker | 用于构建容器镜像，部署项目 | ## 3.2 流程图 > 本次部署Java代码包。 ![image-20220223163453076](../../img/docker_devops/image-20220223163453076.png) # 四、企业业务代码发布系统环境部署 ## 4.1 主机规划 | 序号 | 主机名 | 主机IP | 主机功能 | 软件 | | ---- | -------------- | ------------- | ---------------------------- | -------------------- | | 1 | dev | 192.168.10.20 | 开发者 项目代码 solo | git | | 2 | gitlab-server | 192.168.10.21 | 代码仓库 | gitlab-ce | | 3 | jenkins-server | 192.168.10.22 | 编译代码、打包镜像、项目发布 | jenkins、docker、git | | 4 | harbor-server | 192.168.10.23 | 存储容器镜像 | harbor、docker | | 5 | web-server | 192.168.10.24 | 运行容器，项目上线 | docker | ## 4.2 主机准备 ### 4.2.1 主机名配置 ~~~powershell # hostnamectl set-hostname xxx ~~~ > 根据主机规划实施配置 ### 4.2.2 主机IP地址配置 ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" 配置为静态IP DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.2x" 把2x替换为对应的IP地址 PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ### 4.2.3 主机名与IP地址解析配置 ~~~powershell # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.20 dev 192.168.10.21 gitlab-server 192.168.10.22 jenkins-server 192.168.10.23 harobr-server 192.168.10.24 web-server ~~~ ### 4.2.4 主机安全设置 ~~~powershell # systemctl stop firewalld;systemctl disable firewalld ~~~ ~~~powershell # firewall-cmd --state ~~~ ~~~powershell # sestatus ~~~ ### 4.2.5 主机时间同步 ~~~powershell # crontab -e # crotab -l 0 */1 * * * ntpdate time1.aliyun.com ~~~ ## 4.3 主机中工具安装 ### 4.3.1 dev主机 >下载项目及上传代码至代码仓库 ~~~powershell # yum -y install git ~~~ ### 4.3.2 gitlab-server主机 #### 4.3.2.1 获取YUM源 ![image-20220224134917428](../../img/docker_devops/image-20220224134917428.png) ![image-20220224134941802](../../img/docker_devops/image-20220224134941802.png) ![image-20220224135000130](../../img/docker_devops/image-20220224135000130.png) ![image-20220224135021076](../../img/docker_devops/image-20220224135021076.png) ~~~powershell # cat /etc/yum.repos.d/gitlab.repo [gitlab] name=gitlab-ce baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7 enabled=1 gpgcheck=0 ~~~ #### 4.3.2.2 gitlab-ce安装 ~~~powershell # yum -y install gitlab-ce ~~~ #### 4.3.2.3 gitlab-ce配置 ~~~powershell # vim /etc/gitlab/gitlab.rb 32 external_url 'http://192.168.10.21' ~~~ #### 4.3.2.4 启动gitlab-ce ~~~powershell # gitlab-ctl reconfigure ~~~ ~~~powershell # gitlab-ctl status ~~~ #### 4.3.2.5 访问gitlab-ce ~~~powershell # cat /etc/gitlab/initial_root_password ...... Password: znS4Bqlp0cfYUKg2dHzFiNCAN0GnhtnD4ENjEtEXMVE= ~~~ ![image-20220224140418176](../../img/docker_devops/image-20220224140418176.png) ![image-20220224140436172](../../img/docker_devops/image-20220224140436172.png) ### 4.3.3 jenkins-server主机 #### 4.3.3.1 jdk安装 ~~~powershell # ls jdk-8u191-linux-x64.tar.gz ~~~ ~~~powershell # mv jdk1.8.0_191 /usr/local/jdk ~~~ ~~~powershell # vim /etc/profile # cat /etc/profile ...... export JAVA_HOME=/usr/local/jdk export PATH=${JAVA_HOME}/bin:$PATH ~~~ ~~~powershell # source /etc/profile ~~~ ~~~powershell # java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) ~~~ #### 4.3.3.2 jenkins安装 ##### 4.3.3.2.1 安装 ![image-20220224141530225](../../img/docker_devops/image-20220224141530225.png) ![image-20220224141610569](../../img/docker_devops/image-20220224141610569.png) ![image-20220224141720927](../../img/docker_devops/image-20220224141720927.png) ~~~powershell # wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo ~~~ ~~~powershell # rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key ~~~ ![image-20220224142031988](../../img/docker_devops/image-20220224142031988.png) ~~~powershell # wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo ~~~ ~~~powershell # yum -y install jenkins ~~~ ##### 4.3.3.2.2 jenkins配置 ~~~powershell # vim /etc/init.d/jenkins 在81行下面添加如下内容： 82 /usr/local/jdk/bin/java ~~~ ~~~powershell # vim /etc/sysconfig/jenkins 在19行双引号中添加jdk中java命令路径 19 JENKINS_JAVA_CMD="/usr/local/jdk/bin/java" ~~~ ##### 4.3.3.2.3 jenkins启动 ~~~powershell # chkconfig --list 注：该输出结果只显示 SysV 服务，并不包含 原生 systemd 服务。SysV 配置数据 可能被原生 systemd 配置覆盖。 要列出 systemd 服务，请执行 'systemctl list-unit-files'。 查看在具体 target 启用的服务请执行 'systemctl list-dependencies [target]'。 jenkins 0:关 1:关 2:开 3:开 4:开 5:开 6:关 netconsole 0:关 1:关 2:关 3:关 4:关 5:关 6:关 network 0:关 1:关 2:开 3:开 4:开 5:开 6:关 # chkconfig jenkins on ~~~ ~~~powershell # systemctl start jenkins ~~~ ##### 4.3.3.2.4 jenkins访问 ~~~powershell # cat /var/lib/jenkins/secrets/initialAdminPassword 3363d658a1a5481bbe51a1ece1eb08ab ~~~ ![image-20220224142955854](../../img/docker_devops/image-20220224142955854.png) ##### 4.3.3.2.5 jenkins初始化配置 ![image-20220224173833454](../../img/docker_devops/image-20220224173833454.png) ![image-20220224174018298](../../img/docker_devops/image-20220224174018298.png) ![image-20220224174041874](../../img/docker_devops/image-20220224174041874.png) ![image-20220224174442874](../../img/docker_devops/image-20220224174442874.png) ![image-20220224174507233](../../img/docker_devops/image-20220224174507233.png) ![image-20220224174541367](../../img/docker_devops/image-20220224174541367.png) ![image-20220224174601389](../../img/docker_devops/image-20220224174601389.png) #### 4.3.3.3 git安装 ~~~powershell # yum -y install git ~~~ #### 4.3.3.4 maven安装 ##### 4.3.3.4.1 获取maven安装包 ![image-20220224174735575](../../img/docker_devops/image-20220224174735575.png) ![image-20220224174855779](../../img/docker_devops/image-20220224174855779.png) ~~~powershell # wget https://dlcdn.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz ~~~ ##### 4.3.3.4.2 maven安装 ~~~powershell # ls apache-maven-3.8.4-bin.tar.gz ~~~ ~~~powershell # tar xf apache-maven-3.8.4-bin.tar.gz # ls apache-maven-3.8.4 ~~~ ~~~powershell # mv apache-maven-3.8.4 /usr/local/mvn ~~~ ~~~powershell # vim /etc/profile ...... export JAVA_HOME=/usr/local/jdk export MAVEN_HOME=/usr/local/mvn export PATH=${JAVA_HOME}/bin:${MAVEN_HOME}/bin:$PATH ~~~ ~~~powershell # source /etc/profile ~~~ ~~~powershell # mvn -v Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) Maven home: /usr/local/mvn Java version: 1.8.0_191, vendor: Oracle Corporation, runtime: /usr/local/jdk/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", family: "unix" ~~~ #### 4.3.3.5 docker安装 ~~~powershell # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell # yum -y install docker-ce ~~~ ~~~powershell # systemctl enable docker # systemctl start docker ~~~ ### 4.3.4 harbor-server主机 #### 4.3.4.1 docker安装 ~~~powershell # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell # yum -y install docker-ce ~~~ ~~~powershell # systemctl enable docker # systemctl start docker ~~~ #### 4.3.4.2 docker-compose安装 ##### 4.3.4.2.1 获取docker-compose文件 ![image-20220224180711917](../../img/docker_devops/image-20220224180711917.png) ![image-20220224180732745](../../img/docker_devops/image-20220224180732745.png) ![image-20220224180754614](../../img/docker_devops/image-20220224180754614.png) ![image-20220224180842212](../../img/docker_devops/image-20220224180842212.png) ~~~powershell # wget https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 ~~~ ##### 4.3.4.2.2 docker-compose安装及测试 ~~~powershell # ls docker-compose-linux-x86_64 ~~~ ~~~powershell # chmod +x docker-compose-linux-x86_64 ~~~ ~~~powershell # mv docker-compose-linux-x86_64 /usr/bin/docker-compose ~~~ ~~~powershell # docker-compose version Docker Compose version v2.2.3 ~~~ #### 4.3.4.3 harbor部署 ##### 4.3.4.3.1 harbor部署文件获取 ![image-20220224181602622](../../img/docker_devops/image-20220224181602622.png) ![image-20220224181626286](../../img/docker_devops/image-20220224181626286.png) ![image-20220224181656604](../../img/docker_devops/image-20220224181656604.png) ![image-20220224181746808](../../img/docker_devops/image-20220224181746808.png) ![image-20220224181829179](../../img/docker_devops/image-20220224181829179.png) ~~~powershell # wget https://github.com/goharbor/harbor/releases/download/v2.4.1/harbor-offline-installer-v2.4.1.tgz ~~~ ##### 4.3.4.3.2 harbor部署 ~~~powershell # ls harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell # tar xf harbor-offline-installer-v2.4.1.tgz -C /home ~~~ ~~~powershell # cd /home # ls harbor [root@harbor-server home]# cd harbor/ [root@harbor-server harbor]# ls common.sh harbor.v2.4.1.tar.gz harbor.yml.tmpl install.sh LICENSE prepare ~~~ ~~~powershell # mv harbor.yml.tmpl harbor.yml ~~~ ~~~powershell [root@harbor-server harbor]# vim harbor.yml [root@harbor-server harbor]# cat harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: 192.168.10.23 修改 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config #https: 注释 # https port for harbor, default is 443 # port: 443 注释 # The path of cert and key files for nginx # certificate: /your/certificate/path 注释 # private_key: /your/private/key/path 注释 ~~~ ~~~powershell [root@harbor-server harbor]# ./prepare ~~~ ~~~powershell [root@harbor-server harbor]# ./install.sh ~~~ ~~~powershell [root@harbor-server harbor]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 12605eae32bb goharbor/harbor-jobservice:v2.4.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice 85849b46d56d goharbor/nginx-photon:v2.4.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp nginx 6a18e370354f goharbor/harbor-core:v2.4.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core d115229ef49d goharbor/harbor-portal:v2.4.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal f5436556dd32 goharbor/harbor-db:v2.4.1 "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db 7fb8c4945abe goharbor/harbor-registryctl:v2.4.1 "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl d073e5da1399 goharbor/redis-photon:v2.4.1 "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis 7c09362c986b goharbor/registry-photon:v2.4.1 "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry 55d7f39909e3 goharbor/harbor-log:v2.4.1 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log ~~~ ### 4.3.5 web-server > docker安装 ~~~powershell # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell # yum -y install docker-ce ~~~ ~~~powershell # systemctl enable docker # systemctl start docker ~~~ ## 4.4 工具集成配置 ### 4.4.1 配置docker主机使用harbor #### 4.4.1.1 jenkins-server ~~~powershell [root@jenkins-server ~]# vim /etc/docker/daemon.json [root@jenkins-server ~]# cat /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.23"] } ~~~ ~~~powershell [root@jenkins-server ~]# systemctl restart docker ~~~ ~~~powershell [root@jenkins-server ~]# docker login 192.168.10.23 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ #### 4.4.1.2 harbor-server ~~~powershell [root@harbor-server harbor]# vim /etc/docker/daemon.json [root@harbor-server harbor]# cat /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.23"] } ~~~ ~~~powershell [root@harbor-server harbor]# docker-compose down ~~~ ~~~powershell [root@harbor-server harbor]# systemctl restart docker ~~~ ~~~powershell [root@harbor-server harbor]# docker-compose up -d ~~~ #### 4.4.1.3 web-server ~~~powershell [root@web-server ~]# vim /etc/docker/daemon.json [root@web-server ~]# cat /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.23"] } ~~~ ~~~powershell [root@web-server ~]# systemctl restart docker ~~~ ~~~powershell [root@web-server ~]# docker login 192.168.10.23 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ### 4.4.2 配置jenkins使用docker > 在jenkins-server主机上配置 ~~~powershell 验证系统中是否有jenkins用户 [root@jenkins-server ~]# grep jenkins /etc/passwd jenkins:x:997:995:Jenkins Automation Server:/var/lib/jenkins:/bin/false ~~~ ~~~powershell 验证系统中是否有docker用户及用户组 [root@jenkins-server ~]# grep docker /etc/group docker:x:993: ~~~ ~~~powershell 添加jenkins用户到docker用户组 [root@jenkins-server ~]# usermod -G docker jenkins [root@jenkins-server ~]# grep docker /etc/group docker:x:993:jenkins ~~~ ~~~powershell 重启jenkins服务 [root@jenkins-server ~]# systemctl restart jenkins ~~~ ### 4.4.3 密钥配置 #### 4.4.3.1 dev主机至gitlab-ce ##### 4.4.3.1.1 dev主机生成密钥对 ~~~powershell [root@dev ~]# ssh-keygen ~~~ ##### 4.4.3.1.2 添加公钥至gitlab-ce ~~~powershell [root@dev ~]# cat /root/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy2PdvT9qX55CLZzzaaEf06x8gl3yHGfdJSmAp9L1Fdtcbd3yz3U0lgdOwWpB8fQ/A3HoUUTWCb1iC5WJBOvqkoD8rJ2xC3HJ62zjOjmqcn2fEs09CzJj3bCfahuqPzaPkIOoH42/Y2QdImQ7xZOqqjS7aIc5T2FjDLG3bMhaYFyvx18b1qiPACuh67iniPQnL667MFZ/0QGGVnQKwxop+SezhP9QqV1bvPk94eTdkERIBiY1CNcNmVryk6PzSKY8gfW++3TGN9F+knhMXcswFOu6FzqxcA3G+hYg+Io2HJaDrsfHGZ6CP5T9QiOlIWlNxz05BOK3OFQ5BPeomA+jv root@dev ~~~ ![image-20220224210606310](../../img/docker_devops/image-20220224210606310.png) ![image-20220224210748207](../../img/docker_devops/image-20220224210748207.png) ![image-20220224210823231](../../img/docker_devops/image-20220224210823231.png) #### 4.4.3.2 jenkins-server主机至gitlab-ce ##### 4.4.3.2.1 在jenkins-server生成密钥对 ~~~powershell [root@jenkins-server ~]# ssh-keygen ~~~ ##### 4.4.3.2.2 添加公钥至gitlab-ce ~~~powershell [root@jenkins-server ~]# cat /root/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyg3WaEm5yH9yva8Jm5wfTPwN3ROGMNPpAex8zYj+M1GesoMtE6gkiKHWydAJBiLuu/1fBx6HlgzzxghVj9oK4DmTRZQh2IZY4+zZIGBRaDBuBO1f7+SdVE/jZoLd1a+yZ3FQmy37AlXUcIKxbrDBtefvJ31faziWyZKvT4BGFJCznRU6AOxOg1pe4bWbWI+dGnMIIq7IhtK+6tY/w3OlF7xcWmrJP1oucpq33BYOrnRCL9EO5Zp2jcejDeG5UvXONG7CggT7FDhjwcCRZvX+AutDGAtgBckNXZjV9SDKWgDifCSDtDfV4Be4zb8b3hxtSMsbEY8YHxsThsmHrUkbz root@jenkins-server ~~~ ![image-20220224211329307](../../img/docker_devops/image-20220224211329307.png) #### 4.4.3.3 配置jenkins-sever主机的私钥到凭据列表 ~~~powershell [root@jenkins-server ~]# cat /root/.ssh/id_rsa -----BEGIN RSA PRIVATE KEY----- MIIEogIBAAKCAQEAsoN1mhJuch/cr2vCZucH0z8Dd0ThjDT6QHsfM2I/jNRnrKDL ROoJIih1snQCQYi7rv9Xwceh5YM88YIVY/aCuA5k0WUIdiGWOPs2SBgUWgwbgTtX +/knVRP42aC3dWvsmdxUJst+wJV1HCCsW6wwbXn7yd9X2s4lsmSr0+ARhSQs50VO gDsToNaXuG1m1iPnRpzCCKuyIbSvurWP8NzpRe8XFpqyT9aLnKat9wWDq50Qi/RD uWado3How3huVL1zjRuwoIE+xQ4Y8HAkWb1/gLrQxgLYAXJDV2Y1fUgyloA4nwkg 7Q31eAXuM2/G94cbUjLGxGPGB8bE4bJh61JG8wIDAQABAoIBAEOwy6BPyuelo1Y1 g3LnujTlWRgZ23kCAb7/sPYYFEb/qAxysIGCSVJVi0PO76gQBDM4iftmCsLv/+UI UbolGK5Ybuxj5lB9LeyPfaba0qTOoINhkFxwvvRo7V0Ar3BsKzywqoxHb9nxEoZG 8XSVl4t7zPlgonzK3MqHmAxwk9QrIB/rnjolHGN6HvfK2Cwq5WN1crspwQ+XDbRS J5qoAtv6PJzrU6QhJl/zSMCb0MytlIhZi+V+1yY/QhAYrWJgWypEwGlAXlVC90r4 twX1W/sl63xzFF396WjM1478yqpttvID06dKTC9T3y/k8lLmRNXwqmTCIm7C/jxP 9wjXJUECgYEA4r1N4AML7JpvE7hBRkreSdIIwoppRkBwl6gt/AJj8yt2ydQ8PY4P X3s5ZkCfPzHX2XsVUUcQpsBFcU2PyG29+qzt3XOmlGYgJG11xPQbwi95/u9VSd5u AuaNNa2YPw2teuM0hKVAl5knfy0+YHcOCdU14gHCCWsD4uOz5Zg9jVMCgYEAyYzv SBvCqbZ4d5agTn+ZiOkmgKVT4UVnmZPivFXnCWiIbX2fi3ok7jU1hZAs6lf6VlTU EPV8T1LwjO9yhFmicepzSl9lJCMbMXHt20OsqN0oUQFpoTQ07pbBE2K8c1IuQUEi B2SoLHqv7Ym9jHQqvT3DVhTiC+H2LwsgVRvvi+ECgYAxaID0xJUvnMOBr5ABykTA H1WrVs/z8AzY71v942N2VM1Q07/AxhkRfF+YqZJKCgl4KbsOeAbn31QCiZ1AVrGk U1SOAiqVgd+VMIkOPwdhfEkARZT3QNIGLcktnkNj0g4wjhwen4gAwO37Z5eFG8xi ViSkuC9ZMAmrwmSsLk2TYwKBgHQh0tYXuMiVLUCq99+DQnJS9S53FKfel900Cxc9 4AvZwZJlKgLx9EmVOyukcVzuKH6KDk9fQ6tpPNXYOoHsK9+7mYanBN4XpFmPLeCD U/9QvyQ9ziFmtYEsOD/1SmSgW6qZ3wOnigdnAeu6zA8b+GxmJCF7kuwJ3RIqNQ0V NafBAoGAXyynoTT2zugFq8jYRubxkMk7NdnTRAnGh+mlyrGGMsNLmPvfAw+6yKph 1fVHKXHtSrgtK0CVOIcmaH3r+LfG4Mfrjlq+8qiKcepBFvO9cZLNKn11vqQtzs7m y+ydl4xTcCPoAMDsVeamJ3fv+9nyXe5KqYtw+BJMjpP+PnNN2YQ= -----END RSA PRIVATE KEY----- ~~~ ![image-20220224212308684](../../img/docker_devops/image-20220224212308684.png) ![image-20220224212411414](../../img/docker_devops/image-20220224212411414.png) ![image-20220224212543257](../../img/docker_devops/image-20220224212543257.png) ![image-20220224212622017](../../img/docker_devops/image-20220224212622017.png) ![image-20220224212928853](../../img/docker_devops/image-20220224212928853.png) ![image-20220224213022249](../../img/docker_devops/image-20220224213022249.png) ## 4.5 jenkins插件安装 ### 4.5.1 maven integration > 用于编译JAVA项目 ![image-20220224214012429](../../img/docker_devops/image-20220224214012429.png) ![image-20220224214041517](../../img/docker_devops/image-20220224214041517.png) ![image-20220224214133277](../../img/docker_devops/image-20220224214133277.png) ![image-20220224214217988](../../img/docker_devops/image-20220224214217988.png) ### 4.5.2 git parameter > 用于基于git版本提交进行参数构建项目 ![image-20220224214317316](../../img/docker_devops/image-20220224214317316.png) ![image-20220224214329349](../../img/docker_devops/image-20220224214329349.png) ### 4.5.3 gitlab > 用于jenkins-server拉取项目 ![image-20220224214412283](../../img/docker_devops/image-20220224214412283.png) ![image-20220224214442366](../../img/docker_devops/image-20220224214442366.png) ### 4.5.4 Generic Webhook Trigger > 用于项目自动化构建 ![image-20220224214812077](../../img/docker_devops/image-20220224214812077.png) ![image-20220224214922182](../../img/docker_devops/image-20220224214922182.png) ### 4.5.5 ssh > 用于jenkins-server对web-server实施项目部署 ![image-20220224215008239](../../img/docker_devops/image-20220224215008239.png) ![image-20220224215019932](../../img/docker_devops/image-20220224215019932.png) ## 4.6 jenkins全局工具配置 ![image-20220224215857618](../../img/docker_devops/image-20220224215857618.png) ### 4.6.1 JDK配置 ~~~powershell [root@jenkins-server ~]# java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) [root@jenkins-server ~]# echo $JAVA_HOME /usr/local/jdk ~~~ ![image-20220224220023637](../../img/docker_devops/image-20220224220023637.png) ![image-20220224220323445](../../img/docker_devops/image-20220224220323445.png) ### 4.6.2 Git配置 ~~~powershell [root@jenkins-server ~]# git version git version 1.8.3.1 ~~~ ![image-20220224220502686](../../img/docker_devops/image-20220224220502686.png) ### 4.6.3 Maven配置 ~~~powershell [root@jenkins-server ~]# mvn --version Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) Maven home: /usr/local/mvn Java version: 1.8.0_191, vendor: Oracle Corporation, runtime: /usr/local/jdk/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.49.1.el7.x86_64", arch: "amd64", family: "unix" [root@jenkins-server ~]# echo $MAVEN_HOME /usr/local/mvn ~~~ ![image-20220224220529911](../../img/docker_devops/image-20220224220529911.png) ![image-20220224220733835](../../img/docker_devops/image-20220224220733835.png) ## 4.7 jenkins系统配置 > 主要配置jenkins-server通过ssh协议连接web-server ### 4.7.1 添加jenkins-server访问web-server凭据 ![image-20220224221320500](../../img/docker_devops/image-20220224221320500.png) ![image-20220224221347030](../../img/docker_devops/image-20220224221347030.png) ![image-20220224221416358](../../img/docker_devops/image-20220224221416358.png) ![image-20220224221609980](../../img/docker_devops/image-20220224221609980.png) ![image-20220224221643648](../../img/docker_devops/image-20220224221643648.png) ### 4.7.2 配置ssh协议连接主机 ![image-20220224221754566](../../img/docker_devops/image-20220224221754566.png) ![image-20220224221832480](../../img/docker_devops/image-20220224221832480.png) ![image-20220224221901770](../../img/docker_devops/image-20220224221901770.png) ![image-20220224222146699](../../img/docker_devops/image-20220224222146699.png) # 五、企业业务代码项目发布 ## 5.1 数据库管理系统部署 mariadb及创建项目数据库 ~~~powershell [root@web-server ~]# yum -y install mariadb mariadb-server ~~~ ~~~powershell [root@web-server ~]# systemctl enable mariadb [root@web-server ~]# systemctl start mariadb ~~~ ~~~powershell [root@web-server ~]# mysqladmin -uroot password 'abc123' ~~~ ~~~powershell [root@web-server ~]# mysql -uroot -pabc123 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 3 Server version: 5.5.68-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> ~~~ ~~~powershell MariaDB [(none)]> create database if not exists solo default charset utf8 collate utf8_general_ci; ~~~ ~~~powershell MariaDB [(none)]> grant all on solo.* to 'root'@'%' identified by "123456"; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> grant all on solo.* to 'root'@'localhost' identified by "123456"; Query OK, 0 rows affected (0.00 sec) ~~~ ## 5.2 项目代码获取 ![image-20220224223418318](../../img/docker_devops/image-20220224223418318.png) ~~~powershell # git clone --recurse-submodules https://gitee.com/dl88250/solo.git ~~~ ## 5.3 项目代码修改 ~~~powershell [root@dev ~]# ls solo ~~~ ~~~powershell [root@dev ~]# vim solo/src/main/resources/local.properties [root@dev ~]# cat solo/src/main/resources/local.properties # # Solo - A small and beautiful blogging system written in Java. # Copyright (c) 2010-present, b3log.org # # Solo is licensed under Mulan PSL v2. # You can use this software according to the terms and conditions of the Mulan PSL v2. # You may obtain a copy of Mulan PSL v2 at: # http://license.coscl.org.cn/MulanPSL2 # THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. # See the Mulan PSL v2 for more details. # # # Description: Solo local environment configurations. # Version: 1.1.3.15, Mar 17, 2019 # Author: Liang Ding # #### MySQL runtime #### runtimeDatabase=MYSQL jdbc.username=root jdbc.password=123456 jdbc.driver=com.mysql.cj.jdbc.Driver jdbc.URL=jdbc:mysql://192.168.10.24:3306/solo?useUnicode=yes&characterEncoding=UTF-8&useSSL=false&serverTimezone=UTC&allowPublicKeyRetrieval=true #### H2 runtime #### #runtimeDatabase=H2 #jdbc.username=root #jdbc.password= #jdbc.driver=org.h2.Driver #jdbc.URL=jdbc:h2:~/solo_h2/db;MODE=MYSQL # The minConnCnt MUST larger or equal to 3 jdbc.minConnCnt=5 jdbc.maxConnCnt=10 # The specific table name prefix jdbc.tablePrefix=b3_solo ~~~ ## 5.4 项目代码上传到gitlab ![image-20220224223855433](../../img/docker_devops/image-20220224223855433.png) ![image-20220224223917897](../../img/docker_devops/image-20220224223917897.png) ![image-20220224224014013](../../img/docker_devops/image-20220224224014013.png) ~~~powershell # git config --global user.name "dev" # git config --global user.email "dev@kubemsb.com" ~~~ ~~~powershell [root@dev solo]# git remote remove origin ~~~ ~~~powershell [root@dev solo]# git remote add origin git@192.168.10.21:root/solo.git ~~~ ~~~powershell [root@dev solo]# git add -A . [root@dev solo]# git commit -m "new" [master 3e39b0a] new 1 file changed, 1 insertion(+), 1 deletion(-) ~~~ ~~~powershell [root@dev solo]# git tag 1.0.0 ~~~ ~~~powershell [root@dev solo]# git push origin 1.0.0 ~~~ ~~~powershell [root@dev solo]# git push -u origin --all ~~~ ![image-20220224225051498](../../img/docker_devops/image-20220224225051498.png) ## 5.5 构建项目运行基础应用容器镜像 > 在harbor-server主机上操作 ### 5.5.1 创建项目目录 ~~~powershell [root@harbor-server ~]# mkdir tomcatdir [root@harbor-server ~]# cd tomcatdir ~~~ ### 5.5.2 生成Dockerfile文件 ~~~powershell [root@harbor-server tomcatdir]# echo "tomcat is running" >> index.html ~~~ ~~~powershell [root@harbor-server tomcatdir]# vim Dockerfile [root@harbor-server tomcatdir]# cat Dockerfile FROM centos:centos7 MAINTAINER "www.kubemsb.com" ENV VERSION=8.5.75 ENV JAVA_HOME=/usr/local/jdk ENV TOMCAT_HOME=/usr/local/tomcat RUN yum -y install wget RUN wget https://dlcdn.apache.org/tomcat/tomcat-8/v${VERSION}/bin/apache-tomcat-${VERSION}.tar.gz --no-check-certificate RUN tar xf apache-tomcat-${VERSION}.tar.gz RUN mv apache-tomcat-${VERSION} /usr/local/tomcat RUN rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/* RUN mkdir /usr/local/tomcat/webapps/ROOT ADD ./index.html /usr/local/tomcat/webapps/ROOT/ ADD ./jdk /usr/local/jdk RUN echo "export TOMCAT_HOME=/usr/local/tomcat" >> /etc/profile RUN echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile RUN echo "export PATH=${TOMCAT_HOME}/bin:${JAVA_HOME}/bin:$PATH" >> /etc/profile RUN echo "export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar" >> /etc/profile RUN source /etc/profile EXPOSE 8080 CMD ["/usr/local/tomcat/bin/catalina.sh","run"] ~~~ ~~~powershell [root@harbor-server tomcatdir]# ls Dockerfile index.html jdk ~~~ ### 5.5.3 使用docker build构建容器镜像 ~~~powershell [root@harbor-server tomcatdir]# docker build -t 192.168.10.23/library/tomcat:8575 . ~~~ ### 5.5.4 推送容器镜像至harbor容器镜像仓库 ~~~powershell [root@harbor-server tomcatdir]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.10.23/library/tomcat 8575 01c433f8562d About a minute ago 796MB ~~~ ~~~powershell [root@harbor-server tomcatdir]# docker login 192.168.10.23 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ~~~powershell [root@harbor-server tomcatdir]# docker push 192.168.10.23/library/tomcat:8575 ~~~ ![image-20220224231004637](../../img/docker_devops/image-20220224231004637.png) ### 5.5.5 验证容器镜像可用性 ~~~powershell [root@harbor-server ~]# docker run -d 192.168.10.23/library/tomcat:8575 d5443961ca65311ca0d68d53d44be997f5d6fde2d78772173ac6927112f34579 ~~~ ~~~powershell [root@harbor-server ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d5443961ca65 192.168.10.23/library/tomcat:8575 "/usr/local/tomcat/b…" 3 seconds ago Up 2 seconds 8080/tcp nifty_tesla ~~~ ~~~powershell [root@harbor-server ~]# docker inspect d544 获得：172.17.0.2 ~~~ ~~~powershell [root@harbor-server ~]# curl http://172.17.0.2:8080 tomcat is running ~~~ ## 5.6 项目构建及发布 ### 5.6.1 项目构建及发布步骤 第一步：jenkins获取项目代码 第二步：jenkins对项目代码编译，由maven完成 第三步：jenkins使用docker对编译完成的项目代码进行打包，打包成容器应用镜像 第四步：jenkins把打包的容器应用镜像上传到harbor 第五步：jenkins通过ssh插件完成对web-server进行运行容器应用镜像的操作 ### 5.6.2 创建项目任务 ![image-20220224231659567](../../img/docker_devops/image-20220224231659567.png) ![image-20220224231808502](../../img/docker_devops/image-20220224231808502.png) ![image-20220224233231807](../../img/docker_devops/image-20220224233231807.png) ![image-20220224233433469](../../img/docker_devops/image-20220224233433469.png) ![image-20220225014415364](../../img/docker_devops/image-20220225014415364.png) ![image-20220225014449916](../../img/docker_devops/image-20220225014449916.png) ![image-20220225014608778](../../img/docker_devops/image-20220225014608778.png) ![image-20220225014711014](../../img/docker_devops/image-20220225014711014.png) ~~~powershell Dockerfile： REPOSITORY=192.168.10.23/library/solo:${Tag} # 构建镜像 cat > Dockerfile << EOF FROM 192.168.10.23/library/tomcat:8575 RUN rm -rf /usr/local/tomcat/webapps/ROOT COPY target/*.war /usr/local/tomcat/webapps/ROOT.war CMD ["/usr/local/tomcat/bin/catalina.sh", "run"] EOF docker build -t $REPOSITORY . # 上传镜像 docker login 192.168.10.23 -u admin -p Harbor12345 docker push $REPOSITORY docker logout 192.168.10.23 ~~~ ~~~powershell shell script: REPOSITORY=192.168.10.23/library/solo:${Tag} # 部署 docker rm -f blog-solo |true docker image rm $REPOSITORY |true docker container run -d --name blog-solo -p 80:8080 $REPOSITORY ~~~ ![image-20220225014217956](../../img/docker_devops/image-20220225014217956.png) ![image-20220225014302077](../../img/docker_devops/image-20220225014302077.png) ![image-20220225014131634](../../img/docker_devops/image-20220225014131634.png) ![image-20220225014949917](../../img/docker_devops/image-20220225014949917.png) ![image-20220225013841677](../../img/docker_devops/image-20220225013841677.png) ================================================ FILE: docs/cloud/docker/docker_file.md ================================================ # Dockerfile精讲及新型容器镜像构建技术 # 一、容器与容器镜像之间的关系 说到Docker管理的容器不得不说容器镜像，主要因为容器镜像是容器模板，通过容器镜像我们才能快速创建容器。 如下图所示： ![img](../../img/docker_file/clip_image002.jpg) > Docker Daemon通过容器镜像创建容器。 # 二、容器镜像分类 - 操作系统类 - CentOS - Ubuntu - 在dockerhub下载或自行制作 - 应用类 - Tomcat - Nginx - MySQL - Redis # 三、容器镜像获取的方法 主要有以下几种： 1、在DockerHub直接下载 2、把操作系统中文件系统打包为容器镜像 3、把正在运行的容器打包为容器镜像，即docker commit 4、通过Dockerfile实现容器镜像的自定义及生成 # 四、容器镜像获取方法演示 ## 4.1 在DockerHub直接下载 ~~~powershell # docker pull centos:latest ~~~ ~~~powershell # docker pull nginx:latest ~~~ ## 4.2 把操作系统中文件系统打包为容器镜像 ### 4.2.1 安装一个最化的操作系统 ![image-20220210123047484](../../img/docker_file/image-20220210123047484.png) ### 4.2.2 把操作系统中文件系统进行打包 ~~~powershell # tar --numeric-owner --exclude=/proc --exclude=/sys -cvf centos7u6.tar / ~~~ ### 4.2.3 把打包后文件加载至本地文件系统生成本地容器镜像 ~~~powershell # ls centos7u6.tar ~~~ ~~~powershell # docker import centos7u6.tar centos7u6:v1 ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos7u6 v1 130cb005b2dc 7 seconds ago 1.09GB ~~~ ~~~powershell # docker run -it centos7u6:v1 bash [root@50f24f688b4d /]# ip a s 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 7: eth0@if8: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever ~~~ ## 4.3 把正在运行的容器打包为容器镜像 ### 4.3.1 运行一个容器 ~~~powershell # docker run -it centos7u6:v1 bash ~~~ ### 4.3.2 在容器中安装应用 ~~~powershell [root@064aace45718 /]# yum -y install httpd ~~~ ### 4.3.3 把正在运行的容器打包为容器镜像 ~~~powershell [root@064aace45718 /]# ctrl + p +q ~~~ ~~~powershell # docker commit 064aace45718 centos7u6-httpd:v1 ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos7u6-httpd v1 30ec9d728880 6 seconds ago 1.29GB ~~~ ~~~powershell # docker run -it centos7u6-httpd:v1 bash [root@01a1373b4a3f /]# rpm -qa | grep httpd httpd-tools-2.4.6-97.el7.centos.4.x86_64 httpd-2.4.6-97.el7.centos.4.x86_64 ~~~ ## 4.4 通过Dockerfile实现容器镜像的自定义及生成 ### 4.4.1 Dockerfile介绍 Dockerfile是一种能够被Docker程序解释的剧本。Dockerfile由一条一条的指令组成，并且有自己的书写格式和支持的命令。当我们需要在容器镜像中指定自己额外的需求时，只需在Dockerfile上添加或修改指令，然后通过docker build生成我们自定义的容器镜像（image）。 ![image-20220210123047484](../../img/docker_file/clip_image002-1644469873598.jpg) ### 4.4.2 Dockerfile指令 - 构建类指令 - 用于构建image - 其指定的操作不会在运行image的容器上执行（FROM、MAINTAINER、RUN、ENV、ADD、COPY） - 设置类指令 - 用于设置image的属性 - 其指定的操作将在运行image的容器中执行（CMD、ENTRYPOINT、USER 、EXPOSE、VOLUME、WORKDIR、ONBUILD） - 指令说明 | 指令 | 描述 | | ------- | --------------------------------------------------- | | FROM | 构建新镜像基于的基础镜像 | | LABEL | 标签 | | RUN | 构建镜像时运行的Shell命令 | | COPY | 拷贝文件或目录到镜像中 | | ADD | 解压压缩包并拷贝 | | ENV | 设置环境变量 | | USER | 为RUN、CMD和ENTRYPOINT执行命令指定运行用户 | | EXPOSE | 声明容器运行的服务端口 | | WORKDIR | 为RUN、CMD、ENTRYPOINT、COPY和ADD设置工作目录 | | CMD | 运行容器时默认执行，如果有多个CMD指令，最后一个生效 | - 指令详细解释 通过`man docker_file`可以查看到详细的说明,这里简单的翻译并列出常用的指令 1, **FROM** FROM指令用于指定其后构建新镜像所使用的基础镜像。 FROM指令必是Dockerfile文件中的首条命令。 FROM指令指定的基础image可以是官方远程仓库中的，也可以位于本地仓库，优先本地仓库。 ```powershell 格式:FROM : 例:FROM centos:latest ``` 2, **RUN** RUN指令用于在**构建**镜像中执行命令，有以下两种格式: * shell格式 ~~~powershell 格式:RUN <命令> 例:RUN echo 'kubemsb' > /var/www/html/index.html ~~~ * exec格式 ~~~powershell 格式:RUN ["可执行文件", "参数1", "参数2"] 例:RUN ["/bin/bash", "-c", "echo kubemsb > /var/www/html/index.html"] ~~~ **注意:** 按优化的角度来讲:当有多条要执行的命令,不要使用多条RUN,尽量使用&&符号与\符号连接成一行。因为多条RUN命令会让镜像建立多层(总之就是会变得臃肿了:smiley:)。 ~~~powershell RUN yum install httpd httpd-devel -y RUN echo test > /var/www/html/index.html 可以改成 RUN yum install httpd httpd-devel -y && echo test > /var/www/html/index.html 或者改成 RUN yum install httpd httpd-devel -y \ && echo test > /var/www/html/index.html ~~~ 3, **CMD** CMD不同于RUN,CMD用于指定在容器启动时所要执行的命令,而RUN用于指定镜像构建时所要执行的命令。 ~~~powershell 格式有三种: CMD ["executable","param1","param2"] CMD ["param1","param2"] CMD command param1 param2 ~~~ 每个Dockerfile只能有一条CMD命令。如果指定了多条命令，只有最后一条会被执行。 如果用户启动容器时候指定了运行的命令，则会覆盖掉CMD指定的命令。 ~~~powershell 什么是启动容器时指定运行的命令? # docker run -d -p 80:80 镜像名 运行的命令 ~~~ 4, **EXPOSE** EXPOSE指令用于指定容器在运行时监听的端口 ~~~powershell 格式:EXPOSE [...] 例:EXPOSE 80 3306 8080 ~~~ 上述运行的端口还需要使用docker run运行容器时通过-p参数映射到宿主机的端口. 5, **ENV** ENV指令用于指定一个环境变量. ~~~powershell 格式:ENV 或者 ENV = 例:ENV JAVA_HOME /usr/local/jdkxxxx/ ~~~ 6, **ADD** ADD指令用于把宿主机上的文件拷贝到镜像中 ~~~powershell 格式:ADD 可以是一个本地文件或本地压缩文件，还可以是一个url, 如果把写成一个url，那么ADD就类似于wget命令 路径的填写可以是容器内的绝对路径，也可以是相对于工作目录的相对路径 ~~~ 7, **COPY** COPY指令与ADD指令类似,但COPY的源文件只能是本地文件 ~~~powershell 格式:COPY ~~~ 8, **ENTRYPOINT** ENTRYPOINT与CMD非常类似 相同点： 一个Dockerfile只写一条，如果写了多条，那么只有最后一条生效 都是容器启动时才运行 不同点： 如果用户启动容器时候指定了运行的命令，ENTRYPOINT不会被运行的命令覆盖，而CMD则会被覆盖 ~~~powershell 格式有两种: ENTRYPOINT ["executable", "param1", "param2"] ENTRYPOINT command param1 param2 ~~~ 9, **VOLUME** VOLUME指令用于把宿主机里的目录与容器里的目录映射. 只指定挂载点,docker宿主机映射的目录为自动生成的。 ~~~powershell 格式:VOLUME [""] ~~~ 10, **USER** USER指令设置启动容器的用户(像hadoop需要hadoop用户操作，oracle需要oracle用户操作),可以是用户名或UID ~~~powershell USER daemon USER 1001 ~~~ **注意**：如果设置了容器以daemon用户去运行，那么RUN,CMD和ENTRYPOINT都会以这个用户去运行 镜像构建完成后，通过docker run运行容器时，可以通过-u参数来覆盖所指定的用户 11, **WORKDIR** WORKDIR指令设置工作目录,类似于cd命令。不建议使用`RUN cd /root` ,建议使用WORKDIR ~~~powershell WORKDIR /root ~~~ ### 4.4.3 Dockerfile基本构成 - 基础镜像信息 - 维护者信息 - 镜像操作指令 - 容器启动时执行指令 ### 4.4.4 Dockerfile生成容器镜像方法 ![image-20220210132232826](../../img/docker_file/image-20220210132232826.png) ### 4.4.5 Dockerfile生成容器镜像案例 #### 4.4.5.0 使用Dockerfile生成容器镜像步骤 ~~~powershell 第一步：创建一个文件夹（目录） 第二步：在文件夹（目录）中创建Dockerfile文件(并编写)及其它文件 第三步：使用`docker build`命令构建镜像 第四步：使用构建的镜像启动容器 ~~~ #### 4.4.5.1 使用Dockerfile生成Nginx容器镜像 ~~~powershell [root@localhost ~]# mkdir nginxroot [root@localhost ~]# cd nginxroot [root@localhost nginxroot]# ~~~ ~~~powershell [root@localhost nginxroot]# echo "nginx's running" >> index.html [root@localhost nginxroot]# ls index.html [root@localhost nginxroot]# cat index.html nginx's running ~~~ ~~~powershell [root@localhost nginxroot]# vim Dockerfile [root@localhost nginxroot]# cat Dockerfile FROM centos:centos7 MAINTAINER "www.kubemsb.com" RUN yum -y install wget RUN wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo RUN yum -y install nginx ADD index.html /usr/share/nginx/html/ RUN echo "daemon off;" >> /etc/nginx/nginx.conf EXPOSE 80 CMD /usr/sbin/nginx ~~~ ~~~powershell [root@localhost nginxroot]# docker build -t centos7-nginx:v1 . ~~~ ~~~powershell 输出： Sending build context to Docker daemon 3.072kB 第一步：下载基础镜像 Step 1/9 : FROM centos:centos7 ---> eeb6ee3f44bd 第二步：维护者信息 Step 2/9 : MAINTAINER "www.kubemsb.com" ---> Using cache ---> f978e524772c 第三步：安装wget Step 3/9 : RUN yum -y install wget ---> Running in 4e0fc3854088 Loaded plugins: fastestmirror, ovl Determining fastest mirrors * base: mirrors.huaweicloud.com * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package wget.x86_64 0:1.14-18.el7_6.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: wget x86_64 1.14-18.el7_6.1 base 547 k Transaction Summary ================================================================================ Install 1 Package Total download size: 547 k Installed size: 2.0 M Downloading packages: warning: /var/cache/yum/x86_64/7/base/packages/wget-1.14-18.el7_6.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY Public key for wget-1.14-18.el7_6.1.x86_64.rpm is not installed Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 Importing GPG key 0xF4A80EB5: Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) " Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5 Package : centos-release-7-9.2009.0.el7.centos.x86_64 (@CentOS) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wget-1.14-18.el7_6.1.x86_64 1/1 install-info: No such file or directory for /usr/share/info/wget.info.gz Verifying : wget-1.14-18.el7_6.1.x86_64 1/1 Installed: wget.x86_64 0:1.14-18.el7_6.1 Complete! Removing intermediate container 4e0fc3854088 ---> 369e33a2152a 第四步：使用wget下载YUM源 Step 4/9 : RUN wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo ---> Running in 4bdfc0a1c844 --2022-02-10 06:18:07-- http://mirrors.aliyun.com/repo/epel-7.repo Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 221.195.209.65, 221.195.209.64, 221.195.209.70, ... Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|221.195.209.65|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 664 [application/octet-stream] Saving to: '/etc/yum.repos.d/epel.repo' 0K 100% 158M=0s 2022-02-10 06:18:07 (158 MB/s) - '/etc/yum.repos.d/epel.repo' saved [664/664] Removing intermediate container 4bdfc0a1c844 ---> 1d73faa62447 第五步：安装Nginx Step 5/9 : RUN yum -y install nginx ---> Running in 51b50c2ce841 Loaded plugins: fastestmirror, ovl Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package nginx.x86_64 1:1.20.1-9.el7 will be installed --> Processing Dependency: nginx-filesystem = 1:1.20.1-9.el7 for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_1)(64bit) for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: nginx-filesystem for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: openssl for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: redhat-indexhtml for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: system-logos for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: libcrypto.so.1.1()(64bit) for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: libprofiler.so.0()(64bit) for package: 1:nginx-1.20.1-9.el7.x86_64 --> Processing Dependency: libssl.so.1.1()(64bit) for package: 1:nginx-1.20.1-9.el7.x86_64 --> Running transaction check ---> Package centos-indexhtml.noarch 0:7-9.el7.centos will be installed ---> Package centos-logos.noarch 0:70.0.6-3.el7.centos will be installed ---> Package gperftools-libs.x86_64 0:2.6.1-1.el7 will be installed ---> Package nginx-filesystem.noarch 1:1.20.1-9.el7 will be installed ---> Package openssl.x86_64 1:1.0.2k-24.el7_9 will be installed --> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-24.el7_9 for package: 1:openssl-1.0.2k-24.el7_9.x86_64 --> Processing Dependency: make for package: 1:openssl-1.0.2k-24.el7_9.x86_64 ---> Package openssl11-libs.x86_64 1:1.1.1k-2.el7 will be installed --> Running transaction check ---> Package make.x86_64 1:3.82-24.el7 will be installed ---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated ---> Package openssl-libs.x86_64 1:1.0.2k-24.el7_9 will be an update --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: nginx x86_64 1:1.20.1-9.el7 epel 587 k Installing for dependencies: centos-indexhtml noarch 7-9.el7.centos base 92 k centos-logos noarch 70.0.6-3.el7.centos base 21 M gperftools-libs x86_64 2.6.1-1.el7 base 272 k make x86_64 1:3.82-24.el7 base 421 k nginx-filesystem noarch 1:1.20.1-9.el7 epel 24 k openssl x86_64 1:1.0.2k-24.el7_9 updates 494 k openssl11-libs x86_64 1:1.1.1k-2.el7 epel 1.5 M Updating for dependencies: openssl-libs x86_64 1:1.0.2k-24.el7_9 updates 1.2 M Transaction Summary ================================================================================ Install 1 Package (+7 Dependent packages) Upgrade ( 1 Dependent package) Total download size: 26 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. -------------------------------------------------------------------------------- Total 3.1 MB/s | 26 MB 00:08 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : centos-logos-70.0.6-3.el7.centos.noarch 1/10 Installing : centos-indexhtml-7-9.el7.centos.noarch 2/10 Installing : 1:make-3.82-24.el7.x86_64 3/10 Installing : gperftools-libs-2.6.1-1.el7.x86_64 4/10 Installing : 1:openssl11-libs-1.1.1k-2.el7.x86_64 5/10 Updating : 1:openssl-libs-1.0.2k-24.el7_9.x86_64 6/10 Installing : 1:openssl-1.0.2k-24.el7_9.x86_64 7/10 Installing : 1:nginx-filesystem-1.20.1-9.el7.noarch 8/10 Installing : 1:nginx-1.20.1-9.el7.x86_64 9/10 Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 10/10 Verifying : 1:nginx-filesystem-1.20.1-9.el7.noarch 1/10 Verifying : 1:nginx-1.20.1-9.el7.x86_64 2/10 Verifying : 1:openssl-libs-1.0.2k-24.el7_9.x86_64 3/10 Verifying : 1:openssl11-libs-1.1.1k-2.el7.x86_64 4/10 Verifying : gperftools-libs-2.6.1-1.el7.x86_64 5/10 Verifying : 1:make-3.82-24.el7.x86_64 6/10 Verifying : 1:openssl-1.0.2k-24.el7_9.x86_64 7/10 Verifying : centos-indexhtml-7-9.el7.centos.noarch 8/10 Verifying : centos-logos-70.0.6-3.el7.centos.noarch 9/10 Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 10/10 Installed: nginx.x86_64 1:1.20.1-9.el7 Dependency Installed: centos-indexhtml.noarch 0:7-9.el7.centos centos-logos.noarch 0:70.0.6-3.el7.centos gperftools-libs.x86_64 0:2.6.1-1.el7 make.x86_64 1:3.82-24.el7 nginx-filesystem.noarch 1:1.20.1-9.el7 openssl.x86_64 1:1.0.2k-24.el7_9 openssl11-libs.x86_64 1:1.1.1k-2.el7 Dependency Updated: openssl-libs.x86_64 1:1.0.2k-24.el7_9 Complete! Removing intermediate container 51b50c2ce841 ---> 88a7d7a2c522 第六步：添加文件至容器 Step 6/9 : ADD index.html /usr/share/nginx/html/ ---> a2226a4d6720 第七步：设置nginx服务运行方式 Step 7/9 : RUN echo "daemon off;" >> /etc/nginx/nginx.conf ---> Running in 01d623937807 Removing intermediate container 01d623937807 ---> 53fddea5b491 第八步：暴露端口 Step 8/9 : EXPOSE 80 ---> Running in 9b73fcf7ee1b Removing intermediate container 9b73fcf7ee1b ---> 903377216b23 第九步：运行命令，执行nginx二进制文件 Step 9/9 : CMD /usr/sbin/nginx ---> Running in 58037652952c Removing intermediate container 58037652952c ---> 944d27b80f1f 生成镜像，并为镜像打标记： Successfully built 944d27b80f1f Successfully tagged centos7-nginx:v1 ~~~ ~~~powershell [root@localhost nginxroot]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos7-nginx v1 944d27b80f1f 3 minutes ago 587MB ~~~ ~~~powershell [root@localhost ~]# docker run -d -p 8081:80 centos7-nginx:v1 ~~~ ~~~powershell [root@localhost ~]# curl http://localhost:8081 nginx's running ~~~ #### 4.4.5.2 使用Dockerfile生成Tomcat容器镜像 ~~~powershell [root@localhost ~]# mkdir tomcatdir [root@localhost ~]# cd tomcatdir/ [root@localhost tomcatdir]# ~~~ ~~~powershell [root@localhost tomcatdir]# echo "tomcat is running" >> index.html ~~~ ~~~powershell [root@localhost tomcatdir]# ls Dockerfile jdk index.html jdk为目录 index.html 网站首页 ~~~ ~~~powershell [root@localhost tomcatdir]# vim Dockerfile [root@localhost tomcatdir]# cat Dockerfile FROM centos:centos7 MAINTAINER "www.kubemsb.com" ENV VERSION=8.5.75 ENV JAVA_HOME=/usr/local/jdk ENV TOMCAT_HOME=/usr/local/tomcat RUN yum -y install wget RUN wget https://dlcdn.apache.org/tomcat/tomcat-8/v${VERSION}/bin/apache-tomcat-${VERSION}.tar.gz RUN tar xf apache-tomcat-${VERSION}.tar.gz RUN mv apache-tomcat-${VERSION} /usr/local/tomcat RUN rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/* RUN mkdir /usr/local/tomcat/webapps/ROOT ADD ./index.html /usr/local/tomcat/webapps/ROOT/ ADD ./jdk /usr/local/jdk RUN echo "export TOMCAT_HOME=/usr/local/tomcat" >> /etc/profile RUN echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile RUN echo "export PATH=${TOMCAT_HOME}/bin:${JAVA_HOME}/bin:$PATH" >> /etc/profile RUN echo "export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar" >> /etc/profile RUN source /etc/profile EXPOSE 8080 CMD ["/usr/local/tomcat/bin/catalina.sh","run"] ~~~ ~~~powershell [root@localhost tomcatdir]# docker build -t centos-tomcat:v1 . Sending build context to Docker daemon 398.9MB Step 1/20 : FROM centos:centos7 ---> eeb6ee3f44bd Step 2/20 : MAINTAINER "www.kubemsb.com" ---> Using cache ---> f978e524772c Step 3/20 : ENV VERSION=8.5.75 ---> Using cache ---> 792767bbdb22 Step 4/20 : ENV JAVA_HOME=/usr/local/jdk ---> Using cache ---> 6eb3855650f0 Step 5/20 : ENV TOMCAT_HOME=/usr/local/tomcat ---> Using cache ---> e38bdbbfd19d Step 6/20 : RUN yum -y install wget ---> Using cache ---> 4c6aafa6d8ba Step 7/20 : RUN wget http://dlcdn.apache.org/tomcat/tomcat-8/v${VERSION}/bin/apache-tomcat-${VERSION}.tar.gz ---> Using cache ---> 9bdb6f636a5f Step 8/20 : RUN tar xf apache-tomcat-${VERSION}.tar.gz ---> Using cache ---> 6abe5cb0ef26 Step 9/20 : RUN mv apache-tomcat-${VERSION} /usr/local/tomcat ---> Using cache ---> b3907af15c22 Step 10/20 : RUN rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/* ---> Using cache ---> b775439344e3 Step 11/20 : RUN mkdir /usr/local/tomcat/webapps/ROOT ---> Using cache ---> 149ad46776eb Step 12/20 : ADD ./index.html /usr/local/tomcat/webapps/ROOT/ ---> 064579c39a46 Step 13/20 : ADD ./jdk /usr/local/jdk ---> 477fd38dfbcf Step 14/20 : RUN echo "export TOMCAT_HOME=/usr/local/tomcat" >> /etc/profile ---> Running in 3fc9bc5e8ba5 Removing intermediate container 3fc9bc5e8ba5 ---> 3c43bccd5779 Step 15/20 : RUN echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile ---> Running in 80f8150f0e80 Removing intermediate container 80f8150f0e80 ---> e01307ccb02a Step 16/20 : RUN echo "export PATH=${TOMCAT_HOME}/bin:${JAVA_HOME}/bin:$PATH" >> /etc/profile ---> Running in 92a6a4fd1cbc Removing intermediate container 92a6a4fd1cbc ---> 1d26f53b7095 Step 17/20 : RUN echo "export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar" >> /etc/profile ---> Running in fb5ee1710c36 Removing intermediate container fb5ee1710c36 ---> d2eaff35dce3 Step 18/20 : RUN source /etc/profile ---> Running in 0422af810b35 Removing intermediate container 0422af810b35 ---> fc6d285288ca Step 19/20 : EXPOSE 8080 ---> Running in eeb64d4f9e94 Removing intermediate container eeb64d4f9e94 ---> 05ec1c6d06cf Step 20/20 : CMD ["/usr/local/tomcat/bin/catalina.sh","run"] ---> Running in 66b7851e2772 Removing intermediate container 66b7851e2772 ---> ad338289055c Successfully built ad338289055c Successfully tagged centos-tomcat:v1 ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos-tomcat v1 ad338289055c 6 minutes ago 797MB ~~~ ~~~powershell # docker run -d -p 8082:8080 centos-tomcat:v1 ~~~ ~~~powershell # curl http://localhost:8082 tomcat is running ~~~ ### 4.4.6 使用Dockerfile生成容器镜像优化 #### 4.4.6.1 减少镜像分层 Dockerfile中包含多种指令，如果涉及到部署最多使用的算是RUN命令了，使用RUN命令时，不建议每次安装都使用一条单独的RUN命令，可以把能够合并安装指令合并为一条，这样就可以减少镜像分层。 ~~~powershell FROM centos:latest MAINTAINER www.kubemsb.com RUN yum install epel-release -y RUN yum install -y gcc gcc-c++ make -y RUN wget http://docs.php.net/distributions/php-5.6.36.tar.gz RUN tar zxf php-5.6.36.tar.gz RUN cd php-5.6.36 RUN ./configure --prefix=/usr/local/php RUN make -j 4 RUN make install EXPOSE 9000 CMD ["php-fpm"] ~~~ **优化内容如下：** ~~~powershell FROM centos:latest MAINTAINER www.kubemsb.com RUN yum install epel-release -y && \ yum install -y gcc gcc-c++ make RUN wget http://docs.php.net/distributions/php-5.6.36.tar.gz && \ tar zxf php-5.6.36.tar.gz && \ cd php-5.6.36 && \ ./configure --prefix=/usr/local/php && \ make -j 4 && make install EXPOSE 9000 CMD ["php-fpm"] ~~~ #### 4.4.6.2 清理无用数据 - 一次RUN形成新的一层，如果没有在同一层删除，无论文件是否最后删除，都会带到下一层，所以要在每一层清理对应的残留数据，减小镜像大小。 - 把生成容器镜像过程中部署的应用软件包做删除处理 ~~~powershell FROM centos:latest MAINTAINER www.kubemsb.com RUN yum install epel-release -y && \ yum install -y gcc gcc-c++ make gd-devel libxml2-devel \ libcurl-devel libjpeg-devel libpng-devel openssl-devel \ libmcrypt-devel libxslt-devel libtidy-devel autoconf \ iproute net-tools telnet wget curl && \ yum clean all && \ rm -rf /var/cache/yum/* RUN wget http://docs.php.net/distributions/php-5.6.36.tar.gz && \ tar zxf php-5.6.36.tar.gz && \ cd php-5.6.36 && \ ./configure --prefix=/usr/local/php \ make -j 4 && make install && \ cd / && rm -rf php* ~~~ #### 4.4.6.3 多阶段构建镜像 项目容器镜像有两种，一种直接把项目代码复制到容器镜像中，下次使用容器镜像时即可直接启动；另一种把需要对项目源码进行编译，再复制到容器镜像中使用。 不论是哪种方法都会让制作镜像复杂了些，并也会让容器镜像比较大，建议采用分阶段构建镜像的方法实现。 ~~~powershell $ git clone https://github.com/kubemsb/tomcat-java-demo $ cd tomcat-java-demo $ vi Dockerfile FROM maven AS build ADD ./pom.xml pom.xml ADD ./src src/ RUN mvn clean package FROM kubemsb/tomcat RUN rm -rf /usr/local/tomcat/webapps/ROOT COPY --from=build target/*.war /usr/local/tomcat/webapps/ROOT.war $ docker build -t demo:v1 . $ docker container run -d -v demo:v1 ~~~ ~~~powershell 第一个 FROM 后边多了个 AS 关键字，可以给这个阶段起个名字 第二个 FROM 使用上面构建的 Tomcat 镜像，COPY 关键字增加了 —from 参数，用于拷贝某个阶段的文件到当前阶段。 ~~~ # 五、新型容器镜像构建技术 BuildPacks ================================================ FILE: docs/cloud/docker/docker_image.md ================================================ # Docker容器镜像 # 一、Docker容器镜像操作 ## 2.1 查看本地容器镜像 ### 2.1.1 使用docker images命令查看 ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE bash latest 5557e073f11c 2 weeks ago 13MB nginx latest 605c77e624dd 3 weeks ago 141MB centos latest 5d0da3dc9764 4 months ago 231MB ~~~ ### 2.1.2 使用docker image命令查看 ~~~powershell # docker image list REPOSITORY TAG IMAGE ID CREATED SIZE bash latest 5557e073f11c 2 weeks ago 13MB nginx latest 605c77e624dd 3 weeks ago 141MB centos latest 5d0da3dc9764 4 months ago 231MB ~~~ ### 2.1.3 查看docker容器镜像本地存储位置 > 考虑到docker容器镜像会占用本地存储空间，建议搭建其它存储系统挂载到本地以便解决占用大量本地存储的问题。 ~~~powershell # ls /var/lib/docker buildkit containers image network overlay2 plugins runtimes swarm tmp trust volumes ~~~ ## 2.2 搜索Docker Hub容器镜像 ### 2.2.1 命令行搜索 ~~~powershell # docker search centos ~~~ ~~~powershell 输出 NAME DESCRIPTION STARS OFFICIAL AUTOMATED centos The official build of CentOS. 6987 [OK] ansible/centos7-ansible Ansible on Centos7 135 [OK] consol/centos-xfce-vnc Centos container with "headless" VNC session… 135 [OK] jdeathe/centos-ssh OpenSSH / Supervisor / EPEL/IUS/SCL Repos - … 121 [OK] ~~~ ### 2.2.2 Docker Hub Web界面搜索 ![image-20220124162022990](../../img/docker_image/image-20220124162022990.png) ![image-20220124162116338](../../img/docker_image/image-20220124162116338.png) ![image-20220124162200273](../../img/docker_image/image-20220124162200273.png) ![image-20220124162312918](../../img/docker_image/image-20220124162312918.png) ## 2.3 Docker 容器镜像下载 ~~~powershell # docker pull centos ~~~ ## 2.4 Docker容器镜像删除方法 ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE bash latest 5557e073f11c 2 weeks ago 13MB nginx latest 605c77e624dd 3 weeks ago 141MB centos latest 5d0da3dc9764 4 months ago 231MB ~~~ ~~~powershell # docker rmi centos Untagged: centos:latest Untagged: centos@sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177 Deleted: sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6 Deleted: sha256:74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59 ~~~ 或 ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos latest 5d0da3dc9764 4 months ago 231MB ~~~ ~~~powershell # docker rmi 5d0da3dc9764 ~~~ # 二、Docker容器镜像介绍 ## 2.1 Docker Image - Docker 镜像是只读的容器模板，是Docker容器基础 - 为Docker容器提供了静态文件系统运行环境（rootfs） - 是容器的静止状态 - 容器是镜像的运行状态 ## 2.2 联合文件系统 ### 2.2.1 联合文件系统定义 - 联合文件系统(union filesystem) - 联合文件系统是实现联合挂载技术的文件系统 - 联合挂载技术可以实现在一个挂载点同时挂载多个文件系统，将挂载点的原目录与被挂载内容进行整合，使得最终可见的文件系统包含整合之后的各层文件和目录 ### 2.2.2 图解 ![image-20220125080435098](../../img/docker_image/image-20220125080435098.png) ## 2.3 Docker Overlay2 容器文件系统有多种存储驱动实现方式：aufs，devicemapper，overlay，overlay2 等，本次以overlay2为例进行说明。 ### 2.3.1 概念 - registry/repository： registry 是 repository 的集合，repository 是镜像的集合。 - image：image 是存储镜像相关的元数据，包括镜像的架构，镜像默认配置信息，镜像的容器配置信息等等。它是“逻辑”上的概念，并无物理上的镜像文件与之对应。 - layer：layer(镜像层) 组成了镜像，单个 layer 可以被多个镜像共享。 ![image-20220125082226414](../../img/docker_image/image-20220125082226414.png) ### 2.3.2 查看Docker Host存储驱动方式 ~~~powershell # docker info | grep overlay Storage Driver: overlay2 ~~~ ### 2.3.3 了解images分层 ~~~powershell # docker pull nginx Using default tag: latest latest: Pulling from library/nginx a2abf6c4d29d: Pull complete a9edb18cadd1: Pull complete 589b7251471a: Pull complete 186b1aaa4aa6: Pull complete b4df32aa5a72: Pull complete a0bcbecc962e: Pull complete Digest: sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31 Status: Downloaded newer image for nginx:latest docker.io/library/nginx:latest ~~~ 可以看到上述下载的镜像分为6层，如何找到这6层存储在Docker Host哪个位置呢？ 首先查看nginx镜像 ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 605c77e624dd 3 weeks ago 141MB ~~~ 通过其Image ID 605c77e624dd 就可以找到存储位置 ~~~powershell # ls /var/lib/docker/image/overlay2/ distribution imagedb layerdb repositories.json ~~~ 这个目录是查找的入口，非常重要。它存储了镜像管理的元数据。 - repositories.json 记录了 repo 与镜像 ID 的映射关系 - imagedb 记录了镜像架构，操作系统，构建镜像的容器 ID 和配置以及 rootfs 等信息 - layerdb 记录了每层镜像层的元数据。 通过短 ID 查找 repositories.json 文件，找到镜像 nginx 的长 ID，通过长 ID 在 imagedb 中找到该镜像的元数据： ~~~powershell # cat /var/lib/docker/image/overlay2/repositories.json | grep 605c77e624dd {"Repositories":"nginx":{"nginx:latest":"sha256:605c77e624ddb75e6110f997c58876baa13f8754486b461117934b24a9dc3a85","nginx@sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31":"sha256:605c77e624ddb75e6110f997c58876baa13f8754486b461117934b24a9dc3a85"}}}} ~~~ ~~~powershell # cat /var/lib/docker/image/overlay2/imagedb/content/sha256/605c77e624ddb75e6110f997c58876baa13f8754486b461117934b24a9dc3a85 ...... "os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f","sha256:e379e8aedd4d72bb4c529a4ca07a4e4d230b5a1d3f7a61bc80179e8f02421ad8","sha256:b8d6e692a25e11b0d32c5c3dd544b71b1085ddc1fddad08e68cbd7fda7f70221","sha256:f1db227348d0a5e0b99b15a096d930d1a69db7474a1847acbc31f05e4ef8df8c","sha256:32ce5f6a5106cc637d09a98289782edf47c32cb082dc475dd47cbf19a4f866da","sha256:d874fd2bc83bb3322b566df739681fbd2248c58d3369cb25908d68e7ed6040a6"]}} ~~~ 这里仅保留我们想要的元数据 rootfs。在 rootfs 中看到 layers 有6层，这6层即对应镜像的6层镜像层。并且，自上而下分别映射到容器的底层到顶层。找到了镜像的6层，接下来的问题是每层的文件内容在哪里呢？ layerdb 元数据会给我们想要的信息，通过底层 diff-id: 2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f 我们查到最底层镜像层的 cache_id，通过 cache_id 即可查找到镜像层的文件内容： ~~~powershell # ls /var/lib/docker/image/overlay2/layerdb/sha256/2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f cache-id diff size tar-split.json.gz ~~~ ~~~powershell # cat /var/lib/docker/image/overlay2/layerdb/sha256/2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f/cache-id 85c4c5ecdac6c0d197f899dac227b9d493911a9a5820eac501bb5e9ae361f4c7 ~~~ ~~~powershell # cat /var/lib/docker/image/overlay2/layerdb/sha256/2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f/diff sha256:2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f ~~~ 使用 cacheID 查找文件内容 ~~~powershell # ls /var/lib/docker/overlay2/85c4c5ecdac6c0d197f899dac227b9d493911a9a5820eac501bb5e9ae361f4c7 committed diff link # ls /var/lib/docker/overlay2/85c4c5ecdac6c0d197f899dac227b9d493911a9a5820eac501bb5e9ae361f4c7/diff bin dev home lib64 mnt proc run srv tmp var boot etc lib media opt root sbin sys usr ~~~ 上示例中，镜像元数据和镜像层内容是分开存储的。因此通过 cache-id 我们需要到 /var/lib/docker/overlay2 目录下查看镜像层内容，它就存在 diff 目录下，其中 link 存储的是镜像层对应的短 ID，后面会看到它的用场。 找到了镜像层的最底层，接着查找镜像层的“中间层”，发现在 layerdb 目录下没有 diff-id e379e8aedd4d72bb4c529a4ca07a4e4d230b5a1d3f7a61bc80179e8f02421ad8的镜像层： ~~~powershell # ls /var/lib/docker/image/overlay2/layerdb/sha256/e379e8aedd4d72bb4c529a4ca07a4e4d230b5a1d3f7a61bc80179e8f02421ad8 ls: 无法访问/var/lib/docker/image/overlay2/layerdb/sha256/e379e8aedd4d72bb4c529a4ca07a4e4d230b5a1d3f7a61bc80179e8f02421ad8: 没有那个文件或目录 ~~~ 这是因为 docker 引入了内容寻址机制，该机制会根据文件内容来索引镜像和镜像层。docker 利用 rootfs 中的 diff_id 计算出内容寻址的 chainID，通过 chainID 获取 layer 相关信息，最终索引到镜像层文件内容。 对于最底层镜像层其 diff_id 即是 chainID。因此我们可以查找到它的文件内容。除最底层外，chainID 需通过公式 chainID(n) = SHA256(chain(n-1) diffID(n)) 计算得到，计算“中间层” chainID： ~~~powershell # echo -n "sha256:2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f sha256:e379e8aedd4d72bb4c529a4ca07a4e4d230b5a1d3f7a61bc80179e8f02421ad8" | sha256sum - 780238f18c540007376dd5e904f583896a69fe620876cabc06977a3af4ba4fb5 - ~~~ 根据 “中间层” chainID 查找文件内容： ~~~powershell # ls /var/lib/docker/image/overlay2/layerdb/sha256/780238f18c540007376dd5e904f583896a69fe620876cabc06977a3af4ba4fb5 cache-id diff parent size tar-split.json.gz ~~~ ~~~powershell # cat /var/lib/docker/image/overlay2/layerdb/sha256/780238f18c540007376dd5e904f583896a69fe620876cabc06977a3af4ba4fb5/cache-id 57e1f1b11e26f748161b7fccbf2ba6b24c2f98dc8a821729f0be215ad267498c ~~~ ~~~powershell # cat /var/lib/docker/image/overlay2/layerdb/sha256/780238f18c540007376dd5e904f583896a69fe620876cabc06977a3af4ba4fb5/diff sha256:e379e8aedd4d72bb4c529a4ca07a4e4d230b5a1d3f7a61bc80179e8f02421ad8 ~~~ ~~~powershell # cat /var/lib/docker/image/overlay2/layerdb/sha256/780238f18c540007376dd5e904f583896a69fe620876cabc06977a3af4ba4fb5/parent sha256:2edcec3590a4ec7f40cf0743c15d78fb39d8326bc029073b41ef9727da6c851f ~~~ ~~~powershell 镜像层文件内容 # ls /var/lib/docker/overlay2/57e1f1b11e26f748161b7fccbf2ba6b24c2f98dc8a821729f0be215ad267498c committed diff link lower work # ls /var/lib/docker/overlay2/57e1f1b11e26f748161b7fccbf2ba6b24c2f98dc8a821729f0be215ad267498c/diff/ docker-entrypoint.d etc lib tmp usr var ~~~ ~~~powershell 镜像层文件内容短 ID # cat /var/lib/docker/overlay2/57e1f1b11e26f748161b7fccbf2ba6b24c2f98dc8a821729f0be215ad267498c/link 24GM2IZVPTUROAG7AWJO5ZWE6B ~~~ ~~~powershell “父”镜像层文件内容短 ID # cat /var/lib/docker/overlay2/57e1f1b11e26f748161b7fccbf2ba6b24c2f98dc8a821729f0be215ad267498c/lower l/SICZO4QNVZEVOIJ4HDXVDKNYA2 ~~~ 找到最底层文件内容和“中间层”文件内容，再去找最顶层文件内容就变的不难了 ## 2.4 Docker容器与镜像 通过 docker run 命令启动一个镜像为 nginx的容器： ~~~powershell # docker run -d nginx:latest 3272831107a3499afe8160b0cd423e2ac4223522f1995b7be3504a1d3d272878 # docker ps | grep nginx 3272831107a3 nginx:latest "/docker-entrypoint.…" 11 seconds ago Up 9 seconds 80/tcp angry_beaver ~~~ ~~~powershell # mount | grep overlay overlay on /var/lib/docker/overlay2/b3f5c8b42ac055c715216e376cfe44571f618a876f481533ec1434aa0bc4f8ed/merged type overlay (rw,relatime,seclabel,lowerdir=/var/lib/docker/overlay2/l/MS2X66BYF6UZ7EKUWMZJKCF4HO:/var/lib/docker/overlay2/l/ODJROQUGY3WQMOGQ3BLYZGIAG4:/var/lib/docker/overlay2/l/Q5LOBFJRH5M7M5CMSWW5L4VYOY:/var/lib/docker/overlay2/l/ZR35FN2E3WEARZV4HLRU373FT7:/var/lib/docker/overlay2/l/NSM2PTAT6TIT2H6G3HFNGZJH5N:/var/lib/docker/overlay2/l/24GM2IZVPTUROAG7AWJO5ZWE6B:/var/lib/docker/overlay2/l/SICZO4QNVZEVOIJ4HDXVDKNYA2,upperdir=/var/lib/docker/overlay2/b3f5c8b42ac055c715216e376cfe44571f618a876f481533ec1434aa0bc4f8ed/diff,workdir=/var/lib/docker/overla 2/b3f5c8b42ac055c715216e376cfe44571f618a876f481533ec1434aa0bc4f8ed/work) ~~~ 可以看到，启动容器会 mount 一个 overlay 的联合文件系统到容器内。这个文件系统由三层组成： - lowerdir：只读层，即为镜像的镜像层。 - upperdir：读写层，该层是容器的读写层，对容器的读写操作将反映在读写层。 - workdir： overlayfs 的内部层，用于实现从只读层到读写层的 copy_up 操作。 - merge：容器内作为同一视图联合挂载点的目录。 这里需要着重介绍的是容器的 lowerdir 镜像只读层，查看只读层的短 ID： ~~~powershell lowerdir=/var/lib/docker/overlay2/l/MS2X66BYF6UZ7EKUWMZJKCF4HO /var/lib/docker/overlay2/l/ODJROQUGY3WQMOGQ3BLYZGIAG4 /var/lib/docker/overlay2/l/Q5LOBFJRH5M7M5CMSWW5L4VYOY /var/lib/docker/overlay2/l/ZR35FN2E3WEARZV4HLRU373FT7 /var/lib/docker/overlay2/l/NSM2PTAT6TIT2H6G3HFNGZJH5N /var/lib/docker/overlay2/l/24GM2IZVPTUROAG7AWJO5ZWE6B /var/lib/docker/overlay2/l/SICZO4QNVZEVOIJ4HDXVDKNYA2 ~~~ 镜像层只有6层这里的短 ID 却有7个？ 在 /var/lib/docker/overlay2/l 目录下我们找到了答案： ~~~powershell # cd /var/lib/docker/overlay2/l # pwd /var/lib/docker/overlay2/l # ls 24GM2IZVPTUROAG7AWJO5ZWE6B LZEAXJGRW6HKBBGGB2N4CWMSVJ R2XTGODAA67NQJM44MIKMDUF4W 5OI5WMJ2FP7QI7IFWDMHLBRDDN MS2X66BYF6UZ7EKUWMZJKCF4HO SICZO4QNVZEVOIJ4HDXVDKNYA2 644ISPHLTBSSC2KLP6BGHHHZPR NSM2PTAT6TIT2H6G3HFNGZJH5N ZR35FN2E3WEARZV4HLRU373FT7 6CQUILQSJNVTMFFV3ABCCOGOYG ODJROQUGY3WQMOGQ3BLYZGIAG4 BQENAYC44O2ZCZFT5URMH5OADK Q5LOBFJRH5M7M5CMSWW5L4VYOY ~~~ ~~~powershell # ls -l MS2X66BYF6UZ7EKUWMZJKCF4HO/ 总用量 0 drwxr-xr-x. 4 root root 43 1月 25 01:27 dev drwxr-xr-x. 2 root root 66 1月 25 01:27 etc [root@192 l]# ls -l ODJROQUGY3WQMOGQ3BLYZGIAG4/ 总用量 0 drwxr-xr-x. 2 root root 41 12月 30 03:28 docker-entrypoint.d [root@192 l]# ls -l Q5LOBFJRH5M7M5CMSWW5L4VYOY/ 总用量 0 drwxr-xr-x. 2 root root 41 12月 30 03:28 docker-entrypoint.d [root@192 l]# ls -l ZR35FN2E3WEARZV4HLRU373FT7/ 总用量 0 drwxr-xr-x. 2 root root 45 12月 30 03:28 docker-entrypoint.d [root@192 l]# ls -l NSM2PTAT6TIT2H6G3HFNGZJH5N/ 总用量 4 -rwxrwxr-x. 1 root root 1202 12月 30 03:28 docker-entrypoint.sh [root@192 l]# ls -l 24GM2IZVPTUROAG7AWJO5ZWE6B/ 总用量 4 drwxr-xr-x. 2 root root 6 12月 30 03:28 docker-entrypoint.d drwxr-xr-x. 18 root root 4096 12月 30 03:28 etc drwxr-xr-x. 4 root root 45 12月 20 08:00 lib drwxrwxrwt. 2 root root 6 12月 30 03:28 tmp drwxr-xr-x. 7 root root 66 12月 20 08:00 usr drwxr-xr-x. 5 root root 41 12月 20 08:00 var [root@192 l]# ls -l SICZO4QNVZEVOIJ4HDXVDKNYA2/ 总用量 12 drwxr-xr-x. 2 root root 4096 12月 20 08:00 bin drwxr-xr-x. 2 root root 6 12月 12 01:25 boot drwxr-xr-x. 2 root root 6 12月 20 08:00 dev drwxr-xr-x. 30 root root 4096 12月 20 08:00 etc drwxr-xr-x. 2 root root 6 12月 12 01:25 home drwxr-xr-x. 8 root root 96 12月 20 08:00 lib drwxr-xr-x. 2 root root 34 12月 20 08:00 lib64 drwxr-xr-x. 2 root root 6 12月 20 08:00 media drwxr-xr-x. 2 root root 6 12月 20 08:00 mnt drwxr-xr-x. 2 root root 6 12月 20 08:00 opt drwxr-xr-x. 2 root root 6 12月 12 01:25 proc drwx------. 2 root root 37 12月 20 08:00 root drwxr-xr-x. 3 root root 30 12月 20 08:00 run drwxr-xr-x. 2 root root 4096 12月 20 08:00 sbin drwxr-xr-x. 2 root root 6 12月 20 08:00 srv drwxr-xr-x. 2 root root 6 12月 12 01:25 sys drwxrwxrwt. 2 root root 6 12月 20 08:00 tmp drwxr-xr-x. 11 root root 120 12月 20 08:00 usr drwxr-xr-x. 11 root root 139 12月 20 08:00 var ~~~ 镜像层ODJROQUGY3WQMOGQ3BLYZGIAG4/Q5LOBFJRH5M7M5CMSWW5L4VYOY/ZR35FN2E3WEARZV4HLRU373FT7/NSM2PTAT6TIT2H6G3HFNGZJH5N/24GM2IZVPTUROAG7AWJO5ZWE6B/SICZO4QNVZEVOIJ4HDXVDKNYA2 分别对应镜像的6层镜像层文件内容，它们分别映射到镜像层的 diff 目录。而 MS2X66BYF6UZ7EKUWMZJKCF4HO映射的是容器的初始化层 init，该层内容是和容器配置相关的文件内容，它是只读的。 启动了容器，docker 将镜像的内容 mount 到容器中。那么，如果在容器内写文件会对镜像有什么影响呢？ ## 2.5 容器内写文件 不难理解，镜像层是只读的，在容器中写文件其实是将文件写入到 overlay 的可读写层。 这里有几个 case 可以测试： - 读写层不存在该文件，只读层存在。 - 读写层存在该文件，只读层不存在。 - 读写层和只读层都不存在该文件。 我们简单构建一种读写层和只读层都不存在的场景： ~~~powershell # docker run -it centos:latest bash [root@355e99982248 /]# touch msb.txt [root@355e99982248 /]# ls bin etc lib lost+found mnt opt root sbin sys usr dev home lib64 media msb.txt proc run srv tmp var ~~~ 查看读写层是否有该文件： ~~~powershell 查看镜像是否有变化 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE ubuntu latest d13c942271d6 2 weeks ago 72.8MB bash latest 5557e073f11c 2 weeks ago 13MB nginx latest 605c77e624dd 3 weeks ago 141MB centos latest 5d0da3dc9764 4 months ago 231MB [root@localhost ~]# cat /var/lib/docker/image/overlay2/repositories.json | grep 5d0da3dc9764 {"Repositories"{"centos:latest":"sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6","centos@sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177":"sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6"}}} [root@localhost ~]# cat /var/lib/docker/image/overlay2/imagedb/content/sha256/5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6 {"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59"]}} [root@localhost ~]# ls /var/lib/docker/image/overlay2/layerdb/sha256/74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59: cache-id diff size tar-split.json.gz [root@localhost ~]# cat /var/lib/docker/image/overlay2/layerdb/sha256/74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59/cache-id b17bc5c5103514923a30983c48f909e06f366b7aa1e85f112b67abb3ef5cd0cb [root@localhost ~]# cat /var/lib/docker/image/overlay2/layerdb/sha256/74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59/diff sha256:74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59 [root@localhost ~]# ls /var/lib/docker/overlay2/b17bc5c5103514923a30983c48f909e06f366b7aa1e85f112b67abb3ef5cd0cb committed diff link [root@localhost ~]# ls /var/lib/docker/overlay2/b17bc5c5103514923a30983c48f909e06f366b7aa1e85f112b67abb3ef5cd0cb/diff/ bin etc lib lost+found mnt proc run srv tmp var dev home lib64 media opt root sbin sys usr 查看容器是否有变化 [root@localhost ~]# mount | grep overlay type overlay (rw,relatime,seclabel,lowerdir=/var/lib/docker/overlay2/l/R2W2LEMDPRIUFYDVSLIQSCYTGX:/var/lib/docker/overlay2/l/R2XTGODAA67NQJM44MIKMDUF4W,upperdir=/var/lib/docker overlay2/7f0b54c748171872ce564305e394547555cb1182abf802c2262384be3dc78a8f/diff,workdir=/var/lib/docker/overlay2/7f0b54c748171872ce564305e394547555cb1182abf802c2262384be3dc78a8f/work) [root@localhost ~]# ls -l /var/lib/docker/overlay2/l/ 总用量 0 lrwxrwxrwx. 1 root root 77 1月 25 01:41 R2W2LEMDPRIUFYDVSLIQSCYTGX -> ../7f0b54c748171872ce564305e394547555cb1182abf802c2262384be3dc78a8f-init/diff lrwxrwxrwx. 1 root root 72 1月 25 00:29 R2XTGODAA67NQJM44MIKMDUF4W -> ../b17bc5c5103514923a30983c48f909e06f366b7aa1e85f112b67abb3ef5cd0cb/diff [root@localhost ~]# ls /var/lib/docker/overlay2/7f0b54c748171872ce564305e394547555cb1182abf802c2262384be3dc78a8f/diff msb.txt [root@localhost ~]# ls /var/lib/docker/overlay2/7f0b54c748171872ce564305e394547555cb1182abf802c2262384be3dc78a8f/merged/ bin etc lib lost+found mnt opt root sbin sys usr dev home lib64 media msb.txt proc run srv tmp var ~~~ # 三、Docker容器镜像操作命令 ## 3.1 docker commit 上节提到容器内写文件会反映在 overlay 的可读写层，那么读写层的文件内容可以做成镜像吗？ 可以。docker 通过 commit 和 build 操作实现镜像的构建。commit 将容器提交为一个镜像，build 在一个镜像的基础上构建镜像。 使用 commit 将上节的容器提交为一个镜像： ~~~powershell [root@355e99982248 /]# ctrl+p+q ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 355e99982248 centos:latest "bash" 21 minutes ago Up 21 minutes fervent_perlman ~~~ ~~~powershell # docker commit 355e99982248 sha256:8965dcf23201ed42d4904e2f10854d301ad93b34bea73f384440692e006943de ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE 8965dcf23201 About a minute ago 231MB ~~~ image 短 ID 8965dcf23201 即为容器提交的镜像，查看镜像的 imagedb 元数据： ~~~powershell # cat /var/lib/docker/image/overlay2/imagedb/content/sha256/8965dcf23201ed42d4904e2f10854d301ad93b34bea73f384440692e006943de ...... "os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59","sha256:551c3089b186b4027e949910981ff1ba54114610f2aab9359d28694c18b0203b"]}} ~~~ 可以看到镜像层自上而下的前1个镜像层 diff_id 和 centos 镜像层 diff_id 是一样的，说明每层镜像层可以被多个镜像共享。而多出来的一层镜像层内容即是上节我们写入文件的内容： ~~~powershell # echo -n "sha256:74ddd0ec08fa43d09f32636ba91a0a3053b02cb4627c35051aff89f853606b59 sha256:551c3089b186b4027e949910981ff1ba54114610f2aab9359d28694c18b0203b" | sha256sum - 92f7208b1cc0b5cc8fe214a4b0178aa4962b58af8ec535ee7211f335b1e0ed3b - ~~~ ~~~powershell # cd /var/lib/docker/image/overlay2/layerdb/sha256/92f7208b1cc0b5cc8fe214a4b0178aa4962b58af8ec535ee7211f335b1e0ed3b [root@192 92f7208b1cc0b5cc8fe214a4b0178aa4962b58af8ec535ee7211f335b1e0ed3b]# ls cache-id diff parent size tar-split.json.gz [root@192 92f7208b1cc0b5cc8fe214a4b0178aa4962b58af8ec535ee7211f335b1e0ed3b]# cat cache-id 250dc0b4f2c5f27952241a55cd4c286bfaaf8af4b77c9d0a38976df4c147cb95 [root@192 92f7208b1cc0b5cc8fe214a4b0178aa4962b58af8ec535ee7211f335b1e0ed3b]# ls /var/lib/docker/overlay2/250dc0b4f2c5f27952241a55cd4c286bfaaf8af4b77c9d0a38976df4c147cb95 diff link lower work [root@192 92f7208b1cc0b5cc8fe214a4b0178aa4962b58af8ec535ee7211f335b1e0ed3b]# ls /var/lib/docker/overlay2/250dc0b4f2c5f27952241a55cd4c286bfaaf8af4b77c9d0a38976df4c147cb95/diff msb.txt ~~~ ## 3.2 docker save > 导出容器镜像，方便分享。 ~~~powershell # docker save -o centos.tar centos:latest ~~~ ~~~powershell # ls centos.tar ~~~ ## 3.3 docker load > 把他人分享的容器镜像导入到本地，这通常是容器镜像分发方式之一。 ~~~powershell # docker load -i centos.tar ~~~ ## 3.4 docker export > 把正在运行的容器导出 ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 355e99982248 centos:latest "bash" 7 hours ago Up 7 hours fervent_perlman ~~~ ~~~powershell # docker export -o centos7.tar 355e99982248 ~~~ ~~~powershell # ls centos7.tar ~~~ ## 3.5 docker import > 导入使用docker export导入的容器做为本地容器镜像。 ~~~powershell # ls centos7.tar ~~~ ~~~powershell # docker import centos7.tar centos7:v1 ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos7 v1 3639f9a13231 17 seconds ago 231MB ~~~ 通过docker save与docker load及docker export与docker import分享容器镜像都是非常麻烦的，有没有更方便的方式分享容器镜像呢？ ================================================ FILE: docs/cloud/docker/docker_image_fast.md ================================================ # Docker容器镜像加速器及本地容器镜像仓库 # 一、容器镜像加速器 > 由于国内访问国外的容器镜像仓库速度比较慢，因此国内企业创建了容器镜像加速器，以方便国内用户使用容器镜像。 ## 1.1 获取阿里云容器镜像加速地址 ![image-20220125221631548](../../img/docker_image_fast/image-20220125221631548.png) ![image-20220125221748100](../../img/docker_image_fast/image-20220125221748100.png) ## 1.2 配置docker daemon使用加速器 ~~~powershell 添加daemon.json配置文件 # vim /etc/docker/daemon.json # cat /etc/docker/daemon.json { "registry-mirrors": ["https://s27w6kze.mirror.aliyuncs.com"] } ~~~ ~~~powershell 重启docker # systemctl daemon-reload # systemctl restart docker ~~~ ~~~powershell 尝试下载容器镜像 # docker pull centos ~~~ # 二、容器镜像仓库 ## 2.1 docker hub ### 2.1.1 注册 > 准备邮箱及用户ID ![image-20220125224745376](../../img/docker_image_fast/image-20220125224745376.png) ![image-20220125224838850](../../img/docker_image_fast/image-20220125224838850.png) ![image-20220125225243088](../../img/docker_image_fast/image-20220125225243088.png) ![image-20220125225448956](../../img/docker_image_fast/image-20220125225448956.png) ### 2.1.2 登录 ![image-20220125225753719](../../img/docker_image_fast/image-20220125225753719.png) ![image-20220125225914283](../../img/docker_image_fast/image-20220125225914283.png) ### 2.1.3 创建容器镜像仓库 ![image-20220125230046456](../../img/docker_image_fast/image-20220125230046456.png) ![image-20220125230216488](../../img/docker_image_fast/image-20220125230216488.png) ![image-20220125230307699](../../img/docker_image_fast/image-20220125230307699.png) ### 2.1.4 在本地登录Docker Hub ~~~powershell 默认可以不添加docker hub容器镜像仓库地址 # docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: dockersmartmsb Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded 成功 ~~~ ~~~powershell 登出 # docker logout Removing login credentials for https://index.docker.io/v1/ ~~~ ### 2.1.5 上传容器镜像 > 在登录Docker Hub主机上传容器镜像,向全球用户共享容器镜像。 ~~~powershell 为容器镜像重新打标记 原始容器镜像 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos latest 5d0da3dc9764 4 months ago 231MB 重新为容器镜像打标记 # docker tag centos:latest dockersmartmsb/centos:v1 重新打标记后容器镜像 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE dockersmartmsb/centos v1 5d0da3dc9764 4 months ago 231MB centos latest 5d0da3dc9764 4 months ago 231MB ~~~ ~~~powershell 上传容器镜像至docker hub # docker push dockersmartmsb/centos:v1 The push refers to repository [docker.io/dockersmartmsb/centos] 74ddd0ec08fa: Mounted from library/centos v1: digest: sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc size: 529 ~~~ ![image-20220125231912826](../../img/docker_image_fast/image-20220125231912826.png) ### 2.1.6 下载容器镜像 ~~~powershell 在其它主机上下载 下载 # docker pull dockersmartmsb/centos:v1 v1: Pulling from dockersmartmsb/centos a1d0c7532777: Pull complete Digest: sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc Status: Downloaded newer image for dockersmartmsb/centos:v1 docker.io/dockersmartmsb/centos:v1 查看下载后容器镜像 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE dockersmartmsb/centos v1 5d0da3dc9764 4 months ago 231MB ~~~ ## 2.2 harbor ### 2.2.1 获取 docker compose二进制文件 ~~~powershell 下载docker-compose二进制文件 # wget https://github.com/docker/compose/releases/download/1.25.0/docker-compose-Linux-x86_64 ~~~ ~~~powershell 查看已下载二进制文件 # ls docker-compose-Linux-x86_64 ~~~ ~~~powershell 移动二进制文件到/usr/bin目录，并更名为docker-compose # mv docker-compose-Linux-x86_64 /usr/bin/docker-compose ~~~ ~~~powershell 为二进制文件添加可执行权限 # chmod +x /usr/bin/docker-compose ~~~ ~~~powershell 安装完成后，查看docker-compse版本 # docker-compose version docker-compose version 1.25.0, build 0a186604 docker-py version: 4.1.0 CPython version: 3.7.4 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019 ~~~ ### 2.2.2 获取harbor安装文件 ![image-20220125232445910](../../img/docker_image_fast/image-20220125232445910.png) ![image-20220125232519365](../../img/docker_image_fast/image-20220125232519365.png) ![image-20220125233602760](../../img/docker_image_fast/image-20220125233602760.png) ![image-20220125233652604](../../img/docker_image_fast/image-20220125233652604.png) ![image-20220125233739356](../../img/docker_image_fast/image-20220125233739356.png) ~~~powershell 下载harbor离线安装包 # wget https://github.com/goharbor/harbor/releases/download/v2.4.1/harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell 查看已下载的离线安装包 # ls harbor-offline-installer-v2.4.1.tgz ~~~ ### 2.2.3 获取TLS文件 ~~~powershell 查看准备好的证书 # ls kubemsb.com_nginx.zip ~~~ ~~~powershell 解压证书压缩包文件 # unzip kubemsb.com_nginx.zip Archive: kubemsb.com_nginx.zip Aliyun Certificate Download inflating: 6864844_kubemsb.com.pem inflating: 6864844_kubemsb.com.key ~~~ ~~~powershell 查看解压出的文件 # ls 6864844_kubemsb.com.key 6864844_kubemsb.com.pem ~~~ ### 2.2.4 修改配置文件 ~~~powershell 解压harbor离线安装包 # tar xf harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell 查看解压出来的目录 # ls harbor ~~~ ~~~powershell 移动证书到harbor目录 # # mv 6864844_kubemsb.com.* harbor 查看harbor目录 # ls harbor 6864844_kubemsb.com.key 6864844_kubemsb.com.pem common.sh harbor.v2.4.1.tar.gz harbor.yml.tmpl install.sh LICENSE prepare ~~~ ~~~powershell 创建配置文件 # cd harbor/ # mv harbor.yml.tmpl harbor.yml ~~~ ~~~powershell 修改配置文件内容 # vim harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: www.kubemsb.com 修改为域名，而且一定是证书签发的域名 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /root/harbor/6864844_kubemsb.com.pem 证书 private_key: /root/harbor/6864844_kubemsb.com.key 密钥 # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: 12345 访问密码 ...... ~~~ ### 2.2.5 执行预备脚本 ~~~powershell # ./prepare ~~~ ~~~powershell 输出 prepare base dir is set to /root/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /data/secret/keys/secretkey Successfully called func: create_root_cert Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir ~~~ ### 2.2.6 执行安装脚本 ~~~powershell # ./install.sh ~~~ ~~~powershell 输出 [Step 0]: checking if docker is installed ... Note: docker version: 20.10.12 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.25.0 [Step 2]: loading Harbor images ... [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... prepare base dir is set to /root/harbor [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating registry ... done Creating registryctl ... done Creating redis ... done Creating harbor-portal ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ✔ ----Harbor has been installed and started successfully.---- ~~~ ### 2.2.7 验证运行情况 ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 71c0db683e4a goharbor/nginx-photon:v2.4.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp nginx 4e3b53a86f01 goharbor/harbor-jobservice:v2.4.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice df76e1eabbf7 goharbor/harbor-core:v2.4.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core eeb4d224dfc4 goharbor/harbor-portal:v2.4.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal 70e162c38b59 goharbor/redis-photon:v2.4.1 "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis 8bcc0e9b06ec goharbor/harbor-registryctl:v2.4.1 "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl d88196398df7 goharbor/registry-photon:v2.4.1 "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry ed5ba2ba9c82 goharbor/harbor-db:v2.4.1 "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db dcb4b57c7542 goharbor/harbor-log:v2.4.1 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log ~~~ ### 2.2.8 访问harbor UI界面 #### 2.2.8.1 在物理机通过浏览器访问 ![image-20220126000804490](../../img/docker_image_fast/image-20220126000804490.png) ![image-20220126000825616](../../img/docker_image_fast/image-20220126000825616.png) ![image-20220126000840905](../../img/docker_image_fast/image-20220126000840905.png) #### 2.2.8.2 在Docker Host主机通过域名访问 ~~~powershell 添加域名解析 # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.155 www.kubemsb.com ~~~ ![image-20220126001253192](../../img/docker_image_fast/image-20220126001253192.png) ![image-20220126001447862](../../img/docker_image_fast/image-20220126001447862.png) ![image-20220126001510880](../../img/docker_image_fast/image-20220126001510880.png) # 三、docker镜像上传至Harbor及从harbor下载 ## 3.1 修改docker daemon使用harbor ~~~powershell 添加/etc/docker/daemon.json文件，默认不存在，需要手动添加 # vim /etc/docker/daemon.json # cat /etc/docker/daemon.json { "insecure-registries": ["www.kubemsb.com"] } ~~~ ~~~powershell 重启加载daemon配置 # systemctl daemon-reload ~~~ ~~~powershell 重启docker # systemctl restart docker ~~~ ## 3.2 docker tag ~~~powershell 查看已有容器镜像文件 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos latest 5d0da3dc9764 4 months ago 231MB ~~~ ~~~powershell 为已存在镜像重新添加tag # docker tag centos:latest www.kubemsb.com/library/centos:v1 ~~~ ~~~powershell 再次查看本地容器镜像 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos latest 5d0da3dc9764 4 months ago 231MB www.kubemsb.com/library/centos v1 5d0da3dc9764 4 months ago 231MB ~~~ ## 3.3 docker push ~~~powershell # docker login www.kubemsb.com Username: admin 用户名 admin Password: 密码 12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded 登陆成功 ~~~ ~~~powershell 推送本地容器镜像到harbor仓库 # docker push www.kubemsb.com/library/centos:v1 ~~~ ![image-20220126002747864](../../img/docker_image_fast/image-20220126002747864.png) ## 3.4 docker pull > 在其它主机上下载或使用harbor容器镜像仓库中的容器镜像 ~~~powershell 在本地添加域名解析 # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.155 www.kubemsb.com ~~~ ~~~powershell 在本地添加/etc/docker/daemon.json文件，其中为本地主机访问的容器镜像仓库 # vim /etc/docker/daemon.json # cat /etc/docker/daemon.json { "insecure-registries": ["www.kubemsb.com"] } ~~~ ~~~powershell # systemctl daemon-reload # systemctl restart docker ~~~ ~~~powershell 下载容器镜像 # docker pull www.kubemsb.com/library/centos:v1 v1: Pulling from library/centos Digest: sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc Status: Downloaded newer image for www.kubemsb.com/library/centos:v1 www.kubemsb.com/library/centos:v1 ~~~ ~~~powershell 查看已下载的容器镜像 # docker images REPOSITORY TAG IMAGE ID CREATED SIZE www.kubemsb.com/library/centos v1 5d0da3dc9764 4 months ago 231MB ~~~ ================================================ FILE: docs/cloud/docker/docker_native.md ================================================ # 容器管理工具 Docker生态架构及部署 # 一、Docker生态架构 ## 1.1 Docker Containers Are Everywhere ![image-20220118165726624](../../img/docker_native/image-20220118165726624.png) ## 1.2 生态架构 ![image-20220118170228476](../../img/docker_native/image-20220118170228476.png) ### 1.2.1 Docker Host 用于安装Docker daemon的主机，即为Docker Host，并且该主机中可基于容器镜像运行容器。 ### 1.2.2 Docker daemon 用于管理Docker Host中运行的容器、容器镜像、容器网络等，管理由Containerd.io提供的容器。 ### 1.2.3 Registry 容器镜像仓库，用于存储已生成容器运行模板的仓库，用户使用时，可直接从容器镜像仓库中下载容器镜像，即容器运行模板，就可以运行容器镜像中包含的应用了。例如：Docker Hub,也可以使用Harbor实现企业私有的容器镜像仓库。 ### 1.2.4 Docker client Docker Daemon客户端工具，用于同Docker Daemon进行通信，执行用户指令，可部署在Docker Host上，也可以部署在其它主机，能够连接到Docker Daemon即可操作。 ### 1.2.5 Image 把应用运行环境及计算资源打包方式生成可再用于启动容器的不可变的基础设施的模板文件，主要用于基于其启动一个容器。 ### 1.2.6 Container 由容器镜像生成，用于应用程序运行的环境，包含容器镜像中所有文件及用户后添加的文件，属于基于容器镜像生成的可读写层，这也是应用程序活跃的空间。 ### 1.2.7 Docker Dashboard > 仅限于MAC与Windows操作系统上安装使用。 Docker Dashboard 提供了一个简单的界面，使您能够直接从您的机器管理您的容器、应用程序和映像，而无需使用 CLI 来执行核心操作。 ![image-20220118185507047](../../img/docker_native/image-20220118185507047.png) ## 1.3 Docker版本 - Docker-ce Docker社区版，主要用于个人开发者测试使用，免费版本 - Docker-ee Docker企业版，主要用于为企业开发及应用部署使用，收费版本，免费试用一个月，2020年因国际政治原因曾一度限制中国企业使用。 # 二、Docker部署 > 安装Docker-ce版本。 ## 2.1 使用YUM源部署 > YUM源可以使用官方YUM源、清华大学开源镜像站配置YUM源，也可以使用阿里云开源镜像站提供的YUM源，建议选择使用阿里云开源镜像站提供的YUM源，原因速度快。 ### 2.1.1 获取阿里云开源镜像站YUM源文件 ![image-20220118181640863](../../img/docker_native/image-20220118181640863.png) ![image-20220118181837065](../../img/docker_native/image-20220118181837065.png) ![image-20220118182432607](../../img/docker_native/image-20220118182432607.png) ~~~powershell 在docker host上使用 wget下载到/etc/yum.repos.d目录中即可。 # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ![image-20220118182749037](../../img/docker_native/image-20220118182749037.png) ### 2.1.2 安装Docker-ce > 在docker host上安装即可，本次使用YUM源中稳定版本，由于版本在不断更新，不同的时间安装版本也不相同，使用方法基本一致。 ~~~powershell 直接安装docker-ce，此为docker daemon，所有依赖将被yum自动安装，含docker client等。 # yum -y install docker-ce ~~~ ![image-20220118183627705](../../img/docker_native/image-20220118183627705.png) ### 2.1.3 配置Docker Daemon启动文件 > 由于Docker使用过程中会对Centos操作系统中的Iptables防火墙中的FORWARD链默认规划产生影响及需要让Docker Daemon接受用户自定义的daemon.json文件，需要要按使用者要求的方式修改。 ~~~powershell # vim /usr/lib/systemd/system/docker.service ~~~ ![image-20220118184308554](../../img/docker_native/image-20220118184308554.png) ![image-20220118184420795](../../img/docker_native/image-20220118184420795.png) ### 2.1.4 启动Docker服务并查看已安装版本 ~~~powershell 重启加载daemon文件 # systemctl daemon-reload 启动docker daemon # systemctl start docker 设置开机自启动 # systemctl enable docker ~~~ ~~~powershell 使用docker version客户端命令查看已安装docker软件版本 # docker version Client: Docker Engine - Community 客户端 Version: 20.10.12 API version: 1.41 Go version: go1.16.12 Git commit: e91ed57 Built: Mon Dec 13 11:45:41 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Docker管理引擎 Engine: Version: 20.10.12 API version: 1.41 (minimum version 1.12) Go version: go1.16.12 Git commit: 459d0df Built: Mon Dec 13 11:44:05 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.19.0 GitCommit: de40ad0 ~~~ ## 2.2 使用二进制文件部署 > 官方不建议此种部署方式，主因为不能自动更新，在条件有限制的情况下使用。 二进制安装参考网址：https://docs.docker.com/engine/install/binaries/ ![image-20220118185625614](../../img/docker_native/image-20220118185625614.png) ![image-20220118185711358](../../img/docker_native/image-20220118185711358.png) ![image-20220118185814368](../../img/docker_native/image-20220118185814368.png) ![image-20220118185911155](../../img/docker_native/image-20220118185911155.png) ![image-20220118190235182](../../img/docker_native/image-20220118190235182.png) ![image-20220118190327846](../../img/docker_native/image-20220118190327846.png) ![image-20220118190507974](../../img/docker_native/image-20220118190507974.png) ![image-20220118190622614](../../img/docker_native/image-20220118190622614.png) ![image-20220118190654730](../../img/docker_native/image-20220118190654730.png) ~~~powershell 获取二进制文件，此文件中包含dockerd与docker 2个文件。 # wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz ~~~ ~~~powershell 解压下载的文件 # tar xf docker-20.10.9.tgz 查看解压出的目录 # ls docker containerd containerd-shim-runc-v2 docker docker-init runc containerd-shim ctr dockerd docker-proxy ~~~ ~~~powershell 安装解压后的所有二进制文件 # cp docker/* /usr/bin/ ~~~ ~~~powershell 运行Daemon # dockerd & 会有大量的信息输出，停止后，直接回车即可使用。 ~~~ >如果您需要使用其他选项启动守护程序，请相应地修改上述命令或创建并编辑文件`/etc/docker/daemon.json` 以添加自定义配置选项。 ~~~powershell 确认是否可以使用docker客户端命令 # which docker /usr/bin/docker 使用二进制安装的docker客户端 # docker version Client: Version: 20.10.9 API version: 1.41 Go version: go1.16.8 Git commit: c2ea9bc Built: Mon Oct 4 16:03:22 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.9 API version: 1.41 (minimum version 1.12) Go version: go1.16.8 Git commit: 79ea9d3 Built: Mon Oct 4 16:07:30 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.4.11 GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8 runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2d docker-init: Version: 0.19.0 GitCommit: de40ad0 ~~~ ================================================ FILE: docs/cloud/docker/docker_network.md ================================================ # Docker容器网络与通信原理深度解析 # 一、Docker容器默认网络模型 ## 1.1 原理图 ![image-20220208180516669](../../img/docker_network/image-20220208180516669.png) ## 1.2 名词解释 - docker0 - 是一个二层网络设备，即网桥 - 通过网桥可以将Linux支持的不同的端口连接起来 - 实现类交换机多对多的通信 - veth pair - 虚拟以太网（Ethernet）设备 - 成对出现,用于解决网络命名空间之间的隔离 - 一端连接Container network namespace，另一端连接host network namespace # 二、Docker容器默认网络模型工作原理 ## 2.1 容器访问外网 ![image-20220208190823162](../../img/docker_network/image-20220208190823162.png) ~~~powershell # docker run -d --name web1 -p 8081:80 nginx:latest ~~~ ~~~powershell # iptables -t nat -vnL POSTROUTING ~~~ ~~~powershell 输出： Chain POSTROUTING (policy ACCEPT 7 packets, 766 bytes) pkts bytes target prot opt in out source destination 0 0 MASQUERADE tcp -- * * 172.17.0.2 172.17.0.2 tcp dpt:80 ~~~ ## 2.2 外网访问容器 ![image-20220208191400324](../../img/docker_network/image-20220208191400324.png) ~~~powershell # iptables -t nat -vnL DOCKER ~~~ ~~~powershell 输出： Chain DOCKER (2 references) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8081 to:172.17.0.2:80 ~~~ # 三、Docker容器四种网络模型 ![image-20220209120452509](../../img/docker_network/image-20220209120452509.png) | 模式 | 使用方法 | 说明 | | ------------------------------------------------------------ | ------------------------------------------ | ------------------------------------------------------------ | | bridge [桥接式网络(Bridge container A)] | --network bridge | 桥接容器，除了有一块本地回环接口(Loopback interface)外，还有一块私有接口(Private interface)通过容器虚拟接口(Container virtual interface)连接到桥接虚拟接口(Docker bridge virtual interface)，之后通过逻辑主机接口(Logical host interface)连接到主机物理网络(Physical network interface)。
桥接网卡默认会分配到172.17.0.0/16的IP地址段。
如果我们在创建容器时没有指定网络模型，默认就是(Nat)桥接网络，这也就是为什么我们在登录到一个容器后，发现IP地址段都在172.17.0.0/16网段的原因。 | | host [开放式容器(Open container)] | --network host | 比联盟式网络更开放，我们知道联盟式网络是多个容器共享网络(Net),而开放式容器(Open contaner)就直接共享了宿主机的名称空间。因此物理网卡有多少个，那么该容器就能看到多少网卡信息。我们可以说Open container是联盟式容器的衍生。 | | none [封闭式网络(Closed container)] | --network none | 封闭式容器，只有本地回环接口(Loopback interface，和咱们服务器看到的lo接口类似)，无法与外界进行通信。 | | container [联盟式网络(Joined container A \| Joined container B ] | --network container:c1(容器名称或容器ID) | 每个容器都各有一部分名称空间(Mount,PID,User)，另外一部分名称空间是共享的(UTS,Net,IPC)。
由于它们的网络是共享的，因此各个容器可以通过本地回环接口(Loopback interface)进行通信。
除了共享同一组本地回环接口(Loopback interface)外，还有一块一块私有接口(Private interface)通过联合容器虚拟接口(Joined container virtual interface)连接到桥接虚拟接口(Docker bridge virtual interface)，之后通过逻辑主机接口(Logical host interface)连接到主机物理网络(Physical network interface)。 | # 四、Docker容器四种网络模型应用案例 ## 4.1 查看已有的网络模型 ~~~powershell 查看已有的网络模型 # docker network ls NETWORK ID NAME DRIVER SCOPE a26c79961d8c bridge bridge local d04ce0d0e6ca host host local a369d8e58a41 none null local ~~~ ~~~powershell 查看已有网络模型详细信息 # docker network inspect bridge [ { "Name": "bridge", "Id": "a26c79961d8c3a5f66a7de782b773291e4902badc60d0614745e01b18f506907", "Created": "2022-02-08T11:45:25.607195911+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "dbac5dd601b960c91bee8fafcabc0a6e6091bff14d5fccfa80ca2c74df8891ad": { "Name": "web1", "EndpointID": "2c1d8c66f7f46d6d76e5c384b1729a90441e1398496b3112124ba65d255432a1", "MacAddress": "02:42:ac:11:00:02", "IPv4Address": "172.17.0.2/16", "IPv6Address": "" } }, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ] ~~~ ~~~powershell 查看docker支持的网络模型 # docker info | grep Network Network: bridge host ipvlan macvlan null overlay ~~~ ## 4.2 创建指定类型的网络模型 ### 4.2.1 bridge ~~~powershell 查看创建网络模型的帮助方法 # docker network create --help ~~~ ~~~powershell 创建一个名称为mybr0的网络 # docker network create -d bridge --subnet "192.168.100.0/24" --gateway "192.168.100.1" -o com.docker.network.bridge.name=docker1 mybr0 ~~~ ~~~powershell 查看已创建网络 # docker network ls NETWORK ID NAME DRIVER SCOPE ...... a6a1ad36c3c0 mybr0 bridge local ...... ~~~ ~~~powershell 在docker host主机上可以看到多了一个网桥docker1 # ifconfig docker1: flags=4099 mtu 1500 inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255 ether 02:42:14:aa:f5:04 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 20 bytes 1598 (1.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ~~~ ~~~powershell 启动一个容器并连接到已创建mybr0网络 # docker run -it --network mybr0 --rm busybox / # ifconfig eth0 Link encap:Ethernet HWaddr 02:42:C0:A8:65:02 inet addr:192.168.100.2 Bcast:192.168.100.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:18 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2185 (2.1 KiB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) / # exit ~~~ ### 4.2.2 host ~~~powershell 查看host类型的网络模型 # docker network ls NETWORK ID NAME DRIVER SCOPE ...... d04ce0d0e6ca host host local ...... ~~~ ~~~powershell 查看host网络模型的详细信息 # docker network inspect host [ { "Name": "host", "Id": "d04ce0d0e6ca8e6226937f19033ef2c3f05b47ed63e06492d5c3071904fbb80b", "Created": "2022-01-21T16:12:05.30970114+08:00", "Scope": "local", "Driver": "host", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": {} } ] ~~~ ~~~powershell 创建容器使用host网络模型，并查看其网络信息 # docker run -it --network host --rm busybox / # ifconfig docker0 Link encap:Ethernet HWaddr 02:42:11:B8:9A:C5 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 inet6 addr: fe80::42:11ff:feb8:9ac5/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:53 errors:0 dropped:0 overruns:0 frame:0 TX packets:94 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6924 (6.7 KiB) TX bytes:7868 (7.6 KiB) docker1 Link encap:Ethernet HWaddr 02:42:14:AA:F5:04 inet addr:192.168.100.1 Bcast:192.168.100.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) ens33 Link encap:Ethernet HWaddr 00:0C:29:AF:89:0B inet addr:192.168.255.161 Bcast:192.168.255.255 Mask:255.255.255.0 inet6 addr: fe80::44fc:2662:bfab:2b93/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:157763 errors:0 dropped:0 overruns:0 frame:0 TX packets:50865 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:205504721 (195.9 MiB) TX bytes:3626119 (3.4 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:88 errors:0 dropped:0 overruns:0 frame:0 TX packets:88 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8196 (8.0 KiB) TX bytes:8196 (8.0 KiB) virbr0 Link encap:Ethernet HWaddr 52:54:00:EB:01:E5 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) / # exit ~~~ **运行Nginx服务** ~~~powershell 创建用于运行nginx应用的容器，使用host网络模型 # docker run -d --network host nginx:latest ~~~ ~~~powershell 查看容器运行状态 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f6677b213271 nginx:latest "/docker-entrypoint.…" 7 seconds ago Up 6 seconds youthful_shtern ~~~ ~~~powershell 查看docker host 80端口状态 # ss -anput | grep ":80" tcp LISTEN 0 511 *:80 *:* users:(("nginx",pid=42866,fd=7),("nginx",pid=42826,fd=7)) tcp LISTEN 0 511 :::80 :::* users:(("nginx",pid=42866,fd=8),("nginx",pid=42826,fd=8)) ~~~ ~~~powershell 使用curl命令访问docker host主机IP地址，验证是否可以对nginx进行访问，如可访问，则说明容器与docker host共享网络命名空间 # curl http://192.168.255.161 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ ![image-20220209152555577](../../img/docker_network/image-20220209152555577.png) ### 4.2.3 none ~~~powershell 查看none类型的网络模型 # docker network ls NETWORK ID NAME DRIVER SCOPE ...... a369d8e58a41 none null local ~~~ ~~~powershell 查看none网络模型详细信息 # docker network inspect none [ { "Name": "none", "Id": "a369d8e58a41ce2e3c25f2273b059e984dd561bfa7e79077a0cce9b3a925b9c9", "Created": "2022-01-21T16:12:05.217801814+08:00", "Scope": "local", "Driver": "null", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": {} } ] ~~~ ~~~powershell 创建容器使用none网络模型，并查看其网络状态 # docker run -it --network none --rm busybox:latest / # ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) / # exit ~~~ ### 4.2.4 联盟网络 ~~~powershell 创建c1容器，使用默认网络模型 # docker run -it --name c1 --rm busybox:latest / # ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02 inet addr:172.17.0.2 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:16 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1916 (1.8 KiB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) ~~~ ~~~powershell 查看c1容器状态 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0905bc8ebfb6 busybox:latest "sh" 13 seconds ago Up 11 seconds c1 ~~~ ~~~powershell 创建c2容器，与c1容器共享网络命名空间 # docker run -it --name c2 --network container:c1 --rm busybox:latest / # ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02 inet addr:172.17.0.2 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:22 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2574 (2.5 KiB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) ~~~ ~~~powershell 在c2容器中创建文件并开启httpd服务 / # echo "hello world" >> /tmp/index.html / # ls /tmp index.html / # httpd -h /tmp 验证80端口是否打开 / # netstat -npl Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 :::80 :::* LISTEN 10/httpd ~~~ ~~~powershell 在c1容器中进行访问验证 # docker exec c1 wget -O - -q 127.0.0.1 hello world ~~~ ~~~powershell 查看c1容器/tmp目录，发现没有在c2容器中创建的文件，说明c1与c2仅共享了网络命名空间，没有共享文件系统 # docker exec c1 ls /tmp ~~~ # 五、跨Docker Host容器间通信实现 ## 5.1 跨Docker Host容器间通信必要性 - 由于Docker容器运行的环境类似于在局域网中运行服务一样，无法直接被外界访问，如果采用在Docker Host利用端口映射方式会导致端口被严重消耗。 - 能够实现不同的Docker Host方便访问其它Docker Host之上的容器提供的服务 ## 5.2 跨Docker Host容器间通信实现方案 ### 5.2.1 Docker原生方案 - overlay - 基于VXLAN封装实现Docker原生overlay网络 - macvlan - Docker主机网卡接口逻辑上分为多个子接口，每个子接口标识一个VLAN，容器接口直接连接Docker Host - 网卡接口 - 通过路由策略转发到另一台Docker Host ### 5.2.2 第三方方案 #### 5.2.2.1 隧道方案 - Flannel - 支持UDP和VLAN封装传输方式 - Weave - 支持UDP和VXLAN - OpenvSwitch - 支持VXLAN和GRE协议 #### 5.2.2.2 路由方案 - Calico - 支持BGP协议和IPIP隧道 - 每台宿主机作为虚拟路由，通过BGP协议实现不同主机容器间通信。 ## 5.3 Flannel ### 5.3.1 overlay network介绍 Overlay网络是指在不改变现有网络基础设施的前提下，通过某种约定通信协议，把二层报文封装在IP报文之上的新的数据格式。这样不但能够充分利用成熟的IP路由协议进程数据分发；而且在Overlay技术中采用扩展的隔离标识位数，能够突破VLAN的4000数量限制支持高达16M的用户，并在必要时可将广播流量转化为组播流量，避免广播数据泛滥。 　　因此，Overlay网络实际上是目前最主流的容器跨节点数据传输和路由方案。 ### 5.3.2 Flannel介绍 Flannel是 CoreOS 团队针对 Kubernetes 设计的一个覆盖网络（Overlay Network）工具，其目的在于帮助每一个使用 Kuberentes 的 CoreOS 主机拥有一个完整的子网。 Flannel通过给每台宿主机分配一个子网的方式为容器提供虚拟网络，它基于Linux TUN/TAP，使用UDP封装IP包来创建overlay网络，并借助etcd维护网络的分配情况。 Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes. ### 5.3.3 Flannel工作原理 Flannel是CoreOS团队针对Kubernetes设计的一个网络规划服务，简单来说，它的功能是让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。但在默认的Docker配置中，每个Node的Docker服务会分别负责所在节点容器的IP分配。Node内部的容器之间可以相互访问,但是跨主机(Node)网络相互间是不能通信。Flannel设计目的就是为集群中所有节点重新规划IP地址的使用规则，从而使得不同节点上的容器能够获得"同属一个内网"且"不重复的"IP地址，并让属于不同节点上的容器能够直接通过内网IP通信。 Flannel 使用etcd存储配置数据和子网分配信息。flannel 启动之后，后台进程首先检索配置和正在使用的子网列表，然后选择一个可用的子网，然后尝试去注册它。etcd也存储这个每个主机对应的ip。flannel 使用etcd的watch机制监视`/coreos.com/network/subnets`下面所有元素的变化信息，并且根据它来维护一个路由表。为了提高性能，flannel优化了Universal TAP/TUN设备，对TUN和UDP之间的ip分片做了代理。 如下原理图： ![image-20220209230459054](../../img/docker_network/image-20220209230459054.png) ```text 1、数据从源容器中发出后，经由所在主机的docker0虚拟网卡转发到flannel0虚拟网卡，这是个P2P的虚拟网卡，flanneld服务监听在网卡的另外一端。 2、Flannel通过Etcd服务维护了一张节点间的路由表，该张表里保存了各个节点主机的子网网段信息。 3、源主机的flanneld服务将原本的数据内容UDP封装后根据自己的路由表投递给目的节点的flanneld服务，数据到达以后被解包，然后直接进入目的节点的flannel0虚拟网卡，然后被转发到目的主机的docker0虚拟网卡，最后就像本机容器通信一样的由docker0路由到达目标容器。 ``` ## 5.4 ETCD etcd是CoreOS团队于2013年6月发起的开源项目，它的目标是构建一个高可用的分布式键值(key-value)数据库。etcd内部采用`raft`协议作为一致性算法，etcd基于Go语言实现。 etcd作为服务发现系统，特点： - 简单：安装配置简单，而且提供了HTTP API进行交互，使用也很简单 - 安全：支持SSL证书验证 - 快速：根据官方提供的benchmark数据，单实例支持每秒2k+读操作 - 可靠：采用raft算法，实现分布式系统数据的可用性和一致性 ## 5.5 ETCD部署 > 主机防火墙及SELINUX均关闭。 ### 5.5.1 主机名称配置 ~~~powershell # hostnamectl set-hostname node1 ~~~ ~~~powershell # hostnamectl set-hostname node2 ~~~ ### 5.5.2 主机IP地址配置 ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="6c020cf7-4c6e-4276-9aa6-0661670da705" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.255.154" PREFIX="24" GATEWAY="192.168.255.2" DNS1="119.29.29.29" ~~~ ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="6c020cf7-4c6e-4276-9aa6-0661670da705" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.255.155" PREFIX="24" GATEWAY="192.168.255.2" DNS1="119.29.29.29" ~~~ ### 5.5.3 主机名与IP地址解析 ~~~powershell # vim /etc/hosts [root@node1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.255.154 node1 192.168.255.155 node2 ~~~ ~~~powershell # vim /etc/hosts [root@node2 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.255.154 node1 192.168.255.155 node2 ~~~ ### 5.5.4 开启内核转发 > 所有Docker Host ~~~powershell # vim /etc/sysctl.conf [root@node1 ~]# cat /etc/sysctl.conf ...... net.ipv4.ip_forward=1 ~~~ ~~~powershell # sysctl -p ~~~ ~~~powershell # vim /etc/sysctl.conf [root@node2 ~]# cat /etc/sysctl.conf ...... net.ipv4.ip_forward=1 ~~~ ~~~powershell # sysctl -p ~~~ ### 5.5.5 etcd安装 > etcd集群 ~~~powershell [root@node1 ~]# yum -y install etcd ~~~ ~~~powershell [root@node2 ~]# yum -y install etcd ~~~ ### 5.5.6 etcd配置 ~~~powershell # vim /etc/etcd/etcd.conf [root@node1 ~]# cat /etc/etcd/etcd.conf #[Member] #ETCD_CORS="" ETCD_DATA_DIR="/var/lib/etcd/node1.etcd" #ETCD_WAL_DIR="" ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" ETCD_NAME="node1" #ETCD_SNAPSHOT_COUNT="100000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_QUOTA_BACKEND_BYTES="0" #ETCD_MAX_REQUEST_BYTES="1572864" #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s" #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s" #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s" # #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.255.154:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.255.154:2379,http://192.168.255.155:4001" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_DISCOVERY_SRV="" ETCD_INITIAL_CLUSTER="node1=http://192.168.255.154:2380,node2=http://192.168.255.155:2380" #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_STRICT_RECONFIG_CHECK="true" #ETCD_ENABLE_V2="true" # #[Proxy] ~~~ ~~~powershell # vim /etc/etcd/etcd.conf [root@node2 ~]# cat /etc/etcd/etcd.conf #[Member] #ETCD_CORS="" ETCD_DATA_DIR="/var/lib/etcd/node2.etcd" #ETCD_WAL_DIR="" ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380" ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" ETCD_NAME="node2" #ETCD_SNAPSHOT_COUNT="100000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_QUOTA_BACKEND_BYTES="0" #ETCD_MAX_REQUEST_BYTES="1572864" #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s" #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s" #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s" # #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.255.155:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.255.155:2379,http://192.168.255.155:4001" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_DISCOVERY_SRV="" ETCD_INITIAL_CLUSTER="node1=http://192.168.255.154:2380,node2=http://192.168.255.155:2380" #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_STRICT_RECONFIG_CHECK="true" #ETCD_ENABLE_V2="true" # #[Proxy] ~~~ ### 5.5.7 启动etcd服务 ~~~powershell [root@node1 ~]# systemctl enable etcd [root@node1 ~]# systemctl start etcd ~~~ ~~~powershell [root@node2 ~]# systemctl enable etcd [root@node2 ~]# systemctl start etcd ~~~ ### 5.5.8 检查端口状态 ~~~powershell # netstat -tnlp | grep -E "4001|2380" ~~~ ~~~powershell 输出结果： tcp6 0 0 :::2380 :::* LISTEN 65318/etcd tcp6 0 0 :::4001 :::* LISTEN 65318/etcd ~~~ ### 5.5.9 检查etcd集群是否健康 ~~~powershell # etcdctl -C http://192.168.255.154:2379 cluster-health ~~~ ~~~powershell 输出： member 5be09658727c5574 is healthy: got healthy result from http://192.168.255.154:2379 member c48e6c7a65e5ca43 is healthy: got healthy result from http://192.168.255.155:2379 cluster is healthy ~~~ ~~~powershell # etcdctl member list ~~~ ~~~powershell 输出： 5be09658727c5574: name=node1 peerURLs=http://192.168.255.154:2380 clientURLs=http://192.168.255.154:2379,http://192.168.255.155:4001 isLeader=true c48e6c7a65e5ca43: name=node2 peerURLs=http://192.168.255.155:2380 clientURLs=http://192.168.255.155:2379,http://192.168.255.155:4001 isLeader=false ~~~ ## 5.6 Flannel部署 ### 5.6.1 Flannel安装 ~~~powershell [root@node1 ~]# yum -y install flannel ~~~ ~~~powershell [root@node2 ~]# yum -y install flannel ~~~ ### 5.6.2 修改Flannel配置文件 ~~~powershell [root@node1 ~]# vim /etc/sysconfig/flanneld [root@node1 ~]# cat /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD_ENDPOINTS="http://192.168.255.154:2379,http://192.168.255.155:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/atomic.io/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="" FLANNEL_OPTIONS="--logtostderr=false --log_dir=/var/log/ --etcd endpoints=http://192.168.255.154:2379,http://192.168.255.155:2379 --iface=ens33" ~~~ ~~~powershell [root@node2 ~]# vim /etc/sysconfig/flanneld [root@node2 ~]# cat /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD_ENDPOINTS="http://192.168.255.154:2379,http://192.168.255.155:2379" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_PREFIX="/atomic.io/network" # Any additional options that you want to pass #FLANNEL_OPTIONS="" FLANNEL_OPTIONS="--logtostderr=false --log_dir=/var/log/ --etcd-endpoints=http://192.168.255.154:2379,http://192.168.255.155:2379 --iface=ens33" ~~~ ### 5.6.3 配置etcd中关于flannel的key Flannel使用Etcd进行配置，来保证多个Flannel实例之间的配置一致性，所以需要在etcd上进行如下配置（**'/[http://atomic.io/network/config](https://link.zhihu.com/?target=http%3A//atomic.io/network/config)**'这个key与上面的/etc/sysconfig/flannel中的配置项FLANNEL_ETCD_PREFIX是相对应的，错误的话启动就会出错） >该ip网段可以任意设定，随便设定一个网段都可以。容器的ip就是根据这个网段进行自动分配的，ip分配后，容器一般是可以对外联网的（网桥模式，只要Docker Host能上网即可。） ~~~powershell [root@node1 ~]# etcdctl mk /atomic.io/network/config '{"Network":"172.21.0.0/16"}' {"Network":"172.21.0.0/16"} ~~~ 或 ~~~powershell [root@node1 ~]# etcdctl set /atomic.io/network/config '{"Network":"172.21.0.0/16"}' {"Network":"172.21.0.0/16"} ~~~ ~~~powershell [root@node1 ~]# etcdctl get /atomic.io/network/config {"Network":"172.21.0.0/16"} ~~~ ### 5.6.4 启动Flannel服务 ~~~powershell [root@node1 ~]# systemctl enable flanneld;systemctl start flanneld ~~~ ~~~powershell [root@node2 ~]# systemctl enable flanneld;systemctl start flanneld ~~~ ### 5.6.5 查看各node中flannel产生的配置信息 ~~~powershell [root@node1 ~]# ls /run/flannel/ docker subnet.env [root@node1 ~]# cat /run/flannel/subnet.env FLANNEL_NETWORK=172.21.0.0/16 FLANNEL_SUBNET=172.21.31.1/24 FLANNEL_MTU=1472 FLANNEL_IPMASQ=false ~~~ ~~~powershell [root@node1 ~]# ip a s ...... 5: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:63:d1:9e:0b brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 6: flannel0: mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/none inet 172.21.31.0/16 scope global flannel0 valid_lft forever preferred_lft forever inet6 fe80::edfa:d8b0:3351:4126/64 scope link flags 800 valid_lft forever preferred_lft forever ~~~ ~~~powershell [root@node2 ~]# ls /run/flannel/ docker subnet.env [root@node2 ~]# cat /run/flannel/subnet.env FLANNEL_NETWORK=172.21.0.0/16 FLANNEL_SUBNET=172.21.55.1/24 FLANNEL_MTU=1472 FLANNEL_IPMASQ=false ~~~ ~~~powershell [root@node2 ~]# ip a s ...... 5: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:e1:16:68:de brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 6: flannel0: mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/none inet 172.21.55.0/16 scope global flannel0 valid_lft forever preferred_lft forever inet6 fe80::f895:9b5a:92b1:78aa/64 scope link flags 800 valid_lft forever preferred_lft forever ~~~ ## 5.7 Docker网络配置 > --bip=172.21.31.1/24 --ip-masq=true --mtu=1472 放置于启动程序后 ~~~powershell [root@node1 ~]# vim /usr/lib/systemd/system/docker.service [root@node1 ~]# cat /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket containerd.service [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --bip=172.21.31.1/24 --ip-masq=true --mtu=1472 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always # Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229. # Both the old, and new location are accepted by systemd 229 and up, so using the old location # to make them work for either version of systemd. StartLimitBurst=3 # Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230. # Both the old, and new name are accepted by systemd 230 and up, so using the old name to make # this option work for either version of systemd. StartLimitInterval=60s # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not support it. # Only systemd 226 and above support this option. TasksMax=infinity # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process OOMScoreAdjust=-500 [Install] WantedBy=multi-user.target ~~~ ~~~powershell [root@node2 ~]# vim /usr/lib/systemd/system/docker.service [root@node2 ~]# cat /usr/lib/systemd/system/docker.service [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket containerd.service [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --bip=172.21.55.1/24 --ip-masq=true --mtu=1472 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always # Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229. # Both the old, and new location are accepted by systemd 229 and up, so using the old location # to make them work for either version of systemd. StartLimitBurst=3 # Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230. # Both the old, and new name are accepted by systemd 230 and up, so using the old name to make # this option work for either version of systemd. StartLimitInterval=60s # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not support it. # Only systemd 226 and above support this option. TasksMax=infinity # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process OOMScoreAdjust=-500 [Install] WantedBy=multi-user.target ~~~ ~~~powershell [root@node1 ~]# systemctl daemon-reload [root@node1 ~]# systemctl restart docker ~~~ ~~~powershell [root@node1 ~]# ip a s ...... 5: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:63:d1:9e:0b brd ff:ff:ff:ff:ff:ff inet 172.21.31.1/24 brd 172.21.31.255 scope global docker0 valid_lft forever preferred_lft forever 6: flannel0: mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/none inet 172.21.31.0/16 scope global flannel0 valid_lft forever preferred_lft forever inet6 fe80::edfa:d8b0:3351:4126/64 scope link flags 800 valid_lft forever preferred_lft forever ~~~ ~~~powershell [root@node2 ~]# systemctl daemon-reload [root@node2 ~]# systemctl restart docker ~~~ ~~~powershell [root@node2 ~]# ip a s ...... 5: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:e1:16:68:de brd ff:ff:ff:ff:ff:ff inet 172.21.55.1/24 brd 172.21.55.255 scope global docker0 valid_lft forever preferred_lft forever 6: flannel0: mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/none inet 172.21.55.0/16 scope global flannel0 valid_lft forever preferred_lft forever inet6 fe80::f895:9b5a:92b1:78aa/64 scope link flags 800 valid_lft forever preferred_lft forever ~~~ ## 5.8 跨Docker Host容器间通信验证 ~~~powershell [root@node1 ~]# docker run -it --rm busybox:latest / # ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:15:1F:02 inet addr:172.21.31.2 Bcast:172.21.31.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1472 Metric:1 RX packets:21 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2424 (2.3 KiB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) / # ping 172.21.55.2 PING 172.21.55.2 (172.21.55.2): 56 data bytes 64 bytes from 172.21.55.2: seq=0 ttl=60 time=2.141 ms 64 bytes from 172.21.55.2: seq=1 ttl=60 time=1.219 ms 64 bytes from 172.21.55.2: seq=2 ttl=60 time=0.730 ms ^C --- 172.21.55.2 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.730/1.363/2.141 ms ~~~ ~~~powershell [root@node2 ~]# docker run -it --rm busybox:latest / # ifconfig eth0 Link encap:Ethernet HWaddr 02:42:AC:15:37:02 inet addr:172.21.55.2 Bcast:172.21.55.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1472 Metric:1 RX packets:19 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2246 (2.1 KiB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) / # ping 172.21.31.2 PING 172.21.31.2 (172.21.31.2): 56 data bytes 64 bytes from 172.21.31.2: seq=0 ttl=60 time=1.286 ms 64 bytes from 172.21.31.2: seq=1 ttl=60 time=0.552 ms ^C --- 172.21.31.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.552/0.919/1.286 ms ~~~ ================================================ FILE: docs/cloud/docker/docker_nginx.md ================================================ # 使用容器运行Nginx应用及Docker命令 # 一、使用容器运行Nginx应用 ## 1.1 使用docker run命令运行Nginx应用 ### 1.1.1 观察下载容器镜像过程 > 查找本地容器镜像文件 ~~~powershell 执行命令过程一：下载容器镜像 # docker run -d nginx:latest Unable to find image 'nginx:latest' locally latest: Pulling from library/nginx a2abf6c4d29d: Downloading 1.966MB/31.36MB 下载中 a9edb18cadd1: Downloading 1.572MB/25.35MB 589b7251471a: Download complete 下载完成 186b1aaa4aa6: Download complete b4df32aa5a72: Waiting 等待下载 a0bcbecc962e: Waiting ~~~ ~~~powershell 执行命令过程二：下载容器镜像 [root@localhost ~]# docker run -d nginx:latest Unable to find image 'nginx:latest' locally latest: Pulling from library/nginx a2abf6c4d29d: Downloading 22.87MB/31.36MB a9edb18cadd1: Downloading 22.78MB/25.35MB 589b7251471a: Waiting 186b1aaa4aa6: Waiting b4df32aa5a72: Waiting ~~~ ~~~powershell 执行命令过程三：下载容器镜像 [root@localhost ~]# docker run -d nginx:latest Unable to find image 'nginx:latest' locally latest: Pulling from library/nginx a2abf6c4d29d: Pull complete 下载完成 a9edb18cadd1: Pull complete 589b7251471a: Pull complete 186b1aaa4aa6: Pull complete b4df32aa5a72: Waiting 等待下载 ~~~ ### 1.1.2 观察容器运行情况 ~~~powershell # docker run -d nginx:latest 9834c8c18a7c7c89ab0ea4119d11bafe9c18313c8006bc02ce57ff54d9a1cc0c ~~~ ~~~powershell 命令解释 docker run 启动一个容器 -d 把容器镜像中需要执行的命令以daemon（守护进程）的方式运行 nginx 应用容器镜像的名称，通常表示该镜像为某一个软件 latest 表示上述容器镜像的版本，表示最新版本，用户可自定义其标识，例如v1或v2等 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9834c8c18a7c nginx:latest "/docker-entrypoint.…" 24 seconds ago Up 23 seconds 80/tcp condescending_pare ~~~ ~~~powershell 命令解释 docker ps 类似于Linux系统的ps命令，查看正在运行的容器，如果想查看没有运行的容器，需要在此命令后使用--all ~~~ **输出内容解释** | CONTAINERID | IMAGE | COMMAND | CREATED | STATUS | PORTS | NAMES | | ------------ | ------------ | ---------------------- | -------------- | ------------- | ------ | ------------------ | | 9834c8c18a7c | nginx:latest | "/docker-entrypoint.…" | 24 seconds ago | Up 23 seconds | 80/tcp | condescending_pare | ## 1.2 访问容器中运行的Nginx服务 ### 1.2.1 确认容器IP地址 > 实际工作中不需要此步操作。 ~~~powershell # docker inspect 9834 "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.2", 容器IP地址 "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:02", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "d3de2fdbc30ee36a55c1431ef3ae4578392e552009f00b2019b4720735fe5a60", "EndpointID": "d91f47c9f756ff22dc599a207164f2e9366bd0c530882ce0f08ae2278fb3d50c", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.2", 容器IP地址 "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:02", "DriverOpts": null } } } } ] ~~~ ~~~powershell 命令解释 docker inspect 为查看容器结构信息命令 9834 为前面生成的容器ID号前4位，使用这个ID号时，由于其较长，使用时能最短识别即可。 ~~~ ### 1.2.2 容器网络说明 ![image-20220121172037253](../../img/docker_nginx/image-20220121172037253.png) ~~~powershell # ip a s ...... docker0网桥，用于为容器提供桥接，转发到主机之外的网络 5: docker0: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:d5:c3:d4:cc brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:d5ff:fec3:d4cc/64 scope link valid_lft forever preferred_lft forever 与容器中的虚拟网络设备在同一个命名空间中，用于把容器中的网络连接到主机 9: veth393dece@if8: mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 02:e3:11:58:54:0f brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::e3:11ff:fe58:540f/64 scope link valid_lft forever preferred_lft forever ~~~ ### 1.2.3 使用curl命令访问 ~~~powershell # curl http://172.17.0.2 返回结果，表示访问成功！ Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ # 二、Docker命令 ## 2.1 Docker命令获取帮助方法 ~~~powershell # docker -h Flag shorthand -h has been deprecated, please use --help Usage: docker [OPTIONS] COMMAND 用法 A self-sufficient runtime for containers 功能介绍 Options: 选项 --config string Location of client config files (default "/root/.docker") -c, --context string Name of the context to use to connect to the daemon (overrides DOCKER_HOST env var and default context set with "docker context use") -D, --debug Enable debug mode -H, --host list Daemon socket(s) to connect to -l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info") --tls Use TLS; implied by --tlsverify --tlscacert string Trust certs signed only by this CA (default "/root/.docker/ca.pem") --tlscert string Path to TLS certificate file (default "/root/.docker/cert.pem") --tlskey string Path to TLS key file (default "/root/.docker/key.pem") --tlsverify Use TLS and verify the remote -v, --version Print version information and quit Management Commands: 管理类命令 app* Docker App (Docker Inc., v0.9.1-beta3) builder Manage builds buildx* Docker Buildx (Docker Inc., v0.7.1-docker) config Manage Docker configs container Manage containers context Manage contexts image Manage images manifest Manage Docker image manifests and manifest lists network Manage networks node Manage Swarm nodes plugin Manage plugins scan* Docker Scan (Docker Inc., v0.12.0) secret Manage Docker secrets service Manage services stack Manage Docker stacks swarm Manage Swarm system Manage Docker trust Manage trust on Docker images volume Manage volumes Commands: 未分组命令 attach Attach local standard input, output, and error streams to a running container build Build an image from a Dockerfile commit Create a new image from a container's changes cp Copy files/folders between a container and the local filesystem create Create a new container diff Inspect changes to files or directories on a container's filesystem events Get real time events from the server exec Run a command in a running container export Export a container's filesystem as a tar archive history Show the history of an image images List images import Import the contents from a tarball to create a filesystem image info Display system-wide information inspect Return low-level information on Docker objects kill Kill one or more running containers load Load an image from a tar archive or STDIN login Log in to a Docker registry logout Log out from a Docker registry logs Fetch the logs of a container pause Pause all processes within one or more containers port List port mappings or a specific mapping for the container ps List containers pull Pull an image or a repository from a registry push Push an image or a repository to a registry rename Rename a container restart Restart one or more containers rm Remove one or more containers rmi Remove one or more images run Run a command in a new container save Save one or more images to a tar archive (streamed to STDOUT by default) search Search the Docker Hub for images start Start one or more stopped containers stats Display a live stream of container(s) resource usage statistics stop Stop one or more running containers tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE top Display the running processes of a container unpause Unpause all processes within one or more containers update Update configuration of one or more containers version Show the Docker version information wait Block until one or more containers stop, then print their exit codes ~~~ ## 2.2 Docker官网提供的命令说明 网址链接：https://docs.docker.com/reference/ ![image-20220121173519802](../../img/docker_nginx/image-20220121173519802.png) ![image-20220121173613294](../../img/docker_nginx/image-20220121173613294.png) ![image-20220121173705508](../../img/docker_nginx/image-20220121173705508.png) ## 2.3 docker命令应用 ### 2.3.1 docker run ~~~powershell # docker run -i -t --name c1 centos:latest bash [root@948f234e22a1 /]# ~~~ ~~~powershell 命令解释 docker run 运行一个命令在容器中，命令是主体，没有命令容器就会消亡 -i 交互式 -t 提供终端 --name c1 把将运行的容器命名为c1 centos:latest 使用centos最新版本容器镜像 bash 在容器中执行的命令 ~~~ ~~~powershell 查看主机名 [root@948f234e22a1 /]# ~~~ ~~~powershell 查看网络信息 [root@948f234e22a1 /]# ip a s 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 12: eth0@if13: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever ~~~ ~~~powershell 查看进程 [root@948f234e22a1 /]# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.1 12036 2172 pts/0 Ss 09:58 0:00 bash root 16 0.0 0.0 44652 1784 pts/0 R+ 10:02 0:00 ps aux ~~~ ~~~powershell 查看用户 [root@948f234e22a1 /]# cat /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin nobody:x:65534:65534:Kernel Overflow User:/:/sbin/nologin dbus:x:81:81:System message bus:/:/sbin/nologin systemd-coredump:x:999:997:systemd Core Dumper:/:/sbin/nologin systemd-resolve:x:193:193:systemd Resolver:/:/sbin/nologin ~~~ ~~~powershell 查看目录 [root@948f234e22a1 /]# pwd / [root@948f234e22a1 /]# ls bin etc lib lost+found mnt proc run srv tmp var dev home lib64 media opt root sbin sys usr ~~~ ~~~powershell 退出命令执行，观察容器运行情况 [root@948f234e22a1 /]# exit exit [root@localhost ~]# ~~~ ### 2.3.2 docker ps ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ~~~ ~~~powershell 命令解释 docker ps 查看正在运行的容器，本案例由于没有命令在容器中运行，因此容器被停止了，所以本次查看没有结果。 ~~~ ~~~powershell # docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 948f234e22a1 centos:latest "bash" 10 minutes ago Exited (0) 2 minutes ago c1 ~~~ | CONTAINERID | IMAGE | COMMAND | CREATED | STATUS | PORTS | NAMES | | ------------ | ------------- | ------- | -------------- | ------------------------ | ----- | ----- | | 948f234e22a1 | centos:latest | "bash" | 10 minutes ago | Exited (0) 2 minutes ago | | c1 | ~~~powershell 命令解释 docker ps --all 可以查看正在运行的和停止运行的容器 ~~~ ### 2.3.3 docker inspect ~~~powershell # docker run -it --name c2 centos:latest bash [root@9f2eea16da4c /]# ~~~ ~~~powershell 操作说明 在上述提示符处按住ctrl键，再按p键与q键，可以退出交互式的容器，容器会处于运行状态。 ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 37 seconds ago Up 35 seconds c2 ~~~ ~~~powershell 命令解释 可以看到容器处于运行状态 ~~~ ~~~powershell # docker inspect c2 "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "d3de2fdbc30ee36a55c1431ef3ae4578392e552009f00b2019b4720735fe5a60", "EndpointID": "d1a2b7609f2f73a6cac67229a4395eef293f695c0ac4fd6c9c9e6913c9c85c1c", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:02", "DriverOpts": null } } } } ] ~~~ ~~~powershell 命令解释 docker inpect 查看容器详细信息 ~~~ ### 2.3.4 docker exec ~~~powershell # docker exec -it c2 ls /root anaconda-ks.cfg anaconda-post.log original-ks.cfg ~~~ ~~~powershell 命令解释 docker exec 在容器外实现与容器交互执行某命令 -it 交互式 c2 正在运行的容器名称 ls /root 在正在运行的容器中运行相关的命令 ~~~ ~~~powershell 下面命令与上面命令执行效果一致 # docker exec c2 ls /root anaconda-ks.cfg anaconda-post.log original-ks.cfg ~~~ ### 2.3.5 docker attach ~~~powershell 查看正在运行的容器 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 13 minutes ago Up 13 minutes c2 ~~~ ~~~powershell [root@localhost ~]# docker attach c2 [root@9f2eea16da4c /]# ~~~ ~~~powershell 命令解释 docker attach 类似于ssh命令，可以进入到容器中 c2 正在运行的容器名称 ~~~ ~~~powershell 说明 docker attach 退出容器时，如不需要容器再运行，可直接使用exit退出；如需要容器继续运行，可使用ctrl+p+q ~~~ ### 2.3.6 docker stop ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 22 minutes ago Up 22 minutes c2 ~~~ ~~~powershell # docker stop 9f2eea 9f2eea ~~~ ~~~powershell # docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 22 minutes ago Exited (137) 4 seconds ago c2 ~~~ ### 2.3.7 docker start ~~~powershell # docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 22 minutes ago Exited (137) 4 seconds ago c2 ~~~ ~~~powershell # docker start 9f2eea 9f2eea ~~~ ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 24 minutes ago Up 16 seconds c2 ~~~ ### 2.3.8 docker top >在Docker Host查看容器中运行的进程信息 ~~~powershell # docker top c2 UID PID PPID C STIME TTY TIME CMD root 69040 69020 0 18:37 pts/0 00:00:00 bash ~~~ | UID | PID | PPID | C | STIME | TTY | TIME | CMD | | ---- | ----- | ----- | ---- | ----- | ----- | -------- | ---- | | root | 69040 | 69020 | 0 | 18:37 | pts/0 | 00:00:00 | bash | ~~~powershell 命令解释 docker top 查看container内进程信息，指在docker host上查看，与docker exec -it c2 ps -ef不同。 ~~~ ~~~powershell 输出说明 UID 容器中运行的命令用户ID PID 容器中运行的命令PID PPID 容器中运行的命令父PID，由于PPID是一个容器，此可指为容器在Docker Host中进程ID C 占用CPU百分比 STIME 启动时间 TTY 运行所在的终端 TIME 运行时间 CMD 执行的命令 ~~~ ### 2.3.9 docker rm > 如果容器已停止，使用此命令可以直接删除；如果容器处于运行状态，则需要提前关闭容器后，再删除容器。下面演示容器正在运行关闭后删除的方法。 #### 2.3.9.1 指定删除容器 ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9f2eea16da4c centos:latest "bash" 2 days ago Up 3 seconds c2 ~~~ ~~~powershell # docker stop c2 或 # docker stop 9f2eea16da4c ~~~ ~~~powershell # docker rm c2 或 # docker rm 9f2eea16da4c ~~~ #### 2.3.9.2 批量删除容器 ~~~powershell # docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 948f234e22a1 centos:latest "bash" 2 days ago Exited (0) 2 days ago c1 01cb3e01273c centos:latest "bash" 2 days ago Exited (0) 2 days ago systemimage1 46d950fdfb33 nginx:latest "/docker-ent..." 2 days ago Exited (0) 2 days ago upbeat_goldberg ~~~ ~~~powershell # docker ps --all | awk '{if (NR>=2){print $1}}' | xargs docker rm ~~~ ================================================ FILE: docs/cloud/docker/docker_swarm.md ================================================ # Docker主机集群化方案 Docker Swarm # 一、docker swarm介绍 Docker Swarm是Docker官方提供的一款集群管理工具，其主要作用是把若干台Docker主机抽象为一个整体，并且通过一个入口统一管理这些Docker主机上的各种Docker资源。Swarm和Kubernetes比较类似，但是更加轻，具有的功能也较kubernetes更少一些。 - 是docker host集群管理工具 - docker官方提供的 - docker 1.12版本以后 - 用来统一集群管理的，把整个集群资源做统一调度 - 比kubernetes要轻量化 - 实现scaling 规模扩大或缩小 - 实现rolling update 滚动更新或版本回退 - 实现service discovery 服务发现 - 实现load balance 负载均衡 - 实现route mesh 路由网格，服务治理 # 二、docker swarm概念与架构 > 参考网址：https://docs.docker.com/swarm/overview/ ## 2.1 架构 ![](../../img/docker_swarm/docker swarm架构图.png) ![image-20220216093807929](../../img/docker_swarm/image-20220216093807929.png) ## 2.2 概念 **节点 (node):** 就是一台docker host上面运行了docker engine.节点分为两类: - 管理节点(manager node) 负责管理集群中的节点并向工作节点分配任务 - 工作节点(worker node) 接收管理节点分配的任务，运行任务 ~~~powershell # docker node ls ~~~ **服务(services):** 在工作节点运行的，由多个任务共同组成 ~~~powershell # docker service ls ~~~ **任务(task):** 运行在工作节点上容器或容器中包含应用，是集群中调度最小管理单元 ![image-20220216093706989](../../img/docker_swarm/image-20220216093706989.png) # 三、docker swarm集群部署 > 部署3主2从节点集群，另需提前准备1台本地容器镜像仓库服务器(Harbor) ## 3.1 容器镜像仓库 Harbor准备 ![image-20220216111914692](../../img/docker_swarm/image-20220216111914692.png) ## 3.2 主机准备 ### 3.2.1 主机名 ~~~powershell # hostnamectl set-hostname xxx ~~~ ~~~powershell 说明： sm1 管理节点1 sm2 管理节点2 sm3 管理节点3 sw1 工作节点1 sw2 工作节点2 ~~~ ### 3.2.2 IP地址 ~~~powershell 编辑网卡配置文件 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" 修改为静态 DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" 添加如下内容： IPADDR="192.168.10.xxx" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell 说明： sm1 管理节点1 192.168.10.10 sm2 管理节点2 192.168.10.11 sm3 管理节点3 192.168.10.12 sw1 工作节点1 192.168.10.13 sw2 工作节点2 192.168.10.14 ~~~ ### 3.2.3 主机名与IP地址解析 ~~~powershell 编辑主机/etc/hosts文件，添加主机名解析 # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.10 sm1 192.168.10.11 sm2 192.168.10.12 sm3 192.168.10.13 sw1 192.168.10.14 sw2 ~~~ ### 3.3.4 主机时间同步 ~~~powershell 添加计划任务，实现时间同步，同步服务器为time1.aliyun.com # crontab -e no crontab for root - using an empty one crontab: installing new crontab 查看添加后计划任务 # crontab -l 0 */1 * * * ntpdate time1.aliyun.com ~~~ ### 3.2.5 主机安全设置 ~~~powershell 关闭防火墙并查看其运行状态 # systemctl stop firewalld;systemctl disable firewalld # firewall-cmd --state not running ~~~ ~~~powershell 使用非交互式修改selinux配置文件 # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config 重启所有的主机系统 # reboot 重启后验证selinux是否关闭 # sestatus SELinux status: disabled ~~~ ## 3.3 docker安装 ### 3.3.1 docker安装 ~~~powershell 下载YUM源 # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell 安装docker-ce # yum -y install docker-ce ~~~ ~~~powershell 启动docker服务并设置为开机自启动 # systemctl enable docker;systemctl start docker ~~~ ### 3.3.2 配置docker daemon使用harbor ~~~powershell 添加daemon.json文件，配置docker daemon使用harbor # vim /etc/docker/daemon.json # cat /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.15"] } ~~~ ~~~powershell 重启docker服务 # ystemctl restart docker ~~~ ~~~powershell 深度登录harbor # docker login 192.168.10.15 Username: admin Password: 12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ## 3.4 docker swarm集群初始化 ### 3.4.1 获取docker swarm命令帮助 ~~~powershell 获取docker swarm命令使用帮助 # docker swarm --help Usage: docker swarm COMMAND Manage Swarm Commands: ca Display and rotate the root CA init Initialize a swarm 初始化 join Join a swarm as a node and/or manager 加入集群 join-token Manage join tokens 集群加入时token管理 leave Leave the swarm 离开集群 unlock Unlock swarm unlock-key Manage the unlock key update Update the swarm 更新集群 ~~~ ### 3.4.2 在管理节点初始化 > 本次在sm1上初始化 ~~~powershell 初始化集群 # docker swarm init --advertise-addr 192.168.10.10 --listen-addr 192.168.10.10:2377 Swarm initialized: current node (j42cwubrr70pwxdpmesn1cuo6) is now a manager. To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-6pddlyiq5f1i35w8d7q4bl1co 192.168.10.10:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. ~~~ ~~~powershell 说明： --advertise-addr 当主机有多块网卡时使用其选择其中一块用于广播,用于其它节点连接管理节点使用 --listen-addr 监听地址，用于承载集群流量使用 ~~~ ### 3.4.3 添加工作节点到集群 ~~~powershell 使用初始化过程中生成的token加入集群 [root@sw1 ~]# docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-6pddlyiq5f1i35w8d7q4bl1co 192.168.10.10:2377 This node joined a swarm as a worker. ~~~ ~~~powershell 查看已加入的集群 # docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION j42cwubrr70pwxdpmesn1cuo6 * sm1 Ready Active Leader 20.10.12 4yb34kuma6i9g5hf30vkxm9yc sw1 Ready Active 20.10.12 ~~~ > 如果使用的token已过期，可以再次生成新的加入集群的方法，如下命令所示。 ~~~powershell 重新生成用于添加工作点的token [root@sm1 ~]# docker swarm join-token worker To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-6pddlyiq5f1i35w8d7q4bl1co 192.168.10.10:2377 ~~~ ~~~powershell 加入至集群 [root@sw2 ~]# docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-6pddlyiq5f1i35w8d7q4bl1co 192.168.10.10:2377 This node joined a swarm as a worker. ~~~ ~~~powershell 查看node状态 # docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION j42cwubrr70pwxdpmesn1cuo6 * sm1 Ready Active Leader 20.10.12 4yb34kuma6i9g5hf30vkxm9yc sw1 Ready Active 20.10.12 mekitdu1xbpcttgupwuoiwg91 sw2 Ready Active 20.10.12 ~~~ ### 3.4.4 添加管理节点到集群 ~~~powershell 生成用于添加管理节点加入集群所使用的token [root@sm1 ~]# docker swarm join-token manager To add a manager to this swarm, run the following command: docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-7g85apo82mwz8ttmgdr7onfhu 192.168.10.10:2377 ~~~ ~~~powershell 加入集群 [root@sm2 ~]# docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-7g85apo82mwz8ttmgdr7onfhu 192.168.10.10:2377 This node joined a swarm as a manager. ~~~ ~~~powershell 加入集群 [root@sm3 ~]# docker swarm join --token SWMTKN-1-297iry1n2jeh30oopsjecvsco1uuvl15t2jz6jxabdpf0xkry4-7g85apo82mwz8ttmgdr7onfhu 192.168.10.10:2377 This node joined a swarm as a manager. ~~~ ~~~powershell 查看节点状态 # docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION j42cwubrr70pwxdpmesn1cuo6 * sm1 Ready Active Leader 20.10.12 nzpmehm8n87b9a17or2el10lc sm2 Ready Active Reachable 20.10.12 xc2x9z1b33rwdfxc5sdpobf0i sm3 Ready Active Reachable 20.10.12 4yb34kuma6i9g5hf30vkxm9yc sw1 Ready Active 20.10.12 mekitdu1xbpcttgupwuoiwg91 sw2 Ready Active 20.10.12 ~~~ ### 3.4.5 模拟管理节点出现故障 #### 3.4.5.1 停止docker服务并查看结果 ~~~powershell 停止docker服务 [root@sm1 ~]# systemctl stop docker ~~~ ~~~powershell 查看node状态，发现sm1不可达，状态为未知，并重启选择出leader [root@sm2 ~]# docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION j42cwubrr70pwxdpmesn1cuo6 sm1 Unknown Active Unreachable 20.10.12 nzpmehm8n87b9a17or2el10lc * sm2 Ready Active Leader 20.10.12 xc2x9z1b33rwdfxc5sdpobf0i sm3 Ready Active Reachable 20.10.12 4yb34kuma6i9g5hf30vkxm9yc sw1 Ready Active 20.10.12 mekitdu1xbpcttgupwuoiwg91 sw2 Ready Active 20.10.12 ~~~ #### 3.4.5.2 启动docker服务并查看结果 ~~~powershell 再次重动docker [root@sm1 ~]# systemctl start docker ~~~ ~~~powershell 观察可以得知sm1是可达状态，但并不是Leader [root@sm1 ~]# docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION j42cwubrr70pwxdpmesn1cuo6 * sm1 Ready Active Reachable 20.10.12 nzpmehm8n87b9a17or2el10lc sm2 Ready Active Leader 20.10.12 xc2x9z1b33rwdfxc5sdpobf0i sm3 Ready Active Reachable 20.10.12 4yb34kuma6i9g5hf30vkxm9yc sw1 Ready Active 20.10.12 mekitdu1xbpcttgupwuoiwg91 sw2 Ready Active 20.10.12 ~~~ # 四、docker swarm集群应用 ## 4.1 容器镜像准备 > 准备多个版本的容器镜像，以便于后期使用测试。 ### 4.1.1 v1版本 ~~~powershell 生成网站文件v1版 [root@harbor nginximg]# vim index.html [root@harbor nginximg]# cat index.html v1 ~~~ ~~~powershell 编写Dockerfile文件，用于构建容器镜像 [root@harbor nginximg]# vim Dockerfile [root@harbor nginximg]# cat Dockerfile FROM nginx:latest MAINTAINER 'tom' ADD index.html /usr/share/nginx/html RUN echo "daemon off;" >> /etc/nginx/nginx.conf EXPOSE 80 CMD /usr/sbin/nginx ~~~ ~~~powershell 使用docker build构建容器镜像 [root@harbor nginximg]# docker build -t 192.168.10.15/library/nginx:v1 . ~~~ ~~~powershell 登录harbor # docker login 192.168.10.15 Username: admin Password: 12345 ~~~ ~~~powershell 推送容器镜像至harbor # docker push 192.168.10.15/library/nginx:v1 ~~~ ### 4.1.2 v2版本 ~~~powershell 生成网站文件v2版 [root@harbor nginximg]# vim index.html [root@harbor nginximg]# cat index.html v2 ~~~ ~~~powershell 编写Dockerfile文件，用于构建容器镜像 [root@harbor nginximg]# vim Dockerfile [root@harbor nginximg]# cat Dockerfile FROM nginx:latest MAINTAINER 'tom' ADD index.html /usr/share/nginx/html RUN echo "daemon off;" >> /etc/nginx/nginx.conf EXPOSE 80 CMD /usr/sbin/nginx ~~~ ~~~powershell 使用docker build构建容器镜像 [root@harbor nginximg]# docker build -t 192.168.10.15/library/nginx:v2 . ~~~ ~~~powershell 推送镜像至Harbor [root@harbor nginximg]# docker push 192.168.10.15/library/nginx:v2 ~~~ ![image-20220216163257339](../../img/docker_swarm/image-20220216163257339.png) ## 4.2 发布服务 在docker swarm中,对外暴露的是服务（service)，而不是容器。 为了保持高可用架构，它准许同时启动多个容器共同支撑一个服务，如果一个容器挂了，它会自动使用另一个容器 ### 4.2.1 使用`docker service ls`查看服务 >在管理节点（manager node）上操作 ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS ~~~ ### 4.2.2 发布服务 ~~~powershell [root@sm1 ~]# docker service create --name nginx-svc-1 --replicas 1 --publish 80:80 192.168.10.15/library/nginx:v1 ucif0ibkjqrd7meal6vqwnduz overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged ~~~ ~~~powershell 说明 * 创建一个服务,名为nginx_svc-1 * replicas 1指定1个副本 * --publish 80:80 将服务内部的80端口发布到外部网络的80端口 * 使用的镜像为`192.168.10.15/library/nginx:v1` ~~~ ### 4.2.3 查看已发布服务 ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS ucif0ibkjqrd nginx-svc-1 replicated 1/1 192.168.10.15/library/nginx:v1 *:80->80/tcp ~~~ ### 4.2.4 查看已发布服务容器 ~~~powershell [root@sm1 ~]# docker service ps nginx-svc-1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 47t0s0egf6xf nginx-svc-1.1 192.168.10.15/library/nginx:v1 sw1 Running Running 48 minutes ago ~~~ ~~~powershell [root@sw1 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1bdf8981f511 192.168.10.15/library/nginx:v1 "/docker-entrypoint.…" 53 minutes ago Up 53 minutes 80/tcp nginx-svc-1.1.47t0s0egf6xf1n8m0c0jez3q0 ~~~ ### 4.2.5 访问已发布的服务 ~~~powershell [root@sm1 ~]# curl http://192.168.10.10 v1 [root@sm1 ~]# curl http://192.168.10.11 v1 [root@sm1 ~]# curl http://192.168.10.12 v1 [root@sm1 ~]# curl http://192.168.10.13 v1 [root@sm1 ~]# curl http://192.168.10.14 v1 ~~~ **在集群之外的主机访问** ![image-20220216174612048](../../img/docker_swarm/image-20220216174612048.png) ## 4.3 服务扩展 使用scale指定副本数来扩展 ~~~powershell [root@sm1 ~]# docker service scale nginx-svc-1=2 nginx-svc-1 scaled to 2 overall progress: 2 out of 2 tasks 1/2: running [==================================================>] 2/2: running [==================================================>] verify: Service converged ~~~ ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS ucif0ibkjqrd nginx-svc-1 replicated 2/2 192.168.10.15/library/nginx:v1 *:80->80/tcp ~~~ ~~~powershell [root@sm1 ~]# docker service ps nginx-svc-1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 47t0s0egf6xf nginx-svc-1.1 192.168.10.15/library/nginx:v1 sw1 Running Running about an hour ago oy16nuh5udn0 nginx-svc-1.2 192.168.10.15/library/nginx:v1 sw2 Running Running 57 seconds ago ~~~ ~~~powershell [root@sw1 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1bdf8981f511 192.168.10.15/library/nginx:v1 "/docker-entrypoint.…" About an hour ago Up About an hour 80/tcp nginx-svc-1.1.47t0s0egf6xf1n8m0c0jez3q0 ~~~ ~~~powershell [root@sw2 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0923c0d10223 192.168.10.15/library/nginx:v1 "/docker-entrypoint.…" About a minute ago Up About a minute 80/tcp nginx-svc-1.2.oy16nuh5udn0s1hda5bcpr9hd ~~~ **问题：现在仅扩展为2个副本，如果把服务扩展到3个副本，集群会如何分配主机呢？** ~~~powershell [root@sm1 ~]# docker service scale nginx-svc-1=3 nginx-svc-1 scaled to 3 overall progress: 3 out of 3 tasks 1/3: running [==================================================>] 2/3: running [==================================================>] 3/3: running [==================================================>] verify: Service converged ~~~ ~~~powershell [root@sm1 ~]# docker service ps nginx-svc-1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 47t0s0egf6xf nginx-svc-1.1 192.168.10.15/library/nginx:v1 sw1 Running Running about an hour ago oy16nuh5udn0 nginx-svc-1.2 192.168.10.15/library/nginx:v1 sw2 Running Running 12 minutes ago mn9fwxqbc9d1 nginx-svc-1.3 192.168.10.15/library/nginx:v1 sm1 Running Running 9 minutes ago ~~~ ~~~powershell 说明： 当把服务扩展到一定数量时，管理节点也会参与到负载运行中来。 ~~~ ## 4.4 服务裁减 ~~~powershell [root@sm1 ~]# docker service scale nginx-svc-1=2 nginx-svc-1 scaled to 2 overall progress: 2 out of 2 tasks 1/2: running [==================================================>] 2/2: running [==================================================>] verify: Service converged ~~~ ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS ucif0ibkjqrd nginx-svc-1 replicated 2/2 192.168.10.15/library/nginx:v1 *:80->80/tcp ~~~ ~~~powershell [root@sm1 ~]# docker service ps nginx-svc-1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 47t0s0egf6xf nginx-svc-1.1 192.168.10.15/library/nginx:v1 sw1 Running Running 2 hours ago oy16nuh5udn0 nginx-svc-1.2 192.168.10.15/library/nginx:v1 sw2 Running Running 29 minutes ago ~~~ ## 4.5 负载均衡 > 服务中包含多个容器时，每次访问将以轮询的方式访问到每个容器 ~~~powershell 修改sw1主机中容器网页文件 [root@sw1 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1bdf8981f511 192.168.10.15/library/nginx:v1 "/docker-entrypoint.…" About an hour ago Up About an hour 80/tcp nginx-svc-1.1.47t0s0egf6xf1n8m0c0jez3q0 [root@sw1 ~]# docker exec -it 1bdf bash root@1bdf8981f511:/# echo "sw1 web" > /usr/share/nginx/html/index.html root@1bdf8981f511:/# exit ~~~ ~~~powershell 修改sw2主机中容器网页文件 [root@sw2 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0923c0d10223 192.168.10.15/library/nginx:v1 "/docker-entrypoint.…" 42 minutes ago Up 42 minutes 80/tcp nginx-svc-1.2.oy16nuh5udn0s1hda5bcpr9hd [root@sw2 ~]# docker exec -it 0923 bash root@0923c0d10223:/# echo "sw2 web" > /usr/share/nginx/html/index.html root@0923c0d10223:/# exit ~~~ ~~~powershell [root@sm1 ~]# curl http://192.168.10.10 sw1 web [root@sm1 ~]# curl http://192.168.10.10 sw2 web [root@sm1 ~]# curl http://192.168.10.10 sw1 web [root@sm1 ~]# curl http://192.168.10.10 sw2 web ~~~ ## 4.6 删除服务 ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS ucif0ibkjqrd nginx-svc-1 replicated 2/2 192.168.10.15/library/nginx:v1 *:80->80/tcp ~~~ ~~~powershell [root@sm1 ~]# docker service rm nginx-svc-1 nginx-svc-1 ~~~ ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS ~~~ ## 4.7 服务版本更新 ~~~powershell [root@sm1 ~]# docker service create --name nginx-svc --replicas=1 --publish 80:80 192.168.10.15/library/nginx:v1 yz3wq6f1cgf10vtq5ne4qfwjz overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged ~~~ ~~~powershell [root@sm1 ~]# curl http://192.168.10.10 v1 ~~~ ~~~powershell [root@sm1 ~]# docker service update nginx-svc --image 192.168.10.15/library/nginx:v2 nginx-svc overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged ~~~ ~~~powershell [root@sm1 ~]# curl http://192.168.10.10 v2 ~~~ ## 4.8 服务版本回退 ~~~powershell [root@sm1 ~]# docker service update nginx-svc --image 192.168.10.15/library/nginx:v1 nginx-svc overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged ~~~ ## 4.9 服务版本滚动间隔更新 ~~~powershell # docker service create --name nginx-svc --replicas 60 --publish 80:80 192.168.10.15/library/nginx:v1 pqrt561dckg2wfpect3vf9ll0 overall progress: 60 out of 60 tasks verify: Service converged ~~~ ~~~powershell [root@sm1 ~]# docker service update --replicas 60 --image 192.168.10.15/library/nginx:v2 --update-parallelism 5 --update-delay 30s nginx-svc nginx-svc overall progress: 3 out of 3 tasks 1/3: running [==================================================>] 2/3: running [==================================================>] 3/3: running [==================================================>] verify: Service converged ~~~ ~~~powershell 说明 * --update-parallelism 5 指定并行更新数量 * --update-delay 30s 指定更新间隔时间 ~~~ ~~~powershell docker swarm滚动更新会造成节点上有exit状态的容器,可以考虑清除 命令如下： [root@sw1 ~]# docker container prune WARNING! This will remove all stopped containers. Are you sure you want to continue? [y/N] y ~~~ ## 4.10 副本控制器 > 副本控制器 ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS yz3wq6f1cgf1 nginx-svc replicated 3/3 192.168.10.15/library/nginx:v2 *:80->80/tcp ~~~ ~~~powershell [root@sm1 ~]# docker service ps nginx-svc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS x78l0santsbb nginx-svc.1 192.168.10.15/library/nginx:v2 sw2 Running Running 3 hours ago ura9isskfxku \_ nginx-svc.1 192.168.10.15/library/nginx:v1 sm1 Shutdown Shutdown 3 hours ago z738gvgazish \_ nginx-svc.1 192.168.10.15/library/nginx:v2 sw1 Shutdown Shutdown 3 hours ago 3qsrkkxn32bl \_ nginx-svc.1 192.168.10.15/library/nginx:v1 sm3 Shutdown Shutdown 3 hours ago psbi0mxu3amy nginx-svc.2 192.168.10.15/library/nginx:v2 sw1 Running Running 3 hours ago zpjw39bwhd78 nginx-svc.3 192.168.10.15/library/nginx:v2 sm1 Running Running 3 hours ago ~~~ ~~~~powershell [root@sm1 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 81fffd9132d8 192.168.10.15/library/nginx:v2 "/docker-entrypoint.…" 3 hours ago Up 3 hours 80/tcp nginx-svc.3.zpjw39bwhd78pw49svpy4q8zd [root@sm1 ~]# docker stop 81fffd9132d8;docker rm 81fffd9132d8 81fffd9132d8 81fffd9132d8 ~~~~ ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS yz3wq6f1cgf1 nginx-svc replicated 3/3 192.168.10.15/library/nginx:v2 *:80->80/tcp [root@sm1 ~]# docker service ps nginx-svc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS x78l0santsbb nginx-svc.1 192.168.10.15/library/nginx:v2 sw2 Running Running 3 hours ago ura9isskfxku \_ nginx-svc.1 192.168.10.15/library/nginx:v1 sm1 Shutdown Shutdown 3 hours ago z738gvgazish \_ nginx-svc.1 192.168.10.15/library/nginx:v2 sw1 Shutdown Shutdown 3 hours ago 3qsrkkxn32bl \_ nginx-svc.1 192.168.10.15/library/nginx:v1 sm3 Shutdown Shutdown 3 hours ago psbi0mxu3amy nginx-svc.2 192.168.10.15/library/nginx:v2 sw1 Running Running 3 hours ago qv6ya3crz1fj nginx-svc.3 192.168.10.15/library/nginx:v2 sm1 Running Running 13 seconds ago zpjw39bwhd78 \_ nginx-svc.3 192.168.10.15/library/nginx:v2 sm1 Shutdown Failed 19 seconds ago "task: non-zero exit (137)" ~~~ ## 4.11 在指定网络中发布服务 ~~~powershell [root@sm1 ~]# docker network create -d overlay tomcat-net mrkgccdfddy8zg92ja6fpox7p [root@sm1 ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 5ba369c13795 bridge bridge local 54568abb541a docker_gwbridge bridge local 4edcb5c4a324 host host local l6xmfxiiseqk ingress overlay swarm 5d06d748c9c7 none null local mrkgccdfddy8 tomcat-net overlay swarm [root@sm1 ~]# docker network inspect tomcat-net [ { "Name": "tomcat-net", "Id": "mrkgccdfddy8zg92ja6fpox7p", "Created": "2022-02-16T13:56:52.338589006Z", "Scope": "swarm", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "10.0.1.0/24", "Gateway": "10.0.1.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": null, "Options": { "com.docker.network.driver.overlay.vxlanid_list": "4097" }, "Labels": null } ] ~~~ ~~~powershell 说明： 创建名为tomcat-net的覆盖网络(Overlay Netowork)，这是个二层网络，处于该网络下的docker容器，即使宿主机不一样，也能相互访问 ~~~ ~~~powershell # docker service create --name tomcat \ --network tomcat-net \ -p 8080:8080 \ --replicas 2 \ tomcat:7.0.96-jdk8-openjdk ~~~ ~~~powershell 说明： 创建名为tomcat的服务，使用了刚才创建的覆盖网络 ~~~ ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS wgqkz8vymxkr tomcat replicated 2/2 tomcat:7.0.96-jdk8-openjdk *:8080->8080/tcp ~~~ ~~~powershell [root@sm1 ~]# docker service ps tomcat ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS fsx1fnssbmtg tomcat.1 tomcat:7.0.96-jdk8-openjdk sm3 Running Running 49 seconds ago gq0ogycj7orb tomcat.2 tomcat:7.0.96-jdk8-openjdk sm2 Running Running 58 seconds ago ~~~ ![image-20220216222436884](../../img/docker_swarm/image-20220216222436884.png) ## 4.12 服务网络模式 - 服务模式一共有两种：Ingress和Host，如果不指定，则默认的是Ingress； - Ingress模式下，到达Swarm任何节点的8080端口的流量，都会映射到任何服务副本的内部80端口，就算该节点上没有tomcat服务副本也会映射； ~~~powershell # docker service create --name tomcat \ --network tomcat-net \ -p 8080:8080 \ --replicas 2 \ tomcat:7.0.96-jdk8-openjdk ~~~ ~~~powershell [root@sm1 ~]# docker service ps tomcat ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS fsx1fnssbmtg tomcat.1 tomcat:7.0.96-jdk8-openjdk sm3 Running Running 8 minutes ago gq0ogycj7orb tomcat.2 tomcat:7.0.96-jdk8-openjdk sm2 Running Running 8 minutes ago ~~~ ~~~powershell [root@sm2 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f650498c8e71 tomcat:7.0.96-jdk8-openjdk "catalina.sh run" 9 minutes ago Up 9 minutes 8080/tcp tomcat.2.gq0ogycj7orbu4ua1dwk140as [root@sm2 ~]# docker inspect f650498c8e71 | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "10.0.0.24", ingress IP地址 "IPAddress": "10.0.1.9", 容器IP地址 ~~~ ~~~powershell [root@sm3 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9d0c412717d7 tomcat:7.0.96-jdk8-openjdk "catalina.sh run" 9 minutes ago Up 9 minutes 8080/tcp tomcat.1.fsx1fnssbmtgv3qh84fgqknlh [root@sm3 ~]# docker inspect 9d0c412717d7 | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "10.0.0.23", "IPAddress": "10.0.1.8", ~~~ ~~~powershell [root@sm1 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 [::]:8080 [::]:* users:(("dockerd",pid=2727,fd=54)) ~~~ ~~~powershell [root@sm2 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 [::]:8080 [::]:* users:(("dockerd",pid=1229,fd=26)) ~~~ ~~~powershell [root@sm3 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 [::]:8080 [::]:* users:(("dockerd",pid=1226,fd=22)) ~~~ ~~~powershell [root@sw1 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 [::]:8080 [::]:* users:(("dockerd",pid=1229,fd=39)) ~~~ ~~~powershell [root@sw2 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 [::]:8080 [::]:* users:(("dockerd",pid=1229,fd=22)) ~~~ - Host模式下，仅在运行有容器副本的机器上开放端口，使用Host模式的命令如下： ~~~powershell # docker service create --name tomcat \ --network tomcat-net \ --publish published=8080,target=8080,mode=host \ --replicas 3 \ tomcat:7.0.96-jdk8-openjdk ~~~ ~~~powershell [root@sm1 ~]# docker service ps tomcat ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS x6022h0oungs tomcat.1 tomcat:7.0.96-jdk8-openjdk sw1 Running Running 19 seconds ago *:8080->8080/tcp,*:8080->8080/tcp jmnthwqi6ubf tomcat.2 tomcat:7.0.96-jdk8-openjdk sm1 Running Running 18 seconds ago *:8080->8080/tcp,*:8080->8080/tcp nvcbijnfy2es tomcat.3 tomcat:7.0.96-jdk8-openjdk sw2 Running Running 19 seconds ago *:8080->8080/tcp,*:8080->8080/tcp ~~~ ~~~powershell [root@sm1 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 *:8080 *:* users:(("docker-proxy",pid=20963,fd=4)) tcp LISTEN 0 128 [::]:8080 [::]:* users:(("docker-proxy",pid=20967,fd=4)) ~~~ ~~~powershell [root@sw1 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 *:8080 *:* users:(("docker-proxy",pid=20459,fd=4)) tcp LISTEN 0 128 [::]:8080 [::]:* users:(("docker-proxy",pid=20463,fd=4)) ~~~ ~~~powershell [root@sw2 ~]# ss -anput | grep ":8080" tcp LISTEN 0 128 *:8080 *:* users:(("docker-proxy",pid=19938,fd=4)) tcp LISTEN 0 128 [::]:8080 [::]:* users:(("docker-proxy",pid=19942,fd=4)) ~~~ ~~~powershell 没有被映射端口 [root@sm2 ~]# ss -anput | grep ":8080" [root@sm3 ~]# ss -anput | grep ":8080" ~~~ ## 4.13 服务数据持久化存储 ### 4.13.1 本地存储 #### 4.13.1.1 在集群所有主机上创建本地目录 ~~~powershell # mkdir -p /data/nginxdata ~~~ #### 4.13.1.2 发布服务时挂载本地目录到容器中 ~~~powershell [root@sm1 ~]# docker service create --name nginx-svc --replicas 3 --mount "type=bind,source=/data/nginxdata,target=/usr/share/nginx/html" --publish 80:80 192.168.10.15/library/nginx:v1 s31z75rniv4p53ycbqch3xbqm overall progress: 3 out of 3 tasks 1/3: running [==================================================>] 2/3: running [==================================================>] 3/3: running [==================================================>] verify: Service converged ~~~ #### 4.13.1.3 验证是否使用本地目录 ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS s31z75rniv4p nginx-svc replicated 3/3 192.168.10.15/library/nginx:v1 *:80->80/tcp ~~~ ~~~powershell [root@sm1 ~]# docker service ps nginx-svc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS vgfhk4lksbtp nginx-svc.1 192.168.10.15/library/nginx:v1 sm2 Running Running 54 seconds ago v2bs9araxeuc nginx-svc.2 192.168.10.15/library/nginx:v1 sw2 Running Running 59 seconds ago 1m7fobr3cscz nginx-svc.3 192.168.10.15/library/nginx:v1 sm3 Running Running 59 seconds ago ~~~ ~~~powershell [root@sm2 ~]# ls /data/nginxdata/ [root@sm2 ~]# echo "sm2 web" > /data/nginxdata/index.html ~~~ ~~~powershell [root@sm3 ~]# ls /data/nginxdata/ [root@sm3 ~]# echo "sm3 web" > /data/nginxdata/index.html ~~~ ~~~powershell [root@sw2 ~]# ls /data/nginxdata [root@sw2 ~]# echo "sw2 web" > /data/nginxdata/index.html ~~~ ~~~powershell [root@sm1 ~]# curl http://192.168.10.10 sm2 web [root@sm1 ~]# curl http://192.168.10.10 sm3 web [root@sm1 ~]# curl http://192.168.10.10 sw2 web ~~~ > 存在数据一致性问题 ### 4.13.2 网络存储 * 网络存储卷可以实现跨docker宿主机的数据共享,数据持久保存到网络存储卷中 * 在创建service时添加卷的挂载参数,网络存储卷可以帮助自动挂载(**但需要集群节点都创建该网络存储卷**) #### 4.13.2.1 部署NFS存储 > 本案例以NFS提供远程存储为例 > 在192.168.10.15服务器上部署NFS服务，共享目录为docker swarm集群主机使用。 ~~~powershell [root@harbor ~]# mkdir /opt/dockervolume ~~~ ~~~powershell [root@harbor ~]# yum -y install nfs-utils ~~~ ~~~powershell [root@harbor ~]# vim /etc/exports [root@harbor ~]# cat /etc/exports /opt/dockervolume *(rw,sync,no_root_squash) ~~~ ~~~powershell [root@harbor ~]# systemctl enable nfs-server [root@harbor ~]# systemctl start nfs-server ~~~ ~~~powershell [root@harbor ~]# showmount -e Export list for harbor: /opt/dockervolume * ~~~ #### 4.13.2.2 为集群所有主机安装nfs-utils软件 ~~~powershell # yum -y install nfs-utils ~~~ ~~~powershell # showmount -e 192.168.10.15 Export list for 192.168.10.15: /opt/dockervolume * ~~~ #### 4.13.2.3 创建存储卷 > 集群中所有节点 ~~~powershell # docker volume create --driver local --opt type=nfs --opt o=addr=192.168.10.15,rw --opt device=:/opt/dockervolume nginx_volume nginx_volume ~~~ ~~~powershell # docker volume ls DRIVER VOLUME NAME local nginx_volume ~~~ ~~~powershell # docker volume inspect nginx_volume [ { "CreatedAt": "2022-02-16T23:29:11+08:00", "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/nginx_volume/_data", "Name": "nginx_volume", "Options": { "device": ":/opt/dockervolume", "o": "addr=192.168.10.15,rw", "type": "nfs" }, "Scope": "local" } ] ~~~ #### 4.13.2.4 发布服务 ~~~powershell [root@sm1 ~]# docker service create --name nginx-svc --replicas 3 --publish 80:80 --mount "type=volume,source=nginx_volume,target=/usr/share/nginx/html" 192.168.10.15/library/nginx:v1 uh6k84b87n8vciuirln4zqb4v overall progress: 3 out of 3 tasks 1/3: running [==================================================>] 2/3: running [==================================================>] 3/3: running [==================================================>] verify: Service converged ~~~ #### 4.13.2.5 验证 ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS uh6k84b87n8v nginx-svc replicated 3/3 192.168.10.15/library/nginx:v1 *:80->80/tcp [root@sm1 ~]# docker service ps nginx-svc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS k2vxpav5oadf nginx-svc.1 192.168.10.15/library/nginx:v1 sw2 Running Running 43 seconds ago v8fh0r89wt5i nginx-svc.2 192.168.10.15/library/nginx:v1 sw1 Running Running 43 seconds ago xb0nyft8ou4d nginx-svc.3 192.168.10.15/library/nginx:v1 sm1 Running Running 43 seconds ago ~~~ ~~~powershell [root@sm1 ~]# df -Th | grep nfs :/opt/dockervolume nfs 50G 8.9G 42G 18% /var/lib/docker/volumes/nginx_volume/_data ~~~ ~~~powershell [root@sw1 ~]# df -Th | grep nfs :/opt/dockervolume nfs 50G 8.9G 42G 18% /var/lib/docker/volumes/nginx_volume/_data ~~~ ~~~powershell [root@sw2 ~]# df -Th | grep nfs :/opt/dockervolume nfs 50G 8.9G 42G 18% /var/lib/docker/volumes/nginx_volume/_data ~~~ ~~~powershell [root@harbor ~]# echo "nfs test" > /opt/dockervolume/index.html ~~~ ~~~powershell [root@sm1 ~]# curl http://192.168.10.10 nfs test [root@sm1 ~]# curl http://192.168.10.11 nfs test [root@sm1 ~]# curl http://192.168.10.12 nfs test [root@sm1 ~]# curl http://192.168.10.13 nfs test [root@sm1 ~]# curl http://192.168.10.14 nfs test ~~~ ## 4.14 服务互联与服务发现 如果一个nginx服务与一个mysql服务之间需要连接,在docker swarm如何实现呢? **方法1:** 把mysql服务也使用 `--publish`参数发布到外网,但这样做的缺点是:mysql这种服务发布到外网不安全 **方法2:** 将mysql服务等运行在内部网络,只需要nginx服务能够连接mysql就可以了,在docker swarm中可以使用==**overlay**==网络来实现。 但现在还有个问题,服务副本数发生变化时,容器内部的IP发生变化时,我们希望仍然能够访问到这个服务, 这就是**==服务发现（service discovery)==**. **通过服务发现, service的使用者都不需要知道service运行在哪里,IP是多少,有多少个副本,就能让service通信** 下面使用`docker network ls`查看到的ingress网络就是一个overlay类型的网络,但它不支持服务发现 ~~~powershell [root@sm1 ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 5ba369c13795 bridge bridge local 54568abb541a docker_gwbridge bridge local 4edcb5c4a324 host host local l6xmfxiiseqk ingress overlay swarm 此处 5d06d748c9c7 none null local mrkgccdfddy8 tomcat-net overlay swarm ~~~ 我们**需要自建一个overlay网络来实现服务发现, 需要相互通信的service也必须属于同一个overlay网络** ~~~powershell [root@sm1 ~]# docker network create --driver overlay --subnet 192.168.100.0/24 self-network ejpf8zzig5rjsgefqucopcsdt ~~~ 说明: * --driver overlay指定为overlay类型 * --subnet 分配网段 * self-network 为自定义的网络名称 ~~~powershell [root@sm1 ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 5ba369c13795 bridge bridge local 54568abb541a docker_gwbridge bridge local 4edcb5c4a324 host host local l6xmfxiiseqk ingress overlay swarm 5d06d748c9c7 none null local ejpf8zzig5rj self-network overlay swarm 此处 mrkgccdfddy8 tomcat-net overlay swarm ~~~ **验证自动发现** 1, 发布nignx-svc服务,指定在自建的overlay网络 ~~~powershell [root@sm1 ~]# docker service create --name nginx-svc --replicas 3 --network self-network --publish 80:80 192.168.10.15/library/nginx:v1 rr21tvm1xpi6vk3ic83tfy9e5 overall progress: 3 out of 3 tasks 1/3: running [==================================================>] 2/3: running [==================================================>] 3/3: running [==================================================>] verify: Service converged ~~~ 2, 发布一个busybox服务,也指定在自建的overlay网络 ~~~powershell [root@sm1 ~]# docker service create --name test --network self-network busybox sleep 100000 w14lzhhzdyqwt18lrby4euw98 overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged ~~~ 说明: * 服务名为test * busybox是一个集成了linux常用命令的软件,这里使用它可以比较方便的测试与nginx_service的连通性 * 没有指定副本,默认1个副本 * 因为它并不是长时间运行的daemon守护进程,所以运行一下就会退出.sleep 100000是指定一个长的运行时间,让它有足够的时间给我们测试 3, 查出test服务在哪个节点运行的容器 ~~~powershell [root@sm1 ~]# docker service ps test ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS x8nkifpdtyw5 test.1 busybox:latest sm2 Running Running about a minute ago ~~~ 4, 去运行test服务的容器节点查找容器的名称 ~~~powershell [root@sm2 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8df13819bd5c busybox:latest "sleep 100000" About a minute ago Up About a minute test.1.x8nkifpdtyw5177zhr0r1lxad ~~~ 5, 使用查找出来的容器名称,执行命令测试 ~~~powershell [root@sm2 ~]# docker exec -it test.1.x8nkifpdtyw5177zhr0r1lxad ping -c 2 nginx-svc PING nginx-svc (192.168.100.2): 56 data bytes 64 bytes from 192.168.100.2: seq=0 ttl=64 time=0.093 ms 64 bytes from 192.168.100.2: seq=1 ttl=64 time=0.162 ms --- nginx-svc ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.093/0.127/0.162 ms ~~~ **测试的结果为: test服务可以ping通nginx_service服务,并且返回的IP为自建网络的一个IP(192.168.100.2)** ~~~powershell [root@sm1 ~]# docker service inspect nginx-svc [ ...... "VirtualIPs": [ { "NetworkID": "l6xmfxiiseqkl57wnsm4cykps", "Addr": "10.0.0.36/24" }, { "NetworkID": "ejpf8zzig5rjsgefqucopcsdt", "Addr": "192.168.100.2/24" 与此处IP地址保持一致。 } ] } } ] ~~~ 6, 分别去各个节点查找nginx_service服务的各个容器(3个副本),发现它们的IP与上面ping的IP都不同 ~~~powershell [root@sm1 ~]# docker inspect nginx-svc.1.6nxixaw3tn2ld3vklfjldnpl5 | grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "10.0.0.37", "IPAddress": "192.168.100.3", ~~~ ~~~powershell [root@sw1 ~]# docker inspect nginx-svc.3.steywkaxfboynglx4bsji6jd1 | grep -i ipaddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "10.0.0.39", "IPAddress": "192.168.100.5", ~~~ ~~~powershell [root@sw2 ~]# docker inspect nginx-svc.2.rz1iifb9eg0tos7r59cbesucd | grep -i ipaddress "SecondaryIPAddresses": null, "IPAddress": "", "IPAddress": "10.0.0.38", "IPAddress": "192.168.100.4", ~~~ 7, 后续测试, 将nginx_service服务扩展,裁减,更新,回退.都不影响test服务访问nginx-svc。 **结论: 在自建的overlay网络内,通过服务发现可以实现服务之间通过服务名(不用知道对方的IP)互联,而且不会受服务内副本个数和容器内IP变化等的影响。** ## 4.15 docker swarm网络 在 Swarm Service 中有三个重要的网络概念： - **Overlay networks** 管理 Swarm 中 Docker 守护进程间的通信。你可以将服务附加到一个或多个已存在的 `overlay` 网络上，使得服务与服务之间能够通信。 - **ingress network** 是一个特殊的 `overlay` 网络，用于服务节点间的负载均衡。当任何 Swarm 节点在发布的端口上接收到请求时，它将该请求交给一个名为 `IPVS` 的模块。`IPVS` 跟踪参与该服务的所有IP地址，选择其中的一个，并通过 `ingress` 网络将请求路由到它。 初始化或加入 Swarm 集群时会自动创建 `ingress` 网络，大多数情况下，用户不需要自定义配置，但是 docker 17.05 和更高版本允许你自定义。 - **docker_gwbridge**是一种桥接网络，将 `overlay` 网络（包括 `ingress` 网络）连接到一个单独的 Docker 守护进程的物理网络。默认情况下，服务正在运行的每个容器都连接到本地 Docker 守护进程主机的 `docker_gwbridge` 网络。 `docker_gwbridge` 网络在初始化或加入 Swarm 时自动创建。大多数情况下，用户不需要自定义配置，但是 Docker 允许自定义。 | 名称 | 类型 | 注释 | | --------------- | ------- | ---- | | docker_gwbridge | bridge | none | | ingress | overlay | none | | custom-network | overlay | none | - docker_gwbridge和ingress是swarm自动创建的，当用户执行了docker swarm init/connect之后。 - docker_gwbridge是bridge类型的负责本机container和主机直接的连接 - ingress负责service在多个主机container之间的路由。 - custom-network是用户自己创建的overlay网络，通常我们都需要创建自己的network并把service挂在上面。 ![service ingress image](../../img/docker_swarm/ingress-routing-mesh.png) # 五、docker stack ## 5.1 docker stack介绍 早期使用service发布，每次只能发布一个service。 yaml可以发布多个服务，但是使用docker-compose只能在一台主机发布。 一个stack就是一组有关联的服务的组合，可以一起编排，一起发布, 一起管理 ## 5.2 docker stack与docker compose区别 - Docker stack会忽略了“构建”指令，无法使用stack命令构建新镜像，它是需要镜像是预先已经构建好的。 所以docker-compose更适合于开发场景； - Docker Compose是一个Python项目，在内部，它使用Docker API规范来操作容器。所以需要安装Docker -compose，以便与Docker一起在您的计算机上使用； - Docker Stack功能包含在Docker引擎中。你不需要安装额外的包来使用它，docker stacks 只是swarm mode的一部分。 - Docker stack不支持基于第2版写的docker-compose.yml ，也就是version版本至少为3。然而Docker Compose对版本为2和3的 文件仍然可以处理； - docker stack把docker compose的所有工作都做完了，因此docker stack将占主导地位。同时，对于大多数用户来说，切换到使用docker stack既不困难，也不需要太多的开销。如果您是Docker新手，或正在选择用于新项目的技术，请使用docker stack。 ## 5.3 docker stack常用命令 | 命令 | 描述 | | --------------------- | -------------------------- | | docker stack deploy | 部署新的堆栈或更新现有堆栈 | | docker stack ls | 列出现有堆栈 | | docker stack ps | 列出堆栈中的任务 | | docker stack rm | 删除一个或多个堆栈 | | docker stack services | 列出堆栈中的服务 | ## 5.4 部署wordpress案例 1, 编写YAML文件 ~~~powershell [root@sm1 ~]# vim stack1.yaml [root@sm1 ~]# cat stack1.yaml version: '3' services: db: image: mysql:5.7 environment: MYSQL_ROOT_PASSWORD: somewordpress MYSQL_DATABASE: wordpress MYSQL_USER: wordpress MYSQL_PASSWORD: wordpress deploy: replicas: 1 wordpress: depends_on: - db image: wordpress:latest ports: - "8010:80" environment: WORDPRESS_DB_HOST: db:3306 WORDPRESS_DB_USER: wordpress WORDPRESS_DB_PASSWORD: wordpress WORDPRESS_DB_NAME: wordpress deploy: replicas: 1 placement: constraints: [node.role == manager] ~~~ 说明: * placement的constraints限制此容器在manager节点 2, 使用docker stack发布 ~~~powershell [root@sm1 ~]# docker stack deploy -c stack1.yaml stack1 Creating network stack1_default 创建自建的overlay网络 Creating service stack1_db 创建stack1_db服务 Creating service stack1_wordpress 创建stack1_wordpress服务 ~~~ **如果报错,使用`docker stack rm stack1`删除.排完错再启动** ~~~powershell [root@sm1 ~]# docker stack ls NAME SERVICES ORCHESTRATOR stack1 2 Swarm ~~~ ~~~powershell [root@sm1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS tw1a8rnde2yr stack1_db replicated 1/1 mysql:5.7 zf1h2r4m12li stack1_wordpress replicated 1/1 wordpress:latest *:8010->80/tcp ~~~ 3, 验证 ![image-20220217102033645](../../img/docker_swarm/image-20220217102033645.png) ## 5.5 部署nginx与web管理服务案例 1, 编写YAML文件 ~~~powershell [root@sm1 ~]# vim stack2.yaml [root@sm1 ~]# cat stack2.yaml version: "3" services: nginx: image: 192.168.10.15/library/nginx:v1 ports: - 80:80 deploy: mode: replicated replicas: 3 visualizer: image: dockersamples/visualizer ports: - "9001:8080" volumes: - "/var/run/docker.sock:/var/run/docker.sock" deploy: replicas: 1 placement: constraints: [node.role == manager] portainer: image: portainer/portainer ports: - "9000:9000" volumes: - "/var/run/docker.sock:/var/run/docker.sock" deploy: replicas: 1 placement: constraints: [node.role == manager] ~~~ 说明: stack中共有3个service * nginx服务,3个副本 * visualizer服务: 图形查看docker swarm集群 * portainer服务: 图形管理docker swarm集群 2,使用docker stack发布 ~~~powershell [root@sm1 ~]# docker stack deploy -c stack2.yaml stack2 Creating network stack2_default Creating service stack2_portainer Creating service stack2_nginx Creating service stack2_visualizer 如果报错,使用docker stack rm stack2删除.排完错再启动 ~~~ ~~~powershell [root@sm1 ~]# docker stack ps stack2 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS zpkt1780g2nr stack2_nginx.1 192.168.10.15/library/nginx:v1 sm2 Running Running 54 seconds ago 9iqdgw2fxk0s stack2_nginx.2 192.168.10.15/library/nginx:v1 sm3 Running Running 54 seconds ago 4h0ob7b4ho2a stack2_nginx.3 192.168.10.15/library/nginx:v1 sw2 Running Running 54 seconds ago jpp7h6qheh4j stack2_portainer.1 portainer/portainer:latest sm1 Running Running 21 seconds ago ty0mktx60typ stack2_visualizer.1 dockersamples/visualizer:latest sm1 Running Starting 22 seconds ago ~~~ 3,验证 ![image-20220217102606092](../../img/docker_swarm/image-20220217102606092.png) ![image-20220217102654303](../../img/docker_swarm/image-20220217102654303.png) ## 5.6 nginx+haproxy+nfs案例 1,在docker swarm管理节点上准备配置文件 ~~~powershell [root@sm1 ~]# mkdir -p /docker-stack/haproxy [root@sm1 ~]# cd /docker-stack/haproxy/ [root@sm1 haproxy]# vim haproxy.cfg global log 127.0.0.1 local0 log 127.0.0.1 local1 notice defaults log global mode http option httplog option dontlognull timeout connect 5000ms timeout client 50000ms timeout server 50000ms stats uri /status frontend balancer bind *:8080 mode http default_backend web_backends backend web_backends mode http option forwardfor balance roundrobin server web1 nginx1:80 check server web2 nginx2:80 check server web3 nginx3:80 check option httpchk GET / http-check expect status 200 ~~~ 2, 编写YAML编排文件 ~~~powershell [root@sm1 haproxy]# vim stack3.yml [root@sm1 haproxy]# vim stack3.yaml [root@sm1 haproxy]# cat stack3.yaml version: "3" services: nginx1: image: 192.168.10.15/library/nginx:v1 deploy: mode: replicated replicas: 1 restart_policy: condition: on-failure volumes: - "nginx_vol:/usr/share/nginx/html" nginx2: image: 192.168.10.15/library/nginx:v1 deploy: mode: replicated replicas: 1 restart_policy: condition: on-failure volumes: - "nginx_vol:/usr/share/nginx/html" nginx3: image: 192.168.10.15/library/nginx:v1 deploy: mode: replicated replicas: 1 restart_policy: condition: on-failure volumes: - "nginx_vol:/usr/share/nginx/html" haproxy: image: haproxy:latest volumes: - "./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro" ports: - "8080:8080" deploy: replicas: 1 placement: constraints: [node.role == manager] volumes: nginx_vol: driver: local driver_opts: type: "nfs" o: "addr=192.168.10.15,rw" device: ":/opt/dockervolume" ~~~ 3, 发布 ~~~powershell [root@sm1 haproxy]# docker stack deploy -c stack3.yml stack3 Creating network stack3_default Creating service stack3_nginx3 Creating service stack3_haproxy Creating service stack3_nginx1 Creating service stack3_nginx2 ~~~ 4, 验证 ![image-20220217120725646](../../img/docker_swarm/image-20220217120725646.png) ![image-20220217120746215](../../img/docker_swarm/image-20220217120746215.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_ack.md ================================================ # 阿里云容器服务ACK # 一、配置专有网络 ## 1.1进入阿里云VPC控制台 ![进入VPC控制台](../../img/kubernetes/kubernetes_ack/进入VPC控制台.png) ## 1.2开通阿里云VPC服务 ![vpc控制台](../../img/kubernetes/kubernetes_ack/vpc控制台.png) ![vpc开通成功](../../img/kubernetes/kubernetes_ack/vpc开通成功.png) ## 1.3创建vpc ![创建vpc](../../img/kubernetes/kubernetes_ack/创建vpc.png) ![配置vpc01](../../img/kubernetes/kubernetes_ack/配置vpc01.png) ![配置vpc02](../../img/kubernetes/kubernetes_ack/配置vpc02.png) ![image-20221012160149721](../../img/kubernetes/kubernetes_ack/image-20221012160149721.png) # 二、配置Kubernetes集群 ## 2.1进入阿里云ACK控制台 ![容器服务ack](../../img/kubernetes/kubernetes_ack/容器服务ack.png) ![进入ack控制台](../../img/kubernetes/kubernetes_ack/进入ack控制台.png) ## 2.2创建集群 ### 2.2.1进入集群配置界面 ![创建集群](../../img/kubernetes/kubernetes_ack/创建集群.png) ### 2.2.2选择集群配置 ![配置k8s01](../../img/kubernetes/kubernetes_ack/配置k8s01.png) ![配置k8s02](../../img/kubernetes/kubernetes_ack/配置k8s02.png) ![配置k8s03](../../img/kubernetes/kubernetes_ack/配置k8s03.png) ### 2.2.3选择节点池配置 ![配置nodepool01](../../img/kubernetes/kubernetes_ack/配置nodepool01.png) ![配置nodepool02](../../img/kubernetes/kubernetes_ack/配置nodepool02.png) ### 2.2.4选择组件配置 ![配置组件01](../../img/kubernetes/kubernetes_ack/配置组件01.png) ### 2.2.5确认配置 ![创建集群02](../../img/kubernetes/kubernetes_ack/创建集群02-16655626080871.png) ### 2.2.6创建集群 ![集群创建中](../../img/kubernetes/kubernetes_ack/集群创建中.png) ![集群列表](../../img/kubernetes/kubernetes_ack/集群列表.png) # 三、阿里云控制台测试集群可用性 ## 3.1创建Deployment ![demo01](../../img/kubernetes/kubernetes_ack/demo01.png) ![demo02](../../img/kubernetes/kubernetes_ack/demo02.png) ![demo03](../../img/kubernetes/kubernetes_ack/demo03.png) ## 3.2创建Service ![svcDemo01](../../img/kubernetes/kubernetes_ack/svcDemo01.png) ![svcDemo02](../../img/kubernetes/kubernetes_ack/svcDemo02-16655630626753.png) ![svcDemo03](../../img/kubernetes/kubernetes_ack/svcDemo03.png) ## 3.3创建Ingress ![ingressDemo01](../../img/kubernetes/kubernetes_ack/ingressDemo01.png) ![ingressDemo02](../../img/kubernetes/kubernetes_ack/ingressDemo02.png) ![访问01](../../img/kubernetes/kubernetes_ack/访问01.png) ## 3.4本地pc添加hosts(ingress域名未做备案) 同时按住win键+R键进入运行 ![修改hosts01](../../img/kubernetes/kubernetes_ack/修改hosts01.png) ![修改hosts02](../../img/kubernetes/kubernetes_ack/修改hosts02.png) ![修改hosts03](../../img/kubernetes/kubernetes_ack/修改hosts03.png) ![修改hosts04](../../img/kubernetes/kubernetes_ack/修改hosts04.png) ## 3.5访问应用 ![访问02](../../img/kubernetes/kubernetes_ack/访问02.png) # 四、通过api连接集群，并使用阿里云容器镜像仓库服务进行应用发布 ## 4.1 安装kubectl ~~~powershell [root@kubemsb ~]# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" [root@kubemsb ~]# chmod +x kubectl [root@kubemsb ~]# install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl [root@kubemsb ~]# kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.6-aliyun.1", GitCommit:"6965b276168c8f1e4a8545c52152838a2f433758", GitTreeState:"clean", BuildDate:"2022-09-22T08:06:05Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"} ~~~ ## 4.2 下载kubeConfig文件 ![连接k8s01](../../img/kubernetes/kubernetes_ack/连接k8s01.png) ![连接k8s02](../../img/kubernetes/kubernetes_ack/连接k8s02.png) ~~~powershell [root@kubemsb ~]# mkdir /root/.kube [root@kubemsb ~]# vim /root/.kube/config apiVersion: v1 clusters: - cluster: server: https://你的集群ip:6443 certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUakNDQWphZ0F3SUJBZ0lVWUJSR041Nm5lY3g4aDVLaVhCOUNnVFhUelJjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1BqRW5NQThHQTFVRUNoTUlhR0Z1WjNwb2IzVXdGQVlEVlFRS0V3MWhiR2xpWVdKaElHTnNiM1ZrTVJNdwpFUVlEVlFRREV3cHJkV0psY201bGRHVnpNQ0FYRFRJeU1UQXhNakEwTkRrd01Gb1lEekl3TlRJeE1EQTBNRFEwCk9UQXdXakErTVNjd0R3WURWUVFLRXdob1lXNW5lbWh2ZFRBVUJnTlZCQW9URFdGc2FXSmhZbUVnWTJ4dmRXUXgKRXpBUkJnTlZCQU1UQ210MVltVnlibVYwWlhNd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFSwpBb0lCQVFDbU82S0JQeFR2RGwwQnprdDI1K3hEMENwc3pSR0dpamllWnlWcnErUkxZd0JuN2duODNlRmtVYzJ3ClMzU091QWtjRCsvR3IrSExveTduZVVEbWJvVmZWUG82eEFDZ1A1dGgvVll4bU1GeVVIdytDQ3E1NmxPU1BGMysKQkJXMVRndnNzR25ZaTBhZXJXU211a05jS3lBMkRZTEVmRkw4R3FJRVZxWU1DTjNvcUJvUm5hZUEwSzViU2RFRApTV0JhN0ZtNmlXTlFwMldKUlhhS3Rad2thbzB0QU50NjU1ODBteTdPK3ZhV3E1b3owTXpPcS9jV211eFFvUnBGCmJhOUVSaU9qWFM0bDhsUzFGL010bjQ1bGFiZCtFMjh6R1h6alQzQytwU0FlWTdWLzZsd2tyN0RhdklMSkNSZ0EKMkpRdlZaV3ZRRTJQQWl6RXhMNDZKdHRzemtuYkFnTUJBQUdqUWpCQU1BNEdBMVVkRHdFQi93UUVBd0lDckRBUApCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUlhEL2hHbkFHd0hiZ0J5MXRHU1dGVENhY3NWREFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWZOeFRTbnpwSWFhbVROazBvZW5iQ0pqRyszTCtQMklzTnFPMzdBZkwKVmVlbWdQOWlVYlgxazhqMzFOTDMxU3VOMVFNcmlJeXpBR3hENUgvYnJvQnREWFJ3cFd6S1BQcmw3cnlNNHZsRgp2TkZpZXhFUlB5L1VZNFRnZHVIRlNtY2JETnVpV2xFRVpyd00zMDV2U2VvOGI0WDFtWWpGWHM1aTBDY3ZwOWRmCnZibEFFcFZBV2x3NDZjM0R4UTc2RFFjR1RhUENWTjdPSXBONkRySnc5OE5jRzBkK1B2QnREclZxcGY0TGpuVHgKbEdyMkxBUGN5TVoyMzBuMngvS1h3KzZvc3M0RXo5R2FKajBDdTNhWWRmekgrOGszajZPQWNMVGMvaXJwN3R3RQovekJtMmhqUEpVT085Z0JSbzMwWUxDYnpBeDRiejRTZnlsUlJ6QVJzcCsxOGxRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= name: kubernetes contexts: - context: cluster: kubernetes user: "kubernetes-admin" name: kubernetes-admin-c0aa8c47fbfab4b7ebf610d0214bfacd5 current-context: kubernetes-admin-c0aa8c47fbfab4b7ebf610d0214bfacd5 kind: Config preferences: {} users: - name: "kubernetes-admin" user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQvRENDQXVTZ0F3SUJBZ0lEZUJIRE1BMEdDU3FHU0liM0RRRUJDd1VBTUdveEtqQW9CZ05WQkFvVElXTXcKWVdFNFl6UTNabUptWVdJMFlqZGxZbVkyTVRCa01ESXhOR0ptWVdOa05URVFNQTRHQTFVRUN4TUhaR1ZtWVhWcwpkREVxTUNnR0ExVUVBeE1oWXpCaFlUaGpORGRtWW1aaFlqUmlOMlZpWmpZeE1HUXdNakUwWW1aaFkyUTFNQjRYCkRUSXlNVEF4TWpBME5EZ3dNRm9YRFRJMU1UQXhNVEEwTlRNMU9Wb3dTREVWTUJNR0ExVUVDaE1NYzNsemRHVnQKT25WelpYSnpNUWt3QndZRFZRUUxFd0F4SkRBaUJnTlZCQU1UR3pFNE9UazJPRGd4TXpJME5EQTROVEV0TVRZMgpOVFUxTURRek9UQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvRjFmQ0VZc1FVCmJtODJsYUM3dlNiUTl3WHdNcEJ1ZDQ2ek1PTGtVMU5GcEF3OERUSmw5RGoyZlRFeFZFeGkrT1VFaDBzN0dncysKcitMamlmVVhQT0pRTGxUUGZTdU1iaFdtNmloc3Y1dGlFakg0bXJzRGYwUlpCc2RqMG9oOUVlbDdZT0NTKytuZwpHZFZqNDlIbzBtWUhjRXQ1Z0g4dzQ2UEQwcjBRQi9vVE52SWJFejNMVGpZZEY1ZEYyZG9QcXVKSjFTUDNvODliCmlZR2k0Z2xhL1g2dzVzOWg2TmlNNWhqNGdudHRLeisyNTNsOU4zbEdYWm9xWEVUdW5qS2RYQzlFeThBbklwYW8KVUR4NXlwb0ZqbWc4dTdNWUVBTW9sM0E3c0o4MjFzWUlnMmpsb0FIVGtZZThFNGtDOGRBZnpVaWpuNlRFNVlqTwpMUXZFKzJ0b1RFVUNBd0VBQWFPQnpEQ0J5VEFPQmdOVkhROEJBZjhFQkFNQ0I0QXdFd1lEVlIwbEJBd3dDZ1lJCkt3WUJCUVVIQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JRYks4NTBMcE9aREtoYnpKelIKN0ZSbVAwclhZVEE4QmdnckJnRUZCUWNCQVFRd01DNHdMQVlJS3dZQkJRVUhNQUdHSUdoMGRIQTZMeTlqWlhKMApjeTVoWTNNdVlXeHBlWFZ1TG1OdmJTOXZZM053TURVR0ExVWRId1F1TUN3d0txQW9vQ2FHSkdoMGRIQTZMeTlqClpYSjBjeTVoWTNNdVlXeHBlWFZ1TG1OdmJTOXliMjkwTG1OeWJEQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUEKRW5jWXN1TXNTTGpHZkRMcmJibXg5enhIYmFQcjN5NVVqc3ZjbkFCbDh2Rlh1ZEIzdmtNMFRrWjl6bmZEMFoybAowY2tDRVNkM0FLamV1Zy9XamVaNnp0dWpQNyttMDJiaDU5K3A0V25JRTMxZm43eDVuNmdLVXd0d0hQRUpWdGU1CnptK0QzamxSTXRQWUp0dXpNelBHKzQ2YTdnbWRYVnB1NHdmY0JDNytOOUdRZlVSV28razBQeVU5QXM5YVVrUjcKd1M4RkFydDNQZTRrRkM3SS8wdmptRXVYOGVLMENFWW1KNGdPVWhRSk1OTmRDWEVGaGVFNDdETnhQR0RRTlMrcgp5TW5ibkJNWDJrbUd4WUdLWkxlU3RTbENjMFBXTU1QNWF4b2hwVmphQnRHenJwZG9wQkpmQ1BxSWtua1pnMlJFCm1oUGNjR0pNZk03OHorbUVnWTdQQlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBdjhYVjhJUml4QlJ1YnphVm9MdTlKdEQzQmZBeWtHNTNqck13NHVSVFUwV2tERHdOCk1tWDBPUFo5TVRGVVRHTDQ1UVNIU3pzYUN6NnY0dU9KOVJjODRsQXVWTTk5SzR4dUZhYnFLR3kvbTJJU01maWEKdXdOL1JGa0d4MlBTaUgwUjZYdGc0Skw3NmVBWjFXUGowZWpTWmdkd1MzbUFmekRqbzhQU3ZSQUgraE0yOGhzVApQY3RPTmgwWGwwWFoyZytxNGtuVkkvZWp6MXVKZ2FMaUNWcjlmckRtejJIbzJJem1HUGlDZTIwclA3Ym5lWDAzCmVVWmRtaXBjUk82ZU1wMWNMMFRMd0NjaWxxaFFQSG5LbWdXT2FEeTdzeGdRQXlpWGNEdXduemJXeGdpRGFPV2cKQWRPUmg3d1RpUUx4MEIvTlNLT2ZwTVRsaU00dEM4VDdhMmhNUlFJREFRQUJBb0lCQUVIQnpJSVVZdWxwT0JUYQpXcVFxdThrN2pmaHpUR05KL2FUb3ZMS1FLQ0k5bHVxN0RLdGpXa0xPQWsrRDRxdnVvL21qZStESCt6NHYySStzCkJmb01aY3dVUWs3Q2R1RWYzZjkrcEh3SURndHhyVnVGSzBsRkFiYWw4RFVlZldwZnVHeWU2YkVYdFZtVWhrY3QKWVVqdzFIUTN5TUdZZi84Yy9IR0REQmZZRDJRbjZhTDFWQXJIMU1RanM2QUhsdTBqNmlScmtiZStJQTJmaEhCWgo3SWxNMm1mMWFUQ2UxTFh5VlFLY2lab3FEdG9EaHlGVzlJeFNBRWgyWDJCeHN2aStjbXFFRnMrYVpsQjlOYmVrCmNiaWJNby9BbFBKL0Q5dlUvd2dvOG83eFJKVUVwN0JjNXptQkd1Vk9mOXQyL1FRTHRYMnF3Ym5NbTJ2L09vSmkKcmdDTkt3RUNnWUVBeGtFZ3crelFsdGxIM1dON3FqaTV0SjZoMU54K0FIbXlOVktkMXo4TWVQQlRmWklEYnJwZwpRckk1SGViR0JkRFR3R09xWEFaVjNGMkhNKzVzTmNtRVpMUXgxUFlwUjJKNmRhTDlQcHgyUVYrWG1jcGJFM2h2CnlQdFNxK01jS2dLQTFBWXUvYUpMRUFGYzhkRzZXQSsyVll4S2dHQkoxYWZVd2FkOHNqa01lZjBDZ1lFQTk2Rm4KeG0zcWx5V0NIVVVScGZZbWxHRjcwazhoNEhHbEhQVDhKWFRPUDJ1azhJSG5IcWVHMURHWkVXTjlObGFHNlE2OApwT21kVmpGYUVaRkhsaUgrM2EycjM5ODJZbE0zV2ptdHAwWU1OamRjb0VkNVB2SmhBcDVnanJjanFEbitCRVJRCkhJTlloTHVVTkhldkgybTcxanBMZFc0T3U4RDBtemcwYWZmKzZla0NnWUFqSEsvSDQyNUIrVE9BSXpsb3VXa3kKY3hIb1dEQzh0dEQ0SHF5enpZZlE5NGRoNnZWT3JCV1dXbWhCUW5qNTJMVllFbzYrZzZTN1JabEhla2VSOHlITwprQmR2UnpXaVhlK2dYQUNNRVR2a2NpQ3diU1ZhT205WjRpRVBMa2J6cmNQUlNWTVRPcVllckk4Z2lPdCs0ekdBCnp6cjJwNWNNRHhCZ1IrQ08xQW5BcFFLQmdRQzVidEI3TXd3Y3F5WElMVlNONXNuMUN4cmgzZ2dNUG8zYUpVbzcKOGtBUUZkZXBrbDAwQTFkZnh0Uk14U2VhYlNrUmxNdUc0cm5ETmFtd3hnM0pTeVBEZEsza013NVNpdVpFVmpUSApzaGlxOE1JWDgrM0RTVldEMUgyUjRXcCtLdzhndXRjU3d2aWlUS1VxdFFVYWpON3dDUzlBZkcza0F4WW90ZW5xCkFKK1BJUUtCZ0MxRGFJdS9wWnVNa0FJTUpUZzdKUFZrVVlpMmZ6RkxqSTFQYS9qdWo4NGJ0TCtvRi9BdmZPblkKK1AvTjhHUWJ5UnNqc1owc2JzTzZvWUZrUVd4dlhWY1Y5a3NVNks5ZFpRUkNoSktrM0gxUzFvQ2xnclk2YkViKwowNFpKZlM0bWR0QzVvVVkydVNBdzEyOTI4MUpXdG9XdE5QaHRUeWlDV2xVclRNbDZQZ3R3Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== ~~~ ## 4.3 查看集群信息 ~~~powershell [root@kubemsb ~]# kubectl get nodes --kubeconfig /root/.kube/config NAME STATUS ROLES AGE VERSION cn-chengdu.192.168.10.43 Ready 3h37m v1.24.6-aliyun.1 cn-chengdu.192.168.10.44 Ready 3h37m v1.24.6-aliyun.1 cn-chengdu.192.168.10.45 Ready 3h37m v1.24.6-aliyun.1 ~~~ ## 4.4 创建阿里云容器镜像服务 ![镜像服务](../../img/kubernetes/kubernetes_ack/镜像服务.png) ![镜像服务02](../../img/kubernetes/kubernetes_ack/镜像服务02.png) ![镜像服务03](../../img/kubernetes/kubernetes_ack/镜像服务03.png) ![创建仓库01](../../img/kubernetes/kubernetes_ack/创建仓库01.png) ![创建仓库05](../../img/kubernetes/kubernetes_ack/创建仓库05.png) ![创建仓库03](../../img/kubernetes/kubernetes_ack/创建仓库03.png) ![创建仓库04](../../img/kubernetes/kubernetes_ack/创建仓库04.png) ## 4.5 安装docker ~~~powershell [root@kubemsb ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo [root@kubemsb ~]# yum -y install docker-ce [root@kubemsb ~]# systemctl enable docker [root@kubemsb ~]# systemctl start docker ~~~ ## 4.6 上传镜像至服务器 将资源目录下的“apidemo.tar上传至服务器” ## 4.7 导入镜像 ~~~powershell [root@kubemsb ~]# docker load -i apidemo.tar [root@kubemsb ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE apidemo v1 cae07723a2c3 7 minutes ago 15.5MB ~~~ ## 4.8 登录镜像仓库 ~~~powershell [root@kubemsb ~]# docker login --username=你的用户名 registry.cn-chengdu.aliyuncs.com ~~~ ## 4.9 上传镜像至阿里云镜像仓库 ~~~powershell [root@kubemsb ~]# docker tag apidemo:v1 registry.cn-chengdu.aliyuncs.com/kubemsb/apidemo:v1 [root@kubemsb ~]# docker push registry.cn-chengdu.aliyuncs.com/kubemsb/apidemo:v1 ~~~ ![image-20221012170435435](../../img/kubernetes/kubernetes_ack/image-20221012170435435.png) ## 4.10在阿里云控制台部署项目 ### 4.10.1 部署apidemo_Deployment ![apidemo01](../../img/kubernetes/kubernetes_ack/apidemo01.png) ![apidemo02](../../img/kubernetes/kubernetes_ack/apidemo02.png) ![apidemo03](../../img/kubernetes/kubernetes_ack/apidemo03.png) ![apidemo04](../../img/kubernetes/kubernetes_ack/apidemo04.png) ![apidemo05](../../img/kubernetes/kubernetes_ack/apidemo05.png) ![image-20221012170947145](../../img/kubernetes/kubernetes_ack/image-20221012170947145.png) ![image-20221012171112503](../../img/kubernetes/kubernetes_ack/image-20221012171112503.png) ![apidemo08](../../img/kubernetes/kubernetes_ack/apidemo08.png) ![apidemo09](../../img/kubernetes/kubernetes_ack/apidemo09.png) ### 4.10.2 部署apidemo_SVC ![apidemosvc01](../../img/kubernetes/kubernetes_ack/apidemosvc01.png) ![apidemosvc02](../../img/kubernetes/kubernetes_ack/apidemosvc02.png) ### 4.10.3 部署apidemo_ingress ![apidemoingress01](../../img/kubernetes/kubernetes_ack/apidemoingress01.png) ![apidemoingress02](../../img/kubernetes/kubernetes_ack/apidemoingress02.png) ![apidemoingress03](../../img/kubernetes/kubernetes_ack/apidemoingress03.png) ![apidemoingress04](../../img/kubernetes/kubernetes_ack/apidemoingress04.png) ![apidemoingress05](../../img/kubernetes/kubernetes_ack/apidemoingress05.png) ## 4.11 访问apidemo服务 ### 4.11.1 pc添加hosts ![修改hosts01](../../img/kubernetes/kubernetes_ack/修改hosts01-16655662413204.png) ![修改hosts02](../../img/kubernetes/kubernetes_ack/修改hosts02-16655662470795.png) ![修改hosts03](../../img/kubernetes/kubernetes_ack/修改hosts03-16655662512246.png) ![修改hosts05](../../img/kubernetes/kubernetes_ack/修改hosts05.png) ### 4.11.2 访问apidemo服务 > 浏览器访问apidemo.kubemsb.com/info 返回如下信息则证明服务部署成功 ![image-20221012171836782](../../img/kubernetes/kubernetes_ack/image-20221012171836782.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_calico.md ================================================ # kubernetes网络解决方案 Calico # 一、CNI方案 学习目标 这一节，我们从 基础原理、方案解读、小结 三个方面来学习 **基础原理** 容器访问模式 ![1633650192071](../../img/kubernetes/kubernetes_calico/1633650192071.png) ```powershell 方法1： 虚拟网桥 + 虚拟网卡对 方法2： 多路复用 + 内核级的VLAN模块 方法3： 硬件交换 + 硬件虚拟化 ``` CNI简介 ```powershell 根据我们对于容器的基本了解，他虽然可以借助于虚拟网桥docker0实现一定程度的网络功能，但是在大范围容器访问层面，其没有最好的网络解决方案，只能借助于一些第三方的网络解决方案来实现容器级别的跨网络通信能力。 CNI的全称是Container Network Interface，Google和CoreOS联合定制的多容器通信的网络模型。在Kubernetes中通过一个CNI接口来替代docker0，它在宿主机上的默认名字叫cni0。它没有使用Docker的网络模型的原因有两个：1 不想被Docker要挟，2 自有的网络namespace设计有关。 ``` ![1633651873150](../../img/kubernetes/kubernetes_calico/1633651873150.png) ```powershell CNI的设计思想即为：Kubernetes在启动Pod的pause容器之后，直接调用CNI网络插件，从而实现为Pod内部应用容器所在的Network Namespace配置符合预期的网络信息。这里面需要特别关注两个方面： - Container必须有自己的网络命名空间的环境，也就是endpoint地址。 - Container所在的网段必须能够注册网络地址信息 对容器网络的设置和操作都通过插件（Plugin）进行具体实现，CNI插件包括两种类型：CNIPlugin和IPAM（IP Address Management）Plugin。CNI Plugin负责为容器配置网络资源，IPAM Plugin负责对容器的IP地址进行分配和管理。IPAM Plugin作为CNI Plugin的一部分，与CNI Plugin一起工作。 ``` ```powershell 在Kubernetes中，CNI对于容器网络的设置主要是以CNI Plugin插件的方式来为容器配置网络资源，它主要有三种模式： MainPlugin - 用来创建具体的网络设备的二进制文件 - 比如bridge、ipvlan、vlan、host-device IPAM Plugin - IPAM 就是 IP Address Management - 负责对容器的IP地址进行分配和管理，作为CNI Plugin的一部分，与CNI Plugin一起工作 Meta Plugin - 由CNI社区维护的内部插件功能模块,常见的插件功能模块有以下几种 - flannel 专门为Flannel项目提供的插件 - tuning 通过sysctl调整网络设备参数的二进制文件 - portmap 通过iptables配置端口映射的二进制文件 - bandwidth 使用 Token Bucket Filter (TBF)来进行限流的二进制文件 - firewall 通过iptables或者firewalled添加规则控制容器的进出流量 ``` ```powershell 更多详情查看：https://github.com/containernetworking/cni/blob/main/SPEC.md ``` CNI目前被谁管理？ ```powershell 在 Kubernetes 1.24 之前，CNI 插件也可以由 kubelet 使用命令行参数 cni-bin-dir 和 network-plugin 管理。而在Kubernetes 1.24 移除了这些命令行参数， CNI 的管理不再是 kubelet 的工作。而变成下层的容器引擎需要做的事情了，比如cri-dockerd服务的启动文件 ``` ```powershell 查看服务文件 /etc/systemd/system/cri-docker.service ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin ... 注意： /opt/cni/bin 目录是部署kubernetes的时候，安装的cni-tools软件包自动创建出来的，这里面包含了很多的网络命令工具 ``` ```powershell flannel的CNI配置文件 # cat /etc/cni/net.d/10-flannel.conflist { "name": "cbr0", "plugins": [ { "type": "flannel", # 为Flannel项目提供的插件，配置容器网络 "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", # 通过iptables配置端口映射的二进制文件，配置端口映射 "capabilities": { "portMappings": true } } ] } ``` ```powershell calico的CNI配置文件 # cat /etc/cni/net.d/10-calico.conflist { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "log_level": "info", "log_file_path": "/var/log/calico/cni/cni.log", "datastore_type": "kubernetes", "nodename": "kubernetes-master", "mtu": 0, "ipam": { "type": "host-local", "subnet": "usePodCidr" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} }, { "type": "bandwidth", "capabilities": {"bandwidth": true} } ] } ``` **方案解读** ```powershell kubernetes提供了很多的网络解决方案，相关的资料如下： https://kubernetes.io/zh-cn/docs/concepts/cluster-administration/addons/ ``` flannel方案 ```powershell 项目地址：https://github.com/coreos/flannel Flannel是CoreOS 开发的项目，它是容器编排系统中最成熟的网络方案之一，旨在实现更好的容器间和主机间网络，由于其稳定和配置简单，所以它是CNI最早引入的一套方案。常见的 Kubernetes 集群部署工具和许多 Kubernetes 发行版都可以默认安装 Flannel。 如果你需要一个稳定而又简单的网络功能的话，不那么在乎安全性的话，Flannel是一个很好的选择。 ``` calico方案 ```powershell 项目地址：https://github.com/projectcalico/cni-plugin Calico 是 Kubernetes 生态系统中另一种流行的网络选择。虽然 Flannel 被公认为是最简单的选择，但 Calico 以其性能、灵活性而闻名。Calico 的功能更为全面，除了通用的网络连接，还涉及网络安全和网络策略管理，甚至Calico还可以在微服务的网络治理中进行整合。Calico CNI 插件在 CNI 框架内封装了 Calico 的功能。 如果对你的环境而言，支持网络策略是非常重要的一点，而且你对其他性能和功能也有需求，那么 Calico 会是一个理想的选择。 ``` canal方案 ```powershell 项目地址：https://github.com/projectcalico/canal Canal 试图将 Flannel 提供的网络功能与 Calico 的网络策略功能集成在一起，只不过在研发的时候，发现Flannel 和 Calico 这两个项目的标准化和灵活性都想当标准，那集成就没必要了，所以目前这个项目“烂尾”了。由于Canal的思路很好，业界习惯性地将 Flannel 和 Calico 的功能组合实施称为“Canal”。 对于那些喜欢 Flannel 提供的网络模型，同时喜欢 Calico 的网络策略功能的人来说，Canal是一个选择。 ``` weave方案 ```powershell 项目地址：https://www.weave.works/oss/net/ Weave 是由 Weaveworks 提供的一种 Kubernetes CNI 网络选项，它在集群中的每个节点上部署路由组件，从而实现主机之间创建更加灵活的网状 overlay 网络，借助于内核级别的Open vSwitch 配置，从而实现具有一定程度的智能路由功能。除此之外，weave还具有想当的网络策略功能，网络加密传输功能等。 对于那些寻求功能丰富的网络、同时希望不要增加大量复杂性或管理难度的人来说，Weave 是一个很好的选择。 ``` ![image-20220903022845285](../../img/kubernetes/kubernetes_calico/image-20220903022845285.png) ```powershell bandwidth - 带宽、consumption - 消耗、encryption - 加密 资料来源： https://docs.google.com/spreadsheets/d/12dQqSGI0ZcmuEy48nA0P_bPl7Yp17fNg7De47CYWzaM/edit#gid=404703955 ``` **小结** ``` ``` # 二、 Calico环境 学习目标 这一节，我们从 基础知识、原理解读、小结 三个方面来学习 **基础知识** 简介 ``` Calico是一个开源的虚拟化网络方案,用于为云原生应用实现互联及策略控制.相较于 Flannel 来说,Calico 的优势是对网络策略(network policy),它允许用户动态定义 ACL 规则控制进出容器的数据报文,实现为 Pod 间的通信按需施加安全策略.不仅如此,Calico 还可以整合进大多数具备编排能力的环境,可以为 虚机和容器提供多主机间通信的功能。 Calico 本身是一个三层的虚拟网络方案,它将每个节点都当作路由器,将每个节点的容器都当作是节点路由器的一个终端并为其分配一个 IP 地址,各节点路由器通过 BGP(Border Gateway Protocol)学习生成路由规则,从而将不同节点上的容器连接起来.因此,Calico 方案其实是一个纯三层的解决方案,通过每个节点协议栈的三层（网络层）确保容器之间的连通性,这摆脱了 flannel host-gw 类型的所有节点必须位于同一二层网络的限制,从而极大地扩展了网络规模和网络边界。 官方地址：https://www.tigera.io/project-calico/ 最新版本：v3.24.1 (20220827)、 v3.20.6 (20220802)、v3.21.6 (20220722)、 v3.23.3 (20220720)、v3.22.4 (20220720) ``` 网络模型 ``` Calico为了实现更广层次的虚拟网络的应用场景，它支持多种网络模型来满足需求。 underlay network - BGP(三层虚拟网络解决方案) overlay network - IPIP(双层IP实现跨网段效果)、VXLAN(数据包标识实现大二层上的跨网段通信) ``` 设计思想 ``` Calico不使用隧道或者NAT来实现转发，而是巧妙的把所有二三层流量转换成三层流量，并通过host上路由配置完成跨host转发。 ``` 为什么用calico ![image-20220905120240844](../../img/kubernetes/kubernetes_calico/image-20220905120240844.png) **原理解读** calico ```powershell Calico是一个开源的虚拟化网络方案,用于为云原生应用实现互联及策略控制.相较于 Flannel 来说,Calico 的优势是对网络策略(network policy),它允许用户动态定义 ACL 规则控制进出容器的数据报文,实现为 Pod 间的通信按需施加安全策略.不仅如此,Calico 还可以整合进大多数具备编排能力的环境,可以为 虚机和容器提供多主机间通信的功能。 我们平常使用Calico主要用到的是它的网络策略功能 ``` 软件结构 ![image-20220722175645610](../../img/kubernetes/kubernetes_calico/image-20220722175645610.png) ```powershell Felix 每个节点都有，负责配置路由、ACL、向etcd宣告状态等 BIRD 每个节点都有，负责把 Felix 写入Kernel的路由信息 分发到整个 Calico网络，确保 workload 连通 etcd 存储calico自己的状态数据，可以结合kube-apiserver来工作 官方推荐； < 50节点,可以结合 kube-apiserver 来实现数据的存储 > 50节点,推荐使用独立的ETCD集群来进行处理。 参考资料： https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico Route Reflector 路由反射器，用于集中式的动态生成所有主机的路由表，非必须选项 超过100个节点推荐使用： https://projectcalico.docs.tigera.io/getting-started/kubernetes/rancher#concepts Calico编排系统插件 实现更广范围的虚拟网络解决方案。 参考资料：https://docs.projectcalico.org/reference/architecture/overview ``` 工作模式 ![image-20220722180040443](../../img/kubernetes/kubernetes_calico/image-20220722180040443.png) ```powershell 对于节点量少的情况下，我们推荐使用左侧默认的模式，当节点量多的时候，我们推荐使用右侧的反射器模式 ``` **小结** ``` ``` ## 2.1 Calico部署 学习目标 这一节，我们从 环境解析、简单实践、小结 三个方面来学习 **环境解析** k8s环境上的calico逻辑 ![architecture-calico](../../img/kubernetes/kubernetes_calico/architecture-calico.svg) ```powershell confd - 统一管控 Calico 数据存储以及 BGP 配置的轻量级的配置管理工具。 - 当配置文件发生变化时，动态更新和生成BIRD 配置文件 Dikastes - 为 Istio 服务网格实施网络策略。作为 Istio Envoy 的 sidecar 代理在集群上运行。 CNI plugin - 为 Kubernetes 集群提供 Calico 网络，必须安装在 Kubernetes 集群中的每个节点上。 Datastore plugin - 独立使用etcd作为Calico的数据存储平台，特点在于独立扩展数据存储 - 结合K8s的apiserver实现数据存储到etcd中，特点在于使用 Kubernetes的存储、RBAC、审计功能为Calico服务 IPAM plugin - 使用 Calico 的 IP 池资源来控制 IP 地址如何分配给集群内的 Pod。 kube-controllers - 监控 Kubernetes API 并根据集群状态执行操作 - 主要是策略、命名空间、服务账号、负载、节点等通信策略的控制 Typha - 通过减少每个节点对数据存储的影响来增加规模。在数据存储和 Felix 实例之间作为守护进程运行。 参考资料： https://projectcalico.docs.tigera.io/reference/architecture/overview ``` 基础环境支持 ```powershell linux 主机基本要求： x86-64、arm64、ppc64le 或 s390x 处理器 2CPU 2GB 内存 10GB 可用磁盘空间 RedHat Enterprise Linux 7.x+、CentOS 7.x+、Ubuntu 16.04+ 或 Debian 9.x+ 确保 Calico 可以在主机上进行管理cali和接口 其他需求： kubernetes集群配置了 --pod-network-cidr 属性 参考资料： https://projectcalico.docs.tigera.io/getting-started/kubernetes/quickstart ``` 部署解析 ```powershell 对于calico在k8s集群上的部署来说，为了完成上面四个组件的部署，这里会涉及到两个部署组件 ``` | 组件名 | 组件作用 | | ---------------------- | ------------------------------------------------------------ | | calico-node | 需要部署到所有集群节点上的代理守护进程，提供封装好的Felix和BIRD | | calico-kube-controller | 专用于k8s上对calico所有节点管理的中央控制器。负责calico与k8s集群的协同及calico核心功能实现。 | 部署步骤 ```powershell 1 获取资源配置文件 从calico官网获取相关的配置信息 2 定制CIDR配置 定制calico自身对于pod网段的配置信息，并且清理无关的网络其他插件 3 定制pod的manifest文件分配网络配置 默认的k8s集群在启动的时候，会有一个cidr的配置，有可能与calico进行冲突，那么我们需要修改一下 4 应用资源配置文件 ``` ```powershell 注意事项： 对于calico来说，它自己会生成自己的路由表，如果路由表中存在响应的记录，默认情况下会直接使用，而不是覆盖掉当前主机的路由表 所以如果我们在部署calico之前，曾经使用过flannel，尤其是flannel的host-gw模式的话，一定要注意，在使用calico之前，将之前所有的路由表信息清空，否则无法看到calico的tunl的封装效果 ``` **简单实践** 环境部署 ```powershell 1 获取资源清单文件 mkdir /data/kubernetes/network/calico -p cd /data/kubernetes/network/calico/ curl https://docs.projectcalico.org/manifests/calico.yaml -O cp calico.yaml{,.bak} ``` ```powershell 2 配置资源清单文件 # vim calico.yaml ---- 官网推荐的修改内容 4546 - name: CALICO_IPV4POOL_CIDR 4547 value: "10.244.0.0/16" ---- 方便我们的后续实验，新增调小子网段的分配 4548 - name: CALICO_IPV4POOL_BLOCK_SIZE 4549 value: "24" 配置解析： 开放默认注释的CALICO_IPV4POOL_CIDR变量，然后定制我们当前的pod的网段范围即可 原则上来说，我们修改官方提示的属性即可 ``` ```powershell 3 定制pod的manifest文件分配网络配置 # vim calico.yaml ---- 修改下面的内容 64 "ipam": { 65 "type": "calico-ipam" 66 }, ---- 修改后的内容 64 "ipam": { 65 "type": "host-local", 66 "subnet": "usePodCidr" 67 }, ---- 定制calico使用k8s集群节点的地址 4551 - name: USE_POD_CIDR 4552 value: "true" 配置解析: Calico默认并不会从Node.Spec.PodCIDR中分配地址，但可通过USE_POD_CIDR变量并结合host-local这一IPAM插件以强制从PodCIDR中分配地址 ``` ```powershell 4 定制默认的docker镜像 查看默认的镜像 # grep docker.io calico.yaml | uniq image: docker.io/calico/cni:v3.24.1 image: docker.io/calico/node:v3.24.1 image: docker.io/calico/kube-controllers:v3.24.1 获取镜像 for i in $(grep docker.io calico.yaml | uniq | awk -F'/' '{print $NF}') do docker pull calico/$i docker tag calico/$i kubernetes-register.superopsmsb.com/google_containers/$i docker push kubernetes-register.superopsmsb.com/google_containers/$i docker rmi calico/$i done 修改为定制的镜像 sed -i 's#docker.io/calico#kubernetes-register.superopsmsb.com/google_containers#g' calico.yaml 查看效果 # grep google calico.yaml image: kubernetes-register.superopsmsb.com/google_containers/cni:v3.24.1 image: kubernetes-register.superopsmsb.com/google_containers/cni:v3.24.1 image: kubernetes-register.superopsmsb.com/google_containers/node:v3.24.1 image: kubernetes-register.superopsmsb.com/google_containers/node:v3.24.1 image: kubernetes-register.superopsmsb.com/google_containers/kube-controllers:v3.24.1 ``` ```powershell 5 应用资源配置文件 清理之前的flannel插件 kubectl delete -f kube-flannel.yml kubectl get pod -n kube-system | grep flannel 这个时候，先清除旧网卡，然后最好重启一下主机，直接清空所有的路由表信息 ifconfig flannel.1 reboot 重启后，查看网络效果 注意： 为了避免后续calico网络测试的异常，我们这里最好只留下一个网卡 eth0 ``` ```powershell 应用calico插件 kubectl apply -f calico.yaml 在calico-node部署的时候，会启动多个进程 # kubectl get pod -n kube-system | egrep 'NAME|calico' NAME READY STATUS RESTARTS AGE calico-kube-controllers-549f7748b5-xqz8j 0/1 ContainerCreating 0 9s calico-node-74c5w 0/1 Init:0/3 0 9s ... 环境部署完毕后，查看效果 # kubectl get pod -n kube-system | egrep 'NAME|calico' NAME READY STATUS RESTARTS AGE calico-kube-controllers-549f7748b5-xqz8j 0/1 Running 0 39s calico-node-74c5w 0/1 Running 0 39s ... ``` ```powershell 每个calico节点上都有多个进程，分别来处理不同的功能 ]# ps aux | egrep 'NAME | calico' root 9315 0.5 1.1 1524284 43360 ? Sl 15:29 0:00 calico-node -confd root 9316 0.2 0.8 1155624 32924 ? Sl 15:29 0:00 calico-node -monitor-token root 9317 2.8 1.0 1598528 40992 ? Sl 15:29 0:02 calico-node -felix root 9318 0.3 0.9 1155624 35236 ? Sl 15:29 0:00 calico-node -monitor-addresses root 9319 0.3 0.8 1155624 33460 ? Sl 15:29 0:00 calico-node -status-reporter root 9320 0.2 0.7 1155624 30364 ? Sl 15:29 0:00 calico-node -allocate-tunnel-addrs ``` 测试效果 ```powershell 获取镜像 docker pull nginx docket tag nginx kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.1 docker push kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.1 docker rmi nginx ``` ```powershell 创建一个deployment kubectl create deployment pod-deployment --image=kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.1 --replicas=3 查看pod # kubectl get pod NAME READY STATUS RESTARTS AGE pod-deployment-554dff674-267gq 1/1 Running 0 48s pod-deployment-554dff674-c8cjs 1/1 Running 0 48s pod-deployment-554dff674-pxrwb 1/1 Running 0 48s 暴露一个service kubectl expose deployment pod-deployment --port=80 --target-port=80 确认效果 kubectl get service curl 10.108.138.97 ``` **小结** ``` ``` ## 2.2 简单实践 学习目标 这一节，我们从 网络解析、命令完善、小结 三个方面来学习 **网络解析** calico部署完毕后，会生成一系列的自定义配置属性信息 ```powershell 自动生成一个api版本信息 # kubectl api-versions | grep crd crd.projectcalico.org/v1 该api版本信息中有大量的配置属性 kubectl api-resources | grep crd.pro ``` ```powershell 这里面的 ippools 里面包含了calico相关的网络属性信息 # kubectl get ippools NAME AGE default-ipv4-ippool 37m 查看这里配置的calico相关的信息 # kubectl get ippools default-ipv4-ippool -o yaml apiVersion: crd.projectcalico.org/v1 kind: IPPool ... spec: allowedUses: - Workload - Tunnel blockSize: 24 cidr: 10.244.0.0/16 ipipMode: Always natOutgoing: true nodeSelector: all() vxlanMode: Never 结果显式： 这里的calico采用的模型就是 ipip模型，分配的网段是使我们定制的 cidr网段，而且子网段也是我们定制的 24位掩码 ``` 环境创建完毕后，会生成一个tunl0的网卡，所有的流量会走这个tunl0网卡 ```powershell 确认网卡和路由信息 ]# ifconfig | grep flags ]# ip route list | grep tun 10.244.1.0/24 via 10.0.0.13 dev tunl0 proto bird onlink 10.244.2.0/24 via 10.0.0.14 dev tunl0 proto bird onlink 10.244.3.0/24 via 10.0.0.15 dev tunl0 proto bird onlink 10.244.4.0/24 via 10.0.0.16 dev tunl0 proto bird onlink 10.244.5.0/24 via 10.0.0.17 dev tunl0 proto bird onlink 结果显示： calico模型中，默认使用的是tunl0虚拟网卡实现数据包的专线转发 ``` 测试效果 ```powershell 由于在calico的默认网络模型是 IPIP，所以我们在进行数据包测试的时候，可以通过直接抓取宿主机数据包，来发现双层ip效果 kubectl get pod -o wide 在master1上采用ping的方式来测试 node2上的节点pod [root@kubernetes-master1 /data/kubernetes/network/calico]# ping -c 1 10.244.4.3 PING 10.244.4.3 (10.244.4.3) 56(84) bytes of data. 64 bytes from 10.244.4.3: icmp_seq=1 ttl=63 time=0.794 ms ``` ```powershell 在node2上检测数据包的效果 [root@kubernetes-node2 ~]# tcpdump -i eth0 -nn ip host 10.0.0.16 and host 10.0.0.12 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:38:52.231680 IP 10.0.0.12 > 10.0.0.16: IP 10.244.0.1 > 10.244.4.3: ICMP echo request, id 19189, seq 1, length 64 (ipip-proto-4) 15:38:52.231989 IP 10.0.0.16 > 10.0.0.12: IP 10.244.4.3 > 10.244.0.1: ICMP echo reply, id 19189, seq 1, length 64 (ipip-proto-4) 15:38:54.666538 IP 10.0.0.16.33992 > 10.0.0.12.179: Flags [P.], seq 4281052787:4281052806, ack 3643645463, win 58, length 19: BGP 15:38:54.666962 IP 10.0.0.12.179 > 10.0.0.16.33992: Flags [.], ack 19, win 58, length 0 结果显式： 每个数据包都是基于双层ip嵌套的方式来进行传输，而且协议是 ipip-proto-4 结合路由的分发详情，可以看到具体的操作效果。 具体效果：10.244.0.1 -> 10.0.0.12 -> 10.0.0.16 -> 10.244.6.3 ``` **命令完善** 简介 ```powershell calico本身是一个复杂的系统，复杂到它自己提供一个非常重要的Restful接口，结合calicoctl命令来管理自身的相关属性信息，calicoctl可以直接与etcd进行操作，也可以通过kube-apiserver的方式与etcd来进行操作。默认情况下，它与kube-apiserver通信的认证方式与kubectl的命令使用同一个context。但是我们还是推荐，使用手工定制的一个配置文件。 calicoctl 是运行在集群之外的，用于管理集群功能的一个重要的组件。calicoctl 的安装方式很多，常见的方式有：单主机方式、kubectl命令插件方式、pod方式、主机容器方式。我们需要自己选择一种适合自己的方式 参考资料：https://projectcalico.docs.tigera.io/getting-started/kubernetes/hardway/the-calico-datastore#install ``` ```powershell 获取专用命令 cd /usr/local/bin/ curl -L https://github.com/projectcalico/calico/releases/download/v3.24.1/calicoctl-linux-amd64 -o calicoctl chmod +x calicoctl 查看帮助 # calicoctl --help Usage: calicoctl [options] [...] ``` 命令的基本演示 ```powershell 查看ip的管理 calicoctl ipam --help 查看ip的信息 # calicoctl ipam show +----------+---------------+-----------+------------+--------------+ | GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE | +----------+---------------+-----------+------------+--------------+ | IP Pool | 10.244.0.0/16 | 65536 | 0 (0%) | 65536 (100%) | +----------+---------------+-----------+------------+--------------+ ``` ```powershell 查看信息的显式效果 calicoctl ipam show --help 显式相关的配置属性 # calicoctl ipam show --show-configuration +--------------------+-------+ | PROPERTY | VALUE | +--------------------+-------+ | StrictAffinity | false | | AutoAllocateBlocks | true | | MaxBlocksPerHost | 0 | +--------------------+-------+ ``` 将calico整合到kubectl里面 ```powershell 定制kubectl 插件子命令 # cd /usr/local/bin/ # cp -p calicoctl kubectl-calico 测试效果 # kubectl calico --help Usage: kubectl-calico [options] [...] 后续的操作基本上都一样了，比如获取网络节点效果 [root@kubernetes-master1 /usr/local/bin]# kubectl calico node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.0.0.15 | node-to-node mesh | up | 07:30:48 | Established | | 10.0.0.17 | node-to-node mesh | up | 07:30:48 | Established | | 10.0.0.13 | node-to-node mesh | up | 07:30:48 | Established | | 10.0.0.14 | node-to-node mesh | up | 07:30:51 | Established | | 10.0.0.16 | node-to-node mesh | up | 07:31:41 | Established | +--------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. 注意： 这里查看的是除了自己网段之外的其他节点的路由信息 ``` **小结** ``` ``` ## 2.3 BGP实践 学习目标 这一节，我们从 bgp改造、反射器实践、小结 三个方面来学习 **bgp改造** 简介 ```powershell 对于现有的calico环境，我们如果需要使用BGP环境，我们可以直接使用一个配置清单来进行修改calico环境即可。我们这里先来演示一下如何使用calicoctl修改配置属性。 ``` ```powershell 获取当前的配置属性 # kubectl calico get ipPools NAME CIDR SELECTOR default-ipv4-ippool 10.244.0.0/16 all() # kubectl calico get ipPools default-ipv4-ippool -o yaml apiVersion: projectcalico.org/v3 kind: IPPool ... spec: blockSize: 24 cidr: 10.244.0.0/16 ipipMode: Always natOutgoing: true nodeSelector: all() vxlanMode: Never ``` ```powershell 定制资源配置文件 kubectl calico get ipPools default-ipv4-ippool -o yaml > default-ipv4-ippool.yaml 修改配置文件 # cat default-ipv4-ippool.yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: blockSize: 24 cidr: 10.244.0.0/16 ipipMode: CrossSubnet natOutgoing: true nodeSelector: all() vxlanMode: Never 配置解析： 仅仅将原来的Always 更换成 CrossSubnet(跨节点子网) 模式即可 vxlanMode 的两个值可以实现所谓的 BGP with vxlan的效果 应用资源配置文件 # kubectl calico apply -f default-ipv4-ippool.yaml Successfully applied 1 'IPPool' resource(s) ``` 检查效果 ```powershell 查看路由信息 [root@kubernetes-master1 ~]# ip route list | grep via default via 10.0.0.2 dev eth0 10.244.1.0/24 via 10.0.0.13 dev eth0 proto bird 10.244.2.0/24 via 10.0.0.14 dev eth0 proto bird 10.244.3.0/24 via 10.0.0.15 dev eth0 proto bird 10.244.4.0/24 via 10.0.0.16 dev eth0 proto bird 10.244.5.0/24 via 10.0.0.17 dev eth0 proto bird 结果显式： 更新完毕配置后，动态路由的信息就发生改变了，不再完全是 tunl0 网卡了，而是变成了通过具体的物理网卡eth0 转发出去了 ``` ```powershell 在master1上ping在节点node2上的pod [root@kubernetes-master1 ~]# ping -c 1 10.244.4.3 PING 10.244.4.3 (10.244.4.3) 56(84) bytes of data. 64 bytes from 10.244.4.3: icmp_seq=1 ttl=63 time=0.671 ms 由于这次是直接通过节点进行转发的，所以我们在node2节点上抓包的时候，直接通过内层ip抓取即可。 [root@kubernetes-node2 ~]# tcpdump -i eth0 -nn ip host 10.244.4.3 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:47:32.906462 IP 10.0.0.12 > 10.244.4.3: ICMP echo request, id 28248, seq 1, length 64 15:47:32.906689 IP 10.244.4.3 > 10.0.0.12: ICMP echo reply, id 28248, seq 1, length 64 结果显示： 他们实现了直接的连通，无需进行数据包的转换了，效率更高一点 ``` **反射器实践** 当前节点的网络效果 ```powershell 查看当前节点的网络效果 [root@kubernetes-master1 ~]# kubectl calico node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.0.0.15 | node-to-node mesh | up | 07:30:48 | Established | | 10.0.0.17 | node-to-node mesh | up | 07:30:48 | Established | | 10.0.0.13 | node-to-node mesh | up | 07:30:48 | Established | | 10.0.0.14 | node-to-node mesh | up | 07:30:51 | Established | | 10.0.0.16 | node-to-node mesh | up | 07:31:41 | Established | +--------------+-------------------+-------+----------+-------------+ ``` 需求 ```powershell 当前的calico节点的网络状态是 BGP peer 的模型效果，也就是说 每个节点上都要维护 n-1 个路由配置信息，整个集群中需要维护 n*(n-1) 个路由效果，这在节点量非常多的场景中，不是我们想要的。 所以我们需要实现一种 BGP reflecter 的效果。 ``` ```powershell 如果我们要做 BGP reflecter 效果的话，需要对反射器角色做冗余，如果我们的集群是一个多主集群的话，可以将集群的master节点作为bgp的reflecter角色，从而实现反射器的冗余。 1 定制反射器角色 2 后端节点使用反射器 3 关闭默认的网格效果 ``` 1 创建反射器角色 ```powershell 定制资源定义文件 01-calico-reflector-master.yaml apiVersion: projectcalico.org/v3 kind: Node metadata: labels: route-reflector: true name: kubernetes-master1 spec: bgp: ipv4Address: 10.0.0.12/16 ipv4IPIPTunnelAddr: 10.244.0.1 routeReflectorClusterID: 1.1.1.1 属性解析； metadata.labels 是非常重要的，因为它需要被后面的节点来进行获取 metadata.name 的属性，必须是通过 calicoctl get nodes 获取到的节点名称。 spec.bgp.ipv4Address必须是 指定节点的ip地址 spec.bgp.ipv4IPIPTunnelAddr必须是 指定节点上tunl0的地址 spec.bgp.routeReflectorClusterID 是BGP网络中的唯一标识，所以这里的集群标识只要唯一就可以了 应用资源配置文件 kubectl calico apply -f 01-calico-reflector-master.yaml ``` 2 更改节点的网络模型为 reflecter模型 ```powershell 定制资源定义文件 02-calico-reflector-bgppeer.yaml kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: bgppeer-demo spec: nodeSelector: all() peerSelector: route-reflector=="true" 属性解析； spec.nodeSelector 指定的所有后端节点 spec.peerSelector 指定的是反射器的标签，标识所有的后端节点与这个反射器进行数据交流 应用资源配置文件 kubectl calico apply -f 02-calico-reflector-bgppeer.yaml ``` 3 关闭默认的网格效果 ```powershell 定制资源定义文件 03-calico-reflector-defaultconfig.yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false asNumber: 63400 属性解析； metadata.name 在这里最好是default，因为我们要对BGP默认的网络效果进行关闭 spec.nodeToNodeMeshEnabled 关闭后端节点的BGP peer默认状态 -- 即点对点通信关闭 spec.asNumber 指定的是后端节点间使用反射器的时候，我们要在一个标志号内，这里随意写一个 应用资源配置文件 kubectl calico apply -f 03-calico-reflector-defaultconfig.yaml ``` ```powershell 查看效果 [root@kubernetes-master1 ~/txt]# kubectl calico node status ... IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 10.0.0.13 | node specific | up | 07:51:47 | Established | | 10.0.0.14 | node specific | up | 07:51:49 | Established | | 10.0.0.15 | node specific | up | 07:51:49 | Established | | 10.0.0.16 | node specific | up | 07:51:47 | Established | | 10.0.0.17 | node specific | up | 07:51:47 | Established | +--------------+---------------+-------+----------+-------------+ 结果显式： 默认的点对点通信方式就已经丢失了，剩下了反射器模式 ``` **小结** ``` ``` ## 2.4 策略实践 学习目标 这一节，我们从 属性解读、基本控制、小结 三个方面来学习。 **属性解读** 策略简介 ```powershell 为了使用Network Policy，Kubernetes引入了一个新的资源对象Network Policy，供用户设置Pod间网络访问的策略。策略控制器用于监控指定区域创建对象(pod)时所生成的新API端点，并按需为其附加网络策略。 对于Pod对象来说，网络流量分为 流入(Ingress)和流出(Egress)两个方向，每个方向包含允许和禁止两种控制策略，默认情况下，所有的策略都是允许的，应用策略后，所有未经明确允许的流量都将拒绝。 ``` ![image-20220722193530812](../../img/kubernetes/kubernetes_calico/image-20220722193530812.png) ```powershell 注意： 网络策略的控制，可以是多个级别： 集群级别、namespace级别、Pod级别、ip级别、端口级别 ``` 资源对象属性 ```powershell apiVersion: networking.k8s.io/v1 # 资源隶属的API群组及版本号 kind: NetworkPolicy # 资源类型的名称，名称空间级别的资源； metadata: # 资源元数据 name # 资源名称标识 namespace # NetworkPolicy是名称空间级别的资源 spec: # 期望的状态 podSelector # 当前规则生效的一组目标Pod对象，必选字段；空值表示当前名称空间中的所有Pod资源 policyTypes <[]string> # Ingress表示生效ingress字段；Egress表示生效egress字段，同时提供表示二者均有效 ingress <[]Object> # 入站流量源端点对象列表，白名单，空值表示“所有” - from <[]Object> # 具体的端点对象列表，空值表示所有合法端点 - ipBlock # IP地址块范围内的端点，不能与另外两个字段同时使用 - namespaceSelector # 匹配的名称空间内的端点 podSelector # 由Pod标签选择器匹配到的端点，空值表示 ports <[]Object> # 具体的端口对象列表，空值表示所有合法端口 egress <[]Object> # 出站流量目标端点对象列表，白名单，空值表示“所有” - to <[]Object> # 具体的端点对象列表，空值表示所有合法端点，格式同ingres.from； ports <[]Object> # 具体的端口对象列表，空值表示所有合法端口 注意： 入栈和出栈哪个策略生效，由 policyTypes 来决定。 如果仅配置了podSelector，表明，当前限制仅限于当前的命名空间 ``` 配置示例 ```powershell apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: test-network-policy namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Ingress - Egress ingress: - from: - ipBlock: cidr: 10.244.0.0/16 except: - 10.244.1.0/24 - namespacesSelector: matchLabels: project: develop - podSelector: matchLabels: arch: frontend ports: - protocol: TCP port: 6379 egress: - to: - ipBlock: cidr: 10.244.0.0/24 ports: - protocol: TCP port: 3306 ``` 准备工作 ```powershell 在default命名空间创建应用 [root@kubernetes-master1 ~]# kubectl create deployment nginx-web --image=kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 [root@kubernetes-master1 ~]# kubectl expose deployment nginx-web --port=80 在superopsmsb命名空间创建应用 [root@kubernetes-master1 ~]# kubectl create deployment nginx-web --image=kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 --namespace=superopsmsb [root@kubernetes-master1 ~]# kubectl expose deployment nginx-web --port=80 --namespace=superopsmsb 确认效果 [root@kubernetes-master1 ~]# kubectl get pod -o wide NAME READY... AGE IP ... nginx-web-5865bb968d-759lc 1/1 ... 10s 10.244.1.4 ... [root@kubernetes-master1 ~]# kubectl get pod -o wide -n superopsmsb NAME READY... AGE IP ... nginx-web-65d688fd6-h6sbpp 1/1 ... 10s 10.244.1.5 ... ``` ```powershell 开启superopsmsb的测试容器 [root@kubernetes-master1 ~]# kubectl exec -it nginx-web-65d688fd6-h6sbp -n superopsmsb -- /bin/bash root@nginx-web-65d688fd6-h6sbp:/# apt update; apt install net-tools iputils-ping dnsutils curl -y ``` ```powershell 进入到容器里面查看效果 [root@kubernetes-master1 ~]# kubectl exec -it nginx-web-5865bb968d-759lc -- /bin/bash root@nginx-web-5865bb968d-759lc:/# apt update; apt install net-tools iputils-ping dnsutils curl -y 域名检测 root@nginx-web-5865bb968d-759lc:/# nslookup nginx-web Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx-web.default.svc.cluster.local Address: 10.101.181.105 root@nginx-web-5865bb968d-759lc:/# nslookup nginx-web.superopsmsb.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx-web.superopsmsb.svc.cluster.local Address: 10.105.110.175 ping检测 root@nginx-web-5865bb968d-759lc:/# ifconfig | grep 244 inet 10.244.1.4 netmask 255.255.255.255 broadcast 0.0.0.0 RX packets 22442 bytes 20956160 (19.9 MiB) root@nginx-web-5865bb968d-759lc:/# ping -c 1 10.244.1.5 PING 10.244.1.5 (10.244.1.5) 56(84) bytes of data. 64 bytes from 10.244.1.5: icmp_seq=1 ttl=63 time=0.357 ms ``` **基本控制** 默认拒绝规则 ```powershell 查看资源清单文件 # cat 01_kubernetes_secure_networkpolicy_refuse.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-ingress namespace: superopsmsb spec: podSelector: {} policyTypes: ["Ingress", "Egress"] 应用资源清单文件 # kubectl apply -f 01_kubernetes_secure_networkpolicy_refuse.yaml networkpolicy.networking.k8s.io/deny-all-ingress created ``` ```powershell ``` ```powershell 尝试default空间资源访问superopsmsb空间资源 root@nginx-web-5865bb968d-759lc:/# ping -c 1 10.244.1.5 PING 10.244.1.5 (10.244.1.5) 56(84) bytes of data. --- 10.244.1.5 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms root@nginx-web-5865bb968d-759lc:/# curl nginx-web.superopsmsb.svc.cluster.local curl: (28) Failed to connect to nginx-web.superopsmsb.svc.cluster.local port 80: Connection timed out 结果显示: 访问失败 default空间资源访问非superopsmsb空间资源正常。 root@nginx-web-5865bb968d-759lc:/# curl nginx-web Hello Nginx, nginx-web-5865bb968d-759lc-1.23.0 ``` 默认启用规则 ```powershell 查看资源清单文件 # cat 02_kubernetes_secure_networkpolicy_allow.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-ingress namespace: superopsmsb spec: podSelector: {} policyTypes: ["Ingress", "Egress"] egress: - {} ingress: - {} 应用资源清单文件 # kubectl apply -f 02_kubernetes_secure_networkpolicy_allow.yaml networkpolicy.networking.k8s.io/allow-all-ingress created ``` ```powershell 在default空间访问superopsmsb空间资源 root@nginx-web-5865bb968d-759lc:/# curl nginx-web.superopsmsb.svc.cluster.local Hello Nginx, nginx-web-65d688fd6-h6sbp-1.23.0 root@nginx-web-5865bb968d-759lc:/# ping -c 1 10.244.1.5 PING 10.244.1.5 (10.244.1.5) 56(84) bytes of data. 64 bytes from 10.244.1.5: icmp_seq=1 ttl=63 time=0.181 ms 结果显示： 网络策略成功了，而且多个网络策略可以叠加 注意：仅仅同名策略或者同类型策略可以覆盖 清理网络策略 # kubectl delete -f 02_kubernetes_secure_networkpolicy_allow.yaml ``` 当前namespace的流量限制 ```powershell 查看资源清单文件 # cat 03_kubernetes_secure_networkpolicy_ns.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-current-ns namespace: superopsmsb spec: podSelector: {} policyTypes: ["Ingress", "Egress"] egress: - to: - podSelector: {} ingress: - from: - podSelector: {} 配置解析： 虽然设置了egress和ingress属性，但是下面的podSelector没有选择节点，表示只有当前命名空间所有节点不受限制 应用资源清单文件 # kubectl apply -f 03_kubernetes_secure_networkpolicy_ns.yaml networkpolicy.networking.k8s.io/allow-current-ns created ``` ```powershell default资源访问superopsmsb资源 root@nginx-web-5865bb968d-759lc:/# curl nginx-web.superopsmsb.svc.cluster.local curl: (28) Failed to connect to nginx-web.superopsmsb.svc.cluster.local port 80: Connection timed out root@nginx-web-5865bb968d-759lc:/# ping -c 1 10.244.1.5 PING 10.244.1.5 (10.244.1.5) 56(84) bytes of data. 结果显示： 访问失败 ``` ```powershell superopsmsb资源访问同命名空间的其他资源 root@nginx-web-65d688fd6-h6sbp:/# ping 10.244.1.2 PING 10.244.1.2 (10.244.1.2) 56(84) bytes of data. 64 bytes from 10.244.1.2: icmp_seq=1 ttl=63 time=0.206 ms 结果显示： 同命名空间可以正常访问 ``` **小结** ```powershell ``` ## 2.5 流量管控 学习目标 这一节，我们从 ip限制实践、pod限制实践、小结 三个方面来学习。 **ip限制知识** 准备工作 ```powershell 准备同名空间的两个测试pod 终端1 kubectl run superopsmsb-client1 --image=kubernetes-register.superopsmsb.com/superopsmsb/busybox -n superopsmsb --rm -it -- /bin/sh 终端2 kubectl run superopsmsb-client2 --image=kubernetes-register.superopsmsb.com/superopsmsb/busybox -n superopsmsb --rm -it -- /bin/sh ``` 简介 ```powershell 查看资源清单文件 # cat 04_kubernetes_secure_networkpolicy_ns_pod.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-ns-pod namespace: superopsmsb spec: podSelector: {} policyTypes: ["Ingress", "Egress"] egress: - to: - podSelector: {} ingress: - from: - ipBlock: cidr: 10.244.0.0/16 except: - 10.244.2.0/24 ports: - protocol: TCP port: 80 配置解析： 虽然设置了egress和ingress属性，但是下面的podSelector没有选择节点，表示只有当前命名空间所有节点不受限制 应用资源对象文件 # kubectl apply -f 04_kubernetes_secure_networkpolicy_ns_pod.yaml networkpolicy.networking.k8s.io/deny-ns-pod created ``` ```powershell 3网段测试容器查看效果 / # ifconfig | grep 244 inet addr:10.244.3.7 Bcast:0.0.0.0 Mask:255.255.255.255 / # wget 10.244.1.5 Connecting to 10.244.1.5 (10.244.1.5:80) index.html 100% |*****************************************************| 46 0:00:00 ETA / ``` ```powershell 2网段测试容器查看效果 / # ifconfig | grep 244 inet addr:10.244.2.6 Bcast:0.0.0.0 Mask:255.255.255.255 / # wget 10.244.1.5 Connecting to 10.244.1.5 (10.244.1.5:80) wget: can't connect to remote host (10.244.1.5): Connection timed out / 结果显示： 同namespace可以进行ip级别的访问测试 ``` **pod限制实践** 实践 ```powershell 查看在ip限制的基础上，扩充pod限制资源清单文件 # cat 05_kubernetes_secure_networkpolicy_ns_port.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: egress-controller namespace: superopsmsb spec: podSelector: {} policyTypes: ["Egress"] egress: - to: ports: - protocol: UDP port: 53 - to: ports: - protocol: TCP port: 80 配置解析： 虽然设置了egress和ingress属性，但是下面的podSelector没有选择节点，表示只有当前命名空间所有节点不受限制 应用资源对象文件 # kubectl apply -f 05_kubernetes_secure_networkpolicy_ns_port.yaml networkpolicy.networking.k8s.io/egress-controller created ``` ```powershell 在所有pod测试容器中 / # nslookup nginx-web.default.svc.cluster.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: nginx-web.default.svc.cluster.local Address 1: 10.101.181.105 nginx-web.default.svc.cluster.local / # nslookup nginx-web Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: nginx-web Address 1: 10.105.110.175 nginx-web.superopsmsb.svc.cluster.local 结果显示： 在不影响之前的策略的前提下，扩充了dns的解析功能 ``` **小结** ```powershell ``` ## 2.6 基准测试 学习目标 这一节，我们从 软件简介、简单实践、小结 三个方面来学习 **软件简介** 简介 ```powershell knb 全称 Kubernetes Network Benchmark,它是一个 bash 脚本，它将在目标 Kubernetes 集群上启动网络基准测试。 参考资料： https://github.com/InfraBuilder/k8s-bench-suite ``` ```powershell 它的主要特点如下： 依赖很少的普通 bash 脚本 完成基准测试仅需 2 分钟 能够仅选择要运行的基准测试的子集 测试TCP 和 UDP带宽 自动检测 CNI MTU 在基准报告中包括主机 cpu 和 ram 监控 能够使用 plotly/orca 基于结果数据创建静态图形图像（参见下面的示例） 无需 ssh 访问，只需通过标准 kubectl访问目标集群 不需要高权限，脚本只会在两个节点上启动非常轻量级的 Pod。 它提供了数据的图形生成镜像和配套的服务镜像 ``` 准备工作 ```powershell master节点下载核心服务镜像 docker pull olegeech/k8s-bench-suite 特定node节点下载镜像 docker pull infrabuilder/bench-custom-monitor docker pull infrabuilder/bench-iperf3 docker pull quay.io/plotly/orca ``` 注意事项 ```powershell 由于需要测试k8s节点间的网络通信质量并且将结果输出，所以它在操作的过程中需要提前做一些准备 1 准备k8s集群的认证config文件 2 准备数据文件的映射目录 ``` **简单实践** 命令解析 ```powershell 对于容器内部的knb命令来说，它的基本格式如下： knb --verbose --client-node 客户端节点 --server-node 服务端节点 注意： docker run时候，可以通过NODE_AUTOSELECT从集群中自动选择几个节点来运行测试 - 如果master节点的污点无法容忍，从而导致失败 ``` ```powershell 查看命令帮助 docker run -it --hostname knb --name knb --rm -v /tmp/my-graphs:/my-graphs -v /root/.kube/config:/.kube/config olegeech/k8s-bench-suite --help 注意： 如果网络效果不是很好的话，会因为镜像无法获取而导致失败，需要重试几次即可 所以，如果可以的话，提前下载到特定的工作节点上 ``` 测试效果 ```powershell 生成检测数据 docker run -it --hostname knb --name knb --rm -v /tmp/my-graphs:/my-graphs -v /root/.kube/config:/.kube/config olegeech/k8s-bench-suite --verbose --server-node kubernetes-node1 --client-node kubernetes-node2 -o data -f /my-graphs/mybench.knbdata ``` ```powershell 以json方式显示数据 docker run -it --hostname knb --name knb --rm -v /tmp/my-graphs:/my-graphs -v /root/.kube/config:/.kube/config olegeech/k8s-bench-suite -fd /my-graphs/mybench.knbdata -o json ``` ```powershell 根据数据进行绘图展示 docker run -it --hostname knb --name knb --rm -v /tmp/my-graphs:/my-graphs -v /root/.kube/config:/.kube/config olegeech/k8s-bench-suite -fd /my-graphs/mybench.knbdata --plot --plot-dir /my-graphs/ ``` ```powershell 设定数据的图形样式 docker run -it --hostname knb --name knb --rm -v /tmp/my-graphs:/my-graphs -v /root/.kube/config:/.kube/config olegeech/k8s-bench-suite -fd /my-graphs/mybench.knbdata --plot --plot-args '--width 900 --height 600' --plot-dir /my-graphs ``` **小结** ================================================ FILE: docs/cloud/kubernetes/kubernetes_cluster.md ================================================ # kubernetes集群命令语法 ## kubernetes集群客户端命令 kubectl ## 一、kubectl命令帮助 集群中的管理操作几乎都可以使用`kubectl`命令完成 ```powershell [root@k8s-master1 ~]## kubectl -h ``` ## 二、kubectl命令说明 ![kubernetes集群客户端命令](../../img/kubernetes/kubernetes_cluster/kubectl命令帮助1.png) ![kubernetes集群客户端命令](../../img/kubernetes/kubernetes_cluster/kubectl命令帮助2.png) ## 三、kubectl命令补全 ~~~powershell yum install -y bash-completion source /usr/share/bash-completion/bash_completion source <(kubectl completion bash) kubectl completion bash > ~/.kube/completion.bash.inc source '/root/.kube/completion.bash.inc' source $HOME/.bash_profile ~~~ ## Kubernetes集群Node管理 ## 一、查看集群信息 ~~~powershell [root@k8s-master1 ~]## kubectl cluster-info Kubernetes control plane is running at https://192.168.10.100:6443 CoreDNS is running at https://192.168.10.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ~~~ ## 二、查看节点信息 ### 2.1 查看集群节点信息 ~~~powershell [root@k8s-master1 ~]## kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready 2d20h v1.21.10 k8s-master2 Ready 2d20h v1.21.10 k8s-master3 Ready 2d20h v1.21.10 k8s-worker1 Ready 2d20h v1.21.10 ~~~ ### 2.2 查看集群节点详细信息 ~~~powershell [root@k8s-master1 ~]## kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-master1 Ready 2d20h v1.21.10 192.168.10.12 CentOS Linux 7 (Core) 5.17.0-1.el7.elrepo.x86_64 containerd://1.6.1 k8s-master2 Ready 2d20h v1.21.10 192.168.10.13 CentOS Linux 7 (Core) 5.17.0-1.el7.elrepo.x86_64 containerd://1.6.1 k8s-master3 Ready 2d20h v1.21.10 192.168.10.14 CentOS Linux 7 (Core) 5.17.0-1.el7.elrepo.x86_64 containerd://1.6.1 k8s-worker1 Ready 2d20h v1.21.10 192.168.10.15 CentOS Linux 7 (Core) 5.17.0-1.el7.elrepo.x86_64 containerd://1.6.1 ~~~ ### 2.3 查看节点描述详细信息 ~~~powershell [root@k8s-master1 ~]## kubectl describe node k8s-master1 Name: k8s-master1 Roles: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-master1 kubernetes.io/os=linux Annotations: node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.10.12/24 projectcalico.org/IPv4IPIPTunnelAddr: 10.244.159.128 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 22 Mar 2022 23:47:53 +0800 Taints: Unschedulable: false Lease: HolderIdentity: k8s-master1 AcquireTime: RenewTime: Fri, 25 Mar 2022 20:38:38 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Wed, 23 Mar 2022 00:14:05 +0800 Wed, 23 Mar 2022 00:14:05 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Fri, 25 Mar 2022 20:36:09 +0800 Tue, 22 Mar 2022 23:47:53 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Fri, 25 Mar 2022 20:36:09 +0800 Tue, 22 Mar 2022 23:47:53 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Fri, 25 Mar 2022 20:36:09 +0800 Tue, 22 Mar 2022 23:47:53 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Fri, 25 Mar 2022 20:36:09 +0800 Fri, 25 Mar 2022 00:30:10 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.10.12 Hostname: k8s-master1 Capacity: cpu: 2 ephemeral-storage: 51175Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3994696Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 48294789041 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3892296Ki pods: 110 System Info: Machine ID: a2c5254d78184027930ef5ba59f52d61 System UUID: e9dc4d56-4819-1544-2b93-21af423126d2 Boot ID: e45fcd72-4fc2-45b5-be15-7d944a6b8bcd Kernel Version: 5.17.0-1.el7.elrepo.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.6.1 Kubelet Version: v1.21.10 Kube-Proxy Version: v1.21.10 PodCIDR: 10.244.2.0/24 PodCIDRs: 10.244.2.0/24 Non-terminated Pods: (3 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- default nginx-web-bbh48 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d20h kube-system calico-node-nkxrs 250m (12%) 0 (0%) 0 (0%) 0 (0%) 2d20h kube-system metrics-server-8bb87844c-ptkxm 100m (5%) 0 (0%) 200Mi (5%) 0 (0%) 11h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 350m (17%) 0 (0%) memory 200Mi (5%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: ~~~ ## 三、worker node节点管理集群 * **如果是kubeasz安装，所有节点(包括master与node)都已经可以对集群进行管理** * 如果是kubeadm安装，在node节点上管理时会报如下错误 ~~~powershell [root@k8s-worker1 ~]## kubectl get nodes The connection to the server localhost:8080 was refused - did you specify the right host or port? ~~~ 只要把master上的管理文件`/etc/kubernetes/admin.conf`拷贝到node节点的`$HOME/.kube/config`就可以让node节点也可以实现kubectl命令管理 1, 在node节点的用户家目录创建`.kube`目录 ~~~powershell [root@k8s-worker1 ~]## mkdir /root/.kube ~~~ 2, 在master节点做如下操作 ~~~powershell [root@k8s-worker1 ~]## scp /etc/kubernetes/admin.conf node1:/root/.kube/config ~~~ 3, 在worker node节点验证 ~~~powershell [root@k8s-worker1 ~]## kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready 2d20h v1.21.10 k8s-master2 Ready 2d20h v1.21.10 k8s-master3 Ready 2d20h v1.21.10 k8s-worker1 Ready 2d20h v1.21.10 ~~~ ## 四、节点标签(label) * k8s集群如果由大量节点组成，可将节点打上对应的标签，然后通过标签进行筛选及查看,更好的进行资源对象的相关选择与匹配 ### 4.1 查看节点标签信息 ~~~powershell [root@k8s-master1 ~]## kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master1 Ready 2d20h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master1,kubernetes.io/os=linux k8s-master2 Ready 2d20h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master2,kubernetes.io/os=linux k8s-master3 Ready 2d20h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master3,kubernetes.io/os=linux k8s-worker1 Ready 2d20h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker1,kubernetes.io/os=linux ~~~ ### 4.2 设置节点标签信息 #### 4.2.1 设置节点标签 为节点`k8s-worker1`打一个`region=huanai` 的标签 ~~~powershell [root@k8s-master1 ~]## kubectl label node k8s-worker1 region=huanai node/k8s-worker1 labeled ~~~ #### 4.2.2 查看所有节点标签 ~~~powershell [root@k8s-master1 ~]## kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master1 Ready 2d21h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master1,kubernetes.io/os=linux k8s-master2 Ready 2d21h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master2,kubernetes.io/os=linux k8s-master3 Ready 2d21h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master3,kubernetes.io/os=linux k8s-worker1 Ready 2d21h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker1,kubernetes.io/os=linux,region=huanai ~~~ #### 4.2.3 查看所有节点带region的标签 ~~~powershell [root@k8s-master1 ~]## kubectl get nodes -L region NAME STATUS ROLES AGE VERSION REGION k8s-master1 Ready 2d21h v1.21.10 k8s-master2 Ready 2d21h v1.21.10 k8s-master3 Ready 2d21h v1.21.10 k8s-worker1 Ready 2d21h v1.21.10 huanai ~~~ ### 4.3 多维度标签 #### 4.3.1 设置多维度标签 也可以加其它的多维度标签,用于不同的需要区分的场景 如把`k8s-master3`标签为华南区,A机房,测试环境,游戏业务 ~~~powershell [root@k8s-master1 ~]## kubectl label node k8s-master3 zone=A env=test bussiness=game node/k8s-master3 labeled ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl get nodes k8s-master3 --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master3 Ready 2d21h v1.21.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,bussiness=game,env=test,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master3,kubernetes.io/os=linux,zone=A ~~~ #### 4.3.2 显示节点的相应标签 ~~~powershell [root@k8s-master1 ~]## kubectl get nodes -L region,zone NAME STATUS ROLES AGE VERSION REGION ZONE k8s-master1 Ready 2d21h v1.21.10 k8s-master2 Ready 2d21h v1.21.10 k8s-master3 Ready 2d21h v1.21.10 A k8s-worker1 Ready 2d21h v1.21.10 huanai ~~~ #### 4.3.3 查找`region=huanai`的节点 ~~~powershell [root@k8s-master1 ~]## kubectl get nodes -l region=huanai NAME STATUS ROLES AGE VERSION k8s-worker1 Ready 2d21h v1.21.10 ~~~ #### 4.3.4 标签的修改 ~~~powershell [root@k8s-master1 ~]## kubectl label node k8s-master3 bussiness=ad --overwrite=true node/k8s-master3 labeled 加上--overwrite=true覆盖原标签的value进行修改操作 ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl get nodes -L bussiness NAME STATUS ROLES AGE VERSION BUSSINESS k8s-master1 Ready 2d21h v1.21.10 k8s-master2 Ready 2d21h v1.21.10 k8s-master3 Ready 2d21h v1.21.10 ad k8s-worker1 Ready 2d21h v1.21.10 ~~~ #### 4.3.5 标签的删除 使用key加一个减号的写法来取消标签 ~~~powershell [root@k8s-master1 ~]## kubectl label node k8s-worker1 region- node/k8s-worker1 labeled ~~~ #### 4.3.6 标签选择器 标签选择器主要有2类: * 等值关系: =, != * 集合关系: KEY in {VALUE1, VALUE2......} ~~~powershell [root@k8s-master1 ~]## kubectl label node k8s-master2 env=test1 node/k8s-master2 labeled [root@k8s-master1 ~]## kubectl label node k8s-master3 env=test2 node/k8s-master3 labeled ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl get node -l "env in(test1,test2)" NAME STATUS ROLES AGE VERSION k8s-master2 Ready 2d21h v1.21.10 k8s-master3 Ready 2d21h v1.21.10 ~~~ ## Kubernetes集群声明式文件YAML ## 一、YAML介绍 YAML 的意思是：仍是一种标记语言，但为了强调这种语言以数据做为中心，而不是以标记语言为重点。是一个可读性高，用来表达数据序列的格式。 ## 二、基本语法 ```powershell 1.低版本缩进时不允许使用Tab键，只允许使用空格 2.缩进的空格数目不重要，只要相同层级的元素左侧对齐即可 3.## 标识注释，从这个字符一直到行尾，都会被解释器忽略 ``` ## 三、YAML 支持的数据结构 - 对象：键值对的集合，又称为映射（mapping）/ 哈希（hashes） / 字典（dictionary） - 数组：一组按次序排列的值，又称为序列（sequence） / 列表 （list） - 纯量（scalars）：单个的、不可再分的值 ####### 对象类型：对象的一组键值对，使用冒号结构表示 ```powershell name: Steve age: 18 ``` Yaml 也允许另一种写法，将所有键值对写成一个行内对象 ```powershell hash: { name: Steve, age: 18 } ``` ####### 数组类型：一组连词线开头的行，构成一个数组 ```powershell animal - Cat - Dog ``` 数组也可以采用行内表示法 ```powershell animal: [Cat, Dog] ``` ####### 复合结构：对象和数组可以结合使用，形成复合结构 ```powershell 1 languages: 2 - Ruby 3 - Perl 4 - Python 5 websites: 6 YAML: yaml.org 7 Ruby: ruby-lang.org 8 Python: python.org 9 Perl: use.perl.org ``` ####### **纯量：纯量是最基本的、不可再分的值。以下数据类型都属于纯量** ```powershell 1 字符串 布尔值 整数 浮点数 Null 2 时间 日期 数值直接以字面量的形式表示 number: 12.30 布尔值用true和false表示 isSet: true null用 ~ 表示 parent: ~ 时间采用 ISO8601 格式 iso8601: 2001-12-14t21:59:43.10-05:00 日期采用复合 iso8601 格式的年、月、日表示 date: 1976-07-31 YAML 允许使用两个感叹号，强制转换数据类型 e: !!str 123 f: !!str true ``` ####### 字符串 字符串默认不使用引号表示 ```powershell str: 这是一行字符串 ``` 如果字符串之中包含空格或特殊字符，需要放在引号之中 ```power str: '内容： 字符串' ``` 单引号和双引号都可以使用，双引号不会对特殊字符转义 ```powershell s1: '内容\n字符串' s2: "内容\n字符串" ``` 单引号之中如果还有单引号，必须连续使用两个单引号转义 ```powershell str: 'labor''s day' ``` 字符串可以写成多行，从第二行开始，必须有一个单空格缩进。换行符会被转为 空格 ```powershell str: 这是一段 多行 字符串 ``` 多行字符串可以使用|保留换行符，也可以使用>折叠换行 ```powershell this: | Foo Bar that Foo Bar ``` ## 四、Kubernetes集群中资源对象描述方法 在kubernetes中，一般使用ymal格式的文件来创建符合我们预期期望的pod,这样的yaml文件称为资源清单文件。 ### 4.1 常用字段 | 参数名 | 字段类型 | 说明 | | ------------------------------------------- | -------- | ------------------------------------------------------------ | | version | String | 这里是指的是K8S API的版本，目前基本上是v1，可以用 kubectl api-versions命令查询 | | kind | String | 这里指的是yam文件定义的资源类型和角色，比如:Pod | | metadata | Object | 元数据对象，固定值就写 metadata | | metadata.name | String | 元数据对象的名字，这里由我们编写，比如命名Pod的名字 | | metadata.namespace | String | 元数据对象的命名空间，由我们自身定义 | | Spec | Object | 详细定义对象，固定值就写Spec | | spec. containers[] | list | 这里是Spec对象的容器列表定义，是个列表 | | spec containers [].name | String | 这里定义容器的名字 | | spec.containers [].image | String | 这里定义要用到的镜像名称 | | spec.containers [].imagePullPolicy | String | 定义镜像拉取策路，有 Always、 Never、Ifnotpresent三个值可选：(1) Always:意思是每次都尝试重新拉取镜像；(2) Never:表示仅使用本地镜像；(3) IfNotPresent:如果本地有镜像就使用本地镜像，没有就拉取在线镜像。上面三个值都没设置的话，默认是 Always。 | | spec containers [].command[] | List | 指定容器启动命令，因为是数组可以指定多个。不指定则使用镜像打包时使用的启动命令。 | | spec.containers [].args | List | 指定容器启动命令参数，因为是数组可以指定多个. | | spec.containers [].workDir | String | 指定容器的工作目录 | | spec.containers[]. volumeMounts[] | List | 指定容器内部的存储卷配置 | | spec.containers[]. volumeMounts[].name | String | 指定可以被容器挂载的存储卷的名称 | | spec.containers[]. volumeMounts[].mountPath | String | 指定可以被容器挂载的存储卷的路径 | | spec.containers[]. volumeMounts[].readOnly | String | 设置存储卷路径的读写模式，ture或者 false，默认为读写模式 | | spec.containers [].ports[] | String | 指容器需要用到的端口列表 | | spec.containers [].ports[].name | String | 指定端口名称 | | spec.containers [].ports[].containerPort | String | 指定容器需要监听的端口号 | | spec.containers [].ports[].hostPort | String | 指定容器所在主机需要监听的端口号，默认跟上面 containerPort相同，注意设置了 hostPort同一台主机无法启动该容器的相同副本(因为主机的端口号不能相同，这样会冲突) | | spec.containers [].ports[].protocol | String | 指定端口协议，支持TCP和UDP，默认值为TCP | | spec.containers [].env[] | List | 指定容器运行前需设的环境变量列表 | | spec.containers [].env[].name | String | 指定环境变量名称 | | spec.containers [].env[].value | String | 指定环境变量值 | | spec.containers[].resources | Object | 指定资源 限制和资源请求的值（这里开始就是设置容器的资源上限） | | spec.containers[].resources.limits | Object | 指定设置容器运行时资源的运行上限 | | spec.containers[].resources.limits.cpu | String | 指定CPU限制，单位为core数，将用于docker run -- cpu-shares参数 | | spec.containers[].resources.limits.memory | String | 指定MEM内存的限制，单位为MiB、GiB | | spec.containers[].resources.requests | Object | 指定容器启动和调度时的限制设置 | | spec.containers[].resources.requests.cpu | String | CPU请求，单位为core数，容器启动时初始化可用数量 | | spec.containers[].resources.requests.memory | String | 内存请求，单位为MiB、GiB，容器启动时初始化可用数量 | | sepc.restartPolicy | String | 定义Pod的重启策略，可选值为Always、OnFailure,默认值为Always。1.Always:Pod一旦终止运行，则无论容器时如何终止的，kubelet服务都将重启它。2.OnFailure:只有Pod以非零退出码终止时，kubelet才会重启该容器。如果容器正常结束（退出码为0），则kubelet将不会重启它。3.Never:Pod终止后，kubelet将退出码报告给Master,不会重启该Pod。 | | spec.nodeSelector | Object | 定义Node的Label过滤标签，以key:value格式指定。 | | spec.imagePullSecrets | Object | 定义pull镜像时使用secret名称，以name:secretkey格式指定。 | | spec.hostNetwork | Boolean | 定义是否使用主机网络模式，默认值为false。设置true表示使用宿主机网络，不使用docker网桥，同时设置了true将无法在同一台宿主机上启动第二个副本。 | ### 4.2 举例说明 #### 4.2.1 使用声明式文件YAML创建namespace ~~~powershell apiVersion: v1 kind: Namespace metadata: name: test ~~~ #### 4.2.2 使用声明式文件YAML创建pod ~~~powershell apiVersion: v1 kind: Pod metadata: name: pod1 spec: containers: - name: k8sonline1 image: nginx:latest imagePullPolicy: IfNotPresent ~~~ ## Kubernetes集群命名空间（Namespace） ## 一、命名空间(namespace)作用 * Namespace是对一组资源和对象的抽象集合. * 常见的 pod, service, deployment 等都是属于某一个namespace的（默认是 default） * 不是所有资源都属于namespace，如nodes, persistent volume，namespace 等资源则不属于任何 namespace ## 二、查看namespace ~~~powershell [root@k8s-master1 ~]## kubectl get namespaces ## namespaces可以简写为namespace或ns NAME STATUS AGE default Active 130m ## 所有未指定Namespace的对象都会被默认分配在default命名空间 kube-node-lease Active 130m kube-public Active 130m ## 此命名空间下的资源可以被所有人访问 kube-system Active 130m ## 所有由Kubernetes系统创建的资源都处于这个命名空间 ~~~ ## 三、查看namespace里的资源 **使用`kubectl get all --namespace=命名空间名称`可以查看此命名空间下的所有资源** ~~~powershell [root@k8s-master1 ~]## kubectl get all --namespace=kube-system NAME READY STATUS RESTARTS AGE pod/calico-kube-controllers-7fdc86d8ff-cskfq 1/1 Running 3 5d1h pod/calico-node-9dpc9 1/1 Running 2 5d1h pod/calico-node-jdmxw 1/1 Running 3 5d1h pod/calico-node-krwps 1/1 Running 2 5d1h pod/calico-node-tttlr 1/1 Running 2 5d1h pod/coredns-65dbdb44db-mm7cr 1/1 Running 2 5d1h pod/dashboard-metrics-scraper-545bbb8767-q66bc 1/1 Running 2 5d1h pod/kubernetes-dashboard-65665f84db-nll6k 1/1 Running 4 5d1h pod/metrics-server-869ffc99cd-8f4jd 1/1 Running 3 5d1h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dashboard-metrics-scraper ClusterIP 10.2.246.128 8000/TCP 5d1h service/kube-dns ClusterIP 10.2.0.2 53/UDP,53/TCP,9153/TCP 5d1h service/kubernetes-dashboard NodePort 10.2.213.30 443:21351/TCP 5d1h service/metrics-server ClusterIP 10.2.232.121 443/TCP 5d1h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/calico-node 4 4 4 4 4 beta.kubernetes.io/os=linux 5d1h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/calico-kube-controllers 1/1 1 1 5d1h deployment.apps/coredns 1/1 1 1 5d1h deployment.apps/dashboard-metrics-scraper 1/1 1 1 5d1h deployment.apps/kubernetes-dashboard 1/1 1 1 5d1h deployment.apps/metrics-server 1/1 1 1 5d1h NAME DESIRED CURRENT READY AGE replicaset.apps/calico-kube-controllers-7fdc86d8ff 1 1 1 5d1h replicaset.apps/coredns-65dbdb44db 1 1 1 5d1h replicaset.apps/dashboard-metrics-scraper-545bbb8767 1 1 1 5d1h replicaset.apps/kubernetes-dashboard-65665f84db 1 1 1 5d1h replicaset.apps/metrics-server-869ffc99cd 1 1 1 5d1h ~~~ **使用`kubectl get 资源类型 --namespace=命名空间名称`可以查看此命名空间下的对应的资源** ~~~powershell [root@k8s-master1 ~]## kubectl get pod --namespace=kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-7fdc86d8ff-cskfq 1/1 Running 3 5d1h calico-node-9dpc9 1/1 Running 2 5d1h calico-node-jdmxw 1/1 Running 3 5d1h calico-node-krwps 1/1 Running 2 5d1h calico-node-tttlr 1/1 Running 2 5d1h coredns-65dbdb44db-mm7cr 1/1 Running 2 5d1h dashboard-metrics-scraper-545bbb8767-q66bc 1/1 Running 2 5d1h kubernetes-dashboard-65665f84db-nll6k 1/1 Running 4 5d1h metrics-server-869ffc99cd-8f4jd 1/1 Running 3 5d1h ~~~ ## 四、创建namespace ### 4.1 命令创建 ~~~powershell [root@k8s-master1 ~]## kubectl create namespace ns1 namespace/ns1 created [root@k8s-master1 ~]## kubectl get ns NAME STATUS AGE default Active 5d1h kube-node-lease Active 5d1h kube-public Active 5d1h kube-system Active 5d1h ns1 Active 10s ~~~ ### 4.2 YAML文件创建 * k8s中几乎所有的资源都可以通这YAML编排来创建 * 可以使用`kubectl edit 资源类型 资源名`编辑资源的YAML语法 ~~~powershell [root@k8s-master1 ~]## kubectl edit namespace ns1 ...... ~~~ * 也可使用`kubectl get 资源类型 资源名 -o yaml`来查看 ~~~powershell [root@k8s-master1 ~]## kubectl get ns ns1 -o yaml ...... ~~~ * ==**还可通过`kubectl explain 资源类型`来查看语法文档**== ~~~powershell [root@k8s-master1 ~]## kubectl explain namespace ## 查看namespace相关语法参数 ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl explain namespace.metadata ## 查看namespace下级metadata的相关语法参数 ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl explain namespace.metadata.name ## 查看namespace下级metadata再下级name的相关语法参数 ~~~ 编写创建namespace的YAML文件 ~~~powershell [root@k8s-master1 ~]## vim create_ns2.yml apiVersion: v1 ## api版本号 kind: Namespace ## 类型为namespace metadata: ## 定义namespace的元数据属性 name: ns2 ## 定义name属性为ns2 ~~~ 使用`kubctl apply -f`应用YAML文件 ~~~powershell [root@k8s-master1 ~]## kubectl apply -f create_ns2.yml namespace/ns2 created ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl get ns NAME STATUS AGE default Active 5d2h kube-node-lease Active 5d2h kube-public Active 5d2h kube-system Active 5d2h ns1 Active 10m ns2 Active 46s ~~~ ## 五、删除namespace **注意:** * 删除一个namespace会自动删除所有属于该namespace的资源(类似mysql中drop库会删除库里的所有表一样，请慎重操作) * default,kube-system,kube-public命名空间不可删除 ### 5.1 命令删除 ~~~powershell [root@k8s-master1 ~]## kubectl delete namespace ns1 namespace "ns1" deleted ~~~ ### 5.2 YAML文件删除 ~~~powershell [root@k8s-master1 ~]## kubectl delete -f create_ns2.yml namespace "ns2" deleted ~~~ ~~~powershell [root@k8s-master1 ~]## kubectl get ns NAME STATUS AGE default Active 5d2h kube-node-lease Active 5d2h kube-public Active 5d2h kube-system Active 5d2h ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_cluster_serve.md ================================================ # Kubernetes集群公共服务 # 一、Kubernetes集群核心服务 ![image-20220512190004712](../../img/kubernetes/kubernetes_cluster_serve/image-20220512190004712.png) # 二、Kubernetes公共服务 ![image-20220512190022889](../../img/kubernetes/kubernetes_cluster_serve/image-20220512190022889.png) ![image-20220512190046609](../../img/kubernetes/kubernetes_cluster_serve/image-20220512190046609.png) # 三、Kubernetes集群公共服务 域名解析 DNS ## 3.1 主机IP地址及域名规划 | 序号 | 提供服务 | IP地址 | 域名 | 备注 | | ---- | -------- | -------------- | ------------------ | --------------- | | 1 | DNS | 192.168.10.211 | | | | 2 | Nginx | 192.168.10.212 | yaml.kubemsb.com | | | 3 | Harbor | 192.168.10.213 | harbor.kubemsb.com | www.kubemsb.com | | 4 | NFS | 192.168.10.214 | nfs.kubemsb.com | | ## 3.2 域名解析 DNS配置 - bind9 ~~~powershell [root@localhost ~]# hostnamectl set-hostname dns ~~~ ~~~powershell [root@dns ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="static" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" IPADDR="192.168.10.211" PREFIX="24" GATEWAY="192.168.10.2" DNS1="192.168.10.211" DNS2="119.29.29.29" ~~~ ~~~powershell [root@dns ~]# firewall-cmd --state not running [root@dns ~]# echo $? 252 [root@dns ~]# sestatus SELinux status: disabled [root@dns ~]# echo $? 0 或关闭 [root@dns ~]# systemctl disable firewalld;systemctl stop firewalld [root@dns ~]# sed -ri.bak 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ~~~powershell [root@dns ~]# yum -y install bind ~~~ ~~~powershell [root@dns ~]# cat -n /etc/named.conf 1 // 2 // named.conf 3 // 4 // Provided by Red Hat bind package to configure the ISC BIND named(8) DNS 5 // server as a caching only nameserver (as a localhost DNS resolver only). 6 // 7 // See /usr/share/doc/bind*/sample/ for example named configuration files. 8 // 9 // See the BIND Administrator's Reference Manual (ARM) for details about the 10 // configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html 11 12 options { 13 listen-on port 53 { 127.0.0.1;any; }; 14 listen-on-v6 port 53 { ::1; }; 15 directory "/var/named"; 16 dump-file "/var/named/data/cache_dump.db"; 17 statistics-file "/var/named/data/named_stats.txt"; 18 memstatistics-file "/var/named/data/named_mem_stats.txt"; 19 recursing-file "/var/named/data/named.recursing"; 20 secroots-file "/var/named/data/named.secroots"; 21 allow-query { localhost;any; }; 22 23 /* 24 - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion. 25 - If you are building a RECURSIVE (caching) DNS server, you need to enable 26 recursion. 27 - If your recursive DNS server has a public IP address, you MUST enable access 28 control to limit queries to your legitimate users. Failing to do so will 29 cause your server to become part of large scale DNS amplification 30 attacks. Implementing BCP38 within your network would greatly 31 reduce such attack surface 32 */ 33 recursion yes; 34 35 dnssec-enable yes; 36 dnssec-validation yes; 37 38 /* Path to ISC DLV key */ 39 bindkeys-file "/etc/named.root.key"; 40 41 managed-keys-directory "/var/named/dynamic"; 42 43 pid-file "/run/named/named.pid"; 44 session-keyfile "/run/named/session.key"; 45 }; 46 47 logging { 48 channel default_debug { 49 file "data/named.run"; 50 severity dynamic; 51 }; 52 }; 53 54 zone "." IN { 55 type hint; 56 file "named.ca"; 57 }; 58 59 include "/etc/named.rfc1912.zones"; 60 include "/etc/named.root.key"; ~~~ ~~~powershell [root@dns ~]# tail -6 /etc/named.rfc1912.zones zone "kubemsb.com" IN { type master; file "kubemsb.com.zone"; allow-update { none; }; }; ~~~ ~~~powershell [root@dns ~]# cd /var/named/ [root@dns named]# ls data dynamic named.ca named.empty named.localhost named.loopback slaves [root@dns named]# cp -p named.localhost kubemsb.com.zone [root@dns named]# ll 总用量 20 drwxrwx--- 2 named named 6 4月 7 22:41 data drwxrwx--- 2 named named 6 4月 7 22:41 dynamic -rw-r----- 1 root named 2253 4月 5 2018 named.ca -rw-r----- 1 root named 152 12月 15 2009 named.empty -rw-r----- 1 root named 152 6月 21 2007 named.localhost -rw-r----- 1 root named 168 12月 15 2009 named.loopback drwxrwx--- 2 named named 6 4月 7 22:41 slaves -rw-r----- 1 root named 152 6月 21 2007 kubemsb.com.zone ~~~ ~~~powershell [root@dns named]# cat kubemsb.com.zone $TTL 1D @ IN SOA kubemsb.com admin.kubemsb.com. ( 0 ; serial 1D ; refresh 1H ; retry 1W ; expire 3H ) ; minimum @ NS ns.kubemsb.com. ns A 192.168.10.211 yaml A 192.168.10.212 harbor A 192.168.10.213 nfs A 192.168.10.214 ~~~ ~~~powershell [root@dns named]# systemctl enable named [root@dns named]# systemctl start named ~~~ ## 3.3 域名解析 DNS解析验证 ~~~powershell [root@dns named]# yum -y install bind-utils ~~~ ~~~powershell [root@dns named]# nslookup > server Default server: 192.168.10.211 Address: 192.168.10.211#53 Default server: 119.29.29.29 Address: 119.29.29.29#53 > ns.kubemsb.com Server: 192.168.10.211 Address: 192.168.10.211#53 Name: ns.kubemsb.com Address: 192.168.10.211 > yaml.kubemsb.com Server: 192.168.10.211 Address: 192.168.10.211#53 Name: yaml.kubemsb.com Address: 192.168.10.212 ~~~ # 四、Kubernetes集群公共服务 YAML资源清单文件托管服务 Nginx ~~~powershell [root@localhost ~]# hostnamectl set-hostname nginx ~~~ ~~~powershell [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="static" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" IPADDR="192.168.10.212" PREFIX="24" GATEWAY="192.168.10.2" DNS1="192.168.10.211" DNS2="119.29.29.29" ~~~ ~~~powershell [root@nginx ~]# firewall-cmd --state not running [root@nginx ~]# sestatus SELinux status: disabled ~~~ ![image-20200512094822918](../../img/kubernetes/kubernetes_cluster_serve/image-20200512094822918.png) ![image-20200512094918480](../../img/kubernetes/kubernetes_cluster_serve/image-20200512094918480.png) ~~~powershell [root@nginx ~]# wget http://nginx.org/download/nginx-1.18.0.tar.gz ~~~ ~~~powershell [root@nginx ~]# ls soft/ echo-nginx-module-0.61.tar.gz nginx-1.18.0.tar.gz ngx-fancyindex-0.4.3.tar.gz ~~~ ~~~powershell [root@nginx ~]# yum -y install gcc prce-devel zlib-devel openssl-devel ~~~ ~~~powershell [root@nginx ~]# cd soft/ [root@nginx soft]# ls echo-nginx-module-0.61.tar.gz nginx-1.18.0.tar.gz ngx-fancyindex-0.4.3.tar.gz [root@nginx soft]# tar xf ngx-fancyindex-0.4.3.tar.gz [root@nginx soft]# tar xf nginx-1.18.0.tar.gz [root@nginx soft]# tar xf echo-nginx-module-0.61.tar.gz [root@nginx soft]# ls echo-nginx-module-0.61 nginx-1.18.0 ngx-fancyindex-0.4.3 echo-nginx-module-0.61.tar.gz nginx-1.18.0.tar.gz ngx-fancyindex-0.4.3.tar.gz ~~~ ~~~powershell [root@nginx nginx-1.18.0]# ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --add-module=/root/soft/ngx-fancyindex-0.4.3/ --add-module=/root/soft/echo-nginx-module-0.61 ~~~ ~~~powershell [root@nginx nginx-1.18.0]# make && make install ~~~ ~~~powershell [root@nginx soft]# cat nginx_manager.sh #!/usr/bin/bash # nginx manager # 定义变量 nginx_base="/usr/local/nginx" nginxd="$nginx_base/sbin/nginx" nginx_pid="$nginx_base/logs/nginx.pid" # 调用系统函数，为输出文字添加颜色 if [ -f /etc/init.d/functions ];then source /etc/init.d/functions fi # 检测nginx进程是否存在及进程数量是否正常 if [ -f $nginx_pid ];then nginx_process_pid=`cat $nginx_pid` nginx_process_num=`ps aux | grep "$nginx_process_pid" | wc -l` fi # 封装功能 start () { if [ -f $nginx_pid ] && [ $nginx_process_num -ge 2 ];then echo "nginx already start" elif [ ! -f $nginx_pid ] || [ -z "$nginx_process_num" ];then action "nginx start" $nginxd fi } stop () { if [ -f $nginx_pid ] && [ $nginx_process_num -ge 2 ];then action "nginx stop" kill -s QUIT $nginx_process_pid else echo "nginx already stop" fi } status () { if [ -f $nginx_pid ] && [ $nginx_process_num -ge 2 ];then echo "nginx running" else echo "nginx stopped" fi } reload () { if [ -f $nginx_pid ] && [ $nginx_process_num -ge 2 ];then kill -s HUP $nginx_process_pid else echo "nginx stopped" fi } # 调用函数 case $1 in start) start ;; stop) stop ;; restart) stop sleep 1 start ;; status) status ;; reload) reload ;; *) echo "命令用法: $0 start|stop|restart|status|reload" esac ~~~ ~~~powershell [root@nginx ~]# cat /usr/local/nginx/conf/nginx.conf #user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; server { listen 80; server_name 192.168.10.212; #charset koi8-r; #access_log logs/host.access.log main; root html; location / { fancyindex on; fancyindex_exact_size off; index index; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } # proxy the PHP scripts to Apache listening on 127.0.0.1:80 # #location ~ \.php$ { # proxy_pass http://127.0.0.1; #} # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php$ { # root html; # fastcgi_pass 127.0.0.1:9000; # fastcgi_index index.php; # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; # include fastcgi_params; #} # deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} } # another virtual host using mix of IP-, name-, and port-based configuration # #server { # listen 8000; # listen somename:8080; # server_name somename alias another.alias; # location / { # root html; # index index.html index.htm; # } #} # HTTPS server # #server { # listen 443 ssl; # server_name localhost; # ssl_certificate cert.pem; # ssl_certificate_key cert.key; # ssl_session_cache shared:SSL:1m; # ssl_session_timeout 5m; # ssl_ciphers HIGH:!aNULL:!MD5; # ssl_prefer_server_ciphers on; # location / { # root html; # index index.html index.htm; # } #} } ~~~ > ![image-20200512102547222](../../img/kubernetes/kubernetes_cluster_serve/image-20200512102547222.png) ~~~powershell [root@nginx ~]# cat /etc/rc.d/rc.local #!/bin/bash # THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES # # It is highly advisable to create own systemd services or udev rules # to run scripts during boot instead of using this file. # # In contrast to previous versions due to parallel execution during boot # this script will NOT be run after all other services. # # Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure # that this script will be executed during boot. touch /var/lock/subsys/local /root/soft/nginx_manager.sh start ~~~ ~~~powershell [root@nginx ~]# ll /etc/rc.d/rc.local -rw-r--r-- 1 root root 507 5月 12 10:27 /etc/rc.d/rc.local [root@nginx ~]# chmod 744 /etc/rc.d/rc.local [root@nginx ~]# ll /etc/rc.d/rc.local -rwxr--r-- 1 root root 507 5月 12 10:27 /etc/rc.d/rc.local ~~~ # 五、Kubernetes集群容器镜像仓库 Harbor ## 5.0 主机名及IP地址配置 ~~~powershell [root@localhost ~]# hostnamectl set-hostname harbor ~~~ ~~~powershell [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="static" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" IPADDR="192.168.10.213" PREFIX="24" GATEWAY="192.168.10.2" DNS1="192.168.10.211" DNS2="119.29.29.29" ~~~ ## 5.1 docker ce安装 ### 5.1.1 Docker安装YUM源准备 >使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ### 5.1.2 Docker安装 ~~~powershell # yum -y install docker-ce ~~~ ### 5.1.3 启动Docker服务 ~~~powershell # systemctl enable --now docker ~~~ ## 5.2 获取 docker compose二进制文件 ~~~powershell 下载docker-compose二进制文件 # wget https://github.com/docker/compose/releases/download/1.25.0/docker-compose-Linux-x86_64 ~~~ ~~~powershell 查看已下载二进制文件 # ls docker-compose-Linux-x86_64 ~~~ ~~~powershell 移动二进制文件到/usr/bin目录，并更名为docker-compose # mv docker-compose-Linux-x86_64 /usr/bin/docker-compose ~~~ ~~~powershell 为二进制文件添加可执行权限 # chmod +x /usr/bin/docker-compose ~~~ ~~~powershell 安装完成后，查看docker-compse版本 # docker-compose version docker-compose version 1.25.0, build 0a186604 docker-py version: 4.1.0 CPython version: 3.7.4 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019 ~~~ ## 5.3 获取harbor安装文件 ![image-20220125232445910](../../img/kubernetes/kubernetes_cluster_serve/image-20220125232445910.png) ![image-20220125232519365](../../img/kubernetes/kubernetes_cluster_serve/image-20220125232519365.png) ![image-20220125233602760](../../img/kubernetes/kubernetes_cluster_serve/image-20220125233602760.png) ![image-20220125233652604](../../img/kubernetes/kubernetes_cluster_serve/image-20220125233652604.png) ![image-20220125233739356](../../img/kubernetes/kubernetes_cluster_serve/image-20220125233739356.png) ~~~powershell 下载harbor离线安装包 # wget https://github.com/goharbor/harbor/releases/download/v2.4.1/harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell 查看已下载的离线安装包 # ls harbor-offline-installer-v2.4.1.tgz ~~~ ## 5.4 获取TLS文件 ~~~powershell 查看准备好的证书 # ls kubemsb.com_nginx.zip ~~~ ~~~powershell 解压证书压缩包文件 # unzip kubemsb.com_nginx.zip Archive: kubemsb.com_nginx.zip Aliyun Certificate Download inflating: 6864844_kubemsb.com.pem inflating: 6864844_kubemsb.com.key ~~~ ~~~powershell 查看解压出的文件 # ls 6864844_kubemsb.com.key 6864844_kubemsb.com.pem ~~~ ## 5.5 修改配置文件 ~~~powershell 解压harbor离线安装包 # tar xf harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell 查看解压出来的目录 # ls harbor ~~~ ~~~powershell 移动证书到harbor目录 # # mv 6864844_kubemsb.com.* harbor 查看harbor目录 # ls harbor 6864844_kubemsb.com.key 6864844_kubemsb.com.pem common.sh harbor.v2.4.1.tar.gz harbor.yml.tmpl install.sh LICENSE prepare ~~~ ~~~powershell 创建配置文件 # cd harbor/ # mv harbor.yml.tmpl harbor.yml ~~~ ~~~powershell 修改配置文件内容 # vim harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: www.kubemsb.com 修改为域名，而且一定是证书签发的域名 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /root/harbor/6864844_kubemsb.com.pem 证书 private_key: /root/harbor/6864844_kubemsb.com.key 密钥 # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: 12345 访问密码 ...... ~~~ ## 5.6 执行预备脚本 ~~~powershell # ./prepare ~~~ ~~~powershell 输出 prepare base dir is set to /root/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /data/secret/keys/secretkey Successfully called func: create_root_cert Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir ~~~ ## 5.7 执行安装脚本 ~~~powershell # ./install.sh ~~~ ~~~powershell 输出 [Step 0]: checking if docker is installed ... Note: docker version: 20.10.12 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.25.0 [Step 2]: loading Harbor images ... [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... prepare base dir is set to /root/harbor [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating registry ... done Creating registryctl ... done Creating redis ... done Creating harbor-portal ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ✔ ----Harbor has been installed and started successfully.---- ~~~ ## 5.8 验证运行情况 ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 71c0db683e4a goharbor/nginx-photon:v2.4.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp nginx 4e3b53a86f01 goharbor/harbor-jobservice:v2.4.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice df76e1eabbf7 goharbor/harbor-core:v2.4.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core eeb4d224dfc4 goharbor/harbor-portal:v2.4.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal 70e162c38b59 goharbor/redis-photon:v2.4.1 "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis 8bcc0e9b06ec goharbor/harbor-registryctl:v2.4.1 "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl d88196398df7 goharbor/registry-photon:v2.4.1 "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry ed5ba2ba9c82 goharbor/harbor-db:v2.4.1 "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db dcb4b57c7542 goharbor/harbor-log:v2.4.1 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log ~~~ ## 5.9 访问harbor UI界面 ### 5.9.1 在物理机通过浏览器访问 ![image-20220126000804490](../../img/kubernetes/kubernetes_cluster_serve/image-20220126000804490.png) ![image-20220126000825616](../../img/kubernetes/kubernetes_cluster_serve/image-20220126000825616.png) ![image-20220126000840905](../../img/kubernetes/kubernetes_cluster_serve/image-20220126000840905.png) ### 5.9.2 在虚拟机中通过浏览器访问 ![image-20220126001253192](../../img/kubernetes/kubernetes_cluster_serve/image-20220126001253192.png) ![image-20220126001447862](../../img/kubernetes/kubernetes_cluster_serve/image-20220126001447862.png) ![image-20220126001510880](../../img/kubernetes/kubernetes_cluster_serve/image-20220126001510880.png) ## 5.10 本地docker daemon配置使用本地容器镜像仓库 ~~~powershell [root@harbor ~]# vim /etc/docker/daemon.json [root@harbor ~]# cat /etc/docker/daemon.json { "insecure-registries": ["http://www.kubemsb.com"] } ~~~ ~~~powershell [root@harbor ~]# systemctl restart docker ~~~ ~~~powershell [root@harbor ~]# docker login www.kubemsb.com Username: admin Password: 12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ## 5.11 Kubernetes集群所有节点配置使用本地容器镜像仓库 ~~~powershell [root@k8s-* ~]# vim /etc/docker/daemon.json [root@k8s-* ~]# cat /etc/docker/daemon.json { "insecure-registries": ["http://www.kubemsb.com"] } ~~~ ~~~powershell [root@k8s-* ~]# systemctl restart docker ~~~ ~~~powershell [root@k8s-* ~]# docker login www.kubemsb.com Username: admin Password: 12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ # 六、Kubernetes集群数据持久化存储服务 网络文件系统 NFS ## 6.1 NFS服务准备 ~~~powershell [root@localhost ~]# hostnamectl set-hostname nfs ~~~ ~~~powershell [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="static" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" IPADDR="192.168.10.214" PREFIX="24" GATEWAY="192.168.10.2" DNS1="192.168.10.211" DNS2="119.29.29.29" ~~~ ~~~powershell [root@nfs ~]# firewall-cmd --state not running [root@nfs ~]# sestatus SELinux status: disabled ~~~ ~~~powershell [root@nfs ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 100G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 99G 0 part ├─centos_192-root 253:0 0 50G 0 lvm / ├─centos_192-swap 253:1 0 3.9G 0 lvm [SWAP] └─centos_192-home 253:2 0 45.1G 0 lvm /home sdb 252:16 0 100G 0 disk ~~~ ~~~powershell [root@nfs ~]# mkfs.xfs /dev/sdb meta-data=/dev/vdb isize=512 agcount=4, agsize=6553600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=26214400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=12800, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@nfs ~]# mkdir /sdb [root@nfs ~]# echo -e "/dev/sdb\t/sdb\txfs\tdefaults\t0 0" >> /etc/fstab [root@nfs ~]# tail -1 /etc/fstab /dev/vdb /vdb xfs defaults 0 0 [root@nfs ~]# mount -a [root@nfs ~]# df -Th 文件系统 类型 容量 已用 可用 已用% 挂载点 devtmpfs devtmpfs 2.0G 0 2.0G 0% /dev tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs tmpfs 2.0G 8.6M 2.0G 1% /run tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/mapper/centos_192-root xfs 50G 2.1G 48G 5% / /dev/mapper/centos_192-home xfs 46G 33M 46G 1% /home /dev/vda1 xfs 1014M 163M 852M 16% /boot tmpfs tmpfs 250M 0 250M 0% /run/user/0 /dev/sdb xfs 100G 33M 100G 1% /vdb ~~~ ~~~powershell [root@nfs ~]# yum -y install nfs-utils ~~~ ~~~powershell [root@nfs ~]# cat /etc/exports /sdb *(rw,sync,no_root_squash) ~~~ ~~~powershell [root@nfs ~]# systemctl enable nfs-server;systemctl start nfs-server Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service. [root@nfs ~]# systemctl status nfs-server ~~~ ~~~powershell [root@nfs ~]# showmount -e Export list for nfs: /sdb * ~~~ ~~~powershell 为了模拟在k8s集群主机中验证nfs可用性 [root@k8s-master01 ~]# yum -y install nfs-utils [root@k8s-master01 ~]# showmount -e nfs.kubemsb.com Export list for nfs.kubemsb.com: /sdb * ~~~ ## 6.2 k8s持久存储动态供给准备 1.下载并创建storageclass ~~~powershell [root@k8s-master1 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/class.yaml [root@k8s-master1 ~]# mv class.yaml storageclass-nfs.yml ~~~ ~~~powershell [root@k8s-master1 ~]# cat storageclass-nfs.yml apiVersion: storage.k8s.io/v1 kind: StorageClass # 类型 metadata: name: nfs-client # 名称,要使用就需要调用此名称 provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # 动态供给插件 parameters: archiveOnDelete: "false" # 删除数据时是否存档，false表示不存档，true表示存档 ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f storageclass-nfs.yml storageclass.storage.k8s.io/managed-nfs-storage created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 10s # RECLAIMPOLICY pv回收策略，pod或pvc被删除后，pv是否删除还是保留。 # VOLUMEBINDINGMODE Immediate 模式下PVC与PV立即绑定，主要是不等待相关Pod调度完成，不关心其运行节点，直接完成绑定。相反的 WaitForFirstConsumer模式下需要等待Pod调度完成后进行PV绑定。 # ALLOWVOLUMEEXPANSION pvc扩容 ~~~ 2.下载并创建rbac 因为storage自动创建pv需要经过kube-apiserver，所以需要授权。 ~~~powershell [root@k8s-master1 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/rbac.yaml [root@k8s-master1 ~]# mv rbac.yaml storageclass-nfs-rbac.yaml ~~~ ~~~powershell [root@k8s-master1 ~]# cat storageclass-nfs-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f rbac.yaml serviceaccount/nfs-client-provisioner created clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created ~~~ 3.创建动态供给的deployment 需要一个deployment来专门实现pv与pvc的自动创建 ~~~powershell [root@k8s-master1 ~]# vim deploy-nfs-client-provisioner.yml apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccount: nfs-client-provisioner containers: - name: nfs-client-provisioner image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: k8s-sigs.io/nfs-subdir-external-provisioner - name: NFS_SERVER value: 192.168.10.129 - name: NFS_PATH value: /data/nfs volumes: - name: nfs-client-root nfs: server: 192.168.10.129 path: /data/nfs ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f deploy-nfs-client-provisioner.yml deployment.apps/nfs-client-provisioner created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods |grep nfs-client-provisioner nfs-client-provisioner-5b5ddcd6c8-b6zbq 1/1 Running 0 34s ~~~ **扩展：** ~~~powershell 批量下载文件： # for file in class.yaml deployment.yaml rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/$file ; done ~~~ # 七、应用案例 ## 7.1 YAML应用及镜像下载案例 ~~~powershell 创建Pod资源清单文件 [root@nginx ~]# cat 01-create-pod.yaml apiVersion: v1 kind: Pod metadata: name: pod1 spec: containers: - name: c1 image: www.kubemsb.com/library/nginx:latest ~~~ ~~~powershell [root@nginx ~]# kubectl apply -f http://yaml.kubemsb.com/01-create-pod.yaml ~~~ ## 7.2 k8s持久化存储动态供给应用案例 ~~~powershell 测试存储动态供给是否可用 # vim nginx-sc.yaml --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: selector: matchLabels: app: nginx serviceName: "nginx" replicas: 2 template: metadata: labels: app: nginx spec: imagePullSecrets: - name: huoban-harbor terminationGracePeriodSeconds: 10 containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "nfs-client" resources: requests: storage: 1Gi ~~~ ~~~powershell [root@k8s-master1 nfs]# kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-9c988bc46-pr55n 1/1 Running 0 95s web-0 1/1 Running 0 95s web-1 1/1 Running 0 61s ~~~ ~~~powershell [root@nfsserver ~]# ls /data/nfs/ default-www-web-0-pvc-c4f7aeb0-6ee9-447f-a893-821774b8d11f default-www-web-1-pvc-8b8a4d3d-f75f-43af-8387-b7073d07ec01 ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_configMap_secret.md ================================================ # Kubernetes配置与密钥管理 ConfigMap&Secret # 一、ConfigMap ## 1.1 什么是configmap - **kubernetes集群可以使用ConfigMap来实现对容器中应用的配置管理**。 - **可以把ConfigMap看作是一个挂载到pod中的存储卷** ## 1.2 创建ConfigMap的4种方式 ### 1.2.1 在命令行指定参数创建 > 通过直接在命令行中指定configmap参数创建，即`--from-literal=key=value`； ```powershell [root@k8s-master1 ~]# kubectl create configmap cm1 --from-literal=host=127.0.0.1 --from-literal=port=3306 configmap/cm1 created ``` ```powershell [root@k8s-master1 ~]# kubectl get cm NAME DATA AGE cm1 2 12s ``` ```powershell [root@k8s-master1 ~]# kubectl describe cm cm1 Name: cm1 Namespace: default Labels: Annotations: Data ==== host: ---- 127.0.0.1 port: ---- 3306 Events: ``` ### 1.2.2 在命令行通过多个文件创建 > 通过指定文件创建，即将一个配置文件创建为一个ConfigMap，`--from-file=文件路径`； ```powershell [root@k8s-master1 ~]# echo -n 127.0.0.1 > host [root@k8s-master1 ~]# echo -n 3306 > port [root@k8s-master1 ~]# kubectl create configmap cm2 --from-file=./host --from-file=./port configmap/cm2 created ``` ```powershell [root@k8s-master1 ~]# kubectl get cm NAME DATA AGE cm1 2 3m45s cm2 2 94s ``` ```powershell [root@k8s-master1 ~]# kubectl describe cm cm2 Name: cm2 Namespace: default Labels: Annotations: Data ==== host: ---- 127.0.0.1 port: ---- 3306 Events: ``` ### 1.2.3 在命令行通过文件提供多个键值对创建 > 通过一个文件内多个键值对，`--from-env-file=文件路径`; ```powershell [root@k8s-master1 ~]# vim env.txt host=127.0.0.1 port=3306 [root@k8s-master1 ~]# kubectl create configmap cm3 --from-env-file=env.txt configmap/cm3 created ``` ```powershell [root@k8s-master1 ~]# kubectl get cm NAME DATA AGE cm1 2 4m37s cm2 2 2m26s cm3 2 12s ``` ```powershell [root@k8s-master1 ~]# kubectl describe cm cm3 Name: cm3 Namespace: default Labels: Annotations: Data ==== host: ---- 127.0.0.1 port: ---- 3306 Events: ``` ### 1.2.4 通过YAML资源清单文件创建 > 通过`kubectl create/apply -f YMAL文件` 创建 ```powershell [root@k8s-master1 ~]# vim cm4.yml apiVersion: v1 kind: ConfigMap metadata: name: cm4 data: host: 127.0.0.1 port: "3306" [root@k8s-master1 ~]# kubectl apply -f cm4.yml configmap/cm4 created ``` ```powershell [root@k8s-master1 ~]# kubectl get cm NAME DATA AGE cm1 2 6m18s cm2 2 4m7s cm3 2 113s cm4 2 11s ``` ```powershell [root@k8s-master1 ~]# kubectl describe cm cm4 Name: cm4 Namespace: default Labels: Annotations: Data ==== host: ---- 127.0.0.1 port: ---- 3306 Events: ``` ## 1.3 ConfigMap的2种使用方式 ### 1.3.1 通过环境变量的方式传递给pod ```powershell [root@k8s-master1 ~]# vim pod-cm1.yml apiVersion: v1 kind: Pod metadata: name: pod-cm1 spec: containers: - name: busybox image: busybox args: [ "/bin/sh", "-c", "sleep 10000" ] envFrom: # env方式 - configMapRef: name: cm1 # configmap名称 ``` ```powershell [root@k8s-master1 ~]# kubectl apply -f pod-cm1.yml pod/pod-cm1 created ``` ```powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE pod-cm1 1/1 Running 0 9s ``` ```powershell [root@k8s-master1 ~]# kubectl exec pod-cm1 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=pod-cm1 host=127.0.0.1 # 我们创建的configmap传进去的env port=3306 # 我们创建的configmap传进去的env DEPLOY_NGINX_PORT_80_TCP=tcp://10.2.205.160:80 DEPLOY_NGINX_PORT_80_TCP_ADDR=10.2.205.160 KUBERNETES_PORT=tcp://10.2.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.2.0.1:443 MY_SERVICE_SERVICE_HOST=10.2.52.46 MY_SERVICE_PORT_80_TCP_ADDR=10.2.52.46 KUBERNETES_SERVICE_HOST=10.2.0.1 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=10.2.0.1 MY_SERVICE_SERVICE_PORT=80 MY_SERVICE_PORT=tcp://10.2.52.46:80 MY_SERVICE_PORT_80_TCP=tcp://10.2.52.46:80 DEPLOY_NGINX_SERVICE_HOST=10.2.205.160 DEPLOY_NGINX_SERVICE_PORT=80 KUBERNETES_PORT_443_TCP_PORT=443 MY_SERVICE_PORT_80_TCP_PORT=80 DEPLOY_NGINX_PORT=tcp://10.2.205.160:80 DEPLOY_NGINX_PORT_80_TCP_PROTO=tcp DEPLOY_NGINX_PORT_80_TCP_PORT=80 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 MY_SERVICE_PORT_80_TCP_PROTO=tcp HOME=/root ``` ### 1.3.2 通过volume的方式挂载到pod内 ```powershell [root@k8s-master1 ~]# vim pod-cm2.yml apiVersion: v1 kind: Pod metadata: name: pod-cm2 spec: containers: - name: busybox image: busybox args: [ "/bin/sh", "-c", "sleep 10000" ] volumeMounts: # 用volume挂载方式 - name: vol-cm # 对应下面的volume名 mountPath: "/etc/mysql" # 挂载到容器内部的路径 readOnly: true # 只读 volumes: - name: vol-cm # 卷名称 configMap: name: cm2 # configmap的名称 ``` ```powershell [root@k8s-master1 ~]# kubectl apply -f pod-cm2.yml pod/pod-cm2 created ``` ```powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE pod-cm1 1/1 Running 0 3m51s pod-cm2 1/1 Running 0 49s ``` ```powershell [root@k8s-master1 ~]# kubectl exec pod-cm2 -- ls /etc/mysql host port [root@k8s-master1 ~]# kubectl exec pod-cm2 -- cat /etc/mysql/host 127.0.0.1 [root@k8s-master1 ~]# kubectl exec pod-cm2 -- cat /etc/mysql/port 3306 ``` ## 1.4 ConfigMap的热更新 ### 1.4.1 ConfigMap热更新方式 如果修改了value, 那么容器内部会不会更新? - **通过环境变量的方式传递给pod。这种方式不会热更新** - **通过volume的方式挂载到pod内。这种方式会热更新,大概需要半分钟左右。** ### 1.4.2 ConfigMap热更新验证 #### 1.4.2.1 通过环境变量方式 > 此种方式不会热更新 1.编辑修改对应的configmap ```powershell [root@k8s-master1 ~]# kubectl edit cm cm1 apiVersion: v1 data: host: 127.0.0.1 port: "3307" 3306修改成3307 kind: ConfigMap metadata: creationTimestamp: "2020-11-07T12:07:04Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:host: {} f:port: {} manager: kubectl operation: Update time: "2020-11-07T12:07:04Z" name: cm1 namespace: default resourceVersion: "169386" selfLink: /api/v1/namespaces/default/configmaps/cm1 uid: f06cd44d-2ef9-48f2-9ccc-995f9d9ea2ad ``` 2. 验证对应的pod里的变化,发现很久都不会改变(**环境变量方式**) ```powershell [root@k8s-master1 ~]# kubectl exec pod-cm1 -- env |grep port port=3306 仍然为3306 ``` #### 1.4.2.2 通过volume方式 1. 编辑修改对应的configmap ```powershell [root@k8s-master1 ~]# kubectl edit cm cm2 apiVersion: v1 data: host: 127.0.0.1 port: "3308" 修改成3308 kind: ConfigMap metadata: creationTimestamp: "2020-11-07T12:09:15Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:host: {} f:port: {} manager: kubectl operation: Update time: "2020-11-07T12:09:15Z" name: cm2 namespace: default resourceVersion: "169707" selfLink: /api/v1/namespaces/default/configmaps/cm2 ``` 2.验证对应的pod里的变化,一段时间后会改变(**卷挂载方式**) ```powershell [root@k8s-master1 ~]# kubectl exec pod-cm2 -- cat /etc/mysql/port 3308 大概半分钟后更新 ``` # 二、Secret ## 2.1 什么是secret - Secret与ConfigMap类似,主要的区别是ConfigMap存储的是明文,而Secret存储的是密文。 - ConfigMap可以用配置文件管理,而Secret可用于密码、密钥、token等敏感数据的配置管理。 ## 2.2 secret类型 Secret有4种类型: - Opaque: base64编码格式的Secret，用来存储密码、密钥、信息、证书等，类型标识符为generic - Service Account: 用来访问Kubernetes API，由Kubernetes自动创建，并且会自动挂载到Pod的/var/run/secrets/kubernetes.io/serviceaccount目录中 - kubernetes.io/dockerconfigjson: 用来存储私有docker registry的认证信息，类型标识为docker-registry。 - kubernetes.io/tls: 用于为SSL通信模式存储证书和私钥文件，命令式创建类型标识为tls。 ```powershell [root@k8s-master1 ~]# kubectl create secret -h Create a secret using specified subcommand. Available Commands: docker-registry Create a secret for use with a Docker registry generic Create a secret from a local file, directory or literal value tls Create a TLS secret Usage: kubectl create secret [flags] [options] Use "kubectl --help" for more information about a given command. Use "kubectl options" for a list of global command-line options (applies to all commands). ``` ## 2.3 Secret应用案例 > 创建mysql管理员密码Secret案例，使用Opaque类型来创建mysql密码Secret ### 2.3.1 将明文密码进行base64编码 Opaque类型密码需要进行base64编码 ```powershell [root@k8s-master1 ~]# echo -n 123 |base64 MTIz 假设密码为123,得到的编码为MTIz ``` ### 2.3.2 编写创建secret的YAML文件 ```powershell [root@k8s-master1 ~]# vim secret-mysql.yml apiVersion: v1 kind: Secret metadata: name: secret-mysql data: password: MTIz ``` ### 2.3.3 创建secret并确认 ```powershell [root@k8s-master1 ~]# kubectl apply -f secret-mysql.yml secret/secret-mysql created ``` ```powershell [root@k8s-master1 ~]# kubectl get secret |grep secret-mysql secret-mysql Opaque 1 40s ``` ## 2.4 Secret的2种使用方式 ### 2.4.1 通过环境变量的方式传递给pod ### 1. 编写pod的YAML文件使用Secret ```powershell [root@k8s-master1 ~]# vim pod-mysql-secret.yml apiVersion: v1 kind: Pod metadata: name: pod-mysql-secret1 spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: secret-mysql # 对应创建的secret名字 key: password ``` 2.创建pod ```powershell [root@k8s-master1 ~]# kubectl apply -f pod-mysql-secret.yml pod/pod-mysql-secret1 created ``` 3.验证pod ```powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE pod-mysql-secret1 1/1 Running 0 1m46s ``` 4.验证传入pod的变量效果 ```powershell [root@k8s-master1 ~]# kubectl exec -it pod-mysql-secret1 -- /bin/bash root@pod-mysql-secret1:/# env |grep MYSQL_ROOT_PASSWORD MYSQL_ROOT_PASSWORD=123 root@pod-mysql-secret1:/# mysql -p123 ``` ### 2.4.2 通过volume的方式挂载到pod内 1.编写pod的YAML文件使用Secret ```powershell [root@k8s-master1 ~]# vim pod-mysql-secret2.yml apiVersion: v1 kind: Pod metadata: name: pod-mysql-secret2 spec: containers: - name: busybox image: busybox args: - /bin/sh - -c - sleep 100000 volumeMounts: - name: vol-secret # 定义挂载的卷,对应下面定义的卷名 mountPath: "/opt/passwd" # 挂载目录(支持热更新),也可以使用subPath挂载文件(但不支持热更新) readOnly: true # 只读 volumes: - name: vol-secret # 定义卷名 secret: # 使用secret secretName: secret-mysql # 对应创建好的secret名 ``` 2.创建pod ```powershell [root@k8s-master1 ~]# kubectl apply -f pod-mysql-secret2.yml pod/pod-mysql-secret2 created ``` 3.验证pod ```powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE pod-mysql-secret2 1/1 Running 0 15m ``` 4.验证 ```powershell [root@k8s-master1 ~]# kubectl exec pod-mysql-secret2 -- cat /opt/passwd/password 123 在容器内都被解码了 ``` 5.热更新测试 ```powershell [root@k8s-master1 ~]# echo -n haha123 |base64 aGFoYTEyMw== [root@k8s-master1 ~]# kubectl edit secret secret-mysql apiVersion: v1 data: password: aGFoYTEyMw== 密码改成haha123的base64编码 kind: Secret metadata: ``` ```powershell [root@k8s-master1 ~]# kubectl exec pod-mysql-secret2 -- cat /opt/passwd/password haha123 过一会儿,确认密码确实更新了 ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_core.md ================================================ # Kubernetes核心 # 一、kubernetes核心概念 ## 1.1 Pod Pod是可以在 Kubernetes 中创建和管理的、最小的可部署的计算单元。 Pod就像豌豆荚一样，其中包含着一组（一个或多个）容器； 这些容器共享存储、网络、以及怎样运行这些容器的声明。 ![image-20220330113151804](../../img/kubernetes/kubernetes_core/image-20220330113151804.png) Pod就像一台物理服务器一样，其中包含一个或多个应用容器， 这些容器中运行着用户应用程序。 **举例说明Pod、Container、应用程序三者之间的关系：麻屋子,红帐子,里面住着白胖子。Pod就是麻屋子,Container就是红帐子,应用程序就是里面的白胖子。** ![image-20220330113046572](../../img/kubernetes/kubernetes_core/image-20220330113046572.png) ## 1.2 Controller 在 Kubernetes 中，用于管理和运行Pod的对象 在 Kubernetes 中，控制器通过监控集群的公共状态，并致力于将当前状态转变为期望的状态 **举例说明Controller(控制器)作用：房间里的温度自动调节器** **当你设置了温度，告诉了温度自动调节器你的*期望状态（Desired State）*。 房间的实际温度是*当前状态（Current State）*。 通过对设备的开关控制，温度自动调节器让其当前状态接近期望状态。** 一个控制器至少追踪一种类型的 Kubernetes 资源。这些对象有一个代表期望状态的 `spec` 字段。 该资源的控制器负责确保其当前状态接近期望状态。 不同的类型的控制器所实现的控制方式不一样，例如： - deployment - 部署无状态应用 - 部署无状态应用: 认为pod 都一样，没有顺序要求， 不用考虑在哪个node 运行，随意进行扩展和伸缩 - 管理Pod和 ReplicaSet - 部署、滚动升级等 - 典型的像web服务、分布式服务等 - StatefulSet - 部署有状态应用 - 有状态应用，每个pod 都独立运行，保持pod 启动顺序和唯一性； 有唯一的网络标识符，持久存储； 有序，比如mysql 主从； 主机名称固定。 而且其扩容以及升级等操作也是按顺序进行的操作。 - DaemonSet - 部署守护进程 - DaemonSet保证在每个Node上都运行一个容器副本，常用来部署一些集群的日志、监控或者其他系统管理应用。 新加入的node 也同样运行在一个pod 里面。 - job - 一次性任务 - Job负责批量处理短暂的一次性任务 (short lived one-off tasks)，即仅执行一次的任务，它保证批处理任务的一个或多个Pod成功结束。 - Cronjob - 周期性定时任务 ## 1.3 Label ### 1.3.1 Label介绍 Label是附着到object上（例如Pod）的键值对。可以在创建object的时候指定，也可以在object创建后随时指定。Labels的值对系统本身并没有什么含义，只是对用户才有意义。 一个Label是一个key=value的键值对，其中key与value由用户自己指定。 Label可以附加到各种资源对象上，例如Node、Pod、Service、RC等，一个资源对象可以定义任意数量的Label，同一个Label也可以被添加到任意数量的资源对象上去，Label通常在资源对象定义时确定，也可以在对象创建后动态添加或者删除。 我们可以通过指定的资源对象捆绑一个或多个不同的Label来实现多维度的资源分组管理功能，以便于灵活、方便地进行资源分配、调度、配置、部署等管理工作。例如：部署不同版本的应用到不同的环境中；或者监控和分析应用（日志记录、监控、告警）等。 一些常用abel示例如下所示: 版本标签："release" : "stable" , "release" : "canary"... 环境标签："environment" : "dev" , "environment" : "production" 架构标签："tier" : "frontend" , "tier" : "backend" , "tier" : "middleware" 分区标签："partition" : "customerA" , "partition" : "customerB"... 质量管控标签："track" : "daily" , "track" : "weekly" Label相当于我们熟悉的“标签”，给某个资源对象定义一个Label，就相当于给它打了一个标签，随后可以通过Label Selector（标签选择器）查询和筛选拥有某些Label的资源对象，Kubernetes通过这种方式实现了类似SQL的简单又通用的对象查询机制。 ### 1.3.2 Label语法及字符集 Label key的组成： - 不得超过63个字符 - 可以使用前缀，使用/分隔，前缀必须是DNS子域，不得超过253个字符，系统中的自动化组件创建的label必须指定前缀，`kubernetes.io/`由kubernetes保留 - 起始必须是字母（大小写都可以）或数字，中间可以有连字符、下划线和点 Label value的组成： - 不得超过63个字符 - 起始必须是字母（大小写都可以）或数字，中间可以有连字符、下划线和点 ## 1.4 Label Selector 通过label selector，客户端／用户可以指定一个object集合，通过label selector对object的集合进行操作。 Label selector有两种类型： - *equality-based（基于等式）* ：可以使用`=`、`==`、`!=`操作符，可以使用逗号分隔多个表达式 - *set-based*（基于集合） ：可以使用`in`、`notin`、`!`操作符，另外还可以没有操作符，直接写出某个label的key，表示过滤有某个key的object而不管该key的value是何值，`!`表示没有该label的object **举例说明Label Selector** **Label Selector可以被类比为SQL语句中的where查询条件，例如，name=redis-slave这个label Selector作用于Pod时，可以被类比为select * from pod where pod's name = 'redis-slave'这样的语句。** ## 1.5 Service 将运行在一组 Pods上的应用程序公开为网络服务的抽象方法。 由于Pod是非永久性资源对象，如果你使用Controller运行你的应用程序，你可以动态创建和销毁Pod，这样就会导致无法准确访问到所想要访问的Pod 例如：如果一组 Pod（称为“后端”）为集群内的其他 Pod（称为“前端”）提供功能， 那么前端如何找出并跟踪要连接的 IP 地址，以便前端可以使用提供工作负载的后端部分？ 是一组iptables或ipvs规划，通过把客户端的请求转发到服务端（Pod）,如有多个Pod情况，亦可实现负载均衡的效果。 例如：一个图片处理后端，它运行了 3 个副本（Pod）。这些副本是可互换的 —— 前端不需要关心它们调用了哪个后端副本。 然而组成这一组后端程序的 Pod 实际上可能会发生变化， 前端客户端不应该也没必要知道，而且也不需要跟踪这一组后端的状态。 ## 1.6 Endpoints 为Service管理后端Pod，当后端Pod被创建或销毁时，endpoints列表会更新Pod对应的IP地址，以便Service访问请求能够确保被响应。 ## 1.7 DNS 为kubernetes集群内资源对象的访问提供名称解析，这样就可以实现通过DNS名称而非IP地址来访问服务。 - 实现集群内Service名称解析 - 实现集群内Pod内Container中应用访问互联网提供域名解析 # 二、Kubernetes核心概念之间的关系 ## 2.1 Pod与Controller pod 是通过Controller 实现应用的运维，比如伸缩，滚动升级等待。pod 和 controller 通过label 标签建立关系。 ![](../../img/kubernetes/kubernetes_core/controller与pod.gif) ## 2.2 Pod与Service service 是为了防止pod 失联，提供的服务发现，类似于微服务的注册中心。定义一组pod 的访问策略。可以为一组具有相同功能的容器应用提供一个统一的入口地址，并将请求负载分发到后端的各个容器应用上。 service 通过selector 来管控对应的pod。根据label 和 selector 建立关联，通过service 实现pod 的负载均衡。 ![image-20220330021556081](../../img/kubernetes/kubernetes_core/image-20220330021556081.png) ## 2.3 Service与DNS 通过DNS实现对Service名称解析，以此达到访问后端Pod目的。 # 三、基于kubernetes集群容器化应用的微服务 ## 3.1 服务部署方式介绍 - 单体服务架构 - 所有服务进程运行在同一台主机内 - 分布式服务架构 - 服务进程分布于不同的主机，其中一台主机出现故障，不影响其它主机上的服务运行 - 微服务架构 - 使用容器化技术把分布式服务架构运行起来，并实现对不同的服务进程的高可用及快速发布等。 ## 3.2 微服务架构服务组件（kubernetes核心概念）之间关系举例说明 以在kubernetes集群中运行LNMT应用为例： 把kubernetes集群看做是一个IDC机房，把LNMT Web架构应用以微服务（kubernetes集群资源对象）的方式部署到kubernetes集群中。 ![](../../img/kubernetes/kubernetes_core/kubernetes集群微服务架构.png) # Kubernetes集群核心概念 Pod # 一、工作负载(workloads) 参考链接：https://kubernetes.io/zh/docs/concepts/workloads/ 工作负载（workload）是在kubernetes集群中运行的应用程序。无论你的工作负载是单一服务还是多个一同工作的服务构成，在kubernetes中都可以使用pod来运行它。 ![1604285743110](../../img/kubernetes/kubernetes_core/1.png) workloads分为pod与controllers - pod通过控制器实现应用的运行，如何伸缩，升级等 - controllers 在集群中管理pod - pod与控制器之间通过label-selector相关联，是唯一的关联方式 ![1564844640416](../../img/kubernetes/kubernetes_core/work_load.png) 在pod的YAML里指定pod标签 ~~~powershell 定义标签 labels: app: nginx ~~~ 在控制器的YAML里指定标签选择器匹配标签 ~~~powershell 通过标签选择器选择对应的pod selector: matchLabels: app: nginx ~~~ # 二、pod介绍 ## 2.1 pod定义与分类 参考链接: https://kubernetes.io/zh/docs/concepts/workloads/pods/ ### 2.1.1 Pod定义 * Pod(豌豆荚) 是Kubernetes集群管理（创建、部署）与调度的最小计算单元，表示处于运行状态的一组容器。 * Pod不是进程，而是容器运行的环境。 * 一个Pod可以封装**一个容器或多个容器(主容器或sidecar边车容器)** * 一个pod内的多个容器之间共享部分命名空间，例如：Net Namespace,UTS Namespace,IPC Namespace及存储资源 * 用户pod默认会被调度运行在node节点之上(不运行在master节点上，但也有例外情况) * pod内的IP不是固定的，集群外不能直接访问pod ### 2.1.2 Pod分类 * **静态Pod** 也称之为“无控制器管理的自主式pod”，直接由特定节点上的 `kubelet` 守护进程管理， 不需要API 服务器看到它们，尽管大多数 Pod 都是通过控制面（例如，Deployment） 来管理的，对于静态 Pod 而言，`kubelet` 直接监控每个 Pod，并在其失效时重启之。 * **控制器管理的pod** 控制器可以控制pod的副本数，扩容与裁剪，版本更新与回滚等 ## 2.2 查看pod方法 pod是一种计算资源，可以通过`kubectl get pod`来查看 ~~~powershell [root@k8s-master1 ~]# kubectl get pod # pod或pods都可以，不指定namespace,默认是名为default的namespace [root@k8s-master1 ~]# kubectl get pod -n kube-system ~~~ ## 2.3 pod的YAML资源清单格式 先看一个yaml格式的pod定义文件解释 ~~~powershell # yaml格式的pod定义文件完整内容： apiVersion: v1 #必选，api版本号，例如v1 kind: Pod #必选，Pod metadata: #必选，元数据 name: string #必选，Pod名称 namespace: string #Pod所属的命名空间,默认在default的namespace labels: # 自定义标签 name: string #自定义标签名字 annotations: #自定义注释列表 name: string spec: #必选，Pod中容器的详细定义(期望) containers: #必选，Pod中容器列表 - name: string #必选，容器名称 image: string #必选，容器的镜像名称 imagePullPolicy: [Always | Never | IfNotPresent] #获取镜像的策略 Alawys表示下载镜像 IfnotPresent表示优先使用本地镜像，否则下载镜像，Nerver表示仅使用本地镜像 command: [string] #容器的启动命令列表，如不指定，使用打包时使用的启动命令 args: [string] #容器的启动命令参数列表 workingDir: string #容器的工作目录 volumeMounts: #挂载到容器内部的存储卷配置 - name: string #引用pod定义的共享存储卷的名称，需用volumes[]部分定义的的卷名 mountPath: string #存储卷在容器内mount的绝对路径，应少于512字符 readOnly: boolean #是否为只读模式 ports: #需要暴露的端口库号列表 - name: string #端口号名称 containerPort: int #容器需要监听的端口号 hostPort: int #容器所在主机需要监听的端口号，默认与Container相同 protocol: string #端口协议，支持TCP和UDP，默认TCP env: #容器运行前需设置的环境变量列表 - name: string #环境变量名称 value: string #环境变量的值 resources: #资源限制和请求的设置 limits: #资源限制的设置 cpu: string #Cpu的限制，单位为core数，将用于docker run --cpu-shares参数 memory: string #内存限制，单位可以为Mib/Gib，将用于docker run --memory参数 requests: #资源请求的设置 cpu: string #Cpu请求，容器启动的初始可用数量 memory: string #内存清求，容器启动的初始可用数量 livenessProbe: #对Pod内个容器健康检查的设置，当探测无响应几次后将自动重启该容器，检查方法有exec、httpGet和tcpSocket，对一个容器只需设置其中一种方法即可 exec: #对Pod容器内检查方式设置为exec方式 command: [string] #exec方式需要制定的命令或脚本 httpGet: #对Pod内个容器健康检查方法设置为HttpGet，需要制定Path、port path: string port: number host: string scheme: string HttpHeaders: - name: string value: string tcpSocket: #对Pod内个容器健康检查方式设置为tcpSocket方式 port: number initialDelaySeconds: 0 #容器启动完成后首次探测的时间，单位为秒 timeoutSeconds: 0 #对容器健康检查探测等待响应的超时时间，单位秒，默认1秒 periodSeconds: 0 #对容器监控检查的定期探测时间设置，单位秒，默认10秒一次 successThreshold: 0 failureThreshold: 0 securityContext: privileged:false restartPolicy: [Always | Never | OnFailure] # Pod的重启策略，Always表示一旦不管以何种方式终止运行，kubelet都将重启，OnFailure表示只有Pod以非0退出码退出才重启，Nerver表示不再重启该Pod nodeSelector: obeject # 设置NodeSelector表示将该Pod调度到包含这个label的node上，以key：value的格式指定 imagePullSecrets: #Pull镜像时使用的secret名称，以key：secretkey格式指定 - name: string hostNetwork: false #是否使用主机网络模式，默认为false，如果设置为true，表示使用宿主机网络 volumes: #在该pod上定义共享存储卷列表 - name: string #共享存储卷名称 （volumes类型有很多种） emptyDir: {} #类型为emtyDir的存储卷，与Pod同生命周期的一个临时目录。为空值 hostPath: string #类型为hostPath的存储卷，表示挂载Pod所在宿主机的目录 path: string #Pod所在宿主机的目录，将被用于同期中mount的目录 secret: #类型为secret的存储卷，挂载集群与定义的secret对象到容器内部 scretname: string items: - key: string path: string configMap: #类型为configMap的存储卷，挂载预定义的configMap对象到容器内部 name: string items: - key: string path: string ~~~ **YAML格式查找帮助方法回顾** ~~~powershell [root@k8s-master1 ~]# kubectl explain namespace [root@k8s-master1 ~]# kubectl explain pod [root@k8s-master1 ~]# kubectl explain pod.spec [root@k8s-master1 ~]# kubectl explain pod.spec.containers ~~~ # 三、pod创建与验证 ## 3.1 命令创建pod(v1.18变化) - k8s之前版本中, kubectl run命令用于创建deployment控制器 - 在v1.18版本中, kubectl run命令改为创建pod ### 3.1.1 创建一个名为pod-nginx的pod ```powershell [root@k8s-master1 ~]# kubectl run nginx1 --image=nginx:1.15-alpine pod/nginx1 created ``` ### 3.1.2 验证 ```powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx1 1/1 Running 0 41s ``` ## 3.2 YAML创建pod ### 3.2.1 准备yaml文件 ~~~powershell [root@k8s-master1 ~]# vim pod1.yml apiVersion: v1 # api版本 kind: Pod # 资源类型为Pod metadata: name: pod-stress # 自定义pod的名称 spec: containers: # 定义pod里包含的容器 - name: c1 # 自定义pod中的容器名 image: polinux/stress # 启动容器的镜像名 command: ["stress"] # 自定义启动容器时要执行的命令(类似dockerfile里的CMD) args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] # 自定义启动容器执行命令的参数 # polinux/stress这个镜像用于压力测试,在启动容器时传命令与参数就是相当于分配容器运行时需要的压力 ~~~ 2, 通过yaml文件创建pod ```powershell [root@k8s-master1 ~]# kubectl apply -f pod1.yml pod/pod-stress created ``` ### 3.2.2 查看pod信息 查看pod信息 ```powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE pod-stress 1/1 Running 0 45s ``` 查看pod详细信息 ```powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-stress 1/1 Running 0 71s 10.244.194.72 k8s-worker1 ``` 描述pod详细信息 ```powershell [root@k8s-master1 ~]# kubectl describe pod pod-stress ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 102s default-scheduler Successfully assigned default/pod-stress to k8s-worker1 Normal Pulling 102s kubelet Pulling image "polinux/stress" Normal Pulled 83s kubelet Successfully pulled image "polinux/stress" in 18.944533343s Normal Created 83s kubelet Created container c1 Normal Started 82s kubelet Started container c1 ``` ## 3.3 删除pod ### 3.3.1 单个pod删除 方法1: ```powershell [root@k8s-master1 ~]# kubectl delete pod pod-stress pod "pod-stress" deleted ``` 方法2: ```powershell [root@k8s-master1 ~]# kubectl delete -f pod1.yml ``` ### 3.3.2 多个pod删除 方法1: 后接多个pod名 ```powershell [root@k8s-master1 ~]# kubectl delete pod pod名1 pod名2 pod名3 ...... ``` 方法2: 通过awk截取要删除的pod名称，然后管道给xargs ```powershell [root@k8s-master1 ~]# kubectl get pods |awk 'NR>1 {print $1}' |xargs kubectl delete pod ``` 方法3: 如果要删除的pod都在同一个非default的命名空间，则可直接删除命名空间 ```powershell [root@k8s-master1 ~]# kubectl delete ns xxxx ``` ## 3.4 镜像拉取策略 由imagePullPolicy参数控制 - Always : 不管本地有没有镜像，都要从仓库中下载镜像 - Never : 从来不从仓库下载镜像, 只用本地镜像,本地没有就算了 - IfNotPresent: 如果本地存在就直接使用, 不存在才从仓库下载 默认的策略是： - 当镜像标签版本是latest，默认策略就是Always - 如果指定特定版本默认拉取策略就是IfNotPresent。 1, 将上面的pod删除再创建，使用下面命令查看信息 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod1.yml [root@k8s-master1 ~]# kubectl delete -f pod1.yml ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl describe pod pod-stress ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 102s default-scheduler Successfully assigned default/pod-stress to k8s-worker1 Normal Pulling 102s kubelet Pulling image "polinux/stress" Normal Pulled 83s kubelet Successfully pulled image "polinux/stress" in 18.944533343s Normal Created 83s kubelet Created container c1 Normal Started 82s kubelet Started container c1 ~~~ **说明: 可以看到第二行信息还是`pulling image`下载镜像** 2, 修改YAML ~~~powershell [root@k8s-master1 ~]# vim pod1.yml apiVersion: v1 kind: Pod metadata: name: pod-stress namespace: default spec: containers: - name: c1 image: polinux/stress command: ["stress"] args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] imagePullPolicy: IfNotPresent # 增加了这一句 ~~~ 3，再次删除再创建 ~~~powershell [root@k8s-master1 ~]# kubectl describe pod pod-stress ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 17s default-scheduler Successfully assigned default/pod-stress to k8s-worker1 Normal Pulled 17s kubelet Container image "polinux/stress" already present on machine Normal Created 17s kubelet Created container c1 Normal Started 17s kubelet Started container c1 ~~~ **说明: 第二行信息是说镜像已经存在，直接使用了** ## 3.5 pod的标签 * 为pod设置label,用于控制器通过label与pod关联 * 语法与前面学的node标签几乎一致 ### 3.5.1 通过命令管理Pod标签 1. 查看pod的标签 ~~~powershell [root@k8s-master1 ~]# kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS pod-stress 1/1 Running 0 7m25s ~~~ 2. 打标签,再查看 ~~~powershell [root@k8s-master1 ~]# kubectl label pod pod-stress region=huanai zone=A env=test bussiness=game pod/pod-stress labeled [root@k8s-master1 ~]# kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS pod-stress 1/1 Running 0 8m54s bussiness=game,env=test,region=huanai,zone=A ~~~ 3. 通过等值关系标签查询 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -l zone=A NAME READY STATUS RESTARTS AGE pod-stress 1/1 Running 0 9m22s ~~~ 4. 通过集合关系标签查询 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -l "zone in (A,B,C)" NAME READY STATUS RESTARTS AGE pod-stress 1/1 Running 0 9m55s ~~~ 5. 删除标签后再验证 ~~~powershell [root@k8s-master1 ~]# kubectl label pod pod-stress region- zone- env- bussiness- pod/pod-stress labeled [root@k8s-master1 ~]# kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS pod-stress 1/1 Running 0 16m ~~~ **小结:** * pod的label与node的label操作方式几乎相同 * node的label用于pod调度到指定label的node节点 * pod的label用于controller关联控制的pod ### 3.5.2 通过YAML创建Pod时添加标签 1, 修改yaml ~~~powershell [root@k8s-master1 ~]# vim pod1.yml apiVersion: v1 kind: Pod metadata: name: pod-stress namespace: default labels: env: dev app: nginx # 直接在原来的yaml里加上多个标签 spec: containers: - name: c1 image: polinux/stress command: ["stress"] args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] imagePullPolicy: IfNotPresent ~~~ 2, 直接apply应用 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod1.yaml pod/pod-stress1 configured # 这里是configured,表示修改了 ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS pod-stress 1/1 Running 0 3m5s app=nginx,env=dev # 标签有了 ~~~ ## 3.6 pod资源限制 准备2个不同限制方式的创建pod的yaml文件 ~~~powershell [root@k8s-master1 ~]# vim pod2.yml apiVersion: v1 kind: Namespace metadata: name: namespace1 --- apiVersion: v1 kind: Pod metadata: name: pod-stress2 namespace: namespace1 spec: containers: - name: c1 image: polinux/stress imagePullPolicy: IfNotPresent resources: limits: memory: "200Mi" requests: memory: "100Mi" command: ["stress"] # 启动容器时执行的命令 args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] # 产生1个进程分配150M内存1秒后释放 ~~~ ~~~powershell [root@k8s-master1 ~]# vim pod3.yml apiVersion: v1 kind: Namespace metadata: name: namespace1 --- apiVersion: v1 kind: Pod metadata: name: pod-stress3 namespace: namespace1 spec: containers: - name: c1 image: polinux/stress imagePullPolicy: IfNotPresent resources: limits: memory: "200Mi" requests: memory: "150Mi" command: ["stress"] args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"] ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod2.yml namespace/namespace1 created pod/pod-stress2 created [root@k8s-master1 ~]# kubectl apply -f pod3.yml namespace/namespace1 unchanged pod/pod-stress3 created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get namespace |grep namespace1 namespace1 Active 1m28s [root@k8s-master1 ~]# kubectl get pod -n namespace1 NAME READY STATUS RESTARTS AGE pod-stress2 1/1 Running 0 2m2s pod-stress3 0/1 OOMKilled 4 115s 查看会发现pod-stress3这个pod状态变为OOMKilled，因为它是内存不足所以显示Container被杀死 ~~~ 说明: 一旦pod中的容器挂了，容器会有重启策略， 如下： * Always：表示容器挂了总是重启，这是默认策略 * OnFailures：表容器状态为错误时才重启，也就是容器正常终止时才重启 * Never：表示容器挂了不予重启 * 对于Always这种策略，容器只要挂了，就会立即重启，这样是很耗费资源的。所以Always重启策略是这么做的：第一次容器挂了立即重启，如果再挂了就要延时10s重启，第三次挂了就等20s重启...... 依次类推 测试完后删除 ~~~powershell [root@k8s-master1 ~]# kubectl delete ns namespace1 ~~~ ## 3.7 pod包含多个容器 1, 准备yml文件 ~~~powershell [root@k8s-master1 ~]# vim pod4.yml apiVersion: v1 kind: Pod metadata: name: pod-stress4 spec: containers: - name: c1 image: polinux/stress imagePullPolicy: IfNotPresent resources: limits: memory: "200Mi" requests: memory: "100Mi" command: ["stress"] args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] - name: c2 image: polinux/stress imagePullPolicy: IfNotPresent resources: limits: memory: "200Mi" requests: memory: "100Mi" command: ["stress"] args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] ~~~ 2, 应用yml文件创建pod ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod4.yml pod/pod-stress4 created ~~~ 3, 查看pod在哪个节点 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-stress4 2/2 Running 0 70s 10.244.159.136 k8s-master1 可以看到有2个容器,运行在k8s-master1节点 ~~~ 4,在k8s-master1上验证,确实产生了2个容器 ~~~powershell [root@k8s-master1 ~]# docker ps -a |grep stress d7827a963f9d df58d15b053d "stress --vm 1 --vm-…" 2 hours ago Up 2 hours k8s_c2_pod-stress4_default_a534bce1-3ffe-45f5-8128-34657e289b44_0 ae8e8f8d095b df58d15b053d "stress --vm 1 --vm-…" 2 hours ago Up 2 hours k8s_c1_pod-stress4_default_a534bce1-3ffe-45f5-8128-34657e289b44_0 e66461900426 easzlab/pause-amd64:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_pod-stress4_default_a534bce1-3ffe-45f5-8128-34657e289b44_0 ~~~ 或 ~~~powershell [root@k8s-master1 ~]# crictl ps CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID 08cd23ac9b416 df58d15b053d1 2 hours ago Running c2 0 2dd084491f019 65f0ecba8dec3 df58d15b053d1 2 hours ago Running c1 0 2dd084491f019 ~~~ ## 3.8 对pod里的容器进行操作 ### 3.8.1 命令帮助 ~~~powershell [root@k8s-master1 ~]# kubectl exec -h ~~~ ### 3.8.2 不用交互直接执行命令 格式为: `kubectl exec pod名 -c 容器名 -- 命令` **注意:** * -c 容器名为可选项,如果是1个pod中1个容器,则不用指定; * 如果是1个pod中多个容器,不指定默认为第1个。 ~~~powershell [root@k8s-master1 ~]# kubectl exec pod-stress4 -c c2 -- touch /111 [root@k8s-master1 ~]# kubectl exec pod-stress4 -c c2 -- ls /111 /111 ~~~ 不指定容器名,则默认为pod里的第1个容器 ~~~powershell [root@k8s-master1 ~]# kubectl exec pod-stress4 -- touch /222 Defaulting container name to c1. Use 'kubectl describe pod/pod-stress4 -n default' to see all of the containers in this pod. ~~~ ### 3.8.3 和容器交互操作 和docker exec几乎一样 ~~~powershell [root@k8s-master1 ~]# kubectl exec -it pod-stress4 -c c1 -- /bin/bash bash-5.0# touch /333 bash-5.0# ls 222 bin etc lib mnt proc run srv tmp var 333 dev home media opt root sbin sys usr bash-5.0# exit exit ~~~ ## 3.9 验证pod中多个容器网络共享 1, 编写YAML ```powershell [root@k8s-master1 ~]# vim pod-nginx.yaml apiVersion: v1 kind: Pod metadata: name: nginx2 spec: containers: - name: c1 image: nginx:1.15-alpine - name: c2 image: nginx:1.15-alpine ``` 2, 应用YAML ```powershell [root@k8s-master1 ~]# kubectl apply -f pod-nginx.yaml pod/nginx2 created ``` 3, 查看pod信息与状态 ```powershell [root@k8s-master1 ~]# kubectl describe pod nginx2 ...... ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 25s default-scheduler Successfully assigned default/nginx2 to k8s-worker1 Normal Pulling 24s kubelet Pulling image "nginx:1.15-alpine" Normal Pulled 5s kubelet Successfully pulled image "nginx:1.15-alpine" in 18.928009025s Normal Created 5s kubelet Created container c1 Normal Started 5s kubelet Started container c1 Normal Pulled 2s (x2 over 5s) kubelet Container image "nginx:1.15-alpine" already present on machine Normal Created 2s (x2 over 5s) kubelet Created container c2 Normal Started 2s (x2 over 5s) kubelet Started container c2 ``` ```powershell [root@k8s-master1 ~]# kubectl get pods |grep nginx2 nginx2 1/2 CrashLoopBackOff 3 2m40s 有一个启不来，因为一个容器中两个pod是共用网络的，所以不能两个都占用80端口 ``` 有一个启不来，因为一个pod中两个容器是共用网络的，所以不能两个都占用80端口 通过查找`k8s-worker1`上面的容器，然后`docker logs或crictl logs containerID`查看，得到如下的报错，说明是端口被占用 ```powershell [root@k8s-worker1 ~]# docker logs k8s_c2_nginx2_default_51fd8e81-1c4b-4557-9498-9b25ed8a4c99_4 2020/11/21 04:29:12 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2020/11/21 04:29:12 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2020/11/21 04:29:12 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2020/11/21 04:29:12 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2020/11/21 04:29:12 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2020/11/21 04:29:12 [emerg] 1#1: still could not bind() nginx: [emerg] still could not bind() ``` # 四、pod调度 ## 4.1 pod调度流程 ![1564985023369](../../img/kubernetes/kubernetes_core/pod调度.png) ~~~powershell Step1 通过kubectl命令应用资源清单文件（yaml格式）向api server 发起一个create pod 请求 Step2 api server接收到pod创建请求后，生成一个包含创建信息资源清单文件 Step3 apiserver 将资源清单文件中信息写入etcd数据库 Step4 Scheduler启动后会一直watch API Server，获取 podSpec.NodeName为空的Pod,即判断pod.spec.Node == null? 若为null，表示这个Pod请求是新的，需要创建，因此先进行调度计算（共计2步：1、过滤不满足条件的，2、选择优先级高的），找到合适的node，然后将信息在etcd数据库中更新分配结果：pod.spec.Node = nodeA (设置一个具体的节点) Step5 kubelet 通过watch etcd数据库(即不停地看etcd中的记录)，发现有新的Node出现，如果这条记录中的Node与所在节点编号相同，即这个Pod由scheduler分配给自己，则调用node中的Container Runtime，进而创建container，并将创建后的结果返回到给api server用于更新etcd数据库中数据状态。 ~~~ ## 4.2 调度约束方法 我们为了实现容器主机资源平衡使用, 可以使用约束把pod调度到指定的node节点 - nodeName 用于将pod调度到指定的node名称上 - nodeSelector 用于将pod调度到匹配Label的node上 ### 4.2.1 nodeName 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-nodename.yml apiVersion: v1 kind: Pod metadata: name: pod-nodename spec: nodeName: k8s-worker1 # 通过nodeName调度到k8s-worker1节点 containers: - name: nginx image: nginx:1.15-alpine ~~~ 2, 应用YAML文件创建pod ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-nodename.yml pod/pod-nodename created ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl describe pod pod-nodename |tail -6 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 2m47s kubelet Container image "nginx:1.15-alpine" already present on machine Normal Created 2m47s kubelet Created container nginx Normal Started 2m47s kubelet Started container nginx 倒数第3行没有使用scheduler,而是直接给运行了,说明nodeName约束生效 ~~~ ### 4.2.2 nodeSelector 1, 为k8s-worker1打标签 ~~~powershell [root@k8s-master1 ~]# kubectl label nodes k8s-worker1 bussiness=game node/k8s-worker1 labeled ~~~ 2, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-nodeselector.yml apiVersion: v1 kind: Pod metadata: name: pod-nodeselect spec: nodeSelector: # nodeSelector节点选择器 bussiness: game # 指定调度到标签为bussiness=game的节点 containers: - name: nginx image: nginx:1.15-alpine ~~~ 3, 应用YAML文件创建pod ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-nodeselector.yml pod/pod-nodeselect created ~~~ 4, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl describe pod pod-nodeselect |tail -6 Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 20s default-scheduler Successfully assigned default/pod-nodeselect to k8s-worker1 Normal Pulled 19s kubelet Container image "nginx:1.15-alpine" already present on machine Normal Created 19s kubelet Created container nginx Normal Started 19s kubelet Started container nginx 仍然经过了scheduler,但确实分配到了 k8s-worker1上 ~~~ 有兴趣可以再删除后再创建,重复几次验证 # 五、pod的生命周期 ## 5.1 Pod生命周期 * 有些pod(比如运行httpd服务),正常情况下会一直运行中,但如果手动删除它,此pod会终止 * 也有些pod(比如执行计算任务)，任务计算完后就会自动终止 上面两种场景中,pod从创建到终止的过程就是pod的生命周期。 ![](../../img/kubernetes/kubernetes_core/pod生命周期.png) ### 5.1.1 容器启动 1. pod中的容器在创建前,有初始化容器(init container)来进行初始化环境 2. 初化完后,主容器(main container)开始启动 3. 主容器启动后,有一个**post start**的操作(启动后的触发型操作,或者叫启动后钩子) 4. post start后,就开始做健康检查 * 第一个健康检查叫存活状态检查(liveness probe )，用来检查主容器存活状态的 * 第二个健康检查叫准备就绪检查(readiness probe)，用来检查主容器是否启动就绪 ### 5.1.2 容器终止 1. 可以在容器终止前设置**pre stop**操作(终止前的触发型操作,或者叫终止前钩子) 3. 当出现特殊情况不能正常销毁pod时,大概等待30秒会强制终止 3. 终止容器后还可能会重启容器(视容器重启策略而定)。 ### 5.1.3 回顾容器重启策略 * Always：表示容器挂了总是重启，这是默认策略 * OnFailures：表示容器状态为错误时才重启，也就是容器正常终止时不重启 * Never：表示容器挂了不予重启 * 对于Always这种策略，容器只要挂了，就会立即重启，这样是很耗费资源的。所以Always重启策略是这么做的：第一次容器挂了立即重启，如果再挂了就要延时10s重启，第三次挂了就等20s重启...... 依次类推 ## 5.2 HealthCheck健康检查 当Pod启动时，容器可能会因为某种错误(服务未启动或端口不正确)而无法访问等。 ### 5.2.1 Health Check方式 kubelet拥有两个检测器，它们分别对应不同的触发器(根据触发器的结构执行进一步的动作) | 方式 | 说明 | | ---------------------------- | ------------------------------------------------------------ | | Liveness Probe(存活状态探测) | 指示容器是否正在运行。如果存活态探测失败，则 kubelet 会杀死容器， 并且容器将根据其[重启策略](https://kubernetes.io/zh/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy)决定未来。如果容器不提供存活探针， 则默认状态为 `Success`。 | | readiness Probe(就绪型探测) | 指示容器是否准备好为请求提供服务。如果就绪态探测失败， 端点控制器将从与 Pod 匹配的所有服务的端点列表中删除该 Pod 的 IP 地址。 初始延迟之前的就绪态的状态值默认为 `Failure`。 如果容器不提供就绪态探针，则默认状态为 `Success`。注：检查后不健康，将容器设置为Notready;如果使用service来访问,流量不会转发给此种状态的pod | | startup Probe | 指示容器中的应用是否已经启动。如果提供了启动探针，则所有其他探针都会被 禁用，直到此探针成功为止。如果启动探测失败，`kubelet` 将杀死容器，而容器依其 [重启策略](https://kubernetes.io/zh/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy)进行重启。 如果容器没有提供启动探测，则默认状态为 `Success`。 | ### 5.2.2 Probe探测方式 | 方式 | 说明 | | ------- | ------------------------------------------------------------ | | Exec | 执行命令 | | HTTPGet | http请求某一个URL路径 | | TCP | tcp连接某一个端口 | | gRPC | 使用 [gRPC](https://grpc.io/) 执行一个远程过程调用。 目标应该实现 [gRPC健康检查](https://grpc.io/grpc/core/md_doc_health-checking.html)。 如果响应的状态是 "SERVING"，则认为诊断成功。 gRPC 探针是一个 alpha 特性，只有在你启用了 "GRPCContainerProbe" [特性门控](https://kubernetes.io/zh/docs/reference/command-line-tools-reference/feature-gate/)时才能使用。 | ### 5.2.3 liveness-exec案例 1, 准备YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-liveness-exec.yml apiVersion: v1 kind: Pod metadata: name: liveness-exec namespace: default spec: containers: - name: liveness image: busybox imagePullPolicy: IfNotPresent args: - /bin/sh - -c - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 # pod启动延迟5秒后探测 periodSeconds: 5 # 每5秒探测1次 ~~~ 2, 应用YAML文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-liveness-exec.yml ~~~ 3, 通过下面的命令观察 ~~~powershell [root@k8s-master1 ~]# kubectl describe pod liveness-exec ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 40s default-scheduler Successfully assigned default/liveness-exec to k8s-worker1 Normal Pulled 38s kubelet Container image "busybox" already present on machine Normal Created 37s kubelet Created container liveness Normal Started 37s kubelet Started container liveness Warning Unhealthy 3s kubelet Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory 看到40s前被调度以k8s-worker1节点,3s前健康检查出问题 ~~~ 4, 过几分钟再观察 ~~~powershell [root@k8s-master1 ~]# kubectl describe pod liveness-exec ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m42s default-scheduler Successfully assigned default/liveness-exec to k8s-worker1 Normal Pulled 70s (x3 over 3m40s) kubelet Container image "busybox" already present on machine Normal Created 70s (x3 over 3m39s) kubelet Created container liveness Normal Started 69s (x3 over 3m39s) kubelet Started container liveness Warning Unhealthy 26s (x9 over 3m5s) kubelet Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory Normal Killing 26s (x3 over 2m55s) kubelet Container liveness failed liveness probe, will be restarted ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE liveness-exec 1/1 Running 3 4m12s 看到重启3次,慢慢地重启间隔时间会越来越长 ~~~ ### 拓展: 容器重启策略验证 ~~~powershell apiVersion: v1 kind: Pod metadata: name: liveness-exec namespace: default spec: restartPolicy: Never # 把容器重启策略由默认的always改为Never containers: - name: liveness image: busybox imagePullPolicy: IfNotPresent args: - /bin/sh - -c - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5 ~~~ 请自行验证 验证结果为: * 容器健康检查出现问题后，不再重启，也不会继续sleep 600秒，而是直接关闭了 ### 5.2.4 liveness-httpget案例 1, 编写YMAL文件 ~~~powershell [root@k8s-master1 ~]# vim pod-liveness-httpget.yml apiVersion: v1 kind: Pod metadata: name: liveness-httpget namespace: default spec: containers: - name: liveness image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: # 指定容器端口，这一段不写也行，端口由镜像决定 - name: http # 自定义名称，不需要与下面的port: http对应 containerPort: 80 # 类似dockerfile里的expose 80 livenessProbe: httpGet: # 使用httpGet方式 port: http # http协议,也可以直接写80端口 path: /index.html # 探测家目录下的index.html initialDelaySeconds: 3 # 延迟3秒开始探测 periodSeconds: 5 # 每隔5s钟探测一次 ~~~ 2, 应用YAML文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-liveness-httpget.yml ~~~ 3, 验证查看 ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE liveness-httpget 1/1 Running 0 9s ~~~ 4, 交互删除nginx里的主页文件 ~~~powershell [root@k8s-master1 ~]# kubectl exec -it liveness-httpget -- rm -rf /usr/share/nginx/html/index.html ~~~ 5, 验证查看会发现 ![1564975961146](../../img/kubernetes/kubernetes_core/liveness-httpget.png) ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE liveness-httpget 1/1 Running 1 11m 只restart一次 ~~~ ### 5.2.5 liveness-tcp案例 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-liveness-tcp.yml apiVersion: v1 kind: Pod metadata: name: liveness-tcp namespace: default spec: containers: - name: liveness image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 livenessProbe: tcpSocket: # 使用tcp连接方式 port: 80 # 连接80端口进行探测 initialDelaySeconds: 3 periodSeconds: 5 ~~~ 2, 应用YAML文件创建pod ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-liveness-tcp.yml pod/liveness-tcp created ~~~ 3, 查看验证 ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE liveness-tcp 1/1 Running 0 14s ~~~ 4, 交互关闭nginx ~~~powershell [root@k8s-master1 ~]# kubectl exec -it liveness-tcp -- /usr/sbin/nginx -s stop ~~~ 5, 再次验证查看 ![1564976660465](../../img/kubernetes/kubernetes_core/liveness-tcp.png) ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE liveness-tcp 1/1 Running 1 5m13s 也只重启1次,重启后重新初始化了 ~~~ ### 5.2.6 readiness案例 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-readiness-httpget.yml apiVersion: v1 kind: Pod metadata: name: readiness-httpget namespace: default spec: containers: - name: readiness image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 readinessProbe: # 这里由liveness换成了readiness httpGet: port: http path: /index.html initialDelaySeconds: 3 periodSeconds: 5 ~~~ 2, 应用YAML文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-readiness-httpget.yml pod/readiness-httpget created ~~~ 3, 验证查看 ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE readiness-httpget 1/1 Running 0 10s ~~~ 4, 交互删除nginx主页 ~~~powershell [root@k8s-master1 ~]# kubectl exec -it readiness-httpget -- rm -rf /usr/share/nginx/html/index.html ~~~ 5, 再次验证 ![1564978173894](../../img/kubernetes/kubernetes_core/readiness-httpget.png) ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE readiness-httpget 0/1 Running 0 2m49s READY状态为0/1 ~~~ 6, 交互创建nginx主页文件再验证 ~~~powershell [root@k8s-master1 ~]# kubectl exec -it readiness-httpget -- touch /usr/share/nginx/html/index.html ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE readiness-httpget 1/1 Running 0 3m10s READY状态又为1/1了 ~~~ ### 5.2.7 readiness+liveness综合案例 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-readiness-liveiness.yml apiVersion: v1 kind: Pod metadata: name: readiness-liveness-httpget namespace: default spec: containers: - name: readiness-liveness image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 livenessProbe: httpGet: port: http path: /index.html initialDelaySeconds: 1 periodSeconds: 3 readinessProbe: httpGet: port: http path: /index.html initialDelaySeconds: 5 periodSeconds: 5 ~~~ 2, 应用YAML文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-readiness-liveiness.yml pod/readiness-liveness-httpget created ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get pod |grep readiness-liveness-httpget NAME READY STATUS RESTARTS AGE readiness-liveness-httpget 0/1 Running 0 6s ~~~ ## 5.3 post-start 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pod-poststart.yml apiVersion: v1 kind: Pod metadata: name: poststart namespace: default spec: containers: - name: poststart image: nginx:1.15-alpine imagePullPolicy: IfNotPresent lifecycle: # 生命周期事件 postStart: exec: command: ["mkdir","-p","/usr/share/nginx/html/haha"] ~~~ 2, 应用YMAL文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pod-poststart.yml ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE poststart 1/1 Running 0 25s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl exec -it poststart -- ls /usr/share/nginx/html -l total 8 -rw-r--r-- 1 root root 494 Apr 16 13:08 50x.html drwxr-xr-x 2 root root 6 Aug 5 05:33 haha 有创建此目录 -rw-r--r-- 1 root root 612 Apr 16 13:08 index.html ~~~ ## 5.4 pre-stop 容器终止前执行的命令 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim prestop.yml apiVersion: v1 kind: Pod metadata: name: prestop namespace: default spec: containers: - name: prestop image: nginx:1.15-alpine imagePullPolicy: IfNotPresent lifecycle: # 生命周期事件 preStop: # preStop exec: command: ["/bin/sh","-c","sleep 60000000"] # 容器终止前sleep 60000000秒 ~~~ 2, 应用YAML文件创建pod ~~~powershell [root@k8s-master1 ~]# kubectl apply -f prestop.yml pod/prestop created ~~~ 3, 删除pod验证 ~~~powershell [root@k8s-master1 ~]# kubectl delete -f prestop.yml pod "prestop" deleted 会在这一步等待一定的时间(大概30s-60s左右)才能删除,说明验证成功 ~~~ **结论:** 当出现特殊情况不能正常销毁pod时,大概等待30秒会强制终止 ## 5.5 pod故障排除 | 状态 | 描述 | | ----------------- | ------------------------------------------------------------ | | Pending（悬决） | Pod 已被 Kubernetes 系统接受，但有一个或者多个容器尚未创建亦未运行。此阶段包括等待 Pod 被调度的时间和通过网络下载镜像的时间。 | | Running（运行中） | pod已经绑定到一个节点，并且已经创建了所有容器。至少有一个容器正在运行中，或正在启动或重新启动。 | | completed（完成） | Pod中的所有容器都已成功终止，不会重新启动。 | | Failed（失败） | Pod的所有容器均已终止，且至少有一个容器已在故障中终止。也就是说，容器要么以非零状态退出，要么被系统终止。 | | Unknown（未知） | 由于某种原因apiserver无法获得Pod的状态，通常是由于Master与Pod所在主机kubelet通信时出错。 | | CrashLoopBackOff | 多见于CMD语句错误或者找不到container入口语句导致了快速退出,可以用kubectl logs 查看日志进行排错 | - kubectl describe pod pod名 - kubectl logs pod [-c CONTAINER] - kubectl exec POD [-c CONTAINER] --COMMAND [args...] # kubernetes核心概念 Controller # 一、pod控制器controller ## 1.1 Controller作用及分类 controller用于控制pod 参考: https://kubernetes.io/zh/docs/concepts/workloads/controllers/ ![1564844640416](../../img/kubernetes/kubernetes_core/work_load.png) 控制器主要分为: * Deployments 部署无状态应用，控制pod升级,回退 * ReplicaSet 副本集,控制pod扩容,裁减 * ReplicationController(相当于ReplicaSet的老版本,现在建议使用Deployments加ReplicaSet替代RC) * StatefulSets 部署有状态应用，结合Service、存储等实现对有状态应用部署 * DaemonSet 守护进程集，运行在所有集群节点(包括master), 比如使用filebeat,node_exporter * Jobs 一次性 * Cronjob 周期性 ## 1.2 Deployment ### 1.2.1 Replicaset控制器的功能 - 支持新的基于集合的selector(以前的rc里没有这种功能) - 通过改变Pod副本数量实现Pod的扩容和缩容 ### 1.2.2 Deployment控制器的功能 - Deployment集成了上线部署、滚动升级、创建副本、回滚等功能 - Deployment里包含并使用了ReplicaSet ### 1.2.3 Deployment用于部署无状态应用 无状态应用的特点: - 所有pod无差别 - 所有pod中容器运行同一个image - 所有pod可以运行在集群中任意node上 - 所有pod无启动顺序先后之分 - 随意pod数量扩容或缩容 - 例如简单运行一个静态web程序 ### 1.2.4 创建deployment类型应用 1, 准备YAML文件 ~~~powershell [root@k8s-master1 ~]# vim deployment-nginx.yml apiVersion: apps/v1 kind: Deployment metadata: name: deploy-nginx # deployment名 spec: replicas: 1 # 副本集,deployment里使用了replicaset selector: matchLabels: app: nginx # 匹配的pod标签,表示deployment和rs控制器控制带有此标签的pod template: # 代表pod的配置模板 metadata: labels: app: nginx # pod的标签 spec: containers: # 以下为pod里的容器定义 - name: nginx image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 ~~~ 2, 应用YAML文件创建deployment ~~~powershell [root@k8s-master1 ~]# kubectl apply -f deployment-nginx.yml deployment.apps/deploy-nginx created ~~~ 3, 查看验证 ~~~powershell [root@k8s-master1 ~]# kubectl get deployment # deployment可简写成depoly NAME READY UP-TO-DATE AVAILABLE AGE deploy-nginx 1/1 1 1 19s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE deploy-nginx-6c9764bb69-pbc2h 1/1 Running 0 75s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get replicasets # replicasets可简写成rs NAME DESIRED CURRENT READY AGE deploy-nginx-6c9764bb69 1 1 1 2m6s ~~~ ### 1.2.5 访问deployment 1,查看pod的IP地址 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deploy-nginx-6d9d558bb6-88nr8 1/1 Running 0 39s 10.244.159.155 k8s-master1 pod在k8s-master1节点,pod的IP为10.244.159.155 ~~~ 2, 查看所有集群节点的网卡 ~~~powershell [root@k8s-master1 ~]# ifconfig tunl0 |head -2 tunl0: flags=193 mtu 1480 inet 10.244.159.128 netmask 255.255.255.255 ~~~ ~~~powershell [root@k8s-master2 ~]# ifconfig tunl0 |head -2 tunl0: flags=193 mtu 1480 inet 10.244.224.0 netmask 255.255.255.255 ~~~ ~~~powershell [root@k8s-master3 ~]# ifconfig tunl0 |head -2 tunl0: flags=193 mtu 1480 inet 10.244.135.192 netmask 255.255.255.255 ~~~ ~~~powershell [root@k8s-worker1 ~]# ifconfig tunl0 |head -2 tunl0: flags=193 mtu 1480 inet 10.244.194.64 netmask 255.255.255.255 ~~~ * 可以看到所有集群节点的IP都为`10.244.0.0/16`这个大网段内的子网 3, 在任意集群节点上都可以访问此deploy里pod ~~~powershell # curl 10.244.159.155 结果是任意集群节点都可以访问这个POD,但集群外部是不能访问的 ~~~ ### 1.2.6 删除deployment中的pod 1, 删除pod（**注意: 是删除deployment中的pod**） ~~~powershell [root@k8s-master1 ~]# kubectl delete pod deploy-nginx-6c9764bb69-pbc2h pod "deploy-nginx-6c9764bb69-pbc2h" deleted ~~~ 2, 再次查看,发现又重新启动了一个pod(**节点由k8s-master1转为k8s-worker1 了,IP地址也变化了**) ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deploy-nginx-6d9d558bb6-f2t6r 1/1 Running 0 28s 10.244.194.94 k8s-worker1 ~~~ 也就是说**==pod的IP不是固定的==**,比如把整个集群关闭再启动,pod也会自动启动,但是**IP地址也会变化** **既然IP地址不是固定的,所以需要一个固定的访问endpoint给用户,那么这种方式就是service.** ### 1.2.7 pod版本升级 查看帮助 ~~~powershell [root@k8s-master1 ~]# kubectl set image -h ~~~ 1, 升级前验证nginx版本 ~~~powershell [root@k8s-master1 ~]# kubectl describe pods deploy-nginx-6d9d558bb6-f2t6r | grep Image: Image: nginx:1.15-alpine [root@k8s-master1 ~]# kubectl exec deploy-nginx-6d9d558bb6-f2t6r -- nginx -v nginx version: nginx/1.15.12 ~~~ 2, 升级为1.16版 ~~~powershell [root@k8s-master1 ~]# kubectl set image deployment deploy-nginx nginx=nginx:1.16-alpine --record deployment.apps/deploy-nginx image updated ~~~ 说明: * `deployment deploy-nginx`代表名为deploy-nginx的deployment * `nginx=nginx:1.16-alpine`前面的nginx为容器名 * --record 表示会记录 **容器名怎么查看?** * `kubectl describe pod pod名`查看 * `kubectl edit deployment deployment名`来查看容器名 * `kubectl get deployment deployment名 -o yaml`来查看容器名 3, 验证 如果升级的pod数量较多，则需要一定时间，可通过下面命令查看是否已经成功 ~~~powershell [root@k8s-master1 ~]# kubectl rollout status deployment deploy-nginx deployment "deploy-nginx" successfully rolled out ~~~ 验证 pod ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE deploy-nginx-5f4749c8c8-nskp9 1/1 Running 0 104s 更新后,后面的id变了 ~~~ 验证版本 ~~~powershell [root@k8s-master1 ~]# kubectl describe pod deploy-nginx-5f4749c8c8-nskp9 |grep Image: Image: nginx:1.16-alpine 升级为1.16了 [root@k8s-master1 ~]# kubectl exec deploy-nginx-5f4749c8c8-nskp9 -- nginx -v nginx version: nginx/1.16.1 升级为1.16了 ~~~ **练习:** 再将nginx1升级为1.17版 ~~~powershell [root@k8s-master1 ~]# kubectl set image deployment deploy-nginx nginx=nginx:1.17-alpine --record deployment.apps/deploy-nginx image updated ~~~ ### 1.2.8 pod版本回退 1, 查看版本历史信息 ~~~powershell [root@k8s-master1 ~]# kubectl rollout history deployment deploy-nginx deployment.apps/deploy-nginx REVISION CHANGE-CAUSE 1 原1.15版 2 kubectl set image deployment deploy-nginx nginx=nginx:1.16-alpine --record=true 3 kubectl set image deployment deploy-nginx nginx=nginx:1.17-alpine --record=true ~~~ 2, 定义要回退的版本（还需要执行才是真的回退版本) ~~~powershell [root@k8s-master1 ~]# kubectl rollout history deployment deploy-nginx --revision=1 deployment.apps/deploy-nginx with revision #1 Pod Template: Labels: app=nginx pod-template-hash=6c9764bb69 Containers: nginx: Image: nginx:1.15-alpine 可以看到这是要回退的1.15版本 Port: 80/TCP Host Port: 0/TCP Environment: Mounts: Volumes: ~~~ 3, 执行回退 ~~~powershell [root@k8s-master1 ~]# kubectl rollout undo deployment deploy-nginx --to-revision=1 deployment.apps/deploy-nginx rolled back ~~~ 4, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl rollout history deployment deploy-nginx deployment.apps/deploy-nginx REVISION CHANGE-CAUSE 2 kubectl set image deployment deploy-nginx nginx=nginx:1.16-alpine --record=true 3 kubectl set image deployment deploy-nginx nginx=nginx:1.17-alpine --record=true 4 回到了1.15版,但revision的ID变了 ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE deploy-nginx-6c9764bb69-zgwpj 1/1 Running 0 54s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl describe pod deploy-nginx-6c9764bb69-zgwpj |grep Image: Image: nginx:1.15-alpine 回到了1.15版 [root@k8s-master1 ~]# kubectl exec deploy-nginx-6c9764bb69-zgwpj -- nginx -v nginx version: nginx/1.15.12 回到了1.15版 ~~~ ### 1.2.9 副本扩容 查看帮助 ~~~powershell [root@k8s-master1 ~]# kubectl scale -h ~~~ 1, 扩容为2个副本 ~~~powershell [root@k8s-master1 ~]# kubectl scale deployment deploy-nginx --replicas=2 deployment.apps/deploy-nginx scaled ~~~ 2, 查看 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deploy-nginx-6d9d558bb6-4c64l 1/1 Running 0 27s 10.244.159.157 k8s-master1 deploy-nginx-6d9d558bb6-hkq2b 1/1 Running 0 71s 10.244.194.95 k8s-worker1 在两个node节点上各1个pod ~~~ 3, 继续扩容(我们这里只有2个node,但是可以大于node节点数据) ~~~powershell [root@master ~]# kubectl scale deployment deploy-nginx --replicas=4 deployment.extensions/nginx1 scaled ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deploy-nginx-6d9d558bb6-4c64l 1/1 Running 0 87s 10.244.159.157 k8s-master1 deploy-nginx-6d9d558bb6-586dr 1/1 Running 0 31s 10.244.135.197 k8s-master3 deploy-nginx-6d9d558bb6-hkq2b 1/1 Running 0 2m11s 10.244.194.95 k8s-worker1 deploy-nginx-6d9d558bb6-kvgsc 1/1 Running 0 31s 10.244.224.13 k8s-master2 ~~~ ### 1.2.10 副本裁减 1, 指定副本数为1进行裁减 ~~~powershell [root@k8s-master1 ~]# kubectl scale deployment deploy-nginx --replicas=1 deployment.apps/deploy-nginx scaled ~~~ 2, 查看验证 ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE deploy-nginx-6d9d558bb6-hkq2b 1/1 Running 0 2m56s ~~~ ### 1.2.11 多副本滚动更新 1, 先扩容多点副本 ~~~powershell [root@k8s-master1 ~]# kubectl scale deployment deploy-nginx --replicas=16 deployment.apps/deploy-nginx scaled ~~~ 2, 验证 ~~~powershell [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx1-7d9b8757cf-2hd48 1/1 Running 0 61s nginx1-7d9b8757cf-5m72n 1/1 Running 0 61s nginx1-7d9b8757cf-5w2xr 1/1 Running 0 61s nginx1-7d9b8757cf-5wmdh 1/1 Running 0 61s nginx1-7d9b8757cf-6szjj 1/1 Running 0 61s nginx1-7d9b8757cf-9dgsw 1/1 Running 0 61s nginx1-7d9b8757cf-dc7qj 1/1 Running 0 61s nginx1-7d9b8757cf-l52pr 1/1 Running 0 61s nginx1-7d9b8757cf-m7rt4 1/1 Running 0 26m nginx1-7d9b8757cf-mdkj2 1/1 Running 0 61s nginx1-7d9b8757cf-s79kp 1/1 Running 0 61s nginx1-7d9b8757cf-shhvk 1/1 Running 0 61s nginx1-7d9b8757cf-sv8gb 1/1 Running 0 61s nginx1-7d9b8757cf-xbhf4 1/1 Running 0 61s nginx1-7d9b8757cf-zgdgd 1/1 Running 0 61s nginx1-7d9b8757cf-zzljl 1/1 Running 0 61s nginx2-559567f789-8hstz 1/1 Running 1 114m ~~~ 3, 滚动更新 ~~~powershell [root@k8s-master1 ~]# kubectl set image deployment deploy-nginx nginx=nginx:1.17-alpine --record deployment.apps/deploy-nginx image updated ~~~ 4, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl rollout status deployment deploy-nginx ...... Waiting for deployment "deploy-nginx" rollout to finish: 13 of 16 updated replicas are available... Waiting for deployment "deploy-nginx" rollout to finish: 14 of 16 updated replicas are available... Waiting for deployment "deploy-nginx" rollout to finish: 15 of 16 updated replicas are available... deployment "deploy-nginx" successfully rolled out ~~~ ### 1.2.12 删除deployment 如果使用 `kubectl delete deployment deploy-nginx `命令删除deployment,那么里面的pod也会被自动删除 ## 1.3 Replicaset 1, 编写YAML文件 ~~~powershell [root@master ~]# vim rs-nginx.yml apiVersion: apps/v1 kind: ReplicaSet metadata: name: rs-nginx namespace: default spec: # replicaset的spec replicas: 2 # 副本数 selector: # 标签选择器,对应pod的标签 matchLabels: app: nginx # 匹配的label template: metadata: name: nginx # pod名 labels: # 对应上面定义的标签选择器selector里面的内容 app: nginx spec: # pod的spec containers: - name: nginx image: nginx:1.15-alpine ports: - name: http containerPort: 80 ~~~ 2, 应用YAML文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f rs-nginx.yml replicaset.apps/rs-nginx created ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get rs NAME DESIRED CURRENT READY AGE rs-nginx 2 2 2 26s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE rs-nginx-7j9hz 1/1 Running 0 44s rs-nginx-pncsk 1/1 Running 0 43s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get deployment No resources found. 找不到deployment,说明创建rs并没有创建deployment ~~~ # 二、pod控制器Controller进阶 ## 2.1 DaemonSet ### 2.1.1 DaemonSet介绍 - DaemonSet能够让所有（或者特定）的节点运行同一个pod。 - 当节点加入到K8S集群中，pod会被（DaemonSet）调度到该节点上运行，当节点从K8S集群中被移除，被DaemonSet调度的pod会被移除 - 如果删除DaemonSet，所有跟这个DaemonSet相关的pods都会被删除。 - 如果一个DaemonSet的Pod被杀死、停止、或者崩溃，那么DaemonSet将会重新创建一个新的副本在这台计算节点上。 - DaemonSet一般应用于日志收集、监控采集、分布式存储守护进程等 ### 2.1.2 DaemonSet应用案例 1, 编写YAML文件 ~~~~powershell [root@master ~]# vim daemonset-nginx.yml apiVersion: apps/v1 kind: DaemonSet metadata: name: daemonset-nginx spec: selector: matchLabels: name: nginx-ds template: metadata: labels: name: nginx-ds spec: tolerations: # tolerations代表容忍 - key: node-role.kubernetes.io/master # 能容忍的污点key effect: NoSchedule # kubectl explain pod.spec.tolerations查看(能容忍的污点effect) containers: - name: nginx image: nginx:1.15-alpine imagePullPolicy: IfNotPresent resources: # resources资源限制是为了防止master节点的资源被占太多(根据实际情况配置) limits: memory: 100Mi requests: memory: 100Mi ~~~~ 2, apply应用YAML文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f daemonset-nginx.yml daemonset.apps/daemonset-nginx created ~~~ 3, 验证 ~~~powershell [root@master ~]# kubectl get daemonset # daemonset可简写为ds [root@k8s-master1 ~]# kubectl get ds NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset-nginx 4 4 4 4 4 114s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES daemonset-nginx-94z6d 1/1 Running 0 6s 10.244.194.104 k8s-worker1 daemonset-nginx-hs9mk 1/1 Running 0 6s 10.244.135.206 k8s-master3 daemonset-nginx-jrcf5 1/1 Running 0 6s 10.244.159.167 k8s-master1 daemonset-nginx-sslpl 1/1 Running 0 6s 10.244.224.22 k8s-master2 k8s集群中每个节点都会运行一个pod ~~~ ## 2.2 Job ### 2.2.1 Job介绍 - 对于ReplicaSet而言，它希望pod保持预期数目、持久运行下去，除非用户明确删除，否则这些对象一直存在，它们针对的是耐久性任务，如web服务等。 - 对于非耐久性任务，比如压缩文件，任务完成后，pod需要结束运行，不需要pod继续保持在系统中，这个时候就要用到Job。 - Job负责批量处理短暂的一次性任务 (short lived one-off tasks)，即仅执行一次的任务，它保证批处理任务的一个或多个Pod成功结束。 ### 2.2.2 Job应用案例 #### 2.2.2.1 计算圆周率2000位 1, 编写YAML文件 ~~~powershell [root@master ~]# vim job1.yml apiVersion: batch/v1 kind: Job metadata: name: pi # job名 spec: template: metadata: name: pi # pod名 spec: containers: - name: pi # 容器名 image: perl # 此镜像有800多M,可提前导入到所有节点,也可能指定导入到某一节点然后指定调度到此节点 imagePullPolicy: IfNotPresent command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] restartPolicy: Never # 执行完后不再重启 ~~~ 2, 应用YAML文件创建job ~~~powershell [root@master ~]# kubectl apply -f job1.yml job.batch/pi created ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get jobs NAME COMPLETIONS DURATION AGE pi 1/1 11s 18s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE pi-tjq9b 0/1 Completed 0 27s Completed状态,也不再是ready状态 ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl logs pi-tjq9b 3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901 ~~~ #### 2.2.2.2 创建固定次数job 1, 编写YAML文件 ~~~powershell [root@master ~]# vim job2.yml apiVersion: batch/v1 kind: Job metadata: name: busybox-job spec: completions: 10 # 执行job的次数 parallelism: 1 # 执行job的并发数 template: metadata: name: busybox-job-pod spec: containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ["echo", "hello"] restartPolicy: Never ~~~ 2, 应用YAML文件创建job ~~~powershell [root@k8s-master1 ~]# kubectl apply -f job2.yml job.batch/busybox-job created ~~~ 3, 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get job NAME COMPLETIONS DURATION AGE busybox-job 2/10 9s 9s [root@k8s-master1 ~]# kubectl get job NAME COMPLETIONS DURATION AGE busybox-job 3/10 12s 12s [root@k8s-master1 ~]# kubectl get job NAME COMPLETIONS DURATION AGE busybox-job 4/10 15s 15s [root@k8s-master1 ~]# kubectl get job NAME COMPLETIONS DURATION AGE busybox-job 10/10 34s 48s 34秒左右结束 ~~~ ~~~powershell [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE busybox-job-5zn6l 0/1 Completed 0 34s busybox-job-cm9kw 0/1 Completed 0 29s busybox-job-fmpgt 0/1 Completed 0 38s busybox-job-gjjvh 0/1 Completed 0 45s busybox-job-krxpd 0/1 Completed 0 25s busybox-job-m2vcq 0/1 Completed 0 41s busybox-job-ncg78 0/1 Completed 0 47s busybox-job-tbzz8 0/1 Completed 0 51s busybox-job-vb99r 0/1 Completed 0 21s busybox-job-wnch7 0/1 Completed 0 32s ~~~ #### 2.2.2.3 一次性备份MySQL数据库 > 通过Job控制器创建应用备份MySQL数据库 ##### 2.2.2.3.1 MySQL数据库准备 ~~~powershell [root@nginx jobcontroller]# cat 00_mysql.yaml apiVersion: v1 kind: Service metadata: name: mysql-test namespace: default spec: ports: - port: 3306 name: mysql clusterIP: None selector: app: mysql-dump --- apiVersion: apps/v1 kind: StatefulSet metadata: name: db namespace: default spec: selector: matchLabels: app: mysql-dump serviceName: "mysql-test" template: metadata: labels: app: mysql-dump spec: nodeName: k8s-master3 containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "abc123" ports: - containerPort: 3306 volumeMounts: - mountPath: "/var/lib/mysql" name: mysql-data volumes: - name: mysql-data hostPath: path: /opt/mysqldata ~~~ ##### 2.2.2.3.2 创建用于实现任务的资源清单文件 ~~~powershell [root@nginx jobcontroller]# cat 03_job.yaml apiVersion: batch/v1 kind: Job metadata: name: mysql-dump spec: template: metadata: name: mysql-dump spec: nodeName: k8s-master2 containers: - name: mysql-dump image: mysql:5.7 command: ["/bin/sh","-c","mysqldump --host=mysql-test -uroot -pabc123 --databases mysql > /root/mysql2022.sql"] volumeMounts: - mountPath: "/root" name: mysql-data restartPolicy: Never volumes: - name: mysql-data hostPath: path: /opt/mysqldump ~~~ ## 2.3 CronJob ### 2.3.1 CronJob介绍 * 类似于Linux系统的crontab，在指定的时间周期运行相关的任务 * 时间格式：分时日月周 ### 2.3.2 CronJob应用案例 #### 2.3.2.1 周期性输出字符 1, 编写YAML文件 ~~~powershell [root@k8s-master1 ~]# vim cronjob.yml apiVersion: batch/v1beta1 kind: CronJob metadata: name: cronjob1 spec: schedule: "* * * * *" # 分时日月周 jobTemplate: spec: template: spec: containers: - name: hello image: busybox args: - /bin/sh - -c - date; echo hello kubernetes imagePullPolicy: IfNotPresent restartPolicy: OnFailure ~~~ 2, 应用YAML文件创建cronjob ~~~powershell [root@k8s-master1 ~]# kubectl apply -f cronjob.yml cronjob.batch/cronjob1 created ~~~ 3, 查看验证 ~~~powershell [root@k8s-master1 ~]# kubectl get cronjob NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob1 * * * * * False 0 21s ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE cronjob-1564993080-qlbgv 0/1 Completed 0 2m10s cronjob-1564993140-zbv7f 0/1 Completed 0 70s cronjob-1564993200-gx5xz 0/1 Completed 0 10s 看AGE时间,每分钟整点执行一次 ~~~ #### 2.3.2.2 周期性备份MySQL数据库 ##### 2.3.2.2.1 MySQL数据库准备 ~~~powershell [root@nginx jobcontroller]# cat 00_mysql.yaml apiVersion: v1 kind: Service metadata: name: mysql-test namespace: default spec: ports: - port: 3306 name: mysql clusterIP: None selector: app: mysql-dump --- apiVersion: apps/v1 kind: StatefulSet metadata: name: db namespace: default spec: selector: matchLabels: app: mysql-dump serviceName: "mysql-test" template: metadata: labels: app: mysql-dump spec: nodeName: worker03 containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "abc123" ports: - containerPort: 3306 volumeMounts: - mountPath: "/var/lib/mysql" name: mysql-data volumes: - name: mysql-data hostPath: path: /opt/mysqldata ~~~ ##### 2.3.2.2.2 Cronjob控制器类型应用资源清单文件 ~~~powershell [root@nginx jobcontroller]# cat 05_cronjob.yaml apiVersion: batch/v1beta1 kind: CronJob metadata: name: mysql-dump spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: nodeName: worker02 containers: - name: c1 image: mysql:5.7 command: ["/bin/sh","-c","mysqldump --host=mysql-test -uroot -pabc123 --databases mysql > /root/mysql`date +%Y%m%d%H%M`.sql"] volumeMounts: - name: mysql-data mountPath: "/root" restartPolicy: Never volumes: - name: mysql-data hostPath: path: /opt/mysqldump ~~~ # Kubernetes核心概念 Controller之StatefulSet控制器 # 一、StatefulSet控制器作用 - StatefulSet 是用来管理有状态应用的控制器。 - StatefulSet 用来管理某Pod集合的部署和扩缩， 并为这些 Pod 提供持久存储和持久标识符。 - 参考: https://kubernetes.io/zh/docs/concepts/workloads/controllers/statefulset/ # 二、无状态应用与有状态应用 ## 2.1 无状态应用 - 如nginx * 请求本身包含了响应端为响应这一请求所需的全部信息。每一个请求都像首次执行一样，不会依赖之前的数据进行响应。 * 不需要持久化的数据 * 无状态应用的多个实例之间互不依赖，可以无序的部署、删除或伸缩 ## 2.2 有状态应用 - 如mysql * 前后请求有关联与依赖 * 需要持久化的数据 * 有状态应用的多个实例之间有依赖，不能相互替换：无论怎么调度，每个 Pod 都有一个永久不变的 ID。 # 三、StatefulSet的特点 - 稳定的、唯一的网络标识符。 (通过headless服务实现) - 稳定的、持久的存储。 (通过PV，PVC，storageclass实现) - 有序的、优雅的部署和缩放。 - 有序的、自动的滚动更新。 # 四、StatefulSet的YAML组成 需要三个组成部分: 1. headless service: 实现稳定，唯一的网络标识 2. statefulset类型资源: 写法和deployment几乎一致，就是类型不一样 3. volumeClaimTemplate : 指定存储卷 # 五、创建StatefulSet应用 - 参考: https://kubernetes.io/zh/docs/tutorials/stateful-application/basic-stateful-set/ ## 5.1 编辑YAML资源清单文件 > 创建statelfulset应用来调用名为nfs-client的storageclass,以实现动态供给 ```powershell [root@k8s-master1 ~]# vim nginx-storageclass-nfs.yml apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None # 无头服务 selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web # statefulset的名称 spec: serviceName: "nginx" replicas: 3 # 3个副本 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15-alpine ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "nfs-client" # 与前面定义的storageclass名称对应 resources: requests: storage: 1Gi ``` ```powershell [root@k8s-master1 ~]# kubectl apply -f nginx-storageclass-nfs.yml service/nginx created statefulset.apps/web created ``` ## 5.2 应用部署后验证 ### 5.2.1 验证pod > 产生了3个pod ```powershell [root@k8s-master1 ~]# kubectl get pods |grep web web-0 1/1 Running 0 1m15s web-1 1/1 Running 0 1m7s web-2 1/1 Running 0 57s ``` ### 5.2.2 验证pv > 自动产生了3个pv ```powershell [root@k8s-master1 ~] # kubectl get pv pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6 1Gi RWO Delete Bound default/www-web-0 nfs-client 3m pvc-3114be74-5969-40eb-aeb3-87a3b9ae17bc 1Gi RWO Delete Bound default/www-web-1 nfs-client 2m pvc-43afb71d-1d02-4699-b00c-71679fd75fc3 1Gi RWO Delete ound default/www-web-2 nfs-client 2m ``` ### 5.2.3 验证pvc > 自动产生了3个PVC ```powershell [root@k8s-master1 ~]# kubectl get pvc |grep web www-web-0 Bound pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6 1Gi RWO nfs-client 3m www-web-1 Bound pvc-3114be74-5969-40eb-aeb3-87a3b9ae17bc 1Gi RWO nfs-client 2m www-web-2 Bound pvc-43afb71d-1d02-4699-b00c-71679fd75fc3 1Gi RWO nfs-client 2m ``` ### 5.2.4 验证nfs服务目录 在nfs服务器（这里为hostos)的共享目录中发现自动产生了3个子目录 ```powershell [root@nfsserver ~]# ls /data/nfs/ default-www-web-0-pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6 default-www-web-2-pvc-43afb71d-1d02-4699-b00c-71679fd75fc3 default-www-web-1-pvc-3114be74-5969-40eb-aeb3-87a3b9ae17bc ``` 3个子目录默认都为空目录 ```powershell [root@nfsserver ~]# tree /data/nfs/ /data/nfs/ ├── default-www-web-0-pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6 ├── default-www-web-1-pvc-3114be74-5969-40eb-aeb3-87a3b9ae17bc └── default-www-web-2-pvc-43afb71d-1d02-4699-b00c-71679fd75fc3 ``` ### 5.2.5 验证存储持久性 在3个pod中其中一个创建一个主页文件 ```powershell [root@k8s-master1 ~]# kubectl exec -it web-0 -- /bin/sh / # echo "haha" > /usr/share/nginx/html/index.html / # exit ``` 在nfs服务器上发现文件被创建到了对应的目录中 ```powershell [root@nfsserver ~]# tree /data/nfs/ /data/nfs/ ├── default-www-web-0-pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6 │   └── index.html # 此目录里多了index.html文件，对应刚才在web-0的pod中的创建 ├── default-www-web-1-pvc-3114be74-5969-40eb-aeb3-87a3b9ae17bc └── default-www-web-2-pvc-43afb71d-1d02-4699-b00c-71679fd75fc3 [root@nfsserver ~]# cat /data/nfs/default-www-web-0-pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6/index.html haha # 文件内的内容也与web-0的pod中创建的一致 ``` 删除web-0这个pod,再验证 ```powershell [root@k8s-master1 ~]# kubectl delete pod web-0 pod "web-0" deleted [root@k8s-master1 ~]# kubectl get pods |grep web # 因为控制器的原因，会迅速再拉起web-0这个pod web-0 1/1 Running 0 9s # 时间上看到是新拉起的pod web-1 1/1 Running 0 37m web-2 1/1 Running 0 37m [root@k8s-master1 ~]# kubectl exec -it web-0 -- cat /usr/share/nginx/html/index.html haha # 新拉起的pod仍然是相同的存储数据 [root@nfsserver ~]# cat /data/nfs/default-www-web-0-pvc-2436b20d-1be3-4c2e-87a9-5533e5c5e2c6/index.html haha # nfs服务器上的数据还在 ``` **结论: 说明数据可持久化** ### 5.2.6 访问验证 ~~~powershell 验证Coredns是否可用 # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 6d23h # dig -t a www.baidu.com @10.96.0.10 ~~~ ~~~powershell # dig -t a nginx.default.svc.cluster.local. @10.96.0.10 .... ;; ANSWER SECTION: nginx.default.svc.cluster.local. 30 IN A 10.224.194.75 nginx.default.svc.cluster.local. 30 IN A 10.224.159.141 nginx.default.svc.cluster.local. 30 IN A 10.224.126.6 ~~~ ~~~powershell # dig -t a web-0.nginx.default.svc.cluster.local. @10.96.0.10 ~~~ ~~~powershell 在kubernetes集群内创建pod访问 # kubectl run -it busybox --image=radial/busyboxplus / # curl nginx.default.svc.cluster.local. web-0 / # curl web-0.nginx.default.svc.cluster.local. web-0 ~~~ # 六、已部署应用滚动更新(含金丝雀发布) >它将按照与 Pod 终止相同的顺序（从最大序号到最小序号）进行，每次更新一个 Pod。 > >StatefulSet可以使用partition参数来实现金丝雀更新，partition参数可以控制StatefulSet控制器更新的Pod。下面，我们就进行StatefulSet控制器的金丝雀更新实战。 ~~~powershell kubectl patch sts web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}' ~~~ ~~~powershell 说明： 使用patch参数来指定了StatefulSet控制器的partition参数为2，表示当更新时，只有Pod的编号大于等于2的才更新。 ~~~ ~~~powershell # kubectl exec -it web-0 -- nginx -v nginx version: nginx/1.15.12 ~~~ ~~~powershell kubectl set image sts/web nginx=nginx:latest ~~~ ~~~powershell kubectl get pods -w ~~~ ~~~powershell # kubectl exec -it web-2 -- nginx -v nginx version: nginx/1.21.6 ~~~ ~~~powershell # kubectl get pods -o custom-columns=Name:metadata.name,Image:spec.containers[0].image Name Image web-0 nginx:1.15-alpine web-1 nginx:1.15-alpine web-2 nginx:latest ~~~ **如何实现全部更新呢？** ~~~powershell kubectl patch sts web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}' ~~~ ~~~powershell kubectl set image sts/web nginx=nginx:latest ~~~ ~~~powershell # kubectl get pods -o custom-columns=Name:metadata.name,Image:spec.containers[0].image ~~~ # 七、已部署应用扩容与缩容 在StatefulSet扩容时，会创建一个新的Pod，该Pod与之前的所有Pod都是有顺序的，并且新Pod的序号最大。在缩容时，StatefulSet控制器删除的也是序号最大的Pod。 ~~~powershell # kubectl scale sts web --replicas=4 ~~~ ~~~powershell # kubectl get pods -w ~~~ # kubernetes核心概念 Service # 一、 service作用 使用kubernetes集群运行工作负载时，由于Pod经常处于用后即焚状态，Pod经常被重新生成，因此Pod对应的IP地址也会经常变化，导致无法直接访问Pod提供的服务，Kubernetes中使用了Service来解决这一问题，即在Pod前面使用Service对Pod进行代理，无论Pod怎样变化 ，只要有Label，就可以让Service能够联系上Pod，把PodIP地址添加到Service对应的端点列表（Endpoints）实现对Pod IP跟踪，进而实现通过Service访问Pod目的。 - 通过service为pod客户端提供访问pod方法，即可客户端访问pod入口 - 通过标签动态感知pod IP地址变化等 - 防止pod失联 - 定义访问pod访问策略 - 通过label-selector相关联 - 通过Service实现Pod的负载均衡（TCP/UDP 4层） - 底层实现由kube-proxy通过userspace、iptables、ipvs三种代理模式 # 二、kube-proxy三种代理模式 - kubernetes集群中有三层网络，一类是真实存在的，例如Node Network、Pod Network,提供真实IP地址;一类是虚拟的，例如Cluster Network或Service Network，提供虚拟IP地址，不会出现在接口上，仅会出现在Service当中 - kube-proxy始终watch（监控）kube-apiserver上关于Service相关的资源变动状态，一旦获取相关信息kube-proxy都要把相关信息转化为当前节点之上的，能够实现Service资源调度到特定Pod之上的规则，进而实现访问Service就能够获取Pod所提供的服务 - kube-proxy三种代理模式：UserSpace模式、iptables模式、ipvs模式 ## 2.1 UserSpace模式 userspace 模式是 kube-proxy 使用的第一代模式，该模式在 kubernetes v1.0 版本开始支持使用。 userspace 模式的实现原理图示如下： ![image-20220402102113765](../../img/kubernetes/kubernetes_core/image-20220402102113765.png) kube-proxy 会为每个 Service 随机监听一个端口(proxy port)，并增加一条 iptables 规则。所以通过 ClusterIP:Port 访问 Service 的报文都 redirect 到 proxy port，kube-proxy 从它监听的 proxy port 收到报文以后，走 round robin(默认) 或是 session affinity(会话亲和力，即同一 client IP 都走同一链路给同一 pod 服务)，分发给对应的 pod。 由于 userspace 模式会造成所有报文都走一遍用户态（也就是 Service 请求会先从用户空间进入内核 iptables，然后再回到用户空间，由 kube-proxy 完成后端 Endpoints 的选择和代理工作），需要在内核空间和用户空间转换，流量从用户空间进出内核会带来性能损耗，所以这种模式效率低、性能不高，不推荐使用。 ![](../../img/kubernetes/kubernetes_core/service-userspace.png) ## 2.2 iptables模式 iptables 模式是 kube-proxy 使用的第二代模式，该模式在 kubernetes v1.1 版本开始支持，从 v1.2 版本开始成为 kube-proxy 的默认模式。 iptables 模式的负载均衡模式是通过底层 netfilter/iptables 规则来实现的，通过 Informer 机制 Watch 接口实时跟踪 Service 和 Endpoint 的变更事件，并触发对 iptables 规则的同步更新。 iptables 模式的实现原理图示如下： ![image-20220402102306186](../../img/kubernetes/kubernetes_core/image-20220402102306186.png) 通过图示我们可以发现在 iptables 模式下，kube-proxy 只是作为 controller，而不是 server，真正服务的是内核的 netfilter，体现在用户态的是 iptables。所以整体的效率会比 userspace 模式高。 ![](../../img/kubernetes/kubernetes_core/service-iptables.png) ## 2.3 ipvs模式 ipvs 模式被 kube-proxy 采纳为第三代模式，模式在 kubernetes v1.8 版本开始引入，在 v1.9 版本中处于 beta 阶段，在 v1.11 版本中正式开始使用。 ipvs(IP Virtual Server) 实现了传输层负载均衡，也就是 4 层交换，作为 Linux 内核的一部分。`ipvs`运行在主机上，在真实服务器前充当负载均衡器。ipvs 可以将基于 TCP 和 UDP 的服务请求转发到真实服务器上，并使真实服务器上的服务在单个 IP 地址上显示为虚拟服务。 ipvs 模式的实现原理图示如下： ![image-20220402102614692](../../img/kubernetes/kubernetes_core/image-20220402102614692.png) ![](../../img/kubernetes/kubernetes_core/service-ipvs.png) ipvs 和 iptables 都是基于 netfilter 的，那么 ipvs 模式有哪些更好的性能呢？ - ipvs 为大型集群提供了更好的可拓展性和性能 - ipvs 支持比 iptables 更复杂的负载均衡算法（包括：最小负载、最少连接、加权等） - ipvs 支持服务器健康检查和连接重试等功能 - 可以动态修改 ipset 的集合，即使 iptables 的规则正在使用这个集合 ipvs 依赖于 iptables。ipvs 会使用 iptables 进行包过滤、airpin-masquerade tricks(地址伪装)、SNAT 等功能，但是使用的是 iptables 的扩展 ipset，并不是直接调用 iptables 来生成规则链。通过 ipset 来存储需要 DROP 或 masquerade 的流量的源或目标地址，用于确保 iptables 规则的数量是恒定的，这样我们就不需要关心有多少 Service 或是 Pod 了。 使用 ipset 相较于 iptables 有什么优点呢？iptables 是线性的数据结构，而 ipset 引入了带索引的数据结构，当规则很多的时候，ipset 依然可以很高效的查找和匹配。我们可以将 ipset 简单理解为一个 IP(段) 的集合，这个集合的内容可以是 IP 地址、IP 网段、端口等，iptables 可以直接添加规则对这个“可变的集合进行操作”，这样就可以大大减少 iptables 规则的数量，从而减少性能损耗。 举一个例子，如果我们要禁止成千上万个 IP 访问我们的服务器，如果使用 iptables 就需要一条一条的添加规则，这样会在 iptables 中生成大量的规则；如果用 ipset 就只需要将相关的 IP 地址(网段)加入到 ipset 集合中，然后只需要设置少量的 iptables 规则就可以实现这个目标。 下面的表格是 ipvs 模式下维护的 ipset 表集合： | 设置名称 | 成员 | 用法 | | :----------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | | KUBE-CLUSTER-IP | 所有服务 IP + 端口 | 在 masquerade-all=true 或 clusterCIDR 指定的情况下对 Service Cluster IP 地址进行伪装，解决数据包欺骗问题 | | KUBE-LOOP-BACK | 所有服务 IP + 端口 + IP | 解决数据包欺骗问题 | | KUBE-EXTERNAL-IP | 服务外部 IP + 端口 | 将数据包伪装成 Service 的外部 IP 地址 | | KUBE-LOAD-BALANCER | 负载均衡器入口 IP + 端口 | 将数据包伪装成 Load Balancer 类型的 Service | | KUBE-LOAD-BALANCER-LOCAL | 负载均衡器入口 IP + 端口 以及`externalTrafficPolicy=local` | 接受数据包到 Load Balancer externalTrafficPolicy=local | | KUBE-LOAD-BALANCER-FW | 负载均衡器入口 IP + 端口 以及`loadBalancerSourceRanges` | 使用指定的 loadBalancerSourceRanges 丢弃 Load Balancer 类型 Service 的数据包 | | KUBE-LOAD-BALANCER-SOURCE-CIDR | 负载均衡器入口 IP + 端口 + 源 CIDR | 接受 Load Balancer 类型 Service 的数据包，并指定 loadBalancerSourceRanges | | KUBE-NODE-PORT-TCP | NodePort 类型服务 TCP 端口 | 将数据包伪装成 NodePort（TCP） | | KUBE-NODE-PORT-LOCAL-TCP | NodePort 类型服务 TCP 端口，带有`externalTrafficPolicy=local` | 接受数据包到 NodePort 服务，使用 externalTrafficPolicy=local | | KUBE-NODE-PORT-UDP | NodePort 类型服务 UDP 端口 | 将数据包伪装成 NodePort(UDP) | | KUBE-NODE-PORT-LOCAL-UDP | NodePort 类型服务 UDP 端口，使用`externalTrafficPolicy=local` | 接受数据包到 NodePort 服务，使用 externalTrafficPolicy=local | ## 2.4 iptables与ipvs对比 - iptables - 工作在内核空间 - 优点 - 灵活，功能强大（可以在数据包不同阶段对包进行操作） - 缺点 - 表中规则过多时，响应变慢，即规则遍历匹配和更新，呈线性时延 - ipvs - 工作在内核空间 - 优点 - 转发效率高 - 调度算法丰富：rr，wrr，lc，wlc，ip hash... - 缺点 - 内核支持不全,低版本内核不能使用，需要升级到4.0或5.0以上。 - 使用iptables与ipvs时机 - 1.10版本之前使用iptables(1.1版本之前使用UserSpace进行转发) - 1.11版本之后同时支持iptables与ipvs，默认使用ipvs，如果ipvs模块没有加载时，会自动降级至iptables # 三、 service类型 Service类型决定了访问Service的方法 ## 3.1 service类型 - ClusterIP - 默认，分配一个集群内部可以访问的虚拟IP - NodePort - 在每个Node上分配一个端口作为外部访问入口 - nodePort端口范围为:30000-32767 - LoadBalancer - 工作在特定的Cloud Provider上，例如Google Cloud，AWS，OpenStack - ExternalName - 表示把集群外部的服务引入到集群内部中来，即实现了集群内部pod和集群外部的服务进行通信 ## 3.2 Service参数 - port 访问service使用的端口 - targetPort Pod中容器端口 - nodePort 通过Node实现外网用户访问k8s集群内service (30000-32767) # 四、 Service创建 > Service的创建在工作中有两种方式，一是命令行创建，二是通过资源清单文件YAML文件创建。 ## 4.1 ClusterIP类型 ClusterIP根据是否生成ClusterIP又可分为普通Service和Headless Service Service两类： - 普通Service: 为Kubernetes的Service分配一个集群内部可访问的固定虚拟IP(Cluster IP), 实现集群内的访问。 - Headless Service: 该服务不会分配Cluster IP, 也不通过kube-proxy做反向代理和负载均衡。而是通过DNS提供稳定的网络ID来访问，DNS会将headless service的后端直接解析为pod IP列表。 ![image-20200531203549337](../../img/kubernetes/kubernetes_core/image-20200531203549337.png) ### 4.1.1 普通ClusterIP Service创建 #### 4.1.1.1 命令行创建Service - 创建Deployment类型的应用 ```powershell [root@master01 ~]# cat 01_create_deployment_app_nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server1 spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 ``` - 应用资源清单文件 ~~~powershell [root@master01 ~]# kubectl apply -f 01_create_deployment_app_nginx.yaml ~~~ - 验证Deployment类型的创建情况 ```powershell [root@master01 ~]# kubectl get deployment.apps NAME READY UP-TO-DATE AVAILABLE AGE nginx-server1 2/2 2 2 13s ``` - 创建ClusterIP类型service与Deployment类型应用关联 ```powershell 命令创建service [root@master01 ~]# kubectl expose deployment.apps nginx-server1 --type=ClusterIP --target-port=80 --port=80 ``` ~~~powershell 输出 service/nginx-server1 exposed ~~~ ~~~powershell 说明 expose 创建service deployment.apps 控制器类型 nginx-server1 应用名称，也是service名称 --type=ClusterIP 指定service类型 --target-port=80 指定Pod中容器端口 --port=80 指定service端口 ~~~ #### 4.1.1.2 通过资源清单文件创建Service ~~~powershell [root@master01 ~]# cat 02_create_deployment_app_nginx_with_service.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server1 spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx-smart image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-svc spec: type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 80 selector: app: nginx ~~~ ~~~powershell [root@master01 ~]# kubectl apply -f 02_create_deployment_app_nginx_with_service.yaml ~~~ - 验证 ```powershell 查看service [root@master01 ~]# kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 4d15h nginx-svc ClusterIP 10.101.153.50 80/TCP 3s ``` ```powershell 查看endpoints [root@master01 ~]# kubectl get endpoints NAME ENDPOINTS AGE kubernetes 192.168.122.30:6443 4d15h nginx-svc 172.16.189.74:80,172.16.235.150:80 8s ``` ```powershell 查看Pod [root@master01 ~]# kubectl get pods -l app=nginx NAME READY STATUS RESTARTS AGE nginx-server1-77d4c485d8-gsrmq 1/1 Running 0 12s nginx-server1-77d4c485d8-mmc52 1/1 Running 0 12s ``` #### 4.1.1.3 访问 ```powershell [root@master01 ~]# curl http://10.101.153.50:80 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

``` #### 4.1.1.4 两个pod里做成不同的主页方便测试负载均衡 ```powershell [root@master01 ~]# kubectl exec -it nginx-server1-77d4c485d8-gsrmq -- /bin/bash root@deployment-nginx-6fcfb67547-nv7dn:/# cd /usr/share/nginx/html/ root@deployment-nginx-6fcfb67547-nv7dn:/usr/share/nginx/html# echo web1 > index.html root@deployment-nginx-6fcfb67547-nv7dn:/usr/share/nginx/html# exit exit ``` ```powershell [root@master01 ~]# kubectl exec -it nginx-server1-77d4c485d8-mmc52 -- /bin/bash root@deployment-nginx-6fcfb67547-rqrcw:/# cd /usr/share/nginx/html/ root@deployment-nginx-6fcfb67547-rqrcw:/usr/share/nginx/html# echo web2 > index.html root@deployment-nginx-6fcfb67547-rqrcw:/usr/share/nginx/html# exit exit ``` #### 4.1.1.5 测试 ```powershell [root@master01 ~]# curl 10.101.153.50 或 [root@master01 ~]# while true;do curl 10.101.153.50;sleep 1; done ``` ### 4.1.2 Headless Service - 普通的ClusterIP service是service name解析为cluster ip,然后cluster ip对应到后面的pod ip - Headless service是指service name 直接解析为后面的pod ip #### 4.1.2.1 编写用于创建Deployment控制器类型的资源清单文件 ~~~powershell [root@master01 ~]# cat 03_create_deployment_app_nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server1 spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx-smart image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 ~~~ #### 4.1.2.2 通过资源清单文件创建headless Service ```powershell 编写YAML文件 命令 [root@master ~]# vim 04_headless-service.yml apiVersion: v1 kind: Service metadata: name: headless-service namespace: default spec: type: ClusterIP # ClusterIP类型,也是默认类型 clusterIP: None # None就代表是无头service ports: # 指定service 端口及容器端口 - port: 80 # service ip中的端口 protocol: TCP targetPort: 80 # pod中的端口 selector: # 指定后端pod标签 app: nginx # 可通过kubectl get pod -l app=nginx查看哪些pod在使用此标签 ``` #### 4.1.2.3 应用资源清单文件创建headless Service ```powershell 命令 [root@master ~]# kubectl apply -f 04_headless_service.yml 输出 service/headless-service created ``` #### 4.1.2.4 查看已创建的headless Service ~~~powershell 命令 [root@master ~]# kubectl get svc 输出 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE headless-service ClusterIP None 80/TCP 2m18s kubernetes ClusterIP 10.96.0.1 443/TCP 5d9h 可以看到headless-service没有CLUSTER-IP,用None表示 ~~~ #### 4.1.2.5 DNS DNS服务监视Kubernetes API,为每一个Service创建DNS记录用于域名解析 headless service需要DNS来解决访问问题 DNS记录格式为: ..svc.cluster.local. ##### 4.1.2.5.1 查看kube-dns服务的IP ```powershell 命令 [root@master1 ~]# kubectl get svc -n kube-system 输出 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.2 53/UDP,53/TCP,9153/TCP 5d9h metrics-server ClusterIP 10.105.219.44 443/TCP 45h 查看到coreDNS的服务地址是10.96.0.2 ``` ##### 4.1.2.5.2 在集群主机通过DNS服务地址查找无头服务的dns解析 ```powershell 命令 [root@master01 ~]# dig -t A headless-service.default.svc.cluster.local. @10.96.0.2 输出 ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.2 <<>> -t A headless-service.default.svc.cluster.local. @10.96.0.2 ;; global options: +cmd ;; Got answer: ;; WARNING: .local is reserved for Multicast DNS ;; You are currently testing what happens when an mDNS query is leaked to DNS ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31371 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;headless-service.default.svc.cluster.local. IN A #被解析域名 ;; ANSWER SECTION: headless-service.default.svc.cluster.local. 30 IN A 10.224.235.147 #注意这里IP ;; Query time: 0 msec ;; SERVER: 10.96.0.10#53(10.96.0.2) ;; WHEN: Sun May 17 10:58:50 CST 2020 ;; MSG SIZE rcvd: 129 ``` ##### 4.1.2.5.3 验证pod的IP ```powershell 命令 [root@master ~]# kubectl get pod -o wide 输出 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-deployment-56bf6c9c8c-jmk7r 1/1 Running 0 35m 10.224.235.147 worker1 ``` ##### 4.1.2.5.4 在集群中创建一个pod验证 >创建一个镜像为busyboxplus:curl的pod，pod名称为bb2,用来解析域名 ```powershell 命令 [root@master01 ~]# kubectl run bbp --image=busyboxplus:curl -it 或 [root@master01 ~]# kubectl run bbp --image=1.28 -it 输出 If you don't see a command prompt, try pressing enter. 解析域名 nslookup headless-service.default.svc.cluster.local. 访问命令 [ root@bbp:/ ]$ curl http://headless-service.default.svc.cluster.local. 输出 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

[ root@bbp:/ ]$ exit Session ended, resume using 'kubectl attach bbp -c bbp -i -t' command when the pod is running ``` ## 4.2 NodePort类型 - 创建资源清单文件 ```powershell [root@master01 ~]# cat 05_create_nodeport_service_app.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-app labels: app: nginx-app spec: replicas: 2 selector: matchLabels: app: nginx-app template: metadata: labels: app: nginx-app spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-app spec: type: NodePort selector: app: nginx-app ports: - protocol: TCP nodePort: 30001 port: 8060 targetPort: 80 ``` - 应用资源清单文件 ```powershell [root@master01 ~]# kubectl apply -f 05_create_nodeport_service_app.yaml deployment.apps/nginx-app created service/nginx-app created ``` - 验证service创建 ```powershell [root@master01 ~]# kubectl get deployment.apps NAME READY UP-TO-DATE AVAILABLE AGE nginx-app 2/2 2 2 26s [root@master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 2d22h nginx-app NodePort 10.104.157.20 8060:30001/TCP 36s [root@master01 ~]# kubectl get endpoints NAME ENDPOINTS AGE kubernetes 192.168.122.10:6443 2d22h nginx-app 172.16.1.24:80,172.16.2.20:80 2m10s [root@master01 ~]# ss -anput | grep ":30001" tcp LISTEN 0 128 :::30001 :::* users:(("kube-proxy",pid=5826,fd=9)) [root@worker01 ~]# ss -anput | grep ":30001" tcp LISTEN 0 128 :::30001 :::* users:(("kube-proxy",pid=4937,fd=11)) [root@worker02 ~]# ss -anput | grep ":30001" tcp LISTEN 0 128 :::30001 :::* users:(("kube-proxy",pid=5253,fd=11)) ``` ```powershell [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-app-ffd5ccc78-cnwbx 1/1 Running 0 8m59s nginx-app-ffd5ccc78-mz77g 1/1 Running 0 8m59s [root@master01 ~]# kubectl exec -it nginx-app-ffd5ccc78-cnwbx -- bash root@nginx-app-ffd5ccc78-cnwbx:/# echo "nginx-app-1" > /usr/share/nginx/html/index.html root@nginx-app-ffd5ccc78-cnwbx:/# exit exit [root@master01 ~]# kubectl exec -it nginx-app-ffd5ccc78-mz77g -- bash root@nginx-app-ffd5ccc78-mz77g:/# echo "nginx-app-2" > /usr/share/nginx/html/index.html root@nginx-app-ffd5ccc78-mz77g:/# exit exit ``` - 在与kubernetes 节点同一网络主机中访问k8s集群内service ```powershell [root@bogon ~]# curl http://192.168.10.12:30001 nginx-app-2 [root@bogon ~]# curl http://192.168.10.13:30001 nginx-app-1 [root@bogon ~]# curl http://192.168.10.14:30001 nginx-app-1 [root@bogon ~]# curl http://192.168.10.15:30001 nginx-app-2 ``` ## 4.3 LoadBalancer ### 4.3.1 集群外访问过程 - #### 用户 - #### 域名 - #### 云服务提供商提供LB服务 - #### NodeIP:Port(service IP) - #### Pod IP：端口 ![image-20200531203510735](../../img/kubernetes/kubernetes_core/image-20200531203510735.png) ### 4.3.2 自建Kubernetes的LoadBalancer类型服务方案-MetalLB MetalLB可以为kubernetes集群中的Service提供网络负载均衡功能。 MetalLB两大功能为: - 地址分配，类似于DHCP - 外部通告，一旦MetalLB为服务分配了外部IP地址，它就需要使群集之外的网络意识到该IP在群集中“存在”。MetalLB使用标准路由协议来实现此目的：ARP，NDP或BGP。 #### 4.3.2.1 参考资料 参考网址： https://metallb.universe.tf/installation/ #### 4.3.2.2 应用资源清单文件 ~~~powershell 资源清单文件下载： # kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml # kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml ~~~ #### 4.3.2.3 准备metallb配置文件 ~~~powershell [root@nginx metallb]# cat metallb-conf.yaml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 192.168.10.90-192.168.10.100 192.168.10.90-192.168.10.100是集群节点服务器IP同一段。 ~~~ ~~~powershell 在master01节点应用资源清单文件 [root@master01 ~]# kubectl apply -f metallb-conf.yaml ~~~ ~~~powershell 验证配置 # kubectl describe configmap config -n metallb-system Name: config Namespace: metallb-system Labels: Annotations: Data ==== config: ---- address-pools: - name: default protocol: layer2 addresses: - 192.168.10.90-192.168.10.100 Events: ~~~ #### 4.3.2.4发布Service类型为LoadBalancer的Deployment控制器类型应用 ```powershell 创建Deployment控制器类型应用nginx-metallb及service，service类型为LoadBalancer [root@master01 ~]# vim 02_nginx-metabllb.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-metallb spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx-metallb1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-metallb spec: ports: - port: 8090 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer [root@master01 ~]# kubectl apply -f nginx.yaml ``` #### 4.3.2.5 验证 ```powershell [root@master01 ~]# kubectl get ns NAME STATUS AGE default Active 16d kube-node-lease Active 16d kube-public Active 16d kube-system Active 16d kubernetes-dashboard Active 13d metallb-system Active 130m test1 Active 12d [root@master01 ~]# kubectl get pods -n metallb-system NAME READY STATUS RESTARTS AGE controller-64f8f944d-qdf8m 1/1 Running 0 110m speaker-cwzq7 1/1 Running 0 110m speaker-qk5fb 1/1 Running 0 110m speaker-wsllb 1/1 Running 0 110m speaker-x4bwt 1/1 Running 0 110m [root@master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 16d nginx-metallb LoadBalancer 10.105.239.69 192.168.10.90 8090:31372/TCP 106m [root@master01 ~]# ping 192.168.10.90 PING 192.168.10.90 (192.168.10.90) 56(84) bytes of data. 64 bytes from 192.168.10.90: icmp_seq=1 ttl=64 time=3.45 ms 64 bytes from 192.168.10.90: icmp_seq=2 ttl=64 time=0.040 ms ``` #### 4.3.2.6 访问 ```powershell [root@master01 ~]# curl http://192.168.122.90:8090 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

``` ![image-20220402101421824](../../img/kubernetes/kubernetes_core/image-20220402101421824.png) **注意：使用kubeadm部署kubernetes集群修改方法** ~~~powershell 如果在IPVS模式下使用kube-proxy，从Kubernetes v1.14.2开始，必须启用ARP模式。 可以通过在当前集群中编辑kube-proxy配置来实现： # kubectl edit configmap -n kube-system kube-proxy 并设置： apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "ipvs" ipvs: strictARP: true ~~~ ## 4.4 ExternalName ### 4.4.1 ExternalName作用 - 把集群外部的服务引入到集群内部中来，实现了集群内部pod和集群外部的服务进行通信 - ExternalName 类型的服务适用于外部服务使用域名的方式，缺点是不能指定端口 - 还有一点要注意: 集群内的Pod会继承Node上的DNS解析规则。所以只要Node可以访问的服务，Pod中也可以访问到, 这就实现了集群内服务访问集群外服务 ### 4.4.2 将公网域名引入 1, 编写YAML文件 ```powershell [root@master01 ~]# vim externelname.yml apiVersion: v1 kind: Service metadata: name: my-externalname namespace: default spec: type: ExternalName externalName: www.baidu.com # 对应的外部域名为www.baidu.com ``` 2, 应用YAML文件 ```powershell [root@master01 ~]# kubectl apply -f externelname.yml service/my-externalname created ``` 3, 查看service ```powershell [root@master01 ~]# kubectl get svc |grep exter my-externalname ExternalName www.baidu.com 69s ``` 4, 查看my-service的dns解析 ```powershell [root@master01 ~]# dig -t A my-externalname.default.svc.cluster.local. @10.96.0.2 ; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> -t A my-externalname.default.svc.cluster.local. @10.2.0.2 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31378 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;my-externalname.default.svc.cluster.local. IN A ;; ANSWER SECTION: my-externalname.default.svc.cluster.local. 5 IN CNAME www.baidu.com. www.baidu.com. 5 IN CNAME www.a.shifen.com. www.a.shifen.com. 5 IN A 14.215.177.38 解析的是百度的IP www.a.shifen.com. 5 IN A 14.215.177.39 解析的是百度的IP ;; Query time: 32 msec ;; SERVER: 10.2.0.2#53(10.96.0.2) ;; WHEN: Thu Nov 05 11:23:41 CST 2020 ;; MSG SIZE rcvd: 245 ``` ```powershell [root@master01 ~]# kubectl exec -it deploy-nginx-6c9764bb69-86gwj -- /bin/sh / # nslookup www.baidu.com ...... Name: www.baidu.com Address 1: 14.215.177.39 Address 2: 14.215.177.38 / # nslookup my-externalname.default.svc.cluster.local ...... Name: my-externalname.default.svc.cluster.local Address 1: 14.215.177.38 Address 2: 14.215.177.39 ``` 解析此`my-externalname.default.svc.cluster.local`域名和解析`www.baidu.com`是一样的结果 ### 4.4.3 不同命名空间访问 1， 创建ns1命名空间和相关deploy, pod,service ```powershell [root@master01 ~]# vim ns1-nginx.yml apiVersion: v1 kind: Namespace metadata: name: ns1 # 创建ns1命名空间 --- apiVersion: apps/v1 kind: Deployment metadata: name: deploy-nginx namespace: ns1 # 属于ns1命名空间 spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: svc1 # 服务名 namespace: ns1 # 属于ns1命名空间 spec: selector: app: nginx clusterIP: None # 无头service ports: - port: 80 targetPort: 80 --- kind: Service apiVersion: v1 metadata: name: external-svc1 namespace: ns1 # 属于ns1命名空间 spec: type: ExternalName externalName: svc2.ns2.svc.cluster.local # 将ns2空间的svc2服务引入到ns1命名空间 [root@master1 ~]# kubectl apply -f ns1-nginx.yml namespace/ns1 created deployment.apps/deploy-nginx created service/svc1 created ``` 2， 创建ns2命名空间和相关deploy, pod,service ```powershell [root@master01 ~]# vim ns1-nginx.yml apiVersion: v1 kind: Namespace metadata: name: ns2 # 创建ns2命名空间 --- apiVersion: apps/v1 kind: Deployment metadata: name: deploy-nginx namespace: ns2 # 属于ns2命名空间 spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: svc2 # 服务名 namespace: ns2 # 属于ns2命名空间 spec: selector: app: nginx clusterIP: None # 无头service ports: - port: 80 targetPort: 80 --- kind: Service apiVersion: v1 metadata: name: external-svc1 namespace: ns2 # 属于ns2命名空间 spec: type: ExternalName externalName: svc1.ns1.svc.cluster.local # 将ns1空间的svc1服务引入到ns2命名空间 ``` ```powershell [root@master01 ~]# kubectl apply -f ns2-nginx.yml namespace/ns2 created deployment.apps/deploy-nginx created service/svc2 created service/external-svc2 created ``` 3, 在ns1命名空间的pod里验证 ```powershell [root@master01 ~]# kubectl get pods -n ns1 NAME READY STATUS RESTARTS AGE deploy-nginx-6c9764bb69-g5xl8 1/1 Running 0 8m10s ``` ```powershell [root@master01 ~]# kubectl exec -it -n ns1 deploy-nginx-6c9764bb69-g5xl8 -- /bin/sh / # nslookup svc1 ...... Name: svc1 Address 1: 10.3.166.140 deploy-nginx-6c9764bb69-g5xl8 IP与ns1里的podIP一致(见下面的查询结果) / # nslookup svc2.ns2.svc.cluster.local ..... Name: svc2.ns2.svc.cluster.local Address 1: 10.3.104.17 10-3-104-17.svc2.ns2.svc.cluster.local IP与ns2里的podIP一致(见下面的查询结果) / # exit ``` ```powershell [root@master01 ~]# kubectl get pods -o wide -n ns1 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deploy-nginx-6c9764bb69-g5xl8 1/1 Running 0 70m 10.3.166.140 192.168.122.13 [root@master01 ~]# kubectl get pods -o wide -n ns2 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READI NESS GATES deploy-nginx-6c9764bb69-8psxl 1/1 Running 0 68m 10.3.104.17 192.168.122.14 ``` 反之，在ns2命名空间的pod里访问`svc1.ns1.svc.cluster.local`，解析的IP是ns1命名空间里的pod的IP(请自行验证) 4， 验证ns2中的pod的IP变化, ns1中的pod仍然可以使用`svc2.ns2.svc.cluster.local`访问 ```powershell [root@master01 ~]# kubectl get pod -n ns2 NAME READY STATUS RESTARTS AGE deploy-nginx-6c9764bb69-8psxl 1/1 Running 0 81m [root@master01 ~]# kubectl delete pod deploy-nginx-6c9764bb69-8psxl -n ns2 pod "deploy-nginx-6c9764bb69-8psxl" deleted 因为有replicas控制器，所以删除pod会自动拉一个起来 [root@master01 ~]# kubectl get pod -o wide -n ns2 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES deploy-nginx-6c9764bb69-8qbz2 1/1 Running 0 5m36s 10.3.166.141 192.168.122.13 pod名称变了,IP也变成了10.3.166.141 ``` 回到ns1中的pod验证 ```powershell [root@master01 ~]# kubectl exec -it -n ns1 deploy-nginx-6c9764bb69-g5xl8 -- /bin/sh / # ping svc2.ns2.svc.cluster.local -c 2 PING svc2.ns2.svc.cluster.local (10.3.166.141): 56 data bytes 解析的IP就是ns2中pod的新IP 64 bytes from 10.3.166.141: seq=0 ttl=63 time=0.181 ms 64 bytes from 10.3.166.141: seq=1 ttl=63 time=0.186 ms --- svc2.ns2.svc.cluster.local ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.181/0.183/0.186 ms / # exit ``` # 五、sessionAffinity > 会话粘贴 设置sessionAffinity为Clientip (类似nginx的ip_hash算法,lvs的sh算法) ~~~powershell [root@nginx ~]# cat 02_create_deployment_app_nginx_with_service.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server1 spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-svc spec: type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 80 selector: app: nginx ~~~ ~~~powershell [root@master01 ~]# kubectl apply -f 02_create_deployment_app_nginx_with_service.yaml deployment.apps/nginx-server1 created service/nginx-svc created ~~~ ~~~powershell [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-server1-58845f75f4-9zlnw 1/1 Running 0 2m11s nginx-server1-58845f75f4-ffqdt 1/1 Running 0 2m11s [root@master01 ~]# kubectl exec -it nginx-server1-58845f75f4-9zlnw bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. root@nginx-server1-58845f75f4-9zlnw:/# echo web1 > /usr/share/nginx/html/index.html root@nginx-server1-58845f75f4-9zlnw:/# exit exit [root@master01 ~]# kubectl exec -it nginx-server1-58845f75f4-ffqdt bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. root@nginx-server1-58845f75f4-ffqdt:/# echo web2 > /usr/share/nginx/html/index.html root@nginx-server1-58845f75f4-ffqdt:/# exit exit ~~~ ~~~powershell [root@master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 16d nginx-svc ClusterIP 10.100.53.31 80/TCP 3m53s [root@master01 ~]# curl http://10.100.53.31 web1 [root@master01 ~]# curl http://10.100.53.31 web2 或 [root@master01 ~]# while true;do curl 10.100.53.31;sleep 1; done ~~~ ```powershell [root@master01 ~]# kubectl patch svc nginx-svc -p '{"spec":{"sessionAffinity":"ClientIP"}}' service/nginx-svc patched [root@master01 ~]# curl 10.100.53.31 web1 多次访问,会话粘贴 ``` ```powershell 设置回sessionAffinity为None [root@master01 ~]# kubectl patch svc nginx-svc -p '{"spec":{"sessionAffinity":"None"}}' service/my-service patched ``` ~~~powershell 测试 [root@master01 ~]# curl 10.100.53.31 web1 多次访问,回到负载均衡 或 [root@master01 ~]# while true;do curl 10.100.53.31;sleep 1; done web1 多次访问,会话粘贴 ~~~ # 六、修改为ipvs调度方式（拓展） > 部署方式不同，修改方法不一样。 > > 本次主要介绍使用kubeadm部署集群方式，二进制部署较为简单。 > > 二进制部署修改：/etc/kubernetes/kube-proxy.yaml文件即可。 从kubernetes1.8版本开始，新增了kube-proxy对ipvs的支持，在kubernetes1.11版本中被纳入了GA. ## 6.1 修改为IPVS调度方式前升级内核 > 现使用Centos7u6发布版本，默认内核版本为3.10.0，使用kubernetes为1.18.0时，可升级内核版本至4.18.0或5.6.0版本。 >在所有节点中安装,需要重启操作系统更换内核。以下升级方法供参考。 ~~~powershell [root@localhost ~]# yum -y install perl [root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org [root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm [root@localhost ~]# yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 此处升级为5.0以上版本。 [root@localhost ~]# grub2-set-default 0 [root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg [root@localhost ~]# reboot ~~~ ## 6.2 修改kube-proxy的配置文件 ```powershell [root@master01 ~]# kubectl edit configmap kube-proxy -n kube-system 26 iptables: 27 masqueradeAll: false 28 masqueradeBit: 14 29 minSyncPeriod: 0s 30 syncPeriod: 30s 31 ipvs: 32 excludeCIDRs: null 33 minSyncPeriod: 0s 34 scheduler: "" # 可以在这里修改ipvs的算法,默认为rr轮循算法 35 strictARP: false 36 syncPeriod: 30s 37 kind: KubeProxyConfiguration 38 metricsBindAddress: 127.0.0.1:10249 39 mode: "ipvs" # 默认""号里为空,加上ipvs ``` ## 6.3 查看kube-system的namespace中kube-proxy有关的pod ```powershell [root@master01 ~]# kubectl get pods -n kube-system |grep kube-proxy kube-proxy-69mv6 1/1 Running 6 2d18h kube-proxy-jpc6c 1/1 Running 4 4d16h kube-proxy-kq65l 1/1 Running 4 4d16h kube-proxy-lmphf 1/1 Running 5 4d16h ``` ## 6.4 验证kube-proxy-xxx的pod中的信息 ```powershell [root@master01 ~]# kubectl logs kube-proxy-jpc6c -n kube-system W0517 00:55:10.914754 1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy I0517 00:55:10.923228 1 node.go:136] Successfully retrieved node IP: 192.168.122.32 I0517 00:55:10.923264 1 server_others.go:186] Using iptables Proxier. I0517 00:55:10.923567 1 server.go:583] Version: v1.18.2 I0517 00:55:10.923965 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072 I0517 00:55:10.924001 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0517 00:55:10.924258 1 conntrack.go:83] Setting conntrack hashsize to 32768 I0517 00:55:10.927041 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 I0517 00:55:10.927086 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 I0517 00:55:10.927540 1 config.go:315] Starting service config controller I0517 00:55:10.927556 1 shared_informer.go:223] Waiting for caches to sync for service config I0517 00:55:10.927576 1 config.go:133] Starting endpoints config controller I0517 00:55:10.927594 1 shared_informer.go:223] Waiting for caches to sync for endpoints config I0517 00:55:11.027749 1 shared_informer.go:230] Caches are synced for service config I0517 00:55:11.027858 1 shared_informer.go:230] Caches are synced for endpoints config ``` ## 6.5 > 删除kube-proxy-xxx的所有pod，让它重新拉取新的kube-proxy-xxx的pod ~~~powershell [root@master01 ~]# kubectl delete pod kube-proxy-69mv6 -n kube-system pod "kube-proxy-69mv6" deleted [root@master01 ~]# kubectl delete pod kube-proxy-jpc6c -n kube-system pod "kube-proxy-jpc6c" deleted [root@master01 ~]# kubectl delete pod kube-proxy-kq65l -n kube-system pod "kube-proxy-kq65l" deleted [root@master01 ~]# kubectl delete pod kube-proxy-lmphf -n kube-system pod "kube-proxy-lmphf" deleted ~~~ ~~~powershell [root@master01 ~]# kubectl get pods -n kube-system |grep kube-proxy kube-proxy-2mk2b 1/1 Running 0 2m23s kube-proxy-5bj87 1/1 Running 0 30s kube-proxy-7qq9l 1/1 Running 0 52s kube-proxy-tjtqf 1/1 Running 0 80s ~~~ ~~~powershell 随意查看其中1个或3个kube-proxy-xxx的pod,验证是否为IPVS方式了 [root@master1 ~]# kubectl logs kube-proxy-tjtqf -n kube-system I0517 02:32:26.557696 1 node.go:136] Successfully retrieved node IP: 192.168.122.32 I0517 02:32:26.557745 1 server_others.go:259] Using ipvs Proxier. W0517 02:32:26.557912 1 proxier.go:429] IPVS scheduler not specified, use rr by default I0517 02:32:26.560008 1 server.go:583] Version: v1.18.2 I0517 02:32:26.560428 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0517 02:32:26.561094 1 config.go:315] Starting service config controller I0517 02:32:26.562251 1 shared_informer.go:223] Waiting for caches to sync for service config I0517 02:32:26.561579 1 config.go:133] Starting endpoints config controller I0517 02:32:26.562271 1 shared_informer.go:223] Waiting for caches to sync for endpoints config I0517 02:32:26.662541 1 shared_informer.go:230] Caches are synced for service config I0517 02:32:26.662566 1 shared_informer.go:230] Caches are synced for endpoints config ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_deploy_golang.md ================================================ # Kubernetes集群golang项目上云部署 # 一、项目情况 本次上线部署的是一个基于Golang开发的百万并发的IM系统，提供平台用户基本聊天及群聊功能等。 # 二、项目源码 ~~~powershell [root@harbor ginchat-v1.0]# ls asset config docs go.mod go.sum index.html main.go models router service test utils views ~~~ ~~~powershell 前端： index.html views asset go源码： go.mod go.sum main.go等 配置目录： config/app.yaml 注意修改：mysql数据库地址及redis连接地址 ~~~ ~~~powershell 项目数据库文件 ]# ls sql init_ginchat.sql ~~~ # 三、项目部署第三方服务 ## 3.1 数据库 MySQL ~~~powershell 创建命名空间资源清单文件 [root@nginx 05_go_project]# cat 01_namespace.yaml apiVersion: v1 kind: Namespace metadata: name: ginchat ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f http://yaml.kubemsb.com/04_python_project/01_namespace.yaml ~~~ ~~~powershell 创建部署MySQL数据库管理系统资源清单文件 [root@nginx 05_go_project]# cat 02_mysql.yaml apiVersion: v1 kind: Service metadata: name: ginchatmysql namespace: ginchat spec: ports: - port: 3306 name: mysql clusterIP: None selector: app: mysqlginchat --- apiVersion: apps/v1 kind: StatefulSet metadata: name: ginchatdb namespace: ginchat spec: selector: matchLabels: app: mysqlginchat serviceName: "ginchatmysql" template: metadata: labels: app: mysqlginchat spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "123456" - name: MYSQL_DATABASE value: ginchat ports: - containerPort: 3306 volumeMounts: - mountPath: "/var/lib/mysql" name: mysql-ginchat volumeClaimTemplates: - metadata: name: mysql-ginchat spec: accessModes: ["ReadWriteMany"] storageClassName: "nfs-client" resources: requests: storage: 1Gi ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f http://yaml.kubemsb.com/04_python_project/02_mysql.yaml ~~~ ~~~powershell 查看数据库对应的pod # kubectl get pods -n ginchat NAME READY STATUS RESTARTS AGE ginchatdb-0 1/1 Running 0 9m ~~~ ~~~powershell 导入项目数据 # kubectl exec -it ginchatdb-0 -n ginchat -- mysql -uroot -p123456 ginchat < init_ginchat.sql ~~~ ~~~powershell 检查持久存储 # ls ginchat-mysql-ginchat-ginchatdb-0-pvc-6a01f107-e29a-462a-a24a-64aeb59f6432/ auto.cnf ca.pem client-key.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem ca-key.pem client-cert.pem ginchat ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys ~~~ ## 3.2 Redis ~~~powershell # mkdir -p /opt/redis/conf ~~~ ~~~powershell # touch /opt/redis/conf/redis.conf ~~~ ~~~powershell # docker run -p 6379:6379 --name ginchatredis -v /opt/redis/data:/data -v /opt/redis/conf:/etc/redis -d redis redis-server /etc/redis/redis.conf ~~~ # 四、源码编译 ## 4.1 项目源码 ~~~powershell # ls asset config docs go.mod go.sum index.html main.go models router service test utils views ~~~ ## 4.2 源码编译环境准备 ~~~powershell # wget https://storage.googleapis.com/golang/getgo/installer_linux # chmod +x installer_linux # ./installer_linux # source /root/.bash_profile ~~~ ~~~powershell # go version go version go1.18.3 linux/amd64 ~~~ ## 4.3 源码编译 ~~~powershell [root@harbor ginchat-v1.0]# mkdir bin ~~~ ~~~powershell [root@harbor ginchat-v1.0]# ls asset bin config docs go.mod go.sum index.html main.go models router service test utils views ~~~ ~~~powershell [root@harbor ginchat-v1.0]# go get && go build -o bin/ginchatd ~~~ ~~~powershell [root@harbor ginchat-v1.0]# ls bin ginchatd ~~~ # 五、项目容器镜像制作 ## 5.1 查看制品文件 > 把源码生成的制品文件及前端文件复制到一个新的目录中，内容如下： ~~~powershell [root@harbor ~]## ls ginchat-v1.0/ asset config ginchatd index.html views ~~~ ## 5.2 修改配置文件 ~~~powershell [root@harbor ~]# vim ginchat-v1.0/config/app.yml mysql: dns: root:123456@tcp(ginchatdb-0.ginchatmysql.ginchat.svc.cluster.local:3306)/ginchat?charset=utf8mb4&parseTime=True&loc=Local redis: addr: "192.168.10.213:6379" password: "" DB: 0 poolSize: 30 minIdleConn: 30 oos: Endpoint: "oss-cn-hangzhou.aliyuncs.com" AccessKeyId: "LTAI5tNCXPJwS3MstKoKgixh" AccessKeySecret: "YhHE8OyCMsqfjwOnxQ1oO7paYlDjVH" Bucket : "ginchat" timeout: DelayHeartbeat: 3 #延迟心跳时间 单位秒 HeartbeatHz: 30 #每隔多少秒心跳时间 HeartbeatMaxTime: 30000 #最大心跳时间 ，超过此就下线 RedisOnlineTime: 4 #缓存的在线用户时长 单位H ~~~ ## 5.3 把制品文件打包 ~~~powershell [root@harbor ~]# tar cvzf ginchat.tgz ginchat-v1.0 ~~~ ## 5.4 制作容器镜像 ~~~powershell [root@harbor ~]# mkdir goginchatproject ~~~ ~~~powershell [root@harbor goginchatproject]# vim Dockerfile FROM centos:centos7 ADD ./ginchat.tgz / WORKDIR /ginchat-v1.0 RUN chmod +x /ginchat-v1.0/ginchatd EXPOSE 8081 CMD /ginchat-v1.0/ginchatd ~~~ ~~~powershell [root@harbor ~]# ls goginchatproject/ Dockerfile ginchat.tgz ~~~ ~~~powershell [root@harbor goginchatproject]# docker build -t www.kubemsb.com/library/ginchat:v1 . ~~~ ~~~powershell [root@harbor goginchatproject]# docker push www.kubemsb.com/library/ginchat:v1 ~~~ # 六、项目部署资源清单文件编写 ## 6.1 deployment资源清单文件 ~~~powershell [root@nginx 05_go_project]# cat 04_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: ginchat namespace: ginchat spec: replicas: 1 selector: matchLabels: project: ginchat app: ginchat-demo template: metadata: labels: project: ginchat app: ginchat-demo spec: imagePullSecrets: - name: harborreg #认证信息 containers: - name: ginchat image: www.kubemsb.com/library/ginchat:v2 #镜像 imagePullPolicy: Always ports: - containerPort: 8081 name: web protocol: TCP resources: requests: cpu: 0.5 memory: 1Gi limits: cpu: 1 memory: 2Gi livenessProbe: httpGet: path: / port: 8081 initialDelaySeconds: 30 timeoutSeconds: 20 readinessProbe: httpGet: path: / port: 8081 initialDelaySeconds: 30 timeoutSeconds: 20 ~~~ ## 6.2 service资源清单文件 ~~~powershell [root@nginx 05_go_project]# cat 05_service.yaml apiVersion: v1 kind: Service metadata: name: ginchatsvc namespace: ginchat spec: selector: project: ginchat app: ginchat-demo ports: - name: web port: 80 targetPort: 8081 ~~~ ## 6.3 ingress对象资源清单文件 ~~~powershell [root@nginx 05_go_project]# cat 06_ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ginchatingress namespace: ginchat annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: ginchat.kubemsb.com http: paths: - pathType: Prefix path: / backend: service: name: ginchatsvc port: number: 80 ~~~ # 七、项目部署 ![image-20220708172656543](../../img/kubernetes/kubernetes_deploy_golang/image-20220708172656543.png) ![image-20220708172609736](../../img/kubernetes/kubernetes_deploy_golang/image-20220708172609736.png) ~~~powershell [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/05_go_project/04_deployment.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/05_go_project/05_service.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/05_go_project/06_ingress.yaml ~~~ # 八、项目访问 ~~~powershell [root@master01 ~]# kubectl get pods -n ginchat NAME READY STATUS RESTARTS AGE ginchat-5558d849c5-cq9qq 1/1 Running 0 15m ginchatdb-0 1/1 Running 0 10m ~~~ ![image-20220708154640365](../../img/kubernetes/kubernetes_deploy_golang/image-20220708154640365.png) ![image-20220708154208597](../../img/kubernetes/kubernetes_deploy_golang/image-20220708154208597.png) ![image-20220708154125179](../../img/kubernetes/kubernetes_deploy_golang/image-20220708154125179.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_deploy_java.md ================================================ # kubernetes集群java项目上云部署 # 一、部署前准备工作 ## 1.1 部署项目情况 ### 1.1.1 业务部署架构 - 单体服务架构 - 分布式服务架构 - 微服务架构 - 超微服务架构 ### 1.1.2 项目涉及第三方服务 - 关系型数据库系统 MySQL - 缓存服务 Redis memcache - 协调服务 zookeeper - 消息中间件服务 kafka rabbitmq - 服务注册 服务发现 nacos ### 1.1.3 项目所需要的资源 - 计算资源 - cpu - 内存 - 存储资源 - 本地存储 - 网络存储 ## 1.2 部署项目所需要的k8s资源 ### 1.2.1 多k8s集群资源 - 测试环境 - 开发环境 - 预发布环境 - 生产环境 ### 1.2.2 使用namespace隔离项目或环境 - 每项目独立使用namespace - 每环境独立使用namespace ### 1.2.3 有状态应用部署 > 部署时考虑哪些系统使用有状态应用部署方式 - statefulset - 数据持久化存储动态供给（pv及pvc） - 例如：mysql ### 1.2.4 无状态应用部署 - deployment - 数据持久化存储动态供给 - 例如：tomcat ### 1.2.5 暴露外部访问 - service类型 - 有状态应用部署使用headless service - 无状态应用部署使用ClusterIP - 暴露服务方式 - ingress - api gateway ### 1.2.6 密钥及配置管理 - configmap - mysql配置文件 - secret - k8s集群使用harbor私有项目仓库镜像 ## 1.3 项目基础镜像准备 > 本次发布一个java项目，以war包方式发布，需要使用tomcat做为项目基础镜像。可直接下载使用，也可选择定制。 ### 1.3.1 定制tomcat镜像 #### 1.3.1.1 直接下载 ~~~powershell 直接下载 [root@harborserver ~]# docker pull tomcat ~~~ #### 1.3.1.2 通过Dockerfile文件定制 ~~~powershell [root@harborserver ~]# mkdir tomcatdockerfile [root@harborserver ~]# cd tomcatdockerfile/ [root@harborserver tomcatdockerfile]# cat Dockerfile FROM centos:centos7 MAINTAINER "admin" ENV VERSION=8.5.81 ENV JAVA_HOME=/usr/local/jdk RUN yum -y install wget RUN wget https://dlcdn.apache.org/tomcat/tomcat-8/v${VERSION}/bin/apache-tomcat-${VERSION}.tar.gz --no-check-certificate RUN tar xf apache-tomcat-${VERSION}.tar.gz RUN mv apache-tomcat-${VERSION} /usr/local/tomcat RUN rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/* RUN mkdir /usr/local/tomcat/webapps/ROOT ADD ./jdk /usr/local/jdk RUN echo "export TOMCAT_HOME=/usr/local/tomcat" >> /etc/profile RUN echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile RUN echo "export PATH=$TOMCAT_HOME/bin:$JAVA_HOME/bin:$PATH" >> /etc/profile RUN echo "export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar" >> /etc/profile RUN source /etc/profile EXPOSE 8080 CMD ["/usr/local/tomcat/bin/catalina.sh","run"] ~~~ ~~~powershell [root@harborserver tomcatdockerfile]# ll 总用量 4 -rw-r--r-- 1 root root 865 1月 15 16:44 Dockerfile drwxr-xr-x 7 10 143 245 10月 6 2018 jdk ~~~ ~~~powershell [root@harborserver tomcatdockerfile]# docker build -t www.kubemsb.com/java-project/tomcat:8581 . ...... Successfully built db4db20a6c85 Successfully tagged www.kubemsb.com/java-project/tomcat:8581 ~~~ ~~~powershell [root@harborserver ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE www.kubemsb.com/javap/tomcat 8581 db4db20a6c85 11 minutes ago 817MB ~~~ ~~~powershell [root@harborserver ~]# docker login www.kubemsb.com Username: admin Password: 12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [root@harborserver ~]# docker push www.kubemsb.com/java-project/tomcat:8581 ~~~ ### 1.3.2 在harbor验证tomcat镜像是否上传 ![image-20220705102245766](../../img/kubernetes/kubernetes_deploy_java/image-20220705102245766.png) ## 1.4 项目编排部署 ### 1.4.1 项目资源 - JAVA项目源码 - JAVA项目容器镜像 - JAVA项目资源清单文件 - 数据库系统 MySQL ### 1.4.2 项目镜像构建方法 - 手动构建容器镜像（Dockerfile） - 自动构建容器镜像（jenkins ci/cd） ## 1.5 项目部署工作流程 - 基于项目源码制作容器镜像 - 编写项目部署资源清单文件 - 使用资源清单文件创建项目 Service - 使用ingress暴露服务实现项目对外可访问 ## 1.6 项目部署基础环境架构 ![image-20220705104124475](../../img/kubernetes/kubernetes_deploy_java/image-20220705104124475.png) # 二、存储准备 > 本次使用NFS服务做为K8S集群后端存储，实现kubernetes集群持久存储动态供给，详细可见kubernetes集群公共服务章节。 ~~~powershell [root@master01 ~]# kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 10s ~~~ # 三、项目容器镜像仓库及项目源码准备 ## 3.1 项目容器容器仓库准备 ### 3.1.1 Harbor部署 > 参考kubernetes集群公共服务中容器镜像托管仓库章节 ### 3.1.2 Harbor添加项目容器镜像仓库 ![image-20220705105448685](../../img/kubernetes/kubernetes_deploy_java/image-20220705105448685.png) ## 3.2 项目源码及编译环境准备 ### 3.2.1 项目源码准备 ~~~powershell [root@harbor ~]# ls anaconda-ks.cfg initial-setup-ks.cfg javaproject 公共 模板 视频 图片 文档 下载 音乐 桌面 [root@harbor ~]# cd javaproject/ [root@harbor javaproject]# ls project-source tomcatdockerfile [root@harbor javaproject]# ls project-source/ db pro-source [root@harbor javaproject]# ls project-source/db test.sql [root@harbor javaproject]# ls project-source/pro-source/ java-project [root@harbor javaproject]# ls project-source/pro-source/java-project/ Dockerfile LICENSE pom.xml README.md src [root@harbor javaproject]# ls project-source/pro-source/java-project/src main 源码目录 ~~~ ~~~powershell [root@harbor java-project]# cat Dockerfile FROM www.kubemsb.com/java-project/tomcat:8581 LABEL maintainer "admin " RUN rm -rf /usr/local/tomcat/webapps/* ADD target/*.war /usr/local/tomcat/webapps/ROOT.war ~~~ ~~~powershell [root@harbor java-project]# cat README.md > ### SQL文件: db/test.sql > ### 数据库配置：src/main/resources/application.yml ~~~ ~~~powershell [root@harbor java-project]# cat src/main/resources/application.yml server: port: 8080 spring: datasource: url: jdbc:mysql://db-0.mysql.javaproject:3306/test?characterEncoding=utf-8 username: root password: 123456 driver-class-name: com.mysql.jdbc.Driver freemarker: allow-request-override: false cache: true check-template-location: true charset: UTF-8 content-type: text/html; charset=utf-8 expose-request-attributes: false expose-session-attributes: false expose-spring-macro-helpers: false suffix: .ftl template-loader-path: - classpath:/templates/ ~~~ ### 3.2.2 项目编译环境准备 jdk & maven > 用于对java项目代码进行编译打包代码 #### 3.2.2.1 下载maven ![image-20200115172244243](../../img/kubernetes/kubernetes_deploy_java/image-20200115172244243.png) ![image-20200115172439160](../../img/kubernetes/kubernetes_deploy_java/image-20200115172439160.png) ![image-20200115172545004](../../img/kubernetes/kubernetes_deploy_java/image-20200115172545004.png) ~~~powershell [root@harborserver ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz ~~~ #### 3.2.2.2 安装maven ~~~powershell 必须要有官方的jdk，不然maven会报错 [root@harborserver ~]# ls jdk-8u191-linux-x64.tar.gz [root@harborserver ~]# tar xf jdk-8u191-linux-x64.tar.gz [root@harborserver ~]# ls jdk1.8.0_191 [root@harborserver ~]# mv jdk1.8.0_191 /usr/local/jdk [root@harborserver ~]# ls /usr/local/jdk bin lib src.zip COPYRIGHT LICENSE THIRDPARTYLICENSEREADME-JAVAFX.txt include man THIRDPARTYLICENSEREADME.txt javafx-src.zip README.html jre release [root@harborserver ~]# ls apache-maven-3.6.3-bin.tar.gz [root@harborserver ~]# tar xf apache-maven-3.6.3-bin.tar.gz [root@harborserver ~]# ls apache-maven-3.6.3 [root@harborserver ~]# mv apache-maven-3.6.3 /usr/local/maven [root@harborserver ~]# ls /usr/local/maven/ bin boot conf lib LICENSE NOTICE README.txt [root@harborserver ~]# cat /etc/profile.d/maven.sh export JAVA_HOME=/usr/local/jdk export MAVEN_HOME=/usr/local/maven export PATH=${MAVEN_HOME}/bin:${JAVA_HOME}/bin:$PATH [root@harborserver ~]# source /etc/profile [root@harborserver ~]# mvn -v Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /usr/local/maven Java version: 1.8.0_191, vendor: Oracle Corporation, runtime: /usr/local/jdk/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1062.9.1.el7.x86_64", arch: "amd64", family: "unix" ~~~ # 四、项目部署 ## 4.1 本项目部署思路 ### 4.1.1 本次部署的应用 - 数据库 MySQL - Web应用 Tomcat ### 4.1.2 有状态应用与无状态应用部署规划 - mysql - statefulset控制器 - headless service - pv,pvc 存储资源动态供给 - tomcat（java应用） - deployment控制器 - ClusterIP Service - Ingress 服务暴露实现集群外访问 ## 4.2 项目资源清单文件准备 ### 4.2.1 项目文件总览 ~~~powershell [root@harbor java-project]# pwd /root/javaproject/project-source/pro-source/java-project [root@harbor java-project]# ls Dockerfile LICENSE pom.xml README.md src ~~~ ~~~powershell [root@nginx 03_java_project]# ls 01_ns.yaml 02_deployment.yaml 03_service.yaml 04_ingress.yaml 05_mysql.yaml 06_create_pod_busybox.yaml ~~~ ### 4.2.2 资源清单文件准备 #### 4.2.2.1 namespace资源清单文件 > 用于实现项目隔离 ~~~powershell # cat 01_ns.yaml apiVersion: v1 kind: Namespace metadata: name: javaproject ~~~ #### 4.2.2.2 部署java项目资源清单文件 > 用于部署java项目 ~~~powershell # cat 02_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: java-project namespace: javaproject spec: replicas: 2 selector: matchLabels: project: www app: java-demo template: metadata: labels: project: www app: java-demo spec: imagePullSecrets: - name: harborreg #认证信息 containers: - name: tomcat image: www.kubemsb.com/java-project/java-project:v1 #镜像 imagePullPolicy: Always ports: - containerPort: 8080 name: web protocol: TCP resources: requests: cpu: 0.5 memory: 1Gi limits: cpu: 1 memory: 2Gi livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 60 timeoutSeconds: 20 readinessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 60 timeoutSeconds: 20 ~~~ #### 4.2.2.3 项目服务（service）资源清单文件 > 用于创建service ~~~powershell # cat 03_service.yaml apiVersion: v1 kind: Service metadata: name: java-project namespace: javaproject spec: selector: project: www app: java-demo ports: - name: web port: 80 targetPort: 8080 ~~~ #### 4.2.2.4 Ingress对象资源清单文件 > 需要提前部署 ingress控制器 ~~~powershell # cat 04_ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: java-project namespace: javaproject annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: javaweb.kubemsb.com http: paths: - pathType: Prefix path: / backend: service: name: java-project port: number: 80 ~~~ #### 4.2.2.5 mysql部署资源清单文件 ~~~powershell # cat 05_mysql.yaml apiVersion: v1 kind: Service metadata: name: mysql namespace: javaproject spec: ports: - port: 3306 name: mysql clusterIP: None selector: app: mysql-public --- apiVersion: apps/v1 kind: StatefulSet metadata: name: db namespace: javaproject spec: selector: matchLabels: app: mysql-public serviceName: "mysql" template: metadata: labels: app: mysql-public spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "123456" - name: MYSQL_DATABASE value: test ports: - containerPort: 3306 volumeMounts: - mountPath: "/var/lib/mysql" name: mysql-data volumeClaimTemplates: - metadata: name: mysql-data spec: accessModes: ["ReadWriteMany"] storageClassName: "nfs-client" resources: requests: storage: 5Gi ~~~ #### 4.2.2.6 mysql访问测试pod ~~~powershell # cat 06_create_pod_busybox.yaml apiVersion: v1 kind: Pod metadata: name: busybox-pod spec: containers: - name: busybox-container image: busybox:1.28.4 #此镜像nslookup及ping命令都没有问题，不要下载最新版本和1.31。 imagePullPolicy: IfNotPresent command: - sleep - "3600" restartPolicy: Always ~~~ ## 4.3 项目数据库 Mysql部署 ### 4.3.1 创建命名空间 ~~~powershell # kubectl apply -f http://yaml.kubemsb.com/03_java_project/01_ns.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl get ns NAME STATUS AGE calico-apiserver Active 7d15h calico-system Active 7d16h default Active 12d ingress-nginx Active 2d22h javaproject Active 6s kube-node-lease Active 12d kube-public Active 12d kube-system Active 12d metallb-system Active 2d22h tigera-operator Active 7d16h ~~~ ### 4.3.2 部署数据库 ~~~powershell # kubectl apply -f http://yaml.kubemsb.com/03_java_project/05_mysql.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl get all -n javaproject NAME READY STATUS RESTARTS AGE pod/db-0 1/1 Running 0 19s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP None 3306/TCP 19s NAME READY AGE statefulset.apps/db 1/1 19s ~~~ ~~~powershell [root@nfs ~]# ls /sdb/ javaproject-mysql-data-db-0-pvc-9f616f77-9995-4f11-ad57-833038186777 [root@nfs ~]# ls /sdb/javaproject-mysql-data-db-0-pvc-9f616f77-9995-4f11-ad57-833038186777/ auto.cnf ca.pem client-key.pem ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys ca-key.pem client-cert.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem test ~~~ ### 4.3.3 导入数据库 ~~~powershell [root@harbor project-source]# scp db/test.sql 192.168.10.11:/root ~~~ ~~~powershell [root@master01 ~]# kubectl get pods -n javaproject NAME READY STATUS RESTARTS AGE db-0 1/1 Running 0 15m [root@master01 ~]# kubectl cp test.sql db-0:/ -n javaproject [root@master01 ~]# kubectl exec -it db-0 -n javaproject -- bash root@db-0:/# ls bin dev entrypoint.sh home lib64 mnt proc run srv test.sql usr boot docker-entrypoint-initdb.d etc lib media opt root sbin sys tmp var root@db-0:/# mysql -uroot -p123456 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.38 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> use test; Database changed mysql> source /test.sql; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected, 1 warning (0.01 sec) Query OK, 0 rows affected (0.02 sec) mysql> show tables; +----------------+ | Tables_in_test | +----------------+ | user | +----------------+ 1 row in set (0.00 sec) mysql> desc user; +-------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------+-------------+------+-----+---------+----------------+ | id | bigint(20) | NO | PRI | NULL | auto_increment | | name | varchar(30) | NO | | NULL | | | age | int(11) | YES | | 0 | | | sex | char(1) | YES | | 0 | | +-------+-------------+------+-----+---------+----------------+ 4 rows in set (0.02 sec) ~~~ ### 4.3.4 验证数据库可用性 ~~~powershell # kubectl apply -f http://yaml.kubemsb.com/03_java_project/06_create_pod_busybox.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE busybox-pod 1/1 Running 0 4s nfs-client-provisioner-6446865dff-pqtfj 1/1 Running 1 5d22h [root@master01 ~]# kubectl exec -it busybox-pod -- sh / # nslookup db-0.mysql.javaproject Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: db-0.mysql.javaproject Address 1: 10.224.30.77 db-0.mysql.javaproject.svc.cluster.local / # nslookup db-0.mysql.javaproject.svc.cluster.local. Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: db-0.mysql.javaproject.svc.cluster.local. Address 1: 10.224.30.77 db-0.mysql.javaproject.svc.cluster.local ~~~ ## 4.4 项目源码编译打包 ### 4.4.1 修改项目连接数据库地址 ~~~powershell [root@harbor java-project]# pwd /root/javaproject/project-source/pro-source/java-project [root@harbor java-project]# ls Dockerfile LICENSE pom.xml README.md src [root@harbor java-project]# cd src/ [root@harbor src]# ls main [root@harbor src]# cd main/ [root@harbor main]# ls java resources [root@harbor main]# cd resources/ [root@harbor resources]# ls application.yml log4j.properties static templates [root@harbor resources]# vim application.yml [root@harbor resources]# cat application.yml server: port: 8080 spring: datasource: url: jdbc:mysql://db-0.mysql.javaproject:3306/test?characterEncoding=utf-8 username: root password: 123456 driver-class-name: com.mysql.jdbc.Driver freemarker: allow-request-override: false cache: true check-template-location: true charset: UTF-8 content-type: text/html; charset=utf-8 expose-request-attributes: false expose-session-attributes: false expose-spring-macro-helpers: false suffix: .ftl template-loader-path: - classpath:/templates/ ~~~ ### 4.4.2 使用maven编译项目源码 ~~~powershell [root@harbor java-project]# pwd /root/javaproject/project-source/pro-source/java-project [root@harbor java-project]# mvn clean package [INFO] Scanning for projects... [INFO] [INFO] ---------------------< com.kubemsb:kubemsb-tomcat >--------------------- [INFO] Building kubemsb-tomcat 0.0.1-Test [INFO] --------------------------------[ war ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ kubemsb-tomcat --- [INFO] [INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ kubemsb-tomcat --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 136 resources [INFO] [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ kubemsb-tomcat --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 7 source files to /root/javaproject/project-source/pro-source/java-project/target/classes [INFO] [INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ kubemsb-tomcat --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /root/javaproject/project-source/pro-source/java-project/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ kubemsb-tomcat --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ kubemsb-tomcat --- [INFO] No tests to run. [INFO] [INFO] --- maven-war-plugin:3.1.0:war (default-war) @ kubemsb-tomcat --- [INFO] Packaging webapp [INFO] Assembling webapp [kubemsb-tomcat] in [/root/javaproject/project-source/pro-source/java-project/target/kubemsb-tomcat-0.0.1-Test] [INFO] Processing war project [INFO] Webapp assembled in [166 msecs] [INFO] Building war: /root/javaproject/project-source/pro-source/java-project/target/kubemsb-tomcat-0.0.1-Test.war [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.594 s [INFO] Finished at: 2022-07-05T12:02:11+08:00 [INFO] ------------------------------------------------------------------------ ~~~ ~~~powershell [root@harbor java-project]# ls Dockerfile LICENSE pom.xml README.md src target [root@harbor java-project]# ls target/ classes generated-sources kubemsb-tomcat-0.0.1-Test kubemsb-tomcat-0.0.1-Test.war maven-archiver maven-status ~~~ ## 4.5 生成项目容器镜像制品 ~~~powershell [root@harbor java-project]# pwd /root/javaproject/project-source/pro-source/java-project [root@harbor java-project]# ls Dockerfile LICENSE pom.xml README.md src target [root@harbor java-project]# docker build -t www.kubemsb.com/java-project/java-project:v1 . Sending build context to Docker daemon 42.44MB Step 1/4 : FROM www.kubemsb.com/java-project/tomcat:8581 ---> 7522e6998fbf Step 2/4 : LABEL maintainer "admin " ---> Using cache ---> 4e61597e580c Step 3/4 : RUN rm -rf /usr/local/tomcat/webapps/* ---> Using cache ---> fda908c158cd Step 4/4 : ADD target/*.war /usr/local/tomcat/webapps/ROOT.war ---> c74ffbdeae67 Successfully built c74ffbdeae67 Successfully tagged www.kubemsb.com/java-project/java-project:v1 ~~~ ~~~powershell [root@harbor java-project]# docker login www.kubemsb.com Authenticating with existing credentials... WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ~~~powershell [root@harbor java-project]# docker push www.kubemsb.com/java-project/java-project:v1 ~~~ ![image-20220705121254871](../../img/kubernetes/kubernetes_deploy_java/image-20220705121254871.png) ## 4.6 项目部署 ~~~powershell # kubectl apply -f http://yaml.kubemsb.com/03_java_project/02_deployment.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl get pods -n javaproject NAME READY STATUS RESTARTS AGE db-0 1/1 Running 0 38m java-project-6f74d5b85c-ckq42 0/1 Running 0 43s java-project-6f74d5b85c-dvvcl 0/1 Running 0 43s ~~~ ~~~powershell # kubectl apply -f http://yaml.kubemsb.com/03_java_project/03_service.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl get svc -n javaproject NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE java-project ClusterIP 10.106.90.232 80/TCP 57s mysql ClusterIP None 3306/TCP 39m ~~~ ~~~powershell # kubectl apply -f http://yaml.kubemsb.com/03_java_project/04_ingress.yaml ~~~ ~~~powershell [root@master01 ~]# kubectl get ingress -n javaproject NAME CLASS HOSTS ADDRESS PORTS AGE java-project javaweb.kubemsb.com 192.168.10.13 80 73s ~~~ # 五、访问验证 ~~~powershell [root@master01 ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.97.68.236 192.168.10.90 80:32567/TCP,443:31146/TCP 2d22h ingress-nginx-controller-admission ClusterIP 10.106.171.219 443/TCP 2d22h ~~~ ![image-20220705121941975](../../img/kubernetes/kubernetes_deploy_java/image-20220705121941975.png) ![image-20220705122100186](../../img/kubernetes/kubernetes_deploy_java/image-20220705122100186.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_deploy_python.md ================================================ # kubernetes集群Python项目上云部署 # 一、项目资源及项目代码 ~~~powershell [root@localhost cmdb]# pwd /root/cmdb [root@localhost cmdb]# ls db pipsource requirement syscmdb db用于存储项目数据库 pipsource用于存储pip源 requirement用于存储python项目依赖资源 syscmdb用于存储项目源代码 ~~~ ~~~powershell [root@localhost cmdb]# ls db cmdbdb.sql ~~~ ~~~powershell [root@localhost cmdb]# ls -a pipsource . .. .pip 可以直接复制到基础镜像中，便于安装python项目依赖 ~~~ ~~~powershell [root@localhost cmdb]# ls requirement requirement.txt [root@localhost cmdb]# cat requirement/requirement.txt asn1crypto==0.24.0 astroid==2.1.0 autopep8==1.4.3 backcall==0.1.0 bcrypt==3.1.6 beautifulsoup4==4.7.1 certifi==2019.6.16 cffi==1.12.3 chardet==3.0.4 colorama==0.4.1 cryptography==2.4.2 decorator==4.4.0 Django==1.11.18 django-redis==4.10.0 Faker==2.0.2 geoip2==2.9.0 idna==2.8 ipython==7.6.0 ipython-genutils==0.2.0 isort==4.3.4 itchat==1.3.10 jedi==0.14.0 lazy-object-proxy==1.3.1 maxminddb==1.4.1 mccabe==0.6.1 paramiko==2.4.2 parso==0.5.0 pickleshare==0.7.5 pipenv==2018.11.26 prompt-toolkit==2.0.9 pyasn1==0.4.5 pycodestyle==2.5.0 pycparser==2.19 Pygments==2.4.2 pylint==2.2.2 pymongo==3.8.0 PyMySQL==0.9.3 PyNaCl==1.3.0 pypng==0.0.20 PyQRCode==1.2.1 python-dateutil==2.8.0 pytz==2019.1 redis==3.3.8 requests==2.22.0 six==1.12.0 soupsieve==1.9.1 text-unidecode==1.3 tornado==6.0.2 traitlets==4.3.2 urllib3==1.25.3 virtualenv==16.6.1 virtualenv-clone==0.5.3 wcwidth==0.1.7 webssh==1.4.5 wrapt==1.11.0 ~~~ ~~~powershell [root@localhost cmdb]# ls syscmdb dashboard products resources syscmdb users manage.py README.md static templates util ~~~ # 二、项目基础镜像准备 项目基础镜像主要是为项目提供运行环境，可以考虑直接下载，也可以考虑定制。本次采用定制。 > 可以在harbor主机上准备，准备完成后，再上传到harbor仓库，以备后续使用。 ## 2.1编辑用于生成基础镜像Dockerfile ~~~powershell [root@harborserver ~]# mkdir pythonprojectbaseimage [root@harborserver ~]# cd pythonprojectbaseimage/ [root@harborserver pythonprojectbaseimage]# ls Dockerfile pipsource requirement [root@harborserver pythonprojectbaseimage]# cat Dockerfile FROM centos:centos7 MAINTAINER "admin" WORKDIR / ADD pipsource/.pip /root ADD requirement/* / RUN yum -y install python36 gcc gcc-c++ python3-devel RUN pip3 install -r /requirement.txt ~~~ ## 2.2 使用Dockerfile生成基础项目 ~~~powershell [root@harborserver pythonprojectbaseimage]# docker build -t www.kubemsb.com/library/pythonprojectbaseimage:v1 . Sending build context to Docker daemon 6.144kB Step 1/7 : FROM centos:centos7 ---> 470671670cac Step 2/7 : MAINTAINER "admin" ---> Using cache ---> 9974940d0406 Step 3/7 : WORKDIR / ---> Using cache ---> b804fad33075 Step 4/7 : ADD pipsource/.pip /root ---> Using cache ---> 72d7ea2ef23b Step 5/7 : ADD requirement/* / ---> Using cache ---> 42ebd8ffb203 Step 6/7 : RUN yum -y install python36 gcc gcc-c++ python3-devel ---> Running in 35254e1d7411 Step 7/7 : RUN pip3 install -r /requirement.txt ---> Running in 3f90c1c684a2 Successfully built cbf45af22bda Successfully tagged www.kubemsb.com/library/pythonprojectbaseimage:v1 [root@harborserver pythonprojectbaseimage]# docker push www.kubemsb.com/library/pythonprojectbaseimage:v1 ~~~ # 三、项目数据库部署 ## 3.1 项目数据库文件准备 ~~~powershell [root@localhost cmdb]# ls db pipsource requirement syscmdb db目录用于保存项目数据库 [root@localhost cmdb]# ls db cmdbdb.sql ~~~ ## 3.2 编辑数据库部署的资源清单文件 ~~~powershell [root@nginxk8syaml html]# pwd /usr/share/nginx/html [root@nginxk8syaml html]# mkdir 04_python_project [root@nginxk8syaml html]# cd 04_python_project [root@nginxk8syaml 04_python_project]# ls mysql.yaml namespace.yaml [root@nginxk8syaml 04_python_project]# cat 01_namespace.yaml apiVersion: v1 kind: Namespace metadata: name: cmdb [root@nginxk8syaml pythonproject-yaml]# cat 02_mysql.yaml apiVersion: v1 kind: Service metadata: name: cmdbmysql namespace: cmdb spec: ports: - port: 3306 name: mysql clusterIP: None selector: app: mysqlcmdb --- apiVersion: apps/v1 kind: StatefulSet metadata: name: cmdbdb namespace: cmdb spec: selector: matchLabels: app: mysqlcmdb serviceName: "cmdbmysql" template: metadata: labels: app: mysqlcmdb spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD value: "123456" - name: MYSQL_DATABASE value: syscmdb ports: - containerPort: 3306 volumeMounts: - mountPath: "/var/lib/mysql" name: mysql-cmdb volumeClaimTemplates: - metadata: name: mysql-cmdb spec: accessModes: ["ReadWriteMany"] storageClassName: "nfs-client" resources: requests: storage: 1Gi ~~~ ## 3.3 应用数据库部署的资源清单文件 ![image-20220707111610983](../../img/kubernetes/kubernetes_deploy_python/image-20220707111610983.png) ~~~powershell [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/04_python_project/01_namespace.yaml [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/04_python_project/02_mysql.yaml ~~~ ## 3.4 导入数据库 ### 3.4.1 在k8smaster节点找到数据库文件 ~~~powershell [root@master01 ~]# ls syscmdb.sql ~~~ ### 3.4.2 查找数据库访问地址 ~~~powershell [root@master01 ~]# kubectl get pods -n cmdb NAME READY STATUS RESTARTS AGE cmdbdb-0 1/1 Running 0 3m45s [root@master01 ~]# kubectl get svc -n cmdb NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cmdbmysql ClusterIP None 3306/TCP 3m54s [root@master01 ~]# kubectl get ns NAME STATUS AGE cmdb Active 4m35s 使用命令进入查看 [root@master01 ~]# kubectl exec -it cmdbdb-0 sh -n cmdb # mysql -uroot -p123456 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.29 MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | | syscmdb | +--------------------+ 5 rows in set (0.01 sec) 测试连通性 [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/04_python_project/03_create_pod_busybox.yaml [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE busybox-pod 1/1 Running 55 2d15h [root@master01 ~]# kubectl exec -it busybox-pod sh / # nslookup cmdbdb-0.cmdbmysql.cmdb Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: cmdbdb-0.cmdbmysql.cmdb Address 1: 172.16.215.63 cmdbdb-0.cmdbmysql.cmdb.svc.cluster.local [root@master01 ~]# kubectl exec -it busybox-pod sh / # ping cmdbdb-0.cmdbmysql.cmdb PING cmdbdb-0.cmdbmysql.cmdb (172.16.215.63): 56 data bytes 64 bytes from 172.16.215.63: seq=0 ttl=62 time=0.448 ms 64 bytes from 172.16.215.63: seq=1 ttl=62 time=0.363 ms 64 bytes from 172.16.215.63: seq=2 ttl=62 time=0.543 ms 64 bytes from 172.16.215.63: seq=3 ttl=62 time=0.541 ms --- cmdbdb-0.cmdbmysql.cmdb ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 0.363/0.473/0.543 ms ~~~ **nfs server 验证** ~~~powershell [root@nfsserver ~]# ls /sdb/cmdb-mysql-cmdb-cmdbdb-0-pvc-5aa89483-fc27-428a-a556-0204a8e12e61/syscmdb/ db.opt 里面没有数据表 ~~~ ### 3.4.3 导入数据表 ~~~powershell [root@master1 ~]# kubectl exec cmdbdb-0 -n cmdb -it -- mysql -uroot -p123456 syscmdb < cmdbdb.sql [root@master1 ~]# kubectl exec -it cmdbdb-0 sh -n cmdb # mysql -uroot -p123456 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> use syscmdb; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +----------------------------+ | Tables_in_syscmdb | +----------------------------+ | auth_group | | auth_group_permissions | | auth_permission | | auth_user | | auth_user_groups | | auth_user_user_permissions | | django_admin_log | | django_content_type | | django_migrations | | django_session | | products_product | | resources_disk | | resources_idc | | resources_network | | resources_server | | resources_serverauto | | resources_serveruser | | users_profile | +----------------------------+ 18 rows in set (0.00 sec) ~~~ **nfs server 验证** ~~~powershell [root@nfsserver ~]# ls /sdb/cmdb-mysql-cmdb-cmdbdb-0-pvc-5aa89483-fc27-428a-a556-0204a8e12e61/syscmdb/ auth_group.frm django_admin_log.frm resources_idc.ibd auth_group.ibd django_admin_log.ibd resources_network.frm auth_group_permissions.frm django_content_type.frm resources_network.ibd auth_group_permissions.ibd django_content_type.ibd resources_serverauto.frm auth_permission.frm django_migrations.frm resources_serverauto.ibd auth_permission.ibd django_migrations.ibd resources_server.frm auth_user.frm django_session.frm resources_server.ibd auth_user_groups.frm django_session.ibd resources_serveruser.frm auth_user_groups.ibd products_product.frm resources_serveruser.ibd auth_user.ibd products_product.ibd users_profile.frm auth_user_user_permissions.frm resources_disk.frm users_profile.ibd auth_user_user_permissions.ibd resources_disk.ibd db.opt resources_idc.frm 里面已有数据表 ~~~ # 四、项目镜像准备 ## 4.1 目录及文件准备 ~~~powershell 创建制作镜像文件目录 [root@harborserver ~]# mkdir pythonprojectimage [root@harborserver ~]# cd pythonprojectimage/ [root@harborserver pythonprojectimage]# ls Dockerfile syscmdb 修改项目数据库连接文件 [root@harborserver pythonprojectimage]# vim syscmdb/syscmdb/settings.py ...... 84 DATABASES = { 85 'default': { 86 'ENGINE': 'django.db.backends.mysql', 87 'NAME': 'syscmdb', 88 'USER': 'root', 89 'PASSWORD': '123456', 90 'HOST': 'cmdbdb-0.cmdbmysql.cmdb', 91 'PORT': '3306', 92 } 93 } ...... 编辑制作项目镜像Dockerfile [root@harborserver pythonprojectimage]# cat Dockerfile FROM www.kubemsb.com/python-project/pythonprojectbaseimage:v1 MAINTAINER "admin" ADD . / WORKDIR /syscmdb EXPOSE 8000 CMD ["python3","manage.py","runserver","0.0.0.0:8000"] ~~~ ## 4.2 制定镜像 ~~~powershell [root@harborserver pythonprojectimage]# docker build -t www.kubemsb.com/python-project/pythonprojectimage:v1 . Sending build context to Docker daemon 26.5MB Step 1/6 : FROM www.kubemsb.com/python-project/pythonprojectbaseimage:v1 ---> cbf45af22bda Step 2/6 : MAINTAINER "admin" ---> Running in 5c4842096cd4 Removing intermediate container 5c4842096cd4 ---> ac510a15045d Step 3/6 : ADD syscmdb / ---> 380a9d3815c2 Step 4/6 : WORKDIR /syscmdb ---> Running in 181e423d7b7c Removing intermediate container 181e423d7b7c ---> 0c00978f568c Step 5/6 : EXPOSE 8000 ---> Running in fd34bab88e55 Removing intermediate container fd34bab88e55 ---> c2f30e672d35 Step 6/6 : CMD ["python3","manage.py","runserver","*:8000"] ---> Running in 561d17f2dfba Removing intermediate container 561d17f2dfba ---> b846c51ca3ba Successfully built b846c51ca3ba Successfully tagged www.kubemsb.com/python-project/pythonprojectimage:v1 ~~~ ## 4.3 上传镜像到harbor ~~~powershell [root@harborserver pythonprojectimage]# docker push www.kubemsb.com/python-project/pythonprojectimage:v1 ~~~ # 五、项目部署 ## 5.1 项目部署资源清单准备 ~~~powershell [root@nginxk8syaml 04_python_project]# ls deployment.yaml ingress.yaml mysql.yaml namespace.yaml service.yaml ~~~ ### 5.1.1 deployment.yaml ~~~powershell [root@nginxk8syaml 04_python_project]# cat 04_deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: pythoncmdb namespace: cmdb spec: replicas: 2 selector: matchLabels: project: pythoncmdb app: cmdb-demo template: metadata: labels: project: pythoncmdb app: cmdb-demo spec: imagePullSecrets: - name: harborreg #认证信息 containers: - name: cmdb image: www.kubemsb.com/library/pythonprojectimage:v1 #镜像 imagePullPolicy: Always ports: - containerPort: 8000 name: web protocol: TCP resources: requests: cpu: 0.5 memory: 1Gi limits: cpu: 1 memory: 2Gi livenessProbe: httpGet: path: / port: 8000 initialDelaySeconds: 60 timeoutSeconds: 20 readinessProbe: httpGet: path: / port: 8000 initialDelaySeconds: 60 timeoutSeconds: 20 ~~~ ### 5.1.2 service.yaml ~~~powershell [root@nginxk8syaml 04_python_project]# cat 05_service.yaml apiVersion: v1 kind: Service metadata: name: pythoncmdbsvc namespace: cmdb spec: selector: project: pythoncmdb app: cmdb-demo ports: - name: web port: 80 targetPort: 8000 # 由于使用ingress暴露，所以不使用NodePort ~~~ ### 5.1.3 ingress.yaml ~~~powershell [root@nginxk8syaml 04_python_project]# cat 06_ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: pythoncmdbingress namespace: cmdb annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: cmdb.kubemsb.com http: paths: - pathType: Prefix path: / backend: service: name: pythoncmdbsvc port: number: 80 ~~~ ## 5.2 应用项目部署资源清单 ![image-20220707114242171](../../img/kubernetes/kubernetes_deploy_python/image-20220707114242171.png) ~~~powershell [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/04_python_project/04_deployment.yaml deployment.apps/pythoncmdb created [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/04_python_project/05_service.yaml service/pythoncmdbsvc created [root@master01 ~]# kubectl apply -f http://yaml.kubemsb.com/04_python_project/06_ingress.yaml ingress.networking.k8s.io/pythoncmdbingress created ~~~ # 六、项目部署验证及访问 ## 6.1 验证 ~~~powershell [root@master01 ~]# kubectl get deployment.apps -n cmdb NAME READY UP-TO-DATE AVAILABLE AGE pythoncmdb 2/2 2 2 97s [root@master01 ~]# kubectl get svc -n cmdb NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cmdbmysql ClusterIP None 3306/TCP 27m pythoncmdbsvc ClusterIP 10.101.36.94 80/TCP 89s [root@master01 ~]# kubectl get pods -n cmdb NAME READY STATUS RESTARTS AGE cmdbdb-0 1/1 Running 0 27m pythoncmdb-56d4d84fd4-l927b 1/1 Running 0 116s pythoncmdb-56d4d84fd4-xb6kk 1/1 Running 0 116s [root@master01 ~]# kubectl get ingress -n cmdb NAME CLASS HOSTS ADDRESS PORTS AGE pythoncmdbingress cmdb.kubemsb.com 192.168.10.13 80 87s ~~~ ## 6.2 访问 ![image-20220707114750673](../../img/kubernetes/kubernetes_deploy_python/image-20220707114750673.png) ![image-20220707114850991](../../img/kubernetes/kubernetes_deploy_python/image-20220707114850991.png) ![image-20220707114925356](../../img/kubernetes/kubernetes_deploy_python/image-20220707114925356.png) ![image-20220707114946598](../../img/kubernetes/kubernetes_deploy_python/image-20220707114946598.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_devops.md ================================================ # DevOps ### 一、DevOps介绍 软件开发最开始是由两个团队组成： - 开发计划由[开发团队]()从头开始设计和整体系统的构建。需要系统不停的迭代更新。 - [运维团队]()将开发团队的Code进行测试后部署上线。希望系统稳定安全运行。 这看似两个目标不同的团队需要协同完成一个软件的开发。 在开发团队指定好计划并完成coding后，需要提供到运维团队。 运维团队向开发团队反馈需要修复的BUG以及一些需要返工的任务。 这时开发团队需要经常等待运维团队的反馈。这无疑延长了事件并推迟了整个软件开发的周期。 会有一种方式，在开发团队等待的时候，让开发团队转移到下一个项目中。等待运维团队为之前的代码提供反馈。 可是这样就意味着一个完整的项目需要一个更长的周期才可以开发出最终代码。 ---------- 基于现在的互联网现状，更推崇敏捷式开发，这样就导致项目的迭代速度更快，但是由于开发团队与运维团队的沟通问题，会导致新版本上线的时间成本很高。这又违背的敏捷式开发的最初的目的。 那么如果让开发团队和运维团队整合到成一个团队，协同应对一套软件呢？这就被称为[DevOps]()。 [DevOps]()，字面意思是Development &Operations的缩写，也就是开发&运维。 虽然字面意思只涉及到了开发团队和运维团队，其实QA测试团队也是参与其中的。 网上可以查看到[DevOps]()的符号类似于一个无穷大的符号 | [DevOps]() | | :--------------------------------------: | | ![image-20211124130409521](../../img/kubernetes/kubernetes_devops/image-20211124130409521.png) | 这表明[DevOps]()是一个不断提高效率并且持续不断工作的过程 [DevOps]()的方式可以让公司能够更快地应对更新和市场发展变化，开发可以快速交付，部署也更加稳定。 核心就在于[简化Dev和Ops团队之间的流程，使整体软件开发过程更快速。]() 整体的软件开发流程包括： - PLAN：开发团队根据客户的目标制定开发计划 - CODE：根据PLAN开始编码过程，需要将不同版本的代码存储在一个库中。 - BUILD：编码完成后，需要将代码构建并且运行。 - TEST：成功构建项目后，需要测试代码是否存在BUG或错误。 - DEPLOY：代码经过手动测试和自动化测试后，认定代码已经准备好部署并且交给运维团队。 - OPERATE：运维团队将代码部署到生产环境中。 - MONITOR：项目部署上线后，需要持续的监控产品。 - INTEGRATE：然后将监控阶段收到的反馈发送回PLAN阶段，整体反复的流程就是[DevOps]()的核心，即持续集成、持续部署。 为了保证整体流程可以高效的完成，各个阶段都有比较常见的工具，如下图： | 软件开发过程&涉及工具 | | :--------------------------------------: | | ![2021-11-23_175935](../../img/kubernetes/kubernetes_devops/2021-11-23_175935.png) | 最终可以给[DevOps]()下一个定义：[DevOps 强调的是高效组织团队之间如何通过自动化的工具协作和沟通来完成软件的生命周期管理，从而更快、更频繁地交付更稳定的软件。]() 自动化的工具协作和沟通来完成软件的生命周期管理 ### 二、Code阶段工具 在code阶段，我们需要将不同版本的代码存储到一个仓库中，常见的版本控制工具就是SVN或者Git，这里我们采用Git作为版本控制工具，GitLab作为远程仓库。 #### 2.1 Git安装 https://git-scm.com/（傻瓜式安装） #### 2.2 GitLab安装 单独准备服务器，采用Docker安装 - 查看GitLab镜像 ```sh docker search gitlab ``` - 拉取GitLab镜像 ```sh docker pull gitlab/gitlab-ce ``` - 准备docker-compose.yml文件 ```yml version: '3.1' services: gitlab: image: 'gitlab/gitlab-ce:latest' container_name: gitlab restart: always environment: GITLAB_OMNIBUS_CONFIG: | external_url 'http://192.168.11.11:8929' gitlab_rails['gitlab_shell_ssh_port'] = 2224 ports: - '8929:8929' - '2224:2224' volumes: - './config:/etc/gitlab' - './logs:/var/log/gitlab' - './data:/var/opt/gitlab' ``` - 启动容器（需要稍等一小会……） ```sh docker-compose up -d ``` - 访问GitLab首页 | 首页 | | :--------------------------------------: | | ![image-20211124182140596](../../img/kubernetes/kubernetes_devops/image-20211124182140596.png) | - 查看root用户初始密码 ```sh docker exec -it gitlab cat /etc/gitlab/initial_root_password ``` | 初始密码 | | :--------------------------------------: | | ![image-20211124182921234](../../img/kubernetes/kubernetes_devops/image-20211124182921234.png) | - 登录root用户 | 登录成功后跳转页面 | | :--------------------------------------: | | ![image-20211124183003858](../../img/kubernetes/kubernetes_devops/image-20211124183003858.png) | - 第一次登录后需要修改密码 | 修改密码 | | :--------------------------------------: | | ![image-20211124193444561](../../img/kubernetes/kubernetes_devops/image-20211124193444561.png) | 搞定后，即可像Gitee、GitHub一样使用。 ### 三、Build阶段工具 构建Java项目的工具一般有两种选择，一个是Maven，一个是Gradle。 这里我们选择Maven作为项目的编译工具。 具体安装Maven流程不做阐述，但是需要确保配置好Maven仓库私服以及JDK编译版本。 ### 四、Operate阶段工具 部署过程，会采用Docker进行部署，暂时只安装Docker即可，后续还需安装Kubenetes #### 4.1 Docker安装 - 准备测试环境&生产环境 - 下载Docker依赖组件 ```sh yum -y install yum-utils device-mapper-persistent-data lvm2 ``` - 设置下载Docker的镜像源为阿里云 ```sh yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ``` - 安装Docker服务 ```sh yum -y install docker-ce ``` - 安装成功后，启动Docker并设置开机自启 ```sh # 启动Docker服务 systemctl start docker # 设置开机自动启动 systemctl enable docker ``` - 测试安装成功 ```sh docker version ``` | 效果 | | :--------------------------------------: | | ![image-20211124200317795](../../img/kubernetes/kubernetes_devops/image-20211124200317795.png) | #### 4.2 Docker-Compose安装 - 下载Docker/Compose：https://github.com/docker/compose - 将下载好的[docker-compose-Linux-x86_64]()文件移动到Linux操作系统：…… - 设置[docker-compose-Linux-x86_64]()文件权限，并移动到$PATH目录中 ```sh # 设置文件权限 chmod a+x docker-compose-Linux-x86_64 # 移动到/usr/bin目录下，并重命名为docker-compose mv docker-compose-Linux-x86_64 /usr/bin/docker-compose ``` - 测试安装成功 ```sh docker-compose version ``` | 效果 | | :--------------------------------------: | | ![image-20211124200658107](../../img/kubernetes/kubernetes_devops/image-20211124200658107.png) | ### 五、Integrate工具 持续集成、持续部署的工具很多，其中Jenkins是一个开源的持续集成平台。 Jenkins涉及到将编写完毕的代码发布到测试环境和生产环境的任务，并且还涉及到了构建项目等任务。 Jenkins需要大量的插件保证工作，安装成本较高，下面会基于Docker搭建Jenkins。 #### 5.1 Jenkins介绍 Jenkins是一个开源软件项目，是基于Java开发的一种持续集成工具 Jenkins应用广泛，大多数互联网公司都采用Jenkins配合GitLab、Docker、K8s作为实现[DevOps]()的核心工具。 Jenkins最强大的就在于插件，Jenkins官方提供了大量的插件库，来自动化CI/CD过程中的各种琐碎功能。 | | | | :--------------------------------------: | :--------------------------------------: | | ![image-20211125141950900](../../img/kubernetes/kubernetes_devops/image-20211125141950900.png) | ![image-20211124200317795](../../img/kubernetes/kubernetes_devops/image-20211125141701495.png) | Jenkins最主要的工作就是将GitLab上可以构建的工程代码拉取并且进行构建，再根据流程可以选择发布到测试环境或是生产环境。 一般是GitLab上的代码经过大量的测试后，确定发行版本，再发布到生产环境。 CI/CD可以理解为： - CI过程即是通过Jenkins将代码拉取、构建、制作镜像交给测试人员测试。 - 持续集成：让软件代码可以持续的集成到主干上，并自动构建和测试。 - CD过程即是通过Jenkins将打好标签的发行版本代码拉取、构建、制作镜像交给运维人员部署。 - 持续交付：让经过持续集成的代码可以进行手动部署。 - 持续部署：让可以持续交付的代码随时随地的自动化部署。 | CI、CD | | :--------------------------------------: | | ![image-20211125154112097](../../img/kubernetes/kubernetes_devops/image-20211125154112097.png) | #### 5.2 Jenkins安装 - 拉取Jenkins镜像 ``` docker pull jenkins/jenkins ``` - 编写docker-compose.yml ```yml version: "3.1" services: jenkins: image: jenkins/jenkins container_name: jenkins ports: - 8080:8080 - 50000:50000 volumes: - ./data/:/var/jenkins_home/ ``` - 首次启动会因为数据卷data目录没有权限导致启动失败，设置data目录写权限 | 错误日志 | | :--------------------------------------: | | ![image-20211124202610243](../../img/kubernetes/kubernetes_devops/image-20211124202610243.png) | ```sh chmod -R a+w data/ ``` - 重新启动Jenkins容器后，由于Jenkins需要下载大量内容，但是由于默认下载地址下载速度较慢，需要重新设置下载地址为国内镜像站 ```sh # 修改数据卷中的hudson.model.UpdateCenter.xml文件 default https://updates.jenkins.io/update-center.json # 将下载地址替换为http://mirror.esuni.jp/jenkins/updates/update-center.json default http://mirror.esuni.jp/jenkins/updates/update-center.json # 清华大学的插件源也可以https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json ``` - 再次重启Jenkins容器，访问Jenkins（需要稍微等会） | Jenkins首页 | | :--------------------------------------: | | ![image-20211124204517433](../../img/kubernetes/kubernetes_devops/image-20211124204517433.png) | | ![image-20211124203336300](../../img/kubernetes/kubernetes_devops/image-20211124203336300.png) | - 查看密码登录Jenkins，并登录下载插件 ```sh docker exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword ``` | 登录并下载插件 | | :--------------------------------------: | | ![image-20211124205050484](../../img/kubernetes/kubernetes_devops/image-20211124205050484.png) | | ![image-20211124205513465](../../img/kubernetes/kubernetes_devops/image-20211124205513465.png) | - 选择需要安装的插件 | 选择需要安装的插件 | | :--------------------------------------: | | ![image-20211124205854418](../../img/kubernetes/kubernetes_devops/image-20211124205854418.png) | | ![image-20211124205858730](../../img/kubernetes/kubernetes_devops/image-20211124205858730.png) | | ![image-20211124205917317](../../img/kubernetes/kubernetes_devops/image-20211124205917317.png) | - 下载完毕设置信息进入首页（可能会出现下载失败的插件） | | | ---------------------------------------- | | ![image-20211124211635550](../../img/kubernetes/kubernetes_devops/image-20211124211635550.png) | | ![image-20211124211700999](../../img/kubernetes/kubernetes_devops/image-20211124211700999.png) | | ![image-20211124211720836](../../img/kubernetes/kubernetes_devops/image-20211124211720836.png) | #### 5.3 Jenkins入门配置 由于Jenkins需要从Git拉取代码、需要本地构建、甚至需要直接发布自定义镜像到Docker仓库，所以Jenkins需要配置大量内容。 ##### 5.3.1 构建任务 准备好GitLab仓库中的项目，并且通过Jenkins配置项目的实现当前项目的[DevOps]()基本流程。 - 构建Maven工程发布到GitLab（Gitee、Github均可） | GitLab查看项目 | | :--------------------------------------: | | ![image-20211125195818670](../../img/kubernetes/kubernetes_devops/image-20211125195818670.png) | - Jenkins点击左侧导航新建任务 | 新建任务 | | :--------------------------------------: | | ![image-20211125163541645](../../img/kubernetes/kubernetes_devops/image-20211125163541645.png) | - 选择自由风格构建任务 | 构建任务 | | :--------------------------------------: | | ![image-20211125170350811](../../img/kubernetes/kubernetes_devops/image-20211125170350811.png) | ##### 5.3.1 配置源码拉取地址 Jenkins需要将Git上存放的源码存储到Jenkins服务所在磁盘的本地 - 配置任务源码拉取的地址 | 源码管理 | | :--------------------------------------: | | ![image-20211125170418337](../../img/kubernetes/kubernetes_devops/image-20211125170418337.png) | - Jenkins立即构建 | 点击任务test中的立即构建 | | :--------------------------------------: | | ![image-20211125200218093](../../img/kubernetes/kubernetes_devops/image-20211125200218093.png) | - 查看构建工程的日志，点击上述③的任务条即可 | 查看任务拉取Git源码日志 | | :--------------------------------------: | | ![image-20211125201701443](../../img/kubernetes/kubernetes_devops/image-20211125201701443.png) | 可以看到源码已经拉取带Jenkins本地，可以根据第三行日志信息，查看Jenkins本地拉取到的源码。 - 查看Jenkins容器中[/var/jenkins_home/workspace/test]()的源码 | 源码存放位置 | | :--------------------------------------: | | ![image-20211125201919108](../../img/kubernetes/kubernetes_devops/image-20211125201919108.png) | ##### 5.3.2 配置Maven构建代码 代码拉取到Jenkins本地后，需要在Jenkins中对代码进行构建，这里需要Maven的环境，而Maven需要Java的环境，接下来需要在Jenkins中安装JDK和Maven，并且配置到Jenkins服务。 - 准备JDK、Maven压缩包通过数据卷映射到Jenkins容器内部 | 数据卷存放位置 | | :--------------------------------------: | | ![image-20211125203757232](../../img/kubernetes/kubernetes_devops/image-20211125203757232.png) | - 解压压缩包，并配置Maven的settings.xml ```xml alimaven aliyun maven http://maven.aliyun.com/nexus/content/groups/public/ central jdk-1.8 true 1.8 1.8 1.8 1.8 ``` - Jenkins配置JDK&Maven并保存 | | | ---------------------------------------- | | ![image-20211125204811759](../../img/kubernetes/kubernetes_devops/image-20211125204811759.png) | | ![image-20211125204818869](../../img/kubernetes/kubernetes_devops/image-20211125204818869.png) | - 配置Jenkins任务构建代码 | 配置Maven构建代码 | | :--------------------------------------: | | ![image-20211125205027013](../../img/kubernetes/kubernetes_devops/image-20211125205027013.png) | | ![image-20211125205020738](../../img/kubernetes/kubernetes_devops/image-20211125205020738.png) | - 立即构建测试，查看target下的jar包 | 构建源码 | | :--------------------------------------: | | ![image-20211125205240208](../../img/kubernetes/kubernetes_devops/image-20211125205240208.png) | | ![image-20211125205725948](../../img/kubernetes/kubernetes_devops/image-20211125205725948.png) | ##### 5.3.3 配置Publish发布&远程操作 jar包构建好之后，就可以根据情况发布到测试或生产环境，这里需要用到之前下载好的插件Publish Over SSH。 - 配置Publish Over SSH连接测试、生产环境 | Publish Over SSH配置 | | :--------------------------------------: | | ![image-20211125210148202](../../img/kubernetes/kubernetes_devops/image-20211125210148202.png) | - 配置任务的构建后操作，发布jar包到目标服务 | 配置构建后操作 | | :--------------------------------------: | | ![image-20211125205027013](../../img/kubernetes/kubernetes_devops/image-20211125205027013.png) | | ![image-20211125210424346](../../img/kubernetes/kubernetes_devops/image-20211125210424346.png) | | ![image-20211125210626631](../../img/kubernetes/kubernetes_devops/image-20211125210626631.png) | - 立即构建任务，并去目标服务查看 | 立即构建 | | :--------------------------------------: | | ![image-20211125210755556](../../img/kubernetes/kubernetes_devops/image-20211125210755556.png) | | ![image-20211125210826057](../../img/kubernetes/kubernetes_devops/image-20211125210826057.png) | ### 六、CI、CD入门操作 基于Jenkins拉取GitLab的SpringBoot代码进行构建发布到测试环境实现持续集成 基于Jenkins拉取GitLab指定发行版本的SpringBoot代码进行构建发布到生产环境实现CD实现持续部署 #### 6.1 持续集成 为了让程序代码可以自动推送到测试环境基于Docker服务运行，需要添加Docker配置和脚本文件让程序可以在集成到主干的同时运行起来。 - 添加Dockerfile文件 | 构建自定义镜像 | | :--------------------------------------: | | ![image-20211126161304485](../../img/kubernetes/kubernetes_devops/image-20211126161304485.png) | - 添加docker-compose.yml文件 | 加载自定义镜像启动容器 | | :--------------------------------------: | | ![image-20211126161331991](../../img/kubernetes/kubernetes_devops/image-20211126161331991.png) | - 追加Jenkins构建后操作脚本命令 | 构建后发布并执行脚本命令 | | :--------------------------------------: | | ![image-20211126161408514](../../img/kubernetes/kubernetes_devops/image-20211126161408514.png) | - 发布到GitLab后由Jenkins立即构建并托送到目标服务器 | 构建日志 | | :--------------------------------------: | | ![image-20211126161448527](../../img/kubernetes/kubernetes_devops/image-20211126161448527.png) | - 测试部署到目标服务器程序 | 查看目标服务器并测试接口 | | :--------------------------------------: | | ![image-20211126161504715](../../img/kubernetes/kubernetes_devops/image-20211126161504715.png) | | ![image-20211126161509411](../../img/kubernetes/kubernetes_devops/image-20211126161509411.png) | #### 6.2 持续交付、部署 程序代码在经过多次集成操作到达最终可以交付，持续交付整体流程和持续集成类似，不过需要选取指定的发行版本 - 下载Git Parameter插件 | 下载Git Parameter | | :--------------------------------------: | | ![image-20211126165209057](../../img/kubernetes/kubernetes_devops/image-20211126165209057.png) | - 设置项目参数化构建 | 基于Git标签构建 | | :--------------------------------------: | | ![image-20211126165444124](../../img/kubernetes/kubernetes_devops/image-20211126165444124.png) | | ![image-20211126172828266](../../img/kubernetes/kubernetes_devops/image-20211126172828266.png) | - 给项目添加tag版本 | 添加tag版本 | | :--------------------------------------: | | ![image-20211126165639286](../../img/kubernetes/kubernetes_devops/image-20211126165639286.png) | - 任务构建时，采用Shell方式构建，拉取指定tag版本代码 | 切换指定标签并构建项目 | | :--------------------------------------: | | ![image-20211126174715028](../../img/kubernetes/kubernetes_devops/image-20211126174715028.png) | - 基于Parameter构建任务，任务发布到目标服务器 | 构建任务 | | :--------------------------------------: | | ![image-20211126174159487](../../img/kubernetes/kubernetes_devops/image-20211126174159487.png) | ### 七、集成Sonar Qube #### 7.1 Sonar Qube介绍 Sonar Qube是一个开源的代码分析平台，支持Java、Python、PHP、JavaScript、CSS等25种以上的语言，可以检测出重复代码、代码漏洞、代码规范和安全性漏洞的问题。 Sonar Qube可以与多种软件整合进行代码扫描，比如Maven，Gradle，Git，Jenkins等，并且会将代码检测结果推送回Sonar Qube并且在系统提供的UI界面上显示出来 | Sonar Qube的UI界面 | | :--------------------------------------: | | ![image-20211129190039986](../../img/kubernetes/kubernetes_devops/image-20211129190039986.png) | #### 7.2 Sonar Qube环境搭建 ##### 7.2.1 Sonar Qube安装 Sonar Qube在7.9版本中已经放弃了对MySQL的支持，并且建议在商业环境中采用PostgreSQL，那么安装Sonar Qube时需要依赖PostgreSQL。 并且这里会安装Sonar Qube的长期支持版本[8.9]() - 拉取镜像 ```sh docker pull postgres docker pull sonarqube:8.9.3-community ``` - 编写docker-compoe.yml ```yml version: "3.1" services: db: image: postgres container_name: db ports: - 5432:5432 networks: - sonarnet environment: POSTGRES_USER: sonar POSTGRES_PASSWORD: sonar sonarqube: image: sonarqube:8.9.3-community container_name: sonarqube depends_on: - db ports: - "9000:9000" networks: - sonarnet environment: SONAR_JDBC_URL: jdbc:postgresql://db:5432/sonar SONAR_JDBC_USERNAME: sonar SONAR_JDBC_PASSWORD: sonar networks: sonarnet: driver: bridge ``` - 启动容器 ``` docker-compose up -d ``` - 需要设置sysctl.conf文件信息 | 设置vm.max_map_count | | :--------------------------------------: | | ![image-20211207145215817](../../img/kubernetes/kubernetes_devops/image-20211207145215817.png) | | ![image-20211207145342350](../../img/kubernetes/kubernetes_devops/image-20211207145342350.png) | 并执行命令刷新 ``` sysctl -p ``` - 重新启动需要一定时间启动，可以可以查看容器日志，看到如下内容代表启动成功 | 容器日志 | | :--------------------------------------: | | ![image-20211129191426344](../../img/kubernetes/kubernetes_devops/image-20211129191426344.png) | - 访问Sonar Qube首页 | 登录 | | :--------------------------------------: | | ![image-20211129191537050](../../img/kubernetes/kubernetes_devops/image-20211129191537050.png) | - 还需要重新设置一次密码 | 重新设置密码 | | :--------------------------------------: | | ![image-20211129193824428](../../img/kubernetes/kubernetes_devops/image-20211129193824428.png) | - Sonar Qube首页 | Sonar Qube首页 | | :--------------------------------------: | | ![image-20211129194148239](../../img/kubernetes/kubernetes_devops/image-20211129194148239.png) | ##### 7.2.2 安装中文插件 | 安装中文插件 | | :--------------------------------------: | | ![image-20211129194621820](../../img/kubernetes/kubernetes_devops/image-20211129194621820.png) | 安装成功后需要重启，安装失败重新点击install重装即可。 安装成功后，会查看到重启按钮，点击即可 | 重启按钮 | | :--------------------------------------: | | ![image-20211129194748765](../../img/kubernetes/kubernetes_devops/image-20211129194748765.png) | 重启后查看效果 | 首页效果 | | :--------------------------------------: | | ![image-20211129194931944](../../img/kubernetes/kubernetes_devops/image-20211129194931944.png) | #### 7.3 Sonar Qube基本使用 Sonar Qube的使用方式很多，Maven可以整合，也可以采用sonar-scanner的方式，再查看Sonar Qube的检测效果 ##### 7.3.1 Maven实现代码检测 - 修改Maven的settings.xml文件配置Sonar Qube信息 ```xml sonar true admin 123456789 http://192.168.11.11:9000 ``` - 在代码位置执行命令：mvn sonar:sonar | 执行代码检测 | | :--------------------------------------: | | ![image-20211129195430146](../../img/kubernetes/kubernetes_devops/image-20211129195430146.png) | - 查看Sonar Qube界面检测结果 | Sonar Qube检测结果 | | :--------------------------------------: | | ![image-20211129195503762](../../img/kubernetes/kubernetes_devops/image-20211129195503762.png) | ##### 7.3.2 Sonar-scanner实现代码检测 - 下载Sonar-scanner：https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/ 下载4.6.x版本即可，要求Linux版本 - 解压并配置sonar服务端信息 - 由于是zip压缩包，需要安装unzip解压插件 ```sh yum -y install unzip ``` - 解压压缩包 ```sh unzip sonar-scanner-cli/sonar-scanner-cli-4.6.0.2311-linux.zip ``` - 配置sonarQube服务端地址，修改conf下的sonar-scanner.properties | 配置服务端信息 | | :--------------------------------------: | | ![image-20211130140043382](../../img/kubernetes/kubernetes_devops/image-20211130140043382.png) | - 执行命令检测代码 ```sh # 在项目所在目录执行以下命令 ~/sonar-scanner/bin/sonar-scanner -Dsonar.sources=./ -Dsonar.projectname=demo -Dsonar.projectKey=java -Dsonar.java.binaries=target/ ``` [Ps：主要查看我的sonar-scanner执行命令的位置]() | 查看日志信息 | | :--------------------------------------: | | ![image-20211130141303457](../../img/kubernetes/kubernetes_devops/image-20211130141303457.png) | - 查看SonarQube界面检测结果 | 检测结果 | | :--------------------------------------: | | ![image-20211130144608025](../../img/kubernetes/kubernetes_devops/image-20211130144608025.png) | #### 7.4 Jenkins集成Sonar Qube Jenkins继承Sonar Qube实现代码扫描需要先下载整合插件 ##### 7.4.1 Jenkins安装插件 | 下载Sonar Qube插件 | | :--------------------------------------: | | ![image-20211129201625561](../../img/kubernetes/kubernetes_devops/image-20211129201625561.png) | | ![image-20211129201607240](../../img/kubernetes/kubernetes_devops/image-20211129201607240.png) | | ![image-20211129202147390](../../img/kubernetes/kubernetes_devops/image-20211129202147390.png) | ##### 7.4.2 Jenkins配置Sonar Qube - 开启Sonar Qube权限验证 | 开启Sonar Qube权限校验 | | :--------------------------------------: | | ![image-20211130144850186](../../img/kubernetes/kubernetes_devops/image-20211130144850186.png) | - 获取Sonar Qube的令牌 | 获取令牌 | | :--------------------------------------: | | ![image-20211129203102334](../../img/kubernetes/kubernetes_devops/image-20211129203102334.png) | - 配置Jenkins的Sonar Qube信息 | | | ---------------------------------------- | | ![image-20211129203235019](../../img/kubernetes/kubernetes_devops/image-20211129203235019.png) | | ![image-20211129203342171](../../img/kubernetes/kubernetes_devops/image-20211129203342171.png) | | ![image-20211129203457604](../../img/kubernetes/kubernetes_devops/image-20211129203457604.png) | ##### 7.4.3 配置Sonar-scanner - 将Sonar-scaner添加到Jenkins数据卷中并配置全局配置 | 配置Sonar-scanner | | :--------------------------------------: | | ![image-20211130153628925](../../img/kubernetes/kubernetes_devops/image-20211130153628925.png) | - 配置任务的Sonar-scanner | 配置任务的Sonar-scanner | | :--------------------------------------: | | ![image-20211130155849143](../../img/kubernetes/kubernetes_devops/image-20211130155849143.png) | ##### 7.4.4 构建任务 | 构建任务 | | :--------------------------------------: | | ![image-20211130160017465](../../img/kubernetes/kubernetes_devops/image-20211130160017465.png) | | ![image-20211130160047648](../../img/kubernetes/kubernetes_devops/image-20211130160047648.png) | ### 八、集成Harbor #### 8.1 Harbor介绍 前面在部署项目时，我们主要采用Jenkins推送jar包到指定服务器，再通过脚本命令让目标服务器对当前jar进行部署，这种方式在项目较多时，每个目标服务器都需要将jar包制作成自定义镜像再通过docker进行启动，重复操作比较多，会降低项目部署时间。 我们可以通过Harbor作为私有的Docker镜像仓库。让Jenkins统一将项目打包并制作成Docker镜像发布到Harbor仓库中，只需要通知目标服务，让目标服务统一去Harbor仓库上拉取镜像并在本地部署即可。 Docker官方提供了Registry镜像仓库，但是Registry的功能相对简陋。Harbor是VMware公司提供的一款镜像仓库，提供了权限控制、分布式发布、强大的安全扫描与审查机制等功能 #### 8.2 Harbor安装 这里采用原生的方式安装Harbor。 - 下载Harbor安装包：https://github.com/goharbor/harbor/releases/download/v2.3.4/harbor-offline-installer-v2.3.4.tgz - 拖拽到Linux并解压： ``` tar -zxvf harbor-offline-installer-v2.3.4.tgz -C /usr/local/ ``` - 修改Harbor配置文件： - 首先复制一份harbor.yml配置 ```sh cp harbor.yml.tmpl harbor.yml ``` - 编辑harbor.yml配置文件 | 配置Harbor文件 | | :--------------------------------------: | | ![image-20211130215555218](../../img/kubernetes/kubernetes_devops/image-20211130215555218.png) | - 启动Harbor ```sh ./install.sh ``` | 查看日志 | | :--------------------------------------: | | ![image-20211130215941857](../../img/kubernetes/kubernetes_devops/image-20211130215941857.png) | - 登录Harbor | 登录Harbor | | :--------------------------------------: | | ![image-20211130220028840](../../img/kubernetes/kubernetes_devops/image-20211130220028840.png) | - 首页信息 | 首页信息 | | :--------------------------------------: | | ![image-20211130220111602](../../img/kubernetes/kubernetes_devops/image-20211130220111602.png) | #### 8.3 Harbor使用方式 Harbor作为镜像仓库，主要的交互方式就是将镜像上传到Harbor上，以及从Harbor上下载指定镜像 在传输镜像前，可以先使用Harbor提供的权限管理，将项目设置为私有项目，并对不同用户设置不同角色，从而更方便管理镜像。 ##### 8.3.1 添加用户构建项目 - 创建用户 | 创建用户 | | :--------------------------------------: | | ![image-20211201213427157](../../img/kubernetes/kubernetes_devops/image-20211201213427157.png) | - 构建项目（设置为私有） | 构建项目 | | :--------------------------------------: | | ![image-20211201213751780](../../img/kubernetes/kubernetes_devops/image-20211201213751780.png) | - 给项目追加用户 | 追加用户管理 | | :--------------------------------------: | | ![image-20211201213832458](../../img/kubernetes/kubernetes_devops/image-20211201213832458.png) | - 切换测试用户 | 切换测试用户 | | :--------------------------------------: | | ![image-20211201214008303](../../img/kubernetes/kubernetes_devops/image-20211201214008303.png) | ##### 8.3.2 发布镜像到Harbor - 修改镜像名称 名称要求：[harbor地址/项目名/镜像名:版本]() | 修改镜像名称 | | ---------------------------------------- | | ![image-20211201221040200](../../img/kubernetes/kubernetes_devops/image-20211201221040200.png) | - 修改daemon.json，支持Docker仓库，并重启Docker | 修改daemon.json，支持Docker仓库 | | :--------------------------------------: | | ![image-20211201215931237](../../img/kubernetes/kubernetes_devops/image-20211201215931237.png) | - 设置登录仓库信息 ```sh docker login -u 用户名 -p 密码 Harbor地址 ``` - 推送镜像到Harbor | 推送镜像到Harbor | | :--------------------------------------: | | ![image-20211201221225196](../../img/kubernetes/kubernetes_devops/image-20211201221225196.png) | | ![image-20211201221300055](../../img/kubernetes/kubernetes_devops/image-20211201221300055.png) | ##### 8.3.3 从Harbor拉取镜像ls 跟传统方式一样，不过需要先配置[/etc/docker/daemon.json]()文件 ```json { "registry-mirrors": ["https://pee6w651.mirror.aliyuncs.com"], "insecure-registries": ["192.168.11.11:80"] } ``` | 拉取镜像 | | :--------------------------------------: | | ![image-20211201222450091](../../img/kubernetes/kubernetes_devops/image-20211201222450091.png) | ##### 8.3.4 Jenkins容器使用宿主机Docker 构建镜像和发布镜像到harbor都需要使用到docker命令。而在Jenkins容器内部安装Docker官方推荐直接采用宿主机带的Docker即可。 设置Jenkins容器使用宿主机Docker - 设置宿主机docker.sock权限： ```sh sudo chown root:root /var/run/docker.sock sudo chmod o+rw /var/run/docker.sock ``` - 添加数据卷 ```yml version: "3.1" services: jenkins: image: jenkins/jenkins container_name: jenkins ports: - 8080:8080 - 50000:50000 volumes: - ./data/:/var/jenkins_home/ - /usr/bin/docker:/usr/bin/docker - /var/run/docker.sock:/var/run/docker.sock - /etc/docker/daemon.json:/etc/docker/daemon.json ``` ##### 8.3.5 添加构建操作 | 制作自定义镜像 | | :--------------------------------------: | | ![image-20211229155834500](../../img/kubernetes/kubernetes_devops/image-20211229155834500.png) | ##### 8.3.6 编写部署脚本 部署项目需要通过Publish Over SSH插件，让目标服务器执行命令。为了方便一次性实现拉取镜像和启动的命令，推荐采用脚本文件的方式。 添加脚本文件到目标服务器，再通过Publish Over SSH插件让目标服务器执行脚本即可。 - 编写脚本文件，添加到目标服务器 ```sh harbor_url=$1 harbor_project_name=$2 project_name=$3 tag=$4 port=$5 imageName=$harbor_url/$harbor_project_name/$project_name:$tag containerId=`docker ps -a | grep ${project_name} | awk '{print $1}'` if [ "$containerId" != "" ] ; then docker stop $containerId docker rm $containerId echo "Delete Container Success" fi imageId=`docker images | grep ${project_name} | awk '{print $3}'` if [ "$imageId" != "" ] ; then docker rmi -f $imageId echo "Delete Image Success" fi docker login -u DevOps -p P@ssw0rd $harbor_url docker pull $imageName docker run -d -p $port:$port --name $project_name $imageName echo "Start Container Success" echo $project_name ``` 并设置权限为可执行 ``` chmod a+x deploy.sh ``` | 如图 | | :--------------------------------------: | | ![image-20211203192047357](../../img/kubernetes/kubernetes_devops/image-20211203192047357.png) | ##### 8.3.7 配置构建后操作 | 执行脚本文件 | | :--------------------------------------: | | ![image-20211229155949038](../../img/kubernetes/kubernetes_devops/image-20211229155949038.png) | ### 九、Jenkins流水线 #### 9.1 Jenkins流水线任务介绍 之前采用Jenkins的自由风格构建的项目，每个步骤流程都要通过不同的方式设置，并且构建过程中整体流程是不可见的，无法确认每个流程花费的时间，并且问题不方便定位问题。 Jenkins的Pipeline可以让项目的发布整体流程可视化，明确执行的阶段，可以快速的定位问题。并且整个项目的生命周期可以通过一个Jenkinsfile文件管理，而且Jenkinsfile文件是可以放在项目中维护。 所以Pipeline相对自由风格或者其他的项目风格更容易操作。 #### 9.2 Jenkins流水线任务 ##### 9.2.1 构建Jenkins流水线任务 - 构建任务 | 构建Jenkins流水线任务 | | ---------------------------------------- | | ![image-20211202144429302](../../img/kubernetes/kubernetes_devops/image-20211202144429302.png) | - 生成Groovy脚本 | Hello World脚本生成 | | :--------------------------------------: | | ![image-20211202144531749](../../img/kubernetes/kubernetes_devops/image-20211202144531749.png) | - 构建后查看视图 | 构建后查看视图 | | :--------------------------------------: | | ![image-20211202144616117](../../img/kubernetes/kubernetes_devops/image-20211202144616117.png) | ##### 9.2.2 Groovy脚本 - Groovy脚本基础语法 ```sh // 所有脚本命令包含在pipeline{}中 pipeline { // 指定任务在哪个节点执行（Jenkins支持分布式） agent any // 配置全局环境，指定变量名=变量值信息 environment{ host = '192.168.11.11' } // 存放所有任务的合集 stages { // 单个任务 stage('任务1') { // 实现任务的具体流程 steps { echo 'do something' } } // 单个任务 stage('任务2') { // 实现任务的具体流程 steps { echo 'do something' } } // …… } } ``` - 编写例子测试 ```sh pipeline { agent any // 存放所有任务的合集 stages { stage('拉取Git代码') { steps { echo '拉取Git代码' } } stage('检测代码质量') { steps { echo '检测代码质量' } } stage('构建代码') { steps { echo '构建代码' } } stage('制作自定义镜像并发布Harbor') { steps { echo '制作自定义镜像并发布Harbor' } } stage('基于Harbor部署工程') { steps { echo '基于Harbor部署工程' } } } } ``` | 配置Grovvy脚本 | | :--------------------------------------: | | ![image-20211202145155428](../../img/kubernetes/kubernetes_devops/image-20211202145155428.png) | - 查看效果 | 查看效果 | | :--------------------------------------: | | ![image-20211202145240166](../../img/kubernetes/kubernetes_devops/image-20211202145240166.png) | [Ps：涉及到特定脚本，Jenkins给予了充足的提示，可以自动生成命令]() | 生成命令位置 | | :--------------------------------------: | | ![image-20211202145349043](../../img/kubernetes/kubernetes_devops/image-20211202145349043.png) | ##### 9.2.3 Jenkinsfile实现 Jenkinsfile方式需要将脚本内容编写到项目中的Jenkinsfile文件中，每次构建会自动拉取项目并且获取项目中Jenkinsfile文件对项目进行构建 - 配置pipeline | 配置pipeline | | :--------------------------------------: | | ![image-20211202151127254](../../img/kubernetes/kubernetes_devops/image-20211202151127254.png) | - 准备Jenkinsfile | 准备Jenkinsfile文件 | | :--------------------------------------: | | ![image-20211202151155145](../../img/kubernetes/kubernetes_devops/image-20211202151155145.png) | - 测试效果 | 测试效果 | | :--------------------------------------: | | ![image-20211202151225161](../../img/kubernetes/kubernetes_devops/image-20211202151225161.png) | #### 9.3 Jenkins流水线任务实现 ##### 9.3.1 参数化构建 添加参数化构建，方便选择不的项目版本 | Git参数化构建 | | :--------------------------------------: | | ![image-20211202191944277](../../img/kubernetes/kubernetes_devops/image-20211202191944277.png) | ##### 9.3.2 拉取Git代码 通过流水线语法生成Checkout代码的脚本 | 语法生成 | | :--------------------------------------: | | ![image-20211202192047619](../../img/kubernetes/kubernetes_devops/image-20211202192047619.png) | ![image-20211202192129895](../../img/kubernetes/kubernetes_devops/image-20211202192129895.png)将*/master更改为标签[${tag}]() ```sh pipeline { agent any stages { stage('拉取Git代码') { steps { checkout([$class: 'GitSCM', branches: [[name: '${tag}']], extensions: [], userRemoteConfigs: [[url: 'http://49.233.115.171:8929/root/test.git']]]) } } } } ``` ##### 9.3.3 构建代码 通过脚本执行mvn的构建命令 ```sh pipeline { agent any stages { stage('拉取Git代码') { steps { checkout([$class: 'GitSCM', branches: [[name: '${tag}']], extensions: [], userRemoteConfigs: [[url: 'http://49.233.115.171:8929/root/test.git']]]) } } stage('构建代码') { steps { sh '/var/jenkins_home/maven/bin/mvn clean package -DskipTests' } } } ``` ##### 9.3.4 代码质量检测 通过脚本执行sonar-scanner命令即可 ```sh pipeline { agent any stages { stage('拉取Git代码') { steps { checkout([$class: 'GitSCM', branches: [[name: '${tag}']], extensions: [], userRemoteConfigs: [[url: 'http://49.233.115.171:8929/root/test.git']]]) } } stage('构建代码') { steps { sh '/var/jenkins_home/maven/bin/mvn clean package -DskipTests' } } stage('检测代码质量') { steps { sh '/var/jenkins_home/sonar-scanner/bin/sonar-scanner -Dsonar.sources=./ -Dsonar.projectname=${JOB_NAME} -Dsonar.projectKey=${JOB_NAME} -Dsonar.java.binaries=target/ -Dsonar.login=31388be45653876c1f51ec02f0d478e2d9d0e1fa' } } } } ``` ##### 9.3.5 制作自定义镜像并发布 - 生成自定义镜像脚本 ```sh pipeline { agent any environment{ harborHost = '192.168.11.11:80' harborRepo = 'repository' harborUser = 'DevOps' harborPasswd = 'P@ssw0rd' } // 存放所有任务的合集 stages { stage('拉取Git代码') { steps { checkout([$class: 'GitSCM', branches: [[name: '${tag}']], extensions: [], userRemoteConfigs: [[url: 'http://49.233.115.171:8929/root/test.git']]]) } } stage('构建代码') { steps { sh '/var/jenkins_home/maven/bin/mvn clean package -DskipTests' } } stage('检测代码质量') { steps { sh '/var/jenkins_home/sonar-scanner/bin/sonar-scanner -Dsonar.sources=./ -Dsonar.projectname=${JOB_NAME} -Dsonar.projectKey=${JOB_NAME} -Dsonar.java.binaries=target/ -Dsonar.login=31388be45653876c1f51ec02f0d478e2d9d0e1fa' } } stage('制作自定义镜像并发布Harbor') { steps { sh '''cp ./target/*.jar ./docker/ cd ./docker docker build -t ${JOB_NAME}:${tag} ./''' sh '''docker login -u ${harborUser} -p ${harborPasswd} ${harborHost} docker tag ${JOB_NAME}:${tag} ${harborHost}/${harborRepo}/${JOB_NAME}:${tag} docker push ${harborHost}/${harborRepo}/${JOB_NAME}:${tag}''' } } } } ``` - 生成Publish Over SSH脚本 ```sh pipeline { agent any environment{ harborHost = '192.168.11.11:80' harborRepo = 'repository' harborUser = 'DevOps' harborPasswd = 'P@ssw0rd' } // 存放所有任务的合集 stages { stage('拉取Git代码') { steps { checkout([$class: 'GitSCM', branches: [[name: '${tag}']], extensions: [], userRemoteConfigs: [[url: 'http://49.233.115.171:8929/root/test.git']]]) } } stage('构建代码') { steps { sh '/var/jenkins_home/maven/bin/mvn clean package -DskipTests' } }docker stage('检测代码质量') { steps { sh '/var/jenkins_home/sonar-scanner/bin/sonar-scanner -Dsonar.sources=./ -Dsonar.projectname=${JOB_NAME} -Dsonar.projectKey=${JOB_NAME} -Dsonar.java.binaries=target/ -Dsonar.login=7d66af4b39cfe4f52ac0a915d4c9d5c513207098' } } stage('制作自定义镜像并发布Harbor') { steps { sh '''cp ./target/*.jar ./docker/ cd ./docker docker build -t ${JOB_NAME}:${tag} ./''' sh '''docker login -u ${harborUser} -p ${harborPasswd} ${harborHost} docker tag ${JOB_NAME}:${tag} ${harborHost}/${harborRepo}/${JOB_NAME}:${tag} docker push ${harborHost}/${harborRepo}/${JOB_NAME}:${tag}''' } } stage('目标服务器拉取镜像并运行') { steps { sshPublisher(publishers: [sshPublisherDesc(configName: 'testEnvironment', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: "/usr/bin/deploy.sh $harborHost $harborRepo $JOB_NAME $tag $port ", execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)]) } } } } ``` [Ps：由于采用变量，记得使用双引号]( ) #### 9.4 Jenkins流水线整合钉钉 在程序部署成功后，可以通过钉钉的机器人及时向群众发送部署的最终结果通知 - 安装插件 | 安装插件 | | :--------------------------------------: | | ![image-20211209151549412](../../img/kubernetes/kubernetes_devops/image-20211209151549412.png) | - 钉钉内部创建群组并构建机器人 | 钉钉内部创建群组并构建机器人 | | :--------------------------------------: | | ![image-20211209152217433](../../img/kubernetes/kubernetes_devops/image-20211209152217433.png) | | ![image-20211209152252050](../../img/kubernetes/kubernetes_devops/image-20211209152252050.png) | | ![image-20211209152403312](../../img/kubernetes/kubernetes_devops/image-20211209152403312.png) | 最终或获取到Webhook信息 ``` https://oapi.dingtalk.com/robot/send?access_token=kej4ehkj34gjhg34jh5bh5jb34hj53b4 ``` - 系统配置添加钉钉通知 | 配置钉钉通知 | | :--------------------------------------: | | ![image-20211209162923440](../../img/kubernetes/kubernetes_devops/image-20211209162923440.png) | - 任务中追加流水线配置 ```sh pipeline { agent any environment { sonarLogin = '2bab7bf7d5af25e2c2ca2f178af2c3c55c64d5d8' harborUser = 'admin' harborPassword = 'Harbor12345' harborHost = '192.168.11.12:8888' harborRepo = 'repository' } stages { stage('拉取Git代码'){ steps { checkout([$class: 'GitSCM', branches: [[name: '$tag']], extensions: [], userRemoteConfigs: [[url: 'http://49.233.115.171:8929/root/lsx.git']]]) } } stage('Maven构建代码'){ steps { sh '/var/jenkins_home/maven/bin/mvn clean package -DskipTests' } } stage('SonarQube检测代码'){ steps { sh '/var/jenkins_home/sonar-scanner/bin/sonar-scanner -Dsonar.sources=./ -Dsonar.projectname=${JOB_NAME} -Dsonar.projectKey=${JOB_NAME} -Dsonar.java.binaries=target/ -Dsonar.login=${sonarLogin}' } } stage('制作自定义镜像'){ steps { sh '''cd docker mv ../target/*.jar ./ docker build -t ${JOB_NAME}:$tag . ''' } } stage('推送自定义镜像'){ steps { sh '''docker login -u ${harborUser} -p ${harborPassword} ${harborHost} docker tag ${JOB_NAME}:$tag ${harborHost}/${harborRepo}/${JOB_NAME}:$tag docker push ${harborHost}/${harborRepo}/${JOB_NAME}:$tag''' } } stage('通知目标服务器'){ steps { sshPublisher(publishers: [sshPublisherDesc(configName: 'centos-docker', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: "/usr/bin/deploy.sh $harborHost $harborRepo $JOB_NAME $tag $port", execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)]) } } } post { success { dingtalk ( robot: 'Jenkins-DingDing', type:'MARKDOWN', title: "success: ${JOB_NAME}", text: ["- 成功构建:${JOB_NAME}项目!\n- 版本:${tag}\n- 持续时间:${currentBuild.durationString}\n- 任务:#${JOB_NAME}"] ) } failure { dingtalk ( robot: 'Jenkins-DingDing', type:'MARKDOWN', title: "fail: ${JOB_NAME}", text: ["- 失败构建:${JOB_NAME}项目!\n- 版本:${tag}\n- 持续时间:${currentBuild.durationString}\n- 任务:#${JOB_NAME}"] ) } } } ``` - 查看效果 | 钉钉通知效果 | | :--------------------------------------: | | ![image-20211209163021396](../../img/kubernetes/kubernetes_devops/image-20211209163021396.png) | ###十、Kubernetes编排工具 #### 10.1 Kubernetes介绍 Kubernetes是一个开源的，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效（powerful），Kubernetes提供了应用部署，规划，更新，维护的一种机制。 Kubernetes一个核心的特点就是能够自主的管理容器来保证云平台中的容器按照用户的期望状态运行着，管理员可以加载一个微型服务，让规划器来找到合适的位置，同时，Kubernetes也系统提升工具以及人性化方面，让用户能够方便的部署自己的应用。 Kubernetes主要能帮助我们完成： - 服务发现和负载均衡 Kubernetes 可以使用 DNS 名称或自己的 IP 地址公开容器，如果进入容器的流量很大， Kubernetes 可以负载均衡并分配网络流量，从而使部署稳定。 - 存储编排 Kubernetes 允许你自动挂载你选择的存储系统，比如本地存储，类似Docker的数据卷。 - 自动部署和回滚 你可以使用 Kubernetes 描述已部署容器的所需状态，它可以以受控的速率将实际状态 更改为期望状态。Kubernetes 会自动帮你根据情况部署创建新容器，并删除现有容器给新容器提供资源。 - 自动完成装箱计算 Kubernetes 允许你设置每个容器的资源，比如CPU和内存。 - 自我修复 Kubernetes 重新启动失败的容器、替换容器、杀死不响应用户定义的容器，并运行状况检查的容器。 - 秘钥与配置管理 Kubernetes 允许你存储和管理敏感信息，例如密码、OAuth 令牌和 ssh 密钥。你可以在不重建容器镜像的情况下部署和更新密钥和应用程序配置，也无需在堆栈配置中暴露密钥。 #### 10.2 Kubernetes架构 Kubernetes 搭建需要至少两个节点，一个Master负责管理，一个Slave搭建在工作服务器上负责分配。 | kubernetes架构 | | :--------------------------------------: | | ![image-20211210114507638](../../img/kubernetes/kubernetes_devops/image-20211210114507638.png) | 从图中可以看到各个组件的基本功能： - API Server：作为K8s通讯的核心组件，K8s内部交互以及接收发送指令的组件。 - controller-manager：作为K8s的核心组件，主要做资源调度，根据集群情况分配资源 - etcd：一个key-value的数据库，存储存储集群的状态信息 - scheduler：负责调度每个工作节点 - cloud-controller-manager：负责调度其他云服务产品 - kubelet：管理Pods上面的容器。 - kube-proxy：负责处理其他Slave或客户端的请求。 - Pod：可以理解为就是运行的容器 #### 10.3 Kubernetes安装 这里会采用https://kuboard.cn/提供的方式安装K8s，安装单Master节点 - 要求使用Centos7.8版本：https://vault.centos.org/7.8.2003/isos/x86_64/CentOS-7-x86_64-Minimal-2003.iso - 至少2台 **2核4G** 的服务器 安装流程 | 安装流程 | | :--------------------------------------: | | ![image-20211210190653687](../../img/kubernetes/kubernetes_devops/image-20211210190653687.png) | 准备好服务器后开始安装 - 重新设置hostname，不允许为localhost ```sh # 修改 hostname，名字不允许使用下划线、小数点、大写字母，不能叫master hostnamectl set-hostname your-new-host-name # 查看修改结果 hostnamectl status # 设置 hostname 解析 echo "127.0.0.1 $(hostname)" >> /etc/hosts ``` - 要求2台服务之间可以相互通讯 - 安装软件 ```sh # 阿里云 docker hub 镜像 export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com curl -sSL https://kuboard.cn/install-script/v1.19.x/install_kubelet.sh | sh -s 1.19.5 ``` 首先初始化Master节点 > 关于初始化时用到的环境变量 > > - **APISERVER_NAME** 不能是 master 的 hostname > - **APISERVER_NAME** 必须全为小写字母、数字、小数点，不能包含减号 > - **POD_SUBNET** 所使用的网段不能与 ***master节点/worker节点*** 所在的网段重叠。该字段的取值为一个 [CIDR](https://kuboard.cn/glossary/cidr.html) 值，如果您对 CIDR 这个概念还不熟悉，请仍然执行 export POD_SUBNET=10.100.0.0/16 命令，不做修改 - 设置ip，域名，网段并执行初始化操作 ```sh # 只在 master 节点执行 # 替换 x.x.x.x 为 master 节点实际 IP（请使用内网 IP） # export 命令只在当前 shell 会话中有效，开启新的 shell 窗口后，如果要继续安装过程，请重新执行此处的 export 命令 export MASTER_IP=192.168.11.32 # 替换 apiserver.demo 为 您想要的 dnsName export APISERVER_NAME=apiserver.demo # Kubernetes 容器组所在的网段，该网段安装完成后，由 kubernetes 创建，事先并不存在于您的物理网络中 export POD_SUBNET=10.100.0.1/16 echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts curl -sSL https://kuboard.cn/install-script/v1.19.x/init_master.sh | sh -s 1.19.5 ``` - 检查Master启动状态 ```sh # 只在 master 节点执行 # 执行如下命令，等待 3-10 分钟，直到所有的容器组处于 Running 状态 watch kubectl get pod -n kube-system -o wide # 查看 master 节点初始化结果 kubectl get nodes -o wide ``` `Ps：如果出现NotReady的情况执行（最新版本的BUG，1.19一般没有）` ```sh docker pull quay.io/coreos/flannel:v0.10.0-amd64 mkdir -p /etc/cni/net.d/ cat < /etc/cni/net.d/10-flannel.conf {"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}} EOF mkdir /usr/share/oci-umount/oci-umount.d -p mkdir /run/flannel/ cat < /run/flannel/subnet.env FLANNEL_NETWORK=172.100.0.0/16 FLANNEL_SUBNET=172.100.1.0/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true EOF kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml ``` 安装网络服务插件 ```sh export POD_SUBNET=10.100.0.0/16 kubectl apply -f https://kuboard.cn/install-script/v1.22.x/calico-operator.yaml wget https://kuboard.cn/install-script/v1.22.x/calico-custom-resources.yaml sed -i "s#192.168.0.0/16#${POD_SUBNET}#" calico-custom-resources.yaml kubectl apply -f calico-custom-resources.yaml ``` 初始化worker节点 - 获取Join命令参数，在Master节点执行 ```sh # 只在 master 节点执行 kubeadm token create --print-join-command ``` | 获取命令 | | :--------------------------------------: | | ![image-20211213183025291](../../img/kubernetes/kubernetes_devops/image-20211213183025291.png) | - 在worker节点初始化 ```sh # 只在 worker 节点执行 # 替换 x.x.x.x 为 master 节点的内网 IP export MASTER_IP=192.168.11.32 # 替换 apiserver.demo 为初始化 master 节点时所使用的 APISERVER_NAME export APISERVER_NAME=apiserver.demo echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts # 替换为 master 节点上 kubeadm token create 命令的输出 kubeadm join apiserver.demo:6443 --token vwfilu.3nhndohc5gn1jv9k --discovery-token-ca-cert-hash sha256:22ff15cabfe87ab48a7db39b3bbf986fee92ec92eb8efc7fe9b0abe2175ff0c2 ``` 检查最终运行效果 - 在 master 节点上执行 ```sh # 只在 master 节点执行 kubectl get nodes -o wide ``` `Ps：如果出现NotReady的情况执行（最新版本的BUG，1.19一般没有）` ```sh docker pull quay.io/coreos/flannel:v0.10.0-amd64 mkdir -p /etc/cni/net.d/ cat < /etc/cni/net.d/10-flannel.conf {"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}} EOF mkdir /usr/share/oci-umount/oci-umount.d -p mkdir /run/flannel/ cat < /run/flannel/subnet.env FLANNEL_NETWORK=172.100.0.0/16 FLANNEL_SUBNET=172.100.1.0/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true EOF kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml ``` - 输出结果如下所示： ```sh [root@k8smaster ~]# kubectl get nodes ``` | 搭建成功效果 | | :--------------------------------------: | | ![image-20211210184851810](../../img/kubernetes/kubernetes_devops/image-20211210184851810.png) | 安装Kuboard管理K8s集群 - 安装Kuboard ```yml kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3.yaml # 您也可以使用下面的指令，唯一的区别是，该指令使用华为云的镜像仓库替代 docker hub 分发 Kuboard 所需要的镜像 # kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3-swr.yaml ``` - 查看启动情况 ``` watch kubectl get pods -n kuboard ``` | 查看效果 | | :--------------------------------------: | | ![image-20211213184701784](../../img/kubernetes/kubernetes_devops/image-20211213184701784.png) | - 在浏览器中打开链接 http://your-node-ip-address:30080 | 首页 | | :--------------------------------------: | | ![image-20211213184742709](../../img/kubernetes/kubernetes_devops/image-20211213184742709.png) | - 输入初始用户名和密码，并登录 - 用户名： `admin` - 密码： `Kuboard123` | 首页效果 | | :--------------------------------------: | | ![image-20211213184840120](../../img/kubernetes/kubernetes_devops/image-20211213184840120.png) | #### 10.4 Kubernetes操作 首先我们要了解Kubernetes在运行我们的资源时，关联到了哪些内容 * 资源的构建方式： * 采用kubectl的命令方式 * yaml文件方式 ##### 10.4.1 Namespace * 命名空间：主要是为了对Kubernetes中运行的资源进行过隔离， 但是网络是互通的，类似Docker的容器，可以将多个资源配置到一个NameSpace中。而NameSpace可以对不同环境进行资源隔离，默认情况下Kubernetes提供了default命名空间，在构建资源时，如果不指定资源，默认采用default资源。 命令方式： ```sh # 查看现有的全部命名空间 kubectl get ns # 构建命名空间 kubectl create ns 命名空间名称 # 删除现有命名空间， 并且会删除空间下的全部资源 kubectl delete ns 命名空间名称 ``` yaml文件方式：（构建资源时，设置命名空间） ```yaml apiVersion: v1 kind: Namespace metadata: name: test ``` ##### 10.4.2 Pod * Pod：Kubernetes运行的一组容器，Pod是Kubernetes的最小单位，但是对于Docker而然，Pod中会运行多个Docker容器 * 命令方式： ```bash # 查看所有运行的pod kubectl get pods -A # 查看指定Namespace下的Pod kubectl get pod [-n 命名空间] #（默认default） # 创建Pod kubectl run pod名称 --image=镜像名称 # 查看Pod详细信息 kubectl describe pod pod名称 # 删除pod kubectl delete pod pod名称 [-n 命名空间] #（默认default） # 查看pod输出的日志 kubectl logs -f pod名称 # 进去pod容器内部 kubectl exec -it pod名称 -- bash # 查看kubernetes给Pod分配的ip信息，并且通过ip和容器的端口，可以直接访问 kubectl get pod -owide ``` * yaml方式（推荐） ```yaml apiVersion: v1 kind: Pod metadata: labels: run: 运行的pod名称 name: pod名称 namespace: 命名空间 spec: containers: - image: 镜像名称 name: 容器名称 # 启动Pod：kubectl apply -f yaml文件名称 # 删除Pod：kubectl delete -f yaml文件名称 ``` * Pod中运行多个容器 ```yml apiVersion: v1 kind: Pod metadata: labels: run: 运行的pod名称 name: pod名称 namespace: 命名空间 spec: containers: - image: 镜像名称 name: 容器名称 - image: 镜像名称 name: 容器名称 ………… ``` 启动后可以查看到 | Kuboard效果 | | :--------------------------------------: | | ![image-20220104203155749](../../img/kubernetes/kubernetes_devops/image-20220104203155749.png) | ##### 10.4.3 Deployment 部署时，可以通过Deployment管理和编排Pod Deployment部署实现 - 命令方式 ```sh # 基于Deployment启动容器 kubectl create deployment deployment名称 --image=镜像名称 # 用deployment启动的容器会在被删除后自动再次创建，达到故障漂移的效果 # 需要使用deploy的方式删除deploy # 查看现在的deployment kubectl get deployment # 删除deployment kubectl delete deployment deployment名称 # 基于Deployment启动容器并设置Pod集群数 kubectl create deployment deployment名称 --image=镜像名称 --replicas 集群个数 ``` - [配置文件方式](https://kubernetes.io/zh/docs/concepts/workloads/controllers/deployment/) ```yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 ``` 正常使用kubectl运行yaml即可 弹性伸缩功能 ```sh # 基于scale实现弹性伸缩 kubectl scale deploy/Deployment名称 --replicas 集群个数 # 或者修改yaml文件 kubectl edit deploy Deployment名称 ``` | 图形化页面修改 | | :--------------------------------------: | | ![image-20220104210823057](../../img/kubernetes/kubernetes_devops/image-20220104210823057.png) | 灰度发布 Deploy可以在部署新版本数据时，成功启动一个pod，才会下线一个老版本的Pod ```shell kubectl set image deployment/Deployment名称 容器名=镜像:版本 ``` ##### 10.4.4 Service 可以将多个Pod对外暴露一个Service，让客户端可以通过Service访问到这一组Pod，并且可以实现负载均衡 ClusterIP方式： ClusterIP是集群内部Pod之间的访问方式 - 命令实现效果 ```sh # 通过生成service映射一个Deployment下的所有pod中的某一个端口的容器 kubectl expose deployment Deployment名称 --port=Service端口号 --target-port=Pod内容器端口 ``` 之后通过`kubectl get service`查看Service提供的ip，即可访问 | kubectl get service | | :--------------------------------------: | | ![image-20220104214659229](../../img/kubernetes/kubernetes_devops/image-20220104214659229.png) | 也可以通过`Deployment名称.namespace名称.svc`作为域名访问 | 在服务容器内执行 | | :--------------------------------------: | | ![image-20220104215030265](../../img/kubernetes/kubernetes_devops/image-20220104215030265.png) | NodePort方式 ClusterIP的方式只能在Pod内部实现访问，但是一般需要对外暴露网关，所以需要NodePort的方式Pod外暴露访问 - 命令实现方式 ```sh # 通过生成service映射一个Deployment下的所有pod中的某一个端口的容器 kubectl expose deployment Deployment名称 --port=Service端口号 --target-port=Pod内容器端口 --type=NodePort ``` | 查看Service效果 | | :--------------------------------------: | | ![image-20220104222750733](../../img/kubernetes/kubernetes_devops/image-20220104222750733.png) | | ![image-20220104222819455](../../img/kubernetes/kubernetes_devops/image-20220104222819455.png) | Service也可以通过yaml文件实现 ```yaml apiVersion: v1 kind: Service metadata: labels app: nginx name: nginx spec: selector: app: nginx ports: - port: 8888 protocol: TCP targetPort: 80 ``` 通过apply启动就也可以创建Service 测试效果-Deployment部署，通过Service暴露 ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx-deployment spec: replicas: 2 selector: matchLabels: app: nginx-deployment template: metadata: labels: app: nginx-deployment spec: containers: - name: nginx-deployment image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: labels: app: nginx-service name: nginx-service spec: selector: app: nginx-deployment ports: - port: 8888 protocol: TCP targetPort: 80 type: NodePort ``` 可以查看到暴露的信息 | Service信息 | | :--------------------------------------: | | ![image-20220105205334996](../../img/kubernetes/kubernetes_devops/image-20220105205334996.png) | ##### 10.4.5 Ingress Kubernetes推荐将Ingress作为所有Service的入口，提供统一的入口，避免多个服务之间需要记录大量的IP或者域名，毕竟IP可能改变，服务太多域名记录不方便。 Ingress底层其实就是一个Nginx， 可以在Kuboard上直接点击安装 | Kuboard安装 | | :--------------------------------------: | | ![image-20220105153343642](../../img/kubernetes/kubernetes_devops/image-20220105153343642.png) | | **![image-20220105153416367](../../img/kubernetes/kubernetes_devops/image-20220105153416367.png)** | 因为副本数默认为1，但是k8s整体集群就2个节点，所以显示下面即为安装成功 | 安装成功 | | :--------------------------------------: | | ![image-20220105153502619](../../img/kubernetes/kubernetes_devops/image-20220105153502619.png) | 可以将Ingress接收到的请求转发到不同的Service中。 推荐使用yaml文件方式 ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-ingress spec: ingressClassName: ingress rules: - host: nginx.mashibing.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-service port: number: 8888 ``` | 启动时问题 | | :--------------------------------------: | | ![image-20220105203819715](../../img/kubernetes/kubernetes_devops/image-20220105203819715.png) | Kuboard安装的Ingress有admission的校验配置，需要先删除配置再启动 找到指定的ingress的校验信息，删除即可 | 删除信息 | | :--------------------------------------: | | ![image-20220105204434044](../../img/kubernetes/kubernetes_devops/image-20220105204434044.png) | ```sh # 查看校验webhook的配置 kubectl get -A ValidatingWebhookConfiguration # 删除指定的校验 kubectl delete ValidatingWebhookConfiguration ingress-nginx-admission-my-ingress-controller ``` 配置本地hosts文件 | 配置hosts | | :--------------------------------------: | | ![image-20220105204921272](../../img/kubernetes/kubernetes_devops/image-20220105204921272.png) | 记下来既可以访问在Service中暴露的Nginx信息 | 服通过Ingress访问 | | :--------------------------------------: | | ![image-20220105205407393](../../img/kubernetes/kubernetes_devops/image-20220105205407393.png) | #### 10.5 Jenkins集成Kubernetes ##### 10.5.1 准备部署的yml文件 ```yml apiVersion: apps/v1 kind: Deployment metadata: namespace: test name: pipeline labels: app: pipeline spec: replicas: 2 selector: matchLabels: app: pipeline template: metadata: labels: app: pipeline spec: containers: - name: pipeline image: 192.168.11.102:80/repo/pipeline:v4.0.0 imagePullPolicy: Always ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: namespace: test labels: app: pipeline name: pipeline spec: selector: app: pipeline ports: - port: 8081 targetPort: 8080 type: NodePort --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: namespace: test name: pipeline spec: ingressClassName: ingress rules: - host: mashibing.pipeline.com http: paths: - path: / pathType: Prefix backend: service: name: pipeline port: number: 8081 ``` ##### 10.5.2 Harbor私服配置 在尝试用kubernetes的yml文件启动pipeline服务时，会出现Kubernetes无法拉取镜像的问题，这里需要在kubernetes所在的Linux中配置Harbor服务信息，并且保证Kubernetes可以拉取Harbor上的镜像 - 设置Master和Worker的私服地址信息 | 设置Harbor私服地址 | | :------------------------------------------: | | ![1642498962716](../../img/kubernetes/kubernetes_devops/1642498962716.png) | - 在Kuboard上设置私服密文信息 | 设置密文并测试 | | :------------------------------------------: | | ![1642498994935](../../img/kubernetes/kubernetes_devops/1642498994935.png) | 按照复制指令的位置测试认证，效果如下 | 测试效果 | | :------------------------------------------: | | ![1642499172789](../../img/kubernetes/kubernetes_devops/1642499172789.png) | ##### 10.5.3 测试使用效果 执行kubectl命令，基于yml启动服务，并且基于部署后服务的提示信息以及Ingress的设置，直接访问 | | | -------------------------------------------- | | ![1642499368121](../../img/kubernetes/kubernetes_devops/1642499368121.png) | | ![1642499788199](../../img/kubernetes/kubernetes_devops/1642499788199.png) | ##### 10.5.3 Jenkins远程调用 - 将pipeline.yml配置到Gitlab中 | 配置yml文件 | | :------------------------------------------: | | ![1642499885324](../../img/kubernetes/kubernetes_devops/1642499885324.png) | - 配置Jenkins的目标服务器，可以将yml文件传输到K8s的Master上 | 设置目标服务器 | | :------------------------------------------: | | ![1642499992148](../../img/kubernetes/kubernetes_devops/1642499992148.png) | - 修改Jenkinsfile，重新设置流水线任务脚本，并测试效果 | 传递yml文件脚本 | | :------------------------------------------: | | ![1642500061153](../../img/kubernetes/kubernetes_devops/1642500061153.png) | | ![1642500102996](../../img/kubernetes/kubernetes_devops/1642500102996.png) | - 设置Jenkins无密码登录k8s-master 将Jenkins中公钥信息复制到k8s-master的~/.ssh/authorized_keysz中，保证远程连接无密码 | 远程执行命令无需密码 | | :------------------------------------------: | | ![1642500239406](../../img/kubernetes/kubernetes_devops/1642500239406.png) | - 设置执行kubectl的脚本到Jenkinsfile | 设置Jenkinsfile | | :------------------------------------------: | | ![1642500378788](../../img/kubernetes/kubernetes_devops/1642500378788.png) | - 执行查看效果 | 执行流水线 | | :------------------------------------------: | | ![1642500413802](../../img/kubernetes/kubernetes_devops/1642500413802.png) | 可以查看到yml文件是由变化的， 这样k8s就会重新加载 - 查看效果 | 效果 | | :------------------------------------------: | | ![1642500474036](../../img/kubernetes/kubernetes_devops/1642500474036.png) | [Ps：这种方式更适应与CD操作，将项目将基于某个版本部署到指定的目标服务器]() #### 10.6 基于GitLab的WebHooks 这里要实现自动化的一个CI操作，也就是开发人员Push代码到Git仓库后，Jenkins会自动的构建项目，将最新的提交点代码构建并进行打包部署，这里区别去上述的CD操作，CD操作需要基于某个版本进行部署，而这里每次都是将最新的提交点集成到主干上并测试。 ##### 10.6.1 WebHooks通知 开启Jenkins的自动构建 | 构建触发器 | | :------------------------------------------: | | ![1642500817131](../../img/kubernetes/kubernetes_devops/1642500817131.png) | 设置Gitlab的Webhooks | 设置Gitlab的Webhooks | | :------------------------------------------: | | ![1642500933316](../../img/kubernetes/kubernetes_devops/1642500933316.png) | 需要关闭Jenkins的Gitlab认证 | 关闭Jenkins的Gitlab认证 | | :------------------------------------------: | | ![1642501016474](../../img/kubernetes/kubernetes_devops/1642501016474.png) | 再次测试Gitlab | 再次测试 | | :------------------------------------------: | | ![1642501065243](../../img/kubernetes/kubernetes_devops/1642501065243.png) | ##### 10.6.2 修改配置 修改Jenkinsfile实现基于最新提交点实现持续集成效果，将之前引用${tag}的全部去掉 ```json // 所有的脚本命令都放在pipeline中 pipeline{ // 指定任务再哪个集群节点中执行 agent any // 声明全局变量，方便后面使用 environment { harborUser = 'admin' harborPasswd = 'Harbor12345' harborAddress = '192.168.11.102:80' harborRepo = 'repo' } stages { stage('拉取git仓库代码') { steps { checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[url: 'http://192.168.11.101:8929/root/mytest.git']]]) } } stage('通过maven构建项目') { steps { sh '/var/jenkins_home/maven/bin/mvn clean package -DskipTests' } } stage('通过SonarQube做代码质量检测') { steps { sh '/var/jenkins_home/sonar-scanner/bin/sonar-scanner -Dsonar.source=./ -Dsonar.projectname=${JOB_NAME} -Dsonar.projectKey=${JOB_NAME} -Dsonar.java.binaries=./target/ -Dsonar.login=40306ae8ea69a4792df2ceb4d9d25fe8a6ab1701' } } stage('通过Docker制作自定义镜像') { steps { sh '''mv ./target/*.jar ./docker/ docker build -t ${JOB_NAME}:latest ./docker/''' } } stage('将自定义镜像推送到Harbor') { steps { sh '''docker login -u ${harborUser} -p ${harborPasswd} ${harborAddress} docker tag ${JOB_NAME}:latest ${harborAddress}/${harborRepo}/${JOB_NAME}:latest docker push ${harborAddress}/${harborRepo}/${JOB_NAME}:latest ''' } } stage('将yml文件传到k8s-master上') { steps { sshPublisher(publishers: [sshPublisherDesc(configName: 'k8s', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: '', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: 'pipeline.yml')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)]) } } stage('远程执行k8s-master的kubectl命令') { steps { sh '''ssh root@192.168.11.201 kubectl apply -f /usr/local/k8s/pipeline.yml ssh root@192.168.11.201 kubectl rollout restart deployment pipeline -n test''' } } } post { success { dingtalk( robot: 'Jenkins-DingDing', type: 'MARKDOWN', title: "success: ${JOB_NAME}", text: ["- 成功构建：${JOB_NAME}! \n- 版本：latest \n- 持续时间：${currentBuild.durationString}" ] ) } failure { dingtalk( robot: 'Jenkins-DingDing', type: 'MARKDOWN', title: "success: ${JOB_NAME}", text: ["- 构建失败：${JOB_NAME}! \n- 版本：latest \n- 持续时间：${currentBuild.durationString}" ] ) } } } ``` 修改pipeline.yml，更改镜像版本 ```yml apiVersion: apps/v1 kind: Deployment metadata: namespace: test name: pipeline labels: app: pipeline spec: replicas: 2 selector: matchLabels: app: pipeline template: metadata: labels: app: pipeline spec: containers: - name: pipeline image: 192.168.11.102:80/repo/pipeline:latest # 这里 imagePullPolicy: Always ports: - containerPort: 8080 # 省略其他内容………… ``` ##### 10.6.3 滚动更新 因为pipeline没有改变时，每次不会重新加载，这样会导致Pod中的容器不会动态更新，这里需要使用kubectl的rollout restart命令滚动更新 | 设置Jenkinsfle | | :------------------------------------------: | | ![1642501521065](../../img/kubernetes/kubernetes_devops/1642501521065.png) | | ![1642501549176](../../img/kubernetes/kubernetes_devops/1642501549176.png) | ================================================ FILE: docs/cloud/kubernetes/kubernetes_flannel.md ================================================ # 1 集群环境实践 ## 1.1 单主集群 ### 1.1.1 基础环境部署 学习目标 这一节，我们从 集群规划、主机认证、小结 三个方面来学习。 **集群规划** 简介 ```powershell 在这里，我们以单主分布式的主机节点效果来演示kubernetes的最新版本的集群环境部署。 ``` ![image-20220715092558666](../../img/kubernetes/kubernetes_flannel/image-20220715092558666.png) 节点集群组件规划 ![image-20220715093007357](../../img/kubernetes/kubernetes_flannel/image-20220715093007357.png) ```powershell master节点 kubeadm(集群环境)、kubectl(集群管理)、kubelet(节点状态) kube-apiserver、kube-controller-manager、kube-scheduler、etcd containerd(docker方式部署)、flannel(插件部署) node节点 kubeadm(集群环境)、kubelet(节点状态) kube-proxy、containerd(docker方式部署)、flannel(插件部署) ``` 主机名规划 | 序号 | 主机ip | 主机名规划 | | ---- | --------- | -------------------------------------------------------- | | 1 | 10.0.0.12 | kubernetes-master.superopsmsb.com kubernetes-master | | 2 | 10.0.0.15 | kubernetes-node1.superopsmsb.com kubernetes-node1 | | 3 | 10.0.0.16 | kubernetes-node2.superopsmsb.com kubernetes-node2 | | 4 | 10.0.0.17 | kubernetes-node3.superopsmsb.com kubernetes-node3 | | 5 | 10.0.0.20 | kubernetes-register.superopsmsb.com kubernetes-register | ```powershell 修改master节点主机的hosts文件 [root@localhost ~]# cat /etc/hosts 10.0.0.12 kubernetes-master.superopsmsb.com kubernetes-master 10.0.0.15 kubernetes-node1.superopsmsb.com kubernetes-node1 10.0.0.16 kubernetes-node2.superopsmsb.com kubernetes-node2 10.0.0.17 kubernetes-node3.superopsmsb.com kubernetes-node3 10.0.0.20 kubernetes-register.superopsmsb.com kubernetes-register ``` **主机认证** 简介 ```powershell 因为整个集群节点中的很多文件配置都是一样的，所以我们需要配置跨主机免密码认证方式来定制集群的认证通信机制，这样后续在批量操作命令的时候，就非常轻松了。 ``` ```powershell 为了让ssh通信速度更快一些，我们需要对ssh的配置文件进行一些基本的改造 [root@localhost ~]# egrep 'APIA|DNS' /etc/ssh/sshd_config GSSAPIAuthentication no UseDNS no [root@localhost ~]# systemctl restart sshd 注意： 这部分的操作，应该在所有集群主机环境初始化的时候进行设定 ``` 脚本内容 ```powershell [root@localhost ~]# cat /data/scripts/01_remote_host_auth.sh #!/bin/bash # 功能: 批量设定远程主机免密码认证 # 版本: v0.2 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 user_dir='/root' host_file='/etc/hosts' login_user='root' login_pass='123456' target_type=(部署 免密 同步 主机名 退出) # 菜单 menu(){ echo -e "\e[31m批量设定远程主机免密码认证管理界面\e[0m" echo "=====================================================" echo -e "\e[32m 1: 部署环境 2: 免密认证 3: 同步hosts \e[0m" echo -e "\e[32m 4: 设定主机名 5：退出操作 \e[0m" echo "=====================================================" } # expect环境 expect_install(){ if [ -f /usr/bin/expect ] then echo -e "\e[33mexpect环境已经部署完毕\e[0m" else yum install expect -y >> /dev/null 2>&1 && echo -e "\e[33mexpect软件安装完毕\e[0m" || (echo -e "\e[33mexpect软件安装失败\e[0m" && exit) fi } # 秘钥文件生成环境 create_authkey(){ # 保证历史文件清空 [ -d ${user_dir}/.ssh ] && rm -rf ${user_dir}/.ssh/* # 构建秘钥文件对 /usr/bin/ssh-keygen -t rsa -P "" -f ${user_dir}/.ssh/id_rsa echo -e "\e[33m秘钥文件已经创建完毕\e[0m" } # expect自动匹配逻辑 expect_autoauth_func(){ # 接收外部参数 command="$@" expect -c " spawn ${command} expect { \"yes/no\" {send \"yes\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"} }" } # 跨主机传输文件认证 sshkey_auth_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do # /usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub root@10.0.0.12 cmd="/usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub" remote_host="${login_user}@${ip}" expect_autoauth_func ${cmd} ${remote_host} done } # 跨主机同步hosts文件 scp_hosts_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do remote_host="${login_user}@${ip}" scp ${host_file} ${remote_host}:${host_file} done } # 跨主机设定主机名规划 set_hostname_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do host_name=$(grep ${ip} ${host_file}|awk '{print $NF}') remote_host="${login_user}@${ip}" ssh ${remote_host} "hostnamectl set-hostname ${host_name}" done } # 帮助信息逻辑 Usage(){ echo "请输入有效的操作id" } # 逻辑入口 while true do menu read -p "请输入有效的操作id: " target_id if [ ${#target_type[@]} -ge ${target_id} ] then if [ ${target_type[${target_id}-1]} == "部署" ] then echo "开始部署环境操作..." expect_install create_authkey elif [ ${target_type[${target_id}-1]} == "免密" ] then read -p "请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行免密认证操作..." sshkey_auth_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "同步" ] then read -p "请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行同步hosts文件操作..." scp_hosts_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "主机名" ] then read -p "请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行设定主机名操作..." set_hostname_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "退出" ] then echo "开始退出管理界面..." exit fi else Usage fi done ``` ```powershell 为了更好的把环境部署成功，最好提前更新一下软件源信息 [root@localhost ~]# yum makecache ``` ```powershell 查看脚本执行效果 [root@localhost ~]# /bin/bash /data/scripts/01_remote_host_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 1 开始部署环境操作... expect环境已经部署完毕 Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:u/Tzk0d9sNtG6r9Kx+6xPaENNqT3Lw178XWXQhX1yMw root@kubernetes-master The key's randomart image is: +---[RSA 2048]----+ | .+| | + o.| | E .| | o. | | S + +.| | . . *=+B| | o o+B%O| | . o. +.O+X| | . .o.=+XB| +----[SHA256]-----+ 秘钥文件已经创建完毕 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): 12 开始执行免密认证操作... spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.12 ... Now try logging into the machine, with: "ssh 'root@10.0.0.12'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): {15..17} 开始执行免密认证操作... spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.15 ... Now try logging into the machine, with: "ssh 'root@10.0.0.15'" and check to make sure that only the key(s) you wanted were added. spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.16 ... Now try logging into the machine, with: "ssh 'root@10.0.0.16'" and check to make sure that only the key(s) you wanted were added. spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.17 ... Now try logging into the machine, with: "ssh 'root@10.0.0.17'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): 20 开始执行免密认证操作... spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.20 ... Now try logging into the machine, with: "ssh 'root@10.0.0.20'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 3 请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): 12 开始执行同步hosts文件操作... hosts 100% 470 1.2MB/s 00:00 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 3 请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): {15..17} 开始执行同步hosts文件操作... hosts 100% 470 226.5KB/s 00:00 hosts 100% 470 458.8KB/s 00:00 hosts 100% 470 533.1KB/s 00:00 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 3 请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): 20 开始执行同步hosts文件操作... hosts 100% 470 287.8KB/s 00:00 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 4 请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): 12 开始执行设定主机名操作... 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 4 请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): {15..17} 开始执行设定主机名操作... 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 4 请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): 20 开始执行设定主机名操作... 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 5 开始退出管理界面... ``` 检查效果 ```powershell [root@localhost ~]# exec /bin/bash [root@kubernetes-master ~]# for i in 12 {15..17} 20 > do > name=$(ssh root@10.0.0.$i "hostname") > echo 10.0.0.$i $name > done 10.0.0.12 kubernetes-master 10.0.0.15 kubernetes-node1 10.0.0.16 kubernetes-node2 10.0.0.17 kubernetes-node3 10.0.0.20 kubernetes-register ``` **小结** ``` ``` ### 1.1.2 集群基础环境 学习目标 这一节，我们从 内核调整、软件源配置、小结 三个方面来学习。 **内核调整** 简介 ```powershell 根据kubernetes官方资料的相关信息，我们需要对kubernetes集群是所有主机进行内核参数的调整。 ``` 禁用swap ```powershell 部署集群时，kubeadm默认会预先检查当前主机是否禁用了Swap设备，并在未禁用时强制终止部署过程。因此，在主机内存资源充裕的条件下，需要禁用所有的Swap设备，否则，就需要在后文的kubeadm init及kubeadm join命令执行时额外使用相关的选项忽略检查错误 关闭Swap设备，需要分两步完成。 首先是关闭当前已启用的所有Swap设备： swapoff -a 而后编辑/etc/fstab配置文件，注释用于挂载Swap设备的所有行。 方法一：手工编辑 vim /etc/fstab # UUID=0a55fdb5-a9d8-4215-80f7-f42f75644f69 none swap sw 0 0 方法二： sed -i 's/.*swap.*/#&/' /etc/fstab 替换后位置的&代表前面匹配的整行内容 注意： 只需要注释掉自动挂载SWAP分区项即可，防止机子重启后swap启用 内核(禁用swap)参数 cat >> /etc/sysctl.d/k8s.conf << EOF vm.swappiness=0 EOF sysctl -p /etc/sysctl.d/k8s.conf ``` 网络参数 ```powershell 配置iptables参数，使得流经网桥的流量也经过iptables/netfilter防火墙 cat >> /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF 配置生效 modprobe br_netfilter modprobe overlay sysctl -p /etc/sysctl.d/k8s.conf ``` ```powershell 脚本方式 [root@localhost ~]# cat /data/scripts/02_kubernetes_kernel_conf.sh #!/bin/bash # 功能: 批量设定kubernetes的内核参数调整 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 禁用swap swapoff -a sed -i 's/.*swap.*/#&/' /etc/fstab cat >> /etc/sysctl.d/k8s.conf << EOF vm.swappiness=0 EOF sysctl -p /etc/sysctl.d/k8s.conf # 打开网络转发 cat >> /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF # 加载相应的模块 modprobe br_netfilter modprobe overlay sysctl -p /etc/sysctl.d/k8s.conf ``` 脚本执行 ```powershell master主机执行效果 [root@kubernetes-master ~]# /bin/bash /data/scripts/02_kubernetes_kernel_conf.sh vm.swappiness = 0 vm.swappiness = 0 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 ``` ```powershell node主机执行效果 [root@kubernetes-master ~]# for i in {15..17};do ssh root@10.0.0.$i mkdir /data/scripts -p; scp /data/scripts/02_kubernetes_kernel_conf.sh root@10.0.0.$i:/data/scripts/02_kubernetes_kernel_conf.sh;ssh root@10.0.0.$i "/bin/bash /data/scripts/02_kubernetes_kernel_conf.sh";done 02_kubernetes_kernel_conf.sh 100% 537 160.6KB/s 00:00 vm.swappiness = 0 vm.swappiness = 0 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 02_kubernetes_kernel_conf.sh 100% 537 374.4KB/s 00:00 vm.swappiness = 0 vm.swappiness = 0 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 02_kubernetes_kernel_conf.sh 100% 537 154.5KB/s 00:00 vm.swappiness = 0 vm.swappiness = 0 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 ``` **软件源配置** 简介 ```powershell 由于我们需要在多台主机上初始化kubernetes主机环境，所以我们需要在多台主机上配置kubernetes的软件源，以最简便的方式部署kubernetes环境。 ``` 定制阿里云软件源 ```powershell 定制阿里云的关于kubernetes的软件源 [root@kubernetes-master ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF 更新软件源 [root@kubernetes-master ~]# yum makecache fast ``` 其他节点主机同步软件源 ```powershell [root@kubernetes-master ~]# for i in {15..17};do scp /etc/yum.repos.d/kubernetes.repo root@10.0.0.$i:/etc/yum.repos.d/kubernetes.repo;ssh root@10.0.0.$i "yum makecache fast";done ``` **小结** ``` ``` ### 1.1.3 容器环境 学习目标 这一节，我们从 Docker环境、环境配置、小结 三个方面来学习。 **Docker环境** 简介 ```powershell 由于kubernetes1.24版本才开始将默认支持的容器引擎转换为了Containerd了，所以这里我们还是以Docker软件作为后端容器的引擎，因为目前的CKA考试环境是以kubernetes1.23版本为基准的。 ``` 软件源配置 ```powershell 安装基础依赖软件 [root@kubernetes-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2 定制专属的软件源 [root@kubernetes-master ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ``` 安装软件 ```powershell 确定最新版本的docker [root@kubernetes-master ~]# yum list docker-ce --showduplicates | sort -r 安装最新版本的docker [root@kubernetes-master ~]# yum install -y docker-ce ``` 检查效果 ```powershell 启动docker服务 [root@kubernetes-master ~]# systemctl restart docker 检查效果 [root@kubernetes-master ~]# docker version Client: Docker Engine - Community Version: 20.10.17 API version: 1.41 Go version: go1.17.11 Git commit: 100c701 Built: Mon Jun 6 23:05:12 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.17 API version: 1.41 (minimum version 1.12) Go version: go1.17.11 Git commit: a89b842 Built: Mon Jun 6 23:03:33 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.6 GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc: Version: 1.1.2 GitCommit: v1.1.2-0-ga916309 docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` **环境配置** 需求 ```powershell 1 镜像仓库 默认安装的docker会从官方网站上获取docker镜像，有时候会因为网络因素无法获取，所以我们需要配置国内镜像仓库的加速器 2 kubernetes的改造 kubernetes的创建容器，需要借助于kubelet来管理Docker，而默认的Docker服务进程的管理方式不是kubelet的类型，所以需要改造Docker的服务启动类型为systemd方式。 注意： 默认情况下，Docker的服务修改有两种方式： 1 Docker服务 - 需要修改启动服务文件，需要重载服务文件，比较繁琐。 2 daemon.json文件 - 修改文件后，只需要重启docker服务即可，该文件默认情况下不存在。 ``` 定制docker配置文件 ```powershell 定制配置文件 [root@kubernetes-master ~]# cat >> /etc/docker/daemon.json <<-EOF { "registry-mirrors": [ "http://74f21445.m.daocloud.io", "https://registry.docker-cn.com", "http://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn" ], "insecure-registries": ["10.0.0.20:80"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF 重启docker服务 [root@kubernetes-master ~]# systemctl restart docker ``` 检查效果 ```powershell 查看配置效果 [root@kubernetes-master ~]# docker info Client: ... Server: ... Cgroup Driver: systemd ... Insecure Registries: 10.0.0.20:80 127.0.0.0/8 Registry Mirrors: http://74f21445.m.daocloud.io/ https://registry.docker-cn.com/ http://hub-mirror.c.163.com/ https://docker.mirrors.ustc.edu.cn/ Live Restore Enabled: false ``` docker环境定制脚本 ```powershell 查看脚本内容 [root@localhost ~]# cat /data/scripts/03_kubernetes_docker_install.sh #!/bin/bash # 功能: 安装部署Docker容器环境 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 # 软件源配置 softs_base(){ # 安装基础软件 yum install -y yum-utils device-mapper-persistent-data lvm2 # 定制软件仓库源 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 更新软件源 systemctl restart network yum makecache fast } # 软件安装 soft_install(){ # 安装最新版的docker软件 yum install -y docker-ce # 重启docker服务 systemctl restart docker } # 加速器配置 speed_config(){ # 定制加速器配置 cat > /etc/docker/daemon.json <<-EOF { "registry-mirrors": [ "http://74f21445.m.daocloud.io", "https://registry.docker-cn.com", "http://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn" ], "insecure-registries": ["10.0.0.20:80"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF # 重启docker服务 systemctl restart docker } # 环境监测 docker_check(){ process_name=$(docker info | grep 'p D' | awk '{print $NF}') [ "${process_name}" == "systemd" ] && echo "Docker软件部署完毕" || (echo "Docker软件部署失败" && exit) } # 软件部署 main(){ softs_base soft_install speed_config docker_check } # 调用主函数 main ``` ```powershell 其他主机环境部署docker [root@kubernetes-master ~]# for i in {15..17} 20; do ssh root@10.0.0.$i "mkdir -p /data/scripts"; scp /data/scripts/03_kubernetes_docker_install.sh root@10.0.0.$i:/data/scripts/03_kubernetes_docker_install.sh; done 其他主机各自执行下面的脚本 /bin/bash /data/scripts/03_kubernetes_docker_install.sh ``` **小结** ``` ``` ### 1.1.4 仓库环境 学习目标 这一节，我们从 容器仓库、Harbor实践、小结 三个方面来学习。 **容器仓库** 简介 ```powershell Harbor是一个用于存储和分发Docker镜像的企业级Registry服务器，虽然Docker官方也提供了公共的镜像仓库，但是从安全和效率等方面考虑，部署企业内部的私有环境Registry是非常必要的，Harbor除了存储和分发镜像外还具有用户管理，项目管理，配置管理和日志查询，高可用部署等主要功能。 在本地搭建一个Harbor服务，其他在同一局域网的机器可以使用Harbor进行镜像提交和拉取，搭建前需要本地安装docker服务和docker-compose服务。 最新的软件版本是 2.5.3，我们这里采用 2.5.0版本。 ``` compose部署 ```powershell 安装docker-compose [root@kubernetes-register ~]# yum install -y docker-compose ``` harbor部署 ```powershell 下载软件 [root@kubernetes-register ~]# mkdir /data/{softs,server} -p [root@kubernetes-register ~]# cd /data/softs [root@kubernetes-register ~]# wget https://github.com/goharbor/harbor/releases/download/v2.5.0/harbor-offline-installer-v2.5.0.tgz 解压软件 [root@kubernetes-register ~]# tar -zxvf harbor-offline-installer-v2.5.0.tgz -C /data/server/ [root@kubernetes-register ~]# cd /data/server/harbor/ 加载镜像 [root@kubernetes-register /data/server/harbor]# docker load < harbor.v2.5.0.tar.gz [root@kubernetes-register /data/server/harbor]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE goharbor/harbor-exporter v2.5.0 36396f138dfb 3 months ago 86.7MB goharbor/chartmuseum-photon v2.5.0 eaedcf1f700b 3 months ago 225MB goharbor/redis-photon v2.5.0 1e00fcc9ae63 3 months ago 156MB goharbor/trivy-adapter-photon v2.5.0 4e24a6327c97 3 months ago 164MB goharbor/notary-server-photon v2.5.0 6d5fe726af7f 3 months ago 112MB goharbor/notary-signer-photon v2.5.0 932eed8b6e8d 3 months ago 109MB goharbor/harbor-registryctl v2.5.0 90ef6b10ab31 3 months ago 136MB goharbor/registry-photon v2.5.0 30e130148067 3 months ago 77.5MB goharbor/nginx-photon v2.5.0 5041274b8b8a 3 months ago 44MB goharbor/harbor-log v2.5.0 89fd73f9714d 3 months ago 160MB goharbor/harbor-jobservice v2.5.0 1d097e877be4 3 months ago 226MB goharbor/harbor-core v2.5.0 42a54bc05b02 3 months ago 202MB goharbor/harbor-portal v2.5.0 c206e936f4f9 3 months ago 52.3MB goharbor/harbor-db v2.5.0 d40a1ae87646 3 months ago 223MB goharbor/prepare v2.5.0 36539574668f 3 months ago 268MB ``` ```powershell 备份配置 [root@kubernetes-register /data/server/harbor]# cp harbor.yml.tmpl harbor.yml 修改配置 [root@kubernetes-register /data/server/harbor]# vim harbor.yml.tmpl # 修改主机名 hostname: kubernetes-register.superopsmsb.com http: port: 80 #https: 注释ssl相关的部分 # port: 443 # certificate: /your/certificate/path # private_key: /your/private/key/path # 修改harbor的登录密码 harbor_admin_password: 123456 # 设定harbor的数据存储目录 data_volume: /data/server/harbor/data ``` ```powershell 配置harbor [root@kubernetes-register /data/server/harbor]# ./prepare prepare base dir is set to /data/server/harbor WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https ... Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir 启动harbor [root@kubernetes-register /data/server/harbor]# ./install.sh [Step 0]: checking if docker is installed ... ... [Step 1]: checking docker-compose is installed ... ... [Step 2]: loading Harbor images ... ... Loaded image: goharbor/harbor-exporter:v2.5.0 [Step 3]: preparing environment ... ... [Step 4]: preparing harbor configs ... ... [Step 5]: starting Harbor ... ... ✔ ----Harbor has been installed and started successfully.---- ``` ```powershell 检查效果 [root@kubernetes-register /data/server/harbor]# docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------- harbor-core /harbor/entrypoint.sh Up harbor-db /docker-entrypoint.sh 96 13 Up harbor-jobservice /harbor/entrypoint.sh Up harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up nginx nginx -g daemon off; Up 0.0.0.0:80->8080/tcp,:::80->8080/tcp redis redis-server /etc/redis.conf Up registry /home/harbor/entrypoint.sh Up registryctl /home/harbor/start.sh Up ``` 定制服务启动脚本 ```powershell 定制服务启动脚本 [root@kubernetes-register /data/server/harbor]# cat /lib/systemd/system/harbor.service [Unit] Description=Harbor After=docker.service systemd-networkd.service systemd-resolved.service Requires=docker.service Documentation=http://github.com/vmware/harbor [Service] Type=simple Restart=on-failure RestartSec=5 #需要注意harbor的安装位置 ExecStart=/usr/bin/docker-compose --file /data/server/harbor/docker-compose.yml up ExecStop=/usr/bin/docker-compose --file /data/server/harbor/docker-compose.yml down [Install] WantedBy=multi-user.target ``` ```powershell 加载服务配置文件 [root@kubernetes-register /data/server/harbor]# systemctl daemon-reload 启动服务 [root@kubernetes-register /data/server/harbor]# systemctl start harbor 检查状态 [root@kubernetes-register /data/server/harbor]# systemctl status harbor 设置开机自启动 [root@kubernetes-register /data/server/harbor]# systemctl enable harbor ``` **Harbor实践** windows定制harbor的访问域名 ```powershell 10.0.0.20 kubernetes-register.superopsmsb.com ``` 浏览器访问域名，用户名: admin, 密码：123456 ![image-20220715125357778](../../img/kubernetes/kubernetes_flannel/image-20220715125357778.png) ```powershell 输入用户名和密码后，点击登录，查看harbor的首页效果 ``` ![image-20220715125531167](../../img/kubernetes/kubernetes_flannel/image-20220715125531167.png) 创建工作账号 ```powershell 点击用户管理，进入用户创建界面 ``` ![image-20220715125606669](../../img/kubernetes/kubernetes_flannel/image-20220715125606669.png) ```powershell 点击创建用户，进入用户创建界面 ``` ![image-20220715125758524](../../img/kubernetes/kubernetes_flannel/image-20220715125758524.png) ```powershell 点击确定后，查看创建用户效果 ``` ![image-20220715125834969](../../img/kubernetes/kubernetes_flannel/image-20220715125834969.png) ```powershell 点击左上角的管理员名称，退出终端页面 ``` 仓库管理 ```powershell 采用普通用户登录到harbor中 ``` ![image-20220715130019232](../../img/kubernetes/kubernetes_flannel/image-20220715130019232.png) ```powershell 创建shuji用户专用的项目仓库，名称为 superopsmsb，权限为公开的 ``` ![image-20220715130129241](../../img/kubernetes/kubernetes_flannel/image-20220715130129241.png) ```powershell 点击确定后，查看效果 ``` ![image-20220715130201235](../../img/kubernetes/kubernetes_flannel/image-20220715130201235.png) 提交镜像 ```powershell 准备docker的配置文件 [root@kubernetes-master ~]# grep insecure /etc/docker/daemon.json "insecure-registries": ["kubernetes-register.superopsmsb.com"], [root@kubernetes-master ~]# systemctl restart docker [root@kubernetes-master ~]# docker info | grep -A2 Insecure Insecure Registries: kubernetes-register.superopsmsb.com 127.0.0.0/8 ``` ```powershell 登录仓库 [root@kubernetes-master ~]# docker login kubernetes-register.superopsmsb.com -u shuji Password: # 输入登录密码 A12345678a WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ``` ```powershell 下载镜像 [root@kubernetes-master ~]# docker pull busybox 定制镜像标签 [root@kubernetes-master ~]# docker tag busybox kubernetes-register.superopsmsb.com/superopsmsb/busybox:v0.1 推送镜像 [root@kubernetes-master ~]# docker push kubernetes-register.superopsmsb.com/superopsmsb/busybox:v0.1 The push refers to repository [kubernetes-register.superopsmsb.com/superopsmsb/busybox] 7ad00cd55506: Pushed v0.1: digest: sha256:dcdf379c574e1773d703f0c0d56d67594e7a91d6b84d11ff46799f60fb081c52 size: 527 ``` ```powershell 删除镜像 [root@kubernetes-master ~]# docker rmi busybox kubernetes-register.superopsmsb.com/superopsmsb/busybox:v0.1 下载镜像 [root@kubernetes-master ~]# docker pull kubernetes-register.superopsmsb.com/superopsmsb/busybox:v0.1 v0.1: Pulling from superopsmsb/busybox 19d511225f94: Pull complete Digest: sha256:dcdf379c574e1773d703f0c0d56d67594e7a91d6b84d11ff46799f60fb081c52 Status: Downloaded newer image for kubernetes-register.superopsmsb.com/superopsmsb/busybox:v0.1 kubernetes-register.superopsmsb.com/superopsmsb/busybox:v0.1 结果显示： 我们的harbor私有仓库就构建好了 ``` 同步所有的docker配置 ```powershell 同步所有主机的docker配置 [root@kubernetes-master ~]# for i in 15 16 17;do scp /etc/docker/daemon.json root@10.0.0.$i:/etc/docker/daemon.json; ssh root@10.0.0.$i "systemctl restart docker"; done daemon.json 100% 299 250.0KB/s 00:00 daemon.json 100% 299 249.6KB/s 00:00 daemon.json 100% 299 243.5KB/s 00:00 ``` **小结** ``` ``` ### 1.1.5 master环境部署 学习目标 这一节，我们从 软件安装、环境初始化、小结 三个方面来学习。 **软件安装** 简介 ```powershell 我们已经把kubernetes集群所有主机的软件源配置完毕了，所以接下来，我们需要做的就是如何部署kubernetes环境 ``` 软件安装 ```powershell 查看默认的最新版本 [root@kubernetes-master ~]# yum list kubeadm 已加载插件：fastestmirror Loading mirror speeds from cached hostfile 可安装的软件包 kubeadm.x86_64 1.24.3-0 kubernetes 查看软件的最近版本 [root@kubernetes-master ~]# yum list kubeadm --showduplicates | sort -r | grep 1.2 kubeadm.x86_64 1.24.3-0 kubernetes kubeadm.x86_64 1.24.2-0 kubernetes kubeadm.x86_64 1.24.1-0 kubernetes kubeadm.x86_64 1.24.0-0 kubernetes kubeadm.x86_64 1.23.9-0 kubernetes kubeadm.x86_64 1.23.8-0 kubernetes ... ``` ```powershell 安装制定版本 [root@kubernetes-master ~]# yum install kubeadm-1.23.9-0 kubectl-1.23.9-0 kubelet-1.23.9-0 -y 注意： ``` ![image-20220715160251133](../../img/kubernetes/kubernetes_flannel/image-20220715160251133.png) ```powershell 核心软件解析 kubeadm 主要是对k8s集群来进行管理的，所以在master角色主机上安装 kubelet 是以服务的方式来进行启动，主要用于收集节点主机的信息 kubectl 主要是用来对集群中的资源对象进行管控，一半情况下，node角色的节点是不需要安装的。 依赖软件解析 libnetfilter_xxx是Linux系统下网络数据包过滤的配置工具 kubernetes-cni是容器网络通信的软件 socat是kubelet的依赖 cri-tools是CRI容器运行时接口的命令行工具 ``` 命令解读 ```powershell 查看集群初始化命令 [root@kubernetes-master ~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.9", GitCommit:"c1de2d70269039fe55efb98e737d9a29f9155246", GitTreeState:"clean", BuildDate:"2022-07-13T14:25:37Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"} [root@kubernetes-master ~]# kubeadm --help ┌──────────────────────────────────────────────────────────┐ │ KUBEADM │ │ Easily bootstrap a secure Kubernetes cluster │ │ │ │ Please give us feedback at: │ │ https://github.com/kubernetes/kubeadm/issues │ └──────────────────────────────────────────────────────────┘ Example usage: Create a two-machine cluster with one control-plane node (which controls the cluster), and one worker node (where your workloads, like Pods and Deployments run). ┌──────────────────────────────────────────────────────────┐ │ On the first machine: │ ├──────────────────────────────────────────────────────────┤ │ control-plane# kubeadm init │ └──────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────┐ │ On the second machine: │ ├──────────────────────────────────────────────────────────┤ │ worker# kubeadm join │ └──────────────────────────────────────────────────────────┘ You can then repeat the second step on as many other machines as you like. Usage: kubeadm [command] Available Commands: certs Commands related to handling kubernetes certificates completion Output shell completion code for the specified shell (bash or zsh) config Manage configuration for a kubeadm cluster persisted in a ConfigMap in the cluster help Help about any command init Run this command in order to set up the Kubernetes control plane join Run this on any machine you wish to join an existing cluster kubeconfig Kubeconfig file utilities reset Performs a best effort revert of changes made to this host by 'kubeadm init' or 'kubeadm join' token Manage bootstrap tokens upgrade Upgrade your cluster smoothly to a newer version with this command version Print the version of kubeadm Flags: --add-dir-header If true, adds the file directory to the header of the log messages -h, --help help for kubeadm --log-file string If non-empty, use this log file --log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --one-output If true, only write logs to their native severity level (vs also writing to each lower severity level) --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. --skip-headers If true, avoid header prefixes in the log messages --skip-log-headers If true, avoid headers when opening log files -v, --v Level number for the log level verbosity Additional help topics: kubeadm alpha Kubeadm experimental sub-commands Use "kubeadm [command] --help" for more information about a command. ``` 信息查看 ```powershell 查看集群初始化时候的默认配置 [root@kubernetes-master ~]# kubeadm config print init-defaults apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock imagePullPolicy: IfNotPresent name: node taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: 1.23.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} 注意： 可以重点关注，master和node的注释信息、镜像仓库和子网信息 这条命令可以生成定制的kubeadm.conf认证文件 ``` ```powershell 检查当前版本的kubeadm所依赖的镜像版本 [root@kubernetes-master ~]# kubeadm config images list I0715 16:08:07.269149 7605 version.go:255] remote version is much newer: v1.24.3; falling back to: stable-1.23 k8s.gcr.io/kube-apiserver:v1.23.9 k8s.gcr.io/kube-controller-manager:v1.23.9 k8s.gcr.io/kube-scheduler:v1.23.9 k8s.gcr.io/kube-proxy:v1.23.9 k8s.gcr.io/pause:3.6 k8s.gcr.io/etcd:3.5.1-0 k8s.gcr.io/coredns/coredns:v1.8.6 检查指定版本的kubeadm所依赖的镜像版本 [root@kubernetes-master ~]# kubeadm config images list --kubernetes-version=v1.23.6 k8s.gcr.io/kube-apiserver:v1.23.6 k8s.gcr.io/kube-controller-manager:v1.23.6 k8s.gcr.io/kube-scheduler:v1.23.6 k8s.gcr.io/kube-proxy:v1.23.6 k8s.gcr.io/pause:3.6 k8s.gcr.io/etcd:3.5.1-0 k8s.gcr.io/coredns/coredns:v1.8.6 ``` 拉取环境依赖的镜像 ```powershell 预拉取镜像文件 [root@kubernetes-master ~]# kubeadm config images pull I0715 16:09:24.185598 7624 version.go:255] remote version is much newer: v1.24.3; falling back to: stable-1.23 failed to pull image "k8s.gcr.io/kube-apiserver:v1.23.9": output: Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp 142.250.157.82:443: connect: connection timed out , error: exit status 1 To see the stack trace of this error execute with --v=5 or higher 注意： 由于默认情况下，这些镜像是从一个我们访问不到的网站上拉取的，所以这一步在没有实现科学上网的前提下，不要执行。 推荐在初始化之前，更换一下镜像仓库，提前获取文件,比如我们可以从 registry.aliyuncs.com/google_containers/镜像名称 获取镜像文件 ``` ```powershell 镜像获取脚本内容 [root@localhost ~]# cat /data/scripts/04_kubernetes_get_images.sh #!/bin/bash # 功能: 获取kubernetes依赖的Docker镜像文件 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 定制普通环境变量 ali_mirror='registry.aliyuncs.com' harbor_mirror='kubernetes-register.superopsmsb.com' harbor_repo='google_containers' # 环境定制 kubernetes_image_get(){ # 获取脚本参数 kubernetes_version="$1" # 获取制定kubernetes版本所需镜像 images=$(kubeadm config images list --kubernetes-version=${kubernetes_version} | awk -F "/" '{print $NF}') # 获取依赖镜像 for i in ${images} do docker pull ${ali_mirror}/${harbor_repo}/$i docker tag ${ali_mirror}/${harbor_repo}/$i ${harbor_mirror}/${harbor_repo}/$i docker rmi ${ali_mirror}/${harbor_repo}/$i done } # 脚本的帮助 Usage(){ echo "/bin/bash $0 " } # 脚本主流程 if [ $# -eq 0 ] then read -p "请输入要获取kubernetes镜像的版本(示例: v1.23.9): " kubernetes_version kubernetes_image_get ${kubernetes_version} else Usage fi ``` ```powershell 脚本执行效果 [root@kubernetes-master ~]# /bin/bash /data/scripts/04_kubernetes_get_images.sh 请输入要获取kubernetes镜像的版本(示例: v1.23.9): v1.23.9 ... ``` ```powershell 查看镜像 [root@kubernetes-master ~]# docker images [root@kubernetes-master ~]# docker images | awk '{print $1,$2}' REPOSITORY TAG kubernetes-register.superopsmsb.com/google_containers/kube-apiserver v1.23.9 kubernetes-register.superopsmsb.com/google_containers/kube-controller-manager v1.23.9 kubernetes-register.superopsmsb.com/google_containers/kube-scheduler v1.23.9 kubernetes-register.superopsmsb.com/google_containers/kube-proxy v1.23.9 kubernetes-register.superopsmsb.com/google_containers/etcd 3.5.1-0 kubernetes-register.superopsmsb.com/google_containers/coredns v1.8.6 kubernetes-register.superopsmsb.com/google_containers/pause 3.6 harbor创建仓库 登录harbor仓库，创建一个google_containers的公开仓库 登录仓库 [root@kubernetes-master ~]# docker login kubernetes-register.superopsmsb.com -u shuji Password: # 输入A12345678a 提交镜像 [root@kubernetes-master ~]# for i in $(docker images | grep -v TAG | awk '{print $1":"$2}');do docker push $i;done ``` **环境初始化** master主机环境初始化 ```powershell 环境初始化命令 kubeadm init --kubernetes-version=1.23.9 \ --apiserver-advertise-address=10.0.0.12 \ --image-repository kubernetes-register.superopsmsb.com/google_containers \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --ignore-preflight-errors=Swap ``` ```powershell 参数解析： --apiserver-advertise-address 要设定为当前集群的master地址，而且必须为ipv4|ipv6地址 由于kubeadm init命令默认去外网获取镜像，这里我们使用--image-repository来指定使用国内镜像 --kubernetes-version选项的版本号用于指定要部署的Kubenretes程序版本，它需要与当前的kubeadm支持的版本保持一致；该参数是必须的 --pod-network-cidr选项用于指定分Pod分配使用的网络地址，它通常应该与要部署使用的网络插件（例如flannel、calico等）的默认设定保持一致，10.244.0.0/16是flannel默认使用的网络； --service-cidr用于指定为Service分配使用的网络地址，它由kubernetes管理，默认即为10.96.0.0/12； --ignore-preflight-errors=Swap 如果没有该项，必须保证系统禁用Swap设备的状态。一般最好加上 --image-repository 用于指定我们在安装kubernetes环境的时候，从哪个镜像里面下载相关的docker镜像，如果需要用本地的仓库，那么就用本地的仓库地址即可 ``` 环境初始化过程 ```powershell 环境初始化命令 [root@kubernetes-master ~]# kubeadm init --kubernetes-version=1.23.9 \ > --apiserver-advertise-address=10.0.0.12 \ > --image-repository kubernetes-register.superopsmsb.com/google_containers \ > --service-cidr=10.96.0.0/12 \ > --pod-network-cidr=10.244.0.0/16 \ > --ignore-preflight-errors=Swap # 环境初始化过程 [init] Using Kubernetes version: v1.23.9 [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes-master kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.12] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [kubernetes-master localhost] and IPs [10.0.0.12 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [kubernetes-master localhost] and IPs [10.0.0.12 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 11.006830 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node kubernetes-master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node kubernetes-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: vudfvt.fwpohpbb7yw2qy49 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy # 基本初始化完毕后，需要做的一些事情 Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: # 定制kubernetes的登录权限 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf # 定制kubernetes的网络配置 You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: # node节点注册到master节点 kubeadm join 10.0.0.12:6443 --token vudfvt.fwpohpbb7yw2qy49 \ --discovery-token-ca-cert-hash sha256:110b1efec63971fda17154782dc1179fa93ef90a8804be381e5a83a8a7748545 ``` 确认效果 ```powershell 未设定权限前操作 [root@kubernetes-master ~]# kubectl get nodes The connection to the server localhost:8080 was refused - did you specify the right host or port? 设定kubernetes的认证权限 [root@kubernetes-master ~]# mkdir -p $HOME/.kube [root@kubernetes-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@kubernetes-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config 再次检测 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master NotReady control-plane,master 4m10s v1.23.9 ``` **小结** ``` ``` ### 1.1.6 node环境部署 学习目标 这一节，我们从 节点初始化、网络环境配置、小结 三个方面来学习。 **节点初始化** 简介 ```powershell 对于node节点来说，我们无需对集群环境进行管理，所以不需要安装和部署kubectl软件，其他的正常安装，然后根据master节点的认证通信，我们可以进行节点加入集群的配置。 ``` 安装软件 ```powershell 所有node节点都执行如下步骤： [root@kubernetes-master ~]# for i in {15..17}; do ssh root@10.0.0.$i "yum install kubeadm-1.23.9-0 kubelet-1.23.9-0 -y";done ``` 节点初始化(以node1为例) ```powershell 节点1进行环境初始化 [root@kubernetes-node1 ~]# kubeadm join 10.0.0.12:6443 --token vudfvt.fwpohpbb7yw2qy49 \ > --discovery-token-ca-cert-hash sha256:110b1efec63971fda17154782dc1179fa93ef90a8804be381e5a83a8a7748545 [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` ```powershell 回到master节点主机查看节点效果 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master NotReady control-plane,master 17m v1.23.9 kubernetes-node2 NotReady 2m10s v1.23.9 所有节点都做完后，再次查看master的节点效果 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master NotReady control-plane,master 21m v1.23.9 kubernetes-node1 NotReady 110s v1.23.9 kubernetes-node2 NotReady 6m17s v1.23.9 kubernetes-node3 NotReady 17s v1.23.9 ``` **网络环境配置** 简介 ```powershell 根据master节点初始化的效果，我们这里需要单独将网络插件的功能实现 ``` 插件环境部署 ```powershell 创建基本目录 mkdir /data/kubernetes/flannel -p cd /data/kubernetes/flannel 获取配置文件 wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml ``` ```powershell 获取相关镜像 [root@kubernetes-master /data/kubernetes/flannel]# grep image kube-flannel.yml | grep -v '#' image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 image: rancher/mirrored-flannelcni-flannel:v0.18.1 image: rancher/mirrored-flannelcni-flannel:v0.18.1 定制镜像标签 for i in $(grep image kube-flannel.yml | grep -v '#' | awk -F '/' '{print $NF}') do docker pull rancher/$i docker tag rancher/$i kubernetes-register.superopsmsb.com/google_containers/$i docker push kubernetes-register.superopsmsb.com/google_containers/$i docker rmi rancher/$i done ``` ```powershell 备份配置文件 [root@kubernetes-master /data/kubernetes/flannel]# cp kube-flannel.yml{,.bak} 修改配置文件 [root@kubernetes-master /data/kubernetes/flannel]# sed -i '/ image:/s/rancher/kubernetes-register.superopsmsb.com\/google_containers/' kube-flannel.yml [root@kubernetes-master /data/kubernetes/flannel]# sed -n '/ image:/p' kube-flannel.yml image: kubernetes-register.superopsmsb.com/google_containers/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 image: kubernetes-register.superopsmsb.com/google_containers/mirrored-flannelcni-flannel:v0.18.1 image: kubernetes-register.superopsmsb.com/google_containers/mirrored-flannelcni-flannel:v0.18.1 应用配置文件 [root@kubernetes-master /data/kubernetes/flannel]# kubectl apply -f kube-flannel.yml namespace/kube-flannel created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created ``` 检查效果 ```powershell 查看集群节点效果 [root@kubernetes-master /data/kubernetes/flannel]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready control-plane,master 62m v1.23.9 kubernetes-node1 Ready 42m v1.23.9 kubernetes-node2 Ready 47m v1.23.9 kubernetes-node3 Ready 41m v1.23.9 查看集群pod效果 [root@kubernetes-master /data/kubernetes/flannel]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5d555c984-gt4w9 1/1 Running 0 62m coredns-5d555c984-t4gps 1/1 Running 0 62m etcd-kubernetes-master 1/1 Running 0 62m kube-apiserver-kubernetes-master 1/1 Running 0 62m kube-controller-manager-kubernetes-master 1/1 Running 0 62m kube-proxy-48txz 1/1 Running 0 43m kube-proxy-5vdhv 1/1 Running 0 41m kube-proxy-cblk7 1/1 Running 0 47m kube-proxy-hglfm 1/1 Running 0 62m kube-scheduler-kubernetes-master 1/1 Running 0 62m ``` **小结** ``` ``` ### 1.1.7 集群环境实践 学习目标 这一节，我们从 基础功能、节点管理、小结 三个方面来学习。 **基础功能** 简介 ```powershell 目前kubernetes的集群环境已经部署完毕了，但是有些基础功能配置还是需要来梳理一下的。默认情况下，我们在master上执行命令的时候，没有办法直接使用tab方式补全命令，我们可以采取下面方式来实现。 ``` 命令补全 ```powershell 获取相关环境配置 [root@kubernetes-master ~]# kubectl completion bash 加载这些配置 [root@kubernetes-master ~]# source <(kubectl completion bash) 注意: "<(" 两个符号之间没有空格 放到当前用户的环境文件中 [root@kubernetes-master ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc [root@kubernetes-master ~]# echo "source <(kubeadm completion bash)" >> ~/.bashrc [root@kubernetes-master ~]# source ~/.bashrc 测试效果 [root@kubernetes-master ~]# kubectl get n namespaces networkpolicies.networking.k8s.io nodes [root@kubernetes-master ~]# kubeadm co completion config ``` **命令简介** 简介 ```powershell kubectl是kubernetes集群内部管理各种资源对象的核心命令，一般情况下，只有master主机才有 ``` 命令帮助 ```powershell 注意：虽然kubernetes的kubectl命令的一些参数发生了变化，甚至是移除，但是不影响旧有命令的正常使用。 [root@kubernetes-master ~]# kubectl --help kubectl controls the Kubernetes cluster manager. Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/ Basic Commands (Beginner): 资源对象核心的基础命令 create Create a resource from a file or from stdin expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service run 在集群中运行一个指定的镜像 set 为 objects 设置一个指定的特征 Basic Commands (Intermediate): 资源对象的一些基本操作命令 explain Get documentation for a resource get 显示一个或更多 resources edit 在服务器上编辑一个资源 delete Delete resources by file names, stdin, resources and names, or by resources and label selector Deploy Commands: 应用部署相关的命令 rollout Manage the rollout of a resource scale Set a new size for a deployment, replica set, or replication controller autoscale Auto-scale a deployment, replica set, stateful set, or replication controller Cluster Management Commands: 集群管理相关的命令 certificate 修改 certificate 资源. cluster-info Display cluster information top Display resource (CPU/memory) usage cordon 标记 node 为 unschedulable uncordon 标记 node 为 schedulable drain Drain node in preparation for maintenance taint 更新一个或者多个 node 上的 taints Troubleshooting and Debugging Commands: 集群故障管理相关的命令 describe 显示一个指定 resource 或者 group 的 resources 详情 logs 输出容器在 pod 中的日志 attach Attach 到一个运行中的 container exec 在一个 container 中执行一个命令 port-forward Forward one or more local ports to a pod proxy 运行一个 proxy 到 Kubernetes API server cp Copy files and directories to and from containers auth Inspect authorization debug Create debugging sessions for troubleshooting workloads and nodes Advanced Commands: 集群管理高阶命令，一般用一个apply即可 diff Diff the live version against a would-be applied version apply Apply a configuration to a resource by file name or stdin patch Update fields of a resource replace Replace a resource by file name or stdin wait Experimental: Wait for a specific condition on one or many resources kustomize Build a kustomization target from a directory or URL. Settings Commands: 集群的一些设置性命令 label 更新在这个资源上的 labels annotate 更新一个资源的注解 completion Output shell completion code for the specified shell (bash, zsh or fish) Other Commands: 其他的相关命令 alpha Commands for features in alpha api-resources Print the supported API resources on the server api-versions Print the supported API versions on the server, in the form of "group/version" config 修改 kubeconfig 文件 plugin Provides utilities for interacting with plugins version 输出 client 和 server 的版本信息 Usage: kubectl [flags] [options] Use "kubectl --help" for more information about a given command. Use "kubectl options" for a list of global command-line options (applies to all commands). ``` 命令的帮助 ```powershell 查看子命令的帮助信息 样式1：kubectl 子命令 --help 样式2：kubectl help 子命令 ``` ```powershell 样式1实践 [root@kubernetes-master ~]# kubectl version --help Print the client and server version information for the current context. Examples: # Print the client and server versions for the current context kubectl version ... 样式2实践 [root@kubernetes-master ~]# kubectl help version Print the client and server version information for the current context. Examples: # Print the client and server versions for the current context kubectl version ``` 简单实践 ```powershell 查看当前节点效果 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready control-plane,master 77m v1.23.9 kubernetes-node1 Ready 57m v1.23.9 kubernetes-node2 Ready 62m v1.23.9 kubernetes-node3 Ready 56m v1.23.9 移除节点3 [root@kubernetes-master ~]# kubectl delete node kubernetes-node3 node "kubernetes-node3" deleted 查看效果 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready control-plane,master 77m v1.23.9 kubernetes-node1 Ready 57m v1.23.9 kubernetes-node2 Ready 62m v1.23.9 ``` ```powershell 节点环境重置 [root@kubernetes-node3 ~]# kubeadm reset [root@kubernetes-node3 ~]# systemctl restart kubelet docker [root@kubernetes-node3 ~]# kubeadm join 10.0.0.12:6443 --token vudfvt.fwpohpbb7yw2qy49 --discovery-token-ca-cert-hash sha256:110b1efec63971fda17154782dc1179fa93ef90a8804be381e5a83a8a7748545 ``` ```powershell master节点查看效果 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready control-plane,master 82m v1.23.9 kubernetes-node1 Ready 62m v1.23.9 kubernetes-node2 Ready 67m v1.23.9 kubernetes-node3 Ready 55s v1.23.9 ``` **小结** ``` ``` ### 1.1.8 资源对象解读 学习目标 这一节，我们从 资源对象、命令解读、小结 三个方面来学习。 **资源对象** 简介 ```powershell 根据之前的对于kubernetes的简介，我们知道kubernetes之所以在大规模的容器场景下的管理效率非常好，原因在于它将我们人工对于容器的手工操作全部整合到的各种资源对象里面。 在进行资源对象解读之前，我们需要给大家灌输一个基本的认识： kubernetes的资源对象有两种： 默认的资源对象 - 默认的有五六十种，但是常用的也就那么一二十个。 自定义的资源对象 - 用户想怎么定义就怎么定义，数量无法统计，仅需要了解想要了解的即可 ``` ![image-20220715012035435](../../img/kubernetes/kubernetes_flannel/image-20220715012035435.png) 常见资源缩写 | 最基础资源对象 | | | | | ----------------------- | ------ | -------------------------- | ---- | | 资源对象全称 | 缩写 | 资源对象全称 | 缩写 | | pod/pods | po | node/nodes | no | | **最常见资源对象** | | | | | 资源对象全称 | 缩写 | 资源对象全称 | 缩写 | | replication controllers | rc | horizontal pod autoscalers | hpa | | replica sets | rs | persistent volume | pv | | deployment | deploy | persistent volume claims | pvc | | services | svc | | | | **其他资源对象** | | | | | namespaces | ns | storage classes | sc | | config maps | cm | clusters | | | daemon sets | ds | stateful sets | | | endpoints | ep | secrets | | | events | ev | jobs | | | ingresses | ing | | | **命令解读** 语法解读 ```powershell 命令格式： kubectl [子命令] [资源类型] [资源名称] [其他标识-可选] 参数解析： 子命令：操作Kubernetes资源对象的子命令，常见的有create、delete、describe、get等 create 创建资源对象 describe 查找资源的详细信息 delete 删除资源对象 get 获取资源基本信息 资源类型：Kubernetes资源类型，举例：结点的资源类型是nodes，缩写no 资源名称: Kubernetes资源对象的名称，可以省略。 其他标识: 可选参数，一般用于信息的扩展信息展示 ``` 资源对象基本信息 ```powershell 查看资源类型基本信息 [root@kubernetes-master ~]# kubectl api-resources | head -2 NAME SHORTNAMES APIVERSION NAMESPACED KIND bindings v1 true Binding [root@kubernetes-master ~]# kubectl api-resources | grep -v NAME | wc -l 56 查看资源类型的版本信息 [root@kubernetes-master ~]# kubectl api-versions | head -n2 admissionregistration.k8s.io/v1 apiextensions.k8s.io/v1 ``` 常见资源对象查看 ```powershell 获取资源所在命名空间 [root@kubernetes-master ~]# kubectl get ns NAME STATUS AGE default Active 94m kube-flannel Active 32m kube-node-lease Active 94m kube-public Active 94m kube-system Active 94m 获取命名空间的资源对象 [root@kubernetes-master ~]# kubectl get pod No resources found in default namespace. [root@kubernetes-master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5d555c984-gt4w9 1/1 Running 0 149m coredns-5d555c984-t4gps 1/1 Running 0 149m etcd-kubernetes-master 1/1 Running 0 149m kube-apiserver-kubernetes-master 1/1 Running 0 149m kube-controller-manager-kubernetes-master 1/1 Running 0 149m kube-proxy-48txz 1/1 Running 0 129m kube-proxy-cblk7 1/1 Running 0 134m kube-proxy-ds8x5 1/1 Running 2 (69m ago) 70m kube-proxy-hglfm 1/1 Running 0 149m kube-scheduler-kubernetes-master 1/1 Running 0 149m ``` ```powershell cs 获取集群组件相关资源 [root@kubernetes-master ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true","reason":""} ``` ```powershell sa 和 secrets 获取集群相关的用户相关信息 [root@kubernetes-master ~]# kubectl get sa NAME SECRETS AGE default 1 150m [root@kubernetes-master ~]# kubectl get secrets NAME TYPE DATA AGE default-token-nc8rg kubernetes.io/service-account-token 3 150m ``` 信息查看命令 ```powershell 查看资源对象 kubectl get 资源类型 资源名称 <-o yaml/json/wide | -w> 参数解析： -w 是实时查看资源的状态。 -o 是以多种格式查看资源的属性信息 --raw 从api地址中获取相关资源信息 描述资源对象 kubectl describe 资源类型 资源名称 注意： 这个命令非常重要，一般我们应用部署排错时候，就用它。 查看资源应用的访问日志 kubectl logs 资源类型 资源名称 注意： 这个命令非常重要，一般我们服务排错时候，就用它。 ``` 查看信息基本信息 ```powershell 基本的资源对象简要信息 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready control-plane,master 157m v1.23.9 kubernetes-node1 Ready 137m v1.23.9 kubernetes-node2 Ready 142m v1.23.9 kubernetes-node3 Ready 75m v1.23.9 [root@kubernetes-master ~]# kubectl get nodes kubernetes-node1 NAME STATUS ROLES AGE VERSION kubernetes-node1 Ready 137m v1.23.9 ``` ```powershell 查看资源对象的属性信息 [root@kubernetes-master ~]# kubectl get nodes kubernetes-node1 -o yaml apiVersion: v1 kind: Node metadata: annotations: ... ``` ```powershell 查看资源对象的扩展信息 [root@kubernetes-master ~]# kubectl get nodes kubernetes-node1 -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kubernetes-node1 Ready 139m v1.23.9 10.0.0.15 CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64 docker://20.10.17 ``` 查看资源的描述信息 ```powershell describe 查看资源对象的描述信息 [root@kubernetes-master ~]# kubectl describe namespaces default Name: default Labels: kubernetes.io/metadata.name=default Annotations: Status: Active No resource quota. No LimitRange resource. ``` ```powershell logs 查看对象的应用访问日志 [root@kubernetes-master ~]# kubectl -n kube-system logs kube-proxy-hglfm I0715 08:35:01.883435 1 node.go:163] Successfully retrieved node IP: 10.0.0.12 I0715 08:35:01.883512 1 server_others.go:138] "Detected node IP" address="10.0.0.12" I0715 08:35:01.883536 1 server_others.go:561] "Unknown proxy mode, assuming iptables proxy" proxyMode="" ``` **小结** ``` ``` ### 1.1.9 资源对象实践 学习目标 这一节，我们从 命令实践、应用实践、小结 三个方面来学习。 **命令实践** 准备资源 ```powershell 从网上获取镜像 [root@kubernetes-master ~]# docker pull nginx 镜像打标签 [root@kubernetes-master ~]# docker tag nginx:latest kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.0 提交镜像到仓库 [root@kubernetes-master ~]# docker push kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.0 删除镜像 [root@kubernetes-master ~]# docker rmi nginx ``` 创建资源对象 ```powershell 创建一个应用 [root@kubernetes-master ~]# kubectl create deployment nginx --image=kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.0 查看效果 [root@kubernetes-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-f44f65dc-99dvt 1/1 Running 0 13s 10.244.5.2 kubernetes-node3 访问效果 [root@kubernetes-master ~]# curl 10.244.5.2 -I HTTP/1.1 200 OK Server: nginx/1.23.0 ``` ```powershell 为应用暴露流量入口 [root@kubernetes-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort 查看基本信息 [root@kubernetes-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort service/nginx exposed [root@kubernetes-master ~]# kubectl get service nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx NodePort 10.105.31.160 80:32505/TCP 8s 注意： 这里的 NodePort 代表在所有的节点主机上都开启一个能够被外网访问的端口 32505 访问效果 [root@kubernetes-master ~]# curl 10.105.31.160 -I HTTP/1.1 200 OK Server: nginx/1.23.0 [root@kubernetes-master ~]# curl 10.0.0.12:32505 -I HTTP/1.1 200 OK Server: nginx/1.23.0 ``` 查看容器基本信息 ```powershell 查看资源的访问日志 [root@kubernetes-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-f44f65dc-99dvt 1/1 Running 0 9m47s [root@kubernetes-master ~]# kubectl logs nginx-f44f65dc-99dvt ... 10.244.0.0 - - [15/Jul/2022:12:45:57 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.29.0" "-" ``` ```powershell 查看资源的详情信息 [root@kubernetes-master ~]# kubectl describe pod nginx-f44f65dc-99dvt # 资源对象的基本属性信息 Name: nginx-f44f65dc-99dvt Namespace: default Priority: 0 Node: kubernetes-node3/10.0.0.17 Start Time: Fri, 15 Jul 2022 20:44:17 +0800 Labels: app=nginx pod-template-hash=f44f65dc Annotations: Status: Running IP: 10.244.5.2 IPs: IP: 10.244.5.2 Controlled By: ReplicaSet/nginx-f44f65dc # pod内部的容器相关信息 Containers: nginx: # 容器的名称 Container ID: docker://8c0d89c8ab48e02495a2db4a2b2133c86811bd8064f800a16739f9532670d854 Image: kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.0 Image ID: docker-pullable://kubernetes-register.superopsmsb.com/superopsmsb/nginx@sha256:33cef86aae4e8487ff23a6ca16012fac28ff9e7a5e9759d291a7da06e36ac958 Port: Host Port: State: Running Started: Fri, 15 Jul 2022 20:44:24 +0800 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7sxtx (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True # pod内部数据相关的信息 Volumes: kube-api-access-7sxtx: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s # 资源对象在创建过程中遇到的各种信息，这段信息是非常重要的 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10m default-scheduler Successfully assigned default/nginx-f44f65dc-99dvt to kubernetes-node3 Normal Pulling 10m kubelet Pulling image "kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.0" Normal Pulled 10m kubelet Successfully pulled image "kubernetes-register.superopsmsb.com/superopsmsb/nginx:1.23.0" in 4.335869479s Normal Created 10m kubelet Created container nginx Normal Started 10m kubelet Started container nginx ``` ```powershell 资源对象内部的容器信息 [root@kubernetes-master ~]# kubectl exec -it nginx-f44f65dc-99dvt -- /bin/bash root@nginx-f44f65dc-99dvt:/# env KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 HOSTNAME=nginx-f44f65dc-99dvt ... 修改nginx的页面信息 root@nginx-f44f65dc-99dvt:/# grep -A1 'location /' /etc/nginx/conf.d/default.conf location / { root /usr/share/nginx/html; root@nginx-f44f65dc-99dvt:/# echo $HOSTNAME > /usr/share/nginx/html/index.html root@nginx-f44f65dc-99dvt:/# exit exit 访问应用效果 [root@kubernetes-master ~]# curl 10.244.5.2 nginx-f44f65dc-99dvt [root@kubernetes-master ~]# curl 10.0.0.12:32505 nginx-f44f65dc-99dvt ``` **应用实践** 资源的扩容缩容 ```powershell pod的容量扩充 [root@kubernetes-master ~]# kubectl help scale 调整pod数量为3 [root@kubernetes-master ~]# kubectl scale --replicas=3 deployment nginx deployment.apps/nginx scaled 查看效果 [root@kubernetes-master ~]# kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE nginx 1/3 3 1 20m [root@kubernetes-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-f44f65dc-99dvt 1/1 Running 0 20m 10.244.5.2 kubernetes-node3 nginx-f44f65dc-rskvx 1/1 Running 0 16s 10.244.2.4 kubernetes-node1 nginx-f44f65dc-xpkgq 1/1 Running 0 16s 10.244.1.2 kubernetes-node2 ``` ```powershell pod的容量收缩 [root@kubernetes-master ~]# kubectl scale --replicas=1 deployment nginx deployment.apps/nginx scaled [root@kubernetes-master ~]# kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE nginx 1/1 1 1 20m ``` 资源的删除 ```powershell 删除资源有两种方式 方法1： kubectl delete 资源类型 资源1 资源2 ... 资源n 因为限制了资源类型，所以这种方法只能删除一种资源 方法2： kubectl delete 资源类型/资源 因为删除对象的时候，指定了资源类型，所以我们可以通过这种资源类型限制的方式同时删除多种类型资源 ``` ```powershell 删除deployment资源 [root@kubernetes-master ~]# kubectl delete deployments nginx deployment.apps "nginx" deleted [root@kubernetes-master ~]# kubectl get deployment No resources found in default namespace. 删除svc资源 [root@kubernetes-master ~]# kubectl delete svc nginx service "nginx" deleted [root@kubernetes-master ~]# kubectl get svc nginx Error from server (NotFound): services "nginx" not found ``` **小结** ``` ``` ## 1.2 多主集群 ### 1.2.1 集群环境解读 学习目标 这一节，我们从 集群规划、环境解读、小结 三个方面来学习。 **集群规划** 简介 ```powershell 在生产中，因为kubernetes的主角色主机对于整个集群的重要性不可估量，我们的kubernetes集群一般都会采用多主分布式效果。 另外因为大规模环境下，涉及到的资源对象过于繁多，所以，kubernetes集群环境部署的时候，一般会采用属性高度定制的方式来实现。 为了方便后续的集群环境升级的管理操作，我们在高可用的时候，部署 1.23.8的软件版本(当然只需要1.22+版本都可以，不允许出现大跨版本的出现。) ``` 实验环境的效果图 ![image-20220715213302865](../../img/kubernetes/kubernetes_flannel/image-20220715213302865.png) ```powershell 修改master节点主机的hosts文件 [root@localhost ~]# cat /etc/hosts 10.0.0.12 kubernetes-master1.superopsmsb.com kubernetes-master1 10.0.0.13 kubernetes-master2.superopsmsb.com kubernetes-master2 10.0.0.14 kubernetes-master3.superopsmsb.com kubernetes-master3 10.0.0.15 kubernetes-node1.superopsmsb.com kubernetes-node1 10.0.0.16 kubernetes-node2.superopsmsb.com kubernetes-node2 10.0.0.17 kubernetes-node3.superopsmsb.com kubernetes-node3 10.0.0.18 kubernetes-ha1.superopsmsb.com kubernetes-ha1 10.0.0.19 kubernetes-ha2.superopsmsb.com kubernetes-ha2 10.0.0.20 kubernetes-register.superopsmsb.com kubernetes-register ``` ```powershell 脚本执行实现跨主机免密码认证和hosts文件同步 [root@localhost ~]# /bin/bash /data/scripts/01_remote_host_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 1 ``` 另外两台master主机的基本配置 ```powershell kubernetes的内核参数调整 /bin/bash /data/scripts/02_kubernetes_kernel_conf.sh 底层docker环境的部署 /bin/bash /data/scripts/03_kubernetes_docker_install.sh ``` ```powershell 同步docker环境的基本配置 [root@kubernetes-master ~]# for i in 13 14 > do > scp /etc/docker/daemon.json root@10.0.0.$i:/etc/docker/daemon.json > ssh root@10.0.0.$i "systemctl restart docker" > done daemon.json 100% 299 86.3KB/s 00:00 daemon.json 100% 299 86.3KB/s 00:00 ``` **环境解读** 环境组成 ```powershell 多主分布式集群的效果是在单主分布式的基础上，将master主机扩充到3台，作为一个主角色集群来存在，这个时候我们需要考虑数据的接入和输出。 数据接入： 三台master主机需要一个专属的控制平面来统一处理数据来源 - 在k8s中，这入口一般是以port方式存在 数据流出： 在kubernetes集群中，所有的数据都是以etcd的方式来单独存储的，所以这里无序过度干预。 ``` 高可用梳理 ```powershell 为了保证所有数据都能够访问master主机，同时避免单台master出现异常，我们一般会通过高可用的方式将流量转交个后端，这里采用常见的 keepalived 和 haproxy的方式来实现。 ``` ![1627541947686](../../img/kubernetes/kubernetes_flannel/1627541947686.png) 其他功能 ```powershell 一般来说，对于大型集群来说，其内部的时间同步、域名解析、配置管理、持续交付等功能，都需要单独的主机来进行实现。由于我们这里主要来演示核心的多主集群环境，所以其他的功能，大家可以自由的向当前环境中补充。 ``` **小结** ``` ``` ### 1.2.2 高可用环境实践 学习目标 这一节，我们从 高可用、高负载、高可用升级、小结 四个方面来学习。 **高可用** 简介 ```powershell 所谓的高可用，将核心业务使用多台(一般是2台)主机共同工作，支撑并保障核心业务的正常运行，尤其是业务的对外不间断的对外提供服务。核心特点就是"冗余"，它存在的目的就是为了解决单点故障(Single Point of Failure)问题的。 高可用集群是基于高扩展基础上的一个更高层次的网站稳定性解决方案。网站的稳定性体现在两个方面：网站可用性和恢复能力 ``` keepalived简介 ![1627542432899](../../img/kubernetes/kubernetes_flannel/1627542432899.png) 软件安装 ```powershell 我们在 10.0.0.18和10.0.0.19主机上部署keepalived软件 [root@kubernetes-ha1 ~]# yum install keepalived -y 查看配置文件模板 [root@kubernetes-ha1 ~]# rpm -ql keepalived | grep sample /usr/share/doc/keepalived-1.3.5/samples ``` 软件配置 ```powershell 10.0.0.19主机配置高可用从节点 [root@kubernetes-ha2 ~]# cp /etc/keepalived/keepalived.conf{,.bak} [root@kubernetes-ha2 ~]# cat /etc/keepalived/keepalived.conf global_defs { router_id kubernetes_ha2 } vrrp_instance VI_1 { state BACKUP # 当前节点为高可用从角色 interface eth0 virtual_router_id 51 priority 90 advert_int 1 # 主备通讯时间间隔 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.200 dev eth0 label eth0:1 } } 重启服务后查看效果 [root@kubernetes-ha2 ~]# systemctl restart keepalived.service [root@kubernetes-ha2 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:24:cd:0e txqueuelen 1000 (Ethernet) 结果显示： 高可用功能已经开启了 ``` ```powershell 10.0.0.18主机配置高可用主节点 [root@kubernetes-ha1 ~]# cp /etc/keepalived/keepalived.conf{,.bak} 从10.0.0.19主机拉取配置文件 [root@kubernetes-ha1 ~]# scp root@10.0.0.19:/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf 修改配置文件 [root@kubernetes-ha1 ~]# cat /etc/keepalived/keepalived.conf global_defs { router_id kubernetes_ha1 } vrrp_instance VI_1 { state MASTER # 当前节点为高可用主角色 interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.200 dev eth0 label eth0:1 } } 启动服务 [root@kubernetes-ha1 ~]# systemctl start keepalived.service [root@kubernetes-ha1 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) [root@kubernetes-ha2 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 ether 00:50:56:24:cd:0e txqueuelen 1000 (Ethernet) 结果显示： 高可用的主节点把从节点的ip地址给夺过来了，实现了节点的漂移 ``` ```powershell 主角色关闭服务 [root@kubernetes-ha1 ~]# systemctl stop keepalived.service [root@kubernetes-ha1 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) 从节点查看vip效果 [root@kubernetes-ha2 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:24:cd:0e txqueuelen 1000 (Ethernet) 主角色把vip抢过来 [root@kubernetes-ha1 ~]# systemctl start keepalived.service eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) ``` ```powershell 注意： 在演示实验环境的时候，如果你的主机资源不足，可以只开一台keepalived ``` **高负载** 简介 ```powershell 所谓的高负载集群，指的是在当前业务环境集群中，所有的主机节点都处于正常的工作活动状态，它们共同承担起用户的请求带来的工作负载压力，保证用户的正常访问。支持高可用的软件很多，比如nginx、lvs、haproxy、等，我们这里用的是haproxy。 HAProxy是法国开发者 威利塔罗(Willy Tarreau) 在2000年使用C语言开发的一个开源软件，是一款具备高并发(一万以上)、高性能的TCP和HTTP负载均衡器，支持基于cookie的持久性，自动故障切换，支持正则表达式及web状态统计、它也支持基于数据库的反向代理。 ``` 软件安装 ```powershell 我们在 10.0.0.18和10.0.0.19主机上部署haproxy软件 [root@kubernetes-ha1 ~]# yum install haproxy -y ``` 软件配置 ```powershell 10.0.0.18主机配置高负载 [root@kubernetes-ha1 ~]# cp /etc/haproxy/haproxy.cfg{,.bak} [root@kubernetes-ha1 ~]# cat /etc/haproxy/haproxy.cfg ... listen status bind 10.0.0.200:9999 mode http log global stats enable stats uri /haproxy-status stats auth superopsmsb:123456 listen kubernetes-api-6443 bind 10.0.0.200:6443 mode tcp server kubernetes-master1 10.0.0.12:6443 check inter 3s fall 3 rise 5 server kubernetes-master2 10.0.0.13:6443 check inter 3s fall 3 rise 5 server kubernetes-master3 10.0.0.14:6443 check inter 3s fall 3 rise 5 重启服务后查看效果 [root@kubernetes-ha1 ~]# systemctl start haproxy.service [root@kubernetes-ha1 ~]# netstat -tnulp | head -n4 Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 10.0.0.200:6443 0.0.0.0:* LISTEN 2790/haproxy tcp 0 0 10.0.0.200:9999 0.0.0.0:* LISTEN 2790/haproxy ``` ```powershell 浏览器查看haproxy的页面效果 10.0.0.200:9999/haproxy-status,输入用户名和密码后效果如下 ``` ![image-20220716075320996](../../img/kubernetes/kubernetes_flannel/image-20220716075320996.png) ```powershell 10.0.0.19主机配置高可用主节点 [root@kubernetes-ha2 ~]# cp /etc/haproxy/haproxy.cfg{,.bak} 把10.0.0.18主机配置传递到10.0.0.19主机 [root@kubernetes-ha1 ~]# scp /etc/haproxy/haproxy.cfg root@10.0.0.19:/etc/haproxy/haproxy.cfg 默认情况下，没有vip的节点是无法启动haproxy的 [root@kubernetes-ha2 ~]# systemctl start haproxy [root@kubernetes-ha2 ~]# systemctl status haproxy.service ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) ... 通过修改内核的方式，让haproxy绑定一个不存在的ip地址从而启动成功 [root@kubernetes-ha2 ~]# sysctl -a | grep nonlocal net.ipv4.ip_nonlocal_bind = 0 net.ipv6.ip_nonlocal_bind = 0 开启nonlocal的内核参数 [root@kubernetes-ha2 ~]# echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf [root@kubernetes-ha2 ~]# sysctl -p net.ipv4.ip_nonlocal_bind = 1 注意： 这一步最好在ha1上也做一下 再次启动haproxy服务 [root@kubernetes-ha2 ~]# systemctl start haproxy [root@kubernetes-ha2 ~]# systemctl status haproxy | grep Active Active: active (running) since 六 2062-07-16 08:05:00 CST; 14s ago [root@kubernetes-ha1 ~]# netstat -tnulp | head -n4 Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 10.0.0.200:6443 0.0.0.0:* LISTEN 3062/haproxy tcp 0 0 10.0.0.200:9999 0.0.0.0:* LISTEN 3062/haproxy ``` ```powershell 主角色关闭服务 [root@kubernetes-ha1 ~]# systemctl stop keepalived.service [root@kubernetes-ha1 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) 从节点查看vip效果 [root@kubernetes-ha2 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:24:cd:0e txqueuelen 1000 (Ethernet) ``` **高可用升级** 问题 ```powershell 目前我们实现了高可用的环境，无论keepalived是否在哪台主机存活，都有haproxy能够提供服务，但是在后续处理的时候，会出现一种问题，haproxy一旦故障，而keepalived没有同时关闭的话，会导致服务无法访问。效果如下 ``` ![image-20220716081105875](../../img/kubernetes/kubernetes_flannel/image-20220716081105875.png) ```powershell 所以我们有必要对keeaplived进行升级，需要借助于其内部的脚本探测机制实现对后端haproxy进行探测，如果后端haproxy异常就直接把当前的keepalived服务关闭。 ``` 制作keepalived的脚本文件 ```powershell [root@localhost ~]# cat /etc/keepalived/check_haproxy.sh #!/bin/bash # 功能: keepalived检测后端haproxy的状态 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 检测后端haproxy的状态 haproxy_status=$(ps -C haproxy --no-header | wc -l) # 如果后端没有haproxy服务，则尝试启动一次haproxy if [ $haproxy_status -eq 0 ];then systemctl start haproxy >> /dev/null 2>&1 sleep 3 # 如果重启haproxy还不成功的话，则关闭keepalived服务 if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ] then systemctl stop keepalived fi fi 脚本赋予执行权限 [root@kubernetes-ha1 ~]# chmod +x /etc/keepalived/check_haproxy.sh ``` 改造keepalived的配置文件 ```powershell 查看10.0.0.18主机的高可用配置修改 [root@kubernetes-ha1 ~]# cat /etc/keepalived/keepalived.conf global_defs { router_id kubernetes_ha1 } # 定制keepalive检测后端haproxy服务 vrrp_script chk_haproxy { script "/bin/bash /etc/keepalived/check_haproxy.sh" interval 2 weight -20 # 每次检测失败都降低权重值 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } # 应用内部的检测机制 track_script { chk_haproxy } virtual_ipaddress { 10.0.0.200 dev eth0 label eth0:1 } } 注意： keepalived所有配置后面不允许有任何空格，否则启动有可能出现异常 重启keepalived服务 [root@kubernetes-ha1 ~]# systemctl restart keepalived ``` ```powershell 传递检测脚本给10.0.0.19主机 [root@kubernetes-ha1 ~]# scp /etc/keepalived/check_haproxy.sh root@10.0.0.19:/etc/keepalived/check_haproxy.sh 10.0.0.19主机，也将这两部分修改配置添加上，并重启keepalived服务 [root@kubernetes-ha2 ~]# systemctl restart keepalived ``` 服务检测 ```powershell 查看当前vip效果 [root@kubernetes-ha1 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) ``` ```powershell 因为haproxy内置的强制保证服务启动机制，无法通过stop的方式关闭，通过移除配置文件方式强制关闭haproxy [root@kubernetes-ha1 ~]# mv /etc/haproxy/haproxy.cfg ./ [root@kubernetes-ha1 ~]# systemctl stop haproxy.service [root@kubernetes-ha1 ~]# systemctl status haproxy.service | grep Active Active: failed (Result: exit-code) 10.0.0.18主机检查vip效果 [root@kubernetes-ha1 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) 10.0.0.19主机检测vip效果 [root@kubernetes-ha2 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:24:cd:0e txqueuelen 1000 (Ethernet) 结果显示： 到此为止，我们所有的高可用环境都梳理完毕了 ``` **小结** ``` ``` ### 1.2.3 集群初始化实践 学习目标 这一节，我们从 配置解读、文件实践、小结 三个方面来学习。 **配置解读** 简介 ```powershell 对于kubernetes来说，它提供了高度定制配置的方式来设定集群环境，为了方便后续集群的升级等操作，我们这里安装 1.23.8版本。 ``` 环境准备 ```powershell 所有kubernetes主机节点上都安装1.23.8版本的软件： yum remove kubeadm kubectl kubelet -y yum install kubeadm-1.23.8-0 kubectl-1.23.8-0 kubelet-1.23.8-0 -y ``` ```powershell 执行脚本获取制定版本的kubernetes依赖的docker镜像 [root@kubernetes-master ~]# /bin/bash /data/scripts/04_kubernetes_get_images.sh 请输入要获取kubernetes镜像的版本(示例: v1.23.9): v1.23.8 ... 提交镜像到远程仓库 [root@kubernetes-master ~]# for i in $(docker images | grep -v TAG | awk '{print $1":"$2}');do docker push $i;done ``` 配置文件解读 ```powershell kubernetes的集群初始化两种方式： 方法1：命令方式灵活，但是后期配置不可追溯 方法2：文件方式繁琐，但是所有配置都可以控制 ``` ```powershell 查看集群初始化时候的默认配置 [root@kubernetes-master ~]# kubeadm config print init-defaults apiVersion: kubeadm.k8s.io/v1beta3 # 不同版本的api版本不一样 # 认证授权相关的基本信息 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration # 第一个master节点配置的入口 localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 # node节点注册到master集群的通信方式 nodeRegistration: criSocket: /var/run/dockershim.sock imagePullPolicy: IfNotPresent name: node taints: null --- # 基本的集群属性信息 apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} # kubernetes的数据管理方式 etcd: local: dataDir: /var/lib/etcd # 镜像仓库的配置 imageRepository: k8s.gcr.io kind: ClusterConfiguration # kubernetes版本的定制 kubernetesVersion: 1.23.0 # kubernetes的网络基本信息 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} ``` ```powershell 修改配置文件，让其支持新master的内容 bootstrapTokens.groups.ttl 修改为48h或者更大 localAPIEndpoint.advertiseAddress 修改为当前主机的真实ip地址 controlPlanelEndpoint 如果涉及到haproxy的方式，需要额外添加基于VIP的Endpoint地址 注意：该项一般是在 dns{} 的上一行 imageRepository 修改为我们能够访问到镜像的仓库地址 kubernetesVersion 修改为当前的 k8s 的版本信息 networking.serviceSubnet 改为自己的规划中的service地址信息，最好是大一点的私网地址段 ``` 配置文件变动 ```powershell 创建专属目录 [root@kubernetes-master ~]# mkdir -p /data/kubernetes/cluster_init 在集群初始化节点上生成通信认证的配置信息 [root@kubernetes-master ~]# kubeadm config print init-defaults > /data/kubernetes/cluster_init/kubeadm-init-1-23-8.yml ``` ```powershell 修改配置文件 [root@kubernetes-master ~]# cat /data/kubernetes/cluster_init/kubeadm-init-1-23-8.yml apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 48h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 10.0.0.12 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock imagePullPolicy: IfNotPresent name: kubernetes-master1 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} controlPlaneEndpoint: 10.0.0.200:6443 dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: kubernetes-register.superopsmsb.com/google_containers kind: ClusterConfiguration kubernetesVersion: 1.23.8 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 scheduler: {} ``` **文件实践** 环境初始化 ```powershell 初始化集群环境 [root@kubernetes-master ~]# kubeadm init --config /data/kubernetes/cluster_init/kubeadm-init-1-23-8.yml [root@kubernetes-master ~]# kubeadm init --config /data/kubernetes/cluster_init/kubeadm-init-1-23-8.yml [init] Using Kubernetes version: v1.23.8 [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes-master1 kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.12 10.0.0.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [kubernetes-master1 localhost] and IPs [10.0.0.12 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [kubernetes-master1 localhost] and IPs [10.0.0.12 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 15.554614 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node kubernetes-master1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node kubernetes-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ # 这里多了一个主角色节点加入集群的步骤 You can now join any number of control-plane nodes by copying certificate authorities and service account keys on each node and then running the following as root: kubeadm join 10.0.0.200:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bfaed502274988b5615006e91337db9252707f2dd033e9d32670175bf8445d67 \ --control-plane Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.0.0.200:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bfaed502274988b5615006e91337db9252707f2dd033e9d32670175bf8445d67 ``` ```powershell 设定主角色主机的权限文件 [root@kubernetes-master1 ~]# mkdir -p $HOME/.kube [root@kubernetes-master1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@kubernetes-master1 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` master节点的集群加入 ```powershell 获取master集群节点添加到控制平面的基本认证信息 [root@kubernetes-master1 ~]# kubeadm init phase upload-certs --upload-certs I0716 10:13:59.447562 119631 version.go:255] remote version is much newer: v1.24.3; falling back to: stable-1.23 [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 120fe6a5baad0ec32da4ae185728f5273e415aeb28e36de3b92ad4e1ecd7d69a ``` ```powershell 构造master节点添加到控制平面的命令，在剩余的maste2和master3上执行 [root@kubernetes-master2 ~]# kubeadm join 10.0.0.200:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:bfaed502274988b5615006e91337db9252707f2dd033e9d32670175bf8445d67 --control-plane --certificate-key 120fe6a5baad0ec32da4ae185728f5273e415aeb28e36de3b92ad4e1ecd7d69a ``` ```powershell 所有master节点执行命令完毕后查看效果 [root@kubernetes-master3 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master1 Ready control-plane,master 16m v1.23.8 kubernetes-master2 NotReady control-plane,master 2m13s v1.23.8 kubernetes-master3 NotReady control-plane,master 68s v1.23.8 ``` **小结** ``` ``` ### 1.2.4 工作节点实践 学习目标 这一节，我们从 节点实践、dashboard实践、小结 三个方面来学习。 **节点实践** 将三个工作节点添加到kubernetes集群中 ```powershell 三个节点执行的命令一致 kubeadm join 10.0.0.200:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:bfaed502274988b5615006e91337db9252707f2dd033e9d32670175bf8445d67 ``` ```powershell 节点执行命令完毕后，回到任意一台master上查看效果 [root@kubernetes-master3 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master1 Ready control-plane,master 16m v1.23.8 kubernetes-master2 NotReady control-plane,master 2m13s v1.23.8 kubernetes-master3 NotReady control-plane,master 68s v1.23.8 kubernetes-node1 Ready 44s v1.23.8 kubernetes-node2 Ready 25s v1.23.8 kubernetes-node3 Ready 35s v1.23.8 结果显示： 有的节点状态是Ready，有的不是，原因是我们的flannel环境还没有定制 ``` 定制网络环境 ```powershell master1节点上执行网络配置文件 [root@kubernetes-master1 ~]# kubectl apply -f /data/kubernetes/flannel/kube-flannel.yml namespace/kube-flannel created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created ``` ```powershell 再次查看节点效果 [root@kubernetes-master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master1 Ready control-plane,master 18m v1.23.8 kubernetes-master2 Ready control-plane,master 3m48s v1.23.8 kubernetes-master3 Ready control-plane,master 2m43s v1.23.8 kubernetes-node1 Ready 2m19s v1.23.8 kubernetes-node2 Ready 2m v1.23.8 kubernetes-node3 Ready 2m10s v1.23.8 ``` **dashboard实践** 简介 ```powershell 目前来说，kubernetes的dashboard插件版本没有通用的，只能找到适合当前kubernetes版本的软件。另外，默认的dashboard有可能会受到各种因素的限制，我们还是希望去安装专门的kubernetes的可视化界面软件。 参考资料: https://kubernetes.io/docs/concepts/cluster-administration/addons/ ``` ![image-20220716103544284](../../img/kubernetes/kubernetes_flannel/image-20220716103544284.png) ```powershell 在这里，我们选择v2.5.1版本，这个版本对于1.23.x版本支持没有太大的问题。 [root@kubernetes-master1 ~]# mkdir /data/kubernetes/dashboard -p [root@kubernetes-master1 ~]# cd /data/kubernetes/dashboard/ [root@kubernetes-master1 /data/kubernetes/dashboard]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml [root@kubernetes-master1 /data/kubernetes/dashboard]# mv recommended.yaml dashboard-2.5.1.yaml [root@kubernetes-master1 /data/kubernetes/dashboard]# cp dashboard-2.5.1.yaml{，.bak} ``` 修改配置文件 ```powershell 添加相关配置 [root@kubernetes-master1 /data/kubernetes/dashboard]# vim dashboard-2.5.1.yaml kind: Service ... spec: ports: - port: 443 targetPort: 8443 nodePort: 30443 # 增加此行 type: NodePort # 增加此行 selector: k8s-app: kubernetes-dashboard ``` ```powershell 定制依赖镜像 for i in $(grep 'image:' dashboard-2.5.1.yaml | grep -v '#' | awk -F '/' '{print $NF}') do docker pull kubernetesui/$i docker tag kubernetesui/$i kubernetes-register.superopsmsb.com/google_containers/$i docker push kubernetes-register.superopsmsb.com/google_containers/$i docker rmi kubernetesui/$i done 确认镜像 [root@kubernetes-master1 /data/kubernetes/dashboard]# docker images | egrep 'dashboard|scr' | awk '{print $1,$2}' kubernetes-register.superopsmsb.com/google_containers/dashboard v2.5.1 kubernetes-register.superopsmsb.com/google_containers/metrics-scraper v1.0.7 ``` ```powershell 修改配置文件 [root@kubernetes-master1 /data/kubernetes/dashboard]# sed -i '/ image:/s/kubernetesui/kubernetes-register.superopsmsb.com\/google_containers/' dashboard-2.5.1.yaml [root@kubernetes-master1 /data/kubernetes/dashboard]# sed -n '/ image:/p' dashboard-2.5.1.yaml image: kubernetes-register.superopsmsb.com/google_containers/dashboard:v2.5.1 image: kubernetes-register.superopsmsb.com/google_containers/metrics-scraper:v1.0.7 ``` 应用配置文件 ```powershell [root@kubernetes-master1 /data/kubernetes/dashboard]# kubectl apply -f dashboard-2.5.1.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created 查看环境的准备效果 [root@kubernetes-master1 ~]# kubectl get all -n kubernetes-dashboard ``` ![image-20220716105728014](../../img/kubernetes/kubernetes_flannel/image-20220716105728014.png) 浏览器访问效果 ![image-20220716105728014](../../img/kubernetes/kubernetes_flannel/image-20220716105844575.png) dashboard界面登录 ```powershell 由于涉及到后续我们还没有整理的资料，所以大家执行执行一条命令即可，之余涉及到具体知识点，我们后续再说。 kubectl create serviceaccount dashboard-admin -n kube-system kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin kubernetes_user=$(kubectl -n kube-system get secret | grep dashboard-admin | awk '{print $1}') kubectl describe secrets -n kube-system ${kubernetes_user} ``` ![image-20220716110204077](../../img/kubernetes/kubernetes_flannel/image-20220716110204077.png) ```powershell 拷贝token信息，然后粘贴到浏览器界面的输入框中，然后点击登录，查看效果 ``` ![image-20220716110306390](../../img/kubernetes/kubernetes_flannel/image-20220716110306390.png) **小结** ``` ``` ### 1.2.5 集群初始化解读 学习目标 这一节，我们从 流程简介、流程解读、小结 三个方面来学习。 ```powershell 参考资料: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-workflow ``` **流程简介** 参考资料 ```powershell https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-workflow ``` 流程简介 ```powershell master节点启动： 当前环境检查，确保当前主机可以部署kubernetes 生成kubernetes对外提供服务所需要的各种私钥以及数字证书 生成kubernetes控制组件的kubeconfig文件 生成kubernetes控制组件的pod对象需要的manifest文件 为集群控制节点添加相关的标识，不让主节点参与node角色工作 生成集群的统一认证的token信息，方便其他节点加入到当前的集群 进行基于TLS的安全引导相关的配置、角色策略、签名请求、自动配置策略 为集群安装DNS和kube-porxy插件 ``` ```powershell node节点加入 当前环境检查，读取相关集群配置信息 获取集群相关数据后启动kubelet服务 获取认证信息后，基于证书方式进行通信 ``` **流程解读** master节点初始化流程 ```powershell 1 当前环境检查，确保当前主机可以部署kubernetes [init] Using Kubernetes version: v1.23.8 [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.s ervice' [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubele t.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' ``` ```powershell 2 生成kubernetes对外提供服务所需要的各种私钥以及数字证书 [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes-master1 kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.12 10.0.0.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [kubernetes-master1 localhost] and IPs [10.0.0.12 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [kubernetes-master1 localhost] and IPs [10.0.0.12 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key ``` ```powershell 3 生成kubernetes控制组件的kubeconfig文件及相关的启动配置文件 [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet ``` ```powershell 4 生成kubernetes控制组件的pod对象需要的manifest文件 [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 15.554614 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. [upload-certs] Skipping phase. Please see --upload-certs ``` ```powershell 5 为集群控制节点添加相关的标识，不让主节点参与node角色工作 [mark-control-plane] Marking the node kubernetes-master1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node kubernetes-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] ``` ```powershell 6 生成集群的统一认证的token信息，方便其他节点加入到当前的集群 [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace ``` ```powershell 7 进行基于TLS的安全引导相关的配置、角色策略、签名请求、自动配置策略 [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key ``` ```powershell 8 为集群安装DNS和kube-porxy插件 [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy ``` node节点初始化流程 ```powershell 1 当前环境检查，读取相关集群配置信息 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' ``` ```powershell 2 获取集群相关数据后启动kubelet服务 [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet ``` ```powershell 3 获取认证信息后，基于证书方式进行通信 [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` **小结** ``` ``` ## 1.3 综合实践 ### 1.3.1 应用案例解读 学习目标 这一节，我们从 环境规划、实践解读、小结 三个方面来学习。 **环境规划** 简介 ```powershell 为了让大家对于kubernetes在生产中如何工作，我们这里从0创建容器环境，然后迁移到kubernetes环境上，因为还没有涉及到kubernetes本身的功能学习，所以涉及到部分的资源清单文件，我们会直接粘贴复制，后面的资源清单信息，我们会依次进行手工编写处理。 ``` 环境规划 ![image-20220716135622610](../../img/kubernetes/kubernetes_flannel/image-20220716135622610.png) ```powershell 这里定制一个nginx代理后端的一个简单应用，后面所有的知识点应用，基本上都是基于这个架构进行上下延伸的。 ``` 访问分析 ```powershell 用户访问 10.0.0.201， 如果后面url关键字是 /,则将流量转交给 Nginx,应用返回 Hello Nginx 容器名-版本号 如果后面url关键字是 /django/,则将流量转交给 Django,应用返回 Hello Django 容器名-版本号 如果后面url关键字是 /tomcat/,则将流量转交给 Tomcat,应用返回 Hello Tomcat 容器名-版本号 ``` 基本思路 ```powershell 1 定制各个专用的Docker镜像文件 2 定制kubernetes的专属资源清单文件 3 调试kubernetes的资源清单文件 4 开放外部高可用数据入口 ``` **实践解读** 获取基本镜像 ```powershell 获取依赖的基准镜像 [root@kubernetes-master1 ~]# for i in nginx django tomcat do docker pull $i docker tag $i kubernetes-register.superopsmsb.com/superopsmsb/$i docker push kubernetes-register.superopsmsb.com/superopsmsb/$i docker rmi $i done 查看效果 [root@kubernetes-master1 ~]# docker images | egrep 'nginx|djan|tom' | awk '{print $1,$2}' kubernetes-register.superopsmsb.com/superopsmsb/nginx latest kubernetes-register.superopsmsb.com/superopsmsb/tomcat latest kubernetes-register.superopsmsb.com/superopsmsb/django latest ``` nginx镜像基本环境 ```powershell 启动容器 [root@kubernetes-master1 ~]# docker run -d -P --name nginx-test kubernetes-register.superopsmsb.com/superopsmsb/nginx 538acda8cef9d834915b0953fc353d24f922d592d72a26595c0c6604377c6ab9 进入容器查看基本信息 [root@kubernetes-master1 ~]# docker exec -it nginx-test /bin/bash root@538acda8cef9:/# env | egrep 'HOSTNAME|NGINX_VERSION' HOSTNAME=538acda8cef9 NGINX_VERSION=1.23.0 root@538acda8cef9:/# grep root /etc/nginx/conf.d/default.conf root /usr/share/nginx/html; root@fc943556ab2b:/# echo "Hello Nginx, $HOSTNAME-$NGINX_VERSION" > /usr/share/nginx/html/index.html root@fc943556ab2b:/# curl localhost Hello Nginx, fc943556ab2b-1.23.0 移除容器 [root@kubernetes-master1 ~]# docker rm -f nginx-test nginx-test ``` django基本环境 ```powershell 准备外置目录，方便文件留存 [root@kubernetes-master1 ~]# mkdir /data/code -p 启动容器 [root@kubernetes-master1 ~]# docker run -it -p 8000:8000 -v /data/code:/data/code --name django-test kubernetes-register.superopsmsb.com/superopsmsb/django /bin/bash root@9db574d98e95:/# cd /data/code/ 创建项目和应用 root@9db574d98e95:/data/code# django-admin startproject blog root@9db574d98e95:/data/code# cd blog/ root@9db574d98e95:/data/code/blog# python manage.py startapp app1 ``` ```powershell 由于容器内部环境不足，我们在宿主机环境下编辑文件 [root@kubernetes-master1 ~]# cat /data/code/blog/app1/views.py from django.http import HttpResponse import os def index(request): # 获取系统环境变量 hostname=os.environ.get("HOSTNAME") version=os.environ.get("DJANGO_VERSION") # 定制专属的信息输出 message="{}, {}-{}\n".format("Hello Django", hostname, version) return HttpResponse(message) 定制应用的注册效果 [root@kubernetes-master1 ~]# cat /data/code/blog/blog/settings.py ... ALLOWED_HOSTS = ['*'] ... INSTALLED_APPS = [ ... 'app1', ] 定制数据流转效果 [root@kubernetes-master1 ~]# cat /data/code/blog/blog/urls.py ... from app1.views import * urlpatterns = [ url(r'^admin/', admin.site.urls), url(r'', index), ] ``` ```powershell 容器内部启动服务 root@9db574d98e95:/data/code/blog# python manage.py runserver 0.0.0.0:8000 ... Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. 容器外部访问效果 [root@kubernetes-master1 ~]# curl 10.0.0.12:8000 Hello Django, 2894f12ebd0c 1.10.4 移除容器 [root@kubernetes-master1 ~]# docker rm -f django-test django-test ``` tomcat基本环境 ```powershell 启动容器 [root@kubernetes-master1 ~]# docker run -d -P --name tomcat-test kubernetes-register.superopsmsb.com/superopsmsb/tomcat 8190d3afaa448142e573e826593258316e3480e38c575c864ec5c2708d132a33 进入容器查看基本信息 [root@kubernetes-master1 ~]# docker exec -it tomcat-test /bin/bash root@8190d3afaa44:/usr/local/tomcat# env | egrep 'HOSTNAME|TOMCAT_VERSION' HOSTNAME=8190d3afaa44 TOMCAT_VERSION=10.0.22 root@8190d3afaa44:/usr/local/tomcat# mv webapps webapps-bak root@8190d3afaa44:/usr/local/tomcat# mv webapps.dist webapps root@8190d3afaa44:/usr/local/tomcat# echo "Hello Tomcat, $HOSTNAME-$TOMCAT_VERSION" > webapps/ROOT/index.jsp root@8190d3afaa44:/usr/local/tomcat# curl localhost:8080 Hello Tomcat, 8190d3afaa44-10.0.22 移除容器 [root@kubernetes-master1 ~]# docker rm -f tomcat-test tomcat-test ``` **小结** ``` ``` ### 1.3.2 应用环境定制 学习目标 这一节，我们从 Nginx构建、Django构建、Tomcat构建、小结 四个方面来学习。 **Nginx构建** 创建镜像构建文件 ```powershell 创建基准目录 [root@kubernetes-master1 ~]# mkdir /data/images/web/{nginx,tomcat,django} -p ``` 定制Nginx镜像 创建准备文件 ```powershell 准备基准代码文件 [root@kubernetes-master1 ~]# mkdir /data/images/web/nginx/scripts -p 创建服务启动文件 [root@kubernetes-master1 ~]# cat /data/images/web/nginx/scripts/startup.sh #!/bin/bash # 定制容器里面的nginx服务启动脚本 # 定制tomcat的首页内容 echo "Hello Nginx, $HOSTNAME-$NGINX_VERSION" > /usr/share/nginx/html/index.html # 启动nginx nginx -g "daemon off;" ``` ```powershell 定制Dockerfile文件 [root@kubernetes-master1 ~]# cat /data/images/web/nginx/Dockerfile # 构建一个基于nginx的定制镜像 # 基础镜像 FROM kubernetes-register.superopsmsb.com/superopsmsb/nginx # 镜像作者 MAINTAINER shuji@superopsmsb.com # 添加文件 ADD scripts/startup.sh /data/scripts/startup.sh # 执行命令 CMD ["/bin/bash", "/data/scripts/startup.sh"] ``` ```powershell 定制构造镜像 [root@kubernetes-master1 ~]# docker build -t kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 /data/images/web/nginx/ 测试构造镜像 [root@kubernetes-master1 ~]# docker run -d --name nginx-test -p 666:80 kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 58b84726ff29b87e3c8e6a6489e4bead4d298bdd17ac08d90a05a8ad8674906e [root@kubernetes-master1 ~]# curl 10.0.0.12:666 Hello Nginx, 58b84726ff29-1.23.0 [root@kubernetes-master1 ~]# docker rm -f nginx-test nginx-test 提交镜像到远程仓库 [root@kubernetes-master1 ~]# docker push kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 ``` **Django构建** 创建准备文件 ```powershell 准备基准代码文件 [root@kubernetes-master1 ~]# mv /data/code/blog /data/images/web/django/ [root@kubernetes-master1 ~]# mkdir /data/images/web/nginx/scripts -p 创建服务启动文件 [root@kubernetes-master1 ~]# cat /data/images/web/django/scripts/startup.sh #!/bin/bash # 定制容器里面的django服务启动脚本 # 定制服务启动命令 python /data/code/blog/manage.py runserver 0.0.0.0:8000 ``` 定制Dockerfile ```powershell 定制Dockerfile文件 [root@kubernetes-master1 ~]# cat /data/images/web/django/Dockerfile # 构建一个基于django的定制镜像 # 基础镜像 FROM kubernetes-register.superopsmsb.com/superopsmsb/django # 镜像作者 MAINTAINER shuji@superopsmsb.com # 拷贝文件 ADD blog /data/code/blog ADD scripts/startup.sh /data/scripts/startup.sh # 暴露django端口 EXPOSE 8000 # 定制容器的启动命令 CMD ["/bin/bash", "/data/scripts/startup.sh"] ``` ```powershell 定制构造镜像 [root@kubernetes-master1 ~]# docker build -t kubernetes-register.superopsmsb.com/superopsmsb/django_web:v0.1 /data/images/web/django/ 测试构造镜像 [root@kubernetes-master1 ~]# docker run -d --name django-test -p 666:8000 kubernetes-register.superopsmsb.com/superopsmsb/django_web:v0.1 d8f6a1a237f9276917ffc6e233315d02940437f91c64f43894fb3fab8fd50783 [root@kubernetes-master1 ~]# curl 10.0.0.12:666 Hello Django, d8f6a1a237f9 1.10.4 [root@kubernetes-master1 ~]# docker rm -f django-test nginx-test 提交镜像到远程仓库 [root@kubernetes-master1 ~]# docker push kubernetes-register.superopsmsb.com/superopsmsb/django_web:v0.1 ``` **Tomcat构建** 创建准备文件 ```powershell 准备基准代码文件 [root@kubernetes-master1 ~]# mkdir /data/images/web/tomcat/scripts -p 创建服务启动文件 [root@kubernetes-master1 ~]# cat /data/images/web/tomcat/scripts/startup.sh #!/bin/bash # 定制容器里面的tomcat服务启动脚本 # 定制tomcat的首页内容 echo "Hello Tomcat, $HOSTNAME-$TOMCAT_VERSION" > /usr/local/tomcat/webapps/ROOT/index.jsp # 启动tomcat catalina.sh run ``` 定制Dockerfile ```powershell 定制Dockerfile文件 [root@kubernetes-master1 ~]# cat /data/images/web/tomcat/Dockerfile # 构建一个基于tomcat的定制镜像 # 基础镜像 FROM kubernetes-register.superopsmsb.com/superopsmsb/tomcat # 镜像作者 MAINTAINER shuji@superopsmsb.com # 拷贝文件 RUN mv webapps.dist/* webapps/ # 添加文件 ADD scripts/startup.sh /data/scripts/startup.sh # 执行命令 CMD ["/bin/bash", "/data/scripts/startup.sh"] ``` ```powershell 定制构造镜像 [root@kubernetes-master1 ~]# docker build -t kubernetes-register.superopsmsb.com/superopsmsb/tomcat_web:v0.1 /data/images/web/tomcat/ 测试构造镜像 [root@kubernetes-master1 ~]# docker run -d --name tomcat-test -p 666:8080 kubernetes-register.superopsmsb.com/superopsmsb/tomcat_web:v0.1 e46eb26c49ab873351219e98d6e236fd0445aa39edb8edb0bf86560a808614fb [root@kubernetes-master1 ~]# curl 10.0.0.12:666 Hello Tomcat, e46eb26c49ab-10.0.22 [root@kubernetes-master1 ~]# docker rm -f tomcat-test tomcat-test 提交镜像到远程仓库 [root@kubernetes-master1 ~]# docker push kubernetes-register.superopsmsb.com/superopsmsb/tomcat_web:v0.1 ``` **小结** ``` ``` ### 1.3.3 应用环境实践 学习目标 这一节，我们从 资源清单解读、应用环境实践、小结 三个方面来学习。 **资源清单解读** 创建基准配置文件 ```powershell 创建基准目录 [root@kubernetes-master1 ~]# mkdir /data/kubernetes/cluster_test -p ``` nginx入口资源清单 ```powershell [root@kubernetes-master1 ~]# cat /data/kubernetes/cluster_test/01_kubernetes-nginx-proxy.yml apiVersion: apps/v1 kind: Deployment metadata: name: superopsmsb-nginx-proxy labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: kubernetes-register.superopsmsb.com/superopsmsb/nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: superopsmsb-nginx-proxy labels: app: superopsmsb-nginx-proxy spec: type: NodePort selector: app: nginx ports: - protocol: TCP name: http port: 80 targetPort: 80 nodePort: 30080 ``` nginx静态web资源清单 ```powershell [root@kubernetes-master1 ~]# cat /data/kubernetes/cluster_test/02_kubernetes-nginx-web.yml apiVersion: apps/v1 kind: Deployment metadata: name: superopsmsb-nginx-web labels: app: nginx-web spec: replicas: 1 selector: matchLabels: app: nginx-web template: metadata: labels: app: nginx-web spec: containers: - name: nginx image: kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: superopsmsb-nginx-web labels: app: nginx-web spec: type: NodePort selector: app: nginx-web ports: - protocol: TCP name: http port: 80 targetPort: 80 nodePort: 31080 ``` Django动态web资源清单 ```powershell [root@kubernetes-master1 ~]# cat /data/kubernetes/cluster_test/03_kubernetes-django-web.yml apiVersion: apps/v1 kind: Deployment metadata: name: superopsmsb-django-web labels: app: django spec: replicas: 1 selector: matchLabels: app: django template: metadata: labels: app: django spec: containers: - name: django image: kubernetes-register.superopsmsb.com/superopsmsb/django_web:v0.1 ports: - containerPort: 8000 --- apiVersion: v1 kind: Service metadata: name: superopsmsb-django-web labels: app: django-web spec: type: NodePort selector: app: django ports: - protocol: TCP name: http port: 8000 targetPort: 8000 nodePort: 31800 ``` Tomcat动态web资源清单 ```powershell [root@kubernetes-master1 ~]# cat /data/kubernetes/cluster_test/04_kubernetes-tomcat-web.yml apiVersion: apps/v1 kind: Deployment metadata: name: superopsmsb-tomcat-web labels: app: tomcat spec: replicas: 1 selector: matchLabels: app: tomcat template: metadata: labels: app: tomcat spec: containers: - name: django image: kubernetes-register.superopsmsb.com/superopsmsb/tomcat_web:v0.1 ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: superopsmsb-tomcat-web labels: app: tomcat-web spec: type: NodePort selector: app: tomcat ports: - protocol: TCP name: http port: 8080 targetPort: 8080 nodePort: 31880 ``` **应用环境实践** 创建环境应用 ```powershell 应用所有的资源清单文件 [root@kubernetes-master1 ~]# kubectl apply -f /data/kubernetes/cluster_test deployment.apps/superopsmsb-nginx-proxy created service/superopsmsb-nginx-proxy created deployment.apps/superopsmsb-nginx-web created service/superopsmsb-nginx-web created deployment.apps/superopsmsb-django-web created service/superopsmsb-django-web created deployment.apps/superopsmsb-tomcat-web created service/superopsmsb-tomcat-web created ``` ```powershell 检查资源对象 [root@kubernetes-master1 ~]# kubectl get svc,deployment,pod ``` ![image-20220716142721779](../../img/kubernetes/kubernetes_flannel/image-20220716142721779.png) 检查效果 ```powershell [root@kubernetes-master1 ~]# curl 10.0.0.12:31800 Hello Django, superopsmsb-django-web-5bbcb646df-44hcz-1.10.4 [root@kubernetes-master1 ~]# curl 10.0.0.12:31080 Hello Nginx, 00cfe7d5a6e8-1.23.0 [root@kubernetes-master1 ~]# curl 10.0.0.12:31880 Hello Tomcat, 1734569458ff-10.0.22 [root@kubernetes-master1 ~]# curl -I -s 10.0.0.12:30080 | head -n2 HTTP/1.1 200 OK Server: nginx/1.23.0 ``` **小结** ``` ``` ### 1.3.4 应用环境升级 学习目标 这一节，我们从 需求定制、清单升级、小结 三个方面来学习。 **需求定制** 简介 ```powershell 需求解析： 1 nginx需要实现反向代理的功能 2 nginx、tomcat和django的web应用不对外暴露端口 ``` 命令简介 ```powershell 我们需要在nginx-proxy的pod中定制各个反向代理的基本配置，这就需要连接到pod内部进行基本的操作。我们可以借助于 exec命令来实现。 命令格式： kubectl exec -it 资源对象 -- 容器命令 ``` 进入到容器内容进行操作 ```powershell 进入nginx代理容器 [root@kubernetes-master1 ~]# kubectl exec -it superopsmsb-nginx-proxy-7dcd57d844-z89qr -- /bin/bash root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# 安装基本测试环境命令 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# apt update root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# apt install vim net-tools iputils-ping dnsutils curl -y 在pod内部可以看到所有的后端应用的service地址 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# env | grep '_HOST' SUPEROPSMSB_TOMCAT_WEB_SERVICE_HOST=10.111.147.205 SUPEROPSMSB_NGINX_WEB_SERVICE_HOST=10.111.115.222 KUBERNETES_SERVICE_HOST=10.96.0.1 SUPEROPSMSB_NGINX_PROXY_SERVICE_HOST=10.102.253.91 SUPEROPSMSB_DJANGO_WEB_SERVICE_HOST=10.108.200.190 ``` ```powershell kubernetes内部提供了dns服务的能力 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# nslookup superopsmsb-django-web Server: 10.96.0.10 Address: 10.96.0.10#53 Name: superopsmsb-django-web.default.svc.cluster.local Address: 10.108.200.190 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# nslookup superopsmsb-nginx-web Server: 10.96.0.10 Address: 10.96.0.10#53 Name: superopsmsb-nginx-web.default.svc.cluster.local Address: 10.111.115.222 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# nslookup superopsmsb-tomcat-web Server: 10.96.0.10 Address: 10.96.0.10#53 Name: superopsmsb-tomcat-web.default.svc.cluster.local Address: 10.111.147.205 后端服务可以直接通过域名来访问 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# curl superopsmsb-django-web:8000 Hello Django, superopsmsb-django-web-5bbcb646df-44hcz-1.10.4 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# curl superopsmsb-tomcat-web:8080 Hello Tomcat, 1734569458ff-10.0.22 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# curl superopsmsb-nginx-web Hello Nginx, 00cfe7d5a6e8-1.23.0 ``` 定制反向配置 ```powershell 定制nginx的代理配置文件 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# grep -Env '#|^$' /etc/nginx/conf.d/default.conf 1:server { 2: listen 80; 3: listen [::]:80; 4: server_name localhost; 8: location / { 9: proxy_pass http://superopsmsb-nginx-web/; 10: } 11: location /django/ { 12: proxy_pass http://superopsmsb-django-web:8000/; 13: } 14: location /tomcat/ { 15: proxy_pass http://superopsmsb-tomcat-web:8080/; 16: } 21: error_page 500 502 503 504 /50x.html; 22: location = /50x.html { 23: root /usr/share/nginx/html; 24: } 48:} 检测配置文件 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful 重载配置文件 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# nginx -s reload 2022/07/16 07:08:46 [notice] 634#634: signal process started ``` 外部主机测试效果 ```powershell 测试nginx代理的数据跳转效果 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# curl superopsmsb-nginx-proxy Hello Nginx, 00cfe7d5a6e8-1.23.0 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# curl superopsmsb-nginx-proxy/django/ Hello Django, superopsmsb-django-web-5bbcb646df-44hcz-1.10.4 root@superopsmsb-nginx-proxy-7dcd57d844-z89qr:/# curl superopsmsb-nginx-proxy/tomcat/ Hello Tomcat, 1734569458ff-10.0.22 ``` **清单升级** 镜像构造 ```powershell 定制nginx的配置文件 [root@kubernetes-master1 ~]# mkdir /data/images/proxy/nginx/conf -p [root@kubernetes-master1 ~]# vim /data/images/proxy/nginx/conf/default.conf server { listen 80; listen [::]:80; server_name localhost; location / { proxy_pass http://superopsmsb-nginx-web/; } location /django/ { proxy_pass http://superopsmsb-django-web:8000/; } location /tomcat/ { proxy_pass http://superopsmsb-tomcat-web:8080/; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } } ``` ```powershell 定制镜像的Dockerfile文件 [root@kubernetes-master1 ~]# cat /data/images/proxy/nginx/Dockerfile # 构建一个基于nginx的定制镜像 # 基础镜像 FROM kubernetes-register.superopsmsb.com/superopsmsb/nginx # 镜像作者 MAINTAINER shuji@superopsmsb.com # 增加相关文件 ADD conf/default.conf /etc/nginx/conf.d/default.conf ``` ```powershell 镜像的构建 [root@kubernetes-master1 ~]# docker build -t kubernetes-register.superopsmsb.com/superopsmsb/nginx_proxy:v0.1 /data/images/proxy/nginx/ 镜像提交 [root@kubernetes-master1 ~]# docker push register.superopsmsb.com/superopsmsb/nginx_proxy:v0.1 ``` 改造资源清单文件 ```powershell [root@kubernetes-master1 ~]# cat /data/kubernetes/cluster_test/01_kubernetes-nginx-proxy-update.yml apiVersion: apps/v1 kind: Deployment metadata: name: superopsmsb-nginx-proxy labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: kubernetes-register.superopsmsb.com/superopsmsb/nginx_proxy:v0.1 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: superopsmsb-nginx-proxy labels: app: superopsmsb-nginx-proxy spec: type: NodePort selector: app: nginx ports: - protocol: TCP name: http port: 80 targetPort: 80 nodePort: 30080 ``` 测试效果 ```powershell 应用资源清单文件 [root@kubernetes-master1 /data/kubernetes/cluster_test]# kubectl delete -f 01_kubernetes-nginx-proxy.yml [root@kubernetes-master1 /data/kubernetes/cluster_test]# kubectl apply -f 01_kubernetes-nginx-proxy-update.yml ``` ```powershell 测试效果 [root@kubernetes-master1 /data/kubernetes/cluster_test]# curl 10.0.0.12:30080 Hello Nginx, 00cfe7d5a6e8-1.23.0 [root@kubernetes-master1 /data/kubernetes/cluster_test]# curl 10.0.0.12:30080/django/ Hello Django, superopsmsb-django-web-5bbcb646df-44hcz-1.10.4 [root@kubernetes-master1 /data/kubernetes/cluster_test]# curl 10.0.0.12:30080/tomcat/ Hello Tomcat, 1734569458ff-10.0.22 ``` **高可用** 改造keepalived配置 ```powershell 10.0.0.18主机修改配置 [root@kubernetes-ha1 ~]# cat /etc/keepalived/keepalived.conf ... vrrp_instance VI_2 { state BACKUP interface eth0 virtual_router_id 52 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 2222 } track_script { chk_haproxy } virtual_ipaddress { 10.0.0.201 dev eth0 label eth0:2 } } 10.0.0.18重启服务 [root@kubernetes-ha1 ~]# systemctl restart keepalived.service ``` ```powershell 10.0.0.19主机修改配置 [root@kubernetes-ha2 ~]# cat /etc/keepalived/keepalived.conf ... vrrp_instance VI_2 { state MASTER interface eth0 virtual_router_id 52 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 2222 } track_script { chk_haproxy } virtual_ipaddress { 10.0.0.201 dev eth0 label eth0:2 } } 10.0.0.19重启服务 [root@kubernetes-ha2 ~]# systemctl restart keepalived.service ``` ```powershell 查看效果 [root@kubernetes-ha1 ~]# ifconfig eth0:1 eth0:1: flags=4163 mtu 1500 inet 10.0.0.200 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:2d:d9:0a txqueuelen 1000 (Ethernet) [root@kubernetes-ha2 ~]# ifconfig eth0:2 eth0:2: flags=4163 mtu 1500 inet 10.0.0.201 netmask 255.255.255.255 broadcast 0.0.0.0 ether 00:50:56:24:cd:0e txqueuelen 1000 (Ethernet) ``` 改造haproxy配置 - 两台主机的配置内容一样 ```powershell 以10.0.0.18为例，修改配置文件 [root@kubernetes-ha1 ~]# cat /etc/haproxy/haproxy.cfg ... listen kubernetes-nginx-30080 bind 10.0.0.201:80 mode tcp server kubernetes-master1 10.0.0.12:30080 check inter 3s fall 3 rise 5 server kubernetes-master2 10.0.0.13:30080 check inter 3s fall 3 rise 5 server kubernetes-master3 10.0.0.14:30080 check inter 3s fall 3 rise 5 重启haproxy服务 [root@kubernetes-ha1 ~]# systemctl restart haproxy.service ``` 检查效果 ```powershell haproxy检查效果 [root@kubernetes-ha1 ~]# netstat -tnulp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 10.0.0.200:6443 0.0.0.0:* LISTEN 71198/haproxy tcp 0 0 10.0.0.200:9999 0.0.0.0:* LISTEN 71198/haproxy tcp 0 0 10.0.0.201:80 0.0.0.0:* LISTEN 71198/haproxy ``` ```powershell 浏览器访问haproxy的状态页面效果 ``` ![image-20220716153757411](../../img/kubernetes/kubernetes_flannel/image-20220716153757411.png) ```powershell 找一个客户端访问vip [root@kubernetes-master1 ~]# curl 10.0.0.201 Hello Nginx, superopsmsb-nginx-web-757bcb8fc9-lz6fh-1.23.0 [root@kubernetes-master1 ~]# curl 10.0.0.201/tomcat/ Hello Tomcat, superopsmsb-tomcat-web-66c86cb7bc-l2ml2-10.0.22 [root@kubernetes-master1 ~]# curl 10.0.0.201/django/ Hello Django, superopsmsb-django-web-5bbcb646df-xm8jf-1.10.4 ``` # 1 网络管理 ## 1.1Service ### 1.1.1 网络体系 学习目标 这一节，我们从 应用流程、细节解读、小结 三个方面来学习。 **应用流程** 资源对象体系 ![image-20220715012035435](../../img/kubernetes/kubernetes_flannel/image-20220715012035435.png) 简介 ```powershell 通过对Pod及其管理资源RC和Deployment的实践，我们知道，我们所有的应用服务都是工作在pod资源中，由于每个Pod都有独立的ip地址，而大量的动态创建和销毁操作后，虽然pod资源的数量是控制住了，但是由于pod重新启动，导致他的IP很有可能发生了变化，假设我们这里有前段应用的pod和后端应用的pod，那么再剧烈变动的场景中，两个应用该如何自由通信呢？难道是让我们以类似nginx负载均衡的方式手工定制pod ip然后进行一一管理么？但是这是做不到的。 Kubernetes集群就为我们提供了这样的一个对象--Service，它定义了一组Pod的逻辑集合和一个用于访问它们的策略，它可以基于标签的方式自动找到对应的pod应用，而无需关心pod的ip地址变化与否，从而实现了类似负载均衡的效果. 这个资源在master端的Controller组件中，由Service Controller 来进行统一管理。 ``` service ```powershell service是Kubernetes里最核心的资源对象之一，每一个Service都是一个完整的业务服务，我们之前学到的Pod、RC、Deployment等资源对象都是为Service服务的。他们之间的关系如下图： ``` ![image-20210922135912555](../../img/kubernetes/kubernetes_flannel/image-20210922135912555.png) 解析 ```powershell Kubernetes 的 Service定义了一个服务的访问入口地址，前端的应用Pod通过Service访问其背后一组有Pod副本组成的集群示例，Service通过Label Selector访问指定的后端Pod，RC保证Service的服务能力和服务质量处于预期状态。 Service是Kubernetes中最高一级的抽象资源对象，每个Service提供一个独立的服务，集群Service彼此间使用TCP/IP进行通信，将不同的服务组合在一起运行起来，就行了我们所谓的"系统"，效果如下图 ``` ![image-20220721090648605](../../img/kubernetes/kubernetes_flannel/image-20220721090648605.png) **细节解读** Pod入口 ```powershell 我们知道每个Pod都有一个专用的IP地址，加上Pod内部容器的Port端口，就组成了一个访问Pod专用的EndPoint(Pod IP+Container Port)，从而实现了用户外部资源访问Pod内部应用的效果。这个EndPoint资源在master端的Controller组件中，由EndPoint Controller 来进行统一管理。 ``` kube-proxy ```powershell Pod是工作在不同的Node节点上，而Node节点上有一个kube-proxy组件，它本身就是一个软件负载均衡器，在内部有一套专有的负载均衡与会话保持机制，可以达到，接收到所有对Service请求，进而转发到后端的某个具体的Pod实例上，相应该请求。 -- kube-proxy 其实就是 Service Controller位于各节点上的agent。 ``` service表现 ```powershell Kubernetes给Service分配一个全局唯一的虚拟ip地址--cluster IP，它不存在任何网络设备上，Service通过内容的标签选择器，指定相应该Service的Pod资源，这样以来，请求发给cluster IP，后端的Pod资源收到请求后，就会响应请求。 这种情况下，每个Service都有一个全局唯一通信地址，整个系统的内部服务间调用就变成了最基础的TCP/IP网络通信问题。如果我们的集群内部的服务想要和外部的网络进行通信，方法很多，比如： NodePort类型，通过在所有结点上增加一个对外的端口，用于接入集群外部请求 ingress类型，通过集群附加服务功能，将外部的域名流量转交到集群内部。 ``` service vs endpoint ```powershell 1 当创建 Service资源的时候，最重要的就是为Service指定能够提供服务的标签选择器， 2 Service Controller就会根据标签选择器创建一个同名的Endpoint资源对象。 3 Endpoint Controller开始介入，使用Endpoint的标签选择器(继承自Service标签选择器)，筛选符合条件的pod资源 4 Endpoint Controller 将符合要求的pod资源绑定到Endpoint上，并告知给Service资源，谁可以正常提供服务。 5 Service 根据自身的cluster IP向外提供由Endpoint提供的服务资源。 -- 所以Service 其实就是 为动态的一组pod资源对象 提供一个固定的访问入口。 ``` **小结** ``` ``` ### 1.1.2 工作模型 学习目标 这一节，我们从 模型解读、类型解读、小结 三个方面来学习。 **模型解读** 简介 ```powershell Service对象，对于当前集群的节点来说，本质上就是工作节点的一些iptables或ipvs规则，这些规则由kube-proxy进行实时维护，站在kubernetes的发展脉络上来说，kube-proxy将请求代理至相应端点的方式有三种：userspace/iptables/ipvs。目前我们主要用的是 iptables/ipvs 两种。 ``` 模式解析 ![image-20220721093742953](../../img/kubernetes/kubernetes_flannel/image-20220721093742953.png) ```powershell userspace模型是k8s(1.1-1.2)最早的一种工作模型，作用就是将service的策略转换成iptables规则，这些规则仅仅做请求的拦截，而不对请求进行调度处理。 该模型中，请求流量到达内核空间后，由套接字送往用户空间的kube-proxy，再由它送回内核空间，并调度至后端Pod。因为涉及到来回转发，效率不高，另外用户空间的转发，默认开启了会话粘滞，会导致流量转发给无效的pod上。 ``` ```powershell iptables模式是k8s(1.2-至今)默认的一种模式，作用是将service的策略转换成iptables规则，不仅仅包括拦截，还包括调度，捕获到达ClusterIP和Port的流量，并重定向至当前Service的代理的后端Pod资源。性能比userspace更加高效和可靠 缺点： 不会在后端Pod无响应时自动重定向，而userspace可以 中量级k8s集群(service有几百个)能够承受，但是大量级k8s集群(service有几千个)维护达几万条规则，难度较大 ``` ```powershell ipvs是自1.8版本引入，1.11版本起为默认设置，通过内核的Netlink接口创建相应的ipvs规则 请求流量的转发和调度功能由ipvs实现，余下的其他功能仍由iptables完成。ipvs流量转发速度快，规则同步性能好，且支持众多调度算法，如rr/lc/dh/sh/sed/nq等。 注意： 对于我们kubeadm方式安装k8s集群来说，他会首先检测当前主机上是否已经包含了ipvs模块，如果加载了，就直接用ipvs模式，如果没有加载ipvs模块的话，会自动使用iptables模式。 ``` **类型解读** service类型 ```powershell 对于k8s来说，内部服务的自由通信可以满足我们环境的稳定运行，但是我们作为一个平台，其核心功能还是将平台内部的服务发布到外部环境，那么在k8s环境平台上，Service主要有四种样式来满足我们的需求，种类如下： ``` ```powershell ClusterIP 这是service默认的服务暴露模式，主要针对的对象是集群内部。 NodePort 在ClusterIP的基础上，以:方式对外提供服务，默认端口范围沿用Docker初期的随机端口范围 30000~32767，但是NodePort设定的时候，会在集群所有节点上实现相同的端口。 LoadBalancer 基于NodePort之上，使用运营商负载均衡器方式实现对外提供服务底层是基于IaaS云创建一个k8s云，同时该平台也支持LBaaS产品服务。 ExternalName 当前k8s集群依赖集群外部的服务，那么通过externalName将外部主机引入到k8s集群内部外部主机名以 DNS方式解析为一个 CNAME记录给k8s集群的其他主机来使用这种Service既不会有ClusterIP，也不会有NodePort.而且依赖于内部的CoreDNS功能 ``` **小结** ``` ``` ### 1.1.3 SVC实践 学习目标 这一节，我们从 资源实践、NodePort实践、小结 三个方面来学习。 **资源实践** 资源属性 ```powershell apiVersion: v1 kind: Service metadata: name: … namespace: … labels: key1: value1 key2: value2 spec: type # Service类型，默认为ClusterIP selector # 等值类型的标签选择器，内含“与”逻辑 ports： # Service的端口对象列表 - name # 端口名称 protocol # 协议，目前仅支持TCP、UDP和SCTP，默认为TCP port # Service的端口号 targetPort # 后端目标进程的端口号或名称，名称需由Pod规范定义 nodePort # 节点端口号，仅适用于NodePort和LoadBalancer类型 clusterIP # Service的集群IP，建议由系统自动分配 externalTrafficPolicy # 外部流量策略处理方式，Local表示由当前节点处理，Cluster表示向集群范围调度 loadBalancerIP # 外部负载均衡器使用的IP地址，仅适用于LoadBlancer externalName # 外部服务名称，该名称将作为Service的DNS CNAME值 ``` 手工方法 ```powershell 创建一个应用 [root@kubernetes-master ~]# kubectl create deployment nginx --image=kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 创建多种类型svc [root@kubernetes-master ~]# kubectl expose deployment nginx --port=80 [root@kubernetes-master ~]# kubectl expose deployment nginx --name=svc-default --port=80 [root@kubernetes-master ~]# kubectl expose deployment nginx --name=svc-nodeport --port=80 --type=NodePort [root@kubernetes-master ~]# kubectl expose deployment nginx --name=svc-loadblancer --port=80 --type=LoadBalancer 查看效果 [root@kubernetes-master1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx ClusterIP 10.99.57.240 80/TCP 4m31s svc-default ClusterIP 10.103.54.139 80/TCP 3m2s svc-loadblancer LoadBalancer 10.104.39.177 80:30778/TCP 2m44s svc-nodeport NodePort 10.100.104.140 80:32335/TCP 2m54s ``` 资源对象方式 ```powershell 查看对象标签 [root@kubernetes-master1 ~]# kubectl get pod --show-labels NAME READY STATUS RESTARTS AGE LABELS nginx-f44f65dc-pbht6 1/1 Running 0 8h app=nginx,pod-template-hash=f44f65dc ``` 简单实践 ```powershell 定制资源清单文件 [root@kubernetes-master1 ~]# mkdir /data/kubernetes/service -p ; cd /data/kubernetes/service [root@kubernetes-master1 /data/kubernetes/service]# [root@kubernetes-master1 /data/kubernetes/service]# vim 01_kubernetes-service_test.yml apiVersion: v1 kind: Service metadata: name: superopsmsb-nginx-service spec: selector: app: nginx ports: - name: http port: 80 应用资源清单文件 [root@kubernetes-master1 /data/kubernetes/service]# kubectl apply -f 01_kubernetes-service_test.yml service/superopsmsb-nginx-service created ``` ```powershell 查看service效果 [root@kubernetes-master1 /data/kubernetes/service]# kubectl describe svc superopsmsb-nginx-service Name: superopsmsb-nginx-service ... IP: 10.109.240.56 IPs: 10.109.240.56 Port: http 8000/TCP TargetPort: 8000/TCP Endpoints: 10.244.3.63:8000 ... 访问service [root@kubernetes-master1 /data/kubernetes/service]# curl -s 10.101.25.116 -I | head -n1 HTTP/1.1 200 OK ``` **NodePort实践** 属性解读 ```powershell NodePort会在所有的节点主机上，暴露一个指定或者随机的端口，供外部的服务能够正常的访问pod内部的资源。 ``` 简单实践 ```powershell [root@kubernetes-master1 /data/kubernetes/service]# vim 02_kubernetes-service_nodePort.yml apiVersion: v1 kind: Service metadata: name: superopsmsb-nginx-nodeport spec: selector: app: nginx ports: - name: http port: 80 nodePort: 30080 应用资源清单文件 [root@kubernetes-master1 /data/kubernetes/service]# kubectl apply -f 02_kubernetes-service_nodePort.yml service/superopsmsb-nginx-nodeport created ``` ```powershell 检查效果 [root@kubernetes-master1 /data/kubernetes/service]# kubectl describe svc superopsmsb-nginx-nodeport Name: superopsmsb-nginx-nodeport ... Type: NodePort IP Family Policy: SingleStack IP Families: IPv4 IP: 10.102.1.177 IPs: 10.102.1.177 Port: http 80/TCP TargetPort: 80/TCP NodePort: http 30080/TCP ... 访问效果 [root@kubernetes-master1 /data/kubernetes/service]# curl 10.0.0.12:30080 Hello Nginx, nginx-6944855df5-8zjdn-1.23.0 ``` **小结** ``` ``` ### 1.1.4 IPVS实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ![image-20220721093742953](../../img/kubernetes/kubernetes_flannel/image-20220721093742953.png) ```powershell 关键点： ipvs会在每个节点上创建一个名为kube-ipvs0的虚拟接口，并将集群所有Service对象的ClusterIP和ExternalIP都配置在该接口； - 所以每增加一个ClusterIP 或者 EternalIP，就相当于为 kube-ipvs0 关联了一个地址罢了。 kube-proxy为每个service生成一个虚拟服务器( IPVS Virtual Server)的定义。 ``` ```powershell 基本流程： 所以当前节点接收到外部流量后，如果该数据包是交给当前节点上的clusterIP，则会直接将数据包交给kube-ipvs0，而这个接口是内核虚拟出来的，而kube-proxy定义的VS直接关联到kube-ipvs0上。 如果是本地节点pod发送的请求，基本上属于本地通信，效率是非常高的。 默认情况下，这里的ipvs使用的是nat转发模型，而且支持更多的后端调度算法。仅仅在涉及到源地址转换的场景中，会涉及到极少量的iptables规则(应该不会超过20条) ``` ```powershell 前提：当前操作系统需要提前加载ipvs模块 yum install ipvsadm -y ``` kube-proxy ```powershell 对于k8s来说，默认情况下，支持的规则是 iptables，我们可以通过多种方式对我们的代理模式进行更改，因为这些规则都是基于kube-proxy来定制的，所以，我们如果要更改代理模式的话，就需要调整kube-proxy的属性。 ``` ```powershell 在k8s集群中，关于kube-proxy的所有属性信息，我们可以通过一个 configmap 的资源对象来了解一下 [root@kubernetes-master1 ~]# kubectl describe configmap kube-proxy -n kube-system Name: kube-proxy Namespace: kube-system Labels: app=kube-proxy ... iptables: masqueradeAll: false # 这个属性打开的话，会对所有的请求都进行源地址转换 ... ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" # 调度算法，默认是randomrobin ... kind: KubeProxyConfiguration metricsBindAddress: "" mode: "" # 默认没有指定，就是使用 iptables 规则 ... ``` 查看默认模式 ```powershell 通过kube-proxy-b8dpc的pod日志查看模式 [root@kubernetes-master1 /data/kubernetes/service]# kubectl logs kube-proxy-b8dpc -n kube-system ... I0719 08:39:44.410078 1 server_others.go:561] "Unknown proxy mode, assuming iptables proxy" proxyMode="" I0719 08:39:44.462438 1 server_others.go:206] "Using iptables Proxier" ... ``` **简单实践** 准备 ```powershell 清理所有svc [root@kubernetes-master1 /data/kubernetes/service]# for i in $(kubectl get svc | egrep -v 'NAME|kubernetes' | awk '{print $1}') > do > kubectl delete svc $i > done ``` 修改kube-proxy模式 ```powershell 我们在测试环境中，临时修改一下configmap中proxy的基本属性 - 临时环境推荐 [root@kubernetes-master1 /data/kubernetes/service]# kubectl edit configmap kube-proxy -n kube-system ... mode: "ipvs" ... 重启所有的kube-proxy pod对象 [root@kubernetes-master1 /data/kubernetes/service]# kubectl delete pod -n kube-system -l k8s-app=kube-proxy ``` ```powershell 通过kube-proxy的pod日志查看模式 [root@kubernetes-master1 /data/kubernetes/service]# kubectl logs kube-proxy-fb9pz -n kube-system ... I0721 10:31:40.408116 1 server_others.go:269] "Using ipvs Proxier" I0721 10:31:40.408155 1 server_others.go:271] "Creating dualStackProxier for ipvs" ... ``` 测试效果 ```powershell 安装参考命令 [root@kubernetes-master1 /data/kubernetes/service]# yum install ipvsadm -y 查看规则效果 [root@kubernetes-master1 /data/kubernetes/service]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.17.0.1:30443 rr -> 10.244.3.2:8443 Masq 1 0 0 ... ``` ```powershell 创建一个service [root@kubernetes-master1 /data/kubernetes/service]# kubectl apply -f 02_kubernetes-service_nodePort.yml service/superopsmsb-nginx-nodeport created 查看svc的ip [root@kubernetes-master1 /data/kubernetes/service]# kubectl get svc superopsmsb-nginx-nodeport -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR superopsmsb-nginx-nodeport NodePort 10.106.138.242 80:30080/TCP 23s app=nginx 查看ipvsadm规则 [root@kubernetes-master1 /data/kubernetes/service]# ipvsadm -Ln | grep -A1 10.106.138.242 TCP 10.106.138.242:80 rr -> 10.244.3.64:80 Masq 1 0 0 查看防火墙规则 [root@kubernetes-master1 /data/kubernetes/service]# iptables -t nat -S KUBE-NODE-PORT -N KUBE-NODE-PORT -A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ 结果显示： 没有生成对应的防火墙规则 ``` **小结** ``` ``` ## 1.2 其他资源 ### 1.2.1 域名服务 学习目标 这一节，我们从 场景需求、域名测试、小结 三个方面来学习。 **场景需求** 简介 ```powershell 在传统的系统部署中，服务运行在一个固定的已知的 IP 和端口上，如果一个服务需要调用另外一个服务，可以通过地址直接调用，但是，在虚拟化或容器话的环境中，以我们的k8s集群为例，如果存在个位数个service我们可以很快的找到对应的clusterip地址，进而找到指定的资源，虽然ip地址不容易记住，因为service在创建的时候会为每个clusterip分配一个名称，我们同样可以根据这个名称找到对应的服务。但是，如果我们的集群中有1000个Service，我们如何找到指定的service呢？ ``` ```powershell 虽然我们可以借助于传统的DNS机制来实现，但是在k8s集群中，服务实例的启动和销毁是很频繁的，服务地址在动态的变化，所以传统的方式配置DNS解析记录就不太好实现了。所以针对于这种场景，我们如果需要将请求发送到动态变化的服务实例上，可以通过一下两个步骤来实现： 服务注册 — 创建服务实例后，主动将当前服务实例的信息，存储到一个集中式的服务管理中心。 服务发现 — 当A服务需要找未知的B服务时，先去服务管理中心查找B服务地址，然后根据该地址找到B服务 ``` DNS方案 ```powershell 专用于kubernetes集群中的服务注册和发现的解决方案就是KubeDNS。kubeDNS自从k8s诞生以来，其方案的具体实现样式前后经历了三代，分别是 SkyDNS、KubeDNS、CoreDNS(目前默认的)。 ``` ![image-20220721185041526](../../img/kubernetes/kubernetes_flannel/image-20220721185041526.png) 域名解析记录 ```powershell Kubelet会为创建的每一个容器于/etc/resolv.conf配置文件中生成DNS查询客户端依赖到的必要配置，相关的配置信息源自于kubelet的配置参数，容器的DNS服务器由clusterDNS参数的值设定，它的取值为kube-system名称空间中的Service对象kube-dns的ClusterIP，默认为10.96.0.10. DNS搜索域的值由clusterDomain参数的值设定，若部署Kubernetes集群时未特别指定，其值将为cluster.local、svc.cluster.local和NAMESPACENAME.svc.cluster.local ``` ```powershell kubeadm 1.23.8 环境初始化配置文件中与dns相关的检索信息 [root@kubernetes-master1 ~]# grep -A1 networking /data/kubernetes/cluster_init/kubeadm_init_1.23.8.yml networking: dnsDomain: cluster.local ``` 资源对象的dns记录 ```powershell 对于kubernetes的内部资源对象来说，为了更好的绕过变化频率更高的ip地址的限制，它可以在内部以dns记录的方式进行对象发现，dns记录具有标准的名称格式： 资源对象名.命名空间名.svc.cluster.local ``` **域名测试** 创建一个svc记录 ```pow [root@kubernetes-master1 /data/kubernetes/service]# kubectl apply -f 01_kubernetes-service_test.yml service/superopsmsb-nginx-service created [root@kubernetes-master1 /data/kubernetes/service]# kubectl get svc superopsmsb-nginx-service -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR superopsmsb-nginx-service ClusterIP 10.97.135.126 80/TCP 23s app=nginx ``` 资源对象的名称记录 ```powershell 查看本地的pod效果 [root@kubernetes-master1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-6944855df5-8zjdn 1/1 Running 0 38m 查看pod内部的resolv.conf文件 [root@kubernetes-master1 ~]# kubectl exec -it nginx-6944855df5-8zjdn -- cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local localhost options ndots:5 可以看到： 资源对象的查看dns的后缀主要有四种： default.svc.cluster.local svc.cluster.local cluster.local localhost ``` ```powershell 查看内部的资源对象完整域名 [root@kubernetes-master1 /data/kubernetes/service]# kubectl exec -it nginx-6944855df5-8zjdn -- /bin/bash root@nginx-6944855df5-8zjdn:/# curl curl: try 'curl --help' or 'curl --manual' for more information root@nginx-6944855df5-8zjdn:/# curl superopsmsb-nginx-service Hello Nginx, nginx-6944855df5-8zjdn-1.23.0 root@nginx-6944855df5-8zjdn:/# curl superopsmsb-nginx-service.default.svc.cluster.local Hello Nginx, nginx-6944855df5-8zjdn-1.23.0 root@nginx-6944855df5-8zjdn:/# curl superopsmsb-nginx-service.default.svc.cluster.local. Hello Nginx, nginx-6944855df5-8zjdn-1.23.0 ``` 内部dns测试效果 ```powershell 安装dns测试工具 root@nginx-6944855df5-8zjdn:/# apt update root@nginx-6944855df5-8zjdn:/# apt install dnsutils -y ``` ```powershell 资源对象的查看效果 root@nginx-6944855df5-8zjdn:/# nslookup superopsmsb-nginx-service Server: 10.96.0.10 Address: 10.96.0.10#53 Name: superopsmsb-nginx-service.default.svc.cluster.local Address: 10.97.135.126 root@nginx-6944855df5-8zjdn:/# nslookup 10.97.135.126 126.135.97.10.in-addr.arpa name = superopsmsb-nginx-service.default.svc.cluster.local. root@nginx-6944855df5-8zjdn:/# nslookup kubernetes Server: 10.96.0.10 Address: 10.96.0.10#53 Name: kubernetes.default.svc.cluster.local Address: 10.96.0.1 ``` ```powershell 查看跨命名空间的资源对象 [root@kubernetes-master1 ~]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 2d2h 回到pod终端查看效果 root@nginx-6944855df5-8zjdn:/# nslookup kube-dns.kube-system.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: kube-dns.kube-system.svc.cluster.local Address: 10.96.0.10 root@nginx-6944855df5-8zjdn:/# nslookup 10.96.0.10 10.0.96.10.in-addr.arpa name = kube-dns.kube-system.svc.cluster.local. root@nginx-6944855df5-8zjdn:/# nslookup kube-dns Server: 10.96.0.10 Address: 10.96.0.10#53 ** server can't find kube-dns: NXDOMAIN 结果显示： 对于跨命名空间的资源对象必须使用完整的名称格式 ``` pod资源对象解析 ```powershell 查看pod资源对象 [root@kubernetes-master1 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP ... nginx-6944855df5-8zjdn 1/1 Running 0 56m 10.244.3.64 ... 构造资源对象名，pod的ip名称转换 root@nginx-6944855df5-8zjdn:/# nslookup 10-244-3-64.superopsmsb-nginx-service.default.svc.cluster.local. Server: 10.96.0.10 Address: 10.96.0.10#53 Name: 10-244-3-64.superopsmsb-nginx-service.default.svc.cluster.local Address: 10.244.3.64 ``` **小结** ``` ``` ### 1.2.2 CoreDNS 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell coredns是一个用go语言编写的开源的DNS服务，coredns是首批加入CNCF组织的云原生开源项目，并且作为已经在CNCF毕业的项目，coredns还是目前kubernetes中默认的dns服务。同时，由于coredns可以集成插件，它还能够实现服务发现的功能。 coredns和其他的诸如bind、knot、powerdns、unbound等DNS服务不同的是：coredns非常的灵活，并且几乎把所有的核心功能实现都外包给了插件。如果你想要在coredns中加入Prometheus的监控支持，那么只需要安装对应的prometheus插件并且启用即可。 ``` 配置解析 ```powershell coredns的配置依然是存放在 configmap中 [root@kubernetes-master1 ~]# kubectl get cm coredns -n kube-system NAME DATA AGE coredns 1 2d2h ``` ```powershell 查看配置详情 [root@kubernetes-master1 ~]# kubectl describe cm coredns -n kube-system Name: coredns Namespace: kube-system Labels: Annotations: Data ==== Corefile: ---- .:53 { errors health { # 健康检测 lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { # 解析配置 pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { # 转发配置 max_concurrent 1000 } cache 30 loop reload # 自动加载 loadbalance } BinaryData ==== Events: ``` ```powershell 其他属性和示例： except domain 排除的域名 添加dns解析 hosts { 192.168.8.100 www.example.com fallthrough # 在CoreDNS里面表示如果自己无法处理，则交由下个插件处理。 } ``` **简单实践** 修改实践 ```powershell 修改配置文件 [root@kubernetes-master1 ~]# kubectl edit cm coredns -n kube-system ... forward . /etc/resolv.conf { max_concurrent 1000 except www.baidu.com. } hosts { 10.0.0.20 harbor.superopsmsb.com fallthrough } ... 注意： 多个dns地址间用空格隔开 排除的域名最好在末尾添加 “.”，对于之前的旧版本来说可能会出现无法保存的现象 ``` 测试效果 ```powershell 同步dns的配置信息 [root@kubernetes-master1 ~]# kubectl delete pod -l k8s-app=kube-dns -n kube-system pod "coredns-5d555c984-fmrht" deleted pod "coredns-5d555c984-scvjh" deleted 删除旧pod，使用新pod测试 （可以忽略） [root@kubernetes-master1 /data/kubernetes/service]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-6944855df5-8zjdn 1/1 Running 0 70m [root@kubernetes-master1 /data/kubernetes/service]# kubectl delete pod nginx-6944855df5-8zjdn pod "nginx-6944855df5-8zjdn" deleted ``` ```powershell pod中测试效果 [root@kubernetes-master1 /data/kubernetes/service]# kubectl exec -it nginx-6944855df5-9s4bd -- /bin/bash root@nginx-6944855df5-9s4bd:/# apt update ; apt install dnsutils -y 测试效果 root@nginx-6944855df5-9s4bd:/# nslookup www.baidu.com Server: 10.96.0.10 Address: 10.96.0.10#53 ** server can't find www.baidu.com: SERVFAIL root@nginx-6944855df5-9s4bd:/# nslookup harbor.superopsmsb.com Server: 10.96.0.10 Address: 10.96.0.10#53 Name: harbor.superopsmsb.com Address: 10.0.0.20 ``` 清理环境 ```powershell 将之前的配置清理掉 ``` ``` ``` ### 1.2.3 无头服务 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 在kubernetes中，有一种特殊的无头service，它本身是Service，但是没有ClusterIP，这种svc在一些有状态的场景中非常重要。 ``` ```powershell 无头服务场景下，k8s会将一个集群内部的所有成员提供唯一的DNS域名来作为每个成员的网络标识，集群内部成员之间使用域名通信，这个时候，就特别依赖service的selector属性配置了。 无头服务管理的域名是如下的格式：$(service_name).$(k8s_namespace).svc.cluster.local。 ``` pod资源对象的解析记录 ```powershell dns解析记录 A记录 ---...svc. A PodIP 关键点： svc_name的解析结果从常规Service的ClusterIP，转为各个Pod的IP地址； 反解，则从常规的clusterip解析为service name，转为从podip到hostname, ---...svc. 指的是a-b-c-d格式，而非Pod自己的主机名； ``` **简单实践** 准备工作 ```powershell 扩展pod对象数量 [root@kubernetes-master1 ~]# kubectl scale deployment nginx --replicas=3 deployment.apps/nginx scaled [root@kubernetes-master1 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP ... nginx-6944855df5-9m44b 1/1 Running 0 5s 10.244.2.4 ... nginx-6944855df5-9s4bd 1/1 Running 0 14m 10.244.3.2 kubernetes-node3 nginx-6944855df5-mswl4 1/1 Running 0 5s 10.244.1.3 kubernetes-node2 ``` 定制无头服务 ```powershell 手工定制无头服务 [root@kubernetes-master1 ~]# kubectl create service clusterip service-headless --clusterip="None" service/service-headless-cmd created 查看无头服务 [root@kubernetes-master1 ~]# kubectl get svc service-headless NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service-headless ClusterIP None 6s ``` ```powershell 资源清单文件定制无头服务 [root@kubernetes-master1 /data/kubernetes/service]# cat 03_kubernetes-service_headless.yml apiVersion: v1 kind: Service metadata: name: superopsmsb-nginx-headless spec: selector: app: nginx ports: - name: http port: 80 clusterIP: "None" 应用资源对象 [root@kubernetes-master1 /data/kubernetes/service]# kubectl apply -f 03_kubernetes-service_headless.yml service/superopsmsb-nginx-headless created [root@kubernetes-master1 /data/kubernetes/service]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 6m33s service-headless ClusterIP None 61s superopsmsb-nginx-headless ClusterIP None 80/TCP 7s ``` 测试效果 ```powershell 进入测试容器 [root@kubernetes-master1 /data/kubernetes/service]# kubectl exec -it nginx-fd669dcb-h5d9d -- /bin/bash root@nginx-fd669dcb-h5d9d:/# nslookup superopsmsb-nginx-headless Server: 10.96.0.10 Address: 10.96.0.10#53 Name: superopsmsb-nginx-headless.default.svc.cluster.local Address: 10.244.2.4 Name: superopsmsb-nginx-headless.default.svc.cluster.local Address: 10.244.1.3 Name: superopsmsb-nginx-headless.default.svc.cluster.local Address: 10.244.3.2 结果显式： 由于我们没有对域名做定向解析，那么找请求的时候，就像无头苍蝇似的，到处乱串 ``` **小结** ``` ``` ## 1.3 flannel方案 ### 1.3.1 网络方案 学习目标 这一节，我们从 存储原理、方案解读、小结 三个方面来学习。 **存储原理** 容器实现网络访问样式 ![1633650192071](../../img/kubernetes/kubernetes_flannel/1633650192071.png) ```powershell 1 虚拟网桥： brdige，用纯软件的方式实现一个虚拟网络，用一个虚拟网卡接入到我们虚拟网桥上去。这样就能保证每一个容器和每一个pod都能有一个专用的网络接口，从而实现每一主机组件有网络接口。 2 多路复用： MacVLAN，借助于linux内核级的VLAN模块，在同一个物理网卡上配置多个 MAC 地址，每个 MAC 给一个Pod使用，然后借助物理网卡中的MacVLAN机制进行跨节点之间进行通信了。 3 硬件交换： 很多网卡都已经支持"单根IOV的虚拟化"了，借助于单根IOV（SR-IOV）的方式，直接在物理主机虚拟出多个接口来，通信性能很高，然后每个虚拟网卡能够分配给容器使用。 ``` 容器网络方案 ```powershell 任何一种能够让容器自由通信的网络解决方案，必须包含三个功能： 1 构建一个网络 2 将容器接入到这个网络中 3 实时维护所有节点上的路由信息，实现容器的通信 ``` CNI方案 ```powershell 1 所有节点的内核都启用了VXLAN的功能模块 每个节点都启动一个cni网卡，并维护所有节点所在的网段的路由列表 2 node上的pod发出请求到达cni0 根据内核的路由列表判断对端网段的节点位置 经由 隧道设备 对数据包进行封装标识，对端节点的隧道设备解封标识数据包， 当前数据包一看当前节点的路由表发现有自身的ip地址，这直接交给本地的pod 3 多个节点上的路由表信息维护，就是各种网络解决方案的工作位置 注意： 我们可以部署多个网络解决方案，但是对于CNI来说，只会有一个生效。如果网络信息过多会导致冲突 ``` ![1633651873150](../../img/kubernetes/kubernetes_flannel/1633651873150.png) **方案解读** CNI插件 ```powershell 根据我们刚才对pod通信的回顾，多节点内的pod通信，k8s是通过CNI接口来实现网络通信的。CNI基本思想：创建容器时，先创建好网络名称空间，然后调用CNI插件配置这个网络，而后启动容器内的进程 CNI插件类别：main、meta、ipam main，实现某种特定的网络功能，如loopback、bridge、macvlan、ipvlan meta，自身不提供任何网络实现，而是调用其他插件，如flannel ipam，仅用于分配IP地址，不提供网络实现 ``` 常见方案 ```powershell Flannel 提供叠加网络，基于linux TUN/TAP，使用UDP封装IP报文来创建叠加网络，并借助etcd维护网络分配情况 Calico 基于BGP的三层网络，支持网络策略实现网络的访问控制。在每台机器上运行一个vRouter，利用内核转发数据包，并借助iptables实现防火墙等功能 kube-router K8s网络一体化解决方案，可取代kube-proxy实现基于ipvs的Service，支持网络策略、完美兼容BGP的高级特性 其他网络解决方案： Canal: 由Flannel和Calico联合发布的一个统一网络插件，支持网络策略 Weave Net: 多主机容器的网络方案，支持去中心化的控制平面 Contiv:思科方案，直接提供多租户网络，支持L2(VLAN)、L3(BGP)、Overlay(VXLAN) 更多的解决方案，大家可以参考： https://kubernetes.io/docs/concepts/cluster-administration/addons/ ``` **小结** ``` ``` ### 1.3.2 flannel 学习目标 这一节，我们从 信息查看、原理解读、小结 三个方面来学习。 **信息查看** 查看网络配置 ```powershell [root@kubernetes-master1 ~]# cat /etc/kubernetes/manifests/kube-controller-manager.yaml apiVersion: v1 ... spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true ... - --cluster-cidr=10.244.0.0/16 配置解析： allocate-node-cidrs属性表示，每增加一个新的节点，都从cluster-cidr子网中切分一个新的子网网段分配给对应的节点上。 这些相关的网络状态属性信息，会经过 kube-apiserver 存储到etcd中。 ``` CNI配置 ```powershell 使用CNI插件编排网络，Pod初始化或删除时，kubelet会调用默认CNI插件，创建虚拟设备接口附加到相关的底层网络，设置IP、路由并映射到Pod对象网络名称空间. kubelet在/etc/cni/net.d目录查找cni json配置文件，基于type属性到/opt/cni/bin中查找相关插件的二进制文件，然后调用相应插件设置网络 ``` 网段配置 ```powershell 查看网段配置 [root@kubernetes-master1 ~]# cat /run/flannel/subnet.env FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.0.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true ``` 网段分配原理 ```powershell 分配原理解读 集群的 kube-controller-manager 负责控制每个节点的网段分配 集群的 etcd 负责存储所有节点的网络配置存储 集群的 flannel 负责各个节点的路由表定制及其数据包的拆分和封装 -- 所以flannel各个节点是平等的，仅负责数据平面的操作。网络功能相对来说比较简单。 另外一种插件 calico相对于flannel来说，多了一个控制节点，来管控所有的网络节点的服务进程。 ``` flannel网络 ```powershell pod效果 [root@kubernetes-master1 ~]# kubectl get pod -n kube-flannel -o wide NAME READY STATUS ... IP NODE ... kube-flannel-ds-b6hxm 1/1 Running ... 10.0.0.15 kubernetes-node1 ... kube-flannel-ds-bx7rq 1/1 Running ... 10.0.0.12 kubernetes-master1 ... kube-flannel-ds-hqwrk 1/1 Running ... 10.0.0.13 kubernetes-master2 ... kube-flannel-ds-npcw6 1/1 Running ... 10.0.0.17 kubernetes-node3 ... kube-flannel-ds-sx427 1/1 Running ... 10.0.0.14 kubernetes-master3 ... kube-flannel-ds-v5f4p 1/1 Running ... 10.0.0.16 kubernetes-node2 ... ``` ```powershell 网卡效果 [root@kubernetes-master1 ~]# for i in {12..17};do ssh root@10.0.0.$i ifconfig | grep -A1 flannel ;done flannel.1: flags=4163 mtu 1450 inet 10.244.0.0 netmask 255.255.255.255 broadcast 0.0.0.0 flannel.1: flags=4163 mtu 1450 inet 10.244.4.0 netmask 255.255.255.255 broadcast 0.0.0.0 flannel.1: flags=4163 mtu 1450 inet 10.244.5.0 netmask 255.255.255.255 broadcast 0.0.0.0 flannel.1: flags=4163 mtu 1450 inet 10.244.1.0 netmask 255.255.255.255 broadcast 0.0.0.0 flannel.1: flags=4163 mtu 1450 inet 10.244.2.0 netmask 255.255.255.255 broadcast 0.0.0.0 flannel.1: flags=4163 mtu 1450 inet 10.244.3.0 netmask 255.255.255.255 broadcast 0.0.0.0 注意： flannel.1 后面的.1 就是 vxlan的网络标识。便于隧道正常通信。 ``` ```powershell 路由效果 [root@kubernetes-master1 ~]# ip route list | grep flannel 10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink 10.244.4.0/24 via 10.244.4.0 dev flannel.1 onlink 10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink ``` **原理解读** flannel模型 ```powershell vxlan模型 pod与Pod经由隧道封装后通信，各节点彼此间能通信就行，不要求在同一个二层网络 这是默认采用的方式 host-gw模型 Pod与Pod不经隧道封装而直接通信，要求各节点位于同一个二层网络 vxlan directrouting模型 它是vxlan和host-gw自由组合的一种模型 位于同一个二层网络上的、但不同节点上的Pod间通信，无须隧道封装；但非同一个二层网络上的节点上的Pod间通信，仍须隧道封装 ``` ![image-20220722073310706](../../img/kubernetes/kubernetes_flannel/image-20220722073310706.png) vxlan原理 ```powershell 1 节点上的pod通过虚拟网卡对，连接到cni0的虚拟网络交换机上 当有外部网络通信的时候，借助于 flannel.1网卡向外发出数据包 2 经过 flannel.1 网卡的数据包，借助于flanneld实现数据包的封装和解封 最后送给宿主机的物理接口，发送出去 3 对于pod来说，它以为是通过 flannel.x -> vxlan tunnel -> flannel.x 实现数据通信 因为它们的隧道标识都是".1"，所以认为是一个vxlan，直接路由过去了，没有意识到底层的通信机制。 注意： 由于这种方式，是对数据报文进行了多次的封装，降低了当个数据包的有效载荷。所以效率降低了 ``` host-gw原理 ```powershell 1 节点上的pod通过虚拟网卡对，连接到cni0的虚拟网络交换机上。 2 pod向外通信的时候，到达CNI0的时候，不再直接交给flannel.1由flanneld来进行打包处理了。 3 cni0直接借助于内核中的路由表，通过宿主机的网卡交给同网段的其他主机节点 4 对端节点查看内核中的路由表，发现目标就是当前节点，所以交给对应的cni0，进而找到对应的pod。 ``` **小结** ``` ``` ### 1.3.3 主机网络 学习目标 这一节，我们从 配置解读、简单实践、小结 三个方面来学习。 **配置解读** 配置文件 ```powershell 我们在部署flannel的时候，有一个配置文件，在这个配置文件中的configmap中就定义了虚拟网络的接入功能。 [root@kubernetes-master1 /data/kubernetes/flannel]# cat kube-flannel.yml kind: ConfigMap ... data: cni-conf.json: | cni插件的功能配置 { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", 基于flannel实现网络通信 "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", 来实现端口映射的功能 "capabilities": { "portMappings": true } } ] } net-conf.json: | flannel的网址分配 { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" 来新节点的时候，基于vxlan从network中获取子网 } } ``` 路由表信息 ```powershell [root@kubernetes-master1 ~]# ip route list | grep flannel 10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink 10.244.4.0/24 via 10.244.4.0 dev flannel.1 onlink 10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink 结果显示： 如果数据包的目标是当前节点，这直接通过cni来进行处理 如果数据包的目标是其他节点，这根据路由配置，交给对应节点上的flannel.1网卡来进行处理 然后交给配套的flanneld对数据包进行封装 ``` 数据包转发解析 ```powershell 发现目标地址 [root@kubernetes-master1 /data/kubernetes/flannel]# ip neigh | grep flannel 10.244.2.0 dev flannel.1 lladdr ca:d2:1b:08:40:c2 PERMANENT 10.244.5.0 dev flannel.1 lladdr 92:d8:04:76:cf:af PERMANENT 10.244.1.0 dev flannel.1 lladdr ae:08:68:7d:fa:31 PERMANENT 10.244.4.0 dev flannel.1 lladdr c2:ef:c2:a6:aa:04 PERMANENT 10.244.3.0 dev flannel.1 lladdr ee:dd:92:60:41:8a PERMANENT 转发给指定主机 [root@kubernetes-master1 /data/kubernetes/flannel]# bridge fdb show flannel.1 | grep flannel.1 ee:dd:92:60:41:8a dev flannel.1 dst 10.0.0.17 self permanent ae:08:68:7d:fa:31 dev flannel.1 dst 10.0.0.15 self permanent c2:ef:c2:a6:aa:04 dev flannel.1 dst 10.0.0.13 self permanent ca:d2:1b:08:40:c2 dev flannel.1 dst 10.0.0.16 self permanent 92:d8:04:76:cf:af dev flannel.1 dst 10.0.0.14 self permanent ``` 测试效果 ```powershell 准备工作 [root@kubernetes-master1 ~]# kubectl scale deployment nginx --replicas=1 deployment.apps/nginx scaled 开启测试容器 [root@kubernetes-master1 ~]# kubectl run flannel-test --image="kubernetes-register.superopsmsb.com/superopsmsb/busybox:1.28" -it --rm --command -- /bin/sh pod现状 [root@kubernetes-master1 ~]# kubectl get pod -o wide NAME ... IP NODE ... flannel-test ... 10.244.1.4 kubernetes-node1 ... nginx-fd669dcb-h5d9d ... 10.244.3.2 kubernetes-node3 ... ``` ```powershell 测试容器发送测试数据 [root@kubernetes-master1 ~]# kubectl run flannel-test --image="kubernetes-register.superopsmsb.com/superopsmsb/busybox:1.28" -it --rm --command -- /bin/sh If you don't see a command prompt, try pressing enter. / # ping -c 1 10.244.3.2 PING 10.244.3.2 (10.244.3.2): 56 data bytes 64 bytes from 10.244.3.2: seq=0 ttl=62 time=1.479 ms ... Flannel默认使用8285端口作为UDP封装报文的端口，VxLan使用8472端口,我们在node3上抓取node1的数据包 [root@kubernetes-node3 ~]# tcpdump -i eth0 -en host 10.0.0.15 and udp port 8472 ... 07:46:40.104526 00:50:56:37:14:42 > 00:50:56:37:ed:73, ethertype IPv4 (0x0800), length 148: 10.0.0.17.53162 > 10.0.0.15.otv: OTV, flags [I] (0x08), overlay 0, instance 1 ee:dd:92:60:41:8a > ae:08:68:7d:fa:31, ethertype IPv4 (0x0800), length 98: 10.244.3.2 > 10.244.1.4: ICMP echo reply, id 1792, seq 0, length 64 结果显示： 这里面每一条数据，都包括了两层ip数据包 ``` host-gw实践 ```powershell 修改flannel的配置文件，将其转换为 host-gw 模型。 [root@kubernetes-master1 ~]# kubectl get cm -n kube-flannel NAME DATA AGE kube-flannel-cfg 2 82m 修改资源配置文件 [root@kubernetes-master1 ~]# kubectl edit cm kube-flannel-cfg -n kube-flannel ... net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "host-gw" } } 重启pod [root@kubernetes-master1 ~]# kubectl delete pod -n kube-flannel -l app [root@kubernetes-master1 ~]# kubectl get pod -n kube-flannel NAME READY STATUS RESTARTS AGE kube-flannel-ds-5dw2r 1/1 Running 0 21s kube-flannel-ds-d9bh2 1/1 Running 0 21s kube-flannel-ds-jvn2f 1/1 Running 0 21s kube-flannel-ds-kcrb4 1/1 Running 0 22s kube-flannel-ds-ptlx8 1/1 Running 0 21s kube-flannel-ds-wxqd7 1/1 Running 0 22s ``` 检查信息 ```powershell 查看路由信息 [root@kubernetes-master1 ~]# ip route list | grep 244 10.244.1.0/24 via 10.0.0.15 dev eth0 10.244.2.0/24 via 10.0.0.16 dev eth0 10.244.3.0/24 via 10.0.0.17 dev eth0 10.244.4.0/24 via 10.0.0.13 dev eth0 10.244.5.0/24 via 10.0.0.14 dev eth0 在flannel-test中继续wget测试 / # wget 10.244.3.2 Connecting to 10.244.3.2 (10.244.3.2:80) index.html 100% |***| 41 0:00:00 ETA ``` ```powershell 在node3中继续抓包 [root@kubernetes-node3 ~]# tcpdump -i eth0 -nn host 10.244.1.4 and tcp port 80 ... 08:01:46.930340 IP 10.244.3.2.80 > 10.244.1.4.59026: Flags [FP.], seq 232:273, ack 74, win 56, length 41: HTTP 08:01:46.930481 IP 10.244.1.4.59026 > 10.244.3.2.80: Flags [.], ack 232, win 58, length 0 08:01:46.931619 IP 10.244.1.4.59026 > 10.244.3.2.80: Flags [F.], seq 74, ack 274, win 58, length 0 08:01:46.931685 IP 10.244.3.2.80 > 10.244.1.4.59026: Flags [.], ack 75, win 56, length 0 结果显示： 两个pod形成了直接的通信效果。 ``` **小结** ``` ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_flink.md ================================================ ## 1.1 **Kubernetes 介绍** Kubernetes是Google公司在2014年6月开源的一个容器集群管理系统，使用Go语言开发，也叫K8S（k8s 这个缩写是因为k和s之间有八个字符的关系）。Kubernetes这个名字源于希腊语，意为“舵手”或“飞行员”。Kubernetes的目标是让部署容器化的应用 简单并且高效,提供应用部署，维护，规划，更新。Kubernetes一个核心的特点就是能够自主的管理容器来保证云平台中的容器按照用户的期望状态运行，让用户能够方便的部署自己的应用。 Kubernetes官网地址如下:[https://kubernetes.io/](https://kubernetes.io/)，中文官网地址如下：[https://kubernetes.io/zh-cn/](https://kubernetes.io/zh-cn/) ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/1ffd5c05d5804b12a2835ce752678ae6.png) 企业中应用程序的部署经历了传统部署时代、虚拟化部署时代、容器化部署时代，尤其是今天容器化部署应用在企业中应用非常广泛，Kubernetes作为容器编排管理工具也越来越重要。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/65c590ae4fee44249a01e40aee1ee6aa.png) ### 1.1.1 **传统部署时代** 早期，各个公司是在物理服务器上运行应用程序。由于无法限制在物理服务器中运行的应用程序资源使用，因此会导致资源分配问题。例如，如果在同一台物理服务器上运行多个应用程序，则可能会出现一个应用程序占用大部分资源的情况，而导致其他应用程序的性能下降。一种解决方案是将每个应用程序都运行在不同的物理服务器上，但是当某个应用资源利用率不高时，服务器上剩余资源无法被分配给其他应用，而且维护许多物理服务器的成本很高。 **物理服务器部署应用痛点如下：** * 物理服务器环境部署人力成本大，特别是在自动化手段不足的情况下，依靠人肉运维的方式解决。 * 当物理服务器出现宕机后，服务器重启时间过长，短则1-2分钟，长则3-5分钟，有背于服务器在线时长达到99.999999999%标准的要求。 * 物理服务器在应用程序运行期间硬件出现故障，解决较麻烦。 * 物理服务器计算资源不能有效调度使用，无法发挥其充足资源的优势。 * 物理服务器环境部署浪费时间，没有自动化运维手段，时间是成倍增加的。 * 在物理服务器上进行应用程序配置变更，需要停止之前部署重新实施部署。 ### 1.1.2 **虚拟化部署时代** 由于以上原因，虚拟化技术被引入了，虚拟化技术允许你在单个物理服务器的CPU上运行多台虚拟机（VM）。虚拟化能使应用程序在不同VM之间被彼此隔离，且能提供一定程度的安全性，因为一个应用程序的信息不能被另一应用程序随意访问。每个 VM 是一台完整的计算机，在虚拟化硬件之上运行所有组件，包括其自己的操作系统。 虚拟化技术能够更好地利用物理服务器的资源，并且因为可轻松地添加或更新应用程序，因此具有更高的可扩缩性，以及降低硬件成本等等的好处。通过虚拟化，你可以将一组物理资源呈现为可丢弃的虚拟机集群。 **虚拟机部署应用优点：** * 虚拟机较物理服务器轻量，可借助虚拟机模板实现虚拟机快捷生成及应用。 * 虚拟机中部署应用与物理服务器一样可控性强，且当虚拟机出现故障时，可直接使用新的虚拟机代替。 * 在物理服务器中使用虚拟机可高效使用物理服务器的资源。 * 虚拟机与物理服务器一样可达到良好的应用程序运行环境的隔离。 * 当部署应用程序的虚拟机出现宕机时，可以快速启动，时间通常可达秒级，10秒或20秒即可启动，应用程序可以继续提供服务。 * 在虚拟机中部署应用，容易扩容及缩容实现、应用程序迁移方便。 **虚拟机部署应用缺点：** * 虚拟机管理软件本身占用物理服务器计算资源较多，例如:VMware Workstation Pro就会占用物理服务器大量资源。 * 虚拟机底层硬件消耗物理服务器资源较大，例如：虚拟机操作系统硬盘，会直接占用大量物理服务器硬盘空间。 * 相较于容器技术，虚拟机启动时间过长，容器启动可按毫秒级计算。 * 虚拟机对物理服务器硬件资源调用添加了调链条，存在浪费时间的现象，所以虚拟机性能弱于物理服务器。 * 由于应用程序是直接部署在虚拟机硬盘上，应用程序迁移时，需要连同虚拟机硬盘中的操作系统一同迁移，会导致迁移文件过大，浪费更多的存储空间及时间消耗过长。 ### 1.1.3 **容器化部署时代** 容器类似于VM，但具备更宽松的隔离特性，使容器之间可以共享操作系统（OS），因此，容器比起VM被认为是更轻量级的。容器化技术中常用的就是Docker容器引擎技术，让开发者可以打包应用以及依赖到一个可移植的镜像中，然后发布到任何平台，其与VM类似，每个容器都具有自己的文件系统、CPU、内存、进程空间等。由于它们与基础架构分离，因此可以跨云和OS发行版本进行移植。 ***基于容器化技术部署应用优点* ：** * 不需要为容器安装操作系统，可以节约大量时间。 * 不需要通过手动的方式在容器中部署应用程序的运行环境，直接部署应用就可以了。 * 不需要管理容器网络，以自动调用的方式访问容器中应用提供的服务。 * 方便分享与构建应用容器，一次构建，到处运行，可在 Ubuntu、RHEL、CoreOS、本地、 Google Kubernetes Engine 等地方运行。 * 毫秒级启动。 * 资源隔离与资源高效应用。应用程序被分解成较小的独立部分，并且可以动态部署和管理，而不是在一台大型单机上整体运行，可以在一台物理机上高密度的部署容器。 虽然使用容器有以上各种优点，但是 **使用容器相比于物理机容器可控性不强** ，例如：对容器的访问，总想按物理服务器或虚拟机的方式去管理它，其实容器与物理服务器、虚拟机管理方式上有着本质的区别的，最好不要管理。 ### 1.1.4 **为什么需要Kubernetes** 一般一个容器中部署运行一个服务，一般不会在一个容器运行多个服务，这样会造成容器镜像复杂度的提高，违背了容器初衷。企业中一个复杂的架构往往需要很多个应用，这就需要运行多个容器，并且需要保证这些容器之间有关联和依赖，那么如何保证各个容器自动部署、容器之间服务发现、容器故障后重新拉起正常运行，这就需要容器编排工具。 Kubernetes就是一个容器编排工具，可以实现容器集群的自动化部署、自动扩展、维护等功能，可以基于Docker技术的基础上，为容器化应用提供部署运行、资源调度、服务发现和动态伸缩等一系列完整功能，提高大规模容器集群管理的便捷性。 **Kubernetes**优点如下： * 服务发现和负载均衡 Kubernetes可以使用DNS名称或自己的IP地址来曝露容器。如果进入容器的流量很大，Kubernetes 可以负载均衡并分配网络流量，从而使部署稳定。 * 存储编排 Kubernetes允许你自动挂载你选择的存储系统，例如本地存储、公共云提供商等。 * 自动部署和回滚 你可以使用Kubernetes描述已部署容器的所需状态，它可以以受控的速率将实际状态更改为期望状态。例如，可以通过Kubernetes来为你部署创建新容器、删除现有容器并将它们的所有资源用于新容器。 * 自动完成装箱计算 在Kubernetes集群上运行容器化的任务时，Kubernetes可以根据运行每个容器指定的多少CPU和内存(RAM)将这些容器按实际情况调度到Kubernetes集群各个节点上，以最佳方式利用集群资源。 * 自我修复 Kubernetes将重新启动失败的容器、替换容器、杀死不响应的容器，并且在准备好服务之前不将其通告给客户端。 * 密钥与配置管理 Kubernetes允许你存储和管理敏感信息，例如密码、OAuth令牌和ssh密钥。 你可以在不重建容器镜像的情况下部署和更新密钥和应用程序配置，也无需在堆栈配置中暴露密钥。 ## 1.2 **Kubernetes集群架构及组件** 一个Kubernetes集群至少有一个主控制平面节点（Control Plane）和一台或者多台工作节点（Node）组成，控制面板和工作节点实例可以是物理设备或云中的实例。Kubernetes 架构如下： ![](file:///C:\Temp\ksohtml16136\wps9.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/a574f87c70034ec1b0abb158368cc765.png) ### 1.2.1 **Kubernetes 控制平面（Contorl Plane）** Kubernetes控制平面也称为主节点（Master Node），其管理集群中的工作节点（Worker Node）和Pod，在生产环境中,Master节点可以运行在多台节点实例上形成主备，提供Kubernetes集群的容错和高可用性。我们可以通过CLI或者UI页面中向Master节点输入参数控制Kubernetes集群。 Master节点是Kubernetes集群的管理中心，包含很多组件，这些组件管理Kubernetes集群各个方面，例如集群组件通信、工作负载调度和集群状态持久化。这些组件可以在集群内任意节点运行，但是为了方便会在一台实例上运行Master所有组件，并且不会在此实例上运行用户容器。 **Kubernetes Master主节点包含组件如下：** * **kube-apiserver：** 用于暴露kubernetes API，任何的资源请求/调用操作都是通过kube-apiserver提供的接口进行。例如:通过REST/kubectl 操作Kubernetes集群调用的就是Kube-apiserver。 * **etcd：** etcd是一个一致的、高度可用的键值存储库，是kubernetes提供默认的存储系统，用于存储Kubernetes集群的状态和配置数据。 * **kube-scheduler：** scheduler负责监视新创建、未指定运行节点的Pods并选择节点来让Pod在上面运行。如果没有合适的节点，则将Pod处于挂起的状态直到出现一个健康的Node节点。 * **kube-controller-manager：** controller-manager 负责运行Kubernetes中的Controller控制器，这些Controller控制器包括： > * 节点控制器（Node Controller）：负责在节点出现故障时进行通知和响应。 > * 任务控制器（Job Controller）：监测代表一次性任务的 Job 对象，然后创建 Pods 来运行这些任务直至完成。 > * 端点分片控制器（EndpointSlice controller）：填充端点分片（EndpointSlice）对象（以提供 Service 和 Pod 之间的链接）。 > * 服务账号控制器（ServiceAccount controller）：为新的命名空间创建默认的服务账号（ServiceAccount）。 * **cloud-controller-manager** 云控制器管理器（Cloud Controller Manager）嵌入了特定于云平台的控制逻辑，允许你将你的集群连接到云提供商的 API 之上， 并将与该云平台交互的组件同与你的集群交互的组件分离开来。cloud-controller-manager 仅运行特定于云平台的控制器。 因此如果你在自己的环境中运行 Kubernetes，或者在本地计算机中运行学习环境， 所部署的集群不需要有云控制器管理器。 ### 1.2.2 **Kubernetes Node节点** Kubernetes Node节点又称为工作节点（Worker Node），一个Kubernetes集群至少需要一个工作节点，但通常很多，工作节点也包含很多组件，用于运行以及维护Pod及service等信息, 管理volume(CVI)和网络(CNI)。在Kubernetes集群中可以动态的添加和删除节点来扩展和缩减集群。 **工作节点Node上的组件如下：** * **Kubelet:** Kubelet会在集群中每个Worker节点上运行，负责维护容器（Containers）的生命周期(创建pod，销毁pod)，确保Pod处于运行状态且健康。同时也负责Volume(CVI)和网络(CNI)的管理。Kubelet不会管理不是由Kubernetes创建的容器。 * **kube-proxy:** Kube-proxy是集群中每个Worker节点上运行的网络代理，管理IP转换和路由，确保每个Pod获得唯一的IP地址，维护网络规则，这些网络规则会允许从集群内部或外部的网络会话与Pod进行网络通信。 * **container Runtime:** 容器运行时(Container Runtime)负责运行容器的软件，为了运行容器每个Worker节点都有一个Container Runtime引擎，负责镜像管理以及Pod和容器的启动停止。 ## 1.3 **Kubernetes 核心概念** Kubernetes中有非常多的核心概念，下面主要介绍Kubernetes集群中常见的一些概念。 ### 1.3.1 **Pod** Pod是可以在 Kubernetes 中创建和管理的、最小的可部署的计算单元，是Kubernetes调度的基本单位，Pod设计的理念是每个Pod都有一个唯一的IP。Pod就像豌豆荚一样，其中包含着一组（一个或多个）容器，这些容器共享存储、网络、文件系统以及怎样运行这些容器的声明。 ![](file:///C:\Temp\ksohtml16136\wps10.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/a50dae4997a14440ac578cd042c9a4cc.png) **Node&Pod&Container&应用程序关系如下图所示：** ![](file:///C:\Temp\ksohtml16136\wps11.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/56789990f8a04f7a9d0cc2b75d727654.png) ### 1.3.2 **Label** Label是附着到object上（例如Pod）的键值对。可以在创建object的时候指定，也可以在object创建后随时指定。Labels的值对系统本身并没有什么含义，只是对用户才有意义。 一个Label是一个key=value的键值对，其中key与value由用户自己指定。Label可以附加到各种资源对象上，例如Node、Pod、Service、RC等，一个资源对象可以定义任意数量的Label，同一个Label也可以被添加到任意数量的资源对象上去，Label通常在资源对象定义时确定，也可以在对象创建后动态添加或者删除。 我们可以通过指定的资源对象捆绑一个或多个不同的Label来实现多维度的资源分组管理功能，以便于灵活、方便地进行资源分配、调度、配置、部署等管理工作。例如：部署不同版本的应用到不同的环境中；或者监控和分析应用（日志记录、监控、告警）等。 一些常用abel示例如下所示: * 版本标签："release" : "stable" , "release" : "canary"... * 环境标签："environment" : "dev" , "environment" : "production" * 架构标签："tier" : "frontend" , "tier" : "backend" , "tier" : "middleware" * 分区标签："partition" : "customerA" , "partition" : "customerB"... * 质量管控标签："track" : "daily" , "track" : "weekly" Label相当于我们熟悉的“标签”，给某个资源对象定义一个Label，就相当于给它打了一个标签，随后可以通过Label Selector（标签选择器）查询和筛选拥有某些Label的资源对象，Kubernetes通过这种方式实现了类似SQL的简单又通用的对象查询机制。 ### 1.3.3 **NameSpace** Namespace 命名空间是对一组资源和对象的抽象集合，比如可以用来将系统内部的对象划分为不同的项目组或者用户组。常见的pod、service、replicaSet和deployment等都是属于某一个namespace的(默认是default)，而node, persistentVolumes等则不属于任何namespace。 当删除一个命名空间时会自动删除所有属于该namespace的资源，default和kube-system命名空间不可删除。 ### 1.3.4 **Controller控制器** 在 Kubernetes 中，Contorller用于管理和运行Pod的对象，控制器通过监控集群的公共状态，并致力于将当前状态转变为期望的状态。一个Controller控制器至少追踪一种类型的 Kubernetes 资源。这些对象有一个代表期望状态的spec字段。该资源的控制器负责确保其当前状态接近期望状态。 不同类型的控制器实现的控制方式不一样，以下介绍常见的几种类型的控制器。 #### 1.3.4.1 deployments控制器 deployments控制器用来部署无状态应用。基于容器部署的应用一般分为两种，无状态应用和有状态应用。 * 无状态应用：认为Pod都一样，没有顺序要求，随意进行扩展和伸缩。例如：nginx,请求本身包含了响应端为响应这一请求所@需的全部信息。每一个请求都像首次执行一样，不会依赖之前的数据进行响应，不需要持久化数据，无状态应用的多个实例之间互不依赖，可以无序的部署、删除或伸缩。 * 有状态应用：每个pod都是独立运行，有唯一的网络表示符，持久化存储，有序。例如：mysql主从，主机名称固定，而且其扩容以及升级等操作也是按顺序进行。有状态应用前后请求有关联与依赖，需要持久化数据，有状态应运用的多个实例之间有依赖，不能相互替换。 在Kubernetes中，一般情况下我们不需要手动创建Pod实例，而是采用更高一层的抽象或定义来管理Pod，针对无状态类型的应用，Kubernetes使用Deloyment的Controller对象与之对应，其典型的应用场景包括： > * 定义Deployment来创建Pod和ReplicaSet > * 滚动升级和回滚应用 > * 扩容和缩容 > * 暂停和继续Deployment #### 1.3.4.2 **ReplicaSet控制器** 通过改变Pod副本数量实现Pod的扩容和缩容，一般Deployment里包含并使用了ReplicaSet。对于ReplicaSet而言，它希望pod保持预期数目、持久运行下去，除非用户明确删除，否则这些对象一直存在，它们针对的是耐久性任务，如web服务等。 #### 1.3.4.3 **statefulSet控制器** Deployments和ReplicaSets是为无状态服务设计的，StatefulSet则是为了有状态服务而设计，其应用场景包括： * 稳定的持久化存储，即Pod重新调度后还是能访问到相同的持久化数据，基于PVC（PersistentVolumeClaim，持久存储卷声明）来实现。 * 稳定的网络标志，即Pod重新调度后其PodName和HostName不变，基于Headless Service(即没有Cluster IP的Service)来实现。 注意从写法上来看statefulSet与deployment几乎一致，就是类型不一样。 #### 1.3.4.4 **DaemonSet控制器** DaemonSet保证在每个Node上都运行一个相同Pod实例，常用来部署一些集群的日志、监控或者其他系统管理应用。DaemonSet使用注意以下几点： * 当节点加入到Kubernetes集群中，pod会被（DaemonSet）调度到该节点上运行。 * 当节点从Kubernetes集群中被移除，被DaemonSet调度的pod会被移除。 * 如果删除一个Daemonset，所有跟这个DaemonSet相关的pods都会被删除。 * 如果一个DaemonSet的Pod被杀死、停止、或者崩溃，那么DaemonSet将会重新创建一个新的副本在这台计算节点上。 * DaemonSet一般应用于日志收集、监控采集、分布式存储守护进程等。 #### 1.3.4.5 **Job控制器** ReplicaSet针对的是耐久性任务，对于非耐久性任务，比如压缩文件，任务完成后，pod需要结束运行，不需要pod继续保持在系统中，这个时候就要用到Job。Job负责批量处理短暂的一次性任务 (short lived one-off tasks)，即仅执行一次的任务，它保证批处理任务的一个或多个Pod成功结束。 #### 1.3.4.6 **Cronjob控制器** Cronjob类似于Linux系统的crontab，在指定的时间周期运行相关的任务。 ### 1.3.5 **Service** 使用kubernetes集群运行工作负载时，由于Pod经常处于用后即焚状态，Pod经常被重新生成，因此Pod对应的IP地址也会经常变化，导致无法直接访问Pod提供的服务，Kubernetes中使用了Service来解决这一问题，即在Pod前面使用Service对Pod进行代理，无论Pod怎样变化 ，只要有Label，就可以让Service能够联系上Pod，把PodIP地址添加到Service对应的端点列表（Endpoints）实现对Pod IP跟踪，进而实现通过Service访问Pod目的。 Service有以下几个注意点： * 通过service为pod客户端提供访问pod方法，即可客户端访问pod入口 * 通过标签动态感知pod IP地址变化等 * 防止pod失联 * 定义访问pod访问策略 * 通过label-selector相关联 * 通过Service实现Pod的负载均衡 Service 有如下四种类型： * ClusterIP：默认，分配一个集群内部可以访问的虚拟IP。 * NodePort：在每个Node上分配一个端口作为外部访问入口。nodePort端口范围为:30000-32767 * LoadBalancer：工作在特定的Cloud Provider上，例如Google Cloud，AWS，OpenStack。 * ExternalName：表示把集群外部的服务引入到集群内部中来，即实现了集群内部pod和集群外部的服务进行通信，适用于外部服务使用域名的方式，缺点是不能指定端口。 ### 1.3.6 **Volume 存储卷** 默认情况下容器的数据是非持久化的，容器消亡以后数据也会跟着丢失。Docker容器提供了Volume机制以便将数据持久化存储。Kubernetes提供了更强大的Volume机制和插件，解决了容器数据持久化以及容器间共享数据的问题。 Kubernetes存储卷的生命周期与Pod绑定，容器挂掉后Kubelet再次重启容器时，Volume的数据依然还在，Pod删除时，Volume才会清理。数据是否丢失取决于具体的Volume类型，比如emptyDir的数据会丢失，而PV的数据则不会丢。 目前Kubernetes主要支持以下Volume类型： * emptyDir：Pod存在，emptyDir就会存在，容器挂掉不会引起emptyDir目录下的数据丢失，但是pod被删除或者迁移，emptyDir也会被删除。 * hostPath：hostPath允许挂载Node上的文件系统到Pod里面去。 * NFS（Network File System）：网络文件系统，Kubernetes中通过简单地配置就可以挂载NFS到Pod中，而NFS中的数据是可以永久保存的，同时NFS支持同时写操作。 * glusterfs：同NFS一样是一种网络文件系统，Kubernetes可以将glusterfs挂载到Pod中，并进行永久保存。 * cephfs：一种分布式网络文件系统，可以挂载到Pod中，并进行永久保存。 * subpath：Pod的多个容器使用同一个Volume时，会经常用到。 * secret：密钥管理，可以将敏感信息进行加密之后保存并挂载到Pod中。 * persistentVolumeClaim：用于将持久化存储（PersistentVolume）挂载到Pod中。 除了以上几种Volume类型，Kubernetes还支持很多类型的Volume，详细可以参考：[https://kubernetes.io/docs/concepts/storage/](https://kubernetes.io/docs/concepts/storage/) ### 1.3.7 **PersistentVolume(PV) 持久化存储卷** kubernetes存储卷的分类太丰富了，每种类型都要写相应的接口与参数才行，这就让维护与管理难度加大，PersistentVolume(PV)是集群之中的一块网络存储，跟 Node 一样，也是集群的资源。PV是配置好的一段存储(可以是任意类型的存储卷)，将网络存储共享出来,配置定义成PV。PersistentVolume (PV)和PersistentVolumeClaim (PVC)提供了方便的持久化卷， PV提供网络存储资源，而PVC请求存储资源并将其挂载到Pod中，通过PVC用户不需要关心具体的volume实现细节,只需要关心使用需求。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/fbf2a1687a1e4d3daf73600c19fccaa5.png) ### 1.3.8 **ConfigMap** ConfigMap用于保存配置数据的键值对，可以用来保存单个属性，也可以用来保存配置文件，实现对容器中应用的配置管理，可以把ConfigMap看作是一个挂载到pod中的存储卷。ConfigMap跟secret很类似，但它可以更方便地处理不包含敏感信息的明文字符串。 ### 1.3.9 **Secret** Sercert-密钥解决了密码、token、密钥等敏感数据的配置问题，而不需要把这些敏感数据暴露到镜像或者Pod Spec中。 ### 1.3.10 **ServiceAccount** Service account是为了方便Pod里面的进程调用Kubernetes API或其他外部服务而设计的。Service Account为服务提供了一种方便的认证机制，但它不关心授权的问题。可以配合RBAC(Role Based Access Control)来为Service Account鉴权，通过定义Role、RoleBinding、ClusterRole、ClusterRoleBinding来对sa进行授权。 ## 1.4 **Kubernetes 集群搭建环境准备** 这里使用kubeadm部署工具来进行部署Kubernetes。Kubeadm是为创建Kubernetes集群提供最佳实践并能够“快速路径”构建kubernetes集群的工具。它能够帮助我们执行必要的操作，以获得最小可行的、安全的集群，并以用户友好的方式运行。 ### 1.4.1 **节点划分** kubernetes 集群搭建节点分布： | **节点IP** | **节点名称** | **Master** | **Worker** | | ---------------- | ------------------ | ---------------- | ---------------- | | 192.168.179.4 | node1 | ★ | | | 192.168.179.5 | node2 | | ★ | | 192.168.179.6 | node3 | | ★ | | 192.168.179.7 | node4 | | | | 192.168.179.8 | node5 | | | ### 1.4.2 **升级内核** 升级操作系统内核，升级到6.06内核版本。这里所有主机均操作，包括node4,node5节点。 ``` #导入elrepo gpg key rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org #安装elrepo YUM源仓库 yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm #安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 #设置grub2默认引导为0 grub2-set-default 0 #重新生成grub2引导文件 grub2-mkconfig -o /boot/grub2/grub.cfg #更新后，需要重启，使用升级的内核生效。 reboot #重启后，需要验证内核是否为更新对应的版本 uname -r 6.0.6-1.el7.elrepo.x86_64 ``` ### 1.4.3 **配置内核转发及网桥过滤** 在所有K8S主机配置。添加网桥过滤及内核转发配置文件： ``` vim /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ``` 加载br_netfilter模块： ``` #加载br_netfilter模块 modprobe br_netfilter #查看是否加载 lsmod | grep br_netfilter ``` 加载网桥过滤及内核转发配置文件： ``` sysctl -p /etc/sysctl.d/k8s.conf ``` ### 1.4.4 **安装ipset及ipvsadm** 所有主机均需要操作。主要用于实现service转发。 ``` #安装ipset及ipvsadm yum -y install ipset ipvsadm 配置ipvsadm模块加载方式，添加需要加载的模块 vim /etc/sysconfig/modules/ipvs.modules modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack 授权、运行、检查是否加载 chmod 755 /etc/sysconfig/modules/ipvs.modules bash /etc/sysconfig/modules/ipvs.modules lsmod | grep -e ip_vs -e nf_conntrack ``` ### 1.4.5 **关闭SWAP分区** 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a。永远关闭swap分区，需要重启操作系统。 ``` #永久关闭swap分区 ,在 /etc/fstab中注释掉下面一行 vim /etc/fstab #/dev/mapper/centos-swap swap swap defaults 0 0 #重启机器 reboot ``` ### 1.4.6 **安装docker** 所有集群主机均需操作。 获取docker repo文件 ``` wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ``` 查看docker可以安装的版本： ``` yum list docker-ce.x86_64 --showduplicates | sort -r ``` 安装docker:这里指定docker版本为20.10.9版本 ``` yum -y install docker-ce-20.10.9-3.el7 ``` > 如果安装过程中报错: ``` Error: Package: 3:docker-ce-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: container-selinux >= 2:2.74 Error: Package: docker-ce-rootless-extras-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: fuse-overlayfs >= 0.7 Error: Package: docker-ce-rootless-extras-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: slirp4netns >= 0.4 Error: Package: containerd.io-1.4.9-3.1.el7.x86_64 (docker-ce-stable) ``` > 缺少一些依赖，解决方式：在/etc/yum.repos.d/docker-ce.repo开头追加如下内容: ``` [centos-extras] name=Centos extras - $basearch baseurl=http://mirror.centos.org/centos/7/extras/x86_64 enabled=1 gpgcheck=0 ``` > 然后执行安装命令： ``` yum -y install slirp4netns fuse-overlayfs container-selinux ``` > 执行完以上之后，再次执行yum -y install docker-ce-20.10.9-3.el7安装docker即可。 设置docker 开机启动，并启动docker： ``` systemctl enable docker systemctl start docker ``` 查看docker版本 ``` docker version ``` 修改cgroup方式，并重启docker。 ``` vim /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } #重启docker systemctl restart docker ``` ## 1.5 **Kubernetes 集群搭建** ### 1.5.1 **软件版本** 这里安装Kubernetes版本为1.25.3，在所有主机（node1,node2,node3）安装kubeadm，kubelet，kubectl。 kubeadm：初始化集群、管理集群等。 kubelet:用于接收api-server指令，对pod生命周期进行管理。 kubectl:集群应用命令行管理工具。 ### 1.5.2 **准备阿里yum源** 每台k8s节点vim /etc/yum.repos.d/k8s.repo，写入以下内容： ``` [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ``` ### 1.5.3 **集群软件安装** ``` #查看指定版本 yum list kubeadm.x86_64 --showduplicates | sort -r yum list kubelet.x86_64 --showduplicates | sort -r yum list kubectl.x86_64 --showduplicates | sort -r #安装指定版本 yum -y install --setopt=obsoletes=0 kubeadm-1.25.3-0 kubelet-1.25.3-0 kubectl-1.25.3-0 ``` ### 1.5.4 **配置kubelet** 为了实现docker使用的cgroup driver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ``` #vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ``` 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 ``` systemctl enable kubelet ``` ### 1.5.5 **集群镜像准备** 只需要在node1 Master节点上执行如下下载镜像命令即可，这里先使用kubeadm查询下镜像。 ``` [root@node1 ~]#kubeadm config images list --kubernetes-version=v1.25.3 registry.k8s.io/kube-apiserver:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3 registry.k8s.io/pause:3.8 registry.k8s.io/etcd:3.5.4-0 registry.k8s.io/coredns/coredns:v1.9.3 ``` 编写下载镜像脚本image_download.sh： ``` #!/bin/bash images_list=' registry.k8s.io/kube-apiserver:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3 registry.k8s.io/pause:3.8 registry.k8s.io/etcd:3.5.4-0 registry.k8s.io/coredns/coredns:v1.9.3 ' for i in $images_list do docker pull $i done docker save -o k8s-1-25-3.tar $images_list ``` 以上脚本准备完成之后，执行命令：sh image_download.sh 进行镜像下载 注意：下载时候需要科学上网，否则下载不下来。也可以使用资料中的“k8s-1-25-3.tar”下载好的包。 > #如果下载不下来，使用资料中打包好的k8s-1-25-3.tar，将镜像导入到docker中 > > docker load -i k8s-1-25-3.tar ### 1.5.6 **集群初始化** 只需要在Master节点执行如下初始化命令即可。 ``` [root@node1 ~]# kubeadm init --kubernetes-version=v1.25.3 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.179.4 ``` 注意：--apiserver-advertise-address=192.168.179.4 要写当前主机Master IP > 初始化过程中报错： ``` [init] Using Kubernetes version: v1.25.3 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR CRI]: container runtime is not running: output: E1102 20:14:29.494424 10976 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"time="2022-11-02T20:14:29+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alp ha2.RuntimeService", error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher ``` 执行如下命令，重启containerd后，再次init 初始化。 ``` [root@node1 ~]# rm -rf /etc/containerd/config.toml [root@node1 ~]# systemctl restart containerd ``` 初始化完成后，结果如下： ``` Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 ``` ### 1.5.7 **集群应用客户端管理集群文件准备** 参照初始化的内容来执行如下命令： ``` [root@node1 ~]# mkdir -p $HOME/.kube [root@node1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@node1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@node1 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf ``` ### 1.5.8 **集群网络准备** #### 1.5.8.1 **calico安装** K8s使用calico部署集群网络,安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico，k8s 1.25.3 匹配 calico版本为 v3.24.5。 只需要在Master节点安装即可。 ``` #下载operator资源清单文件 wget https://docs.projectcalico.org/manifests/tigera-operator.yaml --no-check-certificate #应用资源清单文件，创建operator kubectl create -f tigera-operator.yaml #通过自定义资源方式安装 wget https://docs.projectcalico.org/manifests/custom-resources.yaml --no-check-certificate #修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 # vim custom-resources.yaml 【修改和增加以下加粗内容】 apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. calicoNetwork: # Note: The ipPools section cannot be modified post-install. ipPools: - blockSize: 26 cidr: 10.244.0.0/16 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all() nodeAddressAutodetectionV4: interface: ens.* #应用清单文件 kubectl create -f custom-resources.yaml #监视calico-sysem命名空间中pod运行情况 watch kubectl get pods -n calico-system [root@node1 ~]# watch kubectl get pods -n calico-system Every 2.0s: kubectl get pods -n calico-system Thu Nov 3 14:14:30 2022 NAME READY STATUS RESTARTS AGE calico-kube-controllers-65648cd788-flmk4 1/1 Running 0 2m21s calico-node-chnd5 1/1 Running 0 2m21s calico-node-kc5bx 1/1 Running 0 2m21s calico-node-s2cp5 1/1 Running 0 2m21s calico-typha-d76595dfb-5z6mg 1/1 Running 0 2m21s calico-typha-d76595dfb-hgg27 1/1 Running 0 2m19s #删除 master 上的 taint [root@node1 ~]# kubectl taint nodes --all node-role.kubernetes.io/master- taint "node-role.kubernetes.io/master" not found taint "node-role.kubernetes.io/master" not found taint "node-role.kubernetes.io/master" not found #已经全部运行 [root@node1 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-65648cd788-ktjrh 1/1 Running 0 110m calico-node-dvprv 1/1 Running 0 110m calico-node-nhzch 1/1 Running 0 110m calico-node-q44gh 1/1 Running 0 110m calico-typha-6bc9d76554-4bv77 1/1 Running 0 110m calico-typha-6bc9d76554-nkzxq 1/1 Running 0 110m #查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 [root@node1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-565d847f94-bjtlh 1/1 Running 0 19h coredns-565d847f94-wlxmf 1/1 Running 0 19h etcd-node1 1/1 Running 0 19h kube-apiserver-node1 1/1 Running 0 19h kube-controller-manager-node1 1/1 Running 0 19h kube-proxy-bgpz2 1/1 Running 0 19h kube-proxy-jlltp 1/1 Running 0 19h kube-proxy-stfrx 1/1 Running 0 19h kube-scheduler-node1 1/1 Running 0 19h ``` #### 1.5.8.2 **calico客户端安装** 主要用来验证k8s集群节点网络是否正常。这里只需要在Master节点安装就可以。 ``` #下载二进制文件，注意，这里需要检查calico 服务端的版本，客户端要与服务端版本保持一致，这里没有命令验证calico的版本，所以安装客户端的时候安装最新版本即可。 curl -L https://github.com/projectcalico/calico/releases/download/v3.24.3/calicoctl-linux-amd64 -o calicoctl #安装calicoctl mv calicoctl /usr/bin/ #为calicoctl添加可执行权限 chmod +x /usr/bin/calicoctl #查看添加权限后文件 ls /usr/bin/calicoctl #查看calicoctl版本 [root@node1 ~]# calicoctl version Client Version: v3.24.1 Git commit: 83493da01 Cluster Version: v3.24.3 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm 通过~/.kube/config连接kubernetes集群，查看已运行节点 [root@node1 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME node1 ``` ### 1.5.9 **集群工作节点添加** 这里在node2,node3 worker节点上执行命令，将worker节点加入到k8s集群。 ``` [root@node2 ~]# kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 [root@node3 ~]# kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 ``` > 注意：如果以上node2,node3 Worker节点已经错误的加入到Master节点，需要在Worker节点执行如下命令清除对应的信息，然后再次加入即可。 ``` #重置kubeadm [root@node2 ~]# kubeadm reset #删除k8s配置文件和证书文件 [root@node2 kubernetes]# rm -f /etc/kubernetes/kubelet.conf [root@node2 kubernetes]# rm -f /etc/kubernetes/pki/ca.crt #重置kubeadm [root@node3 ~]# kubeadm reset #删除k8s配置文件和证书文件 [root@node3 kubernetes]# rm -f /etc/kubernetes/kubelet.conf [root@node3 kubernetes]# rm -f /etc/kubernetes/pki/ca.crt ``` > 此外如果忘记了node join加入master节点的命令，可以按照以下步骤操作： ``` #查看discovery-token-ca-cert-hash [root@node1 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' #查看token [root@node1 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS1945mk.ved91lifrc8l0zj9 23h 2022-11-10T08:14:46Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token #Node节点加入集群 [root@node1 ~]# kubeadm join 192.168.179.4:6443 --token 查询出来的token \ --discovery-token-ca-cert-hash 查询出来的hash码 ``` 在master节点上操作，查看网络节点是否添加 ``` [root@node1 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME node1 node2 node3 ``` ### 1.5.10 **验证集群可用性** 使用命令查看所有的节点： ``` [root@node1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 20h v1.25.3 node2 Ready 20h v1.25.3 node3 Ready 20h v1.25.3 ``` 查看集群健康情况： ``` [root@node1 ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-0 Healthy {"health":"true","reason":""} scheduler Healthy ok controller-manager Healthy ok ``` 查看kubernetes集群pod运行情况: ``` [root@node1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-565d847f94-bjtlh 1/1 Running 0 20h coredns-565d847f94-wlxmf 1/1 Running 0 20h etcd-node1 1/1 Running 0 20h kube-apiserver-node1 1/1 Running 0 20h kube-controller-manager-node1 1/1 Running 0 20h kube-proxy-bgpz2 1/1 Running 1 20h kube-proxy-jlltp 1/1 Running 1 20h kube-proxy-stfrx 1/1 Running 0 20h kube-scheduler-node1 1/1 Running 0 20h ``` 查看集群信息: ``` [root@node1 ~]# kubectl cluster-info Kubernetes control plane is running at https://192.168.179.4:6443 CoreDNS is running at https://192.168.179.4:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` ### 1.5.11 **K8s集群其他一些配置** 当在Worker节点上执行kubectl命令管理时会报如下错误： ``` The connection to the server localhost:8080 was refused - did you specify the right host or port? ``` 只要把master上的管理文件/etc/kubernetes/admin.conf拷贝到Worker节点的$HOME/.kube/config就可以让Worker节点也可以实现kubectl命令管理。 ``` #在Worker节点创建.kube目录 [root@node2 ~]# mkdir /root/.kube [root@node3 ~]# mkdir /root/.kube #在master节点做如下操作 [root@node1 ~]# scp /etc/kubernetes/admin.conf node2:/root/.kube/config [root@node1 ~]# scp /etc/kubernetes/admin.conf node3:/root/.kube/config #在worker 节点验证 [root@node2 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 24h v1.25.3 node2 Ready 24h v1.25.3 node3 Ready 24h v1.25.3 [root@node3 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 24h v1.25.3 node2 Ready 24h v1.25.3 node3 Ready 24h v1.25.3 ``` 此外，无论在Master节点还是Worker节点使用kubenetes 命令时，默认不能自动补全，例如：kubectl describe 命令中describe不能自动补全，使用非常不方便，那么这里配置命令自动补全功能。 在所有的kubernetes节点上安装bash-completion并source执行，同时配置下开机自动source，每次开机能自动补全命令。 ``` #安装bash-completion 并 source yum install -y bash-completion source /usr/share/bash-completion/bash_completion kubectl completion bash > ~/.kube/completion.bash.inc source '/root/.kube/completion.bash.inc' #实现用户登录主机自动source ,自动使用命令补全 vim ~/.bash_profile 【加入加粗这一句】 # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs source '/root/.kube/completion.bash.inc' PATH=$PATH:$HOME/bin export PATH ``` ## 1.6 **Kubernetes集群UI及主机资源监控** ### 1.6.1 **Kubernetes dashboard作用** 通过Kubernetes dashboard能够直观了解Kubernetes集群中运行的资源对象，通过dashboard可以直接管理（创建、删除、重启等操作）资源对象。 ### 1.6.2 **获取Kubernetes dashboard资源清单文件** 这里只需要在Kubernetes Master节点上来下载应用资源清单文件即可。这里去github.com 搜索“kubernetes dashboard”即可，找到匹配Kubernetes 版本的dashboard，下载对应版本的dashboard yaml文件。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/0cddbf7f85004d91afec4b37a36a382d.png) ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/8c8cde92fb3c4208a8da4ae772c5944c.png) ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/d1c47c507a014c2cab5388b06150c29f.png) ``` [root@node1 ~]# mkdir kube-dashboard [root@node1 ~]# cd kube-dashboard/ [root@node1 kube-dashboard]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml ``` ![](file:///C:\Temp\ksohtml16136\wps26.jpg)对应yaml文件下载完成后，为了方便后续在容器主机上访问，在yaml文件中添加对应的NodePort类型、端口以及修改登录kubernetes dashboard的用户。 ``` #vi recommended.yaml 【只需要添加或修改以下加粗部分】 ... ... kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30000 selector: k8s-app: kubernetes-dashboard ... .... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard ``` 注意：一定要把原来kind为ClusterRole下的name对应值kubernetes-dashboard修改为cluster-admin，不然进入UI后会报错。 ``` #部署Kubernetes dashboard [root@node1 kube-dashboard]# kubectl apply -f recommended.yaml #查看部署是否成功，出现kubenetes-dashboard命名空间即可。 [root@node1 kube-dashboard]# kubectl get ns NAME STATUS AGE calico-apiserver Active 5h33m calico-system Active 5h35m default Active 23h kube-node-lease Active 23h kube-public Active 23h kube-system Active 23h kubernetes-dashboard Active 6s tigera-operator Active 5h36m [root@node1 kube-dashboard]# kubectl get pod,svc -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE pod/dashboard-metrics-scraper-64bcc67c9c-gsqsn 1/1 Running 0 112s pod/kubernetes-dashboard-5c8bd6b59-x4p8r 1/1 Running 0 112s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dashboard-metrics-scraper ClusterIP 10.106.178.119 8000/TCP 112s service/kubernetes-dashboard NodePort 10.96.4.46 443:30000/TCP 113s ``` ### 1.6.3 **访问Kubernetes dashboard** WebUI访问Kubernetes dashboard：[https://192.168.179.4:3000](http://192.168.179.4:3000)0 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/093e784b4abf42f098dfc184a36789c0.png) 选择“Token”，使用如下命令获取Kubernetes的Token: ``` #注意最后的kubernetes-dashboard 为部署dashboard创建的serviceaccounts [root@node1 kube-dashboard]# kubectl -n kubernetes-dashboard create token kubernetes-dashboard eyJhbGciOiJSUzI1NiIsImtpZCI6IlFDVlB4Wng3REtPNGZhd05sYnJvbFBWbE9iS2pDYmtEYUZyQ2VaSjA0MjAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjY3NDgxODQ0LCJpYXQiOjE2Njc0NzgyNDQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInVpZCI6ImY3ZGQ0YTI1LTEwOTAtNDYyZC04N2JhLTM4NjNlM2Q3MjQxNCJ9fSwibmJmIjoxNjY3NDc4MjQ0LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQifQ.b0S3YAJrFTUb-pgiPLp3kuB510sL7r9LPvmeO5kXM86ZRJhbGOFsD-CK-ONQnDF2EVAg76YsV_I7Afv_P_RkSspfy0AnBDUFj-LBufocX1cofCHc1_dErVbCQ5MvUsnw67PvpdcZMWuAndYhMVorOIxOc_RxhUM6tre3kuZJ40r2W-8Kbgd4b3HvLeaE2gJNTofn5ChYLkDd7TQYqRtmZN14l6CFZMUSl1dHqSuWhUncNHELhI8uRRD1pfFmMlYrqkZOTqzkw5_czMFrE9yIFKktMqT3wpvRVWFzYZFd9SpGMoQtshKjR3h508N-KG2Ob3PQYbvpBdoap2UjOjQJVg ``` 将以上生成的Token复制粘贴到登录的WebUI页面中登录Kubernetes DashBoard。注意：以上token复制时不要有空格。 ## 1.7 **Kuberneters 部署案例** 这里为了强化对Kubernetes集群的理解，我们基于Kubernetes集群进行部署nginx服务，nginx服务我们设置2个副本，同时将nginx服务端口80暴露到宿主机上。在kubernetes中，我们可以通过WebUI来添加服务，也可以在命令行中通过应用yaml来部署服务，下面以命令行部署nginx服务为例： 1) **创建资源清单文件** * **nginx.yaml** **：** ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test spec: selector: matchLabels: app: nginx env: test owner: rancher replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx env: test owner: rancher spec: containers: - name: nginx-test image: nginx:1.19.9 ports: - containerPort: 80 ``` * **nginx-service.yaml** **：** ``` apiVersion: v1 kind: Service metadata: name: nginx-test labels: run: nginx spec: type: NodePort ports: - port: 80 protocol: TCP nodePort: 30080 selector: owner: rancher ``` 2) **应用资源清单文件** ``` [root@node1 nginx-test]# kubectl apply -f nginx.yaml [root@node1 nginx-test]# kubectl apply -f nginx-service.yaml ``` 3) **验证** ``` [root@node1 test]# kubectl get all NAME READY STATUS RESTARTS AGE pod/nginx-test-74845c57fb-7tl86 1/1 Running 0 45s pod/nginx-test-74845c57fb-qjc6d 1/1 Running 0 45s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 123m service/nginx-test NodePort 10.107.171.29 80:30204/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-test 2/2 2 2 45s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-test-74845c57fb 2 2 2 45s ``` 4) **访问验证** 访问任意kubernetes集群的节点30080端口查看nginx服务是正常，例如：浏览器输入node1:30080 ![](file:///C:\Temp\ksohtml10300\wps1.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/ce75b5426f07418d8cd0015726d6ac9e.png) 5) **删除nginix服务** ``` [root@node1 nginx-test]# kubectl delete -f nginx-service.yaml [root@node1 nginx-test]# kubectl delete -f nginx.yaml ``` ## 1.8 **Flink基于Kubernetes部署** Flink基于Kubernetes部署时支持两种模式：Kubernetes部署模式和Native Kubernetes 部署模式。Flink从1.2版本开始支持Kubernetes部署模式，从Flink1.10版本开始Flink支持Native Kubernetes部署模式。 * **Kubernetes部署模式** Flink Kubernetes这种部署模式是把JobManager和TaskManager等进程放入容器，在Kubernetes管理和运行，这与我们将Java Web应用做成docker镜像再运行在Kubernetes中道理一样，都是使用kubernetes中的kubectl命令来操作。 * **Native Kubernetes部署模式** Native Kubernetes部署模式是在Flink安装包中有个工具，此工具可以向Kubernetes的Api Server 发送请求，例如:创建Flink Master ，并且可以和FlinkMaster通讯，用于提交任务，我们只要用好Flink 安装包中的工具即可，无需在Kuberbetes上执行kubectl操作。 Flink与Kubernetes整合时要求Kubernetes版本不能低于1.9，我们这里使用的Kubernetes版本是1.25.3版本。 ### 1.8.1 **Kubernetes部署** Flink 基于Kubernetes部署支持Session Cluster模式和Application Cluster模式部署。Session Cluster模式即可以在一个Flink集群中运行多个作业，这些作业公用JobManager和TaskManager。Application Cluster模式即一个作业使用单独的一个专用Flink集群，每个Flink作业的JobManager和TaskManager隔离。无论是Session Cluster模式还是Application Cluster模式都支持JobManager的HA 部署。下面分别介绍并测试。 #### 1.8.1.1 *Session Cluster部署* Flink Session集群作为Kubernetes Deployment来运行的，可以在一个基于K8s 部署的Session Cluster 中运行多个Flink job，在Kubernetes 上部署Flink Session 集群时，一般至少包含三个组件： * 运行JobManager的Deployment * 运行TaskManager的Deployment * 暴露JobManager上的REST和UI端口的Service ##### 1.8.1.1.1 **非HA Session Cluster部署及测试** ###### 1.8.1.1.1.1 **准备deployment文件** * **flink-configuration-configmap.yaml** ``` apiVersion: v1 kind: ConfigMap metadata: name: flink-config labels: app: flink data: flink-conf.yaml: |+ jobmanager.rpc.address: flink-jobmanager taskmanager.numberOfTaskSlots: 2 blob.server.port: 6124 jobmanager.rpc.port: 6123 taskmanager.rpc.port: 6122 queryable-state.proxy.ports: 6125 jobmanager.memory.process.size: 1600m taskmanager.memory.process.size: 1728m parallelism.default: 2 log4j-console.properties: |+ # This affects logging for both user code and Flink rootLogger.level = INFO rootLogger.appenderRef.console.ref = ConsoleAppender rootLogger.appenderRef.rolling.ref = RollingFileAppender # Uncomment this if you want to _only_ change Flink's logging #logger.flink.name = org.apache.flink #logger.flink.level = INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. logger.akka.name = akka logger.akka.level = INFO logger.kafka.name= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper logger.zookeeper.level = INFO # Log all infos to the console appender.console.name = ConsoleAppender appender.console.type = CONSOLE appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Log all infos in the given rolling file appender.rolling.name = RollingFileAppender appender.rolling.type = RollingFile appender.rolling.append = false appender.rolling.fileName = ${sys:log.file} appender.rolling.filePattern = ${sys:log.file}.%i appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n appender.rolling.policies.type = Policies appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size=100MB appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 10 # Suppress the irrelevant (wrong) warnings from the Netty channel handler logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF ``` * jobmanager-service.yaml ``` apiVersion: v1 kind: Service metadata: name: flink-jobmanager spec: type: ClusterIP ports: - name: rpc port: 6123 - name: blob-server port: 6124 - name: webui port: 8081 selector: app: flink component: jobmanager ``` * jobmanager-rest-service.yaml ``` apiVersion: v1 kind: Service metadata: name: flink-jobmanager-rest spec: type: NodePort ports: - name: rest port: 8081 targetPort: 8081 nodePort: 30081 selector: app: flink component: jobmanager ``` * jobmanager-session-deployment-non-ha.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: flink-jobmanager spec: replicas: 1 selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: containers: - name: jobmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 args: ["jobmanager"] ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: localtime #挂载localtime文件，使容器时间与宿主机一致 hostPath: path: /etc/localtime type: '' ``` * taskmanager-session-deployment.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager spec: replicas: 2 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: containers: - name: taskmanager image: flink:1.16.0-scala_2.12-java8 args: ["taskmanager"] ports: - containerPort: 6122 name: rpc - containerPort: 6125 name: query-state livenessProbe: tcpSocket: port: 6122 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/ - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: localtime #挂载localtime文件，使容器时间与宿主机一致 hostPath: path: /etc/localtime type: '' ``` 注意：关于Flink的镜像可以从https://hub.docker.com/网站中搜索下载。以上配置文件可以从资料“flink-nonha-session.zip”中获取。 ###### 1.8.1.1.1.2 **部署yaml 文件** 在对应的目录中执行如下命令： ``` kubectl create -f ./flink-configuration-configmap.yaml kubectl create -f ./jobmanager-rest-service.yaml kubectl create -f ./jobmanager-service.yaml kubectl create -f ./jobmanager-session-deployment-non-ha.yaml kubectl create -f ./taskmanager-session-deployment.yaml ``` 注意：也可以进入对应yaml文件目录，直接执行 kubectl create -f ./ 全部部署也可以。 ###### 1.8.1.1.1.3 **验证部署情况** ``` [root@node1 flink-nonha-session]# kubectl get all NAME READY STATUS RESTARTS AGE pod/flink-jobmanager-6f45bb68f5-rnd5p 1/1 Running 0 81s pod/flink-taskmanager-d5f89bb47-jjtz7 1/1 Running 0 81s pod/flink-taskmanager-d5f89bb47-rmfff 1/1 Running 0 81s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/flink-jobmanager ClusterIP 10.107.162.123 6123/TCP,6124/TCP,8081/TCP 82s service/flink-jobmanager-rest NodePort 10.102.57.189 8081:30081/TCP 82s service/kubernetes ClusterIP 10.96.0.1 443/TCP 2d18h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/flink-jobmanager 1/1 1 1 81s deployment.apps/flink-taskmanager 2/2 2 2 81s NAME DESIRED CURRENT READY AGE replicaset.apps/flink-jobmanager-6f45bb68f5 1 1 1 81s replicaset.apps/flink-taskmanager-d5f89bb47 2 2 2 81s ``` 在浏览器输入：[http://192.168.179.4:30081/即可访问Flink](http://node1:30081/即可访问Flink) Session集群WebUI。浏览器中输入的ip可以是K8s集群中任意节点的IP。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/b41acaf414ed41f4a24f6610f70ce116.png) ###### 1.8.1.1.1.4 **停止集群** ``` kubectl delete -f ./flink-configuration-configmap.yaml kubectl delete -f ./jobmanager-rest-service.yaml kubectl delete -f ./jobmanager-service.yaml kubectl delete -f ./jobmanager-session-deployment-non-ha.yaml kubectl delete -f ./taskmanager-session-deployment.yaml ``` 注意：也可以进入对应yaml文件目录，直接执行 kubectl delete -f ./ 全部部署也可以。 ###### 1.8.1.1.1.5**任务提交与测试** 基于K8s的Flink Standalone 集群我们可以通过Flink WebUI来提交Flink任务，也可以通过Flink客户端命令提交任务。 1) **基于WebUI提交Flink任务** 这里编写读取socket端口数据实时统计WordCount的Flink代码，代码如下： ``` package com.mashibing.flinkjava.code.chapter_k8s; import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.functions.KeySelector; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.streaming.api.datastream.DataStreamSource; import org.apache.flink.streaming.api.datastream.KeyedStream; import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.util.Collector; /** * Flink 基于K8s 测试代码： * 读取socket端口数据实时统计WordCount */ public class WordCount { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStreamSource sourceDS = env.socketTextStream("192.168.179.8", 9999); SingleOutputStreamOperator> tupleDS = sourceDS.flatMap(new FlatMapFunction>() { @Override public void flatMap(String line, Collector> collector) throws Exception { String[] words = line.split(" "); for (String word : words) { collector.collect(Tuple2.of(word, 1L)); } } }); KeyedStream, String> keyedStream = tupleDS.keyBy(new KeySelector, String>() { @Override public String getKey(Tuple2 stringLongTuple2) throws Exception { return stringLongTuple2.f0; } }); keyedStream.sum(1).print(); env.execute(); } } ``` 注意：后续基于Kubernetes运行Flink任务都基于此任务测试。 编写好代码后将以上代码进行打包，在对应的节点上启动对应的socket服务（nc -lk 9999），这里代码中选择的是node5 9999端口。通过WebUI进行上传执行，具体操作如下： ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/3e4ea4d12e334ef28963f7ae67e70665.png) ![](file:///C:\Temp\ksohtml16136\wps27.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/fb329bc208d247b1adbc815f869c74bf.png) 在node5 socket 9999端口输入数据： ``` [root@node5 ~]# nc -lk 9999 hello zhangsan hello lisi hello wangwu ``` 向K8S部署的Flink集群中提交应用程序如果打印结果到控制台不支持在WebUI中的TaskManager中查看对应的Console日志，主要原因是K8S 基于Docker运行Flink TaskExecutor和JobMaster 进程时不会将STDOUT日志重定向到文件中。这里可以通过kubectl logs + pod 来查看对应的输出日志。 首先在WebUI页面中查看对应sink运行所在的TaskManager IP。 ![](file:///C:\Temp\ksohtml16136\wps29.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/6c4c01f62b5b4694ae561771377cc13d.png) 通过TaskManager ip 确定输出结果的TaskManager Pod ： ``` [root@node1 flink]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES flink-jobmanager-6f45bb68f5-4dbkp 1/1 Running 0 14m 10.244.135.27 node3 flink-taskmanager-d5f89bb47-d5ljm 1/1 Running 0 14m 10.244.104.19 node2 flink-taskmanager-d5f89bb47-jmlmr 1/1 Running 0 14m 10.244.135.28 node3 ``` 通过以上查询可以根据TaskManager IP 可以找到名为“flink-taskmanager-d5f89bb47-jmlmr”的Pod。查看该pod日志，就可以看到输出的结果： ``` [root@node1 flink]# kubectl logs flink-taskmanager-d5f89bb47-jmlmr ... ... 2022-11-04 07:49:48,208 INFO org.apache.flink.streaming.api.functions.source.SocketTextStreamFunction [] - Connecting to server socket 192.168.1 79.8:99992022-11-04 07:49:48,210 INFO org.apache.flink.runtime.taskmanager.Task [] - Keyed Aggregation -> Sink: Print to Std. Out (1/1 )#0 (124df880f6de0d84f0d085586c52b6d9_90bea66de1c231edf33913ecd54406c1_0_0) switched from INITIALIZING to RUNNING.(hello,1) (zhangsan,1) (hello,2) (lisi,1) (hello,3) (wangwu,1) ``` 2) **通过客户端命令方式提交任务** 这种方式只需要在任意能连接到Kubernetes集群的节点上通过Flink客户端命令提交Flink任务节即可。该节点需要有Flink的安装包,这里选择node4节点上传Flink的安装包并解压到“/software/flink-1.16.0”路径中。 在node5节点上启动对应的Socket服务，执行如下命令进行客户端命令提交Flink任务： ``` [root@node4 ~]# cd /software/flink-1.16.0/bin/ [root@node4 bin]# ./flink run -m 192.168.179.4:30081 -c com.mashibing.flinkjava.code.chapter_k8s.WordCount /software/FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` 向socket端口输入数据，通过kubectl logs + pods 方式查看对应的实时结果即可。 ##### 1.8.1.1.2 **HA Session Cluster部署及测试** 通过以上基于Kubernetes 部署Flink Session集群，我们部署了1个JobManager和2个TaskManager，这里说的HA Session集群部署，就是当JobManager挂掉时，能正常切换到另外的JobManager中继续调度任务。实际上在Kubernets集群中当JobManager容器挂掉之后，Kubernetes集群会自动重新运行新的JobManager。所以基于Kubernets不部署HA 模式模式也没有问题，这里为了更快的进行恢复Flink任务，我们也可以基于Kubernetes配置HA Session模式。 配置HA Session模式与非HA Session模式相比不再需要jobmanager-server.yaml文件。 ###### 1.8.1.1.2.1 **准备deployment文件** * **flink-configuration-configmap.yaml** 该yaml文件相比于非HA Session模式增加了以下行： ``` kubernetes.cluster-id: myk8s #给kubernets集群取个名字 high-availability: kubernetes #指定高可用模式 high-availability.storageDir: hdfs://mycluster/flink/recovery #指定元数据存储目录为hdfs路径 restart-strategy: fixed-delay #指定重启策略 restart-strategy.fixed-delay.attempts: 10 #指定重启尝试次数 ``` 以上配置使用kubernetes来进行协调FlinkHA，部署相应flink-configuration-configmap.yaml文件后在Kubernetes中会额外生成<cluster-id>-xxx对应的configmap对象，此对象记录Flink 集群中提交的job元数据。 完整的flink-configuration-configmap.yaml文件如下： ``` apiVersion: v1 kind: ConfigMap metadata: name: flink-config labels: app: flink data: flink-conf.yaml: |+ jobmanager.rpc.address: flink-jobmanager taskmanager.numberOfTaskSlots: 2 blob.server.port: 6124 jobmanager.rpc.port: 6123 taskmanager.rpc.port: 6122 queryable-state.proxy.ports: 6125 jobmanager.memory.process.size: 1600m taskmanager.memory.process.size: 1728m parallelism.default: 2 kubernetes.cluster-id: myk8s #给kubernets集群取个名字 high-availability: kubernetes #指定高可用模式 high-availability.storageDir: hdfs://mycluster/flink/recovery #指定元数据存储目录为hdfs路径 restart-strategy: fixed-delay #指定重启策略 restart-strategy.fixed-delay.attempts: 10 #指定重启尝试次数 log4j-console.properties: |+ # This affects logging for both user code and Flink rootLogger.level = INFO rootLogger.appenderRef.console.ref = ConsoleAppender rootLogger.appenderRef.rolling.ref = RollingFileAppender # Uncomment this if you want to _only_ change Flink's logging #logger.flink.name = org.apache.flink #logger.flink.level = INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. logger.akka.name = akka logger.akka.level = INFO logger.kafka.name= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper logger.zookeeper.level = INFO # Log all infos to the console appender.console.name = ConsoleAppender appender.console.type = CONSOLE appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Log all infos in the given rolling file appender.rolling.name = RollingFileAppender appender.rolling.type = RollingFile appender.rolling.append = false appender.rolling.fileName = ${sys:log.file} appender.rolling.filePattern = ${sys:log.file}.%i appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n appender.rolling.policies.type = Policies appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size=100MB appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 10 # Suppress the irrelevant (wrong) warnings from the Netty channel handler logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF ``` * **jobmanager-rest-service.yaml** 该文件与非HA Session模式对应文件部署一样。 ``` apiVersion: v1 kind: Service metadata: name: flink-jobmanager-rest spec: type: NodePort ports: - name: rest port: 8081 targetPort: 8081 nodePort: 30081 selector: app: flink component: jobmanager ``` * jobmanager-cluster-role.yaml 在k8s中是基于角色授权的，创建用户时需要绑定对应的角色，在JobManager HA 部署案例中需要操作对应的配置ConfigMap文件。这里通过jobmanager-cluster-role.yaml创建一个ClusterRole，然后再创建用户，将用户绑定到该集群角色，拥有对ConfigMap操作权限。 ``` kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: configmaps-role #创建ClusterRole的名称 rules: - apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] #指定操作的资源为configmaps verbs: ["create", "edit", "delete","get", "watch", "list","update"] #指定操作的权限 ``` 以上创建完成，再创建一个用户叫flink-service-account，后续将用户绑定到该角色。 ``` kubectl create serviceaccount flink-service-account ``` 把flink-service-account用户绑定到集群角色： ``` kubectl create clusterrolebinding flink-role-binding-serviceaccount --clusterrole=configmaps-role --serviceaccount=default:flink-service-account ``` * **jobmanager-session-deployment-ha.yaml** ``` apiVersion: apps/v1 kind: Deployment metadata: name: flink-jobmanager spec: replicas: 2 #这里配置2个副本# Set the value to greater than 1 to start standby JobManagers selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: hostAliases: #向容器/etc/hosts中加入ip与节点名称映射，pod找HDFS集群时需要使用 - ip: 192.168.179.4 hostnames: - "node1" - ip: 192.168.179.5 hostnames: - "node2" - ip: 192.168.179.6 hostnames: - "node3" - ip: 192.168.179.7 hostnames: - "node4" - ip: 192.168.179.8 hostnames: - "node5" containers: - name: jobmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 env: - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HADOOP_CLASSPATH #这里由于在flink pod内使用到HDFS ，需要把宿主机的HDFS配置文件挂载进来，并配置HADOOP_CLASSPATH环境变量，可以通过在宿主机执行echo `hadoop classpath`来参考这里导入的路径，记得要把Hadoop路径改成挂载到Pod中的路径 value: /opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/* # The following args overwrite the value of jobmanager.rpc.address configured in the configuration config map to POD_IP. args: ["jobmanager", "$(POD_IP)"] ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 mountPath: /opt/hadoop securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary serviceAccountName: flink-service-account #指定seviceAccountName 用户名 volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: localtime #挂载localtime文件，使容器时间与宿主机一致 hostPath: path: /etc/localtime type: '' - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 hostPath: path: /software/hadoop-3.3.4 type: '' ``` * **taskmanager-session-deployment.yaml** ``` apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager spec: replicas: 2 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: hostAliases: #向容器/etc/hosts中加入ip与节点名称映射，pod找HDFS集群时需要使用 - ip: 192.168.179.4 hostnames: - "node1" - ip: 192.168.179.5 hostnames: - "node2" - ip: 192.168.179.6 hostnames: - "node3" - ip: 192.168.179.7 hostnames: - "node4" - ip: 192.168.179.8 hostnames: - "node5" containers: - name: taskmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 env: - name: HADOOP_CLASSPATH #这里由于在flink pod内使用到HDFS ，需要把宿主机的HDFS配置文件挂载进来，并配置HADOOP_CLASSPATH环境变量，可以通过在宿主机执行echo `hadoop classpath`来参考这里导入的路径，记得要把Hadoop路径改成挂载到Pod中的路径 value: /opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/* args: ["taskmanager"] ports: - containerPort: 6122 name: rpc - containerPort: 6125 name: query-state livenessProbe: tcpSocket: port: 6122 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/ - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 mountPath: /opt/hadoop securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary serviceAccountName: flink-service-account #指定seviceAccountName 用户名 volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: localtime #挂载localtime文件，使容器时间与宿主机一致 hostPath: path: /etc/localtime type: '' - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 hostPath: path: /software/hadoop-3.3.4 type: '' ``` 注意：关于Flink的镜像可以从https://hub.docker.com/网站中搜索下载。 由于在flink pod内使用到HDFS ，需要把宿主机的HDFS配置文件挂到Flink JobManager和TaskManager中并配置HADOOP_CLASSPATH环境变量，可以通过在宿主机执行echo `hadoop classpath`来参考HADOOP_CLASSPATH环境变量对应的value值路径，记得要把Hadoop路径改成挂载到Pod中的路径。 以上配置文件可以从资料“flink-ha-session.zip”中获取。 ###### 1.8.1.1.2.2 **部署yaml 文件** 由于HA模式使用到了HDFS集群，所以这里应该首先启动HDFS集群然后再部署对应的yaml文件。 ``` #启动zookeeper [root@node3 ~]# zkServer.sh start [root@node4 ~]# zkServer.sh start [root@node5 ~]# zkServer.sh start #启动HDFS集群 [root@node1 ~]# start-all.sh ``` 部署之前记得执行上一小节中创建用户与绑定用户到角色命令，如果执行过不必重复创建执行。 ``` kubectl create serviceaccount flink-service-account kubectl create clusterrolebinding flink-role-binding-serviceaccount --clusterrole=configmaps-role --serviceaccount=default:flink-service-account ``` 在对应的目录中执行如下命令，部署yaml文件 ``` kubectl create -f ./flink-configuration-configmap.yaml kubectl create -f ./jobmanager-cluster-role.yaml kubectl create -f ./jobmanager-rest-service.yaml kubectl create -f ./jobmanager-session-deployment-ha.yaml kubectl create -f ./taskmanager-session-deployment.yaml ``` 注意：也可以进入对应yaml文件目录，直接执行 kubectl create -f ./ 全部部署也可以。 ###### 1.8.1.1.2.3 **验证部署情况** ``` [root@node1 flink-ha-session]# kubectl get all NAME READY STATUS RESTARTS AGE pod/flink-jobmanager-7bbc68889f-jvp7d 1/1 Running 0 6m40s pod/flink-jobmanager-7bbc68889f-vn7l4 1/1 Running 0 6m41s pod/flink-taskmanager-7766758754-kvg64 1/1 Running 0 6m40s pod/flink-taskmanager-7766758754-mfjgd 1/1 Running 0 6m41s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/flink-jobmanager-rest NodePort 10.100.75.167 8081:30081/TCP 6m41s service/kubernetes ClusterIP 10.96.0.1 443/TCP 2d20h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/flink-jobmanager 2/2 2 2 6m41s deployment.apps/flink-taskmanager 2/2 2 2 6m41s NAME DESIRED CURRENT READY AGE replicaset.apps/flink-jobmanager-7bbc68889f 2 2 2 6m41s replicaset.apps/flink-taskmanager-7766758754 2 2 2 6m41s ``` 在浏览器输入：[http://192.168.179.4:30081/即可访问Flink](http://node1:30081/即可访问Flink) Session集群WebUI。浏览器中输入的ip可以是K8s集群中任意节点的IP。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/61fe95ec75e0431f92de65b96fa28c67.png) ###### 1.8.1.1.2.4 **HA高可用验证** 登录HDFS WebUI查看是否生成“hdfs://mycluster/flink/recovery”目录： ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/55517d6f9ea94fde926c1bd7dc5473a6.png) 下面我们测试JobManager是否能正常切换。首先通过WebUI查看Active JobManager对应的IP信息： ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/a26b98a31a174973872fe36c21eb1c3b.png) 然后在Kubernetes集群中根据这个IP找到对应的的Active JobManager节点并删除： ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/686a99cbdaab4f9699e106a2bca4a70e.png) ``` [root@node1 flink-ha-session]# kubectl delete pod flink-jobmanager-7bbc68889f-vn7l4 pod "flink-jobmanager-7bbc68889f-vn7l4" deleted ``` 以上删除该Active JobManager对应的pod后，Kubernetes机制本身会尝试重启新的Pod，当然由于我们配置了Flink HA ,所以Kubernetes会在新启动的JobManager Pod与原来运行的Standby JobManager Pod中进行自动选主，有一定概率会选择原来一直运行的JobManager Pod当做Active JobManager。 重新查看Flink WebUI中JobManager IP: ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/a5b935cd9a974e4988868eb599f66078.png) 通过以上验证我们发现原来备用的JobManager已经切换成Active JobManager。 注意：删除原来Activate JobManager后有可能将Kubernetes重新启动的JobManager选择为Active JobManager，可以尝试多次delete进行验证HA 切换。 ###### 1.8.1.1.2.5 **任务提交与测试** 这里基于命令行方式来进行任务提交，不再以WebUI方式提交Flink任务。 在任意能连接到Kubernetes集群的节点上上传Flink安装包，并解压，将打包好的Flink WordCountjar包进行上传，同时在node5节点上启动对应的Socket服务，执行如下命令提交Flink任务： ``` [root@node4 ~]# cd /software/flink-1.16.0/bin/ [root@node4 bin]# ./flink run -m 192.168.179.4:30081 -c com.mashibing.flinkjava.code.chapter_k8s.WordCount /software/FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` 向socket端口输入数据，通过kubectl logs + pods 方式查看对应的实时结果即可。 ``` #在node5节点上启动socket 服务，并输入数据 [root@node5 ~]# nc -lk 9999 hello zhangsan hello lisi hello wangwu #通过kubectl logs 查看对应的结果 [root@node1 flink-ha-session]# kubectl logs flink-taskmanager-7766758754-mfjgd ... ... (zhangsan,1) (hello,2) (lisi,1) (hello,3) (wangwu,1) ``` ###### 1.8.1.1.2.6 **停止集群** HA Session Cluster 集群停止时，在对应的目录下执行如下命令停止相应服务： ``` kubectl delete -f ./flink-configuration-configmap.yaml kubectl delete -f ./jobmanager-cluster-role.yaml kubectl delete -f ./jobmanager-rest-service.yaml kubectl delete -f ./jobmanager-service.yaml kubectl delete -f ./jobmanager-session-deployment-ha.yaml kubectl delete -f ./taskmanager-session-deployment.yaml ``` 注意：也可以进入对应yaml文件目录，直接执行 kubectl delete -f ./ 全部部署也可以。除了执行以上命令之外，还需要删除生成的额外的<cluster-id>-xxx configmap对象，命令如下： ``` #查看kubernetes 集群configmap 对象 [root@node1 flink-ha-session]# kubectl get configmap NAME DATA AGE kube-root-ca.crt 1 3d myk8s-cluster-config-map 3 30m #删除对应的configmap对象 [root@node1 flink-ha-session]# kubectl delete configmap myk8s-cluster-config-map configmap "myk8s-cluster-config-map" deleted ``` #### 1.8.1.2 *Application Cluster部署* Application模式即一个作业使用单独的一个专用Flink集群，每个Flink作业的JobManager和TaskManager隔离。在Kubernetes 上部署Application Cluster集群时，与Session Cluster集群部署一样，一般至少包含三个组件： * 运行JobManager的Deployment * 运行TaskManager的Deployment * 暴露JobManager上的REST和UI端口的Service ##### 1.8.1.2.1 **非HA Application Cluster部署及测试** ###### 1.8.1.2.1.1 **准备deployment文件** * **flink-configuration-configmap.yaml** ``` apiVersion: v1 kind: ConfigMap metadata: name: flink-config labels: app: flink data: flink-conf.yaml: |+ jobmanager.rpc.address: flink-jobmanager taskmanager.numberOfTaskSlots: 2 blob.server.port: 6124 jobmanager.rpc.port: 6123 taskmanager.rpc.port: 6122 queryable-state.proxy.ports: 6125 jobmanager.memory.process.size: 1600m taskmanager.memory.process.size: 1728m parallelism.default: 2 log4j-console.properties: |+ # This affects logging for both user code and Flink rootLogger.level = INFO rootLogger.appenderRef.console.ref = ConsoleAppender rootLogger.appenderRef.rolling.ref = RollingFileAppender # Uncomment this if you want to _only_ change Flink's logging #logger.flink.name = org.apache.flink #logger.flink.level = INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. logger.akka.name = akka logger.akka.level = INFO logger.kafka.name= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper logger.zookeeper.level = INFO # Log all infos to the console appender.console.name = ConsoleAppender appender.console.type = CONSOLE appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Log all infos in the given rolling file appender.rolling.name = RollingFileAppender appender.rolling.type = RollingFile appender.rolling.append = false appender.rolling.fileName = ${sys:log.file} appender.rolling.filePattern = ${sys:log.file}.%i appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n appender.rolling.policies.type = Policies appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size=100MB appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 10 # Suppress the irrelevant (wrong) warnings from the Netty channel handler logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF ``` * **jobmanager-rest-service.yaml** ``` apiVersion: v1 kind: Service metadata: name: flink-jobmanager-rest spec: type: NodePort ports: - name: rest port: 8081 targetPort: 8081 nodePort: 30081 selector: app: flink component: jobmanager ``` * **jobmanager-service.yaml** ``` apiVersion: v1 kind: Service metadata: name: flink-jobmanager spec: type: ClusterIP ports: - name: rpc port: 6123 - name: blob-server port: 6124 - name: webui port: 8081 selector: app: flink component: jobmanager ``` * **jobmanager-application-deployment-non-ha.yaml** ``` apiVersion: batch/v1 kind: Job metadata: name: flink-jobmanager spec: template: metadata: labels: app: flink component: jobmanager spec: restartPolicy: OnFailure containers: - name: jobmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 # 以下参数中 standalone-job、--job-classname 是固定的，后面一个参数是运行的Flink 主类，还可以继续跟参数，例如："--input","/xxx/xx" args: ["standalone-job", "--job-classname", "com.mashibing.flinkjava.code.chapter_k8s.WordCount"] # optional arguments: ["--job-id", "", "--fromSavepoint", "/path/to/savepoint", "--allowNonRestoredState"] ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime - name: job-artifacts-volume mountPath: /opt/flink/usrlib #这里必须指定该路径，注意是usrlib ，Flink会从该路径读取用户自己的jar包 securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: job-artifacts-volume hostPath: path: /software/flinkjar #将主类对应的jar包放入到该路径下(该路径要在k8s集群所有节点都要有才可以)，可以自定义路径，直接会挂载到容器中 - name: localtime #同步本机时间到容器 hostPath: path: /etc/localtime type: '' ``` 注意：本地/software/flinkjar 中需要上传运行Flink 主类对应的jar包，并且所有的Kubernetes节点都要有。最好采用nfs公共磁盘挂载。 * **taskmanager-application-deployment.yaml** ``` apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager spec: replicas: 2 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: containers: - name: taskmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 args: ["taskmanager"] ports: - containerPort: 6122 name: rpc - containerPort: 6125 name: query-state livenessProbe: tcpSocket: port: 6122 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/ - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime - name: job-artifacts-volume #这里必须指定该路径，注意是usrlib ，Flink会从该路径读取用户自己的jar包 mountPath: /opt/flink/usrlib securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: job-artifacts-volume #将主类对应的jar包放入到该路径下(该路径要在k8s集群所有节点都要有才可以)，可以自定义路径，直接会挂载到容器中 hostPath: path: /software/flinkjar - name: localtime #挂载localtime文件，使容器时间与宿主机一致 hostPath: path: /etc/localtime type: '' ``` ###### 1.8.1.2.1.2 **部署yaml文件** 由于Application模式部署即执行对应的Flink job，所以部署yaml文件前，需要在Kubernetes集群每台节点创建“/software/flinkjar”目录中上传对应的Flink jar包，然后在对应节点上启动socket服务： ``` [root@node5 ~]# nc -lk 9999 ``` 然后在对应的目录中执行如下命令，进行yaml文件部署 ``` kubectl create -f flink-configuration-configmap.yaml kubectl create -f jobmanager-application-deployment-non-ha.yaml kubectl create -f jobmanager-rest-service.yaml kubectl create -f taskmanager-application-deployment.yaml ``` 注意：也可以进入对应yaml文件目录，直接执行 kubectl create -f ./ 全部部署也可以。 ###### 1.8.1.2.1.3 **验证部署情况** ``` [root@node1 flink-nonha-application]# kubectl get all NAME READY STATUS RESTARTS AGE pod/flink-jobmanager-r86z9 1/1 Running 0 8s pod/flink-taskmanager-7b7b7d9758-7k7bc 1/1 Running 0 7s pod/flink-taskmanager-7b7b7d9758-crkpw 1/1 Running 0 7s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/flink-jobmanager-rest NodePort 10.103.201.10 8081:30081/TCP 8s service/kubernetes ClusterIP 10.96.0.1 443/TCP 2d22h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/flink-taskmanager 2/2 2 2 7s NAME DESIRED CURRENT READY AGE replicaset.apps/flink-taskmanager-7b7b7d9758 2 2 2 7s NAME COMPLETIONS DURATION AGE job.batch/flink-jobmanager 0/1 8s 8s ``` 在浏览器输入：[http://192.168.179.4:30081/即可访问Flink](http://node1:30081/即可访问Flink) Application集群WebUI。浏览器中输入的ip可以是K8s集群中任意节点的IP。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/35aa773ae30d4d5faab61f122df3dd19.png) 向node5 socket端口中输入数据并查询结果： ``` #向node5节点socket输入数据 [root@node5 ~]# nc -lk 9999 hello a hello b hello c #查看结果 [root@node1 flink-nonha-application]# kubectl logs flink-taskmanager-7b7b7d9758-crkpw ... ... 1> (hello,1) 2> (a,1) 1> (hello,2) 1> (b,1) 1> (hello,3) 1> (c,1) ``` ##### 1.8.1.2.2 **HA Application Cluster部署及测试** 同样，基于Kubernetes Application Cluster 部署模式也支持HA模式。配置HA  Application模式与非HA Application模式相比不再需要jobmanager-server.yaml文件。 ###### 1.8.1.2.2.1 **准备deployment文件** * **flink-configuration-configmap.yaml** 该文件相比于非HA Session模式对应文件增加了以下行： ``` kubernetes.cluster-id: myk8s #给kubernets集群取个名字 high-availability: kubernetes #指定高可用模式 high-availability.storageDir: hdfs://mycluster/flink/recovery #指定元数据存储目录为hdfs路径 restart-strategy: fixed-delay #指定重启策略 restart-strategy.fixed-delay.attempts: 10 #指定重启尝试次数 ``` 以上配置使用kubernetes来进行协调FlinkHA，部署相应flink-configuration-configmap.yaml文件后在Kubernetes中会额外生成<cluster-id>-xxx对应的configmap对象，此对象记录Flink 集群中提交的job元数据。完整的flink-configuration-configmap.yaml文件如下： ``` apiVersion: v1 kind: ConfigMap metadata: name: flink-config labels: app: flink data: flink-conf.yaml: |+ jobmanager.rpc.address: flink-jobmanager taskmanager.numberOfTaskSlots: 2 blob.server.port: 6124 jobmanager.rpc.port: 6123 taskmanager.rpc.port: 6122 queryable-state.proxy.ports: 6125 jobmanager.memory.process.size: 1600m taskmanager.memory.process.size: 1728m parallelism.default: 2 kubernetes.cluster-id: myk8s #给kubernets集群取个名字 high-availability: kubernetes #指定高可用模式 high-availability.storageDir: hdfs://mycluster/flink/recovery #指定元数据存储目录为hdfs路径 restart-strategy: fixed-delay #指定重启策略 restart-strategy.fixed-delay.attempts: 10 #指定重启尝试次数 log4j-console.properties: |+ # This affects logging for both user code and Flink rootLogger.level = INFO rootLogger.appenderRef.console.ref = ConsoleAppender rootLogger.appenderRef.rolling.ref = RollingFileAppender # Uncomment this if you want to _only_ change Flink's logging #logger.flink.name = org.apache.flink #logger.flink.level = INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. logger.akka.name = akka logger.akka.level = INFO logger.kafka.name= org.apache.kafka logger.kafka.level = INFO logger.hadoop.name = org.apache.hadoop logger.hadoop.level = INFO logger.zookeeper.name = org.apache.zookeeper logger.zookeeper.level = INFO # Log all infos to the console appender.console.name = ConsoleAppender appender.console.type = CONSOLE appender.console.layout.type = PatternLayout appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n # Log all infos in the given rolling file appender.rolling.name = RollingFileAppender appender.rolling.type = RollingFile appender.rolling.append = false appender.rolling.fileName = ${sys:log.file} appender.rolling.filePattern = ${sys:log.file}.%i appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n appender.rolling.policies.type = Policies appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size=100MB appender.rolling.strategy.type = DefaultRolloverStrategy appender.rolling.strategy.max = 10 # Suppress the irrelevant (wrong) warnings from the Netty channel handler logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline logger.netty.level = OFF ``` * **jobmanager-rest-service.yaml** 改文件与非HA Application模式部署对应的文件一样。 ``` apiVersion: v1 kind: Service metadata: name: flink-jobmanager-rest spec: type: NodePort ports: - name: rest port: 8081 targetPort: 8081 nodePort: 30081 selector: app: flink component: jobmanager ``` * **jobmanager-cluster-role.yaml** 在k8s中是基于角色授权的，创建用户时需要绑定对应的角色，在JobManager HA 部署案例中需要操作对应的配置ConfigMap文件。这里通过jobmanager-cluster-role.yaml创建一个ClusterRole，然后再创建用户，将用户绑定到该集群角色，拥有对ConfigMap操作权限。 ``` kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: configmaps-role #创建ClusterRole的名称 rules: - apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] #指定操作的资源为configmaps verbs: ["create", "edit", "delete","get", "watch", "list","update"] #指定操作的权限 ``` 以上创建完成，再创建一个用户叫flink-service-account，后续将用户绑定到该角色。 ``` kubectl create serviceaccount flink-service-account ``` 把flink-service-account用户绑定到集群角色： ``` kubectl create clusterrolebinding flink-role-binding-serviceaccount --clusterrole=configmaps-role --serviceaccount=default:flink-service-account ``` * **jobmanager-application-deployment-ha.yaml** ``` apiVersion: batch/v1 kind: Job metadata: name: flink-jobmanager spec: #这里配置2个副本 parallelism: 2 # Set the value to greater than 1 to start standby JobManagers template: metadata: labels: app: flink component: jobmanager spec: hostAliases: #向容器/etc/hosts中加入ip与节点名称映射，pod找HDFS集群时需要使用 - ip: 192.168.179.4 hostnames: - "node1" - ip: 192.168.179.5 hostnames: - "node2" - ip: 192.168.179.6 hostnames: - "node3" - ip: 192.168.179.7 hostnames: - "node4" - ip: 192.168.179.8 hostnames: - "node5" restartPolicy: OnFailure containers: - name: jobmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 env: - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HADOOP_CLASSPATH #这里由于在flink pod内使用到HDFS ，需要把宿主机的HDFS配置文件挂载进来，并配置HADOOP_CLASSPATH环境变量，可以通过在宿主机执行echo `hadoop classpath`来参考这里导入的路径，记得要把Hadoop路径改成挂载到Pod中的路径 value: /opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/* # The following args overwrite the value of jobmanager.rpc.address configured in the configuration config map to POD_IP. # 以下参数中 standalone-job、--job-classname 是固定的，后面一个参数是运行的Flink 主类，还可以继续跟参数，例如："--input","/xxx/xx" args: ["standalone-job","--host","$(POD_IP)", "--job-classname", "com.mashibing.flinkjava.code.chapter_k8s.WordCount"] # optional arguments: ["--job-id", "", "--fromSavepoint", "/path/to/savepoint", "--allowNonRestoredState"] ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob-server - containerPort: 8081 name: webui livenessProbe: tcpSocket: port: 6123 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime - name: job-artifacts-volume mountPath: /opt/flink/usrlib #这里必须指定该路径，注意是usrlib ，Flink会从该路径读取用户自己的jar包 - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 mountPath: /opt/hadoop securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary serviceAccountName: flink-service-account # Service account which has the permissions to create, edit, delete ConfigMaps volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: job-artifacts-volume hostPath: path: /software/flinkjar #将主类对应的jar包放入到该路径下(该路径要在k8s集群所有节点都要有才可以)，可以自定义路径，直接会挂载到容器中 - name: localtime #同步本机时间到容器 hostPath: path: /etc/localtime type: '' - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 hostPath: path: /software/hadoop-3.3.4 type: '' ``` * **taskmanager-application-deployment.yaml** ``` apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager spec: replicas: 2 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: hostAliases: #向容器/etc/hosts中加入ip与节点名称映射，pod找HDFS集群时需要使用 - ip: 192.168.179.4 hostnames: - "node1" - ip: 192.168.179.5 hostnames: - "node2" - ip: 192.168.179.6 hostnames: - "node3" - ip: 192.168.179.7 hostnames: - "node4" - ip: 192.168.179.8 hostnames: - "node5" containers: - name: taskmanager image: flink:1.16.0-scala_2.12-java8 #指定Flink的镜像，可以从https://hub.docker.com/ 网站上查找 env: - name: HADOOP_CLASSPATH #这里由于在flink pod内使用到HDFS ，需要把宿主机的HDFS配置文件挂载进来，并配置HADOOP_CLASSPATH环境变量，可以通过在宿主机执行echo `hadoop classpath`来参考这里导入的路径，记得要把Hadoop路径改成挂载到Pod中的路径 value: /opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/* args: ["taskmanager"] ports: - containerPort: 6122 name: rpc - containerPort: 6125 name: query-state livenessProbe: tcpSocket: port: 6122 initialDelaySeconds: 30 periodSeconds: 60 volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/ - name: localtime #挂载localtime文件，使容器时间与宿主机一致 mountPath: /etc/localtime - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 mountPath: /opt/hadoop - name: job-artifacts-volume #这里必须指定该路径，注意是usrlib ，Flink会从该路径读取用户自己的jar包 mountPath: /opt/flink/usrlib securityContext: runAsUser: 9999 # refers to user _flink_ from official flink image, change if necessary serviceAccountName: flink-service-account #指定seviceAccountName 用户名 volumes: - name: flink-config-volume configMap: name: flink-config items: - key: flink-conf.yaml path: flink-conf.yaml - key: log4j-console.properties path: log4j-console.properties - name: job-artifacts-volume #将主类对应的jar包放入到该路径下(该路径要在k8s集群所有节点都要有才可以)，可以自定义路径，直接会挂载到容器中 hostPath: path: /software/flinkjar - name: localtime #挂载localtime文件，使容器时间与宿主机一致 hostPath: path: /etc/localtime type: '' - name: hadoop-conf #将宿主机中的配置好的hadoop的安装包挂载到容器 hostPath: path: /software/hadoop-3.3.4 type: '' ``` 注意：关于Flink的镜像可以从https://hub.docker.com/网站中搜索下载。 由于在flink pod内使用到HDFS ，需要把宿主机的HDFS配置文件挂到Flink JobManager和TaskManager中并配置HADOOP_CLASSPATH环境变量，可以通过在宿主机执行echo `hadoop classpath`来参考HADOOP_CLASSPATH环境变量对应的value值路径，记得要把Hadoop路径改成挂载到Pod中的路径。 以上配置文件可以从资料“flink-ha-application.zip”中获取。 ###### 1.8.1.2.2.2 **部署yaml文件** 由于HA模式使用到了HDFS集群，所以这里应该首先启动HDFS集群然后再部署对应的yaml文件。 ``` #启动zookeeper [root@node3 ~]# zkServer.sh start [root@node4 ~]# zkServer.sh start [root@node5 ~]# zkServer.sh start #启动HDFS集群 [root@node1 ~]# start-all.sh ``` 部署之前记得执行上一小节中创建用户与绑定用户到角色命令，如果执行过不必重复创建执行。 ``` kubectl create serviceaccount flink-service-account kubectl create clusterrolebinding flink-role-binding-serviceaccount --clusterrole=configmaps-role --serviceaccount=default:flink-service-account ``` 在对应的目录中执行如下命令，部署yaml文件 ``` kubectl create -f ./flink-configuration-configmap.yaml kubectl create -f ./jobmanager-application-deployment-ha.yaml kubectl create -f ./jobmanager-cluster-role.yaml kubectl create -f ./jobmanager-rest-service.yaml kubectl create -f ./taskmanager-application-deployment.yaml ``` 注意：也可以进入对应yaml文件目录，直接执行 kubectl create -f ./ 全部部署也可以。 ###### 1.8.1.2.2.3 **验证部署情况** ``` [root@node1 flink-ha-application]# kubectl get all NAME READY STATUS RESTARTS AGE pod/flink-jobmanager-bbp97 1/1 Running 0 60s pod/flink-jobmanager-xvjts 1/1 Running 0 60s pod/flink-taskmanager-58685d568f-7lhhg 1/1 Running 0 60s pod/flink-taskmanager-58685d568f-nj6nf 1/1 Running 0 60s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/flink-jobmanager-rest NodePort 10.97.107.147 8081:30081/TCP 60s service/kubernetes ClusterIP 10.96.0.1 443/TCP 3d1h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/flink-taskmanager 2/2 2 2 60s NAME DESIRED CURRENT READY AGE replicaset.apps/flink-taskmanager-58685d568f 2 2 2 60s NAME COMPLETIONS DURATION AGE job.batch/flink-jobmanager 0/1 of 2 60s 60s ``` 在浏览器输入：[http://192.168.179.4:30081/即可访问Flink](http://node1:30081/即可访问Flink) Application集群WebUI，可以看到对应主类的Flink job已经处于运行状态。 ![](file:///C:\Temp\ksohtml16136\wps30.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/e267196b4441451fb926460c2e92806c.png) ###### 1.8.1.2.2.4 **HA高可用验证** HA验证方式与HA Session Cluster 部署方式验证方式一样。可以参考HA Session Cluster 部署方式HA高可用验证。 ### 1.8.2 **Native Kubernetes部署** Flink的Native Kubernetes集成允许我们直接将Flink部署到正在运行的Kubernetes集群上，Flink能够根据所需的资源通过与Kubernetes 集群通信动态分配和取消分配taskmanager，其部署原理图如下： ![](file:///C:\Temp\ksohtml21784\wps12.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/25b891e0e9d244aabc8dc144b145e393.png) Flink Native kubernetes集成需要一个Kubernetes集群，并且Kubernetes版本需要大于等于1.9 版本，Flink Native kubernetes支持Flink Client 创建Session Cluster和Application Cluster集群部署并进行任务提交，也支持对应HA的任务执行，但是需要自定义pod的yaml资源清单文件实现，这与基于Kubernetes中Session Cluster和Application Cluster类似，这里不再两种模式的HA方式，具体HA方式参考Kubernetes部署。 Flink Native kubernetes的Session Cluster和Application Cluster集群部署两种方式都需要指定具备RBAC(Role-based access control)权限的serviceaccount，以便创建、删除集群内的Pod。 下面首先创建serviceaccount，并绑定到edit的ClusterRole角色，方便创建和删除集群pod。 1) **创建名称为flink的serviceaccount** ``` #创建 serviceaccount ，名称为flink [root@node1 ~]# kubectl create serviceaccount flink ``` 查看创建好的seviceaccount: ``` [root@node1 ~]# kubectl get serviceaccount NAME SECRETS AGE default 0 5d21h flink 0 7m26s ``` 2) **为serviceaccount flink绑定角色** ``` #这里创建clusterrolebinding名称为flink-role-binding-flink，绑定的serviceaccount 名称是flink，绑定到的clusterrole角色为edit [root@node1 ~]# kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=default:flink ``` 查看创建的clusterrolebinding： ``` [root@node1 ~]# kubectl get clusterrolebinding |grep flink-role-binding-flink ``` #### 1.8.2.1 **Session Cluster模式** Session Cluster模式通过Flink Client连接Kubernetes集群进行创建，Flink Client必须在Kubernetes集群内的某台节点，否则Flink Client 连接不上Kubernetes集群。这里选择node3节点当做Flink客户端，将Flink安装包上传至node3节点，并解压。 ``` #将Flink安装包上传到/software目录下，并解压 [root@node3 ~]# cd /software/ [root@node3 software]# tar -zxvf ./flink-1.16.0-bin-scala_2.12.tgz ``` 通过以下步骤创建Session Cluster并提交任务、操作任务。 ##### 1.8.2.1.1 **启动Session集群** ``` [root@node3 ~]# cd /software/flink-1.16.0/bin/ [root@node3 bin]#./kubernetes-session.sh \ -Dkubernetes.container.image=flink:1.16.0-scala_2.12-java8\ -Dkubernetes.jobmanager.service-account=flink \ -Dkubernetes.cluster-id=my-first-flink-cluster\ -Dkubernetes.rest-service.exposed.type=NodePort\ -Dtaskmanager.memory.process.size=1024m \ -Dkubernetes.taskmanager.cpu=1 \ -Dtaskmanager.numberOfTaskSlots=4 \ -Dresourcemanager.taskmanager-timeout=60000 ``` 以上参数解释如下: * kubernetes.container.image：指定Flink image镜像。 * kubernetes.jobmanager.service-account：指定操作的serviceaccount名称 * kubernetes.cluster-id：指定Flink Session Cluster对应Kubernetes集群的id,名字随意，指定后在Kubernetes集群中会有对应名称的Deployment。 * kubernetes.rest-service.exposed.type：指定创建的JobManager rest ui服务对外暴露服务的方式。 * taskmanager.memory.process.size：指定每个TaskManager使用的总内存。 * kubernetes.taskmanager.cpu：指定TaskManager使用的cup个数，默认为1。 * taskmanager.numberOfTaskSlots：指定TaskManager 对应的slot个数，默认为1。 * resourcemanager.taskmanager-timeout:Flink默认会自动取消空闲的TaskManager以避免浪费资源，一旦对应的TaskManager Pod停止，就不能再查看对应Pod的日志，可以设置该参数指定空闲的TaskManager多久可以被销毁，默认30000，即30s。 更多参数可以参考Flink官网配置： [https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/config/#kubernetes](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/config/#kubernetes) 此外，以上flink:1.16.0-scala_2.12-java8 镜像下载需要科学上网工具，如何下载不下来也可以直接导入资料中的“flink1.16.0-scala_2.12-java8.tar”镜像，方法如下： ``` #将flink1.16.0-scala_2.12-java8.tar上传到对应节点，这里上传至Master节点，然后执行如下命令导入镜像 [root@node1 ~]# docker load -i /software/flink1.16.0-scala_2.12-java8.tar f4a670ac65b6: Loading layer [==================================================>] 80.31MB/80.31MB aa2e51f5ab8a: Loading layer [==================================================>] 36.6MB/36.6MB 9b110457aa6d: Loading layer [==================================================>] 108.8MB/108.8MB 6428f4fb9d66: Loading layer [==================================================>] 2.56kB/2.56kB 2da66d05adf1: Loading layer [==================================================>] 11.8MB/11.8MB 2d441efefbd5: Loading layer [==================================================>] 2.3MB/2.3MB 4463bbe0b1a3: Loading layer [==================================================>] 3.254MB/3.254MB 0236f57a147f: Loading layer [==================================================>] 2.56kB/2.56kB 3d6f741552ee: Loading layer [==================================================>] 520.8MB/520.8MB eb617b6c57b4: Loading layer [==================================================>] 7.168kB/7.168kB Loaded image: flink:1.16.0-scala_2.12-java8 #查看导入的镜像 [root@node1 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE flink 1.16.0-scala_2.12-java8 584a51fe68ac 10 days ago 759MB ``` 启动Kubernetes Session Cluster 集群后，可以看到打印出外网访问的IP及端口，可以登录检查启动的Session集群是否正常。 ![](file:///C:\Temp\ksohtml21784\wps17.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/11abf7c8ad4c43f7a8f1fc21aa6b8846.png) ##### 1.8.2.1.2 **停止Session Cluster集群** 当启动Session Cluster后，在Kubernetes集群中会有名称与“kubernetes.cluster-id”参数配置一样的deployment，查看方式如下： ``` [root@node1 ~]# kubectl get all NAME READY STATUS RESTARTS AGE pod/my-first-flink-cluster-577f9d9d9b-wk9c7 1/1 Running 0 6m10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 5d21h service/my-first-flink-cluster ClusterIP None 6123/TCP,6124/TCP 6m10s service/my-first-flink-cluster-rest NodePort 10.101.64.92 8081:32755/TCP 6m10s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/my-first-flink-cluster 1/1 1 1 6m10s NAME DESIRED CURRENT READY AGE replicaset.apps/my-first-flink-cluster-577f9d9d9b 1 1 1 6m10s ``` 停止对应的Session Cluster集群有两种方式，一种是通过Kubernetes命令停止对应的Deployment，另一种是通过Flink Client停止。 * **Kubernetes命令停止Session Cluster集群：** ``` [root@node1 ~]# kubectl delete deployment.apps/my-first-flink-cluster ``` * **Flink Client停止Session Cluster集群：** ``` [root@node3 bin]# echo 'stop' | ./kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster -Dexecution.attached=true ``` ##### 1.8.2.1.3 **提交任务** 这里向启动的Flink Session Cluster 集群中提交任务，任务与之前Flink测试任务一样：读取node5节点上socket 9999 端口数据实时统计WordCount。按照以下步骤来进行任务提交： 1) **在node5节点上启动Socket服务。** ``` [root@node5 ~]# nc -lk 9999 ``` 2) **在node3节点上传用户打好的jar包，放入/software/flinkjar目录中** ``` [root@node3 ~]# mkdir -p /software/flinkjar [root@node3 flinkjar]# ls FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` 3) **执行如下命令提交任务并测试：** ``` [root@node3 ~]# cd /software/flink-1.16.0/bin/ [root@node3 bin]# ./flink run \ --target kubernetes-session \ -Dkubernetes.cluster-id=my-first-flink-cluster \ -c com.mashibing.flinkjava.code.chapter_k8s.WordCount \ /software/flinkjar/FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` ![](file:///C:\Temp\ksohtml21784\wps18.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/34515d8092824fa98889e3c2cb442357.png) session集群中提交任务需要指定--target为kubernetes-session即可，另外可以在提交任务时指定参数-d ，在向Kubernetes中Flink Session Cluster集群提交任务后退出客户端。 ``` [root@node3 bin]# ./flink run -d \ --target kubernetes-session \ -Dkubernetes.cluster-id=my-first-flink-cluster \ -c com.mashibing.flinkjava.code.chapter_k8s.WordCount \ /software/flinkjar/FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` 可以在node5节点socket中输入以下数据： ``` [root@node5 ~]# nc -lk 9999 hello a hello b hello c ``` 查看stdout结果需要使用kubernetes集群命令来查询： ``` [root@node1 software]# kubectl logs my-first-flink-cluster-taskmanager-1-1 ... .... (hello,1) (a,1) (hello,2) (b,1) (hello,3) (c,1) ``` ##### 1.8.2.1.4 **测试资源申请与释放** 默认创建Flink Session Cluster集群指定了每个TaskManager 具备1core和4个slot,当向Flink Session Cluster 集群中提交一个任务时（默认1个并行度）会使用1个slot，当提交多次Flink任务时可以看到FlinkSession Cluster集群可以动态申请TaskManager ![](file:///C:\Temp\ksohtml21784\wps19.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/10030137efbf418489fae3fb3db29b61.png) 通过以上测试会发现，当提交多个Flink 任务时，这些任务共用一个Flink Session Cluster集群，每提交一个Flink任务会分配1个slot，当Session Cluster Slot不够时Kubernetes集群会动态启动新的TaskManager。查看集群中提交的任务： ``` [root@node3 bin]# ./flink list \ --target kubernetes-session \ -Dkubernetes.cluster-id=my-first-flink-cluster ``` 可以通过以下命令取消对应任务执行： ``` ./flink cancel \ --target kubernetes-session \ -Dkubernetes.cluster-id=my-first-flink-cluster \ 任务id ``` 当把集群任务一个个取消后，集群TaskManager会经过“resourcemanager.taskmanager-timeout”参数指定的时间后动态释放以便节省资源，在创建Flink Session Cluster集群时该参数指定60000ms ,即1分钟，Kubernetes集群会在1分钟后自动删除空闲的TaskManager。 ![](file:///C:\Temp\ksohtml21784\wps20.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/4c170c0c0a7f4dea9a6d4e410a5b4d19.png) #### 1.8.2.2 **Application Cluster模式** 在生产环境中，建议基于Application Cluster集群来部署Flink应用程序，因为这些模式为应用程序提供了更好的隔离。Flink基于Native Kubernetes 同样也支持Application Cluster模式，在Flink客户端指定对应的命令可以直接为每个Flink Application 创建一个单独的集群。Flink Application Cluster 中通过Flink客户端提交Flink任务需要自己提前构建flink的镜像，将用户对应执行的jar包上传到镜像内，目前官方没有提供外部指定参数方式来指定用户jar包。 ##### 1.8.2.2.1 **Harbor构建私有镜像仓库** 在企业中使用Kubernetes时，为了方便下载和管理镜像一般都会构建本地的私有镜像仓库。Harbor正是一个用于存储镜像的企业级Registry服务，我们可以通过Harbor来存储容器镜像构建企业本地的镜像仓库。 这里我们选择一台Kubernetes集群外的一台节点搭建Harbor，当然也可以选择Kubernetes集群内的一台节点。 | **节点IP** | **节点名称** | **Harbor** | | ---------------- | ------------------ | ---------------- | | 192.168.179.4 | node1 | | | 192.168.179.5 | node2 | | | 192.168.179.6 | node3 | | | 192.168.179.7 | node4 | ★ | | 192.168.179.8 | node5 | | 1) **安装docker** ``` #准备docker-ce对应的repo文件 [root@node4 ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo #安装docker-ce [root@node4 ~]# yum -y install docker-ce #设置docker开机启动，并启动docker [root@node4 ~]# systemctl enable docker [root@node4 ~]# systemctl start docker ``` 2) **安装docker-compose** 后续需要docker-compose编排工具进行harbor的启停，这里需要安装docker-compose。 ``` #下载docker-compose的二进制文件,改文件如果下载不下来可以参照资料中“docker-compose-Linux-x86_64”文件 [root@node4 ~]# wget https://github.com/docker/compose/releases/download/1.25.0/docker-compose-Linux-x86_64 #移动二进制文件到/usr/bin目录，并更名为docker-compose [root@node4 ~]# mv docker-compose-Linux-x86_64 /usr/bin/docker-compose # 为二进制文件添加可执行权限 chmod +x /usr/bin/docker-compose #以上操作完成后，查看docker-compse版本 [root@node4 ~]# docker-compose version docker-compose version 1.25.0, build 0a186604 docker-py version: 4.1.0 CPython version: 3.7.4 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019 ``` 3. **获取Harbor安装文件并解压** ``` #下载harbor离线安装包，如果下载不下来参考资料中“harbor-offline-installer-v2.5.1.tgz”文件 [root@node4 ~]# wget https://github.com/goharbor/harbor/releases/download/v2.5.1/harbor-offline-installer-v2.5.1.tgz #解压下载好的安装包 [root@node4 ~]# tar -zxvf ./harbor-offline-installer-v2.5.1.tgz ``` 4. **修改Harbor配置文件** 搭建Harbor本地镜像仓库时，可以使用ssl安全访问私有镜像仓库，但是Flink Native Kubernetes中Application模式提交任务下载镜像默认是https方式访问仓库地址，所以这里需要给Harbor配置ssl证书，具体证书参照资料“6864844_kubemsb.com_nginx.zip”文件。 * **准备Harbor的harbor.yml配置文件：** ``` [root@node4 ~]# cd harbor [root@node4 harbor]# mv harbor.yml.tmpl harbor.yml #将资料中“6864844_kubemsb.com_nginx.zip”压缩文件进行解压，然后把证书的pem文件和key文件上传到该目录 [root@node4 harbor]# ls /root/harbor 6864844_kubemsb.com.key 6864844_kubemsb.com.pem ``` * **修改harbor.yml文件内容如下：** ``` # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: www.kubemsb.com # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /root/harbor/6864844_kubemsb.com.pem private_key: /root/harbor/6864844_kubemsb.com.key # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: 123456 ... ... ``` 注意harbor.yml文件中只需要配置hostname、certificate、private_key、harbor_admin_password即可。 5. **执行预备脚本并安装** ``` #执行预备脚本，检查安装所需镜像 [root@node4 harbor]# ./prepare prepare base dir is set to /root/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/core/env Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/registry/passwd Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/db/env Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /config/jobservice/config.yml Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /data/secret/keys/secretkey Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir #执行安装脚本 [root@node4 harbor]# ./install.sh ... [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating registryctl ... done Creating harbor-db ... done Creating registry ... done Creating harbor-portal ... done Creating redis ... done Creating harbor-core ... done Creating nginx ... done Creating harbor-jobservice ... done ✔ ----Harbor has been installed and started successfully.---- ``` 6. **验证Harbor情况** ``` #使用docker ps 命令检查是否是运行9个镜像，如下 [root@node4 harbor]# docker ps CONTAINER ID IMAGE a81fbd05dc13 goharbor/harbor-jobservice:v2.5.1 c374cf3d741a goharbor/nginx-photon:v2.5.1 152c165b0804 goharbor/harbor-core:v2.5.1 4e48926df8b0 goharbor/redis-photon:v2.5.1 6d514441a600 goharbor/harbor-db:v2.5.1 0a170f716955 goharbor/registry-photon:v2.5.1 a8d99c7b2421 goharbor/harbor-portal:v2.5.1 2b808612f108 goharbor/harbor-registryctl:v2.5.1 69c2c7818e6a goharbor/harbor-log:v2.5.1 ``` 注意：安装harbor后，查看对应的 docker images 是否是9个，不是9个需要重新启动harbor。重启Harbor的命令如下： ``` [root@node4 harbor]# cd /root/harbor #停止harbor服务 [root@node4 harbor]# docker-compose down #启动harbor服务 [root@node4 harbor]# docker-compose up -d ``` 7. **配置Kubernetes节点及harbor节点访问harbor服务** 后续Kubernetes各个节点（包括harbor节点本身）需要连接Harbor上传或者下载镜像，所以这里配置各个节点访问harbor。配置各个节点hosts文件配置www.kubemsb.com的映射: ``` #这里在node1-node4各个节点修改/etc/hosts文件，加入以下配置 192.168.179.7 www.kubemsb.com ``` kubernetes集群所有节点配置harbor仓库（包括harbor节点本身也要配置）： ``` #kubernetes 集群各个节点配置/etc/docker/daemon.json文件，追加"insecure-registries": ["https://www.kubemsb.com"] vim /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "insecure-registries": ["https://www.kubemsb.com"] } #harbor节点配置 /etc/docker/daemon.json文件 { "insecure-registries": ["https://www.kubemsb.com"] } ``` 以上配置完成后，配置的每台节点需要重启docker，重点需要注意harbor节点docker重启后是否是对应有9个image，如果不是9个需要使用docker-compose down 停止harbor集群后再次使用命令docker-compse up -d 启动。 ``` systemctl restart docker ``` 检查每个节点是否能正常连接Harbor，这里每台节点连接harbor前必须需要执行如下命令登录下harbor私有镜像仓库。 ``` docker login www.kubemsb.com 输入用户：admin 输入密码：123456 ``` 8. **访问Harbor UI界面** 在window本地“C:\Windows\System32\drivers\etc\hosts”中加入对应的映射。 ``` 192.168.179.7 www.kubemsb.com ``` 然后浏览器访问harbor(只能是域名访问，不能IP访问)，用户名为admin，密码为配置的123456。 ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/3385b90f53db463ba6c742dc2b08ba29.png) ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/b73afe0351014104a0bebd6a4ddfd50e.png) 9. **测试Harbor** 使用docker下载nginx镜像并上传至harbor，然后通过docker从harbor中下载该上传镜像，测试是否能正常从harbor下载存储管理的镜像。具体操作如下： ``` #docker 下载nginx镜像 [root@node1 ~]# docker pull nginx:1.15-alpine #检查镜像 [root@node1 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx 1.15-alpine dd025cdfe837 3 years ago 16.1MB #对nginx镜像进行标记打tag [root@node1 ~]# docker tag nginx:1.15-alpine www.kubemsb.com/library/nginx:v1 #检查镜像 [root@node1 software]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx 1.15-alpine dd025cdfe837 3 years ago 16.1MB www.kubemsb.com/library/nginx v1 dd025cdfe837 3 years ago 16.1MB #推送本地镜像到harbor镜像仓库 [root@node1 ~]# docker push www.kubemsb.com/library/nginx:v1 ``` 将本地镜像推送到harbor镜像仓库后，可以通过WebUI查看对应内容： ![](file:///C:\Temp\ksohtml21784\wps21.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/ed61cd24eb7a4b8aa9c55545d6310941.png) 可以在本地任何一台节点上从Harbor镜像仓库中下载镜像到本地： ``` [root@node2 ~]# docker pull www.kubemsb.com/library/nginx:v1 ``` ##### 1.8.2.2.2 **制作Flink 镜像** Native kubernetes 中Flink Application Cluster 模式提交一个任务会直接执行对应Flink任务，该任务独立生成并使用Flink Application Cluster，Flink 客户端提交命令时没有提供指定外部用户jar包的指令，所以这里需要将用户的jar包打入到flink镜像内，然后在客户端提交任务时直接指定对应的镜像即可。 这里通过Dockerflie将用户jar包打入到镜像内，并将制作好的镜像上传到harbor镜像服务器中，方便后续使用。这里可以在Kubernetes任意一台节点制作Flink镜像。 1) **使用docker下载flink镜像** ``` [root@node1 ~]# docker pull flink:1.16.0-scala_2.12-java8 ``` 2) **准备Dockerflie** ``` #在node1节点创建myflink目录 [root@node1 ~]# mkdir myflink && cd myflink #将用户jar包上传至myflink目录下，方便后续制作镜像 [root@node1 myflink]# ls FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar #创建Dockerfile，写入如下内容 FROM flink:1.16.0-scala_2.12-java8 RUN mkdir -p $FLINK_HOME/usrlib COPY ./FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar $FLINK_HOME/usrlib/ ``` 注意：以上构建的镜像中是将用户的jar包上传到镜像中的$FLINK_HOME/usrlib中，即/opt/flink/usrlib中。 3) **构建docker镜像** ``` #构建docker 镜像 [root@node1 myflink]# docker build -t myflink:v1 . #查看制作好的镜像 [root@node1 myflink]# docker images REPOSITORY TAG myflink v1 ``` 4) **上传docker镜像到Harbor服务器** ``` #对制作好的myflink:v1镜像打标签 [root@node1 myflink]# docker tag myflink:v1 www.kubemsb.com/library/myflink:v1 #查看镜像 [root@node1 myflink]# docker images REPOSITORY TAG www.kubemsb.com/library/myflink v1 myflink v1 #将镜像上传至harbor服务器 [root@node1 myflink]# docker push www.kubemsb.com/library/myflink:v1 ``` 通过Harbor WebUI检查对应的镜像： ![](file:///C:\Temp\ksohtml21784\wps22.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/d3f088c0cee14494bb29bce911af0119.png) ##### 1.8.2.2.3 **提交Flink 任务及测试** Flink Application Cluster模式提交任务后当前任务独自使用集群，Application Cluster集群启动同时任务也就运行了，所以需要在node5节点中启动socket服务，然后再执行任务提交明林，命令如下： ``` [root@node3 ~]# cd /software/flink-1.16.0/bin/ [root@node3 ~]# ./flink run-application \ --target kubernetes-application \ -Dkubernetes.cluster-id=my-first-application-cluster \ -Dkubernetes.container.image=www.kubemsb.com/library/myflink:v1 \ -Dkubernetes.rest-service.exposed.type=NodePort\ -Dkubernetes.jobmanager.service-account=flink\ -Dtaskmanager.memory.process.size=1024m \ -Dkubernetes.taskmanager.cpu=1 \ -Dtaskmanager.numberOfTaskSlots=4 \ -c com.mashibing.flinkjava.code.chapter_k8s.WordCount \ local:///opt/flink/usrlib/FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` 以上命令中--target 指定为kubernetes-application即是Application模式提交任务，其他参数与Session Cluster模式一样。 提交任务之后，可以看到对应的WebUI 的IP及端口，通过浏览器查看WebUI，可以看到一个Flink任务独立使用单独集群： ![](file:///C:\Temp\ksohtml21784\wps23.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/94763877583041f6a7a98440bef5196f.png) 当再次提交Flink任务时，新提交Flink任务同样也会创建新的Flink集群，需要指定的“kubernetes.cluster-id”名称与其他任务不同，例如提交新的Flink任务： ``` [root@node3 ~]# ./flink run-application \ --target kubernetes-application \ -Dkubernetes.cluster-id=my-first-application-cluster1 \ -Dkubernetes.container.image=www.kubemsb.com/library/myflink:v1 \ -Dkubernetes.rest-service.exposed.type=NodePort\ -Dkubernetes.jobmanager.service-account=flink\ -Dtaskmanager.memory.process.size=1024m \ -Dkubernetes.taskmanager.cpu=1 \ -Dtaskmanager.numberOfTaskSlots=4 \ -c com.mashibing.flinkjava.code.chapter_k8s.WordCount \ local:///opt/flink/usrlib/FlinkJavaCode-1.0-SNAPSHOT-jar-with-dependencies.jar ``` 提交新的任务后，同时通过Kubernetes 客户端可以看到不同名称的deployment。 ![](file:///C:\Temp\ksohtml21784\wps24.jpg)![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1667826904074/dcb56b49efbe481b99049e7bcc8ce7ce.png) ##### 1.8.2.2.4 **停止Flink集群** 可以通过Kubernets命令停止Flink Application集群。命令如下： ``` #查看Kubernetes中执行的Flink deployment [root@node1 myflink]# kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE my-first-application-cluster 1/1 1 1 17m my-first-application-cluster1 1/1 1 1 9m59s #停止对应的Flink deployment [root@node1 ~]# kubectl delete deployment/my-first-application-cluster [root@node1 ~]# kubectl delete deployment/my-first-application-cluster1 ``` ## 1.9 **Kubernetes 基于Docker Runtime** 从kubernetes 1.24开始，dockershim已经从kubelet中移除（dockershim 是 Kubernetes 的一个组件，主要目的是为了通过 CRI 操作 Docker），但因为历史问题docker却不支持kubernetes主推的CRI（容器运行时接口）标准，所以docker不能再作为kubernetes的容器运行时了，即从kubernetesv1.24开始不再使用docker了，默认使用的容器运行时是containerd。目前containerd比较新，可能存在一些功能不稳定的情况，所以这里我们也可以选择docker作为kubernetes容器运行时。 如果想继续使用docker的话，可以在kubelet和docker之间加上一个中间层cri-docker。cri-docker是一个支持CRI标准的shim（垫片）。一头通过CRI跟kubelet交互，另一头跟docker api交互，从而间接的实现了kubernetes以docker作为容器运行时。 ### 1.9.1 **节点划分** kubernetes 集群搭建节点分布： | **节点IP** | **节点名称** | **Master** | **Worker** | | ---------------- | ------------------ | ---------------- | ---------------- | | 192.168.179.4 | node1 | ★ | | | 192.168.179.5 | node2 | | ★ | | 192.168.179.6 | node3 | | ★ | | 192.168.179.7 | node4 | | | | 192.168.179.8 | node5 | | | ### 1.9.2 **升级内核** 升级操作系统内核，升级到6.06内核版本。这里所有主机均操作，包括node4,node5节点。 ``` #导入elrepo gpg key rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org #安装elrepo YUM源仓库 yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm #安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 #设置grub2默认引导为0 grub2-set-default 0 #重新生成grub2引导文件 grub2-mkconfig -o /boot/grub2/grub.cfg #更新后，需要重启，使用升级的内核生效。 reboot #重启后，需要验证内核是否为更新对应的版本 uname -r 6.0.6-1.el7.elrepo.x86_64 ``` ### 1.9.3 **配置内核转发及网桥过滤** 在所有K8S主机配置。添加网桥过滤及内核转发配置文件： ``` vim /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ``` 加载br_netfilter模块： ``` #加载br_netfilter模块 modprobe br_netfilter #查看是否加载 lsmod | grep br_netfilter ``` 加载网桥过滤及内核转发配置文件： ``` sysctl -p /etc/sysctl.d/k8s.conf ``` ### 1.9.4 **安装ipset及ipvsadm** 所有主机均需要操作。主要用于实现service转发。 ``` #安装ipset及ipvsadm yum -y install ipset ipvsadm 配置ipvsadm模块加载方式，添加需要加载的模块 vim /etc/sysconfig/modules/ipvs.modules modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack 授权、运行、检查是否加载 chmod 755 /etc/sysconfig/modules/ipvs.modules bash /etc/sysconfig/modules/ipvs.modules lsmod | grep -e ip_vs -e nf_conntrack ``` ### 1.9.5 **关闭SWAP分区** 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a。永远关闭swap分区，需要重启操作系统。 ``` #永久关闭swap分区 ,在 /etc/fstab中注释掉下面一行 vim /etc/fstab #/dev/mapper/centos-swap swap swap defaults 0 0 #重启机器 reboot ``` ### 1.9.6 **安装docker** 所有集群主机均需操作。 获取docker repo文件 ``` wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ``` 查看docker可以安装的版本： ``` yum list docker-ce.x86_64 --showduplicates | sort -r ``` 安装docker:这里指定docker版本为20.10.9版本 ``` yum -y install docker-ce-20.10.9-3.el7 ``` > 如果安装过程中报错: ``` Error: Package: 3:docker-ce-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: container-selinux >= 2:2.74 Error: Package: docker-ce-rootless-extras-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: fuse-overlayfs >= 0.7 Error: Package: docker-ce-rootless-extras-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: slirp4netns >= 0.4 Error: Package: containerd.io-1.4.9-3.1.el7.x86_64 (docker-ce-stable) ``` > 缺少一些依赖，解决方式：在/etc/yum.repos.d/docker-ce.repo开头追加如下内容: ``` [centos-extras] name=Centos extras - $basearch baseurl=http://mirror.centos.org/centos/7/extras/x86_64 enabled=1 gpgcheck=0 ``` > 然后执行安装命令： ``` yum -y install slirp4netns fuse-overlayfs container-selinux ``` > 执行完以上之后，再次执行yum -y install docker-ce-20.10.9-3.el7安装docker即可。 设置docker 开机启动，并启动docker： ``` systemctl enable docker systemctl start docker ``` 查看docker版本 ``` docker version ``` 修改cgroup方式，并重启docker。 ``` vim /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } #重启docker systemctl restart docker ``` ### 1.9.7 **cri-docker安装** 这里需要在全部节点执行cri-docker安装。 1) **下载cri-docker源码** 可以从 [https://github.com/Mirantis/cri-dockerd/archive/refs/tags/v0.2.6.tar.gz地址下载cri-docker源码，然后使用go进行编译安装。](https://github.com/Mirantis/cri-dockerd/archive/refs/tags/v0.2.6.tar.gz地址下载cri-docker源码，然后使用go进行编译安装。) cri-docker源码下载完成后，上传到Master并解压，改名: ``` [root@node1 ~]# tar -zxvf ./cri-dockerd-0.2.6.tar.gz [root@node1 ~]# mv cri-dockerd-0.2.6 cri-dockerd ``` 2) **安装go** ``` [root@node1 ~]# wget https://storage.googleapis.com/golang/getgo/installer_linux [root@node1 ~]# chmod +x ./installer_linux [root@node1 ~]# ./installer_linux [root@node1 ~]# source ~/.bash_profile [root@node1 ~]# go version go version go1.19.3 linux/amd64 ``` 3. **编译安装cri-docker** ``` #进入 cri-dockerd 中，并创建目录bin [root@node1 ~]# cd cri-dockerd && mkdir bin #编译，大概等待1分钟 [root@node1 cri-dockerd]# go build -o bin/cri-dockerd #安装cri-docker，安装-o指定owner -g指定group -m指定指定权限 [root@node1 cri-dockerd]# mkdir -p /usr/local/bin [root@node1 cri-dockerd]# install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd #复制服务管理文件至/etc/systemd/system目录中 [root@node1 cri-dockerd]# cp -a packaging/systemd/* /etc/systemd/system #指定cri-dockerd运行位置 [root@node1 cri-dockerd]# sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service [root@node1 cri-dockerd]# systemctl daemon-reload #启动服务 [root@node1 cri-dockerd]# systemctl enable cri-docker.service [root@node1 cri-dockerd]# systemctl enable --now cri-docker ``` ### 1.9.8 **软件版本** 这里安装Kubernetes版本为1.25.3，在所有主机（node1,node2,node3）安装kubeadm，kubelet，kubectl。 * kubeadm：初始化集群、管理集群等。 * kubelet:用于接收api-server指令，对pod生命周期进行管理。 * kubectl:集群应用命令行管理工具。 ### 1.9.9 **准备阿里yum源** 每台k8s节点vim /etc/yum.repos.d/k8s.repo，写入以下内容： ``` [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ``` ### 1.9.10 **集群软件安装** ``` #查看指定版本 yum list kubeadm.x86_64 --showduplicates | sort -r yum list kubelet.x86_64 --showduplicates | sort -r yum list kubectl.x86_64 --showduplicates | sort -r #安装指定版本 yum -y install --setopt=obsoletes=0 kubeadm-1.25.3-0 kubelet-1.25.3-0 kubectl-1.25.3-0 ``` 安装过程有有如下错误： ``` Error: Package: kubelet-1.25.3-0.x86_64 (kubernetes) Requires: conntrack ``` 解决方式： ``` wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/Centos-7.repo yum install -y conntrack-tools ``` ### 1.9.11 **配置kubelet** 为了实现docker使用的cgroup driver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ``` #vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ``` 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 ``` systemctl enable kubelet ``` ### 1.9.12 **集群镜像准备** 只需要在node1 Master节点上执行如下下载镜像命令即可，这里先使用kubeadm查询下镜像。 ``` [root@node1 ~]#kubeadm config images list --kubernetes-version=v1.25.3 registry.k8s.io/kube-apiserver:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3 registry.k8s.io/pause:3.8 registry.k8s.io/etcd:3.5.4-0 registry.k8s.io/coredns/coredns:v1.9.3 ``` 编写下载镜像脚本image_download.sh： ``` #!/bin/bash images_list=' registry.k8s.io/kube-apiserver:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3 registry.k8s.io/pause:3.8 registry.k8s.io/etcd:3.5.4-0 registry.k8s.io/coredns/coredns:v1.9.3 ' for i in $images_list do docker pull $i done docker save -o k8s-1-25-3.tar $images_list ``` 以上脚本准备完成之后，执行命令：sh image_download.sh 进行镜像下载 注意：下载时候需要科学上网，否则下载不下来。也可以使用资料中的“k8s-1-25-3.tar”下载好的包。 > #如果下载不下来，使用资料中打包好的k8s-1-25-3.tar，将镜像导入到docker中 > > docker load -i k8s-1-25-3.tar ### 1.9.13 **集群初始化** 只需要在Master节点执行如下初始化命令即可。 ``` [root@node1 ~]# kubeadm init --kubernetes-version=v1.25.3 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.179.4 --cri-socket unix:///var/run/cri-dockerd.sock ``` 注意：--apiserver-advertise-address=192.168.179.4 要写当前主机Master IP > 初始化过程中报错： ``` [init] Using Kubernetes version: v1.25.3 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR CRI]: container runtime is not running: output: E1102 20:14:29.494424 10976 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"time="2022-11-02T20:14:29+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alp ha2.RuntimeService", error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher ``` 执行如下命令，重启containerd后，再次init 初始化。 ``` [root@node1 ~]# rm -rf /etc/containerd/config.toml [root@node1 ~]# systemctl restart containerd ``` 初始化完成后，结果如下： ``` Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 ``` ### 1.9.14 **集群应用客户端管理集群文件准备** 参照初始化的内容来执行如下命令： ``` [root@node1 ~]# mkdir -p $HOME/.kube [root@node1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@node1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@node1 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf ``` ### 1.9.15 **集群网络准备** #### 1.9.15.1 **calico安装** K8s使用calico部署集群网络,安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico。 只需要在Master节点安装即可。 ``` #下载operator资源清单文件 wget https://docs.projectcalico.org/manifests/tigera-operator.yaml --no-check-certificate #应用资源清单文件，创建operator kubectl create -f tigera-operator.yaml #通过自定义资源方式安装 wget https://docs.projectcalico.org/manifests/custom-resources.yaml --no-check-certificate #修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 # vim custom-resources.yaml 【修改和增加以下加粗内容】 apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. calicoNetwork: # Note: The ipPools section cannot be modified post-install. ipPools: - blockSize: 26 cidr: 10.244.0.0/16 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all() nodeAddressAutodetectionV4: interface: ens.* #应用清单文件 kubectl create -f custom-resources.yaml #监视calico-sysem命名空间中pod运行情况 watch kubectl get pods -n calico-system [root@node1 ~]# watch kubectl get pods -n calico-system Every 2.0s: kubectl get pods -n calico-system Thu Nov 3 14:14:30 2022 NAME READY STATUS RESTARTS AGE calico-kube-controllers-65648cd788-flmk4 1/1 Running 0 2m21s calico-node-chnd5 1/1 Running 0 2m21s calico-node-kc5bx 1/1 Running 0 2m21s calico-node-s2cp5 1/1 Running 0 2m21s calico-typha-d76595dfb-5z6mg 1/1 Running 0 2m21s calico-typha-d76595dfb-hgg27 1/1 Running 0 2m19s #删除 master 上的 taint [root@node1 ~]# kubectl taint nodes --all node-role.kubernetes.io/master- taint "node-role.kubernetes.io/master" not found taint "node-role.kubernetes.io/master" not found taint "node-role.kubernetes.io/master" not found #已经全部运行 [root@node1 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-65648cd788-ktjrh 1/1 Running 0 110m calico-node-dvprv 1/1 Running 0 110m calico-node-nhzch 1/1 Running 0 110m calico-node-q44gh 1/1 Running 0 110m calico-typha-6bc9d76554-4bv77 1/1 Running 0 110m calico-typha-6bc9d76554-nkzxq 1/1 Running 0 110m #查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 [root@node1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-565d847f94-bjtlh 1/1 Running 0 19h coredns-565d847f94-wlxmf 1/1 Running 0 19h etcd-node1 1/1 Running 0 19h kube-apiserver-node1 1/1 Running 0 19h kube-controller-manager-node1 1/1 Running 0 19h kube-proxy-bgpz2 1/1 Running 0 19h kube-proxy-jlltp 1/1 Running 0 19h kube-proxy-stfrx 1/1 Running 0 19h kube-scheduler-node1 1/1 Running 0 19h ``` #### 1.9.15.2 **calico客户端安装** 主要用来验证k8s集群节点网络是否正常。这里只需要在Master节点安装就可以。 ``` #下载二进制文件，注意，这里需要检查calico 服务端的版本，客户端要与服务端版本保持一致，这里没有命令验证calico的版本，所以安装客户端的时候安装最新版本即可。 curl -L https://github.com/projectcalico/calico/releases/download/v3.24.5/calicoctl-linux-amd64 -o calicoctl #安装calicoctl mv calicoctl /usr/bin/ #为calicoctl添加可执行权限 chmod +x /usr/bin/calicoctl #查看添加权限后文件 ls /usr/bin/calicoctl #查看calicoctl版本 [root@node1 ~]# calicoctl version Client Version: v3.24.5 Git commit: 83493da01 Cluster Version: v3.24.5 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm 通过~/.kube/config连接kubernetes集群，查看已运行节点 [root@node1 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME node1 ``` ### 1.9.16 **集群工作节点添加** 这里在node2,node3 worker节点上执行命令，将worker节点加入到k8s集群。 ``` [root@node2 ~]# kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 --cri-socket unix:///var/run/cri-dockerd.sock [root@node3 ~]# kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 --cri-socket unix:///var/run/cri-dockerd.sock ``` 在master节点上操作，查看网络节点是否添加 ``` [root@node1 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME node1 node2 node3 ``` ### 1.9.17 **验证集群可用性** 使用命令查看所有的节点： ``` [root@node1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 20h v1.25.3 node2 Ready 20h v1.25.3 node3 Ready 20h v1.25.3 ``` 查看集群健康情况： ``` [root@node1 ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-0 Healthy {"health":"true","reason":""} scheduler Healthy ok controller-manager Healthy ok ``` 查看kubernetes集群pod运行情况: ``` [root@node1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-565d847f94-bjtlh 1/1 Running 0 20h coredns-565d847f94-wlxmf 1/1 Running 0 20h etcd-node1 1/1 Running 0 20h kube-apiserver-node1 1/1 Running 0 20h kube-controller-manager-node1 1/1 Running 0 20h kube-proxy-bgpz2 1/1 Running 1 20h kube-proxy-jlltp 1/1 Running 1 20h kube-proxy-stfrx 1/1 Running 0 20h kube-scheduler-node1 1/1 Running 0 20h ``` 查看集群信息: ``` [root@node1 ~]# kubectl cluster-info Kubernetes control plane is running at https://192.168.179.4:6443 CoreDNS is running at https://192.168.179.4:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` ### 1.9.18 **K8s集群其他一些配置** 当在Worker节点上执行kubectl命令管理时会报如下错误： ``` The connection to the server localhost:8080 was refused - did you specify the right host or port? ``` 只要把master上的管理文件/etc/kubernetes/admin.conf拷贝到Worker节点的$HOME/.kube/config就可以让Worker节点也可以实现kubectl命令管理。 ``` #在Worker节点创建.kube目录 [root@node2 ~]# mkdir /root/.kube [root@node3 ~]# mkdir /root/.kube #在master节点做如下操作 [root@node1 ~]# scp /etc/kubernetes/admin.conf node2:/root/.kube/config [root@node1 ~]# scp /etc/kubernetes/admin.conf node3:/root/.kube/config #在worker 节点验证 [root@node2 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 24h v1.25.3 node2 Ready 24h v1.25.3 node3 Ready 24h v1.25.3 [root@node3 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 24h v1.25.3 node2 Ready 24h v1.25.3 node3 Ready 24h v1.25.3 ``` 此外，无论在Master节点还是Worker节点使用kubenetes 命令时，默认不能自动补全，例如：kubectl describe 命令中describe不能自动补全，使用非常不方便，那么这里配置命令自动补全功能。 在所有的kubernetes节点上安装bash-completion并source执行，同时配置下开机自动source，每次开机能自动补全命令。 ``` #安装bash-completion 并 source yum install -y bash-completion source /usr/share/bash-completion/bash_completion kubectl completion bash > ~/.kube/completion.bash.inc source '/root/.kube/completion.bash.inc' #实现用户登录主机自动source ,自动使用命令补全 vim ~/.bash_profile 【加入加粗这一句】 # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs source '/root/.kube/completion.bash.inc' PATH=$PATH:$HOME/bin export PATH ``` 默认K8S我们只要设置了systemctl enable kubelet 后，会在开机自动启动K8S集群，如果想要停止kubernetes集群，我们可以通过systemctl stop kubelet 命令停止集群，但是必须先将节点上的docker停止，命令如下： ### 1.9.19 **K8s集群启停** ``` systemctl stop docker ``` 然后再停止k8s集群： ``` systemctl stop kubelet ``` 启动Kubernetes集群步骤如下： ``` # 先启动docker systemctl start docker # 再启动kubelet systemctl start kubelet ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_gitops.md ================================================ # GitOps # 一、什么是GitOps 说起GitOps，可能马上会联想到我们前面讲的DevOps，那么GitOps和DevOps之间有什么关系、又有什么区别呢？ DevOps是一种文化 DevOps包含了Development和Operations两个部分，是一种重视“软件开发人员（Dev）”和“IT运维技术人员（Ops）”之间沟通合作的文化或惯例。通过自动化“软件交付”和“架构变更”的流程，来使得构建、测试、发布软件能够更加地快捷、频繁和可靠。 如今很多团队当中都在践行着DevOps文化，其中核心的一条理念就是“自动化一切可自动化的”。而GitOps，也正是基于这种理念下而诞生的一种持续交付方式。 # 二、为什么需要GitOps GitOps核心思想是将应用系统的声明性基础架构和应用程序存放在Git版本库中。 将Git作为交付流水线的核心，每个开发人员都可以提交拉取请求（Pull Request）并使用Git来加速和简化应用程序部署和运维任务。通过使用Git，开发人员可以更高效地将注意力集中在创建新功能而不是运维相关任务上（例如，应用系统安装、配置、迁移等）。 同时，GitOps还是一整套CI/CD流水线方案。 在GitOps中可以自由地为流水线的不同部分选择最佳工具。可以从开源生态系统中选择一组工具，也可以从封闭源中选择一组工具，或者根据使用情况，甚至可以将它们组合在一起。 # 三、GitOps怎么实现 ## 3.1实现方式介绍 通过gitlab实现CI流程，将 CD 部分使用 Argo CD 来完成，以实现应用部署及应用部署回滚的可控性。 ![git_argo](../../img/kubernetes/kubernetes_gitops/git_argo.png) ## 3.2 环境准备 ### 3.2.1主机规划 ![image-20220908104352788](../../img/kubernetes/kubernetes_gitops/image-20220908104352788.png) | 序号 | 主机名 | 主机ip | 主机功能 | 软件 | | ---- | ------------ | ------------- | ----------------------- | ------------------------------ | | 1 | dev | 192.168.10.1 | 开发者 项目代码 apidemo | go、golang 、goland | | 2 | gitlab | 192.168.10.11 | 代码仓库，CI操作 | git-lab、git、golang、docker | | 3 | harbor | 192.168.10.21 | 管理和存储容器镜像 | docker、docker-compose、harbor | | 4 | k8s-master01 | 192.168.10.31 | k8s-master | k8s、ArgoCd | | 5 | k8s-worker01 | 192.168.10.32 | k8s-worker | k8s、ArgoCd | | 6 | k8s-worker02 | 192.168.10.33 | k8s-worker | k8s、ArgoCd | ![image-20220908104623080](../../img/kubernetes/kubernetes_gitops/image-20220908104623080.png) ### 3.2.2主机准备 #### 3.2.2.1主机ip地址配置 ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" 配置为静态IP DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.x" 把x替换为对应的IP地址 PREFIX="24" GATEWAY="192.168.10.2" DNS1="223.5.5.5" DNS1="223.6.6.6" ~~~ #### 3.2.2.2主机名准备 ~~~powershell # hostnamectl set-hostname xxx ~~~ > 根据主机规划实施配置 #### 3.2.2.3主机名与ip地址解析配置 ~~~powershell # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.11 gitlab 192.168.10.21 harbor 192.168.10.31 k8s-master01 192.168.10.32 k8s-worker01 192.168.10.33 k8s-worker02 ~~~ #### 3.2.2.4 主机安全设置 ~~~powershell # systemctl stop firewalld;systemctl disable firewalld ~~~ ~~~powershell # firewall-cmd --state ~~~ ~~~powershell # sestatus 关闭selinux ~~~ #### 3.2.2.5主机时间同步 ~~~powershell 编辑 # crontab -e 查看 # crotab -l 0 */1 * * * ntpdate time1.aliyun.com ~~~ ## 3.3主机中工具安装 ### 3.3.1 gitlab主机 #### 3.3.1.1 获取yum源 ~~~powershell [root@gitlab ~]# vim /etc/yum.repos.d/gitlab.repo [gitlab] name=gitlab-ce baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7 enabled=1 gpgcheck=0 ~~~ #### 3.3.1.2 gitlab-ce安装 ~~~powershell [root@gitlab ~]# yum -y install gitlab-ce ~~~ #### 3.3.1.3 gitlab-ce配置 ~~~powershell [root@gitlab ~]# vim /etc/gitlab/gitlab.rb 32 external_url 'http://192.168.10.11' #此ip为在gitlab上创建项目的url前缀 ~~~ #### 3.3.1.4 启动gitlab-ce ~~~powershell [root@gitlab ~]# gitlab-ctl reconfigure ~~~ ~~~powershell [root@gitlab ~]# gitlab-ctl status ~~~ #### 3.3.1.5 访问gitlab-ce 浏览器输入gitlab-ce的服务器地址：http://192.168.10.11 ![登录gitlab-01](../../img/kubernetes/kubernetes_gitops/登录gitlab-01.png) ~~~powershell [root@gitlab ~]# cat /etc/gitlab/initial_root_password #获取gitlab-ce登录初始密码，登录账户名为：root ...... Password: znS4Bqlp0cfYUKg2dHzFiNCAN0GnhtnD4ENjEtEXMVE= ~~~ 通过命令行获取初始化密码后登录gitlab-ce ![登录gitlab-02](../../img/kubernetes/kubernetes_gitops/登录gitlab-02.png) gitlab-ce的UI界面 ![gitlab-ui](../../img/kubernetes/kubernetes_gitops/gitlab-ui.png) #### 3.3.1.6 修改gitlab-ce root用户的密码 方便后续操作，修改root用户的登录密码 ![修改gitlab-ce,root用户密码-01](../../img/kubernetes/kubernetes_gitops/修改gitlab-ce,root用户密码-01.png) ![修改gitlab-ce,root用户密码-02](../../img/kubernetes/kubernetes_gitops/修改gitlab-ce,root用户密码-02.png) ![修改gitlab-ce,root用户密码-03](../../img/kubernetes/kubernetes_gitops/修改gitlab-ce,root用户密码-03.png) ![修改gitlab-ce,root用户密码-04](../../img/kubernetes/kubernetes_gitops/修改gitlab-ce,root用户密码-04.png) 重新登录gitlab-ce ![重新登录gitlab](../../img/kubernetes/kubernetes_gitops/重新登录gitlab.png) #### 3.3.1.7 git安装 当开发者push代码以后，gitlab-runner会在gitlab-ce服务器执行流水线里定义的具体操作，流水线步骤里定义了gitlab-runner通过kustomize客户端工具修改应用部署资源清单文件里的容器镜像版本，并重新push代码到代码仓库，所以需要在gitlab-ce服务器安装git ```shell [root@gitlab ~]# yum -y install git 操作系统自带的源里的git版本可能过低，gitlab-runner无法使用，需要升级下git版本 [root@gitlab ~]# yum install -y http://opensource.wandisco.com/centos/7/git/x86_64/wandisco-git-release-7-2.noarch.rpm [root@gitlab ~]# yum update -y git 查看git版本 [root@gitlab ~]# git --version git version 2.31.1 ``` #### 3.3.1.8 go环境安装 本次应用demo是用go语言开发的，gitlab-runner在编译项目源码时，需要使用go命令，所以需要安装go环境 参考链接：https://golang.google.cn/dl/ ```powershell 安装go解释器，并创建对应目录 [root@gitlab ~]# wget https://dl.google.com/go/go1.19.linux-amd64.tar.gz [root@gitlab ~]# tar xzvf go1.19.linux-amd64.tar.gz -C /usr/local/ [root@gitlab ~]# mkdir /usr/local/go/gopath [root@gitlab ~]# mkdir /usr/local/go/gopath/src [root@gitlab ~]# mkdir /usr/local/go/gopath/bin [root@gitlab ~]# mkdir /usr/local/go/gopath/pkg ``` ~~~powershell [root@gitlab ~]# vim /etc/profile 添加如下几行 export GOROOT="/usr/local/go" export GOPATH="/usr/local/go/gopath" export GOBIN=$GOROOT/bin export PATH=$PATH:$GOBIN 加载环境变量 [root@gitlab ~]# source /etc/profile 配置go环境变量 [root@gitlab ~]# go env -w GOPROXY=https://goproxy.cn [root@gitlab ~]# go env -w GO111MODULE=on 将go可执行文件软链到/usr/bin/下 [root@gitlab ~]# ln -s /usr/local/go/bin/go /usr/bin/go ~~~ ~~~powershell 为GOPATH添加其他用户可写权限，不然gitlab-runner无法执行相关命令 [root@gitlab ~]# chmod -R 757 /usr/local/go/gopath ~~~ #### 3.3.1.9 docker安装 gitlab-runner在执行流水线时需要将源码编译后的可执行文件制作成容器镜像，所以需要安装docker ~~~powershell [root@gitlab ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell [root@gitlab ~]# yum -y install docker-ce ~~~ ~~~powershell [root@gitlab ~]# systemctl enable docker [root@gitlab ~]# systemctl start docker ~~~ ### 3.3.2 harbor主机 #### 3.3.2.1 docker安装 ~~~powershell [root@harbor ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell [root@harbor ~]# yum -y install docker-ce ~~~ ~~~powershell [root@harbor ~]# systemctl enable docker [root@harbor ~]# systemctl start docker ~~~ #### 3.3.2.2 docker-compose安装 参考链接：https://github.com/docker/compose ~~~powershell [root@harbor ~]# wget https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 ~~~ ~~~powershell [root@harbor ~]# ls docker-compose-linux-x86_64 ~~~ ~~~powershell [root@harbor ~]# chmod +x docker-compose-linux-x86_64 ~~~ ~~~powershell [root@harbor ~]# mv docker-compose-linux-x86_64 /usr/bin/docker-compose ~~~ ~~~powershell [root@harbor ~]# docker-compose version Docker Compose version v2.2.3 ~~~ #### 3.3.2.3 部署harbor 参考链接：https://github.com/goharbor/harbor ~~~powershell [root@harbor ~]# wget https://github.com/goharbor/harbor/releases/download/v2.4.1/harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell [root@harbor ~]# ls harbor-offline-installer-v2.4.1.tgz ~~~ ~~~powershell [root@harbor ~]# tar xf harbor-offline-installer-v2.4.1.tgz -C /home ~~~ ~~~powershell [root@harbor ~]# cd /home [root@harbor home]# ls harbor [root@harbor home]# cd harbor/ [root@harbor harbor]# ls common.sh harbor.v2.4.1.tar.gz harbor.yml.tmpl install.sh LICENSE prepare ~~~ ~~~powershell [root@harbor harbor]# mv harbor.yml.tmpl harbor.yml ~~~ ~~~powershell [root@harbor harbor]# vim harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: 192.168.10.21 修改 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config #https: 注释 # https port for harbor, default is 443 # port: 443 注释 # The path of cert and key files for nginx # certificate: /your/certificate/path 注释 # private_key: /your/private/key/path 注释 ~~~ ~~~powershell [root@harbor harbor]# ./prepare ~~~ ~~~powershell [root@harbor harbor]# ./install.sh ...... [+] Running 10/10 ⠿ Network harbor_harbor Created 0.0s ⠿ Container harbor-log Started 0.3s ⠿ Container harbor-portal Started 1.2s ⠿ Container harbor-db Started 1.3s ⠿ Container registry Started 1.3s ⠿ Container registryctl Started 1.3s ⠿ Container redis Started 1.2s ⠿ Container harbor-core Started 2.0s ⠿ Container nginx Started 2.8s ⠿ Container harbor-jobservice Started 2.8s ✔ ----Harbor has been installed and started successfully.---- 输出以上内容，harbor安装成功 ~~~ ~~~powershell [root@harbor harbor]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 18cf67f9557c goharbor/harbor-jobservice:v2.4.1 "/harbor/entrypoint.…" 53 seconds ago Up 50 seconds (healthy) harbor-jobservice 4469424da41a goharbor/nginx-photon:v2.4.1 "nginx -g 'daemon of…" 53 seconds ago Up 50 seconds (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp nginx 813ca518828f goharbor/harbor-core:v2.4.1 "/harbor/entrypoint.…" 53 seconds ago Up 51 seconds (healthy) harbor-core 6119c3a9fe69 goharbor/harbor-db:v2.4.1 "/docker-entrypoint.…" 53 seconds ago Up 52 seconds (healthy) harbor-db 87b262aee8f4 goharbor/harbor-portal:v2.4.1 "nginx -g 'daemon of…" 53 seconds ago Up 52 seconds (healthy) harbor-portal b2dfca1561f4 goharbor/redis-photon:v2.4.1 "redis-server /etc/r…" 53 seconds ago Up 52 seconds (healthy) redis b9462946a5ca goharbor/harbor-registryctl:v2.4.1 "/home/harbor/start.…" 53 seconds ago Up 52 seconds (healthy) registryctl c95cb337f764 goharbor/registry-photon:v2.4.1 "/home/harbor/entryp…" 53 seconds ago Up 51 seconds (healthy) registry 58621f64a41f goharbor/harbor-log:v2.4.1 "/bin/sh -c /usr/loc…" 53 seconds ago Up 53 seconds (healthy) 127.0.0.1:1514->10514/tcp harbor-log ~~~ #### 3.3.2.4 访问harbor 浏览器输入安装harbor配置文件里的hostname，http://192.168.10.21,用户名为admin,密码为配置文件里设置的Harbor12345 ![登录harbor](../../img/kubernetes/kubernetes_gitops/登录harbor.png) harborUI界面 ![harborUI](../../img/kubernetes/kubernetes_gitops/harborUI.png) ### 3.3.3 k8s安装 #### 3.3.3.1 配置主机名与IP地址解析 > 下面解析是管理员添加，sealos在运行过程中，也会自动添加主机名与IP地址解析关系。 ~~~powershell [root@k8s-master01 ~]# cat /etc/hosts 192.168.10.31 k8s-master01 192.168.10.32 k8s-worker01 192.168.10.33 k8s-worker02 [root@k8s-worker01 ~]# cat /etc/hosts 192.168.10.31 k8s-master01 192.168.10.32 k8s-worker01 192.168.10.33 k8s-worker02 [root@k8s-worker02 ~]# cat /etc/hosts 192.168.10.31 k8s-master01 192.168.10.32 k8s-worker01 192.168.10.33 k8s-worker02 ~~~ #### 3.3.3.2 升级操作系统内核 > k8s集群所有机器都需要升级内核 ~~~powershell # rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org # yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm # yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64 # awk -F \' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg # grub2-set-default "CentOS Linux (5.4.204-1.el7.elrepo.x86_64) 7 (Core)" # reboot ~~~ #### 3.3.3.3 kubernetes集群快速部署工具sealos准备 > 参考链接：https://github.com/labring/sealos ~~~powershell [root@k8s-master01 ~]# wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# tar xf sealos_4.0.0_linux_amd64.tar.gz [root@k8s-master01 ~]# ls sealctl sealos ~~~ ~~~powershell [root@k8s-master01 ~]# mv seal* /usr/bin/ ~~~ ~~~powershell [root@k8s-master01 ~]# sealos version {"gitVersion":"4.0.0","gitCommit":"7146cfe","buildDate":"2022-06-30T14:24:31Z","goVersion":"go1.17.11","compiler":"gc","platform":"linux/amd64"} ~~~ #### 3.3.3.4 kubernetes集群快速部署 ~~~powershell [root@k8s-master01 ~]# vim sealos-install-k8s.sh sealos run labring/kubernetes:v1.24.0 labring/calico:v3.22.1 \ --masters 192.168.10.31 \ --nodes 192.168.10.32,192.168.10.33 \ --passwd 123 ~~~ ~~~powershell [root@k8s-master01 ~]# sh sealos-install-k8s.sh ~~~ #### 3.3.3.5 kubernetes集群可用性验证 ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 80s v1.24.0 k8s-worker01 Ready 50s v1.24.0 k8s-worker02 Ready 50s v1.24.0 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE calico-system calico-kube-controllers-6b44b54755-8ps2r 1/1 Running 0 57s calico-system calico-node-q6s64 1/1 Running 0 57s calico-system calico-node-rflzc 1/1 Running 0 57s calico-system calico-node-x2kv6 1/1 Running 0 57s calico-system calico-typha-7df84cd569-sqmnb 1/1 Running 0 57s calico-system calico-typha-7df84cd569-ss49p 1/1 Running 0 48s kube-system coredns-6d4b75cb6d-g67cj 1/1 Running 0 94s kube-system coredns-6d4b75cb6d-knxfk 1/1 Running 0 94s kube-system etcd-k8s-master01 1/1 Running 0 108s kube-system kube-apiserver-k8s-master01 1/1 Running 0 108s kube-system kube-controller-manager-k8s-master01 1/1 Running 0 108s kube-system kube-proxy-5ncc6 1/1 Running 0 95s kube-system kube-proxy-rvntq 1/1 Running 0 81s kube-system kube-proxy-s7gfr 1/1 Running 0 81s kube-system kube-scheduler-k8s-master01 1/1 Running 0 108s kube-system kube-sealyun-lvscare-k8s-worker01 1/1 Running 0 74s kube-system kube-sealyun-lvscare-k8s-worker02 1/1 Running 0 72s tigera-operator tigera-operator-d7957f5cc-4twhq 1/1 Running 1 (60s ago) 79s ~~~ #### 3.3.3.6 安装ingress-nginx 通过配置ingress-nginx对象暴露服务 > 参考链接：https://github.com/kubernetes/ingress-nginx ```shell [root@k8s-master01 ~]# curl -k https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml -o deploy.yaml ``` ```shell [root@k8s-master1 ~]# ls deploy.yaml ``` ```shell [root@k8s-master01 ~]# vim deploy.yaml 334 apiVersion: v1 335 kind: Service 336 metadata: 337 labels: 338 app.kubernetes.io/component: controller 339 app.kubernetes.io/instance: ingress-nginx 340 app.kubernetes.io/name: ingress-nginx 341 app.kubernetes.io/part-of: ingress-nginx 342 app.kubernetes.io/version: 1.3.0 343 name: ingress-nginx-controller 344 namespace: ingress-nginx 345 spec: 346 ipFamilies: 347 - IPv4 348 ipFamilyPolicy: SingleStack 349 ports: 350 - appProtocol: http 351 name: http 352 port: 80 353 protocol: TCP 354 targetPort: http 355 - appProtocol: https 356 name: https 357 port: 443 358 protocol: TCP 359 targetPort: https 360 selector: 361 app.kubernetes.io/component: controller 362 app.kubernetes.io/instance: ingress-nginx 363 app.kubernetes.io/name: ingress-nginx 364 type: LoadBalancer 把364行修改为LoadBalancer 365 --- 366 apiVersion: v1 367 kind: Service ``` ```shell [root@k8s-master01 ~]# kubectl apply -f deploy.yaml ``` ```powershell 验证部署： [root@k8s-master01 ~]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-rksbr 0/1 Completed 0 104s ingress-nginx-admission-patch-zg8f8 0/1 Completed 3 104s ingress-nginx-controller-6dc865cd86-7rvfx 0/1 Running 0 104s [root@k8s-master01 ~]# kubectl get all -n ingress-nginx NAME READY STATUS RESTARTS AGE pod/ingress-nginx-admission-create-rksbr 0/1 Completed 0 111s pod/ingress-nginx-admission-patch-zg8f8 0/1 Completed 3 111s pod/ingress-nginx-controller-6dc865cd86-7rvfx 1/1 Running 0 111s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller LoadBalancer 10.96.2.211 80:32210/TCP,443:32004/TCP 111s service/ingress-nginx-controller-admission ClusterIP 10.96.0.20 443/TCP 111s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 1/1 1 1 111s NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-6dc865cd86 1 1 1 111s NAME COMPLETIONS DURATION AGE job.batch/ingress-nginx-admission-create 1/1 43s 111s job.batch/ingress-nginx-admission-patch 1/1 68s 111s ``` #### 3.3.3.7 安装metallb MetalLB可以为kubernetes集群中的Service提供网络负载均衡功能。 MetalLB两大功能为: - 地址分配，类似于DHCP - 外部通告，一旦MetalLB为服务分配了外部IP地址，它就需要使群集之外的网络意识到该IP在群集中“存在”。MetalLB使用标准路由协议来实现此目的：ARP，NDP或BGP。 参考链接： https://metallb.universe.tf/installation/ ```powershell [root@k8s-master01 ~]# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml [root@k8s-master01 ~]# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml ``` ```powershell [root@k8s-master01 ~]# vim metallb-conf.yaml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 192.168.10.90-192.168.10.100 192.168.10.90-192.168.10.100是集群节点服务器IP同一段。 [root@k8s-master01 ~]# kubectl apply -f metallb-conf.yaml ``` ```powershell 验证配置： [root@k8s-master01 ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.96.2.211 192.168.10.90 80:32210/TCP,443:32004/TCP 7m49s ingress-nginx-controller-admission ClusterIP 10.96.0.20 443/TCP 7m49s ingress-nginx-controller已获取到metallb分配的LB-IP ``` ### 3.3.4 ArgoCd安装 > 参考链接：https://argo-cd.readthedocs.io/en/stable/ #### 3.3.4.1 Arog介绍 Argo([https://argoproj.github.io/projects/argo](https://www.oschina.net/action/GoToLink?url=https%3A%2F%2Fargoproj.github.io%2Fprojects%2Fargo)) 项目是一组 Kubernetes 原生工具集合，用于运行和管理 Kubernetes 上的作业和应用程序。Argo 提供了一种在 Kubernetes 上创建工作和应用程序的三种计算模式 – 服务模式、工作流模式和基于事件的模式 。所有的 Argo 工具都实现为控制器和自定义资源。 **为什么选用Argo CD** 应用程序定义、配置和环境应该是声明性的，并受版本控制。应用程序部署和生命周期管理应该是自动化的、可审计的、易于理解的。 #### 3.3.4.2 Argo安装 ```powershell 创建命名空间 [root@k8s-master01 ~]# kubectl create namespace argocd namespace/argocd created ``` ```powershell 由于后期使用ingress暴露服务，所以不建议直接使用，可下载下来，修改后再执行。 wget https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.11/manifests/install.yaml ``` ```powershell 在10184行下面添加如下内容：默认必须使用TLS证书才能访问，下面案例中，不使用TLS证书。 # vim install.yaml 10184 - command: 10185 - argocd-server 10186 - --insecure 添加此行内容 10187 env: 10188 - name: ARGOCD_SERVER_INSECURE 10189 valueFrom: ``` ```shell [root@k8s-master01 ~]# kubectl apply -f install.yaml -n argocd ``` ```shell [root@k8s-master01 ~]# kubectl get pod -n argocd NAME READY STATUS RESTARTS AGE argocd-application-controller-0 1/1 Running 0 20m argocd-applicationset-controller-7b74965f8c-zvxbc 1/1 Running 0 20m argocd-dex-server-7f75d56bc6-xqzqj 0/1 ImagePullBackOff 0 20m argocd-notifications-controller-54dd686846-gknrm 1/1 Running 0 20m argocd-redis-5dff748d9c-rknln 0/1 ImagePullBackOff 0 20m argocd-repo-server-5576f8d84b-sf24d 1/1 Running 0 20m argocd-server-54b95b8c6-7xxcf 1/1 Running 0 20m ``` **注意：如果argocd-dex-server-7f75d56bc6-xqzqj，argocd-redis-5dff748d9c-rknln两个pod无法拉取镜像的话需要修改Argocd部署方式** ```powershell 以下操作在所有k8s集群服务器上操作 上传“资源”文件夹下的镜像tar包（redis.tar,dex.tar）至所有k8s集群服务器，然后导入镜像 # ctr -n=k8s.io image import redis.tar # ctr -n=k8s.io image import dex.tar # crictl ../../img/kubernetes/kubernetes_gitops list IMAGE TAG IMAGE ID SIZE docker.io/library/redis 7.0.4-alpine 9192ed4e49554 29.7MB ghcr.io/dexidp/dex v2.32.0 cee62569fdc12 80.5MB sealos.hub:5000/calico/cni v3.22.1 2a8ef6985a3e5 80.5MB sealos.hub:5000/calico/node v3.22.1 7a71aca7b60fc 69.6MB sealos.hub:5000/calico/pod2daemon-flexvol v3.22.1 17300d20daf93 8.46MB sealos.hub:5000/coredns/coredns v1.8.6 a4ca41631cc7a 13.6MB sealos.hub:5000/etcd 3.5.3-0 aebe758cef4cd 102MB sealos.hub:5000/kube-apiserver v1.24.0 529072250ccc6 33.8MB sealos.hub:5000/kube-controller-manager v1.24.0 88784fb4ac2f6 31MB sealos.hub:5000/kube-proxy v1.24.0 77b49675beae1 39.5MB sealos.hub:5000/kube-scheduler v1.24.0 e3ed7dee73e93 15.5MB sealos.hub:5000/pause 3.7 221177c6082a8 311kB ``` ```powershell 修改ArgoCd安装资源清单文件中，这两个镜像的拉取策略 9763 containers: 9764 - command: 9765 - /shared/argocd-dex 9766 - rundex 9767 image: ghcr.io/dexidp/dex:v2.32.0 9768 imagePullPolicy: IfNotPresent 修改9768行镜像拉取策略为IfNotPresent 9769 name: dex 9770 ports: 9771 - containerPort: 5556 9772 - containerPort: 5557 9773 - containerPort: 5558 ... 9901 containers: 9902 - args: 9903 - --save 9904 - "" 9905 - --appendonly 9906 - "no" 9907 image: redis:7.0.4-alpine 9908 imagePullPolicy: IfNotPresent 修改9908行镜像拉取策略为IfNotPresent 9909 name: redis 9910 ports: 9911 - containerPort: 6379 ``` ```powershell 然后重新应用ArgoCd部署资源清单文件 [root@k8s-master01 ~]# kubectl apply -f install.yaml -n argocd ``` ```powershell # kubectl get pods -n argocd NAME READY STATUS RESTARTS AGE argocd-application-controller-0 1/1 Running 0 7m27s argocd-applicationset-controller-7b74965f8c-7wsv6 1/1 Running 0 7m28s argocd-dex-server-7f75d56bc6-j6bmk 1/1 Running 0 7m28s argocd-notifications-controller-54dd686846-kj579 1/1 Running 0 7m28s argocd-redis-5dff748d9c-vxlbx 1/1 Running 0 7m28s argocd-repo-server-5576f8d84b-ztv4l 1/1 Running 0 7m28s argocd-server-54b95b8c6-rdc74 1/1 Running 0 7m28s # kubectl get svc -n argocd NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE argocd-applicationset-controller ClusterIP 10.96.2.13 7000/TCP,8080/TCP 7m36s argocd-dex-server ClusterIP 10.96.3.96 5556/TCP,5557/TCP,5558/TCP 7m36s argocd-metrics ClusterIP 10.96.1.126 8082/TCP 7m36s argocd-notifications-controller-metrics ClusterIP 10.96.2.53 9001/TCP 7m36s argocd-redis ClusterIP 10.96.0.83 6379/TCP 7m36s argocd-repo-server ClusterIP 10.96.2.163 8081/TCP,8084/TCP 7m36s argocd-server ClusterIP 10.96.3.54 80/TCP,443/TCP 7m36s argocd-server-metrics ClusterIP 10.96.1.3 8083/TCP 7m36s ``` #### 3.3.4.3 创建Argo Ingress对象 > 使用ingress，方便在k8s集群外访问。 ```powershell [root@k8s-master01 ~]# vim ingress-argocd.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress namespace: argocd annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: argocd.kubemsb.com http: paths: - pathType: Prefix path: "/" backend: service: name: argocd-server port: number: 80 ``` ```shell [root@k8s-master01 ~]# kubectl apply -f ingress-argocd.yaml ingress.networking.k8s.io/ingress created [root@k8s-master01 ~]# kubectl get ingress -n argocd NAME CLASS HOSTS ADDRESS PORTS AGE ingress argocd.alan.com 192.168.10.32 80 14s ``` #### 3.3.4.4 添加argocd.kubemsb.com域名解析 ```powershell 在k8s-master01上添加hosts,通过argocd客户端修改argocd登录密码时，访问argocd [root@k8s-master01 ~]# vim /etc/hosts 192.168.10.90 argocd.kubemsb.com ``` 在pc添加hosts,访问argocdWeb界面 PC按住win+R键，输入drivers,点击确定 ![配置hosts-01](../../img/kubernetes/kubernetes_gitops/配置hosts-01.png) ![配置hosts-02](../../img/kubernetes/kubernetes_gitops/配置hosts-02.png) ![配置hosts-03](../../img/kubernetes/kubernetes_gitops/配置hosts-03.png) ![配置hosts-04](../../img/kubernetes/kubernetes_gitops/配置hosts-04.png) #### 3.3.4.5 访问Argo UI界面 浏览器输入argocd域名：http://argocd.kubemsb.com ![登录argocd-01](../../img/kubernetes/kubernetes_gitops/登录argocd-01.png) #### 3.3.4.6 获取登录密码 > 用户名为：admin，密码需要查询后解密方可知晓 ~~~powershell 查看加密后的密码 [root@k8s-master01 ~]# kubectl get secret argocd-initial-admin-secret -o yaml -n argocd apiVersion: v1 data: password: clpYZ3d4dkY4TGlmMTV4Qw== 此处为加密后密码，需要解密才能使用。 kind: Secret metadata: ~~~ ~~~powershell 把加密后的密码进行解密 [root@k8s-master01 ~]# echo clpYZ3d4dkY4TGlmMTV4Qw== | base64 -d g50ElRK4jlYElORK 此为真正的密码 ~~~ ~~~powershell 或使用下面命令直接获取登录密码： [root@k8s-master01 ~]# kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d g50ElRK4jlYElORK ~~~ 重新登录argo cd:http://argocd.kubemsb.com ![登录argocd-02](../../img/kubernetes/kubernetes_gitops/登录argocd-02.png) ArgoCd web界面 ![argocd-ui](../../img/kubernetes/kubernetes_gitops/argocd-ui.png) #### 3.3.4.7 客户端安装 > 可通过客户端登录Argocd命令行或直接修改admin登录密码等 参考链接：https://github.com/argoproj/argo-cd/releases ```powershell 下载argocd客户端软件 [root@k8s-master01 ~]# wget https://github.com/argoproj/argo-cd/releases/download/v2.4.11/argocd-linux-amd64 ``` ~~~powershell 安装Argocd客户端软件 [root@k8s-master01 ~]# chmod +x argocd-linux-amd64 [root@k8s-master01 ~]# mv argocd-linux-amd64 /usr/local/bin/argocd ~~~ ~~~powershell 使用argocd version查看版本相关信息 [root@k8s-master01 ~]# argocd version argocd: v2.4.11+3d9e9f2 BuildDate: 2022-08-22T09:35:38Z GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b GitTreeState: clean GoVersion: go1.18.5 Compiler: gc Platform: linux/amd64 WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. argocd-server: v2.4.11+3d9e9f2 ~~~ ~~~powershell 在命令行登录 [root@k8s-master01 ~]# argocd login argocd.alan.com WARNING: server certificate had error: x509: certificate is valid for ingress.local, not argocd.alan.com. Proceed insecurely (y/n)? y WARN[0002] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. Username: admin Password: 输入初始密码 'admin:login' logged in successfully Context 'argocd.alan.com' updated ~~~ ~~~powershell 修改admin管理员密码 [root@k8s-master01 ~]# argocd account update-password WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. *** Enter password of currently logged in user (admin): 输入当前密码，可直接复制粘贴 *** Enter new password for user admin: 12345678 必须为8-32位 *** Confirm new password for user admin: 密码确认 Password updated 更新成功，可使用此密码登录WEB UI界面。 Context 'argocd.alan.com' updated ~~~ ~~~powershell 登出argocd [root@k8s-master01 ~]# argocd logout argocd.alan.com Logged out from 'argocd.alan.com' ~~~ 重新登录argocd:http://argocd.kubemsb.com ![重新登录argocd](../../img/kubernetes/kubernetes_gitops/重新登录argocd.png) ## 3.4 工具集成配置 ### 3.4.1 配置主机使用harbor #### 3.4.1.1 harbor ~~~powershell [root@harbor ~]# vim /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.21"], "registry-mirrors": ["https://a237mf1e.mirror.aliyuncs.com"] } ~~~ ~~~powershell [root@harbor ~]# docker-compose down ~~~ ~~~powershell [root@harbor ~]# systemctl restart docker ~~~ ~~~powershell [root@harbor ~]# docker-compose up -d ~~~ #### 3.4.1.2 k8s ```powershell [root@k8s-master01 ~]# cd /etc/containerd/ [root@k8s-master01 containerd]# vim config.toml 添加如下内容 [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.10.21".auth] username = "admin" password = "Harbor12345" [root@k8s-master01 containerd]# cd /etc/containerd/certs.d/ [root@k8s-master01 certs.d]# mkdir 192.168.10.21 [root@k8s-master01 certs.d]# cp sealos.hub\:5000/hosts.toml 192.168.10.21/ [root@k8s-master01 certs.d]# cd 192.168.10.21/ [root@k8s-master01 192.168.10.21]# vim hosts.toml server = "http://192.168.10.21" [host."http://192.168.10.21"] capabilities = ["pull", "resolve", "push"] skip_verify = true [root@k8s-master01 192.168.10.21]# systemctl restart containerd ``` ```powershell 将私有仓库连接配置文件同步到k8s其他节点 [root@k8s-master01 certs.d]# cd /etc/containerd/certs.d/ [root@k8s-master01 certs.d]# scp -r 192.168.10.21/ k8s-worker01:/etc/containerd/certs.d/ [root@k8s-master01 certs.d]# scp -r 192.168.10.21/ k8s-worker02:/etc/containerd/certs.d/ [root@k8s-master01 certs.d]# cd /etc/containerd/ [root@k8s-master01 containerd]# scp -r config.toml k8s-worker01:/etc/containerd/config.toml [root@k8s-master01 containerd]# scp -r config.toml k8s-worker02:/etc/containerd/config.toml k8s其他节点重启containerd [root@k8s-worker01 ~]# systemctl restart containerdcd [root@k8s-worker02 ~]# systemctl restart containerd ``` #### 3.4.1.3 gitlab ~~~powershell [root@gitlab ~]# vim /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.21"], "registry-mirrors": ["https://a237mf1e.mirror.aliyuncs.com"] } ~~~ ~~~powershell [root@harbor ~]# systemctl restart docker ~~~ ### 3.4.2 Kustomize安装 > 安装参考链接：https://kubectl.docs.kubernetes.io/installation/kustomize/ > 使用参考链接：https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/ 当开发者push代码以后，gitlab-runner会在gitlab-ce服务器执行流水线里定义的具体操作，流水线步骤里定义了gitlab-runner通过kustomize客户端工具修改应用部署资源清单文件里的容器镜像版本，并重新push代码到代码仓库，所以需要在gitlab-ce服务器安装kustomize ```powershell [root@gitlab ~]# curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash {Version:kustomize/v4.5.7 GitCommit:56d82a8378dfc8dc3b3b1085e5a6e67b82966bd7 BuildDate:2022-08-02T16:35:54Z GoOs:linux GoArch:amd64} kustomize installed to //root/kustomize [root@gitlab ~]# mv /root/kustomize /usr/bin/ ``` ### 3.4.3 runner安装 #### 3.4.3.1 runner介绍 GitLab Runner是一个开源项目，用于运行您的作业并将结果发送回GitLab。它与GitLab CI一起使用，GitLab CI是GitLab随附的开源持续集成服务，用于协调作业。相当于流水线工人。 #### 3.4.3.2 runner安装 ![安装runner-01](../../img/kubernetes/kubernetes_gitops/安装runner-01.png) ![安装runner-02](../../img/kubernetes/kubernetes_gitops/安装runner-02.png) ![安装runner-03](../../img/kubernetes/kubernetes_gitops/安装runner-03.png) ![安装runner-04](../../img/kubernetes/kubernetes_gitops/安装runner-04.png) ![安装runner-05](../../img/kubernetes/kubernetes_gitops/安装runner-05.png) ![安装runner-06](../../img/kubernetes/kubernetes_gitops/安装runner-06.png) ![安装runner-07](../../img/kubernetes/kubernetes_gitops/安装runner-07.png) ![安装runner-08](../../img/kubernetes/kubernetes_gitops/安装runner-08.png) ![安装runner-09](../../img/kubernetes/kubernetes_gitops/安装runner-09.png) ```powershell [root@gitlab ~]# curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64 [root@gitlab ~]# chmod +x /usr/local/bin/gitlab-runner [root@gitlab ~]# useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash #添加gitlab-runner用户 [root@gitlab ~]# gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner #配置runner工作目录 [root@gitlab ~]# gitlab-runner start [root@gitlab ~]# gitlab-runner register #添加runner实例 Runtime platform arch=amd64 os=linux pid=25519 revision=bbcb5aba version=15.3.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): http://192.168.10.11/ #gitlab地址 Enter the registration token: GR1348941CGPH49ziznkXMcX6sumR #runner配置页面复制的仓库token Enter a description for the runner: [gitlab]: apidemo #runner实例描述，这里设置为apidemo，意为这个runner为apidemo项目的runner Enter tags for the runner (comma-separated): Enter optional maintenance note for the runner: Registering runner... succeeded runner=GR1348941CGPH49zi Enter an executor: virtualbox, docker-ssh+machine, custom, docker, shell, docker+machine, kubernetes, docker-ssh, parallels, ssh: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml" 启动runner [root@gitlab ~]# gitlab-runner run & ``` ```powershell 配置gitlab-runner用户能使用docker [root@gitlab ~]# usermod -G docker gitlab-runner [root@gitlab ~]# grep docker /etc/group docker:x:989:gitlab-runner ``` ```powershell 为runner配置构建目录 [root@gitlab gitlab-runner]# cd /etc/gitlab-runner/ [root@gitlab gitlab-runner]# vim config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "apidemo" url = "http://192.168.10.11/" id = 2 token = "DGzeaCQPGvQHjbH5kUt2" token_obtained_at = 2022-09-04T11:46:00Z token_expires_at = 0001-01-01T00:00:00Z executor = "shell" [runners.custom_build_dir] enabled = true [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] 输出如下内容表示配置生效 Configuration loaded builds=0 ``` #### 3.4.3.3 为项目配置runner ![image-20220907132058971](../../img/kubernetes/kubernetes_gitops/image-20220907132058971.png) ![runner配置01](../../img/kubernetes/kubernetes_gitops/runner配置01.png) ![runner配置02](../../img/kubernetes/kubernetes_gitops/runner配置02.png) ## 3.5 项目代码发布 > 本次项目代码发布使用自己pc作为开发者环境 ### 3.5.1 获取项目源码 项目源代码在“资源”目录下apidemo ### 3.5.2 go环境安装 > 安装git工具,这个是重点 进入“资源”目录下，双击”go1.17.11.windows-386“安装程序，安装go go安装完成后，配置环境变量 ![image-20220907134513508](../../img/kubernetes/kubernetes_gitops/image-20220907134513508.png) ![配置go环境变量-01](../../img/kubernetes/kubernetes_gitops/配置go环境变量-01.png) ![配置go环境变量-02](../../img/kubernetes/kubernetes_gitops/配置go环境变量-02.png) ![配置go环境变量-03](../../img/kubernetes/kubernetes_gitops/配置go环境变量-03.png) ### 3.5.3 安装goIDE(goland) 进入“资源”目录下，双击”goland“安装程序，安装goland ### 3.5.4 IDE中打开项目 #### 3.5.4.1 打开goland，选择项目 ![打开go项目-01](../../img/kubernetes/kubernetes_gitops/打开go项目-01.png) ![打开go项目-02](../../img/kubernetes/kubernetes_gitops/打开go项目-02.png) #### 3.5.4.2 项目结构 ![项目结构](../../img/kubernetes/kubernetes_gitops/项目结构.png) ### 3.5.5 编写Dockerfile文件 ```dockerfile FROM centos:centos7 #基础镜像 ADD ./apidemo /root #将编译后的可执行文件添加至/root目录下 EXPOSE 10088 #镜像实例为容器时，容器暴露的端口 CMD ["/root/apidemo"] #镜像实例为容器时，容器执行的命令 ``` ### 3.5.6 编写应用部署资源清单文件 #### 3.5.6.1 编写项目Deployment资源清单文件 ```yaml apiVersion: apps/v1 #k8s资源版本 kind: Deployment #k8s资源类型 metadata: namespace: apidemo #资源所在命名空间 name: apidemo #资源名称 labels: app: apidemo spec: replicas: 4 #资源副本数 selector: matchLabels: app: apidemo template: metadata: name: apidemo labels: app: apidemo spec: containers: - name: apidemo image: 192.168.10.21/apidemo/apidemo:v3.0 #容器镜像，会通过kustomize进行修改，版本信息会随项目提交而发生变化 imagePullPolicy: Always ports: - containerPort: 10088 #容器端口 ``` #### 3.5.6.2 编写项目Svc资源清单文件 ```yaml apiVersion: v1 #k8s资源版本 kind: Service #k8s资源类型 metadata: namespace: apidemo #资源所在命名空间 name: apidemo #资源名称 labels: app: apidemo spec: type: NodePort #svc类型 ports: - name: apidemoport port: 10088 #svc集群内访问端口 targetPort: 10088 #容器端口 nodePort: 30080 #nodePort暴露端口 selector: app: apidemo ``` #### 3.5.6.3 编写ingress资源清单文件 ```yaml apiVersion: networking.k8s.io/v1 #k8s资源版本 kind: Ingress #k8s资源类型 metadata: name: ingress-apidemo #自定义ingress名称 namespace: apidemo annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: apidemo.alan.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: apidemo # 对应上面创建的service名称 port: number: 10088 # 对应上面创建的service端口 ``` #### 3.5.6.4 编写kustomization.yaml文件，管理资源清单文件 ```yaml resources: #依赖的资源清单文件 - apidemo-deployment.yaml - apidemo-service.yaml - apidemo-ingress.yaml apiVersion: kustomize.config.k8s.io/v1beta1 #k8s资源版本 kind: Kustomization #k8s资源类型 ``` ### 3.5.7 编写gitlabCi流水线文件(.gitlab-ci.yml) 注意：此文件必须放在项目根目录下 ```yaml workflow: #设置工作流 rules: - if: $CI_COMMIT_BRANCH == 'master' #如果代码分支为master则使用以下环境变量 variables: registry: $ci_registry #镜像仓库地址，gitlab配置全局变量 registry_name: $ci_registry_name #镜像仓库登录用户，gitlab配置全局变量 registry_passwd: $ci_registry_passwd #镜像仓库登录密码，gitlab配置全局变量 variables: # app_name: $CI_PROJECT_NAME #项目名，gitlab项目仓库名，gitlab内置变量 app_version: $CI_COMMIT_SHORT_SHA #app版本号，每次push项目的编号，后面会作为镜像版本，gitlab内置变量 namespace: $CI_PROJECT_NAMESPACE #项目所在组，gitlab内置变量 GIT_CLONE_PATH: '$CI_BUILDS_DIR/$CI_JOB_ID/$CI_PROJECT_NAME' #定义gitlab-runner,clone代码的位置 stages: #定义流水线有几个阶段 - build #编译阶段 - deploy #部署阶段 build code: #自定义的步骤名称 stage: build #此步骤为build阶段 script: - go build #具体执行的命令，此处为编译go项目，编译完成会产生apidemo可执行文件 artifacts: #gitlab流水线文件内置关键字，作用为保留制品 paths: #保留制品所在位置 - apidemo #当前步骤需要保留的制品文件，提供个下一步骤使用 docker build: stage: build script: - docker build -t $app_name:$app_version . #此步骤通过项目源码内的Dockerfile文件编译docker镜像 needs: #gitlab流水线文件内置关键字，作用为此步骤所依赖的步骤，只有当被依赖的步骤完成后，此步骤才会执行 - build code #此步骤被依赖的步骤 docker tag: stage: build script: - docker tag $app_name:$app_version $registry/$app_name/$app_name:$app_version #此步骤为上一步骤生成的镜像打上仓库标签 needs: - docker build docker push: stage: build script: - docker login -u $ci_registry_name -p $ci_registry_passwd $ci_registry #登录镜像仓库 - docker push $registry/$app_name/$app_name:$app_version #推送镜像至镜像仓库 - docker logout #登出镜像仓库 needs: - docker tag deploy dev: stage: deploy before_script: #gitlab流水线内置关键字，作用为在该步骤执行流水线操作前所依赖步骤。这里需要runner通过修改kustomization.yaml文件，来修改镜像版本信息 #所以需要在修改镜像版本信息后，重新push代码 - git remote set-url origin http://${CI_USERNAME}:${CI_PASSWORD}@192.168.10.11/apidemo/apidemo.git #设置远程仓库地址，CI_USERNAME为代码仓库登录用户名，需要在gitlab自定义全局变量，CI_PASSWORD为代码仓库登录密码，需要在gitlab自定义全局变量 - git config --global user.name "Administrator" #配置本地仓库用户名信息 - git config --global user.email "admin@example.com" #配置本地仓库邮箱信息 script: - git checkout -B master #切换项目分支 - cd base #进入资源清单文件目录 - kustomize edit set image 192.168.10.21/apidemo/apidemo:v3.0=$registry/$app_name/$app_name:$app_version #runner通过kustomize客户端工具修改镜像版本信息 - cat kustomization.yaml - git commit -am '[skip ci] DEV image update' #git 本地提交，注意“skip ci”为gitlab流水线文件内置关键字，作用为跳过ci流水线操作，未设置可能导致流水线进入死循环 - git push origin master #重新提交修改镜像版本后的代码 needs: - docker push ``` ### 3.5.8 gitlab全局变量配置![gitlab配置环境变量-01](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-01.png) ![gitlab配置环境变量-02](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-02.png) ![gitlab配置环境变量-03](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-03.png) ![gitlab配置环境变量-04](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-04.png) ![gitlab配置环境变量-05](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-05.png) ![gitlab配置环境变量-06](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-06.png) ![gitlab配置环境变量-07](../../img/kubernetes/kubernetes_gitops/gitlab配置环境变量-07.png) ### 3.5.9 开发者环境添加远程地址仓库 ![本地添加远程仓库](../../img/kubernetes/kubernetes_gitops/本地添加远程仓库.png) ![本地添加远程仓库-02](../../img/kubernetes/kubernetes_gitops/本地添加远程仓库-02.png) ### 3.5.10 在ArgoCd中创建项目 ![argo添加项目-01](../../img/kubernetes/kubernetes_gitops/argo添加项目-01.png) ![argo添加项目-02](../../img/kubernetes/kubernetes_gitops/argo添加项目-02.png) ![argo添加代码仓库-01](../../img/kubernetes/kubernetes_gitops/argo添加代码仓库-01.png) ![argo添加代码仓库-02](../../img/kubernetes/kubernetes_gitops/argo添加代码仓库-02.png) ### 3.5.11 ArgoCd中项目配置 ![argo项目配置01](../../img/kubernetes/kubernetes_gitops/argo项目配置01.png) ![argo项目配置02](../../img/kubernetes/kubernetes_gitops/argo项目配置02.png) ![argo项目配置03](../../img/kubernetes/kubernetes_gitops/argo项目配置03.png) ![argo项目配置04](../../img/kubernetes/kubernetes_gitops/argo项目配置04.png) ### 3.5.12 ArgoCd中应用配置 ![image-20220907224523395](../../img/kubernetes/kubernetes_gitops/image-20220907224523395.png) ![image-20220907224746513](../../img/kubernetes/kubernetes_gitops/image-20220907224746513.png) ![image-20220907224925515](../../img/kubernetes/kubernetes_gitops/image-20220907224925515.png) ![image-20220907224948483](../../img/kubernetes/kubernetes_gitops/image-20220907224948483.png) ![image-20220907225007964](../../img/kubernetes/kubernetes_gitops/image-20220907225007964.png) ### 3.5.13 本地提交代码 ![本地提交代码](../../img/kubernetes/kubernetes_gitops/本地提交代码.png) ### 3.5.14 查看pipeline执行情况 ![查看ci](../../img/kubernetes/kubernetes_gitops/查看ci.png) ### 3.5.15 查看ArgoCd执行情况 ![查看cd](../../img/kubernetes/kubernetes_gitops/查看cd.png) ### 3.5.16 查看应用部署情况 ```shell [root@k8s-master01 ~]# kubectl get all -n apidemo NAME READY STATUS RESTARTS AGE pod/apidemo-75f68d8584-6ttqz 1/1 Running 0 55s pod/apidemo-75f68d8584-bznbr 1/1 Running 0 53s pod/apidemo-75f68d8584-fbjtw 1/1 Running 0 50s pod/apidemo-75f68d8584-vvv9m 1/1 Running 0 55s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/apidemo NodePort 10.96.1.218 10088:30080/TCP 12m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/apidemo 4/4 4 4 12m NAME DESIRED CURRENT READY AGE replicaset.apps/apidemo-596865b76 0 0 0 6m22s replicaset.apps/apidemo-75f68d8584 4 4 4 55s replicaset.apps/apidemo-776c87dfb8 0 0 0 4m11s replicaset.apps/apidemo-f5dd44695 0 0 0 12m 访问应用 [root@k8s-master01 ~]# curl http://apidemo.alan.com/info {"version":"1.1.1"} ``` ## 3.6 Argo CD项目回滚 ![image-20220907230140527](../../img/kubernetes/kubernetes_gitops/image-20220907230140527.png) ![image-20220907230348772](../../img/kubernetes/kubernetes_gitops/image-20220907230348772.png) ![image-20220907230413912](../../img/kubernetes/kubernetes_gitops/image-20220907230413912.png) ![image-20220907230610324](../../img/kubernetes/kubernetes_gitops/image-20220907230610324.png) ![image-20220907230632019](../../img/kubernetes/kubernetes_gitops/image-20220907230632019.png) ![image-20220907230654857](../../img/kubernetes/kubernetes_gitops/image-20220907230654857.png) ![image-20220907230703654](../../img/kubernetes/kubernetes_gitops/image-20220907230703654.png) ![image-20220907230715094](../../img/kubernetes/kubernetes_gitops/image-20220907230715094.png) ![image-20220907230729871](../../img/kubernetes/kubernetes_gitops/image-20220907230729871.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_harbor.md ================================================ # Kubernetes集群使用容器镜像仓库Harbor # 一、容器镜像仓库Harbor部署 ## 1.1 在docker主机部署harbor ### 1.1.1 docker-ce安装 #### 1.1.1.1 获取YUM源 > 使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ #### 1.1.1.2 安装并设置启动及开机自启动 ~~~powershell # yum -y install docker-ce ~~~ ~~~powershell # systemctl enable --now docker ~~~ ### 1.1.2 docker compose安装 ~~~powershell 下载docker-compose二进制文件 # wget https://github.com/docker/compose/releases/download/1.25.0/docker-compose-Linux-x86_64 ~~~ ~~~powershell 查看已下载二进制文件 # ls docker-compose-Linux-x86_64 ~~~ ~~~powershell 移动二进制文件到/usr/bin目录，并更名为docker-compose # mv docker-compose-Linux-x86_64 /usr/bin/docker-compose ~~~ ~~~powershell 为二进制文件添加可执行权限 # chmod +x /usr/bin/docker-compose ~~~ ~~~powershell 安装完成后，查看docker-compse版本 # docker-compose version docker-compose version 1.25.0, build 0a186604 docker-py version: 4.1.0 CPython version: 3.7.4 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019 ~~~ ### 1.1.3 获取harbor安装文件 ![image-20220125232445910](../../img/kubernetes/kubernetes_harbor/image-20220125232445910.png) ![image-20220125232519365](../../img/kubernetes/kubernetes_harbor/image-20220125232519365.png) ![image-20220614112423090](../../img/kubernetes/kubernetes_harbor/image-20220614112423090.png) ![image-20220614112444701](../../img/kubernetes/kubernetes_harbor/image-20220614112444701.png) ![image-20220614112508085](../../img/kubernetes/kubernetes_harbor/image-20220614112508085.png) ![image-20220614112527380](../../img/kubernetes/kubernetes_harbor/image-20220614112527380.png) ~~~powershell 下载harbor离线安装包 # wget https://github.com/goharbor/harbor/releases/download/v2.5.1/harbor-offline-installer-v2.5.1.tgz ~~~ ~~~powershell 查看已下载的离线安装包 # ls harbor-offline-installer-v2.5.1.tgz ~~~ ### 1.1.4 修改配置文件 ~~~powershell 解压harbor离线安装包 # tar xf harbor-offline-installer-v2.5.1.tgz ~~~ ~~~powershell 查看解压出来的目录 # ls harbor ~~~ ~~~powershell 查看harbor目录 # ls harbor common.sh harbor.v2.5.1.tar.gz harbor.yml.tmpl install.sh LICENSE prepare ~~~ ~~~powershell 创建配置文件 # cd harbor/ # mv harbor.yml.tmpl harbor.yml ~~~ ~~~powershell 修改配置文件内容 # vim harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: 192.168.10.250 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config # https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /root/harbor/6864844_kubemsb.com.pem # private_key: /root/harbor/6864844_kubemsb.com.key # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: 12345 访问密码 ...... ~~~ ### 1.1.5 执行预备脚本 ~~~powershell # ./prepare ~~~ ~~~powershell 输出 prepare base dir is set to /root/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /data/secret/keys/secretkey Successfully called func: create_root_cert Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir ~~~ ### 1.1.6 执行安装脚本 ~~~powershell # ./install.sh ~~~ ~~~powershell 输出 [Step 0]: checking if docker is installed ... Note: docker version: 20.10.12 [Step 1]: checking docker-compose is installed ... Note: docker-compose version: 1.25.0 [Step 2]: loading Harbor images ... [Step 3]: preparing environment ... [Step 4]: preparing harbor configs ... prepare base dir is set to /root/harbor [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating registry ... done Creating registryctl ... done Creating redis ... done Creating harbor-portal ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ✔ ----Harbor has been installed and started successfully.---- ~~~ ### 1.1.7 验证运行情况 ~~~powershell # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 71c0db683e4a goharbor/nginx-photon:v2.5.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp nginx 4e3b53a86f01 goharbor/harbor-jobservice:v2.5.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice df76e1eabbf7 goharbor/harbor-core:v2.5.1 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core eeb4d224dfc4 goharbor/harbor-portal:v2.5.1 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal 70e162c38b59 goharbor/redis-photon:v2.5.1 "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis 8bcc0e9b06ec goharbor/harbor-registryctl:v2.5.1 "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl d88196398df7 goharbor/registry-photon:v2.5.1 "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry ed5ba2ba9c82 goharbor/harbor-db:v2.5.1 "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db dcb4b57c7542 goharbor/harbor-log:v2.5.1 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log ~~~ ### 1.1.8 访问harbor UI界面 ![image-20220614121150040](../../img/kubernetes/kubernetes_harbor/image-20220614121150040.png) ![image-20220614121218531](../../img/kubernetes/kubernetes_harbor/image-20220614121218531.png) ## 1.2 在kubernetes集群中部署harbor > 由于涉及后面技术内容，例如： pv、pvc、helm等，后期内容做介绍。 # 二、Kubernetes集群使用harbor仓库 ## 2.1 通过secret使用harbor仓库 ### 2.1.1 新建一个harbor私有仓库 ![image-20220614121936871](../../img/kubernetes/kubernetes_harbor/image-20220614121936871.png) ![image-20220614122029572](../../img/kubernetes/kubernetes_harbor/image-20220614122029572.png) ### 2.1.2 kubernetes集群所有节点配置harbor仓库 ```powershell # vim /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.250"] } # systemctl restart docker ``` ### 2.1.3 上传nginx测试镜像到harbor > docker-compose down停止harbor,修改后再启动。 ~~~powershell [root@harbor ~]# cat /etc/docker/daemon.json { "insecure-registries": ["http://192.168.10.250"] } ~~~ ~~~powershell [root@harbor ~]# systemctl restart docker ~~~ ```powershell [root@harbor ~]# docker pull nginx:1.15-alpine [root@harbor ~]# docker login 192.168.10.250 Username: admin Password: 这里密码是12345 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store [root@harbor ~]# docker tag nginx:1.15-alpine 192.168.10.250/test/nginx:v1 [root@harbor ~]# docker push 192.168.10.250/test/nginx:v1 ``` ![image-20220614123555567](../../img/kubernetes/kubernetes_harbor/image-20220614123555567.png) ### 2.1.4 创建docker-registry类型secret ```powershell [root@k8s-master1 ~]# kubectl create secret docker-registry harbor-secret --docker-server=192.168.10.250 --docker-username=admin --docker-password=12345 ``` 说明: - 类型为docker-registry - --docker-server指定harbor仓库的IP - --docker-username指定harbor仓库的登录用户名 - --docker-password指定harbor仓库的登录密码 验证查看 ```powershell [root@k8s-master1 ~]# kubectl get secret |grep harbor-secret harbor-secret kubernetes.io/dockerconfigjson 1 19s ``` ~~~powershell [root@k8s-master1 ~]# kubectl describe secret harbor-secret Name: harbor-secret Namespace: default Labels: Annotations: Type: kubernetes.io/dockerconfigjson Data ==== .dockerconfigjson: 94 bytes ~~~ ### 2.1.5 创建pod并使用secret ```powershell [root@k8s-master1 ~]# vim pod-harbor.yml apiVersion: v1 kind: Pod metadata: name: pod-harbor spec: containers: - name: c1 image: 192.168.10.250/test/nginx:v1 imagePullSecrets: # 定义镜像下载使用的secrets - name: harbor-secret # 与上面的secret一致 ``` ```powershell [root@k8s-master1 ~]# kubectl apply -f pod-harbor.yml pod/pod-harbor created ``` ### 2.1.6 验证pod ```powershell [root@k8s-master1 ~]# kubectl describe pod pod-harbor Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16s default-scheduler Successfully assigned default/pod-harbor to k8s-worker1 Normal Pulling 15s kubelet Pulling image "192.168.10.250/test/nginx:v1" Normal Pulled 14s kubelet Successfully pulled image "192.168.10.250/test/nginx:v1" in 630.869309ms Normal Created 14s kubelet Created container c1 Normal Started 14s kubelet Started container c1 可以看到是从192.168.10.250/test/nginx:v1拉取的镜像 ``` ## 2.2 通过serviceaccout使用harbor仓库 ### 2.2.1 设为serviceaccount默认规则 如果每次编写yaml文件都需要添加imagePullSecrets这2行配置，有点麻烦， 有没有在不需要添加这2行配置就可以实现下载harbor仓库里面的镜像呢？答案是有的，可以把secret配置到serviceAccount中即可。 ### 2.2.2 创建serviceaccount及应用过程 1. 创建serviceaccount ```powershell [root@k8s-master1 ~]# vim serviceaccount-harbor-sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: harbor-sa namespace: default [root@k8s-master1 ~]# kubectl apply -f serviceaccount-harbor-sa.yaml serviceaccount/harbor-sa created [root@k8s-master1 ~]# kubectl get sa |grep harbor-sa harbor-sa 1 14s ``` 2.修改serviceaccount添加使用harbor-secret ~~~powershell [root@k8s-master1 ~]# kubectl describe serviceaccount harbor-sa Name: harbor-sa Namespace: default Labels: Annotations: Image pull secrets: Mountable secrets: harbor-sa-token-thxwq Tokens: harbor-sa-token-thxwq Events: ~~~ ```powershell [root@k8s-master1 ~]# kubectl patch serviceaccount harbor-sa -n default -p '{"imagePullSecrets": [{"name": "harbor-secret"}]}' serviceaccount/harbor-sa patched [root@k8s-master1 ~]# kubectl describe serviceaccount harbor-sa Name: harbor-sa Namespace: default Labels: Annotations: Image pull secrets: harbor-secret Mountable secrets: harbor-sa-token-thxwq Tokens: harbor-sa-token-thxwq Events: [root@k8s-master1 ~]# kubectl get serviceaccount harbor-sa -o yaml apiVersion: v1 imagePullSecrets: - name: harbor-secret # 确认,通过patch方式更新了 kind: ServiceAccount ...... ``` 3.修改yaml使用serviceAccount ```powershell [root@k8s-master1 ~]# vim pod-harbor.yml apiVersion: v1 kind: Pod metadata: name: pod-harbor spec: serviceAccount: harbor-sa # 原来的2句换成使用harbor-sa这个serviceAccount containers: - name: c1 image: 192.168.122.18/test/nginx:v1 ``` 4. 删除先前的重新创建pod验证 ```powershell [root@k8s-master1 ~]# kubectl delete pod pod-harbor pod "pod-harbor" deleted [root@k8s-master1 ~]# kubectl apply -f pod-harbor.yml pod/pod-harbor created [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE pod-harbor 1/1 Running 0 8s ``` ~~~powershell [root@k8s-master1 ~]# kubectl describe pods pod-harbor Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 13s default-scheduler Successfully assigned default/pod-harbor to k8s-worker2 Normal Pulling 13s kubelet Pulling image "192.168.10.250/test/nginx:v1" Normal Pulled 12s kubelet Successfully pulled image "192.168.10.250/test/nginx:v1" in 731.788686ms Normal Created 12s kubelet Created container c1 Normal Started 12s kubelet Started container c1 ~~~ 补充: serviceAccount可以实现不同namespace下载镜像使用访问harbor账号的不同。 ================================================ FILE: docs/cloud/kubernetes/kubernetes_hdfs.md ================================================ ## **1.1 大数据HDFS分布式文件系统搭建** 这里我们使用5台节点来安装分布式文件系统，每台节点给了4G内存，4个core，并且每台节点已经关闭防火墙、配置主机名、设置yum源、各个节点时间同步、各个节点两两免密、安装JDK操作。5台节点信息如下： | **节点IP** | **节点名称** | | ---------------- | ------------------ | | 192.168.179.4 | node1 | | 192.168.179.5 | node2 | | 192.168.179.6 | node3 | | 192.168.179.7 | node4 | | 192.168.179.8 | node5 | 下面一一进行基础技术组件搭建。 备注：修改阿里镜像源： ``` #安装wget，wget是linux最常用的下载命令(有些系统默认安装，可忽略) yum -y install wget #备份当前的yum源 mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup #下载阿里云的yum源配置 wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo #清除原来文件缓存，构建新加入的repo结尾文件的缓存 yum clean all yum makecache ``` ### 1.1.1 **搭建Zookeeper** 这里搭建zookeeper版本为3.4.13，搭建zookeeper对应的角色分布如下： | **节点IP** | **节点名称** | **Zookeeper** | | ---------------- | ------------------ | ------------------- | | 192.168.179.4 | node1 | | | 192.168.179.5 | node2 | | | 192.168.179.6 | node3 | ★ | | 192.168.179.7 | node4 | ★ | | 192.168.179.8 | node5 | ★ | 具体搭建步骤如下: 1) **上传zookeeper并解压,配置环境变量** 在node1,node2,node3,node4,node5各个节点都创建/software目录，方便后期安装技术组件使用。 ``` mkdir /software ``` 将zookeeper安装包上传到node3节点/software目录下并解压： ``` [root@node3 software]# tar -zxvf ./zookeeper-3.4.13.tar.gz ``` 在node3节点配置环境变量： ``` #进入vim /etc/profile，在最后加入： export ZOOKEEPER_HOME=/software/zookeeper-3.4.13 export PATH=$PATH:$ZOOKEEPER_HOME/bin #使配置生效 source /etc/profile ``` 2) **在node3节点配置zookeeper** 进入“/software/zookeeper-3.4.13/conf”修改zoo_sample.cfg为zoo.cfg： ``` [root@node3 conf]# mv zoo_sample.cfg zoo.cfg ``` 配置zoo.cfg中内容如下： ``` tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/data/zookeeper clientPort=2181 server.1=node3:2888:3888 server.2=node4:2888:3888 server.3=node5:2888:3888 ``` 3) **将配置好的zookeeper发送到node4,node5节点** ``` tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/data/zookeeper clientPort=2181 server.1=node3:2888:3888 server.2=node4:2888:3888 server.3=node5:2888:3888 ``` 4) **各个节点上创建数据目录，并配置zookeeper环境变量** 在node3,node4,node5各个节点上创建zoo.cfg中指定的数据目录“/opt/data/zookeeper”。 ``` mkdir -p /opt/data/zookeeper ``` 在node4,node5节点配置zookeeper环境变量 ``` #进入vim /etc/profile，在最后加入： export ZOOKEEPER_HOME=/software/zookeeper-3.4.13 export PATH=$PATH:$ZOOKEEPER_HOME/bin #使配置生效 source /etc/profile ``` 5) **各个节点创建节点ID** 在node3,node4,node5各个节点路径“/opt/data/zookeeper”中添加myid文件分别写入1,2,3: ``` #在node3的/opt/data/zookeeper中创建myid文件写入1 #在node4的/opt/data/zookeeper中创建myid文件写入2 #在node5的/opt/data/zookeeper中创建myid文件写入3 ``` 6) **各个节点启动zookeeper,并检查进程状态** ``` #各个节点启动zookeeper命令 zkServer.sh start #检查各个节点zookeeper进程状态 zkServer.sh status ``` ### 1.1.2 **搭建HDFS** 这里搭建HDFS版本为3.3.4，搭建HDFS对应的角色在各个节点分布如下： | **节点IP** | **节点名称** | **NN** | **DN** | **ZKFC** | **JN** | **RM** | **NM** | | ---------------- | ------------------ | ------------ | ------------ | -------------- | ------------ | ------------ | ------------ | | 192.168.179.4 | node1 | ★ | | ★ | | ★ | | | 192.168.179.5 | node2 | ★ | | ★ | | ★ | | | 192.168.179.6 | node3 | | ★ | | ★ | | ★ | | 192.168.179.7 | node4 | | ★ | | ★ | | ★ | | 192.168.179.8 | node5 | | ★ | | ★ | | ★ | 搭建具体步骤如下： 1) **各个节点安装HDFS HA自动切换必须的依赖** ``` yum -y install psmisc ``` 2) **上传下载好的Hadoop安装包到node1节点上，并解压** ``` [root@node1 software]# tar -zxvf ./hadoop-3.3.4.tar.gz ``` 3) **在node1节点上配置Hadoop的环境变量** ``` [root@node1 software]# vim /etc/profile export HADOOP_HOME=/software/hadoop-3.3.4/ export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin: #使配置生效 source /etc/profile ``` 4) **配置$HADOOP_HOME/etc/hadoop下的hadoop-env.sh文件** ``` #导入JAVA_HOME export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64/ ``` 5) **配置$HADOOP_HOME/etc/hadoop下的hdfs-site.xml文件** ``` dfs.nameservices mycluster dfs.permissions.enabled false dfs.ha.namenodes.mycluster nn1,nn2 dfs.namenode.rpc-address.mycluster.nn1 node1:8020 dfs.namenode.rpc-address.mycluster.nn2 node2:8020 dfs.namenode.http-address.mycluster.nn1 node1:50070 dfs.namenode.http-address.mycluster.nn2 node2:50070 dfs.namenode.shared.edits.dir qjournal://node3:8485;node4:8485;node5:8485/mycluster dfs.client.failover.proxy.provider.mycluster org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence dfs.ha.fencing.ssh.private-key-files /root/.ssh/id_rsa dfs.journalnode.edits.dir /opt/data/journal/node/local/data dfs.ha.automatic-failover.enabled true ``` 6) **配置$HADOOP_HOME/ect/hadoop/core-site.xml** ``` fs.defaultFS hdfs://mycluster hadoop.tmp.dir /opt/data/hadoop/ ha.zookeeper.quorum node3:2181,node4:2181,node5:2181 ``` 7) **配置$HADOOP_HOME/etc/hadoop/yarn-site.xml** ``` yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME yarn.resourcemanager.ha.enabled true yarn.resourcemanager.cluster-id mycluster yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 node1 yarn.resourcemanager.hostname.rm2 node2 yarn.resourcemanager.webapp.address.rm1 node1:8088 yarn.resourcemanager.webapp.address.rm2 node2:8088 yarn.resourcemanager.zk-address node3:2181,node4:2181,node5:2181 yarn.nodemanager.vmem-check-enabled false ``` 8) **配置$HADOOP_HOME/etc/hadoop/mapred-site.xml** ``` mapreduce.framework.name yarn ``` 9) **配置$HADOOP_HOME/etc/hadoop/workers文件** ``` [root@node1 ~]# vim /software/hadoop-3.3.4/etc/hadoop/workers node3 node4 node5 ``` 10) **配置$HADOOP_HOME/sbin/start-dfs.sh 和stop-dfs.sh两个文件中顶部添加以下参数，防止启动错误** ``` HDFS_DATANODE_USER=root HDFS_DATANODE_SECURE_USER=hdfs HDFS_NAMENODE_USER=root HDFS_JOURNALNODE_USER=root HDFS_ZKFC_USER=root ``` 11) **配置$HADOOP_HOME/sbin/start-yarn.sh和stop-yarn.sh两个文件顶部添加以下参数，防止启动错误** ``` YARN_RESOURCEMANAGER_USER=root YARN_NODEMANAGER_USER=root ``` 12) **将配置好的Hadoop安装包发送到其他4个节点** ``` [root@node1 ~]# scp -r /software/hadoop-3.3.4 node2:/software/ [root@node1 ~]# scp -r /software/hadoop-3.3.4 node3:/software/ [root@node1 ~]# scp -r /software/hadoop-3.3.4 node4:/software/ [root@node1 ~]# scp -r /software/hadoop-3.3.4 node5:/software/ ``` 13) **在node2、node3、node4、node5节点上配置HADOOP_HOME** ``` #分别在node2、node3、node4、node5节点上配置HADOOP_HOME vim /etc/profile export HADOOP_HOME=/software/hadoop-3.3.4/ export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin: #最后记得Source source /etc/profile ``` 14) **启动HDFS和Yarn** ``` #在node3,node4,node5节点上启动zookeeper zkServer.sh start #在node1上格式化zookeeper [root@node1 ~]# hdfs zkfc -formatZK #在每台journalnode中启动所有的journalnode,这里就是node3,node4,node5节点上启动 hdfs --daemon start journalnode #在node1中格式化namenode [root@node1 ~]# hdfs namenode -format #在node1中启动namenode,以便同步其他namenode [root@node1 ~]# hdfs --daemon start namenode #高可用模式配置namenode,使用下列命令来同步namenode(在需要同步的namenode中执行，这里就是在node2上执行): [root@node2 software]# hdfs namenode -bootstrapStandby #node1上启动HDFS,启动Yarn [root@node1 sbin]# start-dfs.sh [root@node1 sbin]# start-yarn.sh 注意以上也可以使用start-all.sh命令启动Hadoop集群。 ``` 15) **访问WebUI** ``` #访问HDFS : http://node1:50070 #访问Yarn WebUI ：http://node1:8088 ``` ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1666918405095/222e6bbeeb6942d3bd453576c2ace737.png) ![image.png](https://fynotefile.oss-cn-zhangjiakou.aliyuncs.com/fynote/fyfile/20/1666918405095/ccb112a251d042408f66f52db1155be7.png) 18) **停止集群** ``` #停止集群 [root@node1 ~]# stop-dfs.sh [root@node1 ~]# stop-yarn.sh 注意：以上也可以使用 stop-all.sh 停止集群。 ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_helm.md ================================================ # Kubernetes集群包管理解决方案及应用商店 Helm&Kubeapps # 一、引入helm原因 当今的软件开发，随着云原生技术的普及，我们的工程应用进行微服务化和容器化的现象也变得越来越普遍。而Kubernetes几乎已经成了云原生服务编排绕不开的标准和技术。 假设我们需要在K8s中简单部署一个nginx，必要步骤如下： 1、创建或者编写deployment模板 ~~~powershell # kubectl create deployment nginx --image=nginx --dry-run=client -o yaml > deployment.yaml ~~~ 2、启动nginx的pod ~~~powershell # kubectl apply -f deployment.yaml ~~~ 3、检查pod服务 ~~~powershell # kubectl get pod ~~~ 4、创建service ~~~powershell # kubectl expose deployment nginx --port=8099 --target-port=80 --type=NodePort --dry-run=client -o yaml > service.yaml ~~~ 5、启动service服务 ~~~powershell # kubectl apply -f service.yaml ~~~ 6、检查service端口 ~~~powershell # kubectl get svc ~~~ 7、访问nginx服务 ![image-20220728133433498](../../img/kubernetes/kubernetes_helm/image-20220728133433498.png) 实际生产中，微服务项目可能有十几个模块，若还需要进行安全访问和控制，那么需要创建诸如Role、ServiceAccount等资源。部署和版本升级时也往往需要修改或添加配置文件中的一些参数（例如：服务占用的CPU、内存、副本数、端口等），维护大量的yaml文件极为不便，所以，我们需要将这些YAML文件作为一个**整体**管理，并高效复用。 - 在Linux操作系统软件部署中，我们可以使用批量管理工具完成软件的批量管理等，例如yum、dnf等； - 在容器应用中Docker使用Dockerfile文件解决了容器镜像制作难题； - 在kubernetes应用中，通过YAML格式文件解决容器编排部署难题，例如可以通过YAML格式的资源清单文件，非常方便部署不同控制器类型的应用;但是如何维护大量的，系统性的YAML文件，需要我们拥有更好的工具，不能简单使用YAML资源清单托管服务器就可以解决的， 那么在CNCF的体系中是否存在这样的强力“工具”，能够简化我们部署安装过程呢？答案是存在的，Helm就是这样一款工具。 # 二、helm是什么 - 官方: https://helm.sh/ - 作为CNCF的毕业项目。它的官方的定义是：Helm是一个为K8s进行包管理的工具。Helm将yaml作为一个整体管理并实现了这些yaml的高效复用，就像Linux中的yum或apt-get，它使我们能够在K8s中方便快捷的安装、管理、卸载K8s应用。 ![image-20220728134805687](../../img/kubernetes/kubernetes_helm/image-20220728134805687.png) * Helm(舵柄; 舵轮)是一个Kubernetes的包管理工具，就像Linux下的包管理器，如yum/apt等。 * helm一个命令行客户端工具，主要用于Kubernetes应用chart的创建、打包、发布和管理。 * 通过helm可以很方便的将之前打包好的yaml文件部署到kubernetes上。 * 对于应用发布者而言，可以通过Helm打包应用，管理应用依赖关系，管理应用版本并发布应用到软件仓库。 * 对于使用者而言，使用Helm后不用需要了解Kubernetes的Yaml语法并编写应用部署文件，可以通过Helm下载并在kubernetes上安装需要的应用。 * 除此以外，Helm还提供了kubernetes上的软件部署，删除，升级，回滚应用的强大功能。 * Helm 社区已经维护了一个官方 Helm Hub，我们可以直接使用已经做好的 Helm Chart，部署和管理比较复杂的应用程序 * 早期的hub.helm.dev转移到了https://artifacthub.io/。 # 三、helm作用及核心概念 Helm基于go模板语言，用户只要提供规定的目录结构和模板文件。在真正部署时Helm模板引擎便可以将其渲染成真正的K8s资源配置文件，并按照正确的顺序将它们部署到节点上。 Helm 定义了一套 Chart 格式来描述一个应用。打个比方，一个安卓程序打包成 APK 格式，就可以安装到任意一台运行安卓系统的手机上，如果我们把 kubernetes 集群比做安卓系统，kubernetes 集群内应用比做安卓程序，那么 Chart 就可以比做 APK。这就意味着，kubernetes 集群应用只要打包成 Chart，就可以通过 Helm 部署到任意一个 kubernetes 集群中。 Helm中有三个重要概念，分别为Chart、Repository和Release。 - Chart代表中Helm包。它包含在K8s集群内部运行应用程序，工具或服务所需的所有资源定义，为所有项目资源清单yaml文件的集合，采用TAR格式，可以类比成yum中的RPM。 - Repository就是用来存放和共享Chart的地方，可以类比成YUM仓库。 - Release是运行在K8s集群中的Chart的实例，一个Chart可以在同一个集群中安装多次。Chart就像流水线中初始化好的模板，Release就是这个“模板”所生产出来的各个产品。 Helm作为K8s的包管理软件，每次安装Charts 到K8s集群时，都会创建一个新的 release。你可以在Helm 的Repository中寻找需要的Chart。Helm对于部署过程的优化的点在于简化了原先完成配置文件编写后还需使用一串kubectl命令进行的操作、统一管理了部署时的可配置项以及方便了部署完成后的升级和维护。 # 三、helm架构 ![image-20220728135524226](../../img/kubernetes/kubernetes_helm/image-20220728135524226.png) ![image-20220728140109073](../../img/kubernetes/kubernetes_helm/image-20220728140109073.png) Helm客户端使用REST+JSON的方式与K8s中的apiserver进行交互，进而管理deployment、service等资源，并且客户端本身并不需要数据库，它会把相关的信息储存在K8s集群内的Secrets中。 # 四、helm部署 ![image-20220728140501620](../../img/kubernetes/kubernetes_helm/image-20220728140501620.png) ![image-20220728140538044](../../img/kubernetes/kubernetes_helm/image-20220728140538044.png) ![image-20220728140644543](../../img/kubernetes/kubernetes_helm/image-20220728140644543.png) ![image-20220728141000714](../../img/kubernetes/kubernetes_helm/image-20220728141000714.png) ![image-20220728141101419](../../img/kubernetes/kubernetes_helm/image-20220728141101419.png) ~~~powershell [root@k8s-master01 ~]# wget https://get.helm.sh/helm-v3.9.2-linux-amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# ls helm-v3.9.2-linux-amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# tar xf helm-v3.9.2-linux-amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# ls linux-amd64 ~~~ ~~~powershell [root@k8s-master01 ~]# cd linux-amd64/ [root@k8s-master01 linux-amd64]# ls helm LICENSE README.md ~~~ ~~~powershell [root@k8s-master01 linux-amd64]# mv helm /usr/bin ~~~ ~~~powershell [root@k8s-master01 linux-amd64]# helm version version.BuildInfo{Version:"v3.9.2", GitCommit:"1addefbfe665c350f4daf868a9adc5600cc064fd", GitTreeState:"clean", GoVersion:"go1.17.12"} ~~~ # 五、helm基础使用 ## 5.1 添加及删除仓库 ### 5.1.1 查看仓库 ~~~powershell [root@master1 ~]# helm repo list Error: no repositories to show ~~~ ### 5.1.2 添加新的仓库地址 ``` powershell 微软源 [root@k8s-master01 ~]# helm repo add stable http://mirror.azure.cn/kubernetes/charts/ bitnami源 [root@k8s-master01 ~]# helm repo add bitnami https://charts.bitnami.com/bitnami prometheus源 [root@k8s-master01 ~]# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts ``` ### 5.1.3 查看已经添加的仓库 ~~~powershell [root@k8s-master01 ~]# helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts/ ~~~ ### 5.1.4 更新仓库 ```powershell [root@k8s-master01 ~]# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "stable" chart repository Update Complete. ⎈Happy Helming!⎈ ``` **再查看** ~~~powershell [root@master ~]# helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts/ ~~~ ### 5.1.5删除仓库 ~~~powershell [root@k8s-master01 ~]# helm repo remove stable "stable" has been removed from your repositories ~~~ ~~~powershell [root@k8s-master01 ~]# helm repo list Error: no repositories to show ~~~ ## 5.2 查看charts 使用`helm search repo 关键字`可以查看相关charts ~~~powershell [root@k8s-master01 ~]# helm search repo stable NAME CHART VERSION APP VERSION DESCRIPTION stable/acs-engine-autoscaler 2.2.2 2.1.1 DEPRECATED Scales worker nodes within agent pools stable/aerospike 0.3.5 v4.5.0.5 DEPRECATED A Helm chart for Aerospike in Kubern... stable/airflow 7.13.3 1.10.12 DEPRECATED - please use: https://github.com/air... stable/ambassador 5.3.2 0.86.1 DEPRECATED A Helm chart for Datawire Ambassador stable/anchore-engine 1.7.0 0.7.3 Anchore container analysis and policy evaluatio... stable/apm-server 2.1.7 7.0.0 DEPRECATED The server receives data from the El... stable/ark 4.2.2 0.10.2 DEPRECATED A Helm chart for ark stable/artifactory 7.3.2 6.1.0 DEPRECATED Universal Repository Manager support... stable/artifactory-ha 0.4.2 6.2.0 DEPRECATED Universal Repository Manager support... stable/atlantis 3.12.4 v0.14.0 DEPRECATED A Helm chart for Atlantis https://ww... stable/auditbeat 1.1.2 6.7.0 DEPRECATED A lightweight shipper to audit the a... stable/aws-cluster-autoscaler 0.3.4 DEPRECATED Scales worker nodes within autoscali... stable/aws-iam-authenticator 0.1.5 1.0 DEPRECATED A Helm chart for aws-iam-authenticator stable/bitcoind 1.0.2 0.17.1 DEPRECATED Bitcoin is an innovative payment net... stable/bookstack 1.2.4 0.27.5 DEPRECATED BookStack is a simple, self-hosted, ... ...... ~~~ ~~~powershell [root@k8s-master01 ~]# helm search repo nginx NAME CHART VERSION APP VERSION DESCRIPTION stable/nginx-ingress 1.41.3 v0.34.1 DEPRECATED! An nginx Ingress controller that us... stable/nginx-ldapauth-proxy 0.1.6 1.13.5 DEPRECATED - nginx proxy with ldapauth stable/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego stable/gcloud-endpoints 0.1.2 1 DEPRECATED Develop, deploy, protect and monitor... ~~~ ~~~powershell [root@k8s-master01 ~]# helm search repo tomcat NAME CHART VERSION APP VERSION DESCRIPTION stable/tomcat 0.4.3 7.0 DEPRECATED - Deploy a basic tomcat application ... ~~~ ## 5.3 部署应用 MySQL > 环境说明：k8s集群中存在storageclass:nfs-client 我们现在安装一个 `mysql` 应用： ~~~powershell [root@k8s-master01 ~]# helm search repo mysql NAME CHART VERSION APP VERSION DESCRIPTION stable/mysql 1.6.9 5.7.30 DEPRECATED - Fast, reliable, scalable, and easy... ~~~ ~~~powershell [root@k8s-master01 ~]# helm install stable/mysql --generate-name --set persistence.storageClass=nfs-client --set mysqlRootPassword=test123 ~~~ ~~~powershell 部署过程输出的信息： NAME: mysql-1658996042 LAST DEPLOYED: Thu Jul 28 16:14:03 2022 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: mysql-1658996042.default.svc.cluster.local To get your root password run: MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql-1658996042 -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) To connect to your database: 1. Run an Ubuntu pod that you can use as a client: kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il 2. Install the mysql client: $ apt-get update && apt-get install mysql-client -y 3. Connect using the mysql cli, then provide your password: $ mysql -h mysql-1658996042 -p To connect to your database directly from outside the K8s cluster: MYSQL_HOST=127.0.0.1 MYSQL_PORT=3306 # Execute the following command to route the connection: kubectl port-forward svc/mysql-1658996042 3306 mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD} ~~~ ~~~powershell [root@k8s-master01 ~]# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1658996042 default 1 2022-07-28 16:14:03.530489788 +0800 CST deployed mysql-1.6.9 5.7.30 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-1658996042-755f5f64f6-j5s67 1/1 Running 0 82s ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql-1658996042 Bound pvc-7fcb894e-5b8c-4f3e-945d-21b60b9309e5 8Gi RWO nfs-client 93s ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-7fcb894e-5b8c-4f3e-945d-21b60b9309e5 8Gi RWO Delete Bound default/mysql-1658996042 nfs-client 97s ~~~ **一个 chart 包是可以多次安装到同一个集群中的，每次安装都会产生一个release, 每个release都可以独立管理和升级。** ~~~powershell [root@k8s-master01 ~]# helm install stable/mysql --generate-name --set persistence.storageClass=nfs-client --set mysqlRootPassword=root ~~~ ~~~powershell [root@k8s-master01 ~]# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1658996042 default 1 2022-07-28 16:14:03.530489788 +0800 CST deployed mysql-1.6.9 5.7.30 mysql-1658996297 default 1 2022-07-28 16:18:19.282074215 +0800 CST deployed mysql-1.6.9 5.7.30 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-1658996042-755f5f64f6-j5s67 1/1 Running 0 45m mysql-1658996297-75f6f86d84-5qd8r 1/1 Running 0 41m nfs-client-provisioner-9d46587b5-7n2vf 1/1 Running 0 123m ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl exec -it mysql-1658996042-755f5f64f6-j5s67 -- bash ~~~ ~~~powershell root@mysql-1658996042-755f5f64f6-j5s67:/# mysql -uroot -ptest123 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 547 Server version: 5.7.30 MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.01 sec) ~~~ ## 5.4 查看chart资源 ```powershell [root@k8s-master01 ~]# kubectl get all -l release=mysql-1658996042 NAME READY STATUS RESTARTS AGE pod/mysql-1658996042-755f5f64f6-j5s67 1/1 Running 0 72m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql-1658996042 ClusterIP 10.96.2.136 3306/TCP 72m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql-1658996042 1/1 1 1 72m NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-1658996042-755f5f64f6 1 1 1 72m ``` 我们也可以 `helm show chart` 命令来了解 MySQL 这个 chart 包的一些特性： ```powershell [root@k8s-master01 ~]# helm show chart stable/mysql apiVersion: v1 appVersion: 5.7.30 deprecated: true description: DEPRECATED - Fast, reliable, scalable, and easy to use open-source relational database system. home: https://www.mysql.com/ icon: https://www.mysql.com/common/logos/logo-mysql-170x115.png keywords: - mysql - database - sql name: mysql sources: - https://github.com/kubernetes/charts - https://github.com/docker-library/mysql version: 1.6.9 ``` 如果想要了解更多信息，可以用 `helm show all` 命令： ```powershell [root@k8s-master01 ~]# helm show all stable/mysql ...... ``` ## 5.5 删除Release 如果需要删除这个 release，也很简单，只需要使用 `helm uninstall`或`helm delete` 命令即可： ```powershell [root@k8s-master01 ~]# helm uninstall mysql-1605195227 release "mysql-1605195227" uninstalled ``` `uninstall` 命令会从 Kubernetes 中删除 release，也会删除与 release 相关的所有 Kubernetes 资源以及 release 历史记录。 ~~~powershell [root@k8s-master01 ~]# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1605192239 default 1 ......... deployed mysql-1.6.9 5.7.30 ~~~ 在删除的时候使用 `--keep-history` 参数，则会保留 release 的历史记录，该 release 的状态就是 `UNINSTALLED`， ```powershell [root@k8s-master01 ~]# helm uninstall mysql-1605192239 --keep-history release "mysql-1605192239" uninstalled [root@k8s-master01 ~]# helm ls -a NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1605192239 default 1 ........ uninstalled mysql-1.6.9 5.7.30 状态为uninstalled ``` 审查历史时甚至可以取消删除`release`。 `Usage: helm rollback [REVISION] [flags]` ~~~powershell [root@k8s-master01 ~]# helm rollback mysql-1605192239 1 Rollback was a success! Happy Helming! [root@k8s-master01 ~]# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql-1605192239 default 2 ......... deployed mysql-1.6.9 5.7.30 rollback后，又回到deployed状态 ~~~ ## 5.6 定制参数部署应用 上面我们都是直接使用的 `helm install` 命令安装的 chart 包，这种情况下只会使用 chart 的默认配置选项，但是更多的时候，是各种各样的需求，所以我们希望根据自己的需求来定制 chart 包的配置参数。 我们可以使用 `helm show values` 命令来查看一个 chart 包的所有可配置的参数选项： ```powershell [root@k8s-master01 ~]# helm show values stable/mysql ...... ...... ``` 上面我们看到的所有参数都是可以用自己的数据来覆盖的，可以在安装的时候通过 YAML 格式的文件来传递这些参数 1，准备参数文件 ~~~powershell [root@k8s-master01 ~]# vim mysql-config.yml mysqlDatabase: helm persistence: enabled: true # 没有存储卷情况下，改为false storageClass: nfs-client ~~~ 2, 使用`-f mysql-config.yml`安装应用并覆盖参数 ```powershell [root@k8s-master01 ~]# helm install mysql -f mysql-config.yml stable/mysql ``` ~~~powershell 输出内容： NAME: mysql LAST DEPLOYED: Fri Jul 29 14:07:17 2022 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: mysql.default.svc.cluster.local To get your root password run: MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) To connect to your database: 1. Run an Ubuntu pod that you can use as a client: kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il 2. Install the mysql client: $ apt-get update && apt-get install mysql-client -y 3. Connect using the mysql cli, then provide your password: $ mysql -h mysql -p To connect to your database directly from outside the K8s cluster: MYSQL_HOST=127.0.0.1 MYSQL_PORT=3306 # Execute the following command to route the connection: kubectl port-forward svc/mysql 3306 mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD} ~~~ 3, 查看覆盖的参数 ~~~powershell [root@master ~]# helm get values mysql USER-SUPPLIED VALUES: mysqlDatabase: helm persistence: enabled: true storageClass: nfs-client ~~~ 4, 查看部署的相关资源 ~~~powershell [root@k8s-master01 helmdir]# kubectl get all -l release=mysql NAME READY STATUS RESTARTS AGE pod/mysql-855976764d-npvgm 1/1 Running 0 40m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP 10.96.0.84 3306/TCP 40m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 1/1 1 1 40m NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-855976764d 1 1 1 40m ~~~ 5, 查看pod的IP ~~~powershell [root@k8s-master01 helmdir]# kubectl get pods -o wide -l release=mysql NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-855976764d-npvgm 1/1 Running 0 41m 100.119.84.71 k8s-worker01 得到pod的IP为100.119.84.71 ~~~ 6, 安装mysql客户端并连接测试 ~~~powershell [root@k8s-master01 ~]# yum install mariadb -y ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get secret --namespace default mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo wL2SD0RCsT ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 27h mysql ClusterIP 10.96.0.84 3306/TCP 5m21s ~~~ ~~~powershell [root@k8s-master01 helmdir]# mysql -h 10.96.0.84 -uroot -pwL2SD0RCsT -e "show databases;" +--------------------+ | Database | +--------------------+ | information_schema | | helm | | mysql | | performance_schema | | sys | +--------------------+ ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -o wide -l release=mysql NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-855976764d-npvgm 1/1 Running 0 41m 100.119.84.71 k8s-worker01 ~~~ ~~~powershell [root@k8s-master01 ~]# mysql -h 100.119.84.71 -uroot -pwL2SD0RCsT -e "show databases;" +--------------------+ | Database | +--------------------+ | information_schema | | helm | | mysql | | performance_schema | | sys | +--------------------+ ~~~ ## 5.7 升级和回滚 当新版本的 chart 包发布的时候，或者当你要更改 release 的配置的时候，你可以使用 `helm upgrade` 命令来操作。升级需要一个现有的 release，并根据提供的信息对其进行升级。因为 Kubernetes charts 可能很大而且很复杂，Helm 会尝试以最小的侵入性进行升级，它只会更新自上一版本以来发生的变化： 1, 升级前查看版本 ~~~powershell [root@k8s-master01 helmdir]# mysql -h 10.96.0.84 -uroot -pwL2SD0RCsT -e "select version()" +-----------+ | version() | +-----------+ | 5.7.30 | 版本为5.7.30 +-----------+ [root@k8s-master01 helmdir]# kubectl get deployment mysql -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR mysql 1/1 1 1 54m mysql mysql:5.7.30 app=mysql,release=mysql images版本为5.7.30 ~~~ 2，修改配置并升级 ```powershell [root@k8s-master01 helmdir]# vim mysql-config.yml mysqlDatabase: kubemsb persistence: enabled: true storageClass: nfs-client ``` 升级并且加一个`--set imageTag=5.7.31`参数设置为5.7.31版本 ```powershell [root@k8s-master01 ~]# helm upgrade mysql -f mysql-config.yml --set imageTag=5.7.31 stable/mysql ``` ~~~powershell 升级过程中的输出： WARNING: This chart is deprecated Release "mysql" has been upgraded. Happy Helming! NAME: mysql LAST DEPLOYED: Fri Jul 29 15:04:20 2022 NAMESPACE: default STATUS: deployed REVISION: 2 NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: mysql.default.svc.cluster.local To get your root password run: MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) To connect to your database: 1. Run an Ubuntu pod that you can use as a client: kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il 2. Install the mysql client: $ apt-get update && apt-get install mysql-client -y 3. Connect using the mysql cli, then provide your password: $ mysql -h mysql -p To connect to your database directly from outside the K8s cluster: MYSQL_HOST=127.0.0.1 MYSQL_PORT=3306 # Execute the following command to route the connection: kubectl port-forward svc/mysql 3306 mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD} 注意：更新过程中，密码会被更新，但是实际使用中，密码并未更新。 ~~~ 3, 升级后确认版本 ```powershell [root@k8s-master01 helmdir]# kubectl get deployment mysql -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR mysql 1/1 1 1 58m mysql mysql:5.7.31 app=mysql,release=mysql ``` ~~~powershell [root@k8s-master01 helmdir]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-6f57f64c9d-sc72v 1/1 Running 0 2m20s 100.119.84.72 k8s-worker01 ~~~ ~~~powershell [root@k8s-master01 helmdir]# mysql -h 100.119.84.72 -uroot -pwL2SD0RCsT -e "select version()" +-----------+ | version() | +-----------+ | 5.7.31 | 版本升级为5.7.31 +-----------+ ~~~ 4, 回滚 ~~~powershell [root@k8s-master01 helmdir]# helm history mysql REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Fri Jul 29 14:07:17 2022 superseded mysql-1.6.9 5.7.30 Install complete 2 Fri Jul 29 15:04:20 2022 deployed mysql-1.6.9 5.7.30 Upgrade complete ~~~ ~~~powershell [root@k8s-master01 helmdir]# helm rollback mysql 1 Rollback was a success! Happy Helming! ~~~ 5, 验证 ~~~powershell [root@k8s-master01 helmdir]# kubectl get deployment mysql -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR mysql 1/1 1 1 65m mysql mysql:5.7.30 app=mysql,release=mysql ~~~ ~~~powershell [root@k8s-master01 helmdir]# helm history mysql REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Fri Jul 29 14:07:17 2022 superseded mysql-1.6.9 5.7.30 Install complete 2 Fri Jul 29 15:04:20 2022 superseded mysql-1.6.9 5.7.30 Upgrade complete 3 Fri Jul 29 15:12:24 2022 deployed mysql-1.6.9 5.7.30 Rollback to 1 ~~~ ## 5.8 更多安装方式 和yum命令类似 - chart 仓库 - 本地 chart 压缩包 ~~~powershell [root@k8s-master01 helmdir]# helm pull stable/mysql [root@k8s-master01 helmdir]# ls mysql-1.6.9.tgz mysql-1.6.9.tgz [root@k8s-master01 helmdir]# helm install mysql2 mysql-1.6.9.tgz ~~~ - 在线的 URL（helm install fool https://example.com/charts/foo-1.2.3.tgz） # 六、Chart包开发 ## 6.1 Chart 目录结构 ![1605264781567](../../img/kubernetes/kubernetes_helm/2.png) ~~~powershell [root@k8s-master01 helmdir]# helm create foo [root@k8s-master01 helmdir]# tree foo foo ├── charts ├── Chart.yaml ├── templates │   ├── deployment.yaml │   ├── _helpers.tpl │   ├── hpa.yaml │   ├── ingress.yaml │   ├── NOTES.txt │   ├── serviceaccount.yaml │   ├── service.yaml │   └── tests │   └── test-connection.yaml └── values.yaml ~~~ ~~~powershell [root@master ~]# helm pull stable/mysql [root@master ~]# tar xf mysql-1.6.8.tgz [root@master ~]# ls mysql Chart.yaml README.md templates values.yaml [root@master ~]# ls -l mysql/templates/ total 48 -rwxr-xr-x 1 root root 292 Jan 1 1970 configurationFiles-configmap.yaml -rwxr-xr-x 1 root root 8930 Jan 1 1970 deployment.yaml -rwxr-xr-x 1 root root 1290 Jan 1 1970 _helpers.tpl -rwxr-xr-x 1 root root 295 Jan 1 1970 initializationFiles-configmap.yaml -rwxr-xr-x 1 root root 2036 Jan 1 1970 NOTES.txt -rwxr-xr-x 1 root root 868 Jan 1 1970 pvc.yaml -rwxr-xr-x 1 root root 1475 Jan 1 1970 secrets.yaml -rwxr-xr-x 1 root root 328 Jan 1 1970 serviceaccount.yaml -rwxr-xr-x 1 root root 800 Jan 1 1970 servicemonitor.yaml -rwxr-xr-x 1 root root 1231 Jan 1 1970 svc.yaml drwxr-xr-x 2 root root 50 Nov 13 18:43 tests ~~~ | 文件 | 说明 | | ----------------- | ------------------------------------------------------------ | | Chart.yaml | 用于描述Chart的基本信息; `helm show chart stable/mysql`命令查看的内容就是此文件内容 | | values.yaml | Chart的默认配置文件; `helm show values stable/mysql`命令查看的内容就是此文件内容 | | README.md | [可选] 当前Chart的介绍 | | LICENS | [可选] 协议 | | requirements.yaml | [可选] 用于存放当前Chart依赖的其它Chart的说明文件 | | charts/ | [可选]: 该目录中放置当前Chart依赖的其它Chart | | templates/ | [可选]: 部署文件模版目录 | ## 6.2 创建不可配置的chart ### 1, 创建目录与chart.yaml ```powershell [root@k8s-master01 ~]# mkdir -p /helm/nginx/templates [root@k8s-master01 ~]# cd /helm/nginx ``` ```powershell [root@k8s-master01 nginx]# vim Chart.yaml name: helm-nginx version: 1.0.0 apiVersion: v1 appVersion: "1.0" description: A Helm chart for Kubernetes ``` ### 2, 创建deployment.yaml ```powershell [root@k8s-master01 nginx]# vim templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: helm-nginx spec: replicas: 1 selector: matchLabels: app: helm-nginx template: metadata: labels: app: helm-nginx spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ``` ### **3, 创建service.yaml** ```ruby [root@k8s-master01 nginx]# vim templates/service.yaml apiVersion: v1 kind: Service metadata: name: helm-nginx spec: selector: app: helm-nginx ports: - port: 80 targetPort: 80 protocol: TCP ``` ### 4, 使用chart安装应用 ~~~powershell [root@k8s-master01 nginx]# helm install /helm/nginx --generate-name NAME: nginx-1659144826 LAST DEPLOYED: Sat Jul 30 09:33:46 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ~~~ ### 5, 查看与验证 ~~~powershell [root@k8s-master01 nginx]# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx-1659144826 default 1 2022-07-30 09:33:46.881083524 +0800 CST deployed helm-nginx-1.0.0 ~~~ ~~~powershell [root@k8s-master01 nginx]# kubectl get pods,service NAME READY STATUS RESTARTS AGE pod/helm-nginx-65f57fb758-nrpvf 1/1 Running 0 51s pod/nfs-client-provisioner-9d46587b5-7n2vf 1/1 Running 4 (31m ago) 42h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/helm-nginx ClusterIP 10.96.2.120 80/TCP 51s ~~~ ```powershell [root@k8s-master01 nginx]# curl http://10.96.2.120 Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

Thank you for using nginx.

``` ### 6, 删除 ~~~powershell [root@k8s-master01 ~]# helm uninstall nginx-1659144826 release "nginx-1659144826" uninstalled ~~~ ## 6.3 创建可配置的Chart ### 6.3.1 官方的预定义变量 - Release.Name：发布的名称（不是chart） - Release.Time：chart发布上次更新的时间。这将匹配Last ReleasedRelease对象上的时间。 - Release.Namespace：chart发布到的名称空间。 - Release.Service：进行发布的服务。 - Release.IsUpgrade：如果当前操作是升级或回滚，则设置为true。 - Release.IsInstall：如果当前操作是安装，则设置为true。 - Release.Revision：修订号。它从1开始，每个都递增helm upgrade。 - Chart：内容Chart.yaml。因此，chart版本可以Chart.Version和维护者一样获得 Chart.Maintainers。 - Files：类似于chart的对象，包含chart中的所有非特殊文件。这不会授予您访问模板的权限，但可以访问存在的其他文件（除非使用它们除外.helmignore）。可以使用{{index .Files "file.name"}}或使用{{.Files.Get name}}或 {{.Files.GetStringname}}函数访问文件。您也可以访问该文件的内容，[]byte使用{{.Files.GetBytes}} - Capabilities：类似于地图的对象，包含有关Kubernetes（{{.Capabilities.KubeVersion}}，Tiller（{{.Capabilities.TillerVersion}}和支持的Kubernetes API）版本（{{.Capabilities.APIVersions.Has "batch/v1"）的版本的信息 ### 6.3.2 新增values.yaml文件 ```powershell [root@k8s-master01 nginx]# pwd /helm/nginx [root@k8s-master01 nginx]# vim values.yaml image: repository: nginx tag: '1.15-alpine' replicas: 2 ``` ### 6.3.3 配置deploy引用values的值 ```powershell [root@k8s-master01 nginx]# vim templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: helm-nginx spec: replicas: {{ .Values.replicas }} selector: matchLabels: app: helm-nginx template: metadata: labels: app: helm-nginx spec: containers: - name: helm-nginx image: {{ .Values.image.repository }}:{{ .Values.image.tag }} imagePullPolicy: IfNotPresent ``` ### 6.3.4 测试 #### 6.3.4.1 直接应用测试 > deployment.yaml将直接使用values.yaml中的配置 ~~~powershell [root@k8s-master01 nginx]# helm install helm-nginx-new /helm/nginx NAME: helm-nginx-new LAST DEPLOYED: Sat Jul 30 09:44:21 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ~~~ ~~~powershell [root@k8s-master01 nginx]# kubectl get pods NAME READY STATUS RESTARTS AGE helm-nginx-65f57fb758-pcmkg 1/1 Running 0 38s helm-nginx-65f57fb758-rmmv5 1/1 Running 0 38s ~~~ #### 6.3.4.2 通过命令行设置变量后干运行测试 > 通过在命令行设置变量为deployment.yaml赋值，使用--set选项，使用`--dry-run`选项来打印出生成的清单文件内容，而不执行部署 ~~~powershell [root@k8s-master01 nginx]# helm install helm-nginx --set replicas=3 /helm/nginx/ --dry-run NAME: helm-nginx LAST DEPLOYED: Fri Nov 13 20:57:45 2020 NAMESPACE: default STATUS: pending-install 状态表示是测试，不是真的部署了 REVISION: 1 TEST SUITE: None HOOKS: MANIFEST: --- # Source: helm-nginx/templates/service.yaml apiVersion: v1 kind: Service metadata: name: helm-nginx spec: selector: app: helm-nginx ports: - port: 80 targetPort: 80 protocol: TCP --- # Source: helm-nginx/templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: helm-nginx spec: replicas: 3 副本数量3传参成功 selector: matchLabels: app: helm-nginx template: metadata: labels: app: helm-nginx spec: containers: - name: helm-nginx image: nginx:1.15-alpine 镜像名:TAG 传参成功 imagePullPolicy: IfNotPresent ~~~ ```powershell [root@k8s-master01 nginx]# helm install helm-nginx --set replicas=3 /helm/nginx NAME: helm-nginx LAST DEPLOYED: Sat Jul 30 09:54:00 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ``` ```powershell [root@k8s-master01 nginx]# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION helm-nginx default 1 2022-07-30 09:54:00.744748457 +0800 CST deployed helm-nginx-1.0.0 ``` ~~~powershell [root@k8s-master01 nginx]# kubectl get pods,svc NAME READY STATUS RESTARTS AGE pod/helm-nginx-65f57fb758-j768m 1/1 Running 0 59s pod/helm-nginx-65f57fb758-pscjh 1/1 Running 0 58s pod/helm-nginx-65f57fb758-s6qqj 1/1 Running 0 58s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/helm-nginx ClusterIP 10.96.1.197 80/TCP 59s ~~~ ### 6.3.5将Chart包进行打包 > 将chart打包成一个压缩文件，便于存储与分享。 ```powershell [root@k8s-master01 nginx]# helm package . Successfully packaged chart and saved it to: /helm/nginx/helm-nginx-1.0.0.tgz ``` ~~~powershell [root@k8s-master01 nginx]# ls Chart.yaml helm-nginx-1.0.0.tgz templates values.yaml 打包出mychart-0.1.0.tgz文件 ~~~ ### 6.3.6 使用Chart安装 ~~~powershell [root@master nginx]# helm install helm-nginx2 /helm/nginx/helm-nginx-1.0.0.tgz ~~~ # 七、Chart包托管至Harbor方案 ## 7.1 集群外harbor服务器准备 ### 7.1.1 docker-ce安装 ~~~powershell wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ~~~powershell [root@nfsserver harbor]# yum -y install docker-ce ~~~ ~~~powershell [root@nfsserver harbor]# systemctl enable --now docker ~~~ ### 7.1.2 docker-compose安装 ~~~powershell [root@nfsserver ~]# wget https://github.com/docker/compose/releases/download/1.25.0/docker-compose-Linux-x86_64 ~~~ ~~~powershell [root@nfsserver ~]# mv docker-compose-Linux-x86_64 /usr/bin/docker-compose [root@nfsserver ~]# chmod +x /usr/bin/docker-compose ~~~ ~~~powershell [root@nfsserver ~]# docker-compose version docker-compose version 1.25.0, build 0a186604 docker-py version: 4.1.0 CPython version: 3.7.4 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019 ~~~ ### 7.1.3 harbor服务器安装 ~~~powershell [root@nfsserver ~]# wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz ~~~ ~~~powershell [root@nfsserver harbor]# ls 6864844_kubemsb.com.key 6864844_kubemsb.com.pem common.sh harbor.v2.5.3.tar.gz harbor.yml.tmpl install.sh LICENSE prepare ~~~ ~~~powershell [root@nfsserver harbor]# mv harbor.yml.tmpl harbor.yml ~~~ ~~~powershell # vim harbor.yaml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: www.kubemsb.com # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /home/harbor/6864844_kubemsb.com.pem private_key: /home/harbor/6864844_kubemsb.com.key # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: 12345 # Harbor DB configuration ~~~ ~~~powershell [root@nfsserver harbor]# ./prepare ~~~ ~~~powershell [root@nfsserver harbor]# ./install.sh -h Note: Please set hostname and other necessary attributes in harbor.yml first. DO NOT use localhost or 127.0.0.1 for hostname, because Harbor needs to be accessed by external clients. Please set --with-notary if needs enable Notary in Harbor, and set ui_url_protocol/ssl_cert/ssl_cert_key in harbor.yml bacause notary must run under https. Please set --with-trivy if needs enable Trivy in Harbor Please set --with-chartmuseum if needs enable Chartmuseum in Harbor ~~~ ~~~powershell [root@nfsserver harbor]# ./install.sh --with-chartmuseum ~~~ ~~~powershell [root@nfsserver harbor]# docker ps ~~~ > 在主机上解决域名 192.168.10.146 www.kubemsb.com ![image-20220730112415250](../../img/kubernetes/kubernetes_helm/image-20220730112415250.png) ![image-20220730112450460](../../img/kubernetes/kubernetes_helm/image-20220730112450460.png) ![image-20220730112550547](../../img/kubernetes/kubernetes_helm/image-20220730112550547.png) ![image-20220730112628591](../../img/kubernetes/kubernetes_helm/image-20220730112628591.png) ![image-20220730112729226](../../img/kubernetes/kubernetes_helm/image-20220730112729226.png) ![image-20220730113015244](../../img/kubernetes/kubernetes_helm/image-20220730113015244.png) ![image-20220730113039787](../../img/kubernetes/kubernetes_helm/image-20220730113039787.png) ![image-20220730113133287](../../img/kubernetes/kubernetes_helm/image-20220730113133287.png) ![image-20220730113158610](../../img/kubernetes/kubernetes_helm/image-20220730113158610.png) ~~~powershell [root@k8s-master01 ~]# helm repo add harborhelm https://www.kubemsb.com/chartrepo/nginx --username admin --password 12345 "harborhelm" has been added to your repositories ~~~ ~~~powershell [root@k8s-master01 ~]# helm repo list NAME URL micosoft http://mirror.azure.cn/kubernetes/charts/ prometheus-community https://prometheus-community.github.io/helm-charts harborhelm https://www.kubemsb.com/chartrepo/nginx ~~~ ~~~powershell [root@k8s-master01 ~]# helm search repo helm-nginx NAME CHART VERSION APP VERSION DESCRIPTION harborhelm/helm-nginx 1.0.0 ~~~ ~~~powershell [root@k8s-master01 ~]# helm install helm-nginx-test harborhelm/helm-nginx NAME: helm-nginx-test LAST DEPLOYED: Sat Jul 30 20:32:05 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None ~~~ ~~~powershell [root@k8s-master01 ~]# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION helm-nginx-test default 1 2022-07-30 20:32:05.138180077 +0800 CST deployed helm-nginx-1.0.0 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE helm-nginx-65f57fb758-2hkl6 1/1 Running 0 8s helm-nginx-65f57fb758-v427b 1/1 Running 0 8s ~~~ ### 安装helmpush插件 需要安装helmpush插件才能上传 * 在线直接安装 ~~~powershell [root@k8s-master01 nginx]# helm plugin install https://github.com/chartmuseum/helm-push Downloading and installing helm-push v0.10.3 ... https://github.com/chartmuseum/helm-push/releases/download/v0.10.3/helm-push_0.10.3_linux_amd64.tar.gz Installed plugin: cm-push ~~~ ~~~powershell [root@k8s-master01 nginx]# ls /root/.local/share/helm/plugins/helm-push/bin/ . .. helm-cm-push ~~~ ### 将打包应用push到harbor ~~~powershell [root@k8s-master01 nginx]# ls Chart.yaml templates values.yaml [root@k8s-master01 nginx]# vim Chart.yaml name: helm-nginx version: 1.1.0 [root@k8s-master01 nginx]# helm package . Successfully packaged chart and saved it to: /helm/nginx/helm-nginx-1.1.0.tgz [root@k8s-master01 nginx]# ls Chart.yaml helm-nginx-1.1.0.tgz templates values.yaml ~~~ ~~~powershell [root@k8s-master01 nginx]# helm -h The Kubernetes package manager ... Available Commands: cm-push Please see https://github.com/chartmuseum/helm-push for usage ~~~ ~~~powershell [root@k8s-master01 nginx]# helm cm-push --username admin --password 12345 helm-nginx-1.1.0.tgz harborhelm Pushing helm-nginx-1.1.0.tgz to harborhelm... Done. ~~~ ![image-20220730232227146](../../img/kubernetes/kubernetes_helm/image-20220730232227146.png) # 八、Helm Chart包可视化管理 Kubeapps应用商店 ## 8.1 kubeapps介绍 Kubeapps提供了一个开源的Helm UI界面，方便以图形界面的形式管理Helm应用。 - 部署应用。可从公共或私有仓库中浏览chart并将其部署到集群中； - 管理应用。升级、管理和删除部署在kubernetes集群中的应用程序； - 搜索功能。Kubeapps提供chart搜索页面； ![image-20220730235006506](../../img/kubernetes/kubernetes_helm/image-20220730235006506.png) ## 8.2 使用helm部署kubeapps ~~~powershell [root@k8s-master01 ~]# helm repo add bitnami https://charts.bitnami.com/bitnami "bitnami" has been added to your repositories ~~~ ~~~powershell [root@k8s-master01 ~]# helm repo list NAME URL micosoft http://mirror.azure.cn/kubernetes/charts/ prometheus-community https://prometheus-community.github.io/helm-charts harborhelm https://www.kubemsb.com/chartrepo/nginx bitnami https://charts.bitnami.com/bitnami ~~~ ~~~powershell [root@k8s-master01 ~]# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "harborhelm" chart repository ...Successfully got an update from the "prometheus-community" chart repository ...Successfully got an update from the "micosoft" chart repository ...Successfully got an update from the "bitnami" chart repository Update Complete. ⎈Happy Helming!⎈ ~~~ ~~~powershell [root@k8s-master01 ~]# helm search repo kubeapps NAME CHART VERSION APP VERSION DESCRIPTION bitnami/kubeapps 10.0.2 2.4.6 Kubeapps is a web-based UI for launching and ma... ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl create ns kubeapps namespace/kubeapps created ~~~ ~~~powershell [root@k8s-master01 ~]# helm install kubeapps bitnami/kubeapps --namespace kubeapps ~~~ ~~~powershell 输出信息： NAME: kubeapps LAST DEPLOYED: Sun Jul 31 00:00:03 2022 NAMESPACE: kubeapps STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: kubeapps CHART VERSION: 10.0.2 APP VERSION: 2.4.6** Please be patient while the chart is being deployed ** Tip: Watch the deployment status using the command: kubectl get pods -w --namespace kubeapps Kubeapps can be accessed via port 80 on the following DNS name from within your cluster: kubeapps.kubeapps.svc.cluster.local To access Kubeapps from outside your K8s cluster, follow the steps below: 1. Get the Kubeapps URL by running these commands: echo "Kubeapps URL: http://127.0.0.1:8080" kubectl port-forward --namespace kubeapps service/kubeapps 8080:80 2. Open a browser and access Kubeapps using the obtained URL. ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n kubeapps NAME READY STATUS RESTARTS AGE apprepo-kubeapps-sync-bitnami-w5jlr-6nvsl 1/1 Running 0 30s kubeapps-994b988b5-dlp9r 1/1 Running 0 96s kubeapps-994b988b5-ttxv8 1/1 Running 0 96s kubeapps-internal-apprepository-controller-c78bf86bc-lcpz2 1/1 Running 0 96s kubeapps-internal-dashboard-6445c69c6b-fgkrs 1/1 Running 0 96s kubeapps-internal-dashboard-6445c69c6b-phmxn 1/1 Running 0 96s kubeapps-internal-kubeappsapis-c5d8cbb7f-6rnnl 1/1 Running 0 96s kubeapps-internal-kubeappsapis-c5d8cbb7f-9jthp 1/1 Running 0 96s kubeapps-internal-kubeops-58794f58c8-bkjtc 1/1 Running 0 96s kubeapps-internal-kubeops-58794f58c8-qk655 1/1 Running 0 96s kubeapps-postgresql-0 1/1 Running 0 96s ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get svc -n kubeapps NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubeapps ClusterIP 10.96.2.39 80/TCP 2m18s kubeapps-internal-dashboard ClusterIP 10.96.1.130 8080/TCP 2m18s kubeapps-internal-kubeappsapis ClusterIP 10.96.2.40 8080/TCP 2m18s kubeapps-internal-kubeops ClusterIP 10.96.3.116 8080/TCP 2m18s kubeapps-postgresql ClusterIP 10.96.0.235 5432/TCP 2m18s kubeapps-postgresql-hl ClusterIP None 5432/TCP 2m18s ~~~ ## 8.3 访问kubeapps ~~~powershell [root@k8s-master01 ~]# vim kubeapps-ingress.yaml [root@k8s-master01 ~]# cat kubeapps-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-kubeapps #自定义ingress名称 namespace: kubeapps annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: kubeapps.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: kubeapps # 对应上面创建的service名称 port: number: 80 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl apply -f kubeapps-ingress.yaml ingress.networking.k8s.io/ingress-kubeapps created ~~~ ![image-20220731003858917](../../img/kubernetes/kubernetes_helm/image-20220731003858917.png) ![image-20220731003937933](../../img/kubernetes/kubernetes_helm/image-20220731003937933.png) 参考链接：https://kubeapps.dev/docs/latest/tutorials/getting-started/ ~~~powershell 创建用户 [root@k8s-master01 ~]# kubectl create --namespace default serviceaccount kubeapps-operator serviceaccount/kubeapps-operator created ~~~ ~~~powershell 绑定集群管理员角色 [root@k8s-master01 ~]# kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=default:kubeapps-operator clusterrolebinding.rbac.authorization.k8s.io/kubeapps-operator created ~~~ ~~~powershell [root@k8s-master01 ~]# cat < kube-prometheus-stack.yaml-default ~~~ ~~~powershell # cp kube-prometheus-stack.yaml-default kube-prometheus-stack.yaml ~~~ ~~~powershell # vim kube-prometheus-stack.yaml 2389 serviceMonitorSelectorNilUsesHelmValues: true 把true修改为false ~~~ ~~~powershell # helm install kps prometheus-community/kube-prometheus-stack -f kube-prometheus-stack.yaml -n monitoring --create-namespace --version 37.2.0 --debug ~~~ ~~~powershell kubectl --namespace monitoring get pods -l "release=kps" ~~~ ~~~powershell # kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 23m kps-grafana ClusterIP 10.96.1.75 80/TCP 24m kps-kube-prometheus-stack-alertmanager ClusterIP 10.96.0.115 9093/TCP 24m kps-kube-prometheus-stack-operator ClusterIP 10.96.0.29 443/TCP 24m kps-kube-prometheus-stack-prometheus ClusterIP 10.96.0.220 9090/TCP 24m kps-kube-state-metrics ClusterIP 10.96.1.192 8080/TCP 24m kps-prometheus-node-exporter ClusterIP 10.96.3.21 9100/TCP 24m prometheus-operated ClusterIP None 9090/TCP 23m ~~~ ~~~powershell # kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-kps-kube-prometheus-stack-alertmanager-0 2/2 Running 0 24m kps-grafana-65f5f8f47d-zwt44 3/3 Running 0 24m kps-kube-prometheus-stack-operator-667b454c64-zlgq2 1/1 Running 0 24m kps-kube-state-metrics-7d987f86f9-w5wkq 1/1 Running 0 24m kps-prometheus-node-exporter-7v4c8 1/1 Running 0 24m kps-prometheus-node-exporter-9gmjm 1/1 Running 0 24m kps-prometheus-node-exporter-lmdxp 1/1 Running 0 24m kps-prometheus-node-exporter-pf5kz 1/1 Running 0 24m prometheus-kps-kube-prometheus-stack-prometheus-0 2/2 Running 0 24m ~~~ # 四、配置prometheus及grafana访问 ## 4.1 prometheus ~~~powershell # cat pro.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-prometheus #自定义ingress名称 namespace: monitoring annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: prometheus.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: kps-kube-prometheus-stack-prometheus # 对应上面创建的service名称 port: number: 9090 ~~~ ~~~powershell kubectl apply -f pro.yaml ~~~ > 把prometheus.kubemsb.com域名与ingress nginx controller svc对应的IP进行解析即可访问。 ![image-20220714141949162](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714141949162.png) ## 4.2 grafana ~~~powershell # cat grafana.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-grafana #自定义ingress名称 namespace: monitoring annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: grafana.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: kps-grafana # 对应上面创建的service名称 port: number: 80 ~~~ ~~~powershell kubectl apply -f grafana.yaml ~~~ ~~~powershell # kubectl get secret -n monitoring NAME TYPE DATA AGE kps-grafana Opaque 3 28m ~~~ ~~~powershell # kubectl get secret kps-grafana -n monitoring -o yaml apiVersion: v1 data: admin-password: cHJvbS1vcGVyYXRvcg== admin-user: YWRtaW4= ldap-toml: "" kind: Secret metadata: annotations: meta.helm.sh/release-name: kps meta.helm.sh/release-namespace: monitoring creationTimestamp: "2022-07-14T05:52:02Z" labels: app.kubernetes.io/instance: kps app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: grafana app.kubernetes.io/version: 9.0.2 helm.sh/chart: grafana-6.32.2 name: kps-grafana namespace: monitoring resourceVersion: "4929" uid: 98f435cf-4e27-475c-ade9-3562da262f37 type: Opaque ~~~ ~~~powershell # echo -n "YWRtaW4=" | base64 --decode admin ~~~ ~~~powershell # echo -n "cHJvbS1vcGVyYXRvcg==" | base64 --decode prom-operator ~~~ ![image-20220714142355210](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142355210.png) ![image-20220714142427741](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142427741.png) ![image-20220714142453429](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142453429.png) ![image-20220714142514495](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142514495.png) # 五、使用prometheus监控redis ## 5.1 redis集群部署 ~~~powershell # helm repo add bitnami https://charts.bitnami.com/bitnami "bitnami" has been added to your repositories ~~~ ~~~powershell # helm repo update ~~~ ~~~powershell # helm search repo redis NAME CHART VERSION APP VERSION DESCRIPTION bitnami/redis 17.0.1 7.0.3 Redis(R) is an open source, advanced key-value ... bitnami/redis-cluster 8.0.0 7.0.3 Redis(R) is an open source, scalable, distribut... ~~~ ~~~powershell # helm install redis bitnami/redis --set global.storageClass=managed-nfs-storage --set global.redis.password=root --set architecture=standalone --version 17.0.1 -n redisns --create-namespace --debug ~~~ ~~~powershell # kubectl get all -n redisns NAME READY STATUS RESTARTS AGE pod/redis-master-0 1/1 Running 0 6m19s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/redis-headless ClusterIP None 6379/TCP 6m19s service/redis-master ClusterIP 10.96.3.75 6379/TCP 6m19s NAME READY AGE statefulset.apps/redis-master 1/1 6m19s ~~~ ~~~powershell # dig -t a redis-headless.redisns.svc.cluster.local @10.96.0.10 ~~~ ## 5.2 部署redis-exporter ~~~powershell # cat d.yaml apiVersion: apps/v1 # 版本 kind: Deployment # 资源 metadata: labels: app: redis-exporter # 标签 name: redis-exporter # deploy的名称 namespace: monitoring # 命名空间 spec: replicas: 1 # 副本 selector: matchLabels: k8s-app: redis-exporter # po标签 strategy: {} template: metadata: labels: k8s-app: redis-exporter # po标签 spec: containers: - image: oliver006/redis_exporter:latest name: redis-exporter args: ["-redis.addr", "redis://redis-headless.redisns.svc.cluster.local:6379", "-redis.password", "root"] ports: - containerPort: 9121 # 默认暴露的端口号 name: http ~~~ ~~~powershell kubectl apply -f d.yaml ~~~ ~~~powershell # kubectl get deployment -n monitoring NAME READY UP-TO-DATE AVAILABLE AGE kps-grafana 1/1 1 1 57m kps-kube-prometheus-stack-operator 1/1 1 1 57m kps-kube-state-metrics 1/1 1 1 57m redis-exporter 1/1 1 1 22s ~~~ ~~~powershell # cat svc.yaml apiVersion: v1 kind: Service metadata: name: redis-exporter namespace: monitoring labels: k8s-app: redis-exporter spec: ports: - port: 9121 protocol: TCP name: http selector: k8s-app: redis-exporter type: ClusterIP ~~~ ~~~powershell # kubectl apply -f svc.yaml ~~~ ~~~powershell # kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ..... redis-exporter ClusterIP 10.96.2.78 9121/TCP 8s ~~~ ~~~powershell curl 10.96.2.78:9121/metrics | tail -1 ... redis_uptime_in_seconds 881 ~~~ ~~~powershell # cat sm.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: redis-exporter namespace: monitoring labels: k8s-app: redis-exporter spec: endpoints: - interval: 10s port: http # 这个是port 对应 Service.spec.ports.name scheme: http jobLabel: k8s-app selector: matchLabels: k8s-app: redis-exporter # 跟 svc 的 lables 保持一致 namespaceSelector: matchNames: - monitoring # svc的命名空间 ~~~ ~~~powershell # kubectl apply -f sm.yaml ~~~ ~~~powershell # kubectl get servicemonitor -n monitoring NAME AGE ...... redis-exporter 46s ~~~ ![image-20220715002329243](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715002329243.png) ![image-20220715003418546](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003418546.png) ![image-20220715003450859](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003450859.png) ![image-20220715003558443](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003558443.png) ![image-20220715003335285](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003335285.png) # helm部署prometheus监控系统及应用 # 一、helm安装 ~~~powershell wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz ~~~ ~~~powershell tar xf helm-v3.8.2-linux-amd64.tar.gz ~~~ ~~~powershell mv linux-amd64/helm /bin/helm ~~~ ~~~powershell # helm version version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"} ~~~ # 二、helm添加prometheus仓库 ~~~powershell helm repo add prometheus-community https://prometheus-community.github.io/helm-charts ~~~ ~~~powershell helm repo update ~~~ ~~~powershell # helm search repo prometheus NAME CHART VERSION APP VERSION DESCRIPTION prometheus-community/kube-prometheus-stack 37.2.0 0.57.0 kube-prometheus-stack collects Kubernetes manif. ~~~ # 三、使用helm安装prometheus全家桶 ~~~powershell # helm show values prometheus-community/kube-prometheus-stack > kube-prometheus-stack.yaml-default ~~~ ~~~powershell # cp kube-prometheus-stack.yaml-default kube-prometheus-stack.yaml ~~~ ~~~powershell # vim kube-prometheus-stack.yaml 2389 serviceMonitorSelectorNilUsesHelmValues: true 把true修改为false ~~~ ~~~powershell # helm install kps prometheus-community/kube-prometheus-stack -f kube-prometheus-stack.yaml -n monitoring --create-namespace --version 37.2.0 --debug ~~~ ~~~powershell kubectl --namespace monitoring get pods -l "release=kps" ~~~ ~~~powershell # kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 23m kps-grafana ClusterIP 10.96.1.75 80/TCP 24m kps-kube-prometheus-stack-alertmanager ClusterIP 10.96.0.115 9093/TCP 24m kps-kube-prometheus-stack-operator ClusterIP 10.96.0.29 443/TCP 24m kps-kube-prometheus-stack-prometheus ClusterIP 10.96.0.220 9090/TCP 24m kps-kube-state-metrics ClusterIP 10.96.1.192 8080/TCP 24m kps-prometheus-node-exporter ClusterIP 10.96.3.21 9100/TCP 24m prometheus-operated ClusterIP None 9090/TCP 23m ~~~ ~~~powershell # kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-kps-kube-prometheus-stack-alertmanager-0 2/2 Running 0 24m kps-grafana-65f5f8f47d-zwt44 3/3 Running 0 24m kps-kube-prometheus-stack-operator-667b454c64-zlgq2 1/1 Running 0 24m kps-kube-state-metrics-7d987f86f9-w5wkq 1/1 Running 0 24m kps-prometheus-node-exporter-7v4c8 1/1 Running 0 24m kps-prometheus-node-exporter-9gmjm 1/1 Running 0 24m kps-prometheus-node-exporter-lmdxp 1/1 Running 0 24m kps-prometheus-node-exporter-pf5kz 1/1 Running 0 24m prometheus-kps-kube-prometheus-stack-prometheus-0 2/2 Running 0 24m ~~~ # 四、配置prometheus及grafana访问 ## 4.1 prometheus ~~~powershell # cat pro.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-prometheus #自定义ingress名称 namespace: monitoring annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: prometheus.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: kps-kube-prometheus-stack-prometheus # 对应上面创建的service名称 port: number: 9090 ~~~ ~~~powershell kubectl apply -f pro.yaml ~~~ > 把prometheus.kubemsb.com域名与ingress nginx controller svc对应的IP进行解析即可访问。 ![image-20220714141949162](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714141949162.png) ## 4.2 grafana ~~~powershell # cat grafana.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-grafana #自定义ingress名称 namespace: monitoring annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: grafana.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: kps-grafana # 对应上面创建的service名称 port: number: 80 ~~~ ~~~powershell kubectl apply -f grafana.yaml ~~~ ~~~powershell # kubectl get secret -n monitoring NAME TYPE DATA AGE kps-grafana Opaque 3 28m ~~~ ~~~powershell # kubectl get secret kps-grafana -n monitoring -o yaml apiVersion: v1 data: admin-password: cHJvbS1vcGVyYXRvcg== admin-user: YWRtaW4= ldap-toml: "" kind: Secret metadata: annotations: meta.helm.sh/release-name: kps meta.helm.sh/release-namespace: monitoring creationTimestamp: "2022-07-14T05:52:02Z" labels: app.kubernetes.io/instance: kps app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: grafana app.kubernetes.io/version: 9.0.2 helm.sh/chart: grafana-6.32.2 name: kps-grafana namespace: monitoring resourceVersion: "4929" uid: 98f435cf-4e27-475c-ade9-3562da262f37 type: Opaque ~~~ ~~~powershell # echo -n "YWRtaW4=" | base64 --decode admin ~~~ ~~~powershell # echo -n "cHJvbS1vcGVyYXRvcg==" | base64 --decode prom-operator ~~~ ![image-20220714142355210](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142355210.png) ![image-20220714142427741](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142427741.png) ![image-20220714142453429](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142453429.png) ![image-20220714142514495](../../img/kubernetes/kubernetes_helm_prometheus/image-20220714142514495.png) # 五、使用prometheus监控redis ## 5.1 使用helm实现redis集群部署 ~~~powershell # helm repo add bitnami https://charts.bitnami.com/bitnami "bitnami" has been added to your repositories ~~~ ~~~powershell # helm repo update ~~~ ~~~powershell # helm search repo redis NAME CHART VERSION APP VERSION DESCRIPTION bitnami/redis 17.0.1 7.0.3 Redis(R) is an open source, advanced key-value ... bitnami/redis-cluster 8.0.0 7.0.3 Redis(R) is an open source, scalable, distribut... ~~~ ~~~powershell # helm install redis bitnami/redis --set global.storageClass=managed-nfs-storage --set global.redis.password=root --set architecture=standalone --version 17.0.1 -n redisns --create-namespace --debug ~~~ ~~~powershell # kubectl get all -n redisns NAME READY STATUS RESTARTS AGE pod/redis-master-0 1/1 Running 0 6m19s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/redis-headless ClusterIP None 6379/TCP 6m19s service/redis-master ClusterIP 10.96.3.75 6379/TCP 6m19s NAME READY AGE statefulset.apps/redis-master 1/1 6m19s ~~~ ~~~powershell # dig -t a redis-headless.redisns.svc.cluster.local @10.96.0.10 ~~~ ## 5.2 监控方案一：部署redis-exporter及serviceMonitor ~~~powershell # cat d.yaml apiVersion: apps/v1 # 版本 kind: Deployment # 资源 metadata: labels: app: redis-exporter # 标签 name: redis-exporter # deploy的名称 namespace: monitoring # 命名空间 spec: replicas: 1 # 副本 selector: matchLabels: k8s-app: redis-exporter # po标签 strategy: {} template: metadata: labels: k8s-app: redis-exporter # po标签 spec: containers: - image: oliver006/redis_exporter:latest name: redis-exporter args: ["-redis.addr", "redis://redis-headless.redisns.svc.cluster.local:6379", "-redis.password", "root"] ports: - containerPort: 9121 # 默认暴露的端口号 name: http ~~~ ~~~powershell kubectl apply -f d.yaml ~~~ ~~~powershell # kubectl get deployment -n monitoring NAME READY UP-TO-DATE AVAILABLE AGE kps-grafana 1/1 1 1 57m kps-kube-prometheus-stack-operator 1/1 1 1 57m kps-kube-state-metrics 1/1 1 1 57m redis-exporter 1/1 1 1 22s ~~~ ~~~powershell # cat svc.yaml apiVersion: v1 kind: Service metadata: name: redis-exporter namespace: monitoring labels: k8s-app: redis-exporter spec: ports: - port: 9121 protocol: TCP name: http selector: k8s-app: redis-exporter type: ClusterIP ~~~ ~~~powershell # kubectl apply -f svc.yaml ~~~ ~~~powershell # kubectl get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ..... redis-exporter ClusterIP 10.96.2.78 9121/TCP 8s ~~~ ~~~powershell curl 10.96.2.78:9121/metrics | tail -1 ... redis_uptime_in_seconds 881 ~~~ ~~~powershell # cat sm.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: redis-exporter namespace: monitoring labels: k8s-app: redis-exporter spec: endpoints: - interval: 10s port: http # 这个是port 对应 Service.spec.ports.name scheme: http jobLabel: k8s-app selector: matchLabels: k8s-app: redis-exporter # 跟 svc 的 lables 保持一致 namespaceSelector: matchNames: - monitoring # svc的命名空间 ~~~ ~~~powershell # kubectl apply -f sm.yaml ~~~ ~~~powershell # kubectl get servicemonitor -n monitoring NAME AGE ...... redis-exporter 46s ~~~ ![image-20220715002329243](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715002329243.png) ![image-20220715003418546](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003418546.png) ![image-20220715003450859](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003450859.png) ![image-20220715003558443](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003558443.png) ![image-20220715003335285](../../img/kubernetes/kubernetes_helm_prometheus/image-20220715003335285.png) ![image-20220720144636315](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720144636315.png) ## 5.3 监控方案二：使用helm部署prometheus redis exporter监控redis > 监控helm部署的redis ~~~powershell 查看服务是否存在 # kubectl get svc -n redisns NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redis-headless ClusterIP None 6379/TCP 5d13h redis-master ClusterIP 10.96.0.228 6379/TCP 5d13h ~~~ ~~~powershell 查看是否有redis exporter helm search repo redis ~~~ ~~~powershell 下载prometheus-redis-exporter # helm fetch prometheus-community/prometheus-redis-exporter ~~~ ~~~powershell 解压prometheus-redis-exporter # tar xf prometheus-redis-exporter-5.0.0.tgz ~~~ ~~~powershell 进入prometheus-redis-exporter目录 # cd prometheus-redis-exporter/ ~~~ ~~~powershell 解析访问链接，确认是否可以访问 # dig -t a redis-headless.redisns.svc.cluster.local @10.96.0.10 ~~~ ~~~powershell 修改values.yaml文件，重点看修改部分 49 # If serviceMonitor.multipleTarget is enabled, this configuration is actually not used 50 redisAddress: redis://redis-headless.redisns.svc.cluster.local:6379 配置redis访问地址 51 52 # deployment additional annotations and labels 53 annotations: 把默认｛｝去掉 54 prometheus.io/path: /metrics 默认此位置没有，复制过来即可 55 prometheus.io/port: "9121" 默认此位置没有，复制过来即可 56 prometheus.io/scrape: "true" 默认此位置没有，复制过来即可 ... 67 serviceMonitor: 68 # When set true then use a ServiceMonitor to configure scraping 69 enabled: true 默认为false，修改为true 70 multipleTarget: false 71 targets: 72 # for every targets, url and name must be set, 73 # an individual additionalRelabeling can be set for every target 74 - url: "redis://redis-headless.redisns.svc.cluster.local:6379" 添加上面的redis访问地址 75 name: "myold-redis" 设置redis监控名称 76 # - url: "redis://my-redis-cluster:6379" ... 154 auth: 155 # Use password authentication 156 enabled: false 157 # Use existing secret (ignores redisPassword) 158 secret: 159 name: "" 160 key: "" 161 # Redis password (when not stored in a secret) 162 redisPassword: "root" 设置redis访问密码 163 # Redis user (version 6.X and above) 164 redisUser: "" ~~~ ~~~powershell 安装prometheus-redis-exporter helm install redis-monitor-old ./ -f values.yaml -n monitoring ~~~ ![image-20220720132959064](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720132959064.png) ## 5.4 使用helm部署prometheus redis exporter监控手动部署的redis > 单独复制一份prometheus redis exporter即可 ### 5.4 1 redis部署 ~~~powershell # vim redis.yaml # cat redis.yaml apiVersion: v1 kind: ConfigMap metadata: name: redis namespace: test data: redis.conf: |+ requirepass redis@passwd maxmemory 268435456 --- apiVersion: apps/v1 kind: Deployment metadata: name: redis namespace: test labels: app: redis spec: selector: matchLabels: app: redis template: metadata: labels: app: redis annotations: version/date: "20220720" version/author: "fc" spec: containers: - name: redis image: redis imagePullPolicy: Always command: ["redis-server","/etc/redis/redis.conf"] ports: - containerPort: 6379 volumeMounts: - name: redis-config mountPath: /etc/redis/redis.conf subPath: redis.conf volumes: - name: redis-config configMap: name: redis items: - key: redis.conf path: redis.conf --- kind: Service apiVersion: v1 metadata: name: redis namespace: test spec: selector: app: redis ports: - port: 6379 targetPort: 6379 ~~~ ~~~powershell # kubectl apply -f redis.yaml ~~~ ~~~powershell # kubectl get all -n test NAME READY STATUS RESTARTS AGE pod/redis-75f489d8f4-5gf4r 1/1 Running 0 82m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/redis ClusterIP 10.96.2.16 6379/TCP 82m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis 1/1 1 1 82m NAME DESIRED CURRENT READY AGE replicaset.apps/redis-75f489d8f4 1 1 1 82m ~~~ ~~~powershell 下载prometheus-redis-exporter # helm fetch prometheus-community/prometheus-redis-exporter ~~~ ~~~powershell 解压prometheus-redis-exporter # tar xf prometheus-redis-exporter-5.0.0.tgz ~~~ ~~~powershell # cd prometheus-redis-exporter/ ~~~ ~~~powershell # vim values.yaml ... 49 # If serviceMonitor.multipleTarget is enabled, this configuration is actually not used 50 redisAddress: redis://redis.test.svc.cluster.local:6379 修改为redis访问地址 51 52 # deployment additional annotations and labels 53 annotations: 54 prometheus.io/path: /metrics 默认没有，复制过来即可 55 prometheus.io/port: "9121" 默认没有，复制过来即可 56 prometheus.io/scrape: "true" 默认没有，复制过来即可 57 labels: {} ... 67 serviceMonitor: 68 # When set true then use a ServiceMonitor to configure scraping 69 enabled: true 默认为false,修改为true 70 multipleTarget: false 71 targets: 72 # for every targets, url and name must be set, 73 # an individual additionalRelabeling can be set for every target 74 - url: "redis://redis.test.svc.cluster.local:6379" 默认没有开启，添加访问链接后开启即可 75 name: "my-redis" 默认没有开启，开启即可，可以随意起名 ... 154 auth: 155 # Use password authentication 156 enabled: true 157 # Use existing secret (ignores redisPassword) 158 secret: 159 name: "" 160 key: "" 161 # Redis password (when not stored in a secret) 162 redisPassword: "redis@passwd" 密码要与redis配置文件中密码保持一致即可 163 # Redis user (version 6.X and above) 164 redisUser: "" ~~~ ~~~powershell helm install redis-monitor ./ -f values.yaml -n monitoring ~~~ ![image-20220720134242565](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720134242565.png) ## 5.5 实现helm部署prometheus redis exporter监控helm部署的redis集群 ~~~powershell # helm search repo redis NAME CHART VERSION APP VERSION DESCRIPTION bitnami/redis 17.0.1 7.0.3 Redis(R) is an open source, advanced key-value ... bitnami/redis-cluster 8.0.0 7.0.3 Redis(R) is an open source, scalable, distribut... ~~~ ~~~powershell # helm install redis bitnami/redis --set global.storageClass=managed-nfs-storage --set global.redis.password=root --version 17.0.1 -n redisns --create-namespace --debug ~~~ ~~~powershell 下载prometheus-redis-exporter # helm pull prometheus-community/prometheus-redis-exporter ~~~ ~~~powershell 解压prometheus-redis-exporter # tar xf prometheus-redis-exporter-5.0.0.tgz ~~~ ~~~powershell 进入prometheus-redis-exporter目录 cd prometheus-redis-exporter/ ~~~ ~~~powershell # vim values.yaml ... 49 # If serviceMonitor.multipleTarget is enabled, this configuration is actually not used 50 redisAddress: redis://redis.redisnew.svc.cluster.local:6379 修改为redis访问地址 51 52 # deployment additional annotations and labels 53 annotations: 54 prometheus.io/path: /metrics 默认没有，复制过来即可 55 prometheus.io/port: "9121" 默认没有，复制过来即可 56 prometheus.io/scrape: "true" 默认没有，复制过来即可 57 labels: {} ... 67 serviceMonitor: 68 # When set true then use a ServiceMonitor to configure scraping 69 enabled: true 默认为false,修改为true 70 multipleTarget: false 71 targets: 72 # for every targets, url and name must be set, 73 # an individual additionalRelabeling can be set for every target 74 - url: "redis://redis.redisnew.svc.cluster.local:6379" 默认没有开启，添加访问链接后开启即可 75 name: "my-redis" 默认没有开启，开启即可，可以随意起名 ... 154 auth: 155 # Use password authentication 156 enabled: true 157 # Use existing secret (ignores redisPassword) 158 secret: 159 name: "" 160 key: "" 161 # Redis password (when not stored in a secret) 162 redisPassword: "root" 密码要与redis配置文件中密码保持一致即可 163 # Redis user (version 6.X and above) 164 redisUser: "" ~~~ ~~~powershell # helm install redisnew-exporter ./ -f values.yaml -n monitoring ~~~ ![image-20220722002908512](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722002908512.png) # 六、使用prometheus监控mysql ## 6.1 mysql部署 ~~~powershell # vim mysql.yaml # cat mysql.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: test spec: selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - image: mysql name: mysql env: - name: MYSQL_ROOT_PASSWORD value: root@mysql --- apiVersion: v1 kind: Service metadata: name: mysql namespace: test spec: type: ClusterIP ports: - port: 3306 targetPort: 3306 selector: app: mysql ~~~ ~~~powershell # kubectl apply -f mysql.yaml ~~~ ~~~powershell # kubectl get pods -n test NAME READY STATUS RESTARTS AGE mysql-98797c6bf-jpwvj 1/1 Running 0 72s ~~~ ~~~powershell # dig -t a mysql.test.svc.cluster.local @10.96.0.10 ~~~ ~~~powershell 连接信息汇总 host: mysql.test.svc.cluster.local port: 3306 user: root passwd: root@mysql ~~~ ## 6.2 mysql exporter部署及验证 ~~~powershell # helm pull prometheus-community/prometheus-mysql-exporter ~~~ ~~~powershell # tar xf prometheus-mysql-exporter-1.8.1.tgz ~~~ ~~~powershell # cd prometheus-mysql-exporter ~~~ ~~~powershell # vim values.yaml 20 serviceMonitor: 21 # enabled should be set to true to enable prometheus-operator discovery of this service 22 enabled: true 默认为false，修改为true 23 # interval is the interval at which metrics should be scraped 24 interval: 30s 开启 25 # scrapeTimeout is the timeout after which the scrape is ended 26 scrapeTimeout: 10s 开启 27 #namespace: monitoring 123 # mysql connection params which build the DATA_SOURCE_NAME env var of the docker container 124 mysql: 125 db: "" 126 host: "mysql.test.svc.cluster.local" 127 param: "" 128 pass: "root@mysql" 129 port: 3306 130 protocol: "" 131 user: "root" ~~~ ~~~powershell # helm install prometheus-mysql-exporter ./ -f values.yaml -n monitoring ~~~ ![image-20220720143121566](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720143121566.png) ## 6.3 在prometheus web页面中Graph中查看 ![image-20220720143923630](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720143923630.png) ## 6.4 在grafana中添加dashboard ![image-20220720144246814](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720144246814.png) ![image-20220720144319367](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720144319367.png) ![image-20220720144357335](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720144357335.png) ![image-20220720144419911](../../img/kubernetes/kubernetes_helm_prometheus/image-20220720144419911.png) # 七、使用prometheus监控kafka ## 7.1 kafak部署 ### 7.1.1 下载chart包 ~~~powershell # helm repo add bitnami https://charts.bitnami.com/bitnami ~~~ ~~~powershell # helm repo update ~~~ ~~~powershell # helm search repo zookeeper NAME CHART VERSION APP VERSION DESCRIPTION bitnami/zookeeper 10.0.2 3.8.0 Apache ZooKeeper provides a reliable, centraliz... ~~~ ~~~powershell # helm search repo kafka NAME CHART VERSION APP VERSION DESCRIPTION bitnami/kafka 18.0.3 3.2.0 Apache Kafka is a distributed streaming platfor... ~~~ ### 7.1.2 部署zookeeper ~~~powershell # mkdir kafkadir ~~~ ~~~powershell # cd kafkadir ~~~ ~~~powershell # helm pull bitnami/zookeeper ~~~ ~~~powershell # ls zookeeper-10.0.2.tgz # tar xf zookeeper-10.0.2.tgz # ls zookeeper zookeeper-10.0.2.tgz ~~~ ~~~powershell # cd zookeeper/ # ls Chart.lock charts Chart.yaml README.md templates values.yaml ~~~ > 主要配置时区、持久化存储、副本数等 > > 首次部署时直接修改values中的配置，方便后期更新（upgrade）,也可以使用--set设置 ~~~powershell # vim values.yaml # 手动添加时区 # 改模板没有extraEnvVars字段，直接复制以下3行内容，部署后，可通过logs查看时区是否正确 extraEnvVars: - name: TZ value: "Asia/Shanghai" # 允许任意用户连接（默认开启） allowAnonymousLogin: true # 关闭认证（默认关闭） auth: enable: false # 修改副本数 replicaCount: 3 # 配置持久化存储 persistence: enabled: true storageClass: "managed-nfs-storage" accessModes: - ReadWriteOnce ~~~ ~~~powershell # vim values.yaml 97 auth: 98 client: 99 ## @param auth.client.enabled Enable ZooKeeper client-server authentication. It uses SASL/Digest-MD5 100 ## 101 enabled: false 使用默认值就可以了 ... 238 replicaCount: 3 默认为1，修改为3 ... 添加时区 219 extraEnvVars: 220 - name: TZ 221 value: "Asia/Shanghai" ... 581 persistence: 582 ## @param persistence.enabled Enable ZooKeeper data persistence using PVC. If false, use emptyDir 583 ## 584 enabled: true 585 ## @param persistence.existingClaim Name of an existing PVC to use (only when deploying a single replica) 586 ## 587 existingClaim: "" 588 ## @param persistence.storageClass PVC Storage Class for ZooKeeper data volume 589 ## If defined, storageClassName: 590 ## If set to "-", storageClassName: "", which disables dynamic provisioning 591 ## If undefined (the default) or set to null, no storageClassName spec is 592 ## set, choosing the default provisioner. (gp2 on AWS, standard on 593 ## GKE, AWS & OpenStack) 594 ## 595 storageClass: "nfs-client" 添加一个存储类 596 ## @param persistence.accessModes PVC Access modes 597 ## 598 accessModes: 599 - ReadWriteOnce 600 ## @param persistence.size PVC Storage Request for ZooKeeper data volume 601 ## 602 size: 8Gi 603 ## @param persistence.annotations Annotations for the PVC ... 708 serviceMonitor: 709 ## @param metrics.serviceMonitor.enabled Create ServiceMonitor Resource for scraping metrics using Prometheus Operator 710 ## 711 enabled: true 默认为false，修改为true 712 ## @param metrics.serviceMonitor.namespace Namespace for the ServiceMonitor Resource (defaults to the Release Namespace) 713 ## 714 namespace: "kafka" 默认为空，修改为test 715 ## @param metrics.serviceMonitor.interval Interval at which metrics should be scraped. 716 ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint 717 ## 718 interval: "10" 默认为空，修改为10 719 ## @param metrics.serviceMonitor.scrapeTimeout Timeout after which the scrape is ended 720 ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint 721 ## 722 scrapeTimeout: "20" 默认为空，修改为20 ~~~ ~~~powershell # kubectl create ns kafka ~~~ ~~~powershell # helm install zookeeper ./ -f values.yaml -n kafka ~~~ ~~~powershell # kubectl get pods -n kafka NAME READY STATUS RESTARTS AGE zookeeper-0 1/1 Running 0 62s zookeeper-1 1/1 Running 0 62s zookeeper-2 1/1 Running 0 62s ~~~ ~~~powershell [root@nfsserver ~]# ls /sdb kafka-data-zookeeper-1-pvc-9dbb13e2-3174-429b-90aa-3884459aaa23 kafka-data-zookeeper-2-pvc-0cffe920-2f16-4955-b80e-04e8d1571c82 kafka-data-zookeeper-0-pvc-a6ed8ed3-058a-4260-be1f-12d1be78b2a0 ~~~ ~~~powershell # kubectl get svc -n kafka NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE zookeeper ClusterIP 10.96.2.24 2181/TCP,2888/TCP,3888/TCP 2m9s zookeeper-headless ClusterIP None 2181/TCP,2888/TCP,3888/TCP 2m9s ~~~ ~~~powershell # dig -t a zookeeper-headless.kafka.svc.cluster.local @10.96.0.10 ~~~ ~~~powershell # kubectl get ns NAME STATUS AGE kafka Active 53m [root@k8s-master01 kafkadir]# kubectl get pods -n kafka NAME READY STATUS RESTARTS AGE zookeeper-0 1/1 Running 0 52m zookeeper-1 1/1 Running 0 52m zookeeper-2 1/1 Running 0 52m 查看zookeeper状态 [root@k8s-master01 kafkadir]# kubectl exec -it zookeeper-0 -n kafka -- bash @zookeeper-0:/$ zkServer.sh status /opt/bitnami/java/bin/java ZooKeeper JMX enabled by default Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower ~~~ ### 7.1.3 部署kafka ~~~powershell 拉取kafka chart包 # helm pull bitnami/kafka ~~~ ~~~powershell # ls kafka-18.0.3.tgz ~~~ ~~~powershell # tar xf kafka-18.0.3.tgz ~~~ ~~~powershell # cd kafka ~~~ #### 7.1.3.1 基础配置 > 设置时区、副本数、持久化存储、zookeeper连接等。 ~~~powershell # vim values.yaml 412 extraEnvVars: 413 - name: TZ 414 value: "Asia/Shanghai" ... 426 replicaCount: 3 ... 890 persistence: 891 ## @param persistence.enabled Enable Kafka data persistence using PVC, note that ZooKeeper persistence is unaffected 892 ## 893 enabled: true 894 ## @param persistence.existingClaim A manually managed Persistent Volume and Claim 895 ## If defined, PVC must be created manually before volume will be bound 896 ## The value is evaluated as a template 897 ## 898 existingClaim: "" 899 ## @param persistence.storageClass PVC Storage Class for Kafka data volume 900 ## If defined, storageClassName: 901 ## If set to "-", storageClassName: "", which disables dynamic provisioning 902 ## If undefined (the default) or set to null, no storageClassName spec is 903 ## set, choosing the default provisioner. 904 ## 905 storageClass: "nfs-client" 指定存储类 906 ## @param persistence.accessModes Persistent Volume Access Modes 907 ## 908 accessModes: 909 - ReadWriteOnce 910 ## @param persistence.size PVC Storage Request for Kafka data volume 911 ## 912 size: 8Gi 913 ## @param persistence.annotations Annotations for the PVC ... 1638 zookeeper: 1639 ## @param zookeeper.enabled Switch to enable or disable the ZooKeeper helm chart 1640 ## 1641 enabled: false 默认为true，修改为false,不使用内部zookeeper 1642 ## @param zookeeper.replicaCount Number of ZooKeeper nodes ... 1681 externalZookeeper: 1682 ## @param externalZookeeper.servers List of external zookeeper servers to use. Typically used in combination with 'zookeeperChrootPath'. 1683 ## 1684 servers: zookeeper 使用外部的zookeeper ~~~ #### 7.1.3.2 高可用配置 > 设置默认分区、默认副本数、日志过期时间，需要根据kafka节点数设定。 ~~~powershell # vim values.yaml 允许删除topic 138 deleteTopicEnable: true 默认为false 154 ## @param logRetentionHours The minimum age of a log file to be eligible for deletion due to age 默认日志保留时间，为一周 156 logRetentionHours: 168 166 ## @param defaultReplicationFactor Default replication factors for automatically created topics 自动创建topic默认的副本数 168 defaultReplicationFactor: 2 169 ## @param offsetsTopicReplicationFactor The replication factor for the offsets topic 用于配置offset记录的topic的partition的副本个数 171 offsetsTopicReplicationFactor: 2 172 ## @param transactionStateLogReplicationFactor The replication factor for the transaction topic 事务主题的复制因子 174 transactionStateLogReplicationFactor: 2 175 ## @param transactionStateLogMinIsr Overridden min.insync.replicas config for the transaction topic 177 transactionStateLogMinIsr: 2 默认为1修改为2 184 ## @param numPartitions The default number of log partitions per topic 新建Topic时默认的分区数 186 numPartitions: 3 ~~~ #### 7.1.3.3 kafka部署 ~~~powershell # helm install kafka ./ -f values.yaml -n kafka ~~~ ~~~powershell 输出内容： NAME: kafka LAST DEPLOYED: Wed Jul 20 18:59:51 2022 NAMESPACE: kafka STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: kafka CHART VERSION: 18.0.3 APP VERSION: 3.2.0 ** Please be patient while the chart is being deployed ** Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster: kafka.kafka.svc.cluster.local Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster: kafka-0.kafka-headless.kafka.svc.cluster.local:9092 kafka-1.kafka-headless.kafka.svc.cluster.local:9092 kafka-2.kafka-headless.kafka.svc.cluster.local:9092 To create a pod that you can use as a Kafka client run the following commands: kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.2.0-debian-11-r12 --namespace kafka --command -- sleep infinity kubectl exec --tty -i kafka-client --namespace kafka -- bash PRODUCER: kafka-console-producer.sh \ --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 \ --topic test CONSUMER: kafka-console-consumer.sh \ --bootstrap-server kafka.kafka.svc.cluster.local:9092 \ --topic test \ --from-beginning ~~~ ### 7.1.4 kafka调试 #### 7.1.4.1 查询与创建topic ~~~powershell 进入pod # kubectl exec -it kafka-0 -n kafka -- bash ~~~ ~~~powershell 创建topic,3分区+2副本,注意kafka的版本，本次使用的是3.2.0 @kafka-0:/$ kafka-topics.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test001 --create --partitions 3 --replication-factor 2 Created topic test001. ~~~ ~~~powershell 列出topic @kafka-0:/$ kafka-topics.sh --list --bootstrap-server kafka.kafka.svc.cluster.local:9092 test001 ~~~ ~~~powershell 查看topic详情 @kafka-0:/$ kafka-topics.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --describe --topic test001 输入结果： Topic: test001 TopicId: J5e8gTjPRV2b49NUWYwNaA PartitionCount: 3 ReplicationFactor: 2 Configs: flush.ms=1000,segment.bytes=1073741824,flush.messages=10000,max.message.bytes=1000012,retention.bytes=1073741824 Topic: test001 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: test001 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: test001 Partition: 2 Leader: 2 Replicas: 2,0 Isr: 2,0 ~~~ #### 7.1.4.2 生产者与消费者 ~~~powershell 在一个窗口中打开生产者创建数据 @kafka-0:/$ kafka-console-producer.sh --broker-list kafka:9092 --topic test001 >kubemsb 输入kubemsb后回车 ~~~ ~~~powershell 在另一个窗口中使用消费者访问数据 # kubectl exec -it kafka-0 -n kafka -- bash @kafka-0:/$ kafka-console-consumer.sh --bootstrap-server kafka:9092 --from-beginning --topic test001 kubemsb 显示 ~~~ #### 7.1.4.3 修改与删除topic ~~~powershell 修改topic配置：增加分区至4个（分区只可增不可减） @kafka-0:/$ kafka-topics.sh --alter --bootstrap-server kafka:9092 --partitions 4 --topic test001 查看增加后的结果 @kafka-0:/$ kafka-topics.sh --describe --bootstrap-server kafka:9092 --topic test001 输出结果： Topic: test001 TopicId: J5e8gTjPRV2b49NUWYwNaA PartitionCount: 4 ReplicationFactor: 2 Configs: flush.ms=1000,segment.bytes=1073741824,flush.messages=10000,max.message.bytes=1000012,retention.bytes=1073741824 Topic: test001 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: test001 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: test001 Partition: 2 Leader: 2 Replicas: 2,0 Isr: 2,0 Topic: test001 Partition: 3 Leader: 1 Replicas: 1,2 Isr: 1,2 ~~~ ~~~powershell 删除topic（需要设置deleteTopicEnable: true） @kafka-0:/$ kafka-topics.sh --delete --bootstrap-server kafka:9092 --topic test001 验证是否删除 @kafka-0:/$ kafka-topics.sh --list --bootstrap-server kafka.kafka.svc.cluster.local:9092 __consumer_offsets ~~~ ## 7.2 kafka exporter部署及验证 > 监控kafka之前可以创建topic ~~~powershell 获取kafka-exporter # helm search repo kafka NAME CHART VERSION APP VERSION DESCRIPTION prometheus-community/prometheus-kafka-exporter 1.6.0 v1.4.2 A Helm chart to export the metrics from Kafka i... ~~~ ~~~powershell # helm pull prometheus-community/prometheus-kafka-exporter ~~~ ~~~powershell # tar xf prometheus-kafka-exporter-1.6.0.tgz # ls prometheus-kafka-exporter ~~~ ~~~powershell 链接kafka信息： kafka.kafka.svc.cluster.local:9092 ~~~ ~~~powershell # cd prometheus-kafka-exporter # vim values.yaml ... 8 kafkaServer: 9 #- kafka-server:9092 10 - kafka.kafka.svc.cluster.local:9092 64 prometheus: 65 serviceMonitor: 66 enabled: true 67 namespace: monitoring 68 interval: "30s" 92 annotations: 93 prometheus.io/scrape: "true" 94 prometheus.io/path: "/metrics" 95 prometheus.io/port: "9308" ~~~ ~~~powershell # helm install kafka-exporter ./ -f values.yaml -n monitoring ~~~ ![image-20220722085015022](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722085015022.png) ## 7.3 在prometheus web中Graph查看 ![image-20220722085210137](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722085210137.png) > 如果想采集更多的数据，需要对消费者进行配置(--consumer-property)，以便获取更多的数据 ~~~powershell # kubectl exec -it kafka-0 -n kafka -- bash ~~~ ~~~powershell @kafka-0:/$ kafka-topics.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test002 --create --partitions 3 --replication-factor 2 Created topic test002. ~~~ ~~~powershell 创建生产者生产数据 !@kafka-0:/$ kafka-console-producer.sh --broker-list kafka:9092 --topic test002 >kubemsb >abc >hello ~~~ ~~~powershell 创建消费者获取数据 # kubectl exec -it kafka-2 -n kafka -- bash I have no name!@kafka-2:/$ kafka-console-consumer.sh --bootstrap-server kafka:9092 --from-beginning --topic test002 --consumer-property group.id=test kubemsb hello abc ~~~ ![image-20220722090203802](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722090203802.png) ## 7.4 在grafana中添加kafka监控dashboard > 7589 ![image-20220722090350722](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722090350722.png) ![image-20220722090424326](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722090424326.png) ![image-20220722090446732](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722090446732.png) ![image-20220722090518507](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722090518507.png) ![image-20220722090540207](../../img/kubernetes/kubernetes_helm_prometheus/image-20220722090540207.png) # 八、使用prometheus监控rabbitmq ## 8.1 rabbitmq部署 ### 8.1.1 获取rabbitmq部署文件 ~~~powershell # helm repo add bitnami https://charts.bitnami.com/bitnami ~~~ ~~~powershell # helm repo update ~~~ ~~~powershell # helm search repo rabbitmq ~~~ ~~~powershell # mkdir rabbitmqdir ~~~ ~~~powershell # cd rabbitmqdir ~~~ ~~~powershell # helm pull bitnami/rabbitmq ~~~ ~~~powershell # ls rabbitmq-10.1.15.tgz ~~~ ~~~powershell # tar xf rabbitmq-10.1.15.tgz ~~~ ~~~powershell # ls rabbitmq rabbitmq-10.1.15.tgz ~~~ ### 8.1.2 配置rabbitmq >- 配置持久化存储、副本数等 >- 建议首次部署时直接修改values中的配置，而不是用–set的方式，这样后期upgrade不必重复设置。 ~~~powershell # cd rabbitmq/ ~~~ ~~~powershell # ls Chart.lock charts Chart.yaml README.md templates values.schema.json values.yaml ~~~ #### 8.1.2.1 设置管理员密码 > 方案一:在values.yaml文件中指定 ~~~powershell # vim values.yaml 115 auth: 116 ## @param auth.username RabbitMQ application username 117 ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables 118 ## 119 username: admin 120 ## @param auth.password RabbitMQ application password 121 ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables 122 ## 123 password: "admin@mq" 124 ## @param auth.existingPasswordSecret Existing secret with RabbitMQ credentials (must contain a value for `rabbitmq-password` key) 125 ## e.g: 126 ## existingPasswordSecret: name-of-existing-secret 127 ## 128 existingPasswordSecret: "" 129 ## @param auth.erlangCookie Erlang cookie to determine whether different nodes are allowed to communicate with each other 130 ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables 131 ## 132 erlangCookie: "secretcookie" ~~~ > 方案二：在命令行执行时通过--set直接配置 ~~~powershell --set auth.username=admin,auth.password=admin@mq,auth.erlangCookie=secretcookie ~~~ #### 8.1.2.2 配置rabbitmq强制启动 >当rabbitmq启用持久化存储时，若rabbitmq所有pod同时宕机，将无法重新启动，因此有必要提前开启`clustering.forceBoot` ~~~powershell 211 ## Clustering settings 212 ## 213 clustering: 214 ## @param clustering.enabled Enable RabbitMQ clustering 215 ## 216 enabled: true 217 ## @param clustering.addressType Switch clustering mode. Either `ip` or `hostname` 218 ## 219 addressType: hostname 220 ## @param clustering.rebalance Rebalance master for queues in cluster when new replica is created 221 ## ref: https://www.rabbitmq.com/rabbitmq-queues.8.html#rebalance 222 ## 223 rebalance: false 224 ## @param clustering.forceBoot Force boot of an unexpectedly shut down cluster (in an unexpected order). 225 ## forceBoot executes 'rabbitmqctl force_boot' to force boot cluster shut down unexpectedly in an unknown order 226 ## ref: https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot 227 ## 228 forceBoot: true 由默认的false修改为true ~~~ #### 8.1.2.3 配置时区 ~~~powershell 268 extraEnvVars: 269 - name: TZ 270 value: "Asia/Shanghai" ~~~ #### 8.1.2.4 指定副本数 ~~~powershell 510 replicaCount: 3 由1修改为3 ~~~ #### 8.1.2.5 设置持久化存储 ~~~powershell 776 persistence: 777 ## @param persistence.enabled Enable RabbitMQ data persistence using PVC 778 ## 779 enabled: true 780 ## @param persistence.storageClass PVC Storage Class for RabbitMQ data volume 781 ## If defined, storageClassName: 782 ## If set to "-", storageClassName: "", which disables dynamic provisioning 783 ## If undefined (the default) or set to null, no storageClassName spec is 784 ## set, choosing the default provisioner. (gp2 on AWS, standard on 785 ## GKE, AWS & OpenStack) 786 ## 787 storageClass: "nfs-client" 添加存储类 788 ## @param persistence.selector Selector to match an existing Persistent Volume 789 ## selector: 790 ## matchLabels: 791 ## app: my-app 792 ## 793 selector: {} 794 ## @param persistence.accessModes PVC Access Modes for RabbitMQ data volume 795 ## 796 accessModes: 797 - ReadWriteOnce 798 ## @param persistence.existingClaim Provide an existing PersistentVolumeClaims 799 ## The value is evaluated as a template 800 ## So, for example, the name can depend on .Release or .Chart 801 ## 802 existingClaim: "" 803 ## @param persistence.mountPath The path the volume will be mounted at 804 ## Note: useful when using custom RabbitMQ images 805 ## 806 mountPath: /bitnami/rabbitmq/mnesia 807 ## @param persistence.subPath The subdirectory of the volume to mount to 808 ## Useful in dev environments and one PV for multiple services 809 ## 810 subPath: "" 811 ## @param persistence.size PVC Storage Request for RabbitMQ data volume 812 ## If you change this value, you might have to adjust `rabbitmq.diskFreeLimit` as well 813 ## 814 size: 5Gi ~~~ #### 8.1.2.6 关于service的设置说明 >- 默认通过ClusterIP暴露5672（amqp）和15672（web管理界面）等端口供集群内部使用，也可在外部访问，后面有说明 >- 不建议在values中直接配置nodeport，不方便后期灵活配置 ### 8.1.3 部署rabbitmq #### 8.1.3.1 创建命名空间 ~~~powershell # kubectl create namespace test ~~~ #### 8.1.3.2 安装 > 使用方式一方式安装 ~~~powershell # helm install rabbitmq ./ -f values.yaml -n test ~~~ ~~~powershell NAME: rabbitmq LAST DEPLOYED: Mon Jul 25 10:38:57 2022 NAMESPACE: test STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: rabbitmq CHART VERSION: 10.1.15 APP VERSION: 3.10.6** Please be patient while the chart is being deployed ** Credentials: echo "Username : admin" echo "Password : $(kubectl get secret --namespace test rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 -d)" echo "ErLang Cookie : $(kubectl get secret --namespace test rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 -d)" Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid. This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading. More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases. RabbitMQ can be accessed within the cluster on port 5672 at rabbitmq.test.svc.cluster.local To access for outside the cluster, perform the following steps: To Access the RabbitMQ AMQP port: echo "URL : amqp://127.0.0.1:5672/" kubectl port-forward --namespace test svc/rabbitmq 5672:5672 To Access the RabbitMQ Management interface: echo "URL : http://127.0.0.1:15672/" kubectl port-forward --namespace test svc/rabbitmq 15672:15672 ~~~ > 使用方式二方式安装,通过--set方式指定用户名与密码，方便后期通过upgrade进行更新。 ~~~powershell # helm install rabbitmq ./ -f values.yaml -n test --set auth.username=admin,auth.password=admin@mq,auth.erlangCookie=secretcookie ~~~ #### 8.1.3.3 查看rabbitmq安装状态 ~~~powershell # helm list -n test NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION rabbitmq test 1 2022-07-25 10:38:57.262289654 +0800 CST deployed rabbitmq-10.1.15 3.10.6 ~~~ ~~~powershell # kubectl get sts -n test NAME READY AGE rabbitmq 3/3 4m54s ~~~ ~~~powershell # kubectl get pods -n test NAME READY STATUS RESTARTS AGE rabbitmq-0 1/1 Running 0 5m8s rabbitmq-1 1/1 Running 0 4m6s rabbitmq-2 1/1 Running 0 3m4s ~~~ ~~~powershell # kubectl get svc -n test NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rabbitmq ClusterIP 10.96.3.23 5672/TCP,4369/TCP,25672/TCP,15672/TCP 4m29s rabbitmq-headless ClusterIP None 4369/TCP,5672/TCP,25672/TCP,15672/TCP 4m29s ~~~ #### 8.1.3.4 查看rabbitmq集群状态 ~~~powershell 集群状态 # kubectl exec -it rabbitmq-0 -n test -- bash I have no name!@rabbitmq-0:/$ rabbitmqctl cluster_status ~~~ ~~~powershell 列出策略（默认没有列出镜像模式） I have no name!@rabbitmq-0:/$ rabbitmqctl list_policies Listing policies for vhost "/" ... ~~~ ~~~powershell 设置rabbitmq集群名称 I have no name!@rabbitmq-0:/$ rabbitmqctl set_cluster_name kubemsb_rabbitmq Setting cluster name to kubemsb_rabbitmq ... ~~~ ### 8.1.4 设置RabbitMQ集群外部访问方式 ~~~powershell [root@k8s-master01 rabbitmqdir]# cat 5672.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: rabbitmq-5672 #自定义ingress名称 namespace: test annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: rabbitmq-5672.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: rabbitmq # 对应上面创建的service名称 port: number: 5672 ~~~ ~~~powershell [root@k8s-master01 rabbitmqdir]# kubectl apply -f 5672.yaml ingress.networking.k8s.io/rabbitmq-5672 created ~~~ ~~~powershell [root@k8s-master01 rabbitmqdir]# kubectl get ingress -n test NAME CLASS HOSTS ADDRESS PORTS AGE rabbitmq-5672 rabbitmq-5672.kubemsb.com 192.168.10.147 80 61s ~~~ ~~~powershell [root@k8s-master01 rabbitmqdir]# cat 15672.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: rabbitmq-15672 #自定义ingress名称 namespace: test annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: rabbitmq-15672.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: rabbitmq # 对应上面创建的service名称 port: number: 15672 ~~~ ~~~powershell [root@k8s-master01 rabbitmqdir]# kubectl apply -f 15672.yaml ingress.networking.k8s.io/rabbitmq-15672 created ~~~ ~~~powershell [root@k8s-master01 rabbitmqdir]# kubectl get ingress -n test NAME CLASS HOSTS ADDRESS PORTS AGE rabbitmq-15672 rabbitmq-15672.kubemsb.com 192.168.10.147 80 54s ~~~ ![image-20220725110542672](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725110542672.png) ![image-20220725110703112](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725110703112.png) ![image-20220725110744799](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725110744799.png) ![image-20220725110921083](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725110921083.png) ![image-20220725110831623](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725110831623.png) ### 8.1.5 配置镜像模式实现集群高可用 >镜像模式：将需要消费的队列变为镜像队列，存在于多个节点，这样就可以实现 RabbitMQ 的 HA 高可用性。作用就是消息实体会主动在镜像节点之间实现同步，而不是像普通模式那样，在 consumer 消费数据时临时读取。缺点就是，集群内部的同步通讯会占用大量的网络带宽。 ~~~powershell # kubectl exec -it rabbitmq-0 -n test -- bash ~~~ ~~~powershell I have no name!@rabbitmq-0:/$ rabbitmqctl list_policies Listing policies for vhost "/" ... ~~~ ~~~powershell I have no name!@rabbitmq-0:/$ rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}' Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all","ha-sync-mode":"automatic"}" with priority "0" for vhost "/" ... ~~~ ~~~powershell I have no name!@rabbitmq-0:/$ rabbitmqctl list_policies Listing policies for vhost "/" ... vhost name pattern apply-to definition priority / ha-all ^ all {"ha-mode":"all","ha-sync-mode":"automatic"} 0 ~~~ ![image-20220725113259819](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725113259819.png) ### 8.1.6 卸载rabbitmq > 由于接下来要实现监控，所以这个位置的卸载方法根据个人情况执行即可 ~~~powershell 卸载rabbitmq # kubectl uninstall rabbitmq -n test ~~~ ~~~powershell 删除pvc # kubectl get pvc -n test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-rabbitmq-0 Bound pvc-a1d13494-5dd2-4728-bd89-8b7d03db7a32 5Gi RWO nfs-client 58m data-rabbitmq-1 Bound pvc-224dcc95-1fe5-4ac5-982f-35ba36e92b16 5Gi RWO nfs-client 57m data-rabbitmq-2 Bound pvc-97358dbc-068d-4076-b046-b5efc80fce1a 5Gi RWO nfs-client 56m # kubectl delete pvc data-rabbitmq-0 data-rabbitmq-1 data-rabbitmq-2 -n test ~~~ ~~~powershell # kubectl delete -f 5672.yaml ~~~ ~~~powershell # kubectl delete -f 15672.yaml ~~~ ## 8.2 rabbitmq exporter部署及验证 ### 8.2.1 rabbitmq连接信息 ~~~powershell url:http://rabbitmq.test.svc.cluster.local:15672 username: admin password: 15672 ~~~ ### 8.2.2 rabbitmq exporter chart包下载 ~~~powershell # helm search repo rabbitmq-exporter NAME CHART VERSION APP VERSION DESCRIPTION prometheus-community/prometheus-rabbitmq-exporter 1.2.0 v0.29.0 Rabbitmq metrics exporter for prometheus ~~~ ~~~powershell # helm pull prometheus-community/prometheus-rabbitmq-exporter ~~~ ~~~powershell # ls prometheus-rabbitmq-exporter-1.2.0.tgz ~~~ ### 8.2.3 解压并配置 ~~~powershell # tar xf prometheus-rabbitmq-exporter-1.2.0.tgz ~~~ ~~~powershell # ls prometheus-rabbitmq-exporter ~~~ ~~~powershell # cd prometheus-rabbitmq-exporter/ # ls Chart.yaml README.md templates values.yaml ~~~ ~~~powershell # vim values.yaml 33 loglevel: info 34 rabbitmq: 35 url: http://rabbitmq.test.svc.cluster.local:15672 修改访问地址 36 user: admin 修改用户名 37 password: admin@mq 修改用户名对应的密码 38 # If existingPasswordSecret is set then password is ignored 39 existingPasswordSecret: ~ 40 existingPasswordSecretKey: password 41 capabilities: bert,no_sort 42 include_queues: ".*" 43 include_vhost: ".*" 44 skip_queues: "^$" 45 skip_verify: "false" 46 skip_vhost: "^$" 47 exporters: "exchange,node,overview,queue" 48 output_format: "TTY" 49 timeout: 30 50 max_queues: 0 51 52 ## Additional labels to set in the Deployment object. Together with standard labels from 53 ## the chart 54 additionalLabels: {} 55 56 podLabels: {} 57 58 annotations: 把{}去掉，把下面三行前面的#去掉 59 prometheus.io/scrape: "true" 60 prometheus.io/path: "/metrics" 61 prometheus.io/port: "9419" 63 prometheus: 64 monitor: 65 enabled: true 把false修改为true 66 additionalLabels: {} 67 interval: 15s 68 namespace: [] ~~~ ~~~powershell # helm install rabbitmq-exporter ./ -f values.yaml -n monitoring ~~~ ~~~powershell # helm list -n monitoring NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION rabbitmq-exporter monitoring 1 2022-07-25 12:03:31.769877663 +0800 CST deployed prometheus-rabbitmq-exporter-1.2.0 v0.29.0 ~~~ ### 8.2.4 在prometheus web界面中查看 ![image-20220725120633719](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725120633719.png) ![image-20220725122156046](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122156046.png) ### 8.2.5 在grafana添加dashboard > [RabbitMQ Monitoring](https://grafana.com/grafana/dashboards/4279)4279 > > [RabbitMQ Metrics](https://grafana.com/grafana/dashboards/4371)4371 ![image-20220725122440946](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122440946.png) ![image-20220725122527360](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122527360.png) ![image-20220725122605609](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122605609.png) ![image-20220725122648479](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122648479.png) ![image-20220725122722659](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122722659.png) ![image-20220725122753561](../../img/kubernetes/kubernetes_helm_prometheus/image-20220725122753561.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_hight.md ================================================ # kubeadm部署高可用kubernetes集群 1.21 ​ # 一、kubernetes 1.21发布 ![image-20220119160108054](../../img/kubernetes/kubernetes_hight/image-20220119160108054.png) ## 1.1 介绍 2021年04月，Kubernetes 1.21正式与大家见面，这是我们 2021 年的第一个版本！这个版本包含 51 个增强功能：13 个增强功能升级为稳定版，16 个增强功能升级为 beta 版，20 个增强功能进入 alpha 版，还有 2 个功能已经弃用。 ## 1.2 主要变化 - CronJobs 毕业到稳定！ 自 Kubernetes 1.8 以来，CronJobs一直是一个测试版功能！在 1.21 中，我们终于看到这个广泛使用的 API 毕业到稳定。 CronJobs 用于执行定期计划的操作，如备份、报告生成等。每个任务都应该被配置为无限期地重复出现（例如：一天/一周/一个月）；你可以在该间隔内定义作业应该启动的时间点。 - 不可变的 Secrets 和 ConfigMaps Immutable Secrets和ConfigMaps为这些资源类型添加了一个新字段，如果设置了该字段，将拒绝对这些对象的更改。默认情况下，Secrets 和 ConfigMaps 是可变的，这对能够使用更改的 pod 是有益的。如果将错误的配置推送给使用它们的 pod，可变的 Secrets 和 ConfigMaps 也会导致问题。 通过将 Secrets 和 ConfigMaps 标记为不可变的，可以确保应用程序配置不会改变。如果你希望进行更改，则需要创建一个新的、唯一命名的 Secret 或 ConfigMap，并部署一个新的 pod 来消耗该资源。不可变资源也有伸缩性优势，因为控制器不需要轮询 API 服务器来观察变化。 这个特性在 Kubernetes 1.21 中已经毕业到稳定。 - IPv4/IPv6 双栈支持 IP 地址是一种可消耗的资源，集群操作人员和管理员需要确保它不会耗尽。特别是，公共 IPv4 地址现在非常稀少。双栈支持使原生 IPv6 路由到 pod 和服务，同时仍然允许你的集群在需要的地方使用 IPv4。双堆栈集群网络还改善了工作负载的可能伸缩限制。 Kubernetes 的双栈支持意味着 pod、服务和节点可以获得 IPv4 地址和 IPv6 地址。在 Kubernetes 1.21 中，双栈网络已经从 alpha 升级到 beta，并且已经默认启用了。 - 优雅的节点关闭 在这个版本中，优雅的节点关闭也升级到测试版（现在将提供给更大的用户群）！这是一个非常有益的特性，它允许 kubelet 知道节点关闭，并优雅地终止调度到该节点的 pod。 目前，当节点关闭时，pod 不会遵循预期的终止生命周期，也不会正常关闭。这可能会在许多不同的工作负载下带来问题。接下来，kubelet 将能够通过 systemd 检测到即将发生的系统关闭，然后通知正在运行的 pod，以便它们能够尽可能优雅地终止。 - PersistentVolume 健康监测器 持久卷（Persistent Volumes，PV）通常用于应用程序中获取本地的、基于文件的存储。它们可以以许多不同的方式使用，并帮助用户迁移应用程序，而不需要重新编写存储后端。 Kubernetes 1.21 有一个新的 alpha 特性，允许对 PV 进行监视，以了解卷的运行状况，并在卷变得不健康时相应地进行标记。工作负载将能够对运行状况状态作出反应，以保护数据不被从不健康的卷上写入或读取。 - 减少 Kubernetes 的构建维护 以前，Kubernetes 维护了多个构建系统。这常常成为新贡献者和当前贡献者的摩擦和复杂性的来源。 在上一个发布周期中，为了简化构建过程和标准化原生的 Golang 构建工具，我们投入了大量的工作。这应该赋予更广泛的社区维护能力，并降低新贡献者进入的门槛。 ## 1.3 重大变化 - 弃用 PodSecurityPolicy 在 Kubernetes 1.21 中，PodSecurityPolicy 已被弃用。与 Kubernetes 所有已弃用的特性一样，PodSecurityPolicy 将在更多版本中继续可用并提供完整的功能。先前处于测试阶段的 PodSecurityPolicy 计划在 Kubernetes 1.25 中删除。 接下来是什么？我们正在开发一种新的内置机制来帮助限制 Pod 权限，暂定名为“PSP 替换策略”。我们的计划是让这个新机制覆盖关键的 PodSecurityPolicy 用例，并极大地改善使用体验和可维护性。 - 弃用 TopologyKeys 服务字段 topologyKeys 现在已弃用；所有使用该字段的组件特性以前都是 alpha 特性，现在也已弃用。我们用一种实现感知拓扑路由的方法替换了 topologyKeys，这种方法称为感知拓扑提示。支持拓扑的提示是 Kubernetes 1.21 中的一个 alpha 特性。你可以在拓扑感知提示中阅读关于替换特性的更多细节；相关的KEP解释了我们替换的背景。 # 二、kubernetes 1.21.0 部署工具介绍 ## What is Kubeadm ? `Kubeadm is a tool built to provide best-practice "fast paths" for creating Kubernetes clusters. It performs the actions necessary to get a minimum viable, secure cluster up and running in a user friendly way. Kubeadm's scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools.` Kubeadm是为创建Kubernetes集群提供最佳实践并能够“快速路径”构建kubernetes集群的工具。它能够帮助我们执行必要的操作，以获得最小可行的、安全的集群，并以用户友好的方式运行。 ## Common Kubeadm cmdlets 1. **kubeadm init** to bootstrap the initial Kubernetes control-plane node. `初始化` 2. **kubeadm join** to bootstrap a Kubernetes worker node or an additional control plane node, and join it to the cluster. `添加工作节点到kubernetes集群` 3. **kubeadm upgrade** to upgrade a Kubernetes cluster to a newer version. ` 更新kubernetes版本` 4. **kubeadm reset** to revert any changes made to this host by kubeadm init or kubeadm join. ` 重置kubernetes集群` # 三、kubernetes 1.21.0 部署环境准备 >3主2从 ## 3.1 主机操作系统说明 | 序号 | 操作系统及版本 | 备注 | | :--: | :------------: | :--: | | 1 | CentOS7u6 | | ## 3.2 主机硬件配置说明 | 需求 | CPU | 内存 | 硬盘 | 角色 | 主机名 | | ---- | ---- | ---- | ----- | ------------ | -------- | | 值 | 4C | 8G | 100GB | master | master01 | | 值 | 4C | 8G | 100GB | master | master02 | | 值 | 4C | 8G | 100GB | master | master03 | | 值 | 4C | 8G | 100GB | worker(node) | worker01 | | 值 | 4C | 8G | 100GB | worker(node) | worker02 | | 序号 | 主机名 | IP地址 | 备注 | | ---- | -------- | -------------- | ------ | | 1 | master01 | 192.168.10.11 | master | | 2 | master02 | 192.168.10.12 | master | | 3 | master03 | 192.168.10.13 | master | | 4 | worker01 | 192.168.10.14 | node | | 5 | worker02 | 192.168.10.15 | node | | 6 | master01 | 192.168.10.100 | vip | | 序号 | 主机名 | 功能 | 备注 | | ---- | -------- | ------------------- | ---------------- | | 1 | master01 | haproxy、keepalived | keepalived主节点 | | 2 | master02 | haproxy、keepalived | keepalived从节点 | ## 3.3 主机配置 ### 3.3.1 主机名配置 由于本次使用5台主机完成kubernetes集群部署，其中3台为master节点,名称为master01、master02、master03;其中2台为worker节点，名称分别为：worker01及worker02 ~~~powershell master节点,名称为master01 # hostnamectl set-hostname master01 ~~~ ~~~powershell master节点,名称为master02 # hostnamectl set-hostname master02 ~~~ ~~~powershell master节点,名称为master03 # hostnamectl set-hostname master03 ~~~ ~~~powershell worker1节点,名称为worker01 # hostnamectl set-hostname worker01 ~~~ ~~~powershell worker2节点,名称为worker02 # hostnamectl set-hostname worker02 ~~~ ### 3.3.2 主机IP地址配置 ~~~powershell master01节点IP地址为：192.168.10.11/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.11" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell master02节点IP地址为：192.168.10.12/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.12" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell master03节点IP地址为：192.168.10.13/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.13" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell worker01节点IP地址为：192.168.10.14/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.14" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell worker02节点IP地址为：192.168.10.15/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.15" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ### 3.3.3 主机名与IP地址解析 > 所有集群主机均需要进行配置。 ~~~powershell # cat /etc/hosts ...... 192.168.10.11 master01 192.168.10.12 master02 192.168.10.13 master03 192.168.10.14 worker01 192.168.10.15 worker02 ~~~ ### 3.3.4 防火墙配置 > 所有主机均需要操作。 ~~~powershell 关闭现有防火墙firewalld # systemctl disable firewalld # systemctl stop firewalld # firewall-cmd --state not running ~~~ ### 3.3.5 SELINUX配置 > 所有主机均需要操作。修改SELinux配置需要重启操作系统。 ~~~powershell # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ### 3.3.6 时间同步配置 >所有主机均需要操作。最小化安装系统需要安装ntpdate软件。 ~~~powershell # crontab -l 0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com ~~~ ### 3.3.7 多机互信 > 在master节点上生成证书，复制到其它节点即可。复制完成后，可以相互测试登录。 ~~~powershell # ssh-keygen ~~~ ~~~powershell # cd /root/.ssh [root@master01 .ssh]# ls id_rsa id_rsa.pub known_hosts [root@master01 .ssh]# cp id_rsa.pub authorized_keys ~~~ ~~~powershell # for i in 12 13 14 15; do scp -r /root/.ssh 192.168.10.$i:/root/; done ~~~ ### 3.3.8 升级操作系统内核 > 所有主机均需要操作。 ~~~powershell 导入elrepo gpg key # rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell 安装elrepo YUM源仓库 # yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell 安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 # yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell 设置grub2默认引导为0 # grub2-set-default 0 ~~~ ~~~powershell 重新生成grub2引导文件 # grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ~~~powershell 更新后，需要重启，使用升级的内核生效。 # reboot ~~~ ~~~powershell 重启后，需要验证内核是否为更新对应的版本 # uname -r ~~~ ### 3.3.9 配置内核转发及网桥过滤 >所有主机均需要操作。 ~~~powershell 添加网桥过滤及内核转发配置文件 # cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ ~~~powershell 加载br_netfilter模块 # modprobe br_netfilter ~~~ ~~~powershell 查看是否加载 # lsmod | grep br_netfilter br_netfilter 22256 0 bridge 151336 1 br_netfilter ~~~ ~~~powershell 加载网桥过滤及内核转发配置文件 # sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ ### 3.3.10 安装ipset及ipvsadm > 所有主机均需要操作。主要用于实现service转发。 ~~~powershell 安装ipset及ipvsadm # yum -y install ipset ipvsadm ~~~ ~~~powershell 配置ipvsadm模块加载方式 添加需要加载的模块 # cat > /etc/sysconfig/modules/ipvs.modules < 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a ~~~powershell 永远关闭swap分区，需要重启操作系统 # cat /etc/fstab ...... # /dev/mapper/centos-swap swap swap defaults 0 0 在上一行中行首添加# ~~~ ## 3.4 Docker准备 > 所有集群主机均需操作。 ### 3.4.1 获取YUM源 > 使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ### 3.4.2 查看可安装版本 ~~~powershell # yum list docker-ce.x86_64 --showduplicates | sort -r ~~~ ### 3.4.3 安装指定版本并设置启动及开机自启动 ~~~powershell # yum -y install --setopt=obsoletes=0 docker-ce-20.10.9-3.el7 ~~~ ~~~powershell # systemctl enable docker ; systemctl start docker ~~~ ### 3.4.4 修改cgroup方式 ~~~powershell 在/etc/docker/daemon.json添加如下内容 # cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } ~~~ ### 3.4.5 重启docker ~~~powershell # systemctl restart docker ~~~ # 四、HAProxy及Keepalived部署 ## 4.1 HAProxy及keepalived安装 ~~~powershell [root@master01 ~]# yum -y install haproxy keepalived ~~~ ~~~powershell [root@master02 ~]# yum -y install haproxy keepalived ~~~ ## 4.2 HAProxy配置及启动 ~~~powershell [root@master01 ~]# vim /etc/haproxy/haproxy.cfg [root@master01 ~]# cat /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:16443 bind 127.0.0.1:16443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server master01 192.168.10.11:6443 check server master02 192.168.10.12:6443 check server master03 192.168.10.13:6443 check ~~~ ~~~powershell [root@master01 ~]# systemctl enable haproxy;systemctl start haproxy [root@master01 ~]# systemctl status haproxy ~~~ ![image-20220119174750138](../../img/kubernetes/kubernetes_hight/image-20220119174750138.png) ~~~powershell [root@master01 ~]# scp /etc/haproxy/haproxy.cfg master02:/etc/haproxy/haproxy.cfg ~~~ ~~~powershell [root@master02 ~]# systemctl enable haproxy;systemctl start haproxy [root@master02 ~]# systemctl status haproxy ~~~ ![image-20220119175023889](../../img/kubernetes/kubernetes_hight/image-20220119175023889.png) ## 4.3 Keepalived配置及启动 ~~~powershell [root@master01 ~]# vim /etc/keepalived/keepalived.conf [root@master01 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" #此脚本需要多独定义，并要调用。 interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface ens33 # 修改为正在使用的网卡 mcast_src_ip 192.168.10.11 #为本master主机对应的IP地址 virtual_router_id 51 priority 101 advert_int 2 authentication { auth_type PASS auth_pass abc123 } virtual_ipaddress { 192.168.10.100 #为VIP地址 } track_script { chk_apiserver # 执行上面检查apiserver脚本 } } ~~~ ~~~powershell [root@master01 ~]# vim /etc/keepalived/check_apiserver.sh [root@master01 ~]# cat /etc/keepalived/check_apiserver.sh #!/bin/bash err=0 for k in $(seq 1 3) do check_code=$(pgrep haproxy) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi ~~~ ~~~powershell [root@master01 ~]# chmod +x /etc/keepalived/check_apiserver.sh ~~~ ~~~powershell [root@master01 ~]# scp /etc/keepalived/keepalived.conf master02:/etc/keepalived/ [root@master01 ~]# scp /etc/keepalived/check_apiserver.sh master02:/etc/keepalived/ ~~~ ~~~powershell [root@master02 ~]# vim /etc/keepalived/keepalived.conf [root@master02 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" #此脚本需要多独定义，并要调用。 interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface ens33 # 修改为正在使用的网卡 mcast_src_ip 192.168.10.12 #为本master主机对应的IP地址 virtual_router_id 51 priority 99 # 修改为99 advert_int 2 authentication { auth_type PASS auth_pass abc123 } virtual_ipaddress { 192.168.10.100 #为VIP地址 } track_script { chk_apiserver # 执行上面检查apiserver脚本 } } ~~~ ~~~powershell [root@master01 ~]# systemctl enable keepalived;systemctl start keepalived ~~~ ~~~powershell [root@master02 ~]# systemctl enable keepalived;systemctl start keepalived ~~~ ## 4.4 验证高可用集群可用性 ~~~powershell [root@master01 ~]# ip a s ens33 2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:50:f9:5f brd ff:ff:ff:ff:ff:ff inet 192.168.10.11/24 brd 192.168.10.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.10.100/32 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::adf4:a8bc:a1c:a9f7/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::2b33:40ed:9311:8812/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::8508:20d8:7240:32b2/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever ~~~ ~~~powershell [root@master01 ~]# ss -anput | grep ":16443" tcp LISTEN 0 2000 127.0.0.1:16443 *:* users:(("haproxy",pid=2983,fd=6)) tcp LISTEN 0 2000 *:16443 *:* users:(("haproxy",pid=2983,fd=5)) ~~~ ~~~powershell [root@master02 ~]# ss -anput | grep ":16443" tcp LISTEN 0 2000 127.0.0.1:16443 *:* users:(("haproxy",pid=2974,fd=6)) tcp LISTEN 0 2000 *:16443 *:* users:(("haproxy",pid=2974,fd=5)) ~~~ # 五、kubernetes 1.21.0 集群部署 ## 5.1 集群软件版本说明 | | kubeadm | kubelet | kubectl | | -------- | ---------------------- | --------------------------------------------- | ---------------------- | | 版本 | 1.21.0 | 1.21.0 | 1.21.0 | | 安装位置 | 集群所有主机 | 集群所有主机 | 集群所有主机 | | 作用 | 初始化集群、管理集群等 | 用于接收api-server指令，对pod生命周期进行管理 | 集群应用命令行管理工具 | ## 5.2 kubernetes YUM源准备 > 在/etc/yum.repos.d/目录中创建k8s.repo文件，把下面内容复制进去即可。 ### 5.2.1 谷歌YUM源 ~~~powershell [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg ~~~ ### 5.2.2 阿里云YUM源 ~~~powershell [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ~~~ ## 5.3 集群软件安装 ~~~powershell 查看指定版本 # yum list kubeadm.x86_64 --showduplicates | sort -r # yum list kubelet.x86_64 --showduplicates | sort -r # yum list kubectl.x86_64 --showduplicates | sort -r ~~~ ~~~powershell 安装指定版本 # yum -y install --setopt=obsoletes=0 kubeadm-1.21.0-0 kubelet-1.21.0-0 kubectl-1.21.0-0 ~~~ ## 5.4 配置kubelet >为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ~~~powershell # vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ~~~ ~~~powershell 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 # systemctl enable kubelet ~~~ ## 5.5 集群镜像准备 > 可使用VPN实现下载。 ~~~powershell # kubeadm config images list --kubernetes-version=v1.21.0 k8s.gcr.io/kube-apiserver:v1.21.0 k8s.gcr.io/kube-controller-manager:v1.21.0 k8s.gcr.io/kube-scheduler:v1.21.0 k8s.gcr.io/kube-proxy:v1.21.0 k8s.gcr.io/pause:3.4.1 k8s.gcr.io/etcd:3.4.13-0 k8s.gcr.io/coredns/coredns:v1.8.0 ~~~ ~~~powershell # cat image_download.sh #!/bin/bash images_list=' k8s.gcr.io/kube-apiserver:v1.21.0 k8s.gcr.io/kube-controller-manager:v1.21.0 k8s.gcr.io/kube-scheduler:v1.21.0 k8s.gcr.io/kube-proxy:v1.21.0 k8s.gcr.io/pause:3.4.1 k8s.gcr.io/etcd:3.4.13-0 k8s.gcr.io/coredns/coredns:v1.8.0' for i in $images_list do docker pull $i done docker save -o k8s-1-21-0.tar $images_list ~~~ ## 5.6 集群初始化 > 阿里云镜像仓库中的CoreDNS镜像下载有错误。 ~~~powershell [root@master01 ~]# vim kubeadm-config.yaml [root@master01 ~]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: 7t2weq.bjbawausm0jaxury ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.10.11 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: certSANs: - 192.168.10.100 timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 192.168.10.100:16443 controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.21.0 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {} ~~~ ~~~powershell [root@master01 ~]# vim kubeadm-config.yaml [root@master01 ~]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: 7t2weq.bjbawausm0jaxury ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.10.11 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: certSANs: - 192.168.10.100 timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 192.168.10.100:16443 controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: kind: ClusterConfiguration kubernetesVersion: v1.21.0 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {} ~~~ ~~~powershell [root@master01 ~]# kubeadm init --config /root/kubeadm-config.yaml --upload-certs ~~~ ~~~powershell 输出内容，一定保留，便于后继操作使用。 Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.10.100:16443 --token 7t2weq.bjbawausm0jaxury \ --discovery-token-ca-cert-hash sha256:085fc221ad8b5baffdaa567768a10d21eca2fc1f939fe73578ff725feea70ba4 \ --control-plane --certificate-key 9f74fd2c73a16a79fb9f458cd5874a860564070fd93c3912d910ba2b9c11a2b1 Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.100:16443 --token 7t2weq.bjbawausm0jaxury \ --discovery-token-ca-cert-hash sha256:085fc221ad8b5baffdaa567768a10d21eca2fc1f939fe73578ff725feea70ba4 ~~~ ## 5.7 集群应用客户端管理集群文件准备 ~~~powershell [root@master01 ~]# mkdir -p $HOME/.kube [root@master01 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master01 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@master01 ~]# ls /root/.kube/ config ~~~ ~~~powershell [root@master01 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf ~~~ ## 5.8 集群网络准备 > 使用calico部署集群网络 > > 安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico ### 5.8.1 calico安装 ![image-20220119141547207](../../img/kubernetes/kubernetes_hight/image-20220119141547207.png) ![image-20220119141645676](../../img/kubernetes/kubernetes_hight/image-20220119141645676.png) ![image-20220119141734347](../../img/kubernetes/kubernetes_hight/image-20220119141734347.png) ![image-20220119141830625](../../img/kubernetes/kubernetes_hight/image-20220119141830625.png) ~~~powershell 下载operator资源清多文件 # wget https://docs.projectcalico.org/manifests/tigera-operator.yaml ~~~ ~~~powershell 应用资源清多文件，创建operator # kubectl apply -f tigera-operator.yaml ~~~ ~~~powershell 通过自定义资源方式安装 # wget https://docs.projectcalico.org/manifests/custom-resources.yaml ~~~ ~~~powershell 修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 # vim custom-resources.yaml ...... 11 ipPools: 12 - blockSize: 26 13 cidr: 10.244.0.0/16 14 encapsulation: VXLANCrossSubnet ...... ~~~ ~~~powershell 应用资源清多文件 # kubectl apply -f custom-resources.yaml ~~~ ~~~powershell 监视calico-sysem命名空间中pod运行情况 # watch kubectl get pods -n calico-system ~~~ >Wait until each pod has the `STATUS` of `Running`. ~~~powershell 删除 master 上的 taint # kubectl taint nodes --all node-role.kubernetes.io/master- ~~~ ~~~powershell 已经全部运行 # kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-666bb9949-dzp68 1/1 Running 0 11m calico-node-jhcf4 1/1 Running 4 11m calico-typha-68b96d8d9c-7qfq7 1/1 Running 2 11m ~~~ ~~~powershell 查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 # kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-558bd4d5db-4jbdv 1/1 Running 0 113m coredns-558bd4d5db-pw5x5 1/1 Running 0 113m etcd-master01 1/1 Running 0 113m kube-apiserver-master01 1/1 Running 0 113m kube-controller-manager-master01 1/1 Running 4 113m kube-proxy-kbx4z 1/1 Running 0 113m kube-scheduler-master01 1/1 Running 3 113m ~~~ ### 5.8.2 calico客户端安装 ![image-20220119144207789](../../img/kubernetes/kubernetes_hight/image-20220119144207789.png) ![image-20220119144446449](../../img/kubernetes/kubernetes_hight/image-20220119144446449.png) ~~~powershell 下载二进制文件 # curl -L https://github.com/projectcalico/calico/releases/download/v3.21.4/calicoctl-linux-amd64 -o calicoctl ~~~ ~~~powershell 安装calicoctl # mv calicoctl /usr/bin/ 为calicoctl添加可执行权限 # chmod +x /usr/bin/calicoctl 查看添加权限后文件 # ls /usr/bin/calicoctl /usr/bin/calicoctl 查看calicoctl版本 # calicoctl version Client Version: v3.21.4 Git commit: 220d04c94 Cluster Version: v3.21.4 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm ~~~ ~~~powershell 通过~/.kube/config连接kubernetes集群，查看已运行节点 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME master01 ~~~ ## 5.9 集群其它Master节点加入集群 ~~~powershell [root@master02 ~]# kubeadm join 192.168.10.100:16443 --token 7t2weq.bjbawausm0jaxury \ > --discovery-token-ca-cert-hash sha256:085fc221ad8b5baffdaa567768a10d21eca2fc1f939fe73578ff725feea70ba4 \ > --control-plane --certificate-key 9f74fd2c73a16a79fb9f458cd5874a860564070fd93c3912d910ba2b9c11a2b1 ~~~ ~~~powershell [root@master03 ~]# kubeadm join 192.168.10.100:16443 --token 7t2weq.bjbawausm0jaxury \ > --discovery-token-ca-cert-hash sha256:085fc221ad8b5baffdaa567768a10d21eca2fc1f939fe73578ff725feea70ba4 \ > --control-plane --certificate-key 9f74fd2c73a16a79fb9f458cd5874a860564070fd93c3912d910ba2b9c11a2b1 ~~~ ## 5.10 集群工作节点加入集群 > 因容器镜像下载较慢，可能会导致报错，主要错误为没有准备好cni（集群网络插件），如有网络，请耐心等待即可。 ~~~powershell [root@worker01 ~]# kubeadm join 192.168.10.100:16443 --token 7t2weq.bjbawausm0jaxury \ > --discovery-token-ca-cert-hash sha256:085fc221ad8b5baffdaa567768a10d21eca2fc1f939fe73578ff725feea70ba4 ~~~ ~~~powershell [root@worker02 ~]# kubeadm join 192.168.10.100:16443 --token 7t2weq.bjbawausm0jaxury \ > --discovery-token-ca-cert-hash sha256:085fc221ad8b5baffdaa567768a10d21eca2fc1f939fe73578ff725feea70ba4 ~~~ ## 5.11 验证集群可用性 ~~~powershell 查看所有的节点 [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane,master 13m v1.21.0 master02 Ready control-plane,master 2m25s v1.21.0 master03 Ready control-plane,master 87s v1.21.0 worker01 Ready 3m13s v1.21.0 worker02 Ready 2m50s v1.21.0 ~~~ ~~~powershell 查看集群健康情况,理想状态 [root@master01 ~]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} ~~~ ~~~powershell 真实情况 # kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused etcd-0 Healthy {"health":"true"} ~~~ ~~~powershell 查看kubernetes集群pod运行情况 [root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-558bd4d5db-smp62 1/1 Running 0 13m coredns-558bd4d5db-zcmp5 1/1 Running 0 13m etcd-master01 1/1 Running 0 14m etcd-master02 1/1 Running 0 3m10s etcd-master03 1/1 Running 0 115s kube-apiserver-master01 1/1 Running 0 14m kube-apiserver-master02 1/1 Running 0 3m13s kube-apiserver-master03 1/1 Running 0 116s kube-controller-manager-master01 1/1 Running 1 13m kube-controller-manager-master02 1/1 Running 0 3m13s kube-controller-manager-master03 1/1 Running 0 116s kube-proxy-629zl 1/1 Running 0 2m17s kube-proxy-85qn8 1/1 Running 0 3m15s kube-proxy-fhqzt 1/1 Running 0 13m kube-proxy-jdxbd 1/1 Running 0 3m40s kube-proxy-ks97x 1/1 Running 0 4m3s kube-scheduler-master01 1/1 Running 1 13m kube-scheduler-master02 1/1 Running 0 3m13s kube-scheduler-master03 1/1 Running 0 115s ~~~ ~~~powershell 再次查看calico-system命名空间中pod运行情况。 [root@master01 ~]# kubectl get pod -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-666bb9949-4z77k 1/1 Running 0 10m calico-node-b5wjv 1/1 Running 0 10m calico-node-d427l 1/1 Running 0 4m45s calico-node-jkq7f 1/1 Running 0 2m59s calico-node-wtjnm 1/1 Running 0 4m22s calico-node-xxh2p 1/1 Running 0 3m57s calico-typha-7cd9d6445b-5zcg5 1/1 Running 0 2m54s calico-typha-7cd9d6445b-b5d4j 1/1 Running 0 10m calico-typha-7cd9d6445b-z44kp 1/1 Running 1 4m17s ~~~ ~~~powershell 在master节点上操作，查看网络节点是否添加 [root@master01 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME master01 master02 master03 worker01 worker02 ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_hight_bin1.md ================================================ # Kubernetes高可用集群二进制部署（Runtime Docker） Kubernetes（简称为：k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。 它支持一系列容器工具, 包括Docker、Containerd等。 # 一、集群环境准备 ## 1.1 主机规划 | 主机IP地址 | 主机名 | 主机配置 | 主机角色 | 软件列表 | | -------------- | ----------- | -------- | ----------- | ------------------------------------------------------------ | | 192.168.10.12 | k8s-master1 | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce | | 192.168.10.13 | k8s-master2 | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce | | 192.168.10.14 | k8s-master3 | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce | | 192.168.10.15 | k8s-worker1 | 2C4G | worker | kubelet、kube-proxy、docker-ce | | 192.168.10.10 | ha1 | 1C2G | LB | haproxy、keepalived | | 192.168.10.11 | ha2 | 1C2G | LB | haproxy、keepalived | | 192.168.10.100 | / | / | VIP(虚拟IP) | | ## 1.2 软件版本 | 软件名称 | 版本 | 备注 | | ---------- | ---------------- | --------- | | CentOS7 | kernel版本：5.16 | | | kubernetes | v1.21.10 | | | etcd | v3.5.2 | 最新版本 | | calico | v3.19.4 | | | coredns | v1.8.4 | | | docker-ce | 20.10.13 | YUM源默认 | | haproxy | 5.18 | YUM源默认 | | keepalived | 3.5 | YUM源默认 | ## 1.3 网络分配 | 网络名称 | 网段 | 备注 | | ----------- | --------------- | ---- | | Node网络 | 192.168.10.0/24 | | | Service网络 | 10.96.0.0/16 | | | Pod网络 | 10.244.0.0/16 | | # 二、集群部署 ## 2.1主机准备 ### 2.1.1 主机名设置 ~~~powershell hostnamectl set-hostname xxx ~~~ ~~~powershell 关于主机名参见1.1小节主机规划表 ~~~ ### 2.1.2 主机与IP地址解析 ~~~powershell cat >> /etc/hosts << EOF 192.168.10.10 ha1 192.168.10.11 ha2 192.168.10.12 k8s-master1 192.168.10.13 k8s-master2 192.168.10.14 k8s-master3 192.168.10.15 k8s-worker1 EOF ~~~ ### 2.1.3 主机安全设置 #### 2.1.3.1 关闭防火墙 ~~~powershell systemctl stop firewalld systemctl disable firewalld firewall-cmd --state ~~~ #### 2.1.3.2 关闭selinux ~~~powershell setenforce 0 sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config sestatus ~~~ ### 2.1.4 交换分区设置 ~~~powershell swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab echo "vm.swappiness=0" >> /etc/sysctl.conf sysctl -p ~~~ ### 2.1.5 主机系统时间同步 ~~~powershell 安装软件 yum -y install ntpdate 制定时间同步计划任务 crontab -e 0 */1 * * * ntpdate time1.aliyun.com ~~~ ### 2.1.6 主机系统优化 > limit优化 ~~~powershell ulimit -SHn 65535 ~~~ ~~~powershell cat <> /etc/security/limits.conf * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF ~~~ ### 2.1.7 ipvs管理工具安装及模块加载 > 为集群节点安装，负载均衡节点不用安装 ~~~powershell yum -y install ipvsadm ipset sysstat conntrack libseccomp ~~~ ~~~powershell 所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可： modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack ~~~ ~~~powershell 创建 /etc/modules-load.d/ipvs.conf 并加入以下内容： cat >/etc/modules-load.d/ipvs.conf < 在所有节点中安装,需要重新操作系统更换内核。 ~~~powershell [root@localhost ~]# yum -y install perl ~~~ ~~~powershell [root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell [root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell [root@localhost ~]# yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell [root@localhost ~]# grub2-set-default 0 ~~~ ~~~powershell [root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ### 2.1.9 Linux内核优化 ~~~powershell cat < /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 131072 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF ~~~ ~~~powershell sysctl --system ~~~ ~~~powershell 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载 reboot -h now ~~~ ~~~powershell 重启后查看结果： lsmod | grep --color=auto -e ip_vs -e nf_conntrack ~~~ ### 2.1.10 其它工具安装(选装) ~~~powershell yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y ~~~ ## 2.2 负载均衡器准备 ### 2.2.1 安装haproxy与keepalived ~~~powershell yum -y install haproxy keepalived ~~~ ### 2.2.2 HAProxy配置 ~~~powershell cat >/etc/haproxy/haproxy.cfg<<"EOF" global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:6443 bind 127.0.0.1:6443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server k8s-master1 192.168.10.12:6443 check server k8s-master2 192.168.10.13:6443 check server k8s-master3 192.168.10.14:6443 check EOF ~~~ ### 2.2.3 KeepAlived > 主从配置不一致，需要注意。 ~~~powershell ha1: cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface ens33 mcast_src_ip 192.168.10.10 virtual_router_id 51 priority 100 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.10.100 } track_script { chk_apiserver } } EOF ~~~ ~~~powershell ha2: cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip 192.168.10.11 virtual_router_id 51 priority 99 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.10.100 } track_script { chk_apiserver } } EOF ~~~ ### 2.2.4 健康检查脚本 > ha1及ha2均要配置 ~~~powershell cat > /etc/keepalived/check_apiserver.sh <<"EOF" #!/bin/bash err=0 for k in $(seq 1 3) do check_code=$(pgrep haproxy) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EOF ~~~ ~~~powershell chmod +x /etc/keepalived/check_apiserver.sh ~~~ ### 2.2.5 启动服务并验证 ~~~powershell systemctl daemon-reload systemctl enable --now haproxy systemctl enable --now keepalived ~~~ ~~~powershell ip address show ~~~ ## 2.3 配置免密登录 > 在k8s-master1上操作 ~~~powershell ssh-keygen ~~~ ~~~powershell ssh-copy-id root@k8s-master1 ssh-copy-id root@k8s-master2 ssh-copy-id root@k8s-master3 ssh-copy-id root@k8s-worker1 ~~~ ~~~powershell ssh root@k8s-master1 ~~~ ## 2.4 部署ETCD集群 > 在k8s-master1上操作。 ### 2.4.1 创建工作目录 ~~~powershell mkdir -p /data/k8s-work ~~~ ### 2.4.2 获取cfssl工具 ~~~powershell cd /data/k8s-work wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 ~~~ ~~~powershell 说明： cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有： - cfssl，是CFSSL的命令行工具 - cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。 ~~~ ~~~powershell chmod +x cfssl* ~~~ ~~~powershell mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo ~~~ ~~~powershell # cfssl version Version: 1.2.0 Revision: dev Runtime: go1.6 ~~~ ### 2.4.3 创建CA证书 #### 2.4.3.1 配置ca证书请求文件 ~~~powershell cat > ca-csr.json <<"EOF" { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ], "ca": { "expiry": "87600h" } } EOF ~~~ #### 2.4.3.2 创建ca证书 ~~~powershell cfssl gencert -initca ca-csr.json | cfssljson -bare ca ~~~ #### 2.4.3.3 配置ca证书策略 ~~~powershell cat > ca-config.json <<"EOF" { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF ~~~ ~~~powershell server auth 表示client可以对使用该ca对server提供的证书进行验证 client auth 表示server可以使用该ca对client提供的证书进行验证 ~~~ ### 2.4.4 创建etcd证书 #### 2.4.4.1 配置etcd请求文件 ~~~powershell cat > etcd-csr.json <<"EOF" { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14" ], "key": { "algo": "rsa", "size": 2048 }, "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" }] } EOF ~~~ #### 2.4.4.2 生成etcd证书 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd ~~~ ~~~powershell # ls 输出 ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd.csr etcd-csr.json etcd-key.pem etcd.pem ~~~ ### 2.4.5 部署etcd集群 #### 2.4.5.1 下载etcd软件包 ![image-20220319090935574](../../img/kubernetes/kubernetes_hight_bin/image-20220319090935574.png) ![image-20220319091008943](../../img/kubernetes/kubernetes_hight_bin/image-20220319091008943.png) ![image-20220319091037753](../../img/kubernetes/kubernetes_hight_bin/image-20220319091037753.png) ~~~powershell wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz ~~~ #### 2.4.5.2 安装etcd软件 ~~~powershell tar -xvf etcd-v3.5.2-linux-amd64.tar.gz cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/ ~~~ #### 2.4.5.3 分发etcd软件 ~~~powershell scp etcd-v3.5.2-linux-amd64/etcd* k8s-master2:/usr/local/bin/ scp etcd-v3.5.2-linux-amd64/etcd* k8s-master3:/usr/local/bin/ ~~~ #### 2.4.5.4 创建配置文件 ~~~powershell mkdir /etc/etcd ~~~ ~~~powershell cat > /etc/etcd/etcd.conf <<"EOF" #[Member] ETCD_NAME="etcd1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.12:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.12:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.12:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.12:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.12:2380,etcd2=https://192.168.10.13:2380,etcd3=https://192.168.10.14:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF ~~~ ~~~powershell 说明： ETCD_NAME：节点名称，集群中唯一 ETCD_DATA_DIR：数据目录 ETCD_LISTEN_PEER_URLS：集群通信监听地址 ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址 ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址 ETCD_INITIAL_CLUSTER：集群节点地址 ETCD_INITIAL_CLUSTER_TOKEN：集群Token ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群 ~~~ #### 2.4.5.5 创建服务配置文件 ~~~powershell mkdir -p /etc/etcd/ssl mkdir -p /var/lib/etcd/default.etcd ~~~ ~~~powershell cd /data/k8s-work cp ca*.pem /etc/etcd/ssl cp etcd*.pem /etc/etcd/ssl ~~~ ~~~powershell cat > /etc/systemd/system/etcd.service <<"EOF" [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=-/etc/etcd/etcd.conf WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/local/bin/etcd \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-cert-file=/etc/etcd/ssl/etcd.pem \ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-client-cert-auth \ --client-cert-auth Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.4.5.6 同步etcd配置到集群其它master节点 ~~~powershell 创建目录 mkdir -p /etc/etcd mkdir -p /etc/etcd/ssl mkdir -p /var/lib/etcd/default.etcd ~~~ ~~~powershell 服务配置文件,需要修改etcd节点名称及IP地址 for i in k8s-master2 k8s-master3 \ do \ scp /etc/etcd/etcd.conf $i:/etc/etcd/ \ done ~~~ ~~~powershell k8s-master2: cat /etc/etcd/etcd.conf #[Member] ETCD_NAME="etcd2" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.13:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.13:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.13:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.13:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.12:2380,etcd2=https://192.168.10.13:2380,etcd3=https://192.168.10.14:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ~~~ ~~~powershell k8s-master3: cat /etc/etcd/etcd.conf #[Member] ETCD_NAME="etcd3" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.14:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.14:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.14:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.14:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.12:2380,etcd2=https://192.168.10.13:2380,etcd3=https://192.168.10.14:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ~~~ ~~~powershell 证书文件 for i in k8s-master2 k8s-master3 \ do \ scp /etc/etcd/ssl/* $i:/etc/etcd/ssl \ done ~~~ ~~~powershell 服务启动配置文件 for i in k8s-master2 k8s-master3 \ do \ scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/ \ done ~~~ #### 2.4.5.7 启动etcd集群 ~~~powershell systemctl daemon-reload systemctl enable --now etcd.service systemctl status etcd ~~~ #### 2.4.5.8 验证集群状态 ~~~powershell ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 endpoint health ~~~ ~~~powershell +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://192.168.10.14:2379 | true | 10.393062ms | | | https://192.168.10.12:2379 | true | 15.70437ms | | | https://192.168.10.13:2379 | true | 15.871684ms | | +----------------------------+--------+-------------+-------+ ~~~ ## 2.5 Kubernetes集群部署 ### 2.5.1 Kubernetes软件包下载 ~~~powershell wget https://dl.k8s.io/v1.21.10/kubernetes-server-linux-amd64.tar.gz ~~~ ### 2.5.2 Kubernetes软件包安装 ~~~powershell tar -xvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin/ cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/ ~~~ ### 2.5.3 Kubernetes软件分发 ~~~powershell scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master2:/usr/local/bin/ scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master3:/usr/local/bin/ ~~~ ~~~powershell scp kubelet kube-proxy k8s-master1:/usr/local/bin scp kubelet kube-proxy k8s-master2:/usr/local/bin scp kubelet kube-proxy k8s-master3:/usr/local/bin scp kubelet kube-proxy k8s-worker1:/usr/local/bin ~~~ ### 2.5.4 在集群节点上创建目录 > 所有节点 ~~~powershell mkdir -p /etc/kubernetes/ mkdir -p /etc/kubernetes/ssl mkdir -p /var/log/kubernetes ~~~ ### 2.5.5 部署api-server #### 2.5.5.1 创建apiserver证书请求文件 ~~~powershell cat > kube-apiserver-csr.json << "EOF" { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14", "192.168.10.15", "192.168.10.16", "192.168.10.17", "192.168.10.18", "192.168.10.19", "192.168.10.20", "192.168.10.100", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ] } EOF ~~~ ~~~powershell 说明： 如果 hosts 字段不为空则需要指定授权使用该证书的 IP（含VIP） 或域名列表。由于该证书被 集群使用，需要将节点的IP都填上，为了方便后期扩容可以多写几个预留的IP。 同时还需要填写 service 网络的首个IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.96.0.1)。 ~~~ #### 2.5.5.2 生成apiserver证书及token文件 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver ~~~ ~~~powershell cat > token.csv << EOF $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF ~~~ ~~~powershell 说明： 创建TLS机制所需TOKEN TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。 ~~~ #### 2.5.5.3 创建apiserver服务配置文件 ~~~powershell cat > /etc/kubernetes/kube-apiserver.conf << "EOF" KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.10.12 \ --secure-port=6443 \ --advertise-address=192.168.10.12 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.96.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-32767 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=api \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" EOF ~~~ #### 2.5.5.4 创建apiserver服务管理配置文件 ~~~powershell cat > /etc/systemd/system/kube-apiserver.service << "EOF" [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=etcd.service Wants=etcd.service [Service] EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.5.5.5 同步文件到集群master节点 ~~~powershell cp ca*.pem /etc/kubernetes/ssl/ ~~~ ~~~powershell cp kube-apiserver*.pem /etc/kubernetes/ssl/ ~~~ ~~~powershell cp token.csv /etc/kubernetes/ ~~~ ~~~powershell scp /etc/kubernetes/token.csv k8s-master2:/etc/kubernetes scp /etc/kubernetes/token.csv k8s-master3:/etc/kubernetes ~~~ ~~~powershell scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master2:/etc/kubernetes/ssl scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master3:/etc/kubernetes/ssl ~~~ ~~~powershell scp /etc/kubernetes/ssl/ca*.pem k8s-master2:/etc/kubernetes/ssl scp /etc/kubernetes/ssl/ca*.pem k8s-master3:/etc/kubernetes/ssl ~~~ ~~~powershell scp /etc/kubernetes/kube-apiserver.conf k8s-master2:/etc/kubernetes/kube-apiserver.conf # cat /etc/kubernetes/kube-apiserver.conf KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.10.13 \ --secure-port=6443 \ --advertise-address=192.168.10.13 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.96.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-32767 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=api \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" ~~~ ~~~powershell cp /etc/kubernetes/kube-apiserver.conf k8s-master3:/etc/kubernetes/kube-apiserver.conf # cat /etc/kubernetes/kube-apiserver.conf KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.10.14 \ --secure-port=6443 \ --advertise-address=192.168.10.14 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.96.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-32767 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=api \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" ~~~ ~~~powershell scp /etc/systemd/system/kube-apiserver.service k8s-master2:/etc/systemd/system/kube-apiserver.service scp /etc/systemd/system/kube-apiserver.service k8s-master3:/etc/systemd/system/kube-apiserver.service ~~~ #### 2.5.5.6 启动apiserver服务 ~~~powershell systemctl daemon-reload systemctl enable --now kube-apiserver systemctl status kube-apiserver # 测试 curl --insecure https://192.168.10.12:6443/ curl --insecure https://192.168.10.13:6443/ curl --insecure https://192.168.10.14:6443/ curl --insecure https://192.168.10.100:6443/ ~~~ ### 2.5.6 部署kubectl #### 2.5.6.1 创建kubectl证书请求文件 ~~~powershell cat > admin-csr.json << "EOF" { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:masters", "OU": "system" } ] } EOF ~~~ ~~~powershell 说明： 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权； kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限； O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限； 注： 这个admin 证书，是将来生成管理员用的kubeconfig 配置文件用的，现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制， kubernetes 将证书中的CN 字段 作为User， O 字段作为 Group； "O": "system:masters", 必须是system:masters，否则后面kubectl create clusterrolebinding报错。 ~~~ #### 2.5.6.2 生成证书文件 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin ~~~ #### 2.5.6.3 复制文件到指定目录 ~~~powershell cp admin*.pem /etc/kubernetes/ssl/ ~~~ #### 2.5.6.4 生成kubeconfig配置文件 kube.config 为 kubectl 的配置文件，包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书 ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube.config kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config kubectl config use-context kubernetes --kubeconfig=kube.config ~~~ #### 2.5.6.5 准备kubectl配置文件并进行角色绑定 ~~~powershell mkdir ~/.kube cp kube.config ~/.kube/config kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes --kubeconfig=/root/.kube/config ~~~ #### 2.5.6.6 查看集群状态 ~~~powershell export KUBECONFIG=$HOME/.kube/config ~~~ ~~~powershell kubectl cluster-info kubectl get componentstatuses kubectl get all --all-namespaces ~~~ #### 2.5.6.7 同步kubectl配置文件到集群其它master节点 ~~~powershell k8s-master2: mkdir /root/.kube k8s-master3: mkdir /root/.kube ~~~ ~~~powershell scp /root/.kube/config k8s-master2:/root/.kube/config scp /root/.kube/config k8s-master3:/root/.kube/config ~~~ #### 2.5.6.8 配置kubectl命令补全(可选) ~~~powershell yum install -y bash-completion source /usr/share/bash-completion/bash_completion source <(kubectl completion bash) kubectl completion bash > ~/.kube/completion.bash.inc source '/root/.kube/completion.bash.inc' source $HOME/.bash_profile ~~~ ### 2.5.7 部署kube-controller-manager #### 2.5.7.1 创建kube-controller-manager证书请求文件 ~~~powershell cat > kube-controller-manager-csr.json << "EOF" { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14" ], "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:kube-controller-manager", "OU": "system" } ] } EOF ~~~ ~~~powershell 说明： hosts 列表包含所有 kube-controller-manager 节点 IP； CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限 ~~~ #### 2.5.7.2 创建kube-controller-manager证书文件 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager ~~~ ~~~powershell # ls kube-controller-manager.csr kube-controller-manager-csr.json kube-controller-manager-key.pem kube-controller-manager.pem ~~~ #### 2.5.7.3 创建kube-controller-manager的kube-controller-manager.kubeconfig ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig ~~~ #### 2.5.7.4 创建kube-controller-manager配置文件 ~~~powershell cat > kube-controller-manager.conf << "EOF" KUBE_CONTROLLER_MANAGER_OPTS="--port=10252 \ --secure-port=10257 \ --bind-address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --service-cluster-ip-range=10.96.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --experimental-cluster-signing-duration=87600h \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --leader-elect=true \ --feature-gates=RotateKubeletServerCertificate=true \ --controllers=*,bootstrapsigner,tokencleaner \ --horizontal-pod-autoscaler-use-rest-clients=true \ --horizontal-pod-autoscaler-sync-period=10s \ --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \ --use-service-account-credentials=true \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2" EOF ~~~ #### 2.5.7.5 创建服务启动文件 ~~~powershell cat > kube-controller-manager.service << "EOF" [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.5.7.6 同步文件到集群master节点 ~~~powershell cp kube-controller-manager*.pem /etc/kubernetes/ssl/ cp kube-controller-manager.kubeconfig /etc/kubernetes/ cp kube-controller-manager.conf /etc/kubernetes/ cp kube-controller-manager.service /usr/lib/systemd/system/ ~~~ ~~~powershell scp kube-controller-manager*.pem k8s-master2:/etc/kubernetes/ssl/ scp kube-controller-manager*.pem k8s-master3:/etc/kubernetes/ssl/ scp kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master2:/etc/kubernetes/ scp kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master3:/etc/kubernetes/ scp kube-controller-manager.service k8s-master2:/usr/lib/systemd/system/ scp kube-controller-manager.service k8s-master3:/usr/lib/systemd/system/ ~~~ ~~~powershell #查看证书 openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text ~~~ #### 2.5.7.7 启动服务 ~~~powershell systemctl daemon-reload systemctl enable --now kube-controller-manager systemctl status kube-controller-manager ~~~ ### 2.5.8 部署kube-scheduler #### 2.5.8.1 创建kube-scheduler证书请求文件 ~~~powershell cat > kube-scheduler-csr.json << "EOF" { "CN": "system:kube-scheduler", "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:kube-scheduler", "OU": "system" } ] } EOF ~~~ #### 2.5.8.2 生成kube-scheduler证书 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler ~~~ ~~~powershell # ls kube-scheduler.csr kube-scheduler-csr.json kube-scheduler-key.pem kube-scheduler.pem ~~~ #### 2.5.8.3 创建kube-scheduler的kubeconfig ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig ~~~ #### 2.5.8.4 创建服务配置文件 ~~~powershell cat > kube-scheduler.conf << "EOF" KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \ --leader-elect=true \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2" EOF ~~~ #### 2.5.8.5创建服务启动配置文件 ~~~powershell cat > kube-scheduler.service << "EOF" [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.5.8.6 同步文件至集群master节点 ~~~powershell cp kube-scheduler*.pem /etc/kubernetes/ssl/ cp kube-scheduler.kubeconfig /etc/kubernetes/ cp kube-scheduler.conf /etc/kubernetes/ cp kube-scheduler.service /usr/lib/systemd/system/ ~~~ ~~~powershell scp kube-scheduler*.pem k8s-master2:/etc/kubernetes/ssl/ scp kube-scheduler*.pem k8s-master3:/etc/kubernetes/ssl/ scp kube-scheduler.kubeconfig kube-scheduler.conf k8s-master2:/etc/kubernetes/ scp kube-scheduler.kubeconfig kube-scheduler.conf k8s-master3:/etc/kubernetes/ scp kube-scheduler.service k8s-master2:/usr/lib/systemd/system/ scp kube-scheduler.service k8s-master3:/usr/lib/systemd/system/ ~~~ #### 2.5.8.7 启动服务 ~~~powershell systemctl daemon-reload systemctl enable --now kube-scheduler systemctl status kube-scheduler ~~~ ### 2.5.9 工作节点（worker node）部署 #### 2.5.9.1 docker安装及配置 ~~~powershell wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell yum -y install docker-ce ~~~ ~~~powershell systemctl enable docker systemctl start docker ~~~ ~~~powershell cat < 在k8s-master1上操作 ##### 2.5.9.2.1 创建kubelet-bootstrap.kubeconfig ~~~powershell BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv) kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig ~~~ ~~~powershell kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubelet-bootstrap kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig ~~~ ~~~powershell kubectl describe clusterrolebinding cluster-system-anonymous kubectl describe clusterrolebinding kubelet-bootstrap ~~~ ##### 2.5.9.2.2 创建kubelet配置文件 ~~~powershell cat > kubelet.json << "EOF" { "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1", "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/ssl/ca.pem" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "address": "192.168.10.12", "port": 10250, "readOnlyPort": 10255, "cgroupDriver": "systemd", "hairpinMode": "promiscuous-bridge", "serializeImagePulls": false, "clusterDomain": "cluster.local.", "clusterDNS": ["10.96.0.2"] } EOF ~~~ ##### 2.5.9.2.3 创建kubelet配置文件 ~~~powershell cat > kubelet.service << "EOF" [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/usr/local/bin/kubelet \ --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \ --cert-dir=/etc/kubernetes/ssl \ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \ --config=/etc/kubernetes/kubelet.json \ --network-plugin=cni \ --rotate-certificates \ --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ~~~ ##### 2.5.9.2.4 同步文件到集群节点 ~~~powershell cp kubelet-bootstrap.kubeconfig /etc/kubernetes/ cp kubelet.json /etc/kubernetes/ cp kubelet.service /usr/lib/systemd/system/ ~~~ ~~~powershell for i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet-bootstrap.kubeconfig kubelet.json $i:/etc/kubernetes/;done for i in k8s-master2 k8s-master3 k8s-worker1;do scp ca.pem $i:/etc/kubernetes/ssl/;done for i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet.service $i:/usr/lib/systemd/system/;done ~~~ ~~~powershell 说明： kubelet.json中address需要修改为当前主机IP地址。 ~~~ ##### 2.5.9.2.5 创建目录及启动服务 ~~~powershell mkdir -p /var/lib/kubelet mkdir -p /var/log/kubernetes ~~~ ~~~powershell systemctl daemon-reload systemctl enable --now kubelet systemctl status kubelet ~~~ ~~~powershell # kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 NotReady 2m55s v1.21.10 k8s-master2 NotReady 45s v1.21.10 k8s-master3 NotReady 39s v1.21.10 k8s-worker1 NotReady 5m1s v1.21.10 ~~~ ~~~powershell # kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-b949p 7m55s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-c9hs4 3m34s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-r8vhp 5m50s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-zb4sr 3m40s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued ~~~ ~~~powershell 说明： 确认kubelet服务启动成功后，接着到master上Approve一下bootstrap请求。 ~~~ #### 2.5.9.3 部署kube-proxy ##### 2.5.9.3.1 创建kube-proxy证书请求文件 ~~~powershell cat > kube-proxy-csr.json << "EOF" { "CN": "system:kube-proxy", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ] } EOF ~~~ ##### 2.5.9.3.2 生成证书 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy ~~~ ~~~powershell # ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem ~~~ ##### 2.5.9.3.3 创建kubeconfig文件 ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig ~~~ ##### 2.5.9.3.4 创建服务配置文件 ~~~powershell cat > kube-proxy.yaml << "EOF" apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 192.168.10.12 clientConnection: kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig clusterCIDR: 10.244.0.0/16 healthzBindAddress: 192.168.10.12:10256 kind: KubeProxyConfiguration metricsBindAddress: 192.168.10.12:10249 mode: "ipvs" EOF ~~~ ##### 2.5.9.3.5 创建服务启动管理文件 ~~~powershell cat > kube-proxy.service << "EOF" [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/usr/local/bin/kube-proxy \ --config=/etc/kubernetes/kube-proxy.yaml \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ~~~ ##### 2.5.9.3.6 同步文件到集群工作节点主机 ~~~powershell cp kube-proxy*.pem /etc/kubernetes/ssl/ cp kube-proxy.kubeconfig kube-proxy.yaml /etc/kubernetes/ cp kube-proxy.service /usr/lib/systemd/system/ ~~~ ~~~powershell for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.kubeconfig kube-proxy.yaml $i:/etc/kubernetes/;done for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.service $i:/usr/lib/systemd/system/;done ~~~ ~~~powershell 说明： 修改kube-proxy.yaml中IP地址为当前主机IP. ~~~ ##### 2.5.9.3.7 服务启动 ~~~powershell mkdir -p /var/lib/kube-proxy systemctl daemon-reload systemctl enable --now kube-proxy systemctl status kube-proxy ~~~ ### 2.5.10 网络组件部署 Calico #### 2.5.10.1 下载 ~~~powershell wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml ~~~ #### 2.5.10.2 修改文件 ~~~powershell 3683 - name: CALICO_IPV4POOL_CIDR 3684 value: "10.244.0.0/16" ~~~ #### 2.5.10.3 应用文件 ~~~powershell kubectl apply -f calico.yaml ~~~ #### 2.5.10.4 验证应用结果 ~~~powershell # kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-7cc8dd57d9-tf2m5 1/1 Running 0 72s kube-system calico-node-llw5w 1/1 Running 0 72s kube-system calico-node-mhh6g 1/1 Running 0 72s kube-system calico-node-twj99 1/1 Running 0 72s kube-system calico-node-zh6xl 1/1 Running 0 72s ~~~ ~~~powershell # kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready 55m v1.21.10 k8s-master2 Ready 53m v1.21.10 k8s-master3 Ready 53m v1.21.10 k8s-worker1 Ready 57m v1.21.10 ~~~ ### 2.5.10 部署CoreDNS ~~~powershell cat > coredns.yaml << "EOF" apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/name: "CoreDNS" spec: # replicas: not specified here: # 1. Default is 1. # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname containers: - name: coredns image: coredns/coredns:1.8.4 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.96.0.2 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP EOF ~~~ ~~~powershell kubectl apply -f coredns.yaml ~~~ ~~~powershell # kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-7cc8dd57d9-tf2m5 1/1 Running 0 4m7s kube-system calico-node-llw5w 1/1 Running 0 4m7s kube-system calico-node-mhh6g 1/1 Running 0 4m7s kube-system calico-node-twj99 1/1 Running 0 4m7s kube-system calico-node-zh6xl 1/1 Running 0 4m7s kube-system coredns-675db8b7cc-ncnf6 1/1 Running 0 26s ~~~ ### 2.5.11 部署应用验证 ~~~powershell cat > nginx.yaml << "EOF" --- apiVersion: v1 kind: ReplicationController metadata: name: nginx-web spec: replicas: 2 selector: name: nginx template: metadata: labels: name: nginx spec: containers: - name: nginx image: nginx:1.19.6 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-service-nodeport spec: ports: - port: 80 targetPort: 80 nodePort: 30001 protocol: TCP type: NodePort selector: name: nginx EOF ~~~ ~~~powershell kubectl apply -f nginx.yaml ~~~ ~~~powershell # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-web-qzvw4 1/1 Running 0 58s 10.244.194.65 k8s-worker1 nginx-web-spw5t 1/1 Running 0 58s 10.244.224.1 k8s-master2 ~~~ ~~~powershell # kubectl get all NAME READY STATUS RESTARTS AGE pod/nginx-web-qzvw4 1/1 Running 0 2m2s pod/nginx-web-spw5t 1/1 Running 0 2m2s NAME DESIRED CURRENT READY AGE replicationcontroller/nginx-web 2 2 2 2m2s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 3h37m service/nginx-service-nodeport NodePort 10.96.165.114 80:30001/TCP 2m2s ~~~ ![image-20220319014727332](../../img/kubernetes/kubernetes_hight_bin/image-20220319014727332.png) ## 2.6 Kubernetes集群节点管理 > 本小节主要介绍worker节点管理 ### 2.6.1 主机准备 #### 2.6.1.1 主机名设置 ~~~powershell hostnamectl set-hostname k8s-worker4 ~~~ #### 2.6.1.2 主机与IP地址解析 > 集群中已有节点也需要添加新节点的解析。 ~~~powershell cat >> /etc/hosts << EOF 192.168.10.10 ha1 192.168.10.11 ha2 192.168.10.12 k8s-master1 192.168.10.13 k8s-master2 192.168.10.14 k8s-master3 192.168.10.15 k8s-worker1 192.168.10.19 k8s-worker4 EOF ~~~ #### 2.6.1.3 主机安全设置 ##### 2.6.1.3.1 关闭防火墙 ~~~powershell systemctl stop firewalld systemctl disable firewalld firewall-cmd --state ~~~ ##### 2.6.1.3.2 关闭selinux ~~~powershell setenforce 0 sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config sestatus ~~~ #### 2.6.1.4 交换分区设置 ~~~powershell swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab echo "vm.swappiness=0" >> /etc/sysctl.conf sysctl -p ~~~ #### 2.6.1.5 主机系统时间同步 ~~~powershell 安装软件 yum -y install ntpdate 制定时间同步计划任务 crontab -e 0 */1 * * * ntpdate time1.aliyun.com ~~~ #### 2.6.1.6 主机系统优化 > limit优化 ~~~powershell ulimit -SHn 65535 ~~~ ~~~powershell cat <> /etc/security/limits.conf * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF ~~~ #### 2.6.1.7 ipvs管理工具安装及模块加载 > 为集群节点安装，负载均衡节点不用安装 ~~~powershell yum -y install ipvsadm ipset sysstat conntrack libseccomp ~~~ ~~~powershell 所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可： modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack ~~~ ~~~powershell 创建 /etc/modules-load.d/ipvs.conf 并加入以下内容： cat >/etc/modules-load.d/ipvs.conf < 在所有节点中安装,需要重新操作系统更换内核。 ~~~powershell [root@localhost ~]# yum -y install perl ~~~ ~~~powershell [root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell [root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell [root@localhost ~]# yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell [root@localhost ~]# grub2-set-default 0 ~~~ ~~~powershell [root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ #### 2.6.1.9 Linux内核优化 ~~~powershell cat < /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 131072 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF ~~~ ~~~powershell sysctl --system ~~~ ~~~powershell 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载 reboot -h now ~~~ ~~~powershell 重启后查看结果： lsmod | grep --color=auto -e ip_vs -e nf_conntrack ~~~ #### 2.6.1.10 其它工具安装(选装) ~~~powershell yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y ~~~ ### 2.6.2 配置免密登录 > 在k8s-master1节点操作 ~~~powershell ssh-copy-id root@k8s-worker4 ~~~ ### 2.6.3 Kubernetes软件包获取 #### 2.6.3.1 软件包获取 ~~~powershell [root@k8s-master1 bin]# pwd /data/k8s-work/kubernetes/server/bin ~~~ ~~~powershell scp kubelet kube-proxy k8s-worker4:/usr/local/bin ~~~ ~~~powershell [root@k8s-worker4 ~]# ls /usr/local/bin/kube* /usr/local/bin/kubelet /usr/local/bin/kube-proxy ~~~ #### 2.6.3.2 docker-ce安装及配置 ~~~powershell wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell yum -y install docker-ce ~~~ ~~~powershell systemctl enable docker systemctl start docker ~~~ ~~~powershell cat < 41h v1.21.10 k8s-master2 Ready 41h v1.21.10 k8s-master3 Ready 41h v1.21.10 k8s-worker1 Ready 41h v1.21.10 k8s-worker4 Ready 55s v1.21.10 ~~~ #### 2.6.3.4 部署kube-proxy ~~~powershell scp kube-proxy.kubeconfig kube-proxy.yaml k8s-worker4:/etc/kubernetes/ scp kube-proxy.service k8s-worker4:/usr/lib/systemd/system/ ~~~ ~~~powershell [root@k8s-worker2 ~]# vim /etc/kubernetes/kube-proxy.yaml [root@k8s-worker2 ~]# cat /etc/kubernetes/kube-proxy.yaml apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 192.168.10.19 clientConnection: kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig clusterCIDR: 10.244.0.0/16 healthzBindAddress: 192.168.10.19:10256 kind: KubeProxyConfiguration metricsBindAddress: 192.168.10.19:10249 mode: "ipvs" ~~~ ~~~powershell [root@k8s-worker4 ~]# mkdir -p /var/lib/kube-proxy ~~~ ~~~powershell [root@k8s-worker4 ~]# systemctl daemon-reload [root@k8s-worker4 ~]# systemctl enable --now kube-proxy [root@k8s-worker4 ~]# systemctl status kube-proxy ~~~ ### 2.6.4 验证 ~~~powershell # kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-node-dkndr 1/1 Running 0 19s 192.168.10.19 k8s-worker4 ~~~ ~~~powershell kubectl label nodes k8s-worker3 deploy.type=nginxapp ~~~ ~~~powershell cat > nginx.yaml << EOF --- apiVersion: v1 kind: ReplicationController metadata: name: nginx-web spec: replicas: 1 selector: name: nginx template: metadata: labels: name: nginx spec: nodeSelector: deploy.type: nginxapp containers: - name: nginx image: nginx:1.19.6 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-service-nodeport spec: ports: - port: 80 targetPort: 80 nodePort: 30001 protocol: TCP type: NodePort selector: name: nginx EOF ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_hight_bin2.md ================================================ # Kubernetes高可用集群二进制部署（Runtime Containerd） Kubernetes（简称为：k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。 它支持一系列容器工具, 包括Docker、Containerd等。 # 一、集群环境准备 ## 1.1 主机规划 | 主机IP地址 | 主机名 | 主机配置 | 主机角色 | 软件列表 | | -------------- | ----------- | -------- | ----------- | ------------------------------------------------------------ | | 192.168.10.12 | k8s-master1 | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc | | 192.168.10.13 | k8s-master2 | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc | | 192.168.10.14 | k8s-master3 | 2C4G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc | | 192.168.10.15 | k8s-worker1 | 2C4G | worker | kubelet、kube-proxy、Containerd、runc | | 192.168.10.10 | ha1 | 1C2G | LB | haproxy、keepalived | | 192.168.10.11 | ha2 | 1C2G | LB | haproxy、keepalived | | 192.168.10.100 | / | / | VIP(虚拟IP) | | ## 1.2 软件版本 | 软件名称 | 版本 | 备注 | | ---------- | ---------------- | --------- | | CentOS7 | kernel版本：5.17 | | | kubernetes | v1.21.10 | | | etcd | v3.5.2 | 最新版本 | | calico | v3.19.4 | | | coredns | v1.8.4 | | | containerd | 1.6.1 | | | runc | 1.1.0 | | | haproxy | 5.18 | YUM源默认 | | keepalived | 3.5 | YUM源默认 | ## 1.3 网络分配 | 网络名称 | 网段 | 备注 | | ----------- | --------------- | ---- | | Node网络 | 192.168.10.0/24 | | | Service网络 | 10.96.0.0/16 | | | Pod网络 | 10.244.0.0/16 | | # 二、集群部署 ## 2.1主机准备 ### 2.1.1 主机名设置 ~~~powershell hostnamectl set-hostname xxx ~~~ ~~~powershell 关于主机名参见1.1小节主机规划表 ~~~ ### 2.1.2 主机与IP地址解析 ~~~powershell cat >> /etc/hosts << EOF 192.168.10.10 ha1 192.168.10.11 ha2 192.168.10.12 k8s-master1 192.168.10.13 k8s-master2 192.168.10.14 k8s-master3 192.168.10.15 k8s-worker1 EOF ~~~ ### 2.1.3 主机安全设置 #### 2.1.3.1 关闭防火墙 ~~~powershell systemctl stop firewalld systmctl disable firewalld firewall-cmd --state ~~~ #### 2.1.3.2 关闭selinux ~~~powershell setenforce 0 sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config sestatus ~~~ ### 2.1.4 交换分区设置 ~~~powershell swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab echo "vm.swappiness=0" >> /etc/sysctl.conf sysctl -p ~~~ ### 2.1.5 主机系统时间同步 ~~~powershell 安装软件 yum -y install ntpdate 制定时间同步计划任务 crontab -e 0 */1 * * * ntpdate time1.aliyun.com ~~~ ### 2.1.6 主机系统优化 > limit优化 ~~~powershell ulimit -SHn 65535 ~~~ ~~~powershell cat <> /etc/security/limits.conf * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF ~~~ ### 2.1.7 ipvs管理工具安装及模块加载 > 为集群节点安装，负载均衡节点不用安装 ~~~powershell yum -y install ipvsadm ipset sysstat conntrack libseccomp ~~~ ~~~powershell 所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可： modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack ~~~ ~~~powershell 创建 /etc/modules-load.d/ipvs.conf 并加入以下内容： cat >/etc/modules-load.d/ipvs.conf < /etc/modules-load.d/containerd.conf << EOF overlay br_netfilter EOF ~~~ ~~~powershell 设置为开机启动 systemctl enable --now systemd-modules-load.service ~~~ ### 2.1.9 Linux内核升级 > 在所有节点中安装,需要重新操作系统更换内核。 ~~~powershell [root@localhost ~]# yum -y install perl ~~~ ~~~powershell [root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell [root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell [root@localhost ~]# yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell [root@localhost ~]# grub2-set-default 0 ~~~ ~~~powershell [root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ### 2.1.10 Linux内核优化 ~~~powershell cat < /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 131072 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF ~~~ ~~~powershell sysctl --system ~~~ ~~~powershell 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载 reboot -h now ~~~ ~~~powershell 重启后查看ipvs模块加载情况： lsmod | grep --color=auto -e ip_vs -e nf_conntrack ~~~ ~~~powershell 重启后查看containerd相关模块加载情况： lsmod | egrep 'br_netfilter | overlay' ~~~ ### 2.1.11 其它工具安装(选装) ~~~powershell yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y ~~~ ## 2.2 负载均衡器准备 ### 2.2.1 安装haproxy与keepalived ~~~powershell yum -y install haproxy keepalived ~~~ ### 2.2.2 HAProxy配置 ~~~powershell cat >/etc/haproxy/haproxy.cfg<<"EOF" global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:6443 bind 127.0.0.1:6443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server k8s-master1 192.168.10.12:6443 check server k8s-master2 192.168.10.13:6443 check server k8s-master3 192.168.10.14:6443 check EOF ~~~ ### 2.2.3 KeepAlived > 主从配置不一致，需要注意。 ~~~powershell ha1: cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface ens33 mcast_src_ip 192.168.10.10 virtual_router_id 51 priority 100 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.10.100 } track_script { chk_apiserver } } EOF ~~~ ~~~powershell ha2: cat >/etc/keepalived/keepalived.conf<<"EOF" ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip 192.168.10.11 virtual_router_id 51 priority 99 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.10.100 } track_script { chk_apiserver } } EOF ~~~ ### 2.2.4 健康检查脚本 > ha1及ha2均要配置 ~~~powershell cat > /etc/keepalived/check_apiserver.sh <<"EOF" #!/bin/bash err=0 for k in $(seq 1 3) do check_code=$(pgrep haproxy) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EOF ~~~ ~~~powershell chmod +x /etc/keepalived/check_apiserver.sh ~~~ ### 2.2.5 启动服务并验证 ~~~powershell systemctl daemon-reload systemctl enable --now haproxy systemctl enable --now keepalived ~~~ ~~~powershell ip address show ~~~ ## 2.3 配置免密登录 > 在k8s-master1上操作 ~~~powershell ssh-keygen ~~~ ~~~powershell ssh-copy-id root@k8s-master1 ssh-copy-id root@k8s-master2 ssh-copy-id root@k8s-master3 ssh-copy-id root@k8s-worker1 ~~~ ~~~powershell ssh root@k8s-master1 ~~~ ## 2.4 部署ETCD集群 > 在k8s-master1上操作。 ### 2.4.1 创建工作目录 ~~~powershell mkdir -p /data/k8s-work ~~~ ### 2.4.2 获取cfssl工具 ~~~powershell cd /data/k8s-work wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 ~~~ ~~~powershell 说明： cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有： - cfssl，是CFSSL的命令行工具 - cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。 ~~~ ~~~powershell chmod +x cfssl* ~~~ ~~~powershell mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo ~~~ ~~~powershell # cfssl version Version: 1.2.0 Revision: dev Runtime: go1.6 ~~~ ### 2.4.3 创建CA证书 #### 2.4.3.1 配置ca证书请求文件 ~~~powershell cat > ca-csr.json <<"EOF" { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ], "ca": { "expiry": "87600h" } } EOF ~~~ #### 2.4.3.2 创建ca证书 ~~~powershell cfssl gencert -initca ca-csr.json | cfssljson -bare ca ~~~ #### 2.4.3.3 配置ca证书策略 ~~~powershell cfssl print-defaults config > ca-config.json ~~~ ~~~powershell cat > ca-config.json <<"EOF" { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF ~~~ ~~~powershell server auth 表示client可以对使用该ca对server提供的证书进行验证 client auth 表示server可以使用该ca对client提供的证书进行验证 ~~~ ### 2.4.4 创建etcd证书 #### 2.4.4.1 配置etcd请求文件 ~~~powershell cat > etcd-csr.json <<"EOF" { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14" ], "key": { "algo": "rsa", "size": 2048 }, "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" }] } EOF ~~~ #### 2.4.4.2 生成etcd证书 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd ~~~ ~~~powershell # ls 输出 ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd.csr etcd-csr.json etcd-key.pem etcd.pem ~~~ ### 2.4.5 部署etcd集群 #### 2.4.5.1 下载etcd软件包 ![image-20220319090935574](../../img/kubernetes/kubernetes_hight_bin/image-20220319090935574.png) ![image-20220319091008943](../../img/kubernetes/kubernetes_hight_bin/image-20220319091008943.png) ![image-20220319091037753](../../img/kubernetes/kubernetes_hight_bin/image-20220319091037753.png) ~~~powershell wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz ~~~ #### 2.4.5.2 安装etcd软件 ~~~powershell tar -xvf etcd-v3.5.2-linux-amd64.tar.gz cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/ ~~~ #### 2.4.5.3 分发etcd软件 ~~~powershell scp etcd-v3.5.2-linux-amd64/etcd* k8s-master2:/usr/local/bin/ scp etcd-v3.5.2-linux-amd64/etcd* k8s-master3:/usr/local/bin/ ~~~ #### 2.4.5.4 创建配置文件 ~~~powershell mkdir /etc/etcd ~~~ ~~~powershell cat > /etc/etcd/etcd.conf <<"EOF" #[Member] ETCD_NAME="etcd1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.12:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.12:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.12:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.12:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.12:2380,etcd2=https://192.168.10.13:2380,etcd3=https://192.168.10.14:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF ~~~ ~~~powershell 说明： ETCD_NAME：节点名称，集群中唯一 ETCD_DATA_DIR：数据目录 ETCD_LISTEN_PEER_URLS：集群通信监听地址 ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址 ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址 ETCD_INITIAL_CLUSTER：集群节点地址 ETCD_INITIAL_CLUSTER_TOKEN：集群Token ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群 ~~~ #### 2.4.5.5 创建服务配置文件 ~~~powershell mkdir -p /etc/etcd/ssl mkdir -p /var/lib/etcd/default.etcd ~~~ ~~~powershell cd /data/k8s-work cp ca*.pem /etc/etcd/ssl cp etcd*.pem /etc/etcd/ssl ~~~ ~~~powershell cat > /etc/systemd/system/etcd.service <<"EOF" [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=-/etc/etcd/etcd.conf WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/local/bin/etcd \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-cert-file=/etc/etcd/ssl/etcd.pem \ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-client-cert-auth \ --client-cert-auth Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.4.5.6 同步etcd配置到集群其它master节点 ~~~powershell 创建目录 mkdir -p /etc/etcd mkdir -p /etc/etcd/ssl mkdir -p /var/lib/etcd/default.etcd ~~~ ~~~powershell 服务配置文件,需要修改etcd节点名称及IP地址 for i in k8s-master2 k8s-master3 \ do \ scp /etc/etcd/etcd.conf $i:/etc/etcd/ \ done ~~~ ~~~powershell k8s-master2: cat /etc/etcd/etcd.conf #[Member] ETCD_NAME="etcd2" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.13:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.13:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.13:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.13:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.12:2380,etcd2=https://192.168.10.13:2380,etcd3=https://192.168.10.14:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ~~~ ~~~powershell k8s-master3: cat /etc/etcd/etcd.conf #[Member] ETCD_NAME="etcd3" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.10.14:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.10.14:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.14:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.14:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.12:2380,etcd2=https://192.168.10.13:2380,etcd3=https://192.168.10.14:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ~~~ ~~~powershell 证书文件 for i in k8s-master2 k8s-master3 \ do \ scp /etc/etcd/ssl/* $i:/etc/etcd/ssl \ done ~~~ ~~~powershell 服务启动配置文件 for i in k8s-master2 k8s-master3 \ do \ scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/ \ done ~~~ #### 2.4.5.7 启动etcd集群 ~~~powershell systemctl daemon-reload systemctl enable --now etcd.service systemctl status etcd ~~~ #### 2.4.5.8 验证集群状态 ~~~powershell ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 endpoint health ~~~ ~~~powershell +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://192.168.10.14:2379 | true | 10.393062ms | | | https://192.168.10.12:2379 | true | 15.70437ms | | | https://192.168.10.13:2379 | true | 15.871684ms | | +----------------------------+--------+-------------+-------+ ~~~ ~~~powershell 检查ETCD数据库性能 ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 check perf ~~~ ~~~powershell 59 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom ! 98.33% PASS: Throughput is 151 writes/s PASS: Slowest request took 0.066478s PASS: Stddev is 0.002354s PASS ~~~ ~~~powershell ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 member list ~~~ ~~~powershell +------------------+---------+-------+----------------------------+----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+-------+----------------------------+----------------------------+------------+ | 9b449b0ff1d4c375 | started | etcd1 | https://192.168.10.12:2380 | https://192.168.10.12:2379 | false | | d1fbb74bc6a61e5c | started | etcd2 | https://192.168.10.13:2380 | https://192.168.10.13:2379 | false | | f60b205fb02fe23c | started | etcd3 | https://192.168.10.14:2380 | https://192.168.10.14:2379 | false | +------------------+---------+-------+----------------------------+----------------------------+------------+ ~~~ ~~~powershell ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 endpoint status ~~~ ~~~powershell +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://192.168.10.12:2379 | 9b449b0ff1d4c375 | 3.5.2 | 24 MB | true | false | 2 | 403774 | 403774 | | | https://192.168.10.13:2379 | d1fbb74bc6a61e5c | 3.5.2 | 24 MB | false | false | 2 | 403774 | 403774 | | | https://192.168.10.14:2379 | f60b205fb02fe23c | 3.5.2 | 24 MB | false | false | 2 | 403774 | 403774 | | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ ~~~ ## 2.5 Kubernetes集群部署 ### 2.5.1 Kubernetes软件包下载 ~~~powershell wget https://dl.k8s.io/v1.21.10/kubernetes-server-linux-amd64.tar.gz ~~~ ### 2.5.2 Kubernetes软件包安装 ~~~powershell tar -xvf kubernetes-server-linux-amd64.tar.gz cd kubernetes/server/bin/ cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/ ~~~ ### 2.5.3 Kubernetes软件分发 ~~~powershell scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master2:/usr/local/bin/ scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master3:/usr/local/bin/ ~~~ ~~~powershell scp kubelet kube-proxy k8s-master1:/usr/local/bin scp kubelet kube-proxy k8s-master2:/usr/local/bin scp kubelet kube-proxy k8s-master3:/usr/local/bin scp kubelet kube-proxy k8s-worker1:/usr/local/bin ~~~ ### 2.5.4 在集群节点上创建目录 > 所有节点 ~~~powershell mkdir -p /etc/kubernetes/ mkdir -p /etc/kubernetes/ssl mkdir -p /var/log/kubernetes ~~~ ### 2.5.5 部署api-server #### 2.5.5.1 创建apiserver证书请求文件 ~~~powershell cat > kube-apiserver-csr.json << "EOF" { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14", "192.168.10.15", "192.168.10.16", "192.168.10.17", "192.168.10.18", "192.168.10.19", "192.168.10.20", "192.168.10.100", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ] } EOF ~~~ ~~~powershell 说明： 如果 hosts 字段不为空则需要指定授权使用该证书的 IP（含VIP） 或域名列表。由于该证书被 集群使用，需要将节点的IP都填上，为了方便后期扩容可以多写几个预留的IP。 同时还需要填写 service 网络的首个IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.96.0.1)。 ~~~ #### 2.5.5.2 生成apiserver证书及token文件 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver ~~~ ~~~powershell cat > token.csv << EOF $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF ~~~ ~~~powershell 说明： 创建TLS机制所需TOKEN TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。 ~~~ #### 2.5.5.3 创建apiserver服务配置文件 ~~~powershell cat > /etc/kubernetes/kube-apiserver.conf << "EOF" KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.10.12 \ --secure-port=6443 \ --advertise-address=192.168.10.12 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.96.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-32767 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=api \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" EOF ~~~ #### 2.5.5.4 创建apiserver服务管理配置文件 ~~~powershell cat > /etc/systemd/system/kube-apiserver.service << "EOF" [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=etcd.service Wants=etcd.service [Service] EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.5.5.5 同步文件到集群master节点 ~~~powershell cp ca*.pem /etc/kubernetes/ssl/ ~~~ ~~~powershell cp kube-apiserver*.pem /etc/kubernetes/ssl/ ~~~ ~~~powershell cp token.csv /etc/kubernetes/ ~~~ ~~~powershell scp /etc/kubernetes/token.csv k8s-master2:/etc/kubernetes scp /etc/kubernetes/token.csv k8s-master3:/etc/kubernetes ~~~ ~~~powershell scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master2:/etc/kubernetes/ssl scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master3:/etc/kubernetes/ssl ~~~ ~~~powershell scp /etc/kubernetes/ssl/ca*.pem k8s-master2:/etc/kubernetes/ssl scp /etc/kubernetes/ssl/ca*.pem k8s-master3:/etc/kubernetes/ssl ~~~ ~~~powershell scp /etc/kubernetes/kube-apiserver.conf k8s-master2:/etc/kubernetes/kube-apiserver.conf # cat /etc/kubernetes/kube-apiserver.conf KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.10.13 \ --secure-port=6443 \ --advertise-address=192.168.10.13 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.96.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-32767 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=api \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" ~~~ ~~~powershell cp /etc/kubernetes/kube-apiserver.conf k8s-master3:/etc/kubernetes/kube-apiserver.conf # cat /etc/kubernetes/kube-apiserver.conf KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.10.14 \ --secure-port=6443 \ --advertise-address=192.168.10.14 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.96.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-32767 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=api \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" ~~~ ~~~powershell scp /etc/systemd/system/kube-apiserver.service k8s-master2:/etc/systemd/system/kube-apiserver.service scp /etc/systemd/system/kube-apiserver.service k8s-master3:/etc/systemd/system/kube-apiserver.service ~~~ #### 2.5.5.6 启动apiserver服务 ~~~powershell systemctl daemon-reload systemctl enable --now kube-apiserver systemctl status kube-apiserver # 测试 curl --insecure https://192.168.10.12:6443/ curl --insecure https://192.168.10.13:6443/ curl --insecure https://192.168.10.14:6443/ curl --insecure https://192.168.10.100:6443/ ~~~ ### 2.5.6 部署kubectl #### 2.5.6.1 创建kubectl证书请求文件 ~~~powershell cat > admin-csr.json << "EOF" { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:masters", "OU": "system" } ] } EOF ~~~ ~~~powershell 说明： 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权； kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限； O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限； 注： 这个admin 证书，是将来生成管理员用的kubeconfig 配置文件用的，现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制， kubernetes 将证书中的CN 字段 作为User， O 字段作为 Group； "O": "system:masters", 必须是system:masters，否则后面kubectl create clusterrolebinding报错。 ~~~ #### 2.5.6.2 生成证书文件 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin ~~~ #### 2.5.6.3 复制文件到指定目录 ~~~powershell cp admin*.pem /etc/kubernetes/ssl/ ~~~ #### 2.5.6.4 生成kubeconfig配置文件 kube.config 为 kubectl 的配置文件，包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书 ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube.config kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config kubectl config use-context kubernetes --kubeconfig=kube.config ~~~ #### 2.5.6.5 准备kubectl配置文件并进行角色绑定 ~~~powershell mkdir ~/.kube cp kube.config ~/.kube/config kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes --kubeconfig=/root/.kube/config ~~~ #### 2.5.6.6 查看集群状态 ~~~powershell export KUBECONFIG=$HOME/.kube/config ~~~ ~~~powershell 查看集群信息 kubectl cluster-info 查看集群组件状态 kubectl get componentstatuses 查看命名空间中资源对象 kubectl get all --all-namespaces ~~~ #### 2.5.6.7 同步kubectl配置文件到集群其它master节点 ~~~powershell k8s-master2: mkdir /root/.kube k8s-master3: mkdir /root/.kube ~~~ ~~~powershell scp /root/.kube/config k8s-master2:/root/.kube/config scp /root/.kube/config k8s-master3:/root/.kube/config ~~~ #### 2.5.6.8 配置kubectl命令补全(可选) ~~~powershell yum install -y bash-completion source /usr/share/bash-completion/bash_completion source <(kubectl completion bash) kubectl completion bash > ~/.kube/completion.bash.inc source '/root/.kube/completion.bash.inc' source $HOME/.bash_profile ~~~ ### 2.5.7 部署kube-controller-manager #### 2.5.7.1 创建kube-controller-manager证书请求文件 ~~~powershell cat > kube-controller-manager-csr.json << "EOF" { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14" ], "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:kube-controller-manager", "OU": "system" } ] } EOF ~~~ ~~~powershell 说明： hosts 列表包含所有 kube-controller-manager 节点 IP； CN 为 system:kube-controller-manager; O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限 ~~~ #### 2.5.7.2 创建kube-controller-manager证书文件 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager ~~~ ~~~powershell # ls kube-controller-manager.csr kube-controller-manager-csr.json kube-controller-manager-key.pem kube-controller-manager.pem ~~~ #### 2.5.7.3 创建kube-controller-manager的kube-controller-manager.kubeconfig ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig ~~~ #### 2.5.7.4 创建kube-controller-manager配置文件 ~~~powershell cat > kube-controller-manager.conf << "EOF" KUBE_CONTROLLER_MANAGER_OPTS="--port=10252 \ --secure-port=10257 \ --bind-address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --service-cluster-ip-range=10.96.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --experimental-cluster-signing-duration=87600h \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --leader-elect=true \ --feature-gates=RotateKubeletServerCertificate=true \ --controllers=*,bootstrapsigner,tokencleaner \ --horizontal-pod-autoscaler-use-rest-clients=true \ --horizontal-pod-autoscaler-sync-period=10s \ --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \ --use-service-account-credentials=true \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2" EOF ~~~ #### 2.5.7.5 创建服务启动文件 ~~~powershell cat > kube-controller-manager.service << "EOF" [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.5.7.6 同步文件到集群master节点 ~~~powershell cp kube-controller-manager*.pem /etc/kubernetes/ssl/ cp kube-controller-manager.kubeconfig /etc/kubernetes/ cp kube-controller-manager.conf /etc/kubernetes/ cp kube-controller-manager.service /usr/lib/systemd/system/ ~~~ ~~~powershell scp kube-controller-manager*.pem k8s-master2:/etc/kubernetes/ssl/ scp kube-controller-manager*.pem k8s-master3:/etc/kubernetes/ssl/ scp kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master2:/etc/kubernetes/ scp kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master3:/etc/kubernetes/ scp kube-controller-manager.service k8s-master2:/usr/lib/systemd/system/ scp kube-controller-manager.service k8s-master3:/usr/lib/systemd/system/ ~~~ ~~~powershell #查看证书 openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text ~~~ #### 2.5.7.7 启动服务 ~~~powershell systemctl daemon-reload systemctl enable --now kube-controller-manager systemctl status kube-controller-manager ~~~ ~~~powershell kubectl get componentstatuses ~~~ ### 2.5.8 部署kube-scheduler #### 2.5.8.1 创建kube-scheduler证书请求文件 ~~~powershell cat > kube-scheduler-csr.json << "EOF" { "CN": "system:kube-scheduler", "hosts": [ "127.0.0.1", "192.168.10.12", "192.168.10.13", "192.168.10.14" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:kube-scheduler", "OU": "system" } ] } EOF ~~~ #### 2.5.8.2 生成kube-scheduler证书 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler ~~~ ~~~powershell # ls kube-scheduler.csr kube-scheduler-csr.json kube-scheduler-key.pem kube-scheduler.pem ~~~ #### 2.5.8.3 创建kube-scheduler的kubeconfig ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig ~~~ #### 2.5.8.4 创建服务配置文件 ~~~powershell cat > kube-scheduler.conf << "EOF" KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \ --leader-elect=true \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2" EOF ~~~ #### 2.5.8.5创建服务启动配置文件 ~~~powershell cat > kube-scheduler.service << "EOF" [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ~~~ #### 2.5.8.6 同步文件至集群master节点 ~~~powershell cp kube-scheduler*.pem /etc/kubernetes/ssl/ cp kube-scheduler.kubeconfig /etc/kubernetes/ cp kube-scheduler.conf /etc/kubernetes/ cp kube-scheduler.service /usr/lib/systemd/system/ ~~~ ~~~powershell scp kube-scheduler*.pem k8s-master2:/etc/kubernetes/ssl/ scp kube-scheduler*.pem k8s-master3:/etc/kubernetes/ssl/ scp kube-scheduler.kubeconfig kube-scheduler.conf k8s-master2:/etc/kubernetes/ scp kube-scheduler.kubeconfig kube-scheduler.conf k8s-master3:/etc/kubernetes/ scp kube-scheduler.service k8s-master2:/usr/lib/systemd/system/ scp kube-scheduler.service k8s-master3:/usr/lib/systemd/system/ ~~~ #### 2.5.8.7 启动服务 ~~~powershell systemctl daemon-reload systemctl enable --now kube-scheduler systemctl status kube-scheduler ~~~ ### 2.5.9 工作节点（worker node）部署 #### 2.5.9.1 Containerd安装及配置 ##### 2.5.9.1.1 获取软件包 ![image-20220321233643232](../../img/kubernetes/kubernetes_hight_bin/image-20220321233643232.png) ![image-20220321233718744](../../img/kubernetes/kubernetes_hight_bin/image-20220321233718744.png) ![image-20220321233751488](../../img/kubernetes/kubernetes_hight_bin/image-20220321233751488.png) ![image-20220321233903747](../../img/kubernetes/kubernetes_hight_bin/image-20220321233903747.png) ~~~powershell wget https://github.com/containerd/containerd/releases/download/v1.6.1/cri-containerd-cni-1.6.1-linux-amd64.tar.gz ~~~ ##### 2.5.9.1.2 安装containerd ~~~powershell tar -xf cri-containerd-cni-1.6.1-linux-amd64.tar.gz -C / ~~~ ~~~powershell 默认解压后会有如下目录： etc opt usr 会把对应的目解压到/下对应目录中，这样就省去复制文件步骤。 ~~~ ##### 2.5.9.1.3 生成配置文件并修改 ~~~powershell mkdir /etc/containerd ~~~ ~~~powershell containerd config default >/etc/containerd/config.toml ~~~ ~~~powershell # ls /etc/containerd/ config.toml ~~~ ~~~powershell 下面的配置文件中已修改，可不执行，仅修改默认时执行。 sed -i 's@systemd_cgroup = false@systemd_cgroup = true@' /etc/containerd/config.toml ~~~ ~~~powershell 下面的配置文件中已修改，可不执行，仅修改默认时执行。 sed -i 's@k8s.gcr.io/pause:3.6@registry.aliyuncs.com/google_containers/pause:3.6@' /etc/containerd/config.toml ~~~ ~~~powershell # cat >/etc/containerd/config.toml< 由于上述软件包中包含的runc对系统依赖过多，所以建议单独下载安装。 > > 默认runc执行时提示：runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond ![image-20220322003721996](../../img/kubernetes/kubernetes_hight_bin/image-20220322003721996.png) ![image-20220322003759560](../../img/kubernetes/kubernetes_hight_bin/image-20220322003759560.png) ![image-20220322003823362](../../img/kubernetes/kubernetes_hight_bin/image-20220322003823362.png) ![image-20220322003942106](../../img/kubernetes/kubernetes_hight_bin/image-20220322003942106.png) ~~~powershell wget https://github.com/opencontainers/runc/releases/download/v1.1.0/runc.amd64 ~~~ ~~~powershell chmod +x runc.amd64 ~~~ ~~~powershell 替换掉原软件包中的runc mv runc.amd64 /usr/local/sbin/runc ~~~ ~~~powershell # runc -v runc version 1.1.0 commit: v1.1.0-0-g067aaf85 spec: 1.0.2-dev go: go1.17.6 libseccomp: 2.5.3 ~~~ ~~~powershell systemctl enable containerd systemctl start containerd ~~~ #### 2.5.9.2 部署kubelet > 在k8s-master1上操作 ##### 2.5.9.2.1 创建kubelet-bootstrap.kubeconfig ~~~powershell BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv) kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig ~~~ ~~~powershell kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubelet-bootstrap kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig ~~~ ~~~powershell kubectl describe clusterrolebinding cluster-system-anonymous kubectl describe clusterrolebinding kubelet-bootstrap ~~~ ##### 2.5.9.2.2 创建kubelet配置文件 ~~~powershell cat > kubelet.json << "EOF" { "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1", "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/ssl/ca.pem" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "address": "192.168.10.12", "port": 10250, "readOnlyPort": 10255, "cgroupDriver": "systemd", "hairpinMode": "promiscuous-bridge", "serializeImagePulls": false, "clusterDomain": "cluster.local.", "clusterDNS": ["10.96.0.2"] } EOF ~~~ ##### 2.5.9.2.3 创建kubelet服务启动管理文件 ~~~powershell cat > kubelet.service << "EOF" [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=containerd.service Requires=containerd.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/usr/local/bin/kubelet \ --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \ --cert-dir=/etc/kubernetes/ssl \ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \ --config=/etc/kubernetes/kubelet.json \ --cni-bin-dir=/opt/cni/bin \ --cni-conf-dir=/etc/cni/net.d \ --container-runtime=remote \ --container-runtime-endpoint=unix:///run/containerd/containerd.sock \ --network-plugin=cni \ --rotate-certificates \ --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.6 \ --root-dir=/etc/cni/net.d \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF ~~~ ##### 2.5.9.2.4 同步文件到集群节点 ~~~powershell cp kubelet-bootstrap.kubeconfig /etc/kubernetes/ cp kubelet.json /etc/kubernetes/ cp kubelet.service /usr/lib/systemd/system/ ~~~ ~~~powershell for i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet-bootstrap.kubeconfig kubelet.json $i:/etc/kubernetes/;done for i in k8s-master2 k8s-master3 k8s-worker1;do scp ca.pem $i:/etc/kubernetes/ssl/;done for i in k8s-master2 k8s-master3 k8s-worker1;do scp kubelet.service $i:/usr/lib/systemd/system/;done ~~~ ~~~powershell 说明： kubelet.json中address需要修改为当前主机IP地址。 ~~~ ##### 2.5.9.2.5 创建目录及启动服务 ~~~powershell mkdir -p /var/lib/kubelet mkdir -p /var/log/kubernetes ~~~ ~~~powershell systemctl daemon-reload systemctl enable --now kubelet systemctl status kubelet ~~~ ~~~powershell # kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 NotReady 2m55s v1.21.10 k8s-master2 NotReady 45s v1.21.10 k8s-master3 NotReady 39s v1.21.10 k8s-worker1 NotReady 5m1s v1.21.10 ~~~ ~~~powershell # kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-b949p 7m55s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-c9hs4 3m34s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-r8vhp 5m50s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-zb4sr 3m40s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued ~~~ ~~~powershell 说明： 确认kubelet服务启动成功后，接着到master上Approve一下bootstrap请求。 ~~~ #### 2.5.9.3 部署kube-proxy ##### 2.5.9.3.1 创建kube-proxy证书请求文件 ~~~powershell cat > kube-proxy-csr.json << "EOF" { "CN": "system:kube-proxy", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubemsb", "OU": "CN" } ] } EOF ~~~ ##### 2.5.9.3.2 生成证书 ~~~powershell cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy ~~~ ~~~powershell # ls kube-proxy* kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem ~~~ ##### 2.5.9.3.3 创建kubeconfig文件 ~~~powershell kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig ~~~ ##### 2.5.9.3.4 创建服务配置文件 ~~~powershell cat > kube-proxy.yaml << "EOF" apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 192.168.10.12 clientConnection: kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig clusterCIDR: 10.244.0.0/16 healthzBindAddress: 192.168.10.12:10256 kind: KubeProxyConfiguration metricsBindAddress: 192.168.10.12:10249 mode: "ipvs" EOF ~~~ ##### 2.5.9.3.5 创建服务启动管理文件 ~~~powershell cat > kube-proxy.service << "EOF" [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/usr/local/bin/kube-proxy \ --config=/etc/kubernetes/kube-proxy.yaml \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ~~~ ##### 2.5.9.3.6 同步文件到集群工作节点主机 ~~~powershell cp kube-proxy*.pem /etc/kubernetes/ssl/ cp kube-proxy.kubeconfig kube-proxy.yaml /etc/kubernetes/ cp kube-proxy.service /usr/lib/systemd/system/ ~~~ ~~~powershell for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.kubeconfig kube-proxy.yaml $i:/etc/kubernetes/;done for i in k8s-master2 k8s-master3 k8s-worker1;do scp kube-proxy.service $i:/usr/lib/systemd/system/;done ~~~ ~~~powershell 说明： 修改kube-proxy.yaml中IP地址为当前主机IP. ~~~ ##### 2.5.9.3.7 服务启动 ~~~powershell mkdir -p /var/lib/kube-proxy ~~~ ~~~powershell systemctl daemon-reload systemctl enable --now kube-proxy systemctl status kube-proxy ~~~ ### 2.5.10 网络组件部署 Calico #### 2.5.10.1 下载 ~~~powershell wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml ~~~ #### 2.5.10.2 修改文件 ~~~powershell 3683 - name: CALICO_IPV4POOL_CIDR 3684 value: "10.244.0.0/16" ~~~ #### 2.5.10.3 应用文件 ~~~powershell kubectl apply -f calico.yaml ~~~ #### 2.5.10.4 验证应用结果 ~~~powershell # kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-7cc8dd57d9-tf2m5 1/1 Running 0 72s kube-system calico-node-llw5w 1/1 Running 0 72s kube-system calico-node-mhh6g 1/1 Running 0 72s kube-system calico-node-twj99 1/1 Running 0 72s kube-system calico-node-zh6xl 1/1 Running 0 72s ~~~ ~~~powershell # kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready 55m v1.21.10 k8s-master2 Ready 53m v1.21.10 k8s-master3 Ready 53m v1.21.10 k8s-worker1 Ready 57m v1.21.10 ~~~ ### 2.5.10 部署CoreDNS ~~~powershell cat > coredns.yaml << "EOF" apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/name: "CoreDNS" spec: # replicas: not specified here: # 1. Default is 1. # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname containers: - name: coredns image: coredns/coredns:1.8.4 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.96.0.2 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP EOF ~~~ ~~~powershell kubectl apply -f coredns.yaml ~~~ ~~~powershell # kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-7cc8dd57d9-tf2m5 1/1 Running 0 4m7s kube-system calico-node-llw5w 1/1 Running 0 4m7s kube-system calico-node-mhh6g 1/1 Running 0 4m7s kube-system calico-node-twj99 1/1 Running 0 4m7s kube-system calico-node-zh6xl 1/1 Running 0 4m7s kube-system coredns-675db8b7cc-ncnf6 1/1 Running 0 26s ~~~ ### 2.5.11 部署应用验证 ~~~powershell cat > nginx.yaml << "EOF" --- apiVersion: v1 kind: ReplicationController metadata: name: nginx-web spec: replicas: 2 selector: name: nginx template: metadata: labels: name: nginx spec: containers: - name: nginx image: nginx:1.19.6 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-service-nodeport spec: ports: - port: 80 targetPort: 80 nodePort: 30001 protocol: TCP type: NodePort selector: name: nginx EOF ~~~ ~~~powershell kubectl apply -f nginx.yaml ~~~ ~~~powershell # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-web-qzvw4 1/1 Running 0 58s 10.244.194.65 k8s-worker1 nginx-web-spw5t 1/1 Running 0 58s 10.244.224.1 k8s-master2 ~~~ ~~~powershell # kubectl get all NAME READY STATUS RESTARTS AGE pod/nginx-web-qzvw4 1/1 Running 0 2m2s pod/nginx-web-spw5t 1/1 Running 0 2m2s NAME DESIRED CURRENT READY AGE replicationcontroller/nginx-web 2 2 2 2m2s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 3h37m service/nginx-service-nodeport NodePort 10.96.165.114 80:30001/TCP 2m2s ~~~ ![image-20220319014727332](../../img/kubernetes/kubernetes_hight_bin/image-20220319014727332.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_hybridnet.md ================================================ # k8s集群 underlay 网络方案 hybridnet # 零、容器网络方案介绍 ## 0.1 overlay 网络方案 基于VXLAN、 NVGRE等封装技术实现overlay叠加网络： 1、叠加网络/覆盖网络， 在物理网络的基础之上叠加实现新的虚拟网络， 即可使网络的中的容器可以相互通信。 2、优点是对物理网络的兼容性比较好， 可以实现pod的夸宿主机子网通信。 3、calico与flannel等网络插件都支持overlay网络。 4、缺点是有额外的封装与解封性能开销。 5、目前私有云使用比较多。 ## 0.2 underlay网络方案 Underlay网络就是传统IT基础设施网络， 由交换机和路由器等设备组成， 借助以太网协议、 路由协议和VLAN协议等驱动， 它还是Overlay网络的底层网络， 为Overlay网络提供数据通信服务。 容器网络中的Underlay网络是指借助驱动程序将宿主机的底层网络接口直接暴露给容器使用的一种网络构建技术，较为常见的解决方案有MAC VLAN、 IP VLAN和直接路由等。 Underlay依赖于物理网络进行跨主机通信。 1、Mac Vlan模式： MAC VLAN： 支持在同一个以太网接口上虚拟出多个网络接口(子接口)， 每个虚拟接口都拥有唯一的MAC地址并可配置网卡子接口IP，基于Docker宿主机物理网卡的不同子接口实现多个虚拟vlan，一个子接口就是一个虚拟vlan，容器通过宿主机的路由功能和外网保持通信。 2、IP VLAN模式： IP VLAN类似于MAC VLAN， 它同样创建新的虚拟网络接口并为每个接口分配唯一的IP地址， 不同之处在于， 每个虚拟接口将共享使用物理接口的MAC地址。 ![image-20230404151314708](../../img/kubernetes/kubernetes_hybridnet/image-20230404151314708.png) ## 0.3 K8S Pod通信简介 ![image-20230404151837989](../../img/kubernetes/kubernetes_hybridnet/image-20230404151837989.png) - Overlay网络： Flannel Vxlan、 Calico BGP、 Calico Vxlan 将pod 地址信息封装在宿主机地址信息以内， 实现跨主机且可跨node子网的通信报文。 - 直接路由： Flannel Host-gw、 Flannel VXLAN Directrouting、 Calico Directrouting 基于主机路由， 实现报文从源主机到目的主机的直接转发， 不需要进行报文的叠加封装， 性能比overlay更好。 - Underlay: 不需要为pod启用单独的虚拟机网络， 而是直接使用宿主机物理网络， pod甚至可以在k8s环境之外的节点直接访问(与node节点的网络被打通)， 相当于把pod当桥接模式的虚拟机使用， 比较方便k8s环境以外的访问访问k8s环境中的pod中的服务， 而且由于主机使用的宿主机网络， 其性能最好。 # 一、K8S集群部署 > 基于kubeadm部署K8S 1.26集群，pod network cidr为10.244.0.0/16，service为默认10.96.0.0/12 ~~~powershell 方案1： pod可以选择overlay或者underlay， SVC使用overlay， 如果是underlay需要配置SVC使用宿主机的子网比如以下场景是overlay网络、 后期会用于overlay场景的pod， service会用于overlay的svc场景。 kubeadm init --apiserver-advertise-address=192.168.10.160 \ --apiserver-bind-port=6443 \ --kubernetes-version=v1.26.3 \ --pod-network-cidr=10.244.0.0/16 \ --service-cidr=10.96.0.0/12 \ --service-dns-domain=cluster.local \ --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \ --cri-socket unix:///var/run/cri-dockerd.sock --image-repository=“”为空则使用默认的google容器镜像仓库 --cri-socket 可以使用unix:///var/run/containerd/containerd.sock或unix:///var/run/cri-dockerd.sock ~~~ ~~~powershell 方案2： pod可以选择overlay或者underlay， SVC使用underlay初始化 --pod-network-cidr=10.244.0.0/16会用于后期overlay的场景， underlay的网络CIDR后期单独指定， overlay会与underlay并存 --service-cidr=192.168.200.0/24用于后期的underlay svc， 通过SVC可以直接访问pod。 kubeadm init --apiserver-advertise-address=192.168.10.160 \ --apiserver-bind-port=6443 \ --kubernetes-version=v1.26.3 \ --pod-network-cidr=10.244.0.0/16 \ --service-cidr=192.168.200.0/24 \ --service-dns-domain=cluster.local \ --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \ --cri-socket unix:///var/run/cri-dockerd.sock --service-cidr= 与已存在的网段不能冲突 --image-repository=“”为空则使用默认的google容器镜像仓库 --cri-socket 可以使用unix:///var/run/containerd/containerd.sock或unix:///var/run/cri-dockerd.sock ~~~ # 二、Helm部署 > github链接：https://github.com/helm/helm/releases ![image-20230404120724930](../../img/kubernetes/kubernetes_hybridnet/image-20230404120724930.png) ~~~powershell # wget https://get.helm.sh/helm-v3.11.2-linux-amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# ls helm-v3.11.2-linux-amd64.tar.gz [root@k8s-master01 ~]# tar xf helm-v3.11.2-linux-amd64.tar.gz [root@k8s-master01 ~]# ls helm-v3.11.2-linux-amd64.tar.gz linux-amd64 [root@k8s-master01 ~]# ls linux-amd64/ helm LICENSE README.md ~~~ ~~~powershell [root@k8s-master01 ~]# mv linux-amd64/helm /usr/local/bin/helm ~~~ ~~~powershell [root@k8s-master01 ~]# helm version version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"} ~~~ # 三、部署hybridnet ~~~powershell [root@k8s-master01 ~]# helm repo add hybridnet https://alibaba.github.io/hybridnet/ "hybridnet" has been added to your repositories ~~~ ~~~powershell [root@k8s-master01 ~]# helm repo list NAME URL hybridnet https://alibaba.github.io/hybridnet/ ~~~ ~~~powershell [root@k8s-master01 ~]# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "hybridnet" chart repository Update Complete. ⎈Happy Helming!⎈ ~~~ ~~~powershell [root@k8s-master01 ~]# helm install hybridnet hybridnet/hybridnet -n kube-system --set init.cidr=10.244.0.0/16 ~~~ ~~~powershell 输出： W0404 12:14:47.796075 111159 warnings.go:70] spec.template.spec.nodeSelector[beta.kubernetes.io/os]: deprecated since v1.14; use "kubernetes.io/os" instead W0404 12:14:47.796100 111159 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead NAME: hybridnet LAST DEPLOYED: Tue Apr 4 12:14:47 2023 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None ~~~ ~~~powershell [root@k8s-master01 ~]# helm list -n kube-system NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION hybridnet kube-system 1 2023-04-04 12:14:47.158751157 +0800 CST deployed hybridnet-0.6.0 0.8.0 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-typha-c856d6bfd-7qnkk 1/1 Running 0 107s calico-typha-c856d6bfd-l8nhw 1/1 Running 0 107s calico-typha-c856d6bfd-slppp 1/1 Running 0 109s coredns-787d4945fb-lfk42 1/1 Running 0 15h coredns-787d4945fb-t8x2t 1/1 Running 0 15h etcd-k8s-master01 1/1 Running 0 15h hybridnet-daemon-ls2rh 1/2 Running 1 (19s ago) 114s hybridnet-daemon-lxcb6 1/2 Running 1 (70s ago) 114s hybridnet-daemon-xp7t4 1/2 Running 1 (30s ago) 114s hybridnet-manager-55f5488b46-2x5qw 0/1 Pending 0 114s hybridnet-manager-55f5488b46-ddpjw 0/1 Pending 0 109s hybridnet-manager-55f5488b46-tx78h 0/1 Pending 0 109s hybridnet-webhook-55d848f89c-8zrs2 0/1 Pending 0 114s hybridnet-webhook-55d848f89c-9f9rf 0/1 Pending 0 114s hybridnet-webhook-55d848f89c-q9xgn 0/1 Pending 0 114s kube-apiserver-k8s-master01 1/1 Running 0 15h kube-controller-manager-k8s-master01 1/1 Running 0 15h kube-proxy-5v642 1/1 Running 0 15h kube-proxy-vnwhh 1/1 Running 0 15h kube-proxy-zgrj6 1/1 Running 0 15h kube-scheduler-k8s-master01 1/1 Running 0 15h ~~~ ~~~powershell 此时hybridnet-manager、hybridnet-webhook pod Pending，通过describe查看发现集群没有节点打上master标签 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl describe pods hybridnet-manager-55f5488b46-2x5qw -n kube-system Name: hybridnet-manager-55f5488b46-2x5qw Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Service Account: hybridnet Node: Labels: app=hybridnet component=manager pod-template-hash=55f5488b46 Annotations: Status: Pending IP: IPs: Controlled By: ReplicaSet/hybridnet-manager-55f5488b46 Containers: hybridnet-manager: Image: docker.io/hybridnetdev/hybridnet:v0.8.0 Port: 9899/TCP Host Port: 9899/TCP Command: /hybridnet/hybridnet-manager --default-ip-retain=true --feature-gates=MultiCluster=false,VMIPRetain=false --controller-concurrency=Pod=1,IPAM=1,IPInstance=1 --kube-client-qps=300 --kube-client-burst=600 --metrics-port=9899 Environment: DEFAULT_NETWORK_TYPE: Overlay DEFAULT_IP_FAMILY: IPv4 NAMESPACE: kube-system (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ctkkr (ro) Conditions: Type Status PodScheduled False Volumes: kube-api-access-ctkkr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/master= Tolerations: :NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 3m32s (x2 over 3m34s) default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master01 Ready control-plane 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers= k8s-worker01 Ready 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker01,kubernetes.io/os=linux k8s-worker02 Ready 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker02,kubernetes.io/os=linux ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl label node k8s-master01 node-role.kubernetes.io/master= node/k8s-master01 labeled ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane,master 15h v1.26.3 k8s-worker01 Ready 15h v1.26.3 k8s-worker02 Ready 15h v1.26.3 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master01 Ready control-plane,master 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,networking.alibaba.com/overlay-network-attachment=true,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers= k8s-worker01 Ready 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker01,kubernetes.io/os=linux,networking.alibaba.com/overlay-network-attachment=true k8s-worker02 Ready 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker02,kubernetes.io/os=linux,networking.alibaba.com/overlay-network-attachment=true ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-typha-c856d6bfd-7qnkk 1/1 Running 0 9m17s 192.168.10.160 k8s-master01 calico-typha-c856d6bfd-l8nhw 1/1 Running 0 9m17s 192.168.10.161 k8s-worker01 calico-typha-c856d6bfd-slppp 1/1 Running 0 9m19s 192.168.10.162 k8s-worker02 coredns-787d4945fb-lfk42 1/1 Running 0 15h 10.88.0.3 k8s-master01 coredns-787d4945fb-t8x2t 1/1 Running 0 15h 10.88.0.2 k8s-master01 etcd-k8s-master01 1/1 Running 0 15h 192.168.10.160 k8s-master01 hybridnet-daemon-ls2rh 1/2 Running 1 (7m49s ago) 9m24s 192.168.10.161 k8s-worker01 hybridnet-daemon-lxcb6 1/2 Running 1 (8m40s ago) 9m24s 192.168.10.162 k8s-worker02 hybridnet-daemon-xp7t4 1/2 Running 1 (8m ago) 9m24s 192.168.10.160 k8s-master01 hybridnet-manager-55f5488b46-2x5qw 1/1 Running 0 9m24s 192.168.10.160 k8s-master01 hybridnet-manager-55f5488b46-ddpjw 0/1 Pending 0 9m19s hybridnet-manager-55f5488b46-tx78h 0/1 Pending 0 9m19s hybridnet-webhook-55d848f89c-8zrs2 0/1 Pending 0 9m24s hybridnet-webhook-55d848f89c-9f9rf 1/1 Running 0 9m24s 192.168.10.160 k8s-master01 hybridnet-webhook-55d848f89c-q9xgn 0/1 Pending 0 9m24s kube-apiserver-k8s-master01 1/1 Running 0 15h 192.168.10.160 k8s-master01 kube-controller-manager-k8s-master01 1/1 Running 0 15h 192.168.10.160 k8s-master01 kube-proxy-5v642 1/1 Running 0 15h 192.168.10.160 k8s-master01 kube-proxy-vnwhh 1/1 Running 0 15h 192.168.10.161 k8s-worker01 kube-proxy-zgrj6 1/1 Running 0 15h 192.168.10.162 k8s-worker02 kube-scheduler-k8s-master01 1/1 Running 0 15h 192.168.10.160 k8s-master01 ~~~ # 四、创建hybridnet网络 ~~~powershell [root@k8s-master01 ~]# mkdir /root/hybridnet [root@k8s-master01 ~]# cd hybridnet/ [root@k8s-master01 hybridnet]# ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl label node k8s-master01 network=underlay-nethost node/k8s-master01 labeled [root@k8s-master01 hybridnet]# kubectl label node k8s-worker01 network=underlay-nethost node/k8s-worker01 labeled [root@k8s-master01 hybridnet]# kubectl label node k8s-worker02 network=underlay-nethost node/k8s-worker02 labeled ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master01 Ready control-plane,master 16h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,network=underlay-nethost,networking.alibaba.com/overlay-network-attachment=true,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers= k8s-worker01 Ready 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker01,kubernetes.io/os=linux,network=underlay-nethost,networking.alibaba.com/overlay-network-attachment=true k8s-worker02 Ready 15h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker02,kubernetes.io/os=linux,network=underlay-nethost,networking.alibaba.com/overlay-network-attachment=true ~~~ ~~~powershell [root@k8s-master01 hybridnet]# vim 01-create-underlay-network.yaml [root@k8s-master01 hybridnet]# cat 01-create-underlay-network.yaml --- apiVersion: networking.alibaba.com/v1 kind: Network metadata: name: underlay-network1 spec: netID: 0 type: Underlay nodeSelector: network: "underlay-nethost" --- apiVersion: networking.alibaba.com/v1 kind: Subnet metadata: name: underlay-network1 spec: network: underlay-network1 netID: 0 range: version: "4" cidr: "192.168.10.0/24" gateway: "192.168.10.2" start: "192.168.10.10" end: "192.168.10.20" ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create -f 01-create-underlay-network.yaml ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get network NAME NETID TYPE MODE V4TOTAL V4USED V4AVAILABLE LASTALLOCATEDV4SUBNET V6TOTAL V6USED V6AVAILABLE LASTALLOCATEDV6SUBNET init 4 Overlay 65534 2 65532 init 0 0 0 underlay-network1 0 Underlay 11 0 11 underlay-network1 0 0 0 ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get subnet NAME VERSION CIDR START END GATEWAY TOTAL USED AVAILABLE NETID NETWORK init 4 10.244.0.0/16 65534 2 65532 init underlay-network1 4 192.168.10.0/24 192.168.10.10 192.168.10.20 192.168.10.2 11 11 0 underlay-network1 ~~~ # 五、查看节点Labels信息 ~~~powershell [root@k8s-master01 hybridnet]# kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master01 Ready control-plane,master 16h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,network=underlay-nethost,networking.alibaba.com/dualstack-address-quota=empty,networking.alibaba.com/ipv4-address-quota=nonempty,networking.alibaba.com/ipv6-address-quota=empty,networking.alibaba.com/overlay-network-attachment=true,networking.alibaba.com/underlay-network-attachment=true,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers= k8s-worker01 Ready 16h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker01,kubernetes.io/os=linux,network=underlay-nethost,networking.alibaba.com/dualstack-address-quota=empty,networking.alibaba.com/ipv4-address-quota=nonempty,networking.alibaba.com/ipv6-address-quota=empty,networking.alibaba.com/overlay-network-attachment=true,networking.alibaba.com/underlay-network-attachment=true k8s-worker02 Ready 16h v1.26.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker02,kubernetes.io/os=linux,network=underlay-nethost,networking.alibaba.com/dualstack-address-quota=empty,networking.alibaba.com/ipv4-address-quota=nonempty,networking.alibaba.com/ipv6-address-quota=empty,networking.alibaba.com/overlay-network-attachment=true,networking.alibaba.com/underlay-network-attachment=true ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl describe nodes k8s-master01 Name: k8s-master01 Roles: control-plane,master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-master01 kubernetes.io/os=linux network=underlay-nethost networking.alibaba.com/dualstack-address-quota=empty networking.alibaba.com/ipv4-address-quota=nonempty networking.alibaba.com/ipv6-address-quota=empty networking.alibaba.com/overlay-network-attachment=true networking.alibaba.com/underlay-network-attachment=true node-role.kubernetes.io/control-plane= node-role.kubernetes.io/master= node.kubernetes.io/exclude-from-external-load-balancers= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.10.160/24 projectcalico.org/IPv4VXLANTunnelAddr: 10.244.32.128 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 03 Apr 2023 20:26:44 +0800 Taints: node-role.kubernetes.io/control-plane:NoSchedule Unschedulable: false Lease: HolderIdentity: k8s-master01 AcquireTime: RenewTime: Tue, 04 Apr 2023 12:35:31 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Tue, 04 Apr 2023 11:59:24 +0800 Tue, 04 Apr 2023 11:59:24 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Tue, 04 Apr 2023 12:31:18 +0800 Mon, 03 Apr 2023 20:26:39 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 04 Apr 2023 12:31:18 +0800 Mon, 03 Apr 2023 20:26:39 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 04 Apr 2023 12:31:18 +0800 Mon, 03 Apr 2023 20:26:39 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 04 Apr 2023 12:31:18 +0800 Mon, 03 Apr 2023 20:26:47 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.10.160 Hostname: k8s-master01 Capacity: cpu: 4 ephemeral-storage: 51175Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 4026120Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 48294789041 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3923720Ki pods: 110 System Info: Machine ID: f618107e5de3464bbfc77620a718fdd5 System UUID: B55A4D56-8EBB-7F7D-F774-2CAFA717C713 Boot ID: 289d296b-02fd-4135-a427-81143def0ed9 Kernel Version: 3.10.0-1160.76.1.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.7.0 Kubelet Version: v1.26.3 Kube-Proxy Version: v1.26.3 PodCIDR: 10.244.0.0/24 PodCIDRs: 10.244.0.0/24 Non-terminated Pods: (11 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-typha-c856d6bfd-7qnkk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20m kube-system coredns-787d4945fb-lfk42 100m (2%) 0 (0%) 70Mi (1%) 170Mi (4%) 16h kube-system coredns-787d4945fb-t8x2t 100m (2%) 0 (0%) 70Mi (1%) 170Mi (4%) 16h kube-system etcd-k8s-master01 100m (2%) 0 (0%) 100Mi (2%) 0 (0%) 16h kube-system hybridnet-daemon-xp7t4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20m kube-system hybridnet-manager-55f5488b46-2x5qw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20m kube-system hybridnet-webhook-55d848f89c-9f9rf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20m kube-system kube-apiserver-k8s-master01 250m (6%) 0 (0%) 0 (0%) 0 (0%) 16h kube-system kube-controller-manager-k8s-master01 200m (5%) 0 (0%) 0 (0%) 0 (0%) 16h kube-system kube-proxy-5v642 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16h kube-system kube-scheduler-k8s-master01 100m (2%) 0 (0%) 0 (0%) 0 (0%) 16h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 850m (21%) 0 (0%) memory 240Mi (6%) 340Mi (8%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl describe nodes k8s-worker01 Name: k8s-worker01 Roles: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-worker01 kubernetes.io/os=linux network=underlay-nethost networking.alibaba.com/dualstack-address-quota=empty networking.alibaba.com/ipv4-address-quota=nonempty networking.alibaba.com/ipv6-address-quota=empty networking.alibaba.com/overlay-network-attachment=true networking.alibaba.com/underlay-network-attachment=true Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.10.161/24 projectcalico.org/IPv4VXLANTunnelAddr: 10.244.79.64 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 03 Apr 2023 20:28:44 +0800 Taints: Unschedulable: false Lease: HolderIdentity: k8s-worker01 AcquireTime: RenewTime: Tue, 04 Apr 2023 12:39:22 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Tue, 04 Apr 2023 11:59:42 +0800 Tue, 04 Apr 2023 11:59:42 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Tue, 04 Apr 2023 12:36:30 +0800 Mon, 03 Apr 2023 20:28:43 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 04 Apr 2023 12:36:30 +0800 Mon, 03 Apr 2023 20:28:43 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 04 Apr 2023 12:36:30 +0800 Mon, 03 Apr 2023 20:28:43 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 04 Apr 2023 12:36:30 +0800 Mon, 03 Apr 2023 20:28:47 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.10.161 Hostname: k8s-worker01 Capacity: cpu: 4 ephemeral-storage: 51175Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 4026128Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 48294789041 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3923728Ki pods: 110 System Info: Machine ID: f618107e5de3464bbfc77620a718fdd5 System UUID: 7DD24D56-096D-7853-6E3E-0ED5FAB35AC3 Boot ID: 8f8f964d-a639-4a11-972d-043dca2f718e Kernel Version: 3.10.0-1160.76.1.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.7.0 Kubelet Version: v1.26.3 Kube-Proxy Version: v1.26.3 PodCIDR: 10.244.2.0/24 PodCIDRs: 10.244.2.0/24 Non-terminated Pods: (3 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-typha-c856d6bfd-l8nhw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 24m kube-system hybridnet-daemon-ls2rh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 24m kube-system kube-proxy-vnwhh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 0 (0%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl describe nodes k8s-worker02 Name: k8s-worker02 Roles: Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-worker02 kubernetes.io/os=linux network=underlay-nethost networking.alibaba.com/dualstack-address-quota=empty networking.alibaba.com/ipv4-address-quota=nonempty networking.alibaba.com/ipv6-address-quota=empty networking.alibaba.com/overlay-network-attachment=true networking.alibaba.com/underlay-network-attachment=true Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.10.162/24 projectcalico.org/IPv4VXLANTunnelAddr: 10.244.69.192 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 03 Apr 2023 20:28:39 +0800 Taints: Unschedulable: false Lease: HolderIdentity: k8s-worker02 AcquireTime: RenewTime: Tue, 04 Apr 2023 12:39:49 +0800 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Tue, 04 Apr 2023 12:00:08 +0800 Tue, 04 Apr 2023 12:00:08 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Tue, 04 Apr 2023 12:36:02 +0800 Mon, 03 Apr 2023 20:28:39 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 04 Apr 2023 12:36:02 +0800 Mon, 03 Apr 2023 20:28:39 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 04 Apr 2023 12:36:02 +0800 Mon, 03 Apr 2023 20:28:39 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 04 Apr 2023 12:36:02 +0800 Mon, 03 Apr 2023 20:28:46 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.10.162 Hostname: k8s-worker02 Capacity: cpu: 4 ephemeral-storage: 51175Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 4026120Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 48294789041 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3923720Ki pods: 110 System Info: Machine ID: f618107e5de3464bbfc77620a718fdd5 System UUID: 43224D56-1D27-9147-1CC6-B7FDBED96000 Boot ID: 86cbf2e5-324d-4457-8585-f599ddc38c7c Kernel Version: 3.10.0-1160.76.1.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.7.0 Kubelet Version: v1.26.3 Kube-Proxy Version: v1.26.3 PodCIDR: 10.244.1.0/24 PodCIDRs: 10.244.1.0/24 Non-terminated Pods: (3 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-typha-c856d6bfd-slppp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 24m kube-system hybridnet-daemon-lxcb6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 25m kube-system kube-proxy-zgrj6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 0 (0%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: ~~~ # 六、创建pod使用overlay网络 ~~~powershell [root@k8s-master01 hybridnet]# vim 02-tomcat-app1-overlay.yaml [root@k8s-master01 hybridnet]# cat 02-tomcat-app1-overlay.yaml kind: Deployment apiVersion: apps/v1 metadata: labels: app: myserver-tomcat-app1-deployment-overlay-label name: myserver-tomcat-app1-deployment-overlay namespace: myserver spec: replicas: 1 selector: matchLabels: app: myserver-tomcat-app1-overlay-selector template: metadata: labels: app: myserver-tomcat-app1-overlay-selector spec: containers: - name: myserver-tomcat-app1-container image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/tomcat-app1:v1 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" --- kind: Service apiVersion: v1 metadata: labels: app: myserver-tomcat-app1-service-overlay-label name: myserver-tomcat-app1-service-overlay namespace: myserver spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 8080 nodePort: 30003 selector: app: myserver-tomcat-app1-overlay-selector ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create ns myserver namespace/myserver created ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create -f 02-tomcat-app1-overlay.yaml deployment.apps/myserver-tomcat-app1-deployment-overlay created service/myserver-tomcat-app1-service-overlay created ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get pods -n myserver -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myserver-tomcat-app1-deployment-overlay-5bb44b6bf6-lnp68 1/1 Running 0 2m1s 10.244.0.17 k8s-worker02 ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get svc -n myserver NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE myserver-tomcat-app1-service-overlay NodePort 10.111.170.113 80:30003/TCP 40s ~~~ ![image-20230404131326907](../../img/kubernetes/kubernetes_hybridnet/image-20230404131326907.png) # 七、创建pod并使用underlay网络 ~~~powershell [root@k8s-master01 hybridnet]# vim 03-tomcat-app1-underlay.yaml [root@k8s-master01 hybridnet]# cat 03-tomcat-app1-underlay.yaml kind: Deployment apiVersion: apps/v1 metadata: labels: app: myserver-tomcat-app1-deployment-underlay-label name: myserver-tomcat-app1-deployment-underlay namespace: myserver spec: replicas: 1 selector: matchLabels: app: myserver-tomcat-app1-underlay-selector template: metadata: labels: app: myserver-tomcat-app1-underlay-selector annotations: networking.alibaba.com/network-type: Underlay spec: containers: - name: myserver-tomcat-app1-container image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/tomcat-app1:v2 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" --- kind: Service apiVersion: v1 metadata: labels: app: myserver-tomcat-app1-service-underlay-label name: myserver-tomcat-app1-service-underlay namespace: myserver spec: ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: app: myserver-tomcat-app1-underlay-selector ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create -f 03-tomcat-app1-underlay.yaml deployment.apps/myserver-tomcat-app1-deployment-underlay created service/myserver-tomcat-app1-service-underlay created ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get pods -n myserver -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myserver-tomcat-app1-deployment-overlay-5bb44b6bf6-lnp68 1/1 Running 0 5m13s 10.244.0.17 k8s-worker02 myserver-tomcat-app1-deployment-underlay-7f65f45449-mvkj7 1/1 Running 0 10m 192.168.10.10 k8s-worker01 ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get svc -n myserver NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE myserver-tomcat-app1-service-overlay NodePort 10.111.170.113 80:30003/TCP 5m56s myserver-tomcat-app1-service-underlay ClusterIP 10.99.144.184 80/TCP 100s ~~~ ![image-20230404131724427](../../img/kubernetes/kubernetes_hybridnet/image-20230404131724427.png) ~~~powershell [root@k8s-master01 hybridnet]# curl http://192.168.10.10:8080/myapp/ tomcat app1 v2 ~~~ # 八、创建service并使用underlay网络 ## 8.1 K8S集群初始化注意事项 > 初始化完成后，把所有worker节点加入 ~~~powershell kubeadm init --apiserver-advertise-address=192.168.10.160 \ --apiserver-bind-port=6443 \ --kubernetes-version=v1.26.3 \ --pod-network-cidr=10.244.0.0/16 \ --service-cidr=192.168.200.0/24 \ --service-dns-domain=cluster.local \ --cri-socket unix:///var/run/containerd/containerd.sock ~~~ ~~~powershell --service-cidr=192.168.200.0/24 此位置指定的网段不能与本地已有网络冲突 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 192.168.200.1 443/TCP 82m ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE calico-typha ClusterIP 192.168.200.249 5473/TCP 79m hybridnet-webhook ClusterIP 192.168.200.149 443/TCP 79m kube-dns ClusterIP 192.168.200.10 53/UDP,53/TCP,9153/TCP 83m ~~~ > k8s集群初始化成功后，必须先安装helm，再通过helm部署hybridnet后创建underlay网络。 ## 8.2 创建underlay网络 ~~~powershell [root@k8s-master01 ~]# mkdir /root/hybridnet [root@k8s-master01 ~]# cd hybridnet/ [root@k8s-master01 hybridnet]# ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl label node k8s-master01 network=underlay-nethost node/k8s-master01 labeled [root@k8s-master01 hybridnet]# kubectl label node k8s-worker01 network=underlay-nethost node/k8s-worker01 labeled [root@k8s-master01 hybridnet]# kubectl label node k8s-worker02 network=underlay-nethost node/k8s-worker02 labeled ~~~ ~~~powershell [root@k8s-master01 hybridnet]# vim 01-underlay-network.yaml [root@k8s-master01 hybridnet]# cat 01-underlay-network.yaml --- apiVersion: networking.alibaba.com/v1 kind: Network metadata: name: underlay-network1 spec: netID: 0 type: Underlay nodeSelector: network: "underlay-nethost" --- apiVersion: networking.alibaba.com/v1 kind: Subnet metadata: name: underlay-network1 spec: network: underlay-network1 netID: 0 range: version: "4" cidr: "192.168.200.0/24" gateway: "192.168.200.254" start: "192.168.200.50" end: "192.168.200.200" ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create -f 01-underlay-network.yaml ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get network NAME NETID TYPE MODE V4TOTAL V4USED V4AVAILABLE LASTALLOCATEDV4SUBNET V6TOTAL V6USED V6AVAILABLE LASTALLOCATEDV6SUBNET init 4 Overlay 65534 3 65531 init 0 0 0 underlay-network1 0 Underlay 151 0 151 underlay-network1 0 0 0 ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get subnet NAME VERSION CIDR START END GATEWAY TOTAL USED AVAILABLE NETID NETWORK init 4 10.244.0.0/16 65534 3 65531 init underlay-network1 4 192.168.200.0/24 192.168.200.50 192.168.200.200 192.168.200.254 151 151 0 underlay-network1 ~~~ ## 8.3 创建service使用underlay网络 ~~~powershell [root@k8s-master01 hybridnet]# vim 02-service-underlay.yaml [root@k8s-master01 hybridnet]# cat 02-service-underlay.yaml kind: Deployment apiVersion: apps/v1 metadata: labels: app: myserver-tomcat-app1-deployment-underlay-label name: myserver-tomcat-app1-deployment-underlay namespace: myserver spec: replicas: 1 selector: matchLabels: app: myserver-tomcat-app1-underlay-selector template: metadata: labels: app: myserver-tomcat-app1-underlay-selector spec: containers: - name: myserver-tomcat-app1-container image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/tomcat-app1:v2 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" --- kind: Service apiVersion: v1 metadata: labels: app: myserver-tomcat-app1-service-underlay-label name: myserver-tomcat-app1-service-underlay namespace: myserver annotations: networking.alibaba.com/network-type: Underlay 重点注意这里 spec: ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: app: myserver-tomcat-app1-underlay-selector ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create ns myserver ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl create -f 02-service-underlay.yaml ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get pods -n myserver -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myserver-tomcat-app1-deployment-underlay-849f7f9cf5-x48x8 1/1 Running 0 13m 10.244.0.3 k8s-worker02 ~~~ ~~~powershell [root@k8s-master01 hybridnet]# kubectl get svc -n myserver -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR myserver-tomcat-app1-service-underlay ClusterIP 192.168.200.234 80/TCP 61m app=myserver-tomcat-app1-underlay-selector ~~~ ## 8.4 访问 ### 8.4.1 在K8S集群内节点上访问 ~~~powershell [root@k8s-master01 hybridnet]# curl http://10.244.0.3:8080/myapp/ tomcat app1 v2 ~~~ ~~~powershell [root@k8s-master01 hybridnet]# curl http://192.168.200.234/myapp/ tomcat app1 v2 ~~~ ### 8.4.2 在K8S集群外windows主机上访问 ![image-20230406143957394](../../img/kubernetes/kubernetes_hybridnet/image-20230406143957394.png) ~~~powershell C:\WINDOWS\system32>route add 192.168.200.0 mask 255.255.255.0 -p 192.168.10.160 说明： 去往192.168.200.0/24网段通过192.168.10.160，这个192.168.10.160为k8s集群节点IP地址。 ~~~ ![image-20230406144105191](../../img/kubernetes/kubernetes_hybridnet/image-20230406144105191.png) ![image-20230406144215187](../../img/kubernetes/kubernetes_hybridnet/image-20230406144215187.png) ![image-20230406144245003](../../img/kubernetes/kubernetes_hybridnet/image-20230406144245003.png) ![image-20230406144319092](../../img/kubernetes/kubernetes_hybridnet/image-20230406144319092.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_introduce.md ================================================ # Kubernetes介绍与集群架构 # 一、认识容器编排工具 - docker machine - 主要用于准备docker host - 现已弃用 - 建议使用docker desktop - docker compose - Compose 是一个用于定义和运行多容器 Docker 应用程序的工具。 - 使用 Compose，您可以使用 YAML 文件来配置应用程序的服务。 - 使用一个命令，您可以从您的配置中创建并启动所有服务。 - docker swarm - 内置于docker引擎中 - 对docker引擎进行集群级别的管理 - 分布式设计，可以让集群资源更多，管理更多的主机 - 声明式服务模型，通过YAML文件定义应用程序服务所需状态 - 服务规模可扩大可缩小，保持用户期望状态 - 服务发现 - 负载均衡 - 滚动更新等 - docker service - docker stack - kubernetes - Kubernetes作为一个容器集群管理系统，用于管理容器云平台中多个主机上的容器应用，Kubernetes的目标是让部署容器化的应用变得简单且高效，所以 Kubernetes 提供了应用部署，规划，更新，维护的一整套完整的机制。 - Kubernetes没有固定要求容器的格式，但是Kubernetes使用它自己的API和命令行接口来进行容器编排。 - 除了Docker容器之外，Kubernetes还支持其他多种容器，如 Containerd、rkt、CoreOS 等。 - Kubernetes 是自成体系的管理工具，可以实现容器调度，资源管理，服务发现，健康检查，自动伸缩，更新升级等，也可以在应用模版配置中指定副本数量，服务要求（IO 优先；性能优先等），资源使用区间，标签（Labels等）来匹配特定要求达到预期状态等，这些特征便足以征服开发者，再加上 Kubernetes 有一个非常活跃的社区。它为用户提供了更多的选择以方便用户扩展编排容器来满足他们的需求。但是由于 Kubernetes 使用了自己的 API 接口，所以命令系统是另外一套系统，这也是 kubernetes 应用门槛比较高的原因所在。 - mesos+marathon - Apache Mesos 是一个分布式系统内核的开源集群管理器，Apache Mesos 的出现要远早于 Docker Swarm 和 Kubernetes。 - 其结合Marathon 这个基于容器的应用程序的编排框架，它为 Docker Swarm 和 Kubernetes 提供了一个有效的替代方案。 - Mesos 能够在同样的集群机器上运行多种分布式系统类型，可以更加动态高效的共享资源。 - Messos 也提供服务失败检查，服务发布，服务跟踪，服务监控，资源管理和资源共享。 - Messos 可以扩展伸缩到数千个节点。 - 如果你拥有很多的服务器而且想构建一个大的集群的时候，Mesos 就派上用场了。 - 很多的现代化可扩展性的数据处理应用都可以在 Mesos 上运行，包括大数据框架 Hadoop、Kafka、Spark。 - 但是大而全，往往就是对应的复杂和困难，这一点体现在 Messos 上是完全正确，与Docker 和 Docker Swarm 使用同一种 API 不同的，Mesos 和 Marathon 都有自己的 API，这使得它们比其他编排系统更加的复杂。Apache Mesos 是混合环境的完美编配工具，由于它包含容器和非容器的应用，虽然 Messos 很稳定，但是它的使用户快速学习应用变得更加困难，这也是在应用和部署场景下难于推广的原因之一。 | 分布式资源管理框架 | 主要特点 | 备注 | | ------------------ | ---------------------------------------------- | -------------------------- | | Apache Mesos | 功能强大，但系统复杂不易用 | 2019年Twitter宣布弃用 | | Docker Swarm | 集成在docker引擎中，不需要单独安装，但功能较少 | 2019年阿里云宣布弃用 | | Google Kubernetes | 功能强大，但学习成本与管理成本高 | 目前没有替代者，多领域应用 | # 二、认识kubernetes ## 2.1 kubernetes含义、起源、归属 ### 2.1.1 含义 希腊语：舵手、飞行员 ![1557045795562](../../img/kubernetes/kubernetes_introduce/1557045795562.png) ### 2.1.2 起源 - 源自于谷歌Borg - 使用golang语言开发 - 简称为k8s ### 2.1.3 归属 现归属于CNCF - 云原生(CloudNative)计算基金会 - 是一个开源软件基金会，致力于使云计算普遍性和持续性 - 官方：http://www.cncf.io ![image-20220324160124674](../../img/kubernetes/kubernetes_introduce/image-20220324160124674.png) ## 2.2 kubernetes版本 - 2014年9月第一个正式版本 - 2015年7月1.0版本正式发布 - 现在稳定版本为1.23 - 主要贡献者：Google,Redhat,Microsoft,IBM,Intel - 代码托管github: ![1557046658333](../../img/kubernetes/kubernetes_introduce/1557046658333.png) ## 2.3 Kubernetes用户 - 2017年docker官方宣布原生支持kubernetes - RedHat公司 PaaS平台 OpenShift核心是kubernetes - Rancher平台核心是kubernetes - 现国内大多数公司都可使用kubernetes进行传统IT服务转换，以实现高效管理等。 ## 2.4 Kubernetes网址 - 官方网址 https://kubernetes.io/ ![1557046832480](../../img/kubernetes/kubernetes_introduce/1557046832480.png) https://kubernetes.io/zh/ ![image-20220324145026425](../../img/kubernetes/kubernetes_introduce/image-20220324145026425.png) - 中文社区 ![image-20220324145137982](../../img/kubernetes/kubernetes_introduce/image-20220324145137982.png) # 三、kubernetes架构 ## 3.1 软件架构说明 - 有中心节点分布式架构 - hadoop集群 - ceph集群 - 无中心节点分布式架构 - glusterFS - kubernetes是具有中心节点的分布式架构,也就是说有master管理节点 - Master Node - 中心节点 - manager - 简单叫法 - master节点 - Minion Node - 工作节点 - worker - 简单叫点 - node节点 - worker节点 ## 3.2 Kubernetes架构图示图 ![1557048978763](../../img/kubernetes/kubernetes_introduce/1557048978763.png) ![1604748712024](../../img/kubernetes/kubernetes_introduce/2.png) # 四、Kubernetes集群节点组件 ## 4.1 Master节点组件 master节点是集群管理中心，它的组件可以在集群内任意节点运行，但是为了方便管理所以会在一台主机上运行Master所有组件，**并且不在此主机上运行用户容器** Master组件包括： - kube-apiserver ​ 用于暴露kubernetes API，任何的资源请求/调用操作都是通过kube-apiserver提供的接口进行。 - kube-controller-manager ​ 控制器管理器，用于对控制器进行管理，它们是集群中处理常规任务的后台线程。 - kube-scheduler 监视新创建没有分配到Node的Pod，为Pod选择一个Node运行。 - ETCD 是kubernetes提供默认的存储系统，保存所有集群数据。 ## 4.2 Node节点组件介绍 node节点用于运行以及维护Pod, 管理volume(CVI)和网络(CNI)，维护pod及service等信息 Node组件包括： - kubelet - 负责维护容器的生命周期(创建pod，销毁pod)，同时也负责Volume(CVI)和网络(CNI)的管理 - kube-proxy - 通过在主机上维护网络规则并执行连接转发来实现service(iptables/ipvs) - 随时与apiserver通信，把Service或Pod改变提交给apiserver，保存至etcd（可做高可用集群）中，负责service实现，从内部pod至service和从外部node到service访问。 - Container Runtime - 容器运行时(Container Runtime) - 负责镜像管理以及Pod和容器的真正运行 - 支持docker/containerd/Rkt/Pouch/Kata等多种运行时 ![1564397969713](../../img/kubernetes/kubernetes_introduce/k8s.png) ## 4.3 Add-ons介绍 Add-ons(附件)使功能更丰富，没它并不影响实际使用，可以与主体程序很好结合起来使用 - coredns/kube-dns: 负责为整个集群提供DNS服务 - Ingress Controller 为服务提供集群外部访问 - Heapster/Metries-server 提供集群资源监控(监控容器可以使用prometheus) - Dashboard 提供集群GUI - Federation 提供跨可用区的集群 - Fluentd-elasticsearch 提供集群日志采集、存储与查询 ================================================ FILE: docs/cloud/kubernetes/kubernetes_ipv4_and_ipv6.md ================================================ # K8S 1.22版本双栈协议（IPv4&IPv6）集群部署 # 一、部署说明 > 此笔记主要针对 k8s 1.22，其它版本请自行测试使用。 > > 必须使用Open vSwitch功能。 # 二、主机准备 ## 2.1 主机名配置 由于本次使用3台主机完成kubernetes集群部署，其中1台为master节点,名称为k8s-master01;其中2台为worker节点，名称分别为：k8s-worker01及k8s-worker02 ~~~powershell master节点 # hostnamectl set-hostname k8s-master01 ~~~ ~~~powershell worker01节点 # hostnamectl set-hostname k8s-worker01 ~~~ ~~~powershell worker02节点 # hostnamectl set-hostname k8s-worker02 ~~~ ## 2.2 主机IP地址配置 ### 2.2.1 k8s-master节点 ~~~powershell [root@k8s-master01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@k8s-master01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" NAME="ens33" UUID="063bfc1c-c7c2-4c62-89d0-35ae869e44e7" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.160" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" IPV6INIT="yes" IPV6_AUTOCONF="no" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" IPV6ADDR=2003::11/64 IPV6_DEFAULTGW=2003::1 ~~~ ~~~powershell [root@k8s-master01 ~]# systemctl restart network ~~~ ~~~powershell [root@k8s-master01 ~]# ip a s ~~~ ### 2.2.2 k8s-worker01节点 ~~~powershell [root@k8s-worker01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@k8s-worker01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" NAME="ens33" UUID="063bfc1c-c7c2-4c62-89d0-35ae869e44e7" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.161" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" IPV6INIT="yes" IPV6_AUTOCONF="no" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" IPV6ADDR=2003::12/64 IPV6_DEFAULTGW=2003::1 ~~~ ~~~powershell [root@k8s-worker01 ~]# systemctl restart network ~~~ ~~~powershell [root@k8s-worker01 ~]# ip a s ~~~ ### 2.2.3 k8s-worker02节点 ~~~powershell [root@k8s-worker02 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@k8s-worker02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" NAME="ens33" UUID="063bfc1c-c7c2-4c62-89d0-35ae869e44e7" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.162" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" IPV6INIT="yes" IPV6_AUTOCONF="no" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" IPV6ADDR=2003::13/64 IPV6_DEFAULTGW=2003::1 ~~~ ~~~powershell [root@k8s-worker02 ~]# systemctl restart network ~~~ ~~~powershell [root@k8s-worker02 ~]# ip a s ~~~ ### 2.2.4 在k8s-master上ping通ipv6地址 ~~~powershell [root@k8s-master01 ~]# ping6 -c 4 2003::11 PING 2003::11(2003::11) 56 data bytes 64 bytes from 2003::11: icmp_seq=1 ttl=64 time=0.030 ms 64 bytes from 2003::11: icmp_seq=2 ttl=64 time=0.065 ms 64 bytes from 2003::11: icmp_seq=3 ttl=64 time=0.037 ms 64 bytes from 2003::11: icmp_seq=4 ttl=64 time=0.030 ms --- 2003::11 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3083ms rtt min/avg/max/mdev = 0.030/0.040/0.065/0.015 ms ~~~ ~~~powershell [root@k8s-master01 ~]# ping6 -c 4 2003::12 PING 2003::12(2003::12) 56 data bytes 64 bytes from 2003::12: icmp_seq=1 ttl=64 time=0.323 ms 64 bytes from 2003::12: icmp_seq=2 ttl=64 time=0.557 ms 64 bytes from 2003::12: icmp_seq=3 ttl=64 time=0.552 ms 64 bytes from 2003::12: icmp_seq=4 ttl=64 time=1.30 ms --- 2003::12 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3050ms rtt min/avg/max/mdev = 0.323/0.685/1.308/0.371 ms ~~~ ~~~powershell [root@k8s-master01 ~]# ping6 -c 4 2003::13 PING 2003::13(2003::13) 56 data bytes 64 bytes from 2003::13: icmp_seq=1 ttl=64 time=0.370 ms 64 bytes from 2003::13: icmp_seq=2 ttl=64 time=0.348 ms 64 bytes from 2003::13: icmp_seq=3 ttl=64 time=0.491 ms 64 bytes from 2003::13: icmp_seq=4 ttl=64 time=0.497 ms --- 2003::13 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3059ms rtt min/avg/max/mdev = 0.348/0.426/0.497/0.071 ms ~~~ ## 2.3 主机名解析 ~~~powershell # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.160 k8s-master01 192.168.10.161 k8s-worker01 192.168.10.162 k8s-worker02 2003::11 k8s-master01 2003::12 k8s-worker01 2003::13 k8s-worker02 ~~~ ## 2.4 主机安全设置 > 所有主机均需操作。 ~~~powershell # systemctl stop firewalld && systemctl disable firewalld ~~~ ~~~powershell # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ## 2.5 升级操作系统内核 > 所有主机均需要操作。 ~~~powershell 导入elrepo gpg key # rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell 安装elrepo YUM源仓库 # yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell 安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 # yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64 ~~~ ~~~powershell 设置grub2默认引导为0 # grub2-set-default 0 ~~~ ~~~powershell 重新生成grub2引导文件 # grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ~~~powershell 更新后，需要重启，使用升级的内核生效。 # reboot ~~~ ~~~powershell 重启后，需要验证内核是否为更新对应的版本 # uname -r ~~~ #### 2.6 配置内核转发及网桥过滤 >所有主机均需要操作。 ~~~powershell 添加网桥过滤及内核转发配置文件 # cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 vm.swappiness = 0 ~~~ ~~~powershell 加载br_netfilter模块 # modprobe br_netfilter ~~~ ~~~powershell 查看是否加载 # lsmod | grep br_netfilter br_netfilter 22256 0 bridge 151336 1 br_netfilter ~~~ ## 2.7 调整主机内核参数 ~~~powershell # vim /etc/sysctl.conf 添加如下内容： net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 net.ipv4.ip_nonlocal_bind = 1 ~~~ ~~~powershell 使其生效 # sysctl -p /etc/sysctl.conf ~~~ ## 2.8 主机时钟同步设置 ~~~powershell # crontab -e 0 */1 * * * ntpdate time1.aliyun.com ~~~ ## 2.9 主机swap分区设置 ~~~powershell 临时关闭 # swapoff -a ~~~ ~~~powershell 永远关闭swap分区，需要重启操作系统 # cat /etc/fstab ...... # /dev/mapper/centos-swap swap swap defaults 0 0 在上一行中行首添加# ~~~ ## 2.10 Open vSwitch > 下载地址链接：http://www.openvswitch.org/download/ > > OVS ~~~powershell 安装依赖 # yum -y install openssl openssl-devel ~~~ ~~~powershell 查看是否存在python3 # yum list python3 ~~~ ~~~powershell 安装依赖 # yum install python3 python3-devel -y ~~~ ~~~powershell 下载软件源码包 # wget https://www.openvswitch.org/releases/openvswitch-2.16.2.tar.gz ~~~ ~~~powershell 解压 # tar -xf openvswitch-2.16.2.tar.gz ~~~ ~~~powershell # cd openvswitch-2.16.2/ ~~~ ~~~powershell [root@k8s-XXX openvswitch-2.16.2]# ./configure ~~~ ~~~powershell [root@k8s-XXX openvswitch-2.16.2]# make ~~~ ~~~powershell [root@k8s-XXX openvswitch-2.16.2]# make install ~~~ ~~~powershell 加载模块至内核 # modprobe openvswitch # lsmod | grep openvswitch openvswitch 139264 2 nsh 16384 1 openvswitch nf_conncount 24576 1 openvswitch nf_nat 45056 5 ip6table_nat,xt_nat,openvswitch,iptable_nat,xt_MASQUERADE nf_conntrack 147456 8 xt_conntrack,nf_nat,nfnetlink_cttimeout,xt_nat,openvswitch,nf_conntrack_netlink,nf_conncount,xt_MASQUERADE nf_defrag_ipv6 24576 2 nf_conntrack,openvswitch libcrc32c 16384 4 nf_conntrack,nf_nat,openvswitch,xfs ~~~ # 三、Docker安装 ## 3.1 Docker安装YUM源准备 >使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ## 3.2 Docker安装 ~~~powershell # yum -y install docker-ce-20.10.17 docke-ce-cli-20.10.17 ~~~ ## 3.3 启动Docker服务 ~~~powershell # systemctl enable --now docker ~~~ ## 3.4 修改cgroup方式 >/etc/docker/daemon.json 默认没有此文件，需要单独创建 ~~~powershell 在/etc/docker/daemon.json添加如下内容 # cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } ~~~ ~~~powershell # systemctl restart docker ~~~ # 四、k8s软件安装 ## 4.1 YUM源准备 ~~~powershell # vim /etc/yum.repos.d/k8s.repo # cat /etc/yum.repos.d/k8s.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ~~~ ## 4.2 K8S软件安装及kubelet配置 ### 4.2.1 软件安装 ~~~powershell # yum -y install kubeadm-1.22.11-0 kubelet-1.22.11-0 kubectl-1.22.11-0 ~~~ ### 4.2.2 kubelet配置 >为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ~~~powershell # vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ~~~ ~~~powershell 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 # systemctl enable kubelet ~~~ ## 4.3 K8S集群初始化配置文件准备 >默认 Kubernetes 下每个 node 使用 /24 掩码为 Pod 分配 IPv4 地址，使用 /64 掩码为 Pod 分配 IPv6 地址。可以通过 Controller-manager 参数调整掩码大小，本文会将 IPv4 掩码调整为 /25，IPv6 掩码调整为 /80。 > >（k8s 强制限制 node 掩码不能比 CIDR 掩码小 16 以上，因此当 IPv6 CIDR 使用 /64 得掩码时，node 掩码不能大于 /80 ~~~powershell [root@k8s-master01 ~]# vim kubeadm-config.yaml [root@k8s-master01 ~]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration networking: podSubnet: 10.244.0.0/16,2004::/64 serviceSubnet: 10.96.0.0/16,2005::/110 controllerManager: extraArgs: "node-cidr-mask-size-ipv4": "25" "node-cidr-mask-size-ipv6": "80" imageRepository: "" clusterName: "smartgo-cluster" kubernetesVersion: "v1.22.11" --- apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: "192.168.10.160" bindPort: 6443 nodeRegistration: kubeletExtraArgs: node-ip: 192.168.10.160,2003::11 ~~~ ## 4.4 K8S集群初始化 ~~~powershell [root@k8s-master01 ~]# kubeadm init --config=kubeadm-config.yaml ~~~ ~~~powershell 输出信息： [init] Using Kubernetes version: v1.22.11 [preflight] Running pre-flight checks [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.160] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.160 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.160 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 5.501915 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: tf68fl.pj4xsh62osypb4bj [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.160:6443 --token tf68fl.pj4xsh62osypb4bj \ --discovery-token-ca-cert-hash sha256:e4960afef684bbee72ae904356321997a6eef5bb0394a8d74b72ebaa0b638ecd ~~~ ~~~powershell [root@k8s-master01 ~]# mkdir -p $HOME/.kube [root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady control-plane 2m5s v1.22.11 ~~~ ## 4.5 把工作节点添加到K8S集群 ### 4.5.1 创建kubeadm-config.yaml文件 > 用于添加worker节点 ~~~powershell [root@k8s-worker01 ~]# vim kubeadm-config.yaml [root@k8s-worker01 ~]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration discovery: bootstrapToken: apiServerEndpoint: 192.168.10.160:6443 token: "tf68fl.pj4xsh62osypb4bj" caCertHashes: - "sha256:e4960afef684bbee72ae904356321997a6eef5bb0394a8d74b72ebaa0b638ecd" nodeRegistration: kubeletExtraArgs: node-ip: 192.168.10.161,2003::12 ~~~ ~~~powershell [root@k8s-worker01 ~]# kubeadm join --config=kubeadm-config.yaml [preflight] Running pre-flight checks [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady control-plane 16m v1.22.11 k8s-worker01 NotReady 88s v1.22.11 ~~~ ~~~powershell [root@k8s-worker02 ~]# vim kubeadm-config.yaml [root@k8s-worker02 ~]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration discovery: bootstrapToken: apiServerEndpoint: 192.168.10.160:6443 token: "tf68fl.pj4xsh62osypb4bj" caCertHashes: - "sha256:e4960afef684bbee72ae904356321997a6eef5bb0394a8d74b72ebaa0b638ecd" nodeRegistration: kubeletExtraArgs: node-ip: 192.168.10.162,2003::13 ~~~ ~~~powershell [root@k8s-worker02 ~]# kubeadm join --config=kubeadm-config.yaml [preflight] Running pre-flight checks [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady control-plane 19m v1.22.11 k8s-worker01 NotReady 4m44s v1.22.11 k8s-worker02 NotReady 31s v1.22.11 ~~~ # 五、网络工具Antrea部署 ## 5.0 认识Antrea Antrea 是一个旨在成为 Kubernetes 原生的[Kubernetes网络解决方案。](https://kubernetes.io/)[它在第 3/4 层运行，利用Open vSwitch](https://www.openvswitch.org/)作为网络数据平面，为 Kubernetes 集群提供网络和安全服务 。 ![image-20230324174008962](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324174008962.png) Open vSwitch 是一种广泛采用的高性能可编程虚拟交换机；Antrea 利用它来实现 Pod 网络和安全功能。例如，Open vSwitch 使 Antrea 能够以非常有效的方式实施 Kubernetes 网络策略。 ## 5.1 获取antrea部署文件 ![image-20230324173843525](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324173843525.png) ![image-20230324173901871](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324173901871.png) ![image-20230324174249721](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324174249721.png) ![image-20230324174316268](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324174316268.png) ~~~powershell [root@k8s-master01 ~]# wget https://github.com/antrea-io/antrea/releases/download/v1.11.0/antrea.yml ~~~ ## 5.2 修改antrea部署文件 > 1、禁用overlay封装模式及SNAT模式 > > 2、配置Service的IPv4及IPv6地址段。 ~~~powershell [root@k8s-master01 ~]# vim antrea.yml ~~~ ~~~powershell 3022 trafficEncapMode: "encap" 3023 3024 # Whether or not to SNAT (using the Node IP) the egress traffic from a Pod to the external network. 3025 # This option is for the noEncap traffic mode only, and the default value is false. In the noEncap 3026 # mode, if the cluster's Pod CIDR is reachable from the external network, then the Pod traffic to 3027 # the external network needs not be SNAT'd. In the networkPolicyOnly mode, antrea-agent never 3028 # performs SNAT and this option will be ignored; for other modes it must be set to false. 3029 noSNAT: false 3022 trafficEncapMode: "noencap" 3023 3024 # Whether or not to SNAT (using the Node IP) the egress traffic from a Pod to the external network. 3025 # This option is for the noEncap traffic mode only, and the default value is false. In the noEncap 3026 # mode, if the cluster's Pod CIDR is reachable from the external network, then the Pod traffic to 3027 # the external network needs not be SNAT'd. In the networkPolicyOnly mode, antrea-agent never 3028 # performs SNAT and this option will be ignored; for other modes it must be set to false. 3029 noSNAT: true ~~~ ~~~powershell 3097 serviceCIDR: "" 3098 3099 # ClusterIP CIDR range for IPv6 Services. It's required when using kube-proxy to provide IPv6 Service in a Dual-Stack 3100 # cluster or an IPv6 only cluster. The value should be the same as the configuration for kube-apiserver specified by 3101 # --service-cluster-ip-range. When AntreaProxy is enabled, this parameter is not needed. 3102 # No default value for this field. 3103 serviceCIDRv6: "" 3097 serviceCIDR: "10.96.0.0/16" 3098 3099 # ClusterIP CIDR range for IPv6 Services. It's required when using kube-proxy to provide IPv6 Service in a Dual-Stack 3100 # cluster or an IPv6 only cluster. The value should be the same as the configuration for kube-apiserver specified by 3101 # --service-cluster-ip-range. When AntreaProxy is enabled, this parameter is not needed. 3102 # No default value for this field. 3103 serviceCIDRv6: "2005::/110" ~~~ ## 5.3 应用antrea部署文件 ~~~powershell [root@k8s-master01 ~]# kubectl create -f antrea.yml ~~~ ~~~powershell 输出内容： customresourcedefinition.apiextensions.k8s.io/antreaagentinfos.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/antreacontrollerinfos.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/clustergroups.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/clusternetworkpolicies.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/egresses.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/externalentities.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/externalippools.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/externalnodes.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/ippools.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/supportbundlecollections.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/tiers.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/traceflows.crd.antrea.io created customresourcedefinition.apiextensions.k8s.io/trafficcontrols.crd.antrea.io created serviceaccount/antrea-agent created serviceaccount/antctl created serviceaccount/antrea-controller created secret/antrea-agent-service-account-token created secret/antctl-service-account-token created configmap/antrea-config created customresourcedefinition.apiextensions.k8s.io/groups.crd.antrea.io created clusterrole.rbac.authorization.k8s.io/antrea-agent created clusterrole.rbac.authorization.k8s.io/antctl created clusterrole.rbac.authorization.k8s.io/antrea-cluster-identity-reader created clusterrole.rbac.authorization.k8s.io/antrea-controller created clusterrole.rbac.authorization.k8s.io/aggregate-antrea-policies-edit created clusterrole.rbac.authorization.k8s.io/aggregate-antrea-policies-view created clusterrole.rbac.authorization.k8s.io/aggregate-traceflows-edit created clusterrole.rbac.authorization.k8s.io/aggregate-traceflows-view created clusterrole.rbac.authorization.k8s.io/aggregate-antrea-clustergroups-edit created clusterrole.rbac.authorization.k8s.io/aggregate-antrea-clustergroups-view created clusterrolebinding.rbac.authorization.k8s.io/antrea-agent created clusterrolebinding.rbac.authorization.k8s.io/antctl created clusterrolebinding.rbac.authorization.k8s.io/antrea-controller created service/antrea created daemonset.apps/antrea-agent created deployment.apps/antrea-controller created apiservice.apiregistration.k8s.io/v1beta2.controlplane.antrea.io created apiservice.apiregistration.k8s.io/v1beta1.system.antrea.io created apiservice.apiregistration.k8s.io/v1alpha1.stats.antrea.io created mutatingwebhookconfiguration.admissionregistration.k8s.io/crdmutator.antrea.io created validatingwebhookconfiguration.admissionregistration.k8s.io/crdvalidator.antrea.io created ~~~ ## 5.4 验证antrea部署是否成功 ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE antrea-agent-2fzm4 2/2 Running 0 99s antrea-agent-9jp7g 2/2 Running 0 99s antrea-agent-vkmk4 2/2 Running 0 99s antrea-controller-789587f966-j62zq 1/1 Running 0 99s coredns-787d4945fb-82tmg 1/1 Running 0 37m coredns-787d4945fb-vdsln 1/1 Running 0 37m etcd-k8s-master01 1/1 Running 0 38m kube-apiserver-k8s-master01 1/1 Running 0 38m kube-controller-manager-k8s-master01 1/1 Running 0 38m kube-proxy-4pvpv 1/1 Running 0 37m kube-proxy-4szqs 1/1 Running 0 18m kube-proxy-sl8h5 1/1 Running 0 23m kube-scheduler-k8s-master01 1/1 Running 0 38m ~~~ ## 5.5 查看K8S集群节点主机路由信息 ~~~powershell [root@k8s-master01 ~]# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default gateway 0.0.0.0 UG 100 0 0 ens33 10.244.0.0 0.0.0.0 255.255.255.128 U 0 0 0 antrea-gw0 10.244.0.128 k8s-worker01 255.255.255.128 UG 0 0 0 ens33 10.244.1.0 k8s-worker02 255.255.255.128 UG 0 0 0 ens33 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.10.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 ~~~ ~~~powershell [root@k8s-master01 ~]# route -6 Kernel IPv6 routing table Destination Next Hop Flag Met Ref Use If [::]/96 [::] !n 1024 2 0 lo 0.0.0.0/96 [::] !n 1024 2 0 lo 2002:a00::/24 [::] !n 1024 1 0 lo 2002:7f00::/24 [::] !n 1024 2 0 lo 2002:a9fe::/32 [::] !n 1024 1 0 lo 2002:ac10::/28 [::] !n 1024 2 0 lo 2002:c0a8::/32 [::] !n 1024 1 0 lo 2002:e000::/19 [::] !n 1024 4 0 lo 2003::/64 [::] U 100 7 0 ens33 2004::/80 [::] U 256 3 0 antrea-gw0 2004::1:0:0:0/80 k8s-worker01 UG 1024 1 0 ens33 2004::2:0:0:0/80 k8s-worker02 UG 1024 1 0 ens33 3ffe:ffff::/32 [::] !n 1024 1 0 lo fe80::/64 [::] U 100 1 0 ens33 [::]/0 gateway UG 100 5 0 ens33 localhost/128 [::] Un 0 7 0 lo 2003::/128 [::] Un 0 3 0 ens33 k8s-master01/128 [::] Un 0 8 0 ens33 2004::/128 [::] Un 0 3 0 antrea-gw0 k8s-master01/128 [::] Un 0 3 0 antrea-gw0 fe80::/128 [::] Un 0 3 0 ens33 fe80::/128 [::] Un 0 3 0 antrea-gw0 k8s-master01/128 [::] Un 0 4 0 ens33 k8s-master01/128 [::] Un 0 3 0 antrea-gw0 ff00::/8 [::] U 256 3 0 ens33 ff00::/8 [::] U 256 2 0 antrea-gw0 [::]/0 [::] !n -1 1 0 lo ~~~ > 需要在外部设备上设置静态路由使得pod可路由 # 六、测试双栈协议可用性 ## 6.1 IPv6访问测试 ~~~powershell [root@k8s-master01 ~]# vim deployment.yaml [root@k8s-master01 ~]# cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginxweb name: nginxweb spec: replicas: 2 selector: matchLabels: app: nginxweb template: metadata: labels: app: nginxweb spec: containers: - name: nginxweb image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 name: http protocol: TCP ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl apply -f deployment.yaml deployment.apps/nginxweb created ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE nginxweb-5c77c86f4d-9fdgh 1/1 Running 0 4s nginxweb-5c77c86f4d-hzk6x 1/1 Running 0 4s ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -o yaml | grep ip - ip: 10.244.0.130 - ip: 2004::1:0:0:2 - ip: 10.244.1.2 - ip: 2004::2:0:0:2 ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl describe pods nginxweb-5c77c86f4d-9fdgh Name: nginxweb-5c77c86f4d-9fdgh Namespace: default Priority: 0 Node: k8s-worker01/192.168.10.161 Start Time: Fri, 24 Mar 2023 19:56:16 +0800 Labels: app=nginxweb pod-template-hash=5c77c86f4d Annotations: Status: Running IP: 10.244.0.130 IPs: IP: 10.244.0.130 IP: 2004::1:0:0:2 Controlled By: ReplicaSet/nginxweb-5c77c86f4d Containers: nginxweb: Container ID: docker://68139623df054e7eb47f8a0fdb3891dc36c926ef36edc5b4c4dc25e81ffe3d01 Image: nginx:latest Image ID: docker-pullable://nginx@sha256:617661ae7846a63de069a85333bb4778a822f853df67fe46a688b3f0e4d9cb87 Port: 80/TCP Host Port: 0/TCP State: Running Started: Fri, 24 Mar 2023 19:56:18 +0800 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dgjq4 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-dgjq4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 75s default-scheduler Successfully assigned default/nginxweb-5c77c86f4d-9fdgh to k8s-worker01 Normal Pulled 74s kubelet Container image "nginx:latest" already present on machine Normal Created 74s kubelet Created container nginxweb Normal Started 73s kubelet Started container nginxweb ~~~ ~~~powershell [root@k8s-master01 ~]# ping6 2004::1:0:0:2 PING 2004::1:0:0:2(2004::1:0:0:2) 56 data bytes 64 bytes from 2004::1:0:0:2: icmp_seq=8 ttl=62 time=2049 ms 64 bytes from 2004::1:0:0:2: icmp_seq=9 ttl=62 time=1024 ms 64 bytes from 2004::1:0:0:2: icmp_seq=10 ttl=62 time=2.42 ms 64 bytes from 2004::1:0:0:2: icmp_seq=11 ttl=62 time=0.370 ms 64 bytes from 2004::1:0:0:2: icmp_seq=12 ttl=62 time=0.651 ms 64 bytes from 2004::1:0:0:2: icmp_seq=13 ttl=62 time=1.30 ms ~~~ ~~~powershell [root@k8s-master01 ~]# ping6 2004::2:0:0:2 PING 2004::2:0:0:2(2004::2:0:0:2) 56 data bytes 64 bytes from 2004::2:0:0:2: icmp_seq=1 ttl=62 time=1.35 ms 64 bytes from 2004::2:0:0:2: icmp_seq=2 ttl=62 time=1.24 ms ~~~ ~~~powershell [root@k8s-master01 ~]# curl -g -6 [2004::1:0:0:2] Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ ~~~powershell [root@k8s-master01 ~]# curl -g -6 [2004::2:0:0:2] Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ ## 6.2 双栈service测试 Kubernetes Service 的 IPv4 family 支持下列选项，默认为 IPv4 SingleStack： - SingleStack - PreferDualStack - RequireDualStack ~~~powershell [root@k8s-master01 ~]# vim service.yaml [root@k8s-master01 ~]# cat service.yaml apiVersion: v1 kind: Service metadata: name: nginxweb-v6 spec: selector: app: nginxweb ports: - protocol: TCP port: 80 targetPort: 80 type: NodePort ipFamilyPolicy: RequireDualStack ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl create -f service.yaml service/nginxweb-v6 created ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 63m nginxweb-v6 NodePort 10.96.53.221 80:32697/TCP 22s ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get svc -o yaml apiVersion: v1 items: - apiVersion: v1 kind: Service metadata: creationTimestamp: "2023-03-24T11:12:51Z" labels: component: apiserver provider: kubernetes name: kubernetes namespace: default resourceVersion: "205" uid: d1884296-7060-4dab-961b-3537de7490d0 spec: clusterIP: 10.96.0.1 clusterIPs: - 10.96.0.1 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: https port: 443 protocol: TCP targetPort: 6443 sessionAffinity: None type: ClusterIP status: loadBalancer: {} - apiVersion: v1 kind: Service metadata: creationTimestamp: "2023-03-24T12:16:26Z" name: nginxweb-v6 namespace: default resourceVersion: "6905" uid: c2baa4f5-1ec8-43d3-9fc9-77f9236a4eba spec: clusterIP: 10.96.53.221 clusterIPs: - 10.96.53.221 - 2005::2c47 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 - IPv6 ipFamilyPolicy: RequireDualStack ports: - nodePort: 32697 port: 80 protocol: TCP targetPort: 80 selector: app: nginxweb sessionAffinity: None type: NodePort status: loadBalancer: {} kind: List metadata: resourceVersion: "" selfLink: "" ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl describe svc nginxweb-v6 Name: nginxweb-v6 Namespace: default Labels: Annotations: Selector: app=nginxweb Type: NodePort IP Family Policy: RequireDualStack IP Families: IPv4,IPv6 IP: 10.96.53.221 IPs: 10.96.53.221,2005::2c47 Port: 80/TCP TargetPort: 80/TCP NodePort: 32697/TCP Endpoints: 10.244.0.130:80,10.244.1.2:80 Session Affinity: None External Traffic Policy: Cluster Events: ~~~ ~~~powershell [root@k8s-master01 ~]# curl -g -6 [2005::2c47] Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

~~~ ![image-20230324202324615](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324202324615.png) ![image-20230324202133124](../../img/kubernetes/kubernetes_ipv4_and_ipv6/image-20230324202133124.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_kafka.md ================================================ # kubernetes云原生中间件上云部署 kafka Apache Kafka是一种流行的分布式流式消息平台。Kafka生产者将数据写入分区主题，这些主题通过可配置的副本存储到broker群集上。 消费者来消费存储在broker的分区生成的数据。 # 一、环境说明 - storageclass - ingress # 二、kafka部署及部署验证 ~~~powershell # vim kafka.yaml ~~~ ~~~powershell # cat kafka.yaml apiVersion: v1 kind: Service metadata: name: kafka-svc labels: app: kafka-app spec: clusterIP: None ports: - name: '9092' port: 9092 protocol: TCP targetPort: 9092 selector: app: kafka-app --- apiVersion: apps/v1 kind: StatefulSet metadata: name: kafka labels: app: kafka-app spec: serviceName: kafka-svc replicas: 3 selector: matchLabels: app: kafka-app template: metadata: labels: app: kafka-app spec: containers: - name: kafka-container image: doughgle/kafka-kraft ports: - containerPort: 9092 - containerPort: 9093 env: - name: REPLICAS value: '3' - name: SERVICE value: kafka-svc - name: NAMESPACE value: default - name: SHARE_DIR value: /mnt/kafka - name: CLUSTER_ID value: oh-sxaDRTcyAr6pFRbXyzA - name: DEFAULT_REPLICATION_FACTOR value: '3' - name: DEFAULT_MIN_INSYNC_REPLICAS value: '2' volumeMounts: - name: data mountPath: /mnt/kafka - name: localtime mountPath: /etc/localtime volumes: - name: localtime hostPath: path: /etc/localtime type: '' volumeClaimTemplates: - metadata: name: data spec: accessModes: - "ReadWriteOnce" storageClassName: nfs-client resources: requests: storage: "1Gi" ~~~ ~~~powershell # kubectl apply -f kafka.yaml ~~~ ~~~powershell # kubectl get pods NAME READY STATUS RESTARTS AGE kafka-0 1/1 Running 1 (2m4s ago) 4m22s kafka-1 1/1 Running 0 3m22s kafka-2 1/1 Running 0 2m9s ~~~ ~~~powershell # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kafka-svc ClusterIP None 9092/TCP 4m42s ~~~ # 三、kafka应用测试 ~~~powershell 创建客户端pod # kubectl run kafka-client --rm -it --image bitnami/kafka:3.1.0 -- bash ~~~ ~~~powershell 进入客户端pod I have no name!@kafka-client:/$ ls /opt/bitnami/kafka/bin/ connect-distributed.sh kafka-consumer-perf-test.sh kafka-producer-perf-test.sh kafka-verifiable-consumer.sh connect-mirror-maker.sh kafka-delegation-tokens.sh kafka-reassign-partitions.sh kafka-verifiable-producer.sh connect-standalone.sh kafka-delete-records.sh kafka-replica-verification.sh trogdor.sh kafka-acls.sh kafka-dump-log.sh kafka-run-class.sh windows kafka-broker-api-versions.sh kafka-features.sh kafka-server-start.sh zookeeper-security-migration.sh kafka-cluster.sh kafka-get-offsets.sh kafka-server-stop.sh zookeeper-server-start.sh kafka-configs.sh kafka-leader-election.sh kafka-storage.sh zookeeper-server-stop.sh kafka-console-consumer.sh kafka-log-dirs.sh kafka-streams-application-reset.sh zookeeper-shell.sh kafka-console-producer.sh kafka-metadata-shell.sh kafka-topics.sh kafka-consumer-groups.sh kafka-mirror-maker.sh kafka-transactions.sh ~~~ ~~~powershell 查看默认存在的topic I have no name!@kafka-client:/opt/bitnami/kafka/bin$ kafka-topics.sh --list --bootstrap-server kafka-svc.default.svc.cluster.local:9092 __consumer_offsets test ~~~ ~~~powershell 创建topic I have no name!@kafka-client:/opt/bitnami/kafka/bin$ kafka-topics.sh --bootstrap-server kafka-svc.default.svc.cluster.local:9092 --topic test01 --create --partitions 3 --replication-factor 2 ~~~ ~~~powershell 创建数据生产者，添加数据 I have no name!@kafka-client:/opt/bitnami/kafka/bin$ ./kafka-console-producer.sh --topic test --request-required-acks all --bootstrap-server kafka-svc.default.svc.cluster.local:9092 >hello world ~~~ ~~~powershell 在当前终端或另一个终端中创建数据消费者，消费数据 I have no name!@kafka-client:/opt/bitnami/kafka/bin$ kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka-svc.default.svc.cluster.local:9092 ~~~ ~~~powershell 查看默认test topic相关描述信息 I have no name!@kafka-client:/opt/bitnami/kafka/bin$ kafka-topics.sh --describe --topic test --bootstrap-server kafka-svc.default.svc.cluster.local:9092 Topic: test TopicId: TkbmiTw8S7Om3eVK1LwapQ PartitionCount: 1 ReplicationFactor: 3 Configs: min.insync.replicas=2,segment.bytes=1073741824 Topic: test Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 ~~~ ~~~powershell 查看test01 topic相关描述信息 I have no name!@kafka-client:/opt/bitnami/kafka/bin$ kafka-topics.sh --describe --topic test01 --bootstrap-server kafka-svc.default.svc.cluster.local:9092 Topic: test01 TopicId: JspG5aMhSyewmCWvUaE5ZQ PartitionCount: 3 ReplicationFactor: 2 Configs: min.insync.replicas=2,segment.bytes=1073741824 Topic: test01 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: test01 Partition: 1 Leader: 2 Replicas: 2,0 Isr: 2,0 Topic: test01 Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1 ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_karmada.md ================================================ # karmada实现k8s集群联邦 # 一、karmada介绍 ## 1.1 karmada是什么 Karmada(Kubernetes Armada)是一个Kubernetes管理系统，使您能够跨多个Kubernetes集群和云运行云原生应用程序，而无需更改您的应用程序。通过使用Kubernetes原生API并提供高级调度功能，Karmada实现了真正开放的多云Kubernetes。 Karmada旨在为多云和混合云场景中的多集群应用程序管理提供自动化功能，具有集中式多云管理，高可用性，故障恢复和流量调度等关键功能。 官方网站：https://karmada.io/ 代码地址：https://github.com/karmada-io/karmada ## 1.2 karmada诞生背景 Karmada项目由华为云、工商银行、小红书、中国一汽等8家企业联合发起，沉淀了各企业在多云管理领域的丰富积累，于2021年4月25日在华为开发者大会2021(HDC.Cloud)上正式宣布开源。同年9月，Karmada项目正式捐赠给云原生计算基金会CNCF，成为CNCF首个多云容器编排项目。Karmada项目的加入，也将CNCF的云原生版图进一步扩展至分布式云领域。 CNCF总经理Priyanka Sharma对此表示：“华为一直是云原生社区与开发者生态的重要参与者，Karmada对所有企业构建多云业务架构至关重要，希望未来CNCF与华为云继续密切合作，持续帮助广大云原生开发者。” Karmada自开源以来受到了广泛的关注和支持，目前在代码托管平台上已有超过30家大型企业/机构/高校参与社区开发及贡献。 ## 1.3 karmada优势 Karmada结合了华为云容器平台MCP以及Kubernetes Federation核心实践，并融入了众多新技术：包括Kubernetes原生API支持、多层级高可用部署、多集群自动故障迁移、多集群应用自动伸缩、多集群服务发现等，并且提供原生Kubernetes平滑演进路径，让基于Karmada的多云方案无缝融入云原生技术生态，为企业提供从单集群到多云架构的平滑演进方案。Karmada的优势主要有以下几点： 1、K8s原生API兼容 - 既有应用配置及基础设施无需改造，由单集群架构平滑升级到多集群(多云)架构。 - 无缝集成Kubernetes现有工具链生态。 2、开箱即用 - 面向多场景的内置策略集，包括两地三中心、同城双活、异地容灾等。 - 支持应用的跨集群上的自动伸缩、故障迁移和负载均衡。 3、避免供应商锁定 - 与主流云提供商集成。 - 自动分配，跨集群迁移。 - 不受专有供应商编排的约束。 4、集中管理 - 与位置无关的群集管理。 - 支持公有云、本地或边缘的集群。 5、富有成效的多集群调度策略 - 多集群亲和性调度、应用跨集群拆分、资源重新平衡。 - 多维度多层次的高可用部署：区域/可用区/集群/供应商等。 6、开放和中立 - 由互联网、金融、制造、电信、云提供商等共同发起。 - 使用CNCF实现开放式治理的目标。 ## 1.4 karmada架构 ![](../../img/kubernetes/kubernetes_karmada/karmada架构.png) ![image-20230107171852138](../../img/kubernetes/kubernetes_karmada/image-20230107171852138.png) # 二、karmadactl安装 ![image-20221226111731872](../../img/kubernetes/kubernetes_karmada/image-20221226111731872.png) ![image-20221226111803270](../../img/kubernetes/kubernetes_karmada/image-20221226111803270.png) ![image-20221226111842928](../../img/kubernetes/kubernetes_karmada/image-20221226111842928.png) ![image-20221226111919521](../../img/kubernetes/kubernetes_karmada/image-20221226111919521.png) ![image-20221226111952632](../../img/kubernetes/kubernetes_karmada/image-20221226111952632.png) abc ~~~powershell [root@k8s-master01 ~]# wget https://github.com/karmada-io/karmada/releases/download/v1.4.1/karmadactl-linux-amd64.tgz ~~~ ~~~powershell [root@k8s-master01 ~]# tar xf karmadactl-linux-amd64.tgz ~~~ ~~~powershell [root@k8s-master01 ~]# mv karmadactl /usr/local/bin/ ~~~ # 三、karmada部署 ~~~powershell [root@k8s-master01 ~]# cat karmada.sh karmadactl init \ --namespace='karmada-system' \ --port 30000 \ --etcd-image='registry.k8s.io/etcd:3.5.6-0' \ --etcd-pvc-size='10Gi' \ --etcd-storage-mode='PVC' \ --storage-classes-name='nfs-client' \ --etcd-replicas=1 \ --karmada-aggregated-apiserver-replicas=1 \ --karmada-apiserver-replicas=1 \ --karmada-controller-manager-replicas=1 \ --karmada-kube-controller-manager-replicas=1 \ --karmada-scheduler-replicas=1 \ --karmada-webhook-replicas=1 \ --karmada-aggregated-apiserver-image='docker.io/karmada/karmada-aggregated-apiserver:latest' \ --karmada-apiserver-image='registry.k8s.io/kube-apiserver:v1.24.9' \ --karmada-controller-manager-image='docker.io/karmada/karmada-controller-manager:latest' \ --karmada-kube-controller-manager-image='registry.k8s.io/kube-controller-manager:v1.24.9' \ --karmada-scheduler-image='docker.io/karmada/karmada-scheduler:latest' \ --karmada-webhook-image='docker.io/karmada/karmada-webhook:latest' ~~~ ![image-20230107123014952](../../img/kubernetes/kubernetes_karmada/image-20230107123014952.png) ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n karmada-system NAME READY STATUS RESTARTS AGE etcd-0 1/1 Running 0 13h karmada-agent-7d85f6cc69-x957m 1/1 Running 0 13h karmada-aggregated-apiserver-6577d5769f-h8nmp 1/1 Running 0 13h karmada-apiserver-664446c84c-l5t26 1/1 Running 0 13h karmada-controller-manager-776f8fcbff-fwjqv 1/1 Running 0 13h karmada-scheduler-55dfc748b7-bw2t8 1/1 Running 0 13h karmada-webhook-6cc85d8f86-hr52h 1/1 Running 0 13h kube-controller-manager-7767d948d4-2s4vx 1/1 Running 0 13h ~~~ ~~~powershell Karmada is installed successfully. Register Kubernetes cluster to Karmada control plane. Register cluster with 'Push' mode Step 1: Use "karmadactl join" command to register the cluster to Karmada control plane. --cluster-kubeconfig is kubeconfig of the member cluster. (In karmada)~# MEMBER_CLUSTER_NAME=$(cat ~/.kube/config | grep current-context | sed 's/: /\n/g'| sed '1d') (In karmada)~# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config join ${MEMBER_CLUSTER_NAME} --cluster-kubeconfig=$HOME/.kube/config Step 2: Show members of karmada (In karmada)~# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get clusters Register cluster with 'Pull' mode Step 1: Use "karmadactl register" command to register the cluster to Karmada control plane. "--cluster-name" is set to cluster of current-context by default. (In member cluster)~# karmadactl register 192.168.10.160:30000 --token lw3mto.jv3qeoqtt7hm1vxl --discovery-token-ca-cert-hash sha256:11d5ff60b1fbeba0a4df9926157a4ac86d89ac649b5138a81bd763dbf6a7b7b0 Step 2: Show members of karmada (In karmada)~# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get clusters ~~~ # 四、使用karmadactl创建k8s集群联邦 ## 4.1 karmada push模式创建k8s集群联邦 ~~~powershell [root@k8s-master01 kubedir]# MEMBER_CLUSTER_NAME=$(cat ~/kubeconfigdir/config12 | grep current-context | sed 's/: /\n/g'| sed '1d') [root@k8s-master01 kubedir]# echo ${MEMBER_CLUSTER_NAME} member1 [root@k8s-master01 kubedir]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config join ${MEMBER_CLUSTER_NAME} --cluster-kubeconfig=$HOME/kubeconfigdir/config12 cluster(member1) is joined successfully ~~~ ~~~powershell [root@k8s-master01 kubedir]# kubectl --kubeconfig=config12 config use-context member2 Switched to context "member2". [root@k8s-master01 kubedir]# MEMBER_CLUSTER_NAME=$(cat ~/kubeconfigdir/config12 | grep current-context | sed 's/: /\n/g'| sed '1d') [root@k8s-master01 kubedir]# echo ${MEMBER_CLUSTER_NAME} member2 [root@k8s-master01 kubedir]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config join ${MEMBER_CLUSTER_NAME} --cluster-kubeconfig=$HOME/kubeconfigdir/config12 cluster(member2) is joined successfully ~~~ ~~~powershell [root@k8s-master01 kubedir]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get clusters NAME VERSION MODE READY AGE member1 v1.24.9 Push True 96s member2 v1.24.9 Push True 59s ~~~ ~~~powershell 删除member1集群 [root@k8s-master01 ~]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config unjoin member1 删除member2集群 [root@k8s-master01 ~]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config unjoin member2 ~~~ ## 4.2 karmada pull模式创建k8s集群联邦 ~~~powershell [root@k8s-master01 ~]# karmadactl register 192.168.10.160:30000 --token 2sh3py.li1p4o8gz4wqudrw --discovery-token-ca-cert-hash sha256:b1913d9f6410c2a4e19623223b141000a807cd2b15bbc3096ffc9a1dc39a5fe5 --cluster-name member1 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR]: /etc/karmada/pki/ca.crt already exists [preflight] Please check the above errors [root@k8s-master01 ~]# rm -rf /etc/karmada/pki/ca.crt [root@k8s-master01 ~]# karmadactl register 192.168.10.160:30000 --token 2sh3py.li1p4o8gz4wqudrw --discovery-token-ca-cert-hash sha256:b1913d9f6410c2a4e19623223b141000a807cd2b15bbc3096ffc9a1dc39a5fe5 --cluster-name member1 [preflight] Running pre-flight checks [prefligt] All pre-flight checks were passed [karmada-agent-start] Waiting to perform the TLS Bootstrap [karmada-agent-start] Waiting to construct karmada-agent kubeconfig [karmada-agent-start] Waiting the necessary secret and RBAC [karmada-agent-start] Waiting karmada-agent Deployment W0107 13:21:28.781319 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:29.784425 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:30.785356 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:31.785006 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:32.784396 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:33.784441 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:34.784517 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:35.785503 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:36.784929 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:37.784301 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating W0107 13:21:38.785135 28891 check.go:52] pod: karmada-agent-5dd59f9696-5tt82 not ready. status: ContainerCreating I0107 13:21:39.784795 28891 check.go:49] pod: karmada-agent-5dd59f9696-5tt82 is ready. status: Running cluster(member1) is joined successfully ~~~ ~~~powershell [root@k8s-master01 ~]# karmadactl register 192.168.10.160:30000 --token 2sh3py.li1p4o8gz4wqudrw --discovery-token-ca-cert-hash sha256:b1913d9f6410c2a4e19623223b141000a807cd2b15bbc3096ffc9a1dc39a5fe5 --cluster-name member2 [preflight] Running pre-flight checks [prefligt] All pre-flight checks were passed [karmada-agent-start] Waiting to perform the TLS Bootstrap [karmada-agent-start] Waiting to construct karmada-agent kubeconfig [karmada-agent-start] Waiting the necessary secret and RBAC [karmada-agent-start] Waiting karmada-agent Deployment W0107 13:21:44.795888 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:45.798850 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:46.798943 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:47.800993 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:48.799594 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:49.799841 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:50.799034 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:51.799729 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating W0107 13:21:52.801867 28340 check.go:52] pod: karmada-agent-76b7857fb4-zdnc9 not ready. status: ContainerCreating I0107 13:21:53.799843 28340 check.go:49] pod: karmada-agent-76b7857fb4-zdnc9 is ready. status: Running cluster(member2) is joined successfully ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get clusters NAME VERSION MODE READY AGE member1 v1.24.9 Pull True 119s member2 v1.24.9 Pull True 104s ~~~ # 五、利用karmada使用k8s集群联邦部署应用 ## 5.1 部署应用 ~~~powershell 准备nginx应用部署描述文件 [root@k8s-master01 nginxdir]# cat n1.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx1 # deployment名 spec: replicas: 1 # 副本集,deployment里使用了replicaset selector: matchLabels: app: nginx1 # 匹配的pod标签,表示deployment和rs控制器控制带有此标签的pod template: # 代表pod的配置模板 metadata: labels: app: nginx1 # pod的标签 spec: containers: # 以下为pod里的容器定义 - name: nginx1 image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 ~~~ ~~~powershell 不能这样部署 [root@k8s-master01 nginxdir]# kubectl apply -f n1.yaml deployment.apps/nginx1 created ~~~ ~~~powershell 像下面一样部署，注意这个位置部署的是模板，并没有实际运行在任何集群中。 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config apply -f n1.yaml deployment.apps/nginx1 created ~~~ ~~~powershell 查看已部署应用，没有发现有应用被创建 [root@k8s-master01 nginxdir]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config get pods NAME CLUSTER READY STATUS RESTARTS AGE nfs-client-provisioner-dfd8488d7-hhmwh member1 1/1 Running 3 (24h ago) 30h nfs-client-provisioner-dfd8488d7-rtmdj member2 1/1 Running 2 (24h ago) 30h ~~~ ~~~powershell 准备传播策略文件 [root@k8s-master01 pdir]# cat p1.yaml apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: example-policy # The default namespace is `default`. spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: nginx1 # If no namespace is specified, the namespace is inherited from the parent object scope. placement: clusterAffinity: clusterNames: - member1 ~~~ ~~~powershell 执行传播策略文件 [root@k8s-master01 pdir]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config apply -f p1.yaml propagationpolicy.policy.karmada.io/example-policy created ~~~ ~~~powershell 使用kubectl查看默认命名空间，有 [root@k8s-master01 pdir]# kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-dfd8488d7-hhmwh 1/1 Running 3 (25h ago) 30h nginx1-7bff55d4bd-lcbcn 1/1 Running 0 44s 查看部署deployment [root@k8s-master01 pdir]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config get deployment NAME READY UP-TO-DATE AVAILABLE AGE nginx1 1/1 1 1 3m9s ~~~ ~~~powershell 查看pod [root@k8s-master01 pdir]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config get pod NAME CLUSTER READY STATUS RESTARTS AGE nfs-client-provisioner-dfd8488d7-hhmwh member1 1/1 Running 3 (25h ago) 30h nginx1-7bff55d4bd-lcbcn member1 1/1 Running 0 2m40s nfs-client-provisioner-dfd8488d7-rtmdj member2 1/1 Running 2 (25h ago) 30h ~~~ ## 5.2 更新传播策略 ~~~powershell 更新传播策略至其它集群，此处为member2 [root@k8s-master01 pdir]# cat p2.yaml apiVersion: policy.karmada.io/v1alpha1 kind: PropagationPolicy metadata: name: example-policy spec: resourceSelectors: - apiVersion: apps/v1 kind: Deployment name: nginx1 placement: clusterAffinity: clusterNames: # Modify the selected cluster to propagate the Deployment. - member2 ~~~ ~~~powershell 执行更新后的传播策略文件 [root@k8s-master01 pdir]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config apply -f p2.yaml propagationpolicy.policy.karmada.io/example-policy configured ~~~ ~~~powershell 查看pod是否已在其它集群中运行，此处为member2 [root@k8s-master01 pdir]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config get pod NAME CLUSTER READY STATUS RESTARTS AGE nfs-client-provisioner-dfd8488d7-hhmwh member1 1/1 Running 3 (25h ago) 30h nfs-client-provisioner-dfd8488d7-rtmdj member2 1/1 Running 2 (25h ago) 30h nginx1-7bff55d4bd-fm7kj member2 1/1 Running 0 28s ~~~ ## 5.3 更新模板部署应用 ~~~powershell 更新应用部署模板，把replicas由1修改为2 [root@k8s-master01 nginxdir]# cat n1.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx1 # deployment名 spec: replicas: 2 # 副本集,deployment里使用了replicaset selector: matchLabels: app: nginx1 # 匹配的pod标签,表示deployment和rs控制器控制带有此标签的pod template: # 代表pod的配置模板 metadata: labels: app: nginx1 # pod的标签 spec: containers: # 以下为pod里的容器定义 - name: nginx1 image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 ~~~ ~~~powershell 执行应用部署文件 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /etc/karmada/karmada-apiserver.config apply -f n1.yaml deployment.apps/nginx1 configured ~~~ ~~~powershell 查看pod情况，发现pod由1个变为2个 [root@k8s-master01 nginxdir]# karmadactl --kubeconfig /etc/karmada/karmada-apiserver.config get pod NAME CLUSTER READY STATUS RESTARTS AGE nfs-client-provisioner-dfd8488d7-hhmwh member1 1/1 Running 3 (25h ago) 30h nfs-client-provisioner-dfd8488d7-rtmdj member2 1/1 Running 2 (25h ago) 30h nginx1-7bff55d4bd-fm7kj member2 1/1 Running 0 8m46s nginx1-7bff55d4bd-llhv2 member2 1/1 Running 0 5m12s ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_kubeconfig.md ================================================ # 使用kubeconfig管理多集群方法 # 一、 k8s集群部署 > 准备多套k8s集群，可以使用kubeadm部署多套k8s集群。 ~~~powershell # wget https://github.com/labring/sealos/releases/download/v4.1.7/sealos_4.1.7_linux_amd64.tar.gz && \ tar -zxvf sealos_4.1.7_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin ~~~ ~~~powershell [root@k8s-master01 ~]# cat sealos-install.sh sealos run labring/kubernetes:v1.25.0 labring/helm:v3.8.2 labring/calico:v3.24.1 \ --masters 192.168.10.140 \ --nodes 192.168.10.141,192.168.10.142 -p centos ~~~ ~~~powershell [root@k8s-master01 ~]# cat sealos-install.sh sealos run labring/kubernetes:v1.25.0 labring/helm:v3.8.2 labring/calico:v3.24.1 \ --masters 192.168.10.160 \ --nodes 192.168.10.161,192.168.10.162 -p centos ~~~ # 二、K8S多集群管理 ## 2.1 获取多集群kubeconfig文件 ~~~powershell 创建文件存储目录 [root@k8s-master01 ~]# mkdir kubeconfigdir ~~~ ~~~powershell 获取第一个集群kubeconfig文件 [root@k8s-master01 ~]# cp /root/.kube/config kubeconfigdir/config1 ~~~ ~~~powershell 获取第二个集群kubeconfig文件 [root@k8s-master01 ~]# scp 192.168.10.160:/root/.kube/config kubeconfigdir/config2 ~~~ ## 2.2 合并多集群kubeconfig文件 > 可以选择手动合并，也可以选择自动全并。本案例为手动合并。 > > 自动合并命令：# KUBECONFIG=./config1:./config2 kubectl config view --flatten > ./configX，前提需要修改config文件。 ~~~powershell 合并后kubeconfig文件如下： [root@k8s-master01 kubeconfigdir]# cat config12 apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1ERXdOVEExTWpJek1sb1hEVE16TURFd01qQTFNakl6TWxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTmJjClFkK0ttSkd2NDNJRG9yWXY1UmRSNkNVTkpvUE1CY3kveDd2U3FIM1dEbWJsWnM3NDMvbVJRUDE1NDNRSmlsbDYKNmV1WjQrSDBsZXVsRWNjQU9mN2dPOG5EaDJSTEQ0ZHc4bk51L2JWMnRnRjc5b1FSTUtDczN2TWwzcEwzZmZiVQpweHdQVXAzTktwb1pZZndNVTJvazVIZGxwbkdqQzVLRnkvb2FNcmNrUWFNQjhjaFhhckxoSDBCVHZXS29sWnUxCjJPa2VvRDJKRTl1czVDRkt4MDMxWmtjQVArbVpjM2JYejI0amZYOUI1L05BR3c2Vmx3bjNCR1dHR1pJczd4Q3cKbjdpS1dNMTJGZCsyM1Z0dUtMWDBzdnJsUkQxcHdFdUxvRVVGdHN0dEJqZmN6bHJtQkhTWEE4SHFRazE1bVRCWAo5U0ZSc2NpOEVHaTR1MS9UN0JFQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKRFVvYStGQklSeXduaHFPUDViWk1ZSmdWZEJNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRWc3WmFMLzJNWE9BNi9qem5wTApqVTA4Z2lBV01Jc2pzWk10V3R2WFNEQ3g0UE10LzAzVENXb0hlYi9BTSszMHpkMXNqOWE5V281akNkdFB0WGtQClFIa3gyNWZheVpwVndiN0d1Qk8xNSsxdUFjRWtJWmhKNnhuT25sZlIyUkF2TTVodXV6RUtTeHRoZ3Nlay9wYlAKeFBYWkdzQS90T3dEbm55L1ZpMzNFVjlLampybVY4d0xLR0htRHVhM05hWjh2aUtBeHVscmNITUZ1ZXloSFdaUQpiSlNyS1FZSDJUaGt3ekw4SHd6cEZ3SEQvVDRTTHpQRkF2VVc5ZkxTVm16V0N2Mkd6RzhvQWR0WHYrQ2YyOXZECmEyMFJvWnE1UDcwdnVZdGF5UXpmM0o2aHBMRGRvUFBYUVRmUVo0NlRJZmlKdjM3RTgycXVYQ0JSNUlvOEI5Z0wKWXE4PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.10.140:6443 name: member1 - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1ERXdOVEExTWpNeE5Gb1hEVE16TURFd01qQTFNak14TkZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT0JlCjFRUGhjY3ZPVGRUa1gySE1VOE53WXNiM01CWWNmbG5ERW1ENE9PRE1sNi8xdTRmYVhvc2RFMFNxUTlDYU5pamUKR09OU0lXc01zTlpmWVJ6Tk9mMlBlRXJyeGEwcE12RG5oNnEwSGtqVmQxdVZTNWhrbkNxY00rU0pnYUlhSXl5Zwp0RW00NFFWdFo2MGZhU1cxMFFWQXp5ZWJEYjJ3Wm1zSnhzRE5rbHprYzY4alVIK3N1dmxzMitjb1hnRWo1ditnCnlydjh2SjlPVklIdDlhd2VMU1h1YnhtZVpUOUZ6UDNRandkdTY3NjV3bFVFQ2E5YnZVWEY4MlVuYURnTWhVR1kKR2dqSVQrd09sN0ZGQVVrOFZXMm0xRnNaWlR6bmQwaDloRTI2MkZ1V2YzL29nZlcvbUIyMEo5ZzQ5Q0NoaVFGYQpCY0xUclJuZHhaWU5uZkdIalowQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZIblhUUk52VkVxU1dEWUpROTFROGt3Y0I4ZnFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS3lYMmJ4d2cySVYxUEE0OThaNApaU290VFljNFZMc3hvdUNwNUY0U2dGSVIrc0pEWVc5R1RTRFpXbW9aczlQaE4vOFpEdDFQVjJMdU1waG00YStkCm83OW9PME10OHhvMGNsdGQzaUZXWkNRK0lzcHJoNUVTd0VuSmdCYmpDWGxObCs4T256VUJaQVRtZWw4dnY4LysKektITDdTNE03K1NOclN4ZGE1VHkwU2M4Mi8wS0RoUmZDMWtOZkhFMStuNWw4d2JVa1BmNDJkanY1eEFac2pQUwo1d1VTOCs4QXBQM0ZWNkNhRmJBdUlRbXFUL1pDdG9GYUZVVGxwUEFnRjBPa1AzelVXaTBrQzR3eG5vUVRXaW4yCnhEYWkvdmRPU1Qyd2FnU3dKOXRZd3ZsQmg4ZmQwdmM3bDlvMGREeklBYlh4OTBvdkZ0bzVCZ0VsT0FrQ3hTTisKc0E0PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.10.160:6443 name: member2 contexts: - context: cluster: member1 user: member1-admin name: member1 - context: cluster: member2 user: member2-admin name: member2 current-context: member1 kind: Config preferences: {} users: - name: member1-admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJR3ljSnc3LytXNGt3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBeE1EVXdOVEl5TXpKYUZ3MHlOREF4TURVd05USXlNelJhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXl3NEFmd1RYQllzSUZkZkIKcU9tWWZMbEpzS2VLZDdOS1pSZEZ4TmU5ejB6dTVnc1Zudm9HWXlmdkJXdWxTMDZjcDFuTGdOR1ZWUVN4SndKUQo5blo4cEQyV3h2QkVBRkdBdWoxUWtLaXZnMDkwSG1zbFZHbVE2blhDNkxzTm1QdFRSOVdpWnpmWjQyK2t3SGRJCjM1SGlNeEk2aXIva1BoZHBPc3ZaRWZzUGl1cDZpVDF2d1VKbFR4V00zZVF3d2ZRYW1MSHdGYUhpNFZvOTl4TUkKQllsUktpa2FYMFluQkFGZ2RVcEZ6Rk53TnpTWnFoQy9KaE1qQyt1OTVnakx5SENZVXhWU2xBRDc0TDErb1l2WQppNmZkM0RhQWx5VjRZYXRabytxSFhlbW5RdGMveEhwR3gwNVIxdnBZQmVnNGVXL0lhUER0a29UNEtzR0kyNm9DCithczZaUUlEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JTUTFLR3ZoUVNFY3NKNGFqaitXMlRHQ1lGWApRVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBVGxwRlBzVW5qTGl4L0dINjMrZUY0ZEFYTmZvdW84bWl2cm1pCmVtSTUySzJZMHYzdlhBRkhHWi9GSDdML1VBMTMycmVGSlFHRmhBMHp1NnF5cUMzaHRWTnFsalQwQ3JkdjFVNnYKbE5KT2lVZnVRTXJkV3lDQ3BsdWxMREtQNnpXY0hpRXFacWx0UGVpVHZpdE5LcURvSXBwYXU3NUxCR2JGYVdxNgpESkRZZlBUL2JqM1RHV0V3VzEzWnczdjlnc3NzOVRSbHd6YzJTTmpOOHdSbktnaEh4N3dBSVlPTE51bzRXcUxwCldXdW5jYjJCRFZKQStUc0t1UTVxTUEvVHpLakVZdHR3S0VrTkdTTCt4eS9jRUIvanV4TnR0SXBkbnBhdWxJU3kKMlpuU2pxSkgrbzBxYjdzekJINlVLWVdscFYvZzVnSDlOb1RwVUxHSkdkSWZnaXRwdUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBeXc0QWZ3VFhCWXNJRmRmQnFPbVlmTGxKc0tlS2Q3TktaUmRGeE5lOXowenU1Z3NWCm52b0dZeWZ2Qld1bFMwNmNwMW5MZ05HVlZRU3hKd0pROW5aOHBEMld4dkJFQUZHQXVqMVFrS2l2ZzA5MEhtc2wKVkdtUTZuWEM2THNObVB0VFI5V2laemZaNDIra3dIZEkzNUhpTXhJNmlyL2tQaGRwT3N2WkVmc1BpdXA2aVQxdgp3VUpsVHhXTTNlUXd3ZlFhbUxId0ZhSGk0Vm85OXhNSUJZbFJLaWthWDBZbkJBRmdkVXBGekZOd056U1pxaEMvCkpoTWpDK3U5NWdqTHlIQ1lVeFZTbEFENzRMMStvWXZZaTZmZDNEYUFseVY0WWF0Wm8rcUhYZW1uUXRjL3hIcEcKeDA1UjF2cFlCZWc0ZVcvSWFQRHRrb1Q0S3NHSTI2b0MrYXM2WlFJREFRQUJBb0lCQVFDN3R3UUthTVVISU5LbQpyc0Vma0dRaDJZRWdTS0tmcWlYNmNwdFRNRWNPMzRaek1JZ2FZZldKc1I4c21hbERoemNYRnRJbEVwdkU3d04rCmxvdVdiVThvM3E4RzFwTm0zL0hyT2tmQ2s4ODl5elFEOHZXZHBjSU1ualZEeGJqNlZrMVZPVkJicjZ4RXI2OVYKSm5FK0RiVlpsVjU5YW94c1FtUkxzS3ZLRFpqK3g4SnFhWnhMZkp2eEJPbnBvZ3JpSHhHRjA4Sm5BMTBqZFNlMApnY1Q3dFA4Wk1IeVhiWDVsYkFYQWp3VXJnNTJ2YUFBVEZkOTNUYTkrdDRwcWt5OU93RmZHR0p2bm9VLzRIZ1hqCjVvN0lIdWcxbVNhbXYvSlVvcXRKZW54MzAxVGNNSkxqS1Zlck4wbm85MGExYnpNQThiVzBEaWpiYjl3R043bFkKM0xDdTFMdEJBb0dCQU04NE0rdUJ2c05KbE5pZU9Zd1I1UUh4TjJPV3hnQTBjYSt5S09DYjBsMUpCOE9EK0lhUQpmbVdKTUlLeEpjQUZva0EwT3E3MmFEUlhTeWorcS9hQ2VUUkxPOHFlVjJLSFo5TGRYOW9BNGhqUVFqVm5pbGZSCkJ1dlhKd040bHdLcWR4aVZqN3R2cU0yYmVnZTdRejN2STViQTdIcGNZSnJOZWpmdUdSMDJZcVBaQW9HQkFQcmEKejNjUGFyanN1dGQxK3VQUWdwN0Y1V3Fud05uSUhtWTQ2SVp0YnQ5VnNaN2tRSlJZaHNjdkhMaVhVMmhWQ244agpSOU5WNHNOejQ4azdDU0RpQXdaNFZBYUFOM1N1OFBaNHZ5RFZJZVllK0tVUnJkM1RuRk10NFptc0UvWGlyV29GCmtNSndiK2lLMVJGc1VCU0NTSVBnQjZQOXZ6R3NaV0NnaVczT1I4OXRBb0dBRyt6M0VrWHA3MmQ3SjdZckN6VmUKSjJUYWtoRS9uY2R6aXJuM3lFMDNqRnJMTVE0WDhBcUkvaVgrNDUzNytHVEorTjBSQzRNcGgrUTd2TXFWWWlNegpNbGp4TmQwZzZhWlYxNVQ5MWVOSWxROTczTGFYYmo0OU1JdE9OcW1Kc0ZKSXVvZHRWMVUwNm9DSmNZRkxEbzJyCnZpVkJ1VHU1eVNMbjFhSEF3SzhUbkFFQ2dZRUF2amR6ZitXWjJIWHh5L1d6ZEZJazZmNUgwMU0zSWl6a0dFRm8KMko1Y3AwOVVxNWFLL2JJUEtUU3BRN1BEMUdZLzJsNUhWWkpYckR2UmEwS3Z6bFp6VXRHbGJYU0dHSjJiTEZvdApHOWxocGh5d0VJTlNZdFhXUVNDV1pDK2V4eUhHdTVGU3pvM3gzZFNBY29DK1RIN3FPODJDSGJFSTdNSzc4TVJxCjBXL005aFVDZ1lFQXBWTWxQL24xQ2VKWW12RDZwanNCMy9VaE9UUDlUN0wrWk9LNnlMZEwyNzlqeVRBUm1URGcKOHhWSitZNjZLSXluS3ZFaDlmaGZRZDdtbHJyYmNSK0RwSVdFRXkxN3JQQWpjbkNhVWNZVDJXVGlYcjROWVlmUgpIakdMQ2tLb3hWci9aV2lXUUJ2UVRoemZUd0RRaWVCS2xpT0hoY3h3bnl6Q2dWdm5Wc3hWbDF3PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= - name: member2-admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJTytGWlFKV2RpYUF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBeE1EVXdOVEl6TVRSYUZ3MHlOREF4TURVd05USXpNVFphTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTV3OVB5R3EvMloxRnVtMkkKSkU0TmlONStwUnQyY01vV1RXdi9neTB0NmdMRE5yVTFnRFB1aWZ5SmhLM05veHdwWGJXTXY1dUQ5QzlwS3N2UwpjbW1IdUZQVFF6ZWpleGZuYkEwckhZSk9FK1krSWZhM1FZMTgzL0lJUXZGUGozMkJEdVVNZkNtcHFSbitZejVpCk1TVW1pWCtDelkxRC93REVJZ0FtOUZXTEV5cnkxZi8xMjhkVGRGU3JITFpab2xCbG00aWhIUUMxV0JBVFFsVlMKN2pMMmFNTGNPTDl5OXJGMkNVY3pUTTA1K2R2Z2ZFUGw4aVlVRVNieFRtVUFETmRBS2lGSitvZmFrc0RFaGxpMwpNVXNHNitLdFZVR2E0aTJJc3pYMTlNOHFZYWFNdHZyaDFlTHBaTld5NGwwSUVhd3NlaS9ZVm5OUE5hb0Jld3FnCnJ0bHFwd0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JSNTEwMFRiMVJLa2xnMkNVUGRVUEpNSEFmSAo2akFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBcHJGdkNIS0k2WHlKbzZ2cTRSUmRqdUFjbHhBeTBQV2Fud2VZCnRKR2lKUlFncXJqaUNNQkZDSEdDcjZnTHY3RzUwSFNSL0UwbnpZWGdzQ2FKUFN4U2ZkSFJiRVRRMldWT2ExSVEKN1hUMnUzT1l4bDhSWXkwR2lNMWRyakxYOHQ4K0xXSVhYdFEzVGVkaFJwNEtvR2J5Z3BuVlJlKzNCOXhsbDFZQQoxRWxJL1QxV0t5Qk0waEdhdHB1Z2l2aXFaNFY2RVJkMzBtUmhmT1dqem5rNi9GTmFIRTRtL1J6cFVtNWZPbURBClVSWElBa05JZlZudENRRkVuejhDWmlaMkVyNG5HV0pwOWV1UU9Scyt5eUlGdkFWZjd5REYrUGV6SDVIUCtBL3YKbktGd0JxTUdyaHN0bEEyUHdvSE9wMnVQa01MKzdSSENQYWRMYVltejNRQWFCdGNyRWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBNXc5UHlHcS8yWjFGdW0ySUpFNE5pTjUrcFJ0MmNNb1dUV3YvZ3kwdDZnTEROclUxCmdEUHVpZnlKaEszTm94d3BYYldNdjV1RDlDOXBLc3ZTY21tSHVGUFRRemVqZXhmbmJBMHJIWUpPRStZK0lmYTMKUVkxODMvSUlRdkZQajMyQkR1VU1mQ21wcVJuK1l6NWlNU1VtaVgrQ3pZMUQvd0RFSWdBbTlGV0xFeXJ5MWYvMQoyOGRUZEZTckhMWlpvbEJsbTRpaEhRQzFXQkFUUWxWUzdqTDJhTUxjT0w5eTlyRjJDVWN6VE0wNStkdmdmRVBsCjhpWVVFU2J4VG1VQUROZEFLaUZKK29mYWtzREVobGkzTVVzRzYrS3RWVUdhNGkySXN6WDE5TThxWWFhTXR2cmgKMWVMcFpOV3k0bDBJRWF3c2VpL1lWbk5QTmFvQmV3cWdydGxxcHdJREFRQUJBb0lCQUFNRHNSL0ZQTUdEQ3NEVwo0dnJPUmVEdVBpcTdRLzFPdGFIRzhldHRNSGNvR0JIanBWSUoyMmZUY3B2WGhLSkhJTWNITWxIaG5vUVdCa0kzCnJJUXZta1N1VzBnNk5wakpoQXhsMDVVcitRYkxieTRVUU9uTEJjRUtNRTluUklsenNyWldDS2FxQ0h1YlNqVlQKUUFheUhHR1kxMFVGUGNqYTRyUXEybGMvc25QS2lrMFpkcEZMZlduWUljTTdKSU5TTHZTRkJVWGxHV0hnZi9UbgpBSWNQMFpuWkZ0cHZhdS9MMXk4UUNzY2ZPVWo3YjJFbG5FU2xHRFdnUVcyZDBrdXppQ2dxRmtVZm5HdWVDNExuCnlySUVRSzBSM081TUt4ODM2TWN3TytTcDgrUmtMeitKYlUzTXprelhLVGZBSU5haVFuZ2Q5Zy9JaVpnbUtjUjQKMkFKSFBKa0NnWUVBNjVWTkdORURKaDlsWWlwQWZ6NHVRa1AydmN1VExaajNuVjU0OXlucERZQnFrWnFxSFhXYwpERm1OTWhpdm5VNnprSE02KzY1RWZGOTdQZTkvUmM2QnJrcUhlYWsvcjY2ajdVSVlPVklvR1FvMVZRQ2MyR0NhCnZIVVluL01rOHBRUDdwNHpyUmFxVWFoZm1Jc1JhTlVlZytWMnZsa1krQkNNblRQSTRvRllGSnNDZ1lFQSt4V24KU1RRUmczYWE5N0pwOUFDRTNNVGRHbStPcnExM0lDTU12Q2pKM0c2VFhaOS8wSC96NG1FMlQwMUVIVnIvZStBRQptVmk3VDU1ZWIzRVdRbElsdExKUTdQK1R3YnBheUFMamZqcDVRTGtZUEcvY1UwNnAycFhXOHU0L3dsUk5YbWVmCklkRGFpOTJPNzltOG4rZ1d3b1JzL3g1b21XamE4azkxR1NuV3RPVUNnWUVBa3dIWDJtU1RVbmJGRFR6UWdwYUsKeDA4aDZjM2ZTZFRxcjRrRWN6Zno5ampzUjIvOE4rWHNPc2luRTF2VU9wV2g5OEh3VEoyeW51bjJQZS8xdTluaAprcUZ2YUx2MHdleDQzdFVmeUtVNzRHUStZNHkrVTBmMVJ5VEsrUVVCU1Y2YmtvdW1NNXl4SzhPbDQ4cmtVa2FyCmhDTHN0bHRpK1dsYVZiNjYzSjFhR1lFQ2dZQjU4MWR2OTZrMTkrcG11akk1Ly9LSUk5bmNHQ2p0OEhTMm1DOTgKU0RkYktCM05VRVhOS0FoMWdJL1hUb2p4MVJ2WHQ4T2tFM1BPeFBYTEhOc1oxVHBaSEc2djNhYVBab3JuTUhmNwovRHllVWdoU2VtWkIxQ2d0Y2ErWUNGM1JiZzZ4OXBSVTRWTDBzZVRWM0NTQWFrSjdzY1FhMlZNbWg4WW9BSzI0ClRoanBmUUtCZ0FSUWRsTTZWbVVDTE9iZTgvTFNCckxjbEg1enFwdkx2M2VEeDhuN1BKdTl5aC9jbnAvaFd0ZU0Kd1cyaTUzUTVNUnFManJ5Zk4ra0E0c3BTVk45cTVHVnRYSWE5M1JkOENsc045UEMyMG1OK0pYOUhaUThLb015MQpVNmMybVlyVFhpYm9yNVBPNlB0aUdRaTNkREZWVDhzTnhZd0FzenliankxYjZUQmVDc3lMCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== ~~~ ~~~powershell 注：上述kubeconfig文件中clustername、username、contextname都可以根据企业实际情况进行替换。 ~~~ ~~~powershell 查看可管理的集群 [root@k8s-master01 kubeconfigdir]# kubectl --kubeconfig=config12 config get-clusters NAME member2 member1 ~~~ ~~~powershell 查看可管理的集群上下文 [root@k8s-master01 kubeconfigdir]# kubectl --kubeconfig=config12 config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE member1 member1 member1-admin * member2 member2 member2-admin ~~~ ~~~powershell 查看当前上下文 [root@k8s-master01 kubeconfigdir]# kubectl --kubeconfig=config12 config current-context member2 ~~~ ~~~powershell 切换上下文 [root@k8s-master01 kubeconfigdir]# kubectl --kubeconfig=config12 config use-context member1 Switched to context "member1". ~~~ ~~~powershell 查看切换后当前上下文 [root@k8s-master01 kubeconfigdir]# kubectl --kubeconfig=config12 config current-context member1 ~~~ ## 2.3 使用合并后kubeconfig管理多集群 ~~~powershell 查看member1集群kube-system命名空间中pod信息 [root@k8s-master01 ~]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member1 Switched to context "member1". [root@k8s-master01 ~]# kubectl --kubeconfig /root/kubeconfigdir/config12 config current-context member1 [root@k8s-master01 ~]# kubectl --kubeconfig /root/kubeconfigdir/config12 get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-57575c5f89-4nl9k 1/1 Running 3 (39h ago) 45h coredns-57575c5f89-9hjsn 1/1 Running 3 (39h ago) 45h etcd-k8s-master01 1/1 Running 3 (39h ago) 45h kube-apiserver-k8s-master01 1/1 Running 5 (39h ago) 45h kube-controller-manager-k8s-master01 1/1 Running 3 (39h ago) 45h kube-proxy-4wp45 1/1 Running 3 (39h ago) 45h kube-proxy-pxz6p 1/1 Running 3 (39h ago) 45h kube-proxy-wvx9g 1/1 Running 3 (39h ago) 45h kube-scheduler-k8s-master01 1/1 Running 3 (39h ago) 45h ~~~ ~~~powershell 查看member2集群kube-system命名空间中pod信息 [root@k8s-master01 ~]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member2 Switched to context "member2". [root@k8s-master01 ~]# kubectl --kubeconfig /root/kubeconfigdir/config12 config current-context member2 [root@k8s-master01 ~]# kubectl --kubeconfig /root/kubeconfigdir/config12 get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-57575c5f89-782l5 1/1 Running 3 (14h ago) 45h coredns-57575c5f89-9tjcn 1/1 Running 3 (14h ago) 45h etcd-k8s-master01 1/1 Running 3 (14h ago) 45h kube-apiserver-k8s-master01 1/1 Running 4 (12m ago) 45h kube-controller-manager-k8s-master01 1/1 Running 3 (14h ago) 45h kube-proxy-5nglp 1/1 Running 3 (14h ago) 45h kube-proxy-d7q6c 1/1 Running 3 (14h ago) 45h kube-proxy-dhr9d 1/1 Running 3 (14h ago) 45h kube-scheduler-k8s-master01 1/1 Running 3 (14h ago) 45h ~~~ ## 2.4 kubectx ### 2.4.1 下载 ![image-20230413124940258](../../img/kubernetes/kubernetes_kubeconfig/image-20230413124940258.png) ![image-20230413124955713](../../img/kubernetes/kubernetes_kubeconfig/image-20230413124955713.png) ![image-20230413125024563](../../img/kubernetes/kubernetes_kubeconfig/image-20230413125024563.png) ![image-20230413125115514](../../img/kubernetes/kubernetes_kubeconfig/image-20230413125115514.png) ~~~powershell # wget https://github.com/ahmetb/kubectx/releases/download/v0.9.4/kubectx ~~~ ### 2.4.2 安装及使用 ~~~powershell # ls kubectx # chmod +x kubectx # mv kubectx /usr/local/bin/ ~~~ ~~~powershell # cp configX /root/.kube/config ~~~ ~~~powershell # kubectx member1 member2 ~~~ ~~~powershell 如果默认在member1集群，使用下面命令则会自动切换到member2 # kubectx - Switched to context "member2". ~~~ ~~~powershell 再次使用kubectx将切换到member1 # kubectx - Switched to context "member1". ~~~ # 三、使用kubeconfig实现在不同k8s集群中部署应用 ## 3.1 在member1集群中部署应用 ~~~powershell 查看部署描述文件 [root@k8s-master01 nginxdir]# cat n1.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx1 # deployment名 spec: replicas: 2 # 副本集,deployment里使用了replicaset selector: matchLabels: app: nginx1 # 匹配的pod标签,表示deployment和rs控制器控制带有此标签的pod template: # 代表pod的配置模板 metadata: labels: app: nginx1 # pod的标签 spec: containers: # 以下为pod里的容器定义 - name: nginx1 image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 ~~~ ~~~powershell 确认所在目录 [root@k8s-master01 nginxdir]# pwd /root/nginxdir 确认文件是否存在 [root@k8s-master01 nginxdir]# ls n1.yaml 查看当前上下文 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config current-context member2 切换当前上下文 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member1 Switched to context "member1". 部署nginx应用 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 apply -f n1.yaml deployment.apps/nginx1 created 查看部署的nginx对应的pod [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 get pods NAME READY STATUS RESTARTS AGE nginx1-7bff55d4bd-nwztt 1/1 Running 0 13s nginx1-7bff55d4bd-rvhwp 0/1 ContainerCreating 0 13s 切换上下文 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member2 Switched to context "member2". 确认member2集群中是否有部署nginx应用，结果：没有 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 get pods ~~~ ## 3.2 在member2集群中部署应用 ~~~powershell 查看部署描述文件 [root@k8s-master01 nginxdir]# cat n1.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx1 # deployment名 spec: replicas: 2 # 副本集,deployment里使用了replicaset selector: matchLabels: app: nginx1 # 匹配的pod标签,表示deployment和rs控制器控制带有此标签的pod template: # 代表pod的配置模板 metadata: labels: app: nginx1 # pod的标签 spec: containers: # 以下为pod里的容器定义 - name: nginx1 image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 ~~~ ~~~powershell 切换上下文 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member2 Switched to context "member2". 确认当前上下文 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config current-context member2 执行nginx应用部署 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 apply -f n1.yaml deployment.apps/nginx1 created 确认nginx应用部署对应的pod是否运行，结果：运行。 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-dfd8488d7-rtmdj 1/1 Running 3 (14h ago) 45h nginx1-7bff55d4bd-25vkz 1/1 Running 0 8s nginx1-7bff55d4bd-lfprt 1/1 Running 0 8s ~~~ ~~~powershell 删除不同集群中已部署的应用 删除member2集群中的应用 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member2 Switched to context "member2". [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 delete -f n1.yaml deployment.apps "nginx1" deleted 删除member1集群中的应用 [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 config use-context member1 Switched to context "member1". [root@k8s-master01 nginxdir]# kubectl --kubeconfig /root/kubeconfigdir/config12 delete -f n1.yaml deployment.apps "nginx1" deleted ~~~ # 四、 持久化存储准备 > 提前准备好NFS服务，主要为后续k8s联邦做准备。 ~~~powershell 下载NFS做为后端持久化存储部署文件 # for file in class.yaml deployment.yaml rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/$file ; done ~~~ ~~~powershell 替换的镜像：registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0 ~~~ ~~~powershell 本次NFS服务器IP地址及共享目录： 192.168.10.166 /netshare ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_kubesphere.md ================================================ # 在Linux上以all in one模式安装kubernetes & kubesphere 于刚接触 KubeSphere 并想快速上手该容器平台的用户，All-in-One 安装模式是最佳的选择，它能够帮助您零配置快速部署 KubeSphere 和 Kubernetes。 # 一、Linux主机准备 若要以 All-in-One 模式进行安装，您仅需参考以下对机器硬件和操作系统的要求准备一台主机。 >如果您的机器至少有 8 核 CPU 和 16 GB 内存，则建议启用所有组件。 ## 1.1 硬件推荐配置 | 操作系统 | 最低配置 | | :----------------------------------------------------- | :---------------------------------- | | **Ubuntu** *16.04*, *18.04* | 2 核 CPU，4 GB 内存，40 GB 磁盘空间 | | **Debian** *Buster*, *Stretch* | 2 核 CPU，4 GB 内存，40 GB 磁盘空间 | | **CentOS** *7.x* | 2 核 CPU，4 GB 内存，40 GB 磁盘空间 | | **Red Hat Enterprise Linux 7** | 2 核 CPU，4 GB 内存，40 GB 磁盘空间 | | **SUSE Linux Enterprise Server 15/openSUSE Leap 15.2** | 2 核 CPU，4 GB 内存，40 GB 磁盘空间 | >以上的系统要求和以下的教程适用于没有启用任何可选组件的默认最小化安装。 ![image-20220517012416510](../../img/kubernetes/kubernetes_kubesphere/image-20220517012416510.png) ![image-20220517013504048](../../img/kubernetes/kubernetes_kubesphere/image-20220517013504048.png) ![image-20220517013601517](../../img/kubernetes/kubernetes_kubesphere/image-20220517013601517.png) ## 1.2 节点要求 - 节点必须能够通过 `SSH` 连接。 - 节点上可以使用 `sudo`/`curl`/`openssl` 命令。 ## 1.3 容器运行时 您的集群必须有一个可用的容器运行时。如果您使用 KubeKey 搭建集群，KubeKey 会默认安装最新版本的 Docker。或者，您也可以在创建集群前手动安装 Docker 或其他容器运行时。 | 支持的容器运行时 | 版本 | | :---------------------------- | :------- | | Docker | 19.3.8 + | | containerd | 最新版 | | CRI-O（试验版，未经充分测试） | 最新版 | | iSula（试验版，未经充分测试） | 最新版 | >如果您想在离线环境中部署 KubeSphere，请务必提前安装一个容器运行时。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ~~~powershell # yum -y install docker-ce ~~~ ~~~powershell # systemctl enable --now docker ~~~ ~~~powershell # cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } ~~~ ~~~powershell # systemctl restart docker ~~~ ## 1.4 依赖项要求 >KubeKey 是用 Go 语言开发的一款全新的安装工具，代替了以前基于 ansible 的安装程序。KubeKey 为用户提供了灵活的安装选择，可以分别安装 KubeSphere 和 Kubernetes 或二者同时安装，既方便又高效。 KubeKey 可以将 Kubernetes 和 KubeSphere 一同安装。针对不同的 Kubernetes 版本，需要安装的依赖项可能有所不同。您可以参考以下列表，查看是否需要提前在节点上安装相关的依赖项。 | 依赖项 | Kubernetes 版本 ≥ 1.18 | Kubernetes 版本 < 1.18 | | :---------- | :--------------------- | :--------------------- | | `socat` | 必须 | 可选但建议 | | `conntrack` | 必须 | 可选但建议 | | `ebtables` | 可选但建议 | 可选但建议 | | `ipset` | 可选但建议 | 可选但建议 | ~~~powershell # yum -y install socat conntrack ebtables ipset ~~~ ## 1.5 网络和 DNS 要求 - 请确保 `/etc/resolv.conf` 中的 DNS 地址可用，否则，可能会导致集群中的 DNS 出现问题。 - 如果您的网络配置使用防火墙规则或安全组，请务必确保基础设施组件可以通过特定端口相互通信。建议您关闭防火墙。有关更多信息，请参见[端口要求](https://kubesphere.com.cn/docs/installing-on-linux/introduction/port-firewall/)。 - 支持的 CNI 插件：Calico 和 Flannel。其他插件也适用（例如 Cilium 和 Kube-OVN 等），但请注意它们未经充分测试。 > 提示： > > - 建议您的操作系统处于干净状态（不安装任何其他软件），否则可能会发生冲突。 > - 如果您无法从 dockerhub.io 下载容器镜像，建议提前准备仓库的镜像地址（即加速器）。 ~~~powershell # cat /etc/resolv.conf nameserver 119.29.29.29 ~~~ ~~~powershell # firewall-cmd --state not running ~~~ ~~~powershell # sestatus SELinux status: disabled ~~~ # 二、下载 KubeKey > 2.1与2.2看情况使用。 ## 2.1 通过GitHub/Googleapis获取kk 从 [GitHub Release Page](https://github.com/kubesphere/kubekey/releases) 下载 KubeKey 或直接使用以下命令。 ~~~powershell curl -sfL https://get-kk.kubesphere.io | VERSION=v2.0.0 sh - ~~~ ~~~powershell 输出： Downloading kubekey v2.0.0 from https://github.com/kubesphere/kubekey/releases/download/v2.0.0/kubekey-v2.0.0-linux-amd64.tar.gz ... Kubekey v2.0.0 Download Complete! ~~~ ~~~powershell # ls kk 绿色 ~~~ >执行以上命令会下载最新版 KubeKey (v2.0.0)，您可以修改命令中的版本号下载指定版本。 为 `kk` 添加可执行权限： > 默认不需要添加，注意观察，如果没有执行权限就添加。 ~~~powershell chmod +x kk ~~~ ## 2.2 从正确的区域下载 KubeKey。 ~~~powershell export KKZONE=cn ~~~ 执行以下命令下载kubekey ~~~powershell curl -sfL https://get-kk.kubesphere.io | VERSION=v2.0.0 sh - ~~~ >在您下载 KubeKey 后，如果您将其传至新的机器，且访问 Googleapis 同样受限，在您执行以下步骤之前请务必再次执行 `export KKZONE=cn` 命令。 # 三、开始安装 在本快速入门教程中，您只需执行一个命令即可进行安装，其模板如下所示： ~~~powershell ./kk create cluster [--with-kubernetes version] [--with-kubesphere version] ~~~ 若要同时安装 Kubernetes 和 KubeSphere，可参考以下示例命令： ~~~powershell ./kk create cluster --with-kubernetes v1.21.5 --with-kubesphere v3.2.1 ~~~ >- 安装 KubeSphere 3.2.1 的建议 Kubernetes 版本：1.19.x、1.20.x、1.21.x 或 1.22.x（实验性支持）。如果不指定 Kubernetes 版本，KubeKey 将默认安装 Kubernetes v1.21.5。有关受支持的 Kubernetes 版本的更多信息，请参见[支持矩阵](https://kubesphere.com.cn/docs/installing-on-linux/introduction/kubekey/#支持矩阵)。 >- 一般来说，对于 All-in-One 安装，您无需更改任何配置。 >- 如果您在这一步的命令中不添加标志 `--with-kubesphere`，则不会部署 KubeSphere，KubeKey 将只安装 Kubernetes。如果您添加标志 `--with-kubesphere` 时不指定 KubeSphere 版本，则会安装最新版本的 KubeSphere。 >- KubeKey 会默认安装 [OpenEBS](https://openebs.io/) 为开发和测试环境提供 LocalPV 以方便新用户。对于其他存储类型，请参见[持久化存储配置](https://kubesphere.com.cn/docs/installing-on-linux/persistent-storage-configurations/understand-persistent-storage/)。 执行该命令后，KubeKey 将检查您的安装环境，结果显示在一张表格中。有关详细信息，请参见节点要求和依赖项要求。 ![image-20220517015613194](../../img/kubernetes/kubernetes_kubesphere/image-20220517015613194.png) # 四、验证安装结果 当您看到以下输出时，表明安装已经完成。 ![image-20220517012006124](../../img/kubernetes/kubernetes_kubesphere/image-20220517012006124.png) 输入以下命令以检查安装结果。 ~~~powershell kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f ~~~ 输出信息会显示 Web 控制台的 IP 地址和端口号，默认的 NodePort 是 `30880`。现在，您可以使用默认的帐户和密码 (`admin/P@88w0rd`) 通过 `:30880` 访问控制台。 ~~~powershell ##################################################### ### Welcome to KubeSphere! ### ##################################################### Console: http://192.168.10.223:30880 Account: admin Password: P@88w0rd NOTES： 1. After you log into the console, please check the monitoring status of service components in "Cluster Management". If any service is not ready, please wait patiently until all components are up and running. 2. Please change the default password after login. ##################################################### https://kubesphere.io 20xx-xx-xx xx:xx:xx ##################################################### ~~~ 登录至控制台后，您可以在**系统组件**中查看各个组件的状态。如果要使用相关服务，您可能需要等待部分组件启动并运行。您也可以使用 `kubectl get pod --all-namespaces` 来检查 KubeSphere 相关组件的运行状况。 ![image-20220517020658982](../../img/kubernetes/kubernetes_kubesphere/image-20220517020658982.png) ![image-20220517020739940](../../img/kubernetes/kubernetes_kubesphere/image-20220517020739940.png) ![image-20220517020807274](../../img/kubernetes/kubernetes_kubesphere/image-20220517020807274.png) ![image-20220517020829592](../../img/kubernetes/kubernetes_kubesphere/image-20220517020829592.png) ![image-20220517020857139](../../img/kubernetes/kubernetes_kubesphere/image-20220517020857139.png) # 在kubernetes集群上最小化安装kubesphere 除了在 Linux 机器上安装 KubeSphere 之外，您还可以将其直接部署在现有的 Kubernetes 集群上。 # 一、准备工作 ## 1.1 kubernetes集群节点硬件要求 - 确保您的机器满足最低硬件要求：CPU 大于或等于4 核，内存 大于或等于8 GB。 ## 1.2 部署kubernetes集群 >- 如需在 Kubernetes 上安装 KubeSphere 3.2.1，您的 Kubernetes 版本必须为：1.19.x、1.20.x、1.21.x 或 1.22.x（实验性支持）。 可参考使用kubeadm部署kubernetes集群方法。 > 本案例采用4个节点，其中3个节点用于部署kubernetes集群，1个节点用于提供存储动态供给。 ## 1.3 后端存储动态供给准备 > 在安装之前，需要配置kubernetes个课程上的默认存储类型。 ### 1.3.1 准备硬盘 ~~~powershell 查看准备的磁盘 [root@nfsserver ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 99G 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 2G 0 lvm [SWAP] └─centos-home 253:2 0 47G 0 lvm /home sdb 8:16 0 100G 0 disk ~~~ ### 1.3.2 安装NFS软件 ~~~powershell 安装NFS软件，即是客户端也是服务器端 # yum -y install nfs-utils ~~~ ### 1.3.3 NFS配置 ~~~powershell 创建挂载点 # mkdir /netshare ~~~ ~~~powershell 格式化硬盘 # mkfs.xfs /dev/sdb ~~~ ~~~powershell 编辑文件系统配置文件 # vim /etc/fstab 在文件最后添加此行内容 /dev/sdb /netshare xfs defaults 0 0 ~~~ ~~~powershell 手动挂载全部分区 # mount -a ~~~ ~~~powershell 在本地查看文件系统挂载情况 # df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/sdb 100G 33M 100G 1% /netshare ~~~ ~~~powershell 添加共享目录到配置文件 # vim /etc/exports # cat /etc/exports /netshare *(rw,sync,no_root_squash) ~~~ ~~~powershell 启动服务及设置开机自启动 # systemctl enable nfs-server # systemctl start nfs-server ~~~ ### 1.3.4 验证 ~~~powershell 本地验证目录是否共享 # showmount -e Export list for nfsserver: /netshare * ~~~ ~~~powershell 在k8s master节点验证目录是否共享 # showmount -e nfsserver.kubemsb.com Export list for nfsserver.kubemsb.com: /netshare * ~~~ ~~~powershell 在k8s worker01节点验证目录是否共享 # showmount -e nfsserver.kubemsb.com Export list for nfsserver.kubemsb.com: /netshare * ~~~ ~~~powershell 在k8s worker02节点验证目录是否共享 # showmount -e nfsserver.kubemsb.com Export list for nfsserver.kubemsb.com: /netshare * ~~~ ### 1.3.5 部署存储动态供给 #### 1.3.5.1 获取资源清单文件 ~~~powershell 在k8s master节点获取NFS后端存储动态供给配置资源清单文件 # for file in class.yaml deployment.yaml rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/$file ; done ~~~ ~~~powershell 查看是否下载 # ls class.yaml deployment.yaml rbac.yaml ~~~ #### 1.3.5.2 应用资源清单文件 ~~~powershell 应用rbac资源清单文件 # kubectl apply -f rbac.yaml ~~~ ~~~powershell 应用class（存储类）资源清单文件 # kubectl apply -f class.yaml storageclass.storage.k8s.io/managed-nfs-storage created ~~~ ~~~powershell 应用deployment资源清单文件之前修改其配置，主要配置NFS服务器及其共享的目录 # vim deployment.yaml 注意修改处内容 env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: nfsserver.kubemsb.com 远程NFS服务器地址 - name: NFS_PATH value: /netshare 共享出来的目录 volumes: - name: nfs-client-root nfs: server: nfsserver.kubemsb.com 远程NFS服务器地址 path: /netshare 共享出来的目录 ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f deployment.yaml ~~~ ~~~powershell 查看pod运行情况 # kubectl get pods 出现以下表示成功运行 NAME READY STATUS RESTARTS AGE nfs-client-provisioner-f66d6fbdb-mcrpk 1/1 Running 0 73s ~~~ > 以下为特别注意事项。 ~~~powershell k8s 1.21版本存在问题：无法创建PV，修改api文件，此文件被自动监视，修改后会自动关闭原有Pod，拉起拉的Pod # cat /etc/kubernetes/manifests/kube-apiserver.yaml apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.10.10:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --feature-gates=RemoveSelfLink=false 添加此行内容 - --advertise-address=192.168.10.10 - --allow-privileged=true - --authorization-mode=Node,RBAC ~~~ >使用新的不基于 SelfLink 功能的 provisioner 镜像，重新创建 provisioner 容器。镜像：registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0 ~~~powershell 设置默认存储类 # kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ~~~ #### 1.3.5.3 测试用例验证动态供给是否可用 > 使用测试用例测试NFS后端存储是否可用 ~~~powershell 测试用例： # cat nginx.yaml --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: selector: matchLabels: app: nginx serviceName: "nginx" replicas: 2 template: metadata: labels: app: nginx spec: imagePullSecrets: - name: huoban-harbor terminationGracePeriodSeconds: 10 containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "managed-nfs-storage" resources: requests: storage: 1Gi ~~~ # 二、部署Kubesphere ## 2.1 最小化安装 ~~~powershell kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/kubesphere-installer.yaml kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/cluster-configuration.yaml ~~~ > 如需求开启更多功能，可在cluster-configuration.yaml文件中配置。 ## 2.2 检查安装日志 ~~~powershell kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f ~~~ ## 2.3 检查kubesphere安装是否正常 使用 `kubectl get pod --all-namespaces` 查看所有 Pod 是否在 KubeSphere 的相关命名空间中正常运行。如果是，请通过以下命令检查控制台的端口（默认为 `30880`）： ~~~powershell kubectl get svc/ks-console -n kubesphere-system ~~~ - 通过 NodePort `(IP:30880)` 使用默认帐户和密码 `(admin/P@88w0rd)` 访问 Web 控制台。 ## 2.4 访问kubesphere ![image-20220517132857802](../../img/kubernetes/kubernetes_kubesphere/image-20220517132857802.png) ![image-20220517132926979](../../img/kubernetes/kubernetes_kubesphere/image-20220517132926979.png) ![image-20220517133015625](../../img/kubernetes/kubernetes_kubesphere/image-20220517133015625.png) ![image-20220517133041644](../../img/kubernetes/kubernetes_kubesphere/image-20220517133041644.png) ![image-20220517133221266](../../img/kubernetes/kubernetes_kubesphere/image-20220517133221266.png) # 微服务项目 基于Kubernetes集群PaaS云平台 Kubesphere部署 # 一、Kubernetes集群持久存储准备 NFS > 在安装之前，需要配置kubernetes个课程上的默认存储类型。 ## 1.1 准备硬盘 ~~~powershell 查看准备的磁盘 [root@nfsserver ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 99G 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 2G 0 lvm [SWAP] └─centos-home 253:2 0 47G 0 lvm /home sdb 8:16 0 100G 0 disk ~~~ ## 1.2 安装NFS软件 ~~~powershell 安装NFS软件，即是客户端也是服务器端 # yum -y install nfs-utils ~~~ ## 1.3 NFS配置 ~~~powershell 创建挂载点 # mkdir /netshare ~~~ ~~~powershell 格式化硬盘 # mkfs.xfs /dev/sdb ~~~ ~~~powershell 编辑文件系统配置文件 # vim /etc/fstab 在文件最后添加此行内容 /dev/sdb /netshare xfs defaults 0 0 ~~~ ~~~powershell 手动挂载全部分区 # mount -a ~~~ ~~~powershell 在本地查看文件系统挂载情况 # df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/sdb 100G 33M 100G 1% /netshare ~~~ ~~~powershell 添加共享目录到配置文件 # vim /etc/exports # cat /etc/exports /netshare *(rw,sync,no_root_squash) ~~~ ~~~powershell 启动服务及设置开机自启动 # systemctl enable nfs-server # systemctl start nfs-server ~~~ ## 1.4 验证 ~~~powershell 本地验证目录是否共享 # showmount -e Export list for nfsserver: /netshare * ~~~ ~~~powershell 在k8s master节点验证目录是否共享 # showmount -e 192.168.10.147 Export list for 192.168.10.147: /netshare * ~~~ ~~~powershell 在k8s worker01节点验证目录是否共享 # showmount -e 192.168.10.147 Export list for 192.168.10.147: /netshare * ~~~ ## 1.5 部署存储动态供给 ### 1.5.1 获取资源清单文件 ~~~powershell 在k8s master节点获取NFS后端存储动态供给配置资源清单文件 # for file in class.yaml deployment.yaml rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/$file ; done ~~~ ~~~powershell 查看是否下载 # ls class.yaml deployment.yaml rbac.yaml ~~~ ### 1.5.2 应用资源清单文件 ~~~powershell 应用rbac资源清单文件 # kubectl apply -f rbac.yaml ~~~ ~~~powershell 修改存储类名称 # vim class.yaml # cat class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-client provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false" ~~~ ~~~powershell 应用class（存储类）资源清单文件 # kubectl apply -f class.yaml storageclass.storage.k8s.io/nfs-client created ~~~ ~~~powershell 应用deployment资源清单文件之前修改其配置，主要配置NFS服务器及其共享的目录 # vim deployment.yaml 注意修改处内容 # vim deployment.yaml # cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.10.147 - name: NFS_PATH value: /netshare volumes: - name: nfs-client-root nfs: server: 192.168.10.147 path: /netshare ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f deployment.yaml ~~~ ~~~powershell 查看pod运行情况 # kubectl get pods 出现以下表示成功运行 NAME READY STATUS RESTARTS AGE nfs-client-provisioner-8bcf6c987-7cb8p 1/1 Running 0 74s ~~~ ~~~powershell 设置默认存储类 # kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ~~~ ~~~powershell # kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client (default) fuseim.pri/ifs Delete Immediate false 18m ~~~ ### 1.5.3 测试用例验证动态供给是否可用 > 使用测试用例测试NFS后端存储是否可用 ~~~powershell 测试用例： # vim nginx.yaml # cat nginx.yaml --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: selector: matchLabels: app: nginx serviceName: "nginx" replicas: 2 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "nfs-client" resources: requests: storage: 1Gi ~~~ ~~~powershell # kubectl apply -f nginx.yaml service/nginx created statefulset.apps/web created ~~~ ~~~powershell # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE www-web-0 Bound pvc-57bee742-326b-4d41-b241-7f2b5dd22596 1Gi RWO nfs-client 3m19s ~~~ # 二、Kubesphere部署 ![image-20220809220111447](../../img/kubernetes/kubernetes_kubesphere/image-20220809220111447.png) ~~~powershell # kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/kubesphere-installer.yaml ~~~ ~~~powershell # kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.2.1/cluster-configuration.yaml ~~~ ~~~powershell # kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f ~~~ ~~~powershell ************************************************** Waiting for all tasks to be completed ... task network status is successful (1/4) task openpitrix status is successful (2/4) task multicluster status is successful (3/4) task monitoring status is successful (4/4) ************************************************** Collecting installation results ... ##################################################### ### Welcome to KubeSphere! ### ##################################################### Console: http://192.168.10.141:30880 Account: admin Password: P@88w0rd NOTES： 1. After you log into the console, please check the monitoring status of service components in "Cluster Management". If any service is not ready, please wait patiently until all components are up and running. 2. Please change the default password after login. ##################################################### ~~~ ![image-20220809230741623](../../img/kubernetes/kubernetes_kubesphere/image-20220809230741623.png) ![image-20220809230815158](../../img/kubernetes/kubernetes_kubesphere/image-20220809230815158.png) ![image-20220809230848565](../../img/kubernetes/kubernetes_kubesphere/image-20220809230848565.png) ![image-20220809230905838](../../img/kubernetes/kubernetes_kubesphere/image-20220809230905838.png) ![image-20220809230934784](../../img/kubernetes/kubernetes_kubesphere/image-20220809230934784.png) # 三、Kubesphere开启devops功能 ![image-20220809231058295](../../img/kubernetes/kubernetes_kubesphere/image-20220809231058295.png) ![image-20220809231146157](../../img/kubernetes/kubernetes_kubesphere/image-20220809231146157.png) ![image-20220809231209668](../../img/kubernetes/kubernetes_kubesphere/image-20220809231209668.png) ![image-20220809231245060](../../img/kubernetes/kubernetes_kubesphere/image-20220809231245060.png) ![image-20220809232511974](../../img/kubernetes/kubernetes_kubesphere/image-20220809232511974.png) # kubesphere多租户管理系统应用 # 一、kubesphere平台登录 ![image-20220922173954226](../../img/kubernetes/kubernetes_kubesphere/image-20220922173954226.png) ![image-20220922174311230](../../img/kubernetes/kubernetes_kubesphere/image-20220922174311230.png) ![image-20220922174332481](../../img/kubernetes/kubernetes_kubesphere/image-20220922174332481.png) ![image-20220922174429882](../../img/kubernetes/kubernetes_kubesphere/image-20220922174429882.png) ![image-20220922175512664](../../img/kubernetes/kubernetes_kubesphere/image-20220922175512664.png) ![image-20220922175634327](../../img/kubernetes/kubernetes_kubesphere/image-20220922175634327.png) ![image-20220922175533321](../../img/kubernetes/kubernetes_kubesphere/image-20220922175533321.png) ![image-20220922175658657](../../img/kubernetes/kubernetes_kubesphere/image-20220922175658657.png) ![image-20220922175600095](../../img/kubernetes/kubernetes_kubesphere/image-20220922175600095.png) # 二、Kubesphere应用架构 ## 2.1 架构介绍 - KubeSphere 的多租户系统分**三个**层级，即集群、企业空间和项目。 - **集群**即为kubernetes集群 - **企业空间**是用来管理[项目](https://kubesphere.com.cn/docs/v3.3/project-administration/)、[DevOps 项目](https://kubesphere.com.cn/docs/v3.3/devops-user-guide/)、[应用模板](https://kubesphere.com.cn/docs/v3.3/workspace-administration/upload-helm-based-application/)和应用仓库的一种逻辑单元。可以在企业空间中控制资源访问权限，也可以安全地在团队内部分享资源。 - **项目**即为kubernetes的命名空间 - 您需要创建一个新的企业空间进行操作，而不是使用系统企业空间，系统企业空间中运行着系统资源，绝大部分仅供查看。出于安全考虑，强烈建议给不同的租户授予不同的权限在企业空间中进行协作。 - 可以在一个 KubeSphere 集群中创建多个企业空间，每个企业空间下可以创建多个项目。 ## 2.2 创建企业空间、项目、用户和平台角色 KubeSphere 为每个级别默认设有多个内置角色。此外，您还可以创建拥有自定义权限的角色。KubeSphere 多层次结构适用于具有不同团队或组织以及每个团队中需要不同角色的企业用户。 ### 2.2.1 创建用户 安装 KubeSphere 之后，您需要向平台添加具有不同角色的用户，以便他们可以针对自己授权的资源在不同的层级进行工作。一开始，系统默认只有一个用户 `admin`，具有 `platform-admin` 角色。在本步骤中，您将创建一个示例用户 `user-manager`，然后使用 `user-manager` 创建新用户。 1.以 `admin` 身份使用默认帐户和密码 (`admin/P@88w0rd`,本次使用`admin/Kubemsb123`) 登录 Web 控制台。 >出于安全考虑，强烈建议您在首次登录控制台时更改密码。若要更改密码，在右上角的下拉列表中选择**用户设置**，在**密码设置**中设置新密码，您也可以在**用户设置** > **基本信息**中修改控制台语言。 2.点击左上角的**平台管理**，然后选择**访问控制**。在左侧导航栏中，选择**平台角色**。四个内置角色的描述信息如下表所示。 ![image-20220922185507731](../../img/kubernetes/kubernetes_kubesphere/image-20220922185507731.png) ![image-20220922185532487](../../img/kubernetes/kubernetes_kubesphere/image-20220922185532487.png) ![image-20220922185601902](../../img/kubernetes/kubernetes_kubesphere/image-20220922185601902.png) | 内置角色 | 描述 | | :------------------- | :----------------------------------------------------------- | | `workspaces-manager` | 企业空间管理员，管理平台所有企业空间。 | | `users-manager` | 用户管理员，管理平台所有用户。 | | `platform-regular` | 平台普通用户，在被邀请加入企业空间或集群之前没有任何资源操作权限。 | | `platform-admin` | 平台管理员，可以管理平台内的所有资源。 | >内置角色由 KubeSphere 自动创建，无法编辑或删除。 3.在**用户**中，点击**创建**。在弹出的对话框中，提供所有必要信息（带有*标记），然后在**平台角色**一栏选择 `users-manager` ![image-20220922185654055](../../img/kubernetes/kubernetes_kubesphere/image-20220922185654055.png) ![image-20220922185930626](../../img/kubernetes/kubernetes_kubesphere/image-20220922185930626.png) ![image-20220922185956364](../../img/kubernetes/kubernetes_kubesphere/image-20220922185956364.png) 完成后，点击**确定**。新创建的用户将显示在**用户**页面。 4.切换用户使用 `user-manager` 重新登录，创建如下四个新用户，这些用户将在其他的教程中使用。 >- 帐户登出请点击右上角的用户名，然后选择**登出**。 >- 下面仅为示例用户名，请根据实际情况修改。 ![image-20220922190125715](../../img/kubernetes/kubernetes_kubesphere/image-20220922190125715.png) ![image-20220922190147221](../../img/kubernetes/kubernetes_kubesphere/image-20220922190147221.png) ![image-20220922190235404](../../img/kubernetes/kubernetes_kubesphere/image-20220922190235404.png) ![image-20220922190257617](../../img/kubernetes/kubernetes_kubesphere/image-20220922190257617.png) ![image-20220922190338870](../../img/kubernetes/kubernetes_kubesphere/image-20220922190338870.png) | 用户 | 指定的平台角色 | 用户权限 | | :---------------- | :------------------- | :----------------------------------------------------------- | | `ws-manager` | `workspaces-manager` | 创建和管理所有企业空间。 | | `ws-admin` | `platform-regular` | 被邀请到企业空间后，管理该企业空间中的所有资源（在此示例中，此用户用于邀请新成员加入该企业空间）。 | | `project-admin` | `platform-regular` | 创建和管理项目以及 DevOps 项目，并邀请新成员加入项目。 | | `project-regular` | `platform-regular` | `project-regular` 将由 `project-admin` 邀请至项目或 DevOps 项目。该用户将用于在指定项目中创建工作负载、流水线和其他资源。 | ![image-20220922190428000](../../img/kubernetes/kubernetes_kubesphere/image-20220922190428000.png) ![image-20220922190527754](../../img/kubernetes/kubernetes_kubesphere/image-20220922190527754.png) ![image-20220922190706339](../../img/kubernetes/kubernetes_kubesphere/image-20220922190706339.png) ![image-20220922190804551](../../img/kubernetes/kubernetes_kubesphere/image-20220922190804551.png) ![image-20220922190924240](../../img/kubernetes/kubernetes_kubesphere/image-20220922190924240.png) 5.在**用户**页面，查看创建的四个用户。 ![image-20220922190952728](../../img/kubernetes/kubernetes_kubesphere/image-20220922190952728.png) >您可以点击用户名称后的 ![img](https://kubesphere.com.cn/images/docs/v3.3/common-icons/three-dots.png) 图标选择启用或禁用某个用户。您也可以勾选多个用户进行批量操作。 ### 2.2.2 创建企业空间 需要使用上一个步骤中创建的用户 `ws-manager` 创建一个企业空间。作为管理项目、DevOps 项目和组织成员的基本逻辑单元，企业空间是 KubeSphere 多租户系统的基础。 1.以 `ws-manager` 身份登录 KubeSphere。点击左上角的**平台管理**，选择**访问控制**。在**企业空间**中，可以看到仅列出了一个默认企业空间 `system-workspace`，即系统企业空间，其中运行着与系统相关的组件和服务，您无法删除该企业空间。 ![image-20220922191404163](../../img/kubernetes/kubernetes_kubesphere/image-20220922191404163.png) ![image-20220922191420465](../../img/kubernetes/kubernetes_kubesphere/image-20220922191420465.png) ![image-20220922191436794](../../img/kubernetes/kubernetes_kubesphere/image-20220922191436794.png) 2.点击右侧的**创建**，将新企业空间命名为 `demo-workspace`，并将用户 `ws-admin` 设置为企业空间管理员。完成后，点击**创建**。 ![image-20220922191618983](../../img/kubernetes/kubernetes_kubesphere/image-20220922191618983.png) ![image-20220923113240468](../../img/kubernetes/kubernetes_kubesphere/image-20220923113240468.png) >如果您已启用[多集群功能](https://kubesphere.com.cn/docs/v3.3/multicluster-management/)，您需要为企业空间[分配一个或多个可用集群](https://kubesphere.com.cn/docs/v3.3/cluster-administration/cluster-settings/cluster-visibility-and-authorization/#在创建企业空间时选择可用集群)，以便项目可以在集群中创建。 3.登出控制台，然后以 `ws-admin` 身份重新登录。在**企业空间设置**中，选择**企业空间成员**，然后点击**邀请**。 4.邀请 `project-admin` 和 `project-regular` 进入企业空间，分别授予 `demo-workspace-self-provisioner` 和 `demo-workspace-viewer` 角色，点击**确定**。 > 实际角色名称的格式：`-`。例如，在名为 `demo-workspace` 的企业空间中，角色 `viewer` 的实际角色名称为 `demo-workspace-viewer`。 5.将 `project-admin` 和 `project-regular` 都添加到企业空间后，点击**确定**。在**企业空间成员**中，您可以看到列出的三名成员。 | 用户 | 分配的企业空间角色 | 角色权限 | | :---------------- | :-------------------------------- | :----------------------------------------------------------- | | `ws-admin` | `demo-workspace-admin` | 管理指定企业空间中的所有资源（在此示例中，此用户用于邀请新成员加入企业空间）。 | | `project-admin` | `demo-workspace-self-provisioner` | 创建和管理项目以及 DevOps 项目，并邀请新成员加入项目。 | | `project-regular` | `demo-workspace-viewer` | `project-regular` 将由 `project-admin` 邀请至项目或 DevOps 项目。该用户将用于在指定项目中创建工作负载、流水线和其他资源。 | ### 2.2.3 创建项目 在此步骤中，您需要使用在上一步骤中创建的帐户 `project-admin` 来创建项目。KubeSphere 中的项目与 Kubernetes 中的命名空间相同，为资源提供了虚拟隔离。有关更多信息，请参见[命名空间](https://kubernetes.io/zh/docs/concepts/overview/working-with-objects/namespaces/)。 1.以 `project-admin` 身份登录 KubeSphere Web 控制台，在**项目**中，点击**创建**。 2.输入项目名称（例如 `demo-project`），点击**确定**。您还可以为项目添加别名和描述。 3.在**项目**中，点击刚创建的项目查看其详情页面。 4.在项目的**概览**页面，默认情况下未设置项目配额。您可以点击**编辑配额**并根据需要指定[资源请求和限制](https://v3-2.docs.kubesphere.io/zh/docs/workspace-administration/project-quotas/)（例如：CPU 和内存的限制分别设为 1 Core 和 1000 Gi）。 5.在**项目设置** > **项目成员**中，邀请 `project-regular` 至该项目，并授予该用户 `operator` 角色。 >具有 `operator` 角色的用户是项目维护者，可以管理项目中除用户和角色以外的资源。 6.在创建[应用路由](https://v3-2.docs.kubesphere.io/zh/docs/project-user-guide/application-workloads/routes/)（即 Kubernetes 中的 [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/)）之前，需要启用该项目的网关。网关是在项目中运行的 [NGINX Ingress 控制器](https://github.com/kubernetes/ingress-nginx)。若要设置网关，请转到**项目设置**中的**网关设置**，然后点击**设置网关**。此步骤中仍使用帐户 `project-admin`。 7.选择访问方式 **NodePort**，然后点击**确定**。 8.在**网关设置**下，可以在页面上看到网关地址以及 http/https 的端口。 >如果要使用 `LoadBalancer` 暴露服务，则需要使用云厂商的 LoadBalancer 插件。如果您的 Kubernetes 集群在裸机环境中运行，建议使用 [OpenELB](https://github.com/kubesphere/openelb) 作为 LoadBalancer 插件。 ### 2.2.4 创建角色 完成上述步骤后，您已了解可以为不同级别的用户授予不同角色。先前步骤中使用的角色都是 KubeSphere 提供的内置角色。在此步骤中，您将学习如何创建自定义角色以满足工作需求。 1.再次以 `admin` 身份登录 KubeSphere Web 控制台，转到**访问控制**。 2.点击左侧导航栏中的**平台角色**，再点击右侧的**创建**。 >**平台角色**页面的预设角色无法编辑或删除。 3.在**创建平台角色**对话框中，设置角色标识符（例如，`clusters-admin`）、角色名称和描述信息，然后点击**编辑权限**。 > 本示例演示如何创建负责集群管理的角色。 4.在**编辑权限**对话框中，设置角色权限（例如，选择**集群管理**）并点击**确定**。 >- 在本示例中，角色 `clusters-admin` 包含**集群管理**和**集群查看**权限。 >- 一些权限依赖于其他权限，依赖项由每项权限下的**依赖于**字段指定。 >- 选择权限后，将自动选择它所依赖的权限。 >- 若要取消选择权限，则需要首先取消选择其从属权限。 5.在**平台角色**页面，可以点击所创建角色的名称查看角色详情，点击 ![img](https://v3-2.docs.kubesphere.io/images/docs/zh-cn/quickstart/create-workspaces-projects-accounts/%E6%93%8D%E4%BD%9C%E6%8C%89%E9%92%AE.png) 以编辑角色、编辑角色权限或删除该角色。 6.在**用户**页面，可以在创建帐户或编辑现有帐户时为帐户分配该角色。 ### 2.2.5 创建DevOps >若要创建 DevOps 项目，需要预先启用 KubeSphere DevOps 系统，该系统是个可插拔的组件，提供 CI/CD 流水线、Binary-to-Image 和 Source-to-Image 等功能。有关如何启用 DevOps 的更多信息，请参见 [KubeSphere DevOps 系统](https://v3-2.docs.kubesphere.io/zh/docs/pluggable-components/devops/)。 ![image-20221021235714102](../../img/kubernetes/kubernetes_kubesphere/image-20221021235714102.png) ![image-20221021235838690](../../img/kubernetes/kubernetes_kubesphere/image-20221021235838690.png) ![image-20221021235910131](../../img/kubernetes/kubernetes_kubesphere/image-20221021235910131.png) ![image-20221022000034118](../../img/kubernetes/kubernetes_kubesphere/image-20221022000034118.png) ![image-20221022000103051](../../img/kubernetes/kubernetes_kubesphere/image-20221022000103051.png) ![image-20221022000159140](../../img/kubernetes/kubernetes_kubesphere/image-20221022000159140.png) ![image-20221022000310077](../../img/kubernetes/kubernetes_kubesphere/image-20221022000310077.png) ~~~powershell # kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f ~~~ 1.以 `project-admin` 身份登录控制台，在 **DevOps 项目**中，点击**创建**。 2.输入 DevOps 项目名称（例如 `demo-devops`），然后点击**确定**，也可以为该项目添加别名和描述。 3.点击刚创建的项目查看其详细页面。 4.转到 **DevOps 项目设置**，然后选择 **DevOps 项目成员**。点击**邀请**授予 `project-regular` 用户 `operator` 的角色，允许其创建流水线和凭证。 至此，您已熟悉 KubeSphere 的多租户管理系统。在其他教程中，`project-regular` 帐户还将用于演示如何在项目或 DevOps 项目中创建应用程序和资源。 # KubeSphere应用发布初体验 # 一、WordPress 简介 WordPress（使用 PHP 语言编写）是免费、开源的内容管理系统，用户可以使用 WordPress 搭建自己的网站。完整的 WordPress 应用程序包括以下 Kubernetes 对象，由 MySQL 作为后端数据库。 ![](../../img/kubernetes/kubernetes_kubesphere/WordPress.png) # 二、目的 本教程演示了如何在 KubeSphere 中创建应用程序（以 WordPress 为例）并在集群外进行访问。 # 三、准备工作 您需要准备一个 `project regular` 帐户，并在一个项目中赋予该帐户 `operator` 角色（该用户已被邀请参加该项目）。 # 四、部署过程 ## 4.1 创建密钥 ### 4.1.1 创建MySQL密钥 > 环境变量 `WORDPRESS_DB_PASSWORD` 是连接到 WordPress 数据库的密码。 在此步骤中，您需要创建一个密钥来保存将在 MySQL Pod 模板中使用的环境变量。 1.使用 `project-regular` 帐户登录 KubeSphere 控制台，访问 `demo-project` 的详情页并导航到**配置**。在**保密字典**中，点击右侧的**创建**。 2.输入基本信息（例如，将其命名为 `mysql-secret`）并点击**下一步**。在下一页中，选择**类型**为 **Opaque（默认）**，然后点击**添加数据**来添加键值对。输入如下所示的键 (Key) `MYSQL_ROOT_PASSWORD` 和值 (Value) `123456`，点击右下角 **√** 进行确认。完成后，点击**创建**按钮以继续。 ### 4.1.2 创建WordPress密钥 按照以上相同的步骤创建一个名为 `wordpress-secret` 的 WordPress 密钥，输入键 (Key) `WORDPRESS_DB_PASSWORD` 和值 (Value) `123456`。创建的密钥显示在列表中。 ## 4.2 创建存储卷 1.访问**存储**下的**存储卷**，点击**创建**。 2.输入卷的基本信息（例如，将其命名为 `wordpress-pvc`），然后点击**下一步**。 3.在**存储卷设置**中，需要选择一个可用的**存储类型**，并设置**访问模式**和**存储卷容量**。您可以直接使用默认值，点击**下一步**继续。 4.在**高级设置**中，您无需添加额外的配置，点击**创建**完成即可。 ## 4.3 创建应用程序 ### 4.3.1 添加MySQL后端组件 1.导航到**应用负载**下的**应用**，选择**自制应用** > **创建**。 2.输入基本信息（例如，在应用名称一栏输入 `wordpress`），然后点击**下一步**。 3.在**服务设置**中，点击**创建服务**以在应用中设置组件。 4.设置组件的服务类型为**有状态服务**。 5.输入有状态服务的名称（例如 **mysql**）并点击**下一步**。 6.在**容器组设置**中，点击**添加容器**。 7.在搜索框中输入 `mysql:5.6`，按下**回车键**，然后点击**使用默认端口**。由于配置还未设置完成，请不要点击右下角的 **√** 按钮。 >在**高级设置**中，请确保内存限制不小于 1000 Mi，否则 MySQL 可能因内存不足而无法启动。 8.向下滚动到**环境变量**，点击**引用配置文件或密钥**。输入名称 `MYSQL_ROOT_PASSWORD`，然后选择资源 `mysql-secret` 和前面步骤中创建的密钥 `MYSQL_ROOT_PASSWORD`，完成后点击 **√** 保存配置，最后点击**下一步**继续。 9.选择**存储卷设置**中的**添加存储卷模板**，输入**存储卷名称** (`mysql`) 和**挂载路径**（模式：`读写`，路径：`/var/lib/mysql`）的值。 完成后，点击 **√** 保存设置并点击**下一步**继续。 10.在**高级设置**中，可以直接点击**添加**，也可以按需选择其他选项。 11.现在，MySQL 组件已经添加完成。 ### 4.3.2 添加WordPress前端组件 1.再次点击**创建服务**，选择**无状态服务**。输入名称 `wordpress` 并点击**下一步**。 2.与上述步骤类似，点击**添加容器**，在搜索栏中输入 `wordpress:4.8-apache` 并按下**回车键**，然后点击**使用默认端口**。 3.向下滚动到**环境变量**，点击**引用配置文件或密钥**。这里需要添加两个环境变量，请根据以下截图输入值： - 对于 `WORDPRESS_DB_PASSWORD`，请选择在步骤 1 中创建的 `wordpress-secret` 和 `WORDPRESS_DB_PASSWORD`。 - 点击**添加环境变量**，分别输入 `WORDPRESS_DB_HOST` 和 `mysql` 作为键 (Key) 和值 (Value)。 >对于此处添加的第二个环境变量，该值必须与步骤 5 中创建 MySQL 有状态服务设置的名称完全相同。否则，WordPress 将无法连接到 MySQL 对应的数据库。 点击 **√** 保存配置，再点击**下一步**继续。 4.在**存储卷设置**中，点击**挂载存储卷**，并**选择存储卷**。 5.选择上一步创建的 `wordpress-pvc`，将模式设置为`读写`，并输入挂载路径 `/var/www/html`。点击 **√** 保存，再点击**下一步**继续。 6.在**高级设置**中，可以直接点击**添加**创建服务，也可以按需选择其他选项。 7.现在，前端组件也已设置完成。点击**下一步**继续。 8.您可以**路由设置**中设置路由规则（应用路由 Ingress），也可以直接点击**创建**。 9.创建后，应用将显示在应用列表中。 ## 4.4 验证资源创建情况 在**工作负载**中，分别检查**部署**和**有状态副本集**中 `wordpress-v1` 和 `mysql-v1` 的状态。如果它们的运行状态为**运行中**，就意味着 WordPress 已经成功创建。 ## 4.5 通过NodePort访问WordPress 1.若要在集群外访问服务，请首先导航到**服务**。点击 `wordpress` 右侧的三个点后，选择**编辑外部访问**。 2.在**访问方式**中选择 `NodePort`，然后点击**确定**。 3.点击服务进入详情页，可以在**端口**处查看暴露的端口。 ![image-20221022131423435](../../img/kubernetes/kubernetes_kubesphere/image-20221022131423435.png) 4.通过 `{Node IP}:{NodePort}` 访问此应用程序，可以看到下图： ![image-20221022131403934](../../img/kubernetes/kubernetes_kubesphere/image-20221022131403934.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_kubesphere_devops.md ================================================ # Kubesphere应用前 账号准备 # 一、Docker Hub账号注册 ![image-20221026164044647](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164044647.png) ![image-20221026164119433](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164119433.png) ![image-20221026164158778](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164158778.png) ![image-20221026164540247](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164540247.png) ![image-20221026164602299](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164602299.png) ![image-20221026164638283](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164638283.png) ![image-20221026164728173](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164728173.png) # 二、GitHub账号注册 ![image-20221026164905539](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026164905539.png) ![image-20221026165212818](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165212818.png) ![image-20221026165307549](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165307549.png) ![image-20221026165328295](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165328295.png) ![image-20221026165349757](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165349757.png) ![image-20221026165432544](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165432544.png) ![image-20221026165842535](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165842535.png) ![image-20221026165919225](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165919225.png) ![image-20221026165949811](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026165949811.png) ![image-20221026170055031](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026170055031.png) ![image-20221026170144901](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026170144901.png) ![image-20221026170249245](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026170249245.png) # 基于Kubesphere实现DevOps # 一、DevOps介绍 ## 1.1 项目开发需要考虑的维度 - dev 怎么开发 - ops 怎么运维 > 参考项目链接：https://github.com/kubesphere/devops-maven-sample ## 1.2 DevOps是什么 ![image-20221026122520801](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026122520801.png) # 二、CI/CD介绍 ![image-20221031094406755](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031094406755.png) ## 2.1 持续集成（Continuous Integration） 持续集成是指软件个人的部分向软件整体部分交付，频繁进行集成以便更快地发现其中错误。 CI需要具备这些： - 全面的自动化测试 这是实践持续集成&持续部署的基础，同时，选择合适的自动化测试工具也很重要 - 灵活的基础设施 容器、虚拟机的存在让开发人员和QA人员不必再大费周张去做IT基础设施环境的准备 - 版本控制工具 如：Git、SNV、Gitlab等 - 自动化的构建和软件发布流程工具 如：Jenkins Gitlab Tekton等 - 反馈机制 如果构建/测试失败，可以快速地反馈到相关负责人，以尽快速度解决问题，实现软件开发稳定版本推出。 ## 2.2 持续交付（Continuous Delivery ） 持续交付在持续集成的基础上，将集成后的代码部署到更贴近真实运行环境的类生产环境中，持续交付优先于整个产品生命周期的软件部署，建立在高水平自动化持续集成之上。 - 快速发布 能够应对业务需求，并更快地实现软件价值，实现编码、测试、上线、交付的频繁迭代周期缩短，同时获得迅速反馈 - 高质量软件发布标准 整个交付过程标准化、可重复、可靠 - 整个交付过程进度可视化 方便团队成员了解项目成熟度 - 更先进的团队协作方式 从需求分析、产品的用户体验到交互设计、开发、测试、运维等角色密切协作，相比于传统的的开发模式，减少浪费 ## 2.3 持续部署（Continuous Deployment） 持续部署是指当交付的代码通过评审之后，自动部署到生产环境中。 持续部署是持续交付的最高阶段。这就意味着，所有通过了一系列的自动化测试的发动都将自动部署到生产环境。它也可以被称为Continuous Release 持续部署主要好处是，可以相对独立地部署新的功能，并能快速地收集真实用户的反馈。 ![image-20221026121709183](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221026121709183.png) # 三、KubeSphere DevOps凭证管理 ## 3.1 凭证介绍 凭证是包含敏感信息的对象，例如用户名和密码、SSH 密钥和令牌 (Token)。当 KubeSphere DevOps 流水线运行时，会与外部环境中的对象进行交互，以执行一系列任务，包括拉取代码、推送和拉取镜像以及运行脚本等。此过程中需要提供相应的凭证，而这些凭证不会明文出现在流水线中。 具有必要权限的 DevOps 项目用户可以为 Jenkins 流水线配置凭证。用户在 DevOps 项目中添加或配置这些凭证后，便可以在 DevOps 项目中使用这些凭证与第三方应用程序进行交互。 目前，您可以在 DevOps 项目中创建以下 4 种类型的凭证： - **用户名和密码**：用户名和密码，可以作为单独的组件处理，或者作为用冒号分隔的字符串（格式为 `username:password`）处理，例如 GitHub、GitLab 和 Docker Hub 的帐户。 - **SSH 密钥**：带有私钥的用户名，SSH 公钥/私钥对。 - **访问令牌**：具有访问权限的令牌。 - **kubeconfig**：用于配置跨集群认证。如果选择此类型，将自动获取当前 Kubernetes 集群的 kubeconfig 文件内容，并自动填充在当前页面对话框中。 本教程演示如何在 DevOps 项目中创建和管理凭证。有关如何使用凭证的更多信息，请参见[使用 Jenkinsfile 创建流水线](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/create-a-pipeline-using-jenkinsfile/)和[使用图形编辑面板创建流水线](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/create-a-pipeline-using-graphical-editing-panel/)。 ## 3.2 准备工作 - 您已启用 [KubeSphere DevOps 系统](https://v3-2.docs.kubesphere.io/zh/docs/pluggable-components/devops/)。 - 您需要有一个企业空间、一个 DevOps 项目和一个用户 (`project-regular`)，并已邀请此帐户至 DevOps 项目中且授予 `operator` 角色。如果尚未准备好，请参见[创建企业空间、项目、用户和角色](https://v3-2.docs.kubesphere.io/zh/docs/quick-start/create-workspace-and-project/)。 ## 3.3 创建凭证 以 `project-regular` 身份登录 KubeSphere 控制台。进入您的 DevOps 项目，选择**凭证**，然后点击**创建**。 ### 3.3.1 创建 Docker Hub 凭证 1. 在弹出的对话框中输入以下信息。 - **名称**：设置可以在流水线中使用的 ID，例如 `dockerhub-id`。 - **类型**：选择**用户名和密码**。 - **用户名**：您的 Docker Hub 帐户（即 Docker ID）。 - **密码/令牌**：您的 Docker Hub 密码。 - **描述信息**：凭证的简介。 2. 完成操作后点击**确定**。 ![image-20221027165442735](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027165442735.png) ![image-20221027165531852](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027165531852.png) ![image-20221027165550931](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027165550931.png) ### 3.3.2 创建 GitHub 凭证 同样地，按照上述相同步骤创建 GitHub 凭证。设置不同的**名称**（例如 `github-id`），**类型**同样选择**用户名和密码**。分别在**用户名**和 **密码/令牌**中输入您的 GitHub 用户名和令牌。 >- 自 2021 年 8 月起，GitHub 要求使用基于令牌的身份验证，此处需要输入令牌，而非 GitHub 密码。关于如如何生成令牌，请参阅[创建个人访问令牌](https://docs.github.com/cn/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)。 >- 如果您的帐户或密码中包含任何特殊字符，例如 `@` 和 `$`，可能会因为无法识别而在流水线运行时导致错误。在这种情况下，您需要先在一些第三方网站（例如 [urlencoder](https://www.urlencoder.org/)）上对帐户或密码进行编码，然后将输出结果复制粘贴作为您的凭证信息。 ![image-20221027165713572](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027165713572.png) ![image-20221027170151637](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027170151637.png) ### 3.3.3 创建 kubeconfig 凭证 同样地，按照上述相同步骤创建 kubeconfig 凭证。设置不同的凭证 ID（例如 `demo-kubeconfig`）并选择 **kubeconfig**。 >用于配置集群访问的文件称为 kubeconfig 文件。这是引用配置文件的通用方法。有关更多信息，请参见 [Kubernetes 官方文档](https://kubernetes.io/zh/docs/concepts/configuration/organize-cluster-access-kubeconfig/)。您可以创建 kubeconfig 凭证来访问当前 Kubernetes 集群，该凭证将在流水线中使用。您不需要更改该文件，因为 KubeSphere 会自动使用当前 Kubernetes 集群的 kubeconfig 填充该字段。访问其他集群时，您可能需要更改 kubeconfig。 ![image-20221027170223311](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027170223311.png) ![image-20221027170323070](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027170323070.png) ### 3.3.4 查看和管理凭证 1. 凭证创建后，会在列表中显示。 2. 点击任意一个凭证，进入其详情页面，您可以查看帐户详情和与此凭证相关的所有事件。 3. 您也可以在此页面上编辑或删除凭证。请注意，编辑凭证时，KubeSphere 不会显示现有用户名或密码信息。如果输入新的用户名和密码，则前一个将被覆盖。 # 四、将SonarQube集成到流水线 [SonarQube](https://www.sonarqube.org/) 是一种主流的代码质量持续检测工具。您可以将其用于代码库的静态和动态分析。SonarQube 集成到 KubeSphere 流水线后，如果在运行的流水线中检测到问题，您可以直接在仪表板上查看常见代码问题，例如 Bug 和漏洞。 ## 4.1 准备工作 您需要[启用 KubeSphere DevOps 系统](https://v3-2.docs.kubesphere.io/zh/docs/pluggable-components/devops/)。 ## 4.2 安装 SonarQube 服务器 要将 SonarQube 集成到您的流水线，必须先安装 SonarQube 服务器。 1.请先安装 Helm，以便后续使用该工具安装 SonarQube。例如，运行以下命令安装 Helm 3： ~~~powershell # curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash ~~~ 2.查看helm版本 ~~~powershell # helm version version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.18.7"} ~~~ 3.执行以下命令安装 SonarQube 服务器 ~~~powershell # helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort ~~~ 4.您会获取以下提示内容 ~~~powershell [root@k8s-master01 ~]# kubectl get svc -n kubesphere-devops-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE devops-apiserver ClusterIP 10.96.2.56 9090/TCP 5d16h devops-jenkins NodePort 10.96.1.111 80:30180/TCP 5d16h devops-jenkins-agent ClusterIP 10.96.2.94 50000/TCP 5d16h s2ioperator-metrics-service ClusterIP 10.96.2.41 8080/TCP 5d16h s2ioperator-trigger-service ClusterIP 10.96.2.217 8081/TCP 5d16h sonarqube-postgresql ClusterIP 10.96.2.172 5432/TCP 3m38s sonarqube-postgresql-headless ClusterIP None 5432/TCP 3m38s sonarqube-sonarqube NodePort 10.96.0.120 9000:30058/TCP 3m38s webhook-server-service ClusterIP 10.96.3.74 443/TCP 5d16h ~~~ ## 4.3 获取 SonarQube 控制台地址 1.执行以下命令以获取 SonarQube NodePort ~~~powershell # export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube) ~~~ ~~~powershell # export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}") ~~~ ~~~powershell # echo http://$NODE_IP:$NODE_PORT ~~~ 2.您可以获得如下输出（本示例中端口号为 `30058`，可能与您的端口号不同） ~~~powershell http://192.168.10.141:30058 ~~~ ## 4.4 配置 SonarQube 服务器 ### 4.4.1 访问 SonarQube 控制台 1.执行以下命令查看 SonarQube 的状态。请注意，只有在 SonarQube 启动并运行后才能访问 SonarQube 控制台。 ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n kubesphere-devops-system NAME READY STATUS RESTARTS AGE sonarqube-postgresql-0 1/1 Running 0 37m sonarqube-sonarqube-df8d79d5c-hxxbx 1/1 Running 10 (2m34s ago) 37m ~~~ 2.在浏览器中访问 SonarQube 控制台 `http://:`。 ![image-20221027180700822](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027180700822.png) 3.点击右上角的 **Log in**，然后使用默认帐户 `admin/admin` 登录。 >取决于您的实例的部署位置，您可能需要设置必要的端口转发规则，并在您的安全组中放行该端口，以便访问 SonarQube。 ![image-20221027180719770](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027180719770.png) ![image-20221027180859117](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027180859117.png) ### 4.4.2 创建 SonarQube 管理员令牌 (Token) 1.点击右上角字母 **A**，然后从菜单中选择 **My Account** 以转到 **Profile** 页面。 ![image-20221027181023772](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027181023772.png) ![image-20221027181046300](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027181046300.png) 2.点击 **Security** 并输入令牌名称，例如 `kubesphere`。 ![image-20221027181424797](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027181424797.png) 3.点击 **Generate** 并复制此令牌。 ![image-20221027181543213](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027181543213.png) ~~~powershell c3ee79a4c4b25bd03867db735dc439e3bb726426 ~~~ >如提示所示，您无法再次查看此令牌，因此请确保复制成功。 ### 4.4.3 创建 Webhook 服务器 1.执行以下命令获取 SonarQube Webhook 的地址 ~~~powershell # export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services devops-jenkins) ~~~ ~~~powershell # export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}") ~~~ ~~~powershell # echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/ ~~~ 2.输出结果 ~~~powershell http://192.168.10.141:30180/sonarqube-webhook/ ~~~ 3.依次点击 **Administration**、**Configuration** 和 **Webhooks** 创建一个 Webhook ![image-20221027182301953](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027182301953.png) 4.点击 **Create** ![image-20221027182342776](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027182342776.png) 5.在弹出的对话框中输入 **Name** 和 **Jenkins Console URL**（即 SonarQube Webhook 地址）。点击 **Create** 完成操作。 ![image-20221027182504582](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027182504582.png) ![image-20221027182526782](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027182526782.png) ### 4.4.4 将 SonarQube 配置添加到 ks-installer 1.执行以下命令编辑 `ks-installer` ~~~powershell # kubectl edit cc -n kubesphere-system ks-installer ~~~ 2.搜寻至 `devops`。添加字段 `sonarqube` 并在其下方指定 `externalSonarUrl` 和 `externalSonarToken`。 ~~~powershell devops: enabled: true jenkinsJavaOpts_MaxRAM: 2g jenkinsJavaOpts_Xms: 512m jenkinsJavaOpts_Xmx: 512m jenkinsMemoryLim: 2Gi jenkinsMemoryReq: 1500Mi jenkinsVolumeSize: 8Gi sonarqube: externalSonarUrl: http://192.168.10.141:30058 externalSonarToken: c3ee79a4c4b25bd03867db735dc439e3bb726426 ~~~ 3.完成操作后保存此文件 ### 4.4.5 将 SonarQube 服务器添加至 Jenkins 1.执行以下命令获取 Jenkins 的地址 ~~~powershell # export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services devops-jenkins) ~~~ ~~~powershell # export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}") ~~~ ~~~powershell # echo http://$NODE_IP:$NODE_PORT ~~~ 2.可以获得以下输出，获取 Jenkins 的端口号 ~~~powershell http://192.168.10.141:30180 ~~~ 3.请使用地址 `http://:30180` 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 `admin/P@88w0rd`）登录 Jenkins。有关配置 Jenkins 的更多信息，请参见 [Jenkins 系统设置](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/jenkins-setting/)。 > 取决于您的实例的部署位置，您可能需要设置必要的端口转发规则，并在您的安全组中放行端口 `30180`，以便访问 Jenkins。 ![image-20221027183806808](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027183806808.png) 4.可以前往**系统管理**下的 **Manage Credentials** 并点击 **Stores scoped to Jenkins** 下的 **Jenkins**，再点击**全局凭据 (unrestricted)**，然后点击左侧导航栏的**添加凭据**，参考上方第二张截图用 SonarQube 管理员令牌添加凭证。添加凭证后，从 **Server authentication token** 旁边的下拉列表中选择该凭证。 ![image-20221027185005042](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185005042.png) ![image-20221027185200948](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185200948.png) ![image-20221027185246499](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185246499.png) ![image-20221027185323676](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185323676.png) ![image-20221027185345516](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185345516.png) ![image-20221027185807991](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185807991.png) ![image-20221027185836783](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027185836783.png) 5.点击左侧导航栏中的**系统管理** ![image-20221027184323977](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027184323977.png) 6.向下翻页找到并点击**系统配置** ![image-20221027184409988](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027184409988.png) 6.搜寻到**SonarQube Server**,然后点击Add SonarQube ![image-20221027184516685](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027184516685.png) 7.输入 **Name** 和 **Server URL** (`http://:`)。点击**添加**，选择 **Jenkins**，然后在弹出的对话框中用 SonarQube 管理员令牌创建凭证（如下方第二张截图所示）。创建凭证后，从 **Server authentication token** 旁边的下拉列表中选择该凭证。点击**应用**完成操作。 ![image-20221027190142850](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027190142850.png) ### 4.4.6 将 sonarqubeURL 添加到 KubeSphere 控制台 您需要指定 `sonarqubeURL`，以便可以直接从 KubeSphere 控制台访问 SonarQube 1.执行以下命令 ~~~powershell # kubectl edit cm -n kubesphere-system ks-console-config ~~~ 2.搜寻到 `data.client.enableKubeConfig`，在下方添加 `devops` 字段并指定 `sonarqubeURL` ~~~powershell apiVersion: v1 data: local_config.yaml: | server: http: hostname: localhost port: 8000 static: production: /public: server/public /assets: dist/assets /dist: dist redis: port: 6379 host: redis.kubesphere-system.svc redisTimeout: 5000 sessionTimeout: 7200000 client: version: kubesphere: v3.1.0 kubernetes: v1.24.0 openpitrix: v3.1.0 enableKubeConfig: true devops: 手动添加 sonarqubeURL: http://192.168.10.141:30058 defaultClusterName: default ~~~ 3.保存该文件 ### 4.4.7 重启服务 ~~~powershell # kubectl -n kubesphere-devops-system rollout restart deploy devops-apiserver ~~~ ~~~powershell # kubectl -n kubesphere-system rollout restart deploy ks-console ~~~ ### 4.4.8 为新项目创建 SonarQube Token 需要一个 SonarQube 令牌，以便您的流水线可以在运行时与 SonarQube 通信。 1.在 SonarQube 控制台上，点击 **Create new project**。 ![image-20221027225949333](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027225949333.png) ![image-20221027230313253](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230313253.png) ![image-20221027230416982](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230416982.png) ![image-20221027230540439](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230540439.png) ![image-20221027230601587](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230601587.png) ~~~powershell 4c5c650e18a80f699d5301773230773ad6e17924 ~~~ 2.创建令牌后，点击 **Continue** ![image-20221027230705284](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230705284.png) 3.分别选择 **Java** 和 **Maven**。 > 复制下图所示绿色框中的序列号，如果要在流水线中使用，则需要在[凭证](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/credential-management/#创建凭证)中添加此序列号。 ![image-20221027230847094](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230847094.png) ![image-20221027230917037](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027230917037.png) ~~~powershell mvn sonar:sonar \ -Dsonar.projectKey=java-demo \ -Dsonar.host.url=http://192.168.10.141:30058 \ -Dsonar.login=4c5c650e18a80f699d5301773230773ad6e17924 ~~~ # 五、为 KubeSphere 流水线设置电子邮件服务器 内置 Jenkins 无法与平台通知系统共享相同的电子邮件配置。因此，您需要单独为 KubeSphere DevOps 流水线配置电子邮件服务器设置。 ## 5.1 准备工作 - 您需要启用 [KubeSphere DevOps 系统](https://v3-2.docs.kubesphere.io/zh/docs/pluggable-components/devops/)。 - 您需要一个具有**集群管理**权限的帐户。例如，您可以直接以 `admin` 身份登录控制台或者创建具有该权限的新角色并将该角色分配给一个用户。 ## 5.2 设置电子邮件服务器 1.点击左上角的**平台管理**，然后选择**集群管理**。 2.如果您已经启用[多集群功能](https://v3-2.docs.kubesphere.io/zh/docs/multicluster-management/)并已导入成员集群，那么您可以选择一个特定集群以查看其节点。如果尚未启用该功能，请直接参考下一步。 3.转到**应用负载**下的**工作负载**，然后从下拉列表中选择 **kubesphere-devops-system** 项目。点击 `devops-jenkins` 右侧的 ![img](https://v3-2.docs.kubesphere.io/images/docs/common-icons/three-dots.png) 并选择**编辑 YAML** 以编辑其 YAML 配置文件。 ![image-20221027232346190](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027232346190.png) 4.向下滚动到下图所示的需要指定的字段。完成修改后，点击**确定**以保存。 >在 `devops-jenkins` 部署 (Deployment) 中修改电子邮件服务器后，它会重新启动。因此，DevOps 系统将在几分钟内不可用，请在适当的时候进行此类修改。 ![设置电子邮件](../../img/kubernetes/kubernetes_kubesphere_devops/set-jenkins-email.png) | 环境变量名称 | 描述信息 | | :-------------- | :------------------------ | | EMAIL_SMTP_HOST | SMTP 服务器地址 | | EMAIL_SMTP_PORT | SMTP 服务器端口（如：25） | | EMAIL_FROM_ADDR | 电子邮件发件人地址 | | EMAIL_FROM_NAME | 电子邮件发件人姓名 | | EMAIL_FROM_PASS | 电子邮件发件人密码 | | EMAIL_USE_SSL | 是否启用 SSL 配置 | 实际配置： ![image-20221027232704794](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027232704794.png) ![image-20221027232744625](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221027232744625.png) # 六、使用图形编辑面板创建流水线 KubeSphere 中的图形编辑面板包含用于 Jenkins [阶段 (Stage)](https://www.jenkins.io/zh/doc/book/pipeline/#阶段) 和[步骤 (Step)](https://www.jenkins.io/zh/doc/book/pipeline/#步骤) 的所有必要操作。您可以直接在交互式面板上定义这些阶段和步骤，无需创建任何 Jenkinsfile。 本教程演示如何在 KubeSphere 中使用图形编辑面板创建流水线。KubeSphere 在整个过程中将根据您在编辑面板上的设置自动生成 Jenkinsfile，您无需手动创建 Jenkinsfile。待流水线成功运行，它会相应地在您的开发环境中创建一个部署 (Deployment) 和一个服务 (Service)，并将镜像推送至 Docker Hub。 ## 6.1 准备工作 - 您需要[启用 KubeSphere DevOps 系统](https://v3-2.docs.kubesphere.io/zh/docs/pluggable-components/devops/)。 - 您需要有一个 [Docker Hub](http://www.dockerhub.com/) 帐户。 - 您需要创建一个企业空间、一个 DevOps 项目和一个用户 (`project-regular`)，必须邀请该用户至 DevOps 项目中并赋予 `operator` 角色。如果尚未创建，请参见[创建企业空间、项目、用户和角色](https://v3-2.docs.kubesphere.io/zh/docs/quick-start/create-workspace-and-project/)。 - 设置 CI 专用节点来运行流水线（可选）。有关更多信息，请参见[为缓存依赖项设置 CI 节点](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/set-ci-node/)。 - 配置您的电子邮件服务器用于接收流水线通知（可选）。有关更多信息，请参见[为 KubeSphere 流水线设置电子邮件服务器](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/jenkins-email/)。 - 配置 SonarQube 将代码分析纳入流水线中（可选）。有关更多信息，请参见[将 SonarQube 集成到流水线](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-integrate/sonarqube/)。 ## 6.2 流水线概述 本示例流水线包括以下六个阶段。 ![Pipeline](../../img/kubernetes/kubernetes_kubesphere_devops/20190516091714.png) - **阶段 1：Checkout SCM**：从 GitHub 仓库拉取源代码。 - **阶段 2：单元测试**：待该测试通过后才会进行下一阶段。 - **阶段 3：代码分析**：配置 SonarQube 用于静态代码分析。 - **阶段 4：构建并推送**：构建镜像并附上标签 `snapshot-$BUILD_NUMBER` 推送至 Docker Hub，其中 `$BUILD_NUMBER` 是流水线活动列表中的记录的序列号。 - **阶段 5：制品**：生成一个制品（JAR 文件包）并保存。 - **阶段 6：部署至开发环境**：在开发环境中创建一个部署和一个服务。该阶段需要进行审核，部署成功运行后，会发送电子邮件通知。 ## 6.3 流水线设置 ### 6.3.1 创建凭证 1.以 `project-regular` 身份登录 KubeSphere 控制台。转到您的 DevOps 项目，在 **DevOps 项目设置**下的**凭证**页面创建以下凭证。有关如何创建凭证的更多信息，请参见[凭证管理](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/credential-management/)。 > 如果您的帐户或密码中有任何特殊字符，例如 `@` 和 `$`，可能会因为无法识别而在流水线运行时导致错误。在这种情况下，您需要先在一些第三方网站（例如 [urlencoder](https://www.urlencoder.org/)）上对帐户或密码进行编码，然后将输出结果复制粘贴作为您的凭证信息。 | 凭证 ID | 类型 | 用途 | | :-------------- | :----------- | :--------- | | dockerhub-id | 用户名和密码 | Docker Hub | | demo-kubeconfig | kubeconfig | Kubernetes | 2.您还需要为 SonarQube 创建一个凭证 ID (`sonar-token`)，用于上述的阶段 3（代码分析）。请参阅[为新项目创建 SonarQube 令牌 (Token)](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-integrate/sonarqube/#create-sonarqube-token-for-new-project)，在**访问令牌**类型的凭证的**令牌**字段中输入 SonarQube 令牌。点击**确定**完成操作。 ![image-20221028102436728](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028102436728.png) ![image-20221028102649494](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028102649494.png) ![image-20221028102709409](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028102709409.png) 3.你在列表中可以看到4个凭证 ### 6.3.2 创建项目 在本教程中，示例流水线会将 [sample](https://github.com/kubesphere/devops-maven-sample/tree/sonarqube) 应用部署至一个项目。因此，您必须先创建一个项目（例如 `kubesphere-sample-dev`）。待流水线成功运行，会在该项目中自动创建该应用的部署和服务。 您可以使用 `project-admin` 帐户创建项目。此外，该用户也是 CI/CD 流水线的审核员。请确保将 `project-regular` 帐户邀请至该项目并授予 `operator` 角色。有关更多信息，请参见[创建企业空间、项目、用户和角色](https://v3-2.docs.kubesphere.io/zh/docs/quick-start/create-workspace-and-project/)。 ![image-20221028000558491](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000558491.png) ![image-20221028000639889](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000639889.png) ![image-20221028000734548](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000734548.png) ![image-20221028000754528](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000754528.png) ![image-20221028000845544](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000845544.png) ![image-20221028000919311](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000919311.png) ![image-20221028000959470](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028000959470.png) ![image-20221028001015375](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001015375.png) ![image-20221028001042749](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001042749.png) ### 6.3.3 创建流水线 1.请确保以 `project-regular` 身份登录 KubeSphere 控制台，转到您的 DevOps 项目。在**流水线**页面点击**创建**。 ![image-20221028001349754](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001349754.png) ![image-20221028001415837](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001415837.png) 2.在弹出的对话框中，将它命名为 `graphical-pipeline`，点击**下一步**。 ![image-20221028001512033](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001512033.png) ![image-20221028001554927](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001554927.png) 3.在**高级设置**页面，点击**添加**，添加以下三个字符串参数。这些参数将用于流水线的 Docker 命令。添加完成后，点击**创建**。 ![image-20221028001822672](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001822672.png) | 参数类型 | 名称 | 值 | 描述信息 | | :------- | :------------------ | :-------------- | :----------------------------------------- | | 字符串 | REGISTRY | `docker.io` | 镜像仓库地址。本示例使用 `docker.io`。 | | 字符串 | DOCKERHUB_NAMESPACE | Docker ID | 您的 Docker Hub 帐户或该帐户下的组织名称。 | | 字符串 | APP_NAME | `devops-sample` | 应用名称。 | ![image-20221028001953606](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028001953606.png) ![image-20221028002324916](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028002324916.png) >有关其他字段，请直接使用默认值或者参考[流水线设置](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/pipeline-settings/)以自定义配置。 4.创建的流水线会显示在列表中 ![image-20221028002447067](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028002447067.png) ### 6.3.4 编辑流水线 - 点击流水线进入其详情页面。要使用图形编辑面板，请点击**任务状态**选项卡下的**编辑流水线**。在弹出的对话框中，点击**自定义流水线**。该流水线包括六个阶段，请按照以下步骤设置每个阶段。 - 您也可以点击**持续集成 (CI)** 和**持续集成&交付 (CI/CD)** 来使用 KubeSphere 提供的[内置流水线模板](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/use-pipeline-templates/)。 ![image-20221028003020673](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028003020673.png) ![image-20221028003121067](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028003121067.png) >流水线详情页面会显示**同步状态**，它是 KubeSphere 和 Jenkins 之间的同步结果。若同步成功，您会看到**成功**图标。您也可以点击**编辑 Jenkinsfile** 手动为流水线创建一个 Jenkinsfile。 #### 6.3.4.1 阶段 1：拉取源代码 (Checkout SCM) 图形编辑面板包括两个区域：左侧的**画布**和右侧的**内容**。它会根据您对不同阶段和步骤的配置自动生成一个 Jenkinsfile，为开发者提供更加用户友好的操作体验。 >流水线包括[声明式流水线](https://www.jenkins.io/zh/doc/book/pipeline/syntax/#声明式流水线)和[脚本化流水线](https://www.jenkins.io/zh/doc/book/pipeline/syntax/#脚本化流水线)。目前，您可以使用该面板创建声明式流水线。有关流水线语法的更多信息，请参见 [Jenkins 文档](https://www.jenkins.io/zh/doc/book/pipeline/syntax/)。 1.在图形编辑面板上，从**类型**下拉列表中选择 **node**，从 **Label** 下拉列表中选择 **maven**。 ![image-20221028003238002](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028003238002.png) > `agent` 用于定义执行环境。`agent` 指令指定 Jenkins 执行流水线的位置和方式。有关更多信息，请参见[选择 Jenkins Agent](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-use/choose-jenkins-agent/)。 2.请点击左侧的加号图标来添加阶段。点击**添加步骤**上方的文本框，然后在右侧的**名称**字段中为该阶段设置名称（例如 `Checkout SCM`）。 ![image-20221028003607622](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028003607622.png) 3.点击**添加步骤**。在列表中选择 **git**，以从 GitHub 拉取示例代码。在弹出的对话框中，填写必需的字段。点击**确定**完成操作。 - **URL**：输入 GitHub 仓库地址 `https://github.com/kubesphere/devops-maven-sample.git`。请注意，这里是示例地址，您需要使用您自己的仓库地址。 - **凭证 ID**：本教程中无需输入凭证 ID。 - **分支**：如果您将其留空，则默认为 master 分支。请输入 `sonarqube`，或者如果您不需要代码分析阶段，请将其留空。 ![image-20221028004820884](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028004820884.png) ![image-20221028004912412](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028004912412.png) ![image-20221028004955059](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028004955059.png) ![image-20221028005248406](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028005248406.png) ~~~powershell https://github.com/nextgomsb/devops-maven-sample.git ~~~ **后续执行过程中有问题时，可以尝试修改容器基础镜像** ![image-20221031104243656](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031104243656.png) **创建流水线** ![image-20221028005413460](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028005413460.png) ![image-20221028010524003](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028010524003.png) 4.第一阶段设置完成 ![image-20221028005802017](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028005802017.png) #### 6.3.4.2 阶段 2：单元测试 1.点击阶段 1 右侧的加号图标添加新的阶段，以在容器中执行单元测试。将它命名为 `Unit Test`。 ![image-20221028010703560](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028010703560.png) ![image-20221028010805488](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028010805488.png) 2.点击**添加步骤**，在列表中选择**指定容器**。将其命名为 `maven` 然后点击**确定**。 ![image-20221028010930565](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028010930565.png) ![image-20221028010953830](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028010953830.png) 3.点击**添加嵌套步骤**，在 `maven` 容器下添加一个嵌套步骤。在列表中选择 **shell** 并在命令行中输入以下命令。点击**确定**保存操作。 ![image-20221028011117734](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028011117734.png) ![image-20221028011154458](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028011154458.png) ![image-20221030205259256](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030205259256.png) ~~~powershell mvn clean package ~~~ >您可以在图形编辑面板上指定在给定阶段指令中执行的一系列[步骤](https://www.jenkins.io/zh/doc/book/pipeline/syntax/#steps)。 #### 6.3.4.3 阶段 3：代码分析（可选） 本阶段使用 SonarQube 来测试您的代码。如果您不需要代码分析，可以跳过该阶段。 1.点击 `Unit Test` 阶段右侧的加号图标添加一个阶段，以在容器中进行 SonarQube 代码分析。将它命名为 `Code Analysis`。 ![image-20221028101410290](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028101410290.png) 2.在 **Code Analysis** 中，点击**任务**下的**添加步骤**，选择**指定容器**。将其命名为 `maven` 然后点击**确定**。 ![image-20221028101510119](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028101510119.png) ![image-20221028101631884](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028101631884.png) ![image-20221028101709785](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028101709785.png) 3.点击 `maven` 容器下的**添加嵌套步骤**，以添加一个嵌套步骤。点击**添加凭证**并从**凭证 ID** 列表中选择 SonarQube 令牌 (`sonar-token`)。在**文本变量**中输入 `SONAR_TOKEN`，然后点击**确定**。 ![image-20221028101835008](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028101835008.png) ![image-20221028101905088](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028101905088.png) ![image-20221028102927908](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028102927908.png) 4.在**添加凭证**步骤下，点击**添加嵌套步骤**为其添加一个嵌套步骤。 ![image-20221028103940613](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028103940613.png) 5.点击 **Sonarqube 配置**，在弹出的对话框中保持默认名称 `sonar` 不变，点击**确定**保存操作 ![image-20221028104138270](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104138270.png) ![image-20221028104207816](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104207816.png) 6.在 **Sonarqube 配置**步骤下，点击**添加嵌套步骤**为其添加一个嵌套步骤。 ![image-20221028104342813](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104342813.png) 7.点击 **shell** 并在命令行中输入以下命令，用于 sonarqube 分支和认证，点击**确定**完成操作。 ![image-20221028104524829](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104524829.png) ![image-20221028104701245](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104701245.png) ~~~powershell mvn sonar:sonar -Dsonar.login=$SONAR_TOKEN ~~~ 8.点击**指定容器**步骤下的**添加嵌套步骤**（第三个），选择**超时**。在时间中输入 `1` 并将单位选择为**小时**，点击**确定**完成操作。 ![image-20221028104841540](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104841540.png) ![image-20221028104953045](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028104953045.png) ![image-20221028105020275](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028105020275.png) 9.点击**超时**步骤下的**添加嵌套步骤**，选择**代码质量检查 (SonarQube)**。在弹出的对话框中选择**检查通过后开始后续任务**。点击**确定**保存操作。 ![image-20221028105118051](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028105118051.png) ![image-20221028105202924](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028105202924.png) ![image-20221028105244782](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028105244782.png) ![image-20221028105324072](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028105324072.png) #### 6.3.4.4 阶段 4：构建并推送镜像 点击前一个阶段右侧的加号图标添加一个新的阶段，以构建并推送镜像至 Docker Hub。将其命名为 `Build and Push`。 ![image-20221028105532996](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028105532996.png) 2.点击**任务**下的**添加步骤**，选择**指定容器**，将其命名为 `maven`，然后点击**确定**。 ![image-20221028110037319](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110037319.png) 3.点击 `maven` 容器下的**添加嵌套步骤**添加一个嵌套步骤。在列表中选择 **shell** 并在弹出窗口中输入以下命令，点击**确定**完成操作。 ![image-20221028110434696](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110434696.png) ![image-20221028110514370](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110514370.png) ~~~powershell mvn -Dmaven.test.skip=true clean package ~~~ ![image-20221028110621884](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110621884.png) 4.再次点击**添加嵌套步骤**，选择 **shell**。在命令行中输入以下命令，以根据 [Dockerfile](https://github.com/kubesphere/devops-maven-sample/blob/sonarqube/Dockerfile-online) 构建 Docker 镜像。点击**确定**确认操作。 > 请勿遗漏命令末尾的点 `.` ![image-20221028110746069](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110746069.png) ![image-20221028110832612](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110832612.png) ~~~powershell docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER . ~~~ ![image-20221028110943337](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028110943337.png) 5.再次点击**添加嵌套步骤**，选择**添加凭证**。在弹出的对话框中填写以下字段，点击**确定**确认操作。 - **凭证名称**：选择您创建的 Docker Hub 凭证，例如 `dockerhub-id`。 - **密码变量**：输入 `DOCKER_PASSWORD`。 - **用户名变量**：输入 `DOCKER_USERNAME`。 >出于安全原因，帐户信息在脚本中显示为变量。 ![image-20221028111055981](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028111055981.png) ![image-20221028111123696](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028111123696.png) ![image-20221028111252997](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028111252997.png) 6.在**添加凭证**步骤中点击**添加嵌套步骤**（第一个）。选择 **shell** 并在弹出窗口中输入以下命令，用于登录 Docker Hub。点击**确定**确认操作。 ![image-20221028111539496](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028111539496.png) ![image-20221028111622087](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028111622087.png) ~~~powershell echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin ~~~ ![image-20221028111712445](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028111712445.png) 7.在**添加凭证**步骤中点击**添加嵌套步骤**。选择 **shell** 并输入以下命令，将 SNAPSHOT 镜像推送至 Docker Hub。点击**确定**完成操作。 ![image-20221028112001528](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112001528.png) ~~~powershell docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER ~~~ ![image-20221028112053455](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112053455.png) ![image-20221028112123076](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112123076.png) #### 6.3.4.5 阶段 5：生成制品 1.点击 **Build and Push** 阶段右侧的加号图标添加一个新的阶段，以保存制品，将其命名为 `Artifacts`。本示例使用 JAR 文件包。 ![image-20221028112332849](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112332849.png) 2.选中 **Artifacts** 阶段，点击**任务**下的**添加步骤**，选择**保存制品**。在弹出的对话框中输入 `target/*.jar`，用于设置 Jenkins 中制品的保存路径。点击**确定**完成操作。 ![image-20221028112459764](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112459764.png) ![image-20221028112521317](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112521317.png) ~~~powershell target/*.jar ~~~ ![image-20221028112612608](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112612608.png) #### 6.3.4.6 阶段 6：部署至开发环境 1.点击 **Artifacts** 阶段右侧的加号图标添加最后一个阶段，将其命名为 `Deploy to Dev`。该阶段用于将资源部署至您的开发环境（即 `kubesphere-sample-dev` 项目）。 ![image-20221028112843130](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112843130.png) 2.点击 **Deploy to Dev** 阶段下的**添加步骤**，在列表中选择**审核**，然后在**消息**字段中填入 `@project-admin`，即 `project-admin` 帐户在流水线运行到该阶段时会进行审核。点击**确定**保存操作。 ![image-20221028112947342](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028112947342.png) ![image-20221028113026666](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113026666.png) > 在 KubeSphere 3.2.x 中，能够运行流水线的帐户也能够继续或终止该流水线。此外，流水线创建者、拥有该项目管理员角色的用户或者您指定的帐户也有权限继续或终止流水线。 3.再次点击 **Deploy to Dev** 阶段下的**添加步骤**。在列表中选择**指定容器**，将其命名为 `maven` 然后点击**确定**。 ![image-20221028113214211](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113214211.png) 4.点击 `maven` 容器步骤下的**添加嵌套步骤**。在列表中选择**添加凭证**，在弹出的对话框中填写以下字段，然后点击**确定**。 - 凭证名称：选择您创建的 kubeconfig 凭证，例如 `demo-kubeconfig`。 - kubeconfig 变量：输入 `KUBECONFIG_CONTENT`。 ![image-20221028113256977](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113256977.png) ![image-20221028113408653](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113408653.png) 5.点击**添加凭证**步骤下的**添加嵌套步骤**。在列表中选择 **shell**，在弹出的对话框中输入以下命令，然后点击**确定**。 ![image-20221028113510007](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113510007.png) ![image-20221028113533301](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113533301.png) ~~~powershell mkdir ~/.kube echo "$KUBECONFIG_CONTENT" > ~/.kube/config envsubst < deploy/dev-ol/devops-sample-svc.yaml | kubectl apply -f - envsubst < deploy/dev-ol/devops-sample.yaml | kubectl apply -f - ~~~ ![image-20221028113628854](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113628854.png) 6.如果您想在流水线成功运行时接收电子邮件通知，请点击**添加步骤**，选择**邮件**，以添加电子邮件信息。请注意，配置电子邮件服务器是可选操作，如果您跳过该步骤，依然可以运行流水线。 ![image-20221028113728197](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113728197.png) ![image-20221028113805090](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113805090.png) ![image-20221028113952848](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028113952848.png) 7.待您完成上述步骤，请在右下角点击**保存**。随后，您可以看到该流水线有完整的工作流，并且每个阶段也清晰列示。当您用图形编辑面板定义流水线时，KubeSphere 会自动创建相应的 Jenkinsfile。点击**编辑 Jenkinsfile** 查看该 Jenkinsfile。 >在**流水线**页面，您可以点击该流水线右侧的 ![img](https://v3-2.docs.kubesphere.io/images/docs/common-icons/three-dots.png)，然后选择**复制**来创建该流水线的副本。如果您需要同时运行多个不包含多分支的流水线，您可以全部选中这些流水线，然后点击**运行**来批量运行它们。 ## 6.4 运行流水线 1.您需要手动运行使用图形编辑面板创建的流水线。点击**运行**，您可以在弹出的对话框中看到步骤 3 中已定义的三个字符串参数。点击**确定**来运行流水线。 ![image-20221028114448532](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028114448532.png) ![image-20221031102324133](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031102324133.png) 2.要查看流水线的状态，请转到**运行记录**选项卡，点击您想查看的记录。 ![image-20221028114605201](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221028114605201.png) 3.稍等片刻，流水线如果成功运行，则会在 **Deploy to Dev** 阶段停止。`project-admin` 作为流水线的审核员，需要进行审批，然后资源才会部署至开发环境。 ## 6.5 下载制品 点击**制品**选项卡，然后点击右侧的图标下载该制品。 ![image-20221030131750542](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030131750542.png) ## 6.6 查看代码分析结果 在**代码检查**页面，可以查看由 SonarQube 提供的本示例流水线的代码分析结果。如果您没有事先配置 SonarQube，则该部分不可用。有关更多信息，请参见[将 SonarQube 集成到流水线](https://v3-2.docs.kubesphere.io/zh/docs/devops-user-guide/how-to-integrate/sonarqube/)。 ![SonarQube 详细结果](../../img/kubernetes/kubernetes_kubesphere_devops/sonarqube_result_detail.png) ## 6.7 验证 Kubernetes 资源 1.如果流水线的每个阶段都成功运行，则会自动构建一个 Docker 镜像并推送至您的 Docker Hub 仓库。最终，流水线将在您事先设置的项目中自动创建一个部署和一个服务。 2.前往该项目（本教程中即 `kubesphere-sample-dev`），请点击**应用负载**下的**工作负载**，您可以看到列表中显示的部署。 3.在**服务**页面，您可以看到示例服务通过 NodePort 暴露其端口号。要访问服务，请访问 `:`。 > 访问服务前，您可能需要配置端口转发规则并在安全组中放行该端口。 4.现在流水线已成功运行，将会推送一个镜像至 Docker Hub。登录 Docker Hub 查看结果。 ![DockerHub 镜像](../../img/kubernetes/kubernetes_kubesphere_devops/dockerhub_image.png) 5.该应用的名称为 `devops-sample`，即 `APP_NAME` 的值，标签即 `SNAPSHOT-$BUILD_NUMBER` 的值。`$BUILD_NUMBER` 即**运行记录**选项卡列示的记录的序列号。 6.如果您在最后一个阶段设置了电子邮件服务器并添加了电子邮件通知的步骤，您还会收到电子邮件消息。 # 七、配置流水线把项目发布到生产环境 ![流水线概览](../../img/kubernetes/kubernetes_kubesphere_devops/pipeline-overview.png) ## 7.1 创建GitHub tekens >如需创建 GitHub 个人访问令牌，请转到您 GitHub 帐户的 **Settings**，点击 **Developer settings**，选择 **Personal access tokens**，然后点击 **Generate new token**。 ![image-20221030231946644](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030231946644.png) ![image-20221030232050046](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232050046.png) ![image-20221030232332862](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232332862.png) ![image-20221030232439488](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232439488.png) ![image-20221030232504237](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232504237.png) ![image-20221030232743591](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232743591.png) ![image-20221030232825182](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232825182.png) ![image-20221030232906699](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030232906699.png) **以project-regular用户登录后创建github访问凭证** ![image-20221030233212703](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030233212703.png) ![image-20221030233317075](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030233317075.png) ![image-20221030233338927](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030233338927.png) ## 7.2 GitHub创建代码仓库 ![image-20221031001513477](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031001513477.png) ![image-20221031001740425](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031001740425.png) ## 7.3 创建项目 使用project-admin用户登录WEB控制器，创建kubesphere-sample-prod项目，并邀请project-regular用户成为项目成员，并为operator角色。 ![image-20221030221900997](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030221900997.png) ![image-20221030222059644](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030222059644.png) ![image-20221030222143880](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030222143880.png) ![image-20221030222226462](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030222226462.png) ## 7.4 为发布到生产环境准备TAG_NAME变量 使用project-regular用户登录，并编辑`graphical-pipeline`，添加TAG_NAME,值为v0.0.1 ![image-20221030222449504](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030222449504.png) ![image-20221030222635309](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030222635309.png) ![image-20221030222719291](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030222719291.png) ![image-20221030225055086](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030225055086.png) ## 7.5 阶段7: 提交镜像和代码至仓库(push with tag) ### 7.5.1 添加条件 ![image-20221031004855745](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004855745.png) ![image-20221031004936717](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004936717.png) ![image-20221031005012420](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005012420.png) ~~~powershell scriptBlock return params.TAG_NAME =~ /v.*/ ~~~ ![image-20221031005035219](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005035219.png) ![image-20221031005113742](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005113742.png) ### 7.5.2 添加步骤 ![image-20221031005142758](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005142758.png) ![image-20221031005320532](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005320532.png) ![image-20221031005342665](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005342665.png) #### 7.5.2.1 添加审核 ![image-20221031005743582](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005743582.png) ![image-20221031005819223](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005819223.png) ~~~powershell release image with tag? @project-admin ~~~ ![image-20221031005844113](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031005844113.png) ![image-20221031010317533](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010317533.png) #### 7.5.2.2 添加凭证 ![image-20221031010043691](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010043691.png) ![image-20221030234957787](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030234957787.png) #### 7.5.2.3 添加GitHub配置 ![image-20221031010619701](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010619701.png) ![image-20221031010657851](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010657851.png) ~~~powershell git config --global user.email "nextgo@126.com" ~~~ ![image-20221031010717615](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010717615.png) ![image-20221031010746655](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010746655.png) ~~~powershell git config --global user.name "nextgo" ~~~ ![image-20221031010816798](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010816798.png) ![image-20221031010852017](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010852017.png) ![image-20221031010935558](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031010935558.png) ~~~powershell git tag -a $TAG_NAME -m "$TAG_NAME" ~~~ ![image-20221031011004651](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011004651.png) **提前编辑流水线，添加如下变量** ![image-20221030235404156](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221030235404156.png) ![image-20221031011135117](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011135117.png) ![image-20221031011155995](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011155995.png) ~~~powershell git push http://$GIT_USERNAME:$GITHUB_TOKEN@github.com/$GITHUB_ACCOUNT/devops-java-sample.git --tags --ipv4 ~~~ ![image-20221031011228502](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011228502.png) #### 7.5.2.4 添加DockerHub配置 ![image-20221031011355810](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011355810.png) ![image-20221031011426526](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011426526.png) ~~~powershell docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:v$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ~~~ ![image-20221031011454860](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011454860.png) ![image-20221031011526454](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011526454.png) ![image-20221031011549068](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011549068.png) ~~~powershell docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ~~~ ![image-20221031011618546](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011618546.png) ![image-20221031011650811](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031011650811.png) ## 7.6 阶段8: 发布到生产环境（deploy to production） ### 7.6.1 添加条件 ![image-20221031003324495](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003324495.png) ![image-20221031003349884](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003349884.png) ![image-20221031003416608](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003416608.png) ~~~powershell scriptBlock return params.TAG_NAME =~ /v.*/ ~~~ ![image-20221031003516001](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003516001.png) ### 7.6.2 添加审核 ![image-20221031003611345](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003611345.png) ![image-20221031003641520](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003641520.png) ~~~powershell deploy to production? @project-admin ~~~ ![image-20221031003712803](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003712803.png) ### 7.6.3 添加步骤 ![image-20221031003804371](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031003804371.png) ![image-20221031004025210](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004025210.png) ![image-20221031004052139](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004052139.png) ![image-20221031004237267](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004237267.png) #### 7.6.3.1 添加凭证 ![image-20221031004318639](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004318639.png) ![image-20221031004202774](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004202774.png) #### 7.6.3.2 添加执行步骤 ![image-20221031004500945](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004500945.png) ![image-20221031004533534](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004533534.png) ~~~powershell envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f - ~~~ ![image-20221031004601615](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031004601615.png) ## 7.7 执行流水线 ![image-20221031103744239](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031103744239.png) ![image-20221031103820943](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031103820943.png) ![image-20221031103906025](../../img/kubernetes/kubernetes_kubesphere_devops/image-20221031103906025.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_kurator.md ================================================ # kubernetes集群舰队管理 Kurator # 一、Kurator介绍 ## 1.1 Kurator是什么？ 2023年4月8日，Kurator正式发布v0.3.0版本。 Kurator 是华为云推出的分布式云原生开源套件，通过集成业界主流开源技术栈，帮助用户一站式构建专属的分布式云原生基础设施，助力企业业务跨云跨边、分布式化升级。 Kurator v0.2 版本已具备管理多云基础设施和异构基础设施的核心能力，通过引入Cluster Operator组件，支持“AWS自建集群”和“本地数据中心集群”包括集群创建、清理等在内的生命周期管理特性。 在最新发布的v0.3.0版本中，Cluster Operator不仅分别对两类集群的生命周期管理能力进行了增强，也将v0.2.0版本中的多个API对象抽象成一个API对象cluster，方便用户使用。 同时，在 cluster 对象基础上，Kurator引入了舰队的概念。每个舰队代表一个物理集群组，方便Kurator未来进行统一的编排、调度，统一的流量治理以及统一的监控运维。目前，Kurator的舰队管理支持多个特性，包括舰队控制平面的生命周期管理，以及集群注册和注销到舰队等。 至此，Kurator 通过集群舰队统一了集群视图。这意味着，Kurator开始具备了支持用户体验一致地管理分布在任何云上的集群的能力，从而进一步协助用户的分布式云原生架构的升级。 ## 1.2 Kurator v0.3.0关键特性介绍 - 集群生命周期管理能力增强 Kurator 通过 Cluster Operator组件对集群的生命周期进行管理。基于Cluster API，Cluster Operator 不仅可以管理集群生命周期，还统一并简化了创建集群所需的配置，为用户在不同云平台上管理集群提供了简单易用的API。目前Cluster Operator 支持“本地数据中心集群”和“AWS自建集群”。 - 本地数据中心集群 Kurator基于kubespray对本地数据中心集群的生命周期进行管理。与 kubespray不同的地方在于，Kurator采用了更易于理解和配置的云原生声明式的方法管理集群。 相比较v0.2.0版本，v0.3.0版本的Kurator 带来了以下几个方面的增强特性： - 批量的工作节点扩缩容。 现在Kurator支持以声明式的方式，在原有集群上增加、删除或者替换多个工作节点。 用户只需要声明集群所需要的最终工作节点状态，Kurator即可在没有任何外部干预的情况下完成节点的批量扩缩容。 - 集群版本升级。 用户可以在API对象上声明要升级到的Kubernetes版本，Kurator便会自动对目标集群的各节点进行升级。 - 增强集群控制面高可用。Kurator为用户提供了一种基于VIP的增强集群控制面高可用的方案。在这种方案下，Kurator利用kube-vip的能力，使用VIP实现跨多个控制平面副本的入站流量负载均衡。 ![img](../../img/kubernetes/kubernetes_kurator/v2-298182337030964ef3d83ef70beb747d_r.jpg) ## 1.3 云原生舰队管理 Kurator引入了代表物理集群组的逻辑单元“舰队”，旨在一致地管理一组集群，舰队允许您轻松、一致地管理分布在任何云中的集群。 Kurator通过Fleet Manager实现了对舰队控制平面生命周期管理，并且可以轻松的将集群添加到或从舰队中删除。 未来，Kurator将支持Fleet级的应用编排，并且在Fleet的所有集群中提供统一的命名空间、ServiceAccount、Service，以支持在多个集群之间实现服务发现和通信。此外，Kurator将汇总来自所有集群的监控指标。 Kurator Fleet Manager作为一个Kubernetes Operator运行，负责Fleet控制平面生命周期管理，也负责集群的注册和注销。 ![img](https://pic3.zhimg.com/v2-9b621d72a7b926276742fce0f9e78aae_r.jpg) # 二、Kurator部署 ## 2.1 k8s管理集群准备 > 使用kubespray或kubeasz快速部署一套K8S集群用于实现对其它集群进行管理操作。 ~~~powershell [root@kubespray ~]# cat > /etc/yum.repos.d/k8s.repo < 40d v1.26.3 node5 Ready 40d v1.26.3 ~~~ ## 2.2 安装Kurator客户端 ![image-20230614101114746](../../img/kubernetes/kubernetes_kurator/image-20230614101114746.png) ![image-20230614101147258](../../img/kubernetes/kubernetes_kurator/image-20230614101147258.png) ![image-20230614101231227](../../img/kubernetes/kubernetes_kurator/image-20230614101231227.png) ![image-20230614101304410](../../img/kubernetes/kubernetes_kurator/image-20230614101304410.png) ![image-20230614101336467](../../img/kubernetes/kubernetes_kurator/image-20230614101336467.png) ~~~powershell 下载kurator工具包 [root@kubespray ~]# wget https://github.com/kurator-dev/kurator/releases/download/v0.3.0/kurator-linux-amd64-v0.3.0.tar.gz ~~~ ~~~powershell 解压工具包 [root@kubespray ~]# tar xf kurator-linux-amd64-v0.3.0.tar.gz ~~~ ~~~powershell 查看解压目录包含内容 [root@kubespray ~]# ls out charts linux-amd64 [root@kubespray ~]# ls out/linux-amd64/ cluster-operator Dockerfile fleet-manager kurator ~~~ ~~~powershell 复制kurator至/usr/bin目录完成安装 [root@kubespray ~]# cp out/linux-amd64/kurator /usr/bin/ ~~~ ~~~powershell 验证kurator客户端是否可用 [root@kubespray ~]# kurator Kurator builds distributed cloud-native stacks. Usage: kurator [command] Available Commands: completion Generate the autocompletion script for the specified shell help Help about any command install Install a target component join Register a cluster or node tool Tool information for the component version Print the version of kurator Flags: --context string name of the kubeconfig context to use --dry-run console/log output only, make no changes. -h, --help help for kurator --home-dir string install path, default to $HOME/.kurator (default "/root/.kurator") -c, --kubeconfig string path to the kubeconfig file, default to karmada apiserver config (default "/etc/karmada/karmada-apiserver.config") --wait-interval duration interval used for checking pod ready, default value is 1s. (default 1s) --wait-timeout duration timeout used for checking pod ready, default value is 2m. (default 2m0s) Use "kurator [command] --help" for more information about a command. ~~~ ## 2.3 Cluster Operator安装 > 部署Cluster Operator，用于实现对自建K8S集群进行管理。 ### 2.3.1 helm准备 > 部署Cluster Operator需要使用Helm部署。 ~~~powershell 下载helm源文件 [root@kubespray ~]# wget https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz ~~~ ~~~powershell 解压helm源文件 [root@kubespray ~]# tar xf helm-v3.12.0-linux-amd64.tar.gz ~~~ ~~~powershell 安装helm [root@kubespray ~]# cp linux-amd64/helm /usr/bin/ ~~~ ~~~powershell 验证helm版本 [root@kubespray ~]# helm version version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.3"} ~~~ ### 2.3.2 安装cert manager > 证书管理器，关于cert manager更多介绍，可参考如下链接：https://cert-manager.io/docs/ 由于Kurator cluster operator 需要 webhook 证书，默认情况下，使用Helm安装时会导入所需要的证书，当然也可以由cert manager进行自动注入，所以需要提前准备好cert manager证书管理器。cert manager的作用如下： - cert-manager 将证书和证书颁发者作为资源类型添加到 Kubernetes 集群中，并简化了这些证书的获取、更新和使用过程。 - 它可以从各种受支持的来源颁发证书，包括 [Let's Encrypt](https://letsencrypt.org/)、[HashiCorp Vault](https://www.vaultproject.io/)和[Venafi](https://www.venafi.com/)以及私有 PKI。 - 它将确保证书有效且最新，并尝试在到期前的配置时间更新证书。 ~~~powershell 添加cert manager部署使用的helm仓库 [root@kubespray ~]# helm repo add jetstack https://charts.jetstack.io ~~~ ~~~powershell 更新仓库 [root@kubespray ~]# helm repo update ~~~ ~~~powershell 在管理集群中创建命名空间 [root@kubespray ~]# kubectl create namespace cert-manager ~~~ ~~~powershell 使用helm部署cert manager [root@kubespray ~]# helm install -n cert-manager cert-manager jetstack/cert-manager --set installCRDs=true ~~~ ~~~powershell 输出内容如下： NAME: cert-manager LAST DEPLOYED: Wed Jun 14 08:25:15 2023 NAMESPACE: cert-manager STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: cert-manager v1.12.1 has been deployed successfully! In order to begin issuing certificates, you will need to set up a ClusterIssuer or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer). More information on the different types of issuers and how to configure them can be found in our documentation: https://cert-manager.io/docs/configuration/ For information on how to configure cert-manager to automatically provision Certificates for Ingress resources, take a look at the `ingress-shim` documentation: https://cert-manager.io/docs/usage/ingress/ ~~~ ~~~powershell 查看命名空间创建情况 [root@kubespray ~]# kubectl get ns NAME STATUS AGE cert-manager Active 126m ~~~ ~~~powershell 查看命名空间中pod运行情况 [root@kubespray ~]# kubectl get pods -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-7b5cc56d74-v4zj2 1/1 Running 0 126m cert-manager-cainjector-7d948796d5-bjdl6 1/1 Running 0 126m cert-manager-webhook-68f677967c-czb76 1/1 Running 0 126m ~~~ ### 2.3.3 Install cluster operator from release package > 从发布版中安装Cluster Operator ~~~powershell 查看安装使用的源 [root@kubespray ~]# ls out/ charts linux-amd64 [root@kubespray ~]# ls out/charts/ cluster-operator cluster-operator-v0.3.0.tgz fleet-manager fleet-manager-v0.3.0.tgz ~~~ ~~~powershell 使用helm安装Cluster Operator [root@kubespray ~]# helm install kurator-cluster-operator out/charts/cluster-operator-v0.3.0.tgz -n kurator-system --create-namespace ~~~ ~~~powershell 输出内容如下： NAME: kurator-cluster-operator LAST DEPLOYED: Wed Jun 14 09:02:55 2023 NAMESPACE: kurator-system STATUS: deployed REVISION: 1 TEST SUITE: None ~~~ ~~~powershell 查看命名空间创建情况 [root@kubespray ~]# kubectl get ns NAME STATUS AGE cert-manager Active 38m default Active 40d ingress-nginx Active 40d kube-node-lease Active 40d kube-public Active 40d kube-system Active 40d kurator-system Active 61s ~~~ ~~~powershell 查看命名空间中pod创建情况 [root@kubespray ~]# kubectl get pods -n kurator-system NAME READY STATUS RESTARTS AGE kurator-cluster-operator-67d8bd6b68-t64db 1/1 Running 0 73s ~~~ ## 2.4 安装 Fleet Manager > 安装舰队管理器，用于对K8S集群进行逻辑分组 ### 2.4.1 Install FluxCD with Helm > Fleet Manager借助于FluxCD实现对应用程序的部署。参考：https://fluxcd.io ~~~powershell 添加FluxCD部署Charts库 [root@kubespray ~]# helm repo add fluxcd-community https://fluxcd-community.github.io/helm-charts "fluxcd-community" has been added to your repositories ~~~ ~~~powershell 查看 [root@kubespray ~]# helm repo list NAME URL jetstack https://charts.jetstack.io fluxcd-community https://fluxcd-community.github.io/helm-charts ~~~ ~~~powershell 更新 [root@kubespray ~]# helm repo update ~~~ ~~~powershell 使用helm部署FluxCD [root@kubespray ~]# cat < 从发行版中安装Fleet Manager ~~~powershell 查看安装源 [root@kubespray ~]# ls out/ charts linux-amd64 [root@kubespray ~]# ls out/charts/ cluster-operator cluster-operator-v0.3.0.tgz fleet-manager fleet-manager-v0.3.0.tgz ~~~ ~~~powershell 使用helm对Fleet-manager进行管理 [root@kubespray ~]# helm install --create-namespace kurator-fleet-manager out/charts/fleet-manager-v0.3.0.tgz -n kurator-system ~~~ ~~~powershell 输出内容如下： NAME: kurator-fleet-manager LAST DEPLOYED: Wed Jun 14 09:15:24 2023 NAMESPACE: kurator-system STATUS: deployed REVISION: 1 TEST SUITE: None ~~~ ~~~powershell 查看Fleet Manager部署情况 [root@kubespray ~]# kubectl get pods -n kurator-system NAME READY STATUS RESTARTS AGE kurator-cluster-operator-67d8bd6b68-t64db 1/1 Running 0 12m kurator-fleet-manager-6dbf69dc9d-bh2jm 1/1 Running 0 19s ~~~ # 三、使用Cluster Operator部署企业生产级K8S集群 > Kurator借助KubeSpray实现对K8S集群快速部署，由于KubeSpray版本为2.20.0,目前仅能安装K8S 1.24.6版本。 ## 3.1 需要准备用于部署K8S集群节点主机 > 本次准备3台用于部署K8S集群节点主机，仅需要配置主机名，主机IP地址，主机名与IP地址解析即可 。 ~~~powershell 为新建集群主机准备主机名 # hostnamectl set-hostname XXX 请把XXX替换为 master01 node01 node02 ~~~ ~~~powershell 为新建集群主机准备IP地址 [root@XXX ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@XXX ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="063bfc1c-c7c2-4c62-89d0-35ae869e44e7" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.14X" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" # systemctl restart network ~~~ ~~~powershell 为新建集群主机准备主机名与IP地址解析文件 [root@XXX ~]# vim /etc/hosts [root@XXX ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost6 localhost6.localdomain6 localhost6.localdomain 192.168.10.140 master01 192.168.10.141 node01 192.168.10.142 node02 ~~~ ## 3.2 准备ssh密钥 > 在kubespray节点或kubeasz节点 ~~~powershell 生成密钥，注意：如已生成过密钥，不建议再生成，可使用原有密钥即可 。 [root@kubespray ~]# ssh-keygen ~~~ ~~~powershell 同步密钥到其它主机 [root@kubespray ~]# ssh-copy-id 192.168.10.140 [root@kubespray ~]# ssh-copy-id 192.168.10.141 [root@kubespray ~]# ssh-copy-id 192.168.10.142 ~~~ ~~~powershell 基于ssh密钥创建secret [root@kubespray ~]# kubectl create secret generic cluster-secret --from-file=ssh-privatekey=/root/.ssh/id_rsa secret/cluster-secret created ~~~ ~~~powershell 查看已创建secret [root@kubespray kurator]# kubectl get secret NAME TYPE DATA AGE cluster-secret Opaque 1 9s ~~~ ## 3.3 在虚拟机上部署K8S集群 ### 3.3.1 准备部署文件 ![image-20230614111813744](../../img/kubernetes/kubernetes_kurator/image-20230614111813744.png) ~~~powershell 准备kurator部署K8S集群文件 [root@kubespray ~]# git clone https://github.com/kurator-dev/kurator.git ~~~ ~~~powershell 查看文件内容 [root@kubespray ~]# cd kurator/ [root@kubespray kurator]# ls cmd CODE_OF_CONDUCT.md common CONTRIBUTING.md docs examples go.mod go.sum hack LICENSE Makefile manifests OWNERS pkg README.md ROADMAP.md ~~~ ~~~powershell 部署过程中，如果有报crds错误，可以先执行以下目录中的文件 [root@kubespray ~]# ls out/ charts linux-amd64 [root@kubespray ~]# ls out/charts/ cluster-operator cluster-operator-v0.3.0.tgz fleet-manager fleet-manager-v0.3.0.tgz [root@kubespray ~]# ls out/charts/cluster-operator Chart.yaml crds templates values.yaml [root@kubespray ~]# ls out/charts/cluster-operator/crds/ awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io.yaml eksconfigs.bootstrap.cluster.x-k8s.io.yaml awsclusterroleidentities.infrastructure.cluster.x-k8s.io.yaml eksconfigtemplates.bootstrap.cluster.x-k8s.io.yaml awsclusters.infrastructure.cluster.x-k8s.io.yaml extensionconfigs.runtime.cluster.x-k8s.io.yaml awsclusterstaticidentities.infrastructure.cluster.x-k8s.io.yaml infrastructure.cluster.x-k8s.io_customclusters.yaml awsclustertemplates.infrastructure.cluster.x-k8s.io.yaml infrastructure.cluster.x-k8s.io_custommachines.yaml awsfargateprofiles.infrastructure.cluster.x-k8s.io.yaml ipaddressclaims.ipam.cluster.x-k8s.io.yaml awsmachinepools.infrastructure.cluster.x-k8s.io.yaml ipaddresses.ipam.cluster.x-k8s.io.yaml awsmachines.infrastructure.cluster.x-k8s.io.yaml kubeadmconfigs.bootstrap.cluster.x-k8s.io.yaml awsmachinetemplates.infrastructure.cluster.x-k8s.io.yaml kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io.yaml awsmanagedclusters.infrastructure.cluster.x-k8s.io.yaml kubeadmcontrolplanes.controlplane.cluster.x-k8s.io.yaml awsmanagedcontrolplanes.controlplane.cluster.x-k8s.io.yaml kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io.yaml awsmanagedmachinepools.infrastructure.cluster.x-k8s.io.yaml machinedeployments.cluster.x-k8s.io.yaml clusterclasses.cluster.x-k8s.io.yaml machinehealthchecks.cluster.x-k8s.io.yaml cluster.kurator.dev_clusters.yaml machinepools.cluster.x-k8s.io.yaml clusterresourcesetbindings.addons.cluster.x-k8s.io.yaml machines.cluster.x-k8s.io.yaml clusterresourcesets.addons.cluster.x-k8s.io.yaml machinesets.cluster.x-k8s.io.yaml clusters.cluster.x-k8s.io.yaml ~~~ ~~~powershell 执行crds文件 [root@kubespray ~]# kubectl create -f out/charts/cluster-operator/crds ~~~ ~~~powershell 查看执行后的结果 [root@kubespray ~]# kubectl get crds NAME CREATED AT awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsclusterroleidentities.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsclusters.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsclusterstaticidentities.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsclustertemplates.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsfargateprofiles.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsmachinepools.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsmachines.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsmachinetemplates.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsmanagedclusters.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z awsmanagedcontrolplanes.controlplane.cluster.x-k8s.io 2023-06-15T07:19:01Z awsmanagedmachinepools.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:01Z bgpconfigurations.crd.projectcalico.org 2023-05-04T14:12:39Z bgppeers.crd.projectcalico.org 2023-05-04T14:12:39Z blockaffinities.crd.projectcalico.org 2023-05-04T14:12:39Z buckets.source.toolkit.fluxcd.io 2023-06-15T07:08:25Z caliconodestatuses.crd.projectcalico.org 2023-05-04T14:12:39Z certificaterequests.cert-manager.io 2023-06-15T07:03:55Z certificates.cert-manager.io 2023-06-15T07:03:55Z challenges.acme.cert-manager.io 2023-06-15T07:03:55Z clusterclasses.cluster.x-k8s.io 2023-06-15T07:19:01Z clusterinformations.crd.projectcalico.org 2023-05-04T14:12:39Z clusterissuers.cert-manager.io 2023-06-15T07:03:55Z clusterresourcesetbindings.addons.cluster.x-k8s.io 2023-06-15T07:19:01Z clusterresourcesets.addons.cluster.x-k8s.io 2023-06-15T07:19:01Z clusters.cluster.kurator.dev 2023-06-15T07:19:01Z clusters.cluster.x-k8s.io 2023-06-15T07:19:01Z customclusters.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:02Z custommachines.infrastructure.cluster.x-k8s.io 2023-06-15T07:19:02Z eksconfigs.bootstrap.cluster.x-k8s.io 2023-06-15T07:19:01Z eksconfigtemplates.bootstrap.cluster.x-k8s.io 2023-06-15T07:19:01Z extensionconfigs.runtime.cluster.x-k8s.io 2023-06-15T07:19:02Z felixconfigurations.crd.projectcalico.org 2023-05-04T14:12:39Z fleet.fleet.kurator.dev 2023-06-15T07:09:17Z gitrepositories.source.toolkit.fluxcd.io 2023-06-15T07:08:25Z globalnetworkpolicies.crd.projectcalico.org 2023-05-04T14:12:39Z globalnetworksets.crd.projectcalico.org 2023-05-04T14:12:39Z helmcharts.source.toolkit.fluxcd.io 2023-06-15T07:08:25Z helmreleases.helm.toolkit.fluxcd.io 2023-06-15T07:08:25Z helmrepositories.source.toolkit.fluxcd.io 2023-06-15T07:08:25Z hostendpoints.crd.projectcalico.org 2023-05-04T14:12:39Z ipaddressclaims.ipam.cluster.x-k8s.io 2023-06-15T07:19:02Z ipaddresses.ipam.cluster.x-k8s.io 2023-06-15T07:19:02Z ipamblocks.crd.projectcalico.org 2023-05-04T14:12:39Z ipamconfigs.crd.projectcalico.org 2023-05-04T14:12:39Z ipamhandles.crd.projectcalico.org 2023-05-04T14:12:39Z ippools.crd.projectcalico.org 2023-05-04T14:12:39Z ipreservations.crd.projectcalico.org 2023-05-04T14:12:39Z issuers.cert-manager.io 2023-06-15T07:03:55Z kubeadmconfigs.bootstrap.cluster.x-k8s.io 2023-06-15T07:19:02Z kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io 2023-06-15T07:19:02Z kubeadmcontrolplanes.controlplane.cluster.x-k8s.io 2023-06-15T07:19:02Z kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io 2023-06-15T07:19:02Z kubecontrollersconfigurations.crd.projectcalico.org 2023-05-04T14:12:39Z kustomizations.kustomize.toolkit.fluxcd.io 2023-06-15T07:08:25Z machinedeployments.cluster.x-k8s.io 2023-06-15T07:19:02Z machinehealthchecks.cluster.x-k8s.io 2023-06-15T07:19:02Z machinepools.cluster.x-k8s.io 2023-06-15T07:19:02Z machines.cluster.x-k8s.io 2023-06-15T07:19:02Z machinesets.cluster.x-k8s.io 2023-06-15T07:19:02Z networkpolicies.crd.projectcalico.org 2023-05-04T14:12:39Z networksets.crd.projectcalico.org 2023-05-04T14:12:39Z ocirepositories.source.toolkit.fluxcd.io 2023-06-15T07:08:25Z orders.acme.cert-manager.io 2023-06-15T07:03:55Z ~~~ ~~~powershell 准备部署用K8S集群部署文件 [root@kubespray kurator]# cp -rfp examples/infra/customcluster examples/infra/my-customcluster ~~~ ~~~powershell 进入准备好的K8S集群部署文件所在目录 [root@kubespray kurator]# cd examples/infra/my-customcluster/ [root@kubespray my-customcluster]# ls cc-cluster.yaml cc-customcluster.yaml cc-custommachine.yaml cc-kcp.yaml ~~~ ~~~powershell 修改cc-cluster.yaml文件，主要指定pod与service网络 [root@kubespray my-customcluster]# vim cc-cluster.yaml [root@kubespray kurator]# cat examples/infra/my-customcluster/cc-cluster.yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: cc-cluster namespace: default spec: clusterNetwork: pods: cidrBlocks: - 10.245.0.0/16 services: cidrBlocks: - 10.97.0.0/16 serviceDomain: kurator-service-domain controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane name: cc-kcp namespace: default infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster name: cc-customcluster namespace: default ~~~ ~~~powershell 修改cc-customcluster文件，指定网络插件，默认可以不修改 [root@kubespray my-customcluster]# vim cc-customcluster.yaml [root@kubespray kurator]# cat examples/infra/my-customcluster/cc-customcluster.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster metadata: name: cc-customcluster namespace: default spec: cni: type: cilium machineRef: apiVersion: cluster.kurator.dev/v1alpha1 kind: CustomMachine name: cc-custommachine namespace: default ~~~ ~~~powershell 修改cc-custommachine，指定K8S集群master节点及worker节点 [root@kubespray my-customcluster]# vim cc-custommachine.yaml [root@kubespray my-customcluster]# cat cc-custommachine.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomMachine metadata: name: cc-custommachine namespace: default spec: master: - hostName: master01 publicIP: 192.168.10.140 privateIP: 192.168.10.140 sshKey: apiVersion: v1 kind: Secret name: cluster-secret node: - hostName: node01 publicIP: 192.168.10.141 privateIP: 192.168.10.141 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: node02 publicIP: 192.168.10.142 privateIP: 192.168.10.142 sshKey: apiVersion: v1 kind: Secret name: cluster-secret ~~~ ~~~powershell 修改cc-kcp文件，主要指定K8S集群版本，由于受到kubespray版本影响，本次可部署最新版本为1.24.6 [root@kubespray my-customcluster]# vim cc-kcp.yaml [root@kubespray kurator]# cat examples/infra/my-customcluster/cc-kcp.yaml apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane metadata: name: cc-kcp namespace: default spec: kubeadmConfigSpec: clusterConfiguration: clusterName: kurator-cluster imageRepository: featureGates: PodOverhead: true NodeSwap: true initConfiguration: joinConfiguration: machineTemplate: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: customMachine name: cc-custommachine namespace: default version: v1.24.6 ~~~ ### 3.3.2 Apply resource configuration deploy K8S > 执行部署文件 ~~~powershell [root@kubespray kurator]# kubectl apply -f examples/infra/my-customcluster ~~~ ~~~powershell 输出内容如下： cluster.cluster.x-k8s.io/cc-cluster created customcluster.infrastructure.cluster.x-k8s.io/cc-customcluster created custommachine.infrastructure.cluster.x-k8s.io/cc-custommachine created kubeadmcontrolplane.controlplane.cluster.x-k8s.io/cc-kcp created ~~~ ~~~powershell 查看部署pod状态，需要下载kubespray镜像，共4G大小，需要等待一定时间。 [root@kubespray kurator]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-init 0/1 ContainerCreating 0 4m17s ~~~ ~~~powershell 查看部署过程日志 [root@kubespray kurator]# kubectl logs cc-customcluster-init -f ~~~ ### 3.3.3 验证是否安装成功 ~~~powershell 查看部署pod状态 [root@kubespray ~]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-init 0/1 Completed 0 123m ~~~ ~~~powershell 查看已部署K8S集群 [root@kubespray ~]# kubectl get clusters.cluster.x-k8s.io NAME PHASE AGE VERSION cc-cluster Provisioning 125m ~~~ >在目标集群主机上进行查看。 ~~~powershell [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane 108m v1.24.6 node01 Ready 107m v1.24.6 node02 Ready 107m v1.24.6 ~~~ ~~~powershell [root@kubespray ~]# kubectl get cc NAME AGE cc-customcluster 126m ~~~ # 四、使用Cluster Operator实现对K8S高可用管理 > 使用kube-vip实现master节点高可用。 > > 需要把原集群删除 > > 需要为新添加的master节点准备ssh免密登录证书 ## 4.1 准备多个k8s集群master节点 > master02 IP：192.168.10.143/24；master03 IP:192.168.10.144/24 ~~~powershell [root@localhost ~]# hostnamectl set-hostname master02 [root@localhost ~]# hostnamectl set-hostname master03 ~~~ ~~~powershell [root@XXX ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="063bfc1c-c7c2-4c62-89d0-35ae869e44e7" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.14X" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" # systemctl restart network ~~~ ## 4.2 修改声明文件 ### 4.2.1 集群主机定义文件 >examples/infra/my-customcluster/cc-custommachine.yaml ~~~powershell [root@kubespray kurator]# vim examples/infra/my-customcluster/cc-custommachine.yaml [root@kubespray kurator]# cat examples/infra/my-customcluster/cc-custommachine.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomMachine metadata: name: cc-custommachine namespace: default spec: master: - hostName: master01 publicIP: 192.168.10.140 privateIP: 192.168.10.140 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: master02 添加新主机 publicIP: 192.168.10.143 privateIP: 192.168.10.143 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: master03 添加新主机 publicIP: 192.168.10.144 privateIP: 192.168.10.144 sshKey: apiVersion: v1 kind: Secret name: cluster-secret node: - hostName: node01 publicIP: 192.168.10.141 privateIP: 192.168.10.141 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: node02 publicIP: 192.168.10.142 privateIP: 192.168.10.142 sshKey: apiVersion: v1 kind: Secret name: cluster-secret ~~~ ### 4.2.2 集群定义文件 >examples/infra/my-customcluster/customcluster.yaml ~~~powershell 添加如下内容： # add config controlPlaneConfig: # address is your VIP, assume your VIP is 192.x.x.0 address: 192.168.10.100 # loadBalancerDomainName is an optional field that sets the load balancer domain name. # If not specified, the default name, controlPlaneConfig.address is used. loadBalancerDomainName: lb.kubemsb.com # optional, sets extra Subject Alternative Names for the API Server signing cert. # If you don't have any want to add, you can directly remove this field. certSANs: [192.168.10.140,192.168.10.143,192.168.10.144] ~~~ ~~~powershell [root@kubespray kurator]# vim examples/infra/my-customcluster/cc-customcluster.yaml [root@kubespray kurator]# cat examples/infra/my-customcluster/cc-customcluster.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster metadata: name: cc-customcluster namespace: default spec: cni: type: cilium controlPlaneConfig: address: 192.168.10.100 loadBalancerDomainName: lb.kubemsb.com certSANs: [192.168.10.140,192.168.10.143,192.168.10.144] machineRef: apiVersion: cluster.kurator.dev/v1alpha1 kind: CustomMachine name: cc-custommachine namespace: default ~~~ ## 4.3 应用资源文件创建多master节点集群 ~~~powershell [root@kubespray kurator]# kubectl apply -f examples/infra/my-customcluster/ ~~~ ~~~powershell 输出内容如下： cluster.cluster.x-k8s.io/cc-cluster created customcluster.infrastructure.cluster.x-k8s.io/cc-customcluster created custommachine.infrastructure.cluster.x-k8s.io/cc-custommachine created kubeadmcontrolplane.controlplane.cluster.x-k8s.io/cc-kcp created ~~~ ~~~powershell [root@kubespray kurator]# kubectl logs cc-customcluster-init -f ~~~ > 在k8s集群节点上查看 ~~~powershell [root@master01 ~]# kubectl get po -A | grep kube-vip kube-system kube-vip-master01 1/1 Running 0 9m29s kube-system kube-vip-master02 1/1 Running 0 7m31s kube-system kube-vip-master03 1/1 Running 0 7m31s ~~~ ~~~powershell [root@master01 ~]# cat /root/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EWXhOREEwTkRVeU1sb1hEVE16TURZeE1UQTBORFV5TWxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTmdrCjZFOTdBVjNlYXRwUXh5VG9acFA4c1duSFJoa1pTRVdBd01JSVI0N0w1U0hBcktwSDVuQVZiaS92VFdHaFNEQjIKUEsyb08xQlRYUzFNTmtxUCs4SEtvZ2s4R2RiOUllUUszbm9CaDYrR21nSEpQZkxpL1NHS3VlcC8xZHpDb3hTUgpUbE1PWVd1QmNiRUxibWJ6ZDFjMzBIRkYraHd5cHNCZE9STmlmZlZBZW5SYkx2NzdFZThrQlc5UTNDczJmdmpnClBEdzllWEl0Sitzc1F6aTNzb2I1b3RFREtKSTE0clBCYTFzaUN3eUZSYmhIZFlrSnBIZlJjNHI2NnMzQmE5dVIKMzF6QzI4R1UwRnA0RDZuUFZRMWZkVEZMS0tsYzlTeHJsdTFCcEtpbUY5WnYwYlIzMDNDd2ZVU0V0VEVMQTRvdQpNVDZya1p2cFpKcVhwNkZ1V2RFQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZHQkRldG15OGxOQzM2dnlXU1VnT09JbVNHeERNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS0pMcUljUlBITVBFT1BNN0ltcgpuTExPWElvWlBWd3ZTN01PcGxVL3ZITzBRWjVtZFR5T0pnYk5GL2RpL2ozRTVLbTc2b0gvbXNqTUxYbnNyZmlGCkU5KzZRY3UvNEJUYkpuK21qV05wYk1VOW10OFkvQWdseEorYmpWaGdoMzF1VGdFaUJFYTVOYklmM01Td3h2TjUKL0dBU1VFU3FIaEdTNE5hMkcwQ0wzQ0ZNMW1BUWJIOFpvNk1hWlVHbHhVbE5XSG9lZU5XaFNvRU5SNWFHanM3cgpkTndkK241UkZzV2huR1NzMFFCSzViS2grSjVCSmZEVlp1WHVJVEpIeWxMNTR1cndPOWc3UkVQdmhSWUtJbEVZCms4OG5zempWNjRWdXJHaENScU1sSzFtTlphYlVaL0ZzdCtLa0w0Q1hIZC81VVhQWkRmMUJkeTdaVWVFdEN0OHYKQnhjPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.10.100:6443 name: cluster.local contexts: - context: cluster: cluster.local user: kubernetes-admin name: kubernetes-admin@cluster.local current-context: kubernetes-admin@cluster.local kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJQjZFRnB3NC91UGt3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMk1UUXdORFExTWpKYUZ3MHlOREEyTVRNd05EUTFNak5hTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXZPQ2xVaUFEcDZmQXFIcUYKYUE4SjZLSzdiS0EvV2plNmFhUU1xZ3dvWkJhVG1uVkQyN0J3YWpqcXMva3dHOHZHWG5nNjVhVHNTZ3l2QUczNQpTdFFyeXVNV1ZxRW43SERKYWIrc3VyNVpYeEZyczZSOGUramdjRmJDM3dLT01JNDg0N2taZ2t6L2VxUjREbTkxCkxJLytXT3RFNXJtcGY5OXdhYVRCWnlBK2hSaFdiYzJSOTd0eEdEZU1XTzZpWUdNWS92K2V0R1ZmRm81bUNtUzUKVWpudzM5eGE0VlJtS2xwcUhmbWhCRk1wSDVhdElPNnZUdGE2bEZsS21qeFBDVmI2UmQ2YU9iN1ZkSUhMWC9ubgpBcU8yaHZCQ3J5a2prY2NzWHNGa0QxVldVbjNXeEVrV2tWdmFYaGtvZk1vUUxjMHNhQUE4UUhrZ3pxZDQwNHl1CkVZZllPUUlEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JSZ1EzclpzdkpUUXQrcjhsa2xJRGppSmtocwpRekFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBSjBRZlpnc3UrNXlyemhOa0JqMC9JQm9uYTh5a1dldERhSWc3CktaZURuMHYraXZxTWFzanNtUS82OWlCWU15R0ZwOTlZY2FTcTEvUDV1a2pSV0taWXRIRzRDR0tKV3grdVIzVU8KUHZQcklJUVBkMGp4MFR6dkM5dmQ3NWlodVpmM0VJTmdoZzZBWWxtbDkveXdHV2NhaDltdWduclRwcjBrMm12OApCY0FHYnpZT2RxaHlnQzZjT1YveU1MZEhXcTF1OERHaFkzSHVwdFIydEhTTlIzV3NoQ1RUdHNRQ1NNVE82Sm1NCktFcWpIN25oSit3WWkyNGZJTGdoNWpQR0V4QXRwMTZqSzZkUXpXNUFKdFFsZjhoZURTdCs3Y2NEYkIzRU1GZmsKQmxtOG1DU2l4RzFQdFhYS2FJelhmZDNhQ2NxREpMcldKUVFkbXp2WjE1SlhnczhqaVE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdk9DbFVpQURwNmZBcUhxRmFBOEo2S0s3YktBL1dqZTZhYVFNcWd3b1pCYVRtblZECjI3Qndhampxcy9rd0c4dkdYbmc2NWFUc1NneXZBRzM1U3RRcnl1TVdWcUVuN0hESmFiK3N1cjVaWHhGcnM2UjgKZStqZ2NGYkMzd0tPTUk0ODQ3a1pna3ovZXFSNERtOTFMSS8rV090RTVybXBmOTl3YWFUQlp5QStoUmhXYmMyUgo5N3R4R0RlTVdPNmlZR01ZL3YrZXRHVmZGbzVtQ21TNVVqbnczOXhhNFZSbUtscHFIZm1oQkZNcEg1YXRJTzZ2ClR0YTZsRmxLbWp4UENWYjZSZDZhT2I3VmRJSExYL25uQXFPMmh2QkNyeWtqa2Njc1hzRmtEMVZXVW4zV3hFa1cKa1Z2YVhoa29mTW9RTGMwc2FBQThRSGtnenFkNDA0eXVFWWZZT1FJREFRQUJBb0lCQUc3VWd5M0ZpMTk5bUJkRQpqYk9aMzM2YXZzL20yRVhnSFlxUHNMNjNrNXZuZzRWRytpS1hUZCtHdG5JVTdwM2czTzNpTHUxQjhVVWlYQzRzCldmUUd2QXU1WlFwQVZtVHZhY0Nic0llRDJuMW1nNFJPS0oyczhNWDZveTZwR1RaMHdGWC83Mi8rTG91UXBzSVIKdFBnSkpuNld6ejkwQW5uK2FzQVM1VW5XS2RsK2hGRE5CcExPU05oOWhqMU5OTHNVWVpCaWVTV2p3NFVtaVlpVgpqQkI2NmppMFp1WmVpUFFpdSs4blNPT2kyNkJ3Nm5STGZadzE4RG5GTlF6dS92ZDZteDZaMFVlQ2Yzc1dhOXdTCm5pUXFvY1lVOElML3VCSkQxWEpPSVhHTSsrbXVmcnpZUTlmanpITTFuQ05MamIvS0xMakhSamJlQnZLdnpyUmgKaXVyK0NlRUNnWUVBNGViN1pFaUNaZ3V3a3NZdE9FR1RReWdiL2xvdlJUQUFxNHpIanRSaFBWMXhyZU5yd0ZWdwp5MzZDbmVuUlUyUTlwNTFDRG5BL1RJNkVGMU9VQlVXcld1ci9UeGc4TWpzUjhTZmtFWWFUK0xzcTA3a2E3ckMzCmNWM1dwN1diQmc4QzRCUHFlQ01ESHVhbE1oT2dsblYzZERtcFFxcE5aaHU5OVltd2dCR25ZNHNDZ1lFQTFnclYKTjl0L3k2N080T1dtMDVDTy9nUUY2YTlIL01OL2FWVzdSeHBmSjE2alY2aGxzbFE0NnJaekxXd2ZoNk1XRDFUNwptQWZ5a3U1UGx2VFVhVCtqbmVpMlpOZzBrazRkTk9XYU5uaWw0Z0ZOUHFpZXhuZUpOV1hDWlhCNmVoUURkSUtGCjZyN1JmTHp3R3hrUlNscC90VWtaeW5lN3pPaFYvRUhzWDhOczI4c0NnWUVBMkxMdGhtWjBDQlhBblJ0VDkzb1kKSEx6V0ZhbEhYUlJPSUVJY2tDdTAxOXZTbDk3bjF3bGUrU1h6Z0MzeWFnOW5nV2srRzRHYmh3SFRQbGtqVGVkbQpHQUlEUlVsWFBVd2l5dlhjTnQrbEJGM3ZHRWloeUVSbXhHNGk0cUE0QkpqVHhrWDBqcWZ2YjI0TEc3MXVOOGFZCi96bnMzeEZreThWaTlLM1BUUnNpT2o4Q2dZRUFpN0lVU05sOGhkNDRiWFhWTk4xelJmd3dNNzFQbVh1cXEzL2wKczlsVll5ZVVLZ0RoSnN0clBsMEh2UXlGWTR3Z09LOTVhVmMrallEMTl3c2htdk9MQU9QQ0x2MGhDcC9xMWZ4SQpjdWE1TlZFdUxnZjFZSWpoeHpUNzlDSzkvUEVVOURTLzNGN2Y4SzZMY3RRZ2djdWc4QnVldHBNQXdoS2RFLytlCm16Mjd6Wk1DZ1lCbWY3MkcvVXBVcktFcjNvNGNEL0crSHFvYWNCL09JdklobkRwWXJKNWJxY29TdHZKektldjgKS041WXN2MEZnVE5IQ0VPeEpUOTFuRVRnanU2WDVuaEJiNzhKdzd5cWxiRkh6cU4rZzREZDBvY05mTjdoVVprSQp4THZGdWtxNWYzaWF6RjRHQ3VhdVVPaldaOWhDZDhuSnArY29HWVI3dmpaby8ycnozdVVhOXc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= ~~~ # 五、使用Cluster Operator对K8S进行扩展 >借助 Kurator，可以以声明方式添加、删除或替换 VM 上的多个工作节点。执行缩放时，应避免修改主机名，以防同一 VM 配置了多个名称。只需要在目标 customMachine 上声明想要的最终工作节点状态，Kurator 就可以在没有任何外部干预的情况下完成节点缩放。 ## 5.1 扩展工作节点 ~~~powershell [root@localhost ~]# hostnamectl set-hostname node3 [root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="063bfc1c-c7c2-4c62-89d0-35ae869e44e7" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.145" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" [root@localhost ~]# systemctl restart network ~~~ ~~~powershell [root@kubespray kurator]# ssh-copy-id 192.168.10.145 ~~~ ~~~powershell [root@kubespray kurator]# cp examples/infra/my-customcluster/cc-custommachine.yaml examples/infra/my-customcluster/scale.yaml ~~~ ~~~powershell [root@kubespray kurator]# vi examples/infra/my-customcluster/scale.yaml [root@kubespray kurator]# cat examples/infra/my-customcluster/scale.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomMachine metadata: name: cc-custommachine namespace: default spec: master: - hostName: master01 publicIP: 192.168.10.140 privateIP: 192.168.10.140 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: master02 publicIP: 192.168.10.143 privateIP: 192.168.10.143 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: master03 publicIP: 192.168.10.144 privateIP: 192.168.10.144 sshKey: apiVersion: v1 kind: Secret name: cluster-secret node: - hostName: node01 publicIP: 192.168.10.141 privateIP: 192.168.10.141 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: node02 publicIP: 192.168.10.142 privateIP: 192.168.10.142 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: node03 publicIP: 192.168.10.145 privateIP: 192.168.10.145 sshKey: apiVersion: v1 kind: Secret name: cluster-secret ~~~ ~~~powershell [root@kubespray kurator]# kubectl apply -f examples/infra/my-customcluster/scale.yaml custommachine.infrastructure.cluster.x-k8s.io/cc-custommachine configured ~~~ ~~~powershell [root@kubespray kurator]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-init 0/1 Completed 0 60m cc-customcluster-scale-up 1/1 Running 0 29s ~~~ ~~~powershell [root@kubespray kurator]# kubectl logs cc-customcluster-scale-up -f ~~~ ~~~powershell [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane 85m v1.24.6 master02 Ready control-plane 84m v1.24.6 master03 Ready control-plane 83m v1.24.6 node01 Ready 21m v1.24.6 node02 Ready 21m v1.24.6 node03 Ready 31m v1.24.6 ~~~ ## 5.2 删除节点 > 在examples/infra/my-customcluster/scale.yaml文件中删除或注释相应的集群节点后，再使用kubectl apply -f examples/infra/my-customcluster/scale.yaml即可 。 # 六、使用Cluster Operator对K8S集群版本进行升级 >需要做的就是在 kcp 上声明所需的 Kubernetes 版本，Kurator 将在没有任何外部干预的情况下完成集群升级。由于升级实现依赖kubeadm，建议避免跳过小版本。例如，您可以从v1.22.0 升级到v1.23.9，但不能**从**v1.22.0 一步升级到v1.24.0。要声明所需的升级版本，只需编辑 kcp 的 CRD 以反映所需的升级版本 ~~~powershell [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane 91m v1.24.5 master02 Ready control-plane 90m v1.24.5 master03 Ready control-plane 89m v1.24.5 node01 Ready 27m v1.24.5 node02 Ready 27m v1.24.5 node03 Ready 37m v1.24.5 ~~~ ~~~powershell [root@kubespray kurator]# kubectl get kcp NAME CLUSTER INITIALIZED API SERVER AVAILABLE REPLICAS READY UPDATED UNAVAILABLE AGE VERSION cc-kcp cc-cluster 108m v1.24.5 ~~~ ~~~powershell [root@kubespray kurator]# kubectl edit kcp cc-kcp 原内容： machineTemplate: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: customMachine name: cc-custommachine namespace: default metadata: {} replicas: 1 rolloutStrategy: rollingUpdate: maxSurge: 1 type: RollingUpdate version: v1.24.5 ...... 修改为： machineTemplate: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: customMachine name: cc-custommachine namespace: default metadata: {} replicas: 1 rolloutStrategy: rollingUpdate: maxSurge: 1 type: RollingUpdate version: v1.24.6 ...... ~~~ ~~~powershell [root@kubespray kurator]# kubectl get kcp NAME CLUSTER INITIALIZED API SERVER AVAILABLE REPLICAS READY UPDATED UNAVAILABLE AGE VERSION cc-kcp cc-cluster 110m v1.24.6 ~~~ ~~~powershell [root@kubespray kurator]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-init 0/1 Completed 0 54m cc-customcluster-upgrade 1/1 Running 0 13s ~~~ ~~~powershell 要确认升级工作器正在运行,命令如下： # kubectl get pod -A | grep -i upgrade ~~~ # 七、使用Cluster Operator删除K8S集群 ## 7.1 查找集群资源 ~~~powershell [root@kubespray kurator]# kubectl get clusters.cluster.x-k8s.io NAME PHASE AGE VERSION cc-cluster Provisioning 166m ~~~ ## 7.2 删除自定义集群资源 > 整个集群删除大约需要5分钟 ~~~powershell [root@kubespray kurator]# kubectl delete clusters.cluster.x-k8s.io cc-cluster cluster.cluster.x-k8s.io "cc-cluster" deleted ~~~ ## 7.3 查看是否删除完成 >可以在终端中打开一个新的命令选项卡来检查终止 pod 的状态,当此pod消失时 表示执行完成。 ~~~powershell [root@kubespray ~]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-terminate 1/1 Running 0 88s [root@kubespray ~]# watch kubectl get pods ~~~ # 八、使用kurator实现多集群管理 > 需要准备2套k8s集群主机。 ## 8.1 创建多K8S集群 > 创建K8S集群前需要准备crds ~~~powershell [root@kubespray infra]# pwd /root/kurator/examples/infra [root@kubespray infra]# ls customcluster first-customcluster second-customcluster ~~~ ### 8.1.1 创建first-customcluster集群 ~~~powershell [root@kubespray infra]# cd first-customcluster/ [root@kubespray first-customcluster]# ls cc-cluster.yaml cc-customcluster.yaml cc-custommachine.yaml cc-kcp.yaml [root@kubespray first-customcluster]# vim cc-cluster.yaml [root@kubespray first-customcluster]# cat cc-cluster.yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: first-cluster namespace: default spec: clusterNetwork: pods: cidrBlocks: - 10.245.0.0/16 services: cidrBlocks: - 10.97.0.0/16 serviceDomain: kurator-service-domain controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane name: cc-kcp-first namespace: default infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster name: cc-customcluster-first namespace: default [root@kubespray first-customcluster]# vim cc-customcluster.yaml [root@kubespray first-customcluster]# cat cc-customcluster.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster metadata: name: cc-customcluster-first namespace: default spec: cni: type: cilium machineRef: apiVersion: cluster.kurator.dev/v1alpha1 kind: CustomMachine name: cc-custommachine-first namespace: default [root@kubespray first-customcluster]# vim cc-custommachine.yaml [root@kubespray first-customcluster]# cat cc-custommachine.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomMachine metadata: name: cc-custommachine-first namespace: default spec: master: - hostName: master01 publicIP: 192.168.10.140 privateIP: 192.168.10.140 sshKey: apiVersion: v1 kind: Secret name: cluster-secret node: - hostName: node01 publicIP: 192.168.10.141 privateIP: 192.168.10.141 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: node02 publicIP: 192.168.10.142 privateIP: 192.168.10.142 sshKey: apiVersion: v1 kind: Secret name: cluster-secret [root@kubespray first-customcluster]# vim cc-kcp.yaml [root@kubespray first-customcluster]# cat cc-kcp.yaml apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane metadata: name: cc-kcp-first namespace: default spec: kubeadmConfigSpec: clusterConfiguration: clusterName: kurator-cluster-first imageRepository: featureGates: PodOverhead: true NodeSwap: true initConfiguration: joinConfiguration: machineTemplate: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: customMachine name: cc-custommachine-first namespace: default version: v1.24.6 ~~~ ~~~powershell [root@kubespray kurator]# kubectl apply -f examples/infra/first-customcluster cluster.cluster.x-k8s.io/first-cluster created customcluster.infrastructure.cluster.x-k8s.io/cc-customcluster created custommachine.infrastructure.cluster.x-k8s.io/cc-custommachine created kubeadmcontrolplane.controlplane.cluster.x-k8s.io/cc-kcp-first created ~~~ ~~~powershell [root@kubespray kurator]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-first-init 1/1 Running 0 36s ~~~ ~~~powershell [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane 163m v1.24.6 node01 Ready 162m v1.24.6 node02 Ready 162m v1.24.6 ~~~ ### 8.1.2 创建second-customcluster集群 ~~~powershell [root@kubespray infra]# cd second-customcluster/ [root@kubespray second-customcluster]# ls cc-cluster.yaml cc-customcluster.yaml cc-custommachine.yaml cc-kcp.yaml [root@kubespray second-customcluster]# vim cc-cluster.yaml [root@kubespray second-customcluster]# cat cc-cluster.yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: cc-cluster-second namespace: default spec: clusterNetwork: pods: cidrBlocks: - 10.246.0.0/16 services: cidrBlocks: - 10.98.0.0/16 serviceDomain: kurator-service-domain controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane name: cc-kcp-second namespace: default infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster name: cc-customcluster-second namespace: default [root@kubespray second-customcluster]# vim cc-customcluster.yaml [root@kubespray second-customcluster]# cat cc-customcluster.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomCluster metadata: name: cc-customcluster-second namespace: default spec: cni: type: cilium machineRef: apiVersion: cluster.kurator.dev/v1alpha1 kind: CustomMachine name: cc-custommachine-second namespace: default [root@kubespray second-customcluster]# vim cc-custommachine.yaml [root@kubespray second-customcluster]# cat cc-custommachine.yaml apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: CustomMachine metadata: name: cc-custommachine-second namespace: default spec: master: - hostName: master01 publicIP: 192.168.10.143 privateIP: 192.168.10.143 sshKey: apiVersion: v1 kind: Secret name: cluster-secret node: - hostName: node01 publicIP: 192.168.10.144 privateIP: 192.168.10.144 sshKey: apiVersion: v1 kind: Secret name: cluster-secret - hostName: node02 publicIP: 192.168.10.145 privateIP: 192.168.10.145 sshKey: apiVersion: v1 kind: Secret name: cluster-secret [root@kubespray second-customcluster]# vim cc-kcp.yaml [root@kubespray second-customcluster]# cat cc-kcp.yaml apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane metadata: name: cc-kcp-second namespace: default spec: kubeadmConfigSpec: clusterConfiguration: clusterName: kurator-cluster-second imageRepository: featureGates: PodOverhead: true NodeSwap: true initConfiguration: joinConfiguration: machineTemplate: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1 kind: customMachine name: cc-custommachine-second namespace: default version: v1.24.6 ~~~ ~~~powershell [root@kubespray kurator]# kubectl apply -f examples/infra/second-customcluster cluster.cluster.x-k8s.io/cc-cluster-second created customcluster.infrastructure.cluster.x-k8s.io/cc-customcluster created custommachine.infrastructure.cluster.x-k8s.io/cc-custommachine created kubeadmcontrolplane.controlplane.cluster.x-k8s.io/cc-kcp-second created custommachine.infrastructure.cluster.x-k8s.io/cc-custommachine created ~~~ ~~~powershell [root@kubespray kurator]# kubectl get pods NAME READY STATUS RESTARTS AGE cc-customcluster-second-init 1/1 Running 0 101s ~~~ ~~~powershell [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane 44m v1.24.6 node01 Ready 43m v1.24.6 node02 Ready 43m v1.24.6 ~~~ ## 8.2 创建用于连接集群secret ### 8.2.1 准备kubeconfig文件 ~~~powershell [root@kubespray ~]# scp 192.168.10.140:/root/.kube/config /root/.kube/kurator-member1.config ~~~ ~~~powershell [root@kubespray ~]# scp 192.168.10.143:/root/.kube/config /root/.kube/kurator-member2.config ~~~ ~~~powershell [root@kubespray ~]# ls /root/.kube/ cache config kurator-member1.config kurator-member2.config ~~~ > 注意修改config文件中server地址部分为每一个K8S集群master节点IP地址。注：修改集群名称、上下文名称、管理员名称等。 ~~~powershell [root@kubespray ~]# cat /root/.kube/kurator-member1.config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EWXhOVEE1TURjek9Gb1hEVE16TURZeE1qQTVNRGN6T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTnYyCmd3MXJSeTUrMGRLWE5CRDN4QkR2V0NDc1RnSDJhRUhEQ3pIU3BQaVhab3I4YVhKT3p1QzMvdFZZOXV0N01DbC8KZTBwTDRxMWV4UnBaWDErYUVrUWNhalFXWWQrc2daUWFDQXlPWnJGQVZQOGpNdkhTVEx5N0NJZVg4U3NyR2RPQgoyQjFyenJXRXZ3V1RGUklsSFdKQ0s4ZWZ4N2F3WnhHdDVQQSt5MmVVbno1QWxKVWN2ZDEreHlDV0F1UDY5Ulc3Cm1Bdzd1N2pDRytlby9MeTVOTlVLK1hQQUxhMk9LdmloODNaZi80NkJOeHNKZER3NkZialpRN1BpN2ZLNWZqeTkKb0JBenNhL084andtT0xPQUY2VUJjeCtnK3lQMVJaQldkZGNWeEZMbEJWYlhVZjRUTTZ3NlVvc092cENsMldxcApYcDBCWXhNcTZxVEd6em9VNHVVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZFVTVTZ1dqMWpMSE42OEJWL3hrYmV1c0xBZ0JNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBTkdOWEFtMUozTHJGdnByNVVQRApRV2VWdm92VGJhTC9sQlRwV3Ewb1grOE1mVHRUVEVUbUQrL21TM1MwWWdPalBCbDFxbGtUVnJMWlVHcmxkOUNZCmRaOHViemJkRklMRTU0Q0ZxM3hyMS9yTlJFaGcyZzdkaWZGdkU4TzZyK0hWZ0RnRlNOekl3b2Fsd1kwUzltUWoKbVhSeVh6UC9UeWp3cnQ0S1FQMU9rZlJxNlBGeGFCeGNZV1VHa1ZuRTIzMVRMa01MUHI5MmpWTU00OXBIVEJMMwpCOFlkNE9WNUMyYTNuWkp1bVpJVFFkWkVoNW9rZlEydUphV0diZWhyL05MVmViM05xTXFQQmlNcDc0ZHBYeitNCklKaGN6M1hmaldNbUprOEZlUXMwR29hUmNwb1kyVS85dlpHWjhlaXhWT1BPNU0zTGtGdjIvS3dyQk5ic01uK1EKdS9ZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.10.140:6443 name: member1 contexts: - context: cluster: member1 user: member1-admin name: kurator-member1 current-context: kurator-memeber1 kind: Config preferences: {} users: - name: member1-admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJYzIycW5PcDRjVGt3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMk1UVXdPVEEzTXpoYUZ3MHlOREEyTVRRd09UQTNNemhhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTBPYWpQcFFINDczMGhUbG0KQVVYMXJTc3FOQ1VNQXNzNEgyK1hKZ1V5RnpiL2ZpamN5RDZnQ2h5TXlzbTFtNlNHVXA2c2thMGt3WEE5cGx0Ngo4eTUzeVpHcEFDNHdqWWhVSEUyWjk5TzVwUVR6TElrdVg5SjRpdjA3NzVWRHU1TnZINmsxc3BualhxbkpDVnVQClJJR1pGaFI4TFdhVEVuTDhiRFRoWXdSMHVOSnBOZmRRSGZ4Q0RhS29rbjd1eHlVR0ZTcldRblV5enl2YUMzQmUKeEw2TTlnZ2VnRTRnOGdpWnE3SDFrc3FLeG9TK251Qy9ZTHE0bi9aQXprd013MHQ0TjFBZmhxWGRRZHRNMVduRApEaUZGY0pHNHExUkdiU1BKUTFkeS9IajVSU1Q3UkhYZnN1T1hUSzMySStoZUxWUGJYWVNOV05QZ3h2alAxK2NGCjEyaVpmUUlEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JSRk9Vb0ZvOVl5eHpldkFWZjhaRzNyckN3SQpBVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBVnJQSEFRZ2x0ZTE4QlVNWlJHOGNKS3JXNUhTSGNFZWFPM3NzCmo5QThndk5LNFNFcENJSlJlTzZvMGtnQ09zYWpJKzE1Skc2dERtZTJTMndSOStIWnNCZ3h5eVFLYkZIREEwenQKZGJKL2h5TzBFUGVrNVIvSksxWVpqa01qNHVkRXJLSUlQSHJmbDhxeldSd05Ka2lPVUtzdzRsRjB4OFROVjJ3agpXZFZ6TUxIakR2NmpleHJta3hSYVdrTVBrZUM5ZWdrV1NIZlZoVjRCMFBCWUpHejUzS3BjdkVIaGhWRis2M2MwCkRnOUNjWk9qYUhteG00T05DbFA5emNxOVRCRE82anBneEpENjhHZVlhM3Z3bU54aTQzRStXSXl6em9aS1N6M3kKV0N1QmppYmJiUGEwVmxjZG9uV0RzZGtTWU90YzdhYSs0TXBmb1hDalB5dU9yakJSWEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBME9halBwUUg0NzMwaFRsbUFVWDFyU3NxTkNVTUFzczRIMitYSmdVeUZ6Yi9maWpjCnlENmdDaHlNeXNtMW02U0dVcDZza2Ewa3dYQTlwbHQ2OHk1M3laR3BBQzR3alloVUhFMlo5OU81cFFUekxJa3UKWDlKNGl2MDc3NVZEdTVOdkg2azFzcG5qWHFuSkNWdVBSSUdaRmhSOExXYVRFbkw4YkRUaFl3UjB1TkpwTmZkUQpIZnhDRGFLb2tuN3V4eVVHRlNyV1FuVXl6eXZhQzNCZXhMNk05Z2dlZ0U0ZzhnaVpxN0gxa3NxS3hvUytudUMvCllMcTRuL1pBemt3TXcwdDROMUFmaHFYZFFkdE0xV25ERGlGRmNKRzRxMVJHYlNQSlExZHkvSGo1UlNUN1JIWGYKc3VPWFRLMzJJK2hlTFZQYlhZU05XTlBneHZqUDErY0YxMmlaZlFJREFRQUJBb0lCQVFDVVpUSlRzdDZENXAxNwp3aEYzR0JaMGhEUS94STFxc3FmVWxQVy9PWE9pMEJ2Z3AvQm1neTQyNEo4WE13RENaclpRbkkwa0V1WUlUODVMCmpiS2lhMEVHdkdWY1RuT0pKdTN4Q1NNMXpINlUvaFlzYmVxMmdSSEdVaXV3ZnMwVnc1N3QwOHJrYTdMN3ZRcWwKamhnUTV5c3paeXZyS0hHVGFvRnI4SWluU1RRTlVZbmx0cTY1bEFkWVljdTRuNUZONXc3MUJqaEMxSkR2SVZDYQpwTnpWTm1wa0ZFemRIMFpCMVE5L3NXN2VtWFFJRDI4YmQraEppOHhpNEh5a1hBQjMvYXlLVlkyeFdYT1I2SGRBCmdXdWpSSEdCWWJrOWh4a1lnTXc2T2xjeUM4clFPOE1BaWF3dkkzRm42Vm5tbStQTkVaMjRvLytvR21DNmQwajcKOFFPcTcrRWhBb0dCQVBYZ1BmRldnK0pwSnN0WVFQcnR0YmhqQ3NVNnRlK0xxaHo1NG1BR2J5MWtzaGtGYXBuYwpKWDZudjBzTW1QSzFaK0VhM0Frd1JwK3g4MTloUGhBZm9ndFB3QTlwNFI0VTVvZ1RGZElpNi9IaGt1N2FlR0lFCmYwTVVJM2FLNWd4eVZKeVQ4eFpSUmhONFZXbzhKbnFQVGltU2lPM20xd2tSck42cHN4aW1EVjVGQW9HQkFObUEKcFNZQjVEN1NyMEdOV0VsN0Q2YkJUUm5Jd01QbWhtWEdNZy8wVnpIa01IRkMxL0NBK0cybllWbU5tVk9uRGE2Vwp2MXNTQ2V6dlhzT05MTzRTcFEvWGNlM1JBb0ZMZVlRWHJ1V3JlRzR0K003Q0NhSklFV1J1b290S2pkS3FTNHNYClc2dFFoQTdFZlF5T29KZFpIK1JzMDhiQUtqbkpkSmZCcnI3NW5IM1pBb0dCQUkvZ3pwNkE3OS8zaERvTnhNaVIKZlhzT2llanIrRGJhMVlyck51cldHUXpKRmViS1lwenlWQWFQaHNNVjloSmcwUTdLdHVPeG1rSXMwYXVJcmRhbApqQjl1WlljZmdCb29STlp2eEo4WXpGVFltVENvS3pYQVIvQXF3cTNGamVUT3FuQ05xdnZ1UjBGeGN4b2RaSzduCldPZSsxZDNRN1JoamZYTTJmcFQzNW02NUFvR0FCT2JlVWZYMjgrZUJwYU9PMnRFWU1EVHBET3kyVTNQdGtYdWYKQXpjSmhBbFZnaWYwZitvWUx6aDNmREl4dzIzZlpCRG5yU0RCOG9JT2k2K3gxSWU5Q3BkYmJQV3A3Y29LcWd4eQpsNkIrclZVQjUzNytBcGRITlJFOVBwQm1rOW5jcS96Unh4R2dlRWk3WEw0V0puTU93ZklsNzdQQy85eG5jVWdmCkxSU3BEemtDZ1lBQXl2emUrQnlrWkVnVFcydWExL3NTN2xWMnVOaE5mWTR2RzFaaHJOOXZEckJvMU1wL01meE4KbWo0R2NCRG9wZG15dk5QSXpOc2loaVErdjZtTHdEWkZJckJHNFZadmxLM2d3WEFiMEtzNWpMbVdKa1E1ekVIKwpaTTFHdHVDTjdZVEhHdjc2NmlRSzhFOEdWR05KcDBTSVhvYzdpUzVIcDVRWStMNnBPanA4Rnc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= ~~~ ~~~powershell [root@kubespray ~]# cat /root/.kube/kurator-member2.config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EWXhOVEE1TURRd01Gb1hEVE16TURZeE1qQTVNRFF3TUZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSzViCkZNZXRnRklPUzREekJtcWNyTm1SV0tmZjYxN0ZqRENBVjQrYkxpTEFaUzU0SkU1Yml0SlQzRkJYNU5oaTBneW0KWTdHZHg1NVl6QXJSS0RXbUowbTBKRm0rQXJMSGxUWGNqNzZ5WnBaeGc4MStFdjJ3dnZrcDJINndpaUc5c3diZQplMDhKRTZuQjc5NlpXNnIwRzN5aWdOMmFoOXAxL241OXJoelhYUFUwZXhSZCtwNTlNRW4zK2M5ODdqZnFtTWE4CjA1TFV3bnZnYTNTNGhBWUFGMDFkdDhGVFh4bVBWa01LSU9uTDV2T2EwTUY5LzhHNExRSlpvWUwyOTZsMnFJV0UKV2JQN0phODN1SXVWazN1N1hyZEhub21qVUpENlpHTC8wL0RNdk9EUGQ4a2JSYzN6eDZpek5lVWhnZ0VMMnRzdApOd2s5NlZTNERURXFubkIra0hVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNVVJ3S090VlRid3lpVlludFRBOG9jRy9IdlBNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS0pkRHQwU0pqQjVDczBEYnJLagpEcm8zRkp3RkEzYjd5NzhJN1VYRWtoUFlOYUNGRHdYWUZhOTVPZTRLWm4yZXY5UTRzQy9GZHRaZUpCKzFyR0hTClZXSVBGQzdZK0pJdHUxYUhuaGJRQ2lRM3lmWGozM1MrMXJSUCtTNkpJdU1mRHZ3N0NWNDF0T3JMYnhlSFZhMWsKYVV4RGdwOTJldFl1TWNEWnRwT0pSVTB1TDFlU0FaU000c3c3ejNmTWVtblMzY1dCNmFZTkRxWFlHbUNyK0tkcApVWE9HakpvQkYyRnFnSThHK2VOSWZQcm05QjN3YXljcXZQL3Fod29yK1paL3ZZN2JXeUR4c255aG5BbVEwM2JlCkYrdzVZNnNjaFhmQTZsUzIvL2djRkRucS9RcVAvU3g0Y2MrQ081SWhHYTJoS0NDbGN2NTUwWHVZY2V4N2V2TEMKNUZZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== server: https://192.168.10.143:6443 name: member2 contexts: - context: cluster: member2 user: member2-admin name: kurator-member2 current-context: kurator-member2 kind: Config preferences: {} users: - name: member2-admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJR25sd0QzNVVldmd3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBMk1UVXdPVEEwTURCYUZ3MHlOREEyTVRRd09UQTBNREZhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXlEays0SCtTSkVFV0NKV2wKRG0wb2FxMEZETFZiMjlIaVBQWWgxcDVWZW5CaFZNM2Mrc0pJVjBZR2EwbHpYV252Zy9qazF2bTdPTkZBMzZVUQpLZmlzKy9mS24wWFJYQkUvRjFTREFpU0YrZDRtT1Jkc0lrejRPRTF1eStmWGd4K3RCNllFZ2VRaDBRVzVsTDlxClZoYVJmNzM4MHY4RzB5T1JkaHd6VlB6eGJRUDE5NzBWbWluMm9XanRVNm40a201RGhYOUpMdlBJY0NXMVVLRzEKaGtqSlQ2SCtJK1VZajBiV0Q1bjFYTFhaZXZaYTREaHlCcnlWcVo0bVdtYnVjVkpjRDh0KzlIQkhiTEVnbWhrZQpna2ZoQVRWV043YzBDT0FPU21pdEZsSTF0UGZ1UHZaVmErbzVkMk0yVU81OGM0Uit5SUFFZTZieXZtYlYrMXFOCjVQUkhGd0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JURkVjQ2pyVlUyOE1vbFdKN1V3UEtIQnZ4Nwp6ekFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBUnNXRmoxVkpVZDVaZXlBMU9VZ2RTanJJS0lpMFdYMFFiZTIzCmswb3RISzNxNkdDZ2kxUTdJUGhZM1hJbFBXdmhjd2lLWVYyYUJVcDRwQXZwRmNtMnN5SkViM3FMLzVpQ3A3ZDgKN1BQb1RXS01xemxHQUlIa2NJdE4wQnBxbWxodG8vc1MvbHVqbzlveDBEZ2tvNDBPTHVSWGZoeVZ0QmNlSzY1ZgpYdEVYRlVpdmxhZVFOQ2ZTZmFVcXVMcGxHaHM0ci9TblMvTXIyR01YdnF2QjVOK3ZOK2NBOW1ab2JlMllQYXFtCkszVnZTQ1F6NWpBaE5ZRU4xdFlJSzlWcFgrQ3hrWVhvb0RWS1J3OWxpM1EzS0VhMmtiTGR6RGFXWUN1OC9oYUwKQ2R0K0hhVUJHV0pia0NlQmdMRXd6S0hIS1dhcjJycDQxUEhmWk9CNittZkFiN3Erb0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBeURrKzRIK1NKRUVXQ0pXbERtMG9hcTBGRExWYjI5SGlQUFloMXA1VmVuQmhWTTNjCitzSklWMFlHYTBselhXbnZnL2prMXZtN09ORkEzNlVRS2ZpcysvZktuMFhSWEJFL0YxU0RBaVNGK2Q0bU9SZHMKSWt6NE9FMXV5K2ZYZ3grdEI2WUVnZVFoMFFXNWxMOXFWaGFSZjczODB2OEcweU9SZGh3elZQenhiUVAxOTcwVgptaW4yb1dqdFU2bjRrbTVEaFg5Skx2UEljQ1cxVUtHMWhrakpUNkgrSStVWWowYldENW4xWExYWmV2WmE0RGh5CkJyeVZxWjRtV21idWNWSmNEOHQrOUhCSGJMRWdtaGtlZ2tmaEFUVldON2MwQ09BT1NtaXRGbEkxdFBmdVB2WlYKYStvNWQyTTJVTzU4YzRSK3lJQUVlNmJ5dm1iVisxcU41UFJIRndJREFRQUJBb0lCQUcxbmFrWDlMdnVFWE9lLwo3UHFmQnJDVGJlanR5QTJxOTNYYi84MW5vc2xPVXNDbDVCTll5WEoybnRkTXdTN1B6Y0pNNDYzYnMwTnVnNFl3CmlVNkZ4Ty9CREFSUGdBQlFwQ3E1VVFTQXRIdzl1S0VGZlY2WXRDRC9ub1RvWDhnVUxOVzdEMGR5UWxBV1VydEcKaEhJWnp0TUFHditudmpEc2hIVUhINzFXb1F5eFgwUnZpYkZuN1gxWllrclQ2N2RGcGpLdG9RRTkyUkpRdHpQSApOb0FjMTRCa1VNVEt5ZFpISkUxTmVCRGZ5dStaOXJmeEwxbURuWDBhQWhJdS9XMkdrejZVYitTaFpXL0hsei9XClZSSFgvcHBYWnRQU3RiejJEVlZQZlBrZUtjUVh3QXFWdzMxZnNXNzV4VmZFWWNXR1JYKzZoUlA3SDdRQTV4TnYKaWsramlERUNnWUVBODdLZ3Q0QTJUK3Z5eTduUkpJYjh3ZjdwMVVXajdNWTF3bHBMNTRJUXFWK0pkU2FLRVozdgprait3V3kvUk5NV2pKVlM1M3NVbVgyN0lkWDdoeXIzdU5td2UzSW9pTlg5V09oNTRNdHlWb2E2QXkyNkIxV2wzCmsrZlBMTWxZcGZ4UUtyUDEvV1VtNU10S2dVS2FSbENhVHM0V1hTbG1HL1g3eEQvMWR2NFVMMjBDZ1lFQTBsVEsKSkFSM0RnZHdLRXB2cHBITm9UVUFvV1ovRVJmVXI5VEM0VGtjYnRMYnYxUEdJR1kzcWJHN1V6MzYyVnRRajlJMgpJa2tEd09YZUFQT2xvR2JGR0JkWVNmMmlJOUhJSkQwYm1qaC9UVEI5cU1ZVW9ZMXYrM1J6bUpMbGpYbUkxa1ZDCk5uWXRLQXRlSHNhK0QzMk1LSllWOHY3UGdPY25aNmllSEVUMGloTUNnWUJNdkFaWFFZdllocGNoQnhXUHh1RjMKeEdpa3V4VHV6VXJJTTk2S3V5Mjc1MVZGQ2ltWGVuK2hNV1M2a2NmeU1mazVIQUNhNE1GQWM0V3pXWXlIWmw3SAp6Q3U3amROUHZSVE5vQ0tCQVd3c1NPSk5MREs5bkVRRzBvc3Rzd2l5MjB2eTVrK2l4bU1Feis1WjNUcnV3UnFWCmpkY0U1ckE4UmF4TkUwYWJmVTJsMFFLQmdHY3ovWCtReFU0WDdqdUcwcVA3dE5jZDU3SktWWnp3Z1liSk1kcEwKS0oxa1R6amhCVDFPaW85MFN3TThUVUtyb0tzQkxWakRWTmtvc1Ric1ZsK1BMYlIxZC9mT1FPbXNmTFR6V3dEZwpLZTZaRlg5anVpbk8rWng3ZG1EdDZ6N3czR2l1MFI5aCt5UkQ5Nm5JUU9mZTVZL2duT0ovanhlajFGenppcWs2CldIY0xBb0dBVzJNTDlHeFVwalFmV2EyZW5qWTlybW1yaHVZU0Vmbk4zYlU3ZDBLQzdqWnl4K3RvYVBjQWFZUG4KUTkrN28yWERVa1k0OWw0U0RHZFU5dU1zUnZjaFNseDkydXc0WDNsL3M2U2tERENPUFh4LzBVcmtkbHhweGhGVQoxTUZuRmFxdy8vb2wxaXFmWGxLSnRINTJKdStBZERNUGp1L0cwZmhQY3hoYURJdEoyVjQ9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== ~~~ ### 8.2.2 创建secret ~~~powershell [root@kubespray ~]# kubectl create secret generic kurator-member1 --from-file=kurator-member1.config=/root/.kube/kurator-member1.config ~~~ ~~~powershell [root@kubespray ~]# kubectl create secret generic kurator-member2 --from-file=kurator-member2.config=/root/.kube/kurator-member2.config ~~~ ~~~powershell [root@kubespray ~]# kubectl get secret NAME TYPE DATA AGE cluster-secret Opaque 1 11h kurator-member1 Opaque 1 20s kurator-member2 Opaque 1 7s ~~~ ~~~powershell [root@kubespray ~]# kubectl --kubeconfig=/root/.kube/kurator-member1.config get nodes ~~~ ~~~powershell [root@kubespray ~]# kubectl --kubeconfig=/root/.kube/kurator-member2.config get nodes ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_kustomize.md ================================================ # Kubernetes原生配置管理利器 kustomize # 一、引入kustomize原因 ​ Dcoker是利用集装箱的思想，将依赖和运行环境打包成自包含、轻量级、可移植的容器，它给开发人员带来的切实好处就是一次构建、到处运行，消除了开发、测试、生产环境不一致性。看完之后，不以为然，真的可以完全消除各个环境的不一致性吗？时至今日，Kubernetes 已经上生产，但是各个环境的不一致性，仍然没有解决，大致问题就是，所有服务全部容器化不太现实，比如 MySql、Redis 等，这些服务本身已经存在现有的、稳定的部署方式，且这些服务是不怎么变动的，当然可以使用 Kubernetes 把数据库打成镜像，通过有状态服务资源对象编排，纳入到 Kubernetes 集群管理当中，实现动态扩缩容。但对于中小企业来说，最急切的还是自己业务，对于数据库服务还是使用原有服务器部署，最大程度上降低研发成本。这就带来了如下几个问题： - 其一、开发环境和测试环境连接的数据库地址不是同一个，线上环境更是不同，每次上线都需要维护三份，甚至更多配置即 Kubernetes ConfigMap。 - 其二、通过镜像解决了各个环境的打包问题，但是随之而来的是大量 yaml 编排文件，编排文件如何管理？各个环境虽然镜像一样，但是配置参数可能不同，比如：开发一个副本，但是生产可能需要三个等等。 - 其三、yaml 配置较多，要么一个个执行，或者单独维护脚本批量执行 yaml。 ​ 为了解决不同应用在不同环境中存在使用不同配置参数的复杂问题，容器的生态系统出现了 helm，它大大简化了应用管理的难度，简单来说，helm 类似于 Kubernetes 程序包管理器，用于应用的配置、分发、版本控制、查找等操作。它的核心功能是把 Kubernetes 资源对象（Deployment、ConfigMap、Service）打包到一个 Charts 中，制作完成各个 Charts 保存到 Chart 仓库进行存储和转发，虽然 helm 可以解决 Kubernetes 资源对象生命周期管理以及通过模板的版本控制，但是 helm 使用起来复杂，只想管理几个不同环境 yaml 配置，helm 搞了很多模板等概念，且不支持多租户，虽然 helm v3 抛弃了tiller，同时引入了 lua，本想简单解决 yaml 编排文件问题，却引入更高的复杂度。 ​ 但云原生社区从来不会让我们失望，随之而来的，就是 Kustomize，只有一个 cli 工具，通过这个工具可以打包不同环境的配置，在 Kubernetes 1.14 版本之后，直接集成到 kubectl 命令中，通过执行 kubectl apply -k 命令就可以完成不同环境应用的打包，可以说相当简单。下面对 Kustomize 进行介绍。 # 二、Kustomize 设计理念 Kustomize 允许用户以一个应用描述文件 （YAML 文件）为基础（Base YAML），然后通过 Overlay 的方式生成最终部署应用所需的描述文件。两者都是由 kustomization 文件表示。基础（Base）声明了共享的内容（资源和常见的资源配置），Overlay 则声明了差异。它的设计目的是给 kubernetes 的用户提供一种可以重复使用同一套配置的声明式应用管理，从而在配置工作中用户只需要管理和维护kubernetes的API对象，而不需要学习或安装其它的配置管理工具，也不需要通过复制粘贴来得到新的环境的配置。 # 三、kustomize作用 [Kustomize](https://github.com/kubernetes-sigs/kustomize) 是一个独立的工具，用来通过 [kustomization 文件](https://kubectl.docs.kubernetes.io/references/kustomize/glossary/#kustomization) 定制 Kubernetes 对象。 - 从其他来源生成资源 (依据从其它资源生成资源) - 为资源设置贯穿性字段 (设置资源字段) - 组织和定制资源集合 (组合和定制资源集合) # 四、kustomize使用 ## 4.1 通过kubectl使用 kustomization 文件来管理 Kubernetes 对象介绍 > 从 1.14 版本开始，`kubectl` 也开始支持使用 kustomization 文件来管理 Kubernetes 对象，所以你必须拥有一个 Kubernetes 的集群，同时你的 Kubernetes 集群必须带有 kubectl 命令行工具 ## 4.2 命令格式 ```powershell 查看包含 kustomization 文件的目录中的资源 kubectl kustomize ``` ~~~powershell 应用这些资源，使用 --kustomize 或 -k 参数来执行 kubectl apply kubectl apply -k ~~~ ## 4.3 命令使用 ### 4.3.1 生成资源 ConfigMap 和 Secret 包含其他 Kubernetes 对象（如 Pod）所需要的配置或敏感数据。 ConfigMap 或 Secret 中数据的来源往往是集群外部，例如某个 `.properties` 文件或者 SSH 密钥文件。 Kustomize 提供 `secretGenerator` 和 `configMapGenerator`，可以基于文件或字面值来生成 Secret 和 ConfigMap。 ##### 4.3.1.1 configMapGenerator生成configMap ##### 4.3.1.1.1 基于属性文件来生成 ConfigMap 要基于文件来生成 ConfigMap，可以在 `configMapGenerator` 的 `files` 列表中添加表项。 下面是一个根据 `.properties` 文件中的数据条目来生成 ConfigMap 的示例： ```powershell # 生成一个 application.properties 文件 cat <application.properties FOO=Bar EOF cat <./kustomization.yaml configMapGenerator: - name: example-configmap-1 files: - application.properties EOF ``` 所生成的 ConfigMap 可以使用下面的命令来检查： ```powershell kubectl kustomize ./ ``` 所生成的 ConfigMap 为： ```yaml apiVersion: v1 data: application.properties: | FOO=Bar kind: ConfigMap metadata: name: example-configmap-1-8mbdf7882g ``` ##### 4.3.1.1.2 env 文件生成 ConfigMap 要从 env 文件生成 ConfigMap，请在 `configMapGenerator` 中的 `envs` 列表中添加一个条目。 下面是一个用来自 `.env` 文件的数据生成 ConfigMap 的例子： ```shell # 创建一个 .env 文件 cat <.env FOO=Bar EOF cat <./kustomization.yaml configMapGenerator: - name: example-configmap-1 envs: - .env EOF ``` 可以使用以下命令检查生成的 ConfigMap： ```shell kubectl kustomize ./ ``` 生成的 ConfigMap 为： ```yaml apiVersion: v1 data: FOO: Bar kind: ConfigMap metadata: name: example-configmap-1-42cfbf598f ``` **说明：** `.env` 文件中的每个变量在生成的 ConfigMap 中成为一个单独的键。 这与之前的示例不同，前一个示例将一个名为 `.properties` 的文件（及其所有条目）嵌入到同一个键的值中。 #### 4.3.1.2 使用生成的configMap 要在 Deployment 中使用生成的 ConfigMap，使用 configMapGenerator 的名称对其进行引用。 Kustomize 将自动使用生成的名称替换该名称。 这是使用生成的 ConfigMap 的 deployment 示例： ```yaml # 创建一个 application.properties 文件 cat <application.properties FOO=Bar EOF cat <deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app labels: app: my-app spec: selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: app image: my-app volumeMounts: - name: config mountPath: /config volumes: - name: config configMap: name: example-configmap-1 EOF cat <./kustomization.yaml resources: - deployment.yaml configMapGenerator: - name: example-configmap-1 files: - application.properties EOF ``` 生成 ConfigMap 和 Deployment： ```shell kubectl kustomize ./ ``` 生成的 Deployment 将通过名称引用生成的 ConfigMap： ```yaml apiVersion: v1 data: application.properties: | FOO=Bar kind: ConfigMap metadata: name: example-configmap-1-g4hk9g2ff8 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: my-app name: my-app spec: selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - image: my-app name: app volumeMounts: - mountPath: /config name: config volumes: - configMap: name: example-configmap-1-g4hk9g2ff8 name: config ``` #### 4.3.1.2 设置贯穿性字段 在项目中为所有 Kubernetes 对象设置贯穿性字段是一种常见操作。 贯穿性字段的一些使用场景如下： - 为所有资源设置相同的名字空间 - 为所有对象添加相同的前缀或后缀 - 为对象添加相同的标签(Labels)集合 - 为对象添加相同的注解(annotations)集合 ```shell # 创建一个 deployment.yaml cat <./deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx EOF cat <./kustomization.yaml namespace: my-namespace namePrefix: dev- nameSuffix: "-001" commonLabels: app: bingo commonAnnotations: oncallPager: 400-500-600 resources: - deployment.yaml EOF ``` 执行 `kubectl kustomize ./` 查看这些字段都被设置到 Deployment 资源上： ```yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: oncallPager: 400-500-600 labels: app: bingo name: dev-nginx-deployment-001 namespace: my-namespace spec: selector: matchLabels: app: bingo template: metadata: annotations: oncallPager: 400-500-600 labels: app: bingo spec: containers: - image: nginx name: nginx ``` #### 4.3.1.3 组合和定制资源 一种常见的做法是在项目中构造资源集合并将其放到同一个文件或目录中管理。 Kustomize 提供基于不同文件来组织资源并向其应用补丁或者其他定制的能力。 ##### 4.3.1.3.1 组合 Kustomize 支持组合不同的资源。`kustomization.yaml` 文件的 `resources` 字段定义配置中要包含的资源列表。 你可以将 `resources` 列表中的路径设置为资源配置文件的路径。 下面是由 Deployment 和 Service 构成的 NGINX 应用的示例： ```shell # 创建 deployment.yaml 文件 cat < deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 EOF # 创建 service.yaml 文件 cat < service.yaml apiVersion: v1 kind: Service metadata: name: my-nginx labels: run: my-nginx spec: ports: - port: 80 protocol: TCP selector: run: my-nginx EOF # 创建 kustomization.yaml 来组织以上两个资源 cat <./kustomization.yaml resources: - deployment.yaml - service.yaml EOF ``` `kubectl kustomize ./` 所得到的资源中既包含 Deployment 也包含 Service 对象。 ##### 4.3.1.3.2 定制 ###### 4.3.1.3.2.1 patchesStrategicMerge 补丁文件（Patches）可以用来对资源执行不同的定制。 Kustomize 通过 `patchesStrategicMerge` 和 `patchesJson6902` 支持不同的打补丁机制。 `patchesStrategicMerge` 的内容是一个文件路径的列表，其中每个文件都应可解析为 [策略性合并补丁（Strategic Merge Patch）](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md)。 补丁文件中的名称必须与已经加载的资源的名称匹配。 建议构造规模较小的、仅做一件事情的补丁。 例如，构造一个补丁来增加 Deployment 的副本个数；构造另外一个补丁来设置内存限制。 ```shell # 创建 deployment.yaml 文件 cat < deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 EOF # 生成一个补丁 increase_replicas.yaml cat < increase_replicas.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: replicas: 3 EOF # 生成另一个补丁 set_memory.yaml cat < set_memory.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: template: spec: containers: - name: my-nginx resources: limits: memory: 512Mi EOF cat <./kustomization.yaml resources: - deployment.yaml patchesStrategicMerge: - increase_replicas.yaml - set_memory.yaml EOF ``` 执行 `kubectl kustomize ./` 来查看 Deployment： ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: replicas: 3 selector: matchLabels: run: my-nginx template: metadata: labels: run: my-nginx spec: containers: - image: nginx name: my-nginx ports: - containerPort: 80 resources: limits: memory: 512Mi ``` ###### 4.3.1.3.2.2 patchesJson6902 并非所有资源或者字段都支持策略性合并补丁。为了支持对任何资源的任何字段进行修改， Kustomize 提供通过 `patchesJson6902` 来应用 [JSON 补丁](https://tools.ietf.org/html/rfc6902)的能力。 为了给 JSON 补丁找到正确的资源，需要在 `kustomization.yaml` 文件中指定资源的组（group）、 版本（version）、类别（kind）和名称（name）。 例如，为某 Deployment 对象增加副本个数的操作也可以通过 `patchesJson6902` 来完成： ```shell # 创建一个 deployment.yaml 文件 cat < deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 EOF # 创建一个 JSON 补丁文件 cat < patch.yaml - op: replace path: /spec/replicas value: 3 EOF # 创建一个 kustomization.yaml cat <./kustomization.yaml resources: - deployment.yaml patchesJson6902: - target: group: apps version: v1 kind: Deployment name: my-nginx path: patch.yaml EOF ``` 执行 `kubectl kustomize ./` 以查看 `replicas` 字段被更新： ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: replicas: 3 selector: matchLabels: run: my-nginx template: metadata: labels: run: my-nginx spec: containers: - image: nginx name: my-nginx ports: - containerPort: 80 ``` ###### 4.3.1.3.2.3 通过kustomization.yaml文件中images字段注入 除了补丁之外，Kustomize 还提供定制容器镜像或者将其他对象的字段值注入到容器中的能力，并且不需要创建补丁。 例如，你可以通过在 `kustomization.yaml` 文件的 `images` 字段设置新的镜像来更改容器中使用的镜像。 ```shell cat < deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 EOF cat <./kustomization.yaml resources: - deployment.yaml images: - name: nginx newName: my.image.registry/nginx newTag: 1.4.0 EOF ``` 执行 `kubectl kustomize ./` 以查看所使用的镜像已被更新： ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: replicas: 2 selector: matchLabels: run: my-nginx template: metadata: labels: run: my-nginx spec: containers: - image: my.image.registry/nginx:1.4.0 name: my-nginx ports: - containerPort: 80 ``` ###### 4.3.1.3.2.4 通过变量注入字段 有些时候，Pod 中运行的应用可能需要使用来自其他对象的配置值。 例如，某 Deployment 对象的 Pod 需要从环境变量或命令行参数中读取读取 Service 的名称。 由于在 `kustomization.yaml` 文件中添加 `namePrefix` 或 `nameSuffix` 时 Service 名称可能发生变化，建议不要在命令参数中硬编码 Service 名称。 对于这种使用场景，Kustomize 可以通过 `vars` 将 Service 名称注入到容器中。 ```shell # 创建一个 deployment.yaml 文件（引用此处的文档分隔符） cat <<'EOF' > deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx command: ["start", "--host", "$(MY_SERVICE_NAME)"] EOF # 创建一个 service.yaml 文件 cat < service.yaml apiVersion: v1 kind: Service metadata: name: my-nginx labels: run: my-nginx spec: ports: - port: 80 protocol: TCP selector: run: my-nginx EOF cat <./kustomization.yaml namePrefix: dev- nameSuffix: "-001" resources: - deployment.yaml - service.yaml vars: - name: MY_SERVICE_NAME objref: kind: Service name: my-nginx apiVersion: v1 EOF ``` 执行 `kubectl kustomize ./` 以查看注入到容器中的 Service 名称是 `dev-my-nginx-001`： ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: dev-my-nginx-001 spec: replicas: 2 selector: matchLabels: run: my-nginx template: metadata: labels: run: my-nginx spec: containers: - command: - start - --host - dev-my-nginx-001 image: nginx name: my-nginx ``` ### 4.3.2 基准（Bases）与覆盖（Overlays） Kustomize 中有 **基准（bases）** 和 **覆盖（overlays）** 的概念区分。 **基准** 是包含 `kustomization.yaml` 文件的一个目录，其中包含一组资源及其相关的定制。 基准可以是本地目录或者来自远程仓库的目录，只要其中存在 `kustomization.yaml` 文件即可。 **覆盖** 也是一个目录，其中包含将其他 kustomization 目录当做 `bases` 来引用的 `kustomization.yaml` 文件。 **基准**不了解覆盖的存在，且可被多个覆盖所使用。 覆盖则可以有多个基准，且可针对所有基准中的资源执行组织操作，还可以在其上执行定制。 ```shell # 创建一个包含基准的目录 mkdir base # 创建 base/deployment.yaml cat < base/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx EOF # 创建 base/service.yaml 文件 cat < base/service.yaml apiVersion: v1 kind: Service metadata: name: my-nginx labels: run: my-nginx spec: ports: - port: 80 protocol: TCP selector: run: my-nginx EOF # 创建 base/kustomization.yaml cat < base/kustomization.yaml resources: - deployment.yaml - service.yaml EOF ``` 此基准可在多个覆盖中使用。你可以在不同的覆盖中添加不同的 `namePrefix` 或其他贯穿性字段。 下面是两个使用同一基准的覆盖： ```shell mkdir dev cat < dev/kustomization.yaml bases: - ../base namePrefix: dev- EOF mkdir prod cat < prod/kustomization.yaml bases: - ../base namePrefix: prod- EOF ``` 执行 `kubectl kustomize ./` 以查看覆盖中的资源对象 ### 4.3.3 如何使用 Kustomize 来应用、查看和删除对象 在 `kubectl` 命令中使用 `--kustomize` 或 `-k` 参数来识别被 `kustomization.yaml` 所管理的资源。 注意 `-k` 要指向一个 kustomization 目录。例如： ```shell kubectl apply -k / ``` 假定使用下面的 `kustomization.yaml`： ```shell # 创建 deployment.yaml 文件 cat < deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 EOF # 创建 kustomization.yaml cat <./kustomization.yaml namePrefix: dev- commonLabels: app: my-nginx resources: - deployment.yaml EOF ``` 执行下面的命令来应用 Deployment 对象 `dev-my-nginx`： ```shell kubectl apply -k ./ deployment.apps/dev-my-nginx created ``` 运行下面的命令之一来查看 Deployment 对象 `dev-my-nginx`： ```shell kubectl get -k ./ kubectl describe -k ./ ``` 执行下面的命令删除 Deployment 对象 `dev-my-nginx`： ```shell kubectl delete -k ./ deployment.apps "dev-my-nginx" deleted ``` # 五、kustomize客户端应用 >通过kustomize客户端来使用 kustomization 文件来管理 Kubernetes 对象 ## 5.1 kustomize客户端下载 ```powershell # curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash # mv kustomize /usr/bin ``` 命令格式 ```powershell kustomize [command] ``` ## 5.2 创建资源 `kustomize build`递归地构建指向的 kustomization.yaml，从而生成一组准备好部署的 Kubernetes 资源 ```shell # 创建一个 deployment.yaml cat <./deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx EOF cat <./kustomization.yaml namespace: my-namespace namePrefix: dev- nameSuffix: "-001" commonLabels: app: bingo commonAnnotations: oncallPager: 400-500-600 resources: - deployment.yaml EOF ``` 通过kustomize build ./ 来查看生成的Kubernetes 资源 ## 5.3 修改资源 `kustomize edit`自定义的修改你的kustomization.yaml ```powershell # 创建一个 deployment.yaml cat <./deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx EOF cat <./kustomization.yaml namespace: my-namespace namePrefix: dev- nameSuffix: "-001" commonLabels: app: bingo commonAnnotations: oncallPager: 400-500-600 resources: - deployment.yaml EOF ``` 通过kustomize edit 来修改Kubernetes 资源 ```powershell # kustomize edit set image nginx=mynginx:v1.1 # cat kustomization.yaml namespace: my-namespace namePrefix: dev- nameSuffix: "-001" commonLabels: app: bingo commonAnnotations: oncallPager: 400-500-600 resources: - deployment.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization images: - name: nginx newName: mynginx newTag: v1.1 ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_logs_collect.md ================================================ # kubernetes日志收集方案 ELK # 一、为什么收集日志 收集日志可以用于： - 分析用户行为 - 监控服务器状态 - 增强系统或应用安全性等。 # 二、收集哪些日志 - kubernetes集群节点系统日志 - kubernetes集群节点应用程序日志 - kubernetes集群中部署的应用程序日志 # 三、日志收集方案 ## 3.1 日志收集技术栈ELK(ELKB) + Filebeat ![](../../img/kubernetes/kubernetes_logs_collect/image-20200106175502929.png) ## 3.2 日志收集技术栈 EK(EFK) + fluentd ![image-20220408005109209](../../img/kubernetes/kubernetes_logs_collect/image-20220408005109209.png) # 四、ELK集群部署 > 为了增加ELK集群的运行效率，一般建议在k8s集群之外使用物理机部署ELK集群，当然也可以直接在k8s集群内部署。 ## 4.1 主机准备 | 主机 | 软件 | 版本 | 配置 | IP | | -------- | ------------- | ------ | ---- | -------------- | | kibana | kibana | 7.17.2 | 2C2G | 192.168.10.200 | | elastic | elasticsearch | 7.17.2 | 2C4G | 192.168.10.201 | | logstash | logstash | 7.17.2 | 2C4G | 192.168.10.202 | ~~~powershell # hostname set-hostname xxx ~~~ ~~~powershell # cat /etc/hosts 192.168.10.200 kibana 192.168.10.201 elastic 192.168.10.202 logstash ~~~ ## 4.2 软件安装 > 由于软件下载较慢，请提前准备好以下软件。 ### 4.2.1 安装jdk > 所有主机全部安装，可考虑使用openjdk也可以使用oracle jdk。 ~~~powershell [root@kibana ~]# yum -y install java-11-openjdk [root@elastic ~]# yum -y install java-11-openjdk [root@logstash ~]# yum -y install java-11-openjdk ~~~ ### 4.2.2 安装kibana ![image-20220407154620497](../../img/kubernetes/kubernetes_logs_collect/image-20220407154620497.png) 或 ![image-20220407154735245](../../img/kubernetes/kubernetes_logs_collect/image-20220407154735245.png) ![image-20220407154902092](../../img/kubernetes/kubernetes_logs_collect/image-20220407154902092.png) ![image-20220407154953016](../../img/kubernetes/kubernetes_logs_collect/image-20220407154953016.png) ![image-20220407155103366](../../img/kubernetes/kubernetes_logs_collect/image-20220407155103366.png) ![image-20220407155556894](../../img/kubernetes/kubernetes_logs_collect/image-20220407155556894.png) ![image-20220407155702199](../../img/kubernetes/kubernetes_logs_collect/image-20220407155702199.png) ~~~powershell # wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.2-x86_64.rpm ~~~ ~~~powershell # yum -y install kibana-7.17.2-x86_64.rpm ~~~ ### 4.2.3 安装elasticsearch ![image-20220407160045282](../../img/kubernetes/kubernetes_logs_collect/image-20220407160045282.png) ![image-20220407160127340](../../img/kubernetes/kubernetes_logs_collect/image-20220407160127340.png) ~~~powershell # wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.2-x86_64.rpm ~~~ ~~~powershell # yum -y install elasticsearch-7.17.2-x86_64.rpm ~~~ ### 4.2.4 安装logstash ![image-20220407160405048](../../img/kubernetes/kubernetes_logs_collect/image-20220407160405048.png) ![image-20220407160443555](../../img/kubernetes/kubernetes_logs_collect/image-20220407160443555.png) ~~~powershell # wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.2-x86_64.rpm ~~~ ~~~powershell # yum -y install logstash-7.17.2-x86_64.rpm ~~~ ## 4.3 软件配置及启动 ### 4.3.1 kibana软件配置及启动 ~~~powershell [root@kibana ~]# cat -n /etc/kibana/kibana.yml | grep -v "#" | grep -v "^$" 2 server.port: 5601 7 server.host: "192.168.10.200" 32 elasticsearch.hosts: ["http://192.168.10.201:9200"] 115 i18n.locale: "zh-CN" ~~~ ~~~powershell 说明： server.port 是开启kibana监听端口 server.host 设置远程连接主机IP地址，用于远程访问使用 elasticsearch.hosts 设置elasticsearch.hosts主机IP，用于连接elasticsearch主机，可以为多个值 i18n.locale 设置语言支持，不需要再汉化，直接修改后即可支持中文 ~~~ ~~~powershell [root@kibana ~]# systemctl enable kibana [root@kibana ~]# systemctl start kibana ~~~ ~~~powershell [root@kibana ~]# ss -anput | grep ":5601" tcp LISTEN 0 128 192.168.10.200:5601 *:* users:(("node",pid=2571,fd=71)) ~~~ ### 4.3.2 elasticsearch软件配置及启动 ~~~powershell 修改配置文件 [root@elastic ~]# cat -n /etc/elasticsearch/elasticsearch.yml | grep -v "#" | grep -v "^$" 17 cluster.name: k8s-elastic 23 node.name: elastic 33 path.data: /var/lib/elasticsearch 37 path.logs: /var/log/elasticsearch 56 network.host: 192.168.10.201 61 http.port: 9200 70 discovery.seed_hosts: ["192.168.10.201"] 74 cluster.initial_master_nodes: ["192.168.10.201"] ~~~ ~~~powershell 说明 cluster.name 集群名称 node.name 节点名称 path.data 数据目录 path.logs 日志目录 network.host 主机IP http.port 监听端口 discovery.seed_hosts 主机发现列表 cluster.initial_master_nodes 集群master节点 ~~~ ~~~powershell 启动服务并验证 [root@elastic ~]# systemctl enable elasticsearch [root@elastic ~]# systemctl start elasticsearch ~~~ ~~~powershell [root@elastic ~]# ss -anput | grep ":9200" tcp LISTEN 0 128 [::ffff:192.168.10.201]:9200 [::]:* users:(("java",pid=9726,fd=219)) ~~~ ~~~powershell [root@elastic ~]# curl http://192.168.10.201:9200 { "name" : "elastic", "cluster_name" : "k8s-elastic", "cluster_uuid" : "cW78ZkrhS4OV41DV5CtWWQ", "version" : { "number" : "7.17.2", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "de7261de50d90919ae53b0eff9413fd7e5307301", "build_date" : "2022-03-28T15:12:21.446567561Z", "build_snapshot" : false, "lucene_version" : "8.11.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } ~~~ ### 4.3.3 logstash软件配置及启动 #### 4.3.3.1 修改配置文件 ~~~powershell [root@logstash ~]# cat -n /etc/logstash/logstash.yml | grep -v "#" | grep -v "^$" 19 node.name: logstash 28 path.data: /var/lib/logstash 133 api.http.host: 192.168.10.202 139 api.http.port: 9600-9700 280 path.logs: /var/log/logstash 分布式架构中 api.http.host一定要配置为logstash主机IP，不然无法远程访问。 ~~~ #### 4.3.3.2 启动服务 **logstash进程不用预先启动，使用时启动即可** #### 4.3.3.3 验证logstash可用性 ~~~powershell 标准输入及标准输出验证 [root@logstash ~]# /usr/share/logstash/bin/logstash -e 'input {stdin{} } output {stdout {} }' Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO ] 2022-04-07 16:34:16.332 [main] runner - Starting Logstash {"logstash.version"=>"7.17.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86_64]"} [INFO ] 2022-04-07 16:34:16.339 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true] [INFO ] 2022-04-07 16:34:16.385 [main] settings - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [INFO ] 2022-04-07 16:34:16.423 [main] settings - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [WARN ] 2022-04-07 16:34:16.890 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2022-04-07 16:34:16.956 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"608c9b46-8138-44b6-8cd8-c42d0fb08a90", :path=>"/usr/share/logstash/data/uuid"} [INFO ] 2022-04-07 16:34:18.587 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [INFO ] 2022-04-07 16:34:19.018 [Converge PipelineAction::Create
] Reflections - Reflections took 76 ms to scan 1 urls, producing 119 keys and 419 values [WARN ] 2022-04-07 16:34:19.571 [Converge PipelineAction::Create
] line - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN ] 2022-04-07 16:34:19.590 [Converge PipelineAction::Create
] stdin - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [INFO ] 2022-04-07 16:34:19.839 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["config string"], :thread=>"#"} [INFO ] 2022-04-07 16:34:20.536 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.69} WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [INFO ] 2022-04-07 16:34:20.602 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} The stdin plugin is now waiting for input: [INFO ] 2022-04-07 16:34:20.662 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} abc 输入abc字符，查看其输出 { "@timestamp" => 2022-04-07T08:35:24.663Z, "host" => "logstash", "message" => "abc", "@version" => "1" } 以json格式输出abc内容 ~~~ ~~~powershell 使用logstash输入内容到elasticsearch验证 [root@logstash ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["192.168.10.201:9200"] index => "logstash-%{+YYYY.MM.dd}" } }' hello elasticsearch 此内容将会通过kibana页面中的索引看到，但是需要在kibana页面中添加索引 ~~~ ## 4.4 kibana访问 ![image-20220407163953277](../../img/kubernetes/kubernetes_logs_collect/image-20220407163953277.png) ![image-20220407165920127](../../img/kubernetes/kubernetes_logs_collect/image-20220407165920127.png) ![image-20220407165959162](../../img/kubernetes/kubernetes_logs_collect/image-20220407165959162.png) ![image-20220407170046439](../../img/kubernetes/kubernetes_logs_collect/image-20220407170046439.png) ![image-20220407170142024](../../img/kubernetes/kubernetes_logs_collect/image-20220407170142024.png) ![image-20220407172812976](../../img/kubernetes/kubernetes_logs_collect/image-20220407172812976.png) ![image-20220407172535568](../../img/kubernetes/kubernetes_logs_collect/image-20220407172535568.png) ## 4.5 编写logstash用于收集日志配置文件 > 通过filebeat进行收集 ~~~powershell [root@logstash ~]# cat /etc/logstash/conf.d/logstash-to-elastic.conf input { beats { host => "0.0.0.0" port => "5044" } } filter { } output { elasticsearch { hosts => "192.168.10.201:9200" index => "k8s-%{+YYYY.MM.dd}" } } ~~~ ## 4.6 运行logstash > 如果不涉及多个配置文件，可以直接使用systemctl start logstash;如果有多个配置文件，只想启动一个配置文件，可以使用如下方法。 ### 4.6.1 直接在后台运行 ~~~powershell [root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-elastic.conf --path.data /usr/share/logstash/data1 & ~~~ ### 4.6.2 通过rc.local设置自动后台运行 ~~~powershell [root@logstash ~]# cat /etc/rc.local ... /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-elastic.conf & 查看文件默认权限 [root@logstash ~]# ls -l /etc/rc.d/rc.local -rw-r--r-- 1 root root 562 1月 9 13:40 /etc/rc.d/rc.local 修改文件权限 [root@logstash ~]# chmod +x /etc/rc.d/rc.local 查看修改后文件权限 [root@logstash ~]# ls -l /etc/rc.d/rc.local -rwxr-xr-x 1 root root 562 1月 9 13:40 /etc/rc.d/rc.local ~~~ # 五、收集k8s集群节点系统日志 >通过在work节点以DaemonSet方法运行filebeat应用实现 ## 5.1 下载filebeat镜像 > 所有work节点 ![image-20220407175113144](../../img/kubernetes/kubernetes_logs_collect/image-20220407175113144.png) ![image-20220407175159994](../../img/kubernetes/kubernetes_logs_collect/image-20220407175159994.png) ~~~powershell 下载filebeat镜像 [root@k8s-work1 ~]# docker pull elastic/filebeat:7.17.2 ~~~ ~~~powershell [root@k8s-work1 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.elastic/filebeat 7.17.2 00c5b17745d1 3 weeks ago 359MB ~~~ 或 ~~~powershell 使用containerd时使用 # crictl pull elastic/filebeat:7.17.2 ~~~ ~~~powershell # crictl images IMAGE TAG IMAGE ID SIZE docker.io/elastic/filebeat 7.17.2 2314640a78873 107MB ~~~ ## 5.2 创建filebeat资源清单文件 ~~~powershell [root@k8s-master1 ~]# cat filebeat-to-logstash.yaml apiVersion: v1 kind: ConfigMap metadata: name: k8s-logs-filebeat-config namespace: kube-system data: filebeat.yml: | filebeat.inputs: - type: log paths: - /var/log/messages fields: app: k8s type: module fields_under_root: true setup.ilm.enabled: false setup.template.name: "k8s-module" setup.template.pattern: "k8s-module-*" output.logstash: hosts: ['192.168.10.202:5044'] index: "k8s-module-%{+yyyy.MM.dd}" --- apiVersion: apps/v1 kind: DaemonSet metadata: name: k8s-logs namespace: kube-system spec: selector: matchLabels: project: k8s app: filebeat template: metadata: labels: project: k8s app: filebeat spec: containers: - name: filebeat image: docker.io/elastic/filebeat:7.17.2 args: [ "-c", "/etc/filebeat.yml", "-e", ] resources: requests: cpu: 100m memory: 100Mi limits: cpu: 500m memory: 500Mi securityContext: runAsUser: 0 volumeMounts: - name: filebeat-config mountPath: /etc/filebeat.yml subPath: filebeat.yml - name: k8s-logs mountPath: /var/log/messages volumes: - name: k8s-logs hostPath: path: /var/log/messages - name: filebeat-config configMap: name: k8s-logs-filebeat-config ~~~ ## 5.3 应用filebeat资源清单文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f filebeat-to-logstash.yaml ~~~ ## 5.4 验证结果 ~~~powershell 查看pod # kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE k8s-logs-s8qw6 1/1 Running 0 15s 10.244.194.83 k8s-worker1 ~~~ ~~~powershell 查看pod输出日志 [root@k8s-master1 ~]# kubectl logs k8s-logs-6mqq5 -n kube-system ~~~ ## 5.5 在kibana中添加索引 ![image-20220407182931857](../../img/kubernetes/kubernetes_logs_collect/image-20220407182931857.png) ![image-20220407203219708](../../img/kubernetes/kubernetes_logs_collect/image-20220407203219708.png) ![image-20220407203312600](../../img/kubernetes/kubernetes_logs_collect/image-20220407203312600.png) ![image-20220407203403595](../../img/kubernetes/kubernetes_logs_collect/image-20220407203403595.png) ![image-20220407203454587](../../img/kubernetes/kubernetes_logs_collect/image-20220407203454587.png) ![image-20220407203546362](../../img/kubernetes/kubernetes_logs_collect/image-20220407203546362.png) # 六、收集kubernetes节点应用程序日志 > 本案例在k8s-worker1主机上安装nginx并收集其日志 ## 6.1 安装nginx应用 ~~~powershell # wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo ~~~ ~~~powershell # yum -y install nginx ~~~ ~~~powershell # cd /usr/share/nginx/html/ # ls 404.html 50x.html en-US icons img index.html nginx-logo.png poweredby.png # echo "work1 web page" > index.html ~~~ ~~~powershell # systemctl enable nginx # systemctl start nginx ~~~ ~~~powershell # curl http://192.168.10.15 work1 web page ~~~ ## 6.2 编写filebeat资源清单文件 ~~~powershell [root@k8s-master1 ~]# cat filebeat-to-logstash-nginx.yaml apiVersion: v1 kind: ConfigMap metadata: name: k8s-filebeat-config-nginx-logs namespace: default data: filebeat.yml: | filebeat.inputs: - type: log paths: - /var/log/nginx/access.log fields: app: k8s type: module fields_under_root: true - type: log paths: - /var/log/nginx/error.log fields: app: k8s type: module fields_under_root: true setup.ilm.enabled: false setup.template.name: "k8s-module" setup.template.pattern: "k8s-module-*" output.logstash: hosts: ['192.168.10.202:5055'] --- apiVersion: apps/v1 kind: DaemonSet metadata: name: k8s-logs namespace: default spec: selector: matchLabels: project: k8s app: filebeat template: metadata: labels: project: k8s app: filebeat spec: nodeName: k8s-worker1 containers: - name: filebeat image: docker.io/elastic/filebeat:7.17.2 imagePullPolicy: IfNotPresent args: [ "-c", "/etc/filebeat.yml", "-e", ] resources: requests: cpu: 100m memory: 100Mi limits: cpu: 500m memory: 500Mi securityContext: runAsUser: 0 volumeMounts: - name: filebeat-config mountPath: /etc/filebeat.yml subPath: filebeat.yml - name: nginx-access mountPath: /var/log/nginx/access.log - name: nginx-error mountPath: /var/log/nginx/error.log volumes: - name: nginx-access hostPath: path: /var/log/nginx/access.log - name: nginx-error hostPath: path: /var/log/nginx/error.log - name: filebeat-config configMap: name: k8s-filebeat-config-nginx-logs ~~~ ## 6.3 编写logstash配置文件 ~~~powershell [root@logstash ~]# cat /etc/logstash/conf.d/nginx-logstash-to-elastic.conf input { beats { host => "0.0.0.0" port => "5055" } } filter { } output { elasticsearch { hosts => "192.168.10.201:9200" index => "nginx-%{+YYYY.MM.dd}" } } ~~~ ## 6.4 重启logstash ~~~powershell [root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-logstash-to-elastic.conf --path.data /usr/share/logstash/data2 & ~~~ ~~~powershell [root@logstash ~]# ss -anput | grep ":5055" tcp LISTEN 0 128 [::]:5055 [::]:* users:(("java",pid=14296,fd=106)) ~~~ ## 6.5 应用filebeat资源清单文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f filebeat-to-logstash-nginx.yaml configmap/k8s-filebeat-config-nginx-logs created daemonset.apps/k8s-logs created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES k8s-logs-ndznb 1/1 Running 0 14s 10.244.194.84 k8s-worker1 ~~~ ~~~powershell [root@k8s-master1 ~]# # kubectl logs k8s-logs-ndznb ~~~ ## 6.6 在kibana界面添加索引 ![image-20220407223459697](../../img/kubernetes/kubernetes_logs_collect/image-20220407223459697.png) ![image-20220407223538269](../../img/kubernetes/kubernetes_logs_collect/image-20220407223538269.png) ![image-20220407223613198](../../img/kubernetes/kubernetes_logs_collect/image-20220407223613198.png) ![image-20220407223651714](../../img/kubernetes/kubernetes_logs_collect/image-20220407223651714.png) ![image-20220407223727384](../../img/kubernetes/kubernetes_logs_collect/image-20220407223727384.png) ![image-20220407223838150](../../img/kubernetes/kubernetes_logs_collect/image-20220407223838150.png) # 七、收集kubernetes集群中以Pod方式运行的应用日志 > 通过在应用程序Pod中运行filebeat(sidecar)实现,本次将以tomcat为例进行说明。 ## 7.1 准备tomcat数据目录 > 默认tomcat容器中没有网站首页文件，不添加会导致pod中容器无法正常运行。 ~~~powershell [root@k8s-worker1 ~]# mkdir /opt/tomcatwebroot ~~~ ~~~powershell [root@k8s-worker1 ~]# echo "tomcat running" > /opt/tomcatwebroot/index.html ~~~ ## 7.2 编写tomcat应用资源清单文件 ~~~powershell [root@k8s-master1 ~]# cat tomcat-logs.yaml apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-demo namespace: default spec: replicas: 2 selector: matchLabels: project: www app: tomcat-demo template: metadata: labels: project: www app: tomcat-demo spec: nodeName: k8s-worker1 containers: - name: tomcat image: tomcat:latest imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: web protocol: TCP resources: requests: cpu: 0.5 memory: 1Gi limits: cpu: 1 memory: 2Gi livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 60 timeoutSeconds: 20 readinessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 60 timeoutSeconds: 20 volumeMounts: - name: tomcat-logs mountPath: /usr/local/tomcat/logs - name: tomcatwebroot mountPath: /usr/local/tomcat/webapps/ROOT - name: filebeat image: docker.io/elastic/filebeat:7.17.2 imagePullPolicy: IfNotPresent args: [ "-c", "/etc/filebeat.yml", "-e", ] resources: limits: memory: 500Mi requests: cpu: 100m memory: 100Mi securityContext: runAsUser: 0 volumeMounts: - name: filebeat-config mountPath: /etc/filebeat.yml subPath: filebeat.yml - name: tomcat-logs mountPath: /usr/local/tomcat/logs volumes: - name: tomcat-logs emptyDir: {} - name: tomcatwebroot hostPath: path: /opt/tomcatwebroot type: Directory - name: filebeat-config configMap: name: filebeat-config --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: default data: filebeat.yml: |- filebeat.inputs: - type: log paths: - /usr/local/tomcat/logs/catalina.* fields: app: www type: tomcat-catalina fields_under_root: true multiline: pattern: '^\[' negate: true match: after setup.ilm.enabled: false setup.template.name: "tomcat-catalina" setup.template.pattern: "tomcat-catalina-*" output.logstash: hosts: ['192.168.10.202:5056'] ~~~ ## 7.3 编写logstash配置文件 ~~~powershell 编写logstash配置文件，不影响以往配置文件 [root@logstash ~]# cat /etc/logstash/conf.d/tomcat-logstash-to-elastic.conf input { beats { host => "0.0.0.0" port => "5056" } } filter { } output { elasticsearch { hosts => "192.168.10.201:9200" index => "tomcat-catalina-%{+yyyy.MM.dd}" } } ~~~ ~~~powershell [root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat-logstash-to-elastic.conf --path.data /usr/share/logstash/data3 & ~~~ ~~~powershell 验证端口是否启动 [root@logstash ~]# ss -anput | grep ":5056" tcp LISTEN 0 128 [::]:5056 [::]:* users:(("java",pid=14144,fd=106)) ~~~ ## 7.4 应用tomcat应用资源清单文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f tomcat-logs.yaml [root@k8s-master1 ~]# kubectl get deployment.apps NAME READY UP-TO-DATE AVAILABLE AGE tomcat-demo 2/2 2 2 5m26s [root@k8s-master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-demo-664584f857-k8whd 2/2 Running 0 5m33s tomcat-demo-664584f857-xncpk 2/2 Running 0 5m33s ~~~ ## 7.5 验证Pod 中tomcat及filebeat是否正常 ~~~powershell 查看tomcat产生日志 [root@k8s-master1 ~]# kubectl logs tomcat-demo-664584f857-k8whd -c tomcat 查看filebeat收集日志 [root@k8s-master1 ~]# kubectl logs tomcat-demo-664584f857-k8whd -c filebeat ~~~ ## 7.6 在kibana页面中添加索引 ![image-20220408002236127](../../img/kubernetes/kubernetes_logs_collect/image-20220408002236127.png) ![image-20220408002721389](../../img/kubernetes/kubernetes_logs_collect/image-20220408002721389.png) ![image-20220408003250140](../../img/kubernetes/kubernetes_logs_collect/image-20220408003250140.png) ![image-20220408003550821](../../img/kubernetes/kubernetes_logs_collect/image-20220408003550821.png) ![image-20220408003610470](../../img/kubernetes/kubernetes_logs_collect/image-20220408003610470.png) ![image-20220408003632515](../../img/kubernetes/kubernetes_logs_collect/image-20220408003632515.png) ![image-20220408003733569](../../img/kubernetes/kubernetes_logs_collect/image-20220408003733569.png) # kubernetes 日志收集方案 EFK # 一、EFK ## 1.1 EFK介绍 EFK为elasticsearch、fluentd、kibana的简称，本案例主要对kubernetes集群日志收集。 ## 1.2 Fluentd介绍 fluentd是一款开源的日志收集工具，其于2016年11月8日被云原生计算基金会录取，并于2019年毕业。 ![image-20220408232416685](../../img/kubernetes/kubernetes_logs_collect/image-20220408232416685.png) Fluentd优势： - 使用 JSON 进行统一日志记录 - 其尽可能地把数据结构化为JSON，让下游数据处理容易。 - 可插拔架构 - 利用插件，允许对其功能扩展 - 对计算机资源要求少 - 其使用c语言和ruby结合编写，需要少量系统资源即可运行。 - 内置可靠性 - 支持基于内存和文件的缓冲，防止节点间数据丢失 - 支持强大故障转移并可设置为高可用性 # 二、EFK部署 ## 2.1 获取EFK部署资源清单文件 ~~~powershell 把EFK部署资源清单文件复制到本地主机，本次本地主机主要指k8s master节点 # git clone https://github.com/kubernetes/kubernetes.git ~~~ ~~~powershell 进入目录并查看目录内容 # cd kubernetes/ # ls api cluster docs LICENSE Makefile.generated_files plugin SUPPORT.md build cmd go.mod LICENSES OWNERS README.md test CHANGELOG code-of-conduct.md go.sum logo OWNERS_ALIASES SECURITY_CONTACTS third_party CHANGELOG.md CONTRIBUTING.md hack Makefile pkg staging vendor ~~~ ~~~powershell 查看分支 # git branch ~~~ ~~~powershell 切换对应版本的分支 # git checkout -b v1.21.10 ~~~ ~~~powershell 进入目录并查看目录内容 # cd cluster/addons/fluentd-elasticsearch # ls create-logging-namespace.yaml es-statefulset.yaml fluentd-es-image OWNERS es-image fluentd-es-configmap.yaml kibana-deployment.yaml podsecuritypolicies es-service.yaml fluentd-es-ds.yaml kibana-service.yaml README.md ~~~ ## 2.2 安装ES ### 2.2.1 创建命名空间 ~~~powershell 应用资源清单文件创建命名空间，非必须，可使用资源清单中默认的命名空间 kube-system # kubectl create namespace logging ~~~ ### 2.2.2 部署ES ~~~powershell 部署ES，注意部署前的配置 # kubectl apply -f es-statefulset.yaml ~~~ ~~~powershell 应用前，请注释此文件中ClusterIP:None，并修改type类型为:NodePort，再执行 # kubectl apply -f es-service.yaml ~~~ ### 2.2.3 查看安装情况 ~~~powershell 查看ES部署的pod是否运行 # kubectl get pods -n logging NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 8m elasticsearch-logging-1 1/1 Running 1 5m50s ~~~ ~~~powershell 查看ES部署后的SVC，验证其访问的方法 # kubectl get svc -n logging NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch-logging NodePort 10.107.97.124 9200:31885/TCP,9300:32214/TCP 68s ~~~ ### 2.2.4 验证集群是否健康 ~~~powershell 查看ES集群是否健康，下面状态为健康。 # curl 10.107.97.124:9200/_cat/health?pretty 1640939218 08:26:58 kubernetes-logging green 2 2 6 3 0 0 0 0 - 100.0% ~~~ ## 2.3 部署fluentd ### 2.3.1 部署fluentd ~~~powershell 部署前对fluentd configmap进行配置，主要修改其连接ES的地址及对应的端口，此两项根据使用环境的不同，配置也不相同。 # vim fluentd-es-configmap.yaml 456 output.conf: |- 457 458 @id elasticsearch 459 @type elasticsearch 460 @log_level info 461 type_name _doc 462 include_tag_key true 463 host elasticsearch-logging 修改此处为es主机地址 464 port 9200 使用NodePort时，此处也需要修改对应映射端口 465 logstash_format true 466 ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f fluentd-es-configmap.yaml ~~~ ~~~powershell 修改资源清单文件 # vim fluentd-es-ds.yaml 55 selector: 56 matchLabels: 57 k8s-app: fluentd-es 58 version: v3.1.1 59 template: 60 metadata: 61 labels: 62 k8s-app: fluentd-es 63 version: v3.1.1 64 spec: 65 #securityContext: 66 # seccompProfile: 67 # type: RuntimeDefault ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f fluentd-es-ds.yaml ~~~ ### 2.3.2 查看部署状态 ~~~powershell 查看已部署的组件pod运行情况 # kubectl get pods -n logging ~~~ ~~~powershell 输出结果： NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 20m elasticsearch-logging-1 1/1 Running 1 18m fluentd-es-v3.1.1-2chjb 1/1 Running 0 64s fluentd-es-v3.1.1-5gpmd 1/1 Running 0 64s ~~~ ## 2.4 部署Kibana ### 2.4.1 部署Kibana ~~~powershell 修改资源清单文件 # vim kibana-deployment.yaml 18 spec: 以下三行注释掉 19 # securityContext: 20 # seccompProfile: 21 # type: RuntimeDefault 22 containers: 23 - name: kibana-logging 24 image: docker.elastic.co/kibana/kibana-oss:7.10.2 25 resources: 26 # need more cpu upon initialization, therefore burstable class 27 limits: 28 cpu: 1000m 29 requests: 30 cpu: 100m 31 env: 32 - name: ELASTICSEARCH_HOSTS 33 value: http://elasticsearch-logging.logging.svc.cluster.local.:9200 34 - name: SERVER_NAME 35 value: kibana-logging 以下两行注释掉 36 #- name: SERVER_BASEPATH 37 # value: /api/v1/namespaces/logging/services/kibana-logging/proxy ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f kibana-deployment.yaml ~~~ ~~~powershell 修改kibana service资源清单文件，以NodePort类型暴露服务，供K8S集群外用户访问 # vim kibana-service.yaml spec: ports: - port: 5601 protocol: TCP targetPort: ui selector: k8s-app: kibana-logging type: NodePort 添加此行内容 ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f kibana-service.yaml ~~~ ### 2.4.2 查看Kibana部署状态 ~~~powershell 查看已部署组件pod运行状态 # kubectl get pods -n logging NAME READY STATUS RESTARTS AGE elasticsearch-logging-0 1/1 Running 0 25m elasticsearch-logging-1 1/1 Running 1 22m fluentd-es-v3.1.1-2chjb 1/1 Running 0 5m45s fluentd-es-v3.1.1-5gpmd 1/1 Running 0 5m45s kibana-logging-c46f6b9c5-g9fsl 1/1 Running 0 11s ~~~ ~~~powershell 获取kibana对外提供的主机地址及对应的端口 # kubectl get svc -n logging NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch-logging NodePort 10.107.97.124 9200:31885/TCP,9300:32214/TCP 15m kibana-logging NodePort 10.99.171.38 5601:31739/TCP 7s ~~~ ~~~powershell 在K8S集群任意主机查看是否打开kibana对外的端口（服务类型为NodePort） # ss -anput | grep "31739" tcp LISTEN 0 4096 *:31739 *:* users:(("kube-proxy",pid=4569,fd=23)) ~~~ > 通过浏览器访问kibana web界面。 ![image-20211231164228950](../../img/kubernetes/kubernetes_logs_collect/image-20211231164228950.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_master.md ================================================ # kubeadm部署单Master节点kubernetes集群 1.21 # 一、kubernetes 1.21发布 ![image-20220119160108054](../../img/kubernetes/kubernetes_master/image-20220119160108054.png) ## 1.1 介绍 2021年04月，Kubernetes 1.21正式与大家见面，这是我们 2021 年的第一个版本！这个版本包含 51 个增强功能：13 个增强功能升级为稳定版，16 个增强功能升级为 beta 版，20 个增强功能进入 alpha 版，还有 2 个功能已经弃用。 ## 1.2 主要变化 - CronJobs 毕业到稳定！ 自 Kubernetes 1.8 以来，CronJobs一直是一个测试版功能！在 1.21 中，我们终于看到这个广泛使用的 API 毕业到稳定。 CronJobs 用于执行定期计划的操作，如备份、报告生成等。每个任务都应该被配置为无限期地重复出现（例如：一天/一周/一个月）；你可以在该间隔内定义作业应该启动的时间点。 - 不可变的 Secrets 和 ConfigMaps Immutable Secrets和ConfigMaps为这些资源类型添加了一个新字段，如果设置了该字段，将拒绝对这些对象的更改。默认情况下，Secrets 和 ConfigMaps 是可变的，这对能够使用更改的 pod 是有益的。如果将错误的配置推送给使用它们的 pod，可变的 Secrets 和 ConfigMaps 也会导致问题。 通过将 Secrets 和 ConfigMaps 标记为不可变的，可以确保应用程序配置不会改变。如果你希望进行更改，则需要创建一个新的、唯一命名的 Secret 或 ConfigMap，并部署一个新的 pod 来消耗该资源。不可变资源也有伸缩性优势，因为控制器不需要轮询 API 服务器来观察变化。 这个特性在 Kubernetes 1.21 中已经毕业到稳定。 - IPv4/IPv6 双栈支持 IP 地址是一种可消耗的资源，集群操作人员和管理员需要确保它不会耗尽。特别是，公共 IPv4 地址现在非常稀少。双栈支持使原生 IPv6 路由到 pod 和服务，同时仍然允许你的集群在需要的地方使用 IPv4。双堆栈集群网络还改善了工作负载的可能伸缩限制。 Kubernetes 的双栈支持意味着 pod、服务和节点可以获得 IPv4 地址和 IPv6 地址。在 Kubernetes 1.21 中，双栈网络已经从 alpha 升级到 beta，并且已经默认启用了。 - 优雅的节点关闭 在这个版本中，优雅的节点关闭也升级到测试版（现在将提供给更大的用户群）！这是一个非常有益的特性，它允许 kubelet 知道节点关闭，并优雅地终止调度到该节点的 pod。 目前，当节点关闭时，pod 不会遵循预期的终止生命周期，也不会正常关闭。这可能会在许多不同的工作负载下带来问题。接下来，kubelet 将能够通过 systemd 检测到即将发生的系统关闭，然后通知正在运行的 pod，以便它们能够尽可能优雅地终止。 - PersistentVolume 健康监测器 持久卷（Persistent Volumes，PV）通常用于应用程序中获取本地的、基于文件的存储。它们可以以许多不同的方式使用，并帮助用户迁移应用程序，而不需要重新编写存储后端。 Kubernetes 1.21 有一个新的 alpha 特性，允许对 PV 进行监视，以了解卷的运行状况，并在卷变得不健康时相应地进行标记。工作负载将能够对运行状况状态作出反应，以保护数据不被从不健康的卷上写入或读取。 - 减少 Kubernetes 的构建维护 以前，Kubernetes 维护了多个构建系统。这常常成为新贡献者和当前贡献者的摩擦和复杂性的来源。 在上一个发布周期中，为了简化构建过程和标准化原生的 Golang 构建工具，我们投入了大量的工作。这应该赋予更广泛的社区维护能力，并降低新贡献者进入的门槛。 ## 1.3 重大变化 - 弃用 PodSecurityPolicy 在 Kubernetes 1.21 中，PodSecurityPolicy 已被弃用。与 Kubernetes 所有已弃用的特性一样，PodSecurityPolicy 将在更多版本中继续可用并提供完整的功能。先前处于测试阶段的 PodSecurityPolicy 计划在 Kubernetes 1.25 中删除。 接下来是什么？我们正在开发一种新的内置机制来帮助限制 Pod 权限，暂定名为“PSP 替换策略”。我们的计划是让这个新机制覆盖关键的 PodSecurityPolicy 用例，并极大地改善使用体验和可维护性。 - 弃用 TopologyKeys 服务字段 topologyKeys 现在已弃用；所有使用该字段的组件特性以前都是 alpha 特性，现在也已弃用。我们用一种实现感知拓扑路由的方法替换了 topologyKeys，这种方法称为感知拓扑提示。支持拓扑的提示是 Kubernetes 1.21 中的一个 alpha 特性。你可以在拓扑感知提示中阅读关于替换特性的更多细节；相关的KEP解释了我们替换的背景。 # 二、kubernetes 1.21.0 部署工具介绍 ## What is Kubeadm ? `Kubeadm is a tool built to provide best-practice "fast paths" for creating Kubernetes clusters. It performs the actions necessary to get a minimum viable, secure cluster up and running in a user friendly way. Kubeadm's scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools.` Kubeadm是为创建Kubernetes集群提供最佳实践并能够“快速路径”构建kubernetes集群的工具。它能够帮助我们执行必要的操作，以获得最小可行的、安全的集群，并以用户友好的方式运行。 ## Common Kubeadm cmdlets 1. **kubeadm init** to bootstrap the initial Kubernetes control-plane node. `初始化` 2. **kubeadm join** to bootstrap a Kubernetes worker node or an additional control plane node, and join it to the cluster. `添加工作节点到kubernetes集群` 3. **kubeadm upgrade** to upgrade a Kubernetes cluster to a newer version. ` 更新kubernetes版本` 4. **kubeadm reset** to revert any changes made to this host by kubeadm init or kubeadm join. ` 重置kubernetes集群` # 三、kubernetes 1.21.0 部署环境准备 ## 3.1 主机操作系统说明 | 序号 | 操作系统及版本 | 备注 | | :--: | :------------: | :--: | | 1 | CentOS7u9 | | ## 3.2 主机硬件配置说明 | 需求 | CPU | 内存 | 硬盘 | 角色 | 主机名 | | ---- | ---- | ---- | ----- | ------------ | -------- | | 值 | 4C | 8G | 100GB | master | master01 | | 值 | 4C | 8G | 100GB | worker(node) | worker01 | | 值 | 4C | 8G | 100GB | worker(node) | worker02 | ## 3.3 主机配置 ### 3.3.1 主机名配置 由于本次使用3台主机完成kubernetes集群部署，其中1台为master节点,名称为master01;其中2台为worker节点，名称分别为：worker01及worker02 ~~~powershell master节点,名称为master1 # hostnamectl set-hostname master01 ~~~ ~~~powershell worker1节点,名称为worker1 # hostnamectl set-hostname worker01 ~~~ ~~~powershell worker2节点,名称为worker2 # hostnamectl set-hostname worker02 ~~~ ### 3.3.2 主机IP地址配置 ~~~powershell master节点IP地址为：192.168.10.11/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.11" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell worker1节点IP地址为：192.168.10.12/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.12" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell worker2节点IP地址为：192.168.10.13/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.13" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ### 3.3.3 主机名与IP地址解析 > 所有集群主机均需要进行配置。 ~~~powershell # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.11 master01 192.168.10.12 worker01 192.168.10.13 worker02 ~~~ ### 3.3.4 防火墙配置 > 所有主机均需要操作。 ~~~powershell 关闭现有防火墙firewalld # systemctl disable firewalld # systemctl stop firewalld # firewall-cmd --state not running ~~~ ### 3.3.5 SELINUX配置 > 所有主机均需要操作。修改SELinux配置需要重启操作系统。 ~~~powershell # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ### 3.3.6 时间同步配置 >所有主机均需要操作。最小化安装系统需要安装ntpdate软件。 ~~~powershell # crontab -l 0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com ~~~ ### 3.3.7 升级操作系统内核 > 所有主机均需要操作。 ~~~powershell 导入elrepo gpg key # rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell 安装elrepo YUM源仓库 # yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell 安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 # yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell 设置grub2默认引导为0 # grub2-set-default 0 ~~~ ~~~powershell 重新生成grub2引导文件 # grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ~~~powershell 更新后，需要重启，使用升级的内核生效。 # reboot ~~~ ~~~powershell 重启后，需要验证内核是否为更新对应的版本 # uname -r ~~~ ### 3.3.8 配置内核转发及网桥过滤 >所有主机均需要操作。 ~~~powershell 添加网桥过滤及内核转发配置文件 # cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ ~~~powershell 加载br_netfilter模块 # modprobe br_netfilter ~~~ ~~~powershell 查看是否加载 # lsmod | grep br_netfilter br_netfilter 22256 0 bridge 151336 1 br_netfilter ~~~ ~~~powershell 加载网桥过滤及内核转发配置文件 # sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ ### 3.3.9 安装ipset及ipvsadm > 所有主机均需要操作。主要用于实现service转发。 ~~~powershell 安装ipset及ipvsadm # yum -y install ipset ipvsadm ~~~ ~~~powershell 配置ipvsadm模块加载方式 添加需要加载的模块 # cat > /etc/sysconfig/modules/ipvs.modules < 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a ~~~powershell 永远关闭swap分区，需要重启操作系统 # cat /etc/fstab ...... # /dev/mapper/centos-swap swap swap defaults 0 0 在上一行中行首添加# ~~~ ## 3.4 Docker准备 > 所有集群主机均需操作。 ### 3.4.1 获取YUM源 > 使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ### 3.4.2 查看可安装版本 ~~~powershell # yum list docker-ce.x86_64 --showduplicates | sort -r ~~~ ### 3.4.3 安装指定版本并设置启动及开机自启动 ~~~powershell # yum -y install --setopt=obsoletes=0 docker-ce-20.10.9-3.el7 ~~~ ~~~powershell # systemctl enable docker ; systemctl start docker ~~~ ### 3.4.4 修改cgroup方式 ~~~powershell 在/etc/docker/daemon.json添加如下内容 # cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } ~~~ ### 3.4.5 重启docker ~~~powershell # systemctl restart docker ~~~ # 四、kubernetes 1.21.0 集群部署 ## 4.1 集群软件及版本说明 | | kubeadm | kubelet | kubectl | | -------- | ---------------------- | --------------------------------------------- | ---------------------- | | 版本 | 1.21.0 | 1.21.0 | 1.21.0 | | 安装位置 | 集群所有主机 | 集群所有主机 | 集群所有主机 | | 作用 | 初始化集群、管理集群等 | 用于接收api-server指令，对pod生命周期进行管理 | 集群应用命令行管理工具 | ## 4.2 kubernetes YUM源准备 ### 4.2.1 谷歌YUM源 ~~~powershell [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg ~~~ ### 4.2.2 阿里云YUM源 ~~~powershell [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ~~~ ## 4.3 集群软件安装 ~~~powershell 查看指定版本 # yum list kubeadm.x86_64 --showduplicates | sort -r # yum list kubelet.x86_64 --showduplicates | sort -r # yum list kubectl.x86_64 --showduplicates | sort -r ~~~ ~~~powershell 安装指定版本 # yum -y install --setopt=obsoletes=0 kubeadm-1.21.0-0 kubelet-1.21.0-0 kubectl-1.21.0-0 ~~~ ## 指定阿里云yum源并安装 ```shell cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF setenforce 0 sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable kubelet && systemctl start kubelet ## 另外，你也可以指定版本安装 ## yum install -y kubelet-1.25.6 kubectl-kubelet-1.25.6 kubeadm-kubelet-1.25.6 ``` ## 4.4 配置kubelet >为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ~~~powershell # vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ~~~ ~~~powershell 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 # systemctl enable kubelet ~~~ ## 4.5 集群镜像准备 > 可使用VPN实现下载。 ~~~powershell # kubeadm config images list --kubernetes-version=v1.21.0 k8s.gcr.io/kube-apiserver:v1.21.0 k8s.gcr.io/kube-controller-manager:v1.21.0 k8s.gcr.io/kube-scheduler:v1.21.0 k8s.gcr.io/kube-proxy:v1.21.0 k8s.gcr.io/pause:3.4.1 k8s.gcr.io/etcd:3.4.13-0 k8s.gcr.io/coredns/coredns:v1.8.0 ~~~ ~~~powershell # cat image_download.sh #!/bin/bash images_list=' k8s.gcr.io/kube-apiserver:v1.21.0 k8s.gcr.io/kube-controller-manager:v1.21.0 k8s.gcr.io/kube-scheduler:v1.21.0 k8s.gcr.io/kube-proxy:v1.21.0 k8s.gcr.io/pause:3.4.1 k8s.gcr.io/etcd:3.4.13-0 k8s.gcr.io/coredns/coredns:v1.8.0' for i in $images_list do docker pull $i done docker save -o k8s-1-21-0.tar $images_list ~~~ ## 4.6 集群初始化 ~~~powershell [root@master01 ~]# kubeadm init --kubernetes-version=v1.21.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.10.11 --image-repository registry.aliyuncs.com/google_containers ~~~ ~~~powershell 输出内容，一定保留，便于后继操作使用。 [init] Using Kubernetes version: v1.21.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master01] and IPs [10.96.0.1 192.168.10.11] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost master01] and IPs [192.168.10.11 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost master01] and IPs [192.168.10.11 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. [apiclient] All control plane components are healthy after 57.503834 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master01 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: 9kz5id.pp5rhvzahj51lb5q [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.11:6443 --token 9kz5id.pp5rhvzahj51lb5q \ --discovery-token-ca-cert-hash sha256:86f9c4471b6ef08090ecffadc798040fe5d8ef5975afe527e65d2f0aedf66493 ~~~ ## 4.7 集群应用客户端管理集群文件准备 ~~~powershell [root@master1 ~]# mkdir -p $HOME/.kube [root@master1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@master1 ~]# ls /root/.kube/ config ~~~ ~~~powershell # export KUBECONFIG=/etc/kubernetes/admin.conf ~~~ ## 4.8 集群网络准备 > 使用calico部署集群网络 > > 安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico ### 4.8.1 calico安装 ![image-20220119141547207](../../img/kubernetes/kubernetes_master/image-20220119141547207.png) ![image-20220119141645676](../../img/kubernetes/kubernetes_master/image-20220119141645676.png) ![image-20220119141734347](../../img/kubernetes/kubernetes_master/image-20220119141734347.png) ![image-20220119141830625](../../img/kubernetes/kubernetes_master/image-20220119141830625.png) ~~~powershell 下载operator资源清单文件 # wget https://docs.projectcalico.org/manifests/tigera-operator.yaml ~~~ ~~~powershell 应用资源清单文件，创建operator # kubectl apply -f tigera-operator.yaml ~~~ ~~~powershell 通过自定义资源方式安装 # wget https://docs.projectcalico.org/manifests/custom-resources.yaml ~~~ ~~~powershell 修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 # vim custom-resources.yaml ...... 11 ipPools: 12 - blockSize: 26 13 cidr: 10.244.0.0/16 14 encapsulation: VXLANCrossSubnet ...... ~~~ ~~~powershell 应用资源清单文件 # kubectl apply -f custom-resources.yaml ~~~ ~~~powershell 监视calico-sysem命名空间中pod运行情况 # watch kubectl get pods -n calico-system ~~~ >Wait until each pod has the `STATUS` of `Running`. ~~~powershell 删除 master 上的 taint # kubectl taint nodes --all node-role.kubernetes.io/master- ~~~ ~~~powershell 已经全部运行 # kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-666bb9949-dzp68 1/1 Running 0 11m calico-node-jhcf4 1/1 Running 4 11m calico-typha-68b96d8d9c-7qfq7 1/1 Running 2 11m ~~~ ~~~powershell 查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 # kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-558bd4d5db-4jbdv 1/1 Running 0 113m coredns-558bd4d5db-pw5x5 1/1 Running 0 113m etcd-master01 1/1 Running 0 113m kube-apiserver-master01 1/1 Running 0 113m kube-controller-manager-master01 1/1 Running 4 113m kube-proxy-kbx4z 1/1 Running 0 113m kube-scheduler-master01 1/1 Running 3 113m ~~~ ### 4.8.2 calico客户端安装 ![image-20220119144207789](../../img/kubernetes/kubernetes_master/image-20220119144207789.png) ![image-20220119144446449](../../img/kubernetes/kubernetes_master/image-20220119144446449.png) ~~~powershell 下载二进制文件 # curl -L https://github.com/projectcalico/calico/releases/download/v3.21.4/calicoctl-linux-amd64 -o calicoctl ~~~ ~~~powershell 安装calicoctl # mv calicoctl /usr/bin/ 为calicoctl添加可执行权限 # chmod +x /usr/bin/calicoctl 查看添加权限后文件 # ls /usr/bin/calicoctl /usr/bin/calicoctl 查看calicoctl版本 # calicoctl version Client Version: v3.21.4 Git commit: 220d04c94 Cluster Version: v3.21.4 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm ~~~ ~~~powershell 通过~/.kube/config连接kubernetes集群，查看已运行节点 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME master01 ~~~ ## 4.9 集群工作节点添加 > 因容器镜像下载较慢，可能会导致报错，主要错误为没有准备好cni（集群网络插件），如有网络，请耐心等待即可。 ~~~powershell [root@worker01 ]# kubeadm join 192.168.10.11:6443 --token 9kz5id.pp5rhvzahj51lb5q \ --discovery-token-ca-cert-hash sha256:86f9c4471b6ef08090ecffadc798040fe5d8ef5975afe527e65d2f0aedf66493 ~~~ ~~~powershell [root@worker02 ~]# kubeadm join 192.168.10.11:6443 --token 9kz5id.pp5rhvzahj51lb5q \ --discovery-token-ca-cert-hash sha256:86f9c4471b6ef08090ecffadc798040fe5d8ef5975afe527e65d2f0aedf66493 ~~~ ~~~powershell 在master节点上操作，查看网络节点是否添加 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME master01 worker01 worker02 ~~~ ## 4.10 验证集群可用性 ~~~powershell 查看所有的节点 [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready control-plane,master 169m v1.21.0 worker01 Ready 28m v1.21.0 worker02 Ready 28m v1.21.0 ~~~ ~~~powershell 查看集群健康情况,理想状态 [root@master01 ~]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} ~~~ ~~~powershell 真实情况 # kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused etcd-0 Healthy {"health":"true"} ~~~ ~~~powershell 查看kubernetes集群pod运行情况 [root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-558bd4d5db-4jbdv 1/1 Running 1 169m coredns-558bd4d5db-pw5x5 1/1 Running 1 169m etcd-master01 1/1 Running 1 170m kube-apiserver-master01 1/1 Running 1 170m kube-controller-manager-master01 1/1 Running 14 170m kube-proxy-kbx4z 1/1 Running 1 169m kube-proxy-rgtr8 1/1 Running 0 29m kube-proxy-sq9xv 1/1 Running 0 29m kube-scheduler-master01 1/1 Running 11 170m ~~~ ~~~powershell 再次查看calico-system命名空间中pod运行情况。 [root@master01 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-666bb9949-dzp68 1/1 Running 3 70m calico-node-jhcf4 1/1 Running 15 70m calico-node-jxq9p 1/1 Running 0 30m calico-node-kf78q 1/1 Running 0 30m calico-typha-68b96d8d9c-7qfq7 1/1 Running 13 70m calico-typha-68b96d8d9c-wz2zj 1/1 Running 0 20m ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_nginx_ingress_controller.md ================================================ # Kubernetes集群 服务暴露 Nginx Ingress Controller # 一、ingress控制器 ## 1.1 ingress控制器作用 ingress controller可以为kubernetes 集群外用户访问Kubernetes集群内部pod提供代理服务。 - 提供全局访问代理 - 访问流程 - 用户-->ingress controller-->service-->pod ![image-20220412223010688](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220412223010688.png) ![image-20220412222951979](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220412222951979.png) ## 1.2 ingress控制器种类 ### 1.2.1 Kubernetes Ingress Controller - 参考链接：http://github.com/nginxinc/kubernetes-ingress - 实现：Go/Lua（nginx 是用 C 写的） - 许可证：Apache 2.0 - Kubernetes 的“官方”控制器（之所以称为官方，是想把它区别于 NGINX 公司的控制器）。这是社区开发的控制器，它基于 nginx Web 服务器，并补充了一组用于实现额外功能的 Lua 插件。 - 由于 NGINX 十分流行，再加上把它用作控制器时所需的修改较少，**它对于 K8s 普通工程师来说，可能是最简单和最直接的选择**。 ### 1.2.2 NGINX Ingress Controller - 参考链接：http://github.com/kubernetes/ingress-nginx - 实现：Go - 许可证：Apache 2.0 - 这是 NGINX 公司开发的官方产品，它也有一个基于 NGINX Plus 的商业版。NGINX 的控制器具有很高的稳定性、持续的向后兼容性，且没有任何第三方模块。 - 由于消除了 Lua 代码，和官方控制器相比，它保证了较高的速度，但也因此受到较大限制。相较之下，它的付费版本有更广泛的附加功能，如实时指标、JWT 验证、主动健康检查等。 - NGINX Ingress 重要的优势是对 TCP/UDP 流量的全面支持，最主要缺点是缺乏流量分配功能。 ### 1.2.3 Kong Ingress - 参考链接：http://github.com/Kong/kubernetes-ingress-controller - 实现：Go - 许可证：Apache 2.0 - Kong Ingress 由 Kong Inc 开发，有两个版本：商业版和免费版。它基于 NGINX 构建，并增加了扩展其功能的 Lua 模块。 - 最初，Kong Ingress 主要用作 API 网关，用于 API 请求的处理和路由。现在，它已经成为成熟的 Ingress 控制器，**主要优点是拥有大量易于安装和配置的附加模块、插件（包括第三方插件）**。它开启了控制器具备大量附加功能的先河，其内置函数也提供了许多可能性。Kong Ingress 配置是用 CRD 执行的。 - Kong Ingress 的一个重要特性是它只能在一个环境中运行（而不支持跨命名空间）。这是一个颇有争议的话题：有些人认为这是一个缺点，因为必须为每个环境生成实例；而另一些人认为这是一个特殊特性，因为它是更高级别的隔离，控制器故障的影响仅限于其所在的环境。 ### 1.2.4 Traefik - 参考链接：http://github.com/containous/traefik - 实现：Go - 许可证：MIT - 最初，这个代理是为微服务请求及其动态环境的路由而创建的，因此具有许多有用的功能：**连续更新配置（不重新启动）、支持多种负载均衡算法、Web UI、指标导出、对各种服务的支持协议、REST API、Canary 版本**等。 - 支持开箱即用的 Let’s Encrypt 是它的另一个不错的功能，但它的主要缺点也很明显，就是为了控制器的高可用性，你必须安装并连接其 Key-value store。 - 在 2019 年 9 月发布的 Traefik v2.0 中，虽然它增加许多不错的新功能，如带有 SNI 的 TCP/SSL、金丝雀部署、流量镜像/shadowing 和经过改进的 Web UI，但一些功能（如 WAF 支持）还在策划讨论中。 - 与新版本同期推出的还有一个名叫 Mesh 的服务网格，它建在 Traefik 之上，对kubernetes内部服务访问做到受控及被监控。 ### 1.2.5 HAProxy Ingress - 参考链接：http://github.com/jcmoraisjr/haproxy-ingress - 实现：Go（HAProxy 是用 C 写的） - 许可证：Apache 2.0 - HAProxy 是众所周知的代理服务器和负载均衡器。作为 Kubernetes 集群的一部分，它提供了“软”配置更新（无流量损失）、基于 DNS 的服务发现和通过 API 进行动态配置。 HAProxy 还支持完全自定义配置文件模板（通过替换 ConfigMap）以及在其中使用 Spring Boot 函数。 - 通常，工程师会把重点放在已消耗资源的高速、优化和效率上。而 HAProxy 的优点之一正是支持大量负载均衡算法。值得一提的是，在2020年 6 月发布的 v2.0 中，HAProxy 增加了许多新功能，其即将推出的 v2.1 有望带来更多新功能（包括 OpenTracing 支持）。 ### 1.2.6 Voyager - 参考链接：http://github.com/appscode/voyager - 实现：Go - 许可证：Apache 2.0 - Voyager 基于 HAProxy，并作为一个通用的解决方案提供给大量供应商。它最具代表性的功能包括 L7 和 L4 上的流量负载均衡，其中，**TCP L4 流量负载均衡称得上是该解决方案最关键的功能之一**。 - 在2020年早些时候，尽管 Voyager 在 v9.0.0 中推出了对 HTTP/2 和 gRPC 协议的全面支持，但总的来看，对证书管理（Let’s Encrypt 证书）的支持仍是 Voyager 集成的最突出的新功能。 ### 1.2.7 Contour - 参考链接：http://github.com/heptio/contour - 实现：Go - 许可证：Apache 2.0 - Contour 和 Envoy 由同一个作者开发，它基于 Envoy。**它最特别的功能是可以通过 CRD（IngressRoute）管理 Ingress 资源**，对于多团队需要同时使用一个集群的组织来说，这有助于保护相邻环境中的流量，使它们免受 Ingress 资源更改的影响。 - 它还提供了一组扩展的负载均衡算法（镜像、自动重复、限制请求率等），以及详细的流量和故障监控。对某些工程师而言，它不支持粘滞会话可能是一个严重缺陷。 ### 1.2.8 Istio Ingress - 参考链接：http://istio.io/docs/tasks/traffic-management/ingress - 实现：Go - 许可证：Apache 2.0 - Istio 是 IBM、Google 和 Lyft 的联合开发项目，它是一个全面的服务网格解决方案——**不仅可以管理所有传入的外部流量（作为 Ingress 控制器），还可以控制集群内部的所有流量**。 - Istio 将 Envoy 用作每种服务的辅助代理。从本质上讲，它是一个可以执行几乎所有操作的大型处理器，其中心思想是最大程度的控制、可扩展性、安全性和透明性。 - 通过 Istio Ingress，你可以对流量路由、服务之间的访问授权、均衡、监控、金丝雀发布等进行优化。 ### 1.2.9 Ambassador - 参考链接：http://github.com/datawire/ambassador - 实现：Python - 许可证：Apache 2.0 - Ambassador 也是一个基于 Envoy 的解决方案，它有免费版和商业版两个版本。 - Ambassador 被称为“Kubernetes 原生 API 微服务网关”，它与 K8s 原语紧密集成，拥有你所期望的从 Ingress controller 获得的功能包，它还可以与各种服务网格解决方案，如 Linkerd、Istio 等一起使用。 - 顺便提一下，Ambassador 博客日前发布了一份基准测试结果，比较了 Envoy、HAProxy 和 NGINX 的基础性能。 ### 1.2.10 Gloo - 参考链接：http://github.com/solo-io/gloo - 实现：Go - 许可证：Apache 2.0 - Gloo 是在 Envoy 之上构建的新软件（于 2018 年 3 月发布），由于它的作者坚持认为“网关应该从功能而不是服务中构建 API”，它也被称为“功能网关”。其“功能级路由”的意思是它可以为后端实现是微服务、无服务器功能和遗留应用的混合应用路由流量。 - 由于拥有可插拔的体系结构，Gloo 提供了工程师期望的大部分功能，但是其中一些功能仅在其商业版本（Gloo Enterprise）中可用。 ### 1.2.11 Skipper - 参考链接：http://github.com/zalando/skipper - 实现：Go - 许可证：Apache 2.0 - Skipper 是 HTTP 路由器和反向代理，因此不支持各种协议。从技术上讲，它使用 Endpoints API（而不是 Kubernetes Services）将流量路由到 Pod。它的优点在于其丰富的过滤器集所提供的高级 HTTP 路由功能，工程师可以借此创建、更新和删除所有 HTTP 数据。 - Skipper 的路由规则可以在不停机的情况下更新。正如它的作者所述，Skipper 可以很好地与其他解决方案一起使用，比如 AWS ELB。 # 二、nginx ingress controller ## 2.1 nginx ingress controller位置 ![image-20220412213345991](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220412213345991.png) - 参考链接： ## 2.2 nginx ingress controller部署 - 项目地址：https://github.com/kubernetes/ingress-nginx ### 2.2.1 下载并修改配置文件 ![image-20220412223933840](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220412223933840.png) ![image-20220412224751513](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220412224751513.png) ![image-20220412224823931](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220412224823931.png) ~~~powershell [root@k8s-master1 ~]# curl -k https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml -o deploy.yaml ~~~ ~~~powershell [root@k8s-master1 ~]# ls deploy.yaml ~~~ ~~~powershell [root@k8s-master1 ~]# vim deploy.yaml ...... 323 spec: 324 ports: 325 - appProtocol: http 326 name: http 327 port: 80 328 protocol: TCP 329 targetPort: http 330 - appProtocol: https 331 name: https 332 port: 443 333 protocol: TCP 334 targetPort: https 335 selector: 336 app.kubernetes.io/component: controller 337 app.kubernetes.io/instance: ingress-nginx 338 app.kubernetes.io/name: ingress-nginx 339 type: NodePort 把339行修改为LoadBalancer 323 spec: 324 ports: 325 - appProtocol: http 326 name: http 327 port: 80 328 protocol: TCP 329 targetPort: http 330 - appProtocol: https 331 name: https 332 port: 443 333 protocol: TCP 334 targetPort: https 335 selector: 336 app.kubernetes.io/component: controller 337 app.kubernetes.io/instance: ingress-nginx 338 app.kubernetes.io/name: ingress-nginx 339 type: LoadBalancer ~~~ ### 2.2.2 应用资源清单文件 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f deploy.yaml namespace/ingress-nginx created serviceaccount/ingress-nginx created serviceaccount/ingress-nginx-admission created role.rbac.authorization.k8s.io/ingress-nginx created role.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrole.rbac.authorization.k8s.io/ingress-nginx created clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created rolebinding.rbac.authorization.k8s.io/ingress-nginx created rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created configmap/ingress-nginx-controller created service/ingress-nginx-controller created service/ingress-nginx-controller-admission created deployment.apps/ingress-nginx-controller created job.batch/ingress-nginx-admission-create created job.batch/ingress-nginx-admission-patch created ingressclass.networking.k8s.io/nginx created validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created ~~~ ### 2.2.3 验证部署结果 > 注意镜像较大，可提前下载至集群node节点 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-xdpgp 0/1 Completed 0 91s ingress-nginx-admission-patch-lgnxs 0/1 Completed 1 91s ingress-nginx-controller-9596689c-j9p9l 1/1 Running 0 91s ~~~ ~~~powershell [root@k8s-master1 ng]# kubectl get all -n ingress-nginx NAME READY STATUS RESTARTS AGE pod/ingress-nginx-admission-create-xdpgp 0/1 Completed 0 3m24s pod/ingress-nginx-admission-patch-lgnxs 0/1 Completed 1 3m24s pod/ingress-nginx-controller-9596689c-j9p9l 1/1 Running 0 3m24s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller LoadBalancer 10.96.183.188 192.168.10.91 80:32369/TCP,443:31775/TCP 3m25s service/ingress-nginx-controller-admission ClusterIP 10.96.212.14 443/TCP 3m25s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 1/1 1 1 3m24s NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-9596689c 1 1 1 3m24s NAME COMPLETIONS DURATION AGE job.batch/ingress-nginx-admission-create 1/1 2s 3m24s job.batch/ingress-nginx-admission-patch 1/1 3s 3m24s ~~~ ## 2.3 ingress对象应用案例 ### 2.3 1 ingress-http案例 > 基于名称的负载均衡 #### 2.3.1.1 创建deployment控制器类型应用 ```powershell [root@k8s-master1 ~]# vim nginx.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: ingress-nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ``` 应用YAML ~~~powershell [root@k8s-master1 ~]# kubectl apply -f nginx.yml deployment.extensions/nginx created ~~~ 验证pod ~~~powershell [root@k8s-master1 ~]# kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE nginx-79654d7b8-nhxpm 1/1 Running 0 12s nginx-79654d7b8-tp8wg 1/1 Running 0 13s nginx-ingress-controller-77db54fc46-kwwkt 1/1 Running 0 11m ~~~ #### 2.3.1.2 创建service ```powershell [root@k8s-master1 ~]# vim nginx-service.yml apiVersion: v1 kind: Service metadata: name: nginx-service namespace: ingress-nginx labels: app: nginx spec: ports: - port: 80 targetPort: 80 selector: app: nginx ``` 应用YAML ~~~powershell [root@k8s-master1 ~]# kubectl apply -f nginx-service.yml service/nginx-service created ~~~ 验证service ~~~powershell [root@k8s-master1 ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP 10.2.115.144 80/TCP 5s ~~~ #### 2.3.1.3 创建ingress对象 ```powershell [root@k8s-master1 ~]# vim ingress-nginx.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-nginx #自定义ingress名称 namespace: ingress-nginx annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: www.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: nginx-service # 对应上面创建的service名称 port: number: 80 ``` 应用YAML ~~~powershell [root@k8s-master1 ~]# kubectl apply -f ingress-nginx.yaml ingress.extensions/ingress-nginx created ~~~ 验证ingress ~~~powershell [root@k8s-master1 ~]# kubectl get ingress -n ingress-nginx NAME CLASS HOSTS ADDRESS PORTS AGE ingress-nginx www.kubemsb.com 192.168.10.12 80 113s ~~~ 描述查看ingress信息 ~~~powershell [root@k8s-master1 ~]# kubectl describe ingress ingress-nginx -n ingress-nginx Name: ingress-nginx Namespace: ingress-nginx Address: 192.168.10.12 Default backend: default-http-backend:80 () Rules: Host Path Backends ---- ---- -------- www.kubemsb.com / nginx-service:80 (10.244.159.160:80,10.244.194.110:80) Annotations: kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 2m (x2 over 2m56s) nginx-ingress-controller Scheduled for sync ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide -n ingress-nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-646d5c7b67-mpw9r 1/1 Running 0 4m15s 10.244.194.110 k8s-worker1 nginx-646d5c7b67-v99gz 1/1 Running 0 4m15s 10.244.159.160 k8s-master1 可以看到两个pod的IP正好对应ingress域名对应的IP ~~~ 确认`nginx-ingress-controller`的podIP为`192.168.10.91` #### 2.3.1.4 模拟客户端访问 1, 确认`nginx-ingress-controller`的podIP, 下面命令查询的结果为`192.168.10.91` ~~~powershell [root@k8s-master1 ~]# kubectl get svc -n ingress-nginx |grep ingress ingress-nginx-controller LoadBalancer 10.96.183.188 192.168.10.91 80:32369/TCP,443:31775/TCP 11m ingress-nginx-controller-admission ClusterIP 10.96.212.14 443/TCP 11m ~~~ 2, 在集群之外任一主机中添加上述域名与IP地址解析(模拟公网DNS) ```powershell [root@otherhost ~]# vim /etc/hosts 192.168.10.91 www.kubemsb.com ``` 3, 准备pod内容器运行的web主页 ```powershell [root@k8s-master1 ~]# kubectl get pods -n ingress-nginx nginx-646d5c7b67-mpw9r 1/1 Running 0 8m34s nginx-646d5c7b67-v99gz 1/1 Running 0 8m34s [root@k8s-master1 ~]# kubectl exec -it nginx-646d5c7b67-mpw9r -n ingress-nginx -- /bin/sh / # echo "ingress web1" > /usr/share/nginx/html/index.html / # exit [root@k8s-master1 ~]# kubectl exec -it nginx-646d5c7b67-v99gz -n ingress-nginx -- /bin/sh / # echo "ingress web2" > /usr/share/nginx/html/index.html / # exit ``` 4, 访问及结果展示 ```powershell [root@otherhost ~]# curl www.kubemsb.com ingress web1 [root@otherhost ~]# curl www.kubemsb.com ingress web2 ``` ### 2.3.2 ingress-http案例扩展 > 基于URI的负载均衡 #### 2.3.2.1 创建第一个应用 ~~~powershell [root@k8s-master1 ~]# vim nginx-uri-1.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-uri-1 namespace: ingress-nginx spec: replicas: 2 selector: matchLabels: app: nginx-uri-1 template: metadata: labels: app: nginx-uri-1 spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ~~~ ~~~powershell [root@k8s-master1 ~]# vim nginx-service-uri-1.yml apiVersion: v1 kind: Service metadata: name: nginx-service-uri-1 namespace: ingress-nginx labels: app: nginx-uri-1 spec: ports: - port: 80 targetPort: 80 selector: app: nginx-uri-1 ~~~ ~~~powershell # kubectl apply -f nginx-uri-1.yaml ~~~ ~~~powershell # kubectl apply -f nginx-service-uri-1.yaml ~~~ #### 2.3.2.2 创建第二个应用 ~~~powershell [root@k8s-master1 ~]# vim nginx-uri-2.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-uri-2 namespace: ingress-nginx spec: replicas: 2 selector: matchLabels: app: nginx-uri-2 template: metadata: labels: app: nginx-uri-2 spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ~~~ ~~~powershell [root@k8s-master1 ~]# vim nginx-service-uri-2.yml apiVersion: v1 kind: Service metadata: name: nginx-service-uri-2 namespace: ingress-nginx labels: app: nginx-uri-2 spec: ports: - port: 80 targetPort: 80 selector: app: nginx-uri-2 ~~~ ~~~powershell # kubectl apply -f nginx-uri-2.yaml ~~~ ~~~powershell # kubectl apply -f nginx-service-uri-2.yaml ~~~ ~~~powershell # kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service-uri-1 ClusterIP 10.96.171.135 80/TCP 7m24s nginx-service-uri-2 ClusterIP 10.96.234.164 80/TCP 4m11s ~~~ #### 2.3.2.3 创建ingress对象 ```powershell [root@k8s-master1 ~]# vim ingress-nginx.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-uri namespace: ingress-nginx annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: www.kubemsburi.com http: paths: - path: /svc1 pathType: Prefix backend: service: name: nginx-service-uri-1 port: number: 80 - path: /svc2 pathType: Prefix backend: service: name: nginx-service-uri-2 port: number: 80 ``` 应用YAML ~~~powershell [root@master1 ~]# kubectl apply -f ingress-nginx-uri.yaml ingress.networking.k8s.io/ingress-uri created ~~~ 验证ingress ~~~powershell [root@master1 ~]# kubectl get ingress -n ingress-nginx NAME CLASS HOSTS ADDRESS PORTS AGE ingress-uri www.kubemsburi.com 80 13s ~~~ 描述查看ingress信息 ~~~powershell [root@master1 ~]# kubectl describe ingress ingress-uri -n ingress-nginx Name: ingress-uri Namespace: ingress-nginx Address: 192.168.10.12 Default backend: default-http-backend:80 () Rules: Host Path Backends ---- ---- -------- www.kubemsburi.com /svc1 nginx-service-uri-1:80 (10.244.159.158:80,10.244.194.111:80) /svc2 nginx-service-uri-2:80 (10.244.159.159:80,10.244.194.112:80) Annotations: kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 4s (x2 over 32s) nginx-ingress-controller Scheduled for sync ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide -n ingress-nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-uri-1-7d7d75f86-dws96 1/1 Running 0 14m 10.244.159.158 k8s-master1 nginx-uri-1-7d7d75f86-s8js4 1/1 Running 0 14m 10.244.194.111 k8s-worker1 nginx-uri-2-7cdf7f89b7-8s4mg 1/1 Running 0 10m 10.244.194.112 k8s-worker1 nginx-uri-2-7cdf7f89b7-gj8x6 1/1 Running 0 10m 10.244.159.159 k8s-master1 ~~~ 确认`nginx-ingress-controller`的podIP为`192.168.10.91` #### 2.3.1.4 模拟客户端访问 1, 确认`nginx-ingress-controller`的podIP, 下面命令查询的结果为`192.168.10.91` ~~~powershell [root@k8s-master1 ~]# kubectl get svc -n ingress-nginx |grep ingress ingress-nginx-controller LoadBalancer 10.96.183.188 192.168.10.91 80:32369/TCP,443:31775/TCP 11m ingress-nginx-controller-admission ClusterIP 10.96.212.14 443/TCP 11m ~~~ 2, 在集群之外任一主机中添加上述域名与IP地址解析(模拟公网DNS) ```powershell [root@otherhost ~]# vim /etc/hosts 192.168.10.91 www.kubemsburi.com ``` 3, 准备pod内容器运行的web主页 ```powershell [root@k8s-master1 ~]# kubectl exec -it nginx-uri-1-7d7d75f86-dws96 -n ingress-nginx -- /bin/sh / # mkdir /usr/share/nginx/html/svc1 / # echo "sssvc1" > /usr/share/nginx/html/svc1/index.html / # exit [root@k8s-master1 ~]# kubectl exec -it nginx-uri-1-7d7d75f86-s8js4 -n ingress-nginx -- /bin/sh / # mkdir /usr/share/nginx/html/svc1 / # echo "sssvc1" > /usr/share/nginx/html/svc1/index.html / # exit [root@k8s-master1 ~]# kubectl exec -it nginx-uri-2-7cdf7f89b7-8s4mg -n ingress-nginx -- /bin/sh / # mkdir /usr/share/nginx/html/svc2 / # echo "sssvc2" > /usr/share/nginx/html/svc1/index.html / # exit [root@k8s-master1 ~]# kubectl exec -it nginx-uri-2-7cdf7f89b7-gj8x6 -n ingress-nginx -- /bin/sh / # mkdir /usr/share/nginx/html/svc2 / # echo "sssvc2" > /usr/share/nginx/html/svc1/index.html / # exit ``` 4, 访问及结果展示 ```powershell [root@otherhost ~]# curl www.kubemsburi.com/svc1/index.html sssvc1 [root@otherhost ~]# curl www.kubemsburi.com/svc2/index.html sssvc2 ``` ### 2.3.3 ingress-https案例 #### 2.3.3.1 创建自签证书 ~~~powershell [root@k8s-master1 ~]# mkdir ingress-https [root@k8s-master1 ~]# cd ingress-https/ [root@k8s-master1 ingress-https]# openssl genrsa -out nginx.key 2048 [root@k8s-master1 ingress-https]# openssl req -new -x509 -key nginx.key -out nginx.pem -days 365 ...... ...... Country Name (2 letter code) [XX]:CN State or Province Name (full name) []:GD Locality Name (eg, city) [Default City]:SZ Organization Name (eg, company) [Default Company Ltd]:IT Organizational Unit Name (eg, section) []:it Common Name (eg, your name or your server's hostname) []:kubemsbhost Email Address []:admin@kubemsbhost.com [root@k8s-master1 ingress-https]# ls nginx.key nginx.pem ~~~ #### 2.3.3.2 将证书创建成secret ~~~powershell [root@k8s-master1 ingress-https]# kubectl create secret tls nginx-tls-secret --cert=nginx.pem --key=nginx.key -n ingress-nginx secret/nginx-tls-secret created [root@k8s-master1 ingress-https]# kubectl get secrets -n ingress-nginx |grep nginx-tls-secret nginx-tls-secret kubernetes.io/tls 2 38s ~~~ #### 2.3.3.3 编排YAML并创建 ~~~powershell [root@k8s-master1 ingress-https]# vim ingress-https.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx2 namespace: ingress-nginx spec: replicas: 2 selector: matchLabels: app: nginx2 template: metadata: labels: app: nginx2 spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 - name: https containerPort: 443 --- apiVersion: v1 kind: Service metadata: name: nginx-service2 namespace: ingress-nginx labels: app: nginx2 spec: ports: - name: http port: 80 targetPort: 80 - name: https port: 443 targetPort: 443 selector: app: nginx2 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-nginx2 namespace: ingress-nginx annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: tls: - hosts: - www.kubemsbhost.com # 域名 secretName: nginx-tls-secret # 调用前面创建的secret rules: - host: www.kubemsbhost.com # 域名 http: paths: - pathType: Prefix path: "/" backend: service: name: nginx-service2 # 对应服务名 port: number: 80 ~~~ ~~~powershell [root@k8s-master1 ingress-https]# kubectl apply -f ingress-https.yml deployment.apps/nginx2 created service/nginx-service2 created ingress.extensions/ingress-nginx2 created ~~~ 验证 ~~~powershell [root@k8s-master1 ~]# kubectl get ingress -n ingress-nginx NAME CLASS HOSTS ADDRESS PORTS AGE ingress-nginx2 www.kubemsbhost.com 192.168.10.12 80, 443 2m14s ~~~ #### 2.3.3.4 模拟客户端访问 ~~~powershell [root@otherhost ~]# vim /etc/hosts 192.168.10.91 www.kubemsbhost.com 添加这行模拟DNS [root@otherhost ~]# firefox https://www.kubemsbhost.com & [1] 10892 ~~~ ![image-20220413013418685](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220413013418685.png) ![image-20220413013457091](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220413013457091.png) ~~~powershell 关于可信任证书的说明： 如果需要在互联网中访问kubernetes集群中的服务是可信的，建议使用互联网中申请的SSL证书。 ~~~ ### 2.3.4 ingress+nodeport服务 ~~~powershell [root@k8s-master1 ~]# vim ingress-nodeport.yml apiVersion: apps/v1 kind: Deployment metadata: name: nginx3 namespace: ingress-nginx spec: replicas: 2 selector: matchLabels: app: nginx3 template: metadata: labels: app: nginx3 spec: containers: - name: c1 image: nginx:1.15-alpine imagePullPolicy: IfNotPresent --- apiVersion: v1 kind: Service metadata: name: nginx-service3 namespace: ingress-nginx labels: app: nginx3 spec: type: NodePort # NodePort类型服务 ports: - port: 80 targetPort: 80 selector: app: nginx3 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-nginx3 namespace: ingress-nginx annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: www.kubemsb3.com # 域名 http: paths: - pathType: Prefix path: "/" backend: service: name: nginx-service3 # 对应服务名 port: number: 80 ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f ingress-nodeport.yml ~~~ ~~~powershell root@k8s-master1 ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP 10.2.115.144 80/TCP 22h nginx-service2 ClusterIP 10.2.237.70 80/TCP,443/TCP 22h nginx-service3 NodePort 10.2.75.250 80:26765/TCP 3m51s nginx-service3是nodeport类型 ~~~ ~~~powershell [root@otherhost ~]# vim /etc/hosts 192.168.10.91 www.kubemsb3.com 添加这行模拟DNS [root@otherhost ~]# curl www.kubemsb3.com ~~~ # ingress nginx controller 1.4.0 最新版本部署方法 # 一、获取ingress nginx controller ![image-20220918114949361](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220918114949361.png) ![image-20220918115048813](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220918115048813.png) ![image-20220918115146998](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220918115146998.png) ![image-20220918115213271](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220918115213271.png) ![image-20220918115253857](../../img/kubernetes/kubernetes_nginx_ingress_controller/image-20220918115253857.png) ~~~powershell # wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/cloud/deploy.yaml ~~~ # 二、部署ingress nginx controller ~~~powershell # vim deploy.yaml ...... 361 spec: 362 externalTrafficPolicy: Cluster 默认为local，修改为Cluster 363 ipFamilies: 364 - IPv4 365 ipFamilyPolicy: SingleStack 366 ports: 367 - appProtocol: http 368 name: http 369 port: 80 370 protocol: TCP 371 targetPort: http 372 - appProtocol: https 373 name: https 374 port: 443 375 protocol: TCP 376 targetPort: https 377 selector: 378 app.kubernetes.io/component: controller 379 app.kubernetes.io/instance: ingress-nginx 380 app.kubernetes.io/name: ingress-nginx 381 type: LoadBalancer 此处默认即为LoadBalancer,此版本不用修改。 ~~~ ~~~powershell # kubectl apply -f deploy.yaml ~~~ ~~~powershell # kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-szjlp 0/1 Completed 0 13m ingress-nginx-admission-patch-b8vnr 0/1 Completed 0 13m ingress-nginx-controller-b4fcbcc8f-mg9vm 1/1 Running 0 13m ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_openfass.md ================================================ # Serverless之OpenFaaS 函数即服务 # 一、OpenFaaS介绍 ![image-20221124112009308](../../img/kubernetes/kubernetes_openfass/image-20221124112009308.png) Openfaas是一种serverless提供方式，可以使开发人员轻松地将事件驱动的功能和微服务部署到Kubernetes中，能够自动将你的代码或已有的二进制文件打包到Docker容器镜像中，快速实现部署并提供即时访问接口，且在运行过程中具有自动缩放和指标的高度扩展性。 **OpenFaas 特性** - 开源函数框架 - 在任何云上运行函数，无需担心锁定 - 用任何语言编写函数并将它们打包到 Docker/OCI 格式的容器中 - 易于使用 - 内置 UI、强大的 CLI 和一键安装 - 随心所欲地扩展 - 处理流量高峰，并在空闲时缩减 - 活跃的社区 - 贡献和归属 - 开发人员社区版、专业版和生产支持 **Openfaas架构设计** ![img](../../img/kubernetes/kubernetes_openfass/15383821-b2406d917f0c17ec.png) **function调用路径** ![image-20221125102936359](../../img/kubernetes/kubernetes_openfass/image-20221125102936359.png) ![img](../../img/kubernetes/kubernetes_openfass/image-20221124.png) API网关为你的函数提供了一个外部路由，并通过Prometheus收集云原生指标。网关还有一个内置的UI界面，使得你可以从OpenFaaS的函数商店部署并调用自己的函数。 网关会根据需要自动伸缩函数，依据 Kubernetes的API来改变服务的副本数来做到。 在/system/alert端点接收AlertManager生成的自定义通知。 - 内置UI - 部署自定义函数，或者函数商店中的函数 - Prometheus的仪表盘 - 通过AlertManager自动伸缩 - REST API 当Gateway作为一个入口，当CLI或者web页面发来要部署或者调用一个函数的时候，Gateway会将请求转发给Provider，同时会将监控指标发给Prometheus。AlterManager会根据需求，调用API自动伸缩函数。 nats是一个开源的，云原生的消息系统，类似于ActiveMQ、KafKa、RabbitMq。Apcera，百度，西门子，VMware，HTC和爱立信等公司都有在使用。 核心基于EventMachine开发，原理是基于消息发布订阅机制，每台服务器上的每个模块会根据自己的消息类别向MessageBus发布多个消息主题，而同时也向自己需要交互的模块，按照需要的主题订阅消息。能够达到每秒8-11百万个消息，整个程序很小只有3M Docker image，它不支持持久化消息，如果你离线，你就不能获得消息。使用nats streaming可以做到持久化，缓存等功能。 ~~~powershell # kubectl get pods -n openfaas NAME READY STATUS RESTARTS AGE alertmanager-646f7b5cf4-w96hm 1/1 Running 1 (18h ago) 18h basic-auth-plugin-957dfbfb7-v5jv7 1/1 Running 1 (18h ago) 18h gateway-79458985bd-9g9xw 2/2 Running 4 (18h ago) 18h nats-7f8c77747c-dbqrm 1/1 Running 1 (18h ago) 18h prometheus-6f69c7b564-x8p8d 1/1 Running 1 (18h ago) 18h queue-worker-78bd96497-dnvt5 1/1 Running 1 (18h ago) 18h ~~~ ![img](../../img/kubernetes/kubernetes_openfass/15383821-3c8659e46094f5d1.png) # 二、OpenFaaS运行基础环境 > 官网链接：www.openfaas.com kubernetes现在已经成了环境管理及容器调度的实时标准了。openfaas的部署章节，推荐使用kubernetes进行部署和使用。 ![image-20221124104359974](../../img/kubernetes/kubernetes_openfass/image-20221124104359974.png) ![image-20221124105200142](../../img/kubernetes/kubernetes_openfass/image-20221124105200142.png) # 三、使用helm进行部署 ## 3.1 helm安装 > 参考链接：https://github.com/openfaas/faas-netes/blob/master/chart/openfaas/README.md ![image-20221124105412280](../../img/kubernetes/kubernetes_openfass/image-20221124105412280.png) ~~~powershell 下载helm curl -sSLf https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash ~~~ ~~~powershell 验证helm版本 helm version ~~~ ## 3.2 使用helm部署openfaas > 参考链接：https://github.com/openfaas/faas-netes/blob/master/chart/openfaas/README.md ![image-20221124105646619](../../img/kubernetes/kubernetes_openfass/image-20221124105646619.png) ~~~powershell helm repo add openfaas https://openfaas.github.io/faas-netes/ ~~~ ~~~powershell helm repo update ~~~ ~~~powershell kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml ~~~ >- openfaas：用于核心组件 >- openfaas-fn：用于functions ~~~powershell # kubectl get ns NAME STATUS AGE ...... openfaas Active 8s openfaas-fn Active 8s ~~~ ![image-20221124111657736](../../img/kubernetes/kubernetes_openfass/image-20221124111657736.png) ~~~powershell helm repo update \ && helm upgrade openfaas --install openfaas/openfaas \ --namespace openfaas \ --set functionNamespace=openfaas-fn \ --set generateBasicAuth=true ~~~ ~~~powershell Release "openfaas" does not exist. Installing it now. NAME: openfaas LAST DEPLOYED: Thu Nov 24 11:21:14 2022 NAMESPACE: openfaas STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: To verify that openfaas has started, run: kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas" To retrieve the admin password, run: echo $(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) ~~~ ## 3.3 openfaas部署验证 ~~~powershell # kubectl get all -n openfaas NAME READY STATUS RESTARTS AGE pod/alertmanager-646f7b5cf4-mpkxp 1/1 Running 0 112s pod/basic-auth-plugin-957dfbfb7-6wbwb 1/1 Running 0 112s pod/gateway-79458985bd-pd5lw 2/2 Running 0 112s pod/nats-7f8c77747c-tsxlg 1/1 Running 0 112s pod/prometheus-6f69c7b564-fn8vs 1/1 Running 0 112s pod/queue-worker-78bd96497-gwr7m 1/1 Running 1 (99s ago) 112s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/alertmanager ClusterIP 10.96.0.255 9093/TCP 112s service/basic-auth-plugin ClusterIP 10.96.2.167 8080/TCP 112s service/gateway ClusterIP 10.96.3.110 8080/TCP 112s service/gateway-external NodePort 10.96.3.20 8080:31112/TCP 112s service/gateway-provider ClusterIP 10.96.1.95 8081/TCP 112s service/nats ClusterIP 10.96.3.5 4222/TCP 112s service/prometheus ClusterIP 10.96.2.177 9090/TCP 112s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/alertmanager 1/1 1 1 112s deployment.apps/basic-auth-plugin 1/1 1 1 112s deployment.apps/gateway 1/1 1 1 112s deployment.apps/nats 1/1 1 1 112s deployment.apps/prometheus 1/1 1 1 112s deployment.apps/queue-worker 1/1 1 1 112s NAME DESIRED CURRENT READY AGE replicaset.apps/alertmanager-646f7b5cf4 1 1 1 112s replicaset.apps/basic-auth-plugin-957dfbfb7 1 1 1 112s replicaset.apps/gateway-79458985bd 1 1 1 112s replicaset.apps/nats-7f8c77747c 1 1 1 112s replicaset.apps/prometheus-6f69c7b564 1 1 1 112s replicaset.apps/queue-worker-78bd96497 1 1 1 112s ~~~ # 四、访问gateway管理界面 >可以看到已经部署了NodePort service，暴露端口为`31112`，可以通过`http://:31112`打开gateway的管理界面了。 ## 4.1 获取访问密码 ~~~powershell echo $(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) 输出密码为： 8hG1vZOzFfBs ~~~ 或 ~~~powershell PASSWORD=$(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) && \ echo "OpenFaaS admin password: $PASSWORD" 输出密码为： OpenFaaS admin password: 8hG1vZOzFfBs ~~~ ## 4.2 访问gateway管理界面 >打开gateway界面，用户名为admin，密码为上面查到内容。 ![image-20221124113114148](../../img/kubernetes/kubernetes_openfass/image-20221124113114148.png) ![image-20221124113230431](../../img/kubernetes/kubernetes_openfass/image-20221124113230431.png) ![image-20221124113610809](../../img/kubernetes/kubernetes_openfass/image-20221124113610809.png) # 五、OpenFaaS初体验 ![image-20221124114009593](../../img/kubernetes/kubernetes_openfass/image-20221124114009593.png) ![image-20221124114309331](../../img/kubernetes/kubernetes_openfass/image-20221124114309331.png) ~~~powershell # kubectl get all -n openfaas-fn NAME READY STATUS RESTARTS AGE pod/env-5f5d786cc7-wx9md 1/1 Running 0 7m43s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/env ClusterIP 10.96.0.160 8080/TCP 7m43s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/env 1/1 1 1 7m43s NAME DESIRED CURRENT READY AGE replicaset.apps/env-5f5d786cc7 1 1 1 7m43s ~~~ 获取访问信息 ![image-20221124114506374](../../img/kubernetes/kubernetes_openfass/image-20221124114506374.png) ![image-20221124114909763](../../img/kubernetes/kubernetes_openfass/image-20221124114909763.png) ![image-20221124114541763](../../img/kubernetes/kubernetes_openfass/image-20221124114541763.png) ![image-20221124120155418](../../img/kubernetes/kubernetes_openfass/image-20221124120155418.png) # 六、开发和部署function ![image-20221124182027663](../../img/kubernetes/kubernetes_openfass/image-20221124182027663.png) ## 6.1 安装命令行工具 faas-cli ![image-20221124121217520](../../img/kubernetes/kubernetes_openfass/image-20221124121217520.png) ~~~powershell curl -sSL https://cli.openfaas.com | sh ~~~ ![image-20221124121404135](../../img/kubernetes/kubernetes_openfass/image-20221124121404135.png) ~~~powershell # faas-cli version ___ _____ ____ / _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___| | | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \ | |_| | |_) | __/ | | | _| (_| | (_| |___) | \___/| .__/ \___|_| |_|_| \__,_|\__,_|____/ |_| CLI: commit: 0074051aeb837f5f160ee8736341460468b5c190 version: 0.15.4 ~~~ ## 6.2 登录faas-cli工具 ~~~powershell 设置OpenFaas gateway的地址 export OPENFAAS_URL=192.168.10.160:31112 ~~~ ~~~powershell 获取OpenFaas gateway admin的密码 $ PASSWORD=$(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) && echo "OpenFaaS admin password: $PASSWORD" ~~~ ~~~powershell 登录faas-cli $ echo -n $PASSWORD | faas-cli login -g $OPENFAAS_URL -u admin --password-stdin ~~~ ~~~powershell 登录提示： Calling the OpenFaaS server to validate the credentials... WARNING! You are not using an encrypted connection to the gateway, consider using HTTPS. credentials saved for admin http://192.168.10.160:31112 ~~~ ## 6.3 获取模板列表 OpenFaas提供了各种语言的模板列表，可以通过以下命令获取最新的列表 ~~~powershell # faas-cli template pull ~~~ ~~~powershell 下载提示： Fetch templates from repository: https://github.com/openfaas/templates.git at 2022/11/24 12:25:08 Attempting to expand templates from https://github.com/openfaas/templates.git 2022/11/24 12:25:11 Fetched 17 template(s) : [csharp dockerfile go java11 java11-vert-x node node12 node12-debian node14 node16 node17 php7 php8 python python3 python3-debian ruby] from https://github.com/openfaas/templates.git ~~~ ~~~powershell 查看模板列表 # faas-cli template store list NAME SOURCE DESCRIPTION csharp openfaas Classic C# template dockerfile openfaas Classic Dockerfile template go openfaas Classic Golang template java11 openfaas Java 11 template java11-vert-x openfaas Java 11 Vert.x template node17 openfaas HTTP-based Node 17 template node16 openfaas HTTP-based Node 16 template node14 openfaas HTTP-based Node 14 template node12 openfaas HTTP-based Node 12 template node openfaas Classic NodeJS 8 template puppeteer-nodelts alexellis A puppeteer template for headless Chrome php7 openfaas Classic PHP 7 template php8 openfaas Classic PHP 8 template python openfaas Classic Python 2.7 template python3 openfaas Classic Python 3.6 template python3-dlrs intel Deep Learning Reference Stack v0.4 for ML workloads ruby openfaas Classic Ruby 2.5 template ruby-http openfaas Ruby 2.4 HTTP template python27-flask openfaas Python 2.7 Flask template python3-flask openfaas Python 3.7 Flask template python3-flask-debian openfaas Python 3.7 Flask template based on Debian python3-http openfaas Python 3.7 with Flask and HTTP python3-http-debian openfaas Python 3.7 with Flask and HTTP based on Debian golang-http openfaas Golang HTTP template golang-middleware openfaas Golang Middleware template python3-debian openfaas Python 3 Debian template powershell-template openfaas-incubator Powershell Core Ubuntu:16.04 template powershell-http-template openfaas-incubator Powershell Core HTTP Ubuntu:16.04 template rust booyaa Rust template crystal tpei Crystal template csharp-httprequest distantcam C# HTTP template csharp-kestrel burtonr C# Kestrel HTTP template vertx-native pmlopes Eclipse Vert.x native image template swift affix Swift 4.2 Template lua53 affix Lua 5.3 Template vala affix Vala Template vala-http affix Non-Forking Vala Template quarkus-native pmlopes Quarkus.io native image template perl-alpine tmiklas Perl language template based on Alpine image crystal-http koffeinfrei Crystal HTTP template rust-http openfaas-incubator Rust HTTP template bash-streaming openfaas-incubator Bash Streaming template cobol devries COBOL Template ~~~ ## 6.4 根据模板创建项目 > 需要本地容器镜像仓库harbor ~~~powershell # faas-cli new helloworld --lang node17 -p 192.168.10.163/library Folder: hellowoard created. ___ _____ ____ / _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___| | | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \ | |_| | |_) | __/ | | | _| (_| | (_| |___) | \___/| .__/ \___|_| |_|_| \__,_|\__,_|____/ |_| Function created in folder: helloworld Stack file written: helloworld.yml Notes: You have created a new function which uses Node.js 17 and the OpenFaaS of-watchdog which gives greater control over HTTP responses. npm i --save can be used to add third-party packages like request or cheerio npm documentation: https://docs.npmjs.com/ Unit tests are run at build time via "npm run", edit package.json to specify how you want to execute them. ~~~ ~~~powershell # ls hellowoard.yml ~~~ ~~~powershell # vim hellowoard.yml # cat hellowoard.yml version: 1.0 provider: name: openfaas gateway: http://192.168.10.160:31112 functions: hellowoard: lang: node17 handler: ./hellowoard image: 192.168.10.163/library/helloworld:latest ~~~ ## 6.5 构建并发布function ~~~powershell # docker login 192.168.10.163 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ~~~powershell # faas-cli build -f hellowoard.yml ~~~ ~~~powershell 输出内容：现场制作容器镜像 [0] > Building hellowoard. Clearing temporary build folder: ./build/hellowoard/ Preparing: ./hellowoard/ build/hellowoard/function Building: www.kubemsb.com/library/hellowoard:latest with node17 template. Please wait.. Sending build context to Docker daemon 12.8kB Step 1/31 : FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.9.10 as watchdog ---> 9f97468a4531 Step 2/31 : FROM --platform=${TARGETPLATFORM:-linux/amd64} node:17-alpine as ship ---> 57488723f087 Step 3/31 : ARG TARGETPLATFORM ---> Running in 0b7f8643414c Removing intermediate container 0b7f8643414c ---> 6ca3a7a51d47 Step 4/31 : ARG BUILDPLATFORM ---> Running in 82454128d8c4 Removing intermediate container 82454128d8c4 ---> 638ddef53476 Step 5/31 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog ---> e653b619f59e Step 6/31 : RUN chmod +x /usr/bin/fwatchdog ---> Running in 897382feb255 Removing intermediate container 897382feb255 ---> 59c994b15db8 Step 7/31 : RUN apk --no-cache add curl ca-certificates && addgroup -S app && adduser -S -g app app ---> Running in c4f01a1bc53f fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/5) Installing ca-certificates (20220614-r0) (2/5) Installing brotli-libs (1.0.9-r5) (3/5) Installing nghttp2-libs (1.46.0-r0) (4/5) Installing libcurl (7.80.0-r4) (5/5) Installing curl (7.80.0-r4) Executing busybox-1.34.1-r5.trigger Executing ca-certificates-20220614-r0.trigger OK: 10 MiB in 21 packages Removing intermediate container c4f01a1bc53f ---> 5bf3224bb8a3 Step 8/31 : ENV NPM_CONFIG_LOGLEVEL warn ---> Running in 59f234b7e326 Removing intermediate container 59f234b7e326 ---> a2d3940460fb Step 9/31 : RUN chmod 777 /tmp ---> Running in 46a686ba987a Removing intermediate container 46a686ba987a ---> 7ea73920ff98 Step 10/31 : USER app ---> Running in 13b8c0e7d772 Removing intermediate container 13b8c0e7d772 ---> df386a70e76a Step 11/31 : RUN mkdir -p /home/app/function ---> Running in 633412ec863d Removing intermediate container 633412ec863d ---> 78bd26cddd01 Step 12/31 : WORKDIR /home/app ---> Running in 59176c2c8794 Removing intermediate container 59176c2c8794 ---> 01d66992d0bb Step 13/31 : COPY --chown=app:app package.json ./ ---> 34e943bc9eff Step 14/31 : RUN npm i ---> Running in e96cdf2a1bc7 added 57 packages, and audited 58 packages in 4s 7 packages are looking for funding run `npm fund` for details found 0 vulnerabilities npm notice npm notice New major version of npm available! 8.11.0 -> 9.1.2 npm notice Changelog: npm notice Run `npm install -g npm@9.1.2` to update! npm notice Removing intermediate container e96cdf2a1bc7 ---> 2e8f491b4ee5 Step 15/31 : COPY --chown=app:app index.js ./ ---> d101cde972b3 Step 16/31 : WORKDIR /home/app/function ---> Running in fe5700f37a20 Removing intermediate container fe5700f37a20 ---> b1d49b003ebb Step 17/31 : COPY --chown=app:app function/*.json ./ ---> 3e7f5a4fe5ae Step 18/31 : RUN npm i ---> Running in 37135d3aa520 up to date, audited 1 package in 137ms found 0 vulnerabilities Removing intermediate container 37135d3aa520 ---> 997da5f36cd4 Step 19/31 : COPY --chown=app:app function/ ./ ---> 7ca14d566a1d Step 20/31 : RUN npm test ---> Running in 8ee1030b112f > openfaas-function@1.0.0 test > echo "Error: no test specified" && exit 0 Error: no test specified Removing intermediate container 8ee1030b112f ---> 7b65e89a46eb Step 21/31 : WORKDIR /home/app/ ---> Running in 9cd82539cef8 Removing intermediate container 9cd82539cef8 ---> d55568f994ac Step 22/31 : ENV cgi_headers="true" ---> Running in 7640c091a499 Removing intermediate container 7640c091a499 ---> 1d4a00728497 Step 23/31 : ENV fprocess="node index.js" ---> Running in f7fbb1c52b44 Removing intermediate container f7fbb1c52b44 ---> bb1d909c9c06 Step 24/31 : ENV mode="http" ---> Running in 1fa2d2113def Removing intermediate container 1fa2d2113def ---> 20398232e2c1 Step 25/31 : ENV upstream_url="http://127.0.0.1:3000" ---> Running in 5c9cf3fda8cb Removing intermediate container 5c9cf3fda8cb ---> 15fd083830c6 Step 26/31 : ENV exec_timeout="10s" ---> Running in 686ede8f15b2 Removing intermediate container 686ede8f15b2 ---> 4b061fc53091 Step 27/31 : ENV write_timeout="15s" ---> Running in d7cd8dcbb89c Removing intermediate container d7cd8dcbb89c ---> 6415ecf24827 Step 28/31 : ENV read_timeout="15s" ---> Running in a9c708e4fa6c Removing intermediate container a9c708e4fa6c ---> 92836ac38c09 Step 29/31 : ENV prefix_logs="false" ---> Running in 4fd948efe2ba Removing intermediate container 4fd948efe2ba ---> a0c9c4a83ad6 Step 30/31 : HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1 ---> Running in a7ea4775b045 Removing intermediate container a7ea4775b045 ---> 0a8359a2c326 Step 31/31 : CMD ["fwatchdog"] ---> Running in 13af34489af2 Removing intermediate container 13af34489af2 ---> 58ef3dc504a2 Successfully built 58ef3dc504a2 Successfully tagged 192.168.10.163/library/helloworld:latest Image: 192.168.10.163/library/helloworld:latest built. [0] < Building hellowoard done in 12.64s. [0] Worker done. Total build time: 12.64s ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.10.163/library/helloworld latest 58ef3dc504a2 53 seconds ago 194MB ~~~ ~~~powershell # # faas-cli push -f helloworld.yml [0] > Pushing helloworld [192.168.10.163/library/helloworld:latest] The push refers to repository [192.168.10.163/library/helloworld] 2f8ab7175cca: Pushed 72580dd28d28: Pushed 22e68554b4c4: Pushed e9970376b1c4: Pushed 616e13af0ac0: Pushed 43158d5b82e4: Pushed e272465983a4: Pushed 64af5fc26ddf: Pushed 896605964f7e: Pushed 79b014d8db02: Pushed 4607252a98e4: Pushed e6a74996eabe: Pushed db2e1fd51a80: Pushed 19ebba8d6369: Pushed 4fc242d58285: Pushed latest: digest: sha256:0f8253da4a46655c9bf34a65c538a31efd62634e89f886668f734cd0a4270fd3 size: 3660 [0] < Pushing helloworld [192.168.10.163/library/helloworld:latest] done. [0] Worker done. ~~~ ![image-20221124175920457](../../img/kubernetes/kubernetes_openfass/image-20221124175920457.png) ~~~powershell # echo -n $PASSWORD | faas-cli login -g $OPENFAAS_URL -u admin --password-stdin ~~~ ~~~powershell # faas-cli deploy -f helloworld.yml Deploying: helloworld. WARNING! You are not using an encrypted connection to the gateway, consider using HTTPS. Deployed. 202 Accepted. URL: http://192.168.10.160:31112/function/helloworld ~~~ ~~~powershell # kubectl get pods -n openfaas-fn NAME READY STATUS RESTARTS AGE helloworld-8668999bcc-qph5w 1/1 Running 0 17s ~~~ ![image-20221124180309309](../../img/kubernetes/kubernetes_openfass/image-20221124180309309.png) ![image-20221124181118239](../../img/kubernetes/kubernetes_openfass/image-20221124181118239.png) ~~~powershell # curl http://192.168.10.160:31112/function/helloworld -X POST -d 'kubemsb' {"body":"{\"kubemsb\":\"\"}","content-type":"application/x-www-form-urlencoded"} ~~~ ~~~powershell # faas-cli remove -f helloworld.yml Deleting: helloworld. Removing old function. ~~~ # 七、部署其它编程语言模板function ~~~powershell # mkdir pl ~~~ ~~~powershell # faas-cli new python-demo --lang python -p 192.168.10.163/library 2022/11/24 18:25:01 No templates found in current directory. 2022/11/24 18:25:01 Attempting to expand templates from https://github.com/openfaas/templates.git 2022/11/24 18:25:04 Fetched 17 template(s) : [csharp dockerfile go java11 java11-vert-x node node12 node12-debian node14 node16 node17 php7 php8 python python3 python3-debian ruby] from https://github.com/openfaas/templates.git Folder: python-demo created. ___ _____ ____ / _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___| | | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \ | |_| | |_) | __/ | | | _| (_| | (_| |___) | \___/| .__/ \___|_| |_|_| \__,_|\__,_|____/ |_| Function created in folder: python-demo Stack file written: python-demo.yml ~~~ ~~~powershell # ls python-demo python-demo.yml template ~~~ ~~~powershell # vim python-demo.yml # cat python-demo.yml version: 1.0 provider: name: openfaas gateway: http://192.168.10.160:31112 functions: python-demo: lang: python handler: ./python-demo image: 192.168.10.163/library/python-demo:latest ~~~ ~~~powershell # cd python-demo/ [root@k8s-master01 python-demo]# ls handler.py requirements.txt [root@k8s-master01 python-demo]# cat handler.py def handle(req): """handle a request to the function Args: req (str): request body """ return req [root@k8s-master01 python-demo]# vim handler.py [root@k8s-master01 python-demo]# cat handler.py def handle(req): """handle a request to the function Args: req (str): request body """ return "hello function " + req 添加了内容 ~~~ ~~~powershell [root@k8s-master01 pl]# ls python-demo python-demo.yml template [root@k8s-master01 pl]# faas-cli build -f python-demo.yml ~~~ ~~~powershell 输出内容： [0] > Building python-demo. Clearing temporary build folder: ./build/python-demo/ Preparing: ./python-demo/ build/python-demo/function Building: 192.168.10.163/library/python-demo:latest with python template. Please wait.. Sending build context to Docker daemon 8.192kB Step 1/31 : FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/classic-watchdog:0.2.1 as watchdog 0.2.1: Pulling from openfaas/classic-watchdog e1749af664ff: Pulling fs layer e1749af664ff: Verifying Checksum e1749af664ff: Download complete e1749af664ff: Pull complete Digest: sha256:640de69b1d683cbfa73fd3b2d707d33a4e4570164c9795c3be028949688e5c61 Status: Downloaded newer image for ghcr.io/openfaas/classic-watchdog:0.2.1 ---> 021a98fdbddd Step 2/31 : FROM --platform=${TARGETPLATFORM:-linux/amd64} python:2.7-alpine 2.7-alpine: Pulling from library/python aad63a933944: Pulling fs layer 259d822268fb: Pulling fs layer 10ba96d218d3: Pulling fs layer 44ba9f6a4209: Pulling fs layer 44ba9f6a4209: Waiting aad63a933944: Download complete aad63a933944: Pull complete 10ba96d218d3: Verifying Checksum 10ba96d218d3: Download complete 44ba9f6a4209: Verifying Checksum 44ba9f6a4209: Download complete 259d822268fb: Retrying in 5 seconds 259d822268fb: Retrying in 4 seconds 259d822268fb: Retrying in 3 seconds 259d822268fb: Retrying in 2 seconds 259d822268fb: Retrying in 1 second 259d822268fb: Verifying Checksum 259d822268fb: Download complete 259d822268fb: Pull complete 10ba96d218d3: Pull complete 44ba9f6a4209: Pull complete Digest: sha256:724d0540eb56ffaa6dd770aa13c3bc7dfc829dec561d87cb36b2f5b9ff8a760a Status: Downloaded newer image for python:2.7-alpine ---> 8579e446340f Step 3/31 : ARG TARGETPLATFORM ---> Running in c26ca1bbf850 Removing intermediate container c26ca1bbf850 ---> f40c5f5b8fd1 Step 4/31 : ARG BUILDPLATFORM ---> Running in a4151dd3cf5a Removing intermediate container a4151dd3cf5a ---> 75659f182395 Step 5/31 : ARG ADDITIONAL_PACKAGE ---> Running in 76a400f5de60 Removing intermediate container 76a400f5de60 ---> 737eaa099199 Step 6/31 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog ---> 3b3e3e9fd519 Step 7/31 : RUN chmod +x /usr/bin/fwatchdog ---> Running in a714a0aa618b Removing intermediate container a714a0aa618b ---> 73061af0ecbb Step 8/31 : RUN apk --no-cache add ca-certificates ${ADDITIONAL_PACKAGE} ---> Running in d53508362f5b fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz OK: 11 MiB in 32 packages Removing intermediate container d53508362f5b ---> 96dbdb8cba4c Step 9/31 : RUN addgroup -S app && adduser app -S -G app ---> Running in e5a46359d098 Removing intermediate container e5a46359d098 ---> d0e784d71ee6 Step 10/31 : WORKDIR /home/app/ ---> Running in 46793fbe6fda Removing intermediate container 46793fbe6fda ---> 05e976bdcd71 Step 11/31 : COPY index.py . ---> d626bc3acb48 Step 12/31 : COPY requirements.txt . ---> 7b9ba7660a76 Step 13/31 : RUN chown -R app /home/app && mkdir -p /home/app/python && chown -R app /home/app ---> Running in a35db473c903 Removing intermediate container a35db473c903 ---> 8e262945a3f2 Step 14/31 : USER app ---> Running in f3a65f2901a9 Removing intermediate container f3a65f2901a9 ---> cc12b9a6286e Step 15/31 : ENV PATH=$PATH:/home/app/.local/bin:/home/app/python/bin/ ---> Running in 3e11289adbb2 Removing intermediate container 3e11289adbb2 ---> 54ad4e19a031 Step 16/31 : ENV PYTHONPATH=$PYTHONPATH:/home/app/python ---> Running in dbdeae97cf27 Removing intermediate container dbdeae97cf27 ---> 666a540d411c Step 17/31 : RUN pip install -r requirements.txt --target=/home/app/python ---> Running in a0e7f7380801 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support WARNING: You are using pip version 20.0.2; however, version 20.3.4 is available. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command. Removing intermediate container a0e7f7380801 ---> 8f7a6ad3a148 Step 18/31 : RUN mkdir -p function ---> Running in 5bb68121e113 Removing intermediate container 5bb68121e113 ---> fa0cff1f7478 Step 19/31 : RUN touch ./function/__init__.py ---> Running in 7da2f055434a Removing intermediate container 7da2f055434a ---> fee8adecc37d Step 20/31 : WORKDIR /home/app/function/ ---> Running in 55c057bd93d4 Removing intermediate container 55c057bd93d4 ---> a608325243c0 Step 21/31 : COPY function/requirements.txt . ---> 34f27e58e110 Step 22/31 : RUN pip install -r requirements.txt --target=/home/app/python ---> Running in 676540bb1990 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support WARNING: You are using pip version 20.0.2; however, version 20.3.4 is available. You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command. Removing intermediate container 676540bb1990 ---> 4d7d1bbe8e3a Step 23/31 : WORKDIR /home/app/ ---> Running in 16e2280c32cb Removing intermediate container 16e2280c32cb ---> d23c17c42492 Step 24/31 : USER root ---> Running in d1862e78e61b Removing intermediate container d1862e78e61b ---> ebc5ad55b29f Step 25/31 : COPY function function ---> e88be2f10c60 Step 26/31 : RUN chown -R app:app ./ && chmod -R 777 /home/app/python ---> Running in 234f8fa7cf73 Removing intermediate container 234f8fa7cf73 ---> 56489e89ba3f Step 27/31 : USER app ---> Running in 5ac2c43773bf Removing intermediate container 5ac2c43773bf ---> b1250cb46911 Step 28/31 : ENV fprocess="python index.py" ---> Running in 17abf9c8a7fb Removing intermediate container 17abf9c8a7fb ---> b6046bf381f4 Step 29/31 : EXPOSE 8080 ---> Running in 1a04f0d42be5 Removing intermediate container 1a04f0d42be5 ---> fee47369368e Step 30/31 : HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1 ---> Running in 50ea1f87b1e9 Removing intermediate container 50ea1f87b1e9 ---> 13b59f25fbc1 Step 31/31 : CMD ["fwatchdog"] ---> Running in 27403d3a56b2 Removing intermediate container 27403d3a56b2 ---> 568fbf0171e5 Successfully built 568fbf0171e5 Successfully tagged 192.168.10.163/library/python-demo:latest Image: 192.168.10.163/library/python-demo:latest built. [0] < Building python-demo done in 154.40s. [0] Worker done. Total build time: 154.40s ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.10.163/library/python-demo latest 568fbf0171e5 54 seconds ago 87.3MB ~~~ ~~~powershell # faas-cli push -f python-demo.yml [0] > Pushing python-demo [192.168.10.163/library/python-demo:latest] The push refers to repository [192.168.10.163/library/python-demo] 4e80d6112849: Pushed fb6d78c26a34: Pushed 820eb676a69e: Pushed d587ee49c6a4: Pushed 8a8abd3ceddc: Pushed d82f40d56cc4: Pushed 6c8a6f5c151c: Pushed ff049ec38e28: Pushed a3a372e107c7: Pushed 140e08eabde8: Pushed 4d263f56c46d: Pushed cf79dfe6afa5: Pushed 1b5baba80737: Pushed 879c0d8666e3: Pushed 20a7b70bdf2f: Pushed 3fc750b41be7: Pushed beee9f30bc1f: Pushed latest: digest: sha256:cbcb9b1bbae25f13f335eabd08657fc40beb9039d26d1a07f129fa61c68ebca5 size: 4074 [0] < Pushing python-demo [192.168.10.163/library/python-demo:latest] done. [0] Worker done. ~~~ ![image-20221124184045647](../../img/kubernetes/kubernetes_openfass/image-20221124184045647.png) ~~~powershell # faas-cli deploy -f python-demo.yml Deploying: python-demo. WARNING! You are not using an encrypted connection to the gateway, consider using HTTPS. Deployed. 202 Accepted. URL: http://192.168.10.160:31112/function/python-demo ~~~ ~~~powershell # faas-cli list Function Invocations Replicas python-demo 0 1 ~~~ ![image-20221124184414872](../../img/kubernetes/kubernetes_openfass/image-20221124184414872.png) ![image-20221124184515057](../../img/kubernetes/kubernetes_openfass/image-20221124184515057.png) # 八、OpenFaaS自动扩缩容 ## 8.1 自动扩缩容测试 > Auto-scaling > 部署成功的Function是具有扩缩容能力的,可以使用 ab命令压测 ![img](../../img/kubernetes/kubernetes_openfass/15383821-248bb7249f586186.png) - 压测之前，先观察pod的情况 ~~~powershell # kubectl get pods -n openfaas-fn NAME READY STATUS RESTARTS AGE python-demo-67564cf477-xh8nw 1/1 Running 0 4h22m ~~~ - 执行压测命令 ~~~powershell # yum -y install httpd-tools ~~~ ~~~powershell # ab -c 10 -n 10000 -k http://192.168.10.160:31112/function/python-demo This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 192.168.10.160 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests Server Software: Server Hostname: 192.168.10.160 Server Port: 31112 Document Path: /function/python-demo Document Length: 16 bytes Concurrency Level: 10 Time taken for tests: 89.722 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 10000 Total transferred: 2710000 bytes HTML transferred: 160000 bytes Requests per second: 111.46 [#/sec] (mean) Time per request: 89.722 [ms] (mean) Time per request: 8.972 [ms] (mean, across all concurrent requests) Transfer rate: 29.50 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 2 Processing: 33 90 31.7 88 502 Waiting: 32 90 31.7 87 502 Total: 34 90 31.7 88 502 Percentage of the requests served within a certain time (ms) 50% 88 66% 98 75% 105 80% 110 90% 128 95% 146 98% 170 99% 188 100% 502 (longest request) ~~~ - 观察压测过程中pod变化 > 再打开一个终端 ![image-20221124231043648](../../img/kubernetes/kubernetes_openfass/image-20221124231043648.png) - 压测完成后pod变化 ![image-20221124231203352](../../img/kubernetes/kubernetes_openfass/image-20221124231203352.png) ## 8.2 自定义参数使用Auto-Scaling实现自动扩缩容 >Function的扩缩容由Label来设置。 ~~~powershell $ faas-cli deploy -f helloworld.yml \ --label com.openfaas.scale.max=10 \ --label com.openfaas.scale.target=5 \ --label com.openfaas.scale.type=capacity \ --label com.openfaas.scale.target-proportion=1.0 \ --label com.openfaas.scale.zero=true \ --label com.openfaas.scale.zero-duration=5m ~~~ 标签说明文档如下： | 标签 | 含义 | 默认值 | | ------------------------------------ | ------------------------------------------------------------ | ------ | | com.openfaas.scale.max | 最多扩容到几个replicaset | 20 | | com.openfaas.scale.min | 最小缩容到几个replicaset | 1 | | com.openfaas.scale.zero | 是否允许缩容到0个，即关闭所有的replicaset | false | | com.openfaas.scale.zero-duration | 空闲多久之后，缩容为0 | 15m | | com.openfaas.scale.target | 当每个replicaset的压力达到多少时进行扩容 | 50 | | com.openfaas.scale.target-proportion | 扩容时的缩放系数，用来避免爆炸式扩容或扩容不足的发生。为一个浮点数 | 0.90 | | com.openfaas.scale.type | 扩缩容的模式，可选择rps、capacity、cpu | rps | ~~~powershell # faas-cli deploy -f helloworld.yml \ > --label com.openfaas.scale.max=10 \ > --label com.openfaas.scale.target=5 \ > --label com.openfaas.scale.type=capacity \ > --label com.openfaas.scale.target-proportion=1.0 \ > --label com.openfaas.scale.zero=true \ > --label com.openfaas.scale.zero-duration=5m Deploying: helloworld. WARNING! You are not using an encrypted connection to the gateway, consider using HTTPS. Deployed. 202 Accepted. URL: http://192.168.10.160:31112/function/helloworld ~~~ ~~~powershell ab -c 10 -n 10000 -k http://192.168.10.160:31112/function/helloworld ~~~ ![image-20221124235014191](../../img/kubernetes/kubernetes_openfass/image-20221124235014191.png) > 扩缩容计算公式： > > 扩缩容Replicaset个数 = 当前的replicaset个数 * (当前的压力 / ( (`com.openfaas.scale.target` * 当前的replicaset个数 ) * `com.openfaas.scale.target-proportion` ) ) `当前的压力`: 当扩缩容的模式为`capacity`时，可以理解为排队请求的个数，或者正在等待响应的请求个数。 **扩缩容模式** 一个关键的参数`com.openfaas.scale.type`的三种类型也需要精心选择。 - rps 每秒钟完成的请求个数（Requests Per Second），适用于快速响应的函数类型 - capacity 排队的请求个数，或者正在等待响应的请求个数，适用于响应较慢的函数，调用量不大但是容易发生堵塞的调用。 - cpu 根据CPU的负载进行扩缩容，适用于CPU消耗很高的函数，比如AI计算函数。 ================================================ FILE: docs/cloud/kubernetes/kubernetes_rancher.md ================================================ # Rancher容器云管理平台 # 一、主机硬件说明 | 序号 | 硬件 | 操作及内核 | | ---- | ------------------------- | ---------- | | 1 | CPU 4 Memory 4G Disk 100G | CentOS7 | | 2 | CPU 4 Memory 4G Disk 100G | CentOS7 | | 3 | CPU 4 Memory 4G Disk 100G | CentOS7 | | 4 | CPU 4 Memory 4G Disk 100G | CentOS7 | # 二、主机配置 ## 2.1 主机名 ~~~powershell # hostnamectl set-hostname rancherserver ~~~ ~~~powershell # hostnamectl set-hostname k8s-master01 ~~~ ~~~powershell # hostnamectl set-hostname k8s-worker01 ~~~ ~~~powershell # hostnamectl set-hostname k8s-worker02 ~~~ ## 2.2 IP地址 ~~~powershell [root@rancherserver ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.130" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell [root@k8s-master01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@k8s-master01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.131" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell [root@k8s-worker01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@k8s-worker01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.132" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell [root@k8s-worker02 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 [root@k8s-worker02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.133" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ## 2.3 主机名与IP地址解析 ~~~powershell # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.130 rancherserver 192.168.10.131 k8s-master01 192.168.10.132 k8s-worker01 192.168.10.133 k8s-worker02 ~~~ ## 2.4 主机安全设置 ~~~powershell # systemctl stop firewalld;systemctl disable firewalld # firewall-cmd --state not running ~~~ ~~~powershell # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ## 2.5 主机时钟同步 ~~~powershell # crontab -l 0 */1 * * * ntpdate time1.aliyun.com ~~~ ## 2.6 关闭swap > 关闭k8s集群节点swap ~~~powershell # cat /etc/fstab 默认开启，修改后关闭 #/dev/mapper/centos-swap swap swap defaults 0 0 ~~~ ~~~powershell 临时关闭所有 # swapoff -a ~~~ ## 2.7 配置内核路由转发 ~~~powershell # vim /etc/sysctl.conf # cat /etc/sysctl.conf ... net.ipv4.ip_forward=1 ~~~ ~~~powershell # sysctl -p net.ipv4.ip_forward = 1 ~~~ # 三、docker-ce安装 > 所有主机安装docker-ce ~~~powershell # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ~~~powershell # yum -y install docker-ce ~~~ ~~~powershell # systemctl enable --now docker ~~~ # 四、rancher安装 ![image-20220816144830488](../../img/kubernetes/kubernetes_rancher/image-20220816144830488.png) ![image-20220816144953656](../../img/kubernetes/kubernetes_rancher/image-20220816144953656.png) ~~~powershell [root@rancherserver ~]# docker pull rancher/rancher:v2.5.15 ~~~ ~~~powershell [root@rancherserver ~]# mkdir -p /opt/data/rancher_data ~~~ ~~~powershell [root@rancherserver ~]# docker run -d --privileged -p 80:80 -p 443:443 -v /opt/data/rancher_data:/var/lib/rancher --restart=always --name rancher-2.5.15 rancher/rancher:v2.5.15 ~~~ ~~~powershell [root@rancherserver ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 99e367eb35a3 rancher/rancher:v2.5.15 "entrypoint.sh" 26 seconds ago Up 26 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp rancher-2.5.15 ~~~ # 五、通过Rancher部署kubernetes集群 ## 5.1 Rancher访问 ![image-20220816150151634](../../img/kubernetes/kubernetes_rancher/image-20220816150151634.png) ![image-20220816150209913](../../img/kubernetes/kubernetes_rancher/image-20220816150209913.png) ![image-20220816150231704](../../img/kubernetes/kubernetes_rancher/image-20220816150231704.png) ![image-20220816150305037](../../img/kubernetes/kubernetes_rancher/image-20220816150305037.png) > 本次密码为Kubemsb123 ![image-20220816150322211](../../img/kubernetes/kubernetes_rancher/image-20220816150322211.png) ![image-20220816150450409](../../img/kubernetes/kubernetes_rancher/image-20220816150450409.png) ## 5.2 通过Rancher创建Kubernetes集群 ![image-20220816150706088](../../img/kubernetes/kubernetes_rancher/image-20220816150706088.png) ![image-20220816150736103](../../img/kubernetes/kubernetes_rancher/image-20220816150736103.png) ![image-20220816150900071](../../img/kubernetes/kubernetes_rancher/image-20220816150900071.png) ![image-20220816151001001](../../img/kubernetes/kubernetes_rancher/image-20220816151001001.png) ![image-20220816151039880](../../img/kubernetes/kubernetes_rancher/image-20220816151039880.png) ![image-20220816151117519](../../img/kubernetes/kubernetes_rancher/image-20220816151117519.png) ![image-20220816151131738](../../img/kubernetes/kubernetes_rancher/image-20220816151131738.png) ![image-20220816151144464](../../img/kubernetes/kubernetes_rancher/image-20220816151144464.png) ![image-20220816151157176](../../img/kubernetes/kubernetes_rancher/image-20220816151157176.png) ![image-20220816151225471](../../img/kubernetes/kubernetes_rancher/image-20220816151225471.png) ![image-20220816151259299](../../img/kubernetes/kubernetes_rancher/image-20220816151259299.png) 添加master节点主机 ![image-20220816151352377](../../img/kubernetes/kubernetes_rancher/image-20220816151352377.png) ~~~powershell [root@k8s-master01 ~]# docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.5.15 --server https://192.168.10.130 --token 7985nkpc48854kwmgh6pnfb7hcrkwhlcmx6wxq8tb4vszxn2qv9xdd --ca-checksum f6d5f24fcd41aa057a205d4f6922dfd309001126d040431222bfba7aa93517b7 --etcd --controlplane --worker ~~~ ~~~powershell [root@k8s-master01 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8e7e73b477dc rancher/rancher-agent:v2.5.15 "run.sh --server htt…" 20 seconds ago Up 18 seconds brave_ishizaka ~~~ 添加 worker节点 ![image-20220816151709536](../../img/kubernetes/kubernetes_rancher/image-20220816151709536.png) ![image-20220816151731937](../../img/kubernetes/kubernetes_rancher/image-20220816151731937.png) ![image-20220816151746413](../../img/kubernetes/kubernetes_rancher/image-20220816151746413.png) ~~~powershell [root@k8s-worker01 ~]# docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.5.15 --server https://192.168.10.130 --token 7985nkpc48854kwmgh6pnfb7hcrkwhlcmx6wxq8tb4vszxn2qv9xdd --ca-checksum f6d5f24fcd41aa057a205d4f6922dfd309001126d040431222bfba7aa93517b7 --worker ~~~ ~~~powershell [root@k8s-worker02 ~]# docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.5.15 --server https://192.168.10.130 --token 7985nkpc48854kwmgh6pnfb7hcrkwhlcmx6wxq8tb4vszxn2qv9xdd --ca-checksum f6d5f24fcd41aa057a205d4f6922dfd309001126d040431222bfba7aa93517b7 --worker ~~~ ![image-20220816152002056](../../img/kubernetes/kubernetes_rancher/image-20220816152002056.png) 所有节点激活后状态 ![image-20220816152858892](../../img/kubernetes/kubernetes_rancher/image-20220816152858892.png) ![image-20220816152917840](../../img/kubernetes/kubernetes_rancher/image-20220816152917840.png) ![image-20220816153009799](../../img/kubernetes/kubernetes_rancher/image-20220816153009799.png) # 六、配置通过命令行访问Kubernetes集群 ![image-20220816153241393](../../img/kubernetes/kubernetes_rancher/image-20220816153241393.png) ![image-20220816153301099](../../img/kubernetes/kubernetes_rancher/image-20220816153301099.png) ![image-20220816153357410](../../img/kubernetes/kubernetes_rancher/image-20220816153357410.png) 在集群节点命令行，如果访问呢？ ![image-20220816153917499](../../img/kubernetes/kubernetes_rancher/image-20220816153917499.png) ![image-20220816154006380](../../img/kubernetes/kubernetes_rancher/image-20220816154006380.png) ~~~powershell [root@k8s-master01 ~]# cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF ~~~ 修改gpgcheck=0及修改repo_gpgcheck=0 ~~~powershell [root@k8s-master01 ~]# vim /etc/yum.repos.d/kubernetes.repo [root@k8s-master01 ~]# cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ~~~ ~~~powershell [root@k8s-master01 ~]# yum -y install kubectl ~~~ ~~~powershell [root@k8s-master01 ~]# mkdir ~/.kube ~~~ ![image-20220816155110578](../../img/kubernetes/kubernetes_rancher/image-20220816155110578.png) ![image-20220816155140075](../../img/kubernetes/kubernetes_rancher/image-20220816155140075.png) ~~~powershell [root@k8s-master01 ~]# vim ~/.kube/config [root@k8s-master01 ~]# cat ~/.kube/config apiVersion: v1 kind: Config clusters: - name: "kubemsb-smart-1" cluster: server: "https://192.168.10.130/k8s/clusters/c-5jtsf" certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJwekNDQ\ VUyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQTdNUnd3R2dZRFZRUUtFeE5rZVc1aGJXbGoKY\ kdsemRHVnVaWEl0YjNKbk1Sc3dHUVlEVlFRREV4SmtlVzVoYldsamJHbHpkR1Z1WlhJdFkyRXdIa\ GNOTWpJdwpPREUyTURZMU9UVTBXaGNOTXpJd09ERXpNRFkxT1RVMFdqQTdNUnd3R2dZRFZRUUtFe\ E5rZVc1aGJXbGpiR2x6CmRHVnVaWEl0YjNKbk1Sc3dHUVlEVlFRREV4SmtlVzVoYldsamJHbHpkR\ 1Z1WlhJdFkyRXdXVEFUQmdjcWhrak8KUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVJMbDZKOWcxMlJQT\ G93dnVHZkM0YnR3ZmhHUDBpR295N1U2cjBJK0JZeAozZCtuVDBEc3ZWOVJWV1BDOGZCdGhPZmJQN\ GNYckx5YzJsR081RHkrSXRlM28wSXdRREFPQmdOVkhROEJBZjhFCkJBTUNBcVF3RHdZRFZSMFRBU\ UgvQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVNnZYWXBRYm9IdXF0UlBuS1FrS3gKMjBSZzJqMHdDZ\ 1lJS29aSXpqMEVBd0lEU0FBd1JRSWdMTUJ6YXNDREU4T2tCUk40TWRuZWNRU0xjMFVXQmNseApGO\ UFCem1MQWQwb0NJUUNlRWFnRkdBa1ZsUnV1czllSE1VRUx6ODl6VlY5L096b3hvY1ROYnA5amlBP\ T0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==" - name: "kubemsb-smart-1-k8s-master01" cluster: server: "https://192.168.10.131:6443" certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM0VENDQ\ WNtZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFTTVJBd0RnWURWUVFERXdkcmRXSmwKT\ FdOaE1CNFhEVEl5TURneE5qQTNNVFV3TmxvWERUTXlNRGd4TXpBM01UVXdObG93RWpFUU1BNEdBM\ VVFQXhNSAphM1ZpWlMxallUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ\ 0VCQUt4Qkh3S0RFZE5WCm1tU2JFZDZXaTZzRFNXcklPZEZ5dFN5Z1puVmk2cXFkYmxXODZRQ1Y1U\ WdEckI5QU43aDJ1RHRZMlFiNGZOTmEKQWZkSVVhS2tjZ0taNnplS1Z3eEdRYkptcEovSk5yYWw2N\ ENldng5QTU1UUFBL29FSzBVbkdackliSjQ5dEl4awp1TnMwNFVIRWxVVUZpWjlmckdBQU9sK3lVa\ GxXQUlLQzhmMUZSeVhpaVZEN2FTcTVodHNWeC9qczBBUUo3R2dFCjMxQUdRcmF4S2s0STVCN1hBY\ 1pybHdrS1ljaGFPZnBlNkV6Ly9HZXVFekR5VnN3a09uK2h0ZGNIMlJVSHozUlcKWWVTMGw2ZzZpO\ HcyUXRUakNwTUtId1FRTmQwSjdFM0k1aS9CRVA0azhXSHZIYjBkQk8ydVkyTml1cmNMWWw4dgpHO\ DUyZ2ZibWt2OENBd0VBQWFOQ01FQXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93U\ UZNQU1CCkFmOHdIUVlEVlIwT0JCWUVGQkg4VzVBbmxKYVNrVXowSzNobms1Vm55MVNnTUEwR0NTc\ UdTSWIzRFFFQkN3VUEKQTRJQkFRQmE0WmtsWmRINUFCOTNWaXhYOUhnMEYwYXdVZWduNkVSRGtRQ\ VBlcHZNaG5ON1lyVGlFN3lUSGxvWApLNS9ROTJ5Y2FnRGVlNjlEbHpvWEppTlNzdEZWYmtaSVN0O\ HVRZFhCYjFoSUtzbXBVYWlSeXFoRmVjbnRaSi85CmhCWmZMRjZnNitBNUlvVGxYOThqMERVU21IV\ is2Q29raXhPV3ZESmJ6dkI2S3VXdnhQbTF5WFgveVpBTDd1U1gKcUNnTC84UjJjSm53dUZhTnFvS\ 3I3STE5bDBRNi9VWWQ0bWhralpmUTdqdGlraEpmQXpWRUFtWlVza0hZSkRtdwp6bzJKMUJLL0Jxb\ m8rSFplbThFTExpK1ZhRXVlR280blF4ZmpaSlF2MWFXZHhCMnRrUWovdWNUa1QxRU1kUFBsCm0rd\ kh2MWtYNW5BaHl5eWR0dG12UGRlOWFHOUwKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=" users: - name: "kubemsb-smart-1" user: token: "kubeconfig-user-9cn9x.c-5jtsf:x57644qvmbfqpmh78fb4cbdnm8zbbxk9hmjb2bjggl5j2hvwnvj4c9" contexts: - name: "kubemsb-smart-1" context: user: "kubemsb-smart-1" cluster: "kubemsb-smart-1" - name: "kubemsb-smart-1-k8s-master01" context: user: "kubemsb-smart-1" cluster: "kubemsb-smart-1-k8s-master01" current-context: "kubemsb-smart-1" ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready controlplane,etcd,worker 35m v1.20.15 k8s-worker01 Ready worker 31m v1.20.15 k8s-worker02 Ready worker 27m v1.20.15 ~~~ # 七、通过Rancher部署Nginx应用 ![image-20220816155537732](../../img/kubernetes/kubernetes_rancher/image-20220816155537732.png) ![image-20220816155741848](../../img/kubernetes/kubernetes_rancher/image-20220816155741848.png) ![image-20220816155932470](../../img/kubernetes/kubernetes_rancher/image-20220816155932470.png) ![image-20220816160056177](../../img/kubernetes/kubernetes_rancher/image-20220816160056177.png) ![image-20220816160200746](../../img/kubernetes/kubernetes_rancher/image-20220816160200746.png) ![image-20220816160254842](../../img/kubernetes/kubernetes_rancher/image-20220816160254842.png) ![image-20220816160423255](../../img/kubernetes/kubernetes_rancher/image-20220816160423255.png) ![image-20220816160443666](../../img/kubernetes/kubernetes_rancher/image-20220816160443666.png) ![image-20220816160733605](../../img/kubernetes/kubernetes_rancher/image-20220816160733605.png) ![image-20220816160825424](../../img/kubernetes/kubernetes_rancher/image-20220816160825424.png) ~~~powershell [root@k8s-master01 ~]# kubectl get svc -n kubemsbf-1 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE myapp-1 ClusterIP 10.43.15.240 80/TCP 4m35s myapp-1-nodeport NodePort 10.43.214.118 80:32406/TCP 4m35s ~~~ ![image-20220816162243693](../../img/kubernetes/kubernetes_rancher/image-20220816162243693.png) # 八、通过Rancher部署mysql数据库 ## 8.1 持久化存储类准备 ### 8.1.1 NFS服务 ~~~powershell [root@nfsserver ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 99G 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 2G 0 lvm └─centos-home 253:2 0 47G 0 lvm /home sdb 8:16 0 100G 0 disk /sdb ~~~ ~~~powershell [root@nfsserver ~]# mkdir /sdb ~~~ ~~~powershell [root@nfsserver ~]# mkfs.xfs /dev/sdb ~~~ ~~~powershell [root@nfsserver ~]#vim /etc/fstab [root@nfsserver ~]# cat /etc/fstab ...... /dev/sdb /sdb xfs defaults 0 0 ~~~ ~~~powershell [root@nfsserver ~]# mount -a ~~~ ~~~powershell [root@nfsserver ~]# vim /etc/exports [root@nfsserver ~]# cat /etc/exports /sdb *(rw,sync,no_root_squash) ~~~ ~~~powershell [root@nfsserver ~]# systemctl enable --now nfs-server ~~~ ~~~powershell [root@nfsserver ~]# showmount -e Export list for nfs-server: /sdb * ~~~ ### 8.1.2 存储卷 ~~~powershell [root@k8s-master01 ~]# for file in class.yaml deployment.yaml rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/$file ; done ~~~ > 需要修改class.yaml中资源对象名称为nfs-client > > 需要修改deployment.yaml中nfs server及其共享的目录、容器对应的镜像。 ~~~powershell [root@k8s-master01 ~]# kubectl apply -f class.yaml ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl apply -f rbac.yaml ~~~ ~~~powershell [root@k8s-master01 nfsdir]# vim deployment.yaml [root@k8s-master01 nfsdir]# cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.10.133 - name: NFS_PATH value: /sdb volumes: - name: nfs-client-root nfs: server: 192.168.10.133 path: /sdb ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl apply -f deployment.yaml ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client fuseim.pri/ifs Delete Immediate false 109m ~~~ ![image-20220816185710948](../../img/kubernetes/kubernetes_rancher/image-20220816185710948.png) ## 8.2 MySQL数据库部署 ### 8.2.1 PVC准备 ![image-20220816202438365](../../img/kubernetes/kubernetes_rancher/image-20220816202438365.png) ![image-20220816202505925](../../img/kubernetes/kubernetes_rancher/image-20220816202505925.png) ![image-20220816202814643](../../img/kubernetes/kubernetes_rancher/image-20220816202814643.png) ![image-20220816202832741](../../img/kubernetes/kubernetes_rancher/image-20220816202832741.png) ![image-20220816202919580](../../img/kubernetes/kubernetes_rancher/image-20220816202919580.png) ![image-20220816202953810](../../img/kubernetes/kubernetes_rancher/image-20220816202953810.png) ![image-20220816203153592](../../img/kubernetes/kubernetes_rancher/image-20220816203153592.png) ![image-20220816203212769](../../img/kubernetes/kubernetes_rancher/image-20220816203212769.png) ~~~powershell [root@k8s-master01 ~]# kubectl get pvc -n kubemsbdata NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myvolume Bound pvc-52460d7f-db89-40ab-b09e-ab9d0cfcaa17 5Gi RWO nfs-client 80s ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-52460d7f-db89-40ab-b09e-ab9d0cfcaa17 5Gi RWO Delete Bound kubemsbdata/myvolume nfs-client 84s ~~~ ~~~powershell [root@nfsserver ~]# ls /sdb kubemsbdata-myvolume-pvc-52460d7f-db89-40ab-b09e-ab9d0cfcaa17 ~~~ ### 8.2.2 MySQL部署 ![image-20220816203527686](../../img/kubernetes/kubernetes_rancher/image-20220816203527686.png) ![image-20220816205034201](../../img/kubernetes/kubernetes_rancher/image-20220816205034201.png) ![image-20220816204138975](../../img/kubernetes/kubernetes_rancher/image-20220816204138975.png) ![image-20220816204242066](../../img/kubernetes/kubernetes_rancher/image-20220816204242066.png) ![image-20220816204323541](../../img/kubernetes/kubernetes_rancher/image-20220816204323541.png) ![image-20220816204356332](../../img/kubernetes/kubernetes_rancher/image-20220816204356332.png) ![image-20220816204448683](../../img/kubernetes/kubernetes_rancher/image-20220816204448683.png) ![image-20220816205248456](../../img/kubernetes/kubernetes_rancher/image-20220816205248456.png) ### 8.2.2 MySQL访问 #### 8.2.2.1 方案一 通过Rancher web界面访问 ![image-20220816201739371](../../img/kubernetes/kubernetes_rancher/image-20220816201739371.png) ![image-20220816201826018](../../img/kubernetes/kubernetes_rancher/image-20220816201826018.png) #### 8.2.2.2 方案二 通过主机访问 ~~~powershell [root@k8s-master01 ~]# ss -anput | grep ":32666" tcp LISTEN 0 128 *:32666 *:* users:(("kube-proxy",pid=7654,fd=3)) ~~~ ~~~powershell [root@k8s-master01 nfsdir]# mysql -h 192.168.10.131 -uroot -p123456 -P 32666 ...... MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | kubemsb | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.01 sec) ~~~ # 九、部署wordpress ![image-20220816210148864](../../img/kubernetes/kubernetes_rancher/image-20220816210148864.png) ![image-20220816210220210](../../img/kubernetes/kubernetes_rancher/image-20220816210220210.png) ![image-20220816210402733](../../img/kubernetes/kubernetes_rancher/image-20220816210402733.png) ![image-20220816210440574](../../img/kubernetes/kubernetes_rancher/image-20220816210440574.png) ![image-20220816211312775](../../img/kubernetes/kubernetes_rancher/image-20220816211312775.png) ![image-20220816211412434](../../img/kubernetes/kubernetes_rancher/image-20220816211412434.png) ~~~powershell [root@k8s-master01 ~]# dig -t a mysqldata1-0.mysqldata1.kubemsbdata.svc.cluster.local @10.43.0.10 ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> -t a mysqldata1-0.mysqldata1.kubemsbdata.svc.cluster.local @10.43.0.10 ;; global options: +cmd ;; Got answer: ;; WARNING: .local is reserved for Multicast DNS ;; You are currently testing what happens when an mDNS query is leaked to DNS ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63314 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;mysqldata1-0.mysqldata1.kubemsbdata.svc.cluster.local. IN A ;; ANSWER SECTION: mysqldata1-0.mysqldata1.kubemsbdata.svc.cluster.local. 5 IN A 10.42.1.4 ;; Query time: 0 msec ;; SERVER: 10.43.0.10#53(10.43.0.10) ;; WHEN: 二 8月 16 21:20:18 CST 2022 ;; MSG SIZE rcvd: 151 ~~~ ![image-20220816212308034](../../img/kubernetes/kubernetes_rancher/image-20220816212308034.png) ![image-20220816212409188](../../img/kubernetes/kubernetes_rancher/image-20220816212409188.png) ![image-20220816212543225](../../img/kubernetes/kubernetes_rancher/image-20220816212543225.png) ![image-20220816212701687](../../img/kubernetes/kubernetes_rancher/image-20220816212701687.png) ![image-20220816212725300](../../img/kubernetes/kubernetes_rancher/image-20220816212725300.png) ![image-20220816212928917](../../img/kubernetes/kubernetes_rancher/image-20220816212928917.png) ![image-20220816212956908](../../img/kubernetes/kubernetes_rancher/image-20220816212956908.png) ![image-20220816213049117](../../img/kubernetes/kubernetes_rancher/image-20220816213049117.png) ![image-20220816213120382](../../img/kubernetes/kubernetes_rancher/image-20220816213120382.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_rke.md ================================================ # 使用RKE构建企业生产级Kubernetes集群 # 一、RKE工具介绍 - RKE是一款经过CNCF认证的开源Kubernetes发行版，可以在Docker容器内运行。 - 它通过删除大部分主机依赖项，并为部署、升级和回滚提供一个稳定的路径，从而解决了Kubernetes最常见的安装复杂性问题。 - 借助RKE，Kubernetes可以完全独立于正在运行的操作系统和平台，轻松实现Kubernetes的自动化运维。 - 只要运行受支持的Docker版本，就可以通过RKE部署和运行Kubernetes。仅需几分钟，RKE便可通过单条命令构建一个集群，其声明式配置使Kubernetes升级操作具备原子性且安全。 # 二、集群主机准备 ## 2.1 集群主机配置要求 ### 2.1.1 部署集群环境说明 部署Kubernetes集群机器需要满足以下几个条件： 1）一台或多台机器，操作系统 CentOS7 2）硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘100GB或更多 3） 集群中所有机器之间网络互通 4）可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点 5）禁止swap分区 ### 2.1.2 软件环境 | 软件 | 版本 | | ---------- | -------- | | 操作系统 | CentOS7 | | docker-ce | 20.10.12 | | kubernetes | 1.22.5 | ### 2.1.3 集群主机名称、IP地址及角色规划 | 主机名称 | IP地址 | 角色 | | -------- | ------------- | -------------------------- | | master01 | 192.168.10.10 | controlplane、rancher、rke | | master02 | 192.168.10.11 | controlpane | | worker01 | 192.168.10.12 | worker | | worker02 | 192.168.10.13 | worker | | etcd01 | 192.168.10.14 | etcd | ## 2.2 集群主机名称配置 > 所有集群主机均要配置对应的主机名称即可。 ~~~powershell # hostnamectl set-hostname xxx ~~~ ~~~powershell 把xxx替换为对应的主机名 192.168.10.10 master01 192.168.10.11 master02 192.168.10.12 worker01 192.168.10.13 worker02 192.168.10.14 etcd01 ~~~ ## 2.3 集群主机IP地址配置 > 所有集群主机均要配置对应的主机IP地址即可。 ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" 修改为静态 DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" 添加如下内容： IPADDR="192.168.10.XXX" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ## 2.4 主机名与IP地址解析 > 所有主机均要配置。 ~~~powershell # vim /etc/hosts # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.10 master01 192.168.10.11 master02 192.168.10.12 worker01 192.168.10.13 worker02 192.168.10.14 etcd01 ~~~ ## 2.5 配置ip_forward及过滤机制 > 所有主机均要配置 >将桥接的IPv4流量传递到iptables的链 ~~~powershell # vim /etc/sysctl.conf # cat /etc/sysctl.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 ~~~ ~~~powershell # modprobe br_netfilter ~~~ ~~~powershell # sysctl -p /etc/sysctl.conf ~~~ ## 2.6 主机安全设置 > 所有主机均要设置 ### 2.6.1 防火墙 ~~~powershell # systemctl stop firewalld # systemctl disable firewalld ~~~ ~~~powershell # firewall-cmd --state ~~~ ### 2.6.2 selinux > 修改完成后一定要重启操作系统 ~~~powershell 永久关闭，一定要重启操作系统后生效。 sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ~~~powershell 临时关闭，不重启操作系统，即刻生效。 # setenforce 0 ~~~ ## 2.7 主机swap分区设置 > 所有主机均要配置 ~~~powershell 永久关闭，需要重启操作系统生效。 # sed -ri 's/.*swap.*/#&/' /etc/fstab ~~~ ~~~powershell # cat /etc/fstab ...... #/dev/mapper/centos_192-swap swap swap defaults 0 0 ~~~ ~~~powershell 临时关闭，不需要重启操作系统，即刻生效。 # swapoff -a ~~~ ## 2.8 时间同步 > 所有主机均要配置 ~~~powershell # yum -y insall ntpdate ~~~ ~~~powershell # crontab -e 0 */1 * * * ntpdate time1.aliyun.com ~~~ # 三、Docker部署 > 所有主机均要配置 ## 3.1 配置Docker YUM源 ![image-20220130193936921](../../img/kubernetes/kubernetes_rke/image-20220130193936921.png) ~~~powershell # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ ## 3.2 安装Docker CE ~~~powershell # yum -y install docker-ce ~~~ ## 3.3 启动Docker服务 ~~~powershell # systemctl enable docker # systemctl start docker ~~~ ## 3.4 配置Docker容器镜像加速器 ~~~powershell # vim /etc/docker/daemon.json # cat /etc/docker/daemon.json { "registry-mirrors": ["https://s27w6kze.mirror.aliyuncs.com"] } ~~~ # 四、docker compose安装 ~~~powershell # curl -L "https://github.com/docker/compose/releases/download/1.28.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose ~~~ ~~~powershell # chmod +x /usr/local/bin/docker-compose ~~~ ~~~powershell # ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose ~~~ ~~~powershell # docker-compose --version ~~~ # 五、添加rancher用户 >使用CentOS时，不能使用 root 账号，因此要添加专用的账号进行 docker相关 操作。 >所有集群主机均需要操作 ~~~powershell # useradd rancher ~~~ ~~~powershell # usermod -aG docker rancher ~~~ ~~~powershell # echo 123 | passwd --stdin rancher ~~~ # 六、生成ssh证书用于部署集群 > rke二进制文件安装主机上创建密钥，即为control主机，用于部署集群。 ## 6.1 生成ssh证书 ~~~powershell # ssh-keygen ~~~ ## 6.2 复制证书到集群中所有主机 ~~~powershell # ssh-copy-id rancher@master01 ~~~ ~~~powershell # ssh-copy-id rancher@master02 ~~~ ~~~powershell # ssh-copy-id rancher@worker01 ~~~ ~~~powershell # ssh-copy-id rancher@worker02 ~~~ ~~~powershell # ssh-copy-id rancher@etcd01 ~~~ ## 6.3 验证ssh证书是否可用 > 本次在master01上部署rke二进制文件。 > 在rke二进制文件安装主机机测试连接其它集群主机，验证是否可使用docker ps命令即可。 ~~~powershell # ssh rancher@主机名 ~~~ ~~~powershell 远程主机# docker ps ~~~ # 七、rke工具下载 > 本次在master01上部署rke二进制文件。 ![image-20220222170808926](../../img/kubernetes/kubernetes_rke/image-20220222170808926.png) ~~~powershell # wget https://github.com/rancher/rke/releases/download/v1.3.7/rke_linux-amd64 ~~~ ~~~powershell # mv rke_linux-amd64 /usr/local/bin/rke ~~~ ~~~powershell # chmod +x /usr/local/bin/rke ~~~ ~~~powershell # rke --version rke version v1.3.7 ~~~ # 八、初始化rke配置文件 ~~~powershell # mkdir -p /app/rancher ~~~ ~~~powershell # cd /app/rancher ~~~ ~~~powershell # rke config --name cluster.yml [+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: 集群私钥路径 [+] Number of Hosts [1]: 3 集群中有3个节点 [+] SSH Address of host (1) [none]: 192.168.10.10 第一个节点IP地址 [+] SSH Port of host (1) [22]: 22 第一个节点SSH访问端口 [+] SSH Private Key Path of host (192.168.10.10) [none]: ~/.ssh/id_rsa 第一个节点私钥路径 [+] SSH User of host (192.168.10.10) [ubuntu]: rancher 远程用户名 [+] Is host (192.168.10.10) a Control Plane host (y/n)? [y]: y 是否为k8s集群控制节点 [+] Is host (192.168.10.10) a Worker host (y/n)? [n]: n 不是worker节点 [+] Is host (192.168.10.10) an etcd host (y/n)? [n]: n 不是etcd节点 [+] Override Hostname of host (192.168.10.10) [none]: 不覆盖现有主机名 [+] Internal IP of host (192.168.10.10) [none]: 主机局域网IP地址 [+] Docker socket path on host (192.168.10.10) [/var/run/docker.sock]: 主机上docker.sock路径 [+] SSH Address of host (2) [none]: 192.168.10.12 第二个节点 [+] SSH Port of host (2) [22]: 22 远程端口 [+] SSH Private Key Path of host (192.168.10.12) [none]: ~/.ssh/id_rsa 私钥路径 [+] SSH User of host (192.168.10.12) [ubuntu]: rancher 远程访问用户 [+] Is host (192.168.10.12) a Control Plane host (y/n)? [y]: n 不是控制节点 [+] Is host (192.168.10.12) a Worker host (y/n)? [n]: y 是worker节点 [+] Is host (192.168.10.12) an etcd host (y/n)? [n]: n 不是etcd节点 [+] Override Hostname of host (192.168.10.12) [none]: 不覆盖现有主机名 [+] Internal IP of host (192.168.10.12) [none]: 主机局域网IP地址 [+] Docker socket path on host (192.168.10.12) [/var/run/docker.sock]: 主机上docker.sock路径 [+] SSH Address of host (3) [none]: 192.168.10.14 第三个节点 [+] SSH Port of host (3) [22]: 22 远程端口 [+] SSH Private Key Path of host (192.168.10.14) [none]: ~/.ssh/id_rsa 私钥路径 [+] SSH User of host (192.168.10.14) [ubuntu]: rancher 远程访问用户 [+] Is host (192.168.10.14) a Control Plane host (y/n)? [y]: n 不是控制节点 [+] Is host (192.168.10.14) a Worker host (y/n)? [n]: n 不是worker节点 [+] Is host (192.168.10.14) an etcd host (y/n)? [n]: y 是etcd节点 [+] Override Hostname of host (192.168.10.14) [none]: 不覆盖现有主机名 [+] Internal IP of host (192.168.10.14) [none]: 主机局域网IP地址 [+] Docker socket path on host (192.168.10.14) [/var/run/docker.sock]: 主机上docker.sock路径 [+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: 使用的网络插件 [+] Authentication Strategy [x509]: 认证策略 [+] Authorization Mode (rbac, none) [rbac]: 认证模式 [+] Kubernetes Docker image [rancher/hyperkube:v1.21.9-rancher1]: 集群容器镜像 [+] Cluster domain [cluster.local]: 集群域名 [+] Service Cluster IP Range [10.43.0.0/16]: 集群中Servic IP地址范围 [+] Enable PodSecurityPolicy [n]: 是否开启Pod安装策略 [+] Cluster Network CIDR [10.42.0.0/16]: 集群Pod网络 [+] Cluster DNS Service IP [10.43.0.10]: 集群DNS Service IP地址 [+] Add addon manifest URLs or YAML files [no]: 是否增加插件manifest URL或配置文件 ~~~ ~~~powershell [root@master01 rancher]# ls cluster.yml ~~~ > 在cluster.yaml文件中 > > ~~~powershell > kube-controller: > image: "" > extra_args: > # 如果后面需要部署kubeflow或istio则一定要配置以下参数 > cluster-signing-cert-file: "/etc/kubernetes/ssl/kube-ca.pem" > cluster-signing-key-file: "/etc/kubernetes/ssl/kube-ca-key.pem" > ~~~ > > # 九、集群部署 ~~~powershell # pwd /app/rancher ~~~ ~~~powershell # rke up ~~~ ~~~powershell 输出： INFO[0000] Running RKE version: v1.3.7 INFO[0000] Initiating Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [192.168.10.14] INFO[0000] [dialer] Setup tunnel for host [192.168.10.10] INFO[0000] [dialer] Setup tunnel for host [192.168.10.12] INFO[0000] Checking if container [cluster-state-deployer] is running on host [192.168.10.14], try #1 INFO[0000] Checking if container [cluster-state-deployer] is running on host [192.168.10.10], try #1 INFO[0000] Checking if container [cluster-state-deployer] is running on host [192.168.10.12], try #1 INFO[0000] [certificates] Generating CA kubernetes certificates INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating Kubernetes API server certificates INFO[0000] [certificates] Generating Service account token key INFO[0000] [certificates] Generating Kube Controller certificates INFO[0000] [certificates] Generating Kube Scheduler certificates INFO[0000] [certificates] Generating Kube Proxy certificates INFO[0001] [certificates] Generating Node certificate INFO[0001] [certificates] Generating admin certificates and kubeconfig INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates INFO[0001] [certificates] Generating kube-etcd-192-168-10-14 certificate and key INFO[0001] Successfully Deployed state file at [./cluster.rkestate] INFO[0001] Building Kubernetes cluster INFO[0001] [dialer] Setup tunnel for host [192.168.10.12] INFO[0001] [dialer] Setup tunnel for host [192.168.10.14] INFO[0001] [dialer] Setup tunnel for host [192.168.10.10] INFO[0001] [network] Deploying port listener containers INFO[0001] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0001] Starting container [rke-etcd-port-listener] on host [192.168.10.14], try #1 INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [192.168.10.14] INFO[0001] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0001] Starting container [rke-cp-port-listener] on host [192.168.10.10], try #1 INFO[0002] [network] Successfully started [rke-cp-port-listener] container on host [192.168.10.10] INFO[0002] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0002] Starting container [rke-worker-port-listener] on host [192.168.10.12], try #1 INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [192.168.10.12] INFO[0002] [network] Port listener containers deployed successfully INFO[0002] [network] Running control plane -> etcd port checks INFO[0002] [network] Checking if host [192.168.10.10] can connect to host(s) [192.168.10.14] on port(s) [2379], try #1 INFO[0002] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0002] Starting container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0002] [network] Successfully started [rke-port-checker] container on host [192.168.10.10] INFO[0002] Removing container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0002] [network] Running control plane -> worker port checks INFO[0002] [network] Checking if host [192.168.10.10] can connect to host(s) [192.168.10.12] on port(s) [10250], try #1 INFO[0002] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0003] Starting container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0003] [network] Successfully started [rke-port-checker] container on host [192.168.10.10] INFO[0003] Removing container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0003] [network] Running workers -> control plane port checks INFO[0003] [network] Checking if host [192.168.10.12] can connect to host(s) [192.168.10.10] on port(s) [6443], try #1 INFO[0003] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0003] Starting container [rke-port-checker] on host [192.168.10.12], try #1 INFO[0003] [network] Successfully started [rke-port-checker] container on host [192.168.10.12] INFO[0003] Removing container [rke-port-checker] on host [192.168.10.12], try #1 INFO[0003] [network] Checking KubeAPI port Control Plane hosts INFO[0003] [network] Removing port listener containers INFO[0003] Removing container [rke-etcd-port-listener] on host [192.168.10.14], try #1 INFO[0003] [remove/rke-etcd-port-listener] Successfully removed container on host [192.168.10.14] INFO[0003] Removing container [rke-cp-port-listener] on host [192.168.10.10], try #1 INFO[0003] [remove/rke-cp-port-listener] Successfully removed container on host [192.168.10.10] INFO[0003] Removing container [rke-worker-port-listener] on host [192.168.10.12], try #1 INFO[0003] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.10.12] INFO[0003] [network] Port listener containers removed successfully INFO[0003] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0003] Checking if container [cert-deployer] is running on host [192.168.10.14], try #1 INFO[0003] Checking if container [cert-deployer] is running on host [192.168.10.10], try #1 INFO[0003] Checking if container [cert-deployer] is running on host [192.168.10.12], try #1 INFO[0003] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0003] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0003] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0004] Starting container [cert-deployer] on host [192.168.10.14], try #1 INFO[0004] Starting container [cert-deployer] on host [192.168.10.12], try #1 INFO[0004] Starting container [cert-deployer] on host [192.168.10.10], try #1 INFO[0004] Checking if container [cert-deployer] is running on host [192.168.10.14], try #1 INFO[0004] Checking if container [cert-deployer] is running on host [192.168.10.12], try #1 INFO[0004] Checking if container [cert-deployer] is running on host [192.168.10.10], try #1 INFO[0009] Checking if container [cert-deployer] is running on host [192.168.10.14], try #1 INFO[0009] Removing container [cert-deployer] on host [192.168.10.14], try #1 INFO[0009] Checking if container [cert-deployer] is running on host [192.168.10.12], try #1 INFO[0009] Removing container [cert-deployer] on host [192.168.10.12], try #1 INFO[0009] Checking if container [cert-deployer] is running on host [192.168.10.10], try #1 INFO[0009] Removing container [cert-deployer] on host [192.168.10.10], try #1 INFO[0009] [reconcile] Rebuilding and updating local kube config INFO[0009] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] WARN[0009] [reconcile] host [192.168.10.10] is a control plane node without reachable Kubernetes API endpoint in the cluster WARN[0009] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found INFO[0009] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0009] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.10.10] INFO[0009] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0009] Starting container [file-deployer] on host [192.168.10.10], try #1 INFO[0009] Successfully started [file-deployer] container on host [192.168.10.10] INFO[0009] Waiting for [file-deployer] container to exit on host [192.168.10.10] INFO[0009] Waiting for [file-deployer] container to exit on host [192.168.10.10] INFO[0009] Container [file-deployer] is still running on host [192.168.10.10]: stderr: [], stdout: [] INFO[0010] Waiting for [file-deployer] container to exit on host [192.168.10.10] INFO[0010] Removing container [file-deployer] on host [192.168.10.10], try #1 INFO[0010] [remove/file-deployer] Successfully removed container on host [192.168.10.10] INFO[0010] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0010] [reconcile] Reconciling cluster state INFO[0010] [reconcile] This is newly generated cluster INFO[0010] Pre-pulling kubernetes images INFO[0010] Pulling image [rancher/hyperkube:v1.21.9-rancher1] on host [192.168.10.10], try #1 INFO[0010] Pulling image [rancher/hyperkube:v1.21.9-rancher1] on host [192.168.10.14], try #1 INFO[0010] Pulling image [rancher/hyperkube:v1.21.9-rancher1] on host [192.168.10.12], try #1 INFO[0087] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0090] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.12] INFO[0092] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.14] INFO[0092] Kubernetes images pulled successfully INFO[0092] [etcd] Building up etcd plane.. INFO[0092] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0092] Starting container [etcd-fix-perm] on host [192.168.10.14], try #1 INFO[0092] Successfully started [etcd-fix-perm] container on host [192.168.10.14] INFO[0092] Waiting for [etcd-fix-perm] container to exit on host [192.168.10.14] INFO[0092] Waiting for [etcd-fix-perm] container to exit on host [192.168.10.14] INFO[0092] Container [etcd-fix-perm] is still running on host [192.168.10.14]: stderr: [], stdout: [] INFO[0093] Waiting for [etcd-fix-perm] container to exit on host [192.168.10.14] INFO[0093] Removing container [etcd-fix-perm] on host [192.168.10.14], try #1 INFO[0093] [remove/etcd-fix-perm] Successfully removed container on host [192.168.10.14] INFO[0093] Image [rancher/mirrored-coreos-etcd:v3.5.0] exists on host [192.168.10.14] INFO[0093] Starting container [etcd] on host [192.168.10.14], try #1 INFO[0093] [etcd] Successfully started [etcd] container on host [192.168.10.14] INFO[0093] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [192.168.10.14] INFO[0093] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0094] Starting container [etcd-rolling-snapshots] on host [192.168.10.14], try #1 INFO[0094] [etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.10.14] INFO[0099] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0099] Starting container [rke-bundle-cert] on host [192.168.10.14], try #1 INFO[0099] [certificates] Successfully started [rke-bundle-cert] container on host [192.168.10.14] INFO[0099] Waiting for [rke-bundle-cert] container to exit on host [192.168.10.14] INFO[0099] Container [rke-bundle-cert] is still running on host [192.168.10.14]: stderr: [], stdout: [] INFO[0100] Waiting for [rke-bundle-cert] container to exit on host [192.168.10.14] INFO[0100] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.10.14] INFO[0100] Removing container [rke-bundle-cert] on host [192.168.10.14], try #1 INFO[0100] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0100] Starting container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0100] [etcd] Successfully started [rke-log-linker] container on host [192.168.10.14] INFO[0100] Removing container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0100] [remove/rke-log-linker] Successfully removed container on host [192.168.10.14] INFO[0100] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0101] Starting container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0101] [etcd] Successfully started [rke-log-linker] container on host [192.168.10.14] INFO[0101] Removing container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0101] [remove/rke-log-linker] Successfully removed container on host [192.168.10.14] INFO[0101] [etcd] Successfully started etcd plane.. Checking etcd cluster health INFO[0101] [etcd] etcd host [192.168.10.14] reported healthy=true INFO[0101] [controlplane] Building up Controller Plane.. INFO[0101] Checking if container [service-sidekick] is running on host [192.168.10.10], try #1 INFO[0101] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0101] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0101] Starting container [kube-apiserver] on host [192.168.10.10], try #1 INFO[0101] [controlplane] Successfully started [kube-apiserver] container on host [192.168.10.10] INFO[0101] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.10.10] INFO[0106] [healthcheck] service [kube-apiserver] on host [192.168.10.10] is healthy INFO[0106] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0107] Starting container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0107] [controlplane] Successfully started [rke-log-linker] container on host [192.168.10.10] INFO[0107] Removing container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0107] [remove/rke-log-linker] Successfully removed container on host [192.168.10.10] INFO[0107] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0107] Starting container [kube-controller-manager] on host [192.168.10.10], try #1 INFO[0107] [controlplane] Successfully started [kube-controller-manager] container on host [192.168.10.10] INFO[0107] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.10.10] INFO[0112] [healthcheck] service [kube-controller-manager] on host [192.168.10.10] is healthy INFO[0112] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0113] Starting container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0113] [controlplane] Successfully started [rke-log-linker] container on host [192.168.10.10] INFO[0113] Removing container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0113] [remove/rke-log-linker] Successfully removed container on host [192.168.10.10] INFO[0113] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0113] Starting container [kube-scheduler] on host [192.168.10.10], try #1 INFO[0113] [controlplane] Successfully started [kube-scheduler] container on host [192.168.10.10] INFO[0113] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.10.10] INFO[0118] [healthcheck] service [kube-scheduler] on host [192.168.10.10] is healthy INFO[0118] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0119] Starting container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0119] [controlplane] Successfully started [rke-log-linker] container on host [192.168.10.10] INFO[0119] Removing container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0119] [remove/rke-log-linker] Successfully removed container on host [192.168.10.10] INFO[0119] [controlplane] Successfully started Controller Plane.. INFO[0119] [authz] Creating rke-job-deployer ServiceAccount INFO[0119] [authz] rke-job-deployer ServiceAccount created successfully INFO[0119] [authz] Creating system:node ClusterRoleBinding INFO[0119] [authz] system:node ClusterRoleBinding created successfully INFO[0119] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding INFO[0119] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully INFO[0119] Successfully Deployed state file at [./cluster.rkestate] INFO[0119] [state] Saving full cluster state to Kubernetes INFO[0119] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state INFO[0119] [worker] Building up Worker Plane.. INFO[0119] Checking if container [service-sidekick] is running on host [192.168.10.10], try #1 INFO[0119] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0119] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0119] [sidekick] Sidekick container already created on host [192.168.10.10] INFO[0119] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0119] Starting container [kubelet] on host [192.168.10.10], try #1 INFO[0119] [worker] Successfully started [kubelet] container on host [192.168.10.10] INFO[0119] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.10.10] INFO[0119] Starting container [nginx-proxy] on host [192.168.10.14], try #1 INFO[0119] [worker] Successfully started [nginx-proxy] container on host [192.168.10.14] INFO[0119] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0119] Starting container [nginx-proxy] on host [192.168.10.12], try #1 INFO[0119] [worker] Successfully started [nginx-proxy] container on host [192.168.10.12] INFO[0119] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0119] Starting container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0120] Starting container [rke-log-linker] on host [192.168.10.12], try #1 INFO[0120] [worker] Successfully started [rke-log-linker] container on host [192.168.10.14] INFO[0120] Removing container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0120] [remove/rke-log-linker] Successfully removed container on host [192.168.10.14] INFO[0120] Checking if container [service-sidekick] is running on host [192.168.10.14], try #1 INFO[0120] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0120] [worker] Successfully started [rke-log-linker] container on host [192.168.10.12] INFO[0120] Removing container [rke-log-linker] on host [192.168.10.12], try #1 INFO[0120] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.14] INFO[0120] [remove/rke-log-linker] Successfully removed container on host [192.168.10.12] INFO[0120] Checking if container [service-sidekick] is running on host [192.168.10.12], try #1 INFO[0120] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0120] Starting container [kubelet] on host [192.168.10.14], try #1 INFO[0120] [worker] Successfully started [kubelet] container on host [192.168.10.14] INFO[0120] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.10.14] INFO[0120] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.12] INFO[0120] Starting container [kubelet] on host [192.168.10.12], try #1 INFO[0120] [worker] Successfully started [kubelet] container on host [192.168.10.12] INFO[0120] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.10.12] INFO[0124] [healthcheck] service [kubelet] on host [192.168.10.10] is healthy INFO[0124] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0124] Starting container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0125] [worker] Successfully started [rke-log-linker] container on host [192.168.10.10] INFO[0125] Removing container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0125] [remove/rke-log-linker] Successfully removed container on host [192.168.10.10] INFO[0125] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0125] Starting container [kube-proxy] on host [192.168.10.10], try #1 INFO[0125] [worker] Successfully started [kube-proxy] container on host [192.168.10.10] INFO[0125] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.10.10] INFO[0125] [healthcheck] service [kubelet] on host [192.168.10.14] is healthy INFO[0125] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0125] Starting container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0125] [healthcheck] service [kubelet] on host [192.168.10.12] is healthy INFO[0125] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0125] [worker] Successfully started [rke-log-linker] container on host [192.168.10.14] INFO[0125] Starting container [rke-log-linker] on host [192.168.10.12], try #1 INFO[0125] Removing container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0126] [remove/rke-log-linker] Successfully removed container on host [192.168.10.14] INFO[0126] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.14] INFO[0126] Starting container [kube-proxy] on host [192.168.10.14], try #1 INFO[0126] [worker] Successfully started [rke-log-linker] container on host [192.168.10.12] INFO[0126] Removing container [rke-log-linker] on host [192.168.10.12], try #1 INFO[0126] [worker] Successfully started [kube-proxy] container on host [192.168.10.14] INFO[0126] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.10.14] INFO[0126] [remove/rke-log-linker] Successfully removed container on host [192.168.10.12] INFO[0126] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.12] INFO[0126] Starting container [kube-proxy] on host [192.168.10.12], try #1 INFO[0126] [worker] Successfully started [kube-proxy] container on host [192.168.10.12] INFO[0126] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.10.12] INFO[0130] [healthcheck] service [kube-proxy] on host [192.168.10.10] is healthy INFO[0130] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0130] Starting container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0130] [worker] Successfully started [rke-log-linker] container on host [192.168.10.10] INFO[0130] Removing container [rke-log-linker] on host [192.168.10.10], try #1 INFO[0130] [remove/rke-log-linker] Successfully removed container on host [192.168.10.10] INFO[0131] [healthcheck] service [kube-proxy] on host [192.168.10.14] is healthy INFO[0131] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0131] Starting container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0131] [healthcheck] service [kube-proxy] on host [192.168.10.12] is healthy INFO[0131] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0131] [worker] Successfully started [rke-log-linker] container on host [192.168.10.14] INFO[0131] Removing container [rke-log-linker] on host [192.168.10.14], try #1 INFO[0131] Starting container [rke-log-linker] on host [192.168.10.12], try #1 INFO[0131] [remove/rke-log-linker] Successfully removed container on host [192.168.10.14] INFO[0131] [worker] Successfully started [rke-log-linker] container on host [192.168.10.12] INFO[0131] Removing container [rke-log-linker] on host [192.168.10.12], try #1 INFO[0131] [remove/rke-log-linker] Successfully removed container on host [192.168.10.12] INFO[0131] [worker] Successfully started Worker Plane.. INFO[0131] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0131] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0131] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0132] Starting container [rke-log-cleaner] on host [192.168.10.14], try #1 INFO[0132] Starting container [rke-log-cleaner] on host [192.168.10.12], try #1 INFO[0132] Starting container [rke-log-cleaner] on host [192.168.10.10], try #1 INFO[0132] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.14] INFO[0132] Removing container [rke-log-cleaner] on host [192.168.10.14], try #1 INFO[0132] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.12] INFO[0132] Removing container [rke-log-cleaner] on host [192.168.10.12], try #1 INFO[0132] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.10] INFO[0132] Removing container [rke-log-cleaner] on host [192.168.10.10], try #1 INFO[0132] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.14] INFO[0132] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.12] INFO[0132] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.10] INFO[0132] [sync] Syncing nodes Labels and Taints INFO[0132] [sync] Successfully synced nodes Labels and Taints INFO[0132] [network] Setting up network plugin: canal INFO[0132] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes INFO[0132] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes INFO[0132] [addons] Executing deploy job rke-network-plugin INFO[0137] [addons] Setting up coredns INFO[0137] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0137] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0137] [addons] Executing deploy job rke-coredns-addon INFO[0142] [addons] CoreDNS deployed successfully INFO[0142] [dns] DNS provider coredns deployed successfully INFO[0142] [addons] Setting up Metrics Server INFO[0142] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0142] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0142] [addons] Executing deploy job rke-metrics-addon INFO[0147] [addons] Metrics Server deployed successfully INFO[0147] [ingress] Setting up nginx ingress controller INFO[0147] [ingress] removing admission batch jobs if they exist INFO[0147] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0147] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0147] [addons] Executing deploy job rke-ingress-controller INFO[0152] [ingress] removing default backend service and deployment if they exist INFO[0152] [ingress] ingress controller nginx deployed successfully INFO[0152] [addons] Setting up user addons INFO[0152] [addons] no user addons defined INFO[0152] Finished building Kubernetes cluster successfully ~~~ # 十、安装kubectl客户端 > 在master01主机上操作。 ## 10.1 kubectl客户端安装 ~~~powershell # wget https://storage.googleapis.com/kubernetes-release/release/v1.21.9/bin/linux/amd64/kubectl ~~~ ~~~powershell # chmod +x kubectl ~~~ ~~~powershell # mv kubectl /usr/local/bin/kubectl ~~~ ~~~powershell # kubectl version --client Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.9", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:33:06Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"} ~~~ ## 10.2 kubectl客户端配置集群管理文件及应用验证 ~~~powershell [root@master01 ~]# ls /app/rancher/ cluster.rkestate cluster.yml kube_config_cluster.yml ~~~ ~~~powershell [root@master01 ~]# mkdir ./.kube ~~~ ~~~powershell [root@master01 ~]# cp /app/rancher/kube_config_cluster.yml /root/.kube/config ~~~ ~~~powershell [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.10.10 Ready controlplane 9m13s v1.21.9 192.168.10.12 Ready worker 9m12s v1.21.9 192.168.10.14 Ready etcd 9m12s v1.21.9 ~~~ ~~~powershell [root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5685fbd9f7-gcwj7 1/1 Running 0 9m36s canal-fz2bg 2/2 Running 0 9m36s canal-qzw4n 2/2 Running 0 9m36s canal-sstjn 2/2 Running 0 9m36s coredns-8578b6dbdd-ftnf6 1/1 Running 0 9m30s coredns-autoscaler-f7b68ccb7-fzdgc 1/1 Running 0 9m30s metrics-server-6bc7854fb5-kwppz 1/1 Running 0 9m25s rke-coredns-addon-deploy-job--1-x56w2 0/1 Completed 0 9m31s rke-ingress-controller-deploy-job--1-wzp2b 0/1 Completed 0 9m21s rke-metrics-addon-deploy-job--1-ltlgn 0/1 Completed 0 9m26s rke-network-plugin-deploy-job--1-nsbfn 0/1 Completed 0 9m41s ~~~ # 十一、集群web管理 rancher > 在master01运行 rancher控制面板主要方便用于控制k8s集群，查看集群状态，编辑集群等。 ## 11.1 使用docker run启动一个rancher ~~~powershell [root@master01 ~]# docker run -d --restart=unless-stopped --privileged --name rancher -p 80:80 -p 443:443 rancher/rancher:v2.5.9 ~~~ ~~~powershell [root@master01 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0fd46ee77655 rancher/rancher:v2.5.9 "entrypoint.sh" 5 seconds ago Up 3 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp rancher ~~~ ## 11.2 访问rancher ~~~powershell [root@master01 ~]# ss -anput | grep ":80" tcp LISTEN 0 128 *:80 *:* users:(("docker-proxy",pid=29564,fd=4)) tcp LISTEN 0 128 [::]:80 [::]:* users:(("docker-proxy",pid=29570,fd=4)) ~~~ ![image-20220222120525765](../../img/kubernetes/kubernetes_rke/image-20220222120525765.png) ![image-20220222120615706](../../img/kubernetes/kubernetes_rke/image-20220222120615706.png) ![image-20220222120657384](../../img/kubernetes/kubernetes_rke/image-20220222120657384.png) ![image-20220222121036160](../../img/kubernetes/kubernetes_rke/image-20220222121036160.png) ![image-20220222121132181](../../img/kubernetes/kubernetes_rke/image-20220222121132181.png) ## 11.3 在rancher web界面添加kubernetes集群 ![image-20220222121419801](../../img/kubernetes/kubernetes_rke/image-20220222121419801.png) ![image-20220222121500369](../../img/kubernetes/kubernetes_rke/image-20220222121500369.png) ![image-20220222121639207](../../img/kubernetes/kubernetes_rke/image-20220222121639207.png) ![image-20220222121800708](../../img/kubernetes/kubernetes_rke/image-20220222121800708.png) ~~~powershell 使用第一条报错： [root@master01 ~]# kubectl apply -f https://192.168.10.10/v3/import/vljtg5srnznpzts662q6ncs4jm6f8kd847xqs97d6fbs5rhn7kfzvk_c-ktwhn.yaml Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 172.17.0.2, not 192.168.10.10 ~~~ ~~~powershell 使用第二条： 第一次报错： [root@master01 ~]# curl --insecure -sfL https://192.168.10.10/v3/import/vljtg5srnznpzts662q6ncs4jm6f8kd847xqs97d6fbs5rhn7kfzvk_c-ktwhn.yaml | kubectl apply -f - error: no objects passed to apply 第二次成功： [root@master01 ~]# curl --insecure -sfL https://192.168.10.10/v3/import/vljtg5srnznpzts662q6ncs4jm6f8kd847xqs97d6fbs5rhn7kfzvk_c-ktwhn.yaml | kubectl apply -f - Warning: resource clusterroles/proxy-clusterrole-kubeapiserver is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver configured Warning: resource clusterrolebindings/proxy-role-binding-kubernetes-master is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master configured namespace/cattle-system created serviceaccount/cattle created clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding created secret/cattle-credentials-0619853 created clusterrole.rbac.authorization.k8s.io/cattle-admin created Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead deployment.apps/cattle-cluster-agent created ~~~ ![image-20220222124826542](../../img/kubernetes/kubernetes_rke/image-20220222124826542.png) ![image-20220222124844668](../../img/kubernetes/kubernetes_rke/image-20220222124844668.png) # 十二、集群节点更新 ## 12.1 增加worker节点 RKE 支持为 worker 和 controlplane 主机添加或删除节点。 可以通过修改cluster.yml文件的内容，添加额外的节点，并指定它们在 Kubernetes 集群中的角色；或从cluster.yml中的节点列表中删除节点信息，以达到删除节点的目的。 通过运行rke up --update-only，您可以运行rke up --update-only命令，只添加或删除工作节点。这将会忽略除了cluster.yml中的工作节点以外的其他内容。 > 使用--update-only添加或删除 worker 节点时，可能会触发插件或其他组件的重新部署或更新。 > 添加一台节点环境也需要一致。安装docker, 创建用户，关闭swap等 ### 12.1.1 集群主机名称配置 ~~~powershell # hostnamectl set-hostname xxx ~~~ ### 12.1.2 集群主机IP地址配置 ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" 修改为静态 DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth0" DEVICE="eth0" ONBOOT="yes" 添加如下内容： IPADDR="192.168.10.XXX" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ### 12.1.3 主机名与IP地址解析 ~~~powershell # vim /etc/hosts # cat /etc/hosts ...... 192.168.10.10 master01 192.168.10.11 master02 192.168.10.12 worker01 192.168.10.13 worker02 192.168.10.14 etcd01 ~~~ ### 12.1.4 配置ip_forward及过滤机制 ~~~powershell # vim /etc/sysctl.conf # cat /etc/sysctl.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 ~~~ ~~~powershell # modprobe br_netfilter ~~~ ~~~powershell # sysctl -p /etc/sysctl.conf ~~~ ### 12.1.5 主机安全设置 #### 12.1.5.1 防火墙 ~~~powershell # systemctl stop firewalld # systemctl disable firewalld ~~~ ~~~powershell # firewall-cmd --state ~~~ #### 12.1.5.2 selinux > 修改完成后一定要重启操作系统 ~~~powershell 永久关闭，一定要重启操作系统后生效。 sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ~~~powershell 临时关闭，不重启操作系统，即刻生效。 # setenforce 0 ~~~ ### 12.1.6 主机swap分区设置 ~~~powershell 永久关闭，需要重启操作系统生效。 # sed -ri 's/.*swap.*/#&/' /etc/fstab ~~~ ~~~powershell # cat /etc/fstab ...... #/dev/mapper/centos_192-swap swap swap defaults 0 0 ~~~ ~~~powershell 临时关闭，不需要重启操作系统，即刻生效。 # swapoff -a ~~~ ### 12.1.7 时间同步 ~~~powershell # yum -y insall ntpdate ~~~ ~~~powershell # crontab -e 0 */1 * * * ntpdate time1.aliyun.com ~~~ ### 12.1.8 Docker部署 #### 12.1.8.1 配置Docker YUM源 ~~~powershell # wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ~~~ #### 12.1.8.2 安装Docker CE ~~~powershell # yum -y install docker-ce ~~~ #### 12.1.8.3 启动Docker服务 ~~~powershell # systemctl enable docker # systemctl start docker ~~~ #### 12.1.8.4 配置Docker容器镜像加速器 ~~~powershell # vim /etc/docker/daemon.json # cat /etc/docker/daemon.json { "registry-mirrors": ["https://s27w6kze.mirror.aliyuncs.com"] } ~~~ ### 12.1.9 docker compose安装 ~~~powershell # curl -L "https://github.com/docker/compose/releases/download/1.28.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose ~~~ ~~~powershell # chmod +x /usr/local/bin/docker-compose ~~~ ~~~powershell # ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose ~~~ ~~~powershell # docker-compose --version ~~~ ### 12.1.10 添加rancher用户 >使用CentOS时，不能使用 root 账号，因此要添加专用的账号进行 docker相关 操作。重启系统以后才能生效，只重启Docker服务是不行的！重启后，rancher用户也可以直接使用docker ps命令 ~~~powershell # useradd rancher ~~~ ~~~powershell # usermod -aG docker rancher ~~~ ~~~powershell # echo 123 | passwd --stdin rancher ~~~ ### 12.1.11 复制ssh证书 > 从rke二进制文件安装主机上复制，如果已经复制，则可不需要重复性复制。 #### 12.1.11.1 复制证书 ~~~powershell # ssh-copy-id rancher@worker02 ~~~ #### 12.1.11.2 验证ssh证书是否可用 > 在rke二进制文件安装主机机测试连接其它集群主机，验证是否可使用docker ps命令即可。 ~~~powershell # ssh rancher@worker02 ~~~ ~~~powershell 远程主机# docker ps ~~~ ### 12.1.12 编辑cluster.yml文件 > 在文件中添加worker节点信息 ~~~powershell # vim cluster.yml ...... - address: 192.168.10.13 port: "22" internal_address: "" role: - worker hostname_override: user: "rancher" docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] ...... ~~~ ~~~powershell # rke up --update-only ~~~ ~~~powershell 输出 INFO[0000] Running RKE version: v1.3.7 INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating admin certificates and kubeconfig INFO[0000] Successfully Deployed state file at [./cluster.rkestate] INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [192.168.10.13] INFO[0000] [dialer] Setup tunnel for host [192.168.10.10] INFO[0000] [dialer] Setup tunnel for host [192.168.10.14] INFO[0000] [dialer] Setup tunnel for host [192.168.10.12] INFO[0000] [network] Deploying port listener containers INFO[0000] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0000] Starting container [rke-etcd-port-listener] on host [192.168.10.14], try #1 INFO[0000] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0000] Starting container [rke-cp-port-listener] on host [192.168.10.10], try #1 INFO[0000] Pulling image [rancher/rke-tools:v0.1.78] on host [192.168.10.13], try #1 INFO[0000] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0001] Starting container [rke-worker-port-listener] on host [192.168.10.12], try #1 INFO[0031] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0032] Starting container [rke-worker-port-listener] on host [192.168.10.13], try #1 INFO[0033] [network] Successfully started [rke-worker-port-listener] container on host [192.168.10.13] INFO[0033] [network] Port listener containers deployed successfully INFO[0033] [network] Running control plane -> etcd port checks INFO[0033] [network] Checking if host [192.168.10.10] can connect to host(s) [192.168.10.14] on port(s) [2379], try #1 INFO[0033] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0033] Starting container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0033] [network] Successfully started [rke-port-checker] container on host [192.168.10.10] INFO[0033] Removing container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0033] [network] Running control plane -> worker port checks INFO[0033] [network] Checking if host [192.168.10.10] can connect to host(s) [192.168.10.12 192.168.10.13] on port(s) [10250], try #1 INFO[0033] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0033] Starting container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0033] [network] Successfully started [rke-port-checker] container on host [192.168.10.10] INFO[0033] Removing container [rke-port-checker] on host [192.168.10.10], try #1 INFO[0033] [network] Running workers -> control plane port checks INFO[0033] [network] Checking if host [192.168.10.12] can connect to host(s) [192.168.10.10] on port(s) [6443], try #1 INFO[0033] [network] Checking if host [192.168.10.13] can connect to host(s) [192.168.10.10] on port(s) [6443], try #1 INFO[0033] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0033] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0034] Starting container [rke-port-checker] on host [192.168.10.13], try #1 INFO[0034] Starting container [rke-port-checker] on host [192.168.10.12], try #1 INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.10.12] INFO[0034] Removing container [rke-port-checker] on host [192.168.10.12], try #1 INFO[0034] [network] Successfully started [rke-port-checker] container on host [192.168.10.13] INFO[0034] Removing container [rke-port-checker] on host [192.168.10.13], try #1 INFO[0034] [network] Checking KubeAPI port Control Plane hosts INFO[0034] [network] Removing port listener containers INFO[0034] Removing container [rke-etcd-port-listener] on host [192.168.10.14], try #1 INFO[0034] [remove/rke-etcd-port-listener] Successfully removed container on host [192.168.10.14] INFO[0034] Removing container [rke-cp-port-listener] on host [192.168.10.10], try #1 INFO[0034] [remove/rke-cp-port-listener] Successfully removed container on host [192.168.10.10] INFO[0034] Removing container [rke-worker-port-listener] on host [192.168.10.12], try #1 INFO[0034] Removing container [rke-worker-port-listener] on host [192.168.10.13], try #1 INFO[0034] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.10.12] INFO[0034] [remove/rke-worker-port-listener] Successfully removed container on host [192.168.10.13] INFO[0034] [network] Port listener containers removed successfully INFO[0034] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0034] Checking if container [cert-deployer] is running on host [192.168.10.13], try #1 INFO[0034] Checking if container [cert-deployer] is running on host [192.168.10.14], try #1 INFO[0034] Checking if container [cert-deployer] is running on host [192.168.10.12], try #1 INFO[0034] Checking if container [cert-deployer] is running on host [192.168.10.10], try #1 INFO[0034] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0034] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0034] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0034] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0034] Starting container [cert-deployer] on host [192.168.10.13], try #1 INFO[0034] Starting container [cert-deployer] on host [192.168.10.14], try #1 INFO[0034] Starting container [cert-deployer] on host [192.168.10.12], try #1 INFO[0034] Starting container [cert-deployer] on host [192.168.10.10], try #1 INFO[0034] Checking if container [cert-deployer] is running on host [192.168.10.13], try #1 INFO[0035] Checking if container [cert-deployer] is running on host [192.168.10.12], try #1 INFO[0035] Checking if container [cert-deployer] is running on host [192.168.10.14], try #1 INFO[0035] Checking if container [cert-deployer] is running on host [192.168.10.10], try #1 INFO[0039] Checking if container [cert-deployer] is running on host [192.168.10.13], try #1 INFO[0039] Removing container [cert-deployer] on host [192.168.10.13], try #1 INFO[0040] Checking if container [cert-deployer] is running on host [192.168.10.12], try #1 INFO[0040] Checking if container [cert-deployer] is running on host [192.168.10.14], try #1 INFO[0040] Removing container [cert-deployer] on host [192.168.10.12], try #1 INFO[0040] Removing container [cert-deployer] on host [192.168.10.14], try #1 INFO[0040] Checking if container [cert-deployer] is running on host [192.168.10.10], try #1 INFO[0040] Removing container [cert-deployer] on host [192.168.10.10], try #1 INFO[0040] [reconcile] Rebuilding and updating local kube config INFO[0040] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0040] [reconcile] host [192.168.10.10] is a control plane node with reachable Kubernetes API endpoint in the cluster INFO[0040] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0040] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.10.10] INFO[0040] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0040] Starting container [file-deployer] on host [192.168.10.10], try #1 INFO[0040] Successfully started [file-deployer] container on host [192.168.10.10] INFO[0040] Waiting for [file-deployer] container to exit on host [192.168.10.10] INFO[0040] Waiting for [file-deployer] container to exit on host [192.168.10.10] INFO[0040] Container [file-deployer] is still running on host [192.168.10.10]: stderr: [], stdout: [] INFO[0041] Waiting for [file-deployer] container to exit on host [192.168.10.10] INFO[0041] Removing container [file-deployer] on host [192.168.10.10], try #1 INFO[0041] [remove/file-deployer] Successfully removed container on host [192.168.10.10] INFO[0041] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0041] [reconcile] Reconciling cluster state INFO[0041] [reconcile] Check etcd hosts to be deleted INFO[0041] [reconcile] Check etcd hosts to be added INFO[0041] [reconcile] Rebuilding and updating local kube config INFO[0041] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0041] [reconcile] host [192.168.10.10] is a control plane node with reachable Kubernetes API endpoint in the cluster INFO[0041] [reconcile] Reconciled cluster state successfully INFO[0041] max_unavailable_worker got rounded down to 0, resetting to 1 INFO[0041] Setting maxUnavailable for worker nodes to: 1 INFO[0041] Setting maxUnavailable for controlplane nodes to: 1 INFO[0041] Pre-pulling kubernetes images INFO[0041] Pulling image [rancher/hyperkube:v1.21.9-rancher1] on host [192.168.10.13], try #1 INFO[0041] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.12] INFO[0041] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.14] INFO[0041] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.10] INFO[0130] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.13] INFO[0130] Kubernetes images pulled successfully INFO[0130] [etcd] Building up etcd plane.. INFO[0130] [etcd] Successfully started etcd plane.. Checking etcd cluster health INFO[0130] [etcd] etcd host [192.168.10.14] reported healthy=true INFO[0130] [controlplane] Now checking status of node 192.168.10.10, try #1 INFO[0130] [authz] Creating rke-job-deployer ServiceAccount INFO[0130] [authz] rke-job-deployer ServiceAccount created successfully INFO[0130] [authz] Creating system:node ClusterRoleBinding INFO[0130] [authz] system:node ClusterRoleBinding created successfully INFO[0130] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding INFO[0130] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully INFO[0130] Successfully Deployed state file at [./cluster.rkestate] INFO[0130] [state] Saving full cluster state to Kubernetes INFO[0130] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state INFO[0130] [worker] Now checking status of node 192.168.10.14, try #1 INFO[0130] [worker] Now checking status of node 192.168.10.12, try #1 INFO[0130] [worker] Upgrading Worker Plane.. INFO[0155] First checking and processing worker components for upgrades on nodes with etcd role one at a time INFO[0155] [workerplane] Processing host 192.168.10.14 INFO[0155] [worker] Now checking status of node 192.168.10.14, try #1 INFO[0155] [worker] Getting list of nodes for upgrade INFO[0155] [workerplane] Upgrade not required for worker components of host 192.168.10.14 INFO[0155] Now checking and upgrading worker components on nodes with only worker role 1 at a time INFO[0155] [workerplane] Processing host 192.168.10.12 INFO[0155] [worker] Now checking status of node 192.168.10.12, try #1 INFO[0155] [worker] Getting list of nodes for upgrade INFO[0155] [workerplane] Upgrade not required for worker components of host 192.168.10.12 INFO[0155] [workerplane] Processing host 192.168.10.13 INFO[0155] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0156] Starting container [nginx-proxy] on host [192.168.10.13], try #1 INFO[0156] [worker] Successfully started [nginx-proxy] container on host [192.168.10.13] INFO[0156] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0156] Starting container [rke-log-linker] on host [192.168.10.13], try #1 INFO[0156] [worker] Successfully started [rke-log-linker] container on host [192.168.10.13] INFO[0156] Removing container [rke-log-linker] on host [192.168.10.13], try #1 INFO[0156] [remove/rke-log-linker] Successfully removed container on host [192.168.10.13] INFO[0156] Checking if container [service-sidekick] is running on host [192.168.10.13], try #1 INFO[0156] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0156] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.13] INFO[0156] Starting container [kubelet] on host [192.168.10.13], try #1 INFO[0156] [worker] Successfully started [kubelet] container on host [192.168.10.13] INFO[0156] [healthcheck] Start Healthcheck on service [kubelet] on host [192.168.10.13] INFO[0162] [healthcheck] service [kubelet] on host [192.168.10.13] is healthy INFO[0162] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0162] Starting container [rke-log-linker] on host [192.168.10.13], try #1 INFO[0162] [worker] Successfully started [rke-log-linker] container on host [192.168.10.13] INFO[0162] Removing container [rke-log-linker] on host [192.168.10.13], try #1 INFO[0162] [remove/rke-log-linker] Successfully removed container on host [192.168.10.13] INFO[0162] Image [rancher/hyperkube:v1.21.9-rancher1] exists on host [192.168.10.13] INFO[0162] Starting container [kube-proxy] on host [192.168.10.13], try #1 INFO[0162] [worker] Successfully started [kube-proxy] container on host [192.168.10.13] INFO[0162] [healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.10.13] INFO[0167] [healthcheck] service [kube-proxy] on host [192.168.10.13] is healthy INFO[0167] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0168] Starting container [rke-log-linker] on host [192.168.10.13], try #1 INFO[0168] [worker] Successfully started [rke-log-linker] container on host [192.168.10.13] INFO[0168] Removing container [rke-log-linker] on host [192.168.10.13], try #1 INFO[0168] [remove/rke-log-linker] Successfully removed container on host [192.168.10.13] INFO[0168] [worker] Successfully upgraded Worker Plane.. INFO[0168] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.14] INFO[0168] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.13] INFO[0168] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.12] INFO[0168] Image [rancher/rke-tools:v0.1.78] exists on host [192.168.10.10] INFO[0168] Starting container [rke-log-cleaner] on host [192.168.10.13], try #1 INFO[0168] Starting container [rke-log-cleaner] on host [192.168.10.14], try #1 INFO[0168] Starting container [rke-log-cleaner] on host [192.168.10.12], try #1 INFO[0168] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.13] INFO[0168] Removing container [rke-log-cleaner] on host [192.168.10.13], try #1 INFO[0168] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.12] INFO[0168] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.14] INFO[0168] Removing container [rke-log-cleaner] on host [192.168.10.14], try #1 INFO[0168] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.13] INFO[0168] Starting container [rke-log-cleaner] on host [192.168.10.10], try #1 INFO[0168] Removing container [rke-log-cleaner] on host [192.168.10.12], try #1 INFO[0168] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.12] INFO[0168] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.14] INFO[0169] [cleanup] Successfully started [rke-log-cleaner] container on host [192.168.10.10] INFO[0169] Removing container [rke-log-cleaner] on host [192.168.10.10], try #1 INFO[0169] [remove/rke-log-cleaner] Successfully removed container on host [192.168.10.10] INFO[0169] [sync] Syncing nodes Labels and Taints INFO[0169] [sync] Successfully synced nodes Labels and Taints INFO[0169] [network] Setting up network plugin: canal INFO[0169] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes INFO[0169] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes INFO[0169] [addons] Executing deploy job rke-network-plugin INFO[0169] [addons] Setting up coredns INFO[0169] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0169] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0169] [addons] Executing deploy job rke-coredns-addon INFO[0169] [addons] CoreDNS deployed successfully INFO[0169] [dns] DNS provider coredns deployed successfully INFO[0169] [addons] Setting up Metrics Server INFO[0169] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0169] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0169] [addons] Executing deploy job rke-metrics-addon INFO[0169] [addons] Metrics Server deployed successfully INFO[0169] [ingress] Setting up nginx ingress controller INFO[0169] [ingress] removing admission batch jobs if they exist INFO[0169] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0169] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0169] [addons] Executing deploy job rke-ingress-controller INFO[0169] [ingress] removing default backend service and deployment if they exist INFO[0169] [ingress] ingress controller nginx deployed successfully INFO[0169] [addons] Setting up user addons INFO[0169] [addons] no user addons defined INFO[0169] Finished building Kubernetes cluster successfully ~~~ ~~~powershell [root@master01 rancher]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.10.10 Ready controlplane 51m v1.21.9 192.168.10.12 Ready worker 51m v1.21.9 192.168.10.13 Ready worker 62s v1.21.9 192.168.10.14 Ready etcd 51m v1.21.9 ~~~ ## 12.2 移除worker节点 >修改cluster.yml文件,将对应节点信息删除即可。 ~~~powershell # vim cluster.yml ...... - address: 192.168.10.13 port: "22" internal_address: "" role: - worker hostname_override: user: "rancher" docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] ...... ~~~ ~~~powershell [root@master01 rancher]# rke up --update-only ~~~ ~~~powershell [root@master01 rancher]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.10.10 Ready controlplane 53m v1.21.9 192.168.10.12 Ready worker 53m v1.21.9 192.168.10.14 Ready etcd 53m v1.21.9 ~~~ >但是worker节点上的pod是没有结束运行的。如果节点被重复使用，那么在创建新的 Kubernetes 集群时，将自动删除 pod。 ~~~powershell [root@worker02 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b96aa2ac2c25 rancher/nginx-ingress-controller "/usr/bin/dumb-init …" 3 minutes ago Up 3 minutes k8s_controller_nginx-ingress-controller-wxzv4_ingress-nginx_2f6d0569-6a92-4208-8fae-f46b23f2b123_0 f8e7f496e9af rancher/mirrored-coredns-coredns "/coredns -conf /etc…" 3 minutes ago Up 3 minutes k8s_coredns_coredns-8578b6dbdd-xqqdd_kube-system_f10a7413-1f1a-44bf-9070-b7420c296a39_0 7df4ce7aad96 rancher/mirrored-coreos-flannel "/opt/bin/flanneld -…" 3 minutes ago Up 3 minutes k8s_kube-flannel_canal-6m2wj_kube-system_5a55b012-e6ba-4b41-aee4-323a7ce99871_0 38693983ea9c rancher/mirrored-calico-node "start_runit" 4 minutes ago Up 3 minutes k8s_calico-node_canal-6m2wj_kube-system_5a55b012-e6ba-4b41-aee4-323a7ce99871_0 c45bdddaba81 rancher/mirrored-pause:3.5 "/pause" 4 minutes ago Up 3 minutes k8s_POD_nginx-ingress-controller-wxzv4_ingress-nginx_2f6d0569-6a92-4208-8fae-f46b23f2b123_29 7d97152ec302 rancher/mirrored-pause:3.5 "/pause" 4 minutes ago Up 3 minutes k8s_POD_coredns-8578b6dbdd-xqqdd_kube-system_f10a7413-1f1a-44bf-9070-b7420c296a39_31 ea385d73aab9 rancher/mirrored-pause:3.5 "/pause" 5 minutes ago Up 5 minutes k8s_POD_canal-6m2wj_kube-system_5a55b012-e6ba-4b41-aee4-323a7ce99871_0 ~~~ ## 12.3 增加etcd节点 ### 12.3.1 主机准备 > 与添加worker节点主机准备一致,可单独准备2台新主机。 ### 12.3.2 修改cluster.yml文件 ~~~powershell # vim cluster.yml ...... - address: 192.168.10.15 port: "22" internal_address: "" role: - etcd hostname_override: "" user: rancher docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] - address: 192.168.10.16 port: "22" internal_address: "" role: - etcd hostname_override: "" user: rancher docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] ...... ~~~ ### 12.3.3 执行rke up命令 ~~~powershell # rke up --update-only ~~~ ### 12.3.4 查看添加结果 ~~~powershell [root@master01 rancher]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.10.10 Ready controlplane 114m v1.21.9 192.168.10.12 Ready worker 114m v1.21.9 192.168.10.14 Ready etcd 114m v1.21.9 192.168.10.15 Ready etcd 99s v1.21.9 192.168.10.16 Ready etcd 85s v1.21.9 ~~~ ~~~powershell [root@etcd01 ~]# docker exec -it etcd /bin/sh # etcdctl member list 746b681e35b1537c, started, etcd-192.168.10.16, https://192.168.10.16:2380, https://192.168.10.16:2379, false b07954b224ba7459, started, etcd-192.168.10.15, https://192.168.10.15:2380, https://192.168.10.15:2379, false e94295bf0a471a67, started, etcd-192.168.10.14, https://192.168.10.14:2380, https://192.168.10.14:2379, false ~~~ # 十三、部署应用 ## 13.1 创建资源清单文件 ~~~powershell # vim nginx.yaml # cat nginx.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test spec: selector: matchLabels: app: nginx env: test owner: rancher replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx env: test owner: rancher spec: containers: - name: nginx-test image: nginx:1.19.9 ports: - containerPort: 80 ~~~ ~~~powershell # kubectl apply -f nginx.yaml ~~~ ~~~powershell # vim nginx-service.yaml # cat nginx-service.yaml --- apiVersion: v1 kind: Service metadata: name: nginx-test labels: run: nginx spec: type: NodePort ports: - port: 80 protocol: TCP selector: owner: rancher ~~~ ~~~powershell # kubectl apply -f nginx-service.yaml ~~~ ## 13.2 验证 ~~~powershell # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-test-7d95fb4447-6k4p9 1/1 Running 0 55s 10.42.2.11 192.168.10.12 nginx-test-7d95fb4447-sfnsk 1/1 Running 0 55s 10.42.2.10 192.168.10.12 ~~~ ~~~powershell # kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.43.0.1 443/TCP 120m nginx-test NodePort 10.43.158.143 80:32669/TCP 2m22s owner=rancher ~~~ ![image-20220222144422235](../../img/kubernetes/kubernetes_rke/image-20220222144422235.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_rokectmq.md ================================================ # rocketmq部署 # 一、rokectmq介绍 Apache RocketMQ是一个分布式消息传递和流媒体平台，具有低延迟、高性能和可靠性、万亿级别的容量和灵活的可伸缩性。 # 二、rokectmq特性 - 发布/订阅消息传递模型 - 定期消息传递 - 按时间或偏移量进行消息回溯 - 日志中心流 - 大数据集成 - 在同一队列中可靠的FIFO和严格的有序消息传递 - 有效的拉伸消费模式 - 在一个队列中有百万级的消息积累容量 - 多种消息传递协议，如JMS和OpenMessaging - 灵活的分布式扩展部署体系结构 - 快速批量消息交换系统。 - 各种消息过滤机制，如SQL和标记 - 用于隔离测试和云隔离集群的Docker映像。 - 用于配置、度量和监视的功能丰富的管理仪表板 # 三、使用rocketmq理由 - 强调集群无单点，可扩展，任意一点高可用，水平可扩展 ​ 方便集群配置，而且容易扩展（横向和纵向），通过slave的方式每一点都可以实现高可用 - 支持上万个队列，顺序消息 ​ 顺序消费是实现在同一队列的，如果高并发的情况就需要队列的支持，rocketmq可以满足上万个队列同时存在 - 任性定制你的消息过滤 ​ rocketmq提供了两种类型的消息过滤，也可以说三种可以通过topic进行消息过滤、可以通过tag进行消息过滤、还可以通过filter的方式任意定制过滤 - 消息的可靠性（无Buffer，持久化，容错，回溯消费） ​ 消息无buffer就不用担心buffer回满的情况，rocketmq的所有消息都是持久化的，生产者本身可以进行错误重试，发布者也会按照时间阶梯的方式进行消息重发，消息回溯说的是可以按照指定的时间进行消息的重新消费，既可以向前也可以向后（前提条件是要注意消息的擦除时间） - 海量消息堆积能力，消息堆积后，写入低延迟 ​ 针对于provider需要配合部署方式，对于consumer，如果是集群方式一旦master返现消息堆积会向consumer下发一个重定向指令，此时consumer就可以从slave进行数据消费了 - 分布式事务 rocketmq对这一块说的不是很清晰，而且官方也说现在这块存在缺陷（会令系统pagecache过多），所以线上建议还是少用为好 - 消息失败重试机制 ​ 针对provider的重试，当消息发送到选定的broker时如果出现失败会自动选择其他的broker进行重发，默认重试三次，当然重试次数要在消息发送的超时时间范围内。 ​ 针对consumer的重试，如果消息因为各种原因没有消费成功，会自动加入到重试队列，一般情况如果是因为网络等问题连续重试也是照样失败，所以rocketmq也是采用阶梯重试的方式。 - 定时消费 ​ 除了上面的配置，在发送消息是也可以针对message设置setDelayTimeLevel - 活跃的开源社区 ​ 现在rocketmq成为了apache的一款开源产品，活跃度也是不容怀疑的 - 成熟度 ​ 经过双十一考验 # 四、rocketmq核心概念 - NameServer 这里我们可以理解成类似于zk的一个注册中心，而且rocketmq最初也是基于zk作为注册中心的，现在相当于为rocketmq自定义了一个注册中心，代码不超过1000行。RocketMQ 有多种配置方式可以令客户端找到 Name Server, 然后通过 Name Server 再找到 Broker，分别如下，优先级由高到低，高优先级会覆盖低优先级。客户端提供http和ip:端口号的两种方式，推荐使用http的方式可以实现nameserver的热部署。 - Push Consumer ​ Consumer 的一种，应用通常通过 Consumer 对象注册一个 Listener 接口，一旦收到消息，Consumer 对象立刻回调 Listener 接口方法，类似于activemq的方式 - Pull Consumer ​ Consumer 的一种，应用通常主动调用 Consumer 的拉消息方法从 Broker 拉消息，主动权由应用控制 - Producer Group ​ 一类producer的集合名称，这类producer通常发送一类消息，且发送逻辑一致 - Consumer Group ​ 同上，consumer的集合名称 - Broker ​ 消息中转的角色，负责存储消息（实际的存储是调用的store组件完成的），转发消息，一般也称为server，同jms中的provider - Message Filter ​ 可以实现高级的自定义的消息过滤 - Master/Slave ​ 集群的主从关系，broker的name相同，brokerid=0的为主master，大于0的为从slave，可以一主多从，但一从只能有一主 # 五、rocketmq角色 RocketMQ由四部分构成：Producer、Consumer、Broker和NameServer 启动顺序：NameServer->Broker 为了消除单点故障，增加可靠性或增大吞吐量，可以在多台机器上部署多个nameserver和broker，并且为每个broker部署1个或多个slave ![image-20221207102436799](../../img/kubernetes/kubernetes_rokectmq/image-20221207102436799.png) >Topic & message queue：一个分布式消息队列中间件部署好以后，可以给很多个业务提供服务，同一个业务也有不同类型的消息要投递，这些不同类型的消息以不同的 Topic 名称来区分。所以发送和接收消息前，先创建topic，针对某个 Topic 发送和接收消息。有了 Topic 以后，还需要解决性能问题 。 如果一个Topic 要发送和接收的数据量非常大， 需要能支持增加并行处理的机器来提高处理速度，这时候一个 Topic 可以根据需求设置一个或多个 Message Queue, Message Queue 类似分区或 Partition 。Topic有了多个 Message Queue 后，消息可以并行地向各个Message Queue 发送，消费者也可以并行地从多个 Message Queue 读取消息并消费 。 # 六、rocketmq集群部署方式 - 单Master模式 ​ 只有一个 Master节点 ​ 优点：配置简单，方便部署 ​ 缺点：这种方式风险较大，一旦Broker重启或者宕机时，会导致整个服务不可用，不建议线上环境使用 - 多Master模式 ​ 一个集群无 Slave，全是 Master，例如 2 个 Master 或者 3 个 Master ​ 优点：配置简单，单个Master 宕机或重启维护对应用无影响，在磁盘配置为RAID10 时，即使机器宕机不可恢复情况下，由与 RAID10磁盘非常可靠，消息也不会丢（异步刷盘丢失少量消息，同步刷盘一条不丢）。性能最高。 ​ 缺点：单台机器宕机期间，这台机器上未被消费的消息在机器恢复之前不可订阅，消息实时性会受到受到影响 - 多Master多Slave模式（异步复制） ​ 每个 Master 配置一个 Slave，有多对Master-Slave， HA，采用异步复制方式，主备有短暂消息延迟，毫秒级。 ​ 优点：即使磁盘损坏，消息丢失的非常少，且消息实时性不会受影响，因为Master 宕机后，消费者仍然可以从 Slave消费，此过程对应用透明。不需要人工干预。性能同多 Master 模式几乎一样。 ​ 缺点： Master 宕机，磁盘损坏情况，会丢失少量消息。 - 多Master多Slave模式（同步双写） ​ 每个 Master 配置一个 Slave，有多对Master-Slave， HA采用同步双写方式，主备都写成功，向应用返回成功。 ​ 优点：数据与服务都无单点， Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高 ​ 缺点：性能比异步复制模式略低，大约低 10%左右，发送单个消息的 RT会略高。目前主宕机后，备机不能自动切换为主机，后续会支持自动切换功能 # 七、rocketmq集群部署 >rocketmq-operator都是不能很灵活的调整副本数，毕竟每个副本对应其唯一的配置的文件，不同的broker实例使用的配置文件都是不一样的，都存在差异，一旦副本变化了，就不能很好的和broker 的配置文件内容一 一对应起来。 >接下来我们创建一个基于k8s部署单master以及多master部署rocketmq集群，并且只需要一个broker配置文件，多个broker实例会自动基于该broker配置文件模板，自动生成不同broker实例的broker配置文件，扩容或者伸缩rocketmq集群 nameserver或者broker副本数的时候不需要理会配置文件，仅仅是调整实例的副本即可。 > 备注：该方案不适合有slave节点的部署方式。 ## 7.1 环境说明 - 提供持久存储动态供给功能 ~~~powershell [root@k8s-master01 ~]# kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client fuseim.pri/ifs Delete Immediate false 8d ~~~ - 提供metallb功能 > 按官网提供的方式修改即可，链接：https://metallb.universe.tf/ ~~~powershell # kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.5/config/manifests/metallb-native.yaml ~~~ ~~~powershell # vim ippool.yaml # cat ippool.yaml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: first-pool namespace: metallb-system spec: addresses: - 192.168.10.100-192.168.100.110 ~~~ ~~~powershell # kubectl create -f ippool.yaml ~~~ ~~~powershell # vim l2.yaml # cat l2.yaml apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: example namespace: metallb-system ~~~ ~~~powershell # kubectl create -f l2.yaml ~~~ - 提供ingress nginx controller ~~~powershell # wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/cloud/deploy.yaml ~~~ ~~~powershell 部署前修改362行,把Local修改为Cluster # kubectl apply -f deploy.yaml ~~~ ~~~powershell # kubectl get ns NAME STATUS AGE ingress-nginx Active 8s ~~~ ~~~powershell # kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-r9gqt 0/1 Completed 0 62s ingress-nginx-admission-patch-r24d6 0/1 Completed 0 62s ingress-nginx-controller-7844b9db77-mxrgn 1/1 Running 0 62s ~~~ ~~~powershell # kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.96.21.124 192.168.10.100 80:31844/TCP,443:32294/TCP 49m ingress-nginx-controller-admission ClusterIP 10.109.18.235 443/TCP 49m ~~~ - 容器镜像仓库 > 没有本地容器镜像仓库，可以使用hub.dockcer.com。 ## 7.2 构建rocketmq镜像 rocketmq-namesrv和 rocketmq-broker共用同一个镜像，仅仅是启动命令和启动参数不一样，后期可灵活的通过调整启动命令和启动参数来实现不同的效果（比如通过挂载configMap的方式自定义rocketmq的配置文件，而不需要重建rocketmq的镜像。 ~~~powershell # vim Dockerfile # cat Dockerfile FROM docker.io/library/openjdk:8u102-jdk AS JDK LABEL mail=admin@kubemsb.com RUN rm -vf /etc/localtime \ && ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \ && echo "Asia/Shanghai" > /etc/timezone \ && export LANG=zh_CN.UTF-8 RUN curl -k https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/4.9.4/rocketmq-all-4.9.4-bin-release.zip \ -o /tmp/rocketmq-all-4.9.4-bin-release.zip \ && unzip /tmp/rocketmq-all-4.9.4-bin-release.zip -d /tmp/ \ && mv /tmp/rocketmq-all-4.9.4-bin-release /opt/rocketmq \ && rm -rf /tmp/* RUN sed -ir '/-Xmx/c JAVA_OPT=${JAVA_OPT}' /opt/rocketmq/bin/runserver.sh \ && sed -ir '/-Xmx/c JAVA_OPT=${JAVA_OPT}' /opt/rocketmq/bin/runbroker.sh ## 运行 MQ 应用时候可以通过环境变量设置 jvm 数值，如：JAVA_OPT="-server -Xms2g -Xmx2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m" ENV ROCKETMQ_HOME=/opt/rocketmq WORKDIR $ROCKETMQ_HOME ~~~ ~~~powershell # docker build -t docker.io/nextgomsb/rocketmq:v1 . --no-cache ~~~ ~~~powershell # docker images REPOSITORY TAG IMAGE ID CREATED SIZE nextgomsb/rocketmq v1 ed01df462eb3 31 seconds ago 677MB ~~~ ~~~powershell # docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: nextgomsb 替换为自己仓库的用户名 Password: 替换为自己仓库的用户名 WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ~~~ ~~~powershell # docker push docker.io/nextgomsb/rocketmq:v1 ~~~ ![image-20221207115538883](../../img/kubernetes/kubernetes_rokectmq/image-20221207115538883.png) ## 7.3 获取rocketmq-dashboard镜像 rocketmq-dashboard是一个可视化的rocketmq集群运维监控工具。 ![image-20221207104445050](../../img/kubernetes/kubernetes_rokectmq/image-20221207104445050.png) ![image-20221207104516944](../../img/kubernetes/kubernetes_rokectmq/image-20221207104516944.png) ![image-20221207104540676](../../img/kubernetes/kubernetes_rokectmq/image-20221207104540676.png) ## 7.4 rocketmq部署描述文件编写 ### 7.4.1 rocketmq-namesrv部署描述文件 ~~~powershell # vim rocketmq-namesrv.yaml # cat rocketmq-namesrv.yaml --- apiVersion: v1 kind: Namespace metadata: name: rocketmq --- apiVersion: apps/v1 kind: StatefulSet metadata: name: rocketmq-namesrv namespace: rocketmq spec: serviceName: rocketmq-namesrv replicas: 2 selector: matchLabels: app: rocketmq-namesrv template: metadata: labels: app: rocketmq-namesrv spec: containers: - name: rocketmq-namesrv-container image: docker.io/nextgomsb/rocketmq:v1 imagePullPolicy: IfNotPresent command: - bin/mqnamesrv env: - name: JAVA_OPT value: -server -Xms2g -Xmx2g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m --- apiVersion: v1 kind: Service metadata: name: rocketmq-namesrv namespace: rocketmq labels: app: rocketmq-namesrv spec: ports: - port: 9876 protocol: TCP targetPort: 9876 selector: app: rocketmq-namesrv type: ClusterIP ~~~ ~~~powershell # dig -t a rocketmq-namesrv.rocketmq.svc.cluster.local. @10.96.0.10 ~~~ ### 7.4.2 rocketmq-broker部署描述文件 ~~~powershell # vim rocketmq-broker.yaml # cat rocketmq-broker.yaml --- apiVersion: apps/v1 kind: StatefulSet metadata: name: rocketmq-broker namespace: rocketmq spec: serviceName: rocketmq-broker replicas: 2 selector: matchLabels: app: rocketmq-broker template: metadata: labels: app: rocketmq-broker spec: containers: - name: rocketmq-broker image: nextgomsb/rocketmq:v1 imagePullPolicy: IfNotPresent command: - bin/mqbroker - --namesrvAddr=rocketmq-namesrv.rocketmq.svc.cluster.local.:9876 env: - name: JAVA_OPT value: -server -Xms1g -Xmx1g dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler terminationGracePeriodSeconds: 30 updateStrategy: rollingUpdate: partition: 0 type: RollingUpdate ~~~ ### 7.4.3 rocketmq-dashboard部署描述文件 > 部署一个能实现运维监控rocketmq的可视化web应用。 部署rocketmq-dashboard应用时候重点关注部署文件里面的env环境变量参数JAVA_OPTS，该env环境变量（JAVA_OPTS）决定了应用是否能成功连接到 rocketmq-namesrv 服务。 ~~~powershell # vim rocketmq-dashboard.yaml # cat rocketmq-dashboard.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: rocketmq-dashboard namespace: rocketmq labels: app: rocketmq-dashboard spec: replicas: 1 selector: matchLabels: app: rocketmq-dashboard template: metadata: labels: app: rocketmq-dashboard spec: containers: - name: rocketmq-dashboard image: apacherocketmq/rocketmq-dashboard:latest imagePullPolicy: IfNotPresent env: - name: JAVA_OPTS value: -Drocketmq.namesrv.addr=rocketmq-namesrv.rocketmq.svc.cluster.local.:9876 dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 --- apiVersion: v1 kind: Service metadata: name: rocketmq-dashboard namespace: rocketmq labels: app: rocketmq-dashboard spec: ports: - port: 8080 protocol: TCP targetPort: 8080 selector: app: rocketmq-dashboard type: ClusterIP ~~~ ## 7.5 执行部署描述文件 ### 7.5.1 rocketmq-namesrv部署 ~~~powershell # kubectl create -f rocketmq-namesrv.yaml ~~~ ~~~powershell # kubectl get pods -n rocketmq NAME READY STATUS RESTARTS AGE ... rocketmq-namesrv-0 1/1 Running 0 14m rocketmq-namesrv-1 1/1 Running 0 13m ~~~ ### 7.5.2 rocketmq-broker部署 ~~~powershell # kubectl create -f rocketmq-broker.yaml ~~~ ~~~powershell # kubectl get pods -n rocketmq NAME READY STATUS RESTARTS AGE rocketmq-broker-0 1/1 Running 0 4m16s rocketmq-broker-1 1/1 Running 0 4m15s ~~~ ### 7.5.3 rocketmq-dashboard部署 ~~~powershell # kubectl create -f rocketmq-dashboard.yaml ~~~ ~~~powershell # kubectl get pods -n rocketmq NAME READY STATUS RESTARTS AGE rocketmq-dashboard-f4ccdf496-sv984 1/1 Running 0 73s ~~~ ## 7.6 调整副本 ~~~powershell 调整rocket-namesrv副本数 kubectl scale sts rocketmq-namesrv --replicas=3 -n rocketmq ~~~ ~~~powershell # kubectl get pods -n rocketmq NAME READY STATUS RESTARTS AGE rocketmq-namesrv-0 1/1 Running 0 15m rocketmq-namesrv-1 1/1 Running 0 14m rocketmq-namesrv-2 1/1 Running 0 4s ~~~ ~~~powershell 调整rocket-broker副本数,集群节点内存要大于副本数中内存。 kubectl scale sts rocketmq-broker --replicas=3 -n rocketmq ~~~ ## 7.7 创建ingress资源对象实现域名访问dashboard ~~~powershell # vim rocketmq-dashboard-ingress.yaml # cat rocketmq-dashboard-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-rocketmq-dashboard #自定义ingress名称 namespace: rocketmq annotations: ingressclass.kubernetes.io/is-default-class: "true" kubernetes.io/ingress.class: nginx spec: rules: - host: rocketmq-dashboard.kubemsb.com # 自定义域名 http: paths: - pathType: Prefix path: "/" backend: service: name: rocketmq-dashboard # 对应上面创建的service名称 port: number: 8080 ~~~ ~~~powershell # kubectl create -f rocketmq-dashboard-ingress.yaml ~~~ ~~~powershell # kubectl get ingress -n rocketmq NAME CLASS HOSTS ADDRESS PORTS AGE ingress-rocketmq-dashboard rocketmq-dashboard.kubemsb.com 80 31s ~~~ ![image-20221207135747845](../../img/kubernetes/kubernetes_rokectmq/image-20221207135747845.png) ![image-20221207135848816](../../img/kubernetes/kubernetes_rokectmq/image-20221207135848816.png) ![image-20221207135956814](../../img/kubernetes/kubernetes_rokectmq/image-20221207135956814.png) ![image-20221207140035693](../../img/kubernetes/kubernetes_rokectmq/image-20221207140035693.png) ![image-20221207181406628](../../img/kubernetes/kubernetes_rokectmq/image-20221207181406628.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_safety.md ================================================ # 1 安全管理 ## 1.1 安全框架 ### 1.1.1 认证框架 **基础知识** 资源操作 ![image-20220722144813760](../../img/kubernetes/kubernetes_safety/image-20220722144813760.png) ```powershell 用户访问k8s业务应用的流程： 方法一：无需api_server认证 用户 -- ingress|service -- pod 方法二：基于api_server认证 管理k8s平台上各种应用对象 ``` **认证流程** 简介 ```powershell 对APIServer的访问一般都要经过的三个步骤，认证(Authn)+授权(Authz)、准入控制(Admission)，它也能在一定程度上提高安全性，不过更多是资源管理方面的作用。 注意：认证和授权功能都是以插件化的方式来实现的，这样可以最大化的用户自定义效果。 ``` ![1633439274496](../../img/kubernetes/kubernetes_safety/1633439274496.png) ```powershell 认证(Authn) 对用户身份进行基本的认证，只允许被当前系统许可的人进入集群内部 授权(Authz) 不同的用户可以获取不同的资源操作权限，比如普通用户、超级用户、等 准入控制(Admission) 类似于审计，主要侧重于 操作动作的校验、语法规范的矫正等写操作场景。 ``` 解析 ```powershell 左侧： 对于k8s来说，它主要面对两种用户： 普通人类用户(交互模式) - User Account 集群内部的pod用户(服务进程) - Service Account 中间： 每个部分都是以插件的方式来整合到 k8s 集群环境中： - 认证和授权是按照 插件的顺序进行依次性的检测，并且遵循 "短路模型" 的认证方式 所谓的"短路"即，失败的时候，到此结束,后续的规则不会进行。 - 准入控制 由于仅用户写操作场景，所以它不遵循 "短路模型"，而是会全部检测 原因在于，它要记录为什么不执行，原因是什么，方便后续审计等动作。 ``` **小结** ```powershell ``` ### 1.1.2 框架解读 学习目标 这一节，我们从 流程解读、对象梳理、小结 三个方面来学习。 **流程解读** 认证 ```powershell 认证用户 在我们的认证范围中，有一个术语叫Subject，它表示我们在集群中基于某些规则和动作尝试操作的对象，在k8s集群中定义了两种类型的subject资源： User Account： 有外部独立服务进行管理的，对于用户的管理集群内部没有一个关联的资源对象 如果需要操作k8s资源，需要集群内部证书的认证签名。 Service Account 通过Kubernetes API 来管理的一些用户帐号，和 namespace 进行关联的 一般是内部资源对象操作资源的一种方式 ``` ```powershell 用户组 单个控制用户的权限台繁琐，可以借助于用户组的机制实现批量用户的自动化管理，主要有以下几种 system:unauthenticated 未能通过任何一个授权插件检验的账号的所有未通过认证测试的用户统一隶属的用户组； system:authenticated 认证成功后的用户自动加入的一个专用组，用于快捷引用所有正常通过认证的用户账号； system:serviceaccounts 所有名称空间中的所有ServiceAccount对象 system:serviceaccounts: 特定名称空间内所有的ServiceAccount对象 ``` ```powershell 认证方式 kubernetes的认证方式主要有两种： 证书认证 - 本质上就是TLS双向认证 令牌认证 - 大量手动配置TLS认证比较麻烦，可以将证书生成token进行间接使用。 ``` 授权 ```powershell 授权主要是在认证的基础上，用于对集群资源的访问控制权限设置，通过检查请求包含的相关属性值，与相对应的访问策略相比较，API请求必须满足某些策略才能被处理。常见授权机制有： ``` ```powershell Node 主要针对节点的基本通信授权认证，比如kubelet的正常通信。 ABAC(Attribute Based Access Control)： 基于属性的访问控制，与apiserver服务的本地策略规则紧密相关，一旦变动需要重启。 RBAC(Role Based Access Control)： 基于角色的访问控制，这是1.6+版本主推的授权策略，可以使用API自定义角色和集群角色，并将角色和特定的用户，用户组，Service Account关联起来，可以用来实现多租户隔离功能（基于namespace资源） ``` 准入控制 ```powershell 准入控制(Admission Control)，实际上是一个准入控制器插件列表，发送到APIServer的请求都需要经过这个列表中的每个准入控制器插件的检查，如果某一个控制器插件准入失败，就准入失败。它主要涉及到pod和容器对特定用户和特定权限之间的关联关系。 比如操作的对象是否存在依赖关系、被操作的对象是否能够增删改查等限制。 ``` **对象梳理** 资源对象 ![image-20220715012035435](../../img/kubernetes/kubernetes_safety/image-20220715012035435.png) 对象简介 ```powershell 认证对象：SA、token 授权对象：Role、ClusterRole、RoleBinding、ClusterRoleBinding 准入对象：LimitRanger、ResourceQuota、PodSecurityPolicy等 ``` **小结** ```powershell ``` ## 1.2 认证实践 ### 1.2.1 用户实践 学习目标 这一节，我们从 SA实践、UA实践、小结 三个方面来学习。 **SA实践** 资源属性 ```powershell apiVersion: v1 # ServiceAccount所属的API群组及版本 kind: ServiceAccount # 资源类型标识 metadata: name # 资源名称 namespace # ServiceAccount是名称空间级别的资源 automountServiceAccountToken # 是否让Pod自动挂载API令牌 secrets <[]Object> # 以该SA运行的Pod所要使用的Secret对象组成的列表 apiVersion # 引用的Secret对象所属的API群组及版本，可省略 kind # 引用的资源的类型，这里是指Secret，可省略 name # 引用的Secret对象的名称，通常仅给出该字段即可 namespace # 引用的Secret对象所属的名称空间 uid # 引用的Secret对象的标识符； imagePullSecrets <[]Object> # 引用的用于下载Pod中容器镜像的Secret对象列表 name # docker-registry类型的Secret资源的名称 ``` 命令解析 ```powershell 命令格式：kubectl create serviceaccount NAME [--dry-run] [options] 作用：创建一个"服务账号" 参数详解 --dry-run=false 模拟创建模式 --generator='serviceaccount/v1' 设定api版本信息 -o, --output='' 设定输出信息格式，常见的有： json|yaml|name|template|... --save-config=false 保存配置信息 --template='' 设定配置模板文件 ``` 简单实践 ```powershell 创建专属目录 [root@kubernetes-master1 ~]# mkdir /data/kubernetes/secure -p; cd /data/kubernetes/secure ``` ```powershell 手工创建SA账号 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl create serviceaccount mysa serviceaccount/mysa created 检查效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get sa mysa NAME SECRETS AGE mysa 1 9s ``` ```powershell 资源清单创建SA [root@kubernetes-master1 /data/kubernetes/secure]# 01_kubernetes_secure_sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: superopsmsb --- apiVersion: v1 kind: Pod metadata: name: nginx-web spec: containers: - name: nginx-web image: kubernetes-register.superopsmsb.com/superopsmsb/nginx_web:v0.1 serviceAccountName: superopsmsb 应用资源清单 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl apply -f 01_kubernetes_secure_sa.yaml serviceaccount/superopsmsb created pod/nginx-web created ``` ```powershell 查看资源属性 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl describe pod nginx-web Name: nginx-web ... Containers: nginx-web: ... Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wqb6w (ro) ... 结果显示： 用户信息挂载到容器的 /var/run/secrets/kubernetes.io/serviceaccount 目录下了 ``` ```powershell 容器内部的账号身份信息 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl exec -it nginx-web -- /bin/sh # ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token ``` **UA实践** 简介 ```powershell 所谓的UA其实就是我们平常做CA时候的基本步骤，主要包括如下几步 1 创建私钥文件 2 基于私钥文件创建证书签名请求 3 基于私钥和签名请求生成证书文件 ``` 1 创建私钥文件 ```powershell 给用户 superopsmsb 创建一个私钥，命名成：superopsmsb.key(无加密) [root@kubernetes-master1 ~]# cd /etc/kubernetes/pki/ [root@kubernetes-master1 /etc/kubernetes/pki]# (umask 077; openssl genrsa -out superopsmsb.key 2048) G 命令解析： genrsa 该子命令用于生成RSA私钥，不会生成公钥，因为公钥提取自私钥 -out filename 生成的私钥保存至filename文件，若未指定输出文件，则为标准输出 -numbits 指定私钥的长度，默认1024，该项必须为命令行的最后一项参数 ``` 2 签名请求 ```powershell 用刚创建的私钥创建一个证书签名请求文件：superopsmsb.csr [root@kubernetes-master1 /etc/kubernetes/pki]# openssl req -new -key superopsmsb.key -out superopsmsb.csr -subj "/CN=superopsmsb/O=superopsmsb" 参数说明： -new 生成证书请求文件 -key 指定已有的秘钥文件生成签名请求，必须与-new配合使用 -out 输出证书文件名称 -subj 输入证书拥有者信息，这里指定 CN 以及 O 的值，/表示内容分隔 CN以及O的值对于kubernetes很重要，因为kubernetes会从证书这两个值对应获取相关信息： "CN"：Common Name，用于从证书中提取该字段作为请求的用户名 (User Name)； 浏览器使用该字段验证网站是否合法； "O"：Organization，用于分组认证 检查效果： [root@kubernetes-master1 /etc/kubernetes/pki]# ls superopsmsb.* superopsmsb.csr superopsmsb.key 结果显示： *.key 是我们的私钥，*.csr是我们的签名请求文件 ``` 3 生成证书 ```powershell 利用Kubernetes集群的CA相关证书对UA文件进行认证 [root@kubernetes-master1 /etc/kubernetes/pki]# openssl x509 -req -in superopsmsb.csr -CA ./ca.crt -CAkey ./ca.key -CAcreateserial -out superopsmsb.crt -days 365 Signature ok subject=/CN=superopsmsb/O=superopsmsb Getting CA Private Key 参数说明： -req 产生证书签发申请命令 -in 指定需要签名的请求文件 -CA 指定CA证书文件 -CAkey 指定CA证书的秘钥文件 -CAcreateserial 生成唯一的证书序列号 -x509 表示输出一个X509格式的证书 -days 指定证书过期时间为365天 -out 输出证书文件 检查文件效果 [root@kubernetes-master1 /etc/kubernetes/pki]# ls superopsmsb.* superopsmsb.crt superopsmsb.csr superopsmsb.key 结果显示： *.crt就是我们最终生成的签证证书 ``` ```powershell 提取信息效果 ]# openssl x509 -in superopsmsb.crt -text -noout Certificate: ... Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes ... Subject: CN=superopsmsb, O=superopsmsb 结果显示： Issuer: 表示是哪个CA机构帮我们认证的 我们关注的重点在于Subject内容中的请求用户所属的组信息 ``` **小结** ```powershell ``` ### 1.2.2 集群用户 学习目标 这一节，我们从 config基础、简单实践、小结 三个方面来学习。 **config基础** 简介 ```powershell 根据我们之前对kubernetes集群的了解，我们主要是通过config文件来进入到kubernetes集群内部的，然后具有操作相关资源的权限。 ``` ![image-20220722152956452](../../img/kubernetes/kubernetes_safety/image-20220722152956452.png) ```powershell 在config文件内部主要做了四件事情： 1 创建登录k8s集群的用户 基于证书和秘钥信息创建用户 2 创建登录k8s集群的地址 3 将登录用户和目标k8s集群关联在一起，形成k8s集群入口 4 设定默认的k8s集群入口 注意： 这里的k8s集群入口其实就类似于我们 通过ssh登录远程主机的一个入口，示例如下： ssh root@10.0.0.12 ``` 命令解读 ```powershell 集群操作 get-clusters 显示 kubeconfig 文件中定义的集群 delete-cluster 删除 kubeconfig 文件中指定的集群 set-cluster Set a cluster entry in kubeconfig 用户操作 delete-user Delete the specified user from the kubeconfig get-users Display users defined in the kubeconfig set-credentials Set a user entry in kubeconfig 上下文操作 current-context Display the current-context delete-context 删除 kubeconfig 文件中指定的 context get-contexts 描述一个或多个 contexts rename-context Rename a context from the kubeconfig file set-context Set a context entry in kubeconfig use-context Set the current-context in a kubeconfig file 其他操作 set Set an individual value in a kubeconfig file unset Unset an individual value in a kubeconfig file view 显示合并的 kubeconfig 配置或一个指定的 kubeconfig 文件 ``` **简单实践** 创建用户 ```powershell 创建用户 [root@kubernetes-master1 /etc/kubernetes/pki]# kubectl config set-credentials superopsmsb --client-certificate=./superopsmsb.crt --client-key=./superopsmsb.key --embed-certs=true --kubeconfig=/tmp/superopsmsb.conf 参数详解： set-credentials 子命令的作用就是给kubeconfig认证文件创建一个用户条目 --client-certificate=path/to/certfile 指定用户的签证证书文件 --client-key=path/to/keyfile 指定用户的私钥文件 --embed-certs=true，在kubeconfig中为用户条目嵌入客户端证书/密钥，默认值是false， --kubeconfig=/path/to/other_config.file 表示将属性信息单独输出到一个文件，不指定的话，表示存到默认的 ~/.kube/config文件中 ``` 创建集群 ```powershell 创建集群 [root@kubernetes-master1 /etc/kubernetes/pki]# kubectl config set-cluster mycluster --server="https://10.0.0.200:6443" --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true --kubeconfig=/tmp/superopsmsb.conf 参数详解： --server=cluster_api_server --certificate-authority=path/to/certificate/authority ``` 关联用户和集群 ```powershell 配置上下文信息 [root@kubernetes-master1 /etc/kubernetes/pki]# kubectl config set-context superopsmsb@mycluster --cluster=mycluster --user=superopsmsb --kubeconfig=/tmp/superopsmsb.conf 属性详解 --cluster=cluster_nickname 关联的集群名称 --user=user_nickname 关联的用户名称 --namespace=namespace 可以设置该生效的命名空间 ``` 设定默认的入口 ```powershell 更改默认登录kubernetes的用户 [root@kubernetes-master1 /etc/kubernetes/pki]# kubectl config use-context superopsmsb@mycluster --kubeconfig=/tmp/superopsmsb.conf ``` 测试效果 ```powershell 查看配置文件内容 [root@kubernetes-master1 /etc/kubernetes/pki]# kubectl config view --kubeconfig=/tmp/superopsmsb.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.200:6443 name: mycluster contexts: - context: cluster: mycluster user: superopsmsb name: superopsmsb@mycluster current-context: superopsmsb@mycluster kind: Config preferences: {} users: - name: superopsmsb user: client-certificate-data: REDACTED client-key-data: REDACTED ``` ```powershell 查看资源测试 [root@kubernetes-master1 /etc/kubernetes/pki]# kubectl get pod --kubeconfig=/tmp/superopsmsb.conf Error from server (Forbidden): pods is forbidden: User "superopsmsb" cannot list resource "pods" in API group "" in the namespace "default" 结果显示： 虽然提示报错，但是这也证明了用户已经被kubernetes承认了 ``` **小结** ```powershell ``` ### 1.2.3 授权基础 学习目标 这一节，我们从 基础知识、Role实践、clusterRole实践、小结 四个方面来学习。 **基础知识** 授权场景 ![image-20220722152956452](../../img/kubernetes/kubernetes_safety/image-20220722152956452.png) 授权机制 ```powershell RBAC使用rbac.authorization.k8s.io API Group 来实现授权决策，允许管理员通过 Kubernetes API 动态配置策略，要启用RBAC，需要在 apiserver 中添加参数--authorization-mode=RBAC，kubeadm安装的集群默认开启了RBAC，我们可以通过查看 Master 节点上 apiserver 的静态Pod定义文件： [root@kubernetes-master1 ~]# grep authorization-mode /etc/kubernetes/manifests/kube-apiserver.yaml - --authorization-mode=Node,RBAC ``` ```powershell Kubernetes的基本特性就是它的所有资源对象都是模型化的 API 对象，我们可以基于api-server对各种资源进行增、删、改、查等操作，但是这些操作涉及到的不仅仅是资源本身和动作，而且还涉及到资源和动作之间的内容，比如api的分组版本和资源和api的关联即权限授权等。 ``` 授权 ![1633487979722](../../img/kubernetes/kubernetes_safety/1633487979722.png) ```powershell 所谓的授权，其实指的是，将某些subject对象赋予执行某些资源动作的权限。我们有时候会将其称为Group(权限组)，而这个组其实是有两部分组成：组名和组关联(也称绑定)。 简单来说：所谓的授权，其实是为用户授予xx角色 关于kubernetes的授权属性，主要包括如下三个层次：namespace级别、cluster级别、混合级别 ``` 授权级别 ![image-20220722160151434](../../img/kubernetes/kubernetes_safety/image-20220722160151434.png) ```powershell namespace级别 rules - 规则，是一组属于不同 API Group 资源上的权限操作的集合 role - 表示在一个namespace中基于rules使用资源的权限，主要涉及到操作和对象 RoleBingding - 将Subject和Role绑定在一起，表示Subject具有指定资源的role操作权限 ``` ```powershell cluster级别 ClusterRole - 表示在一个Cluster中基于rules使用资源的权限 ClusterRoleBingding - 将Subject和ClusterRole绑定在一起，表示Subject指定资源的ClusterRole操作权限 ``` ```powershell 混合级别 RoleBingding - 将Subject基于RoleBingding与ClusterRole绑定在一起 - 表示Subject可以使用所有ClusterRole中指定资源的role角色 - 但是限制在了只能操作某个命名空间的资源。 - 也就是所谓的"权限降级" ``` ``` 场景1： 多个namespace中的role角色都一致，如果都使用内部的RoleBingding的话，每个namespace都必须单独创建role，而使用ClusterRole的话，只需要一个就可以了，大大的减轻批量使用namespace中的RoleBingding 操作。 场景2： 我们对A用户要提升权限，但是，由于A处于考察期，那么我们暂时给他分配一个区域，测试一下它的运行效果。生活中的场景，提升张三为公司副总裁，但是由于是新手，所以加了一个限制 -- 主管销售范围的副总裁。 ``` **简单实践** 属性解析 ```powershell 我们可以使用 kubectl explain role 的方式来查看一下Role的属性信息： apiVersion kind metadata rules <[]Object> apiGroups <[]string> nonResourceURLs <[]string> resourceNames <[]string> resources <[]string> verbs <[]string> -required- 结果显示： 对于role来说，其核心的内容主要是rules的权限规则 在这么多rules属性中，最重要的是verbs权限条目，而且所有的属性都是可以以列表的形式累加存在 ``` ```powershell 命令行创建role，查看具有pod资源的get、list权限的属性信息 [root@kubernetes-master1 ~]# kubectl create role pods-reader --verb=get,list --resource=pods --dry-run -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null 时间信息 name: pods-reader role的名称 rules: 授权规则 - apiGroups: 操作的对象 - "" 所有权限 resources: 资源对象 - pods pod的对象 verbs: 对pod允许的权限 - get 获取 - list 查看 结果显示： 对于一个role必备的rules来说，他主要有三部分组成：apiGroup、resources、verbs apiGroups 设定包含资源的api组，如果是多个，表示只要属于api组范围中的任意资源都可以操作 resources 位于apiGroup范围中的某些具体的资源对象 verbs 针对具体资源对象的一些具体操作 注意： 关于api组的信息获取，可以参照https://kubernetes.io/docs/reference/#api-reference ``` 简单实践 ```powershell 按照上面的role格式，我们写一个role资源文件,允许用户操作 Deployment、Pod、RS 的所有权限 [root@kubernetes-master1 /data/kubernetes/secure]# 02_kubernetes_secure_role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: myrole rules: - apiGroups: ["", "extensions", "apps"] resources: ["pods", "deployments", "replicasets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] 属性解析： Pod属于 core 的 API Group，在YAML中用空字符就可以，Deployment 属于 apps 的 API Group，ReplicaSets属于extensions这个 API Group，所以 rules 下面的 apiGroups 的内容：["", "extensions", "apps"] verbs是可以对这些资源对象执行的操作，如果是所有动作，也可以使用["*"]来代替。 ``` 查看资源对象 ```powershell 初始化实例对象 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl apply -f 02_kubernetes_secure_role.yaml role.rbac.authorization.k8s.io/myrole created 查看效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl describe role myrole ``` ![image-20220722160724920](../../img/kubernetes/kubernetes_safety/image-20220722160724920.png) Role用户授权实践 ```powershell 限定用户只能访问命名空间资源 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl create rolebinding super-rolebind --role=myrole --user=superopsmsb rolebinding.rbac.authorization.k8s.io/super-rolebind created 查看资源效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get pod --kubeconfig=/tmp/superopsmsb.conf NAME READY STATUS RESTARTS AGE nginx-web 1/1 Running 0 56m [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get svc --kubeconfig=/tmp/superopsmsb.conf Error from server (Forbidden): services is forbidden: User "superopsmsb" cannot list resource "services" in API group "" in the namespace "default" [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get svc --kubeconfig=/tmp/superopsmsb.conf -n kube-system Error from server (Forbidden): services is forbidden: User "superopsmsb" cannot list resource "services" in API group "" in the namespace "kube-system" 清理授权 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl delete rolebindings super-rolebind rolebinding.rbac.authorization.k8s.io "super-rolebind" deleted ``` **ClusterRole实践** 属性解析 ```powershell [root@kubernetes-master1 /data/kubernetes/secure]# kubectl explain clusterrole aggregationRule apiVersion kind metadata rules <[]Object> apiGroups <[]string> nonResourceURLs <[]string> resourceNames <[]string> resources <[]string> verbs <[]string> -required- 结果显示： clusterrole相对于role的属性多了一个集中控制器的属性aggregationRule，而这是一个可选的属性 ``` ```powershell 命令行创建clusterrole，查看具有pod资源的get、list权限的属性信息 [root@kubernetes-master1 ~]# kubectl create clusterrole myclusterrole --verb=get,list --resource=pods --dry-run -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null 时间信息 name: myclusterrole role的名称 rules: 授权规则 - apiGroups: 操作的对象 - "" 所有权限 resources: 资源对象 - pods pod的对象 verbs: 对pod允许的权限 - get 获取 - list 查看 结果显示： ClusterRole与role的配置一样，也由三部分组成：apiGroup、resources、verbs ``` 简单实践 ```powershell 按照上面的clusterrole格式，我们写一个clusterrole资源文件,允许用户操作 Deployment、Pod、RS 的所有权限 [root@kubernetes-master1 /data/kubernetes/secure]# 03_kubernetes_secure_clusterrole.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: myclusterrole rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] 创建资源对象 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl apply -f 03_kubernetes_secure_clusterrole.yaml clusterrole.rbac.authorization.k8s.io/myclusterrole created 查看效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl describe clusterrole myclusterrole Name: myclusterrole Labels: Annotations: PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- pods [] [] [get list watch] ``` ClusterRole用户授权实践 ```powershell 限定用户只能访问命名空间资源 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl create clusterrolebinding super-clusterrolebind --clusterrole=myclusterrole --user=superopsmsb 查看资源效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get pod --kubeconfig=/tmp/superopsmsb.conf NAME READY STATUS RESTARTS AGE nginx-web 1/1 Running 0 68m [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get svc --kubeconfig=/tmp/superopsmsb.conf Error from server (Forbidden): services is forbidden: User "superopsmsb" cannot list resource "services" in API group "" in the namespace "default" [root@kubernetes-master1 /data/kubernetes/secure]# kubectl get pod --kubeconfig=/tmp/superopsmsb.conf -n kube-system NAME READY STATUS RESTARTS AGE coredns-5d555c984-hzq8q 1/1 Running 0 9h 清理授权 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl delete clusterrolebinding super-clusterrolebind rolebinding.rbac.authorization.k8s.io "super-clusterrolebind" deleted ``` **小结** ```powershell ``` ### 1.2.4 授权案例 学习目标 这一节，我们从 案例解读、文件实践、小结 三个方面来学习。 **案例解读** 简介 ```powershell 接下来我们在kubernetes的默认dashboard上面，来综合演示一下用户的安全管理操作。 ``` ![image-20220716105844575](../../img/kubernetes/kubernetes_safety/image-20220716105844575.png) ```powershell 对于dashboard来说，他的认证方法主要有两种：令牌认证和文件认证。由于涉及面比较多，我们通过两节的内容来进行讲解，在这里我们用令牌的方式来学习完整的服务认证流程。 令牌认证 - 基于认证用户的唯一令牌来进行认证，有默认的超时机制，一会儿就失效了 文件认证 - 基于账号的token信息，创建kubeconfig文件，实现长久的认证方式。 ``` ```powershell 根据我们之前对serviceaccount的工作流程的学习，对于dashboard的配置也应该遵循相应的操作流程： 1、创建专用的serviceaccount 2、创建对应的权限角色 3、将serviceaccount和权限角色进行关联绑定 ``` **令牌认证** 集群级别 ```powershell 资源定义文件方式 [root@kubernetes-master1 /data/kubernetes/secure]# 04_kubernetes_secure_dashboard_cluster.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: dashboard-admin annotations: rbac.authorization.kubernetes.io/autoupdate: "true" roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: dashboard-admin namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: dashboard-admin namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile 创建资源对象 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl apply -f 04_kubernetes_secure_dashboard_cluster.yaml clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created serviceaccount/dashboard-admin created ``` ```powershell 获取token信息 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') Name: dashboard-admin-token-4cdrd ... token: eyJhbG...qU3__b9ITbLHEytrA 复制token到浏览器查看效果 ``` ![image-20220722165215675](../../img/kubernetes/kubernetes_safety/image-20220722165215675.png) 用户级别 ```powershell 资源定义文件方式 [root@kubernetes-master1 /data/kubernetes/secure]# 05_kubernetes_secure_dashboard_namespace.yaml apiVersion: v1 kind: ServiceAccount metadata: name: dashboard-ns namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: dashboard-ns namespace: default roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: admin subjects: - kind: ServiceAccount name: dashboard-ns namespace: default 创建资源对象 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl apply -f 05_kubernetes_secure_dashboard_namespace.yaml serviceaccount/dashboard-ns created rolebinding.rbac.authorization.k8s.io/dashboard-ns created ``` ```powershell 获取token信息 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl describe secrets $(kubectl get secret | awk '/dashboard-ns/{print $1}') Name: dashboard-ns-token-btq6w ... token: eyJhbG...qU3__8Kqc0Q 复制token到浏览器查看效果 ``` ![image-20220722170114980](../../img/kubernetes/kubernetes_safety/image-20220722170114980.png)**文件实践** namespace文件实践 ```powershell 设置集群信息 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --server="https://10.0.0.200:6443" --embed-certs=true --kubeconfig=/root/dashboard-ns.conf ``` ```powershell 获取秘钥信息 [root@kubernetes-master1 /data/kubernetes/secure]# NAMESPACE_TOKEN=$(kubectl get secret $(kubectl get secret | awk '/dashboard-ns/{print $1}') -o jsonpath={.data.token} |base64 -d) 设置用户信息 kubectl config set-credentials dashboard-ns --token=$NAMESPACE_TOKEN --kubeconfig=/root/dashboard-ns.conf ``` ```powershell 配置上下文信息 kubectl config set-context dashboard-ns@kubernetes --cluster=kubernetes --user=def-ns-admin --kubeconfig=/root/dashboard-ns.conf 切换用户 kubectl config use-context dashboard-ns@kubernetes --kubeconfig=/root/dashboard-ns.conf ``` ```powershell 查看效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl config view --kubeconfig=/root/dashboard-ns.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.200:6443 name: kubernetes contexts: - context: cluster: kubernetes user: def-ns-admin name: dashboard-ns@kubernetes current-context: dashboard-ns@kubernetes kind: Config preferences: {} users: - name: dashboard-ns user: token: REDACTED - name: def-ns-admin user: token: REDACTED ``` ```powershell 下载dashboard-ns.conf到windows主机，然后在浏览器上，以kubeconfig文件方式登录dashboard ``` ![image-20220722170114980](../../img/kubernetes/kubernetes_safety/image-20220722170114980.png) cluster文件实践 ```powershell 设置集群信息 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --server="https://10.0.0.200:6443" --embed-certs=true --kubeconfig=/root/dashboard-cluster.conf ``` ```powershell 获取秘钥信息 [root@kubernetes-master1 /data/kubernetes/secure]# CLUSTER_TOKEN=$(kubectl get secret -n kube-system $(kubectl get secret -n kube-system | awk '/dashboard-admin/{print $1}') -o jsonpath={.data.token} |base64 -d) 设置用户信息 kubectl config set-credentials dashboard-cluster --token=$CLUSTER_TOKEN --kubeconfig=/root/dashboard-cluster.conf ``` ```powershell 配置上下文信息 kubectl config set-context dashboard-cluster@kubernetes --cluster=kubernetes --user=dashboard-cluster --kubeconfig=/root/dashboard-cluster.conf 切换用户 kubectl config use-context dashboard-cluster@kubernetes --kubeconfig=/root/dashboard-cluster.conf ``` ```powershell 查看效果 [root@kubernetes-master1 /data/kubernetes/secure]# kubectl config view --kubeconfig=/root/dashboard-cluster.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.200:6443 name: kubernetes contexts: - context: cluster: kubernetes user: dashboard-cluster name: dashboard-cluster@kubernetes current-context: dashboard-cluster@kubernetes kind: Config preferences: {} users: - name: dashboard-cluster user: token: REDACTED ``` ```powershell 下载dashboard-cluster.conf到windows主机，然后在浏览器上，以kubeconfig文件方式登录dashboard ``` ![image-20220722172012447](../../img/kubernetes/kubernetes_safety/image-20220722172012447.png) **小结** ```powershell ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_sealos.md ================================================ # 使用sealos部署kubernetes集群并实现集群管理 > 本次使用4台主机完成，其中3台主机为master节点，1台主机为worker节点。 # 一、主机准备 ## 1.1 配置主机名 ~~~powershell # hostnamectl set-hostname xxx k8s-master01 k8s-master02 k8s-master03 k8s-worker01 ~~~ ## 1.2 设置静态IP地址 | 序号 | 主机名 | 主机IP | | ---- | ------------ | -------------- | | 1 | k8s-master01 | 192.168.10.142 | | 2 | k8s-master02 | 192.168.10.143 | | 3 | k8s-master03 | 192.168.10.144 | | 4 | k8s-worker01 | 192.168.10.145 | ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="ec87533a-8151-4aa0-9d0f-1e970affcdc6" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.xxx" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ## 1.3 配置主机名与IP地址解析 > 下面解析是管理员添加，sealos在运行过程中，也会自动添加主机名与IP地址解析关系。 ~~~powershell # /etc/hosts 192.168.10.142 k8s-master01 192.168.10.143 k8s-master02 192.168.10.144 k8s-master03 192.168.10.145 k8s-worker01 ~~~ ## 1.4 升级内核 ~~~powershell rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64 awk -F \' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg grub2-set-default "CentOS Linux (5.4.204-1.el7.elrepo.x86_64) 7 (Core)" reboot ~~~ # 二、sealos准备 ~~~powershell wget -c https://sealyun-home.oss-cn-beijing.aliyuncs.com/sealos-4.0/latest/sealos-amd64 -O sealos && chmod +x sealos && mv sealos /usr/bin ~~~ ~~~powershell # sealos version {"gitVersion":"4.0.0","gitCommit":"7146cfe","buildDate":"2022-06-30T14:24:31Z","goVersion":"go1.17.11","compiler":"gc","platform":"linux/amd64"} ~~~ # 三、使用sealos部署kubernetes集群 > kubernetes集群默认使用containerd ~~~powershell sealos run labring/kubernetes:v1.24.0 labring/calico:v3.22.1 --masters 192.168.10.142,192.168.10.143,192.168.10.144 --nodes 192.168.10.145 --passwd centos ~~~ ~~~powershell # kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 16h v1.24.0 k8s-master02 Ready control-plane 16h v1.24.0 k8s-master03 Ready control-plane 16h v1.24.0 k8s-worker01 Ready 16h v1.24.0 ~~~ ~~~powershell # kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d4b75cb6d-59ph5 1/1 Running 1 (15h ago) 16h coredns-6d4b75cb6d-wz6tx 1/1 Running 1 (15h ago) 16h etcd-k8s-master01 1/1 Running 1 (15h ago) 16h etcd-k8s-master02 1/1 Running 1 (15h ago) 16h etcd-k8s-master03 1/1 Running 1 (15h ago) 16h kube-apiserver-k8s-master01 1/1 Running 3 (15h ago) 16h kube-apiserver-k8s-master02 1/1 Running 1 (15h ago) 16h kube-apiserver-k8s-master03 1/1 Running 1 (15h ago) 16h kube-controller-manager-k8s-master01 1/1 Running 3 (15h ago) 16h kube-controller-manager-k8s-master02 1/1 Running 1 (15h ago) 16h kube-controller-manager-k8s-master03 1/1 Running 1 (15h ago) 16h kube-proxy-5l26r 1/1 Running 1 (15h ago) 16h kube-proxy-cfbkh 1/1 Running 1 (15h ago) 16h kube-proxy-g92fs 1/1 Running 1 (15h ago) 16h kube-proxy-zsjxv 1/1 Running 1 (15h ago) 16h kube-scheduler-k8s-master01 1/1 Running 3 (15h ago) 16h kube-scheduler-k8s-master02 1/1 Running 1 (15h ago) 16h kube-scheduler-k8s-master03 1/1 Running 1 (15h ago) 16h kube-sealyun-lvscare-k8s-worker01 1/1 Running 1 (15h ago) 16h ~~~ # 四、使用kuboard实现k8s集群托管 | 序号 | 主机名 | 主机IP | | ---- | -------------- | -------------- | | 1 | kuboard-server | 192.168.10.146 | ## 4.1 kuboard部署及访问 ~~~powershell wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ~~~powershell yum -y install docker-ce ~~~ ~~~powershell systemctl enable --now docker ~~~ ~~~powershell docker run -d --restart=unless-stopped --name=kuboard -p 80:80/tcp -p 10081:10081/tcp -e KUBOARD_ENDPOINT="http://192.168.10.146:80" -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" -v /root/kuboard-data:/data eipwork/kuboard:v3 ~~~ > 用户名和密码分别为：admin及Kuboard123 ![image-20220712165328787](../../img/kubernetes/kubernetes_sealos/image-20220712165328787.png) ## 4.2 kuboard添加k8s集群 ![image-20220712170948576](../../img/kubernetes/kubernetes_sealos/image-20220712170948576.png) ![image-20220712171014736](../../img/kubernetes/kubernetes_sealos/image-20220712171014736.png) ![image-20220712171110532](../../img/kubernetes/kubernetes_sealos/image-20220712171110532.png) ![image-20220712171148271](../../img/kubernetes/kubernetes_sealos/image-20220712171148271.png) ![image-20220712171456289](../../img/kubernetes/kubernetes_sealos/image-20220712171456289.png) ~~~powershell [root@k8s-master01 ~]# kubectl apply -f kuboard-agent.yaml namespace/kuboard created serviceaccount/kuboard-admin created clusterrolebinding.rbac.authorization.k8s.io/kuboard-admin-crb created serviceaccount/kuboard-viewer created clusterrolebinding.rbac.authorization.k8s.io/kuboard-viewer-crb created deployment.apps/kuboard-agent-du7gv7 created deployment.apps/kuboard-agent-du7gv7-2 created ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n kuboard NAME READY STATUS RESTARTS AGE kuboard-agent-du7gv7-2-84f65f77b8-rcb4x 1/1 Running 0 54s kuboard-agent-du7gv7-56c7cb9564-m78qx 1/1 Running 0 54s ~~~ ![image-20220712171651634](../../img/kubernetes/kubernetes_sealos/image-20220712171651634.png) ![image-20220712171756799](../../img/kubernetes/kubernetes_sealos/image-20220712171756799.png) ![image-20220712171828626](../../img/kubernetes/kubernetes_sealos/image-20220712171828626.png) ![image-20220712171924216](../../img/kubernetes/kubernetes_sealos/image-20220712171924216.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_spark.md ================================================ ## 1.1 **Kubernetes 介绍** Kubernetes是Google公司在2014年6月开源的一个容器集群管理系统，使用Go语言开发，也叫K8S（k8s 这个缩写是因为k和s之间有八个字符的关系）。Kubernetes这个名字源于希腊语，意为“舵手”或“飞行员”。Kubernetes的目标是让部署容器化的应用 简单并且高效,提供应用部署，维护，规划，更新。Kubernetes一个核心的特点就是能够自主的管理容器来保证云平台中的容器按照用户的期望状态运行，让用户能够方便的部署自己的应用。 Kubernetes官网地址如下:[https://kubernetes.io/](https://kubernetes.io/)，中文官网地址如下：[https://kubernetes.io/zh-cn/](https://kubernetes.io/zh-cn/) ![image.png](../../img/kubernetes/kubernetes_spark/1ffd5c05d5804b12a2835ce752678ae6.png) 企业中应用程序的部署经历了传统部署时代、虚拟化部署时代、容器化部署时代，尤其是今天容器化部署应用在企业中应用非常广泛，Kubernetes作为容器编排管理工具也越来越重要。 ![image.png](../../img/kubernetes/kubernetes_spark/65c590ae4fee44249a01e40aee1ee6aa.png) ### 1.1.1 **传统部署时代** 早期，各个公司是在物理服务器上运行应用程序。由于无法限制在物理服务器中运行的应用程序资源使用，因此会导致资源分配问题。例如，如果在同一台物理服务器上运行多个应用程序，则可能会出现一个应用程序占用大部分资源的情况，而导致其他应用程序的性能下降。一种解决方案是将每个应用程序都运行在不同的物理服务器上，但是当某个应用资源利用率不高时，服务器上剩余资源无法被分配给其他应用，而且维护许多物理服务器的成本很高。 **物理服务器部署应用痛点如下：** * 物理服务器环境部署人力成本大，特别是在自动化手段不足的情况下，依靠人肉运维的方式解决。 * 当物理服务器出现宕机后，服务器重启时间过长，短则1-2分钟，长则3-5分钟，有背于服务器在线时长达到99.999999999%标准的要求。 * 物理服务器在应用程序运行期间硬件出现故障，解决较麻烦。 * 物理服务器计算资源不能有效调度使用，无法发挥其充足资源的优势。 * 物理服务器环境部署浪费时间，没有自动化运维手段，时间是成倍增加的。 * 在物理服务器上进行应用程序配置变更，需要停止之前部署重新实施部署。 ### 1.1.2 **虚拟化部署时代** 由于以上原因，虚拟化技术被引入了，虚拟化技术允许你在单个物理服务器的CPU上运行多台虚拟机（VM）。虚拟化能使应用程序在不同VM之间被彼此隔离，且能提供一定程度的安全性，因为一个应用程序的信息不能被另一应用程序随意访问。每个 VM 是一台完整的计算机，在虚拟化硬件之上运行所有组件，包括其自己的操作系统。 虚拟化技术能够更好地利用物理服务器的资源，并且因为可轻松地添加或更新应用程序，因此具有更高的可扩缩性，以及降低硬件成本等等的好处。通过虚拟化，你可以将一组物理资源呈现为可丢弃的虚拟机集群。 **虚拟机部署应用优点：** * 虚拟机较物理服务器轻量，可借助虚拟机模板实现虚拟机快捷生成及应用。 * 虚拟机中部署应用与物理服务器一样可控性强，且当虚拟机出现故障时，可直接使用新的虚拟机代替。 * 在物理服务器中使用虚拟机可高效使用物理服务器的资源。 * 虚拟机与物理服务器一样可达到良好的应用程序运行环境的隔离。 * 当部署应用程序的虚拟机出现宕机时，可以快速启动，时间通常可达秒级，10秒或20秒即可启动，应用程序可以继续提供服务。 * 在虚拟机中部署应用，容易扩容及缩容实现、应用程序迁移方便。 **虚拟机部署应用缺点：** * 虚拟机管理软件本身占用物理服务器计算资源较多，例如:VMware Workstation Pro就会占用物理服务器大量资源。 * 虚拟机底层硬件消耗物理服务器资源较大，例如：虚拟机操作系统硬盘，会直接占用大量物理服务器硬盘空间。 * 相较于容器技术，虚拟机启动时间过长，容器启动可按毫秒级计算。 * 虚拟机对物理服务器硬件资源调用添加了调链条，存在浪费时间的现象，所以虚拟机性能弱于物理服务器。 * 由于应用程序是直接部署在虚拟机硬盘上，应用程序迁移时，需要连同虚拟机硬盘中的操作系统一同迁移，会导致迁移文件过大，浪费更多的存储空间及时间消耗过长。 ### 1.1.3 **容器化部署时代** 容器类似于VM，但具备更宽松的隔离特性，使容器之间可以共享操作系统（OS），因此，容器比起VM被认为是更轻量级的。容器化技术中常用的就是Docker容器引擎技术，让开发者可以打包应用以及依赖到一个可移植的镜像中，然后发布到任何平台，其与VM类似，每个容器都具有自己的文件系统、CPU、内存、进程空间等。由于它们与基础架构分离，因此可以跨云和OS发行版本进行移植。 ***基于容器化技术部署应用优点* ：** * 不需要为容器安装操作系统，可以节约大量时间。 * 不需要通过手动的方式在容器中部署应用程序的运行环境，直接部署应用就可以了。 * 不需要管理容器网络，以自动调用的方式访问容器中应用提供的服务。 * 方便分享与构建应用容器，一次构建，到处运行，可在 Ubuntu、RHEL、CoreOS、本地、 Google Kubernetes Engine 等地方运行。 * 毫秒级启动。 * 资源隔离与资源高效应用。应用程序被分解成较小的独立部分，并且可以动态部署和管理，而不是在一台大型单机上整体运行，可以在一台物理机上高密度的部署容器。 虽然使用容器有以上各种优点，但是 **使用容器相比于物理机容器可控性不强** ，例如：对容器的访问，总想按物理服务器或虚拟机的方式去管理它，其实容器与物理服务器、虚拟机管理方式上有着本质的区别的，最好不要管理。 ### 1.1.4 **为什么需要Kubernetes** 一般一个容器中部署运行一个服务，一般不会在一个容器运行多个服务，这样会造成容器镜像复杂度的提高，违背了容器初衷。企业中一个复杂的架构往往需要很多个应用，这就需要运行多个容器，并且需要保证这些容器之间有关联和依赖，那么如何保证各个容器自动部署、容器之间服务发现、容器故障后重新拉起正常运行，这就需要容器编排工具。 Kubernetes就是一个容器编排工具，可以实现容器集群的自动化部署、自动扩展、维护等功能，可以基于Docker技术的基础上，为容器化应用提供部署运行、资源调度、服务发现和动态伸缩等一系列完整功能，提高大规模容器集群管理的便捷性。 **Kubernetes**优点如下： * 服务发现和负载均衡 Kubernetes可以使用DNS名称或自己的IP地址来曝露容器。如果进入容器的流量很大，Kubernetes 可以负载均衡并分配网络流量，从而使部署稳定。 * 存储编排 Kubernetes允许你自动挂载你选择的存储系统，例如本地存储、公共云提供商等。 * 自动部署和回滚 你可以使用Kubernetes描述已部署容器的所需状态，它可以以受控的速率将实际状态更改为期望状态。例如，可以通过Kubernetes来为你部署创建新容器、删除现有容器并将它们的所有资源用于新容器。 * 自动完成装箱计算 在Kubernetes集群上运行容器化的任务时，Kubernetes可以根据运行每个容器指定的多少CPU和内存(RAM)将这些容器按实际情况调度到Kubernetes集群各个节点上，以最佳方式利用集群资源。 * 自我修复 Kubernetes将重新启动失败的容器、替换容器、杀死不响应的容器，并且在准备好服务之前不将其通告给客户端。 * 密钥与配置管理 Kubernetes允许你存储和管理敏感信息，例如密码、OAuth令牌和ssh密钥。 你可以在不重建容器镜像的情况下部署和更新密钥和应用程序配置，也无需在堆栈配置中暴露密钥。 ## 1.2 **Kubernetes集群架构及组件** 一个Kubernetes集群至少有一个主控制平面节点（Control Plane）和一台或者多台工作节点（Node）组成，控制面板和工作节点实例可以是物理设备或云中的实例。Kubernetes 架构如下： ![image.png](../../img/kubernetes/kubernetes_spark/a574f87c70034ec1b0abb158368cc765.png) ### 1.2.1 **Kubernetes 控制平面（Contorl Plane）** Kubernetes控制平面也称为主节点（Master Node），其管理集群中的工作节点（Worker Node）和Pod，在生产环境中,Master节点可以运行在多台节点实例上形成主备，提供Kubernetes集群的容错和高可用性。我们可以通过CLI或者UI页面中向Master节点输入参数控制Kubernetes集群。 Master节点是Kubernetes集群的管理中心，包含很多组件，这些组件管理Kubernetes集群各个方面，例如集群组件通信、工作负载调度和集群状态持久化。这些组件可以在集群内任意节点运行，但是为了方便会在一台实例上运行Master所有组件，并且不会在此实例上运行用户容器。 **Kubernetes Master主节点包含组件如下：** * **kube-apiserver：** 用于暴露kubernetes API，任何的资源请求/调用操作都是通过kube-apiserver提供的接口进行。例如:通过REST/kubectl 操作Kubernetes集群调用的就是Kube-apiserver。 * **etcd：** etcd是一个一致的、高度可用的键值存储库，是kubernetes提供默认的存储系统，用于存储Kubernetes集群的状态和配置数据。 * **kube-scheduler：** scheduler负责监视新创建、未指定运行节点的Pods并选择节点来让Pod在上面运行。如果没有合适的节点，则将Pod处于挂起的状态直到出现一个健康的Node节点。 * **kube-controller-manager：** controller-manager 负责运行Kubernetes中的Controller控制器，这些Controller控制器包括： > * 节点控制器（Node Controller）：负责在节点出现故障时进行通知和响应。 > * 任务控制器（Job Controller）：监测代表一次性任务的 Job 对象，然后创建 Pods 来运行这些任务直至完成。 > * 端点分片控制器（EndpointSlice controller）：填充端点分片（EndpointSlice）对象（以提供 Service 和 Pod 之间的链接）。 > * 服务账号控制器（ServiceAccount controller）：为新的命名空间创建默认的服务账号（ServiceAccount）。 * **cloud-controller-manager** 云控制器管理器（Cloud Controller Manager）嵌入了特定于云平台的控制逻辑，允许你将你的集群连接到云提供商的 API 之上， 并将与该云平台交互的组件同与你的集群交互的组件分离开来。cloud-controller-manager 仅运行特定于云平台的控制器。 因此如果你在自己的环境中运行 Kubernetes，或者在本地计算机中运行学习环境， 所部署的集群不需要有云控制器管理器。 ### 1.2.2 **Kubernetes Node节点** Kubernetes Node节点又称为工作节点（Worker Node），一个Kubernetes集群至少需要一个工作节点，但通常很多，工作节点也包含很多组件，用于运行以及维护Pod及service等信息, 管理volume(CVI)和网络(CNI)。在Kubernetes集群中可以动态的添加和删除节点来扩展和缩减集群。 **工作节点Node上的组件如下：** * **Kubelet:** Kubelet会在集群中每个Worker节点上运行，负责维护容器（Containers）的生命周期(创建pod，销毁pod)，确保Pod处于运行状态且健康。同时也负责Volume(CVI)和网络(CNI)的管理。Kubelet不会管理不是由Kubernetes创建的容器。 * **kube-proxy:** Kube-proxy是集群中每个Worker节点上运行的网络代理，管理IP转换和路由，确保每个Pod获得唯一的IP地址，维护网络规则，这些网络规则会允许从集群内部或外部的网络会话与Pod进行网络通信。 * **container Runtime:** 容器运行时(Container Runtime)负责运行容器的软件，为了运行容器每个Worker节点都有一个Container Runtime引擎，负责镜像管理以及Pod和容器的启动停止。 ## 1.3 **Kubernetes 核心概念** Kubernetes中有非常多的核心概念，下面主要介绍Kubernetes集群中常见的一些概念。 ### 1.3.1 **Pod** Pod是可以在 Kubernetes 中创建和管理的、最小的可部署的计算单元，是Kubernetes调度的基本单位，Pod设计的理念是每个Pod都有一个唯一的IP。Pod就像豌豆荚一样，其中包含着一组（一个或多个）容器，这些容器共享存储、网络、文件系统以及怎样运行这些容器的声明。 ![image.png](../../img/kubernetes/kubernetes_spark/a50dae4997a14440ac578cd042c9a4cc.png) **Node&Pod&Container&应用程序关系如下图所示：** ![image.png](../../img/kubernetes/kubernetes_spark/56789990f8a04f7a9d0cc2b75d727654.png) ### 1.3.2 **Label** Label是附着到object上（例如Pod）的键值对。可以在创建object的时候指定，也可以在object创建后随时指定。Labels的值对系统本身并没有什么含义，只是对用户才有意义。 一个Label是一个key=value的键值对，其中key与value由用户自己指定。Label可以附加到各种资源对象上，例如Node、Pod、Service、RC等，一个资源对象可以定义任意数量的Label，同一个Label也可以被添加到任意数量的资源对象上去，Label通常在资源对象定义时确定，也可以在对象创建后动态添加或者删除。 我们可以通过指定的资源对象捆绑一个或多个不同的Label来实现多维度的资源分组管理功能，以便于灵活、方便地进行资源分配、调度、配置、部署等管理工作。例如：部署不同版本的应用到不同的环境中；或者监控和分析应用（日志记录、监控、告警）等。 一些常用abel示例如下所示: * 版本标签："release" : "stable" , "release" : "canary"... * 环境标签："environment" : "dev" , "environment" : "production" * 架构标签："tier" : "frontend" , "tier" : "backend" , "tier" : "middleware" * 分区标签："partition" : "customerA" , "partition" : "customerB"... * 质量管控标签："track" : "daily" , "track" : "weekly" Label相当于我们熟悉的“标签”，给某个资源对象定义一个Label，就相当于给它打了一个标签，随后可以通过Label Selector（标签选择器）查询和筛选拥有某些Label的资源对象，Kubernetes通过这种方式实现了类似SQL的简单又通用的对象查询机制。 ### 1.3.3 **NameSpace** Namespace 命名空间是对一组资源和对象的抽象集合，比如可以用来将系统内部的对象划分为不同的项目组或者用户组。常见的pod、service、replicaSet和deployment等都是属于某一个namespace的(默认是default)，而node, persistentVolumes等则不属于任何namespace。 当删除一个命名空间时会自动删除所有属于该namespace的资源，default和kube-system命名空间不可删除。 ### 1.3.4 **Controller控制器** 在 Kubernetes 中，Contorller用于管理和运行Pod的对象，控制器通过监控集群的公共状态，并致力于将当前状态转变为期望的状态。一个Controller控制器至少追踪一种类型的 Kubernetes 资源。这些对象有一个代表期望状态的spec字段。该资源的控制器负责确保其当前状态接近期望状态。 不同类型的控制器实现的控制方式不一样，以下介绍常见的几种类型的控制器。 #### 1.3.4.1 deployments控制器 deployments控制器用来部署无状态应用。基于容器部署的应用一般分为两种，无状态应用和有状态应用。 * 无状态应用：认为Pod都一样，没有顺序要求，随意进行扩展和伸缩。例如：nginx,请求本身包含了响应端为响应这一请求所@需的全部信息。每一个请求都像首次执行一样，不会依赖之前的数据进行响应，不需要持久化数据，无状态应用的多个实例之间互不依赖，可以无序的部署、删除或伸缩。 * 有状态应用：每个pod都是独立运行，有唯一的网络表示符，持久化存储，有序。例如：mysql主从，主机名称固定，而且其扩容以及升级等操作也是按顺序进行。有状态应用前后请求有关联与依赖，需要持久化数据，有状态应运用的多个实例之间有依赖，不能相互替换。 在Kubernetes中，一般情况下我们不需要手动创建Pod实例，而是采用更高一层的抽象或定义来管理Pod，针对无状态类型的应用，Kubernetes使用Deloyment的Controller对象与之对应，其典型的应用场景包括： > * 定义Deployment来创建Pod和ReplicaSet > * 滚动升级和回滚应用 > * 扩容和缩容 > * 暂停和继续Deployment #### 1.3.4.2 **ReplicaSet控制器** 通过改变Pod副本数量实现Pod的扩容和缩容，一般Deployment里包含并使用了ReplicaSet。对于ReplicaSet而言，它希望pod保持预期数目、持久运行下去，除非用户明确删除，否则这些对象一直存在，它们针对的是耐久性任务，如web服务等。 #### 1.3.4.3 **statefulSet控制器** Deployments和ReplicaSets是为无状态服务设计的，StatefulSet则是为了有状态服务而设计，其应用场景包括： * 稳定的持久化存储，即Pod重新调度后还是能访问到相同的持久化数据，基于PVC（PersistentVolumeClaim，持久存储卷声明）来实现。 * 稳定的网络标志，即Pod重新调度后其PodName和HostName不变，基于Headless Service(即没有Cluster IP的Service)来实现。 注意从写法上来看statefulSet与deployment几乎一致，就是类型不一样。 #### 1.3.4.4 **DaemonSet控制器** DaemonSet保证在每个Node上都运行一个相同Pod实例，常用来部署一些集群的日志、监控或者其他系统管理应用。DaemonSet使用注意以下几点： * 当节点加入到Kubernetes集群中，pod会被（DaemonSet）调度到该节点上运行。 * 当节点从Kubernetes集群中被移除，被DaemonSet调度的pod会被移除。 * 如果删除一个Daemonset，所有跟这个DaemonSet相关的pods都会被删除。 * 如果一个DaemonSet的Pod被杀死、停止、或者崩溃，那么DaemonSet将会重新创建一个新的副本在这台计算节点上。 * DaemonSet一般应用于日志收集、监控采集、分布式存储守护进程等。 #### 1.3.4.5 **Job控制器** ReplicaSet针对的是耐久性任务，对于非耐久性任务，比如压缩文件，任务完成后，pod需要结束运行，不需要pod继续保持在系统中，这个时候就要用到Job。Job负责批量处理短暂的一次性任务 (short lived one-off tasks)，即仅执行一次的任务，它保证批处理任务的一个或多个Pod成功结束。 #### 1.3.4.6 **Cronjob控制器** Cronjob类似于Linux系统的crontab，在指定的时间周期运行相关的任务。 ### 1.3.5 **Service** 使用kubernetes集群运行工作负载时，由于Pod经常处于用后即焚状态，Pod经常被重新生成，因此Pod对应的IP地址也会经常变化，导致无法直接访问Pod提供的服务，Kubernetes中使用了Service来解决这一问题，即在Pod前面使用Service对Pod进行代理，无论Pod怎样变化 ，只要有Label，就可以让Service能够联系上Pod，把PodIP地址添加到Service对应的端点列表（Endpoints）实现对Pod IP跟踪，进而实现通过Service访问Pod目的。 Service有以下几个注意点： * 通过service为pod客户端提供访问pod方法，即可客户端访问pod入口 * 通过标签动态感知pod IP地址变化等 * 防止pod失联 * 定义访问pod访问策略 * 通过label-selector相关联 * 通过Service实现Pod的负载均衡 Service 有如下四种类型： * ClusterIP：默认，分配一个集群内部可以访问的虚拟IP。 * NodePort：在每个Node上分配一个端口作为外部访问入口。nodePort端口范围为:30000-32767 * LoadBalancer：工作在特定的Cloud Provider上，例如Google Cloud，AWS，OpenStack。 * ExternalName：表示把集群外部的服务引入到集群内部中来，即实现了集群内部pod和集群外部的服务进行通信，适用于外部服务使用域名的方式，缺点是不能指定端口。 ### 1.3.6 **Volume 存储卷** 默认情况下容器的数据是非持久化的，容器消亡以后数据也会跟着丢失。Docker容器提供了Volume机制以便将数据持久化存储。Kubernetes提供了更强大的Volume机制和插件，解决了容器数据持久化以及容器间共享数据的问题。 Kubernetes存储卷的生命周期与Pod绑定，容器挂掉后Kubelet再次重启容器时，Volume的数据依然还在，Pod删除时，Volume才会清理。数据是否丢失取决于具体的Volume类型，比如emptyDir的数据会丢失，而PV的数据则不会丢。 目前Kubernetes主要支持以下Volume类型： * emptyDir：Pod存在，emptyDir就会存在，容器挂掉不会引起emptyDir目录下的数据丢失，但是pod被删除或者迁移，emptyDir也会被删除。 * hostPath：hostPath允许挂载Node上的文件系统到Pod里面去。 * NFS（Network File System）：网络文件系统，Kubernetes中通过简单地配置就可以挂载NFS到Pod中，而NFS中的数据是可以永久保存的，同时NFS支持同时写操作。 * glusterfs：同NFS一样是一种网络文件系统，Kubernetes可以将glusterfs挂载到Pod中，并进行永久保存。 * cephfs：一种分布式网络文件系统，可以挂载到Pod中，并进行永久保存。 * subpath：Pod的多个容器使用同一个Volume时，会经常用到。 * secret：密钥管理，可以将敏感信息进行加密之后保存并挂载到Pod中。 * persistentVolumeClaim：用于将持久化存储（PersistentVolume）挂载到Pod中。 除了以上几种Volume类型，Kubernetes还支持很多类型的Volume，详细可以参考：[https://kubernetes.io/docs/concepts/storage/](https://kubernetes.io/docs/concepts/storage/) ### 1.3.7 **PersistentVolume(PV) 持久化存储卷** kubernetes存储卷的分类太丰富了，每种类型都要写相应的接口与参数才行，这就让维护与管理难度加大，PersistentVolume(PV)是集群之中的一块网络存储，跟 Node 一样，也是集群的资源。PV是配置好的一段存储(可以是任意类型的存储卷)，将网络存储共享出来,配置定义成PV。PersistentVolume (PV)和PersistentVolumeClaim (PVC)提供了方便的持久化卷， PV提供网络存储资源，而PVC请求存储资源并将其挂载到Pod中，通过PVC用户不需要关心具体的volume实现细节,只需要关心使用需求。 ![image.png](../../img/kubernetes/kubernetes_spark/fbf2a1687a1e4d3daf73600c19fccaa5.png) ### 1.3.8 **ConfigMap** ConfigMap用于保存配置数据的键值对，可以用来保存单个属性，也可以用来保存配置文件，实现对容器中应用的配置管理，可以把ConfigMap看作是一个挂载到pod中的存储卷。ConfigMap跟secret很类似，但它可以更方便地处理不包含敏感信息的明文字符串。 ### 1.3.9 **Secret** Sercert-密钥解决了密码、token、密钥等敏感数据的配置问题，而不需要把这些敏感数据暴露到镜像或者Pod Spec中。 ### 1.3.10 **ServiceAccount** Service account是为了方便Pod里面的进程调用Kubernetes API或其他外部服务而设计的。Service Account为服务提供了一种方便的认证机制，但它不关心授权的问题。可以配合RBAC(Role Based Access Control)来为Service Account鉴权，通过定义Role、RoleBinding、ClusterRole、ClusterRoleBinding来对sa进行授权。 ## 1.4 **Kubernetes 集群搭建环境准备** 这里使用kubeadm部署工具来进行部署Kubernetes。Kubeadm是为创建Kubernetes集群提供最佳实践并能够“快速路径”构建kubernetes集群的工具。它能够帮助我们执行必要的操作，以获得最小可行的、安全的集群，并以用户友好的方式运行。 ### 1.4.1 **节点划分** kubernetes 集群搭建节点分布： | **节点IP** | **节点名称** | **Master** | **Worker** | | ---------------- | ------------------ | ---------------- | ---------------- | | 192.168.179.4 | node1 | ★ | | | 192.168.179.5 | node2 | | ★ | | 192.168.179.6 | node3 | | ★ | | 192.168.179.7 | node4 | | | | 192.168.179.8 | node5 | | | ### 1.4.2 **升级内核** 升级操作系统内核，升级到3.10内核版本以上。这里所有主机均操作，包括node4,node5节点。 ``` #导入elrepo gpg key rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org #安装elrepo YUM源仓库 yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm #安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64 #设置grub2默认引导为0 grub2-set-default 0 #重新生成grub2引导文件 grub2-mkconfig -o /boot/grub2/grub.cfg #更新后，需要重启，使用升级的内核生效。 reboot #重启后，需要验证内核是否为更新对应的版本 uname -r 6.0.6-1.el7.elrepo.x86_64 ``` ### 1.4.3 **配置内核转发及网桥过滤** 在所有K8S主机配置。添加网桥过滤及内核转发配置文件： ``` vim /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ``` 加载br_netfilter模块： ``` #加载br_netfilter模块 modprobe br_netfilter #查看是否加载 lsmod | grep br_netfilter ``` 加载网桥过滤及内核转发配置文件： ``` sysctl -p /etc/sysctl.d/k8s.conf ``` ### 1.4.4 **安装ipset及ipvsadm** 所有主机均需要操作。主要用于实现service转发。 ``` #安装ipset及ipvsadm yum -y install ipset ipvsadm 配置ipvsadm模块加载方式，添加需要加载的模块 vim /etc/sysconfig/modules/ipvs.modules modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack 授权、运行、检查是否加载 chmod 755 /etc/sysconfig/modules/ipvs.modules bash /etc/sysconfig/modules/ipvs.modules lsmod | grep -e ip_vs -e nf_conntrack ``` ### 1.4.5 **关闭SWAP分区** 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a。永远关闭swap分区，需要重启操作系统。 ``` #永久关闭swap分区 ,在 /etc/fstab中注释掉下面一行 vim /etc/fstab #/dev/mapper/centos-swap swap swap defaults 0 0 #重启机器 reboot ``` ### 1.4.6 **安装docker** 所有集群主机均需操作。 获取docker repo文件 ``` wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ``` 查看docker可以安装的版本： ``` yum list docker-ce.x86_64 --showduplicates | sort -r ``` 安装docker:这里指定docker版本为20.10.9版本 ``` yum -y install docker-ce-20.10.9-3.el7 ``` > 如果安装过程中报错: ``` Error: Package: 3:docker-ce-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: container-selinux >= 2:2.74 Error: Package: docker-ce-rootless-extras-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: fuse-overlayfs >= 0.7 Error: Package: docker-ce-rootless-extras-20.10.9-3.el7.x86_64 (docker-ce-stable) Requires: slirp4netns >= 0.4 Error: Package: containerd.io-1.4.9-3.1.el7.x86_64 (docker-ce-stable) ``` > 缺少一些依赖，解决方式：在/etc/yum.repos.d/docker-ce.repo开头追加如下内容: ``` [centos-extras] name=Centos extras - $basearch baseurl=http://mirror.centos.org/centos/7/extras/x86_64 enabled=1 gpgcheck=0 ``` > 然后执行安装命令： ``` yum -y install slirp4netns fuse-overlayfs container-selinux ``` > 执行完以上之后，再次执行yum -y install docker-ce-20.10.9-3.el7安装docker即可。 设置docker 开机启动，并启动docker： ``` systemctl enable docker systemctl start docker ``` 查看docker版本 ``` docker version ``` 修改cgroup方式，并重启docker。 ``` vim /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } #重启docker systemctl restart docker ``` ### 1.4.7 **cri-docker安装** 从kubernetes 1.24开始，dockershim已经从kubelet中移除（dockershim 是 Kubernetes 的一个组件，主要目的是为了通过 CRI 操作 Docker），但因为历史问题docker却不支持kubernetes主推的CRI（容器运行时接口）标准，所以docker不能再作为kubernetes的容器运行时了，即从kubernetesv1.24开始不再使用docker了，默认使用的容器运行时是containerd。目前containerd比较新，可能存在一些功能不稳定的情况，所以这里我们使用容器运行时还是选择docker。 如果想继续使用docker的话，可以在kubelet和docker之间加上一个中间层cri-docker。cri-docker是一个支持CRI标准的shim（垫片）。一头通过CRI跟kubelet交互，另一头跟docker api交互，从而间接的实现了kubernetes以docker作为容器运行时。这里需要在全部节点执行cri-docker安装。 1) **下载cri-docker源码** 可以从[https://github.com/Mirantis/cri-dockerd/archive/refs/tags/v0.3.1.tar.gz地址下载cri-docker源码，然后使用go进行编译安装。](https://github.com/Mirantis/cri-dockerd/archive/refs/tags/v0.2.6.tar.gz地址下载cri-docker源码，然后使用go进行编译安装。) cri-docker源码下载完成后，上传到Master并解压，改名: ``` [root@node1 ~]# tar -zxvf ./cri-dockerd-0.2.6.tar.gz [root@node1 ~]# mv cri-dockerd-0.2.6 cri-dockerd ``` 2) **安装go** ``` [root@node1 ~]# wget https://storage.googleapis.com/golang/getgo/installer_linux [root@node1 ~]# chmod +x ./installer_linux [root@node1 ~]# ./installer_linux [root@node1 ~]# source ~/.bash_profile [root@node1 ~]# go version go version go1.19.3 linux/amd64 ``` 如果在线安装go安装不上，可以选择离线安装go，步骤如下： ``` #在资料中获取“go1.19.3.linux-amd64.tar.gz”并解压，或者通过wget下载: wget https://go.dev/dl/go1.19.3.linux-amd64.tar.gz [root@node1 ~]# tar -C /usr/local -zxvf go1.19.3.linux-amd64.tar.gz #解压完成后，配置环境变量 [root@node1 ~]# echo 'export PATH=$PATH:/usr/local/go/bin' >> /etc/profile #环境变量生效 [root@node1 ~]# source /etc/profile #检查go 版本 [root@node1 ~]# go version go version go1.19.3 linux/amd64 ``` 4) **编译安装cri-docker** ``` #进入 cri-dockerd 中，并创建目录bin [root@node1 ~]# cd cri-dockerd && mkdir bin #编译，大概等待1分钟 [root@node1 cri-dockerd]# go build -o bin/cri-dockerd #安装cri-docker，安装-o指定owner -g指定group -m指定指定权限 [root@node1 cri-dockerd]# mkdir -p /usr/local/bin [root@node1 cri-dockerd]# install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd #复制服务管理文件至/etc/systemd/system目录中 [root@node1 cri-dockerd]# cp -a packaging/systemd/* /etc/systemd/system #指定cri-dockerd运行位置 [root@node1 cri-dockerd]# sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service [root@node1 cri-dockerd]# systemctl daemon-reload #启动服务 [root@node1 cri-dockerd]# systemctl enable cri-docker.service [root@node1 cri-dockerd]# systemctl enable --now cri-docker ``` ## 1.5 **Kubernetes 基于Docker Runtime 集群部署** ### 1.5.1 **软件版本** 这里安装Kubernetes版本为1.25.3，在所有主机（node1,node2,node3）安装kubeadm，kubelet，kubectl。 * kubeadm：初始化集群、管理集群等。 * kubelet:用于接收api-server指令，对pod生命周期进行管理。 * kubectl:集群应用命令行管理工具。 ### 1.5.2 **准备阿里yum源** 每台k8s节点vim /etc/yum.repos.d/k8s.repo，写入以下内容： ``` [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ``` ### 1.5.3 **集群软件安装** ``` #查看指定版本 yum list kubeadm.x86_64 --showduplicates | sort -r yum list kubelet.x86_64 --showduplicates | sort -r yum list kubectl.x86_64 --showduplicates | sort -r #安装指定版本 yum -y install --setopt=obsoletes=0 kubeadm-1.25.3-0 kubelet-1.25.3-0 kubectl-1.25.3-0 ``` 安装过程有有如下错误： ``` Error: Package: kubelet-1.25.3-0.x86_64 (kubernetes) Requires: conntrack ``` 解决方式： ``` wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/Centos-7.repo yum install -y conntrack-tools ``` ### 1.5.4 **配置kubelet** 为了实现docker使用的cgroup driver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ``` #vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ``` 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 ``` systemctl enable kubelet ``` ### 1.5.5 **集群镜像准备** 只需要在node1 Master节点上执行如下下载镜像命令即可，这里先使用kubeadm查询下镜像。 ``` [root@node1 ~]#kubeadm config images list --kubernetes-version=v1.25.3 registry.k8s.io/kube-apiserver:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3 registry.k8s.io/pause:3.8 registry.k8s.io/etcd:3.5.4-0 registry.k8s.io/coredns/coredns:v1.9.3 ``` 编写下载镜像脚本image_download.sh： ``` #!/bin/bash images_list=' registry.k8s.io/kube-apiserver:v1.25.3 registry.k8s.io/kube-controller-manager:v1.25.3 registry.k8s.io/kube-scheduler:v1.25.3 registry.k8s.io/kube-proxy:v1.25.3 registry.k8s.io/pause:3.8 registry.k8s.io/etcd:3.5.4-0 registry.k8s.io/coredns/coredns:v1.9.3 ' for i in $images_list do docker pull $i done docker save -o k8s-1-25-3.tar $images_list ``` 以上脚本准备完成之后，执行命令：sh image_download.sh 进行镜像下载 注意：下载时候需要科学上网，否则下载不下来。也可以使用资料中的“k8s-1-25-3.tar”下载好的包。 > #如果下载不下来，使用资料中打包好的k8s-1-25-3.tar，将镜像导入到docker中 > > docker load -i k8s-1-25-3.tar ### 1.5.6 **集群初始化** 只需要在Master节点执行如下初始化命令即可。 ``` [root@node1 ~]# kubeadm init --kubernetes-version=v1.25.3 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.179.4 --cri-socket unix:///var/run/cri-dockerd.sock ``` 注意：--apiserver-advertise-address=192.168.179.4 要写当前主机Master IP > 初始化过程中报错： ``` [init] Using Kubernetes version: v1.25.3 [preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR CRI]: container runtime is not running: output: E1102 20:14:29.494424 10976 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"time="2022-11-02T20:14:29+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alp ha2.RuntimeService", error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher ``` 执行如下命令，重启containerd后，再次init 初始化。 ``` [root@node1 ~]# rm -rf /etc/containerd/config.toml [root@node1 ~]# systemctl restart containerd ``` 初始化完成后，结果如下： ``` Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 ``` ### 1.5.7 **集群应用客户端管理集群文件准备** 参照初始化的内容来执行如下命令： ``` [root@node1 ~]# mkdir -p $HOME/.kube [root@node1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@node1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@node1 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf ``` ### 1.5.8 **集群网络准备** #### 1.5.8.1 **calico安装** K8s使用calico部署集群网络,安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico。 只需要在Master节点安装即可。 ``` #下载operator资源清单文件 wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml --no-check-certificate #应用资源清单文件，创建operator kubectl create -f tigera-operator.yaml #通过自定义资源方式安装 wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml --no-check-certificate #修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 # vim custom-resources.yaml 【修改和增加以下加粗内容】 apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. calicoNetwork: # Note: The ipPools section cannot be modified post-install. ipPools: - blockSize: 26 cidr: 10.244.0.0/16 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all() nodeAddressAutodetectionV4: interface: ens.* #应用清单文件 kubectl create -f custom-resources.yaml #监视calico-sysem命名空间中pod运行情况 watch kubectl get pods -n calico-system [root@node1 ~]# watch kubectl get pods -n calico-system Every 2.0s: kubectl get pods -n calico-system Thu Nov 3 14:14:30 2022 NAME READY STATUS RESTARTS AGE calico-kube-controllers-65648cd788-flmk4 1/1 Running 0 2m21s calico-node-chnd5 1/1 Running 0 2m21s calico-node-kc5bx 1/1 Running 0 2m21s calico-node-s2cp5 1/1 Running 0 2m21s calico-typha-d76595dfb-5z6mg 1/1 Running 0 2m21s calico-typha-d76595dfb-hgg27 1/1 Running 0 2m19s #已经全部运行 [root@node1 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-65648cd788-ktjrh 1/1 Running 0 110m calico-node-dvprv 1/1 Running 0 110m calico-node-nhzch 1/1 Running 0 110m calico-node-q44gh 1/1 Running 0 110m calico-typha-6bc9d76554-4bv77 1/1 Running 0 110m calico-typha-6bc9d76554-nkzxq 1/1 Running 0 110m #查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 [root@node1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-565d847f94-bjtlh 1/1 Running 0 19h coredns-565d847f94-wlxmf 1/1 Running 0 19h etcd-node1 1/1 Running 0 19h kube-apiserver-node1 1/1 Running 0 19h kube-controller-manager-node1 1/1 Running 0 19h kube-proxy-bgpz2 1/1 Running 0 19h kube-proxy-jlltp 1/1 Running 0 19h kube-proxy-stfrx 1/1 Running 0 19h kube-scheduler-node1 1/1 Running 0 19h ``` #### 1.5.8.2 **calico客户端安装** 主要用来验证k8s集群节点网络是否正常。这里只需要在Master节点安装就可以。 ``` #下载二进制文件，注意，这里需要检查calico 服务端的版本，客户端要与服务端版本保持一致，这里没有命令验证calico的版本，所以安装客户端的时候安装最新版本即可。 curl -L https://github.com/projectcalico/calico/releases/download/v3.25.0/calicoctl-linux-amd64 -o calicoctl #安装calicoctl mv calicoctl /usr/bin/ #为calicoctl添加可执行权限 chmod +x /usr/bin/calicoctl #查看添加权限后文件 ls /usr/bin/calicoctl #查看calicoctl版本 [root@node1 ~]# calicoctl version Client Version: v3.25.0 Git commit: 3f7fe4d29 Cluster Version: v3.25.0 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm 通过~/.kube/config连接kubernetes集群，查看已运行节点 [root@node1 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME node1 ``` ### 1.5.9 **集群工作节点添加** 这里在node2,node3 worker节点上执行命令，将worker节点加入到k8s集群。 ``` [root@node2 ~]# kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 --cri-socket unix:///var/run/cri-dockerd.sock [root@node3 ~]# kubeadm join 192.168.179.4:6443 --token tpynmm.7picylv5i83q9ghw \ --discovery-token-ca-cert-hash sha256:2924026774d657b8860fbac4ef7698e90a3811137673af45e533c91e567a1529 --cri-socket unix:///var/run/cri-dockerd.sock ``` 注意：如果以上node2,node3 Worker节点已经错误的加入到Master节点，需要再Worker节点执行如下命令清除对应的信息，然后再次加入即可。 ``` #重置kubeadm [root@node2 ~]# kubeadm reset -cri-socket unix:///var/run/cri-dockerd.sock #删除k8s配置文件和证书文件 [root@node2 kubernetes]# rm -f /etc/kubernetes/kubelet.conf [root@node2 kubernetes]# rm -f /etc/kubernetes/pki/ca.crt #重置kubeadm [root@node3 ~]# kubeadm reset #删除k8s配置文件和证书文件 [root@node3 kubernetes]# rm -f /etc/kubernetes/kubelet.conf [root@node3 kubernetes]# rm -f /etc/kubernetes/pki/ca.crt ``` 此外如果忘记了node join加入master节点的命令，可以按照以下步骤操作： ``` #查看discovery-token-ca-cert-hash [root@node1 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' #查看token [root@node1 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS1945mk.ved91lifrc8l0zj9 23h 2022-11-10T08:14:46Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token #节点加入集群 [root@node1 ~]# kubeadm join 192.168.179.4:6443 --token 查询出来的token \ --discovery-token-ca-cert-hash 查询出来的hash码 ``` 在master节点上操作，查看网络节点是否添加 ``` [root@node1 ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME node1 node2 node3 ``` ### 1.5.10 **验证集群可用性** 使用命令查看所有的节点： ``` [root@node1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 20h v1.25.3 node2 Ready 20h v1.25.3 node3 Ready 20h v1.25.3 ``` 查看集群健康情况： ``` [root@node1 ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-0 Healthy {"health":"true","reason":""} scheduler Healthy ok controller-manager Healthy ok ``` 查看kubernetes集群pod运行情况: ``` [root@node1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-565d847f94-bjtlh 1/1 Running 0 20h coredns-565d847f94-wlxmf 1/1 Running 0 20h etcd-node1 1/1 Running 0 20h kube-apiserver-node1 1/1 Running 0 20h kube-controller-manager-node1 1/1 Running 0 20h kube-proxy-bgpz2 1/1 Running 1 20h kube-proxy-jlltp 1/1 Running 1 20h kube-proxy-stfrx 1/1 Running 0 20h kube-scheduler-node1 1/1 Running 0 20h ``` 查看集群信息: ``` [root@node1 ~]# kubectl cluster-info Kubernetes control plane is running at https://192.168.179.4:6443 CoreDNS is running at https://192.168.179.4:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` ### 1.5.11 **K8s集群其他一些配置** 当在Worker节点上执行kubectl命令管理时会报如下错误： ``` The connection to the server localhost:8080 was refused - did you specify the right host or port? ``` 只要把master上的管理文件/etc/kubernetes/admin.conf拷贝到Worker节点的$HOME/.kube/config就可以让Worker节点也可以实现kubectl命令管理。 ``` #在Worker节点创建.kube目录 [root@node2 ~]# mkdir /root/.kube [root@node3 ~]# mkdir /root/.kube #在master节点做如下操作 [root@node1 ~]# scp /etc/kubernetes/admin.conf node2:/root/.kube/config [root@node1 ~]# scp /etc/kubernetes/admin.conf node3:/root/.kube/config #在worker 节点验证 [root@node2 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 24h v1.25.3 node2 Ready 24h v1.25.3 node3 Ready 24h v1.25.3 [root@node3 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready control-plane 24h v1.25.3 node2 Ready 24h v1.25.3 node3 Ready 24h v1.25.3 ``` 此外，无论在Master节点还是Worker节点使用kubenetes 命令时，默认不能自动补全，例如：kubectl describe 命令中describe不能自动补全，使用非常不方便，那么这里配置命令自动补全功能。 在所有的kubernetes节点上安装bash-completion并source执行，同时配置下开机自动source，每次开机能自动补全命令。 ``` #安装bash-completion 并 source yum install -y bash-completion source /usr/share/bash-completion/bash_completion kubectl completion bash > ~/.kube/completion.bash.inc source '/root/.kube/completion.bash.inc' #实现用户登录主机自动source ,自动使用命令补全 vim ~/.bash_profile 【加入加粗这一句】 # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs source '/root/.kube/completion.bash.inc' PATH=$PATH:$HOME/bin export PATH ``` 默认K8S我们只要设置了systemctl enable kubelet 后，会在开机自动启动K8S集群，如果想要停止kubernetes集群，我们可以通过systemctl stop kubelet 命令停止集群，但是必须先将节点上的docker停止，命令如下： ### 1.5.12 **K8s集群启停** ``` systemctl stop docker ``` 然后再停止k8s集群： ``` systemctl stop kubelet ``` 启动Kubernetes集群步骤如下： ``` # 先启动docker systemctl start docker # 再启动kubelet systemctl start kubelet ``` ## 1.6 **Kubernetes集群UI及主机资源监控** ### 1.6.1 **Kubernetes dashboard作用** 通过Kubernetes dashboard能够直观了解Kubernetes集群中运行的资源对象，通过dashboard可以直接管理（创建、删除、重启等操作）资源对象。 ### 1.6.2 **获取Kubernetes dashboard资源清单文件** 这里只需要在Kubernetes Master节点上来下载应用资源清单文件即可。这里去github.com 搜索“kubernetes dashboard”即可，找到匹配Kubernetes 版本的dashboard，下载对应版本的dashboard yaml文件。 ![image.png](../../img/kubernetes/kubernetes_spark/0cddbf7f85004d91afec4b37a36a382d.png) ![image.png](../../img/kubernetes/kubernetes_spark/8c8cde92fb3c4208a8da4ae772c5944c.png) ![image.png](../../img/kubernetes/kubernetes_spark/d1c47c507a014c2cab5388b06150c29f.png) ``` [root@node1 ~]# mkdir kube-dashboard [root@node1 ~]# cd kube-dashboard/ [root@node1 kube-dashboard]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml ``` 对应yaml文件下载完成后，为了方便后续在容器主机上访问，在yaml文件中添加对应的NodePort类型、端口以及修改登录kubernetes dashboard的用户。 ``` #vi recommended.yaml 【只需要添加或修改以下加粗部分】 ... ... kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30000 selector: k8s-app: kubernetes-dashboard ... .... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard ``` 注意：一定要把原来kind为ClusterRole下的name对应值kubernetes-dashboard修改为cluster-admin，不然进入UI后会报错。 ``` #部署Kubernetes dashboard [root@node1 kube-dashboard]# kubectl apply -f recommended.yaml #查看部署是否成功，出现kubenetes-dashboard命名空间即可。 [root@node1 kube-dashboard]# kubectl get ns NAME STATUS AGE calico-apiserver Active 5h33m calico-system Active 5h35m default Active 23h kube-node-lease Active 23h kube-public Active 23h kube-system Active 23h kubernetes-dashboard Active 6s tigera-operator Active 5h36m [root@node1 kube-dashboard]# kubectl get pod,svc -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE pod/dashboard-metrics-scraper-64bcc67c9c-gsqsn 1/1 Running 0 112s pod/kubernetes-dashboard-5c8bd6b59-x4p8r 1/1 Running 0 112s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dashboard-metrics-scraper ClusterIP 10.106.178.119 8000/TCP 112s service/kubernetes-dashboard NodePort 10.96.4.46 443:30000/TCP 113s ``` ### 1.6.3 **访问Kubernetes dashboard** WebUI访问Kubernetes dashboard：[https://192.168.179.4:3000](http://192.168.179.4:3000)0 ![image.png](../../img/kubernetes/kubernetes_spark/093e784b4abf42f098dfc184a36789c0.png) 选择“Token”，使用如下命令获取Kubernetes的Token: ``` #注意最后的kubernetes-dashboard 为部署dashboard创建的serviceaccounts [root@node1 kube-dashboard]# kubectl -n kubernetes-dashboard create token kubernetes-dashboard eyJhbGciOiJSUzI1NiIsImtpZCI6IlFDVlB4Wng3REtPNGZhd05sYnJvbFBWbE9iS2pDYmtEYUZyQ2VaSjA0MjAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjY3NDgxODQ0LCJpYXQiOjE2Njc0NzgyNDQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInVpZCI6ImY3ZGQ0YTI1LTEwOTAtNDYyZC04N2JhLTM4NjNlM2Q3MjQxNCJ9fSwibmJmIjoxNjY3NDc4MjQ0LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQifQ.b0S3YAJrFTUb-pgiPLp3kuB510sL7r9LPvmeO5kXM86ZRJhbGOFsD-CK-ONQnDF2EVAg76YsV_I7Afv_P_RkSspfy0AnBDUFj-LBufocX1cofCHc1_dErVbCQ5MvUsnw67PvpdcZMWuAndYhMVorOIxOc_RxhUM6tre3kuZJ40r2W-8Kbgd4b3HvLeaE2gJNTofn5ChYLkDd7TQYqRtmZN14l6CFZMUSl1dHqSuWhUncNHELhI8uRRD1pfFmMlYrqkZOTqzkw5_czMFrE9yIFKktMqT3wpvRVWFzYZFd9SpGMoQtshKjR3h508N-KG2Ob3PQYbvpBdoap2UjOjQJVg ``` 将以上生成的Token复制粘贴到登录的WebUI页面中登录Kubernetes DashBoard。注意：以上token复制时不要有空格。 ## 1.7 **Kuberneters 部署案例** 这里为了强化对Kubernetes集群的理解，我们基于Kubernetes集群进行部署nginx服务，nginx服务我们设置2个副本，同时将nginx服务端口80暴露到宿主机上。在kubernetes中，我们可以通过WebUI来添加服务，也可以在命令行中通过应用yaml来部署服务，下面以命令行部署nginx服务为例： 1) **创建资源清单文件** * **nginx.yaml** **：** ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test spec: selector: matchLabels: app: nginx env: test owner: rancher replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx env: test owner: rancher spec: containers: - name: nginx-test image: nginx:1.19.9 ports: - containerPort: 80 ``` * **nginx-service.yaml** **：** ``` apiVersion: v1 kind: Service metadata: name: nginx-test labels: run: nginx spec: type: NodePort ports: - port: 80 protocol: TCP nodePort: 30080 selector: owner: rancher ``` 2) **应用资源清单文件** ``` [root@node1 nginx-test]# kubectl apply -f nginx.yaml [root@node1 nginx-test]# kubectl apply -f nginx-service.yaml ``` 3) **验证** ``` [root@node1 test]# kubectl get all NAME READY STATUS RESTARTS AGE pod/nginx-test-74845c57fb-7tl86 1/1 Running 0 45s pod/nginx-test-74845c57fb-qjc6d 1/1 Running 0 45s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 123m service/nginx-test NodePort 10.107.171.29 80:30204/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-test 2/2 2 2 45s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-test-74845c57fb 2 2 2 45s ``` 4) **访问验证** 访问任意kubernetes集群的节点30080端口查看nginx服务是正常，例如：浏览器输入node1:30080 ![image.png](../../img/kubernetes/kubernetes_spark/ce75b5426f07418d8cd0015726d6ac9e.png) 5) **删除nginix服务** ``` [root@node1 nginx-test]# kubectl delete -f nginx-service.yaml [root@node1 nginx-test]# kubectl delete -f nginx.yaml ``` ## 1.8 **搭建nfs服务** 后续基于Native Kubernetes部署Spark后，提交任务的jar包可以是本地编写的jar，在提交命令时，jar包存放位置不能是单独的本地位置，因为executor以pod方式运行在找对应的jar时无法获取提交任务节点的jar，提交任务的jar路径可以放在hdfs或者nfs路径中，但是需要executor pod能访问到。后续提交jar时选择nfs服务器，所以这里需要搭建nfs服务。 nfs搭建分为nfs-server和nfs-client两部分，节点划分如下： | **节点IP** | **节点名称** | **nfs-server** | **nfs-client** | | ---------------- | ------------------ | -------------------- | -------------------- | | 192.168.179.4 | node1 | | ★ | | 192.168.179.5 | node2 | | ★ | | 192.168.179.6 | node3 | | ★ | | 192.168.179.7 | node4 | ★ | ★ | | 192.168.179.8 | node5 | | | 详细搭建nfs步骤如下： 1. **在**nfs-server ****节点上安装**** nfs**并配置目录** ``` #node4节点安装nfs-server [root@node4 harbor]# yum install nfs-utils -y #node4节点创建/localfolder 目录并配置 [root@node4 ~]# mkdir -p /localfolder [root@node4 ~]# vim /etc/exports /localfolder *(rw,no_root_squash,sync) #重启并设置开机启动 [root@node4 ~]# systemctl restart nfs-server [root@node4 ~]# systemctl enable nfs-server ``` 2. **在**nfs-client ****节点上安装**** nfs ``` #在node1~node3节点上安装nfs [root@node1 ~]# yum install nfs-utils -y [root@node2 ~]# yum install nfs-utils -y [root@node3 ~]# yum install nfs-utils -y ``` 3. **所有节点验证**nfs**可用性** ``` #node1~node4节点验证nfs可用性 showmount -e 192.168.179.7 ``` 至此，nfs集群搭建完成。 ## 1.9**Harbor构建私有镜像仓库** 在企业中使用Kubernetes时，为了方便下载和管理镜像一般都会构建本地的私有镜像仓库。Harbor正是一个用于存储镜像的企业级Registry服务，我们可以通过Harbor来存储容器镜像构建企业本地的镜像仓库。 这里我们选择一台Kubernetes集群外的一台节点搭建Harbor，当然也可以选择Kubernetes集群内的一台节点。 | **节点IP** | **节点名称** | **Harbor** | | ---------------- | ------------------ | ---------------- | | 192.168.179.4 | node1 | | | 192.168.179.5 | node2 | | | 192.168.179.6 | node3 | | | 192.168.179.7 | node4 | ★ | | 192.168.179.8 | node5 | | 1) **安装docker** ``` #准备docker-ce对应的repo文件 [root@node4 ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo #安装docker-ce [root@node4 ~]# yum -y install docker-ce-20.10.9-3.el7 #设置docker开机启动，并启动docker [root@node4 ~]# systemctl enable docker [root@node4 ~]# systemctl start docker ``` 2) **安装docker-compose** 后续需要docker-compose编排工具进行harbor的启停，这里需要安装docker-compose。 ``` #下载docker-compose的二进制文件,改文件如果下载不下来可以参照资料中“docker-compose-Linux-x86_64”文件 [root@node4 ~]# wget https://github.com/docker/compose/releases/download/1.25.0/docker-compose-Linux-x86_64 #移动二进制文件到/usr/bin目录，并更名为docker-compose [root@node4 ~]# mv docker-compose-Linux-x86_64 /usr/bin/docker-compose # 为二进制文件添加可执行权限 chmod +x /usr/bin/docker-compose #以上操作完成后，查看docker-compse版本 [root@node4 ~]# docker-compose version docker-compose version 1.25.0, build 0a186604 docker-py version: 4.1.0 CPython version: 3.7.4 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019 ``` 3. **获取Harbor安装文件并解压** ``` #下载harbor离线安装包，如果下载不下来参考资料中“harbor-offline-installer-v2.5.1.tgz”文件 [root@node4 ~]# wget https://github.com/goharbor/harbor/releases/download/v2.5.1/harbor-offline-installer-v2.5.1.tgz #解压下载好的安装包 [root@node4 ~]# tar -zxvf ./harbor-offline-installer-v2.5.1.tgz ``` 4. **修改Harbor配置文件** 搭建Harbor本地镜像仓库时，可以使用ssl安全访问私有镜像仓库，也可以不使用，直接使用ip访问。 * **准备Harbor的harbor.yml配置文件：** ``` [root@node4 ~]# cd harbor [root@node4 harbor]# mv harbor.yml.tmpl harbor.yml ``` * **修改harbor.yml文件内容如下：** ``` # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: 192.168.179.7 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config #https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /root/harbor/6864844_kubemsb.com.pem # private_key: /root/harbor/6864844_kubemsb.com.key # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: 123456 ... ... ``` 注意harbor.yml文件中只需要配置hostname、certificate、private_key、harbor_admin_password即可。 5. **执行预备脚本并安装** ``` #执行预备脚本，检查安装所需镜像 [root@node4 harbor]# ./prepare prepare base dir is set to /root/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/core/env Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/registry/passwd Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/db/env Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /config/jobservice/config.yml Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /data/secret/keys/secretkey Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir #执行安装脚本 [root@node4 harbor]# ./install.sh ... [Step 5]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating registryctl ... done Creating harbor-db ... done Creating registry ... done Creating harbor-portal ... done Creating redis ... done Creating harbor-core ... done Creating nginx ... done Creating harbor-jobservice ... done ✔ ----Harbor has been installed and started successfully.---- ``` 6. **验证Harbor情况** ``` #使用docker ps 命令检查是否是运行9个镜像，如下 [root@node4 harbor]# docker ps CONTAINER ID IMAGE a81fbd05dc13 goharbor/harbor-jobservice:v2.5.1 c374cf3d741a goharbor/nginx-photon:v2.5.1 152c165b0804 goharbor/harbor-core:v2.5.1 4e48926df8b0 goharbor/redis-photon:v2.5.1 6d514441a600 goharbor/harbor-db:v2.5.1 0a170f716955 goharbor/registry-photon:v2.5.1 a8d99c7b2421 goharbor/harbor-portal:v2.5.1 2b808612f108 goharbor/harbor-registryctl:v2.5.1 69c2c7818e6a goharbor/harbor-log:v2.5.1 ``` 注意：安装harbor后，查看对应的 docker images 是否是9个，不是9个需要重新启动harbor。重启Harbor的命令如下： ``` [root@node4 harbor]# cd /root/harbor #停止harbor服务 [root@node4 harbor]# docker-compose down #启动harbor服务 [root@node4 harbor]# docker-compose up -d ``` 7. **配置Kubernetes节点及harbor节点访问harbor服务** 后续Kubernetes各个节点（包括harbor节点本身）需要连接Harbor上传或者下载镜像，所以这里配置各个节点访问harbor。kubernetes集群所有节点配置harbor仓库（包括harbor节点本身也要配置）： ``` #kubernetes 集群各个节点配置/etc/docker/daemon.json文件，追加"insecure-registries": ["https://www.kubemsb.com"] vim /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "insecure-registries": ["https://www.kubemsb.com"] } #harbor节点配置 /etc/docker/daemon.json文件 { "insecure-registries": ["http://192.168.179.7"] } ``` 以上配置完成后，配置的每台节点需要重启docker，重点需要注意harbor节点docker重启后是否是对应有9个image，如果不是9个需要使用docker-compose down 停止harbor集群后再次使用命令docker-compse up -d 启动。 ``` systemctl restart docker ``` 检查每个节点是否能正常连接Harbor，这里每台节点连接harbor前必须需要执行如下命令登录下harbor私有镜像仓库。 ``` docker login 192.168.179.7 输入用户：admin 输入密码：123456 ``` 8. **访问Harbor UI界面** 然后浏览器访问harbor(只能是ip访问)，用户名为admin，密码为配置的123456。 ![image.png](../../img/kubernetes/kubernetes_spark/3385b90f53db463ba6c742dc2b08ba29.png) ![image.png](../../img/kubernetes/kubernetes_spark/b73afe0351014104a0bebd6a4ddfd50e.png) 9. **测试Harbor** 使用docker下载nginx镜像并上传至harbor，然后通过docker从harbor中下载该上传镜像，测试是否能正常从harbor下载存储管理的镜像。具体操作如下： ``` #docker 下载nginx镜像 [root@node1 ~]# docker pull nginx:1.15-alpine #检查镜像 [root@node1 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx 1.15-alpine dd025cdfe837 3 years ago 16.1MB #对nginx镜像进行标记打tag [root@node1 ~]# docker tag nginx:1.15-alpine 192.168.179.7/library/nginx:v1 #检查镜像 [root@node1 software]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx 1.15-alpine dd025cdfe837 3 years ago 16.1MB www.kubemsb.com/library/nginx v1 dd025cdfe837 3 years ago 16.1MB #推送本地镜像到harbor镜像仓库 [root@node1 ~]# docker push www.kubemsb.com/library/nginx:v1 ``` 将本地镜像推送到harbor镜像仓库后，可以通过WebUI查看对应内容： ![image.png](../../img/kubernetes/kubernetes_spark/ed61cd24eb7a4b8aa9c55545d6310941.png) 可以在本地任何一台节点上从Harbor镜像仓库中下载镜像到本地： ``` [root@node2 ~]# docker pull 192.168.179.7/library/nginx:v1 ``` ## 1.10 **Spark基于Kubernetes部署优势** Spark是新一代分布式内存计算框架，Apache开源的顶级项目。相比于Hadoop Map-Reduce计算框架，Spark将中间计算结果保留在内存中，速度提升10~100倍；同时它还提供更丰富的算子，采用弹性分布式数据集(RDD)实现迭代计算，更好地适用于数据挖掘、机器学习算法，极大提升开发效率。相比于在物理机上部署，在Kubernetes集群上部署Spark集群，具有以下优势： - **快速部署** ：安装1000台级别的Spark集群，在Kubernetes集群上只需设定worker副本数目replicas=1000，即可一键部署。 - **快速升级** ：升级Spark版本，只需替换Spark镜像，一键升级。 - **弹性伸缩** ：需要扩容、缩容时，自动修改worker副本数目replicas即可。 - **高一致性** ：各个Kubernetes节点上运行的Spark环境一致、版本一致。 - **高可用性** ：如果Spark所在的某些node或pod死掉，Kubernetes会自动将计算任务，转移到其他node或创建新pod。 - **强隔离性** ：通过设定资源配额等方式，可与Web、大数据应用部署在同一集群，提升机器资源使用效率，从而降低服务器成本。 ## 1.11 **Spark角色介绍** 下图是Spark框架中对应的角色，每个角色作用如下: ![image.png](../../img/kubernetes/kubernetes_spark/8457fc89c24741a3a22e4f90796bbb50.png) - Client ：客户端进程，负责提交作业到Master。 - Master ：Standalone模式中主节点，负责接收Client提交的作业，管理Worker，并命令Worker启动Driver和Executor。 - Worker ：Standalone模式中slave节点上的守护进程，负责管理本节点的资源，定期向Master汇报心跳，接收Master的命令，启动Driver和Executor。 - Driver ： 一个Spark作业运行时包括一个Driver进程，也是作业的主进程，负责作业的解析、生成Stage并调度Task到Executor上，包括 DAGScheduler ， TaskScheduler 。 ## 1.12 **Spark任务提交模式** Spark任务可以运行在多种资源调度框架上，可以基于Standalone、Yarn、Kubernetes进行任务调度，Standalone、Yarn、Kubernetes中运行Spark任务时这些框架主要提供任务所需的计算资源，任务运行也都有Client模式和Cluster模式。 下面以Spark 基于 Standalone集群提交任务为例来介绍Spark任务提交的模式,Spark Standalone集群是Spark自带的资源调度框架，资源由Spark自己管理。 ### 1.12.1**Standalone集群搭建** Standalone集群中有Master和Worker，Standalone集群搭建节点划分如下： | **节点IP** | **节点名称** | **Master** | **Worker** | **客户端** | | ---------------- | ------------------ | ---------------- | ---------------- | ---------------- | | 192.168.179.4 | node1 | ★ | | | | 192.168.179.5 | node2 | | ★ | | | 192.168.179.6 | node3 | | ★ | | | 192.168.179.7 | node4 | | | ★ | | 192.168.179.8 | node5 | | | | 详细的搭建步骤如下： 1. **下载** Spark **安装包** 这里在Spark官网中现在Spark安装包，安装包下载地址：https://www.apache.org/dyn/closer.lua/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz/ 2. **上传、解压、修改名称** 这里将下载好的安装包上传至node1节点的""路径，进行解压，修改名称： ``` #解压 [root@node1 ~]# tar -zxvf /software/spark-3.3.1-bin-hadoop3.tgz -C /software/ #修改名称 [root@node1 software]# mv spark-3.3.1-bin-hadoop3 spark-3.3.1 ``` 3. **配置**conf **文件** ``` #进入conf路径 [root@node1 ~]# cd /software/spark-3.3.1/conf/ #改名 [root@node1 conf]# cp spark-env.sh.template spark-env.sh [root@node1 conf]# cp workers.template workers #配置spark-env.sh，在改文件中写入如下配置内容 export SPARK_MASTER_HOST=node1 export SPARK_MASTER_PORT=7077 export SPARK_WORKER_CORES=3 export SPARK_WORKER_MEMORY=3g #配置workers，在workers文件中写入worker节点信息 node2 node3 ``` 将以上配置好Spark解压包发送到node2、node3节点上： ``` [root@node1 ~]# cd /software/ [root@node1 software]# scp -r ./spark-3.3.1 node2:/software/ [root@node1 software]# scp -r ./spark-3.3.1 node3:/software/ ``` 4. **启动集群** 在node1节点上进入"$SPARK\_HOME/sbin"目录中执行如下命令启动集群： ``` #启动集群 [root@node1 ~]# cd /software/spark-3.3.1/sbin/ [root@node1 sbin]# ./start-all.sh ``` 5. **访问**webui Spark集群启动完成之后，可以在浏览器中输入"http://node1:8080"来查看Spark WebUI： ![image.png](../../img/kubernetes/kubernetes_spark/7bfb13b30d5e4efbb09487fb1c22dbfc.png) 在浏览器中输入地址出现以上页面，并且对应的worker状态为Alive，说明Spark Standalone集群搭建成功。 6. Spark Pi 任务提交测试 这里在客户端提交Spark PI任务来进行任务测试，首先在node4节点上传Spark安装包，然后进行任务提交，这里只需要将Spark安装包在node4节点进行解压即可，操作如下： ``` #上传安装包，解压 [root@node4 ~]# cd /software/ [root@node4 software]# tar -zxvf ./spark-3.3.1-bin-hadoop3.tgz #提交Spark Pi任务 [root@node4 ~]# cd /software/spark-3.3.1-bin-hadoop3/bin/ [root@node4 bin]# ./spark-submit --master spark://node1:7077 --class org.apache.spark.examples.SparkPi ../examples/jars/spark-examples_2.12-3.3.1.jar ... Pi is roughly 3.1410557052785264 ... ``` ### 1.12.2 **Client模式** Spark基于Standalone模式提交任务流程如下： ![image.png](../../img/kubernetes/kubernetes_spark/b0c38807aa9f411098af69ba48b81072.png) 1. client模式提交任务后，会在客户端启动Driver进程。 2. Driver会向Master申请启动Application启动的资源。 3. Master收到请求之后会在对应的Worker节点上启动Executor 4. Executor启动之后，会注册给Driver端，Driver掌握一批计算资源。 5. Driver端将task发送到worker端执行,worker将task执行结果返回到Driver端。 client模式适用于测试调试程序，Driver进程是在客户端启动的，这里的客户端就是指提交应用程序的当前节点。在Driver端可以看到task执行的情况。生产环境下不能使用client模式，原因是假设要提交100个application到集群运行，Driver每次都会在client端启动，那么就会导致客户端100次网卡流量暴增的问题。client模式适用于程序测试，不适用于生产环境，在客户端可以看到task的执行和结果。 standalone-client模式提交任务命令如下： ``` spark-submit --master spark://spark_master_ip:7077 --class xxx jar 参数 或者 spark-submit --master spark://spark_master_ip:7077 --deploy-mode client --class xxx jar 参数 ``` ### 1.12.3 **Cluster模式** Spark基于Standalone cluster模式提交任务流程如下: ![image.png](../../img/kubernetes/kubernetes_spark/638b6c046c5b4ae9b782e60cf95e7038.png) 1. cluster模式提交应用程序后，会向Master请求启动Driver. 2. Master接受请求，随机在集群一台节点启动Driver进程。 3. Driver启动后为当前的应用程序申请资源。 4. Driver端发送task到worker节点上执行。 5. worker将执行情况和执行结果返回给Driver端。 Spark基于Standalone-Cluster模式提交任务，Driver是随机在一台Worker节点启动的，如果在客户端提交多个Spark application，每个application 都会启动独立的Driver，这些Driver是分散到集群中启动，如果一次提交多个任务，会将单节点网卡流量激增问题分散到集群中，这种模式适用于生产环境，在客户端看不到task执行和结果。可以去webui中查看日志和结果。 standalone-client模式提交任务命令如下： ``` spark-submit --master spark://spark_master_ip:7077 --deploy-mode cluster --class xxx jar 参数 ``` ## 1.13 **Spark 基于Kubernetes部署** Spark基于Kubernetes部署分为两种方式。第一种是Spark Standalone集群中的各个角色(Master、Worker)以pod的方式运行在Kubernetes集群中，这种方式这里称作Standalone on Kubernetes 部署；另外一种方式是将Kuberntes 看成资源调度平台，类似于Yarn，直接基于Kubernetes提交任务运行，这种方式这里成为Spark Native Kubernetes部署。 以上两种方式，无论哪一种方式，目前Spark 官方没有提供构建好的Spark镜像，需要我们自己构建镜像。在介绍每种部署方式时进行再介绍镜像构建步骤。 ### 1.13.1 **Standalone on Kubernetes部署** Spark Standalone 集群中有Master和Worker两种角色，基于Kubernetes进行部署，即将两种对象以pod的方式部署到Kubernetes集群中，Master和Worker所需要的资源由Kubernetes集群提供。 #### 1.13.1.1 **构建镜像** Spark官方没有提供Spark的容器镜像，这里需要自己构建，构建Spark镜像的步骤如下： 1. **下载**Spark**安装包** 在自己使用Dockerfile构建Spark容器镜像时，我们需要添加Spark的安装包，也可以在Dockfile中直接下载，由于速度慢，这里选择手动下载安装包后，通过ADD 命令加入到容器中,通过Dockerfile构建镜像时还需要下载Hadoop安装包并添加到镜像中。在官网下载Spark3.3.1安装包和hadoop3.1.4，下载地址分别为：https://www.apache.org/dyn/closer.lua/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz/ [https://archive.apache.org/dist/hadoop/common/hadoop-3.1.4/](https://archive.apache.org/dist/hadoop/common/hadoop-3.1.4/) ``` #node1创建/root/spark目录，将下载的Spark安装包和hadoop安装包上传到此路径 [root@node1 ~]# mkdir -p /root/spark && cd spark [root@node1 spark]# ls hadoop-3.1.4.tar.gz spark-3.3.1-bin-hadoop3.tgz ``` 2. **编写**Docker File 在/root/spark目录中创建Dockerfile，内容如下： ``` FROM openjdk:8u151 ENV hadoop_version 3.1.4 ENV spark_version 3.3.1 ADD hadoop-3.1.4.tar.gz /opt ADD spark-3.3.1-bin-hadoop3.tgz /opt RUN mv /opt/hadoop-3.1.4 /opt/hadoop && mv /opt/spark-3.3.1-bin-hadoop3 /opt/spark && \ echo HADOOP ${hadoop_version} installed in /opt/hadoop && \ echo Spark ${spark_version} installed in /opt/spark ENV SPARK_HOME=/opt/spark ENV PATH=$PATH:$SPARK_HOME/bin ENV HADOOP_HOME=/opt/hadoop ENV PATH=$PATH:$HADOOP_HOME/bin ENV LD_LIBRARY_PATH=$HADOOP_HOME/lib/native ADD start-common.sh start-worker start-master / ADD spark-defaults.conf /opt/spark/conf/spark-defaults.conf ENV PATH $PATH:/opt/spark/bin ENV SPARK_WORKER_MEMORY=1024m ENV SPARK_WORKER_CORES=2 ``` 3. **上传**Dockerfile构建需要的文件 start-common.sh，内容如下： ``` #!/bin/sh unset SPARK_MASTER_PORT export SPARK_DIST_CLASSPATH=$(hadoop classpath) export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/hadoop/lib/native ``` start-master，内容如下： ``` #!/bin/sh . /start-common.sh echo "$(hostname -i) spark-master" >> /etc/hosts /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master --ip spark-master --port 7077 --webui-port 8080 ``` start-worker，内容如下： ``` #!/bin/sh . /start-common.sh if ! getent hosts spark-master; then echo "=== Cannot resolve the DNS entry for spark-master. Has the service been created yet, and is SkyDNS functional?" echo "=== See http://kubernetes.io/v1.1/docs/admin/dns.html for more details on DNS integration." echo "=== Sleeping 10s before pod exit." sleep 10 exit 0 fi /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077 --webui-port 8081 ``` spark-defaults.conf，内容如下： ``` spark.master spark://spark-master:7077 spark.driver.extraLibraryPath /opt/hadoop/lib/native #spark.driver.extraClassPath /opt/spark/jars/hadoop-aws-2.8.2.jar:/opt/spark/jars/aws-java-sdk-1.11.712.jar #spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem #spark.fs.s3a.connection.ssl.enabled false #spark.executor.extraJavaOptions -Dcom.amazonaws.sdk.disableCertChecking=1 spark.app.id KubernetesSpark spark.executor.memory 512m spark.executor.cores 1 ``` 以上start-common.sh、start-master、start-worker三个文件上传到/root/spark目录后，需要通过"chmod + x ./对应文件"来增加可执行权限。 ``` [root@node1 spark]# chmod +x ./start-common.sh ./start-master ./start-worker ``` 4. **构建**Spark**容器镜像** 执行如下命令，构建Spark容器镜像: ``` #构建spark 容器镜像 [root@node1 spark]# docker build -t myspark:v1 . ... Successfully built 46cfc6d5f975 Successfully tagged myspark:v1 ... #查看spark 容器镜像 [root@node1 spark]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE myspark v1 46cfc6d5f975 5 minutes ago 1.21GB ``` 注意:以上构建过程中下载openjdk比较慢，可以将资料中打包好的"openjdk.tar"加载到docker中： ``` [root@node1 ~]# docker load -i openjdk.tar [root@node1 ~]# docker images openjdk 8u151 a30a1e547e6d ... ... ``` 5. **将镜像上传到** harbor 后续基于Kubernetes部署Spark standalone集群时需要使用该spark容器镜像，需要从harbor私有镜像仓库进行拉取，这里我们将myspark:v1镜像上传至Harbor私有镜像仓库，步骤如下： ``` #给当前镜像打标签 [root@node1 spark]# docker tag myspark:v1 192.168.179.7/library/spark:v1 #上传到harbor [root@node1 spark]# docker push 192.168.179.7/library/spark:v1 ``` 登录harbor观察对应的镜像是否上传成功，通过webui观察，上传已经成功。 ![image.png](../../img/kubernetes/kubernetes_spark/ad532b75beec49a283ff6489bc24e83c.png) 资料中"myspark\_v1.tar"为打包好自定义spark镜像，可以通过以下命令加载到docker中： ``` [root@node1 ~]# docker load -i myspark_v1.tar [root@node1 ~]# docker images ... REPOSITORY TAG IMAGE ID 192.168.179.7/library/spark v1 f61c90cf4473 ... ``` #### 1.13.1.2 **yaml资源清单文件** 创建/root/spark-standalone-deployment目录: ``` [root@node1 ~]# mkdir -p /root/spark-standalone-deployment [root@node1 ~]# cd /root/spark-standalone-deployment/ ``` 在该目录中创建如下yaml资源清单文件： spark-master-controller.yaml: ``` kind: ReplicationController apiVersion: v1 metadata: name: spark-master-controller spec: replicas: 1 selector: component: spark-master template: metadata: labels: component: spark-master spec: hostname: spark-master-hostname subdomain: spark-master-nodeport containers: - name: spark-master image: 192.168.179.7/library/spark:v1 imagePullPolicy: Always command: ["/start-master"] ports: - containerPort: 7077 - containerPort: 8080 resources: requests: cpu: 100m ``` spark-master-service.yaml: ``` kind: Service apiVersion: v1 metadata: name: spark-master-nodeport spec: ports: - name: rest port: 8080 targetPort: 8080 nodePort: 30080 - name: submit port: 7077 targetPort: 7077 nodePort: 30077 type: NodePort selector: component: spark-master --- kind: Service apiVersion: v1 metadata: name: spark-master spec: ports: - port: 7077 name: spark - port: 8080 name: http selector: component: spark-master ``` spark-worker-controller.yaml: ``` kind: ReplicationController apiVersion: v1 metadata: name: spark-worker-controller spec: replicas: 2 selector: component: spark-worker template: metadata: labels: component: spark-worker spec: containers: - name: spark-worker image: 192.168.179.7/library/spark:v1 imagePullPolicy: Always command: ["/start-worker"] ports: - containerPort: 8081 resources: requests: cpu: 100m ``` #### 1.13.1.3 **部署yaml资源清单文件** 通过以下命令进行部署以上yaml资源清单文件： ``` #部署yaml资源清单文件 [root@node1 spark-standalone-deployment]# kubectl create -f . #查看pod启动情况 [root@node1 spark-standalone-deployment]# watch kubectl get all Every 2.0s: kubectl get all Thu Feb 16 15:09:54 2023 NAME READY STATUS RESTARTS AGE pod/spark-master-controller-gsvdt 1/1 Running 0 13s pod/spark-worker-controller-hhl2c 1/1 Running 0 13s pod/spark-worker-controller-lrprt 1/1 Running 0 13s ``` 部署完成之后，我们可以登录webui查看对应的Spark Standalone集群启动情况，node1:8080: ![image.png](../../img/kubernetes/kubernetes_spark/d73ae86fdde049f3ab8cfcc81bfe10d7.png) #### 1.13.1.4 **任务提交及日志查看** 这种Standalone集群部署方式想要提交Spark任务只能进入到pod中进行任务提交。所以这种方式在真实的生产环境中我们也不会使用，官网中也没有给出这种方式的部署方式，这里只用作测试。 1. **Spark Shell** **执行任务：** ``` #进入容器，执行spark shell [root@node1 ~]# kubectl exec -it spark-master-controller-gsvdt -- bash root@spark-master-hostname:/# /opt/spark/bin/spark-shell ... #编程scala代码进行测试 scala> val rdd = sc.makeRDD(List("hello k8s","hello k8s","hello spark")) rdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at makeRDD at :23 scala> rdd.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect() res0: Array[(String, Int)] = Array((spark,1), (k8s,2), (hello,3)) ... ``` 2. **client**模式提交 ****SparkPi**** 任务： ``` #登录master pod [root@node1 spark-standalone-deployment]#kubectl exec -it spark-master-controller-gsvdt -- bash #client模式提交任务 root@spark-master-hostname:cd /opt/spark/bin # ./spark-submit --master spark://spark-master:7077 --deploy-mode client --class org.apache.spark.examples.SparkPi ../examples/jars/spark-examples_2.12-3.3.1.jar ... Pi is roughly 3.13439567197836 ... ``` 3. **cluster**模式提交 ****SparkPi**** 任务： cluster模式运行，目前只能在webui中看到任务执行完成，在worker节点上运行的Driver(Exector)的日志没有挂载出来，所以 **对应最终的** pi **结果无法查看** ，这里只关注集群模式提交任务命令即可。 ``` #登录master pod [root@node1 ~]# kubectl exec -it spark-master-controller-gsvdt -- bash root@spark-master-hostname:/# cd /opt/spark/bin/ root@spark-master-hostname:/opt/spark/bin# ./spark-submit --master spark://spark-master:7077 --deploy-mode cluster --driver-memory 512m --class org.apache.spark.examples.SparkPi ../examples/jars/spark-examples_2.12-3.3.1.jar ``` 以上命令执行完成之后结果如下： ![image.png](../../img/kubernetes/kubernetes_spark/eee6425d23a5459ab4631528746f63e3.png) 注意：执行过程中如果delete对应的yaml资源清单后，pod状态一直为Terminating，可以通过强制删除pod命令删除,命令如下： ``` # --grace-period=0 直接删除 pod kubectl delete pod xxx -n xxx --force --grace-period=0 ``` ### 1.13.2 **Spark Native Kubernetes部署** Spark Native Kubernetes 部署类似于Spark 基于Yarn部署，Kubernetes看成是资源调度中心负责资源调度，运行executor。这种方式部署提交Spark任务可以是client模式提交任务也可以cluster模式提交任务，Spark Native Kuberntes提交任务原理图如下： ![image.png](../../img/kubernetes/kubernetes_spark/7d3a575ba7d747a194261aa257d7412c.png) ![image.png](../../img/kubernetes/kubernetes_spark/80392b13372c457b861c8bdb39168890.png) ![image.png](../../img/kubernetes/kubernetes_spark/426325be79c845c4858701e337d8454f.png) Client提交任务模式中，Driver运行在提交Spark任务的服务器上，也就是客户端，提交任务申请启动的executor以pod方式运行在Kubernetes集群中，当应用程序执行完成后，executor pod被终止被清理，Spark应用程序的日志会在客户端显示。 Cluster提交任务模式中，Driver会以pod的方式运行在Kubernetes集群中，Driver向Kubernetes申请启动Executor，Executor以pod方式运行在Kubernetes集群中，应用程序的日志不能在客户端显示，需要在Kubernetes集群对应的pod中查看日志，当应用程序完成时，执行程序pod将终止并被清理，但Driver pod会保存日志并在Kubernetes API中保持“已完成”状态。 #### 1.13.2.1 **构建并上传镜像** Spark Native Kubernetes部署模式中官方也没有提供对应的镜像，但是从Spark2.3版本后官方提供了Dockerfile，默认路径在”SPARK_HOME/kubernetes/dockerfiles/spark”路径下，要基于此Dockerfile构建Spark镜像，用户无需关心该Dockerfile文件位置，Spark提供了自动构建Spark镜像脚本，脚本为“SPARK_HOME/bin/docker-image-tool.sh”,只需要运行该脚本会自动构架Spark镜像。 ``` #运行docker-image-tool.sh构建镜像 [root@node1 ~]# cd /software/spark-3.3.1-bin-hadoop3/bin/ [root@node1 bin]# ./docker-image-tool.sh -r 192.168.179.7/library -t v2 build ... Successfully built c96b8604936f Successfully tagged 192.168.179.7/library/spark:v2 #检查构建的镜像 [root@node1 bin]# docker images REPOSITORY TAG IMAGE ID 192.168.179.7/library/spark v2 c96b8604936f ``` 注意以上构建命令参数解释如下: - -r :指定镜像仓库名称 - -t :指定构建镜像的tag 镜像构建完成后通过以下命令将镜像上传到私有镜像仓库： ``` [root@node1 bin]# docker push 192.168.179.7/library/spark:v2 ``` 检查上传镜像： ![image.png](../../img/kubernetes/kubernetes_spark/bf834dfc56d34c55aede4f37ffb29801.png) #### 1.13.2.2 **配置用户权限** 后续基于Kubernetes提交Spark任务时需要指定用户，Kubernetes是基于角色进行授权，所以这里创建对应的serviceaccount，然后给serviceaccount进行角色赋权。 ``` #创建命名空间 [root@node1 ~]# kubectl create ns spark namespace/spark created #创建serviceaccount [root@node1 ~]# kubectl create serviceaccount spark -n spark serviceaccount/spark created #给serviceaccount进行角色赋权 [root@node1 ~]# kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=spark:spark clusterrolebinding.rbac.authorization.k8s.io/spark-role created ``` #### 1.13.2.3 **client模式任务提交** client模式提交任务需要在节点上有Spark安装包，使用spark-submit命令进行任务提交，client模式进行任务提交在生产环境中使用较少，作为了解即可。以运行SparkPi任务为例，client模式提交任务命令如下： ``` [root@node1 ~]# cd /software/spark-3.3.1-bin-hadoop3/bin/ ./spark-submit \ --master k8s://https://192.168.179.4:6443 \ --deploy-mode client \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.kubernetes.namespace=spark \ --conf spark.executor.instances=2 \ --conf spark.kubernetes.container.image=192.168.179.7/library/spark:v2 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.driver.host=192.168.179.4 \ /software/spark-3.3.1-bin-hadoop3/examples/jars/spark-examples_2.12-3.3.1.jar ``` 以上提交任务命令参数解释如下： - --master：指定Kubernetes集群，格式为k8s://https://\:\，可以在k8s集群中执行"kubectl cluster-info"来获取k8s-apiserver-host和k8s-apiserver-port。 - --deploy-ment:指定部署模式 - --name:指定应用程序名称 - --class:指定运行主类 - --conf:指定配置 - spark.kubernetes.namespace：指定使用的Kubernetes的命名空间。 - spark.executor.instances：运行该Application时申请启动的executor的实例个数。 - spark.kubernetes.container.image：指定启动Container使用的镜像。 - spark.kubernetes.authenticate.driver.serviceAccountName：指定操作pod使用的用户。 - spark.driver.host：指定提交任务的driver地址，这里在哪个节点提交spark任务就写哪个节点的ip即可。 以上命令使用的jar包为节点上用户提供的jar包，提交之后，可以通过以下命令查看kubernetes中启动的executor情况： ``` [root@node1 ~]# watch kubectl get all -n spark Every 2.0s: kubectl get all -n spark Thu Feb 16 19:09:49 2023 NAME READY STATUS RESTARTS AGE pod/spark-pi-9a5ebe8659e9d1f0-exec-1 1/1 Running 0 46s pod/spark-pi-9a5ebe8659e9d1f0-exec-2 1/1 Running 0 45s ``` 任务执行完成之后，可以在提交任务的节点(客户端)看到对应PI结果： ``` ... Pi is roughly 3.142755713778569 ... ``` #### 1.13.2.4 **cluster模式任务提交** 1. **使用** pod ****内的**** jar **包** Cluster模式提交任务中，Driver与Exector都是以Pod方式运行在Kubernetes集群中,以运行SparkPi为例，Cluster模式提交任务命令如下： ``` [root@node1 ~]# cd /software/spark-3.3.1-bin-hadoop3/bin/ ./spark-submit \ --master k8s://https://192.168.179.4:6443 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.kubernetes.namespace=spark \ --conf spark.executor.instances=5 \ --conf spark.kubernetes.container.image=192.168.179.7/library/spark:v2 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ local:///opt/spark/examples/jars/spark-examples_2.12-3.3.1.jar ``` 以上执行SparkPi任务jar包以local://xxx 方式指定，local方式是pod镜像内的路径，这里指Driver对应pod内的jar包路径，在对应启动的executor pod中由于使用镜像都一样，所以该jar同样都存在于各个exeutor pod中。 命令执行之后，可以观察，kubernetes集群spark命名空间下的pod： ``` [root@node1 ~]# watch kubectl get all -n spark Every 2.0s: kubectl get all -n spark Thu Feb 16 19:31:43 2023 NAME READY STATUS RESTARTS AGE pod/spark-pi-467c6b8659fe633c-exec-1 1/1 Running 0 12s pod/spark-pi-467c6b8659fe633c-exec-2 1/1 Running 0 12s pod/spark-pi-467c6b8659fe633c-exec-3 1/1 Running 0 12s pod/spark-pi-467c6b8659fe633c-exec-4 0/1 Pending 0 12s pod/spark-pi-467c6b8659fe633c-exec-5 0/1 Pending 0 12s pod/spark-pi-d3e33d8659fe3a9b-driver 1/1 Running 0 22s #等待任务执行完成后，可以最终看到 NAME READY STATUS RESTARTS AGE pod/spark-pi-d3e33d8659fe3a9b-driver 0/1 Completed 0 2m40s ``` Spark Application执行的结果在客户端看不到，需要通过Kubernetes集群pod日志来查看，WebUI访问Kubernetes dashboard：[https://192.168.179.4:3000](http://192.168.179.4:3000/)0： ![image.png](../../img/kubernetes/kubernetes_spark/29ec6a81c8cf42e58a02c42e75751aca.png) 需要提供token，在Kubernetes集群中任意一台节点执行如下命令查询临时token: ``` #注意最后的kubernetes-dashboard 为部署dashboard创建的serviceaccounts [root@node1 kube-dashboard]# kubectl -n kubernetes-dashboard create token kubernetes-dashboard eyJhbGciOiJSUzI1NiIsImtpZCI6IktKSkJla1plMTJ6VHpNTmgxVG42OC1jaktuR1dOSzNqeHpWajRBQUZLZ1kifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjc2NTUxMjUzLCJpYXQiOjE2NzY1NDc2NTMsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInVpZCI6ImM2ZDg0NjJiLWI0YTItNDQyZC04ZDg2LTAyODc5ZDhkZmZiNCJ9fSwibmJmIjoxNjc2NTQ3NjUzLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQifQ.XRLNTx39s1hRcy8hbYFEwFp7JKoDBqd7XzKGieB5SPFeXXDiK6i4DHorg51H0CrxIejGeRJclwiCMk4UPtWM2Yxd07gchCouxsfxMxs4HEK5r1X-eeBv7BlBboZjAjldlhCV6v5cE2yxJkm0BSWOU91zMruRy0q4bzCFxP9mYPPovcaVAWah7EtBytqwuo0aGBM_XeJSmoXKrpApSTaeD5sBzfdsNMu4nhx4Tg8piBlaOALUKgko96ZusyrNokpEwMFyKX5q33vYNFs28tNQR8Nf45xLDTnCdGnOVmBWCSMYD7fxRe56fsPtHkZs7SHOfUWOvRu0QZM2KxntHceThw ``` 登录webui后查看对应pod日志： ![image.png](../../img/kubernetes/kubernetes_spark/f416db97747047b1b2a715efeeff599a.png) ![image.png](../../img/kubernetes/kubernetes_spark/a37255eda96142ae8c0a7e4220733665.png) ![image.png](../../img/kubernetes/kubernetes_spark/e9a10f64114249c3b2f3561678dfb4fb.png) 2. **使用**pod ****外用户提供的**** jar**包** 如果想要使用用户自己的jar包运行任务，需要保证Driver和Executor对应的pod能访问到该jar包，一般可以基于Kubernetes构建通用的HDFS集群，将jar包提交到基于K8s的集群中，那么运行的Driver和Executor pod自然就能访问到该jar。也可以通过一些命令将k8s集群外的jar包通过挂载方式挂载到Driver和Executor对应的pod中。 目前k8s集群中没有HDFS集群，这里我们使用挂载的方式将用户jar包挂载到各个pod中，由于每个pod不一定启动在k8s集群哪个节点，所以这里挂载的文件类型我们选择nfs，这样无论pod启动在k8s集群哪个节点都能访问到统一的nfs目录。 前面我们已经搭建过nfs集群，nfs-server节点是node4节点，nfs公共目录为：/localfolder，现在我们将位于k8s集群外的"spark-examples\_2.12-3.3.1.jar"看做用户提供的jar，上传到node4节点的nfs公共目录/localfolder中，这样在基于Kubernetes以Cluster模式提交Spark任务时，可以通过参数方式指定挂载到Driver和Executor的路径为该nfs公共目录路径，这样各个pod就能访问到用户提供的jar包。 首先将"spark-examples\_2.12-3.3.1.jar"放在node4节点/localfolder目录下： ``` [root@node1 ~]# scp /software/spark-3.3.1-bin-hadoop3/examples/jars/spark-examples_2.12-3.3.1.jar node4:/localfolder ``` 以运行SparkPi任务为例，Cluster模式提交任务命令如下： ``` [root@node1 ~]# cd /software/spark-3.3.1-bin-hadoop3/bin/ ./spark-submit \ --master k8s://https://192.168.179.4:6443 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.kubernetes.namespace=spark \ --conf spark.executor.instances=5 \ --conf spark.kubernetes.container.image=192.168.179.7/library/spark:v2 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.driver.volumes.nfs.up-jars.options.server=192.168.179.7 \ --conf spark.kubernetes.driver.volumes.nfs.up-jars.options.path=/localfolder \ --conf spark.kubernetes.driver.volumes.nfs.up-jars.mount.path=/localfolder \ --conf spark.kubernetes.executor.volumes.nfs.up-jars.options.server=192.168.179.7 \ --conf spark.kubernetes.executor.volumes.nfs.up-jars.options.path=/localfolder \ --conf spark.kubernetes.executor.volumes.nfs.up-jars.mount.path=/localfolder \ local:///localfolder/spark-examples_2.12-3.3.1.jar ``` 以上任务提交--conf执行的参数解释如下： - spark.kubernetes.namespace：指定使用的命名空间 - spark.executor.instances：指定启动executor pod实例个数 - spark.kubernetes.container.image:指定container使用的镜像地址 - spark.kubernetes.authenticate.driver.serviceAccountName:指定操作pod的用户 - spark.kubernetes.driver.volumes.nfs.up-jars.options.server:指定driver pod 挂载使用的nfs服务器地址 - spark.kubernetes.driver.volumes.nfs.up-jars.options.path:指定driver pod 挂载使用的nfs服务器共享目录 - spark.kubernetes.driver.volumes.nfs.up-jars.mount.path：指定nfs 目录挂载到driver pod 的路径 - spark.kubernetes.executor.volumes.nfs.up-jars.options.server:指定executor pod 挂载使用的nfs服务器地址 - spark.kubernetes.executor.volumes.nfs.up-jars.options.path:指定executor pod 挂载使用的nfs服务器共享目录 - spark.kubernetes.executor.volumes.nfs.up-jars.mount.path:指定nfs 目录挂载到executor pod 的路径 更多参数可以参考官网：https://spark.apache.org/docs/latest/running-on-kubernetes.html#configuration 任务提交之后可以观察kubernetes集群spark命名空间中pod启动情况: ``` [root@node1 ~]# watch kubectl get all -n spark Every 2.0s: kubectl get all -n spark Thu Feb 16 20:19:58 2023 NAME READY STATUS RESTARTS AGE pod/spark-pi-e29c77865a2ab033-exec-1 1/1 Running 0 3s pod/spark-pi-e29c77865a2ab033-exec-2 1/1 Running 0 3s pod/spark-pi-e29c77865a2ab033-exec-3 0/1 Pending 0 3s pod/spark-pi-e29c77865a2ab033-exec-4 0/1 Pending 0 3s pod/spark-pi-e29c77865a2ab033-exec-5 0/1 Pending 0 3s pod/spark-pi-eee459865a2a8a5e-driver 1/1 Running 0 13s #等待一段时间后，可以看到driver运行完成 pod/spark-pi-eee459865a2a8a5e-driver 0/1 Completed 0 52s ``` 查看对应运行结果，也需要登录Kubernetes webui中找到对应的pod查看日志。登录Kubernetes Webui https://node1:30000查看日志: ![image.png](../../img/kubernetes/kubernetes_spark/ce20c6f171eb4ef58c6f073d6c331b1d.png) 执行如下命令获取token: ``` #注意最后的kubernetes-dashboard 为部署dashboard创建的serviceaccounts [root@node1 kube-dashboard]# kubectl -n kubernetes-dashboard create token kubernetes-dashboard eyJhbGciOiJSUzI1NiIsImtpZCI6IktKSkJla1plMTJ6VHpNTmgxVG42OC1jaktuR1dOSzNqeHpWajRBQUZLZ1kifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjc2NTUxMjUzLCJpYXQiOjE2NzY1NDc2NTMsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInVpZCI6ImM2ZDg0NjJiLWI0YTItNDQyZC04ZDg2LTAyODc5ZDhkZmZiNCJ9fSwibmJmIjoxNjc2NTQ3NjUzLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQifQ.XRLNTx39s1hRcy8hbYFEwFp7JKoDBqd7XzKGieB5SPFeXXDiK6i4DHorg51H0CrxIejGeRJclwiCMk4UPtWM2Yxd07gchCouxsfxMxs4HEK5r1X-eeBv7BlBboZjAjldlhCV6v5cE2yxJkm0BSWOU91zMruRy0q4bzCFxP9mYPPovcaVAWah7EtBytqwuo0aGBM_XeJSmoXKrpApSTaeD5sBzfdsNMu4nhx4Tg8piBlaOALUKgko96ZusyrNokpEwMFyKX5q33vYNFs28tNQR8Nf45xLDTnCdGnOVmBWCSMYD7fxRe56fsPtHkZs7SHOfUWOvRu0QZM2KxntHceThw ``` 登录之后，找打对应spark命名空间，查看对应pod日志如下： ![image.png](../../img/kubernetes/kubernetes_spark/0bf82f31aa7643b4941996f47b304b9b.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_storage_ceph.md ================================================ # 存储解决方案 Ceph # 1 快速入门 ## 1.1 基础知识 ### 1.1.1 存储基础 学习目标 这一节，我们从 基础知识、文件系统、小结 三个方面来学习。 **基础知识** 存储基础 ```powershell 我们知道，对于一个计算机设备来说，其存储功能是非常重要的，也就是说，任何一台电脑上，必须包含一个设备--磁盘。这个磁盘就是用于数据存储的目的的。 ``` ```powershell 常见的存储设备接口： DAS设备：IDE、SATA、SCSI、SAS、USB 无论是那种接口，他都是存储设备驱动下的磁盘设备，而磁盘设备其实就是一种存储，这种存储是直接接入到主板总线上去的。 - 基于数据块来进行访问 - 基于服务器方式实现互联访问，操作简单、成本低。 NAS设备：NFS、CIFS 几乎所有的网络存储设备基本上都是以文件系统样式进行使用，无法进一步格式化操作。 - 基于文件系统方式访问 - 没有网络区域限制，支持多种协议操作文件。 SAN:scsi协议、FC SAN、iSCSI 基于SAN方式提供给客户端操作系统的是一种块设备接口，所以这些设备间主要是通过scsi协议来完成正常的通信。 scsi的结构类似于TCP/IP协议，也有很多层，但是scsi协议主要是用来进行存储数据操作的。既然是分层方式实现的，那就是说，有部分层可以被替代。比如将物理层基于FC方式来实现，就形成了FCSAN，如果基于以太网方式来传递数据，就形成了iSCSI模式。 - 基于数据块来实现访问 - 不受服务器约束，通过存储池实现资源的高效利用，扩展性好 ``` ![image-20220923150326956](../../img/kubernetes/kubernetes_storage_ceph/image-20220923150326956.png) 存储使用 ```powershell 对于存储的使用，我们需要借助于文件系统的方式来实现，而存储在使用前往往需要进行格式化。 ``` **文件系统** 简介 ```powershell 文件系统的基本数据单位是文件，它的目的是对磁盘上的文件进行组织管理，那组织的方式不同，就会形成不同的文件系统。 Linux 文件系统会为每个文件分配两个数据结构：索引节点(index node)和目录项(directory entry)，它们主要用来记录文件的元信息和目录层次结构。 索引节点 - 用来记录文件的元信息，比如 inode 编号、文件大小、访问权限、创建时间、等信息。 - 索引节点是文件的唯一标识，它们之间一一对应，也同样都会被存储在硬盘中，所以索引节点同样占用磁盘空间。 - 用户查找的时候，会根据inode的信息，找到对应的数据块，我们可以将inode理解为数据块的路由信息。 目录项 - 用来记录文件的名字、索引节点指针以及与其他目录项的层级关联关系。多个目录项关联起来，形成目录结构 - 它与索引节点不同，目录项是由内核维护的一个数据结构，不存放于磁盘，而是缓存在内存。 - 目录项和索引节点的关系是多对一 ``` 数据存储 ```powershell 数据块 磁盘读写的最小单位是扇区，扇区的大小只有 512B 大小，文件系统把多个扇区组成了一个逻辑块，每次读写的最小单位就是逻辑块（数据块），Linux 中的逻辑块大小为 4KB，也就是一次性读写 8 个扇区，这将大大提高了磁盘的读写的效率。 磁盘想要被文件系统使用，需要进行格式化，此时磁盘会被分成三个存储区域 - 超级块，用来存储文件系统的详细信息，比如块个数、块大小、空闲块等等。 - 索引节点区，用来存储索引节点； - 数据块区，用来存储文件或目录数据； ``` 存储应用的基本方式 ![1636079494896](../../img/kubernetes/kubernetes_storage_ceph/1636079494896.png) ```powershell 为了加速文件的访问，通常会把相关信息加载到内存中，但是考虑到内存的容量限制，它们加载进内存的时机是不同的： 超级块：当文件系统挂载时进入内存； 索引节点区：当文件被访问时进入内存； 数据块：文件数据使用的时候进入内； ``` 文件存储 ```powershell 文件的数据是要存储在硬盘上面的，数据在磁盘上的存放方式，有两种方式： 连续空间存放方式 - 同一个文件存放到一个连续的存储空间 - 一旦文件删除可能导致磁盘空间碎片 - 文件内容长度扩展不方便，综合效率是非常低的。 非连续空间存放方式 - 同一个文件存放到一个不连续的存储空间，每个空间会关联下一个空间 - 可以消除磁盘碎片，可大大提高磁盘空间的利用率，同时文件的长度可以动态扩展。 - 查找效率低，需要额外的资源消耗。 ``` ![image-20220923154844319](../../img/kubernetes/kubernetes_storage_ceph/image-20220923154844319.png) **小结** ``` ``` ### 1.1.2 DFS概述 学习目标 这一节，我们从 DFS简介、原理解读、小结 三个方面来学习。 **DFS简介** 存储基础 ```powershell 存储处理能力不足： 传统的IDE的io值是100次/秒，SATA固态磁盘500次/秒，NVMe固态硬盘达到2000-4000次/秒。即时磁盘的io能力再大数十倍，难道能够抗住网站访问高峰期数十万、数百万甚至上亿用户的同时访问么？这还受到网络io能力的限制。 存储空间能力不足： 单块磁盘的容量再大，也无法满足用户的正常访问所需的数据容量限制。 需求： 可以实现横向扩展的存储系统，这种存储系统在市面中的表现样式很多，不过他们有一个统一的称呼 -- 分布式存储系统。 ``` 分布式文件系统 ```powershell 随着传输技术发展，操作系统读写数据的方式，不再局限于本地I/O技术，开始支持远距离的TCP/IP方式获取数据。它相当于新增一种可以远距离传输的I/O技术，使得分散的存储设备和用户操作系统可以通过网络方式接入联合在一起，形成更大容量，更易于拓展伸缩的存储系统，对此人们引入分布式文件系统的概念。 ``` ```powershell 分布式文件系统（Distributed File System，DFS）是指文件系统管理的物理存储资源不一定直接连接在本地节点上，而是通过计算机网络与节点相连；或是若干不同的逻辑磁盘分区或卷标组合在一起而形成的完整的有层次的文件系统。 分布式文件系统的发展现后经历了三个阶段：网络文件系统、共享SAN文件系统、面向对象的并行文件系统。 ``` ```powershell 从本质上来说，分布式文件系统跟传统的文件系统没本质的差别，只不过是需要额外考虑多节点网络连接的可靠性、接入的存储设备的复杂性，需要文件系统、网络环境、存储策略共同协作而已。 由于文件系统与传统一致，网络环境不受控制，所以，我们平常所说的分布式文件系统，也等同于分布式存储系统。 ``` DSS简介 ```powershell 分布式存储系统，是将数据分散存储在多台独立的设备上，从而解决传统的存储系统的容量和性能限制。所以如果存储服务器的可靠性和安全性无法满足大规模存储应用的需要，那么它就会成为分布式文件系统的性能瓶颈。 其实，分布式存储系统可以理解为多台单机存储系统的各司其职、协同合作，统一的对外提供存储的服务。 ``` ```powershell 分布式网络存储系统采用可扩展的系统结构，利用多台存储服务器分担存储负荷，利用位置服务器定位存储信息，它不但提高了系统的可靠性、可用性和存取效率，还易于扩展。 按照分布式存储系统的作用场景，我们可以将其划分为 存储非结构化数据的分布式文件系统、存储结构化数据的分布式数据库、存储半结构化数据的分布式NoSQL数据库等。 ``` 常见的文件系统 ![image-20220923162020799](../../img/kubernetes/kubernetes_storage_ceph/image-20220923162020799.png) **原理解读** 分布式数据存储 ![1636082742521](../../img/kubernetes/kubernetes_storage_ceph/1636082742521.png) 存储角色 ```powershell 节点角色 当我们将数据存储到分布式系统上的时候，就需要有一个路由机制，能够将我们的请求转交给对应的存储节点上。所以，根据我们对数据在单节点上的存储原理，我们就需要有一个单独的节点来存储所有数据的元数据信息，然后，分布式存储就作为block的存储区域，专门用户数据存储。 存储元素据的节点我们把它称为NameNode，存储具体数据的节点我们称为DataNode。 数据拆分： 当我们要存储的数据非常大，比如说5个G，所以我们在存储的时候，将存储的数据信息发送给元数据控制节点，然后元数据控制节点根据自定义的存储策略，将要存储的数据进行拆分(64M一块)--也就是数据切片，将切分后的数据作为一个独立的文件，然后基于同样的路由逻辑，将其分散存储到不同的存储节点上。 元数据控制节点，在进行数据切分的时候，还需要能够组合起来，所以拆分后的数据块大小、组合时候的偏移信息、路由到的节点等信息都应该有针对性的记录。 在切片的时候，还可以实现并行的存储逻辑效果。每一个数据块都称为一个shard。 ``` 数据高可用 ```powershell 元数据高可用 由于NameNode保存了非常重要的数据信息，所以为了避免因为NameNode故障导致的问题，我们一定要对NameNode进行高可用处理。 由于元数据非常小(几k)，所以NameNode上的数据是 非常密集而且io量非常小的。所以为了提高数据的查询和存储效率，我们一般将这些数据保存到内存中。所以为了防止主机断电导致数据丢失，所以我们需要随时的进行数据同步到磁盘中，因为没有办法判断每一次到底是哪种数据被访问或更改，所以在磁盘数据同步的时候，随机io是非常大的。 同时，为了避免单节点的数据文件丢失，我们需要通过共享存储的方式将数据保存在第三方的存储设备上，同时还需要对数据存储主机进行高可用 ``` ```powershell 数据高可用 由于我们对元数据进行了数据切片的方式，实现了数据的高效率存取，但是我们知道，一旦这些数据块中，任意丢弃一块，就会导致所有的数据无法正常的使用。所以有必要对这些数据进行高可用的操作。对于高可用的方式，我们一般会有两种方式来实现数据的冗余。 节点级： 通过对主机节点进行高可用，从而实现数据的冗余，这种机制，成本太高了，不值得。 数据级： 我们对拆分后的数据块进行副本操作，而且还可以根据节点的数量，自定义冗余的副本数量。这是推荐的。 主角色的数据块称为 primary shard，副本数据块称为 replica shard。 ``` ```powershell 在进行数据块数据冗余的时候，这些副本的策略机制是有元数据节点来进行控制的，当一个DataNode故障的时候： - 如果主shard没有了，从所有的副本shard中选择一个主。 - 如果副本shard没有了，再创建一个副本shard即可。 - 一旦DataNode节点数量多于副本数量，控制副本数据在另一个DataNode节点上复制一个新的副本 从而保证副本的总体是满足预期的。 为了防止数据副本节点在同一个物理机架上的时候，因为机架故障，导致所有副本无效，所以我们在考虑冗余的时候，还需要考虑地理位置区域的冗余。 ``` **小结** ``` ``` ### 1.1.3 存储简介 学习目标 这一节，我们从 存储类型、ceph简介、小结 三个方面来学习。 **存储类型** 三种存储 ![1636084901961](../../img/kubernetes/kubernetes_storage_ceph/1636084901961.png) 块存储 ```powershell 块存储是将裸磁盘空间整个映射给主机使用的，主机层面操作系统识别出硬盘，与我们服务器内置的硬盘没有什么差异,可以自由的进行磁盘进行分区、格式化， 块存储不仅仅是直接使用物理设备，间接使用物理设备的也叫块设备，比如虚机创建虚拟磁盘、虚机磁盘格式包括raw、qcow2等。 常见解决方案： ABS(Azure Block Storage)、GBS(Google Block Storage)、EBS(Elastic Block Storag) Cinder、Ceph RBD、sheepdog ``` 文件系统存储 ```powershell 最常见、最容易、最经典实现的一种存储接口。它可以直接以文件系统挂载的方式，为主机提供存储空间资源。它有别于块存储的特点就在于，可以实现共享效果，但是有可能受网络因素限制导致速度较慢。 常见解决方案： AFS(Azure FileStorage)、GFS(Google FileStorage)、EFS(Elastic File Storage) Swift、CephFS、HDFS、NFS、CIFS、Samba、FTP ``` 对象存储 ```powershell 其核心是将 数据读或写 和 元数据 分离，并且基于对象存储设备(Object-based Storage Device，OSD)构建存储系统，然后借助于对象存储设备的管理功能，自动管理该设备上的数据分布。 对象存储主要包括四部分：对象、对象存储设备、元数据服务器、对象存储系统的客户端 简单地说，块存储读写块，不利于共享，文件存储读写慢，利于共享，而对象存储综合了两者的优点。 常见的解决方案： AS(Azure Storage)、GCS(Google Cloud Storage)、S3(Simple Storage Service) Swift、Ceph OSD ``` **Ceph简介** 官方介绍 ``` Ceph is the future of storage; where traditional(传统的) systems fail to deliver, Ceph is designed to excel(出色). Leverage(利用) your data for better business(业务) decisions(决策) and achieve(实现) operational(运营) excellence(卓越) through scalable, intelligent(智能), reliable(可靠) and highly available(可用) storage software. Ceph supports object, block and file storage, all in one unified(统一) storage system. ``` ```powershell 来源： Ceph项目最早起源于Sage就读博士期间的工作（最早的成果于2004年发表，论文发表于2006年），并随后贡献给开源社区。 ``` ```powershell 参考资料： 官方地址：https://ceph.com/en/ 官方文档：https://docs.ceph.com/en/latest/# github地址：https://github.com/ceph/ceph 最新版本：Quincy-17.2.3(2022-04-19)、Pacific-16.2.10(2021-03-31)、Octopus-15.2.17(2020-03-23) 系统支持： 全系列最好的系统支持是Ubuntu Ceph的发展对于Centos系列和Redhat系列不友好，新版本不支持旧版本系统。 ``` ![image-20220923170114091](../../img/kubernetes/kubernetes_storage_ceph/image-20220923170114091.png) 软件特点 ![image-20220923163720189](../../img/kubernetes/kubernetes_storage_ceph/image-20220923163720189.png) ```powershell 为什么很多人用ceph？ 1 创新，Ceph能够同时提供对象存储、块存储和文件系统存储三种存储服务的统一存储架构 2 算法，Ceph得以摒弃了传统的集中式存储元数据寻址方案，通过Crush算法的寻址操作，有相当强大的扩展性。 3 可靠，Ceph中的数据副本数量可以由管理员自行定义，并可以通过Crush算法指定副本的物理存储位置以分隔故障域，支持数据强一致性的特性也使Ceph具有了高可靠性，可以忍受多种故障场景并自动尝试并行修复。 ``` 基本结构 ![1636088249032](../../img/kubernetes/kubernetes_storage_ceph/1636088249032.png) ```powershell Ceph是一个多版本存储系统，它把每一个待管理的数据流（例如一个文件） 切分为一到多个固定大小的对象数据，并以其为原子单元完成数据存取。 对象数据的底层存储服务是由多个主机（host）组成的存储集群，该集群也被称之为 RADOS（Reliable Automatic Distributed Object Store）存储集群，即可靠、自动化、 分布式对象存储系统 librados是RADOS存储集群的API，它支持C、C++、Java、Python、Ruby和PHP等编程语言。 ``` 应用场景 ``` Ceph uniquely(独特的) delivers object, block, and file storage in one unified(统一) system. 注意：这个介绍在官网这几年基本没有变化 ``` ![1636099032730](../../img/kubernetes/kubernetes_storage_ceph/1636099032730.png) ``` RadosGW、RBD和CephFS都是RADOS存储服务的客户端，它们把RADOS的存储服务接口(librados)分别从不同的角度做了进一步抽象，因而各自适用于不同的应用场景。 也就是说，ceph将三种存储类型统一在一个平台中，从而实现了更强大的适用性。 ``` ![1636089258096](../../img/kubernetes/kubernetes_storage_ceph/1636089258096.png) ``` LIBRADOS - 通过自编程方式实现数据的存储能力 RADOSGW - 通过标准的RESTful接口，提供一种云存储服务 RBD - 将ceph提供的空间，模拟出来一个一个的独立块设备使用。 而且，ceph环境部署完毕后，服务端就准备好了rbd接口 CFS - 通过一个标准的文件系统接口来进行数据的存储 参考图例：https://docs.ceph.com/en/pacific/_images/stack.png ``` ![image-20211110140906074](../../img/kubernetes/kubernetes_storage_ceph/image-20211110140906074.png) **小结** ``` ``` ### 1.1.4 组件解析 学习目标 这一节，我们从 组件介绍、流程解读、小结 三个方面来学习。 **组件介绍** 组件简介 ``` 无论您是想向云平台提供 Ceph 对象存储和 Ceph 块设备服务、部署 Ceph 文件系统还是将 Ceph 用于其他目的，所有 Ceph 存储集群部署都从设置每个 Ceph 节点、您的网络和 Ceph 开始 存储集群。 一个 Ceph 存储集群至少需要一个 Ceph Monitor、Ceph Manager 和 Ceph OSD（对象存储守护进程）。 运行 Ceph 文件系统客户端时也需要 Ceph 元数据服务器。 ``` ![1636099270839](../../img/kubernetes/kubernetes_storage_ceph/1636099270839.png) | 组件 | 解析 | | --------- | ------------------------------------------------------------ | | Monitors | Ceph Monitor (守护进程ceph-mon) 维护集群状态的映射，包括监视器映射、管理器映射、OSD 映射、MDS 映射和 CRUSH 映射。 这些映射是 Ceph 守护进程相互协调所需的关键集群状态。 监视器还负责管理守护进程和客户端之间的身份验证。 通常至少需要三个监视器才能实现冗余和高可用性。基于 paxos 协议实现节点间的信息同步。 | | Managers | Ceph 管理器 (守护进程ceph-mgr) 负责跟踪运行时指标和 Ceph 集群的当前状态，包括存储利用率、当前性能指标和系统负载。 Ceph 管理器守护进程还托管基于 Python 的模块来管理和公开 Ceph 集群信息，包括基于 Web 的 Ceph 仪表板和 REST API。 高可用性通常至少需要两个管理器。基于 raft 协议实现节点间的信息同步。 | | Ceph OSDs | Ceph OSD（对象存储守护进程，ceph-osd）存储数据，处理数据复制、恢复、重新平衡，并通过检查其他 Ceph OSD 守护进程的心跳来向 Ceph 监视器和管理器提供一些监控信息。 通常至少需要 3 个 Ceph OSD 来实现冗余和高可用性。本质上osd就是一个个host主机上的存储磁盘。 | | MDSs | Ceph 元数据服务器（MDS[Metadata Server]、ceph-mds）代表 Ceph 文件系统存储元数据。 Ceph 元数据服务器允许 POSIX（为应用程序提供的接口标准） 文件系统用户执行基本命令（如 ls、find 等），而不会给 Ceph 存储集群带来巨大负担。 | | PG | PG全称Placement Grouops，是一个逻辑的概念，一个PG包含多个OSD。引入PG这一层其实是为了更好的分配数据和定位数据。写数据的时候，写入主osd，冗余两份。 | **流程解读** 综合效果图 ![1636104082951](../../img/kubernetes/kubernetes_storage_ceph/1636104082951.png) ```powershell Ceph 将数据作为对象存储在逻辑存储池中，主要包括两步： 把对象数据 映射给 PG - 计算出哪个归置组应该包含该对象 - 这部分功能是由Hash算法实现的 把 PG 映射到 host的OSD - 计算出哪个 Ceph OSD Daemon 应该存储该归置组 - 这部分由 CRUSH算法来决定的，CRUSH 算法使 Ceph 存储集群能够动态扩展、重新平衡和恢复。 ``` 数据存储逻辑 ![1636103641572](../../img/kubernetes/kubernetes_storage_ceph/1636103641572.png) **小结** ``` ``` ### 1.1.5 存储原理 学习目标 这一节，我们从 存储解读、案例解读、小结 三个方面来学习。 **存储解读** 存储数据 ```powershell Ceph 存储集群从 Ceph 客户端接收数据,无论是通过 Ceph 块设备、Ceph 对象存储、Ceph 文件系统还是您使用 librados 创建的自定义实现, 这些数据存储为 RADOS 对象, 每个对象都存储在一个对象存储设备上。 Ceph OSD 守护进程处理存储驱动器上的读、写和复制操作。它主要基于两种文件方式实现： - Filestore方式，每个 RADOS 对象都作为一个单独的文件存储在传统文件系统（通常是 XFS）上。 - BlueStore方式，对象以类似整体数据库的方式存储，这是新版本ceph默认的存储方式。 注意：在Ceph中，每一个文件都会被拆分为多个独立的object，然后按照上面的逻辑进行持久化。 ``` ![1636104261047](../../img/kubernetes/kubernetes_storage_ceph/1636104261047.png) ```powershell Ceph OSD 守护进程将"数据"作为对象存储在平面命名空间中,该对象包括如下部分： 标识符 - 在内存中唯一查找的标识 二进制数据 - 每个对象的真实数据 属性数据 - 由一组名称/值对组成的元数据，语义完全取决于 Ceph 客户端。 例如，CephFS 使用元数据来存储文件属性，例如文件所有者、创建日期、上次修改日期等。 注意： 对象 ID 在整个集群中是唯一的，而不仅仅是本地文件系统. ``` ![1636104309671](../../img/kubernetes/kubernetes_storage_ceph/1636104309671.png) **案例解读** 存储示例 ```powershell 存储一个大小为16M的文件，存储的时候，需要首先进行切割，假设每个对象(object)大小为4M,然后通过hash方式将object存储到对应的PG上，然后通过 CRUSH 策略将对应的数据关联到不同的host上。 这个时候遇到一个问题：osd是一个磁盘设备，那么如何来进行存储数据 常见的处理措施主要有两种： 第一种：将osd格式化为文件系统，然后挂载到对应目录，然后就可以使用了 第二种：osd有自己的守护进程，那么直接在守护进程空间中实现数据的处理 ``` 方法1 - FileStore ![1636105592444](../../img/kubernetes/kubernetes_storage_ceph/1636105592444.png) ```powershell 这个时候，osd就变成一个文件系统中的文件(目录)了。 因此，OSD需要维护object的对象属性信息，object的元数据保存到 osd(文件系统)的元数据区。 但是文件系统的元数据区只能存放文件的 属主、属组、权限、时间戳等信息。对于ceph来说，object的元数据信息却包含很多相关信息，那么这些数据保存到哪里？ ``` ```powershell 那么为了保存文件系统默认能够保存的数据之外的元数据(object对象的其他元数据信息)，我们就需要将OSD做成一个XFS文件系统，在这个文件系统中，需要有一个单独的数据库(LevelDB)，来维护ceph的元数据信息，效果如下 ``` ![1636106662704](../../img/kubernetes/kubernetes_storage_ceph/1636106662704.png) ```powershell 由于这种方式是基于文件系统映射的方式来实现 ceph的属性存储，所以我们把这种文件的存储方式称为 FiltStore。 劣势： 由于我们的object本来已经称为对象了，而在FileStore中，我们需要将其转换为文件方式来进行数据属性的存储，所以效率有些慢 附注： XFS是被开发用于高容量磁盘以及高性能文件系统之用的。主要规划为三个部分： 资料区（data section）- 存储包括inode、block、superblock等数据 文件系统活动登录区（log section） - 用来记录文件系统的变化 实时运作（realtime section）- 数据真正存储的地方 ``` 方法2 - BlueStore ```powershell 因为osd对象是由一个ceph-osd守护进程来实现存储数据、处理数据复制、恢复、重新平衡等管理操作的。所以新版的Ceph的osd进程中，维护了一个RocksDB用于存储objects的元数据属性信息. RocksDB为了实现数据的存储还需要一个文件系统，所以Ceph就为它开发了一个文件系统 BlueFS。 RocksDB + BlueFS 共同实现了objects的元数据的存储，所以我们把这种存储方式称为 BlueStore。新版的Ceph的存储机制，默认采用的就是 BlueStore机制。 ``` ![1636107225026](../../img/kubernetes/kubernetes_storage_ceph/1636107225026.png) **小结** ``` ``` ## 1.2 集群部署 ### 1.2.1 环境概述 学习目标 这一节，我们从 基础知识、环境规划、小结 三个方面来学习。 **基础知识** 注意事项 ```powershell 在Ceph系统的搭建过程中,会出现各种意想不到或者预想到问题，就算整个过程中每一步都没问题，还是会出现各种问题，这些问题不仅仅在网上找不到，甚至在官网中找不到，甚至玩ceph数年的人都解决不了。 尤其是，就算你第一次成功后，第二次重试就会出现问题。所以，如果出现问题怎么办？一步一步踏踏实实的进行研究，分析解决问题，并进行总结并梳理成册就可以了。 ``` 简介 ```powershell ceph的环境部署是非常繁琐的，所以，官方帮我们提供了很多的快捷部署方式。 参考资料： https://docs.ceph.com/en/pacific/install/ ``` ```powershell 推荐方法： Cephadm 使用容器和 systemd 安装和管理 Ceph 集群，并与 CLI 和仪表板 GUI 紧密集成。 仅支持 Octopus 和更新版本，需要容器和 Python3支持 与新的编排 API 完全集成 Rook 在 Kubernetes 中运行的 Ceph 集群，同时还支持通过 Kubernetes API 管理存储资源和配置。 仅支持 Nautilus 和较新版本的 Ceph 其他方法 ceph-ansible 使用Ansible部署Ceph集群，对于新的编排器功能、管理功能和仪表板支持不好。 ceph-deploy 是一个快速部署集群的工具，不支持Centos8 DeepSea 使用 Salt 安装 Ceph ceph-mon 使用 Juju 安装 Ceph Puppet-ceph 通过 Puppet 安装 Ceph 二进制源码 手工安装 windows图形 在windows主机上，通过鼠标点点点的方式进行部署。 ``` 版本的选择 ```powershell 版本地址：https://docs.ceph.com/en/latest/releases/ 最新版本：官网版本 v16.2.10 Pacific 版本特性：x.0.z(开发版)、x.1.z(候选版)、x.2.z(稳定、修正版) ``` **环境规划** 网络规划 ![image-20211110140641473](../../img/kubernetes/kubernetes_storage_ceph/image-20211110140641473.png) ```powershell 公有网络(public)： - 用于用户的数据通信 - 10.0.0.0/16 集群网络(cluster)： - 用于集群内部的管理通信 - 192.168.8.0/16 ``` 主机规划 ```powershell 磁盘规划 磁盘1 - VM的系统盘 磁盘2和磁盘3 - Ceph的OSD ``` ```powershell 主机名规划 主机名 公有网络 私有网络 磁盘 其他角色 admin 10.0.0.12 192.168.8.12 sdb、sdc stor01 10.0.0.13 192.168.8.13 sdb、sdc mon01 stor02 10.0.0.14 192.168.8.14 sdb、sdc mon02 stor03 10.0.0.15 192.168.8.15 sdb、sdc mon03 stor04 10.0.0.16 192.168.8.16 sdb、sdc stor05 10.0.0.17 192.168.8.17 sdb、sdc stor06 10.0.0.18 192.168.8.18 sdb、sdc ``` ```powershell 注意： 由于生产中，ceph的集群角色是非常多的，当我们的主机量少的时候，只能让一台主机节点运行多个角色。 stor01~03 这三台主机，还同时兼具 mon的角色，视情况兼容mgr角色 主机名的完整格式是： xxx.superopsmsb.com ``` 其他准备 ```powershell 管理用户 由于我们接下来的所有操作，基本上都在 admin这个主机上来运行，所以，我们不推荐直接使用root用户来管理，倾向于通过一个普通用户来操作接下来的操作。 由于后续的安装软件，涉及到root用户权限的操作，所以这个普通用户最好具备sudo的权限。 ``` ```powershell 时间同步 对于任何一个集群来说，时间同步是非常重要的。 ``` ```powershell 主机名规划 随着生产中的主机节点越来越多，我们通过手工定制主机名的方式就不太适合集群的主机管理了。所以在企业中，我们的主机名相关的信息，倾向于通过内网dns来进行管理。 尤其是等我们到 radosgw的时候，必须通过泛域名解析的机制来实现，更强大的面向客户端的主机名管理体系。 ``` VM主机准备 ![image-20211108105810433](../../img/kubernetes/kubernetes_storage_ceph/image-20211108105810433.png) ```powershell 所有节点都准备三块盘，两块网卡 虚拟网络设置： VMnet1 设定为 192.168.8.0 网段， VMnet8 设定为 10.0.0.0 网段 虚拟机设置 额外添加两块盘，每个根据自己的情况设定容量，我这里设定为20G 额外更加一块网络适配器，使用仅主机模式 -- VMnet1，mac地址必须要重新生成，避免冲突 ``` **小结** ```powershell ``` ### 1.2.2 准备工作 学习目标 这一节，我们从 基本环境、软件安装、小结 三个方面来学习。 **基本环境** 主机名管理 ```powershell 编辑 /etc/hosts 文件 10.0.0.12 admin.superopsmsb.com admin 10.0.0.13 stor01.superopsmsb.com stor01 mon01.superopsmsb.com mon01 10.0.0.14 stor02.superopsmsb.com stor02 mon02.superopsmsb.com mon02 10.0.0.15 stor03.superopsmsb.com stor03 mon03.superopsmsb.com mon03 10.0.0.16 stor04.superopsmsb.com stor04 10.0.0.17 stor05.superopsmsb.com stor05 10.0.0.18 stor06.superopsmsb.com stor06 注意： 后续可能会涉及到k8s环境的部署，所以hosts文件有可能会发生变动。 ``` 防火墙管理 ```powershell 关闭防火墙 systemctl stop iptables firewalld systemctl disable iptables firewalld systemctl status iptables firewalld ``` 跨主机通信 ```powershell 脚本文件名称 01_remote_host_auth.sh #!/bin/bash # 功能: 批量设定远程主机免密码认证 # 版本: v0.2 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 user_dir='/root' host_file='/etc/hosts' login_user='root' login_pass='123456' target_type=(部署 免密 同步 主机名 退出) # 菜单 menu(){ echo -e "\e[31m批量设定远程主机免密码认证管理界面\e[0m" echo "=====================================================" echo -e "\e[32m 1: 部署环境 2: 免密认证 3: 同步hosts \e[0m" echo -e "\e[32m 4: 设定主机名 5：退出操作 \e[0m" echo "=====================================================" } # expect环境 expect_install(){ if [ -f /usr/bin/expect ] then echo -e "\e[33mexpect环境已经部署完毕\e[0m" else yum install expect -y >> /dev/null 2>&1 && echo -e "\e[33mexpect软件安装完毕\e[0m" || (echo -e "\e[33mexpect软件安装失败\e[0m" && exit) fi } # 秘钥文件生成环境 create_authkey(){ # 保证历史文件清空 [ -d ${user_dir}/.ssh ] && rm -rf ${user_dir}/.ssh/* || mkdir -p ${user_dir}/.ssh # 构建秘钥文件对 /usr/bin/ssh-keygen -t rsa -P "" -f ${user_dir}/.ssh/id_rsa echo -e "\e[33m秘钥文件已经创建完毕\e[0m" } # expect自动匹配逻辑 expect_autoauth_func(){ # 接收外部参数 command="$@" expect -c " spawn ${command} expect { \"yes/no\" {send \"yes\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"} }" } # 跨主机传输文件认证 sshkey_auth_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do # /usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub root@10.0.0.12 cmd="/usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub" remote_host="${login_user}@${ip}" expect_autoauth_func ${cmd} ${remote_host} done } # 跨主机同步hosts文件 scp_hosts_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do remote_host="${login_user}@${ip}" scp ${host_file} ${remote_host}:${host_file} done } # 跨主机设定主机名规划 set_hostname_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do host_name=$(grep ${ip} ${host_file}|awk '{print $NF}') remote_host="${login_user}@${ip}" ssh ${remote_host} "hostnamectl set-hostname ${host_name}" done } # 帮助信息逻辑 Usage(){ echo "请输入有效的操作id" } # 逻辑入口 while true do menu read -p "请输入有效的操作id: " target_id if [ ${#target_type[@]} -ge ${target_id} ] then if [ ${target_type[${target_id}-1]} == "部署" ] then echo "开始部署环境操作..." expect_install create_authkey elif [ ${target_type[${target_id}-1]} == "免密" ] then read -p "请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行免密认证操作..." sshkey_auth_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "同步" ] then read -p "请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行同步hosts文件操作..." scp_hosts_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "主机名" ] then read -p "请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行设定主机名操作..." set_hostname_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "退出" ] then echo "开始退出管理界面..." exit fi else Usage fi done ``` ```powershell 执行脚本文件 [root@localhost ~]# /bin/bash /data/scripts/01_remote_host_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 1 开始部署环境操作... expect环境已经部署完毕 Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:DNQAfWAD3MYPywK4W0BrNsgK+u3ysliuI6QdF3BOuyc root@localhost The key's randomart image is: +---[RSA 2048]----+ |.o .o**+ | |= + +o*.o | |oO * +.= | |B o = oo. | |oo + S | |.o..E . | |o.oo.o | |+++. | |+oo=o | +----[SHA256]-----+ 秘钥文件已经创建完毕 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): {12..18} 开始执行免密认证操作... ... Now try logging into the machine, with: "ssh 'root@10.0.0.12'" and check to make sure that only the key(s) you wanted were added. ... Now try logging into the machine, with: "ssh 'root@10.0.0.18'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 3 请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): {12..18} 开始执行同步hosts文件操作... hosts 100% 520 1.2MB/s 00:00 ... hosts 100% 520 403.3KB/s 00:00 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 4 请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): {12..18} 开始执行设定主机名操作... 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 5 开始退出管理界面... ``` ```powershell 测试效果 [root@localhost ~]# for i in {12..18}; do hostname=$(ssh root@10.0.0.$i "hostname"); echo "10.0.0.$i - $hostname"; done 10.0.0.12 - admin 10.0.0.13 - mon01 10.0.0.14 - mon02 10.0.0.15 - mon03 10.0.0.16 - stor04 10.0.0.17 - stor05 10.0.0.18 - stor06 ``` **软件安装** 用户管理 ```powershell 创建普通用户 useradd -m cephadm -s /bin/bash echo cephadm:123456 | chpasswd 为用户配置root权限 echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm chmod 0440 /etc/sudoers.d/cephadm ``` ```powershell 切换用户时候不输出最近登录信息 [root@admin ~]# grep "se.*postlogin" /etc/pam.d/su # session include postlogin ``` ```powershell 脚本方法 02_create_ceph_user.sh #!/bin/bash # 功能: 创建专属的ceph管理用户 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 login_user='cephadm' login_pass='123456' # 设定普通用户 useradd -m ${login_user} -s /bin/bash echo ${login_user}:${login_pass} | chpasswd echo "${login_user} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/${login_user} chmod 0440 /etc/sudoers.d/${login_user} ``` ```powershell 批量执行 for i in {13..18} do ssh root@10.0.0.$i "mkdir /data/scripts -p" scp /data/scripts/02_create_ceph_user.sh root@10.0.0.$i:/data/scripts/02_create_ceph_user.sh ssh root@10.0.0.$i "/bin/bash /data/scripts/02_create_ceph_user.sh" done ``` ```powershell 确认效果 [root@localhost ~]# for i in {12..18}; do usermsg=$(ssh root@10.0.0.$i "id cephadm"); echo "10.0.0.$i - $usermsg"; done 10.0.0.12 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) 10.0.0.13 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) 10.0.0.14 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) 10.0.0.15 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) 10.0.0.16 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) 10.0.0.17 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) 10.0.0.18 - uid=1001(cephadm) gid=1001(cephadm) 组=1001(cephadm) ``` 跨主机免密码认证 ```powershell 脚本文件内容 /data/scripts/03_remote_cephadm_auth.sh #!/bin/bash # 功能: 批量设定远程主机免密码认证 # 版本: v0.3 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 user_dir='/home/cephadm' login_user='cephadm' login_pass='123456' host_file='/etc/hosts' target_type=(部署 免密 退出) # 菜单 menu(){ echo -e "\e[31m批量设定远程主机免密码认证管理界面\e[0m" echo "=====================================================" echo -e "\e[32m 1: 部署环境 2: 免密认证 3: 退出操作 \e[0m" echo "=====================================================" } # expect环境 expect_install(){ if [ -f /usr/bin/expect ] then echo -e "\e[33mexpect环境已经部署完毕\e[0m" else sudo yum install expect -y >> /dev/null 2>&1 && echo -e "\e[33mexpect软件安装完毕\e[0m" || (echo -e "\e[33mexpect软件安装失败\e[0m" && exit) fi } # 秘钥文件生成环境 create_authkey(){ # 保证历史文件清空 [ -d ${user_dir}/.ssh ] && rm -rf ${user_dir}/.ssh/* # 构建秘钥文件对 /usr/bin/ssh-keygen -t rsa -P "" -f ${user_dir}/.ssh/id_rsa echo -e "\e[33m秘钥文件已经创建完毕\e[0m" } # expect自动匹配逻辑 expect_autoauth_func(){ # 接收外部参数 command="$@" expect -c " spawn ${command} expect { \"yes/no\" {send \"yes\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"} }" } # 跨主机传输文件认证 sshkey_auth_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do # /usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub root@10.0.0.12 cmd="/usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub" remote_host="${login_user}@${ip}" host_name=$(grep ${ip} ${host_file}|awk '{print $NF}') remote_host1="${login_user}@${host_name}" remote_host2="${login_user}@${host_name}.superopsmsb.com" expect_autoauth_func ${cmd} ${remote_host} expect_autoauth_func ${cmd} ${remote_host1} expect_autoauth_func ${cmd} ${remote_host2} done } # 帮助信息逻辑 Usage(){ echo "请输入有效的操作id" } # 逻辑入口 while true do menu read -p "请输入有效的操作id: " target_id if [ ${#target_type[@]} -ge ${target_id} ] then if [ ${target_type[${target_id}-1]} == "部署" ] then echo "开始部署环境操作..." expect_install create_authkey elif [ ${target_type[${target_id}-1]} == "免密" ] then read -p "请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行免密认证操作..." sshkey_auth_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "退出" ] then echo "开始退出管理界面..." exit fi else Usage fi done ``` ```powershell 更改文件权限 chown cephadm:cephadm /data/scripts/03_remote_cephadm_auth.sh 切换用户 su - cephadm ``` ```powershell 执行脚本文件 [cephadm@admin ~]$ /bin/bash /data/scripts/03_remote_cephadm_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 退出操作 ===================================================== 请输入有效的操作id: 1 开始部署环境操作... expect环境已经部署完毕 Generating public/private rsa key pair. Your identification has been saved in /home/cephadm/.ssh/id_rsa. Your public key has been saved in /home/cephadm/.ssh/id_rsa.pub. The key fingerprint is: SHA256:EvrdtcF4q+e2iyVVOovFkoHOisorCem+1eSG7WO+NCQ cephadm@admin The key's randomart image is: +---[RSA 2048]----+ | | | . | | . . . . | | . + * o | | . E.o. S + @ | |o Oo + . B * | |o. o.Bo . + = | |oo..+o. =o | |.+=.o+o o++o | +----[SHA256]-----+ 秘钥文件已经创建完毕 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): {12..18} 开始执行免密认证操作... ... Now try logging into the machine, with: "ssh 'cephadm@10.0.0.18'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 退出操作 ===================================================== 请输入有效的操作id: 3 开始退出管理界面... ``` ```powershell 测试效果 [cephadm@admin ~]$ for i in {12..18}; do hostname=$(ssh cephadm@10.0.0.$i "hostname"); echo "10.0.0.$i - $hostname"; done 10.0.0.12 - admin 10.0.0.13 - mon01 10.0.0.14 - mon02 10.0.0.15 - mon03 10.0.0.16 - stor04 10.0.0.17 - stor05 10.0.0.18 - stor06 ``` ```powershell 因为10.0.0.13-15 有两个角色，所以我们需要将相关角色的免密认证 num_list={1..3} for i in $(eval echo stor0$num_list stor0$num_list.superopsmsb.com); do ssh-copy-id cephadm@$i; done ``` 定制软件源 ```powershell 对于ceph-deploy方式部署Ceph来说，Ceph的官方仓库路径是http://download.ceph.com/，包括各种ceph版本，比如octopus、pacific、quincy等，它根据不同OS系统环境，分别位于rpm-版本号 或者 debian-版本号 的noarch目录下。比如pacific版本的软件相关源在 rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm。 注意： el7代表支持Red Hat 7.x、CentOS 7.x 系统的软件 el8代表支持Red Hat 8.x、CentOS 8.x 系统的软件 pacific版本及其更新版本，只支持CentOS 8.x环境 ``` ```powershell Ceph的pacific和Quincy版本，仅仅支持CentOS8.X，Octopus版本虽然有CentOS7的版本，不仅仅软件不全，而且对于底层GCC库和GLIBC库要求比较高，如果升级CentOS7的底层库，会导致其他软件受到影响，无法正常使用，另外没有配套的ceph-deploy。所以对于CentOS7来说，只能部署Nautilus版本和更低版本。 对于Ubuntu系统来说，即使多个版本对于底层环境要求有些区别，但是经过测试，问题不大，也就是说Ubuntu系统可以安装Ceph的全系列 ``` ```powershell 安装软件源 yum install -y https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/ceph-release-1-1.el7.noarch.rpm 更新软件源 yum makecache fast ``` ```powershell 所有ceph节点部署软件源 for i in {12..18} do ssh root@10.0.0.$i yum install -y https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/ceph-release-1-1.el7.noarch.rpm ssh root@10.0.0.$i yum makecache fast done ``` 部署依赖 ```powershell admin主机安装ceph软件 yum update -y yum install ceph-deploy python-setuptools python2-subprocess32 -y 测试效果 su - cephadm -c "ceph-deploy --help" ``` 命令解析 ```powershell 查看命令帮助 [root@admin ~]# su - cephadm [cephadm@admin ~]$ ceph-deploy --help usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME] [--overwrite-conf] [--cluster NAME] [--ceph-conf CEPH_CONF] COMMAND ... Easy Ceph deployment -^- / \ |O o| ceph-deploy v1.5.25 ).-.( '/|||\` | '|` | '|` Full documentation can be found at: http://ceph.com/ceph-deploy/docs optional arguments: -h, --help show this help message and exit -v, --verbose be more verbose -q, --quiet be less verbose --version the current installed version of ceph-deploy --username USERNAME the username to connect to the remote host --overwrite-conf overwrite an existing conf file on remote host (if present) --cluster NAME name of the cluster --ceph-conf CEPH_CONF use (or reuse) a given ceph.conf file commands: COMMAND description # 创建一个集群 new Start deploying a new cluster, and write a CLUSTER.conf and keyring for it. install Install Ceph packages on remote hosts. rgw Deploy ceph RGW on remote hosts. mds Deploy ceph MDS on remote hosts. mon Deploy ceph monitor on remote hosts. gatherkeys Gather authentication keys for provisioning new nodes. disk Manage disks on a remote host. osd Prepare a data disk on remote host. # 同步admin秘钥信息 admin Push configuration and client.admin key to a remote host. # 同步ceph.conf文件 config Push configuration file to a remote host. uninstall Remove Ceph packages from remote hosts. purgedata Purge (delete, destroy, discard, shred) any Ceph data from /var/lib/ceph purge Remove Ceph packages from remote hosts and purge all data. forgetkeys Remove authentication keys from the local directory. pkg Manage packages on remote hosts. calamari Install and configure Calamari nodes. Assumes that a repository with Calamari packages is already configured. Refer to the docs for examples (http://ceph.com/ceph-deploy/docs/conf.html) ``` **小结** ``` ``` ### 1.2.3 Ceph部署 学习目标 这一节，我们从 集群创建、Mon环境、小结 三个方面来学习。 **集群创建** 准备工作 ```powershell 首先在管理节点上以cephadm用户创建集群相关的配置文件目录： su - cephadm mkdir ceph-cluster && cd ceph-cluster ``` 初始化集群解析 ```powershell 操作解析 ceph-deploy new --help 初始化第一个MON节点的命令格式为”ceph-deploy new {initial-monitor-node(s)}“， - mon01即为第一个MON节点名称，其名称必须与节点当前实际使用的主机名称(uname -n)保存一致 - 可以是短名称，也可以是长名称，但是最终用的仍然是短名称,但是会导致如下报错： ceph-deploy new: error: hostname:  xxx is not resolvable - 推荐使用完整写法： 格式 hostname:fqdn，比如 mon01:mon01.superopsmsb.com 注意： 如果初始化的时候，希望同时部署多个节点的换，使用空格隔开hostname:fqdn即可 如果部署过程出现问题，需要清空 ceph-deploy forgetkeys ceph-deploy purge mon01 ceph-deploy purgedata mon01 rm ceph.* ``` 集群初始化 ```powershell 部署3个mon节点 [cephadm@admin ceph-cluster]$ ceph-deploy new --public-network 10.0.0.0/24 --cluster-network 192.168.8.0/24 mon01:mon01.superopsmsb.com mon02:mon02.superopsmsb.com mon03:mon03.superopsmsb.com --no-ssh-copykey ... [ceph_deploy.new][DEBUG ] Resolving host mon03.superopsmsb.com [ceph_deploy.new][DEBUG ] Monitor mon03 at 10.0.0.15 [ceph_deploy.new][DEBUG ] Monitor initial members are ['mon01', 'mon02', 'mon03'] [ceph_deploy.new][DEBUG ] Monitor addrs are [u'10.0.0.13', u'10.0.0.14', u'10.0.0.15'] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... ``` ```powershell 注意： 如果出现如下报错： [ceph_deploy][ERROR ] AttributeError: 'module' object has no attribute 'needs_ssh' 在执行命令的时候，添加一个 --no-ssh-copykey 参数即可 这主要是因为免密认证的时候，没有进行 ssh cephadm@主机名 导致的 ``` 查看效果 ```powershell 查看初始化后的文件内容 [cephadm@admin ceph-cluster]$ ls ceph.conf ceph.log ceph.mon.keyring 查看集群的配置文件 [cephadm@admin ceph-cluster]$ cat ceph.conf [global] fsid = 7738ce65-2d87-481c-8253-9d0d4b29e8eb # 这个地方很重要的，每次都不一样，不要乱动 public_network = 10.0.0.0/24 cluster_network = 192.168.8.0/24 mon_initial_members = mon01, mon02, mon03 mon_host = 10.0.0.13,10.0.0.14,10.0.0.15 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true 查看集群通信的认证信息 [cephadm@admin ceph-cluster]$ cat ceph.mon.keyring [mon.] key = AQBZ9y1jAAAAABAAKnVONZ3l+EEpjUOjtK8Xmw== caps mon = allow * 查看集群初始化的日志信息 [cephadm@admin ceph-cluster]$ cat ceph.log [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf ... ``` **部署Mon** 部署mon软件 ```powershell 操作解析： ceph-deploy命令能够以远程的方式连入Ceph集群各节点完成程序包安装等操作 命令格式： ceph-deploy install {ceph-node} [{ceph-node} ...] 示例：ceph-deploy install --nogpgcheck admin mon01 mon02 mon03 注意： 这里主要是ceph的工作角色的的节点 一般情况下，不推荐使用这种直接的方法来进行安装，效率太低，而且容易干扰其他主机环境 ``` ```powershell 注意： 上面会在所有节点上都来进行正常的安装部署其实还有另外一种方法，手工在所有节点上安装ceph软件-- 推荐 yum install -y ceph ceph-osd ceph-mds ceph-mon ceph-radosgw 最后在admin角色主机上安装 ceph-deploy install --no-adjust-repos --nogpgcheck admin mon01 mon02 mon03 ``` ```powershell 执行过程 [cephadm@admin ceph-cluster]$ ceph-deploy install --no-adjust-repos --nogpgcheck admin mon01 mon02 mon03 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.25): /bin/ceph-deploy install --no-adjust-repos admin mon01 mon02 mon03 [ceph_deploy.install][DEBUG ] Installing stable version hammer on cluster ceph hosts admin mon01 mon02 mon03 [ceph_deploy.install][DEBUG ] Detecting platform for host mon01 ... [admin][DEBUG ] connection detected need for sudo [admin][DEBUG ] connected to host: admin ... [mon03][DEBUG ] 完毕！ [mon03][INFO ] Running command: sudo ceph --version [mon03][DEBUG ] ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) ``` 集群通信认证 ```powershell 配置初始MON节点,同时向所有节点同步配置 ceph-deploy mon create-initial 注意： 为了避免因为认证方面导致的通信失败，尤其是在现有环境上，推荐使用 --overwrite-conf 参数 ceph-deploy --overwrite-conf config push mon01 mon02 mon03 ``` ```powershell 执行效果 [cephadm@admin ceph-cluster]$ ceph-deploy mon create-initial [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy mon create-initial ... [mon01][DEBUG ] ******************************************************************************** [mon01][DEBUG ] status for monitor: mon.mon01 [mon01][DEBUG ] { [mon01][DEBUG ] "election_epoch": 0, [mon01][DEBUG ] "extra_probe_peers": [ ... [mon01][DEBUG ] "feature_map": { ... [mon01][DEBUG ] }, [mon01][DEBUG ] "features": { ... [mon01][DEBUG ] }, [mon01][DEBUG ] "monmap": { ... [mon01][DEBUG ] "mons": [ [mon01][DEBUG ] { [mon01][DEBUG ] "addr": "10.0.0.13:6789/0", [mon01][DEBUG ] "name": "mon01", [mon01][DEBUG ] "public_addr": "10.0.0.13:6789/0", [mon01][DEBUG ] "public_addrs": { [mon01][DEBUG ] "addrvec": [ [mon01][DEBUG ] { [mon01][DEBUG ] "addr": "10.0.0.13:3300", [mon01][DEBUG ] "nonce": 0, [mon01][DEBUG ] "type": "v2" [mon01][DEBUG ] }, [mon01][DEBUG ] { [mon01][DEBUG ] "addr": "10.0.0.13:6789", [mon01][DEBUG ] "nonce": 0, [mon01][DEBUG ] "type": "v1" [mon01][DEBUG ] } [mon01][DEBUG ] ] [mon01][DEBUG ] }, [mon01][DEBUG ] "rank": 0 [mon01][DEBUG ] }, [mon01][DEBUG ] { [mon01][DEBUG ] "addr": "0.0.0.0:0/1", [mon01][DEBUG ] "name": "mon02", ... [mon01][DEBUG ] "rank": 1 [mon01][DEBUG ] }, [mon01][DEBUG ] { [mon01][DEBUG ] "addr": "0.0.0.0:0/2", [mon01][DEBUG ] "name": "mon03", ... [mon01][DEBUG ] "rank": 2 [mon01][DEBUG ] } [mon01][DEBUG ] ] [mon01][DEBUG ] }, [mon01][DEBUG ] "name": "mon01", [mon01][DEBUG ] "outside_quorum": [ [mon01][DEBUG ] "mon01" [mon01][DEBUG ] ], [mon01][DEBUG ] "quorum": [], [mon01][DEBUG ] "rank": 0, [mon01][DEBUG ] "state": "probing", [mon01][DEBUG ] "sync_provider": [] [mon01][DEBUG ] } [mon01][DEBUG ] ******************************************************************************** [mon01][INFO ] monitor: mon.mon01 is running [mon01][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.mon01.asok mon_status ... [ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring [ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring [ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpyal7qW ``` ```powershell 到mon的节点上查看mon的守护进程 [cephadm@admin ceph-cluster]$ for i in {13..15}; do ssh cephadm@10.0.0.$i "ps aux | grep -v grep | grep ceph-mon"; done ceph 2189 0.1 1.8 504004 34644 ? Ssl 11:55 0:00 /usr/bin/ceph-mon -f --cluster ceph --id mon01 --setuser ceph --setgroup ceph ceph 2288 0.1 1.7 502984 33328 ? Ssl 11:55 0:00 /usr/bin/ceph-mon -f --cluster ceph --id mon02 --setuser ceph --setgroup ceph ceph 2203 0.1 1.7 502988 32912 ? Ssl 11:55 0:00 /usr/bin/ceph-mon -f --cluster ceph --id mon03 --setuser ceph --setgroup ceph 结果显示： 在所有的节点主机上，都有一套ceph-mon的进程在进行。 ``` ```powershell 集群在初始化的时候，会为对应的mon节点生成配套的认证信息 [cephadm@admin ceph-cluster]$ ls /home/cephadm/ceph-cluster ceph.bootstrap-mds.keyring ceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph-deploy-ceph.log ceph.bootstrap-mgr.keyring ceph.bootstrap-rgw.keyring ceph.conf ceph.mon.keyring ``` ```powershell 结果显示： 这里生成了一系列的与ceph集群相关的 认证文件 ceph.bootstrap-mds.keyring 引导启动 mds的秘钥文件 ceph.bootstrap-osd.keyring 引导启动 osd的秘钥文件 ceph.client.admin.keyring ceph客户端和管理端通信的认证秘钥，是最重要的 ceph-deploy-ceph.log ceph.bootstrap-mgr.keyring 引导启动 mgr的秘钥文件 ceph.bootstrap-rgw.keyring 引导启动 rgw的秘钥文件 ceph.conf ceph.mon.keyring 注意： ceph.client.admin.keyring 拥有ceph集群的所有权限，一定不能有误。 ``` **小结** ``` ``` ### 1.2.4 Ceph部署2 学习目标 这一节，我们从 Mon认证、Mgr环境、小结、三个方面来学习 **Mon认证** ```powershell 为了方便后续的监控环境认证操作，在admin角色主机上，把配置文件和admin密钥拷贝Ceph集群各监控角色节点,拷贝前秘钥文件前的各个mon节点效果 [cephadm@admin ceph-cluster]$ for i in {13..15}; do ssh cephadm@10.0.0.$i "echo -----$i-----; ls /etc/ceph"; done -----13----- ceph.conf rbdmap tmpc8lO0G -----14----- ceph.conf rbdmap tmpkmxmh3 -----15----- ceph.conf rbdmap tmp4GwYSs ``` ```powershell 原则上要求，所有mon节点上的 ceph.conf 内容必须一致，如果不一致的话，可以通过下面命令同步 ceph-deploy --overwrite-conf config push mon01 mon02 mon03 执行集群的认证文件的拷贝动作 ceph-deploy admin mon01 mon02 mon03 ``` ```powershell 执行认证文件信息同步 [cephadm@admin ceph-cluster]$ ceph-deploy admin mon01 mon02 mon03 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf ... [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mon01 ... [mon01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mon02 ... [mon02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mon03 ... [mon03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf ``` ```powershell 查看效果 [cephadm@admin ceph-cluster]$ for i in {13..15}; do ssh cephadm@10.0.0.$i "echo -----$i-----; ls /etc/ceph"; done -----13----- ceph.client.admin.keyring ceph.conf rbdmap tmpc8lO0G -----14----- ceph.client.admin.keyring ceph.conf rbdmap tmpkmxmh3 -----15----- ceph.client.admin.keyring ceph.conf rbdmap tmp4GwYSs 结果显示： 所有的mon节点上多了一个 ceph的客户端与服务端进行认证的秘钥文件了。 ceph.client.admin.keyring 主要用于 ceph客户端与管理端的一个通信认证。 注意：如果我们不做交互式操作的话，这个文件可以不用复制。 ``` 认证文件权限 ```powershell 虽然我们把认证文件传递给对应的监控角色主机了，但是我们的服务式通过普通用户cephadm来进行交流的。而默认情况下，传递过去的认证文件，cephadm普通用户是无法正常访问的 [cephadm@admin ceph-cluster]$ for i in {13..15}; do ssh cephadm@10.0.0.$i "echo -----$i-----; ls /etc/ceph/ceph.cl* -l"; done -----13----- -rw------- 1 root root 151 9月 25 12:06 /etc/ceph/ceph.client.admin.keyring -----14----- -rw------- 1 root root 151 9月 25 12:06 /etc/ceph/ceph.client.admin.keyring -----15----- -rw------- 1 root root 151 9月 25 12:06 /etc/ceph/ceph.client.admin.keyring ``` ```powershell 我们需要在Ceph集群中需要运行ceph命令的的节点上，以root用户的身份设定普通用户cephadm能够读 取/etc/ceph/ceph.client.admin.keyring文件的权限。 [cephadm@admin ceph-cluster]$ for i in {13..15}; do ssh cephadm@10.0.0.$i "sudo setfacl -m u:cephadm:r /etc/ceph/ceph.client.admin.keyring"; done 查看文件权限效果 [cephadm@admin ceph-cluster]$ for i in {13..15}; do ssh cephadm@10.0.0.$i "echo -----$i-----; ls /etc/ceph/ceph.cl* -l"; done -----13----- -rw-r-----+ 1 root root 151 9月 25 12:06 /etc/ceph/ceph.client.admin.keyring -----14----- -rw-r-----+ 1 root root 151 9月 25 12:06 /etc/ceph/ceph.client.admin.keyring -----15----- -rw-r-----+ 1 root root 151 9月 25 12:06 /etc/ceph/ceph.client.admin.keyring 查看文件的授权信息 [root@mon01 ~]# getfacl /etc/ceph/ceph.client.admin.keyring getfacl: Removing leading '/' from absolute path names # file: etc/ceph/ceph.client.admin.keyring # owner: root # group: root user::rw- user:cephadm:r-- group::--- mask::r-- other::--- ``` ```powershell 监控节点就可以自己来收集相关的数据了，比如我们在mon01上执行如下命令 [root@mon01 ~]# ceph -s cluster: id: 1d4e5773-619a-479d-861a-66ba451ce945 health: HEALTH_WARN mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 18m) mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 结果显示： 我们的cluster状态不是正常的 对于service来说，有三个mon服务，选举的节点有三个，其他的服务没有。 ``` ```powershell 集群状态不正常的原因，我们可以通过 ceph health命令来进行确认，效果如下 [root@mon01 ~]# ceph health HEALTH_WARN mons are allowing insecure global_id reclaim [root@mon01 ~]# ceph health detail HEALTH_WARN mons are allowing insecure global_id reclaim AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED mons are allowing insecure global_id reclaim mon.mon01 has auth_allow_insecure_global_id_reclaim set to true mon.mon02 has auth_allow_insecure_global_id_reclaim set to true mon.mon03 has auth_allow_insecure_global_id_reclaim set to true 结果显示： 我们在所有的mon节点上进行提示属性的设定 ``` ```powershell [root@mon01 ~]# ceph config set mon auth_allow_insecure_global_id_reclaim false [root@mon01 ~]# ceph health detail HEALTH_OK 结果显示： 集群状态问题已经解决了 ``` **Mgr环境** 需求 ```powershell Ceph-MGR工作的模式是事件驱动型的，简单来说，就是等待事件，事件来了则处理事件返回结果，又继续等待。Ceph MGR 是 Ceph 12.2 依赖主推的功能之一，它负责 Ceph 集群管理的组件，它主要功能是把集群的一些指标暴露给外界使用。根据官方的架构原则上来说，mgr要有两个节点来进行工作。 对于我们的学习环境来说，其实一个就能够正常使用了,为了节省资源的使用，我们这里将mon01 和mon02主机节点兼做MGR节点，为了后续的节点扩充实践，我们暂时先安装一个节点，后面再安装一个节点。 ``` ```powershell 未部署mgr节点的集群状态效果 [cephadm@admin ceph-cluster]$ ssh mon01 ceph -s cluster: id: 1d4e5773-619a-479d-861a-66ba451ce945 health: HEALTH_OK services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 29m) mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: ``` mgr服务配置 ```powershell 配置Manager节点，启动ceph-mgr进程： [cephadm@admin ceph-cluster]$ ceph-deploy mgr create mon01 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy mgr create mon01 [ceph_deploy.cli][INFO ] ceph-deploy options: ... [ceph_deploy.cli][INFO ] mgr : [('mon01', 'mon01')] ... [mon01][INFO ] Running command: sudo systemctl start ceph-mgr@mon ``` ```powershell 在指定的mgr节点上，查看守护进程 [cephadm@admin ceph-cluster]$ ssh mon01 ps aux | grep -v grep | grep ceph-mgr ceph 3065 2.8 6.6 1037824 124244 ? Ssl 12:27 0:01 /usr/bin/ceph-mgr -f --cluster ceph --id mon01 --setuser ceph --setgroup ceph 结果显示： 在mon01上，部署了一个mgr的服务进程 ``` ```powershell 查看集群服务的运行状态 [cephadm@admin ceph-cluster]$ ssh mon01 ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 33m) mgr: mon01(active, since 86s) osd: 0 osds: 0 up, 0 in data: ... 结果显示： 这个时候，service上，多了一个mgr的服务，在mon01节点上，服务状态是 active。 ``` admin查看状态 ```powershell 远程查看状态方式不太方便，我们可以在admin主机上进行一下操作来实现admin主机查看集群状态 sudo yum -y install ceph-common ceph-deploy admin admin sudo setfacl -m u:cephadm:rw /etc/ceph/ceph.client.admin.keyring ``` ```powershell 确认效果 [cephadm@admin ceph-cluster]$ ceph -s cluster: id: 1d4e5773-619a-479d-861a-66ba451ce945 health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3 services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 37m) mgr: mon01(active, since 4m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: ``` **小结** ``` ``` ### 1.2.5 OSD环境 学习目标 这一节，我们从 基本环境、OSD实践、小结、三个方面来学习。 **基本环境** 简介 ```powershell 我们知道对于osd来说，它进行真正数据的存储的引擎有两种：BlueStore和FileStore，自动Ceph L版之后，默认都是BlueStore了。 ``` 基本流程 ```powershell 一般来说，我们可以通过一下四个步骤来设置OSD环境： 1 要知道对应的主机上有哪些磁盘可以提供给主机来进行正常的使用。 2 格式化磁盘(非必须) 3 ceph擦除磁盘上的数据 4 添加osd ``` 1 确保提供专属数据磁盘，然后进行格式化 ```powershell 根据我们的了解，我们为所有的节点主机都准备了两块额外的磁盘， fdisk -l ``` 2 进行磁盘格式化 ```powershell 我们在所有的osd角色的主机上，进行磁盘的格式化操作,对所有的osd节点主机进行磁盘格式化。 mkfs.ext4 /dev/sdb mkfs.ext4 /dev/sdc ``` ```powershell 查看磁盘格式化效果，以mon03为例 [root@mon03 ~]# blkid | egrep "sd[bc]" /dev/sdb: UUID="49b5be7c-76dc-4ac7-a3e6-1f54526c83df" TYPE="ext4" /dev/sdc: UUID="ecdc0ce6-8045-4caa-b272-2607e70700ee" TYPE="ext4" ``` 3 ceph擦除磁盘上的数据 ```powershell 保证所有包含OSD磁盘上主机上，安装ceph的命令 yum install -y ceph radosgw ``` ```powershell 检查并列出OSD节点上所有可用的磁盘的相关信息 [cephadm@admin ceph-cluster]$ ceph-deploy disk list stor01 stor02 stor03 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk list stor01 stor02 stor03 ... [stor01][DEBUG ] connection detected need for sudo ... [stor01][INFO ] Running command: sudo fdisk -l [stor02][DEBUG ] connection detected need for sudo ... [stor02][INFO ] Running command: sudo fdisk -l [stor03][DEBUG ] connection detected need for sudo ... [stor03][INFO ] Running command: sudo fdisk -l ``` ```powershell 在管理节点上使用ceph-deploy命令擦除计划专用于OSD磁盘上的所有分区表和数据以便用于OSD， [cephadm@admin ceph-cluster]$ for i in {1..3} do ceph-deploy disk zap stor0$i /dev/sdb /dev/sdc done ... [stor03][WARNIN] stderr: 记录了10+0 的读入 [stor03][WARNIN] 记录了10+0 的写出 [stor03][WARNIN] 10485760字节(10 MB)已复制 [stor03][WARNIN] stderr: ，0.018883 秒，555 MB/秒 [stor03][WARNIN] --> Zapping successful for: ``` **OSD实践** 命令解析 ```powershell 对于osd的相关操作，可以通过 ceph-deploy osd 命令来进行，帮助信息如下 [cephadm@admin ceph-cluster]$ ceph-deploy osd --help usage: ceph-deploy osd [-h] {list,create} ... Create OSDs from a data disk on a remote host: ceph-deploy osd create {node} --data /path/to/device For bluestore, optional devices can be used:: ceph-deploy osd create {node} --data /path/to/data --block-db /path/to/db-device ceph-deploy osd create {node} --data /path/to/data --block-wal /path/to/wal-device ceph-deploy osd create {node} --data /path/to/data --block-db /path/to/db-device --block-wal /path/to/wal-device For filestore, the journal must be specified, as well as the objectstore:: ceph-deploy osd create {node} --filestore --data /path/to/data --journal /path/to/journal For data devices, it can be an existing logical volume in the format of: vg/lv, or a device. For other OSD components like wal, db, and journal, it can be logical volume (in vg/lv format) or it must be a GPT partition. positional arguments: {list,create} list List OSD info from remote host(s) create Create new Ceph OSD daemon by preparing and activating a device ... ``` ```powershell 帮助显示：这里提示了两类的存储机制： 对于bluestore来说，它主要包括三类数据： --data /path/to/data ceph 保存的对象数据 --block-db /path/to/db-device ceph 保存的对象数据 --block-wal /path/to/wal-device 数据库的 wal 日志 对于filestore来说，它主要包括两类数据 --data /path/to/data ceph的文件数据 --journal /path/to/journal 文件系统日志数据 对于 osd来说，它主要有两个动作： list 列出osd相关的信息 create 创建osd设备 ``` 添加OSD命令解读 ```powershell 对于osd的创建来说，我们来看一下他的基本格式 [cephadm@admin ceph-cluster]$ ceph-deploy osd create --help usage: ceph-deploy osd create [-h] [--data DATA] [--journal JOURNAL] [--zap-disk] [--fs-type FS_TYPE] [--dmcrypt] [--dmcrypt-key-dir KEYDIR] [--filestore] [--bluestore] [--block-db BLOCK_DB] [--block-wal BLOCK_WAL] [--debug] [HOST] positional arguments: HOST Remote host to connect optional arguments: -h, --help show this help message and exit --data DATA The OSD data logical volume (vg/lv) or absolute path to device --journal JOURNAL Logical Volume (vg/lv) or path to GPT partition --zap-disk DEPRECATED - cannot zap when creating an OSD --fs-type FS_TYPE filesystem to use to format DEVICE (xfs, btrfs) --dmcrypt use dm-crypt on DEVICE --dmcrypt-key-dir KEYDIR directory where dm-crypt keys are stored --filestore filestore objectstore --bluestore bluestore objectstore --block-db BLOCK_DB bluestore block.db path --block-wal BLOCK_WAL bluestore block.wal path --debug Enable debug mode on remote ceph-volume calls 结果显示： 对于osd的创建，默认情况下用的就是 bluestore类型， ``` 4 添加osd ```powershell 创建osd，我们这里全部用于存储数据。 ceph-deploy --overwrite-conf osd create stor01 --data /dev/sdb ceph-deploy --overwrite-conf osd create stor01 --data /dev/sdc 注意： 这里只能一个磁盘一个磁盘的添加 ``` ```powershell 查看效果 [cephadm@admin ceph-cluster]$ ceph-deploy --overwrite-conf osd create stor01 --data /dev/sdc [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy --overwrite-conf osd create stor01 --data /dev/sdc ... [stor01][WARNIN] --> ceph-volume lvm activate successful for osd ID: 1 [stor01][WARNIN] --> ceph-volume lvm create successful for: /dev/sdc [stor01][INFO ] checking OSD status... [stor01][DEBUG ] find the location of an executable [stor01][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json [ceph_deploy.osd][DEBUG ] Host stor01 is now ready for osd use. ``` ```powershell 查看命令执行后的ceph的集群状态 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 59m) mgr: mon01(active, since 27m) osd: 2 osds: 2 up (since 98s), 2 in (since 98s) data: ... 结果显示： 在services部分，的osd多了信息，有两个osd是up的状态,而且都加入到了集群中。 ``` ```powershell 接下来，我们通过批量操作的方式，将其他节点主机的磁盘都格式化 for i in 2 3 do ceph-deploy --overwrite-conf osd create stor0$i --data /dev/sdb ceph-deploy --overwrite-conf osd create stor0$i --data /dev/sdc done 执行后的效果如下： ``` ```powershell 再次查看集群状态 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 66m) mgr: mon01(active, since 34m) osd: 6 osds: 6 up (since 5s), 6 in (since 5s) data: ... 结果显示： osd的磁盘数量达到了 6个，都是处于up的状态。 对于ceph集群的数据容量来说，一共有120G的磁盘空间可以使用 ``` 查看节点磁盘状态 ```powershell [cephadm@admin ceph-cluster]$ ceph-deploy osd list stor01 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy osd list stor01 ... [stor01][DEBUG ] connection detected need for sudo [stor01][DEBUG ] connected to host: stor01 [stor01][DEBUG ] detect platform information from remote host [stor01][DEBUG ] detect machine type [stor01][DEBUG ] find the location of an executable [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.9.2009 Core [ceph_deploy.osd][DEBUG ] Listing disks on stor01... [stor01][DEBUG ] find the location of an executable [stor01][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm list [stor01][DEBUG ] [stor01][DEBUG ] [stor01][DEBUG ] ====== osd.0 ======= [stor01][DEBUG ] ... [stor01][DEBUG ] type block [stor01][DEBUG ] vdo 0 [stor01][DEBUG ] devices /dev/sdb [stor01][DEBUG ] [stor01][DEBUG ] ====== osd.1 ======= [stor01][DEBUG ] ... [stor01][DEBUG ] type block [stor01][DEBUG ] vdo 0 [stor01][DEBUG ] devices /dev/sdc ``` OSD的磁盘状态查看 ```powershell 对于osd来说，它还有一个专门用于osd管理的 命令 ceph，相关的帮助信息如下 ceph osd --help 帮助解析： 这里面有几个是与osd信息查看相关的 ls 查看所有OSD的id值 dump 查看 OSD 的概述性信息 status 查看 OSD 的详细的状态信息 stat 查看 OSD 的精简的概述性信息 ``` ```powershell 查看所有OSD的id值 [cephadm@admin ceph-cluster]$ ceph osd ls 0 1 2 3 4 5 查看 OSD 的概述性信息 [cephadm@admin ceph-cluster]$ ceph osd dump epoch 25 ... max_osd 6 osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:10.0.0.13:6802/5544,v1:10.0.0.13:6803/5544] [v2:192.168.8.13:6800/5544,v1:192.168.8.13:6801/5544] exists,up 78bd7a7e-9dcf-4d91-be20-d5de2fbec7dd ... 查看 OSD 的精简的概述性信息 [cephadm@admin ceph-cluster]$ ceph osd stat 6 osds: 6 up (since 4m), 6 in (since 4m); epoch: e25 ``` **小结** ``` ``` ### 1.2.6 OSD操作 学习目标 这一节，我们从 基本实践、进阶实践、小结、三个方面来学习。 **基本实践** 简介 ```powershell OSD全称Object Storage Device，负责响应客户端请求返回具体数据的进程。一个Ceph集群中，有专门的osd角色主机，在这个主机中一般有很多个OSD设备。 ``` ```powershell 对于osd来说，它还有一个专门用于osd管理的 命令 ceph，相关的帮助信息如下 ceph osd --help 帮助解析： 这里面有几个是与osd信息查看相关的 ls 查看所有OSD的id值 dump 查看 OSD 的概述性信息 status 查看 OSD 的详细的状态信息 stat 查看 OSD 的精简的概述性信息 tree 查看 OSD 在主机上的分布信息 perf 查看 OSD 磁盘的延迟统计信息 df 查看 OSD 磁盘的使用率信息 ``` 命令查看 ```powershell 查看所有OSD的id值 [cephadm@admin ceph-cluster]$ ceph osd ls 0 1 2 3 4 5 ``` ```powershell 查看 OSD 的数据映射信息 [cephadm@admin ceph-cluster]$ ceph osd dump epoch 25 ... max_osd 6 osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:10.0.0.13:6802/5544,v1:10.0.0.13:6803/5544] [v2:192.168.8.13:6800/5544,v1:192.168.8.13:6801/5544] exists,up 78bd7a7e-9dcf-4d91-be20-d5de2fbec7dd ... 查看指定OSD节点的信息 [cephadm@admin ceph-cluster]$ ceph osd dump 3 epoch 3 fsid 1d4e5773-619a-479d-861a-66ba451ce945 ... osd.0 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists,new 78bd7a7e-9dcf-4d91-be20-d5de2fbec7dd ``` ```powershell 查看 OSD 的精简的概述性信息 [cephadm@admin ceph-cluster]$ ceph osd stat 6 osds: 6 up (since 4m), 6 in (since 4m); epoch: e25 状态解析： OSD节点数量(osds) 集群内(in)、集群外(out) 运行(up)、不再运行(down) OSD 的每一次状态变更的历史信息(epoch) ``` ```powershell 查看 OSD 的详细的状态信息 [cephadm@admin ceph-cluster]$ ceph osd status +----+-------+-------+-------+--------+---------+--------+---------+-----------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-------+-------+-------+--------+---------+--------+---------+-----------+ | 0 | mon01 | 1027M | 18.9G | 0 | 0 | 0 | 0 | exists,up | | 1 | mon01 | 1027M | 18.9G | 0 | 0 | 0 | 0 | exists,up | | 2 | mon02 | 1027M | 18.9G | 0 | 0 | 0 | 0 | exists,up | | 3 | mon02 | 1027M | 18.9G | 0 | 0 | 0 | 0 | exists,up | | 4 | mon03 | 1027M | 18.9G | 0 | 0 | 0 | 0 | exists,up | | 5 | mon03 | 1027M | 18.9G | 0 | 0 | 0 | 0 | exists,up | +----+-------+-------+-------+--------+---------+--------+---------+-----------+ ``` ```powershell 查看OSD在各个主机上的分布情况 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 up 1.00000 1.00000 1 hdd 0.01949 osd.1 up 1.00000 1.00000 -5 0.03897 host mon02 2 hdd 0.01949 osd.2 up 1.00000 1.00000 3 hdd 0.01949 osd.3 up 1.00000 1.00000 -7 0.03897 host mon03 4 hdd 0.01949 osd.4 up 1.00000 1.00000 5 hdd 0.01949 osd.5 up 1.00000 1.00000 ``` ```powershell 查看OSD磁盘的延迟统计信息 [cephadm@admin ceph-cluster]$ ceph osd perf osd commit_latency(ms) apply_latency(ms) 5 0 0 4 0 0 0 0 0 1 0 0 2 0 0 3 0 0 结果显示： 主要解决单块磁盘问题，如果有问题及时剔除OSD。统计的是平均值 commit_latency 表示从接收请求到设置commit状态的时间间隔 apply_latency 表示从接收请求到设置apply状态的时间间隔 ``` ```powershell 查看OSD磁盘的使用率信息 [cephadm@admin ceph-cluster]$ ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.01949 1.00000 20 GiB 1.0 GiB 3.2 MiB 0 B 1 GiB 19 GiB 5.02 1.00 0 up 1 hdd 0.01949 1.00000 20 GiB 1.0 GiB 3.2 MiB 0 B 1 GiB 19 GiB 5.02 1.00 0 up 2 hdd 0.01949 1.00000 20 GiB 1.0 GiB 3.2 MiB 0 B 1 GiB 19 GiB 5.02 1.00 0 up 3 hdd 0.01949 1.00000 20 GiB 1.0 GiB 3.2 MiB 0 B 1 GiB 19 GiB 5.02 1.00 0 up 4 hdd 0.01949 1.00000 20 GiB 1.0 GiB 3.2 MiB 0 B 1 GiB 19 GiB 5.02 1.00 0 up 5 hdd 0.01949 1.00000 20 GiB 1.0 GiB 3.2 MiB 0 B 1 GiB 19 GiB 5.02 1.00 0 up TOTAL 120 GiB 6.0 GiB 20 MiB 0 B 6 GiB 114 GiB 5.02 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 ``` **进阶实践** osd 暂停开启 ```powershell 命令格式： ceph osd pause 集群暂停接收数据 ceph osd unpause 集群开始接收数据 ``` ```powershell [cephadm@admin ceph-cluster]$ ceph osd pause pauserd,pausewr is set [cephadm@admin ceph-cluster]$ ceph -s ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 2h) mgr: mon01(active, since 2h) osd: 6 osds: 6 up (since 107m), 6 in (since 107m) flags pauserd,pausewr # 可以看到，多了pause的标签 ... [cephadm@admin ceph-cluster]$ ceph osd unpause pauserd,pausewr is unset [cephadm@admin ceph-cluster]$ ceph -s ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 2h) mgr: mon01(active, since 2h) osd: 6 osds: 6 up (since 107m), 6 in (since 107m) ... # 可以看到，pause的标签已经被移除 ``` osd数据操作比重 ```powershell 命令格式： osd节点上线：ceph osd crush reweight osd.编号 权重值 ``` ```powershell 查看默认的OSD操作权重值 [cephadm@admin ceph-cluster]$ ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 1 hdd 0.01949 osd.1 ... 修改osd的数据操作权重值 [cephadm@admin ceph-cluster]$ ceph osd crush reweight osd.0 0.1 reweighted item id 0 name 'osd.0' to 0.1 in crush map [cephadm@admin ceph-cluster]$ ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 0.19742 root default -3 0.11948 host mon01 0 hdd 0.09999 osd.0 1 hdd 0.01949 osd.1 ... 恢复osd的数据操作权重值 [cephadm@admin ceph-cluster]$ ceph osd crush reweight osd.0 0.01949 reweighted item id 0 name 'osd.0' to 0.01949 in crush map [cephadm@admin ceph-cluster]$ ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 1 hdd 0.01949 osd.1 ``` osd上下线 ```powershell 命令格式： osd节点上线：ceph osd down osd编号 osd节点下线：ceph osd up osd编号 注意： 由于OSD有专门的管理服务器控制，一旦发现被下线，会尝试启动它 ``` ```powershell 将磁盘快速下线，然后查看状态 [cephadm@admin ceph-cluster]$ ceph osd down 0 ; ceph osd tree marked down osd.0. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 down 1.00000 1.00000 ... 等待一秒钟后查看状态，指定的节点又上线了 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 up 1.00000 1.00000 ... ``` 驱逐加入OSD对象 ``` 命令格式： ceph osd out osd编号 ceph osd in osd编号 注意： 所谓的从OSD集群中驱离或者加入OSD对象，本质上Ceph集群数据操作的权重值调整 ``` ```powershell 将0号OSD下线 [cephadm@admin ceph-cluster]$ ceph osd out 0 marked out osd.0. [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 up 0 1.00000 ... 将0号OSD上线 [cephadm@admin ceph-cluster]$ ceph osd in 0; marked in osd.0. [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 up 1.00000 1.00000 ... 结果显示： OSD在上下线实践的时候，所谓的REWEIGHT会进行调整，1代表上线，空代表下线 ``` **小结** ```powershell ``` ### 1.2.7 OSD节点 学习目标 这一节，我们从 OSD删除、OSD添加、小结 三个方面来学习 **OSD删除** 基本步骤 ```powershell 将OSD删除需要遵循一定的步骤： 1 修改osd的数据操作权重值，让数据不分布在这个节点上 2 到指定节点上，停止指定的osd进程 3 将移除OSD节点状态标记为out 4 从crush中移除OSD节点，该节点不作为数据的载体 5 删除OSD节点 6 删除OSD节点的认证信息 ``` 删除OSD节点实践 ```powershell 修改osd的数据操作权重值 [cephadm@admin ceph-cluster]$ ceph osd crush reweight osd.5 0 reweighted item id 5 name 'osd.5' to 0 in crush map [cephadm@admin ceph-cluster]$ ceph osd crush tree ID CLASS WEIGHT TYPE NAME ... -7 0.01949 host mon03 4 hdd 0.01949 osd.4 5 hdd 0 osd.5 ``` ```powershell 到指定节点上，停止指定的osd进程 [cephadm@admin ceph-cluster]$ ssh mon03 sudo systemctl disable ceph-osd@5 Removed symlink /etc/systemd/system/ceph-osd.target.wants/ceph-osd@5.service. [cephadm@admin ceph-cluster]$ ssh mon03 sudo systemctl stop ceph-osd@5 [cephadm@admin ceph-cluster]$ ssh mon03 sudo systemctl status ceph-osd@5 ... 9月 25 15:25:32 mon03 systemd[1]: Stopped Ceph object storage daemon osd.5. ``` ```powershell 将移除OSD节点状态标记为out [cephadm@admin ceph-cluster]$ ceph osd out osd.5 marked out osd.5. [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.01949 host mon03 4 hdd 0.01949 osd.4 up 1.00000 1.00000 5 hdd 0 osd.5 down 0 1.00000 ``` ```powershell 从crush中移除OSD节点，该节点不作为数据的载体 [cephadm@admin ceph-cluster]$ ceph osd crush remove osd.5 removed item id 5 name 'osd.5' from crush map 查看效果 [cephadm@admin ceph-cluster]$ ceph osd crush tree ID CLASS WEIGHT TYPE NAME ... -7 0.01949 host mon03 4 hdd 0.01949 osd.4 结果显示： osd.5 已经被移除了 ``` ```powershell 删除OSD节点前查看效果 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.01949 host mon03 4 hdd 0.01949 osd.4 up 1.00000 1.00000 5 0 osd.5 down 0 1.00000 移除无效的osd节点 [cephadm@admin ceph-cluster]$ ceph osd rm osd.5 removed osd.5 再次确认效果 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.01949 host mon03 4 hdd 0.01949 osd.4 up 1.00000 1.00000 结果显示： osd节点已经被移除了 ``` ```powershell 查看历史认证信息 [cephadm@admin ceph-cluster]$ ceph auth ls ... osd.5 key: AQCy4C9jI1wzDhAAaPjSPSQpMdu8NUPIRecctQ== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * ... 删除OSD节点的认证信息 [cephadm@admin ceph-cluster]$ ceph auth del osd.5 updated [cephadm@admin ceph-cluster]$ ceph auth ls ... 结果显示： 已经没有历史的节点信息了 ``` **OSD添加** 基本步骤 ```powershell 将OSD增加到集群需要遵循一定的步骤： 1 确定OSD节点没有被占用 2 磁盘格式化 3 ceph擦除磁盘上的数据 4 添加OSD到集群 ``` 添加OSD节点实践简介 ```powershell 确定OSD节点没有被占用 [root@mon03 ~]# blkid | egrep 'sd[bc]' /dev/sdb: UUID="h3NNr5-Sgr8-nJ8k-QWbY-UiRu-Pbmu-nxs3pZ" TYPE="LVM2_member" /dev/sdc: UUID="VdOW5k-zxMN-4Se5-Bxlc-8h93-uhPA-R30P3u" TYPE="LVM2_member" 格式化磁盘失败 [root@mon03 ~]# mkfs.ext4 /dev/sdc mke2fs 1.42.9 (28-Dec-2013) /dev/sdc is entire device, not just one partition! 无论如何也要继续? (y,n) y /dev/sdc is apparently in use by the system; will not make a 文件系统 here! ``` ```powershell 查看被占用的磁盘 [root@mon03 ~]# dmsetup status ceph--9dd352be--1b6e--4d60--957e--81c59eafa7b3-osd--block--49251ee9--b1cc--432f--aed7--3af897911b60: 0 41934848 linear ceph--3712fe04--cc04--4ce4--a75e--f7f746d62d6f-osd--block--99787a7e--40ee--48ae--802a--491c4f326d0c: 0 41934848 linear 1 确定OSD节点没有被占用，注意ceph的osd挂载目录 [root@mon03 ~]# cat /var/lib/ceph/osd/ceph-4/fsid 49251ee9-b1cc-432f-aed7-3af897911b60 移除被占用磁盘 [root@mon03 ~]# dmsetup remove ceph--3712fe04--cc04--4ce4--a75e--f7f746d62d6f-osd--block--99787a7e--40ee--48ae--802a--491c4f326d0c [root@mon03 ~]# dmsetup status | grep 99787a7e ``` ```powershell 2 磁盘格式化 [root@mon03 ~]# mkfs.ext4 /dev/sdc ``` ```powershell 3 ceph擦除磁盘上的数据 [cephadm@admin ceph-cluster]$ ceph-deploy disk zap stor03 /dev/sdc ``` ```powershell 4 添加OSD到集群 ceph-deploy --overwrite-conf osd create stor03 --data /dev/sdc ``` ```powershell 确认效果 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -7 0.03897 host mon03 4 hdd 0.01949 osd.4 up 1.00000 1.00000 5 hdd 0.01949 osd.5 up 1.00000 1.00000 结果显示： 之前被移除的osd节点已经被找回来了 ``` **小结** ``` ``` ### 1.2.8 存储实践 学习目标 这一节，我们从 基本环境、OSD实践、小结、三个方面来学习。 **基本环境** 存储术语 ```powershell Pool RADOS存储集群提供的基础存储服务需要由“存储池（pool）”分割为逻辑存储 区域，此类的逻辑区域亦是对象数据的名称空间。 PG 归置组（Placement Group）是用于跨OSD将数据存储在某个存储池中的内部数据 结构 相对于存储池来说，PG是一个虚拟组件，它是对象映射到存储池时使用的虚拟层 是实现大容量集群的关键效率技术 PGP (Placement Group for Placement)是用于维持PG和OSD的一种策略。 防止OSD重新分配时候，PG找不到之前的OSD，从而引起大范围的数据迁移 CRUSH 把对象直接映射到OSD之上会导致二者之间的紧密耦合关系，在OSD设备变动时不可避免地对整个集群产生扰动。所以需要一种策略算法来处理这种问题。 Ceph将一个对象映射进RADOS集群的过程分为两步 - 首先是以一致性哈希算法将对象名称映射到PG - 而后是将PG ID基于CRUSH算法映射到OSD CRUSH(Controlled Replication Under Scalable Hashing),它是一种数据分布式算法，类似于一致性哈希算法，用于为RADOS存储集群控制数据分布。 ``` 基本逻辑图 ![1636103641572](../../img/kubernetes/kubernetes_storage_ceph/1636103641572-1664092679864.png) 需求 ```powershell OSD平台的目的就是数据的存在，我们这里就先来简单的演示一下，OSD环境的数据操作。这里主要临时演示两个功能: 数据存储 - 客户端连接至RADOS集群上某存储池,根据相关的CRUSH规则完成数据对象存储. 数据删除 - 集合配套的命令，实现数据的移除功能。 ``` **OSD实践** 创建存储池 ```powershell 命令格式 ceph osd pool create [pgp-num] [replicated] \ [crush-rule-name] [expected-num-objects] 参数解析： pool-name：存储池名称，在一个RADOS存储集群上必须具有唯一性； pg-num：当前存储池中的PG数量，一定要合理 pgp-num ：用于归置的PG数量，其值应该等于PG的数量 replicated：存储池类型；副本存储池需更多原始存储空间，但已实现Ceph支持的所有操 作 crush-ruleset-name：此存储池所用的CRUSH规则集的名称，引用的规则集必须事先存在 查看命令 ceph osd pool ls rados lspools ``` ```powershell 创建一个存储池，名称为mypool，pg数量和pgp数量都是16 [cephadm@admin ceph-cluster]$ ceph osd pool create mypool 16 16 pool 'mypool' created 查看存储池的列表 [cephadm@admin ceph-cluster]$ ceph osd pool ls mypool [cephadm@admin ceph-cluster]$ rados lspools mypool ``` 数据的上传 ```powershell 命令格式： 虽然我们目前没有形成专用的数据接口，但是ceph提供了一个原理的文件测试接口 -- rados命令 rados put 文件对象名(id) /path/to/file --pool=存储池 ``` ```powershell 提交文件到对应的osd里面 [cephadm@admin ceph-cluster]$ rados put ceph-file /home/cephadm/ceph-cluster/ceph.conf --pool=mypool 确认上传数据效果 [cephadm@admin ceph-cluster]$ rados ls --pool=mypool ceph-file ``` 查看数据的存储关系 ```powershell 命令格式： 通过属性的方式获取到存储池中数据对象的具体位置信息 ceph osd map 存储池 文件对象名(id) ``` ```powershell 查看ceph-file文件对象的内部属性关系 [cephadm@admin ceph-cluster]$ ceph osd map mypool ceph-file osdmap e51 pool 'mypool' (1) object 'ceph-file' -> pg 1.7753490d (1.d) -> up ([2,1,5], p2) acting ([2,1,5], p2) 结果解析： 可以看到文件对象的内部属性关系 [ num ] 是副本所存储的osd id的值 ``` 数据删除实践 ```powershell 命令格式： 将文件对象从pool里面删除 rados rm 文件对象名(id) --pool=存储池 ``` ```powershell 将刚才添加的文件对象从存储池里面移除 [cephadm@admin ceph-cluster]$ rados rm ceph-file --pool=mypool 查看存储池的内容 [cephadm@admin ceph-cluster]$ rados ls --pool=mypool [cephadm@admin ceph-cluster]$ ``` **小结** ``` ``` ### 1.2.9 存储解析 学习目标 这一节，我们从 存储解析、存储删除、小结、三个方面来学习。 **基本环境** 数据存储逻辑 ![1636103641572](../../img/kubernetes/kubernetes_storage_ceph/1636103641572.png) ```powershell pool 是ceph存储数据时的逻辑分区，它起到据不同的用户场景，基于namespace实现隔离故障域的作用。 每个pool包含一定数量的PG, PG里的对象被映射到不同的OSD上。 OSD分散到所有的主机磁盘上 ``` 存储池基本信息 ```powershell 命令格式 ceph osd pool ls [detail] ceph osd pool stats {} ``` ```powershell 查看存储池名称 [cephadm@admin ceph-cluster]$ ceph osd pool ls mypool 查看存储池详情 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail pool 1 'mypool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 51 flags hashpspool stripe_width 0 确认存储池状态 [cephadm@admin ceph-cluster]$ ceph osd pool stats mypool pool mypool id 1 nothing is going on 结果显示： 对于mypool来说，它的id是1，该存储池里面的所有pg都是以1为开头的 ``` 存储池&PG ```powershell 通过 pg 查找 pool ceph pg dump | grep "^{poolid}\." 通过 pool 查找 pg ceph pg ls-by-pool {poolname} ceph pg ls {poolid} ``` ```powershell 根据pg查找pools [cephadm@admin ceph-cluster]$ ceph pg dump | grep "^1. " ... [cephadm@admin ceph-cluster]$ ceph pg dump pools POOLID OBJECTS ... 1 0 ... 根据存储池找PG [cephadm@admin ceph-cluster]$ ceph pg ls-by-pool mypool | awk '{print $1,$2,$15}' PG OBJECTS ACTING 1.0 0 [3,5,0]p3 ... 1.e 0 [2,1,5]p2 1.f 0 [3,1,4]p3 看出mypool的pg都是以1开头的,确定pg的分布情况 [cephadm@admin ceph-cluster]$ ceph pg dump pgs|grep ^1|awk '{print $1,$2,$15,$19}' dumped pgs 1.f 0 0'0 [3,1,4] ... 1.9 0 0'0 [3,4,1] 结果显示： 每个pg都会分布在三个osd上，整个集群有6个osd ``` PG & OSD ```powershell 通过 pg 查找 osd ceph pg map {pgid} 通过 osd 查找 pg ceph pg ls-by-osd osd.{osdid} ``` ```powershell 根据pg找osd的分布 [cephadm@admin ceph-cluster]$ ceph pg map 1.1 osdmap e51 pg 1.1 (1.1) -> up [5,0,2] acting [5,0,2] 根据osd找pg的分布 [cephadm@admin ceph-cluster]$ ceph pg ls-by-osd osd.1 | awk '{print $1,$2,$10,$14,$15}' PG OBJECTS STATE UP ACTING 1.3 0 active+clean [1,2,5]p1 [1,2,5]p1 ... 1.f 0 active+clean [3,1,4]p3 [3,1,4]p3 * NOTE: and soon afterwards ``` **存储删除** 存储池删除 ```powershell 命令格式： 删除存储池命令存在数据丢失的风险，Ceph于是默认禁止此类操作。 管理员需要在ceph.conf配置文件中启用支持删除动作 ceph osd pool rm 存储池名 存储池名 --yes-i-really-really-mean-it 注意： 存储池名称必须出现两遍，后面的 参数代表是强制 ``` 删除实践 ```powershell 默认情况下删除存储池 [cephadm@admin ceph-cluster]$ ceph osd pool rm mypool mypool --yes-i-really-really-mean-it Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool ``` ```powershell 主节点上修改ceph.conf文件，增加两行配置，让其运行删除pool [cephadm@admin ceph-cluster]$ tail -n2 ceph.conf [mon] mon allow pool delete = true 同步ceph.conf文件到所有的ceph节点上 [cephadm@admin ceph-cluster]$ ceph-deploy --overwrite-conf config push admin mon01 mon02 mon03 重启所有的mon节点上的ceph-mon服务 [cephadm@admin ceph-cluster]$ for i in mon0{1..3}; do ssh $i "sudo systemctl restart ceph-mon.target"; done ``` ```powershell 将刚才添加的文件对象从存储池里面移除 [cephadm@admin ceph-cluster]$ ceph osd pool rm mypool mypool --yes-i-really-really-mean-it pool 'mypool' removed 确认效果 [cephadm@admin ceph-cluster]$ ceph osd pool stats there are no pools! [cephadm@admin ceph-cluster]$ ceph osd pool ls [cephadm@admin ceph-cluster]$ ``` **小结** ```powershell ``` ### 1.2.10 环境完善 学习目标 这一节，我们从 扩展mon、扩展mgr、小结、三个方面来学习。 **扩展mon** 操作mon节点基础 ```powershell 命令格式： ceph-deploy mon add mon节点名称 注意：如果add换成destroy，则变成移除mon节点 ``` ```powershell 查看ceph的mon状态 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 3m) mgr: mon01(active, since 4h) osd: 6 osds: 6 up (since 60m), 6 in (since 60m) [cephadm@admin ceph-cluster]$ ceph mon stat e1: 3 mons at {mon01=[v2:10.0.0.13:3300/0,v1:10.0.0.13:6789/0],mon02=[v2:10.0.0.14:3300/0,v1:10.0.0.14:6789/0],mon03=[v2:10.0.0.15:3300/0,v1:10.0.0.15:6789/0]}, election epoch 6, leader 0 mon01, quorum 0,1,2 mon01,mon02,mon03 ``` 移除实践 ```powershell 移除mon节点 [cephadm@admin ceph-cluster]$ ceph-deploy mon destroy mon03 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf ... 查看效果 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 2 daemons, quorum mon01,mon02 (age 9s) mgr: mon01(active, since 4h) osd: 6 osds: 6 up (since 63m), 6 in (since 63m) 结果显示： mon03已经被移除了 ``` 扩展实践 ```powershell 扩展mon实践 [cephadm@admin ceph-cluster]$ ceph-deploy mon add mon03 ... 确认效果 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 2s) mgr: mon01(active, since 4h) osd: 6 osds: 6 up (since 65m), 6 in (since 65m) 结果显示： mon03已经被添加到集群了 ``` **扩展mgr** 简介 ```powershell Ceph Manager在高可用的场景下，守护进程以“Active/Standby”模式运行，部署其它ceph-mgr守护程序可确保在Active节点或其上的ceph-mgr守护进程故障时，其中的一个Standby实例可以在不中断服务的情况下接管其任务。如果只有一个mgr服务器的话，守护进程的状态是Active。 ``` ```powershell 确认效果 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 2s) mgr: mon01(active, since 4h) osd: 6 osds: 6 up (since 65m), 6 in (since 65m) ``` 扩展实践 ```powershell 在当前的环境中，添加mgr节点 [cephadm@admin ceph-cluster]$ ceph-deploy mgr create mon02 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf ... 查看效果 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 3m) mgr: mon01(active, since 4h), standbys: mon02 osd: 6 osds: 6 up (since 69m), 6 in (since 69m) 结果显示： mon01节点就是我们的主角色节点，mon02是我们的从角色节点。 ``` **小结** ``` ``` # 1 核心实践 ## 1.1 认证管理 ### 1.1.1 认证基础 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell Ceph作为一个分布式存储系统，支持对象存储、块设备和文件系统。为了在网络传输中防止数据被篡改，做到较高程度的安全性，加入了Cephx加密认证协议。其目的是客户端和管理端之间的身份识别，加密、验证传输中的数据。 注意：所谓的client就是使用ceph命令的客户端 ``` ```powershell ceph集群默认开启了cephx协议功能 [cephadm@admin ceph-cluster]$ cat ceph.conf [global] ... auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx ... 如果我们提供的是公共的信息 -- 不需要进行认证，所以我们可以禁用CephX功能。将对应的属性值设置为 none即可，比如 auth_cluster_required = none ``` 认证场景 ```powershell 命令行与Ceph集群操作，包括： RadosGW 对象网关认证（对象网关认证系统+cephx） RBD 认证 CephFS 用户认证（文件路径+cephx） Ceph集群内部组件之间通信 执行ceph命令时，会使用client.admin 用户并加载 /etc/ceph/ceph.client.admin.keyring文件 ``` 认证和授权 ```powershell 在ceph系统中，所有元数据保存在mon节点的ceph-mon的进程中，mon保存了系统中重要的认证相关元数据，例如每个用户的key以及权限。样式结构如下： 名称 key Caps（权限） client.admin xxxxxx osd allow rw, mon allow rw ``` ```powershell 对于ceph的认证和授权来说，主要涉及到三个内容：ceph用户、资源权限、用户授权。 - Ceph用户必须拥有执行权限才能执行Ceph的管理命令 - Ceph用户需要拥有存储池访问权限才能到ceph中读取和写入数据 ``` 基本概念 ```powershell 用户 - ceph创建出来的用户，可以进入到ceph集群里面 - ceph支持多种类型，可管理的用户属于Client类型 - MON、OSD和MDS等系统组件属于系统参与者客户端 授权 - 将某些资源的使用权限交给特定的用户。 - allow 权限 - 描述用户可针对MON、OSD或MDS等资源的使用权限范围或级别 - MON具备的权限包括r、w、x和allow profile cap - OSD具备的权限包括r、w、x和class-read、class-write和profile osd、存储池和名称空间设置 - OSD具备的权限包括allow ``` 权限解析 ```powershell allow 需先于守护进程的访问设置指定，标识赋予的xx权限 r：读取权限，访问MON以检索CRUSH时依赖此使能 w：对象写入权限 x：调用类方法（读取和写入）的能力，以及在MON上执行auth操作的能力 class-read：x能力的子集，授予用户调用类读取方法的能力 class-write：x的子集，授予用户调用类写入方法的能力 *：授予用户对特定守护进程/存储池的读取、写入和执行权限，以及执行管理命令的能力 ``` ```powershell profile osd 授予用户以某个OSD身份连接到其他OSD或监视器的权限 profile mds 授予用户以某个MDS身份连接到其他MDS或监视器的权限 profile bootstrap-osd 授予用户引导OSD的权限，在部署时候产生 profile bootstrap-mds 授予用户引导元数据服务器的权限，在部署时候产生 ``` **简单实践** CephX身份验正流程 ![image-20211206175437937](../../img/kubernetes/kubernetes_storage_ceph/image-20211206175437937.png) ```powershell client和osd都有一个叫做monclient的模块负责认证和密钥交换。而monitor上有一个AuthMonitor的服务模块负责与monclient对话。 ``` ```powershell Ceph使用cephx协议对客户端进行身份认证 每个MON都可以对客户端进行身份验正并分发密钥，不存在单点故障和性能瓶颈,MON会返回用于身份验正的数据结构，其包含获取Ceph服务时用到的session key. - session key通过客户端密钥进行加密 - 客户端使用session key向MON请求所需的服务 - MON向客户端提供一个ticket，用于向实际处理数据的OSD等验正客户端身份 - MON和OSD共享同一个secret，因此OSD会信任由MON发放的ticket - ticket存在有效期限 注意： CephX身份验正功能仅限制Ceph的各组件之间，它不能扩展到其它非Ceph组件； 它并不解决数据传输加密的问题； ``` CephX身份验正-MDS和OSD ![image-20211206175909027](../../img/kubernetes/kubernetes_storage_ceph/image-20211206175909027.png) ```powershell 1 当client与Monitor实现基本的认证后，monitor与后端的mds和osd会自动进行认证信息的同步 2 当client与Monitor实现基本的通信认证后 可以独立与后端的MDS服务发送请求。 可以独立与后端的OSD服务发送请求 ``` **小结** ``` ``` ### 1.1.2 用户基础 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell Ceph集群管理员能够直接在Ceph集群中创建、更新和删除用户 注意： 创建用户时，可能需要将密钥分发到客户端，以便将密钥添加到密钥环 ``` 常见命令 ```powershell 列出用户 命令：ceph auth list 用户标识：TYPE.ID，因此，osd.0表示OSD类型的用户，用户ID为0 检索特定用户 命令：ceph auth get TYPE.ID或者ceph auth export TYPE.ID 列出用户的密钥 命令：ceph auth print-key TYPE.ID ``` **简单实践** 信息查看 ```powershell 查看用户账号 [cephadm@admin ceph-cluster]$ ceph auth list osd.0 key: AQCv3i9j34BWKhAAII3+THENuLJTv5Yr2NwDxA== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * ... 结果显示： 每个osd都有类似的caps权限信息 client.admin具备的权限是非常多的 ``` ```powershell 查看制定的认证信息 [cephadm@admin ceph-cluster]$ ceph auth get osd.0 [osd.0] key = AQCv3i9j34BWKhAAII3+THENuLJTv5Yr2NwDxA== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" exported keyring for osd.0 ``` ```powershell 查看秘钥文件信息 [cephadm@admin ceph-cluster]$ ls /etc/ceph/ ceph.client.admin.keyring ceph.conf rbdmap tmpr0oj5H 查看秘钥环相关的信息 [cephadm@admin ceph-cluster]$ cat /etc/ceph/ceph.client.admin.keyring [client.admin] key = AQAr0S9jK8RNLxAAD/xpDZgnAx1kQEV3+/utWw== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" 注意： 一个秘钥环里面可以存放很多秘钥信息的 ``` ```powershell 列出用户秘钥信息 [cephadm@admin ceph-cluster]$ ceph auth print-key mgr.mon01 AQC12C9j/hAgKhAA6lOZiUpz5kqA55PFv3eHng== ``` **小结** ``` ``` ### 1.1.3 用户实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 在ceph中，针对用户主要有两种场景：增删和导出导入 ``` 命令解析 ```powershell 添加用户 ceph auth add：创建用户、生成密钥并添加指定的caps ceph auth get-or-create：创建用户并返回密钥文件格式的密钥信息，用户存在时返回密钥信息 ceph auth get-or-create-key：创建用户并返回密钥信息，用户存在时返回密钥信息 注意： 典型的用户至少对 Ceph monitor 具有读取功能，并对 Ceph OSD 具有读取和写入功能； 另外，用户的 OSD 权限通常应该限制为只能访问特定的存储池， 否则，他将具有访问集群中所有存储池的权限 ``` ```powershell 导入用户 命令：ceph auth import 修改用户caps 命令：ceph auth caps 会覆盖用户现有的caps，因此建立事先使用ceph auth get TYPE.ID命令查看用户的caps 若是为添加caps，则需要先指定现有的caps,命令格式如下： ceph auth caps TYPE.ID daemon 'allow [r|w|x|*|...] [pool=pool-name]' ... 删除用户 命令：ceph auth del TYPE.ID ``` **简单实践** 添加用户 ```powershell 查看帮助信息 [cephadm@admin ceph-cluster]$ ceph auth --help General usage: ============== usage: ceph [-h] [-c CEPHCONF] [-i INPUT_FILE] [-o OUTPUT_FILE] [--setuser SETUSER] [--setgroup SETGROUP] [--id CLIENT_ID] [--name CLIENT_NAME] [--cluster CLUSTER] [--admin-daemon ADMIN_SOCKET] [-s] [-w] [--watch-debug] [--watch-info] [--watch-sec] [--watch-warn] [--watch-error] [--watch-channel {cluster,audit,*}] [--version] [--verbose] [--concise] [-f {json,json-pretty,xml,xml-pretty,plain}] [--connect-timeout CLUSTER_TIMEOUT] [--block] [--period PERIOD] ``` ```powershell 创建普通用户 [cephadm@admin ceph-cluster]$ ceph auth add client.testuser mon 'allow r' osd 'allow rw pool=rdbpool' added key for client.testuser 获取创建的用户信息 [cephadm@admin ceph-cluster]$ ceph auth get client.testuser [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== caps mon = "allow r" caps osd = "allow rw pool=rdbpool" exported keyring for client.testuser 列出用户的秘钥信息 [cephadm@admin ceph-cluster]$ ceph auth print-key client.testuser AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== ``` 用户授权 ```powershell 修改用户的授权 [cephadm@admin ceph-cluster]$ ceph auth caps client.testuser mon 'allow rw' osd 'allow rw pool=rdbpool1' updated caps for client.testuser 查看修改后的授权信息 [cephadm@admin ceph-cluster]$ ceph auth get client.testuser [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== caps mon = "allow rw" caps osd = "allow rw pool=rdbpool1" exported keyring for client.testuser ``` 导出用户 ```powershell 查看导出信息 [cephadm@admin ceph-cluster]$ ceph auth export client.testuser [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== caps mon = "allow rw" caps osd = "allow rw pool=rdbpool1" export auth(key=AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg==) ``` ```powershell 导出信息到一个备份文件 [cephadm@admin ceph-cluster]$ ceph auth export client.testuser > testuser.file export auth(key=AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg==) [cephadm@admin ceph-cluster]$ cat testuser.file [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== caps mon = "allow rw" caps osd = "allow rw pool=rdbpool1" ``` 删除用户 ```powershell 删除用户信息 [cephadm@admin ceph-cluster]$ ceph auth del client.testuser updated 查看效果 [cephadm@admin ceph-cluster]$ ceph auth get client.testuser Error ENOENT: failed to find client.testuser in keyring ``` 导入用户 ```powershell 导入用户文件 [cephadm@admin ceph-cluster]$ ceph auth import -i testuser.file imported keyring 查看文件效果 [cephadm@admin ceph-cluster]$ ceph auth get client.testuser [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== caps mon = "allow rw" caps osd = "allow rw pool=rdbpool1" exported keyring for client.testuser ``` 尝试创建一个未知的用户 ```powershell 创建一个已知的用户 [cephadm@admin ceph-cluster]$ ceph auth get-or-create client.testuser [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== 结果显示： 如果是一个已存在的用户名，则会返回具体的信息，而且不会覆盖现有的用户信息 [cephadm@admin ceph-cluster]$ ceph auth get client.testuser [client.testuser] key = AQDfOTBjsmRaEhAATcpXwDluXBpshy4420/zgg== caps mon = "allow rw" caps osd = "allow rw pool=rdbpool1" exported keyring for client.testuser ``` ```powershell 创建一个未知的用户 [cephadm@admin ceph-cluster]$ ceph auth get-or-create client.testuser2 mon 'allow r' osd 'allow rw pool=rdbpool' [client.testuser2] key = AQAwPDBjHPByABAA+nMEvAXztrzE8FSCZkNGgg== 如果从未出现过的用户，则直接创建新的用户 [cephadm@admin ceph-cluster]$ ceph auth get client.testuser2 [client.testuser2] key = AQAwPDBjHPByABAA+nMEvAXztrzE8FSCZkNGgg== caps mon = "allow r" caps osd = "allow rw pool=rdbpool" exported keyring for client.testuser2 ``` **小结** ``` ``` ### 1.1.4 秘钥管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** keyring ```powershell 密钥环文件是“存储机密、密码、密钥、证书并使它们可用于应用程序的组件的集合”。密钥环文件存储一个或多个 Ceph 身份验证密钥以及可能的相关功能规范。 注意： 每个键都与一个实体名称相关联，形式为 {client,mon,mds,osd}.name。 ceph-authtool 是一个用于创建、查看和修改 Ceph 密钥环文件的实用程序。 ``` 秘钥环文件信息 ```powershell 访问Ceph集群时，客户端会于本地查找密钥环，只有，认证成功的对象，才可以正常使用。 默认情况下，Ceph会使用以下四个密钥环名称预设密钥环 /etc/ceph/cluster-name.user-name.keyring：保存单个用户的keyring /etc/ceph/cluster.keyring：保存多个用户的keyring /etc/ceph/keyring /etc/ceph/keyring.bin 注意： cluster-name是为集群名称，user-name是为用户标识（TYPE.ID） client.admin用户的在名为ceph的集群上的密钥环文件名为ceph.client.admin.keyring ``` keyring的管理 ```powershell 创建keyring ceph auth add等命令添加的用户还需要额外使用ceph-authtool命令为其创建用户密钥环文件,ceph客户端通过keyring文件查找用户名并检索密钥 命令：ceph-authtool --create-keyring /path/to/kerying 注意 keyring文件一般应该保存于/etc/ceph目录中，以便客户端能自动查找 创建包含多个用户的keyring文件时，应该使用cluster-name.keyring作为文件名 创建仅包含单个用户的kerying文件时，应该使用cluster-name.user-name.keyring作为文件名 ``` ```powershell 将用户添加至keyring 可将某个用户从包含多个用户的keyring中导出，并保存于一个专用的keyring文件 命令：ceph auth get TYPE.ID -o /etc/ceph/cluster-name.user-name.keyring 也可将用户的keyring合并至一个统一的keyring文件中 命令：ceph-authtool /etc/ceph/cluster-name.keyring --import-key /etc/ceph/clustername. user-name.keyring ``` ```powershell ceph-authtool命令可直接创建用户、授予caps并创建keyring ceph-authtool keyringfile [-C | --create-keyring] [-n | --name entityname] [--genkey] [-a | --add-key base64_key] [--cap | --caps capfile] 注意： 此种方式添加的用户仅存在于keyring文件中，管理员还需要额外将其添加至Ceph集群上； 命令： ceph auth add TYPE.ID -i /PATH/TO/keyring ``` **简单实践** 创建携带秘钥环的账号 ```powershell 创建普通格式的用户 [cephadm@admin ceph-cluster]$ ceph auth get-or-create client.kube mon 'allow r' osd 'allow * pool=kube' [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== ``` ```powershell 查看用户信息 [cephadm@admin ceph-cluster]$ ceph auth get client.kube [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== caps mon = "allow r" caps osd = "allow * pool=kube" exported keyring for client.kube 查看文件 [cephadm@admin ceph-cluster]$ ls ceph.bootstrap-mds.keyring ceph.bootstrap-rgw.keyring ceph-deploy-ceph.log ceph.bootstrap-mgr.keyring ceph.client.admin.keyring ceph.mon.keyring ceph.bootstrap-osd.keyring ceph.conf testuser.file 结果显示： 没有生成对应的用户秘钥环文件 ``` 导出秘钥环文件 ```powershell 将普通的用户导出为keyring [cephadm@admin ceph-cluster]$ ceph auth get client.kube -o ceph.client.kube.keyring exported keyring for client.kube ``` ```powershell 查看效果 [cephadm@admin ceph-cluster]$ ll *kube* -rw-rw-r-- 1 cephadm cephadm 117 9月 26 06:26 ceph.client.kube.keyring [cephadm@admin ceph-cluster]$ cat ceph.client.kube.keyring [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== caps mon = "allow r" caps osd = "allow * pool=kube" ``` 合并秘钥环文件 ```powershell 创建要合并的文件 [cephadm@admin ceph-cluster]$ ceph-authtool --create-keyring cluster.keyring creating cluster.keyring [cephadm@admin ceph-cluster]$ ll cluster.keyring -rw------- 1 cephadm cephadm 0 9月 26 06:28 cluster.keyring 合并要导入的keyring文件 [cephadm@admin ceph-cluster]$ ceph-authtool cluster.keyring --import-keyring ceph.client.kube.keyring importing contents of ceph.client.kube.keyring into cluster.keyring [cephadm@admin ceph-cluster]$ cat cluster.keyring [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== caps mon = "allow r" caps osd = "allow * pool=kube" ``` ```powershell 再来合并一个用户 c[cephadm@admin ceph-cluster]$ ceph-authtool cluster.keyring --import-keyring ceph.client.admin.keyring importing contents of ceph.client.admin.keyring into cluster.keyring 查看合并后效果 [cephadm@admin ceph-cluster]$ cat cluster.keyring [client.admin] key = AQAr0S9jK8RNLxAAD/xpDZgnAx1kQEV3+/utWw== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== caps mon = "allow r" caps osd = "allow * pool=kube" ``` ```powershell 专用查看keyring的内容 [cephadm@admin ceph-cluster]$ ceph-authtool -l cluster.keyring [client.admin] key = AQAr0S9jK8RNLxAAD/xpDZgnAx1kQEV3+/utWw== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== caps mon = "allow r" caps osd = "allow * pool=kube" ``` **小结** ``` ``` ## 1.2 RBD接口 ### 1.2.1 基础知识 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ![1636089258096](../../img/kubernetes/kubernetes_storage_ceph/1636089258096.png) ```powershell Ceph块设备，也称为RADOS块设备（简称RBD），是一种基于RADOS存储系统支持超配（thin-provisioned）、可伸缩的条带化数据存储系统，它通过librbd库与OSD进行交互。RBD为KVM等虚拟化技术和云OS（如OpenStack和CloudStack）提供高性能和无限可扩展性的存储后端，这些系统依赖于libvirt和QEMU实用程序与RBD进行集成。 ``` RBD ```powershell RBD，全称RADOS Block Devices，是一种建构在RADOS存储集群之上为客户端提供块设备接口的存储服务中间层. - 这类的客户端包括虚拟化程序KVM（结合qemu）和云的计算操作系统OpenStack和CloudStack等 RBD基于RADOS存储集群中的多个OSD进行条带化，支持存储空间的简配（thinprovisioning）和动态扩容等特性，并能够借助于RADOS集群实现快照、副本和一致性。同时，RBD自身也是RADOS存储集群的客户端，它通过将存储池提供的存储服务抽象为一到多个image（表现为块设备）向客户端提供块级别的存储接口 - RBD支持两种格式的image，不过v1格式因特性较少等原因已经处于废弃状态 - 当前默认使用的为v2格式 ``` ```powershell 客户端访问RBD设备的方式有两种 - 通过内核模块rbd.ko将image映射为节点本地的块设备，相关的设备文件一般为/dev/rdb#（#为设备编号，例如rdb0等） - 通过librbd提供的API接口，它支持C/C++和Python等编程语言，qemu即是此类接口的客户端 ``` 操作逻辑 ```powershell RBD接口在ceph环境创建完毕后，就在服务端自动提供了; 客户端基于librbd库即可将RADOS存储集群用作块设备。不过，RADOS集群体用块设备需要经过一系列的"额外操作"才能够为客户端提供正常的存储功能。 这个过程步骤，主要有以下几步： - 1 创建一个专用的存储池 ceph osd pool create {pool-name} {pg-num} {pgp-num} - 2 对存储池启用rbd功能 ceph osd pool application enable {pool-name} rbd - 3 对存储池进行环境初始化 rbd pool init -p {pool-name} - 4 基于存储池创建专用的磁盘镜像 rbd create --size --pool ``` ```powershell 其他命令 存储池中的各image名称需要惟一，“rbd ls”命令能够列出指定存储池中的image rbd ls [-p ] [--format json|xml] [--pretty-format] 要获取指定image的详细信息，则通常使用“rbd info”命令 rbd info [--pool ] [--image ] [--format ] ... ``` **简单实践** 创建专用存储池 ```powershell [cephadm@admin ceph-cluster]$ ceph osd pool create rbddata 64 pool 'rbddata' created 注意： 这里面的pg数量，我们定制为64个 ``` 对存储池启用rbd功能 ```powershell 启用存储池的rbd功能 [cephadm@admin ceph-cluster]$ ceph osd pool application enable rbddata rbd enabled application 'rbd' on pool 'rbddata' 注意： 如果关闭应用的话，使用disable 查看rbc的效果 [cephadm@admin ceph-cluster]$ ceph osd pool application get rbddata { "rbd": {} } ``` 对存储池进行环境初始化 ```powershell 环境初始化 [cephadm@admin ceph-cluster]$ rbd pool init -p rbddata 查看效果 [cephadm@admin ceph-cluster]$ rbd pool stats rbddata Total Images: 0 Total Snapshots: 0 Provisioned Size: 0 B ``` 基于存储池创建专用的磁盘镜像 ```powershell 创建镜像 [cephadm@admin ceph-cluster]$ rbd create img1 --size 1024 --pool rbddata [cephadm@admin ceph-cluster]$ rbd ls -p rbddata img1 查看状态 [cephadm@admin ceph-cluster]$ rbd pool stats rbddata Total Images: 1 Total Snapshots: 0 Provisioned Size: 1 GiB ``` ```powershell 注意： 这个时候，我们创建出来的磁盘影响文件，就可以在客户端上，通过内核机制，直接导入到内核中，在内核中被当成一个磁盘设备来进行使用，样式就是 /dev/xxx，然后就可以针对这个rdb磁盘设备，进行各种后续分区、格式化等操作。 ``` 查看磁盘信息 ```powershell 查看方法1： [cephadm@admin ceph-cluster]$ rbd info rbddata/img1 [cephadm@admin ceph-cluster]$ rbd --image img1 --pool rbddata info rbd image 'img1': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 3817a3738fac block_name_prefix: rbd_data.3817a3738fac format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Sep 26 10:46:25 2062 access_timestamp: Mon Sep 26 10:46:25 2062 modify_timestamp: Mon Sep 26 10:46:25 2062 ``` ```powershell size : 就是这个块的大小，即1024MB=1G，1024MB/256 = 4M,共分成了256个对象(object)，每个对象4M。 order 22： 有效范围为12-25。22是个幂编号，4M是22， 8M是23，也就是2^22 bytes = 4MB, 2^23 bytes = 8MB block_name_prefix : 这个是块的最重要的属性了，这是每个块在ceph中的唯一前缀编号，有了这个前缀，把主机上的OSD都拔下来带回家，就能复活所有的VM了 format : 格式有两种，1和2 features：当前image启用的功能特性，其值是一个以逗号分隔的字符串列表，例如 layering, exclusive-lock, object-map op_features：可选的功能特性； ``` 磁盘设备的删除 ```powershell 删除设备 [cephadm@admin ceph-cluster]$ rbd rm rbddata/img1 Removing image: 100% complete...done. 查看效果 [cephadm@admin ceph-cluster]$ rbd ls -p rbddata [cephadm@admin ceph-cluster]$ rbd pool stats rbddata Total Images: 0 Total Snapshots: 0 Provisioned Size: 0 B 删除存储池 [cephadm@admin ceph-cluster]$ ceph osd pool rm rbddata rbddata --yes-i-really-really-mean-it pool 'rbddata' removed ``` **小结** ``` ``` ### 1.2.2 镜像管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 镜像属性 | 属性 | 解析 | | -------------- | ------------------------------------------------------------ | | layering | 分层克隆机制，磁盘的数据分层获取克隆机制 | | striping | 是否支持数据对象间的数据条带化 | | exclusive-lock | 排它锁的机制，磁盘应用于多路读写机制场景。限制同时仅能有一个客户端访问当前image | | object-map | 对象位图机制，主要用于加速导入、导出及已用容量统计等操作，依赖于exclusive-lock特性 | | fast-diff | 快照定制机制，快速对比数据差异，便于做快照管理，依赖于object-map特性 | | deep-flatten | 数据处理机制，解除父子image及快照的依赖关系 | | journaling | 磁盘日志机制，将image的所有修改操作进行日志化，便于异地备份，依赖于exclusive-lock特性 | | data-pool | 是否支持将image的数据对象存储于纠删码存储池，主要用于将image的元数据与数据放置于不同的存储池； | 属性操作 ```powershell 镜像功能命令 cephadm@admin:~/ceph-cluster$ rbd --help | grep feature feature disable Disable the specified image feature. feature enable Enable the specified image feature. ``` **简单实践** 准备工作 ```powershell 创建存储池 [cephadm@admin ceph-cluster]$ ceph osd pool create kube 64 64 pool 'kube' created [cephadm@admin ceph-cluster]$ ceph osd pool ls kube 启动rbd功能 [cephadm@admin ceph-cluster]$ ceph osd pool application enable kube rbd enabled application 'rbd' on pool 'kube' rbd环境初始化 [cephadm@admin ceph-cluster]$ rbd pool init kube ``` 创建镜像 ```powershell 方法2：通过参数属性方式，创建一个2G的镜像文件 [cephadm@admin ceph-cluster]$ rbd create --pool super --size 2G --image vol01 查看镜像 [cephadm@admin ceph-cluster]$ rbd ls --pool kube vol01 ``` ```powershell 方法2：通过方式，创建一个2G的镜像文件 [cephadm@admin ceph-cluster]$ rbd create --size 2G kube/vol02 查看镜像 [cephadm@admin ceph-cluster]$ rbd ls --pool kube vol01 vol02 ``` ```powershell 查看详情信息 -l [cephadm@admin ceph-cluster]$ rbd ls --pool kube -l NAME SIZE PARENT FMT PROT LOCK vol01 2 GiB 2 vol02 2 GiB 2 以json格式查看详情 --format json --pretty-format [cephadm@admin ceph-cluster]$ rbd ls --pool kube -l --format json --pretty-format [ { "image": "vol01", "size": 2147483648, "format": 2 }, { "image": "vol02", "size": 2147483648, "format": 2 } ] ``` ```powershell 查看镜像属性 [cephadm@admin ceph-cluster]$ rbd info kube/vol01 rbd image 'vol01': size 2 GiB in 512 objects order 22 (4 MiB objects) snapshot_count: 0 id: 386ed735e96b block_name_prefix: rbd_data.386ed735e96b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten ... ``` 镜像属性 ```powershell 禁用磁盘镜像功能 [cephadm@admin ceph-cluster]$ rbd feature disable kube/vol01 object-map fast-diff deep-flatten [cephadm@admin ceph-cluster]$ rbd info --pool kube --image vol01 rbd image 'vol01': size 2 GiB in 512 objects ... features: layering, exclusive-lock ... ``` ```powershell 启用磁盘镜像功能 [cephadm@admin ceph-cluster]$ rbd feature enable kube/vol01 object-map fast-diff [cephadm@admin ceph-cluster]$ rbd info --pool kube --image vol01 rbd image 'vol01': size 2 GiB in 512 objects ... flags: object map invalid, fast diff invalid ... ``` **小结** ``` ``` ### 1.2.3 镜像实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 磁盘使用 ```powershell 使用方法1：通过内核级别的ceph模块，将rdb设备提供给相关主机使用 初始化存储池后，创建image 最好禁用 object-map，fast-diff，deep-flatten功能 需要直接与monitor角色进行通信 - 若存储集群端启用了CephX认证，还需要指定用户名和keyring文件 使用rdb命令的map子命令，进行磁盘文件的映射 注意： 在客户端上可以通过 rbd showmapped 查看已经映射的image 在客户端上可以通过 rbd unmap 命令断开已经映射的image ``` admin主机管理image镜像文件的权限 ```powershell 禁用无效的属性 [cephadm@admin ceph-cluster]$ rbd feature disable kube/vol01 object-map fast-diff [cephadm@admin ceph-cluster]$ rbd info --pool kube --image vol01 rbd image 'vol01': size 2 GiB in 512 objects ... features: layering, exclusive-lock ``` 映射命令的帮助 ```powershell [cephadm@admin ceph-cluster]$ rbd help map usage: rbd map [--device-type ] [--pool ] [--namespace ] [--image ] [--snap ] [--read-only] [--exclusive] [--options ] ... ``` **简单实践** 客户端主机安装基本环境 ```powershell 客户端主机安装环境 [root@stor06 ~]# yum install ceph ceph-common -y ``` ```powershell 查看模块效果 [root@stor06 ~]# modinfo ceph filename: /lib/modules/3.10.0-1160.el7.x86_64/kernel/fs/ceph/ceph.ko.xz license: GPL description: Ceph filesystem for Linux author: Patience Warnick author: Yehuda Sadeh author: Sage Weil alias: fs-ceph retpoline: Y rhelversion: 7.9 srcversion: EB765DDC1F7F8219F09D34C depends: libceph intree: Y vermagic: 3.10.0-1160.el7.x86_64 SMP mod_unload modversions signer: CentOS Linux kernel signing key sig_key: E1:FD:B0:E2:A7:E8:61:A1:D1:CA:80:A2:3D:CF:0D:BA:3A:A4:AD:F5 sig_hashalgo: sha256 ``` 管理端主机传递相关文件 ```powershell 查看认证信息 [cephadm@admin ceph-cluster]$ ceph auth get client.kube [client.kube] key = AQA11TBjzq6NEhAAF2tyvEY0L+XYMP7IOykkcQ== caps mon = "allow r" caps osd = "allow * pool=super" exported keyring for client.kube [cephadm@admin ceph-cluster]$ ls *kube* ceph.client.kube.keyring 传递认证文件 [cephadm@admin ceph-cluster]$ sudo scp ceph.client.kube.keyring root@stor06:/etc/ceph/ ``` ```powershell 传递正常的ceph集群的配置文件 [cephadm@admin ceph-cluster]$ sudo scp ceph.conf root@stor06:/etc/ceph/ ``` 客户端简单使用 ```powershell 默认情况下，是无法正常连接ceph集群的 root@stor06:~# ceph -s [root@stor06 ~]# ceph -s 2062-09-26 11:05:04.530 7f7cca27f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory ... [errno 2] error connecting to the cluster 结果显示： 默认采用的认证用户是 client.admin账号，而我们没有。 通过--user参数，使用指定的用户来访问ceph [root@stor06 ~]# ceph --user kube -s cluster: id: 1d4e5773-619a-479d-861a-66ba451ce945 health: HEALTH_OK ... ``` rdb文件的使用 ```powershell 查看当前的系统磁盘效果 [root@stor06 ~]# fdisk -l | grep '磁盘 ' 磁盘 /dev/sda：21.5 GB, 21474836480 字节，41943040 个扇区 磁盘 /dev/sdb：21.5 GB, 21474836480 字节，41943040 个扇区 磁盘 /dev/sdc：21.5 GB, 21474836480 字节，41943040 个扇区 ``` ```powershell 映射远程ceph的磁盘文件到本地 root@stor06:~# rbd --user kube map kube/vol01 /dev/rbd0 查看效果 [root@stor06 ~]# fdisk -l | grep '磁盘 ' ... 磁盘 /dev/rbd0：2147 MB, 2147483648 字节，4194304 个扇区 ``` ```powershell 格式化磁盘 [root@stor06 ~]# mkfs.ext4 /dev/rbd0 mke2fs 1.45.5 (07-Jan-2020) ... 写入超级块和文件系统账户统计信息： 已完成 ``` ```powershell 挂载操作 [root@stor06 ~]# mount /dev/rbd0 /mnt [root@stor06 ~]# mount | grep rbd /dev/rbd0 on /mnt type ext4 (rw,relatime,stripe=1024,data=ordered) 尝试使用磁盘文件 [root@stor06 ~]# cp /etc/issue /mnt/ [root@stor06 ~]# cat /mnt/issue ``` ```powershell 卸载操作 [root@stor06 ~]# umount /mnt ``` 我们还可以在客户端上，使用 rbd命令对rbd设备进行管理 ```powershell 客户端挂载操作 [root@stor06 ~]# mount /dev/rbd0 /mnt 确认效果 [root@stor06 ~]# rbd showmapped id pool namespace image snap device 0 kube vol01 - /dev/rbd0 ``` ```powershell 主服务器上，查看挂载效果 [cephadm@admin ceph-cluster]$ rbd ls --pool kube -l NAME SIZE PARENT FMT PROT LOCK vol01 2 GiB 2 excl vol02 2 GiB 2 ``` 磁盘卸载操作 ```powershell 卸载操作 [root@stor06 ~]# rbd unmap /dev/rbd0 rbd: sysfs write failed rbd: unmap failed: (16) Device or resource busy 卸载磁盘 [root@stor06 ~]# umount /mnt [root@stor06 ~]# rbd unmap /dev/rbd0 查看挂载效果 [root@stor06 ~]# rbd showmapped 主节点查看挂载效果 [cephadm@admin ceph-cluster]$ rbd ls --pool kube -l NAME SIZE PARENT FMT PROT LOCK vol01 2 GiB 2 vol02 2 GiB 2 ``` **小结** ``` ``` ### 1.2.4 容量管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 容量 ```powershell 如果有必要的话，我们可以针对镜像文件的大小进行容量的调整。 [cephadm@admin ceph-cluster]$ rbd help resize usage: rbd resize [--pool ] [--namespace ] [--image ] --size [--allow-shrink] [--no-progress] ``` 调整image的大小 ```powershell 增大： rbd resize [--pool ] [--image ] --size 减少： rbd resize [--pool ] [--image ] --size [--allow-shrink] ``` 删除image ```powershell 命令格式 rbd remove [--pool ] [--image ] ... 注意： 删除image会导致数据丢失，且不可恢复； ``` ```powershell 如果删除一些敏感的image，为了保险，推荐使用回收站功能trash，确定不再需要时再从trash中删除； 命令格式： rbd trash {list|move|purge|remove|restore} ``` **简单实践** image的容量扩展 ```powershell 查看当前的image容量 [cephadm@admin ceph-cluster]$ rbd ls -p kube -l NAME SIZE PARENT FMT PROT LOCK vol01 2 GiB 2 vol02 2 GiB 2 调整image容量 [cephadm@admin ceph-cluster]$ rbd resize -s 5G kube/vol01 Resizing image: 100% complete...done. 再次确认image容量大小 [cephadm@admin ceph-cluster]$ rbd ls -p kube -l NAME SIZE PARENT FMT PROT LOCK vol01 5 GiB 2 vol02 2 GiB 2 ``` image容量的缩小 ```powershell 调整image容量 [cephadm@admin ceph-cluster]$ rbd resize -s 3G kube/vol01 --allow-shrink Resizing image: 100% complete...done. 再次确认image容量大小 [cephadm@admin ceph-cluster]$ rbd ls -p kube -l NAME SIZE PARENT FMT PROT LOCK vol01 3 GiB 2 vol02 2 GiB 2 ``` 删除image文件 ```powershell 删除image文件 [cephadm@admin ceph-cluster]$ rbd rm kube/vol02 Removing image: 100% complete...done. 查看image文件效果 [cephadm@admin ceph-cluster]$ rbd ls -p kube -l NAME SIZE PARENT FMT PROT LOCK vol01 3 GiB 2 注意： image文件的删除是不可修复的 ``` image文件恢复 ```powershell 查看trash命令帮助 [cephadm@admin ceph-cluster]$ rbd trash --help | grep trash trash list (trash ls) List trash images. trash move (trash mv) Move an image to the trash. trash purge Remove all expired images from trash. trash remove (trash rm) Remove an image from trash. trash restore Restore an image from trash. ``` 简介 ```powershell 查看回收站 [cephadm@admin ceph-cluster]$ rbd trash ls -p kube 移动文件到回收站 [cephadm@admin ceph-cluster]$ rbd trash move kube/vol01 查看回收站 [cephadm@admin ceph-cluster]$ rbd trash ls -p kube 386ed735e96b vol01 查看文件 [cephadm@admin ceph-cluster]$ rbd ls -p kube -l [cephadm@admin ceph-cluster]$ ``` ```powershell 恢复image文件 [cephadm@admin ceph-cluster]$ rbd trash restore -p kube --image vol01 --image-id 386ed735e96b 查看回收站 [cephadm@admin ceph-cluster]$ rbd trash ls -p kube 查看image文件 [cephadm@admin ceph-cluster]$ rbd ls -p kube -l NAME SIZE PARENT FMT PROT LOCK vol01 3 GiB 2 ``` **小结** ``` ``` ### 1.2.5 快照管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 快照 ```powershell RBD支持image快照技术，借助于快照可以保留image的状态历史。Ceph还支持快照“分层”机制，从而可实现快速克隆VM映像。 ``` 常见命令 ```powershell 创建快照 rbd snap create [--pool ] --image --snap 或者 rbd snap create [/]@ 注意： 在创建映像快照之前应停止image上的IO操作，且image上存在文件系统时，还要确保其处于一致状态； ``` ```powershell 列出快照 rbd snap ls [--pool ] --image ... ``` ```powershell 回滚快照 rbd snap rollback [--pool ] --image --snap ... 注意： 意味着会使用快照中的数据重写当前版本的image，回滚所需的时间将随映像大小的增加而延长 ``` ```powershell 限制快照数量 快照数量过多，必然会导致image上的原有数据第一次修改时的IO压力恶化 rbd snap limit set [--pool ] [--image ] ... 解除限制 rbd snap limit clear [--pool ] [--image ] ``` ```powershell 删除快照 rbd snap rm [--pool ] [--image ] [--snap ] [--no-progress] [--force] 提示： Ceph OSD会以异步方式删除数据，因此删除快照并不能立即释放磁盘空间; ``` ```powershell 清理快照 删除一个image的所有快照，可以使用rbd snap purge命令 rbd snap purge [--pool ] --image [--no-progress] ``` **简单实践** 查看快照命令 ```powershell [cephadm@admin ceph-cluster]$ rbd --help | grep ' snap ' snap create (snap add) Create a snapshot. snap limit clear Remove snapshot limit. snap limit set Limit the number of snapshots. snap list (snap ls) Dump list of image snapshots. snap protect Prevent a snapshot from being deleted. snap purge Delete all unprotected snapshots. snap remove (snap rm) Delete a snapshot. snap rename Rename a snapshot. snap rollback (snap revert) Rollback image to snapshot. snap unprotect Allow a snapshot to be deleted. ``` 客户端准备rbd文件 ```powershell 客户端挂载rbd文件 [root@stor06 ~]# rbd --user kube map kube/vol01 /dev/rbd0 查看效果 [root@stor06 ~]# rbd showmapped id pool namespace image snap device 0 kube vol01 - /dev/rbd0 挂载操作 [root@stor06 ~]# mount /dev/rbd0 /mnt [root@stor06 ~]# ls /mnt/ issue lost+found 结果显示： 虽然容量调整过了，但是就有的文件仍然存在 ``` 管理端创建快照 ```powershell 查看快照 [cephadm@admin ceph-cluster]$ rbd snap list kube/vol01 创建快照 [cephadm@admin ceph-cluster]$ rbd snap create kube/vol01@snap01 查看快照 [cephadm@admin ceph-cluster]$ rbd snap list kube/vol01 SNAPID NAME SIZE PROTECTED TIMESTAMP 4 snap01 3 GiB Mon Sep 26 11:34:20 2062 ``` 恢复快照动作 ```powershell 客户端将文件删除 [root@stor06 ~]# ls /mnt/ issue lost+found [root@stor06 ~]# rm -f /mnt/issue [root@stor06 ~]# ls /mnt/ lost+found 客户端解除磁盘的占用 root@stor06:~# umount /mnt root@stor06:~# rbd unmap /dev/rbd0 root@stor06:~# rbd showmapped ``` ```powershell 服务端恢复快照 [cephadm@admin ceph-cluster]$ rbd snap rollback kube/vol01@snap01 Rolling back to snapshot: 100% complete...done. 客户端重新加载磁盘文件 [root@stor06 ~]# rbd --user kube map kube/vol01 /dev/rbd0 [root@stor06 ~]# mount /dev/rbd0 /mnt [root@stor06 ~]# ls /mnt/ issue lost+found 结果显示： 快照数据恢复过来了 ``` 删除快照 ```powershell 删除之前查看效果 [cephadm@admin ceph-cluster]$ rbd snap list kube/vol01 SNAPID NAME SIZE PROTECTED TIMESTAMP 4 snap01 3 GiB Mon Sep 26 11:34:20 2062 删除快照 [cephadm@admin ceph-cluster]$ rbd snap rm kube/vol01@snap01 Removing snap: 100% complete...done. 确认删除效果 [cephadm@admin ceph-cluster]$ rbd snap list kube/vol01 [cephadm@admin ceph-cluster]$ ``` 快照数量的限制 ```powershell 设定快照数量的限制 [cephadm@admin ceph-cluster]$ rbd snap limit set --pool kube --image vol01 --limit 5 确认效果 [cephadm@admin ceph-cluster]$ rbd snap create kube/vol01@snap01 ... [cephadm@admin ceph-cluster]$ rbd snap create kube/vol01@snap06 rbd: failed to create snapshot: (122) Disk quota exceeded 解除快照限制 [cephadm@admin ceph-cluster]$ rbd snap limit clear --pool kube --image vol01 确认效果 [cephadm@admin ceph-cluster]$ rbd snap create kube/vol01@snap06 Creating snap: 100% complete...done. [cephadm@admin ceph-cluster]$ rbd snap list kube/vol01 SNAPID NAME SIZE PROTECTED TIMESTAMP 6 snap01 3 GiB Mon Sep 26 11:38:25 2022 ... 13 snap06 3 GiB Mon Sep 26 11:39:07 2022 ``` 清理所有快照 ```powershell 清理所有快照 [cephadm@admin ceph-cluster]$ rbd snap purge --pool kube --image vol01 确认效果 [cephadm@admin ceph-cluster]$ rbd snap list kube/vol01 ``` **小结** ``` ``` ### 1.2.6 快照分层 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell Ceph支持在一个块设备快照的基础上创建一到多个COW或COR（Copy-On-Read）类型的克隆，这种中间快照层机制提了一种极速创建image的方式.用户可以创建一个基础image并为其创建一个只读快照层，而后可以在此快照层上创建任意个克隆进行读写操作，甚至能够进行多级克隆 例如: 实践中可以为Qemu虚拟机创建一个image并安装好基础操作系统环境作为模板，对其创建创建快照层后，便可按需创建任意多个克隆作为image提供给多个不同的VM（虚拟机）使用，或者每创建一个克隆后进行按需修改，而后对其再次创建下游的克隆. ``` ```powershell 通过克隆生成的image在其功能上与直接创建的image几乎完全相同，它同样支持读、写、克隆、空间扩缩容等功能，惟一的不同之处是克隆引用了一个只读的上游快照，而且此快照必须要置于“保护”模式之下 ``` ```powershell Ceph的快照支持COW和COR两种类型 COW是为默认的类型，仅在数据首次写入时才需要将它复制到克隆的image中 COR则是在数据首次被读取时复制到当前克隆中，随后的读写操作都将直接基于此克隆中的对象进行 ``` 分层快照使用 ```powershell 在RBD上使用分层克隆的方法非常简单： 创建一个image，对image创建一个快照并将其置入保护模式，而克隆此快照即可 创建克隆的image时， 需要指定引用的存储池、镜像和镜像快照，以及克隆的目标image的存储池和镜像名称， 因此，克隆镜像支持跨存储池进行 ``` ![image-20211210111915782](../../img/kubernetes/kubernetes_storage_ceph/image-20211210111915782.png) ```powershell 简单来说，这里的快照分层，主要说的就是： 基础的模板快照和特有应用快照 ``` 命令实践 ```powershell 保护上游的原始快照 下游image需要引用上游快照中的数据，快照的意外删除必将导致数据服务的中止，因此在克隆操作之外，必须将上游的快照置于保护模式 rbd snap protect [--pool ] --image --snap ``` ```powershell 克隆快照 rbd clone [--pool ] --image --snap --dest-pool ... rbd clone [/]@ [/] ``` ```powershell 列出快照的子项 rbd children [--pool ] --image --snap ``` ```powershell 展平克隆的image 克隆的映像会保留对父快照的引用，删除子克隆对父快照的引用时，可通过将信息从快照复制到克隆，进行image的“展平”操作 展平克隆所需的时间随着映像大小的增加而延长 要删除某拥有克隆子项的快照，必须先平展其子image 命令： rbd flatten [--pool ] --image --no-progress ``` ```powershell 取消快照保护 必须先取消保护快照，然后才能删除它 用户无法删除克隆所引用的快照，需要先平展其每个克隆，然后才能删除快照 命令：rbd snap unprotect [--pool ] --image --snap ``` **简单实践** 清理环境 ```powershell 在客户端将磁盘取消应用 [root@stor06 ~]# umount /mnt [root@stor06 ~]# rbd unmap /dev/rbd0 [root@stor06 ~]# rbd showmapped ``` 服务端定制基础快照 ```powershell 定制基础快照 [cephadm@admin ceph-cluster]$ rbd snap create kube/vol01@clonetpl01 [cephadm@admin ceph-cluster]$ rbd snap ls kube/vol01 SNAPID NAME SIZE PROTECTED TIMESTAMP 20 clonetpl01 3 GiB Mon Sep 26 11:42:51 2062 ``` ```powershell 将快照置于保护模式 [cephadm@admin ceph-cluster]$ rbd snap protect kube/vol01@clonetpl01 [cephadm@admin ceph-cluster]$ rbd snap ls kube/vol01 SNAPID NAME SIZE PROTECTED TIMESTAMP 20 clonetpl01 3 GiB yes Mon Sep 26 11:42:51 2062 结果显示： 该快照已经被处于保护模式了 ``` 基于快照来进行克隆操作 ```powershell 基于基础模板克隆一个镜像 [cephadm@admin ceph-cluster]$ rbd clone kube/vol01@clonetpl01 kube/myimg01 [cephadm@admin ceph-cluster]$ rbd clone kube/vol01@clonetpl01 kube/myimg02 查看效果 [cephadm@admin ceph-cluster]$ rbd ls -p kube myimg01 myimg02 vol01 查看模板镜像的所有子镜像文件 [cephadm@admin ceph-cluster]$ rbd children kube/vol01@clonetpl01 kube/myimg01 kube/myimg02 ``` ```powershell 客户端尝试应用一下clone文件 [root@stor06 ~]# rbd --user kube map kube/myimg01 /dev/rbd0 查看效果 [root@stor06 ~]# rbd showmapped id pool namespace image snap device 0 kube myimg01 - /dev/rbd0 挂载操作 [root@stor06 ~]# mount /dev/rbd0 /mnt [root@stor06 ~]# ls /mnt issue lost+found [root@stor06 ~]# echo myimg1 >> /mnt/issue 结果显示： 该clone镜像文件可以正常的使用 ``` ```powershell 客户端再次应用clone文件 [root@stor06 ~]# rbd --user kube map kube/myimg02 /dev/rbd1 查看效果 [root@stor06 ~]# rbd showmapped id pool namespace image snap device 0 kube myimg01 - /dev/rbd0 1 kube myimg02 - /dev/rbd1 挂载操作 [root@stor06 ~]# mkdir /mnt1 [root@stor06 ~]# mount /dev/rbd1 /mnt1 [root@stor06 ~]# cat /mnt1/issue \S Kernel \r on an \m 结果显示： 多个clone镜像文件可以独立的正常使用 ``` 卸载操作 ```powershell 客户端卸载myimg02 [root@stor06 ~]# umount /mnt1 [root@stor06 ~]# rbd unmap kube/myimg02 管理端删除image镜像 [cephadm@admin ceph-cluster]$ rbd rm kube/myimg02 Removing image: 100% complete...done. [cephadm@admin ceph-cluster]$ rbd children kube/vol01@clonetpl01 kube/myimg01 ``` 展平快照文件 ```powershell 如果我们要删除底层的模板快照文件，为了避免对上层镜像的文件产生不好的影响，我们可以将底层快照的文件转移到上层clone文件中一份，这个动作叫展平快照文件 ``` ```powershell 展平数据 [cephadm@admin ceph-cluster]$ rbd flatten kube/myimg01 Image flatten: 100% complete...done. 取消保护基础快照模板文件 [cephadm@admin ceph-cluster]$ rbd snap unprotect kube/vol01@clonetpl01 移除底层的模板文件 [cephadm@admin ceph-cluster]$ rbd snap rm kube/vol01@clonetpl01 Removing snap: 100% complete...done. 查看效果 [cephadm@admin ceph-cluster]$ rbd snap ls kube/vol01 [cephadm@admin ceph-cluster]$ ``` ```powershell 在客户端上不影响子clone文件的操作 [root@stor06 ~]# ls /mnt [root@stor06 ~]# cat /mnt/issue ``` **小结** ``` ``` ### 1.2.7 RBD实践 学习目标 这一节，我们从 案例需求、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell ceph提供的块存储能够可以为其他虚拟机提供磁盘设备来进行使用，所以接下来我们准备将ceph的磁盘文件提供给kvm进行使用。 ``` ```powershell 我们需要按照如下的步骤来进行相关的实践： 1 kvm环境和ceph环境准备 2 ceph和kvm认证集成 3 kvm到ceph中创建镜像文件 4 启动kvm虚拟机测试ceph磁盘使用效果 ``` 准备系统文件 ```powershell 获取镜像文件 mkdir /data/images/ && cd /data/images/ wget http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img ``` 部署kvm环境 ```powershell 判断CPU是否支持硬件虚拟化 [root@stor05 ~]# egrep '(vmx|svm)' --color=always /proc/cpuinfo | wc -l 2 检查kvm模块是否加载 [root@stor05 ~]# lsmod | grep kvm kvm_intel 188740 0 kvm 637289 1 kvm_intel irqbypass 13503 1 kvm ``` ```powershell 安装软件 [root@stor05 ~]# yum install -y virt-manager libvirt qemu-kvm kvm 注意： libvirt 虚拟机管理 virt 虚拟机安装克隆 qemu-kvm 管理虚拟机磁盘 ``` ```powershell 启动服务 systemctl start libvirtd.service 确认效果 systemctl status libvirtd.service ``` ```powershell 查看虚拟网络 [root@stor05 ~]# virsh net-list 名称 状态 自动开始 持久 ---------------------------------------------------------- default 活动 是 是 查看虚拟主机 [root@stor05 ~]# virsh list Id 名称 状态 ---------------------------------------------------- ``` 部署ceph环境 ```powershell 安装ceph软件 [root@stor05 ~]# yum install ceph ceph-common -y ``` **简单实践** 确认kvm环境支持rdb磁盘 ```powershell 内置的qemu-kvm包已经支持RBD块设备 [root@stor05 ~]# qemu-img --help | grep rbd Supported formats: vvfat vpc vmdk vhdx vdi ssh sheepdog rbd ... [root@stor05 ~]# ldd /usr/libexec/qemu-kvm | grep rbd librbd.so.1 => /lib64/librbd.so.1 (0x00007f9badca9000) 结果显示： qemu-kvm具备访问RBD的能力。 ``` ceph创建专属的pool和秘钥 ```powershell Ceph建立一个Pool， [cephadm@admin ceph-cluster]$ ceph osd pool create superopsmsb 16 16 pool 'superopsmsb' created 启动rbd功能 [cephadm@admin ceph-cluster]$ ceph osd pool application enable superopsmsb rbd enabled application 'rbd' on pool 'superopsmsb' rbd环境初始化 [cephadm@admin ceph-cluster]$ rbd pool init superopsmsb ``` ```powershell 创建superopsmsb的pool专属认证权限 [cephadm@admin ceph-cluster]$ ceph auth get-or-create client.superopsmsb mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=superopsmsb' [client.superopsmsb] key = AQD5JzFjymapKxAASoWCRgp/yQt4dFGRmZDI8w== ``` ```powershell 创建专属用户的秘钥文件 [cephadm@admin ceph-cluster]$ ceph auth get client.superopsmsb -o ceph.client.superopsmsb.keyring exported keyring for client.superopsmsb ``` ```powershell 传递认证文件 [cephadm@admin ceph-cluster]$ sudo scp ceph.client.superopsmsb.keyring root@stor05:/etc/ceph/ [cephadm@admin ceph-cluster]$ sudo scp ceph.client.admin.keyring root@stor05:/etc/ceph/ 传递正常的ceph集群的配置文件 [cephadm@admin ceph-cluster]$ sudo scp ceph.conf root@stor05:/etc/ceph/ ``` ```powershell 确认效果 [root@stor05 ~]# ceph --user superopsmsb -s cluster: id: 1d4e5773-619a-479d-861a-66ba451ce945 ... ``` kvm 集成ceph ```powershell 创建认证文件 ceph-client-superopsmsb-secret.xml client.superopsmsb secret ``` ```powershell 使用virsh创建secret [root@stor05 /data/conf]# virsh secret-define --file ceph-client-superopsmsb-secret.xml 生成 secret caf4336b-d2de-4699-8aee-94569531559b ``` ```powershell 将ceph的client.superopsmsb的密钥导⼊secret中 [root@stor05 /data/conf]# virsh secret-set-value --secret caf4336b-d2de-4699-8aee-94569531559b --base64 $(ceph auth get-key client.superopsmsb) secret 值设定 确认效果 [root@stor05 /data/conf]# virsh secret-list UUID 用量 -------------------------------------------------------------------------------- caf4336b-d2de-4699-8aee-94569531559b ceph client.superopsmsb secret ``` 镜像创建 ```powershell 创建空的image，或者导⼊已经的磁盘镜像内容 方法1：导入镜像 qemu-img convert -f qcow2 -O raw /path/to/image.img rbd:/ 方法2：创建镜像 qemu-img xxx create -f rbd rbd:/ size ``` ```powershell 创建镜像 [root@stor05 /data/conf]# qemu-img create -f rbd rbd:superopsmsb/superopsmsb-image 1G Formatting 'rbd:superopsmsb/superopsmsb-image', fmt=rbd size=10737418240 cluster_size=0 ``` ```powershell 确认镜像文件 [root@stor05 /data/conf]# qemu-img info /data/images/cirros-0.5.2-x86_64-disk.img image: cirros-0.5.2-x86_64-disk.img file format: qcow2 ... 导入镜像文件到rdb [root@stor05 /data/conf]# qemu-img convert -f qcow2 -O raw /data/images/cirros-0.5.2-x86_64-disk.img rbd:superopsmsb/cirros-0.5.2 ``` ```powershell 确认效果 [root@stor05 ~]# rbd --user superopsmsb ls superopsmsb -l NAME SIZE PARENT FMT PROT LOCK cirros-0.5.2 112 MiB 2 superopsmsb-image 10 GiB 2 注意： 因为是远程查看，所以信息展示有可能缓慢，我们可以直接在admin主机上进行快速查看 ``` 虚拟机创建 ```powershell 定制虚拟机配置文件 /data/conf/node1.xml node1 512000 512000 1 hvm /usr/libexec/qemu-kvm ``` ```powershell 创建虚拟机 [root@stor05 /data/conf]# virsh define node1.xml 定义域 node1（从 node1.xml） [root@stor05 /data/conf]# virsh start node1 域 node1 已开始 ``` ```powershell 查看主机状态 [root@stor05 /data/conf]# virsh list --all Id 名称 状态 ---------------------------------------------------- 2 node1 running 查看设备信息 [root@stor05 /data/conf]# virsh domblklist node1 目标 源 ------------------------------------------------ vda superopsmsb/cirros-0.5.2 可以借助于vnc viewer 连接到虚拟机查看效果 注意： 启动很慢，但是运行需要等待3分钟左右 ``` 磁盘挂载 ```powershell 关闭虚拟机 [root@stor05 /data/conf]# virsh destroy node1 域 node1 被删除 ``` ```powershell 编写专有设备文件 device.xml ``` ```powershell 将设备追加到虚拟机中 [root@stor05 /data/conf]# virsh attach-device node1 device.xml --persistent 成功附加设备 确认磁盘挂载效果 [root@stor05 /data/conf]# virsh domblklist node1 目标 源 ------------------------------------------------ vda superopsmsb/cirros-0.5.2 vdc superopsmsb/superopsmsb-image ``` 环境还原 ```powershell 删除虚拟机 [root@stor05 /data/conf]# virsh undefine node1 ``` ```powershell 删除pool [cephadm@admin ceph-cluster]$ ceph osd pool rm superopsmsb superopsmsb --yes-i-really-really-mean-it pool 'superopsmsb' removed [cephadm@admin ceph-cluster]$ ceph osd pool rm kube kube --yes-i-really-really-mean-it pool 'kube' removed ``` **小结** ``` ``` ## 1.3 RGW接口 ### 1.3.1 基础知识 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** OSS简介 ```powershell 对象存储(Object Storage) 是无层次结构的数据存储方法，通常用于云计算环境中。不同于其他数据存储方法，基于对象的存储不使用目录树 数据作为单独的对象进行存储 数据并不放置在目录层次结构中，而是存在于平面地址空间内的同一级别 应用通过唯一地址来识别每个单独的数据对象 每个对象可包含有助于检索的元数据 专为使用API在应用级别（而非用户级别）进行访问而设计 ``` 对象 ```powershell 对象是对象存储系统中数据存储的基本单位，每个Object是数据和数据属性集的综合体，数据属性可以根据应用的需求进行设置，包括数据分布、服务质量等 每个对象自我维护其属性，从而简化了存储系统的管理任务 对象的大小可以不同，甚至可以包含整个数据结构，如文件、数据库表项等 ``` ```powershell 对象存储系统一般是一类智能设备，它具有自己的存储介质、处理器、内存以及网络系统等，负责管理本地的对象，是对象存储系统的核心. ``` ![image-20220926182809531](../../img/kubernetes/kubernetes_storage_ceph/image-20220926182809531.png) 术语解析 ```powershell 一般说来，一个对象存储系统的核心资源类型应该包括用户（User）、存储桶（ bucket）和对象（object） 它们之间的关系是： - User将Object存储到存储系统上的Bucket - 存储桶属于某个用户并可以容纳对象，一个存储桶用于存储多个对象 - 同一个用户可以拥有多个存储桶，不同用户允许使用相同名称的bucket ``` ![image-20211210163113908](../../img/kubernetes/kubernetes_storage_ceph/image-20211210163113908.png) 常见方案 ```powershell 各种存储方案，虽然在设计与实现上有所区别，但大多数对象存储系统对外呈现的核心资源类型大同小异 ``` | 方案 | 描述 | | --------------- | ------------------------------------------------------------ | | Amazon S3 | 提供了user、bucket和object分别表示用户、存储桶和对象，其中bucket隶属于user，因此user名称即可做为bucket的名称空间，不同用户允许使用相同名称的bucket | | OpenStack Swift | 提供了user、container和object分别对应于用户、存储桶和对象，不过它还额外为user提供了父级组件account，用于表示一个项目或租户，因此一个account中可包含一到多个user，它们可共享使用同一组container，并为container提供名称空间 | | RadosGW | 提供了user、subuser、bucket和object，其中的user对应于S3的user，而subuser则对应于Swift的user，不过user和subuser都不支持为bucket提供名称空间，因此 ，不同用户的存储桶也不允许同名；不过，自Jewel版本起，RadosGW引入了tenant（租户）用于为user和bucket提供名称空间，但它是个可选组件
Jewel版本之前，radosgw的所有user位于同一名称空间，它要求所有user的ID必须惟一，并且即便是不同user的bucket也不允许使用相同的bucket ID | **简单实践** 简介 ```powershell Ceph对象存储使用Ceph对象网关守护进程（radosgw），它是用于与Ceph存储群集进行交互的HTTP服务器。由于它提供与OpenStack Swift和Amazon S3兼容的接口，因此Ceph对象网关具有自己的用户管理。Ceph对象网关可以将数据存储在用于存储来自Ceph文件系统客户端或Ceph块设备客户端的数据的同一Ceph存储群集中。S3和Swift API共享一个公共的名称空间，因此可以使用一个API编写数据，而使用另一个API检索数据。 ``` ```powershell Ceph RGW基于librados，是为应用提供RESTful类型的对象存储接口。RGW提供两种类型的接口： 　　1) S3：兼容Amazon S3RESTful API； 　　2) Swift：兼容OpenStack Swift API。 　　 同时，RGW为了实现RESTful接口的功能，默认使用Civetweb作为其Web Sevice，而Civetweb默认使用端口7480提供服务，如果想修改端口（如80端口），就需要修改Ceph的配置文件。 ``` 要点 ```powershell RGW在创建的时候，自动会初始化自己的存储池，而且RGW还需要自己独有的守护进程服务才可以正常的使用 RGW并非必须的接口，仅在需要用到与S3和Swift兼容的RESTful接口时才需要部署RGW实例 ``` ![image-20220926220715586](../../img/kubernetes/kubernetes_storage_ceph/image-20220926220715586.png) RGW内部逻辑处理层级结构图 ![image-20220928111659491](../../img/kubernetes/kubernetes_storage_ceph/image-20220928111659491.png) ```powershell HTTP 前端: 接收数据操作请求。 REST API接口： 从http请求中解析出 S3 或 Swift 数据并进行一系列检查。 API操作逻辑: 经Restful 接口检查通过后，根据内部业务逻辑交由不同业务模块进行数据处理。 RADOS接口： 如果需要写入或者操作ceph集群数据，则经由RADOS + librados 调用将数据发送给集群的后端主机 ``` 常见属性 ```powershell 自0.80版本起，Ceph放弃了基于apache和fastcgi提供radosgw服务的传统而代之以默认嵌入在ceph-radosgw进程中的Citeweb，这种新的实现方式更加轻便和简洁，但直到Ceph 11.0.1版本，Citeweb才开始支持SSL协议. Citeweb默认监听于TCP协议的7480端口提供http服务，修改配置需要编辑ceph.conf配置文件，以如下格式进行定义 [client.rgw.] rgw_host = rgw_frontends = "civetweb port=80" 配置完成后需要重启ceph-radosgw进程以生效新配置 ceph-radosgw@rgw. ``` ```powershell 其他相关的配置 配置https 额外添加参数ssl_certificate=/PATH/TO/PEM_FILE 定义port=443s，或者port=80+443s 其它配置参数 num_threads：Citeweb以线程模型处理客户端请求，它为每个连接请求分配一个专用线 程，因而此参数定义了其支持的最大并发连接数，默认值为50 request_timeout_ms：网络发送与接收操作的超时时长，以ms为单位，默认值为30000 ；可以在必要时通过增大此值实现长连接的效果 access_log_file：访问日志的文件路径，默认为空 error_log_file：错误日志的文件路径，默认为空 rgw_dns_name： 定制专属的dns服务解析域名 ``` **小结** ``` ``` ### 1.3.2 基础操作 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 部署RGW主机 ```powershell 到stor04主机上的部署ceph环境 [root@stor04 ~]# yum install -y ceph-radosgw ``` ```powershell admin节点上，创建rgw的主机环境 [cephadm@admin ceph-cluster]$ ceph-deploy rgw create stor04 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf ... [ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host stor04 and default port 7480 ``` 查看效果 ```powershell 查看集群的状态 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 3h) mgr: mon01(active, since 30h), standbys: mon02 osd: 6 osds: 6 up (since 26h), 6 in (since 26h) rgw: 1 daemon active (stor04) ... ``` ```powershell 查看服务状况 [root@stor04 ~]# netstat -tnulp | grep radosgw tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 4295/radosgw tcp6 0 0 :::7480 :::* LISTEN 4295/radosgw 查看服务进程 [root@stor04 ~]# ps aux | grep -v grep | grep radosgw ceph 4295 0.5 2.1 5145976 39612 ? Ssl 18:37 0:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.stor04 --setuser ceph --setgroup ceph ``` ```powershell RGW会在rados集群上生成包括如下存储池的一系列存储池 [cephadm@admin ceph-cluster]$ ceph osd pool ls .rgw.root default.rgw.control default.rgw.meta default.rgw.log 结果显示： 默认情况下，rgw自动创建4个存储池 ``` RGW提供了一个简单的web接口，用于数据的交流 ```powershell web访问接口 RGW提供的是REST接口，客户端通过http与其进行交互，完成数据的增删改查等管理操作。浏览器查看 10.0.0.16:7480 的web简单应用效果 ``` ![image-20211206115942508](../../img/kubernetes/kubernetes_storage_ceph/image-20211206115942508.png) **简单实践** 更改端口实践 ```powershell 默认情况下，RGW实例监听于TCP协议的7480端口7480，需要定制的话，可以通过在运行RGW的节点上编辑其主配置文件ceph.conf进行修改，相关参数如下所示： [root@stor04 ~]# cat /etc/ceph/ceph.conf ... [client.rgw.stor04] rgw_frontends = "civetweb port=8080" ``` ```powershell 重启服务 [root@stor04 ~]# systemctl restart ceph-radosgw@rgw.stor04 [root@stor04 ~]# systemctl status ceph-radosgw@rgw.stor04 查看效果 [root@stor04 ~]# netstat -tnulp | grep 8080 tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 152636/radosgw ``` ![image-20211210173544875](../../img/kubernetes/kubernetes_storage_ceph/image-20211210173544875.png) ssl通信 ``` 需要提供pem证书文件，需要将 证书和私钥文件放在同一个文件里面 ``` ```powershell 生成秘钥 [root@stor04 ~]# mkdir /etc/ceph/ssl && cd /etc/ceph/ssl [root@stor04 /etc/ceph/ssl]# openssl genrsa -out civetweb.key 2048 生成证书 [root@stor04 /etc/ceph/ssl]# openssl req -new -x509 -key civetweb.key -out civetweb.crt -days 3650 -subj "/CN=stor04.superopsmsb.com" [root@stor04 /etc/ceph/ssl]# ls civetweb.crt civetweb.key 合并证书信息 root@stor04:/etc/ceph/ssl# cat civetweb.key civetweb.crt > civetweb.pem ``` 修改认证信息 ```powershell 在运行RGW的节点上编辑其主配置文件ceph.conf进行修改，相关参数如下所示： [root@stor04 ~]# cat /etc/ceph/ceph.conf ... [client.rgw.stor04] rgw_frontends = "civetweb port=8443s ssl_certificate=/etc/ceph/ssl/civetweb.pem" 注意： 这里的8443s，最后的s代表必须使用ssl通信方式 ``` ```powershell 重启服务 [root@stor04 ~]# systemctl restart ceph-radosgw@rgw.stor04 [root@stor04 ~]# systemctl status ceph-radosgw@rgw.stor04 查看效果 [root@stor04 ~]# netstat -tnulp | grep 8443 tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 153319/radosgw ``` ![image-20211210174100106](../../img/kubernetes/kubernetes_storage_ceph/image-20211210174100106.png) 同时开启 https和http方式来进行访问 ```powershell 在运行RGW的节点上编辑其主配置文件ceph.conf进行修改，相关参数如下所示： [root@stor04 ~]# cat ceph.conf ... [client.rgw.stor04] rgw_frontends = "civetweb port=7480+8443s ssl_certificate=/etc/ceph/ssl/civetweb.pem" 注意： 通过 port+port 的方式实现多端口开启服务 ``` ```powershell 重启服务 [root@stor04 ~]# systemctl restart ceph-radosgw@rgw.stor04 [root@stor04 ~]# systemctl status ceph-radosgw@rgw.stor04 查看效果 [root@stor04 ~]# netstat -tnulp | grep radosgw tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 153941/radosgw tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 153941/radosgw ``` ![image-20211210174340551](../../img/kubernetes/kubernetes_storage_ceph/image-20211210174340551.png) 高并发配置 ``` 对于ceph来说，它内部的高并发主要是通过线程方式定义的，而且默认值是50，如果我们要调整并发数量的话，可以调整num_threads的属性值 配置示例： [client.rgw.stor04] rgw_frontends = "civetweb port=7480+8443s ssl_certificate=/etc/ceph/ssl/civetweb.pem num_threads=2000" ``` **小结** ``` ``` ### 1.3.3 泛域名实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell S3的存储桶是用于存储对象的容器，每个对象都必须储存在一个特定的存储桶中，且每个对象都要直接通过RESTful API基于URL进行访问，URL格式为 “http(s)://bucket-name.radowgw-host[:port]/key” ``` ```powershell 例如，对于存储在rgw01.superopsmsb.io上的S3 API对象存储系统上eshop存储桶中的名为images/1.jpg 的对象，可通过 http://eshop.rgw01.superopsmsb.io/images/1.jpg对其进行寻址 ``` ```powershell 因此，radosgw的S3 API接口的功能强依赖于DNS的泛域名解析服务，它必须能够正常解析任何“.”格式的名称至radosgw主机。 除此之外，我们还需要配置每个radowgw守护进程的rgw_dns_name为其DNS名称 ``` dns环境配置 ```powershell 安装软件 [root@admin ~]# yum -y install bind bind-chroot bind-utils bind-libs 备注： bind 提供了域名服务的主要程序及相关文件 bind-utils 提供了对DNS服务器的测试工具程序（如nslookup、dig等） bind-chroot 为bind提供一个伪装的根目录以增强安全性 启动服务 [root@admin ~]# systemctl start named [root@admin ~]# systemctl status named ``` ```powershell 查看版本信息 [root@admin ~]# named -v BIND 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.9 (Extended Support Version) 查看配置文件 [root@admin ~]# rpm -ql bind | grep named.conf /etc/named.conf /usr/lib/tmpfiles.d/named.conf /usr/share/doc/bind-9.11.4/man.named.conf.html /usr/share/doc/bind-9.11.4/named.conf.default /usr/share/doc/bind-9.11.4/sample/etc/named.conf /usr/share/man/man5/named.conf.5.gz 查看暴露端口 [root@admin ~]# netstat -tnulp | grep 53 tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 17563/named tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 17563/named tcp6 0 0 ::1:53 :::* LISTEN 17563/named tcp6 0 0 ::1:953 :::* LISTEN 17563/named udp 0 0 127.0.0.1:53 0.0.0.0:* 17563/named udp6 0 0 ::1:53 :::* 17563/named ``` **简单实践** 定制zone的配置 ```powershell 定制正向解析zone的配置 [root@admin ~]# tail /etc/named.conf -n 9 options { listen-on port 53 { any; }; // 监听任何ip对53端口的请求 ... allow-query { any; }; // 接收任何来源查询dns记录 ... }; ... // 定制网站主域名的zone配置 zone "superopsmsb.com" IN { type master; file "superopsmsb.com.zone"; }; // 定制泛域名网站的zone配置 zone "stor04.superopsmsb.com" IN { type master; file "stor04.superopsmsb.com.zone"; }; ... include "/etc/named.rfc1912.zones"; include "/etc/named.root.key"; ``` ```powershell 定制主域名的zone文件 [root@admin ~]# cat /var/named/superopsmsb.com.zone ; ; BIND reverse data file for local loopback interface ; $TTL 604800 @ IN SOA ns.superopsmsb.com. admin.superopsmsb.com. ( 1 ; Serial 604800 ; Refresh 86400 ; Retry 2419200 ; Expire 604800 ) ; Negative Cache TTL ; IN NS ns ns IN A 10.0.0.12 stor02 IN A 10.0.0.14 stor03 IN A 10.0.0.15 stor04 IN NS ns ``` ```powershell 定制泛域名的zone文件 [root@admin ~]# cat /var/named/stor04.superopsmsb.com.zone ; ; BIND reverse data file for local loopback interface ; $TTL 604800 @ IN SOA ns.superopsmsb.com. admin.superopsmsb.com. ( 1 ; Serial 604800 ; Refresh 86400 ; Retry 2419200 ; Expire 604800 ) ; Negative Cache TTL ; IN NS ns ns IN A 10.0.0.12 * IN A 10.0.0.16 ``` ```powershell 检查配置文件 [root@admin ~]# named-checkconf 重启dns服务 [root@admin ~]# systemctl restart named [root@admin ~]# systemctl status named 定制本地主机专属的dns域名服务器 [root@admin ~]# cat /etc/resolv.conf # Generated by NetworkManager search localhost nameserver 10.0.0.12 ``` ``` 确认dns解析效果 [root@admin ~]# host -a stor02.superopsmsb.com [root@admin ~]# host -a stor02.superopsmsb.com 10.0.0.12 [root@admin ~]# host -a stor02.superopsmsb.com 10.0.0.12 Trying "stor02.superopsmsb.com" Using domain server: Name: 10.0.0.12 Address: 10.0.0.12#53 Aliases: ... ;; QUESTION SECTION: ;stor02.superopsmsb.com. IN ANY ;; ANSWER SECTION: stor02.superopsmsb.com. 604800 IN A 10.0.0.14 ;; AUTHORITY SECTION: superopsmsb.com. 604800 IN NS ns.superopsmsb.com. ;; ADDITIONAL SECTION: ns.superopsmsb.com. 604800 IN A 10.0.0.12 Received 89 bytes from 10.0.0.12#53 in 0 ms ``` ```powershell [root@admin ~]# nslookup stor02.superopsmsb.com [root@admin ~]# nslookup stor02.superopsmsb.com 10.0.0.12 Server: 10.0.0.12 Address: 10.0.0.12#53 Name: stor02.superopsmsb.com Address: 10.0.0.14 ``` ```powershell 确认dns解析效果 [root@admin ~]# dig file.stor04.superopsmsb.com [root@admin ~]# dig img.stor04.superopsmsb.com @10.0.0.12 ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.9 <<>> img.stor04.superopsmsb.com @10.0.0.12 ... ;; ANSWER SECTION: img.stor04.superopsmsb.com. 604800 IN A 10.0.0.16 ;; AUTHORITY SECTION: stor04.superopsmsb.com. 604800 IN NS ns.stor04.superopsmsb.com. ;; ADDITIONAL SECTION: ns.stor04.superopsmsb.com. 604800 IN A 10.0.0.12 ;; Query time: 0 msec ;; SERVER: 10.0.0.12#53(10.0.0.12) ;; WHEN: 一 9月 26 19:46:57 CST 2022 ;; MSG SIZE rcvd: 104 ``` 配置rgw存储节点的dns相关配置 ```powershell 定制stor04的dns域名配置 [root@stor04 ~]# tail -n4 /etc/ceph/ceph.conf [client.rgw.stor04] rgw_host = stor04 rgw_frontends = "civetweb port=7480+8443s ssl_certificate=/etc/ceph/ssl/civetweb.pem" rgw_dns_name = stor04.superopsmsb.com 属性解析： 通过 rgw_dns_name 属性制定当前节点的域名 重启当前节点的服务效果 [root@stor04 ~]# systemctl restart ceph-radosgw@rgw.stor04 [root@stor04 ~]# systemctl status ceph-radosgw@rgw.stor04 ``` **小结** ``` ``` ### 1.3.4 S3测试 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** S3 API接口简介 ```powershell S3服务的REST API使用用户账号（user）、存储桶（bucket）和对象（object）三个组件来组织存储的数据对象，对象保存于存储桶中，而存储桶则支持授权给特定账号进行读写及创建/删除等操作 ``` ```powershell radosgw-admin是用于管理radowgw服务的命令行接口，它有着众多的分别用于不同管理功能的命令，例如user、subuser、key、bucket和object等 命令格式： radosgw-admin user create --uid="s3user" --display-name="S3 Testing User" ``` ```powershell 而后即可通过其API接口进行访问测试，或者使用s3cmd命令进行 - 使用s3cmd命令之前需要事先配置其工作环境，包括指定Access Key和Secret Key，以及S3服务的访问端点和默认的Region（Ceph的新版本中称作zonegroup）等 命令格式： s3cmd --configure - 配置的结果将保存于~/.s3cmd.cfg配置文件中，用户随后可通过编辑此文件修改配置参数，或者再次运行此配置命令为其指定新的配置信息 s3cmd有众多的子命令 ``` ```powershell 参考资料： https://docs.ceph.com/en/latest/radosgw/s3/python/#using-s3-api-extensions ``` **简单实践** S3 API接口专用账号 ```powershell 创建专属账号 [cephadm@admin ceph-cluster]$ radosgw-admin user create --uid='s3user' --display-name='S3 Testing user' { "user_id": "s3user", "display_name": "S3 Testing user", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "s3user", "access_key": "BX2IDCO5ADZ8IWZ4AVX7", "secret_key": "wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf" } ], ... } ``` ```powershell 查看认证用户 [cephadm@admin ceph-cluster]$ radosgw-admin user list [ "s3user" ] 查看详情 [cephadm@admin ceph-cluster]$ radosgw-admin user info --uid s3user { "user_id": "s3user", ... "keys": [ { "user": "s3user", "access_key": "BX2IDCO5ADZ8IWZ4AVX7", "secret_key": "wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf" } ], ... } ``` 客户端配置 ```powershell 在客户端上安装测试命令 s3cmd [root@stor04 ~]# yum install s3cmd -y ``` ```powershell 使用s3之前，需要做一些预设的配置 [root@stor04 ~]# s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. # 下面的两个属性是来自于之前的账号信息 Access Key: BX2IDCO5ADZ8IWZ4AVX7 Secret Key: wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf Default Region [US]: # 此处可以使用默认的 Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. # 此处的域名是否用，我们自己定义的内容 S3 Endpoint [s3.amazonaws.com]: stor04.superopsmsb.com:7480 Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. # 关于存储桶的定制，需要使用自己的域名，但是格式一样 DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.stor04.superopsmsb.com:7480 Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: # 这里不需要密码 Path to GPG program [/usr/bin/gpg]: # 使用默认的gpg命令路径 When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: No # 不使用HTTPs协议 On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: # 不使用代理，因为是本地访问 New settings: Access Key: NOF182FGP8Q38CWLZ8WQ Secret Key: zP2ZWJPXTsZWQXg2hKY4Wv4OpoolKZEOwfheJlx3 Default Region: US S3 Endpoint: stor04.superopsmsb.com:7480 DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.stor04.superopsmsb.com:7480 Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y # 确认最后的信息 Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Not configured. Never mind. Save settings? [y/N] y # 保存信息 Configuration saved to '/root/.s3cfg' ``` ```powershell 查看常见的帮助信息 [root@stor04 ~]# s3cmd Usage: s3cmd [options] COMMAND [parameters] ... Commands: Make bucket s3cmd mb s3://BUCKET Remove bucket s3cmd rb s3://BUCKET List objects or buckets s3cmd ls [s3://BUCKET[/PREFIX]] List all object in all buckets s3cmd la Put file into bucket s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX] Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE Delete file from bucket s3cmd del s3://BUCKET/OBJECT Delete file from bucket (alias for del) s3cmd rm s3://BUCKET/OBJECT Restore file from Glacier storage s3cmd restore s3://BUCKET/OBJECT ... ``` 域名解析 ```powershell 定制专属的dns域名，让客户端找到 stor04.superopsmsb.com 主机名。 [root@stor04 ~]# cat /etc/resolv.conf # Generated by NetworkManager nameserver 10.0.0.12 ... [root@stor04 ~]# nslookup file.stor04.superopsmsb.com 10.0.0.12 Server: 10.0.0.12 Address: 10.0.0.12#53 Name: file.stor04.superopsmsb.com Address: 10.0.0.16 ``` 存储桶创建实践 ```powershell 创建存储桶 [root@stor04 ~]# s3cmd mb s3://images Bucket 's3://images/' created [root@stor04 ~]# s3cmd mb s3://dir Bucket 's3://dir/' created 查看存储桶 [root@stor04 ~]# s3cmd ls 2062-09-26 14:21 s3://dir 2062-09-26 13:57 s3://images ``` 文件上传下载实践 ```powershell 上传文件到存储桶中，存储同目录可以随意玩耍 [root@stor04 ~]# s3cmd put /etc/issue s3://images/linux/issue upload: '/etc/issue' -> 's3://images/linux/issue' [1 of 1] 23 of 23 100% in 1s 12.79 B/s done 查看文件对象效果 [root@stor04 ~]# s3cmd ls s3://images/linux DIR s3://images/linux/ [root@stor04 ~]# s3cmd ls s3://images/linux/issue 2062-09-26 13:58 23 s3://images/linux/issue ``` ```powershell 下载文件对象 [root@stor04 ~]# s3cmd get s3://images/linux/issue download: 's3://images/linux/issue' -> './issue' [1 of 1] 23 of 23 100% in 0s 3.42 KB/s done [root@stor04 ~]# s3cmd get s3://images/linux/issue /tmp/issue-bak download: 's3://images/linux/issue' -> '/tmp/issue-bak' [1 of 1] 23 of 23 100% in 0s 3.38 KB/s done 查看效果 [root@stor04 ~]# ls anaconda-ks.cfg issue [root@stor04 ~]# ls /tmp/issue-bak /tmp/issue-bak ``` ```powershell 对于一个大的RGW Object，会被切割成多个独立的RGW Object上传，称为multipart。multipar的优势是断点续传。s3接口默认切割大小为15MB。 [root@stor04 /data/scripts]# dd if=/dev/zero of=a.txt bs=1M count=60 记录了60+0 的读入 记录了60+0 的写出 62914560字节(63 MB)已复制，0.0495739 秒，1.3 GB/秒 [root@stor04 /data/scripts]# ll a.txt -rw-r--r-- 1 root root 62914560 9月 27 11:06 a.txt [root@stor04 /data/scripts]# s3cmd put a.txt s3://images/linux/ upload: 'a.txt' -> 's3://images/linux/a.txt' [part 1 of 4, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 25.99 MB/s done upload: 'a.txt' -> 's3://images/linux/a.txt' [part 2 of 4, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 27.90 MB/s done upload: 'a.txt' -> 's3://images/linux/a.txt' [part 3 of 4, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 28.93 MB/s done upload: 'a.txt' -> 's3://images/linux/a.txt' [part 4 of 4, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 31.85 MB/s done ``` 目录上传下载实践 ```powershell 准备目录结构 [root@stor04 ~]# mkdir /data/test/{scripts,conf,file} -p [root@stor04 ~]# touch /data/test/{scripts/{1..4}.sh,conf/{1..3}.conf,file/{1..5}.txt} [root@stor04 ~]# echo scripts-1 > /data/test/scripts/1.sh [root@stor04 ~]# echo conf-2 > /data/test/conf/2.conf [root@stor04 ~]# echo file-5 > /data/test/file/5.txt 上传目录 [root@stor04 ~]# s3cmd put /data/test/ s3://dir/etc/ --recursive upload: '/data/test/conf/1.conf' -> 's3://dir/etc/conf/1.conf' [1 of 12] 0 of 0 0% in 0s 0.00 B/s done ... upload: '/data/test/scripts/4.sh' -> 's3://dir/etc/scripts/4.sh' [12 of 12] 0 of 0 0% in 0s 0.00 B/s done 查看效果 [root@stor04 ~]# s3cmd ls s3://dir DIR s3://dir/etc/ [root@stor04 ~]# s3cmd ls s3://dir/etc # 查看目录的时候，必须在最后添加/ DIR s3://dir/etc/ [root@stor04 ~]# s3cmd ls s3://dir/etc/ DIR s3://dir/etc/conf/ DIR s3://dir/etc/file/ DIR s3://dir/etc/scripts/ [root@stor04 ~]# s3cmd ls s3://dir/etc/conf/ 2062-09-26 14:23 0 s3://dir/etc/conf/1.conf 2062-09-26 14:23 7 s3://dir/etc/conf/2.conf 2062-09-26 14:23 0 s3://dir/etc/conf/3.conf ``` ```powershell 下载目录对象 [root@stor04 ~]# s3cmd get s3://dir/ --recursive download: 's3://dir/etc/conf/1.conf' -> './etc/conf/1.conf' [1 of 12] 0 of 0 0% in 0s 0.00 B/s done ... 注意： 如果按照上传目录结构方式下载文件的时候，不要加子目录， 下载子目录 [root@stor04 ~]# s3cmd get s3://dir/etc/ --recursive download: 's3://dir/etc/conf/1.conf' -> './conf/1.conf' [1 of 12] 0 of 0 0% in 0s 0.00 B/s done ... 结果显示： 它是将子目录下载到当前目录了 ``` 文件删除 ```powershell 删除文件文件对象 [root@stor04 ~]# s3cmd del s3://images/linux/issue delete: 's3://images/linux/issue' [root@stor04 ~]# s3cmd ls s3://images/linux/issue [root@stor04 ~]# 删除目录对象 [root@stor04 ~]# s3cmd del s3://images/linux/issue delete: 's3://images/linux/issue' [root@stor04 ~]# s3cmd del s3://dir/etc/ --recursive delete: 's3://dir/etc/conf/1.conf' ... delete: 's3://dir/etc/scripts/4.sh' 删除存储桶 [root@stor04 ~]# s3cmd rb s3://images Bucket 's3://images/' removed [root@stor04 ~]# s3cmd rb s3://dir Bucket 's3://dir/' removed ``` python测试 ```powershell 安装测试模块 [root@stor04 ~]# yum -y install python-boto ``` ```powershell 定制测试脚本 [root@stor04 ~]# cat /data/scripts/tests3.py import boto.s3.connection access_key = 'BX2IDCO5ADZ8IWZ4AVX7' secret_key = 'wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf' conn = boto.connect_s3( aws_access_key_id=access_key, aws_secret_access_key=secret_key, host='10.0.0.16', port=7480, is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(), ) bucket = conn.create_bucket('ceph-s3-bucket') for bucket in conn.get_all_buckets(): print "{name} {created}".format( name=bucket.name, created=bucket.creation_date, ) ``` ```powershell 测试效果 [root@stor04 ~]# python /data/scripts/tests3.py ceph-s3-bucket 2062-09-26T14:41:42.505Z ``` **小结** ``` ``` ### 1.3.5 Swift测试 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** Swift API接口简介 ```powershell Swift的用户账号对应于radosgw中的subuser（子用户），它隶属于某个事先存在的user（用户账号） radosgw-admin user create --uid="swiftuser" --display-name="Swift Testing User" radosgw-admin subuser create --uid=swiftuser --subuser=swiftuser:swift --access=full Swift API的上下文中，存储桶以container表示，而非S3中的bucket，但二者在功用上类同，都是对象数据的容器 ``` ```powershell Python Swiftclient是一个用于与Swift API交互的Python客户端程序，它包含了Python API（swift 模块）和一个命令行工具swift. 环境安装如下 pip instal --upgrade python-swiftclient swift命令可以通过Swift API完成容器和对象数据的管理操作，其基础语法格式为“ swift [-A Auth URL] [-U username] [-K password] subcommand ``` **简单实践** 用户创建 ```powershell swift的实践用户，我们直接使用上面s3创建的用户来创建 [cephadm@admin ceph-cluster]$ radosgw-admin user list [ "s3user" ] ``` ```powershell 基于s3user用户创建swift拥有所有权限的用户 [cephadm@admin ceph-cluster]$ radosgw-admin subuser create --uid s3user --subuser=s3user:swift --access=full { "user_id": "s3user", ... "subusers": [ { "id": "s3user:swift", "permissions": "full-control" } ], "keys": [ { "user": "s3user", "access_key": "BX2IDCO5ADZ8IWZ4AVX7", "secret_key": "wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf" } ], "swift_keys": [ { "user": "s3user:swift", "secret_key": "pZ5DGT3YDBqdAtb10aFiNGeO2AtJKJN4rLKuI4wU" } ], ... } ``` ```powershell 生成 s3user:swift 对应的secret_key [cephadm@admin ceph-cluster]$ radosgw-admin key create --subuser=s3user:swift --key-type=swift --gen-secret { "user_id": "s3user", "display_name": "S3 Testing user", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [ { "id": "s3user:swift", "permissions": "full-control" } ], "keys": [ { "user": "s3user", "access_key": "BX2IDCO5ADZ8IWZ4AVX7", "secret_key": "wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf" } ], "swift_keys": [ { "user": "s3user:swift", "secret_key": "O7On31ApRWE6aZkSPby9bTp4y19srGTfDKl7IA04" } ], ... } ``` 客户端环境 ```powershell 安装swift命令 [root@stor04 ~]# yum -y install python-setuptools python-pip [root@stor04 ~]# pip install python-swiftclient==3.4.0 注意： 如果需要安装最新版本的话，需要提前将Python版本提升到3.6+ https://pypi.org/project/python-swiftclient/ ``` ```powershell 命令解析： swift -A 认证URL/auth -U 用户名称:swift -K secret_key 命令 命令测试效果 [root@stor04 ~]# swift -A http://10.0.0.16:7480/auth -U s3user:swift -K O7On31ApRWE6aZkSPby9bTp4y19srGTfDKl7IA04 list ceph-s3-bucket ``` ```powershell 配置环境变量 cat > ~/.swift << EOF export ST_AUTH=http://10.0.0.16:7480/auth export ST_USER=s3user:swift export ST_KEY=O7On31ApRWE6aZkSPby9bTp4y19srGTfDKl7IA04 EOF ``` ```powershell 加载环境变量 [root@stor04 ~]# source ~/.swift 确认效果 [root@stor04 ~]# swift list ceph-s3-bucket ``` ```powershell 查看状态信息 [root@stor04 ~]# swift stat Account: v1 Containers: 2 Objects: 1 Bytes: 23 Objects in policy "default-placement-bytes": 0 Bytes in policy "default-placement-bytes": 0 Containers in policy "default-placement": 2 Objects in policy "default-placement": 1 Bytes in policy "default-placement": 23 X-Openstack-Request-Id: tx00000000000000000012c-0063324b45-63ad-default X-Account-Bytes-Used-Actual: 4096 X-Trans-Id: tx00000000000000000012c-0063324b45-63ad-default X-Timestamp: 1664240453.17581 Content-Type: text/plain; charset=utf-8 Accept-Ranges: bytes ``` 存储桶增删测试 ```powershell 创建存储桶 [root@stor04 ~]# swift post swift-super 删除存储桶 [root@stor04 ~]# swift delete ceph-s3-bucket ceph-s3-bucket 查看存储桶 [root@stor04 ~]# swift list swift-super ``` 文件上传操作 ```powershell 上传文件 [root@stor04 ~]# swift upload swift-super /etc/passwd etc/passwd 查看效果 [root@stor04 ~]# swift list swift-super etc/passwd ``` 目录上传操作 ```powershell 上传目录操作 [root@stor04 ~]# swift upload swift-super /data/test data/test/scripts/4.sh ... data/test/file/3.txt 查看效果 [root@stor04 ~]# swift list swift-super data/test/conf/1.conf ... etc/passwd ``` 文件下载操作 ```powershell 下载文件 [root@stor04 ~]# swift download swift-super etc/passwd etc/passwd [auth 0.014s, headers 0.023s, total 0.024s, 0.120 MB/s] [root@stor04 ~]# ls etc/passwd etc/passwd 下载所有 [root@stor04 ~]# swift download --all swift-super/data/test/file/4.txt [auth 0.008s, headers 0.019s, total 0.019s, 0.000 MB/s] ... 注意： 它对于目录的递归下载不是太友好 ``` 文件删除操作 ```powershell 删除文件 [root@stor04 ~]# swift delete swift-super etc/passwd etc/passwd 注意： 它不支持目录删除，但是可以通过批量删除来实现 批量删除 [root@stor04 ~]# swift delete swift-super --prefix=data data/test/file/4.txt ... data/test/scripts/1.sh 批量删除 [root@stor04 ~]# swift delete swift-super --all ``` **小结** ``` ``` ### 1.3.6 对象访问 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 作为OSS对象存储，我们可以在互联网上以http(s):// 的方式来访问对象文件，基本格式如下： 对于存储在rgw01.superopsmsb.com上的S3 API对象存储系统上bucket存储桶中的名为images/1.jpg 的对象，可通过 http://bucket.rgw01.superopsmsb.com/images/1.jpg对其进行访问。 ``` ```powershell 默认情况下，这种情况是拒绝的 [root@stor04 ~]# s3cmd ls s3://images/linux/issue 2062-09-27 00:49 23 s3://images/linux/issue [root@stor04 ~]# curl http://images.stor04.superopsmsb.com:7480/linux/issue AccessDeniedimagestx00000000000000000012f-0063324d15-63ad-default63ad-default-default 结果显示： 这里拒绝的原因就是 AccessDenied，是需要我们通过专用的访问策略方式来进行功能实现。 ``` 解析 ```powershell 在使用http方式访问对象存储的时候，我们需要注意的是： 1 资源对象的访问方式 http 还是 https，依赖于rgw的基本配置 2 资源对象的访问控制 通过定制策略的方式来实现 3 资源对象的跨域问题 通过定制cors的方式来实现 4 资源对象在浏览器端的缓存机制 rgw的基本配置定制 ``` ```powershell 参考资料： https://docs.ceph.com/en/latest/radosgw/bucketpolicy/ https://docs.ceph.com/en/latest/radosgw/s3/python/ ``` 命令解析 ```powershell [root@stor04 /data/conf]# s3cmd --help Usage: s3cmd [options] COMMAND [parameters] ... Modify Bucket Policy s3cmd setpolicy FILE s3://BUCKET Delete Bucket Policy s3cmd delpolicy s3://BUCKET Modify Bucket CORS s3cmd setcors FILE s3://BUCKET Delete Bucket CORS s3cmd delcors s3://BUCKET ... Upload a lifecycle policy for the bucket s3cmd setlifecycle FILE s3://BUCKET Get a lifecycle policy for the bucket s3cmd getlifecycle s3://BUCKET Remove a lifecycle policy for the bucket s3cmd dellifecycle s3://BUCKET ... ``` 查看默认策略 ```powershell [root@stor04 /data/conf]# s3cmd info s3://swift-super s3://swift-super/ (bucket): Location: default Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: S3 Testing user: FULL_CONTROL ``` **简单实践** 定制策略 ```powershell 定制访问策略文件 cat /data/conf/policy.json { "Statement": [{ "Effect": "Allow", "Principal": "*", "Action": ["s3:GetObject"], "Resource": "*" }] } 属性解析： Resource参数匹配某一种文件：Resource:["*.jpg","*.png"] Action可以设置较多参数， 参考资料： https://docs.ceph.com/docs/master/radosgw/bucketpolicy/ ``` ```powershell 应用访问策略 [root@stor04 /data/conf]# s3cmd setpolicy policy.json s3://images --acl-public s3://images/: Policy updated 确认效果 [root@stor04 /data/conf]# s3cmd info s3://images s3://images/ (bucket): Location: default Payer: BucketOwner Expiration Rule: none Policy: { "Statement": [{ "Effect": "Allow", "Principal": "*", "Action": ["s3:GetObject"], "Resource": "*" }] } CORS: none ACL: S3 Testing user: FULL_CONTROL ``` 设置访问限制 ```powershell 定制cors的策略文件 /data/conf/rules.xml Allow everything * GET HEAD PUT POST DELETE * 30 ``` ```powershell 应用浏览器跨域设置文件 [root@stor04 /data/conf]# s3cmd setcors rules.xml s3://images 查看效果 [root@stor04 /data/conf]# s3cmd info s3://images s3://images/ (bucket): Location: default Payer: BucketOwner Expiration Rule: none Policy: { ... } CORS: ACL: S3 Testing user: FULL_CONTROL ``` 访问效果 ```powershell [root@stor04 /data/conf]# curl http://images.stor04.superopsmsb.com:7480/linux/issue \S Kernel \r on an \m ``` 脚本测试 ```powershell 定制访问测试脚本 /data/scripts/tests4.py import boto.s3.connection access_key = 'BX2IDCO5ADZ8IWZ4AVX7' secret_key = 'wKXAiBBspTx9kl1NpcAi4eOCHxvQD7ORHnrmvBlf' conn = boto.connect_s3( aws_access_key_id=access_key, aws_secret_access_key=secret_key, host='10.0.0.16', port=7480, is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(), ) def get_object_url(bucket_name, object_name): try: bucket = conn.get_bucket(bucket_name) plans_key = bucket.get_key(object_name) plans_url = plans_key.generate_url(0, query_auth=False, force_http=False) return plans_url except Exception as e: print("get {} error:{}".format(object_name, e)) return False print(get_object_url("images", "linux/issue")) ``` ```powershell 获取资源对象的url效果 [root@stor04 /data/scripts]# python tests4.py http://10.0.0.16:7480/images/linux/issue?x-amz-meta-s3cmd-attrs=atime%3A1664200700/ctime%3A1654585109/gid%3A0/gname%3Aroot/md5%3Af078fe086dfc22f64b5dca2e1b95de2c/mode%3A33188/mtime%3A1603464839/uid%3A0/uname%3Aroot 访问url的效果 [root@stor04 /data/scripts]# curl http://10.0.0.16:7480/images/linux/issue?x-amz-meta-s3cmd-attrs=atime%3A1664200700/ctime%3A1654585109/gid%3A0/gname%3Aroot/md5%3Af078fe086dfc22f64b5dca2e1b95de2c/mode%3A33188/mtime%3A1603464839/uid%3A0/uname%3Aroot \S Kernel \r on an \m ``` **小结** ``` ``` ### 1.3.7 底层数据 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell RGW是一个对象处理网关。数据实际存储在ceph集群中。利用librados的接口，与ceph集群通信。RGW主要存储三类数据：元数据(metadata)、索引数据(bucket index)、数据(data)。 - 元数据信息：包括user(用户信息)，bucket(存储桶关系)，bucket.instance(存储桶实例)。 - 索引数据信息: 主要k/v的结构来维护bucket中rgw对象的索引信息，val是关于rgw对象的元数据信息 - 数据信息: 其实就是rgw object内容，它存放在一个或多个rados object中 ``` ```powershell 这三类数据一般存储在不同的pool中，元数据也分多种元数据，存在不同的ceph pool中。 ``` ```powershell 默认情况下，RGW安装完毕后，会在rados集群上生成包括如下存储池的一系列存储池 [cephadm@admin ceph-cluster]$ ceph osd pool ls .rgw.root default.rgw.control default.rgw.meta default.rgw.log 当我们开始基于对象存储的方式实现数据访问后，它就会生成另外一些存储池 [cephadm@admin ~]$ ceph osd pool ls .rgw.root # 包含的都是zone,zonegroup,realm等信息 default.rgw.control # pool中对象的控制信息 default.rgw.meta # 包含的是元数据信息 default.rgw.log # 包含的是日志处理信息，包括垃圾回收机制 default.rgw.buckets.index # 包含的是存储桶和对象的映射信息 default.rgw.buckets.data # 包含的是存储桶的对象数据信息 注意： 这里的default指的是 ceph的zone区域。 control利用librados提供的对象watch-notify功能，当有数据更新时，通知其他RGW刷新cache。 ``` 元数据 ```powershell 查看元数据 [cephadm@admin ~]$ radosgw-admin metadata list [ "bucket", "bucket.instance", "otp", # One-time password 一次性密码机制 "user" ] ``` ```powershell 查看用户 [cephadm@admin ~]$ radosgw-admin metadata list user [ "s3user" ] 获取用户细节信息 [cephadm@admin ~]$ radosgw-admin metadata list user:s3user ... ``` ```powershell 查看存储桶 [cephadm@admin ~]$ radosgw-admin metadata list bucket [ "images", "swift-super" ] 获取存储桶信息 [cephadm@admin ~]$ radosgw-admin metadata get bucket:images: ... ``` ```powershell 获取存储桶实例 [cephadm@admin ~]$ radosgw-admin metadata list bucket.instance [ "images:09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7", "swift-super:09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.4" ] 获取存储桶实例信息 [cephadm@admin ~]$ radosgw-admin metadata get bucket.instance:images:09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7 ... ``` **简单实践** 元数据存储池 ```pow {zone}.rgw.meta 存放的是元数据 root: bucket及bucket-instance users.keys: 用户key users.email:用户Email,object的key值=email users.swift: swift账号 users.uid: s3用户及用户的Bucket信息 roles： heap： ``` ```powershell 查看用户的uid信息 [cephadm@admin ~]$ rados -p default.rgw.meta -N users.uid ls s3user s3user.buckets ``` ```powershell 获取用户管理的buckets [cephadm@admin ~]$ rados -p default.rgw.meta -N users.uid listomapkeys s3user.buckets images swift-super 默认查看bucket信息是二进制，所以我们需要将其保存到一个文件中，通过专用命令进行查看 [cephadm@admin ~]$ rados -p default.rgw.meta -N users.uid getomapval s3user.buckets images images_bucket Writing to images_bucket 查看加密文件信息 [cephadm@admin ~]$ ceph-dencoder import images_bucket type cls_user_bucket_entry decode dump_json { "bucket": { "name": "images", "marker": "09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7", "bucket_id": "09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7" }, "size": 23, "size_rounded": 4096, "creation_time": "2022-09-27 00:49:53.839482Z", "count": 1, "user_stats_sync": "true" } ``` ```powershell 其他信息获取 bucket信息： rados -p default.rgw.meta -N root ls bucket元数据：rados -p default.rgw.meta -N root get {bucket_id} meta_file ``` 索引数据存储池 ```powershell 获取buckets的索引信息 [cephadm@admin ~]$ rados -p default.rgw.buckets.index ls .dir.09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.4 .dir.09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7 ``` ```powershell 查看存储桶的对象信息 [cephadm@admin ~]$ rados -p default.rgw.buckets.index listomapkeys .dir.09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7 linux/issue ``` ```powershell 查看对象元数据信息 [cephadm@admin ~]$ rados -p default.rgw.buckets.index getomapval .dir.09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7 linux/issue object_key Writing to object_key 查看文件内容信息 [cephadm@admin ~]$ ceph-dencoder type rgw_bucket_dir_entry import object_key decode dump_json { "name": "linux/issue", "instance": "", "ver": { "pool": 11, "epoch": 1 }, "locator": "", "exists": "true", ... } ``` ```powershell 文件对象的header中记录了一些统计信息 [cephadm@admin ~]$ rados -p default.rgw.buckets.index getomapheader .dir.09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7 index_object_header Writing to index_object_header 查看统计信息 [cephadm@admin ~]$ ceph-dencoder type rgw_bucket_dir_header import index_object_header decode dump_json { "ver": 2, "master_ver": 0, "stats": [ 1, { "total_size": 23, "total_size_rounded": 4096, "num_entries": 1, "actual_size": 23 } ], "new_instance": { "reshard_status": "not-resharding", "new_bucket_instance_id": "", "num_shards": -1 } } ``` 数据信息存储池 ```powershell 查看数据对象 [cephadm@admin ~]$ rados -p default.rgw.buckets.data ls 09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7_linux/issue 查看对象数据的属性信息 [cephadm@admin ~]$ rados -p default.rgw.buckets.data listxattr 09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7_linux/issue user.rgw.acl user.rgw.content_type user.rgw.etag user.rgw.idtag user.rgw.manifest user.rgw.pg_ver user.rgw.source_zone user.rgw.storage_class user.rgw.tail_tag user.rgw.x-amz-content-sha256 user.rgw.x-amz-date user.rgw.x-amz-meta-s3cmd-attrs ``` ```powershell 获取对象的manifest的信息 [cephadm@admin ~]$ rados -p default.rgw.buckets.data getxattr 09d0b6f1-fc49-4838-bda1-08cc60553e29.15577.7_linux/issue user.rgw.manifest > manifest 查看信息详情 [cephadm@admin ~]$ ceph-dencoder type RGWObjManifest import manifest decode dump_json { "objs": [], ... ``` **小结** ``` ``` ### 1.3.8 进阶方案 学习目标 这一节，我们从 基础知识、案例解读、小结 三个方面来学习。 **基础知识** 简介 ```powershell ceph分布式存储机制，随着大量应用的普及，在生产中的存储地位越来越高了，但是自我搭建存储云的成本太高了，所以我们在实际应用的过程中，往往是混合云(公有云+私有云)的方式来实现,借助于公有云的可扩展、低成本实现大量普通数据的存储，借助于私有云的高度可控实现敏感数据的安全控制。 ``` ```powershell 混合云存储的特点： 高性能：敏感活动数据存储在私有云存储中，普通数据存储到公有云存储中。利用公有云的特性，实现负载的分散，从而提供一个较高的综合访问性能； 安全可控：混合云存储中的私有云部分的软硬件资源都是高度控制的，所以可以实现敏感数据的可控制性和安全性； 高度扩展：借助于公有云存储的扩展特性，混合云存储也具备了高度扩展容量的特性； 相对低成本：普通数据存储到公有云存储中，节省了私有云存储的成本，还拥有公有云存储的成本优势，混合云存储具有低成本的优势。 ``` ceph解决方案 ```powershell 混合云存储相较于公有云存储和私有云存储会更加全面，更加完善。Ceph 的对象存储针对混合云的场景，也相应的提供了解决方案，即云同步Cloud Sync功能。Ceph RGW 的Cloud Sync 功能是基于RGW Multisite 机制实现的，先看下RGW Multisite 机制。 ``` ```powershell Ceph RGW 的 Multisite 机制用于实现多个Ceph 对象存储集群间数据同步，其涉及到的核心概念包括： zone：对应于一个独立的集群，由一组 RGW 对外提供服务。 zonegroup：每个 zonegroup 可以对应多个zone，zone 之间同步数据和元数据。 realm：每个realm都是独立的命名空间，可以包含多个 zonegroup，zonegroup 之间同步元数据。 ``` ![image-20220927100024375](../../img/kubernetes/kubernetes_storage_ceph/image-20220927100024375.png) ```powershell Multisite 是一个zone 层面的功能处理机制，所以默认情况下，是 zone 级的数据同步，所以说操作粒度过于粗糙。而基于RGW multisite 实现了 Cloud Sync，支持将Ceph 中的对象数据同步到支持 S3 接口的公有云存储中，作为slave zone不再是一个具体的对象存储集群，而是一个抽象程度更高的概念，可以代表任何一个对象存储集群，从而实现了混合云的基本能力。 Cloud Sync 功能正是将支持 S3 接口的存储集群，抽象为 slave zone 的概念，然后通过Multisite 机制，实现将 Ceph 中的对象数据同步到外部对象存储中。 ``` 局限性 ```powershell 在使用 Ceph 对象存储时， RGW 的 Cloud Sync 功能实际上是基本可以满足混合云存储的应用场景的，当然劣势是不可避免的： 1 支持的同步粒度最细为存储桶级，在某些应用场景下，存储桶级的同步粒度是不太合适的； 2 RGW Multisite 的数据同步因为涉及到多云环境，所以时间控制要求高，一些时间敏感的场景并不适用。 ``` **案例解读** 存储分级管理 ```powershell 存储介质： 出于对访问性能、成本等因素的考虑，同时引入多种存储介质的分级管理，实现高效数据存储。 存储策略： 主要针对的是数据级别的副本设定(默认3副本)和存储厂商的指定。 存放规则： 针对应用场景的不同，对不同的数据实现各自的数据存放规则。但是因为是zone级别的，所以有些场景不合适。 ``` 对象周期管理 ```powershell 作为对象存储，它的最大特点就是基于web形式来进行访问，不可避免的涉及到缓存的管理，所以对象生命周期管理是非常重要的事项。 目前来说，对象存储的周期管理主要有两种方式：迁移处理和过期删除处理 ``` 自动迁移管理 ```powershell 根据存储桶日志中的操作记录等属性参数，我们可以实现对存储桶中的对象数据的热度进行分析，并按照分析结果自动生成迁移策略，从而实现对象数据的自动化管理。 从 target bucket 中读取存储桶日志； 对日记记录进行过滤、分析，得到用户配置的规则中所标定的对象数据的访问热度； 生成相应的生命周期管理规则； 将生成的生命周期管理规则配置到相应的存储桶上。 ``` **小结** ``` ``` ## 1.4 CephFS接口 ### 1.4.1 基础知识 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 场景 ```powershell 有了 RBD，远程磁盘挂载的问题就解决了，但 RBD 的问题是不能多个主机共享一个磁盘，如果有一份数据很多客户端都要读写该怎么办呢？这时 CephFS 作为文件系统存储解决方案就派上用场了。 ``` 简介 ```powershell Ceph 文件系统（Ceph FS）是个 POSIX 兼容的文件系统，它直接使用 Ceph 存储集群来存储数据。 Ceph 文件系统与 Ceph 块设备、同时提供 S3 和 Swift API 的 Ceph 对象存储、或者原生库（ librados ）的实现机制稍显不同。 ``` 要点 ![image-20211117114047214](../../img/kubernetes/kubernetes_storage_ceph/image-20211117114047214.png) ```powershell CephFS需要至少运行一个元数据服务器（MDS）守护进程（ceph-mds），此进程管理与CephFS上存储的文件相关的元数据，并协调对Ceph存储集群的访问。 因此，若要使用CephFS接口，需要在存储集群中至少部署一个MDS实例。 ``` ```powershell 注意： MDS虽然称为元数据服务，但是它却不存储任何元数据信息，它存在的目的仅仅是 让我们rados集群提供的存储接口能够兼容POSIX 的文件系统接口 -- 简单来说，它是真正元数据信息的索引服务。 客户端在访问文件接口的时候，首先连接到 MDS上，在MDS的内存里面维持元数据的索引信息，这些信息真正的保存在RADOS集群的metadata pool上面，而对应的数据，保存在对应的 data pool上面。 而且以将元数据服务作为一个非状态的效果，便于扩展。 ``` ```powershell 相较于NFS来说，它主要有以下特点优势： 1 底层数据冗余的功能，底层的roados 提供了基本的数据冗余功能，因此不存在nfs的单点故障因素。 2 底层的roados系统有n个存储节点组成，所以数据的存储可以分散IO，吞吐量较高。 3 底层的roados系统有n个存储节点组成，所以ceph提供的扩展性要想当的高 ``` **简单实践** 部署mds主机 ```powershell 在admin主机上为stor05上部署mds服务 [cephadm@admin ceph-cluster]$ ceph-deploy mds create stor05 在stor05上，并确认是否开启了守护进程 [cephadm@admin ceph-cluster]$ ssh stor05 "ps aux | grep ceph-mds | grep -v grep" ceph 14471 0.1 1.2 384316 22504 ? Ssl 11:48 0:00 /usr/bin/ceph-mds -f --cluster ceph --id stor05 --setuser ceph --setgroup ceph ``` ```powershell 在admin主机上通过 ceph -s 中查看mds的状态 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 6m) mgr: mon01(active, since 47h), standbys: mon02 mds: 1 up:standby # 一个mds部署成功 osd: 6 osds: 6 up (since 6m), 6 in (since 43h) rgw: 1 daemon active (stor04) 注意： 在没有为fs准备存储池的前提下，是看不到效果的 ``` ```powershell 需要专用的命令来进行查看mds的状态效果 [cephadm@admin ceph-cluster]$ ceph mds stat 1 up:standby 结果显示： 查看MDS的相关状态可以发现，刚添加的MDS处于Standby模式 ``` 创建专属存储池 ```powershell 创建元数据存储池 ceph osd pool create cephfs-metadata 32 32 ceph osd pool create cephfs-data 64 64 ``` ```powershell 将刚才创建的存储池组合为fs专属存储池 [cephadm@admin ceph-cluster]$ ceph fs new cephfs cephfs-metadata cephfs-data new fs with metadata pool 14 and data pool 15 ``` 确认效果 ```powershell 确认效果 [cephadm@admin ceph-cluster]$ ceph -s cluster: ... services: mon: 3 daemons, quorum mon01,mon02,mon03 (age 11m) mgr: mon01(active, since 47h), standbys: mon02 mds: cephfs:1 {0=stor05=up:active} osd: 6 osds: 6 up (since 11m), 6 in (since 43h) rgw: 1 daemon active (stor04) ... ``` ```powershell 查看状态 [cephadm@admin ceph-cluster]$ ceph fs status cephfs cephfs - 0 clients ====== +------+--------+--------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+--------+---------------+-------+-------+ | 0 | active | stor05 | Reqs: 0 /s | 10 | 13 | +------+--------+--------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs-metadata | metadata | 1536k | 35.8G | | cephfs-data | data | 0 | 35.8G | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) ``` **小结** ``` ``` ### 1.4.2 接口使用 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 客户端使用cephFS方式 ![image-20211210121816739](../../img/kubernetes/kubernetes_storage_ceph/image-20211210121816739.png) ```powershell 客户端使用cephFS，主要有两种方式： 内核文件系统 - Ceph、libcephfs 用户空间文件系统 - FUSE(Filesystem in USErspace) 前提： 1 至少一个节点运行ceph-mds守护进程 2 创建存储池 ceph-metadata、ceph-data 3 激活文件系统 4 无论那种客户端使用cephFS的方式，都需要有专门的认证机制。 ``` 准备认证文件 ```powershell 在admin节点上，准备认证账号 [cephadm@admin ceph-cluster]$ ceph auth get-or-create client.fsclient mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data' -o ceph.client.fsclient.keyring 注意：我们只需要对data的存储池进行操作即可，不需要对metadata存储池进行授权。 ``` ```powershell 查看账号信息 [cephadm@admin ceph-cluster]$ ceph auth get client.fsclient 查看认证文件信息 [cephadm@admin ceph-cluster]$ ls *fsclient* ceph.client.fsclient.keyring [cephadm@admin ceph-cluster]$ cat ceph.client.fsclient.keyring [client.fsclient] key = AQBgdTJj+LVdFBAAZOjGIsw4t+o1swZlW4CvKQ== ``` 秘钥管理 ```powershell 客户端挂载cephFS的时候，需要指定用户名和key，而这两项是有不同的两个选项来指定的，所以我们应该将这两个内容分开存储 获取相关的秘钥信息 [cephadm@admin ceph-cluster]$ ceph-authtool -p -n client.fsclient ceph.client.fsclient.keyring AQBgdTJj+LVdFBAAZOjGIsw4t+o1swZlW4CvKQ== [cephadm@admin ceph-cluster]$ ceph auth print-key client.fsclient AQBgdTJj+LVdFBAAZOjGIsw4t+o1swZlW4CvKQ== ``` ```powershell 保存用户账号的密钥信息于secret文件，用于客户端挂载操作认证之用 [cephadm@admin ceph-cluster]$ ceph auth print-key client.fsclient > fsclient.key [cephadm@admin ceph-cluster]$ cat fsclient.key AQBgdTJj+LVdFBAAZOjGIsw4t+o1swZlW4CvKQ== ``` **简单实践** ceph客户端的需求 ```powershell 客户端主机环境准备 内核模块ceph.ko yum install ceph 安装ceph-common程序包 yum install ceph-common 提供ceph.conf配置文件 ceph-deploy --overwrite-conf config push stor06 获取到用于认证的密钥文件 sudo scp fsclient.key stor06:/etc/ceph/ ``` cephFS客户端准备 ```powershell 我们准备将 stor06 作为cephFS的客户端主机，检查stor06的ceph模块加载情况 [root@stor06 ~]# modinfo ceph ... 保证 stor06上有ceph相关的配置文件 [root@stor06 ~]# ls /etc/ceph/ ceph.client.kube.keyring ... ceph.conf fsclient.key rbdmap ``` 挂载cephFS ```powershell 创建专属的数据目录 [root@stor06 ~]# mkdir /cephfs-data/ 挂载cephFS对象 [root@stor06 ~]# mount -t ceph mon01:6789,mon02:6789,mon03:6789:/ /cephfs-data/ -o name=fsclient,secretfile=/etc/ceph/fsclient.key 注意： 内核空间挂载ceph文件系统，要求必须指定ceph文件系统的挂载路径 确认挂载效果 [root@stor06 ~]# mount | grep cephfs 10.0.0.13:6789,10.0.0.14:6789,10.0.0.15:6789:/ on /cephfs-data type ceph (rw,relatime,name=fsclient,secret=,acl,wsize=16777216) ``` ```powershell 查看挂载点的效果 [root@stor06 ~]# stat -f /cephfs-data/ 文件："/cephfs-data/" ID：c931549bd8ae8624 文件名长度：255 类型：ceph 块大小：4194304 基本块大小：4194304 块：总计：9182 空闲：9182 可用：9182 Inodes: 总计：0 空闲：-1 结果显示： 挂载点的类型是 ceph。 确认文件系统使用效果 [root@stor06 ~]# echo nihao cephfs > /cephfs-data/cephfs.txt [root@stor06 ~]# cat /cephfs-data/cephfs.txt nihao cephfs ``` 开机自启动挂载cephFS ```powershell 卸载刚才挂载的cephFS [root@stor06 ~]# umount /cephfs-data [root@stor06 ~]# mount | grep ceph 设置开机自启动 [root@stor06 ~]# echo 'mon01:6789,mon02:6789,mon03:6789:/ /cephfs-data ceph name=fsclient,secretfile=/etc/ceph/fsclient.key,_netdev,noatime 0 0' >> /etc/fstab 属性解析： _netdev - 告诉挂载程序，一旦超时，自动跳过去。 noatime - 在读文件时不去更改文件的access time属性，提高性能 ``` ```powershell 挂载磁盘 [root@stor06 ~]# mount -a [root@stor06 ~]# mount | grep ceph 10.0.0.13:6789,10.0.0.14:6789,10.0.0.15:6789:/ on /cephfs-data type ceph (rw,noatime,name=fsclient,secret=,acl) 查看效果 [root@stor06 ~]# cat /cephfs-data/cephfs.txt nihao cephfs ``` **小结** ``` ``` ### 1.4.3 FUSE实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 对于某些操作系统来说，它没有提供对应的ceph内核模块，我们还需要使用cephFS的话，可以通过 FUSE方式来实现，FUSE全称Filesystem in Userspace，用于非特权用户能够无需操作内核而创建文件系统。 ``` ```powershell 如果我们在非内核的环境下使用cephFS的话，需要对客户端主机的环境做一些准备工作： 安装ceph-fuse程序包 yum install -y ceph-fuse ceph-common 获取到客户端账号的keyring文件和ceph.conf配置文件 ceph-deploy --overwrite-conf config push stor06 sudo scp ceph.client.fsclient.keyring root@stor06:/etc/ceph/ ``` **简单实践** 客户端环境准备 ```powershell 我们准备将 stor06 作为cephFS的客户端主机，安装对应的软件 [root@stor06 ~]# yum install ceph-fuse ceph-common -y 保证 stor06上有ceph相关的配置文件 [root@stor06 ~]# ls /etc/ceph/ ceph.client.kube.keyring ... ceph.conf fsclient.key rbdmap ``` 挂载cephFS ```powershell 创建专属的数据目录 [root@stor06 ~]# mkdir /cephfs-data-user/ 挂载cephFS对象 [root@stor06 ~]# ceph-fuse -n client.fsclient -m mon01:6789,mon02:6789,mon03:6789 /cephfs-data-user/ ceph-fuse[7963]: starting ceph client2062-09-27 12:23:47.905 7f6e97de4f80 -1 init, newargv = 0x5637c9545540 newargc=9 ceph-fuse[7963]: starting fuse 确认挂载效果 [root@stor06 ~]# mount | grep cephfs ceph-fuse on /cephfs-data-user type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) ``` ```powershell 查看挂载点的效果 [root@stor06 ~]# stat -f /cephfs-data-user/ 文件："/cephfs-data-user/" ID：0 文件名长度：255 类型：fuseblk 块大小：4194304 基本块大小：4194304 块：总计：9182 空闲：9182 可用：9182 Inodes: 总计：1 空闲：0 确认文件系统使用效果 [root@stor06 ~]# echo nihao cephfs user > /cephfs-data-user/cephfs.txt [root@stor06 ~]# cat /cephfs-data-user/cephfs.txt nihao cephfs -user ``` 开机自启动挂载cephFS ```powershell 卸载刚才挂载的cephFS [root@stor06 ~]# umount /cephfs-data-user [root@stor06 ~]# mount | grep ceph 设置开机自启动 [root@stor06 ~]# echo 'none /cephfs-data-user fuse.ceph ceph.id=fsclient,ceph.conf=/etc/ceph/ceph.conf,_netdev,noatime 0 0' >> /etc/fstab 属性解析： _netdev - 告诉挂载程序，一旦超时，自动跳过去。 noatime - 在读文件时不去更改文件的access time属性，提高性能 ``` ```powershell 挂载磁盘 [root@stor06 ~]# mount -a [root@stor06 ~]# mount | grep ceph ceph-fuse on /cephfs-data-user type fuse.ceph-fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,allow_other) 查看效果 [root@stor06 ~]# cat /cephfs-data-user/cephfs.txt nihao cephfs user 专用的卸载命令 [root@stor06 ~]# fusermount -u /cephfs-data-user [root@stor06 ~]# mount | grep cephfs ``` **小结** ``` ``` ## 1.5 集群管理 ### 1.5.1 集群状态 学习目标 这一节，我们从 基础知识、集群管理、小结 三个方面来学习。 **基础知识** 检查集群状态 ```powershell [cephadm@admin ceph-cluster]$ ceph -s cluster: 查看集群ID 和 集群运行状况 id: 1d4e5773-619a-479d-861a-66ba451ce945 health: HEALTH_OK services: 各种服务的状态监控 mon: 3 daemons, quorum mon01,mon02,mon03 (age 48m) mgr: mon01(active, since 2d), standbys: mon02 mds: cephfs:1 {0=stor05=up:active} osd: 6 osds: 6 up (since 48m), 6 in (since 44h) rgw: 1 daemon active (stor04) task status: 任务状态 data: 存储数据相关的监控，包括存储池、存储对象、存储统计数量等 pools: 10 pools, 352 pgs objects: 270 objects, 60 MiB usage: 6.3 GiB used, 114 GiB / 120 GiB avail pgs: 352 active+clean ``` 检查集群即时状态 ```powershell 查看pg的状态 [cephadm@admin ceph-cluster]$ ceph pg stat 352 pgs: 352 active+clean; 60 MiB data, 314 MiB used, 114 GiB / 120 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 6 op/s 查看pool的状态 [cephadm@admin ceph-cluster]$ ceph osd pool stats pool .rgw.root id 6 nothing is going on ... ``` ```powershell 查看ceph的存储空间状态 [cephadm@admin ceph-cluster]$ ceph df [detail] RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 120 GiB 114 GiB 314 MiB 6.3 GiB 5.26 TOTAL 120 GiB 114 GiB 314 MiB 6.3 GiB 5.26 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .rgw.root 6 32 1.2 KiB 4 768 KiB 0 36 GiB ... ``` 查看osd状态 ```powershell 查看osd的基本状态 [cephadm@admin ceph-cluster]$ ceph osd stat 6 osds: 6 up (since 52m), 6 in (since 44h); epoch: e150 查看osd的属性详情 [cephadm@admin ceph-cluster]$ ceph osd dump epoch 150 fsid 1d4e5773-619a-479d-861a-66ba451ce945 ... 查看osd的归属结构Ceph将列显CRUSH树及主机、它的OSD、OSD是否已启动及其权重 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.11691 root default -3 0.03897 host mon01 0 hdd 0.01949 osd.0 up 1.00000 1.00000 1 hdd 0.01949 osd.1 up 1.00000 1.00000 ... ``` 查看mon状态 ```powershell 集群中存在多个Mon主机时，应该在启动集群之后读取或写入数据之前检查Mon的仲裁状态；事实上，管理员也应该定期检查这种仲裁结果 显示监视器映射 [cephadm@admin ceph-cluster]$ ceph mon stat e3: 3 mons at {mon01=[v2:10.0.0.13:3300/0,v1:10.0.0.13:6789/0],mon02=[v2:10.0.0.14:3300/0,v1:10.0.0.14:6789/0],mon03=[v2:10.0.0.15:3300/0,v1:10.0.0.15:6789/0]}, election epoch 36, leader 0 mon01, quorum 0,1,2 mon01,mon02,mon03 [cephadm@admin ceph-cluster]$ ceph mon dump ... 0: [v2:10.0.0.13:3300/0,v1:10.0.0.13:6789/0] mon.mon01 1: [v2:10.0.0.14:3300/0,v1:10.0.0.14:6789/0] mon.mon02 2: [v2:10.0.0.15:3300/0,v1:10.0.0.15:6789/0] mon.mon03 dumped monmap epoch 3 ``` ```powershell 普通方式查看多个mon节点的选举状态 [cephadm@admin ceph-cluster]$ ceph quorum_status 以格式化方式查看选举状态 [cephadm@admin ceph-cluster]$ ceph quorum_status -f json-pretty ``` **集群管理** 集群管理套接字 ```powershell Ceph的管理套接字接口常用于查询守护进程 资源对象文件保存的目录 /var/lib/ceph目录 套接字默认保存 于/var/run/ceph目录 此接口的使用不能以远程方式进程 ``` 借助于socket文件实现ceph属性的动态管理 ```powershell 命令的使用格式： ceph --admin-daemon /var/run/ceph/socket-name 获取使用帮助： ceph --admin-daemon /var/run/ceph/socket-name help ``` ```powershell 简单示例 ceph --admin-daemon /var/run/ceph/ceph-mgr.mon01.asok help ceph --admin-daemon /var/run/ceph/ceph-mon.mon01.asok help ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok help ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok help ``` **小结** ``` ``` ### 1.5.2 配置文件 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 文件目录 ```powershell ceph软件安装完成后，就会生成专用的家目录 root@mon01:~# ls /etc/ceph/ ceph.client.admin.keyring ceph.conf rbdmap tmpzvyyulv9 ``` ```powershell ceph配置文件就是 ceph.conf，该文件遵循 ini文件的语法格式。 root@mon01:~# cat /etc/ceph/ceph.conf [global] fsid = 22fc9b8e-6d30-463d-b312-89dc91ee0969 public_network = 10.0.0.0/24 cluster_network = 192.168.8.0/24 ... [mon] mon allow pool delete = true ``` 配置结构 ```powershell 对于ceph.conf文件来说，它主要有四部分组成： [global] ceph存储的全局性配置 [osd] 所有影响 Ceph 存储集群中 ceph-osd 守护进程的相关配置，会覆盖global的相同属性 [mon] 所有影响 Ceph 存储集群中 ceph-mon 守护进程的相关配置，会覆盖global的相同属性 [client] 所有影响 Ceph 客户端管理的相关配置，比如 块设备客户端、fs客户端、oss客户端等 注意：ceph.conf主要是针对整体来进行设置的，如果我们需要针对某一个对象来进行修改的话，可以使用 "对象.主机名|id" 的方式来进行针对性定制。 比如：[mon.mon01]、[osd.1] ``` 配置文件的加载 ```powershell 对于ceph来说，它的配置文件，不仅仅有 /etc/ceph/ceph.conf 一种样式，他还有更多的样式 -- 主要是作用的范围不一样 全局范围： /etc/ceph/ceph.conf $CEPH_CONF -c path/to/ceph.conf 局部范围 ~/.ceph/config ./ceph/conf ``` **简单实践** 元参数 ``` 在ceph环境中，它提供了很多可以直接拿过来使用的元数据变量。这些变量可以直接从环境中获取到指定的信息，方便我们进行后续处理。 这些信息，不仅仅可以在运行时使用，还可以在配置文件中进行定制。 ``` ```powershell 常见的变量信息有： - $cluster：当前Ceph集群的名称 - $type：当前服务的类型名称，可能会展开为osd或mon； - $id：进程的标识符，例如对osd.0来说，其标识符为0； - $host：守护进程所在主机的主机名； - $name：其值为$type.$id ``` 运行时使用 ```powershell 我们在ceph运行过程中，可以通过 以下方式来获取不同的属性信息内容 查看配置信息 ceph daemon {daemon-type}.{id} config show 查看帮助信息 ceph daemon {daemon-type}.{id} help 查看指定参数信息 ceph daemon {daemon-type}.{id} config get {parameter} 修改特定的信息 ceph tell {daemon-type}.{daemon id or *} injectargs --{name} {value} [--{name} {value}] 设定特定的属性 ceph daemon {daemon-type}.{id} set {name} {value} ``` ```powershell 获取所有配置属性 [root@mon01 ~]# ceph daemon osd.0 config show { "name": "osd.0", "cluster": "ceph", "admin_socket": "/var/run/ceph/ceph-osd.0.asok", "admin_socket_mode": "", ... "threadpool_empty_queue_max_wait": "2", "throttler_perf_counter": "true" } [root@mon01 ~]# ceph daemon osd.0 config show | wc -l 1644 ``` ```powershell 获取帮助信息 [root@mon01 ~]# ceph daemon mon.mon01 help { "add_bootstrap_peer_hint": "add peer address as potential bootstrap peer for cluster bringup", ... "sessions": "list existing sessions", "smart": "Query health metrics for underlying device", "sync_force": "force sync of and clear monitor store", "version": "get ceph version" } ``` ```powershell 获取制定信息 [root@mon01 ~]# ceph daemon mon.mon01 config get admin_socket { "admin_socket": "/var/run/ceph/ceph-mon.mon01.asok" } [root@mon01 ~]# ceph daemon osd.0 config get admin_socket { "admin_socket": "/var/run/ceph/ceph-osd.0.asok" } ``` 更改属性 ```powershell 查看当前的属性信息 [root@mon01 ~]# ceph daemon osd.0 config get debug_osd { "debug_osd": "1/5" } 修改制定的属性 [root@mon01 ~]# ceph tell osd.0 injectargs '--debug-osd 0/5' 查看修改后的属性信息 [root@mon01 ~]# ceph daemon osd.0 config get debug_osd { "debug_osd": "0/5" } ``` ```powershell 修改制定的属性 ceph daemon osd.0 config set debug_osd 1/5 ceph daemon osd.0 config get debug_osd ``` **小结** ``` ``` ### 1.5.3 磁盘管理 学习目标 这一节，我们从 场景需求、简单实践、小结 三个方面来学习。 **场景需求** 简介 ```powershell 在ceph分布式存储集群里，数以千计万计的磁盘数量规模下，磁盘出现故障是不可避免，所以服务器更换故障磁盘就成为大规模机房的日常运维工作之一，而ceph存储集群中就需要在物理磁盘更换的前提下，需要不断重复的完成OSD磁盘的更换的操作。 ``` 操作步骤 ```powershell 在ceph集群环境中，主要有三套方法来实现磁盘的更换操作，但是这些操作各有优劣，甚至对于某些场景来说，不能够使用xx方式的磁盘更换操作，所以我们需要了解一下这些常见的磁盘管理操作。 注意： 这里主要介绍多种磁盘更换的思路和区别，里面的一些数据，仅供参考。 ``` ```powershell 更换方式1： 1 到指定节点上，停止指定的osd进程 2 将移除OSD节点状态标记为out 3 从crush中移除OSD节点，该节点不作为数据的载体 4 删除OSD节点和对应的认证信息 5 增加一个新的 OSD 注意： 1、2、3、4、5 这几步都会发生数据迁移的动作 ``` ```powershell 更换方式2： 1 修改osd的数据操作权重值，让数据不分布在这个节点上 2 到指定节点上，停止指定的osd进程 3 将移除OSD节点状态标记为out 4 从crush中移除OSD节点，该节点不作为数据的载体 5 删除OSD节点和对应的认证信息 6 增加一个新的 OSD 注意： 1、5、6 这几步会发生数据迁移动作 ``` ```powershell 更换方式3： 1 对ceph集群的osd设置禁止数据迁移的标记 2 修改osd的数据操作权重值，让数据不分布在这个节点上 3 到指定节点上，停止指定的osd进程 4 从crush中移除OSD节点，该节点不作为数据的载体 5 删除OSD节点和对应的认证信息 6 增加一个新的 OSD 7 移除ceph集群的osd禁止数据迁移标记 注意： 因为添加了标记，所以1-6这几步的数据迁移都不会执行，只有7这一步会发生数据迁移动作。 ``` ```powershell 注意： 如果想要查看每次数据迁移的状态，可以查看当前pg的状态效果 ceph pg dump pgs|awk '{print $1,$15}'|grep -v pg > pg_base.txt 每次执行数据迁移动作后，可以再次保存，然后对比此次数据迁移的效果 diff -y -W 100 pg_end.txt pg_base.txt --suppress-common-lines | wc -l ``` **简单实践** 磁盘更换方式1实践 ```powershell 1 到指定节点上，停止指定的osd进程 [root@mon01 ~]# systemctl stop ceph-osd@1 [root@mon01 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... 1 hdd 0.01949 osd.1 down 1.00000 1.00000 ``` ```powershell 2 将移除OSD节点状态标记为out [root@mon01 ~]# ceph osd out 1 marked out osd.1. [root@mon01 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... 1 hdd 0.01949 osd.1 down 0 1.00000 ``` ```powershell 标记状态的改变，会导致pg的迁移，查看pgs的状态 [root@mon01 ~]# ceph pg dump pgs 10.1d 0 ... 0 active+clean ... ... 注意： pg迁移过程中，会有一些异常状态 active+recovering+undersized+remapped 我们还有另外一种方式来进行检查数据迁移效果 [cephadm@admin ceph-cluster]$ ceph -s ... services: ... osd: 7 osds: 6 up (since 20s), 6 in (since 20s); 17 remapped pgs rgw: 1 daemon active (stor04) task status: data: pools: 10 pools, 352 pgs objects: 270 objects, 60 MiB usage: 6.4 GiB used, 114 GiB / 120 GiB avail pgs: 0.284% pgs not active 158/810 objects degraded (19.506%) 14/810 objects misplaced (1.728%) 327 active+clean 17 active+recovery_wait+undersized+degraded+remapped 3 active+recovery_wait+degraded 1 active+recovery_wait 1 active+recovery_wait+remapped 1 active+recovering+degraded 1 activating 1 active+recovering+undersized+remapped io: recovery: 3.1 MiB/s, 4 objects/s ``` ```powershell 3 从crush中移除OSD节点，该节点不作为数据的载体 [root@mon01 ~]# ceph osd crush remove osd.1 removed item id 1 name 'osd.1' from crush map ``` ```powershell 4 移除无效的osd节点,并移除认证信息 [cephadm@admin ceph-cluster]$ ceph osd rm osd.1 removed osd.1 [cephadm@admin ceph-cluster]$ ceph auth del osd.1 updated ``` ```powershell 5-1 清理磁盘 dmsetup status cat /var/lib/ceph/osd/ceph-1/fsid dmsetup remove ceph-1的id mkfs.ext4 /dev/sdc 5-2 擦除磁盘上的数据 [cephadm@admin ceph-cluster]$ ceph-deploy disk zap mon01 /dev/sdc 5-3 添加磁盘数据 [cephadm@admin ceph-cluster]$ ceph-deploy --overwrite-conf osd create mon01 --data /dev/sdc ``` 磁盘更换方式2实践 ```powershell 这次以mon02 主机为例 [root@mon02 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -5 0.03897 host mon02 2 hdd 0.01949 osd.2 up 1.00000 1.00000 3 hdd 0.01949 osd.3 up 1.00000 1.00000 ... ``` ```powershell 1 修改osd的数据操作权重值，让数据不分布在这个节点上 [root@mon02 ~]# ceph osd crush reweight osd.3 0 reweighted item id 3 name 'osd.3' to 0 in crush map ``` ```powershell 2 到指定节点上，停止指定的osd进程 [root@mon02 ~]# sudo systemctl stop ceph-osd@3 3 将移除OSD节点状态标记为out [cephadm@admin ceph-cluster]$ ceph osd out osd.3 marked out osd.3. ``` ```powershell 4 从crush中移除OSD节点，该节点不作为数据的载体 [root@mon02 ~]# ceph osd crush remove osd.3 removed item id 3 name 'osd.3' from crush map ``` ```powershell 5 移除无效的osd节点后删除认证信息 [cephadm@admin ceph-cluster]$ ceph osd rm osd.3 removed osd.3 [cephadm@admin ceph-cluster]$ ceph auth del osd.3 updated ``` ```powershell 6-1 清理磁盘 dmsetup status cat /var/lib/ceph/osd/ceph-2/fsid dmsetup remove ceph-3的id mkfs.ext4 /dev/sdc 6-2 擦除磁盘上的数据后添加磁盘 [cephadm@admin ceph-cluster]$ ceph-deploy disk zap mon02 /dev/sdc [cephadm@admin ceph-cluster]$ ceph-deploy --overwrite-conf osd create mon02 --data /dev/sdc ``` 磁盘更换方式3实践 ```powershell 1 对ceph集群的osd设置禁止数据迁移的标记 [cephadm@admin ceph-cluster]$ ceph osd set norebalance norebalance is set [cephadm@admin ceph-cluster]$ ceph osd set nobackfill nobackfill is set [cephadm@admin ceph-cluster]$ ceph osd set norecover norecover is set 查看状态后，osd会有相应标签 [cephadm@admin ceph-cluster]$ ceph -s ... services: ... osd: 6 osds: 6 up (since 16m), 6 in (since 16m) flags nobackfill,norebalance,norecover rgw: 1 daemon active (stor04) ``` ```powershell 2 修改osd的数据操作权重值，让数据不分布在这个节点上 [cephadm@admin ceph-cluster]$ ceph osd crush reweight osd.5 0 reweighted item id 5 name 'osd.5' to 0 in crush map 因为集群已经禁止数据迁移了，所以这个时候，并不会发生数据迁移动作 [cephadm@admin ceph-cluster]$ ceph -s cluster: id: 1d4e5773-619a-479d-861a-66ba451ce945 health: HEALTH_WARN nobackfill,norebalance,norecover flag(s) set Reduced data availability: 16 pgs inactive, 20 pgs peering Degraded data redundancy: 104/810 objects degraded (12.840%), 30 pgs degraded, 1 pg undersized ... data: pools: 10 pools, 352 pgs objects: 270 objects, 60 MiB usage: 6.5 GiB used, 114 GiB / 120 GiB avail pgs: 27.273% pgs not active 104/810 objects degraded (12.840%) 6/810 objects misplaced (0.741%) 226 active+clean 95 peering 16 active+recovery_wait+degraded 14 active+recovery_wait+undersized+degraded+remapped 1 remapped+peering ``` ```powershell 3 到指定节点上，停止指定的osd进程 [root@mon03 ~]# sudo systemctl stop ceph-osd@5 4 从crush中移除OSD节点，该节点不作为数据的载体 [root@mon02 ~]# ceph osd crush remove osd.5 removed item id 5 name 'osd.5' from crush map 5 移除无效的osd节点后删除认证信息 [cephadm@admin ceph-cluster]$ ceph osd rm osd.5 removed osd.5 [cephadm@admin ceph-cluster]$ ceph auth del osd.5 updated ``` ```powershell 6-1 清理磁盘 dmsetup status cat /var/lib/ceph/osd/ceph-5/fsid dmsetup remove ceph-5的id mkfs.ext4 /dev/sdc 6-2 擦除磁盘上的数据后添加磁盘 [cephadm@admin ceph-cluster]$ ceph-deploy disk zap mon03 /dev/sdc [cephadm@admin ceph-cluster]$ ceph-deploy --overwrite-conf osd create mon03 --data /dev/sdc ``` ```powershell 7 移除ceph集群的osd禁止数据迁移标记 [cephadm@admin ceph-cluster]$ ceph osd unset norebalance norebalance is unset [cephadm@admin ceph-cluster]$ ceph osd unset nobackfill nobackfill is unset [cephadm@admin ceph-cluster]$ ceph osd unset norecover norecover is unset ``` ```powershell 稍等一会查看ceph的状态 [cephadm@admin ceph-cluster]$ ceph -s ... data: pools: 10 pools, 352 pgs objects: 270 objects, 60 MiB usage: 6.5 GiB used, 113 GiB / 120 GiB avail pgs: 352 active+clean 结果显示： 这个时候，才会发生数据迁移的动作 ``` 生产技巧 ```powershell 当我们遇到ceph宿主机重启的场景，如果不想发生频繁的数据迁移动作，可以手工临时禁用ceph自带数据恢复平衡功能即可 禁用ceph集群数据迁移： ceph osd set noout ceph osd set nobackfill ceph osd set norecover 启用ceph集群数据迁移： ceph osd unset noout ceph osd unset nobackfill ceph osd unset norecover ``` **小结** ``` ``` ### 1.5.4 性能调优 学习目标 这一节，我们从 基础知识、常见策略、小结 三个方面来学习。 **基础知识** 简介 ```powershell 性能优化没有一个标准的定义，如果用大白话来说的话，他就是在现有主机资源的前提下，发挥业务最大的处理能力。也理解为：通过空间(资源消耗)优化换取时间(业务处理)效益 它主要体现在几个方面：改善业务逻辑处理的速度、提升业务数据吞吐量、优化主机资源的消耗量。 ``` ```powershell 性能优化没有一个基准值，我们只能通过自我或者普遍的场景工作经验，设定一个阶段性的目标，然后通过物理手段、配置手段、代码手段等方式提高主机资源的业务处理能力。 ``` 通用处理措施 ```powershell 基础设施 合适设备 - 多云环境、新旧服务器结合、合适网络等 研发环境 - 应用架构、研发平台、代码规范等 应用软件 多级缓存 - 浏览器缓存、缓存服务器、服务器缓存、应用缓存、代码缓存、数据缓存等。 数据压缩 - 数据构建压缩、数据传输压缩、缓存数据压缩、数据对象压缩、清理无效数据等。 数据预热 - 缓冲数据、预取数据、预置主机、数据同步等。 业务处理 削峰填谷 - 延时加载、分批发布、限流控制、异步处理、超时降级等 任务批处理 - 数据打包传输、数据批量传输、延迟数据处理等 ``` ceph环境 ```powershell Ceph分布式存储环境是以集群服务器来进行承载存储服务的，所以对于Ceph的性能优化主要体现在基础设施层的合适设备、应用项目的多级缓存和数据压缩、业务处理的任务批处理。 ``` **常见策略** 基础设施 ```powershell Ceph集群的各个服务守护进程在 Linux 服务器上运行，所以适当调优操作系统能够对ceph存储集群的性能产生积极的影响。它主要涉及到以下几个方面： - 选择合适的CPU和内存。 不同角色具有不同的 CPU 需求，MDS(4)>OSD(2)>MON(1)，纠删代码功能需要更多 CPU。 不同角色具有不同的 内存 需求，MON>OSD - 选择合适的磁盘容量 合理评估阶段性业务量，计算节点数量及各个节点的磁盘容量需求。 选择合适的 SAS、SATA、SSD，实现分级存储的能力。 OS系统、数据、日志使用独立的SSD硬盘，使整体吞吐量最大化 - 选择合适的网络设备 Ceph集群对于网络数据传输能力要求高，需要支持巨型帧(9000字节，默认1500字节)功能 Ceph集群数据传输量很大，需要所有设备的 MTU 设置必须一致 如果可以的话，网络带宽一定要大，最好以3倍需求量的方式采购 - 配置合适的文件系统 推荐使用vfs文件系统，ceph借助于xfs扩展属性在创建和挂载文件系统场景中提高索引数据缓存 osd_mkfs_options_xfs、osd_mount_options_xfs - 搭建内部时间服务器 ceph集群对于时间要求比较高，为了保证数据的通信一致性，最好采用内部时间服务器 - 合理采用多云环境 将合适的业务负载转移到公有云环境。 ``` 应用软件 ```powershell Ceph集群的很多功能都是以各种服务配置的方式来进行实现的，所以我们有必要结合不同的配置属性，实现ceph集群环境级别的性能优化： - 数据访问 采用public网络和cluster网络机制，实现数据操作网络的可靠性，避免外部网络攻击的安全性。 配置属性：public_network、cluster_network、max_open_file等 - 数据存储 设定合理的存储池的PG 数量，集群PGs总数 = (OSD总数 * 100 / 最大副本数) 更多数据存储规划：https://docs.ceph.com/en/latest/rados/operations/placement-groups/#choosing-the-number-of-placement-groups - 多级缓存 根据实际情况开启RDBCache(rbd_cache_xxx)、RGWCache(rgw_cache_xxx)、OSDCache、MDSCache 建立合理的SSD缓存pool，设定合理的cache age、target size 等参数 理清业务场景，根据缓存模式(write-back、readproxy、readonly等)，启用cache-tiering机制。 ``` 业务处理 ```powershell ceph集群在大规模数据处理的时候，可以通过数据批处理能力来实现数据的高效传输。 - 数据批处理 filestore_max|min_sync_interval 从日志到数据盘最大|小同步间隔(seconds) filestore_queue_max_ops|bytes 数据盘单次最大接受的操作数|字节数 filestore_queue_committing_max_ops|bytes 数据盘单次最大处理的操作数|字节数 journal_max_write_ops|bytes 日志一次性写入的最大操作数|字节数(bytes) journal_queue_max_ops|bytes 日志一次性查询的最大操作数|字节数(bytes) osd_max_write_size OSD一次可写入的最大值(MB) osd_recovery_max_active 同一时间内活跃的恢复请求数 ``` **小结** ``` ``` ### 1.5.5 性能测试 学习目标 这一节，我们从 基准测试、rados测试、小结 三个方面来学习。 **基准测试** 磁盘测试 ```powershell 准备工作-清理缓存 [root@mon03 ~]# echo 3 > /proc/sys/vm/drop_caches^C [root@mon03 ~]# cat /proc/sys/vm/drop_caches 0 [root@mon03 ~]# echo 3 > /proc/sys/vm/drop_caches 查看磁盘 [root@mon03 ~]# ls /var/lib/ceph/osd/ ceph-4 ceph-5 ``` ```powershell OSD 磁盘写性能 [root@mon03 ~]# dd if=/dev/zero of=/var/lib/ceph/osd/ceph-4/write_test bs=1M count=100 ... 104857600字节(105 MB)已复制，0.4009 秒，262 MB/秒 OSD 磁盘读性能 [root@mon03 ~]# dd if=/var/lib/ceph/osd/ceph-4/write_test of=/dev/null bs=1M count=100 ... 104857600字节(105 MB)已复制，0.0233439 秒，4.5 GB/秒 清理环境 [root@mon03 ~]# rm -f /var/lib/ceph/osd/ceph-4/write_test ``` ```powershell 参考资料： https://www.thomas-krenn.com/en/wiki/Linux_I/O_Performance_Tests_using_dd ``` 网络测试 ```powershell 在多台主机上准备网络基准测试工具 yum install iperf -y 服务端命令参数 -s 以server模式启动 -p 指定服务器端使用的端口或客户端所连接的端口 客户端命令参数 -d 同时进行双向传输测试 -n 指定传输的字节数 ``` ```powershell 节点1执行命令 [root@admin ~]# iperf -s -p 6900 ``` ```powershell 客户端执行命令 [root@mon03 ~]# iperf -c admin -p 6900 ------------------------------------------------------------ Client connecting to admin, TCP port 6900 TCP window size: 552 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.15 port 48376 connected with 10.0.0.12 port 6900 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 2.58 GBytes 2.21 Gbits/sec ``` ```powershell 注意： 无论是磁盘测试还是网络测试，都需要多次测试后，获取平均值效果 ``` **rados测试** bench性能测试工具 ```powershell 它是Ceph 自带的 rados bench 测试工具，命令格式如下： rados bench -p <存储池> <秒数> <操作模式> -b <块大小> -t 并行数 --no-cleanup 参数解读： 操作模式：包括三种，分别是write：写，seq：顺序读；rand：随机读 块大小：默认为4M 并行数: 读/写并行数，默认为 16 --no-cleanup 表示测试完成后不删除测试用数据。 - 在做读测试之前，需要使用该参数来运行一遍写测试来产生测试数据， - 在测试结束后运行 rados -p <存储池> cleanup 清理测试数据。 注意： 曾经的bench-write已经被整合到bench中了 ``` ```powershell 创建测试存储池 [root@admin ~]# ceph osd pool create pool-test 32 32 pool 'pool-test' created ``` ```powershell 写测试： [root@admin ~]# rados bench -p pool-test 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_admin_5683 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 ... 10 16 147 131 52.3689 64 1.18095 1.16048 Total time run: 10.9577 ... Min latency(s): 0.202 ``` ```powershell 顺序读测试 [root@admin ~]# rados bench -p pool-test 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 ... 10 15 139 124 45.0608 8 9.09797 0.93913 Total time run: 11.5093 ... Min latency(s): 0.0240264 ``` ```powershell 随机读测试 [root@admin ~]# rados bench -p pool-test 10 rand hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 ... 10 16 589 573 229.081 244 0.20209 0.272096 Total time run: 10.2428 ... Min latency(s): 0.0225745 ``` ```powershell 清理环境 [root@admin ~]# rados -p pool-test cleanup Removed 148 objects ``` load-gen性能测试工具 ```powershell Ceph 自带的 rados 性能测试工具，可在集群内产生混合类型的负载，相较于bench，load-gen只做吞吐量测试。命令格式如下： rados -p 存储池 load-gen 参数解读： ... --num-objects # 初始生成测试用的对象数 --min-object-size # 最小对象大小 --max-object-size # 最大对象大小 --max-ops # 最大操作数目 --min-op-len # 最小操作长度，相当于iodepth --max-op-len # 最大操作长度 --read-percent # 读操作的百分比 --target-throughput # 单次提交IO的累计吞吐量上线，单位 MB/s --run-length # 运行时长，单位秒 --percent # 读写混合中，读操作占用比例 --max_backlog # 单次提交IO的吞吐量上限 注意： iodepth就是io队列深度(io队列里面允许出现几个命令等待操作)，它的作用是 - 控制硬盘操作的过程中的负载情况，避免系统一直发送指令，出现磁盘受不了导致系统崩溃的现象 ``` ```powershell 在pool-test存储池中，初始对象50个，单次io的最大数量16，顺序写入4M的数据块,测试时间10秒。 [root@mon03 ~]# rados -p pool-test load-gen --num-objects 50 --min-object-size 4M --max-object-size 4M --max-object-size 4M --max-ops 16 --min-op-len 4M --max-op-len 4M --percent 5 --target-throughput 40 --run-length 10 run length 10 seconds preparing 50 objects load-gen will run 10 seconds 1: throughput=0MB/sec pending data=0 READ : oid=obj-AF5uiLtTtxSNSNR off=0 len=4194304 op 0 completed, throughput=3.86MB/sec ... 9: throughput=0.442MB/sec pending data=0 waiting for all operations to complete cleaning up objects ``` **小结** ``` ``` # 1 综合实践 ## 1.1 存储池 ### 1.1.1 创建实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 存储的常用管理操作包括列出、创建、重命名和删除等操作，常用相关的工具都是 “ceph osd pool”的子命令，包括ls、create、rename和rm等 ``` 创建存储池 ```powershell 副本型存储池创建命令 ceph osd pool create [pgp-num] [replicated] \ [crush-rule-name] [expected-num-objects] 纠删码池创建命令 ceph osd pool create erasure \ [erasure-code-profile] [crush-rule-name] [expected-num-objects] ``` ```powershell 命令格式中常用的参数 pool-name：存储池名称，在一个RADOS存储集群上必须具有唯一性； pg-num：当前存储池中的PG数量，合理的PG数量对于存储池的数据分布及性能表现来说至关重要； pgp-num ：用于归置的PG数量，其值应该等于PG的数量 replicated|erasure：存储池类型；副本存储池需更多原始存储空间，但已实现Ceph支持的所有操作，而纠删码存储池所需原始存储空间较少，但目前仅实现了Ceph的部分操作 crush-ruleset-name：此存储池所用的CRUSH规则集的名称，不过，引用的规则集必须事先存在 ``` 获取存储池 ```powershell 列出存储池 ceph osd pool ls [detail] 获取存储池的统计数据 ceph osd pool stats [pool-name] 显示存储池的用量信息 rados df ``` 重命名存储池 ```powershell 重命名存储池 ceph osd pool rename old-name new-name ``` **简单实践** 查看存储池信息 ```powershell 列出所有存储池名称 [cephadm@admin ceph-cluster]$ ceph osd pool ls .rgw.root default.rgw.control default.rgw.meta default.rgw.log default.rgw.buckets.index default.rgw.buckets.data default.rgw.otp default.rgw.buckets.non-ec cephfs-metadata cephfs-data 获取所有存储池的详细信息 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail pool 6 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 117 flags hashpspool stripe_width 0 application rgw ... 结果显示： 可以看到，所有存储池的详情信息 ``` ```powershell 获取所有存储池的统计数据 [cephadm@admin ceph-cluster]$ ceph osd pool stats pool .rgw.root id 6 nothing is going on ... pool cephfs-data id 15 nothing is going on 获取制定存储池的统计数据 [cephadm@admin ceph-cluster]$ ceph osd pool stats cephfs-data pool cephfs-data id 15 nothing is going on ``` ```powershell 显示存储池的用量信息 [cephadm@admin ceph-cluster]$ rados df POOL_NAME USED OBJECTS ... RD_OPS RD WR_OPS WR ... .rgw.root 768 KiB 4 ... 0 0 B 4 4 KiB ... ... default.rgw.otp 0 B 0 ... 0 0 B 0 0 B ... total_objects 328 total_used 7.0 GiB total_avail 113 GiB total_space 120 GiB ``` 创建副本存储池 ```powershell 创建副本型存储池 [cephadm@admin ceph-cluster]$ ceph osd pool create replicated-pool 32 32 pool 'replicated-pool' created 查看副本统计信息 [cephadm@admin ceph-cluster]$ ceph osd pool stats replicated-pool pool replicated-pool id 18 nothing is going on 查看副本详情信息 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep replicated-pool pool 18 'replicated-pool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 507 flags hashpspool stripe_width 0 ``` 创建纠偏码池 ```powershell 创建纠删码池 [cephadm@admin ceph-cluster]$ ceph osd pool create erasure-pool 16 16 erasure pool 'erasure-pool' created 查看副本统计信息 [cephadm@admin ceph-cluster]$ ceph osd pool stats erasure-pool pool erasure-pool id 19 nothing is going on 查看副本详情信息 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep erasure-pool pool 19 'erasure-pool' erasure size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 511 flags hashpspool stripe_width 8192 ``` **小结** ``` ``` ### 1.1.2 删除实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 意外删除存储池会导致数据丢失，因此 Ceph 实施了两个机制来防止删除存储池，要删除存储池，必须先禁用这两个机制 ``` ```powershell 第一个机制是NODELETE标志，其值需要为false，默认也是false 查看命令：ceph osd pool get pool-name nodelete 修改命令：ceph osd pool set pool-name nodelete false 第二个机制是集群范围的配置参数mon allow pool delete，其默认值为 “false”，这表示默认不能删除存储池，临时设定的方法如下 ceph tell mon.* injectargs --mon-allow-pool-delete={true|false} 建议删除之前将其值设置为true，删除完成后再改为false ``` ```powershell 删除存储池命令格式 ceph osd pool rm pool-name pool-name --yes-i-really-really-mean-it ``` **简单实践** 删除存储池 ```powershell 查看删除标识 [cephadm@admin ceph-cluster]$ ceph osd pool get replicated-pool nodelete nodelete: false 删除存储池 [cephadm@admin ceph-cluster]$ ceph osd pool rm replicated-pool replicated-pool --yes-i-really-really-mean-it Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool 结果显示: 默认无法删除存储池 ``` ```powershell 修改删除标识 [cephadm@admin ceph-cluster]$ ceph tell mon.* injectargs --mon-allow-pool-delete=true mon.mon01: injectargs:mon_allow_pool_delete = 'true' mon.mon02: injectargs:mon_allow_pool_delete = 'true' mon.mon03: injectargs:mon_allow_pool_delete = 'true' 再次删除存储池 [cephadm@admin ceph-cluster]$ ceph osd pool rm replicated-pool replicated-pool --yes-i-really-really-mean-it pool 'replicated-pool' removed ``` 永久删除方法 ```powershell 将所有的ceph.conf文件中，增加默认允许删除存储池的配置 [cephadm@admin ceph-cluster]$ cat /etc/ceph/ceph.conf ... [mon] mon_allow_pool_delete = true 然后同步到所有节点，重启mon服务 [cephadm@admin ceph-cluster]$ ceph-deploy --overwrite-conf config push admin mon01 mon02 mon03 [cephadm@admin ceph-cluster]$ for i in mon0{1..3}; do ssh $i "sudo systemctl restart ceph-mon.target"; done ``` ```powershell 修改删除标识 [cephadm@admin ceph-cluster]$ ceph tell mon.* injectargs --mon-allow-pool-delete=false 创建存储池 [cephadm@admin ceph-cluster]$ ceph osd pool create replicated-pool 16 16 pool 'replicated-pool' created 删除存储池 [cephadm@admin ceph-cluster]$ ceph osd pool rm replicated-pool replicated-pool --yes-i-really-really-mean-it pool 'replicated-pool' removed ``` **小结** ``` ``` ### 1.1.3 配额管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 设置存储池配额 ```powershell Ceph支持为存储池设置可存储对象的最大数量（max_objects）和可占用的最大空间（ max_bytes）两个纬度的配额 ceph osd pool set-quota max_objects|max_bytes 获取存储池配额的相关信息 ceph osd pool get-quota ``` 配置存储池参数 ```powershell 存储池的诸多配置属性保存于配置参数中 获取配置：ceph osd pool get 设定配置：ceph osd pool set ``` ```powershell 常用的可配置参数 size： 存储池中的对象副本数； min_size： I/O所需要的最小副本数； pg_num： 存储池的PG数量； pgp_num： 计算数据归置时要使用的PG的有效数量； crush_ruleset： 用于在集群中映射对象归置的规则组； nodelete： 控制是否可删除存储池； nopgchange： 控制是否可更改存储池的pg_num和pgp_num； nosizechange： 控制是否可更改存储池的大小； noscrub和nodeep-scrub： 控制是否可整理或深层整理存储池以解决临时高I/O负载的问题 scrub_min_interval： 集群负载较低时整理存储池的最小时间间隔； 默认值为0，表示其取值来自于配置文件中的osd_scrub_min_interval参数； scrub_max_interval：整理存储池的最大时间间隔； 默认值为0，表示其取值来自于配置文件中的osd_scrub_max_interval参数； deep_scrub_interval：深层整理存储池的间隔； 默认值为0，表示其取值来自于配置文件中的osd_deep_scrub参数； ``` **简单实践** 设置存储池配额 ```powershell 获取存储池配额的相关信息 [cephadm@admin ceph-cluster]$ ceph osd pool get-quota erasure-pool quotas for pool 'erasure-pool': max objects: N/A # 表示没有限制 max bytes : N/A # 表示没有限制 设置存储池配额的值 [cephadm@admin ceph-cluster]$ ceph osd pool set-quota erasure-pool max_objects 10000 set-quota max_objects = 10000 for pool erasure-pool [cephadm@admin ceph-cluster]$ ceph osd pool set-quota erasure-pool max_bytes 1024000 set-quota max_bytes = 1024000 for pool erasure-pool [cephadm@admin ceph-cluster]$ ceph osd pool get-quota erasure-pool quotas for pool 'erasure-pool': max objects: 10k objects max bytes : 1000 KiB ``` 配置存储池参数 ```powershell 创建存储池 [cephadm@admin ceph-cluster]$ ceph osd pool create replicated-pool 16 16 pool 'replicated-pool' created 获取配置 [cephadm@admin ceph-cluster]$ ceph osd pool get replicated-pool size size: 3 ``` ```powershell 调整配置属性 [cephadm@admin ceph-cluster]$ ceph osd pool set replicated-pool size 4 set pool 21 size to 4 [cephadm@admin ceph-cluster]$ ceph osd pool get replicated-pool size size: 4 调整配置属性 [cephadm@admin ceph-cluster]$ ceph osd pool set replicated-pool size 3 set pool 21 size to 3 [cephadm@admin ceph-cluster]$ ceph osd pool get replicated-pool size size: 3 ``` **小结** ``` ``` ### 1.1.4 快照管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 关于存储池快照 存储池快照是指整个存储池的状态快照 通过存储池快照，可以保留存储池状态的历史 创建存储池快照可能需要大量存储空间，具体取决于存储池的大小 ``` 常见操作 ```powershell 创建存储池快照 ceph osd pool mksnap rados -p mksnap 列出存储池的快照 rados -p lssnap 回滚存储池至指定的快照 rados -p rollback 删除存储池快照 ceph osd pool rmsnap rados -p rmsnap ``` **简单实践** 查看快照 ```powershell 查看制定存储池的快照数量 [cephadm@admin ceph-cluster]$ rados -p replicated-pool lssnap 0 snaps [cephadm@admin ceph-cluster]$ rados -p erasure-pool lssnap 0 snaps ``` 制作快照 ```powershell 创建快照 [cephadm@admin ceph-cluster]$ ceph osd pool mksnap replicated-pool replicated-snap created pool replicated-pool snap replicated-snap [cephadm@admin ceph-cluster]$ ceph osd pool mksnap erasure-pool erasure-snap created pool erasure-pool snap erasure-snap 查看效果 [cephadm@admin ceph-cluster]$ rados -p replicated-pool lssnap 1 replicated-snap 2062.09.28 22:28:33 1 snaps [cephadm@admin ceph-cluster]$ rados -p erasure-pool lssnap 1 erasure-snap 2062.09.28 22:28:43 1 snaps 结果分析： 创建过image的存储池无法再创建存储池快照了，因为存储池当前已经为unmanaged snaps mode 或者 selfmanaged_snaps模式，而没有创建image的，就可以做存储池快照。 ``` 快照还原 ```powershell 添加文件 [cephadm@admin ceph-cluster]$ rados put ceph-file /home/cephadm/ceph-cluster/ceph.conf --pool=erasure-pool [cephadm@admin ceph-cluster]$ rados ls --pool=erasure-pool ceph-file 添加快照 [cephadm@admin ceph-cluster]$ ceph osd pool mksnap erasure-pool erasure-snap1 created pool erasure-pool snap erasure-snap1 [cephadm@admin ceph-cluster]$ rados -p erasure-pool lssnap 1 erasure-snap 2062.09.28 22:28:43 2 erasure-snap1 2062.09.28 22:30:17 2 snaps 回滚快照 [cephadm@admin ceph-cluster]$ rados -p erasure-pool rollback ceph.conf erasure-snap rolled back pool erasure-pool to snapshot erasure-snap ``` 删除快照 ```powershell 删除快照 [cephadm@admin ceph-cluster]$ ceph osd pool rmsnap erasure-pool erasure-snap1 removed pool erasure-pool snap erasure-snap1 [cephadm@admin ceph-cluster]$ ceph osd pool rmsnap erasure-pool erasure-snap removed pool erasure-pool snap erasure-snap [cephadm@admin ceph-cluster]$ rados --pool=erasure-pool lssnap 0 snaps ``` **小结** ``` ``` ### 1.1.5 压缩管理 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell BlueStore存储引擎提供即时数据压缩，以节省磁盘空间 ``` 启用压缩 ```powershell ceph osd pool set compression_algorithm snappy 压缩算法有none、zlib、lz4、zstd和snappy等几种，默认为snappy； zstd有较好的压缩比，但比较消耗CPU； lz4和snappy对CPU占用比例较低； 不建议使用zlib； ceph osd pool set compression_mode aggressive 压缩模式：none、aggressive、passive和force，默认值为none； none：不压缩 passive：若提示COMPRESSIBLE，则压缩 aggressive：除非提示INCOMPRESSIBLE，否则就压缩； force：始终压缩 注意：如果需要全局压缩，最好在配置文件中定制 ``` ```powershell 其它可用的压缩参数 compression_required_ratio：指定压缩比，取值格式为双精度浮点型，其值为 SIZE_COMPRESSED/SIZE_ORIGINAL，即压缩后的大小与原始内容大小的比值，默认为 .875； compression_max_blob_size：压缩对象的最大体积，无符号整数型数值，默认为0； compression_min_blob_size：压缩对象的最小体积，无符号整数型数值，默认为0； ``` ```powershell 全局压缩选项 可在ceph配置文件中设置压缩属性，它将对所有的存储池生效，可设置的相关参数如下 bluestore_compression_algorithm bluestore_compression_mode bluestore_compression_required_ratio bluestore_compression_min_blob_size bluestore_compression_max_blob_size bluestore_compression_min_blob_size_ssd bluestore_compression_max_blob_size_ssd bluestore_compression_min_blob_size_hdd bluestore_compression_max_blob_size_hdd ``` **简单实践** 查看默认的算法 ```powershell [root@mon01 ~]# ceph daemon osd.0 config show | grep compression "bluestore_compression_algorithm": "snappy", ... "bluestore_compression_mode": "none", "bluestore_compression_required_ratio": "0.875000", ... "rbd_compression_hint": "none", 结果显示： 默认的压缩算法是 snappy,默认的压缩模式是 node,压缩比例是0.875 注意： 这条命令最好指定节点下执行 ``` 更改压缩算法 ```powershell 管理节点设定压缩算法 [cephadm@admin ceph-cluster]$ ceph osd pool set erasure-pool compression_algorithm zstd set pool 19 compression_algorithm to zstd 存储池查看效果 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep erasure-pool pool 19 'erasure-pool' erasure ... compression_algorithm zstd ``` 更改算法模式 ```powershell 更改算法模式 [cephadm@admin ceph-cluster]$ ceph osd pool set erasure-pool compression_mode aggressive set pool 19 compression_mode to aggressive 查看存储池算法模式 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep erasure-pool pool 19 'erasure-pool' erasure ... compression_mode aggressive ``` 还原算法模式和算法 ```powershell 还原算法和模式 ceph osd pool set erasure-pool compression_algorithm snappy ceph osd pool set erasure-pool compression_mode none 查看效果 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep erasure-pool pool 19 'erasure-pool' erasure ... compression_algorithm snappy compression_mode none ``` **小结** ``` ``` ### 1.1.6 纠删码基础 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 场景介绍 ```powershell 纠删码理论(erasure coding，EC)始于 20 世纪 60 年代,它是一种数据保护方法。从原理上说，它将数据分割成片段，把冗余数据块扩展、编码，并将其存储在不同的位置，比如磁盘、存储节点或者其它地理位置。 总数据块 = 原始数据块 + 校验块， 常见表示为，n = k + m 当冗余级别为n时，将这些数据块分别存放在n个硬盘上，这样就能容忍小于m个（假设初始数据有k个）硬盘发生故障。当不超过m个硬盘发生故障时，只需任意选取k个正常的数据块就能计算得到所有的原始数据。 在云存储中，我们通常会使用副本的方式来保证系统的可用性。问题是当存储达到 PB 级别后要求的容量将会非常高。通过使用纠删码技术可以在保证相同可用性的情况下，节省大量存储空间，从而大大的降低 TCO（总体成本）。Ceph 从 Firefly 版本开始支持纠删码。 ``` ```powershell 比如8块磁盘的总数(N)，5个数据盘(K), 通过5个数据块计算出3个校验块(m)，我们可以用RS(5,3)来代表。在这样的场景中，每份数据都会切分5份--5*5的逆向矩阵，它可以容忍任意 3 块磁盘的故障。如果某些数据块丢失，可以通过剩下的可用数据恢复出原始的数据内容。 ``` ![image-20220928232215966](../../img/kubernetes/kubernetes_storage_ceph/image-20220928232215966.png) ceph纠偏码 ```powershell Ceph纠删码实现了高速的计算，但有2个缺点：速度慢、只支持对象的部分操作。ceph中的EC编码是以插件的形式来提供的。EC编码有三个指标：空间利用率、数据可靠性和恢复效率。ceph提供以下几种纠删码插件：clay(coupled-layer)、jerasure、lrc、shec、isa等 ``` ```powershell Ceph支持以插件方式加载使用的纠删编码插件，存储管理员可根据存储场景的需要优化选择合用的插件。 jerasure： 最为通用的和灵活的纠删编码插件，它也是纠删码池默认使用的插件；不过，任何一个OSD成员的丢失，都需要余下的所有成员OSD参与恢复过程；另外，使用此类插件时，管理员还可以通过technique选项指定要使用的编码技术： reed_sol_van：最灵活的编码技术，管理员仅需提供k和m参数即可； cauchy_good：更快的编码技术，但需要小心设置PACKETSIZE参数； reed_sol_r6_op、liberation、blaum_roth或liber8tion： 仅支持使用m=2的编码技术，功能特性类同于RAID 6； lrc： 全称为Locally Repairable Erasure Code，即本地修复纠删码，除了默认的m个编码块之外，它会额外在本地创建指定数量（l）的奇偶校验块，从而在一个OSD丢失时，可以仅通过l个奇偶校验块完成恢复 isa： 仅支持运行在intel CPU之上的纠删编码插件，它支持reed_sol_van和cauchy两种技术 shec： shec(k,m,l)，k为数据块，m为校验块，l代表计算校验块时需要的数据块数量。 其最大允许失效数据块为：ml/k，恢复失效的单个数据块（data chunk）只需要额外读取l个数据块。 ``` 基础概念 ```powershell 块（chunk） 基于纠删码编码时，每次编码将产生若干大小相同的块(有序)。 ceph通过数量相等的PG将这些分别存储在不同的osd中。 条带（strip） 如果编码对象太大，可分多次进行编码，每次完成编码的部分称为条带。同一个对内的条带时有序的。 分片（shared） 同一个对象中所有序号相同的块位于同一个PG上，他们组成对象的一个分片，分片的编号就是块的序号。 ``` ```powershell 空间利用率（rate）： 通过k/n计算。 对象尺寸： Ceph 存储集群里的对象有最大可配置尺寸，对象尺寸必须足够大,而且应该是条带单元的整数倍。 条带数量（strip_size）： Ceph 客户端把一系列条带单元写入由条带数量所确定的一系列对象，这一系列的对象称为一个对象集。 客户端写到对象集内的最后一个对象时，再返回到第一个。 条带宽度（strip_width）： 条带都有可配置的单元尺寸。Ceph 客户端把数据等分成适合写入对象的条带单元。 strip_width = chunk_size * strip_size ``` ```powershell 假设有EC（k=4，m=2），strip_size=4，chunk_size=1K，那么strip_width=4K。在ceph中，strip_width默认为4K。 假如object对象内容是 ABCDEFGHIJKL；将该数据对象写入pool时，纠删码函数把object分4个数据块：第1个是ABC,第2个是DEF，第3个是GHI，第4个是JKL；如果object的长度不是K的倍数，object会被填充一些内容；纠删码函数同时创建2个编码块：第4个是xxx，第5个是yyy； ``` ![image-20220928234255383](../../img/kubernetes/kubernetes_storage_ceph/image-20220928234255383.png) **简单实践** 创建纠删码池 ```powershell 命令格式 ceph osd pool create erasure [erasure-code-profile] [crush-rule-name] [expected-num-objects] 要点解析 未指定要使用的纠删编码配置文件时，创建命令会为其自动创建一个，并在创建相关的CRUSH规则集时使用到它 默认配置文件自动定义k=2和m=1，这意味着Ceph将通过三个OSD扩展对象数据，并且可以丢失其中一个OSD而不会丢失数据，因此，在冗余效果上，它相当于一个大小为2的副本池 ，不过，其存储空间有效利用率为2/3而非1/2。 ``` 其他命令 ```powershell 列出纠删码配置文件 ceph osd erasure-code-profile ls 获取指定的配置文件的相关内容 ceph osd erasure-code-profile get default 自定义纠删码配置文件 ceph osd erasure-code-profile set [] [] [] [] [ ...] [--force] - directory：加载纠删码插件的目录路径，默认为/usr/lib/ceph/erasure-code； - plugin：用于生成及恢复纠删码块的插件名称，默认为jerasure； - crush-device-class：设备类别，例如hdd或ssd，默认为none，即无视类别； - crush-failure-domain：故障域，默认为host，支持使用的包括osd、host、rack、row和room等 ； - --force：强制覆盖现有的同名配置文件； ``` ```powershell 示例： 例如，如果所需的体系结构必须承受两个OSD的丢失，并且存储开销为40％ ceph osd erasure-code-profile set myprofile k=4 m=2 crush-failure-domain=osd 例如，下面的命令创建了一个使用lrc插件的配置文件LRCprofile，其本地奇偶校验块为3，故障域为osd ceph osd erasure-code-profile set LRCprofile plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd ``` 查看纠删码池 ```powershell 列出纠删码配置文件 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile ls default 获取指定的配置文件的相关内容 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile get default k=2 m=1 plugin=jerasure technique=reed_sol_van ``` 自定义纠删码配置 ```powershell 创建纠删码配置 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile set myprofile k=4 m=2 crush-failure-domain=osd 查看纠删码配置信息 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile ls default myprofile [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile get myprofile crush-device-class= crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 ``` ```powershell 创建定制的纠偏码池 [cephadm@admin ceph-cluster]$ ceph osd pool create erasure-pool-myprofile 8 8 erasure myprofile pool 'erasure-pool-myprofile' created 查看效果 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep erasure-pool-myprofile pool 22 'erasure-pool-myprofile' erasure size 6 min_size 5 crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 540 flags hashpspool stripe_width 16384 解析： erasure size 6 代表有(K+m)6个osd磁盘 ``` 基于插件定制纠偏码配置 ```powershell 基于纠偏码算法定制专属配置 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile set LRCprofile plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 查看纠删码配置信息 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile ls LRCprofile default myprofile [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile get LRCprofile crush-device-class= crush-failure-domain=osd crush-root=default k=4 l=3 m=2 plugin=lrc ``` ```powershell 创建定制配置的纠偏码池 [cephadm@admin ceph-cluster]$ ceph osd pool create erasure-pool-LRCprofile 1 1 erasure LRCprofile pool 'erasure-pool-LRCprofile' created 查看效果 [cephadm@admin ceph-cluster]$ ceph osd pool ls detail | grep erasure-pool-LRCprofile pool 23 'erasure-pool-LRCprofile' ... flags hashpspool stripe_width 16384 ``` **小结** ``` ``` ### 1.1.7 纠删码实践 学习目标 这一节，我们从 数据实践、异常实践、小结三个方面来学习。 **数据实践** 纠删码池数据实践 ```powershell 创建存储池 [cephadm@admin ceph-cluster]$ ceph osd pool create ecpool 12 12 erasure pool 'ecpool' created ``` ```powershell 数据写入实践 [cephadm@admin ceph-cluster]$ echo ABCDEFGHI | rados --pool ecpool put object_data - 数据读取实践 [cephadm@admin ceph-cluster]$ rados --pool ecpool get object_data - ABCDEFGHI 注意： 数据写入和读取的实践命令末尾必须有一个 - ``` ```powershell 文件写入实践 [cephadm@admin ceph-cluster]$ echo test > test.txt [cephadm@admin ceph-cluster]$ rados -p ecpool put test test.txt 文件读取实践 [cephadm@admin ceph-cluster]$ rados -p ecpool get test file.txt [cephadm@admin ceph-cluster]$ cat file.txt test ``` 纠删码池不支持部分image功能 ```powershell 启用rbd功能 [cephadm@admin ceph-cluster]$ ceph osd pool application enable ecpool rbd enabled application 'rbd' on pool 'ecpool' 创建image操作 [cephadm@admin ceph-cluster]$ rbd create myimg -p ecpool --size 1024 2062-09-29 01:01:01.144 7faafbfff700 -1 librbd::image::ValidatePoolRequest: handle_overwrite_rbd_info: pool missing required overwrite support 2062-09-29 01:01:01.144 7faafbfff700 -1 librbd::image::CreateRequest: 0x55a667ec7190 handle_validate_data_pool: pool does not support RBD images rbd: create error: (22) Invalid argument 结果显示： 纠偏码池不支持RBD images的参数操作 ``` **异常实践** ```powershell 创建纠删码配置 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile set Ecprofile crush-failure-domain=osd k=3 m=2 查看纠删码配置信息 [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile ls Ecprofile [cephadm@admin ceph-cluster]$ ceph osd erasure-code-profile get Ecprofile crush-device-class= crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 ``` ```powershell 基于纠删码配置文件创建erasure类型的Ceph池： [cephadm@admin ceph-cluster]$ ceph osd pool create Ecpool 16 16 erasure Ecprofile pool 'Ecpool' created 确认效果 [cephadm@admin ceph-cluster]$ ceph osd dump | grep Ecpool pool 25 'Ecpool' erasure size 5 min_size 4 crush_rule 4 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 566 flags hashpspool stripe_width 12288 结果显示： erasure size 5 表示 3+2=5 个osd磁盘来存储数据和校验码 ``` ```powershell 创建文件后提交文件到纠删码池 [cephadm@admin ceph-cluster]$ echo test_ecpool > test_file.txt [cephadm@admin ceph-cluster]$ rados put -p Ecpool object1 test_file.txt [cephadm@admin ceph-cluster]$ rados -p Ecpool ls object1 ``` ```powershell 检查纠删码池中和object1的OSDmap [cephadm@admin ceph-cluster]$ ceph osd map Ecpool object1 osdmap e566 pool 'Ecpool' (25) object 'object1' -> pg 25.bac5debc (25.c) -> up ([2,1,0,4,5], p2) acting ([2,1,0,4,5], p2) 结果显示： 0-5的osd磁盘上都有我们的数据库和校验块 ``` 测试效果 ```powershell 找到两台节点，将两个被用到的osd移除，比如mon02主机上的osd2，mon03主机上的osd5 [cephadm@admin ceph-cluster]$ ssh mon03 sudo systemctl stop ceph-osd@5 [cephadm@admin ceph-cluster]$ ssh mon02 sudo systemctl stop ceph-osd@2 [cephadm@admin ceph-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ... -5 0.03897 host mon02 2 hdd 0.01949 osd.2 down 1.00000 1.00000 3 hdd 0.01949 osd.3 up 1.00000 1.00000 -7 0.03897 host mon03 4 hdd 0.01949 osd.4 up 1.00000 1.00000 5 hdd 0.01949 osd.5 down 1.00000 1.00000 ``` ```powershell 检查EC池和object1的OSDmap [cephadm@admin ceph-cluster]$ ceph osd map Ecpool object1 osdmap e574 pool 'Ecpool' (25) object 'object1' -> pg 25.bac5debc (25.c) -> up ([NONE,1,0,4,NONE], p1) acting ([NONE,1,0,4,NONE], p1) 结果显示： osd.5和osd.2 已经不管用了 获取文件查看 [cephadm@admin ceph-cluster]$ rados -p Ecpool get object1 ok.txt 结果显示： 查看文件失败，可以看到，不允许有超过2个osd失败的效果 ``` ```powershell ``` **小结** ``` ``` ## 1.2 存储进阶 ### 1.2.1 原理解读 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 存储池 ```powershell RADOS存储集群提供的基础存储服务需要由“pool”分割为逻辑存储区域 -- 对象数据的名称空间。我们在部署ceph集群的过程中，涉及到大量的应用程序，而这些应用程序在创建的时候，需要专属的数据存储空间--存储池，比如rbd存储池、rgw存储池、cephfs存储池等。 存储池还可以再进一步细分为一至多个子名称空间，比如我们在看pool的时候 [cephadm@admin ceph-cluster]$ ceph osd pool ls .rgw.root default.rgw.control {根命名空间}.{应用命名空间}.{子空间} ``` ```powershell 对于Ceph集群来说，它的存储池主要由默认的副本池(replicated pool)和纠删码池(erasure code) 两种类型组成。 客户端在使用这些存储池的时候，往往依赖于相关用户信息的认证机制。 ``` 数据存储逻辑 ![1636103641572](../../img/kubernetes/kubernetes_storage_ceph/1636103641572.png) ```powershell Ceph 在 RADOS 集群中动态存储、复制和重新平衡数据对象。由于许多不同的用户在无数 OSD 上出于不同目的将对象存储在不同的池中，因此 Ceph 操作需要一些数据放置规划。Ceph 中主要的数据放置规划概念包括： ``` ```powershell Pools（池）： Ceph 将数据存储在pool中，pool是用于存储对象的逻辑组。首次部署集群而不创建pool时，Ceph 使用默认pool来存储数据。 pool主要管理的内容有：PG的数量、副本的数量和pool的 CRUSH 规则。 当然了。要将数据存储在池中，您必须有一个经过身份验证的用户，该用户具有该池的权限。 参考资料：https://docs.ceph.com/en/quincy/rados/operations/pools/ ``` ```powershell Placement Groups（归置组）： 归置组 (PG) 是 Ceph 如何分发数据的内部实现细节。 Ceph 将对象映射到归置组(PG)，归置组(PG)是逻辑对象池的分片或片段，它们将对象作为一个组放置到OSD中。当 Ceph 将数据存储在 OSD 中时，放置组会减少每个对象的元数据量。 每个 PG 都属于一个特定的Pool，因此当多个Pool使用相同的 OSD 时，必须注意每个OSD的PG副本的总和不能超过OSD自身允许的PG数量阈值。 参考资料：https://docs.ceph.com/en/quincy/rados/operations/placement-groups ``` ```powershell CRUSH Maps（CRUSH 映射）： CRUSH 是让 Ceph 在正常运行情况下进行数据扩展的重要部分。CRUSH 映射为 CRUSH 算法提供集群的物理拓扑数据，以确定对象及其副本的数据应该存储在哪里，以及如何跨故障域以增加数据安全等。 参考资料：https://docs.ceph.com/en/quincy/rados/operations/crush-map。 ``` ```powershell Balancer： 平衡器是一个功能，它会自动优化PG跨设备的分布，以实现数据的均衡分布，最大化集群中可以存储的数据量，并在OSD之间平均分配工作负载。 ``` ```powershell Pool数量、CRUSH规则和PG数量决定了 Ceph 将如何放置数据 ``` **简单实践** ```powershell Ceph 的 RADOS中，引入了 PG 的概念用以更好地管理数据。PG 的引入降低对数量巨大的对象的管理难度。 1、对象数量时刻处于变化中。而PG的数量经过人工规划因而严格可控。 2、PG数量远小于对象，且生命周期稳定。因此以PG为单位进行数据同步或者迁移，相比较对象难度更小。 PG 最引人注目之处在于其可以在OSD之间（根据CRUSH的实时计算结果）自由迁移，这是ceph赖以实现自动数据恢复、自动数据平衡等高级特性的基础。 ``` ```powershell RADOS 提供的是基于 Object 的存储功能，每个 Object 会先通过简单的 Hash 算法归到一个 PG 中，PGID 再作为"参数"通过 CRUSH 计算置入到多个 OSD 中。 Object - 可以理解为一个文件 PG - 可以理解为一个目录 OSD - 可以理解我一个PG目录下的简易文件系统 ``` ```powershell 存储池在操作资源对象的时候，需要设置：对象的所有权/访问权、归置组的数量、CRUSH 规则 ``` PG 映射到 OSD ![image-20220930074909586](../../img/kubernetes/kubernetes_storage_ceph/image-20220930074909586.png) ```powershell 每个Pool都有许多归置组。当 Ceph 客户端存储对象时，CRUSH 会将每个对象映射到一个归置组,然后CRUSH 将 PG 动态映射到 OSD。 所以说，PG相当于 OSD守护进程 和 Ceph客户端之间的一个中间层： 1 通过CRUSH实现数据对象动态平衡的分散到所有OSD中，同时也可以让客户端知道数据在哪个OSD中 2 中间层还允许新加入的OSD和其他OSD在数据负载时候实现动态平衡 ``` PG id ```powershell 当 Ceph客户端绑定到 Ceph Mon节点时，它会检索 Cluster Map的最新副本。客户端可以通过Cluster Map了解集群中的所有监视器、OSD和元数据服务器信息。由于数据对象与Cluster Map无关，所以它对对象位置一无所知。 客户端操作数据对象的时候，需要指定 对象ID(就是对象名称)和Pool。 ``` ```powershell CRUSH算法允许客户端计算对象应该存储在哪里，从而实现客户端能够快速联系主 OSD 以存储或检索对象。 1 Ceph 客户端输入Pool名称和对象ID rados put {object_name} /path/to/local_file --pool={pool_name} 2 Ceph 获取对象ID后对其进行hash处理 hash(对象ID名称)%PG_num 3 Ceph 基于PG数为模对PG进行哈希计算后获取 PG ID，比如 58 4 Ceph 根据pool名称获取Pool ID，比如 4 5 Ceph 将pool ID 附加到 PG ID，比如 4.58 ``` ```powershell Ceph OSD 守护进程会检查彼此的心跳并向 Ceph Mon节点报告状态信息。 Ceph OSD 守护进程会将同一组PG中所有对象的状态进行一致性同步，同步异常往往会自动解决 ``` 数据一致性 ```powershell 为了保证Ceph 存储集群的数据安全向，它被设计为存储至少两个对象副本，以便它可以在保持数据安全的同时继续在某个状态下运行。 [root@mon01 ~]# ceph daemon osd.0 config show | grep default_size "osd_pool_default_size": "3", [root@mon01 ~]# ceph osd pool get ecpool size size: 3 所以PG在关联的时候，往往会关联多个OSD的id值，我们会将同一个PG关联的多个OSD集合称为 Acting Set，注意该Set是一个有序集合。其中第一个OSD，我们将其称为 Primary OSD，然后依次类推为 Secondary OSD等等。注意Primary OSD守护进程会维护同一组PG的所有OSD副本数据状态的一致性。 ``` ```powershell 对于Acting Set的 Ceph OSD 守护进程状态主要有四种： UP(启动已运行)、Down(关闭未运行)、In(集群中)、Out(集群外), 我们可以通过 ceph -s、ceph osd tree、ceph osd stat来查看OSD的状态信息 [root@mon01 ~]# ceph osd stat 6 osds: 6 up (since 14h), 6 in (since 14h); epoch: e631 注意： OSD和PG上的每一次状态变更的历史信息，我们称为epoch ``` ![image-20220930111350189](../../img/kubernetes/kubernetes_storage_ceph/image-20220930111350189.png) **小结** ``` ``` ### 1.2.2 归置组 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** PG简介 ```powershell 当用户在Ceph存储集群中创建存储池Pool的时候，我们往往会为它创建PG和PGS，如果我们没有指定PG和PGP的话，则Ceph使用配置文件中的默认值来创建Pool的PG和PGP。通常情况下，我们建议用户根据实际情况在配置文件中自定义pool的对象副本数量和PG数目。 参考资料： https://docs.ceph.com/en/latest/rados/configuration/pool-pg-config-ref/ ``` ```powershell 关于对象副本数目，用户可以根据自身对于数据安全性的要求程度进行设置，Ceph默认存储一份主数据对象和两个副本数据(osd pool default size = 3)。对于PG数目，假如数据对象副本数目为N，集群OSD数量为M，则每个OSD上的PG数量为X，官方提供了一个默认的PG数量计算公式。 PG|PGP 数量 = M * X / N 注意： 官方推荐X默认的值为100 再一个，PG算出来的数据往往不是一个整数，但是我们往往将该值取为最接近2的幂次方值。 ``` ```powershell PG数量计数： 假如Ceph集群有100个OSD，数据副本为3，则PG数量值为 100*100/3=3333.33,该值不是一个整数，我们将该值取为4096(2^12),所以我们可以将PG相关的属性设置为 [global] osd pool default size = 4 osd pool default min size = 1 osd pool default pg mum = 4096 osd pool default pgp mum = 4096 常见数量统计： OSD<5个，pg_num 设置为 256；5个 pg_num 256 [cephadm@admin cephadm-cluster]$ ceph osd pool set mypool pgp_num 256 set pool 7 pgp_num to 256 ``` ```powershell 获取完整的PG的统计信息 [cephadm@admin cephadm-cluster]$ ceph pg dump [cephadm@admin cephadm-cluster]$ ceph pg dump --format=json [cephadm@admin cephadm-cluster]$ ceph pg dump --format=json-pretty ``` ```powershell 获取精简的PG统计信息 [cephadm@admin cephadm-cluster]$ ceph -s | grep pgs pools: 1 pools, 256 pgs pgs: 256 active+clean [cephadm@admin cephadm-cluster]$ ceph pg dump_stuck ok 查看所有PG的状态 [cephadm@admin cephadm-cluster]$ ceph pg stat 256 pgs: 256 active+clean; 0 B data, 122 MiB used, 114 GiB / 120 GiB avail ``` ```powershell 查看指定PG值的统计信息 [cephadm@admin cephadm-cluster]$ ceph pg 7.c4 query ``` 状态 ```powershell 检查集群状态时(ceph -s)， Ceph 会报告归置组状态,其最优状态为 active+clean。其他常见状态有： Creating Ceph 仍在创建归置组。 Active Ceph 可处理到归置组的请求。 Clean Ceph 把归置组内的对象复制了规定次数。 Down 包含必备数据的副本挂了，所以归置组离线。 Replay 某 OSD 崩溃后，归置组在等待客户端重放操作。 Splitting Ceph 正在把一归置组分割为多个。（实现了？） Scrubbing Ceph 正在检查归置组的一致性。 Degraded 归置组内的对象还没复制到规定次数。 Inconsistent Ceph 检测到了归置组内一或多个副本间不一致现象。 Peering 归置组正在互联。 Repair Ceph 正在检查归置组、并试图修复发现的不一致（如果可能的话）。 Recovering Ceph 正在迁移/同步对象及其副本。 Backfill Ceph 正在扫描并同步整个归置组的内容，Backfill 是恢复的一种特殊情况。 Wait-backfill 归置组正在排队，等候回填。 Backfill-toofull 一回填操作在等待，因为目标 OSD 使用率超过了占满率。 Incomplete Ceph 探测到某一归置组异常。 Stale 归置组处于一种未知状态，从归置组运行图变更起就没再收到它的更新。 Remapped 归置组被临时映射到了另外一组 OSD ，它们不是 CRUSH 算法指定的。 Undersized 此归置组的副本数小于配置的存储池副本水平。 Peered 此归置组已互联，因为副本数没有达到标准，不能向客户端提供服务 异常状态标识 Inactive 归置组不能处理读写，因为它们在等待一个有最新数据的 OSD 复活且进入集群。 Unclean 归置组含有复制数未达到期望数量的对象，它们应该在恢复中。 Stale 归置组处于未知状态：存储它们的 OSD 有段时间没向监视器报告了。 ``` **简单实践** 获取特殊状态的PG ```powershell [cephadm@admin cephadm-cluster]$ ceph pg dump_stuck stale ok 注意： 其他的异常状态有：inactive|unclean|stale|undersized|degraded ``` 列举不一致pg信息 ```powershell 列出不一致的PG： [cephadm@admin cephadm-cluster]$ rados list-inconsistent-pg mypool [] 列出不一致的 rados 对象： [cephadm@admin cephadm-cluster]$ rados list-inconsistent-obj 7.c4 {"epoch":156,"inconsistents":[]} 列出给定置放群组中不一致的快照集： [cephadm@admin cephadm-cluster]$ rados list-inconsistent-snapset 7.c4 {"epoch":156,"inconsistents":[]} ``` 修复损坏pg信息 ```powershell [cephadm@admin cephadm-cluster]$ ceph pg repair 7.c4 instructing pg 7.c4 on osd.4 to repair ``` 临时PG ```powershell 假设一个PG的acting set为[0,1,2]列表。此时如果osd0出现故障，导致CRUSH算法重新分配该PG的acting set为[3,1,2]。此时osd3为该PG的主OSD，但是OSD3为新加入的OSD，并不能负担该PG上的读操作。所以PG向Monitor申请一个临时的PG，osd1为临时的主OSD，这是up set变为[1,3,2]，acting set依然为[3,1,2]，导致acting set和up set不同。当osd3完成Backfill过程之后，临时PG被取消，该PG的up set修复为acting set，此时acting set和up set都是[3,1,2]列表。 ``` **小结** ``` ``` ### 1.2.3 运行图 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** map简介 ```powershell 对于Ceph集群来说，有个非常重要的特点就是高性能，而高性能有一个非常突出的特点就是单位时间内处理业务数据的量。Ceph为了实现业务数据的精准高效操作，它主要是通过五种Map分别来实现特定的功能： Monitor Map Mon节点和所有节点的连接信息，包括ceph集群ID，monitor 节点名称，IP地址和端口等。 CRUSH Map 让 Ceph 在正常运行情况下进行高效数据操作的重要支撑部分，包括数据的写入和查询用到的设备列表、存储桶信息、故障域结构，故障域规则等。 OSD Map 保存OSD的基本信息，包括ID，状态，副本、PG、OSD信息等，便于数据的均衡性操作。 MDS Map 保存MDS的基本信息，包括版本号、创建和修改时间、数据和元数据存储池、数量、MDS状态等。 PG Map 保存PG的基本信息，包括PG的ID、数量、状态、版本号、时间戳、容量百分比等。 ``` ![image-20220930074909586](../../img/kubernetes/kubernetes_storage_ceph/image-20220930074909586.png) **简单实践** 基本map关联关系 ```powershell 查看mon相关的关联关系 [cephadm@admin cephadm-cluster]$ ceph mon dump epoch 3 fsid d932ded6-3765-47c1-b0dc-e6957051e31a ... 2: [v2:10.0.0.15:3300/0,v1:10.0.0.15:6789/0] mon.mon03 dumped monmap epoch 3 ``` ```powershell 查看osd相关信息 [cephadm@admin cephadm-cluster]$ ceph osd dump epoch 158 fsid d932ded6-3765-47c1-b0dc-e6957051e31a ... osd.5 up in weight 1 up_from 146 up_thru 156 down_at 145 last_clean_interval [125,142) [v2:10.0.0.15:6800/1274,v1:10.0.0.15:6805/1274] [v2:192.168.8.15:6804/1274,v1:192.168.8.15:6805/1274] exists,up 67282205-0c58-49c8-9af6-878198d05f2e ``` ```powershell 查看mds相关信息 [cephadm@admin cephadm-cluster]$ ceph mds dump ``` ```powershell 查看crush相关信息 [cephadm@admin cephadm-cluster]$ ceph osd crush dump { "devices": [ # 设备列表信息 { "id": 0, "name": "osd.0", "class": "hdd" }, ... ], "types": [ # 资源类型列表12类，主要有 { # osd、host、chassis、rack "type_id": 0, # row、pdu、pod、room "name": "osd" # datacenter、zone、region、root }, ... ], "buckets": [ # 存储桶列表 { "id": -1, "name": "default", ... }, ... ], "rules": [ # 数据映射规则列表 { "rule_id": 0, "rule_name": "replicated_rule", ... ], "tunables": { # 可调节的相关属性 "choose_local_tries": 0, ... }, "choose_args": {} # 可选择的其他参数 } ``` 查看PG的相关信息 ```powershell 查看pg相关的信息 [cephadm@admin cephadm-cluster]$ ceph pg dump version 3481 stamp 2062-10-08 20:54:41.007647 last_osdmap_epoch 0 last_pg_scan 0 ... OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 5 20 MiB 19 GiB 1.0 GiB 20 GiB [0,1,2,3,4] 137 44 ... sum 122 MiB 114 GiB 6.1 GiB 120 GiB ... ``` ```powershell 查看PG-OSD关系图 [cephadm@admin cephadm-cluster]$ ceph pg map 7.c4 osdmap e158 pg 7.c4 (7.c4) -> up [4,2,1] acting [4,2,1] ``` ```powershell 提交文件到对应的osd里面 [cephadm@admin ceph-cluster]$ rados put ceph-file /home/cephadm/ceph-cluster/ceph.conf --pool=mypool object-PG-OSD关系图 查看ceph-file文件对象的内部属性关系 [cephadm@admin ceph-cluster]$ ceph osd map mypool ceph-file osdmap e51 pool 'mypool' (1) object 'ceph-file' -> pg 1.7753490d (1.d) -> up ([2,1,5], p2) acting ([2,1,5], p2) ``` **小结** ``` ``` ### 1.2.4 CRUSH 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell CRUSH(Controlled Replication Under Scalable Hashing)是ceph的核心设计之一，它本质上是Ceph存储集群使用的一种数据分发算法，类似于OpenStack的Swift和AQS的对象存储所使用的哈希和一致性hash数据分布算法。 CRUSH算法通过接受多维参数，通过一定的计算对客户端对象数据进行分布存储位置的确定，来解决数据动态分发的问题。因此ceph客户端无需经过传统查表的方式来获取数据的索引，进而根据索引来读写数据，只需通过crush算法计算后直接和对应的OSD交互进行数据读写。这样，ceph就避免了查表这种传统中心化架构存在的单点故障、性能瓶颈以及不易扩展的缺陷。这也是Ceph相较于其他分布式存储系统具有高扩展性、高可用和高性能特点的主要原因。 ``` CRUSH Map简介 ```powershell Ceph中的寻址至少要经历以下三次映射： File 和 object 映射：文件数据object的数据块切片操作，便于多数据的并行化处理。 Object 和 PG 映射：将文件数据切分后的每一个Object通过简单的 Hash 算法归到一个 PG 中。 PG 和 OSD 映射：将PG映射到主机实际的OSD数据磁盘上。 ``` ```powershell CRUSH算法提供了配置和更改和数据动态再平衡等关键特性，而CRUSH算法存储数据对象的过程可通过CRUSH Map控制并进行自定义修改，CRUSH Map是Ceph集群物理拓扑结构、副本策略以及故障域等信息抽象配置段，借助于CRUSH Map可以将数据伪随机地分布到集群的各个OSD上。 ``` ![image-20221008220724118](../../img/kubernetes/kubernetes_storage_ceph/image-20221008220724118.png) ```powershell CRUSH Map 由不同层次的逻辑Buckets 和 Devices组成： Buckets - Root指的是多区域，datacenter是数据中心，room是机房、rack是机柜，host是主机 Devices - 主要指各种OSD存储设备 注意： 对于每一个Ceph集群来说，CRUSH Map在正式上线前已经确定好了，如果用户需要自定义更改CRUSH Map的话，必须在集群上线前进行更改和核实，然后应用到CRUSH算法中。 ``` Buckets ```powershell CRUSH Map中的Buckets是用户自定义增加的，每个层级的Bucket对应不同的故障域，在实际应用中，为了更加精细化地隔离故障域，用户还可以增加PDU、POD、ROW、CHASSIS等，这些名称是用户随意定义的。 对于Ceph N版本来说，它默认声明了12种Buckets。 root-根分区、region-可用区域、zone-数据区域、datacenter-数据中心 room-机房、pod-机房单间、pdu-电源插座、row-机柜排 rack-机柜、chassis-机箱、host-主机、osd-磁盘 ``` 配置文件解读 ```powershell # begin crush map 设定修正bug、优化算法、以及向后兼容老版本等属性信息 tunable choose_local_tries 0 # 为做向后兼容应保持为0 tunable choose_local_fallback_tries 0 # 为做向后兼容应保持为0 tunable choose_total_tries 50 # 选择bucket的最大重试次数 tunable chooseleaf_descend_once 1 # 为做向后兼容应保持为1 tunable chooseleaf_vary_r 1 # 修复旧bug，为做向后兼容应保持为1 tunable chooseleaf_stable 1 # 避免不必要的pg迁移，为做向后兼容应保持为1 tunable straw_calc_version 1 # straw算法版本，为做向后兼容应保持为1 tunable allowed_bucket_algs 54 # 允许使用的bucket选择算法，通过位运算计算得出的值 # devices 该部分保存了 Ceph 集群中所有 OSD 设备和 ceph-osd 守护进程的映射关系。 # 格式： device {num} {osd.name} [class {class}] device 0 osd.0 class hdd ... device 5 osd.5 class hdd # types 该部分定义了在 CRUSH 层次结构中用到的 buckets 类型。 # 格式：type {num} {bucket-name} type 0 osd # OSD守护进程编号（如：osd.1,osd.2等） type 1 host # OSD所在主机名称 type 2 chassis # host所在机箱名称 type 3 rack # 机箱所在机柜名称 type 4 row # 机柜所在排名称 type 5 pdu # 机柜排所在的电源插座 type 6 pod # 电源插座专属的单间 type 7 room # 房间所属的机房 type 8 datacenter # 机房所属的数据中心 type 9 zone # 数据中心所属的数据区域 type 10 region # 数据区域所属的可用区域 type 11 root # 设备管理的根路径 # buckets 该部分定义了一个个具体的type类型的设备区域 host mon01 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 0.039 alg straw2 # straw2算法减少了集群发生了改变后的数据移动 hash 0 # bucket使用的hash算法，默认是rjenkins1 item osd.0 weight 0.019 # 低一层级的bucket名称，以及其对应的weight item osd.1 weight 0.019 } ... # rules 部分定义了存储池的属性，以及存储池中数据的存放方式，尤其是复制(replication)和放置(placement)数据的策略。默认的 CRUSH map 包含了适用于默认存储池 rbd 的一条规则。 rule replicated_rule { id 0 # 定制所属规则集 type replicated # 作用副本存储池范围 min_size 1 # 副本少于1个，规则失效 max_size 10 # 副本大于10个，规则失效 step take default # 作用于default类型的bucket step chooseleaf firstn 0 type host # 作用于包含3个子bucket的host step emit # 表示数据处理的方式，处理完数据后，清理处理过程 } # end crush map ``` ```powershell bucket 实例的属性解析 [bucket类型] [bucket名称] { id [负整数表示bucket唯一ID] weight [表示设备容量之间的权重差异，权重1.00代表1T容量，权重0.5代表500G容量] alg [bucket类型的算法选择: uniform|list|tree|straw|straw2，它是性能和效率之间的一种妥协] hash [hash算法类型: 0 是默认的rjenkins1算法] item [当前bucket的子项元素] weight [相对权重值] } 算法解读： uniform 每次设备变动，数据都会进行均衡处理，要求所有设备的权重一致，效率极低 list 设备的变动场景以链表方式实现，大规模设备场景下，效率极其低下。 tree 设备的变动场景以二叉树方式实现，综合数据处理性能叫均衡。 straw 设备的变动场景在list和tree之间选择最优方式进行，同时支持副本放置时公平竞争处理和二级权重处理 straw2 进阶版的straw，即使出现数据变动，不会影响到跨高层的数据操作。 ``` ```powershell rule 实例的属性解析 rule { ruleset <当前规则所属的规则集，整数标识> type [指定rule作用的存储池类型 replicated|erasure] min_size <存储池的副本份数小于min_size值后，rule不起作用> max_size <存储池的副本份数大于max_size值后，rule不起作用> step take <获取一个 bucket类型名称，开始遍历其结构> step [choose|chooseleaf] [firstn|indep] step emit } 注意： choose：选择到预期数量和类型的bucket即可结束 chooseleaf：选择到预期数量和类型的bucket，并最终从这些bucket中选出叶子节点 --- 当出现osd 异常情况时， 副本池选择firstn方式，表示新节点以追加方式追加到osd列表 纠删码池选择indep方式，表示新节点以原位置插入添加到osd列表，其他保持有序状态 --- 如果 num==0，选择 N 个 bucket。 如果 num > 0 并且 num 1，2，4，5，6。 在indep模式下： CRUSH算法发现3其down掉后，会在3位置插入一个新的未down的OSD 6，OSD变迁过程为： 1，2，3，4，5 -> 1，2，6，4，5。 ``` **简单实践** Crush map操作步骤解读 ```powershell crush 相关的信息，我们可以通过两种方法来进行操作： 1 获取crush相关信息 ceph osd crush dump 2 操作crush相关信息 获取crush map信息后进行格式转换，编辑文件后再次应用crush map数据 ``` 操作crush信息 ```powershell 1、从monitor节点上获取CRUSH map [cephadm@admin cephadm-cluster]$ ceph osd getcrushmap -o crushmap_file 23 默认获取的文件并不是普通的文本文件，无法直接查看 [cephadm@admin cephadm-cluster]$ file crushmap_file crushmap_file: MS Windows icon resource - 8 icons, 1-colors ``` ```powershell 2、获取该crushmap文件后，编译为可读文件 [cephadm@admin cephadm-cluster]$ crushtool -d crushmap_file -o crushmap_file.txt [cephadm@admin cephadm-cluster]$ file crushmap_file.txt crushmap_file.txt: ASCII text ``` ```powershell 查看文件内容 [cephadm@admin cephadm-cluster]$ cat crushmap_file.txt ... # 参考上述的文件格式解读 ``` crush map信息修改 ```powershell 修改crushmap_file文件内容 [cephadm@admin cephadm-cluster]$ vim crushmap_file.txt ... rule replicated_rule { ... max_size 20 # 修改该值 ... } 3、将修改后的crushmap_file.txt编译为二进制文件 [cephadm@admin cephadm-cluster]$ crushtool -c crushmap_file.txt -o new_crushmap_file.txt ``` ```powershell 4、将新的crushmap注入到ceph集群 [cephadm@admin cephadm-cluster]$ ceph osd setcrushmap -i new_crushmap_file.txt 24 确认效果 [cephadm@admin cephadm-cluster]$ ceph osd crush dump | grep max_size "max_size": 20, [cephadm@admin cephadm-cluster]$ ceph osd crush rule dump | grep max_size "max_size": 20, 结果显示： crush map的数据修改成功了 ``` **小结** ``` ``` ### 1.2.5 实践解读 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 案例需求 ```powershell 随着存储技术的发展，目前存储平台中的存储介质的类型也越来越多了，目前主要有两大类型：SSD磁盘和SAS|SATA磁盘。我们可以根据应用对于场景的使用特点，高性能场景的数据存储使用SSD磁盘，而普通数据的存储我们采用SAS磁盘，所以在SSD场景中，我们就可以基于SSD磁盘组成高性能POOL，将基于SAS|SATA磁盘组成常规POOL。 以OpenStack场景为例，对于VM实例来说，Nova对于实时数据IO要求较高，所以推荐使用SSD存储池；VM实例创建过程中不高的冷数据，比如Glance镜像数据和Cinder块设备备份数据，推荐使用SAS|SATA的常规POOL。 ``` 场景规划 ![image-20221009234113283](../../img/kubernetes/kubernetes_storage_ceph/image-20221009234113283.png) ```powershell 为了区分SSD和SAS磁盘，需要在CRUSH Map中增加Root层，增加SAS和SSD区域。 业务A对性能要求较高，将SSD作为数据盘，需创建3副本的SSD存储池 业务B对性能要求不高，但数据量较大，将SAS作为数据盘降低成本，需创建3副本的SAS存储池 ``` **简单实践** 定制crush map ```powershell 定制专属文件 [cephadm@admin cephadm-cluster]$ cat crushmap_file_case.txt # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class ssd device 1 osd.1 class sas device 2 osd.2 class ssd device 3 osd.3 class sas device 4 osd.4 class ssd device 5 osd.5 class sas # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host mon01-ssd { id -3 # do not change unnecessarily id -4 class ssd # do not change unnecessarily # weight 0.039 alg straw2 hash 0 # rjenkins1 item osd.0 weight 0.019 } host mon02-ssd { id -5 # do not change unnecessarily id -6 class ssd # do not change unnecessarily # weight 0.039 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.019 } host mon03-ssd { id -7 # do not change unnecessarily id -8 class ssd # do not change unnecessarily # weight 0.039 alg straw2 hash 0 # rjenkins1 item osd.4 weight 0.019 } host mon01-sas { id -9 class sas alg straw2 hash 0 item osd.1 weight 0.019 } host mon02-sas { id -10 class sas alg straw2 hash 0 item osd.3 weight 0.019 } host mon03-sas { id -11 class sas alg straw2 hash 0 item osd.5 weight 0.019 } root ssd { id -1 id -2 class ssd alg straw2 hash 0 item mon01-ssd weight 0.019 item mon02-ssd weight 0.019 item mon03-ssd weight 0.019 } root sas { id -127 id -128 class sas alg straw2 hash 0 item mon01-sas weight 0.019 item mon02-sas weight 0.019 item mon03-sas weight 0.019 } # rules rule ssd_rule { id 0 type replicated min_size 1 max_size 10 step take ssd step chooseleaf firstn 0 type host step emit } rule sas_rule { id 1 type replicated min_size 1 max_size 10 step take sas step chooseleaf firstn 0 type host step emit } # end crush map 注意： 每个元素都应该有自己的id值 原则上来说，每个bucket名称最好不要一致，即使是同一台主机 ``` ```powershell 3、将修改后的crushmap_file.txt编译为二进制文件 [cephadm@admin cephadm-cluster]$ crushtool -c crushmap_file_case.txt -o crushmap_file_case ``` ```powershell 4、将新的crushmap注入到ceph集群 [cephadm@admin cephadm-cluster]$ ceph osd setcrushmap -i crushmap_file_case 25 ``` ```powershell 确认效果 [cephadm@admin cephadm-cluster]$ ceph osd crush dump ... [cephadm@admin cephadm-cluster]$ ceph osd crush rule dump ... 结果显示： crush map的数据修改成功了 ``` ```powershell 确认osd的状态树 [cephadm@admin cephadm-cluster]$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -127 0.05699 root sas -12 0.01900 host mon01-sas 1 sas 0.01900 osd.1 up 1.00000 1.00000 -13 0.01900 host mon02-sas 3 sas 0.01900 osd.3 up 1.00000 1.00000 -14 0.01900 host mon03-sas 5 sas 0.01900 osd.5 up 1.00000 1.00000 -1 0.05699 root ssd -3 0.01900 host mon01-ssd 0 ssd 0.01900 osd.0 up 1.00000 1.00000 -5 0.01900 host mon02-ssd 2 ssd 0.01900 osd.2 up 1.00000 1.00000 -7 0.01900 host mon03-ssd 4 ssd 0.01900 osd.4 up 1.00000 1.00000 ``` ```powershell 创建ssd存储池 [cephadm@admin cephadm-cluster]$ ceph osd pool create ssd_pool 16 16 replicated ssd_rule pool 'ssd_pool' created 创建sas存储池 [cephadm@admin cephadm-cluster]$ ceph osd pool create sas_pool 16 16 replicated sas_rule pool 'sas_pool' created ``` ```powershell 确认不同的pool所使用的osd效果 [cephadm@admin cephadm-cluster]$ ceph pg ls-by-pool ssd_pool |awk '{print $1,$2,$15}' PG OBJECTS ACTING 15.0 0 [4,0,2]p4 ... 15.f 0 [2,4,0]p2 [cephadm@admin cephadm-cluster]$ ceph pg ls-by-pool sas_pool |awk '{print $1,$2,$15}' PG OBJECTS ACTING 16.0 0 [3,5,1]p3 ... 16.f 0 [5,3,1]p5 ``` **小结** ```powershell ``` ## 1.3 可视化 ### 1.3.1 dashboard 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell Ceph dashboard 是通过一个web界面，对已经运行的ceph集群进行状态查看以及功能配置等功能，早起ceph使用的是第三方的dashboard组件。 关于dashboard是通过模块的方式来进行加载的，而且默认情况下，该模块是具备输出所有ceph集群状态的一个模块，因为这里面涉及到某些敏感信息，所以默认情况下，使用https协议来进行访问。 ``` ```powershell 参考资料： https://docs.ceph.com/en/latest/mgr/dashboard/ ``` 常见 监控模块工具 | 工具 | 解析 | | --------- | ------------------------------------------------------------ | | calamari | 对外提供了十分漂亮的web管理和监控界面，以及一套改进的REST API接口，在一定程度上简化了ceph管理，最初calamari是作为lnktank公司的ceph企业级商业产品来销售，红帽2015年收购后为了更好地推动ceph的发展，对外宣布calamari开源 | | VSM | Virtual Storage Manager是Inter公司研发并且开源的一款ceph集群管理和监控软件，简化了一些ceph集群部署的一些步骤，可以简单的通过web页面来操作 | | Inksope | Inksope是一个ceph的管理和监控系统，依赖于ceph提供的API，使用MongoDB来存储实时的监控数据和历史信息 | | dashboard | 是用python开发的一个ceph的监控面板，用来监控ceph的运行状态。同时提供REST API来访问状态数据，该插件必须安装在mgr节点上。 | 查看模块相关的命令 ```powershell 查与模块相关的功能 [cephadm@admin cephadm-cluster]$ ceph mgr --help | grep module mgr module disable disable mgr module mgr module enable {--force} enable mgr module mgr module ls list active mgr modules ... ``` ```powershell 查看模块功能 [cephadm@admin cephadm-cluster]$ ceph mgr module ls { "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator_cli", "progress", "rbd_support", "status", "volumes" ], "enabled_modules": [ "iostat", "restful" ], "disabled_modules": { "name": "alerts", ... ``` 部署dashboard模块 ```powershell 默认情况下，ceph是没有dashboard模块的 [cephadm@admin cephadm-cluster]$ ceph mgr module ls | grep dashboard [cephadm@admin cephadm-cluster]$ ``` ```powershell 在所有的mgr节点上部署dashoard模块 [cephadm@admin cephadm-cluster]$ for i in mon01 mon02 > do > ssh cephadm@$i "sudo yum install ceph-mgr-dashboard -y" > done ``` ```powershell 确认ceph集群是否启动了dashboard的模块 [cephadm@admin cephadm-cluster]$ ceph mgr module ls | grep dashboard "name": "dashboard", ``` **简单实践** 启用dashboard ```powershell 启用dashboard模块 [cephadm@admin cephadm-cluster]$ ceph mgr module enable dashboard 查看已经启用的功能模块 [cephadm@admin cephadm-cluster]$ ceph mgr module ls | grep -A4 "enabled_modules" "enabled_modules": [ "dashboard", "iostat", "restful" ], ``` 查看状态 ```powershell 关闭dashboard的tls功能 [cephadm@admin cephadm-cluster]$ ceph config set mgr mgr/dashboard/ssl false ``` ```powershell 确认dashboard的服务效果 [cephadm@admin cephadm-cluster]$ ceph mgr services { "dashboard": "http://stor01.superopsmsb.com:8080/" } 查看服务器启动端口 [cephadm@admin cephadm-cluster]$ for i in mon01 mon02; do ssh cephadm@$i "sudo netstat -tnulp | grep 8080"; done tcp6 0 0 :::8080 :::* LISTEN 1963/ceph-mgr tcp6 0 0 :::8080 :::* LISTEN 1903/ceph-mgr ``` 配置模块 ```powershell 配置dashboard监听的mon节点地址和端⼝： [cephadm@admin ceph-cluster]$ ceph config set mgr mgr/dashboard/mon01/server_addr 10.0.0.13 [cephadm@admin ceph-cluster]$ ceph config set mgr mgr/dashboard/mon01/server_port 8080 [cephadm@admin ceph-cluster]$ ceph config set mgr mgr/dashboard/mon02/server_addr 10.0.0.14 [cephadm@admin ceph-cluster]$ ceph config set mgr mgr/dashboard/mon02/server_port 8080 ``` ```powershell 重启mgr服务 [root@mon01 ~]# systemctl restart ceph-mgr.target [root@mon02 ~]# systemctl restart ceph-mgr.target ``` ```powershell 再次确认监听端口的效果 [cephadm@admin cephadm-cluster]$ for i in mon01 mon02; do ssh cephadm@$i "sudo netstat -tnulp | grep 8080"; done tcp 0 0 10.0.0.13:8080 0.0.0.0:* LISTEN 2549/ceph-mgr tcp 0 0 10.0.0.14:8080 0.0.0.0:* LISTEN 2211/ceph-mgr ``` ```powershell 浏览器访问：10.0.0.13:8080 或者 10.0.0.14:8080 查看效果 ``` ![image-20221013195313762](../../img/kubernetes/kubernetes_storage_ceph/image-20221013195313762.png) 设定登录密码 ```powershell 设定登录的用户名和密码 [cephadm@admin ceph-cluster]$ echo "12345678" >> dashboard_passwd.txt [cephadm@admin ceph-cluster]$ ceph dashboard set-login-credentials admin -i dashboard_passwd.txt ****************************************************************** *** WARNING: this command is deprecated. *** *** Please use the ac-user-* related commands to manage users. *** ****************************************************************** Username and password updated ``` ```powershell 使用用户(admin)和密码(12345678),浏览器登录dashboard的效果 ``` ![image-20221013195513390](../../img/kubernetes/kubernetes_storage_ceph/image-20221013195513390.png) **小结** ``` ``` ### 1.3.2 tls实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 如果我们希望ceph具有更加安全的访问能力的话，我们可以为dashboard能力提供tls能力。对于ceph来说，它的tls能力主要有两种方式： - 使用默认的tls能力 - 使用自签证书实现tls能力 ``` ```powershell 注意： 对于tls能力来说，我们需要提前对于ceph启用tls的功能 ceph config set mgr mgr/dashboard/ssl true ``` 基本步骤 ```powershell 方法1：使用默认的tls能力 ceph dashboard create-self-signed-cert 方法2：使用自签证书实现tls能力 openssl req -new -nodes -x509 -subj "/O=IT/CN=ceph.superopsmsb.com" -days 3650 -keyout dashboard.key -out dashboard.crt -extensions v3_ca 配置dashboard加载证书 ceph config-key set mgr mgr/dashboard/crt -i dashboard.crt ceph config-key set mgr mgr/dashboard/key -i dashboard.key ``` **简单实践** 使用自动生成证书 ```powershell 重启模块 [cephadm@admin ceph-cluster]$ mkdir tls && cd tls [cephadm@admin tls]$ ceph mgr module disable dashboard [cephadm@admin tls]$ ceph mgr module enable dashboard 启用tls服务 [cephadm@admin tls]$ ceph config set mgr mgr/dashboard/ssl true ``` ```powershell 生成自签证书 [cephadm@admin tls]$ ceph dashboard create-self-signed-cert Self-signed certificate created 创建 web 登录用户密码 [cephadm@admin tls]$ echo "12345678" >> dashboard_passwd.txt [cephadm@admin tls]$ ceph dashboard set-login-credentials admin -i dashboard_passwd.txt ``` ```powershell 重启mgr服务 [root@mon01 ~]# systemctl restart ceph-mgr.target [root@mon02 ~]# systemctl restart ceph-mgr.target ``` ```powershell 再次确认监听端口的效果 [cephadm@admin tls]$ for i in mon01 mon02; do ssh cephadm@$i "sudo netstat -tnulp | grep ceph-mgr"; done ... tcp 0 0 10.0.0.13:8443 0.0.0.0:* LISTEN 3093/ceph-mgr tcp 0 0 10.0.0.14:8443 0.0.0.0:* LISTEN 2491/ceph-mgr ``` ```powershell 使用用户(admin)和密码(12345678),浏览器访问：https://10.0.0.13:8443 或者 https://10.0.0.14:8443 查看效果 ``` ![image-20221013201943746](../../img/kubernetes/kubernetes_storage_ceph/image-20221013201943746.png) 使用自签名证书 ```powershell 重启模块 [cephadm@admin tls]$ ceph mgr module disable dashboard [cephadm@admin tls]$ ceph mgr module enable dashboard ``` ```powershell 生成自签证书 [cephadm@admin tls]$ openssl req -new -nodes -x509 -subj "/O=IT/CN=ceph.superopsmsb.com" -days 3650 -keyout dashboard.key -out dashboard.crt -extensions v3_ca Generating a 2048 bit RSA private key .......+++ ...................+++ writing new private key to 'dashboard.key' ----- 查看效果 [cephadm@admin tls]$ ls dashboard.crt dashboard.key dashboard_passwd.txt ``` ```powershell 应用自签证书 [cephadm@admin tls]$ ceph config-key set mgr mgr/dashboard/crt -i dashboard.crt [cephadm@admin tls]$ ceph config-key set mgr mgr/dashboard/key -i dashboard.key 创建 web 登录用户密码 [cephadm@admin tls]$ echo "12345678" >> dashboard_passwd.txt [cephadm@admin tls]$ ceph dashboard set-login-credentials admin-openssl -i dashboard_passwd.txt ****************************************************************** *** WARNING: this command is deprecated. *** *** Please use the ac-user-* related commands to manage users. *** ****************************************************************** Username and password updated ``` ```powershell 宿主机hosts 文件定制 10.0.0.13 ceph.superopsmsb.com ``` ```powershell 使用用户(admin-openssl)和密码(12345678),浏览器访问 https://ceph.superopsmsb.com:8443 效果： ``` ![image-20221013203249531](../../img/kubernetes/kubernetes_storage_ceph/image-20221013203249531.png) **小结** ``` ``` ### 1.3.3 RGW实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell ceph的dashboard很多都是可以直接使用的，但是对于 rgw,cephfs,iscsi,监控等功能，需要基于手工方式启用功能。 ``` ![image-20221013204353270](../../img/kubernetes/kubernetes_storage_ceph/image-20221013204353270.png) 准备rgw环境 ```powershell 查看rgw的效果 [cephadm@admin cephadm-cluster]$ ceph -s | grep rgw rgw: 1 daemon active (stor04) ``` **简单实践** 用户准备 ```powershell 创建专属的用户信息 [cephadm@admin cephadm-cluster]$ radosgw-admin user create --uid=rgw --display-name=rgw --system { "user_id": "rgw", ... "keys": [ { "user": "rgw", "access_key": "NAS972R98010Q2HUYW1F", "secret_key": "AvQO7AepS0017A75ClRAZxkhFmzFwXxahkyQFHdX" } ], ... 确认效果 [cephadm@admin cephadm-cluster]$ radosgw-admin user info --uid=rgw ``` 为Dashboard设置access_key 和 secret_key ```powershell 注意：ceph的rgw属性定制是使用文件方式来实现 ``` ```powershell 定制access-key认证信息 [cephadm@admin cephadm-cluster]$ echo 'NAS972R98010Q2HUYW1F' > access-key.file [cephadm@admin cephadm-cluster]$ ceph dashboard set-rgw-api-access-key -i access-key.file Option RGW_API_ACCESS_KEY updated 定制secret_key认证信息 [cephadm@admin cephadm-cluster]$ echo 'AvQO7AepS0017A75ClRAZxkhFmzFwXxahkyQFHdX' > secret_key.file [cephadm@admin cephadm-cluster]$ ceph dashboard set-rgw-api-secret-key -i secret_key.file Option RGW_API_SECRET_KEY updated ``` 重启mgr服务 ```powershell [root@mon01 ~]# systemctl restart ceph-mgr.target [root@mon02 ~]# systemctl restart ceph-mgr.target ``` ```powershell 查看dashboard的RGW效果 ``` ![image-20221013210433473](../../img/kubernetes/kubernetes_storage_ceph/image-20221013210433473.png) **小结** ``` ``` ### 1.3.4 NFS实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell ceph的dashboard很多都是可以直接使用的，但是对于 rgw,nfs,iscsi,监控等功能，需要基于手工方式启用功能。自从Ceph 的J版本开始，ceph引入了 nfs-ganesha软件，ganesha通过rgw和cephfs两种方式实现ceph以nfs的方式实现外部功能访问。从Ceph Nautilus版本开始，Ceph Dashboard中可以直接支持配置这两种方式的NFS。 - FSAL_RGW 调用librgw2将NFS协议转义为S3协议再通过RGW存入到Ceph中 - FSAL_CEPH 调用libcephfs2将NFS转义为Cephfs协议再存入到Ceph 中 ``` ![image-20221013211051686](../../img/kubernetes/kubernetes_storage_ceph/image-20221013211051686.png) ```powershell 参考资料： https://docs.ceph.com/en/latest/cephfs/nfs/#nfs ``` 我们以stor04主机为ganesha节点 ```powershell 查看是否安装librgw2和libcephfs2软件包 [root@stor04 ~]# rpm -qa |grep librgw librgw2-14.2.22-0.el7.x86_64 [root@stor04 ~]# rpm -qa |grep libcephfs libcephfs2-14.2.22-0.el7.x86_64 ``` ```powershell 定制软件源 cat >> /etc/yum.repos.d/nfs-ganesha.repo<< eof [nfs-ganesha] name=nfs-ganesha baseurl=http://us-west.ceph.com/nfs-ganesha/rpm-V2.7-stable/nautilus/x86_64/ enabled=1 priority=1 eof ``` ```powershell 部署ganesha服务 [root@stor04 ~]# yum install nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rgw -y ``` ```powershell 启动服务 [root@stor04 ~]# systemctl start nfs-ganesha.service [root@stor04 ~]# systemctl status nfs-ganesha.service [root@stor04 ~]# systemctl enable nfs-ganesha.service ``` **简单实践** 定制存储池 ```powershell 新建 ganesha_data的pool，用来存放一些dashboard的nfs相关配置文件 [cephadm@admin cephadm-cluster]$ ceph osd pool create ganesha_data 16 16 pool 'ganesha_data' created ``` ```powershell 新建空的daemon.txt文本文件。 [cephadm@admin cephadm-cluster]$ touch daemon.txt 导入daemon文件到ganesha_data pool中 [cephadm@admin cephadm-cluster]$ rados -p ganesha_data put conf-stor04.localdomain daemon.txt ``` ```powershell 注意事项： 存入rados的文件名必须要是conf-格式，其中对应于运行此服务的节点名称。 后续Dashboard创建NFS后，conf-会有内容，每个conf-都包含指向NFS-Ganesha守护程序应服务的导出的RADOS URL。 格式为：%url rados://[/]/export- conf-和export-对象必须存储在同一个RADOS池/命名空间 ``` ```powershell 确认效果 [cephadm@admin cephadm-cluster]$ rados -p ganesha_data ls conf-mon02.localdomain conf-mon03.localdomain conf-mon01.localdomain ``` 确定rgw认证信息 ```powershell 查看当前Ceph节点的rgw认证信息 [cephadm@admin cephadm-cluster]$ ceph auth get client.rgw.stor04 [client.rgw.stor04] key = AQCkOEJj/uBgIRAALPAxf3NIp0EbGiCDgLegig== caps mon = "allow rw" caps osd = "allow rwx" exported keyring for client.rgw.stor04 ``` 配置ganesha配置文件 ```powershell 修改所有rgw节点上的ganesha配置文件 [root@stor04 ~]# cat /etc/ganesha/ganesha.conf ... # 定制rados的连接配置 RADOS_URLS { ceph_conf = "/etc/ceph/ceph.conf"; Userid = "admin"; watch_url = "rados://ganesha_data/conf-stor04.localdomain"; } %url rados://ganesha_data/conf-stor04.localdomain # 定制rgw的连接配置 RGW { ceph_conf = "/etc/ceph/ceph.conf"; name = "client.rgw.stor04.localdomain"; cluster = "ceph"; } ``` ```powershell 配置解析： RADOS_URLS 定制rados的连接配置 watch_url 指定rados的专属文件地址 RGW 定制rgw的连接配置 name 定制rgw的专属认证账户信息 ``` ```powershell 重启ganesha服务 [root@stor04 /etc/ceph]# systemctl restart nfs-ganesha ``` 启用NFS-Ganesha能力 ```powershell 设定dashboard的nfs配置所在位置 [cephadm@admin cephadm-cluster]$ ceph dashboard set-ganesha-clusters-rados-pool-namespace ganesha_data Option GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE updated ``` ```powershell 重启mgr服务 [root@mon01 ~]# systemctl restart ceph-mgr.target [root@mon02 ~]# systemctl restart ceph-mgr.target ``` ```powershell 重新查看Dashboard的NFS功能效果 ``` ![image-20221013222600422](../../img/kubernetes/kubernetes_storage_ceph/image-20221013222600422.png) **小结** ``` ``` ## 1.4 监控 ### 1.4.1 prom基础 学习目标 这一节，我们从 基础知识、术语解析、小结 三个方面来学习。 **基础知识** 软件简介 ```powershell Prometheus 作为生态圈 Cloud Native Computing Foundation（CNCF）中的重要一员 Prometheus 本身基于Go语言开发的一套开源的系统监控报警框架和时序列数据库(TSDB)。它启发于 Google 的 borgmon 监控系统，在一定程度上可以理解为，Google BorgMon监控系统的开源版本。 该软件由工作在 SoundCloud 的 google 前员工在 2012 年创建，作为社区开源项目进行开发，并于 2015 年正式发布。2016 年，Prometheus 正式加入 Cloud Native Computing Foundation，随着容器技术的迅速发展，Kubernetes 已然成为大家追捧的容器集群管理系统。其活跃度仅次于 Kubernetes的开源项目, 现已广泛用于容器集群的监控系统中，当然不仅限于容器集群。。 Prometheus功能更完善、更全面，性能也足够支撑上万台规模的集群。 ``` ```powershell 网站：https://prometheus.io/ github：https://github.com/prometheus 最新版本：2.39.1 / 2022-10-07 ``` prometheus的架构效果图如下： ![architecture](../../img/kubernetes/kubernetes_storage_ceph/1.1.3-2.png) ``` 从上图可以看出，Prometheus 的主要模块包括：Prometheus server, exporters, Pushgateway, PromQL, Alertmanager 以及图形界面。 ``` 其大概的工作流程是： 1 Prometheus server 定期从配置好的 jobs 或者 exporters 中拉 metrics，或者接收来自 Pushgateway 发过来的 metrics，或者从其他的 Prometheus server 中拉 metrics。 2 Prometheus server 在本地存储收集到的 metrics，并运行已定义好的 alert.rules，记录新的时间序列或者向 Alertmanager 推送警报，实现一定程度上的完全冗余功能。 3 Alertmanager 根据配置文件，对接收到的警报进行去重分组，根据路由配置，向对应主机发出告警。 4 集成Grafana或其他API作为图形界面，用于可视化收集的数据。 Prometheus 由几个主要的软件组件组成，其职责概述如下。 | 组件 | 解析 | | ------------------ | ------------------------------------------------------------ | | Prometheus Server | 彼此独立运行，仅依靠其本地存储来实现其核心功能：抓取时序数据，规则处理和警报等。 | | Client Library | 客户端库，为需要监控的服务生成相应的 metrics 并暴露给 Prometheus server。当 Prometheus server 来 pull 时，直接返回实时状态的 metrics。 | | Push Gateway | 主要用于短期的 jobs。由于这类 jobs 存在时间较短，可能在 Prometheus 来 pull 之前就消失了。为此，这次 jobs 可以直接向 Prometheus server 端推送它们的 metrics。这种方式主要用于服务层面的 metrics，对于机器层面的 metrices，需要使用 node exporter。 | | Exporters | 部署到第三方软件主机上，用于暴露已有的第三方服务的 metrics 给 Prometheus。 | | Alertmanager | 从 Prometheus server 端接收到 alerts 后，会进行去除重复数据，分组，并路由到对应的接受方式，以高效向用户完成告警信息发送。常见的接收方式有：电子邮件，pagerduty，OpsGenie, webhook 等,一些其他的工具。 | | Data Visualization | Prometheus Web UI （Prometheus Server内建），及Grafana等 | | Service Discovery | 动态发现待监控的Target，从而完成监控配置的重要组件，在容器化环境中尤为有用；该组件目前由Prometheus Server内建支持； | 在上诉的组件中，大多数都是用Go编写的，因此易于构建和部署为静态二进制文件。 **术语解析** metrics ```powershell Prometheus中存储的数据为时间序列，即基于同一度量标准或者同一时间维度的数据流。除了时间序列数据的正常存储之外，Prometheus还会基于原始数据临时生成新的时间序列数据，用于后续查询的依据或结果。 每个时间序列都由metric名称和标签(可选键值对)组合成唯一标识。 查询语言允许基于这些维度进行过滤和聚合。更改任何标签值，包括添加或删除标签，都会创建一个新的时间序列。 ``` ![image-20211013134435251](../../img/kubernetes/kubernetes_storage_ceph/image-20211013134435251.png) 数据获取 ```powershell 这些metric数据，是基于HTTP call方式来进行获取的，从对方的配置文件中指定的网络端点(endpoint)上周期性获取指标数据。每一个端点上几乎不可能只有一个数据指标。 用Prometheus术语来说，一个单独 scrape(host:port;)的目标称为instance，通常对应于单个进程。 一组同种类型的 instances集合称为jobs，主要用于保证可扩展性和可靠性。 ``` ![image-20211013111443902](../../img/kubernetes/kubernetes_storage_ceph/image-20211013111443902.png) **小结** ``` ``` ### 1.4.2 prom部署 学习目标 这一节，我们从 prometheus部署、grafana部署、exporter部署、小结 四个方面来学习。 **prometheus部署** 准备工作 ```powershell 创建软件目录 mkdir /data/{server,softs} -p cd /data/softs 注意： 以上所有的prometheus软件都去https://prometheus.io/download/网站下载。 https://github.com/prometheus/prometheus/releases/download/v2.39.1/prometheus-2.39.1.linux-amd64.tar.gz ``` ```powershell 解压软件 tar xf prometheus-2.39.1.linux-amd64.tar.gz -C /data/server/ cd /data/server ln -s /data/server/prometheus-2.39.1.linux-amd64 /data/server/prometheus ``` 安装软件 ```powershell 配置准备 mkdir /data/server/prometheus/{data,cfg,logs,bin} -p cd /data/server/prometheus mv prometheus promtool bin/ mv prometheus.yml cfg/ ``` ```powershell 环境变量定制 ]# cat /etc/profile.d/prometheus.sh #!/bin/bash # set prometheus env export PROMETHEUS_HOME=/data/server/prometheus export PATH=$PATH:${PROMETHEUS_HOME}/bin 应用环境变量文件 ]# source /etc/profile.d/prometheus.sh ]# chmod +x /etc/profile.d/prometheus.sh ``` 服务定制 ```powershell 定制prometheus服务文件 ]# cat /usr/lib/systemd/system/prometheus.service [Unit] Description=prometheus server project After=network.target [Service] Type=simple ExecStart=/data/server/prometheus/bin/prometheus --config.file=/data/server/prometheus/cfg/prometheus.yml --storage.tsdb.path=/data/server/prometheus/data Restart=on-failure [Install] WantedBy=multi-user.target 配置解析： 在这里，我们需要将定制的prometheus的配置文件和数据目录作为启动参数配置好 其它的参数，我们可以基于prometheus --help 查看更多 启动服务 systemctl daemon-reload systemctl start prometheus.service systemctl status prometheus.service systemctl enable prometheus.service ``` ```powershell 检查效果 ]# netstat -tnulp | egrep 'Pro|pro' Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 :::9090 :::* LISTEN 25082/prometheus 结果显示： 可以看到当前主机上可以看到一个端口9090，我们可通过可以看到prometheus的服务页面 浏览器访问 http://10.0.0.18:9090/，查看效果 ``` ![image-20221013232913546](../../img/kubernetes/kubernetes_storage_ceph/image-20221013232913546.png) **grafana部署** 准备工作 ```powershell 创建软件目录 cd /data/softs wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.2.0-1.x86_64.rpm 注意： grafana软件从 https://grafana.com/grafana/download 下载 ``` 安装软件 ```powershell 安装软件 sudo yum install grafana-enterprise-9.2.0-1.x86_64.rpm 注意： 这里安装的是本地文件，所以要加文件路径 安装插件 grafana-cli plugins list-remote grafana-cli plugins install grafana-piechart-panel grafana-cli plugins ls 启动服务 systemctl daemon-reload systemctl start grafana-server.service systemctl status grafana-server.service systemctl enable grafana-server.service ``` ```powershell 检查效果 ]# netstat -tnulp | egrep 'Pro|gra' Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 :::3000 :::* LISTEN 25839/grafana-serve 结果显示： 当前主机上出现了一个端口3000 浏览器访问 http://10.0.0.18:3000/，查看效果 ``` ![image-20221013233232284](../../img/kubernetes/kubernetes_storage_ceph/image-20221013233232284.png) ```powershell 输入用户名和密码：admin/admin，就会进入到更改密码的页面，重置过密码后，查看效果 ``` ![image-20221013233326269](../../img/kubernetes/kubernetes_storage_ceph/image-20221013233326269.png) ```powershell 添加数据源: 点击 "Add your data source" 选择 "Prometheus" 出现添加界面 ``` ![image-20221013233442609](../../img/kubernetes/kubernetes_storage_ceph/image-20221013233442609.png) ```powershell 点击最下面的 save & test ``` **exporter部署** 准备工作 ```powershell 解压软件 cd /data/softs tar xf node_exporter-1.4.0.linux-amd64.tar.gz -C /data/server/ ``` ```powershell 配置命令 ln -s /data/server/node_exporter-1.4.0.linux-amd64 /data/server/node_exporter mkdir /data/server/node_exporter/bin cd /data/server/node_exporter mv node_exporter bin/ 配置环境变量 # cat /etc/profile.d/node_exporter.sh #!/bin/bash # set node_exporter env export PROMETHEUS_HOME=/data/server/node_exporter export PATH=$PATH:${PROMETHEUS_HOME}/bin ``` ```powershell 服务文件 ]# cat /usr/lib/systemd/system/node_exporter.service [Unit] Description=node exporter project After=network.target [Service] Type=simple ExecStart=/data/server/node_exporter/bin/node_exporter Restart=on-failure [Install] WantedBy=multi-user.target 启动服务 systemctl daemon-reload systemctl start node_exporter.service systemctl status node_exporter.service systemctl enable node_exporter.service ``` ```powershell 检查效果 ]# netstat -tnulp | egrep 'Pro|node' Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 :::9100 :::* LISTEN 24650/node_exporter 结果显示： 可以看到当前主机上可以看到一个端口9100 ``` ```powershell 访问http://10.0.0.18:9100/metrics 就可以看到node export所在节点的所有定制好的监控项。 [root@stor06 /data/server/node_exporter]# curl -s http://10.0.0.18:9100/metrics | head # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 0.000104469 go_gc_duration_seconds{quantile="0.25"} 0.000104469 ``` 环境集成 ```powershell 编辑prometheus配置文件 ]# vim /data/server/prometheus/cfg/prometheus.yml ... scrape_configs: ... - job_name: 'node_exporter' static_configs: - targets: ['10.0.0.12:9100'] 属性解析： 新增一个job_name 和 static_configs的属性 targets 就是我们在基本概念中讲到的instance，格式就是"ip:port" 重启服务 systemctl restart prometheus.service ``` ```powershell 浏览器访问 http://10.0.0.18:9090/targets 就可以看到node_exporter已经被prometheus加载到监控中了，效果如下 ``` ![image-20221013234058265](../../img/kubernetes/kubernetes_storage_ceph/image-20221013234058265.png) ```powershell 登录到Grafana界面，点击左侧边栏"+",选择"import",在dashboard的输入框中输入 https://grafana.com/dashboards/8919,查看效果 ``` ![image-20221013234413884](../../img/kubernetes/kubernetes_storage_ceph/image-20221013234413884.png) ```powershell 点击完 load后，在当前界面的下方找到数据集 prometheus，最后点击 import，效果如下 ``` ![image-20221013234455785](../../img/kubernetes/kubernetes_storage_ceph/image-20221013234455785.png) **小结** ``` ``` ### 1.4.3 ceph监控 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell Ceph Manager内置了众多模块，包括prometheus模块，⽤于直接输出Prometheus⻛格的指标数据。我们只需要开启该功能即可，Prometheus模块默认监听于TCP协议的9283端⼝。 ``` 开启ceph的prometheus模块 ````powershell 开启ceph的监听模块 [cephadm@admin ceph-cluster]$ ceph mgr module enable prometheus 查看模块效果 [cephadm@admin ceph-cluster]$ ceph mgr module ls { ... "enabled_modules": [ "dashboard", "iostat", "nfs", "prometheus", "restful" ], 检查mgr的暴露端口 root@mon02:~# netstat -tnulp | grep mgr tcp 0 0 10.0.0.14:8443 0.0.0.0:* LISTEN 8111/ceph-mgr tcp6 0 0 :::9283 :::* LISTEN 8111/ceph-mgr ```` 配置prometheus的监控配置 ```powershell ]# vim /data/server/prometheus/cfg/prometheus.yml ... scrape_configs: ... - job_name: 'ceph' static_configs: - targets: ['10.0.0.13:9283'] 属性解析： 新增一个job_name 和 static_configs的属性 targets 就是我们在基本概念中讲到的instance，格式就是"ip:port" 重启服务 systemctl restart prometheus.service 浏览器访问 http://10.0.0.18:9090/targets 就可以看到node_exporter已经被prometheus加载到监控中了，效果如下 ``` ![image-20221013235206075](../../img/kubernetes/kubernetes_storage_ceph/image-20221013235206075.png) ```powershell 登录到Grafana界面，点击左侧边栏"+",选择"import",在dashboard的输入框中输入 https://grafana.com/dashboards/2842,查看效果 ``` ![image-20221013235316552](../../img/kubernetes/kubernetes_storage_ceph/image-20221013235316552.png) **小结** ``` ``` ## 1.5 k8s实践 ### 1.5.1 基础环境 学习目标 这一节，我们从 集群规划、主机认证、小结 三个方面来学习。 **集群规划** 简介 ```powershell 在这里，我们以单主分布式的主机节点效果来演示kubernetes的最新版本的集群环境部署。 ``` ![image-20220715092558666](../../img/kubernetes/kubernetes_storage_ceph/image-20220715092558666.png) 节点集群组件规划 ![image-20220715093007357](../../img/kubernetes/kubernetes_storage_ceph/image-20220715093007357.png) ```powershell master节点 kubeadm(集群环境)、kubectl(集群管理)、kubelet(节点状态) kube-apiserver、kube-controller-manager、kube-scheduler、etcd containerd(docker方式部署)、flannel(插件部署) node节点 kubeadm(集群环境)、kubelet(节点状态) kube-proxy、containerd(docker方式部署)、flannel(插件部署) ``` 主机名规划 | 序号 | 主机ip | 主机名规划 | | ---- | --------- | ---------------------------------------------------- | | 1 | 10.0.0.19 | kubernetes-master.superopsmsb.com kubernetes-master | | 2 | 10.0.0.20 | kubernetes-node1.superopsmsb.com kubernetes-node1 | | 3 | 10.0.0.21 | kubernetes-node2.superopsmsb.com kubernetes-node2 | ```powershell 修改master节点主机的hosts文件 [root@localhost ~]# cat /etc/hosts 10.0.0.19 kubernetes-master.superopsmsb.com kubernetes-master 10.0.0.20 kubernetes-node1.superopsmsb.com kubernetes-node1 10.0.0.21 kubernetes-node2.superopsmsb.com kubernetes-node2 ``` **主机认证** 简介 ```powershell 因为整个集群节点中的很多文件配置都是一样的，所以我们需要配置跨主机免密码认证方式来定制集群的认证通信机制，这样后续在批量操作命令的时候，就非常轻松了。 ``` 脚本内容 ```powershell [root@localhost ~]# cat /data/scripts/01_remote_host_auth.sh #!/bin/bash # 功能: 批量设定远程主机免密码认证 # 版本: v0.2 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 user_dir='/root' host_file='/etc/hosts' login_user='root' login_pass='123456' target_type=(部署 免密 同步 主机名 退出) # 菜单 menu(){ echo -e "\e[31m批量设定远程主机免密码认证管理界面\e[0m" echo "=====================================================" echo -e "\e[32m 1: 部署环境 2: 免密认证 3: 同步hosts \e[0m" echo -e "\e[32m 4: 设定主机名 5：退出操作 \e[0m" echo "=====================================================" } # expect环境 expect_install(){ if [ -f /usr/bin/expect ] then echo -e "\e[33mexpect环境已经部署完毕\e[0m" else yum install expect -y >> /dev/null 2>&1 && echo -e "\e[33mexpect软件安装完毕\e[0m" || (echo -e "\e[33mexpect软件安装失败\e[0m" && exit) fi } # 秘钥文件生成环境 create_authkey(){ # 保证历史文件清空 [ -d ${user_dir}/.ssh ] && rm -rf ${user_dir}/.ssh/* # 构建秘钥文件对 /usr/bin/ssh-keygen -t rsa -P "" -f ${user_dir}/.ssh/id_rsa echo -e "\e[33m秘钥文件已经创建完毕\e[0m" } # expect自动匹配逻辑 expect_autoauth_func(){ # 接收外部参数 command="$@" expect -c " spawn ${command} expect { \"yes/no\" {send \"yes\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"; exp_continue} \"*password*\" {send \"${login_pass}\r\"} }" } # 跨主机传输文件认证 sshkey_auth_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do # /usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub root@10.0.0.12 cmd="/usr/bin/ssh-copy-id -i ${user_dir}/.ssh/id_rsa.pub" remote_host="${login_user}@${ip}" expect_autoauth_func ${cmd} ${remote_host} done } # 跨主机同步hosts文件 scp_hosts_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do remote_host="${login_user}@${ip}" scp ${host_file} ${remote_host}:${host_file} done } # 跨主机设定主机名规划 set_hostname_func(){ # 接收外部的参数 local host_list="$*" for ip in ${host_list} do host_name=$(grep ${ip} ${host_file}|awk '{print $NF}') remote_host="${login_user}@${ip}" ssh ${remote_host} "hostnamectl set-hostname ${host_name}" done } # 帮助信息逻辑 Usage(){ echo "请输入有效的操作id" } # 逻辑入口 while true do menu read -p "请输入有效的操作id: " target_id if [ ${#target_type[@]} -ge ${target_id} ] then if [ ${target_type[${target_id}-1]} == "部署" ] then echo "开始部署环境操作..." expect_install create_authkey elif [ ${target_type[${target_id}-1]} == "免密" ] then read -p "请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行免密认证操作..." sshkey_auth_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "同步" ] then read -p "请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行同步hosts文件操作..." scp_hosts_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "主机名" ] then read -p "请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): " num_list ip_list=$(eval echo 10.0.0.$num_list) echo "开始执行设定主机名操作..." set_hostname_func ${ip_list} elif [ ${target_type[${target_id}-1]} == "退出" ] then echo "开始退出管理界面..." exit fi else Usage fi done ``` ```powershell 为了更好的把环境部署成功，最好提前更新一下软件源信息 [root@localhost ~]# yum makecache ``` ```powershell 查看脚本执行效果 [root@localhost ~]# /bin/bash /data/scripts/01_remote_host_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 1 开始部署环境操作... expect环境已经部署完毕 Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:u/Tzk0d9sNtG6r9Kx+6xPaENNqT3Lw178XWXQhX1yMw root@kubernetes-master The key's randomart image is: +---[RSA 2048]----+ | .+| | + o.| | E .| | o. | | S + +.| | . . *=+B| | o o+B%O| | . o. +.O+X| | . .o.=+XB| +----[SHA256]-----+ 秘钥文件已经创建完毕 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): {19..21} 开始执行免密认证操作... spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.19 ... Now try logging into the machine, with: "ssh 'root@10.0.0.21'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 3 请输入需要批量远程主机同步hosts的主机列表范围(示例: {12..19}): {19..21} 开始执行同步hosts文件操作... hosts 100% 470 226.5KB/s 00:00 hosts 100% 470 458.8KB/s 00:00 hosts 100% 470 533.1KB/s 00:00 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 4 请输入需要批量设定远程主机主机名的主机列表范围(示例: {12..19}): {19..21} 开始执行设定主机名操作... 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 5 开始退出管理界面... ``` 检查效果 ```powershell [root@localhost ~]# exec /bin/bash [root@kubernetes-master ~]# for i in {19..21} > do > name=$(ssh root@10.0.0.$i "hostname") > echo 10.0.0.$i $name > done 10.0.0.19 kubernetes-master 10.0.0.20 kubernetes-node1 10.0.0.21 kubernetes-node2 ``` **小结** ``` ``` ### 1.5.2 集群准备 学习目标 这一节，我们从 内核调整、软件源配置、小结 三个方面来学习。 **内核调整** 简介 ```powershell 根据kubernetes官方资料的相关信息，我们需要对kubernetes集群是所有主机进行内核参数的调整。 ``` 禁用swap ```powershell 部署集群时，kubeadm默认会预先检查当前主机是否禁用了Swap设备，并在未禁用时强制终止部署过程。因此，在主机内存资源充裕的条件下，需要禁用所有的Swap设备，否则，就需要在后文的kubeadm init及kubeadm join命令执行时额外使用相关的选项忽略检查错误 关闭Swap设备，需要分两步完成。 首先是关闭当前已启用的所有Swap设备： swapoff -a 而后编辑/etc/fstab配置文件，注释用于挂载Swap设备的所有行。 方法一：手工编辑 vim /etc/fstab # UUID=0a55fdb5-a9d8-4215-80f7-f42f75644f69 none swap sw 0 0 方法二： sed -i 's/.*swap.*/#&/' /etc/fstab 替换后位置的&代表前面匹配的整行内容 注意： 只需要注释掉自动挂载SWAP分区项即可，防止机子重启后swap启用 内核(禁用swap)参数 cat >> /etc/sysctl.d/k8s.conf << EOF vm.swappiness=0 EOF sysctl -p /etc/sysctl.d/k8s.conf ``` 网络参数 ```powershell 配置iptables参数，使得流经网桥的流量也经过iptables/netfilter防火墙 cat >> /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF 配置生效 modprobe br_netfilter modprobe overlay sysctl -p /etc/sysctl.d/k8s.conf ``` ```powershell 脚本方式 [root@localhost ~]# cat /data/scripts/02_kubernetes_kernel_conf.sh #!/bin/bash # 功能: 批量设定kubernetes的内核参数调整 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 禁用swap swapoff -a sed -i 's/.*swap.*/#&/' /etc/fstab cat >> /etc/sysctl.d/k8s.conf << EOF vm.swappiness=0 EOF sysctl -p /etc/sysctl.d/k8s.conf # 打开网络转发 cat >> /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF # 加载相应的模块 modprobe br_netfilter modprobe overlay sysctl -p /etc/sysctl.d/k8s.conf ``` 脚本执行 ```powershell master主机执行效果 [root@kubernetes-master ~]# /bin/bash /data/scripts/02_kubernetes_kernel_conf.sh vm.swappiness = 0 vm.swappiness = 0 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 ``` ```powershell 所有node主机执行效果 [root@kubernetes-master ~]# for i in 20 21;do ssh root@10.0.0.$i mkdir /data/scripts -p; scp /data/scripts/02_kubernetes_kernel_conf.sh root@10.0.0.$i:/data/scripts/02_kubernetes_kernel_conf.sh;ssh root@10.0.0.$i "/bin/bash /data/scripts/02_kubernetes_kernel_conf.sh";done ``` **软件源配置** 简介 ```powershell 由于我们需要在多台主机上初始化kubernetes主机环境，所以我们需要在多台主机上配置kubernetes的软件源，以最简便的方式部署kubernetes环境。 ``` 定制阿里云软件源 ```powershell 定制阿里云的关于kubernetes的软件源 [root@kubernetes-master ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF 更新软件源 [root@kubernetes-master ~]# yum makecache fast [root@kubernetes-master ~]# yum install kubeadm-1.23.9-0 kubectl-1.23.9-0 kubelet-1.23.9-0 -y ``` 其他节点主机同步软件源 ```powershell [root@kubernetes-master ~]# for i in 20 21;do scp /etc/yum.repos.d/kubernetes.repo root@10.0.0.$i:/etc/yum.repos.d/kubernetes.repo;ssh root@10.0.0.$i "yum makecache fast; yum install kubeadm-1.23.9-0 kubelet-1.23.9-0 -y";done ``` **小结** ``` ``` ### 1.5.3 容器环境 学习目标 这一节，我们从 Docker环境、环境配置、小结 三个方面来学习。 **Docker环境** 简介 ```powershell 由于kubernetes1.24版本才开始将默认支持的容器引擎转换为了Containerd了，所以这里我们还是以Docker软件作为后端容器的引擎，因为目前的CKA考试环境是以kubernetes1.23版本为基准的。 ``` 软件源配置 ```powershell 安装基础依赖软件 [root@kubernetes-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2 定制专属的软件源 [root@kubernetes-master ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo ``` 安装软件 ```powershell 确定最新版本的docker [root@kubernetes-master ~]# yum list docker-ce --showduplicates | sort -r 安装最新版本的docker [root@kubernetes-master ~]# yum install -y docker-ce ``` 检查效果 ```powershell 启动docker服务 [root@kubernetes-master ~]# systemctl restart docker 检查效果 [root@kubernetes-master ~]# docker version Client: Docker Engine - Community Version: 20.10.17 API version: 1.41 Go version: go1.17.11 Git commit: 100c701 Built: Mon Jun 6 23:05:12 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.17 API version: 1.41 (minimum version 1.12) Go version: go1.17.11 Git commit: a89b842 Built: Mon Jun 6 23:03:33 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.6 GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc: Version: 1.1.2 GitCommit: v1.1.2-0-ga916309 docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` **环境配置** 需求 ```powershell 1 镜像仓库 默认安装的docker会从官方网站上获取docker镜像，有时候会因为网络因素无法获取，所以我们需要配置国内镜像仓库的加速器 2 kubernetes的改造 kubernetes的创建容器，需要借助于kubelet来管理Docker，而默认的Docker服务进程的管理方式不是kubelet的类型，所以需要改造Docker的服务启动类型为systemd方式。 注意： 默认情况下，Docker的服务修改有两种方式： 1 Docker服务 - 需要修改启动服务文件，需要重载服务文件，比较繁琐。 2 daemon.json文件 - 修改文件后，只需要重启docker服务即可，该文件默认情况下不存在。 ``` 定制docker配置文件 ```powershell 定制配置文件 [root@kubernetes-master ~]# cat >> /etc/docker/daemon.json <<-EOF { "registry-mirrors": [ "http://74f21445.m.daocloud.io", "https://registry.docker-cn.com", "http://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn" ], "insecure-registries": ["10.0.0.20:80"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF 重启docker服务 [root@kubernetes-master ~]# systemctl restart docker ``` 检查效果 ```powershell 查看配置效果 [root@kubernetes-master ~]# docker info Client: ... Server: ... Cgroup Driver: systemd ... Insecure Registries: 10.0.0.20:80 127.0.0.0/8 Registry Mirrors: http://74f21445.m.daocloud.io/ https://registry.docker-cn.com/ http://hub-mirror.c.163.com/ https://docker.mirrors.ustc.edu.cn/ Live Restore Enabled: false ``` docker环境定制脚本 ```powershell 查看脚本内容 [root@localhost ~]# cat /data/scripts/03_kubernetes_docker_install.sh #!/bin/bash # 功能: 安装部署Docker容器环境 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 准备工作 # 软件源配置 softs_base(){ # 安装基础软件 yum install -y yum-utils device-mapper-persistent-data lvm2 # 定制软件仓库源 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 更新软件源 systemctl restart network yum makecache fast } # 软件安装 soft_install(){ # 安装最新版的docker软件 yum install -y docker-ce # 重启docker服务 systemctl restart docker } # 加速器配置 speed_config(){ # 定制加速器配置 cat > /etc/docker/daemon.json <<-EOF { "registry-mirrors": [ "http://74f21445.m.daocloud.io", "https://registry.docker-cn.com", "http://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn" ], "insecure-registries": ["10.0.0.20:80"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF # 重启docker服务 systemctl restart docker } # 环境监测 docker_check(){ process_name=$(docker info | grep 'p D' | awk '{print $NF}') [ "${process_name}" == "systemd" ] && echo "Docker软件部署完毕" || (echo "Docker软件部署失败" && exit) } # 软件部署 main(){ softs_base soft_install speed_config docker_check } # 调用主函数 main ``` ```powershell 其他主机环境部署docker [root@kubernetes-master ~]# for i in {19..21}; do ssh root@10.0.0.$i "mkdir -p /data/scripts"; scp /data/scripts/03_kubernetes_docker_install.sh root@10.0.0.$i:/data/scripts/03_kubernetes_docker_install.sh; done 其他主机各自执行下面的脚本 /bin/bash /data/scripts/03_kubernetes_docker_install.sh ``` **小结** ``` ``` ### 1.5.4 k8s环境 学习目标 这一节，我们从 集群部署、其他配置、小结 三个方面来学习。 **集群部署** 镜像获取 ```powershell 镜像获取脚本内容 [root@localhost ~]# cat /data/scripts/04_kubernetes_get_images.sh #!/bin/bash # 功能: 获取kubernetes依赖的Docker镜像文件 # 版本: v0.1 # 作者: 书记 # 联系: superopsmsb.com # 定制普通环境变量 ali_mirror='registry.aliyuncs.com' harbor_mirror='kubernetes-register.superopsmsb.com' harbor_repo='google_containers' # 环境定制 kubernetes_image_get(){ # 获取脚本参数 kubernetes_version="$1" # 获取制定kubernetes版本所需镜像 images=$(kubeadm config images list --kubernetes-version=${kubernetes_version} | awk -F "/" '{print $NF}') # 获取依赖镜像 for i in ${images} do docker pull ${ali_mirror}/${harbor_repo}/$i docker tag ${ali_mirror}/${harbor_repo}/$i ${harbor_mirror}/${harbor_repo}/$i docker rmi ${ali_mirror}/${harbor_repo}/$i done } # 脚本的帮助 Usage(){ echo "/bin/bash $0 " } # 脚本主流程 if [ $# -eq 0 ] then read -p "请输入要获取kubernetes镜像的版本(示例: v1.23.9): " kubernetes_version kubernetes_image_get ${kubernetes_version} else Usage fi ``` ```powershell 脚本执行效果 [root@kubernetes-master ~]# /bin/bash /data/scripts/04_kubernetes_get_images.sh 请输入要获取kubernetes镜像的版本(示例: v1.23.9): v1.23.9 ... ``` ```powershell node节点获取镜像 docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.23.9 docker pull registry.aliyuncs.com/google_containers/pause:3.6 ``` master环境初始化 ```powershell 环境初始化命令 kubeadm init --kubernetes-version=1.23.9 \ --apiserver-advertise-address=10.0.0.19 \ --image-repository registry.aliyuncs.com/google_containers \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --ignore-preflight-errors=Swap ``` ```powershell 设定kubernetes的认证权限 [root@kubernetes-master ~]# mkdir -p $HOME/.kube [root@kubernetes-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@kubernetes-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config 再次检测 [root@kubernetes-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master NotReady control-plane,master 4m10s v1.23.9 ``` node环境初始化 ```powershell node节点环境初始化 kubeadm join 10.0.0.19:6443 --token vfiiwc.se99g4ai8wl2md9r --discovery-token-ca-cert-hash sha256:0c552edf7b884431dfc1877e6f09a0660460ddbdc567d2366dae43dbc10e9554 ``` **其他配置** 网络环境配置 ```powershell 创建基本目录 mkdir /data/kubernetes/flannel -p cd /data/kubernetes/flannel 获取配置文件 wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml ``` ```powershell 所有节点获取镜像 docker pull docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 docker pull docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2 ``` ```powershell 应用网络环境配置文件 kubectl apply -f kube-flannel.yml ``` ```powershell 确认效果 [root@localhost /data/kubernetes/flannel]# kubectl get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready control-plane,master 10m v1.23.9 kubernetes-node1 Ready 9m56s v1.23.9 kubernetes-node2 Ready 9m52s v1.23.9 ``` 命令优化 ```powershell 放到当前用户的环境文件中 [root@kubernetes-master ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc [root@kubernetes-master ~]# echo "source <(kubeadm completion bash)" >> ~/.bashrc [root@kubernetes-master ~]# source ~/.bashrc 测试效果 [root@kubernetes-master ~]# kubectl get n namespaces networkpolicies.networking.k8s.io nodes [root@kubernetes-master ~]# kubeadm co completion config ``` **小结** ``` ``` ### 1.5.5 rbd实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell 自从k8s1.13版本以来，ceph与k8s的集成越来越紧密。ceph为k8s提供存储服务主要有两种方式，cephfs和ceph rdb；cephfs方式支持k8s的pv的3种访问模式ReadWriteOnce，ReadOnlyMany，ReadWriteMany ，RBD支持ReadWriteOnce，ReadOnlyMany。 ``` 准备工作 ```powershell 跨主机免密码认证 [root@admin /data/scripts]# /bin/bash 01_remote_host_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): {19..21} 开始执行免密认证操作... spawn /usr/bin/ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.0.0.19 ... Now try logging into the machine, with: "ssh 'root@10.0.0.21'" and check to make sure that only the key(s) you wanted were added. 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 同步hosts 4: 设定主机名 5：退出操作 ===================================================== 请输入有效的操作id: 5 开始退出管理界面... ``` ```powershell ceph环境准备工作 [root@admin /data/scripts]# for i in {19..21} do scp /etc/yum.repos.d/ceph.repo root@10.0.0.$i:/etc/yum.repos.d/ ssh root@10.0.0.$i "yum install ceph -y" done ``` ```powershell 准备文件 [root@admin /data/scripts]# for i in {19..21};do scp 02_create_ceph_user.sh 03_remote_cephadm_auth.sh root@10.0.0.$i:/data/scripts; done 准备专属用户 /bin/bash /data/scripts/02_create_ceph_user.sh ``` ```powershell 定制数据目录的认证 [cephadm@admin cephadm-cluster]$ /bin/bash /data/scripts/03_remote_cephadm_auth.sh 批量设定远程主机免密码认证管理界面 ===================================================== 1: 部署环境 2: 免密认证 3: 退出操作 ===================================================== 请输入有效的操作id: 2 请输入需要批量远程主机认证的主机列表范围(示例: {12..19}): {19..21} 开始执行免密认证操作... spawn /usr/bin/ssh-copy-id -i /home/cephadm/.ssh/id_rsa.pub cephadm@10.0.0.19 ... ``` ```powershell 同步配置文件 [cephadm@admin cephadm-cluster]$ ceph-deploy config push kubernetes-master kubernetes-node1 kubernetes-node2 ``` 创建rbd相关的存储池 ```powershell 创建存储池 [cephadm@admin ceph-cluster]$ ceph osd pool create kube-rbddata 16 16 pool 'kube-rbddata' created 启用rbd功能 [cephadm@admin ceph-cluster]$ ceph osd pool application enable kube-rbddata rbd enabled application 'rbd' on pool 'kube-rbddata' 存储池初始化 [cephadm@admin ceph-cluster]$ rbd pool init -p kube-rbddata 创建存储镜像 [cephadm@admin ceph-cluster]$ rbd create kube-rbddata/kubeimg --size 2Gi 查看效果 [cephadm@admin ceph-cluster]$ rbd ls -p kube-rbddata -l NAME SIZE PARENT FMT PROT LOCK kubeimg 2 GiB 2 查看镜像属性 [cephadm@admin cephadm-cluster]$ rbd info kube-rbddata/kubeimg rbd image 'kubeimg': size 2 GiB in 512 objects ... features: layering, exclusive-lock, object-map, fast-diff, deep-flatten ``` ```powershell 注意： 有可能在k8s使用rbd的时候，提示不支持 object-map, fast-diff, deep-flatten 属性，那么我们到时候执行如下命令： rbd feature disable kube-rbddata/kubeimg exclusive-lock object-map fast-diff deep-flatten ``` 创建专属的访问存储池的用户账号 ```powershell 创建专属的账号信息 [cephadm@admin ceph-cluster]$ ceph auth get-or-create client.k8s mon 'allow r' osd 'allow * pool=kube-rbddata' -o ceph.client.k8s.keyring 查看秘钥环文件 [cephadm@admin ceph-cluster]$ cat ceph.client.k8s.keyring [client.k8s] key = AQCAR0hj6rCdNhAAFPIDxBTzujdX3SpFDE+pOQ== 完善秘钥环文件 [cephadm@admin cephadm-cluster]$ ceph auth get client.k8s -o ceph.client.k8s.keyring exported keyring for client.k8s [cephadm@admin cephadm-cluster]$ cat ceph.client.k8s.keyring [client.k8s] key = AQCAR0hj6rCdNhAAFPIDxBTzujdX3SpFDE+pOQ== caps mon = "allow r" ``` ```powershell 传递秘钥环文件到所有k8s节点 for i in {19..21} do scp ceph.client.k8s.keyring root@10.0.0.$i:/etc/ceph/ done ``` ```powershell 查看效果 [root@kubernetes-master ~]# rbd --user k8s -p kube-rbddata ls kubeimg [root@kubernetes-master ~]# ceph --user k8s -s cluster: id: d932ded6-3765-47c1-b0dc-e6957051e31a health: HEALTH_WARN application not enabled on 1 pool(s) ... ``` ceph查看效果 ```powershell 查看当前ceph的pool信息 [root@kubernetes-master ~]# ceph --user k8s osd pool ls | grep kube kube-rbddata 创建对应的镜像信息 [root@kubernetes-master ~]# rbd --user k8s create kube-rbddata/podimg02 --size 1G [root@kubernetes-master ~]# rbd --user k8s ls -p kube-rbddata | grep od podimg01 ``` **简单实践** 准备工作 ```powershell 待测试客户端准备镜像 [root@kubernetes-node1 ~]# docker pull redis:6.2.5 ``` 定制pod测试1 ```powershell 定制专属的pod测试资源清单文件 01-ceph-k8s-pod-test.yaml apiVersion: v1 kind: Pod metadata: name: redis-with-rdb-test spec: nodeName: kubernetes-node1 containers: - name: redis image: redis:6.2.5 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/data" name: redis-cephrbd-vol volumes: - name: redis-cephrbd-vol rbd: monitors: - 10.0.0.13:6789 - 10.0.0.14:6789 - 10.0.0.15:6789 pool: kube-rbddata image: kubeimg fsType: xfs user: k8s keyring: /etc/ceph/ceph.client.k8s.keyring 应用资源清单文件 kubectl apply -f 01-ceph-k8s-pod-test.yaml ``` ```powershell 检查效果 [root@kubernetes-master /data/kubernetes/ceph]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE ... redis-with-rdb-test 1/1 Running 0 22s 10.244.1.2 k8s-node1 ... 查看信息 [root@kubernetes-master /data/kubernetes/ceph]# kubectl describe pod redis-with-rdb-test ``` ```powershell 查看宿主机绑定效果 [root@kubernetes-node1 ~]# rbd showmapped id pool namespace image snap device 0 kube-rbddata kubeimg - /dev/rbd0 [root@kubernetes-node1 ~]# mount | grep rbd /dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-rbddata-image-kubeimg type xfs (rw,relatime,attr2,inode64,sunit=8192,swidth=8192,noquota) /dev/rbd0 on /var/lib/kubelet/pods/006fb6bd-6623-4f29-88d9-53e6f4fe5222/volumes/kubernetes.io~rbd/redis-cephrbd-vol type xfs (rw,relatime,attr2,inode64,sunit=8192,swidth=8192,noquota) ``` ```powershell 查看pod [root@kubernetes-master /data/kubernetes/ceph]# kubectl get pod NAME READY STATUS RESTARTS AGE redis-with-rdb-test 1/1 Running 0 7m39s 进入pod查看效果 [root@kubernetes-master /data/kubernetes/ceph]# kubectl exec -it redis-with-rdb-test -- bash root@redis-with-rdb-test:/data# mount | grep rbd /dev/rbd0 on /data type xfs (rw,relatime,attr2,inode64,sunit=8192,swidth=8192,noquota) redis使用rdb效果 root@redis-with-rdb-test:/data# ls lost+found root@redis-with-rdb-test:/data# redis-cli 127.0.0.1:6379> set nihao nihao OK 127.0.0.1:6379> set buhao buhao OK 127.0.0.1:6379> keys * 1) "buhao" 2) "nihao" 127.0.0.1:6379> BGSAVE Background saving started 127.0.0.1:6379> exit root@redis-with-rdb-test:/data# ls dump.rdb lost+found ``` ```powershell 宿主机确认效果 [root@kubernetes-node1 ~]# ls /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-rbddata-image-kubeimg dump.rdb ``` ```powershell 收尾动作 [root@kubernetes-master /data/kubernetes/ceph]# kubectl delete -f 01-ceph-k8s-pod-test.yaml pod "redis-with-rdb-test" deleted ``` 定制pod实践2 ```powershell 创建专属的镜像磁盘 [cephadm@admin ceph-cluster]$ rbd create kube-rbddata/kubeimg02 --size 2Gi [cephadm@admin ceph-cluster]$ rbd ls -p kube-rbddata -l NAME SIZE PARENT FMT PROT LOCK kubeimg 2 GiB 2 excl kubeimg02 2 GiB 2 清理磁盘格式 [cephadm@admin ceph-cluster]$ rbd feature disable kube-rbddata/kubeimg02 exclusive-lock object-map fast-diff deep-flatten ``` ```powershell 查看ceph的keyring的信息 [cephadm@admin ceph-cluster]$ cat ceph.client.k8s.keyring [client.k8s] key = AQDh47ZhqWklMRAAY+6UM0wkbSRc2DeLctQY+A== [cephadm@admin ceph-cluster]$ ceph auth print-key client.k8s AQDh47ZhqWklMRAAY+6UM0wkbSRc2DeLctQY+A== 生成对应的base64编码信息 [cephadm@admin ceph-cluster]$ ceph auth print-key client.k8s | base64 QVFEaDQ3WmhxV2tsTVJBQVkrNlVNMHdrYlNSYzJEZUxjdFFZK0E9PQ== ``` ```powershell 定制专属的pod测试资源清单文件 02-ceph-k8s-pod-secret.yaml apiVersion: v1 kind: Secret metadata: name: ceph-secret type: "kubernetes.io/rbd" data: key: QVFEaDQ3WmhxV2tsTVJBQVkrNlVNMHdrYlNSYzJEZUxjdFFZK0E9PQ== --- apiVersion: v1 kind: Pod metadata: name: redis-with-secret-test spec: nodeName: kubernetes-node1 containers: - name: redis image: redis:6.2.5 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/data" name: redis-cephrbd-vol volumes: - name: redis-cephrbd-vol rbd: monitors: - 10.0.0.13:6789 - 10.0.0.14:6789 - 10.0.0.15:6789 pool: kube-rbddata image: kubeimg02 fsType: xfs readOnly: false user: k8s secretRef: name: ceph-secret 应用资源清单文件 kubectl apply -f 02-ceph-k8s-pod-secret.yaml ``` ```powershell 查看效果 [root@kubernetes-master /data/kubernetes/ceph]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE ... redis-with-secret-test 1/1 Running 0 13s 10.244.2.3 k8s-node2 ... ``` ```powershell 收尾动作 [root@kubernetes-master /data/kubernetes/ceph]# kubectl delete -f 02-ceph-k8s-pod-secret.yaml secret "ceph-secret" deleted pod "redis-with-secret-test" deleted ``` **小结** ``` ``` ### 1.5.6 CephFS实践 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 需求 ```powershell k8s对ceph rbd模式不支持ReadWriteMany（RWX）,为了满足k8s的灵活性需求,采用支持多点挂载的cephfs工作模式 ``` 准备工作 ```powershell 检查cephfs环境 [cephadm@admin cephadm-cluster]$ ceph fs ls name: cephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data ] [cephadm@admin cephadm-cluster]$ ceph -s | grep mds mds: cephfs:1 {0=stor05=up:active} ``` ```powershell 保存用户账号的密钥信息于secret文件，用于客户端挂载操作认证之用 [cephadm@admin ceph-cluster]$ ceph auth print-key client.fsclient > fsclient.key [cephadm@admin ceph-cluster]$ cat fsclient.key AQBgdTJj+LVdFBAAZOjGIsw4t+o1swZlW4CvKQ== ``` ```powershell 将秘钥文件传递给所有的k8s节点主机 for i in {19..21} do scp fsclient.key root@10.0.0.$i:/etc/ceph/ done ``` ```powershell 创建专属的数据目录 [root@kubernetes-node2 ~]# mkdir /cephfs-data/ 挂载cephFS对象 [root@kubernetes-node2 ~]# mount -t ceph 10.0.0.13:6789,10.0.0.14:6789,10.0.0.15:6789:/ /cephfs-data/ -o name=fsclient,secretfile=/etc/ceph/fsclient.key 注意： 内核空间挂载ceph文件系统，要求必须指定ceph文件系统的挂载路径 确认挂载效果 [root@kubernetes-node2 ~]# mount | grep cephfs 10.0.0.13:6789,10.0.0.14:6789,10.0.0.15:6789:/ on /cephfs-data type ceph (rw,relatime,name=fsclient,secret=,acl,wsize=16777216) ``` **简单实践** 创建资源清单文件 ```powershell 获取秘钥信息 [root@kubernetes-master /data/kubernetes/ceph]# cat /etc/ceph/fsclient.key AQDYCUpjmLjUDxAAYkfNGUG5FUYaFVE6jlFgrQ== 生成专属的秘钥信息 [root@kubernetes-master /data/kubernetes/ceph]# echo -n 'AQDYCUpjmLjUDxAAYkfNGUG5FUYaFVE6jlFgrQ==' | base64 QVFEWUNVcGptTGpVRHhBQVlrZk5HVUc1RlVZYUZWRTZqbEZnclE9PQ== ``` ```powershell 创建secret资源清单文件 03-cephfs-k8s-test.yaml apiVersion: v1 kind: Secret metadata: name: ceph-secret data: key: QVFEWUNVcGptTGpVRHhBQVlrZk5HVUc1RlVZYUZWRTZqbEZnclE9PQ== --- apiVersion: v1 kind: PersistentVolume metadata: name: cephfs-pv labels: pv: cephfs-pv spec: capacity: storage: 1Gi accessModes: - ReadWriteMany cephfs: monitors: - 10.0.0.13:6789 - 10.0.0.14:6789 - 10.0.0.15:6789 user: fsclient secretRef: name: ceph-secret readOnly: false persistentVolumeReclaimPolicy: Delete --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: cephfs-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi selector: matchLabels: pv: cephfs-pv ``` ```powershell 定制资源清单 04-cephfs-k8s-pod-redis.yaml apiVersion: v1 kind: Pod metadata: name: superopsmsb-redis spec: nodeName: kubernetes-node1 volumes: - name: redis-cephfs-vol persistentVolumeClaim: claimName: cephfs-pvc containers: - name: redis image: redis:6.2.5 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: "/data" name: redis-cephfs-vol ``` 测试效果 ```powershell 应用资源清单文件 [root@kubernetes-master /data/kubernetes/ceph]# kubectl apply -f 03-cephfs-k8s-test.yaml -f 04-cephfs-k8s-pod-redis.yaml 确认效果 [root@kubernetes-master /data/kubernetes/ceph]# kubectl describe pod superopsmsb-redis ``` ```powershell 客户端确认效果 [root@kubernetes-node1 ~]# mount | grep ceph 10.0.0.13:6789,10.0.0.14:6789,10.0.0.15:6789:/ on /var/lib/kubelet/pods/41a1af23-da66-486c-b80d-a2e084f75edc/volumes/kubernetes.io~cephfs/cephfs-pv type ceph (rw,relatime,name=fsclient,secret=,acl,wsize=16777216) ``` ```powershell 收尾操作 [root@kubernetes-master /data/kubernetes/ceph]# kubectl delete -f 04-cephfs-k8s-pod-redis.yaml -f 03-cephfs-k8s-test.yaml ``` **小结** ``` ``` ### 1.5.7 SC基础 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 需求 ```powershell 对于手工方式为应用提供指定的存储能力是可行的，但是在k8s动态变化的场景中，尤其是面对无状态的集群服务，如何自由的提供存储能力，是想当要考量的一个话题，对于这种场景，我们可以在SC的基础上，实现动态的为应用服务提供存储能力。 ``` ceph-csi ```powershell ceph-csi扩展各种存储类型的卷的管理能力，实现第三方存储ceph的各种操作能力与k8s存储系统的结合。调用第三方存储ceph的接口或命令，从而提供ceph数据卷的创建/删除、挂载/解除挂载的具体操作实现。前面分析组件中的对于数据卷的创建/删除、挂载/解除挂载操作，全是调用ceph-csi，然后由ceph-csi调用ceph提供的命令或接口来完成最终的操作。 自从kubernetes 1.13版本之后，ceph-csi为k8s环境动态地提供相关的存储能力。 ``` 版本支持 | Ceph CSI Version | Container Orchestrator Name | Version Tested | | ---------------- | --------------------------- | ------------------- | | v3.7.1 | Kubernetes | v1.22, v1.23, v1.24 | | v3.7.0 | Kubernetes | v1.22, v1.23, v1.24 | | v3.6.1 | Kubernetes | v1.21, v1.22, v1.23 | | v3.6.0 | Kubernetes | v1.21, v1.22, v1.23 | **简单实践** 组件功能 ```powershell 创建 pvc: external-provisioner组件监听到pvc创建事件后，负责拼接请求，然后调用ceph-csi的CreateVolume方法来创建存储； 删除 pvc: pv对象状态由bound变为release，external-provisioner监听到pv更新事件后，负责调用ceph-csi的DeleteVolume方法来删除存储。 pod关联pvc: kubelet会调用ceph-csi组件将创建好的存储从ceph集群挂载到pod所在的node上，然后再挂载到pod相应的目录上； pod断开pvc: kubelet会调用ceph-csi组件相应方法，解除存储在pod目录上的挂载，再解除存储在node上的挂载。 ``` 服务组成 ```powershell ceph-csi含有rbdType、cephfsType、livenessType三大类型服务,它们可以通过启动参数指定一种服务来进行启动。 - rbdType主要进行rbd的操作完成与ceph的交互 - cephfsType主要进行cephfs的操作完成与ceph交互 - livenessType该服务主要是定时向csi endpoint探测csi组件的存活，然后统计到prometheus指标中。 ``` ```powershell rbdType、cephfsType类型的服务可以细分为如下三个子服务： - ControllerServer：主要负责创建、删除cephfs/rbd存储等操作。 - NodeServer：部署在k8s中的每个node上，主要负责cephfs、rbd在node节点上相关的操作。 - IdentityServer：主要是返回自身服务的相关信息，如返回服务身份信息、服务能力、暴露存活探测接口等 注意： 其中NodeServer与ControllerServer只能选其一进行启动 IdentityServer会伴随着NodeServer或ControllerServer的启动而启动。 ``` ![image-20221015174850734](../../img/kubernetes/kubernetes_storage_ceph/image-20221015174850734.png) 应用组件 ```powershell ceph-csi 在部署的时候，主要涉及到以下组件 external-provisioner（csi-provisioner）： - 创建provision环境，然后负责监听pvc是否需要动态创建和删除存储卷 external-attacher（csi-attacher）： - 负责获取PV的所有信息，然后判断是否调用CSI Plugin的ControllerPublish做attach，还是调用CntrollerUnpublish接口做detach external-snapshotter： - 对镜像文件进行快照相关操作，同时负责判断PVC中是否调整需求的存储容量空间大小 node-driver-registrar 调用CSI Plugin的接口获取插件信息，通过Kubelet的插件注册机制将CSI Plugin注册到kubelet livenessprobe 调用CSI Plugin的Probe接口，同时在/healthz暴露HTTP健康检查探针 ``` ![image-20221015175059810](../../img/kubernetes/kubernetes_storage_ceph/image-20221015175059810.png) 部署解读 ```powershell 对于ceph-csi环境的部署来说，主要涉及到两个文件： csi-xxxplugin-provisioner.yaml部署的相关组件： csi-provisioner 核心sc服务能力，负责根据pvc来管理pv对象，动态的管理ceph中的image csi-snapshotter 负责处理存储快照相关的操作 csi-attacher 负责操作VolumeAttachment对象，并没有操作存储。 csi-resizer 负责处理存储扩容相关的操作 csi-xxxplugin 核心的ceph-csi组件，负责与kubernetes对接存储相关的操作 liveness-prometheus 负责探测并上报csi-rbdplugin服务的存活情况 csi-xxxplugin.yaml部署的相关组件： driver-registrar 向kubelet传入csi-xxxplugin容器提供服务的socket地址等信息。 csi-xxxplugin 负责与kubernetes对接存储相关的操作 liveness-prometheus 负责探测并上报csi-xxxplugin服务的存活情况 注意： xxx 主要有两类：rdb和cephfs ``` **小结** ``` ``` ### 1.5.8 SC-rbd实践 学习目标 这一节，我们从 基础环境、简单实践、小结 三个方面来学习。 **基础知识** ```powershell 参考资料： https://docs.ceph.com/en/quincy/rbd/rbd-kubernetes/ ``` 准备工作 ```powershell 获取代码环境 cd /data/kubernetes/ git clone https://github.com/ceph/ceph-csi.git cd ceph-csi/deploy/rbd/kubernetes/ ``` ```powershell 获取基本状态信息 [cephadm@admin cephadm-cluster]$ ceph mon dump epoch 3 fsid d932ded6-3765-47c1-b0dc-e6957051e31a last_changed 2062-09-30 23:25:47.516433 created 2062-09-30 19:05:03.907227 min_mon_release 14 (nautilus) 0: [v2:10.0.0.13:3300/0,v1:10.0.0.13:6789/0] mon.mon01 1: [v2:10.0.0.14:3300/0,v1:10.0.0.14:6789/0] mon.mon02 2: [v2:10.0.0.15:3300/0,v1:10.0.0.15:6789/0] mon.mon03 这里重点关注： fsid-ceph集群的id 和 mon节点的列表 ``` 部署rbd环境 ```powershell 修改核心配置文件 csi-config-map.yaml --- apiVersion: v1 kind: ConfigMap metadata: name: "ceph-csi-config" data: config.json: |- [ { "clusterID": "d932ded6-3765-47c1-b0dc-e6957051e31a", "monitors": [ "10.0.0.13:6789", "10.0.0.14:6789", "10.0.0.15:6789" ] } ] 注意： 修改 clusterID 和 monitors 部分内容 ``` ```powershell 创建依赖的configmap文件 cat < ceph-config-map.yaml --- apiVersion: v1 kind: ConfigMap data: config.json: |- {} metadata: name: ceph-csi-encryption-kms-config --- apiVersion: v1 kind: ConfigMap data: ceph.conf: | [global] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx keyring: | metadata: name: ceph-config EOF ``` ```powershell 修改部署资源清单文件 csi-rbdplugin-provisioner.yaml ... kind: Deployment apiVersion: apps/v1 metadata: name: csi-rbdplugin-provisioner # replace with non-default namespace name namespace: default spec: replicas: 2 # 将3修改为2 ... ``` ```powershell 部署资源清单文件 kubectl apply -f csi-config-map.yaml kubectl apply -f ceph-config-map.yaml kubectl apply -f csidriver.yaml kubectl apply -f csi-provisioner-rbac.yaml kubectl apply -f csi-nodeplugin-rbac.yaml kubectl apply -f csi-rbdplugin-provisioner.yaml kubectl apply -f csi-rbdplugin.yaml 注意： csidriver.yaml 对于pod挂载pvc是非常重要的，如果不执行会导致挂载失败 ``` 确认效果 ```powershell 检查效果 [root@kubernetes-master /data/kubernetes/ceph-csi/deploy/rbd/kubernetes]# kubectl get pod NAME READY STATUS RESTARTS AGE csi-rbdplugin-8xq6m 3/3 Running 0 53s csi-rbdplugin-nbf8g 3/3 Running 0 53s csi-rbdplugin-provisioner-8599998d64-77zqd 7/7 Running 0 54s csi-rbdplugin-provisioner-8599998d64-p48wm 7/7 Running 0 54s ``` **简单实践** ceph环境准备 ```powershell 创建专属的存储池 [cephadm@admin cephadm-cluster]$ ceph osd pool create kubernetes 64 pool 'kubernetes' created [cephadm@admin cephadm-cluster]$ rbd pool init kubernetes ``` ```powershell 创建专属的用户 -- 可选 [cephadm@admin cephadm-cluster]$ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes' [client.kubernetes] key = AQA7VUpjyHt/OxAAh1GTn8eWEWBmW2lJ61NmjQ== ``` ```powershell 我们这里以admin用户为例 [cephadm@admin cephadm-cluster]$ ceph auth get client.admin [client.admin] key = AQByzTZjaEaCCRAAolAFknx1x/LgTYhNyy50cw== ... ``` 应用测试 ```powershell 进入专属测试文件目录 cd /data/kubernetes/ceph-csi/examples/rbd/ ``` ```powershell 修改secret文件 secret.yaml ... stringData: userID: admin userKey: AQByzTZjaEaCCRAAolAFknx1x/LgTYhNyy50cw== ... 修改： 根据提示信息修改 userID 和 userKey 的信息 这里的userKey无需进行base64加密 ``` ```powershell 修改SC配置 storageclass.yaml ... parameters: ... clusterID: d932ded6-3765-47c1-b0dc-e6957051e31a ... pool: kubernetes ... 修改： 根据提示信息修改 clusterID 和 pool 的信息 ``` ```powershell 应用资源清单文件 kubectl apply -f secret.yaml kubectl apply -f storageclass.yaml kubectl apply -f pvc.yaml kubectl apply -f pod.yaml ``` 测试效果 ```powershell 确定pvc和pv的效果 [root@kubernetes-master /data/kubernetes/ceph-csi/examples/rbd]# kubectl get pvc NAME STATUS VOLUME CAPACITY ... rbd-pvc Bound pvc-441dbf43-5051-4130-b8d7-a539e5395ba1 1Gi ... [root@kubernetes-master /data/kubernetes/ceph-csi/examples/rbd]# kubectl get pv NAME CAPACITY ACCESS MODES ... pvc-441dbf43-5051-4130-b8d7-a539e5395ba1 1Gi RWO ... ``` ```powershell 确定pod效果 [root@kubernetes-master /data/kubernetes/ceph-csi/examples/rbd]# kubectl describe pod csi-rbd-demo-pod | grep -A3 Volumes: Volumes: mypvc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: rbd-pvc ``` ```powershell ceph环境中确认效果 [cephadm@admin cephadm-cluster]$ rbd ls --pool kubernetes -l NAME SIZE PARENT FMT PROT LOCK csi-vol-0a8bcde0-ddd0-4ca8-a194-70bf61aaf691 1 GiB 2 ``` 移除应用 ```powershell 移除应用 kubectl delete -f pod.yaml kubectl delete -f pvc.yaml ceph环境中确认效果 [cephadm@admin cephadm-cluster]$ rbd ls --pool kubernetes -l [cephadm@admin cephadm-cluster]$ ``` 环境收尾 ```powershell 清理应用环境 cd /data/kubernetes/ceph-csi/examples/rbd kubectl delete -f storageclass.yaml -f secret.yaml 清理sc环境 cd /data/kubernetes/ceph-csi/deploy/rbd/kubernetes kubectl delete -f ./ ``` ```powershell 清理ceph环境 ceph osd pool rm kubernetes kubernetes --yes-i-really-really-mean-it ceph auth rm client.kubernetes ``` **小结** ``` ``` ### 1.5.9 SC-cephfs实践 学习目标 这一节，我们从 基础环境、简单实践、小结 三个方面来学习。 **基础知识** 准备工作 ```powershell 获取代码环境 cd /data/kubernetes/ceph-csi/deploy/cephfs/kubernetes/ ``` ```powershell 获取基本状态信息 [cephadm@admin cephadm-cluster]$ ceph mon dump epoch 3 fsid d932ded6-3765-47c1-b0dc-e6957051e31a last_changed 2062-09-30 23:25:47.516433 created 2062-09-30 19:05:03.907227 min_mon_release 14 (nautilus) 0: [v2:10.0.0.13:3300/0,v1:10.0.0.13:6789/0] mon.mon01 1: [v2:10.0.0.14:3300/0,v1:10.0.0.14:6789/0] mon.mon02 2: [v2:10.0.0.15:3300/0,v1:10.0.0.15:6789/0] mon.mon03 这里重点关注： fsid-ceph集群的id 和 mon节点的列表 ``` 部署rbd环境 ```powershell 修改核心配置文件 csi-config-map.yaml --- apiVersion: v1 kind: ConfigMap metadata: name: "ceph-csi-config" data: config.json: |- [ { "clusterID": "d932ded6-3765-47c1-b0dc-e6957051e31a", "monitors": [ "10.0.0.13:6789", "10.0.0.14:6789", "10.0.0.15:6789" ] } ] 注意： 修改 clusterID 和 monitors 部分内容 ``` ```powershell 创建依赖的configmap文件 cat < ceph-config-map.yaml --- apiVersion: v1 kind: ConfigMap data: ceph.conf: | [global] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx keyring: | metadata: name: ceph-config EOF ``` ```powershell 修改部署资源清单文件 csi-cephfsplugin-provisioner.yaml ... kind: Deployment apiVersion: apps/v1 metadata: name: csi-cephfsplugin-provisioner spec: selector: matchLabels: app: csi-cephfsplugin-provisioner replicas: 2 # 将3修改为2 ... containers: - name: csi-provisioner ... - "--extra-create-metadata=false" - name: csi-snapshotter ... - "--extra-create-metadata=false" 注意： 一定要将 --extra-create-metadata 的值设为 false，否则后续对于subvolume要求较高，报错信息： API call not implemented server-side: No handler found for 'fs subvolume metadata set' ``` ```powershell 部署资源清单文件 kubectl apply -f ./ 注意： csidriver.yaml 对于pod挂载pvc是非常重要的，如果不执行会导致挂载失败 ``` 确认效果 ```powershell 检查效果 [root@kubernetes-master /data/kubernetes/ceph-csi/deploy/cephfs/kubernetes]# kubectl get pod NAME READY STATUS RESTARTS AGE csi-cephfsplugin-dtdsn 3/3 Running 0 27s csi-cephfsplugin-provisioner-65d8546b95-k2rzt 5/5 Running 0 27s csi-cephfsplugin-provisioner-65d8546b95-zwhws 5/5 Running 0 26s csi-cephfsplugin-sd9rk 3/3 Running 0 27s ``` **简单实践** ceph环境准备 ```powershell 如果没有cephfs环境的话，执行如下命令 ceph osd pool create cephfs-data 16 16 ceph osd pool create cephfs-metadata 16 16 ceph fs new cephfs cephfs-metadata cephfs-data ``` ```powershell 确认当前的cephfs环境 [cephadm@admin cephadm-cluster]$ ceph fs ls name: cephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data ] ``` ```powershell 创建专属的用户 -- 可选 [cephadm@admin cephadm-cluster]$ ceph auth get-or-create client.fsclient mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data' -o ceph.client.fsclient.keyring ``` ```powershell 我们这里以admin 和 fsclient用户为例 [cephadm@admin cephadm-cluster]$ ceph auth get-key client.fsclient AQDYCUpjmLjUDxAAYkfNGUG5FUYaFVE6jlFgrQ== [cephadm@admin cephadm-cluster]$ ceph auth get-key client.admin AQByzTZjaEaCCRAAolAFknx1x/LgTYhNyy50cw== ``` 应用测试 ```powershell 进入专属测试文件目录 cd /data/kubernetes/ceph-csi/examples/cephfs/ ``` ```powershell 修改secret文件 secret.yaml ... stringData: # Required for statically provisioned volumes userID: fsclient userKey: AQDYCUpjmLjUDxAAYkfNGUG5FUYaFVE6jlFgrQ== # Required for dynamically provisioned volumes adminID: admin adminKey: AQByzTZjaEaCCRAAolAFknx1x/LgTYhNyy50cw== ... 修改： 根据提示信息修改 userID 和 userKey 的信息 这里的userKey无需进行base64加密 ``` ```powershell 修改SC配置 storageclass.yaml ... parameters: ... clusterID: d932ded6-3765-47c1-b0dc-e6957051e31a ... fsName: cephfs # (optional) Ceph pool into which volume data shall be stored pool: cephfs-data ... mounter: kernel ... mountOptions: - discard 修改： 根据提示信息修改 clusterID、fsName、pool、mounter 的信息 ``` ```powershell 应用资源清单文件 kubectl apply -f secret.yaml kubectl apply -f storageclass.yaml kubectl apply -f pvc.yaml kubectl apply -f pod.yaml ``` 测试效果 ```powershell 确定pvc和pv的效果 [root@kubernetes-master /data/kubernetes/ceph-csi/examples/cephfs]# kubectl get pvc NAME STATUS VOLUME CAPACITY ... csi-cephfs-pvc Bound pvc-6868b7d4-5daa-4c56-ba38-02334bc7a3aa 1Gi ... [root@kubernetes-master /data/kubernetes/ceph-csi/examples/cephfs]# kubectl get pv NAME CAPACITY ACCESS MODES ... pvc-6868b7d4-5daa-4c56-ba38-02334bc7a3aa 1Gi RWX ... ``` ```powershell 确定pod效果 [root@kubernetes-master /data/kubernetes/ceph-csi/examples/rbd]# kubectl describe pod csi-cephfs-demo-pod | grep -A3 Volumes: Volumes: mypvc: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: csi-cephfs-pvc ``` ```powershell pod环境中确认效果 [root@kubernetes-master /data/kubernetes/ceph-csi/deploy/cephfs/kubernetes]# kubectl exec -it csi-cephfs-demo-pod -- mount | grep ceph 10.0.0.13:6789,10.0.0.14:6789,10.0.0.15:6789:/volumes/csi/csi-vol-b7cd7860-8d68-4a20-91ee-b3031ef8c3a9/78b1871c-1274-4c47-8bd6-1f255a5275ed on /var/lib/www type ceph (rw,relatime,name=admin,secret=,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=cephfs,wsize=16777216) ``` 环境收尾 ```powershell 清理应用环境 cd /data/kubernetes/ceph-csi/examples/rbd kubectl delete -f pod.yaml -f pvc.yaml kubectl delete -f storageclass.yaml -f secret.yaml 清理sc环境 cd /data/kubernetes/ceph-csi/deploy/rbd/kubernetes kubectl delete -f ./ ``` ```powershell 清理ceph环境 ceph osd pool rm kubernetes kubernetes --yes-i-really-really-mean-it ceph auth rm client.kubernetes ``` **小结** ``` ``` ### 1.5.10 集群部署 学习目标 这一节，我们从 基础知识、简单实践、小结 三个方面来学习。 **基础知识** 简介 ```powershell Rook 是 Kubernetes 的开源云原生存储编排器，为各种存储解决方案提供平台、框架和支持，以与云原生环境进行原生集成。 Rook 将存储软件转变为自我管理、自我扩展和自我修复的存储服务。它通过自动化部署、引导、配置、供应、扩展、升级、迁移、灾难恢复、监控和资源管理来实现这一点。Rook 使用底层云原生容器管理、调度和编排平台提供的设施来履行其职责。 Rook 利用扩展点深度集成到云原生环境中，并为调度、生命周期管理、资源管理、安全性、监控和用户体验提供无缝体验。 Rook 是用 golang 实现的。Ceph 是用 C++ 实现的，其中数据路径经过高度优化。 ``` ```powershell Rook是一个自管理的分布式存储编排系统，可以为Kubernetes提供便利的存储解决方案，Rook本身并不提供存储，而是在Kubernetes和存储之间提供适配层，简化存储系统的部署和维护工作。目前，主要支持存储系统包括但不限于Ceph(主推)、Cassandra、NFS。 从本质上来说，Rook 是一个可以提供 Ceph 集群管理能力的 Operator。Rook 使用 CRD 一个控制器来对 Ceph 之类的资源进行部署和管理。 ``` ```powershell 最新版本: 1.10.3 官方地址：https://rook.io/docs/rook/v1.9/ceph-storage.html github: https://github.com/rook/rook ``` ![img](../../img/kubernetes/kubernetes_storage_ceph/kubernetes.png) 环境要求 ```powershell 1) K8s集群，1.16版本+ 2) K8s至少3个工作节点 3) 每个工作节点有一块未使用的硬盘 4) Rook支持部署Ceph Nautilus以上版本 ``` 准备工作 ```powershell 查看master节点的污点 [root@kubernetes-master ~]# kubectl describe nodes kubernetes-master | grep Taints: Taints: node-role.kubernetes.io/master:NoSchedule 移除master节点的污点 [root@kubernetes-master ~]# kubectl taint nodes kubernetes-master node-role.kubernetes.io/master:NoSchedule- node/kubernetes-master untainted 确认效果 [root@kubernetes-master ~]# kubectl describe nodes kubernetes-master | grep Taints: Taints: ``` 部署环境 ```powershell 获取软件 cd /data/kubernetes git clone https://github.com/rook/rook.git cd rook/deploy/examples ``` ```powershell 创建rook kubectl apply -f crds.yaml -f common.yaml -f operator.yaml ``` ```powershell 部署ceph kubectl apply -f cluster.yaml ``` ```powershell 部署Rook Ceph工具 kubectl apply -f toolbox.yaml ``` ```powershell 部署Ceph UI kubectl apply -f dashboard-external-https.yaml ``` 检查效果 ```powershell 检查pod [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get pod -n rook-ceph NAME READY ... csi-cephfsplugin-cs9z7 2/2 ... csi-cephfsplugin-l2bgm 2/2 ... csi-cephfsplugin-provisioner-657868fc8c-66nsc 5/5 ... csi-cephfsplugin-provisioner-657868fc8c-phdxm 5/5 ... csi-cephfsplugin-x7d7k 2/2 ... csi-rbdplugin-425nc 2/2 ... csi-rbdplugin-kthc2 2/2 ... csi-rbdplugin-provisioner-9d6787bcd-4pzvf 5/5 ... csi-rbdplugin-provisioner-9d6787bcd-q6gnf 5/5 ... csi-rbdplugin-zt2qc 2/2 ... rook-ceph-crashcollector-kubernetes-master-6795cf77b5-k6m98 1/1 ... rook-ceph-crashcollector-kubernetes-node1-7986cf44b-m7fdv 1/1 ... rook-ceph-crashcollector-kubernetes-node2-7d666d4f7c-lhvcw 1/1 ... rook-ceph-mgr-a-567fc6fbf9-tr9kq 2/2 ... rook-ceph-mgr-b-56496f5b65-7mtzb 2/2 ... rook-ceph-mon-a-74dd858bcf-rn6bs 1/1 ... rook-ceph-mon-b-7dbffc8dd8-ddvsl 1/1 ... rook-ceph-mon-c-6f7567b6dc-n7c47 1/1 ... rook-ceph-operator-785cc8f794-l8dc5 1/1 ... rook-ceph-osd-0-98876dbd6-49wl4 1/1 ... rook-ceph-osd-1-c867b6ddc-clc6c 1/1 ... rook-ceph-osd-2-d74bd646-rhwbs 1/1 ... rook-ceph-osd-3-779b6dc974-q6lbm 1/1 ... rook-ceph-osd-4-5646549855-crjqf 1/1 ... rook-ceph-osd-5-7b96ffbf6c-jsdsm 1/1 ... rook-ceph-osd-prepare-kubernetes-master-fxtbp 0/1 ... rook-ceph-osd-prepare-kubernetes-node1-l4f2s 0/1 ... rook-ceph-osd-prepare-kubernetes-node2-k6wx4 0/1 ... rook-ceph-tools-7c8ddb978b-sqlj2 1/1 ... ``` ```powershell 检查svc [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get svc -n rook-ceph NAME TYPE ... PORT(S) AGE rook-ceph-mgr ClusterIP ... 9283/TCP 26m rook-ceph-mgr-dashboard ClusterIP ... 8443/TCP 26m rook-ceph-mgr-dashboard-external-https NodePort ... 8443:31825/TCP 44m rook-ceph-mon-a ClusterIP ... 6789/TCP,3300/TCP 36m rook-ceph-mon-b ClusterIP ... 6789/TCP,3300/TCP 35m rook-ceph-mon-c ClusterIP ... 6789/TCP,3300/TCP 29m ``` 查看dashboard ```powershell 获取Ceph Dashboard登录密码 [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 -d ^B7+jmpcdh+}u^[*@n)K ``` ```powershell 浏览器访问： https://10.0.0.19:31825，输入用户名admin，密码^B7+jmpcdh+}u^[*@n)K ``` ![image-20221016004235785](../../img/kubernetes/kubernetes_storage_ceph/image-20221016004235785.png) 命令环境监测 ```powershell 获取ceph-tools工具 [root@kubernetes-master ~]# kubectl get pod -n rook-ceph | grep tools rook-ceph-tools-7c8ddb978b-sqlj2 1/1 Running 0 18m 进入到专用工具环境 [root@kubernetes-master ~]# kubectl -n rook-ceph exec -it rook-ceph-tools-7c8ddb978b-sqlj2 -- /bin/bash bash-4.4$ ``` ```powershell 检查ceph环境 [root@kubernetes-master ~]# kubectl -n rook-ceph exec -it rook-ceph-tools-7c8ddb978b-sqlj2 -- /bin/bash bash-4.4$ ceph -s cluster: id: d20c23c6-4d4f-4926-a0ca-fed583a907a0 health: HEALTH_WARN clock skew detected on mon.c services: mon: 3 daemons, quorum a,b,c (age 17m) mgr: b(active, since 19m), standbys: a osd: 6 osds: 6 up (since 18m), 6 in (since 22m) data: pools: 1 pools, 1 pgs objects: 2 objects, 449 KiB usage: 92 MiB used, 120 GiB / 120 GiB avail pgs: 1 active+clean ``` ```powershell bash-4.4$ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 120 GiB 120 GiB 92 MiB 92 MiB 0.08 TOTAL 120 GiB 120 GiB 92 MiB 92 MiB 0.08 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 449 KiB 0 38 GiB ``` 部署rbd 和 cephfs环境 ```powershell 部署rbd环境 kubectl apply -f csi/rbd/storageclass.yaml 部署cephfs环境 kubectl apply -f filesystem.yaml kubectl apply -f csi/cephfs/storageclass.yaml ``` ```powershell 检查环境 [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY ... rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete ... rook-cephfs rook-ceph.cephfs.csi.ceph.com Delete ... ``` **简单实践** 存储测试 ```powershell 定制资源清单文件 pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pv-claim labels: app: wordpress spec: storageClassName: rook-ceph-block accessModes: - ReadWriteOnce resources: requests: storage: 20Gi ``` ```powershell 应用资源对象文件 kubectl apply -f pvc.yaml ``` ```powershell 确定效果 [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get pvc NAME STATUS VOLUME ... mysql-pv-claim Bound pvc-fa8d837b-a087-471f-8d00-0c8a9fe65835 ... [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl -n rook-ceph exec -it rook-ceph-tools-7c8ddb978b-sqlj2 -- rbd ls --pool replicapool -l NAME SIZE PARENT FMT PROT LOCK csi-vol-c7d28742-4ca9-11ed-bffb-d22d32d7a8e9 20 GiB 2 ``` wordpress测试 ```powershell 应用资源对象文件 kubectl apply -f mysql.yaml -f wordpress.yaml ``` ```powershell 获取pvc的效果 [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get pvc -l app=wordpress NAME STATUS VOLUME ... AGE mysql-pv-claim Bound pvc-38487272-e3cf-46a2-be06-0e20af42054f ... 25s wp-pv-claim Bound pvc-17da7479-cb6d-46b5-a4fe-0f230f0ae780 ... 25s 获取pod的效果 [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get pods -l app=wordpress NAME READY STATUS RESTARTS AGE wordpress-b98c66fff-gbxbc 1/1 Running 0 38s wordpress-mysql-79966d6c5b-6g82t 1/1 Running 0 38s 获取svc的效果 [root@kubernetes-master /data/kubernetes/rook/deploy/examples]# kubectl get svc -l app=wordpress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE wordpress LoadBalancer 10.98.101.125 80:31606/TCP 47s wordpress-mysql ClusterIP None 3306/TCP 47s ``` ```powershell 浏览器访问： http://10.0.0.19:31606，可以看到wordpress的部署页面 ``` ![image-20221016011450922](../../img/kubernetes/kubernetes_storage_ceph/image-20221016011450922.png) **小结** ``` ``` ================================================ FILE: docs/cloud/kubernetes/kubernetes_storage_volume.md ================================================ # kubernetes持久化存储卷 # 一、存储卷介绍 pod有生命周期，生命周期结束后pod里的数据会消失(如配置文件,业务数据等)。 **解决: 我们需要将数据与pod分离,将数据放在专门的存储卷上** pod在k8s集群的节点中是可以调度的, 如果pod挂了被调度到另一个节点,那么数据和pod的联系会中断。 **解决: 所以我们需要与集群节点分离的存储系统才能实现数据持久化** 简单来说: **volume提供了在容器上挂载外部存储的能力** # 二、存储卷的分类 kubernetes支持的存储卷类型非常丰富,使用下面的命令查看: ```powershell # kubectl explain pod.spec.volumes ``` 或者参考: https://kubernetes.io/docs/concepts/storage/ kubernetes支持的存储卷列表如下: - [awsElasticBlockStore](https://kubernetes.io/docs/concepts/storage/#awselasticblockstore) - [azureDisk](https://kubernetes.io/docs/concepts/storage/#azuredisk) - [azureFile](https://kubernetes.io/docs/concepts/storage/#azurefile) - [cephfs](https://kubernetes.io/docs/concepts/storage/#cephfs) - [cinder](https://kubernetes.io/docs/concepts/storage/#cinder) - [configMap](https://kubernetes.io/docs/concepts/storage/#configmap) - [csi](https://kubernetes.io/docs/concepts/storage/#csi) - [downwardAPI](https://kubernetes.io/docs/concepts/storage/#downwardapi) - [emptyDir](https://kubernetes.io/docs/concepts/storage/#emptydir) - [fc (fibre channel)](https://kubernetes.io/docs/concepts/storage/#fc) - [flexVolume](https://kubernetes.io/docs/concepts/storage/#flexVolume) - [flocker](https://kubernetes.io/docs/concepts/storage/#flocker) - [gcePersistentDisk](https://kubernetes.io/docs/concepts/storage/#gcepersistentdisk) - [gitRepo (deprecated)](https://kubernetes.io/docs/concepts/storage/#gitrepo) - [glusterfs](https://kubernetes.io/docs/concepts/storage/#glusterfs) - [hostPath](https://kubernetes.io/docs/concepts/storage/#hostpath) - [iscsi](https://kubernetes.io/docs/concepts/storage/#iscsi) - [local](https://kubernetes.io/docs/concepts/storage/#local) - [nfs](https://kubernetes.io/docs/concepts/storage/#nfs) - [persistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/#persistentvolumeclaim) - [projected](https://kubernetes.io/docs/concepts/storage/#projected) - [portworxVolume](https://kubernetes.io/docs/concepts/storage/#portworxvolume) - [quobyte](https://kubernetes.io/docs/concepts/storage/#quobyte) - [rbd](https://kubernetes.io/docs/concepts/storage/#rbd) - [scaleIO](https://kubernetes.io/docs/concepts/storage/#scaleio) - [secret](https://kubernetes.io/docs/concepts/storage/#secret) - [storageos](https://kubernetes.io/docs/concepts/storage/#storageos) - [vsphereVolume](https://kubernetes.io/docs/concepts/storage/#vspherevolume) 我们将上面的存储卷列表进行简单的分类: - 本地存储卷 - emptyDir pod删除,数据也会被清除, 用于数据的临时存储 - hostPath 宿主机目录映射(本地存储卷) - 网络存储卷 - NAS类 nfs等 - SAN类 iscsi,FC等 - 分布式存储 glusterfs,cephfs,rbd,cinder等 - 云存储 aws,azurefile等 # 三、存储卷的选择 市面上的存储产品种类繁多, 但按应用角度主要分为三类: - 文件存储 如:nfs,glusterfs,cephfs等 - 优点: 数据共享(多pod挂载可以同读同写) - 缺点: 性能较差 - 块存储 如: iscsi,rbd等 - 优点: 性能相对于文件存储好 - 缺点: 不能实现数据共享（部分） - 对象存储 如: ceph对象存储 - 优点: 性能好, 数据共享 - 缺点: 使用方式特殊,支持较少 面对kubernetes支持的形形色色的存储卷,如何选择成了难题。在选择存储时,我们要抓住核心需求: - 数据是否需要持久性 - 数据可靠性 如存储集群节点是否有单点故障,数据是否有副本等 - 性能 - 扩展性 如是否能方便扩容,应对数据增长的需求 - 运维难度 存储的运维难度是比较高的,尽量选择稳定的开源方案或商业产品 - 成本 总之, 存储的选择是需要考虑很多因素的, 熟悉各类存储产品, 了解它们的优缺点，结合自身需求才能选择合适自己的。 # 四、本地存储卷之emptyDir - 应用场景 实现pod内容器之间数据共享 - 特点 随着pod被删除，该卷也会被删除 1.创建yaml文件 ```powershell [root@k8s-master1 ~]# vim volume-emptydir.yml apiVersion: v1 kind: Pod metadata: name: volume-emptydir spec: containers: - name: write image: centos imagePullPolicy: IfNotPresent command: ["bash","-c","echo haha > /data/1.txt ; sleep 6000"] volumeMounts: - name: data mountPath: /data - name: read image: centos imagePullPolicy: IfNotPresent command: ["bash","-c","cat /data/1.txt; sleep 6000"] volumeMounts: - name: data mountPath: /data volumes: - name: data emptyDir: {} ``` 2.基于yaml文件创建pod ```powershell [root@k8s-master1 ~]# kubectl apply -f volume-emptydir.yml pod/volume-emptydir created ``` 3.查看pod启动情况 ```powershell [root@k8s-master1 ~]# kubectl get pods |grep volume-emptydir NAME READY STATUS RESTARTS AGE volume-emptydir 2/2 Running 0 15s ``` 4.查看pod描述信息 ```powershell [root@k8s-master1 ~]# kubectl describe pod volume-emptydir | tail -10 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 50s default-scheduler Successfully assigned default/volume-emptydir to k8s-worker1 Normal Pulling 50s kubelet Pulling image "centos:centos7" Normal Pulled 28s kubelet Successfully pulled image "centos:centos7" in 21.544912361s Normal Created 28s kubelet Created container write Normal Started 28s kubelet Started container write Normal Pulled 28s kubelet Container image "centos:centos7" already present on machine Normal Created 28s kubelet Created container read Normal Started 28s kubelet Started container read ``` 5.验证 ```powershell [root@k8s-master1 ~]# kubectl logs volume-emptydir -c write [root@k8s-master1 ~]# kubectl logs volume-emptydir -c read haha ``` # 五、本地存储卷之hostPath - 应用场景 pod内与集群节点目录映射（pod中容器想访问节点上数据，例如监控，只有监控访问到节点主机文件才能知道集群节点主机状态） - 缺点 如果集群节点挂掉，控制器在另一个集群节点拉起容器，数据就会变成另一台集群节点主机的了（无法实现数据共享) 1.创建yaml文件 ~~~powershell [root@k8s-master1 ~]# vim volume-hostpath.yml apiVersion: v1 kind: Pod metadata: name: volume-hostpath spec: containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ["/bin/sh","-c","echo haha > /data/1.txt ; sleep 600"] volumeMounts: - name: data mountPath: /data volumes: - name: data hostPath: path: /opt type: Directory ~~~ 2.基于yaml文件创建pod ~~~powershell [root@k8s-master1 ~]# kubectl apply -f volume-hostpath.yml pod/volume-hostpath created ~~~ 3.查看pod状态 ~~~powershell [root@k8s-master1 ~]# kubectl get pods -o wide |grep volume-hostpath volume-hostpath 1/1 Running 0 29s 10.224.194.120 k8s-worker1 可以看到pod是在k8s-worker1节点上 ~~~ 4.验证pod所在机器上的挂载文件 ~~~powershell [root@k8s-worker1 ~]# cat /opt/1.txt haha ~~~ # 六、网络存储卷之nfs 1.搭建nfs服务器 ~~~powershell [root@nfsserver ~]# mkdir -p /data/nfs [root@nfsserver ~]# vim /etc/exports /data/nfs *(rw,no_root_squash,sync) [root@nfsserver ~]# systemctl restart nfs-server [root@nfsserver ~]# systemctl enable nfs-server ~~~ 2.所有node节点安装nfs客户端相关软件包 ```powershell [root@k8s-worker1 ~]# yum install nfs-utils -y [root@k8s-worker2 ~]# yum install nfs-utils -y ``` 3.验证nfs可用性 ```powershell [root@node1 ~]# showmount -e 192.168.10.129 Export list for 192.168.10.129: /data/nfs * [root@node2 ~]# showmount -e 192.168.10.129 Export list for 192.168.10.129: /data/nfs * ``` 4.master节点上创建yaml文件 ```powershell [root@k8s-master1 ~]# vim volume-nfs.yml apiVersion: apps/v1 kind: Deployment metadata: name: volume-nfs spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15-alpine imagePullPolicy: IfNotPresent volumeMounts: - name: documentroot mountPath: /usr/share/nginx/html ports: - containerPort: 80 volumes: - name: documentroot nfs: server: 192.168.10.129 path: /data/nfs ``` 5.应用yaml创建 ```powershell [root@k8s-master1 ~]# kubectl apply -f volume-nfs.yml deployment.apps/nginx-deployment created ``` 6.在nfs服务器共享目录中创建验证文件 ```powershell [root@nfsserver ~]# echo "volume-nfs" > /data/nfs/index.html ``` 7.验证pod ```powershell [root@k8s-master1 ~]# kubectl get pod |grep volume-nfs volume-nfs-649d848b57-qg4bz 1/1 Running 0 10s volume-nfs-649d848b57-wrnpn 1/1 Running 0 10s [root@k8s-master1 ~]# kubectl exec -it volume-nfs-649d848b57-qg4bz -- /bin/sh / # ls /usr/share/nginx/html/ index.html / # cat /usr/share/nginx/html/index.html volume-nfs # 文件内容与nfs服务器上创建的一致 / # exit [root@k8s-master1 ~]# kubectl exec -it volume-nfs-649d848b57-wrnpn -- /bin/sh / # ls /usr/share/nginx/html/ index.html / # cat /usr/share/nginx/html/index.html volume-nfs # 文件内容与nfs服务器上创建的一致 / # exit ``` # 七、PV(持久存储卷)与PVC(持久存储卷声明) ## 7.1 认识pv与pvc kubernetes存储卷的分类太丰富了,每种类型都要写相应的接口与参数才行，这就让维护与管理难度加大。 persistenvolume(**PV**) 是配置好的一段存储(可以是任意类型的存储卷) - 也就是说将网络存储共享出来,配置定义成PV。 PersistentVolumeClaim(**PVC**)是用户pod使用PV的申请请求。 - 用户不需要关心具体的volume实现细节,只需要关心使用需求。 ## 7.2 pv与pvc之间的关系 - pv提供存储资源(生产者) - pvc使用存储资源(消费者) - 使用pvc绑定pv ![](../../img/kubernetes/kubernetes_storage_volume/6.png) ## 7.3 实现nfs类型pv与pvc 1.编写创建pv的YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pv-nfs.yml apiVersion: v1 kind: PersistentVolume # 类型为PersistentVolume(pv) metadata: name: pv-nfs # 名称 spec: capacity: storage: 1Gi # 大小 accessModes: - ReadWriteMany # 访问模式 nfs: path: /data/nfs # nfs共享目录 server: 192.168.10.129 # nfs服务器IP ~~~ 访问模式有3种 参考: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes - ReadWriteOnce 单节点读写挂载 - ReadOnlyMany 多节点只读挂载 - ReadWriteMany 多节点读写挂载 cephfs存储卷3种类型都支持, 我们要实现多个nginx跨节点之间的数据共享,所以选择ReadWriteMany模式。 2.创建pv并验证 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pv-nfs.yml persistentvolume/pv-nfs created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-nfs 1Gi RWX Retain Available 81s ~~~ 说明: - RWX为ReadWriteMany的简写 - Retain是回收策略 - Retain表示需要不使用了需要手动回收 - 参考: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy 3.编写创建pvc的YAML文件 ~~~powershell [root@k8s-master1 ~]# vim pvc-nfs.yml apiVersion: v1 kind: PersistentVolumeClaim # 类型为PersistentVolumeClaim(pvc) metadata: name: pvc-nfs # pvc的名称 spec: accessModes: - ReadWriteMany # 访问模式 resources: requests: storage: 1Gi # 大小要与pv的大小保持一致 ~~~ 4.创建pvc并验证 ~~~powershell [root@k8s-master1 ~]# kubectl apply -f pvc-nfs.yml persistentvolumeclaim/pvc-nfs created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-nfs Bound pv-nfs 1Gi RWX 38s ~~~ **注意:** STATUS必须为Bound状态(Bound状态表示pvc与pv绑定OK) 5.编写deployment的YMAL ~~~powershell [root@k8s-master1 ~]# vim deploy-nginx-nfs.yml apiVersion: apps/v1 kind: Deployment metadata: name: deploy-nginx-nfs spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15-alpine imagePullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - name: www mountPath: /usr/share/nginx/html volumes: - name: www persistentVolumeClaim: claimName: pvc-nfs ~~~ 6.应用YAML创建deploment ~~~powershell [root@k8s-master1 ~]# kubectl apply -f deploy-nginx-nfs.yml deployment.apps/deploy-nginx-nfs created ~~~ 7.验证pod ~~~powershell [root@k8s-master1 ~]# kubectl get pod |grep deploy-nginx-nfs deploy-nginx-nfs-6f9bc4546c-gbzcl 1/1 Running 0 1m46s deploy-nginx-nfs-6f9bc4546c-hp4cv 1/1 Running 0 1m46s ~~~ 8.验证pod内卷的数据 ~~~powershell [root@k8s-master1 ~]# kubectl exec -it deploy-nginx-nfs-6f9bc4546c-gbzcl -- /bin/sh / # ls /usr/share/nginx/html/ index.html / # cat /usr/share/nginx/html/index.html volume-nfs / # exit [root@k8s-master1 ~]# kubectl exec -it deploy-nginx-nfs-6f9bc4546c-hp4cv -- /bin/sh / # ls /usr/share/nginx/html/ index.html / # cat /usr/share/nginx/html/index.html volume-nfs / # exit ~~~ ## 7.4 subpath使用 subpath是指可以把相同目录中不同子目录挂载到容器中不同的目录中使用的方法。以下通过案例演示： ~~~powershell 编辑文件 # vim 01_create_pod.yaml 编辑后查看，保持内容一致即可 # cat 01_create_pod.yaml apiVersion: v1 kind: Pod metadata: name: pod1 spec: containers: - name: c1 image: busybox command: ["/bin/sleep","100000"] volumeMounts: - name: data mountPath: /opt/data1 subPath: data1 - name: data mountPath: /opt/data2 subPath: data2 volumes: - name: data persistentVolumeClaim: claimName: pvc-nfs ~~~ ~~~powershell 执行文件，创建pod # kubectl apply -f 01_create_pod.yaml pod/pod1 created ~~~ ~~~powershell 编辑文件 # vim 02_create_pvc.yaml 查看编辑后文件，保持内容一致即可 # cat 02_create_pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim # 类型为PersistentVolumeClaim(pvc) metadata: name: pvc-nfs # pvc的名称 spec: accessModes: - ReadWriteMany # 访问模式 resources: requests: storage: 1Gi # 大小要与pv的大小保持一致 ~~~ ~~~powershell 执行文件，创建pvc # kubectl apply -f 02_create_pvc.yaml persistentvolumeclaim/pvc-nfs created ~~~ ~~~powershell 编辑文件 # vim 03_create_pv_nfs.yaml 查看编辑后文件，保持内容一致，注意修改nfs服务器及其共享的目录 # cat 03_create_pv_nfs.yaml apiVersion: v1 kind: PersistentVolume # 类型为PersistentVolume(pv) metadata: name: pv-nfs # 名称 spec: capacity: storage: 1Gi # 大小 accessModes: - ReadWriteMany # 访问模式 nfs: path: /sdb # nfs共享目录 server: 192.168.10.214 ~~~ ~~~powershell 执行文件，创建pv # kubectl apply -f 03_create_pv_nfs.yaml persistentvolume/pv-nfs created ~~~ ~~~powershell 在nfs服务器查看pod中目录是否自动添加到nfs服务器/sdb目录中 [root@nfsserver ~]# ls /sdb data1 data2 ~~~ # 八、存储的动态供给 ## 8.1 什么是动态供给 每次使用存储要先创建pv, 再创建pvc，真累! 所以我们可以实现使用存储的动态供给特性。 * 静态存储需要用户申请PVC时保证容量和读写类型与预置PV的容量及读写类型完全匹配, 而动态存储则无需如此. * 管理员无需预先创建大量的PV作为存储资源 Kubernetes从1.4版起引入了一个新的资源对象StorageClass，可用于将存储资源定义为具有显著特性的类（Class）而不是具体 的PV。用户通过PVC直接向意向的类别发出申请，匹配由管理员事先创建的PV，或者由其按需为用户动态创建PV，这样就免去 了需要先创建PV的过程。 ## 8.2 使用NFS文件系统创建存储动态供给 PV对存储系统的支持可通过其插件来实现，目前，Kubernetes支持如下类型的插件。 官方地址：https://kubernetes.io/docs/concepts/storage/storage-classes/ 官方插件是不支持NFS动态供给的，但是我们可以用第三方的插件来实现 第三方插件地址: https://github.com/kubernetes-retired/external-storage ![1604838285045](../../img/kubernetes/kubernetes_storage_volume/1.png) ![1604838445318](../../img/kubernetes/kubernetes_storage_volume/2.png) ![1604838485281](../../img/kubernetes/kubernetes_storage_volume/3.png) ![1604838524969](../../img/kubernetes/kubernetes_storage_volume/4.png) 1.下载并创建storageclass ~~~powershell [root@k8s-master1 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/class.yaml [root@k8s-master1 ~]# mv class.yaml storageclass-nfs.yml ~~~ ~~~powershell [root@k8s-master1 ~]# cat storageclass-nfs.yml apiVersion: storage.k8s.io/v1 kind: StorageClass # 类型 metadata: name: nfs-client # 名称,要使用就需要调用此名称 provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # 动态供给插件 parameters: archiveOnDelete: "false" # 删除数据时是否存档，false表示不存档，true表示存档 ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f storageclass-nfs.yml storageclass.storage.k8s.io/managed-nfs-storage created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 10s # RECLAIMPOLICY pv回收策略，pod或pvc被删除后，pv是否删除还是保留。 # VOLUMEBINDINGMODE Immediate 模式下PVC与PV立即绑定，主要是不等待相关Pod调度完成，不关心其运行节点，直接完成绑定。相反的 WaitForFirstConsumer模式下需要等待Pod调度完成后进行PV绑定。 # ALLOWVOLUMEEXPANSION pvc扩容 ~~~ 2.下载并创建rbac 因为storage自动创建pv需要经过kube-apiserver，所以需要授权。 ~~~powershell [root@k8s-master1 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/rbac.yaml [root@k8s-master1 ~]# mv rbac.yaml storageclass-nfs-rbac.yaml ~~~ ~~~powershell [root@k8s-master1 ~]# cat storageclass-nfs-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f rbac.yaml serviceaccount/nfs-client-provisioner created clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created ~~~ 3.创建动态供给的deployment 需要一个deployment来专门实现pv与pvc的自动创建 ~~~powershell [root@k8s-master1 ~]# vim deploy-nfs-client-provisioner.yml apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccount: nfs-client-provisioner containers: - name: nfs-client-provisioner image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: k8s-sigs.io/nfs-subdir-external-provisioner - name: NFS_SERVER value: 192.168.10.129 - name: NFS_PATH value: /data/nfs volumes: - name: nfs-client-root nfs: server: 192.168.10.129 path: /data/nfs ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl apply -f deploy-nfs-client-provisioner.yml deployment.apps/nfs-client-provisioner created ~~~ ~~~powershell [root@k8s-master1 ~]# kubectl get pods |grep nfs-client-provisioner nfs-client-provisioner-5b5ddcd6c8-b6zbq 1/1 Running 0 34s ~~~ ~~~powershell 测试存储动态供给是否可用 # vim nginx-sc.yaml --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: selector: matchLabels: app: nginx serviceName: "nginx" replicas: 2 template: metadata: labels: app: nginx spec: imagePullSecrets: - name: huoban-harbor terminationGracePeriodSeconds: 10 containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "nfs-client" resources: requests: storage: 1Gi ~~~ ~~~powershell [root@k8s-master1 nfs]# kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-9c988bc46-pr55n 1/1 Running 0 95s web-0 1/1 Running 0 95s web-1 1/1 Running 0 61s ~~~ ~~~powershell [root@nfsserver ~]# ls /data/nfs/ default-www-web-0-pvc-c4f7aeb0-6ee9-447f-a893-821774b8d11f default-www-web-1-pvc-8b8a4d3d-f75f-43af-8387-b7073d07ec01 ~~~ **扩展：** ~~~powershell 批量下载文件： # for file in class.yaml deployment.yaml rbac.yaml ; do wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/$file ; done ~~~ # k8s集群存储解决方案 GlusterFS # 一、存储解决方案介绍 ## 1.1 GlusterFS - GlusterFS是一个开源的分布式文件系统 - 具有强大的横向扩展能力 - 通过扩展能够支持数PB存储容量和处理数千客户端 - GlusterFS借助TCP/IP或InfiniBandRDMA网络将物理分布的存储资源聚集在一起，使用单一全局命名空间来管理数据。 ## 1.2 Heketi - Heketi（https://github.com/heketi/heketi），是一个基于RESTful API的GlusterFS卷管理框架。 - Heketi可以方便地和云平台整合，提供RESTful API供Kubernetes调用，实现多GlusterFS集群的卷管理 - Heketi还有保证bricks和它对应的副本均匀分布在集群中的不同可用区的优点。 # 二、环境说明 ## 2.1 k8s集群 | kubeadm | kubelet | kubectl | docker | 节点数 | | -------- | -------- | -------- | -------- | --------------- | | v1.21.10 | v1.21.10 | v1.21.10 | 20.10-17 | 3;1master2slave | | 主机 | IP地址 | Heketi | | :------: | :-----------: | -------------------- | | master01 | 192.168.10.11 | heketi heketi-client | | worker01 | 192.168.10.12 | heketi-client | | worker02 | 192.168.10.13 | heketi-client | ## 2.2 GlusterFS集群 | 主机 | IP地址 | 硬盘 | 硬盘容量 | | ---- | ------------- | -------- | -------- | | g1 | 192.168.10.60 | /dev/sdb | 100G | | g2 | 192.168.10.61 | /dev/sdb | 100G | | g3 | 192.168.10.62 | /dev/sdb | 100G | # 三、GlusterFS集群部署 ## 3.1 主机准备 ### 3.1.1 主机名配置 ~~~powershell [root@localhost ~]# hostnamectl set-hostname gX X为1，2，3 ~~~ ### 3.1.2 IP配置 ~~~powershell [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=static IPADDR=192.168.10.6X NETMASK=255.255.255.0 GATEWAY=192.168.10.2 DNS1=119.29.29.29 X为0，1，2 ~~~ ### 3.1.3 主机名解析设置 ~~~powershell [root@localhost ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.60 g1 192.168.10.61 g2 192.168.10.62 g3 ~~~ ### 3.1.4 主机间免密登录设置 > 在g1主机操作，然后copy到其它主机即可。 ~~~powershell [root@g1 ~]# ssh-keygen -t rsa -f /root/.ssh/id_rsa -N '' Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:FuKHUe57eCqUjIwH9q9zXNsQt/6BHDNZcSOixGBHXs0 root@g1 The key's randomart image is: +---[RSA 2048]----+ | o+= ooo o | | . *.o .E+ . | | o = . | | o . =...o | | . = oo.So=. | | . = +oo+.= | | . + .o== . | | . = .+o . | | .+ .. .. | +----[SHA256]-----+ [root@g1 ~]# cd /root/.ssh [root@g1 .ssh]# ls id_rsa id_rsa.pub [root@g1 .ssh]# cp id_rsa.pub authorized_keys [root@g1 .ssh]# ls authorized_keys id_rsa id_rsa.pub [root@g1 .ssh]# cd .. [root@g1 ~]# scp -r /root/.ssh g2:/root [root@g1 ~]# scp -r /root/.ssh g3:/root ~~~ ### 3.1.5 硬盘准备 #### 3.1.5.1 查看硬盘 > 所有GlusterFS集群节点全部操作，仅在g1主机演示操作方法。 ~~~powershell [root@gX ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 19G 0 part ├─centos-root 253:0 0 17G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 252:16 0 100G 0 disk ~~~ #### 3.1.5.2 格式化硬盘 ~~~powershell [root@gX ~]# mkfs.xfs /dev/sdb meta-data=/dev/sdb isize=512 agcount=4, agsize=6553600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=26214400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=12800, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 ~~~ ### 3.1.6 硬盘自动挂载准备 #### 3.1.6.1 准备挂载目录 ~~~powershell [root@g1 ~]# mkdir /glustersdb [root@g2 ~]# mkdir /glustersdb [root@g3~]# mkdir /glustersdb ~~~ #### 3.1.6.2 修改/etc/fstab文件实现自动挂载 ~~~powershell [root@gX ~]# cat /etc/fstab ...... /dev/sdb /glustersdb xfs defaults 0 0 ~~~ ~~~powershell 挂载所有 [root@gX ~]# mount -a 查看文件系统挂载情况 [root@gX ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/mapper/centos-root 17G 1.1G 16G 7% / devtmpfs 988M 0 988M 0% /dev tmpfs 1000M 0 1000M 0% /dev/shm tmpfs 1000M 8.6M 991M 1% /run tmpfs 1000M 0 1000M 0% /sys/fs/cgroup /dev/vda1 1014M 133M 882M 14% /boot tmpfs 98M 0 98M 0% /run/user/0 /dev/sdb 100G 33M 100G 1% /glustersdb ~~~ ## 3.2 安全设置 ### 3.2.1 firewalld设置 ~~~powershell [root@gX ~]# systemctl disable firewalld [root@gX ~]# systemctl stop firewalld [root@gX ~]# firewall-cmd --state not running ~~~ ### 3.2.2 SELinux设置 > 所有主机均要修改，修改后，请重启系统让修改生效。 ~~~powershell [root@gX ~]# sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ## 3.3 时间同步设置 ~~~powershell [root@gX ~]# crontab -l 0 */1 * * * ntpdate time1.aliyun.com ~~~ ## 3.4 GlusterFS安装 ### 3.4.1 YUM源准备 ~~~powershell [root@gX ~]# yum -y install centos-release-gluster [root@gX ~]# ls /etc/yum.repos.d/ CentOS-Gluster-7.repo CentOS-Storage-common.repo tuna.repo ~~~ ### 3.4.2 GlusterFS安装 > 关于软件的依赖，待补充。 ~~~powershell [root@gX ~]# yum -y install glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma fuse ~~~ ### 3.4.3 启动GlusterFS 服务 ~~~powershell [root@gX ~]# systemctl enable glusterd [root@gX ~]# systemctl start glusterd ~~~ ## 3.5 GlusterFS集群配置 > 在GlusterFS集群g1主机上添加g2和g3 2台主机。 ~~~powershell [root@g1 ~]# gluster peer probe g2 peer probe: success. [root@g1 ~]# gluster peer probe g3 peer probe: success. ~~~ ~~~powershell [root@g1 ~]# gluster peer status Number of Peers: 2 Hostname: g2 Uuid: 7660736f-056b-414e-8b0c-b5272265946c State: Peer in Cluster (Connected) Hostname: g3 Uuid: 75b7c358-edbe-438c-ad72-2ce16ffabf9d State: Peer in Cluster (Connected) [root@g2 ~]# gluster peer status Number of Peers: 2 Hostname: g1 Uuid: 920e9070-1336-4bff-8bfd-eb6161d035d3 State: Peer in Cluster (Connected) Hostname: g3 Uuid: 75b7c358-edbe-438c-ad72-2ce16ffabf9d State: Peer in Cluster (Connected) [root@g3 ~]# gluster peer status Number of Peers: 2 Hostname: g1 Uuid: 920e9070-1336-4bff-8bfd-eb6161d035d3 State: Peer in Cluster (Connected) Hostname: g2 Uuid: 7660736f-056b-414e-8b0c-b5272265946c State: Peer in Cluster (Connected) ~~~ ## 3.6 添加复制卷验证GlusterFS集群可用性 > 如果是为K8S集群提供持久化存储，请不要再继续验证GlusterFS集群可用性或验证完成后，重新添加硬盘。 > 在GlusterFS集群任意节点均可完成 ### 3.6.1 创建复制卷 ~~~powershell [root@g1 ~]# gluster volume create k8s-test-volume replica 3 g1:/glustersdb/r1 g2:/glustersdb/r2 g3:/glustersdb/r3 volume create: k8s-test-volume: success: please start the volume to access data [root@g1 ~]# ls /glustersdb r1 [root@g2 ~]# ls /glustersdb r2 [root@g3 ~]# ls /glustersdb r3 ~~~ ### 3.6.2 启动复制卷 ~~~powershell [root@g1 ~]# gluster volume start k8s-test-volume volume start: k8s-test-volume: success ~~~ ### 3.6.3 查询复制卷状态 ~~~powershell [root@g1 ~]# gluster volume status k8s-test-volume Status of volume: k8s-test-volume Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick g1:/glustersdb/r1 49152 0 Y 6622 Brick g2:/glustersdb/r2 49152 0 Y 6518 Brick g3:/glustersdb/r3 49152 0 Y 6518 Self-heal Daemon on localhost N/A N/A Y 6643 Self-heal Daemon on g3 N/A N/A Y 6539 Self-heal Daemon on g2 N/A N/A Y 6539 Task Status of Volume k8s-test-volume ------------------------------------------------------------------------------ There are no active volume tasks ~~~ ### 3.6.4 查看复制卷信息 ~~~powershell [root@g1 ~]# gluster volume info k8s-test-volume Volume Name: k8s-test-volume Type: Replicate Volume ID: 0529c5f6-1ac0-40ea-a29c-6c4f85dc54cb Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: g1:/glustersdb/r1 Brick2: g2:/glustersdb/r2 Brick3: g3:/glustersdb/r3 Options Reconfigured: transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: off ~~~ ### 3.6.5 如果某一个brick不在线会影响客户端挂载(可选) > 设置后，可以允许volume中的某块brick不在线的情况 ~~~powershell [root@g1 glusterfs]# gluster volume set k8s-test-volume cluster.server-quorum-type none volume set: success [root@g1 glusterfs]# gluster volume set k8s-test-volume cluster.quorum-type none volume set: success ~~~ ### 3.6.6 限额问题(可选) ~~~powershell [root@g1 ~]# gluster volume quota k8s-test-volume enable volume quota : success [root@g1 ~]# gluster volume quota k8s-test-volume limit-usage / 10GB volume quota : success ~~~ ## 3.7 在k8s集群工作节点验证GlusterFS集群可用性 > 由于仅使用一个工作节点验证GlusterFS集群可用性，因此没有必要为所有工作节点全部安装GlusterFS客户端。 ### 3.7.1 准备YUM源 ~~~powershell [root@worker01 ~]# yum -y install centos-release-gluster ~~~ ### 3.7.2 在k8s集群work1节点安装glusterfs ~~~powershell [root@worker01 ~]# yum -y install glusterfs glusterfs-fuse ~~~ ### 3.7.3 创建用于挂载目录 ~~~powershell [root@worker01 ~]# mkdir /k8s-glusterfs-test-volume ~~~ ### 3.7.4 手动挂载GlusterFS集群中的复制卷 > 如果使用主机名挂载，g1,g2,g3主机名需要添加到解析。 ~~~powershell [root@worker01 ~]# mount -t glusterfs g1:/k8s-test-volume /k8s-glusterfs-test-volume ~~~ ### 3.7.5 验证挂载情况 ~~~powershell [root@worker01 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 ...... g1:/k8s-test-volume 100G 1.1G 99G 2% /k8s-glusterfs-test-volume ~~~ ### 3.7.6 验证完成后需要卸载 ~~~powershell [root@worker01 ~]# umount /k8s-glusterfs-test-volume ~~~ # 四、Heketi安装 >heketi是为glusterfs提供RESETFUL的API， 相当于给glusterfs和k8s之间架通了桥梁。k8s集群可以通过heketi提供的RESETFUL API完成对Glusterfs的PV申请和管理。 ## 4.1 配置Heketi YUM源 > k8s集群所有节点均需要 ~~~powershell [root@master01 ~]# yum -y install centos-release-gluster [root@worker01 ~]# yum -y install centos-release-gluster [root@worker02 ~]# yum -y install centos-release-gluster ~~~ ## 4.2 安装Heketi ### 4.2.1 k8s集群master节点安装 ~~~powershell [root@master01 ~]# yum -y install heketi heketi-client ~~~ ### 4.2.2 k8s集群工作节点安装 ~~~powershell [root@worker01 ~]# yum -y install heketi-client [root@worker02 ~]# yum -y install heketi-client ~~~ ## 4.3 在k8s集群master节点修改Heketi配置文件 ### 4.3.1 在k8s集群master节点查看并备份文件 ~~~powershell [root@master01 ~]# ls /etc/heketi/ heketi.json [root@master01 ~]# cp /etc/heketi/heketi.json{,.bak} [root@master01 ~]# ls /etc/heketi/ heketi.json heketi.json.bak ~~~ ### 4.3.2 在k8s集群master节点修改配置文件 ~~~powershell [root@master01 ~]# cat /etc/heketi/heketi.json { "_port_comment": "Heketi Server Port Number", "port": "18080", 修改为18080，防止与其它端口冲突 "_use_auth": "Enable JWT authorization. Please enable for deployment", "use_auth": true, 开启用户认证 "_jwt": "Private keys for access", "jwt": { "_admin": "Admin has access to all APIs", "admin": { "key": "adminkey" 用户认证的key }, "_user": "User only has access to /volumes endpoint", "user": { "key": "My Secret" } }, "_glusterfs_comment": "GlusterFS Configuration", "glusterfs": { "_executor_comment": [ "Execute plugin. Possible choices: mock, ssh", "mock: This setting is used for testing and development.", " It will not send commands to any node.", "ssh: This setting will notify Heketi to ssh to the nodes.", " It will need the values in sshexec to be configured.", "kubernetes: Communicate with GlusterFS containers over", " Kubernetes exec api." ], "executor": "ssh", 访问glusterfs集群的方法 "_sshexec_comment": "SSH username and private key file information", "sshexec": { "keyfile": "/etc/heketi/heketi_key", 访问glusterfs集群使用的私钥，需要提前在k8s集群master节点生成并copy到glusterfs集群所有节点,需要从/root/.ssh/id_rsa复制到此处才可以使用。 "user": "root", 认证使用的用户 "port": "22", ssh连接使用的端口 "fstab": "/etc/fstab" 挂载的文件系统 }, "_kubeexec_comment": "Kubernetes configuration", "kubeexec": { "host" :"https://kubernetes.host:8443", "cert" : "/path/to/crt.file", "insecure": false, "user": "kubernetes username", "password": "password for kubernetes user", "namespace": "OpenShift project or Kubernetes namespace", "fstab": "Optional: Specify fstab file on node. Default is /etc/fstab" }, "_db_comment": "Database file name", "db": "/var/lib/heketi/heketi.db", 数据库位置 "_loglevel_comment": [ "Set log level. Choices are:", " none, critical, error, warning, info, debug", "Default is warning" ], "loglevel" : "warning" 修改日志级别 } } ~~~ > 需要说明的是，heketi有三种executor，分别为mock、ssh、kubernetes，建议在测试环境使用mock，生产环境使用ssh，当glusterfs以容器的方式部署在kubernetes上时，才使用kubernetes。我们这里将glusterfs和heketi独立部署，使用ssh的方式。 ## 4.4 配置ssh密钥 在上面我们配置heketi的时候使用了ssh的executor，那么就需要heketi服务器能通过ssh密钥的方式连接到所有glusterfs节点进行管理操作，所以需要先生成ssh密钥 ### 4.4.1 生成密钥 ~~~powershell [root@master01 ~]# ssh-keygen -t rsa -f /root/.ssh/id_rsa -N '' Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:EG1ql3V+ExAqMTubL4AqSgCMGZ0mZliSsVH2n83TT28 root@master01 The key's randomart image is: +---[RSA 2048]----+ |**+. ..o oo | |*X+. .o+... . | |B+ . .o+o.o . | |. oo=o* . o | |. ..+.S . .. . | |. . . o o . | |... . . . E | |o. . . | |. | +----[SHA256]-----+ ~~~ ### 4.4.2 复制密钥到远程主机 ~~~powershell [root@master01 ~]# ssh-copy-id 192.168.10.60 [root@master01 ~]# ssh-copy-id 192.168.10.61 [root@master01 ~]# ssh-copy-id 192.168.10.62 ~~~ ### 4.4.3 验证密钥的可用性 ~~~powershell [root@master01 ~]# ssh 192.168.10.60 Last login: Wed Jan 29 20:17:39 2020 from 192.168.10.1 [root@g1 ~]# exit 登出 Connection to 192.168.10.60 closed. [root@master01 ~]# ssh 192.168.10.61 Last login: Wed Jan 29 20:17:51 2020 from 192.168.10.1 [root@g2 ~]# exit 登出 Connection to 192.168.10.61 closed. [root@master01 ~]# ssh 192.168.10.62 Last login: Wed Jan 29 20:18:04 2020 from 192.168.10.1 [root@g3 ~]# exit 登出 Connection to 192.168.10.62 closed. ~~~ ### 4.4.4 复制私密到/etc/heketi目录 ~~~powershell [root@master01 ~]# cp .ssh/id_rsa /etc/heketi/heketi_key [root@master01 ~]# ls /etc/heketi/ heketi.json heketi.json.bak heketi_key ~~~ ## 4.5 启动Heketi >默认yum安装后，/etc/heketi及/var/lib/heketi目录所有者是root, 但是安装提供的service文件的user又是heketi. 导致不修改权限就是启动不起来,因此需要修改权限再启动服务。 ~~~powershell [root@master01 heketi]# chown heketi:heketi /etc/heketi/ -R || chown heketi:heketi /var/lib/heketi -R ~~~ ~~~powershell [root@master01 ~]# systemctl enable heketi [root@master01 ~]# systemctl start heketi [root@master01 ~]# systemctl status heketi ● heketi.service - Heketi Server Loaded: loaded (/usr/lib/systemd/system/heketi.service; enabled; vendor preset: disabled) Active: active (running) since 三 2020-01-29 22:13:52 CST; 2min 31s ago Main PID: 23664 (heketi) Tasks: 11 Memory: 8.8M CGroup: /system.slice/heketi.service └─23664 /usr/bin/heketi --config=/etc/heketi/heketi.json 1月 29 22:13:52 master01 systemd[1]: Started Heketi Server. 1月 29 22:13:52 master01 heketi[23664]: Heketi 9.0.0 1月 29 22:13:52 master01 heketi[23664]: Authorization loaded 1月 29 22:13:52 master01 heketi[23664]: Listening on port 18080 ~~~ ## 4.6 验证Heketi ~~~powershell 验证是否可以创建集群 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 --json cluster create {"id":"1c8824939237ea79aa17a127e958fc92","nodes":[],"volumes":[],"block":true,"file":true,"blockvolumes":[]} 删除已创建的集群 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 --json cluster delete 1c8824939237ea79aa17a127e958fc92 Cluster 1c8824939237ea79aa17a127e958fc92 deleted ~~~ ## 4.7 创建集群 ~~~powershell [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 --json cluster create {"id":"dd456dbc15f1206e980fdb5345117085","nodes":[],"volumes":[],"block":true,"file":true,"blockvolumes":[]} 说明 192.168.10.11 为在k8s集群master节点IP ~~~ ## 4.8 添加节点 ~~~powershell 添加g1 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 --json node add --cluster "dd456dbc15f1206e980fdb5345117085" --management-host-name 192.168.10.60 --storage-host-name 192.168.10.60 --zone 1 {"zone":1,"hostnames":{"manage":["192.168.10.60"],"storage":["192.168.10.60"]},"cluster":"dd456dbc15f1206e980fdb5345117085","id":"217899105fa01434f9f29625e7ad9cfb","state":"online","devices":[]} 添加g2 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 --json node add --cluster "dd456dbc15f1206e980fdb5345117085" --management-host-name 192.168.10.61 --storage-host-name 192.168.10.61 --zone 1 {"zone":1,"hostnames":{"manage":["192.168.10.61"],"storage":["192.168.10.61"]},"cluster":"dd456dbc15f1206e980fdb5345117085","id":"b8cb7ce3f753fea41bb170f2639a1554","state":"online","devices":[]} 添加g3 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 --json node add --cluster "dd456dbc15f1206e980fdb5345117085" --management-host-name 192.168.10.62 --storage-host-name 192.168.10.62 --zone 1 {"zone":1,"hostnames":{"manage":["192.168.10.62"],"storage":["192.168.10.62"]},"cluster":"dd456dbc15f1206e980fdb5345117085","id":"bd7637215a852092583d7e5cd84b6c9e","state":"online","devices":[]} ~~~ ~~~powershell 查看集群中node列表 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 node list Id:217899105fa01434f9f29625e7ad9cfb Cluster:dd456dbc15f1206e980fdb5345117085 Id:b8cb7ce3f753fea41bb170f2639a1554 Cluster:dd456dbc15f1206e980fdb5345117085 Id:bd7637215a852092583d7e5cd84b6c9e Cluster:dd456dbc15f1206e980fdb5345117085 ~~~ ## 4.9 添加设备 ### 4.9.1 错误的示范 ~~~powershell [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 device add --name "/dev/sdb" --node 217899105fa01434f9f29625e7ad9cfb Error: Setup of device /dev/sdb failed (already initialized or contains data?): Can't open /dev/sdb exclusively. Mounted filesystem? Can't open /dev/sdb exclusively. Mounted filesystem? ~~~ ### 4.9.2 添加新硬盘 > 如果没有做使用测试，可以不操作此步骤。 ~~~powershell [root@g1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 19G 0 part ├─centos-root 253:0 0 17G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 252:16 0 100G 0 disk /glustersdb sdc 252:32 0 50G 0 disk [root@g2 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 19G 0 part ├─centos-root 253:0 0 17G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 252:16 0 100G 0 disk /glustersdb sdc 252:32 0 50G 0 disk [root@g3 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 19G 0 part ├─centos-root 253:0 0 17G 0 lvm / └─centos-swap 253:1 0 2G 0 lvm [SWAP] sdb 252:16 0 100G 0 disk /glustersdb sdc 252:32 0 50G 0 disk ~~~ ### 4.9.3 添加GlusterFS集群节点中的设备到Heketi集群 ~~~powershell [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 device add --name "/dev/sdc" --node 217899105fa01434f9f29625e7ad9cfb Device added successfully [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 device add --name "/dev/sdc" --node b8cb7ce3f753fea41bb170f2639a1554 Device added successfully [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 device add --name "/dev/sdc" --node bd7637215a852092583d7e5cd84b6c9e Device added successfully ~~~ ### 4.9.4 验证节点及设备添加情况 ~~~powershell [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 topology info 或 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 topology info --json ~~~ ## 4.10 测试通过Heketi在GlusterFS集群中添加volume ### 4.10.1 在k8s集群master节点查看是否有volume ~~~powershell [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 volume list ~~~ ### 4.10.2 在k8s集群master节点创建volume ~~~powershell 获取帮助 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 volume create -h ~~~ ~~~powershell 创建一个复制卷，共5G大小。卷的名称自动生成。 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 volume create --size=5 --replica=2 Name: vol_80539c6510a73f70ad3453c221901334 Size: 5 Volume Id: 80539c6510a73f70ad3453c221901334 Cluster Id: dd456dbc15f1206e980fdb5345117085 Mount: 192.168.10.60:vol_80539c6510a73f70ad3453c221901334 Mount Options: backup-volfile-servers=192.168.10.61,192.168.10.62 Block: false Free Size: 0 Reserved Size: 0 Block Hosting Restriction: (none) Block Volumes: [] Durability Type: replicate Distribute Count: 1 Replica Count: 2 ~~~ ~~~powershell 验证卷是否创建 [root@master01 ~]# heketi-cli --user admin --secret adminkey --server http://192.168.10.11:18080 volume list Id:80539c6510a73f70ad3453c221901334 Cluster:dd456dbc15f1206e980fdb5345117085 Name:vol_80539c6510a73f70ad3453c221901334 在GlusterFS集群节点中验证即可看到已创建的卷。 [root@g1 ~]# gluster volume list k8s-test-volume vol_80539c6510a73f70ad3453c221901334 ~~~ # 五、K8S集群使用GlusterFS集群 > 提示：k8s中使用glusterfs的时候， 会根据pvc的申请自动创建对应的pv， 然后绑定。 ## 5.1 在k8s集群master节点创建storageclass资源清单文件 ~~~powershell [root@master01 yaml]# cat storageclass-gluserfs.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: glusterfs provisioner: kubernetes.io/glusterfs #表示存储分配器，需要根据后端存储的不同而变更 parameters: resturl: "http://192.168.10.11:18080" #heketi API服务提供的URL,为k8s集群master节点IP restauthenabled: "true" #可选参数，默认为"false",heketi服务开启认证时，必须设置为"true" restuser: "admin" #可选参数，开启认证时设置相应用户名 restuserkey: "adminkey" #可选，开启认证时设置密码 volumetype: "replicate:2" #可选参数，设置卷类型及其参数，如果未分配卷类型，则有分配器决定卷类型；如”volumetype: replicate:3”表示3副本的replicate卷，”volumetype: disperse:4:2”表示disperse卷，其中‘4’是数据，’2’是冗余校验，”volumetype: none”表示distribute卷 ~~~ ## 5.2 在k8s集群master节点应用上述资源清单文件 ~~~powershell [root@master01 yaml]# kubectl apply -f storageclass-gluserfs.yaml storageclass.storage.k8s.io/glusterfs created ~~~ ## 5.3 在k8s集群master节点验证是否创建storageclass存储对象 ~~~powershell [root@master01 yaml]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE glusterfs kubernetes.io/glusterfs Delete Immediate false 48s ~~~ ## 5.4 在k8s集群master节点创建用于创建PVC的资源清单文件 ~~~powershell [root@master01 yaml]# cat glusterfs-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: glusterfs-mysql namespace: default annotations: volume.beta.kubernetes.io/storage-class: "glusterfs" spec: accessModes: - ReadWriteMany resources: requests: storage: 2Gi ~~~ ## 5.5 在k8s集群master节点应用上述资源清单文件 ~~~powershell [root@master01 yaml]# kubectl apply -f glusterfs-pvc.yaml persistentvolumeclaim/glusterfs-mysql created ~~~ ## 5.6 在k8s集群master节点验证是否创建PVC ~~~powershell [root@master01 yaml]# kubectl get pv,pvc NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-77d6fca6-f284-49fb-a0f3-8f5664690562 2Gi RWX Delete Bound default/glusterfs-mysql glusterfs 2s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/glusterfs-mysql Bound pvc-77d6fca6-f284-49fb-a0f3-8f5664690562 2Gi RWX glusterfs 3s ~~~ ## 5.7 在k8s集群master节点创建Pod时使用上述创建的PVC ~~~powershell [root@master01 yaml]# cat mysql.yaml apiVersion: v1 kind: Service metadata: name: mysql-svc labels: app: mysql-svc spec: ports: - port: 3306 name: mysql clusterIP: None selector: name: mysql --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql namespace: default spec: serviceName: mysql-svc selector: matchLabels: name: mysql replicas: 1 template: metadata: labels: name: mysql spec: containers: - name: mysql image: mysql:5.7 imagePullPolicy: IfNotPresent env: - name: MYSQL_ROOT_PASSWORD value: "123456" ports: - containerPort: 3306 name: mysql volumeMounts: - name: glusterfs-mysql-data mountPath: "/var/lib/mysql" volumes: - name: glusterfs-mysql-data persistentVolumeClaim: claimName: glusterfs-mysql ~~~ ~~~powershell [root@master01 yaml]# kubectl apply -f mysql.yaml service/mysql-svc created statefulset.apps/mysql created ~~~ ~~~powershell [root@master01 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE busybox-pod 1/1 Running 247 14d mysql-0 1/1 Running 0 27s nfs-client-provisioner-5786f95795-x7bcs 1/1 Running 1 30h [root@master01 ~]# kubectl exec -it mysql-0 sh # mysql -uroot -p123456 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.29 MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.01 sec) mysql> create database k8sonline; Query OK, 1 row affected (0.01 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | k8sonline | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.01 sec) ~~~ ~~~powershell 查看GlusterFS集群数据存储位置 在g1节点 [root@g1 ~]# gluster volume list vol_80539c6510a73f70ad3453c221901334 [root@g1 ~]# gluster volume info vol_80539c6510a73f70ad3453c221901334 Volume Name: vol_80539c6510a73f70ad3453c221901334 Type: Replicate Volume ID: 5df33cf0-093d-4a6c-9a2c-d2b4ec195c9e Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 192.168.10.61:/var/lib/heketi/mounts/vg_d62a7a4a632dd4864edc367c952d0fa9/brick_f7d134a34348c334a369b84604db9a40/brick Brick2: 192.168.10.60:/var/lib/heketi/mounts/vg_6e8d391aec35995a4ee82e53e986bf70/brick_b4caa8e338233c536fd98966eeccce98/brick Options Reconfigured: user.heketi.id: 80539c6510a73f70ad3453c221901334 transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: off 在g2节点 [root@g2 ~]# ls /var/lib/heketi/mounts/vg_d62a7a4a632dd4864edc367c952d0fa9/brick_834718f2a0236b913b3aa14609b34819/brick/ auto.cnf ib_buffer_pool k8sonline server-cert.pem ca-key.pem ibdata1 mysql server-key.pem ca.pem ib_logfile0 performance_schema sys client-cert.pem ib_logfile1 private_key.pem client-key.pem ibtmp1 public_key.pem ~~~ ## 5.8 关于storageclass资源清单的扩展 以上将userkey明文写入配置文件创建storageclass的方式，官方推荐将key使用secret保存。 ~~~powershell # glusterfs-secret.yaml内容如下： apiVersion: v1 kind: Secret metadata: name: heketi-secret namespace: default data: # base64 encoded password. E.g.: echo -n "mypassword" | base64 key: TFRTTkd6TlZJOEpjUndZNg== type: kubernetes.io/glusterfs ~~~ ~~~powershell # storageclass-glusterfs.yaml内容修改如下： apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: glusterfs provisioner: kubernetes.io/glusterfs parameters: resturl: "http://192.168.10.11:18080" clusterid: "dd456dbc15f1206e980fdb5345117085" restauthenabled: "true" restuser: "admin" secretNamespace: "default" secretName: "heketi-secret" #restuserkey: "adminkey" gidMin: "40000" gidMax: "50000" volumetype: "replicate:2" ~~~ ## 5.9 FAQ ### 问题 - heketi有些卷明明存在但是却删不了 直接删除heketi存储目录/var/lib/heketi/ 下的mounts/文件夹，然后> heketi.db 清空db文件，重新来 - Can't initialize physical volume "/dev/sdb1" of volume group "vg1" without –ff 这是因为没有卸载之前的vg和pv,使用命令vgremove,pvremove依次删除卷组，逻辑卷即可 ================================================ FILE: docs/cloud/kubernetes/kubernetes_traefik.md ================================================ # Kubernetes集群 服务暴露 Traefik # 一、认识traefik ## 1.1 traefik简介 - 参考链接: https://traefik.cn/ - [traefik的yaml文件](../../img/kubernetes/kubernetes_traefik/traefik.zip) - 是一个为了让部署微服务更加便捷而诞生的现代HTTP反向代理、负载均衡工具。 - 它支持多种后台 ([Docker](https://www.docker.com/), [Swarm](https://docs.docker.com/swarm), [Kubernetes](https://kubernetes.io/), [Marathon](https://mesosphere.github.io/marathon/), [Mesos](https://github.com/apache/mesos), [Consul](https://www.consul.io/), [Etcd](https://coreos.com/etcd/), [Zookeeper](https://zookeeper.apache.org/), [BoltDB](https://github.com/boltdb/bolt), Rest API, file…) 来自动化、动态的应用它的配置文件设置。 ![](../../img/kubernetes/kubernetes_traefik/traefik功能.png) ## 1.2 traefix 特性 - 非常快 - 无需安装其他依赖，通过Go语言编写的单一可执行文件 - 支持 Rest API - 多种后台支持：Docker, Swarm, Kubernetes, Marathon, Mesos, Consul, Etcd, 并且还会更多 - 后台监控, 可以监听后台变化进而自动化应用新的配置文件设置 - 配置文件热更新。无需重启进程 - 正常结束http连接 - 后端断路器 - 轮询，rebalancer 负载均衡 - Rest Metrics - 支持最小化官方docker 镜像 - 前、后台支持SSL - 清爽的AngularJS前端页面 - 支持Websocket - 支持HTTP/2 - 网络错误重试 - 支持[Let’s Encrypt](https://letsencrypt.org/) (自动更新HTTPS证书) - 高可用集群模式 ## 1.3 traefik与nginx ingress对比 ![20220419021708805](../../img/kubernetes/kubernetes_traefik/image-20220419021708805.png) ## 1.4 traefik核心概念及能力 Traefik是一个边缘路由器，它会拦截外部的请求并根据逻辑规则选择不同的操作方式，这些规则决定着这些请求到底该如何处理。Traefik提供自动发现能力，会实时检测服务，并自动更新路由规则。 ![](../../img/kubernetes/kubernetes_traefik/traefik能力.png) 从上图可知，请求首先会连接到entrypoints，然后分析这些请求是否与定义的rules匹配，如果匹配，则会通过一系列middlewares，再到对应的services上。 这就涉及到以下几个重要的核心组件。 - Providers - Entrypoints - Routers - Services - Middlewares 下面分别介绍一下： - Providers Providers是基础组件，Traefik的配置发现是通过它来实现的，它可以是协调器，容器引擎，云提供商或者键值存储。 Traefik通过查询Providers的API来查询路由的相关信息，一旦检测到变化，就会动态的更新路由。 - Entrypoints Entrypoints是Traefik的网络入口，它定义接收请求的接口，以及是否监听TCP或者UDP。 - Routers Routers主要用于分析请求，并负责将这些请求连接到对应的服务上去，在这个过程中，Routers还可以使用Middlewares来更新请求，比如在把请求发到服务之前添加一些Headers。 - Services Services负责配置如何到达最终将处理传入请求的实际服务。 - Middlewares Middlewares用来修改请求或者根据请求来做出一些判断（authentication, rate limiting, headers, …），中间件被附件到路由上，是一种在请求发送到你的服务之前（或者在服务的响应发送到客户端之前）调整请求的一种方法。 # 二、traefik部署 ## 2.1 获取traefik部署前置资源清单文件 ### 2.1.1 创建CRD资源 >本次Traefik 是部署在 kube-system Namespace 下，如果不想部署到配置的 Namespace，需要修改下面部署文件中的 Namespace 参数。 > >此yaml资源清单文件可在traefik.io网站直接复制使用：https://doc.traefik.io/traefik/v2.5/reference/dynamic-configuration/kubernetes-crd/#definitions ~~~powershell # vim traefik-crd.yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: ingressroutes.traefik.containo.us spec: group: traefik.containo.us names: kind: IngressRoute listKind: IngressRouteList plural: ingressroutes singular: ingressroute scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: IngressRoute is an Ingress CRD specification. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IngressRouteSpec is a specification for a IngressRouteSpec resource. properties: entryPoints: items: type: string type: array routes: items: description: Route contains the set of routes. properties: kind: enum: - Rule type: string match: type: string middlewares: items: description: MiddlewareRef is a ref to the Middleware resources. properties: name: type: string namespace: type: string required: - name type: object type: array priority: type: integer services: items: description: Service defines an upstream to proxy traffic. properties: kind: enum: - Service - TraefikService type: string name: description: Name is a reference to a Kubernetes Service object (for a load-balancer of servers), or to a TraefikService object (service load-balancer, mirroring, etc). The differentiation between the two is specified in the Kind field. type: string namespace: type: string passHostHeader: type: boolean port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true responseForwarding: description: ResponseForwarding holds configuration for the forward of the response. properties: flushInterval: type: string type: object scheme: type: string serversTransport: type: string sticky: description: Sticky holds the sticky configuration. properties: cookie: description: Cookie holds the sticky configuration based on cookie. properties: httpOnly: type: boolean name: type: string sameSite: type: string secure: type: boolean type: object type: object strategy: type: string weight: description: Weight should only be specified when Name references a TraefikService object (and to be precise, one that embeds a Weighted Round Robin). type: integer required: - name type: object type: array required: - kind - match type: object type: array tls: description: "TLS contains the TLS certificates configuration of the routes. To enable Let's Encrypt, use an empty TLS struct, e.g. in YAML: \n \t tls: {} # inline format \n \t tls: \t secretName: # block format" properties: certResolver: type: string domains: items: description: Domain holds a domain name with SANs. properties: main: type: string sans: items: type: string type: array type: object type: array options: description: Options is a reference to a TLSOption, that specifies the parameters of the TLS connection. properties: name: type: string namespace: type: string required: - name type: object secretName: description: SecretName is the name of the referenced Kubernetes Secret to specify the certificate details. type: string store: description: Store is a reference to a TLSStore, that specifies the parameters of the TLS store. properties: name: type: string namespace: type: string required: - name type: object type: object required: - routes type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: ingressroutetcps.traefik.containo.us spec: group: traefik.containo.us names: kind: IngressRouteTCP listKind: IngressRouteTCPList plural: ingressroutetcps singular: ingressroutetcp scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: IngressRouteTCP is an Ingress CRD specification. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IngressRouteTCPSpec is a specification for a IngressRouteTCPSpec resource. properties: entryPoints: items: type: string type: array routes: items: description: RouteTCP contains the set of routes. properties: match: type: string middlewares: description: Middlewares contains references to MiddlewareTCP resources. items: description: ObjectReference is a generic reference to a Traefik resource. properties: name: type: string namespace: type: string required: - name type: object type: array services: items: description: ServiceTCP defines an upstream to proxy traffic. properties: name: type: string namespace: type: string port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true proxyProtocol: description: ProxyProtocol holds the ProxyProtocol configuration. properties: version: type: integer type: object terminationDelay: type: integer weight: type: integer required: - name - port type: object type: array required: - match type: object type: array tls: description: "TLSTCP contains the TLS certificates configuration of the routes. To enable Let's Encrypt, use an empty TLS struct, e.g. in YAML: \n \t tls: {} # inline format \n \t tls: \t secretName: # block format" properties: certResolver: type: string domains: items: description: Domain holds a domain name with SANs. properties: main: type: string sans: items: type: string type: array type: object type: array options: description: Options is a reference to a TLSOption, that specifies the parameters of the TLS connection. properties: name: type: string namespace: type: string required: - name type: object passthrough: type: boolean secretName: description: SecretName is the name of the referenced Kubernetes Secret to specify the certificate details. type: string store: description: Store is a reference to a TLSStore, that specifies the parameters of the TLS store. properties: name: type: string namespace: type: string required: - name type: object type: object required: - routes type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: ingressrouteudps.traefik.containo.us spec: group: traefik.containo.us names: kind: IngressRouteUDP listKind: IngressRouteUDPList plural: ingressrouteudps singular: ingressrouteudp scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: IngressRouteUDP is an Ingress CRD specification. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IngressRouteUDPSpec is a specification for a IngressRouteUDPSpec resource. properties: entryPoints: items: type: string type: array routes: items: description: RouteUDP contains the set of routes. properties: services: items: description: ServiceUDP defines an upstream to proxy traffic. properties: name: type: string namespace: type: string port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true weight: type: integer required: - name - port type: object type: array type: object type: array required: - routes type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: middlewares.traefik.containo.us spec: group: traefik.containo.us names: kind: Middleware listKind: MiddlewareList plural: middlewares singular: middleware scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Middleware is a specification for a Middleware resource. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: MiddlewareSpec holds the Middleware configuration. properties: addPrefix: description: AddPrefix holds the AddPrefix configuration. properties: prefix: type: string type: object basicAuth: description: BasicAuth holds the HTTP basic authentication configuration. properties: headerField: type: string realm: type: string removeHeader: type: boolean secret: type: string type: object buffering: description: Buffering holds the request/response buffering configuration. properties: maxRequestBodyBytes: format: int64 type: integer maxResponseBodyBytes: format: int64 type: integer memRequestBodyBytes: format: int64 type: integer memResponseBodyBytes: format: int64 type: integer retryExpression: type: string type: object chain: description: Chain holds a chain of middlewares. properties: middlewares: items: description: MiddlewareRef is a ref to the Middleware resources. properties: name: type: string namespace: type: string required: - name type: object type: array type: object circuitBreaker: description: CircuitBreaker holds the circuit breaker configuration. properties: expression: type: string type: object compress: description: Compress holds the compress configuration. properties: excludedContentTypes: items: type: string type: array type: object contentType: description: ContentType middleware - or rather its unique `autoDetect` option - specifies whether to let the `Content-Type` header, if it has not been set by the backend, be automatically set to a value derived from the contents of the response. As a proxy, the default behavior should be to leave the header alone, regardless of what the backend did with it. However, the historic default was to always auto-detect and set the header if it was nil, and it is going to be kept that way in order to support users currently relying on it. This middleware exists to enable the correct behavior until at least the default one can be changed in a future version. properties: autoDetect: type: boolean type: object digestAuth: description: DigestAuth holds the Digest HTTP authentication configuration. properties: headerField: type: string realm: type: string removeHeader: type: boolean secret: type: string type: object errors: description: ErrorPage holds the custom error page configuration. properties: query: type: string service: description: Service defines an upstream to proxy traffic. properties: kind: enum: - Service - TraefikService type: string name: description: Name is a reference to a Kubernetes Service object (for a load-balancer of servers), or to a TraefikService object (service load-balancer, mirroring, etc). The differentiation between the two is specified in the Kind field. type: string namespace: type: string passHostHeader: type: boolean port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true responseForwarding: description: ResponseForwarding holds configuration for the forward of the response. properties: flushInterval: type: string type: object scheme: type: string serversTransport: type: string sticky: description: Sticky holds the sticky configuration. properties: cookie: description: Cookie holds the sticky configuration based on cookie. properties: httpOnly: type: boolean name: type: string sameSite: type: string secure: type: boolean type: object type: object strategy: type: string weight: description: Weight should only be specified when Name references a TraefikService object (and to be precise, one that embeds a Weighted Round Robin). type: integer required: - name type: object status: items: type: string type: array type: object forwardAuth: description: ForwardAuth holds the http forward authentication configuration. properties: address: type: string authRequestHeaders: items: type: string type: array authResponseHeaders: items: type: string type: array authResponseHeadersRegex: type: string tls: description: ClientTLS holds TLS specific configurations as client. properties: caOptional: type: boolean caSecret: type: string certSecret: type: string insecureSkipVerify: type: boolean type: object trustForwardHeader: type: boolean type: object headers: description: Headers holds the custom header configuration. properties: accessControlAllowCredentials: description: AccessControlAllowCredentials is only valid if true. false is ignored. type: boolean accessControlAllowHeaders: description: AccessControlAllowHeaders must be used in response to a preflight request with Access-Control-Request-Headers set. items: type: string type: array accessControlAllowMethods: description: AccessControlAllowMethods must be used in response to a preflight request with Access-Control-Request-Method set. items: type: string type: array accessControlAllowOriginList: description: AccessControlAllowOriginList is a list of allowable origins. Can also be a wildcard origin "*". items: type: string type: array accessControlAllowOriginListRegex: description: AccessControlAllowOriginListRegex is a list of allowable origins written following the Regular Expression syntax (https://golang.org/pkg/regexp/). items: type: string type: array accessControlExposeHeaders: description: AccessControlExposeHeaders sets valid headers for the response. items: type: string type: array accessControlMaxAge: description: AccessControlMaxAge sets the time that a preflight request may be cached. format: int64 type: integer addVaryHeader: description: AddVaryHeader controls if the Vary header is automatically added/updated when the AccessControlAllowOriginList is set. type: boolean allowedHosts: items: type: string type: array browserXssFilter: type: boolean contentSecurityPolicy: type: string contentTypeNosniff: type: boolean customBrowserXSSValue: type: string customFrameOptionsValue: type: string customRequestHeaders: additionalProperties: type: string type: object customResponseHeaders: additionalProperties: type: string type: object featurePolicy: description: 'Deprecated: use PermissionsPolicy instead.' type: string forceSTSHeader: type: boolean frameDeny: type: boolean hostsProxyHeaders: items: type: string type: array isDevelopment: type: boolean permissionsPolicy: type: string publicKey: type: string referrerPolicy: type: string sslForceHost: description: 'Deprecated: use RedirectRegex instead.' type: boolean sslHost: description: 'Deprecated: use RedirectRegex instead.' type: string sslProxyHeaders: additionalProperties: type: string type: object sslRedirect: description: 'Deprecated: use EntryPoint redirection or RedirectScheme instead.' type: boolean sslTemporaryRedirect: description: 'Deprecated: use EntryPoint redirection or RedirectScheme instead.' type: boolean stsIncludeSubdomains: type: boolean stsPreload: type: boolean stsSeconds: format: int64 type: integer type: object inFlightReq: description: InFlightReq limits the number of requests being processed and served concurrently. properties: amount: format: int64 type: integer sourceCriterion: description: SourceCriterion defines what criterion is used to group requests as originating from a common source. If none are set, the default is to use the request's remote address field. All fields are mutually exclusive. properties: ipStrategy: description: IPStrategy holds the ip strategy configuration. properties: depth: type: integer excludedIPs: items: type: string type: array type: object requestHeaderName: type: string requestHost: type: boolean type: object type: object ipWhiteList: description: IPWhiteList holds the ip white list configuration. properties: ipStrategy: description: IPStrategy holds the ip strategy configuration. properties: depth: type: integer excludedIPs: items: type: string type: array type: object sourceRange: items: type: string type: array type: object passTLSClientCert: description: PassTLSClientCert holds the TLS client cert headers configuration. properties: info: description: TLSClientCertificateInfo holds the client TLS certificate info configuration. properties: issuer: description: TLSClientCertificateDNInfo holds the client TLS certificate distinguished name info configuration. cf https://tools.ietf.org/html/rfc3739 properties: commonName: type: boolean country: type: boolean domainComponent: type: boolean locality: type: boolean organization: type: boolean province: type: boolean serialNumber: type: boolean type: object notAfter: type: boolean notBefore: type: boolean sans: type: boolean serialNumber: type: boolean subject: description: TLSClientCertificateDNInfo holds the client TLS certificate distinguished name info configuration. cf https://tools.ietf.org/html/rfc3739 properties: commonName: type: boolean country: type: boolean domainComponent: type: boolean locality: type: boolean organization: type: boolean province: type: boolean serialNumber: type: boolean type: object type: object pem: type: boolean type: object plugin: additionalProperties: x-kubernetes-preserve-unknown-fields: true type: object rateLimit: description: RateLimit holds the rate limiting configuration for a given router. properties: average: format: int64 type: integer burst: format: int64 type: integer period: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true sourceCriterion: description: SourceCriterion defines what criterion is used to group requests as originating from a common source. If none are set, the default is to use the request's remote address field. All fields are mutually exclusive. properties: ipStrategy: description: IPStrategy holds the ip strategy configuration. properties: depth: type: integer excludedIPs: items: type: string type: array type: object requestHeaderName: type: string requestHost: type: boolean type: object type: object redirectRegex: description: RedirectRegex holds the redirection configuration. properties: permanent: type: boolean regex: type: string replacement: type: string type: object redirectScheme: description: RedirectScheme holds the scheme redirection configuration. properties: permanent: type: boolean port: type: string scheme: type: string type: object replacePath: description: ReplacePath holds the ReplacePath configuration. properties: path: type: string type: object replacePathRegex: description: ReplacePathRegex holds the ReplacePathRegex configuration. properties: regex: type: string replacement: type: string type: object retry: description: Retry holds the retry configuration. properties: attempts: type: integer initialInterval: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true type: object stripPrefix: description: StripPrefix holds the StripPrefix configuration. properties: forceSlash: type: boolean prefixes: items: type: string type: array type: object stripPrefixRegex: description: StripPrefixRegex holds the StripPrefixRegex configuration. properties: regex: items: type: string type: array type: object type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: middlewaretcps.traefik.containo.us spec: group: traefik.containo.us names: kind: MiddlewareTCP listKind: MiddlewareTCPList plural: middlewaretcps singular: middlewaretcp scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: MiddlewareTCP is a specification for a MiddlewareTCP resource. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: MiddlewareTCPSpec holds the MiddlewareTCP configuration. properties: ipWhiteList: description: TCPIPWhiteList holds the TCP ip white list configuration. properties: sourceRange: items: type: string type: array type: object type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: serverstransports.traefik.containo.us spec: group: traefik.containo.us names: kind: ServersTransport listKind: ServersTransportList plural: serverstransports singular: serverstransport scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: ServersTransport is a specification for a ServersTransport resource. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: ServersTransportSpec options to configure communication between Traefik and the servers. properties: certificatesSecrets: description: Certificates for mTLS. items: type: string type: array disableHTTP2: description: Disable HTTP/2 for connections with backend servers. type: boolean forwardingTimeouts: description: Timeouts for requests forwarded to the backend servers. properties: dialTimeout: anyOf: - type: integer - type: string description: The amount of time to wait until a connection to a backend server can be established. If zero, no timeout exists. x-kubernetes-int-or-string: true idleConnTimeout: anyOf: - type: integer - type: string description: The maximum period for which an idle HTTP keep-alive connection will remain open before closing itself. x-kubernetes-int-or-string: true responseHeaderTimeout: anyOf: - type: integer - type: string description: The amount of time to wait for a server's response headers after fully writing the request (including its body, if any). If zero, no timeout exists. x-kubernetes-int-or-string: true type: object insecureSkipVerify: description: Disable SSL certificate verification. type: boolean maxIdleConnsPerHost: description: If non-zero, controls the maximum idle (keep-alive) to keep per-host. If zero, DefaultMaxIdleConnsPerHost is used. type: integer peerCertURI: description: URI used to match against SAN URI during the peer certificate verification. type: string rootCAsSecrets: description: Add cert file for self-signed certificate. items: type: string type: array serverName: description: ServerName used to contact the server. type: string type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: tlsoptions.traefik.containo.us spec: group: traefik.containo.us names: kind: TLSOption listKind: TLSOptionList plural: tlsoptions singular: tlsoption scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: TLSOption is a specification for a TLSOption resource. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: TLSOptionSpec configures TLS for an entry point. properties: alpnProtocols: items: type: string type: array cipherSuites: items: type: string type: array clientAuth: description: ClientAuth defines the parameters of the client authentication part of the TLS connection, if any. properties: clientAuthType: description: ClientAuthType defines the client authentication type to apply. enum: - NoClientCert - RequestClientCert - RequireAnyClientCert - VerifyClientCertIfGiven - RequireAndVerifyClientCert type: string secretNames: description: SecretName is the name of the referenced Kubernetes Secret to specify the certificate details. items: type: string type: array type: object curvePreferences: items: type: string type: array maxVersion: type: string minVersion: type: string preferServerCipherSuites: type: boolean sniStrict: type: boolean type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: tlsstores.traefik.containo.us spec: group: traefik.containo.us names: kind: TLSStore listKind: TLSStoreList plural: tlsstores singular: tlsstore scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: TLSStore is a specification for a TLSStore resource. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: TLSStoreSpec configures a TLSStore resource. properties: defaultCertificate: description: DefaultCertificate holds a secret name for the TLSOption resource. properties: secretName: description: SecretName is the name of the referenced Kubernetes Secret to specify the certificate details. type: string required: - secretName type: object required: - defaultCertificate type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.6.2 creationTimestamp: null name: traefikservices.traefik.containo.us spec: group: traefik.containo.us names: kind: TraefikService listKind: TraefikServiceList plural: traefikservices singular: traefikservice scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: TraefikService is the specification for a service (that an IngressRoute refers to) that is usually not a terminal service (i.e. not a pod of servers), as opposed to a Kubernetes Service. That is to say, it usually refers to other (children) services, which themselves can be TraefikServices or Services. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: ServiceSpec defines whether a TraefikService is a load-balancer of services or a mirroring service. properties: mirroring: description: Mirroring defines a mirroring service, which is composed of a main load-balancer, and a list of mirrors. properties: kind: enum: - Service - TraefikService type: string maxBodySize: format: int64 type: integer mirrors: items: description: MirrorService defines one of the mirrors of a Mirroring service. properties: kind: enum: - Service - TraefikService type: string name: description: Name is a reference to a Kubernetes Service object (for a load-balancer of servers), or to a TraefikService object (service load-balancer, mirroring, etc). The differentiation between the two is specified in the Kind field. type: string namespace: type: string passHostHeader: type: boolean percent: type: integer port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true responseForwarding: description: ResponseForwarding holds configuration for the forward of the response. properties: flushInterval: type: string type: object scheme: type: string serversTransport: type: string sticky: description: Sticky holds the sticky configuration. properties: cookie: description: Cookie holds the sticky configuration based on cookie. properties: httpOnly: type: boolean name: type: string sameSite: type: string secure: type: boolean type: object type: object strategy: type: string weight: description: Weight should only be specified when Name references a TraefikService object (and to be precise, one that embeds a Weighted Round Robin). type: integer required: - name type: object type: array name: description: Name is a reference to a Kubernetes Service object (for a load-balancer of servers), or to a TraefikService object (service load-balancer, mirroring, etc). The differentiation between the two is specified in the Kind field. type: string namespace: type: string passHostHeader: type: boolean port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true responseForwarding: description: ResponseForwarding holds configuration for the forward of the response. properties: flushInterval: type: string type: object scheme: type: string serversTransport: type: string sticky: description: Sticky holds the sticky configuration. properties: cookie: description: Cookie holds the sticky configuration based on cookie. properties: httpOnly: type: boolean name: type: string sameSite: type: string secure: type: boolean type: object type: object strategy: type: string weight: description: Weight should only be specified when Name references a TraefikService object (and to be precise, one that embeds a Weighted Round Robin). type: integer required: - name type: object weighted: description: WeightedRoundRobin defines a load-balancer of services. properties: services: items: description: Service defines an upstream to proxy traffic. properties: kind: enum: - Service - TraefikService type: string name: description: Name is a reference to a Kubernetes Service object (for a load-balancer of servers), or to a TraefikService object (service load-balancer, mirroring, etc). The differentiation between the two is specified in the Kind field. type: string namespace: type: string passHostHeader: type: boolean port: anyOf: - type: integer - type: string x-kubernetes-int-or-string: true responseForwarding: description: ResponseForwarding holds configuration for the forward of the response. properties: flushInterval: type: string type: object scheme: type: string serversTransport: type: string sticky: description: Sticky holds the sticky configuration. properties: cookie: description: Cookie holds the sticky configuration based on cookie. properties: httpOnly: type: boolean name: type: string sameSite: type: string secure: type: boolean type: object type: object strategy: type: string weight: description: Weight should only be specified when Name references a TraefikService object (and to be precise, one that embeds a Weighted Round Robin). type: integer required: - name type: object type: array sticky: description: Sticky holds the sticky configuration. properties: cookie: description: Cookie holds the sticky configuration based on cookie. properties: httpOnly: type: boolean name: type: string sameSite: type: string secure: type: boolean type: object type: object type: object type: object required: - metadata - spec type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] ~~~ ### 2.1.2 创建RBAC权限 > 基于角色的访问控制（RBAC）策略，方便对Kubernetes资源和API进行细粒度控制 > > traefik需要一定的权限，需要提前创建ServiceAccount并分配一定的权限。 ~~~powershell # vim traefik-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: namespace: kube-system name: traefik-ingress-controller --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: traefik-ingress-controller rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions - networking.k8s.io resources: - ingresses - ingressclasses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update - apiGroups: - traefik.containo.us resources: - middlewares - middlewaretcps - ingressroutes - traefikservices - ingressroutetcps - ingressrouteudps - tlsoptions - tlsstores - serverstransports verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: traefik-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: traefik-ingress-controller subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: kube-system ~~~ ### 2.1.3 创建traefik配置文件 > 由traefik配置很多，通过CLI定义不方便，一般都通过配置文件对traefik进行参数配置，例如使用ConfigMap将配置挂载至traefik中 ~~~powershell # vim traefik-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: traefik namespace: kube-system data: traefik.yaml: |- serversTransport: insecureSkipVerify: true ## 略验证代理服务的 TLS 证书 api: insecure: true ## 允许 HTTP 方式访问 API dashboard: true ## 启用 Dashboard debug: true ## 启用 Debug 调试模式 metrics: prometheus: "" ## 配置 Prometheus 监控指标数据，并使用默认配置 entryPoints: web: address: ":80" ## 配置 80 端口，并设置入口名称为 web websecure: address: ":443" ## 配置 443 端口，并设置入口名称为 websecure metrics: address: ":8082" ## 配置 8082端口，并设置入口名称为 metrics tcpep: address: ":8083" ## 配置 8083端口，并设置入口名称为 tcpep,做为tcp入口 udpep: address: ":8084/udp" ## 配置 8084端口，并设置入口名称为 udpep，做为udp入口 providers: kubernetesCRD: "" ## 启用 Kubernetes CRD 方式来配置路由规则 kubernetesingress: "" ## 启用 Kubernetes Ingress 方式来配置路由规则 kubernetesGateway: "" ## 启用 Kubernetes Gateway API experimental: kubernetesGateway: true ## 允许使用 Kubernetes Gateway API log: filePath: "" ## 设置调试日志文件存储路径，如果为空则输出到控制台 level: error ## 设置调试日志级别 format: json ## 设置调试日志格式 accessLog: filePath: "" ## 设置访问日志文件存储路径，如果为空则输出到控制台 format: json ## 设置访问调试日志格式 bufferingSize: 0 ## 设置访问日志缓存行数 filters: retryAttempts: true ## 设置代理访问重试失败时，保留访问日志 minDuration: 20 ## 设置保留请求时间超过指定持续时间的访问日志 fields: ## 设置访问日志中的字段是否保留（keep 保留、drop 不保留） defaultMode: keep ## 设置默认保留访问日志字段 names: ClientUsername: drop headers: defaultMode: keep ## 设置 Header 中字段是否保留,设置默认保留 Header 中字段 names: ## 针对 Header 中特别字段特别配置保留模式 User-Agent: redact Authorization: drop Content-Type: keep ~~~ ### 2.1.4 应用上述资源清单文件 ~~~powershell # kubectl apply -f traefik-crd.yaml ~~~ ~~~powershell # kubectl apply -f traefik-rbac.yaml ~~~ ~~~powershell # kubectl apply -f traefik-config.yaml ~~~ ### 2.1.5 设置节点Label > 由于使用DaemonSet方式部署Traefik，所以需要为节点设置label，当应用部署时会根据节点Label进行选择。 ~~~powershell 设置节点标签 # kubectl label nodes --all IngressProxy=true ~~~ ~~~powershell 查看节点标签 # kubectl get nodes --show-labels ~~~ ~~~powershell 如需要取消时，可执行下述命令 # kubectl label nodes --all IngressProxy- ~~~ ## 2.2 部署Traefik ### 2.2.1 deploy资源清单文件准备 > 本次将用Daemonset方式部署traefik,便于后期扩展 > > 本次部署通过hostport方式把Pod中容器内的80、443映射到物理机，方便集群外访问。 ~~~powershell # vim traefik-deploy.yaml apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-system name: traefik labels: app: traefik spec: selector: matchLabels: app: traefik template: metadata: labels: app: traefik spec: serviceAccountName: traefik-ingress-controller containers: - name: traefik image: traefik:v2.5.7 args: - --configfile=/config/traefik.yaml volumeMounts: - mountPath: /config name: config ports: - name: web containerPort: 80 hostPort: 80 ## 将容器端口绑定所在服务器的 80 端口 - name: websecure containerPort: 443 hostPort: 443 ## 将容器端口绑定所在服务器的 443 端口 - name: admin containerPort: 8080 ## Traefik Dashboard 端口 - name: tcpep containerPort: 8083 hostPort: 8083 ## 将容器端口绑定所在服务器的 8083 端口 - name: udpep containerPort: 8084 hostPort: 8084 ## 将容器端口绑定所在服务器的 8084 端口 protocol: UDP volumes: - name: config configMap: name: traefik tolerations: ## 设置容忍所有污点，防止节点被设置污点 - operator: "Exists" nodeSelector: ## 设置node筛选器，在特定label的节点上启动 IngressProxy: "true" ~~~ ~~~powershell 验证端口配置是否正确 # kubectl get daemonset traefik -n kube-system -o yaml ~~~ ### 2.2.2 应用deploy资源清单文件 ~~~powershell # kubectl apply -f traefix-deploy.yaml ~~~ ### 2.2.3 service资源清单文件准备 ~~~powershell # vim traefix-service.yaml apiVersion: v1 kind: Service metadata: name: traefik namespace: kube-system spec: ports: - protocol: TCP name: web port: 80 - protocol: TCP name: admin port: 8080 - protocol: TCP name: websecure port: 443 - protocol: TCP name: tcpep port: 8083 - protocol: UDP name: udpep port: 8084 selector: app: traefik ~~~ ### 2.2.4 应用service资源清单文件 ~~~powershell # kubectl apply -f traefik-service.yaml ~~~ ## 2.3 配置访问traefik dashboard路由规则 >Traefik 应用已经部署完成，但是想让外部访问 Kubernetes 内部服务，还需要配置路由规则，上面部署 Traefik 时开启了 Traefik Dashboard，这是 Traefik 提供的视图看板，所以，首先配置基于 HTTP 的 Traefik Dashboard 路由规则，使外部能够访问 Traefik Dashboard。这里使用 IngressRoute方式进行演示。 ### 2.3.1 Traefik创建路由规则方法 - 原生ingress - CRD IngressRoute - Gateway API ![image-20220418185227777](../../img/kubernetes/kubernetes_traefik/image-20220418185227777.png) ### 2.3.2 通过原生ingress方式暴露traefik dashboard ~~~powershell # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE traefik ClusterIP 10.96.174.88 80/TCP,8080/TCP,443/TCP,8083/TCP,8084/UDP 6h44m ~~~ ~~~powershell # kubectl get endpoints -n kube-system NAME ENDPOINTS AGE traefik 10.244.135.204:80,10.244.159.141:80,10.244.194.92:80 + 17 more... 6h44m ~~~ ~~~powershell # cat traefik-dashboard-native-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: traefik-dashboard-ingress namespace: kube-system annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/router.entrypoints: web spec: rules: - host: tfni.kubemsb.com http: paths: - pathType: Prefix path: / backend: service: name: traefik port: number: 8080 ~~~ ~~~powershell # kubectl apply -f traefik-dashboard-native-ingress.yaml ~~~ ~~~powershell # kubectl get ingress -n kube-system NAME CLASS HOSTS ADDRESS PORTS AGE traefik-dashboard-ingress tfni.kubemsb.com 80 56m ~~~ ~~~powershell # kubectl describe ingress traefik-dashboard-ingress -n kube-system Name: traefik-dashboard-ingress Namespace: kube-system Address: Default backend: default-http-backend:80 () Rules: Host Path Backends ---- ---- -------- tfni.kubemsb.com / traefik:8080 (10.244.135.204:8080,10.244.159.141:8080,10.244.194.92:8080 + 1 more...) Annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/router.entrypoints: web Events: ~~~ **在集群之外主机访问** ~~~powershell # vim /etc/hosts ...... 192.168.10.12 tfni.kubemsb.com ~~~ ![image-20220418185030751](../../img/kubernetes/kubernetes_traefik/image-20220418185030751.png) ### 2.3.3 创建dashboard ingress route资源清单文件 ~~~powershell # vim traefik-dashboard-ingress-route.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: traefik namespace: kube-system spec: entryPoints: - web routes: - match: Host(`traefik.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: traefik port: 8080 ~~~ ### 2.3.4 应用资源清单文件 ~~~powershell # kubectl apply -f traefik-dashboard-ingress-route.yaml ~~~ ### 2.3.5 在集群内或集群外主机配置域名解析 ~~~powershell 把域名解析为kubernetes集群任意节点IP既可。 # vim /etc/hosts 192.168.10.12 traefik.kubemsb.com ~~~ ### 2.3.6 通过域名访问traefik dashboard ~~~powershell # firefox http://traefik.kubemsb.com & ~~~ ![image-20220416095227915](../../img/kubernetes/kubernetes_traefik/image-20220416095227915.png) # 三、traefik基础应用 ## 3.1 配置kubernetes dashboard路由规则 > 必须使用SSL证书创建secret密钥 ### 3.1.1 查看kubernetes dashboard service状态 ~~~powershell # kubectl get svc -n kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.96.100.4 8000/TCP 21d kubernetes-dashboard NodePort 10.96.19.1 443:30000/TCP 21d ~~~ ### 3.1.2 编写访问kubernetes dashboard 路由规则 ~~~powershell # ls 6864844_kubemsb.com_nginx.zip kubemsb.key kubemsb.pem ~~~ ~~~powershell # kubectl create secret tls kubemsb-tls --cert=kubemsb.pem --key=kubemsb.key secret/kubemsb-tls created ~~~ ~~~powershell # kubectl get secret NAME TYPE DATA AGE default-token-x4qbc kubernetes.io/service-account-token 3 24d kubemsb-tls kubernetes.io/tls 2 11s ~~~ ~~~powershell # vim kubernetes-dashboard-ir.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: kubernetes-dashboard namespace: kubernetes-dashboard spec: entryPoints: - websecure routes: - match: Host(`dashboard.kubemsb.com`) kind: Rule services: - name: kubernetes-dashboard port: 443 tls: secretName: kubemsb-tls ~~~ ### 3.1.3 应用上述路由规则文件 ~~~powershell # kubectl apply -f kubernetes-dashboard-ir.yaml ~~~ ![image-20220416090128886](../../img/kubernetes/kubernetes_traefik/image-20220416090128886.png) ![image-20220416090950402](../../img/kubernetes/kubernetes_traefik/image-20220416090950402.png) ### 3.1.4 配置域名解析及访问 ~~~powershell # vim /etc/hosts 192.168.10.12 dashboard.kubemsb.com ~~~ ![image-20220416091040903](../../img/kubernetes/kubernetes_traefik/image-20220416091040903.png) ![image-20220416192909766](../../img/kubernetes/kubernetes_traefik/image-20220416192909766.png) ~~~powershell # kubectl get secret -n kubernetes-dashboard NAME TYPE DATA AGE ...... kubernetes-dashboard-token-8tm5n kubernetes.io/service-account-token 3 21d ~~~ ~~~powershell # kubectl describe secret kubernetes-dashboard-token-8tm5n -n kubernetes-dashboard Name: kubernetes-dashboard-token-8tm5n Namespace: kubernetes-dashboard Labels: Annotations: kubernetes.io/service-account.name: kubernetes-dashboard kubernetes.io/service-account.uid: 47d19e14-e3ed-4733-9799-02396dfb436a Type: kubernetes.io/service-account-token Data ==== ca.crt: 1359 bytes namespace: 20 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImVGc2xjT05uekl0MlVOZ0VCSlhHSURfOXd6WGFvVnZFZmNwREwtVk1STlEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi04dG01biIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ3ZDE5ZTE0LWUzZWQtNDczMy05Nzk5LTAyMzk2ZGZiNDM2YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.rY5YFoJD3ZUg1e31uxrPhKl0a_mslAmVG5r1ZDia0z7WKKtGjs8XwSu5cIDqKa-1FUV6ixxgfDwm5Rd-nK8TCMVRrDY_7r2I5-u2ebh4rEAXCBvQw0gtJu16-e1Z_TQqNuc73E9fDS0AqHr2qGOWAQcz_FjWGAGjZKzYlKPcA3mFI2wsAIRgxh29-S2f4SxB2zgWyYQdbW-fiHDHWK6dQ-a3glIlCk4jnPtzrX1HNK3BSRPoKaZg_9Ot0dABex2ro5QkQ0wyuT3NLio-n8v21MbhKG5ehBEFRNrTcTPM2CLt4XaUJezphKHShdc3VbUlizPge5DleAJcp9JrFzyqBQ ~~~ ![image-20220416192957609](../../img/kubernetes/kubernetes_traefik/image-20220416192957609.png) ![image-20220416193057872](../../img/kubernetes/kubernetes_traefik/image-20220416193057872.png) ## 3.2 配置HTTP路由规则 ### 3.2.1 创建应用及服务资源清单文件并应用 ~~~powershell # vim whoami.yaml kind: Deployment apiVersion: apps/v1 metadata: name: whoami namespace: default labels: app: traefiklabs name: whoami spec: replicas: 2 selector: matchLabels: app: traefiklabs task: whoami template: metadata: labels: app: traefiklabs task: whoami spec: containers: - name: whoami image: traefik/whoami ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: whoami namespace: default spec: ports: - name: http port: 80 selector: app: traefiklabs task: whoami --- kind: Deployment apiVersion: apps/v1 metadata: name: whoamitcp namespace: default labels: app: traefiklabs name: whoamitcp spec: replicas: 2 selector: matchLabels: app: traefiklabs task: whoamitcp template: metadata: labels: app: traefiklabs task: whoamitcp spec: containers: - name: whoamitcp image: traefik/whoamitcp ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: whoamitcp namespace: default spec: ports: - protocol: TCP port: 8080 selector: app: traefiklabs task: whoamitcp --- kind: Deployment apiVersion: apps/v1 metadata: name: whoamiudp namespace: default labels: app: traefiklabs name: whoamiudp spec: replicas: 2 selector: matchLabels: app: traefiklabs task: whoamiudp template: metadata: labels: app: traefiklabs task: whoamiudp spec: containers: - name: whoamiudp image: traefik/whoamiudp:latest ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: whoamiudp namespace: default spec: ports: - port: 8080 protocol: UDP selector: app: traefiklabs task: whoamiudp ~~~ ~~~powershell # kubectl apply -f whoami.yaml ~~~ ~~~powershell # kubectl get all NAME READY STATUS RESTARTS AGE pod/whoami-7d666f84d8-ffbdv 1/1 Running 0 35m pod/whoami-7d666f84d8-n75wb 1/1 Running 0 35m pod/whoamitcp-744cc4b47-g98fv 1/1 Running 0 35m pod/whoamitcp-744cc4b47-s2m7h 1/1 Running 0 35m pod/whoamiudp-58f6cf7b8-54552 1/1 Running 0 35m pod/whoamiudp-58f6cf7b8-dzxpg 1/1 Running 0 35m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 25d service/nginx-metallb LoadBalancer 10.96.109.6 192.168.10.90 8090:30259/TCP 14d service/whoami ClusterIP 10.96.251.213 80/TCP 35m service/whoamitcp ClusterIP 10.96.20.1 8080/TCP 35m service/whoamiudp ClusterIP 10.96.85.175 8080/UDP 35m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/whoami 2/2 2 2 35m deployment.apps/whoamitcp 2/2 2 2 35m deployment.apps/whoamiudp 2/2 2 2 35m NAME DESIRED CURRENT READY AGE replicaset.apps/whoami-7d666f84d8 2 2 2 35m replicaset.apps/whoamitcp-744cc4b47 2 2 2 35m replicaset.apps/whoamiudp-58f6cf7b8 2 2 2 35m ~~~ ### 3.2.2 创建whoami应用ingress route资源清单文件并应用 ~~~powershell # vim whoami-ingressroute.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: myingressroute namespace: default spec: entryPoints: - web routes: - match: Host(`whoami.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: whoami port: 80 ~~~ ~~~powershell # kubectl apply -f whoami-ingressroute.yaml ~~~ ~~~powershell # kubectl get ingressroute NAME AGE myingressroute 29m ~~~ ![image-20220416182934718](../../img/kubernetes/kubernetes_traefik/image-20220416182934718.png) ![image-20220416182828875](../../img/kubernetes/kubernetes_traefik/image-20220416182828875.png) ## 3.3 配置HTTPS路由规则 如果我们需要用 HTTPS 来访问我们这个应用的话，就需要监听 websecure 这个入口点，也就是通过 443 端口来访问，同样用 HTTPS 访问应用必然就需要证书 ### 3.3.1 自签名证书 ~~~powershell # openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=whoamissl.kubemsb.com" ~~~ ### 3.3.2 创建secret ~~~powershell # kubectl create secret tls who-tls --cert=tls.crt --key=tls.key ~~~ ### 3.3.3 创建https应用路由规则 ~~~powershell # vim whoamissl-ingressroute.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: ingressroutetls spec: entryPoints: - websecure routes: - match: Host(`whoamissl.kubemsb.com`) kind: Rule services: - name: whoami port: 80 tls: secretName: who-tls ~~~ ~~~powershell # kubectl apply -f whoamissl-ingressroute.yaml ~~~ ~~~powershell # kubectl get ingressroute NAME AGE ingressroutetls 17s ~~~ ![image-20220416184943571](../../img/kubernetes/kubernetes_traefik/image-20220416184943571.png) ~~~powershell # cat /etc/hosts 192.168.10.12 whoamissl.kubemsb.com ~~~ ![image-20220416185151502](../../img/kubernetes/kubernetes_traefik/image-20220416185151502.png) ![image-20220416185217631](../../img/kubernetes/kubernetes_traefik/image-20220416185217631.png) ## 3.4 配置TCP路由规则 > SNI为服务名称标识，是 TLS 协议的扩展。因此，只有 TLS 路由才能使用该规则指定域名。但是，非 TLS 路由必须使用带有 `*` 的规则（每个域）来声明每个非 TLS 请求都将由路由进行处理。 ### 3.4.1 实验案例配置 ~~~powershell # vim whoami-ingressroutetcp.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRouteTCP metadata: name: ingressroutetcpwho spec: entryPoints: - tcpep routes: - match: HostSNI(`*`) services: - name: whoamitcp port: 8080 ~~~ ~~~powershell # kubectl get ingressroutetcp NAME AGE ingressroutetcpwho 17s ~~~ ![image-20220416194607990](../../img/kubernetes/kubernetes_traefik/image-20220416194607990.png) ![image-20220416194637687](../../img/kubernetes/kubernetes_traefik/image-20220416194637687.png) ![image-20220416194801740](../../img/kubernetes/kubernetes_traefik/image-20220416194801740.png) ### 3.4.2 生产案例配置 MySQL部署及traefik代理 #### 3.4.2.1 修改traefik-configmap.yaml ~~~powershell # cat traefik-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: traefik namespace: kube-system data: traefik.yaml: |- serversTransport: insecureSkipVerify: true api: insecure: true dashboard: true debug: true metrics: prometheus: "" entryPoints: web: address: ":80" websecure: address: ":443" metrics: address: ":8082" tcpep: address: ":8083" udpep: address: ":8084/udp" mysql: address: ":3312" providers: kubernetesCRD: "" kubernetesingress: "" log: filePath: "" level: error format: json accessLog: filePath: "" format: json bufferingSize: 0 filters: retryAttempts: true minDuration: 20 fields: defaultMode: keep names: ClientUsername: drop headers: defaultMode: keep names: User-Agent: redact Authorization: drop Content-Type: keep ~~~ #### 3.4.2.2 修改traefik-deploy.yaml ~~~powershell # vim traefik-deploy.yaml apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-system name: traefik labels: app: traefik spec: selector: matchLabels: app: traefik template: metadata: labels: app: traefik spec: serviceAccountName: traefik-ingress-controller containers: - name: traefik image: traefik:v2.5.7 args: - --configfile=/config/traefik.yaml volumeMounts: - mountPath: /config name: config ports: - name: web containerPort: 80 hostPort: 80 - name: websecure containerPort: 443 hostPort: 443 - name: admin containerPort: 8080 - name: tcpep containerPort: 8083 hostPort: 8083 - name: udpep containerPort: 8084 hostPort: 8084 protocol: UDP - name: mysql containerPort: 3312 hostPort: 3306 volumes: - name: config configMap: name: traefik ~~~ ~~~powershell # vim traefik-service.yaml apiVersion: v1 kind: Service metadata: name: traefik namespace: kube-system spec: ports: - protocol: TCP name: web port: 80 - protocol: TCP name: admin port: 8080 - protocol: TCP name: websecure port: 443 - protocol: TCP name: tcpep port: 8083 - protocol: UDP name: udpep port: 8084 - protocol: TCP name: mysql port: 3312 selector: app: traefik ~~~ ~~~powershell 删除以前的配置及应用 # kubectl delete -f traefix-configmap.yaml # kubectl delete -f traefix-deploy.yaml # kubectl delete -f traefix-service.yaml ~~~ ~~~powershell 重新部署 # kubectl apply -f traefix-configmap.yaml # kubectl apply -f traefix-deploy.yaml # kubectl apply -f traefix-service.yaml ~~~ ![image-20220416214209085](../../img/kubernetes/kubernetes_traefik/image-20220416214209085.png) #### 3.4.2.3 部署mysql应用 > 关于端口的说明： >traefik Pod：3312:3306（traefik Pod:k8s Node），3312是traefik配置的mysql入口点的端口，3306是k8s Node的端口，traefik请求入口 ~~~powershell # vim mysql-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: mysql labels: app: mysql namespace: default data: my.cnf: | [mysqld] character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci skip-character-set-client-handshake = 1 default-storage-engine = INNODB max_allowed_packet = 500M explicit_defaults_for_timestamp = 1 long_query_time = 10 ~~~ ~~~powershell # vim mysql-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: mysql name: mysql namespace: default spec: selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:5.7 imagePullPolicy: IfNotPresent env: - name: MYSQL_ROOT_PASSWORD value: abc123 ports: - containerPort: 3306 volumeMounts: - mountPath: /var/lib/mysql name: pv - mountPath: /etc/mysql/conf.d/my.cnf subPath: my.cnf name: cm volumes: - name: pv hostPath: path: /opt/mysqldata - name: cm configMap: name: mysql ~~~ ~~~powershell # vim mysql-service.yaml apiVersion: v1 kind: Service metadata: name: mysql namespace: default spec: ports: - port: 3306 protocol: TCP targetPort: 3306 selector: app: mysql ~~~ ~~~powershell # kubectl apply -f mysql-configmap.yaml configmap/mysql created ~~~ ~~~powershell # kubectl apply -f mysql-deploy.yaml deployment.apps/mysql created ~~~ ~~~powershell # kubectl apply -f mysql-service.yaml service/mysql created ~~~ #### 3.4.2.4 为mysql应用创建ingressroute ~~~powershell # vim mysql-ingressroutetcp.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRouteTCP metadata: name: mysql namespace: default spec: entryPoints: - mysql routes: - match: HostSNI(`*`) services: - name: mysql port: 3306 ~~~ ~~~powershell # kubectl apply -f mysql-ingressroutetcp.yaml ingressroutetcp.traefik.containo.us/mysql created ~~~ ~~~powershell # kubectl get ingressroutetcp NAME AGE mysql 8s ~~~ #### 3.4.2.5 验证 ![image-20220416223942162](../../img/kubernetes/kubernetes_traefik/image-20220416223942162.png) **在集群之外主机上执行如下操作** ~~~powershell # cat /etc/hosts ...... 192.168.10.12 mysql.kubemsb.com ~~~ ~~~powershell # mysql -h mysql.kubemsb.com -uroot -pabc123 -P3306 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.7.36 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> ~~~ ### 3.4.3 生产案例 Redis部署及traefik代理 ~~~powershell # vim redis.yaml apiVersion: apps/v1 kind: Deployment metadata: name: redis namespace: default spec: selector: matchLabels: app: redis template: metadata: labels: app: redis spec: containers: - name: redis image: redis:6.2.6 ports: - containerPort: 6379 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: redis namespace: default spec: ports: - port: 6379 targetPort: 6379 selector: app: redis ~~~ ~~~powershell # kubectl apply -f redis.yaml deployment.apps/redis created service/redis created ~~~ ~~~powershell # vim traefik-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: traefik namespace: kube-system data: traefik.yaml: |- serversTransport: insecureSkipVerify: true api: insecure: true dashboard: true debug: true metrics: prometheus: "" entryPoints: web: address: ":80" websecure: address: ":443" metrics: address: ":8082" tcpep: address: ":8083" udpep: address: ":8084/udp" mysql: address: ":3312" redis: address: ":6379" providers: kubernetesCRD: "" kubernetesingress: "" log: filePath: "" level: error format: json accessLog: filePath: "" format: json bufferingSize: 0 filters: retryAttempts: true minDuration: 20 fields: defaultMode: keep names: ClientUsername: drop headers: defaultMode: keep names: User-Agent: redact Authorization: drop Content-Type: keep ~~~ ~~~powershell 删除原configmap,再重新创建 # kubectl delete -f traefik-configmap.yaml # kubectl apply -f traefik-configmap.yaml ~~~ ~~~powershell # vim traefix-deploy.yaml apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-system name: traefik labels: app: traefik spec: selector: matchLabels: app: traefik template: metadata: labels: app: traefik spec: serviceAccountName: traefik-ingress-controller containers: - name: traefik image: traefik:v2.5.7 args: - --configfile=/config/traefik.yaml volumeMounts: - mountPath: /config name: config ports: - name: web containerPort: 80 hostPort: 80 - name: websecure containerPort: 443 hostPort: 443 - name: admin containerPort: 8080 - name: tcpep containerPort: 8083 hostPort: 8083 - name: udpep containerPort: 8084 hostPort: 8084 protocol: UDP - name: mysql containerPort: 3312 hostPort: 3306 - name: redis containerPort: 6379 hostPort: 6379 volumes: - name: config configMap: name: traefik ~~~ ~~~powershell 删除原deploy,再重新创建 # kubectl delete -f traefik-deploy.yaml # kubectl apply -f traefik-deploy.yaml ~~~ ~~~powershell 验证是否添加成功 # kubectl get daemonset traefik -n kube-system -o yaml ... - containerPort: 6379 hostPort: 6379 name: redis protocol: TCP ... ~~~ ~~~powershell # vim redis-ingressroutetcp.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRouteTCP metadata: name: redis namespace: default spec: entryPoints: - redis routes: - match: HostSNI(`*`) services: - name: redis port: 6379 ~~~ ~~~powershell # kubectl get ingressroutetcp NAME AGE redis 8s ~~~ ![image-20220417011156699](../../img/kubernetes/kubernetes_traefik/image-20220417011156699.png) ![image-20220417011339455](../../img/kubernetes/kubernetes_traefik/image-20220417011339455.png) **在集群之外主机添加域名解析** ~~~powershell # cat /etc/hosts 192.168.10.15 redis.kubemsb.com ~~~ ~~~powershell # wget http://download.redis.io/releases/redis-3.2.8.tar.gz # tar xf redis-3.2.8.tar.gz # make ~~~ ~~~powershell # ./src/redis-cli -h redis.kubemsb.com -p 6379 redis.kubemsb.com:6379> ping PONG redis.kubemsb.com:6379> set hello world OK redis.kubemsb.com:6379> get hello "world" ~~~ ## 3.5 配置UDP路由规则 ~~~powershell # vim whoami-ingressrouteudp.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRouteUDP metadata: name: ingressrouteudpwho spec: entryPoints: - udpep routes: - services: - name: whoamiudp port: 8080 ~~~ ~~~powershell # kubectl get ingressrouteudp NAME AGE ingressrouteudpwho 11s ~~~ ![image-20220416195402099](../../img/kubernetes/kubernetes_traefik/image-20220416195402099.png) ![image-20220416195433680](../../img/kubernetes/kubernetes_traefik/image-20220416195433680.png) ![image-20220416195459722](../../img/kubernetes/kubernetes_traefik/image-20220416195459722.png) **验证可有性** ~~~powershell # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE whoamiudp ClusterIP 10.96.85.175 8080/TCP 6h35m ~~~ ~~~powershell echo "WHO" | socat - udp4-datagram:10.96.85.175:8080 ~~~ ~~~powershell echo "othermessage" | socat - udp4-datagram:10.96.85.175:8080 ~~~ **输出** ~~~powershell # echo "WHO" | socat - udp4-datagram:192.168.10.12:8084 Hostname: whoamiudp-58f6cf7b8-54552 IP: 127.0.0.1 IP: ::1 IP: 10.244.159.187 IP: fe80::1cb4:e1ff:fe66:d9b ~~~ ~~~powershell # echo "othermessage" | socat - udp4-datagram:192.168.10.12:8084 Received: othermessage ~~~ # 四、traefik中间件 MiddleWare ## 4.1 traefik中间件介绍 MiddleWare 中间件是 Traefik2.0 中一个非常有特色的功能，我们可以根据自己的各种需求去选择不同的中间件来满足服务，Traefik 官方已经内置了许多不同功能的中间件，其中包括修改请求头信息；重定向；身份验证等等，而且中间件还可以通过链式组合的方式来适用各种情况。例如：强制跳转https、去除访问前缀、访问白名单等。 ![](../../img/kubernetes/kubernetes_traefik/traefik-middleware.png) ## 4.2 traefik 中间件应用案例 ipWhiteList > 在工作 中，有一些URL并不希望对外暴露，比如prometheus、grafana等，我们就可以通过白名单IP来过到要求，可以使用Traefix中的ipWhiteList中间件来完成。 ~~~powershell # vim deploy-service.yaml kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web-middleware namespace: default spec: replicas: 1 selector: matchLabels: app: middle template: metadata: labels: app: middle spec: containers: - name: nginx-web-c image: nginx:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: service-middle namespace: default spec: ports: - name: http port: 80 selector: app: middle ~~~ ~~~powershell # kubectl apply -f deploy-service.yaml deployment.apps/nginx-web-middleware created service/service-middle created ~~~ ~~~powershell # vim middleware-ipwhitelist.yaml apiVersion: traefik.containo.us/v1alpha1 kind: Middleware metadata: name: gs-ipwhitelist spec: ipWhiteList: sourceRange: - 127.0.0.1 - 10.244.0.0/16 - 10.96.0.0/12 - 192.168.10.0/24 ~~~ ~~~powershell # kubectl apply -f middleware-ipwhitelist.yaml middleware.traefik.containo.us/gs-ipwhitelist created ~~~ ~~~powershell # vim deploy-service-middle.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: ingressroutemiddle namespace: default spec: entryPoints: - web routes: - match: Host(`middleware.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: service-middle port: 80 namespace: default middlewares: - name: gs-ipwhitelist ~~~ ~~~powershell # kubectl apply -f deploy-service-middle.yaml ingressroute.traefik.containo.us/ingressroutemiddle created ~~~ ![image-20220417144538252](../../img/kubernetes/kubernetes_traefik/image-20220417144538252.png) ![image-20220417144726075](../../img/kubernetes/kubernetes_traefik/image-20220417144726075.png) **在集群之外的主机上访问** ~~~powershell # vim /etc/hosts 192.168.10.15 middleware.kubemsb.com ~~~ ~~~powershell # curl http://middleware.kubemsb.com ~~~ **把集群外主机所在的网段从白名单中删除，发现无法访问。** ~~~powershell # cat middleware-ipwhitelist.yaml apiVersion: traefik.containo.us/v1alpha1 kind: Middleware metadata: name: gs-ipwhitelist spec: ipWhiteList: sourceRange: - 127.0.0.1 - 10.244.0.0/16 - 10.96.0.0/12 ~~~ ~~~powershell # kubectl apply -f middleware-ipwhitelist.yaml middleware.traefik.containo.us/gs-ipwhitelist configured ~~~ ![image-20220417145252982](../../img/kubernetes/kubernetes_traefik/image-20220417145252982.png) **在集群外主机访问** ~~~powershell # curl http://middleware.kubemsb.com Forbidden ~~~ # 五、traefik高级应用 在实际的生产环境，除了上线业务之外，还有更复杂的使用要求。 在开始traefik的高级用法之前，还需要了解一个TraefikService，通过把TraefikService注册到CRD来实现更复杂的请求设置。 ~~~powershell TraefikService 目前能用于以下功能 servers load balancing.（负载均衡） services Weighted Round Robin load balancing.（权重轮询） services mirroring.（镜像） ~~~ ## 5.1 负载均衡 ### 5.1.1 创建deployment控制器类型应用 ~~~powershell # vim loadbalancer-deploy.yaml kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web1 namespace: default spec: replicas: 1 selector: matchLabels: app: v1 template: metadata: labels: app: v1 spec: containers: - name: nginx-web-c image: nginx:latest ports: - containerPort: 80 --- kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web2 namespace: default spec: replicas: 1 selector: matchLabels: app: v2 template: metadata: labels: app: v2 spec: containers: - name: nginx-web-c image: nginx:latest ports: - containerPort: 80 ~~~ ~~~powershell # kubectl apply -f loadbalancer-deploy.yaml ~~~ **修改容器中网站页面信息** ~~~powershell # kubectl get pods NAME READY STATUS RESTARTS AGE nginx-web1-856c759646-58snd 1/1 Running 0 2m34s nginx-web2-5d547f7c5f-vd8n7 1/1 Running 0 2m34s ~~~ ~~~powershell # kubectl exec -it nginx-web1-856c759646-58snd -- sh # echo "svc1" > /usr/share/nginx/html/index.html # exit # kubectl exec -it nginx-web2-5d547f7c5f-vd8n7 -- sh # echo "svc2" > /usr/share/nginx/html/index.html # exit ~~~ ~~~powershell # vim loadbalancer-deploy.yaml kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web1 namespace: default spec: replicas: 1 selector: matchLabels: app: v1 template: metadata: labels: app: v1 spec: containers: - name: nginx-web-c image: nginx:latest lifecycle: postStart: exec: command: ["/bin/sh", "-c", "echo svc1 > /usr/share/nginx/html/index.html"] ports: - containerPort: 80 --- kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web2 namespace: default spec: replicas: 1 selector: matchLabels: app: v2 template: metadata: labels: app: v2 spec: containers: - name: nginx-web-c image: nginx:latest lifecycle: postStart: exec: command: ["/bin/sh", "-c", "echo svc2 > /usr/share/nginx/html/index.html"] ports: - containerPort: 80 ~~~ ### 5.1.2 创建service ~~~powershell # vim loadbalancer-deploy-service.yaml apiVersion: v1 kind: Service metadata: name: svc1 namespace: default spec: ports: - name: http port: 80 selector: app: v1 --- apiVersion: v1 kind: Service metadata: name: svc2 namespace: default spec: ports: - name: http port: 80 selector: app: v2 ~~~ ~~~powershell # kubectl apply -f loadbalancer-deploy-service.yaml ~~~ ~~~powershell # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc1 ClusterIP 10.96.136.245 80/TCP 3s svc2 ClusterIP 10.96.2.4 80/TCP 3s ~~~ ~~~powershell # curl http://10.96.136.245 svc1 # curl http://10.96.2.4 svc2 ~~~ ### 5.1.3 创建ingressroute ~~~powershell # vim loadbalancer-deploy-service-ingressroute.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: ingressrouteweblb namespace: default spec: entryPoints: - web routes: - match: Host(`lb.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: svc1 port: 80 namespace: default - name: svc2 port: 80 namespace: default ~~~ ~~~powershell # kubectl apply -f loadbalancer-deploy-service-ingressroute.yaml ~~~ ~~~powershell # kubectl get ingressroute NAME AGE ingressrouteweblb 9s ~~~ ![image-20220417091705273](../../img/kubernetes/kubernetes_traefik/image-20220417091705273.png) ![image-20220417091742077](../../img/kubernetes/kubernetes_traefik/image-20220417091742077.png) ### 5.1.4 访问验证 > 在集群之外的主机上访问 ~~~powershell # cat /etc/hosts 192.168.10.15 lb.kubemsb.com ~~~ ~~~powershell # curl http://lb.kubemsb.com svc1 # curl http://lb.kubemsb.com svc2 ~~~ ## 5.2 灰度发布 > 基于上述负载均衡案例基础之上实施。 > > 灰度发布也称为金丝雀发布，让一部分即将上线的服务发布到线上，观察是否达到上线要求，主要通过权重轮询的方式实现。 ![image-20220419015820945](../../img/kubernetes/kubernetes_traefik/image-20220419015820945.png) ### 5.2.1 创建TraefikService ~~~powershell # vim traefikservice.yaml apiVersion: traefik.containo.us/v1alpha1 kind: TraefikService metadata: name: wrr namespace: default spec: weighted: services: - name: svc1 port: 80 weight: 3 # 定义权重 kind: Service # 可选，默认就是 Service - name: svc2 port: 80 weight: 1 ~~~ ~~~powershell # kubectl apply -f traefikservice.yaml traefikservice.traefik.containo.us/wrr created ~~~ ### 5.2.2 创建ingressroute >需要注意的是现在我们配置的 Service 不再是直接的 Kubernetes 对象了，而是上面我们定义的 TraefikService 对象 ~~~powershell # vim traefik-wrr.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: ingressroutewrr namespace: default spec: entryPoints: - web routes: - match: Host(`lb.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: wrr namespace: default kind: TraefikService ~~~ ~~~powershell # kubectl apply -f traefik-wrr.yaml ingressroute.traefik.containo.us/ingressroutewrr created ~~~ ![image-20220417093350378](../../img/kubernetes/kubernetes_traefik/image-20220417093350378.png) ![image-20220417093427267](../../img/kubernetes/kubernetes_traefik/image-20220417093427267.png) ~~~powershell 第一次 # curl http://lb.kubemsb.com svc1 第二次 # curl http://lb.kubemsb.com svc1 第三次 # curl http://lb.kubemsb.com svc2 第四次 # curl http://lb.kubemsb.com svc1 ~~~ ## 5.3 流量复制 > 在负责均衡案例基础之上实施 > > 所谓的流量复制，也称为镜像服务是指将请求的流量按规则复制一份发送给其它服务，并且会忽略这部分请求的响应，这个功能在做一些压测或者问题复现的时候很有用。 ### 5.3.1 指定流量来自己于kubernetes service对象 ~~~powershell # vim mirror_from_service.yaml apiVersion: traefik.containo.us/v1alpha1 kind: TraefikService metadata: name: mirror-from-service namespace: default spec: mirroring: name: svc1 # 发送 100% 的请求到 K8S 的 Service "v1" port: 80 mirrors: - name: svc2 # 然后复制 20% 的请求到 v2 port: 80 percent: 20 ~~~ ~~~powershell # kubectl apply -f mirror_from_service.yaml traefikservice.traefik.containo.us/mirror-from-service created ~~~ ~~~powershell # vim mirror-from-service-ingressroute.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: ingressroute-mirror namespace: default spec: entryPoints: - web routes: - match: Host(`lb.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: mirror-from-service namespace: default kind: TraefikService ~~~ ~~~powershell # kubectl apply -f mirror-from-service-ingressroute.yaml ingressroute.traefik.containo.us/ingressroute-mirror created ~~~ ![image-20220417095009228](../../img/kubernetes/kubernetes_traefik/image-20220417095009228.png) ![image-20220417095043019](../../img/kubernetes/kubernetes_traefik/image-20220417095043019.png) ![image-20220417095553462](../../img/kubernetes/kubernetes_traefik/image-20220417095553462.png) ~~~powershell # while true do curl http://lb.kubemsb.com sleep 1 done ~~~ ### 5.3.2 通过traefikservice导入流量 ~~~powershell # vim mirror-from-traefikservice.yaml apiVersion: traefik.containo.us/v1alpha1 kind: TraefikService metadata: name: mirror-from-traefikservice namespace: default spec: mirroring: name: mirror-from-service #流量入口从TraefikService 来 kind: TraefikService mirrors: - name: svc2 port: 80 percent: 20 ~~~ ~~~powershell # kubectl apply -f mirror-from-traefikservice.yaml traefikservice.traefik.containo.us/mirror-from-traefikservice created ~~~ ~~~powershell # vim mirror-from-traefikservice-ingressroute.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: ingressroute-mirror-traefikservice namespace: default spec: entryPoints: - web routes: - match: Host(`lb.kubemsb.com`) && PathPrefix(`/`) kind: Rule services: - name: mirror-from-traefikservice namespace: default kind: TraefikService ~~~ ~~~powershell # kubectl apply -f mirror-from-tradfikservice-ingressroute.yaml ingressroute.traefik.containo.us/ingressroute-mirror-traefikservice created ~~~ ![image-20220417101212376](../../img/kubernetes/kubernetes_traefik/image-20220417101212376.png) ![image-20220417101255133](../../img/kubernetes/kubernetes_traefik/image-20220417101255133.png) ![image-20220417101321919](../../img/kubernetes/kubernetes_traefik/image-20220417101321919.png) ~~~powershell # while true; do curl http://lb.kubemsb.com; sleep 1; done ~~~ ### 5.3.3 流量复制小结 通过上述的演示我们会发现所有的流量100%发送了svc1,有20%的流量被复制到svc2，且用户收到响应均来自svc1，svc2并没有响应，可通过查看svc1及svc2应用日志获取访问日志。 # 六、Kubernetes Gateway API ## 6.1 Gateway API介绍 ### 6.1.1 Gateway API架构 Gateway API（之前叫 Service API）是由 SIG-NETWORK 社区管理的开源项目，项目地址：https://gateway-api.sigs.k8s.io/。主要原因是 Ingress 资源对象不能很好的满足网络需求，很多场景下 Ingress 控制器都需要通过定义 annotations 或者 crd 来进行功能扩展，这对于使用标准和支持是非常不利的，新推出的 Gateway API 旨在通过可扩展的面向角色的接口来增强服务网络。 Gateway API 是 Kubernetes 中的一个 API 资源集合，包括 GatewayClass、Gateway、HTTPRoute、TCPRoute、Service 等，这些资源共同为各种网络用例构建模型。 ![image-20220418105312470](../../img/kubernetes/kubernetes_traefik/image-20220418105312470.png) Gateway API 的改进比当前的 Ingress 资源对象有很多更好的设计： - 面向角色 - Gateway 由各种 API 资源组成，这些资源根据使用和配置 Kubernetes 服务网络的角色进行建模。 - 通用性 - 和 Ingress 一样是一个具有众多实现的通用规范，Gateway API 是一个被设计成由许多实现支持的规范标准。 - 更具表现力 - Gateway API 资源支持基于 Header 头的匹配、流量权重等核心功能，这些功能在 Ingress 中只能通过自定义注解才能实现。 - 可扩展性 - Gateway API 允许自定义资源链接到 API 的各个层，这就允许在 API 结构的适当位置进行更精细的定制。 还有一些其他值得关注的功能： - GatewayClasses - GatewayClasses 将负载均衡实现的类型形式化，这些类使用户可以很容易了解到通过 Kubernetes 资源可以获得什么样的能力。 - 共享网关和跨命名空间支持 - 它们允许共享负载均衡器和 VIP，允许独立的路由资源绑定到同一个网关，这使得团队可以安全地共享（包括跨命名空间）基础设施，而不需要直接协调。 - 规范化路由和后端 - Gateway API 支持类型化的路由资源和不同类型的后端，这使得 API 可以灵活地支持各种协议（如 HTTP 和 gRPC）和各种后端服务（如 Kubernetes Service、存储桶或函数）。 ### 6.1.2 面向角色设计 无论是道路、电力、数据中心还是 Kubernetes 集群，基础设施都是为了共享而建的，然而共享基础设施提供了一个共同的挑战，那就是如何为基础设施用户提供灵活性的同时还能被所有者控制。 Gateway API 通过对 Kubernetes 服务网络进行面向角色的设计来实现这一目标，平衡了灵活性和集中控制。它允许共享的网络基础设施（硬件负载均衡器、云网络、集群托管的代理等）被许多不同的团队使用，所有这些都受到集群运维设置的各种策略和约束。下面的例子显示了是如何在实践中运行的。 ![image-20220418133026558](../../img/kubernetes/kubernetes_traefik/image-20220418133026558.png) 一个集群运维人员创建了一个基于 GatewayClass 的 Gateway 资源，这个 Gateway 部署或配置了它所代表的基础网络资源，集群运维和特定的团队必须沟通什么可以附加到这个 Gateway 上来暴露他们的应用。集中的策略，如 TLS，可以由集群运维在 Gateway 上强制执行，同时，Store 和 Site 应用在他们自己的命名空间中运行，但将他们的路由附加到相同的共享网关上，允许他们独立控制他们的路由逻辑。 这种关注点分离的设计可以使不同的团队能够管理他们自己的流量，同时将集中的策略和控制留给集群运维。 ### 6.1.3 Gateway API概念 在整个 Gateway API 中涉及到3个角色：基础设施提供商、集群管理员、应用开发人员，在某些场景下可能还会涉及到应用管理员等角色。Gateway API 中定义了3种主要的资源模型：GatewayClass、Gateway、Route。 #### 6.1.3.1 GatewayClass GatewayClass 定义了一组共享相同配置和动作的网关。每个GatewayClass 由一个控制器处理，是一个集群范围的资源，必须至少有一个 GatewayClass 被定义。 这与 Ingress 的 IngressClass 类似，在 Ingress v1beta1 版本中，与 GatewayClass 类似的是 ingress-class 注解，而在Ingress V1 版本中，最接近的就是 IngressClass 资源对象。 #### 6.1.3.2 Gateway Gateway 网关描述了如何将流量转化为集群内的服务，也就是说，它定义了一个请求，要求将流量从不了解 Kubernetes 的地方转换到集群内的服务。例如，由云端负载均衡器、集群内代理或外部硬件负载均衡器发送到 Kubernetes 服务的流量。 它定义了对特定负载均衡器配置的请求，该配置实现了 GatewayClass 的配置和行为规范，该资源可以由管理员直接创建，也可以由处理 GatewayClass 的控制器创建。 Gateway 可以附加到一个或多个路由引用上，这些路由引用的作用是将流量的一个子集导向特定的服务。 #### 6.1.3.3 Route 资源 路由资源定义了特定的规则，用于将请求从网关映射到 Kubernetes 服务。 从 v1alpha2 版本开始，API 中包含四种 Route 路由资源类型，对于其他未定义的协议，鼓励采用特定实现的自定义路由类型，当然未来也可能会添加新的路由类型。 ##### 6.1.3.3.1 HTTPRoute HTTPRoute 适用于 HTTP 或 HTTPS 连接，适用于我们想要检查 HTTP 请求并使用 HTTP 请求进行路由或修改的场景，比如使用 HTTP Headers 头进行路由，或在请求过程中对它们进行修改。 ##### 6.1.3.3.2 TLSRoute TLSRoute 用于 TLS 连接，通过 SNI 进行区分，它适用于希望使用 SNI 作为主要路由方法的地方，并且对 HTTP 等更高级别协议的属性不感兴趣，连接的字节流不经任何检查就被代理到后端。 ##### 6.1.3.3.3 TCPRoute 和 UDPRoute TCPRoute（和UDPRoute）旨在用于将一个或多个端口映射到单个后端。在这种情况下，没有可以用来选择同一端口的不同后端的判别器，所以每个 TCPRoute 在监听器上需要一个不同的端口。你可以使用 TLS，在这种情况下，未加密的字节流会被传递到后端，当然也可以不使用 TLS，这样加密的字节流将传递到后端。 ### 6.1.4 Gateway API概念之间关系 GatewayClass、Gateway、xRoute 和 Service 的组合定义了一个可实施的负载均衡器。下图说明了不同资源之间的关系: ![image-20220418133554117](../../img/kubernetes/kubernetes_traefik/image-20220418133554117.png) 使用反向代理实现的网关的典型客户端/网关 API 请求流程如下所示： 1. 客户端向 http://foo.example.com 发出请求 2. DNS 将域名解析为 Gateway 网关地址 3. 反向代理在监听器上接收请求，并使用 Host Header 来匹配HTTPRoute 4. (可选)反向代理可以根据 HTTPRoute 的匹配规则进行路由 5. (可选)反向代理可以根据 HTTPRoute 的过滤规则修改请求，即添加或删除 headers 6. 最后，反向代理根据 HTTPRoute 的 forwardTo 规则，将请求转发给集群中的一个或多个对象，即服务。 ## 6.2 kubernetes gateway CRD安装 要在 Traefik 中使用 Gateway API，首先我们需要先手动安装 Gateway API 的 CRDs，使用如下命令即可安装，这将安装包括 GatewayClass、Gateway、HTTPRoute、TCPRoute 等 CRDs： ~~~powershell # kubectl apply -k "github.com/kubernetes-sigs/service-apis/config/crd?ref=v0.3.0" 输出： customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/referencepolicies.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io created ~~~ ![image-20220418121243955](../../img/kubernetes/kubernetes_traefik/image-20220418121243955.png) ## 6.3 为traefik授权（RBAC） ~~~powershell # vim traefik-gatewayapi-rbac.yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: gateway-role rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - networking.x-k8s.io resources: - gatewayclasses - gateways - httproutes - tcproutes - tlsroutes verbs: - get - list - watch - apiGroups: - networking.x-k8s.io resources: - gatewayclasses/status - gateways/status - httproutes/status - tcproutes/status - tlsroutes/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gateway-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: gateway-role subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: kube-system ~~~ ## 6.4 Traefik开启Gateway api支持 ~~~powershell # vim traefik-configmap.yaml ...... providers: kubernetesCRD: "" kubernetesingress: "" kubernetesGateway: "" 添加此行 experimental: 添加此行 kubernetesGateway: true 添加此行 ...... ~~~ ~~~powershell 删除 # kubectl delete -f traefik-configmap.yaml # kubectl delete -f traefik-deploy.yaml ~~~ ~~~powershell 重新运行 # kubectl apply -f traefik-configmap.yaml # kubectl apply -f traefik-deploy.yaml ~~~ ![image-20220418134757719](../../img/kubernetes/kubernetes_traefik/image-20220418134757719.png) ## 6.5 创建Gateway API的GatewayClass ~~~powershell # vim gatewayclass.yaml apiVersion: networking.x-k8s.io/v1alpha1 kind: GatewayClass metadata: name: traefik spec: controller: traefik.io/gateway-controller ~~~ ~~~powershell # kubectl apply -f gatewayclass.yaml ~~~ ~~~powershell # kubectl get gatewayclass NAME CONTROLLER AGE traefik traefik.io/gateway-controller 2m59s ~~~ ## 6.6 Gateway API应用案例 ### 6.6.1 通过Gateway API方式暴露traefik dashboard #### 6.6.1.1 创建gateway ~~~powershell # vim gateway.yaml apiVersion: networking.x-k8s.io/v1alpha1 kind: Gateway metadata: name: http-gateway namespace: kube-system spec: gatewayClassName: traefik listeners: - protocol: HTTP port: 80 routes: kind: HTTPRoute namespaces: from: All selector: matchLabels: app: traefik ~~~ ~~~powershell # kubectl apply -f gateway.yaml gateway.networking.x-k8s.io/http-gateway created ~~~ ~~~powershell # kubectl get gateway NAME CLASS AGE http-gateway traefik 6s ~~~ #### 6.5.1.2 创建HTTPRoute ~~~powershell # vim httproute.yaml apiVersion: networking.x-k8s.io/v1alpha1 kind: HTTPRoute metadata: name: traefix-dashboard-gateway-api-route namespace: kube-system labels: app: traefik spec: hostnames: - "traefikdashboard.kubemsb.com" rules: - matches: - path: type: Prefix value: / forwardTo: - serviceName: traefik port: 8080 weight: 1 ~~~ ~~~powershell # kubectl apply -f httproute.yaml httproute.networking.x-k8s.io/traefix-dashboard-gateway-api-route created ~~~ ~~~powershell # kubectl get httproute NAME HOSTNAMES AGE traefix-dashboard-gateway-api-route ["traefikdashboard.kubemsb.com"] 6s ~~~ #### 6.5.1.3 在集群之外主机访问 ~~~powershell # vim /etc/hosts 192.168.10.12 traefikdashboard.kubemsb.com ~~~ ![image-20220418162326429](../../img/kubernetes/kubernetes_traefik/image-20220418162326429.png) ![image-20220418162355210](../../img/kubernetes/kubernetes_traefik/image-20220418162355210.png) ![image-20220418162539822](../../img/kubernetes/kubernetes_traefik/image-20220418162539822.png) ### 6.6.2 通过Gateway API方式暴露WEB应用 ~~~powershell # cat gatewayapi-web.yaml kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web-gatewayapi namespace: default spec: replicas: 1 selector: matchLabels: app: gatewayweb template: metadata: labels: app: gatewayweb spec: containers: - name: nginx-web image: nginx:latest lifecycle: postStart: exec: command: ["/bin/sh", "-c", "echo gatewayweb > /usr/share/nginx/html/index.html"] ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-web-gatewayapi-svc namespace: default spec: ports: - name: http port: 80 selector: app: gatewayweb ~~~ ~~~powershell # kubectl apply -f gatewayapi-web.yaml deployment.apps/nginx-web-gatewayapi created service/nginx-web-gatewayapi-svc created ~~~ ~~~powershell # vim gatewayapi-web-gateway.yaml apiVersion: networking.x-k8s.io/v1alpha1 kind: Gateway metadata: name: nginx-web-gateway spec: gatewayClassName: traefik listeners: - protocol: HTTP port: 80 routes: kind: HTTPRoute namespaces: from: All selector: matchLabels: app: gatewayweb ~~~ ~~~powershell # kubectl apply -f gatewayapi-web-gateway.yaml gateway.networking.x-k8s.io/nginx-web-gateway created ~~~ ~~~powershell # kubectl get gateway NAME CLASS AGE nginx-web-gateway traefik 7s ~~~ ~~~powershell # kubectl describe gateway nginx-web-gateway ~~~ ~~~powershell # cat gatewayapi-web-httproute.yaml apiVersion: networking.x-k8s.io/v1alpha1 kind: HTTPRoute metadata: name: nginx-web-gateway-api-route labels: app: gatewayweb spec: hostnames: - "nginx.kubemsb.com" rules: - matches: - path: type: Prefix value: / forwardTo: - serviceName: nginx-web-gatewayapi-svc port: 80 weight: 1 ~~~ ~~~powershell # kubectl apply -f gatewayapi-web-httproute.yaml httproute.networking.x-k8s.io/nginx-web-gateway-api-route created ~~~ ~~~powershell kubectl get httproute NAME HOSTNAMES AGE nginx-web-gateway-api-route ["nginx.kubemsb.com"] 6s ~~~ **在集群之外主机访问** ~~~powershell # vim /etc/hosts 192.168.10.12 nginx.kubemsb.com ~~~ ~~~powershell # curl http://nginx.kubemsb.com gatewayweb ~~~ ### 6.6.3 金丝雀发布 >Gateway APIs 规范可以支持的另一个功能是金丝雀发布，假设你想在一个端点上运行两个不同的服务（或同一服务的两个版本），并将一部分请求路由到每个端点，则可以通过修改你的 HTTPRoute 来实现。 ~~~powershell # vim gateway-cn-deploy.yaml kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web1 namespace: kube-system spec: replicas: 1 selector: matchLabels: app: v1 template: metadata: labels: app: v1 spec: containers: - name: nginx-web-c image: nginx:latest lifecycle: postStart: exec: command: ["/bin/sh", "-c", "echo svc1 > /usr/share/nginx/html/index.html"] ports: - containerPort: 80 --- kind: Deployment apiVersion: apps/v1 metadata: name: nginx-web2 namespace: kube-system spec: replicas: 1 selector: matchLabels: app: v2 template: metadata: labels: app: v2 spec: containers: - name: nginx-web-c image: nginx:latest lifecycle: postStart: exec: command: ["/bin/sh", "-c", "echo svc2 > /usr/share/nginx/html/index.html"] ports: - containerPort: 80 ~~~ ~~~powershell # kubectl apply -f gateway-cn-deploy.yaml ~~~ ~~~powershell # vim gateway-cn-deploy-svc.yaml apiVersion: v1 kind: Service metadata: name: svc1 namespace: kube-system spec: ports: - name: http port: 80 selector: app: v1 --- apiVersion: v1 kind: Service metadata: name: svc2 namespace: kube-system spec: ports: - name: http port: 80 selector: app: v2 ~~~ ~~~powershell # kubectl apply -fgateway-cn-deploy-svc.yaml ~~~ ~~~powershell # vim gateway-cn-httproute.yaml apiVersion: networking.x-k8s.io/v1alpha1 kind: HTTPRoute metadata: labels: app: traefik name: nginxweb-app namespace: kube-system spec: hostnames: - canary.kubemsb.com rules: - forwardTo: - port: 80 serviceName: svc1 weight: 3 # 3/4 的请求到svc1 - port: 80 serviceName: svc2 weight: 1 # 1/4 的请求到svc2 ~~~ ~~~powershell # kubectl apply -f gateway-cn-httproute.yaml httproute.networking.x-k8s.io/nginxweb-app created ~~~ ~~~powershell # kubectl get httproute -n kube-system NAME HOSTNAMES AGE nginxweb-app ["canary.kubemsb.com"] 7s ~~~ **在 集群之外主机访问** ~~~powershell # vim /etc/hosts 192.168.10.12 canary.kubemsb.com ~~~ ~~~powershell [root@dockerhost ~]# curl http://canary.kubemsb.com svc1 [root@dockerhost ~]# curl http://canary.kubemsb.com svc1 [root@dockerhost ~]# curl http://canary.kubemsb.com svc1 [root@dockerhost ~]# curl http://canary.kubemsb.com svc2 ~~~ >以上使用 Traefik 来测试了 Kubernetes Gateway APIs 的使用。目前，Traefik 对 Gateway APIs 的实现是基于 v1alpha1 版本的规范，目前最新的规范是 v1alpha2，所以和最新的规范可能有一些出入的地方。 ================================================ FILE: docs/cloud/kubernetes/kubernetes_ui.md ================================================ # Kubernetes集群UI及主机资源监控 # 一、Kubernetes dashboard作用 - 通过dashboard能够直观了解Kubernetes集群中运行的资源对象 - 通过dashboard可以直接管理（创建、删除、重启等操作）资源对象 ![](../../img/kubernetes/kubernetes_ui/1573974396340.png) # 二、获取Kubernetes dashboard资源清单文件 ![image-20220324233108575](../../img/kubernetes/kubernetes_ui/image-20220324233108575.png) ![image-20220324233134810](../../img/kubernetes/kubernetes_ui/image-20220324233134810.png) ![image-20220324233255036](../../img/kubernetes/kubernetes_ui/image-20220324233255036.png) ~~~powershell wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml ~~~ # 三、修改并部署kubernetes dashboard资源清单文件 ~~~powershell [root@k8s-master1 ~]# vi recommended.yaml ...... 上述内容不变 为了方便在容器主机上访问，下面的service需要添加NodePort类型及端口 --- kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30000 selector: k8s-app: kubernetes-dashboard 此证书不注释，对于早期版本需要注释。 --- apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque --- ...... 中间内容不用改变 需要修改登录kubernetes dashboard后用户的身份，不然无法显示资源情况 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin 一定要把原来的kubernetes-dashboard修改为cluster-admin，不然进入UI后会报错。 subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard ...... 以下内容暂不修改 ~~~ ~~~powershell kubectl apply -f recommended.yaml ~~~ # 四、访问Kubernetes dashboard 使用：https://192.168.10.12:30000 访问 ![image-20220324233800297](../../img/kubernetes/kubernetes_ui/image-20220324233800297.png) ~~~powershell kubectl get secret -n kubernetes-dashboard NAME TYPE DATA AGE default-token-dzr9f kubernetes.io/service-account-token 3 3m59s kubernetes-dashboard-certs Opaque 0 3m59s kubernetes-dashboard-csrf Opaque 1 3m59s kubernetes-dashboard-key-holder Opaque 2 3m59s kubernetes-dashboard-token-g6pq7 kubernetes.io/service-account-token 3 3m59s 用此token ~~~ ~~~powershell kubectl describe secret kubernetes-dashboard-token-g6pq7 -n kubernetes-dashboard Name: kubernetes-dashboard-token-g6pq7 Namespace: kubernetes-dashboard Labels: Annotations: kubernetes.io/service-account.name: kubernetes-dashboard kubernetes.io/service-account.uid: 46292f26-3046-411a-a9df-eac200290722 Type: kubernetes.io/service-account-token Data ==== namespace: 20 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImVGc2xjT05uekl0MlVOZ0VCSlhHSURfOXd6WGFvVnZFZmNwREwtVk1STlEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1nNnBxNyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ2MjkyZjI2LTMwNDYtNDExYS1hOWRmLWVhYzIwMDI5MDcyMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.qJMnfpKvpZNziXfkamdtYyIHrPnwisJBjlyCg_XWHoVPs5gNouGfrYkcxUKMdP9pBa7n1TwrurL3ppZTOAJRNSGO94F7BOXIFZ8O1-Ff1LZicWQrikSXDDyyWWEWypPHBIgOTGN_HMFJnIF98JnYd8vzrVVZBfiXco6lkVOK4eTQY87FgB0iJtXWh5LITefkNJm2d8o0tn2zrVnRUZ_TYisnirJOOrlx-GzfnwlXQxdaQRxdgEHHK3-lNZli54XtjB7IwP5jaER4mQ_sMTxrEMC-If46_ftMQqKn3R6YTGTG8UP49Xji_tPp--L3RUQI7vakr0x5-Cv_y0JlKEmlog 复制token全部内容 ca.crt: 1359 bytes ~~~ ![image-20220324234042124](../../img/kubernetes/kubernetes_ui/image-20220324234042124.png) ![image-20220324234116954](../../img/kubernetes/kubernetes_ui/image-20220324234116954.png) # 五、使用metrics-server实现主机资源监控 ## 5.1 获取metrics-server资源清单文件 ![image-20220325021720671](../../img/kubernetes/kubernetes_ui/image-20220325021720671.png) ![image-20220325021757986](../../img/kubernetes/kubernetes_ui/image-20220325021757986.png) ![image-20220325021838791](../../img/kubernetes/kubernetes_ui/image-20220325021838791.png) ![image-20220325021932343](../../img/kubernetes/kubernetes_ui/image-20220325021932343.png) ~~~powershell wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.1/components.yaml ~~~ ## 5.2 修改metrics-server资源清单文件 ~~~powershell # vim components.yaml spec: containers: - args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-preferred-address-types=InternalIP,InternalDNS,ExternalDNS,ExternalIP,Hostname - --kubelet-use-node-status-port - --metric-resolution=15s - --kubelet-insecure-tls 添加此行内容 ~~~ ## 5.3 部署metrics-server资源清单文件 ~~~powershell # kubectl top nodes error: Metrics API not available ~~~ ~~~powershell # kubectl top pods error: Metrics API not available ~~~ ~~~powershell kubectl apply -f components.yaml ~~~ ## 5.4 验证及授权 ~~~powershell kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-7cc8dd57d9-fk777 1/1 Running 1 2d1h calico-node-57vrc 1/1 Running 0 2d1h calico-node-7828d 1/1 Running 0 2d1h calico-node-n264t 1/1 Running 0 2d1h calico-node-nkxrs 1/1 Running 0 2d1h coredns-675db8b7cc-jp54h 1/1 Running 0 2d1h metrics-server-8bb87844c-4ttp9 1/1 Running 0 9m52s 此pod ~~~ ~~~powershell # kubectl top nodes Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io) ~~~ ~~~powershell kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous ~~~ ~~~powershell # kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-master1 97m 4% 2497Mi 65% k8s-master2 133m 6% 2290Mi 60% k8s-master3 95m 4% 2215Mi 58% k8s-worker1 45m 2% 1062Mi 27% ~~~ ~~~powershell ]# kubectl top pods NAME CPU(cores) MEMORY(bytes) nginx-web-bbh48 0m 1Mi nginx-web-x85nl 0m 1Mi ~~~ ![image-20220325022749621](../../img/kubernetes/kubernetes_ui/image-20220325022749621.png) ================================================ FILE: docs/cloud/kubernetes/kubernetes_velero.md ================================================ # kubernetes集群备份与恢复管理利器Velero # 一、Velero简介 Velero 是一款可以安全的备份、恢复和迁移 Kubernetes 集群资源和持久卷等资源的备份恢复软件。 Velero 实现的 kubernetes 资源备份能力，可以轻松实现 Kubernetes 集群的数据备份和恢复、复制 kubernetes 集群资源到其他 kubernetes 集群或者快速复制生产环境到测试环境等功能，这种备份就类似于把资源的 yaml 文件进行整体备份，从而保障资源的完整性。 Velero 对存储的支持较好，可以支持很多种存储资源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等. # 二、Velero工作流程 ![img](../../img/kubernetes/kubernetes_velero/16200606_61bb2bae6996764949.png) # 三、Velero备份过程 - 本地 Velero 客户端发送备份指令。 - Kubernetes 集群内就会创建一个 Backup 对象。 - BackupController 监测 Backup 对象并开始备份过程。 - BackupController 会向 API Server 查询相关数据。 - BackupController 将查询到的数据备份到远端的对象存储。 # 四、Velero特性 Velero 目前包含以下特性： - 支持 Kubernetes 集群数据备份和恢复 - 支持复制当前 Kubernetes 集群的资源到其它 Kubernetes 集群 - 支持复制生产环境到开发以及测试环境 # 五、Velero组件 Velero 组件一共分两部分，分别是服务端和客户端。 - 服务端：运行在你 Kubernetes 的集群中 - 客户端：是一些运行在本地的命令行的工具，需要已配置好 kubectl 及集群 kubeconfig 的机器上 # 六、Velero支持备份存储 - AWS S3 以及兼容 S3 的存储，比如：Minio - Azure BloB 存储 - Google Cloud 存储 - 阿里云OSS # 七、Velero适用场景 - 灾备场景：提供备份恢复k8s集群的能力 - 迁移场景：提供拷贝集群资源到其他集群的能力（复制同步开发，测试，生产环境的集群配置，简化环境配置） # 八、Velero备份与etcd备份的区别 - 与 Etcd 备份相比，直接备份 Etcd 是将集群的全部资源备份起来。 - Velero 可以对 Kubernetes 集群内对象级别进行备份。 - 除了对 Kubernetes 集群进行整体备份外，Velero 还可以通过对 Type、Namespace、Label 等对象进行分类备份或者恢复。 > 注意：备份过程中创建的对象是不会被备份的。 # 九、Velero部署 | | | | | -------------- | ----------------------------------------- | -------------------------------------- | | Velero version | Expected Kubernetes version compatibility | Tested on Kubernetes version | | 1.11 | 1.18-latest | 1.23.10, 1.24.9, 1.25.5, and 1.26.1 | | 1.10 | 1.18-latest | 1.22.5, 1.23.8, 1.24.6 and 1.25.1 | | 1.9 | 1.18-latest | 1.20.5, 1.21.2, 1.22.5, 1.23, and 1.24 | | 1.8 | 1.18-latest | | ~~~powershell [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 55d v1.25.0 k8s-worker01 Ready 55d v1.25.0 k8s-worker02 Ready 55d v1.25.0 ~~~ ## 9.1 Velero工具获取 ![image-20230607153118915](../../img/kubernetes/kubernetes_velero/image-20230607153118915.png) ![image-20230607153146246](../../img/kubernetes/kubernetes_velero/image-20230607153146246.png) ![image-20230607153425305](../../img/kubernetes/kubernetes_velero/image-20230607153425305.png) ![image-20230607153540407](../../img/kubernetes/kubernetes_velero/image-20230607153540407.png) ![image-20230607153639779](../../img/kubernetes/kubernetes_velero/image-20230607153639779.png) ~~~powershell [root@k8s-master01 ~]# wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.0/velero-v1.11.0-linux-amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# ls velero-v1.11.0-linux-amd64.tar.gz ~~~ ## 9.2 部署Velero依赖后端存储MinIo ~~~powershell [root@k8s-master01 ~]# tar xf velero-v1.11.0-linux-amd64.tar.gz ~~~ ~~~powershell [root@k8s-master01 ~]# ls velero-v1.11.0-linux-amd64 [root@k8s-master01 ~]# cd velero-v1.11.0-linux-amd64/ [root@k8s-master01 velero-v1.11.0-linux-amd64]# ls examples LICENSE velero [root@k8s-master01 velero-v1.11.0-linux-amd64]# cd examples/ [root@k8s-master01 examples]# ls minio nginx-app ~~~ ~~~powershell [root@k8s-master01 examples]# cd minio/ [root@k8s-master01 minio]# ls 00-minio-deployment.yaml ~~~ ~~~powershell [root@k8s-master01 minio]# vim 00-minio-deployment.yaml [root@k8s-master01 minio]# cat 00-minio-deployment.yaml # Copyright 2017 the Velero contributors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. --- apiVersion: v1 kind: Namespace metadata: name: velero --- apiVersion: apps/v1 kind: Deployment metadata: namespace: velero name: minio labels: component: minio spec: strategy: type: Recreate selector: matchLabels: component: minio template: metadata: labels: component: minio spec: volumes: - name: storage emptyDir: {} - name: config emptyDir: {} containers: - name: minio image: minio/minio:latest imagePullPolicy: IfNotPresent args: - server - /storage - --config-dir=/config - --console-address - ":30605" env: - name: MINIO_ACCESS_KEY value: "minio" - name: MINIO_SECRET_KEY value: "minio123" ports: - name: web containerPort: 9000 - name: console containerPort: 30605 volumeMounts: - name: storage mountPath: "/storage" - name: config mountPath: "/config" --- apiVersion: v1 kind: Service metadata: namespace: velero name: minio labels: component: minio spec: # ClusterIP is recommended for production environments. # Change to NodePort if needed per documentation, # but only if you run Minio in a test/trial environment, for example with Minikube. type: NodePort ports: - name: web port: 9000 targetPort: 9000 nodePort: 31900 protocol: TCP - name: console port: 30605 targetPort: 30605 nodePort: 30605 protocol: TCP selector: component: minio --- apiVersion: batch/v1 kind: Job metadata: namespace: velero name: minio-setup labels: component: minio spec: template: metadata: name: minio-setup spec: restartPolicy: OnFailure volumes: - name: config emptyDir: {} containers: - name: mc image: minio/mc:latest imagePullPolicy: IfNotPresent command: - /bin/sh - -c - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero" volumeMounts: - name: config mountPath: "/config" ~~~ ~~~powershell [root@k8s-master01 minio]# pwd /root/velero-v1.11.0-linux-amd64/examples/minio [root@k8s-master01 minio]# ls 00-minio-deployment.yaml [root@k8s-master01 minio]# kubectl apply -f 00-minio-deployment.yaml namespace/velero created deployment.apps/minio created service/minio created job.batch/minio-setup created ~~~ > 注意：访问用户名为minio，密码为minio123 ~~~powershell [root@k8s-master01 minio]# kubectl get all -n velero NAME READY STATUS RESTARTS AGE pod/minio-6959795f76-fc4ls 1/1 Running 0 2m52s pod/minio-setup-fsfvm 0/1 Completed 0 2m51s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/minio NodePort 10.96.2.7 9000:31900/TCP,30605:30605/TCP 2m52s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/minio 1/1 1 1 2m52s NAME DESIRED CURRENT READY AGE replicaset.apps/minio-6959795f76 1 1 1 2m52s NAME COMPLETIONS DURATION AGE job.batch/minio-setup 1/1 4s 2m52s ~~~ ![image-20230607173840236](../../img/kubernetes/kubernetes_velero/image-20230607173840236.png) ![image-20230607173939822](../../img/kubernetes/kubernetes_velero/image-20230607173939822.png) ![image-20230607174053103](../../img/kubernetes/kubernetes_velero/image-20230607174053103.png) ## 9.3 安装Velero ### 9.3.1 创建访问minio密钥 ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# vim cred-velero [root@k8s-master01 velero-v1.11.0-linux-amd64]# cat cred-velero [default] aws_access_key_id = minio aws_secret_access_key = minio123 ~~~ 或 ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# cat > cred-velero << EOF [default] aws_access_key_id = minio aws_secret_access_key = minio123 EOF ~~~ ### 9.3.2 安装Velero客户端到当前系统 ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# pwd /root/velero-v1.11.0-linux-amd64 [root@k8s-master01 velero-v1.11.0-linux-amd64]# ls examples LICENSE velero [root@k8s-master01 velero-v1.11.0-linux-amd64]# cp velero /usr/bin/ [root@k8s-master01 velero-v1.11.0-linux-amd64]# ls /usr/bin/velero /usr/bin/velero ~~~ ### 9.3.3 在K8S集群快速安装Velero ~~~powershell [root@k8s-master01 ~]# velero completion bash ~~~ ~~~powershell [root@k8s-master01 ~]# dig -t a minio.velero.svc.cluster.local. @10.96.0.10 ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.9 <<>> -t a minio.velero.svc.cluster.local. @10.96.0.10 ;; global options: +cmd ;; Got answer: ;; WARNING: .local is reserved for Multicast DNS ;; You are currently testing what happens when an mDNS query is leaked to DNS ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43950 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;minio.velero.svc.cluster.local. IN A ;; ANSWER SECTION: minio.velero.svc.cluster.local. 30 IN A 10.96.2.7 ;; Query time: 0 msec ;; SERVER: 10.96.0.10#53(10.96.0.10) ;; WHEN: 三 6月 07 18:06:18 CST 2023 ;; MSG SIZE rcvd: 105 ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# ls cred-velero examples LICENSE velero velero.sh [root@k8s-master01 velero-v1.11.0-linux-amd64]# vim velero.sh [root@k8s-master01 velero-v1.11.0-linux-amd64]# cat velero.sh velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.0.0 \ --bucket velero \ --secret-file ./cred-velero \ --use-volume-snapshots=false \ --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc.cluster.local.:9000 ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# sh velero.sh CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client CustomResourceDefinition/backuprepositories.velero.io: created CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: attempting to create resource client CustomResourceDefinition/backups.velero.io: created CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client CustomResourceDefinition/backupstoragelocations.velero.io: created CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client CustomResourceDefinition/deletebackuprequests.velero.io: created CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client CustomResourceDefinition/downloadrequests.velero.io: created CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client CustomResourceDefinition/podvolumebackups.velero.io: created CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client CustomResourceDefinition/podvolumerestores.velero.io: created CustomResourceDefinition/restores.velero.io: attempting to create resource CustomResourceDefinition/restores.velero.io: attempting to create resource client CustomResourceDefinition/restores.velero.io: created CustomResourceDefinition/schedules.velero.io: attempting to create resource CustomResourceDefinition/schedules.velero.io: attempting to create resource client CustomResourceDefinition/schedules.velero.io: created CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client CustomResourceDefinition/serverstatusrequests.velero.io: created CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client CustomResourceDefinition/volumesnapshotlocations.velero.io: created Waiting for resources to be ready in cluster... Namespace/velero: attempting to create resource Namespace/velero: attempting to create resource client Namespace/velero: already exists, proceeding Namespace/velero: created ClusterRoleBinding/velero: attempting to create resource ClusterRoleBinding/velero: attempting to create resource client ClusterRoleBinding/velero: created ServiceAccount/velero: attempting to create resource ServiceAccount/velero: attempting to create resource client ServiceAccount/velero: created Secret/cloud-credentials: attempting to create resource Secret/cloud-credentials: attempting to create resource client Secret/cloud-credentials: created BackupStorageLocation/default: attempting to create resource BackupStorageLocation/default: attempting to create resource client BackupStorageLocation/default: created Deployment/velero: attempting to create resource Deployment/velero: attempting to create resource client Deployment/velero: created Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status. ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# kubectl api-versions | grep velero velero.io/v1 ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# kubectl get pods -n velero NAME READY STATUS RESTARTS AGE minio-6959795f76-fc4ls 1/1 Running 0 38m minio-setup-fsfvm 0/1 Completed 0 38m velero-84fff5d559-rr6bw 1/1 Running 0 93s ~~~ # 十、velero命令介绍 ~~~powershell [root@k8s-master01 ~]# velero create backup --help Create a backup Usage: velero create backup NAME [flags] Examples: # Create a backup containing all resources. velero backup create backup1 # Create a backup including only the nginx namespace. velero backup create nginx-backup --include-namespaces nginx # Create a backup excluding the velero and default namespaces. velero backup create backup2 --exclude-namespaces velero,default # Create a backup based on a schedule named daily-backup. velero backup create --from-schedule daily-backup # View the YAML for a backup that doesn't snapshot volumes, without sending it to the server. velero backup create backup3 --snapshot-volumes=false -o yaml # Wait for a backup to complete before returning from the command. velero backup create backup4 --wait ~~~ ~~~powershell # 剔除 namespace --exclude-namespaces stringArray namespaces to exclude from the backup # 剔除资源类型 --exclude-resources stringArray resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io # 包含集群资源类型 --include-cluster-resources optionalBool[=true] include cluster-scoped resources in the backup # 包含 namespace --include-namespaces stringArray namespaces to include in the backup (use '*' for all namespaces) (default *) # 包含 namespace 资源类型 --include-resources stringArray resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources) # 给这个备份加上标签 --labels mapStringString labels to apply to the backup -o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command. # 对指定标签的资源进行备份 -l, --selector labelSelector only back up resources matching this label selector (default ) # 对 PV 创建快照 --snapshot-volumes optionalBool[=true] take snapshots of PersistentVolumes as part of the backup # 指定备份的位置 --storage-location string location in which to store the backup # 备份数据多久删掉 --ttl duration how long before the backup can be garbage collected (default 720h0m0s) # 指定快照的位置，也就是哪一个公有云驱动 --volume-snapshot-locations strings list of locations (at most one per provider) where volume snapshots should be stored ~~~ # 十一、使用velero实现k8s资源对象备份 ## 11.1 创建用于备份测试应用 ~~~powershell [root@k8s-master01 ~]# ls velero-v1.11.0-linux-amd64 cred-velero examples LICENSE velero velero.sh [root@k8s-master01 ~]# ls velero-v1.11.0-linux-amd64/examples/ minio nginx-app [root@k8s-master01 ~]# ls velero-v1.11.0-linux-amd64/examples/nginx-app/ base.yaml README.md with-pv.yaml [root@k8s-master01 ~]# cat velero-v1.11.0-linux-amd64/examples/nginx-app/base.yaml # Copyright 2017 the Velero contributors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. --- apiVersion: v1 kind: Namespace metadata: name: nginx-example labels: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment namespace: nginx-example spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx:1.17.6 name: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: labels: app: nginx name: my-nginx namespace: nginx-example spec: ports: - port: 80 targetPort: 80 selector: app: nginx type: LoadBalancer ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl apply -f velero-v1.11.0-linux-amd64/examples/nginx-app/base.yaml namespace/nginx-example created deployment.apps/nginx-deployment created service/my-nginx created ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get all -n nginx-example NAME READY STATUS RESTARTS AGE pod/nginx-deployment-747864f4b5-5r7kd 1/1 Running 0 39s pod/nginx-deployment-747864f4b5-9n5gz 1/1 Running 0 39s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/my-nginx LoadBalancer 10.96.1.113 80:32403/TCP 39s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-deployment 2/2 2 2 39s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-deployment-747864f4b5 2 2 2 39s ~~~ ## 11.2 创建备份 ~~~powershell [root@k8s-master01 ~]# velero backup create nginx-backup --include-namespaces nginx-example Backup request "nginx-backup" submitted successfully. Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details. ~~~ ## 11.3 查看备份 ~~~powershell [root@k8s-master01 ~]# velero backup describe nginx-backup Name: nginx-backup Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/source-cluster-k8s-gitversion=v1.25.5 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=25 Phase: Completed Namespaces: Included: nginx-example Excluded: Resources: Included: * Excluded: Cluster-scoped: auto Label selector: Storage Location: default Velero-Native Snapshot PVs: auto TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 1h0m0s Hooks: Backup Format Version: 1.1.0 Started: 2023-06-08 10:49:07 +0800 CST Completed: 2023-06-08 10:49:08 +0800 CST Expiration: 2023-07-08 10:49:07 +0800 CST Velero-Native Snapshots: ~~~ ~~~powershell 查看备份位置 [root@k8s-master01 ~]# velero backup-location get NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT default aws velero Available 2023-06-09 10:16:49 +0800 CST ReadWrite true ~~~ ~~~powershell 查看备份文件 [root@k8s-master01 ~]# kubectl get backups.velero.io -n velero NAME AGE nginx-backup 23h ~~~ ![image-20230608110056263](../../img/kubernetes/kubernetes_velero/image-20230608110056263.png) ![image-20230608110346005](../../img/kubernetes/kubernetes_velero/image-20230608110346005.png) # 十二、使用velero实现对K8S资源对象进行恢复 ## 12.1 删除备份测试应用 ~~~powershell [root@k8s-master01 ~]# kubectl delete -f velero-v1.11.0-linux-amd64/examples/nginx-app/base.yaml namespace "nginx-example" deleted deployment.apps "nginx-deployment" deleted service "my-nginx" deleted ~~~ ## 12.2 恢复备份测试应用 ~~~powershell [root@k8s-master01 ~]# velero restore create --from-backup nginx-backup --wait Restore request "nginx-backup-20230608111349" submitted successfully. Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background. Restore completed with status: Completed. You may check for more information using the commands `velero restore describe nginx-backup-20230608111349` and `velero restore logs nginx-backup-20230608111349`. ~~~ ~~~powershell [root@k8s-master01 ~]# kubectl get pods -n nginx-example NAME READY STATUS RESTARTS AGE nginx-deployment-747864f4b5-5r7kd 1/1 Running 0 37s nginx-deployment-747864f4b5-9n5gz 1/1 Running 0 37s ~~~ # 十三、周期性备份任务 ~~~powershell # Create a backup every 6 hours velero create schedule NAME --schedule="0 */6 * * *" # Create a backup every 6 hours with the @every notation velero create schedule NAME --schedule="@every 6h" # Create a daily backup of the web namespace velero create schedule NAME --schedule="@every 24h" --include-namespaces web # Create a weekly backup, each living for 90 days (2160 hours) velero create schedule NAME --schedule="@every 168h" --ttl 2160h0m0s # 每日对anchnet-devops-dev/anchnet-devops-test/anchnet-devops-prod/xxxxx-devops-common-test 名称空间进行备份 velero create schedule anchnet-devops-dev --schedule="@every 24h" --include-namespaces xxxxx-devops-dev velero create schedule anchnet-devops-test --schedule="@every 24h" --include-namespaces xxxxx-devops-test velero create schedule anchnet-devops-prod --schedule="@every 24h" --include-namespaces xxxxx-devops-prod velero create schedule anchnet-devops-common-test --schedule="@every 24h" --include-namespaces xxxxx-devops-common-test ~~~ ~~~powershell 案例： [root@k8s-master01 ~]# velero create schedule nginx-backups --schedule="0 */1 * * *" --include-namespaces nginx-example Schedule "nginx-backups" created successfully. [root@k8s-master01 ~]# velero get schedules NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR PAUSED nginx-backups Enabled 2023-06-09 10:36:08 +0800 CST 0 */1 * * * 0s n/a false ~~~ > 注意事项： > > - 在velero备份的时候，备份过程中创建的对象是不会被备份的。 > - velero restore 恢复不会覆盖已有的资源，只恢复当前集群中不存在的资源。已有的资源不会回滚到之前的版本，如需要回滚，需在restore之前提前删除现有的资源。 > - velero也可作为一个crontjob来运行，定期备份数据。 # 十四、在其它K8S集群中部署velero及恢复应用 ## 14.1 在其它K8S集群中部署velero ~~~powershell 从第一个K8S集群中复制文件到第二个K8S集群master节点 [root@k8s-master01 ~]# scp -r velero-v1.11.0-linux-amd64 192.168.10.160:/root ~~~ ~~~powershell [root@k8s-master01 ~]# cp velero-v1.11.0-linux-amd64/velero /usr/bin/ ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# cat cred-velero [default] aws_access_key_id = minio aws_secret_access_key = minio123 [root@k8s-master01 velero-v1.11.0-linux-amd64]# cat velero.sh velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.0.0 \ --bucket velero \ --secret-file ./cred-velero \ --use-volume-snapshots=false \ --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc.cluster.local.:9000 [root@k8s-master01 velero-v1.11.0-linux-amd64]# vim velero.sh [root@k8s-master01 velero-v1.11.0-linux-amd64]# cat velero.sh velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.0.0 \ --bucket velero \ --secret-file ./cred-velero \ --use-volume-snapshots=false \ --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.10.140:31900 ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# sh velero.sh CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client CustomResourceDefinition/backuprepositories.velero.io: created CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: attempting to create resource client CustomResourceDefinition/backups.velero.io: created CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client CustomResourceDefinition/backupstoragelocations.velero.io: created CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client CustomResourceDefinition/deletebackuprequests.velero.io: created CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client CustomResourceDefinition/downloadrequests.velero.io: created CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client CustomResourceDefinition/podvolumebackups.velero.io: created CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client CustomResourceDefinition/podvolumerestores.velero.io: created CustomResourceDefinition/restores.velero.io: attempting to create resource CustomResourceDefinition/restores.velero.io: attempting to create resource client CustomResourceDefinition/restores.velero.io: created CustomResourceDefinition/schedules.velero.io: attempting to create resource CustomResourceDefinition/schedules.velero.io: attempting to create resource client CustomResourceDefinition/schedules.velero.io: created CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client CustomResourceDefinition/serverstatusrequests.velero.io: created CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client CustomResourceDefinition/volumesnapshotlocations.velero.io: created Waiting for resources to be ready in cluster... Namespace/velero: attempting to create resource Namespace/velero: attempting to create resource client Namespace/velero: already exists, proceeding Namespace/velero: created ClusterRoleBinding/velero: attempting to create resource ClusterRoleBinding/velero: attempting to create resource client ClusterRoleBinding/velero: created ServiceAccount/velero: attempting to create resource ServiceAccount/velero: attempting to create resource client ServiceAccount/velero: created Secret/cloud-credentials: attempting to create resource Secret/cloud-credentials: attempting to create resource client Secret/cloud-credentials: created BackupStorageLocation/default: attempting to create resource BackupStorageLocation/default: attempting to create resource client BackupStorageLocation/default: created Deployment/velero: attempting to create resource Deployment/velero: attempting to create resource client Deployment/velero: created Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status. ~~~ ~~~powershell [root@k8s-master01 velero-v1.11.0-linux-amd64]# kubectl get pods -n velero NAME READY STATUS RESTARTS AGE velero-84fff5d559-x82gh 1/1 Running 0 50s ~~~ ## 14.2 恢复应用到新K8S集群 ~~~powershell 查看备份位置 [root@k8s-master01 velero-v1.11.0-linux-amd64]# velero backup-location get NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT default aws velero Available 2023-06-09 10:16:44 +0800 CST ReadWrite true ~~~ ~~~powershell 查看备份文件 [root@k8s-master01 velero-v1.11.0-linux-amd64]# kubectl get backups.velero.io -n velero NAME AGE nginx-backup 4m7s ~~~ ~~~powershell 恢复备份到新K8S集群 [root@k8s-master01 ~]# velero restore create \ --namespace velero \ --from-backup nginx-backup --wait ~~~ ~~~powershell 查看备份恢复情况 [root@k8s-master01 ~]# kubectl get ns NAME STATUS AGE calico-apiserver Active 56d calico-system Active 56d default Active 56d kube-node-lease Active 56d kube-public Active 56d kube-system Active 56d nginx-example Active 7s tigera-operator Active 56d velero Active 6m17s [root@k8s-master01 ~]# kubectl get pods -n nginx-example NAME READY STATUS RESTARTS AGE nginx-deployment-747864f4b5-5r7kd 1/1 Running 0 22s nginx-deployment-747864f4b5-9n5gz 1/1 Running 0 22s ~~~ ================================================ FILE: docs/cloud/kubernetes/kubernetes_way.md ================================================ # Kubernetes集群部署方式说明 # 一、本地化部署 ## 1.1 kubeadm ![image-20220324204203839](../../img/kubernetes/kubernetes_way/image-20220324204203839.png) - Kubeadm 是一个工具，旨在提供创建 Kubernetes 集群`kubeadm init`的`kubeadm join`最佳实践“快速路径”。 - kubeadm 执行必要的操作以启动并运行最小的可行集群。 - 按照设计，它只关心引导，而不关心配置机器。 - 同样，安装各种不错的插件，如 Kubernetes 仪表板、监控解决方案和特定于云的插件，也不在范围内。 - 相反，我们希望在 kubeadm 之上构建更高级别和更量身定制的工具，理想情况下，使用 kubeadm 作为所有部署的基础将更容易创建符合要求的集群。 - 用法参考链接 - https://kubernetes.io/docs/reference/setup-tools/kubeadm/ ## 1.2 minikube - minikube适合用于部署本地kubernetes集群，此集群主要用于测试目的 - minikube可以快速让你在单主机上部署kubernetes集群 - 可以跨平台部署kubernetes集群（Linux、MAC、Windowns） ## 1.3 二进制部署方式 - 纯人肉方式部署 - 企业生产级别的部署方式 - 部署时间长 - 需要配置内容： - 证书 - 服务配置文件 - 使用systemd管理服务管理文件 - kubeconfig ## 1.4 国内第三方部署工具 - rke - 是一个快速的，多功能的kubernetes集群部署工具 - 仅通过一个配置文件就可以快速完成kubernetes集群部署 - 方便在kubernetes中添加任何节点数量的主机 - kubekey ![image-20220324205307087](../../img/kubernetes/kubernetes_way/image-20220324205307087.png) - kubeykey是KubeSphere基于Go语言开发的kubernetes集群安装工具，可以轻松、高效、灵活地单独或整体安装Kubernetes和KubeSphere，底层使用Kubeadm在多个节点上并行安装Kubernetes 集群，支持创建、扩缩和升级 Kubernetes 集群。 - KubeKey 提供内置高可用模式，支持一键安装高可用 Kubernetes 集群。 - KubeKey 不仅能帮助用户在线创建集群，还能作为离线安装解决方案。 - KubeKey可以用于以下三种安装场景： - 仅安装 Kubernetes集群 - 一键安装 Kubernetes 和 KubeSphere - 已有Kubernetes集群，使用ks-installer 在其上部署 KubeSphere - kubeasz ![image-20220324205820518](../../img/kubernetes/kubernetes_way/image-20220324205820518.png) 项目致力于提供快速部署高可用k8s集群的工具, 同时也努力成为k8s实践、使用的参考书；基于二进制方式部署和利用ansible-playbook实现自动化；既提供一键安装脚本, 也可以根据安装指南分步执行安装各个组件。 - 集群特性 TLS双向认证、RBAC授权、多Master高可用、支持Network Policy、备份恢复、离线安装 - 集群版本 kubernetes v1.20, v1.21, v1.22, v1.23 - 操作系统 CentOS/RedHat 7, Debian 9/10, Ubuntu 16.04/18.04/20.04 - 运行时 docker 19.03.x, 20.10.x containerd v1.5.8 - 网络 calico, cilium, flannel, kube-ovn, kube-router - 项目代码托管网址：https://github.com/easzlab/kubeasz # 二、公有云平台部署 ## 2.1 公有云平台提供容器云服务 - 阿里云 - ACK ![image-20220324210529285](../../img/kubernetes/kubernetes_way/image-20220324210529285.png) - 华为云 - CCE ![image-20220324210634327](../../img/kubernetes/kubernetes_way/image-20220324210634327.png) - 腾讯云 - EKS ![image-20220324210727688](../../img/kubernetes/kubernetes_way/image-20220324210727688.png) ## 2.2 在公有云平台部署 - kubeadm - minikube - 二进制方式部署 - rke - kubekey - kubeasz ================================================ FILE: docs/cloud/kubernetes/kubernetes_zookeeper.md ================================================ # 企业级中间件上云部署 zookeeper # 一、环境说明 - storageclass - ingress # 二、zookeeper部署YAML资源清单准备 ~~~powershell # vim zookeeper.yaml # cat zookeeper.yaml --- apiVersion: v1 kind: Namespace metadata: name: smart --- apiVersion: v1 kind: Service metadata: name: zk-headless namespace: smart labels: app: zk spec: ports: - port: 2888 name: server - port: 3888 name: leader-election clusterIP: None selector: app: zk --- apiVersion: v1 kind: Service metadata: name: zk-np namespace: smart labels: app: zk spec: type: NodePort ports: - port: 2181 targetPort: 2181 name: client nodePort: 32181 selector: app: zk --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: zk-pdb namespace: smart spec: selector: matchLabels: app: zk maxUnavailable: 1 --- apiVersion: apps/v1 kind: StatefulSet metadata: name: zookeeper namespace: smart spec: serviceName: zk-headless replicas: 3 updateStrategy: type: RollingUpdate selector: matchLabels: app: zk template: metadata: labels: app: zk spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - zk topologyKey: "kubernetes.io/hostname" containers: - name: kubernetes-zookeeper image: k8s.gcr.io/kubernetes-zookeeper:1.0-3.4.10 imagePullPolicy: Always resources: requests: memory: "1Gi" cpu: "0.5" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election command: - sh - -c - "start-zookeeper \ --servers=3 \ --data_dir=/var/lib/zookeeper/data \ --data_log_dir=/var/lib/zookeeper/data/log \ --conf_dir=/opt/zookeeper/conf \ --client_port=2181 \ --election_port=3888 \ --server_port=2888 \ --tick_time=2000 \ --init_limit=10 \ --sync_limit=5 \ --heap=512M \ --max_client_cnxns=60 \ --snap_retain_count=3 \ --purge_interval=12 \ --max_session_timeout=40000 \ --min_session_timeout=4000 \ --log_level=INFO" readinessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 10 timeoutSeconds: 5 livenessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 10 timeoutSeconds: 5 volumeMounts: - name: datadir mountPath: /var/lib/zookeeper - name: localtime mountPath: /etc/localtime volumes: - name: localtime hostPath: path: /etc/localtime type: '' volumeClaimTemplates: - metadata: name: datadir annotations: volume.alpha.kubernetes.io/storage-class: anything spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "nfs-client" resources: requests: storage: 1Gi ~~~ **注意：** Pod Disruption Budget (pod 中断 预算)，含义其实是 终止pod前 通过labelSelector机制获取正常运行的pod数目的限制，目的是对主动驱逐的保护措施。 - 场景 节点维护或升级时(kubectl drain) 对应用的自动缩容操作(autoscaling down) 由于节点不可用(not ready)导致的Pod驱逐就不能称之为主动 - 特性 PDB指定一个pod集合在一段时间内存活的最小实例数量或者百分比 作用于一组被同一个控制器管理的pod。例如：RC或者statefulapp 使用PodDisruptionBudget控制器本身无法真正保障指定数量或者百分比的pod存活，PodDisruptionBudget控制器只能保证POD主动逃离的情况下业务不中断或者业务SLA不降级 - 场景局限于：主动驱逐 主动驱逐的场景，如果能够保持存活pod数量，将会非常有用。通过使用Pod Disruption Budget 对象，应用可以保证那些主动移除pod的集群操作永远不会同一时间停掉太多pod，导致服务中断或者服务降级。 kubectl drain 操作时遵循PDB对象的设定，如果在该节点上运行了属于统一服务的多个pod，则为了保证最少存活数量，系统会确保每终止一个pod就会在健康的node上启动新的pod后，再继续终止下一个pod容器。 从版本1.7开始可以通过两个参数来配置PodDisruptionBudget： 1、 MinAvailable参数：表示最小可用POD数，表示应用POD集群处于运行状态的最小POD数量，或者是运行状态的POD数同总POD数的最小百分比。 2、 MaxUnavailable参数：表示最大不可用PO数，表示应用POD集群处于不可用状态的最大POD数，或者是不可用状态的POD数同总POD数的最大百分比。 这里需要注意的是，MinAvailable参数和MaxUnavailable参数是互斥的，也就是说如果使用了其中一个参数，那么就不能使用另外一个参数了。 比如当进行kubectl drain或者POD主动逃离的时候，kubernetes可以通过下面几种情况来判断是否允许： 1、 minAvailable设置成了数值5：应用POD集群中最少要有5个健康可用的POD，那么就可以进行操作。 2、 minAvailable设置成了百分数30%：应用POD集群中最少要有30%的健康可用POD，那么就可以进行操作。 3、 maxUnavailable设置成了数值5：应用POD集群中最多只能有5个不可用POD，才能进行操作。 4、 maxUnavailable设置成了百分数30%：应用POD集群中最多只能有30%个不可用POD，才能进行操作。 在极端的情况下，比如将maxUnavailable设置成0，或者设置成100%，那么就表示不能进行kubectl drain操作。同理将minAvailable设置成100%，或者设置成应用POD集群最大副本数，也表示不能进行kubectl drain操作。 这里面需要注意的是，使用PodDisruptionBudget控制器并不能保证任何情况下都对业务POD集群进行约束，PodDisruptionBudget控制器只能保证POD主动逃离的情况下业务不中断或者业务SLA不降级，例如在执行kubectldrain命令时。 # 三、zookeeper部署及部署验证 ~~~powershell # kubectl apply -f zookeeper.yaml ~~~ ~~~powershell # kubectl get sts -n smart NAME READY AGE zookeeper 3/3 21m ~~~ ~~~powershell # kubectl get pods -n smart NAME READY STATUS RESTARTS AGE zookeeper-0 1/1 Running 0 22m zookeeper-1 1/1 Running 0 21m zookeeper-2 1/1 Running 0 21m ~~~ ~~~powershell # kubectl get svc -n smart NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE zk-headless ClusterIP None 2888/TCP,3888/TCP 22m zk-np NodePort 10.96.0.124 2181:32181/TCP 22m ~~~ # 四、zookeeper应用验证 zookeeper访问连接信息验证 ~~~powershell # dig -t a zk-headless.smart.svc.cluster.local @10.96.0.10 ~~~ ~~~powershell # dig -t a zk-np.smart.svc.cluster.local @10.96.0.10 ~~~ 在kubernetes集群内访问验证 ~~~powershell # kubectl get pods -n smart NAME READY STATUS RESTARTS AGE zookeeper-0 1/1 Running 0 24m zookeeper-1 1/1 Running 0 24m zookeeper-2 1/1 Running 0 23m ~~~ ~~~powershell # kubectl exec -it zookeeper-0 -n smart -- bash ~~~ ~~~powershell root@zookeeper-0:/# zkCli.sh [zk: localhost:2181(CONNECTED) 0] create /key100 values100 Created /key100 [zk: localhost:2181(CONNECTED) 1] get /key100 values100 [zk: localhost:2181(CONNECTED) 2] quit ~~~ 在kubernetes集群外访问验证 ![image-20220725223714016](../../img/kubernetes/kubernetes_zookeeper/image-20220725223714016.png) ![image-20220725223754534](../../img/kubernetes/kubernetes_zookeeper/image-20220725223754534.png) ![image-20220725223823061](../../img/kubernetes/kubernetes_zookeeper/image-20220725223823061.png) ~~~powershell # wget https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz ~~~ ~~~powershell # ls apache-zookeeper-3.6.3-bin.tar.gz # tar xf apache-zookeeper-3.6.3-bin.tar.gz # ls apache-zookeeper-3.6.3-bin.tar.gz apache-zookeeper-3.6.3-bin ~~~ ~~~powershell # cd apache-zookeeper-3.6.3-bin/ [root@localhost apache-zookeeper-3.6.3-bin]# cd bin/ [root@localhost bin]# ls README.txt zkCli.cmd zkEnv.cmd zkServer.cmd zkServer.sh zkSnapShotToolkit.sh zkTxnLogToolkit.sh zkCleanup.sh zkCli.sh zkEnv.sh zkServer-initialize.sh zkSnapShotToolkit.cmd zkTxnLogToolkit.cmd ~~~ ~~~powershell [root@localhost bin]# ./zkCli.sh -server 192.168.10.142:32181 ~~~ ~~~powershell [zk: 192.168.10.142:32181(CONNECTED) 0] create /key200 values200 Created /key200 [zk: 192.168.10.142:32181(CONNECTED) 1] get /key200 values200 ~~~ ================================================ FILE: docs/cloud/kubernetes.md ================================================ # kubeadm极速部署Kubernetes 1.24版本集群 # 一、Kubernetes 1.24版本发布及重磅改动 ## 1.1 Kubernetes 1.24 发布 2022 年 5 月 3 日，Kubernetes 1.24 正式发布，在新版本中，我们看到 Kubernetes 作为容器编排的事实标准，正愈发变得成熟，**有 12 项功能都更新到了稳定版本**，同时引入了很多实用的功能，例如 **StatefulSets 支持批量滚动更新**，**NetworkPolicy 新增 NetworkPolicyStatus 字段方便进行故障排查等** ## 1.2 Kubernetes 1.24 重磅改动 Kubernetes 正式移除对 Dockershim 的支持，讨论很久的 “弃用 Dockershim” 也终于在这个版本画上了句号。 ![image-20220508094844868](../img/kubernetes/image-20220508094844868.png) ![image-20220508094933949](../img/kubernetes/image-20220508094933949.png) ![image-20220507134711296](../img/kubernetes/image-20220507134711296.png) # 二、Kubernetes 1.24版本集群部署 ## 2.1 Kubernetes 1.24版本集群部署环境准备 ### 2.1.1 主机操作系统说明 | 序号 | 操作系统及版本 | 备注 | | :--: | :------------: | :--: | | 1 | CentOS7u9 | | ### 2.1.2 主机硬件配置说明 | 需求 | CPU | 内存 | 硬盘 | 角色 | 主机名 | | ---- | ---- | ---- | ----- | ------------ | ------------ | | 值 | 4C | 8G | 100GB | master | k8s-master01 | | 值 | 4C | 8G | 100GB | worker(node) | k8s-worker01 | | 值 | 4C | 8G | 100GB | worker(node) | k8s-worker02 | ### 2.1.3 主机配置 #### 2.1.3.1 主机名配置 由于本次使用3台主机完成kubernetes集群部署，其中1台为master节点,名称为k8s-master01;其中2台为worker节点，名称分别为：k8s-worker01及k8s-worker02 ~~~powershell master节点 # hostnamectl set-hostname k8s-master01 ~~~ ~~~powershell worker01节点 # hostnamectl set-hostname k8s-worker01 ~~~ ~~~powershell worker02节点 # hostnamectl set-hostname k8s-worker02 ~~~ #### 2.1.3.2 主机IP地址配置 ~~~powershell k8s-master节点IP地址为：192.168.10.200/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.200" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell k8s-worker1节点IP地址为：192.168.10.201/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.201" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell k8s-worker2节点IP地址为：192.168.10.202/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.202" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ #### 2.1.3.3 主机名与IP地址解析 > 所有集群主机均需要进行配置。 ~~~powershell # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.200 k8s-master01 192.168.10.201 k8s-worker01 192.168.10.202 k8s-worker02 ~~~ #### 2.1.3.4 防火墙配置 > 所有主机均需要操作。 ~~~powershell 关闭现有防火墙firewalld # systemctl disable firewalld # systemctl stop firewalld # firewall-cmd --state not running ~~~ #### 2.1.3.5 SELINUX配置 > 所有主机均需要操作。修改SELinux配置需要重启操作系统。 ~~~powershell # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ #### 2.1.3.6 时间同步配置 >所有主机均需要操作。最小化安装系统需要安装ntpdate软件。 ~~~powershell # crontab -l 0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com ~~~ #### 2.1.3.7 升级操作系统内核 > 所有主机均需要操作。 ~~~powershell 导入elrepo gpg key # rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell 安装elrepo YUM源仓库 # yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell 安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 # yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell 设置grub2默认引导为0 # grub2-set-default 0 ~~~ ~~~powershell 重新生成grub2引导文件 # grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ~~~powershell 更新后，需要重启，使用升级的内核生效。 # reboot ~~~ ~~~powershell 重启后，需要验证内核是否为更新对应的版本 # uname -r ~~~ #### 2.1.3.8 配置内核转发及网桥过滤 >所有主机均需要操作。 ~~~powershell 添加网桥过滤及内核转发配置文件 # cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ ~~~powershell 加载br_netfilter模块 # modprobe br_netfilter ~~~ ~~~powershell 查看是否加载 # lsmod | grep br_netfilter br_netfilter 22256 0 bridge 151336 1 br_netfilter ~~~ ~~~powershell 加载网桥过滤及内核转发配置文件 # sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ #### 2.1.3.9 安装ipset及ipvsadm > 所有主机均需要操作。 ~~~powershell 安装ipset及ipvsadm # yum -y install ipset ipvsadm ~~~ ~~~powershell 配置ipvsadm模块加载方式 添加需要加载的模块 # cat > /etc/sysconfig/modules/ipvs.modules < 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a ~~~powershell 永远关闭swap分区，需要重启操作系统 # cat /etc/fstab ...... # /dev/mapper/centos-swap swap swap defaults 0 0 在上一行中行首添加# ~~~ ## 2.2 Docker准备 ### 2.2.1 Docker安装YUM源准备 >使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ### 2.2.2 Docker安装 ~~~powershell # yum -y install docker-ce ~~~ ### 2.2.3 启动Docker服务 ~~~powershell # systemctl enable --now docker ~~~ ### 2.2.4 修改cgroup方式 >/etc/docker/daemon.json 默认没有此文件，需要单独创建 ~~~powershell 在/etc/docker/daemon.json添加如下内容 # cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } ~~~ ~~~powershell # systemctl restart docker ~~~ ### 2.2.5 cri-dockerd安装 #### 2.2.5.1 golang环境准备 > 下载链接地址：https://golang.google.cn/dl/ ~~~powershell 获取golang安装包 # wget https://golang.google.cn/dl/go1.16.10.linux-amd64.tar.gz ~~~ ~~~powershell 解压golang至指定目录 # tar -xzf go1.16.10.linux-amd64.tar.gz -C /usr/local ~~~ ~~~powershell 添加环境变量 # cat /etc/profile ...... export GOROOT=/usr/local/go export GOPATH=$HOME/go export PATH=$PATH:$GOROOT/bin:$GOPATH/bin ~~~ ~~~powershell 加载/etc/profile文件 # source /etc/profile ~~~ ~~~powershell 验证golang是否安装完成 # go version 输出 go version go1.16.10 linux/amd64 ~~~ ~~~powershell 创建gopath目录 # mkdir -p ~/go/bin ~/go/src ~/go/pkg ~~~ #### 2.2.5.2 构建并安装cri-dockerd ![image-20220507120653090](../img/kubernetes/image-20220507120653090.png) ![image-20220507120725815](../img/kubernetes/image-20220507120725815.png) ![image-20220507120808122](../img/kubernetes/image-20220507120808122.png) ![image-20220507120849669](../img/kubernetes/image-20220507120849669.png) ~~~powershell 克隆cri-dockerd源码 # git clone https://github.com/Mirantis/cri-dockerd.git ~~~ ~~~powershell 查看克隆下来的目录 # ls cri-dockerd ~~~ ~~~powershell # cd cri-dockerd ~~~ ~~~powershell 创建bin目录并构建cri-dockerd二进制文件 # go build -o ./cri-dockerd ~~~ ~~~powershell 创建/usr/local/bin,默认存在时，可不用创建 # mkdir -p /usr/local/bin 安装cri-dockerd # install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd 复制服务管理文件至/etc/systemd/system目录中 # cp -a packaging/systemd/* /etc/systemd/system 指定cri-dockerd运行位置 #sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service 启动服务 # systemctl daemon-reload # systemctl enable cri-docker.service # systemctl enable --now cri-docker.socket ~~~ ## 2.3 kubernetes 1.24.0 集群部署 ### 2.3.1 集群软件及版本说明 | | kubeadm | kubelet | kubectl | | -------- | ---------------------- | --------------------------------------------- | ---------------------- | | 版本 | 1.24.0 | 1.24.0 | 1.24.0 | | 安装位置 | 集群所有主机 | 集群所有主机 | 集群所有主机 | | 作用 | 初始化集群、管理集群等 | 用于接收api-server指令，对pod生命周期进行管理 | 集群应用命令行管理工具 | ### 2.3.2 kubernetes YUM源准备 #### 2.3.2.1 谷歌YUM源 ~~~powershell [kubernetes] vi /etc/yum.repos.d/kubernetes.repo name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg ~~~ #### 2.3.2.2 阿里云YUM源 ~~~powershell [kubernetes] vi /etc/yum.repos.d/kubernetes.repo name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ~~~ ### 2.3.3 集群软件安装 > 所有节点均可安装 ~~~powershell 安装 # yum -y install kubeadm kubelet kubectl ~~~ ### 2.3.4 配置kubelet >为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ~~~powershell # vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ~~~ ~~~powershell 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 # systemctl enable kubelet ~~~ ### 2.3.5 使用阿里云拉取所需镜像 ~~~powershell kubeadm config images pull --image-repository=registry.aliyuncs.com/google_containers --cri-socket unix:///run/cri-dockerd.sock ~~~ ### 2.3.6 集群初始化 ~~~powershell [root@k8s-master01 ~] kubeadm init --kubernetes-version=v1.26.2 --pod-network-cidr=10.224.0.0/16 --apiserver-advertise-address=192.168.10.200 --cri-socket unix:///run/cri-dockerd.sock --image-repository registry.aliyuncs.com/google_containers ~~~ ~~~powershell 如果不添加--cri-socket选项，则会报错，内容如下： Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock To see the stack trace of this error execute with --v=5 or higher ~~~ ~~~powershell 初始化过程输出 [init] Using Kubernetes version: v1.24.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.200 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.200 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 13.006785 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: 8x4o2u.hslo8xzwwlrncr8s [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.200:6443 --token 8x4o2u.hslo8xzwwlrncr8s \ --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 ~~~ ### 2.3.7 集群应用客户端管理集群文件准备 ~~~powershell [root@k8s-master01 ~]# mkdir -p $HOME/.kube [root@k8s-master01 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master01 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@k8s-master01 ~]# ls /root/.kube/ config ~~~ ~~~powershell [root@k8s-master01 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf ~~~ ### 2.3.8 集群网络准备 > 使用calico部署集群网络 > > 安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico #### 2.3.8.1 calico安装 ![image-20220119141547207](../img/kubernetes/image-20220119141547207.png) ![image-20220119141645676](../img/kubernetes/image-20220119141645676.png) ![image-20220119141734347](../img/kubernetes/image-20220119141734347.png) ![image-20220119141830625](../img/kubernetes/image-20220119141830625.png) ~~~powershell 下载operator资源清单文件 [root@k8s-master01 ~]# wget https://docs.projectcalico.org/manifests/tigera-operator.yaml ~~~ ~~~powershell 应用资源清单文件，创建operator [root@k8s-master01 ~]# kubectl apply -f tigera-operator.yaml ~~~ ~~~powershell 通过自定义资源方式安装 [root@k8s-master01 ~]# wget https://docs.projectcalico.org/manifests/custom-resources.yaml ~~~ ~~~powershell 修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 [root@k8s-master01 ~]# vim custom-resources.yaml ...... 11 ipPools: 12 - blockSize: 26 13 cidr: 10.224.0.0/16 14 encapsulation: VXLANCrossSubnet ...... ~~~ ~~~powershell 应用资源清单文件 [root@k8s-master01 ~]# kubectl apply -f custom-resources.yaml ~~~ ~~~powershell 监视calico-sysem命名空间中pod运行情况 [root@k8s-master01 ~]# watch kubectl get pods -n calico-system ~~~ >Wait until each pod has the `STATUS` of `Running`. ~~~powershell 删除 master 上的 taint [root@k8s-master01 ~]# kubectl taint nodes --all node-role.kubernetes.io/master- ~~~ ~~~powershell 已经全部运行 [root@k8s-master01 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-666bb9949-dzp68 1/1 Running 0 11m calico-node-jhcf4 1/1 Running 4 11m calico-typha-68b96d8d9c-7qfq7 1/1 Running 2 11m ~~~ ~~~powershell 查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 [root@k8s-master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d4b75cb6d-js5pl 1/1 Running 0 12h coredns-6d4b75cb6d-zm8pc 1/1 Running 0 12h etcd-k8s-master01 1/1 Running 0 12h kube-apiserver-k8s-master01 1/1 Running 0 12h kube-controller-manager-k8s-master01 1/1 Running 0 12h kube-proxy-7nhr7 1/1 Running 0 12h kube-proxy-fv4kr 1/1 Running 0 12h kube-proxy-vv5vg 1/1 Running 0 12h kube-scheduler-k8s-master01 1/1 Running 0 12h ~~~ #### 2.3.8.2 calico客户端安装 ![image-20220119144207789](../img/kubernetes/image-20220119144207789.png) ![image-20220119144446449](../img/kubernetes/image-20220119144446449.png) ~~~powershell 下载二进制文件 # curl -L https://github.com/projectcalico/calico/releases/download/v3.21.4/calicoctl-linux-amd64 -o calicoctl ~~~ ~~~powershell 安装calicoctl # mv calicoctl /usr/bin/ 为calicoctl添加可执行权限 # chmod +x /usr/bin/calicoctl 查看添加权限后文件 # ls /usr/bin/calicoctl /usr/bin/calicoctl 查看calicoctl版本 # calicoctl version Client Version: v3.21.4 Git commit: 220d04c94 Cluster Version: v3.21.4 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm ~~~ ~~~powershell 通过~/.kube/config连接kubernetes集群，查看已运行节点 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME k8s-master01 ~~~ ### 2.3.9 集群工作节点添加 > 因容器镜像下载较慢，可能会导致报错，主要错误为没有准备好cni（集群网络插件），如有网络，请耐心等待即可。 ~~~powershell [root@k8s-worker01 ~]# kubeadm join 192.168.10.200:6443 --token 8x4o2u.hslo8xzwwlrncr8s \ --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 --cri-socket unix:///var/run/cri-dockerd.sock ~~~ ~~~powershell [root@k8s-worker02 ~]# kubeadm join 192.168.10.200:6443 --token 8x4o2u.hslo8xzwwlrncr8s \ --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 --cri-socket unix:///var/run/cri-dockerd.sock ~~~ ~~~powershell 在master节点上操作，查看网络节点是否添加 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME k8s-master01 k8s-worker01 k8s-worker02 ~~~ ### 2.3.10 验证集群可用性 ~~~powershell 查看所有的节点 [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION# kubeadm极速部署Kubernetes 1.24版本集群 ~~~ # 一、Kubernetes 1.24版本发布及重磅改动 ## 1.1 Kubernetes 1.24 发布 2022 年 5 月 3 日，Kubernetes 1.24 正式发布，在新版本中，我们看到 Kubernetes 作为容器编排的事实标准，正愈发变得成熟，**有 12 项功能都更新到了稳定版本**，同时引入了很多实用的功能，例如 **StatefulSets 支持批量滚动更新**，**NetworkPolicy 新增 NetworkPolicyStatus 字段方便进行故障排查等** ## 1.2 Kubernetes 1.24 重磅改动 Kubernetes 正式移除对 Dockershim 的支持，讨论很久的 “弃用 Dockershim” 也终于在这个版本画上了句号。 ![image-20220508094844868](../img/kubernetes/image-20220508094844868.png) ![image-20220508094933949](../img/kubernetes/image-20220508094933949.png) ![image-20220507134711296](../img/kubernetes/image-20220507134711296.png) # 二、Kubernetes 1.24版本集群部署 ## 2.1 Kubernetes 1.24版本集群部署环境准备 ### 2.1.1 主机操作系统说明 | 序号 | 操作系统及版本 | 备注 | | :--: | :------------: | :--: | | 1 | CentOS7u9 | | ### 2.1.2 主机硬件配置说明 | 需求 | CPU | 内存 | 硬盘 | 角色 | 主机名 | | ---- | ---- | ---- | ----- | ------------ | ------------ | | 值 | 4C | 8G | 100GB | master | k8s-master01 | | 值 | 4C | 8G | 100GB | worker(node) | k8s-worker01 | | 值 | 4C | 8G | 100GB | worker(node) | k8s-worker02 | ### 2.1.3 主机配置 #### 2.1.3.1 主机名配置 由于本次使用3台主机完成kubernetes集群部署，其中1台为master节点,名称为k8s-master01;其中2台为worker节点，名称分别为：k8s-worker01及k8s-worker02 ~~~powershell master节点 # hostnamectl set-hostname k8s-master01 ~~~ ~~~powershell worker01节点 # hostnamectl set-hostname k8s-worker01 ~~~ ~~~powershell worker02节点 # hostnamectl set-hostname k8s-worker02 ~~~ #### 2.1.3.2 主机IP地址配置 ~~~powershell k8s-master节点IP地址为：192.168.10.200/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.200" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell k8s-worker1节点IP地址为：192.168.10.201/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.201" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ~~~powershell k8s-worker2节点IP地址为：192.168.10.202/24 # vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.202" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ #### 2.1.3.3 主机名与IP地址解析 > 所有集群主机均需要进行配置。 ~~~powershell # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.200 k8s-master01 192.168.10.201 k8s-worker01 192.168.10.202 k8s-worker02 ~~~ #### 2.1.3.4 防火墙配置 > 所有主机均需要操作。 ~~~powershell 关闭现有防火墙firewalld # systemctl disable firewalld # systemctl stop firewalld # firewall-cmd --state not running ~~~ #### 2.1.3.5 SELINUX配置 > 所有主机均需要操作。修改SELinux配置需要重启操作系统。 ~~~powershell # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ #### 2.1.3.6 时间同步配置 >所有主机均需要操作。最小化安装系统需要安装ntpdate软件。 ~~~powershell # crontab -l 0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com ~~~ #### 2.1.3.7 升级操作系统内核 > 所有主机均需要操作。 ~~~powershell 导入elrepo gpg key # rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org ~~~ ~~~powershell 安装elrepo YUM源仓库 # yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm ~~~ ~~~powershell 安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本 # yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64 ~~~ ~~~powershell 设置grub2默认引导为0 # grub2-set-default 0 ~~~ ~~~powershell 重新生成grub2引导文件 # grub2-mkconfig -o /boot/grub2/grub.cfg ~~~ ~~~powershell 更新后，需要重启，使用升级的内核生效。 # reboot ~~~ ~~~powershell 重启后，需要验证内核是否为更新对应的版本 # uname -r ~~~ #### 2.1.3.8 配置内核转发及网桥过滤 >所有主机均需要操作。 ~~~powershell 添加网桥过滤及内核转发配置文件 # cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ ~~~powershell 加载br_netfilter模块 # modprobe br_netfilter ~~~ ~~~powershell 查看是否加载 # lsmod | grep br_netfilter br_netfilter 22256 0 bridge 151336 1 br_netfilter ~~~ ~~~powershell 加载网桥过滤及内核转发配置文件 # sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 ~~~ #### 2.1.3.9 安装ipset及ipvsadm > 所有主机均需要操作。 ~~~powershell 安装ipset及ipvsadm # yum -y install ipset ipvsadm ~~~ ~~~powershell 配置ipvsadm模块加载方式 添加需要加载的模块 # cat > /etc/sysconfig/modules/ipvs.modules < 修改完成后需要重启操作系统，如不重启，可临时关闭，命令为swapoff -a ~~~powershell 永远关闭swap分区，需要重启操作系统 # cat /etc/fstab ...... # /dev/mapper/centos-swap swap swap defaults 0 0 在上一行中行首添加# ~~~ ## 2.2 Docker准备 ### 2.2.1 Docker安装YUM源准备 >使用阿里云开源软件镜像站。 ~~~powershell # wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo ~~~ ### 2.2.2 Docker安装 ~~~powershell # yum -y install docker-ce ~~~ ### 2.2.3 启动Docker服务 ~~~powershell # systemctl enable --now docker ~~~ ### 2.2.4 修改cgroup方式 >/etc/docker/daemon.json 默认没有此文件，需要单独创建 ~~~powershell 在/etc/docker/daemon.json添加如下内容 # cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"] } ~~~ ~~~powershell # systemctl restart docker ~~~ ### 2.2.5 cri-dockerd安装 #### 2.2.5.1 golang环境准备 > 下载链接地址：https://golang.google.cn/dl/ ~~~powershell 获取golang安装包 # wget https://golang.google.cn/dl/go1.16.10.linux-amd64.tar.gz ~~~ ~~~powershell 解压golang至指定目录 # tar -xzf go1.16.10.linux-amd64.tar.gz -C /usr/local ~~~ ~~~powershell 添加环境变量 # cat /etc/profile ...... export GOROOT=/usr/local/go export GOPATH=$HOME/go export PATH=$PATH:$GOROOT/bin:$GOPATH/bin ~~~ ~~~powershell 加载/etc/profile文件 # source /etc/profile ~~~ ~~~powershell 验证golang是否安装完成 # go version 输出 go version go1.16.10 linux/amd64 ~~~ ~~~powershell 创建gopath目录 # mkdir -p ~/go/bin ~/go/src ~/go/pkg ~~~ #### 2.2.5.2 构建并安装cri-dockerd ![image-20220507120653090](../img/kubernetes/image-20220507120653090.png) ![image-20220507120725815](../img/kubernetes/image-20220507120725815.png) ![image-20220507120808122](../img/kubernetes/image-20220507120808122.png) ![image-20220507120849669](../img/kubernetes/image-20220507120849669.png) ~~~powershell 克隆cri-dockerd源码 # git clone https://github.com/Mirantis/cri-dockerd.git ~~~ ~~~powershell 查看克隆下来的目录 # ls cri-dockerd ~~~ ~~~powershell 查看目录中内容 # ls cri-dockerd/ LICENSE Makefile packaging README.md src VERSION ~~~ ~~~powershell # cd cri-dockerd ~~~ ~~~powershell 创建bin目录并构建cri-dockerd二进制文件 # mkdir bin # cd src && go get && go build -o ../bin/cri-dockerd ~~~ ~~~powershell 创建/usr/local/bin,默认存在时，可不用创建 # mkdir -p /usr/local/bin 安装cri-dockerd # install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd 复制服务管理文件至/etc/systemd/system目录中 # cp -a packaging/systemd/* /etc/systemd/system 指定cri-dockerd运行位置 #sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service 启动服务 # systemctl daemon-reload # systemctl enable cri-docker.service # systemctl enable --now cri-docker.socket ~~~ ## 2.3 kubernetes 1.24.0 集群部署 ### 2.3.1 集群软件及版本说明 | | kubeadm | kubelet | kubectl | | -------- | ---------------------- | --------------------------------------------- | ---------------------- | | 版本 | 1.24.0 | 1.24.0 | 1.24.0 | | 安装位置 | 集群所有主机 | 集群所有主机 | 集群所有主机 | | 作用 | 初始化集群、管理集群等 | 用于接收api-server指令，对pod生命周期进行管理 | 集群应用命令行管理工具 | ### 2.3.2 kubernetes YUM源准备 #### 2.3.2.1 谷歌YUM源 ~~~powershell [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg ~~~ #### 2.3.2.2 阿里云YUM源 ~~~powershell [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg ~~~ ### 2.3.3 集群软件安装 > 所有节点均可安装 ~~~powershell 安装 # yum -y install kubeadm kubelet kubectl ~~~ ### 2.3.4 配置kubelet >为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。 ~~~powershell # vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" ~~~ ~~~powershell 设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动 # systemctl enable kubelet ~~~ ### 2.3.5 集群镜像准备 > 可使用VPN实现下载。 ~~~powershell # kubeadm config images list --kubernetes-version=v1.24.0 k8s.gcr.io/kube-apiserver:v1.24.0 k8s.gcr.io/kube-controller-manager:v1.24.0 k8s.gcr.io/kube-scheduler:v1.24.0 k8s.gcr.io/kube-proxy:v1.24.0 k8s.gcr.io/pause:3.7 k8s.gcr.io/etcd:3.5.3-0 k8s.gcr.io/coredns/coredns:v1.8.6 ~~~ ~~~powershell # cat image_download.sh #!/bin/bash images_list=' k8s.gcr.io/kube-apiserver:v1.24.0 k8s.gcr.io/kube-controller-manager:v1.24.0 k8s.gcr.io/kube-scheduler:v1.24.0 k8s.gcr.io/kube-proxy:v1.24.0 k8s.gcr.io/pause:3.7 k8s.gcr.io/etcd:3.5.3-0 k8s.gcr.io/coredns/coredns:v1.8.6' for i in $images_list do docker pull $i done docker save -o k8s-1-24-0.tar $images_list ~~~ ### 2.3.6 集群初始化 ~~~powershell [root@k8s-master01 ~]# kubeadm init --kubernetes-version=v1.24.0 --pod-network-cidr=10.224.0.0/16 --apiserver-advertise-address=192.168.10.200 --cri-socket unix:///var/run/cri-dockerd.sock ~~~ ~~~powershell 如果不添加--cri-socket选项，则会报错，内容如下： Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock To see the stack trace of this error execute with --v=5 or higher ~~~ ~~~powershell 初始化过程输出 [init] Using Kubernetes version: v1.24.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.200 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.10.200 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 13.006785 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: 8x4o2u.hslo8xzwwlrncr8s [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.10.200:6443 --token 8x4o2u.hslo8xzwwlrncr8s \ --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 ~~~ ### 2.3.7 集群应用客户端管理集群文件准备 ~~~powershell [root@k8s-master01 ~]# mkdir -p $HOME/.kube [root@k8s-master01 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@k8s-master01 ~]# chown $(id -u):$(id -g) $HOME/.kube/config [root@k8s-master01 ~]# ls /root/.kube/ config ~~~ ~~~powershell [root@k8s-master01 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf ~~~ ### 2.3.8 集群网络准备 > 使用calico部署集群网络 > > 安装参考网址：https://projectcalico.docs.tigera.io/about/about-calico #### 2.3.8.1 calico安装 ![image-20220119141547207](../img/kubernetes/image-20220119141547207.png) ![image-20220119141645676](../img/kubernetes/image-20220119141645676.png) ![image-20220119141734347](../img/kubernetes/image-20220119141734347.png) ![image-20220119141830625](../img/kubernetes/image-20220119141830625.png) ~~~powershell 下载operator资源清单文件 [root@k8s-master01 ~]# wget https://docs.projectcalico.org/manifests/tigera-operator.yaml ~~~ ~~~powershell 应用资源清单文件，创建operator [root@k8s-master01 ~]# kubectl apply -f tigera-operator.yaml ~~~ ~~~powershell 通过自定义资源方式安装 [root@k8s-master01 ~]# wget https://docs.projectcalico.org/manifests/custom-resources.yaml ~~~ ~~~powershell 修改文件第13行，修改为使用kubeadm init ----pod-network-cidr对应的IP地址段 [root@k8s-master01 ~]# vim custom-resources.yaml ...... 11 ipPools: 12 - blockSize: 26 13 cidr: 10.224.0.0/16 14 encapsulation: VXLANCrossSubnet ...... ~~~ ~~~powershell 应用资源清单文件 [root@k8s-master01 ~]# kubectl apply -f custom-resources.yaml ~~~ ~~~powershell 监视calico-sysem命名空间中pod运行情况 [root@k8s-master01 ~]# watch kubectl get pods -n calico-system ~~~ >Wait until each pod has the `STATUS` of `Running`. ~~~powershell 删除 master 上的 taint [root@k8s-master01 ~]# kubectl taint nodes --all node-role.kubernetes.io/master- ~~~ ~~~powershell 已经全部运行 [root@k8s-master01 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-666bb9949-dzp68 1/1 Running 0 11m calico-node-jhcf4 1/1 Running 4 11m calico-typha-68b96d8d9c-7qfq7 1/1 Running 2 11m ~~~ ~~~powershell 查看kube-system命名空间中coredns状态，处于Running状态表明联网成功。 [root@k8s-master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d4b75cb6d-js5pl 1/1 Running 0 12h coredns-6d4b75cb6d-zm8pc 1/1 Running 0 12h etcd-k8s-master01 1/1 Running 0 12h kube-apiserver-k8s-master01 1/1 Running 0 12h kube-controller-manager-k8s-master01 1/1 Running 0 12h kube-proxy-7nhr7 1/1 Running 0 12h kube-proxy-fv4kr 1/1 Running 0 12h kube-proxy-vv5vg 1/1 Running 0 12h kube-scheduler-k8s-master01 1/1 Running 0 12h ~~~ #### 2.3.8.2 calico客户端安装 ![image-20220119144207789](../img/kubernetes/image-20220119144207789.png) ![image-20220119144446449](../img/kubernetes/image-20220119144446449.png) ~~~powershell 下载二进制文件 # curl -L https://github.com/projectcalico/calico/releases/download/v3.21.4/calicoctl-linux-amd64 -o calicoctl ~~~ ~~~powershell 安装calicoctl # mv calicoctl /usr/bin/ 为calicoctl添加可执行权限 # chmod +x /usr/bin/calicoctl 查看添加权限后文件 # ls /usr/bin/calicoctl /usr/bin/calicoctl 查看calicoctl版本 # calicoctl version Client Version: v3.21.4 Git commit: 220d04c94 Cluster Version: v3.21.4 Cluster Type: typha,kdd,k8s,operator,bgp,kubeadm ~~~ ~~~powershell 通过~/.kube/config连接kubernetes集群，查看已运行节点 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME k8s-master01 ~~~ ### 2.3.9 集群工作节点添加 > 因容器镜像下载较慢，可能会导致报错，主要错误为没有准备好cni（集群网络插件），如有网络，请耐心等待即可。 ~~~powershell [root@k8s-worker01 ~]# kubeadm join 192.168.10.200:6443 --token 8x4o2u.hslo8xzwwlrncr8s \ --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 --cri-socket unix:///var/run/cri-dockerd.sock ~~~ ~~~powershell [root@k8s-worker02 ~]# kubeadm join 192.168.10.200:6443 --token 8x4o2u.hslo8xzwwlrncr8s \ --discovery-token-ca-cert-hash sha256:7323a8b0658fc33d89e627f078f6eb16ac94394f9a91b3335dd3ce73a3f313a0 --cri-socket unix:///var/run/cri-dockerd.sock ~~~ ~~~powershell 在master节点上操作，查看网络节点是否添加 # DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes NAME k8s-master01 k8s-worker01 k8s-worker02 ~~~ ### 2.3.10 验证集群可用性 ~~~powershell 查看所有的节点 [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 12h v1.24.0 k8s-worker01 Ready 12h v1.24.0 k8s-worker02 Ready 12h v1.24.0 ~~~ ~~~powershell 查看集群健康情况 [root@k8s-master01 ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true","reason":""} ~~~ ~~~powershell 查看kubernetes集群pod运行情况 [root@k8s-master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d4b75cb6d-js5pl 1/1 Running 0 12h coredns-6d4b75cb6d-zm8pc 1/1 Running 0 12h etcd-k8s-master01 1/1 Running 0 12h kube-apiserver-k8s-master01 1/1 Running 0 12h kube-controller-manager-k8s-master01 1/1 Running 0 12h kube-proxy-7nhr7 1/1 Running 0 12h kube-proxy-fv4kr 1/1 Running 0 12h kube-proxy-vv5vg 1/1 Running 0 12h kube-scheduler-k8s-master01 1/1 Running 0 12h ~~~ ~~~powershell 再次查看calico-system命名空间中pod运行情况。 [root@k8s-master01 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5b544d9b48-xgfnk 1/1 Running 0 12h calico-node-7clf4 1/1 Running 0 12h calico-node-cjwns 1/1 Running 0 12h calico-node-hhr4n 1/1 Running 0 12h calico-typha-6cb6976b97-5lnpk 1/1 Running 0 12h calico-typha-6cb6976b97-9w9s8 1/1 Running 0 12h ~~~ k8s-master01 Ready control-plane 12h v1.24.0 k8s-worker01 Ready 12h v1.24.0 k8s-worker02 Ready 12h v1.24.0 ~~~ ~~~powershell 查看集群健康情况 [root@k8s-master01 ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true","reason":""} ~~~ ~~~powershell 查看kubernetes集群pod运行情况 [root@k8s-master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d4b75cb6d-js5pl 1/1 Running 0 12h coredns-6d4b75cb6d-zm8pc 1/1 Running 0 12h etcd-k8s-master01 1/1 Running 0 12h kube-apiserver-k8s-master01 1/1 Running 0 12h kube-controller-manager-k8s-master01 1/1 Running 0 12h kube-proxy-7nhr7 1/1 Running 0 12h kube-proxy-fv4kr 1/1 Running 0 12h kube-proxy-vv5vg 1/1 Running 0 12h kube-scheduler-k8s-master01 1/1 Running 0 12h ~~~ ~~~powershell 再次查看calico-system命名空间中pod运行情况。 [root@k8s-master01 ~]# kubectl get pods -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5b544d9b48-xgfnk 1/1 Running 0 12h calico-node-7clf4 1/1 Running 0 12h calico-node-cjwns 1/1 Running 0 12h calico-node-hhr4n 1/1 Running 0 12h calico-typha-6cb6976b97-5lnpk 1/1 Running 0 12h calico-typha-6cb6976b97-9w9s8 1/1 Running 0 12h ~~~ ================================================ FILE: docs/cloud/native.md ================================================ # 云原生（CloudNative） # 一、云原生定义 ## 1.1 理解云原生 ### 1.1.1 从字面理解 云原生从字面意思上来看可以分成： - 云(Cloud) - 原生(Native) - 云原生（CloudNative） ### 1.1.2 由CNCF定义 > 云原生计算基金会(Cloud Native Computing Foundation,CNCF)成立于2015年12月11日，由谷歌与Linux基金会联合创办，成立这个非盈利组织的目的是为了推广、孵化和标准化云原生相关的技术。 云原生计算基金会(Cloud Native Computing Foundation,CNCF)认为：云原生是一类技术的统称，通过云原生技术，我们可以构建出更易于弹性扩展的应用程序，其包含容器、服务网格、微服务、不可变基础设施和声明式API等相关技术，这些技术能够构建容错性好、易于管理和便于观察的松耦合系统，结合可靠的自动化手段，相关工程师能够轻松对系统作出频繁和可预测的重大变更。 ## 1.2 云与原生之间的关系 - 云是指云计算技术或云计算平台 - 原生就是土生土长 - 云原生表示业务应用原生化，例如:Kubernetes使用声明式部署业务应用，所以众多的产品都在使用声明式方式部署应用 - 使用云原生的好处： - 业务应用被设计为在云上以最佳方式运行 - 充分发挥云的优势，例如：资源的无限化、扩缩容便利化等特点 ## 1.3 云原生概念由来及最佳实践三个层面 ### 1.3.1 概念由来 - Pivotal公司的Matt Stine于2013年首次提出云原生(CloudNative)的概念. - 2015年云原生计算基金会(CNCF)成立，最初把云原生定义为包括：容器化封装+自动化管理+面向微服务。 - 到了2018年，CNCF又更新了云原生的定义，把服务网格(Service Mesh)和声明式API给加了进来。 - 云原生技术有利于各组织在公有云、私有云和混合云等新型动态环境中，构建和运行可弹性扩展的应用。 - 这些技术能够构建容错性好、易于管理和便于观察的松耦合系统。 - 结合可靠的自动化手段，云原生技术使工程师能够轻松地对系统作出频繁和可预测的重大变更。 ### 1.3.2 最佳实践三个层面 1）服务编排要实现计算资源弹性化 2） 服务构建和部署要实现高度自动化 3）事件驱动基础设施标准化 ## 1.4 云原生代表技术 ![image-20220110110617840](../img/cloudnative/image-20220110110617840.png) ### 1.4.1 微服务 微服务的定义：原有单体应用拆分为多个独立自治的组件，每个组件都可以独立设计、开发、测试、部署和运维，这个组件可以单独提供对外服务，我们称之为微服务。 例如：早期的LNMT WEB部署架构，使用微服务后，每一个组件都可以独立自治、运行、扩容、缩容等 **各组件之间可通过轻量的Restful风格接口进行交互和协同** ### 1.4.2 容器化 #### 1.4.2.1 Docker容器 Docker容器，容器属于it基础设施层概念，是比虚拟机更轻量化的隔离工具，是微服务最佳载体。 ![image-20220110105253169](../img/cloudnative/image-20220110105253169.png) ![image-20220110105313084](../img/cloudnative/image-20220110105313084.png) #### 1.4.2.2 Kubernetes资源调度与容器编排 使用kubernetes的资源调度与容器编排，可以实现Docker容器更优管理，进一步实现其PaaS层能力。 ### 1.4.3 服务网格 服务网格存在的目的，就是去中心化的服务治理框架。 以往需要对微服务或对api接口去做治理和管控，一般会用类似于ESB服务总线或 API网关，将API接口注册和接入到API网关，由于API网关本身是一个中心化的架构，所以所有的请求流量都可以通过API网关，由API网关实现对流量拦截，同时对拦截以后的流量进行安全，日志，限流熔断，链路监控等各种管控治理，去中心化以后就没有这种集中化的流量管控点了，所以对流量的拦截就从ESB服务总线或API网关下沉到各个微服务中去了，这就是为什么我们需要在微服务端增加一个代理包的原因，通过这个代理包来做流量的拦截，同时实现对流量的管控，当前在微服务网格中也是用同样的思路来对服务进行治理的。例如：istio服务治理，它会在微服务应用中添加一个边车容器（Envoy）来实现流量的拦截和管控。这个属于微服务服务网格治理的核心技术。 ![image-20220110105634552](../img/cloudnative/image-20220110105634552.png) 去中心化的服务治理依然有一个控制中心，而控制中心依然是中心化的，但实际的控制流和接口数据访问的消息流是实现分离的，控制中心仅处理服务注册发现，实际的接口调用、服务访问是不通过控制中心的，即使控制中心出现问题，例如控制中心服务不可用等，也不会影响实际服务接口调用。 ### 1.4.4 不可变基础设施 ![image-20220110113843903](../img/cloudnative/image-20220110113843903.png) 传统开发过程中，做一个软件程序的部署，当它部署到一个生产环境，如果我们要做变更，不管是程序的变更还是配置的变更，都需要在原来的生产环境上面重新部署或对某一个配置直接进行修改，但是在云原生应用中，任何一个应用当你部署到生产环境中，形成一个容器实例以后，这个容器实例不应该再做任何变化，如果软件程序需要重新部署或修改配置时怎么办呢？可以利用基础容器镜像，重新生成一个新的容器实例，同时把旧的容器实例销毁掉，这个就是云原生技术中要求的不可变技术点。 ### 1.4.5 声明式API > 应用部署大体上分为两种执行方式：命令式和声明式。 - 命令式 - 在命令行执行命令创建容器， ~~~powershell # kubectl run -it busyboxapp --image=busyboxapp:1.28.4 ~~~ - 声明式 - 使用yaml资源清单文件 - 在yaml文件中声明要做的操作、需要的配置信息有哪些、用户期望达到的状态 ~~~powershell 创建yaml资源清单文件 # cat nginx.yaml --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: selector: matchLabels: app: nginx serviceName: "nginx" replicas: 2 template: metadata: labels: app: nginx spec: imagePullSecrets: - name: huoban-harbor terminationGracePeriodSeconds: 10 containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "managed-nfs-storage" resources: requests: storage: 1Gi 应用yaml资源清单文件 # kubectl apply -f nginx.yaml ~~~ ***IT基础设施获取声明式文件的后续操作*** 当IT基础设施获取到声明文件后，首先要解析声明式文件中声明的内容，再去后端做出相应的操作，操作完成后，把各个底层技术组件协调到应用需要的一个状态。 使用声明式API，任何对生产环境、配置都不是操作一条命令来完成的，都需要先写声明，或配置文件，这些操作都可以纳入配置管理中进行集中管理，这样有利于在生产环境出现问题时，能够快速了解前述操作，及对生产环境产生的影响，易于做版本回退、回滚等操作。 ### 1.4.6 DevOps 借助于云原生相关技术，DevOps时代才真正地到来了。 - 实现开发、运维、测试协同全作 - 构建自动化发布管道，实现代码快速部署(测试环境、预发布环境、生产环境等) - 频繁发布、快速交付、快速反馈、降低发布风险 # 二、云原生发展 我们知道计算资源应用技术一直是在不断地往前发展的，从物理机演进为虚拟机，从虚拟机再演进到容器化，由容器化再演变到我们今天所看到云原生技术，其中充满了机会与竞争，就拿容器化技术演进过程来说： 在这个发展历程中, 各家公司都出现了商业竞争： - 容器引擎之争: docker公司的docker VS CoreOS公司的rocket(简称 rkt) - 容器编排之争: Docker swarm VS Kubernetes VS Apache Mesos 在竞争合作之间寻找平衡从而导致了标准规范的诞生，而标准规范的诞生是整个云原生生态最重要的基石。 2015年6月，Docker带头成立OCI，旨在“制定并维护容器镜像格式和容器运行时的正式规范（OCI Specifications）”，其核心产出是OCI Runtime Spec（容器运行时规范）、OCI Image Spec（镜像格式规范）、OCI Distribution Spec（镜像分发规范）。所以**OCI组织解决的是容器的构建、分发和运行问题**。 一个月之后，Google带头成立了Cloud Native Computing Foundation（CNCF），旨在“构建云原生计算 :一种围绕着微服务、容器和应用动态调度的、以基础设施为中心的架构，并促进其广泛使用”。所以**CNCF组织解决的是应用管理及容器编排问题**。 # 三、CNCF云原生全景图 ## 3.1 CNCF官方网址 网址：https://www.cncf.io/ ![image-20220110132736535](../img/cloudnative/image-20220110132736535.png) ## 3.2 CNCF已毕业项目及孵化中项目 ![image-20220110133025174](../img/cloudnative/image-20220110133025174.png) ![image-20220110134328548](../img/cloudnative/image-20220110134328548.png) ![image-20220110133146979](../img/cloudnative/image-20220110133146979.png) ![image-20220110134417204](../img/cloudnative/image-20220110134417204.png) ## 3.3 CNCF云原生全景图 CNCF的项目全景图网址链接: https://landscape.cncf.io/ ![image-20220110133821857](../img/cloudnative/image-20220110133821857.png) ![image-20220110133846106](../img/cloudnative/image-20220110133846106.png) ![image-20220110134011998](../img/cloudnative/image-20220110134011998.png) ![image-20220110134037012](../img/cloudnative/image-20220110134037012.png) ## 3.4 CNCF云原生全景图分层 ### 3.4.1 云原生应用领域 云原生应用生态已覆盖到:大数据、人工智能、边缘计算、区块链等领域。 ![image-20211228163349479](../img/cloudnative/image-20211228163349479.png) ![image-20211228163330949](../img/cloudnative/image-20211228163330949.png) ### 3.4.2 云原生应用编排及管理 #### 3.4.2.1 编排与调度 ![image-20211228162902138](../img/cloudnative/image-20211228162902138.png) ![image-20211228162846749](../img/cloudnative/image-20211228162846749.png) #### 3.4.2.2 远程调用 ![image-20211228162820445](../img/cloudnative/image-20211228162820445.png) ![image-20211228162806110](../img/cloudnative/image-20211228162806110.png) #### 3.4.2.3 服务代理 ![image-20211228162740463](../img/cloudnative/image-20211228162740463.png) ![image-20211228162721662](../img/cloudnative/image-20211228162721662.png) #### 3.4.2.4 API网关 ![image-20211228162612934](../img/cloudnative/image-20211228162612934.png) ![image-20211228162536821](../img/cloudnative/image-20211228162536821.png) #### 3.4.2.5 服务网格 ![image-20211228162652358](../img/cloudnative/image-20211228162652358.png) ![image-20211228162637366](../img/cloudnative/image-20211228162637366.png) #### 3.4.2.6 服务发现 ![image-20211228164445629](../img/cloudnative/image-20211228164445629.png) ![image-20211228164429739](../img/cloudnative/image-20211228164429739.png) #### 3.4.2.7 消息和流式处理 ![image-20211228164319351](../img/cloudnative/image-20211228164319351.png) ![image-20211228164257206](../img/cloudnative/image-20211228164257206.png) #### 3.4.2.8 Serverless ![image-20211228164159596](../img/cloudnative/image-20211228164159596.png) ![image-20211228164126687](../img/cloudnative/image-20211228164126687.png) #### 3.4.2.9 CI/CD ![image-20211228163000639](../img/cloudnative/image-20211228163000639.png) ![image-20211228163024373](../img/cloudnative/image-20211228163024373.png) ![image-20211228162943111](../img/cloudnative/image-20211228162943111.png) #### 3.4.2.10 自动化配置 ![image-20211228164755110](../img/cloudnative/image-20211228164755110.png) ![image-20211228164738124](../img/cloudnative/image-20211228164738124.png) #### 3.4.2.11 数据库 ![image-20211228163550767](../img/cloudnative/image-20211228163550767.png) ![image-20211228163527606](../img/cloudnative/image-20211228163527606.png) #### 3.4.2.12 容器镜像仓库 ![image-20211228163450565](../img/cloudnative/image-20211228163450565.png) ![image-20211228163429237](../img/cloudnative/image-20211228163429237.png) #### 3.4.2.13 应用定义及镜像制作 ![image-20211228163237565](../img/cloudnative/image-20211228163237565.png) ![image-20211228163211829](../img/cloudnative/image-20211228163211829.png) #### 3.4.2.14 密钥管理 ![image-20211228163851230](../img/cloudnative/image-20211228163851230.png) ![image-20211228163823615](../img/cloudnative/image-20211228163823615.png) ### 3.4.3 云原生底层技术 #### 3.4.3.1 容器技术 ![image-20211228161918933](../img/cloudnative/image-20211228161918933.png) ![image-20211228161901341](../img/cloudnative/image-20211228161901341.png) #### 3.4.3.2 存储技术 ![image-20211228162018405](../img/cloudnative/image-20211228162018405.png) ![image-20211228162000448](../img/cloudnative/image-20211228162000448.png) #### 3.4.3.3 网络技术 ![image-20211228162136775](../img/cloudnative/image-20211228162136775.png) ![image-20211228162121504](../img/cloudnative/image-20211228162121504.png) ### 3.4.4 云原生监测分析 #### 3.4.4.1 主机状态及服务状态监控 ![image-20211228161605415](../img/cloudnative/image-20211228161605415.png) ![image-20211228161548694](../img/cloudnative/image-20211228161548694.png) #### 3.4.4.2 日志收集分析 ![image-20211228161522569](../img/cloudnative/image-20211228161522569.png) ![image-20211228161419286](../img/cloudnative/image-20211228161419286.png) #### 3.4.4.3 全链路状态跟踪 ![image-20211228161355406](../img/cloudnative/image-20211228161355406.png) ![image-20211228161326104](../img/cloudnative/image-20211228161326104.png) ### 3.4.5 云原生安全技术 ![image-20211228165028007](../img/cloudnative/image-20211228165028007.png) ![image-20211228165053215](../img/cloudnative/image-20211228165053215.png) ![image-20211228164913741](../img/cloudnative/image-20211228164913741.png) > **基础设施安全**：存储安全（加密存储、容灾备份）、网络安全（网络策略管理、访问控制）、计算安全（系统加固、资源隔离） > **应用安全**：应用数据安全、应用配置安全、应用环境安全 > **云原生研发安全**：代码托管、代码审计、软件管理、可信测试、可信构建 > **容器生命周期安全**：运行时安全、容器构建（镜像扫描、镜像签名）、部署安全（合规部署）、组件安全 > **安全管理**：身份认证、访问授权、账号管理、审计日志、密钥管理、监控告警 > 以上所有云原生相关技术都是围绕着Kubernetes展开。 # 四、serverless 无服务器架构 > 云原生2.0时代，由On Cloud变为In Cloud，生于云，长于云，且立而不破。 ## 4.1 计算资源控制力与抽象化 ![image-20220110112117912](../img/cloudnative/image-20220110112117912.png) ## 4.2 计算资源成本发展 ![image-20220110113626429](../img/cloudnative/image-20220110113626429.png) ## 4.3 云原生技术未来发展方向 由于业务的逐利性，我们需要对计算资源和服务进行不断的抽象，在抽象的过程中我们会发现越来越少的能够接触到IT基础设施层，仅能够接触到各种技术的服务能力，这种服务能力我们称之BaaS后端能力及服务，对于开发人员来说，不要去接触到最底层的资源，这也不是其应该去做的事，开发人员应该专注于代码业务逻辑的实现即可。 其实在从IT基础设施层(IaaS)到PaaS层，我们一直在强调这种变化，那么serverless能够给我们带来了什么呢？ ![image-20220110140011904](../img/cloudnative/image-20220110140011904.png) 在传统的云原生架构开发下面，我们基于DevOps，基于微服务和容器云，开发应用的时候，你依旧会选择一个开发框架，开发所使用的底层的基础平台，依旧会涉及到开发一个应用的时候的数据层，逻辑层，展现层，例如我们常说的三层架构和五层架构： ![image-20220110112731134](../img/cloudnative/image-20220110112731134.png) 进入serverless时代后，对于开发人员来说不需要去了解底层的基础设施及多层架构，全部不需要了解，任何一个功能的实现，简单来说，就变成一个个代码片段，通过代码片段去实现功能，通过代码片段的组合组装，来实现复杂一点的流程，这就是serverless所要达到的目的，因此Serverless有两个部分组成，一个是前面我们提到的BaaS，另一个是FaaS（funtion as a Service）层，函数即服务，当我们的BaaS能力足够强以后，我们去实现代码的功能非常简单，只要写一个个函数，并让其执行即可,即可达到项目上线的目的。 ![image-20220110112747843](../img/cloudnative/image-20220110112747843.png) ================================================ FILE: docs/cloud/virtual.md ================================================ # 虚拟化技术 # 一、计算机系统虚拟化定义 ## 1.1 物理机向虚拟机演进过程 当物理机资源足够某一应用使用，并有多余的富余时，可以把多余的资源提供给其它的应用使用，这时就需要一种计算资源“打包技术”，2001年vmware公司提出来虚拟机，即在一台物理机上运行多个虚拟机，让不同或相同的应用使用虚拟机运行，这就达到了共享一台物理机计算资源的方式。 ![image-20220110200045146](../img/virtual/image-20220110200045146.png) ![image-20220110200109732](../img/virtual/image-20220110200109732.png) ## 1.2 计算机系统虚拟化定义 计算机系统虚拟化就是对计算机资源的抽像。在物理机中创建软件或虚拟表示形式的应用、服务器、存储和网络，以减少 IT 开销，同时提高效率和敏捷性。 虚拟化可以提高 IT 敏捷性、灵活性和可扩展性，同时大幅节约成本。更高的工作负载移动性、更高的性能和资源可用性、自动化运维 - 这些都是虚拟化的优势，虚拟化技术可以使 IT 部门更轻松地进行管理以及降低拥有成本和运维成本。 # 二、虚拟化技术分类 ## 2.1 虚拟化管理程序 Hypervisor（VMM） 一种运行在物理机和虚拟机操作系统之间的中间==软件层==，可以允许多个操作系统和应用共享硬件，即虚拟机监视器，也可称之为VMM。 ![image-20220110155353456](../img/virtual/image-20220110155353456.png) ## 2.2 虚拟化管理程序 Hypervisors作用 Hypervisor是所有虚拟化技术的核心。 非中断地支持多工作负载迁移的能力是Hypervisor的基本功能。 Hypervisors是一种在虚拟环境中的“元”操作系统。他们可以访问服务器上包括磁盘和内存在内的所有物理设备。Hypervisors不但协调着这些硬件资源的访问，而且在各个虚拟机之间施加防护。当服务器启动并执行Hypervisor时，它会加载所有虚拟机客户端的操作系统同时会分配给每一台虚拟机适量的内存，CPU，网络和磁盘。 ## 2.3 虚拟化管理程序Hypervisors分类 目前市场上各种x86 管理程序(hypervisor)的架构存在差异，三个最主要的架构类别包括： ### 2.3.1 半虚拟化（Hypervisor Type I） 虚拟机直接运行在系统硬件上，创建硬件全仿真实例，被称为“裸机”型。 裸机型在虚拟化中Hypervisor直接管理调用硬件资源，不需要底层操作系统，也可以将Hypervisor看作一个很薄的操作系统。这种方案的性能处于主机虚拟化与操作系统虚拟化之间。 例如：Xen ![image-20220110194319016](../img/virtual/image-20220110194319016.png) ![image-20220110194348034](../img/virtual/image-20220110194348034.png) ### 2.3.2 硬件辅助全虚拟化(HypervisorTyep II) 虚拟机运行在传统操作系统(HOST OS)上，同样创建的是硬件全仿真实例，被称为“托管（宿主）”型。托管型/主机型Hypervisor运行在基础操作系统上，构建出一整套虚拟硬件平台（CPU/Memory/Storage/Adapter），使用者根据需要安装新的操作系统和应用软件，底层和上层的操作系统可以完全无关化，如Windows运行Linux操作系统。主机虚拟化中VM的应用程序调用硬件资源时需要经过:VM内核->Hypervisor->主机内核，因此相对来说，性能是三种虚拟化技术中最差的。 例如：kvm ![image-20220110194527729](../img/virtual/image-20220110194527729.png) ![image-20220110194550287](../img/virtual/image-20220110194550287.png) ### 2.3.3 软件全虚拟化(Hypervisor Type III) 所谓的软件全虚拟化，即非硬件辅助全虚拟化，模拟CPU让VM使用，效率低，例如：QEMU ### 2.3.4 操作系统虚拟化 称为轻量级虚拟化，允许操作系统内核拥有彼此隔离和分割的多用户空间实例（instance），这些实例也被称之为容器。其是基于Linux内核中的namespace,cgroup实现 例如：LXC,Docker ![image-20220110195509448](../img/virtual/image-20220110195509448.png) # 三、虚拟化技术管理工具 ![image-20220110151928337](../img/virtual/image-20220110151928337.png) # 四、虚拟机管理工具 VMware Workstation pro ## 4.1 安装前准备 - 创建安装目录 ![image-20220111141447039](../img/virtual/image-20220111141447039.png) - 准备操作系统ISO镜像文件 ![image-20220111141519483](../img/virtual/image-20220111141519483.png) - 虚拟机软件准备 ![image-20220111141943956](../img/virtual/image-20220111141943956.png) ## 4.2 安装Linux操作系统虚拟机 ### 4.2.1 创建虚拟机 ![image-20220111152601888](../img/virtual/image-20220111152601888.png) ![image-20220111142255315](../img/virtual/image-20220111142255315.png) ![image-20220111142334823](../img/virtual/image-20220111142334823.png) ![image-20220111142511897](../img/virtual/image-20220111142511897.png) ![image-20220111142535948](../img/virtual/image-20220111142535948.png) ![image-20220111142606674](../img/virtual/image-20220111142606674.png) ![image-20220111142811078](../img/virtual/image-20220111142811078.png) ![image-20220111142844069](../img/virtual/image-20220111142844069.png) ![image-20220111142910855](../img/virtual/image-20220111142910855.png) ![image-20220111142933018](../img/virtual/image-20220111142933018.png) ![image-20220111142952608](../img/virtual/image-20220111142952608.png) ![image-20220111143014339](../img/virtual/image-20220111143014339.png) ![image-20220111143033619](../img/virtual/image-20220111143033619.png) ![image-20220111143138494](../img/virtual/image-20220111143138494.png) ![image-20220111143204016](../img/virtual/image-20220111143204016.png) ![image-20220111143237484](../img/virtual/image-20220111143237484.png) ### 4.2.2 安装操作系统 ![image-20220111143507489](../img/virtual/image-20220111143507489.png) ![image-20220111143631795](../img/virtual/image-20220111143631795.png) ![image-20220111143747401](../img/virtual/image-20220111143747401.png) ![image-20220111143830169](../img/virtual/image-20220111143830169.png) ![image-20220111144005531](../img/virtual/image-20220111144005531.png) ![image-20220111144154956](../img/virtual/image-20220111144154956.png) ![image-20220111144254002](../img/virtual/image-20220111144254002.png) ![image-20220111144409337](../img/virtual/image-20220111144409337.png) ![image-20220111144453627](../img/virtual/image-20220111144453627.png) ![image-20220111144547786](../img/virtual/image-20220111144547786.png) ![image-20220111144638457](../img/virtual/image-20220111144638457.png) ![image-20220111144720723](../img/virtual/image-20220111144720723.png) ![image-20220111144745098](../img/virtual/image-20220111144745098.png) ![image-20220111144823978](../img/virtual/image-20220111144823978.png) ![image-20220111145005563](../img/virtual/image-20220111145005563.png) ![image-20220111144921587](../img/virtual/image-20220111144921587.png) ![image-20220111145053730](../img/virtual/image-20220111145053730.png) ![image-20220111145809091](../img/virtual/image-20220111145809091.png) ![image-20220111145937589](../img/virtual/image-20220111145937589.png) ![image-20220111150021029](../img/virtual/image-20220111150021029.png) ![image-20220111150042642](../img/virtual/image-20220111150042642.png) ![image-20220111150130846](../img/virtual/image-20220111150130846.png) ![image-20220111150430571](../img/virtual/image-20220111150430571.png) ![image-20220111150452529](../img/virtual/image-20220111150452529.png) ![image-20220111150516285](../img/virtual/image-20220111150516285.png) ![image-20220111150542618](../img/virtual/image-20220111150542618.png) ![image-20220111150600954](../img/virtual/image-20220111150600954.png) ![image-20220111150708634](../img/virtual/image-20220111150708634.png) ![image-20220111150735190](../img/virtual/image-20220111150735190.png) ## 4.3 VMware Workstation虚拟机网络设置 ![image-20220111152914753](../img/virtual/image-20220111152914753.png) ![image-20220111153030283](../img/virtual/image-20220111153030283.png) ~~~powershell # vim /etc/sysconfig/network-scripts/ifcfg-ens33 # cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="ens33" UUID="c8838cc3-83d3-45fb-8ddd-863e5f6c7c2c" DEVICE="ens33" ONBOOT="yes" IPADDR="192.168.10.130" PREFIX="24" GATEWAY="192.168.10.2" DNS1="119.29.29.29" ~~~ ![image-20220111153523824](../img/virtual/image-20220111153523824.png) ~~~powershell # systemctl restart network # ip a s ens33 2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:2d:2e:6b brd ff:ff:ff:ff:ff:ff inet 192.168.10.130/24 brd 192.168.10.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::81a2:2c89:8b27:a756/64 scope link noprefixroute valid_lft forever preferred_lft forever ~~~ ~~~powershell ping网关，检查网络连通性 # ping 192.168.10.2 PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data. 64 bytes from 192.168.10.2: icmp_seq=1 ttl=128 time=0.186 ms 64 bytes from 192.168.10.2: icmp_seq=2 ttl=128 time=0.369 ms 64 bytes from 192.168.10.2: icmp_seq=3 ttl=128 time=0.145 ms ~~~ ~~~powershell ping互联网服务器，检查网络连通性 # ping -c 4 www.baidu.com PING www.a.shifen.com (110.242.68.4) 56(84) bytes of data. 64 bytes from 110.242.68.4 (110.242.68.4): icmp_seq=1 ttl=128 time=11.7 ms 64 bytes from 110.242.68.4 (110.242.68.4): icmp_seq=2 ttl=128 time=11.6 ms 64 bytes from 110.242.68.4 (110.242.68.4): icmp_seq=3 ttl=128 time=12.1 ms 64 bytes from 110.242.68.4 (110.242.68.4): icmp_seq=4 ttl=128 time=11.6 ms --- www.a.shifen.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 11.629/11.778/12.118/0.239 ms ~~~ ## 4.4 VMware Workstation虚拟机系统安全设置及更新 ### 4.4.1 关闭防火墙 ~~~powershell 关闭防火墙 # systemctl stop firewalld 设置防火墙为开机禁用状态 # systemctl disable firewalld ~~~ ### 4.4.2 关闭SELINUX ~~~powershell 关闭SELINUX后，必须重启系统才能使其修改生效。 # sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config ~~~ ~~~powershell 确认是否关闭 # cat /etc/selinux/config ...... SELINUX=disabled ...... ~~~ ### 4.4.3 更新系统 ~~~powershell 更新系统 # yum -y update ~~~ ~~~powershell 重启操作系统 # reboot ~~~ ## 4.5 VMware Workstation虚拟机快照设置 > 为了防止部署应用过程中多次使用同一环境，可以考虑使用虚拟机快照功能。建议关闭虚拟机后再做快照。 ![image-20220111153951801](../img/virtual/image-20220111153951801.png) ![image-20220111154024872](../img/virtual/image-20220111154024872.png) ![image-20220111154107331](../img/virtual/image-20220111154107331.png) ![image-20220111154248279](../img/virtual/image-20220111154248279.png) ## 4.6 VMware Workstation Pro模板机 > 在后期应用中，可能会同时需要多台虚拟机，建议使用虚拟机模板机创建虚拟机使用。虚拟机模板机安装完成后，关闭防火墙、关闭SELINUX、更新操作系统即可，不需要配置IP地址，后期使用过程中再设置IP地址。 ### 4.6.1 新安装的虚拟机硬盘文件注意事项 ![image-20220111143138494](../img/virtual/image-20220111143138494.png) ### 4.6.2 利用模板机快速创建虚拟机 ![image-20220111154718283](../img/virtual/image-20220111154718283.png) ![image-20220111154818419](../img/virtual/image-20220111154818419.png) ![image-20220111154951242](../img/virtual/image-20220111154951242.png) ![image-20220111155010629](../img/virtual/image-20220111155010629.png) ![image-20220111155037173](../img/virtual/image-20220111155037173.png) ![image-20220111155059999](../img/virtual/image-20220111155059999.png) ![image-20220111155159931](../img/virtual/image-20220111155159931.png) ![image-20220111155237982](../img/virtual/image-20220111155237982.png) ![image-20220111155300651](../img/virtual/image-20220111155300651.png) ![image-20220111155316649](../img/virtual/image-20220111155316649.png) ![image-20220111155338372](../img/virtual/image-20220111155338372.png) ![image-20220111155402773](../img/virtual/image-20220111155402773.png) ![image-20220111155423525](../img/virtual/image-20220111155423525.png) ![image-20220111155445712](../img/virtual/image-20220111155445712.png) ![image-20220111155549589](../img/virtual/image-20220111155549589.png) ![image-20220111155615710](../img/virtual/image-20220111155615710.png) ![image-20220111155640584](../img/virtual/image-20220111155640584.png) # 五、虚拟机管理工具KVM ## 5.1 KVM系统需求 ~~~powershell 官方推荐宿主机硬件需求 Host system requirements 1.1核心 2.2G内存 3.6G硬盘 ~~~ ~~~powershell 运行KVM虚拟机需求 KVM hypervisor requirements [root@localhost ~]# lscpu 虚拟化： VT-x #intel虚拟技术 ~~~ ~~~powershell 检查CPU是否有以下特性，此特性支持CPU虚拟化 [root@localhost ~]# egrep 'svm|vmx' /proc/cpuinfo vmx ~~~ ~~~powershell 在BIOS中开启虚拟化功能 BIOS Enable Virtualization ~~~ ~~~powershell 不同CPU厂商 CPU虚拟化技术名称 Intel CPU: VT-x AMD CPU: AMD -V ~~~ ## 5.2 安装KVM虚拟机管理工具 > 如果需要在vmware workstation 测试环境上安装kvm: ![](../img/virtual/vmware_workstation_install_option.png) ~~~powershell 管理KVM虚拟机管理工具组件 [root@localhost ~]# yum grouplist [root@localhost ~]# yum -y groupinstall "虚拟化*" ~~~ ~~~powershell 安装完成后验证libvirtd状态及系统模块是否添加 [root@localhost ~]#systemctl status libvirtd [root@localhost ~]#lsmod | grep kvm ~~~ ~~~powershell 在firewalld中配置default默认区域中masquerade，以便虚拟机通过连接外网。 [root@localhost ~]# firewall-cmd --permanent --zone=public --add-masquerade 加载配置，使上述配置生效。 [root@localhost ~]# firewall-cmd --reload ~~~ ~~~powershell 通过命令查看虚拟机列表 [root@localhost ~]# virsh list --all ~~~ ## 5.3 使用virt-manager安装KVM虚拟机 ### 5.3.1 准备 - 准备iso镜像文件 - 准备一套系统预备工具 PXE&kickstart || Cobbler ### 5.3.2 KVM虚拟机组成 - 磁盘镜像文件 /var/lib/libvirt/images - 配置文件 /etc/libvirt/qemu/ ### 5.3.3 Linux主机安装 ``` virt-manager ``` ![1545964085410](../img/virtual/1545964085410.png) ![1545964256167](../img/virtual/1545964256167.png) ![1545964311912](../img/virtual/1545964311912.png) ![1545964480043](../img/virtual/1545964480043.png) ![1545964650939](../img/virtual/1545964650939.png) ![1545964871080](../img/virtual/1545964871080.png) ![1545965036341](../img/virtual/1545965036341.png) ![1545965229134](../img/virtual/1545965229134.png) ![1545965315019](../img/virtual/1545965315019.png) ![1545965440953](../img/virtual/1545965440953.png) ![1545965688832](../img/virtual/1545965688832.png) ![1545965872625](../img/virtual/1545965872625.png) ![1545965938349](../img/virtual/1545965938349.png) ### 5.3.4 查看已安装的虚拟机 ~~~powershell 通过命令查看虚拟机列表 [root@localhost ~]# virsh list --all ~~~ ================================================ FILE: docs/dart/syntax.md ================================================ # Dart语法学习 ## 目录 * 参考资料 * 语言特性 * 关键字 * 变量与常量 * 数据类型 * 运算符 operators * 控制流程语句 * 异常 Exceptions * 函数 Function * 类 Class * 类-方法 * 类-抽象类 * 类-隐式接口 * 类-扩展一个类（重写） * 库和可见性 * 异步支持 ## 参考资料 * [【官方文档】](https://www.dartlang.org/guides/language/language-tour ) * [【极客学院】](http://wiki.jikexueyuan.com/project/dart-language-tour/) * [【author：AWeiLoveAndroid】](https://www.jianshu.com/p/3d927a7bf020) * [【author：soojade】](https://www.jianshu.com/p/a7cc623132b0) * [【author：优腾爱乐】](https://www.jianshu.com/p/8a62b1a2fd75) ## 语言特性 * Dart所有的东西都是对象， 即使是数字numbers、函数function、null也都是对象，所有的对象都继承自Object类。 * Dart动态类型语言, 尽量给变量定义一个类型，会更安全，没有显示定义类型的变量在 debug 模式下会类型会是 dynamic(动态的)。 * Dart 在 running 之前解析你的所有代码，指定数据类型和编译时的常量，可以提高运行速度。 * Dart中的类和接口是统一的，类即接口，你可以继承一个类，也可以实现一个类（接口），自然也包含了良好的面向对象和并发编程的支持。 * Dart 提供了顶级函数(如：main())。 * Dart 没有 public、private、protected 这些关键字，变量名以"_"开头意味着对它的 lib 是私有的。 * 没有初始化的变量都会被赋予默认值 null。 * final的值只能被设定一次。const 是一个编译时的常量，可以通过 const 来创建常量值，var c=const[];，这里 c 还是一个变量，只是被赋值了一个常量值，它还是可以赋其它值。实例变量可以是 final，但不能是 const。 * 编程语言并不是孤立存在的，Dart也是这样，他由语言规范、虚拟机、类库和工具等组成： * SDK：SDK 包含 Dart VM、dart2js、Pub、库和工具。 * Dartium：内嵌 Dart VM 的 Chromium ，可以在浏览器中直接执行 dart 代码。 * Dart2js：将 Dart 代码编译为 JavaScript 的工具。 * Dart Editor：基于 Eclipse 的全功能 IDE，并包含以上所有工具。支持代码补全、代码导航、快速修正、重构、调试等功能。 ## 关键字（56个） 关键字|-|-|- -|-|-|- abstract | do | import | super as | dynamic | in | switch assert | else | interface | sync enum | implements | is | this async | export | library | throw await | external | mixin | true break | extends | new | try case | factory | null | typedef catch | false | operator | var class | final | part | void const | finally | rethrow | while continue | for | return | with covariant | get | set | yield default | if | static | deferred ## 变量与常量 1. 变量声明与初始化 * 调用的变量name包含对String值为“张三” 的对象的引用，name推断变量的类型是String，但可以通过指定它来更改该类型，如果对象不限于单一类型（没有明确的类型），请使用Object或dynamic关键字。 ```dart // 没有明确类型，编译的时候根据值明确类型 var name = ‘Bob’; Object name = '张三'; dynamic name = '李四'; // 显示声明将被推断类型, 可以使用String显示声明字符串类型 String name = 'Bob' ; ``` 2. 默认值 * 未初始化的变量的初始值为null（包括数字），因此数字、字符串都可以调用各种方法 ```dart //测试 数字类型的初始值是什么? int lineCount; // 为false的时候抛出异常 assert(lineCount == null); print(lineCount); //打印结果为null，证明数字类型初始化值是null ``` 3. final and const * 如果您从未打算更改一个变量，那么使用 final 或 const，不是var，也不是一个类型。 一个 final 变量只能被初始化一次; const变量是一个编译时常量，(Const变量是隐式的final) final的顶级或类变量在第一次使用时被初始化。 * 被final修饰的顶级变量或类变量在第一次声明的时候就需要初始化。 ``` dart // The final variable 'outSideFinalName' must be initialized. final String outSideFinalName ``` * 被final或者const修饰的变量，变量类型可以省略，建议指定数据类型。 ``` dart //可以省略String这个类型声明 final name = "Bob"; final String name1 = "张三"; const name2 = "alex"; const String name3 = "李四"; ``` * 被 final 或 const 修饰的变量无法再去修改其值。 ``` dart final String outSideFinalName = "Alex"; // outSideFinalName', a final variable, can only be set once // 一个final变量，只能被设置一次。 outSideFinalName = "Bill"; const String outSideName = 'Bill'; // 这样写，编译器提示：Constant variables can't be assigned a value // const常量不能赋值 // outSideName = "小白"; ``` * flnal 或者 const 不能和 var 同时使用 ``` dart // Members can't be declared to be both 'const' and 'var' const var String outSideName = 'Bill'; // Members can't be declared to be both 'final' and 'var' final var String name = 'Lili'; ``` * 常量如果是类级别的，请使用 static const ``` dart // 常量如果是类级别的，请使用 static const static const String name3 = 'Tom'; // 这样写保存 // Only static fields can be declared as const // 只有静态字段可以声明为const //const String name3 = 'Tom'; ``` * 常量的运算 ``` dart const speed = 100; //速度（km/h） const double distance = 2.5 * speed; // 距离 = 时间 * 速度 final speed2 = 100; //速度（km/h） final double distance2 = 2.5 * speed2; // 距离 = 时间 * 速度 ``` * const关键字不只是声明常数变量，您也可以使用它来创建常量值，以及声明创建常量值的构造函数，任何变量都可以有一个常量值。 ``` dart // 注意: [] 创建的是一个空的list集合 // const []创建一个空的、不可变的列表（EIL）。 var varList = const []; // varList 当前是一个EIL final finalList = const []; // finalList一直是EIL const constList = const []; // constList 是一个编译时常量的EIL // 可以更改非final,非const变量的值 // 即使它曾经具有const值 varList = ["haha"]; // 不能更改final变量或const变量的值 // 这样写，编译器提示：a final variable, can only be set once // finalList = ["haha"]; // 这样写，编译器提示：Constant variables can't be assigned a value // constList = ["haha"]; ``` * 在常量表达式中，该运算符的操作数必须为'bool'、'num'、'String'或'null', const常量必须用conat类型的值初始化。 ``` dart const String outSideName = 'Bill'; final String outSideFinalName = 'Alex'; const String outSideName2 = 'Tom'; const aConstList = const ['1', '2', '3']; // In constant expressions, operands of this operator must be of type 'bool', 'num', 'String' or 'null' // 在常量表达式中，该运算符的操作数必须为'bool'、'num'、'String'或'null'。 const validConstString = '$outSideName $outSideName2 $aConstList'; // Const variables must be initialized with a constant value // const常量必须用conat类型的值初始化 const validConstString = '$outSideName $outSideName2 $outSideFinalName'; var outSideVarName='Cathy'; // Const variables must be initialized with a constant value. // const常量必须用conat类型的值初始化 const validConstString = '$outSideName $outSideName2 $outSideVarName'; // 正确写法 const String outSideConstName = 'Joy'; const validConstString = '$outSideName $outSideName2 $outSideConstName'; ``` ## 数据类型 1. num * num 是数字类型的父类，有两个子类 int 和 double。 * int 根据平台的不同，整数值不大于64位。在Dart VM上，值可以从-263到263 - 1，编译成JavaScript的Dart使用JavaScript代码，允许值从-253到253 - 1。 * double 64位（双精度）浮点数，如IEEE 754标准所规定。 ``` dart int a = 1; print(a); double b = 1.12; print(b); // String -> int int one = int.parse('1'); // 输出3 print(one + 2); // String -> double var onePointOne = double.parse('1.1'); // 输出3.1 print(onePointOne + 2); // int -> String String oneAsString = 1.toString(); // The argument type 'int' can't be assigned to the parameter type 'String' //print(oneAsString + 2); // 输出 1 + 2 print('$oneAsString + 2'); // 输出 1 2 print('$oneAsString 2'); // double -> String 注意括号中要有小数点位数，否则报错 String piAsString = 3.14159.toStringAsFixed(2); // 截取两位小数, 输出3.14 print(piAsString); String aString = 1.12618.toStringAsFixed(2); // 检查是否四舍五入，输出1.13，发现会做四舍五入 print(aString); ``` 2. String * Dart里面的String是一系列 UTF-16 代码单元。 * Dart里面的String是一系列 UTF-16 代码单元。 * 单引号或者双引号里面嵌套使用引号。 * 用 或{} 来计算字符串中变量的值，需要注意的是如果是表达式需要${表达式} ``` dart String singleString = 'abcdddd'; String doubleString = "abcsdfafd"; String sdString = '$singleString a "bcsd" ${singleString}'; String dsString = "abc 'aaa' $sdString"; print(sdString); print(dsString); String singleString = 'aaa'; String doubleString = "bbb"; // 单引号嵌套双引号 String sdString = '$singleString a "bbb" ${doubleString}'; // 输出 aaa a "bbb" bbb print(sdString); // 双引号嵌套单引号 String dsString = "${singleString.toUpperCase()} abc 'aaa' $doubleString.toUpperCase()"; // 输出 AAA abc 'aaa' bbb.toUpperCase(), 可以看出 ”$doubleString.toUpperCase()“ 没有加“{}“，导致输出结果是”bbb.toUpperCase()“ print(dsString); ``` 3. bool * Dart 是强 bool 类型检查，只有bool 类型的值是true 才被认为是true。 * 只有两个对象具有bool类型：true和false，它们都是编译时常量。 * Dart的类型安全意味着您不能使用 if（nonbooleanValue） 或 assert（nonbooleanValue） 等代码, 相反Dart使用的是显式的检查值。 * assert 是语言内置的断言函数，仅在检查模式下有效在开发过程中， 除非条件为真，否则会引发异常。(断言失败则程序立刻终止)。 ``` dart // 检查是否为空字符串 var fullName = ''; assert(fullName.isEmpty); // 检查0 var hitPoints = 0; assert(hitPoints <= 0); // 检查是否为null var unicorn; assert(unicorn == null); // 检查是否为NaN var iMeantToDoThis = 0 / 0; assert(iMeantToDoThis.isNaN); ``` 4. List集合 * 在Dart中，数组是List对象，因此大多数人只是将它们称为List。Dart list文字看起来像JavaScript数组文字 ``` dart //创建一个int类型的list List list = [10, 7, 23]; // 输出[10, 7, 23] print(list); // 使用List的构造函数，也可以添加int参数，表示List固定长度，不能进行添加 删除操作 var fruits = new List(); // 添加元素 fruits.add('apples'); // 添加多个元素 fruits.addAll(['oranges', 'bananas']); List subFruits = ['apples', 'oranges', 'banans']; // 添加多个元素 fruits.addAll(subFruits); // 输出： [apples, oranges, bananas, apples, oranges, banans] print(fruits); // 获取List的长度 print(fruits.length); // 获取第一个元素 print(fruits.first); // 获取元素最后一个元素 print(fruits.last); // 利用索引获取元素 print(fruits[0]); // 查找某个元素的索引号 print(fruits.indexOf('apples')); // 删除指定位置的元素，返回删除的元素 print(fruits.removeAt(0)); // 删除指定元素,成功返回true，失败返回false // 如果集合里面有多个“apples”, 只会删除集合中第一个改元素 fruits.remove('apples'); // 删除最后一个元素，返回删除的元素 fruits.removeLast(); // 删除指定范围(索引)元素，含头不含尾 fruits.removeRange(start,end); // 删除指定条件的元素(这里是元素长度大于6) fruits.removeWhere((item) => item.length >6)； // 删除所有的元素 fruits.clear(); ``` * 注意事项： 1. 可以直接打印list包括list的元素，list也是一个对象。但是java必须遍历才能打印list，直接打印是地址值。 2. 和java一样list里面的元素必须保持类型一致，不一致就会报错。 3. 和java一样list的角标从0开始。 4. 如果集合里面有多个相同的元素“X”, 只会删除集合中第一个改元素 5. Map集合 * 一般来说，map是将键和值相关联的对象。键和值都可以是任何类型的对象。每个键只出现一次，但您可以多次使用相同的值。Dart支持map由map文字和map类型提供。 * 初始化Map方式一： 直接声明，用{}表示，里面写key和value，每组键值对中间用逗号隔开。 ``` dart // Two keys in a map literal can't be equal. // Map companys = {'Alibaba': '阿里巴巴', 'Tencent': '腾讯', 'baidu': '百度', 'Alibaba': '钉钉', 'Tenect': 'qq-music'}; Map companys = {'Alibaba': '阿里巴巴', 'Tencent': '腾讯', 'baidu': '百度'}; // 输出：{Alibaba: 阿里巴巴, Tencent: 腾讯, baidu: 百度} print(companys); ``` * 创建Map方式二：先声明，再去赋值。 ``` dart Map schoolsMap = new Map(); schoolsMap['first'] = '清华'; schoolsMap['second'] = '北大'; schoolsMap['third'] = '复旦'; // 打印结果 {first: 清华, second: 北大, third: 复旦} print(schoolsMap); var fruits = new Map(); fruits["first"] = "apple"; fruits["second"] = "banana"; fruits["fifth"] = "orange"; //换成双引号,换成var 打印结果 {first: apple, second: banana, fifth: orange} print(fruits); ``` * Map API ``` dart // 指定键值对的参数类型 var aMap = new Map(); // Map的赋值，中括号中是Key，这里可不是数组 aMap[1] = '小米'; //Map中的键值对是唯一的 //同Set不同，第二次输入的Key如果存在，Value会覆盖之前的数据 aMap[1] = 'alibaba'; // map里面的value可以相同 aMap[2] = 'alibaba'; // map里面value可以为空字符串 aMap[3] = ''; // map里面的value可以为null aMap[4] = null; print(aMap); // 检索Map是否含有某Key assert(aMap.containsKey(1)); //删除某个键值对 aMap.remove(1); print(aMap); ``` * 注意事项 1. map的key类型不一致也不会报错。 2. 添加元素的时候，会按照你添加元素的顺序逐个加入到map里面，哪怕你的key，比如分别是 1,2,4，看起来有间隔，事实上添加到map的时候是{1:value,2:value,4:value} 这种形式。 3. map里面的key不能相同。但是value可以相同,value可以为空字符串或者为null。 ## 运算符 描述 | 操作符 -|- 一元后置操作符 | expr++ expr-- () [] . ?. 一元前置操作符 | expr !expr ~expr ++expr --expr 乘除 | * / % ~/ 加减 | + - 位移 | << >> 按位与 | & 按位或 | 按位异或 | ^ 逻辑与 | && 逻辑或 | 关系和类型判断 | >= > <= < as is is! 等 | == != 如果为空 | ?? 条件表达式 | expr1 ? expr2 : expr3 赋值 | = *= /= ~/= %= += -= <<= >>= &= ^= = ??= 级联 | .. ## 流程控制语句（Control flow statements） * if...else * for * while do-whild * break continue * break continue * assert（仅在checked模式有效） ## 异常（Exceptions） 1. throw * 抛出固定类型的异常 ``` dart throw new FormatException('Expected at least 1 section'); ``` * 抛出任意类型的异常 ``` dart throw 'Out of llamas!'; ``` * 因为抛出异常属于表达式，可以将throw语句放在=>语句中，或者其它可以出现表达式的地方 ``` dart distanceTo(Point other) => throw new UnimplementedError(); ``` 2. catch * 将可能出现异常的代码放置到try语句中，可以通过 on语句来指定需要捕获的异常类型，使用catch来处理异常。 ``` dart try { breedMoreLlamas(); } on OutOfLlamasException { // A specific exception buyMoreLlamas(); } on Exception catch (e) { // Anything else that is an exception print('Unknown exception: $e'); } catch (e, s) { print('Exception details:\n $e'); print('Stack trace:\n $s'); } ``` 3. rethrow * rethrow语句用来处理一个异常，同时希望这个异常能够被其它调用的部分使用。 ``` dart final foo = ''; void misbehave() { try { foo = "1"; } catch (e) { print('2'); rethrow;// 如果不重新抛出异常，main函数中的catch语句执行不到 } } void main() { try { misbehave(); } catch (e) { print('3'); } } ``` 4. finally * Dart的finally用来执行那些无论异常是否发生都执行的操作。 ``` dart final foo = ''; void misbehave() { try { foo = "1"; } catch (e) { print('2'); } } void main() { try { misbehave(); } catch (e) { print('3'); } finally { print('4'); // 即使没有rethrow最终都会执行到 } } ``` ## 函数 Function * 以下是一个实现函数的例子： ``` dart bool isNoble(int atomicNumber) { return _nobleGases[atomicNumber] != null; } ``` 1. main()函数 * 每个应用程序都必须有一个顶层main()函数，它可以作为应用程序的入口点。该main()函数返回void并具有List参数的可选参数。 ``` dart void main() { querySelector('#sample_text_id') ..text = 'Click me!' ..onClick.listen(reverseText); } ``` * 级联符号..允许您在同一个对象上进行一系列操作。除了函数调用之外，还可以访问同一对象上的字段。这通常会为您节省创建临时变量的步骤，并允许您编写更流畅的代码。 ``` dart querySelector('#confirm') // Get an object. ..text = 'Confirm' // Use its members. ..classes.add('important') ..onClick.listen((e) => window.alert('Confirmed!')); ``` * 上述例子相对于： ``` dart var button = querySelector('#confirm'); button.text = 'Confirm'; button.classes.add('important'); button.onClick.listen((e) => window.alert('Confirmed!')); ``` * 级联符号也可以嵌套使用。 例如： ``` dart final addressBook = (AddressBookBuilder() ..name = 'jenny' ..email = 'jenny@example.com' ..phone = (PhoneNumberBuilder() ..number = '415-555-0100' ..label = 'home') .build()) .build(); ``` * 当返回值是void时不能构建级联。 例如，以下代码失败： ``` dart var sb = StringBuffer(); sb.write('foo') // 返回void ..write('bar'); // 这里会报错 ``` * 注意： 严格地说，级联的..符号不是操作符。它只是Dart语法的一部分。 2. 可选参数 * 可选的命名参数, 定义函数时，使用{param1, param2, …}，用于指定命名参数。例如： ``` dart //设置[bold]和[hidden]标志 void enableFlags({bool bold, bool hidden}) { // ... } enableFlags(bold: true, hidden: false); ``` * 可选的位置参数，用[]它们标记为可选的位置参数： ``` dart String say(String from, String msg, [String device]) { var result = '$from says $msg'; if (device != null) { result = '$result with a $device'; } return result; } ``` * 下面是一个不带可选参数调用这个函数的例子： ``` dart say('Bob', 'Howdy'); //结果是： Bob says Howdy ``` * 下面是用第三个参数调用这个函数的例子： ``` dart say('Bob', 'Howdy', 'smoke signal'); //结果是：Bob says Howdy with a smoke signal ``` 3. 默认参数 * 函数可以使用=为命名参数和位置参数定义默认值。默认值必须是编译时常量。如果没有提供默认值，则默认值为null。 * 下面是为命名参数设置默认值的示例: ``` dart // 设置 bold 和 hidden 标记的默认值都为false void enableFlags2({bool bold = false, bool hidden = false}) { // ... } // 调用的时候：bold will be true; hidden will be false. enableFlags2(bold: true); ``` * 下一个示例显示如何为位置参数设置默认值： ``` dart String say(String from, String msg, [String device = 'carrier pigeon', String mood]) { var result = '$from says $msg'; if (device != null) { result = '$result with a $device'; } if (mood != null) { result = '$result (in a $mood mood)'; } return result; } //调用方式： say('Bob', 'Howdy'); //结果为：Bob says Howdy with a carrier pigeon; ``` * 您还可以将list或map作为默认值传递。下面的示例定义一个函数doStuff()，该函数指定列表参数的默认list和gifts参数的默认map。 ``` dart // 使用list 或者map设置默认值 void doStuff( {List list = const [1, 2, 3], Map gifts = const {'first': 'paper', 'second': 'cotton', 'third': 'leather' }}) { print('list: $list'); print('gifts: $gifts'); } ``` 4. 作为一个类对象的功能 * 您可以将一个函数作为参数传递给另一个函数。 ``` dart void printElement(int element) { print(element); } var list = [1, 2, 3]; // 把 printElement函数作为一个参数传递进来 list.forEach(printElement); ``` * 您也可以将一个函数分配给一个变量。 ``` dart var loudify = (msg) => '!!! ${msg.toUpperCase()} !!!'; assert(loudify('hello') == '!!! HELLO !!!'); ``` 5. 匿名函数 * 大多数函数都能被命名为匿名函数，如 main() 或 printElement()。您还可以创建一个名为匿名函数的无名函数，有时也可以创建lambda或闭包。您可以为变量分配一个匿名函数，例如，您可以从集合中添加或删除它。 * 一个匿名函数看起来类似于一个命名函数 - 0或更多的参数，在括号之间用逗号和可选类型标注分隔。 * 下面的代码块包含函数的主体: ``` dart ([[Type] param1[, …]]) { codeBlock; }; ``` * 下面的示例定义了一个具有无类型参数的匿名函数item，该函数被list中的每个item调用，输出一个字符串，该字符串包含指定索引处的值。 ``` dart var list = ['apples', 'bananas', 'oranges']; list.forEach((item) { print('${list.indexOf(item)}: $item'); }); ``` * 如果函数只包含一条语句，可以使用箭头符号=>来缩短它, 比如上面的例2可以简写成： ``` dart list.forEach((item) => print('${list.indexOf(item)}: $item')); ``` 6. 返回值 * 所有函数都返回一个值，如果没有指定返回值，则语句return null，隐式地附加到函数体。 ``` dart foo() {} assert(foo() == null); ``` ## 类（Classes） 1. 对象 * Dart 是一种面向对象的语言，并且支持基于mixin的继承方式。 * Dart 语言中所有的对象都是某一个类的实例,所有的类有同一个基类--Object。 * 基于mixin的继承方式具体是指：一个类可以继承自多个父类。 * 使用new语句来构造一个类，构造函数的名字可能是ClassName，也可以是ClassName.identifier， 例如： ``` dart var jsonData = JSON.decode('{"x":1, "y":2}'); // Create a Point using Point(). var p1 = new Point(2, 2); // Create a Point using Point.fromJson(). var p2 = new Point.fromJson(jsonData); ``` * 使用.（dot）来调用实例的变量或者方法。 ``` dart var p = new Point(2, 2); // Set the value of the instance variable y. p.y = 3; // Get the value of y. assert(p.y == 3); // Invoke distanceTo() on p. num distance = p.distanceTo(new Point(4, 4)); ``` * 使用?.来确认前操作数不为空, 常用来替代. , 避免左边操作数为null引发异常。 ``` dart // If p is non-null, set its y value to 4. p?.y = 4; ``` * 使用const替代new来创建编译时的常量构造函数。 ``` dart var p = const ImmutablePoint(2, 2); ``` * 使用runtimeType方法，在运行中获取对象的类型。该方法将返回Type 类型的变量。 ``` dart print('The type of a is ${a.runtimeType}'); ``` 2. 实例化变量(Instance variables) * 在类定义中，所有没有初始化的变量都会被初始化为null。 ``` dart class Point { num x; // Declare instance variable x, initially null. num y; // Declare y, initially null. num z = 0; // Declare z, initially 0. } ``` * 类定义中所有的变量, Dart语言都会隐式的定义 setter 方法，针对非空的变量会额外增加 getter 方法。 ``` dart class Point { num x; num y; } main() { var point = new Point(); point.x = 4; // Use the setter method for x. assert(point.x == 4); // Use the getter method for x. assert(point.y == null); // Values default to null. } ``` 3. 构造函数(Constructors) * 声明一个和类名相同的函数，来作为类的构造函数。 ``` dart class Point { num x; num y; Point(num x, num y) { // There's a better way to do this, stay tuned. this.x = x; this.y = y; } } ``` * this关键字指向了当前类的实例, 上面的代码可以简化为： ``` dart class Point { num x; num y; // Syntactic sugar for setting x and y // before the constructor body runs. Point(this.x, this.y); } ``` 4. 构造函数不能继承(Constructors aren’t inherited) * Dart 语言中，子类不会继承父类的命名构造函数。如果不显式提供子类的构造函数，系统就提供默认的构造函数。 5. 命名的构造函数(Named constructors) * 使用命名构造函数从另一类或现有的数据中快速实现构造函数。 ``` dart class Point { num x; num y; Point(this.x, this.y); // 命名构造函数Named constructor Point.fromJson(Map json) { x = json['x']; y = json['y']; } } ``` * 构造函数不能被继承，父类中的命名构造函数不能被子类继承。如果想要子类也拥有一个父类一样名字的构造函数，必须在子类是实现这个构造函数。 6. 调用父类的非默认构造函数 * 默认情况下，子类只能调用父类的无名，无参数的构造函数; 父类的无名构造函数会在子类的构造函数前调用; 如果initializer list 也同时定义了，则会先执行initializer list 中的内容，然后在执行父类的无名无参数构造函数，最后调用子类自己的无名无参数构造函数。即下面的顺序： 1. initializer list（初始化列表） 2. super class’s no-arg constructor（父类无参数构造函数） 3. main class’s no-arg constructor （主类无参数构造函数） * 如果父类不显示提供无名无参数构造函数的构造函数，在子类中必须手打调用父类的一个构造函数。这种情况下，调用父类的构造函数的代码放在子类构造函数名后，子类构造函数体前，中间使用:(colon) 分割。 ``` dart class Person { String firstName; Person.fromJson(Map data) { print('in Person'); } } class Employee extends Person { // 父类没有无参数的非命名构造函数，必须手动调用一个构造函数 super.fromJson(data) Employee.fromJson(Map data) : super.fromJson(data) { print('in Employee'); } } main() { var emp = new Employee.fromJson({}); // Prints: // in Person // in Employee if (emp is Person) { // Type check emp.firstName = 'Bob'; } (emp as Person).firstName = 'Bob'; } ``` 7. 初始化列表 * 除了调用父类的构造函数，也可以通过初始化列表在子类的构造函数体前（大括号前）来初始化实例的变量值，使用逗号,分隔。如下所示： ``` dart class Point { num x; num y; Point(this.x, this.y); // 初始化列表在构造函数运行前设置实例变量。 Point.fromJson(Map jsonMap) : x = jsonMap['x'], y = jsonMap['y'] { print('In Point.fromJson(): ($x, $y)'); } } ``` **注意：上述代码，初始化程序无法访问 this 关键字。** 8. 静态构造函数 * 如果你的类产生的对象永远不会改变，你可以让这些对象成为编译时常量。为此，需要定义一个 const 构造函数并确保所有的实例变量都是 final 的。 ``` dart class ImmutablePoint { final num x; final num y; const ImmutablePoint(this.x, this.y); static final ImmutablePoint origin = const ImmutablePoint(0, 0); } ``` 9. 重定向构造函数 * 有时候构造函数的目的只是重定向到该类的另一个构造函数。重定向构造函数没有函数体，使用冒号:分隔。 ``` dart class Point { num x; num y; // 主构造函数 Point(this.x, this.y) { print("Point($x, $y)"); } // 重定向构造函数，指向主构造函数，函数体为空 Point.alongXAxis(num x) : this(x, 0); } void main() { var p1 = new Point(1, 2); var p2 = new Point.alongXAxis(4); } ``` 10. 常量构造函数 * 如果类的对象不会发生变化，可以构造一个编译时的常量构造函数。定义格式如下： * 定义所有的实例变量是final。 * 使用const声明构造函数。 ``` dart class ImmutablePoint { final num x; final num y; const ImmutablePoint(this.x, this.y); static final ImmutablePoint origin = const ImmutablePoint(0, 0); } ``` 11. 工厂构造函数 * 当实现一个使用 factory 关键词修饰的构造函数时，这个构造函数不必创建类的新实例。例如，工厂构造函数可能从缓存返回实例，或者它可能返回子类型的实例。 下面的示例演示一个工厂构造函数从缓存返回的对象： ``` dart class Logger { final String name; bool mute = false; // _cache 是一个私有库,幸好名字前有个 _ 。 static final Map _cache = {}; factory Logger(String name) { if (_cache.containsKey(name)) { return _cache[name]; } else { final logger = new Logger._internal(name); _cache[name] = logger; return logger; } } Logger._internal(this.name); void log(String msg) { if (!mute) { print(msg); } } } ``` **注意：工厂构造函数不能用 this。** ## 方法 * 方法就是为对象提供行为的函数。 1. 实例方法 * 对象的实例方法可以访问实例变量和 this 。以下示例中的 distanceTo() 方法是实例方法的一个例子： ``` dart import 'dart:math'; class Point { num x; num y; Point(this.x, this.y); num distanceTo(Point other) { var dx = x - other.x; var dy = y - other.y; return sqrt(dx * dx + dy * dy); } } ``` 2. setters 和 Getters * 是一种提供对方法属性读和写的特殊方法。每个实例变量都有一个隐式的 getter 方法，合适的话可能还会有 setter 方法。你可以通过实现 getters 和 setters 来创建附加属性，也就是直接使用 get 和 set 关键词： ``` dart class Rectangle { num left; num top; num width; num height; Rectangle(this.left, this.top, this.width, this.height); // 定义两个计算属性: right and bottom. num get right => left + width; set right(num value) => left = value - width; num get bottom => top + height; set bottom(num value) => top = value - height; } main() { var rect = new Rectangle(3, 4, 20, 15); assert(rect.left == 3); rect.right = 12; assert(rect.left == -8); } ``` * 借助于 getter 和 setter ，你可以直接使用实例变量，并且在不改变客户代码的情况下把他们包装成方法。 * 注： 不论是否显式地定义了一个 getter，类似增量（++）的操作符，都能以预期的方式工作。为了避免产生任何向着不期望的方向的影响，操作符一旦调用 getter ，就会把他的值存在临时变量里。 3. 抽象方法 * Instance ， getter 和 setter 方法可以是抽象的，也就是定义一个接口，但是把实现交给其他的类。要创建一个抽象方法，使用分号（；）代替方法体： ``` dart abstract class Doer { // ...定义实例变量和方法... void doSomething(); // 定义一个抽象方法。 } class EffectiveDoer extends Doer { void doSomething() { // ...提供一个实现，所以这里的方法不是抽象的... } } ``` 4. 枚举类型 * 枚举类型，通常被称为 enumerations 或 enums ，是一种用来代表一个固定数量的常量的特殊类。 * 声明一个枚举类型需要使用关键字 enum ： ``` dart enum Color { red, green, blue } ``` * 在枚举中每个值都有一个 index getter 方法，它返回一个在枚举声明中从 0 开始的位置。例如，第一个值索引值为 0 ，第二个值索引值为 1 。 ``` dart assert(Color.red.index == 0); assert(Color.green.index == 1); assert(Color.blue.index == 2); ``` * 要得到枚举列表的所有值，可使用枚举的 values 常量。 ``` dart List colors = Color.values; assert(colors[2] == Color.blue); ``` * 你可以在 switch 语句 中使用枚举。如果 e 在 switch (e) 是显式类型的枚举，那么如果你不处理所有的枚举值将会弹出警告： ``` dart ***枚举类型有以下限制*** * 你不能在子类中混合或实现一个枚举。 * 你不能显式实例化一个枚举。 enum Color { red, green, blue } // ... Color aColor = Color.blue; switch (aColor) { case Color.red: print('Red as roses!'); break; case Color.green: print('Green as grass!'); break; default: // Without this, you see a WARNING. print(aColor); // 'Color.blue' } ``` 5. 为类添加特征：mixins * mixins 是一种多类层次结构的类的代码重用。 * 要使用 mixins ，在 with 关键字后面跟一个或多个 mixin 的名字。下面的例子显示了两个使用mixins的类： ``` dart class Musician extends Performer with Musical { // ... } class Maestro extends Person with Musical, Aggressive, Demented { Maestro(String maestroName) { name = maestroName; canConduct = true; } } ``` * 要实现 mixin ，就创建一个继承 Object 类的子类，不声明任何构造函数，不调用 super 。例如： ``` dart abstract class Musical { bool canPlayPiano = false; bool canCompose = false; bool canConduct = false; void entertainMe() { if (canPlayPiano) { print('Playing piano'); } else if (canConduct) { print('Waving hands'); } else { print('Humming to self'); } } } ``` 6. 类的变量和方法 * 使用 static 关键字来实现类变量和类方法。 * 只有当静态变量被使用时才被初始化。 * 静态变量, 静态变量（类变量）对于类状态和常数是有用的： ``` dart class Color { static const red = const Color('red'); // 一个恒定的静态变量 final String name; // 一个实例变量。 const Color(this.name); // 一个恒定的构造函数。 } main() { assert(Color.red.name == 'red'); } ``` * 静态方法, 静态方法（类方法）不在一个实例上进行操作，因而不必访问 this 。例如： ``` dart import 'dart:math'; class Point { num x; num y; Point(this.x, this.y); static num distanceBetween(Point a, Point b) { var dx = a.x - b.x; var dy = a.y - b.y; return sqrt(dx * dx + dy * dy); } } main() { var a = new Point(2, 2); var b = new Point(4, 4); var distance = Point.distanceBetween(a, b); assert(distance < 2.9 && distance > 2.8); } ``` * 注：考虑到使用高阶层的方法而不是静态方法，是为了常用或者广泛使用的工具和功能。 * 你可以将静态方法作为编译时常量。例如，你可以把静态方法作为一个参数传递给静态构造函数。 ## 抽象类 * 使用 abstract 修饰符来定义一个抽象类，该类不能被实例化。抽象类在定义接口的时候非常有用，实际上抽象中也包含一些实现。如果你想让你的抽象类被实例化，请定义一个 工厂构造函数 。 * 抽象类通常包含 抽象方法。下面是声明一个含有抽象方法的抽象类的例子： ``` dart // 这个类是抽象类，因此不能被实例化。 abstract class AbstractContainer { // ...定义构造函数，域，方法... void updateChildren(); // 抽象方法。 } ``` * 下面的类不是抽象类，因此它可以被实例化，即使定义了一个抽象方法： ``` dart class SpecializedContainer extends AbstractContainer { // ...定义更多构造函数，域，方法... void updateChildren() { // ...实现 updateChildren()... } // 抽象方法造成一个警告，但是不会阻止实例化。 void doSomething(); } ``` ## 类-隐式接口 * 每个类隐式的定义了一个接口，含有类的所有实例和它实现的所有接口。如果你想创建一个支持类 B 的 API 的类 A，但又不想继承类 B ，那么，类 A 应该实现类 B 的接口。 * 一个类实现一个或更多接口通过用 implements 子句声明，然后提供 API 接口要求。例如： ``` dart // 一个 person ，包含 greet() 的隐式接口。 class Person { // 在这个接口中，只有库中可见。 final _name; // 不在接口中，因为这是个构造函数。 Person(this._name); // 在这个接口中。 String greet(who) => 'Hello, $who. I am $_name.'; } // Person 接口的一个实现。 class Imposter implements Person { // 我们不得不定义它，但不用它。 final _name = ""; String greet(who) => 'Hi $who. Do you know who I am?'; } greetBob(Person person) => person.greet('bob'); main() { print(greetBob(new Person('kathy'))); print(greetBob(new Imposter())); } ``` * 这里是具体说明一个类实现多个接口的例子： ``` dart class Point implements Comparable, Location { // ... } ``` ## 类-扩展一个类 * 使用 extends 创建一个子类，同时 supper 将指向父类： ``` dart class Television { void turnOn() { _illuminateDisplay(); _activateIrSensor(); } // ... } class SmartTelevision extends Television { void turnOn() { super.turnOn(); _bootNetworkInterface(); _initializeMemory(); _upgradeApps(); } // ... } ``` * 子类可以重载实例方法， getters 方法， setters 方法。下面是个关于重写 Object 类的方法 noSuchMethod() 的例子,当代码企图用不存在的方法或实例变量时，这个方法会被调用。 ``` dart class A { // 如果你不重写 noSuchMethod 方法, 就用一个不存在的成员，会导致NoSuchMethodError 错误。 void noSuchMethod(Invocation mirror) { print('You tried to use a non-existent member:' + '${mirror.memberName}'); } } ``` * 你可以使用 @override 注释来表明你重写了一个成员。 ``` dart class A { @override void noSuchMethod(Invocation mirror) { // ... } } ``` * 如果你用 noSuchMethod() 实现每一个可能的 getter 方法，setter 方法和类的方法，那么你可以使用 @proxy 标注来避免警告。 ``` dart @proxy class A { void noSuchMethod(Invocation mirror) { // ... } } ``` ## 库和可见性 1. import,part,library指令可以帮助创建一个模块化的，可共享的代码库。库不仅提供了API，还提供隐私单元：以下划线（_）开头的标识符只对内部库可见。每个Dartapp就是一个库，即使它不使用库指令。 2. 库可以分布式使用包。见 Pub Package and Asset Manager 中有关pub(SDK中的一个包管理器）。 3. 使用库 * 使用 import 来指定如何从一个库命名空间用于其他库的范围。 * 使用 import 来指定如何从一个库命名空间用于其他库的范围。 ``` dart import 'dart:html'; ``` * 唯一需要 import 的参数是一个指向库的 URI。对于内置库，URI中具有特殊dart:scheme。对于其他库，你可以使用文件系统路径或package:scheme。包 package：scheme specifies libraries ，如pub工具提供的软件包管理器库。例如： ``` dart import 'dart:io'; import 'package:mylib/mylib.dart'; import 'package:utils/utils.dart'; ``` 4. 指定库前缀 * 如果导入两个库是有冲突的标识符，那么你可以指定一个或两个库的前缀。例如，如果 library1 和 library2 都有一个元素类，那么你可能有这样的代码： ``` dart import 'package:lib1/lib1.dart'; import 'package:lib2/lib2.dart' as lib2; // ... var element1 = new Element(); // 使用lib1里的元素 var element2 = new lib2.Element(); // 使用lib2里的元素 ``` 5. 导入部分库 * 如果想使用的库一部分，你可以选择性导入库。例如： ``` dart // 只导入foo库 import 'package:lib1/lib1.dart' show foo; //导入所有除了foo import 'package:lib2/lib2.dart' hide foo; ``` 6. 延迟加载库 * 延迟(deferred)加载（也称为延迟(lazy)加载）允许应用程序按需加载库。下面是当你可能会使用延迟加载某些情况： * 为了减少应用程序的初始启动时间； * 执行A / B测试-尝试的算法的替代实施方式中； * 加载很少使用的功能，例如可选的屏幕和对话框。 * 为了延迟加载一个库，你必须使用 deferred as 先导入它。 ``` dart import 'package:deferred/hello.dart' deferred as hello; ``` * 当需要库时，使用该库的调用标识符调用 LoadLibrary（）。 ``` dart greet() async { await hello.loadLibrary(); hello.printGreeting(); } ``` * 在前面的代码，在库加载好之前，await关键字都是暂停执行的。有关 async 和 await 见 asynchrony support 的更多信息。 您可以在一个库调用 LoadLibrary（） 多次都没有问题。该库也只被加载一次。 * 当您使用延迟加载，请记住以下内容： * 延迟库的常量在其作为导入文件时不是常量。记住，这些常量不存在，直到迟库被加载完成。 * 你不能在导入文件中使用延迟库常量的类型。相反，考虑将接口类型移到同时由延迟库和导入文件导入的库。 * Dart隐含调用LoadLibrary（）插入到定义deferred as namespace。在调用LoadLibrary（）函数返回一个Future。 7. 库的实现 * 用 library 来来命名库，用part来指定库中的其他文件。 注意：不必在应用程序中（具有顶级main（）函数的文件）使用library，但这样做可以让你在多个文件中执行应用程序。 8. 声明库 * 利用library identifier（库标识符）指定当前库的名称： ``` dart // 声明库，名ballgame library ballgame; // 导入html库 import 'dart:html'; // ...代码从这里开始... ``` 9. 关联文件与库 * 添加实现文件，把part fileUri放在有库的文件，其中fileURI是实现文件的路径。然后在实现文件中，添加部分标识符（part of identifier），其中标识符是库的名称。下面的示例使用的一部分，在三个文件来实现部分库。 * 第一个文件，ballgame.dart，声明球赛库，导入其他需要的库，并指定ball.dart和util.dart是此库的部分： ``` dart library ballgame; import 'dart:html'; // ...其他导入在这里... part 'ball.dart'; part 'util.dart'; // ...代码从这里开始... ``` * 第二个文件ball.dart，实现了球赛库的一部分： ``` dart part of ballgame; // ...代码从这里开始... ``` * 第三个文件，util.dart，实现了球赛库的其余部分： ``` dart part of ballgame; // ...Code goes here... ``` 10. 重新导出库(Re-exporting libraries) *可以通过重新导出部分库或者全部库来组合或重新打包库。例如，你可能有实现为一组较小的库集成为一个较大库。或者你可以创建一个库，提供了从另一个库方法的子集。 ``` dart // In french.dart: library french; hello() => print('Bonjour!'); goodbye() => print('Au Revoir!'); // In togo.dart: library togo; import 'french.dart'; export 'french.dart' show hello; // In another .dart file: import 'togo.dart'; void main() { hello(); //print bonjour goodbye(); //FAIL } ``` ## 异步的支持 1. Dart 添加了一些新的语言特性用于支持异步编程。最通常使用的特性是 async 方法和 await 表达式。Dart 库大多方法返回 Future 和 Stream 对象。这些方法是异步的：它们在设置一个可能的耗时操作（比如 I/O 操作）之后返回，而无需等待操作完成 2. 当你需要使用 Future 来表示一个值时，你有两个选择。 * 使用 async 和 await * 使用 Future API 3. 同样的，当你需要从 Stream 获取值的时候，你有两个选择。 使用 async 和一个异步的 for 循环 (await for) 使用 Stream API 4. 使用 async 和 await 的代码是异步的，不过它看起来很像同步的代码。比如这里有一段使用 await 等待一个异步函数结果的代码： ``` dart await lookUpVersion() ``` 5. 要使用 await，代码必须用 await 标记 ``` dart checkVersion() async { var version = await lookUpVersion(); if (version == expectedVersion) { // Do something. } else { // Do something else. } } ``` 6. 你可以使用 try, catch, 和 finally 来处理错误并精简使用了 await 的代码。 ``` dart try { server = await HttpServer.bind(InternetAddress.LOOPBACK_IP_V4, 4044); } catch (e) { // React to inability to bind to the port... } ``` 7. 声明异步函数 * 一个异步函数是一个由 async 修饰符标记的函数。虽然一个异步函数可能在操作上比较耗时，但是它可以立即返回-在任何方法体执行之前。 ``` dart checkVersion() async { // ... } lookUpVersion() async => /* ... */; ``` * 在函数中添加关键字 async 使得它返回一个 Future，比如，考虑一下这个同步函数，它将返回一个字符串。 * String lookUpVersionSync() => '1.0.0'; * 如果你想更改它成为异步方法-因为在以后的实现中将会非常耗时-它的返回值是一个 Future 。 * Future lookUpVersion() async => '1.0.0'; * 请注意函数体不需要使用 Future API，如果必要的话 Dart 将会自己创建 Future 对象 8. 使用带 future 的 await 表达式 * 一个 await表达式具有以下形式 ``` dart await expression ``` * 在异步方法中你可以使用 await 多次。比如，下列代码为了得到函数的结果一共等待了三次。 ``` dart var entrypoint = await findEntrypoint(); var exitCode = await runExecutable(entrypoint, args); await flushThenExit(exitCode); ``` * 在 await 表达式中， 表达式 的值通常是一个 Future 对象；如果不是，那么这个值会自动转为 Future。这个 Future 对象表明了表达式应该返回一个对象。await 表达式 的值就是返回的一个对象。在对象可用之前，await 表达式将会一直处于暂停状态。 * 如果 await 没有起作用，请确认它是一个异步方法。比如，在你的 main() 函数里面使用await，main() 的函数体必须被 async 标记： ``` dart main() async { checkVersion(); print('In main: version is ${await lookUpVersion()}'); } ``` 9. 结合 streams 使用异步循环 * 一个异步循环具有以下形式： ``` dart await for (variable declaration in expression) { // Executes each time the stream emits a value. } ``` * 表达式 的值必须有Stream 类型（流类型）。执行过程如下： * 在 stream 发出一个值之前等待 * 执行 for 循环的主体，把变量设置为发出的值。 * 重复 1 和 2，直到 Stream 关闭 * 如果要停止监听 stream ，你可以使用 break 或者 return 语句，跳出循环并取消来自 stream 的订阅 。 * 如果一个异步 for 循环没有正常运行，请确认它是一个异步方法。 比如，在应用的 main() 方法中使用异步的 for 循环时，main() 的方法体必须被 async 标记。 ``` dart main() async { ... await for (var request in requestServer) { handleRequest(request); } ... } ``` 更多关于异步编程的信息，请看 dart:async 库部分的介绍。你也可以看文章 [Dart Language Asynchrony Support: Phase 1 ](https://www.dartlang.org/articles/await-async/) ================================================ FILE: docs/docker/docker-compose.md ================================================ # docker和docker-compose 配置 mysql mssql mongodb redis nginx jenkins 环境 ## 磁盘挂载 ``` shell fdisk -l #查看磁盘列表 mkfs.ext4 /dev/vdb #格式化磁盘 mount /dev/vdb /data #挂载磁盘在/data echo '/dev/vdb /data ext4 defaults,nofail 0 1'>> /etc/fstab # 启动服务器自动挂载 mount -a #校验自动挂载脚本 df -h #查看磁盘挂载后信息 ``` ## docker ### 安装 docker ``` shell yum update #更新系统包 yum install -y yum-utils device-mapper-persistent-data lvm2 #安装yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo #为yum源添加docker仓库位置 yum install docker-ce #安装docker systemctl enable docker #设置开机自动启动 systemctl start docker #启动docker systemctl stop docker #暂停docker mv /var/lib/docker /data/docker # 修改Docker镜像的存放位置 ln -s /data/docker /var/lib/docker #建立软连接 echo '{ "registry-mirrors": [ "https://dockerhub.azk8s.cn", "https://hub-mirror.c.163.com", "https://registry.docker-cn.com" ] } '>> /etc/docker/daemon.json # 镜像下载代理 ``` ### 拉取 Java 镜像 ``` shell docker pull java ``` ### 拉取SqlServer镜像 ``` shell docker pull microsoft/mssql-server-linux # 拉取SqlServer镜像 docker run -p 1433:1433 --name mssql \ # run 运行容器 -p 将容器的1433端口映射到主机的1433端口 --name 容器运行的名字 --restart=always \ # 挂断自动重新启动 -v /data/sqlserver:/var/opt/mssql \ # 挂载mssql文件夹到主机 -e ACCEPT_EULA=Y \ # 同意协议 -e MSSQL_SA_PASSWORD=mssql-MSSQL \ # 初始化sa密码 -u root \ # 指定容器为root运行 -d microsoft/mssql-server-linux # -d 后台运行 ``` ### 拉取 MySql 镜像 ``` shell docker pull mysql #拉取 MySql docker run -p 3306:3306 --name mysql \ # run 运行容器 -p 将容器的3306端口映射到主机的3306端口 --name 容器运行的名字 --restart=always \ # 挂断自动重新启动 -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /data/mysql/log:/var/log/mysql \ # 将日志文件夹挂载到主机 -v /data/mysql/data:/var/lib/mysql \ # 将数据文件夹挂载到主机 -v /data/mysql/mysql-files:/var/lib/mysql-files \ # 将数据文件夹挂载到主机 -v /data/mysql/conf:/etc/mysql \ # 将配置文件夹挂在到主机 -e MYSQL_ROOT_PASSWORD=xiujingmysql. \ # 初始化root用户的密码 -d mysql # -d 后台运行 docker exec -it mysql /bin/bash # 进入Docker容器内部的bash ``` ### 拉取 Mongodb 镜像 ``` shell docker pull mongo #拉取 mongodb docker run -p 27017:27017 --name mongo \ # run 运行容器 -p 将容器的27017端口映射到主机的27017端口 --name 容器运行的名字 --restart=always \ # 挂断自动重新启动 -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /data/mongodb/db:/data/db \ # 将数据文件夹挂载到主机 -v /data/mongodb/configdb:/data/configdb \ # 将数据库配置文件挂载到主机 -v /data/mongodb/initdb:/docker-entrypoint-initdb.d # 通过/docker-entrypoint-initdb.d/将更复杂的用户设置显式留给用户 当容器首次启动时它会执行/docker-entrypoint-initdb.d 目录下的sh 和js脚本 。 以脚本字母顺序执行 -e MONGO_INITDB_ROOT_USERNAME=admin \ # 设置admin数据库账户名称 如果使用了此项，则不需要 --auth 参数 -e MONGO_INITDB_ROOT_PASSWORD=admin \ # 设置admin数据库账户密码 如果使用了此项，则不需要 --auth 参数 -d mongo \ # -d 后台运行 --auth # --auth 需要密码才能访问容器服务 docker exec -it mongo mongo admin # 进入mongo db.createUser({ user:'admin',pwd:'123456',roles:[ { role:'userAdminAnyDatabase', db: 'admin'}]}); #创建一个名为 admin，密码为 123456 的用户。 db.auth('admin', '123456') # 尝试使用上面创建的用户信息进行连接。 ``` ### 拉取 Redis 镜像 ``` shell docker pull redis #拉取 redis docker run -p 6379:6379 --name redis \ # run 运行容器 -p 将容器的6379端口映射到主机的6379端口 --name 容器运行的名字 --restart=always \ # 挂断自动重新启动 -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /data/redis:/data \ # 将数据文件夹挂载到主机 -d redis \ # -d 后台运行 redis-server --appendonly yes \ # 在容器执行redis-server启动命令，并打开redis持久化配置 --requirepass "123456" # 设置密码123456 ``` ### 拉取 Nginx 镜像 ``` shell docker pull nginx #拉取 nginx docker run -p 80:80 -p 443:443 --name nginx -d nginx # 运行容器 docker container cp nginx:/etc/nginx /data/nginx/ #拷贝容器配置 docker rm -f nginx # 删除容器 ``` nginx 配置文件 ``` conf user nginx; worker_processes 1; error_log /var/log/nginx/error_log.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; #gzip on; gzip on; #开启gzip gzip_disable "msie6"; #IE6不使用gzip gzip_vary on; #设置为on会在Header里增加 "Vary: Accept-Encoding" gzip_proxied any; #代理结果数据的压缩 gzip_comp_level 6; #gzip压缩比（1~9），越小压缩效果越差，但是越大处理越慢，所以一般取中间值 gzip_buffers 16 8k; #获取多少内存用于缓存压缩结果 gzip_http_version 1.1; #识别http协议的版本 gzip_min_length 1k; #设置允许压缩的页面最小字节数，超过1k的文件会被压缩 gzip_types application/javascript text/css; #对特定的MIME类型生效,js和css文件会被压缩 include /etc/nginx/conf.d/*.conf; server { #nginx同时开启http和https listen 80 default backlog=2048; listen 443 ssl; server_name ysf.djtlpay.com; ssl_certificate /ssl/1_ysf.djtlpay.com_bundle.crt; ssl_certificate_key /ssl/2_ysf.djtlpay.com.key; location / { root /usr/share/nginx/html; index index.html index.htm; } } } ``` 运行 nginx ``` shell docker run -p 80:80 -p 443:443 --name nginx \ # run 运行容器 -p 将容器的80,443端口映射到主机的80,443端口 --name 容器运行的名字 --restart=always \ # 挂断自动重新启动 -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /data/nginx/html:/usr/share/nginx/html \ # nginx 静态资源 -v /data/nginx/logs:/var/log/nginx \ # 将日志文件夹挂载到主机 -v /data/nginx/conf:/etc/nginx \ # 将配置文件夹挂在到主机 -v /data/nginx/conf/ssl:/ssl \ # 将证书文件夹挂在到主机 -d nginx # ``` ### 拉取Jenkins镜像： ```shell docker pull jenkins/jenkins:lts # 拉取 jenkins docker run -p 8080:8080 -p 50000:50000 --name jenkins \ # run 运行容器 -p 将容器的8080,50000端口映射到主机的8080,50000端口 --name 容器运行的名字 --restart=always \ # 挂断自动重新启动 -u root \ # 运行的用户为root -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /data/jenkins_home:/var/jenkins_home \ # 将jenkins_home文件夹挂在到主机 -e JAVA_OPTS=-Duser.timezone=Asia/Shanghai \ #设置jenkins运行环境时区 -d jenkins/jenkins:lts # -d 后台运行 ``` ### 拉取MinIO镜像 ```shell docker pull minio/minio # 拉取MinIO镜像 docker run -p 9000:9000 --name minio \ # run 运行容器 -p 将容器的9000,9000端口映射到主机的9000,9000端口 --name 容器运行的名 --restart=always \ # 挂断自动重新启动 -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /data/minio/data:/data \ # 将data文件夹挂在到主机 -v /data/minio/config:/root/.minio \ # 将配置文件夹挂在到主机 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \ # 设置MINIO_ACCESS_KEY的值 -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \ # 设置MINIO_SECRET_KEY值 -d minio/minio server /data # -d 后台运行 server /data 导出/data目录 ``` ### 拉取Portainer镜像 ``` shell docker pull portainer/portainer # 拉取MinIO镜像 docker run -p 8001:8000 -p 9001:9000 --name portainer \ # run 运行容器 -p 将容器的8000,9000端口映射到主机的8000,9000端口 --name 容器运行的名 --restart=always \ # 挂断自动重新启动 -v /etc/localtime:/etc/localtime \ # 将主机本地时间夹挂在到容器 -v /var/run/docker.sock:/var/run/docker.sock \ # 将docker.sock文件夹挂在到主机 -v /data/portainer/data:/data \ # 将配置文件夹挂在到主机 -d portainer/portainer portainer # -d 后台运行 ``` ### 拉取Phabricator镜像 ``` shell docker pull redpointgames/phabricator # 拉取Phabricator镜像 docker run \ --rm -p 80:80 -p 443:443 -p 2222:22 \ --env PHABRICATOR_HOST=localhost \ --env MYSQL_HOST=localhost \ --env MYSQL_USER=root \ --env MYSQL_PASS=root \ --env PHABRICATOR_REPOSITORY_PATH=/repos \ -v /data/phabricator/repos:/repos \ --name phabricator \ -d redpointgames/phabricator ``` [redpointgames/phabricator go to github](https://github.com/RedpointGames/phabricator) [bitnami-docker-phabricator go to github](https://github.com/bitnami/bitnami-docker-phabricator) ### Docker 开启远程API * 用vi编辑器修改docker.service文件 ``` shell vi /usr/lib/systemd/system/docker.service #需要修改的部分： ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock #修改后的部分： ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock ``` ### Docker 常用命令 ``` shell systemctl start docker #启动docker systemctl enable docker #将docker服务设为开机启动 systemctl stop docker #停止容器 systemctl restart docker #重启docker服务 docker images # 列出镜像 docker rmi --name # 删除镜像 -f 强制删除 docker ps # 列出容器 -a 所有 docker start --name # 启动容器 docker stop --name # 停止容器 docker restart --name # 重启docker容器 docker rm --name # 删除容器 -f 强制删除 docker stats -a # 查看所有容器情况 docker system df # 查看Docker磁盘使用情况 docker exec -it --name /bin/bash #进入Docker容器内部的bash docker cp 主机文件 容器名称:容器路径 #复制文件到docker容器中 docker logs --name #查看docker镜像日志 docker rm $(docker ps -a -q) # 删除所有容器 -f 强制删除 docker rmi $(docker images -a -q) # 删除所有镜像 -f 强制删除 docker rm -f `docker ps -a | grep -vE 'mysql|nginx|redis|jenkins' | awk '{print $1}'` # 删除mysql|nginx|redis|jenkins非容器 -f 强制删除 docker rmi -f `docker images | grep none | awk '{print $3}'` # 删除镜像none镜像 -f 强制删除 ``` ## docker-compose ### 安装 docker-compose ```shell # 下载Docker Compose curl -L https://get.daocloud.io/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose # 修改该文件的权限为可执行 chmod +x /usr/local/bin/docker-compose # 查看是否已经安装成功 docker-compose --version ``` ### 使用Docker Compose的步骤 * 使用Dockerfile定义应用程序环境，一般需要修改初始镜像行为时才需要使用； * 使用docker-compose.yml定义需要部署的应用程序服务，以便执行脚本一次性部署； * 使用docker-compose up命令将所有应用服务一次性部署起来。 ### docker-compose.yml常用命令 ``` shell # 指定运行的镜像名称 image: name:version # 配置容器名称 container_name: name # 指定宿主机和容器的端口映射 ports: - 3306:3306 # 将宿主机的文件或目录挂载到容器中 volumes: - /etc/localtime:/etc/localtime - /data/mysql/log:/var/log/mysql - /data/mysql/data:/var/lib/mysql - /data/mysql/conf:/etc/mysql - /data/mysql/mysql-files:/var/lib/mysql-files # 配置环境变量 environment: - MYSQL_ROOT_PASSWORD=xiujingmysql. # 连接其他容器的服务 links: - db:database #可以以database为域名访问服务名称为db的容器 # 挂断自动重新启动 restart: always # 指定容器执行命令 command: redis-server --requirepass xiujingredis. ``` ### Docker Compose常用命令 ```shell # 构建、创建、启动相关容器 docker-compose up -d # -d表示在后台运行 # 停止所有相关容器 docker-compose stop # 删除容器文件 docker-compose rm -f # -f 强制删除 # 重启容器 docker-compose restart # 列出所有容器信息 docker-compose ps # 查看容器日志 docker-compose logs ``` ### 使用Docker Compose 部署应用 编写docker-compose.yml文件 ``` yml version: '3' services: # 指定服务名称 nginx: # 指定服务使用的镜像 image: nginx # 指定容器名称 container_name: nginx # 指定服务运行的端口 ports: - 80:80 - 443:443 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime # 挂断自动重新启动 restart: always # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 # 指定服务名称 sqlserver: # 指定服务使用的镜像 image: mcr.microsoft.com/mssql/server # 指定容器名称 container_name: sqlserver # 指定服务运行的端口 ports: - "1433" # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/sqlserver:/var/opt/mssql # 挂断自动重新启动 restart: always environment: - TZ=Asia/Shanghai - SA_PASSWORD=mssql-MSSQL - ACCEPT_EULA=Y # 指定容器运行的用户为root user: root # 指定服务名称 mysql: # 指定服务使用的镜像 image: mysql # 指定容器名称 container_name: mysql # 指定服务运行的端口 ports: - 3306:3306 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/mysql/log:/var/log/mysql - /data/mysql/data:/var/lib/mysql - /data/mysql/mysql-files:/var/lib/mysql-files - /data/mysql/conf:/etc/mysql # 挂断自动重新启动 restart: always # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 - MYSQL_ROOT_PASSWORD=xiujingmysql. # 设置root密码 # 指定容器运行的用户为root user: root # 指定服务名称 redis: # 指定服务使用的镜像 image: redis # 指定容器名称 container_name: redis # 指定服务运行的端口 ports: - 6379:6379 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/redis:/data - /data/redis/redis.conf:/etc/redis.conf # 挂断自动重新启动 restart: always # 指定容器执行命令 command: redis-server /etc/redis.conf --requirepass xiujingredis. --appendonly yes # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 # 指定服务名称 mongo: # 指定服务使用的镜像 image: mongo # 指定容器名称 container_name: mongo # 指定服务运行的端口 ports: - 27017:27017 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/mongodb/db:/data/db - /data/mongodb/configdb:/data/configdb - /data/mongodb/initdb:/docker-entrypoint-initdb.d # 挂断自动重新启动 restart: always # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 - AUTH=yes - MONGO_INITDB_ROOT_USERNAME=admin - MONGO_INITDB_ROOT_PASSWORD=admin # 指定服务名称 jenkins: # 指定服务使用的镜像 image: jenkins # 指定容器名称 container_name: jenkins # 指定服务运行的端口 ports: - 8080:8080 - 50000:50000 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/jenkins_home:/var/jenkins_home # 挂断自动重新启动 restart: always # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 - JAVA_OPTS=-Duser.timezone=Asia/Shanghai # 指定容器运行的用户为root user: root # 指定服务名称 minio: # 指定服务使用的镜像 image: minio # 指定容器名称 container_name: minio # 指定服务运行的端口 ports: - 9000:9000 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/minio/data:/data - /data/minio/config:/root/.minio # 挂断自动重新启动 restart: always # 指定容器执行命令 command: server /data # 指定服务名称 portainer : # 指定服务使用的镜像 image: portainer # 指定容器名称 container_name: portainer # 指定服务运行的端口 ports: - 8001:8000 - 9001:9000 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /var/run/docker.sock:/var/run/docker.sock - /data/portainer/data:/data # 挂断自动重新启动 restart: always ``` 运行Docker Compose命令启动所有服务 ``` shell docker-compose up -d ``` ================================================ FILE: docs/docker/docker-jenkins.md ================================================ # Docker部署Jenkins ## Jenkins简介 Jenkins是开源CI&CD软件领导者，提供超过1000个插件来支持构建、部署、自动化，满足任何项目的需要。我们可以用Jenkins来构建和部署我们的项目，比如说从我们的代码仓库获取代码，然后将我们的代码打包成可执行的文件，之后通过远程的ssh工具执行脚本来运行我们的项目。 ## Jenkins的安装及配置 ### Docker环境下的安装 * 下载Jenkins的Docker镜像： ```shell docker pull jenkins/jenkins:lts ``` * 在Docker容器中运行Jenkins： ``` shell docker run -p 8080:8080 -p 50000:50000 --name jenkins \ -u root \ -v /etc/localtime:/etc/localtime \ -v /data/jenkins_home:/var/jenkins_home \ -e JAVA_OPTS=-Duser.timezone=Asia/Shanghai \ -d jenkins/jenkins:lts ``` * echo 'Asia/Shanghai' >/etc/timezone ### Jenkins的配置 * 运行成功后访问该地址登录Jenkins，第一次登录需要输入管理员密码：http://localhost:8080/ ![jenkins](../img/docker/jenkins-1.png) * 使用管理员密码进行登录，可以使用以下命令从容器启动日志中获取管理密码： ``` shell docker logs jenkins ``` ![jenkins](../img/docker/jenkins-2.png) * 选择安装插件方式，这里我们直接安装推荐的插件： ![jenkins](../img/docker/jenkins-3.png) * 进入插件安装界面，联网等待插件安装： ![jenkins](../img/docker/jenkins-4.png) * 安装完成后，创建管理员账号： ![jenkins](../img/docker/jenkins-5.png) * 进行实例配置，配置Jenkins的URL： ![jenkins](../img/docker/jenkins-6.png) * 点击系统管理->插件管理，进行一些自定义的插件安装： ![jenkins](../img/docker/jenkins-7.png) * 确保以下插件被正确安装： 1. 根据角色管理权限的插件：Role-based Authorization Strategy 2. 远程使用ssh的插件：SSH plugin * 通过系统管理->全局工具配置来进行全局工具的配置，比如maven的配置： ![jenkins](../img/docker/jenkins-8.png) * 新增maven的安装配置： ![jenkins](../img/docker/jenkins-9.png) * 在系统管理->系统配置中添加全局ssh的配置，这样Jenkins使用ssh就可以执行远程的linux脚本了： ![jenkins](../img/docker/jenkins-10.png) ## 角色权限管理 我们可以使用Jenkins的角色管理插件来管理Jenkins的用户，比如我们可以给管理员赋予所有权限，运维人员赋予执行任务的相关权限，其他人员只赋予查看权限。 * 在系统管理->全局安全配置中启用基于角色的权限管理： ![jenkins](../img/docker/jenkins-11.png) * 进入系统管理->Manage and Assign Roles界面： ![jenkins](../img/docker/jenkins-12.png) * 添加角色与权限的关系： ![jenkins](../img/docker/jenkins-13.png) * 给用户分配角色： ![jenkins](../img/docker/jenkins-14.png) ================================================ FILE: docs/docker/docker-phabricator.md ================================================ # Docker部署Phabricator ## 安装 1. 拉取镜像启动容器 ``` docker run --name phabricator \ -p 443:443 \ -p 80:80 \ -v /etc/localtime:/etc/localtime \ -v /data/phabricator/conf/local:/var/www/phabricator/conf/local \ -v /data/phabricator/repos:/var/repo \ -d phabricator/phabricator ``` 2. 进入容器内部配置 ``` shell docker exec -it phabricator bash cd phabricator ``` 3. 配置Mysql ## ``` shell ./bin/config set mysql.host localhost ./bin/config set mysql.port 3306 ./bin/config set mysql.user root ./bin/config set mysql.pass root ./bin/storage upgrade #设置数据库 ``` cmkmxlgplozydhie ## 页面引导 1. 创建账号 ![pha](../img/docker/pha-1.jpg) 2. 九项需要配置处理 ![pha](../img/docker/pha-2.jpg) 3. 配置身份验证提供程序 ![pha](../img/docker/pha-3.jpg) ![pha](../img/docker/pha-4.jpg) ![pha](../img/docker/pha-5.jpg) ![pha](../img/docker/pha-6.jpg) ![pha](../img/docker/pha-7.jpg) ![pha](../img/docker/pha-8.jpg) ```shell docker exec -it phabricator bash cd phabricator ./bin/auth lock #锁定身份验证提供者配置 ``` ![pha](../img/docker/pha-9.jpg) 4. 配置Email ![pha](../img/docker/pha-10.jpg) ``` shell vi cluster.mailers ``` ``` json [{ "key":"qq-email", "type":"smtp", "options":{ "host":"smtp.qq.com", "port":465, "user":"qhhjsj.qq.com", "password":"cmkmxlgplozydhie", "protocol":"ssl" } }] ``` ``` shell ./bin/config set cluster.mailers --stdin < cluster.mailers ./bin/config set metamta.default-address qhhjsj@qq.com ./bin/config set metamta.can-send-as-user false ./bin/mail show-outbound --id 1 # 查看邮件发送文本 ``` ================================================ FILE: docs/docker/docker.md ================================================ # Docker ## Docker简介 * Docker是开源应用容器引擎，轻量级容器技术。 * 基于Go语言，并遵循Apache2.0协议开源 * Docker可以让开发者打包他们的应用以及依赖包到一个轻量级、可移植的容器中，然后发布到任何流行的Linux系统上，也可以实现虚拟化 * 容器完全使用沙箱技术，相互之间不会有任何接口 * 类似于虚拟机技术(vmware、vitural)，但docker直接运行在操作系统(Linux)上，而不是运行在虚拟机中，速度快，性能开销极低 白话文，简介就是: >Docker支持将软件编译成一个镜像，然后在镜像中各种软件做好配置，将镜像发布出去，其他使用者可以直接使用这个镜像。 运行中的这个镜像称为容器，容器启动是非常快速的。类似windows里面的ghost操 作系统，安装好后什么都有了。 ## 什么是Docker Docker 是一个开源的应用容器引擎，基于Go语言，诞生于2013年初，最初发起者是dotCloud公司，开发者可以打包应用到一个轻量级、可移植的容器中，然后发布到主流Linux系统上运行。 为什么用Docker * 持续交付和部署：使用Docker可以通过定制应用镜像来实现持续集成，持续交付，部署。开发人员构建后的镜像，结合持续集成系统进行集成测试，而运维人员则可以在生产环境中快速部署该镜像，也可以结合持续部署系统进行自动部署。 * 更高效的资源利用：Docker是基于内核级的虚拟化，可以实现更高效的性能，同时对资源的额外需求很低，相比传统虚拟机方式，相同配置的主机能够运行更多的应用。 更轻松的迁移和扩展：Docker容器几乎可以在任何平台上运行，同时支持主流的操作系统发行版本。 * 更快速的启动时间：传统的虚拟机技术启动应用服务往往需要数分钟，而Docker容器应用，由于直接运行于宿主内核，无需启动完整的操作系统，因此可以做到妙级，甚至毫秒级的启动时间，大大的节约了开发，测试，部署的时间。 Docker与传统虚拟机差异 传统虚拟化方式 ![传统虚拟化方式](./../img/docker/docker1.png "传统虚拟化方式") Docker虚拟化方式 ![Docker虚拟化方式](./../img/docker/docker2.png "Docker虚拟化方式") 传统虚拟化是在硬件层面实现虚拟化，需要有额外的虚拟机管理应用和虚拟机操作系统层，而Docker容器是在操作系统层面实现虚拟化，直接复用本地主机操作系统，更加轻量级。 ## 核心概念 * Docker镜像：类似于虚拟机里的镜像，是一个只读的模板，一个独立的文件系统，使用镜像可以创建容器，可以理解为镜像是容器的基石。 * Docker容器：是由Docker镜像创建的运行实例，类似于轻量级的沙箱，每个容器之间都是相互隔离的。支持的操作有启动，停止，删除等。 * docker客户端(Client)：客户端通过命令行或其他工具使用Docker API(https://docs.docker.com/reference/api/docker_remote_api)与Docker的守护进程进行通信 * docker主机(Host)：一个物理或虚拟的机器用来执行Docker守护进程和容器 * docker仓库(Registry)：Docker仓库用来存储镜像，可以理解为代码控制中的代码仓库，Docker Hub(https://hub.docker.com) 提供了庞大的镜像集合供使用 ## Docker安装及启停 1. 查看centos版本 ``` cmd Docker 要求 CentOS 系统的内核版本高于 3.10 通过命令：uname -r 查看当前centos版本，如版本不符，需升级系统版本 ``` 2. 升级软件包及内核(可选) ``` s yum update ``` 3. 安装docker ``` s yum install docker ``` 4. 启动docker ``` s systemctl start docker ``` 5. 将docker服务设为开机启动 ``` s systemtctl enable docker ``` 6. 停止docker ```s systemtctl stop docker ``` ## Docker常用命令及操作 docker镜像命令 ``` s docker search mysql ## 搜索镜像 docker pull mysql ## 下载镜像 下载命名为：docker pull 镜像名:tag，其中tag多为系统的版本，可选的，默认为least docker images ## 镜像列表 RESPOSITORY为镜像名 TAG为镜像版本，least代表最新 IMAGE_ID 为该镜像唯一ID CREATED 为该镜像创建时间 SIZE 为该镜像大小 docker rmi image-id ##删除指定镜像 docker rmi $(docker images -q) ##删除所有镜像 docker run --name container-name -d image-name ##根据镜像启动容器 -- name：为容器起一个名称 -d：detached，执行完这句命令后，控制台将不会阻塞，可以继续输入命令操作 image-name：要运行的镜像名称 docker ps ##查看运行中容器 CONTAINER ID：启动时生成的ID IMAGE：该容器使用的镜像 COMMAND：容器启动时执行的命令 CREATED：容器创建时间 STATUS：当前容器状态 PORTS：当前容器所使用的默认端口号 NAMES：启动时给容器设置的名称 docker stop container-name/container-id ## 停止运行中容器 docker ps -a ##查看所有的容器 docker start container-name/container-id ##启动容器 docker rm container-id ## 删除单个容器 docker rm $(docker ps -a -q ) ## 删除所有容器 docker run --name tomcat2 -d -p 8888:8080 tomcat ## 启动做端口映射的容器 docker logs container-id/container-name ##查看容器日志 docker port container-id ## 查看端口映射 docker exec -it container-id/container-name bash ##容器登录命令为 exit ##容器退出命令 ``` [更多命令可以参考](https://docs.docker.com/engine/reference/commandline/docker/ "更多命令可以参考") ## 镜像操作指令 * 获取镜像： docker pull centos (默认获取centos最新的镜像) docker pull centos:7 (获取指定标签镜像) * 查看本地镜像： docker images * 查看镜像详细信息： docker inspect centos:7 * 查看镜像历史： docker history centos:7 * 删除镜像： A:使用标签删除：docker rmi centos B:使用ID删除：docker rimi * 构建镜像： A:使用docker commit命令 B:使用Dockerfile构建 ## 使用docker commit 例：构建一个带有jdk的镜像 按照如下步骤操作 ``` s [root@localhost ~]# docker run -it centos:7 /bin/bash [root@060793baf536 /]# yum install wget [root@060793baf536 /]# wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm [root@060793baf536 /]# rpm -ivh jdk-8u131-linux-x64.rpm Preparing... ################################# [100%] Updating / installing... 1:jdk1.8.0_131-2000:1.8.0_131-fcs ################################# [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... [root@060793baf536 /]# exit [root@localhost ~]# docker commit 060793baf536 centos/jdk:2.0 ``` 通过docker images命令可以看到新增了centos/jdk 标签为2.0的镜像 ## 使用Dockerfile构建 实际使用中不推荐使用docker commit构建，应使用更灵活和强大的Dockerfile构建docker镜像，直接举例来认识Dockerfile。 例：构建一个带有jdk的centos7镜像 ``` s [root@localhost Dockerfile]# mkdir Dockerfile [root@localhost Dockerfile]# cd Dockerfile 编写Dockerfile： FROM centos:7 MAINTAINER Java-Road "Java-Road@qq.com" RUN mkdir /usr/local/jdk COPY jdk-8u171-linux-x64.rpm /usr/local/jdk/ RUN rpm -ivh /usr/local/jdk/jdk-8u171-linux-x64.rpm 执行如下指令： [root@localhost Dockerfile]# docker build -t centos/jdk . ``` 运行结果如下： ![运行结果如下：](./../img/docker/docker3.png "运行结果如下：") docker images可以看到新生成的centos/jdk镜像 ## 容器操作指令 * 创建启动容器： ``` s [root@localhost ~]# docker run centos:7 /bin/echo'hello world' 容器运行完后直接退出 ``` * 交互形式创建启动容器 ``` s [root@localhost ~]# docker run -it centos:7 /bin/bash [root@802e3623e566 /]# ps PID TTY TIME CMD 1 ? 00:00:00 bash 13 ? 00:00:00 ps [root@802e3623e566 /]# exit 执行exit才能退出容器 ``` * 守护状态运行容器 ``` s [root@localhost ~]# docker run -d centos:7 /bin/sh -c "while true; do echo hello world; sleep 1; done" ``` * 启动已有的容器： ``` s * docker start 容器ID # 例： [root@localhost ~]# docker start 802e3623e566 ``` * 停止运行的容器 ``` s * docker stop 容器ID # 例： [root@localhost ~]# docker stop 802e3623e566 ``` * 删除容器： ``` s [root@localhost ~]# docker stop 89566e38c7fb [root@localhost ~]# docker rm 89566e38c7fb ``` * 进入运行的容器： ``` s [root@localhost ~]# docker exec -it cbd8b1f35dcc /bin/bash ``` * 导出容器： ``` s # 导出容器cbd8b1f35dcc到centos_test.tar文件 [root@localhost ~]# docker export -o centos_test.tar cbd8b1f35dcc # 导出的tar文件可以在其他机器上，通过导入来重新运行 ``` * 导入容器： ``` s # 把导出的文件centos_test.tar通过docker import导入变成镜像 [root@localhost ~]# docker import centos_test.tar test/centos # 通过docker images命令可以看到增加了个test/centos镜像 ``` ## 实例：制作自己的 Docker 容器 >下面我以 [koa-demos](http://www.ruanyifeng.com/blog/2017/08/koa.html)项目为例，介绍怎么写 Dockerfile 文件，实现让用户在 Docker 容器里面运行 Koa 框架。 作为准备工作，请先下载源码。 ``` s $ git clone https://github.com/ruanyf/koa-demos.git $ cd koa-demos ``` ## Dockerfile 文件 首先，在项目的根目录下，新建一个文本文件.dockerignore，写入下面的内容。 ``` s .git node_modules npm-debug.log ``` 上面代码表示，这三个路径要排除，不要打包进入 image 文件。如果你没有路径要排除，这个文件可以不新建。 然后，在项目的根目录下，新建一个文本文件 Dockerfile，写入下面的内容。 ``` s FROM node:8.4 COPY . /app WORKDIR /app RUN npm install --registry=https://registry.npm.taobao.org EXPOSE 3000 ``` 上面代码一共五行，含义如下。 * FROM node:8.4：该 image 文件继承官方的 node image，冒号表示标签，这里标签是8.4，即8.4版本的 node。 * COPY . /app：将当前目录下的所有文件（除了.dockerignore排除的路径），都拷贝进入 image 文件的/app目录。 * WORKDIR /app：指定接下来的工作路径为/app。 * RUN npm install：在/app目录下，运行npm install命令安装依赖。注意，安装后所有的依赖，都将打包进入 image 文件。 * EXPOSE 3000：将容器 3000 端口暴露出来， 允许外部连接这个端口。 ## 创建 image 文件 有了 Dockerfile 文件以后，就可以使用docker image build命令创建 image 文件了。 ``` s $ docker image build -t koa-demo . # 或者 $ docker image build -t koa-demo:0.0.1 . ``` s 上面代码中，-t参数用来指定 image 文件的名字，后面还可以用冒号指定标签。如果不指定，默认的标签就是latest。最后的那个点表示 Dockerfile 文件所在的路径，上例是当前路径，所以是一个点。 如果运行成功，就可以看到新生成的 image 文件koa-demo了。 ``` s $ docker image ls ``` ## 生成容器 ``` s docker container run命令会从 image 文件生成容器。 $ docker container run -p 8000:3000 -it koa-demo /bin/bash # 或者 $ docker container run -p 8000:3000 -it koa-demo:0.0.1 /bin/bash ``` 上面命令的各个参数含义如下： * -p参数：容器的 3000 端口映射到本机的 8000 端口。 * -it参数：容器的 Shell 映射到当前的 Shell，然后你在本机窗口输入的命令，就会传入容器。 * koa-demo:0.0.1：image 文件的名字（如果有标签，还需要提供标签，默认是 latest 标签）。 * /bin/bash：容器启动以后，内部第一个执行的命令。这里是启动 Bash，保证用户可以使用 Shell。 如果一切正常，运行上面的命令以后，就会返回一个命令行提示符。 ``` s root@66d80f4aaf1e:/app# ``` 这表示你已经在容器里面了，返回的提示符就是容器内部的 Shell 提示符。执行下面的命令。 ```s root@66d80f4aaf1e:/app# node demos/01.js ``` 这时，Koa 框架已经运行起来了。打开本机的浏览器，访问 http://127.0.0.1:8000，网页显示"Not Found"，这是因为这个 demo 没有写路由。 这个例子中，Node 进程运行在 Docker 容器的虚拟环境里面，进程接触到的文件系统和网络接口都是虚拟的，与本机的文件系统和网络接口是隔离的，因此需要定义容器与物理机的端口映射（map）。 现在，在容器的命令行，按下 Ctrl + c 停止 Node 进程，然后按下 Ctrl + d （或者输入 exit）退出容器。此外，也可以用docker container kill终止容器运行。 ```s # 在本机的另一个终端窗口，查出容器的 ID $ docker container ls # 停止指定的容器运行 $ docker container kill [containerID] # 容器停止运行之后，并不会消失，用下面的命令删除容器文件。 # 查出容器的 ID $ docker container ls --all # 删除指定的容器文件 $ docker container rm [containerID] #也可以使用docker container run命令的--rm参数，在容器终止运行后自动删除容器文件。 $ docker container run --rm -p 8000:3000 -it koa-demo /bin/bash ``` ## CMD 命令 上一节的例子里面，容器启动以后，需要手动输入命令node demos/01.js。我们可以把这个命令写在 Dockerfile 里面，这样容器启动以后，这个命令就已经执行了，不用再手动输入了。 ``` s FROM node:8.4 COPY . /app WORKDIR /app RUN npm install --registry=https://registry.npm.taobao.org EXPOSE 3000 CMD node demos/01.js ``` 上面的 Dockerfile 里面，多了最后一行CMD node demos/01.js，它表示容器启动后自动执行node demos/01.js。 你可能会问，RUN命令与CMD命令的区别在哪里？简单说，RUN命令在 image 文件的构建阶段执行，执行结果都会打包进入 image 文件；CMD命令则是在容器启动后执行。另外，一个 Dockerfile 可以包含多个RUN命令，但是只能有一个CMD命令。 注意，指定了CMD命令以后，docker container run命令就不能附加命令了（比如前面的/bin/bash），否则它会覆盖CMD命令。现在，启动容器可以使用下面的命令。 ``` s $ docker container run --rm -p 8000:3000 -it koa-demo:0.0.1 ``` ## 发布 image 文件 容器运行成功后，就确认了 image 文件的有效性。这时，我们就可以考虑把 image 文件分享到网上，让其他人使用。 首先，去 hub.docker.com 或 cloud.docker.com 注册一个账户。然后，用下面的命令登录。 ``` s $ docker login # 接着，为本地的 image 标注用户名和版本。 $ docker image tag [imageName] [username]/[repository]:[tag] # 实例 $ docker image tag koa-demos:0.0.1 ruanyf/koa-demos:0.0.1 #也可以不标注用户名，重新构建一下 image 文件。 $ docker image build -t [username]/[repository]:[tag] . # 最后，发布 image 文件。 $ docker image push [username]/[repository]:[tag] ``` 发布成功以后，登录 hub.docker.com，就可以看到已经发布的 image 文件。 ## 其他有用的命令 docker 的主要用法就是上面这些，此外还有几个命令，也非常有用。 1. docker container start 前面的docker container run命令是新建容器，每运行一次，就会新建一个容器。同样的命令运行两次，就会生成两个一模一样的容器文件。如果希望重复使用容器，就要使用docker container start命令，它用来启动已经生成、已经停止运行的容器文件。 ``` s $ docker container start [containerID] ``` 2. docker container stop 前面的docker container kill命令终止容器运行，相当于向容器里面的主进程发出 SIGKILL 信号。而docker container stop命令也是用来终止容器运行，相当于向容器里面的主进程发出 SIGTERM 信号，然后过一段时间再发出 SIGKILL 信号。 ``` s $ bash container stop [containerID] ``` 这两个信号的差别是，应用程序收到 SIGTERM 信号以后，可以自行进行收尾清理工作，但也可以不理会这个信号。如果收到 SIGKILL 信号，就会强行立即终止，那些正在进行中的操作会全部丢失。 3. docker container logs docker container logs命令用来查看 docker 容器的输出，即容器里面 Shell 的标准输出。如果docker run命令运行容器的时候，没有使用-it参数，就要用这个命令查看输出。 ``` s $ docker container logs [containerID] ``` 4. docker container exec docker container exec命令用于进入一个正在运行的 docker 容器。如果docker run命令运行容器的时候，没有使用-it参数，就要用这个命令进入容器。一旦进入了容器，就可以在容器的 Shell 执行命令了。 ``` s $ docker container exec -it [containerID] /bin/bash ``` 5. docker container cp docker container cp命令用于从正在运行的 Docker 容器里面，将文件拷贝到本机。下面是拷贝到当前目录的写法。 ``` s $ docker container cp [containID]:[/path/to/file] . ``` ## Docker命令详解（run篇） 命令格式：docker run [OPTIONS] IMAGE [COMMAND] [ARG...] Usage: Run a command in a new container 中文意思为：通过run命令创建一个新的容器（container） ### 常用选项说明 -d, --detach=false， 指定容器运行于前台还是后台，默认为false -i, --interactive=false， 打开STDIN，用于控制台交互 -t, --tty=false， 分配tty设备，该可以支持终端登录，默认为false -u, --user=""， 指定容器的用户 -a, --attach=[]， 登录容器（必须是以docker run -d启动的容器） -w, --workdir=""， 指定容器的工作目录 -c, --cpu-shares=0， 设置容器CPU权重，在CPU共享场景使用 -e, --env=[]， 指定环境变量，容器中可以使用该环境变量 -m, --memory=""， 指定容器的内存上限 -P, --publish-all=false， 指定容器暴露的端口 -p, --publish=[]， 指定容器暴露的端口 -h, --hostname=""， 指定容器的主机名 -v, --volume=[]， 给容器挂载存储卷，挂载到容器的某个目录 --volumes-from=[]， 给容器挂载其他容器上的卷，挂载到容器的某个目录 --cap-add=[]， 添加权限，权限清单详见：http://linux.die.net/man/7/capabilities --cap-drop=[]， 删除权限，权限清单详见：http://linux.die.net/man/7/capabilities --cidfile=""， 运行容器后，在指定文件中写入容器PID值，一种典型的监控系统用法 --cpuset=""， 设置容器可以使用哪些CPU，此参数可以用来容器独占CPU --device=[]， 添加主机设备给容器，相当于设备直通 --dns=[]， 指定容器的dns服务器 --dns-search=[]， 指定容器的dns搜索域名，写入到容器的/etc/resolv.conf文件 --entrypoint=""， 覆盖image的入口点 --env-file=[]， 指定环境变量文件，文件格式为每行一个环境变量 --expose=[]， 指定容器暴露的端口，即修改镜像的暴露端口 --link=[]， 指定容器间的关联，使用其他容器的IP、env等信息 --lxc-conf=[]， 指定容器的配置文件，只有在指定--exec-driver=lxc时使用 --name=""， 指定容器名字，后续可以通过名字进行容器管理，links特性需要使用名字 --net="bridge"， 容器网络设置: bridge 使用docker daemon指定的网桥 host //容器使用主机的网络 container:NAME_or_ID >//使用其他容器的网路，共享IP和PORT等网络资源 none 容器使用自己的网络（类似--net=bridge），但是不进行配置 --privileged=false， 指定容器是否为特权容器，特权容器拥有所有的capabilities --restart="no"， 指定容器停止后的重启策略: no：容器退出时不重启 on-failure：容器故障退出（返回值非零）时重启 always：容器退出时总是重启 --rm=false， 指定容器停止后自动删除容器(不支持以docker run -d启动的容器) --sig-proxy=true， 设置由代理接受并处理信号，但是SIGCHLD、SIGSTOP和SIGKILL不能被代理 ### 示例 * 运行一个在后台执行的容器，同时，还能用控制台管理： ``` shell docker run -i -t -d ubuntu:latest ``` * 运行一个带命令在后台不断执行的容器，不直接展示容器内部信息： ``` shell docker run -d ubuntu:latest ping www.docker.com ``` * 运行一个在后台不断执行的容器，同时带有命令，程序被终止后还能重启继续跑，还能用控制台管理， ``` shell docker run -d --restart=always ubuntu:latest ping www.docker.com ``` * 为容器指定一个名字， ``` shell docker run -d --name=ubuntu_server ubuntu:latest ``` * 容器暴露80端口，并指定宿主机80端口与其通信(: 之前是宿主机端口，之后是容器需暴露的端口)， ``` shell docker run -d --name=ubuntu_server -p 80:80 ubuntu:latest ``` * 指定容器内目录与宿主机目录共享(: 之前是宿主机文件夹，之后是容器需共享的文件夹)， ```shell docker run -d --name=ubuntu_server -v /etc/www:/var/www ubuntu:latest ``` ================================================ FILE: docs/docker/docker_core.md ================================================ # Docker 核心技术 ## 1. 从系统架构谈起 ### 传统分层架构 vs 微服务 ![docker_core_1](../img/docker/docker_core_1.jpg) ![docker_core_2](../img/docker/docker_core_2.jpg) ### 微服务改造 分离微服务的方法建议： - 审视并发现可以分离的业务逻辑业务逻辑 - 寻找天生隔离的代码模块，可以借助于静态代码分析工具 - 不同并发规模，不同内存需求的模块都可以分离出不同的微服务，此方法可提高资源利用率，节省成本 一些常用的可微服务化的组件： - 用户和账户管理 - 授权和会话管理 - 系统配置 - 通知和通讯服务 - 照片，多媒体，元数据等 分解原则：基于size, scope and capabilities ### 微服务间通讯 点对点： - 多用于系统内部多组件之间通讯； - 有大量的重复模块如认证授权； - 缺少统一规范，如监控，审计等功能； - 后期维护成本高，服务和服务的依赖关系错综复杂难以管理。 ![docker_core_3](../img/docker/docker_core_3.jpg) API 网关 - 基于一个轻量级的message gateway - 新API通过注册至Gateway实现 - 整合实现Common function ![docker_core_4](../img/docker/docker_core_4.jpg) ## 2. 理解 Docker ### Docker - 基于Linux 内核的Cgroup，Namespace，以及Union FS等技术，对进程进行封装隔离，属于操作系统 层面的虚拟化技术，由于隔离的进程独立于宿主和其它的隔离的进程，因此也称其为容器。 - 最初实现是基于LXC，从0.7 以后开始去除LXC，转而使用自行开发的Libcontainer，从1.11 开始，则 进一步演进为使用runC和Containerd。 - Docker 在容器的基础上，进行了进一步的封装，从文件系统、网络互联到进程隔离等等，极大的简化了容 器的创建和维护，使得Docker 技术比虚拟机技术更为轻便、快捷。 ### 为什么要用 Docker - 更高效地利用系统资源 - 更快速的启动时间 - 一致的运行环境 - 持续交付和部署 - 更轻松地迁移 - 更轻松地维护和扩展 ...... ### 虚拟机和容器运行态的对比 ![docker_core_5](../img/docker/docker_core_5.jpg) ### 性能对比 ![docker_core_6](../img/docker/docker_core_6.jpg) ### 安装 Docker 在ubuntu上安装Docker运行时，参考https://docs.docker.com/engine/install/ubuntu/ $ sudoapt-get update $ sudoapt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common $ curl -fsSLhttps://download.docker.com/linux/ubuntu/gpg| sudoapt-key add - $ sudoadd-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release-cs) \ stable" $ sudoapt-get update $ sudoapt-get install docker-cedocker-ce-cli containerd.io ### 容器操作 启动： - docker run - it 交互 - d后台运行 - p端口映射 - v磁盘挂载 - 启动已终止容器 ``` docker start ``` - 停止容器 ``` docker stop ``` - 查看容器进程 ``` dockerps ``` - 查看容器细节： dockerinspect - 进入容器； - Docker attach： 通过 nsenter PID=$(docker inspect --format "{{ .State.Pid}}" ) $ nsenter--target $PID --mount --uts--ipc--net --pid - 拷贝文件至容器内： dockercp file1 :/file-to-path ### 初识容器 - cat Dockerfile ``` FROM ubuntu ENV MY_SERVICE_PORT=80 ADD bin/amd64/httpserver /httpserver ENTRYPOINT /httpserver ``` - 将Dockerfile打包成镜像 ``` docker build -t cncamp/httpserver:${tag}. docker push cncamp/httpserver:v1. ``` - 运行容器 ``` docker run -d cncamp/httpserver:v1. ``` ### 容器标准 - Open Container Initiative（OCI） - 轻量级开放式管理组织（项目） - OCI主要定义两个规范 - Runtime Specification - 文件系统包如何解压至硬盘，共运行时运行。 - Image Specification - 如何通过构建系统打包，生成镜像清单（Manifest）、文件系统序列化文件、镜像配置。 ### 容器主要特性 - 隔离性 - 便携性 - 可配额 - 安全性 ### Namespace - Linux Namespace是一种Linux Kernel提供的资源隔离方案： - 系统可以为进程分配不同的Namespace； - 并保证不同的Namespace资源独立分配、进程彼此隔离，即不同的Namespace下的进程互不干扰。 ##### Linux 内核代码中 Namespace 的实现 - 进程数据结构 ``` struct task_struct { ... /* namespaces */ struct nsproxy *nsproxy; ... } ``` - Namespace数据结构 ``` struct nsproxy { atomic_t count; struct uts_namespace *uts_ns; struct ipc_namespace *ipc_ns; struct mnt_namespace *mnt_ns; struct pid_namespace *pid_ns_for_children; struct net *net_ns; } ``` #### Linux 对 Namespace 操作方法 - clone 在创建新进程的系统调用时，可以通过flags参数指定需要新建的Namespace类型： // CLONE_NEWCGROUP/ CLONE_NEWIPC/ CLONE_NEWNET/ CLONE_NEWNS/ CLONE_NEWPID/ CLONE_NEWUSER/ CLONE_NEWUTS ``` int clone(int (*fn)(void *), void *child_stack, int flags, void *arg) ``` - setns 该系统调用可以让调用进程加入某个已经存在的Namespace中： ``` Int setns(int fd, int nstype) ``` - unshare 该系统调用可以将调用进程移动到新的Namespace下： ``` int unshare(int flags) ``` #### 隔离性 – Linux Namespace ``` Namespace类型 隔离资源 Kernel版本 ``` ``` IPC System V IPC和POSIX消息队列 2.6.19 ``` ``` Network 网络设备、网络协议栈、网络端口等 2.6.29 ``` ``` PID 进程 2.6.14 ``` ``` Mount 挂载点 2.4.19 ``` ``` UTS 主机名和域名 2.6.19 ``` ``` USR 用户和用户组 3.8 ``` ![docker_core_7](../img/docker/docker_core_7.jpg) Pidnamespace - 不同用户的进程就是通过Pidnamespace隔离开的，且不同namespace 中可以有相同Pid。 - 有了Pidnamespace, 每个namespace中的Pid能够相互隔离。 net namespace - 网络隔离是通过net namespace实现的，每个net namespace有独立的network devices, IP addresses, IP routing tables, /proc/net 目录。 - Docker默认采用veth的方式将container中的虚拟网卡同host上的一个docker bridge: docker0连接在一起。 ipcnamespace - Container中进程交互还是采用linux常见的进程间交互方法（interprocesscommunication –IPC）, 包括常见的信号量、消息队列和共享内存。 - container 的进程间交互实际上还是host上具有相同Pidnamespace中的进程间交互，因此需要在IPC资源申请时加入namespace信息-每个IPC资源有一个唯一的 32 位ID。 mntnamespace - mntnamespace允许不同namespace的进程看到的文件结构不同，这样每个namespace 中的进程所看到的文件目录就被隔离开了。 utsnamespace -UTS(“UNIX Time-sharing System”) namespace允许每个container拥有独立的hostname和 domain name, 使其在网络上可以被视作一个独立的节点而非Host上的一个进程。 user namespace - 每个container可以有不同的user 和group id, 也就是说可以在container内部用container内部的用户执行程序而非Host上的用户。 #### 关于 namespace 的常用操作 - 查看当前系统的namespace： ``` lsns –t ``` - 查看某进程的namespace： ``` ls -la /proc//ns/ ``` - 进入某namespace运行命令： ``` nsenter-t -n ipaddr ``` #### Namespace 练习 - 在新networknamespace执行sleep指令： unshare-fnsleep 60 - 查看进程信息 ps-ef|grepsleep root 32882 4935 0 10:00 pts/0 00:00:00 unshare-fnsleep 60 root 32883 32882 0 10:00 pts/0 00:00:00 sleep 60 - 查看网络Namespace lsns-t net 4026532508 net 2 32882 root unassigned unshare - 进入改进程所在Namespace查看网络配置，与主机不一致 nsenter-t 32882 -n ipa 1: lo: mtu 65536 qdiscnoopstate DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00 ### Cgroups - Cgroups（Control Groups）是Linux下用于对一个或一组进程进行资源控制和监控的机制； - 可以对诸如CPU使用时间、内存、磁盘I/O等进程所需的资源进行限制； - 不同资源的具体管理工作由相应的Cgroup子系统（Subsystem）来实现； - 针对不同类型的资源限制，只要将限制策略在不同的的子系统上进行关联即可； - Cgroups在不同的系统资源管理子系统中以层级树（Hierarchy）的方式来组织管理：每个Cgroup都可以 包含其他的子Cgroup，因此子Cgroup能使用的资源除了受本Cgroup配置的资源参数限制，还受到父 Cgroup设置的资源限制。 ##### Linux 内核代码中 Cgroups 的实现 - 进程数据结构 ``` struct task_struct { #ifdef CONFIG_CGROUPS struct css_set __rcu *cgroups; struct list_head cg_list; #endif } ``` - css_set 是 cgroup_subsys_state 对象的集合数据结构 ``` struct css_set { /* * Set of subsystem states, one for each subsystem. This array is * immutable after creation apart from the init_css_set during * subsystem registration (at boot time). */ struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT]; }; ``` ### 可配额/可度量 - Control Groups (cgroups) ![docker_core_8](../img/docker/docker_core_8.jpg) cgroups实现了对资源的配额和度量 - blkio： 这个子系统设置限制每个块设备的输入输出控制。例如:磁盘，光盘以及USB等等。 - CPU： 这个子系统使用调度程序为cgroup任务提供CPU的访问。 - cpuacct： 产生 cgroup任务的CPU资源报告。 - cpuset： 如果是多核心的CPU，这个子系统会为cgroup任务分配单独的CPU和内存。 - devices： 允许或拒绝cgroup任务对设备的访问。 - freezer： 暂停和恢复cgroup任务。 - memory： 设置每个cgroup的内存限制以及产生内存资源报告。 - net_cls： 标记每个网络包以供cgroup方便使用。 - ns： 名称空间子系统。 - pid: 进程标识子系统。 #### CPU 子系统 cpu.shares： 可出让的能获得CPU使用时间的相对值。 cpu.cfs_period_us：cfs_period_us用来配置时间周期长度，单位为us（微秒）。 cpu.cfs_quota_us：cfs_quota_us用来配置当前Cgroup在cfs_period_us时间内最多能使用的CPU时间数，单 位为us（微秒）。 cpu.stat： Cgroup内的进程使用的CPU时间统计。 nr_periods： 经过cpu.cfs_period_us的时间周期数量。 nr_throttled： 在经过的周期内，有多少次因为进程在指定的时间周期内用光了配额时间而受到限制。 throttled_time： Cgroup中的进程被限制使用CPU的总用时，单位是ns（纳秒）。 #### Linux 调度器 内核默认提供了 5 个调度器，Linux内核使用struct sched_class来对调度器进行抽象： - Stop调度器，stop_sched_class：优先级最高的调度类，可以抢占其他所有进程，不能被其他进程抢占； - Deadline调度器，dl_sched_class：使用红黑树，把进程按照绝对截止期限进行排序，选择最小进程进 行调度运行； - RT调度器，rt_sched_class：实时调度器，为每个优先级维护一个队列； - CFS调度器，cfs_sched_class：完全公平调度器，采用完全公平调度算法，引入虚拟运行时间概念； - IDLE-Task调度器，idle_sched_class：空闲调度器，每个CPU都会有一个idle线程，当没有其他进程 可以调度时，调度运行idle线程。 #### CFS 调度器 - CFS是Completely Fair Scheduler简称，即完全公平调度器。 - CFS 实现的主要思想是维护为任务提供处理器时间方面的平衡，这意味着应给进程分配相当数量的处理器。 - 分给某个任务的时间失去平衡时，应给失去平衡的任务分配时间，让其执行。 - CFS通过虚拟运行时间（vruntime）来实现平衡，维护提供给某个任务的时间量。 - vruntime= 实际运行时间*1024 / 进程权重 - 进程按照各自不同的速率在物理时钟节拍内前进，优先级高则权重大，其虚拟时钟比真实时钟跑得慢，但 获得比较多的运行时间。 #### vruntime 红黑树 CFS调度器没有将进程维护在运行队列中，而是维护了一个以虚拟运行时间为顺序的红黑树。红黑树的主要 特点有： 1. 自平衡，树上没有一条路径会比其他路径长出俩倍。 2. O(log n) 时间复杂度，能够在树上进行快速高效地插入或删除进程。 ![docker_core_9](../img/docker/docker_core_9.jpg) #### CFS进程调度 - 在时钟周期开始时，调度器调用__schedule()函数来开始调度的运行。 - __schedule()函数调用pick_next_task()让进程调度器从就绪队列中选择一个最合适的进程next，即红 黑树最左边的节点。 - 通过context_switch()切换到新的地址空间，从而保证next进程运行。 - 在时钟周期结束时，调度器调用entity_tick()函数来更新进程负载、进程状态以及vruntime（当前vruntime+ 该时钟周期内运行的时间）。 - 最后，将该进程的虚拟时间与就绪队列红黑树中最左边的调度实体的虚拟时间做比较，如果小于坐左边的时间，则不用触发调度，继续调度当前调度实体。 ![docker_core_10](../img/docker/docker_core_10.jpg) #### CPU 子系统练习 - 在cgroupcpu子系统目录中创建目录结构 cd /sys/fs/cgroup/cpu mkdircpudemo cd cpudemo - 运行busyloop - 执行top查看CPU使用情况，CPU占用200% - 通过cgroup限制cpu cd /sys/fs/cgroup/cpu/cpudemo - 把进程添加到cgroup进程配置组 echo ps-ef|grepbusyloop|grep-v grep|awk'{print $2}' > cgroup.procs - 设置cpuquota echo 10000 > cpu.cfs_quota_us - 执行top查看CPU使用情况，CPU占用变为10% ##### cpuacct 子系统 用于统计Cgroup及其子Cgroup下进程的CPU的使用情况。 - cpuacct.usage 包含该Cgroup及其子Cgroup下进程使用CPU的时间，单位是ns（纳秒）。 - cpuacct.stat 包含该Cgroup及其子Cgroup下进程使用的CPU时间，以及用户态和内核态的时间。 #### Memory 子系统 - memory.usage_in_bytes ``` cgroup下进程使用的内存，包含cgroup及其子cgroup下的进程使用的内存 ``` - memory.max_usage_in_bytes ``` cgroup下进程使用内存的最大值，包含子cgroup的内存使用量。 ``` - memory.limit_in_bytes ``` 设置Cgroup下进程最多能使用的内存。如果设置为- 1 ，表示对该cgroup的内存使用不做限制。 ``` - memory.soft_limit_in_bytes ``` 这个限制并不会阻止进程使用超过限额的内存，只是在系统内存足够时，会优先回收超过限额的内存，使之向限定值靠拢。 ``` - memory.oom_control 设置是否在Cgroup中使用OOM（Out of Memory）Killer，默认为使用。当属于该cgroup的进程使用的内存超过最大的限定值时， 会立刻被OOM Killer处理。 #### Cgroup driver systemd: - 当操作系统使用systemd作为initsystem时，初始化进程生成一个根cgroup目录结构并作为cgroup 管理器。 - systemd与cgroup紧密结合，并且为每个systemdunit分配cgroup。 cgroupfs: - docker默认用cgroupfs作为cgroup驱动。 存在问题： - 在systemd作为initsystem的系统中，默认并存着两套groupdriver。 - 这会使得系统中Docker和kubelet管理的进程被cgroupfs驱动管，而systemd拉起的服务由 systemd驱动管，让cgroup管理混乱且容易在资源紧张时引发问题。 因此kubelet会默认--cgroup-driver=systemd，若运行时cgroup不一致时，kubelet会报错。 ### 文件系统 Union FS - 将不同目录挂载到同一个虚拟文件系统下（unite several directories into a single virtual filesystem） 的文件系统 - 支持为每一个成员目录（类似GitBranch）设定readonly、readwrite 和whiteout-able 权限 - 文件系统分层, 对readonly权限的branch 可以逻辑上进行修改(增量地, 不影响readonly部分的)。 - 通常Union FS 有两个用途, 一方面可以将多个disk挂到同一个目录下, 另一个更常用的就是将一个 readonly的branch 和一个writeable 的branch 联合在一起。 #### 容器镜像 ![docker_core_11](../img/docker/docker_core_11.jpg) ##### Docker 的文件系统 典型的Linux文件系统组成： - Bootfs（boot file system） - Bootloader-引导加载kernel， - Kernel-当kernel被加载到内存中后umount bootfs。 - rootfs（root file system） - /dev，/proc，/bin，/etc等标准目录和文件。 - 对于不同的linux发行版, bootfs基本是一致的， 但rootfs会有差别。 ##### Docker 启动 Linux - 在启动后，首先将rootfs设置为readonly, 进行一系列检查, 然后将其切换为“readwrite”供用户使用。 Docker启动 - 初始化时也是将rootfs以readonly方式加载并检查，然而接下来利用union mount 的方式将一个 readwrite文件系统挂载在readonly的rootfs之上； - 并且允许再次将下层的FS（file system）设定为readonly并且向上叠加。 - 这样一组readonly和一个writeable的结构构成一个container的运行时态, 每一个FS被称作一个FS层。 #### 写操作 由于镜像具有共享特性，所以对容器可写层的操作需要依赖存储驱动提供的写时复制和用时分配机制，以此来 支持对容器可写层的修改，进而提高对存储和内存资源的利用率。 - 写时复制 - 写时复制，即Copy-on-Write。 - 一个镜像可以被多个容器使用，但是不需要在内存和磁盘上做多个拷贝。 - 在需要对镜像提供的文件进行修改时，该文件会从镜像的文件系统被复制到容器的可写层的文件系统 进行修改，而镜像里面的文件不会改变。 - 不同容器对文件的修改都相互独立、互不影响。 - 用时分配 按需分配空间，而非提前分配，即当一个文件被创建出来后，才会分配空间。 #### 容器存储驱动 ![docker_core_12](../img/docker/docker_core_12.jpg) ![docker_core_13](../img/docker/docker_core_13.jpg) #### 以 OverlayFS 为例 OverlayFS也是一种与AUFS类似的联合文件系统，同样属于文件级的存储驱动，包含了最初的Overlay和 更新更稳定的overlay2。 Overlay只有两层：upper层和lower层，Lower层代表镜像层，upper层代表容器可写层。 ![docker_core_14](../img/docker/docker_core_14.jpg) ##### OverlayFS 文件系统练习 $ mkdirupper lower merged work $ echo "from lower" > lower/in_lower.txt $ echo "from upper" > upper/in_upper.txt $ echo "from lower" > lower/in_both.txt $ echo "from upper" > upper/in_both.txt $ sudomount -t overlay overlay -o lowerdir=`pwd`/lower,upperdir=`pwd`/upper,workdir=`pwd`/work `pwd`/merged $ cat merged/in_both.txt $ delete merged/in_both.txt $ delete merged/in_lower.txt $ delete merged/in_upper.txt #### OCI 容器标准 Open Container Initiative - OCI组织于 2015 年创建，是一个致力于定义容器镜像标准和运行时标准的开放式组织。 - OCI定义了镜像标准（Runtime Specification）、运行时标准（Image Specification）和分发标准 （DistributionSpecification） - 镜像标准定义应用如何打包 - 运行时标准定义如何解压应用包并运行 - 分发标准定义如何分发容器镜像 #### Docker 引擎架构 ![docker_core_15](../img/docker/docker_core_15.jpg) #### 网络 ![docker_core_16](../img/docker/docker_core_16.jpg) ![docker_core_17](../img/docker/docker_core_17.jpg) ##### Null 模式 - Null 模式是一个空实现； - 可以通过Null模式启动容器并在宿主机上通 过命令为容器配置网络。 ``` mkdir-p /var/run/netns find -L /var/run/netns-type l -delete ln -s /proc/$pid/ns/net /var/run/netns/$pid iplink add A type vethpeer name B brctladdifbr0 A iplink set A up iplink set B netns$pid ipnetnsexec $pidiplink set dev B name eth0 ipnetnsexec $pidiplink set eth0 up ipnetnsexec $pidipaddradd $SETIP/$SETMASK dev eth0 ipnetnsexec $pidiproute add default via $GATEWAY ``` ##### 默认模式– 网桥和 NAT 为主机eth0分配IP192.168.0.101； 启动dockerdaemon，查看主机iptables； - POSTROUTING-A POSTROUTING -s 172.17.0.0/16! -o docker0 -j MASQUERADE 在主机启动容器： - docker run -d --name ssh-p 2333:22 centos-ssh - Docker会以标准模式配置网络： - 创建vethpair； - 将vethpair的一端连接到docker0网桥； - vethpair的另外一端设置为容器名空间的eth0； - 为容器名空间的eth0分配ip； - 主机上的Iptables 规则：PREROUTING-A DOCKER! -idocker0 -p tcp-m tcp--dport 2333 - j DNAT --to- destination 172.17.0.2:22。 ![docker_core_18](../img/docker/docker_core_18.jpg) #### Underlay - 采用Linux 网桥设备（sbrctl），通过物理网络连通容器； - 创建新的网桥设备mydr0； - 将主机网卡加入网桥； - 把主机网卡的地址配置到网桥，并把默认路由规则转移到网桥mydr0； - 启动容器； - 创建veth对，并且把一个peer添加到网桥mydr0； - 配置容器把veth的另一个peer分配给容器网卡； ![docker_core_19](../img/docker/docker_core_19.jpg) #### Docker Libnetwork Overlay l Dockeroverlay网络驱动原生支持多主机网络； l Libnetwork是一个内置的基于VXLAN的网络驱动。 #### VXLAN ![docker_core_20](../img/docker/docker_core_20.jpg) #### Overlay network sample – Flannel - 同一主机内的 Pod可以使用网桥进行通信。 - 不同主机上的 Pod将通过flanneld将其流量封装在UDP数据包中。 ![docker_core_21](../img/docker/docker_core_21.jpg) #### Flannel packet sample ![docker_core_22](../img/docker/docker_core_22.jpg) ##### 创建 docker 镜像 定义dockerfile ``` FROM ubuntu # so aptENV DEBIAN_FRONTEND=-get doesn't complainnoninteractive RUN sed-i's/^exit 101/exit 0/' /usr/sbin/policy-rc.d RUN \ aptapt--get update get install -y ca&& \-certificates && \ apt-get install -y curl && \ rm-rf/var/lib/apt/lists/* ADD ENTRYPOINT ./bin/eiceic["/eic"] ``` docker build ``` docker build ``` ### 3. Dockerfile 的最佳实践 #### 回顾 12 Factor 之进程 - 运行环境中，应用程序通常是以一个和多个进程运行的。 - 12 - Factor 应用的进程必须无状态（Stateless）且无共享（Share nothing）。 - 任何需要持久化的数据都要存储在后端服务内，比如数据库。 - 应在构建阶段将源代码编译成待执行应用。 - Session Sticky是 12 - Factor 极力反对的。 - Session 中的数据应该保存在诸如Memcached或Redis这样的带有过期时间 的缓存中。 Docker遵循以上原则管理和构建应用。 #### 理解构建上下文（Build Context） - 当运行dockerbuild命令时，当前工作目录被称为构建上下文。 - dockerbuild 默认查找当前目录的Dockerfile作为构建输入，也可以通过–f指定Dockerfile。 - docker build –f ./Dockerfile - 当dockerbuild运行时，首先会把构建上下文传输给dockerdaemon，把没用的文件包含在构建上下文时，会 导致传输时间长，构建需要的资源多，构建出的镜像大等问题。 - 试着到一个包含文件很多的目录运行下面的命令，会感受到差异； - docker build -f $GOPATH/src/github.com/cncamp/golang/httpserver/Dockerfile； - docker build $GOPATH/src/github.com/cncamp/golang/httpserver/； - 可以通过.dockerignore文件从编译上下文排除某些文件。 - 因此需要确保构建上下文清晰，比如创建一个专门的目录放置Dockerfile，并在目录中运行dockerbuild。 #### 镜像构建日志 docker build $GOPATH/src/github.com/cncamp/golang/httpserver/ Sending build context to Docker daemon 14.57MB Step 1/4 : FROM ubuntu ---> cf0f3ca922e0 Step 2/4 : ENV MY_SERVICE_PORT=80 ---> Using cache ---> a7d824f74410 Step 3/4 : ADD bin/amd64/httpserver/httpserver ---> Using cache ---> 00bb47fce704 Step 4/4 : ENTRYPOINT /httpserver ---> Using cache ---> f77ee3366d08 Successfully built f77ee3366d08 #### Build Cache 构建容器镜像时，Docker依次读取Dockerfile中的指令，并按顺序依次执行构建指令。 Docker读取指令后，会先判断缓存中是否有可用的已存镜像，只有已存镜像不存在时才会重新构建。 - 通常Docker简单判断Dockerfile中的指令与镜像。 - 针对ADD和COPY指令，Docker判断该镜像层每一个文件的内容并生成一个checksum，与现存镜 像比较时，Docker比较的是二者的checksum。 - 其他指令，比如RUN apt-get -y update，Docker简单比较与现存镜像中的指令字串是否一致。 - 当某一层cache失效以后，所有所有层级的cache均一并失效，后续指令都重新构建镜像。 #### 多段构建（Multi-stage build） - 有效减少镜像层级的方式 FROM golang:1.16-alpine AS build RUN apkadd --no-cache git RUN go get github.com/golang/dep/cmd/dep COPY Gopkg.lockGopkg.toml/go/src/project/ WORKDIR /go/src/project/ RUN dep ensure -vendor-only COPY. /go/src/project/ RUN go build -o /bin/project（只有这个二进制文件是产线需要的，其他都是waste） FROM scratch COPY --from=build /bin/project /bin/project ENTRYPOINT ["/bin/project"] CMD ["--help"] #### Dockerfile 常用指令 - FROM：选择基础镜像，推荐alpine FROM [--platform=] [@] [AS ] - LABELS：按标签组织项目 LABEL multi.label1="value1" multi.label2="value2" other="value3” 配合labelfilter可过滤镜像查询结果 docker images -f label=multi.label1="value1" - RUN：执行命令 最常见的用法是RUN apt-get update && apt-get install，这两条命令应该永远用&&连接，如果分开执行，RUN apt-get update 构建层被缓存，可能会导致新package无法安装 - CMD：容器镜像中包含应用的运行命令，需要带参数 CMD ["executable", "param1", "param2"...] #### Dockerfile 常用指令 - EXPOSE：发布端口 ``` EXPOSE [/...] - 是镜像创建者和使用者的约定 - 在dockerrun –P时，docker会自动映射expose的端口到主机大端口，如0.0.0.0:32768->80/tcp ``` - ENV 设置环境变量 ``` ENV = ... ``` - ADD：从源地址（文件，目录或者URL）复制文件到目标路径 ``` ADD [--chown=:] ... ADD [--chown=:] [“”,... “”] （路径中有空格时使用） ``` - ADD支持Go风格的通配符，如ADD check* /testdir/ - src如果是文件，则必须包含在编译上下文中，ADD 指令无法添加编译上下文之外的文件 - src如果是URL - 如果dest结尾没有/，那么dest是目标文件名，如果dest结尾有/，那么dest是目标目录名 - 如果src是一个目录，则所有文件都会被复制至dest - 如果src是一个本地压缩文件，则在ADD 的同时完整解压操作 - 如果dest不存在，则ADD 指令会创建目标目录 - 应尽量减少通过ADDURL 添加remote 文件，建议使用curl 或者wget&& untar - COPY：从源地址（文件，目录或者URL）复制文件到目标路径 ``` COPY [--chown=:] ... COPY [--chown=:] ["",... ""]//路径中有空格时使用 ``` - COPY的使用与ADD类似，但有如下区别 - COPY 只支持本地文件的复制，不支持URL - COPY 不解压文件 - COPY可以用于多阶段编译场景，可以用前一个临时镜像中拷贝文件 - COPY --from=build /bin/project /bin/project COPY 语义上更直白，复制本地文件时，优先使用COPY - ENTRYPOINT：定义可以执行的容器镜像入口命令 ``` ENTRYPOINT ["executable", "param1", "param2"] // docker run参数追加模式 ENTRYPOINT command param1 param2 // docker run参数替换模式 ``` - docker run –entrypoint可替换Dockerfile中定义的ENTRYPOINT - ENTRYPOINT的最佳实践是用ENTRYPOINT定义镜像主命令，并通过CMD定义主要参数，如下所示 - ENTRYPOINT ["s3cmd"] - CMD ["--help"] ##### Dockerfile 常用指令 - VOLUME：将指定目录定义为外挂存储卷，Dockerfile中在该指令之后所有对同一目录的修改都无效 ``` VOLUME ["/data"] ``` 等价于docker run –v /data，可通过dockerinspect查看主机的mountpoint， /var/lib/docker/volumes//_data - USER：切换运行镜像的用户和用户组，因安全性要求，越来越多的场景要求容器应用要以non-root身份运行 ``` USER [:] ``` - WORKDIR：等价于cd，切换工作目录 ``` WORKDIR /path/to/workdir ``` - 其他非常用指令 - ARG - ONBUILD - STOPSIGNAL - HEALTHCHECK - SHELL #### Dockerfile 最佳实践 - 不要安装无效软件包。 - 应简化镜像中同时运行的进程数，理想状况下，每个镜像应该只有一个进程。 - 当无法避免同一镜像运行多进程时，应选择合理的初始化进程（initprocess）。 - 最小化层级数 - 最新的docker只有RUN，COPY，ADD创建新层，其他指令创建临时层，不会增加镜像大小。 - 比如EXPOSE指令就不会生成新层。 - 多条RUN命令可通过连接符连接成一条指令集以减少层数。 - 通过多段构建减少镜像层数。 - 把多行参数按字母排序，可以减少可能出现的重复参数，并且提高可读性。 - 编写dockerfile的时候，应该把变更频率低的编译指令优先构建以便放在镜像底层以有效利用buildcache。 - 复制文件时，每个文件应独立复制，这确保某个文件变更时，只影响改文件对应的缓存。 目标：易管理、少漏洞、镜像小、层级少、利用缓存。 #### 多进程的容器镜像 - 选择适当的init进程 - 需要捕获SIGTERM信号并完成子进程的优雅终止 - 负责清理退出的子进程以避免僵尸进程 开源项目 https://github.com/krallin/tini #### Docker 镜像管理 dockersave/load dockertag dockerpush/pull #### 基于 Docker 镜像的版本管理 - Docker tag docker tag命令可以为容器镜像添加标签 docker tag 0e5574283393 hub.docker.com/cncamp/httpserver:v1.0 hub.docker.com：镜像仓库地址，如果不填，则默认为hub.docker.com cncamp:repositry httpserver：镜像名 v1.0：tag，常用来记录版本信息 #### Docker tag 与 github 的版本管理合力 - 以Kubernetes为例 - 开发分支 - git checkout master - Release分支 - git checkout –b release-1.21 - 在并行期，所有的变更同时放进master和releasebranch - 版本发布 - 以releasebranch为基础构建镜像，并为镜像标记版本信息：docker tag 0e5574283393 k8s.io/kubernetes/apiserver:v1.21 - 在github中保存release代码快照 - git tag v1.21 #### 镜像仓库 Docker hub - https://hub.docker.com/ 创建私有镜像仓库 - sudodocker run -d -p 5000:5000 registry ##### Docker 优势 封装性： - 不需要再启动内核，所以应用扩缩容时可以秒速启动。 - 资源利用率高，直接使用宿主机内核调度资源，性能损失 小。 - 方便的CPU、内存资源调整。 - 能实现秒级快速回滚。 - 一键启动所有依赖服务，测试不用为搭建环境犯愁，PE也不用为 建站复杂担心。 - 镜像一次编译，随处使用。 - 测试、生产环境高度一致（数据除外）。 镜像增量分发： - 由于采用了Union FS，简单来说就是支持将不同的目录 挂载到同一个虚拟文件系统下，并实现一种layer的概 念，每次发布只传输变化的部分，节约带宽。 隔离性： - 应用的运行环境和宿主机环境无关，完全由镜像控制，一台物理 机上部署多种环境的镜像测试。 - 多个应用版本可以并存在机器上。 社区活跃： - Docker命令简单、易用，社区十分活跃，且周边组件丰富。 ================================================ FILE: docs/emotion/emotion.md ================================================ # 情绪管理 ### （一）要不要管理好情绪 有一个男孩脾气很坏，于是他的父亲就给了他一袋钉子，并且告诉他，当他想发脾气的时候，就钉一根钉子在后院的围篱上。第一天，这个男孩钉下了40根钉子。慢慢地，男孩可以控制他的情绪，不再乱发脾气，所以每天钉下的钉子也跟着减少了，他发现控制自己的脾气比钉下那些钉子来得容易一些。终于，父亲告诉他，现在开始每当他能控制自己的脾气的时候，就拔出一根钉子。一天天过去了，最后男孩告诉他的父亲，他终于把所有的钉子都拔出来了。于是，父亲牵着他的手来到后院，告诉他说：“孩子，你做得很好。但看看那，此时围墙上的坑坑洞洞，这些围篱将永远不能回复从前的样子了，当你生气时所说的话就像这些钉子一样，会留下很难弥补的疤痕，有些是难以磨灭的呀！”从此，男孩终于懂得管理情绪的重要性了。 你现在的心情如何！是欢乐、烦恼、生气、担心、害怕、难过、失望或者是平静无常呢？还是你根本不懂自己的心情！一早起来，也许你看到阳光普照而心情愉快，也可能因为细雨绵绵而心情低落；你也许因为逃课没被点到名而高兴，然而考试快到又让你担心；谈恋爱的你，心花怒放，失恋的你却又垂头丧气……我们拥有许多不同的情绪，而它们似乎也为我们的生活增添了许多色彩。然而，有情绪好不好呢？一个成功的人应不应该流露情绪？怕不怕被人说你太情绪化？所以宁愿不要有情绪……其实真正的问题并不存情绪本身，而在情绪的表达方式，如果能以适当的方式在适当的情境表达适度的情绪，就是健康的情绪管理之道。 通常因为面子、情境等场合因素所致，我们会不自觉地在他人面前隐藏真实的情绪，或者我们会将强烈的情绪转换为较不具杀伤力、震撼力的方式表现，而只让别人看到一部分真实的情绪，这都是非常符合社会规范、十足社会化的行为表现。或许我们未必完全清楚自己有这样的行为表现，但是我们确实是适宜地掌握自身的情绪，决定了合适的呈现形态，在不扭曲事实的情况下，在掌控范围之内，我们作出于人、于己皆恰当的决定与行为反应。 如果我们能够明白这些行为都是经过自己抉择、决定的结果，进而能为自己的情绪负责任，则不必要的情绪问题便可以减少。问题是，生活中所面临的许多事对个人而言是具有威胁性的，有些大到足以引起个人内心焦虑的事，旁人看来却只是芝麻绿豆大，然而对个人而言，却可能意味着个人形象、价值感、自尊心的折损和破坏，令人方寸大乱！此时，恐怕就得费些工夫才能对情绪操控自如了。 关于情绪能力的“软糖实验” 实验人员把一组4岁儿童分别领入空荡荡的大房间，只在一张桌子上放着非常显眼的东西：软糖。这些孩子进来前，实验人员告诉过他们，允许你走出大厅之前吃掉这颗软糖，但如果你能坚持在走出大厅之前不吃这颗糖，就会有奖励，能再得到一块软糖。结果当然是两种情况都有。专家们把坚持下来得到第二块糖的孩子归为一组，没有坚持下来只吃一块糖的孩子归为另一组，并对这两组孩子进行了14年的追踪研究。结果发现，那些向往未来而能克制眼前诱惑的孩子，在学业、品质、行为、操守方面，与另一组相比有显著优越的表现。这说明，决定人生成功的因素并非只有传统智商理论所认定的那些东西，非智力因素特别是情绪智力对个人的成功有着极为重要的影响。 人的自控能力大小跟人生成功与否有着密切的关系。心理学家经过长期研究认为：人与人之间的智商并没有明显的差别，但有的人之所以成功，有的人之所以未能成功，与各自的情商有密切关系。情商的要素之一就是人的自控能力，从某种意义上讲，情商表现的是人们通过控制自己的情绪来提高生活品质的能力，即如何激活自己的潜能，如何克制自己的情绪冲动，如何使自己始终对未来充满希望，等等。 ### （二） 能接纳自己的情绪吗 1. 你有没有对情绪的迷失？ 一般而言，我们会将情绪简单分为正向情绪和负向情绪，即爱的反面是恨，喜悦、高兴的反面是难过、沮丧。然而，有些人却过度夸大情绪的负面影响，产生对情绪的迷失。例如，认为如果你对别人生气就表示你不喜欢他；或者认为表达生气就表示不尊重或者没有爱。所以当父母对你生气而大骂时，你可能就会全盘否定他们对你的关爱；或者父母的某些做法令你不高兴时，你可能会觉得有很深的罪恶感，觉得自己不应该这样；等等。其实爱恨是可以并存的，所有正向与负向的感受都可以同时存在，情绪只是反映出我们内在的感受，并没有好坏之分，每种情绪都有它独特的价值，如果仅仅为了某种情绪而忽略其他情绪，我们就无法完整地体验生活。 此外，我们对情绪还存在错觉，认为可以选择性地去掉一些情绪，可以完全不生气或者完全不难过，可以永远快乐。实际上当我们压抑情绪时，就累积了一些紧张，而压制了我们感受快乐与爱的能力。情绪的能力是整体的，只有自由地体验各种情绪，才能感受更多流畅的情绪。就如纪伯伦所说：“悲伤在你心中切割得愈深，你便能容纳更多的快乐。”“当你快乐时，深察你的内心吧。你将发现，只有那曾使你悲伤的，正给你快乐。当你悲伤时，再深察你的内心吧。你将明白，事实上你正为曾使你快乐的事物哭泣。” 2. 你怎样对待和处理自己的情绪？ 阿强前两天因为考试没考好，心理有挫折感。他一直责怪自己平时不够用功，考前没好好准备，考试的时候没仔细看，他觉得自己不是一块念书的料，自己比别人差，因而很灰心！他开始垂头丧气，故意远离人群，一个人躲在角落，心情很沮丧。 酷妹把她的朋友小顽子的心爱的偶像签名照弄丢了。那是小顽子千辛万苦、排了两三个小时队，才得到的偶像的亲笔签名，现在却被醋妹弄丢了，小顽子真的很生气。可是酷妹是她最好的朋友，怎么可以对她生气呢？而且生气是不好的，万一失控，不晓得会不会伤害到其他人呢？而且酷妹也许就不再跟她做朋友了，所以她告诉自己：“算了！丢了就丢了，生气也无济于事。”虽然这样，她还是心有疙瘩，无法再像以前一样对待酷妹。 小东的学习成绩不好，不喜欢上课，经常逃课去网吧玩游戏。父母经常教育他，希望他专心读书，将来能凭自己的本领自食其力。但是，小东总是嫌父母啰嗦，与他们争吵。有一次他跟父母吵架，一气之下把电视机给砸了，令父母很伤心。 上面三个故事中呈现出来的是某些同学对情绪的态度与处理方式，比较有代表性。如果主角换成是你，你会有什么感受？你会怎么处理你的情绪呢？有些人在面对情绪时，完全被情绪所控制，当负面情绪产生时，就任由情绪牵制他们的一切思想、感受和行为。影响层面小一点的包括个人心情不愉快、生活功能受到限制；影响层面广泛一点的包括人际关系出现问题，更严重的是他们可能因一时冲动，做出严重的举动，造成生命、财产的损失，后悔莫及。另外，有些人则是对负面情绪感到害怕、恐惧，担心自己若感受到生气、愤怒、悲伤、沮丧、紧张、焦虑等情绪，情况会更加糟糕，甚至会发生无法预测的后果，因而就极力压抑、控制自己的情绪；但是，没有表现出的情绪，并不表示没有情绪，所以原本被引发的情绪仍会间接地影响自己或者人际关系等。也有些人不满于负向情绪的控制和预防，他们认为情绪是非理性的，所以一个理性成熟的人不应该表现出自己的情绪。他们不允许自己处在负面的情绪中，拼命告诫自己“要理性”，“要控制情绪”，“我不应该焦虑，焦虑只会让我表现得更糟”，“我不应该沮丧，沮丧只会侵蚀我的斗志”，“我不能生气，生气代表我是一个不能把情绪管理好的人”。因此，他们塑造自己成为有修养的人，预防可能会引出负面情绪的情境。然而如果我们一味地否认、压抑或控制负面情绪，我们将失去适当地反映真实情绪的能力，所以也将无法真实感受到快乐等正面情绪，而变成一个单调无情绪的人。 其实，当我们失去感受负面情绪的能力，也就失去感受正面情绪的能力，然而许多人却很排斥负面情绪的发生或存在，对它敬而远之，除了因为它带给人们不愉快的感受之外，也因为它会使我们其他方面的运作和表现受到影响。然而排斥并不能防止这些负面情绪的出现，只是徒增自己适应上的困难而已。所以有效管理情绪的方法，绝不是压抑或控制，而是学习接纳情绪，允许自己有情绪，然后通过适当的方法加以表达或纾解。（改编自：《情绪管理》，第163－164页，（台湾）蔡秀玲、杨智馨著，安徽人民出版社，2001年版） 3. 能不能对惹你生气的人发怒？ 那些在不应当愤怒时而愤怒的人，被视为无能；愤怒的方式，愤怒发作的时刻，以及愤怒的对象不适合时，也被视为无能的表现。（——亚里士多德） 下面请听听在重要公正事务所工作的年轻人让·马克的诉说： 我从来没愤怒过，童年时期除外。那时我对金属玩具发火时，我的家长立即告诉我：“人不能对东西生气。”而且按照他们的观点，对人也不能生气。进入青春期以后，我心情经常不好，他们都会要求我：“不能用自己的情绪影响别人，要控制自己。”除了最小的妹妹阿丽思，我的姐妹们比我更服从，后来阿丽思与一个个性很强的人结为夫妻。我父母履行自己的说教：总是彬彬有礼，笑容满面，心平气和，即使与他人争辩时也是这样。我还是能想起父亲驾驶汽车时，有人在前方“甩鱼尾”他愤怒的样子（他字字句句按照交通规则行事）。在生活中，无能力愤怒的表现，不久就给我带来许多麻烦。同龄青少年向我挑衅时，我无能力反击。因此我常常跑到女孩子那里躲避，因此成为女孩子们的知心人。我学习成绩很好，很容易就找到了工作，也因为我的性格类型令雇主们喜欢，我平和、礼貌，而且很能干。但是仔细想一下，我很苦恼，因为我常常被雄心更强或咄咄逼人的人欺负。他们故意这样做，我想是因为他们不害怕我。有时，我因对某个同事霸占了有意思的项目，或因跟我开让我不高兴的玩笑而反复琢磨，心里很不舒服。但是只要一面对他们，我有教养的“好孩子”表现就占了上风，我表现出彬彬有礼，只不过与之拉开点距离而已。妻子因此常常谴责我，因为她是我无能力对进攻做出反应的直接见证入，她非常愤怒。这种无能力发怒的压力变得越来越大，于是我决定去和心理治疗师谈一谈。其实不是我害怕对方的反应，只是他人进攻时，我感到内心的退缩，于是变得无动于衷，可事后我非常愤怒。家长把我训练得太有教养了！ 让·马克对不能将愤怒表达出来而造成的不良后果描写得淋漓尽致； ――他人对自己的任意摆弄； ――私下里常常对自己的无力反击而痛苦烦恼； ――一个男人缺少表达愤怒的能力，被看做唯命是从，缺乏男子汉气魄。 因此，不要因为自己表达了愤怒而觉得自己不好，你有权愤怒。当你愤怒时，请你这样想： ――最好能够控制，但是我不一定总能做到； ――我情愿不伤害别人，但是如果发生了，我也能承受； ――最好在我有道理的时候愤怒，但是我也有权做错事； ――我喜欢被人接纳，但是我不可能让所有的人喜欢我。 ### （三）如何有效地管理自己的情绪 诺贝尔文学奖得主赫曼赫塞说：“痛苦让你觉得苦恼的，只是因为你惧怕它、责怪它；痛苦会紧追你不舍，是因为你想逃离它。所以，你不可逃避，不可责怪，不可惧怕。你自己知道，在心的深处完全知道——世界上只有一个魔术、一种力量和一个幸福，它就叫爱。因此，去爱痛苦吧。不要违逆痛苦，不要逃避痛苦，去品尝痛苦深处的甜美吧。”要记住，其实情绪本身并无是非、好坏之分，每一种情绪都有它的价值和功能。因此，一个心理健康的人不否定自己情绪的存在，而且会给它一个适当的空间允许自己有负面的情绪。只要我们能成为情绪的主人，不是完全让它左右我们的思想和行为，就可以善用情绪的价值和功能。 在许多情境下，一个人应该泰然接受自己的情绪，把它视为正常。例如，我们不必为了想家而感到羞耻，不必因为害怕某物而感到不安，对触怒你的人生气也没有什么不对。这些感觉与情绪都是自然的，应该允许他们适时适地存在，并缓解出来。这远比压抑、否认有益多了，接纳自己内心感受的存在，才能谈及有效管理情绪。 至于管理情绪的方法，就是要能清楚自己当时的感受，认清引发情绪的理由，再找出适当的方法缓解或表达情绪，我们可以归纳成为以下三部曲。 1. ＷＨＡＴ――我现在有什么情绪？ 由于我们平常比较容易压抑感觉或者常认为有情绪是不好的，因此常常忽略我们真实的感受，因此，情绪管理第一步就是要先能察觉我们的情绪，并且接纳我们的情绪。情绪没有好坏之分，只要是我们真实的感受，我们要学习正视并接受它。只有当我们认清我们的情绪，知道自己现在的感受，才有机会掌握情绪，也才能为自己的情绪负责，而不会被情绪所左右。 2. ＷＨＹ——我为什么会有这种感觉（情绪）？ 我为什么生气？我为什么难过？我为什么觉得挫折无助？我为什么……找出原因我们才知道这样的反应是否正常，找出引发情绪的原因，我们才能对症下药。 3. ＨＯＷ――如何有效处理情绪？ 想想看，可以用什么方法来纾解自己的情绪呢？平常当你心情不好的时候，你会怎么处理？什么方法对你是比较有效的呢？也许可以通过深呼吸。肌肉松弛法、静坐冥想、运动、到郊外走走、听音乐等来让心情平静，也许会大哭一场、找人聊聊、涂鸦、用笔抒情等方式，来宣泄一下或者换个乐观的想法来改变心情。 ### 四）、有趣的情绪小故事 #### （一）笑是良药 1. 笑声护士。 据美国芝加哥《医学生活周报》报道，美国一些大型医院和心理诊所已经开始雇用“幽默护士”。她们陪同重病患者看幽默漫画并谈笑风生，以此作为心理治疗的方法之一。幽默与笑声，帮助不少重病患者或情绪障碍者解除了烦恼与痛苦。 笑声一般都是人们所喜欢的，每个人都不愿意看到朋友愁眉苦脸。最新的医学研究发现，笑口常开可以防止传染病、头痛、高血压，可以减轻过度的精神压力，因为欢笑可以增加血液中的氧分，并刺激体内免疫物质的分泌，对抵御病菌的侵袭大有帮助。而不笑的人，患病几率较高，而且一旦生病之后，也常是重病。美国医学界将欢笑称为“静态的慢地”。笑能使肌肉松弛，对心脏和肝脏都有好处。如果生活中没有时间去慢跑，我们可以每天多笑一笑，甚至哈哈大笑几十次，以调节身体状态，增进健康。 耶鲁大学心理学教授列文博士说：“笑表达了人类征服忧虑的能力。”笑又往往是人欢乐的一种表达，之所以欢乐，是人体在生理上产生了某种愉悦的缘故。 赶紧笑起来吧！别等到生病以后才咧开嘴。 2. 笑的妙用。 名医张子和曾采用使人发笑疏导法治愈了一个人的怪病。当时有个官吏的妻子，精神失常，不吃不喝，只是胡叫乱骂，不少医生使用各种药物治疗了半年也无效。张子和则叫来两个老妇人，在病人面前涂脂抹粉，故意做出各种滑稽的样子，这个病人看了不禁大笑起来。第二天，张子和又让那两个老妇人做摔跤表演，病人看了又大笑不止。后来张子和又让两个食欲旺盛的妇人在身边进餐，一边吃一边对食物的鲜美味道赞不绝口，这个病人看见她俩吃得津津有味便要求尝一尝。从此她开始正常进食，怒气平息，病全好了。 著名科学家法拉第年轻时，由于工作十分紧张，导致精神失调、身体非常虚弱，虽然长期进行药物治疗却毫无起色。后来一位名医对他进行了仔细的检查，但未开药方，临走时只说了一句话：“一个小丑进城胜过一打医生！”法拉第对这句话仔细琢磨，终于明白了其中的奥秘。从此以后，他经常抽空去看马戏、滑稽戏与喜剧，经常高兴得开怀大笑，愉快的心情使他恢复了健康。 3. 欢笑诊所。 据说现在每天早上，在印度孟买的大小公园里，可以看见许多男女老少站成一圈，一遍又一遍地哈哈大笑，这是在进行“欢笑晨练”。印度的马丹·卡塔里亚医生开设了150家“欢笑诊所”，人们可以在诊所里学到各种各样的笑：“哈哈”开怀大笑；“吃吃”抿嘴偷笑；抱着胳膊会心微笑……以此来治疗心情压抑等心理疾病。 4. 笑能拯救生命。 加利福尼亚大学的诺曼教授，40多岁时患上了胶原病，医生说，这种病康复的可能性是五百分之一。他按照医生的吩咐，经常看滑稽有趣的文娱体育节目，有的节目使他捧腹大笑，有的节目使他从心底发出微笑。他除了看有趣的节目，平时还有意识地和家人开开玩笑。一年后医生对他进行血沉检查，发现指标开始好转了。两年以后，他身上的胶原病竟然自然消失了。为此，他撰写了一本《五百分之一的奇迹》，书中提出：“……如果消极情绪能引起肉体的消极化学反应的话，那么，积极向上的情绪就可以引起积极的化学反应……爱、希望、信仰、笑、信赖、对生的渴望等等，也具有医疗价值。”中外许多心理学家、运动学家认为，一般性的笑，能使隔膜、咽喉。腹部、心脏、两肺，甚至连肝脏都能获得一次短暂的运动。捧腹大笑，它还能牵动脸部、手臂和两腿肌肉的运动。当笑停止之后，脉搏的跳动会低于正常的频率，骨骼肌也会变得非常松弛。 5. 装笑也管用 美国一广告公司的部门经理弗雷德工作一向很出色。有一天，他感到心情很差。但由于这天他要在开会时和客户见面谈话，所以不能有情绪低落。萎靡不振的神情表现。于是，他在会议上笑容可掬，谈笑风生，装成心情愉快而又和蔼可亲的样子。令人惊奇的是，他的这种心情“装扮”却带来了意想不到的结果——随后不久，他就发现自己不再抑郁不振了。 美国心理学家霍特指出，弗雷德在无意中采用了心理学的一项重要规律：装着有某种心情，模仿着某种心情，往往能帮助我们真的获得这种心情。 有些人通常在情绪低落的时候避不见人，直到这种心情消散为止。这么做果真是好办法吗？ 多年来，心理学家都认为，除非人们能改变自己的情绪，否则通常不会改变行为。当然，情绪、行为的改变也不是说变就变、想变就变的“瞬间”现象，而是有一个心理变化的内在过程。心理学家艾克曼的最新实验表明，一个人老是想像自己进入了某种情境，并感受某种情绪时，结果这种情绪十之八九果真会到来。需要注意的是：随着年龄、性别、职业、性格等因素的不同，情绪变化的程度和时间也不一样。情绪有了变化之后，伴随每一种情绪的外在表现，生理反应也会出现变化。华东师范大学心理学系孔教授研究后认为：一个故意装作愤怒的实验者，由于“角色”行为的潜移默化影响，他真的也会愤怒起来，表现在待人接物、言谈举止等方面；同时，他的心率和体温（心率和体温都是愤怒的生理反应指标）也会上升。 为了调控好情绪，不妨偶尔对自己的心情进行一番“乔装打扮”。 #### （二）有害的情绪 1. 可怜的小羊。 历史上有个著名的医师叫阿维林纳，他对动物的生存环境做过一个试验。他把两只小羊同样喂养，其中一只放在离狼笼子不远的地方，由于经常恐惧，这只小羊逐渐消瘦，身体衰弱，不久即死了；而另一只小羊因为放在比较安静的地方，没有狼的恐吓，而健康地生存下来了。 2. 致命杀手“生气水”。 最近，美国一些心理学家做了一项实验，他们把正在生气的人的血液中所含物质注射到小老鼠身上，并观察其反应。初期，这些小老鼠表现呆滞，整天不思饮食。几天后，它们就默默地死掉了。美国生理学家爱尔玛为了研究情绪状态对健康的影响，设计了一个很简单的实验：他把一支支玻璃管插在正好是0℃的冰水混合物容器里，然后分别注入人们在不同情况下的“气水”，即用人们在悲痛、悔恨、生气时呼出的水汽和他们在心平气和时呼出的水汽作对比实验。结果表明，当一个人心平气和时呼出的水汽冷凝成水后，水是澄清透明、无杂质的；悲痛时呼出的水汽冷凝后则有白色沉淀；悔恨时呼出的水汽沉淀物为乳白色；而生气时呼出的“生气水”沉淀物为紫色。他把“生气水”注射到大白鼠身上，几十分钟后，大白鼠就死了。由此可见，生气对健康的危害非同一般。 有分析表明：人生气10分钟会耗费大量精力，其程度不亚于参加一次3000米的赛跑；而且生气时的生理反应也十分剧烈，分泌物比其他任何情绪状态下的分泌物都复杂，且更具毒性。因此，动辄生气的人很难健康长寿（很多人都是给气死的）。生气发怒引起的后果有： 伤心损肺：气愤必然引起心跳加快，心律失常，使心脏受到邪气的侵袭，诱发心慌心痛，呼吸急促，引发气逆胸闷、肺胀、咳嗽及哮喘。 伤脾伤肝：生气时除了伤脾脏外，还会导致尿道受阻或失禁，并使肝胆不和、肝部疼痛。 伤脑失神：人在发怒时心理状况失衡，情绪高度紧张，神志恍惚。在这样恶劣的心理状态和强烈的不良情绪下，大脑中的“脑岛皮质”受到刺激，时间久了就会改变大脑的正常运作。 ================================================ FILE: docs/emotion/eq.md ================================================ # 提高情商从哪几方面入手 情商的重要性，想必你一定知道。 情商高的人，他们往往是最受欢迎的人，而他们，也拥有许多的支持者。 那你知道，高情商的表现都有哪些吗？ ## 不抱怨、不指责、不批评，有着正面的积极情绪，且充满热情 生活、工作、感情，确实有时候很苦，有时候会让你觉得很烦。 而在这些因素的影响下，你所表现出的种种行为，都能体现出你情商的高低。 一个高情商的男人，即使生活再苦，再烦，他都能保持积极的情绪，让好的心情常伴左右，也不会被这些因素影响到生活、工作。 当你遇到这些情况的时候，你的抱怨、指责，就是负面情绪。也许，在你发泄的时候，你感觉挺好，但事后，你一定会反思。 对于解决问题来说，负面情绪起不到任何作用，甚至还会传染给别人，影响别人。 所以，你是否为高情商男人，就看你是否能控制与释放自己的负面情绪。 ## 不轻易麻烦别人 在你遇到问题的时候，你是否会咨询他人呢？ 但，如果次数多，太频繁的去麻烦别人，这会给别人带来不适，他可能就会不愉快，甚至不想帮你。 每个人都是独立的个体，没有谁有义务去帮助你，即便那个是你的朋友、亲人，他们帮，是情分，不帮，是本分。 所以，能够百度解决的问题，就不要总是麻烦别人了，不轻易麻烦别人，是一种基本美德。 ## 懂得聆听 你喜欢倾诉吗？如果别人不认真听，边听边说，不给你解决方案，甚至嘲笑，你喜欢这种人吗？ 懂得倾听和善于倾听，能够让身边的人更加舒适和信任你，这不仅是高情商，更是卓越的社交技能。 有时候，少说多听真的很重要，别人也许是真的想倾诉，想找一个能听得进去，能理解他的人，如果你连听都听不进去，你还怎么获取信息，怎么了解，怎么帮他。 ## 善于观察的赞美 生活中，你喜欢赞美他人吗？ 我们常常能看到，那些喜欢赞美他人的人，他们很受欢迎，也很讨喜，而那些总觉得自己才是最好的那些人，往往是被人讨厌的，因为总感觉他看不起我们。 如果你要赞美，那就发自内心的，真诚的去赞美，当你能够看到别人的优点时，你才会进步得更快。 每个人都喜欢赞美，读书的孩子得到了夸赞，他会努力学习，工作的员工得到了夸赞，他会做得更好，在恋爱中更是如此。 谈恋爱，最后必然是要和对方走一生的，如果在这段感情中，没有赞美，只有看不起，那么，这样的感情又怎么会舒服和自在呢。 ## 记住别人的名字 一个见了很多次的人，你记得他的名字，但他却不记得你的名字，甚至每次都要问你，你会觉得不好受是不是？ 但是，如果在一群人里面，有人能记住你的名字，但记不住别人的名字，那他给你的感觉也会不一样。（这感觉太棒了，所有人的名字你都记得不清楚，但唯独我） 这件事情虽然小，但是，这也是高情商的一种表现，你记住别人的名字，别人会因此更加愿意与你亲近，和你做朋友，而这，也有助于你有更好的发展，有更好的人生。 ## 责任心 别人交给你的任务，你会把这当作是一种责任，并按时完成吗？ 生活中，有太多事情需要我们用到责任心，自己愿意为这件事情负责，或者别人嘱托的，我也会去承担。 当你承担一件事情后，遇到问题，不推卸责任，不逃避，懂得分析问题，解决问题，这就是一个好的责任心。 有时候，你承担的责任比别人多，你觉得亏了，但是往远处看，你在这个过程中获得的成长是比别人多的，而你的责任越大，就说明了你的能力越强。 ## 停止争端 在很多事情上，每个人都有不同的意见，因此，我们可能会吵起来，这个时候，懂得放弃也是高情商的一种表现。 意见不符合，那我就说几句，结果你越来越激动，那我就放弃争端，停止争吵，而这，会让你更加舒适。 我们总会遇到那些不讲道理的人，你能够很明显的感觉到他的情商和智商很低，所以，如果你与他纠缠了起来，你也只会把自己和他拉到同一个水平，有时候，不和傻瓜争论，是件好事。 ## 能独处、懂沉默 每个人都有自己独立的空间，有时候，不去打扰他人，不过分依赖他人，能够合理安排自己的时间，这也是高情商与高智商的一种表现。 你会被别人误解，有时候，不需要积极的去争辩是非，因为别人并不了解你，你也没必要向全世界喊话解释。 在很多社交场合里，你如果懂得适当的沉默，那么，这是一种优秀的品质，因为你要知道，你并不需要通过语言来证明什么。 ## 后记 如何提升情商 1. 停止无意义的抱怨 2. 接受自己的不完美 3. 永远不要停止学习和阅读 4. 你的朋友不需要多，“质”才重要 5. 有时候，换一种生活方式，能帮你拓宽视野，提高情商 6. 做事情时，制定清晰的目标，以扫除浪费精力的事物 每个人都不完美，但你可以努力让自己接近完美，此刻的你，也许情商不高，但人是会变的，一时的不优秀，不代表你一辈子都不优秀，所以，为了更好的明天，去努力吧。 #我见过情商最低的行为，就是不停地讲道理 最近我对“情商”这个词有了新的理解——高情商的人，原来最不讲道理。 ### 情商高的老板少和员工讲道理 梁宁看纸牌屋都跟我们不一样，她看完学到一点：管人的本质，就是管理情绪。 克莱尔一路辅佐伍德，直到当上总统，她永远知道如何管理伍德的情绪。 比如第一集，伍德被他一路扶上马的总统耍了，他在街头闷坐一天，回家后，克莱尔应该干什么？ 给他安慰，给他讲道理？ NO，她直接刺激他，让伍德将桌子上的东西扫到地下摔碎.... 这是干什么？让伍德的情绪快速到达顶点释放。然后，递给他一支烟，让他站在熟悉的窗前。 伍德呢？他的对手都是高智商生物，讲道理、PK逻辑，伍德肯定赢不了，但是他懂得抓住他们的心理，每个掉进伍德坑里的，都是输在情绪上、心理上。 梁宁说：反思我当年创业的时候，管人的能力很差，因为我只会一招——讲道理。这就是情商低啊情商低。 我朋友在一家新媒体公司上班，说特别烦：老板特别喜欢开会，一开会就特别喜欢讲道理，讲道理上瘾，让我们学习某某公司，说人家公司晚上都加班到凌晨....你给我多少钱啊，我给你加班到凌晨.... 所以，那个老板天天讲道理也没用，大家到点就下班走人。 可你想想，马云当年带领18罗汉创业，不光工作环境差，还工资低，又天天加班，可是大家还是不愿意离开。为什么？ 因为马云情商高啊，他能把人都搞得很嗨。 高情商的老板，不跟员工讲道理，聪明的老板讲故事，善良的老板讲KPI(绩效考核方案)，讲利益分配。 当然，对于那些只讲故事画大饼的老板，我想把张小龙的话送给你：我劝你善良。 另外，别以为天天讲很多道理，能帮多大忙。 不能讲道理么？能。 讲一遍，他懂了。他去做，说明他信。他不去做，不是他不懂，是他不信，你再唠叨十遍他也不信。 那怎么办？让他自己去撞南墙，让他去犯错，去经历，然后他就懂了。 **时尚芭莎的前执行主编说：世界上最无效的努力，就是对年轻人掏心掏肺讲道理。** ### 高情商的父母少和孩子讲道理,高情商的孩子少和父母讲道理 管人就是管情绪，管理其实无处不在，人与人相处，其实就是在管理对方的情绪，而管理情绪，最好不要通过讲道理来实现。 前段时间知识星球社群里有个星友提问： > 我女儿上初二，个性外向，喜欢吹牛，智商中上水平。但没有上进心，读书积极性不高，成绩在班上和全校是中上水平，父母的管教没多少效果，请问如何突破，争取有所成长和进步。 01. 你看到的不一定是真的。 初中的孩子都很叛逆，即便内心有上进心，也不愿意表现出来。 按我的记忆，那个阶段的我们，以展示不求上进为酷和潇洒。 02. 初中的学业并不轻松。 没上进心，读书积极性不高，感觉很难把成绩维持在中上水平吧。 03. 爱吹牛，这个没有准确的定义。 我高中时说要考到北京，我大学时说我要留在北京，毕业后我经常说以后我要在北京买房子，我要创业怎样怎样…… 我妈老说你这孩子怎么老吹牛…… 后来，我想做啥时都不愿意跟我妈说了…… 说不定你以为的吹牛，正是她有追求有上进心的表现。 04. 你女儿成绩不错，少管教，多赞美。 成绩中上，性格开朗，还没有学坏，对这个阶段的孩子来说，挺好了。 按这节奏走下去，不会差。有条件的话，寒暑假多带出来旅行，看看更大的世界，体验过更好的东西后，她自然想追求更好的。 反过来说，我们也不要老跟父母讲道理，否则也情商太低了，你讲不通的。 我刚毕业那会儿，我的爸妈经常想说服我让我考研、考公务员。 一开始我会跟他们讲道理，为什么我包括我们这样的家庭不适合考研、考公务员。 公务员可能是最不适合我们家这样的孩子做的工作。 我也可以把知进退和边际效益这样的东西讲得通俗易懂，但是他们能听懂就会理解你么？ 父母很多时候不理解你，原因仅仅是因为观念不同，立场不同，价值观不同。 而且他们“爱你”、“为了你好”...... 所以，我每次高高兴兴地回家，吃完饭就要不高兴的吵一会儿，我讲各种道理来论证。 其实最后我说服他们了，他们也只是表面被说服，心里还是固执的认为自己是对的，不然为什么下次又劝你一番？好像上次白讨论了…… 和父母讲道理，情商太低了。 那怎么办呢？ 他们说，你乖乖听着就行了，非争论做什么。然后剩下的，你全力以赴去用行动和结果证明你按你的路径走是对的，大家皆大欢喜。 ***父母最终希望的就是你有个好的结果，过得好，当你没有、过得不好时，他们当然会按照他们的经验想给你指一条路径，他们没有错。** 但是当你用自己的路径走出一条路来时，他们看到你好的结果，自然也不会去干涉了。 所以后来我就不跟父母争了，我拼命按照我的想法去努力，让他们看到，我可以有体面的工作，体面的收入，有开心的生活...... 现在他们再也不说我了，因为他们看到了。 ### 高情商的爱不讲道理 我最见不得一种事：在北京熙熙攘攘的地铁站里，看到一对情侣大声吵架。我觉得女生都很要面子，你在公共场合这样吼太不合适了。 所以每次看到，我都恨不得过去跟那个男的说：她只是需要一个拥抱，你这个傻子却跟她讲道理。 人家跟你谈恋爱，你跟人家讲道理：讲道理要你干嘛，不如去看书。 所以，恋爱里面更多不是“对与错”，而是“爱与不爱”。 老是讲道理的恋爱不会长久，因为虽然你对，但你不够爱。 高情商的爱，要少讲道理，多哄她，你不想哄，放开，让我来嘛。 结了婚更不能老讲道理，否则你们家，除了吵架就是冷战，尤其是没钱的时候，我小时候在农村学到一个词叫“穷叽歪”，就是越穷的两口子，越爱吵架讲道理。 过穷日子的时候互相扶持，过富日子的时候互相信任，这里面靠的都不是道理，而是爱。 ![emotion](./../img/emotion-1.png) 中国人好客，出于礼貌在外面不会轻易释放自己的情绪，却喜欢回到家里跟自己的亲人讲道理，一言不合就吵架。 两个人情商都低，互不相让，最后就变成互相讲道理，但道理其实是永远讲不清的，这就是家庭矛盾激化的源泉。 **所以，讲道理，请对外人讲，对自己人，讲感情就好。** ### 高情商的人不跟傻子讲道理 前几天，在后台收到一条恶意谩骂的留言。 记得刚入行时候，运营一个10万的大号，自己也在上面写文章，评论区里什么人都有。 大家在网上口无遮拦不用负责，所以经常有那种不和谐的声音，我每次看完都会忍不住回复，给他讲道理，有时候碰到杠的，我们能来来回回说十条，气的吃不下饭，晚上刷着评论区睡不着觉。 现在回想起来，真的是情商低啊。 **人家就是来恶心你的，你还一本正经的跟人家讲道理。** 其实不光是运营公众号，生活中工作中，我们也经常遇到情商低的人，千万不要跟他们讲道理，他们本来就想拉你下水，你别真上当。 高情商的人，会把时间花在喜欢自己的人身上。对讨厌的人，不浪费一点生命。 ### 高情商的人不跟朋友讲道理 我们刚才说了，不跟情商低的人讲道理，为什么跟朋友也要不讲道理？ 其实是从另一个角度讲。 **朋友之间出来吃个饭、聊个天，最怕你喜欢杠，一杠就情商低了。** 朋友就是出来一起放松一下，聊天大部分也是互相吹吹牛扯扯淡，本来很多话就是经不起细细推敲和严密论证的，结果你老是跟打辩论赛似的，动不动就揪住一个点想给大家讲道理。 你讲的对又怎样，你让朋友们不开心呐。 大家是出来玩的，不是出来听你讲课的。 这么好为人师，你怎么不去做自媒体啊哈哈哈哈哈。 **情商高的人，不讲道理，因为他懂人性。** 高情商的人为什么最不讲道理？因为他们都懂一个更底层的道理，这也是我从梁宁老师那里学到的： 人非机器，人有情绪。人与人的相处，充沛的道理是最没趣味的部分。真正让人和人的关系有机互动的，是情绪，彼此能挑动情绪，就是对眼了，就是气味相投了。 大多数情况下，你和你的爱人、同学、朋友、同事，都有类似的知识背景、经验背景，这种东西都有一定的圈层属性，也就是说，这个圈层里的常识性的道理和逻辑，大家都懂，用不着你讲。 比如老板给员工讲的大道理，其实员工大部分都懂，听得耳朵都起老茧了。 人的底层操作系统是情绪，一个人由情绪构成，靠情绪驱动，所以高情商的人，手里把玩的不是道理，而是情绪，他们善于观察、管理双方的情绪，让双方都舒服，让关系趋于持久共赢。 **我见过情商最低的行为，就是不停地讲道理。** ================================================ FILE: docs/emotion/lifetime.md ================================================ # 心态好的人，一辈子都好 ### 01 昨天微信收到一条私信，一位朋友问我： “三十而立的年纪了，没钱、没车、没房，每天麻木的上班下班，浑浑噩噩地过日子，经常会不想活了，但是又没那个勇气，觉得自己特孤独、特没用，茶茶你说，我怎么才能让自己正常点？” 看完他的问题，我反问他：“如果现在你有钱、有车、有房子，你就一定会远离孤独，找到快乐吗？” 他说：“我觉得应该会吧，至少会比现在好。” 对话让我想起了最近特别喜欢的一首歌，是赵轲的《有人》，通篇的歌词都让我很有感触，印象最深刻的是这几句： 有人家财万贯却还失声痛哭，有人身无分文却也活得舒服，有人入不敷出，半杯酒便再无贪图。 我把这段歌词发给他，他回了我一个苦笑的表情，说：“又要费上一盒烟，还有半宿的时间了。” 莫名被逗笑，我想，他其实也许是个很幽默的人，只是在那个时间段，被负情绪迷了心吧。 听过很多人说，越长大越发现，钱真的是个好东西，钱能解决大多数的烦恼，也能换来大多数的快乐。 必须承认，这句话在某些时候的确有些道理，我们可以用钱买到自己想要的东西，朝自己想要的物质生活更近一步，但只是某些时候而已。 生而为人，个人有个人的难，个人有个人的甜。 有钱有富贵的活法，没钱有穷困的活法，但没钱的人，却并非一定不比有钱人幸福。 就像狄更斯说的：一个健全的心态，比一百种智慧更有力量。 我们都会有想不通、看不开、放不下的时候，这是我们的业障，也是幸福和快乐前面的遮挡，这样的时候，往往一叶障目，不见泰山，而当我们拨开叶子，才发现天高云阔，豁然开朗。 好的心态，就是我们拨开叶子的，最有力的力量。 ### 02 生活中的我们，有时候只是被一点点乌云挡住了心里的阳光，却没能及时驱赶开，结果自己的世界，亮度变得越来越暗。 情绪和心魔其实都不可怕，可怕的是受其影响却不自知，甚至自甘堕落其中，而不是想着如何调整自己。 人生苦累，大多都是跟自己较劲，有一些负担在心上，明明是可以及早舍弃掉的。 重要的并不是你有多少钱，也不是你有多少牛逼的朋友，更不是你有多显赫的名利富贵，这些外在的东西，撑住的只是你外在的面子。 面子的状态随时都在变，而里子，也就是你的心，你的心态，才是人生大方向的主宰。 心小了，小事也成了西瓜大事，郁结在心里化不开；心大了，大事也成了芝麻小事，动动手就掸走了。 心态好的人，才能一辈子都好。 ### 03 常会这样的留言，说日子苦，工作累，赚钱难，或者是老公不懂事，孩子不省心等等。 其实每个人当初的选择，都是为了让自己更开心，如果同行者能增加你的快乐，那就聊得多一些，如果不能，也别放任自己打烂一手好牌。 毕竟，说白了，人都是为自己活的。 所以，能用手拥抱的，就别用嘴去争吵，纵使改变不了别人，也别轻易让自己变得暴躁易怒。 要记得，工作也好，生活也罢，都是为了幸福，一定别被它们反过来束缚住了。 很喜欢罗兰的一句话：个人有个人理想的乐园，有自己所乐于安详的世界，朝自己所乐于追求的方向去追寻，就是你一生的道路，不必抱怨环境，也无需艳羡他人。 人生是没有回头路的，做再多的设想也是无意义的。 与其纠结抱怨，不如踏实努力，但前提是，你不能在这条自己选择的路上，让心变得贫瘠和困苦。 我知道做起来很难，但人生不就是这样一场漫长的修行吗？见天地，见众生，更要见最真实最平和的自我。 境随心转，物由心造，相由心生，你的心态越好，生活就对你，越发温柔。 愿余生，不为琐事扰，之一笑而过，用最好的心态，过最好的生活。 共勉之。 ================================================ FILE: docs/emotion/livefail.md ================================================ # 如果你觉得自己活得很失败 我之前也觉得自己特失败，就是，身边朋友，不是去了名校深造，就是拿到了漂亮的offer，噢，也包括我这个朋友，我当时给他发“恭喜”的时候，还在想，大家都混得蛮好。 **谁不想要成功呢。往往是现状越难言，越想要180度转变的电影式人生——“我也想正儿八经地觉得自己了不起啊，我也想成为父母的骄傲啊”。** 我有段时间，我也很焦虑，觉得迷茫。想着想着就难受得很了，我要的那个“成功”，离我真的太远了。 >我望着跑道终点想，太远了，我怎么离那里那么远啊。焦虑得什么都做不了，一点用都没有。 **但这种情绪其实没那么可怕。我怎么走出这种情绪的呢，不难，就是，push自己，做点确定有用的事。不管多小，开始做了，你的状态自然就会不一样。烦闷一个下午，边刷手机边“恨自己没用”，但坐到书桌前拿起笔列一下to do list，你的心情就会不同。** 比如看书，比如学习，可能今天和明天一样，但是“量变累积质变”，我跑得不管多慢，总归能离终点近一点啊。 我当然不是说我现在就是“成功”了，我只是不那么纠结“成功”了，我喜欢自己写东西，而且觉得明天我会写得更好，我比昨天好一点儿，就算一次圆满。 越是讨厌现状，越不能去望金字塔尖，为了看高处就停下脚步，是最蠢的事。自怨自艾本质就是浪费时间，而且陷在里头也不快活，多傻啊。 觉得自己失败这件事，太容易了。有几个人真正觉得自己是成功的？我甚至怀疑，是不是其实每个人心里都有一小块儿，是觉得自己是失败的。 **但感到自己活得失败，其实没那么糟，觉得自己成功的那一刻，就到头了** 怎么说呢，大家都是听着“你将来会有出息的”这样的话长大的，年岁渐长，每个人都或多或少开始觉得，“我不止活成这样”。 但这是没关系的。对现状不满根本是常态，因为永远有“更好”，所以才会觉得现在“失败”。 **试着把失败感当成礼物。你觉得失败，是因为你把很棒的事情当成参照物。你觉得自己还能变得更好。** 我学人力资源遇见一个女孩，很认真一姑娘，但记忆力很差，我考出来的时候，她还差一科。 她觉得自己差劲，逼自己每早晚记忆。 要知道，“匮乏只是暂时的阶段”。世界上再没有比改变更稳固的状态，多读一本书，多背一个单词，多跑一千米，只要做就对了，多做一点，就多改变一点，和刚刚那个陷在失败感里的自己分道扬镳。 **觉得自己失败，其实是一件值得骄傲的事，因为是你首先意识到这一点，是你自己，而不是其他任何人。这是你的自省，领悟，决心，是赶在真的“失败”之前的反应机制。** 没有人为自己的恰得其所而不甘心，觉得失败，是因为心里知道自己的最佳水平在更高的位置，却还没能走到那里去。 知道自己的潜力，并且为之努力，不是好事吗？只要不停下来，就在进步，就离脱离失败，更进一步。 我今天觉得自己活得很失败。但只要明天进步一点儿，我就离想要的那个成功更近了。 **我的人生起点并不重要，重要的是我的人生终点能在哪里** ================================================ FILE: docs/emotion/lookbook.md ================================================ # 我们为什么要读书？这是让我印象最深的答案 我一直有每天看书的习惯，即便工作再忙，也仍然坚持着。比如在地铁拥挤嘈杂的环境中，我掏出一本书看，就像肩上卸下一个大包袱。在书籍的世界里，我的心灵获得了放松。 于我而言，看书的行为很单纯。你不需要跟谁打招呼，说客套、不必要的话，你只管直来直去，看想看的内容，学想学的东西。你可以随时打开它，与它对话交流，也可以随时关闭它，与它分开或隔离。 你去看它时，它是欢迎你的。你走时，它也不会挽留。知识的大门，随时向你敞开，而你却可以来去自由，不必顾虑太多。 曾听过这样一种说法：“读书好比‘隐身’的串门儿。要参见钦佩的老师或拜谒有名的学者，不必事前打招呼，也不怕搅扰主人。翻开书面就闯进大门，翻过几页就登堂入室；而且可以经常去，时刻去，如果不得要领，还可以不辞而别，或者干脆另找高明，和他对质。” 如果说我们为人处世有太多需要周全的地方，但读书这件事不仅可以增长见识、拓宽视野、打开思维，更重要的是，在整个过程中我们可以随心所欲、无拘无束、毫无压力。 我一个朋友在小县城安家，已结婚生子，终日与锅碗瓢盆相伴。虽过的是普通人的生活，看书这个爱好却始终未变。 刚开始许多人都投来嘲笑的目光，甚至有人觉得她装。因为读书多，好像也没有让她表现出特别的优势。 但她自己知道，因为读书，她的虚荣心变少了，她很少因为跟别人攀比而嫉妒或自卑；因为读书，她变得更低调了，她以前特别喜欢炫耀，现在却很少显摆卖弄；因为读书，她的心态更乐观、状态更积极了，她以前遇事易慌张、斤斤计较，现在学会了大度和淡定。 就算最终跌入繁琐，同样的工作，却有不一样的心境；同样的家庭，却有不一样的情调；同样的后代，却有不一样的素养。读书，就是让不一样发生的变量。某种意义上说，读书就是为了让我们尽量摆脱平庸。 虽然生活朴素，但我们内心丰富；即使深陷泥泞，也依然可以仰望星空。 成年后，我们读书或许会有了更多功利性。刚开始看书时，我也是为写文章积累素材，可后来发现自己越来越离不开书。 在这里，我可以感受到纯粹的快乐。书籍就像我的知己、导师，当我迷茫、焦躁、不知所措时，它给我指明方向，可以安抚我、引领我、激励我。 在这里，我可以认识和探索世界。眼睛看不到的，读书可以；脚步不能丈量的，读书可以；身体无法抵达的，读书也可以。哪怕我们活在方寸之地，依旧可以拥有大境界和大格局。 在这里，我们可以见天地、见众生、见自己，即便身处柴米油盐中，也不会被细碎和庸常的生活捆绑。我们会变得越来越通达坦荡，不偏执、不世故。 我们为什么要读书？大概就是为了遇见更好的自己，成为一个有温度、懂情趣、会思考的人吧。 ================================================ FILE: docs/emotion/losecome.md ================================================ # 所有的失去，都会以另一种方式归来 `` 曾有人问我，失去的东西回来了还要吗？我说，就像曾经丢了一粒扣子，等找到那粒扣子的时候，我早已经换了一件新衣服。 `` 大部分的痛苦，都是不肯换场的结果，大部分的哀伤，只是源于执念太深。 好在，上天是公平的。 有人惹你哭就有另一个人来逗你笑，所有的失去都会以另一种方式得到补偿。 所有的痛苦都会过去的，所有的失去，都会以另一种方式重新拥有。 ## 离散 我的好朋友年前结婚了，他的新娘在一边满面春风地和我们寒暄，递邀请函。 然而，去年的这个时候，我还记得他一个人眼里噙着泪，拖着行李漂洋过海去和前任谈分手。 与其说是谈分手倒不如说是尽最后的努力挽留。毕竟那是一场从青涩到成熟一共7年，横跨了他几乎整个青春的爱情。但是女孩在出国的时候，还是另结新欢。接到分手电话的那一刻，他形容说感觉心口硬生生被挖走了一块肉。 紧接着，抱着非要问清楚为什么的不甘心，倔强的他花光了为数不多的积蓄，非要绕大半个地球去要个答案。 你永远挽回不了一个不爱你的人，他亦是。 其实谁都没有错，都是命定的缘分，来的时候无法躲避，走的时候无法挽留。 他跟你终究不合适，就像37码的鞋遇上42号的脚一样不合适，不怪鞋也不怪脚。 回来的时候，他在飞机的角落里泣不成声。 原以为此生不会再爱，但兜兜转转，还是遇上了一个更好的人。 **能走开的，都不是最爱；走不开的，是命定。** 好在，当他绝望得以为此生不会再爱，转角还是遇到了另一个人；你觉得心口扎上一刀痛得快要死去，时间还是将它治愈了。 宫崎骏老先生说：没有不可以治愈的伤痛，没有不能结束的沉沦。 很多时候，很多事情，走不圆满的结局，错的不是人，是时间，是命运。 但所有失去的，会以另一种方式归来。比如，那个让你曾放下所有骄傲去迁就的人离开了，没关系，自有另一个人帮你重塑起你骄傲，护你一生安好。 你有过的每一场心碎，最后都会换一个人来，帮你一片片拾起，拼凑到如初。 就像张小娴说的那般： > 总有一天，你会对着过去的伤痛微笑，你会感谢离开你的那个人，他配不上你爱、你的好，你的痴心。他终究不是命定的那个人。幸好他不是。 ## 世事难料 其实，生命本身就是个在“不断失去”与“不断得到”中循环往复的过程。 往往你以为是失去，回过头来看，都会发出“幸好当初没有”的感叹。 甚至当你被逼到绝境不得不放手一搏时，不远的将来你也会感恩当时没有死于安逸。 你以为的失去，其实不过是为了迎接你一种“得到”。 只是，只有到最后关头，你才会恍然大悟生命到底给你布下了什么套路。 比如那个《塞翁失马》的故事： 塞翁的马走失了。邻居们都来宽慰他，塞翁则笑笑说:“丢了一匹马损失不大，没准还会带来福气。” 过了没几天，丢马不仅回家了，还带回一匹骏马。 大家纷纷来向塞翁道贺，塞翁反倒一点高兴的样子都没有，忧虑地说:“白白得了一匹好马，不一定是什么福气，也许惹出什么麻烦来。” 结果没几天他的独生子就因为骑那匹骏马摔断了腿。大家同情塞翁时，他却很乐观:“或许是福气呢。” 不久，匈奴兵大举入侵，青年人被应征入伍结果战死，然而塞翁的儿子因为摔断了腿不能入伍，逃过一劫。 很多事情，自有天机。 你看不透天机，但是，与其在那种困境中垂头丧气，一蹶不振，还不如振作起来去寻找新的可能。 抱着积极乐观的心态，不畏艰难险阻，或许一个转身，你就撞上机遇了。 我有个朋友，自青年时立志要当作家。 但连续考了三年的文学研究生都没考上，一度沮丧到几乎万念俱灰。 当他以为此生与梦想无缘，正闲赋在家思考余生去处时，恰逢那时新媒体兴起。 因为热爱写作，所以他闲来无事就开了账号写写这三年的感悟，结果赶上了风口早班车，加上扎实的文学功底，文字和经历都引起很多人的共鸣，后来便成了一名小有名气的作家。 世事就是这么难料，但付出和收获永远是对等的，只是，你不知道努力最终会以什么形式回报到你的身上。 所以，没有什么事是绝对的好与坏的。 人生就像个万花筒，有时候换个视角就可以看到不一样的风景。 换个角度看问题，柳暗花明又一村。 ## 相信 作家耿帅说过： > 生活还没有教会我们一笑而过的本领，我们还是会一次次地摔倒，只是不会再那么害怕疼痛了，没有了软肋，也就不需要铠甲，爬起来拍拍土，继续向前走，伤口总会愈合。 其实，在成年的世界，没有谁是容易的。 在电影《这个杀手不太冷》里，玛蒂尔达问杀手“是不是人生总是如此艰难，还是只有童年如此？” 杀手莱昂回答她“ Always like this（总是如此）。” 每个人都在自己的世界里，承受着这样或那样的痛。 所幸，你永远都有时间重头再来。 失去一个人的时候，你一定也以为此生注定孤独。但你也可以转角就遇到一个爱你如命的人。 当你被生活折磨得痛不欲生时，可当你迈过那个坎，回顾过往突然意识到那也是命运的另一种馈赠。 **人，正是在承受一次次的失去中，一点点成长为更为强大的自己。** **你所遇见的每一个人和看到的这个世界，只是为了让你完成一场人生的修行。** 遇到的人，历经的事，都有其意义。 请相信，如果事与愿违，那一定是另有安排；所有失去的，都会以另一种方式归来。 ================================================ FILE: docs/emotion/onepath.md ================================================ # 余生，学会一个人走，不管有没有人陪 ### 孤独 很多人说，人生总是有诸多无常，起伏波折是上帝给每个人的考验。 是啊，一帆风顺，大多只是一种奢望。 到了一定的年龄，岁月总是如期而来，磨难总是不请自到，孤独总是突如而至。 有时候，面对困难，连个可以寻求帮助的人都没有。 生命中的人，来来往往，路过的多，留下的少，漠不关己多，感同深受少。 我们不得不承认，人生的路途注定是单枪匹马，没有人一直陪伴在我们身侧，就如没有一朵花会永远盛开。 ### 阿甘正传 不由想起电影《阿甘正传》的男主阿甘： 先天心智不足，饱受同龄人的歧视，幸亏有母亲的照顾，爱人的陪伴。 可后来母亲去世，爱人出走，朋友牺牲，他们都只陪伴了他极为短暂的时光，更多的时候是他一个人在走、在跑、在闯荡、在前行。 越年长越觉得，很多事都只能依靠我们自己。 心里的委屈，只能说给自己听，生活的困难，只能自己一个人过，人生的道路，只能自己一个人走。 因为除了我们自己以外，没有人能永远被我们依赖，生我们养我们的人，会老，疼我们爱我们的人，会走。 从始至终面对这个世界荣辱悲欢的，只有我们自己，人生的酸甜苦辣，也只能我们自己体会。 因此我们必须一个人昂首挺胸走在自己的道路上，像这句话说的那样： “把自己当成一支队伍，对着自己的头脑和心灵招兵买马，不气馁。” 只有学会一个人面对生活的苦难，才能真正学会独立。 ### 勇往直前 当我们不再因为生活的苦难抱怨命运，不再因为独处感到孤单，以微笑面对生活。 相信此后无论是否有人陪伴，我们都已强大到能一个人面对生活的考验。 别抱怨命运对自己的不公，别被生活带来的失意打倒。 有时候，我们以为一个人跨不过去的坎，一咬牙，可能就已经跨过去了，我们以为等不来的阳光，一回头，也已经驱散了漫漫长夜。 回头发现，那些难熬的日子，其实是自己美好未来的奠基石，孤独带给我们的并不只是阴郁和失落，还有渐渐沉淀下来的沉稳与坚定。 人活一辈子总要经历些许坎坷，没有什么是扛不过去的。 你只需一路向前，披荆斩棘，总有一天，你会一个人熬过那些苦难。 欲戴王冠，必承其重。 相信命运不会辜负每一个用力奔跑的人，暴雨的终点，是一片草原。 答应自己，余生，无论前行路上是否有人陪伴，都要学会一个人走，好吗？ ================================================ FILE: docs/emotion/selfdiscipline.md ================================================ # 关于自律，80%的人都理解错了 ### 立Flag 2019年的份额已经过去了超过1/24。也就是说，如果你计划今年要跑100公里的步，到今天应该已经跑了4公里。你的Flag还好吗？ 看过一项研究说，只有8%的人可以实现自己的新年计划，而大部分人都会在新年的第二周放弃计划。我们那么渴望自律，却也那么容易放弃。 不知道你有没有这样的经历，越想戒掉的垃圾食品，越会在自己的舌尖回味；睡前越提醒自己不该刷朋友圈、微博，越会刷到停不下来。 太多事实证明，强行的自我压制往往是无效的。甚至有时候，这种自控可能是失控的开始。在我们强行压制欲望的时候，往往会产生相反的效果，让我们更想去做这件我们要求自己不要去做的事。 我们很多时候做不到长久的自律，恰恰是因为我们对自己压制得太狠，消耗掉了我们的意志力。 ### 实例 我有个同学，办理了5年的健身卡，据说比很多3年的健身卡都便宜。办完卡后，他再也没去过那家健身房，甚至忘记了自己还有这么张卡。 卖时间长且算下来相对便宜的年卡，是健身房的惯有招数。但对我们来说，如果你也下意识觉得“这卡那么便宜”时，就是在暗暗给自己的行为贴上了一张“便宜”的标签。如果你把自己的自律，从一开始就贬低得如此廉价，那你心里就会默认自己接下来的坚持一样低廉。 有一个朋友做了一个阶段阅读计划，发了条朋友圈：求监督。 隔段时间我问他进展如何，他说，当时以为这么多人点赞会给自己动力和压力，觉得实现不了会很丢脸，可结果好像也没什么改变，计划依旧失败了。 表面看来，公开目标的行为的确提高了违约成本，但这种违约成本是建立在他人对我们的评价之上。心理测试证明，告诉别人你的目标，大家也承认你的目标，会使你产生一种目标已经达到的满足感，反而会让你对完成目标这件事本身的动力降低。 当我们在朋友圈晒下今日运动成果后，我们错误地暗示了自己“我今天已经达到目标了”，于是往往下一步就是大吃一顿。我们每年都满腔热情地立志，要坚持读书、坚持早起、坚持锻炼……可似乎失败却周而复始。 激情并不可控，如果立下的豪言壮志只是凭一腔激情，没过几天就会被打回原形。 ### 目标 目标，是比激情更靠谱的东西，因为它是固定而明确的。 一个切实的目标，会告诉我们从手中最简单最基础，也是最繁琐最无趣的起点开始努力。此时我们迈出的每一步，都是对目标的承诺。只有把模棱两可的热爱变成斩钉截铁的目标，把模糊的“我喜欢”转变成清晰的“我要完成”，我们才能到达终点。 自律，只是一个手段。如果我们执泥于这个手段之中，总想着坚持的痛苦，那最终自律只能沦为一种形式。透过自律行为本身，而关注自律的结果，我们会发现，我们自律，终究是为了实现一个目标。 每天跑步3公里，是为了拥有一个健康的体魄；每个月读1本书，是为了让自己更多地了解这个世界。如果我们摆脱对过程的执着，专注于目标本身，自律便不再是一种负担，目标会引导我们一步一步地完成该做的事。 真正能自律的人，可能不是那些最有意志力的人，而是在面对行为和意志的冲突时仍然不迷失方向的人。 自律也不一定就是每天坚持做某一件事情，而是长期朝着一个目标前进，把实现目标需要做的事情当成自己的习惯。哪怕累了要歇一会儿也没关系，只要你坚持自己的方向，养足精神后依然可以所向披靡。 只要是离目标越来越近，你就做到了自律。 ### 后记 自律，简单来说，就是朝着自己心中的目标去做，付出的或多或少，进步或大或小都无所谓，只要能越来越接近自己既定的目标就好。 ================================================ FILE: docs/emotion/threeheart.md ================================================ # 人生有这三种好心态 ### 第一，放宽心。 许多人会因为人与人之间的隔阂或矛盾而感到不愉快，但随着年岁渐长就会发现，许多事真的没必要。 比如职场上的明争暗斗、生意上的尔虞我诈、竞争对手的互相拆台，我们以为战胜了别人，就赢得了想要的名和利。等到了后来才明白，人与人之间的互相信任、理解和包容，远比多挣来的一分半文更有价值。 我们活着，并不是为了一些虚名浮利。能让人感到快乐的，往往是一些看似无足轻重的小事，比如同事之间的互相理解、同行之间的互相帮忙，甚至是对手之间的互相鼓励。一个人好不算好，大家好才是真的好。 不要因为贪婪，故意去挡别人的路，因为大家一起走，路才会越来越宽。不要因为嫉妒，故意去坏别人的事，因为别人过不好，你也不会很好过。不要因为好胜，故意去争不该你得的东西，因为德不配位，往往有余殃。 钱永远也挣不完，福永远也享不尽。但为人厚道善良，吃点亏、让几步，却可以给你带来源源不断的好运气。 ### 第二，不纠缠。 你有没有为打翻的牛奶哭泣过，为选错的路沮丧过，为不值得的人伤心过？人生总有各种大大小小的遗憾，比起继续纠缠，更好的选择其实是及时止损。 失去的东西，无论你再怎么挽回，也都失去了。爱错的人，无论你再怎么反悔，也都爱错了。大部分时候，我们把太多值得去追求、探索、体验的时间和精力，耗费在了不值得、已经错过，或是无法改变的人、事、物上。 但与其和过去的错误较劲，不如接受它、面对它、改变它。 首先你要知道，时光不会倒退，只会继续往前，你无法把一切都重来。其次你要懂得，没有谁足够完美，错了就要认，认了以后才可能改。如果只是一味逃避，那么日子只会越过越糟。即便有些事已无法改变，我们仍可改变看待事情的心态。 ### 第三，少计较。 你有没有为了什么事，耿耿于怀过？比如领导多给你安排了一点工作，你就表现出各种不高兴；比如朋友少给了一顿饭钱，你总是铭记在心，试图让对方还回来；比如爱人多说了一句狠话，你就拿着鸡毛当令箭，随时旧事重提，嘴巴不饶人。 其实，很多事我们大可不必去计较，甚至计较起来也没有多大意义。有句话说得好，世界上最宽阔的是海洋，比海洋更宽阔的是天空，比天空更宽阔的是人的心灵。 当你有能力时，多干点事，不仅锻炼了你，领导也会看在眼里。当一顿饭委屈不了你时，多结几次账，朋友更忘记不了你。当几句气话对你产生不了多大影响时，不去较真好强，爱人才会把你的好放在心底。 如果过度计较一些鸡毛蒜皮的事，反而会因小失大。当你把格局放大一点、境界提高一点、心量放宽一点，你的纠结、痛苦、不如意也会少很多。 就如你的眼睛，如果只盯着脚下的方寸之地，自然没有瞭望远方时看到的风景那么多、那么美、那么开阔。 ================================================ FILE: docs/emotion/twopath.md ================================================ # 往后余生还很精彩，别被熬夜拖垮了 ### 熬夜前因后果 我特别理解熬夜人的苦衷：也许白天时间不够用，只能利用晚上充实自己；也许工作所迫，不得不通宵干活；也许平时有太多身不由己，只能在夜深人静时自我放逐。 有人把熬夜细分为“被迫式熬夜”“习惯性熬夜”和“报复性熬夜”。比如“报复性熬夜”，意思是白天没有自己的时间，被工作、学业和社交所累，只能在深夜做自己想做的事。 许多人对此很有共鸣：“工作真的太忙了，到晚上才有时间打会儿游戏。”“终于放松下来了，有自己的时间了，必须做点事情来度过。” 其实不管哪种熬夜，最终报复的还是自己的健康和人生。 以前，我对熬夜不以为然。直到身边一个同事倒下后，我才心有余悸。因为赶一个项目，所有事情他一手操办，夜夜加班到凌晨两三点。连续熬夜一个多月后，突然有一天他晕倒在地。送到医院的时候，医生说再晚来一步就迟了。 同事跟我说：怕了。以前总觉得身体扛得住，可这样在鬼门关走一遭你就会明白，死神把你撂倒时，一声招呼都不会打。 事关生死，不应存侥幸之心。好好睡觉，本就应当是人生的头等大事。 ### 不熬夜的人 我一个大学舍友，晚上十一点睡觉，早上六点起来，雷打不动。白天该做完的事决不拖到晚上，把自己的时间安排得妥妥帖帖，大学四年过得不慌不乱、不忧不惧。 当我们笑他错过了夜生活的丰富多彩，笑他没看过夜里十二点繁星璀璨的星空时，他却笑我们从没看到早上六七点的太阳初升，未曾体会把一天时间拉长的感觉。 知乎上有个问题：曾长期熬夜的人，成功地养成早睡早起的习惯，是一种怎样的体验？ 网友们都不吝啬赞美之词：“以前上午的时间都是荒废的，如果是上班的日子也是没精神的。现在是早上不到6点起来跑步，上午的时间拉长了好多。”“整个人看起来精神抖擞，不会再脱发，不会再病恹恹。” 如果我们想掌控好自己的生活，不如先试着从早睡早起开始。 有人说，“每天清晨的第一道曙光，可以神奇地治愈昨日的伤痛。”是的，与深夜相比，清晨另有一种美好。 ### 如何做 有人毫不留情地指出许多人熬夜的状态：白天，关注养生，了解人类极限，贪生怕死，惜命如金。夜晚，追最好的剧，聊最嗨的天，毫不畏惧，视死如归。 其实不少人都尝试过早睡早起，但有的以失败告终。那到底如何才能做到早睡早起呢？ 首先，提高效率。扪心自问一下，你在工作学习时有全身心地投入吗？是不是总得玩一下手机，是不是没一会儿就要嚼些零食磨蹭半天？有句话说得好：无论你做什么，百分之百地做，工作就工作，笑就笑。当你把手头上的事情都尽可能地高效去完成时，你会发现生命是可以延展的。真正属于自己的时间从来都不是用熬夜换来的。 其次，加强自律。熬夜的罪魁祸首往往是自制力不够。你说再玩会儿手机就睡，谁知一玩就到一两点；你发誓电视剧只看一集，却一口气看完了全部……很多人熬夜，就是败在自己的不自律。 当然，自律从来都不是一蹴而就的事，生活方式的改变需要一步一步来。你可以每天提早五分钟睡觉，提早五分钟起床，循序渐进。自律这件事最难解决，也最好解决，靠的就是自己的毅力和坚持。 最后，自我调节。生活很难事事如愿，有时不得不向现实妥协。有一些工种的确需要值夜班或是加班，当没法避免熬夜时，就应该懂得调整好时间，让自己的身体充分休息。 ### 后记 虽然熬夜的理由有千万种，但再重要的理由都比不上拥有一个健康的身体。 只要身体健康，生活再闹心，都还有希望改善。若是身体垮了，一切都不过是过眼云烟。 希望你早点睡觉，心满意足地结束这一天；希望你早点起床，自信满满地开始新一天。 往后余生还很精彩，别被熬夜拖垮了。 ================================================ FILE: docs/emotion/workheard.md ================================================ # 工作心得：资质平庸的人该怎么办？引人深思！ 天赋秉异的人永远是少数，剩下的都是资质平庸的芸芸众生。相信即使只是普通人，也有一颗不甘于平庸的心。那么资质平庸的人该如何在职场上做出一番成就呢？ 其实以大多数人的努力程度之低，根本轮不到拼天赋。你需要的是比别人更加熟悉职场的规则、遵守规则、利用规则，这个过程也是“被社会与职场的规律驯化”的过程。“被驯化”是无法避免的，只有熟悉“游戏规则”，才能更好地“玩游戏”。 ### 你有同理心吗？ 什么叫“同理心”？ 说复杂点儿，同理心就是站在当事人的角度和位置上，客观地理解当事人的内心感受，且把这种理解传达给当事人的一种沟通交流方式。 说简单点儿，同理心就是“己所不欲，勿施于人”。将心比心，也就是设身处地去感受、去体谅他人。说白了，同理心就是“情商”。 具体点说：同理心就是，领导交办一项工作，你要读懂他的目的、看清他的用意。 我经常遇到这样的情况：给团队成员安排工作时，一再询问“我说明白了吗”“有没有问题”，再三确认后，提交上来的东西仍然答非所问。所以我在接受任务时，总会向领导确认：你想要的是什么？你的目的是什么？了解这个以后，就可以站在他的角度，有效的帮他解决问题。 ### 听话，出活 什么叫“听话”？有句老话叫“干活不由东，累死也无功”，谁是“东”啊？你的直属领导就是“东”，大部分时候，听他的话准没错儿。 根据我的经验，一般来说，领导都比你水平高，起码在一点上是这样：他比你信息更全面、判断的更准确。因为领导更容易接触到更高层，更了解更高层的意图，他知道的你不知道，你在自己的角度上认为“这么做对”，但领导在更高的层面，并不一定这么看。 什么叫“出活”？就是领导给你的工作，你得按时完成并且汇报总结。如果这个工作要持续较长时间，那么你需要阶段性的给领导反馈。 我们经常犯一个错误，领导安排的工作，他不问你也不说，黑不提白不提这事儿就算过去了。过去了？哪儿那么容易啊！领导都记着呢，你等他问你的时候——“诶小陈，上次安排你做的那事儿怎么样了？”——他就已经在心里给你写上了标签：“不靠谱”。一个“不靠谱”需要用十个“靠谱”来扭转，两个“不靠谱”就很难转变印象，三个“不靠谱”你就没有机会了。 ### 要想人前显贵，必须背地受罪 道理并不难懂，就是真到受苦的时候就含糊了。 有的人会说，我年纪轻轻的为什么不好好享受生活啊？这种想法很普遍，这本是一个价值观的问题，没什么可说的，一个人想怎么生活都对。但是有一些朋友是在追求理想和享受生活中纠结的，和这些朋友，是可以聊的。 马云曾经说过：我们追求的应是人生的大平衡，而不是一时一日的小平衡（大意如此）。新东方也有一句话说：怕吃苦吃苦一辈子，不怕苦吃苦半辈子。两句话大意相同，值得深思。 能忍多大事儿，就能成多大事儿 有一天加班，晚上2点钟到家，收到老板的一封邮件，批评我工作不到位。我收到邮件后就很崩溃，还很委屈。于是当即奋笔疾书，回邮件！解释我是如何工作的，我做的如何有道理，我做的如何有效果……写了2000多字。 写完了，我好像冷静了一些，我就琢磨一个事儿：如果我是老板，我对一个员工工作不满意，于是我给他写了封邮件批评他，我想看到的是他洋洋洒洒的解释和辩解吗？显然不是啊。然后我就突然明白了，于是我把那2000多字都删了，简单回复了一句话，大意是：我会反思工作的问题，然后尽快整改。 两个月后我晋升了。在我的晋升仪式上，我对我老板说起这件事，他对我说，我知道你很委屈，我就是想看看你在面对委屈和压力时，会有怎样的反应，这体现了一个人的成熟程度。 ### 总躲着领导，你就危险了 不少人躲着领导，尽量少跟领导说话、绕着领导走。因为跟领导近了事儿就多，不跟领导多接触，事儿少，多清闲。这是“一叶障目，不见泰山”。 如果你想在工作上取得一些成绩，我建议还是应该主动的多和领导沟通。领导在平时开会时说的多是大面儿上的话，真话、有用的话、有价值的话不一定说。这并不是他不想说，而是没机会说。 有心的员工会随时抽时间和领导沟通、增加私人交流的机会：一起吃饭、一起抽烟、一起上下班、甚至一起打球K歌……通过这样的机会，你可以了解领导对于你的看法、对于工作的观点，这些都有益于你调整自己的工作的方式。 有朋友担心这样做会引起领导反感，其实完全不会，领导们多是孤独的，如果他发现有一个员工虚心向他请教、积极分享工作的思考，他是非常高兴的。 老板也是人，大家用人类的方式沟通，一切会变得简单很多。 ### 帮助别人千万别吝啬 当你正在忙于某项工作时，有同事来向你“求助”，很多时候我们会很直接、甚至粗暴的拒绝，殊不知这样做正在给你今后的工作种下麻烦的种子。 风水轮流转，在一家公司里，大家的工作互相交叉的几率很大，说不定你会用上谁，这些人脉关系需要平时去维护。今天你帮助了人家，说不定明天对方就会成为你的救命稻草，这非常可能。 ### 目标再目标，量化再量化 没有目标的，都不叫工作；没有量化的，都不叫目标。 在接受一项工作时，先问目标是什么；在布置一项工作时，先交代目标是什么。这个不说清楚，都是扯淡。 不想成为蒙着眼睛拉磨的驴？那么除了清楚的知道自己的目标外，还得知道你的部门、你的公司的目标，最关键的，你需要知道，你的工作在总体目标中处在什么地位、扮演什么角色。如果你发现，你工作的目标和总体目标关系很小、甚至没有关系，那么你就很容易被拿掉。 ### 总结 （什么是工作到位？(很深刻)） 1. 汇报工作说结果 不要告诉老板工作多么艰辛，你多么的不容易。举重若轻的人老板最喜欢，一定要把结果给老板，结果思维是第一思维。 2. 请示工作说方案 不要让老板做问答题，而是让老板做选择题。请示工作至少给老板两个方案，并表达自己的看法。 3. 总结工作说流程 做工作总结要说流程，不只是先后顺序逻辑清楚，还要找出流程中的关键、失误点、反思点。 4. 布置工作说标准 工作又布置就有考核，考核就要建立工作标准，否则下属不知道怎么做，做到什么程度才最适合。标准不但确立了规范，又划定了工作边界。 5. 关心下属问的过程 关心下属要注意聆听他们的问题，让其阐述越详细越好，明确让下属感动的点面。 6. 交接工作讲道德 把工作中形成的经验教训毫不保留的交接给继承者，把完成的与未竟的工作分类逐一交接，不要设置障碍，使其迅速进入工作角色 8. 回忆工作说感受 交流多说自己的感悟，那些是学的，那些是悟到的，那些是反思的，那些事努力的。 9. 领导工作别瞎忙 比尔.盖茨说过：“一个领袖如果每天很忙，就证明能力不足。”如果作为领导者的你很忙，不妨问自己四个问题: * 我在忙什么？ * 我忙的事情有多大价值？ * 我做的事情别人会不会做？ * 我为什么会这么忙？ 不管什么原因，如果此刻你很忙，请你真实的面对自己，该停一停，给自己充电！ ================================================ FILE: docs/exp/ai.md ================================================ # 人工智能、机器学习、神经网络、深度学习、TensorFlow、图像处理必备书籍(附PDF百度盘下载链接) --- 在学习人工智能相关相关知识中往往不理解其中相关术语意义和知识原理的组成，下面书籍是阿拉灯神丁君在阅读了大量书籍后觉得很不错的一部分，特此分享出来，以供大家学习之便利。内容链接如有侵犯到您的权益，请联系删除。 在学习人工智能相关相关知识中往往不理解其中相关术语意义和知识原理的组成，下面书籍是阿拉灯神丁君在阅读了大量书籍后觉得很不错的一部分，特此分享出来，以供大家学习之便利。内容链接如有侵犯到您的权益，请联系删除。 1. 机器学习 周志华.pdf [链接:](https://pan.baidu.com/s/1P5Owh7YoZ6ncQz9dXanwCA) 密码:wzst 2. 推荐系统实践.pdf [链接:](https://pan.baidu.com/s/11-VA6RE0RY3XRBQkEomMMg) 密码:te31 3. 《自然语言处理综论》.pdf [链接:](https://pan.baidu.com/s/1KP6wTUyRvtpDGC84bltoyA) 密码:l3wl 4、《计算机视觉：一种现代方法》.pdf [链接:](https://pan.baidu.com/s/1XXY5AP7hHOb6XKtLv36yvg) 密码:l3fb 5、图解机器学习.pdf [链接:](https://pan.baidu.com/s/1X3qsFL1rSL_bhqsAHQ7fPg) 密码:k59g 6、《决策知识自动化》.pdf [链接:](https://pan.baidu.com/s/13kalbqzHI8G7F_dprWytkw) 密码:bu7l 7、《人工智能：一种现代的方法(第3版)》.pdf [链接:](https://pan.baidu.com/s/1cHPk3gVUGJFKP-xL90vEFA) 密码:8iyo 8、Python数据分析与挖掘实战.pdf [链接:](https://pan.baidu.com/s/1OjCLV_hhCcFO5BBCKUDxHA) 密码:gnzc 9、机器学习导论.pdf [链接:](https://pan.baidu.com/s/1hmD-VeuzlOy-qakENc-77g) 密码:2wpg 10、面向机器智能的TensorFlow实践 (智能系统与技术丛书)_.pdf [链接:](https://pan.baidu.com/s/1BQJLFh7te0ggyD538JgQWg) 密码:03lo 11、图像处理、分析与机器视觉（第三版）.pdf [链接:](https://pan.baidu.com/s/17VmF6AHL_WsLr6g8_oaHmw) 密码:104g 12、TensorFlow实战_黄文坚（完整）.pdf [链接:](https://pan.baidu.com/s/15voQk0MxMYySwhTy2UTVSQ) 密码:6syh 13、Tensorflow 实战Google深度学习框架.pdf [链接:](https://pan.baidu.com/s/1L85W0wOCdNBM-a-QDYceTA) 密码:ejj3 14、统计学习方法.pdf [链接:](https://pan.baidu.com/s/1tKUWBK-eZHwSPSUf15PGuQ) 密码:zjme 作者李航，是国内机器学习领域的几个大家之一，曾在MSRA任高级研究员，现在华为诺亚方舟实验室。书中写了十个算法，每个算法的介绍都很干脆，直接上公式，是彻头彻尾的“干货书”。每章末尾的参考文献也方便了想深入理解算法的童鞋直接查到经典论文；本书可以与上面两本书互为辅助阅读。 15、数学之美.pdf [链接:](https://pan.baidu.com/s/1sWFH8lkzWpHLDxw0zBvyyw) 密码:y7xe 作者吴军大家都很熟悉。以极为通俗的语言讲述了数学在机器学习和自然语言处理等领域的应用。 16、区块链新经济概论.pdf [链接:](https://pan.baidu.com/s/10ugwGsJI9yf-z5yzKNon0g) 密码:m35l 最近也是在学习入门阶段，也就一个感觉“一如侯门深似海，从此节操是路人”，看的我是头晕眼花，公式，概念，金星星眼前飘过~~~.....((/- -)/ 以上电子书也基本都是高清版，本人对电子书的质量要求也是比较高的，影印版太垃圾了，更是伤银镜。 人工智能领域涵盖的知识非常的广：算法、深度学习、机器学习、自然语言处理、数据结构、Tensorflow、Python 、数据挖掘、搜索开发、神经网络、视觉度量、图像识别、语音识别、推荐系统、系统算法、图像算法、数据分析、概率编程、计算机数学、数据仓库、建模等关键词，基本涵盖了现阶段人工智能细分领域的人才结构。 ================================================ FILE: docs/exp/cl.md ================================================ # 集群和负载均衡 在“高并发，海量数据，分布式，NoSql，云计算......”概念满天飞的年代，相信不少朋友都听说过甚至常与人提起“集群，负载均衡”等， 但不是所有人都有机会真正接触到这些技术，也不是所有人都真正理解了这些“听起来很牛的”技术名词。下面简单解释一下吧。 要了解这些概念首先要了解一下项目架构的演进，我这里应用一张Dubbo的文档图片如图 ## 一：项目架构的演进 ![cl](./../img/cl1.png) ### ORM与MVC： 早期的架构都集中在一台服务器上，这样对于小型的业务访问量是完全可以的，但是随着业务的增多，我们引进的MVC的架构，这种架构是将整个业务分成不同的层（表现层，业务层，数据访问层）维护也更加方面了，开发更加方便。 ### PRC架构： 但是业务如果继续增大，项目会出现臃肿，一台服务器已经完全没办法支持了，所以出现了RPC分布式的架构，RPC架构就是将服务进行合理拆分，分别放入多台服务器执行，服务器与服务器之间通过远程调用的方式进行通信。 服务提供者：运行在服务器端，提供服务接口与服务实现类 服务中心：运行在服务器端，负责将本地服务发布成远程服务，管理远程服务，提供服务给消费者使用。 服务消费者：运行在客户端，通过远程代理对象调用远程服务 ### SOA架构： 但是业务继续增加，对RPC架构来说，各个服务与服务之间的通信越来越多，依赖越来越多，越来越混乱，给开发带来了困难，于是SOA架构应运而生，SOA架构将服务与服务集中起来进行管理，加上一个服务治理中心。谁发布了服务来中心进行注册，谁需要依赖什么服务来中心进行请求。 而最近很火的微服务，则是将业务拆分更加精细，每一个可以成为一个完整的服务。演变肯定会演变，但是过程得多久谁也不好说。 ![cl](./../img/cl2.png) ## 二：名词解释 接下来进入正题，解释让外行看起来高大上的名词 ### 集群(Cluster) 所谓集群是指一组独立的计算机系统构成的一个松耦合的多处理器系统，它们之间通过网络实现进程间的通信。应用程序可以通过网络共享内存进行消息传送，实现分布式计算机。通俗一点来说，就是让若干台计算机联合起来工作(服务)，可以是并行的，也可以是做备份。 大规模集群，通常具备以下一些特点： 1. 高可靠性(HA) 利用集群管理软件，当主服务器故障时，备份服务器能够自动接管主服务器的工作，并及时切换过去，以实现对用户的不间断服务。 2. 高性能计算(HP) 即充分利用集群中的每一台计算机的资源，实现复杂运算的并行处理，通常用于科学计算领域，比如基因分析、化学分析等。 3. 负载平衡(LB) 即把负载压力根据某种算法合理分配到集群中的每一台计算机上，以减轻主服务器的压力，降低对主服务器的硬件和软件要求。 #### 常用的集群又分以下几种 1. load balance cluster(负载均衡集群) 一共有四兄弟开裁缝铺，生意特别多，一个人做不下来，老是延误工期，于是四个兄弟商量：老大接订单, 三个兄弟来干活。 客户多起来之后，老大根据一定的原则(policy) 根据三兄弟手上的工作量来分派新任务。 2. High availability cluster(高可用集群) 两兄弟开早餐铺，生意不大，但是每天早上7点到9点之间客户很多并且不能中断。为了保证2个小时内这个早餐铺能够保证持续提供服务，两兄弟商量几个方法: 方法一：平时老大做生意，老二这个时间段在家等候，一旦老大无法做生意了，老二就出来顶上，这个叫做 Active/Standby.(双机热备) 方法二：平时老大做生意，老二这个时候就在旁边帮工，一旦老大无法做生意，老二就马上顶上，这个叫做Active/Passive.(双机双工) 方法三：平时老大卖包子，老二也在旁边卖豆浆，老大有问题，老二就又卖包子，又卖豆浆，老二不行了，老大就又卖包子，又卖豆浆.这个叫做Active/Active (dual Active)(双机互备) 3. high computing clustering(高性能计算集群) 10个兄弟一起做手工家具生意，一个客户来找他们的老爹要求做一套非常复杂的仿古家具，一个人做也可以做，不过要做很久很久，为了1个星期就交出这一套家具，10个兄弟决定一起做。 老爹把这套家具的不同部分分开交给儿子们作，然后每个儿子都在做木制家具的加工，最后拼在一起叫货。 老爹是scheduler任务调度器，儿子们是compute node. 他们做的工作叫做作业。 ### 负载均衡 #### HTTP重定向负载均衡 当用户发来请求的时候，Web服务器通过修改HTTP响应头中的Location标记来返回一个新的url，然后浏览器再继续请求这个新url，实际上就是页面重定向。通过重定向，来达到“负载均衡”的目标。例如，我们在下载JAVA源码包的时候，点击下载链接时，为了解决不同国家和地域下载速度的问题，它会返回一个离我们近的下载地址。重定向的HTTP返回码是302。 优点：比较简单。 缺点：浏览器需要两次请求服务器才能完成一次访问，性能较差。重定向服务自身的处理能力有可能成为瓶颈，整个集群的伸缩性规模有限；使用HTTP302响应码重定向，有可能使搜索引擎判断为SEO作弊，降低搜索排名。 #### DNS域名解析负载均衡 DNS（Domain Name System）负责域名解析的服务，域名url实际上是服务器的别名，实际映射是一个IP地址，解析过程，就是DNS完成域名到IP的映射。而一个域名是可以配置成对应多个IP的。因此，DNS也就可以作为负载均衡服务。 事实上，大型网站总是部分使用DNS域名解析，利用域名解析作为第一级负载均衡手段，即域名解析得到的一组服务器并不是实际提供Web服务的物理服务器，而是同样提供负载均衡服务的内部服务器，这组内部负载均衡服务器再进行负载均衡，将请求分发到真实的Web服务器上。 优点：将负载均衡的工作转交给DNS，省掉了网站管理维护负载均衡服务器的麻烦，同时许多DNS还支持基于地理位置的域名解析，即会将域名解析成举例用户地理最近的一个服务器地址，这样可以加快用户访问速度，改善性能。 缺点：不能自由定义规则，而且变更被映射的IP或者机器故障时很麻烦，还存在DNS生效延迟的问题。而且DNS负载均衡的控制权在域名服务商那里，网站无法对其做更多改善和更强大的管理。 #### 反向代理负载均衡 反向代理服务可以缓存资源以改善网站性能。实际上，在部署位置上，反向代理服务器处于Web服务器前面（这样才可能缓存Web响应，加速访问），这个位置也正好是负载均衡服务器的位置，所以大多数反向代理服务器同时提供负载均衡的功能，管理一组Web服务器，将请求根据负载均衡算法转发到不同的Web服务器上。Web服务器处理完成的响应也需要通过反向代理服务器返回给用户。由于web服务器不直接对外提供访问，因此Web服务器不需要使用外部ip地址，而反向代理服务器则需要配置双网卡和内部外部两套IP地址。 优点：和反向代理服务器功能集成在一起，部署简单。 缺点：反向代理服务器是所有请求和响应的中转站，其性能可能会成为瓶颈。 #### 负载均衡策略 1. 轮询 2. 加权轮询 3. 最少连接数 4. 最快响应 5. Hash法 ![cl](./../img/cl3.png) ### 缓存 缓存就是将数据存放在距离计算最近的位置以加快处理速度。缓存是改善软件性能的第一手段，现在CPU越来越快的一个重要因素就是使用了更多的缓存，在复杂的软件设计中，缓存几乎无处不在。大型网站架构设计在很多方面都使用了缓存设计。 #### CDN缓存 内容分发网络，部署在距离终端用户最近的网络服务商，用户的网络请求总是先到达他的网络服务商哪里，在这里缓存网站的一些静态资源（较少变化的数据），可以就近以最快速度返回给用户，如视频网站和门户网站会将用户访问量大的热点内容缓存在CDN中。 #### 反向代理缓存 反向代理属于网站前端架构的一部分，部署在网站的前端，当用户请求到达网站的数据中心时，最先访问到的就是反向代理服务器，这里缓存网站的静态资源，无需将请求继续转发给应用服务器就能返回给用户。 #### 本地缓存 在应用服务器本地缓存着热点数据，应用程序可以在本机内存中直接访问数据，而无需访问数据库。 #### 分布式缓存 大型网站的数据量非常庞大，即使只缓存一小部分，需要的内存空间也不是单机能承受的，所以除了本地缓存，还需要分布式缓存，将数据缓存在一个专门的分布式缓存集群中，应用程序通过网络通信访问缓存数据。 ### 流控（流量控制） #### 流量丢弃 通过单机内存队列来进行有限的等待，直接丢弃用户请求的处理方式显得简单而粗暴，并且如果是I/O密集型应用（包括网络I/O和磁盘I/O），瓶颈一般不再CPU和内存。因此，适当的等待，既能够提升用户体验，又能够提高资源利用率。 通过分布式消息队列来将用户的请求异步化。 ================================================ FILE: docs/exp/code-principle.md ================================================ # 编码原则 ## 吻（保持最简单愚蠢） 如果保持简单而不是复杂，大多数系统都能发挥最佳性能。 为什么： * 更少的代码花费更少的时间来编写，具有更少的错误，并且更容易修改。 * 简约是最终的成熟。 * 似乎没有任何东西可以添加，但是当没有什么可以带走时，达到完美。 资源： * [Kiss原则](https://en.wikipedia.org/wiki/KISS_principle) * [保持简单愚蠢（KISS）](http://principles-wiki.net/principles:keep_it_simple_stupid) ## YAGNI(你不需要它) YAGNI代表“你不会需要它”：在必要之前不要实施某些东西。 为什么： * 任何仅用于明天需要的功能的工作意味着从当前迭代需要完成的功能中失去工作量。 * 它导致代码膨胀; 软件变得更大，更复杂。 怎么样： * 当你真正需要它们时，总是要实现它们，而不是在你预见到需要它们的时候。 资源： *[你不会需要它](http://c2.com/xp/YouArentGonnaNeedIt.html) *[你不需要它！](http://www.xprogramming.com/Practices/PracNotNeed.html) *[你不需要它](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it) ## 做最简单的事可能有效 为什么: * 如果我们只是解决问题的真正原因，那么真正问题的真正进展就会最大化。 怎么样: * 问问自己：“最简单的事情是什么？” 资源: * [做最简单的事可能有效](http://c2.com/xp/DoTheSimplestThingThatCouldPossiblyWork.html) ## 关注点分离 关注点分离是将计算机程序分成不同部分的设计原则，这样每个部分都解决了一个单独的问题。例如，应用程序的业务逻辑是一个问题，用户界面是另一个问题。更改用户界面不应要求更改业务逻辑，反之亦然。 引用[Edsger W. Dijkstra](https://en.wikipedia.org/wiki/Edsger_W._Dijkstra)（1974）： ``` 这就是我有时称之为“关注点的分离”，即使不完全可能，它仍然是我所知道的有效排序一个人思想的唯一可行技术。这就是我所说的“将注意力集中在某些方面”：它并不意味着忽视其他方面，它只是公正地从这个方面来看，另一个是无关紧要的事实。 ``` 为什么: * 简化软件应用程序的开发和维护。 * 当问题分离时，各个部分可以重复使用，也可以独立开发和更新。 怎么样: * 将程序功能分解为尽可能少重叠的单独模块。 资源: * [关注点分离](https://en.wikipedia.org/wiki/Separation_of_concerns) ## 保持干燥 每一条知识都必须在系统中具有单一，明确，权威的表示。 程序中的每个重要功能都应该在源代码中的一个位置实现。在通过不同的代码片段执行类似的功能的情况下，通过抽象出变化的部分将它们组合成一个通常是有益的。 为什么: * 重复（无意或有目的的重复）可能导致维护噩梦，差的因素和逻辑矛盾。 * 对系统的任何单个元素的修改不需要改变其他逻辑上不相关的元素。 * 另外，逻辑上相关的元素都可以预测和统一地改变，因此保持同步。 怎么样: * 只在一个地方放置业务规则，长表达式，if语句，数学公式，元数据等。 * 确定系统中使用的每一条知识的单一，权威来源，然后使用该源生成该知识的适用实例（代码，文档，测试等）。 * 适用[三条规则](https://en.wikipedia.org/wiki/Rule_of_three_(computer_programming))。 资源: * [别重复自己](http://wiki.c2.com/?DontRepeatYourself) * [不要重复自己](https://en.wikipedia.org/wiki/Don't_repeat_yourself) 有关: * [抽象原则](https://en.wikipedia.org/wiki/Abstraction_principle_(computer_programming)) * [Once And Only Once](http://wiki.c2.com/?OnceAndOnlyOnce)是DRY的一个子集（也称为重构的目标）。 * [单一事实来源](https://en.wikipedia.org/wiki/Single_source_of_truth) 违反DRY是[WET](http://thedailywtf.com/articles/The-WET-Cart)（写一切两次） ## 维护者代码 为什么： * 到目前为止，维护是任何项目中最昂贵的阶段。 怎么样： * 做维护者。 * 总是编码好像最终维护你的代码的人是一个知道你住在哪里的暴力精神病患者。 * 始终以这样一种方式编码和评论：如果有人在少数几个级别上接受代码，他们会乐于阅读并从中学习。 * [不要让我思考](http://www.sensible.com/dmmt.html)。 * 使用[最小惊讶原则](http://en.wikipedia.org/wiki/Principle_of_least_astonishment)。 资源: * [维护者代码](http://wiki.c2.com/?CodeForTheMaintainer) * [高尚的维护编程艺术](https://blog.codinghorror.com/the-noble-art-of-maintenance-programming/) ## 避免过早优化 引用[唐纳德克努特](https://en.wikiquote.org/wiki/Donald_Knuth)： ``` 程序员浪费了大量时间来思考或担心程序中非关键部分的速度，而这些效率尝试实际上在考虑调试和维护时会产生很大的负面影响。我们应该忘记小的效率，大约97％的时间说：过早的优化是所有邪恶的根源。然而，我们不应该放弃那个至关重要的3％的机会。 ``` 理解什么是“不成熟”并不是“过早”是至关重要的。 为什么: * 事先不知道瓶颈在哪里。 * 优化后，可能更难以阅读并因此维护。 怎么样: * [让它工作正确使它快速](http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast) * 在您需要之前不要进行优化，并且只有在进行性能分析后才发现瓶颈才能优化。 资源: * [程序优化](https://en.wikipedia.org/wiki/Program_optimization) * [过早优化](http://wiki.c2.com/?PrematureOptimization) ## 最小化耦合 模块/组件之间的耦合是它们相互依赖的程度; 较低的耦合更好。换句话说，耦合是代码单元“A”在对代码单元“A”的未知改变之后将“破坏”的概率。 为什么: * 一个模块的变化通常会对其他模块的变化产生连锁反应。 * 由于模块间依赖性的增加，模块的组装可能需要更多的努力和/或时间。 * 特定模块可能更难以重用和/或测试，因为必须包含依赖模块。 * 开发人员可能害怕更改代码，因为他们不确定可能会受到什么影响。 怎么样: * 消除，最小化并减少必要关系的复杂性。 * 通过隐藏实现细节，减少了耦合。 * 应用得墨忒耳定律。 资源: * [耦合](https://en.wikipedia.org/wiki/Coupling_(computer_programming)) * [耦合与凝聚力](http://wiki.c2.com/?CouplingAndCohesion) ## 得墨忒耳定律 不要和陌生人说话。 为什么: * 它通常会收紧耦合 * 它可能会揭示太多的实现细节 怎么样: 对象的方法只能调用以下方法： * 对象本身。 * 方法的一个论点。 * 在方法中创建的任何对象。 * 对象的任何直接属性/字段。 资源: * [得墨忒耳定律](https://en.wikipedia.org/wiki/Law_of_Demeter) * [得墨忒耳定律不是一个点数运动](https://haacked.com/archive/2009/07/14/law-of-demeter-dot-counting.aspx/) ## 继承的组合 为什么: * 类之间的耦合较少。 * 使用继承，子类很容易做出假设，并破坏LSP。 怎么样: * 测试LSP（可替代性）以决定何时继承。 * 当存在“具有”（或“使用”）关系时，在“是”时继承。 资源: * [赞成组合而不是继承](https://blogs.msdn.microsoft.com/thalesc/2012/09/05/favor-composition-over-inheritance/) ## 正交 正交性的基本思想是，在概念上不相关的事物不应该与系统相关。 来源：[Be Orthogonal](https://www.artima.com/intv/dry3.html) 它与简单相关; 设计越正交，异常越少。这使得用编程语言更容易学习，读写程序。正交特征的含义与上下文无关; 关键参数是对称性和一致性。 资料来源：[正交性](https://en.wikipedia.org/wiki/Orthogonality_(programming)) ## 稳健性原则 保守你的所作所为，要接受别人的自由 协作服务取决于彼此的接口。通常接口需要进化，导致另一端接收未指定的数据。如果收到的数据不严格遵循规范，那么天真的实现就会拒绝协作。更复杂的实现仍然会忽略它无法识别的数据。 为什么: * 为了能够发展服务，您需要确保提供商可以进行更改以支持新需求，同时最大限度地减少对现有客户的破坏。 怎么样: * 将命令或数据发送到其他机器（或同一台机器上的其他程序）的代码应完全符合规范，但只要含义明确，接收输入的代码就应接受不符合要求的输入。 资源: * [维基百科中的稳健性原则](https://en.wikipedia.org/wiki/Robustness_principle) * [宽容读者](https://martinfowler.com/bliki/TolerantReader.html) ## 控制反转 控制倒置也被称为好莱坞原则，“不要打电话给我们，我们会打电话给你”。这是一种设计原则，其中计算机程序的定制编写部分从通用框架接收控制流。控制反转具有强烈的含义，即可重用代码和特定于问题的代码即使在应用程序中一起操作也是独立开发的。 为什么: * 控制反转用于增加程序的模块性并使其可扩展。 * 将任务的执行与实现分离。 * 将模块集中在它所针对的任务上。 * 使模块免于假设其他系统如何做他们所做的事情，而是依赖合同。 * 更换模块时防止副作用。 怎么样: * 使用工厂模式 * 使用服务定位器模式 * 使用依赖注入 * 使用上下文查找 * 使用模板方法模式 * 使用策略模式 资源: * [维基百科中的控制反转](https://en.wikipedia.org/wiki/Inversion_of_control) * [控制容器的反转和依赖注入模式](https://www.martinfowler.com/articles/injection.html) ## 最大化凝聚力 单个模块/组件的凝聚力是其职责构成有意义单元的程度; 更高的凝聚力更好。 为什么: * 理解模块的难度增加。 * 维护系统的难度增加，因为域中的逻辑更改会影响多个模块，并且因为一个模块中的更改需要更改相关模块。 * 由于大多数应用程序不需要模块提供的随机操作集，因此增加了重用模块的难度。 怎么样: * 集团相关的功能共享一个单一的职责（例如在一个班级）。 资源: * [凝聚](https://en.wikipedia.org/wiki/Cohesion_(computer_science)) * [耦合与凝聚力](http://wiki.c2.com/?CouplingAndCohesion) ## 利斯科夫替代原则(里氏替换原则) LSP完全是关于对象的预期行为： 程序中的对象应该可以替换其子类型的实例，而不会改变该程序的正确性。 资源： * [利斯科夫替代原则](https://en.wikipedia.org/wiki/Liskov_substitution_principle) * [利斯科夫替代原则](http://www.blackwasp.co.uk/lsp.aspx) ## 开放/封闭原则 软件实体（例如类）应该是可以扩展的，但是关闭以进行修改。即，这样的实体可以允许在不改变其源代码的情况下修改其行为。 为什么: * 通过最小化对现有代码的更改来提高可维护性和稳定性 怎么样: * 编写可以扩展的类（与可以修改的类相对）。 * 仅暴露需要更改的移动部件，隐藏其他所有内容。 资源: * [开放封闭原则](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle) * [开放封闭原则](https://blog.cleancoder.com/uncle-bob/2014/05/12/TheOpenClosedPrinciple.html) ## 单一责任原则 一个班级永远不应该有多个改变的理由。 长版本：每个班级都应该承担一个责任，并且该责任应该由班级完全封装。责任可以被定义为改变的理由，因此一个类或模块应该只有一个改变的理由。 为什么: * 可维护性：仅在一个模块或类中需要进行更改。 怎么样 * 应用[Curly定律](https://blog.codinghorror.com/curlys-law-do-one-thing/)。 资源: * [单一责任原则](https://en.wikipedia.org/wiki/Single_responsibility_principle) ## 隐藏实施细节 软件模块通过提供接口隐藏信息（即实现细节），而不泄漏任何不必要的信息。 为什么: * 当实现更改时，使用的接口客户端不必更改。 怎么样: * 最小化类和成员的可访问性。 * 不要公开公开成员数据。 * 避免将私有实现细节放入类的接口中。 * 减少耦合以隐藏更多实现细节。 资源: * [信息隐藏](https://en.wikipedia.org/wiki/Information_hiding) ## 卷毛定律 Curly定律是为任何特定的代码选择一个明确定义的目标：做一件事。 * [卷毛定律：做一件事](https://blog.codinghorror.com/curlys-law-do-one-thing/) * 一个规则或卷曲定律 ## 封装了哪些变化 一个好的设计可以识别最有可能改变的热点，并将它们封装在API之后。当发生预期的变化时，修改保持在本地。 为什么: * 在发生更改时最小化所需的修改 怎么样: * 封装API背后不同的概念 * 可能将不同的概念分成它自己的模块 资源: * [封装变化的概念](http://principles-wiki.net/principles:encapsulate_the_concept_that_varies) * [封装什么变化](https://blogs.msdn.microsoft.com/steverowe/2007/12/26/encapsulate-what-varies/) * [信息隐藏](https://en.wikipedia.org/wiki/Information_hiding) ## 接口隔离原理 将胖接口减少为多个更小，更具体的客户端特定接口。接口应该更多地依赖于调用它的代码而不是实现它的代码。 为什么: * 如果一个类实现了不需要的方法，则调用者需要知道该类的方法实现。例如，如果一个类实现一个方法但只是抛出，那么调用者将需要知道实际上不应该调用此方法。 怎么样: * 避免胖接口。类不应该实现违反[单一责任原则的方法。](https://en.wikipedia.org/wiki/Single_responsibility_principle) 资源: * [界面隔离原理](https://en.wikipedia.org/wiki/Interface_segregation_principle) ## 童子军规则 美国童子军有一个简单的规则，我们可以适用于我们的职业：“让露营地更清洁，而不是你发现它”。童子军规则规定我们应该始终保持代码比我们发现的更干净。 为什么: * 在对现有代码库进行更改时，代码质量往往会降低，从而累积技术债务。按照boyscout规则，我们应该注意每次提交的质量。无论多么小，技术债务都会受到持续重构的抵制。 怎么样: * 每次提交都要确保它不会降低代码库质量。 * 每当有人看到一些不太清晰的代码时，他们应该抓住机会在那里修复它。 资源: * [机会重构](https://martinfowler.com/bliki/OpportunisticRefactoring.html) ## 命令查询分离 命令查询分离原则指出每个方法应该是执行操作的命令或将数据返回给调用者而不是两者都返回的查询。提出问题不应该修改答案。 应用此原则后，程序员可以更自信地编写代码。查询方法可以在任何地方以任何顺序使用，因为它们不会改变状态。使用命令必须更加小心。 为什么: * 通过将方法明确地分离为查询和命令，程序员可以在不知道每个方法的实现细节的情况下进行编码。 怎么样: * 将每个方法实现为查询或命令 * 将命名约定应用于方法名称，该方法名称暗示该方法是查询还是命令 资源: * [维基百科中的命令查询分离](https://en.wikipedia.org/wiki/Command%E2%80%93query_separation) * [Martin Fowler命令查询分离](https://martinfowler.com/bliki/CommandQuerySeparation.html) ================================================ FILE: docs/exp/cto.md ================================================ # CTO 技能图谱 ### 岗位职责 * 建立技术团队文化 * 规划技术发展路线 * 落地产品研发成果 * 宣传公司技术品牌 * 吸引优秀技术人才 ### 基本素质 * 正直诚实的道德修养 * 谦虚谨慎的工作态度 * 随机应变的处事风格 * 统领全局的战略思维 ### 硬技能 #### 技术能力 * 具备一定的技术深度 * 具备较强的技术广度 * 追求技术选型合理性 * 对技术发展嗅觉敏锐 #### 业务能力 * 能深度理解业务本质 * 能用技术来帮助业务 * 让技术驱动业务发展 #### 架构能力 * 能站在业务上设计架构 * 架构规范合理且易落地 * 能为架构设计提出意见 #### 产品能力 * 具备一定的产品价值观 * 能准确地抓住用户刚需 * 能为产品设计提出意见 #### 管理能力 - 团队管理 * 招人：知道如何吸引所需的人？ * 识人：知道如何识别已有队员？ * 育人：知道如何让队员们成长？ * 开人：知道如何请人愉快离开？ * 留人：知道如何留住该留的人？ * 挖人：知道如何挖到需要的人？ - 项目管理 * 估算：知道项目成本如何估算？ * 分工：知道工作任务如何分配？ * 排期：知道项目计划如何制定？ * 实施：知道项目实施如何开展？ * 发布：知道项目发布如何执行？ * 回顾：知道项目回顾如何进行？ - 绩效管理 * 制定：知道如何制定考核标准？ * 执行：知道如何执行绩效考核？ * 优化：知道如何提升团队绩效？ - 时间管理 * 制定：知道如何制定团队计划？ * 管理：知道如何管理团队任务？ * 权衡：知道如何权衡优先级别？ - 情绪管理 * 控制：知道如何控制自我情绪？ * 转化：知道如何化悲观为乐观？ * 使用：知道如何善用自我情绪？ ### 软技能 #### 领导能力 - 决策能力 * 不要害怕做决定 * 做出正确的决定 * 敢于为决定负责 - 影响能力 * 不要改变别人而是激发别人 * 用自己的行为和态度去激发 * 持续不断提高自己的影响力 - 沟通能力 + 向上沟通（与公司创始人） * 领会老板真实意图 * 站在老板角度思考 * 不要强迫改变老板 + 向下沟通（与本部门同事） * 是沟通而不是命令 * 站在下属立场沟通 * 不要吝啬夸赞队员 - 横向沟通（与跨部门同事） * 突出对方的重要性 * 先要共识才能共赢 * 懂得圆满大于完美 #### 执行能力 * 理解执行目标 * 提高执行效率 * 确保执行效果 #### 学习能力 * 对新知识充满好奇心 * 能够快速学习新技能 * 拥有触类旁通的能力 #### 组织能力 * 积极主动并有活力 * 做事情敢于放得开 * 能够调用所需资源 #### 洞察能力 * 善于抓住事物本质 * 善于观察人性心理 * 善于预见未来变化 #### 抗压能力 * 学会释放压力 * 化压力为动力 * 化消极为积极 #### 自省能力 * 能够不断自我反省 * 能够从失败中总结 * 能够在总结中提高 #### 战略能力 * 能够深刻理解公司战略 * 能够站在更高层次思考 * 结合战略形成技术壁垒 #### 社交能力 * 善于表达自己 * 善于结交朋友 * 善于公众演讲 #### 谈判能力 * 能够通过谈判得到对方认同 * 能够从谈判中找到共赢方式 * 能够在谈判场合中保持镇定 #### 政治能力 * 能够对政治持有敏感度 * 能够处理办公室小政治 * 能够主动避免政治风险 ================================================ FILE: docs/exp/devops.md ================================================ # 一分钟告诉你究竟DevOps是什么鬼？ ![devops](./../img/devops1.png "devops") ## 历史回顾 为了能够更好的理解什么是DevOps，我们很有必要对当时还只有程序员(此前还没有派生出开发者，前台工程师，后台工程师之类)这个称号存在的历史进行一下回顾。 ## 如编程之道中所言： 老一辈的程序员是神秘且深奥的。我们没法揣摩他们的想法，我们所能做的只是描述一下他们的表象。 * 清醒的像一只游过水面的狐狸 * 警惕的像一位战场上的将军 * 友善的像一位招待客人的女主人 * 单纯的像一块未经雕琢的木头 * 深邃的像一潭幽深洞穴中漆黑的池水 程序员开发了机器语言，机器语言又产生了汇编语言，汇编语言产生了编译器，如今的语言已经多不胜数。每一种语言都有其各自的谦卑用途。每一种语言都表达出软件的阴和阳。每一种语言都在此道之中有其一席之地。 遥想当年，软件程序员的大部分办公司那时还被称作实验室，程序员那时还叫做科学家。为了开发出一套优秀的软件，程序员们必须深入了解他们需要的应用相关的所有问题。他们必须清楚知道这个软件应用在什么场合，这个软件是必须在什么系统上运行。本质上说，程序员对所要开发的软件的所有环节都有透彻的了解，从规格说明书编写、到软件开发、到测试、到部署、再到技术支持。 过了不久，人类(客户)贪婪的特性就开始表现出来，他们开始不断的进行更多的索求。更快的速度，更多的功能，更多的用户，更多的所有所有。 作为一类谦虚、谦卑、且平静的生物，我们的老一辈程序员们将很难在这种爆发性的过度的需求索取中幸存。最好的取胜办法就是往不同的方向进化成不同的新物种。很快，程序员这个称号就开始绝迹于江湖，而那些叫做开发者、软件工程师、网络管理员、数据库开发者、网页开发者、系统架构师、测试工程师等等更多的新物种就开始诞生。快速进化和快速适应外界的挑战成为了他们的DNA的一部分。这些新的种族可以在几个星期内就完成进化。网页开发者很快就能进化成后台开发者，前台开发者，PHP开发者，Ruby开发者，Angular开发者…多得让人侧目。 很快他们就都忘却了他们都是起源于程序员这个共同的祖先的事实，忘却了曾经有过这么一个单纯且平静的，想要让这个世界变得更好的科学家。然后他们开始不断的剑拔弩张，都声称自己才是“程序员”的纯血统继承人。 随着时间的转移，各门各派开始独占山头，很少进行交流互动，只有在迫不得已的时刻才会进行沟通。他们开始不再为同源的遥远的同宗兄弟们的成功而欢呼雀跃，甚至再也不会时把的遥寄张明信片进行嘘寒问暖。 但是在深夜仰望星空的时候，他们还是会发现他们的心底深处的程序员基因还是会不停的闪烁着，期盼着这闪烁的火花能照亮整个银河系并带来和平。 ![devops](./../img/devops2.png "devops") 在这场自私且以自我为中心的欲征服世界的赛跑旅程里，程序员的子孙们早把他们真正的工作目标置之脑后-为客户解决问题。面对一拖再拖的项目交付日期，昂贵的开发代价，甚至最终失败的项目，客户们开始对这种情况深恶痛绝。 偶尔，也会有一个闪亮的明星站出来，灵机一动的提供一种办法来尝试结束这种混乱并带来和平。所以瀑布开发流程就应运而生了。这是一个非常了不起的创意，因为它利用了不同团队的开发者们只在必须的时候才进行沟通的这个事实。当一个团队完成了他们的工作的时候，它就会和下游的团队进行交流并把任务进行往下传，如此一级接一级的传递下去，永不回首。 ![devops](./../img/devops3.png "devops") 这种方式在一段时间内发挥了效用，但很快，一如既往，贪婪的人们(客户)又开始提出更多的诉求。他们希望能够更多地参加到整个软件的开发流程中来，不时的提出他们的建议，甚至在很晚的时候还提出改需求这种丧心病狂的事情来。 结果就是如大家有目共睹的事实一样，软件项目非常容易失败这个说法已经作为一个行业标准被人们所接受。数据表明超过50%的项目最终都是以失败告终的。更可悲的是，在当时看来，人们对这种情况是束手无策。 值得庆幸的是，每一个时代总会有那么几个思想开放的英雄如漆黑中的萤火虫般冒出来。他们知道这些不同团队的开发者们必须要找到一个可以协同工作、进行交流、并且能够弹性的向客户保证对方将会拿到最优的解决方案的方式。这种尝试最早可以追溯到1957年，伟大的约翰·冯·诺依曼和同行们的努力。但是我们最终却是等到2001年才收获到革命的果实，当时行业的十多个精英创造出了如今闻名世界的“敏捷宣言”。 ## 敏捷宣言基于以下十二条原则： 1. 我们的首要任务是通过尽早地、持续地交付可评价的软件来使客户满意。 2. 乐于接受需求变更，即使是在开发后期也应如此。敏捷过程能够驾驭变化，从而为客户赢得竞争优势。 3. 频繁交付可使用的软件，交付间隔越短越好，可以从几个星期到几个月。 4. 在整个项目开发期间，业务人员和开发人员必须朝夕工作在一起。 5. 围绕那些有推动力的人们来构建项目。给予他们所需的环境和支持，并且信任他们能够把工作完成好。 6. 与开发团队以及在开发团队内部最快速、有效的传递信息的方法就是，面对面的交谈。 7. 可使用的软件是进度的主要衡量指标。 8. 敏捷过程提倡可持续发展。出资人、开发人员以及使用者应该总是共同维持稳定的开发速度。 9. 为了增强敏捷能力，应持续关注技术上的杰出成果和良好的设计。 10. 简洁——最大化不必要工作量的艺术——是至关重要的。 11. 最好的架构、需求和设计都源自自我组织的团队。 12. 团队应该定期反思如何能变得更有战斗力，然后相应地转变并调整其行为。 敏捷宣言是为银河系带来和平以及维护各自的平衡所迈出的很重要的第一步。在很长的时间里，相比此前基于流程和机械化的方式，这是第一次基于文化和“人性”来将不同的关键项目关系人连接在一起的方式。人们开始互相交流，进行基本的碰头会议，并开始不断的交流意见和看法。他们开始意识到他们是有着很多比想象中还多的共同点的，客户也开始成为他们之中的一员，而不再是像以往一样只是往项目砸钱然后开始求神拜佛祈求一切顺利如愿。 ![devops](./../img/devops4.png "devops") 尽管前面还是有不少的障碍需要克服，但是未来已经光明了许多。敏捷意味着开放和拥抱(需求)改变。但是，如果改变过多的话，人们就很难专注到最终的目标和交付上来。此时精益软件开发就开始破土而出了。 因为对精益软件开发的着迷以及为了达成放逐和驱赶风险的目的，一些程序员的子孙们就开始探首窗外，开始向软件之外的行业进行取经。他们从一家主要的汽车生产商身上找到了救赎。丰田生产系统在精益上面的成就是不可思议的，同时它们的精益生产的经验也是很容易应用到软件开发上来的。 ## 精益有以下7个原则： 1. 杜绝浪费 2. 内建质量 3. 创建知识(放大学习) 4. 延迟决策(尽量延迟决定) 5. 快速交付 6. 尊重人员(团队授权) 7. 全局优化 将这些放到敏捷上去的话，精益原则就能让人们在从精神上关注做正确的事情，同时还能够让整个开发流程拥有足够的弹性。 一旦敏捷和精益软件开发被软件开发团队采纳，那么下一步就是把这一套原则应用到IT团队上来。把IT也纳入到整体战略上，然后我们就来到了DevOps跟前了！ ## 进入DevOps – 高速公路的三条车道 老一派的软件开发团队成员会包含业务分析员，系统架构师，前端开发者，后端开发者，测试员，等等。优化如敏捷和精益原则等的软件开发流程的关注点就在这些地方。比如，软件一旦达到”可以生产“的程度，就会发到系统工程师、发布工程师、DBA、网络工程师，安全专家这些“运维人员”的手上。这里该如何将横在Dev(开发)和Ops(运维)之间的鸿沟给填平，这就是DevOps的主要关注点了。 DevOps是在整个IT价值流中实施精益原则的结果。IT价值流将开发延伸至生产，将由程序员这个遥远的祖宗所繁衍的所有子孙给联合在一起。 这是来自Gene Kim的对DevOps的最好的解析，如果你还没有看过他的《凤凰项目》这本书的话，我建议你真的该好好花时间看看。 你不应该重新招聘DevOps工程师，且DevOps也不应该是一个IT的新部门。DevOps是一种文化，一种理念，且是和IT糅合成一整体的。世间没有任何工具可以把你的IT变成一个DevOps组织，也没有任何自动化方式可以指引你该如何为你的客户提供最大化的效益。 DevOps通常作为下面这三个方式而为人所熟知，而在我眼里我是把它们看成是一条高速公路上的三条车道。你从第一条车道开始，然后加速进入到第二条车道，最终在第三车道上高速行驶。 车道1 – 系统级别的整体效率考量是最主要的关注点，这超过对系统中任何一个单独个体元素的考虑 车道2 – 确保能提供持续不断的反馈循环，且这些反馈不被忽视。 车道3 – 持续的学习和吸取经验，不停的进步，快速的失败。 ## 车道1 – 获取速度 要采纳DevOps的原则，理解整个运作系统的重要性并对工作事项进行合适的优先级排序是组织首先要学的事情。在整个价值流中不能允许任何人产生瓶颈并降低整个工作流程。 ![devops](./../img/devops5.png "devops") 确保工作流程的不可中断是身处流程中的所有成员的终极目标。无论一个成员或者团队的角色是什么，他们都必须力图对整个系统进行深入的理解。这种思维方式对质量会有着直接的影响，因为缺陷永远不会被下放到“下游“中，这样做的话将会导致瓶颈的产生。 确保整个工作流程不会被瓶颈堵塞住还不够。一个高产的组织应该时常考虑该如何提升整个工作流程。有很多方法论可以做到这一点，你不妨去看下“约束理论”，“六西格玛”，精益，或者丰田生产系统。 DevOps原则不关心你身处哪个团队，你是否是系统架构师，DBA，QA，或者是网络管理员。相同的规则覆盖所有的成员，每个成员都应该遵循两个简单的原则： 保持系统运作流程不可中断 随时提升和优化工作流程 ## 车道2 – 换挡加速 不可中断的系统流程是定向的，且预期是从开发流向运维。在一个理想的世界中，这就意味着快速的开发出高质量的软件，部署，并为客户提供价值。 但是，DevOps并非乌托邦式的理想国。如果单向的交付方式是可行的话，我们的瀑布模式早就能胜任了。评估可交付产品和整个流程中的交流对确保质量是至关重要的。这里首个必须实现的”面向上游”的交流通道是从Ops到Dev。 ![devops](./../img/devops6.png "devops") 我们独自意淫是件非常容易的事情，但是获取别人的反馈和提供反馈给别人才是探究事实真相的正确方法。下游的每一步(反馈)都必须紧跟着有一个上游的确定。 你如何建立反馈循环机制并不重要。你可以邀请开发人员加入技术支持团队的会议，或者将网络管理员放到Sprint计划会议中去。一旦你的反馈机制就绪，反馈能够被接收并被处理，你就已经可以说是走到了DevOps高速车道上来了。 ## 车道3 – 飞速前进 DevOps这条快速车道并不适合意志脆弱的人。为了进入这条车道，你的组织必须要足够的成熟。这里充满了冒险和对失败教训的学习，不断的尝试，并认同屡败屡战和不断的实践是走向成功这条康庄大道的前提条件。在这里你应该会经常听到”套路“这个词，这是有原因的。不断的训练和重复所以能培养出大师，是因为其让复杂的动作常规化。 但是在你要将这些复杂的动作连接起来之前，你很有必要先去掌握好每一个单独步骤。 “适合大师的动作并不适合新手，脱胎换骨之前你必须先要明白道的真谛。“ ![devops](./../img/devops7.png "devops") DevOps的第三个方式/快速车道包括每天分配时间来持续的进行试验，时常的奖励敢于冒险的团队，并将缺陷特意引入到运作系统上来以增加系统的抗击打能力。 为了确保你的组织能够消化好这些方法，你必须在每个团队之间建立好频繁的反馈循环，同时需要确保所有的瓶颈都能够及时的被清理掉，并确保整个系统的运作流程是不可中断的。 实施好这些措施可以让你的组织时刻保持警惕，并能够快速且高效的应对挑战。 ## 概要 – DevOps清单 下面是一张你可以用来检验你的组织对DevOps的应用情况的清单。当然你也可以在文章评论后面给出你的观点。 1. 开发团队和运维团队之间没有障碍。两者皆是DevOps统一流程的一部分。 2. 从一个团队流到另一个团队的工作都能够得到高质量的验证 3. 工作没有堆积，所有的瓶颈都已经被处理好。 4. 开发团队没有占用运维团队的时间，因为部署和维护都是处于同一个时间盒里面的。 5. 开发团队不会在周五下午5点后把代码交付进行部署，剩下运维团队周末加班加点来给他们擦屁股 6. 开发环境标准化，运维人员可以很容易將之扩展并进行部署 7. 开发团队可以找到合适的方式交付新版本，且运维团队可以轻易的进行部署。 8. 每个团队之间的通信线路都很明确 9. 所有的团队成员都有时间去为改善系统进行试验和实践 10. 常规性的引入(或者模拟)缺陷到系统中来并得到处理。每次学习到的经验都应该文档化下来并分享给相关人员。事故处理成为日常工作的一部分，且处理方式是已知的 ## 总结 使用现代化的DevOps工具，如Chef、Docker、Ansible、Packer、Troposphere、Consul、Jenkins、SonarQube、AWS等，并不代表你就在正确的应用DevOps的原则。DevOps是一种思维方式。我们所有人都是该系统流程的一部分，我们一起分享共同的时光和交付价值。每个参加到这个软件交付流程上来的成员都能够加速或减缓整个系统的运作速度。系统出现的一个缺陷，以及错误配置的团队之间的“防火墙”，都可能会使得整个系统瘫痪， 所有的人都是DevOps的一部分，一旦你的组织明白了这一点，能够帮你管理好这些的工具和技术栈就自然而然的会出现在你眼前了。 ================================================ FILE: docs/exp/four_deep_learning.md ================================================ # 数据科学家必知的 5 个深度学习框架 从出道起，我就一直是一名程序员。我喜欢从头开始编写代码，这有助于我清楚地理解主题（或技巧）。当我们刚开始学习数据科学时，这种方法尤为有用。 尝试从无到有地实现一个神经网络，你将会明白很多有趣的事情。但是当需要为现实世界的数据集构建深度学习模型时，这还是一个不错的主意吗？如果你需要几天或几周的时间来建立起模型，这是完全不可能的。 对于那些无法访问无限计算资源的人来说，你们已经来到了正确的地方。 ![deeplearn](./../img/deeplearning.png) 值得庆幸的是，我们现在已经有了易于使用的开源深度学习框架，旨在简化复杂和大规模深度学习模型的实现。使用这些神奇的框架，我们可以实现诸如卷积神经网络这样复杂的模型。 在本文中，将介绍5种非常有用的深度学习框架、它们的优点以及应用。我们将对每个框架进行比较，以了解何时何地可以使用它们。 **我们还创建了一个非常酷的针对每个深度学习框架的信息图表，附在在文章的末尾，为每个数据科学家所必备。** ## 目录 * 一、什么是深度学习框架？ * 二、TensorFlow * 三、Keras * 四、PyTorch * 五、Caffe * 六、Deeplearning4j * 七、五个深度学习框架之间的对比 ### 一、什么是深度学习框架？ 让我们用一个例子来理解这个概念，来看以下图像集合： ![deep](./../img/deep1.png) 在这个图像中有不同的分类：猫，骆驼，鹿，大象等。我们的任务是将这些图像归到相应的类（或类别）中。用Google搜索一下就能知道：**卷积神经网络（CNNs）**对于这类图像分类任务十分有效。 我们要做的工作就是实现这个模型，对吗？如果从头开始编写一个卷积神经网络，则需要几天（甚至几周）才能得到一个有效的模型，我们却没法等这么长的时间！ 这正是深度学习框架真正改变了局面的地方。 ![deep](./../img/deep2.png) >深度学习框架是一种界面、库或工具，它使我们在无需深入了解底层算法的细节的情况下，能够更容易、更快速地构建深度学习模型。深度学习框架利用预先构建和优化好的组件集合定义模型，为模型的实现提供了一种清晰而简洁的方法。 利用恰当的框架来快速构建模型，而无需编写数百行代码，一个良好的深度学习框架具备以下关键特征： * 优化的性能 * 易于理解和编码 * 良好的社区支持 * 并行化的进程，以减少计算 * 自动计算梯度 这五点也是我用来挑选五大顶级深度学习框架的标准。下面让我们详细研究一下它们。 ### 二、TensorFlow ![deep](./../img/TensorFlow.png) TensorFlow是由谷歌大脑团队的研究人员和工程师开发的，它是深度学习领域中最常用的软件库（尽管其他软件正在迅速崛起）。 ![deep](./../img/deep3.png) >我喜欢TensorFlow的原因有两点：它完全是开源的，并且有出色的社区支持。TensorFlow为大多数复杂的深度学习模型预先编写好了代码，比如递归神经网络和卷积神经网络。 TensorFlow如此流行的最大原因之一是支持多种语言来创建深度学习模型，比如Python、C和R，并且有不错的文档和指南。 TensorFlow有许多组件，其中最为突出的是： * Tensorboard：帮助使用数据流图进行有效的数据可视化 * TensorFlow：用于快速部署新算法/试验 TensorFlow的灵活架构使我们能够在一个或多个CPU（以及GPU）上部署深度学习模型。下面是一些典型的TensorFlow用例： * 基于文本的应用：语言检测、文本摘要 * 图像识别：图像字幕、人脸识别、目标检测 * 声音识别 * 时间序列分析 * 视频分析 用例远远不止这些，如果你知道TensorFlow还有以上所述之外的其他应用，我很乐意知道！可以在本文的评论部分告诉我，我们再做讨论。 安装TensorFlow也是一个非常简单的任务。 对于CPU： ``` python pip install tensorflow ``` 对于启用CUDA的GPU卡： ``` python pip install tensorflow-gpu ``` 通过以下综合教程了解如何使用TensorFlow建立神经网络模型： * [利用TensorFlow实现神经网络简介](https://www.analyticsvidhya.com/blog/2016/10/an-introduction-to-implementing-neural-networks-using-tensorflow/?utm_source=blog&utm_medium=comparison-deep-learning-framework) * [TensorFlow教程](https://www.tensorflow.org/tutorials) ### 三、Keras ![deep](./../img/keras.png) 你习惯使用Python吗？如果是，那么可以立即连接到Keras。这是一个开启你的深度学习之旅的完美的框架。 Keras用Python编写，可以在TensorFlow（以及CNTK和Theano）之上运行。TensorFlow的接口具备挑战性，因为它是一个低级库，新用户可能会很难理解某些实现。 而Keras是一个高层的API，它为快速实验而开发。因此，如果希望获得快速结果，Keras会自动处理核心任务并生成输出。Keras支持卷积神经网络和递归神经网络，可以在CPU和GPU上无缝运行。 深度学习的初学者经常会抱怨：无法正确理解复杂的模型。如果你是这样的用户，Keras便是你的正确选择！它的目标是最小化用户操作，并使其模型真正容易理解。 可以将Keras中的模型大致分为两类： 1. 序列化 模型的层是按顺序定义的。这意味着当我们训练深度学习模型时，这些层次是按顺序实现的。下面是一个顺序模型的示例： ``` python from keras.models import Sequential from keras.layers import Dense model = Sequential() # we can add multiple layers to the model using .add() model.add(Dense(units=64, activation='relu', input_dim=100)) model.add(Dense(units=10, activation='softmax')) ``` 2. Keras 函数API 用于定义复杂模型，例如多输出模型或具有共享层的模型。请查看下面的代码来理解这一点： ``` python from keras.layers import Input, Dense from keras.models import Model inputs = Input(shape=(100,)) # specify the input shape x = Dense(64, activation='relu')(inputs) predictions = Dense(10, activation='softmax')(x) model = Model(inputs=inputs, outputs=predictions) ``` Keras有多种架构，如下所述，用于解决各种各样的问题，其中包括我的最爱之一：图像分类！ * VGG 16 * VGG 19 * InceptionV 3 * Mobilenet及更多 可以参考官方的Keras文档来详细了解框架是如何工作的。 [Keras官方中文文档](https://keras.io/zh/​) 仅需一行代码即可安装Keras： ```python pip install keras ``` 对Keras感兴趣？可以继续学习以下教程，了解如何使用Keras实现神经网络： [基于Keras的神经网络优化](https://www.analyticsvidhya.com/blog/2016/10/tutorial-optimizing-neural-networks-using-keras-with-image-recognition-case-study/?utm_source=blog&utm_medium=comparison-deep-learning-framework) ### 四、PyTorch 还记得我们说过TensorFlow是目前最常用的深度学习框架吗？但是如果考虑到数据科学家和开发者们拥抱Facebook的PyTorch的速度，那它可能很快就要落伍了。 我是PyTorch的拥护者，在我所研究过的框架中，PyTorch最富灵活性。 PyTorch是Torch深度学习框架的一个接口，可用于建立深度神经网络和执行张量计算。Torch是一个基于Lua的框架，而PyTorch则运行在Python上。 PyTorch是一个Python包，它提供张量计算。张量是多维数组，就像numpy的ndarray一样，它也可以在GPU上运行。PyTorch使用动态计算图，PyTorch的Autograd软件包从张量生成计算图，并自动计算梯度。 与特定功能的预定义的图表不同，PyTorch提供了一个框架，用于在运行时构建计算图形，甚至在运行时也可以对这些图形进行更改。当不知道创建神经网络需要多少内存的情况下，这个功能便很有价值。 可以使用PyTorch处理各种来自深度学习的挑战，包括： * 影像（检测、分类等） * 文本(NLP) * 增强学习 想知道如何在机器上安装PyTorch，请稍等片刻。安装步骤取决于操作系统、需要安装的PyTorch包、正在使用的工具/语言、CUDA等其他一些因素。 根据此链接的内容检查PyTorch安装步骤，准备好框架之后，再检查以下两个资源，利用PyTorch构建第一个神经网络： * [学习如何使用PyTorch来构建快速和准确的神经网络-4个不错的案例研究](https://www.analyticsvidhya.com/blog/2019/01/guide-pytorch-neural-networks-case-studies/https://www.analyticsvidhya.com/blog/2019/01/guide-pytorch-neural-networks-case-studies/?utm_source=blog&utm_medium=comparison-deep-learning-framework) * [PyTorch教程](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) ### 五、Caffe CAFE是另一个面向图像处理领域的、比较流行的深度学习框架，它是由贾阳青（Yangqing Jia）在加利福尼亚伯克利大学读博士期间开发的。同样，它也是开源的！ 首先，Caffe对递归网络和语言建模的支持不如上述三个框架。但是Caffe最突出的地方是它的处理速度和从图像中学习的速度。 ![deep](./../img/deep4.png) >Caffe可以每天处理超过六千万张图像，只需单个NVIDIA K40 GPU，其中 1毫秒/图像用于推理，4毫秒/图像用于学习。 它为C、Python、MATLAB等接口以及传统的命令行提供了坚实的支持。 通过Caffe Model Zoo框架可访问用于解决深度学习问题的预训练网络、模型和权重。这些模型可完成下述任务： * 简单的递归 * 大规模视觉分类 * 用于图像相似性的SiameSE网络 * 语音和机器人应用 有关更多细节，您可以查看Caffe相关文档。 [Caffe安装文档](http://caffe.berkeleyvision.org/installation.html) [Caffe文档](http://caffe.berkeleyvision.org/) ### 六、Deeplearning4j ![deep](./../img/deeplearning4j.png) 我们社区中有Java程序员吗？这是你理想的深度学习框架！Deeplearning4j是用Java实现的，因此与Python相比效率更高。它使用称为ND4J的张量库，提供了处理n维数组（也称为张量）的能力。该框架还支持CPU和GPU。 Deeplearning4j将加载数据和训练算法的任务作为单独的过程处理，这种功能分离提供了很大的灵活性。谁都喜欢这样，尤其是在深度学习中！ Deeplearning4j也适用于不同的数据类型： * 图像 * CSV * 纯文本等 可以使用Deeplearning4j构建的深度学习模型有： * 卷积神经网络(CNNs) * 递归神经网络(RNNs) * 长短时记忆(LSTM)等多种结构 阅读Deeplearning4j的安装步骤和文档，开始使用这个框架。 [Deeplearning4j安装步骤](https://deeplearning4j.org/docs/latest/deeplearning4j-config-gpu-cpu) [Deeplearning4j文档](https://deeplearning4j.org/docs/latest/deeplearning4j-quickstart) ### 七、五种深度学习框架之间的对比 上面已经讨论了五个最流行的深度学习框架，每一个都独具特性，那么数据科学家会如何做出选择呢。 你决定用哪一种了吗？或者你打算换一个全新的框架？不管是什么情况，了解每个框架的优点和局限性非常重要。如果选对了正确的框架，当遇到错误时，便不会感到惊讶了！ 某些框架在处理图像数据时工作得非常好，但无法解析文本数据；某些框架在处理图像和文本数据时，性能很好，但是它们的内部工作原理很难理解。 在本节中，将使用以下标准比较这五个深度学习框架： * 社区支持力度 * 使用的语言 * 接口 * 对预训练的模型的支持 下表对这些框架进行了比较： ![deep](./../img/deepmore.png) 对于选择使用的框架来说，这是一个非常方便的对比表！ 所有这些框架都是开源的，支持CUDA，并有预训练的模型。**但是，应该如何正确开始，应该选择哪个框架来构建（初始）深度学习模型？**让我们来做详细的讨论！ * [TensorFlow](https://www.tensorflow.org) 我们先来说说TensortFlow。TensorFlow能处理图像以及基于序列的数据，如果你是深度学习的初学者，或者对线性代数和微积分等数学概念没有坚实的基础，那么TensortFlow的学习曲线将会令人畏惧地陡峭。 我完全理解，对于刚起步的人来说，这可能太复杂。但我建议你不断练习，不断探索社区，并继续阅读文章以掌握TensorFlow的诀窍。一旦对这个框架有了一个很好的理解，实现一个深度学习模型对你来说将是易如反掌。 * [Keras](https://keras.io/zh/​) Keras是一个非常坚实的框架，可以开启深度学习之旅。如果你熟悉Python，并且没有进行一些高级研究或开发某种特殊的神经网络，那么Keras适合你。 Keras的重点更多地放在取得成果上，而不是被模型的复杂之处所困扰。因此，如果有一个与图像分类或序列模型相关的项目，可以从Keras开始，很快便可以构建出一个工作模型。 Keras也集成在TensorFlow中，因此也可以使用tf.keras.构建模型。 * [PyTorch](https://pytorch.org/) 与特定功能的预定义的图表不同，PyTorch提供了一个框架，用于在运行时构建计算图形，甚至在运行时也可以对这些图形进行更改。当不知道创建神经网络需要多少内存的情况下，这个功能便很有价值。 * [Caffe](http://caffe.berkeleyvision.org/) 在图像数据上构建深度学习模型时，Caffe是不错的选择。但是，当谈到递归神经网络和语言模型时，Caffe落后于我们讨论过的其他框架。**Caffe的主要优点是，即使没有强大的机器学习或微积分知识，也可以构建出深度学习模型。** Caffe主要用于建立和部署移动电话和其他计算受限平台的深度学习模型。 * [Deeplearning4j](https://deeplearning4j.org/docs/latest/deeplearning4j-quickstart) 正如之前所述，DeepleEarning4J是Java程序员的天堂。它为CNNS、RNN和LSTMS等不同的神经网络提供了大量的支持，它在不牺牲速度的情况下可以处理大量数据。听起来不错，有机会通过！ ### 后记及图示信息图 记住，这些框架基本上只是帮助我们实现最终目标的工具，正确地选择它们可以减少大量的精力和时间。 除了文中提及的五种深度学习框架之外，你有没有其他的深度学习框架？很想听听你的想法和反馈。请issues me。 ================================================ FILE: docs/exp/learnweetout.md ================================================ # 程序员如何技术成长 在浩大的软件世界里，作为一名普通程序员，显得十分渺小，甚至会感到迷茫。 我们内心崇拜技术，却也对日新月异的技术抱有深深的恐惧。有时候我会思考难道在 技术领域内不断紧跟新潮，不断提升技能就是我的价值所在？那么我是技术的主人还 是技术的奴隶？ 人之所以迷茫往往是找不到工作生活的重心，感受不到工作或生活的价值。那么 什么是价值呢？说的大一点就是我改变了世界，说的小一点就是我的所作所为改善了 某些问题。如果不清楚自己的行为、目标、价值三者的关系，那么又何来重心？又如 何能分得清重要性与优先级呢？ 程序员的迷茫不仅仅是面对技术繁杂的无力感，更重要的是因为长期埋没于软件 世界的浩大的分工体系中，无法看清从业务到软件架构的价值链条，无法清楚定位自 己在分工体系的位置，处理不好自身与技术、业务的关系所致。 很多程序员打心底不喜欢业务，这一点我曾经也经历过，我更宁愿从事框架工 具、技术组件研究的相关事情。我有个朋友经常吐槽我说：”你们天天加班加点写了 那么多代码，然后呢？有改变什么吗？还不是写出了一堆垃圾。”仔细想想很多时候 业务在我们脑海中存留的只是逻辑和流程，我们丢失的是对业务场景的感受，对用 户痛点的体会，对业务发展的思考。这些都是与价值紧密相关的部分。我们很自然的 用战术的勤快掩盖战略的懒惰！那么这样的后果就是我们把自己限死在流水线的工位 上，阉割了自己能够发现业务价值的能力，而过多关注新技术对职场竞争力的价值。 这也就是我们面对繁杂技术，而产生技术学习焦虑症的根本原因。 ### 业务、技术与软件系统的价值链 那么什么是业务呢？就是指某种有目的的工作或工作项目，业务的目的就是解 决人类社会与吃喝住行息息相关的领域问题，包括物质的需求和精神的需求，使开展 业务活动的主体和受众都能得到利益。通俗的讲业务就是用户的痛点，是业务提供方 （比如公司）的盈利点。而技术则是解决问题的工具和手段。比如为了解决用户随时随 地购物的业务问题时，程序员利用 web 技术构建电子商务 App，而当需求升级为帮 助用户快速选购商品时，程序员会利用数据算法等技术手段构建推荐引擎。 技术如果 脱离了业务，那么技术应用就无法很好的落地，技术的研究也将失去场景和方向。而 业务脱离了技术，那么业务的开展就变得极其昂贵和低效。 所以回过头来我们想想自己没日没夜写了那么多的代码从而构建起来的软件系 统，它的价值何在呢？说白了就是为了解决业务问题，所以当你所从事的工作内容并 不能为解决业务问题带来多大帮助的时候，你应该要及时做出调整。那么软件系统又 是如何体现它自身的价值呢？在我看来有如下几个方面的体现： **业务领域与功能：** 比如支付宝立足支付领域而推出的转账、收款功能等，比如人 工智能自动驾驶系统等。 **服务能力：** 这就好比火车站购票窗口，评判它的服务能力的标准就是它能够同时 处理多少用户的购票业务，能不能在指定时间内完成购票业务，能不能 7*8 小时持续 工作。对应到软件系统领域，则表现为以下三个方面： * 系统正确性 ( 程序能够正确表述业务流程，没有 Bug) * 可用性（可以 7 ＊ 24 小时＊ 365 不间歇工作） * 大规模（高并发，高吞吐量） 互联网公司正是借助大规模的软件系统承载着繁多的业务功能，使其拥有巨大的 服务能力并借助互联网技术突破了空间限制，高效低廉解决了业务问题，创造了丰厚的 利润，这是人肉所不可比拟的。 理解了这一层面的概念，你就可以清楚这个价值链条：公司依靠软件系统提供业 务服务而创造价值，程序员则是通过构建并持续演进软件系统服务能力以及业务功能 以支撑公司业务发展从而创造价值。 有了这个价值链条，我们就可以反思自己的工作学习对软件系统的服务能力提升 起到了多大的推动作用？可以反思自己的工作学习是否切实在解决领域的业务问题， 还是只是做一些意义不大的重复性工作。 前两天面试了一个候选人，他的工作是从事票务系统开发，他说自己在研究 linux 内核与汇编语言，我就问他 linux 内核和汇编语言的学习对你的工作产生了哪些 帮助？能否举一个例子？他哑口无言，我内心就觉得这样一个热爱学习的好苗子正迷 茫找不到重心，正在做一件浪费精力的事情。正确的学习方式应该是将学习与具体业 务场景结合起来，和公司通过软件系统开展业务服务而创造价值，程序员通过提升软 件系统服务能力创造价值这一链条串接起来，从对这些价值产生帮助的程度去思考优 先级。学习本身没有错，错的往往就是那颗初心。 现在你再来看高并发分布式相关的知识，你会发现并不是因为这些知识比较高深、 比较时髦，很多公司有需求才值得学习，而是他们对价值链条有着实实在在的贡献。 ### 价值驱动的架构 一谈到软件系统，人们免不了想起架构这件事来。之所以此处去谈及架构是因为 每一个程序员本质都是软件架构体系中的一分子，我们可能深埋于体系流水线之中， 感受不到位置和价值。但如果站在架构这一高度去看这些问题则将会非常透彻。那么 架构究竟是什么？和上述的价值链又有什么关系呢？ ### 什么是架构？ 在我看来软件架构就是将人员、技术等资源组织起来以解决业务问题，支撑业务 增长的一种活动。可能比较抽象，我想我们可以从架构师的一些具体工作任务来理解 这句话含义： * 组织业务：架构师通过探索和研究业务领域的知识，构建自身看待业务的”世界 观”。他会基于这种认识拆分业务生命周期，确立业务边界，构建出了一套解决特定 业务问题的领域模型，并且确认模型之间、领域之间的关系与协作方式，完成了对业 务领域内的要素的组织工作。 * 组织技术：为了能在计算机世界中运作人类社会的业务模型，架构师需要选用计 算机世界中合适的框架、中间件、编程语言、网络协议等技术工具依据之前设计方 案组织起来形成一套软件系统方案，在我看来软件系统就像是一种技术组织，即技 术组件、技术手段依据某种逻辑被组织起来了，这些技术工具被确定了职责，有了明 确分工，并以实现业务功能为目标集合在了一起。比如 RPC 框架或消息队列被用 于内部系统之间的通信服务就如同信使一般，而数据库则负责记录结果，它更像是 一名书记员。 * 组织人员：为了能够实现利用软件系统解决业务问题的目标，架构师还需要关注 软件系统的构建过程，他以实现软件系统为号召，从公司组织中聚集一批软件工程 师，并将这些人员按不同工种、不同职责、不同系统进行组织，确定这些人员之间的 协作方式，并关注这个组织系统是否运作良好比如沟通是否顺畅、产出是否达到要 求、能否按时间完成等。 * 组织全局，对外输出：架构师的首要目标是解决业务问题，推动业务增长。所以 他非常关心软件的运行状况。因为只有在软件系统运行起来后，才能对外提供服务， 才能在用户访问的过程中，解决业务问题。架构师需要关注运行过程中产生的数据比 如业务成功率，系统运行资源占用数据、用户反馈信息、业务增长情况等，这些信息 将会帮助架构师制定下一步架构目标和方向。 所以软件架构不仅仅只是选用什么框架、选用什么技术组件这么简单。它贯穿了 对人的组织、对技术的组织、对业务的组织，并将这三种组织以解决业务问题这一目 标有机的结合在了一起。 很多面试的候选人在被问及他所开发的系统采用什么架构的问题时，只会罗列出 一些技术组件、技术框架等技术要素，这样看来其根本没有理清架构的深层含义。也 有一些架构师只专注对底层技术的研究，以为打造一个卓越的系统是非常牛逼的事 情，可是他忽略了软件系统的价值是以解决业务问题的能力、支撑业务增长的能力为 衡量标准，所以最后生产出了很多对组织，对业务没有帮助的系统。 ### 成本与收益 正如之前所说软件系统只有在运行的时候才能创造价值，也就是说软件系统能否 7*24 小时＊ 365 天稳定的工作关系到公司的收益水平。所以开发团队对生产环境的 发布总是小心翼翼，对解决生产环境的问题总是加班加点。而软件系统的成本则体现 在软件构建过程，这时候我们就能理解那些工程技术如项目管理、敏捷开发、 单元测 试、持续集成、持续构建，版本管理等的价值了，他们有的是保证软件系统正确性， 有的是为了降低沟通成本，有的是为了提升开发效率等但总的来说就是为了降低软件 的构建成本。所以在提升系统服务能力，创造更多业务收益的同时，降低构建成本也 是一种提升收益的有效手段。 作为一名软件工程师而言，我们往往处在软件构建过程体系中的某个环节，我们 可以基于成本与收益的关系去思考自己每一项技能的价值，学习新的有价值的技能， 甚至在工作中基于成本与收益的考量选择合适的技术。比如在逻辑不大发生变化的地 方，没有必要去做过多的设计，应用各种花俏的设计模式等浪费时间。这样我们才能 成为技术的主人。 ### 架构目标需要适应业务的发展 架构的目标就是为了支撑业务增长，就是提升软件系统的服务能力。可是话虽说 如此，但真实却要做很多取舍。比如对初创团队而言，其产品是否解决业务问题这一 设想还没得到确认，就立即去构造一个高性能、高可用的分布式系统，这样的架构目 标远超出业务发展的需求，最后的结果就是浪费大量人力物力，却得不到任何起色。 架构师需要审时度势，仔细衡量正确性、大规模、可用性三者的关系，比如今年业务 蓬勃发展日均订单 300 万，基于对未来的可能预测，明年可能有 3000 万的订单，那 么架构师应该要着重考虑大规模和可用性。而且每一点提升的程度，也需要架构师衡 量把握，比如可用性要达到 2 个 9 还是 3 个 9。 回顾自己以往的工作很多时候就是因为没有确立架构目标导致浪费了组织很多资 源，比如在之前的创业团队中，由于本人有一定的代码洁癖，经常会花费很多时间和 同事计较代码质量，这样本可以更快上线的功能却需要被延迟，当时过度追求正确性 的行为是与创业团队快速验证想法的业务需求不匹配的。 另外一点比较深刻的案例则是在本人担任一个技术团队负责人的时候，在一次述 职报告的时候，leader 问我对接下来团队工作有什么计划？我当时说了一堆什么改进 代码质量，每天晨会，任务透明化，建立迭代机制等等，然后就被各种批驳一通。当 时团队基本以外包人员为主，人员水平较差，开发出来的金融系统也是千疮百孔而这 条业务线最重要的业务价值则是按计划实现潜在投资方的需求，争取拉到投资。所以 不久leader就召集测试架构的相关人员与我这边一同梳理对核心功能的测试工作， 将研发、测试、上线的流程自动化。 当时并不理解这样做核心价值是什么。但回过头来看这样的工作方式恰好符合了 业务发展的需求，即确保系统是符合设计需求的，保证系统达到可接受的正确性，为 后续能过快速前进打下基础，最重要的是为企业降低了构建成本。所以程序员想要工 作出业绩，必须认清楚系统背后的业务价值，按价值去梳理工作优先级，而不是像我 一般过度纠结细节，追求技术理想化。 ### 成也分工，败也分工 正如在程序员的迷茫那一章节提到的：程序员的迷茫因为长期埋没于软件世界的 浩大的分工体系中，无法看清从业务到软件架构的价值链条，无法清楚定位自己在分 工体系的位置，处理不好自身与技术、业务的关系所致，所以在这里我想谈谈分工。 架构师为了使软件系统更好的服务业务，必然将软件系统生命周期进行拆分，比如分 出开发生命周期、测试生命周期、用户访问生命周期、软件运维生命周期，并根据不 同的生命周期划分出不同的职责与角色。 比如开发人员负责开发周期负责完成软件研发，测试人员负责对开发人员交付的 成果进行测试等，于是就形成了分工。一旦分工形成，每一个分工组织都会有自己的 价值追求，架构师关注的顶层的价值即软件系统能否支撑业务增长被分工的形式打 碎到各个组织中。分工是有其价值的，他使得复杂昂贵的任务可以被简单、并行、 可替换的流水线方式解决。但久而久之，价值碎片化的问题就出现了，比如测试人 员只关注找出更多问题，开发人员只关注快速开发更多的系统，运维人员只关注保障 系统稳定。 三者之间常常都只站在自己的立场去要求对方怎么做，没有人再关注整体价值， 产生诸多矛盾增加软件实施成本。而身处流水线中的一员，又因为困扰于重复性工 作， 迷茫于工作的意义，甚至感觉自己做为了人的创意与灵感都被扼杀了。所以我的 朋友吐槽我说你写了那么多代码然后并没有怎么样是非常有道理的，那是因为我只关 注着做为流水工人的价值要求，看不到生态链最顶端的价值。 我们仔细想想那些团队领导，精英领袖哪一个不是为着更广大的价值所负责，比如 项目经理只需要关心自身项目的商业价值，而公司 CEO 则关心公司范畴内所有业务 的总体商业价值。所以关注的价值越大且职位也就越高。这些高层领导者们把控着整 体的价值链条，及时纠正底层分工组织的价值目标与整体价值目标出现偏差的问题。 ### 从价值出发－找寻学习与工作的新思路 迷茫能引发思考，架构则塑造了视野，而价值则是我们之所以存活，之所以工作 的逻辑起点。基于这样一种价值思维，对我们的学习和工作又可以有哪些改启示呢？ * **明确自身的业务相关主体：**找出你工作的协作关系网内的业务方和客户方，这样 你就可以从客户方中找到离你最近的业务价值点，从你的业务方中挖掘更多的资源。 甚至你可以按这个思路顺着网络向上或向下挖掘价值链条，整合更多的上下游资源以 实现更大的价值。 * **向前一步，为更大的价值负责：**不要因为自己是开发人员就不去关注软件运维， 不要因为只是测试就不关注软件开发，因为你关注的越多你越能看清全局的价值目 标。如果只关注一亩三分地，那么注定这辈子只能困守在这一亩三分地里，成为一名 流水线上焦虑至死的码农。试着转变思维，从架构师的角度思考价值问题，看看能否 将技术贯穿到业务、到用户、到最终的价值去。之前我的朋友说过要把产品经理踢到 运营位置去，把程序员踢到产品经理位置去，这样才是正确做事方式。这句话也是类 似的意思，向前一步才能懂得怎么做的更好。 * **像架构师一样思考，用价值找寻重心：**人的迷茫是因为找不到重心，而价值的意 义在于引导我们思考做哪些事情才能实现价值，先做哪些事情会比后做哪些事情更能 创造收益。像架构师那样全局性思考，把遇到问题进行拆分，把学习到的事物串联起 来，努力构成完整的价值链条。 * **学会连接，构建体系：**前几天看到一篇文章对今日头条的产品形态极尽批判之 词，指责它的智能算法将人类封死在自己的喜好之中，将人类社会进一步碎片化。这 似乎很有道理，有趣的是互联网将我们连接至广袤的世界，却也把我们封闭在独属于 自己的小世界里。依旧是我的那位朋友，他说他的最大价值在于连接，将不同的人连 接在一起，有趣的事情可能就会即将发生。 或许算法的天性就是顺从与迎合，但人最终想理解这个世界还是需要依靠自身 的行动与不同人之间建立联系，这也是一种摆脱流水线限制的有效方式。另外，我们 自身也是某种事物连接的产物，比如架构师，他是业务、技术、管理连接在一起的一 种产物。所以我们应当树立自身的知识体系以吸收融合新知识，将孤立的概念连接起 来，形成自身的价值链条。比如这篇文章将我从事技术开发经验、与对架构的理解以 及自身过往经历结合起来，这也是一种内在的体系梳理。 ================================================ FILE: docs/exp/micro-service.md ================================================ # 微服务架构技术栈选型手册 2014~2018，微服务经过三年的发展，现状如何？这是一份为让你更好使用微服务的技术站选型手册。除此之外，你还可以按需选用配套的微服务架构视频内容。 ## 一、前言 2014 年可以认为是微服务 1.0 的元年，当年有几个标志性事件，一是 Martin Fowler 在其博客上发表了”Microservices”一文，正式提出微服务架构风格；二是 Netflix 微服务架构经过多年大规模生产验证，最终抽象落地形成一整套开源的微服务基础组件，统称 NetflixOSS，Netflix 的成功经验开始被业界认可并推崇；三是 Pivotal 将 NetflixOSS 开源微服务组件集成到其 Spring 体系，推出 Spring Cloud 微服务开发技术栈。 一晃三年过去，微服务技术生态又发生了巨大变化，容器，PaaS，Cloud Native，gRPC，ServiceMesh，Serverless 等新技术新理念你方唱罢我登场，不知不觉我们又来到了微服务 2.0 时代。 基于近年在微服务基础架构方面的实战经验和平时的学习积累，我想总结并提出一些构建微服务 2.0 技术栈的选型思路，供各位在一线实战的架构师、工程师参考借鉴。对于一些暂时还没有成熟开源产品的微服务支撑模块，我也会给出一些定制自研的设计思路。 ## 二、选型准则 对于技术选型，我个人有很多标准，其中下面三项是最重要的： ### 1. 生产级 我们选择的技术栈是要解决实际业务问题和上生产抗流量的（选择不慎可能造成生产级事故），而不是简单做个 POC 或者 Demo 展示，所以生产级（Production Ready），可运维（Ops Ready），可治理，成熟稳定的技术才是我们的首选； ### 2. 一线互联网公司落地产品 我们会尽量采用在一线互联网公司落地并且开源的，且在社区内形成良好口碑的产品，它们已经在这些公司经过流量冲击，坑已经基本被填平，且被社区接受形成一个良好的社区生态（本文附录部分会给出所有推荐使用或参考的开源项目的 GitHub 链接）。 ### 3. 开源社区活跃度 GitHub 上的 stars 的数量是一个重要指标，同时会参考其代码和文档更新频率（尤其是近年），这些指标直接反应开源产品的社区活跃度或者说生命力。 另外，对于不同业务体量和团队规模的公司，技术选型标准往往是不同的，创业公司的技术选型和 BAT 级别公司的技术选型标准可能完全不同。本文主要针对日流量千万以上，研发团队规模不少于 50 人的公司，如果小于这个规模我建议认真评估是否真的需要采用微服务架构。考虑到 Java 语言在国内的流行度和我个人的背景经验，本文主要针对采用 Java 技术栈的企业。本文也假定自建微服务基础架构，有些产品其实有对应的云服务可以直接使用，自建和采用云服务各有利弊，架构师需要根据场景上下文综合权衡。 ## 三、微服务基础架构关键点 下面脑图中芒果色标注的七个模块，我认为是构建微服务 2.0 技术栈的核心模块，本文后面的选型会分别基于这些模块展开。对于每个模块我也列出一些核心架构关注点，在选择具体产品时，需要尽可能覆盖到这些关注点。 ![框架](./../img/exp1.png "框架") 下图是我近期工作总结和参考的一个微服务技术体系，我想同时分享给一线架构师或者工程师参考，其中粉红色标注的模块是和微服务关系最密切的模块，大家在做技术选型时，可以同时对照这个体系。 ![框架](./../img/exp2.png "框架") ## 四、服务框架选型 服务框架是一个比较成熟的领域，有太多可选项。Spring Boot/Cloud[附录 12.1] 由于 Spring 社区的影响力和 Netflix 的背书，目前可以认为是构建 Java 微服务的一个社区标准，Spring Boot 目前在 GitHub 上有超过 20k 星。 基于 Spring 的框架本质上可以认为是一种 RESTful 框架（不是 RPC 框架），序列化协议主要采用基于文本的 JSON，通讯协议一般基于 HTTP。RESTful 框架天然支持跨语言，任何语言只要有 HTTP 客户端都可以接入调用，但是客户端一般需要自己解析 payload。目前 Spring 框架也支持 Swagger 契约编程模型，能够基于契约生成各种语言的强类型客户端，极大方便不同语言栈的应用接入，但是因为 RESTful 框架和 Swagger 规范的弱契约特性，生成的各种语言客户端的互操作性还是有不少坑的。 Dubbo[附录 12.2] 是阿里多年构建生产级分布式微服务的技术结晶，服务治理能力非常丰富，在国内技术社区具有很大影响力，目前 github 上有超过 16k 星。Dubbo 本质上是一套基于 Java 的 RPC 框架，当当 Dubbox 扩展了 Dubbo 支持 RESTful 接口暴露能力。 Dubbo 主要面向 Java 技术栈，跨语言支持不足是它的一个弱项，另外因为治理能力太丰富，以至于这个框架比较重，完全用好这个框架的门槛比较高，但是如果你的企业基本上投资在 Java 技术栈上，选 Dubbo 可以让你在服务框架一块站在较高的起点上，不管是性能还是企业级的服务治理能力，Dubbo 都做的很出色。新浪微博开源的 Motan（GitHub 4k stars）也不错，功能和 Dubbo 类似，可以认为是一个轻量裁剪版的 Dubbo。 gRPC[附录 12.3] 是谷歌近年新推的一套 RPC 框架，基于 protobuf 的强契约编程模型，能自动生成各种语言客户端，且保证互操作。支持 HTTP2 是 gRPC 的一大亮点，通讯层性能比 HTTP 有很大改进。Protobuf 是在社区具有悠久历史和良好口碑的高性能序列化协议，加上 Google 公司的背书和社区影响力，目前 gRPC 也比较火，GitHub 上有超过 13.4k 星。 目前看 gRPC 更适合内部服务相互调用场景，对外暴露 RESTful 接口可以实现，但是比较麻烦（需要 gRPC Gateway 配合），所以对于对外暴露 API 场景可能还需要引入第二套 RESTful 框架作为补充。总体上 gRPC 这个东西还比较新，社区对于 HTTP2 带来的好处还未形成一致认同，建议谨慎投入，可以做一些试点。 ## 五、运行时支撑服务选型 运行时支撑服务主要包括服务注册中心，服务路由网关和集中式配置中心三个产品。 服务注册中心，如果采用 Spring Cloud 体系，则选择 Eureka[附录 12.4] 是最佳搭配，Eureka 在 Netflix 经过大规模生产验证，支持跨数据中心，客户端配合 Ribbon 可以实现灵活的客户端软负载，Eureka 目前在 GitHub 上有超过 4.7k 星；Consul[附录 12.5] 也是不错选择，天然支持跨数据中心，还支持 KV 模型存储和灵活健康检查能力，目前在 GitHub 上有超过 11k 星。 服务网关也是一个比较成熟的领域，有很多可选项。如果采用 Spring Cloud 体系，则选择 Zuul[附录 12.6] 是最佳搭配，Zuul 在 Netflix 经过大规模生产验证，支持灵活的动态过滤器脚本机制，异步性能不足（基于 Netty 的异步 Zuul 迟迟未能推出正式版）。Zuul 网关目前在 github 上有超过 3.7k 星。基于 Nginx/OpenResty 的 API 网关 Kong[附录 12.7] 目前在 github 上比较火，有超过 14.1k 星。因为采用 Nginx 内核，Kong 的异步性能较强，另外基于 lua 的插件机制比较灵活，社区插件也比较丰富，从安全到限流熔断都有，还有不少开源的管理界面，能够集中管理 Kong 集群。 配置中心，Spring Cloud 自带 Spring Cloud Config[附录 12.8]（GitHub 0.75k stars），个人认为算不上生产级，很多治理能力缺失，小规模场景可以试用。个人比较推荐携程的 Apollo[附录 12.9] 配置中心，在携程经过生产级验证，具备高可用，配置实时生效（推拉结合），配置审计和版本化，多环境多集群支持等生产级特性，建议中大规模需要对配置集中进行治理的企业采用。Apollo 目前在 github 上有超过 3.4k 星。 ## 六、服务监控选型 主要包括日志监控，调用链监控，Metrics 监控，健康检查和告警通知等产品。 ELK 目前可以认为是日志监控的标配，功能完善开箱即用，ElasticSearch[附录 12.10] 目前在 GitHub 上有超过 28.4k 星。Elastalert[附录 12.11] (GitHub 4k stars) 是 Yelp 开源的针对 ELK 的告警通知模块。 调用链监控目前社区主流是点评 CAT[附录 12.12]（GitHub 4.3k stars），Twitter 之前开源现在由 OpenZipkin 社区维护的 Zipkin[附录 12.13]（GitHub 7.5k stars）和 Naver 开源的 Pinpoint[附录 12.14]（GitHub 5.3k stars）。个人比较推荐点评开源的 CAT，在点评和国内多家互联网公司有落地案例，生产级特性和治理能力较完善，另外 CAT 自带告警模块。下面是我之前对三款产品的评估表，供参考。 ![框架](./../img/exp3.png "框架") Metrics 监控主要依赖于时间序列数据库 (TSDB)，目前较成熟的产品是 StumbleUpon 公司开源的基于 HBase 的 OpenTSDB[附录 12.15]（基于 Cassandra 的 KariosDB[附录 12.16] 也是一个选择，GitHub 1.1k stars，它基本上是 OpenTSDB 针对 Cassandra 的一个改造版），OpenTSDB 具有分布式能力可以横向扩展，但是相对较重，适用于中大规模企业，OpenTSDB 目前在 GitHub 上有近 2.9k 星。 OpenTSDB 本身不提供告警模块，Argus[附录 12.17]（GitHub 0.29k 星）是 Salesforce 开源的基于 OpenTSDB 的统一监控告警平台，支持丰富的告警函数和灵活的告警配置，可以作为 OpenTSDB 的告警补充。近年也出现一些轻量级的 TSDB，如 InfluxDB[附录 12.18]（GitHub 12.4k stars）和 Prometheus[附录 12.19]（GitHub 14.3k stars），这些产品函数报表能力丰富，自带告警模块，但是分布式能力不足，适用于中小规模企业。Grafana[附录 12.20]（GitHub 19.9k stars）是 Metrics 报表展示的社区标配。 社区还有一些通用的健康检查和告警产品，例如 Sensu[附录 12.21]（GitHub 2.7k stars），能够对各种服务（例如 Spring Boot 暴露的健康检查端点，时间序列数据库中的 metrics，ELK 中的错误日志等）定制灵活的健康检查 (check)，然后用户可以针对 check 结果设置灵活的告警通知策略。Sensu 在 Yelp 等公司有落地案例。其它类似产品还有 Esty 开源的 411[附录 12.22]（GitHub 0.74k 星）和 Zalando 的 ZMon[附录 12.23] (GitHub 0.15k 星)，它们是分别在 Esty 和 Zalando 落地的产品，但是定制 check 和告警配置的使用门槛比较高，社区不热，建议有定制自研能力的团队试用。ZMon 后台采用 KairosDB 存储，如果企业已经采用 KariosDB 作为时间序列数据库，则可以考虑 ZMon 作为告警通知模块。 ## 七、服务容错选型 针对 Java 技术栈，Netflix 的 Hystrix[附录 12.24]（github 12.4k stars）把熔断、隔离、限流和降级等能力封装成组件，任何依赖调用（数据库，服务，缓存）都可以封装在 Hystrix Command 之内，封装后自动具备容错能力。Hystrix 起源于 Netflix 的弹性工程项目，经过 Netflix 大规模生产验证，目前是容错组件的社区标准，GitHub 上有超 12k 星。其它语言栈也有类似 Hystrix 的简化版本组件。 Hystrix 一般需要在应用端或者框架内埋点，有一定的使用门槛。对于采用集中式反向代理（边界和内部）做服务路由的公司，则可以集中在反向代理上做熔断限流，例如采用 Nginx[附录 12.25]（GitHub 5.1k stars）或者 Kong[附录 12.7]（GitHub 11.4k stars）这类反向代理，它们都插件支持灵活的限流容错配置。Zuul 网关也可以集成 Hystrix 实现网关层集中式限流容错。集中式反向代理需要有一定的研发和运维能力，但是可以对限流容错进行集中治理，可以简化客户端。 ## 八、后台服务选型 后台服务主要包括消息系统，分布式缓存，分布式数据访问层和任务调度系统。后台服务是一个相对比较成熟的领域，很多开源产品基本可以开箱即用。 消息系统，对于日志等可靠性要求不高的场景，则 Apache 顶级项目 Kafka[附录 12.26]（GitHub 7.2k stars）是社区标配。对于可靠性要求较高的业务场景，Kafka 其实也是可以胜任，但企业需要根据具体场景，对 Kafka 的监控和治理能力进行适当定制完善，Allegro 公司开源的 hermes[附录 12.27]（GitHub 0.3k stars）是一个可参考项目，它在 Kafka 基础上封装了适合业务场景的企业级治理能力。阿里开源的 RocketMQ[附录 12.28]（GitHub 3.5k 星）也是一个不错选择，具备更多适用于业务场景的特性，目前也是 Apache 顶级项目。RabbitMQ[附录 12.29]（GitHub 3.6k 星）是老牌经典的 MQ，队列特性和文档都很丰富，性能和分布式能力稍弱，中小规模场景可选。 对于缓存治理，如果倾向于采用客户端直连模式（个人认为缓存直连更简单轻量），则 SohuTv 开源的 cachecloud[附录 12.30]（GitHub 2.5k stars）是一款不错的 Redis 缓存治理平台，提供诸如监控统计，一键开启，自动故障转移，在线伸缩，自动化运维等生产级治理能力，另外其文档也比较丰富。如果倾向采用中间层 Proxy 模式，则 Twitter 开源的 twemproxy[附录 12.31]（GitHub 7.5k stars）和 CodisLab 开源的 codis[附录 12.32]（GitHub 6.9k stars）是社区比较热的选项。 对于分布式数据访问层，如果采用 Java 技术栈，则当当开源的 shardingjdbc[附录 12.33]（GitHub 3.5k stars）是一个不错的选项，分库分表逻辑做在客户端 jdbc driver 中，客户端直连数据库比较简单轻量，建议中小规模场景采用。如果倾向采用数据库访问中间层 proxy 模式，则从阿里 Cobar 演化出来的社区开源分库分表中间件 MyCAT[附录 12.34]（GitHub 3.6k stars）是一个不错选择 。proxy 模式运维成本较高，建议中大规模场景，有一定框架自研和运维能力的团队采用。 任务调度系统，个人推荐徐雪里开源的 xxl-job[附录 12.35]（GitHub 3.4k stars），部署简单轻量，大部分场景够用。当当开源的 elastic-job[附录 12.36]（GitHub 3.2k stars）也是一个不错选择，相比 xxl-job 功能更强一些也更复杂。 ## 九、服务安全选型 对于微服务安全认证授权机制一块，目前业界虽然有 OAuth 和 OpenID connect 等标准协议，但是各家具体实现的做法都不太一样，企业一般有很多特殊的定制需求，整个社区还没有形成通用生产级开箱即用的产品。有一些开源授权服务器产品，比较知名的如 Apereo CAS[附录 12.37]（GitHub 3.6k stars），JBoss 开源的 keycloak[附录 12.38]（GitHub 1.9 stars），spring cloud security[附录 12.39] 等，大都是 opinionated（一家观点和做法）的产品，同时因支持太多协议造成产品复杂，也缺乏足够灵活性。个人建议基于 OAuth 和 OpenID connect 标准，在参考一些开源产品的基础上（例如 Mitre 开源的 OpenID-Connect-Java-Spring-Server[附录 12.40]，GitHub 0.62k stars），定制自研轻量级授权服务器。Wso2 提出了一种微服务安全的参考方案 [附录 12.45]，建议参考，该方案的关键步骤如下： ![框架](./../img/exp4.png "框架") 1. 使用支持 OAuth 2.0 和 OpenID Connect 标准协议的授权服务器（个人建议定制自研）； 2. 使用 API 网关作为单一访问入口，统一实现安全治理； 3. 客户在访问微服务之前，先通过授权服务器登录获取 access token，然后将 access token 和请求一起发送到网关； 4. 网关获取 access token，通过授权服务器校验 token，同时做 token 转换获取 JWT token。 5. 网关将 JWT Token 和请求一起转发到后台微服务； 6. JWT 中可以存储用户会话信息，该信息可以传递给后台的微服务，也可以在微服务之间传递，用作认证授权等用途； 7. 每个微服务包含 JWT 客户端，能够解密 JWT 并获取其中的用户会话信息。 8. 整个方案中，access token 是一种 by reference token，不包含用户信息可以直接暴露在公网上；JWT token 是一种 by value token，可以包含用户信息但不暴露在公网上。 ## 十、服务部署平台选型 容器已经被社区接受为交付微服务的一种理想手段，可以实现不可变（immutable）发布模式。一个轻量级的基于容器的服务部署平台主要包括容器资源调度，发布系统，镜像治理，资源治理和 IAM 等模块。 集群资源调度系统：屏蔽容器细节，将整个集群抽象成容器资源池，支持按需申请和释放容器资源，物理机发生故障时能够实现自动故障迁移 (fail over)。目前 Google 开源的 Kubernetes[附录 12.41]，在 Google 背书和社区的强力推动下，基本已经形成市场领导者地位，GitHub 上有 31.8k 星，社区的活跃度已经远远超过了 mesos[附录 12.42]（GitHub 3.5k stars）和 swarm 等竞争产品，所以容器资源调度建议首选 K8s。当然如果你的团队有足够定制自研能力，想深度把控底层调度算法，也可以基于 Mesos 做定制自研。 镜像治理：基于 Docker Registry，封装一些轻量级的治理功能。VMware 开源的 harbor[附录 12.43] (GitHub 3.5k stars) 是目前社区比较成熟的企业级产品，在 Docker Registry 基础上扩展了权限控制，审计，镜像同步，管理界面等治理能力，可以考虑采用。 资源治理：类似于 CMDB 思路，在容器云环境中，企业仍然需要对应用 app，组织 org，容器配额和数量等相关信息进行轻量级的治理。目前这块还没有生产级的开源产品，一般企业需要根据自己的场景定制自研。 发布平台：面向用户的发布管理控制台，支持发布流程编排。它和其它子系统对接交互，实现基本的应用发布能力，也实现如蓝绿，金丝雀和灰度等高级发布机制。目前这块生产级的开源产品很少，Netflix 开源的 spinnaker[附录 12.44]（github 4.2k stars）是一个，但是这个产品比较复杂重量（因为它既要支持适配对接各种 CI 系统，同时还要适配对接各种公有云和容器云，使得整个系统异常复杂），一般企业建议根据自己的场景定制自研轻量级的解决方案。 IAM：是 identity & access management 的简称，对发布平台各个组件进行身份认证和安全访问控制。社区有不少开源的 IAM 产品，比较知名的有 Apereo CAS（GitHub 3.6k stars），JBoss 开源的 keycloak（GitHub 1.9 stars）等。但是这些产品一般都比较复杂重量，很多企业考虑到内部各种系统灵活对接的需求，都会考虑定制自研轻量级的解决方案。 考虑到服务部署平台目前还没有端到端生产级解决方案，企业一般需要定制集成，下面给出一个可以参考的具备轻量级治理能力的发布体系： ![框架](./../img/exp5.png "框架") 简化发布流程如下： 1. 应用通过 CI 集成后生成镜像，用户将镜像推到镜像治理中心； 2. 用户在资产治理中心申请发布，填报应用，发布和配额相关信息，然后等待审批通过； 3. 发布审批通过，开发人员通过发布控制台发布应用； 4. 发布系统通过查询资产治理中心获取发布规格信息； 5. 发布系统向容器云发出启动容器实例指令； 6. 容器云从镜像治理中心拉取镜像并启动容器； 7. 容器内服务启动后自注册到服务注册中心，并保持定期心跳； 8. 用户通过发布系统调用服务注册中心调拨流量，实现蓝绿，金丝雀或灰度发布等机制； 9. 网关和内部微服务客户端定期同步服务注册中心上的服务路由表，将流量按负载均衡策略分发到新的服务实例上。 另外，持续交付流水线（CD Pipeline）也是微服务发布重要环节，这块主要和研发流程相关，一般需要企业定制，下面是一个可供参考的流水线模型，在镜像治理中心上封装一些轻量级的治理流程，例如只有通过测试环境测试的镜像才能升级发布到 UAT 环境，只有通过 UAT 环境测试的镜像才能升级发布到生产环境，通过在流水线上设置一些质量门，保障应用高质量交付到生产。 ![框架](./../img/exp6.png "框架") ## 十一、写在最后 注意，本文限于篇幅，对测试和 CI 等环节没有涉及，但它们同样是构建微服务架构的重要环节，也有众多成熟的开源产品可选。 技术选型虽然重要，但还只是微服务建设的一小部分工作，选型后的产品要在企业内部真正落地，形成完整的微服务技术栈体系，则后续还有大量集成、定制、治理、运维和推广等工作。 本文仅限个人经验视角，选型思路仅供参考借鉴。每个企业的具体上下文（业务场景，团队组织，技术架构等）各不相同，每个架构师的背景经验也各不相同，大家要结合实际自己做出选型，没有最好的技术栈，只有相对较合适的技术栈。另外，好的技术选型是相互借鉴甚至 PK 出来的，欢迎大家讨论，给出自己的微服务 2.0 技术栈选型意见。 ## 十二、附录链接 1. Spring Boot https://github.com/spring-projects/spring-boot 2. Alibaba Dubbo https://github.com/alibaba/dubbo 3. Google gRPC https://github.com/grpc/grpc 4. NetflixOSS Eureka https://github.com/Netflix/eureka 5. Hashicorp Consul https://github.com/hashicorp/consul 6. NetflixOSS Zuul https://github.com/Netflix/zuul 7. Kong https://github.com/Kong/kong 8. Spring Cloud Config https://github.com/spring-cloud/spring-cloud-config 9. CTrip Apollo https://github.com/ctripcorp/apollo 10. ElasticSearch https://github.com/elastic/elasticsearch 11. Yelp Elastalert https://github.com/Yelp/elastalert 12. Dianping CAT https://github.com/dianping/cat 13. Zipkin https://github.com/openzipkin/zipkin 14. Naver Pinpoint https://github.com/naver/pinpoint 15. OpenTSDB https://github.com/OpenTSDB/opentsdb 16. KairosDB https://github.com/kairosdb/kairosdb 17. Argus https://github.com/salesforce/Argus 18. InfluxDB https://github.com/influxdata/influxdb 19. Prometheus https://github.com/prometheus/prometheus 20. Grafana https://github.com/grafana/grafana 21. Sensu https://github.com/sensu/sensu 22. Esty 411 https://github.com/etsy/411 23. Zalando ZMon https://github.com/zalando/zmon 24. NetflixOSS Hystrix https://github.com/Netflix/Hystrix 25. Nginx https://github.com/nginx/nginx 26. Apache Kafka https://github.com/apache/kafka 27. Allegro Hermes https://github.com/allegro/hermes 28. Apache Rocketmq https://github.com/apache/rocketmq 29. Rabbitmq https://github.com/rabbitmq/rabbitmq-server 30. Sohutv CacheCloud https://github.com/sohutv/cachecloud 31. Twitter twemproxy https://github.com/twitter/twemproxy 32. CodisLab codis https://github.com/CodisLabs/codis 33. Dangdang Sharding-jdbc https://github.com/shardingjdbc/sharding-jdbc 34. MyCAT https://github.com/MyCATApache/Mycat-Server 35. Xxl-job https://github.com/xuxueli/xxl-job 36. Dangdang elastic-job https://github.com/elasticjob/elastic-job-lite 37. Apereo CAS https://github.com/apereo/cas 38. JBoss keycloak https://github.com/keycloak/keycloak 39. Spring cloud security https://github.com/spring-cloud/spring-cloud-security 40. OpenID-Connect-Java-Spring-Server https://github.com/mitreid-connect/OpenID-Connect-Java-Spring-Server 41. Google Kubernetes https://github.com/kubernetes/kubernetes 42. Apache Mesos https://github.com/apache/mesos 43. Vmware Harbor https://github.com/vmware/harbor 44. Netflix Spinnaker https://github.com/spinnaker/spinnaker 45. Microservices in Practice – Key Architecture Concepts of an MSA https://wso2.com/whitepapers/microservices-in-practice-key-architectural-concepts-of-an-msa/ 46. Serf https://github.com/hashicorp/serf ================================================ FILE: docs/exp/pt.md ================================================ # 需求又变了，要不要怼回去？ 需求变更，让每一个技术人头疼的问题，应该以怎么样的态度来面对需求变更，是今天要讨论的话题。 为什么技术人讨厌需求变更？ 一个典型的互联网产品项目的流程是： 1. 调研，产品经理设计需求； 2. 产品经理和技术进行需求评审； 3. 技术进行方案设计； 4. 技术进行编码实现； 5. 技术进行功能联调； 6. 技术进行提测，开始进行测试； 7. 若干轮测试与BUG修复，达到准上线状态； 8. 发布沙箱环境，进行最后一轮沙箱测试，达到准发布状态； 9. 将产品系统发布到线上； 不管使用敏捷看板，还是传统瀑布式项目管理方法，就单个项目来看，这流程是串行的。 画外音：如果够敏捷，产品、测试、研发等角色是流水线的。 可以看到，产品方案是在项目流程的最上游，产品方案确定后，技术同学会按照2-3-4...-7-8-9等步骤将产品发布上线。 试想，现在项目流程进行到了第x个步骤，产品经理突然说，产品方案要变化，这意味着什么呢？ 这可能意味着一系列工作要推到重来： * （9）线上发布白干了，产品要回滚； * （8）沙箱发布白干了，沙箱要回滚； * （6-7）若干轮测试白干了； * （5）联调白干了； * （4）代码白写了； * （3）技术方案白设计了； * （2）需求要重新评审； * （3）技术方案重新设计； * （4）代码重新开发 * （5）系统重新联调； * （6-7）重新提测，迭代测试； * （8）重新发布沙箱； * （9）重新发布线上； 产品经理们，尝试着体会一下技术们的心里感受，就知道为什么技术这么痛恨“需求变更”了。 看到这里，一定有产品经理嘀咕“有这么夸张么，只变了10%的需求，需要全部重来么，技术同学也太矫情了”。有必要全部重来么？能不能省去联调、提测、测试的几个环节？ 很多人把产品设计比喻成建设高楼大厦，产品经理是大厦设计师，技术是工程师。大厦快建成了，设计师突然说，图纸要修改10%，工程师要不要重来？还是说省去几个环节？ 那么，需求变更了，技术同学要不要怼回去？ 在一家互联网公司，产品技术是很难分割的一个整体，他们虽有分工，各有侧重，但归根结底其最终目标是一致的：为达成公司业务目标共同努力。 为了这个共同的业务目标，产品经理的工作思路与逻辑应该是： “在资源有限的情况下，做什么产品、工具、后台、运营活动，对业务目标的达成最有帮助，就最先做什么”。 然而，有些产品经理的工作思路与逻辑却是： “想一出是一出，拍脑袋决策，凭直觉决策”： 登录这里改一下，能够提高转化率 详情页这里改一下，能够提高留存率 过节发个红包，能够拉新，能够提高活跃度 无穷多的需求上线后，无非是这几个结果： 产品成功了，哇，天才的构想 产品效果不好，嘘，技术们在忙着做后面的需求，也不会有人注意到 有人想起来要复盘，可以用“互联网产品，还不允许试错啦”搪塞过去 在“人人都是产品经理”的时代，有多少没有调研，没有分析，凭借直觉的产品经理，拍脑袋出来的需求，让一大帮技术专家为之操劳花好几月去实现。 画外音：这句话是对产品经理的侮辱，还是？产品经理的门槛很高，专业要求很高的。 不仅如此，项目半途，有多少需求是因为产品经理“之前没有想清楚”变更了需求，让“几百人日”的研发投入付之东流。 即使有些非常资深，非常厉害的产品经理，但人数一多，每个产品经理都想要业绩，在研发资源有限的情况下，为了争抢资源： 我这提了N个需求，总得排期一个吧 竞品有了这个功能，我们也必须有一个 老板说了，这个功能必须在节前上线 画外音： （1）既然负责这个产品模块，必须提对应的需求呀； （2）劣币驱逐良币； 一将无能，累死三军。 所以，这不是一个简单的“需求变更，要不要怼回去”的问题，甚至不是一个“需求提过来，要不要承接”的问题： 产品技术作为一个整体，共同为公司的业务目标服务 画外音：有些公司甚至有“技术不允许拒绝需求，不允许拒绝需求变更”的畸形文化。 产品作为技术侧的上游，需要系统性思考问题，而不是拍脑袋提需求 技术侧作为产品侧的下游，不能只是一味的接需求，要帮助他们想清楚，少为业务埋坑 技术侧能够如何帮助产品，让他们把需求提得更靠谱呢？ 至少，“过节发个红包”这类需求评审时，多问这么一些问题： 哪些产品指标，和业务的最终目标相关 画外音，假设有：新客，转化，留存，下单等。 这些产品指标中，哪些是主要矛盾 画外音：假设是新客。 对于这个产品指标，做哪些事情可以改善 画外音：假设有12345五件事。 对于这五件事，哪件事投入产出比最高 画外音：假设真的是“过节发个红包”这个活动。 对于这个活动，活动方案有几种优缺点是什么，为什么最终选择了这一种，活动逻辑是怎么样的，活动效果有没有预估，有没有埋点，活动结果怎么评估... 画外音：啥？活动只有一种方案？ 事先问的问题越多，讨论得越充分，需求就越靠谱，需求变更的几率就越小，埋坑的几率就越小。 # 总结： 产品技术作为一个整体，共同为公司的业务目标服务 产品经理要系统性思考问题 技术要帮助产品经理系统性思考问题 有技术的同学说，我天天在做需求，没时间想这些问题，没时间和产品经理讨论这些问题，怎么办？ 你MB，你活该天天加班。 ================================================ FILE: docs/exp/raft-gossip.md ================================================ # Raft算法和Gossip协议 简单介绍下集群数据同步，集群监控用到的两种常见算法。 ## Raft算法 raft 集群中的每个节点都可以根据集群运行的情况在三种状态间切换：follower, candidate 与 leader。leader 向 follower 同步日志，follower 只从 leader 处获取日志。在节点初始启动时，节点的 raft 状态机将处于 follower 状态并被设定一个 election timeout，如果在这一时间周期内没有收到来自 leader 的 heartbeat，节点将发起选举：节点在将自己的状态切换为 candidate 之后，向集群中其它 follower 节点发送请求，询问其是否选举自己成为 leader。当收到来自集群中过半数节点的接受投票后，节点即成为 leader，开始接收保存 client 的数据并向其它的 follower 节点同步日志。leader 节点依靠定时向 follower 发送 heartbeat 来保持其地位。任何时候如果其它 follower 在 election timeout 期间都没有收到来自 leader 的 heartbeat，同样会将自己的状态切换为 candidate 并发起选举。每成功选举一次，新 leader 的步进数都会比之前 leader 的步进数大1。 Raft一致性算法处理日志复制以保证强一致性。 ## follower 节点不可用 follower 节点不可用的情况相对容易解决。因为集群中的日志内容始终是从 leader 节点同步的，只要这一节点再次加入集群时重新从 leader 节点处复制日志即可。 ## leader 不可用 一般情况下，leader 节点定时发送 heartbeat 到 follower 节点。 ![rg](./../img/rg1.png) 由于某些异常导致 leader 不再发送 heartbeat ，或 follower 无法收到 heartbeat 。 ![rg](./../img/rg2.png) 当某一 follower 发生 election timeout 时，其状态变更为 candidate，并向其他 follower 发起投票。 ![rg](./../img/rg3.png) 当超过半数的 follower 接受投票后，这一节点将成为新的 leader，leader 的步进数加1并开始向 follower 同步日志。 ![rg](./../img/rg4.png) 当一段时间之后，如果之前的 leader 再次加入集群，则两个 leader 比较彼此的步进数，步进数低的 leader 将切换自己的状态为 follower。 ![rg](./../img/rg5.png) 较早前 leader 中不一致的日志将被清除，并与现有 leader 中的日志保持一致。 ![rg](./../img/rg6.png) ## Gossip协议 传统的监控，如ceilometer，由于每个节点都会向server报告状态，随着节点数量的增加server的压力随之增大。分布式健康检查可以解决这类性能瓶颈，降节点数量从数百台扩至数千台，甚至更多。 Agent在每台节点上运行，可以在每个Agent上添加一些健康检查的动作，Agent会周期性的运行这些动作。用户可以添加脚本或者请求一个URL链接。一旦有健康检查报告失败，Agent就把这个事件上报给服务器节点。用户可以在服务器节点上订阅健康检查事件，并处理这些报错消息。 在所有的Agent之间（包括服务器模式和普通模式）运行着Gossip协议。服务器节点和普通Agent都会加入这个Gossip集群，收发Gossip消息。每隔一段时间，每个节点都会随机选择几个节点发送Gossip消息，其他节点会再次随机选择其他几个节点接力发送消息。这样一段时间过后，整个集群都能收到这条消息。示意图如下。 ![rg](./../img/rg7.png) Gossip协议已经是P2P网络中比较成熟的协议了。Gossip协议的最大的好处是，即使集群节点的数量增加，每个节点的负载也不会增加很多，几乎是恒定的。这就允许Consul管理的集群规模能横向扩展到数千个节点。 Consul的每个Agent会利用Gossip协议互相检查在线状态，本质上是节点之间互Ping，分担了服务器节点的心跳压力。如果有节点掉线，不用服务器节点检查，其他普通节点会发现，然后用Gossip广播给整个集群。 Gossip算法又被称为反熵（Anti-Entropy），熵是物理学上的一个概念，代表杂乱无章，而反熵就是在杂乱无章中寻求一致，这充分说明了Gossip的特点：在一个有界网络中，每个节点都随机地与其他节点通信，经过一番杂乱无章的通信，最终所有节点的状态都会达成一致。每个节点可能知道所有其他节点，也可能仅知道几个邻居节点，只要这些节可以通过网络连通，最终他们的状态都是一致的，当然这也是疫情传播的特点。 要注意到的一点是，即使有的节点因宕机而重启，有新节点加入，但经过一段时间后，这些节点的状态也会与其他节点达成一致，也就是说，Gossip天然具有分布式容错的优点。 Gossip协议实现 [serf](https://github.com/hashicorp/serf) ================================================ FILE: docs/exp/techbig.md ================================================ # 如何成为技术大牛 不管是开发、测试、运维，每个技术人员心里多多少少都有一个成为技术大牛的 梦，毕竟“梦想总是要有的，万一实现了呢”！正是对技术梦的追求，促使我们不断 地努力和提升自己。 然而“梦想是美好的，现实却是残酷的”，很多同学在实际工作后就会发现，梦 想是成为大牛，但做的事情看起来跟大牛都不沾边，例如，程序员说“天天写业务代 码还加班，如何才能成为技术大牛”，测试说“每天都有执行不完的测试用例”，运维 说“扛机器接网线敲 shell 命令，这不是我想要的运维人生”。 我也是一位程序员，所以我希望通过以下基于程序开发的一些例子，帮助大家解 决这些困惑。大道理是相通的，测试、运维都可以借鉴。 ## 几个典型的误区 有人认为想成为技术大牛最简单直接、快速有效的方式是“拜团队技术大牛为 师”，让他们平时给你开小灶，给你分配一些有难度的任务。 我个人是反对这种方法的，主要的原因有几个： ### 拜大牛为师 * 大牛很忙，不太可能单独给你开小灶，更不可能每天都给你开1个小时的小 灶；而且一个团队里面，如果大牛平时经常给你开小灶，难免会引起其他团队 成员的疑惑， 我个人认为如果团队里的大牛如果真正有心的话，多给团队培训 是最好的。然而做过培训的都知道，准备一场培训是很耗费时间的，课件和材 料至少 2 个小时（还不能是碎片时间），讲解 1 个小时，大牛们一个月做一次 培训已经是很高频了。 * 因为第一个原因，所以一般要找大牛，都是带着问题去请教或者探讨。因为 回答或者探讨问题无需太多的时间，更多的是靠经验和积累，这种情况下大 牛们都是很乐意的，毕竟影响力是大牛的一个重要指标嘛。然而也要特别注 意：如果经常问那些书本或者 google 能够很容易查到的知识，大牛们也会很 不耐烦的，毕竟时间宝贵。经常有网友问我诸如“jvm 的 -Xmn 参数如何配 置”这类问题，我都是直接回答“请直接去 google”，因为这样的问题实在 是太多了，如果自己不去系统学习，每个都要问是非常浪费自己和别人的时 间的。 * 大牛不多，不太可能每个团队都有技术大牛，只能说团队里面会有比你水平高 的人，即使他每天给你开小灶，最终你也只能提升到他的水平；而如果是跨团 队的技术大牛，由于工作安排和分配的原因，直接请教和辅导的机会是比较少 的，单凭参加几次大牛的培训，是不太可能就成为技术大牛的。 综合上述的几个原因，我认为对于大部分人来说，要想成为技术大牛，首先还是 要明白“主要靠自己”这个道理，不要期望有个像武功师傅一样的大牛手把手一步一 步地教你。适当的时候可以通过请教大牛或者和大牛探讨来提升自己，但大部分时间 还是自己系统性、有针对性的提升。 ### 业务代码一样很牛逼 有人认为写业务代码一样可以很牛逼，理由是业务代码一样可以有各种技巧，例 如可以使用封装和抽象使得业务代码更具可扩展性，可以通过和产品多交流以便更好 的理解和实现业务，日志记录好了问题定位效率可以提升 10 倍等等。 业务代码一样有技术含量，这点是肯定的，业务代码中的技术是每个程序员的基 础，但只是掌握了这些技巧，并不能成为技术大牛，就像游戏中升级打怪一样，开始 打小怪，经验值很高，越到后面经验值越少，打小怪已经不能提升经验值了，这个时 候就需要打一些更高级的怪，刷一些有挑战的副本了，没看到哪个游戏只要一直打小 怪就能升到顶级的。成为技术大牛的路也是类似的，你要不断的提升自己的水平，然 后面临更大的挑战，通过应对这些挑战从而使自己水平更上一级，然后如此往复，最 终达到技术大牛甚至业界大牛的境界，写业务代码只是这个打怪升级路上的一个挑战 而已，而且我认为是比较初级的一个挑战。 所以我认为：业务代码都写不好的程序员肯定无法成为技术大牛，但只把业务代 码写好的程序员也还不能成为技术大牛。 ### 上班太忙没时间自己学习 很多人认为自己没有成为技术大牛并不是自己不聪明，也不是自己不努力， 而是中国的这个环境下，技术人员加班都太多了，导致自己没有额外的时间进行 学习。 这个理由有一定的客观性，毕竟和欧美相比，我们的加班确实要多一些，但这个 因素只是一个需要克服的问题，并不是不可逾越的鸿沟，毕竟我们身边还是有那么多 的大牛也是在中国这个环境成长起来的。 我认为有几个误区导致了这种看法的形成： 1. 上班做的都是重复工作，要想提升必须自己额外去学习 形成这个误区的主要原因还是在于认为“写业务代码是没有技术含量的”，而我 现在上班就是写业务代码，所以我在工作中不能提升。 2. 学习需要大段的连续时间 很多人以为要学习就要像学校上课一样，给你一整天时间来上课才算学习，而我 们平时加班又比较多，周末累的只想睡懒觉，或者只想去看看电影打打游戏来放松， 所以就没有时间学习了。 实际上的做法正好相反：首先我们应该在工作中学习和提升，因为学以致用或者 有实例参考，学习的效果是最好的；其次工作后学习不需要大段时间，而是要挤出时 间，利用时间碎片来学习。 ### 正确的做法 ### Do more(做更多) ***做的更多，做的比你主管安排给你的任务更多。** 我在HW的时候，负责一个版本的开发，这个版本的工作量大约是2000行左 右，但是我除了做完这个功能，还将关联的功能全部掌握清楚了，代码（大约 10000 行）也全部看了一遍，做完这个版本后，我对这个版本相关的整套业务全部很熟悉了。 经过一两次会议后，大家发现我对这块掌握最熟了，接下来就有趣了：产品讨论需求 找我、测试有问题也找我、老大对外支撑也找我；后来，不是我负责的功能他们也找 我，即使我当时不知道，我也会看代码或者找文档帮他们回答。最后我就成了我这个 系统的“专家”了。虽然这个时候我还是做业务的，还是写业务代码，但是我已经对 整个业务都很熟悉了。 以上只是一个简单的例子，其实就是想说：**要想有机会，首先你得从人群中冒出 来，要想冒出来，你就必须做到与众不同，要做到与众不同，你就要做得更多！** 怎么做得更多呢？可以从以下几个方面着手： 1. 熟悉更多业务，不管是不是你负责的；熟悉更多代码，不管是不是你写的 这样做有很多好处，举几个简单的例子： * 需求分析的时候更加准确，能够在需求阶段就识别风险、影响、难点 * 问题处理的时候更加快速，因为相关的业务和代码都熟悉，能够快速的判断问 题可能的原因并进行排查处理 * 方案设计的时候考虑更加周全，由于有对全局业务的理解，能够设计出更好 的方案 2. 熟悉端到端 比如说你负责 web 后台开发，但实际上用户发起一个 http 请求，要经过很多中 间步骤才到你的服务器（例如浏览器缓存、DNS、nginx 等），服务器一般又会 经过很多处理才到你写的那部分代码（路由、权限等）这整个流程中的很多系统或者 步骤，绝大部分人是不可能去参与写代码的，但掌握了这些知识对你的综合水平有很 大作用，例如方案设计、线上故障处理这些更加有含金量的技术工作都需要综合技术 水平。 “系统性”、“全局性”、“综合性”这些字眼看起来比较虚，但其实都是技术大牛 的必备的素质，要达到这样的境界，必须去熟悉更多系统、业务、代码。 3. 自学 一般在比较成熟的团队，由于框架或者组件已经进行了大量的封装，写业务代码 所用到的技术确实也比较少，但我们要明白“唯一不变的只有变化”，框架有可能要 改进，组件可能要替换，或者你换了一家公司，新公司既没有组件也没有框架，要你 从头开始来做。这些都是机会，也是挑战，而机会和挑战只会分配给有准备的人， 所 以这种情况下我们更加需要自学更多东西，因为真正等到要用的时候再来学已经没有 时间了。 以java为例，大部分业务代码就是if-else加个数据库操作，但我们完全可以 自己学些更多 java 的知识，例如垃圾回收，调优，网络编程等，这些可能暂时没用， 但真要用的时候，不是 google 一下就可以了，这个时候谁已经掌握了相关知识和技 能，机会就是谁的。 以垃圾回收为例，我自己平时就抽时间学习了这些知识，学了 1 年都没用上，但 后来用上了几次，每次都解决了卡死的大问题，而有的同学，写了几年的 java 代码， 对于 stop-the-world 是什么概念都不知道，更不用说去优化了。 ### Do better(做的更好) 要知道这个世界上没有完美的东西，你负责的系统和业务，总有不合理和可以改 进的地方，这些“不合理”和“可改进”的地方，都是更高级别的怪物，打完后能够 增加更多的经验值。识别出这些地方，并且给出解决方案，然后向主管提出，一次不 行两次，多提几次，只要有一次落地了，这就是你的机会。 例如： * 重复代码太多，是否可以引入设计模式？ * 系统性能一般，可否进行优化？ * 目前是单机，如果做成双机是否更好？ * 版本开发质量不高，是否引入高效的单元测试和集成测试方案？ * 目前的系统太庞大，是否可以通过重构和解耦改为 3 个系统？ * 中间件有一些系统感觉我们也可以用，是否可以引入 ？ 只要你去想，其实总能发现可以改进的地方的；如果你觉得系统哪里都没有改进 的地方，那就说明你的水平还不够，可以多学习相关技术，多看看业界其它优秀公司 怎么做。 我 2015 年调配到武汉起源技术有限公司，刚开始接手了一个简单的后台系统，每天就是配合前台 做数据增删改查，看起来完全没意思，是吧？如果只做这些确实没意思，但我们接手 后做了很多事情： 1. 解耦，将一个后台拆分为 2 个后台，提升可扩展性和稳定性； 2. 双机，将单机改为双机系统，提高可靠性； 3. 优化，将原来一个耗时 5 小时的接口优化为耗时 5 分钟 还有其它很多优化，后来我们这个组承担了更多的系统，后来这个小组 5 个人， 负责了 6 个系统。 ### Do exercise(多练习) 在做职业等级沟通的时候，发现有很多同学确实也在尝试 Do more、Do better， 但在执行的过程中，几乎每个人都遇到同一个问题：**光看不用效果很差，怎么办？** 例如： 1. 学习了 jvm 的垃圾回收，但是线上比较少出现 FGC 导致的卡顿问题，就算出 现了，恢复业务也是第一位的，不太可能线上出现问题然后让每个同学都去练 一下手，那怎么去实践这些 jvm 的知识和技能呢？ 2. Netty我也看了，也了解了Reactor的原理，但是我不可能参与Netty开发， 怎么去让自己真正掌握 Reactor 异步模式呢？ 3. 看了《高性能 MySQL》，但是线上的数据库都是 DBA 管理的，测试环境的数 据库感觉又是随便配置的，我怎么去验证这些技术呢？ 4. 框架封装了 DAL 层，数据库的访问我们都不需要操心，我们怎么去了解分库 分表实现？ 诸如此类问题还有很多，我这里分享一下个人的经验，其实就是3个词： learning、trying、teaching ！ 1. Learning 这个是第一阶段，看书、google、看视频、看别人的博客都可以，但要注意一 点是“系统化”，特别是一些基础性的东西，例如 JVM 原理、Java 编程、网络编程， HTTP 协议等等，这些基础技术不能只通过 google 或者博客学习，我的做法一般是 先完整的看完一本书全面的了解，然后再通过 google、视频、博客去有针对性的查 找一些有疑问的地方，或者一些技巧。 2. Trying 这个步骤就是解答前面提到的很多同学的疑惑的关键点，形象来说就是“自己动 手丰衣足食”，也就是自己去尝试搭建一些模拟环境，自己写一些测试程序。例如： * Jvm 垃圾回收：可以自己写一个简单的测试程序，分配内存不释放，然后调整 各种jvm启动参数，再运行的过程中使用jstack、jstat等命令查看jvm的堆内存分布和垃圾回收情况。这样的程序写起来很简单，简单一点的就几行，复 杂一点的也就几十行。 * Reactor原理：自己真正去尝试写一个Reactor模式的Demo，不要以为这 个很难，最简单的 Reactor 模式代码量（包括注释）不超过 200 行（可以参考 [java-design-patterns](https://github.com/iluwatar/java-design-patterns) 的 Reactor）。自己写完后，再去看看 别人怎么做，一对比理解就更 加深刻了。 * MySQL：既然有线上的配置可以参考，那可以直接让 DBA 将线上配置发给我 们（注意去掉敏感信息），直接学习；然后自己搭建一个 MySQL 环境，用线上 的配置启动；要知道很多同学用了很多年 MySQL，但是连个简单的 MySQL 环境都搭不起来。 * 框架封装了DAL层：可以自己用JDBC尝试去写一个分库分表的简单实现， 然后与框架的实现进行对比，看看差异在哪里。 * 用浏览器的工具查看HTTP缓存实现，看看不同种类的网站，不同类型的资 源，具体是如何控制缓存的；也可以自己用 Python 写一个简单的 HTTP 服务 器，模拟返回各种 HTTP Headers 来观察浏览器的反应。 还有很多方法，这里就不一一列举，简单来说，就是要将学到的东西真正试 试，才能理解更加深刻，印第安人有一句谚语：I hear and I forget. I see and I remember. I do and I understand，而且“试试”其实可以比较简单，很多时候 我们都可以自己动手做。 当然，如果能够在实际工作中使用，效果会更好，毕竟实际的线上环境和业务复 杂度不是我们写个模拟程序就能够模拟的，但这样的机会可遇不可求，大部分情况我 们还真的只能靠自己模拟，然后等到真正业务要用的时候，能够信手拈来。 3. Teaching 一般来说，经过 Learning 和 Trying，能掌握 70% 左右，但要真正掌握，我觉 得一定要做到能够跟别人讲清楚。因为在讲的时候，我们既需要将一个知识点系统 化，也需要考虑各种细节，这会促使我们进一步思考和学习。同时，讲出来后看或者听 的人可以有不同的理解，或者有新的补充，这相当于继续完善了整个知识技能体系。 这样的例子很多，包括我自己写博客的时候经常遇到，本来我觉得自己已经掌 握很全面了，但一写就发现很多点没考虑到；组内培训的时候也经常看到，有的同学 写了 PPT，但是讲的时候，大家一问，或者一讨论，就会发现很多点还没有讲清楚， 或者有的点其实是理解错了。写 PPT、讲 PPT、讨论 PPT，这个流程全部走一遍， 基本上对一个知识点掌握就比较全面了。 ### 后记 成为技术大牛梦想虽然很美好，但是要付出很多，不管是Do more还是Do better 还是 Do exercise，都需要花费时间和精力，这个过程中可能很苦逼，也可能 很枯燥，这里我想特别强调一下：前面我讲的都是一些方法论的东西，但真正起决定 作用的，其实还是我们对技术的热情和兴趣！ ================================================ FILE: docs/exp/tl.md ================================================ # 在阿里做了5年技术Leader，我总结出这些套路！ 本文的作者是阿里的技术Leader——**云狄**，他将从管理的角度分享技术 TL 的核心职责，主要分为如下几个方面与大家共同探讨、交流： * 团队建设 * 团队管理 * 团队文化 * 沟通与辅导 * 招聘与解雇 互联网公司的技术团队管理通常分为两个方向：技术管理和团队管理，互联网公司的技术 TL 与传统软件公司的 PM 还是有很大的区别。 传统软件公司的 PM 更多注重于对项目的管理，包括项目任务拆解、项目进度以及风险等。 对于多数互联网公司而言，技术 TL 更多的职责不再局限于项目角度，而是对业务与技术都要有深入的了解，就像黑夜里的灯塔，能够引导和修正团队成员前进的航向。 综合技术和业务角度去深度思考问题，具备一定的前瞻性，并在技术领域投入持续的学习热情，向团队成员传道，补齐短板，提高整个团队的战斗力。 技术 TL 职责不仅需要制定日常规范，包括开发规范、流程规范等，推动规范的落地，以公有的强制约定来避免不必要的内耗。 另外一多半的时间可能花在了开发任务分解分配、开发实践、技术架构评审、代码审核和风险识别上，剩余的时间则花在为了保障系统按时交付所需的各种计划、协作、沟通、管理上。 管理大师彼得·德鲁克说：“组织的目的，就是让平凡的人做出不平凡的事。” 然而，不是任何一群平凡的人聚集到一起，都能做出不平凡的事。甚至一群优秀的人聚集到一起，也可能只是一个平庸的组织。 大到一个国家，小到一个团队，任何一个卓越的组织，都必须有一个卓越的领导者。领导者是一个组织的灵魂，领导者在很大程度上决定了组织所能达到的高度。 阿里有句土话“平凡人、非凡事”，技术团队同样如此，管理者的战略眼光、管理方法、人格魅力等，都会给团队的工作结果带来决定性的影响。 其实每个公司、每个团队的背景不太一样，从管理学的角度探讨一些问题，没有统一标准的答案，本文中一些观点仅是个人观点，更多是我个人成长为技术 TL 的一些观点理念。 同时我也是吸取了前辈们一些优秀的管理理念，包括我最为尊敬的通用电气 CEO 杰克·韦尔奇、苹果 CEO 乔布斯、Intel CEO 格鲁夫，国内我最推崇的技术管理者 Robbin（丁香园的技术副总裁）。 ## 团队建设 从 2014 年开始带这块业务技术团队，至今有 5 个年头。回想起来，团队管理中所有能遇上的问题都遇到过了。 其中的磕磕绊绊数不胜数，完全是在实践当中吸取教训，团队建设这块在这里和大家简单分享一下，当然这里面也有做得不够好的地方。 在阿里每个人都能感受到拥抱变化，基本上每年组织架构都会调整，甚至有些团队每半年都会调整一次。 2014 年我也算是被分配到这个团队负责这块业务，这块业务是集团收购一家子公司的业务，整个团队文化和技术体系与阿里有很大的差异化。 一般来说新官上任三把火，新的技术 TL 空降之后往往会大肆招人，快速推进改革，而且有些技术 TL 喜欢把原来的一些旧将搬进来。 当时我没有急于这么去做，没有招过一个新员工，而是立足于稳定现有的团队，主要基于以下原因： * **团队和业务了解不够深**：对于目前团队的人员以及业务，我不够了解，不清楚这里面有哪些坑和陷阱，一旦初战不利，领导的信任度被透支，在公司恐怕难有立足之地，更不用谈论改造团队，发挥自己的才能了。 * **流程与制度**：针对团队现状存在的一些问题，我初步判断并不是人的问题，很多问题是一些组织、流程、制度上的问题。 我认为只有好的制度才能造就好的团队，在没有解决现有团队的痼疾之前招聘新人，不但不会带来新的生产力，反而会造成团队的混乱，应该先打下一个好的根基，再招人，才能事半功倍。 * **团队安全感**：不想让团队现有的成员感觉一朝天子一朝臣，担心自己在团队中会被边缘化，成为弃儿。另外一方面能够让现有团队心理比较安全，可以安心地好好工作，不至于发生更多的动荡。 经过了几个月的摸底了解，大概清楚当时团队存在的一些问题和原因： * **业务配合不规范**：产品、运营、研发部门之间配合没有建立合理的工作流程，比如对于产品需求的 PRD 评审没有标准，对于运营需求没有量化指标，大家都是疲于奔命做需求，导致大家的积极性不够高。 * **跨团队协作混乱**：跨部门之间的工作配合毫无规范可言，部门之间相互推诿，随便什么业务人员都会随时给研发人员下命令，长此以往，伤害了研发团队的积极性。 针对以上问题，我主要把协作流程规范梳理了一番，制定了相对合理、规范的产品合作流程，同产品同学约法三章，明确了 PRD 输出的标准和规范，运营的业务需求也统一由产品输出，杜绝一句话需求。 同产品、前端、UED、QA 团队的协作统一标准流程，下游对上游依赖方输出的工作必须有明确的标准规范，口头说的统统无效，拒绝合作。 针对跨团队协作乱的情况，我特别想说明一下，由于研发部门不是直接创造收入的业务部门，而是承担业务部门的服务者角色。 作为一个服务者，往往站在一个被动和弱势的位置上，很容易被业务人员举着收入的大棒指挥你无条件的服从。 业务部门人员随便指派任务，随意变更需求，团队同学无所适从。这样一来，部门内部无论怎样合理的计划都会被外部的力量轻易打破，让团队同学无所适从，导致大家的工作积极性不高，喜欢互相推卸责任。 久而久之，员工就产生了自我保护意识，凡工作尽量往后退，凡责任尽量往别处推，不求有功但求无过。 为打破员工养成的这种自我封闭的保护意识，鼓励员工更加积极主动做事情，我能够做的就是把这些责任都扛在自己身上，亲自去协调每项工作。 让团队成员没有后顾之忧，让团队同学相信我可以搞定他们担心的事情，出了任何问题我可以来背锅，给自己的团队创造一个相对宽松和自由的工作空间，保护团队不被外部的各种杂事伤害到。 ## 团队管理 人往往会高估自己而低估别人，很多管理者都会觉得手下交上来的工作做得不够完美，这里考虑不周那里做的啰嗦，但很多时候你只是看到了他人不擅长的地方，或者只是对方和你的出发点不同给出了不同的解决方案而已。 很多时候，我们并不如自己想象的那么强。管理者在充分理解一些管理的理念之后，不断地在实际的管理工作中去实践并收集反馈和迭代，这样才能够形成自己的管理风格，并找到最适合当前团队的管理方法。 作为一个团队的管理者，通常会有两种风格管理策略，简要概括为： * 集权式的管理风格 * 放权式的管理风格 **集权式管理**：管理者的风格是偏细节的，定义清晰的工作目标，并且把工作目标分解得非常细致，让手下的团队能按照整个计划步步为营往前推进，这是一种风格，相对来讲比较集权。 可以说我带这个团队的第一年是这种风格，我甚至会参加每一次需求评审，无论需求大小，会和研发同学一起去写代码。 对研发团队我会做详细的 Code Review，亲自带领研发团队做技术交流和分享，参与技术讨论确认架构方案，这样以来和大家建立起了充分的信任。 **放权式管理**：定义大的目标，把握大的方向，做关键性的决策。但是并不深入每个细节去管控手下团队的执行细节，以结果为导向。 我到这个团队一年后，业务流程已经清晰的建立起来了，骨干员工在业务上能够完全领会并且达到我的要求，这个时候放权可以充分调动团队的自主性和创造性，多数技术人员他们喜欢被领导，不喜欢被管理。 以上这两类管理风格没有对错之分，究竟哪种方式更适合完全取决于团队的状况。 其实这里我更想说一下关于放权式的管理风格，对于一个制度刚刚建立，流程还没有跑顺畅，团队残缺，骨干员工业务能力不及格的团队，采用放权式管理是错误的。 你必须事无巨细，从第一线的业务细节抓起，手把手的带员工，教会他们怎么正确的做事情，怎样达到你的要求，手把手的培养业务骨干，搭建团队核心架构。 这些年我看到过太多的案例，管理层自己从不真正深入业务，也缺乏对业务的深刻理解，没有找到问题的本质原因。 总是寄希望于招人来解决问题，结果换了一茬又一茬人，问题永远解决不了，而且从来不深刻反思自己是否亲自尝试解决业务问题。很多时候架构反映出来的问题，其实是组织、流程的问题。 总之，作为管理层，如果自己没有深入一线去发现问题，自己动手去解决问题的决心和勇气的话，那这个团队很难有新的突破和成功。 ## 团队文化 在我刚参加工作的前几年，就听过一些关于团队文化和企业文化的一些概念，并没有特别深刻的印象。 尤其我读了《基业长青》这本书后，让我感受到对于一个企业而言，决定短期的是技巧，决定中期的是战略，决定长期的是文化。 企业文化对一家公司来说真的很重要，同样团队文化对于一个团队来说也很重要，我在带团队之初也曾经忽视了团队文化的冲击。 在带领这个团队之初，我私下找一些团队同学做 1on1 沟通，我发现这里面的问题还是比较严重的。 很多人为了避免故障遭受惩罚，不敢去重构优化代码，把自己封闭到一个很小的圈子，也没有过多的追求和理想，以前也没有末位淘汰机制，大家觉得可以继续吃大锅饭。 当时部门都是工作多年的老人，老的风气和习惯已经形成了很顽固的不良文化，工作情绪受到很大的影响。 老的不良的文化包括： * 做事情没有积极性。 * 永远不承认自己的错误，永远找借口推卸责任，永远都是别人的问题。 * 不求有功但求无过；责任心差，对待工作自我要求低。 * 对工作安排喜欢讨价还价。 在一个不好的文化氛围下，优秀的员工会被排挤，团队没有向心力，也很难留住好的人才，员工流失率会非常高。 我认为衡量一个团队文化氛围是否有吸引力，有一个很重要的指标，是新员工的流失率： 如果一个团队氛围非常好，新员工入职以后往往能够快速融入进来，流失率很低。 如果团队氛围差，新员工入职以后比较茫然难以融入，往往会很快离职，流失率非常高，实际上留不住新员工远远比留不住老员工更可怕。 接下来我希望给团队树立的文化是： * 坦诚，公开，透明。 * 平等相处，消除等级感。 * 工作气氛轻松，团队关系和谐。 * 敢于担当，主动承担责任。 * 成就他人，乐于分享。 关于团队文化这个话题其实很泛，可以单独写一篇文章出来的。这里我主要基于团队文化以上几点，谈一下我的一些个人的看法。 ### 坦诚的力量 首先，我觉得坦诚无论对于一个 TL 还是团队成员来说，坦诚也是一种价值观，对于一个团队的发展来说是非常重要的。 作为一个 TL，带领一支团队，我觉得最重要的是 TL 本人必须做到坦诚的态度，只有对团队坦诚，才能和团队之间形成信任，只有和团队形成了信任，才能成为一支默契的团队。 通用电气 CEO 杰克·韦尔奇说过：什么是信任？当一个领导真诚、坦率、言出必行的时候，信任就出现了，事情就是这么简单。为什么坦诚精神能行得通？很简单，因为坦诚有化繁为简的力量！ 坦诚的性格是管理者最基本的要求，只有管理者坦诚，才能获得团队的信任，作秀式的演讲和奖励并不能够真正获得团队的心，还是需要在工作中脚踏实地一点一滴去做好最平凡普通的事情。 坦诚能够让你直面自身的缺陷，有针对性地改变自己，解决团队的问题，造就一个互相信任的团队氛围。 我见过一个比较典型的案例，日常工作中主管对于下属不够坦诚，下属与主管的平时一些工作沟通中，下属做的不够好的地方，主管不及时进行沟通与辅导，结果最后 KPI 考核被打了低绩效。 换位思考一下，这个被打低绩效的人是我，我也会不服气，有问题你为啥不提前告诉我，让我提前去改正。 对待下属要有勇气，敢于指出他们的问题，对于表现不好的员工要敢于批评和管理，例如为什么解雇你。 这些谈话和冲突往往让人感到不舒服，我也承认每次谈低绩效是硬着头皮的，但是你必须有这样的勇气，坦诚不仅仅要对那些表现良好的人，还要对那些表现糟糕的人。 苹果创始人乔布斯是一个对自己、对别人坦诚得可怕的人，坦诚的残酷，直面事情最真实的一面。 的确坦诚的态度在很多时候会让别人感觉不舒服，乔布斯粗暴的坦诚态度也备受争议，但我觉得，如果你是一个结果导向的人，还是应该尽量坚持坦诚的态度，否则最终的结果可能远远偏离你的目标。 ### 允许你的下属 challenge 你 其次，我再聊一下关于平等相处，消除等级感，这点我觉得最重要的是让大家感受到你的亲和力，不是一个高高在上的领导。 比如很多时候团队一些技术方案的决策不是你一个人来决定，有时候还是要善于倾听一下团队成员的意见，要允许团队成员 challenge 你。 其实，国内外要求下属服从的企业文化很普遍，这不一定是坏事，特别是公司如果有想法的人太多，想法又无法统一起来，公司的整体战略呈现精神分裂状态，那基本上就离死不远了。 所以管理层统一公司战略，一线员工强调使命必达。 国内的外企格外强调下属的服从性，把这一点作为员工的基本职业素养来培训，常用来讲解的故事就是《把信送给加西亚》，强调上司安排一项工作以后，下属不允许谈任何条件，不允许 challenge 上司，必须无条件服从，克服一切困难也要完成工作任务，以解领导之忧。 这种执行力让上司感觉很舒服，而且公司管理实施难度也比较低。 多数管理者都喜欢比较听话的下属，认为顺从的下属更好用。心态上高人一等，不会放低心态倾听下属的意见，即使自己错了也不会承认错误，一方面害怕自己的权威被挑战，另外害怕向下属认错，觉得抹不开面子。 我不是圣人，作为 TL 曾经也犯过一些错误，我也曾私下里和个别同学道过歉。放开心态，不需要过多的太在意别人的看法，这些我觉得都是无所谓的小事。 从我个人自身的一些经历来看，其实一味地要求下属服从是有害的，要适当允许你的下属 challenge 你。 如果一味地要求下属服从，不能进行任何反驳，长时间下来会导致团队的人缺乏思考，只是一味的按照 TL 的想法去执行，当下属内心并不认可工作本身，仅仅出于职业性完成工作，成绩最多是合格，很难达到卓越。 同时会导致下属缺乏工作积极性主动性，容易养成下属逃避责任的习惯。 相反我觉得作为 TL 一定要鼓励下属积极主动地思考，让下属能够自己设定成长目标，对工作拥有归属感和责任感。 尽量给予下属更自由的空间，不要设置过多形式主义的约束；要允许下属去 challenge 你，参与你的决策，甚至质疑你的决策。 用这种方式增加下属对工作的归属感，工作责任心更强，更积极主动，能够自我驱动。 当你的决策错误的时候，下属可以帮你纠错，集体的智慧毕竟高于个人，俗话说“三个臭皮匠赛过诸葛亮”。 ### Owner 意识 “Owner 意识”主要体现在两个层面： * 认真负责的态度。认真负责是工作的底线。 * 积极主动的精神。积极主动是“Owner 意识”更高一级的要求。 自私确实是人的天性，不是自己的东西，很难谈什么责任感，更不用说主动性了。 因此，团队管理就是要努力地培养大家的责任感，主人翁意识，想做到这一点，就需要增强团队成员的参与感，让他们知晓并理解所做事情的价值、来龙去脉，不断地强化使命感。 例如可以将系统、业务范围等根据团队成员的兴趣点、以往项目经历等多种因素划分给指定人负责，并明确赏罚机制。 要清晰地传达一种思想，那就是：这块东西就是你的，干好了评优、升职、加薪等都会优先考虑；干不好，出事情了，你要负责，我也会负责。 如果有一天你看到团队成员像呵护自己的孩子一样，去对待自己的工作，那么你的目的已经达到了，他已经完全具备 Owner 意识了。 ### 建立学习型的组织 最后一点我要谈的是建立学习型的组织，团队成员要尽可能地分享自己的知识和想法，大家互相学习，也通过分享能够总结自己学习过程中零散的知识点。 如何建立人才梯队的，其实就是要建立学习型组织，让大家积极参与学习与分享。 具体做法 KPI 里设置一项技术分享与团队贡献，团队内部轮流进行技术分享，一方面让大家去学习、研究一些前沿技术。 尤其是团队可能会用到的一些技术储备，如果他真的能把这个技术给大家讲明白的话，那他就是真的掌握了，同时也让其他人开始了解并学习这项技术，同时还能够锻炼其演讲与口才。 鼓励团队成员敢于去分享，乐于去分享，开放心态成就他人。把技术培训和分享坚持下去，形成这样一种学习型的文化以后，你就会发现整个研发团队的技术能力的提升速度是非常惊人的，并且不会再占用太多额外的时间。 当你再招一个资历较浅的新员工时，他也能在这种环境中快速提升，通常半年左右时间就能达到非常好的水平。 当然，一开始的团队可能没有这样的意识，就需要你作为管理者强行去推动，把要求列入 KPI，很认真地考核他，慢慢地，团队就会形成这样的氛围和文化。 当然建立这种学习型的组织，也可以建立一些读书分享会，把读的一些书籍感受分享给大家，另外一点团队的 Wiki 知识库一定要建立起来，让团队同学把一些日常的技术方案、项目总结、故障总结通过文档的形式积累起来。 ## 沟通与辅导 根据美国普林斯顿大学的调查报告，在所有对工作产生影响的因素中，沟通占的比例高达 75%。而我们工作中出现的 80% 问题都是由沟通不当造成的，可见沟通的重要性。 多数时候，我们只想着表达自己的观点，只关注自己想说什么，我们会尽量使用漂亮的 PPT、华美的语言、一堆的数据、甚至引章据典，而不关心别人听懂没有，没有思考别人是否想听，别人是否听得懂。 沟通在我们的工作中无处不在，你会发现尤其在技术这个圈子里，能够进行高效沟通的人占比会更少一些。 沟通按照沟通对象类型通常分为向下沟通(同下属沟通)、横向沟通(跨团队沟通)、向上沟通(同老板沟通)，接下来只讨论如何同下属进行沟通，最为有效沟通方式：一对一沟通。 一对一沟通，又被称作一对一会议、One-on-one 等，是互联网公司常用的沟通方式。 一对一沟通虽然被广泛使用，但是涉及的文章却很少，这里我给大家推荐《格鲁夫给经理人的第一课》、《创业维艰 : 如何完成比难更难的事》，这两本书有更多关于一对一沟通的介绍。 格鲁夫是 Intel 公司的总裁，成功带领 Intel 公司完成了从半导体存储器到微处理器的转型，也是我非常欣赏的一位 CEO。 《创业维艰》的作者本·霍洛维茨是硅谷的顶级 VC，投资了 Facebook、Twitter 等公司。 在《格鲁夫给经理人的第一课》一书中，格鲁夫对「一对一沟通」的介绍如下： 在英特尔，一对一会议通常是由经理人召集他的部属召开的，这也是维系双方从属关系最主要的方法。一对一会议主要的目的在于互通信息以及彼此学习。经过对特定事项的讨论，上司可以将其技能以及经验传授给下属，并同时建议他切入问题的方式；而下属也能对工作中碰到的问题进行汇报。 在我看来，技术研发同学多数比较内向，不轻易向别人表达自己内心的一些想法。 一对一沟通的意义是可以使得信息从下而上地传递，同时可以把一些疑问、想法、意见、问题、规划等等和管理者做沟通，从而获得在其它渠道不易获得的信息，保证透明。 ### 1on1 沟通聊什么 在《创业维艰：如何完成比难更难的事》这本书中专门拿出了一节提到了一对一沟通（1on1），具体聊那些内容给了一些建议。 作为 TL 我通常会与团队的人聊以下话题： * 你有没有认为自己的价值和能力被低估了吗？为什么？ * 你觉得在工作中能学到东西吗？你最近学到了什么？你还希望在哪些领域进行学习？ * 近期这段时间，对自己有哪些满意、不满意的地方？ * 目前工作中，有哪些困惑？希望我如何去帮助你？ * 对团队和我的一些期待和建议。 * 在公司战略和目标方面，你最不清楚的是什么? 以上这些内容，除了在一对一沟通中交流之外，很难找到别的渠道来有效解决。 通过这些 1on1 的沟通，真的可以得到很多反馈信息，甚至得到的一些信息令我感到吃惊，原来还有这些细节问题我没有做好。 一对一沟通构造了一个渠道，这个渠道自下而上，使得以上这些内容都能够被倾听，从而被解决。 ### 1on1 沟通的一些注意点 ***找个私密的环境**:找个空会议室或者别人听不到谈话的角落，不要在工位或嘈杂的环境中进行，因为私密的环境才能降低沟通中某些话被他人听到的心理压力，才能更轻松和真实的表达自己。 ***最好提前告知 1on1 的团队成员**:一般需要提前 1 周把 1on1 沟通的话题、具体时间通知到团队成员，这样的好处是团队成员可以提前准备下聊的内容，因为临时性的沟通很容易出现因为人类记忆力的问题，导致一些想聊的问题在当时没想到。 ***定期进行**:在《创业维艰》一书中，本·霍洛维茨认为一对一沟通需要保证至少一个月一次。而格鲁夫认为，需要根据部属对工作的熟悉度，而进行不同程度的掌控。 另外，格鲁夫还认为，事情变化的速度也是影响一对一沟通频率的因素。作为技术研发部门，我通常会 1-2 月进行一次 1on1 沟通。 ***用心倾听并行动**:沟通要有效，用心倾听、保持真诚是必要的前提，否则员工不可能将心中的问题提出来。 保持真诚需要不敷衍任何团队同学提出的问题，不管这个问题有多尖锐。如果你也不知道如何解决这个问题，不妨和团队同学一起讨论讨论，看看大家能不能一起寻找可行的办法。 切忌不要讲空话和套话，一旦团队同学发现这是一个无效的沟通渠道之后，「自下而上」的通道就被关闭了。 ***适当引导**:并不是每一个员工都懂得一对一沟通的重要性，也不是每一个员工都能主动倾述问题，寻求帮助。很多程序员的性格都是比较内向的，有一些甚至不善于表达自己。 所以，虽然员工是一对一沟通的「主角」，但是上司也是需要进行适当的引导。 对于上司已经发现的员工工作中的困难，可以适当的主动提出来，以便于更好地讨论，这也会让员工感到很体贴。 ## 招聘与解雇 对于一个团队来说，人才是最核心、关键的。招聘和解雇尤其对于一个新上任的技术 TL，都是一个很大的挑战，接下来我们重点讨论这两个话题。 ### 招聘 招聘很多时候取决于公司在什么发展时期，需要招聘什么样的人。在初创时期，本不太可能招聘到清一色的专家人才，这个时候活下来比啥都重要，态度和味道是重点看的。 在高速发展之后，可能需要引进能带来新思路的一些人才，这取决于业务、技术、组织三者的对齐。那么这个时候，就是既要高技能，又要好的做事态度、习惯。 在搭建技术团队招聘前，要先明确所搭团队的类型，一般来说有三种不同类型的技术团队，即： * 项目驱动型 * 业务驱动型 * 技术驱动型 不同类型的技术团队在招聘时也有很大的不同，比如技术驱动型团队你可能需要一个在中间件、语言功底非常深厚、有大局观的人，业务驱动型的团队可能需要有业务 Sense，并且具备良好技术和业务架构能力的人。 在招聘这条路上，我也走过弯路，一开始我对候选人的背景、语言功底、架构能力以及运维、数据库方面比较关注，希望能招到全栈的技术人才，后来发现我忽视了一个很重要的点沟通与协作能力、态度。 后来导致新人来到团队后，虽然技术牛 B，但喜欢闭门造车，不喜欢和别人沟通，团队协作能力不够好，整体产出和效率不高。 所以在招聘新人的过程中，不能够只盯着候选人有什么经验，会什么框架等技术面，也需要着重考量他们的综合素质，一个领导力好的候选人，能够非常快速地融入团队，也能够非常快的学习一些知识。 招聘步骤： ***根据搭建团队的目标，做好招聘计划。**根据团队自身的定位，招聘合适的人才。 有几点需要 TL 特别关注的，作为 TL 要对候选人的成长负责，切忌因人设岗、因单独项目而招人。 比如前端团队招聘一些后端开发，工程团队招聘算法，这样以来可能会导致候选人进来后很难融入到团队，没有存在感，长时间下来会导致新人离职。 ***确定招聘需求（定岗定责）**：列出每个岗位的职责、需要具备的技能及其他要求。招聘需求归根结底是需要什么样的人，与整体业务和组织发展匹配。 ***合理利用人才招聘渠道。**从我自身的经历来看，人才招聘渠道多数通过互联网招聘渠道以及朋友推荐更可靠一些，对于高级别的人才可以采用猎头定向挖人。 **人才筛选：**作为技术面试官，对于人才的筛选也是非常重要关键的一个环节，要根据自己团队的目标来选取合适的人才，设定完成的时间期限，将面试的重点放在专业技能、管理能力、价值观（公司认同）等方面。 一般要求如下： ***和岗位需要的专业技能高度匹配：**专业技术技能面试过关，定岗定责。 ***沟通力强：**理解公司的业务，知晓管理层，了解公司的发展方向。 ***责任心：**凡事有交代，件件有着落，事事有回音。 ***靠谱并自带正能量：**不抱怨，主动解决问题，懂得纪律的重要性，一诺千金。 ***价值观认同：**认同公司，有目标有理想、有激情有冲劲。 ***背景调查：**非常有用的一个办法，可以大幅度降低选人风险，不用怕麻烦，这个工作的付出永远都是值得的。 另外，我想说的对于技术面试官需要有一定甄别人才的能力，同时有意识地提高这方面的能力。 我提供以下几点建议给技术面试官： * 如果对候选人有些犹豫和纠结，请你放弃这个候选人，你最担心的问题往往很大概率上会发生。 * 明确我们招聘的候选人标准，比如后端 Java 研发：Java 基础和分布式领域知识技能考察是必须的，少问记忆性问题和太理论性问题，更多地从候选人的一些实践经历中，提取出对这个候选人的更有价值的判断。 * 一面非常重要，要保证客观、公平，后面的交叉和终面往往参考前面的评价反馈，我们今天不仅是为我们的团队选拔人才，更是为公司选拔人才，还是要高标准的要求。 * 从心理学角度讲，必须要交叉面试，而且交叉面试官给出的反馈往往是比较客观、中肯的，而且要以交叉面试官的评价为主。 * 面试官切忌拿自己擅长的东西去考察候选人，需要认真的看候选人的简历，从候选人的经历中去考察这个人的综合能力。 一个团队的健康发展，最重要的是核心技术人，所以招聘工作必须谨慎，一旦有人加入就等于同上了一艘船，其中的纠结、痛苦、欢喜都要一起面对。 招募一个不合适人员的成本不仅仅是薪资那么简单。所以请一定要放过那些经验不错、资质不错但是很犹豫匹配度、落地融入堪忧的面试者，其结局大部分都是彼此痛苦。 作为技术 TL 最成功的是招到比你更优秀的人，你不需要担心自己会不会被取代： * 一是成就个人和成就团队，作为 TL 应该抱着如何成就团队的发展思路，不能让自己成为天花板，本身技术就不应该是你最擅长的事情！ * 二是兼容并蓄，发展多样性。刘邦善用汉初三杰，单项能力不如韩信、张良。TL 不要以自己的长短来衡量招聘的人，而是看团队技能视图的缺口和发展项。 ## 解雇 解雇员工通常更多的是针对触犯公司文化、原则红线，或者持续无法跟上公司节奏的员工进行的处理。 在阿里也有这么一句土话：“如果你没有开除或解雇过一位员工，算不上真正合格的管理者”，大多数技术管理者性格比较随和，不喜欢开除员工。 但是出现触犯红线的员工或者跟不上节奏的员工，尤其是不认可团队价值观的同学，会把一些负面情绪、行为影响到团队其他同学，因此需要杀伐决断，当机立断采用合适的方法让员工离开。 当然，如果只是能力跟不上的员工，你也可以推荐给其他公司适合的岗位，让和自己一起奋战过的兄弟有一个好归宿，也会让在职的员工感觉温暖。 整体上“慈不掌兵”，在开人这件事情上，高级管理者不要过于犹豫，为了一两个人最后影响整体团队的士气反而得不偿失。 多数互联网公司对于技术人员都有相应的 KPI 考核，对于达不到预期的人员会进行淘汰。 解雇尤其对于新上任的技术主管还是有一定挑战的，我相信人的本性还是善良的，作为技术 TL 不想让团队成员面对这一难题，包括我自己在内。 一家公司在成长，组织肯定要升级，人员的新老交替也是正常的。如果团队成员的表现达不到预期，不通过 KPI 考核机制告诉他，也许他不会意识到自身的一些问题，他永远不会成长起来，相对短期这些经济回报而言，个人的成长更为关键重要。 在阿里有这么一句土话：**“不经历 3.25 的人生，不是完美的阿里之旅”**，当你处于发展的低谷时，经历一次末位考核结果也许能够让他彻底清醒，认识到自己的不足，彻底激发自己的潜能，能够触底反弹。 心理学上有个著名的邓宁-克鲁格效应，又称达克效应。大意是，人很容易对自我产生认知偏差，最简单来说，就是会过于高估自己。 达克效应的曲线图: ![tl](./../img/tl.png) 上面的图片上反映出，大部分人其实都处于愚昧之巅。人能够成长为智者和大师要先从愚昧之巅，掉到绝望之谷，然后辛苦攀爬，积累知识和经验，成为智者和大师。 有担当的管理者的一个重要责任，就是把下属从愚昧之巅推向绝望之谷，至于能否爬上开悟之坡，看个人造化。 一个合格的技术 TL 必须要给团队成员塑造一个绝望山谷，同时还要让他看到一个开悟之坡，这样员工会不断突破自我。 作为一个有担当的管理者，我们不应该是一个老好人的角色，也要有冷酷无情的一面。 阿里也有一句土话：“心要仁慈，刀要快”，当团队中出现一些达不到团队要求的人，管理者应该主动去拉他一把，如果多次尝试，最终达不到预期，应该请他离开。 因为到了中途，再被残酷淘汰，无论对组织，还是对个人，损失都更大。 每一位开发者眼中，都有自己理想中的技术管理者。你认为优秀的技术 TL 身上有哪些特质？欢迎在留言区讨论，与大家分享交流。 ================================================ FILE: docs/framework/agility.md ================================================ # 如何理解敏捷开发 ## 为什么要敏捷开发 “没有人喜欢敏捷，但我们不得不敏捷。就像没有人喜欢工作，但你必须工作。”这是我经常用来调侃敏捷的一句话。试想一下，拿到一份完整详尽的需求文档，逐个功能Coding，测试部署上线。不需要再次确认需求，不会有人打断思路。没有需求更改，只要自己不犯错，不存在推倒重来这才是大部分开发人员最舒服的工作方式吧，简直太完美了。但它很像瀑布，一点都不敏捷。既然我们喜欢的工作方式是传统的、瀑布的，为什么要敏捷？这还不都是市场环境逼的。今天的市场向我们提出了三个问题： ## 01如何做出真正满足用户需求的产品？ 我不确定这世上是否有人可以做到，他所做出的全部决定都是对的。但绝大多数人是无法一次性做出真正满足用户需求的产品的。我们无法预测未来，我们必须通过一次次的测试，去寻找用户需要的产品是什么？想要更快地获得好的产品，必须迅速将产品推向市场，快速试验，避免走弯路。 ## 02如何满足不断变化的用户需求？ 今天所有的事情的都在快速变化，用户的需求也在变化。毫不夸张地说，我们正在全力开发的一些功能，当它们上线的时候，我们的用户已经不再需要了。我们没有办法让一切不变，我们只能选择去拥抱变化。 ## 03如何同时满足不同层次用户的需求？ 过去我们的产品会遵循产品生命周期，早期追求新奇的尝鲜者，中期普通大众，后期落后者。今天，产品的生命周期变化非常地快，我们可能会同时面对尝鲜者、普通大众、落后者，不同的用户类型的需求是不一样的，你如何满足他们？我们必须快速响应，没法快速响应，就没法留住用户，没有用户就没有一切。迅速将产品推向市场，拥抱变化、快速响应。今天的市场向所有的从业者提出了一个要求：拥有应对快速变化的需求的软件开发能力。而敏捷就是赋予团队应对快速变化的需求的软件开发能力的方法，而这就是敏捷流行的原因。 ## 什么是敏捷开发 敏捷绝非某一种特定的开发方法，它只是一种应对快速变化的需求的一种软件开发能力。敏捷本身只包含了《敏捷软件开发宣言》和《敏捷软件的十二条原则》两份文档。敏捷相信，只要符合这两份文档的开发方法，就能让开发团队拥有应对快速变化需求的能力，这样的开发方法都叫做敏捷开发方法。 ## 最流行的敏捷开发方式是什么 敏捷开发的方法有很多：ASD、AUP、DSDM、XP、FDD、Kanban、RAD、Scrum。目前国内最流行的应属Scrum。Scrum本意是指橄榄球运动中的“带球过人”，它可以解决非常复杂的项目，并且能高效并创造性地交付高价值的产品。Scrum包含3个角色、3个工件、5个价值观、5个事件 [详情https://www.scrum.org](https://www.scrum.org/)。相比其他敏捷方法，Scrum既不简单又不复杂。它有完善的指南，你可以按照指南，轻易在团队内推广尝试。但想要实践好Scrum，又需要很好的技巧。这种易上手难精通的方法，特别适合培训机构。所以近几年，关于Scrum的培训机构遍地都是。可以说，在Scrum流行这件事上，这些培训机构做了很大的贡献。 ## 如何实施Scrum 在团队中实施敏捷开发是一个很有挑战性的工作，尤其是Scrum这种描述比较详细、有很多规则的方法。 **如果你打算把所有人召集到一起，宣布我们要实施敏捷开发Scrum，然后就照着Scrum的描述开发方法直接实施，那么，你离失败不远了。**想要实施好Scrum，你必须一步步来，一点一点改变团队的习惯，当一项新制度已经变成本能之后，再推行下一项制度。 ### >>>>第一步，先把团队的开发任务可视化。 你可以用一块白板，让团队成员把所有的开发任务都用便签写好放上去，也可以找一个专业的看板工具。我推荐日事清，当所有任务都在看板上，尽收眼底，那种感觉还是很棒的。团队中每个人的工作都会被所有人看到，会形成一种无形的监督。 ### >>>>第二步，引入站会制度。 当团队所有人的任务都在看板后，这些任务的状态是需要有人更新的。无论是专人更新，还是每个人自己更新，都会出现忘记更新，更新不及时的情况。这时，就可以顺势引入站会制度了。每天上班的时候，或者每天下班的时候，所有人站到一起，花15分钟，说一下自己做了什么，准备做什么，所有人一起把看板更新了。 ### >>>>第三步，引入回顾会议 当站会制度成熟后，就可以挑一个版本发布的日子。等版本发布后，把团队成员叫到一起，开一个回顾会。让团队成员对这个版本开发的情况做个总结，同时可以让大家用便签把这个开发版本遇到的问题、不满写下来。针对这些问题不满，大家一起讨论一些改进方案。 ### >>>>第四步，引入计划会议 如果再开回顾会的时候，有人提出需求不清，拆解不够细致，工时估算不准之类的问题，那你真是太幸运了。碰到这种情况，你顺势提出这个制度：每个版本开发前，产品经理要和团队人员开个会，一起逐条对需求。产品经理要向团队解释每个需求，并且对团队提出的疑问进行解释和澄清。开发团队要将需求分解成任务，估算工时，最终形成一个开发清单。 ### >>>>第五步，引入评审会议 当计划会成功开过几次之后，就可以考虑引入评审会了。你可以在某个重大功能发布的时候，把所有项目相关的人员召集起来，向大家展示团队做了哪些了不起的功能，也可以和大家讨论讨论还可以做哪些优化。 ### >>>>第六步，宣布引入Scrum 当整个流程都跑顺以后，就可以在某次全体会议上，提出我们要实施Scrum了。你需要任命PO、SM，对团队培训Scrum知识，讲解Scrum的价值观。 ## 实施敏捷开发会面临怎样的挑战 1.团队成员对变化的恐惧 实施Scrum最大的挑战来自内部，来自团队对变化的恐惧。每个团队在运营一段时间后都会形成固有的默契和习惯，引入Scrum必然会打破这种习惯。这会让团队很不舒服，团队也会抵制，甚至是反抗。而且实施敏捷的过程，一定不是一帆风顺的，尤其是实施敏捷的早期阶段，不仅不会对团队产生太大的价值，反而会引起混乱。所以，在实施敏捷的过程中，千万要小心，每次只改变一点点，敏捷地完成敏捷。 2.组织人员的专业化水 平敏捷并不会直接提高团队的专业化水平，反而对团队专业化水平有一定的要求。必要的专业分工、合适的专业能力是必须的。敏捷会让开发整个流程持续运转，当某一个环节专业化能力出现问题的时候，会极大地导致整个流程运转停滞。平时团队中的问题会在敏捷中快速暴露。一旦出现专业水平问题，要及时解决，补充人员，或者帮助其提高能力。即使问题并不严重，也要在最短的时间解决，否则会影响团队对敏捷的信心。 3.产品经理这个组织Bug 国内在实施Scrum的时候经常会碰到的问题就是团队中会有产品经理这个角色。 **一般的处理方法是产品经理当PO**，但PO是需要领导开发团队的。国内的产品经理往往不是技术出身，领导开发团队会出很多问题，所以产品经理并不适合当PO。 **另一种做法是产品经理做利益相关者**，这种做法更不靠谱，利益相关者离团队太远，会出现无法及时响应的问题，时间长了也会导致产品和开发之间关系冷漠，出现隔阂。我们目前的做法是尝试是引入一个非官方版本的Scrum（Scrum 3.0），他将PO拆分成了业务拥有者（BO）和团队队长（TC）。产品经理担任BO，开发经理担任TC，目前看效果还不错 我们目前的做法是尝试是引入一个非官方版本的Scrum（Scrum 3.0），他将PO拆分成了业务拥有者（BO）和团队队长（TC）。产品经理担任BO，开发经理担任TC，目前看效果还不错 另外腾讯企业微信里[**TAPD**](https://www.tapd.cn)也是不错的 ================================================ FILE: docs/framework/algorithm-ten.md ================================================ # 必学的 10 大算法 >10 大常用机器学习算法，包括线性回归、Logistic 回归、线性判别分析、朴素贝叶斯、KNN、随机森林等。 ## 线性回归 在统计学和机器学习领域，线性回归可能是最广为人知也最易理解的算法之一。 预测建模主要关注的是在牺牲可解释性的情况下，尽可能最小化模型误差或做出最准确的预测。我们将借鉴、重用来自许多其它领域的算法（包括统计学）来实现这些目标。 线性回归模型被表示为一个方程式，它为输入变量找到特定的权重（即系数 B），进而描述一条最佳拟合了输入变量（x）和输出变量（y）之间关系的直线。 ![algorithm-ten](../img/alorithm/algorithm-ten-1.png) 例如：y = B0 + B1 * x 我们将在给定输入值 x 的条件下预测 y，线性回归学习算法的目的是找到系数 B0 和 B1 的值。 我们可以使用不同的技术来从数据中学习线性回归模型，例如普通最小二乘法的线性代数解和梯度下降优化。 线性回归大约有 200 多年的历史，并已被广泛地研究。在使用此类技术时，有一些很好的经验规则：我们可以删除非常类似（相关）的变量，并尽可能移除数据中的噪声。线性回归是一种运算速度很快的简单技术，也是一种适合初学者尝试的经典算法。 ## Logistic 回归 Logistic 回归是机器学习从统计学领域借鉴过来的另一种技术。它是二分类问题的首选方法。 像线性回归一样，Logistic 回归的目的也是找到每个输入变量的权重系数值。但不同的是，Logistic 回归的输出预测结果是通过一个叫作「logistic 函数」的非线性函数变换而来的。 logistic 函数的形状看起来像一个大的「S」，它会把任何值转换至 0-1 的区间内。这十分有用，因为我们可以把一个规则应用于 logistic 函数的输出，从而得到 0-1 区间内的捕捉值（例如，将阈值设置为 0.5，则如果函数值小于 0.5，则输出值为 1），并预测类别的值。 ![algorithm-ten](../img/alorithm/algorithm-ten-2.png) 由于模型的学习方式，Logistic 回归的预测结果也可以用作给定数据实例属于类 0 或类 1 的概率。这对于需要为预测结果提供更多理论依据的问题非常有用。 与线性回归类似，当删除与输出变量无关以及彼此之间非常相似（相关）的属性后，Logistic 回归的效果更好。该模型学习速度快，对二分类问题十分有效。 ## 线性判别分析 Logistic 回归是一种传统的分类算法，它的使用场景仅限于二分类问题。如果你有两个以上的类，那么线性判别分析算法（LDA）是首选的线性分类技术。 LDA 的表示方法非常直接。它包含为每个类计算的数据统计属性。对于单个输入变量而言，这些属性包括： * 每个类的均值。 * 所有类的方差。 ![algorithm-ten](../img/alorithm/algorithm-ten-3.png) ## 分类和回归树 决策树是一类重要的机器学习预测建模算法。 决策树可以被表示为一棵二叉树。这种二叉树与算法设计和数据结构中的二叉树是一样的，没有什么特别。每个节点都代表一个输入变量（x）和一个基于该变量的分叉点（假设该变量是数值型的）。 ![algorithm-ten](../img/alorithm/algorithm-ten-4.png) 决策树的叶子结点包含一个用于做出预测的输出变量（y）。预测结果是通过在树的各个分叉路径上游走，直到到达一个叶子结点并输出该叶子结点的类别值而得出。 决策树的学习速度很快，做出预测的速度也很快。它们在大量问题中往往都很准确，而且不需要为数据做任何特殊的预处理准备。 ## 朴素贝叶斯 朴素贝叶斯是一种简单而强大的预测建模算法。 该模型由两类可直接从训练数据中计算出来的概率组成：1）数据属于每一类的概率；2）给定每个 x 值，数据从属于每个类的条件概率。一旦这两个概率被计算出来，就可以使用贝叶斯定理，用概率模型对新数据进行预测。当你的数据是实值的时候，通常假设数据符合高斯分布（钟形曲线），这样你就可以很容易地估计这些概率。 ![algorithm-ten](../img/alorithm/algorithm-ten-5.png) 朴素贝叶斯之所以被称为「朴素」，是因为它假设每个输入变量相互之间是独立的。这是一种很强的、对于真实数据并不现实的假设。不过，该算法在大量的复杂问题中十分有效。 ## K 最近邻算法 K 最近邻（KNN）算法是非常简单而有效的。KNN 的模型表示就是整个训练数据集。这很简单吧？ 对新数据点的预测结果是通过在整个训练集上搜索与该数据点最相似的 K 个实例（近邻）并且总结这 K 个实例的输出变量而得出的。对于回归问题来说，预测结果可能就是输出变量的均值；而对于分类问题来说，预测结果可能是众数（或最常见的）的类的值。 关键之处在于如何判定数据实例之间的相似程度。如果你的数据特征尺度相同（例如，都以英寸为单位），那么最简单的度量技术就是使用欧几里得距离，你可以根据输入变量之间的差异直接计算出该值。 ![algorithm-ten](../img/alorithm/algorithm-ten-6.png) KNN 可能需要大量的内存或空间来存储所有数据，但只有在需要预测时才实时执行计算（或学习）。随着时间的推移，你还可以更新并管理训练实例，以保证预测的准确率。 使用距离或接近程度的度量方法可能会在维度非常高的情况下（有许多输入变量）崩溃，这可能会对算法在你的问题上的性能产生负面影响。这就是所谓的维数灾难。这告诉我们，应该仅仅使用那些与预测输出变量最相关的输入变量。 ## 学习向量量化 KNN 算法的一个缺点是，你需要处理整个训练数据集。而学习向量量化算法（LVQ）允许选择所需训练实例数量，并确切地学习这些实例。 ![algorithm-ten](../img/alorithm/algorithm-ten-7.png) LVQ 的表示是一组码本向量。它们在开始时是随机选择的，经过多轮学习算法的迭代后，最终对训练数据集进行最好的总结。通过学习，码本向量可被用来像 K 最近邻那样执行预测。通过计算每个码本向量与新数据实例之间的距离，可以找到最相似的邻居（最匹配的码本向量）。然后返回最匹配单元的类别值（分类）或实值（回归）作为预测结果。如果将数据重新放缩放到相同的范围中（例如 0 到 1 之间），就可以获得最佳的预测结果。 如果你发现 KNN 能够在你的数据集上得到不错的预测结果，那么不妨试一试 LVQ 技术，它可以减少对内存空间的需求，不需要像 KNN 那样存储整个训练数据集。 ## 支持向量机 支持向量机（SVM）可能是目前最流行、被讨论地最多的机器学习算法之一。 超平面是一条对输入变量空间进行划分的「直线」。支持向量机会选出一个将输入变量空间中的点按类（类 0 或类 1）进行最佳分割的超平面。在二维空间中，你可以把他想象成一条直线，假设所有输入点都可以被这条直线完全地划分开来。SVM 学习算法旨在寻找最终通过超平面得到最佳类别分割的系数。 ![algorithm-ten](../img/alorithm/algorithm-ten-8.png) 超平面与最近数据点之间的距离叫作间隔（margin）。能够将两个类分开的最佳超平面是具有最大间隔的直线。只有这些点与超平面的定义和分类器的构建有关，这些点叫作支持向量，它们支持或定义超平面。在实际应用中，人们采用一种优化算法来寻找使间隔最大化的系数值。 支持向量机可能是目前可以直接使用的最强大的分类器之一，值得你在自己的数据集上试一试。 ## 袋装法和随机森林 随机森林是最流行也最强大的机器学习算法之一，它是一种集成机器学习算法。 自助法是一种从数据样本中估计某个量（例如平均值）的强大统计学方法。你需要在数据中取出大量的样本，计算均值，然后对每次取样计算出的均值再取平均，从而得到对所有数据的真实均值更好的估计。 Bagging 使用了相同的方法。但是最常见的做法是使用决策树，而不是对整个统计模型进行估计。Bagging 会在训练数据中取多个样本，然后为每个数据样本构建模型。当你需要对新数据进行预测时，每个模型都会产生一个预测结果，Bagging 会对所有模型的预测结果取平均，以便更好地估计真实的输出值。 ![algorithm-ten](../img/alorithm/algorithm-ten-9.png) 随机森林是这种方法的改进，它会创建决策树，这样就不用选择最优分割点，而是通过引入随机性来进行次优分割。 因此，为每个数据样本创建的模型比在其它情况下创建的模型更加独特，但是这种独特的方式仍能保证较高的准确率。结合它们的预测结果可以更好地估计真实的输出值。 如果你使用具有高方差的算法（例如决策树）获得了良好的结果，那么你通常可以通过对该算法执行 Bagging 获得更好的结果。 ## Boosting 和 AdaBoost Boosting 是一种试图利用大量弱分类器创建一个强分类器的集成技术。要实现 Boosting 方法，首先你需要利用训练数据构建一个模型，然后创建第二个模型（它企图修正第一个模型的误差）。直到最后模型能够对训练集进行完美地预测或加入的模型数量已达上限，我们才停止加入新的模型。 AdaBoost 是第一个为二分类问题开发的真正成功的 Boosting 算法。它是人们入门理解 Boosting 的最佳起点。当下的 Boosting 方法建立在 AdaBoost 基础之上，最著名的就是随机梯度提升机。 ![algorithm-ten](../img/alorithm/algorithm-ten-10.png) AdaBoost 使用浅层决策树。在创建第一棵树之后，使用该树在每个训练实例上的性能来衡量下一棵树应该对每个训练实例赋予多少权重。难以预测的训练数据权重会增大，而易于预测的实例权重会减小。模型是一个接一个依次创建的，每个模型都会更新训练实例权重，影响序列中下一棵树的学习。在构建所有的树之后，我们就可以对新的数据执行预测，并根据每棵树在训练数据上的准确率来对其性能进行加权。 由于算法在纠正错误上投入了如此多的精力，因此删除数据中的异常值在数据清洗过程中是非常重要的。 ================================================ FILE: docs/framework/data_middle.md ================================================ # 数据中台的思考与总结 ## 数据中台 ![data-middle](../img/data_middle/data-middle-1.png) ### 数据汇聚 数据汇聚是数据中台必须提供的核心工具，把各种异构网络、异构数据源的数据方便地采集到数据中台中进行集中存储，为后续的加工建模做准备。数据汇聚方式一般有数据库同步、埋点、网络爬虫、消息队列等；从汇聚的时效性来分，有离线批量汇聚和实时采集。 数据采集工具: * Canal * DataX * Sqoop ### 数据开发 数据开发模块主要面向开发人员、分析人员，提供离线、实时、算法开发工具。 #### 离线开发 1. 作业调度 * 依赖调度：所有父作业运行完成后，当前作业才能开始运行。图64中的作业B，只有父作业A和C运行完成后，才能开始被调度。 * 时间调度：可指定作业的调度开始时间。图64中的作业B，只有到达05：00后才能开始被调度。 2. 基线控制 在大数据离线作业中，作业执行时间较长，经常遇到急着用数据发现数据还没出来的情况。采用算法对作业完成时间进行智能预测，根据预测，当作业无法正常产出且动态调整无法完成时，调度中心会及时通过监控告警通知运维值班人员提前介入处理，为大数据作业执行留出充裕的时间。 3. 异构存储 企业内部的存储计算引擎呈多元化趋势。离线开发中心针对每种类型的计算引擎会开发不同的组件，例如，针对Oracle开发Oracle插件，针对Hadoop体系分别开发出Hive、Spark、MR等插件。用户在界面新建各种作业类型，在执行时自动根据作业的类型寻找相应的插件来运行作业。 4. 代码校验 对于常见的SQL任务类型，SQL检查器会做好严格的管控，做到事前发现问题。 5. 多环境级联 通过环境级联的方式灵活支持企业的各类环境需求，方便对资源、权限进行控制和隔离。每个环境有独立的Hive数据库、Yarn调度队列，甚至不同的Hadoop集群。常见的环境如下： * 单一环境：只有一个生产环境，内部管理简单。 * 经典环境：开发环境中存放脱敏数据、供开发测试使用，上生产环境走发布流程，用于真实数据生产。 任务、资源和函数必须在开发环境下进行新建、修改或删除，再经过提交、创建发布包、同意发布三个操作后，才能同步到生产环境。 * 复杂环境：企业有外部人员和内部人员，会给外部人员提供一个脱敏管控的环境，外部人员开发完的数据模型经过测试后发布到内部开发环境。 6. 推荐依赖 随着业务的不断深入，数据开发人员需要开发的作业会不断累加。既能保证准确找到需要定位的上游作业，又能保证不会形成环路。 ![data-middle](../img/data_middle/data-middle-2.png) 获取推荐依赖的核心原理在于上下游作业输入和输出的表级血缘依赖图； 通过血缘分析当前作业的输入和输出，找到合适的上游作业； 对合适的作业进行环路检测，剔除存在闭环的作业； 返回合适的节点列表。 7. 数据权限 企业内部计算引擎多样化，数据权限管理面临如下问题： * 部分引擎拥有独立的权限管理系统（例如Oracle、HANA、LibrA），导致权限申请需要到每一种引擎上单独操作，让使用变得复杂。 * 同一种计算引擎，不同厂商的权限系统有多种，例如Hadoop自身无数据权限系统，由不同厂商各自去实现，目前主要有两种策略： * RBAC（Role-Based Access Control）：如Cloudera用的是Sentry，华为的FI也是类似的机制 * PBAC（Policy-Based Access Control）：如Hortonworks用的Ranger * 数据权限是由大数据集群或数据库运维人员管理的，开发人员无法直接操作或者接触，所有的权限申请都需要运维人员开通，造成运维人员负担过重。在实际开发中，一般需要运维人员把整个库的权限授权给某个开发负责人，然后库里面的表、字段、函数的权限管理由开发负责人负责就行。 数据权限管理中心提供界面化操作，数据申请方直接在页面上进行各种权限的申请，数据管理方在界面上审核权限，执行同意或拒绝操作。同时，所有权限的申请、审批都会有记录，便于进行权限审计。在统一数据权限服务中，会对接底层的各种权限管理系统，例如Sentry、Ranger、Oracle，同时对数据权限管理中心提供服务，执行权限的申请、授权、撤销等操作。 #### 实时开发 1. 元数据管理 2. SQL驱动 3. 组件化开发 ### 智能运维 任务的管理、代码发布、运维、监控、告警等一系列集成工具，方便使用，提升效率。重跑、重跑下游、补数据。 ### 数据体系 有了数据汇聚、数据开发模块，中台已经具备传统数据仓库（后面简称：数仓）平台的基本能力，可以做数据的汇聚以及各种数据开发，就可以建立企业的数据体系。之前说数据体系是中台的血肉，开发、管理、使用的都是数据。 中台数据体系应具备以下特征： * 覆盖全域数据：数据集中建设、覆盖所有业务过程数据，业务中台在数据体系中总能找到需要的数据。 * 结构层次清晰：纵向的数据分层、横向主题域、业务过程划分，让整个层次结构清晰易理解。 * 数据准确一致：定义一致性指标，统一命名、统一业务含义、统一计算口径，并有专业团队负责建模，保证数据的准确一致。 * 性能提升：统一的规划设计，选用合理的数据模型，清晰的定义并统一规范，并且考虑使用场景，使整体性能更好。 * 降低成本：数据体系的建设使得数据能被业务共享，这避免了大量烟囱式的重复建设，节约了计算、存储和人力成本。 * 方便易用：易用的总体原则是越往后越能方便地直接使用数据，把一些复杂的处理尽可能前置，必要时做适当的冗余处理。 不同行业的数据体系建设： * 地产行业 ![data-middle](../img/data_middle/data-middle-3.png) * 证券行业 ![data-middle](../img/data_middle/data-middle-4.png) * 零售行业 ![data-middle](../img/data_middle/data-middle-5.png) * 制造行业 ![data-middle](../img/data_middle/data-middle-6.png) * 传媒行业 ![data-middle](../img/data_middle/data-middle-7.png) * 检务行业 ![data-middle](../img/data_middle/data-middle-8.png) 1. 贴源数据层ODS 对各业务系统数据进行采集、汇聚，尽可能保留原始业务流程数据，与业务系统基本保持一致，仅做简单整合、非结构化数据结构化处理或者增加标识数据日期描述信息，不做深度清洗加工。 表名：ODS_系统简称_业务系统表名 字段名：与业务系统字段名保持一致，字段类型也尽可能保持一致 对于数据量比较大的业务表，采用增量同步的方式，则要同时建立增量表和全量表，增量表命名加后缀：ODS_系统简称_业务系统表名_delta。 对于日志、文件等半结构数据，不仅要存储原始数据，还要存储结构化之后的数据。 使用DataX同步数据步骤： 1）确定业务系统源表与贴源数据层目标表 2）配置数据字段映射关系，目标表可能会增加采集日期、分区、原系统标识等必要信息，业务相关内容不做转换 3）如果是增量同步或着有条件的同步部分数据，则配置数据同步条件 4）清理目标表对应数据 5）启动同步任务，往贴源数据层目标表导入数据 6）验证任务是否可以正确运行，并且采集到准确数据 7）发布采集任务，加入生产调度，并配置相关限速、容错、质量监控、告警机制 2. 统一数仓层DW * 明细数据层DWD * 汇总数据层DWS 与传统数据仓库功能基本一致，对全历史业务过程数据进行建模存储。对来源于业务系统的数据进行重新组织。业务系统是按照业务流程方便操作的方式来组织数据的，而统一数仓层从业务易理解的视角来重新组织，定义一致的指标、维度，各业务板块、业务域按照统一规范独立建设，从而形成统一规范的标准业务数据体系。 3. 标签数据层TDM 面向对象建模，对跨业务板块、跨数据域的特定对象数据进行整合，通过IDMapping把各个业务板块、各个业务过程中的同一对象的数据打通，形成对象的全域标签体系，方便深度分析、挖掘、应用。 ![data-middle](../img/data_middle/data-middle-9.png) 4. 应用数据层ADS 按照业务的需要从统一数仓层、标签数据层抽取数据，并面向业务的特殊需要加工业务特定数据，以满足业务及性能需求，向特定应用组装应用数据。 ### 数据资产管理 数据资产管理包括对数据资产目录、元数据、数据质量、数据血缘、数据生命周期等进行管理和展示，以一种更直观的方式展现企业的数据资产，提升企业的数据意识。 数据资产对上支持以价值挖掘和业务赋能为导向的数据应用开发，对下依托大数据平台实现数据全生命周期的管理，并对企业数据资产的价值、质量进行评估，促进企业数据资产不断自我完善，持续向业务输出动力。 #### 数据治理 传统的数据治理通常包含数据标准管理、元数据管理、数据质量管理、数据安全管理、数据生命周期管理等内容。 ## 数据服务体系 前面利用数据汇聚、数据开发建设企业的数据资产，利用数据管理展现企业的数据资产，但是并没有发挥数据的价值。数据服务体系就是把数据变为一种服务能力，通过数据服务让数据参与到业务， 快速开发企业的业务中台等。 ### 查询服务 输入特定的查询条件，返回该条件下的数据，以API形式供上层应用调用。 1）支持配置查询标识，底层数据组织一般会对该标识建立索引，以加快查询速度 2）支持配置过滤项 3）支持查询结果配置，包括数据排序规则和分页规则。 ### 分析服务 借助分析组件高效的大数据分析能力，对数据进行关联分析，分析结果通过API形式供上层应用调用。 1）支持多源数据接入：企业的数据经过清洗加工转换成数据资产后，最终通过服务作用于业务系统，基于企业异构存储的现状，要求分析服务能够支持与Hive、ES、Greenplum、MySQL、Oracle、本地文件等多种数据源进行连接。 2）高性能即席查询：随着企业数据爆发式增长，传统的数据分析工具遇到分析能力的瓶颈，也就是对大数据量的分析越来越乏力。因此，这就要求分析服务内置高速计算引擎，以对数据进行高性能的即席计算，实现亿级数据毫秒级（至多秒级）分析和计算，减少用户等待时间。 3）多维数据分析 分析服务除了支持常规的数据分析、上卷下钻、切片切块之外，还应该支持多维的数据分析以及深层次的数据挖掘，发现数据背后的关联关系。 4）灵活对接业务系统 ### 推荐服务 按约定的格式提供历史日志行为数据和实时访问数据，推荐模型就会生成相应的推荐API，从而为上层应用提供推荐服务。 推荐服务即所谓的千人千面，对不同的人对物的行为进行数据挖掘，构建每个人与物之间的关系程度，来推荐人、物以满足用户的兴趣爱好，以提升用户对业务的粘性。每个人打开手机淘宝看到的内容都不一样，这就是一种基于人的兴趣爱好的推荐服务能力。 1）支持不同行业的推荐：不同行业背后的推荐逻辑是有区别的 2）支持不同场景的推荐：以内容资讯为例，在用户冷启动场景下，应该推荐哪些资讯？在用户已有浏览行为的场景下，又该为其推荐哪些资讯？ 3）支持推荐效果优化：从导入的原始数据开始，经过推荐组件生成推荐数据，再根据用户的浏览数据不断修正推荐模型，从而使推荐效果不断优化 ### 圈人服务 从全量用户数据中，基于标签组合筛选符合指定特征条件的人群，并通过API形式供上层应用调用。 1）支持人群圈选：通过SQL代码或标签取值组合等多种方式，实现人员查找，帮用户找到对的人群 2）支持人群计量：营销部门或者广告公司使用圈人服务圈选出目标人群后，往往还要考虑人群量是否符合预期，因为预算有限，不可能不计成本的对人群进行营销。 3）支持多渠道对接：将人群名单导出到相应的下游系统。最简单的名单导出方式是先下载文件，再由业务人员导入相应的业务系统中。或者直接对接到短信系统、微信投放接口、营销活动系统等。 ## 离线平台 ### 苏宁 [苏宁大数据离线任务开发调度平台实践](https://www.infoq.cn/article/suning-big-data) [苏宁大数据离线任务开发调度平台实践：任务调度模块架构设计](https://www.infoq.cn/article/xTvBg1_9iUL0z5Pjf0Os) 苏宁离线平台产品功能图： ![data-middle](../img/data_middle/data-middle-10.png) 苏宁调度模块功能图： ![data-middle](../img/data_middle/data-middle-11.png) 苏宁离线平台整体架构图： ![data-middle](../img/data_middle/data-middle-12.png) 跨任务流依赖的实现： FTP事件机制，即在 FTP 服务器上建立标识文件，一个事件对应一个标识文件地址，当 FTP 服务器上的标识文件生成的时候，我们认为业务系统已经完成作业，需要触发平台任务执行。 “华佗”平台，实施任务诊断： ![data-middle](../img/data_middle/data-middle-13.png) 立即触发的任务，放入DelayQueue的队列头部 周期调度的任务，使用Quartz 依赖触发的任务，使用zk，各个子节点监听自己的父节点，所有父节点执行完毕则可触发执行 ## 实时平台 ### 美团点评 ![data-middle](../img/data_middle/data-middle-14.png) 使用了Grafana，可以内嵌到自己的平台。 ### bilibili SQL化编程 DAG拖拽编程 一体化托管运维 实时平台由实时传输和实时计算两部分组成，平台底层统一管理元数据、血缘、权限以及作业运维等。实时传输主要负责将数据传入到大数据体系中。实时计算基于 BSQL 提供各种应用场景支持。 如下图所示，实时传输有 APP 日志、数据库 Binlog、服务端日志或系统日志。bilibili 内部的 Lancer 系统解决数据落地到 Kafka 或 HDFS。计算体系主要围绕 Saber 构建一套 BSQL，底层基于 YARN 进行调度管理。 上层核心基于 Flink 构建运行池。再向上一层满足多种维表场景，包括 MySQL、Redis、HBase。状态（State）部分在 RocksDB 基础上，还扩展了 MapDB、Redis。Flink 需要 IO 密集是很麻烦的问题，因为 Flink 的资源调度体系内有内存和 CPU，但 IO 单位未做统一管理。当某一个作业对 IO 有强烈的需求时，需要分配很多以 CPU 或内存为单位的资源，且未必能够很好的满足 IO 的扩展。所以本质上 bilibili 现阶段是将 IO 密集的资源的 State 转移到 Redis 上做缓解。数据经过 BSQL 计算完成之后传输到实时数仓，如 Kafka、HBase、ES 或 MySQL、TiDB。最终到 AI 或 BI、报表以及日志中心。 ![data-middle](../img/data_middle/data-middle-15.png) #### 场景 * AI工程方向，解决了广告、搜索、推荐的流式Joiner和维表Joiner * 实时计算的特征支持，支持 Player 以及 CDN 的质量监控。包括直播、PCU、卡顿率、CDN 质量等； * 用户增长，即如何借助实时计算进行渠道分析、调整渠道投放效果； * 实时 ETL，包括 Boss 实时播报、实时大屏、看板等。 ### 网易 目前网易流计算覆盖了绝大多数场景，包括广告、电商大屏、ETL、数据分析、推荐、风控、搜索、直播等。 #### 事件管理 对于分布式平台的任务操作而言，当前任务启动过程中只允许一个人操作，而不允许两个人同时操作，这就需要以下几个模块来共同配合： * Server：事件执行的发起者，接受事件的请求，进行数据校验，拼装，将事件发送给 Kernel 执行。 * Kernel：事件具体逻辑的执行者，根据请求向集群发送指令(Shell 脚本方式)。 * Admin：事件执行结果的确认者，根据事件类型，获取事件的最终结果，保证结果的正确性。 ![data-middle](../img/data_middle/data-middle-16.png) 以启动场景为例： 首先，Server 会接收到来自用户的启动请求，之后会创建一个分布式锁，Admin 会监控这个锁。 然后， Server 向 Kernel 提交任务，提交之后会立即返回，返回之后就会立即更新数据库中的状态，将状态更新为启动中，这样在页面上用户就能够看到任务是启动中的状态了。 接下来，Server 就会等待内核的 Shell 脚本的执行结果，如果 Shell 脚本执行成功了，就会去写 Zookeeper，写完 Zookeeper 之后 Admin 模块就会马上检测到 Zookeeper 节点有状态发生了修改，Admin 会立即去获取 YARN 上的任务状态，如果获取到任务状态是运行中，就将数据库的任务状态更新为运行中，这会在前端看到任务就已经是运行状态了。 最后一步是 Admin 更为完数据库之后，会释放掉 Zookeeper 上的锁，其他人这时候就可以操作这个任务了。 Server、Kernel 和 Admin 这三个模块都是不可靠的，那么如何保证其稳定和高可用呢？Server 可以通过部署多个，水平扩展来实现，Kernel 则会由 Server 来进行监听，当发现 Kernel 挂了，可以由 Server 重新拉起或者重新创建。而 Admin 的高可用则是通过热备来实现的，如果主 Admin 挂掉了，可以马上迁移到备 Admin，备 Admin 可以迅速将元数据以及任务信息全部加载进来接替工作，进而实现高可用。 #### 平台任务状态管理 平台的任务状态主要由 Server 和 Admin 来控制。Server 主要控制初始状态的执行，Admin 则主要负责控制所有与 YARN 相关的状态交互。 ![data-middle](../img/data_middle/data-middle-17.png) #### 任务调试 SQL 类型的任务支持调试功能，用户可以根据不同的 source 表和 dim 表，上传不同的 csv 文件作为输入数据，进行调试。调试执行由指定的 kernel 来完成，sloth-server 负责组装请求，调用 kernel，返回结果，搜集日志。 ![data-middle](../img/data_middle/data-middle-18.png) #### 日志检索 在 YARN 集群的每个节点上面部署 Filebeat，通过 Filebeat 将节点上面的任务日志写入到 Kafka 消息队列中，然后通过 Logstash 进行解析处理，之后写入 ES 集群中。主要用于两个用途，一个是通过界面 Kibana 来提供给开发和运维人员使用，另外一个就是将运行时状态的任务日志直接在界面上展示供用户进行搜索和查看。 ![data-middle](../img/data_middle/data-middle-19.png) #### 监控 在监控方面，使用的是 influxdb metric report 组件对于指标进行监控。时序数据库使用的是网易自研的 ntsdb 时序数据库，其能够支持动态扩展和高可用等功能。监控指标的使用方式有两种： 一种是通过 Grafana 的界面来查看指标； 另外一种是报警模块会从Ntsdb中获取相关指标数据并进行监控报警。 ![data-middle](../img/data_middle/data-middle-20.png) #### 报警 Sloth 流计算平台支持常见的任务失败，数据滞留延迟，failover 报警，也支持用户自定义规则报警，包括对于输入 QPS、输出 QPS，户自定义延迟的监控等。以输入 QPS 为例，可以设置当连续几个周期内 QPS 低于某一值时就触发报警。此外，报警方式也支持多样化的工具，比如各种网易内部的聊天工具、邮件、电话以及短信等，对于任务调试阶段，为了避免被骚扰，可以设置任务报警抑制时间间隔。 ![data-middle](../img/data_middle/data-middle-21.png) #### 实时数仓 目前网易很多产品已经开始实时数仓的建设了，但仍旧处于持续完善过程中。实时数仓的建设和离线数仓大致相同，只不过实时数仓是经过实时计算平台进行处理的。大致的过程就是首先收集日志、埋点数据等，将其写入到 Kafka 里面，经过实时计算平台进行处理，将 ODS 层中的明细数据抽取出来，在进行汇总以及维度关联等操作，将结果写入到 Redis，Kudu 等，再通过数据服务提供给前端的业务使用。x ![data-middle](../img/data_middle/data-middle-22.png) #### 电商应用-数据分析 实时活动分析、首页资源分析、流量漏斗以及实时毛利计算等。 #### 电商应用-搜索推荐 电商的搜索推荐场景则主要包括用户实时足迹、用户实时特征、商品实时特征、实时 CTR CVR 样本组建、首页 A 区轮播、B 区活动精选等 UV、PV 实时统计等。 网络营销中的常见名词解释： ``` c CPC (Cost Per Click): 按点击计费 CPA (Cost Per Action): 按成果数计费 CPM (Cost Per Mille): 按千次展现计费 CVR (Click Value Rate): 转化率，衡量CPA广告效果的指标 CTR (Click Through Rate): 点击率 PV (Page View): 流量 ADPV (Advertisement Page View): 载有广告的pageview流量ADimp (ADimpression): 单个广告的展示次数 PV单价: 每PV的收入，衡量页面流量变现能力的指标 ``` ## 离线数仓与实时数仓 ### 从0建设离线数仓 #### 建设数仓 数据仓库定义：在企业管理和决策中面向主题的、集成的、与时间相关的、不可修改的数据集合。 数据仓库目标：数据资产、决策信息。 * ETL过程：打通你的任督二脉（离线+实时），让数据在整个环节中流通起来 * 数据分层：一套低耦合、高内聚的层级，是十分重要的，总不想业务、数据等一变化，数仓像又投胎了一次 * 数据集成：多业务场景下，打破数据信息壁垒，避免数据歧义，统一数据服务 * 规范化：良好的流程化、规范化设计，易维护、高扩展 * 监控与辅助：质量监控、调度管理、元数据管理、信息安全管理 * 走向服务：对外api服务/自助查询平台/OLAP分析平台 #### ETL 业务数据往往涉及多种数据源，数据存储也常常会有多种选择。文本数据、日志数据、RMDB、Nosql等。则要求etl工具能够覆盖这些业务场景。 工具有datax/sqoop/kettle/informatica等等。 ETL一般为最开始的部分，凌晨之后的时间点。a：避免集中式的对某个jdbc海量同步，影响业务(部分从库可能提供查询服务)、b：明确调度的时间，应尽可能的在某个时间段内完成(不能仅依靠调度，实现任务流的串行；为后期的大作业空间，占用等待的系统资源) #### 分层 ![data-middle](../img/data_middle/data-middle-23.png) 1. Stage缓冲层 事务性数据，每日增量方式进行数据同步。需要注意数据同步时的边界问题，避免脏数据。 对于非事务性数据，一般通过快照/全量更新。不对外开放数据查询。 2. ods层 一般场景下，我们认为该层数据与线上保持一致。实际处理过程中，为了处理时间维度上的数据变化，会记录数据的变化轨迹。对于该部分数据，应该有选择的实施，避免业务处理过程变得复杂和问题发生后难以回溯。 3. dim/dw层 (模型层) dim：维度层 dw：主题事实及业务宽表 在ods基础上，设计一个宽表/模型层，通过维度建模的方式，实现维度数据与事实数据的分离（星型模型）。 4. da层（应用层） 面向不同的应用，聚合类的数据层。该层对于dim/dw层的使用，是对模型层的一个检视维度。 #### 代码规范 1. 脚本格式规范：脚本头部注释编码规范、注释规范、sql规范参考goole规范 2. 文件/表命名规范：一个文件中，只应该有一张表，其余只能是临时表；表名称应与文件名相同 3. 字段命名规范：去除多词同义，和同词多义的问题。尤其是在模型层（一般也叫做一致性维度） ### 区别 离线数仓主要基于sqoop、datax、hive等技术来构建 T+1 的离线数据，通过定时任务每天垃取增量数据导入到hive表中，然后创建各个业务相关的主题，对外提供T+1的数据查询接口。 实时数仓主要是基于数据采集工具，如canal等原始数据写入到kafka这样的数据通道中，最后一般都是写入到类似于HBase这样的OLAP存储系统中。对外提供分钟级别，甚至秒级别的查询方案。 数仓类型|准确性|实时性|稳定性 --|--|--|-- 离线数仓|准确度高|时延一般在一天|稳定性好，方便重算 实时数仓|准确度低，数据延迟、数据乱序造成数据准确度低|分钟级延迟|稳定性差，需要考虑数据回溯处理 ![data-middle](../img/data_middle/data-middle-24.png) 数据仓库的建设主要包括数据的采集、数据的处理、数据归档、数据应用四个方面。 当前主要的应用场景包括报表展示、即席查询、BI展示、数据分析、数据挖掘、模型训练等方面。 数据仓库的建设是面向主题的、集成性的、不可更新的、时许变化的。 实时数仓的实施关键点： 1. 端到端数据延迟、数据流量的监控 2. 故障的快速恢复能力 3. 数据的回溯处理，系统支持消费指定时间段内的数据 4. 实时数据从实时数仓中查询，T+1数据借助离线通道修正 5. 数据地图、数据血缘关系的梳理 6. 业务数据质量的实时监控，初期可以根据规则的方式来识别质量状况 其实，你需要的不是实时数仓，需要的是一款合适且强大的OLAP数据库。 在实时数仓的建设中，OLAP数据库的选型直接制约实时数仓的可用性和功能性。 原始层 明细层 汇总层 应用层 * ods：原始数据层，事实数据，存储在kafka中 * dwd：数据明细层，可以做一些join等加宽处理，可以存储在kafka和redis中 * dim：维度数据，如存储在HBase中的数据 * dm：MySQL -> 汇总指标模型；Greenplum -> 明细，多维分析关联；HBase -> 汇总指标(大量并发)；Redis -> 汇总、大列表TopN ## 数据中台解决方案 ### 零售行业 ![data-middle](../img/data_middle/data-middle-25.png) ``` c RPS (Revenue Per Search): 每搜索产生的收入，衡量搜索结果变现能力指标 ``` 　 ROI： 投资回报率（ROI）是指通过投资而应返回的价值，它涵盖了企业的获利目标。利润和投入的经营所必备的财产相关，因为管理人员必须通过投资和现有财产获得利润。又称会计收益率、投资利润率。 ================================================ FILE: docs/framework/fgb.md ================================================ # 怎么理解分布式、高并发、多线程？ 是不是很多人都认为分布式=高并发=多线程？ 当面试官问到高并发系统可以采用哪些手段来解决，或者被问到分布式系统如何解决一致性的问题，是不是一脸懵逼？ 一开始，不少人都会将三者混淆，误以为所谓的分布式高并发的系统就是能同时供海量用户访问，而采用多线程手段不就是可以提供系统的并发能力吗？实际上，**他们三个总是相伴而生，但侧重点又有不同。** ### 什么是分布式？ 分布式更多的一个概念，**是为了解决单个物理服务器容量和性能瓶颈问题而采用的优化手段**。该领域需要解决的问题极多，在不同的技术层面上，又包括：分布式文件系统、分布式缓存、分布式数据库、分布式计算等，一些名词如Hadoop、zookeeper、MQ等都跟分布式有关。从理念上讲，分布式的实现有两种形式： * 水平扩展：当一台机器扛不住流量时，就通过添加机器的方式，将流量平分到所有服务器上，所有机器都可以提供相当的服务； * 垂直拆分：前端有多种查询需求时，一台机器扛不住，可以将不同的需求分发到不同的机器上，比如A机器处理余票查询的请求，B机器处理支付的请求。 ### 什么是高并发？ **相对于分布式来讲，高并发在解决的问题上会集中一些，其反应的是同时有多少量**：比如在线直播服务，同时有上万人观看。 高并发可以通过分布式技术去解决，将并发流量分到不同的物理服务器上。但除此之外，还可以有很多其他优化手段：比如使用缓存系统，将所有的，静态内容放到CDN等；还可以使用多线程技术将一台服务器的服务能力最大化。 ![fgb](./../img/fgb1.png) ### 什么是多线程？ **多线程是指从软件或者硬件上实现多个线程并发执行的技术**，它更多的是解决CPU调度多个进程的问题，从而让这些进程看上去是同时执行（实际是交替运行的）。 这几个概念中，**多线程解决的问题是最明确的，手段也是比较单一的，基本上遇到的最大问题就是线程安全**。在JAVA语言中，需要对JVM内存模型、指令重排等深入了解，才能写出一份高质量的多线程代码。 ### 总结一下： * 分布式是从物理资源的角度去将不同的机器组成一个整体对外服务，技术范围非常广且难度非常大，有了这个基础，高并发、高吞吐等系统很容易构建； * 高并发是从业务角度去描述系统的能力，实现高并发的手段可以采用分布式，也可以采用诸如缓存、CDN等，当然也包括多线程； * 多线程则聚焦于如何使用编程语言将CPU调度能力最大化。 分布式与高并发系统，涉及到大量的概念和知识点，如果没有系统的学习，很容易会杂糅概念而辨识不清，在面试与实际工作中都会遇到困难。如果你从事Java开发，希望深入浅出了解Java分布式、高并发等技术要点，渴望实现技术和职业成长上的双重突破，那么以下福利就很适合 ================================================ FILE: docs/framework/fwork.md ================================================ # 走向架构师必备的技能 中国有很多年轻人，他们18,9岁或21,2岁，通过自学也写了不少代码，他们有的代码写的很漂亮，一些技术细节相当出众，也很有钻研精神，但是他们被一些错误的认识和观点左右，缺乏对系统，对程序的整体理解能力，这些人，一个网上的朋友说得很好，他们实际fans，压根没有资格称为程序员，但是据我所知，不少小网络公司的Cfans,拿着吓人的工资，做着吓人的项目，项目的结局通常也很吓人。 ## 　程序员基本素质： 作一个真正合格的程序员，或者说就是可以真正合格完成一些代码工作的程序员，应该具有的素质。 1. 团队精神和协作能力 把它作为基本素质，并不是不重要，恰恰相反，这是程序员应该具备的最基本的，也是最重要的安身立命之本。把高水平程序员说成独行侠的都是在呓语，任何个人的力量都是有限的，即便如linus这样的天才，也需要通过组成强大的团队来创造奇迹，那些遍布全球的为linux写核心的高手们，没有协作精神是不可想象的。独行侠可以作一些赚钱的小软件发点小财，但是一旦进入一些大系统的研发团队，进入商业化和产品化的开发任务，缺乏这种素质的人就完全不合格了。 2. 文档习惯 说高水平程序员从来不写文档的肯定是乳臭未干的毛孩子，良好的文档是正规研发流程中非常重要的环节，作为代码程序员，30％的工作时间写技术文档是很正常的，而作为高级程序员和系统分析员，这个比例还要高很多。缺乏文档，一个软件系统就缺乏生命力，在未来的查错，升级以及模块的复用时就都会遇到极大的麻烦。 3. 规范化，标准化的代码编写习惯 作为一些外国知名软件公司的规矩，代码的变量命名，代码内注释格式，甚至嵌套中行缩进的长度和函数间的空行数字都有明确规定，良好的编写习惯，不但有助于代码的移植和纠错，也有助于不同技术人员之间的协作。 fans叫嚣高水平程序员写的代码旁人从来看不懂，这种叫嚣只能证明他们自己压根不配自称程序员。代码具有良好的可读性，是程序员基本的素质需求。 再看看整个linux的搭建，没有规范化和标准化的代码习惯，全球的研发协作是绝对不可想象的。 4. 需求理解能力 程序员需要理解一个模块的需求，很多小朋友写程序往往只关注一个功能需求，他们把性能指标全部归结到硬件，操作系统和开发环境上，而忽视了本身代码的性能考虑，有人曾经放言说写一个广 告交换程序很简单，这种人从来不知道在百万甚至千万数量级的访问情况下的性能指标是如何实现的，对于这样的程序员，你给他深蓝那套系统，他也做不出太极链的并访能力。性能需求指标中，稳定性，并访支撑能力以及安全性都很重要，作为程序员需要评估该模块在系统运营中所处的环境，将要受到的负荷压力以及各种潜在的危险和恶意攻击的可能性。就这一点，一个成熟的程序员至少需要2到3年的项目研发和跟踪经验才有可能有心得。 5. 复用性，模块化思维能力 经常可以听到一些程序员有这样的抱怨，写了几年程序，变成了熟练工，每天都是重复写一些没有任何新意的代码，这其实是中国软件人才最大浪费的地方，一些重复性工作变成了熟练程序员的主要工作，而这些，其实是完全可以避免的。 复用性设计，模块化思维就是要程序员在完成任何一个功能模块或函数的时候，要多想一些，不要局限在完成当前任务的简单思路上，想想看该模块是否可以脱离这个系统存在，是否可以通过简单的修改参数的方式在其他系统和应用环境下直接引用，这样就能极大避免重复性的开发工作，如果一个软件研发单位和工作组能够在每一次研发过程中都考虑到这些问题，那么程序员就不会在重复性的工作中耽误太多时间，就会有更多时间和精力投入到创新的代码工作中去。 一些好的程序模块代码，即便是70年代写成的，拿到现在放到一些系统里面作为功能模块都能适合的很好，而现在我看到的是，很多小公司软件一升级或改进就动辄全部代码重写，大部分重复性工作无谓的浪费了时间和精力。 6. 测试习惯 作为一些商业化正规化的开发而言，专职的测试工程师是不可少的，但是并不是说有了专职的测试工程师程序员就可以不进行自测；软件研发作为一项工程而言，一个很重要的特点就是问题发现的越早，解决的代价就越低，程序员在每段代码，每个子模块完成后进行认真的测试，就可以尽量将一些潜在的问题最早的发现和解决，这样对整体系统建设的效率和可靠性就有了最大的保证。 测试工作实际上需要考虑两方面，一方面是正常调用的测试，也就是看程序是否能在正常调用下完成基本功能，这是最基本的测试职责，可惜在很多公司这成了唯一的测试任务，实际上还差的远那；第二方面就是异常调用的测试，比如高压力负荷下的稳定性测试，用户潜在的异常输入情况下的测试，整体系统局部故障情况下该模块受影响状况的测试，频发的异常请求阻塞资源时的模块稳定测试等等。当然并不是程序员要对自己的每段代码都需要进行这种完整测试，但是程序员必须清醒认识自己的代码任务在整体项目中的地位和各种性能需求，有针对性的进行相关测试并尽早发现和解决问题，当然这需要上面提到需求理解能力。 7. 学习和总结的能力 程序员是人才很容易被淘汰，很容易落伍的职业，因为一种技术可能仅仅在三两年内具有领先性，程序员如果想安身立命，就必须不断跟进新的技术，学习新的技能。 善于学习，对于任何职业而言，都是前进所必需的动力，对于程序员，这种要求就更加高了。但是学习也要找对目标，一些小coding有些codingTO就是这样的coding上只是一些Cfans们，他们也津津乐道于他们的学习能力，一会学会了asp，一会儿学会了php，一会儿学会了jsp，他们把这个作为炫耀的资本，盲目的追逐一些肤浅的，表面的东西和名词，做网络程序不懂通讯传输协议，做应用程序不懂中断向量处理，这样的技术人员，不管掌握了多少所谓的新语言，永远不会有质的提高。 善于总结，也是学习能力的一种体现，每次完 成一个研发任务，完成一段代码，都应当有目的的跟踪该程序的应用状况和用户反馈，随时总结，找到自己的不足，这样逐步提高，一个程序员才可能成长起来。 一个不具备成长性的程序员，即便眼前看是个高手，建议也不要选用，因为他落伍的时候马上就到了。具备以上全部素质的人，应当说是够格的程序员了，请注意以上的各种素质都不是由IQ决定的，也不是大学某些课本里可以学习到的，需要的仅仅是程序员对自己工作的认识，是一种意识上的问题。 ## 高级程序员 那么作为高级程序员，以至于系统分析员，也就是对于一个程序项目的设计者而言，除了应该具备上述全部素质之外，还需要具备以下素质： 1. 需求分析能力 对于程序员而言，理解需求就可以完成合格的代码，但是对于研发项目的组织和管理者，他们不但要理解客户需求，更多时候还要自行制定一些需求，为什么这么说呢？ 一般而言，进行研发任务，也许是客户提出需求，也许是市场和营销部门提出的需求，这时候对于研发部门，他们看到的不是一个完整的需求，通常而言，该需求仅仅是一些功能上的要求，或者更正规些，可能获得一个完整的用户视图；但是这都不够，因为客户由于非技术因素多一些，他们可能很难提出完整和清晰，或者说专业性的性能需求，但是对于项目组织者和规划者，他必须能够清醒认识到这些需求的存在并在完成 需求分析报告的时候适当的提出，同时要完整和清晰的体现在设计说明书里面，以便于程序员编码时不会失去这些准则。 程序设计者必须正确理解用户需求所处的环境，并针对性做出需求的分析，举例而言，同样一个软件通过ASP租用方式发布和通过License方式发布，性能需求可能就是有区别的，前者强调的是更好的支撑能力和稳定性，而后者则可能更强调在各种平台下的普适性和安装使用的简捷性。 2. 项目设计方法和流程处理能力 程序设计者必须能够掌握不少于两到三种的项目设计方法（比如自顶至下的设计方法，比如快速原型法等等），并能够根据项目需求和资源搭配来选择合适的设计方法进行项 目的整体设计。设计方法上选择不当，就会耽误研发周期，浪费研发资源，甚至影响研发效果。 一个程序设计者还需要把很多功夫用在流程图的设计和处理上，他需要做数据流图以确立数据词典；他需要加工逻辑流图以形成整体的系统处理流程。一个流程有问题的系统，就算代码多漂亮，每个模块多精致，也不会成为一个好的系统。当然，做好流程分析并选择好项目设计方法，都需要在需求分析能力上具有足够的把握。 3. 复用设计和模块化分解能力 这个似乎又是老调重谈，前面基本素质上不是已经说明了这个问题吗？作为一个从事模块任务的程序员，他需要对他所面对的特定功能模块的 复用性进行考虑，而作为一个系统分析人员，他要面对的问题复杂的多，需要对整体系统按照一种模块化的分析能力分解为很多可复用的功能模块和函数，并针对每一模块形成一个独立的设计需求。举个例子，好比是汽车生产，最早每辆汽车都是独立安装的，每个部件都是量身定做的，但是后来不一样了，机器化大生产了，一个汽车厂开始通过流水线来生产汽车，独立部件开始具有一定的复用性，在后来标准化成为大趋势，不同型号，品牌甚至不同厂商的汽车部件也可以进行方便的换装和升级，这时候，汽车生产的效率达到最大化。软件工程也是同样的道理，一个成熟的软件行业，在一些相关项目和系统中，不同的部件是可以随意换装的，比如微软的许多桌面软件，在很多操作模块（如打开文件，保存文件等等）都是复用的同一套功能模块，而这些接口又通过一些类库提供给了桌面应用程序开发者方便挂接，这就是复用化的模块设计明显的一个佐证。 将一个大型的，错综复杂的应用系统分解成一些相对独立的，具有高度复用性的，并能仅仅依靠几个参数完成数据联系的模块组合，是作为高级程序员和系统分析员一项最重要的工作，合适的项目设计方法，清晰的流程图，是实现这一目标的重要保证。 4. 整体项目评估能力 作为系统设计人员，必须能够从全局出发，对项目又整体的清醒认识，比如公司的资源配置是否合理和到位，比如工程进度安排是否能最大化体现效率又不至于无法按期完成。评估项 目整体和各个模块的工作量，评估项目所需的资源，评估项目可能遇到的困难，都需要大量的经验积累，换言之，这是一种不断总结的累计才能达到的境界。在西方一些软件系统设计的带头人都是很年长的，比如4，50岁，甚至更老，他们在编码方面已经远远不如年轻人那样活络，但是就项目评估而言，他们几十年的经验积累就是最重要和宝贵的财富。中国缺这么一代程序员，主要还不是缺那种年纪的程序员，而是那种年纪的程序员基本上都是研究单位作出来的，都不是从专业的产品化软件研发作出来的，他们没有能积累那种产品化研发的经验，这也是没有办法的事情。 5. 团队组织管理能力 完成一个项目工程，需要团队的齐心协力，作为项目设计者或研发的主管人，就应当有能力最大化发挥团队的整体力量，技术管理由于其专业性质，不大同于一般的人事管理，因为这里面设计了一些技术性的指标和因素。 首先是工作的量化，没有量化就很难做到合适的绩效考核，而程序量化又不是简单的代码行数可以计算的，因此要求技术管理人员需要能真正评估一个模块的复杂性和工作量。 其次是对团队协作模式的调整，一般而言，程序开发的协作通常分为小组进行，小组有主程序员方式的，也有民主方式的，根据程序员之间的能力水平差距，以及根据项目研发的需求，选择合适的组队方式，并能将责权和成员的工作任务紧密结合，这样才能最大发挥组队的效率。 一个代码水平高的人，未必能成为一个合格的项目研发主管，这方面的能力欠缺往往是容易被忽视的。 综上可以看到，作为一个主管研发的负责人，一个项目设计者，所需要具备的素质和能力并不是程序代码编写的能力，当然一般情况下，一个程序员通过不断的总结提高达到了这种素质的时候，他所具有的代码编写能力也已经相当不简单了，但是请注意这里面的因果关系，一个高水平的项目设计者通常已经是代码编写相当优秀的人了，但是并不是一个代码相当优秀的程序员就可以胜任项目设计的工作，这里面存在的也不是智商和课本的问题，还是在于一个程序员在积累经验，逐步提升的时候没有意识到应当思考哪方面的东西，没有有意识的就项目的组织和复用设计进行揣摩，没有经常性的文档习惯和总结习惯， 不改变这些，我们的合格的项目设计者还是非常欠缺。 另外，为防止有无聊的人和我较真，补充一点，本文针对目标是作商业化的软件项目和工程，那些科研机构的编程高手，比如算法高手，比如图象处理高手，他们的工作是研究课题而非直接完成商业软件（当然最终间接成为商业产品，比如微软研究院在作的研究课题），因此他们强调的素质可能是另外的东西，这些人（专家），并不能说是程序员，不能用程序员的标准去衡量。 最后补充一点东西，一个软件项目研发的设计流程是怎样的呢？以通常标准的设计方法为例，（不过笔者喜欢快速原型法）。 * 第一个步骤是市场调研，技术和市场要结合才能体现最大价值。 * 第二个步骤是需求分析，这个阶段需要出三样东西，用户视图，数据词典和用户操作手册。用户视图是该软件用户（包括终端用户和管理用户）所能看到的页面样式，这里面包含了很多操作方面的流程和条件。数据词典是指明数据逻辑关系并加以整理的东东，完成了数据词典，数据库的设计就完成了一半多。用户操作手册是指明了操作流程的说明书。 注意，用户操作流程和用户视图是由需求决定的，因此应该在软件设计之前完成，完成这些，就为程序研发提供了约束和准绳，很遗憾太多公司都不是这样做的，因果颠倒，顺序不分，开发工作和实际需求往往因此 产生隔阂脱节的现象。 需求分析，除了以上工作，笔者以为作为项目设计者应当完整的做出项目的性能需求说明书，因为往往性能需求只有懂技术的人才可能理解，这就需要技术专家和需求方（客户或公司市场部门）能够有真正的沟通和了解。 * 第三个步骤是概要设计，将系统功能模块初步划分，并给出合理的研发流程和资源要求。作为快速原型设计方法，完成概要设计就可以进入编码阶段了，通常采用这种方法是因为涉及的研发任务属于新领域，技术主管人员一上来无法给出明确的详细设计说明书，但是并不是说详细设计说明书不重要，事实上快速原型法在完成原型代码后，根据评测结果和经验教训的总结，还要重新进行详细设计的步骤。 * 第四个步骤是详细设计，这是考验技术专家设计思维的重 要关卡，详细设计说明书应当把具体的模块以最‘干净’的方式(黑箱结构）提供给编码者，使得系统整体模块化达到最大；一份好的详细设计说明书，可以使编码的复杂性减低到最低，实际上，严格的讲详细设计说明书应当把每个函数的每个参数的定义都精精细细的提供出来，从需求分析到概要设计到完成详细设计说明书，一个软件项目就应当说完成了一半了。换言之，一个大型软件系统在完成了一半的时候，其实还没有开始一行代码工作。那些把作软件的程序员简单理解为写代码的，就从根子上犯了错误了。 * 第五个步骤是编码，在规范化的研发流程中，编码工作在整个项目流程里最多不会超过1/2，通常在1/3的时间，所谓磨刀不误砍柴功，设计过程完成的好，编码效率就会极大提高，编码时不同模块之间的进度协调和协作是最需要小心的，也许一个小模块的问题就可能影响了整体进度，让很多程序员因此被迫停下工作等待，这种问题在很多研发过程中都出现过。编码时的相互沟通和应急的解决手段都是相当重要的，对于程序员而言，bug永远存在，你必须永远面对这个问题，大名鼎鼎的微软，可曾有连续三个月不发补丁的时候吗？从来没有！ * 第六个步骤是测试 测试有很多种：按照测试执行方，可以分为内部测试和外部测试；按照测试范围，可以分为模块测试和整体联调；按照测试条 件，可以分为正常操作情况测试和异常情况测试；按照测试的输入范围，可以分为全覆盖测试和抽样测试。以上都很好理解，不再解释。 总之，测试同样是项目研发中一个相当重要的步骤，对于一个大型软件，3个月到1年的外部测试都是正常的，因为永远都会又不可预料的问题存在。 完成测试后，完成验收并完成最后的一些帮助文档，整体项目才算告一段落，当然日后少不了升级，修补等等工作，只要不是想通过一锤子买卖骗钱，就要不停的跟踪软件的运营状况并持续修补升级，知道这个软件被彻底淘汰为止。 写这些步骤算不上卖弄什么，因为实话讲我手边是一本《软件工程》，在大学里这是计算机专业的必修课程，但是我知道很多程序员似乎从来都只是热衷于什么《30天精通VC》之类的，他们有些和我一样游击队出身，没有正规学过这个专业，还有一些则早就在混够学分后就把这些真正有用的东西还给了老师。 fans乱嚷嚷，混淆视听，实际上真正的技术专家很少在网上乱发帖子的，如笔者这样不知天高地厚的，其实实在是算不上什么高手，只不过看不惯这种对技术，对程序员的误解和胡说，只好挺身而出，做拨乱反正之言，也希望那些还fans们能认真想想，走到正途上，毕竟那些聪明的头脑还远远没有发挥应有的价值。沉迷于一些错误人士的coding 从程序员升级到工程师大多数象我这样对软件有浓厚兴趣的人，毕业后义无反顾地走进了企业，开始了程序员的生涯。那时，我们迷恋“大全”、“秘籍”一类的书籍，心中只有代码。当我看到一行行枯燥的代码变成了能够打电话的设备，变成了屏幕上漂亮的表格，变成了动听的音乐，成就感油然而生。我觉得自己也是一个出色的程序员了。在用户的机房中苦熬三昼夜解决软件的bug，也成了一种可以夸耀的资历。五年前的某一天，我把曾经让我兴奋自豪的大量代码和少得可怜的文档移交之后，来到了华为。这里有更多的年轻人，我如鱼得水，可以充分发挥自己的想象力。依然是代码，依然是匆匆地在纸上记下稍纵即逝的灵感（我们把它称作文档），依然是无休止地和bug作斗争。当有一天，一个新来 的同事拿着署着我的大名的文档，小心翼翼地来问我时，我发现自己好象有点不认识它了。我心里有点沮丧，再看看代码，发现文档上记录的一些灵感已面目全非。我当时不知道那位新来的同事感受如何，但我从那时起，好象意识到什么。现在来看，那时的很多事情都是事倍功半。 去年年底，我学习印度从事项目开发，学习印度的软件开发管理方法。一种久违的冲动在心底升起。我在学习过程中和印度的工程师交流过，他们言谈中透着自信。他们给我讲解正在做的软件的测试环境，给我看他们写的单元测试文档。当我看到一个软件模块的单元测试用例有三百多页时，我觉得心里很是沉重。当我第三次踏上这片土地时，我又见到了熟悉的人们，明亮的眼睛，温和的笑容，随意的穿着，风驰电掣的摩托，还有大学校园中穿着拖鞋，手抱书本的年轻人。 我也见到了我的项目经理，一个个子较高，瘦瘦的年轻人，据说刚从美国回来，已工作了五、六年。我听了心里很高兴，这回要一招一式地学两手。需求分析的时间是一个月，项目经理和我们（实际上代表客户 ）讨论了proposal中的内容，确定每一项都是需要的。然后他把模块大致划分了一下，开始进入计划中的学习阶段。每个人在学习阶段要写出功能描述的胶片，给其他人讲解，不知不觉中，项目组的所有人对项目有了整体的了解。 　他还安排了一些培训，如他们公司的软件开发模型、项目组中各角色的定义，以后及时的培训不断，只要项目组中有需求，他总是把qa或相关的人请来，培训很专业。需求分析完成后提交了一份四十多页的文档，当我看到这份英文文档中我写的部分整整齐齐地列在其中时，我的感觉很复杂，有些喜悦，但更多的是苦涩，我以前怎么就从来没有这样做过需求分析呢。 在我写文档的过程中，qa给我们培训过srs的写作模板，后来我还是不放心，让他们一个有经验的工程师写了一段，我们再琢磨着照着写。这份srs虽然是多个人合写，但风格一致，内容详实。更为可贵的是，一直到最后，这份需求分析的内容都没有改过，以至于我们没有机会走一下他们的需求更改流程。 需求分析是项目的第一阶段，第二阶段的开发时间要根据需求分析的结果来确定。当对方的首席技术官（相当于我们业务部的总体组长）来和我们讨论计划时，他们已列出了对每个 模块的代码行数的预测，可能存在的风险。根据他们公司的生产率－－300行/人月，他得出了项目第二阶段需要多少周。 我们当时就提出了异议：1）公司对该项目需求很急；2）每月300行是否太少；3）我们还有下载的源代码参考。他解释说，300行/人月是使得项目能达到他们质量标准的经验数据，考虑到有源代码参考，生产率最多不能超过350行/人月。 　当他问我们公司的生产率时，我脑袋里转了三个圈，没敢多说，大概六、七百行吧。他沉默了一会儿，然后坚定地说，我们这个计划是建立在确保质量的基础上的，我想你们到印度来开发软件，首先看中的应该是我们印度公司的质量保证。我知道你们不缺乏软件开发人员，你们为什么不选择下载的软件呢。几句话说到了我的痛处，现在国内的弟兄们还在为使用下载软件移植的产品四处奔波呢！ 　随后的开发活动有条不紊，我们老老实实地跟着做。系统测试计划、用例，概要设计，集成测试计划、用例，详细设计，单元测试计划、用例，编码，单元测试，集成测试，系统测试。一个完整的v模型开发过程，其中每个过程都有review。当我们对一些设计的方法不太明白时，项目经理给我们发来了相关的资料，我不知道他当时是怎么想的，一些基本的分析、设计方法是十年，甚至二十年前的软件工程书中就讲到的，印度每个计算机专业的人员都是必修这些内容的。而我们除了对一些具体协议的代码很熟之外，对这些常用的方法似乎一无所知。我感到一些羞愧，进城直奔书店，把他给我开列的书找了出来，晚上躺在床上，仔细研读，我仿佛突然又遇到了能给我指点迷津的良师益友。现在印度所已形成了强烈的学习风气。我回来后也推销了700多本书，这些书教我们如何用工程化的方法开发软件，是成为一个软件工程师必读的资料。 我们的项目经理的计划控制能力很强，当有什么影响到项目计划的事情发生时，如人员辞职、实验室搬家、某一模块预测不准（该模块是我们预测的），他总是采取必要的措施，减少延期，调整计划。刚开始，我们对他们每天上午11点，下午4点下楼喝咖啡还有点意见，后来也跟着喝去了，原来，喝咖啡时的交流非常丰富，从项目管理到设计方法，从技术发展到风土人情，无所不包，对我们互相之间的理解，对团队的气氛很有帮助。我们项目的qa也在适当的时候出现在我们的面前，我们对她的工作只有一些感性认识。她每次参加会议时，手里时常拿着一个check list，项目经理准备相应的资料，回答一些问题，她打着勾，或写着项目经理的解释。她给我们做培训时也很耐心，体现出很好的职业素养，我至今还在怀念她给我们的帮助。 我从事软件开发已有四个年头了，可我现在仍然不能说自己是个合格的软件工程师，更不用谈什么合格的管理者。我看到一份报道说，瑞士洛桑一权威机构把中国的科技综合竞争力从原来的第十三位调到二十多位，原因是他们调整了一些评估标准，其中有一条是中国合格工程师的可获得性非常低。想着弟兄们熬红的双眼，四处奔波升级的疲惫身影，我有一个强烈的愿望：快把我们自己升级成合格的工程师吧! ================================================ FILE: docs/framework/split.md ================================================ # 架构拆分的代价之一二三 架构拆分可以用来节省人们执行事务生命周期活动所需的时间，这就是传说中购买“寸光阴”的方式，也是传说中的“银弹”。不过要获得这个“银弹”，也不是没有代价的。但既然节省的是时间，而时间又是无价的，因此人们也愿意采用架构拆分，哪怕要因此而付出巨大的代价。因为相比“无价”而言，“巨大的代价”也是可以接受的。 可是架构也不是万能的。那么在采用架构拆分之前，了解这些“巨大的代价” 是非常有必要的，可以帮助我们认清架构的局限，厘清对作架构拆分的人的要求。因为不合理的架构拆分，反而会浪费更多的时间；哪怕是合理的架构拆分， 但是如果不能够重新组装回原有生命周期的话，也无法达到节省时间的目的， 最终也会导致浪费时间。 没有上下文的同学看到这里，可能会对“架构”和“架构拆分”这两个词有迷惑。一会儿说“架构”，一会儿说“架构拆分”，那么架构和架构拆分究竟是什么关系呢？架构拆分是实现架构的一个手段，是架构落地的方法，是架构的核心生命周期活动。 本文尝试描述架构拆分所产生的代价。观点不一定正确，也不一定适合所有人，仅供参考。如能引发读者的哪怕一丝思考，那就幸甚至哉! ## 空间拆分的代价 架构拆分的本质就是空间拆分，这在之前的文章中已经探讨过了。一个事物在进行空间拆分之后，必然会变成一个一个不同的个体。每个个体从原有事物中独立出来之后，会形成各自一个一个的独立生命周期。 既然是独立生命周期，那么就意味着对原有事物做架构拆分的人 ---- 也就是 架构拆分主体，他必须要管理这些新出现的一个一个独立生命周期。也可以看到，有了架构才会出现管理，做管理的人不懂架构是不行的。 但是大部分做架构拆分的人，都没有意识到这一点。大多数人只注意获得架构拆分的好处，而忽略了拆分出来的子生命周期也是需要管理的，只有管理好子生命周期才能够得到架构拆分的好处。 比如做技术的人，在企业经过一段时间的锻炼之后，掌握了某种技术。往往他们就会开始去思考如何把技术变现，以求自身利益最大化。比较流行的方式当然就是去创业。 而创业则等于形成一种新的社会分工，也就是对社会的一个架构拆分。拆分的结果，形成了一个新的公司，也就形成了一个新的生命周期。这个新的生命周期必须先要健康的运转起来，技术才能够派得上用场。而要推动运转这个新生命周期，则需要先注册公司，准备资金，需要和税务、法务打交道，要招人，要社保......。这个时候，已经累的够呛了，但是离运用想要变现的技术还远着呢。 公司成立好了，接下来为了要形成订单，还需要先找到客户。为此，还需要先把技术形成产品，以便形成商品进行销售。为了形成产品，则需要巨大的投入。为了找到客户，则需要巨大的营销成本。而这些工作，光只是一个人是不够的，还需要先做自身的架构拆分后，再招人来帮忙。如果没有受过架构方面的训练，这个架构拆分本身就很容易产生新的问题。即便成功的对自身工作 进行架构拆分之后，则又形成了新的生命周期，新招来的人则会负责推进这些生命周期。那么这些新招来的人当然也是需要管理的，他们的生命周期也是需要推进的，需要与他们签订合同，发工资，交社保，等等等等。 回头再看看，在企业内部掌握并运用这个技术，只需要关心这个技术本身的生命周期就足够，事情简单太多了。 很多技术创业者满腔热情，只看到了自己的技术的好处，没有注意到架构拆分后，要管理子生命周期的成本，最后往往因此不堪重负，最终导致失败。 所以做架构拆分的人，他必须要管理拆分之后子生命周期的全生命周期，才能够享受到架构拆分的好处，大家对此要有清醒的认识。他必须要了解拆分后子生命周期的全生命周期活动，而不是仅仅关心自己想要的其中那一个活动。 所以做架构拆分时，要谋定而后动：只有管理好每个独立子生命周期的成本自 己可以承受，并且有应对的办法，才可以做架构拆分。 再举个例子，比如有些人不想挤地铁，想要买车代步，那么这也是在做自己的空间架构拆分。这也就意味着他要付出管理这辆车全生命周期的代价：例如定期需要保险、保养、年检等，坏了还需要修理，没油了要找加油站加油，或者是电车的话，没电了还要充电，等等之类，必须要保持这辆车的健康。何况附带着还有练习开车，获得驾照，并维持驾照的生命周期。而且还得自己管理开车的生命周期，比如还需要自己认路，还要注意不要违章，要时刻小心自己的生命安全，等等等等。虽然人们对车的实际需求只是开车上路，到达目的地而已，但是只有把这些子生命周期的全生命周期活动管理好之后，才能够达成这一点。否则的话，车子非但不能够产生帮助，反而会带来麻烦，影响用车的目的达成。比如车况不好，经常半路抛锚，那谁也不敢开了。 当然还可以进一步做架构拆分，请一个人来开车，让他来执行并推进这些生命周期。这又意味着进一步的成本的付出，因为还要新增负担管理司机的生命周期，需要支付工资等等。 不清楚这些新出现的子生命周期的代价，贸然的去买车，那么也只会给自己带来麻烦。这种情况之下，坐地铁反而是经济实惠的。 对于软件开发工作者而言，在对代码做架构拆分时，同样也必须管理好架构拆分后所形成新组件的全生命周期。比如把一个应用拆分为两个子应用后，那么两个子应用的全生命周期都要管理起来，哪怕只是需要子应用的某一个服务。 当然，人们常用“非功能性需求”来说明这一点，但是往往得不到重视，也 不够全面。或许是因为“非功能”这一概念仍然仅仅是附带在“功能”上，没脱离“功能”的阴影。因此我也很少用“非功能性需求”这个词汇，因为无法用“功能”去定义它。 只有回归到软件自身的生命周期上，才能够正确理解所谓的“非功能性需求”：其关注点并不在于“功能”，而是在于保障功能自身的健康，并保证功能可以被正常访问到。因为想要得到子应用的某一个功能 ---- 这只是充分条件，只是决定了该子应用具备独立生存的资格；但是这还远远不够，先必须得让这个子应用能够正常生存下去 ---- 这是必要条件，因为只有健康活着才能够被访问。 只有充分条件和必要条件同时都满足，这个子应用的生命周期才算是完整了， 该子应用的功能才能够正常运转并接受访问。 因此，所谓的“非功能性需求”，指的就是软件得以正常生存并对外提供服务的必要条件。详细展开来，该必要条件主要体现在以下两个方面。 首先，承载所需功能的软件自身必须要健康才行。 达到这一点，需要从计算机硬件的边界完整考虑其生命周期，如机房、电源、计算机资源、网络资源等硬件资源的生命周期；还需要从计算机软件的边界完整考虑其生命周期，如启动、到服务、到关闭等。二者都需要完整考虑，也都需要进行监控。这是软件功能得以正常运行的必要条件 ---- 软件先要能够健康的活着，并且能够被用户访问到。 ## 首先，承载所需功能的软件自身必须要健康才行。 达到这一点，需要从计算机硬件的边界完整考虑其生命周期，如机房、电源、计算机资源、网络资源等硬件资源的生命周期；还需要从计算机软件的边界完整考虑其生命周期，如启动、到服务、到关闭等。二者都需要完整考虑，也都需要进行监控。这是软件功能得以正常运行的必要条件 ---- 软件先要能够健康的活着，并且能够被用户访问到。 ## 其次，软件是提供给用户访问的，要保证合法用户的合法访问可以正常到达软件。 要满足这一点，需要解决以下两方面的问题： * 一、识别合法的访问。 比如哪些是合法用户的访问，哪些是非法用户的访问？合法用户的访问哪些是 合法的，哪些是非法的？合法的访问中，哪些是正常的访问，哪些是非正常的访问？等等。只有屏蔽了不合法用户的访问，也排除了合法用户的非正常访问， 那么合法用户的合法访问才会不受影响。这些问题往往被称作安全需求。 * 二、 满足访问量要求。该软件所在机器可接受的访问量范围是多少？该软件在该机 器上可接受的访问量是多少？访问量超过正常的范围怎么处理？承载用户对软件访问的网络设备带宽是多少，能承受多大的访问量？超过这个范围怎么处理？等等。当超过流量范围时，是否要优先确保已经在正常访问的那些用户访问？如何确保或者根据业务的需求，优先保证高价值用户的访问？等等。这些问题往往被称作容量需求。这是软件功能能够被正常访问的必要条件 ---- 对访问 安全问题和访问容量问题的合适处理，保障了软件访问的健康。 保障软件自身的健康，基本上是技术问题居多，容易被技术人员识别出来。 但是保障软件访问的健康，基本上都是业务问题，也难怪容易被技术人员忽略， 其优先级也容易被排的很低。因此，提升技术人员的业务意识至关重要。 所以，软件架构分拆并不是简单的把代码拆分出来。架构师不考虑清楚分拆出来软件的完整生命周期，仅关注所需的功能，那么架构拆分只会带来无尽的麻烦，浪费更多的时间，而得不到任何好处。 ## 连续的流程所产⽣生的单点 架构拆分之后，会在空间上形成不同的个体。原本只是在一个个体上完成的事情，则变成了需要通过多个不同的个体组合来完成。为了组合这些个体完成原有的事务，则需要按一定的顺序组合并遍历这些个体，这就形成了一个流程。这也意味着需要在空间上按照时间顺序一个一个地访问这些个体，才能够得到与原有事务相同的结果。 这么做是有代价的。当流程中的任何一个个体出现问题，这个事务就会整体出现问题，这就是单点故障。比方说工厂中的流水线，其中一个节点失效或者堵塞，整个流程都会受到影响而停止下来。因此，所拆分出来的个体越多，所形成的流程则越长，那么单个点失败出现的可能性就越大。 在数学上，这种情况称之为串联。串联的稳定性是这么计算的，如果每个个体稳定性为 N%，那么整个链条的稳定性是该链条上所有个体稳定性的乘积。 这个怎么来理解呢？ ![split](./../img/split1.png) 比如上图，一个事务本身的稳定性是 90%，在拆分为两个个体组合来完成之后，形成了一个链条，这个链条上有两个个体。假设拆分后，每个个体的稳定性保持不变，每个个体的稳定性仍然是 90%，那么拆分之后，整体稳定性变成了 90%×90%=81%，稳定性反而下降了 9 个百分点。拆出来的个体越多，流程越长，稳定性下降得就越明显。 这就是架构拆分后，单点的代价。所有做架构拆分的人 ---- 人们称他们为架构师，都必须要意识到这个代价。 同时，如果没有注意管理好代价 1 所带来的代价，那么在把原有事务拆分成为串联的子节点后，每个子节点的稳定性反而会下降。结合代价 2 导致的串联引入，其架构拆分的最终结果，也就可想而知了。 但是单点既是代价，也是一个机会。比如在现实生活中，一个城市只有一家餐馆，那么没有哪个人愿意只依赖这家餐馆，而放弃在家做饭，因为这一个餐馆就是一个单点。如果这个餐馆过忙了，或者歇业，大家就没处吃饭了。但是如果大家都很忙，没空自己做饭的时候，那么这个单点就成了一个商机，餐馆也会变得越来越多。一旦有了多个可备份的选择，找餐馆吃饭也就变得容易了，那么越来越多的人也就敢于放弃在家做饭，在餐馆就餐。这就是所谓的并联。 ![split](./../img/split2.png) 这一特性，也可以在数学上表示出来。如上例，如果把相同的组件并联起 来，只有在两个并联组件同时都失败后，才会导致整体失败，因此其失败的可能性为 (1-90%)×(1-90%)。其稳定性则为 1-(1-90%)×(1-90%)=99.99%，提升了近 10 个百分点。由此可见，并联可以提升整个系统的稳定性。因此，人们常用这个办法来改善架构拆分后系统稳定性会降低这一弱点。 再比如古人远行，庄子云：“适莽苍者，三湌而反，腹犹果然；适百里者宿舂粮；适千里者三月聚粮”。走的越远，所需储备的粮食就越多。这是一个人独立完成的事务，在没有作架构拆分的情况下，只需要提升个体的稳定性即可。 有了马匹等代步工具的丰富，有了沿路餐馆的丰富，交易的发达，人们不再依靠自己储备的粮食就可以出门，且在相同时间内可以行更远的路，节省了时间。 行步通过采用代步工具，形成了架构拆分，饮食也形成了架构拆分，且这些拆 分都有并联来保障其稳定性。这就是社会的架构拆分所带来的好处，节省了每一个人的时间，且利用了并联，避免了拆分后形成串联所带来稳定性下降这一坏处。 再比如代步工具已经进化到现代的汽车，从人的油门一踩，到轮子前行，其中经历了非常多的串行机构。整个链条上的任何一个机构失效，都会导致传动的失败。方向盘一转、刹车一踩，也是一样的。这些是属于没有可能并联的地方，因此必须要把传动链条上的每一个机构都做的非常的稳定、健壮，所以这些机构都非常的粗壮，用的都是非常好的材料，这也是避免串行缺点的一个处理方法。通过把串行节点上的每一个节点都做的异常的健壮、可靠，也可以提升整体稳定性，以降低架构拆分所带来的负面影响。反观自行车，类似的串联机构则没那么粗壮，一方面速度慢、自重轻，另一方面串联的机构数量也更少。 对于软件的架构拆分，每个拆分出来的子软件也要形成并联来保障整体的稳定性，也就是集群的方式，这是同样的道理。如果架构师在拆分时不了解这一限制，也是会付出极大的代价的。可是为何不通过提升每个节点的健壮性来提升呢？这个问题留给大家思考。 ## 组合的代价 一个事务拆分成为多个子生命周期之后，最终还是需要把这些子生命周期整合成为原事务的，这样才算完成了架构拆分。 许多人作架构拆分时，只考虑拆分之后的好处，却在组合回原事务时，忘记了拆分的初衷。轻则导致完成原事务变得更加困难，重则导致原事务无法完成，导致原事务的失败。比如代价 1 和代价 2 的处理不当，都能够使得原事务无法得到正确还原。 而且每当一个新个体拆分出来之后，这个新的个体就有了它自身的生命周期。负责这个新个体的人，就会有自己的诉求，有他自己的利益。作架构拆分 的人，必须在作架构拆分时就充分考虑好拆分出来的每个个体的利益。用通俗的话说，在做架构拆分时就要对这些拆分出来的子个体设定好 KPI，明确这些子个体的目标，使得这些子个体不得违反整个架构目标，要能够顺利组合为原有事务，共同为完成原有事务作出贡献。 对于软件也是同样的，拆分时就要想好怎么把拆分出来的软件组合成原有的系统。这些软件在拆分前处在同一个应用的时候，它们共享计算机的内存、 CPU、网络、磁盘等硬件，操作系统等软件。一旦分离开来，每个应用就有了自己的计算机资源，有了自己的对外访问，有了自己的利益，有了自己的计算机硬件和软件的边界。虽然这些刚分离出来的软件暂时仅仅服务于原来的业务， 但是随着时间的变化，这些软件一定会有自己独立的业务出现，这些软件的拥 有者一定会有自己的诉求。因为一个事物只要出生，它必然要长大，这是不可避免的。 所以在组合这些软件时要明白，这是针对不同个体的访问，需要考虑跨越网络，需要考虑这些个体自身利益，比如它们的流量分配、访问边界、安全与隔离等等。如果仅仅是简单粗暴的组合，忽视不同应用各自的利益，一定会带来极大的沟通成本，甚至会导致组合的失败。 总结一下，也就是说架构拆分是有成本的，哪怕它能够节省时间。如果不合适的架构拆分，或者架构拆分时考虑不周，则会导致人们整体花费更多的时间成本，那么这个架构拆分则是不值得的。只有能够管理好架构拆分的时间成本，并能够让架构拆分后的时间成本低于原有的时间成本，这个拆分才是值得的。所以架构拆分也不是想怎么拆就怎么拆，也不是拆完了就结束了，架构拆分有其自身的规律。架构师必须要先考虑好架构拆分的代价再进行拆分，这样就不容易掉进坑里。 ## 后记 以上略略谈了一下架构拆分的代价，仅仅算是抛砖引玉。对于这个问题，每个人都应该好好思考一下，或许不仅仅只是以上三点。 纵观当今深度学习框架市场，各种框架如雨后春笋般崛起，碎片化程度堪比如今的安卓终端系统。为了了解开发者对深度学习和人工智能的看法，选出开发者最喜爱的深度学习平台，并探索未来深度学习平台的发展方向。我们在此邀请您作为代表，提供有价值的意见。 ================================================ FILE: docs/go/go_base.md ================================================ # Go基础语法 ## 模块一：Go 语言特性 ##### 统一思想- 12 factors I. 基准代码 一份基准代码，多份部署 II. 依赖 显式声明依赖关系 III. 配置 在环境中存储配置 IV. 后端服务 把后端服务当作附加资源 V. 构建，发布，运行 严格分离构建和运行 VI. 进程 以一个或多个无状态进程运行应用 ``` VII. 端口绑定 通过端口绑定提供服务 VIII. 并发 通过进程模型进行扩展 IX. 易处理 快速启动和优雅终止可最大化健壮性 X. 开发环境与线上环境等价 尽可能的保持开发，预发布，线上环境相同 XI. 日志 把日志当作事件流 XII. 管理进程 后台管理任务当作一次性进程运行 ``` ``` ``` # 目录 1. 为什么需要另外一种语言？ 2. Go 语言编译环境设置 3. 控制结构 4. Go 语言常用数据结构 5. Go 语言函数调用 6. 常用语法 7. 多线程 - 深入理解 channel - 基于 channel 编写一个生产者消费者程序 #### 1. 为什么需要 Go 语言 ``` ``` ###### Less is exponentially more - Rob Pike, Go Designer ###### Do Less, Enable More - Russ Cox, Go Tech Lead ##### Go 语言的原则 ##### 为什么需要 Go 语言 - 其他编程语言的弊端。 - 硬件发展速度远远超过软件。 - C 语言等原生语言缺乏好的依赖管理 (依赖头文件）。 - Java 和 C++ 等语言过于笨重。 - 系统语言对垃圾回收和并行计算等基础功能缺乏支持。 - 对多核计算机缺乏支持。 - Go 语言是一个可以编译高效，支持高并发的，面向垃圾回收的全新语言。 - 秒级完成大型程序的单节点编译。 - 依赖管理清晰。 - 不支持继承，程序员无需花费精力定义不同类型之间的关系。 - 支持垃圾回收，支持并发执行，支持多线程通讯。 - 对多核计算机支持友好。 ##### Go 语言不支持的特性 - 不支持函数重载和操作符重载 - 为了避免在 C/C++ 开发中的一些 Bug 和混乱，不支持隐式转换 - 支持接口抽象，不支持继承 - 不支持动态加载代码 - 不支持动态链接库 - 通过 recover 和 panic 来替代异常机制 - 不支持断言 - 不支持静态变量 如果你没做过其他语言的开发， 那么恭喜，以上大部分复杂的问题，在 Go 语言里不存在，你也无需关心。 ##### Go 语言特性衍生来源 ``` golang ``` ``` Java Java, c# C 语言 ``` ``` Limbo ``` ``` Docker Kubernetes Istio ``` ``` Javascript, Ruby 以及其 他动态语言 ``` ``` 包定义 ``` ``` Unix ``` ``` Plan 9 ``` CSP多线程模型 (Communication Sequential Process) ``` 多态支持 ``` ``` 接口抽象 基本语法和数据结构 ``` ``` 应用 ``` ``` 参考：The way to go ``` #### 2. Go 语言环境搭建 ##### 下载 Go - Go 安装文件以及源代码 https://golang.google.cn/dl/ - 下载对应平台的二进制文件并安装 - 环境变量 - GOROOT l go的安装目录 - GOPATH l src：存放源代码 l pkg：存放依赖包 l bin：存放可执行文件 - 其他常用变量 l GOOS，GOARCH，GOPROXY l 国内用户建议设置 goproxy：export GOPROXY=https://goproxy.cn ##### IDE 设置（VS Code） - 下载并安装 Visual Studio Code https://code.visualstudio.com/download - 安装 Go 语言插件 https://marketplace.visualstudio.com/items?itemName=golang.go - 其他可选项 - Intellj goland，收费软件 - vim，sublime等 ``` bug start a bug report build compile packages and dependencies clean remove object files and cached files doc show documentation for package or symbol env print Go environment information fix update packages to use new APIs fmt gofmt (reformat) package sources generate generate Go files by processing source get add dependencies to current module and install them ``` ##### 一些基本命令 ``` install compile and install packages and dependencies list list packages or modules mod module maintenance run compile and run Go program test test packages tool run specified go tool version print Go version vet report likely mistakes in packages ``` ##### 一些基本命令 ##### Go build - Go 语言不支持动态链接，因此编译时会将所有依赖编译进同一个二进制文件。 - 指定输出目录。 - go build –o bin/mybinary. - 常用环境变量设置编译操作系统和 CPU 架构。 - GOOS=linux GOARCH=amd64 go build - 全支持列表。 - $GOROOT/src/go/build/syslist.go ##### Go test Go 语言原生自带测试 import "testing" func TestIncrease(t *testing.T) { t.Log("Start testing") increase( 1 , 2 ) } go test ./... -v 运行测试 go test 命令扫描所有*_test.go为结尾的文件，惯例是将测试代码与正式代码放在同目录， 如 foo.go 的测试代码一般写在 foo_test.go ##### Go vet 代码静态检查，发现可能的 bug 或者可疑的构造 - Print-format 错误，检查类型不匹配的print str := “hello world!” fmt.Printf("%d\n", str) - Boolean 错误，检查一直为 true、false 或者冗余的表达式 - Range 循环，比如如下代码主协程会先退出，go routine无法被执行 - Unreachable的代码，如 return 之后的代码 - 其他错误，比如变量自赋值，error 检查滞后等 ``` fmt.Println(i != 0 || i != 1 ) ``` ``` words := []string{"foo", "bar", "baz"} for _, word := range words { go func() { fmt.Println(word). }() } ``` ``` res, err := http.Get("https://www.spreadsheetdb.io/") defer res.Body.Close() if err != nil { log.Fatal(err) } ``` ##### 代码版本控制 - 下载并安装 Git Command Line - https://git-scm.com/downloads - Github - 本课程示例代码均上传在 https://github.com/cncamp/golang - 创建代码目录 - mkdir –p $GOPATH/src/github.com/cncamp - cd $GOPATH/src/github.com/cncamp - 代码下载 - git clone https://github.com/cncamp/golang.git - 修改代码 - 上传代码 - git add filename - git commit –m 'change logs' - git push ##### Golang playground 官方 playground https://play.golang.org/ 可直接编写和运行 Go 语言程序 国内可直接访问的 playground https://goplay.tools/ #### 3. 控制结构 ##### If - 基本形式 if condition1 { // do something } else if condition2 { // do something else } else { // catch-all or default } - if 的简短语句 - 同 for 一样， if 语句可以在条件表达式前执行一个简单的语句。 if v := x - 100 ; v < 0 { return v } ##### switch switch var1 { case val1: //空分支 case val2: fallthrough //执行case3中的f() case val3: f() default: //默认分支 ... } ##### For Go 只有一种循环结构：for 循环。 - 计入计数器的循环 for 初始化语句; 条件语句; 修饰语句{} for i := 0 ; i < 10 ; i++ { sum += i } - 初始化语句和后置语句是可选的，此场景与 while 等价（Go 语言不支持 while） for ; sum < 1000 ; { sum += sum } - 无限循环 for { ifcondition1 { break } } ##### for-range 遍历数组，切片，字符串，Map 等 for index, char := range myString { ... } for key, value := range MyMap { ... } for index, value := range MyArray { ... } 需要注意：如果 for range 遍历指针数组，则 value 取出的指 针地址为原指针地址的拷贝。 #### 4. 常用数据结构 ##### 变量与常量 - 常量 ``` const identifier type ``` - 变量 ``` var identifier type ``` ##### 变量定义 - 变量 - var 语句用于声明一个变量列表，跟函数的参数列表一样，类型在最后。 - var c, python, java bool - 变量的初始化 - 变量声明可以包含初始值，每个变量对应一个。 - 如果初始化值已存在，则可以省略类型；变量会从初始值中获得类型。 - var i, j int = 1, 2 - 短变量声明 - 在函数中，简洁赋值语句 := 可在类型明确的地方代替 var 声明。 - 函数外的每个语句都必须以关键字开始（var, func 等等），因此 := 结构不能在函数外使用。 - c, python, java := true, false, "no!" ##### 类型转换与推导 - 类型转换 - 表达式 T(v) 将值 v 转换为类型 T。 l 一些关于数值的转换： l var i int = 42 l var f float64 = float64(i) l var u uint = uint(f) l 或者，更加简单的形式： l i := 42 l f := float64(i) l u := uint(f) - 类型推导 - 在声明一个变量而不指定其类型时（即使用不带类型的 := 语法或 var = 表达式语法），变量的类型由右值推导得出。 l var i int l j := i // j 也是一个 int ##### 数组 - 相同类型且长度固定连续内存片段 - 以编号访问每个元素 - 定义方法 - var identifier [len]type - 示例 - myArray := [3]int{1,2,3} ##### 切片(slice) - 切片是对数组一个连续片段的引用 - 数组定义中不指定长度即为切片 - var identifier []type - 切片在未初始化之前默认为nil， 长度为 0 - 常用方法 ``` func main() { myArray := [ 5 ]int{ 1 , 2 , 3 , 4 , 5 } mySlice := myArray[ 1 : 3 ] fmt.Printf("mySlice %+v\n", mySlice) fullSlice := myArray[:] remove3rdItem := deleteItem(fullSlice, 2 ) fmt.Printf("remove3rdItem %+v\n", remove3rdItem) } func deleteItem(slice []int, index int) []int { return append(slice[:index], slice[index+ 1 :]...) } ``` ##### Make 和 New - New 返回指针地址 - Make 返回第一个元素，可预设内存空间，避免未来的内存拷贝 - 示例 mySlice1 := new([]int) mySlice2 := make([]int, 0) mySlice3 := make([]int, 10) mySlice4 := make([]int, 10, 20) ##### 关于切片的常见问题 - 切片是连续内存并且可以动态扩展，由此引发的问题？ a := []int b := []int{1,2,3} c := a a = append(b, 1) - 修改切片的值？ mySlice := []int{ 10 , 20 , 30 , 40 , 50 } for _, value := range mySlice { value *= 2 } fmt.Printf("mySlice %+v\n", mySlice) for index, _ := range mySlice { mySlice[index] *= 2 } fmt.Printf("mySlice %+v\n", mySlice) ##### Map - 声明方法 - var map1 map[keytype]valuetype - 示例 myMap := make(map[string]string, 10 ) myMap["a"] = "b" myFuncMap := map[string]func() int{ "funcA": func() int { return 1 }, } fmt.Println(myFuncMap) f := myFuncMap["funcA"] fmt.Println(f()) ##### 访问 Map 元素 按 Key 取值 value, exists := myMap["a"] if exists { println(value) } 遍历 Map for k, v := range myMap { println(k, v) } ##### 结构体和指针 - 通过 type ... struct 关键字自定义结构体 - Go 语言支持指针，但不支持指针运算 - 指针变量的值为内存地址 - 未赋值的指针为 nil type MyType struct { Name string } func printMyType(t *MyType){ println(t.Name) } func main(){ t := MyType{Name: "test"} printMyType(&t) } ##### 结构体标签 - 结构体中的字段除了有名字和类型外，还可以有一个可选的标签（tag） - 使用场景：Kubernetes APIServer 对所有资源的定义都用 Json tag 和 protoBuff tag - NodeName string `json:"nodeName,omitempty" protobuf:"bytes,10,opt,name=nodeName"` type MyType struct { ``` Name string `json:"name"` ``` } func main() { ``` mt := MyType{Name: "test"} myType := reflect.TypeOf(mt) name := myType.Field( 0 ) tag := name.Tag.Get("json") println(tag) ``` } ##### 类型别名 ``` // Service Type string describes ingress methods for a service type ServiceType string const (// ServiceTypeClusterIP means a service will only be accessible inside the // cluster, via the ClusterIP. ServiceTypeClusterIP ServiceType = "ClusterIP" // ServiceTypeNodePort means a service will be exposed on one port of // every node, in addition to 'ClusterIP' type. ServiceTypeNodePort ServiceType = "NodePort" // ServiceTypeLoadBalancer means a service will be exposed via an // external load balancer (if the cloud provider supports it), in addition// to 'NodePort' type. ServiceTypeLoadBalancer ServiceType = "LoadBalancer" // // an external name that ServiceTypeExternalNamekubednsmeans a service consists of only a reference toor equivalent will return as a CNAME // record, with no exposing or proxying of any pods involved. ServiceTypeExternalName ServiceType = "ExternalName" ) ``` #### 5. 函数 ##### Main 函数 - 每个 Go 语言程序都应该有个 main package - Main package 里的 main 函数是 Go 语言程序入口 package main func main() { args := os.Args if len(args) != 0 { println("Do not accept any argument") os.Exit( 1 ) } println("Hello world") } ##### 参数解析 - 请注意 main 函数与其他语言不同，没有类似 java 的 []string args 参数 - Go 语言如何传入参数呢？ - 方法 1 ： l fmt.Println("os args is:", os.Args) - 方法 2 ： l name := flag.String("name", "world", "specify the name you want to say hi") l flag.Parse() ##### Init 函数 - Init 函数：会在包初始化时运行 - 谨慎使用 init 函数 - 当多个依赖项目引用统一项目，且被引用项目的初始化在 init 中完成，并且不可重复运行时，会导 致启动错误 package main var myVariable = 0 func init() { myVariable = 1 } ##### 返回值 - 多值返回 - 函数可以返回任意数量的返回值 - 命名返回值 - Go 的返回值可被命名，它们会被视作定义在函数顶部的变量。 - 返回值的名称应当具有一定的意义，它可以作为文档使用。 - 没有参数的 return 语句返回已命名的返回值。也就是直接返回。 - 调用者忽略部分返回值 ``` result, _ = strconv.Atoi(origStr) ``` ##### 传递变长参数 Go 语言中的可变长参数允许调用方传递任意多个相同类型的参数 - 函数定义 func append(slice []Type, elems ...Type) []Type - 调用方法 myArray := []string{} myArray = append(myArray, "a","b","c") ##### 内置函数 ``` 函数名 作用 close 管道关闭 len, cap 返回数组、切片，Map 的长度或容量 new, make 内存分配 copy, append 操作切片 panic, recover 错误处理 print, println 打印 complex, real, imag 操作复数 ``` ##### 回调函数(Callback) ``` 示例： func main() { DoOperation( 1 , increase) DoOperation( 1 , decrease) } func increase(a, b int) { println(“increase result is:”, a+b) } func DoOperation(y int, f func(int, int)) { f(y, 1 ) } func decrease(a, b int) { println("decrease result is:", a-b) } ``` - 函数作为参数传入其它函数，并在其他函数内部调用执行 - strings.IndexFunc(line, unicode.IsSpace) - Kubernetes controller的leaderelection ##### 闭包 - 匿名函数 - 不能独立存在 - 可以赋值给其他变量 l x:= func(){} - 可以直接调用 l func(x,y int){println(x+y)}(1,2) - 可作为函数返回值 l func Add() (func(b int) int) - 使用场景 defer func() { if r := recover(); r != nil { println(“recovered in FuncX”) } }() ##### 方法 - 方法：作用在接收者上的函数 - func (recv receiver_type) methodName(parameter_list) (return_value_list) - 使用场景 - 很多场景下，函数需要的上下文可以保存在receiver属性中，通过定义 receiver 的方法，该方法可以直接 访问 receiver 属性，减少参数传递需求 // StartTLS starts TLS on a server from NewUnstartedServer. func (s *Server) StartTLS() { if s.URL != “” { panic(“Server already started”) } if s.client == nil { s.client = &http.Client{Transport: &http.Transport{}} } ##### 传值还是传指针 - Go 语言只有一种规则-传值 - 函数内修改参数的值不会影响函数外原始变量的值 - 可以传递指针参数将变量地址传递给调用函数，Go 语言会 ``` 复制该指针作为函数内的地址，但指向同一地址 ``` - 思考：当我们写代码的时候，函数的参数传递应该用struct ``` 还是pointer？ ``` ##### 接口 - 接口定义一组方法集合 type IF interface { Method1(param_list) return_type } - 适用场景：Kubernetes 中有大量的接口抽象和多种实现 - Struct 无需显示声明实现 interface，只需直接实现方法 - Struct 除实现 interface 定义的接口外，还可以有额外的方法 - 一个类型可实现多个接口（Go 语言的多重继承） - Go 语言中接口不接受属性定义 - 接口可以嵌套其他接口 ##### 接口 type IF interface { getName() string } type Human struct { firstName, lastName string } func (h *Human) getName() string { return h.firstName + "," + h.lastName } type Car struct { factory, model string } func (c *Car) getName() string { return c.factory + "-" + c.model } ``` func main() { interfaces := []IF{} h := new(Human) h.firstName = "first" h.lastName = "last" interfaces = append(interfaces, h) c := new(Car) c.factory = "benz" c.model = "s" interfaces = append(interfaces, c) for _, f := range interfaces { fmt.Println(f.getName()) } } ``` ##### 注意事项 - Interface 是可能为 nil 的，所以针对 interface 的使用一定要预 先判空，否则会引起程序 crash(nil panic) - Struct 初始化意味着空间分配，对 struct 的引用不会出现空指针 ##### 反射机制 - reflect.TypeOf ()返回被检查对象的类型 - reflect.ValueOf()返回被检查对象的值 - 示例 myMap := make(map[string]string, 10 ) myMap["a"] = "b" t := reflect.TypeOf(myMap) fmt.Println("type:", t) v := reflect.ValueOf(myMap) fmt.Println("value:", v) ##### 基于 struct 的反射 // struct myStruct:= T{A: "a"} v1:= reflect.ValueOf(myStruct) fori := 0 ; i < v1.NumField(); i++ { fmt.Printf("Field %d: %v\n", i, v1.Field(i)) } fori := 0 ; i < v1.NumMethod(); i++ { fmt.Printf("Method %d: %v\n", i, v1.Method(i)) } // 需要注意 receive是struct还是指针 result:= v1.Method( 0 ).Call(nil) fmt.Println("result:", result) ##### Go 语言中的面向对象编程 - 可见性控制 - public - 常量、变量、类型、接口、结构、函数等的名称大写 - private - 非大写就只能在包内使用 - 继承 - 通过组合实现，内嵌一个或多个 struct - 多态 - 通过接口实现，通过接口定义方法集，编写多套实现 ##### Json 编解码 - Unmarshal: 从 string 转换至 struct func unmarshal2Struct(humanStr string)Human { h := Human{} err := json.Unmarshal([]byte(humanStr), &h) if err != nil { println(err) } return h } - Marshal: 从 struct 转换至 string func marshal2JsonString(h Human) string { h.Age = 30 updatedBytes, err := json.Marshal(&h) if err != nil { println(err) } return string(updatedBytes) } ##### Json 编解码 - json 包使用 map[string]interface{} 和 []interface{} 类型保存任意对象 - 可通过如下逻辑解析任意 json var obj interface{} err := json.Unmarshal([]byte(humanStr), &obj) objMap, ok := obj.(map[string]interface{}) for k, v := range objMap { switch value := v.(type) { case string: fmt.Printf("type of %s is string, value is %v\n", k, value) case interface{}: fmt.Printf("type of %s is interface{}, value is %v\n", k, value) default: fmt.Printf("type of %s is wrong, value is %v\n", k, value) } } #### 6. 常用语法 ##### 错误处理 - Go 语言无内置 exceptio 机制，只提供 error接口供定义错误 typeerror interface { Error() string } - 可通过 errors.New 或fmt.Errorf 创建新的error - var errNotFounderror = errors.New("NotFound") - 通常应用程序对 error 的处理大部分是判断error 是否为 nil 如需将 error归类，通常交给应用程序自定义，比如 kubernetes 自定义了与 apiserver 交互的不同类型错误 type StatusErrorstruct { ErrStatus metav1.Status } var _ error = &StatusError{} // Error implements the Error interface. func (e *StatusError) Error() string { return e.ErrStatus.Message } ##### defer - 函数返回之前执行某个语句或函数 - 等同于 Java 和 C# 的 finally - 常见的 defer 使用场景：记得关闭你打开的资源 - defer file.Close() - defer mu.Unlock() - defer println("") ##### Panic 和 recover - panic: 可在系统出现不可恢复错误时主动调用 panic, panic 会使当前线程直接 crash - defer: 保证执行并把控制权交还给接收到 panic 的函数调用者 - recover: 函数从 panic 或 错误场景中恢复 defer func() { fmt.Println("defer func is called") if err := recover(); err != nil { fmt.Println(err) } }() panic("a panic is triggered") #### 7. 多线程 ##### 并发和并行 - 并发（concurrency） - 两个或多个事件在同一时间间隔发生 - 并行（parallellism） - 两个或者多个事件在同一时刻发生 ``` 线程 1 线程 2 线程 1 线程 2 线程 3 ``` ``` 线程 1 ``` ``` 线程 2 ``` ``` 线程 3 ``` ##### 协程 - 进程： - 分配系统资源（CPU 时间、内存等）基本单位 - 有独立的内存空间，切换开销大 - 线程：进程的一个执行流，是 CPU 调度并能独立运行的的基本单位 - 同一进程中的多线程共享内存空间，线程切换代价小 - 多线程通信方便 - 从内核层面来看线程其实也是一种特殊的进程，它跟父进程共享了打开的文件和文件系统信息，共 享了地址空间和信号处理函数 - 协程 - Go 语言中的轻量级线程实现 - Golang 在 runtime、系统调用等多方面对 goroutine 调度进行了封装和处理，当遇到长时间执行 或者进行系统调用时，会主动把当前 goroutine 的 CPU (P) 转让出去，让其他 goroutine 能被调度 并执行，也就是 Golang 从语言层面支持了协程 ##### Communicating Sequential Process - CSP - 描述两个独立的并发实体通过共享的通讯 channel 进行通信的并发模型。 - Go 协程 goroutine - 是一种轻量线程，它不是操作系统的线程，而是将一个操作系统线程分段使用，通过调度器实现协 作式调度。 - 是一种绿色线程，微线程，它与 Coroutine 协程也有区别，能够在发现堵塞后启动新的微线程。 - 通道 channel - 类似 Unix 的 Pipe，用于协程之间通讯和同步。 - 协程之间虽然解耦，但是它们和 Channel 有着耦合。 ##### 线程和协程的差异 - 每个 goroutine (协程) 默认占用内存远比 Java 、C 的线程少 - goroutine：2KB - 线程： 8 MB - 线程/goroutine 切换开销方面，goroutine 远比线程小 - 线程：涉及模式切换(从用户态切换到内核态)、 16 个寄存器、PC、SP...等寄存器的刷新 - goroutine：只有三个寄存器的值修改 - PC / SP / DX. - GOMAXPROCS - 控制并行线程数量 ##### 协程示例 - 启动新协程：go functionName() for i := 0 ; i < 10 ; i++ { ``` go fmt.Println(i) ``` } time.Sleep(time.Second) ##### channel - 多线程通信 - Channel 是多个协程之间通讯的管道 - 一端发送数据，一端接收数据 - 同一时间只有一个协程可以访问数据，无共享内存模式可能出现的内存竞争 - 协调协程的执行顺序 - 声明方式 - var identifier chan datatype - 操作符<- - 示例 ch := make(chan int) go func() { fmt.Println("hello from goroutine") ch <- 0 //数据写入Channel }() i := <-ch//从Channel中取数据并赋值 ##### 通道缓冲 - 基于 Channel 的通信是同步的 - 当缓冲区满时，数据的发送是阻塞的 - 通过 make 关键字创建通道时可定义缓冲区容量，默认缓冲区容量为 0 - 下面两个定义的区别？ - ch := make(chan int) - ch := make(chan int,1) ##### 遍历通道缓冲区 ch := make(chan int, 10 ) go func() { for i := 0 ; i < 10 ; i++ { rand.Seed(time.Now().UnixNano()) n := rand.Intn( 10 ) // n will be between 0 and 10 fmt.Println("putting: ", n) ch <- n } close(ch) }() fmt.Println("hello from main") for v := range ch { fmt.Println("receiving: ", v) } ##### 单向通道 - 只发送通道 - var sendOnlychan<-int - 只接收通道 - var readOnly<-chanint - Istio webhook controller - func(w *WebhookCertPatcher) runWebhookController(stopChan <-chanstruct{}) {} - 如何用: 双向通道转换 var c = make(chan int) go prod(c) go consume(c) func prod(ch chan<- int){ for { ch <- 1 } } func consume(ch <-chan int) { for { <-ch } } ##### 关闭通道 - 通道无需每次关闭 - 关闭的作用是告诉接收者该通道再无新数据发送 - 只有发送方需要关闭通道 ``` ch := make(chan int) defer close(ch) if v, notClosed := <-ch; notClosed { fmt.Println(v) } ``` ##### select - 当多个协程同时运行时，可通过 select 轮询多个通道 - 如果所有通道都阻塞则等待，如定义了 default 则执行 default - 如多个通道就绪则随机选择 select { case v:= <- ch1: ... case v:= <- ch2: ... default: ... } ##### 定时器 Timer - time.Ticker 以指定的时间间隔重复的向通道 C 发送时间值 - 使用场景 - 为协程设定超时时间 timer := time.NewTimer(time.Second) select { // check normal channel case <-ch: fmt.Println("received from ch") case <-timer.C: fmt.Println("timeout waiting from channel ch") } ##### 上下文 Context - 超时、取消操作或者一些异常情况，往往需要进行抢占操作或者中断后续操作 - Context 是设置截止日期、同步信号，传递请求相关值的结构体 type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} } - 用法 - context.Background - context.TODO - context.WithDeadline - context.WithValue - context.WithCancel ##### 如何停止一个子协程 done := make(chan bool) go func() { for { select { case <-done: fmt.Println("done channel is triggerred, exit child go routine") return } } }() close(done) ##### 基于 Context 停止子协程 - Context 是 Go 语言对 go routine 和 timer 的封装 ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() go process(ctx, 100 *time.Millisecond) <-ctx.Done() fmt.Println("main:", ctx.Err()) ## 模块二：Go 语言进阶 # 目录 •• 线程加锁线程调度 - 内存管理 - 包引用与依赖管理 ### 1. 线程加锁 #### 锁 - Go 语言不仅仅提供基于 CSP 的通讯模型，也支持基于共享内存的多线程数据访问 - Sync 包提供了锁的基本原语 - sync.Mutex 互斥锁 - Lock()加锁，Unlock 解锁 - sync.RWMutex 读写分离锁 - 不限制并发读，只限制并发写和并发读写 - sync.WaitGroup - 等待一组 goroutine 返回 - sync.Once - 保证某段代码只执行一次 - sync.Cond - 让一组 goroutine 在满足特定条件时被唤醒 #### Mutex 示例 ###### Kubernetes 中的 informer factory // Start initializes all requested informers. func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) { f.lock.Lock() defer f.lock.Unlock() for informerType, informer := range f.informers { if !f.startedInformers[informerType] { go informer.Run(stopCh) f.startedInformers[informerType] = true } } } // CreateBatch create a batch of pods. All pods are created before waiting. func (c *PodClient) CreateBatch(pods []*v1.Pod) []*v1.Pod { psvar:= wgmakesync.WaitGroup([]*v1.Pod, len(pods)) for wg.i, Addpod( 1 := ) range pods { go funcdefer(i wg.intDone, pod *v1.Pod) {() deferps[i] = GinkgoRecoverc.CreateSync()(pod) } }(i, pod) wg.returnWait()ps } #### WaitGroup 示例 #### Cond 示例 Kubernetescond: sync.NewCond中的队列，标准的生产者消费者模式(&sync.Mutex{}), // Add marks item as needing processing.func(q *Type) Add(item interface{}) { q.cond.L.deferq.cond.L.Lock()Unlock() ifq.shuttingDownreturn { }ifq.dirty.has(item) { } return q.metrics.q.dirty.insertadd(item)(item) ifq.processing.return has(item) { }q.queue= append(q.queue, item) } q.cond.Signal() #### Cond 示例 // Get blocks until it can return an item to be processed. If shutdown = true,// the caller should end their goroutine. You must call Done with item when you // have finished processing it.func (q *Type) Get() (item interface{}, shutdown bool) { q.cond.L.defer q.cond.L.Lock()Unlock() for lenq.cond.(q.queueWait) == () 0 && !q.shuttingDown { }if len(q.queue) == 0 { // We must be shutting down.return nil, true }item, q.queue = q.queue[ 0 ], q.queue[ 1 :] q.metrics.q.processing.get(item)insert(item) q.dirty.return item, deletefalse(item) } ### 2. 线程调度 ### 深入理解 Go 语言线程调度 - 进程：资源分配的基本单位 - 线程：调度的基本单位 - 无论是线程还是进程，在 linux 中都以 task_struct 描述，从内核角度看，与进程无本质区别 - Glibc 中的 pthread 库提供 NPTL（Native POSIX Threading Library）支持 Processmm fs files signal ``` Processmm fs files signal ``` ``` Process Process mm fs files signal ``` ``` 创建进程：fork 创建线程：pthread_create ``` #### Linux 进程的内存使用 Kernel space Stack 未分配内存 Heap BSS（未初始化数据） Data（初始化数据） Text（程序代码） ``` PGD PUD PMD PT PGD: page global directoryPUD: page upper directory PMP: page middle directoryPT: page table ``` ``` 虚拟地址 物理内存 磁盘（虚拟内存） ``` ``` 参数环境变量 ``` #### CPU 对内存的访问 - CPU 上有个 Memory Management Unit（MMU） 单元 - CPU 把虚拟地址给 MMU，MMU 去物理内存中查询页表，得到实际的物理地址 - CPU 维护一份缓存 Translation Lookaside Buffer（TLB），缓存虚拟地址和物理地址的映射关系 ``` CPU ``` ``` TLB MMU PGD PUD PMD PT ``` ``` 物理内存 ``` 总线 #### 进程切换开销 - 直接开销 - 切换页表全局目录（PGD） - 切换内核态堆栈 - 切换硬件上下文（进程恢复前，必须装入寄存器的数据统称为硬件上下文） - 刷新 TLB - 系统调度器的代码执行 - 间接开销 - CPU 缓存失效导致的进程需要到内存直接访问的 IO 操作变多 #### 线程切换开销 - 线程本质上只是一批共享资源的进程，线程切换本质上依然需要内核进行进程切换 - 一组线程因为共享内存资源，因此一个进程的所有线程共享虚拟地址空间，线程切换相比进程 切换，主要节省了虚拟地址空间的切换 #### 用户线程 无需内核帮助，应用程序在用户空间创建的可执行单元，创建销毁完全在用户态完成。 ``` CPU ``` ``` Kernel Thread Kernel Thread Kernel Thread ``` ``` User Thread User Thread User Thread ``` ``` User Thread User Thread ``` ``` User Thread User Thread User Thread ``` ``` Process Process Process ``` ``` 用户态 内核态 ``` #### Goroutine Go 语言基于 GMP 模型实现用户态线程 - G：表示 goroutine，每个 goroutine 都有自己的栈空间，定时器， 初始化的栈空间在 2 k 左右，空间会随着需求增长。 - M：抽象化代表内核线程，记录内核线程栈信息，当 goroutine 调度 到线程时，使用该 goroutine 自己的栈信息。 - P：代表调度器，负责调度 goroutine，维护一个本地 goroutine 队 列，M 从 P 上获得 goroutine 并执行，同时还负责部分内存的管理。 M G ``` P ``` ``` G G G G ``` #### GMP 模型细节 ``` G M ``` ``` P ``` ``` G ``` ``` G ``` ``` G G G G G ``` ``` G M ``` ``` P ``` ``` G ``` ``` G ``` ``` G M ``` ``` P ``` ``` G ``` ``` G ``` ``` G ``` ``` P P ``` pidle（全局空闲P列表） ``` GRQ（全局可运行G队列） ``` ``` G G G ``` gFree（全局自由G列表） ``` G G G ``` sudog（阻塞队列） ``` CPU CPU CPU ``` ``` G Grunning G Grunnable G Gwaiting G Gsyscall G Gdead P Pidle P Prunning P Psyscall ``` ``` LRQ LRQ LRQ ``` #### G 所处的位置 - 进程都有一个全局的 G 队列 - 每个 P 拥有自己的本地执行队列 - 有不在运行队列中的 G - 处于 channel 阻塞态的 G 被放在 sudog - 脱离 P 绑定在 M 上的 G，如系统调用 - 为了复用，执行结束进入 P 的 gFree 列表中的 G #### Goroutine 创建过程 - 获取或者创建新的 Goroutine 结构体 - 从处理器的 gFree 列表中查找空闲的 Goroutine - 如果不存在空闲的 Goroutine，会通过 runtime.malg 创建一个栈大小足够的新结构体 - 将函数传入的参数移到 Goroutine 的栈上 - 更新 Goroutine 调度相关的属性，更新状态为_Grunnable - 返回的 Goroutine 会存储到全局变量 allgs 中 #### 将 Goroutine 放到运行队列上 - Goroutine 设置到处理器的 runnext 作为下一个处理器 执行的任务 - 当处理器的本地运行队列已经没有剩余空间时，就会把 本地队列中的一部分 Goroutine 和待加入的 Goroutine 通过 runtime.runqputslow 添加到调度器持有的全局 运行队列上 #### 调度器行为 - 为了保证公平，当全局运行队列中有待执行的 Goroutine 时，通过 schedtick 保证有一定 几率会从全局的运行队列中查找对应的 Goroutine - 从处理器本地的运行队列中查找待执行的 Goroutine - 如果前两种方法都没有找到 Goroutine，会通过 runtime.findrunnable 进行阻塞地查找 Goroutine - 从本地运行队列、全局运行队列中查找 - 从网络轮询器中查找是否有 Goroutine 等待运行 - 通过 runtime.runqsteal 尝试从其他随机的处理器中窃取待运行的 Goroutine ### 3. 内存管理 #### 关于内存管理的争论 内存管理太重要了！手动管理麻烦且容易出错，所以我们应该交给机器去管理！ 内存管理太重要了！所以如果交给机器管理我不能放心！ ``` Java/golang ``` ``` c/c++ ``` #### 堆内存管理 ``` Heap ``` ``` Mutator Allocator ``` ``` Collector ``` ``` 内存分配器，处理动态内存分配请求 ``` 用户程序，通过 Allocator 创建对象 ``` 垃圾回收器，回收内存空间 对象头，Collector 和 Allocator 同步对象元数据 ``` ``` Object Header ``` #### 堆内存管理 - 初始化连续内存块作为堆 - 有内存申请的时候，Allocator 从堆内存的未分配区域分割小内存块 - 用链表将已分配内存连接起来 - 需要信息描述每个内存块的元数据：大小，是否使用，下一个内存块的地址等 ``` size used next data size used next data unmapped Object Header ``` ``` align Object Header ``` ### TCMalloc 概览 ``` CentralCache ``` ``` ThreadCache 1ThreadCache 2 ThreadCache n ``` ``` Application ``` ``` PageHeap ``` MemoryVirtual ``` Size class 0 Size class 1 Size class 2 Size class n ``` ``` Size class 0 Size class 1 Size class 2 Size class n ``` ``` Span list 1Span list 2 Span list 128 ``` ``` Large span set ``` ``` ... Small object ``` ``` Large and medium object ``` ``` Pages Span ``` ``` Free Object ``` ``` Span list 1: 1 pageSpan list 2: 2 pagesSpan list 128: 128 pages = 1M ``` #### TCMalloc - page:内存页，一块 8 K 大小的内存空间。Go 与操作系统之间的内存申请和释放，都是以 page 为单位的 - span: 内存块，一个或多个连续的 page 组成一个 span - sizeclass : 空间规格，每个 span 都带有一个 sizeclass ，标记着该 span 中的 page 应该如何 使用 - object : 对象，用来存储一个变量数据内存空间，一个 span 在初始化时，会被切割成一堆等大 的 object ；假设 object 的大小是 16 B ，span 大小是 8 K ，那么就会把 span 中的 page 就会 被初始化 8 K / 16B = 512 个 object 。所谓内存分配，就是分配一个 object 出去 #### TCMalloc - 对象大小定义 - 小对象大小：0~256KB - 中对象大小：256KB~1MB - 大对象大小：>1MB - 小对象的分配流程 - ThreadCacheCentralCache -和> CentralCacheHeapPage，无系统调用配合无锁分配，分配效率是非常高的-> HeapPage，大部分时候，ThreadCache 缓存都是足够的，不需要去访问 - 中对象分配流程 - 直接在 PageHeap 中选择适当的大小即可， 128 Page 的 Span 所保存的最大内存就是 1 MB - 大对象分配流程 - 从 large span set 选择合适数量的页面组成 span，用来存储数据 #### Go 语言内存分配 ``` mcentral ``` ``` mmcache of P1cache of P2 mcache of P3 ``` ``` Application ``` ``` mheap ``` MemoryVirtual ``` span class 0 span class 0 spanclass 1 span 134 class ``` ``` spanspanclass 1class 133 ``` ``` free ``` ``` Tiny object ``` ``` Large and medium object ``` ``` Pages Span ``` ``` span class 0 Free Object span class 0spanclass 1 ``` ``` span 134 class ``` ``` span class 1 ``` ``` scav ``` ``` heapArenaarenas heapArenaheapArena ``` #### Go 语言内存分配 - mcache：小对象的内存分配直接走 - size class 从 1 到 66 ，每个 class 两个 span - Span 大小是 8 KB，按 span class 大小切分 - mcentral - Spanspan，内的所有内存块都被占用时，没有剩余空间继续分配对象，mcache 拿到 span 后继续分配对象 mcache 会向 mcentral 申请 1 个 - 当spanmcentral 向 mcache 提供 span 时，如果没有符合条件的 span，mcentral 会向 mheap 申请 - mheap - 当 mheap 没有足够的内存时，mheap 会向 OS 申请内存 - Mheap 把 Span 组织成了树结构，而不是链表 - 然后把 Span 分配到 heapArena 进行管理，它包含地址映射和 span 是否包含指针等位图 - 为了更高效的分配、回收和再利用内存 #### 内存回收 - 引用计数（Python，PHP，Swift） - 对每一个对象维护一个引用计数，当引用该对象的对象被销毁的时候，引用计数减收该对象 1 ，当引用计数为 0 的时候，回 - 优点：对象可以很快的被回收，不会出现内存耗尽或达到某个阀值时才回收 - 缺点：不能很好的处理循环引用，而且实时维护引用计数，有也一定的代价 - 标记-清除（Golang） - 从根变量开始遍历所有引用的对象，引用的对象标记为"被引用"，没有被标记的进行回收 - 优点：解决引用计数的缺点 - 缺点：需要 STW（stop the word），即要暂停程序运行 - 分代收集（Java） - 按照生命周期进行划分不同的代空间，生命周期长的放入老年代，短的放入新生代，新生代的回收频率高于老年代的频率 #### mspan - allocBits - 记录了每块内存分配的情况 - gcmarkBits - 记录了每块内存的引用情况，标记阶段对每块内存进行标记，有对象引用的内存标记为 1 ，没有的标 记为 0 #### mspan - 这两个位图的数据结构是完全一致的，标记结束则进行内存回收，回收的时候，将 allocBits 指 向 gcmarkBits，标记过的则存在，未进行标记的则进行回收 ``` startAddrmspan allocBitsnpages gcmarkBitsallocCount spanclass ``` ``` 1 1 0 1 0 0 1 ... 1 1 0 0 0 0 0 ... ``` #### GC 工作流程 Golang GC 的大部分处理是和用户代码并行的 - Mark： - Mark Prepare: 务数量等。这个过程需要初始化 GCSTW任务，包括开启写屏障 (write barrier) 和辅助 GC(mutator assist)，统计root对象的任 - GC Drains: 加入标记队列扫描所有(灰色队列root)，并循环处理灰色队列的对象，直到灰色队列为空。该过程后台并行执行对象，包括全局指针和 goroutine(G) 栈上的指针（扫描对应 G 栈时需停止该 G)，将其 - Mark Termination程中可能会有新的对象分配和指针赋值，这个时候就需要通过写屏障（：完成标记工作，重新扫描(re-scan)全局指针和栈。因为write barrierMark 和用户程序是并行的，所以在）记录下来，re-scan 再检查一下，这Mark 过 个过程也是会 STW 的 - Sweep：按照标记结果回收所有的白色对象，该过程后台并行执行 - Sweep Termination：对未清扫的 span 进行清扫, 只有上一轮的 GC 的清扫工作完成才可以开始新一轮的 GC #### GC 工作流程 ``` 关闭 •• GC写操作是正常的赋值关闭 栈扫描 开启写屏障 STW •• 开启写屏障等准备工作，短暂开启从全局空间和 goroutine 栈空间上收集变量STW 标记 • 三色标记法，直到没有灰色对象 标记结束 STW • 开启 STW，回头重新扫描 root 区域新变量，对他们进行标记 清除 • 关闭 STW 和 写屏障，对白色对象进行清除 关闭 • 循环结束，重启下一阶段GC ``` #### 三色标记 - GC 开始时，认为所有 object 都是 白色，即垃圾。 - 从 root 区开始遍历，被触达的 object 置成 灰色。 - 遍历所有灰色 object，将他们内部的引用变量置成 灰色，自身置成 黑色 - 循环第 3 步，直到没有灰色 object 了，只剩下了黑白两种，白色的都是垃圾。 - 对于黑色 object，如果在标记期间发生了写操作，写屏障会在真正赋值前将新对象标记为 灰色。 - 标记过程中，mallocgc 新分配的 object，会先被标记成 黑色 再返回。 a b c ``` d a b c ``` ``` d a b c ``` ``` d a b c ``` ``` d a b c ``` ``` d ``` #### 垃圾回收触发机制 - 内存分配量达到阀值触发 GC - 每次内存分配时都会检查当前内存分配量是否已达到阀值，如果达到阀值则立即启动 GC。 - 阀值 = 上次 GC 内存分配量 * 内存增长率 - 内存增长率由环境变量 GOGC 控制，默认为 100 ，即每当内存扩大一倍时启动 GC。 - 定期触发 GC - 默认情况下，最长 2 分钟触发一次 GC，这个间隔在 src/runtime/proc.go:forcegcperiod 变量中 被声明 - 手动触发 - 程序代码中也可以使用 runtime.GC()来手动触发 GC。这主要用于 GC 性能测试和统计。 ### 4. 包引用与依赖管理 #### Go 语言依赖管理的演进 - 回顾 GOPATH - 通过环境变量设置系统级的 Go 语言类库目录 - GOPATH 的问题？ - 不同项目可能依赖不同版本 - 代码被 clone 以后需要设置 GOPATH 才能编译 - vendor - 自 1.6 版本，支持 vendor 目录，在每个 Go 语言项目中，创建一个名叫 vendor 的目录，并将依赖拷贝至该目录。 - Go 语言项目会自动将 vendor 目录作为自身的项目依赖路径 - 好处？ - 每个项目的 vendor目录是独立的，可以灵活的选择版本 - Vendor 目录与源代码一起 check in 到 github，其他人 checkout 以后可直接编译 - 无需在编译期间下载依赖包，所有依赖都已经与源代码保存在一起 #### vendor 管理工具 通过声明式配置，实现 vendor 管理的自动化 - 在早期，Go 语言无自带依赖管理工具，社区方案鱼龙混杂比较出名的包括 - Godeps, Glide - Go 语言随后发布了自带的依赖管理工具 Gopkg - 很快用新的工具 gomod 替换掉了 gopkg - 切换 mod 开启模式：export GO111MODULE=on/off/auto - Go mod 相比之前的工具更灵活易用，以基本统一了 Go 语言依赖管理 思考：用依赖管理工具的目的？ - 版本管理 - 防篡改 #### Go mod 使用 - 创建项目 - 初始化 Go 模块 - go mod init - 下载依赖包 - go mod download（下载的依赖包在$GOPATH/pkg，如果没有设置 GOPATH，则下载在项目根目录/pkg） - 在源代码中使用某个依赖包，如 github.com/emicklei/go-restful - 添加缺少的依赖并为依赖包瘦身 - go mod tidy - 把 Go 依赖模块添加到 vendor 目录 - go mod vendor 配置细节会被保存在项目根目录的 go.mod 中 可在 require 或者 replacement 中指定版本 #### go.mod sample module k8s.io/go 1.13 apiserver require (github.com/evanphx/json-patch v4.9.0+incompatible github.comgithub.com/go/go--openapiopenapi//spec v0.19.3jsonreferencev0.19.3 // indirect github.comgithub.com//google/gogogo/protobuf-cmpv1.3.2v0.3.0 github.comk8s.io/apimachinery/google/gofuzzv0.0.0-v1.1.0 20210518100737 - 44f1264f7b6b ) replace (golang.org/x/crypto => golang.org/x/crypto v0.0.0- 20200220183623 - bac4c82f6975 golang.orgk8s.io/api=> k8s.io//x/text => apigolang.orgv0.0.0- 20210518101910 /x/text v0.3.2 -53468e23a787 k8s.io/k8s.io/clientapimachinery-go => k8s.io/client=> k8s.io/apimachinery-go v0.0.0- 20210518104342 v0.0.0- (^20210518100737) - fa3acefe68f3-44f1264f7b6b ) k8s.io/component-base => k8s.io/component-base v0.0.0-^20210518111421 - 67c12a31a26a #### GOPROXY 和 GOPRIVATE - GOPROXY - 为拉取 Go 依赖设置代理 - export GOPROXY=https://goproxy.cn - 在设置 GOPROXY 以后，默认所有依赖拉取都需要经过 proxy 连接 git repo，拉取代码，并做 checksum 校验 - 某些私有代码仓库是 goproxy.cn 无法连接的，因此需要设置 GOPRIVATE 来声明私有代码仓库 GOPRIVATE=*.corp.example.com GOPROXY=proxy.example.com GONOPROXY=myrepo.corp.example.com ### 5. Makefile #### Go 语言项目多采用 Makefile 组织项目编译 root: export ROOT=github.com/cncamp/golang; .PHONY: root release: echo "building httpserver binary" mkdir -p bin/amd64 CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/amd64. .PHONY: release ### 6. 动手编写一个 HTTP Server #### 理解 net.http 包 - 注册 handle 处理函数 http.HandleFunc("/healthz", healthz) //Use the default DefaultServeMux. ListenAndService err := http.ListenAndServe(":80", nil) if err != nil { log.Fatal(err) } - 定义 handle 处理函数 func*http.Requesthealthz(w ) {http.ResponseWriter, r io.WriteString(w, "ok") } #### 阻塞 IO 模型 ``` 应用进程 系统调用 系统内核 recvfrom 数据报文尚未就绪 ``` ``` 数据报文就绪 拷贝数据 ``` ``` 处理数据报 系统调用返回 拷贝完成 ``` recvfrom进程阻塞于调用 #### 非阻塞 IO 模型 ``` 应用进程 系统内核 recvfrom 系统调用 数据报文尚未就绪 ``` ``` 数据报文就绪 拷贝数据 ``` ``` 处理数据报 拷贝完成 ``` recvfrom，直至返回进程重复调用OK ``` recvfrom recvfrom ``` ``` 系统调用返回错误 返回错误 ``` ``` 返回OK ``` ``` 系统调用 ``` #### IO 多路复用 ``` 应用进程 系统内核 select/poll 系统调用 数据报文尚未就绪 ``` ``` 数据报文就绪 拷贝数据 ``` ``` 处理数据报文 拷贝完成 ``` 进程阻塞于等待有可读的select/pollsocket 调用， ``` recvfrom ``` ``` 返回可读 ``` ``` 返回OK ``` 系统调用 数据复制到进程缓冲区期间进程阻塞 ， ### 异步 IO ``` 应用进程 系统内核 异步 IO 读 系统调用 数据报文尚未就绪 ``` ``` 数据报文就绪 拷贝数据 ``` ``` 处理数据报文 拷贝完成 ``` ``` 程序继续运行 ``` ``` 系统调用返回 ``` ``` 发送信号 ``` #### Linux epoll epoll_create1 ... ``` rdlistwq ovflistrbr epitem ``` ``` epitem epitem ``` ``` epitem epitem epitem epitem ``` ``` epitem epitem epitem ``` ``` epoll_ctl(epfd, op, fd, event) ``` ``` sk_wait... ... ``` ``` epoll_wait(epfd, event, maxevents, timeout) copy ``` ``` Data from network stack ``` #### Go 语言高性能 httpserver 的实现细节 - Go 语言将协程与 fd 资源绑定 - 一个 socket fd 与一个协程绑定 - 当 socket fd 未就绪时，将对应协程设置为 Gwaiting 状态，将 CPU 时间片让给其他协程 - GoGrunnable语言 runtime 并加入执行队列调度器进行调度唤醒协程时，检查 fd 是否就绪，如果就绪则将协程置为 - 协程被调度后处理 fd 数据 Server G G G ``` fd fd fd ``` ``` client client client ``` ``` data ready data not ready ``` ``` runnable waiting runnable data ready ``` #### 代码实现细节 https://pouncing-waterfall-7c4.notion.site/http-server-socket-detail- e1f350d63c7c4d9f86ce140949bd90c2 ### 7. 调试 #### debug - gdb: - Gccgo 原生支持 gdb，因此可以用 gdb 调试 Go 语言代码，但 dlv 对 Go 语言 debug 的支持比 gdb 更好 - Gdb 对 Go 语言的栈管理，多线程支持等方面做的不够好，调试代码时可能有错乱现象 - dlv： - Go 语言的专有 debugger #### dlv 的配置 - 配置 - 在 vscode 中配置 dlb - 菜单：View -> Command Palette - 选择 Go : Install/Update Tools，选择安装 - 安装完后，从改入口列表中可以看到 dlv 和 dlv-dap 已经安装好 - Debug 方法 - 在代码中设置断点 - 菜单中选择 Run -> Start Debugging 即可进入调试 #### 更多 debug 方法 - 添加日志 - 在关键代码分支中加入日志 - 基于fmt包将日志输出到标准输出 stdout - fmt.Println() - fmt 无日志重定向，无日志分级 - 即与日志框架将日志输出到对应的 appender - 比如可利用 glog 进行日志输出 - 可配置 appender，将标准输出转至文件 - 支持多级日志输出，可修改配置调整日志等级 - 自带时间戳和代码行，方便调试 #### Glog 使用方法示例 import"flag" ( ")github.com/golang/glog" func flag.main() {Set("v", "4") glog.mux := V( (^2) http.).InfoNewServeMux("Starting http server..."() ) mux.err HandleFunc:= http.ListenAndServe("/", rootHandler(":80"), mux) if err != log.Fatalnil (err){ } } #### 性能分析（Performance Profiling） - CPU Profiling: 在代码中添加 CPUProfile 代码，runtime/pprof 包提供支持 varfunccpuprofilemain() { = flag.String("cpuprofile", "", "write cpu profile to file") flag.if *cpuprofileParse() != "" { f, errif err != := os.Createnil { (*cpuprofile) } log.Fatal(err) deferpprof.pprof.StartCPUProfileStopCPUProfile(f)() } } #### 分析 CPU 瓶颈 - 运行 cpuprofilie 代码后，会在 /tmp/cpuprofile 中记录 cpu 使用时间 - 运行 go tool pprof /tmp/cpuprofile 进入分析模式 - 运行 top10 查看 top 10 线程，显示 30ms 花费在 main.main Showing nodes accounting for 30ms, 100% of 30ms total flat flat% sum% cum cum% 30ms 100% 100% 30ms 100% main.main 0 0% 100% 30ms 100% runtime.main - (pprof) list main.main 显示 30 毫秒都花费在循环上 Total: 30ms 30ms 30ms (flat, cum) 100% of Total 20ms 20ms 21: for i := 0; i < 100000000; i++ { 10ms 10ms 22: result += I - 可执行 web 命令生成 svg 文件，在通过浏览器打开 svg 文件查看图形化分析结果 #### 其他可用 profiling 工具分析的问题 - CPU profile - 程序的 CPU 使用情况，每 100 毫秒采集一次 CPU 使用情况 - Memory Profile - 程序的内存使用情况 - Block Profiling - 非运行态的 goroutine 细节，分析和查找死锁 - Goroutine Profiling - 所有 goroutines 的细节状态，有哪些 goroutine，它们的调用关系是怎样的 #### 针对 http 服务的 pprof ``` Import ("net/http/pprof" ) funcmuxstartHTTP:= http.(addrNewServeMuxstring, s *tnetd.Server() ) { mux.mux.HandleFuncHandleFunc((“/debug/“/debug/pprofpprof/”/profile”, pprof.Index, pprof.Profile) ) mux.mux.HandleFuncHandleFunc((“/debug/“/debug/pprofpprof/symbol”/trace”, pprof.Trace, pprof.Symbol) ) serverAddr:= &: addrhttp.Server, { } Handler: mux, } server.ListenAndServe() ``` - net/http/pprof 包提供支持 - 如果采用默认 mux handle，则只需 import _ "net/http/pprof” - 如果采用自定义 mux handle，则需要注册 pprof handler #### 分析 go profiling 结果 在运行了开启 pprof 的服务器以后，可以通过访问对应的 URL 获得 profile 结果 - allocs: A sampling of all past memory allocations - block: Stack traces that led to blocking on synchronization primitives - cmdline: The command line invocation of the current program - goroutine: Stack traces of all current goroutines - heap: A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample. - mutex: Stack traces of holders of contended mutexes - profile: CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile. - threadcreate: Stack traces that led to the creation of new OS threads - trace: A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace. #### 分析 go profiling 结果 #### 结果分析示例 - 分析 goroutine 运行情况 - curl localhost/debug/pprof/goroutine?debug=2 - 分析堆内存使用情况 - curl localhost/debug/pprof/heap?debug=2 ### 8. Kubernetes 中常用代码解读 #### Rate Limit Queue func (r *ItemExponentialFailureRateLimiter) When(item interface{}) time.Duration { r.failuresLock.defer r.failuresLock.Lock()Unlock() expr.failures:= r.failures[item] = [item]r.failures[item] + 1 // The backoff is capped such that ‘calculated’ value never overflows.backoff := float64(r.baseDelay.Nanoseconds()) * math.Pow( 2 , float64(exp)) if backoff > math.MaxInt64 {return r.maxDelay } calculatedif calculated > := time.r.maxDelayDuration(backoff){ } return r.maxDelay } return calculated ### 9. Kubernetes 日常运维中的代码调试场景 #### 案例 1 ：空指针 - 问题描述 Kubenetes 调度器在调度有外挂存储需求的 pod 的时候，在获取节点信息失败 时会异常退出 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x105e2 83] #### 案例 1 ：空指针 - 根因分析 nil pointe 是 Go 语言中最常出现的一类错误，也最容易判断，通常在 call stack 中就会告诉 你哪行代码有问题 在调度器 csi.go 中的如下代码，当 node 为 nil 的时候，对 node 的引用 node.Name 就会 引发空指针 nodeif:= node == nodeInfo.nilNode{ () not found: return framework.%s", node.NameNewStatus)) (framework.Error, fmt.Sprintf("node } - 解决办法 当指针为空时，不要继续引用。 https://github.com/kubernetes/kubernetes/pull/102229 #### 案例 2 ：Map 的读写冲突 - 问题描述： 程序在遍历 Kubernetes 对象的 Annotation 时异常退出 - 根因分析 Kubernetes 对象中 Label 和 Annotation 是 map[string]string 经常有代码需要修改这两个 Map 同时可能有其他线程 for...range 遍历 - 解决方法： - 用 sync.RWMutex 加锁 - 使用线程安全 Map，比如 sync.Map{} #### 案例 3 ：kube-proxy 消耗 10 个 CPU - 问题描述 客户汇报问题，kube-proxy 消耗了主机 10 个 CPU - 根因分析 - 登录问题机器，执行top命令查看 cpu消耗，可以看到kube-proxy的cpu消耗和pid信息 - 对说明程序创建了大量可回收对象。kube-proxy进程运行System profiling tool，发现 10 个CPU中，超过 60%的CPU都在做垃圾回收，这说明GC需要回收的对象太多了， - perf top –p Overhead Shared Obj Symbol 26.48% kube-proxy [.] runtime.gcDrain 13.86% kube-proxy [.] runtime.greyobject 10.71% kube-proxy [.] runtime.(*lfstack).pop 10.04% kube-proxy [.] runtime.scanobject #### 案例 3 ：kube-proxy 消耗 10 个 CPU 通过 pprof 分析内存占用情况 curl 127.0.0.1:10249/debug/pprof/heap?debug=2 1: 245760 [301102: 73998827520] @ 0x11ddcda 0x11f306e 0x11f35f5 0x11fbdce 0x1204a8a 0x114ed76 0x114eacb 0x11 # 0x11ddcd9 k8s.io/kubernetes/vendor/github.com/vishvananda/netlink.(*Handle).RouteListFiltered+0x679 # 0x11f306d k8s.io/kubernetes/pkg/proxy/ipvs.(*netlinkHandle).GetLocalAddresses+0xed # 0x11f35f4 k8s.io/kubernetes/pkg/proxy/ipvs.(*realIPGetter).NodeIPs+0x64 # 0x11fbdcd k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncProxyRules+0x47dd #### 案例 3 ：kube-proxy 消耗 10 个 - heap dump 分析 - GetLocalAddresses 函数调用创建了 301102 个对象，占用内存 73998827520 - 如此多的对象被创建，显然会导致 kube-proxy 进程忙于 GC，占用大量 CPU - 对照代码分析通过 ip route GetLocalAddresses命令获得当前节点所有的实现local，发现该函数的主要目的是获取节点本机路由信息并转换成 go struct 并过滤掉 ipvs0IP 地址，获取的方法是网口上的路由信息 - ip route show table local type local proto kernel - 因为集群规模较大，该命令返回 5000 条左右记录，因此每次函数调用都会有数万个对象被生成 - 而反复调用该函数创建大量临时对象kube-proxy 在处理每一个服务的时候都会调用该方法，因为集群有数千个服务，因此，kube-proxy在 - 修复方法 - 函数调用提取到循环外 - https://github.com/kubernetes/kubernetes/pull/79444 #### 案例 4 ：线程池耗尽 - 问题描述： 在 Kubernetes 中有一个控制器，叫做 endpoint controller，该控制器符合生产者消费者模式，默认有 5 个 worker 线程作为消费者。该消费者在处理请求时，可能调用的 LBaaS 的 API 更新负载均衡配置。我们 发现该控制器会时不时不工作，具体表现为，该做的配置变更没发生，相关日志也不打印了。 - 根因分析： 通过 pprof 打印出该进程的所有 go routine 信息，发现 worker 线程都卡在 http 请求调用处。 当worker线程调用 LBaaS API 时，底层是 net/http 包调用，而客户端在发起连接请求时，未设置客户端 超时时间。这导致当出现某些网络异常时，客户端会永远处于等待状态。 - 解决方法： 修改代码加入客户端超时控制。 [Go 语言设计与实现](https://draveness.me/golang/) ================================================ FILE: docs/index.md ================================================ # 朽木自雕 活在当下，活着就要发挥，就要创造 ## 个人爱好，纯属扯蛋 ## 捐赠 如果你觉得这写文章能帮助到了你，你可以帮作者买一杯果汁表示鼓励 ![pay](./img/pay.png) [Paypal Me](https://paypal.me/yangfubing) ## 关联 本仓库下存放个人博客的源文件。持续更新，欢迎 star。 如果大家觉得那里写的不合适的可以给我提 Issue [Github](https://github.com/burningmyself) [Gitee](https://gitee.com/burningmyself) ================================================ FILE: docs/java/bio-nio.md ================================================ # 关于BIO和NIO的理解 摘要: 关于BIO和NIO的理解 最近大概看了ZooKeeper和Mina的源码发现都是用Java NIO实现的，所以有必要搞清楚什么是NIO。下面是我结合网络资料自己总结的，为了节约时间图示随便画的，能达意就行。 ### 简介： BIO：同步阻塞式IO，服务器实现模式为一个连接一个线程，即客户端有连接请求时服务器端就需要启动一个线程进行处理，如果这个连接不做任何事情会造成不必要的线程开销，当然可以通过线程池机制改善。 NIO：同步非阻塞式IO，服务器实现模式为一个请求一个线程，即客户端发送的连接请求都会注册到多路复用器上，多路复用器轮询到连接有I/O请求时才启动一个线程进行处理。 AIO(NIO.2)：异步非阻塞式IO，服务器实现模式为一个有效请求一个线程，客户端的I/O请求都是由OS先完成了再通知服务器应用去启动线程进行处理。 ### BIO 同步阻塞式IO，相信每一个学习过操作系统网络编程或者任何语言的网络编程的人都很熟悉，在while循环中服务端会调用accept方法等待接收客户端的连接请求，一旦接收到一个连接请求，就可以建立通信套接字在这个通信套接字上进行读写操作，此时不能再接收其他客户端连接请求，只能等待同当前连接的客户端的操作执行完成。 如果BIO要能够同时处理多个客户端请求，就必须使用多线程，即每次accept阻塞等待来自客户端请求，一旦受到连接请求就建立通信套接字同时开启一个新的线程来处理这个套接字的数据读写请求，然后立刻又继续accept等待其他客户端连接请求，即为每一个客户端连接请求都创建一个线程来单独处理，大概原理图就像这样： ![bio-nio](./../img/bio-nio-1.png) 虽然此时服务器具备了高并发能力，即能够同时处理多个客户端请求了，但是却带来了一个问题，随着开启的线程数目增多，将会消耗过多的内存资源，导致服务器变慢甚至崩溃，NIO可以一定程度解决这个问题。 ### NIO 同步非阻塞式IO，关键是采用了事件驱动的思想来实现了一个多路转换器。 NIO与BIO最大的区别就是只需要开启一个线程就可以处理来自多个客户端的IO事件，这是怎么做到的呢？ 就是多路复用器，可以监听来自多个客户端的IO事件： A. 若服务端监听到客户端连接请求，便为其建立通信套接字(java中就是通道)，然后返回继续监听，若同时有多个客户端连接请求到来也可以全部收到，依次为它们都建立通信套接字。 B. 若服务端监听到来自已经创建了通信套接字的客户端发送来的数据，就会调用对应接口处理接收到的数据，若同时有多个客户端发来数据也可以依次进行处理。 C. 监听多个客户端的连接请求和接收数据请求同时还能监听自己时候有数据要发送。 ![bio-nio](./../img/bio-nio-2.png) 总之就是在一个线程中就可以调用多路复用接口（java中是select）阻塞同时监听来自多个客户端的IO请求，一旦有收到IO请求就调用对应函数处理。 #### 轮询方式 如果将套接字读操作换成非阻塞的，那么只需要一个线程就可以同时处理套接字，每次检查一个套接字，有数据则读取，没有则检查下一个，因为是非阻塞的，所以执行read操作时若没有数据准备好则立即返回，不会发生阻塞。 #### I/O多路复用 这种轮询的方式缺点是浪费CPU资源，大部分时间可能都是无数据可读的，不必仍不间断的反复执行read操作，I/O多路复用（IOmultiplexing）是一种更好的方法，调用select函数时，其内部会维护一张监听的套接字的列表，其会一直阻塞直到其中某一个套接字有数据准备好才返回，并告诉是哪个套接字可读，这时再调用该套接字的read函数效率更高。 所以基本可以认为 “NIO = I/O多路复用 + 非阻塞式I/O”，大部分情况下是单线程，但也有超过一个线程实现NIO的情况 #### NIO三种模型 上面所讲到的只需要一个线程就可以同时处理多个套接字，这只是其中的一种单线程模型，是一种较为极端的情况，NIO主要包含三种线程模型： 1. Reactor单线程模型 2. Reactor多线程模型 3. 主从Reactor多线程模型   ##### Reactor单线程模型： 单个线程完成所有事情包括接收客户端的TCP连接请求，读取和写入套接字数据等。 对于一些小容量应用场景，可以使用单线程模型。但是对于高负载、大并发的应用却不合适，主要原因如下： 1. 一个NIO线程同时处理成百上千的链路，性能上无法支撑，即便NIO线程的CPU负荷达到100%，也无法满足海量消息的编码、解码、读取和发送； 2. 当NIO线程负载过重之后，处理速度将变慢，这会导致大量客户端连接超时，超时之后往往会进行重发，这更加重了NIO线程的负载，最终会导致大量消息积压和处理超时，NIO线程会成为系统的性能瓶颈； 3. 可靠性问题：一旦NIO线程意外跑飞，或者进入死循环，会导致整个系统通信模块不可用，不能接收和处理外部消息，造成节点故障。 为了解决这些问题，演进出了Reactor多线程模型。   ##### Reactor多线程模型： Reactor多线程模型与单线程模型最大的区别就是有一组NIO线程处理真实的IO操作。 Reactor多线程模型的特点： 1. 有专门一个NIO线程-Acceptor线程用于监听服务端，接收客户端的TCP连接请求； 2. 网络IO操作-读、写等由一个NIO线程池负责，线程池可以采用标准的JDK线程池实现，它包含一个任务队列和N个可用的线程，由这些NIO线程负责消息的读取、解码、编码和发送； 3. 1个NIO线程可以同时处理N条链路，但是1个链路只对应1个NIO线程，防止发生并发操作问题。 在绝大多数场景下，Reactor多线程模型都可以满足性能需求；但是，在极特殊应用场景中，一个NIO线程负责监听和处理所有的客户端连接可能会存在性能问题。例如百万客户端并发连接，或者服务端需要对客户端的握手消息进行安全认证，认证本身非常损耗性能。在这类场景下，单独一个Acceptor线程可能会存在性能不足问题，为了解决性能问题，产生了第三种Reactor线程模型-主从Reactor多线程模型。 即从单线程中由一个线程即监听连接事件、读写事件、由完成数据读写，拆分为由一个线程专门监听各种事件，再由专门的线程池负责处理真正的IO数据读写。   ##### 主从Reactor多线程模型 主从Reactor线程模型与Reactor多线程模型的最大区别就是有一组NIO线程处理连接、读写事件。 主从Reactor线程模型的特点是：服务端用于接收客户端连接的不再是个1个单独的NIO线程，而是一个独立的NIO线程池。Acceptor接收到客户端TCP连接请求处理完成后（可能包含接入认证等），将新创建的SocketChannel注册到IO线程池（sub reactor线程池）的某个IO线程上，由它负责SocketChannel的读写和编解码工作。Acceptor线程池仅仅只用于客户端的登陆、握手和安全认证，一旦链路建立成功，就将链路注册到后端subReactor线程池的IO线程上，由IO线程负责后续的IO操作。 即从多线程模型中由一个线程来监听连接事件和数据读写事件，拆分为一个线程监听连接事件，线程池的多个线程监听已经建立连接的套接字的数据读写事件，另外和多线程模型一样有专门的线程池处理真正的IO操作。 ### 各自应用场景 到这里你也许已经发现，一旦有请求到来(不管是几个同时到还是只有一个到)，都会调用对应IO处理函数处理，所以： （1）NIO适合处理连接数目特别多，但是连接比较短（轻操作）的场景，Jetty，Mina，ZooKeeper等都是基于java nio实现。 （2）BIO方式适用于连接数目比较小且固定的场景，这种方式对服务器资源要求比较高，并发局限于应用中。 ================================================ FILE: docs/java/feature.md ================================================ # Java 新特性总结 总结的这些新特性，都是自己觉得在开发中实际用得上的。 简单概括下就是： * JAVA1.3：普通的原始的JAVA，基本语法相信大家都见过了 * JAVA1.4：assert关键字 * JAVA5：枚举类型、泛型、自动拆装箱 * JAVA6： @Override注解 * JAVA7： <>符号、ARM支持、支持多catch * JAVA8：Lamda表达式，类型注解等 * JAVA9： 模块化、接口中的私有方法等 * JAVA10: 局部变量推断、整个JDK代码仓库、统一的垃圾回收接口、并行垃圾回收器G1、线程局部管控 ## Java5 新特性总结 ### 泛型 Generics 引用泛型之后，允许指定集合里元素的类型，免去了强制类型转换，并且能在编译时刻进行类型检查的好处。Parameterized Type作为参数和返回值，Generic是vararg、annotation、enumeration、collection的基石。 泛型可以带来如下的好处总结如下： 1. 类型安全：抛弃List、Map，使用List、Map给它们添加元素或者使用Iterator遍历时，编译期就可以给你检查出类型错误 2. 方法参数和返回值加上了Type: 抛弃List、Map，使用List、Map 3. 不需要类型转换：List list=new ArrayList(); 4. 类型通配符“?”： 假设一个打印List中元素的方法printList,我们希望任何类型T的List都可以被打印 ### 枚举类型 引入了枚举类型 ### 自动装箱拆箱（自动类型包装和解包）autoboxing & unboxing 简单的说是类型自动转换。自动装包：基本类型自动转为包装类（int ——Integer）自动拆包：包装类自动转为基本类型（Integer——int） ### 可变参数varargs(varargs number of arguments) 参数类型相同时，把重载函数合并到一起了。如： ```java public void test(object... objs){ for(Object obj:objs){ System.out.println(obj); } } ``` ### Annotations（重要） 它是java中的metadata（注释） 注解在JAVA5中就引入了。这是非常重要的特性。现在注解的应用已经随处可见。不过JAVA5的注解还不成熟，没法自定义注解。 ### 新的迭代语句 ```java for(int n:numbers){ //process } ``` ### 静态导入（import static ） 导入静态对象，可以省略些代码。不过这个也不常用。 ```java import static java.lang.System.out;//导入java.lang包下的System类的静态方法out; public class HelloWorld{ public static void main(String[] args){ out.print("Hello World!");//既是在这里不用再写成System.out.println("Hello World!")了，因为已经导入了这个静态方法out。 } } ``` ### 新的格式化方法java.util.Formatter） ```java formatter.format("Remaining account balance: $%.2f", balance); ``` ### 新的线程模型和并发库Thread Framework(重要) 最主要的就是引入了java.util.concurrent包，这个都是需要重点掌握的。 HashMap的替代者ConcurrentHashMap和ArrayList的替代者CopyOnWriteArrayList在大并发量读取时采用java.util.concurrent包里的一些类会让大家满意BlockingQueue、Callable、Executor、Semaphore ## Java6 新特性总结 ### Web Services 优先支持编写 XML web service 客户端程序。你可以用过简单的annotaion将你的API发布成.NET交互的web services. Mustang 添加了新的解析和 XML 在 Java object-mapping APIs中, 之前只在Java EE平台实现或者Java Web Services Pack中提供. ### Scripting 现在你可以在Java源代码中混入JavaScript了，这对开发原型很有有用，你也可以插入自己的脚本引擎。 ### JDBC4.0 JAVA6将联合绑定 Java DB (Apache Derby). JDBC 4.0 增加了许多特性例如支持XML作为SQL数据类型，更好的集成Binary Large OBjects (BLOBs) 和 Character Large OBjects (CLOBs) . ### UI优化 1. GUI 开发者可以有更多的技巧来使用 SwingWorker utility ，以帮助GUI应用中的多线程。, JTable 分类和过滤，以及添加splash闪屏。 2. Swing拥有更好的 look-and-feel , LCD 文本呈现, 整体GUI性能的提升。Java应用程序可以和本地平台更好的集成，例如访问平台的系统托盘和开始菜单。Mustang将Java插件技术和Java Web Start引擎统一了起来。 ### 监控管理增强 添加更多的诊断信息，绑定了不是很知名的 memory-heap 分析工具Jhat 来查看内核导出。 ### 编译API compiler API提供编程访问javac，可以实现进程内编译，动态产生Java代码 ### 自定义注解 Java tool和framework 提供商可以定义自己的 annotations ，并且内核支持自定义annotation的插件和执行处理器 ### 安全性 XML-数字签名(XML-DSIG) APIs 用于创建和操纵数字签名); 新的方法来访问本地平台的安全服务，例如本地Microsoft Windows for secure authentication and communicationnative 的Public Key Infrastructure (PKI) 和 cryptographic services， Java Generic Security Services (Java GSS) 和 Kerberos services for authentication, 以及访问 LDAP servers 来认证用户. ## Java7 新特性总结 ### switch中使用String java7以前在switch中只能使用number或enum,现在可以使用string了。 示例： ```java String s = "a"; switch (s) { case "a": System.out.println("is a"); break; case "b": System.out.println("is b"); break; default: System.out.println("is c"); break; } ``` ### 异常处理 * Throwable类增加addSuppressed方法和getSuppressed方法，支持原始异常中加入被抑制的异常。 * 异常抑制：在try和finally中同时抛出异常时，finally中抛出的异常会在异常栈中向上传递，而try中产生的原始异常会消失。 * 在Java7之前的版本，可以将原始异常保存，在finally中产生异常时抛出原始异常： ```java //java 7 以前 public void read(String filename) throws BaseException { FileInputStream input = null; IOException readException = null; try { input = new FileInputStream(filename); } catch (IOException ex) { readException = ex; //保存原始异常 } finally { if (input != null) { try { input.close(); } catch (IOException ex) { if (readException == null) { readException = ex; } } } if (readException != null) { throw new BaseException(readException); } } } //在Java7中的版本，可以使用addSuppressed方法记录被抑制的异常： public void read(String filename) throws IOException { FileInputStream input = null; IOException readException = null; try { input = new FileInputStream(filename); } catch (IOException ex) { readException = ex; } finally { if (input != null) { try { input.close(); } catch (IOException ex) { if (readException != null) { //此处的区别 readException.addSuppressed(ex); } else { readException = ex; } } } if (readException != null) { throw readException; } } } ``` ### try-with-resources java7以前对某些资源的操作是需要手动关闭，如InputStream，Writes，Sockets，Sql等，需要在finally中进行关闭资源的操作，现在不需要使用finally来保证打开的流被正确关闭，现在是自动完成的，会自动释放资源,确保每一个资源在处理完成后都会关闭，就不需要我们代码去close(); * 在采用try-with-resources方式后，不需要再次声明流的关闭。 * 可以使用try-with-resources的资源有：任何实现了java.lang.AutoCloseable接口和java.io.Closeable接口的对象。为了支持这个行为，所有可关闭的类将被修改为可以实现一个Closable（可关闭的）接口。 ``` java public interface Closeable extends AutoCloseable{} public abstract class Reader implements Readable, Closeable{} ``` 如果在try语句中写入了没有实现该接口的类，会提示： >The resource type File does not implement java.lang.AutoCloseable 示例： ```java //java 7 以前 OutputStream fos = null; try { fos = new FileOutputStream("D:/file"); } finally { fos.close(); } //java 7 以后 try(OutputStream fos = new FileOutputStream("D:/file");){ // 不需要再次指明fos.close(); } //try子句中可以管理多个资源 public void copyFile(String fromPath, String toPath) throws IOException { try ( InputStream input = new FileInputStream(fromPath); OutputStream output = new FileOutputStream(toPath) ) { byte[] buffer = new byte[8192]; int len = -1; while( (len=input.read(buffer))!=-1 ) { output.write(buffer, 0, len); } } } ``` ### 捕获多个异常 java7以前在一个方法抛出多个异常时，只能一个个的catch，这样代码会有多个catch,显得很不友好，现在只需一个catch语句，多个异常类型用"|"隔开。 示例： ```java //java 7 以前 try { result = field.get(obj); } catch (IllegalArgumentException e) { e.printStackTrace(); } catch (IllegalAccessException e) { e.printStackTrace(); } //java 7 以后 try { result = field.get(obj); } catch (IllegalArgumentException | IllegalAccessException e) { e.printStackTrace(); } ``` ### 泛型实例化类型自动推断 运用泛型实例化类型自动推断，对通用实例创建(diamond)的type引用进行了改进 示例： ```java //java 7 以前 List list = new ArrayList(); //java 7 以后 List list = new ArrayList<>(); ``` ### 增加二进制表示 Java7前支持十进制（123）、八进制（0123）、十六进制（0X12AB） Java7增加二进制表示（0B11110001、0b11110001） 示例： ```java int binary = 0b0001_1001; System.out.println("binary is :"+binary);                 binary is :25 ``` ### 数字中可添加分隔符 Java7中支持在数字中间增加'_'作为分隔符，分隔长int以及long（也支持double,float），显示更直观，如（12_123_456）。 下划线只能在数字中间，编译时编译器自动删除数字中的下划线。 示例： ```java int intOne = 1_000_000; long longOne = 1_000_000; double doubleOne = 1_000_000; float floatOne = 1_000_000; ``` ### 变长参数方法的优化 参数类型相同时，把重载函数合并到一起了 使用可变参数时，提升编译器的警告和错误信息 ```java public int sum(int... args) { int result = 0; for (int value : args) { result += value; } return result; } ``` ### 集合类的语法支持 ```java //java 7 以前 List list = new ArrayList(); list.add("item"); String item = list.get(0); Set set = new HashSet(); set.add("item"); Map map = new HashMap(); map.put("key", 1); int value = map.get("key"); //java 7 以后 List list = ["item"]; String item = list[0]; Set set = {"item"}; Map map = {"key" : 1}; int value = map["key"]; ``` ### 自动资源管理 Java中某些资源是需要手动关闭的，如InputStream，Writes，Sockets，Sql classes等。这个新的语言特性允许try语句本身申请更多的资源，这些资源作用于try代码块，并自动关闭。 ```java //java 7 以前 BufferedReader br = new BufferedReader(new FileReader(path)); try { return br.readLine(); } finally { br.close(); } //java 7 以后 try (BufferedReader br = new BufferedReader(new FileReader(path)) { return br.readLine(); } ``` ### 新增一些取环境信息的工具方法 ```java File System.getJavaIoTempDir() // IO临时文件夹 File System.getJavaHomeDir() // JRE的安装目录 File System.getUserHomeDir() // 当前用户目录 File System.getUserDir() // 启动java进程时所在的目录 ``` ## Java8 新特性总结 Java8 新增了非常多的特性，我们主要讨论以下几个： 1. Lambda 表达式 − Lambda允许把函数作为一个方法的参数(函数作为参数传递进方法中)。 2. 方法引用 − 方法引用提供了非常有用的语法，可以直接引用已有Java类或对象（实例）的方法或构造器。与lambda联合使用，方法引用可以使语言的构造更紧凑简洁，减少冗余代码。 3. 默认方法 − 默认方法就是一个在接口里面有了一个实现的方法。 4. 新工具 − 新的编译工具，如：Nashorn引擎 jjs、 类依赖分析器jdeps。 5. Stream API −新添加的Stream API（java.util.stream） 把真正的函数式编程风格引入到Java中。 6. Date Time API − 加强对日期与时间的处理。 7. Optional 类 − Optional 类已经成为 Java 8 类库的一部分，用来解决空指针异常。 8. Nashorn, JavaScript 引擎 − Java 8提供了一个新的Nashorn javascript引擎，它允许我们在JVM上运行特定的javascript应用。 ### Lambda表达式和函数式接口 Lambda表达式（也称为闭包）是Java 8中最大和最令人期待的语言改变。它允许我们将函数当成参数传递给某个方法，或者把代码本身当作数据处理：函数式开发者非常熟悉这些概念。很多JVM平台上的语言（Groovy、Scala等）从诞生之日就支持Lambda表达式，但是Java开发者没有选择，只能使用匿名内部类代替Lambda表达式。 Lambda的设计耗费了很多时间和很大的社区力量，最终找到一种折中的实现方案，可以实现简洁而紧凑的语言结构。最简单的Lambda表达式可由逗号分隔的参数列表、->符号和语句块组成，例如： ```java Arrays.asList( "a", "b", "d" ).forEach( e -> System.out.println( e ) ); ``` 在上面这个代码中的参数e的类型是由编译器推理得出的，你也可以显式指定该参数的类型，例如： ```java Arrays.asList( "a", "b", "d" ).forEach( ( String e ) -> System.out.println( e ) ); ``` 如果Lambda表达式需要更复杂的语句块，则可以使用花括号将该语句块括起来，类似于Java中的函数体，例如： ```java Arrays.asList( "a", "b", "d" ).forEach( e -> { System.out.print( e ); System.out.print( e ); } ); ``` Lambda表达式可以引用类成员和局部变量（会将这些变量隐式得转换成final的），例如下列两个代码块的效果完全相同： ```java String separator = ","; Arrays.asList( "a", "b", "d" ).forEach( ( String e ) -> System.out.print( e + separator ) ); //和 final String separator = ","; Arrays.asList( "a", "b", "d" ).forEach( ( String e ) -> System.out.print( e + separator ) ); ``` Lambda表达式有返回值，返回值的类型也由编译器推理得出。如果Lambda表达式中的语句块只有一行，则可以不用使用return语句，下列两个代码片段效果相同： ```java Arrays.asList( "a", "b", "d" ).sort( ( e1, e2 ) -> e1.compareTo( e2 ) ); //和 Arrays.asList( "a", "b", "d" ).sort( ( e1, e2 ) -> { int result = e1.compareTo( e2 ); return result; } ); ``` Lambda的设计者们为了让现有的功能与Lambda表达式良好兼容，考虑了很多方法，于是产生了函数接口这个概念。函数接口指的是只有一个函数的接口，这样的接口可以隐式转换为Lambda表达式。java.lang.Runnable和java.util.concurrent.Callable是函数式接口的最佳例子。在实践中，函数式接口非常脆弱：只要某个开发者在该接口中添加一个函数，则该接口就不再是函数式接口进而导致编译失败。为了克服这种代码层面的脆弱性，并显式说明某个接口是函数式接口，Java 8 提供了一个特殊的注解@FunctionalInterface（Java 库中的所有相关接口都已经带有这个注解了），举个简单的函数式接口的定义： ```java @FunctionalInterface public interface Functional { void method(); } ``` 不过有一点需要注意，默认方法和静态方法不会破坏函数式接口的定义，因此如下的代码是合法的。 ```java @FunctionalInterface public interface FunctionalDefaultMethods { void method(); default void defaultMethod() { System.out.print("defaultMethod"); } static void staticMethod(){ System.out.print("staticMethod"); } } ``` Lambda表达式作为Java 8的最大卖点，它有潜力吸引更多的开发者加入到JVM平台，并在纯Java编程中使用函数式编程的概念。如果你需要了解更多Lambda表达式的细节，可以参考[官方文档](https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html "官方文档")。 ### 接口的默认方法和静态方法 Java 8使用两个新概念扩展了接口的含义：默认方法和静态方法。默认方法使得接口有点类似traits，不过要实现的目标不一样。默认方法使得开发者可以在 不破坏二进制兼容性的前提下，往现存接口中添加新的方法，即不强制那些实现了该接口的类也同时实现这个新加的方法。 默认方法和抽象方法之间的区别在于抽象方法需要实现，而默认方法不需要。接口提供的默认方法会被接口的实现类继承或者覆写，例子代码如下： ```java private interface Defaulable { // Interfaces now allow default methods, the implementer may or // may not implement (override) them. default String notRequired() { return "Default implementation"; } } private static class DefaultableImpl implements Defaulable { } private static class OverridableImpl implements Defaulable { @Override public String notRequired() { return "Overridden implementation"; } } ``` Defaulable接口使用关键字default定义了一个默认方法notRequired()。DefaultableImpl类实现了这个接口，同时默认继承了这个接口中的默认方法；OverridableImpl类也实现了这个接口，但覆写了该接口的默认方法，并提供了一个不同的实现。 Java 8带来的另一个有趣的特性是在接口中可以定义静态方法，例子代码如下： ```java private interface DefaulableFactory { // Interfaces now allow static methods static Defaulable create( Supplier< Defaulable > supplier ) { return supplier.get(); } } ``` 下面的代码片段整合了默认方法和静态方法的使用场景： ```java public static void main( String[] args ) { Defaulable defaulable = DefaulableFactory.create( DefaultableImpl::new ); System.out.println( defaulable.notRequired() ); defaulable = DefaulableFactory.create( OverridableImpl::new ); System.out.println( defaulable.notRequired() ); } ``` 这段代码的输出结果如下： >Default implementation >Overridden implementation 由于JVM上的默认方法的实现在字节码层面提供了支持，因此效率非常高。默认方法允许在不打破现有继承体系的基础上改进接口。该特性在官方库中的应用是：给java.util.Collection接口添加新方法，如stream()、parallelStream()、forEach()和removeIf()等等。 尽管默认方法有这么多好处，但在实际开发中应该谨慎使用：在复杂的继承体系中，默认方法可能引起歧义和编译错误。如果你想了解更多细节，可以参考[官方文档](https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html "官方文档")。 ### 方法引用 方法引用使得开发者可以直接引用现存的方法、Java类的构造方法或者实例对象。方法引用和Lambda表达式配合使用，使得java类的构造方法看起来紧凑而简洁，没有很多复杂的模板代码。 西门的例子中，Car类是不同方法引用的例子，可以帮助读者区分四种类型的方法引用。 ```java public static class Car { public static Car create( final Supplier< Car > supplier ) { return supplier.get(); } public static void collide( final Car car ) { System.out.println( "Collided " + car.toString() ); } public void follow( final Car another ) { System.out.println( "Following the " + another.toString() ); } public void repair() { System.out.println( "Repaired " + this.toString() ); } } ``` 第一种方法引用的类型是构造器引用，语法是Class::new，或者更一般的形式：Class::new。注意：这个构造器没有参数。 ```java final Car car = Car.create( Car::new ); final List< Car > cars = Arrays.asList( car ); ``` 第二种方法引用的类型是静态方法引用，语法是Class::static_method。注意：这个方法接受一个Car类型的参数。 ```java cars.forEach( Car::collide ); ``` 第三种方法引用的类型是某个类的成员方法的引用，语法是Class::method，注意，这个方法没有定义入参： ```java cars.forEach( Car::repair ); ``` 第四种方法引用的类型是某个实例对象的成员方法的引用，语法是instance::method。注意：这个方法接受一个Car类型的参数： ```java final Car police = Car.create( Car::new ); cars.forEach( police::follow ); ``` 运行上述例子，可以在控制台看到如下输出（Car实例可能不同）： >Collided com.javacodegeeks.java8.method.references.MethodReferences$Car@7a81197d >Repaired com.javacodegeeks.java8.method.references.MethodReferences$Car@7a81197d >Following the com.javacodegeeks.java8.method.references.MethodReferences$Car@7a81197d 如果想了解和学习更详细的内容，可以参考[官方文档](https://docs.oracle.com/javase/tutorial/java/javaOO/methodreferences.html "官方文档")。 ### 重复注解 自从Java 5中引入注解以来，这个特性开始变得非常流行，并在各个框架和项目中被广泛使用。不过，注解有一个很大的限制是：在同一个地方不能多次使用同一个注解。Java 8打破了这个限制，引入了重复注解的概念，允许在同一个地方多次使用同一个注解。 在Java 8中使用@Repeatable注解定义重复注解，实际上，这并不是语言层面的改进，而是编译器做的一个trick，底层的技术仍然相同。可以利用下面的代码说明： ```java package com.javacodegeeks.java8.repeatable.annotations; import java.lang.annotation.ElementType; import java.lang.annotation.Repeatable; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; public class RepeatingAnnotations { @Target( ElementType.TYPE ) @Retention( RetentionPolicy.RUNTIME ) public @interface Filters { Filter[] value(); } @Target( ElementType.TYPE ) @Retention( RetentionPolicy.RUNTIME ) @Repeatable( Filters.class ) public @interface Filter { String value(); }; @Filter( "filter1" ) @Filter( "filter2" ) public interface Filterable { } public static void main(String[] args) { for( Filter filter: Filterable.class.getAnnotationsByType( Filter.class ) ) { System.out.println( filter.value() ); } } } ``` 正如我们所见，这里的Filter类使用@Repeatable(Filters.class)注解修饰，而Filters是存放Filter注解的容器，编译器尽量对开发者屏蔽这些细节。这样，Filterable接口可以用两个Filter注解注释（这里并没有提到任何关于Filters的信息）。 另外，反射API提供了一个新的方法：getAnnotationsByType()，可以返回某个类型的重复注解，例如Filterable.class.getAnnoation(Filters.class)将返回两个Filter实例，输出到控制台的内容如下所示： >filter1 >filter2 如果想了解和学习更详细的内容，可以参考[官方文档](https://docs.oracle.com/javase/tutorial/java/annotations/repeating.html "官方文档")。 ### 更好的类型推断 Java 8编译器在类型推断方面有很大的提升，在很多场景下编译器可以推导出某个参数的数据类型，从而使得代码更为简洁。例子代码如下： ```java package com.javacodegeeks.java8.type.inference; public class Value< T > { public static< T > T defaultValue() { return null; } public T getOrDefault( T value, T defaultValue ) { return ( value != null ) ? value : defaultValue; } } ``` 下列代码是Value类型的应用： ```java package com.javacodegeeks.java8.type.inference; public class TypeInference { public static void main(String[] args) { final Value< String > value = new Value<>(); value.getOrDefault( "22", Value.defaultValue() ); } } ``` 参数Value.defaultValue()的类型由编译器推导得出，不需要显式指明。在Java 7中这段代码会有编译错误，除非使用Value.defaultValue()。 ### 拓宽注解的应用场景 Java 8拓宽了注解的应用场景。现在，注解几乎可以使用在任何元素上：局部变量、接口类型、超类和接口实现类，甚至可以用在函数的异常定义上。下面是一些例子： ```java package com.javacodegeeks.java8.annotations; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import java.util.ArrayList; import java.util.Collection; public class Annotations { @Retention( RetentionPolicy.RUNTIME ) @Target( { ElementType.TYPE_USE, ElementType.TYPE_PARAMETER } ) public @interface NonEmpty { } public static class Holder< @NonEmpty T > extends @NonEmpty Object { public void method() throws @NonEmpty Exception { } } @SuppressWarnings( "unused" ) public static void main(String[] args) { final Holder< String > holder = new @NonEmpty Holder< String >(); @NonEmpty Collection< @NonEmpty String > strings = new ArrayList<>(); } } ``` ElementType.TYPE_USER和ElementType.TYPE_PARAMETER是Java 8新增的两个注解，用于描述注解的使用场景。Java 语言也做了对应的改变，以识别这些新增的注解。 ### Java编译器的新特性 为了在运行时获得Java程序中方法的参数名称，老一辈的Java程序员必须使用不同方法，例如Paranamer library。Java 8终于将这个特性规范化，在语言层面（使用反射API和Parameter.getName()方法）和字节码层面（使用新的javac编译器以及-parameters参数）提供支持。 ```java package com.javacodegeeks.java8.parameter.names; import java.lang.reflect.Method; import java.lang.reflect.Parameter; public class ParameterNames { public static void main(String[] args) throws Exception { Method method = ParameterNames.class.getMethod( "main", String[].class ); for( final Parameter parameter: method.getParameters() ) { System.out.println( "Parameter: " + parameter.getName() ); } } } ``` 在Java 8中这个特性是默认关闭的，因此如果不带-parameters参数编译上述代码并运行，则会输出如下结果： >Parameter: arg0 如果带-parameters参数，则会输出如下结果（正确的结果）： >Parameter: args 如果你使用Maven进行项目管理，则可以在maven-compiler-plugin编译器的配置项中配置-parameters参数： ``` yml org.apache.maven.plugins maven-compiler-plugin 3.1 -parameters 1.8 1.8 ``` ### Optional Java应用中最常见的bug就是空值异常。在Java 8之前，Google Guava引入了Optionals类来解决NullPointerException，从而避免源码被各种null检查污染，以便开发者写出更加整洁的代码。Java 8也将Optional加入了官方库。 Optional仅仅是一个容器：存放T类型的值或者null。它提供了一些有用的接口来避免显式的null检查，可以参考Java 8官方文档了解更多细节。 接下来看一点使用Optional的例子：可能为空的值或者某个类型的值： ```java Optional< String > fullName = Optional.ofNullable( null ); System.out.println( "Full Name is set? " + fullName.isPresent() ); System.out.println( "Full Name: " + fullName.orElseGet( () -> "[none]" ) ); System.out.println( fullName.map( s -> "Hey " + s + "!" ).orElse( "Hey Stranger!" ) ); ``` 如果Optional实例持有一个非空值，则isPresent()方法返回true，否则返回false；orElseGet()方法，Optional实例持有null，则可以接受一个lambda表达式生成的默认值；map()方法可以将现有的Opetional实例的值转换成新的值；orElse()方法与orElseGet()方法类似，但是在持有null的时候返回传入的默认值。 上述代码的输出结果如下： >Full Name is set? false >Full Name: [none] >Hey Stranger! 再看下另一个简单的例子： ```java Optional< String > firstName = Optional.of( "Tom" ); System.out.println( "First Name is set? " + firstName.isPresent() ); System.out.println( "First Name: " + firstName.orElseGet( () -> "[none]" ) ); System.out.println( firstName.map( s -> "Hey " + s + "!" ).orElse( "Hey Stranger!" ) ); System.out.println(); ``` 如果想了解和学习更详细的内容，可以参考[官方文档](https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html "官方文档")。 ### Streams 新增的Stream API（java.util.stream）将生成环境的函数式编程引入了Java库中。这是目前为止最大的一次对Java库的完善，以便开发者能够写出更加有效、更加简洁和紧凑的代码。 Steam API极大得简化了集合操作（后面我们会看到不止是集合），首先看下这个叫Task的类： ```java public class Streams { private enum Status { OPEN, CLOSED }; private static final class Task { private final Status status; private final Integer points; Task( final Status status, final Integer points ) { this.status = status; this.points = points; } public Integer getPoints() { return points; } public Status getStatus() { return status; } @Override public String toString() { return String.format( "[%s, %d]", status, points ); } } } ``` Task类有一个分数（或伪复杂度）的概念，另外还有两种状态：OPEN或者CLOSED。现在假设有一个task集合： ```java final Collection< Task > tasks = Arrays.asList( new Task( Status.OPEN, 5 ), new Task( Status.OPEN, 13 ), new Task( Status.CLOSED, 8 ) ); ``` 首先看一个问题：在这个task集合中一共有多少个OPEN状态的点？在Java 8之前，要解决这个问题，则需要使用foreach循环遍历task集合；但是在Java 8中可以利用steams解决：包括一系列元素的列表，并且支持顺序和并行处理。 ```java // Calculate total points of all active tasks using sum() final long totalPointsOfOpenTasks = tasks .stream() .filter( task -> task.getStatus() == Status.OPEN ) .mapToInt( Task::getPoints ) .sum(); System.out.println( "Total points: " + totalPointsOfOpenTasks ); ``` 运行这个方法的控制台输出是： >Total points: 18 这里有很多知识点值得说。首先，tasks集合被转换成steam表示；其次，在steam上的filter操作会过滤掉所有CLOSED的task；第三，mapToInt操作基于每个task实例的Task::getPoints方法将task流转换成Integer集合；最后，通过sum方法计算总和，得出最后的结果。 在学习下一个例子之前，还需要记住一些steams（[点此更多细节](https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html "点此更多细节")）的知识点。Steam之上的操作可分为中间操作和晚期操作。 中间操作会返回一个新的steam——执行一个中间操作（例如filter）并不会执行实际的过滤操作，而是创建一个新的steam，并将原steam中符合条件的元素放入新创建的steam。 晚期操作（例如forEach或者sum），会遍历steam并得出结果或者附带结果；在执行晚期操作之后，steam处理线已经处理完毕，就不能使用了。在几乎所有情况下，晚期操作都是立刻对steam进行遍历。 steam的另一个价值是创造性地支持并行处理（parallel processing）。对于上述的tasks集合，我们可以用下面的代码计算所有任务的点数之和： ```java // Calculate total points of all tasks final double totalPoints = tasks .stream() .parallel() .map( task -> task.getPoints() ) // or map( Task::getPoints ) .reduce( 0, Integer::sum ); System.out.println( "Total points (all tasks): " + totalPoints ); ``` 这里我们使用parallel方法并行处理所有的task，并使用reduce方法计算最终的结果。控制台输出如下： >Total points（all tasks）: 26.0 对于一个集合，经常需要根据某些条件对其中的元素分组。利用steam提供的API可以很快完成这类任务，代码如下： ```java // Group tasks by their status final Map< Status, List< Task > > map = tasks .stream() .collect( Collectors.groupingBy( Task::getStatus ) ); System.out.println( map ); ``` 控制台的输出如下： >{CLOSED=[[CLOSED, 8]], OPEN=[[OPEN, 5], [OPEN, 13]]} 最后一个关于tasks集合的例子问题是：如何计算集合中每个任务的点数在集合中所占的比重，具体处理的代码如下： ```java // Calculate the weight of each tasks (as percent of total points) final Collection< String > result = tasks .stream() // Stream< String > .mapToInt( Task::getPoints ) // IntStream .asLongStream() // LongStream .mapToDouble( points -> points / totalPoints ) // DoubleStream .boxed() // Stream< Double > .mapToLong( weigth -> ( long )( weigth * 100 ) ) // LongStream .mapToObj( percentage -> percentage + "%" ) // Stream< String> .collect( Collectors.toList() ); // List< String > System.out.println( result ); ``` 控制台输出结果如下： >[19%, 50%, 30%] 最后，正如之前所说，Steam API不仅可以作用于Java集合，传统的IO操作（从文件或者网络一行一行得读取数据）可以受益于steam处理，这里有一个小例子： ```java final Path path = new File( filename ).toPath(); try( Stream< String > lines = Files.lines( path, StandardCharsets.UTF_8 ) ) { lines.onClose( () -> System.out.println("Done!") ).forEach( System.out::println ); } ``` Stream的方法onClose 返回一个等价的有额外句柄的Stream，当Stream的close（）方法被调用的时候这个句柄会被执行。Stream API、Lambda表达式还有接口默认方法和静态方法支持的方法引用，是Java 8对软件开发的现代范式的响应。 [点此更多细节](https://docs.oracle.com/javase/8/docs/api/java/util/stream/package-summary.html "点此更多细节") ### Date/Time API(JSR 310) Java 8引入了新的Date-Time API(JSR 310)来改进时间、日期的处理。时间和日期的管理一直是最令Java开发者痛苦的问题。java.util.Date和后来的java.util.Calendar一直没有解决这个问题（甚至令开发者更加迷茫）。 因为上面这些原因，诞生了第三方库Joda-Time，可以替代Java的时间管理API。Java 8中新的时间和日期管理API深受Joda-Time影响，并吸收了很多Joda-Time的精华。新的java.time包包含了所有关于日期、时间、时区、Instant（跟日期类似但是精确到纳秒）、duration（持续时间）和时钟操作的类。新设计的API认真考虑了这些类的不变性（从java.util.Calendar吸取的教训），如果某个实例需要修改，则返回一个新的对象。 我们接下来看看java.time包中的关键类和各自的使用例子。首先，Clock类使用时区来返回当前的纳秒时间和日期。Clock可以替代System.currentTimeMillis()和TimeZone.getDefault()。 ```java // Get the system clock as UTC offset final Clock clock = Clock.systemUTC(); System.out.println( clock.instant() ); System.out.println( clock.millis() ); ``` 这个例子的输出结果是： >2014-04-12T15:19:29.282Z >1397315969360 第二，关注下LocalDate和LocalTime类。LocalDate仅仅包含ISO-8601日历系统中的日期部分；LocalTime则仅仅包含该日历系统中的时间部分。这两个类的对象都可以使用Clock对象构建得到。 ```java // Get the local date and local time final LocalDate date = LocalDate.now(); final LocalDate dateFromClock = LocalDate.now( clock ); System.out.println( date ); System.out.println( dateFromClock ); // Get the local date and local time final LocalTime time = LocalTime.now(); final LocalTime timeFromClock = LocalTime.now( clock ); System.out.println( time ); System.out.println( timeFromClock ); ``` 上述例子的输出结果如下： >2014-04-12 >2014-04-12 >11:25:54.568 >15:25:54.568 LocalDateTime类包含了LocalDate和LocalTime的信息，但是不包含ISO-8601日历系统中的时区信息。这里有一些关于LocalDate和LocalTime的例子： ```java // Get the local date/time final LocalDateTime datetime = LocalDateTime.now(); final LocalDateTime datetimeFromClock = LocalDateTime.now( clock ); System.out.println( datetime ); System.out.println( datetimeFromClock ); ``` 上述这个例子的输出结果如下： >2014-04-12T11:37:52.309 >2014-04-12T15:37:52.309 如果你需要特定时区的data/time信息，则可以使用ZoneDateTime，它保存有ISO-8601日期系统的日期和时间，而且有时区信息。下面是一些使用不同时区的例子： ```java // Get the zoned date/time final ZonedDateTime zonedDatetime = ZonedDateTime.now(); final ZonedDateTime zonedDatetimeFromClock = ZonedDateTime.now( clock ); final ZonedDateTime zonedDatetimeFromZone = ZonedDateTime.now( ZoneId.of( "America/Los_Angeles" ) ); System.out.println( zonedDatetime ); System.out.println( zonedDatetimeFromClock ); System.out.println( zonedDatetimeFromZone ); ``` 这个例子的输出结果是： >2014-04-12T11:47:01.017-04:00[America/New_York] >2014-04-12T15:47:01.017Z >2014-04-12T08:47:01.017-07:00[America/Los_Angeles] 最后看下Duration类，它持有的时间精确到秒和纳秒。这使得我们可以很容易得计算两个日期之间的不同，例子代码如下： ```java // Get duration between two dates final LocalDateTime from = LocalDateTime.of( 2014, Month.APRIL, 16, 0, 0, 0 ); final LocalDateTime to = LocalDateTime.of( 2015, Month.APRIL, 16, 23, 59, 59 ); final Duration duration = Duration.between( from, to ); System.out.println( "Duration in days: " + duration.toDays() ); System.out.println( "Duration in hours: " + duration.toHours() ); ``` 这个例子用于计算2014年4月16日和2015年4月16日之间的天数和小时数，输出结果如下： >Duration in days: 365 >Duration in hours: 8783 对于Java 8的新日期时间的总体印象还是比较积极的，一部分是因为Joda-Time的积极影响，另一部分是因为官方终于听取了开发人员的需求。如果想了解和学习更详细的内容，可以参考[官方文档](https://docs.oracle.com/javase/tutorial/datetime/index.html "官方文档")。 ### Nashorn JavaScript引擎 Java 8提供了新的Nashorn JavaScript引擎，使得我们可以在JVM上开发和运行JS应用。Nashorn JavaScript引擎是javax.script.ScriptEngine的另一个实现版本，这类Script引擎遵循相同的规则，允许Java和JavaScript交互使用，例子代码如下： ```java ScriptEngineManager manager = new ScriptEngineManager(); ScriptEngine engine = manager.getEngineByName( "JavaScript" ); System.out.println( engine.getClass().getName() ); System.out.println( "Result:" + engine.eval( "function f() { return 1; }; f() + 1;" ) ); ``` 这个代码的输出结果如下： >jdk.nashorn.api.scripting.NashornScriptEngine >Result: 2 ### Base64 对Base64编码的支持已经被加入到Java 8官方库中，这样不需要使用第三方库就可以进行Base64编码，例子代码如下： ```java package com.javacodegeeks.java8.base64; import java.nio.charset.StandardCharsets; import java.util.Base64; public class Base64s { public static void main(String[] args) { final String text = "Base64 finally in Java 8!"; final String encoded = Base64 .getEncoder() .encodeToString( text.getBytes( StandardCharsets.UTF_8 ) ); System.out.println( encoded ); final String decoded = new String( Base64.getDecoder().decode( encoded ), StandardCharsets.UTF_8 ); System.out.println( decoded ); } } ``` 这个例子的输出结果如下： >QmFzZTY0IGZpbmFsbHkgaW4gSmF2YSA4IQ== >Base64 finally in Java 8! 新的Base64API也支持URL和MINE的编码解码。 (Base64.getUrlEncoder() / Base64.getUrlDecoder(), Base64.getMimeEncoder() / Base64.getMimeDecoder())。 ### 并行数组 Java8版本新增了很多新的方法，用于支持并行数组处理。最重要的方法是parallelSort()，可以显著加快多核机器上的数组排序。下面的例子论证了parallexXxx系列的方法： ```java package com.javacodegeeks.java8.parallel.arrays; import java.util.Arrays; import java.util.concurrent.ThreadLocalRandom; public class ParallelArrays { public static void main( String[] args ) { long[] arrayOfLong = new long [ 20000 ]; Arrays.parallelSetAll( arrayOfLong, index -> ThreadLocalRandom.current().nextInt( 1000000 ) ); Arrays.stream( arrayOfLong ).limit( 10 ).forEach( i -> System.out.print( i + " " ) ); System.out.println(); Arrays.parallelSort( arrayOfLong ); Arrays.stream( arrayOfLong ).limit( 10 ).forEach( i -> System.out.print( i + " " ) ); System.out.println(); } } ``` 上述这些代码使用parallelSetAll()方法生成20000个随机数，然后使用parallelSort()方法进行排序。这个程序会输出乱序数组和排序数组的前10个元素。上述例子的代码输出的结果是： >Unsorted: 591217 891976 443951 424479 766825 351964 242997 642839 119108 552378 >Sorted: 39 220 263 268 325 607 655 678 723 793 ### 并发性 基于新增的lambda表达式和steam特性，为Java 8中为java.util.concurrent.ConcurrentHashMap类添加了新的方法来支持聚焦操作；另外，也为java.util.concurrentForkJoinPool类添加了新的方法来支持通用线程池操作 Java 8还添加了新的java.util.concurrent.locks.StampedLock类，用于支持基于容量的锁——该锁有三个模型用于支持读写操作（可以把这个锁当做是java.util.concurrent.locks.ReadWriteLock的替代者）。 在java.util.concurrent.atomic包中也新增了不少工具类，列举如下： 1. DoubleAccumulator 2. DoubleAdder 3. LongAccumulator 4. LongAdder ### 新的Java工具 #### Nashorn引擎：jjs jjs是一个基于标准Nashorn引擎的命令行工具，可以接受js源码并执行。例如，我们写一个func.js文件，内容如下： ```javascript function f() { return 1; }; print( f() + 1 ); ``` 可以在命令行中执行这个命令：jjs func.js，控制台输出结果是： >2 如果需要了解细节，可以参考[官方文档](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jjs.html "官方文档")。 #### 类依赖分析器：jdeps jdeps是一个相当棒的命令行工具，它可以展示包层级和类层级的Java类依赖关系，它以.class文件、目录或者Jar文件为输入，然后会把依赖关系输出到控制台。 我们可以利用jedps分析下Spring Framework库，为了让结果少一点，仅仅分析一个JAR文件：org.springframework.core-3.0.5.RELEASE.jar。 ```java jdeps org.springframework.core-3.0.5.RELEASE.jar ``` 这个命令会输出很多结果，我们仅看下其中的一部分：依赖关系按照包分组，如果在classpath上找不到依赖，则显示"not found". ``` s org.springframework.core-3.0.5.RELEASE.jar -> C:\Program Files\Java\jdk1.8.0\jre\lib\rt.jar org.springframework.core (org.springframework.core-3.0.5.RELEASE.jar) -> java.io -> java.lang -> java.lang.annotation -> java.lang.ref -> java.lang.reflect -> java.util -> java.util.concurrent -> org.apache.commons.logging not found -> org.springframework.asm not found -> org.springframework.asm.commons not found org.springframework.core.annotation (org.springframework.core-3.0.5.RELEASE.jar) -> java.lang -> java.lang.annotation -> java.lang.reflect -> java.util ``` 如果需要了解细节，可以参考[官方文档](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jdeps.html "官方文档")。 ### JVM的新特性 使用Metaspace（JEP 122）代替持久代（PermGen space）。在JVM参数方面，使用-XX:MetaSpaceSize和-XX:MaxMetaspaceSize代替原来的-XX:PermSize和-XX:MaxPermSize。 ## Java 9 新特性总结 1. 模块系统：模块是一个包的容器，Java 9 最大的变化之一是引入了模块系统（Jigsaw 项目）。 2. REPL (JShell)：交互式编程环境。 3. HTTP 2 客户端：HTTP/2标准是HTTP协议的最新版本，新的 HTTPClient API 支持 WebSocket 和 HTTP2 流以及服务器推送特性。 4. 改进的 Javadoc：Javadoc 现在支持在 API 文档中的进行搜索。另外，Javadoc 的输出现在符合兼容 HTML5 标准。 5. 多版本兼容 JAR 包：多版本兼容 JAR 功能能让你创建仅在特定版本的 Java 环境中运行库程序时选择使用的 class 版本。 6. 集合工厂方法：List，Set 和 Map 接口中，新的静态工厂方法可以创建这些集合的不可变实例。 7. 私有接口方法：在接口中使用private私有方法。我们可以使用 private 访问修饰符在接口中编写私有方法。 8. 进程 API: 改进的 API 来控制和管理操作系统进程。引进 java.lang.ProcessHandle 及其嵌套接口 Info 来让开发者逃离时常因为要获取一个本地进程的 PID 而不得不使用本地代码的窘境。 9. 改进的 Stream API：改进的 Stream API 添加了一些便利的方法，使流处理更容易，并使用收集器编写复杂的查询。 10. 改进 try-with-resources：如果你已经有一个资源是 final 或等效于 final 变量,您可以在 try-with-resources 语句中使用该变量，而无需在 try-with-resources 语句中声明一个新变量。 11. 改进的弃用注解 @Deprecated：注解 @Deprecated 可以标记 Java API 状态，可以表示被标记的 API 将会被移除，或者已经破坏。 12. 改进钻石操作符(Diamond Operator) ：匿名类可以使用钻石操作符(Diamond Operator)。 13. 改进 Optional 类：java.util.Optional 添加了很多新的有用方法，Optional 可以直接转为 stream。 14. 多分辨率图像 API：定义多分辨率图像API，开发者可以很容易的操作和展示不同分辨率的图像了。 15. 改进的 CompletableFuture API ： CompletableFuture 类的异步机制可以在 ProcessHandle.onExit 方法退出时执行操作。 16. 轻量级的 JSON API：内置了一个轻量级的JSON API 17. 响应式流（Reactive Streams) API: Java 9中引入了新的响应式流 API 来支持 Java 9 中的响应式编程。 ### Java9 新特性之---目录结构 包含jdk8及以前的jdk版本，所有目录结构以及目录含义如图： ![java](./../img/java1.png "java") ![java](./../img/java2.png "java") jdk9之后，目录结构发生变化如图： ![java](./../img/java3.png "java") 这个新特性只要了解下就可以了，这个目录结构是方便为了接下来新特性做保证 ### 模块化 一个大型的项目，比如淘宝商城等，都会包含多个模块，比如订单模块，前台模块，后台管理模块，广告位模块，会员模块.....等等，各个模块之间会相互调用，不过这种情况下会很少，只针对特殊情况，如果一个项目有30个模块系统进行开发，但是只要某个单独模块运行时，都会带动所有的模块，这样对于jvm来说在内存和性能上会很低，所以，java9提供了这一个特性，某一个模块运行的时候，jvm只会启动和它有依赖的模块，并不会加载所有的模块到内存中，这样性能大大的提高了。写法上如下： ![java](./../img/java4.png "java") 一个项目中的两个模块，模块之间通过module-info.java来关联，在IDEA编辑器右键创建package-info.java ![java](./../img/java5.png "java") 在这个两个模块java9Demo和java9Test中，java9demo编写一个实体类Person,在java9Test调用这样一个过程 这个是java9Demo 将 java9Test 模块需要的文件导出 exports 把它所在的包导出 ```java module java9Demo{ requires com.mdxl.layer_cj.entity; } ``` 然后在java9Test模块中创建一个package-info.java,引入java9Demo模块导出包名 ```java module java9Test{ requires java9Demo; } ``` 这样就可以直接在java9Test中引入Person实体类了，这只是一个简单的例子。exports 控制着那些包可以被模块访问，所以不被导出的包不能被其他模块访问 ### JShell工具 怎么理解，怎么用呢？这个只是针对于java9来说，相当于cmd工具，你可以和cmd一样，直接写方法等等，不过我认为只是适用于初学者做一些最简单的运算和写一些方法： 在cmd中打开这个工具： ```s $ jshell | Welcome to JShell -- Version 9-ea | For an introduction type: /help intro jshell> ``` 查看 JShell 命令 输入 /help 可以查看 JShell相关的命令： ``` s jshell> /help | Type a Java language expression, statement, or declaration. | Or type one of the following commands: | /list [|-all|-start] | list the source you have typed | /edit | edit a source entry referenced by name or id | /drop | delete a source entry referenced by name or id | /save [-all|-history|-start] | Save snippet source to a file. | /open | open a file as source input | /vars [|-all|-start] | list the declared variables and their values | /methods [|-all|-start] | list the declared methods and their signatures | /types [|-all|-start] | list the declared types | /imports | list the imported items ``` 执行 JShell 命令 /imports 命令用于查看已导入的包： ``` s jshell> /imports | import java.io.* | import java.math.* | import java.net.* | import java.nio.file.* | import java.util.* | import java.util.concurrent.* | import java.util.function.* | import java.util.prefs.* | import java.util.regex.* | import java.util.stream.* jshell> ``` 等等，我认为只适用于初学者学习java不用其他编辑工具就可以学习java ### HTTP 2 客户端 JDK9之前提供HttpURLConnection API来实现Http访问功能，但是这个类基本很少使用，一般都会选择Apache的Http Client，此次在Java 9的版本中引入了一个新的package:java.net.http，里面提供了对Http访问很好的支持，不仅支持Http1.1而且还支持HTTP2（什么是HTTP2？请参见HTTP2的时代来了...），以及WebSocket，据说性能特别好。 ![java](./../img/java6.png "java") 注意：新的 HttpClient API 在 Java 9 中以所谓的孵化器模块交付。也就是说，这套 API 不能保证 100% 完成。 ### 改进 Javadoc javadoc 工具可以生成 Java 文档， Java 9 的 javadoc 的输出现在符合兼容 HTML5 标准。 ``` s //java 9 之前 C:\JAVA>javadoc -d C:/JAVA Tester.java //java 9 之后 C:\JAVA> javadoc -d C:/JAVA -html5 Tester.java ``` ### 多版本兼容 jar 包 多版本兼容 JAR 功能能让你创建仅在特定版本的 Java 环境中运行库程序时选择使用的 class 版本。 通过 --release 参数指定编译版本。 具体的变化就是 META-INF 目录下 MANIFEST.MF 文件新增了一个属性： >Multi-Release: true 然后 META-INF 目录下还新增了一个 versions 目录，如果是要支持 java9，则在 versions 目录下有 9 的目录。 ``` s multirelease.jar ├── META-INF │ └── versions │ └── 9 │ └── multirelease │ └── Helper.class ├── multirelease ├── Helper.class └── Main.class ``` ### 集合工厂方法 Java 9 List，Set 和 Map 接口中，新的静态工厂方法可以创建这些集合的不可变实例。 这些工厂方法可以以更简洁的方式来创建集合。 旧方法创建集合: ```java import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; public class Tester { public static void main(String []args) { Set set = new HashSet<>(); set.add("A"); set.add("B"); set.add("C"); set = Collections.unmodifiableSet(set); System.out.println(set); List list = new ArrayList<>(); list.add("A"); list.add("B"); list.add("C"); list = Collections.unmodifiableList(list); System.out.println(list); Map map = new HashMap<>(); map.put("A","Apple"); map.put("B","Boy"); map.put("C","Cat"); map = Collections.unmodifiableMap(map); System.out.println(map); } } ``` 执行输出结果为： ``` s [A, B, C] [A, B, C] {A=Apple, B=Boy, C=Cat} ``` 新方法创建集合： Java 9 中，以下方法被添加到 List，Set 和 Map 接口以及它们的重载对象。 ```java static List of(E e1, E e2, E e3); static Set of(E e1, E e2, E e3); static Map of(K k1, V v1, K k2, V v2, K k3, V v3); static Map ofEntries(Map.Entry... entries) ``` * List 和 Set 接口, of(...) 方法重载了 0 ~ 10 个参数的不同方法 。 * Map 接口, of(...) 方法重载了 0 ~ 10 个参数的不同方法 。 * Map 接口如果超过 10 个参数, 可以使用 ofEntries(...) 方法。 新方法创建集合: ```java import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.AbstractMap; import java.util.Map; import java.util.Set; public class Tester { public static void main(String []args) { Set set = Set.of("A", "B", "C"); System.out.println(set); List list = List.of("A", "B", "C"); System.out.println(list); Map map = Map.of("A","Apple","B","Boy","C","Cat"); System.out.println(map); Map map1 = Map.ofEntries ( new AbstractMap.SimpleEntry<>("A","Apple"), new AbstractMap.SimpleEntry<>("B","Boy"), new AbstractMap.SimpleEntry<>("C","Cat")); System.out.println(map1); } } ``` 输出结果为： ``` s [A, B, C] [A, B, C] {A=Apple, B=Boy, C=Cat} {A=Apple, B=Boy, C=Cat} ``` ### 私有接口方法 在 Java 8之前，接口可以有常量变量和抽象方法。 我们不能在接口中提供方法实现。如果我们要提供抽象方法和非抽象方法（方法与实现）的组合，那么我们就得使用抽象类。 ```java public interface Tests{ //java 7 及以前特性 常量 抽象方法 String str='hello wrold'; void show(T str); //java 8 特性 默认方法 静态方法 default void def(){ System.out.print("default method"); } static void sta(){ System.out.print("static method"); } //java 9 特性 私有方法 私有静态方法 private void pri(){ System.out.print("private method"); } private static void pri_sta(){ System.out.print("private static method"); } } ``` ### 改进的进程 API 在 Java 9 之前，Process API 仍然缺乏对使用本地进程的基本支持，例如获取进程的 PID 和所有者，进程的开始时间，进程使用了多少 CPU 时间，多少本地进程正在运行等。 Java 9 向 Process API 添加了一个名为 ProcessHandle 的接口来增强 java.lang.Process 类。 ProcessHandle 接口的实例标识一个本地进程，它允许查询进程状态并管理进程。 ProcessHandle 嵌套接口 Info 来让开发者逃离时常因为要获取一个本地进程的 PID 而不得不使用本地代码的窘境。 我们不能在接口中提供方法实现。如果我们要提供抽象方法和非抽象方法（方法与实现）的组合，那么我们就得使用抽象类。 ProcessHandle 接口中声明的 onExit() 方法可用于在某个进程终止时触发某些操作。 ```java import java.time.ZoneId; import java.util.stream.Stream; import java.util.stream.Collectors; import java.io.IOException; public class Tester { public static void main(String[] args) throws IOException { ProcessBuilder pb = new ProcessBuilder("notepad.exe"); String np = "Not Present"; Process p = pb.start(); ProcessHandle.Info info = p.info(); System.out.printf("Process ID : %s%n", p.pid()); System.out.printf("Command name : %s%n", info.command().orElse(np)); System.out.printf("Command line : %s%n", info.commandLine().orElse(np)); System.out.printf("Start time: %s%n", info.startInstant().map(i -> i.atZone(ZoneId.systemDefault()) .toLocalDateTime().toString()).orElse(np)); System.out.printf("Arguments : %s%n", info.arguments().map(a -> Stream.of(a).collect( Collectors.joining(" "))).orElse(np)); System.out.printf("User : %s%n", info.user().orElse(np)); } } ``` 以上实例执行输出结果为： ``` s Process ID : 5800 Command name : C:\Windows\System32\notepad.exe Command line : Not Present Start time: 2017-11-04T21:35:03.626 Arguments : Not Present User: administrator ``` ### 改进的 Stream API Java 9 改进的 Stream API 添加了一些便利的方法，使流处理更容易，并使用收集器编写复杂的查询。 Java 9 为 Stream 新增了几个方法：dropWhile、takeWhile、ofNullable，为 iterate 方法新增了一个重载方法。 #### takeWhile 方法 语法 ```java default Stream takeWhile(Predicate predicate) ``` takeWhile() 方法使用一个断言作为参数，返回给定 Stream 的子集直到断言语句第一次返回 false。如果第一个值不满足断言条件，将返回一个空的 Stream。 takeWhile() 方法在有序的 Stream 中，takeWhile 返回从开头开始的尽量多的元素；在无序的 Stream 中，takeWhile 返回从开头开始的符合 Predicate 要求的元素的子集。 ```java import java.util.stream.Stream; public class Tester { public static void main(String[] args) { Stream.of("a","b","c","","e","f").takeWhile(s->!s.isEmpty()) .forEach(System.out::print); } } ``` 以上实例 takeWhile 方法在碰到空字符串时停止循环输出，执行输出结果为： ```s abc ``` #### dropWhile 方法 语法: ```java default Stream dropWhile(Predicate predicate) ``` dropWhile 方法和 takeWhile 作用相反的，使用一个断言作为参数，直到断言语句第一次返回 true 才返回给定 Stream 的子集。 ```java import java.util.stream.Stream; public class Tester { public static void main(String[] args) { Stream.of("a","b","c","","e","f").dropWhile(s-> !s.isEmpty()) .forEach(System.out::print); } } ``` 以上实例 dropWhile 方法在碰到空字符串时开始循环输出，执行输出结果为： ```s ef ``` #### iterate 方法 语法: ```java static Stream iterate(T seed, Predicate hasNext, UnaryOperator next) ``` 方法允许使用初始种子值创建顺序（可能是无限）流，并迭代应用指定的下一个方法。 当指定的 hasNext 的 predicate 返回 false 时，迭代停止。 ```java java.util.stream.IntStream; public class Tester { public static void main(String[] args) { IntStream.iterate(3, x -> x < 10, x -> x+ 3).forEach(System.out::println); } } ``` 执行输出结果为： ``` s 3 6 9 ``` #### ofNullable 方法 语法: ```java static Stream ofNullable(T t) ``` ofNullable 方法可以预防 NullPointerExceptions 异常， 可以通过检查流来避免 null 值。 如果指定元素为非 null，则获取一个元素并生成单个元素流，元素为 null 则返回一个空流。 ```java import java.util.stream.Stream; public class Tester { public static void main(String[] args) { long count = Stream.ofNullable(100).count(); System.out.println(count); count = Stream.ofNullable(null).count(); System.out.println(count); } } ``` 执行输出结果为： ``` s 1 0 ``` ### 改进的 try-with-resources try-with-resources 是 JDK 7 中一个新的异常处理机制，它能够很容易地关闭在 try-catch 语句块中使用的资源。所谓的资源（resource）是指在程序完成后，必须关闭的对象。try-with-resources 语句确保了每个资源在语句结束时关闭。所有实现了 java.lang.AutoCloseable 接口（其中，它包括实现了 java.io.Closeable 的所有对象），可以使用作为资源。 try-with-resources 声明在 JDK 9 已得到改进。如果你已经有一个资源是 final 或等效于 final 变量,您可以在 try-with-resources 语句中使用该变量，而无需在 try-with-resources 语句中声明一个新变量。 ```java import java.io.BufferedReader; import java.io.IOException; import java.io.Reader; import java.io.StringReader; public class Tester { public static void main(String[] args) throws IOException { System.out.println(readData("test")); } static String readData(String message) throws IOException { Reader inputString = new StringReader(message); BufferedReader br = new BufferedReader(inputString); try (BufferedReader br1 = br) { return br1.readLine(); } } } ``` 输出结果为： ``` s test ``` 以上实例中我们需要在 try 语句块中声明资源 br1，然后才能使用它。 在 Java 9 中，我们不需要声明资源 br1 就可以使用它，并得到相同的结果。 ```java import java.io.BufferedReader; import java.io.IOException; import java.io.Reader; import java.io.StringReader; public class Tester { public static void main(String[] args) throws IOException { System.out.println(readData("test")); } static String readData(String message) throws IOException { Reader inputString = new StringReader(message); BufferedReader br = new BufferedReader(inputString); try (br) { return br.readLine(); } } } ``` 执行输出结果为： ``` s test ``` 在处理必须关闭的资源时，使用try-with-resources语句替代try-finally语句。 生成的代码更简洁，更清晰，并且生成的异常更有用。 try-with-resources语句在编写必须关闭资源的代码时会更容易，也不会出错，而使用try-finally语句实际上是不可能的。 ### 改进的 @Deprecated 注解 注解 @Deprecated 可以标记 Java API 状态，可以是以下几种： 使用它存在风险，可能导致错误 可能在未来版本中不兼容 可能在未来版本中删除 一个更好和更高效的方案已经取代它。 Java 9 中注解增加了两个新元素：since 和 forRemoval。 * since: 元素指定已注解的API元素已被弃用的版本。 * forRemoval: 元素表示注解的 API 元素在将来的版本中被删除，应该迁移 API。 ### 钻石操作符的升级 钻石操作符是在 java 7 中引入的，可以让代码更易读，但它不能用于匿名的内部类。 在 java 9 中， 它可以与匿名的内部类一起使用，从而提高代码的可读性。 ```java //java6 及之前 Map map6=new HashMap(); //java7和8 <>没有了数据类型 Map map6=new HashMap<>(); //java9 添加了匿名内部类的功能 后面添加大括号{} 可以做一些细节操作 Map map6=new HashMap<>(){}; ``` ### 改进的 Optional 类 Optional 类在 Java 8 中引入，Optional 类的引入很好的解决空指针异常。。在 java 9 中, 添加了三个方法来改进它的功能： * stream() * ifPresentOrElse() * or() #### stream() 方法 语法: ```java public Stream stream() ``` stream 方法的作用就是将 Optional 转为一个 Stream，如果该 Optional 中包含值，那么就返回包含这个值的 Stream，否则返回一个空的 Stream（Stream.empty()）。 ```java import java.util.Arrays; import java.util.List; import java.util.Optional; import java.util.stream.Collectors; import java.util.stream.Stream; public class Tester { public static void main(String[] args) { List> list = Arrays.asList ( Optional.empty(), Optional.of("A"), Optional.empty(), Optional.of("B")); //filter the list based to print non-empty values //if optional is non-empty, get the value in stream, otherwise return empty List filteredList = list.stream() .flatMap(o -> o.isPresent() ? Stream.of(o.get()) : Stream.empty()) .collect(Collectors.toList()); //Optional::stream method will return a stream of either one //or zero element if data is present or not. List filteredListJava9 = list.stream() .flatMap(Optional::stream) .collect(Collectors.toList()); System.out.println(filteredList); System.out.println(filteredListJava9); } } ``` 执行输出结果为： ``` s [A, B] [A, B] ``` #### ifPresentOrElse() 方法 语法: ```java public void ifPresentOrElse(Consumer action, Runnable emptyAction) ``` ifPresentOrElse 方法的改进就是有了 else，接受两个参数 Consumer 和 Runnable。 ifPresentOrElse 方法的用途是，如果一个 Optional 包含值，则对其包含的值调用函数 action，即 action.accept(value)，这与 ifPresent 一致；与 ifPresent 方法的区别在于，ifPresentOrElse 还有第二个参数 emptyAction —— 如果 Optional 不包含值，那么 ifPresentOrElse 便会调用 emptyAction，即 emptyAction.run()。 ```java import java.util.Optional; public class Tester { public static void main(String[] args) { Optional optional = Optional.of(1); optional.ifPresentOrElse( x -> System.out.println("Value: " + x),() -> System.out.println("Not Present.")); optional = Optional.empty(); optional.ifPresentOrElse( x -> System.out.println("Value: " + x),() -> System.out.println("Not Present.")); } } ``` 执行输出结果为： ``` s Value: 1 Not Present. ``` #### or() 方法 语法: ```java public Optional or(Supplier> supplier) ``` 如果值存在，返回 Optional 指定的值，否则返回一个预设的值。 ```java import java.util.Optional; import java.util.function.Supplier; public class Tester { public static void main(String[] args) { Optional optional1 = Optional.of("Mahesh"); Supplier> supplierString = () -> Optional.of("Not Present"); optional1 = optional1.or( supplierString); optional1.ifPresent( x -> System.out.println("Value: " + x)); optional1 = Optional.empty(); optional1 = optional1.or( supplierString); optional1.ifPresent( x -> System.out.println("Value: " + x)); } } ``` 执行输出结果为： ``` s Value: Mahesh Value: Not Present ``` ### 多分辨率图像 API Java 9 定义多分辨率图像 API，开发者可以很容易的操作和展示不同分辨率的图像了。 以下是多分辨率图像的主要操作方法： * Image getResolutionVariant(double destImageWidth, double destImageHeight) − 获取特定分辨率的图像变体-表示一张已知分辨率单位为DPI的特定尺寸大小的逻辑图像，并且这张图像是最佳的变体。。 * List getResolutionVariants() − 返回可读的分辨率的图像变体列表。 ### 改进的 CompletableFuture API Java 8 引入了 CompletableFuture 类，可能是 java.util.concurrent.Future 明确的完成版（设置了它的值和状态），也可能被用作java.util.concurrent.CompleteStage 。支持 future 完成时触发一些依赖的函数和动作。Java 9 引入了一些CompletableFuture 的改进： Java 9 对 CompletableFuture 做了改进： * 支持 delays 和 timeouts * 提升了对子类化的支持 * 新的工厂方法 #### 支持 delays 和 timeouts ```java public CompletableFuture completeOnTimeout(T value, long timeout, TimeUnit unit) ``` 在 timeout（单位在 java.util.concurrent.Timeunits units 中，比如 MILLISECONDS ）前以给定的 value 完成这个 CompletableFutrue。返回这个 CompletableFutrue。 ```java public CompletableFuture orTimeout(long timeout, TimeUnit unit) ``` 如果没有在给定的 timeout 内完成，就以 java.util.concurrent.TimeoutException 完成这个 CompletableFutrue，并返回这个 CompletableFutrue #### 增强了对子类化的支持 做了许多改进使得 CompletableFuture 可以被更简单的继承。比如，你也许想重写新的 public Executor defaultExecutor() 方法来代替默认的 executor。 另一个新的使子类化更容易的方法是： ```java public CompletableFuture newIncompleteFuture() ``` ### 新的工厂方法 ``` s Java 8引入了 CompletableFuture completedFuture(U value) 工厂方法来返回一个已经以给定 value 完成了的 CompletableFuture。Java 9以 一个新的 CompletableFuture failedFuture(Throwable ex) 来补充了这个方法，可以返回一个以给定异常完成的 CompletableFuture。 除此以外，Java 9 引入了下面这对 stage-oriented 工厂方法，返回完成的或异常完成的 completion stages: * CompletionStage completedStage(U value): 返回一个新的以指定 value 完成的CompletionStage ，并且只支持 CompletionStage 里的接口。 * CompletionStage failedStage(Throwable ex): 返回一个新的以指定异常完成的CompletionStage ，并且只支持 CompletionStage 里的接口。 ``` ## Java 10 新特性 1. 局部变量推断 2. 整个JDK代码仓库 3. 统一的垃圾回收接口 4. 并行垃圾回收器G1 5. 线程局部管控 ### 局部变量推断 它向 Java 中引入在其他语言中很常见的 var ，比如 JavaScript 。只要编译器可以推断此种类型，你不再需要专门声明一个局部变量的类型。 开发者将能够声明变量而不必指定关联的类型。比如： ``` java List list = new ArrayList (); Stream stream = getStream(); ``` 它可以简化为： ``` java var list = new ArrayList(); var stream = getStream(); ``` 局部变量类型推断将引入“ var ”关键字的使用，而不是要求明确指定变量的类型，我们俗称“语法糖”。 这就消除了我们之前必须执行的 ArrayList 类型定义的重复。 其实我们在JDK7，我们需要： ``` java List list = new ArrayList (); ``` 但是在JDK8，我们只需要： ``` java List list = new ArrayList <>(); ``` 所以这是一个逐步的升级。也是人性化的表现与提升。 有趣的是，需要注意 var 不能成为一个关键字，而是一个保留字。这意味着你仍然可以使用 var 作为一个变量，方法或包名，但是现在（尽管我确定你绝不会）你不能再有一个类被调用。 局部变量类型推荐仅限于如下使用场景： * 局部变量初始化 * for循环内部索引变量 * 传统的for循环声明变量 Java官方表示，它不能用于以下几个地方： * 方法参数 * 构造函数参数 * 方法返回类型 * 字段 * 捕获表达式（或任何其他类型的变量声明） 注意： Java的var和JavaScript的完全不同，不要这样去类比。Java的var是用于局部类型推断的，而不是JS那样的动态类型，所以下面这个样子是不行的： ``` java var a = 10; a = "abc"; //error! ``` 其次，这个var只能用于局部变量声明，在其他地方使用都是错误的。 ``` java class C {     public var a = 10; //error     public var f() { //error         return 10;     } } ``` ### 整合 JDK 代码仓库 为了简化开发流程，Java 10 中会将多个代码库合并到一个代码仓库中。 在已发布的 Java 版本中，JDK 的整套代码根据不同功能已被分别存储在多个 Mercurial 存储库，这八个 Mercurial 存储库分别是：root、corba、hotspot、jaxp、jaxws、jdk、langtools、nashorn。 虽然以上八个存储库之间相互独立以保持各组件代码清晰分离，但同时管理这些存储库存在许多缺点，并且无法进行相关联源代码的管理操作。其中最重要的一点是，涉及多个存储库的变更集无法进行原子提交 （atomic commit）。例如，如果一个 bug 修复时需要对独立存储两个不同代码库的代码进行更改，那么必须创建两个提交：每个存储库中各一个。这种不连续性很容易降低项目和源代码管理工具的可跟踪性和加大复杂性。特别是，不可能跨越相互依赖的变更集的存储库执行原子提交这种多次跨仓库的变化是常见现象。 为了解决这个问题，JDK 10 中将所有现有存储库合并到一个 Mercurial 存储库中。这种合并的一次生效应，单一的 Mercurial 存储库比现有的八个存储库要更容易地被镜像(作为一个 Git 存储库)，并且使得跨越相互依赖的变更集的存储库运行原子提交成为可能，从而简化开发和管理过程。虽然在整合过程中，外部开发人员有一些阻力，但是 JDK 开发团队已经使这一更改成为 JDK 10 的一部分。 ### 统一的垃圾回收接口 在当前的 Java 结构中，组成垃圾回收器（GC）实现的组件分散在代码库的各个部分。尽管这些惯例对于使用 GC 计划的 JDK 开发者来说比较熟悉，但对新的开发人员来说，对于在哪里查找特定 GC 的源代码，或者实现一个新的垃圾收集器常常会感到困惑。更重要的是，随着 Java modules 的出现，我们希望在构建过程中排除不需要的 GC，但是当前 GC 接口的横向结构会给排除、定位问题带来困难。 为解决此问题，需要整合并清理 GC 接口，以便更容易地实现新的 GC，并更好地维护现有的 GC。Java 10 中，hotspot/gc 代码实现方面，引入一个干净的 GC 接口，改进不同 GC 源代码的隔离性，多个 GC 之间共享的实现细节代码应该存在于辅助类中。这种方式提供了足够的灵活性来实现全新 GC 接口，同时允许以混合搭配方式重复使用现有代码，并且能够保持代码更加干净、整洁，便于排查收集器问题。 ### 并行垃圾回收器 G1 大家如果接触过 Java 性能调优工作，应该会知道，调优的最终目标是通过参数设置来达到快速、低延时的内存垃圾回收以提高应用吞吐量，尽可能的避免因内存回收不及时而触发的完整 GC（Full GC 会带来应用出现卡顿）。 G1 垃圾回收器是 Java 9 中 Hotspot 的默认垃圾回收器，是以一种低延时的垃圾回收器来设计的，旨在避免进行 Full GC，但是当并发收集无法快速回收内存时，会触发垃圾回收器回退进行 Full GC。之前 Java 版本中的 G1 垃圾回收器执行 GC 时采用的是基于单线程标记扫描压缩算法（mark-sweep-compact）。为了最大限度地减少 Full GC 造成的应用停顿的影响，Java 10 中将为 G1 引入多线程并行 GC，同时会使用与年轻代回收和混合回收相同的并行工作线程数量，从而减少了 Full GC 的发生，以带来更好的性能提升、更大的吞吐量。 Java 10 中将采用并行化 mark-sweep-compact 算法，并使用与年轻代回收和混合回收相同数量的线程。具体并行 GC 线程数量可以通过：-XX：ParallelGCThreads 参数来调节，但这也会影响用于年轻代和混合收集的工作线程数。 ### 线程局部管控 在已有的 Java 版本中，JVM 线程只能全部启用或者停止，没法做到对单独某个线程的操作。为了能够对单独的某个线程进行操作，Java 10 中线程管控引入 JVM 安全点的概念，将允许在不运行全局 JVM 安全点的情况下实现线程回调，由线程本身或者 JVM 线程来执行，同时保持线程处于阻塞状态，这种方式使得停止单个线程变成可能，而不是只能启用或停止所有线程。通过这种方式显著地提高了现有 JVM 功能的性能开销，并且改变了到达 JVM 全局安全点的现有时间语义。 增加的参数为：-XX:ThreadLocalHandshakes (默认为开启)，将允许用户在支持的平台上选择安全点。 [参考]("https://segmentfault.com/a/1190000018588495") ================================================ FILE: docs/java/java-simple.md ================================================ # Java 代码精简 ## 利用语法 ### 利用三元表达式 普通 ``` java String title; if (isMember(phone)) { title = "会员"; } else { title = "游客"; } ``` 精简 ``` java String title = isMember(phone) ? "会员" : "游客"; ``` >注意：对于包装类型的算术计算，需要注意避免拆包时的空指针问题。 ### 利用 for-each 语句 >从 Java 5 起，提供了 for-each 循环，简化了数组和集合的循环遍历。for-each 循环允许你无需保持传统 for 循环中的索引就可以遍历数组，或在使用迭代器时无需在 while 循环中调用 hasNext 方法和 next 方法就可以遍历集合。 普通 ``` java double[] values = ...; for(int i = 0; i < values.length; i++) { double value = values[i]; // TODO: 处理value } List valueList = ...; Iterator iterator = valueList.iterator(); while (iterator.hasNext()) { Double value = iterator.next(); // TODO: 处理value } ``` 精简 ``` java double[] values = ...; for(double value : values) { // TODO: 处理value } List valueList = ...; for(Double value : valueList) { // TODO: 处理value } ``` ### 利用 try-with-resource 语句 >所有实现 Closeable 接口的“资源”，均可采用 try-with-resource 进行简化。 普通 ``` java BufferedReader reader = null; try { reader = new BufferedReader(new FileReader("cities.csv")); String line; while ((line = reader.readLine()) != null) { // TODO: 处理line } } catch (IOException e) { log.error("读取文件异常", e); } finally { if (reader != null) { try { reader.close(); } catch (IOException e) { log.error("关闭文件异常", e); } } } ``` 精简 ``` java try (BufferedReader reader = new BufferedReader(new FileReader("test.txt"))) { String line; while ((line = reader.readLine()) != null) { // TODO: 处理line } } catch (IOException e) { log.error("读取文件异常", e); } ``` ### 利用 return 关键字 >利用 return 关键字，可以提前函数返回，避免定义中间变量。 普通 ``` java public static boolean hasSuper(@NonNull List userList) { boolean hasSuper = false; for (UserDO user : userList) { if (Boolean.TRUE.equals(user.getIsSuper())) { hasSuper = true; break; } } return hasSuper; } ``` 精简 ``` java public static boolean hasSuper(@NonNull List userList) { for (UserDO user : userList) { if (Boolean.TRUE.equals(user.getIsSuper())) { return true; } } return false; } ``` ### 利用 static 关键字 >利用 static 关键字，可以把字段变成静态字段，也可以把函数变为静态函数，调用时就无需初始化类对象。 普通 ``` java public final class GisHelper { public double distance(double lng1, double lat1, double lng2, double lat2) { // 方法实现代码 } } GisHelper gisHelper = new GisHelper(); double distance = gisHelper.distance(116.178692D, 39.967115D, 116.410778D, 39.899721D); ``` 精简 ``` java public final class GisHelper { public static double distance(double lng1, double lat1, double lng2, double lat2) { // 方法实现代码 } } double distance = GisHelper.distance(116.178692D, 39.967115D, 116.410778D, 39.899721D); ``` ### 利用 lambda 表达式 >Java 8 发布以后，lambda 表达式大量替代匿名内部类的使用，在简化了代码的同时，更突出了原有匿名内部类中真正有用的那部分代码。 普通 ``` java new Thread(new Runnable() { public void run() { // 线程处理代码 } }).start(); ``` 精简 ``` java new Thread(() -> { // 线程处理代码 }).start(); ``` ### 利用方法引用 >方法引用（::），可以简化 lambda 表达式，省略变量声明和函数调用。 普通 ``` java Arrays.sort(nameArray, (a, b) -> a.compareToIgnoreCase(b)); List userIdList = userList.stream() .map(user -> user.getId()) .collect(Collectors.toList()); ``` 精简 ``` java Arrays.sort(nameArray, String::compareToIgnoreCase); List userIdList = userList.stream() .map(UserDO::getId) .collect(Collectors.toList()); ``` ### 利用静态导入 > 静态导入（import static），当程序中大量使用同一静态常量和函数时，可以简化静态常量和函数的引用。 普通 ``` java List areaList = radiusList.stream().map(r -> Math.PI * Math.pow(r, 2)).collect(Collectors.toList()); ``` 精简 ``` java import static java.lang.Math.PI; import static java.lang.Math.pow; import static java.util.stream.Collectors.toList; List areaList = radiusList.stream().map(r -> PI * pow(r, 2)).collect(toList()); ``` * 注意：静态引入容易造成代码阅读困难，所以在实际项目中应该警慎使用。 ### 利用 unchecked 异常 >Java 的异常分为两类：Checked 异常和 Unchecked 异常。Unchecked 异常继承了RuntimeException ，特点是代码不需要处理它们也能通过编译，所以它们称作 Unchecked 异常。利用 Unchecked 异常，可以避免不必要的 try-catch 和 throws 异常处理。 普通 ``` java @Service public class UserService { public void createUser(UserCreateVO create, OpUserVO user) throws BusinessException { checkOperatorUser(user); ... } private void checkOperatorUser(OpUserVO user) throws BusinessException { if (!hasPermission(user)) { throw new BusinessException("用户无操作权限"); } ... } ... } @RestController @RequestMapping("/user") public class UserController { @Autowired private UserService userService; @PostMapping("/createUser") public Result createUser(@RequestBody @Valid UserCreateVO create, OpUserVO user) throws BusinessException { userService.createUser(create, user); return Result.success(); } ... } ``` 精简 ``` java @Service public class UserService { public void createUser(UserCreateVO create, OpUserVO user) { checkOperatorUser(user); ... } private void checkOperatorUser(OpUserVO user) { if (!hasPermission(user)) { throw new BusinessRuntimeException("用户无操作权限"); } ... } ... } @RestController @RequestMapping("/user") public class UserController { @Autowired private UserService userService; @PostMapping("/createUser") public Result createUser(@RequestBody @Valid UserCreateVO create, OpUserVO user) { userService.createUser(create, user); return Result.success(); } ... } ``` ## 利用注解 ### 利用 Lombok 注解 >Lombok 提供了一组有用的注解，可以用来消除Java类中的大量样板代码。 普通 ``` java public class UserVO { private Long id; private String name; public Long getId() { return this.id; } public void setId(Long id) { this.id = id; } public String getName() { return this.name; } public void setName(String name) { this.name = name; } ... } ``` 精简 ``` java @Getter @Setter @ToString public class UserVO { private Long id; private String name; ... } ``` ### 利用 Validation 注解 普通 ``` java @Data public class UserCreateVO { private String name; private Long companyId; } @Service public class UserService { public Long createUser(UserCreateVO create) { // TODO: 创建用户 if(create.getName()==null){ ... } if(create.getCompanyId()==null){ ... } return null; } } ``` 精简 ``` java @Getter @Setter @ToString public class UserCreateVO { @NotBlank(message = "用户名称不能为空") private String name; @NotNull(message = "公司标识不能为空") private Long companyId; ... } @Service @Validated public class UserService { public Long createUser(@Valid UserCreateVO create) { // TODO: 创建用户 return null; } } ``` ### 利用 @NonNull 注解 >Spring 的 @NonNull 注解，用于标注参数或返回值非空，适用于项目内部团队协作。只要实现方和调用方遵循规范，可以避免不必要的空值判断，这充分体现了阿里的“新六脉神剑”提倡的“因为信任，所以简单”。 普通 ``` java public List queryCompanyUser(Long companyId) { // 检查公司标识 if (companyId == null) { return null; } // 查询返回用户 List userList = userDAO.queryByCompanyId(companyId); return userList.stream().map(this::transUser).collect(Collectors.toList()); } Long companyId = 1L; List userList = queryCompanyUser(companyId); if (CollectionUtils.isNotEmpty(userList)) { for (UserVO user : userList) { // TODO: 处理公司用户 } } ``` 精简 ``` java public @NonNull List queryCompanyUser(@NonNull Long companyId) { List userList = userDAO.queryByCompanyId(companyId); return userList.stream().map(this::transUser).collect(Collectors.toList()); } Long companyId = 1L; List userList = queryCompanyUser(companyId); for (UserVO user : userList) { // TODO: 处理公司用户 } ``` ### 利用注解特性 注解有以下特性可用于精简注解声明： 1、当注解属性值跟默认值一致时，可以删除该属性赋值； 2、当注解只有value属性时，可以去掉value进行简写； 3、当注解属性组合等于另一个特定注解时，直接采用该特定注解。 普通 ``` java @Lazy(true); @Service(value = "userService") @RequestMapping(path = "/getUser", method = RequestMethod.GET) ``` 精简 ``` java @Lazy @Service("userService") @GetMapping("/getUser") ``` ## 利用泛型 ### 泛型接口 >在 Java 没有引入泛型前，都是采用 Object 表示通用对象，最大的问题就是类型无法强校验并且需要强制类型转换。 普通 ``` java public interface Comparable { public int compareTo(Object other); } @Getter @Setter @ToString public class UserVO implements Comparable { private Long id; @Override public int compareTo(Object other) { UserVO user = (UserVO)other; return Long.compare(this.id, user.id); } } ``` 精简 ``` java public interface Comparable { public int compareTo(T other); } @Getter @Setter @ToString public class UserVO implements Comparable { private Long id; @Override public int compareTo(UserVO other) { return Long.compare(this.id, other.id); } } ``` ### 泛型类 普通 ``` java @Getter @Setter @ToString public class IntPoint { private Integer x; private Integer y; } @Getter @Setter @ToString public class DoublePoint { private Double x; private Double y; } ``` 精简 ``` java @Getter @Setter @ToString public class Point { private T x; private T y; } ``` ### 泛型方法 普通 ``` java public static Map newHashMap(String[] keys, Integer[] values) { // 检查参数非空 if (ArrayUtils.isEmpty(keys) || ArrayUtils.isEmpty(values)) { return Collections.emptyMap(); } // 转化哈希映射 Map map = new HashMap<>(); int length = Math.min(keys.length, values.length); for (int i = 0; i < length; i++) { map.put(keys[i], values[i]); } return map; } ``` 精简 ``` java public static Map newHashMap(K[] keys, V[] values) { // 检查参数非空 if (ArrayUtils.isEmpty(keys) || ArrayUtils.isEmpty(values)) { return Collections.emptyMap(); } // 转化哈希映射 Map map = new HashMap<>(); int length = Math.min(keys.length, values.length); for (int i = 0; i < length; i++) { map.put(keys[i], values[i]); } return map; } ``` ## 利用自身方法 ### 利用构造方法 >构造方法，可以简化对象的初始化和设置属性操作。对于属性字段较少的类，可以自定义构造方法。 普通 ``` java @Getter @Setter @ToString public class PageDataVO { private Long totalCount; private List dataList; } PageDataVO pageData = new PageDataVO<>(); pageData.setTotalCount(totalCount); pageData.setDataList(userList); return pageData; ``` 精简 ``` java @Getter @Setter @ToString @NoArgsConstructor @AllArgsConstructor public class PageDataVO { private Long totalCount; private List dataList; } return new PageDataVO<>(totalCount, userList); ``` * 注意：如果属性字段被替换时，存在构造函数初始化赋值问题。比如把属性字段title替换为 nickname ，由于构造函数的参数个数和类型不变，原有构造函数初始化语句不会报错，导致把原title值赋值给 nickname 。如果采用 Setter 方法赋值，编译器会提示错误并要求修复。 ### 利用 Set 的 add 方法 >利用 Set 的 add 方法的返回值，可以直接知道该值是否已经存在，可以避免调用 contains 方法判断存在。 普通 以下案例是进行用户去重转化操作，需要先调用 contains 方法判断存在，后调用add方法进行添加。 ``` java Set userIdSet = new HashSet<>(); List userVOList = new ArrayList<>(); for (UserDO userDO : userDOList) { if (!userIdSet.contains(userDO.getId())) { userIdSet.add(userDO.getId()); userVOList.add(transUser(userDO)); } } ``` 精简 ``` java SSet userIdSet = new HashSet<>(); List userVOList = new ArrayList<>(); for (UserDO userDO : userDOList) { if (userIdSet.add(userDO.getId())) { userVOList.add(transUser(userDO)); } } ``` ### 利用 Map 的 computeIfAbsent 方法 >利用 Map 的 computeIfAbsent 方法，可以保证获取到的对象非空，从而避免了不必要的空判断和重新设置值。 普通 ``` java Map> roleUserMap = new HashMap<>(); for (UserDO userDO : userDOList) { Long roleId = userDO.getRoleId(); List userList = roleUserMap.get(roleId); if (Objects.isNull(userList)) { userList = new ArrayList<>(); roleUserMap.put(roleId, userList); } userList.add(userDO); } ``` 精简 ``` java Map> roleUserMap = new HashMap<>(); for (UserDO userDO : userDOList) { roleUserMap.computeIfAbsent(userDO.getRoleId(), key -> new ArrayList<>()) .add(userDO); } ``` ### 利用链式编程 >链式编程，也叫级联式编程，调用对象的函数时返回一个this对象指向对象本身，达到链式效果，可以级联调用。链式编程的优点是：编程性强、可读性强、代码简洁。 普通 ``` java StringBuilder builder = new StringBuilder(96); builder.append("select id, name from "); builder.append(T_USER); builder.append(" where id = "); builder.append(userId); builder.append(";"); ``` 精简 ``` java StringBuilder builder = new StringBuilder(96); builder.append("select id, name from ") .append(T_USER) .append(" where id = ") .append(userId) .append(";"); ``` ## 利用工具方法 ### 避免空值判断 普通 ``` java if (userList != null && !userList.isEmpty()) { // TODO: 处理代码 } ``` 精简 ``` java if (CollectionUtils.isNotEmpty(userList)) { // TODO: 处理代码 } ``` ### 避免条件判断 普通 ``` java double result; if (value <= MIN_LIMIT) { result = MIN_LIMIT; } else { result = value; } ``` 精简 ``` java double result = Math.max(MIN_LIMIT, value); ``` ### 简化赋值语句 普通 ``` java public static final List ANIMAL_LIST; static { List animalList = new ArrayList<>(); animalList.add("dog"); animalList.add("cat"); animalList.add("tiger"); ANIMAL_LIST = Collections.unmodifiableList(animalList); } ``` 精简 ``` java // JDK流派 public static final List ANIMAL_LIST = Arrays.asList("dog", "cat", "tiger"); // Guava流派 public static final List ANIMAL_LIST = ImmutableList.of("dog", "cat", "tiger"); ``` * 注意：Arrays.asList 返回的 List 并不是 ArrayList ，不支持 add 等变更操作。 ### 简化数据拷贝 普通 ``` java UserVO userVO = new UserVO(); userVO.setId(userDO.getId()); userVO.setName(userDO.getName()); ... userVO.setDescription(userDO.getDescription()); userVOList.add(userVO); ``` 精简 ``` java UserVO userVO = new UserVO(); BeanUtils.copyProperties(userDO, userVO); userVOList.add(userVO); ``` 反例 ``` java List userVOList = JSON.parseArray(JSON.toJSONString(userDOList), UserVO.class); ``` * 精简代码，但不能以过大的性能损失为代价。例子是浅层拷贝，用不着 JSON 这样重量级的武器。 ### 简化异常断言 普通 ``` java if (Objects.isNull(userId)) { throw new IllegalArgumentException("用户标识不能为空"); } ``` 精简 ``` java Assert.notNull(userId, "用户标识不能为空"); ``` * 注意：可能有些插件不认同这种判断，导致使用该对象时会有空指针警告。 ### 简化测试用例 >把测试用例数据以 JSON 格式存入文件中，通过 JSON 的 parseObject 和 parseArray 方法解析成对象。虽然执行效率上有所下降，但可以减少大量的赋值语句，从而精简了测试代码。 普通 ``` java @Test public void testCreateUser() { UserCreateVO userCreate = new UserCreateVO(); userCreate.setName("Changyi"); userCreate.setTitle("Developer"); userCreate.setCompany("AMAP"); ... Long userId = userService.createUser(OPERATOR, userCreate); Assert.assertNotNull(userId, "创建用户失败"); } ``` 精简 ``` java @Test public void testCreateUser() { String jsonText = ResourceHelper.getResourceAsString(getClass(), "createUser.json"); UserCreateVO userCreate = JSON.parseObject(jsonText, UserCreateVO.class); Long userId = userService.createUser(OPERATOR, userCreate); Assert.assertNotNull(userId, "创建用户失败"); } ``` * 建议：JSON 文件名最好以被测试的方法命名，如果有多个版本可以用数字后缀表示。 ### 简化算法实现 >一些常规算法，已有现成的工具方法，我们就没有必要自己实现了。 普通 ``` java int totalSize = valueList.size(); List> partitionList = new ArrayList<>(); for (int i = 0; i < totalSize; i += PARTITION_SIZE) { partitionList.add(valueList.subList(i, Math.min(i + PARTITION_SIZE, totalSize))); } ``` 精简 ``` java List> partitionList = ListUtils.partition(valueList, PARTITION_SIZE); ``` ### 封装工具方法 >一些特殊算法，没有现成的工具方法，我们就只好自己亲自实现了。 普通 比如，SQL 设置参数值的方法就比较难用，setLong 方法不能设置参数值为 null 。 ``` java // 设置参数值 if (Objects.nonNull(user.getId())) { statement.setLong(1, user.getId()); } else { statement.setNull(1, Types.BIGINT); } ``` 精简 我们可以封装为一个工具类 SqlHelper ，简化设置参数值的代码。 ``` java /** SQL辅助类 */ public final class SqlHelper { /** 设置长整数值 */ public static void setLong(PreparedStatement statement, int index, Long value) throws SQLException { if (Objects.nonNull(value)) { statement.setLong(index, value.longValue()); } else { statement.setNull(index, Types.BIGINT); } } ... } // 设置参数值 SqlHelper.setLong(statement, 1, user.getId()); ``` ## 利用数据结构 ### 利用数组简化 >对于固定上下限范围的 if-else 语句，可以用数组+循环来简化。 普通 ``` java public static int getGrade(double score) { if (score >= 90.0D) { return 1; } if (score >= 80.0D) { return 2; } if (score >= 60.0D) { return 3; } if (score >= 30.0D) { return 4; } return 5; } ``` 精简 ``` java private static final double[] SCORE_RANGES = new double[] {90.0D, 80.0D, 60.0D, 30.0D}; public static int getGrade(double score) { for (int i = 0; i < SCORE_RANGES.length; i++) { if (score >= SCORE_RANGES[i]) { return i + 1; } } return SCORE_RANGES.length + 1; } ``` 思考：上面的案例返回值是递增的，所以用数组简化是没有问题的。但是，如果返回值不是递增的，能否用数组进行简化呢？答案是可以的，请自行思考解决。 ### 利用 Map 简化 >对于映射关系的 if-else 语句，可以用Map来简化。此外，此规则同样适用于简化映射关系的 switch 语句。 普通 ``` java public static String getBiologyClass(String name) { switch (name) { case "dog" : return "animal"; case "cat" : return "animal"; case "lavender" : return "plant"; ... default : return null; } } ``` 精简 ``` java private static final Map BIOLOGY_CLASS_MAP = ImmutableMap.builder() .put("dog", "animal") .put("cat", "animal") .put("lavender", "plant") ... .build(); public static String getBiologyClass(String name) { return BIOLOGY_CLASS_MAP.get(name); } ``` 已经把方法简化为一行代码，其实都没有封装方法的必要了。 ### 利用容器类简化 >Java 不像 Python 和 Go ，方法不支持返回多个对象。如果需要返回多个对象，就必须自定义类，或者利用容器类。常见的容器类有 Apache 的 Pair 类和 Triple 类， Pair 类支持返回 2 个对象， Triple 类支持返回 3 个对象。 普通 ``` java @Setter @Getter @ToString @AllArgsConstructor public static class PointAndDistance { private Point point; private Double distance; } public static PointAndDistance getNearest(Point point, Point[] points) { // 计算最近点和距离 ... // 返回最近点和距离 return new PointAndDistance(nearestPoint, nearestDistance); } ``` 精简 ``` java public static Pair getNearest(Point point, Point[] points) { // 计算最近点和距离 ... // 返回最近点和距离 return ImmutablePair.of(nearestPoint, nearestDistance); } ``` ### 利用 ThreadLocal 简化 > ThreadLocal 提供了线程专有对象，可以在整个线程生命周期中随时取用，极大地方便了一些逻辑的实现。用 ThreadLocal 保存线程上下文对象，可以避免不必要的参数传递。 普通 由于 DateFormat 的 format 方法线程非安全（建议使用替代方法），在线程中频繁初始化 DateFormat 性能太低，如果考虑重用只能用参数传入 DateFormat 。例子如下： ``` java public static String formatDate(Date date, DateFormat format) { return format.format(date); } public static List getDateList(Date minDate, Date maxDate, DateFormat format) { List dateList = new ArrayList<>(); Calendar calendar = Calendar.getInstance(); calendar.setTime(minDate); String currDate = formatDate(calendar.getTime(), format); String maxsDate = formatDate(maxDate, format); while (currDate.compareTo(maxsDate) <= 0) { dateList.add(currDate); calendar.add(Calendar.DATE, 1); currDate = formatDate(calendar.getTime(), format); } return dateList; } ``` 精简 可能你会觉得以下的代码量反而多了，如果调用工具方法的地方比较多，就可以省下一大堆 DateFormat 初始化和传入参数的代码。 ``` java private static final ThreadLocal LOCAL_DATE_FORMAT = new ThreadLocal() { @Override protected DateFormat initialValue() { return new SimpleDateFormat("yyyyMMdd"); } }; public static String formatDate(Date date) { return LOCAL_DATE_FORMAT.get().format(date); } public static List getDateList(Date minDate, Date maxDate) { List dateList = new ArrayList<>(); Calendar calendar = Calendar.getInstance(); calendar.setTime(minDate); String currDate = formatDate(calendar.getTime()); String maxsDate = formatDate(maxDate); while (currDate.compareTo(maxsDate) <= 0) { dateList.add(currDate); calendar.add(Calendar.DATE, 1); currDate = formatDate(calendar.getTime()); } return dateList; } ``` * 注意：ThreadLocal 有一定的内存泄露的风险，尽量在业务代码结束前调用 remove 方法进行数据清除。 ## 利用 Optional >在 Java 8 里，引入了一个 Optional 类，该类是一个可以为 null 的容器对象。 ### 保证值存在 普通 ``` java Integer thisValue; if (Objects.nonNull(value)) { thisValue = value; } else { thisValue = DEFAULT_VALUE; } ``` 精简 ``` java Integer thisValue = Optional.ofNullable(value).orElse(DEFAULT_VALUE); ``` ### 保证值合法 普通 ``` java Integer thisValue; if (Objects.nonNull(value) && value.compareTo(MAX_VALUE) <= 0) { thisValue = value; } else { thisValue = MAX_VALUE; } ``` 精简 ``` java Integer thisValue = Optional.ofNullable(value) .filter(tempValue -> tempValue.compareTo(MAX_VALUE) <= 0).orElse(MAX_VALUE); ``` ### 避免空判断 普通 ``` java String zipcode = null; if (Objects.nonNull(user)) { Address address = user.getAddress(); if (Objects.nonNull(address)) { Country country = address.getCountry(); if (Objects.nonNull(country)) { zipcode = country.getZipcode(); } } } ``` 精简 ``` java String zipcode = Optional.ofNullable(user).map(User::getAddress) .map(Address::getCountry).map(Country::getZipcode).orElse(null); ``` ## 利用 Stream > 流（Stream）是Java 8的新成员，允许你以声明式处理数据集合，可以看成为一个遍历数据集的高级迭代器。流主要有三部分构成：获取一个数据源→数据转换→执行操作获取想要的结果。每次转换原有 Stream 对象不改变，返回一个新的 Stream 对象，这就允许对其操作可以像链条一样排列，形成了一个管道。流（Stream）提供的功能非常有用，主要包括匹配、过滤、汇总、转化、分组、分组汇总等功能。 ### 匹配集合数据 普通 ``` java boolean isFound = false; for (UserDO user : userList) { if (Objects.equals(user.getId(), userId)) { isFound = true; break; } } ``` 精简 ``` java boolean isFound = userList.stream() .anyMatch(user -> Objects.equals(user.getId(), userId)); ``` ### 过滤集合数据 普通 ``` java List resultList = new ArrayList<>(); for (UserDO user : userList) { if (Boolean.TRUE.equals(user.getIsSuper())) { resultList.add(user); } } ``` 精简 ``` java List resultList = userList.stream() .filter(user -> Boolean.TRUE.equals(user.getIsSuper())) .collect(Collectors.toList()); ``` ### 汇总集合数据 普通 ``` java double total = 0.0D; for (Account account : accountList) { total += account.getBalance(); } ``` 精简 ``` java double total = accountList.stream().mapToDouble(Account::getBalance).sum(); ``` ### 转化集合数据 普通 ``` java List userVOList = new ArrayList<>(); for (UserDO userDO : userDOList) { userVOList.add(transUser(userDO)); } ``` 精简 ``` java List userVOList = userDOList.stream() .map(this::transUser).collect(Collectors.toList()); ``` ### 分组集合数据 普通 ``` java Map> roleUserMap = new HashMap<>(); for (UserDO userDO : userDOList) { roleUserMap.computeIfAbsent(userDO.getRoleId(), key -> new ArrayList<>()) .add(userDO); } ``` 精简 ``` java Map> roleUserMap = userDOList.stream() .collect(Collectors.groupingBy(UserDO::getRoleId)); ``` ### 分组汇总集合 普通 ``` java Map roleTotalMap = new HashMap<>(); for (Account account : accountList) { Long roleId = account.getRoleId(); Double total = Optional.ofNullable(roleTotalMap.get(roleId)).orElse(0.0D); roleTotalMap.put(roleId, total + account.getBalance()); } ``` 精简 ``` java roleTotalMap = accountList.stream().collect(Collectors.groupingBy(Account::getRoleId, Collectors.summingDouble(Account::getBalance))); ``` ### 生成范围集合 > Python 的 range 非常方便，Stream 也提供了类似的方法。 普通 ``` java int[] array1 = new int[N]; for (int i = 0; i < N; i++) { array1[i] = i + 1; } int[] array2 = new int[N]; array2[0] = 1; for (int i = 1; i < N; i++) { array2[i] = array2[i - 1] * 2; } ``` 精简 ``` java int[] array1 = IntStream.rangeClosed(1, N).toArray(); int[] array2 = IntStream.iterate(1, n -> n * 2).limit(N).toArray(); ``` ## 利用程序结构 ### 返回条件表达式 >条件表达式判断返回布尔值，条件表达式本身就是结果。 普通 ``` java public boolean isSuper(Long userId) UserDO user = userDAO.get(userId); if (Objects.nonNull(user) && Boolean.TRUE.equals(user.getIsSuper())) { return true; } return false; } ``` 精简 ``` java public boolean isSuper(Long userId) UserDO user = userDAO.get(userId); return Objects.nonNull(user) && Boolean.TRUE.equals(user.getIsSuper()); } ``` ### 最小化条件作用域 >最小化条件作用域，尽量提出公共处理代码。 普通 ``` java Result result = summaryService.reportWorkDaily(workDaily); if (result.isSuccess()) { String message = "上报工作日报成功"; dingtalkService.sendMessage(user.getPhone(), message); } else { String message = "上报工作日报失败:" + result.getMessage(); log.warn(message); dingtalkService.sendMessage(user.getPhone(), message); } ``` 精简 ``` java String message; Result result = summaryService.reportWorkDaily(workDaily); if (result.isSuccess()) { message = "上报工作日报成功"; } else { message = "上报工作日报失败:" + result.getMessage(); log.warn(message); } dingtalkService.sendMessage(user.getPhone(), message); ``` ### 调整表达式位置 >调整表达式位置，在逻辑不变的前提下，让代码变得更简洁。 普通1 ``` java String line = readLine(); while (Objects.nonNull(line)) { ... // 处理逻辑代码 line = readLine(); } ``` 普通2 ``` java for (String line = readLine(); Objects.nonNull(line); line = readLine()) { ... // 处理逻辑代码 } ``` 精简 ``` java String line; while (Objects.nonNull(line = readLine())) { ... // 处理逻辑代码 } ``` * 注意：有些规范可能不建议这种精简写法。 ### 利用非空对象 >在比较对象时，交换对象位置，利用非空对象，可以避免空指针判断。 普通 ``` java private static final int MAX_VALUE = 1000; boolean isMax = (value != null && value.equals(MAX_VALUE)); boolean isTrue = (result != null && result.equals(Boolean.TRUE)); ``` 精简 ``` java private static final Integer MAX_VALUE = 1000; boolean isMax = MAX_VALUE.equals(value); boolean isTrue = Boolean.TRUE.equals(result); ``` ## 利用设计模式 ### 模板方法模式 >模板方法模式（Template Method Pattern）定义一个固定的算法框架，而将算法的一些步骤放到子类中实现，使得子类可以在不改变算法框架的情况下重定义该算法的某些步骤。 普通 ``` java @Repository public class UserValue { /** 值操作 */ @Resource(name = "stringRedisTemplate") private ValueOperations valueOperations; /** 值模式 */ private static final String KEY_FORMAT = "Value:User:%s"; /** 设置值 */ public void set(Long id, UserDO value) { String key = String.format(KEY_FORMAT, id); valueOperations.set(key, JSON.toJSONString(value)); } /** 获取值 */ public UserDO get(Long id) { String key = String.format(KEY_FORMAT, id); String value = valueOperations.get(key); return JSON.parseObject(value, UserDO.class); } ... } @Repository public class RoleValue { /** 值操作 */ @Resource(name = "stringRedisTemplate") private ValueOperations valueOperations; /** 值模式 */ private static final String KEY_FORMAT = "Value:Role:%s"; /** 设置值 */ public void set(Long id, RoleDO value) { String key = String.format(KEY_FORMAT, id); valueOperations.set(key, JSON.toJSONString(value)); } /** 获取值 */ public RoleDO get(Long id) { String key = String.format(KEY_FORMAT, id); String value = valueOperations.get(key); return JSON.parseObject(value, RoleDO.class); } ... } ``` 精简 ``` java public abstract class AbstractDynamicValue { /** 值操作 */ @Resource(name = "stringRedisTemplate") private ValueOperations valueOperations; /** 设置值 */ public void set(I id, V value) { valueOperations.set(getKey(id), JSON.toJSONString(value)); } /** 获取值 */ public V get(I id) { return JSON.parseObject(valueOperations.get(getKey(id)), getValueClass()); } ... /** 获取主键 */ protected abstract String getKey(I id); /** 获取值类 */ protected abstract Class getValueClass(); } @Repository public class UserValue extends AbstractValue { /** 获取主键 */ @Override protected String getKey(Long id) { return String.format("Value:User:%s", id); } /** 获取值类 */ @Override protected Class getValueClass() { return UserDO.class; } } @Repository public class RoleValue extends AbstractValue { /** 获取主键 */ @Override protected String getKey(Long id) { return String.format("Value:Role:%s", id); } /** 获取值类 */ @Override protected Class getValueClass() { return RoleDO.class; } } ``` ### 建造者模式 > 建造者模式（Builder Pattern）将一个复杂对象的构造与它的表示分离，使同样的构建过程可以创建不同的表示，这样的设计模式被称为建造者模式。 普通 ``` java public interface DataHandler { /** 解析数据 */ public T parseData(Record record); /** 存储数据 */ public boolean storeData(List dataList); } public long executeFetch(String tableName, int batchSize, DataHandler dataHandler) throws Exception { // 构建下载会话 DownloadSession session = buildSession(tableName); // 获取数据数量 long recordCount = session.getRecordCount(); if (recordCount == 0) { return 0; } // 进行数据读取 long fetchCount = 0L; try (RecordReader reader = session.openRecordReader(0L, recordCount, true)) { // 依次读取数据 Record record; List dataList = new ArrayList<>(batchSize); while ((record = reader.read()) != null) { // 解析添加数据 T data = dataHandler.parseData(record); if (Objects.nonNull(data)) { dataList.add(data); } // 批量存储数据 if (dataList.size() == batchSize) { boolean isContinue = dataHandler.storeData(dataList); fetchCount += batchSize; dataList.clear(); if (!isContinue) { break; } } } // 存储剩余数据 if (CollectionUtils.isNotEmpty(dataList)) { dataHandler.storeData(dataList); fetchCount += dataList.size(); dataList.clear(); } } // 返回获取数量 return fetchCount; } // 使用案例 long fetchCount = odpsService.executeFetch("user", 5000, new DataHandler() { /** 解析数据 */ @Override public T parseData(Record record) { UserDO user = new UserDO(); user.setId(record.getBigint("id")); user.setName(record.getString("name")); return user; } /** 存储数据 */ @Override public boolean storeData(List dataList) { userDAO.batchInsert(dataList); return true; } }); ``` 精简 ``` java public long executeFetch(String tableName, int batchSize, Function dataParser, Function, Boolean> dataStorage) throws Exception { // 构建下载会话 DownloadSession session = buildSession(tableName); // 获取数据数量 long recordCount = session.getRecordCount(); if (recordCount == 0) { return 0; } // 进行数据读取 long fetchCount = 0L; try (RecordReader reader = session.openRecordReader(0L, recordCount, true)) { // 依次读取数据 Record record; List dataList = new ArrayList<>(batchSize); while ((record = reader.read()) != null) { // 解析添加数据 T data = dataParser.apply(record); if (Objects.nonNull(data)) { dataList.add(data); } // 批量存储数据 if (dataList.size() == batchSize) { Boolean isContinue = dataStorage.apply(dataList); fetchCount += batchSize; dataList.clear(); if (!Boolean.TRUE.equals(isContinue)) { break; } } } // 存储剩余数据 if (CollectionUtils.isNotEmpty(dataList)) { dataStorage.apply(dataList); fetchCount += dataList.size(); dataList.clear(); } } // 返回获取数量 return fetchCount; } // 使用案例 long fetchCount = odpsService.executeFetch("user", 5000, record -> { UserDO user = new UserDO(); user.setId(record.getBigint("id")); user.setName(record.getString("name")); return user; }, dataList -> { userDAO.batchInsert(dataList); return true; }); ``` 普通的建造者模式，实现时需要定义 DataHandler 接口，调用时需要实现 DataHandler 匿名内部类，代码较多较繁琐。而精简后的建造者模式，充分利用了函数式编程，实现时无需定义接口，直接使用 Function 接口；调用时无需实现匿名内部类，直接采用 lambda 表达式，代码较少较简洁。 ### 代理模式 >Spring 中最重要的代理模式就是 AOP (Aspect-Oriented Programming，面向切面的编程)，是使用 JDK 动态代理和 CGLIB 动态代理技术来实现的。 普通 ``` java @Slf4j @RestController @RequestMapping("/user") public class UserController { /** 用户服务 */ @Autowired private UserService userService; /** 查询用户 */ @PostMapping("/queryUser") public Result queryUser(@RequestBody @Valid UserQueryVO query) { try { PageDataVO pageData = userService.queryUser(query); return Result.success(pageData); } catch (Exception e) { log.error(e.getMessage(), e); return Result.failure(e.getMessage()); } } ... } ``` 精简1 基于 @ControllerAdvice 的异常处理： ``` java @RestController @RequestMapping("/user") public class UserController { /** 用户服务 */ @Autowired private UserService userService; /** 查询用户 */ @PostMapping("/queryUser") public Result> queryUser(@RequestBody @Valid UserQueryVO query) { PageDataVO pageData = userService.queryUser(query); return Result.success(pageData); } ... } @Slf4j @ControllerAdvice public class GlobalControllerAdvice { /** 处理异常 */ @ResponseBody @ExceptionHandler(Exception.class) public Result handleException(Exception e) { log.error(e.getMessage(), e); return Result.failure(e.getMessage()); } } ``` 精简2 基于 AOP 的异常处理： ``` java // UserController代码同"精简1" @Slf4j @Aspect public class WebExceptionAspect { /** 点切面 */ @Pointcut("@annotation(org.springframework.web.bind.annotation.RequestMapping)") private void webPointcut() {} /** 处理异常 */ @AfterThrowing(pointcut = "webPointcut()", throwing = "e") public void handleException(Exception e) { Result result = Result.failure(e.getMessage()); writeContent(JSON.toJSONString(result)); } ... } ``` ## 利用删除代码 >“少即是多”，“少”不是空白而是精简，“多”不是拥挤而是完美。删除多余的代码，才能使代码更精简更完美。 ### 删除已废弃的代码 >删除项目中的已废弃的包、类、字段、方法、变量、常量、导入、注解、注释、已注释代码、Maven包导入、MyBatis的SQL语句、属性配置字段等，可以精简项目代码便于维护。 普通 ``` java import lombok.extern.slf4j.Slf4j; @Slf4j @Service public class ProductService { @Value("discardRate") private double discardRate; ... private ProductVO transProductDO(ProductDO productDO) { ProductVO productVO = new ProductVO(); BeanUtils.copyProperties(productDO, productVO); // productVO.setPrice(getDiscardPrice(productDO.getPrice())); return productVO; } private BigDecimal getDiscardPrice(BigDecimal originalPrice) { ... } } ``` 精简 ``` java @Service public class ProductService { ... private ProductVO transProductDO(ProductDO productDO) { ProductVO productVO = new ProductVO(); BeanUtils.copyProperties(productDO, productVO); return productVO; } } ``` ### 删除接口方法的public >对于接口(interface)，所有的字段和方法都是 public 的，可以不用显式声明为 public 。 普通 ``` java public interface UserDAO { public Long countUser(@Param("query") UserQuery query); public List queryUser(@Param("query") UserQuery query); } ``` 精简 ``` java public interface UserDAO { Long countUser(@Param("query") UserQuery query); List queryUser(@Param("query") UserQuery query); } ``` ### 删除枚举构造方法的 private >对于枚举(menu)，构造方法都是 private 的，可以不用显式声明为 private 。 普通 ``` java public enum UserStatus { DISABLED(0, "禁用"), ENABLED(1, "启用"); private final Integer value; private final String desc; private UserStatus(Integer value, String desc) { this.value = value; this.desc = desc; } ... } ``` 精简 ``` java public enum UserStatus { DISABLED(0, "禁用"), ENABLED(1, "启用"); private final Integer value; private final String desc; UserStatus(Integer value, String desc) { this.value = value; this.desc = desc; } ... } ``` ### 删除 final 类方法的 final > 对于 final 类，不能被子类继承，所以其方法不会被覆盖，没有必要添加 final 修饰。 普通 ``` java public final Rectangle implements Shape { ... @Override public final double getArea() { return width * height; } } ``` 精简 ``` java public final Rectangle implements Shape { ... @Override public double getArea() { return width * height; } } ``` ### 删除基类 implements 的接口 > 如果基类已 implements 某接口，子类没有必要再 implements 该接口，只需要直接实现接口方法即可。 普通 ``` java public interface Shape { ... double getArea(); } public abstract AbstractShape implements Shape { ... } public final Rectangle extends AbstractShape implements Shape { ... @Override public double getArea() { return width * height; } } ``` 精简 ``` java ... public final Rectangle extends AbstractShape { ... @Override public double getArea() { return width * height; } } ``` ### 删除不必要的变量 >不必要的变量，只会让代码看起来更繁琐。 普通 ``` java public Boolean existsUser(Long userId) { Boolean exists = userDAO.exists(userId); return exists; } ``` 精简 ``` java public Boolean existsUser(Long userId) { return userDAO.exists(userId); } ``` ================================================ FILE: docs/java/java-string.md ================================================ # String性能提升10倍的几个方法 String 类型是我们使用最频繁的数据类型，没有之一。那么提高 String 的运行效率，无疑是提升程序性能的最佳手段。 我们本文将从 String 的源码入手，一步步带你实现字符串优化的小目标。**不但教你如何有效的使用字符串，还为你揭晓这背后的深层次原因。** 本文涉及的知识点，如下图所示： ![java-stirng](../img/java-string-1.png) 在看如何优化 String 之前，我们先来了解一下 String 的特性，毕竟知己知彼，才能百战不殆。 ## 字符串的特性 想要了解 String 的特性就必须从它的源码入手，如下所示： ``` java // 源码基于 JDK 1.8 public final class String implements java.io.Serializable, Comparable, CharSequence { // String 值的实际存储容器 private final char value[]; public String() { this.value = "".value; } public String(String original) { this.value = original.value; this.hash = original.hash; } // 忽略其他信息 } ``` 从他的源码我们可以看出，String 类以及它的 value[] 属性都被 final 修饰了，其中 value[] 是实现字符串存储的最终结构，而 final 则表示“最后的、最终的”。 我们知道，被 final 修饰的类是不能被继承的，也就是说此类将不能拥有子类，而被 final 修饰的变量即为常量，**它的值是不能被改变的。这也就说当 String 一旦被创建之后，就不能被修改了。** ## String 为什么不能被修改？ String 的类和属性 value[] 都被定义为 final 了，这样做的好处有以下三点： * 安全性：当你在调用其他方法时，比如调用一些系统级操作指令之前，可能会有一系列校验，如果是可变类的话，可能在你校验过后，它的内部的值又被改变了，这样有可能会引起严重的系统崩溃问题，所以迫使 String 设计为 final 类的一个重要原因就是出于安全考虑； * 高性能：String 不可变之后就保证的 hash 值的唯一性，这样它就更加高效，并且更适合做 HashMap 的 key- value 缓存； * 节约内存：String 的不可变性是它实现字符串常量池的基础，字符串常量池指的是字符串在创建时，先去“常量池”查找是否有此“字符串”，如果有，则不会开辟新空间创作字符串，而是直接把常量池中的引用返回给此对象，这样就能更加节省空间。例如，通常情况下 String 创建有两种方式，直接赋值的方式，如 String str="Java"；另一种是 new 形式的创建，如 String str = new String("Java")。当代码中使用第一种方式创建字符串对象时，JVM 首先会检查该对象是否在字符串常量池中，如果在，就返回该对象引用，否则新的字符串将在常量池中被创建。这种方式可以减少同一个值的字符串对象的重复创建，节约内存。String str = new String("Java") 这种方式，首先在编译类文件时，“Java”常量字符串将会放入到常量结构中，在类加载时，“Java”将会在常量池中创建；其次，在调用 new 时，JVM 命令将会调用 String 的构造函数，同时引用常量池中的“Java”字符串，在堆内存中创建一个 String 对象，最后 str 将引用 String 对象。 ## 不要直接+=字符串 通过上面的内容，我们知道了 String 类是不可变的，那么在使用 String 时就不能频繁的 += 字符串了。 优化前代码： ``` java public static String doAdd() { String result = ""; for (int i = 0; i < 10000; i++) { result += (" i:" + i); } return result; } ``` 有人可能会问，我的业务需求是这样的，那我该如何实现？ 官方为我们提供了两种字符串拼加的方案：StringBuffer 和 StringBuilder，其中 StringBuilder 为非线程安全的，而 StringBuffer 则是线程安全的，StringBuffer 的拼加方法使用了关键字 synchronized 来保证线程的安全，源码如下： ``` java @Override public synchronized StringBuffer append(CharSequence s) { toStringCache = null; super.append(s); return this; } ``` 也因为使用 synchronized 修饰，所以 StringBuffer 的拼加性能会比 StringBuilder 低。 那我们就用 StringBuilder 来实现字符串的拼加，优化后代码： ``` java public static String doAppend() { StringBuilder sb = new StringBuilder(); for (int i = 0; i < 10000; i++) { sb.append(" i:" + i); } return sb.toString(); } ``` 我们通过代码测试一下，两个方法之间的性能差别： ``` java public class StringTest { public static void main(String[] args) { for (int i = 0; i < 5; i++) { // String long st1 = System.currentTimeMillis(); // 开始时间 doAdd(); long et1 = System.currentTimeMillis(); // 开始时间 System.out.println("String 拼加，执行时间：" + (et1 - st1)); // StringBuilder long st2 = System.currentTimeMillis(); // 开始时间 doAppend(); long et2 = System.currentTimeMillis(); // 开始时间 System.out.println("StringBuilder 拼加，执行时间：" + (et2 - st2)); System.out.println(); } } public static String doAdd() { String result = ""; for (int i = 0; i < 10000; i++) { result += ("Java中文社群:" + i); } return result; } public static String doAppend() { StringBuilder sb = new StringBuilder(); for (int i = 0; i < 10000; i++) { sb.append("Java中文社群:" + i); } return sb.toString(); } } ``` 以上程序的执行结果如下： ``` String 拼加，执行时间：429 StringBuilder 拼加，执行时间：1 String 拼加，执行时间：325 StringBuilder 拼加，执行时间：1 String 拼加，执行时间：287 StringBuilder 拼加，执行时间：1 String 拼加，执行时间：265 StringBuilder 拼加，执行时间：1 String 拼加，执行时间：249 StringBuilder 拼加，执行时间：1 ``` 从结果可以看出，优化前后的性能相差很大。 >注意：此性能测试的结果与循环的次数有关，也就是说循环的次数越多，他们性能相除的结果也越大。 接下来，我们要思考一个问题：**为什么 StringBuilder.append() 方法比 += 的性能高？而且拼接的次数越多性能的差距也越大？** 当我们打开 StringBuilder 的源码，就可以发现其中的“小秘密”了，StringBuilder 父类 AbstractStringBuilder 的实现源码如下： ``` java abstract class AbstractStringBuilder implements Appendable, CharSequence { char[] value; int count; @Override public AbstractStringBuilder append(CharSequence s, int start, int end) { if (s == null) s = "null"; if ((start < 0) || (start > end) || (end > s.length())) throw new IndexOutOfBoundsException( "start " + start + ", end " + end + ", s.length() " + s.length()); int len = end - start; ensureCapacityInternal(count + len); for (int i = start, j = count; i < end; i++, j++) value[j] = s.charAt(i); count += len; return this; } // 忽略其他信息... } ``` 而 StringBuilder 使用了父类提供的 char[] 作为自己值的实际存储单元，每次在拼加时会修改 char[] 数组，StringBuilder toString() 源码如下： ``` java @Override public String toString() { // Create a copy, don't share the array return new String(value, 0, count); } ``` 综合以上源码可以看出：**StringBuilder 使用了 char[] 作为实际存储单元，每次在拼加时只需要修改 char[] 数组即可，只是在 toString() 时创建了一个字符串；而 String 一旦创建之后就不能被修改，因此在每次拼加时，都需要重新创建新的字符串，所以 StringBuilder.append() 的性能就会比字符串的 += 性能高很多。** ## 善用 intern 方法 善用 String.intern() 方法可以有效的节约内存并提升字符串的运行效率，先来看 intern() 方法的定义与源码： ``` java /** * Returns a canonical representation for the string object. *

* A pool of strings, initially empty, is maintained privately by the * class {@code String}. *

* When the intern method is invoked, if the pool already contains a * string equal to this {@code String} object as determined by * the {@link #equals(Object)} method, then the string from the pool is * returned. Otherwise, this {@code String} object is added to the * pool and a reference to this {@code String} object is returned. *

* It follows that for any two strings {@code s} and {@code t}, * {@code s.intern() == t.intern()} is {@code true} * if and only if {@code s.equals(t)} is {@code true}. *

* All literal strings and string-valued constant expressions are * interned. String literals are defined in section 3.10.5 of the * The Java™ Language Specification. * * @return a string that has the same contents as this string, but is * guaranteed to be from a pool of unique strings. */ public native String intern(); ``` 可以看出 intern() 是一个高效的本地方法，它的定义中说的是，当调用 intern 方法时，如果字符串常量池中已经包含此字符串，则直接返回此字符串的引用，如果不包含此字符串，先将字符串添加到常量池中，再返回此对象的引用。 那什么情况下适合使用 intern() 方法？ Twitter 工程师曾分享过一个 String.intern() 的使用示例，Twitter 每次发布消息状态的时候，都会产生一个地址信息，以当时 Twitter 用户的规模预估，服务器需要 32G 的内存来存储地址信息。 ``` java public class Location { private String city; private String region; private String countryCode; private double longitude; private double latitude; } ``` 考虑到其中有很多用户在地址信息上是有重合的，比如，国家、省份、城市等，这时就可以将这部分信息单独列出一个类，以减少重复，代码如下： ``` java public class SharedLocation { private String city; private String region; private String countryCode; } public class Location { private SharedLocation sharedLocation; double longitude; double latitude; } ``` 通过优化，数据存储大小减到了 20G 左右。但对于内存存储这个数据来说，依然很大，怎么办呢？ Twitter 工程师使用 String.intern() 使重复性非常高的地址信息存储大小从 20G 降到几百兆，从而优化了 String 对象的存储。 实现的核心代码如下： ``` java SharedLocation sharedLocation = new SharedLocation(); sharedLocation.setCity(messageInfo.getCity().intern()); sharedLocation.setCountryCode(messageInfo.getRegion().intern()); sharedLocation.setRegion(messageInfo.getCountryCode().intern()); ``` 从 JDK1.7 版本以后，常量池已经合并到了堆中，所以不会复制字符串副本，只是会把首次遇到的字符串的引用添加到常量池中。此时只会判断常量池中是否已经有此字符串，如果有就返回常量池中的字符串引用。 这就相当于以下代码： ``` java String s1 = new String("Java中文社群").intern(); String s2 = new String("Java中文社群").intern(); System.out.println(s1 == s2); ``` 执行的结果为：true 此处如果有人问为什么不直接赋值（使用 String s1 = "Java中文社群"），是因为这段代码是简化了上面 Twitter 业务代码的语义而创建的，他使用的是对象的方式，而非直接赋值的方式。 ## 慎重使用 Split 方法 之所以要劝各位慎用 Split 方法，是因为 Split 方法大多数情况下使用的是正则表达式，这种分割方式本身没有什么问题，但是由于正则表达式的性能是非常不稳定的，使用不恰当会引起回溯问题，很可能导致 CPU 居高不下。 例如以下正则表达式： ``` java String badRegex = "^([hH][tT]{2}[pP]://|[hH][tT]{2}[pP][sS]://)(([A-Za-z0-9-~]+).)+([A-Za-z0-9-~\\\\/])+$"; String bugUrl = "http://www.apigo.com/dddp-web/pdf/download?request=6e7JGxxxxx4ILd-kExxxxxxxqJ4-CHLmqVnenXC692m74H38sdfdsazxcUmfcOH2fAfY1Vw__%5EDadIfJgiEf"; if (bugUrl.matches(badRegex)) { System.out.println("match!!"); } else { System.out.println("no match!!"); } ``` 执行效果如下图所示： ![java-string](../img/java-string-2.png) 可以看出，此代码导致了 CPU 使用过高。 **Java 正则表达式使用的引擎实现是 NFA（Non deterministic Finite Automaton，不确定型有穷自动机）自动机，这种正则表达式引擎在进行字符匹配时会发生回溯（backtracking），而一旦发生回溯，那其消耗的时间就会变得很长，有可能是几分钟，也有可能是几个小时，时间长短取决于回溯的次数和复杂度。** 为了更好地解释什么是回溯，我们使用以下面例子进行解释： ``` java text = "abbc"; regex = "ab{1,3}c"; ``` 上面的这个例子的目的比较简单，匹配以 a 开头，以 c 结尾，中间有 1-3 个 b 字符的字符串。 NFA 引擎对其解析的过程是这样子的： * 首先，读取正则表达式第一个匹配符 a 和 字符串第一个字符 a 比较，匹配上了，于是读取正则表达式第二个字符； * 读取正则表达式第二个匹配符 b{1,3} 和字符串的第二个字符 b 比较，匹配上了。但因为 b{1,3} 表示 1-3 个 b 字符串，以及 NFA 自动机的贪婪特性（也就是说要尽可能多地匹配），所以此时并不会再去读取下一个正则表达式的匹配符，而是依旧使用 b{1,3} 和字符串的第三个字符 b 比较，发现还是匹配上了，于是继续使用 b{1,3} 和字符串的第四个字符 c 比较，发现不匹配了，此时就会发生回溯； * 发生回溯后，我们已经读取的字符串第四个字符 c 将被吐出去，指针回到第三个字符串的位置，之后程序读取正则表达式的下一个操作符 c，然后再读取当前指针的下一个字符 c 进行对比，发现匹配上了，于是读取下一个操作符，然后发现已经结束了。 这就是正则匹配执行的流程和简单的回溯执行流程，而上面的示例在匹配到“com/dzfp-web/pdf/download?request=6e7JGm38jf.....”时因为贪婪匹配的原因，所以程序会一直读后面的字符串进行匹配，最后发现没有点号，于是就一个个字符回溯回去了，于是就会导致了 CPU 运行过高。 **所以我们应该慎重使用 Split() 方法，我们可以用 String.indexOf() 方法代替 Split() 方法完成字符串的分割。如果实在无法满足需求，你就在使用 Split() 方法时，对回溯问题加以重视就可以了。** ## 总结 本文通过 String 源码分析，发现了 String 的不可变特性，以及不可变特性的 3 大优点讲解；然后讲了字符串优化的三个手段：不要直接 += 字符串、善用 intern() 方法和慎重使用 Split() 方法。并且通过 StringBuilder 的源码分析，了解了 append() 性能高的主要原因，以及正则表达式不稳定性导致回溯问题，进入导致 CPU 使用过高的案例分析，希望可以切实的帮助到你。 ================================================ FILE: docs/java/java-utils.md ================================================ # java常用工具库使用 ![java-utils](../img/java-utils-1.png) ## 字符串相关工具类 Java 中 String 应该是日常用的最多一个类吧，平常我们很多代码需要围绕 String ，做一些处理。 JDK 提供 String API 虽然比较多，但是功能比较基础，通常我们需要结合 String 多个方法才能完成一个业务功能。 下面介绍一下 Apache 提供的一个工具类 StringUtils. Maven Pom 信息如下： ``` pom org.apache.commons commons-lang3 3.10 ``` > commons-lang 有两个版本，一个是 commons-lang3 ，一个是 commons-lang 。 commons-lang 是老版本，已经很久没有维护了。 commons-lang3 是一直在维护的版本，推荐直接使用这个版本。 注意：如果你系统已经有 commons-lang，注意如果直接替换成 commons-lang3，将会编译错误。commons-lang3 中相关类与 commons-lang 一样，但是包名不一样。 ### 判断字符串是否为空 判断字符串是否为空，想必每个人应该都写过吧： ``` java if (null == str || str.isEmpty()) { } ``` 虽然这段代码非常简单，但是说实话，以前还是在这里犯过空指针的异常的。 使用 StringUtils ，上面代码可以替换下面这样： ``` java if (StringUtils.isEmpty(str)) { } ``` StringUtils 内部还有一个方法 isBlank，也是用来判断字符串是否为空，两个方法比较相近，比较搞混，主要区别如下： ``` java // 如果字符串都是空格的话， StringUtils.isBlank(" ") = true; StringUtils.isEmpty(" ") = false； ``` ### 字符串固定长度 这个通常用于字符串需要固定长度的场景，比如需要固定长度字符串作为流水号，若流水号长度不足，，左边补 0 。 这里当然可以使用 String#format 方法，不过阿粉觉得比较麻烦，这里可以这样使用： ``` java // 字符串固定长度 8位，若不足，往左补 0 StringUtils.leftPad("test", 8, "0"); ``` 另外还有一个 StringUtils#rightPad,这个方法与上面方法正好相反。 ### 字符串关键字替换 StringUtils 提供一些列的方法，可以替换某些关键字： ``` java // 默认替换所有关键字 StringUtils.replace("aba", "a", "z") = "zbz"; // 替换关键字，仅替换一次 StringUtils.replaceOnce("aba", "a", "z") = "zba"; // 使用正则表达式替换 StringUtils.replacePattern("ABCabc123", "[^A-Z0-9]+", "") = "ABC123"； .... ``` ### 字符串拼接 字符串拼接是个常见的需求，简单办法使用 StringBuilder 循环遍历拼接： ``` java String[] array = new String[]{"test", "1234", "5678"}; StringBuilder stringBuilder = new StringBuilder(); for (String s : array) { stringBuilder.append(s).append(";"); } // 防止最终拼接字符串为空 if (stringBuilder.length() > 0) { stringBuilder.deleteCharAt(stringBuilder.length() - 1); } System.out.println(stringBuilder.toString()); ``` 上面业务代码不太难，但是需要注意一下上面这段代码非常容易出错，容易抛出 StringIndexOutOfBoundsException。 这里我们可以直接使用以下方法获取拼接之后字符串： ``` java StringUtils.join(["a", "b", "c"], ",") = "a,b,c" ``` StringUtils 只能传入数组拼接字符串，不过我比较喜欢集合拼接，所以再推荐下 Guava 的 Joiner。 实例代码如下： ``` java String[] array = new String[]{"test", "1234", "5678"}; List list=new ArrayList<>(); list.add("test"); list.add("1234"); list.add("5678"); StringUtils.join(array, ","); // 逗号分隔符，跳过 null Joiner joiner=Joiner.on(",").skipNulls(); joiner.join(array); joiner.join(list); ``` ### 字符串拆分 有字符串拼接，就会有拆分字符串的需求，同样的 StringUtils 也有拆分字符串的方法。 ``` java StringUtils.split("a..b.c", '.') = ["a", "b", "c"] StringUtils.splitByWholeSeparatorPreserveAllTokens("a..b.c", ".")= ["a","", "b", "c"] ``` >ps:注意以上两个方法区别。 StringUtils 拆分之后得到是一个数组，我们可以使用 Guava 的 ``` java Splitter splitter = Splitter.on(","); // 返回是一个 List 集合，结果：[ab, , b, c] splitter.splitToList("ab,,b,c"); // 忽略空字符串，输出结果 [ab, b, c] splitter.omitEmptyStrings().splitToList("ab,,b,c") ``` StringUtils 内部还有其他常用的方法，小伙伴可以自行查看其 API。 ## 日期相关工具类 ### DateUtils/DateFormatUtils JDK8 之前，Java 只提供一个 Date 类，平常我们需要将 Date 按照一定格式转化成字符串，我们需要使用 SimpleDateFormat。 ``` java SimpleDateFormat simpleDateFormat=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); // Date 转 字符串 simpleDateFormat.format(new Date()); // 字符串 转 Date simpleDateFormat.parse("2020-05-07 22:00:00"); ``` 代码虽然简单，但是这里需要注意 SimpleDateFormat，不是线程安全的，多线程环境一定要注意使用安全。 这里阿粉推荐 commons-lang3 下的时间工具类DateUtils/DateFormatUtils,解决 Date 与字符串转化问题。 >ps:吐槽一下，你们工程中有没有多个叫 DateUtils 类？阿粉发现我们现有工程，多个模块有提供这个类，每个实现大同小异。 使用方法非常简单： ``` java // Date 转化为字符串 DateFormatUtils.format(new Date(),"yyyy-MM-dd HH:mm:ss"); // 字符串 转 Date DateUtils.parseDate("2020-05-07 22:00:00","yyyy-MM-dd HH:mm:ss"); ``` 除了格式转化之外，DateUtils 还提供时间计算的相关功能。 ``` java Date now = new Date(); // Date 加 1 天 Date addDays = DateUtils.addDays(now, 1); // Date 加 33 分钟 Date addMinutes = DateUtils.addMinutes(now, 33); // Date 减去 233 秒 Date addSeconds = DateUtils.addSeconds(now, -233); // 判断是否 Wie 同一天 boolean sameDay = DateUtils.isSameDay(addDays, addMinutes); // 过滤时分秒,若 now 为 2020-05-07 22:13:00 调用 truncate 方法以后 // 返回时间为 2020-05-07 00:00:00 Date truncate = DateUtils.truncate(now, Calendar.DATE); ``` ## JDK8 时间类 JDK8 之后，Java 将日期与时间分为 LocalDate，LocalTime，功能定义更加清晰，当然其也提供一个 LocalDateTime，包含日期与时间。这些类相对于 Date 类优点在于，这些类与 String 类一样都是不变类型，不但线程安全，而且不能修改。 >ps：仔细对比 mysql 时间日期类型 DATE，TIME，DATETIME，有没有感觉差不多 现在 mybatis 等 ORM 框架已经支持 LocalDate 与 JDBC 时间类型转化，所以大家可以直接将时间字段实际类型定义为 JDK8 时间类型，然后再进行相关转化。 如果依然使用的是 Date 类型，如果需要使用新的时间类型，我们需要进行相关转化。两者之间进行转化， 稍微复杂一点，我们需要显示指定当前时区。 ``` java Date now = new Date(); // Date-----> LocalDateTime 这里指定使用当前系统默认时区 LocalDateTime localDateTime = now.toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime(); // LocalDateTime------> Date 这里指定使用当前系统默认时区 Date date = Date.from(localDateTime.atZone(ZoneId.systemDefault()).toInstant()); ``` 接下来我们使用 LocalDateTime 进行字符串格式化。 ``` java // 按照 yyyy-MM-dd HH:mm:ss 转化时间 LocalDateTime dateTime = LocalDateTime.parse("2020-05-07 22:34:00", DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")); // 将 LocalDateTime 格式化字符串 String format = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss").format(dateTime); ``` 另外我们使用 LocalDateTime 获取当前时间年份，月份特别简单： ``` java LocalDateTime now = LocalDateTime.now(); // 年 int year = now.getYear(); // 月 int month = now.getMonthValue(); // 日 int day = now.getDayOfMonth(); ``` 最后我们还可以使用 LocalDateTime 进行日期加减，获取下一天的时间： ``` java LocalDateTime now = LocalDateTime.now(); // 当前时间加一天 LocalDateTime plusDays = now.plusDays(1l); // 当前时间减一个小时 LocalDateTime minusHours = now.minusHours(1l); // 还有很多其他方法 ``` 总之 JDK8 提供的时间类非常好用，还没用过小伙伴，可以尝试下。 ## 集合/数组相关 集合与数组我们日常也需要经常使用，也需要对其进行判空： ``` java if (null == list || list.isEmpty()) { } ``` >ps: 数组、Map 集合与其类似 上面代码如字符串判空一样写起来都非常简单，但是也比较容易写出会抛出空指针异常的代码。这里我们可以使用 commons-collections 提供工具类。 pom 信息： ``` pom org.apache.commons commons-collections4 4.4 ``` > ps: 还有一个低版本的 ，artifactId 为 commons-collections 我们可以使用 CollectionUtils/MapUtils进行判空判断。 ``` java // List/Set 集合判空 if(CollectionUtils.isEmpty(list)){ } // Map 等集合进行判空 if (MapUtils.isEmpty(map)) { } ``` 至于数组判空判断需要使用 commons-lang 下的 ArrayUtils进行判断: ``` java // 数组判空 if (ArrayUtils.isEmpty(array)) { } ``` 除此之外还有一些列的对于集合增强方法，比如快速将数组加入到现有集合中： ``` java List listA = new ArrayList<>(); listA.add("1"); listA.add("2"); listA.add("3"); String[] arrays = new String[]{"a", "b", "c"}; CollectionUtils.addAll(listA, arrays); ``` ## I/O 相关 JDK 有提供一系列的类可以读取文件等，不过阿粉觉得那些类有些晦涩难懂，实现一个小功能可能还要写好多代码，而且还不一定能写对。 阿粉推荐一下 Apache 提供的 commons-io 库，增强 I/O 操作，简化操作难度。pom 信息： ``` pom commons-io commons-io 2.6 ``` ### FileUtils-文件操作工具类 文件操作工具类提供一系列方法，可以让我们快速读取写入文件。 快速实现文件/文件夹拷贝操作 ,FileUtils.copyDirectory/FileUtils.copyFile ``` java // 拷贝文件 File fileA = new File("E:\\test\\test.txt"); File fileB = new File("E:\\test1\\test.txt"); FileUtils.copyFile(fileA,fileB); ``` 使用 FileUtils.listFiles 获取指定文件夹上所有文件 ``` java // 按照指定文件后缀如java,txt等去查找指定文件夹的文件 File directory = new File("E:\\test"); FileUtils.listFiles(directory, new String[]{"txt"}, false); ``` 使用 FileUtils.readLines 读取该文件所有行。 ``` java // 读取指定文件所有行 不需要使用 while 循环读取流了 List lines = FileUtils.readLines(fileA) ``` 有读就存在写，可以使用 FileUtils.writeLines，直接将集合中数据，一行行写入文本。 ``` java // 可以一行行写入文本 List lines = new ArrayList<>(); ..... FileUtils.writeLines(lines) ``` ### IOUtils-I/O 操作相关工具类 FileUtils 主要针对相关文件操作，IOUtils 更加针对底层 I/O,可以快速读取 InputStream。实际上 FileUtils 底层操作依赖就是 IOUtils。 IOUtils可以适用于一个比较试用的场景，比如支付场景下，HTTP 异步通知场景。如果我们使用 JDK 原生方法写: >从 Servlet 获取异步通知内容 ``` java byte[] b = null; ByteArrayOutputStream baos = null; String respMsg = null; try { byte[] buffer = new byte[1024]; baos = new ByteArrayOutputStream(); // 获取输入流 InputStream in = request.getInputStream(); for (int len = 0; (len = in.read(buffer)) > 0; ) { baos.write(buffer, 0, len); } b = baos.toByteArray(); baos.close(); // 字节数组转化成字符串 String reqMessage = new String(b, "utf-8"); } catch (IOException e) { } finally { if (baos != null) { try { baos.close(); } catch (IOException e) { } } } ``` 上面代码说起来还是挺复杂的。不过我们使用 IOUtils，一个方法就可以简单搞定： ``` java // 将输入流信息全部输出到字节数组中 byte[] b = IOUtils.toByteArray(request.getInputStream()); // 将输入流信息转化为字符串 String resMsg = IOUtils.toString(request.getInputStream()); ``` >ps: InputStream 不能被重复读取 ### 计时 编程中有时需要统计代码的的执行耗时，当然执行代码非常简单，结束时间与开始时间相减即可。 ``` java long start = System.currentTimeMillis(); //获取开始时间 //其他代码 //... long end = System.currentTimeMillis(); //获取结束时间 System.out.println("程序运行时间： " + (end - start) + "ms"); ``` 虽然代码很简单，但是非常不灵活，默认情况我们只能获取 ms 单位，如果需要转换为秒，分钟，就需要另外再计算。 这里我们介绍 Guava Stopwatch 计时工具类，借助他统计程序执行时间，使用方式非常灵活。 > commons-lang3 与 Spring-core 也有这个工具类，使用方式大同小异，大家根据情况选择。 ``` java // 创建之后立刻计时，若想主动开始计时 Stopwatch stopwatch = Stopwatch.createStarted(); // 创建计时器，但是需要主动调用 start 方法开始计时 // Stopwatch stopwatch = Stopwatch.createUnstarted(); // stopWatch.start(); // 模拟其他代码耗时 TimeUnit.SECONDS.sleep(2l); // 当前已经消耗的时间 System.out.println(stopwatch.elapsed(TimeUnit.SECONDS));; TimeUnit.SECONDS.sleep(2l); // 停止计时 未开始的计时器调用 stop 将会抛错 IllegalStateException stopwatch.stop(); // 再次统计总耗时 System.out.println(stopwatch.elapsed(TimeUnit.SECONDS));; // 重新开始，将会在原来时间基础计算，若想重新从 0开始计算，需要调用 stopwatch.reset() stopwatch.start(); TimeUnit.SECONDS.sleep(2l); System.out.println(stopwatch.elapsed(TimeUnit.SECONDS)); ``` 输出结果为： ``` 2 4 6 ``` ================================================ FILE: docs/java/java-validator.md ================================================ # Validator 注解使用 ## 为什么要用validator * javax.validation的一系列注解可以帮我们完成参数校验,免去繁琐的串行校验 不然我们的代码就像下面这样: ``` java /** * 走串行校验 * * @param userVO * @return */ @PostMapping("/save/serial") public Object save(@RequestBody UserVO userVO) { String mobile = userVO.getMobile(); //手动逐个 参数校验~ 写法 if (StringUtils.isBlank(mobile)) { return RspDTO.paramFail("mobile:手机号码不能为空"); } else if (!Pattern.matches("^[1][3,4,5,6,7,8,9][0-9]{9}$", mobile)) { return RspDTO.paramFail("mobile:手机号码格式不对"); } //抛出自定义异常等~写法 if (StringUtils.isBlank(userVO.getUsername())) { throw new BizException(Constant.PARAM_FAIL_CODE, "用户名不能为空"); } // 比如写一个map返回 if (StringUtils.isBlank(userVO.getSex())) { Map result = new HashMap<>(5); result.put("code", Constant.PARAM_FAIL_CODE); result.put("msg", "性别不能为空"); return result; } //.........各种写法 ... userService.save(userVO); return RspDTO.success(); } ``` 这被大佬看见,一定说,都9102了还这么写,然后被劝退了 * 什么是javax.validation >JSR303 是一套JavaBean参数校验的标准，它定义了很多常用的校验注解，我们可以直接将这些注解加在我们JavaBean的属性上面(面向注解编程的时代)，就可以在需要校验的时候进行校验了,在SpringBoot中已经包含在starter-web中,再其他项目中可以引用依赖,并自行调整版本: ``` java javax.validation validation-api 1.1.0.Final org.hibernate hibernate-validator 5.2.0.Final ``` ![java-validator](../img/java-validator-1.png) * 注解说明 ``` java 1.@NotNull：不能为null，但可以为empty(""," "," ") 2.@NotEmpty：不能为null，而且长度必须大于0 (" "," ") 3.@NotBlank：只能作用在String上，不能为null，而且调用trim()后，长度必须大于0("test") 即：必须有实际字符 ``` 验证注解|验证的数据类型|说明 --|:--:|--: @AssertFalse|Boolean,boolean|验证注解的元素值是false @AssertTrue|Boolean,boolean|验证注解的元素值是true @NotNull|任意类型|验证注解的元素值不是null @Null|任意类型|验证注解的元素值是null @Min(value=值)|BigDecimal，BigInteger, byte,short, int, long，等任何Number或CharSequence（存储的是数字）子类型|验证注解的元素值大于等于@Min指定的value值 @Max（value=值）|和@Min要求一样|验证注解的元素值小于等于@Max指定的value值 @DecimalMin(value=值)|和@Min要求一样|验证注解的元素值大于等于@ DecimalMin指定的value值 @DecimalMax(value=值)|和@Min要求一样|验证注解的元素值小于等于@ DecimalMax指定的value值 @Digits(integer=整数位数, fraction=小数位数)|和@Min要求一样|验证注解的元素值的整数位数和小数位数上限 @Size(min=下限, max=上限)|字符串、Collection、Map、数组等|验证注解的元素值的在min和max（包含）指定区间之内，如字符长度、集合大小 @Past|java.util.Date,java.util.Calendar;Joda Time类库的日期类型|验证注解的元素值（日期类型）比当前时间早 @Future|与@Past要求一样|验证注解的元素值（日期类型）比当前时间晚 @NotBlank|CharSequence子类型|验证注解的元素值不为空（不为null、去除首位空格后长度为0），不同于@NotEmpty，@NotBlank只应用于字符串且在比较时会去除字符串的首位空格 @Length(min=下限, max=上限)|CharSequence子类型|验证注解的元素值长度在min和max区间内 @NotEmpty|CharSequence子类型、Collection、Map、数组|验证注解的元素值不为null且不为空（字符串长度不为0、集合大小不为0） @Range(min=最小值, max=最大值)|BigDecimal,BigInteger,CharSequence, byte, short, int, long等原子类型和包装类型|验证注解的元素值在最小值和最大值之间 @Email(regexp=正则表达式,flag=标志的模式)|CharSequence子类型（如String）|验证注解的元素值是Email，也可以通过regexp和flag指定自定义的email格式 @Pattern(regexp=正则表达式,flag=标志的模式)|String，任何CharSequence的子类型|验证注解的元素值与指定的正则表达式匹配 @Valid|任何非原子类型|指定递归验证关联的对象如用户对象中有个地址对象属性，如果想在验证用户对象时一起验证地址对象的话，在地址对象上加@Valid注解即可级联验证 此处只列出Hibernate Validator提供的大部分验证约束注解，请参考hibernate validator官方文档了解其他验证约束注解和进行自定义的验证约束注解定义。 ## 实战演练 * @Validated 声明要检查的参数 这里我们在控制器层进行注解声明 ``` java /** * 走参数校验注解 * * @param userDTO * @return */ @PostMapping("/save/valid") public RspDTO save(@RequestBody @Validated UserDTO userDTO) { userService.save(userDTO); return RspDTO.success(); } ``` * 对参数的字段进行注解标注 ``` java import lombok.Data; import org.hibernate.validator.constraints.Length; import javax.validation.constraints.*; import java.io.Serializable; import java.util.Date; /** * @author LiJing * @ClassName: UserDTO * @Description: 用户传输对象 * @date 2019/7/30 13:55 */ @Data public class UserDTO implements Serializable { private static final long serialVersionUID = 1L; /*** 用户ID*/ @NotNull(message = "用户id不能为空") private Long userId; /** 用户名*/ @NotBlank(message = "用户名不能为空") @Length(max = 20, message = "用户名不能超过20个字符") @Pattern(regexp = "^[\\u4E00-\\u9FA5A-Za-z0-9\\*]*$", message = "用户昵称限制：最多20字符，包含文字、字母和数字") private String username; /** 手机号*/ @NotBlank(message = "手机号不能为空") @Pattern(regexp = "^[1][3,4,5,6,7,8,9][0-9]{9}$", message = "手机号格式有误") private String mobile; /**性别*/ private String sex; /** 邮箱*/ @NotBlank(message = "联系邮箱不能为空") @Email(message = "邮箱格式不对") private String email; /** 密码*/ private String password; /*** 创建时间 */ @Future(message = "时间必须是将来时间") private Date createTime; } ``` * 在全局校验中增加校验异常 > MethodArgumentNotValidException是springBoot中进行绑定参数校验时的异常,需要在springBoot中处理,其他需要 处理ConstraintViolationException异常进行处理. * 为了优雅一点,我们将参数异常,业务异常,统一做了一个全局异常,将控制层的异常包装到我们自定义的异常中 * 为了优雅一点,我们还做了一个统一的结构体,将请求的code,和msg,data一起统一封装到结构体中,增加了代码的复用性 ``` java import com.boot.lea.mybot.dto.RspDTO; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.dao.DuplicateKeyException; import org.springframework.web.bind.MethodArgumentNotValidException; import org.springframework.web.bind.annotation.ExceptionHandler; import org.springframework.web.bind.annotation.RestControllerAdvice; import org.springframework.web.servlet.NoHandlerFoundException; import javax.validation.ConstraintViolationException; import javax.validation.ValidationException; /** * @author LiJing * @ClassName: GlobalExceptionHandler * @Description: 全局异常处理器 * @date 2019/7/30 13:57 */ @RestControllerAdvice public class GlobalExceptionHandler { private Logger logger = LoggerFactory.getLogger(getClass()); private static int DUPLICATE_KEY_CODE = 1001; private static int PARAM_FAIL_CODE = 1002; private static int VALIDATION_CODE = 1003; /** * 处理自定义异常 */ @ExceptionHandler(BizException.class) public RspDTO handleRRException(BizException e) { logger.error(e.getMessage(), e); return new RspDTO(e.getCode(), e.getMessage()); } /** * 方法参数校验 */ @ExceptionHandler(MethodArgumentNotValidException.class) public RspDTO handleMethodArgumentNotValidException(MethodArgumentNotValidException e) { logger.error(e.getMessage(), e); return new RspDTO(PARAM_FAIL_CODE, e.getBindingResult().getFieldError().getDefaultMessage()); } /** * ValidationException */ @ExceptionHandler(ValidationException.class) public RspDTO handleValidationException(ValidationException e) { logger.error(e.getMessage(), e); return new RspDTO(VALIDATION_CODE, e.getCause().getMessage()); } /** * ConstraintViolationException */ @ExceptionHandler(ConstraintViolationException.class) public RspDTO handleConstraintViolationException(ConstraintViolationException e) { logger.error(e.getMessage(), e); return new RspDTO(PARAM_FAIL_CODE, e.getMessage()); } @ExceptionHandler(NoHandlerFoundException.class) public RspDTO handlerNoFoundException(Exception e) { logger.error(e.getMessage(), e); return new RspDTO(404, "路径不存在，请检查路径是否正确"); } @ExceptionHandler(DuplicateKeyException.class) public RspDTO handleDuplicateKeyException(DuplicateKeyException e) { logger.error(e.getMessage(), e); return new RspDTO(DUPLICATE_KEY_CODE, "数据重复，请检查后提交"); } @ExceptionHandler(Exception.class) public RspDTO handleException(Exception e) { logger.error(e.getMessage(), e); return new RspDTO(500, "系统繁忙,请稍后再试"); } } ``` * 测试 如下文:确实做到了参数校验时返回异常信息和对应的code,方便了我们不再繁琐的处理参数校验 ![java-validatior](../img/java-validatior-2.png) ## 自定义参数注解 * 比如我们来个 自定义身份证校验 注解 ``` java @Documented @Target({ElementType.PARAMETER, ElementType.FIELD}) @Retention(RetentionPolicy.RUNTIME) @Constraint(validatedBy = IdentityCardNumberValidator.class) public @interface IdentityCardNumber { String message() default "身份证号码不合法"; Class[] groups() default {}; Class[] payload() default {}; } ``` 这个注解是作用在Field字段上，运行时生效，触发的是IdentityCardNumber这个验证类。 >message 定制化的提示信息，主要是从ValidationMessages.properties里提取，也可以依据实际情况进行定制 >groups 这里主要进行将validator进行分类，不同的类group中会执行不同的validator操作 >payload 主要是针对bean的，使用不多。 * 然后自定义Validator 这个是真正进行验证的逻辑代码： ``` java public class IdentityCardNumberValidator implements ConstraintValidator { @Override public void initialize(IdentityCardNumber identityCardNumber) { } @Override public boolean isValid(Object o, ConstraintValidatorContext constraintValidatorContext) { return IdCardValidatorUtils.isValidate18Idcard(o.toString()); } } ``` * 使用自定义的注解 ``` java @NotBlank(message = "身份证号不能为空") @IdentityCardNumber(message = "身份证信息有误,请核对后提交") private String clientCardNo; ``` * 使用groups的校验 有的宝宝说同一个对象要复用,比如UserDTO在更新时候要校验userId,在保存的时候不需要校验userId,在两种情况下都要校验username,那就用上groups了: >先定义groups的分组接口Create和Update ``` java import javax.validation.groups.Default; public interface Create extends Default { } import javax.validation.groups.Default; public interface Update extends Default{ } ``` >再在需要校验的地方@Validated声明校验组 ```java /** * 走参数校验注解的 groups 组合校验 * * @param userDTO * @return */ @PostMapping("/update/groups") public RspDTO update(@RequestBody @Validated(Update.class) UserDTO userDTO) { userService.updateById(userDTO); return RspDTO.success(); } ``` > 在DTO中的字段上定义好groups = {}的分组类型 ``` java @Data public class UserDTO implements Serializable { private static final long serialVersionUID = 1L; /*** 用户ID*/ @NotNull(message = "用户id不能为空", groups = Update.class) private Long userId; /** * 用户名 */ @NotBlank(message = "用户名不能为空") @Length(max = 20, message = "用户名不能超过20个字符", groups = {Create.class, Update.class}) @Pattern(regexp = "^[\\u4E00-\\u9FA5A-Za-z0-9\\*]*$", message = "用户昵称限制：最多20字符，包含文字、字母和数字") private String username; /** * 手机号 */ @NotBlank(message = "手机号不能为空") @Pattern(regexp = "^[1][3,4,5,6,7,8,9][0-9]{9}$", message = "手机号格式有误", groups = {Create.class, Update.class}) private String mobile; /** * 性别 */ private String sex; /** * 邮箱 */ @NotBlank(message = "联系邮箱不能为空") @Email(message = "邮箱格式不对") private String email; /** * 密码 */ private String password; /*** 创建时间 */ @Future(message = "时间必须是将来时间", groups = {Create.class}) private Date createTime; } ``` 注意:在声明分组的时候尽量加上 extend javax.validation.groups.Default否则,在你声明@Validated(Update.class)的时候,就会出现你在默认没添加groups = {}的时候的校验组@Email(message = "邮箱格式不对"),会不去校验,因为默认的校验组是groups = {Default.class}. * restful风格用法 > 在多个参数校验,或者@RequestParam 形式时候,需要在controller上加注@Validated ``` java @GetMapping("/get") public RspDTO getUser(@RequestParam("userId") @NotNull(message = "用户id不能为空") Long userId) { User user = userService.selectById(userId); if (user == null) { return new RspDTO().nonAbsent("用户不存在"); } return new RspDTO().success(user); } @RestController @RequestMapping("user/") @Validated public class UserController extends AbstractController { @GetMapping("/get") public RspDTO getUser(@RequestParam("userId") @NotNull(message = "用户id不能为空") Long userId) { User user = userService.selectById(userId); if (user == null) { return new RspDTO().nonAbsent("用户不存在"); } return new RspDTO().success(user); } } ``` ================================================ FILE: docs/java/javavm.md ================================================ # Java 虚拟机

## 一、基本概念 ### 1.1 OpenJDK 自 1996 年 `JDK 1.0` 发布以来，Sun 公司在大版本上发行了 `JDK 1.1`、`JDK 1.2`、`JDK 1.3`、`JDK 1.4`、`JDK 5`，`JDK 6` ，这些版本的 JDK 都可以统称为 SunJDK 。之后在 2006 年的 JavaOne 大会上，Sun 公司宣布将 Java 开源，在随后的一年多里，它陆续将 JDK 的各个部分在 GPL v2（GNU General Public License，version 2）协议下开源，并建立了 OpenJDK 组织来对这些代码进行独立的管理，这就是 OpenJDK 的来源，此时的 OpenJDK 拥有当时 sunJDK 7 的几乎全部代码。 ### 1.2 OracleJDK 在 JDK 7 的开发期间，由于各种原因的影响 Sun 公司市值一路下跌，已无力推进 JDK 7 的开发，JDK 7 的发布一直被推迟。之后在 2009 年 Sun 公司被 Oracle 公司所收购，为解决 JDK 7 长期跳票的问题，Oracle 将 JDK 7 中大部分未能完成的项目推迟到 JDK 8 ，并于 2011 年发布了JDK 7，在这之后由 Oracle 公司正常发行的 JDK 版本就由 SunJDK 改称为 Oracle JDK。 在 2017 年 JDK 9 发布后，Oracle 公司宣布从此以后 JDK 将会在每年的 3 月和 9 月各发布一个大版本，即半年发行一个大版本，目的是为了避免众多功能被捆绑到一个 JDK 版本上而引发的无法交付的风险。 在 JDK 11 发布后，Oracle 同步调整了 JDK 的商业授权，宣布从 JDK 11 起将以前的商业特性全部开源给 OpenJDK ，这样 OpenJDK 11 和 OracleJDK 11 的代码和功能，在本质上就完全相同了。同时还宣布以后都会发行两个版本的 JDK ： + 一个是在 GPLv2 + CE 协议下由 Oracle 开源的 OpenJDK； + 一个是在 OTN 协议下正常发行的 OracleJDK。 两者共享大部分源码，在功能上几乎一致。唯一的区别是 Oracle OpenJDK 可以在开发、测试或者生产环境中使用，但只有半年的更新支持；而 OracleJDK 对个人免费，但在生产环境中商用收费，可以有三年时间的更新支持。 ### 1.3 HotSpot VM 它是 Sun/Oracle JDK 和 OpenJDK 中默认的虚拟机，也是目前使用最为广泛的虚拟机。最初由 Longview Technologies 公司设计发明，该公司在 1997 年被 Sun 公司收购，随后 Sun 公司在 2006 年开源 SunJDK 时也将 HotSpot 虚拟机一并进行了开源。之后 Oracle 收购 Sun 以后，建立了 HotRockit 项目，用于将其收购的另外一家公司（BEA）的 JRockit 虚拟机中的优秀特性集成到 HotSpot 中。HotSpot 在这个过程里面移除掉永久代，并吸收了 JRockit 的 Java Mission Control 监控工具等功能。到 JDK 8 发行时，采用的就是集两者之长的 HotSpot VM 。 我们可以在自己的电脑上使用 `java -version` 来获得 JDK 的信息： ```shell C:\Users> java -version java version "1.8.0_171" # 如果是openJDK, 则这里会显示：openjdk version Java(TM) SE Runtime Environment (build 1.8.0_171-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) # 使用的是HotSpot虚拟机，默认为服务端模式 ``` ## 二、Java 内存区域
### 2.1 程序计数器 程序计数器（Program Counter Register）是一块较小的内存空间，它可以看做是当前线程所执行的字节码的行号指示器。字节码解释器通过改变程序计数器的值来选取下一条需要执行的字节码指令，分支、循环、跳转、异常处理、线程恢复等基础功能都需要该计数器来完成。每条线程都拥有一个独立的程序计数器，各条线程之间的计数器互不影响，独立存储。 ### 2.2 Java虚拟机栈 Java 虚拟机栈（Java Virtual Machine Stack）也为线程私有，它描述的是 Java 方法执行的线程内存模型：每个方法被执行的时候，Java 虚拟机都会同步创建一个栈帧，用于存储局部变量表、操作数栈、动态连接、方法出口等信息。方法从调用到结束就对应着一个栈帧从入栈到出栈的过程。在《Java 虚拟机规范》中，对该内存区域规定了两类异常： + 如果线程请求的栈深度大于虚拟机所允许的栈深度，将抛出 `StackOverflowError` 异常； + 如果 Java 虚拟机栈的容量允许动态扩展，当栈扩展时如果无法申请到足够的内存会抛出 `OutOfMemoryError` 异常。 ### 2.3 本地方法栈 本地方法栈（Native Method Stacks）与虚拟机栈类似，其区别在于：Java 虚拟机栈是为虚拟机执行 Java 方法（也就是字节码）服务，而本地方法栈则是为虚拟机使用到的本地（Native）方法服务。 ### 2.4 Java堆 Java 堆（Java Heap）是虚拟机所管理的最大一块的内存空间，它被所有线程所共享，用于存放对象实例。Java 堆可以处于物理上不连续的内存空间中，但在逻辑上它应该被视为是连续的。Java 堆可以被实现成固定大小的，也可以是可扩展的，当前大多数主流的虚拟机都是按照可扩展来实现的，即可以通过最大值参数 `-Xmx` 和最小值参数 `-Xms` 进行设定。如果 Java 堆中没有足够的内存来完成实例分配，并且堆也无法再扩展时，Java 虚拟机将会抛出 `OutOfMemoryError` 异常。 ### 2.5 方法区 方法区（Method Area）也是各个线程共享的内存区域，它用于存储已被虚拟机加载的类型信息、常量、静态变量、即时编译器编译后的代码缓存等数据。方法区也被称为 “非堆”，目的是与 Java 堆进行区分。《Java 虚拟机规范》规定，如果方法区无法满足新的内存分配需求时，将会抛出 `OutOfMemoryError` 异常。 运行时常量池（Runtime Constant Pool）是方法区的一部分，用于存放常量池表（Constant Pool Table），常量池表中存放了编译期生成的各种符号字面量和符号引用。 ## 三、对象 ### 3.1 对象的创建 当我们在代码中使用 `new` 关键字创建一个对象时，其在虚拟机中需要经过以下步骤： **1. 类加载过程** 当虚拟机遇到一条字节码 `new` 指令时，首先将去检查这个指令的参数是否能在常量池中定位到一个符号引用，并且检查这个符号引用代表的类是否已被加载、解析和初始化过。如果没有，就必须先执行相应的类加载过程。 **2. 分配内存** 在类加载检查通过后，虚拟机需要新生对象分配内存空间。根据 Java 堆是否规整，可以有以下两种分配方案： + **指针碰撞**：假设 Java 堆中内存是绝对规整的，所有使用的内存放在一边，所有未被使用的内存放在另外一边，中间以指针作为分界点指示器。此时内存分配只是将指针向空闲方向偏移出对象大小的空间即可，这种方式被称为指针碰撞。
+ **空闲列表**：如果 Java 堆不是规整的，此时虚拟机需要维护一个列表，记录哪些内存块是可用的，哪些是不可用的。在进行内存分配时，只需要从该列表中选取出一块足够的内存空间划分给对象实例即可。 > 注：Java 堆是否规整取决于其采用的垃圾收集器是否带有空间压缩整理能力，后文将会介绍。 除了分配方式外，由于对象创建在虚拟机中是一个非常频繁的行为，此时需要保证在并发环境下的线程安全：如果一个线程给对象 A 分配了内存空间，但指针还没来得及修改，此时就可能出现另外一个线程使用原来的指针来给对象 B 分配内存空间的情况。想要解决这个问题有两个方案： + **方式一**：采用同步锁定，或采用 CAS 配上失败重试的方式来保证更新操作的原子性。 + **方式二**：为每个线程在 Java 堆中预先分配一块内存，称为本地线程分配缓冲（Thread Local Allocation Buffer，TLAB）。线程在进行内存分配时优先使用本地缓冲，当本地缓冲使用完成后，再向 Java 堆申请分配，此时 Java 堆采用同步锁定的方式来保证分配行为的线程安全。 **3. 对象头设置** 将对象有关的元数据信息、对象的哈希码、分代年龄等信息存储到对象头中。 **4. 对象初始化** 调用对象的构造函数，即 Class 文件中的 `()` 来初始化对象，为相关字段赋值。 ### 3.2 对象的内存布局 在 HotSpot 虚拟机中，对象在堆内存中的存储布局可以划分为以下三个部分： **1. 对象头 (Header)** 对象头包括两部分信息： + **Mark Word**：对象自身的运行时数据，如哈希码、GC 分代年龄、锁状态标志、线程持有的锁、偏向线程 ID、偏向时间戳等，官方统称为 Mark Word 。 + **类型指针**：对象指向它类型元数据的指针，Java 虚拟机通过这个指针来确定该对象是哪个类的示例。需要说明的是并非所有的虚拟机都必须要在对象数据上保留类型指针，这取决于对象的访问定位方式（详见下文）。 **2. 实例数据 (Instance Data)** 即我们在程序代码中定义的各种类型的字段的内容，无论是从父类继承而来，还是子类中定义的都需要记录。 **3. 对其填充 (Padding)** 主要起占位符的作用。HotSpot 虚拟机要求对象起始地址必须是 8 字节的整倍数，即间接要求了任何对象的大小都必须是 8 字节的整倍数。对象头部分在设计上就是 8 字节的整倍数，如果对象的实例数据不是 8 字节的整倍数，则由对齐填充进行补全。 ### 3.3 对象的访问定位 对象创建后，Java 程序就可以通过栈上的 `reference` 来操作堆上的具体对象。《Java 虚拟机规范》规定 `reference` 是一个指向对象的引用，但并未规定其具体实现方式。主流的方式方式有以下两种： + **句柄访问**：Java 堆将划分出一块内存来作为句柄池， `reference` 中存储的是对象的句柄地址，而句柄则包含了对象实例数据和类型数据的地址信息。 + **指针访问**：`reference` 中存储的直接就是对象地址，而对象的类型数据则由上文介绍的对象头中的类型指针来指定。 通过句柄访问对象：
通过直接指针访问对象：
句柄访问的优点在于对象移动时（垃圾收集时移动对象是非常普遍的行为）只需要改变句柄中实例数据的指针，而 `reference` 本生并不需要修改；指针访问则反之，由于其 `reference` 中存储的直接就是对象地址，所以当对象移动时， `reference` 需要被修改。但针对只需要访问对象本身的场景，指针访问则可以减少一次定位开销。由于对象访问是一项非常频繁的操作，所以这类减少的效果会非常显著，基于这个原因，HotSpot 主要使用的是指针访问的方式。 ## 四、垃圾收集算法 在 Java 虚拟机内存模型中，程序计数器、虚拟机栈、本地方法栈这 3 个区域都是线程私有的，会随着线程的结束而销毁，因此在这 3 个区域当中，无需过多考虑垃圾回收问题。垃圾回收问题主要发生在 Java 堆和方法区上。 ### 4.1 Java 堆回收 在 Java 堆上，垃圾回收的主要内容是死亡对象（不可能再被任何途径使用的对象）。而判断对象是否死亡有以下两种方法： #### 1. 引用计数法 在对象中添加一个引用计数器，对象每次被引用时，该计数器加一；当引用失效时，计数器的值减一；只要计数器的值为零，则代表对应的对象不可能再被使用。该方法的缺点在于无法避免相互循环引用的问题： ```java objA.instance = objB objB.instance = objA objA = null; objB = null; System.gc(); ``` 如上所示，此时两个对象已经不能再被访问，但其互相持有对对方的引用，如果采用引用计数法，则两个对象都无法被回收。 #### 2. 可达性分析 上面的代码在大多数虚拟机中都能被正确的回收，因为大多数主流的虚拟机都是采用的可达性分析方法来判断对象是否死亡。可达性分析是通过一系列被称为 `GC Roots` 的根对象作为起始节点集，从这些节点开始，根据引用关系向下搜索，搜索过程所走过的路径被称为引用链（Reference Chain），如果某个对象到 `GC Roots` 间没有任何引用链相连，这代表 `GC Roots` 到该对象不可达， 此时证明此该对象不可能再被使用。
在 Java 语言中，固定可作为 `GC Roots` 的对象包括以下几种： + 在虚拟机栈（栈帧中的本地变量表）中引用的对象，譬如各个线程被调用的方法堆栈中使用到的参数、局部变量、临时变量等； + 在方法区中类静态属性引用的对象，譬如 Java 类中引用类型的静态变量； + 在方法区中常量引用的对象，譬如字符串常量池（String Table）里的引用； + 在本地方法栈中的 JNI（即 Native 方法）引用的对象； + Java 虚拟机内部的引用，如基本数据类型对应的 Class 对象，一些常驻的异常对象（如 NullPointException，OutOfMemoryError 等）及系统类加载器； + 所有被同步锁（synchronized 关键字）持有的对象； + 反应 Java 虚拟机内部情况的 JMXBean，JVMTI 中注册的回调，本地代码缓存等。 除了这些固定的 `GC Roots` 集合以外，根据用户所选用的垃圾收集器以及当前回收的内存区域的不同，还可能会有其他对象 “临时性” 地加入，共同构成完整的 `GC Roots` 集合。 #### 3. 对象引用 可达性分析是基于引用链进行判断的，在 JDK 1.2 之后，Java 将引用关系分为以下四类： + **强引用 (Strongly Reference)** ：最传统的引用，如 `Object obj = new Object()` 。无论任何情况下，只要强引用关系还存在，垃圾收集器就永远不会回收掉被引用的对象。 + **软引用 (Soft Reference)** ：用于描述一些还有用，但非必须的对象。只被软引用关联着的对象，在系统将要发生内存溢出异常之前，会被列入回收范围内进行第二次回收，如果这次回收后还没有足够的内存，才会抛出内存溢出异常。 + **弱引用 (Weak Reference)** ：用于描述那些非必须的对象，强度比软引用弱。被弱引用关联对象只能生存到下一次垃圾收集发生时，无论当前内存是否足够，弱引用对象都会被回收。 + **虚引用 (Phantom Reference)** ：最弱的引用关系。为一个对象设置虚引用关联的唯一目的只是为了能在这个对象被回收时收到一个系统通知。 #### 4. 对象真正死亡 要真正宣告一个对象死亡，需要经过至少两次标记过程： + 如果对象在进行可达性分析后发现 `GC Roots` 不可达，将会进行第一次标记； + 随后进行一次筛选，筛选的条件是此对象是否有必要执行 `finalized()` 方法。如果对象没有覆盖 `finalized()` 方法，或者 `finalized()` 已经被虚拟机调用过，这两种情况都会视为没有必要执行。如果判定结果是有必要执行，此时对象会被放入名为 `F-Queue` 的队列，等待 Finalizer 线程执行其 `finalized()` 方法。在这个过程中，收集器会进行第二次小规模的标记，如果对象在 `finalized()` 方法中重新将自己与引用链上的任何一个对象进行了关联，如将自己（this 关键字）赋值给某个类变量或者对象的成员变量，此时它就实现了自我拯救，则第二次标记会将其移除 “即将回收” 的集合，否则该对象就将被真正回收，走向死亡。 ### 4.2 方法区回收 在 Java 堆上进行对象回收的性价比通常比较高，因为大多数对象都是朝生夕灭的。而方法区由于回收条件比较苛刻，对应的回收性价比通常比较低，主要回收两部分内容：废弃的常量和不再使用的类型。 ### 4.3 垃圾收集算法 #### 1. 分代收集理论 当前大多数虚拟机都遵循 “分代收集” 的理论进行设计，它建立在强弱两个分代假说下： + **弱分代假说 (Weak Generational Hypothesis)**：绝大多数对象都是朝生夕灭的。 + **强分代假说 (Strong Generational Hypothesis)**：熬过越多次垃圾收集过程的对象就越难以消亡。 + **跨带引用假说 (Intergenerational Reference Hypothesis)**：基于上面两条假说还可以得出的一条隐含推论：存在相互引用关系的两个对象，应该倾向于同时生存或者同时消亡。 强弱分代假说奠定了垃圾收集器的设计原则：收集器应该将 Java 堆划分出不同的区域，然后将回收对象依据其年龄（年龄就是对象经历垃圾收集的次数）分配到不同的区域中进行存储。之后如果一个区域中的对象都是朝生夕灭的，那么收集器只需要关注少量对象的存活而不是去标记那些大量将要被回收的对象，此时就能以较小的代价获取较大的空间。最后再将难以消亡的对象集中到一块，根据强分代假说，它们是很难消亡的，因此虚拟机可以使用较低的频率进行回收，这就兼顾了时间和内存空间的开销。 #### 2. 回收类型 根据分代收集理论，收集范围可以分为以下几种类型： + **部分收集 (Partial GC)**：具体分为： + 新生代收集（Minor GC / Young GC）：只对新生代进行垃圾收集； + 老年代收集（Major GC / Old GC）：只对老年代进行垃圾收集。需要注意的是 Major GC 在有的语境中也用于指代整堆收集； + 混合收集（Mixed GC）：对整个新生代和部分老年代进行垃圾收集。 + **整堆收集 (Full GC)**：收集整个 Java 堆和方法区。 #### 3. 标记-清除算法 它是最基础的垃圾收集算法，收集过程分为两个阶段：首先标记出所有需要回收的对象，在标记完成后，统一回收掉所有被标记的对象；也可以反过来，标记存活对象，统一回收所有未被标记的对象。
它主要有以下两个缺点： + 执行效率不稳定：如果 Java 堆上包含大量需要回收的对象，则需要进行大量标记和清除动作； + 内存空间碎片化：标记清除后会产生大量不连续的空间，从而可能导致无法为大对象分配足够的连续内存。 #### 4. 标记-复制算法 标记-复制算法基于 ”半区复制“ 算法：它将可用内存按容量划分为大小相等的两块，每次只使用其中一块，当这一块的内存使用完了，就将还存活着的对象复制到另外一块上面，然后再把已经使用过的那块内存空间一次性清理掉。其优点在于避免了内存空间碎片化的问题，其缺点如下： + 如果内存中多数对象都是存活的，这种算法将产生大量的复制开销； + 浪费内存空间，内存空间变为了原有的一半。
基于新生代 “朝生夕灭” 的特点，大多数虚拟机都不会按照 1:1 的比例来进行内存划分，例如 HotSpot 虚拟机会将内存空间划分为一块较大的 `Eden` 和 两块较小的 `Survivor` 空间，它们之间的比例是 8:1:1 。 每次分配时只会使用 `Eden` 和其中的一块 `Survivor` ，发生垃圾回收时，只需要将存活的对象一次性复制到另外一块 `Survivor` 上，这样只有 10% 的内存空间会被浪费掉。当 `Survivor` 空间不足以容纳一次 `Minor GC` 时，此时由其他内存区域（通常是老年代）来进行分配担保。 #### 5. 标记-整理算法 标记-整理算法是在标记完成后，让所有存活对象都向内存的一端移动，然后直接清理掉边界以外的内存。其优点在于可以避免内存空间碎片化的问题，也可以充分利用内存空间；其缺点在于根据所使用的收集器的不同，在移动存活对象时可能要全程暂停用户程序：
## 五、经典垃圾收集器 并行与并发是并发编程中的专有名词，在谈论垃圾收集器的上下文语境中，它们的含义如下： + **并行 (Parallel)**：并行描述的是多条垃圾收集器线程之间的关系，说明同一时间有多条这样的线程在协同工作，此时通常默认用户线程是处于等待状态。 + **并发 (Concurrent)**：并发描述的是垃圾收集器线程与用户线程之间的关系，说明同一时间垃圾收集器线程与用户线程都在运行。但由于垃圾收集器线程会占用一部分系统资源，所以程序的吞吐量依然会受到一定影响。 HotSpot 虚拟机中一共存在七款经典的垃圾收集器：
> 注：收集器之间存在连线，则代表它们可以搭配使用。 ### 5.1 Serial 收集器 Serial 收集器是最基础、历史最悠久的收集器，它是一个单线程收集器，在进行垃圾回收时，必须暂停其他所有的工作线程，直到收集结束，这是其主要缺点。它的优点在于单线程避免了多线程复杂的上下文切换，因此在单线程环境下收集效率非常高，由于这个优点，迄今为止，其仍然是 HotSpot 虚拟机在客户端模式下默认的新生代收集器：
### 5.2 ParNew 收集器 他是 Serial 收集器的多线程版本，可以使用多条线程进行垃圾回收：
### 5.3 Parallel Scavenge 收集器 Parallel Scavenge 也是新生代收集器，基于 标记-复制 算法进行实现，它的目标是达到一个可控的吞吐量。这里的吞吐量指的是处理器运行用户代码的时间与处理器总消耗时间的比值： $$ 吞吐量 = \frac{运行用户代码时间}{运行用户代码时间+运行垃圾收集时间} $$ Parallel Scavenge 收集器提供两个参数用于精确控制吞吐量： + **-XX:MaxGCPauseMillis**：控制最大垃圾收集时间，假设需要回收的垃圾总量不变，那么降低垃圾收集的时间就会导致收集频率变高，所以需要将其设置为合适的值，不能一味减小。 + **-XX:MaxGCTimeRatio**：直接用于设置吞吐量大小，它是一个大于 0 小于 100 的整数。假设把它设置为 19，表示此时允许的最大垃圾收集时间占总时间的 5%（即 1/(1+19) ）；默认值为 99 ，即允许最大 1%（ 1/(1+99) ）的垃圾收集时间。 ### 5.4 Serial Old 收集器 从名字也可以看出来，它是 Serial 收集器的老年代版本，同样是一个单线程收集器，采用 标记-整理 算法，主要用于给客户端模式下的 HotSpot 虚拟机使用：
### 5.5 Paralled Old 收集器 Paralled Old 是 Parallel Scavenge 收集器的老年代版本，支持多线程并发收集，采用 标记-整理 算法实现：
### 5.6 CMS 收集器 CMS（Concurrent Mark Sweep）收集器是一种以获取最短回收停顿时间为目标的收集器，基于 标记-清除 算法实现，整个收集过程分为以下四个阶段： 1. **初始标记 (inital mark)**：标记 `GC Roots` 能直接关联到的对象，耗时短但需要暂停用户线程； 2. **并发标记 (concurrent mark)**：从 `GC Roots` 能直接关联到的对象开始遍历整个对象图，耗时长但不需要暂停用户线程； 3. **重新标记 (remark)**：采用增量更新算法，对并发标记阶段因为用户线程运行而产生变动的那部分对象进行重新标记，耗时比初始标记稍长且需要暂停用户线程； 4. **并发清除 (inital sweep)**：并发清除掉已经死亡的对象，耗时长但不需要暂停用户线程。
其优点在于耗时长的 并发标记 和 并发清除 阶段都不需要暂停用户线程，因此其停顿时间较短，其主要缺点如下： + 由于涉及并发操作，因此对处理器资源比较敏感。 + 由于是基于 标记-清除 算法实现的，因此会产生大量空间碎片。 + 无法处理浮动垃圾（Floating Garbage）：由于并发清除时用户线程还是在继续，所以此时仍然会产生垃圾，这些垃圾就被称为浮动垃圾，只能等到下一次垃圾收集时再进行清理。 ### 5.7 Garbage First 收集器 Garbage First（简称 G1）是一款面向服务端的垃圾收集器，也是 JDK 9 服务端模式下默认的垃圾收集器，它的诞生具有里程碑式的意义。G1 虽然也遵循分代收集理论，但不再以固定大小和固定数量来划分分代区域，而是把连续的 Java 堆划分为多个大小相等的独立区域（Region）。每一个 Region 都可以根据不同的需求来扮演新生代的 `Eden` 空间、`Survivor` 空间或者老年代空间，收集器会根据其扮演角色的不同而采用不同的收集策略。
上面还有一些 Region 使用 H 进行标注，它代表 Humongous，表示这些 Region 用于存储大对象（humongous object，H-obj），即大小大于等于 region 一半的对象。G1 收集器的运行大致可以分为以下四个步骤： 1. **初始标记 (Inital Marking)**：标记 `GC Roots` 能直接关联到的对象，并且修改 TAMS（Top at Mark Start）指针的值，让下一阶段用户线程并发运行时，能够正确的在 Reigin 中分配新对象。G1 为每一个 Reigin 都设计了两个名为 TAMS 的指针，新分配的对象必须位于这两个指针位置以上，位于这两个指针位置以上的对象默认被隐式标记为存活的，不会纳入回收范围； 2. **并发标记 (Concurrent Marking)**：从 `GC Roots` 能直接关联到的对象开始遍历整个对象图。遍历完成后，还需要处理 SATB 记录中变动的对象。SATB（snapshot-at-the-beginning，开始阶段快照）能够有效的解决并发标记阶段因为用户线程运行而导致的对象变动，其效率比 CMS 重新标记阶段所使用的增量更新算法效率更高； 3. **最终标记 (Final Marking)**：对用户线程做一个短暂的暂停，用于处理并发阶段结束后仍遗留下来的少量的 STAB 记录。虽然并发标记阶段会处理 SATB 记录，但由于处理时用户线程依然是运行中的，因此依然会有少量的变动，所以需要最终标记来处理； 4. **筛选回收 (Live Data Counting and Evacuation)**：负责更新 Regin 统计数据，按照各个 Regin 的回收价值和成本进行排序，在根据用户期望的停顿时间进行来指定回收计划，可以选择任意多个 Regin 构成回收集。然后将回收集中 Regin 的存活对象复制到空的 Regin 中，再清理掉整个旧的 Regin 。此时因为涉及到存活对象的移动，所以需要暂停用户线程，并由多个收集线程并行执行。
### 5.8 内存分配原则 #### 1. 对象优先在 Eden 分配 大多数情况下，对象在新生代的 `Eden` 区中进行分配，当 `Eden` 区没有足够空间时，虚拟机将进行一次 Minor GC。 #### 2. 大对象直接进入老年代 大对象就是指需要大量连续内存空间的 Java 对象，最典型的就是超长的字符串或者元素数量很多的数组，它们将直接进入老年代。主要是因为如果在新生代分配，因为其需要大量连续的内存空间，可能会导致提前触发垃圾回收；并且由于新生代的垃圾回收本身就很频繁，此时复制大对象也需要额外的性能开销。 #### 3. 长期存活的对象将进入老年代 虚拟机会给每个对象在其对象头中定义一个年龄计数器。对象通常在 `Eden` 区中诞生，如果经历第一次 Minor GC 后仍然存活，并且能够被 Survivor 容纳的话，该对象就会被移动到 Survivor 中，并将其年龄加 1。对象在 Survivor 中每经过一次 Minor GC，年龄就加 1，当年龄达到一定程度后（由 `-XX:MaxTenuringThreshold` 设置，默认值为 15）就会进入老年代中。 #### 4. 动态年龄判断 如果在 Survivor 空间中相同年龄的所有对象大小的总和大于 Survivor 空间的一半，那么年龄大于或等于该年龄的对象就可以直接进入老年代，而无需等待年龄到达 `-XX:MaxTenuringThreshold` 设置的值。 #### 5. 空间担保分配 在发生 Minor GC 之前，虚拟机必须先检查老年代最大可用的连续空间是否大于新生代所有对象的总空间，如果条件成立，那么这一次的 Minor GC 可以确认是安全的。如果不成立，虚拟机会查看 `-XX:HandlePromotionFailure` 的值是否允许担保失败，如果允许那么就会继续检查老年代最大可用的连续空间是否大于历次晋升到老年代对象的平均大小，如果大于，将尝试着进行一次 Minor GC；如果小于或者 `-XX:HandlePromotionFailure` 的值设置不允许冒险，那么就要改为进行一次 Full GC 。 ## 六、虚拟机类加载机制 Java 虚拟机把描述类的数据从 Class 文件加载到内存，并对数据进行校验、转换解析和初始化，最终形成可以被虚拟机直接使用的 Java 类型，这个过程被称为虚拟机的类加载机制。 ### 6.1 类加载时机 一个类型从被加载到虚拟机内存中开始，到卸载出内存为止，它的整个生命周期将会经历加载、验证、准备、卸载、解析、初始化、使用、卸载七个阶段，其中验证、准备、解析三个部分统称为连接：
《Java 虚拟机规范》严格规定了有且只有六种情况必须立即对类进行初始化： 1. 遇到 `new`、 `getstatic`、 `putstatic`、 `invokestatic` 这四条字节码指令，如果类型进行过初始化，则需要先触发其进行初始化，能够生成这四条指令码的典型 Java 代码场景有： + 使用 `new` 关键字实例化对象时； + 读取或设置一个类型的静态字段时（被 final 修饰，已在编译期把结果放入常量池的静态字段除外）； + 调用一个类型的静态方法时。 2. 使用 `java.lang.reflect` 包的方法对类型进行反射调用时，如果类型没有进行过初始化、则需要触发其初始化； 3. 当初始化类时，如发现其父类还没有进行过初始化、则需要触发其父类进行初始化； 4. 当虚拟机启动时，用户需要指定一个要执行的主类（包含 main() 方法的那个类），虚拟机会先初始化这个主类； 5. 当使用 JDK 7 新加入的动态语言支持时，如果一个 `java.lang.invoke.MethodHandle` 实例最后解析的结果为 `REF_getStatic` ， `REF_putStatic` ， `REF_invokeStatic` ， `REF_newInvokeSpecial` 四种类型的方法句柄，并且这个方法句柄对应的类没有进行过初始化，则需要先触发其初始化； 6. 当一个接口中定义了 JDK 8 新加入的默认方法（被 default 关键字修饰的接口方法）时，如果有这个接口的实现类发生了初始化，那么该接口要在其之前被初始化。 ### 6.2 类加载过程 #### 1. 加载 在加载阶段，虚拟机需要完成以下三件事： + 通过一个类的全限定名来获取定义此类的二进制字节流 ； + 将这个字节流所代表的静态存储结构转换为方法区的运行时数据结构； + 在内存中生成一个代表这个类的 `java.lang.Class` 对象，作为方法区这个类的各种数据的访问入口。 《Java 虚拟机规范》并没有限制从何处获取二进制流，因此可以从 JAR 包、WAR 包获取，也可以从 JSP 生成的 Class 文件等处获取。 #### 2. 验证 这一阶段的目的是确保 Class 文件的字节流中包含的信息符合《Java 虚拟机规范》的全部约束要求，从而保证这些信息被当做代码运行后不会危害虚拟机自身的安全。验证阶段大致会完成下面四项验证： + **文件格式验证**：验证字节流是否符合 Class 文件格式的规范； + **元数据验证**：对字节码描述的信息进行语义分析，以保证其描述的信息符合《Java 语言规范》的要求（如除了 `java.lang.Object` 外，所有的类都应该有父类）； + **字节码验证**：通过数据流分析和控制流分析，确定程序语义是合法的，符合逻辑的（如允许把子类对象赋值给父类数据类型，但不能把父类对象赋值给子类数据类型）； + **符号引用验证**：验证类是否缺少或者被禁止访问它依赖的某些外部类、方法、字段等资源。如果无法验证通过，则会抛出一个`java.lang.IncompatibleClassChangeError` 的子类异常，如 `java.lang.NoSuchFieldError` 、 `java.lang.NoSuchMethodError` 等。 #### 3. 准备 准备阶段是正式为类中定义的变量（即静态变量，被 static 修饰的变量）分配内存并设置类变量初始值的阶段。 #### 4. 解析 解析是 Java 虚拟机将常量池内的符号引用替换为直接引用的过程： + **符号引用**：符号引用用一组符号来描述所引用的目标，符号可以是任何形式的字面量，只要使用时能无歧义地定位到目标即可。 + **直接引用**：直接引用是指可以直接指向目标的指针、相对偏移量或者一个能间接定位到目标的句柄。 整个解析动作主要针对类或接口、字段、类方法、接口方法、方法类型、方法句柄和调用点限定符这 7 类符号引用进行解析。 #### 5. 初始化 初始化阶段就是执行类构造器的 `()` 方法的过程，该方法具有以下特点： + `()` 方法由编译器自动收集类中所有类变量的赋值动作和静态语句块中的语句合并产生，编译器收集顺序由语句在源文件中出现的顺序决定。 + `()` 方法与类的构造器函数（即在虚拟机视角中的实例构造器 `()`方法）不同，它不需要显示的调用父类的构造器，Java 虚拟机会保证在子类的 `()` 方法执行前，父类的 `()` 方法已经执行完毕。 + 由于父类的 `()` 方法先执行，也就意味着父类中定义的静态语句块要优先于子类变量的赋值操作。 + `()` 方法对于类或者接口不是必须的，如果一个类中没有静态语句块，也没有对变量进行赋值操作，那么编译器可以不为这个类生成 `()` 方法。 + 接口中不能使用静态语句块，但仍然有变量初始化的赋值操作，因此接口与类一样都会生成 `()` 方法。 + Java 虚拟机必须保证一个类的 `()` 方法在多线程环境中被正确的加锁同步，如果多个线程同时去初始化一个类，那么只会有其中一个线程去执行这个类的 `()` 方法，其他线程都需要阻塞等待。 ### 6.3 类加载器 能够通过一个类的全限定名来获取描述该类的二进制字节流的工具称为类加载器。每一个类加载器都拥有一个独立的类名空间，因此对于任意一个类，都必须由加载它的类加载器和这个类本身来共同确立其在 Java 虚拟机中的唯一性。这意味着要想比较两个类是否相等，必须在同一类加载器加载的前提下；如果两个类的类加载器不同，则它们一定不相等。 ### 6.4 双亲委派模型 从 Java 虚拟机角度而言，类加载器可以分为以下两类： + **启动类加载器**：启动类加载器（Bootstrap ClassLoader）由 C++ 语言实现（以 HotSpot 为例），它是虚拟机自身的一部分； + **其他所有类的类加载器**：由 Java 语言实现，独立存在于虚拟机外部，并且全部继承自 `java.lang.ClassLoader` 。 从开发人员角度而言，类加载器可以分为以下三类： + **启动类加载器 (Boostrap Class Loader)**：负责把存放在 `\lib` 目录中，或被 `-Xbootclasspath` 参数所指定的路径中存放的能被 Java 虚拟机识别的类库加载到虚拟机的内存中； + **扩展类加载器 (Extension Class Loader)**：负责加载 `\lib\ext` 目录中，或被 `java.ext.dirs` 系统变量所指定的路径中的所有类库。 + **应用程序类加载器 (Application Class Loader)**：负责加载用户类路径（ClassPath）上的所有的类库。 JDK 9 之前的 Java 应用都是由这三种类加载器相互配合来完成加载：
上图所示的各种类加载器之间的层次关系被称为类加载器的 “双亲委派模型”，“双亲委派模型” 要求除了顶层的启动类加载器外，其余的类加载器都应该有自己的父类加载器，需要注意的是这里的加载器之间的父子关系一般不是以继承关系来实现的，而是使用组合关系来复用父类加载器的代码。 双亲委派模型的工作过程如下：如果一个类加载器收到了类加载的请求，它首先不会自己去尝试加载这个类，而是把这个请求委派给父类加载器去完成，每一个层次的类加载器都是如此，因此所有的加载请求最终都应该传送到最顶层的启动类加载器，只有当父加载器反馈自己无法完成这个加载请求（它的搜索范围中没有找到所需的类）时，子加载器才会尝试自己去完成加载。基于双亲委派模型可以保证程序中的类在各种类加载器环境中都是同一个类，否则就有可能出现一个程序中存在两个不同的 `java.lang.Object` 的情况。 ### 6.5 模块化下的类加载器 JDK 9 之后为了适应模块化的发展，类加载器做了如下变化： + 仍维持三层类加载器和双亲委派的架构，但扩展类加载器被平台类加载器所取代； + 当平台及应用程序类加载器收到类加载请求时，要首先判断该类是否能够归属到某一个系统模块中，如果可以找到这样的归属关系，就要优先委派给负责那个模块的加载器完成加载； + 启动类加载器、平台类加载器、应用程序类加载器全部继承自 `java.internal.loader.BuiltinClassLoader` ，BuiltinClassLoader 中实现了新的模块化架构下类如何从模块中加载的逻辑，以及模块中资源可访问性的处理。
## 七、程序编译 ### 7.1 编译器分类 + **前端编译器**：把 `*.java` 文件转变成 `.class` 文件的过程；如 JDK 的 Javac，Eclipse JDT 中的增量式编译器。 + **即使编译器**：常称为 JIT 编译器（Just In Time Complier），在运行期把字节码转变成本地机器码的过程；如 HotSpot 虚拟机中的 C1、C2 编译器，Graal 编译器。 + **提前编译器**：直接把程序编译成目标机器指令集相关的二进制代码的过程。如 JDK 的 jaotc，GUN Compiler for the Java（GCJ），Excelsior JET 。 ### 7.2 解释器与编译器 在 HotSpot 虚拟机中，Java 程序最初都是通过解释器（Interpreter）进行解释执行的，其优点在于可以省去编译时间，让程序快速启动。当程序启动后，如果虚拟机发现某个方法或代码块的运行特别频繁，就会使用编译器将其编译为本地机器码，并使用各种手段进行优化，从而提高执行效率，这就是即时编译器。HotSpot 内置了两个（或三个）即时编译器： + **客户端编译器 (Client Complier)**：简称 C1； + **服务端编译器 (Servier Complier)**：简称 C2，在有的资料和 JDK 源码中也称为 Opto 编译器； + **Graal 编译器**：在 JDK 10 时才出现，长期目标是替代 C2。 在分层编译的工作模式出现前，采用客户端编译器还是服务端编译器完全取决于虚拟机是运行在客户端模式还是服务端模式下，可以在启动时通过 `-client` 或 `-server` 参数进行指定，也可以让虚拟机根据自身版本和宿主机性能来自主选择。 ### 7.3 分层编译 要编译出优化程度越高的代码通常都需要越长的编译时间，为了在程序启动速度与运行效率之间达到最佳平衡，HotSpot 虚拟机在编译子系统中加入了分层编译（Tiered Compilation）： + **第 0 层**：程序纯解释执行，并且解释器不开启性能监控功能； + **第 1 层**：使用客户端编译器将字节码编译为本地代码来运行，进行简单可靠的稳定优化，不开启性能监控功能； + **第 2 层**：仍然使用客户端编译执行，仅开启方法及回边次数统计等有限的性能监控； + **第 3 层**：仍然使用客户端编译执行，开启全部性能监控； + **第 4 层**：使用服务端编译器将字节码编译为本地代码，其耗时更长，并且会根据性能监控信息进行一些不可靠的激进优化。 以上层次并不是固定不变的，根据不同的运行参数和版本，虚拟机可以调整分层的数量。各层次编译之间的交互转换关系如下图所示：
实施分层编译后，解释器、客户端编译器和服务端编译器就会同时工作，可以用客户端编译器获取更高的编译速度、用服务端编译器来获取更好的编译质量。 ### 7.4 热点探测 即时编译器编译的目标是 “热点代码”，它主要分为以下两类： + 被多次调用的方法。 + 被多次执行循环体。这里指的是一个方法只被少量调用过，但方法体内部存在循环次数较多的循环体，此时也认为是热点代码。但编译器编译的仍然是循环体所在的方法，而不会单独编译循环体。 判断某段代码是否是热点代码的行为称为 “热点探测” （Hot Spot Code Detection），主流的热点探测方法有以下两种： + **基于采样的热点探测 (Sample Based Hot Spot Code Detection)**：采用这种方法的虚拟机会周期性地检查各个线程的调用栈顶，如果发现某个（或某些）方法经常出现在栈顶，那么就认为它是 “热点方法”。 + **基于计数的热点探测 (Counter Based Hot Spot Code Detection)**：采用这种方法的虚拟机会为每个方法（甚至是代码块）建立计数器，统计方法的执行次数，如果执行次数超过一定的阈值就认为它是 “热点方法”。 ## 八、代码优化 即时编译器除了将字节码编译为本地机器码外，还会对代码进行一定程度的优化，它包含多达几十种优化技术，这里选取其中代表性的四种进行介绍： ### 8.1 方法内联 最重要的优化手段，它会将目标方法中的代码原封不动地 “复制” 到发起调用的方法之中，避免发生真实的方法调用，并采用名为类型继承关系分析（Class Hierarchy Analysis，CHA）的技术来解决虚方法（Java 语言中默认的实例方法都是虚方法）的内联问题。 ### 8.2 逃逸分析 逃逸行为主要分为以下两类： + **方法逃逸**：当一个对象在方法里面被定义后，它可能被外部方法所引用，例如作为调用参数传递到其他方法中，此时称为方法逃逸； + **线程逃逸**：当一个对象在方法里面被定义后，它可能被外部线程所访问，例如赋值给可以在其他线程中访问的实例变量，此时称为线程，其逃逸程度高于方法逃逸。 ```java public static StringBuilder concat(String... strings) { StringBuilder sb = new StringBuilder(); for (String string : strings) { sb.append(string); } return sb; // 发生了方法逃逸 } public static String concat(String... strings) { StringBuilder sb = new StringBuilder(); for (String string : strings) { sb.append(string); } return sb.toString(); // 没有发生方法逃逸 } ``` 如果能证明一个对象不会逃逸到方法或线程之外，或者逃逸程度比较低（只逃逸出方法而不会逃逸出线程），则可以为这个对象实例采取不同程序的优化： + **栈上分配 (Stack Allocations)**：如果一个对象不会逃逸到线程外，那么将会在栈上分配内存来创建这个对象，而不是 Java 堆上，此时对象所占用的内存空间就会随着栈帧的出栈而销毁，从而可以减轻垃圾回收的压力。 + **标量替换 (Scalar Replacement)**：如果一个数据已经无法再分解成为更小的数据类型，那么这些数据就称为标量（如 int、long 等数值类型及 reference 类型等）；反之，如果一个数据可以继续分解，那它就被称为聚合量（如对象）。如果一个对象不会逃逸外方法外，那么就可以将其改为直接创建若干个被这个方法使用的成员变量来替代，从而减少内存占用。 + **同步消除 (Synchronization Elimination)**：如果一个变量不会逃逸出线程，那么对这个变量实施的同步措施就可以消除掉。 ### 8.3 公共子表达式消除 如果一个表达式 E 之前已经被计算过了，并且从先前的计算到现在 E 中所有变量的值都没有发生过变化，那么 E 这次的出现就称为公共子表达式。对于这种表达式，无需再重新进行计算，只需要直接使用前面的计算结果即可。 ### 8.4 数组边界检查消除 对于虚拟机执行子系统来说，每次数组元素的读写都带有一次隐含的上下文检查以避免访问越界。如果数组的访问发生在循环之中，并且使用循环变量来访问数据，即循环变量的取值永远在 [0，list.length) 之间，那么此时就可以消除整个循环的数据边界检查，从而避免多次无用的判断。 ## 参考资料 + 主要参考自：周志明 . 深入理解Java虚拟机（第3版）. 机械工业出版社 , 2019-12 ，想要深入了解虚拟机的话，推荐阅读原书。 + [美团技术团队 —— Java Hotspot G1 GC的一些关键技术](https://tech.meituan.com/2016/09/23/g1.html) ================================================ FILE: docs/java/load-class.md ================================================ # 谈谈 Java 类加载机制 最近在学习 Tomcat 架构，其中很重要的一个模块是类加载器，因为以前学习的不够深入，所以趁这个机会好好把类加载机制搞明白。 - [Overview](#overview) - [API for class loading](#api-for-class-loading) - [`java.lang.ClassLoader`](#javalangclassloader) - [`java.security.SecureClassLoader`](#javasecuritysecureclassloader) - [`java.net.URLClassLoader`](#javaneturlclassloader) - [Tomcat 8.5.15 class loading Mechanism](#tomcat-8515-class-loading-mechanism) - [Reference](#reference) --- ## Overview ![](./../img/Class-loader.png) 类加载器主要分为两类，一类是 JDK 默认提供的，一类是用户自定义的。 JDK 默认提供三种类加载器 1. `Bootstrap ClassLoader`，启动类加载器，每次执行 `java` 命令时都会使用该加载器为虚拟机加载核心类。该加载器是由 `native code` 实现，而不是 Java 代码，加载类的路径为 `/jre/lib`。特别的 `/jre/lib/rt.jar` 中包含了 `sun.misc.Launcher` 类， 而 `sun.misc.Launcher$ExtClassLoader` 和 `sun.misc.Launcher$AppClassLoader` 都是 `sun.misc.Launcher` 的内部类，所以拓展类加载器和系统类加载器都是由启动类加载器加载的。 1. `Extension ClassLoader`, 拓展类加载器，用于加载拓展库中的类。拓展库路径为 `/jre/lib/ext/`。实现类为 `sun.misc.Launcher$ExtClassLoader` 1. `System ClassLoader`, 系统类加载器。用于加载 CLASSPATH 中的类。实现类为 `sun.misc.Launcher$AppClassLoader` 用户自定义的类加载器 1. `Custom ClassLoader`, 一般都是 `java.lang.ClassLoder` 的子类 正统的类加载机制是基于双亲委派的，也就是当调用类加载器加载类时，首先将加载任务委派给双亲，若双亲无法加载成功时，自己才进行类加载。 在实例化一个新的类加载器时，我们可以为其指定一个 `parent`，即双亲，若未显式指定，则 `System ClassLoader` 就作为默认双亲。 具体的说，类加载任务是由 `ClassLoader` 的 `loadClass()` 方法来执行的，他会按照以下顺序加载类： 1. 通过 `findLoadedClass()` 看该类是否已经被加载。该方法为 native code 实现，若已加载则返回。 1. 若未加载则委派给双亲，`parent.loadClass()`，若成功则返回 1. 若未成功，则调用 `findClass()` 方法加载类。`java.lang.ClassLoader` 中该方法只是简单的抛出一个 `ClassNotFoundException` 所以，自定义的 ClassLoader 都需要 Override `findClass()` 方法 ## API for class loading ### `java.lang.ClassLoader` - `ClassLoader` 是一个抽象类。 - 待加载的类必须用 `The Java™ Language Specification` 定义的全类名，全类名的定义请查阅 [The Form of a Binary](https://docs.oracle.com/javase/specs/jls/se8/html/jls-13.html#jls-13.1)。 - 给定一个全类名，类加载器应该去定位该类所在的位置。通用的策略是将全类名转换为类文件路径，然后通过类文件路径在文件系统中定位。 - 每一个加载到内存的类都由一个 Class 对象来表示，每一个 Class 对象都有一个指向加载该类的类加载器的引用。但是数组的 Class 对象是由 Java 运行时环境创建的，通过 `Class.getClassLoader()` 方法返回的是数组元素的类加载器，若数组元素是基本类型，则返回 `null`，若类是由 `Bootstrap ClassLoader` 加载的话也是返回 `null` ```java public class Main { public static void main(String[] args) { // Object 类在 /jre/lib/rt.jar 中， // 由 Bootstrap ClassLoader 加载，由于该类加载器是由 native code 编写 // 所以输出为 null Object[] objects = new Object[5]; System.out.println(); System.out.println(objects.getClass().getClassLoader()); // ZipFileAttributes 类在 /jre/lib/ext/zipfs.jar 中， // 由 Extension ClassLoader 加载， // 输出为 sun.misc.Launcher$ExtClassLoader@4b67cf4d ZipFileAttributes[] attributes = new ZipFileAttributes[5]; System.out.println(); System.out.println(attributes.getClass().getClassLoader()); // Main 类是自定义的类， // 默认由 System ClassLoader 加载， // 输出为 sun.misc.Launcher$AppClassLoader@18b4aac2 Main[] array = new Main[5]; array[0] = new Main(); System.out.println(); System.out.println(array.getClass().getClassLoader()); } } ``` - `ClassLoader` 默认支持并行加载，但是其子类必须调用 `ClassLoader.registerAsParallelCapable()` 来启用并行加载 - 一般来说,JVM 从本地文件系统加载类的行为是与平台有关的。 - `defineClass()` 方法可以将字节流转换成一个 `Class` 对象。然后调用 `Class.newInstance()` 来创建类的实例 ### `java.security.SecureClassLoader` 增加了一层权限验证，因为关注点不在安全，所以暂不讨论。 ### `java.net.URLClassLoader` 该类加载器用来加载 URL 指定的 JAR 文件或目录中的类和资源，以 `/` 结尾的 URL 认为是目录，否则认为是 JAR 文件。 ```java // 尝试通过 URLClassLoader 来加载桌面下的 Test 类。 public class Main { public static void main(String[] args) { try { URL[] urls = new URL[1]; URLStreamHandler streamHandler = null; File classPath = new File("/home/chen/Desktop/"); String repository = (new URL("file", null, classPath.getCanonicalPath() + File.separator)) .toString(); urls[0] = new URL(null, repository, streamHandler); ClassLoader loader = new URLClassLoader(urls); Class testClass = loader.loadClass("Test"); // output: java.net.URLClassLoader@7f31245a System.out.println(testClass.getClassLoader()); } catch (MalformedURLException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } catch (ClassNotFoundException e) { e.printStackTrace(); } } } ``` ## Tomcat 8.5.15 class loading Mechanism ![](./../img/tomcat-classloader.png) Tomcat 使用正统的类加载机制(双亲委派),但部分地方做了改动。 - `Bootstrap classLoader` 和 `Extension classLoader` 的作用不变 - `System classLoader` 正常情况下加载的是 `CLASSPATH` 下的类，但是 Tomcat 的启动脚本并未使用该变量，而是从以下仓库下加载类： 1. `$CATALINA_HOME/bin/bootstrap.jar` 包含了 Tomcat 的启动类。在该启动类中创建了 `Common classLoader`、`Catalina classLoader`、`shared classLoader`。因为 `$CATALINA_BASE/conf/catalina.properties` 中只对 `common.loader` 属性做了定义，`server.loader` 和 `shared.loader` 属性为空，所以默认情况下，这三个 classLoader 都是 `CommonLoader`。具体的代码逻辑可以查阅 `org.apache.catalina.startup.Bootstrap` 类的 `initClassLoaders()` 方法和 `createClassLoader()` 方法。 1. `$CATALINA_BASE/bin/tomcat-juli.jar` 包含了 Tomcat 日志模块所需要的实现类 1. `$CATALINA_HOME/bin/commons-daemon.jar` - `Common classLoader` 是位于 Tomcat 应用服务器顶层的公用类加载器。由其加载的类可以由 Tomcat 自身类和所有应用程序使用。扫描路径由 `$CATALINA_BASE/conf/catalina.properties` 文件中的 `common.loader` 属性定义。默认是 `$CATALINA_HOME/lib`. - `catalina classLoader` 用于加载服务器内部可见类，这些类应用程序不能访问。 - `shared classLoader` 用于加载应用程序共享类，这些类服务器不会依赖。 - `Webapp classLoader` 。每个应用程序都会有一个独一无二的 `webapp classloader`，他用来加载本应用程序 `/WEB-INF/classes` 和 `/WEB-INF/lib` 下的类。 特别的： `Webapp classLoader` 的默认行为会与正常的双亲委派模式不同： 1. 从 `Bootstrap classloader` 加载 1. 若没有，从 `/WEB-INF/classes` 加载 1. 若没有，从 `/WEB-INF/lib/*.jar` 加载 1. 若没有，则依次从 `System`、`Common`、`shared` 加载（该步骤使用双亲委派） 当然了，我们也可以通过配置来使 `Webapp classLoader` 严格按照双亲委派模式加载类： 1. 通过在工程的 `META-INF/context.xml`（和 `WEB-INF/classes` 在同一目录下） 配置文件中添加 `` 1. 因为 `Webapp classLoader` 的实现类是 `org.apache.catalina.loader.WebappLoader`，他有一个属性叫 `delegate`， 用来控制类加载器的加载行为，默认为 `false`，我们可以使用 `set` 方法，将其设为 `true` 来启用严格双亲委派加载模式。 严格双亲委派模式加载步骤： 1. 从 `Bootstrap classloader` 加载 1. 若没有，则依次从 `System`、`Common`、`shared` 加载 1. 若没有，从 `/WEB-INF/classes` 加载 1. 若没有，从 `/WEB-INF/lib/*.jar` 加载 ## Reference 1. [The Java Class Loading Mechanism](https://docs.oracle.com/javase/tutorial/ext/basics/load.html) 1. [Java Classloader](https://en.wikipedia.org/wiki/Java_Classloader) 1. [Class Loader HOW-TO - Apache Tomcat 8](https://tomcat.apache.org/tomcat-8.5-doc/class-loader-howto.html) 1. [《Tomcat 架构解析》]() 1. [《深入理解 Java 虚拟机》]() ================================================ FILE: docs/java/orm.md ================================================ # 浅析 Mybatis 与 Hibernate 的区别与用途 有很长一段时间对mybatis是比较陌生的，只知道与Hibernate一样是个orm数据库框架。随着使用熟练度的增加，发现它与Hibernate区别是非常大的，应当结合不同的情况分析选用。结合至今为止的经验，总结出以下几点： 1. hibernate是全自动，而mybatis是半自动 hibernate完全可以通过对象关系模型实现对数据库的操作，拥有完整的JavaBean对象与数据库的映射结构来自动生成sql。而mybatis仅有基本的字段映射，对象数据以及对象实际关系仍然需要通过手写sql来实现和管理。 2. hibernate数据库移植性远大于mybatis hibernate通过它强大的映射结构和hql语言，大大降低了对象与数据库（oracle、mysql等）的耦合性，而mybatis由于需要手写sql，因此与数据库的耦合性直接取决于程序员写sql的方法，如果sql不具通用性而用了很多某数据库特性的sql语句的话，移植性也会随之降低很多，成本很高。 3. hibernate拥有完整的日志系统，mybatis则欠缺一些 hibernate日志系统非常健全，涉及广泛，包括：sql记录、关系异常、优化警告、缓存提示、脏数据警告等；而mybatis则除了基本记录功能外，功能薄弱很多。 4. mybatis相比hibernate需要关心很多细节 hibernate配置要比mybatis复杂的多，学习成本也比mybatis高。但也正因为mybatis使用简单，才导致它要比hibernate关心很多技术细节。mybatis由于不用考虑很多细节，开发模式上与传统jdbc区别很小，因此很容易上手并开发项目，但忽略细节会导致项目前期bug较多，因而开发出相对稳定的软件很慢，而开发出软件却很快。hibernate则正好与之相反。但是如果使用hibernate很熟练的话，实际上开发效率丝毫不差于甚至超越mybatis。 5. sql直接优化上，mybatis要比hibernate方便很多 由于mybatis的sql都是写在xml里，因此优化sql比hibernate方便很多。而hibernate的sql很多都是自动生成的，无法直接维护sql；虽有hql，但功能还是不及sql强大，见到报表等变态需求时，hql也歇菜，也就是说hql是有局限的；hibernate虽然也支持原生sql，但开发模式上却与orm不同，需要转换思维，因此使用上不是非常方便。总之写sql的灵活度上hibernate不及mybatis。 ## 随着使用情况的不断增多，我又做了进一步的总结总结： mybatis：小巧、方便、高效、简单、直接、半自动 hibernate：强大、方便、高效、复杂、绕弯子、全自动 mybatis： 1. 入门简单，即学即用，提供了数据库查询的自动对象绑定功能，而且延续了很好的SQL使用经验，对于没有那么高的对象模型要求的项目来说，相当完美。 2. 可以进行更为细致的SQL优化，可以减少查询字段。 3. 缺点就是框架还是比较简陋，功能尚有缺失，虽然简化了数据绑定代码，但是整个底层数据库查询实际还是要自己写的，工作量也比较大，而且不太容易适应快速数据库修改。 4. 二级缓存机制不佳。 hibernate： 1. 功能强大，数据库无关性好，O/R映射能力强，如果你对Hibernate相当精通，而且对Hibernate进行了适当的封装，那么你的项目整个持久层代码会相当简单，需要写的代码很少，开发速度很快，非常爽。 2. 有更好的二级缓存机制，可以使用第三方缓存。 3. 缺点就是学习门槛不低，要精通门槛更高，而且怎么设计O/R映射，在性能和对象模型之间如何权衡取得平衡，以及怎样用好Hibernate方面需要你的经验和能力都很强才行。 举个形象的比喻： mybatis：机械工具，使用方便，拿来就用，但工作还是要自己来作，不过工具是活的，怎么使由我决定。 hibernate：智能机器人，但研发它（学习、熟练度）的成本很高，工作都可以摆脱他了，但仅限于它能做的事。 ================================================ FILE: docs/java/rocketmq/rmq-1.md ================================================ # RocketMq下载与安装 本文讲解下载并安装单机版RocketMq，linux环境。 ### 正文介绍： 目前 RocketMQ 已经成为Apache 顶级项目 。 在阿里内部， RocketMQ 很好地服务了 集 团大大小小上千个应 用，在每年的双十一当天，更有不可思议的万亿级消息通过 RocketMQ 流转(在 2017 年的双十一当天，整个阿里巴巴集团通过 RocketMQ 流转的线上消息达到了 万亿级，峰值 TPS 达到 5600 万)，在阿里大中台策略上发挥着举足轻重的作用 。 此外， RocketMQ 是使用 Java语言开发的，比起 Kafka 的 Scala语言和 RabbitMQ 的 Erlang 语 言，更容易找 到技术人员进行定制开发 。 #### 1.RocketMQ 由四部分组成 * 发信者 ------------> Producer * 收信者 -------------> Consumer * 负责暂存 --------------> Broker * 传输的邮局 -------------> NameServer 启动 RocketMQ 的顺序是先启动 NameServer，再启动 Broker，这时候消 息队列已 经可以提供服务了，想发送消息就使用 Producer来发送，想接收消息 就使用 Consumer来接收 。 很多应用程序既要发送，又要接收，可以启动多个Producer 和 Consumer 来发送多种消息，同时接收多种消息 。 #### 2.那么RocketMq有什么用？ 1. 应用解藕 2. 流量消峰：把一秒内下的订单分散成一段时间来处理，这时有些用户可 能在下单后十几秒才能收到下单成功的状态，但是也比不能下单的体验要好。 3. 消息分发：数据的产生方只 需要把各自的数据写人一个消息队列即可 数据使用方根据各自需求订阅感兴 趣的数据，不同数据团队所订阅的数据可以重复也可以不重复，互不干扰，也 不必和数据产生方关联 除了上面列出的应用解棉、流量消峰、消息分发等功能外，消息队列还有保证最终一致性、方便动态扩容等功能。 #### 3.安装使用 ``` shell -- 下载 wget -c http://mirror.bit.edu.cn/apache/rocketmq/4.4.0/rocketmq-all-4.4.0-bin-release.zip -- 解压 unzip rocketmq-all-4.4.0-bin-release.zip -- Start Name Server nohup sh bin/mqnamesrv -- Start Broker nohup sh bin/mqbroker -n localhost:9876 -- 发送 rocketmq-all-4.3.0/distribution/target/apache-rocketmq 当前目录下 执行 export NAMESRV ADDR=localhost:9876 执行 sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer 效果SendResult [sendStatus=SEND OK, msgid= 省略很多 -- 接收 sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer 效果 ConsumeMessageThread 主d Receive New Messages : [MessageExt 省略 -- 闭消息队列 sh bin/mqshutdown broker sh bin/mqshutdown namesrv ``` ================================================ FILE: docs/java/spring-cloud.md ================================================ # 什么是Spring Cloud >Spring Cloud为开发人员提供了工具，以快速构建分布式系统中的某些常见模式（例如，配置管理，服务发现，断路器，智能路由，微代理，控制总线，一次性令牌，全局锁，领导选举，分布式会话，群集状态）。分布式系统的协调导致样板式样，并且使用Spring Cloud开发人员可以快速站起来实现这些样板的服务和应用程序。它们可以在任何分布式环境中正常工作，包括开发人员自己的笔记本电脑，裸机数据中心以及Cloud Foundry等托管平台。 >Spring Cloud致力于为典型的用例和扩展机制提供良好的开箱即用体验，以涵盖其他用例。有如下特点 * 分布式/版本化配置 * 服务注册和发现 * 路由 * 服务到服务的通话 * 负载均衡 * 断路器 * 全局锁 * 领导选举和集群状态 * 分布式消息传递 官方果然官方，介绍都这么有板有眼的。 我所理解的 Spring Cloud 就是微服务系统架构的一站式解决方案，在平时我们构建微服务的过程中需要做如 服务发现注册 、配置中心 、消息总线 、负载均衡 、断路器 、数据监控 等操作，而 Spring Cloud 为我们提供了一套简易的编程模型，使我们能在 Spring Boot 的基础上轻松地实现微服务项目的构建。 ## Spring Cloud 的版本 Spring Cloud 的版本号并不是我们通常见的数字版本号，而是一些很奇怪的单词。这些单词均为英国伦敦地铁站的站名。同时根据字母表的顺序来对应版本时间顺序，比如：最早 的 Release 版本 Angel，第二个 Release 版本 Brixton（英国地名），然后是 Camden、 Dalston、Edgware、Finchley、Greenwich、Hoxton。 ## Spring Cloud 的服务发现框架——Eureka >Eureka是基于REST（代表性状态转移）的服务，主要在AWS云中用于定位服务，以实现负载均衡和中间层服务器的故障转移。我们称此服务为Eureka服务器。Eureka还带有一个基于Java的客户端组件Eureka Client，它使与服务的交互变得更加容易。客户端还具有一个内置的负载平衡器，可以执行基本的循环负载平衡。在Netflix，更复杂的负载均衡器将Eureka包装起来，以基于流量，资源使用，错误条件等多种因素提供加权负载均衡，以提供出色的弹性。 总的来说，Eureka 就是一个服务发现框架。何为服务，何又为发现呢？ 举一个生活中的例子，就比如我们平时租房子找中介的事情。 在没有中介的时候我们需要一个一个去寻找是否有房屋要出租的房东，这显然会非常的费力，一你找凭一个人的能力是找不到很多房源供你选择，再者你也懒得这么找下去(找了这么久，没有合适的只能将就)。**这里的我们就相当于微服务中的 Consumer ，而那些房东就相当于微服务中的 Provider 。消费者 Consumer 需要调用提供者 Provider 提供的一些服务，就像我们现在需要租他们的房子一样。** 但是如果只是租客和房东之间进行寻找的话，他们的效率是很低的，房东找不到租客赚不到钱，租客找不到房东住不了房。所以，后来房东肯定就想到了广播自己的房源信息(比如在街边贴贴小广告)，这样对于房东来说已经完成他的任务(将房源公布出去)，但是有两个问题就出现了。第一、其他不是租客的都能收到这种租房消息，这在现实世界没什么，但是在计算机的世界中就会出现资源消耗的问题了。第二、租客这样还是很难找到你，试想一下我需要租房，我还需要东一个西一个地去找街边小广告，麻不麻烦？ 那怎么办呢？我们当然不会那么傻乎乎的，第一时间就是去找 中介 呀，它为我们提供了统一房源的地方，我们消费者只需要跑到它那里去找就行了。而对于房东来说，他们也只需要把房源在中介那里发布就行了。 那么现在，我们的模式就是这样的了。 ![spring-cloud](../img/java/spring-cloud-1.png) 但是，这个时候还会出现一些问题。 1.房东注册之后如果不想卖房子了怎么办？我们是不是需要让房东定期续约？如果房东不进行续约是不是要将他们从中介那里的注册列表中移除。 2.租客是不是也要进行注册呢？不然合同乙方怎么来呢？ 3.中介可不可以做连锁店呢？如果这一个店因为某些不可抗力因素而无法使用，那么我们是否可以换一个连锁店呢？ 针对上面的问题我们来重新构建一下上面的模式图 ![spring-cloud](../img/java/spring-cloud-2.png) 好了，举完这个例子我们就可以来看关于 Eureka 的一些基础概念了，你会发现这东西理解起来怎么这么简单。 服务发现：其实就是一个“中介”，整个过程中有三个角色：服务提供者(出租房子的)、服务消费者(租客)、服务中介(房屋中介)。 服务提供者：就是提供一些自己能够执行的一些服务给外界。 服务消费者：就是需要使用一些服务的“用户”。 服务中介：其实就是服务提供者和服务消费者之间的“桥梁”，服务提供者可以把自己注册到服务中介那里，而服务消费者如需要消费一些服务(使用一些功能)就可以在服务中介中寻找注册在服务中介的服务提供者。 ## Eureka 工作原理 >Eureka 作为 Spring Cloud 体系中最核心、默认的注册中心组件，研究它的运行机制，有助于我们在工作中更好地使用它。 ### Eureka 核心概念 回到上节的服务注册调用示意图，服务提供者和服务的消费者，本质上也是 Eureka Client 角色。整体上可以分为两个主体：Eureka Server 和 Eureka Client。 ![spring-cloud](../img/java/spring-cloud-3.png) #### Eureka Server：注册中心服务端 注册中心服务端主要对外提供了三个功能： * 服务注册 服务提供者启动时，会通过 Eureka Client 向 Eureka Server 注册信息，Eureka Server 会存储该服务的信息，Eureka Server 内部有二层缓存机制来维护整个注册表 * 提供注册表 服务消费者在调用服务时，如果 Eureka Client 没有缓存注册表的话，会从 Eureka Server 获取最新的注册表 * 同步状态 Eureka Client 通过注册、心跳机制和 Eureka Server 同步当前客户端的状态。 #### Eureka Client：注册中心客户端 > Eureka Client 是一个 Java 客户端，用于简化与 Eureka Server 的交互。Eureka Client 会拉取、更新和缓存 Eureka Server 中的信息。因此当所有的 Eureka Server 节点都宕掉，服务消费者依然可以使用缓存中的信息找到服务提供者，但是当服务有更改的时候会出现信息不一致。 * Register: 服务注册 服务的提供者，将自身注册到注册中心，服务提供者也是一个 Eureka Client。当 Eureka Client 向 Eureka Server 注册时，它提供自身的元数据，比如 IP 地址、端口，运行状况指示符 URL，主页等。 * Renew: 服务续约 Eureka Client 会每隔 30 秒发送一次心跳来续约。 通过续约来告知 Eureka Server 该 Eureka Client 运行正常，没有出现问题。 默认情况下，如果 Eureka Server 在 90 秒内没有收到 Eureka Client 的续约，Server 端会将实例从其注册表中删除，此时间可配置，一般情况不建议更改 * 服务续约的两个重要属性 ``` 服务续约任务的调用间隔时间，默认为30秒 eureka.instance.lease-renewal-interval-in-seconds=30 服务失效的时间，默认为90秒。 eureka.instance.lease-expiration-duration-in-seconds=90 ``` * Eviction 服务剔除 当 Eureka Client 和 Eureka Server 不再有心跳时，Eureka Server 会将该服务实例从服务注册列表中删除，即服务剔除。 * Cancel: 服务下线 Eureka Client 在程序关闭时向 Eureka Server 发送取消请求。 发送请求后，该客户端实例信息将从 Eureka Server 的实例注册表中删除。该下线请求不会自动完成，它需要调用以下内容： ``` DiscoveryManager.getInstance().shutdownComponent()； ``` * GetRegisty: 获取注册列表信息 Eureka Client 从服务器获取注册表信息，并将其缓存在本地。客户端会使用该信息查找其他服务，从而进行远程调用。该注册列表信息定期（每30秒钟）更新一次。每次返回注册列表信息可能与 Eureka Client 的缓存信息不同，Eureka Client 自动处理。 如果由于某种原因导致注册列表信息不能及时匹配，Eureka Client 则会重新获取整个注册表信息。 Eureka Server 缓存注册列表信息，整个注册表以及每个应用程序的信息进行了压缩，压缩内容和没有压缩的内容完全相同。Eureka Client 和 Eureka Server 可以使用 JSON/XML 格式进行通讯。在默认情况下 Eureka Client 使用压缩 JSON 格式来获取注册列表的信息。 * 获取服务是服务消费者的基础，所以必有两个重要参数需要注意： ``` # 启用服务消费者从注册中心拉取服务列表的功能 eureka.client.fetch-registry=true # 设置服务消费者从注册中心拉取服务列表的间隔 eureka.client.registry-fetch-interval-seconds=30 ``` * Remote Call: 远程调用 当 Eureka Client 从注册中心获取到服务提供者信息后，就可以通过 Http 请求调用对应的服务；服务提供者有多个时，Eureka Client 客户端会通过 Ribbon 自动进行负载均衡。 #### 自我保护机制 默认情况下，如果 Eureka Server 在一定的 90s 内没有接收到某个微服务实例的心跳，会注销该实例。但是在微服务架构下服务之间通常都是跨进程调用，网络通信往往会面临着各种问题，比如微服务状态正常，网络分区故障，导致此实例被注销。 固定时间内大量实例被注销，可能会严重威胁整个微服务架构的可用性。为了解决这个问题，Eureka 开发了自我保护机制，那么什么是自我保护机制呢？ Eureka Server 在运行期间会去统计心跳失败比例在 15 分钟之内是否低于 85%，如果低于 85%，Eureka Server 即会进入自我保护机制。 ##### Eureka Server 触发自我保护机制后，页面会出现提示： ![spring-cloud](../img/java/spring-cloud-4.png) #### Eureka Server 进入自我保护机制，会出现以下几种情况： (1 Eureka 不再从注册列表中移除因为长时间没收到心跳而应该过期的服务 (2 Eureka 仍然能够接受新服务的注册和查询请求，但是不会被同步到其它节点上(即保证当前节点依然可用) (3 当网络稳定时，当前实例新的注册信息会被同步到其它节点中 Eureka 自我保护机制是为了防止误杀服务而提供的一个机制。当个别客户端出现心跳失联时，则认为是客户端的问题，剔除掉客户端；当 Eureka 捕获到大量的心跳失败时，则认为可能是网络问题，进入自我保护机制；当客户端心跳恢复时，Eureka 会自动退出自我保护机制。 如果在保护期内刚好这个服务提供者非正常下线了，此时服务消费者就会拿到一个无效的服务实例，即会调用失败。对于这个问题需要服务消费者端要有一些容错机制，如重试，断路器等。 * 通过在 Eureka Server 配置如下参数，开启或者关闭保护机制，生产环境建议打开： ``` eureka.server.enable-self-preservation=true ``` #### Eureka 集群原理 再来看看 Eureka 集群的工作原理。我们假设有三台 Eureka Server 组成的集群，第一台 Eureka Server 在北京机房，另外两台 Eureka Server 在深圳和西安机房。这样三台 Eureka Server 就组建成了一个跨区域的高可用集群，只要三个地方的任意一个机房不出现问题，都不会影响整个架构的稳定性。 ![spring-cloud](../img/java/spring-cloud-5.png) 从图中可以看出 Eureka Server 集群相互之间通过 Replicate 来同步数据，相互之间不区分主节点和从节点，所有的节点都是平等的。在这种架构中，节点通过彼此互相注册来提高可用性，每个节点需要添加一个或多个有效的 serviceUrl 指向其他节点。 如果某台 Eureka Server 宕机，Eureka Client 的请求会自动切换到新的 Eureka Server 节点。当宕机的服务器重新恢复后，Eureka 会再次将其纳入到服务器集群管理之中。当节点开始接受客户端请求时，所有的操作都会进行节点间复制，将请求复制到其它 Eureka Server 当前所知的所有节点中。 另外 Eureka Server 的同步遵循着一个非常简单的原则：只要有一条边将节点连接，就可以进行信息传播与同步。所以，如果存在多个节点，只需要将节点之间两两连接起来形成通路，那么其它注册中心都可以共享信息。每个 Eureka Server 同时也是 Eureka Client，多个 Eureka Server 之间通过 P2P 的方式完成服务注册表的同步。 Eureka Server 集群之间的状态是采用异步方式同步的，所以不保证节点间的状态一定是一致的，不过基本能保证最终状态是一致的。 #### Eureka 分区 Eureka 提供了 Region 和 Zone 两个概念来进行分区，这两个概念均来自于亚马逊的 AWS: region：可以理解为地理上的不同区域，比如亚洲地区，中国区或者深圳等等。没有具体大小的限制。根据项目具体的情况，可以自行合理划分 region。 zone：可以简单理解为 region 内的具体机房，比如说 region 划分为深圳，然后深圳有两个机房，就可以在此 region 之下划分出 zone1、zone2 两个 zone。 上图中的 us-east-1c、us-east-1d、us-east-1e 就代表了不同的 Zone。Zone 内的 Eureka Client 优先和 Zone 内的 Eureka Server 进行心跳同步，同样调用端优先在 Zone 内的 Eureka Server 获取服务列表，当 Zone 内的 Eureka Server 挂掉之后，才会从别的 Zone 中获取信息。 #### Eurka 保证 AP Eureka Server 各个节点都是平等的，几个节点挂掉不会影响正常节点的工作，剩余的节点依然可以提供注册和查询服务。而 Eureka Client 在向某个 Eureka 注册时，如果发现连接失败，则会自动切换至其它节点。只要有一台 Eureka Server 还在，就能保证注册服务可用(保证可用性)，只不过查到的信息可能不是最新的(不保证强一致性)。 ### Eurka 工作流程 了解完 Eureka 核心概念，自我保护机制，以及集群内的工作原理后，我们来整体梳理一下 Eureka 的工作流程： 1、Eureka Server 启动成功，等待服务端注册。在启动过程中如果配置了集群，集群之间定时通过 Replicate 同步注册表，每个 Eureka Server 都存在独立完整的服务注册表信息 2、Eureka Client 启动时根据配置的 Eureka Server 地址去注册中心注册服务 3、Eureka Client 会每 30s 向 Eureka Server 发送一次心跳请求，证明客户端服务正常 4、当 Eureka Server 90s 内没有收到 Eureka Client 的心跳，注册中心则认为该节点失效，会注销该实例 5、单位时间内 Eureka Server 统计到有大量的 Eureka Client 没有上送心跳，则认为可能为网络异常，进入自我保护机制，不再剔除没有上送心跳的客户端 6、当 Eureka Client 心跳请求恢复正常之后，Eureka Server 自动退出自我保护模式 7、Eureka Client 定时全量或者增量从注册中心获取服务注册表，并且将获取到的信息缓存到本地 8、服务调用时，Eureka Client 会先从本地缓存找寻调取的服务。如果获取不到，先从注册中心刷新注册表，再同步到本地缓存 9、Eureka Client 获取到目标服务器信息，发起服务调用 10、Eureka Client 程序关闭时向 Eureka Server 发送取消请求，Eureka Server 将实例从注册表中删除 这就是Eurka基本工作流程 ### Eurka 总结 讲了 Eureka 核心概念、Eureka 自我保护机制和 Eureka 集群原理。通过分析 Eureka 工作原理，我可以明显地感觉到 Eureka 的设计之巧妙，通过一些列的机制，完美地解决了注册中心的稳定性和高可用性。 Eureka 为了保障注册中心的高可用性，容忍了数据的非强一致性，服务节点间的数据可能不一致， Client-Server 间的数据可能不一致。比较适合跨越多机房、对注册中心服务可用性要求较高的使用场景。 ## 负载均衡之 Ribbon 什么是 RestTemplate? 不是讲 Ribbon 么？怎么扯到了 RestTemplate 了？你先别急，听我慢慢道来。 我不听我不听我不听🙉🙉🙉。 我就说一句！RestTemplate是Spring提供的一个访问Http服务的客户端类，怎么说呢？就是微服务之间的调用是使用的 RestTemplate 。比如这个时候我们 消费者B 需要调用 提供者A 所提供的服务我们就需要这么写。如我下面的伪代码。 ```java @Autowired private RestTemplate restTemplate; // 这里是提供者A的ip地址，但是如果使用了 Eureka 那么就应该是提供者A的名称 private static final String SERVICE_PROVIDER_A = "http://localhost:8081"; @PostMapping("/judge") public boolean judge(@RequestBody Request request) { String url = SERVICE_PROVIDER_A + "/service1"; return restTemplate.postForObject(url, request, Boolean.class); } ``` 如果你对源码感兴趣的话，你会发现上面我们所讲的 Eureka 框架中的 注册、续约 等，底层都是使用的 RestTemplate 。 为什么需要 Ribbon？ Ribbon 是 Netflix 公司的一个开源的负载均衡 项目，是一个客户端/进程内负载均衡器，运行在消费者端 我们再举个例子，比如我们设计了一个秒杀系统，但是为了整个系统的 高可用 ，我们需要将这个系统做一个集群，而这个时候我们消费者就可以拥有多个秒杀系统的调用途径了，如下图。 ![spring-cloud](../img/java/spring-cloud-6.png) 如果这个时候我们没有进行一些 均衡操作 ，如果我们对 秒杀系统1 进行大量的调用，而另外两个基本不请求，就会导致 秒杀系统1 崩溃，而另外两个就变成了傀儡，那么我们为什么还要做集群，我们高可用体现的意义又在哪呢？ 所以 Ribbon 出现了，注意我们上面加粗的几个字——运行在消费者端。指的是，Ribbon 是运行在消费者端的负载均衡器，如下图。 ![spring-cloud](../img/java/spring-cloud-7.png) 其工作原理就是 Consumer 端获取到了所有的服务列表之后，在其内部使用负载均衡算法，进行对多个系统的调用。 #### Nginx 和 Ribbon 的对比 提到 负载均衡 就不得不提到大名鼎鼎的 Nignx 了，而和 Ribbon 不同的是，它是一种集中式的负载均衡器。 何为集中式呢？简单理解就是 将所有请求都集中起来，然后再进行负载均衡。如下图。 ![spring-cloud](../img/java/spring-cloud-8.png) 我们可以看到 Nginx 是接收了所有的请求进行负载均衡的，而对于 Ribbon 来说它是在消费者端进行的负载均衡。如下图。 ![spring-cloud](../img/java/spring-cloud-9.png) >请注意 Request 的位置，在 Nginx 中请求是先进入负载均衡器，而在 Ribbon 中是先在客户端进行负载均衡才进行请求的。 #### Ribbon 的几种负载均衡算法 负载均衡，不管 Nginx 还是 Ribbon 都需要其算法的支持，如果我没记错的话 Nginx 使用的是 轮询和加权轮询算法。而在 Ribbon 中有更多的负载均衡调度算法，其默认是使用的 RoundRobinRule 轮询策略。 RoundRobinRule：轮询策略。Ribbon 默认采用的策略。若经过一轮轮询没有找到可用的 provider，其最多轮询 10 轮。若最终还没有找到，则返回 null。 RandomRule: 随机策略，从所有可用的 provider 中随机选择一个。 RetryRule: 重试策略。先按照 RoundRobinRule 策略获取 provider，若获取失败，则在指定的时限内重试。默认的时限为 500 毫秒。 🐦🐦🐦 还有很多，这里不一一举🌰了，你最需要知道的是默认轮询算法，并且可以更换默认的负载均衡算法，只需要在配置文件中做出修改就行。 ``` providerName: ribbon: NFLoadBalancerRuleClassName: com.netflix.loadbalancer.RandomRule ``` 当然，在 Ribbon 中你还可以自定义负载均衡算法，你只需要实现 IRule 接口，然后修改配置文件或者自定义 Java Config 类。 ### 什么是 Open Feign 有了 Eureka，RestTemplate，Ribbon 我们就可以😃愉快地进行服务间的调用了，但是使用 RestTemplate 还是不方便，我们每次都要进行这样的调用 ``` java @Autowired private RestTemplate restTemplate; // 这里是提供者A的Ip地址，但是如果使用了Eureka那么就应该是提供者A的名称 private static final String SERVICE_PROVIDER_A="http://localhost:8081"; @PostMapping("judge") public boolean judge(@RequestBody Request request){ String url=SERVICE_PROVIDER_A+"/service1"; //是不是太麻烦了？？？每次都要url、请求、返回类型的 return restTemplate.postForObject(url,request,Boolean.class) } ``` 这样每次都调用 RestRemplate 的 API 是否太麻烦，我能不能像调用原来代码一样进行各个服务间的调用呢？ 💡💡💡聪明的小朋友肯定想到了，那就用 映射 呀，就像域名和IP地址的映射。我们可以将被调用的服务代码映射到消费者端，这样我们就可以 “无缝开发”啦。 >OpenFeign 也是运行在消费者端的，使用 Ribbon 进行负载均衡，所以 OpenFeign 直接内置了 Ribbon。 在导入了 Open Feign 之后我们就可以进行愉快编写 Consumer 端代码了。 ``` java //使用@FeignClient 注解来指定提供者的名字 @FeignClient("eureka-client-provider") public interface TestClient{ // 这里一定要注意需要使用的是提供者那端的请求相对路径，这里就相对与映射了 @RequestMapping(value="provider/xxx",method=RequestMethod.POST) CommonResponse> getPlans(@RequestBody planGetRequest request); } ``` 然后我们在 Controller 就可以像原来调用 Service 层代码一样调用它了。 ```java @RestController public class TestController { //这里就相当于原来的service层代码 @Autowired private TestClient testClient; // controller 调用 service 层代码 @RequestMapping(value="/test",method=RequestMethod.POST) public CommonResponse> get (@RequestBody){ return testClient.getPlans(request); } } ``` ### 必不可少的 Hystrix >在分布式环境中，不可避免地会有许多服务依赖项中的某些失败。Hystrix是一个库，可通过添加等待时间容限和容错逻辑来帮助您控制这些分布式服务之间的交互。Hystrix通过隔离服务之间的访问点，停止服务之间的级联故障并提供后备选项来实现此目的，所有这些都可以提高系统的整体弹性。 总体来说 Hystrix 就是一个能进行 熔断 和 降级 的库，通过使用它能提高整个系统的弹性。 那么什么是 熔断和降级 呢？再举个🌰，此时我们整个微服务系统是这样的。服务A调用了服务B，服务B再调用了服务C，但是因为某些原因，服务C顶不住了，这个时候大量请求会在服务C阻塞。 ![spring-cloud-10](../img/java/spring-cloud-10.png) 服务C阻塞了还好，毕竟只是一个系统崩溃了。但是请注意这个时候因为服务C不能返回响应，那么服务B调用服务C的的请求就会阻塞，同理服务B阻塞了，那么服务A也会阻塞崩溃。 >请注意，为什么阻塞会崩溃。因为这些请求会消耗占用系统的线程、IO 等资源，消耗完你这个系统服务器不就崩了么。 ![spring-cloud-10](../img/java/spring-cloud-11.png) 这就叫 服务雪崩。妈耶，上面两个 熔断 和 降级 你都没给我解释清楚，你现在又给我扯什么 服务雪崩 ？😵😵😵 别急，听我慢慢道来。 不听我也得讲下去！ 所谓 熔断 就是服务雪崩的一种有效解决方案。当指定时间窗内的请求失败率达到设定阈值时，系统将通过 断路器 直接将此请求链路断开。 也就是我们上面服务B调用服务C在指定时间窗内，调用的失败率到达了一定的值，那么 Hystrix 则会自动将 服务B与C 之间的请求都断了，以免导致服务雪崩现象。 其实这里所讲的 熔断 就是指的 Hystrix 中的 断路器模式 ，你可以使用简单的 @HystrixCommand 注解来标注某个方法，这样 Hystrix 就会使用 断路器 来“包装”这个方法，每当调用时间超过指定时间时(默认为1000ms)，断路器将会中断对这个方法的调用。 当然你可以对这个注解的很多属性进行设置，比如设置超时时间，像这样。 ```java @HystrixCommand( commandProperties = {@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",value = "1200")} ) public List getXxxx() { // ...省略代码逻辑 } ``` 但是，我查阅了一些博客，发现他们都将 熔断 和 降级 的概念混淆了，以我的理解，降级是为了更好的用户体验，当一个方法调用异常时，通过执行另一种代码逻辑来给用户友好的回复。这也就对应着 Hystrix 的 后备处理 模式。你可以通过设置 fallbackMethod 来给一个方法设置备用的代码逻辑。比如这个时候有一个热点新闻出现了，我们会推荐给用户查看详情，然后用户会通过id去查询新闻的详情，但是因为这条新闻太火了(比如最近什么*易对吧)，大量用户同时访问可能会导致系统崩溃，那么我们就进行 服务降级 ，一些请求会做一些降级处理比如当前人数太多请稍后查看等等。 ``` java //指定后备方案调用 @HystrixCommand(fallbackMethod="getHystrixNews") @GetMapping("/get/news") public News getNews(@PathVariable("id") int id){ //调用新闻api 代码逻辑 } public News getHystrixNews(@PathVariable("id") int id){ //做服务降级 //返回自定信息 } ``` #### 什么是Hystrix之其他 我在阅读 《Spring微服务实战》这本书的时候还接触到了一个舱壁模式的概念。在不使用舱壁模式的情况下，服务A调用服务B，这种调用默认的是使用同一批线程来执行的，而在一个服务出现性能问题的时候，就会出现所有线程被刷爆并等待处理工作，同时阻塞新请求，最终导致程序崩溃。而舱壁模式会将远程资源调用隔离在他们自己的线程池中，以便可以控制单个表现不佳的服务，而不会使该程序崩溃。 具体其原理我推荐大家自己去了解一下，本篇文章中对舱壁模式不做过多解释。当然还有 Hystrix 仪表盘，它是用来实时监控 Hystrix 的各项指标信息的，这里我将这个问题也抛出去，希望有不了解的可以自己去搜索一下。 ### 微服务网关——Zuul > ZUUL 是从设备和 web 站点到 Netflix 流应用后端的所有请求的前门。作为边界服务应用，ZUUL 是为了实现动态路由、监视、弹性和安全性而构建的。它还具有根据情况将请求路由到多个 Amazon Auto Scaling Groups（亚马逊自动缩放组，亚马逊的一种云计算方式） 的能力 在上面我们学习了 Eureka 之后我们知道了 服务提供者 是 消费者 通过 Eureka Server 进行访问的，即 Eureka Server 是 服务提供者 的统一入口。那么整个应用中存在那么多 消费者 需要用户进行调用，这个时候用户该怎样访问这些 消费者工程 呢？当然可以像之前那样直接访问这些工程。但这种方式没有统一的消费者工程调用入口，不便于访问与管理，而 Zuul 就是这样的一个对于 消费者 的统一入口。 > 如果学过前端的肯定都知道 Router 吧，比如 Flutter 中的路由，Vue，React中的路由，用了 Zuul 你会发现在路由功能方面和前端配置路由基本是一个理。😁 我偶尔撸撸 Flutter。 大家对网关应该很熟吧，简单来讲网关是系统唯一对外的入口，介于客户端与服务器端之间，用于对请求进行鉴权、限流、 路由、监控等功能。 ![spring-cloud](../img/java/spring-cloud-12.png) 没错，网关有的功能，Zuul 基本都有。而 Zuul 中最关键的就是 路由和过滤器 了，在官方文档中 Zuul 的标题就是 >Router and Filter : Zuul #### Zuul 的路由功能 * 简单配置 本来想给你们复制一些代码，但是想了想，因为各个代码配置比较零散，看起来也比较零散，我决定还是给你们画个图来解释吧。 >请不要因为我这么好就给我点赞 👍 。疯狂暗示。 ![spring-cloud](../img/java/spring-cloud-13.png) 比如这个时候我们已经向 Eureka Server 注册了两个 Consumer 、三个 Provicer ，这个时候我们再加个 Zuul 网关应该变成这样子了。 emmm，信息量有点大，我来解释一下。关于前面的知识我就不解释了😐 。 首先，Zuul 需要向 Eureka 进行注册，注册有啥好处呢？ 你傻呀，Consumer 都向 Eureka Server 进行注册了，我网关是不是只要注册就能拿到所有 Consumer 的信息了？ 拿到信息有什么好处呢？ 我拿到信息我是不是可以获取所有的 Consumer 的元数据(名称，ip，端口)？ 拿到这些元数据有什么好处呢？拿到了我们是不是直接可以做路由映射？比如原来用户调用 Consumer1 的接口 localhost:8001/studentInfo/update 这个请求，我们是不是可以这样进行调用了呢？localhost:9000/consumer1/studentInfo/update 呢？你这样是不是恍然大悟了？ > 这里的url为了让更多人看懂所以没有使用 restful 风格。 上面的你理解了，那么就能理解关于 Zuul 最基本的配置了，看下面。 ``` server: port: 9000 eureka: client: service-url: # 这里只要注册 Eureka 就行了 defaultZone: http://localhost:9997/eureka ``` 然后在启动类上加入 @EnableZuulProxy 注解就行了。没错，就是那么简单😃。 * 统一前缀 这个很简单，就是我们可以在前面加一个统一的前缀，比如我们刚刚调用的是 localhost:9000/consumer1/studentInfo/update，这个时候我们在 yaml 配置文件中添加如下。 ``` zuul: prefix: /zuul ``` 这样我们就需要通过 localhost:9000/zuul/consumer1/studentInfo/update 来进行访问了。 * 路由策略配置 你会发现前面的访问方式(直接使用服务名)，需要将微服务名称暴露给用户，会存在安全性问题。所以，可以自定义路径来替代微服务名称，即自定义路由策略。 ``` zuul: routes: consumer1: /FrancisQ1/** consumer2: /FrancisQ2/** ``` 这个时候你就可以使用 localhost:9000/zuul/FrancisQ1/studentInfo/update 进行访问了。 * 服务名屏蔽 这个时候你别以为你好了，你可以试试，在你配置完路由策略之后使用微服务名称还是可以访问的，这个时候你需要将服务名屏蔽。 ``` zuul: ignore-services: "*" ``` * 路径屏蔽 Zuul 还可以指定屏蔽掉的路径 URI，即只要用户请求中包含指定的 URI 路径，那么该请求将无法访问到指定的服务。通过该方式可以限制用户的权限。 ``` zuul: ignore-patterns: **/auto/** ``` 这样关于 auto 的请求我们就可以过滤掉了。 >** 代表匹配多级任意路径 *代表匹配一级任意路径 * 敏感请求头屏蔽 默认情况下，像 Cookie、Set-Cookie 等敏感请求头信息会被 zuul 屏蔽掉，我们可以将这些默认屏蔽去掉，当然，也可以添加要屏蔽的请求头。 * Zuul 的过滤功能 如果说，路由功能是 Zuul 的基操的话，那么过滤器就是 Zuul的利器了。毕竟所有请求都经过网关(Zuul)，那么我们可以进行各种过滤，这样我们就能实现 限流，灰度发布，权限控制 等等。 * 简单实现一个请求时间日志打印 要实现自己定义的 Filter 我们只需要继承 ZuulFilter 然后将这个过滤器类以 @Component 注解加入 Spring 容器中就行了。 在给你们看代码之前我先给你们解释一下关于过滤器的一些注意点。 ![spring-cloud](../img/java/spring-cloud-14.png) 过滤器类型：Pre、Routing、Post。前置Pre就是在请求之前进行过滤，Routing路由过滤器就是我们上面所讲的路由策略，而Post后置过滤器就是在 Response 之前进行过滤的过滤器。你可以观察上图结合着理解，并且下面我会给出相应的注释。 ``` java // 加入Spring容器 @Component public class PreRequestFilter extends ZuulFilter { // 返回过滤器类型 这里是前置过滤器 @Override public String filterType() { return FilterConstants.PRE_TYPE; } // 指定过滤顺序 越小越先执行，这里第一个执行 // 当然不是只真正第一个 在Zuul内置中有其他过滤器会先执行 // 那是写死的 比如 SERVLET_DETECTION_FILTER_ORDER = -3 @Override public int filterOrder() { return 0; } // 什么时候该进行过滤 // 这里我们可以进行一些判断，这样我们就可以过滤掉一些不符合规定的请求等等 @Override public boolean shouldFilter() { return true; } // 如果过滤器允许通过则怎么进行处理 @Override public Object run() throws ZuulException { // 这里我设置了全局的RequestContext并记录了请求开始时间 RequestContext ctx = RequestContext.getCurrentContext(); ctx.set("startTime", System.currentTimeMillis()); return null; } } ``` ``` java // lombok的日志 @Slf4j // 加入 Spring 容器 @Component public class AccessLogFilter extends ZuulFilter { // 指定该过滤器的过滤类型 // 此时是后置过滤器 @Override public String filterType() { return FilterConstants.POST_TYPE; } // SEND_RESPONSE_FILTER_ORDER 是最后一个过滤器 // 我们此过滤器在它之前执行 @Override public int filterOrder() { return FilterConstants.SEND_RESPONSE_FILTER_ORDER - 1; } @Override public boolean shouldFilter() { return true; } // 过滤时执行的策略 @Override public Object run() throws ZuulException { RequestContext context = RequestContext.getCurrentContext(); HttpServletRequest request = context.getRequest(); // 从RequestContext获取原先的开始时间 并通过它计算整个时间间隔 Long startTime = (Long) context.get("startTime"); // 这里我可以获取HttpServletRequest来获取URI并且打印出来 String uri = request.getRequestURI(); long duration = System.currentTimeMillis() - startTime; log.info("uri: " + uri + ", duration: " + duration / 100 + "ms"); return null; } } ``` 上面就简单实现了请求时间日志打印功能，你有没有感受到 Zuul 过滤功能的强大了呢？ 没有？好的、那我们再来。 * 令牌桶限流 当然不仅仅是令牌桶限流方式，Zuul 只要是限流的活它都能干，这里我只是简单举个🌰。 ![spring-cloud](../img/java/spring-cloud-15.png) 我先来解释一下什么是 令牌桶限流 吧。 首先我们会有个桶，如果里面没有满那么就会以一定 固定的速率 会往里面放令牌，一个请求过来首先要从桶中获取令牌，如果没有获取到，那么这个请求就拒绝，如果获取到那么就放行。关注公众号Java面试那些事儿，回复关键字面试，获取最新得面试资料。很简单吧，啊哈哈、 下面我们就通过 Zuul 的前置过滤器来实现一下令牌桶限流。 ```java @Component @Slf4j public class RouteFilter extends ZuulFilter { // 定义一个令牌桶，每秒产生2个令牌，即每秒最多处理2个请求 private static final RateLimiter RATE_LIMITER = RateLimiter.create(2); @Override public String filterType() { return FilterConstants.PRE_TYPE; } @Override public int filterOrder() { return -5; } @Override public Object run() throws ZuulException { log.info("放行"); return null; } @Override public boolean shouldFilter() { RequestContext context = RequestContext.getCurrentContext(); if(!RATE_LIMITER.tryAcquire()) { log.warn("访问量超载"); // 指定当前请求未通过过滤 context.setSendZuulResponse(false); // 向客户端返回响应码429，请求数量过多 context.setResponseStatusCode(429); return false; } return true; } } ``` 这样我们就能将请求数量控制在一秒两个，有没有觉得很酷？ * 关于 Zuul 的其他 Zuul 的过滤器的功能肯定不止上面我所实现的两种，它还可以实现 权限校验，包括我上面提到的 灰度发布 等等。 当然，Zuul 作为网关肯定也存在 单点问题 ，如果我们要保证 Zuul 的高可用，我们就需要进行 Zuul 的集群配置，这个时候可以借助额外的一些负载均衡器比如 Nginx 。 ## Spring Cloud配置管理——Config ### 为什么要使用进行配置管理？ 当我们的微服务系统开始慢慢地庞大起来，那么多 Consumer 、Provider 、Eureka Server 、Zuul 系统都会持有自己的配置，这个时候我们在项目运行的时候可能需要更改某些应用的配置，如果我们不进行配置的统一管理，我们只能去每个应用下一个一个寻找配置文件然后修改配置文件再重启应用。 首先对于分布式系统而言我们就不应该去每个应用下去分别修改配置文件，再者对于重启应用来说，服务无法访问所以直接抛弃了可用性，这是我们更不愿见到的。 那么有没有一种方法既能对配置文件统一地进行管理，又能在项目运行时动态修改配置文件呢？ 那就是我今天所要介绍的 Spring Cloud Config 。 >能进行配置管理的框架不止 Spring Cloud Config 一种，大家可以根据需求自己选择（disconf，阿波罗等等）。而且对于 Config 来说有些地方实现的不是那么尽人意。 ### Config 是什么 >Spring Cloud Config 为分布式系统中的外部化配置提供服务器和客户端支持。使用 Config 服务器，可以在中心位置管理所有环境中应用程序的外部属性。 简单来说，Spring Cloud Config 就是能将各个 应用/系统/模块 的配置文件存放到 统一的地方然后进行管理(Git 或者 SVN)。 你想一下，我们的应用是不是只有启动的时候才会进行配置文件的加载，那么我们的 Spring Cloud Config 就暴露出一个接口给启动应用来获取它所想要的配置文件，应用获取到配置文件然后再进行它的初始化工作。就如下图。 ![spring-cloud](../img/java/spring-cloud-16.png) 当然这里你肯定还会有一个疑问，如果我在应用运行时去更改远程配置仓库(Git)中的对应配置文件，那么依赖于这个配置文件的已启动的应用会不会进行其相应配置的更改呢？ 答案是不会的。 什么？那怎么进行动态修改配置文件呢？这不是出现了 配置漂移 吗？你个渣男🤬，你又骗我！ 别急嘛，你可以使用 Webhooks ，这是 github 提供的功能，它能确保远程库的配置文件更新后客户端中的配置信息也得到更新。 噢噢，这还差不多。我去查查怎么用。 慢着，听我说完，Webhooks 虽然能解决，但是你了解一下会发现它根本不适合用于生产环境，所以基本不会使用它的。 而一般我们会使用 Bus 消息总线 + Spring Cloud Config 进行配置的动态刷新。 ## 引出 Spring Cloud Bus >用于将服务和服务实例与分布式消息系统链接在一起的事件总线。在集群中传播状态更改很有用（例如配置更改事件）。 你可以简单理解为 Spring Cloud Bus 的作用就是管理和广播分布式系统中的消息，也就是消息引擎系统中的广播模式。关注公众号Java面试那些事儿，回复关键字面试，获取最新得面试资料。当然作为 消息总线 的 Spring Cloud Bus 可以做很多事而不仅仅是客户端的配置刷新功能。 而拥有了 Spring Cloud Bus 之后，我们只需要创建一个简单的请求，并且加上 @ResfreshScope 注解就能进行配置的动态修改了，下面我画了张图供你理解。 ![spring-cloud](../img/java/spring-cloud-17.png) ## 总结 这篇文章中我带大家初步了解了 Spring Cloud 的各个组件，他们有 * Eureka 服务发现框架 * Ribbon 进程内负载均衡器 * Open Feign 服务调用映射 * Hystrix 服务降级熔断器 * Zuul 微服务网关 * Config 微服务统一配置中心 * Bus 消息总线 如果你能这个时候能看懂下面那张图，也就说明了你已经对 Spring Cloud 微服务有了一定的架构认识。 ![spring-cloud](../img/java/spring-cloud-18.png) ================================================ FILE: docs/java/springAnnotation.md ================================================ # Spring 中的 18 个注解 ## 1 @Controller 标识一个该类是Spring MVC controller处理器，用来创建处理http请求的对象. ```java @Controller public class TestController{ public String test(Map map){ return "hello"; } } ``` ## 2 @RestController Spring4之后加入的注解，原来在@Controller中返回json需要@ResponseBody来配合，如果直接用@RestController替代@Controller就不需要再配置@ResponseBody，默认返回json格式。 ## 3 @Service 用于标注业务层组件，说白了就是加入你有一个用注解的方式把这个类注入到spring配置中 ## 4 @Autowired 用来装配bean，都可以写在字段上，或者方法上。 默认情况下必须要求依赖对象必须存在，如果要允许null值，可以设置它的required属性为false，例如： ```java @Autowired(required=false) ``` ## 5 @RequestMapping 类定义处: 提供初步的请求映射信息，相对于 WEB 应用的根目录。 方法处: 提供进一步的细分映射信息，相对于类定义处的 URL。 ## 6 @RequestParam 用于将请求参数区数据映射到功能处理方法的参数上 例如 ``` java public Resp test(@RequestParam Integer id){ return Resp.success(customerInfoService.fetch(id)); } ``` 这个id就是要接收从接口传递过来的参数id的值的，如果接口传递过来的参数名和你接收的不一致，也可以如下 ``` java public Resp test (@RequestParam(value="course_id") Integer id){ return Resp.success(customerInfoService.fetch(id)); } ``` 其中course_id就是接口传递的参数，id就是映射course_id的参数名 ## 7 @ModelAttribute 使用地方如下： 1. 标记在方法上 标记在方法上，会在每一个@RequestMapping标注的方法前执行，如果有返回值，则自动将该返回值加入到ModelMap中。 (1). 在有返回的方法上:当ModelAttribute设置了value，方法返回的值会以这个value为key，以参数接受到的值作为value，存入到Model中，如下面的方法执行之后，最终相当于 model.addAttribute("user_name", name);假如 @ModelAttribute没有自定义value，则相当于 model.addAttribute("name", name); ``` java @ModelAttribute(value="user_name") public String before(@RequestParam(required = false) String Name,Model model){ System.out.println("name is "+name); } ``` (2) 在没返回的方法上： 需要手动model.add方法 ``` java @ModelAttribute public void before(@RequestParam(required = false) Integer age,Model model) { model.addAttribute("age",age); System.out.println("age is "+age); } ``` 2. 标记在方法的参数上 标记在方法的参数上，会将客户端传递过来的参数按名称注入到指定对象中，并且会将这个对象自动加入ModelMap中，便于View层使用.我们在上面的类中加入一个方法如下 ``` java public Resp model(@ModelAttribute("user_name") String user_name, @ModelAttribute("name") String name, @ModelAttribute("age") Integer age,Model model){ System.out.println("user_name="+user_name+" name="+name+" age="+age); System.out.println("model="+model); } ``` 用在方法参数中的@ModelAttribute注解，实际上是一种接受参数并且自动放入Model对象中，便于使用。 ## 8 @Cacheable 用来标记缓存查询。可用用于方法或者类中，当标记在一个方法上时表示该方法是支持缓存的，当标记在一个类上时则表示该类所有的方法都是支持缓存的。 参数列表 参数|解释|例子 -|-|-|- value|名称|@Cacheable(value={"c1","c2"}) key|key|@Cacheable(value="c1",key="#id") condition|条件|@Cacheable(value="c1",condition="#id=1") 比如@Cacheable(value="UserCache") 标识的是当调用了标记了这个注解的方法时，逻辑默认加上从缓存中获取结果的逻辑，如果缓存中没有数据，则执行用户编写查询逻辑，查询成功之后，同时将结果放入缓存中。 但凡说到缓存，都是key-value的形式的，因此key就是方法中的参数（id），value就是查询的结果，而命名空间UserCache是在spring*.xml中定义。 ```java @Cacheable(value="UserCache") public int getUserAge(int id){ int age=getAgeById(id); return age; } ``` ## 9 @CacheEvict @CacheEvict用来标记要清空缓存的方法，当这个方法被调用后，即会清空缓存。 @CacheEvict(value=”UserCache”) 参数列表 参数|解释|例子 -|-|-|- value|名称|@CacheEvict(value={"c1","c2"}) key|key|@CacheEvict(value="c1",key="#id") condition|缓存得条件可为空| allEntries|是否清空所有内容|@CacheEvict(value="c1",allEntries=true) beforeInvocation|是否在方法执行前清空|@CacheEvict(value="c1",beforeInvocation=true) ## 10 @Resource @Resource的作用相当于@Autowired 只不过@Autowired按byType自动注入， 而@Resource默认按 byName自动注入罢了。 @Resource有两个属性是比较重要的，分是name和type，Spring将@Resource注解的name属性解析为bean的名字，而type属性则解析为bean的类型。所以如果使用name属性，则使用byName的自动注入策略，而使用type属性时则使用byType自动注入策略。如果既不指定name也不指定type属性，这时将通过反射机制使用byName自动注入策略。 @Resource装配顺序: 1、如果同时指定了name和type，则从Spring上下文中找到唯一匹配的bean进行装配，找不到则抛出异常 2、如果指定了name，则从上下文中查找名称（id）匹配的bean进行装配，找不到则抛出异常 3、如果指定了type，则从上下文中找到类型匹配的唯一bean进行装配，找不到或者找到多个，都会抛出异常 4、如果既没有指定name，又没有指定type，则自动按照byName方式进行装配；如果没有匹配，则回退为一个原始类型进行匹配，如果匹配则自动装配； ## 11 @PostConstruct @PostConstruct用来标记是在项目启动的时候执行这个方法。用来修饰一个非静态的void()方法 也就是spring容器启动时就执行，多用于一些全局配置、数据字典之类的加载 被@PostConstruct修饰的方法会在服务器加载Servlet的时候运行，并且只会被服务器执行一次。PostConstruct在构造函数之后执行,init()方法之前执行。PreDestroy（）方法在destroy()方法执行之后执行 ## 12 @PreDestory 被@PreDestroy修饰的方法会在服务器卸载Servlet的时候运行，并且只会被服务器调用一次，类似于Servlet的destroy()方法。被@PreDestroy修饰的方法会在destroy()方法之后运行，在Servlet被彻底卸载之前 ## 13 @Repository @Repository用于标注数据访问组件，即DAO组件 ## 14 @Component @Component泛指组件，当组件不好归类的时候，我们可以使用这个注解进行标注 ## 15 @Scope @Scope用来配置 spring bean 的作用域，它标识 bean 的作用域。 默认值是单例 1、singleton:单例模式,全局有且仅有一个实例 2、prototype:原型模式,每次获取Bean的时候会有一个新的实例 3、request:request表示该针对每一次HTTP请求都会产生一个新的bean，同时该bean仅在当前HTTP request内有效 4、session:session作用域表示该针对每一次HTTP请求都会产生一个新的bean，同时该bean仅在当前HTTP session内有效 5、global session:只在portal应用中有用，给每一个 global http session 新建一个Bean实例。 ## 16 @SessionAttributes 默认情况下Spring MVC将模型中的数据存储到request域中。当一个请求结束后，数据就失效了。如果要跨页面使用。那么需要使用到session。而@SessionAttributes注解就可以使得模型中的数据存储一份到session域中 参数： 1、names：这是一个字符串数组。里面应写需要存储到session中数据的名称。 2、types：根据指定参数的类型，将模型中对应类型的参数存储到session中 3、value：和names是一样的。 ```java @Controller @SessionAttributes(value={"names"},types={Integer.class}) public class ScopeService{ @RequestMapping("/testSession") public String test(Map map){ map.put("names",Arrays.asList("a","b","c")); map.put("age",12); return "hello"; } } ``` ## 17 @Required 适用于bean属性setter方法，并表示受影响的bean属性必须在XML配置文件在配置时进行填充。否则，容器会抛出一个BeanInitializationException异常。 ## 18 @Qualifier @Qualifier当你创建多个具有相同类型的 bean 时，并且想要用一个属性只为它们其中的一个进行装配，在这种情况下，你可以使用 @Qualifier 注释和 @Autowired 注释通过指定哪一个真正的 bean 将会被装配来消除混乱。 ['更多']("https://mp.weixin.qq.com/s/Lq_iBz9cV9g11OvAwFmo-A") ================================================ FILE: docs/java/springdesign.md ================================================ # Spring 常用三种设计模式 关于设计模式，如果使用得当，将会使我们的代码更加简洁，并且更具扩展性。本文主要讲解Spring中如何使用策略模式，工厂方法模式以及Builder模式。 ## 策略模式 关于策略模式的使用方式，在Spring中其实比较简单，从本质上讲，策略模式就是一个接口下有多个实现类，而每种实现类会处理某一种情况。 我们以发奖励为例进行讲解，比如我们在抽奖系统中，有多种奖励方式可供选择，比如积分，虚拟币和现金等。在存储时，我们必然会使用一个类似于type的字段用于表征这几种发放奖励的，那么这里我们就可以使用多态的方式进行奖励的发放。比如我们抽象出一个PrizeSender的接口，其声明如下： ``` java public interface PrizeSender { /** * 用于判断当前实例是否支持当前奖励的发放 */ boolean support(SendPrizeRequest request); /** * 发放奖励 */ void sendPrize(SendPrizeRequest request); } ``` 该接口中主要有两个方法：support()和sendPrize()，其中support()方法主要用于判断各个子类是否支持当前类型数据的处理，而sendPrize()则主要是用于进行具体的业务处理的，比如这里奖励的发放。下面就是我们三种不同类型的奖励发放的具体代码： ```java // 积分发放 @Component public class PointSender implements PrizeSender { @Override public boolean support(SendPrizeRequest request) { return request.getPrizeType() == PrizeTypeEnum.POINT; } @Override public void sendPrize(SendPrizeRequest request) { System.out.println("发放积分"); } } // 虚拟币发放 @Component public class VirtualCurrencySender implements PrizeSender { @Override public boolean support(SendPrizeRequest request) { return PrizeTypeEnum.VIRTUAL_CURRENCY == request.getPrizeType(); } @Override public void sendPrize(SendPrizeRequest request) { System.out.println("发放虚拟币"); } } // 现金发放 @Component public class CashSender implements PrizeSender { @Override public boolean support(SendPrizeRequest request) { return PrizeTypeEnum.CASH == request.getPrizeType(); } @Override public void sendPrize(SendPrizeRequest request) { System.out.println("发放现金"); } } ``` 这里可以看到，在每种子类型中，我们只需要在support()方法中通过request的某个参数来控制当前request是否是当前实例能够处理的类型，如果是，则外层的控制逻辑就会将request交给当前实例进行处理。关于这个类的设计，有几个点需要注意： * 使用@Component注解对当前类进行标注，将其声明为Spring容器所管理的一个bean； * 声明一个返回boolean值的类似于support()的方法，通过这个方法来控制当前实例是否为处理目标request的实例； * 声明一个类似于sendPrize()的方法用于处理业务逻辑，当然根据各个业务的不同声明的方法名肯定是不同的，这里只是一个对统一的业务处理的抽象； * 无论是support()方法还是sendPrize()方法，都需要传一个对象进行，而不是简简单单的基本类型的变量，这样做的好处是后续如果要在Request中新增字段，那么就不需要修改接口的定义和已经实现的各个子类的逻辑； ## 工厂方法模式 上面我们讲解了如何使用Spring来声明一个策略模式，那么如何为不同的业务逻辑来注入不同的bean呢，或者说外层的控制逻辑是什么样的，这里我们就可以使用工厂方法模式了。 所谓的工厂方法模式，就是定义一个工厂方法，通过传入的参数，返回某个实例，然后通过该实例来处理后续的业务逻辑。一般的，工厂方法的返回值类型是一个接口类型，而选择具体子类实例的逻辑则封装到了工厂方法中了。通过这种方式，来将外层调用逻辑与具体的子类的获取逻辑进行分离。如下图展示了工厂方法模式的一个示意图： ![spring-design](../img/springdesign.png) 可以看到，工厂方法将具体实例的选择进行了封装，而客户端，也就是我们的调用方只需要调用工厂的具体方法获取到具体的事例即可，而不需要管具体的实例实现是什么。 上面我们讲解了Spring中是如何使用策略模式声明处理逻辑的，而没有讲如何选择具体的策略，这里我们就可以使用工厂方法模式。 如下是我们声明的一个PrizeSenderFactory： ```java @Component public class PrizeSenderFactory { @Autowired private List prizeSenders; public PrizeSender getPrizeSender(SendPrizeRequest request) { for (PrizeSender prizeSender : prizeSenders) { if (prizeSender.support(request)) { return prizeSender; } } throw new UnsupportedOperationException("unsupported request: " + request); } } ``` 这里我们声明一个了一个工厂方法getPrizeSender()，其入参就是SendPrizeRequest，而返回值是某个实现了PrizeSender接口的实例，可以看到，通过这种方式，我们将具体的选择方式下移到了具体的子类中的，因为当前实现了PrizeSender的bean是否支持当前request的处理，是由具体的子类实现的。 在该工厂方法中，我们也没有任何与具体子类相关的逻辑，也就是说，该类实际上是可以动态检测新加入的子类实例的。这主要是通过Spring的自动注入来实现的，主要是因为我们这里注入的是一个List，也就是说，如果有新的PrizeSender的子类实例，只要其是Spring所管理的，那么都会被注入到这里来。下面就是我们编写的一段用于测试的代码来模拟调用方的调用： ``` java @Service public class ApplicationService { @Autowired private PrizeSenderFactory prizeSenderFactory; public void mockedClient() { SendPrizeRequest request = new SendPrizeRequest(); request.setPrizeType(PrizeTypeEnum.POINT); // 这里的request一般是根据数据库或外部调用来生成的 PrizeSender prizeSender = prizeSenderFactory.getPrizeSender(request); prizeSender.sendPrize(request); } } ``` 在客户端代码中，首先通过PrizeSenderFactory获取一个PrizeSender实例，然后通过其sendPrize()方法发放具体的奖励，通过这种方式，将具体的奖励发放逻辑与客户端调用进行了解耦。而且根据前面的讲解，我们也知道，如果新增了一种奖励方式，我们只需要声明一个新的实现了PrizeSender的bean即可，而不需要对现有代码进行任何修改。 ## Builder模式 关于Builder模式，我想使用过lombok的同学肯定会说builder模式非常的简单，只需要在某个bean上使用@Builder注解进行声明即可，lombok可以自动帮我们将其声明为一个Builder的bean。关于这种使用方式，本人不置可否，不过就我的理解，这里主要有两个点我们需要理解： 1、Builder模式就其名称而言，是一个构建者，我更倾向于将其理解为通过一定的参数，通过一定的业务逻辑来最终生成某个对象。如果仅仅只是使用lombok的这种方式，其本质上也还是创建了一个简单的bean，这个与通过getter和setter方式构建一个bean是没有什么大的区别的； 2、在Spring框架中，使用设计模式最大的问题在于如果在各个模式bean中能够注入Spring的bean，如果能够注入，那么将大大的扩展其使用方式。因为我们就可以真的实现通过传入的简单的几个参数，然后结合Spring注入的bean进行一定的处理后，以构造出我们所需要的某个bean。显然，这是lombok所无法实现的； 关于Builder模式，我们可以以前面奖励发放的SendPrizeRequest的构造为例进行讲解。在构造request对象的时候，必然是通过前台传如的某些参数来经过一定的处理，最后生成一个request对象。那么我们就可以使用Builder模式来构建一个SendPrizeRequest。 这里假设根据前台调用，我们能够获取到prizeId和userId，那么我们就可以创建一个如下的SendPrizeRequest： ```java public class SendPrizeRequest { private final PrizeTypeEnum prizeType; private final int amount; private final String userId; public SendPrizeRequest(PrizeTypeEnum prizeType, int amount, String userId) { this.prizeType = prizeType; this.amount = amount; this.userId = userId; } @Component @Scope("prototype") public static class Builder { @Autowired PrizeService prizeService; private int prizeId; private String userId; public Builder prizeId(int prizeId) { this.prizeId = prizeId; return this; } public Builder userId(String userId) { this.userId = userId; return this; } public SendPrizeRequest build() { Prize prize = prizeService.findById(prizeId); return new SendPrizeRequest(prize.getPrizeType(), prize.getAmount(), userId); } } public PrizeTypeEnum getPrizeType() { return prizeType; } public int getAmount() { return amount; } public String getUserId() { return userId; } } ``` 这里就是使用Spring维护一个Builder模式的示例，具体的 维护方式就是在Builder类上使用@Component和@Scope注解来标注该Builder类，这样我们就可以在Builder类中注入我们所需要的实例来进行一定的业务处理了。关于该模式，这里有几点需要说明： * 在Builder类上必须使用@Scope注解来标注该实例为prototype类型，因为很明显，我们这里的Builder实例是有状态的，无法被多线程共享； * 在Builder.build()方法中，我们可以通过传入的参数和注入的bean来进行一定的业务处理，从而得到构建一个SendPrizeRequest所需要的参数； * Builder类必须使用static修饰，因为在Java中，如果内部类不用static修饰，那么该类的实例必须依赖于外部类的一个实例，而我们这里本质上是希望通过内部类实例来构建外部类实例，也就是说内部类实例存在的时候，外部类实例是还不存在的，因而这里必须使用static修饰； * 根据标准的Builder模式的使用方式，外部类的各个参数都必须使用final修饰，然后只需要为其声明getter方法即可。 上面我们展示了如何使用Spring的方式来声明一个Builder模式的类，那么我们该如何进行使用呢，如下是我们的一个使用示例： ```java @Service public class ApplicationService { @Autowired private PrizeSenderFactory prizeSenderFactory; @Autowired private ApplicationContext context; public void mockedClient() { SendPrizeRequest request = newPrizeSendRequestBuilder() .prizeId(1) .userId("u4352234") .build(); PrizeSender prizeSender = prizeSenderFactory.getPrizeSender(request); prizeSender.sendPrize(request); } public Builder newPrizeSendRequestBuilder() { return context.getBean(Builder.class); } } ``` 上述代码中，我们主要要看一下newPrizeSendRequestBuilder()方法，在Spring中，如果一个类是多例类型，也即使用@Scope("prototype")进行了标注，那么每次获取该bean的时候就必须使用ApplicationContext.getBean()方法获取一个新的实例，至于具体的原因，读者可查阅相关文档。 我们这里就是通过一个单独的方法来创建一个Builder对象，然后通过流式来为其设置prizeId和userId等参数，最后通过build()方法构建得到了一个SendPrizeRequest实例，通过该实例来进行后续的奖励发放。 ## 小结 本文主要通过一个奖励发放的示例来对Spring中如何使用工厂方法模式，策略模式和Builder模式的方式进行讲解，并且着重强调了实现各个模式时我们所需要注意的点。 ================================================ FILE: docs/js/baidu-tongji.js ================================================ var _hmt = _hmt || []; (function() { let hm = document.createElement("script"); // rosemarys.gitee.io hm.src = "https://hm.baidu.com/hm.js?08563f1498cfbdc12f55e3fc8b133b4b"; let s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); // burningmyself.cn hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?ea33cbaa80e12b2127ea3d44494c0870"; s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); // burningmyself.gitee.io hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?10b96d1600d054d9d9519e5512e0af19"; s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); // burningmyself.github.io hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?14446117c56cf1fdab44c58081833af8"; s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); // yangfubing.gitee.io hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?66401a22d7cba55cbbdd060204e0f29d"; s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); // yangfubing.github.io hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?dd0db5e74ea516ec6bee7d759094aaaa"; s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); })(); ================================================ FILE: docs/js/extra.js ================================================ (function () { let generator = document.getElementsByTagName("meta").generator; generator.name="keywords"; generator.content="net,java,php,python,docker,web"; let meta = document.createElement("meta"); let s = document.getElementsByTagName("meta")[0]; meta.name="google-site-verification"; meta.content="le9TAKnSKhLDEEGnDu2ofXi3taLVIxmKNT0bEIsetNE"; s.parentNode.insertBefore(meta, s); let copyright = document.getElementsByClassName("md-footer-copyright") copyright[0].outerHTML=document.getElementsByClassName("md-footer-copyright__highlight")[0].outerHTML })(); ================================================ FILE: docs/js/google.js ================================================ // burningmyself.gitee.io // Global site tag (gtag.js) - Google Analytics /* */ window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-155084439-1'); //yangfubing.gitee.io /* */ window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-155132293-1'); ================================================ FILE: docs/kubernetes/k8s_controller_manager.md ================================================ # kubernetes 控制平面组件 - kube-scheduler - Controller Manager - kubelet - CRI - CNI - CSI ## kuber-scheduler kube-scheduler负责分配调度Pod到集群内的节点上，它监听kube-apiserver,查询还未分配Node的Pod,然后根据调度策略为这些Pod分配节点（更新Pod的NodeName字段）。 - 公平调度 - 资源高效利用 - Qos - affinity和anti-affinity - 数据本地化（inter-workload interference） - deadlines ### 调度器 kube-scheduler调度分为两个阶段，predicate和priority - predicate：过滤不符合条件的节点 - priority: 优先级排序，选择优先级最高的节点 #### Predicates 策略 ![k8s_controller_manager_1](../img/k8s/k8s_controller_manager/k8s_controller_manager_1.jpg) ![k8s_controller_manager_2](../img/k8s/k8s_controller_manager/k8s_controller_manager_2.jpg) #### Priorities plugin工作原理 ![k8s_controller_manager_3](../img/k8s/k8s_controller_manager/k8s_controller_manager_3.jpg) #### Priorities策略 ![k8s_controller_manager_4](../img/k8s/k8s_controller_manager/k8s_controller_manager_4.jpg) ![k8s_controller_manager_5](../img/k8s/k8s_controller_manager/k8s_controller_manager_5.jpg) ### 资源需求 - CPU - requests - Kubernetes调度Pod时，会判断当前节点正在运行的Pod的CPU Request的总和，再加上当前调度Pod的CPU request，计算其是否超过节点的CPU的可分配资源 - limits - 配置cgroup以限制资源上限 - 内存 - requests - 判断节点的剩余内存是否满足Pod的内存请求量，以确定是否可以将Pod调度到该节点 - limits - 配置cgroup以限制资源上限 - 磁盘资源需求 - 容器临时存储（epheme「凯storage）包含日志和可写层数据，可以通过定义Pod Spec中的 Limits.ephemeral-storage 和 requests. ephemevL-stonage 来申请 - Pod调度完成后,计算节点对临时存储的限制不是基于CGroup的，而是由kubelet定时获取容器的日志 和容器可写层的磁盘使用情况，如果超过限制，则会对Pod进行驱逐 ### Init Container的资源需求 - 当kube-scheduleri|§度带有多个init:容器的Pod时，只计算cpu.request最多的init容器，而不是计算所有的init容器总和。 - 由于多个init容器按顺序执行，并且执行完成立即退出，所以申请最多的资源init容器中的所需资源, 即可满足所有init容器需求。 - kube-scheduler在计算该节点被占用的资源时，init容器的资源依然会被纳入计算。因为init容器在 特定情况下可能会被再次执行，比如由于更换镜像而引起Sandbox重建时。 ### 把Pod调度到指定Node上 - 可以通过 nodeSeLector、nodeAffinitys podAffinity 以及Taints和tolerations等来将Pod调度动需要的Node 上。 - 也可以通过设置nodeName参数，将Pod调度到指定node节点上。 比如，使用nodeselector,首先给Node加上标签: kubectl label nodes disktype=ssd 接着，指定该Pod只想运行在带有disktype=ssd标签的 Node上 ```yaml apiVersion: vl kind: Pod metadata: name: nginx labels: env: test spec: containers: -name: nginx image: nginx imagePullPolicy: IfNotPresent nodeSelector: disktype: ssd ``` ### nodeSelector 首先给Node打上标签： kubectl label nodes node-01 disktype=ss 然后在 daemonset 中指定 nodeselector 为 disktype=ssd: spec: nodeSelector: disktype: ssd ### NodeAffinity NodeAffinity 目前支持两种：requiredDuringSchedulinglgnoredDuringExecution 和 preferredDuringSchedulinglgnoredDuringExecution,分别代表必须满足条件和优选条件。 比如下面的例子代表调度到包含标签Kubernetes.io/e2e-az-name并且值为e2e-az1或e2e-az2的 Node 上,并且优选还帯自标签 another-node-label-key=another-node-label-value 的 Node。 ``` yaml apiVersion: vl kind: Pod metadata: name: with-node-affinity spec: affin ity: no deAffi nity: requiredDuri ngSchedulinglg no redDuringExecution: no deSelectorTerms: -matchExpressions: -key: kubernetes.io/e2e-az-name operator: In values: -e2e-azl -e2e-az2 preferredDuringSchedulinglgnoredDuringExecution: -weight: 1 pref ere nee: matchExpressi ons: -key: another-node-label-key operator: In values: -ano ther-no de-label-value containers: -name: with-node-affinity image: gcr.io/google_c ontain ers/pause:2.0 ``` ### podAffinity podAffinity基于Pod的标签来选择Nod巳仅调度到满足条件Pod所在的Node上，支持 podAffinity和podAntiAffinity。这个功能比较绕'以下面的例子为例： 如果一个“Node所在zone中包含至少一个带有security=Sl标签且运行中的Pod” ,那么可以调度 到该Nod巳不调度到“包含至少一个带有security=S2标签且运行中Pod〃的Node上。 ### podAffinity 示例 ``` yaml apiVersion: vl kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulinglgnoredDuringExecution: -labelselector: matchExpressi ons: -key: security operator: In values: -SI topologyKey: failure-domain.beta.kubernetes.io/zone podAntiAffinity: preferredDuringSchedulinglgnoredDuringExecution: -weight: 100 podAffinityTerm: labelselector: matchExpressions: -key: security operator: In values: -S2 topologyKey: kubernetes.io/hostname containers: -name: with-pod-affinity image: gcr.io/google_containers/pause:2.0 ``` ### Taints 和 Tolerations Taints和Tolerations用于保证Pod不被调度到不合适的Node上,其中Taint应用于Node上，而 Toleration则应用于Pod上。 目前支持的Taint类型： - NoSchedule:新的Pod不调度到该Node ±,不影响正在运行的Pod; - PreferNoSchedule: soft版的 NoSchedule,尽量不调度到该 Node 上; - NoExecute:新的Pod不调度到该Node±,并且删除(evict)已在运行的Pod。Pod可以 增加一个时间(tolerationSeconds)。 然而，当Pod的Tolerations匹配Node的所有Taints的时候可以调度到该Node±;当Pod是已 经运行的时候，也不会被删除(evicted)，另外对于NoExecute,如果Pod增加了一个 tolerationSeconds,则会在该时间之后才删除Pod ### 多租户Kubernetes集群-计算资源隔离 Kubernetes集群一般是通用集群，可被所有用户共享，用户无需关心计算节点细节。 但往往某些自带计算资源的客户要求： - 带着计算资源加入Kubernetes集群； - 要求资源隔离。 实现方案： - 将要隔离的计算节点打上Taints; - 在用户创建创建Pod时，定义tolerations来指定要调度到nodetaints0 该方案有漏洞吗？如何堵住？ - 其他用户如果可以get nodes或者pods,可以看到taints信息，也可以用用同的tolerations占用资源。 - 不让用户get node detail? - 不让用户get别人的pod detail? - 企业内部，也可以通过规范管理，通过统计数据看谁占用了哪些node; - 数据平面上的隔离还需要其他方案配合。 ### 来自生产系统的经验 - 用户会忘记打tolerance,导致pod无法调度，pending; - 新员工常犯的错误，通过聊天机器人的Q&A解决； - 其他用户会get node detail,查到Taints,偷用资源。 - 通过dashboard,能看到哪些用户的什么应用跑在哪些节点上; - 对于违规用户'批评教育为主。 ### 优先级调度 从v1.8开始，kube-scheduler支持定义Pod的优先级，从而保证高优先级的Pod优先调度。开启方法为： apiserve配置--feature-gates=PodPriority=true --runtime-config=scheduling.k8s.io/v1alpha1=true kube-scheduler配置--feature-gates=PodPriority=true ### Priorityclass 在指定Pod的优先级之前需要先定义一个Priorityclass （非namespace资源）,如： ``` yaml apiVersion: vl kind: PriorityCless metadata: name: high-priority value: 1000000 globalDefault: false description: "This priority class should be used for XYZ service pods only." ``` ### 为Pod设置priority ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: env: test spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent priorityClassName: high-priority ``` ### 多调度器 如果默认的调度器不满足要求，还可以部署自定义的调度器。并且，在整个集群中还可以同时运行多个调度器实例，通过podSpec.schedulerName来选择使用哪一个调度器（默认使用内置的调度器）。 ### 来自生产的一些经验 调度器可以说是运营过程中稳定性最好的组件之一，基本没有太大的维护effort ![k8s_controller_manager_6](../img/k8s/k8s_controller_manager/k8s_controller_manager_6.jpg) ## Controller Manager ### 控制工作流程 ![k8s_controller_manager_7](../img/k8s/k8s_controller_manager/k8s_controller_manager_7.jpg) ### Informer 的内部机制 ![k8s_controller_manager_8](../img/k8s/k8s_controller_manager/k8s_controller_manager_8.jpg) ### 控制器的协同工作原理 ![k8s_controller_manager_9](../img/k8s/k8s_controller_manager/k8s_controller_manager_9.jpg) ### 通用 Controller - Job Controller:处理 job。 - Pod AutoScaler:处理pod为自动缩容/扩容。 - RelicaSet:依据 ReplicasetSpec 创建 Pod。 - Service Controller:为 LoadBalancertype 的 service 创建 LB VIP - ServiceAccount Controller:确保 serviceaccount在当前 namespace存在。 - StatefulSet Controller:处理 statefulset 中的 pod。 - Volume Controller:依据 PV spec 创建 volume。 - Resource quota Controller:在用户使用资源之后，更新状态。 - Namespace Controller:保证namespace删除时，该namespace下的所有资源都先被删除 - Replication Controller:创建 RC 后，负责创建 Pod。 - Node Controller:维护node状态，处理evict请求等。 - Daemon Controller:依据 damonset 创建 Pod。 - Deployment Controller:依据 deployment spec 创建 replicaset。 - Endpoint Controller:依据 service spec 创建 endpoint,依据 podip 更新 endpoint。 - Garbage Collector:处理级联删除，比如删除deployment的同时删除replicaset以及Pod - Cronjob Controller:处理 cronjob ### Cloud Controller Manager 什么时候需要 cloud controller manager? Cloud Controller Manager 自 Kubernetesl.6 开始，从 kube-controller-manager 中分离出来,主 要因为 Cloud Controller Manager 往往需要跟企业 cloud 做深度集成,release cycle 跟Kubernetes 相对独立。 与Kubernetes核心管理组件一起升级是一件费时费力的事。 通常 cloud controller manager 需要： - 认证授权：企业cloud往往需要认证信息，Kubernetes要与Cloud API通信，需要获取 cloud系统里的 ServiceAccount; - Cloud controller manager 本身作为一个用户态的 component,需要在 Kubernetes 中有 正确的RBAC设置，获得资源操作权限； - 高可用：需要通过leader election来确保cloud controller manger高可用。 ### Cloud controller manager 的配置 - cloud controller manager是从老版本的APIServer分离出来的。 - Kube-APIServer 和 kube-controller-manager 中一定不能指定 cloud-provider,否则会加载 内置的cloud controller manager - Kubelet 要配置--cloud-provider=external - Cloud Controller Manager 主要支持： - Node controller:访问cloud API,来更新node状态；在cloud删除该节点以后，从 - kubernetes 删除 node; - Service controller:负责配置为loadbalancer类型的服务配置LB VIP; - Route Controller:在 cloud 环境配置路由； - 可以自定义任何需要的Cloud Controller。 ### 需要定制的Cloud controller - Ingress controller; - Service Controller; - 自主研发的controller,比如之前提到的: - RBAC controller; - Accountcontroller。 ### 来自生产的经验 - 保护好controller manager 的 kubeconfig： - 此kubeconfig拥有所有资源的所有操作权限，防止普通用户通过kubectl exec kube-controller-manager cat 获取该文件。 - 用户可能做任何你想象不到的操作，然后来找你support - Podevict后IP发生变化,但endpoint中的address更新失败： - 分析 stacktrace 发现 endpoint 在更新 LoadBalancer 时调用 gophercloud 连接 hang 住，导 致endpoint worker线程全部卡死。 ### 确保scheduler和controller的高可用 Leader Election Kubenetes 提供基于 configmap 和 endpoint 的 leader election 类库 Kubernetes采用leader election模式启动component后，会创建对应endpoint,并把当前的 leader 信息、annotate 到 endponit 上 ```yaml apiVersion: vl kind: Endpoints metadata: annotations: control-plane.alpha.kuber netes.io/leader: ‘{"holderldentity":"minikube"/"leaseDurationSeconds":15/"acquireTimen:"2023-04- 05T17:31:29Z","renewTime":"2023-04-07T07:31:29Z”，“leaderTransitions”:0}' creationTimestamp: 2023-04-05T17:31:29Z name：kube-scheduler namespace: kube-system resourceversion: "138930" selfLink: /api/vl/namespaces/kube-system/endpoints/kube-scheduler uid: 2d12578d-38f7-11 e8-8df0-0800275259e5 subsets: null ``` ### Leader Election ![k8s_controller_manager_10](../img/k8s/k8s_controller_manager/k8s_controller_manager_10.jpg) ## kubelet ### kubelet构架 ![k8s_controller_manager_11](../img/k8s/k8s_controller_manager/k8s_controller_manager_11.jpg) ### kubelet管理Pod的核心流程 ![k8s_controller_manager_12](../img/k8s/k8s_controller_manager/k8s_controller_manager_12.jpg) ### Kuberlet 每个节点上都运行一个kubelet服务进程，默认监听10250端口。 - 接收并执行master发来的指令； - 管理Pod及Pod中的容器； - 每个kubelet进程会在API Server上注册节点自身信息，定期向master节点汇报节点的资 源使用情况，并通过cAdvisor监控节点和容器的资源。 ### 节点管理 节点管理主要是节点自注册和节点状态更新： - Kubelet可以通过设置启动参数-register-node来确定是否向API Server注册自己 - 如果Kubelet没有选择自注册模式，则需要用户自己配置Node资源信息，同时需要告知Kubelet集群上的API Server的位置； - Kubelet在启动时通过API Server注册节点信息,并定时向API Server发送节点新消息，API Server在接收到新消息后，将信息写入etcd。 ### Pod管理 获取Pod清单： - 文件：启动参数--config指定的配置目录下的文件(默认/etc/Kubernetes/manifests/)。该文件每20秒重新检查一次(可配置)。 - HTTP endpoint (URL):启动参数一manifest-url设置。每20秒检查一次这个端点(可配置 )。 - API Server:通过 API Server 监听 etcd 目录，同步 Pod 清单。 - HTTP server: kubelet侦听HTTP请求，并响应简单的API以提交新的Pod清单。 ### Pod启动流程 ![k8s_controller_manager_13](../img/k8s/k8s_controller_manager/k8s_controller_manager_13.jpg) ### Kubelet启动Pod流程 ![k8s_controller_manager_14](../img/k8s/k8s_controller_manager/k8s_controller_manager_14.jpg) ## CRI 容器运行时(Container Runtime),运行于Kubernetes (k8s)集群的每个节点中,负责容器的整 个生命周期。其中Docker是目前应用最广的。随着容器云的发展，越来越多的容器运行时涌现。为了 解决这些容器运行时和Kubernetes的集成问题，在Kubernetes 1.5版本中，社区推出了 CRI ( Container Runtime Interface,容器运行时接口)以支持更多的容器运行时。 ![k8s_controller_manager_15](../img/k8s/k8s_controller_manager/k8s_controller_manager_15.jpg) CRI是Kubernetes定义的一组gRPC服务。kubelet作为客户端，基于gRPC框架，通过Socket和 容器运行时通信。它包括两类服务：镜像服务(Image Service)和运行时服务(Runtime Service) o镜像服务提供下载、检查和删除镜像的远程程序调用。运行时服务包含用于管理容器生命周期，以 及与容器交互的调用(exec/attach/port-forward)的远程程序调用。 ![k8s_controller_manager_16](../img/k8s/k8s_controller_manager/k8s_controller_manager_16.jpg) ### 运行时的层级 Dockershim, containerd和CRI-0都是遵循CRI的容器运行时，我们称他们为高层级运行时 (High-Level Runtime)。 OCI (Open Container Initiative,开放容器计划)定义了创建容器的格式和运行时的开源行业标准， 包括镜像规范(Image Spec币cation)和运行时规范(Runtime Spec币cation)。 镜像规范定义了 OCI镜像的标准。高层级运行时将会下载一个OCI镜像，并把它解压成OCI运行时文 件系统包(filesystem bundle)。 运行时规范则描述了如何从OCI运行时文件系统包运行容器程序’并且定义它的配置、运行环境和生 命周期。如何为新容器设置命名空间(namepsaces)和控制组(cgroups),以及挂载根文件系统 等等操作，都是在这里定义的。它的一个参考实现是runC。我们称其为低层级运行时(Low-level R untime)。除runC以外，也有很多其他的运行时遵循OCI标准，例如kata-runtime。 容器运行时是真正起删和管理容器的组件。容器运行时可以分为高层和低层的运行时。高层运行时主要包括Docker, containerd和CRI-O,低层的运行时，包含了 runc, kata,以及gVisor。低层运行时kata和gVisor都还处于小 规模落地或者实验阶段，其生态成熟度和使用案例都比较欠缺，所以除非有特殊的需求，否则runc几乎是必然的选择。因此在对容器运行时的选择上，主要是聚焦于上层运行时的选择。 Docker内部关于容器运行时功能的核心组件是containerd,后来containerd也可直接和kubelet通过CRI对接， 独立在Kubernetes中使用。相对于Docker而言，containerd减少了 Docker所需的处理模块Dockerd和 Docker-shim,并且对Docker支持的存储驱动进行了优化，因此在容器的创建启动停止和删除，以及对镜像的拉取上，都具有性能上的优势。架构的简化同时也带来了维护的便利。当然Docker也具有很多containerd不具有的功能，例如支持zfs存储驱动，支持对日志的大小和文件限制，在以overlayfs2做存储驱动的情况下，可以通过 xfs_quota来对容器的可写层进行大小限制等。尽管如此，containerd目前也基本上能够满足容器的众多管理需求, 所以将它作为运行时的也越来越多。 CRI ![k8s_controller_manager_17](../img/k8s/k8s_controller_manager/k8s_controller_manager_17.jpg) ![k8s_controller_manager_18](../img/k8s/k8s_controller_manager/k8s_controller_manager_18.jpg) ### 开源运行时的比较 Docker的多层封装和调用，导致其在可维护性上略逊一筹，增加了线上问题的定位难度；几乎除了重 启Docker,我们就毫无施法了。 containerd和CRI-O的方案比起Docker简洁很多。 ![k8s_controller_manager_19](../img/k8s/k8s_controller_manager/k8s_controller_manager_19.jpg) ### Docker和containerd差异细节 ![k8s_controller_manager_20](../img/k8s/k8s_controller_manager/k8s_controller_manager_20.jpg) ### 多种运行时的性能比较 containerd在各个方面都表现良好，除了启动容器这项。从总用时来看，containerd的用时还是要比 CRI-O要短的 ![k8s_controller_manager_21](../img/k8s/k8s_controller_manager/k8s_controller_manager_21.jpg) ### 运行时优劣对比 功能性来讲,containerd和CRI-O都符合CRI和OCI的标准; 在稳定性上，containerd略胜一筹； 从性能上讲，containerd胜出. ![k8s_controller_manager_22](../img/k8s/k8s_controller_manager/k8s_controller_manager_22.jpg) ## CNI Kubernetes网络模型设计的基础原则是： - 所有的Pod能够不通过NAT就能相互访问。 - 所有的节点能够不通过NAT就能相互访问。 - 容器内看见的IP地址和外部组件看到的容器IP是一样的。 Kubernetes的集群里，IP地址是以Pod为单位进行分配的，每个Pod都拥有一个独立的IP地址。一 个Pod内部的所有容器共享一个网络栈，即宿主机上的一个网络命名空间’包括它们的IP地址、网络 设备、配置等都是共享的。也就是说，Pod里面的所有容器能通过localhost:port来连接对方。在 Kubernetes中，提供了一个轻量的通用容器网络接口 CNI (Container Network Interface),专门 用于设置和删除容器的网络连通性。容器运行时通过CNI调用网络插件来完成容器的网络设置。 ### CNI插件分类和常见插件 - IPAM： IP地址分配 - 主插件：网卡设置 - bridge:创建一个网桥，并把主机端口和容器端口插入网桥 - ipvlan:为容器添加ipvlan网口 - loopback:设置Loopback网口 - Meta:附加功能 - portmap:设置主机端口和容器端口映射 - bandwidth:利用Linux Traffic Control限流 - firewall:通过iptdbles或firewaHd为容器设置防火墙规则 https://qithub.com/containernetworkinq/pluqins ### CNI插件运行机制 容器运行时在启动时会从CNI的配置目录中读取JSON格式的配置文件，文件后缀为".conf" ".conflist",".json"。如果配置目录中包含多个文件，一般情况下,会以名字排序选用第一个配置文件作为默认的网络配置，并加载获取其中指定的CNI插件名称和配置参数。 ![k8s_controller_manager_23](../img/k8s/k8s_controller_manager/k8s_controller_manager_23.jpg) ### CNI的运行机制 关于容器网络管理，容器运行时一般需要配置两个参--cni-bin-dir和--cni-conf-dir。有一种特殊情况，kubelet内置的Docker作为容器运行时，是由kubelet来查找CNI插件的，运行插件来为容器 设置网络，这两个参数应该配置在kubelet处： - cni-bin-dir:网络插件的可执行文件所在目录。默认是/opt/cni/bin - cni-conf-dir:网络插件的配置文件所在目录。默认是/etc/cni/net.d ### CNI 插件设计考量 ![k8s_controller_manager_24](../img/k8s/k8s_controller_manager/k8s_controller_manager_24.jpg) ![k8s_controller_manager_25](../img/k8s/k8s_controller_manager/k8s_controller_manager_25.jpg) ### 打通主机层网络 CNI插件外，Kubernetes还需要标准的CNI插件Lo,最低版本为0.2.0版本。网络插件除支持设置和 清理Pod网络接口外，该插件还需要支持lptables。如果Kube-proxy工作在Iptables模式，网络插件需要确保容器流量能使用Iptables转发。例如，如果网络插件将容器连接到Linux网桥，必须将net/bridge/bridge-nf-call-iptables参数sysctl设置为1,网桥上数据包将遍历Iptables规则。如果插件不使用Linux桥接器（而是类似Open vSwitch或其他某种机制的插件）,则应确保容器流量被正确设置了路由。 ### CNI Plugin ContainerNetworking 组维护了一些 CNI 插件，包括网络接口创建的 bridge, ipvlan. Loopback, macvlan、ptp、host-device 等，IP 地址分酉己的 DHCP、host-local 和 static,其他的 Flannel, tunning、portmaps firewall等。 社区还有些第三方网络策略方面的插件，例如Calico. Cilium和Weave等。可用选项的多样性意味 着大多数用户将能够找到适合其当前需求和部署环境的CNI插件，并在情况变化时迅捷转换解决方案。 ### Flannel Flannel是由CoreOS开发的项目，是CNI插件早期的入门产品,简单易用。 Flannel使用Kubernetes集群的现有etcd集群来存储其状态信息，从而不必提供专用的数据存储，只需要在每个节点上运行 flanneld来守护进程。 每个节点都被分配一个子网，为该节点上的Pod分配IP地址。 同一主机内的Pod可以使用网桥进行通信，而不同主机上的Pod将通过fbanneld将其流量封装在UDP数据包中，以路由到适当的目的地。 封装方式默认和推荐的方法是使用VxLAN,因为它具有良好的性能，并且比其他选项要少些人为干预。虽然使用VxLAN之类的技术 封装的解决方案效果很好，但缺点就是该过程使流量跟踪变得困难。 ![k8s_controller_manager_26](../img/k8s/k8s_controller_manager/k8s_controller_manager_26.jpg) ### Calico Calico以其性能、灵活性和网络策略而闻名，不仅涉及在主机和Pod之间提供网络连接，而且还涉及网络安全性和 策略管理。 对于同网段通信，基于第3层，Calico使用BGP路由协议在主机之间路由数据包，使用BGP路由协议也意味着数据 包在主机之间移动时不需要包装在额外的封装层中。 对于跨网段通信，基于IPinlP使用虚拟网卡设备tunlO,用一个IP数据包封装另一个IP数据包，外层IP数据包头的 源地址为隧道入口设备的IP地址，目标地址为隧道出口设备的IP地址。 网络策略是Calico最受欢迎的功能之一，使用ACLs协议和kube-proxy来创建iptables过滤规则，从而实现隔离 容器网络的目的。 此外，Calico还可以与服务网格Istio集成，在服务网格层和网络基础结构层上解释和实施集群中工作负载的策略。 这意味着您可以配置功能强大的规则，以描述Pod应该如何发送和接收流量，提高安全性及加强对网络环境的控制。 Calico属于完全分布式的横向扩展结构，允许开发人员和管理员快速和平稳地扩展部署规模。对于性能和功能（如 网络策略）要求高的环境，Calico是一个不错选择。 ### Calico组件 ![k8s_controller_manager_27](../img/k8s/k8s_controller_manager/k8s_controller_manager_27.jpg) ### Calico初始化 配置和CNI二进制文件由initContainer推送 ``` yaml -comma nd: -/opt/cni/bin/install env: -name: CNI_CONF_NAME value: 10-calico.conflist -name: SLEEP value: "false" -name: CNI_NET_DIR value: /etc/cni/net.d -name: CNI_NETWORK_CONFIG valueFrom: con figMapKeyRef: key: config name: cni-config -name: KUBERNETES_SERVICE_HOST value: 10.96.0.1 -name: KUBERNETES_SERVICE_PORT value: "443" image: docker.io/calico/cni:vM.20.1 imagePullPolicy: IfNotPresent name: install-cni ``` ### Calico配置一览 ``` json "name": "kSs-pod-network", "cniVersion": "0.3.1", "plugins":[ { "type": "calico", "datastore_type": "kubernetes", "mtu": 0, "noden ame_file_opti on al": false, "log_level": "Info", "log_file_path": "/var/log/calico/cni/cni.log", :ype": "calico-ipam", "assign ipv4" : "true", a Ise"}, "container_settings": { "allow ip forwarding": false }, "policy": { "type": "k8s" }, "kubernetes": { "k8s_api_root":"https://10.96.0.1:443", "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } { "type": "bandwidth", "capabilities": {"bandwidth": true} }, {"type": "portmap", "snat": true, "capabilities": {"portMappings": true}} ``` ### Calico VXLan ![k8s_controller_manager_28](../img/k8s/k8s_controller_manager/k8s_controller_manager_28.jpg) ### IPPool IPPool用来定义一个集群的预定义IP段 apiVersion: crd.projectcalico.org/vl kind: IPPool ``` yaml metadata: name: default-ipv4-ippool spec: blocksize: 26 cidr: 192.168.0.0/16 ipipMode: Never natOutgoing: true nodeSelector: all() vxlanMode: CrossSubnet ``` ### IPAMBlock IPAMBLock用来定义每个主机预分配的IP段 ```yaml apiVersion: crd.projectcalico.org/vl kind: IPAMBlock metadata: anno tations: name: 192-168-119-64-26 spec: affinity: host:cadmin allocati ons: -null -0 -null -1 -2 -3 attributes: -handle_id: vxlan-tunnel-addr-cadmin secondary: no de: cadmin type: vxlanTunnelAddress -handle_id: k8s-pod-network.6680d3883d6150e75ffbd031f86c689a97a5be0f260c6442b2bb46b567c2ca40 secondary: namespace: calico-apiserver node: cadmin pod: calico-apiserver-77dffffcdf-g2tcx timestamp: 2021 -09-30 09:46:57.45651816 +0000 UTC -handle_id: k8s-pod-network.bl0d7702bf334fc55a5e399a731 ab3201 ea9990扪e?bc79894abddd712646699 secondary: namespace: calico-system node: cadmin pod: calico-kube-controllers-bdd5f97c5-554z5 timestamp: 2021 -09-30 09:46:57.502351 346 +0000 UTC ``` ### IPAMHandle IPAMHandle用来记录IP分配的具体细节 ``` yaml apiVersion: crd.projectcalico.org/vl kind: IPAMHancILe metadata: name: k8s-pod-network.8d75b941 d85c4998016b72c83f9c5a75512c82c052357daf0ec8e67365635d93 spec: block: 192.168.119.64/26： 1 deleted: false handlelD: k8s-pod-network.8d75b941 d85c4998016b72c83f9c5a75512c82c052357daf0ec8e67365635d93 ``` ### 创建Pod并查看IP配置情况 容器 namespace ``` shell nsenter -t 720702 -n ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1 /8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if27: vBROADCAST,MULTICAST,UP丄OWER_UPA mtu 1450 qdisc noqueue state UP group default link/ether f2:86:d2:4f:1f:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.119.84/32 brd 192.168.119.84 scope global ethO valid_lft forever preferred_lft forever nsenter -t 720702 -n ip r default via 169.254.1.1 dev ethO 169.254.1.1 dev ethO scope Link nsenter -t 720702 -n arp Address HWtype HWaddress Flags Mask Iface 169.254.1.1 ether ee:ee:ee:ee:ee:ee C ethO 10.0.2.15 ether ee:ee:ee:ee:ee:ee C ethO ``` 主机 Namespace ``` shell ip link 27: cali23a582ef038@if3: 《BROADCAST,MULTICAST,UP丄OWER_UP> mtu 1450 qdisc no queue state UP group default link/ether ee:ee:ee:ee:ee:ee brd link-netnsid 9 inet6 fe80::ecee:eeff:feee:eeee/64 scope link validjft forever preferred」ft forever ip route 192.168.119.84 dev cali23a582ef038 scope link ``` ### CNI Plugin的对比 ![k8s_controller_manager_29](../img/k8s/k8s_controller_manager/k8s_controller_manager_29.jpg) ## CSI ### 容器运行时存储 - 除外挂存储卷外，容器启动后，运行时所需文件 系统性能直接影响容器性能； - 早期的Docker采用Device Mapper作为容器 运行时存储驱动，因为0verlayFS尚未合并进 Kernel; - 目前Docker和containerd都默认以 OverlayFS作为运行时存储驱动； - OverlayFS目前已经有非常好的性能，与 DeviceMapper相比优20%,与操作主机文件 性能几乎一致。 ![k8s_controller_manager_30](../img/k8s/k8s_controller_manager/k8s_controller_manager_30.jpg) ### 存储卷插件管理 Kubernetes支持以插件的形式来实现对不同存储的支持和扩展，这些扩展基于如下三种方式: - in-tree 插件:Kubernetes社区已不再接受 新的in-tree存储插件，新的存储必须通过out-of-tree插件进行支持 - out-of-tree FlexVolume 插件:FlexVolume 是指 Kubernetes 通过调用计算节点的本地可执行文件与存储插件进行交互FlexVolume插件需要宿主机用root权限来安装插件驱动FlexVolume存储驱动需要宿主机安装attach、mount等工具，也需要具有root访问权限。 - out-of-tree CSI插件 #### out-of-tree CSI插件 - CSI通过RPC与存储驱动进行交互。 - 在设计CSI的时候，Kubernetes对CSI存储驱动的打包和部署要求很少，主要定义了 Kubernetes的两个相关模块： - kube-controller-manager : - kube-controller-manager模块用于感知CSI驱动存在。 - Kubernetes的主控模块通过Unix domain socket (而不是CSI驱动)或者其他方式进行直接地 交互。 - Kubernetes的主控模块只与Kubernetes相关的API进行交互。 - 因此CSI驱动若有依赖于Kubernetes API的操作，例如卷的创建、卷的attach、卷的快照等，需要在CSI驱动里面通过Kubernetes的API,来触发相关的CSI操作。 - kubelet: - kubelet模块用于与CSI驱动进行交互。 - kubelet 通过 Unix domain socket 向 CSI 驱动发起 CSI 调用(如 NodeStageVolume、NodePublishVolume等),再发起 mount 卷和 umount 卷。 - kubelet通过插件注册机制发现CSI驱动及用于和CSI驱动交互的Unix Domain Socket。 - 所有部署在Kubernetes集群中的CSI驱动都要通过kubelet的插件注册机制来注册自己。 ### CSI驱动 CSI 的驱动一般包含 externaL-attacher、external-provisioner、external-resizer、 external- snapshotter、node-driver-register、CSI driver等模块，可以根据实际的存储类型和需求进行不同方式的部署。 ![k8s_controller_manager_31](../img/k8s/k8s_controller_manager/k8s_controller_manager_31.jpg) ### 临时存储 常见的临时存储主要就是emptyDir卷。 emptyDir是一种经常被用户使用的卷类型，顾名思义，〃卷〃最初是空的。当Pod从节点上删除时’ emptyDir卷中的数据也会被永久删除。但当Pod的容器因为某些原因退出再重启时，emptyDir卷内的数据并不会丢失。 默认情况下，emptyDir卷存储在支持该节点所使用的存储介质上，可以是本地磁盘或网络存储。 emptyDir也可以通过将emptyDir.medium字段设置为"Memory"来通知Kubernetes为容器安装tmpfs,此时数据被存储在内存中，速度相对于本地存储和网络存储快很多。但是在节点重启的时候，内存数据会被清除；而如果存在磁盘上，则重启后数据依然存在。另外，使用tmpfs的内存也会计入容器的使用内存总量中，受系统的Cgroup限制。 emptyDir设计的初衷主要是给应用充当缓存空间，或者存储中间数据，用于快速恢复。然而,这并不 是说满足以上需求的用户都被推荐使用emptyDir,我们要根据用户业务的实际特点来判断是否使用 emptyDir。因为emptyDir的空间位于系统根盘，被所有容器共享’所以在磁盘的使用率较高时会触 发Pod的eviction操宿 从而影响业务的稳定。 ### 半持久化存储 常见的半持久化存储主要是hostPath卷。hostPath卷能将主机节点文件系统上的文件或目录挂载到指 定Pod中。对普通用户而言一般不需要这样的卷，但是对很多需要获取节点系统信息的Pod而言，却 是非常必要的。 例如，hostPath的用法举例如下： - 某个Pod需要获取节点上所有Pod的tog7可以通过hostPath访问所有Pod的stdout输出 存储目录’例如/var/tog/pods路径。 - 某个Pod需要统计系统相关的信息’可以通过hostPath访问系统的/proc目录。 使用hostPath的时候，除设置必需的path属性外，用户还可以有选择性地为hostPath卷指定类型, 支持类型包含目录、字符设备、块设备等。 ### hostPath卷需要注意 使用同一个目录的Pod可能会由于调度到不同的节点，导致目录中的内容有所不同。 Kubernetes在调度时无法顾及由hostPath使用的资源。 Pod被删除后，如果没有特别处理，那么hostPath±写的数据会遗留到节点上，占用磁盘空间。 ### 持久化存储 支持持久化的存储是所有分布式系统所必备的特性。针对持久化存储，Kubernetes引入了 Storageclass、Volume、PVC (Persistent Volume Claim) 、PV (Persitent Volume)的概念，将存储独立于Pod的生命周期来进行管理。 Kuberntes目前支持的持久化存储包含各种主流的块存储和文件存储，譬如awsElasticBlockStore、azureDisk、cinder、 NFS、cephfs、 iscsi等，在大类上可以将其分为网络存储和本地存储两种类型。 ### StorageCLass Storageclass用于指示存储的类型，不同的存储类型可以通过不同的StorageCLass来为用户提供服务。 Storageclass主要包含存储插件provisioned、卷的创建和mount参数等字段。 ``` yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: Storageclass metadata: annotations: storageclass.kubernetes.io/is-default-class: "false" name: rook-ceph-block parameters: clusterlD: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/no de-stage-secret-n amespace: rook-ceph csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph imageFeatures: layering imageFormat: "2" pool: replicapool provisioner: rook-ceph.rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate ``` ### PVC 由用户创建，代表用户对存储需求的声明，主要包含需要的存储大小、存储卷的访问模式、 StroageClass等类型，其中存储卷的访问模式必须与存储的类型一致 ![k8s_controller_manager_32](../img/k8s/k8s_controller_manager/k8s_controller_manager_32.jpg) ### PV 由集群管理员提前创建’或者根据PVC的申请需求动态地创建,它代表系统后端的真实的存储空间, 可以称之为卷空间。 ### 存储对象关系 用户通过创建PVC来申请存储。控制器通过PVC的storageclass和请求的大小声明来存储后端创建 卷，进而创建PV, Pod通过指定PVC来引用存储。 ![k8s_controller_manager_33](../img/k8s/k8s_controller_manager/k8s_controller_manager_33.jpg) ### 生产实践经验分享 不同介质类型的磁盘'需要设置不同的Storageclass,以便让用户做区分。Storageclass需要设置磁盘介质的类 型，以便用户了解该类存储的属性。 在本地存储的PV静态部署模式下，每个物理磁盘都尽量只创建一个PV,而不是划分为多个分区来提供多个本地存 储PV,避免在使用时分区之间的I/O干扰。 本地存储需要配合磁盘检测来使用。当集群部署规模化后，每个集群的本地存储PV可能会超过几万个，如磁盘损坏 将是频发事件。此时，需要在检测到磁盘损坏、丢盘等问题后，对节点的磁盘和相应的本地存储PV进行特定的处理, 例如触发告警、自动cordon节点、自动通知用户等。 对于提供本地存储节点的磁盘管理，需要做到灵活管理和自动化。节点磁盘的信息可以归一、集中化管理。在 local-volume-provisioner中增加部署逻辑，当容器运行起来时，拉取该节点需要提供本地存储的磁盘信息，例如 磁盘的设备路径，以Filesystem或Block的模式提供本地存储，或者是否需要加入某个LVM的虚拟组(VG)等。 local-volume-provisioner根据获取的磁盘信息对磁盘进行格式化，或者加入到某个VG,从而形成对本地存储支 持的自动化闭环。 ### 独占的 Local Volume - 创建 PV:通过Local-volume-provisioner DaemonSet 创建本地存储 的PV。 - 创建PVC:用户创建PVC,由于它处于pending状态，所以kube-controller-manager并不会对该PVC做任何操作。 - 创建Pod:用户创建Pod。 - Pod挑选节点:kube-scheduler开始调度Pod,通过PVC的 resources.request.storage 和 volumeMode 选择满足条件的 PV, 并且为Pod选择一个合适的节点。 - 更新 PV: kube-scheduler 将 PV 的 pv.Spec.claimRef 设置为对应的 PVC,并且设置 dnnotdtion pv.kubernetes.io/bound-by- controller 的值为"yes"。 - PVC和PV绑定:pv_controller同步PVC和PV的状态，并将PVC和 PV进行绑定。 - 监听PVC对象：kube-scheduler等待PVC的状态变成Bound状态。 - Pod调度到节点：如果PVC的状态变为Bound则说明调度成功，而 如果PVC 一直处于pending状态，超时后会再次进行调度。 - Mount卷启动容器：kubelet监听到有Pod已经调度到节点上，对本 地存储进行mount操作，并启动容器。 ![k8s_controller_manager_34](../img/k8s/k8s_controller_manager/k8s_controller_manager_34.jpg) ### Dynamic Local Volume CSI驱动需要汇报节点上相关存储的资源信息，以便用于调度 但是机器的厂家不同，汇报方式也不同。 例如，有的厂家的机器节点上具有NVMe、SSD、HDD等多种存储介质，希望将这些存储介质分别进 行汇报。 这种需求有别于其他存储类型的CSI驱动对接口的需求，因此如何汇报节点的存储信息，以及如何让节 点的存储信息应用于调度，目前并没有形成统一的意见。 集群管理员可以基于节点存储的实际情况对开源CSI驱动和调度进行一些代码修改,再进行部署和使用 ### Local Dynamic的挂载流程 - 创建PVC:用户创建PVC, PVC处于pending状态。 - 创建Pod:用户创建Pod。 - Pod选择节点：kube-scheduLer开始调度Pod,通过PVC的 pvc.spec.resources.request.storage 等选择满足条件的节点。 - 更新PVC:选择节点后，kube-scheduler会给PVC添加包含节 点信息的 annotation: volume.kubernetes.io/selected- node: ＜节点名字〉。 - 创建卷：运行在节点上的容器external-provisioner监听到 PVC带有该节点相关的annotation,向相应的CSI驱动申请分 配卷。 - 创建PV: PVC申请到所需的存储空间后，external-provisioner 创建PV,该PV的pv.Spec.claimRef设置为对应的PVCO - PVC和PV绑定：kube-controller-manager 将 PVC 和 PV 进行 绑定，状态修改为Bound0 - 监听PVC状态：kube-scheduler等待PVC变成Bound状态。 - Pod调度到节点：当PVC的状态为Bound时，Pod才算真正调 度成功了。如果PVC-直处于Pending状态，超时后会再次进 行调度。 - Mount卷：kubelet监听到有Pod已经调度到节点上，对本地存 储进行mount操作。 - 启动容器：启动容器。 ![k8s_controller_manager_35](../img/k8s/k8s_controller_manager/k8s_controller_manager_35.jpg) ### Local Dynamic 的挑战 如果将磁盘空间作为一个存储池（例如LVM）来动态分配，那么在分配出来的逻辑卷空间的使用上, 可能会受到其他逻辑卷的I/O干扰，因为底层的物理卷可能是同一个。 如果PV后端的磁盘空间是一块独立的物理磁盘，则I/O就不会受到干扰。 ### 生产实践经验分享 不同介质类型的磁盘，需要设置不同的StorageCLass,以便让用户做区分。Storageclass需要设置磁 盘介质的类型，以便用户了解该类存储的属性。 在本地存储的PV静态部署模式下，每个物理磁盘都尽量只创建一个PV,而不是划分为多个分区来提供 多个本地存储PV,避免在使用时分区之间的I/O干扰。 本地存储需要配合磁盘检测来使用。当集群部署规模化后’每个集群的本地存储PV可能会超过几万个, 如磁盘损坏将是频发事件。此时，需要在检测到磁盘损坏、丢盘等问题后，对节点的磁盘和相应的本地 存储PV进行特定的处理，例如触发告警、自动cordon节点、自动通知用户等。 对于提供本地存储节点的磁盘管理，需要做到灵活管理和自动化。节点磁盘的信息可以归一、集中化管 理。在local-volume-provisioner中增加部署逻辑’当容器运行起来时，拉取该节点需要提供本地存 储的磁盘信息，例如磁盘的设备路径，以Filesystem或Block的模式提供本地存储，或者是否需要加 入某个LVM的虚拟组(VG)等。local-volume-provisioner根据获取的磁盘信息对磁盘进行格式化， 或者加入到某个VG,从而形成对本地存储支持的自动化闭环。 ### Rook Rook是一款云原生环境下的开源分布式存储编排系统，目前支持Ceph、NFS、EdgeFS. Cassandra. CockroachDB等存储系统。它实现了一个自动管理的、自动扩容的、自动修复的分布 式存储服务。Rook支持自动部署、启动、配置、分配、扩容/缩容、升级、迁移、灾难恢复、监控以 及资源管理。 ### Rook 构架 ![k8s_controller_manager_36](../img/k8s/k8s_controller_manager/k8s_controller_manager_36.jpg) ### Rook Operator Rook Operater 是 Rook 的大脑，以 deployment 形式存在。 其利用Kubernetes的controller-runtime框架实现了 CRD,并进而接受Kubernetes创建资源的请 求并创建相关资源（集群，pool,块存储服务，文件存储服务等）。 Rook Operater监控存储守护进程，来确保存储集群的健康。 监听Rook Discovers收集到的存储磁盘设备，并创建相应服务（Ceph的话就是OSD 了）。 ### Rook Discover Rook Discover是以DaemonSet形式部署在所有的存储机上的，其检测挂接到存储节点上的存储设 备。把符合要求的存储设备记录下来，这样Rook Operate感知到以后就可以基于该存储设备创建相应 服务了。 ``` shell ## discover device $ Isblk --all --noheadings --list --output KNAME $ Isblk /dev/vdd --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME $ udevadm info --query二property /dev/vdd $ Isblk —noheadings —pairs /dev/vdd ## discover ceph inventory $ ceph-volume inventory —format json if device has ceph inv, device.CephVolumeData = CVData ## put device info into con figmap per node ``` ### CSIDriver 发现 CSI驱动发现： 如果一个CSI驱动创建CSIDriver对象'Kubernetes用户可以通过get CSIDriver命令发现它们; CSI对象有如下特点： - 自定义的Kubernetes逻辑； - Kubernetes对存储卷有一些列操作，这些CSIDriver可以自定义支持哪些操作？ ### Provisioner CSI external-provisioner 是一个监控 Kubernetes PVC 对象的 Sidecar 容器。 当用户创建PVC后，Kubernetes会监测PVC对应的Storageclass,如果Storageclass中的provisioner与某 插件匹配，该容器通过CSI Endpoint （通常是unix socket）调用CreateVolume方法。 如果 CreateVolume 方法调用成功,则 Provisioner sidecar 创建 Kubernetes PV 对象。 ### CSI External Provisioner ```yaml containers: -args: - --csi-address=$(ADDRESS) - --v=0 - --timeout=150s - --retry-i ntervdL-st provisionClaimOperation-> provisioner.Provision pkg/operator/ceph/provisioner/provisioner.go Provision-> createVolume ceph.Createlmage pkg/daem on/ceph/clie nt/image.go rbd create poolName/ndme -size sizeMB volume Sto re. Sto reVolume controller/volume_store.go doSaveVolume client.CoreV1().PersistentVolumes().Create(volume) ``` ### Provisioner log ``` log 10816 10:17:32.535207 1 connection.go:153] Connecting to unix:///csi/csi-provisi on er.sock 10816 10:17:54.361911 1 volume_store.go:97] Starting save volume queue 10816 10:17:54.461930 1 controller.go:!284] provision "defauIt/mysqI-pv-cLaim" class"rook-ceph-block": started 10816 10:17:54.462078 1 controller.go:848] Started provisi on er controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-677577c77c-vwkzz_ca5971ab-2293-4e52- 9bc9-c490f7f50b07! - ~ 10816 10:17:54.465752 1 event.go:281]Event(v1 .ObjectReferencefKind/'PersistentVolumeClaim", Namespace:"default", Name:"mysql-pv-claim"y UID:"24449707-6738-425c-ac88-de3c470cf91 a"z APIVersion:"v1", Resourceversion:"45668", FieldPath:""})： type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "defauIt/mysqI-pv-claim" ``` ### Rook Agent Rook Agent是以DaemonSet形式部署在所有的存储机上的’其处理所有的存储操作’例如挂卸载存 储卷以及格式化文件系统等。 ### CSI插件注册 ``` yaml spec: hostNetwork: true hostPID: true -args: - --v=0 - --csi-address=/csi/csi.sock - --kubelet-registrati on-pdth=/vdr/lib/kubeLet/plugins/ro ok-ceph. rbd.csi.ceph.com/csi.sock image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0 name: driver-registrar resources: {} securitycontext: privileged: true volumeM oun ts: -mountPath: /csi name: plugin-dir -mountPath: /registration name: registration-dir ``` ### CSI Driver ```yaml apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: rook-ceph.rbd.csi.ceph.com spec: attachRequired: true podlnfoOnMount: false volumeLifecycleModes: -Persistent ls /var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com csi.sock -args: - --nodeid=$(NODE_ID) - --endpoint=$(CSlbDPOINT) - --v=0 - --type=rbd - --no deserver=true - --driver name=rook-ceph. rbd.csi.ceph.com - --pidlimit=-1 - --metricsport=9090 - --metricspath=/metrics - --en ablegrpcmetrics=true env: -name: CSI_ENDPOINT value: unix:///csi/csi.sock image: quay.io/cephcsi/cephcsi:v3.0.0 name: csi-rbdplugin securitycontext: allowPrivilegeEscalation: true capabilities: add: - SYS_ADMIN privileged: true -hostPath: path: /var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com type: DirectoryOrCreate name: plugin-dir ``` ### Agent ```go pkg/daem on/ceph/agent/agent.go flexvolume.NewController(a.context, volumeAttachmentController, volumeManager) rpc.Register(flexvolumeController) flexvolumeServer.Start ``` ### Cluster 针对不同ceph cluster, rook启动一组管理 组件，包括： mon, mgr, osd, mds, rgw ``` yaml apiVersion: ceph.rook.io/vl kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: ceph/ceph:vl4.2.10 dataDirHostPath: /var/lib/rook mon: count: 3 allowMultiplePerNode: false mgr: modules: -name: pg_autoscaler enabled: true dashboard: enabled: true storage: useAUNodes: true useAUDevices: true ``` ### Pool —个ceph cluster可以有多个pool, 定义副本数量，故障域等多个属性。 ```yaml apiVersion: ceph.rook.io/vl kind: CephBlockPool metadata: name: replicapool namespace: rook-ceph spec: compressionMode: "" crushRoot: "" deviceclass: “” erasureCoded: algorithm: “” codingChunks: 0 dataChunks: 0 failureDomain: host replicated: requireSafeReplicaSize: false size: 1 targetSizeRatio: 0 status: phase: Ready ``` ### Storage Class Storageclass是Kubernetes用来自动创建PV的对象。 ```yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: Storageclass name: rook-ceph-block parameters: clusterlD: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph imageFeatures: layering imageFormat: "2" pool: replicapool provisioner: rook-ceph.rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate ``` [参考资料](https://dramasamy.medium.com/life-of-a-packet-i n-kuber netes-p3rt-2-307f5bf0ff14) ================================================ FILE: docs/kubernetes/k8s_etcd.md ================================================ # Kubernetes 控制平面组件：etcd ## etcd Etcd是CoreOS基于Raft开发的分布式key-value存储，可用于服务发现、共享配置以及一致性 保障（如数据库选主、分布式锁等）。 在分布式系统中，如何管理节点间的状态一直是一个难题，etcd像是专门为集群环境的服务发现 和注册而设计，它提供了数据TTL失效、数据改变监视、多值、目录监听、分布式锁原子操作等 功能，可以方便的跟踪并管理集群节点的状态。 - 键值对存储：将数据存储在分层组织的目录中，如同在标准文件系统中 - 监测变更：监测特定的键或目录以进行更改，并对值的更改做出反应 - 简单:curl可访问的用户的API（HTTP+JSON） - 安全: 可选的SSL客户端证书认证 - 快速: 单实例每秒 1000 次写操作，2000+次读操作 - 可靠: 使用Raft算法保证一致性 ### 主要功能 - 基本的key-value存储 - 监听机制 - key的过期及续约机制，用于监控和服务发现 - 原子Compare And Swap和Compare And Delete，用于分布式锁和leader选举 ### 使用场景 - 也可以用于键值对存储，应用程序可以读取和写入etcd中的数据 - etcd比较多的应用场景是用于服务注册与发现 - 基于监听机制的分布式异步系统 ### 键值对存储 etcd是一个 **键值存储** 的组件，其他的应用都是基于其键值存储的功能展开。 - 采用kv型数据存储，一般情况下比关系型数据库快。 - 支持动态存储(内存)以及静态存储(磁盘)。 - 分布式存储，可集成为多节点集群。 - 存储方式，采用类似目录结构。（B+tree） - 只有叶子节点才能真正存储数据，相当于文件。 - 叶子节点的父节点一定是目录，目录不能存储数据。 ### 服务注册与发现 - 强一致性、高可用的服务存储目录。 - 基于Raft 算法的etcd 天生就是这样一个强一致性、高可用的服务存储目录。 - 一种注册服务和服务健康状况的机制。 - 用户可以在etcd中注册服务，并且对注册的服务配置 key TTL，定时保持服务的心跳以达 到监控健康状态的效果。 ### 消息发布与订阅 - 在分布式系统中，最适用的一种组件间通信方式就是消息发布与订阅。 - 即构建一个配置共享中心，数据提供者在这个配置中心发布消息，而消息使用者则订阅他们 关心的主题，一旦主题有消息发布，就会实时通知订阅者。 - 通过这种方式可以做到分布式系统配置的集中式管理与动态更新。 - 应用中用到的一些配置信息放到etcd上进行集中管理。 - 应用在启动的时候主动从etcd获取一次配置信息，同时，在etcd节点上注册一个Watcher并 等待，以后每次配置有更新的时候，etcd都会实时通知订阅者，以此达到获取最新配置信息 的目的。 ### Etcd的安装 下载安装包，参考https://github.com/etcd-io/etcd/releases - ETCD_VER=v3.4. - DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download - rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz - rm -rf /tmp/etcd-download-test && mkdir-p /tmp/etcd-download-test - curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}- linux-amd64.tar.gz - tar xzvf/tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components= - rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz 更多信息 https://github.com/cncamp/101/blob/master/module5/1.etcd-member-list.MD ### 第三方库和客户端工具 目前有很多支持etcd的库和客户端工具 - 命令行客户端工具etcdctl - Go客户端go-etcd - Java客户端jetcd - Python客户端python-etcd ### etcd 命令 查看集群成员状态 etcdctlmember list --write-out=table +------------------+---------+---------+-----------------------+-----------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+---------+-----------------------+-----------------------+------------+ | 8e9e05c52164694d | started | default | [http://localhost:2380](http://localhost:2380) | [http://localhost:2379](http://localhost:2379) | false | +------------------+---------+---------+-----------------------+-----------------------+------------+ 基本的数据读写操作 - 写入数据 etcdctl--endpoints=localhost:12379 put /a b OK - 读取数据 etcdctl--endpoints=localhost:12379 get /a /a b - 按key的前缀查询数据 etcdctl--endpoints=localhost:12379 get --prefix / - 只显示键值 etcdctl--endpoints=localhost:12379 get --prefix / --keys-only--debug ### 核心：TTL & CAS TTL（timeto live）指的是给一个key设置一个有效期，到期后这个key就会被自动删掉，这在 很多分布式锁的实现上都会用到，可以保证锁的实时有效性。 Atomic Compare-and-Swap（CAS）指的是在对key进行赋值的时候，客户端需要提供一些条 件，当这些条件满足后，才能赋值成功。这些条件包括： - prevExist：key当前赋值前是否存在 - prevValue：key当前赋值前的值 - prevIndex：key当前赋值前的Index 这样的话，key的设置是有前提的，需要知道这个key当前的具体情况才可以对其设置。 ## Raft协议 ### Raft协议概览 Raft协议基于quorum机制，即大多数同意原则，任何的变更都需超过半数的成员确认 ### 理解Raft协议 [http://thesecretlivesofdata.com/raft/](http://thesecretlivesofdata.com/raft/) ### learner Raft 4.2.1引入的新角色 当出现一个etcd集群需要增加节点时，新节点与 Leader的数据差异较大，需要较多数据同步才能跟 上leader的最新的数据。 此时Leader的网络带宽很可能被用尽，进而使得 leader无法正常保持心跳。 进而导致follower重新发起投票。 进而可能引发etcd集群不可用。 Learner角色只接收数据而不参与投票，因此增加 learner节点时，集群的quorum不变。 ![k8s_etcd_1](../img/k8s/k8s_etcd/k8s_etcd_1.jpg) ### etcd基于Raft的一致性 选举方法 - 初始启动时，节点处于follower状态并被设定一个election timeout，如果在这一时间周期内没有收到 来自leader 的heartbeat，节点将发起选举：将自己切换为candidate 之后，向集群中其它follower 节点发送请求，询问其是否选举自己成为 leader。 - 当收到来自集群中过半数节点的接受投票后，节点即成为 leader，开始接收保存client 的数据并向其它 的follower 节点同步日志。如果没有达成一致，则candidate随机选择一个等待间隔（150ms ~ 300ms）再次发起投票，得到集群中半数以上follower接受的candidate将成为leader - leader节点依靠定时向 follower 发送heartbeat来保持其地位。 - 任何时候如果其它follower 在election timeout 期间都没有收到来自leader 的heartbeat，同样会将 自己的状态切换为candidate 并发起选举。每成功选举一次，新 leader 的任期（Term）都会比之前 leader 的任期大 1 。 ### 日志复制 当接Leader收到客户端的日志（事务请求）后先把该日志追加到本地的Log中，然后通过 heartbeat把该Entry同步给其他Follower，Follower接收到日志后记录日志然后向Leader发送 ACK，当Leader收到大多数（n/2+1）Follower的ACK信息后将该日志设置为已提交并追加到 本地磁盘中，通知客户端并在下个heartbeat中Leader将通知所有的Follower将该日志存储在自 己的本地磁盘中。 ### 安全性 安全性是用于保证每个节点都执行相同序列的安全机制，如当某个Follower在当前Leader commit Log时变得不可用了，稍后可能该Follower又会被选举为Leader，这时新Leader可能 会用新的Log覆盖先前已committed的Log，这就是导致节点执行不同序列；Safety就是用于保 证选举出来的Leader一定包含先前committed Log的机制； 选举安全性（Election Safety）：每个任期（Term）只能选举出一个Leader Leader完整性（Leader Completeness）：指Leader日志的完整性，当Log在任期Term1被 Commit后，那么以后任期Term2、Term3...等的Leader必须包含该Log；Raft在选举阶段就使 用Term的判断用于保证完整性：当请求投票的该Candidate的Term较大或Term相同Index更大 则投票，否则拒绝该请求。 ### 失效处理 1) Leader失效：其他没有收到heartbeat的节点会发起新的选举，而当Leader恢复后由于步进 数小会自动成为follower（日志也会被新leader的日志覆盖） 2 ）follower节点不可用：follower 节点不可用的情况相对容易解决。因为集群中的日志内容始 终是从leader 节点同步的，只要这一节点再次加入集群时重新从 leader 节点处复制日志即可。 3 ）多个candidate：冲突后candidate将随机选择一个等待间隔（150ms ~ 300ms）再次发起 投票，得到集群中半数以上follower接受的candidate将成为leader ### wal日志 wal日志是二进制的，解析出来后是以上数据结构LogEntry。其中第一个字段type，只有两种， 一种是 0 表示Normal， 1 表示ConfChange（ConfChange表示 Etcd本身的配置变更同步，比 如有新的节点加入等）。第二个字段是term，每个term代表一个主节点的任期，每次主节点变 更term就会变化。第三个字段是index，这个序号是严格有序递增的，代表变更序号。第四个字 段是二进制的data，将raft request对象的pb结构整个保存下。etcd 源码下有个tools/etcd- dump-logs，可以将wal日志dump成文本查看，可以协助分析Raft协议。 Raft协议本身不关心应用数据，也就是data中的部分，一致性都通过同步wal日志来实现，每个 节点将从主节点收到的data apply到本地的存储，Raft只关心日志的同步状态，如果本地存储实 现的有bug，比如没有正确的将data apply到本地，也可能会导致数据不一致。 ## etcd v3 存储，Watch以及过期机制 ![k8s_etcd_2](../img/k8s/k8s_etcd/k8s_etcd_2.jpg) ### 存储机制 etcdv3 store 分为两部分，一部分是内存中的索引，kvindex，是基于Google开源的一个 Golang的btree实现的，另外一部分是后端存储。按照它的设计，backend可以对接多种存储， 当前使用的boltdb。boltdb是一个单机的支持事务的kv存储，etcd 的事务是基于boltdb的事务 实现的。etcd 在boltdb中存储的key是reversion，value是 etcd 自己的key-value组合，也就 是说etcd 会在boltdb中把每个版本都保存下，从而实现了多版本机制。 reversion主要由两部分组成，第一部分main rev，每次事务进行加一，第二部分sub rev，同一 个事务中的每次操作加一。 etcd 提供了命令和设置选项来控制compact，同时支持put操作的参数来精确控制某个key的历 史版本数。 内存kvindex保存的就是key和reversion之前的映射关系，用来加速查询。 ![k8s_etcd_3](../img/k8s/k8s_etcd/k8s_etcd_3.jpg) ### Watch机制 etcdv3 的watch机制支持watch某个固定的key，也支持watch一个范围（可以用于模拟目录的 结构的watch），所以 watchGroup包含两种watcher，一种是 key watchers，数据结构是每 个key对应一组watcher，另外一种是 range watchers, 数据结构是一个IntervalTree，方便通 过区间查找到对应的watcher。 同时，每个WatchableStore 包含两种 watcherGroup，一种是synced，一种是unsynced， 前者表示该group的watcher数据都已经同步完毕，在等待新的变更，后者表示该group的 watcher数据同步落后于当前最新变更，还在追赶。 当etcd 收到客户端的watch请求，如果请求携带了revision参数，则比较请求的revision和 store当前的revision，如果大于当前revision，则放入synced组中，否则放入unsynced组。同 时etcd会启动一个后台的goroutine持续同步unsynced的watcher，然后将其迁移到synced组。 也就是这种机制下，etcd v3 支持从任意版本开始watch，没有v2的 1000 条历史event表限制的 问题（当然这是指没有compact的情况下） ## etcd 命令 查看集群成员状态 etcdctlmember list --write-out=table +------------------+---------+---------+-----------------------+-----------------------+----- -------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+---------+-----------------------+-----------------------+----- -------+ | 8e9e05c52164694d | started | default | [http://localhost:2380](http://localhost:2380) | [http://localhost:2379](http://localhost:2379) | false | +------------------+---------+---------+-----------------------+-----------------------+----- -------+ - 启动新etcd集群 docker run -d registry.aliyuncs.com/google_containers/etcd:3.5.0-0 /usr/local/bin/etcd - 进入etcd容器 dockerps|grepetcd docker exec –it sh - 存入数据 etcdctlput x 0 - 读取数据 etcdctlget x -w=json {"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"r evision":2,"raft_term":2},"kvs":[{"key":"eA==","create_revision":2,"mod_revision":2,"versi on":1,"value":"MA=="}],"count":1} - 修改值 etcdctlput x 1 - 查询最新值 etcdctlget x x 1 - 查询历史版本值 etcdctlget x --rev=2 x 0 ## etcd 成员重要参数 成员相关参数 ``` --name 'default' Human-readable name for this member. --data-dir'${name}.etcd' Path to the data directory. --listen-peer-urls 'http://localhost:2380' List of URLs to listen on for peer traffic. --listen-client-urls 'http://localhost:2379' List of URLs to listen on for client traffic. ``` 集群相关参数 ``` --initial-advertise-peer-urls 'http://localhost:2380' List of this member's peer URLs to advertise to the rest of the cluster. --initial-cluster 'default=http://localhost:2380' Initial cluster configuration for bootstrapping. --initial-cluster-state 'new' Initial cluster state ('new' or 'existing'). --initial-cluster-token 'etcd-cluster' Initial cluster token for the etcd cluster during bootstrap. --advertise-client-urls 'http://localhost:2379' List of this member's client URLs to advertise to the public. ``` etcd安全相关参数 ``` --cert-file '' Path to the client server TLS cert file. --key-file '' Path to the client server TLS key file. --client-crl-file '' Path to the client certificate revocation list file. --trusted-ca-file '' Path to the client server TLS trusted CA cert file. --peer-cert-file '' Path to the peer server TLS cert file. --peer-key-file '' Path to the peer server TLS key file. --peer-trusted-ca-file '' Path to the peer server TLS trusted CA file. ``` ## 灾备 - 创建Snapshot ``` etcdctl--endpoints https://127.0.0.1:3379 --cert /tmp/etcd-certs/certs/127.0.0.1.pem -- key /tmp/etcd-certs/certs/127.0.0.1-key.pem --cacert/tmp/etcd-certs/certs/ca.pem snapshot save snapshot.db ``` - 恢复数据 ``` etcdctlsnapshot restore snapshot.db\ --name infra2 \ --data-dir=/tmp/etcd/infra2 \ --initial-cluster infra0=http://127.0.0.1:3380,infra1=http://127.0.0.1:4380,infra2=http://127.0.0.1:5380 \ --initial-cluster-token etcd-cluster- 1 \ --initial-advertise-peer-urls http://127.0.0.1:5380 ``` ## 容量管理 单个对象不建议超过1.5M 默认容量2G 不建议超过8G ## Alarm & Disarm Alarm - 设置etcd存储大小 $ etcd--quota-backend-bytes =$((16 * 1024 * 1024)) - 写爆磁盘 $ while [ 1 ]; do dd if=/dev/urandombs=1024 count=1024 | ETCDCTL_API=3 etcdctlput key || break; done - 查看endpoint状态 $ ETCDCTL_API **=3** etcdctl--write-out **=** table endpoint status - 查看alarm $ ETCDCTL_API **=3** etcdctlalarm list - 清理碎片 $ ETCDCTL_API **=3** etcdctldefrag - 清理alarm $ ETCDCTL_API **=3** etcdctlalarm disarm - 碎片整理 //keep one hour of history $ etcd --auto-compaction-retention=1 //compact up to revision 3 $ etcdctlcompact 3 $ etcdctl defrag Finished defragmenting etcd member [127.0.0.1:2379] ## 高可用etcd解决方案 etcd-operator: coreos开源的，基于kubernetes CRD完成etcd集群配置。Archived https://github.com/coreos/etcd-operator Etcdstatefulset Helm chart: Bitnami(powered by vmware) https://bitnami.com/stack/etcd/helm https://github.com/bitnami/charts/blob/master/bitnami/etcd ### Etcd Operator ![k8s_etcd_4](../img/k8s/k8s_etcd/k8s_etcd_4.jpg) https://github.com/coreos/etcd-operator.git ### 基于 Bitnami 安装etcd高可用集群 - 安装helm https://github.com/helm/helm/releases - 通过helm安装etcd helm repo add bitnamihttps://charts.bitnami.com/bitnami helm install my-release bitnami/etcd - 通过客户端与serve交互 kubectlrun my-release-etcd-client --restart='Never' --image docker.io/bitnami/etcd:3.5.0-debian- 10 - r94 --env ROOT_PASSWORD=$(kubectlget secret --namespace default my-release-etcd -o jsonpath="{.data.etcd-root-password}" | base64 --decode) --env ETCDCTL_ENDPOINTS="my-release- etcd.default.svc.cluster.local:2379" --namespace default --command --sleep infinity ## Kubernetes如何使用etcd etcd是kubernetes的后端存储 对于每一个kubernetesObject，都有对应的storage.go负责对象的存储操作 - pkg/registry/core/pod/storage/storage.go API server 启动脚本中指定etcdservers集群 ```yaml spec: containers: - command: - kube-apiserver - --advertise-address=192.168.34.2 - --enable-bootstrap-token-auth=true - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 ``` 早期APIserver 对etcd做简单的Ping check，现在已经改为真实的etcd api call ## Kubernets对象在etcd中的存储路径 - ksexec -it etcd-cadmin sh - ETCDCTL_API=3 - alias ectl='etcdctl --endpoints https://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key - ectlget --prefix --keys-only / /registry/namespaces/calico-apiserver /registry/networkpolicies/calico-apiserver/allow-apiserver /registry/operator.tigera.io/tigerastatuses/apiserver /registry/pods/calico-apiserver/calico-apiserver-77dffffcdf-g2tcx /registry/pods/default/toolbox-68f79dd5f8-4664n ## etcd在集群中所处的位置 ![k8s_etcd_5](../img/k8s/k8s_etcd/k8s_etcd_5.jpg) ## Kubernetes如何使用etcd etcd是kubernetes的后端存储 对于每一个kubernetes Object，都有对应的storage.go 负责对象的存储操作 - pkg/registry/core/pod/storage/storage.go API server 启动脚本中指定etcd servers集群 - /usr/local/bin/kube-apiserver --etcd_servers=https://localhost:4001 --etcd- cafile=/etc/ssl/kubernetes/ca.crt--storage-backend=etcd3 --etcd-servers- overrides=/events#https:// localhost:4002 ### 堆叠式etcd集群的高可用拓扑 - 这种拓扑将相同节点上的控制平面和etcd成员耦合在一起。优点在于建立起来非常容易，并且 对副本的管理也更容易。但是，堆叠式存在耦合失败的风险。如果一个节点发生故障，则etcd 成员和控制平面实例都会丢失，并且集群冗余也会受到损害。可以通过添加更多控制平面节点 来减轻这种风险。因此为实现集群高可用应该至少运行三个堆叠的Master节点。 ![k8s_etcd_6](../img/k8s/k8s_etcd/k8s_etcd_6.jpg) ### 外部etcd集群的高可用拓扑 - 该拓扑将控制平面和etcd成员解耦。如果丢失一个Master节点，对etcd成员的影响较小，并 且不会像堆叠式拓扑那样对集群冗余产生太大影响。但是，此拓扑所需的主机数量是堆叠式拓 扑的两倍。具有此拓扑的群集至少需要三个主机用于控制平面节点，三个主机用于etcd集群。 ![k8s_etcd_7](../img/k8s/k8s_etcd/k8s_etcd_7.jpg) ### 实践 - etcd集群高可用 多少个peer最适合？ - 1 个？ 3 个？ 5 个？ - 保证高可用是首要目标 - 所有写操作都要经过leader - peer多了是否能提升集群并读操作的并发能力？ ➢ apiserver的配置只连本地的etcd peer ➢ apiserver的配置指定所有etcd peers，但只有当前连接的etcd member异常， apiserver才会换目标 - 需要动态flexup吗？ 保证apiserver和etcd之间的高效性通讯 - apiserver和etcd部署在同一节点 - apiserver和etcd之间的通讯基于gRPC ➢ 针对每一个object，apiserver和etcd之间的Connection -> stream共享 ➢ http2的特性 ➢ Streamquota ➢ 带来的问题？对于大规模集群，会造成链路阻塞 ➢ 10000 个pod，一次list操作需要返回的数据可能超过100M ➢ k get pod --all-namespaces|wc–l ➢ 8520 ➢ k get pod -oyaml--all-namespaces>pods ➢ ls -l pods ➢ -rw-r--r--1 root root 75339736 Apr 5 03:13 pods ### 实践 – etcd存储规划 - 本地vs远程？ ➢ RemoteStorage ➢ 优势是假设永远可用，现实真是如此吗？ ➢ 劣势是IO效率，可能带来的问题？ ➢ 最佳实践： ➢ LocalSSD ➢ 利用localvolume分配空间 - 多少空间？ ➢ 与集群规模相关，思考：为什么每个member的DB size不一致？ ### etcd 安全性 ➢ peer和peer之间的通讯加密 ➢ 是否有需求 ➢ TLS的额外开销 ➢ 运营复杂度增加 ➢ 数据加密 ➢ 是否有需求？ ➢ Kubernetes提供了针对secret的加密 ➢ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ 事件分离 - 对于大规模集群，大量的事件会对etcd造成压力 - API server 启动脚本中指定etcdservers集群 ➢ /usr/local/bin/kube-apiserver --etcd_servers=https://localhost:4001 --etcd- cafile=/etc/ssl/kubernetes/ca.crt--storage-backend=etcd3 --etcd-servers- overrides=/events#https://localhost:4002 如何监控? ### 减少网络延迟 减少网络延迟 - 数据中心内的RTT大概是数毫秒，国内的典型RTT约为50ms，两大洲之间的RTT可能慢至 400ms。因此建议etcd集群尽量同地域部署。 - 当客户端到Leader的并发连接数量过多，可能会导致其他Follower节点发往Leader的请求因 为网络拥塞而被延迟处理。在Follower节点上，可能会看到这样的错误： ➢ dropped MsgPropto 247ae21ff9436b2d since streamMsg'ssending buffer is full - 可以在节点上通过流量控制工具（Traffic Control）提高etcd成员之间发送数据的优先级来避 免。 ### 减少磁盘I/O延迟 对于磁盘延迟，典型的旋转磁盘写延迟约为 10 毫秒。对于SSD（Solid State Drives，固态硬 盘），延迟通常低于 1 毫秒。HDD（Hard Disk Drive，硬盘驱动器）或者网盘在大量数据读写 操作的情况下延时会不稳定。因此强烈建议使用SSD。 同时为了降低其他应用程序的I/O操作对etcd的干扰，建议将etcd的数据存放在单独的磁盘内。 也可以将不同类型的对象存储在不同的若干个etcd集群中，比如将频繁变更的event对象从主 etcd集群中分离出来，以保证主集群的高性能。在APIServer处这是可以通过参数配置的。这些 etcd集群最好也分别能有一块单独的存储磁盘。 如果不可避免地，etcd和其他的业务共享存储磁盘，那么就需要通过下面ionice命令对etcd服务 设置更高的磁盘I/O优先级，尽可能避免其他进程的影响。 ``` $ ionice -c2 -n0 -p 'pgrepetcd' ``` ### 保持合理的日志文件大小 etcd以日志的形式保存数据，无论是数据创建还是修改，它都将操作追加到日志文件，因此日志 文件大小会随着数据修改次数而线性增长。 当Kubernetes集群规模较大时，其对etcd集群中的数据更改也会很频繁，集群日记文件会迅速 增长。 为了有效降低日志文件大小，etcd会以固定周期创建快照保存系统的当前状态，并移除旧日志文 件。另外当修改次数累积到一定的数量（默认是 10000 ，通过参数“--snapshot-count”指 定），etcd也会创建快照文件。 如果etcd的内存使用和磁盘使用过高，可以先分析是否数据写入频度过大导致快照频度过高，确 认后可通过调低快照触发的阈值来降低其对内存和磁盘的使用。 ### 设置合理的存储配额 存储空间的配额用于控制etcd数据空间的大小。合理的存储配额可保证集群操作的可靠性。如果 没有存储配额，也就是etcd可以利用整个磁盘空间，etcd的性能会因为存储空间的持续增长而 严重下降，甚至有耗完集群磁盘空间导致不可预测集群行为的风险。如果设置的存储配额太小， 一旦其中一个节点的后台数据库的存储空间超出了存储配额，etcd就会触发集群范围的告警，并 将集群置于只接受读和删除请求的维护模式。只有在释放足够的空间、消除后端数据库的碎片和 清除存储配额告警之后，集群才能恢复正常操作。 ### 自动压缩历史版本 etcd会为每个键都保存了历史版本。为了避免出现性能问题或存储空间消耗完导致写不进去的问 题，这些历史版本需要进行周期性地压缩。压缩历史版本就是丢弃该键给定版本之前的所有信息， 节省出来的空间可以用于后续的写操作。etcd支持自动压缩历史版本。在启动参数中指定参数 “--auto-compaction”，其值以小时为单位。也就是etcd会自动压缩该值设置的时间窗口之 前的历史版本。 ### 定期消除碎片化 压缩历史版本，相当于离散地抹去etcd存储空间某些数据，etcd存储空间中将会出现碎片。这 些碎片无法被后台存储使用，却仍占据节点的存储空间。因此定期消除存储碎片，将释放碎片化 的存储空间，重新调整整个存储空间。 ### 数据备份 - 备份方案 ➢ 基于事件重放 ➢ etcd备份：备份完整的集群信息，灾难恢复 ➢ etcdctlsnapshot save - 频度？ ➢ 时间间隔太长： ➢ 能否接受userdatalost？ ➢ 如果有外部资源配置，如负载均衡等，能否接受数据丢失导致的leak？ ➢ 时间间隔太短： ➢ 对etcd的影响 ➢ 做snapshot的时候，etcd会锁住当前数据 ➢ 并发的写操作需要开辟新的空间进行增量写，导致磁盘空间增长 - 如何保证备份的时效性，同时防止磁盘爆掉？ ➢ Autodefrag？ ### 优化运行参数 当网络延迟和磁盘延迟固定的情况下，可以优化etcd运行参数来提升集群的工作效率。etcd基于Raft 协议进行Leader选举，当Leader选定以后才能开始数据读写操作，因此频繁的Leader选举会导致数 据读写性能显著降低。可以通过调整心跳周期（HeatbeatInterval）和选举超时时间（Election Timeout），来降低Leader选举的可能性。 心跳周期是控制Leader以何种频度向Follower发起心跳通知。心跳通知除表明Leader活跃状态之外， 还带有待写入数据信息，Follower依据心跳信息进行数据写入，默认心跳周期是100ms。选举超时时 间定义了当Follower多久没有收到Leader心跳，则重新发起选举，该参数的默认设置是1000ms。 如果etcd集群的不同实例部署在延迟较低的相同数据中心，通常使用默认配置即可。如果不同实例部 署在多数据中心或者网络延迟较高的集群环境，则需要对心跳周期和选举超时时间进行调整。建议心 跳周期参数推荐设置为接近etcd多个成员之间平均数据往返周期的最大值，一般是平均RTT的0.55- 1.5倍。如果心跳周期设置得过低，etcd会发送很多不必要的心跳信息，从而增加CPU和网络的负担。 如果设置得过高，则会导致选举频繁超时。选举超时时间也需要根据etcd成员之间的平均RTT时间来 设置。选举超时时间最少设置为etcd成员之间RTT时间的 10 倍，以便对网络波动。 心跳间隔和选举超时时间的值必须对同一个etcd集群的所有节点都生效，如果各个节点配置不同，就 会导致集群成员之间协商结果不可预知而不稳定。 ### etcd备份存储 etcd的默认工作目录下会生成两个子目录：wal和snap。wal是用于存放预写式日志，其最大的 作用是记录整个数据变化的全部历程。所有数据的修改在提交前，都要先写入wal中。 snap是用于存放快照数据。为防止wal文件过多，etcd会定期（当wal中数据超过 10000 条记录 时，由参数“--snapshot-count”设置）创建快照。当快照生成后，wal中数据就可以被删除了。 如果数据遭到破坏或错误修改需要回滚到之前某个状态时，方法就有两个：一是从快照中恢复数 据主体，但是未被拍入快照的数据会丢失；而是执行所有WAL中记录的修改操作，从最原始的 数据恢复到数据损坏之前的状态，但恢复的时间较长。 ### 备份方案实践 官方推荐etcd集群的备份方式是定期创建快照。和etcd内部定期创建快照的目的不同，该备份方式依赖外部 程序定期创建快照，并将快照上传到网络存储设备以实现etcd数据的冗余备份。上传到网络设备的数据，都 应进行了加密。即使当所有etcd实例都丢失了数据，也能允许etcd集群从一个已知的良好状态的时间点在任 一地方进行恢复。根据集群对etcd备份粒度的要求，可适当调节备份的周期。在生产环境中实测，拍摄快照 通常会影响集群当时的性能，因此不建议频繁创建快照。但是备份周期太长，就可能导致大量数据的丢失。 这里可以使用增量备份的方式。如图 3 - 8 所示，备份程序每 30 分钟触发一次快照的拍摄。紧接着它从快照结 束的版本（Revision）开始，监听etcd集群的事件，并每 10 秒钟将事件保存到文件中，并将快照和事件文件 上传到网络存储设备中。 30 分钟的快照周期对集群性能影响甚微。当大灾难来临时，也至多丢失 10 秒的数据。 至于数据修复，首先把数据从网络存储设备中下载下来，然后从快照中恢复大块数据，并在此基础上依次应 用存储的所有事件。这样就可以将集群数据恢复到灾难发生前。 ![k8s_etcd_8](../img/k8s/k8s_etcd/k8s_etcd_8.jpg) ## etcd 常见问题 ### 多少个peer最适合？ - 1 个？ 3 个？ 5 个？ - 保证高可用是首要目标 - 所有写操作都要经过leader - peer多了是否能提升集群并读操作的并发能力？ ➢ apiserver的配置只连本地的etcd peer ➢ apiserver的配置指定所有etcd peers，但只有当前连接的etcd member异常， apiserver才会换目标 - 需要动态flexup吗？ ### 保证apiserver和etcd之间的高效性通讯 保证apiserver和etcd之间的高效性通讯 - apiserver和etcd部署在同一节点 - apiserver和etcd之间的通讯基于gRPC ➢ 针对每一个object，apiserver和etcd之间的Connection -> stream共享 ➢ HTTP/2的特性 ➢ Streamquota ➢ 带来的问题？对于大规模集群，会造成链路阻塞 ➢ 10000 个pod，一次list操作需要返回的数据可能超过100M ➢ k get pod --all-namespaces|wc–l ➢ 8520 ➢ k get pod -oyaml--all-namespaces>pods ➢ ls -l pods ➢ -rw-r--r--1 root root 75339736 Apr 5 03:13 pods ## 增强版backup方案 ![k8s_etcd_9](../img/k8s/k8s_etcd/k8s_etcd_9.jpg) ## etcd数据加密 ➢ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ ``` yaml apiVersion: API Server.config.k 8 s.io/v 1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aesgcm: keys: - name: key 1 secret: c 2 VjcmV 0 IGlzIHNlY 3 VyZQ== - name: key 2 secret: dGhpcyBpcyBwYXNzd 29 yZA== - aescbc: keys: - name: key 1 secret: c 2 VjcmV 0 IGlzIHNlY 3 VyZQ== - secretbox: keys: - name: key 1 secret: YWJjZGVmZ 2 hpamtsbW 5 vcHFyc 3 R 1 dnd 4 eXoxMjM 0 NTY= - kms: name : myKmsPlugin endpoint: unix:///tmp/socketfile.sock cachesize: 100 ``` ## Kubernetes中数据分离 - 对于大规模集群，大量的事件会对etcd造成压力 - API server 启动脚本中指定etcd servers集群 /usr/local/bin/kube-apiserver--etcd-servers=https://localhost:4001 --etcd- cafile=/etc/ssl/kubernetes/ca.crt--storage-backend=etcd3 --etcd-servers- overrides=/events#https://localhost:4002 ## 查询APIServer 返回某namespace中的所有Pod GET /api/v1/namespaces/test/pods --- 200 OK Content-Type: application/json { "kind": "PodList", "apiVersion": "v1", "metadata": {"resourceVersion":"10245"}, "items": [...] } ## 从 12345 版本开始，监听所有对象变化 GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245 --- 200 OK Transfer-Encoding: chunked Content-Type: application/json { "type": "ADDED", "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...} } { "type": "MODIFIED", "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...} } ... ## 分页查询 GET /api/v1/pods?limit=500 --- 200 OK Content-Type: application/json { "kind": "PodList", "apiVersion": "v1", "metadata": { "resourceVersion":"10245", "continue": "ENCODED_CONTINUE_TOKEN", }, "items": [...] // returns pods 1- 500 } GET /api/v1/pods?limit=500&continue=ENCODED_CONTINUE_TOKEN --- 200 OK Content-Type: application/json { "kind": "PodList", "apiVersion": "v1", "metadata": { "resourceVersion":"10245", "continue": "ENCODED_CONTINUE_TOKEN_2", }, "items": [...] // returns pods 501- 1000 } ## ResourceVersion - 单个对象的resourceVersion - 对象的最后修改resourceVersion - List对象的resourceVersion - 生成listresponse时的resourceVersion - List行为 - List对象时，如果不加resourceVersion，意味着需要MostRecent数据，请求会击穿 APIServer缓存，直接发送至etcd - APIServer通过Label过滤对象查询时，过滤动作是在APIServer端，APIServer需要向 etcd发起全量查询请求 ## 遭遇到的陷阱 频繁的leaderelection etcd 分裂 etcd 不响应 与apiserver之间的链路阻塞 磁盘暴涨 ## 少数etcd 成员Down ![k8s_etcd_10](../img/k8s/k8s_etcd/k8s_etcd_10.jpg) ## Master节点出现网络分区 **Case: 网络分区出现** Group#1: master-1, master- 2 Group#2: master-3, master-4, master- 5 ![k8s_etcd_11](../img/k8s/k8s_etcd/k8s_etcd_11.jpg) ## 参考资料 B树和B+树 https://segmentfault.com/a/1190000020416577 ================================================ FILE: docs/kubernetes/k8s_fw_rule_and_object_design.md ================================================ # Kubernet ## Kubernetes 架构原则和对象设计 ### 什么是云计算 ![k8s_fw_rule_and_object_design_1](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_1.jpg) #### 云计算平台的分类 以Openstack为典型的虚拟化平台 - 虚拟机构建和业务代码部署分离。 - 可变的基础架构使后续维护风险变大。 以谷歌borg为典型的基于进程的作业调度平台 - 技术的迭代引发borg的换代需求。 - 早期的隔离依靠chrootjail实现，一些不合理的设计需要在新产品中改进。 - 对象之间的强依赖job和task是强包含关系，不利于重组。 - 所有容器共享IP，会导致端口冲突，隔离困难等问题。 - 为超级用户添加复杂逻辑导致系统过于复杂。 #### Kubernetes 架构基础 ![k8s_fw_rule_and_object_design_2](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_2.jpg) #### Google Borg 简介 特性 - 物理资源利用率高。 - 服务器共享，在进程级别做隔离。 - 应用高可用，故障恢复时间短。 - 调度策略灵活。 - 应用接入和使用方便，提供了完备的Job描述语言，服务发现，实时状态监控和诊断工具。 优势 - 对外隐藏底层资源管理和调度、故障处理等。 - 实现应用的高可靠和高可用。 - 足够弹性，支持应用跑在成千上万的机器上。 ##### 基本概念 Workload - prod：在线任务，长 期运行、对延时敏感、 面向终端用户等，比如 Gmail, Google Docs, WebSearch服务等。 - non-prod ：离线任 务，也称为批处理任务 （Batch），比如一些 分布式计算服务等。 Cell - 一个 Cell 上跑一个集 群管理系统Borg。 - 通过定义 Cell 可以让 Borg 对服务器资源进 行统一抽象，作为用户 就无需知道自己的应用 跑在哪台机器上，也不 用关心资源分配、程序 安装、依赖管理、健康 检查及故障恢复等。 Job和Task Naming - 用户以 Job 的形式提 交应用部署请求。一个 Job 包含一个或多个相 同的Task，每个Task 运行相同的应用程序， Task 数量就是应用的 副本数。 - 每个 Job 可以定义属 性、元信息和优先级， 优先级涉及到抢占式调 度过程。 - Borg 的服务发现通过 BNS （ Borg Name Service）来实现。 - 50 .jfoo.ubar.cc.borg .google.com 可表示 在一个名为 cc 的 Cell 中由用户 uBar 部署的 一个名为 jFoo 的 Job 下的第 50 个Task。 #### Borg 架构 Borgmaster主进程： - 处理客户端RPC请求，比如创建Job，查询Job等。 - 维护系统组件和服务的状态，比如服务器、Task等。 - 负责与Borglet通信。 Scheduler进程： - 调度策略 - WorstFit - BestFit - Hybrid - 调度优化 - Scorecaching: 当服务器或者任务的状态未发生变更或者变更很少时，直接采用缓存数据，避免重复计算。 - Equivalenceclasses:调度同一Job下多个相同的Task只需计算一次。 - Relaxedrandomization:引入一些随机性，即每次随机选择一些机器，只要符合需求的服务器数量达到一定值时，就可 以停止计算，无需每次对Cell中所有服务器进行feasibilitychecking。 Borglet： Borglet是部署在所有服务器上的Agent，负责接收Borgmaster进程的指令。 #### 应用高可用 - 被抢占的non-prod任务放回pendingqueue，等待重新调度。 - 多副本应用跨故障域部署。所谓故障域有大有小，比如相同机器、相同机架或相同电源插座等，一挂全挂。 - 对于类似服务器或操作系统升级的维护操作，避免大量服务器同时进行。 - 支持幂等性，支持客户端重复操作。 - 当服务器状态变为不可用时，要控制重新调度任务的速率。因为Borg 无法区分是节点故障还是出现了短暂的 网络分区，如果是后者，静静地等待网络恢复更利于保障服务可用性。 - 当某种“任务 @ 服务器”的组合出现故障时，下次重新调度时需避免这种组合再次出现，因为极大可能会再 次出现相同故障。 - 记录详细的内部信息，便于故障排查和分析。 - 保障应用高可用的关键性设计原则：无论何种原因，即使 Borgmaster 或者 Borglet 挂掉、失联，都不能杀 掉正在运行的服务（Task）。 #### Borg 系统自身高可用 - Borgmaster组件多副本设计。 - 采用一些简单的和底层（low-level）的工具来部署Borg系统实例，避免引入过多的外部依赖。 - 每个Cell的Borg均独立部署，避免不同Borg系统相互影响。 #### 资源利用率 - 通过将在线任务（prod）和离线任务（non-prod，Batch）混合部署，空闲时，离线任务可以充分利用计 算资源；繁忙时，在线任务通过抢占的方式保证优先得到执行，合理地利用资源。 - 98 %的服务器实现了混部。 - 90 %的服务器中跑了超过 25 个Task和 4500 个线程。 - 在一个中等规模的 Cell 里，在线任务和离线任务独立部署比混合部署所需的服务器数量多出约 20 %- 30 %。 可以简单算一笔账，Google 的服务器数量在千万级别，按 20 % 算也是百万级别，大概能省下的服务器采 购费用就是百亿级别了，这还不包括省下的机房等基础设施和电费等费用。 #### Brog 调度原理 ![k8s_fw_rule_and_object_design_3](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_3.jpg) #### 隔离性 安全性隔离： - 早期采用Chrootjail，后期版本基于Namespace。 性能隔离： - 采用基于Cgroup的容器技术实现。 - 在线任务（prod）是延时敏感（latency-sensitive）型的，优先级高，而离线任务（non-prod， Batch）优先级低。 - Borg通过不同优先级之间的抢占式调度来优先保障在线任务的性能，牺牲离线任务。 - Borg将资源类型分成两类： - 可压榨的（compressible），CPU是可压榨资源，资源耗尽不会终止进程； - 不可压榨的（non-compressible），内存是不可压榨资源，资源耗尽进程会被终止。 ### 什么是 Kubernetes（K8s） Kubernetes是谷歌开源的容器集群管理系统，是Google多年大规模容器管理技术Borg的开源版本，主要功能包括： - 基于容器的应用部署、维护和滚动升级； - 负载均衡和服务发现； - 跨机器和跨地区的集群调度； - 自动伸缩； - 无状态服务和有状态服务； - 插件机制保证扩展性。 ![k8s_fw_rule_and_object_design_4](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_4.jpg) #### 命令式（ Imperative）vs 声明式（ Declarative） 命令式系统关注 “如何做” - 在软件工程领域，命令式系统是写出解决某个问题、 完成某个任务或者达到某个目标的明确步骤。此方法 明确写出系统应该执行某指令，并且期待系统返回期 望结果。 声明式系统关注“做什么” - 在软件工程领域，声明式系统指程序代码描述系统应该 做什么而不是怎么做。仅限于描述要达到什么目的，如 何达到目的交给系统。 #### 声明式（Declaritive）系统规范 命令式： - 我要你做什么，怎么做，请严格按照我说的做。 声明式： - 我需要你帮我做点事，但是我只告诉你我需要你做什么，不是你应该怎么做。 - 直接声明：我直接告诉你我需要什么。 - 间接声明：我不直接告诉你我的需求，我会把我的需求放在特定的地方，请在方便的时候拿出来处理。 幂等性： - 状态固定，每次我要你做事，请给我返回相同结果。 面向对象的： - 把一切抽象成对象。 #### Kubernetes：声明式系统 Kubernetes的所有管理能力构建在对象抽象的基础上，核心对象包括： - Node：计算节点的抽象，用来描述计算节点的资源抽象、健康状态等。 - Namespace：资源隔离的基本单位，可以简单理解为文件系统中的目录结构。 - Pod：用来描述应用实例，包括镜像地址、资源需求等。 Kubernetes 中最核心 的对象，也是打通应用和基础架构的秘密武器。 - Service：服务如何将应用发布成服务，本质上是负载均衡和域名服务的声明。 #### Kubernetes 采用与 Borg 类似的架构 ![k8s_fw_rule_and_object_design_5](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_5.jpg) #### 主要组件 ![k8s_fw_rule_and_object_design_6](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_6.jpg) #### Kubernetes 的主节点（Master Node） API服务器API Server: - 这是Kubernetes控制面板中唯一带有用户可访问API以及用户可交互的组件。API服 务器会暴露一个RESTful的Kubernetes API并使用JSON格式的清单文件（manifest files）。 群的数据存储Cluster Data Store: - Kubernetes 使用“etcd”。这是一个强大的、稳定的、高可用的键值存储，被 Kubernetes用于长久储存所有的API对象。 控制管理器ControllerManager: - 被称为“kube-controller manager”，它运行着所有处理集群日常任务的控制器。包 括了节点控制器、副本控制器、端点（endpoint）控制器以及服务账户等。 调度器Scheduler: - 调度器会监控新建的pods（一组或一个容器）并将其分配给节点。 #### Kubernetes 的工作节点（Worker Node） Kubelet: - 负责调度到对应节点的Pod 的生命周期管理，执行任务并将Pod状态报告给主节 点的渠道，通过容器运行时（拉取镜像、启动和停止容器等）来运行这些容器。它 还会定期执行被请求的容器的健康探测程序。 Kube-proxy: - 它负责节点的网络，在主机上维护网络规则并执行连接转发。它还负责对正在服务 的pods进行负载平衡。 #### etcd etcd 是 CoreOS 基于 Raft 开发的分布式 key-value 存储，可用于服务发现、共享配置以及一致性保障（如 数据库选主、分布式锁等）。 - 基本的key-value存储； - 监听机制； - key的过期及续约机制，用于监控和服务发现； - 原子CAS和CAD，用于分布式锁和leader选举。 ![k8s_fw_rule_and_object_design_7](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_7.jpg) 直接访问 etcd 的数据: - 通过etcd进程查看启动参数 - 进入容器 - ps-ef|grepetcd - sh: ps: command not found - 怎么办？到主机Namespace查看cert信息 - 进入容器查询数据 ``` shell export ETCDCTL_API=3 etcdctl--endpoints https://localhost:2379 --cert /etc/kubernetes/pki/etcd/server.crt--key /etc/kubernetes/pki/etcd/server.key--cacert/etc/kubernetes/pki/etcd/ca.crtget --keys-only --prefix / ``` - 监听对象变化 ``` shell etcdctl--endpoints https://localhost:2379 --cert /etc/kubernetes/pki/etcd/server.crt--key /etc/kubernetes/pki/etcd/server.key--cacert/etc/kubernetes/pki/etcd/ca.crtwatch --prefix /registry/services/specs/default/mynginx ``` #### APIServer Kube-APIServer是Kubernetes最重要的核心组件之一，主要提供以下功能： - 提供集群管理的RESTAPI接口，包括: - 认证Authentication； - 授权Authorization； - 准入Admission（Mutating&Valiating）。 - 提供其他模块之间的数据交互和通信的枢纽（其他模块通过APIServer查询或 修改数据，只有APIServer才直接操作etcd）。 - APIServer提供etcd数据缓存以减少集群对etcd的访问。 ![k8s_fw_rule_and_object_design_8](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_8.jpg) ##### APIServer 展开 ![k8s_fw_rule_and_object_design_9](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_9.jpg) #### Controller Manager - ControllerManager是集群的大脑，是确保整个集群动起来的关键； - 作用是确保 Kubernetes遵循声明式系统规范，确保系统的真实状态（Actual State）与用户定义的期望状态（DesiredState）一致； - Controller Manager是多个控制器的组合，每个Controller 事实上都是一个 controlloop，负责侦听其管控的对象，当对象发生变更时完成配置； - Controller 配置失败通常会触发自动重试，整个集群会在控制器不断重试的机 制下确保最终一致性（EventualConsistency）。 ##### 控制器的工作流程 ![k8s_fw_rule_and_object_design_10](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_10.jpg) ##### Informer 的内部机制 ![k8s_fw_rule_and_object_design_11](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_11.jpg) ##### 控制器的协同工作原理 ![k8s_fw_rule_and_object_design_12](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_12.jpg) #### Scheduler 特殊的Controller，工作原理与其他控制器无差别。 Scheduler 的特殊职责在于监控当前集群所有未调度的 Pod，并且获取当前集群所有节点的健康状况和资源 使用情况，为待调度Pod选择最佳计算节点，完成调度。 调度阶段分为： - Predict：过滤不能满足业务需求的节点，如资源不足、端口冲突等。 - Priority：按既定要素将满足调度需求的节点评分，选择最佳节点。 - Bind：将计算节点与Pod绑定，完成调度。 ![k8s_fw_rule_and_object_design_13](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_13.jpg) #### Kubelet Kubernetes的初始化系统（initsystem） - 从不同源获取Pod清单，并按需求启停Pod的核心组件： - Pod清单可从本地文件目录，给定的HTTPServer或Kube-APIServer等源头获取； - Kubelet将运行时，网络和存储抽象成了CRI，CNI，CSI。 - 负责汇报当前节点的资源信息和健康状态； - 负责Pod的健康检查和状态汇报。 ![k8s_fw_rule_and_object_design_14](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_14.jpg) #### Kube-Proxy - 监控集群中用户发布的服务，并完成负载均衡配置。 - 每个节点的 Kube-Proxy 都会配置相同的负载均衡策略，使得整个集群的服务发现建立在分布式负载均衡器之上，服务调用无需经过额外的网络跳转（NetworkHop）。 - 负载均衡配置基于不同插件实现： - userspace。 - 操作系统网络协议栈不同的Hooks点和插件： - iptables； - ipvs。 ![k8s_fw_rule_and_object_design_15](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_15.jpg) #### 推荐的 Add-ons - kube-dns：负责为整个集群提供DNS服务； - IngressController：为服务提供外网入口； - MetricsServer：提供资源监控； - Dashboard：提供GUI； - Fluentd-Elasticsearch：提供集群日志采集、存储与查询。 ### 了解 kubectl #### Kubectl 命令和 kubeconfig - kubectl是一个Kubernetes的命令行工具，它允许Kubernetes用户以命令行的方式与Kubernetes交 互，其默认读取配置文件~/.kube/config。 - kubectl会将接收到的用户请求转化为rest调用以rest client的形式与apiserver通讯。 - apiserver的地址，用户信息等配置在 kubeconfig。 ``` yaml apiVersion: v1 clusters: - cluster: certificate-authority-data: REDACTED server: https://127.0.0.1:54729 name: kind-kind contexts: - context: cluster: kind-kind user: kind-kind name: kind-kind current-context: kind-kind kind: Config users: - name: kind-kind user: client-certificate-data: REDACTED client-key-data: REDACTED ``` #### kubectl 常用命令 kubectl get po –oyaml -w kubectl 可查看对象。 - oyaml 输出详细信息为yaml格式。 - wwatch 该对象的后续变化。 - owide 以详细列表的格式查看对象。 #### Kubectl describe kubectldescribe展示资源的详细信息和相关Event。 ```shell kubectldescribe po ubuntu-6fcf6c67db-xvmjh .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8m13s default-scheduler Successfully assigned ubuntu-6fcf6c67db-xvmjh to k8smaster Normal Pulling 7m56s kubelet, k8smaster pulling image "ubuntu:16.04" Normal Pulled 7m50s kubelet, k8smaster Successfully pulled image "ubuntu:16.04" Normal Created 7m50s kubelet, k8smaster Created container Normal Started 7m50s kubelet, k8smaster Started container ``` #### kubectl exec kubectlexec提供进入运行容器的通道，可以进入容器进行debug操作。 ``` shell # kubectlexec -it ubuntu-6fcf6c67db-xvmjh bash root@ubuntu-6fcf6c67db-xvmjh:/# hostname -f ubuntu-6fcf6c67db-xvmjh root@ubuntu-6fcf6c67db-xvmjh:/# ... ``` #### kubectl logs Kubectllogs可查看pod的标准输入（stdout, stderr），与tail用法类似。 ```shell **jianqli:~# kubectl logs ubuntu-6fcf6c67db-xvmjh Mon Mar 25 14:56:02 UTC 2019 Mon Mar 25 14:56:05 UTC 2019 Mon Mar 25 14:56:08 UTC 2019 Mon Mar 25 14:56:11 UTC 2019 Mon Mar 25 14:56:14 UTC 2019 ...** ``` ### 深入理解 Kubernetes #### 云计算的传统分类 ![k8s_fw_rule_and_object_design_16](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_16.jpg) #### Kubernetes 生态系统 ![k8s_fw_rule_and_object_design_17](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_17.jpg) #### Kubernetes 设计理念 可扩展性: - 基于CRD的扩展 - 插件化的生态系统 可移植性: - 可移植性 - 多种基础架构的选择 - 多云和混合云 高可用: - 基于 replicaset，statefulset 的应用高可用 - Kubernetes 组件本身高可用 安全: - 基于 TLS 提供服务 - Serviceaccount 和 user - 基于 Namespace 的隔离 - secret - Taints，psp， networkpolicy #### Kubernetes Master ![k8s_fw_rule_and_object_design_18](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_18.jpg) #### 分层架构 - 核心层：Kubernetes最核心的功能，对外提供API构建高层的应用，对内提供插件式应用执行环境。 - 应用层：部署（无状态应用、有状态应用、批处理任务、集群应用等）和路由（服务发现、DNS 解析 等）。 - 管理层：系统度量（如基础设施、容器和网络的度量）、自动化（如自动扩展、动态 Provision 等）、 策略管理（RBAC、Quota、PSP、NetworkPolicy等）。 - 接口层：Kubectl命令行工具、客户端SDK以及集群联邦。 - 生态系统：在接口层之上的庞大容器集群管理调度的生态系统，可以划分为两个范畴： - Kubernetes 外部：日志、监控、配置管理、CI、CD、Workflow、FaaS、OTS 应用、 ChatOps等； - Kubernetes内部：CRI、CNI、CVI、镜像仓库、CloudProvider、集群自身的配置和管理等。 ![k8s_fw_rule_and_object_design_19](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_19.jpg) ![k8s_fw_rule_and_object_design_20](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_20.jpg) #### API 设计原则 - 所有API都应是声明式的 - 相对于命令式操作，声明式操作对于重复操作的效果是稳定的，这对于容易出现数据丢失或重复的分布式环境来说是很重要的。 - 声明式操作更易被用户使用，可以使系统向用户隐藏实现的细节，同时也保留了系统未来持续优化的可能性。 - 此外，声明式的API还隐含了所有的API对象都是名词性质的，例如Service、Volume这些API都是名词，这些名词描述了用户所 期望得到的一个目标对象。 - API对象是彼此互补而且可组合的 - 这实际上鼓励API对象尽量实现面向对象设计时的要求，即“高内聚，松耦合”，对业务相关的概念有一个合适的分解，提高分解出 来的对象的可重用性。 - 高层API以操作意图为基础设计 - 如何能够设计好 API，跟如何能用面向对象的方法设计好应用系统有相通的地方，高层设计一定是从业务出发，而不是过早的从技术 实现出发。 - 因此，针对Kubernetes的高层API设计，一定是以Kubernetes的业务为基础出发，也就是以系统调度管理容器的操作意图为基础 设计。 - 低层 API根据高层API的控制需要设计 - 设计实现低层 API 的目的，是为了被高层API 使用，考虑减少冗余、提高重用性的目的，低层API 的设计也要以需求为基础，要尽量抵抗受技术实现影响的诱惑。 - 尽量避免简单封装，不要有在外部API无法显式知道的内部隐藏的机制 - 简单的封装，实际没有提供新的功能，反而增加了对所封装API的依赖性。 - 例如 StatefulSet 和 ReplicaSet，本来就是两种Pod 集合，那么Kubernetes就用不同 API 对象 来定义它们，而不会说只用同一个 ReplicaSet，内部通过特殊的算法再来区分这个 ReplicaSet 是 有状态的还是无状态。 - API操作复杂度与对象数量成正比 - API的操作复杂度不能超过O(N)，否则系统就不具备水平伸缩性了。 - API对象状态不能依赖于网络连接状态 - 由于众所周知，在分布式环境下，网络连接断开是经常发生的事情，因此要保证API对象状态能应 对网络的不稳定，API对象的状态就不能依赖于网络连接状态。 - 尽量避免让操作机制依赖于全局状态 - 因为在分布式系统中要保证全局状态的同步是非常困难的。 #### Kubernetes 如何通过对象的组合完成业务描述 ![k8s_fw_rule_and_object_design_21](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_21.jpg) #### 架构设计原则 - 只有APIServer可以直接访问etcd存储，其他服务必须通过KubernetesAPI来访问集群状态； - 单节点故障不应该影响集群的状态； - 在没有新请求的情况下，所有组件应该在故障恢复后继续执行上次最后收到的请求比如网络分区或服务重启等）； - 所有组件都应该在内存中保持所需要的状态，APIServer将状态写入etcd存储，而其他组件则通过APIServer更新并监听所有的变化； - 优先使用事件监听而不是轮询。 #### 引导（Bootstrapping）原则 - Self-hosting是目标。 - 减少依赖，特别是稳态运行的依赖。 - 通过分层的原则管理依赖。 - 循环依赖问题的原则： - 同时还接受其他方式的数据输入（比如本地文件等），这样在其他服务不可 用时还可以手动配置引导服务； - 状态应该是可恢复或可重新发现的； - 支持简单的启动临时实例来创建稳态运行所需要的状态，使用分布式锁或文 件锁等来协调不同状态的切换（通常称为pivoting技术）； - 自动重启异常退出的服务，比如副本或者进程管理器等。 ### 核心技术概念和 API 对象 API对象是Kubernetes集群中的管理操作单元。 Kubernetes集群系统每支持一项新功能，引入一项新技术，一定会新引入对应的API对象，支持对该功能的管理操 作。 每个API对象都有四大类属性： - TypeMeta - MetaData - Spec - Status #### TypeMeta Kubernetes对象的最基本定义，它通过引入GKV（Group，Kind，Version）模型定义了一个对象的类型。 1.Group： - Kubernetes定义了非常多的对象，如何将这些对象进行归类是一门学问，将对象依据其功能范围归入不同的分组， 比如把支撑最基本功能的对象归入core组，把与应用部署有关的对象归入apps组，会使这些对象的可维护性和可 理解性更高。 2.Kind： - 定义一个对象的基本类型，比如Node、Pod、Deployment等。 3.Version： - 社区每个季度会推出一个Kubernetes版本，随着Kubernetes版本的演进，对象从创建之初到能够完全生产化就 绪的版本是不断变化的。与软件版本类似，通常社区提出一个模型定义以后，随着该对象不断成熟，其版本可能会 从v1alpha1到v1alpha2，或者到v1beta1，最终变成生产就绪版本v1。 #### Metadata Metadata中有两个最重要的属性：Namespace和Name，分别定义了对象的 Namespace归属及名字，这两个属性唯一定义了某个对象实例。 Label： 顾名思义就是给对象打标签，一个对象可以有任意对标签，其存在形式是键值对。 Label定义了对象的可识别属性，Kubernetes API支持以Label作为过滤条件 查询对象。 - Label是识别Kubernetes对象的标签，以key/value的方式附加到对象上。 - key最长不能超过 63 字节，value可以为空，也可以是不超过 253 字节的字符串。 - Label不提供唯一性，并且实际上经常是很多对象（如Pods）都使用相同的label来标志具 体的应用。 - Label定义好后其他对象可以使用Label Selector来选择一组相同label的对象 - Label Selector支持以下几种方式： - 等式，如app=nginx和env!=production； - 集合，如env in (production, qa)； - 多个label（它们之间是AND关系），如app=nginx,env=test。 Annotation： Annotation与Label一样用键值对来定义，但Annotation是作为属性扩展， 更多面向于系统管理员和开发人员，因此需要像其他属性一样做合理归类。 - Annotations是key/value形式附加于对象的注解。 - 不同于 Labels 用于标志和选择对象，Annotations 则是用来记录一些附加信息，用来辅助应用部署、安 全策略以及调度策略等。 - 比如deployment使用annotations来记录rollingupdate的状态。 Finalizer： - Finalizer本质上是一个资源锁，Kubernetes在接收某对象的删除请求时，会检 查Finalizer是否为空，如果不为空则只对其做逻辑删除，即只会更新对象中的 metadata.deletionTimestamp字段。 ResourceVersion： - ResourceVersion可以被看作一种乐观锁，每个对象在任意时刻都有其 ResourceVersion，当Kubernetes对象被客户端读取以后，ResourceVersion 信息也被一并读取。此机制确保了分布式系统中任意多线程能够无锁并发访问对 象，极大提升了系统的整体效率。 #### Spec 和 Status - Spec和Status才是对象的核心。 - Spec是用户的期望状态，由创建对象的用户端来定义。 - Status是对象的实际状态，由对应的控制器收集实际状态并更新。 - 与TypeMeta和Metadata等通用属性不同，Spec和Status是每个对象独有的。 #### 常用 Kubernetes 对象及其分组 ![k8s_fw_rule_and_object_design_22](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_22.jpg) ### 核心对象概览 #### Node - Node是Pod真正运行的主机，可以物理机，也可以是虚拟机。 - 为了管理Pod，每个Node节点上至少要运行container runtime （比如Docker或者Rkt）、Kubelet和Kube-proxy服务。 #### Namespace Namespace是对一组资源和对象的抽象集合，比如可以用来将系统内部的对 象划分为不同的项目组或用户组。 常见的pods, services, replication controllers和deployments等都是属于 某一个Namespace的（默认是default），而Node, persistentVolumes 等则不属于任何Namespace。 #### 什么是 Pod - Pod是一组紧密关联的容器集合，它们共享PID、IPC、Network和UTS namespace，是Kubernetes 调度的基本单位。 - Pod的设计理念是支持多个容器在一个Pod中共享网络和文件系统，可以通过进程间通信和文件共享这 种简单高效的方式组合完成服务。 - 同一个Pod中的不同容器可共享资源： - 共享网络Namespace； - 可通过挂载存储卷共享存储； - 共享SecurityContext。 #### 如何通过 Pod 对象定义支撑应用运行 - 环境变量： - 直接设置值； - 读取PodSpec的某些属性； - 从ConfigMap读取某个值； - 从Secret读取某个值。 #### 存储卷 - 通过存储卷可以将外挂存储挂载到Pod内部使用。 - 存储卷定义包括两个部分: Volume和VolumeMounts。 - Volume：定义Pod可以使用的存储卷来源； - VolumeMounts：定义存储卷如何Mount到容器内部。 #### Pod 网络 Pod的多个容器是共享网络Namespace的，这意味着： - 同一个Pod中的不同容器可以彼此通过Loopback地址访问： - 在第一个容器中起了一个服务http://127.0.0.1。 - 在第二个容器内，是可以通过httpGethttp://172.0.0.1访问到该地址的。 - 这种方法常用于不同容器的互相协作。 #### 资源限制 Kubernetes通过Cgroups提供容器资源管理的功能，可以限制每个容器的 CPU和内存使用，比如对于刚才创建的deployment，可以通过下面的命令限制 nginx容器最多只用50%的CPU和128MB的内存： ```shell $ kubectl set resources deployment nginx-app -c=nginx-- limits=cpu=500m,memory=128Mi deployment "nginx" resource requirements updated ``` 等同于在每个 Pod 中设置 resources limits ``` yaml apiVersion:v 1 kind:Pod metadata: labels: app:nginx name:nginx spec: containers: - image:nginx name:nginx resources: limits: cpu:" 500 m" memory:" 128 Mi" ``` #### 健康检查 Kubernetes作为一个面向应用的集群管理工具，需要确保容器在部署后确实处在正常的运行状态。 1.探针类型： - LivenessProbe - 探测应用是否处于健康状态，如果不健康则删除并重新创建容器。 - ReadinessProbe - 探测应用是否就绪并且处于正常服务状态，如果不正常则不会接收来自Kubernetes Service 的流量。 - StartupProbe - 探测应用是否启动完成，如果在failureThreshold*periodSeconds周期内未就绪，则会应用进程会被重启。 2.探活方式： - Exec - TCP socket - HTTP 健康检查 spec ```yaml apiVersion:extensions/v 1 beta 1 kind:Deployment metadata: labels: app:nginx name:nginx-default spec: replicas: 3 selector: matchLabels: app:nginx template: metadata: labels: app:nginx ``` ``` yaml spec: containers: - image:nginx imagePullPolicy:Always name:http resources:{} terminationMessagePath: /dev/termination-log terminationMessagePolicy:File resources: limits: cpu:" 500 m" memory:" 128 Mi" ``` yaml livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 15 timeoutSeconds: 1 readinessProbe: httpGet: path: /ping port: 80 initialDelaySeconds: 5 timeoutSeconds: 1 ``` #### ConfigMap - ConfigMap用来将非机密性的数据保存到键值对中。 - 使用时，Pods可以将其用作环境变量、命令行参数或者存储卷中的配置文件。 - ConfigMap将环境配置信息和容器镜像解耦，便于应用配置的修改。 #### 密钥对象（Secret） - Secret是用来保存和传递密码、密钥、认证凭证这些敏感信息的对象。 - 使用Secret的好处是可以避免把敏感信息明文写在配置文件里。 - Kubernetes集群中配置和使用服务不可避免的要用到各种敏感信息实现登录、认 证等功能，例如访问AWS存储的用户名密码。 - 为了避免将类似的敏感信息明文写在所有需要使用的配置文件中，可以将这些信息 存入一个Secret对象，而在配置文件中通过Secret对象引用这些敏感信息。 - 这种方式的好处包括：意图明确，避免重复，减少暴漏机会。 #### 用户（User Account）& 服务帐户（Service Account） - 顾名思义，用户帐户为人提供账户标识，而服务账户为计算机进程和Kubernetes集群中运行的Pod提供 账户标识。 - 用户帐户和服务帐户的一个区别是作用范围： - 用户帐户对应的是人的身份，人的身份与服务的Namespace无关，所以用户账户是跨 Namespace的； - 而服务帐户对应的是一个运行中程序的身份，与特定Namespace是相关的。 #### Service Service是应用服务的抽象，通过labels为应用提供负载均衡和服务发现。匹 配labels的PodIP和端口列表组成endpoints，由Kube-proxy负责将服务 IP负载均衡到这些endpoints上。 每个 Service 都会自动分配一个 cluster IP（仅在集群内部可访问的虚拟地址） 和DNS名，其他容器可以通过该地址或DNS来访问服务，而不需要了解后端 容器的运行。 ![k8s_fw_rule_and_object_design_23](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_23.jpg) #### Service Spec ``` yaml apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - port: 8078 # the port that this service should serve on name: http # the container on each pod to connect to, can be a name # (e.g. 'www') or a number (e.g. 80) targetPort: 80 protocol: TCP selector: app: nginx ``` #### 副本集（Replica Set） - Pod只是单个应用实例的抽象，要构建高可用应用，通常需要构建多个同样的副本，提供同一个服务。 - Kubernetes为此抽象出副本集ReplicaSet，其允许用户定义Pod的副本数，每一个Pod都会被当作一 个无状态的成员进行管理，Kubernetes保证总是有用户期望的数量的Pod正常运行。 - 当某个副本宕机以后，控制器将会创建一个新的副本。 - 当因业务负载发生变更而需要调整扩缩容时，可以方便地调整副本数量。 #### 部署（Deployment） - 部署表示用户对Kubernetes集群的一次更新操作。 - 部署是一个比RS应用模式更广的API对象，可以是创建一个新的服务，更新一个新的服务，也可以是滚动升级一个服务。 - 滚动升级一个服务，实际是创建一个新的RS，然后逐渐将新RS中副本数增加到理想状态，将旧RS中的副本数减小到 0 的 复合操作。 - 这样一个复合操作用一个RS是不太好描述的，所以用一个更通用的Deployment来描述。 - 以Kubernetes的发展方向，未来对所有长期伺服型的的业务的管理，都会通过Deployment来管理。 ![k8s_fw_rule_and_object_design_24](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_24.jpg) #### 有状态服务集（StatefulSet） - 对于StatefulSet中的Pod，每个Pod挂载自己独立的存储，如果一个Pod出现故障，从其他节点启动一个同样名字的 Pod，要挂载上原来Pod的存储继续以它的状态提供服务。 - 适合于StatefulSet的业务包括数据库服务MySQL和PostgreSQL，集群化管理服务ZooKeeper、etcd等有状态服务。 - 使用StatefulSet，Pod仍然可以通过漂移到不同节点提供高可用，而存储也可以通过外挂的存储来提供高可靠性， StatefulSet 做的只是将确定的Pod与确定的存储关联起来保证状态的连续性。 ![k8s_fw_rule_and_object_design_25](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_25.jpg) #### Statefulset 与 Deployment 的差异 - 身份标识 - StatefulSetController为每个Pod编号，序号从 0 开始。 - 数据存储 - StatefulSet允许用户定义volumeClaimTemplates，Pod被创建的同时，Kubernetes会以 volumeClaimTemplates中定义的模板创建存储卷，并挂载给Pod。 - StatefulSet的升级策略不同 - onDelete - 滚动升级 - 分片升级 #### 任务（Job） - Job是Kubernetes用来控制批处理型任务的API对象。 - Job管理的Pod根据用户的设置把任务成功完成后就自动退出。 - 成功完成的标志根据不同的spec.completions策略而不同： - 单Pod型任务有一个Pod成功就标志完成； - 定数成功型任务保证有N个任务全部成功； - 工作队列型任务根据应用确认的全局成功而标志成功。 ![k8s_fw_rule_and_object_design_26](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_26.jpg) #### 后台支撑服务集（DaemonSet） - 长期伺服型和批处理型服务的核心在业务应用，可能有些节点运行多个同类业务的Pod，有些节点上又没有这类Pod运行； - 而后台支撑型服务的核心关注点在Kubernetes集群中的节点（物理机或虚拟机），要保证每个节点上都有一个此类Pod运行。 - 节点可能是所有集群节点也可能是通过nodeSelector选定的一些特定节点。 - 典型的后台支撑型服务包括存储、日志和监控等在每个节点上支撑Kubernetes集群运行的服务。 ![k8s_fw_rule_and_object_design_27](../img/k8s/k8s_fw_rule_and_object_design/k8s_fw_rule_and_object_design_27.jpg) #### 存储 PV 和 PVC - PersistentVolume（PV）是集群中的一块存储卷，可以由管理员手动设置， 或当用户创建PersistentVolumeClaim（PVC）时根据StorageClass动 态设置。 - PV和PVC与Pod生命周期无关。也就是说，当Pod中的容器重新启动、 Pod重新调度或者删除时，PV和PVC不会受到影响，Pod存储于PV里 的数据得以保留。 - 对于不同的使用场景，用户通常需要不同属性（例如性能、访问模式等） 的PV。 #### CustomResourceDefinition - CRD就像数据库的开放式表结构，允许用户自定义Schema。 - 有了这种开放式设计，用户可以基于CRD定义一切需要的模型，满足不同 业务的需求。 - 社区鼓励基于CRD的业务抽象，众多主流的扩展应用都是基于CRD构建 的，比如Istio、Knative。 - 甚至基于CRD推出了Operator Mode和Operator SDK，可以以极低的 开发成本定义新对象，并构建新对象的控制器。 ================================================ FILE: docs/kubernetes/k8s_kube_APIServer.md ================================================ # 深入理解Kube-APIServer 目录 - 认证 - 鉴权 - 准入 Mutating Validating Admission - 限流 - APIServer对象的实现 ## API Server kube-apiserver是Kubernetes最重要的核心组件之一，主要提供以下的功能 - 提供集群管理的REST API接口，包括认证授权、数据校验以及集群状态变更 等 - 提供其他模块之间的数据交互和通信的枢纽（其他模块通过API Server查询或 修改数据，只有API Server才直接操作etcd） ## 访问控制概览 Kubernetes API的每个请求都会经过多阶段的访问控制之后才会被接受，这包括认证、授权以及 准入控制（Admission Control）等。 ![k8s_kube_APIServer_1](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_1.jpg) ## 访问控制细节 ![k8s_kube_APIServer_2](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_2.jpg) ## 认证 开启TLS时，所有的请求都需要首先认证。Kubernetes支持多种认证机制，并支持同时 开启多个认证插件（只要有一个认证通过即可）。如果认证成功，则用户的username会 传入授权模块做进一步授权验证；而对于认证失败的请求则返回HTTP 401。 ### 认证插件 - X509证书 - 使用X509客户端证书只需要API Server启动时配置--client-ca-file=SOMEFILE。在证书认证时，其CN域用作用户名，而组织 机构域则用作group名。 - 静态Token文件 - 使用静态Token文件认证只需要APIServer启动时配置--token-auth-file=SOMEFILE。 - 该文件为csv格式，每行至少包括三列token,username,userid， token,user,uid,"group1,group2,group3” - 引导Token - 为了支持平滑地启动引导新的集群，Kubernetes 包含了一种动态管理的持有者令牌类型，称作启动引导令牌（Bootstrap Token）。 - 这些令牌以Secret 的形式保存在kube-system名字空间中，可以被动态管理和创建。 - 控制器管理器包含的TokenCleaner控制器能够在启动引导令牌过期时将其删除。 - 在使用kubeadm部署Kubernetes时，可通过kubeadmtoken list命令查询。 - 静态密码文件 - 需要APIServer启动时配置--basic-auth-file=SOMEFILE，文件格式为csv，每行至少三列password, user, uid，后面是可选 的group名 password,user,uid,"group1,group2,group3” - ServiceAccount - ServiceAccount是Kubernetes自动生成的，并会自动挂载到容器的/run/secrets/kubernetes.io/serviceaccount目录中。 - OpenID - OAuth 2.0的认证机制 - Webhook 令牌身份认证 - --authentication-token-webhook-config-file指向一个配置文件，其中描述如何访问远程的Webhook 服务。 - --authentication-token-webhook-cache-ttl用来设定身份认证决定的缓存时间。默认时长为 2 分钟。 - 匿名请求 - 如果使用AlwaysAllow以外的认证模式，则匿名请求默认开启，但可用--anonymous-auth=false禁止匿名请求。 ### 基于webhook的认证服务集成 https://github.com/appscode/guard #### 构建符合Kubernetes规范的认证服务 需要依照Kubernetes规范，构建认证服务，用来认证tokenreviewrequest 构建认证服务 - 认证服务需要满足如下Kubernetes的规范 ➢ URL：https://authn.example.com/authenticate ➢ Method：POST ➢ Input: ➢ Output: ``` json { "apiVersion": "authentication.k8s.io/v1beta1", "kind": "TokenReview", "spec": { "token": "(BEARERTOKEN)" } } { "apiVersion": "authentication.k8s.io/v1beta1", "kind": "TokenReview", "status": { "authenticated": true, "user": { "username": "janedoe@example.com", "uid": "42", "groups": [ "developers", "qa" ]}} } ``` #### 开发认证服务 解码认证请求 ``` go decoder := json.NewDecoder(r.Body) var tr authentication.TokenReview err := decoder.Decode(&tr) if err != nil { log.Println("[Error]", err.Error()) w.WriteHeader(http.StatusBadRequest) json.NewEncoder(w).Encode(map[string]interface{}{ "apiVersion": "authentication.k8s.io/v1beta1", "kind": "TokenReview", "status": authentication.TokenReviewStatus{ Authenticated: false, }, }) return } ``` 转发认证请求至认证服务器 ``` go //CheckUser ts:=oauth2.StaticTokenSource( &oauth2.Token{AccessToken:tr.Spec.Token}, ) tc:=oauth2.NewClient(oauth2.NoContext,ts) client:=github.NewClient(tc) user,_,err:=client.Users.Get(context.Background(),"") iferr!=nil{ log.Println("[Error]",err.Error()) w.WriteHeader(http.StatusUnauthorized) json.NewEncoder(w).Encode(map[string]interface{}{ "apiVersion":"authentication.k8s.io/v1beta1", "kind": "TokenReview", "status":authentication.TokenReviewStatus{ Authenticated:false, }, }) return } ``` 认证结果返回给APIServer ``` go w.WriteHeader(http.StatusOK) trs := authentication.TokenReviewStatus{ Authenticated: true, User: authentication.UserInfo{ Username: *user.Login, UID: *user.Login, }, } json.NewEncoder(w).Encode(map[string]interface{}{ "apiVersion": "authentication.k8s.io/v1beta1", "kind": "TokenReview", "status": trs, }) ``` #### 配置认证服务 ``` json { "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "github-authn", "cluster": { "server": "http://localhost:3000/authenticate" } }], "users": [ { "name": "authn-apiserver", "user": { "token": "secret" } }], "contexts": [ { "name": "webhook", "context": { "cluster": "github-authn", "user": "authn-apiserver" } }], "current-context": "webhook" } ``` #### 配置apiserver 可以是任何认证系统 - 但在用户认证完成后，生成代表用户身份的token - 该token通常是有失效时间的 - 用户获取该token以后以后，将token配置进 kubeconfig 修改apiserver设置，开启认证服务，apiserver保证将所有收到的 请求中的token信息，发给认证服务进行验证 - --authentication-token-webhook-config-file，该 文件描述如何访问认证服务 - --authentication-token-webhook-cache-ttl，默 认 2 分钟 配置文件需要mount进Pod 配置文件中的服务器地址需要指向authService ```json { "kind": "Config", "apiVersion": "v1", "preferences": {}, "clusters": [ { "name": "github-authn", "cluster": { "server": "http://localhost:3000/authenticate" } }], "users": [ { "name": "authn-apiserver", "user": { "token": "secret" } }], "contexts": [ { "name": "webhook", "context": { "cluster": "github-authn", "user": "authn-apiserver" } }], "current-context": "webhook" } ``` ### 生产系统中遇到的陷阱 ``` 基于Keystone的认证插件导致Keystone故障且无法恢复 Keystone是企业关键服务 Kubernetes以Keystone作为认证插件 Keystone在出现故障后会抛出 401 错误 Kubernetes发现 401 错误后会尝试重新认证 大多数controller都有指数级backoff，重试间隔越来越慢 但gophercloud针对过期token会一直retry 大量的request积压在Keystone导致服务无法恢复 Kubernetes成为压死企业认证服务的最后一根稻草 ``` 解决方案？ - Circuitbreak - Ratelimit ## 2. 鉴权 ### 授权 授权主要是用于对集群资源的访问控制，通过检查请求包含的相关属性值，与相对应的访问策略相比较，API请求必须满足某 些策略才能被处理。跟认证类似，Kubernetes也支持多种授权机制，并支持同时开启多个授权插件（只要有一个验证通过即 可）。如果授权成功，则用户的请求会发送到准入控制模块做进一步的请求验证；对于授权失败的请求则返回HTTP 403。 Kubernetes授权仅处理以下的请求属性： - user, group, extra - API、请求方法（如get、post、update、patch和delete）和请求路径（如/api） - 请求资源和子资源 - Namespace - API Group 目前，Kubernetes支持以下授权插件： - ABAC - RBAC - Webhook - Node ### RBAC vs ABAC ABAC（Attribute Based Access Control）本来是不错的概念，但是在Kubernetes 中的实现比较 难于管理和理解，而且需要对Master 所在节点的SSH 和文件系统权限，要使得对授权的变更成 功生效，还需要重新启动API Server。 而RBAC 的授权策略可以利用kubectl 或者Kubernetes API 直接进行配置。RBAC 可以授权给用户， 让用户有权进行授权管理，这样就可以无需接触节点，直接进行授权管理。RBAC 在Kubernetes 中被映射为API 资源和操作。 #### RBAC老图 ![k8s_kube_APIServer_3](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_3.jpg) #### RBAC新解 ![k8s_kube_APIServer_4](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_4.jpg) ### Role与ClusterRole Role（角色）是一系列权限的集合，例如一个角色可以包含读取Pod 的权限和列出Pod 的权限。 Role只能用来给某个特定namespace中的资源作鉴权，对多namespace和集群级的资源或者是非 资源类的API（如/healthz）使用ClusterRole。 ``` yaml # Role示例 kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["get", "watch", "list"] # ClusterRole示例 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: # "namespace" omitted since ClusterRoles are not namespaced name: secret-reader rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "watch", "list"] ``` #### binding ``` yaml # RoleBinding示例（引用ClusterRole） # This role binding allows "dave" to read secrets in the "development" namespace. kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-secrets namespace: development # This only grants permissions within the "development" namespace. subjects: - kind: User name: dave apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io ``` ![k8s_kube_APIServer_5](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_5.jpg) ### 账户／组的管理 角色绑定（Role Binding）是将角色中定义的权限赋予一个或者一组用户。 它包含若干主体（用户、组或服务账户）的列表和对这些主体所获得的角色的引用。 组的概念： - 当与外部认证系统对接时，用户信息（UserInfo）可包含Group信息，授权可针对用户群组 - 当对ServiceAccount授权时，Group代表某个Namespace下的所有ServiceAccount ![k8s_kube_APIServer_6](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_6.jpg) ### 针对群租授权 ``` yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: read-secrets-global subjects: - kind: Group name: manager # 'name' 是区分大小写的 apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: read-secrets-global subjects: - kind: Group name: system:serviceaccounts:qa apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io ``` ### 规划系统角色 User - 管理员 ➢所有资源的所有权限？？ - 普通用户 ➢ 是否有该用户创建的namespace下的所有object的操作权限？ ➢ 对其他用户的namespace资源是否可读，是否可写？ SystemAccount - SystemAccount是开发者（kubernetesdeveloper或者domaindeveloper）创建应用后，应 用于apiserver通讯需要的身份 - 用户可以创建自定的ServiceAccount，kubernetes也为每个namespace创建default ServiceAccount - DefaultServiceAccount通常需要给定权限以后才能对apiserver做写操作 ### 实现方案 在cluster创建时，创建自定义的role，比如namespace-creator Namespace-creatorrole定义用户可操作的对象和对应的读写操作。 创建自定义的namespaceadmissioncontroller - 当namespace创建请求被处理时，获取当前用户信息并annotate到namespace 创建RBAC controller - Watchnamespace的创建事件 - 获取当前namespace的创建者信息 - 在当前namespace创建rolebinding对象，并将namespace-creator 角色和用户绑定 ### 与权限相关的其他最佳实践 ClusterRole是非namespace绑定的，针对整个集群生效 通常需要创建一个管理员角色，并且绑定给开发运营团队成员 ThirdPartyResource和CustomResourceDefinition是全局资源，普通用户创建 ThirdPartyResource以后，需要管理员授予相应权限后才能真正操作该对象 针对所有的角色管理，建议创建spec，用源代码驱动 - 虽然可以通过edit操作来修改权限，但后期会导致权限管理混乱，可能会有很多临时创建出来的 角色和角色绑定对象，重复绑定某一个资源权限 权限是可以传递的，用户A可以将其对某对象的某操作，抽取成一个权限，并赋给用户B 防止海量的角色和角色绑定对象，因为大量的对象会导致鉴权效率低，同时给apiserver增加负担 ServiceAccount也需要授权的，否则你的component可能无法操作某对象 Tips：SSH到master节点通过insecureport访问apiserver可绕过鉴权，当需要做管理操作又没 有权限时可以使用（不推荐） ### 运营过程中出现的陷阱 案例1: - 研发人员为提高系统效率，将update方法修改为patch - 研发人员本地非安全测试环境测试通过 - 上生产，发现不work - 原因：忘记更新rolebinding，对应的serviceaccount没有patch权限 案例2: - 研发人员创建CRD，并针对该CRD编程 - 上生产后不工作 - 原因，该CRD未授权，对应的组件get不到对应的CRD资源 ## 3. 准入 ### 准入控制 为资源增加自定义属性 - 作为多租户集群方案中的一环，我们需要在namespace的准入控制中，获取用户信息，并将用 户信息更新的namespace的annotation 只有当namespace中有有效用户信息时，我们才可以在namespace创建时，自动绑定用户权限， namespace才可用。 准入控制（Admission Control）在授权后对请求做进一步的验证或添加默认参数。不同于授权 和认证只关心请求的用户和操作，准入控制还处理请求的内容，并且仅对创建、更新、删除或连 接（如代理）等有效，而对读操作无效。 准入控制支持同时开启多个插件，它们依次调用，只有全部插件都通过的请求才可以放过进入系 统。 ### 准入控制插件 AlwaysAdmit: 接受所有请求。 AlwaysPullImages: 总是拉取最新镜像。在多租户场景下非常有用。 DenyEscalatingExec: 禁止特权容器的exec和attach操作。 ImagePolicyWebhook: 通过webhook决定image策略，需要同时配置--admission-control- config-file ServiceAccount：自动创建默认ServiceAccount，并确保Pod引用的ServiceAccount已经存在 SecurityContextDeny：拒绝包含非法SecurityContext配置的容器 ResourceQuota：限制Pod的请求不会超过配额，需要在namespace中创建一个 ResourceQuota对象 LimitRanger：为Pod设置默认资源请求和限制，需要在namespace中创建一个LimitRange对 象 InitialResources：根据镜像的历史使用记录，为容器设置默认资源请求和限制 NamespaceLifecycle：确保处于termination状态的namespace不再接收新的对象创建请求， 并拒绝请求不存在的namespace DefaultStorageClass：为PVC设置默认StorageClass DefaultTolerationSeconds：设置Pod的默认forgivenesstoleration为 5 分钟 PodSecurityPolicy：使用PodSecurity Policies时必须开启 NodeRestriction：限制kubelet仅可访问node、endpoint、pod、service以及secret、 configmap、PV和PVC等相关的资源 除默认的准入控制插件以外，Kubernetes预留了准入控制插件的扩展点，用户可自定义准入控制 插件实现自定义准入功能 MutatingWebhookConfiguration：变形插件，支持对准入对象的修改 ValidatingWebhookConfiguration：校验插件，只能对准入对象合法性进行校验，不能修改 ![k8s_kube_APIServer_7](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_7.jpg) ``` yaml # {{if eq .k8snode_validating "enabled"}} apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: ns-mutating.webhook.k8s.io webhooks: - clientConfig: caBundle: {{.serverca_base64}} url: https://admission.local.tess.io/apis/admissio n.k8s.io/v1alpha1/ ns-mutating failurePolicy: Fail name: ns-mutating.webhook.k8s.io namespaceSelector: {} rules: - apiGroups: - "" apiVersions: - '*' operations: - CREATE resources: - nodes sideEffects: Unknown # {{end}} ``` 准入控制 配额管理 - 原因：资源有限，如何限定某个用户有多少资源？ 方案： - 预定义每个Namespace的ResourceQuota，并把spec保存为configmap ➢ 用户可以创建多少个Pod ➢BestEffortPod ➢ QoSPod ➢ 用户可以创建多少个service ➢ 用户可以创建多少个ingress ➢ 用户可以创建多少个serviceVIP - 创建ResourceQuotaController ➢ 监控namespace创建事件，当namespace创建时，在该namespace创建对应的 ResourceQuota对象 - apiserver中开启ResourceQuota的admissionplugin ## 4. 限流 ### 计数器固定窗口算法 原理就是对一段固定时间窗口内的请求进行计数，如果请求数超过了阈值，则舍弃该请求； 如果没有达到设定的阈值，则接受该请求，且计数加 1 。 当时间窗口结束时，重置计数器为 0 。 ![k8s_kube_APIServer_8](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_8.jpg) ### 计数器滑动窗口算法 在固定窗口的基础上，将一个计时窗口分成了若干个小窗口，然后每个小窗口维护一个独立的计 数器。 当请求的时间大于当前窗口的最大时间时，则将计时窗口向前平移一个小窗口。 平移时，将第一个小窗口的数据丢弃，然后将第二个小窗口设置为第一个小窗口，同时在最后面 新增一个小窗口，将新的请求放在新增的小窗口中。 同时要保证整个窗口中所有小窗口的请求数目之后不能超过设定的阈值。 ![k8s_kube_APIServer_9](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_9.jpg) ### 漏斗算法 漏斗算法的原理也很容易理解。请求来了之后会首先进到漏斗里，然后漏斗以恒定的速率将请求 流出进行处理，从而起到平滑流量的作用。 当请求的流量过大时，漏斗达到最大容量时会溢出，此时请求被丢弃。 在系统看来，请求永远是以平滑的传输速率过来，从而起到了保护系统的作用。 ![k8s_kube_APIServer_10](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_10.jpg) ### 令牌桶算法 令牌桶算法是对漏斗算法的一种改进，除了能够起到限流的作用外，还允许一定程度的流量突发。 在令牌桶算法中，存在一个令牌桶，算法中存在一种机制以恒定的速率向令牌桶中放入令牌。 令牌桶也有一定的容量，如果满了令牌就无法放进去了。 当请求来时，会首先到令牌桶中去拿令牌，如果拿到了令牌，则该请求会被处理，并消耗掉拿到 的令牌； 如果令牌桶为空，则该请求会被丢弃。 ![k8s_kube_APIServer_11](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_11.jpg) ### APIServer中的限流 max-requests-inflight：在给定时间内的最大non-mutating 请求数 max-mutating-requests-inflight：在给定时间内的最大mutating 请求数，调整apiserver的 流控qos 代码 staging/src/k8s.io/apiserver/pkg/server/filters/maxinflight.go:WithMaxInFlightLimit() ![k8s_kube_APIServer_12](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_12.jpg) ### 传统限流方法的局限性 - 粒度粗 - 无法为不同用户，不同场景设置不通的限流 - 单队列 - 共享限流窗口/桶，一个坏用户可能会将整个系统堵塞，其他正常用户的请求无法被及时处 理 - 不公平 - 正常用户的请求会被排到队尾，无法及时处理而饿死 - 无优先级 - 重要的系统指令一并被限流，系统故障难以恢复 ### API Priority and Fairness - APF 以更细粒度的方式对请求进行分类和隔离。 - 它还引入了空间有限的排队机制，因此在非常短暂的突发情况下，API 服务器不会拒绝任何请 求。 - 通过使用公平排队技术从队列中分发请求，这样， 一个行为不佳的控制器就不会饿死其他控制 器（即使优先级相同）。 - APF的核心 - 多等级 - 多队列 - APF 的实现依赖两个非常重要的资源FlowSchema, PriorityLevelConfiguration - APF 对请求进行更细粒度的分类，每一个请求分类对应一个FlowSchema(FS) - FS 内的请求又会根据distinguisher 进一步划分为不同的Flow. - FS 会设置一个优先级(Priority Level, PL)，不同优先级的并发资源是隔离的。所以不同优先级的资源不会相互排挤。特定优先级的请求可以被高优处理。 - 一个PL 可以对应多个FS，PL 中维护了一个QueueSet，用于缓存不能及时处理的请求，请求不会因为超出PL 的并发限制而被丢弃。 - FS 中的每个Flow 通过shuffle sharding算法从QueueSet选取特定的queues 缓存请求。 - 每次从QueueSet中取请求执行时，会先应用fair queuing 算法从QueueSet中选中一个queue，然后从这个queue中取出oldest 请求执行。所以即使是同一个PL 内的请求，也不会出现一个Flow 内的请求一直占用资源的不公平现象。 ### 概念 - 传入的请求通过FlowSchema按照其属性分类，并分配优先级。 - 每个优先级维护自定义的并发限制，加强了隔离度，这样不同优先级的请求，就不会相互饿死。 - 在同一个优先级内，公平排队算法可以防止来自不同flow的请求相互饿死。 - 该算法将请求排队，通过排队机制，防止在平均负载较低时，通信量突增而导致请求失败。 ### 优先级 - 如果未启用APF，API 服务器中的整体并发量将受到kube-apiserver的参数--max- requests-inflight和--max-mutating-requests-inflight的限制。 - 启用 APF 后，将对这些参数定义的并发限制进行求和，然后将总和分配到一组可配置的 优先 级中。每个传入的请求都会分配一个优先级； - 每个优先级都有各自的配置，设定允许分发的并发请求数。 - 例如，默认配置包括针对领导者选举请求、内置控制器请求和Pod 请求都单独设置优先级。 这表示即使异常的Pod 向API 服务器发送大量请求，也无法阻止领导者选举或内置控制器的 操作执行成功。 ### 排队 - 即使在同一优先级内，也可能存在大量不同的流量源。 - 在过载情况下，防止一个请求流饿死其他流是非常有价值的（尤其是在一个较为常见的场景中，一个有故 障的客户端会疯狂地向kube-apiserver发送请求，理想情况下，这个有故障的客户端不应对其他客户端 产生太大的影响）。 - 公平排队算法在处理具有相同优先级的请求时，实现了上述场景。 - 每个请求都被分配到某个流中，该流由对应的FlowSchema的名字加上一个流区分项（Flow Distinguisher）来标识。 - 这里的流区分项可以是发出请求的用户、目标资源的名称空间或什么都不是。 - 系统尝试为不同流中具有相同优先级的请求赋予近似相等的权重。 - 将请求划分到流中之后，APF 功能将请求分配到队列中。 - 分配时使用一种称为混洗分片（Shuffle-Sharding）的技术。该技术可以相对有效地利用队列隔离低强 度流与高强度流。 - 排队算法的细节可针对每个优先等级进行调整，并允许管理员在内存占用、公平性（当总流量超标时，各 个独立的流将都会取得进展）、突发流量的容忍度以及排队引发的额外延迟之间进行权衡。 ### 豁免请求 某些特别重要的请求不受制于此特性施加的任何限制。这些豁免可防止不当的流控配置完全禁用 API 服务器。 ### 默认配置 - system - 用于system:nodes组（即kubelets）的请求；kubelets必须能连上API 服务器，以便工作负 载能够调度到其上。 - leader-election - 用于内置控制器的领导选举的请求（特别是来自kube-system名称空间中system:kube- controller-manager和system:kube-scheduler用户和服务账号，针对endpoints、 configmaps或leases的请求）。 - 将这些请求与其他流量相隔离非常重要，因为领导者选举失败会导致控制器发生故障并重新启动， 这反过来会导致新启动的控制器在同步信息时，流量开销更大。 - workload-high - 优先级用于内置控制器的请求。 - workload-low - 优先级适用于来自任何服务帐户的请求，通常包括来自Pods 中运行的控制器的所有请求。 - global-default - 优先级可处理所有其他流量，例如：非特权用户运行的交互式kubectl命令。 - exempt - 优先级的请求完全不受流控限制：它们总是立刻被分发。特殊的exemptFlowSchema 把system:masters组的所有请求都归入该优先级组。 - catch-all - 优先级与特殊的catch-allFlowSchema结合使用，以确保每个请求都分类。 - 一般不应该依赖于catch-all的配置，而应适当地创建自己的catch-allFlowSchema和 PriorityLevelConfigurations（或使用默认安装的global-default配置）。 - 为了帮助捕获部分请求未分类的配置错误，强制要求catch-all优先级仅允许 5 个并发份额， 并且不对请求进行排队，使得仅与catch-allFlowSchema匹配的流量被拒绝的可能性更高， 并显示HTTP 429 错误。 ### PriorityLevelConfiguration 一个PriorityLevelConfiguration表示单个隔离类型。 每个PriorityLevelConfigurations对未完成的请求数有各自的限制，对排队中的请求数也有限制。 ![k8s_kube_APIServer_13](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_13.jpg) ### FlowSchema FlowSchema匹配一些入站请求，并将它们分配给优先级。 每个入站请求都会对所有FlowSchema测试是否匹配，首先从matchingPrecedence数值最低的匹配开始 （我们认为这是逻辑上匹配度最高），然后依次进行，直到首个匹配出现 ![k8s_kube_APIServer_14](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_14.jpg) ### 调试 - /debug/api_priority_and_fairness/dump_priority_levels——所有优先级及其当前状态 的列表 kubectlget --raw /debug/api_priority_and_fairness/dump_priority_levels - /debug/api_priority_and_fairness/dump_queues——所有队列及其当前状态的列表 kubectlget --raw /debug/api_priority_and_fairness/dump_queues - /debug/api_priority_and_fairness/dump_requests——当前正在队列中等待的所有请求 的列表 kubectlget --raw /debug/api_priority_and_fairness/dump_requests ## 5. 高可用APIServer ### 启动apiserver示例 ``` shell kube-apiserver--feature-gates=AllAlpha=true --runtime-config=api/all=true \ --requestheader-allowed-names=front-proxy-client \ --client-ca-file=/etc/kubernetes/pki/ca.crt \ --allow-privileged=true \ --experimental-bootstrap-token-auth=true \ --storage-backend=etcd3 \ --requestheader-username-headers=X-Remote-User \ --requestheader-extra-headers-prefix=X-Remote-Extra-\ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \ --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \ --enabled-hooks=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota \ --requestheader-group-headers=X-Remote-Group \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\ --secure-port=6443 \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname\ --service-cluster-ip-range=10.96.0.0/12 \ --advertise-address=192.168.0.20 --etcd-servers=http://127.0.0.1:2379 ``` ### 构建高可用的多副本apiserver apiserver是无状态的RestServer 无状态所以方便ScaleUp／down 负载均衡 - 在多个apiserver实例之上，配置负载均衡 - 证书可能需要加上LoadbalancerVIP重新生成 ### 预留充足的CPU、内存资源 随着集群中节点数量不断增多，APIServer对CPU和内存的开销也不断增大。过少的CPU资源会降 低其处理效率，过少的内存资源会导致Pod被OOMKilled，直接导致服务不可用。在规划 APIServer资源时，不能仅看当下需求，也要为未来预留充分。 ### 善用速率限制（RateLimit） APIServer的参数“--max-requests-inflight”和“--max-mutating-requests-inflight”支持 在给定时间内限制并行处理读请求（包括Get、List和Watch操作）和写请求（包括Create、 Delete、Update和Patch操作）的最大数量。当APIServer接收到的请求超过这两个参数设定的 值时，再接收到的请求将会被直接拒绝。通过速率限制机制，可以有效地控制APIServer内存的使 用。如果该值配置过低，会经常出现请求超过限制的错误，如果配置过高，则APIServer可能会因 为占用过多内存而被强制终止，因此需要根据实际的运行环境，结合实时用户请求数量和 APIServer的资源配置进行调优。 客户端在接收到拒绝请求的返回值后，应等待一段时间再发起重试，无间隔的重试会加重 APIServer的压力，导致性能进一步降低。针对并行处理请求数的过滤颗粒度太大，在请求数量比 较多的场景，重要的消息可能会被拒绝掉，自1.18版本开始，社区引入了优先级和公平保证 （Priority and Fairness）功能，以提供更细粒度地客户端请求控制。该功能支持将不同用户或 不同类型的请求进行优先级归类，保证高优先级的请求总是能够更快得到处理，从而不受低优先 级请求的影响。 ### 设置合适的缓存大小 APIServer与etcd之间基于gRPC协议进行通信，gRPC协议保证了二者在大规模集群中的数据高速 传输。gRPC基于连接复用的HTTP/2协议，即针对相同分组的对象，APIServer和etcd之间共享相 同的TCP连接，不同请求由不同的stream传输。 一个HTTP/2连接有其stream配额，配额的大小限制了能支持的并发请求。APIServer提供了集 群对象的缓存机制，当客户端发起查询请求时，APIServer默认会将其缓存直接返回给客户端。缓 存区大小可以通过参数“--watch-cache-sizes”设置。针对访问请求比较多的对象，适当设置 缓存的大小，极大降低对etcd的访问频率，节省了网络调用，降低了对etcd集群的读写压力，从 而提高对象访问的性能。 但是APIServer也是允许客户端忽略缓存的，例如客户端请求中ListOption中没有设置 resourceVersion，这时APIServer直接从etcd拉取最新数据返回给客户端。客户端应尽量避免此 操作，应在ListOption中设置resourceVersion为 0 ，APIServer则将从缓存里面读取数据，而不 会直接访问etcd。 ### 客户端尽量使用长连接 当查询请求的返回数据较大且此类请求并发量较大时，容易引发TCP链路的阻塞，导致其他查询 操作超时。因此基于Kubernetes开发组件时，例如某些DaemonSet和Controller，如果要查询 某类对象，应尽量通过长连接ListWatch监听对象变更，避免全量从APIServer获取资源。如果在 同一应用程序中，如果有多个Informer监听APIServer资源变化，可以将这些Informer合并，减 少和APIServer的长连接数，从而降低对APIServer的压力。 ### 如何访问APIServer 对外部客户（user/client/admin)，永远只通过LoadBalancer访问 只有当负载均衡出现故障时，管理员才切换到apiserverIP进行管理 内部客户端，优先访问cluster IP？（是否一定如此？） ![k8s_kube_APIServer_15](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_15.jpg) ### 搭建多租户的Kubernetes集群 授信 - 认证： ➢ 禁止匿名访问，只允许可信用户做操作。 - 授权： ➢ 基于授信的操作，防止多用户之间互相影响，比如普通用户删除Kubernetes核心服务，或者A用户删除或修改B用户 的应用。 隔离 - 可见行隔离： ➢ 用户只关心自己的应用，无需看到其他用户的服务和部署。 - 资源隔离： ➢ 有些关键项目对资源需求较高，需要专有设备，不与其他人共享。 - 应用访问隔离： ➢ 用户创建的服务，按既定规则允许其他用户访问。 资源管理 - Quota管理 ➢ 谁能用多少资源？ ### 认证 与企业现有认证系统集成 - 很多企业基于MicrosoftActiveDirectory提供认证服务 选择认证插件 ➢ 选择webhook作为认证插件（*以此为例展开） ➢ 也可以选择Keystone作为认证插件，以MicrosoftAd作为backend搭建keystone服务 一旦认证完成，Kubernetes即可获取当前用户信息（主要是用户名），并针对该用户做授权。授权和准入控 制完成后，该用户的请求完成。 ### 注册APIService ``` yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: labels: kube-aggregator.kubernetes.io/automanaged: onstart name: v1. spec: groupPriorityMinimum: 18000 version: v1 versionPriority: 1 status: conditions: - lastTransitionTime: "2020- 08 - 16T05:35:33Z" message: Local APIServices are always available reason: Local status: "True" type: Available ``` ### 授权 ABAC有期局限性，针对每个account都需要做配置，并且需要重启apiserver RBAC更灵活，更符合我们通常熟知的权限管理 ### RBAC ![k8s_kube_APIServer_16](../img/k8s/k8s_kube_APIServer/k8s_kube_APIServer_16.jpg) ### 规划系统角色 User - 管理员 ➢ 所有资源的所有权限？？ - 普通用户 ➢ 是否有该用户创建的namespace下的所有object的操作权限？ ➢ 对其他用户的namespace资源是否可读，是否可写？ SystemAccount - SystemAccount是开发者（kubernetesdeveloper或者domaindeveloper）创建应用后，应 用于apiserver通讯需要的身份 - 用户可以创建自定的ServiceAccount，kubernetes也为每个namespace创建default ServiceAccount - DefaultServiceAccount通常需要给定权限以后才能对apiserver做写操作 ### 实现方案 在cluster创建时（kube-up.sh），创建自定义的role，比如namespace-creator namespace-creatorrole定义用户可操作的对象和对应的读写操作。 创建自定义的namespaceadmissioncontroller - 当namespace创建请求被处理时，获取当前用户信息并annotate（注释）到namespace 创建RBAC controller - Watchnamespace的创建事件 - 获取当前namespace的创建者信息 - 在当前namespace创建rolebinding对象，并将namespace-creator 角色和用户绑定 ### 与权限相关的其他最佳实践 ClusterRole是非namespace绑定的，针对整个集群生效 通常需要创建一个管理员角色，并且绑定给开发运营团队成员 ThirdPartyResource和CustomResourceDefinition是全局资源，普通用户创建ThirdPartyResource以后， 需要管理员授予相应权限后才能真正操作该对象 针对所有的角色管理，建议创建spec，用源代码驱动 - 虽然可以通过edit操作来修改权限，但后期会导致权限管理混乱，可能会有很多临时创建出来的 角色和角色绑定对象，重复绑定某一个资源权限 权限是可以传递的，用户A可以将其对某对象的某操作，抽取成一个权限，并赋给用户B 防止海量的角色和角色绑定对象，因为大量的对象会导致鉴权效率低，同时给apiserver增加负担 ServiceAccount也需要授权的，否则你的component可能无法操作某对象 Tips：SSH到master节点通过insecureport访问apiserver可绕过鉴权，当需要做管理操作又没有权限时可以 使用（不推荐） ## apimachinery **https://github.com/kubernetes/apimachineryhttps://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery->.** ### 回顾GKV Group Kind Version - Internelversion和Externalversion - 版本转换 ### 如何定义Group pkg/apis/core/register.go 定义group GroupName 定义groupversion ``` go var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: runtime.APIVersionInternal} ``` 定义SchemeBuilder ``` go var ( SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes) AddToScheme = SchemeBuilder.AddToScheme ) ``` 将对象加入SchemeBuild ``` go func addKnownTypes(scheme *runtime.Scheme) error { if err := scheme.AddIgnoredConversionType(&metav1.TypeMeta{}, &metav1.TypeMeta{}); err != nil { return err } scheme.AddKnownTypes(SchemeGroupVersion, &Pod{}, &PodList{}, }} ``` ### 定义对象类型 types.go List 单一对象数据结构 - TypeMeta - ObjectMeta - Spec - Status ### 代码生成Tags GlobalTags - 定义在doc.go中 ➢ // +k8s:deepcopy-gen=package LocalTags - 定义在types.go中的每个对象里 ➢ // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object ➢ // +genclient // +genclient:nonNamespaced // +genclient:noVerbs // +genclient:onlyVerbs=create,delete // +genclient:skipVerbs=get,list,create,update,patch,delete,deleteCollection,watc h // +genclient:method=Create,verb=create,result=k8s.io/apimachinery/pkg/apis/ meta/v1.Status ### 实现etcd storage pkg/registry/core/configmap/storage/storage.go ``` go funcNewREST(optsGetter generic.RESTOptionsGetter) *REST { store := &genericregistry.Store{ NewFunc: func() runtime.Object { return&api.ConfigMap{} }, NewListFunc: func() runtime.Object { return&api.ConfigMapList{} }, DefaultQualifiedResource: api.Resource("configmaps"), CreateStrategy: configmap.Strategy, UpdateStrategy: configmap.Strategy, DeleteStrategy: configmap.Strategy, TableConvertor: printerstorage.TableConvertor{TableGenerator: printers.NewTableGenerator().With(printersinternal.AddHandlers)}, } options := &generic.StoreOptions{RESTOptions: optsGetter} if err := store.CompleteWithOptions(options); err != nil { panic(err) // TODO: Propagate error up } return &REST{store} } ``` ### 创建和更新对象时的业务逻辑-Strategy ``` go func (strategy) PrepareForCreate(ctx context.Context, obj runtime.Object) { _ = obj.(*api.ConfigMap) } func (strategy) Validate(ctx context.Context, obj runtime.Object) field.ErrorList { cfg := obj.(*api.ConfigMap) return validation.ValidateConfigMap(cfg) } func (strategy) PrepareForUpdate(ctx context.Context, newObj, oldObj runtime.Object) { _ = oldObj.(*api.ConfigMap) _ = newObj.(*api.ConfigMap) } ``` ### subresource 什么是subresource，内嵌在kubernetes对象中，有独立的操作逻辑的属性集合，如podstatus ``` go statusStore.UpdateStrategy = pod.StatusStrategy var StatusStrategy = podStatusStrategy{Strategy} func (podStatusStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) { newPod := obj.(*api.Pod) oldPod := old.(*api.Pod) newPod.Spec = oldPod.Spec newPod.DeletionTimestamp = nil // don't allow the pods/status endpoint to touch owner references since old kubelets corrupt them in a way // that breaks garbage collection newPod.OwnerReferences = oldPod.OwnerReferences } ``` ### 注册APIGroup 定义Storage ``` go configMapStorage := configmapstore.NewREST(restOptionsGetter) restStorageMap := map[string]rest.Storage{ "configMaps": configMapStorage, } ``` 定义对象的StorageMap ``` go apiGroupInfo.VersionedResourcesStorageMap["v1"] = restStorageMap ``` 将对象注册至APIServer（挂载handler） ``` go if err := m.GenericAPIServer.InstallLegacyAPIGroup(genericapiserver.DefaultLegacyAPIPrefix, &apiGroupInfo); err != nil { klog.Fatalf("Error in registering group versions: %v", err) } ``` ### 代码生成 ``` deepcopy-gen ``` - 为对象生成DeepCopy方法，用于创建对象副本 client-gen - 创建Clientset，用于操作对象的CRUD informer-gen - 为对象创建Informer框架，用于监听对象变化 lister-gen - 为对象构建Lister框架，用于为Get和List操作，构建客户端缓存 coversion-gen - 为对象构建Conversion方法，用于内外版本转换以及不同版本号的转换 **https://github.com/kubernetes/code-generator**^83 ### hack/update-codegen.sh 依赖 BUILD_TARGETS=( vendor/k8s.io/code-generator/cmd/client-gen vendor/k8s.io/code-generator/cmd/lister-gen vendor/k8s.io/code-generator/cmd/informer-gen ) 生成命令 ${GOPATH}/bin/deepcopy-gen --input-dirs{versioned-package-pach} - O zz_generated.deepcopy\ --bounding-dirs{output-package-path} \ --go-header-file ${SCRIPT_ROOT}/hack/boilerplate.go.txt ### APIServer代码走读 https://cncamp.notion.site/kube-apiserver-10d5695cbbb14387b60c6d622005583d ================================================ FILE: docs/linux/linux.md ================================================ # Linux 学习笔记 ## 一、 常用命令 #### LINUX 常用操作命令和命令行编辑快捷键 #### 终端快捷键： ``` Ctrl + a/Home 切换到命令行开始 Ctrl + e/End 切换到命令行末尾 Ctrl + l 清除屏幕内容，效果等同于clear Ctrl + u 清除剪切光标之前的内容 Ctrl + k 剪切清除光标之后的内容 Ctrl + y 粘贴刚才所删除的字符 Ctrl + r 在历史命令中查找 （这个非常好用，输入关键字就调出以前的命令了） Ctrl + c 终止命令 ctrl + o 重复执行命令 Ctrl + d 退出shell，logout Ctrl + z 转入后台运行,但在当前用户退出后就会终止 Ctrl + t 颠倒光标所在处及其之前的字符位置，并将光标移动到下一个字符 Alt + t 交换当前与以前单词的位置 Alt + d 剪切光标之后的词 Ctrl+w 剪切光标所在处之前的一个词（以空格、标点等为分隔符） Ctrl+（x u） 按住Ctrl的同时再先后按x和u，撤销刚才的操作 Ctrl+s 锁住终端 Ctrl+q 解锁终端 !! 重复执行最后一条命令 history 显示你所有执行过的编号+历史命令。这个可以配合!编辑来执行某某命令 !$ 显示系统最近的一条参数 最后这个比较有用，比如我先用cat /etc/sysconfig/network-scripts/ifconfig-eth0，然后我想用vim编辑。 一般的做法是先用↑ 显示最后一条命令，然后用Home移动到命令最前，删除cat，然后再输入vim命 令。其实完全可以用vim !$来代替。 ``` #### gnome 快捷键 ``` Alt + F1 类似Windows下的Win键，在GNOME中打开"应用程序"菜单(Applications) Alt + F2 类似Windows下的Win + R组合键，在GNOME中运行应用程序 Ctrl + Alt + D 类似Windows下的Win + D组合键，显示桌面 Ctrl + Alt + L 锁定桌面并启动屏幕保护程序 Alt + Tab 同Windows下的Alt + Tab组合键，在不同程序窗口间切换 PrintScreen 全屏抓图 Alt + PrintScreen 当前窗口抓图 Ctrl + Alt + → / ← 在不同工作台间切换 Ctrl + Alt + Shift + → / ← 移动当前窗口到不同工作台 Ctrl+Alt+Fn 终端N或模拟终端N(n和N为数字 1 －6) ``` ``` Ctrl+Alt+F7 返回桌面 ``` #### 窗口操作快捷键 ``` Alt + F4 关闭窗口 Alt + F5 取消最大化窗口 (恢复窗口原来的大小) Alt + F7 移动窗口 (注: 在窗口最大化的状态下无效) Alt + F8 改变窗口大小 (注: 在窗口最大化的状态下无效) Alt + F9 最小化窗口 Alt + F10 最大化窗口 Alt + 空格键 打开窗口的控制菜单 (点击窗口左上角图标出现的菜单） ``` #### 文件浏览器 ``` Ctrl+N 新建窗口 Ctrl + Shift + W 关闭所有文件浏览器 Ctrl + 1/2 改变文件夹视图查看方式，图标视图/列表视图 Alt + → / ← 后退/前进 Alt + ↑/↓ 移动到父文件夹/选择的文件夹 Alt + Home 直接移动到主文件夹 F9 开关显示隐藏Nautilus侧边栏 Ctrl+H 显示隐藏文件（切换键） Shift+Ctrl+N 新建文件夹， 很有用 Alt + Enter 查看选择文件/文件夹的属性，代替单击右键选择属性 Ctrl+Page Up 上一个标签 Ctrl+Page Down 下一个标签 Alt+N 切换到第N个标签（N为数字） ``` #### 关机和重启命令 Shutdown Reboot Halt poweroff #### grep 和管道符 ``` 昨天的时候leader给我出了道问题： 找出文件夹下包含 “aaa” 同时不包含 “bbb”的文件，然后把他们重新生成一下。要求只能用一行命令。 我对Linux 是个白痴，工作了之后才开始接触的，会用的命令只有那几个常用的。这个问题对我来说就有 点难度，我只是大概知道查找文件用 grep ，其他的就不知道了。不过没关系，用Google ，查找到 grep 的完整用法： 1 、grep -l ‘boss’ * 显示所有包含boss的文件名。 2 、grep -n ‘boss’ file在匹配行之前加行号。 3 、grep -i ‘boss’ file显示匹配行，boss不区分大小写。 4 、grep -v ‘boss’ file显示所有不匹配行。 5 、grep -q ‘boss’ file找到匹配行，但不显示，但可以检查grep的退出状态。（ 0 为匹配成功） 6 、grep -c ‘boss’ file只显示匹配行数（包括 0 ）。 ``` ``` 7 、grep “$boss” file扩展变量boss的值再执行命令。 8 、ps -ef|grep “^*user1′′ 搜索user1的命令，即使它前面有零个或多个空格。 9 、ps -e|grep -E ‘grant_server|commsvr|tcpsvr|dainfo’ 查找多个字符串的匹配（grep -E相当于egrep） （来源：蓝森林） 了解了grep的参数之后，问题就解决了一半了，因为可以搜索出符合条件的文件了。不过光有grep 还是 不行，因为要把搜索出来的文件名作为参数传给 generate 命令。OK，接下来该管道符出场了。 即使是像我这样对Linux只是有一点了解的人也经常用到管道符，比如“|”，示例： ls -a | more 。但是对 于管道符的具体意义和它做了什么我就不知道了，没关系，Google 一下，找到一些资料： 利用 Linux 所提供的管道符 “|” 将两个命令隔开，管道符左边命令的输出就会作为管道符右边命令的输入。 连续使用管道意味着第一个命令的输出会作为第二个命令的输入，第二个命令的输出又会作为第三个命 令的输入，依此类推。 所以查找的时候可以这样写： grep -rl “aaa” * | grep -v “bbb” 这样右边的命令就可以从前面的结果中筛选了。然后还有 generate 命令，因为生成文件的命令格式是这 样的： generate 文件名 不过如果直接使用 generate grep -rl “aaa” * | grep -v “bbb” 的话会出错，因为命令会从左向右执行，这条 命令就会把grep作为一个文件名来看待。怎么办呢？这个时候就要使用 · (键盘上数字键 1 旁边的那个符 号，和“~”在一个按键上)来做命令替换了，用 · 把后面的 grep 命令包起来就好了，这样： generate ·grep -rl “aaa” * | grep -v “bbb”· 然后就搞定了。 工作一段时间之后，越来越喜欢Linux的哲学了，它有很多命令，看起来功能都不是那么强劲，但是如果 你开动脑筋把这些命令组合起来的话，就能实现 很多让你意想不到的功能，有时候你忍不住惊呼：实在 是太coooool了！ 这对于像我这种被Windows的傻瓜式操作惯坏了的人来说，是个福音，以后要多多开 动生锈了的大脑。如果单纯使用电脑的话，还是Windows好 用 ，但 是对于程序员，最好还是多玩玩 Linux。 BTW，现在也越来越喜欢使用VIM 了，虽然刚开始用的时候就觉得它是个记事本~囧~~ 以前总听说“真正 的牛人编码都是用记事本编写的”，当时就觉得这些人实在太厉害了，代码提示和自动补全都不用，现在 想想，可能是外行看到他们使用灵活+强 大的VIM或者 EMACS 了吧。^_^ 我的补充: 查找包含logField又包含open的文件： 用grep "logFileId" *.tbc|grep "open" ``` ## 二、 磁盘管理 #### 文件系统配置文件 /etc/filesystems：系统指定的测试挂载文件系统类型 /proc/filesystems：Linux系统已经加载的文件系统类型 /lib/modules/2.6.18-274.el5/kernel/fs/ 文件系统类型的驱动所在目录 /etc/fstab /etc/mtab #### linux 文件类型的颜色 linux文件颜色的含义：蓝色代表目录绿色代表可执行文件 红色表示压缩文件 浅蓝色表示链接文件 灰 色表示其他文件 红色闪烁表示链接的文件有问题了 黄色表示设备文件 ##### 蓝色文件----------目录 ##### 白色文件----------一般性文件，如文本文件，配置文件，源码文件等 浅蓝色文件----------链接文件，主要是使用ln命令建立的文件 绿色文件----------可执行文件，可执行的程序 红色文件-----------压缩文件或者包文件 Linux下用字符表示的文件类型 - ：普通文件 d：目录文件 l：链接文件 b：块设备文件 c：字符设备文件 p：管道文件 #### 文件系统操作命令 #### df ：列出文件系统的整体磁盘使用情况 [root@centos57 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/mapper/VolGroup00-LogVol 16G 4.2G 11G 28% / /dev/sda1 99M 13M 81M 14% /boot tmpfs 1005M 0 1005M 0% /dev/shm [root@centos57 ~]# df -i 文件系统 Inode (I)已用 (I)可用 (I)已用% 挂载点 /dev/mapper/VolGroup00-LogVol 4186112 154441 4031671 4% / /dev/sda1 26104 36 26068 1% /boot tmpfs 257210 1 257209 1% /dev/shm .host:/ 0 0 0 - /mnt/hgfs #### du ：列出目录所占空间 du -sh 显示当前目录大小 du –sh / 显示/目录下的所有目录大小 #### dumpe2fs ：显示当前的磁盘状态 #### ln ：连接文件（快捷方式） ln –sf 源文件 目标文件 不加任何参数就进行连接，就是hard link，加 上 -s就是Symbolic link，hard link不支持目录和跨文件系统。 #### Fdisk Fdisk不支持大于2T的磁盘 Fdisk –l 显示系统中的所有分区内容 **[root@centos57 ~]# fdisk -l** Disk /dev/sda: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders 总扇区数，可以和下面的最后扇区数比较，看剩余 Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 13 104391 83 Linux /dev/sda2 14 2610 20860402+ 8e Linux LVM **[root@centos57 ~]# fdisk /dev/sda** #### Parted ： 2T 以上磁盘分区工具 支持大于2T的磁盘 ，2T以下的最好还是用Fdisk来分区。 [root@centos57 aixi]# parted /dev/hda print Model: VMware Virtual IDE Hard Drive (ide) Disk /dev/hda: 2147MB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system 标志 1 32.3kB 101MB 101MB 主分区 ext **2** 101MB 357MB 256MB 主分区 linux-swap # parted /dev/hda rm **2** 删除第 **2** 个分区 **# parted /dev/hda mkpart primary ext3 120MB 200MB** 创建分区， **primary** 代表主分区，还可以是 **extended** 扩展分区， **logical** 逻辑分区 **;ext3** 代表分区类型， **120MB** 是开始位置，最好是接上一分区的结束位 置， **200M** 是结束位置 #### partprobe : 更新分区表 / 磁盘 用于重读分区表，当出现删除文件后,出现仍然占用空间。可以partprobe在不重启的情况下重读分区 # partprobe 这个命令执行完毕之后不会输出任何返回信息，你可以使用mke2fs命令在新的分区上创建文件系统。 #### Mkfs: 磁盘格式化 **Mkfs –t** 文件系统格式 设备文件名（盘符） [root@centos57 ~]# mkfs -t ext3 /dev/hda #### e2label ：设置磁盘卷标 e2label 设备名称 新label名称，可以用dumpe2fs查看卷标 [root@centos57 ~]# e2label /dev/hda1 aixi #### Mount: 挂载磁盘 ##### 命令格式： mount [-t vfstype] [-o options] device dir **mount –o remount,rw,auto /** 重新挂载 **mount –n –o remount,rw /** 重新挂载根目录，设置为可读写 其中： 1. -t vfstype 指定文件系统的类型，通常不必指定。mount 会自动选择正确的类型。常用类型有： 光盘或光盘镜像：iso DOS fat16文件系统：msdos Windows 9x fat32文件系统：vfat Windows NT ntfs文件系统：ntfs Mount Windows文件网络共享：smbfs UNIX(LINUX) 文件网络共享：nfs 2. -o options 主要用来描述设备或档案的挂接方式。常用的参数有： loop：用来把一个文件当成硬盘分区挂接上系统 ro：采用只读方式挂接设备 rw：采用读写方式挂接设备 iocharset：指定访问文件系统所用字符集 3.device 要挂接(mount)的设备。 4.dir设备在系统上的挂接点(mount point)。 挂接光盘镜像文件 1 、从光盘制作光盘镜像文件。将光盘放入光驱，执行下面的命令。 #cp /dev/cdrom /home/sunky/mydisk.iso 或 #dd if=/dev/cdrom of=/home/sunky/mydisk.iso 注：执行上面的任何一条命令都可将当前光驱里的光盘制作成光盘镜像文件/home/sunky/mydisk.iso 2 、将文件和目录制作成光盘镜像文件，执行下面的命令。 #mkisofs -r -J -V mydisk -o /home/sunky/mydisk.iso /home/sunky/ mydir 注：这条命令将/home/sunky/mydir目录下所有的目录和文件制作成光盘镜像文件 /home/sunky/mydisk.iso，光盘卷标为：mydisk 3 、光盘镜像文件的挂接(mount) #mkdir /mnt/vcdrom 注：建立一个目录用来作挂接点(mount point) **#mount -o loop -t iso9660 /home/sunky/mydisk.iso /mnt/vcdrom** 注：使用/mnt/vcdrom就可以访问盘镜像文件mydisk.iso里的所有文件了。 ##### 挂接移动硬盘 对linux系统而言，USB接口的移动硬盘是当作SCSI设备对待的。插入移动硬盘之前，应先用fdisk –l 或 more /proc/partitions查看系统的硬盘和硬盘分区情况。 [root at pldyrouter /]# fdisk -l Disk /dev/sda: 73 dot 4 GB, 73407820800 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 4 32098+ de Dell Utility /dev/sda2 * 5 2554 20482875 7 HPFS/NTFS /dev/sda3 2555 7904 42973875 83 Linux /dev/sda4 7905 8924 8193150 f Win95 Ext'd (LBA) /dev/sda5 7905 8924 8193118+ 82 Linux swap 在这里可以清楚地看到系统有一块SCSI硬盘/dev/sda和它的四个磁盘分区/dev/sda1 -- /dev/sda4, /dev/sda5是分区/dev/sda4的逻辑分区。接好移动硬盘后，再用fdisk –l 或 more /proc/partitions查看系统 的硬盘和硬盘分区情况 [root at pldyrouter /]# fdisk -l Disk /dev/sda: 73 dot 4 GB, 73407820800 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 4 32098+ de Dell Utility /dev/sda2 * 5 2554 20482875 7 HPFS/NTFS /dev/sda3 2555 7904 42973875 83 Linux /dev/sda4 7905 8924 8193150 f Win95 Ext'd (LBA) /dev/sda5 7905 8924 8193118+ 82 Linux swap Disk /dev/sdc: 40.0 GB, 40007761920 bytes 255 heads, 63 sectors/track, 4864 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 510 4096543+ 7 HPFS/NTFS /dev/sdc2 511 4864 34973505 f Win95 Ext'd (LBA) /dev/sdc5 511 4864 34973473+ b Win95 FAT 大家应该可以发现多了一个SCSI硬盘/dev/sdc和它的两个磁盘分区/dev/sdc1?、/dev/sdc2,其中/dev/sdc 是/dev/sdc2分区的逻辑分区。我们可以使用下面的命令挂接/dev/sdc1和/dev/sdc5。 #mkdir -p /mnt/usbhd #mkdir -p /mnt/usbhd 注：建立目录用来作挂接点(mount point) #mount -t ntfs /dev/sdc1 /mnt/usbhd #mount -t vfat /dev/sdc5 /mnt/usbhd 注：对ntfs格式的磁盘分区应使用-t ntfs 参数，对fat32格式的磁盘分区应使用-t vfat参数。若汉字 文件名显示为乱码或不显示，可以使用下面的命令格式。 #mount -t ntfs -o iocharset=cp936 /dev/sdc1 /mnt/usbhd #mount -t vfat -o iocharset=cp936 /dev/sdc5 /mnt/usbhd linux系统下使用fdisk分区命令和mkfs文件系统创建命令可以将移动硬盘的分区制作成linux系统所特 有的ext2、ext3格式。这样，在linux下使用就更方便了。使用下面的命令直接挂接即可。 #mount /dev/sdc1 /mnt/usbhd ##### 挂接 U 盘 和USB接口的移动硬盘一样对linux系统而言U盘也是当作SCSI设备对待的。使用方法和移动硬盘完全 一样。插入U盘之前，应先用fdisk –l 或 more /proc/partitions查看系统的硬盘和硬盘分区情况。 [root at pldyrouter root]# fdisk -l Disk /dev/sda: 73 dot 4 GB, 73407820800 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 4 32098+ de Dell Utility /dev/sda2 * 5 2554 20482875 7 HPFS/NTFS /dev/sda3 2555 7904 42973875 83 Linux /dev/sda4 7905 8924 8193150 f Win95 Ext'd (LBA) /dev/sda5 7905 8924 8193118+ 82 Linux swap 插入U盘后，再用fdisk –l 或 more /proc/partitions查看系统的硬盘和硬盘分区情况。 [root at pldyrouter root]# fdisk -l Disk /dev/sda: 73 dot 4 GB, 73407820800 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 4 32098+ de Dell Utility /dev/sda2 * 5 2554 20482875 7 HPFS/NTFS /dev/sda3 2555 7904 42973875 83 Linux /dev/sda4 7905 8924 8193150 f Win95 Ext'd (LBA) /dev/sda5 7905 8924 8193118+ 82 Linux swap Disk /dev/sdd: 131 MB, 131072000 bytes 9 heads, 32 sectors/track, 888 cylinders Units = cylinders of 288 * 512 = 147456 bytes Device Boot Start End Blocks Id System /dev/sdd1 * 1 889 127983+ b Win95 FAT Partition 1 has different physical/logical endings: phys=(1000, 8, 32) logical=(888, 7, 31) 系统多了一个SCSI硬盘/dev/sdd和一个磁盘分区/dev/sdd1,/dev/sdd1就是我们要挂接的U盘。 #mkdir -p /mnt/usb 注：建立一个目录用来作挂接点(mount point) #mount -t vfat /dev/sdd1 /mnt/usb 注：现在可以通过/mnt/usb来访问U盘了, 若汉字文件名显示为乱码或不显示，可以使用下面的命令。 #mount -t vfat -o iocharset=cp936 /dev/sdd1 /mnt/usb 挂接 **Windows** 文件共享 Windows网络共享的核心是SMB/CIFS，在linux下要挂接(mount)windows的磁盘共享，就必须安装和使 用samba 软件包。现在流行的linux发行版绝大多数已经包含了samba软件包，如果安装linux系统时未安 装samba请首先安装samba。当然也可以到 [http://www.samba.org网站下载......新的版本是3.0.10版。](http://www.samba.org网站下载......新的版本是3.0.10版。) 当windows系统共享设置好以后，就可以在linux客户端挂接(mount)了，具体操作如下： # mkdir –p /mnt/samba 注：建立一个目录用来作挂接点(mount point) # mount -t smbfs -o username=administrator,password=pldy123 //10.140.133.23/c$ /mnt/samba 注：administrator 和 pldy123 是ip地址为10.140.133.23 windows计算机的一个用户名和密码，c$是这 台计算机的一个磁盘共享 如此就可以在linux系统上通过/mnt/samba来访问windows系统磁盘上的文件了。以上操作在redhat as server 3、redflag server 4.1、suse server 9以及windows NT 4.0、windows 2000、windows xp、windows 2003 环境下测试通过。 挂接 **UNIX** 系统 **NFS** 文件共享 类似于windows的网络共享，UNIX(Linux)系统也有自己的网络共享，那就是NFS(网络文件系统)，下面 我们就以SUN Solaris2.8和REDHAT as server 3 为例简单介绍一下在linux下如何mount nfs网络共享。 在linux客户端挂接(mount)NFS磁盘共享之前，必须先配置好NFS服务端。 1 、Solaris系统NFS服务端配置方法如下： (1)修改 /etc/dfs/dfstab, 增加共享目录 share -F nfs -o rw /export/home/sunky (2)启动nfs服务 # /etc/init.d/nfs.server start ##### (3)NFS服务启动以后，也可以使用下面的命令增加新的共享 # share /export/home/sunky # share /export/home/sunky 注：/export/home/sunky和/export/home/sunky1是准备共享的目录 2 、linux系统NFS服务端配置方法如下： (1)修改 /etc/exports,增加共享目录 /export/home/sunky 10.140.133.23(rw) /export/home/sunky1 *(rw) /export/home/sunky2 linux-client(rw) 注：/export/home/目录下的sunky、sunky1、sunky2是准备共享的目录，10.140.133.23、*、 linux-client 是被允许挂接此共享linux客户机的IP地址或主机名。如果要使用主机名linux-client必须在服务端主机 /etc/hosts文件里增加linux-client主机ip定义。格式如下： 10.140.133.23 linux-client (2)启动与停止NFS服务 /etc/rc.d/init.d/portmap start (在R E D H AT中PORTMAP是默认启动的) /etc/rc.d/init.d/nfs start 启动NFS服务 /etc/rc.d/init.d/nfs stop 停止NFS服务 注：若修改/etc/export文件增加新的共享，应先停止NFS服务，再启动NFS服务方能使新增加的共享 起作用。使用命令exportfs -rv也可以达到同样的效果。 3 、linux客户端挂接(mount)其他linux系统或UNIX系统的NFS共享 # mkdir –p /mnt/nfs 注：建立一个目录用来作挂接点(mount point) #mount -t nfs -o rw 10.140.133.9:/export/home/sunky /mnt/nfs 注：这里我们假设10.140.133.9是NFS服务端的主机IP地址，当然这里也可以使用主机名，但必须 在本机/etc/hosts文件里增加服务端ip定义。/export/home/sunky为服务端共享的目录。 如此就可以在linux客户端通过/mnt/nfs来访问其它linux系统或UNIX系统以NFS方式共享出来的文件 了。以上操作在 redhat as server 3、redflag server4.1、suse server 9以及Solaris 7、Solaris 8、Solaris 9 for x86&sparc环境下测试通过。 权限问题： 假设 server 端的使用者 jack, user id 为 1818, gid 为 1818, client 端也有一个使用者 jack，但 是 uid 及 gid 是 1818 。client 端的 jack 希望能完全读写 server 端的 /home/jack 这个目录。server 端的 /etc/exports 是 这样写的： /home/jack *(rw,all_squash,anonuid=1818,anongid=1818) 这个的配置文件的意思是，所有 client 端的使用者存取 server 端 /home/jack 这 目录时，都会 map 成 server 端的 jack (uid,gid=1818)。我 mount 的结果是 1. client 端的 root 可以完全存取该目录, 包括读、写、杀......等 2. client 端的 jack (uid,gid=1818) 我可以做： rm -rf server_jack/* cp something server_jack/ mkdir server_jack/a #### umount ：将文件设备卸载 [root@centos57 ~]# umount /dev/hda1 用设备文件名来卸载 [root@centos57 ~]# umount /aixi 用挂载点来卸载 umount: /aixi: device is busy 如果提示设备忙，不急可以使用如下命令卸载 **#** umount -l /mnt/hda 1 选项 – l 并不是马上umount，而是在该目录空闲后再umount。 如果比较急，可用如下命令： **#** umount -f /mnt/hda 1 -f代表强制 卸载 如果还不行，可使用fuser -m –v /dev/hda1来查询是哪些程序在占用，结束这些程序进程即可卸载 [root@centos57 aixi]# sync && fuser -m /dev/hda1 –k 使用这条命令后一定可以卸载 #### 交换分区 ##### 交换分区最大容量为64G，最多只能建 32 个, ##### 创建交换分区 #fdisk /dev/hdan+容量pt(修改系统ID)分区号 82 pw #mkswap /dev/hda2（以上划分的分区号） 构建swap格式 #swapon /dev/hda2 加载即完成增加swap #swapon –s 显示目前使用的Swap设备 创建交换文件 # dd if=/dev/hda1 of=/aixi/swap bs=1M count=64 创建大文件 #mkswap /aixi/swap #swapon /aixi/swap 完成 取消交换分区或者交换文件 #swapon –s 显示目前使用的Swap设备 #swapoff /aixi/swap #swapoff /dev/hda #free –m查看 ## 三、 用户管理 #### 用户和用户组操作命令 #### Id #### Finger #### Pwck 检查/etc/passwd配置文件内的信息与实际主文件夹是否存在，还可比较/etc/passwd和/etc/shadow的信 息是否一致，另外如果/etc/passwd中的数据字段错误也会提示。 #### Grpck 和pwck功能相近，这是检查用户组的。 #### Groups #### newgrp #### useradd #### usermod usermod 不仅能改用户的SHELL类型，所归属的用户组，也能改用户密码的有效期，还能改登录名。usermod 如此看来就 是能做到用户帐号大转移；比如我把用户A改为新用户B； usermod [-u uid [-o]] [-g group] [-G group,...] [-d 主目录 [-m]] [-s shell] [-c 注释] [-l 新名称] [-f 失效日] [-e 过期日] [- p 密码] [- L|-U] 用户名 usermod 命令会参照你命令列上指定的部份修改系统帐号档。下列为usermod可选用的参数。 -c comment 更新用户帐号password档中的注解栏，一般是使用chfn(1)来修改。 -d home_dir 更新用户新的登入目录。如果给定-m选项，用户旧目录会搬到新的目录去，如旧目录不存在则建个新的。 -e expire_date 加上用户帐号停止日期。日期格式为MM/DD/YY. -f inactive_days 帐号过期几日后永久停权。当值为 0 时帐号则立刻被停权。而当值为- 1 时则关闭此功能。预设值为- 1 。 -g initial_group 更新用户新的起始登入用户组。用户组名须已存在。用户组ID必须参照既有的的用户组。用户组ID预设 值为 1 。 -G group,[...] 定义用户为一堆groups的成员。每个用户组使用","区格开来，不可以夹杂空白字元。用户组名同-g选项的 限制。如果用户现在的用户组不再此列，则将用户由该用户组中移除。 -l login_name 变更用户login时的名称为login_name。其它不变。特别是，用户目录名应该也会跟着更动成新的登入名。 -s shell 指定新登入shell。如此栏留白，系统将选用系统预设shell。 -u uid用户ID值。必须为唯一的ID值，除非用-o选项。数字不可为负值。预设为最小不得小于/etc/login.defs中定义的 UID_MIN值。 0 到UID_MIN值之间是传统上保留给系统帐号使用。用户目录树下所有的档案目录其userID会自动改变。放在 用户目录外的档案则要自行手动更动。 警告:usermod不允许你改变正在线上的用户帐号名称。当usermod用来改变userID,必须确认这名user没在电脑上执行任何 程序。你需手动更改用户的crontab档。也需手动更改用户的at工作档。采用NISserver须在server上更动相关的NIS设 定。 举个简单的例子，我们在前面说了关于useradd的工具，而usermod 工具和useradd的参数差不多；两者不同之处在于 useradd是添加，usermod 是修改； [root@localhost ~]# usermod -d /opt/linuxfish -m -l fishlinux -U linuxfish 注：把linuxfish 用户名改为fishlinux ，并且把其家目录转移到 /opt/linuxfish ； [root@localhost ~]# ls -la /opt/linuxfish/ 注：查看用户fishlinux的家目录下的文件及属主； 总用量 48 drwxr-xr-x 3 fishlinux linuxfish 4096 11月 5 16:. drwxrwxrwx 29 root root 4096 11月 5 16: .. -rw-r--r-- 1 fishlinux linuxfish 24 11月 5 16:46 .bash_logout -rw-r--r-- 1 fishlinux linuxfish 191 11月 5 16:46 .bash_profile -rw-r--r-- 1 fishlinux linuxfish 124 11月 5 16:46 .bashrc -rw-r--r-- 1 fishlinux linuxfish 5619 11月 5 16:46 .canna -rw-r--r-- 1 fishlinux linuxfish 438 11月 5 16:46 .emacs -rw-r--r-- 1 fishlinux linuxfish 120 11月 5 16:46 .gtkrc drwxr-xr-x 3 fishlinux linuxfish 4096 11月 5 16:46 .kde -rw-r--r-- 1 fishlinux linuxfish 0 11月 5 16:46 mydoc.txt -rw-r--r-- 1 fishlinux linuxfish 658 11月 5 16:46 .zshrc [root@localhost ~]# more /etc/passwd |grep fishlinux 注：查看有关fishlinux的记录； fishlinux:x:512:512::/opt/linuxfish:/bin/bash 通过上面的例子，我们发现文件的用户组还没有变，如果您想改变为fishlinux用户组，如果想用通过 usermod来修改，就要 先添加fishlinux用户组；然后用usermod -g 来修改 ，也可以用chown -R fishlinux:fishlinux /opt/finshlinux 来改； 警告： usermod 最好不要用它来改用户的密码，因为他在/etc/shadow中显示的是明口令；修改用户的口令最好用passwd ； [root@localhost ~]# usermod -p 123456 fishlinux 注：修改fishlinux的口令是 123456 ； [root@localhost ~]# more /etc/shadow |grep fishlinux 注：查询/etc/shadow文件中fishlinux的口令；我们看到明显 是没有加密； fishlinux:123456:13092:0:99999:7::: #### userdel userdel很简单，只有一个参数可选 -r ；如果加参数-r ，表示在删除用户的同时，一并把用户的家目录及本地邮件存储的目 录或文件也一同删除；比如我们现在有两个用户bnnb和lanhaitun，其家目录都位于/home目录中，现在我们来删除这两个 用户； [root@localhost ~]# userdel bnnb 注：删除用户bnnb，但不删除其家目录及文件； [root@localhost ~]# ls -ld /home/bnnb 注：查看其家目录是否存在； drwxr-xr-x 14 501 501 4096 8 月 29 16:33 /home/bnnb 注：存在； [root@localhost ~]# ls -ld /home/lanhaitun 注：查看lanhaitun家目录是否存在； drwx------ 4 lanhaitun lanhaitun 4096 11月 5 14:50 /home/lanhaitun 注：存在； [root@localhost ~]# userdel -r lanhaitun 注：删除用户lanhaitun，其家目录及文件一并删除； [root@localhost ~]# ls -ld /home/lanhaitun 注：查看是否在删除lanhaitun 用户的同时，也一并把其家目录和文件一 同删除； ls: /home/lanhaitun: 没有那个文件或目录 注：已经删除； 警告： 请不要轻易用-r参数；他会删除用户的同时删除用户所有的文件和目录，切记；如果用户目录下有重要的文件，在删 除前请备份； 其实也有最简单的办法，但这种办法有点不安全，也就是直接在/etc/passwd中删除您想要删除用户的记录；但最好不要这样 做，/etc/passwd 是极为重要的文件，可能您一不小心会操作失误； #### Groupadd #### groupmod #### groupdel 是用来删除用户组的； 语法格式：groupdel 用户组 比如： [root@localhost ~]# groupdel lanhaitun #### passwd passwd 作为普通用户和超级权限用户都可以运行，但作为普通用户只能更改自己的用户密码，但前提是没有被root用户锁 定；如果root用户运行passwd ，可以设置或修改任何用户的密码； passwd 命令后面不接任何参数或用户名，则表示修改当前用户的密码；请看下面的例子； [root@localhost ~]# passwd 注：没有加任何用户，我是用root用户来执行的passwd 表示修改root用户的密码；下面 也有提示； Changing password for user root. New UNIX password: 注：请输入新密码； Retype new UNIX password: 注：验证新密码； passwd: all authentication tokens updated successfully. 注：修改root密码成功； 如果是普通用户执行passwd 只能修改自己的密码； 如果新建用户后，要为新用户创建密码，则用 passwd 用户名 ，注意要以root用户的权限来创建； [root@localhost ~]# passwd beinan 注：更改或创建beinan用户的密码； Changing password for user beinan. New UNIX password: 注：请输入新密码； Retype new UNIX password: 注：再输入一次； passwd: all authentication tokens updated successfully. 注：成功； 普通用户如果想更改自己的密码，直接运行passwd即可；比如当前操作的用户是beinan； [beinan@localhost ~]$ passwd Changing password for user beinan. 注：更改beinan用户的密码； (current) UNIX password: 注：请输入当前密码； New UNIX password: 注：请输入新密码； Retype new UNIX password: 注：确认新密码； passwd: all authentication tokens updated successfully. 注：更改成功； #### passwd 几个比较重要的参数； [root@localhost beinan]# passwd --help Usage: passwd [OPTION...] -k, --keep -tokens keep non-expired authentication tokens 注：保留即将过期的用户在期满后能仍能使用； -d, --delete delete the password for the named account (root only) 注：删除用户密码，仅能以root权限操作； -l, --lock lock the named account (root only) 注：锁住用户无权更改其密码，仅能通过root权限操作； -u, --unlock unlock the named account (root only) 注：解除锁定； -f, --force force operation 注：强制操作；仅root权限才能操作； -x, --maximum=DAYS maximum password lifetime (root only) 注：两次密码修正的最大天数，后面接数字；仅能 root权限操作； -n, --minimum=DAYS minimum password lifetime (root only) 注：两次密码修改的最小天数，后面接数字，仅能 root权限操作； -w, --warning=DAYS number of days warning users receives before 注：在距多少天提醒用户修改密码；仅能 root权限操作； password expiration (root only) -i, --inactive=DAYS number of days after password expiration when an 注：在密码过期后多少天，用户被禁 掉，仅能以root操作； account becomes disabled (root only) -S, --status report password status on the named account (root 注：查询用户的密码状态，仅能root 用户操作； only) --stdin read new tokens from stdin (root only) 比如我们让某个用户不能修改密码，可以用-l 参数来锁定： [root@localhost ~]# passwd -l beinan 注：锁定用户beinan不能更改密码； Locking password for user beinan. passwd: Success 注：锁定成功； [beinan@localhost ~]# su beinan 注：通过su切换到beinan用户； [beinan@localhost ~]$ passwd 注：beinan来更改密码； Changing password for user beinan. Changing password for beinan (current) UNIX password: 注：输入beinan的当前密码； passwd: Authentication token manipulation error 注：失败，不能更改密码； 再来一例： [root@localhost ~]# passwd -d beinan 注：清除beinan用户密码； Removing password for user beinan. passwd: Success 注：清除成功； [root@localhost ~]# passwd -S beinan 注：查询beinan用户密码状态； Empty password. 注：空密码，也就是没有密码； 注意： 当我们清除一个用户的密码时，登录时就无需密码；这一点要加以注意； #### chage 修改用户密码有效期限的命令； chage 用语法格式： chage [-l] [ -m 最小天数] [- M 最大天数] [-W 警告] [-I 失效日] [-E 过期日] [-d 最后日] 用户 前面已经说的好多了，这个只是一笔带过吧，知道有这个命令就行，自己实践实践再说，大体和psswd有些参数的用法差不 多； #### id 工具 : 查询用户所对应的 UID 和 GID 及 GID 所对应的用户组； id 工具是用来查询用户信息，比如用户所归属的用户组，UID 和GID等；id 用法极为简单；我们举个例子说明一下； 语法格式： id [参数] [用户名] 至于有哪些参数，自己查一下 id --help 或man id ；如 果id后面不接任何参数和任何用户，默认显示当前操作用户的用户名、 所归属的用户组、UID和GID等； 实例一：不加任何参数和用户名； [beinan@localhost ~]$ id uid=500(beinan) gid=500(beinan) groups=500(beinan) 注解：在没有加任何参数的情况下，查询的是当前操作用户的用户名、UID 、GID 和所处的主用户组和附属用户组；在本例 中，用户名是beinan，UID是500,所归属的主用户组是beinan，GID是 500 ； 实例二： id 后面接用户名； 如果我们想查询系统中用户的UID和GID 相应的内容，可以直接接用户名，但用户名必须是真实的 ，能在/etc/passwd中查 到的； [beinan@localhost ~]$ id linuxsir uid=505(linuxsir) gid=502(linuxsir) groups=502(linuxsir),0(root),500(beinan) 注解：查询用户linuxsir 的信息，用户linuxsir ，UID 为 505 ，所归属的主用户组是linuxsir，主用户组的GID是 502 ；同时 linuxsir用户也是GID为 0 的root用户组成员，也是GID为 500 用户组beinan的成员； 这个例子和实例一在用户组方面有所不同，我们在 《Linux 用户（user）和用户组（group）管理概述》 中有提到；用户和 用户组的对应关系，可以是一对一、一对多、多对一、或多对多的交叉关系，请参考之；另外您还需要掌握《用户（user） 和用户组（group）配置文件详解》一文； #### 2 、finger 工具：用来查询用户信息，侧重用户家目录、登录SHELL等； finger 工具侧重于用户信息的查询；查询的内容包括用户名（也被称为登录名Login），家目录，用户真实的名字（Name）... ... 办公地址、办公电话；也包括登录终端、写状态、空闭时间等； 我们最常用finger 来查询用户家目录、用户真实名、所用SHELL 类型、以及办公地址和电话，这是以参数 -l 长格式输出的； 而修改用户的家目录、真实名字、办公地址及办公电话，我们一般要能过chfn命令进行； 语法格式： finger [参数选项] [用户名］ -l 采用长格式（默认），显示由-s选项所包含的所有信息，以及主目录、办公地址、办公电话、登录SHELL、邮件状 态、.plan、.project和.forward； -m 禁止对用户真实名字进行匹配； -p 把.plan和.project文件中的内容省略； -s 显示短格式，用户名（也被称为登录名Login）、真实名字（NAME）、在哪个终端登录（Tty）、写状态、空闲时间（Idle）、 登录时间（Login Time）、办公地点、办公电话等； 至于finger 有哪些参数，您可以通过 finger --help 或man finger 来获取，我们在本文中以实例讲述最常用的参数； 实例一：不接任何参数，也不指定查询用户名；默认为加了-s参数； [beinan@localhost ~]$ finger Login Name Tty Idle Login Time Office Office Phone beinan beinan sun tty1 1:39 Nov 2 08: linuxsir linuxsir open tty2 2 Nov 2 10:03 linuxsir o +1-389- 866- 771 等价命令 [beinan@localhost ~]$ finger -s 注解：不加任何参数，也没有指定查询哪个用户，finger 会以默认以短格-s 来输出登录本机的所有用户的用户名（也被称为 登录名Login）、真实名字（NAME）、在哪个终端登录（Tty）、写状态、空闲时间（Idle）、登录时间（Login Time）、办公地 点、办公电话等； 在这个例子中，有beinan用户登录，真实名字是beinan sun （这个名字是用户的真实名字，如果在添加用户时没有设置， 是不会显示的），在tty1终端登录，空闭时间是 1 分 39 秒，登录时间是Nov /2/08:27 ，没有办公室名称，没有办公电话； 请对照本例中beinan用户记录的解说，我们来看看本例中的 linuxsir用户信息；应该不难。 关于写状态，如果在Tty 后面 没有任何输出，表示正在写入，如果有*出现，表示没有写入或被禁止，比如下面的例子，ftp 用户没有通过终端登录系统，因为Tty是*，同时Tty后面还有一个* ，表示禁止写入或没有写入状态(当用户没有登录时）； [beinan@localhost ~]$ finger -s ftp Login Name Tty Idle Login Time Office Office Phone ftp FTP User * * No logins 我们可以以短格式的来查询某个用户信息以短格式输出，比如下面的例子； [beinan@localhost ~]$ finger -s beinan 实例二：关于长格式的用户信息的输出 -l 参数的实例； finger -l 如果不加用户名的情况下，可以列出所有通过tty登录的用户信息；如果您想查询某个用户，就直接指定用户，可以 指定一个或多个；什么是tty登录？如果您在全屏文本界面操作的话，您可以通过按CTRL+F2或CTRL+F3 或CTRL+F4等， 以几个不同的用户登录到主机上，您就会看到，每个用户都有不同的tty； [beinan@localhost ~]$ finger -l [beinan@localhost ~]$ finger -l beinan linuxsir 注：可以同时查询几个用户信息，以长格式输出； [beinan@localhost ~]$ finger beinan Login: beinan Name: beinan sun Directory: /home/beinan Shell: /bin/bash On since Wed Nov 2 08:27 (CST) on tty1 2 hours 29 minutes idle On since Wed Nov 2 10:50 (CST) on pts/0 from :0. No mail. No Plan. 在本例中，所查询的用户是beinan，真实名字是beinan sun ，家目录位于 /home/beinan ，所用SHELL类型是bash ；然 后就是通过哪个终端登录的，登录时间，是不是有mail ，有Plan 等； 实例三：参数组合的例子； [beinan@localhost ~]$ finger -lp beinan Login: beinan Name: beinan sun Directory: /home/beinan Shell: /bin/bash On since Wed Nov 2 08:27 (CST) on tty1 2 hours 36 minutes idle On since Wed Nov 2 10:50 (CST) on pts/0 from :0. No mail. 注解：查询beinan用户信息，以长格式输出，并且不输出.Plan和.Project的内容； 实例四： finger -s 和w 及who的比较； 对于finger 就说这么多吧，极为简单的工具，当用到-s 参数时，您最好和w和who工具对照，看看finger -s 和w 及who 的输出有什么异同，w和who是查询哪些用户登录主机的；而finger -s 呢，无论是登录还是不登录的用户都可以查；但所查 到的内容侧重有所不同；自己看看例子； [beinan@localhost ~]$ finger -s Login Name Tty Idle Login Time Office Office Phone beinan beinan sun tty1 3:03 Nov 2 08: beinan beinan sun pts/0 Nov 2 10:50 (:0.0) linuxsir linuxsir open tty2 1:26 Nov 2 10:03 linuxsir o +1-389- 866- 771 [beinan@localhost ~]$ w 11:30:36 up 3:04, 3 users, load average: 0.30, 0.15, 0. USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT beinan tty1 - 08:27 3:03m 2:52 0.00s /bin/sh /usr/X11R6/bin/startx linuxsir tty2 - 10:03 1:26m 0.01s 0.01s -bash beinan pts/0 :0.0 10:50 0.00s 0.16s 0.00s w [beinan@localhost ~]$ who beinan tty1 Nov 2 08: linuxsir tty2 Nov 2 10: beinan pts/0 Nov 2 10:50 (:0.0) #### 3 、查询登录主机的用户工具：w 、who 、users w、who和users工具，是查询已登录当前主机的用户；另外finger -s 也同样能查询；侧重点不一样；请自己对比着看；毕 竟简单，这里只是介绍 ； [beinan@localhost ~]$ w 12:09:56 up 3:43, 7 users, load average: 0.16, 0.10, 0. USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT beinan tty1 - 08:27 3:42m 3:09 0.00s /bin/sh /usr/X11R6/bin/startx linuxsir tty2 - 10:03 2:06m 0.01s 0.01s -bash beinan pts/0 :0.0 11:36 1:09 0.15s 0.15s bash beinan pts/1 :0.0 11:37 1:12 0.21s 0.21s bash beinan pts/2 :0.0 12:02 6:52 0.09s 0.09s bash beinan pts/3 :0.0 12:05 12.00s 0.11s 0.06s ssh xmbnnbdl@linuxsir.org -p 17007 beinan pts/4 :0.0 12:06 0.00s 0.21s 0.00s w [beinan@localhost ~]$ who beinan tty1 Nov 2 08: linuxsir tty2 Nov 2 10: beinan pts/0 Nov 2 11:36 (:0.0) beinan pts/1 Nov 2 11:37 (:0.0) beinan pts/2 Nov 2 12:02 (:0.0) beinan pts/3 Nov 2 12:05 (:0.0) beinan pts/4 Nov 2 12:06 (:0.0) [beinan@localhost ~]$ users beinan beinan beinan beinan beinan beinan linuxsir #### 4 、groups 用户所归属的用户组查询； groups 用法很简单，就是查询用户所归属哪个或哪些用户组； 语法格式： groups 用户名 实例： [beinan@localhost ~]$ groups beinan 注：查询beinan所归属的用户组； beinan : beinan 注：beinan 是beinan用户组下的成员； [beinan@localhost ~]$ groups linuxsir 注：查询linuxsir用户所归属的用户组； linuxsir : linuxsir root beinan 注：linuxsir用户是linuxsir用户组、beinan用户组、root用户组成员； groups 主要是查询用户所归属的用户组名，最好和id命令相对比；这样对这两个工具都有所了解 #### 相关配置文件 /etc/passwd /etc/shadow /etc/gshadow /etc/group #### Linux 用户密码策略 Linux用户密码的有效期,是否可以修改密码可以通过login.defs文件控制.对login.defs文件修只影响后续 建立的用户,如果要改变以前建立的用户的有效期等可以使用chage命令. Linux用户密码的复杂度可以通过pam pam_cracklib module或pam_passwdqc module控制,两者不能同时 使用. 个人感觉pam_passwdqc更好用. /etc/login.defs密码策略 PASS_MAX_DAYS 99999 #密码的最大有效期, 99999:永久有期 PASS_MIN_DAYS 0 #是否可修改密码,0可修改,非 0 多少天后可修改 PASS_MIN_LEN 5 #密码最小长度,使用pam_cracklib module,该参数不再有效 PASS_WARN_AGE 7 #密码失效前多少天在用户登录时通知用户修改密码 pam_cracklib主要参数说明: tretry=N:重试多少次后返回密码修改错误 difok=N:新密码必需与旧密码不同的位数 dcredit=N: N >= 0:密码中最多有多少个数字;N < 0密码中最少有多少个数字. lcredit=N:小宝字母的个数 ucredit=N大宝字母的个数 credit=N:特殊字母的个数 minclass=N:密码组成(大/小字母,数字,特殊字符) pam_passwdqc主要参数说明: mix:设置口令字最小长度，默认值是mix=disabled。 max:设置口令字的最大长度，默认值是max=40。 passphrase:设置口令短语中单词的最少个数，默认值是passphrase=3，如果为 0 则禁用口令短语。 atch:设置密码串的常见程序，默认值是match=4。 similar:设置当我们重设口令时，重新设置的新口令能否与旧口令相似，它可以是similar=permit允 许相似或similar=deny不允许相似。 random:设置随机生成口令字的默认长度。默认值是random=42。设为 0 则禁止该功能。 enforce:设置约束范围，enforce=none表示只警告弱口令字，但不禁止它们使用；enforce=users将 对系统上的全体非根用户实行这一限制；enforce=everyone将对包括根用户在内的全体用户实行这 一限制。 non-unix:它告诉这个模块不要使用传统的getpwnam函数调用获得用户信息， retry:设置用户输入口令字时允许重试的次数，默认值是retry= 密码复杂度通过/etc/pam.d/system-auth实施 如: 要使用pam_cracklib将注释去掉,把pam_passwdqc.so注释掉即可. #password requisite /lib/security/$ISA/pam_cracklib.so retry=3 difok= password requisite /lib/security/$ISA/pam_passwdqc.so min=disabled,24,12,8,7 passphrase= password sufficient /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow #password requisite /lib/security/$ISA/pam_cracklib.so retry=3 difok= 新密码至少有一位与原来的不同 PASS_MIN_DAYS参数则设定了在本次密码修改后，下次允许更改密码之前所需的最少天数。 PASS_WARN_AGE的设定则指明了在口令失效前多少天开始通知用户更改密码（一般在用户刚刚登陆系统时 就会收到警告通知）。 你也会编辑/etc/default/useradd文件，寻找INACTIVE和EXPIRE两个关键词： INACTIVE= EXPIRE= ##### 这会指明在口令失效后多久时间内，如果口令没有进行更改，则将账户更改为失效状态。在本例中， ##### 这个时间是 14 天。而EXPIRE的设置则用于为所有新用户设定一个密码失效的明确时间（具体格式为“年份- ##### 月份-日期”）。 ##### 显然，上述这些设定更改之后，只能影响到新建立的用户。要想修改目前已存在的用户具体设置， 需要使用chage工具。 # chage -M 60 joe 这条命令将设置用户joe的PASS_MAX_DAYS为 60 ，并修改对应的shadow文件。 你可以使用chage -l的选项，列出当前的账户时效情况，而使用-m选项是设置PASS_MIN_DAYS，用-W 则是设置PASS_WARN_AGE，等等。chage工具可以让你修改特定账户的所有密码时效状态。 注意，chage仅仅适用于本地系统的账户，如果你在使用一个类似LDAP这样的认证系统时，该工具 会失效。如果你在使用LDAP作为认证，而你又打算使用chage，那么，哪怕仅仅是试图列出用户密码的时 效信息，你也会发现chage根本不起作用。 制定一项策略，定义多长时间一个密码必须进行更改，然后强制执行该策略，是非常不错的一个做 法。在解雇了某个雇员后，口令时效策略会保证该雇员不可能在被解雇 3 个月后发现他的口令依然可用。即 使系统管理员忽略了删除他的帐号，该帐号也会因密码时效策略而被自动锁定。当然，这一点并不能成为不 及时删除该雇员帐号的理由，但是这个策略的确提供了一层额外的安全防护，尤其是在过去常常忽视及时清 理帐号的情况下。 #### ACL 权限设置 ACL是Access Control List 的缩写，主要用于在提供传统的owner、group、others的read、write、execute 权限之外进行细部权限设置。 #### 启动 ACL ##### 让 / 目录支持 ACL ： ``` #mount –o remount ,acl / #mount |grep / //查看是否有挂载 开机启动 ACL ： 将要启动ACL的分区写入/etc/fstab中： #vi /etc/fstab /dev/hda5 / ext3 default,acl 1 2 ``` #### ACL 相关命令 **Getfacl :** 取得某个文件 **/** 目录的 **ACL** 权限； **Setfacl :** 设置某个文件 **/** 目录的 **ACL** 权限； setfacl [-bkndRLPvh] [{-m|-x} acl_spec] [{-M|-X} acl_file] file ... setfacl --restore=file 描述 setfacl用来在命令行里设置ACL。在命令行里，一系列的命令跟随以一系列的文件名。 选项-m和-x后边跟以acl规则。多条acl规则以逗号(,)隔开。选项-M和-X用来从文件或标准输入读取 acl规则。 选项--set和--set-file用来设置文件或目录的acl规则，先前的设定将被覆盖。 选项-m(--modify)和-M(--modify-file)选项修改文件或目录的acl规则。 选项-x(--remove)和-X(--remove-file)选项删除acl规则。 当使用-M，-X选项从文件中读取规则时，setfacl接受getfacl命令输出的格式。每行至少一条规则，以# 开始的行将被视为注释。 当在不支持ACLs的文件系统上使用setfacl命令时，setfacl将修改文件权限位。如果acl规则并不完全匹 配文件权限位，setfacl将会修改文件权限位使其尽可能的反应acl规则，并会向standard error发送错误消息， 以大于 0 的状态返回。 权限 文件的所有者以及有CAP_FOWNER的用户进程可以设置一个文件的acl。（在目前的linux系统上，root 用户是唯一有CAP_FOWNER能力的用户） 选项 -b, --remove-all 删除所有扩展的acl规则，基本的acl规则(所有者，群组，其他）将被保留。 -k,--remove-default 删除缺省的acl规则。如果没有缺省规则，将不提示。 -n，--no-mask 不要重新计算有效权限。setfacl默认会重新计算ACL mask，除非mask被明确的制定。 --mask 重新计算有效权限，即使ACL mask被明确指定。 -d，--default 设定默认的acl规则。 --restore=file 从文件恢复备份的acl规则（这些文件可由getfacl -R产生）。通过这种机制可以恢复整个目录树的acl 规则。此参数不能和除--test以外的任何参数一同执行。 --test 测试模式，不会改变任何文件的acl规则，操作后的acl规格将被列出。 -R，--recursive 递归的对所有文件及目录进行操作。 -L，--logical 跟踪符号链接，默认情况下只跟踪符号链接文件，跳过符号链接目录。 -P，--physical 跳过所有符号链接，包括符号链接文件。 --version 输出setfacl的版本号并退出。 --help 输出帮助信息。 -- 标识命令行参数结束，其后的所有参数都将被认为是文件名 - 如果文件名是-，则setfacl将从标准输入读取文件名。 ACL规则 setfacl命令可以识别以下的规则格式。 [d[efault]:] [u[ser]:]uid [:perms] 指定用户的权限，文件所有者的权限（如果uid没有指定）。 [d[efault]:] g[roup]:gid [:perms] 指定群组的权限，文件所有群组的权限（如果gid未指定） [d[efault]:] m[ask][:] [:perms] 有效权限掩码 [d[efault]:] o[ther] [:perms] 其他的权限 恰当的acl规则被用在修改和设定的操作中。 对于uid和gid，可以指定一个数字，也可指定一个名字。 perms域是一个代表各种权限的字母的组合：读-r 写-w 执行-x，执行只适合目录和一些可执行的文件。 pers域也可设置为八进制格式。 自动创建的规则 最初的，文件目录仅包含 3 个基本的acl规则。为了使规则能正常执行，需要满足以下规则。 *3个基本规则不能被删除。 *任何一条包含指定的用户名或群组名的规则必须包含有效的权限组合。 *任何一条包含缺省规则的规则在使用时，缺省规则必须存在。 用法举例如下： acl 全称 Access Control Lists 翻译成中文叫"访问控制列表", 传统的 Linux 文件系统的权限控制是通过 user、group、other 与 r(读)、w(写)、x(执行) 的不同组合来 实现的。随着应用的发展，这些权限组合已不能适应现时复杂的文件系统权限控制要求。 例如，目录 /data 的权限为：drwxr-x---，所有者与所属组均为 root，在不改变所有者的前提下，要求用户 tom 对该目录有完 全访问权限 (rwx).考虑以下 2 种办法 (这里假设 tom 不属于 root group) (1) 给 /data 的 other 类别增加 rwx permission，这样由于 tom 会被归为 other 类别，那么他也将拥 有 rwx 权限。 (2) 将 tom 加入到 root group，为 root group 分配 rwx 权限，那么他也将拥有 rwx 权限。 以上 2 种方法其实都不合适，所以传统的权限管理设置起来就力不从心了。 为了解决这些问题，Linux 开发出了一套新的文件系统权限管理方法，叫文件访问控制列表 (Access Control Lists, ACL)。简单地来说，ACL 就是可以设置特定用户或者用户组对于一个文件的操作权限。 ACL 有两种，一种是存取 ACL (access ACLs)，针对文件和目录设置访问控制列表。一种是默认 ACL (default ACLs)，只能针对目录设置。如果目录中的文件没有设置 ACL，它就会使用该目录的默认 ACL. 首先我来讲一下getfacl ( 显示文件或目录的 ACL) 在我的电脑里首先有一个用户叫NEU.我们学校的简称.同时还有一个用户,software,我的专业名称. 我以neu用户进行操作,在其目录下建立一个文件fileofneu. 可以看到它的初始权限为-rw-rw-r--然后我把这个文件权限进行下修改.使用的命令为chmod,修改后的文 件权限为-rw-rw----现在这个文件的权限就不允许其它用户访问了. 然后切换到sofeware用户,来证实这个文件的不可访问性. 下面我们就通过getfacl命令来查看.这时候得进入neu用户下操作了.其命令格式很简单:getfacl fileofneu 权限一目了然.不多介绍了,下面就要用Setfacl来进行修改了.使其在对于其它用户的权限里,只对 software用户只读只写. **setfacl -m u:softeware:rw- fileofneu setfacl -R -m u:softeware:rw- fileofneu (-R** 一定要在 **-m** 前面，表示目录下所有文件） **setfacl -x u:softeware fileofneu** （去掉单个权限） **setfacl -b** （去掉所有 **acl** 权限） 如果我们希望在一个目录中新建的文件和目录都使用同一个预定的ACL，那么我们可以使用默认(Default) ACL。在对一个目录设置了默认的ACL以后，每个在目录中创建的文件都会自动继承目录的默认ACL作为自 己的ACL。用setfacl的-d选项就可以做到这一点： [root@FC3-vm mnt]# setfacl -d --set g:testg1:rwx dir1 [root@FC3-vm mnt]# getfacl dir1 然后用getfacl命令来进行查看.我们就可以看到多了一行user:software:rw- 这说明其对用户software开 放了读写的权限. 为了证实其可用性,再切换到software用户下访问这个文件,发现与前面不同的是,这回可以读写了. 今天就讲这一个吧,讲多了,大家也记不住. 对了,刚才我进行了一下这个操作,发现进不去,原来是我没有给software用户授与访问/home/neu这个目 录的进入的权力,所以,我们还得应用 setfacl命令来使得software用户拥有进入这个目录的权力.其操作与上 面基本一致。 #### 用户身份切换 #### Su ##### 命令作用 ``` su的作用是变更为其它使用者的身份，超级用户除外，需要键入该使用者的密码。 使用方式 su [-fmp] [-c command] [-s shell] [--help] [--version] [-] [USER [ARG]] 参数说明 -f ， – fast：不必读启动文件（如 csh.cshrc 等），仅用于csh或tcsh两种Shell。 -l ， – login：加了这个参数之后，就好像是重新登陆一样，大部分环境变量(例如HOME、SHELL 和USER等)都是以该使用者(USER)为主，并且工作目录也会改变。如果没有指定USER，缺省情况是root。 -m， -p ，–preserve-environment：执行su时不改变环境变数。 -c command：变更账号为USER的使用者，并执行指令（command）后再变回原来使用者。 ``` - help 显示说明文件 - version 显示版本资讯 USER：欲变更的使用者账号， ARG: 传入新的Shell参数。 例子 **su -c ls root** 变更帐号为 **root** 并在执行 **ls** 指令后退出变回原使用者。 **[user1@centos6 ~]$ su - root -c "head -n 3 /etc/passwd"** 对于命令参数要加上引号 su [用户名] a>在root用户下, 输入 su 普通用户. 则切换至普通用户, 从root切换到变通用户不需要密码 b>在普通用户下, 输入 su [用户名] 提示 password: 输入用户的PASSWORD, 则切换至该用户 #### Sudo /etc/sudoers 谁能作什么的一个列表，Sudo能用需要在这个文件中定义 #visudo 增加如下，加%代表用户组，ALL=(ALL)表示登录者的来源主机名，最后的ALL代表可执行 的命令。NOPASSWD代表不需要密码直接可运行Sudo,限制多命令一定要写绝对路径，用逗号分开，多行用 ‘\’，用！代表不能执行 %aixi ALL=(ALL) NOPASSWD: ALL %aixi ALL=(ALL) NOPASSWD: /bin/ls,/bin/mkdir,/bin/rmdir,\ /usr/bin/who,!/usr/bin/passwd root #### 查询用户命令 #### W ##### 可显示开机多久，当前登录的所有用户，平均负载 #### Who ##### 显示当前登录的所有用户 #### Last ##### 显示每个用户最后的登录时间 #### Lastlog ##### 显示每个用户最后的登录时间 ## 四、 文件权限 #### 1 、文件类型 ``` Linux广泛的被很多用户所接受，它强大的功能受到很多人喜欢，Linux文件一般是用一些相关的应 用程序创建，比如图像工具、文档工具、归档工具... .... 或 cp工具等。Linux文件的删除方式是用rm 命 令。 Linux文件类型和Linux文件的文件名所代表的意义是两个不同的概念。我们通过一般应用程序而创建的 比如file.txt、file.tar.gz ，这些文件虽然要用不同的程序来打开，但放在Linux文件类型中衡量的话，大 多是常规文件（也被称为普通文件）。 Linux文件类型常见的有：普通文件、目录、字符设备文件、块设备文件、符号链接文件等；现在我们 进行一个简要的说明； 1 普通文件 ``` 1. [root@localhost ~]# ls -lh install.log 2. -rw-r--r-- 1 root root 53K 03 -16 08:54 install.log 我 们用 ls -lh 来查看某个文件的属性，可以看到有类似 -rw-r--r-- ，值得注意的是第一个符号是 - ， 这样的文件在Linux中就是普通文件。这些文件一般是用一些相关的应用程序创建，比如图像工具、文 档工具、归档工具... .... 或 cp工具等。这类文件的删除方式是用rm 命令； 2 目录 1. [root@localhost ~]# ls -lh 2. 总计 14M 3. -rw-r--r-- 1 root root 2 03-27 02:00 fonts.scale 4. -rw-r--r-- 1 root root 53K 03- 16 08:54 install.log 5. -rw-r--r-- 1 root root 14M 03 -16 07:53 kernel-6.15-1.2025_FC5.i686.rpm 6. drwxr-xr-x 2 1000 users 4.0K 04 -04 23:30 mkuml-2004.07.17 7. drwxr-xr-x 2 root root 4.0K 04 -19 10:53 mydir 8. drwxr-xr-x 2 root root 4.0K 03 -17 04:25 Public 当 我们在某个目录下执行，看到有类似 drwxr-xr-x ，这样的文件就是目录，目录在Linux是一个比较 特殊的文件。注意它的第一个字符是d。创建目录的命令可以用 mkdir 命令，或cp命令，cp可以把一 个目录复制为另一个目录。删除用rm 或rmdir命令。 3 字符设备或块设备文件 如时您进入/dev目录，列一下文件，会看到类似如下的； 1. [root@localhost ~]# ls -la /dev/tty 2. crw-rw-rw- 1 root tty 5, 0 04- 19 08:29 /dev/tty 3. [root@localhost ~]# ls -la /dev/hda1 4. brw-r----- 1 root disk 3, 1 2006-04-19 /dev/hda1 我们看到/dev/tty的属性是 crw-rw-rw- ，注意前面第一个字符是 c ，这表示字符设备文件。比如猫等 串口设备 我们看到 /dev/hda1 的属性是 brw-r----- ，注意前面的第一个字符是b，这表示块设备，比如硬盘，光 驱等设备； 这个种类的文件，是用mknode来创建，用rm来删除。目前在最新的Linux发行版本中，我们一般不 用自己来创建设备文件。因为这些文件是和内核相关联的。 4 套接口文件 当我们启动MySQL服务器时，会产生一个mysql.sock的文件。 1. [root@localhost ~]# ls -lh /var/lib/mysql/mysql.sock 2. srwxrwxrwx 1 mysql mysql 0 04-19 11:12 /var/lib/mysql/mysql.sock 注意这个文件的属性的第一个字符是 s。我们了解一下就行了。 5 符号链接文件 1. [root@localhost ~]# ls -lh setup.log 2. lrwxrwxrwx 1 root root 11 04 -19 11:18 setup.log - **>** install.log 当我们查看文件属性时，会看到有类似 lrwxrwxrwx,注意第一个字符是l，这类文件是链接文件。是通过 ln -s 源文件名 新文件名。上面是一个例子，表示setup.log是install.log的软链接文件。怎么理解呢？ 这和Windows操作系统中的快捷方式有点相似。 符号链接文件的创建方法举例； 1. [root@localhost ~]# ls -lh kernel-6.15-1.2025_FC5.i686.rpm 2. -rw-r--r-- 1 root root 14M 03- 16 07:53 kernel-6.15-1.2025_FC5.i686.rpm 3. [root@localhost ~]# ln -s kernel-6.15-1.2025_FC5.i686.rpm kernel.rpm 4. [root@localhost ~]# ls -lh kernel* 5. -rw-r--r-- 1 root root 14M 03- 16 07:53 kernel-6.15-1.2025_FC5.i686.rpm 6. lrwxrwxrwx 1 root root 33 04 -19 11:27 kernel.rpm - **>** kernel-6.15-1.2025_FC5 #### 2 、文件权限 Linux系统是一个典型的多用户系统，不同的用户处于不同的地位。为了保护系统的安全性，Linux 系统对不同用户访问同一文件的权限做了不同的规定。 对于一个Linux系统中的文件来说，它的权限可以分为三种：读的权限、写的权限和执行的权限， 分别用r、w和x表示。不同的用户具有不同的读、写和执行的权限。 对于一个文件来说，它都有一个特定的所有者，也就是对文件具有所有权的用户。同时，由于在 Linux系统中，用户是按组分类的，一个用户属于一个或多个组。文件所有者以外的用户又可以分为文 件所有者的同组用户和其它用户。因此，Linux系统按文件所有者、文件所有者同组用户和其它用户三 类规定不同的文件访问权限。 权限的概念 Linux文件系统安全模型是通过给系统中的文件赋予两个属性来起作用的，这两个赋予每个文件的 属性称为所有者(ownership)和访问权限(access rights)。Linux下的每一个文件必须严格地属于一个用户和 一个组。 下图是在我机器上的/root目录下运行ls -l命令的情况。 ``` -rw-r--r-- 这些符号用来描述文件的访问权限类别，也就是常说的文件权限。这些访问权限指导Linux根据文 件的用户和组所有权来处理所有访问文件的用户请求。总共有 10 种权限属性，因此一个权限列表总是 10 个字符的长度。它的格式遵循下列规则： ◆ 第 1 个字符表示一种特殊的文件类型。其中字符可为d(表示该文件是一个目录)、b(表示该文件 是一个系统设备，使用块输入/输出与外界交互，通常为一个磁盘)、c(表示该文件是一个系统设备，使 用连续的字符输入/输出与外界交互，如串口和声音设备)，“.”表示该文件是一个普通文件，没有特殊 属性。 ◆ 2 ～ 4 个字符用来确定文件的用户(user)权限， 5 ～ 7 个字符用来确定文件的组(group)权限， 8 ～ 10 个字符用来确定文件的其它用户(other user，既不是文件所有者，也不是组成员的用户)的权限。其中， 2 、 5 、 8 个字符是用来控制文件的读权限的，该位字符为r表示允许用户、组成员或其它人可从该文件 中读取数据。短线“-”则表示不允许该成员读取数据。与此类似， 3 、 6 、 9 位的字符控制文件的写权 限，该位若为w表示允许写，若为“-”表示不允许写。 4 、 7 、 10 位的字符用来控制文件的制造权限， 该位若为x表示允许执行，若为“-”表示不允许执行。 任何列在/etc/passwd文件中的用户都可以是一个文件的所有者，也称为该文件的用户。同样任何 列在/etc/group文件中的组都可以是文件组的所有者，也简称为文件的组。 drwxrwxr-- 2 root root 4096 2月 11 10:36 guo 因为guo的第 1 个位置的字符是d，所以由此知道guo是一个目录。第 2 至 4 位置上的属性是rwx， 表示用户root拥有权限列表显示guo中所有的文件、创建新文件或者删除guo中现有的文件，或者将 guo作为当前工作目录。第 5 至 7 个位置上的权限是rwx，表示root组的成员拥有和root一样的权限。 第 8 至 10 位上的权限仅是r--，表示不是root的用户及不属于root组的成员只有对guo目录列表的权 限。这些用户不能创建或者删除guo中的文件、执行junk中的可执行文件，或者将junk作为他们的当 前工作目录。 ``` ``` -rwxr-xr-- 1 user admin 20480 11月 11 09:23 Readme.txt 在该项中，第 1 个位置是短线“-”，表示该文件是一个普通文件，没有特殊属性。该文件对任何人 都可读，只对user可写，user和admin的组成员可以执行该文件。 另外需要注意的是，当用户访问一个文件时，权限检查是从左到右的。假设上述的readme.txt文件 具有以下权限： -r--rw-r-- 那么即使user是属于admin组的一个成员，也不能对该文件进行写操作，因为已经被左边的写权 限设置拒绝了。 ``` ### 一般文件权限读（R）， 写 (W)，执行（X）权限比较简单。一般材料上面都 ### 有介绍。这里介绍一下一些特殊的文件权限——SUID，SGID，Stick bit。 ### 如果你检查一下/usr/bin/passwd和/tmp/的文件权限你就会发现和普 ### 通的文件权限有少许不同，如下图所示： 这里就涉及到SUID和Stick bit。 SUID和SGID 我们首先来谈一下passwd程序特殊的地方。大家都知道，Linux把用户的密码信息存放 在/etc/shadow里面，该文件属性如下： 可以看到Shadow 的只有所有者可读写，所有者是root，所以该文件对普通用户是不可读写的。但是普通用户调 用passwd程序是可以修改自己的密码的，这又是为什么呢？难道普通用户可以读写shadow 文件？难道Linux有漏洞？当然不是啦。password可以修改shadow文件的原因是他设置了 SUID文件权限。 SUID文件权限作用于可执行文件。一般的可执行文件在执行期的所有者是当前用户，比 如当前系统用户是simon，simon运行程序 a.out，a.out执行期的所有者应该是simon。但是 如果我们给可执行文件设置了SUID权限，则该程序的执行期所有者，就是该文件所有者。还 以 前面的a.out为例，假如a.out设置了SUID，并且其所有者是root，系统当前用户是simon， 当simon运行a.out的时 候，a.out在运行期的所有者就是root，这 时 a.out可以存取只有root 权限才能存取的资源，比如读写shadow文件。当a.out执行结束 的时候当前用户的权限又回 到了simon的权限了。 passwd就是设置了SUID权限，并且passwd的所有者是root，所以所有的用户都可以 执行他，在passwd运行期，程序获得临时的root权限，这时其可以存取shadow文件。当passwd 运行完成，当前用户又回到普通权限。 同理，设置程序的SGID，可以使程序运行期可以临时获得所有者组的权限。在团队开发 的时候，这个文件权限比较有用，一般系统用SUID比较多。 SGID可以用于目录，当目录设置了SGID之后，在该目录下面建立的所有文件和目录都 具有和该目录相同的用户组。 Stick bit(粘贴位) 对程序，该权限告诉系统在程序完成后在内存中保存一份运行程序的备份，如该程序常 用，可为系统节省点时间，不用每次从磁盘加载到内存。Linux当前对文件没有实现这个功能， 一些其他的UNIX系统实现了这个功能。 Stick bit可以作用于目录，在设置了粘贴位的目录下面的文件和目录，只有所有者和root 可以删除他。现在我们可以回头去看看/tmp/目录的情况，这个目录 设置了粘贴位。所以说， 并且所有人都可以对该目录读写执行（ 777 ），这样意味着所有人都可以在/tmp/下面创建临时 目录。因为设置Stick bit只有所有者和root才能删除目录。这样普通用户只能删除属于自己 的文件，而不能删除其他人的文件。如下图所示： 设置SUID，SGID，Stick bit 前面介绍过SUID与SGID的功能，那么，如何打开文件使其成为具有SUID与SGID的 权限呢？这就需要使用数字更改权限了。现在应该知道，使用数字 更改权限的方式为“ 3 个数 字”的组合，那么，如果在这 3 个数字之前再加上一个数字，最前面的数字就表示这几个属性 了（注：通常我们使用chmod xyz filename的方式来设置filename的属性时，则是假设没有 SUID、SGID及Sticky bit）。 4 为SUID 2 为SGID 1 为Sticky bit 假设要将一个文件属性改为“-rwsr-xr-x”，由于s在用户权限中，所以是SUID，因此，在 原先的 755 之前还要加上 4 ，也就是使用“chmod 4755 filename”来设置。 SUID也可以用“chmod u+s filename”来设置，“chmod u-s filename”来取消SUID设 置；同样，SGID可以用“chmod g+s filename”，“chmod g-s filename”来取消SGID设置。 一般来说，使用过Linux的同学都知道，Linux文件的权限有rwx，所有者、所有组、其它用户的rwx权限 是彼此独立的。为此，经常会听到如果某个web文件需要被修改的话，需要加上 777 的权限，这就是让所有 用户可写。 但仔细一想，这样的权限未免有些想得比较天真，没有考滤特殊情况。例如/tmp目录默认权限是 777 ， 而且有些文件也是允许所有用户访问修改的，那么是不 是任何一个用户都可以将这些删除呢？再如 /etc/shadow保存的是用户密码文件，默认情况下它的权限是 640 ，那么只有shadow的 owner(root)才能修 改它，按照常规理解，这是不可理解的，因为每个用户都可能修改密码，也就是会修改这个文件。 为了把这些情况解释清楚，需要引入Linux特殊文件权限的概念。Linux特殊文件权限有三个玩意： sticky bit、SGID、SUID，以下一一道来。 sticky bit sticky bit只对目录有效，使目录下的文件，只有文件拥有者才能删除（如果他不属于owner，仅属于 group或者other，就算他有w权限，也不能删除文件）。 加sticky bit的方法： chmod o+t /tmp或者 chmod 1777 /tmp 查看是否加了sticky bit，用ls -l，可以看到有类似这样的权限：“-rwxrwxrwt”，t就代表已经加 上了sticky bit，而且生效了，如果显示的是“-rwxrwxrwT”，说明也已经加上了sticky bit，但没有生 效（因为本来other就没有写的权限）。 看看/tmp目录的权限，就是drwxrwxrwt吧 SGID(The Set GroupID ) 加上SGID的文件，表示运行这个程序时，是临时以这个文件的拥有组的身份运行的；加上SGID的文件 夹，表示在这个目录下创建的文件属于目录所有的组，而不是创建人所在的组，在这个目录下创建的目录继 承本目录的SGID。 加SGID的方法： chmod g+s /tmp或 chmod 2777 /tmp 查看是否加了SGID，用ls -l，可以看到类似这样的权限“drwxrwsrwx”，s就代表已经加上了SGID， 而且生效，如果显示“drwxrwSrwx”，说明已经加上了SGID，但没有生效（因为本来group就没有执行的 权限）。 SUID(The Set UserID) SUID与SGID是一样的，惟一不同的是，运行时是以这个文件的拥有者身份来运行。 加SUID的方法： chmod o+s /tmp或 chmod 4777 /tmp 同样的，加了SUID的文件权限有这类似这两种： “drwsrwxrwx”、“drwSrwxrwx”。 看看passwd命令的权限：ll /usr/bin/passwd，是"-rwsr-x-rx"，终于知道为什么执行passwd时，可 以修改/etc/shadow文件了吧 SUID：置于 u 的 x 位，原位置有执行权限，就置为 s，没有了为 S. SGID：置于 g 的 x 位，原位置有执行权限，就置为 s，没有了为 S. STICKY：粘滞位，置于 o 的 x 位，原位置有执行权限，就置为 t ，否则为T. #### 3 、超级权限控制 #### 在Linux操作系统中，root的权限是最高的，也被称为超级权限的拥有者。普通用户无法执行的操作，root用户都能完 成，所以也被称之为超级管理用户。 在系统中，每个文件、目录和进程，都归属于某一个用户，没有用户许可其它普通用户是无法操作的，但对root除外。root 用户的特权性还表现在root可以超越任何用户和用户组来对文件或目录进行读取、修改或删除（在系统正常的许可范围内）； 对可执行程序的执行、终止；对硬件设备的添加、创建和移除等；也可以对文件和目录进行属主和权限进行修改，以适合系 统管理的需要（因为root是系统中权限最高的特权用户）； #### 3.1 、对超级用户和普通用户的理解 #### 3.1.1、什么是超级用户； 在所有Linux系统中，系统都是通过UID来区分用户权限级别的，而UID为 0 的用户被系统约定为是具有超级权限。超级用 户具有在系统约定的最高权限满园内操作，所以说超级用户可以完成系统管理的所有工具；我们可以通过/etc/passwd 来查得 UID为 0 的用户是root，而且只有root对应的UID为 0 ，从这一点来看，root用户在系统中是无可替代的至高地位和无限制 权限。root用户在系统中就是超级用户； 3.1. 2 、理解 UID 和用户的对应关系 当系统默认安装时，系统用户和UID 是一对一的对关系，也就是说一个UID 对应一个用户。我们知道用户身份是通过UID 来 确认的，我们在 《用户（user）和用户组（group）配置文件详解》中的UID 的解说中有谈到“UID 是确认用户权限的标识， 用户登录系统所处的角色是通过UID 来实现的，而非用户名；把几个用户共用一个UID 是危险的，比如我们把普通用户的 UID 改为 0 ，和root共用一个UID ，这事实上就造成了系统管理权限的混乱。如果我们想用root权限，可以通过su或sudo 来实现；切不可随意让一个用户和root分享同一个UID ；” 在系统中，能不能让UID 和用户是一对多的关系？是可以的，比如我们可以把一个UID为 0 这个值分配给几个用户共同使用， 这就是UID 和用户的一对多的关系。但这样做的确有点危险；相同UID的用户具有相同的身份和权限。比如我们在系统中把 beinan这个普通用户的UID改为 0 后，事实上这个普通用户就具有了超级权限，他的能力和权限和root用户一样；用户beinan 所有的操作都将被标识为root的操作，因为beinan的UID为0,而UID为 0 的用户是root ，是不是有点扰口？也可以理解为 UID为 0 的用户就是root ，root用户的UID就是 0 ； UID和用户的一对一的对应关系 ，只是要求管理员进行系统管理时，所要坚守的准则，因为系统安全还是第一位的。所以我 们还是把超级权限保留给root这唯一的用户是最好的选择； 如果我们不把UID的 0 值的分享给其它用户使用，只有root用户是唯一拥有UID=0的话，root用户就是唯一的超级权限用户； #### 3.1. 3 、普通用户和伪装用户 与超级用户相对的就是普通用户和虚拟（也被称为伪装用户），普通和伪装用户都是受限用户；但为了完成特定的任务，普 通用户和伪装用户也是必须的；Linux是一个多用户、多任务的操作系统，多用户主要体现在用户的角色的多样性，不同的用 户所分配的权限也不同；这也是Linux系统比Windows系统更为安全的本质所在，即使是现在最新版本的Windows 2003 ， 也无法抹去其单用户系统的烙印； #### 3.2. 超级用户（权限）在系统管理中的作用 超级权限用户（UID为 0 的用户）到底在系统管理中起什么作用呢？主要表现在以下两点； #### 3.2. 1 、对任何文件、目录或进程进行操作； 但值得注意的是这种操作是在系统最高许可范围内的操作；有些操作就是具有超级权限的root也无法完成； 比如/proc 目录，/proc 是用来反应系统运行的实时状态信息的，因此即便是root也无能为力；它的权限如下 [root@localhost ~]# pwd /root [root@localhost ~]# cd / [root@localhost /]# ls -ld /proc/ dr- xr-xr-x 134 root root 0 2005-10-27 /proc/ 就是这个目录，只能是读和执行权限，但绝对没有写权限的；就是我们把/proc 目录的写权限打开给root，root用户也是不能 进行写操作； [root@localhost ~]# chmod 755 /proc [root@localhost /]# ls -ld /proc/ drwxr-xr-x 134 root root 0 2005-10-27 /proc/ [root@localhost /]# cd /proc/ [root@localhost proc]# mkdir testdir mkdir: 无法创建目录‘testdir’: 没有那个文件或目录 #### 3.2. 2 、对于涉及系统全局的系统管理； 硬件管理、文件系统理解、用户管理以及涉及到的系统全局配置等等......如果您执行某个命令或工具时，提示您无权限，大多 是需要超级权限来完成； 比如用adduser来添加用户，这个只能用通过超级权限的用户来完成； ### 3.2. 3 、超级权限的不可替代性； 由于超级权限在系统管理中的不可缺少的重要作用，为了完成系统管理任务，我们必须用到超级权限；在一般情况下，为了 系统安全，对于一般常规级别的应用，不需要root用户来操作完成，root用户只是被用来管理和维护系统之用；比如系统日 志的查看、清理，用户的添加和删除...... 在不涉及系统管理的工作的环境下，普通用户足可以完成，比如编写一个文件，听听音乐；用gimp 处理一个图片等...... 基 于普通应用程序的调用，大多普通用户就可以完成； 当我们以普通权限的用户登录系统时，有些系统配置及系统管理必须通过超级权限用户完成，比如对系统日志的管理，添加 和删除用户。而如何才能不直接以root登录，却能从普通用户切换到root用户下才能进行操作系统管理需要的工作，这就涉 及到超级权限管理的问题； 获取超级权限的过程，就是切换普通用户身份到超级用户身份的过程；这个过程主要是通过su和sudo 来解决； #### 3.3 、使用 su 命令临时切换用户身份； #### 3.3. 1 、su 的适用条件和威力 su命令就是切换用户的工具，怎么理解呢？比如我们以普通用户beinan登录的，但要添加用户任务，执行useradd ，beinan 用户没有这个权限，而这个权限恰恰由root所拥有。解决办法无法有两个，一是退出beinan用户，重新以root用户登录， 但这种办法并不是最好的；二是我们没有必要退出beinan用户，可以用su来切换到root下进行添加用户的工作，等任务完 成后再退出root。我们可以看到当然通过su 切换是一种比较好的办法； 通过su可以在用户之间切换，如果超级权限用户root向普通或虚拟用户切换不需要密码，什么是权力？这就是！而普通用户 切换到其它任何用户都需要密码验证； #### 3.3. 2 、su 的用法： su [OPTION选项参数] [用户] -, -l, --login 登录并改变到所切换的用户环境； -c, --commmand=COMMAND 执行一个命令，然后退出所切换到的用户环境； 至于更详细的，请参看man su ； #### 3.3. 3 、su 的范例： su 在不加任何参数，默认为切换到root用户，但没有转到root用户家目录下，也就是说这时虽然是切换为root用户了，但 并没有改变root登录环境；用户默认的登录环境，可以在/etc/passwd 中查得到，包括家目录，SHELL定义等； [beinan@localhost ~]$ su Password: [root@localhost beinan]# pwd /home/beinan su 加参数 - ，表示默认切换到root用户，并且改变到root用户的环境； [beinan@localhost ~]$ pwd /home/beinan [beinan@localhost ~]$ su - Password: [root@localhost ~]# pwd /root su 参数 - 用户名 [beinan@localhost ~]$ su - root 注：这个和su - 是一样的功能； Password: [root@localhost ~]# pwd /root [beinan@localhost ~]$ su - linuxsir 注：这是切换到 linuxsir用户 Password: 注：在这里输入密码； [linuxsir@localhost ~]$ pwd 注：查看用户当前所处的位置； /home/linuxsir [linuxsir@localhost ~]$ id 注：查看用户的UID和GID信息，主要是看是否切换过来了； uid=505(linuxsir) gid=502(linuxsir) groups=0(root),500(beinan),502(linuxsir) [linuxsir@localhost ~]$ [beinan@localhost ~]$ su - -c ls 注：这是su的参数组合，表示切换到root用户，并且改变到root环境，然后列出 root家目录的文件，然后退出root用户； Password: 注：在这里输入root的密码； anaconda-ks.cfg Desktop install.log install.log.syslog testgroup testgroupbeinan testgrouproot [beinan@localhost ~]$ pwd 注：查看当前用户所处的位置； /home/beinan [beinan@localhost ~]$ id 注：查看当前用户信息； uid=500(beinan) gid=500(beinan) groups=500(beinan) #### 3.3. 4 、su的优缺点； su 的确为管理带来方便，通过切换到root下，能完成所有系统管理工具，只要把root的密码交给任何一个普通用户，他都 能切换到root来完成所有的系统管理工作； 但通过su切换到root后，也有不安全因素；比如系统有 10 个用户，而且都参与管理。如果这 10 个用户都涉及到超级权限 的运用，做为管理员如果想让其它用户通过su来切换到超级权限的root，必须把root权限密码都告诉这 10 个用户；如果这 10 个用户都有root权限，通过root权限可以做任何事，这在一定程度上就对系统的安全造成了威协；想想Windows吧，简 直就是恶梦； “没有不安全的系统，只有不安全的人”，我们绝对不能保证这 10 个用户都能按正常操作流程来管理系统，其中任何一人对系 统操作的重大失误，都可能导致系统崩溃或数据损失； 所以su 工具在多人参与的系统管理中，并不是最好的选择，su只适用于一两个人参与管理的系统，毕竟su并不能让普通用 户受限的使用； 超级用户root密码应该掌握在少数用户手中，这绝对是真理！所以集权而治的存在还是有一定道理的； #### 3.4 、 sudo 授权许可使用的 su ，也是受限制的 su #### 3.4.1. sudo 的适用条件； 由于su 对切换到超级权限用户root后，权限的无限制性，所以su并不能担任多个管理员所管理的系统。如果用su 来切换 到超级用户来管理系统，也不能明确哪些工作是由哪个管理员进行的操作。特别是对于服务器的管理有多人参与管理时，最 好是针对每个管理员的技术特长和管理范围，并且有针对性的下放给权限，并且约定其使用哪些工具来完成与其相关的工作， 这时我们就有必要用到 sudo。 通过sudo，我们能把某些超级权限有针对性的下放，并且不需要普通用户知道root密码，所以sudo 相对于权限无限制性的 su来说，还是比较安全的，所以sudo 也能被称为受限制的su ；另外sudo 是需要授权许可的，所以也被称为授权许可的 su； sudo 执行命令的流程是当前用户切换到root（或其它指定切换到的用户），然后以root（或其它指定的切换到的用户）身份 执行命令，执行完成后，直接退回到当前用户；而这些的前提是要通过sudo的配置文件/etc/sudoers来进行授权； #### 3.4. 2 、从编写 sudo 配置文件/etc/sudoers开始； sudo的配置文件是/etc/sudoers ，我们可以用他的专用编辑工具visodu ，此工具的好处是在添加规则不太准确时，保存退 出时会提示给我们错误信息；配置好后，可以用切换到您授权的用户下，通过sudo -l 来查看哪些命令是可以执行或禁止的； /etc/sudoers 文件中每行算一个规则，前面带有#号可以当作是说明的内容，并不执行；如果规则很长，一行列不下时，可以 用\号来续行，这样看来一个规则也可以拥有多个行； /etc/sudoers 的规则可分为两类；一类是别名定义，另一类是授权规则；别名定义并不是必须的，但授权规则是必须的； #### 3.4. 3 、/etc/sudoers 配置文件中别名规则 别名规则定义格式如下： Alias_Type NAME = item1, item2, ... 或 Alias_Type NAME = item1, item2, item3 : NAME = item4, item5 别名类型（Alias_Type）：别名类型包括如下四种 Host_Alias 定义主机别名； User_Alias 用户别名，别名成员可以是用户，用户组（前面要加%号） Runas_Alias 用来定义runas别名，这个别名指定的是“目的用户”，即sudo 允许切换至的用户； Cmnd_Alias 定义命令别名； NAME 就是别名了，NMAE的命名是包含大写字母、下划线以及数字，但必须以一个大写字母开头，比如SYNADM、SYN_ADM 或SYNAD0是合法的，sYNAMDA或1SYNAD是不合法的； item 按中文翻译是项目，在这里我们可以译成成员，如果一个别名下有多个成员，成员与成员之间，通过半角,号分隔；成员 在必须是有效并事实存在的。什么是有效的呢？比如主机名，可以通过w查看用户的主机名（或ip地址），如果您只是本地 机操作，只通过hostname 命令就能查看；用户名当然是在系统中存在的，在/etc/paswd中必须存在；对于定义命令别名， 成员也必须在系统中事实存在的文件名（需要绝对路径）； item成员受别名类型 Host_Alias、User_Alias、Runas_Alias、Cmnd_Alias 制约，定义什么类型的别名，就要有什么类型的 成员相配。我们用Host_Alias定义主机别名时，成员必须是与主机相关相关联，比如是主机名（包括远程登录的主机名）、 ip地址（单个或整段）、掩码等；当用户登录时，可以通过w命令来查看登录用户主机信息；用User_Alias和Runas_Alias 定义时，必须要用系统用户做为成员；用Cmnd_Alias 定义执行命令的别名时，必须是系统存在的文件，文件名可以用通配 符表示，配置Cmnd_Alias时命令需要绝对路径； 其中 Runas_Alias 和User_Alias 有点相似，但与User_Alias 绝对不是同一个概念，Runas_Alias 定义的是某个系统用户 可以sudo 切换身份到Runas_Alias 下的成员；我们在授权规则中以实例进行解说； 别名规则是每行算一个规则，如果一个别名规则一行容不下时，可以通过\来续行；同一类型别名的定义，一次也可以定义几 个别名，他们中间用:号分隔， Host_Alias HT01=localhost,st05,st04,10,0,0,4,255.255.255.0,192.168.1.0/24 注：定义主机别名HT01，通过=号列 出成员 Host_Alias HT02=st09,st10 注：主机别名HT02，有两个成员； Host_Alias HT01=localhost,st05,st04,10,0,0,4,255.255.255.0,192.168.1.0/24:HT02=st09,st 10 注：上面的两条对 主机的定义，可以通过一条来实现，别名之间用:号分割； 注：我们通过Host_Alias 定义主机别名时，项目可以是主机名、可以是单个ip（整段ip地址也可以），也可以是网络掩码； 如果是主机名，必须是多台机器的网络中，而且这些机器得能通过主机名相互通信访问才有效。那什么才算是通过主机名相 互通信或访问呢？比如 ping 主机名，或通过远程访问主机名来访问。在我们局域网中，如果让计算机通过主机名访问通信， 必须设置/etc/hosts，/etc/resolv.conf ，还要有DNS做解析，否则相互之间无法通过主机名访问；在设置主机别名时，如果 项目是中某个项目是主机名的话，可以通过hostname 命令来查看本地主机的主机名，通过w命令查来看登录主机是来源， 通过来源来确认其它客户机的主机名或ip地址；对于主机别名的定义，看上去有点复杂，其实是很简单。 如果您不明白Host_Alias 是怎么回事，也可以不用设置主机别名，在定义授权规则时通过ALL来匹配所有可能出现的主机情 况。如果您把主机方面的知识弄的更明白，的确需要多多学习。 User_Alias SYSAD=beinan,linuxsir,bnnnb,lanhaitun 注：定义用户别名，下有四个成员；要在系统中确实在存在的； User_Alias NETAD=beinan,bnnb 注：定义用户别名NETAD ，我想让这个别名下的用户来管理网络，所以取了NETAD的 别名； User_Alias WEBMASTER=linuxsir 注：定义用户别名WEBMASTER，我想用这个别名下的用户来管理网站； User_Alias SYSAD=beinan,linuxsir,bnnnb,lanhaitun:NETAD=beinan,bnnb:WEBMASTER=linuxsir 注：上面三行的别名 定义，可以通过这一行来实现，请看前面的说明，是不是符合？ Cmnd_Alias USERMAG=/usr/sbin/adduser,/usr/sbin/userdel,/usr/bin/passwd [A-Za-z]*,/bin/chown,/bin/chmod 注意：命令别名下的成员必须是文件或目录的绝对路径； Cmnd_Alias DISKMAG=/sbin/fdisk,/sbin/parted Cmnd_Alias NETMAG=/sbin/ifconfig,/etc/init.d/network Cmnd_Alias KILL = /usr/bin/kill Cmnd_Alias PWMAG = /usr/sbin/reboot,/usr/sbin/halt Cmnd_Alias SHELLS = /usr/bin/sh, /usr/bin/csh, /usr/bin/ksh, \ /usr/local/bin/tcsh, /usr/bin/rsh, \ /usr/local/bin/zsh 注：这行定义命令别名有点长，可以通过 \ 号断行； Cmnd_Alias SU = /usr/bin/su,/bin,/sbin,/usr/sbin,/usr/bin 在上面的例子中，有KILL和PWMAG的命令别名定义，我们可以合并为一行来写，也就是等价行； Cmnd_Alias KILL = /usr/bin/kill:PWMAG = /usr/sbin/reboot,/usr/sbin/halt 注：这一行就代表了KILL和PWMAG 命令别名，把KILL和PWMAG的别名定义合并在一行写也是可以的； Runas_Alias OP = root, operator Runas_Alias DBADM=mysql:OP = root, operator 注：这行是上面两行的等价行；至于怎么理解Runas_Alias ，我们必须 得通过授权规则的实例来理解； #### 3.4. 4 、/etc/sudoers中的授权规则： 授权规则是分配权限的执行规则，我们前面所讲到的定义别名主要是为了更方便的授权引用别名；如果系统中只有几个用户， 其实下放权限比较有限的话，可以不用定义别名，而是针对系统用户直接直接授权，所以在授权规则中别名并不是必须的； 授权规则并不是无章可寻，我们只说基础一点的，比较简单的写法，如果您想详细了解授权规则写法的，请参看man sudoers 授权用户 主机=命令动作 这三个要素缺一不可，但在动作之前也可以指定切换到特定用户下，在这里指定切换的用户要用( )号括起来，如果不需要密 码直接运行命令的，应该加NOPASSWD:参数，但这些可以省略；举例说明； 实例一： beinan ALL=/bin/chown,/bin/chmod 如果我们在/etc/sudoers 中添加这一行，表示beinan 可以在任何可能出现的主机名的系统中，可以切换到root用户下执行 /bin/chown 和/bin/chmod 命令，通过sudo -l 来查看beinan 在这台主机上允许和禁止运行的命令； 值得注意的是，在这里省略了指定切换到哪个用户下执行/bin/shown 和/bin/chmod命令；在省略的情况下默认为是切换到root 用户下执行；同时也省略了是不是需要beinan用户输入验证密码，如果省略了，默认为是需要验证密码。 为了更详细的说明这些，我们可以构造一个更复杂一点的公式； 授权用户 主机=[(切换到哪些用户或用户组)] [是否需要密码验证] 命令1,[(切换到哪些用户或用户组)] [是否需要密码验证] [命令2],[(切换到哪些用户或用户组)] [是否需要密码验证] [命令3]...... 注解： 凡是[ ]中的内容，是可以省略；命令与命令之间用,号分隔；通过本文的例子，可以对照着看哪些是省略了，哪些地方需要有 空格； 在[(切换到哪些用户或用户组)] ，如果省略，则默认为root用户；如果是ALL ，则代表能切换到所有用户；注意要切换到的 目的用户必须用()号括起来，比如(ALL)、(beinan) 实例二： beinan ALL=(root) /bin/chown, /bin/chmod 如果我们把第一个实例中的那行去掉，换成这行；表示的是beinan 可以在任何可能出现的主机名的主机中，可以切换到root 下执行 /bin/chown ，可以切换到任何用户招执行/bin/chmod 命令，通过sudo -l 来查看beinan 在这台主机上允许和禁止运 行的命令； 实例三： beinan ALL=(root) NOPASSWD: /bin/chown,/bin/chmod 如果换成这个例子呢？表示的是beinan 可以在任何可能出现的主机名的主机中，可以切换到root下执行 /bin/chown ，不需 要输入beinan用户的密码；并且可以切换到任何用户下执行/bin/chmod 命令，但执行chmod时需要beinan输入自己的密 码；通过sudo -l 来查看beinan 在这台主机上允许和禁止运行的命令； 关于一个命令动作是不是需要密码，我们可以发现在系统在默认的情况下是需要用户密码的，除非特加指出不需要用户需要 输入自己密码，所以要在执行动作之前加入NOPASSWD: 参数； 有可能有的弟兄对系统管理的命令不太懂，不知道其用法，这样就影响了他对 sudoers定义的理解，下面我们再举一个最简 单，最有说服务力的例子； 实例四： 比如我们想用beinan普通用户通过more /etc/shadow文件的内容时，可能会出现下面的情况； [beinan@localhost ~]$ more /etc/shadow /etc/shadow: 权限不够 这时我们可以用sudo more /etc/shadow 来读取文件的内容；就就需要在/etc/soduers中给beinan授权； 于是我们就可以先su 到root用户下通过visudo 来改/etc/sudoers ；（比如我们是以beinan用户登录系统的） [beinan@localhost ~]$ su Password: 注：在这里输入root密码 下面运行visodu； [root@localhost beinan]# visudo 注：运行visudo 来改 /etc/sudoers 加入如下一行，退出保存；退出保存，在这里要会用vi，visudo也是用的vi编辑器；至于vi的用法不多说了； beinan ALL=/bin/more 表示beinan可以切换到root下执行more 来查看文件； 退回到beinan用户下，用exit命令； [root@localhost beinan]# exit exit [beinan@localhost ~]$ 查看beinan的通过sudo能执行哪些命令？ [beinan@localhost ~]$ sudo -l Password: 注：在这里输入beinan用户的密码 User beinan may run the following commands on this host: 注：在这里清晰的说明在本台主机上，beinan用户可以以 root权限运行more ；在root权限下的more ，可以查看任何文本文件的内容的； (root) /bin/more 最后，我们看看是不是beinan用户有能力看到/etc/shadow文件的内容； [beinan@localhost ~]$ sudo more /etc/shadow beinan 不但能看到 /etc/shadow文件的内容，还能看到只有root权限下才能看到的其它文件的内容，比如； [beinan@localhost ~]$ sudo more /etc/gshadow 对于beinan用户查看和读取所有系统文件中，我只想把/etc/shadow 的内容可以让他查看；可以加入下面的一行； beinan ALL=/bin/more /etc/shadow 题外话：有的弟兄会说，我通过su 切换到root用户就能看到所有想看的内容了，哈哈，对啊。但咱们现在不是在讲述sudo 的用法吗？如果主机上有多个用户并且不知道root用户的密码，但又想查看某些他们看不到的文件，这时就需要管理员授权 了；这就是sudo的好处； 实例五：练习用户组在/etc/sudoers中写法； 如果用户组出现在/etc/sudoers 中，前面要加%号，比如%beinan ，中间不能有空格； %beinan ALL=/usr/sbin/*,/sbin/* 如果我们在 /etc/sudoers 中加上如上一行，表示beinan用户组下的所有成员，在所有可能的出现的主机名下，都能切换到 root用户下运行 /usr/sbin和/sbin目录下的所有命令； 实例六：练习取消某类程序的执行； 取消程序某类程序的执行，要在命令动作前面加上!号； 在本例中也出现了通配符的*的用法； beinan ALL=/usr/sbin/*,/sbin/*,!/usr/sbin/fdisk 注：把这行规则加入到/etc/sudoers中；但您得有beinan这个用 户组，并且beinan也是这个组中的才行； 本规则表示beinan用户在所有可能存在的主机名的主机上运行/usr/sbin和/sbin下所有的程序，但fdisk 程序除外； [beinan@localhost ~]$ sudo -l Password: 注：在这里输入beinan用户的密码； User beinan may run the following commands on this host: (root) /usr/sbin/* (root) /sbin/* (root) !/sbin/fdisk [beinan@localhost ~]$ sudo /sbin/fdisk -l Sorry, user beinan is not allowed to execute '/sbin/fdisk -l' as root on localhost. 注：不能切换到root用户下运行fdisk 程序； 实例七：别名的运用的实践； 假如我们就一台主机localhost，能通过hostname 来查看，我们在这里就不定义主机别名了，用ALL来匹配所有可能出现的 主机名；并且有beinan、linuxsir、lanhaitun 用户；主要是通过小例子能更好理解；sudo虽然简单好用，但能把说的明白的 确是件难事；最好的办法是多看例子和man soduers ； User_Alias SYSADER=beinan,linuxsir,%beinan User_Alias DISKADER=lanhaitun Runas_Alias OP=root Cmnd_Alias SYDCMD=/bin/chown,/bin/chmod,/usr/sbin/adduser,/usr/bin/passwd [A-Za- z]*,!/usr/bin/passwd root Cmnd_Alias DSKCMD=/sbin/parted,/sbin/fdisk 注：定义命令别名DSKCMD，下有成员parted和fdisk ； SYSADER ALL= SYDCMD,DSKCMD DISKADER ALL=(OP) DSKCMD 注解： 第一行：定义用户别名SYSADER 下有成员 beinan、linuxsir和beinan用户组下的成员，用户组前面必须加%号； 第二行：定义用户别名 DISKADER ，成员有lanhaitun 第三行：定义Runas用户，也就是目标用户的别名为OP，下有成员root 第四行：定义SYSCMD命令别名，成员之间用,号分隔，最后的!/usr/bin/passwd root 表示不能通过passwd 来更改root密 码； 第五行：定义命令别名DSKCMD，下有成员parted和fdisk ； 第六行： 表示授权SYSADER下的所有成员，在所有可能存在的主机名的主机下运行或禁止 SYDCMD和DSKCMD下定义 的命令。更为明确遥说，beinan、linuxsir和beinan用户组下的成员能以root身份运行 chown 、chmod 、adduser、passwd， 但不能更改root的密码；也可以以root身份运行 parted和fdisk ，本条规则的等价规则是； beinan,linuxsir,%beinan ALL=/bin/chown,/bin/chmod,/usr/sbin/adduser,/usr/bin/passwd [A- Za-z]*,!/usr/bin/passwd root,/sbin/parted,/sbin/fdisk 第七行：表示授权DISKADER 下的所有成员，能以OP的身份，来运行 DSKCMD ，不需要密码；更为明确的说 lanhaitun 能以root身份运行 parted和fdisk 命令；其等价规则是： lanhaitun ALL=(root) /sbin/parted,/sbin/fdisk 可能有的弟兄会说我想不输入用户的密码就能切换到root并运行SYDCMD和DSKCMD 下的命令，那应该把把NOPASSWD: 加在哪里为好？理解下面的例子吧，能明白的； SYSADER ALL= NOPASSWD: SYDCMD, NOPASSWD: DSKCMD #### 3.4. 5 、/etc/sudoers中其它的未尽事项； 在授权规则中，还有 NOEXEC:和EXEC的用法，自己查man sudoers 了解；还有关于在规则中通配符的用法，也是需要了 解的。这些内容不多说了，毕竟只是一个入门性的文档。soduers配置文件要多简单就有多简单，要多难就有多难，就看自己 的应用了。 #### 3.4. 6 、sudo的用法； 我们在前面讲的/etc/sudoers 的规则写法，最终的目的是让用户通过sudo读取配置文件中的规则来实现匹配和授权，以便替 换身份来进行命令操作，进而完成在其权限下不可完成的任务； 我们只说最简单的用法；更为详细的请参考man sudo sudo [参数选项] 命令 -l 列出用户在主机上可用的和被禁止的命令；一般配置好/etc/sudoers后，要用这个命令来查看和测试是不是配置正确的； -v 验证用户的时间戳；如果用户运行sudo 后，输入用户的密码后，在短时间内可以不用输入口令来直接进行sudo 操作； 用-v 可以跟踪最新的时间戳； -u 指定以以某个用户执行特定操作； -k 删除时间戳，下一个sudo 命令要求用求提供密码； 举列： 首先我们通过visudo 来改/etc/sudoers 文件，加入下面一行； beinan,linuxsir,%beinan ALL=/bin/chown,/bin/chmod,/usr/sbin/adduser,/usr/bin/passwd [A- Za-z]*,!/usr/bin/passwd root,/sbin/parted,/sbin/fdisk 然后列出beinan用户在主机上通过sudo 可以切换用户所能用的命令或被禁止用的命令； [beinan@localhost ~]$ sudo -l 注：列出用户在主机上能通过切换用户的可用的或被禁止的命令； Password: 注：在这里输入您的用户密码； User beinan may run the following commands on this host: (root) /bin/chown 注：可以切换到root下用chown命令； (root) /bin/chmod 注：可以切换到root下用chmod命令； (root) /usr/sbin/adduser 注：可以切换到root下用adduser命令； (root) /usr/bin/passwd [A-Za-z]* 注：可以切换到root下用 passwd 命令； (root) !/usr/bin/passwd root 注：可以切换到root下，但不能执行passwd root 来更改root密码； (root) /sbin/parted 注：可以切换到 root下执行parted ； (root) /sbin/fdisk 注：可以切换到root下执行 fdisk ； 通过上面的sudo -l 列出可用命令后，我想通过chown 命令来改变/opt目录的属主为beinan ； [beinan@localhost ~]$ ls -ld /opt 注：查看/opt的属主； drwxr-xr-x 26 root root 4096 10月 27 10:09 /opt 注：得到的答案是归属root用户和root用户组； [beinan@localhost ~]$ sudo chown beinan:beinan /opt 注：通过chown 来改变属主为beinan用户和beinan用户组； [beinan@localhost ~]$ ls -ld /opt 注：查看/opt属主是不是已经改变了； drwxr-xr-x 26 beinan beinan 4096 10月 27 10:09 /opt 我们通过上面的例子发现beinan用户能切换到root后执行改变用户口令的passwd命令；但上面的sudo -l 输出又明文写着 不能更改root的口令；也就是说除了root的口令，beinan用户不能更改外，其它用户的口令都能更改。下面我们来测试； 对于一个普通用户来说，除了更改自身的口令以外，他不能更改其它用户的口令。但如果换到root身份执行命令，则可以更 改其它用户的口令； 比如在系统中有linuxsir这个用户, 我们想尝试更改这个用户的口令， [beinan@localhost ~]$ passwd linuxsir 注：不通过sudo 直接运行passwd 来更改linuxsir用户的口令； passwd: Only root can specify a user name. 注：失败，提示仅能通过 root来更改； [beinan@localhost ~]$ sudo passwd linuxsir 注：我们通过/etc/sudoers 的定义，让beinan切换到root下执行 passwd 命令来改变linuxsir的口令； Changing password for user linuxsir. New UNIX password: 注：输入新口令； Retype new UNIX password: 注：再输入一次； passwd: all authentication tokens updated successfully. 注：改变成功； ### 后记： 本文是用户管理的文档的重要组成部份，我计划在明天开始写用户管理控制工具，比如 useradd、userdel、usermod ，也就 是管理用户的工具介绍；当然我还会写用户查询工具等与用户管理相关的 #### 4 、权限命令 #### 4.1 、 chmod 文件或者目录的用户能够使用chmod命令修改文件的权限。Chmod命令有两种方式：一种是字符方 式，使用字符来修改文件的权限；另外一种是数字方式，使用 3 个数字的组合来修改文件的权限。 使用方式 : chmod [-cfvR] [--help] [--version] mode file... 说明 : Linux/Unix 的档案调用权限分为三级 : 档案拥有者、群组、其他。利用 chmod 可以借以控制档 案如何被他人所调用。 参数 : mode : 权限设定字串，格式如下 : [ugoa...][[+-=][rwxX]...][,...]，其中 u 表示该档案的拥有者，g 表示与该档案的拥有者属于同一个群体(group)者，o 表示其他以外的人，a 表 示这三者皆是。 + 表示增加权限、- 表示取消权限、= 表示唯一设定权限。 r 表示可读取，w 表示可写入，x 表示可执行，X 表示只有当该档案是个子目录或者该档案已经被设定 过为可执行。 -c : 若该档案权限确实已经更改，才显示其更改动作 -f : 若该档案权限无法被更改也不要显示错误讯息 -v : 显示权限变更的详细资料 -R : 对目前目录下的所有档案与子目录进行相同的权限变更(即以递回的方式逐个变更) --help : 显示辅助说明 --version : 显示版本 范例 : 将档案 file1.txt 设为所有人皆可读取 : chmod ugo+r file1.txt 将档案 file1.txt 设为所有人皆可读取 : chmod a+r file1.txt 将档案 file1.txt 与 file2.txt 设为该档案拥有者，与其所属同一个群体者可写入，但其他以外的人则不 可写入 : chmod ug+w,o-w file1.txt file2.txt 将 ex1.py 设定为只有该档案拥有者可以执行 : chmod u+x ex1.py 将目前目录下的所有档案与子目录皆设为任何人可读取 : chmod -R a+r * 数字方式的基本语法是：chmod nnn 文件 其中第 1 、 2 、 3 个n分别表示用户、组成员和所有其它用户。各个位置上的n要么是一个 0 ，或者是一 个由赋予权限的相关值相加得到的单个阿拉伯数字之和。这些数字的意义如表 1 所示。 值 表示的意义 4 表示文件或者目录的读权限 2 表示文件或者目录的写权限 1 表示文件或者目录的执行权限 很显然，当使用数字方式时，这 3 个数字必须为 0 至 7 中的一个。 若要rwx属性则4+2+1=7； 若要rw-属性则4+2=6； 若要r-x属性则4+1=7。 范例： chmod a=rwx file 和 chmod 777 file 效果相同 chmod ug=rwx,o=x file 和 chmod 771 file 效果相同 若用chmod 4755 filename可使此程序具有root的权限 #### 4.2 、 umask ##### 很显然，系统中各种文件的权限设置对特定用户的数据安全有很大影响。但是要求用户逐一明确设 置系统中每个文件的权限也是不现实的，为此，需要使用umask命令，该命令可以为用户账号中新文 件的创建进行缺省设置。系统管理员必须要为你设置一个合理的u m a s k值，以确保你创建的文件具有 所希望的缺省权限，防止其他非同组用户对你的文件具有写权限。具体来说，umask是用来设置权限掩 码的，权限掩码由 3 个数字组成，将现有的存取权限减掉权限掩码后，即可产生建立文件时默认的权限。 语 法：umask [-S][权限掩码] 补充说明：umask可用来设定[权限掩码]。[权限掩码]是由 3 个八进制的数字所组成，将现有的存 取权限减掉权限掩码后，即可产生建立文件时预设的权限。 参 数： -S 以文字的方式来表示权限掩码。 登录之后，可以按照个人的偏好使用u m a s k命令来改变文件创建的缺省权限。相应的改变直到退 出该s h e l l或使用另外的u m a s k命令之前一直有效。一般来说，u m a s k命令是在/ e t c / p r o f i l e 文件中设置的，每个用户在登录时都会引用这个文件，所以如果希望改变所有用户的u m a s k，可以在 该文件中加入相应的条目。如果希望永久性地设置自己的u m a s k值，那么就把它放在自己$ H O M E 目录下的. p r o f i l e或. b a s h _ p r o f i l e文件中。 如何计算umask值 u m a s k命令允许你设定文件创建时的缺省模式，对应每一类用户(文件属主、同组用户、其他用 户)存在一个相应的u m a s k值中的数字。对于文件来说，这一数字的最大值分别是 6 。系统不允许你 在创建一个文本文件时就赋予它执行权限，必须在创建后用c h m o d命令增加这一权限。目录则允许 设置执行权限，这样针对目录来说， u m a s k中各个数字最大可以到 7 。该命令的一般形式为： umask nnn 其中n n n为u m a s k置0 0 0 - 7 7 7。 计算u m a s k值：可以有几种计算u m a s k值的方法，通过设置u m a s k值，可以为新创建的文 件和目录设置缺省权限。 例如，对于u m a s k值0 0 2，相应的文件和目录缺省创建权限是什么呢？ 第一步，我们首先写下具有全部权限的模式，即7 7 7 (所有用户都具有读、写和执行权限)。 第二步，在下面一行按照u m a s k值写下相应的位，在本例中是0 0 2。 第三步，在接下来的一行中记下上面两行中没有匹配的位。这就是目录的缺省创建权限。稍加练习 就能够记住这种方法。 第四步，对于文件来说，在创建时不能具有文件权限，只要拿掉相应的执行权限比特即可。 这就是上面的例子，其中u m a s k值为0 0 2： 1) 文件的最大权限rwx rwx rwx (777) 2) umask值为0 0 2 - - - - - - -w- 3) 目录权限rwx rwx r-x (775) 这就是目录创建缺省权限 4) 文件权限rw- rw- r-- (664) 这就是文件创建缺省权限 系统默认的umask码是 0022 也就是：目录 755(rwx,rx,rx) ，文件：644(rw,r,r) 。 umask码的换算 0022 + 0755 = 0777 对应默认目录权限 反之 0777 - 0755 = 0022 0022 + 0644 +0111 = 0777 对应默认文件权限 反之 0777 - 0111 - 0644 = 0022 哈哈~~ 简单吧！假如我们要将默认目录权限设置为 744 那么对应的umask 是 0777 - 0744 = 0033 ，然后执行umask 0033命令就将umask码改成 0033 了。 下面是另外一个例子，假设这次u m a s k值为0 2 2： 1) 文件的最大权限rwx rwx rwx (777) 2 ) u m a s k值为0 2 2 - - - -w- -w- 3) 目录权限rwx r-x r-x (755) 这就是目录创建缺省权限 4) 文件权限rw- r-- r-- (644) 这就是文件创建缺省权限 下面是常用的u m a s k值及对应的文件和目录权限 umask值 目录 文件 0 2 2 7 5 5 6 4 4 0 2 7 7 5 0 6 4 0 0 0 2 7 7 5 6 6 4 0 0 6 7 7 1 6 6 0 0 0 7 7 7 0 6 6 0 如果想知道当前的umask 值，可以使用u m a s k命令：如果想要改变u m a s k值，只要使用u m a s k命令设置一个新的值即可： $ umask 002 确认一下系统是否已经接受了新的u m a s k值：在使用u m a s k命令之前一定要弄清楚到底希望 具有什么样的文件/目录创建缺省权限。否则可能会得到一些非常奇怪的结果；例如，如果将u m a s k 值设置为6 0 0，那么所创建的文件/目录的缺省权限就是0 6 6！除非你有特殊需要，否则没有必要去管 他，系统默认的值“ 022 ”umask是用的掩码，至于掩码的概念，从基础学吧，这里不说了。 ``` linux中的文件/目录许可是用 4 位八进制数表示的。其中第一个八进制数用来表示特殊许可设置， 第二个数字用来设置文件所有者的许可，第三个数字用来设置组许可，第四个数字用来设置所有人的许 可。 例如，root的权限为 777 ，若权限掩码设为 022 ，那么两都相减后可得 755 。下面是在我的系统更 改umask的一些情况： [root@linuxserver root]# umask 022 上述命令显示表示我的系统的umask值为 022 。 [root@linuxserver root]# umask -S u=rwx,g=rx,o=rx 当umask值为 022 时，默认情况下各用户的权限。注意这里的参数“S”是大写。 [root@linuxserver root]# umask 177 [root@linuxserver root]# umask -S u=rw,g=,o= 上述两行命令把umask值改为 177 ，结果只有文件所有者具有读写文件的权限，其它用户不能访问 该文件。这显然是一种非常安全的状态。 ``` #### 4.3 、 chown chown 命令用途更改与文件关联的所有者或组。 语法chown[ -f ] [ -h] [ -R ] Owner [ :Group ] { File ... | Directory ... } 描述chown命令将 File 参数指定的文件的所有者更改为 Owner 参数指定的用户。Owner 参数的值可 以是可在 /etc/passwd 文件中找到的用户标识或登录名。还可以选择性地指定组。Group 参数的值可以是可 在 /etc/group 文件中找到的组标识或组名。 只有 root 用户可以更改文件的所有者。只在您是 root 用户或拥有该文件的情况下才可以更改文件的 组。如果拥有文件但不是 root 用户，则只可以将组更改为您是其成员的组。 虽然 -H、-L 和 -P 标志是互斥的，指定不止一个也不认为是错误。指定的最后一个标志确定命令拟稿 将演示的操作。 参数： -f 禁止除用法消息之外的所有错误消息。 -h 更改遇到的符号链接的所有权，而非符号链接指向的文件或目录的所有权。 当遇到符号链接而您未 指定 -h 标志时，chown 命令更改链接指向的文件或目录的所有权，而非链接本身的所有权。如果指定 -R 标志，chown 命令递归地降序指定的目录。-H 如果指定了 -R 选项，并且引用类型目录的文件的符 号链接在命令行上指定，chown 变量会更改由符号引用的目录的用户标识（和组标识，如果已指定） 和所有在该目录下的文件层次结构中的所有文件。 -L 如果指定了 -R 选项，并且引用类型目录的文件的符号在命令行上指定或在遍历文件层次结构期间遇 到，chown 命令会更改由符号链接引用的目录的用户标识（和组标识，如果已指定）和在该目录之下 的文件层次结构中的所有文件。 -P 如果指定了 -R 选项并且符号链接在命令行上指定或者在遍历文件层次结构期间遇到，则如果系统 支持该操作，则 chown 命令会更改符号链接的所有者标识（和组标识，如果已指定）。chown 命令不 会执行至文件层次结构的任何其它部分的符号链接。 -R 递归地降序目录，更改每个文件的所有权。当遇到符号链接并且链接指向目录时，更改该目录的所 有权，但不进一步遍历目录。不过 -h、-H、-L or -P 标志也未指定，则当遇到符号链接并且该链接指向 到目录时，该目录的组所有权更改但不会进一步遍历目录。 安全性访问控制：此程序应该作为“可信计 算基”中的正常用户程序安装。退出状态该命令返回以下出口值： 0 命令执行成功并已执行所有请求的 更改。 >0 发生错误。 ##### 示例： 要更改文件 program.c 的所有者： chown jim program.cprogram.c的用户访问权限现在应用到 jim。作 为所有者，jim 可以使用 chmod命令允许或拒绝其他用户访问 program.c。 要将目录 /tmp/src 中所有文件的所有者和组更改为用户 john 和组 build： chown -R john:build /tmp/src文件 将档案 file1.txt 的拥有者设为 users 群体的使用者 jessie : chown jessie:users file1.txt 将目前目录下的所有档案与子目录的拥有者皆设为 users 群体的使用者 lamport : chmod -R lamport:users * #### 4.4 、 chgrp ##### 功能说明：变更文件或目录的所属群组。 语 法：chgrp [-cfhRv][--help][--version][所属群组][文件或目录...] 或 chgrp [-cfhRv][--help][--reference=<参 考文件或目录>][--version][文件或目录...] 补充说明：在UNIX系统家族里，文件或目录权限的掌控以拥有者及所属群组来管理。您可以使用chgrp指 令去变更文件与目录的所属群组，设置方式采用群组名称或群组识别码皆可。 参 数： -c或--changes 效果类似"-v"参数，但仅回报更改的部分。 -f或--quiet或--silent 不显示错误信息。 -h或--no-dereference 只对符号连接的文件作修改，而不更动其他任何相关文件。 -R或--recursive 递归处理，将指定目录下的所有文件及子目录一并处理。 -v或--verbose 显示指令执行过程。 --help 在线帮助。 --reference=<参考文件或目录> 把指定文件或目录的所属群组全部设成和参考文件或目录的所属群组相 同。 --version 显示版本信息。 范例： [root@linux ~]# chgrp users install.log [root@linux ~]#ls –l -rw-r--r-- 1 root users 28490 Jun 25 08:53 install.log [root@linux ~]#chgrp testing install.log chgrp:invalid group name ‘testing’ <==出现错误信息～找不到这个用户组名～ 发现了吗？文件的用户组被改成了users了，但要改成testing的时候，就会发生错误。注意，出现错误信 息后，要检查错误信息的内容。 ## 五、 目录结构 ##### 目录结构及主要内容“/”根目录部分有以下子目录： ``` ◆/usr 目录包含所有的命令、程序库、文档和其它文件。这些文件在正常操作中不会被改变的。这个 目录也包含你的Linux发行版本的主要的应用程序，譬如，Netscape。 ◆/var 目录包含在正常操作中被改变的文件：假脱机文件、记录文件、加锁文件、临时文件和页格式 化文件等。这个目录中存放着那些不断在扩充着的东西，为了保持/usr的相对稳定，那些经常被修改的 目录可以放在这个目录下，实际上许多系统管理员都是这样干的。顺带说一下系统的日志文件就在 /var/log目录中。 ``` ◆/home 目录包含用户的文件：参数设置文件、个性化文件、文档、数据、EMAIL、缓存数据等。这个 目录在系统省级时应该保留。 ◆/proc 目录整个包含虚幻的文件。它们实际上并不存在磁盘上，也不占用任何空间。（用ls –l 可以 显示它们的大小）当查看这些文件时，实际上是在访问存在内存中的信息，这些信息用于访问系统 ◆/bin 系统启动时需要的执行文件（二进制），这些文件可以被普通用户使用。 ◆/sbin 系统执行文件（二进制），这些文件不打算被普通用户使用。（普通用户仍然可以使用它们，但 要指定目录。） ◆/etc 操作系统的配置文件目录。 ◆/root 系统管理员（也叫超级用户或根用户）的Home目录。 ◆/dev 设备文件目录。LINUX下设备被当成文件，这样一来硬件被抽象化，便于读写、网络共享以及 需要临时装载到文件系统中。正常情况下，设备会有一个独立的子目 录。这些设备的内容会出现在独 立的子目录下。LINUX没有所谓的驱动符。 ◆/lib 根文件系统目录下程序和核心模块的共享库。 ◆/boot 用于自举加载程序（LILO或GRUB）的文件。当计算机启动时（如果有多个操作系统，有可能 允许你选择启动哪一个操作系统），这些文件首先被装载。这个目录也会包含LINUX核（压缩文件vmlinuz）， 但LINUX核也可以存在别处，只要配置LILO并且LILO知道LINUX核在哪儿。 ◆/opt 可选的应用程序，譬如，REDHAT 5.2下的KDE （REDHAT 6 .0下，KDE放在其它的XWINDOWS应 用程序中，主执行程序在/usr/bin目录下） ◆/tmp 临时文件。该目录会被自动清理干净。 ◆/lost+found 在文件系统修复时恢复的文件 ◆/usr目录下比较重要的部分有： ◆/usr/X11R6 X-WINDOWS系统（version 11, release 6) ◆/usr/X11 同/usr/X11R6 （/usr/X11R6的符号连接） ◆/usr/X11R6/bin 大量的小X-WINDOWS应用程序（也可能是一些在其它子目录下大执行文件的符号连 接 ）。 ◆/usr/doc LINUX的文档资料（在更新的系统中，这个目录移到/usr/share/doc）。 ◆/usr/share 独立与你计算机结构的数据，譬如，字典中的词。 ◆/usr/bin和/usr/sbin 类似与“/”根目录下对应的目录（/bin和/sbin），但不用于基本的启动（譬如， 在紧急维护中）。大多数命令在这个目录下。 ◆/usr/local 本地管理员安装的应用程序（也可能每个应用程序有单独的子目录）。在“main”安装后， 这个目录可能是空的。这个目录下的内容在重安装或升级操作系统后应该存在。 ◆/usr/local/bin 可能是用户安装的小的应用程序，和一些在/usr/local目录下大应用程序的符号连接。 ◆/proc目录的内容： ◆/proc/cpuinfo 关于处理器的信息，如类型、厂家、型号和性能等。 ◆/proc/devices 当前运行内核所配置的所有设备清单。 ◆/proc/dma 当前正在使用的DMA通道。/proc/filesystems 当前运行内核所配置的文件系统。 ◆/proc/interrupts 正在使用的中断，和曾经有多少个中断。 ◆/proc/ioports 当前正在使用的I/O端口。 举例，使用下面的命令能读出系统的CPU信息。 cat /proc/cpuinfo ``` /bin bin是binary的缩写。这个目录沿袭了UNIX系统的结构，存放着使用者最经常使 用的命令。例如cp、ls、cat，等等。 /boot 这里存放的是启动Linux时使用的一些核心文件。 ``` ``` /dev dev是device（设备）的缩写。这个目录下是所有Linux的外部设备，其功能类似 DOS下的.sys和Win下的.vxd。在 Linux中设备和文件是用同种方法访问的。例如： /dev/hda代表第一个物理IDE硬盘。 /etc 这个目录用来存放系统管理所需要的配置文件和子目录。 /home 用户的主目录，比如说有个用户叫wang，那他的主目录就是/home/wang也可以 用~wang表示。 /lib 这个目录里存放着系统最基本的动态链接共享库，其作用类似于Windows里的.dll 文件。几乎所有的应用程序都须要用到这些共享库。 /lost+found 这个目录平时是空的，当系统不正常关机后，这里就成了一些无家可归的文件的 避难所。对了，有点类似于DOS下的.chk文件。 /mnt 这个目录是空的，系统提供这个目录是让用户临时挂载别的文件系统。 /proc 这个目录是一个虚拟的目录，它是系统内存的映射，我们可以通过直接访问这个 目录来获取系统信息。也就是说，这个目录的内容不在硬盘上而是在内存里。 /root 系统管理员（也叫超级用户）的主目录。作为系统的拥有者，总要有些特权啊！ 比如单独拥有一个目录。 /sbin s就是Super User的意思，也就是说这里存放的是系统管理员使用的管理程序。 /tmp 这个目录不用说，一定是用来存放一些临时文件的地方了。 ``` /usr (^) 这是最庞大的目录，我们要用到的应用程序和文件几乎都存放在这个目录下。其 中包含以下子目录； /usr/X11R6 存放X-Window的目录； /usr/bin 存放着许多应用程序； /usr/sbin 给超级用户使用的一些管理程序就放在这里； /usr/doc 这是Linux文档的大本营； /usr/include (^) Linux下开发和编译应用程序需要的头文件，在这里查找； /usr/lib 存放一些常用的动态链接共享库和静态档案库； /usr/local 这是提供给一般用户的/usr目录，在这里安装软件最适合； /usr/man (^) man在Linux中是帮助的同义词，这里就是帮助文档的存放目录； /usr/src Linux开放的源代码就存在这个目录，爱好者们别放过哦！ /var 这个目录中存放着那些不断在扩充着的东西，为了保持/usr的相对稳定，那些经 常被修改的目录可以放在这个目录下，实际上许多系统管理员都是这样干的。顺 带说一下系统的日志文件就在/var/log目录中。 ##### 总结来说： ◆用户应该将文件存在/home/user_login_name目录下(及其子目录下)。 ◆本地管理员大多数情况下将额外的软件安装在/usr/local目录下并符号连接在/usr/local/bin下的主执 行程序。 ◆系统的所有设置在/etc目录下。 ◆不要修改根目录（“/”） 或/usr目录下的任何内容，除非真的清楚要做什么。这些目录最好和LINUX 发布时保持一致。 ``` ◆大多数工具和应用程序安装在目录：/bin, /usr/sbin, /sbin, /usr/x11/bin,/usr/local/bin。 ◆所有的文件在单一的目录树下。没有所谓的“驱动符”。 ``` ## 六、 软件安装 #### RPM RPM软件的安装、删除、更新只有root权限才能使用；对于查询功能任何用户都可以操作；如果普通 用户拥有安装目录的权限，也可以进行安装。 初始化rpm 数据库 通过rpm 命令查询一个rpm 包是否安装了，也是要通过rpm 数据库来完成的；所以我们要经常用下面 的两个命令来初始化rpm 数据库； [root@localhost beinan]# rpm --initdb [root@localhost beinan]# rpm --rebuilddb 注：这个要花好长时间； 注：这两个参数是极为有用，有时rpm系统出了问题，不能安装和查询，大多是这里出了问题； /var/lib/rpm目录下的数据库记录所有软件的升级需求，记录已经安装的所有软件，数字证书记录等，这 个目录下的文件非常重要。 #### RPM 软件包管理的查询功能： ##### 命令格式 ``` rpm {-q|-- query} [select-options] [query-options] RPM的查询功能是极为强大，是极为重要的功能之一；举几个常用的例子，更为详细的具体的，请参考 #man rpm ``` **1** 、查询系统已安装的软件； ``` 语法：rpm -q 软件名 举例： [root@localhost beinan]# rpm –q gaim gaim-1.3.0-1.fc4 -q就是 -- query ，中文意思是“问”，此命令表示的是，是不是系统安装了gaim ；如果已安装会有 信息输出；如果没有安装，会输出gaim 没有安装的信息；查看系统中所有已经安装的包，要加 -a 参 数； [root@localhost RPMS]# rpm -qa 如果分页查看，再加一个管道 |和more命令； [root@localhost RPMS]# rpm -qa |more 在所有已经安装的软件包中查找某个软件，比如说 gaim ；可以用 grep 抽取出来； [root@localhost RPMS]# rpm -qa |grep gaim 上面这条的功能和 rpm -q gaim 输出的结果是一样的； 等 37448 ``` **2** 、查询一个已经安装的文件属于哪个软件包； ``` 语法 rpm -qf 文件名 注：文件名所在的绝对路径要指出举例： [root@localhost RPMS]# rpm -qf /usr/lib/libacl.la libacl-devel-2.2.23-8 ``` **3** 、查询已安装软件包都安装到何处； ``` 语法：rpm -ql 软件名 或 rpm rpmquery -ql 软件名 ``` ##### 举例： ``` [root@localhost RPMS]# rpm -ql lynx [root@localhost RPMS]# rpmquery -ql lynx ``` **4** 、查询一个已安装软件包的信息 ``` 语法格式： rpm -qi 软件名 举例： [root@localhost RPMS]# rpm -qi lynx ``` **5** 、查看一下已安装软件的配置文件； ``` 语法格式：rpm -qc 软件名 举例： [root@localhost RPMS]# rpm -qc lynx ``` **6** 、查看一个已经安装软件的文档安装位置： ``` 语法格式： rpm -qd 软件名 举例： [root@localhost RPMS]# rpm -qd lynx ``` **7** 、查看一下已安装软件所依赖的软件包及文件； ``` 语法格式： rpm -qR 软件名 举例： [root@localhost beinan]# rpm -qR rpm -python 查询已安装软件的总结：对于一个软件包已经安装，我们可以把一系列的参数组合起来用；比如 rpm -qil ；比如： [root@localhost RPMS]# rpm -qil lynx 对已安装软件包查询的一点补充； ``` ``` [root@localhost RPMS]# updatedb [root@localhost RPMS]# locate 软件名或文件名 通过updatedb，我们可以用 locate 来查询一些软件安装到哪里了；系统初次安装时要执行updatedb ， 每隔一段时间也要执行一次；以保持已安装软件库最新；updatedb 是slocate软件包所有；如果您没 有这个命令，就得安装slocate ；举例： [root@localhost RPMS]# locate gaim ``` #### 对于未安装的软件包的查看： 查看的前提是您有一个.rpm 的文件，也就是说对既有软件file.rpm的查看等； **1** 、查看一个软件包的用途、版本等信息； ``` 语法： rpm -qpi file.rpm 举例： [root@localhost RPMS]# rpm -qpi lynx-2.8.5-23.i386.rpm ``` **2** 、查看一件软件包所包含的文件； ``` 语法： rpm -qpl file.rpm 举例： [root@localhost RPMS]# rpm -qpl lynx-2.8.5-23.i386.rpm ``` ##### 3 、查看软件包的文档所在的位置； ``` 语法： rpm -qpd file.rpm 举例： [root@localhost RPMS]# rpm -qpd lynx-2.8.5-23.i386.rpm ``` **4** 、查看一个软件包的配置文件； ``` 语法： rpm -qpc file.rpm 举例： [root@localhost RPMS]# rpm -qpc lynx-2.8.5-23.i386.rpm ``` **5** 、查看一个软件包的依赖关系 ``` 语法： rpm -qpR file.rpm 举例： [root@localhost archives]# rpm -qpR yumex_0.42-3.0.fc4_noarch.rpm /bin/bash /usr/bin/python config(yumex) = 0.42-3.0.fc4 pygtk2 pygtk2-libglade rpmlib(CompressedFileNames) <= 3.0.4- 1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 usermode yum >= 2.3.2 ``` #### 软件包的安装、升级、删除等； **1** 、安装和升级一个 **rpm** 包； [root@localhost beinan]#rpm -vih file.rpm 注：这个是用来安装一个新的rpm 包； [root@localhost beinan]#rpm -Uvh file.rpm 注：这是用来升级一个rpm 包； 如果有依赖关系的，请解决依赖关系，其实软件包管理器能很好的解决依赖关系，请看前面的软件包管理器 的介绍；如果您在软件包管理器中也找不到依赖关系的包；那只能通过编译他所依赖的包来解决依赖关系， 或者强制安装；语法结构： [root@localhost beinan]# rpm -ivh file.rpm --nodeps --force [root@localhost beinan]# rpm -Uvh file.rpm --nodeps --force 更多的参数，请查看 man rpm 举例应用： [root@localhost RPMS]# rpm -ivh lynx-2.8.5-23.i386.rpm Preparing... ########################################### [100%] 1:lynx ########################################### [100%] [root@localhost RPMS]# rpm -ivh --replacepkgs lynx-2.8.5-23.i386.rpm Preparing... ########################################### [100%] 1:lynx ########################################### [100%] 注： --replacepkgs 参数是以已安装的软件再安装一次；有时没有太大的必要；测试安装参数 --test ， 用来检查依赖关系；并不是真正的安装； [root@localhost RPMS]# rpm -ivh --test gaim-1.3.0-1.fc4.i386.rpm Preparing... ########################################### [100%] 由新版本降级为旧版本，要加 --oldpackage 参数； [root@localhost RPMS]# rpm -qa gaim gaim-1.5.0-1.fc4 [root@localhost RPMS]# rpm -Uvh --oldpackage gaim-1.3.0-1.fc4.i386.rpm Preparing... ########################################### [100%] 1:gaim ########################################### [100%] [root@localhost RPMS]# rpm -qa gaim gaim-1.3.0-1.fc4 为软件包指定安装目录：要加 -relocate 参数；下面的举例是把gaim-1.3.0-1.fc4.i386.rpm指定安装在 /opt/gaim 目录中； [root@localhost RPMS]# rpm -ivh --relocate /=/opt/gaim gaim-1.3.0-1.fc4.i386.rpm Preparing... ########################################### [100%] 1:gaim ########################################### [100%] [root@localhost RPMS]# ls /opt/ gaim 为软件包指定安装目录：要加 -relocate 参数；下面的举例是把lynx-2.8.5-23.i386.rpm 指定安装在 /opt/lynx 目录中； [root@localhost RPMS]# rpm -ivh --relocate /=/opt/lynx --badreloc lynx-2.8.5-23.i386.rpm Preparing... ########################################### [100%] 1:lynx ########################################### [100%] 我们安装在指定目录中的程序如何调用呢？一般执行程序，都放在安装目录的bin或者sbin目录中；看下 面的例子；如果有错误输出，就做相应的链接，用 ln -s ； [root@localhost RPMS]# /opt/lynx/usr/bin/lynx Configuration file /etc/lynx.cfg is not available. [root@localhost RPMS]# ln -s /opt/lynx/etc/lynx.cfg /etc/lynx.cfg [root@localhost RPMS]# /opt/lynx/usr/bin/lynx [http://www.linuxsir.org](http://www.linuxsir.org) RPM管理包管理器支持网络安装和查询； 比如我们想通过 Fedora Core 4.0 的一个镜像查询、安装软件包；地址： [http://mirrors.kernel.org/fedora/core/4/i386/os/Fedora/RPMS/](http://mirrors.kernel.org/fedora/core/4/i386/os/Fedora/RPMS/) 举例：命令格式： rpm 参数 rpm包文件的http或者ftp的地址 # rpm -qpi [http://mirrors.kernel.org/fedora/core/4/i386/os/](http://mirrors.kernel.org/fedora/core/4/i386/os/) Fedora/RPMS/gaim-1.3.0-1.fc4.i386.rpm # rpm -ivh [http://mirrors.kernel.org/fedora/core/4/i386/os/](http://mirrors.kernel.org/fedora/core/4/i386/os/) Fedora/RPMS/gaim-1.3.0-1.fc4.i386.rpm **2** 、删除一个 **rpm** 包； 首先您要学会查询rpm 包 ；请看前面的说明； [root@localhost beinan]#rpm -e 软件包名 举例：我想 移除lynx 包，完整的操作应该是： [root@localhost RPMS]# rpm -e lynx 如果有依赖关系，您也可以用--nodeps 忽略依赖的检查来删除。但尽可能不要这么做，最好用软件包管理 器 systerm-config-packages 来删除或者添加软件； [root@localhost beinan]# rpm -e lynx --nodeps #### RPM 验证与数字证书： ##### 导入签名： [root@localhost RPMS]# rpm --import 签名文件 举例： [root@localhost fc40]# rpm --import RPM-GPG-KEY [root@localhost fc40]# rpm --import RPM-GPG-KEY -fedora RPM验证作用是使用/var/lib/rpm下面的数据库内容来比较目前linux系统的环境下的所有软件文件，也就 是说当你有数据不小心丢失，或者不小心修改到某个软件的文件内容，就用这个简单的方法验证一下原本的 文件系统 #rpm –Va 列出目前系统上面所有可能被改动过的文件 #### 从 rpm 软件包抽取文件； 命令格式： rpm2cpio file.rpm |cpio -div 举例： [root@localhost RPMS]# rpm2cpio gaim-1.3.0-1.fc4.i386.rpm |cpio -div 抽取出来的文件就在当用操作目录中的 usr 和etc中；其实这样抽到文件不如指定安装目录来安装软件来 的方便；也一样可以抽出文件；为软件包指定安装目录：要加 -relocate 参数；下面的举例是把 gaim-1.3.0-1.fc4.i386.rpm指定安装在 /opt/gaim 目录中； [root@localhost RPMS]# rpm -ivh --relocate /=/opt/gaim gaim-1.3.0-1.fc4.i386.rpm Preparing... ########################################### [100%] 1:gaim ########################################### [100%] [root@localhost RPMS]# ls /opt/ gaim 这样也能一目了然；gaim的所有文件都是安装在 /opt/gaim 中，我们只是把gaim 目录备份一下，然后卸 掉gaim；这样其实也算提取文件的一点用法； #### RPM 的配置文件； RPM包管理，的配置文件是 rpmrc ，我们可以在自己的系统中找到；比如Fedora Core 4.0中的rpmrc 文 件位于； [root@localhost RPMS]# locate rpmrc /usr/lib/rpm/rpmrc /usr/lib/rpm/redhat/rpmrc 我们可以通过 rpm --showrc 查看；具体的还得我们自己来学习。呵。。。不要问我，我也不懂；只要您 看了这篇文章，认为对您有用，您的水平就和我差不多；咱们水平是一样的，所以我不能帮助您了；请 理解。 #### YUM #### YUM 配置文件 创建容器，位置在 **/etc/yum.repos.d** ，扩展名必须是 **.repo** ``` #cd /etc/yum.repos.d #vim yum.repo 新建一个仓库文件，名字可以随便定义，在文件中写如下内容 [base] #代表容器名称，中括号一定要存在，里面的名字可随便取 name=base #说明这个容器的意义，随便写都可以 baseurl=ftp://192.168.0.6/pub/Server #192. 168. 0. 6是你的YUM源地址，这个很重要。 enabled=1 #是否启动，=0则不启动，不启动就无法使用该源 gpgcheck=0 #是否验证. 可不要 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release ＃验证的密钥. 可不要 ``` 命令 **:#yum repolist all** 显示目前所使用的容器有哪些 如果查询出来的容器，status为disabled，要将配置文件，如上enabled=1 **/etc/yum.conf** **yum.conf** 这个配置文件主要是 **yum** 客户端使用，里面主要规定了要去用的 **rpm** 包的 **yum** 服务器的信 息。 [main] #main开头的块用于对客户端进行配置，在main后也可以指定yum源（不推荐这样做），与 /etc/yum.repo.d中指定yum源相同 cachedir=/var/cache/yum #cachedir：yum缓存的目录，yum在此存储下载的rpm包和数据库，一般是/var/cache/yum。 keepcache=0 #0表示不保存下载的文件， 1 表示保存下载的文件，默认为不保存 debuglevel=2 #debuglevel：除错级别， 0 ──10,默认是 2 貌似只记录安装和删除记录 logfile=/var/log/yum.log #指定yum的日志文件 pkgpolicy=newest #包的策略，如果配置多了yum源，同一软件在不同的yum源中有不同版本， newest则安装最新版本，该值为lastest，则yum会将服务器上ID按照字母序排列，选择最后那个服务器上 的软件安装 distroverpkg=centos-release #指定一个软件包，yum会根据这个包判断你的发行版本，默认是redhat-release，也可以是安装的任何 针对自己发行版的rpm包。 tolerant=1 #tolerent，也有 1 和 0 两个选项，表示yum是否容忍命令行发生与软件包有关的错误，比如你要安装 1,2,3三个包，而其中 3 此前已经 安装了，如果你设为1,则yum不会出现错误信息。默认是 0 。 exactarch=1 #exactarch，有两个选项 1 和0,代表是否只升级和你安装软件包cpu体系一致的包，如果设为 1 ，则如 你安装了一个i386的rpm，则yum不会用i686的包来升级。 retries=20 #retries，网络连接发生错误后的重试次数，如果设为 0 ，则会无限重试。 obsoletes=1 gpgcheck=1 #gpgchkeck= 有 1 和 0 两个选择，分别代表是否是否进行gpg校验，如果没有这一项，默认是检查的。 pl ugins = 1 #是否启用插件，默认 1 为允许， 0 表示不允许 reposdir=/etc/yy.rm #默认是 /etc/yum.repos.d/ 低下的 xx.repo后缀文件 #默认都会被include 进来 也就是说 /etc/yum.repos.d/xx.repo 无论配置文件有多少个 每个里面有多少 个[name]最后其实都被整合到 一个里面看就是了 重复的[name]应该是前面覆盖后面的--还是后面的覆盖前 面的呢？enabled 测试是后面覆盖前面 exclude=xxx #exclude 排除某些软件在升级名单之外，可以用通配符，列表中各个项目要用空格隔开，这个对于安 装了诸如美化包，中文补丁的朋友特别有用。 keepcache=[1 or 0] #设置 keepcache=1，yum 在成功安装软件包之后保留缓存的头文件 (headers) 和软件包。默认值为 keepcache=0 不保存 reposdir=[包含 .repo 文件的目录的绝对路径] #该选项用户指定 .repo 文件的绝对路径。.repo 文件包含软件仓库的信息 (作用与 /etc/yum.conf 文件 中的 [repository] 片段相同)。中 #### YUM 命令 用YUM安装删除软件yum install xxx，yum会查询数据库，有无这一软件包，如果有，则检查其依赖冲 突关系，如果没有依赖冲突，那么最好，下载安装;如果有，则会给出提示，询问是否要同时安装依赖，或 删除冲突的包，你可以自己作出判断。 删除的命令是，yum remove xxx，同安装一样，yum也会查询数据库，给出解决依赖关系的提示。 **YUM** 安装软件包 命令：yum install **YUM** 删除软件包 命令：yum remove 用YUM查询软件信息，我们常会碰到这样的情况，想要安装一个软件，只知道它和某方面有关，但又 不能确切知道它的名字。这时yum的查询功能就起作用了。你可以用yum search keyword这样的命令来进 行搜索，比如我们要则安装一个Instant Messenger，但又不知到底有哪些，这时不妨用yum search messenger 这样的指令进行搜索，yum会搜索所有可用rpm的描述，列出所有描述中和messeger有关的rpm包，于是 我们可能得到gaim，kopete等等，并从中选择。有时我们还会碰到安装了一个包，但又不知道其用途，我 们可以用yum info packagename这个指令来获取信息。 1.使用YUM查找软件包 命令：yum search 2.列出所有可安装的软件包 命令：yum list 3.列出所有可更新的软件包 命令：yum list updates 4.列出所有已安装的软件包 命令：yum list installed 5.列出所有已安装但不在 Yum Repository 內的软件包 命令：yum list extras 6.列出所指定的软件包 命令：yum list 7.使用YUM获取软件包信息 命令：yum info 8.列出所有软件包的信息 命令：yum info 9.列出所有可更新的软件包信息 命令：yum info updates 10.列出所有已安裝的软件包信息 命令：yum info installed 11.列出所有已安裝但不在 Yum Repository 內的软件包信息 命令：yum info extras 12.列出软件包提供哪些文件 命令：yum provides #### 清除 YUM 缓存 yum会把下载的软件包和header存储在cache中，而不会自动删除。如果我们觉得它们占用了磁盘空间， 可以使用yum clean指令进行清除，更精确的用法是yum clean headers清除header，yum clean packages清 除下载的rpm包，yum clean all一股脑儿端 1.清除缓存目录(/var/cache/yum)下的软件包 命令：yum clean packages 2.清除缓存目录(/var/cache/yum)下的 headers 命令：yum clean headers 3.清除缓存目录(/var/cache/yum)下旧的 headers 命令：yum clean Oldheaders 4.清除缓存目录(/var/cache/yum)下的软件包及旧的headers 命令：yum clean, yum clean all (= yum clean packages; yum clean oldheaders) ## 七、 时间管理 #### 1 、 Linux 时间介绍： ``` Linux时钟分为系统时钟（System Clock）和硬件（Real Time Clock，简称RTC）时钟。系统时钟 是指当前Linux Kernel中的时钟，而硬件时钟则是主板上由电池供电的时钟，这个硬件时钟可以在 BIOS中进行设置。当Linux启动时，硬件时钟会去读取系统时钟的 设置，然后系统时钟就会独立 于硬件运作。 Linux中的所有命令（包括函数）都是采用的系统时钟设置。在Linux中，用于时钟查看和 设置的命令主要有date、hwclock和clock。其中，clock和hwclock用法相近，只用一个就行，只不 过clock命令除了支持x86硬 件体系外，还支持Alpha硬件体系。 ``` #### 2 、 Linux 时间设置命令 #### 2.1 、 date: ``` 语法格式：date [-u] [-d datestr] [-s datestr] [--utc] [--universal] [--date=datestr] [--set=datestr] [--help] [--version] [+FORMAT] [MMDDhhmm[[CC]YY][.ss]] 说明： 可用来设置系统日期与时间。只有管理员才有设置日期与时间的权限，一般用户只能用 date命令显示时间。若不加任何参数，data会显示目前的日期与时间。 例 1 ：显示当前系统时间 [root@Test2 ~]# date 2010 年 06 月 17 日 星期四 00:00:04 CST 例 2 ：设置日期和时间为 2010 年 6 月 18 号12:00 [root@Test2 ~]# date -s "20100618 12:00:00" 2010 年 06 月 18 日 星期五 12:00:00 CST 例 3 ：设置日期为 2010 年年 6 月 18 号 [root@Test2 ~]# date -s 20100618 2010 年 06 月 18 日 星期五 00:00:00 CST 例 4 ：设置时间为12:00:00 ``` [root@Test2 ~]# date 12:00:00 date: invalid date “12:00:00” 例 **5** ：显示时区 [root@Test2 ~]# date -R Thu, 17 Jun 2010 00:01:36 +0800 或者： [root@Test2 ~]# cat /etc/sysconfig/clock # The ZONE parameter is only evaluated by system-config-date. # The timezone of the system is defined by the contents of /etc/localtime. ZONE="Asia/Shanghai" UTC=true ARC=false #### 2.2 、 hwclock/clock 语法格式：hwclock [--adjust][--debug][--directisa][--hctosys][--show][--systohc][--test] [--utc][--version][--set --date=<日期与时间>] 参数： --adjust hwclock每次更改硬件时钟时，都会记录在/etc/adjtime文件中。使用--adjust参数，可使hwclock 根据先前的记录来估算硬件时钟的偏差，并用来校正目前的硬件时钟。 --debug 显示hwclock执行时详细的信息。 --directisa hwclock预设从/dev/rtc设备来存取硬件时钟。若无法存取时，可用此参数直接以I/O指令 来存取硬件时钟。 --hctosys 将系统时钟调整为与目前的硬件时钟一致。 --set --date=<日期与时间> 设定硬件时钟。 --show 显示硬件时钟的时间与日期。 --systohc 将硬件时钟调整为与目前的系统时钟一致。 --test 仅测试程序，而不会实际更改硬件时钟。 --utc 若要使用格林威治时间，请加入此参数，hwclock会执行转换的工作。 --version 显示版本信息。 例 **1** ：查看硬件时间 # hwclock --show 或者 # clock --show 例 **2** ：设置硬件时间 # hwclock --set --date="07/07/06 10:19" （月/日/年 时:分:秒） 或者 # clock --set --date="07/07/06 10:19" （月/日/年 时:分:秒） 例 **3** ：硬件时间和系统时间的同步 按照前面的说法，重新启动系统，硬件时间会读取系统时间，实现同步，但是在不重新启动的时候，需 要用hwclock或clock命令实现同步。 硬件时钟与系统时钟同步： # hwclock --hctosys（hc代表硬件时间，sys代表系统时间） 或者 # clock –hctosys 例 **4** ：系统时钟和硬件时钟同步： ``` # hwclock --systohc 或者 # clock –systohc 例 5 ：强制将系统时间写入 CMOS ，使之永久生效，避免系统重启后恢复成原时间 # clock –w 或者 # hwclock -w ``` #### 2.3 、时区的设置 # tzselect Please identify a location so that time zone rules can be set correctly. Please select a continent or ocean. 1) Africa 2) Americas 3) Antarctica 4) Arctic Ocean 5) Asia 6) Atlantic Ocean 7) Australia 8) Europe 9) Indian Ocean 10) Pacific Ocean 11) none - I want to specify the time zone using the Posix TZ format. #? 输入 5 ，亚洲 Please select a country. 1) Afghanistan 18) Israel 35) Palestine 2) Armenia 19) Japan 36) Philippines 3) Azerbaijan 20) Jordan 37) Qatar 4) Bahrain 21) Kazakhstan 38) Russia 5) Bangladesh 22) Korea (North) 39) Saudi Arabia 6) Bhutan 23) Korea (South) 40) Singapore 7) Brunei 24) Kuwait 41) Sri Lanka 8) Cambodia 25) Kyrgyzstan 42) Syria 9) China 26) Laos 43) Taiwan 10) Cyprus 27) Lebanon 44) Tajikistan 11) East Timor 28) Macau 45) Thailand 12) Georgia 29) Malaysia 46) Turkmenistan 13) Hong Kong 30) Mongolia 47) United Arab Emirates 14) India 31) Myanmar (Burma) 48) Uzbekistan 15) Indonesia 32) Nepal 49) Vietnam 16) Iran 33) Oman 50 ) Yeme n 17) Iraq 34) Pakistan #? 输入 9 ，中国 Please select one of the following time zone regions. 1) east China - Beijing, Guangdong, Shanghai, etc. 2) Heilongjiang 3) central China - Gansu, Guizhou, Sichuan, Yunnan, etc. 4) Tibet & most of Xinjiang Uyghur 5) southwest Xinjiang Uyghur #? 输入 1 ，北京时间 The following information has been given: China east China - Beijing, Guangdong, Shanghai, etc. Therefore TZ='Asia/Shanghai' will be used. Local time is now: Fri Jul 7 10:32:18 CST 2006. Universal Time is now: Fri Jul 7 02:32:18 UTC 2006. Is the above information OK? 1 ) Ye s 2) No #? 输入 1 ，确认 如果不用tzselect命令，可以修改文件变更时区。 # vi /etc/sysconfig/clock Z/Shanghai（查/usr/share/zoneinfo下面的文件） UTC=false ARC=false # rm /etc/localtime # ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime 重新启动即可。 #### 2.4 、图形界面设置时区命令 timeconfig #### 2.5 、时间同步 ##### 例 1 ：同步时间 # ntpdate 210.72.145.44 （210.72.145.44是中国国家授时中心的官方服务器） 例 **2** ：定时同步时间 # crontab –e添加脚本例子如下： */20 * * * * /usr/sbin/ntpdate 210.72.145.44 //每 20 分钟执行一次 30 5 * * * /usr/sbin/ntpdate 210.72.145.44 //每天早晨 5 点半执行 ※ 前面五个*号代表五个数字，数字的取值范围和含义如下：分钟（0-59） 小 時（0-23） 日 期（1-31） 月份（1-12） 星期（ 0 -6）//0代表星期天设定完毕后，可使用# crontab –l 查看上面的设定。 ## 八、 启动引导 #### 1 、 Linux 的启动流程 ##### 1) BIOS 自检 ##### 2) 启动 GRUB/LILO ``` 3) 运行 Linux kernel 并检测硬件 4) 挂载根文件系统 5) 运行 Linux 系统的第一个进程 init(其 PID 永远为 1 ，是所有其它进程的父进程) 6) init 读取系统引导配置文件 /etc/inittab 中的信息进行初始化 ``` ``` 7) 执行系统初始化脚本－ /etc/rc.d/rc.sysinit，执行系统初始化(包括很多内容) 8) 根据指定的运行级别(runlevel)来运行服务器脚本程序，再执行脚本 /etc/rc.d/rc.local 9) 运行一些其他的特别服务，一般为 /sbin/mingetty 和 /etc/X11/prefdm 10) Linux 控制台(console)提示用户输入用户名、密码进行登陆。 总结：BIOS初始化检查外围设备检查启动设备读区MBR ``` #### 2 、在 Linux 中常用的启动引导工具： grub 和 lilo 在Linux和WINDOWS两系统并存时就需要安装GRUB（Grand Unified Bootloader），GRUB被广泛地 用于替代lilo,GRUB支持在启动时使用命令行模式,支持md5加密保护 还可以从ext2/ext3、ReiseFS、JFS、 FAT、minix及FFS文件系统上启动其配置文件为/boot/grub/grub.conf,更改grub.conf即可立时生效如果 硬盘上的MBR被更动过，可以用/sbin/grub-install /dev/hda来重安装grub现在我们打开 /boot/grub/grub.conf查看一下 # vim /boot/grub/grub.conf 内容如下： # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-ve rsion ro root=/dev/sda2 # initrd /initrd-version.img #boot=/dev/sda default=0 #default=0表示默认启动第一个系统,如果系统有两个系统是用什么做为分隔符的呢？title就是系统的分 隔符,第一个title后面就是第一个系统,用 0 表示。 timeout=5 #timout=5，就是默认在启动选择界面停留的时间，单位是秒。等待 5 秒自动进入默认操作系统 splashimage=(hd0,0)/grub/splash.xpm.gz #splashimage是grub启动背景画面，如果是自己写grub.conf文件，这个可以不用写。 hiddenmenu title Red Hat Enterprise Linux Server (2.6.18-53.el5) #title后面就是系统在启动时候显示的名字 root (hd0,0) #root 启动文件所在位置 kernel /vmlinuz-2.6.18-53.el5 ro root=LABEL=/ rhgb quiet #kernel 内核所在位置和名字 initrd /initrd-2.6.18-53.el5.img #initrd内核镜象的名字 grub.conf的范例： timeout=10 #等待 10 秒自动进入默认操作系统 splashimage=(hd0,0)/grub/splash.xpm.gz #grub启动背景画面 default=0 #默认进入第一个标题 title Red Hat Linux (2.4.20-18) #Red Hat Linux标题 root (hd0,0) #根文件系统位置 kernel /vmlinuz-2.4.20-18 ro root=LABEL=/ #核心位置与核心加载参数 initrd /initrd-2.4.20-18.img #启动initrd ram盘 title windows #另一个操作系统的标题 rootnoverify (hd0,1) #操作系统存放在hd0,1上，不要在grub里mount chainloader +1 #从hd0,1的第一个扇面启动 ## 九、 运行级别 #### 1 、 Linux 系统的运行级别 (runlevel) ``` Linux 系统有 7 个运行级别，Linux 系统任何时候都运行在一个指定的运行级别上，不同的运行级 别所运行的程序和服务不尽相同，所要完成的工作和要达到的目的也不相同 · 运行级别 0 系统停机(halt)状态，系统的默认运行级别不能设为 0 ，否则不能正常启动 · 运行级别 1 单用户工作(single user)状态，root 权限，用于系统维护，禁止远程登陆 · 运行级别 2 多用户(multiuser)状态 (没有 NFS) · 运行级别 3 完全的多用户(multiuser)状态 (有 NFS)，登陆后进入控制台命令行模式 · 运行级别 4 系统未使用，保留 · 运行级别 5 X11 控制台 (xdm、gdm、kdm)，登陆后进入图形 GUI 模式 ``` ``` · 运行级别 6 系统正常关闭并重启(reboot)，默认运行级别不能设为 6 ，否则不能正常启动 ``` #### 2 、运行级别的原理 ``` 在目录 /etc/rc.d/init.d 下有许多服务器脚本程序，一般称为服务(service)， 在 /etc/rc.d 下 有 7 个名为 rcN.d 的目录，其中 N=0-6，对应于系统的 7 个运行级别， rcN.d 目录下，都是 一些符号链接文件，这些链接文件都指向 init.d 目录下的 service 脚本文件，这些链接文件的命 名规则是 "K+nn+服务名" 或 "S+nn+服务名"，其中 nn 为 2 位数字： 例: rc3.d 目录下的链接文件 S80sendmail 就指向 service 脚本文件 ../init.d/sendmail 系统会根据指定的 runlevel 进入对应的 rcN.d 目录，并按照文件名顺序检索目录下的链接文件 ``` - 对于以 K 为开头的链接文件，系统将终止对应的服务 - 对于以 S 为开头的链接文件，系统将启动对应的服务 通过这种方式来实现 "不同的运行级别运行不同的程序和服务" #### 3 、 /etc/inittab 配置文件详解 init的进程号是 1 ，从这一点就能看出，init进程是系统所有进程的起点，Linux在完成核内引导以后， 就开始运行init程序 ，init程序需要读取设置文件/etc/inittab。inittab是个不可执行的文本文件，他有若 干行指令所组成。在Redhat系统中，inittab的内容如下所示(以“###"开始的中注释为笔者增加的)： （如果你改变了 **inittab** 文件，那么要使他立即生效，需要使用一个命令： **init q** ） # # inittab This file describes how the INIT process should set up # the system in a certain run-level. # # Author: Miquel van Smoorenburg, # Modified for RHS Linux by Marc Ewing and Donnie Barnes # # Default runlevel. The runlevels used by RHS are: # 0 - halt (Do NOT set initdefault to this) # 1 - Single user mode # 2 - Multiuser, without NFS (The same as 3, if you do not have networking) # 3 - Full multiuser mode # 4 - unused # 5 - X11 # 6 - reboot (Do NOT set initdefault to this) # ###表示当前缺省运行级别为5(initdefault)； id:5:initdefault: ###启动时自动执行/etc/rc.d/rc.sysinit脚本(sysinit) # System initialization. si::sysinit:/etc/rc.d/rc.sysinit l0:0:wait:/etc/rc.d/rc 0 l1:1:wait:/etc/rc.d/rc 1 l2:2:wait:/etc/rc.d/rc 2 l3:3:wait:/etc/rc.d/rc 3 l4:4:wait:/etc/rc.d/rc 4 ###当运行级别为 5 时，以 5 为参数运行/etc/rc.d/rc脚本，init将等待其返回(wait) l5:5:wait:/etc/rc.d/rc 5 l6:6:wait:/etc/rc.d/rc 6 ###在启动过程中允许按CTRL-ALT-DELETE重启系统 # Trap C TR L-A LT-DELETE ca::ctrlaltdel:/sbin/shutdown -t3 -r now # When our UPS tells us power has failed, assume we have a few minutes # of power left. Schedule a shutdown for 2 minutes from now. # This does, of course, assume you have powerd installed and your # UPS connected and working correctly. pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down" # If power was restored before the shutdown kicked in, cancel it. pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Cancelled" ###在 2 、 3 、 4 、 5 级别上以ttyX为参数执行/sbin/mingetty程序，打开ttyX终端用于用户登录， ###如果进程退出则再次运行mingetty程序(respawn) # Run gettys in standard runlevels 1:2345:respawn:/sbin/mingetty tty1 2:2345:respawn:/sbin/mingetty tty2 3:2345:respawn:/sbin/mingetty tty3 4:2345:respawn:/sbin/mingetty tty4 5:2345:respawn:/sbin/mingetty tty5 6:2345:respawn:/sbin/mingetty tty6 ###在 5 级别上运行xdm程序，提供xdm图像方式登录界面，并在退出时重新执行(respawn) # Run xdm in runlevel 5 x:5:respawn:/etc/X11/prefdm -nodaemon 以上面的inittab文件为例，来说明一下inittab的格式。其中以#开始的行是注释行，除了注释行之 外，每一行都有以下格式： id:runlevel:action:process 对上面各项的周详解释如下： 1. id id是指入口标识符，他是个字符串，对于getty或mingetty等其他login程序项，需求id和tty的编 号相同，否则getty程序将不能正常工作。 2. runlevel runlevel是init所处于的运行级别的标识，一般使用 0 － 6 及S或s。 0 、 1 、 6 运行级别被系统保留： 其中 0 作为shutdown动作， 1 作为重启至单用户模式， 6 为重启；S和s意义相同，表示单用户模式， 且无需inittab文件，因此也不在inittab中出现，实际上，进入单用户模式时，init直接在控制台 （/dev/console）上运行/sbin/sulogin。在一般的系统实现中，都使用了 2 、 3 、 4 、 5 几个级别，在 Redhat 系统中， 2 表示无NFS支持的多用户模式， 3 表示完全多用户模式（也是最常用的级别）， 4 保留给用户 自定义， 5 表示XDM图像登录方式。 7 － 9 级别也是能使用的，传统的Unix系统没有定义这几个级别。 runlevel能是并列的多个值，以匹配多个运行级别，对大多数action来说，仅当runlevel和当前运行级 别匹配成功才会执行。 3. action action是描述其后的process的运行方式的。action可取的值包括：initdefault、sysinit、boot、bootwait 等： initdefault是个特别的action值，用于标识缺省的启动级别；当init由核心激活以后，他将读取inittab 中的 initdefault项，取得其中的runlevel，并作为当前的运行级别。如果没有inittab文件，或其中没有 initdefault项， init将在控制台上请求输入runlevel。 sysinit、boot、bootwait等action将在系统启动时无条件运行，而忽略其中的runlevel。 其余的action（不含initdefault）都和某个runlevel相关。各个action的定义在inittab的man手册 中有周详的描述。 4. process process为具体的执行程序。程序后面能带参数。 第三部分：系统初始化 在init的设置文件中有这么一行： si::sysinit:/etc/rc.d/rc.sysinit 他调用执行了/etc/rc.d/rc.sysinit，而rc.sysinit是个bash shell的脚本，他主要 在init的设置文件中有这么一行： si::sysinit:/etc/rc.d/rc.sysinit 他 调用执行了/etc/rc.d/rc.sysinit，而rc.sysinit是个bash shell的脚本，他主要是完成一些系统初始化的 工作，rc.sysinit是每一个运行级别都要首先运行的重要脚本。他主要完成的工作有：激活交换分区，检 查磁盘，加载硬件模块及其他一些需要优先执行任务。 rc.sysinit约有 850 多行，不过每个单一的功能还是比较简单，而且带有注释，建议有兴趣的用户能自行 阅读自己机器上的该文件，以了解系统初始化所周详情况。由于此文件较长，所以不在本文中列出来， 也不做具体的介绍。 当rc.sysinit程序执行完毕后，将返回init继续下一步。 第四部分：启动对应运行级别的守护进程 在rc.sysinit执行后，将返回init继续其他的动作，通常接下来会执行到/etc/rc.d/rc程序。以运行级别 5 为例，init将执行设置文件inittab中的以下这行： l5:5:wait:/etc/rc.d/rc 5 这一行表示以 5 为参数运行/etc/rc.d/rc，/etc/rc.d/rc是个Shell脚本，他接受 5 作为参数，去执行/etc/rc.d /rc5.d/目录下的所有的rc启动脚本，/etc/rc.d/rc5.d/目录中的这些启动脚本实际上都是一些链接文件， 而不是真正的rc启动脚本，真正的rc启动脚本实际上都是放在/etc/rc.d/init.d/目录下。而这些rc启动 脚本有着类似的用法，他们一般能接受start、stop、 restart、status等参数。 /etc/rc.d/rc5.d/中的rc启动脚本通常是K或S开头的链接文件，对于以以S开头的启动脚本，将以start 参数来运行。而如果发现存在相应的脚本也存在K打头的链接，而且已处于运行态了(以/var/lock/subsys/ 下的文件作为标志)，则将首先以stop为参数停止这些已启动了的守护进程，然后再重新运行。这样做 是为了确保是当init改动运行级别时，所有相关的守护进程都将重启。 至于在每个运行级中将运行哪些守护进程，用户能通过chkconfig或setup中的"System Services"来自行 设定。常见的守护进程有： amd：自动安装NFS守护进程 apmd:高级电源管理守护进程 arpwatch：记录日志并构建一个在LAN接口上看到的以太网地址和IP地址对数据库 autofs：自动安装管理进程automount，和NFS相关，依赖于NIS crond：Linux下的计划任务的守护进程 named：DNS服务器 netfs：安装NFS、Samba和NetWare网络文件系统 network：激活已设置网络接口的脚本程序 nfs：打开NFS服务 portmap：RPC portmap管理器，他管理基于RPC服务的连接 sendmail：邮件服务器sendmail smb：Samba文件共享/打印服务 syslog：一个让系统引导时起动syslog和klogd系统日志守候进程的脚本 xfs：X Window字型服务器，为本地和远程X服务器提供字型集 Xinetd：支持多种网络服务的核心守护进程，能管理wuftp、sshd、telnet等服务 这些守护进程也启动完成了，rc程序也就执行完了，然后又将返回init继续下一步。 第五部分：建立终端 rc执行完毕后，返回init。这时基本系统环境已设置好了，各种守护进程也已启动了。init接下来会打 开 6 个终端，以便用户登录系统。通过按Alt+Fn(n对应1-6)能在这 6 个终端中转换。在inittab中的以下 6 行就是定义了 6 个终端： 1:2345:respawn:/sbin/mingetty tty1 2:2345:respawn:/sbin/mingetty tty2 3:2345:respawn:/sbin/mingetty tty3 4:2345:respawn:/sbin/mingetty tty4 5:2345:respawn:/sbin/mingetty tty5 6:2345:respawn:/sbin/mingetty tty6 从上面能看出在 2 、 3 、 4 、 5 的运行级别中都将以respawn方式运行mingetty程序，mingetty程序能打 开终端、设置模式。同时他会显示一个文本登录界面，这个界面就是我们经常看到的登录界面，在这个 登录界面中会提示用户输入用户名，而用户输入的用户将作为参数传给login程序来验证用户的身份。 第六部分：登录系统，启动完成 对于运行级别为 5 的图像方式用户来说，他们的登录是通过一个图像化的登录界面。登录成功后能直接 进入KDE、Gnome等窗口管理器。而本文主要讲的还是文本方式登录的情况： 当我们看到mingetty的登录界面时，我们就能输入用户名和密码来登录系统了。 Linux的账号验证程序是login，login会接收mingetty传来的用户名作为用户名参数。然后login会 对用户名进行分析：如果用户名不是root，且存在/etc/nologin文件，login将输出nologin文件的内容， 然后退出。这通常用来系统维护时防止非root用户登录。只有/etc/securetty中登记了的终端才允许root 用户登录，如果不存在这个文件，则root能在所有终端上登录。/etc /usertty文件用于对用户作出附加 访问限制，如果不存在这个文件，则没有其他限制。 在分析完用户名后，login将搜索/etc/passwd及/etc/shadow来验证密码及设置账户的其他信息，比如： 主目录是什么、使用何种shell。如果没有指定主目录，将默认为根目录；如果没有指定shell，将默 认 为/bin/bash。 login 程序成功后，会向对应的终端在输出最近一次登录的信息(在/var/log/lastlog中有记录)，并检查用 户是否有新邮件(在/usr/spool /mail/的对应用户名目录下)。然后开始设置各种环境变量：对于bash来说， 系统首先寻找/etc/profile脚本文件，并执行他；然后如果用户的主目录中存在.bash_profile文件，就执 行他，在这些文件中又可能调用了其他设置文件，所有的设置文件执行后后，各种环境变量也设好了， 这时会出现大家熟悉的命令行提示符，到此整个启动过程就结束了。 #### 4 、相关命令 #### 4.1 、查看当前系统运行等级 [root@test ~]# runlevel N 5 //’N’代表先前的Runlevel; ‘5’代表目前的Runlevel #### 4.2 、切换系统运行等级 #init N //切换到运行级别 N # init 0 //关机 # init 6 //重启动系统 ## 十、 进程管理 进程就是运行中的程序，一个运行着的程序，可能有多个进程。 比如 LinuxSir.Org 所用的WWW 服务器是apache服务器，当管理员启动服务后，可能会有好多人来访问，也就是说许多用户来同时请 求httpd服务，apache服务器将会创建有多个httpd进程来对其进行服务。 #### 1 、 进程分类 ##### 进程一般分为交互进程、批处理进程和守护进程三类。 ##### 值得一提的是守护进程总是活跃的，一般是后台运行，守护进程一般是由系统在开机时通过脚本自 动激活启动或超级管理用户root来启动。比如在Fedora或Redhat中，我们可以定义httpd服务器的启 动脚本的运行级别，此文件位于/etc/init.d目录下，文件名是httpd，/etc/init.d/httpd就是httpd服务器 的守护程序，当把它的运行级别设置为 3 和 5 时，当系统启动时，它会跟着启动。 [root@localhost ~]# chkconfig --level 35 httpd on ##### 由于守护进程是一直运行着的，所以它所处的状态是等待请求处理任务。比如，我们是不是访问 LinuxSir.Org ，LinuxSir.Org 的httpd服务器都在运行，等待着用户来访问，也就是等待着任务处理。 #### 2 、进程的属性 ##### 进程ID（PID)：是唯一的数值，用来区分进程； ##### 父进程和父进程的ID（PPID)； ##### 启动进程的用户ID（UID）和所归属的组（GID）； ##### 进程状态：状态分为运行R、休眠S、僵尸Z； ##### 进程执行的优先级； ##### 进程所连接的终端名； ##### 进程资源占用：比如占用资源大小（内存、CPU占用量）； #### 3 、父进程和子进程 ##### 他们的关系是管理和被管理的关系，当父进程终止时，子进程也随之而终止。但子进程终止，父进 程并不一定终止。比如httpd服务器运行时，我们可以杀掉其子进程，父进程并不会因为子进程的终止 而终止。 在进程管理中，当我们发现占用资源过多，或无法控制的进程时，应该杀死它，以保护系统的稳定 安全运行 #### 4 、进程管理命令 #### 4.1 、 ps ps为我们提供了进程的一次性的查看，它所提供的查看结果并不动态连续的；如果想对进程时间监控， 应该用top工具。 **4.1.1** 、 **ps** 的参数说明： ps 提供了很多的选项参数，常用的有以下几个； l 长格式输出； u 按用户名和启动时间的顺序来显示进程； j 用任务格式来显示进程； f 用树形格式来显示进程； a 显示所有用户的所有进程（包括其它用户）； x 显示无控制终端的进程； r 显示运行中的进程； ww 避免详细参数被截断； 我们常用的选项是组合是aux 或lax，还有参数f的应用； ps aux 或lax输出的解释； ##### USER表示启动进程用户。PID表示进程标志号。%CPU表示运行该进程占用CPU的时间与该进 ##### 程总的运行时间的比例。%MEM表示该进程占用内存和总内存的比例。VSZ表示占用的虚拟内存大 ##### 小，以KB为单位。RSS为进程占用的物理内存值，以KB为单位。TTY表示该进程建立时所对应的 ##### 终端，"?"表示该进程不占用终端。STAT表示进程的运行状态，包括以下几种代码：D，不可中断的 ##### 睡眠；R，就绪（在可运行队列中）；S，睡眠；T，被跟踪或停止；Z，终止（僵死）的进程，Z不 ##### 存在，但暂时无法消除；W，没有足够的内存分页可分配；<高优先序的进程；N，低优先序的进程； ##### L，有内存分页分配并锁在内存体内（实时系统或I/O）。STA RT为进程开始时间。TIME为执行的时 ##### 间。COMMAND是对应的命令名。 **4.1.2 ps** 应用举例 实例一：ps aux 最常用 [root@localhost ~]# ps -aux |more 可以用 | 管道和 more 连接起来分页查看； [root@localhost ~]# ps aux > ps001.txt [root@localhost ~]# more ps001.txt 这里是把所有进程显示出来，并输出到ps001.txt文件，然后再通过more 来分页查看； 实例二：和grep 结合，提取指定程序的进程； [root@localhost ~]# ps aux |grep httpd root 4187 0.0 1.3 24236 10272? Ss 11:55 0:00 /usr/sbin/httpd apache 4189 0.0 0.6 24368 4940? S 11:55 0:00 /usr/sbin/httpd apache 4190 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd apache 4191 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd apache 4192 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd apache 4193 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd apache 4194 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd apache 4195 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd apache 4196 0.0 0.6 24368 4932? S 11:55 0:00 /usr/sbin/httpd root 4480 0.0 0.0 5160 708 pts/3 R+ 12:20 0:00 grep httpd 实例二：父进和子进程的关系友好判断的例子 [root@localhost ~]# ps auxf |grep httpd root 4484 0.0 0.0 5160 704 pts/3 S+ 12:21 0:00 \_ grep httpd root 4187 0.0 1.3 24236 10272? Ss 11:55 0:00 /usr/sbin/httpd apache 4189 0.0 0.6 24368 4940? S 11:55 0:00 \_ /usr/sbin/httpd apache 4190 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd apache 4191 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd apache 4192 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd apache 4193 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd apache 4194 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd apache 4195 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd apache 4196 0.0 0.6 24368 4932? S 11:55 0:00 \_ /usr/sbin/httpd 这里用到了f参数；父与子关系一目了然； 例三：找出消耗内存最多的前 10 名进程 # ps -auxf | sort -nr -k 4 | head - 10 例四：找出使用CPU最多的前 10 名进程 # ps -auxf | sort -nr -k 3 | head - 10 #### 4.2 、 pstree 功能：pstree命令列出当前的进程，以及它们的树状结构。 格式：pstree [选项] [pid|user] 主要选项如下： ``` -a：显示执行程序的命令与完整参数。 -c：取消同名程序，合并显示。 -h：对输出结果进行处理，高亮显示正在执行的程序。 -l：长格式显示。 -n：以PID大小排序。 -p：显示PID。 -u：显示UID信息。 -G：使用VT100终端编码显示。 -U：使用UTF-8（Unicode）编码显示。 说明：使用ps命令得到的数据精确，但数据庞大，这一点对掌握系统整体概况来说是不容易的。pstree 正好可以弥补这个缺憾。它能将当前的执行程序以树状结构显示。pstree支持指定特定程序（PID） 或使用者（USER）作为显示的起始。 应用实例如下。 进程启动的时候可能会产生自己的一个子进程。运行pstree命令就可以很容易地看到这些信息。以超 级用户权限运行pstree： ＃init-+-apmd ``` ``` |-atd ``` ``` |-bdflush ``` ``` |-gconfd-2 ``` ``` |-gdm-binary---gdm-binary-+-X ``` ``` | `-startkde-+-kwrapper ``` ``` | `-ssh-agent ``` ``` |-gpm ``` ``` |-httpd---8*[httpd] ``` ``` ......下略 ``` ``` 命令对程序名称相同的会自动合并，所有"|-httpd---8*[httpd]"即表示系统中有 8 个httpd进程产生的 子进程。 ``` #### 4.3 、 top top命令用来显示系统当前的进程状况。 格式：top [选项] 主要选项如下。 d：指定更新的间隔，以秒计算。 q：没有任何延迟的更新。如果使用者有超级用户，则top命令将会以最高的优先序执行。 c：显示进程完整的路径与名称。 S：累积模式，会将已完成或消失的子进程的CPU时间累积起来。 s：安全模式。 i：不显示任何闲置（Idle）或无用（Zombie）的进程。 n：显示更新的次数，完成后将会退出top。 说明：top命令和ps命令的基本作用是相同的，都显示系统当前的进程状况。但是top是一个动态显 示过程，即可以通过用户按键来不断刷新当前状态。这里结合下图来说明它给出的信息。 ##### 第一行表示的项目依次为当前时间、系统启动时间、当前系统登录用户数目、平均负载。 第二行显示的是Tasks: 114 total 进程总数、2 running 正在运行的进程数、110 sleeping 睡眠的进程数、 0 stopped 停止的进程数、2 zombie 僵尸进程数 第三行显示的是目前CPU的使用情况，Cpu(s): 0.3% us 用户空间占用CPU百分比、1.0% sy 内核空间 占用CPU百分比、0.0% ni 用户进程空间内改变过优先级的进程占用CPU百分比、98.7% id 空 闲CPU百分比、0.0% wa 等待输入输出的CPU时间百分比、0.0% hi、0.0% si 第四行显示物理内存的使用情况，Mem: 191272k total 物理内存总量、173656k used 使用的物理内存 总量、17616k free 空闲内存总量、22052k buffers 用作内核缓存的内存量 第五行显示交换分区使用情况，Swap: 192772k total 交换区总量、0k used 使用的交换区总量、192772k free 空闲交换区总量、123988k cached 缓冲的交换区总量、内存中的内容被换出到交换区， 而后又被换入到内存，但使用过的交换区尚未被覆盖，该数值即为这些内容已存在于内存中的 交换区的大小。相应的内存再次被换出时可不必再对交换区写入。 第六行显示的项目最多，下面列出了详细解释。 PID（Process ID）：进程标志号，是非零正整数。USER：进程所有者的用户名。PR：进程的优先级别。 NI：进程的优先级别数值。VIRT：进程占用的虚拟内存值。RES：进程占用的物理内存值。SHR：进程 使用的共享内存值。S TAT：进程的状态，其中S表示休眠，R表示正在运行，Z表示僵死状态，N表 示该进程优先值是负数。%CPU：该进程占用的CPU使用率。%MEM：该进程占用的物理内存和总内 ##### 存的百分比。TIME：该进程启动后占用的总的CPU时间。COMMAND：进程启动的启动命令名称，如 ##### 果这一行显示不下，进程会有一个完整的命令行。 ``` top命令使用过程中，还可以使用一些交互的命令来完成其他参数的功能。这些命令是通过快捷键启 动的。 <空格>：立刻刷新。 A 分类显示系统不同资源的使用大户。有助于快速识别系统中资源消耗多的任务。 f 添加删除所要显示栏位. o 调整所要显示栏位的顺序. r 调整一个正在运行的进程Nice值. k 结束一个正在运行的进程. z 彩色/黑白显示开关 P：根据CPU使用大小进行排序。 T：根据时间、累计时间排序。 q：退出top命令。 m：切换显示内存信息。 t：切换显示进程和CPU状态信息。 c：切换显示命令名称和完整命令行。 M：根据使用内存大小进行排序。 W：将当前设置写入～/.toprc文件中。这是写top配置文件的推荐方法。 可以看到，top命令是一个功能十分强大的监控系统的工具，对于系统管理员而言尤其重要。但是， 它的缺点是会消耗很多系统资源。 ``` ## 十一、 资源监控 #### 1 、 free 内存监控 语 法： free [-bkmotV][-s <间隔秒数>] 补充说明：free指令会显示内存的使用情况，包括实体内存，虚拟的交换文件内存，共享内存区段，以 及系统核心使用的缓冲区等。 参 数： -b 以Byte为单位显示内存使用情况。 -k 以KB为单位显示内存使用情况。 -m 以MB为单位显示内存使用情况。 -o 不显示缓冲区调节列。 -s<间隔秒数> 持续观察内存使用状况。 -t 显示内存总和列。 -V 显示版本信息。 Mem：表示物理内存统计 -/+ buffers/cached：表示物理内存的缓存统计 Swap：表示硬盘上交换分区的使用情况 第 1 行 Mem： total：表示物理内存总量。 used：表示总计分配给缓存（包含buffers 与cache ）使用的数量，但其中可能部分缓存并未实际使 用。 free：未被分配的内存。 shared：共享内存，一般系统不会用到，这里也不讨论。 buffers：系统分配但未被使用的buffers 数量。 cached：系统分配但未被使用的cache 数量。buffer 与cache 的区别见后面。 total = used + free 第 2 行 -/+ buffers/cached： used：也就是第一行中的used – buffers-cached 也是实际使用的内 存总量。 free：未被使用的buffers 与cache 和未被分配的内存之和，这就是系统当前实际可用内存。 free 2= buffers1 + cached1 + free1 //free2为第二行、buffers1等为第一行 A buffer is something that has yet to be “written” to disk. A cache is something that has been “read” from the disk and stored for later use 第 3 行： 第三行所指的是从应用程序角度 来看，对于应用程序来说，buffers/cached 是等于可用的，因为buffer/cached是为了提高文件读取 的性能，当应用程序需在用到内存的时候，buffer/cached会很快地被回收。 所以从应用程序的角度来说，可用内存=系统free memory+buffers+cached. 接下来解释什么时候内存会被交换，以及按什么方交换。 当可用内存少于额定值的时候，就会开会进行交换，如何看额定值（RHEL4.0）： #cat /proc/meminfo 交换将通过三个途径来减少系统中使用的物理页面的个数： 1.减少缓冲与页面cache的大小， 2.将系统V类型的内存页面交换出去， 3.换出或者丢弃页面。(Application 占用的内存页，也就是物理内存不足）。 事实上，少量地使用swap是不是影响到系统性能的。 下面是buffers与cached的区别： buffers是指用来给块设备做的缓冲大小，他只记录文件系统的metadata以及 tracking in-flight pages. cached是用来给文件做缓冲。 那就是说：buffers是用来存储，目录里面有什么内容，权限等等。 而cached直接用来记忆我们打开的文件 ，如果你想知道他是不是真的生效，你可以试一下，先后执行 两次命令#man X ,你就可以明显的感觉到第二次的开打的速度快很多。 实验：在一台没有什么应用的机器上做会看得比较明显。记得实验只能做一次，如果想多做请换一个文 件名。 #free #man X #free #man X #free 你可以先后比较一下free后显示buffers的大小。 另一个实验： #free #ls /dev #free 你比较一下两个的大小，当然这个buffers随时都在增加，但你有ls过的话，增加的速度会变得快， 这个就是buffers/chached的区别。 因为Linux将你暂时不使用的内存作为文件和数据缓存，以提高系统性能，当你需要这些内存时， 系统会自动释放（不像windows那样，即使你有很多空闲内存,他也要访问一下磁盘中的pagefiles） 使用free命令 将used的值减去 buffer和cache的值就是你当前真实内存使用 ————– 对操作系统 来讲是 Mem的参数.buffers/cached 都是属于被使用,所以它认为free只有16936. 对应用程序 来讲是(-/+ buffers/cach).buffers/cached 是等同可用的，因为buffer/cached是为 了 提高 程序执行的性能， 当程序使用内存时，buffer/cached会很快地被使用。 所以,以应用来看看, 以(-/+ buffers/cache)的free和used为主.所以我们看这个就好了.另外告诉大家 一些常识.Linux 为了提高磁盘和内存存取效率, Linux做了很多精心的设计, 除了对dentry进行缓存(用于 VFS,加速 文件路径名到inode的转换), 还采取了两种主要Cache方式：Buffer Cache和Page Cache。 前者针 对磁盘块的读写，后者针对文件inode的读写。这些Cache能有效缩短了 I/O系统调用(比如 read,write,getdents)的时间。 记住内存是拿来用的,不是拿来看的. 不象windows,无论你的真实物 理内存有多少,他都要拿硬盘交换 文 件来读.这也就是windows为什么常常提示虚拟空间不足的原因. 你们想想,多无聊,在内存还有大部分 的时候,拿出一部分硬盘空间来充当内存.硬盘怎么会快过内存. 所以我们看linux,只要不用swap的交换 空间,就不用担心自己的内存太少.如果常常swap用很多,可 能你就要考虑加物理内存了.这也是linux看 内存是否够用的标准哦. [root@scs-2 tmp]# free total used free shared buffers cached Mem: 3266180 3250004 16176 0 110652 2668236 -/+ buffers/cache: 471116 2795064 Swap: 2048276 80160 1968116 下面是对这些数值的解释： total:总计物理内存的大小。 used:已使用多大。 free:可用有多少。 Shared:多个进程共享的内存总额。 Buffers/cached:磁盘缓存 的大小。 第三行(-/+ buffers/cached): used:已使用多大。 free:可用有多少。 第四行就不多解释了。 区别：第二行(mem)的used/free与第三行(-/+ buffers/cache) used/free的区别。 这两个的区别在于使用 的角度来看，第一行是从OS的角度来看，因为对于OS，buffers/cached 都是属于被使用，所以他的可 用内存是16176KB,已用内存是3250004KB,其中包括，内核（OS）使用+Application(X, oracle,etc)使用的 +buffers+cached. 第三行所指的是从应用程序角度来看，对于应用程序来说，buffers/cached 是等于可用的，因为 buffer/cached是为了提高文件读取的性能，当应用程序需在用到内存的时候，buffer/cached会很快地被 回收。 所以从应用程序的角度来说，可用内存=系统free memory+buffers+cached。 如上例： 2795064=16176+110652+2668236 接下来解释什么时候内存会被交换，以及按什么方交换。 当可用内存少于额定值的时候，就会开会进 行交换。 如何看额定值： cat /proc/meminfo [root@scs-2 tmp]# cat /proc/meminfo MemTotal: 3266180 kB MemFree: 17456 kB Buffers: 111328 kB Cached: 2664024 kB SwapCached: 0 kB Active: 467236 kB Inactive: 2644928 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 3266180 kB LowFree: 17456 kB SwapTotal: 2048276 kB SwapFree: 1968116 kB Dirty: 8 kB Writeback: 0 kB Mapped: 345360 kB Slab: 112344 kB Committed_AS: 535292 kB PageTables: 2340 kB VmallocTotal: 536870911 kB VmallocUsed: 272696 kB VmallocChunk: 536598175 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB 用free -m查看的结果： [root@scs-2 tmp]# free -m total used free shared buffers cached Mem: 3189 3173 16 0 107 2605 -/+ buffers/cache: 460 2729 Swap: 2000 78 1921 查看/proc/kcore文件的大小（内存镜像）： [root@scs-2 tmp]# ll -h /proc/kcore -r-------- 1 root root 4.1G Jun 12 12:04 /proc/kcore 备注： 占用内存的测量 测量一个进程占用了多少内存，linux为我们提供了一个很方便的方法，/proc目录为我们提供了所有的 信息，实际上top等工具也通过这里来获取相应的信息。 /proc/meminfo 机器的内存使用信息 /proc/pid/maps pid为进程号，显示当前进程所占用的虚拟地址。 /proc/pid/statm 进程所占用的内存 [root@localhost ~]# cat /proc/self/statm 654 57 44 0 0 334 0 输出解释 CPU 以及CPU0。。。的每行的每个参数意思（以第一行为例）为： 参数 解释 /proc//status Size (pages) 任务虚拟地址空间的大小 VmSize/4 Resident(pages) 应用程序正在使用的物理内存的大小 VmRSS/4 Shared(pages) 共享页数 0 Trs(pages) 程序所拥 有的可执行虚拟内存的大小 VmExe/4 Lrs(pages) 被映像到任务的虚拟内存空间的库的大小 VmLib/4 Drs(pages) 程序数据段和用户态的栈的大小 （VmData+ VmStk ） 4 dt(pages) 04 查看机器可用内存 /proc/28248/>free total used free shared buffers cached Mem: 1023788 926400 97388 0 134668 503688 -/+ buffers/cache: 288044 735744 Swap: 1959920 89608 1870312 我们通过free命令查看机器空闲内存时，会发现free的值很小。这主要是因为，在linux中有这么一种 思想，内存不用白不用，因此它尽可能的cache和buffer一些数据，以方便下次使用。但实际上这些内 存也是可以立刻拿来使用的。 所以 空闲内存=free+buffers+cached=total-used #### 2 、 vmstat 很显然从名字中我们就可以知道vmstat是一个查看虚拟内存（Virtual Memory）使用状况的工具，但 是怎样通过vmstat来发现系统中的瓶颈呢？在回答这个问题前，还是让我们回顾一下Linux中关于虚拟 内存相关内容。 #### 2.1 、虚拟内存运行原理 ##### 在系统中运行的每个进程都需要使用到内存，但不是每个进程都需要每时每刻使用系统分配的内存 ##### 空间。当系统运行所需内存超过实际的物理内存，内核会释放某些进程所占用但未使用的部分或所有物 ##### 理内存，将这部分资料存储在磁盘上直到进程下一次调用，并将释放出的内存提供给有需要的进程使用。 在Linux内存管理中，主要是通过“调页Paging”和“交换Swapping”来完成上述的内存调度。调页算 法是将内存中最近不常使用的页面换到磁盘上，把活动页面保留在内存中供进程使用。交换技术是将整 个进程，而不是部分页面，全部交换到磁盘上。 分页(Page)写入磁盘的过程被称作Page-Out，分页(Page)从磁盘重新回到内存的过程被称作Page-In。当 内核需要一个分页时，但发现此分页不在物理内存中(因为已经被Page-Out了)，此时就发生了分页错误 （Page Fault）。 当系统内核发现可运行内存变少时，就会通过Page-Out来释放一部分物理内存。经管Page-Out不是经 常发生，但是如果Page-out频繁不断的发生，直到当内核管理分页的时间超过运行程式的时间时，系 统效能会急剧下降。这时的系统已经运行非常慢或进入暂停状态，这种状态亦被称作thrashing(颠簸)。 #### 2.2 、使用 vmstat ##### 1.用法 ``` vmstat [-a] [-n] [-S unit] [delay [ count]] ``` vmstat [-s] [-n] [-S unit] vmstat [-m] [-n] [delay [ count]] vmstat [-d] [-n] [delay [ count]] vmstat [-p disk partition] [-n] [delay [ count]] vmstat [-f] vmstat [-V] -a：显示活跃和非活跃内存 -f：显示从系统启动至今的fork数量 。 -m：显示slabinfo -n：只在开始时显示一次各字段名称。 -s：显示内存相关统计信息及多种系统活动数量。 delay：刷新时间间隔。如果不指定，只显示一条结果。 count：刷新次数。如果不指定刷新次数，但指定了刷新时间间隔，这时刷新次数为无穷。 -d：显示磁盘相关统计信息。 -p：显示指定磁盘分区统计信息 -S：使用指定单位显示。参数有 k 、K 、m 、M ，分别代表 1000 、 1024 、 1000000 、 1048576 字 节（byte）。 默认单位为K（1024 bytes） -V：显示vmstat版本信息。 #### 2.3 、实例 ##### 例子 1 ：每 2 秒输出一条结果 ##### 字段说明： **Procs** （进程）： r: 运行的和等待(CPU时间片)运行的进程数，这个值也可以判断是否需要增加CPU(长期大于1) b: 等待IO的进程数量，处于不可中断状态的进程数，常见的情况是由IO引起的 **Memory** （内存）： swpd: 使用虚拟内存大小，切换到交换内存上的内存(默认以KB为单位) 如果 swpd 的值不为 0 ，或者还比较大，比如超过100M了，但是 si, so 的值长期为 0 ，这种情况我 们可以不用担心，不会影响系统性能。 free: 空闲的物理内存 buff: 用作缓冲的内存大小 cache: 用作缓存的内存大小，文件系统的cache，如果 cache 的值大的时候，说明cache住的文件数 多，如果频繁访问到的文件都能被cache住，那么磁盘的读IO bi 会非常小 **Swap** ： si: 每秒从交换区写到内存的大小，交换内存使用，由磁盘调入内存 so: 每秒写入交换区的内存大小，交换内存使用，由内存调入磁盘 内存够用的时候，这 2 个值都是 0 ，如果这 2 个值长期大于 0 时，系统性能会受到影响。磁盘IO和 CPU资源都会被消耗。 常有人看到空闲内存(free)很少或接近于 0 时，就认为内存不够用了，实际上不能光看这一点的，还 要结合si,so，如果free很少，但是si,so也很少(大多时候是0)，那么不用担心，系统性能这时不会受 到影响的。 **IO** ：（现在的 **Linux** 版本块的大小为 **1024bytes** ） bi: 每秒读取的块数，从块设备读入的数据总量(读磁盘) (KB/s) bo: 每秒写入的块数，写入到块设备的数据总理(写磁盘) (KB/s) 随机磁盘读写的时候，这 2 个 值越大（如超出1M），能看到CPU在IO等待的值也会越大 **system** ： in: 每秒中断数，包括时钟中断。 cs: 每秒上下文切换数。 上面这 2 个值越大，会看到由内核消耗的CPU时间会越多 **CPU** （以百分比表示）： us: 用户进程消耗的CPU时间百分比，us 的值比较高时，说明用户进程消耗的CPU时间多，但是如 果长期超过50% 的使用，那么我们就该考虑优化程序算法或者进行加速了 sy: 内核进程消耗的CPU时间百分比，sy 的值高时，说明系统内核消耗的CPU资源多，这并不是良 性的表现，我们应该检查原因。 id: CPU处在空闲状态时间百分比(包括IO等待时间) wa: IO等待消耗的CPU时间百分比，wa 的值高时，说明IO等待比较严重，这可能是由于磁盘大量 作随机访问造成，也有可能是磁盘的带宽出现瓶颈(块操作)。 例子 **2** ：显示活跃和非活跃内存 使用-a选项显示活跃和非活跃内存时，所显示的内容除增加inact和active外，其他显示内容与例子 1 相同。 字段说明： **Memory** （内存）： inact: 非活跃内存大小（当使用-a选项时显示） active: 活跃的内存大小（当使用-a选项时显示） 注：如果r经常大于 4 ，且id经常少于 40 ，表示cpu的负荷很重，如果bi，bo长期不等于 0 ，表示内 存不足，如果disk经常不等于 0 ，且在b中的队列大于 3 ，表示io性能不好。 CPU问题现象： 1.)如果在processes中运行的序列(process r)是连续的大于在系统中的CPU的个数表示系统现在运行比较 慢,有多数的进程等待CPU. 2.)如果r的输出数大于系统中可用CPU个数的 4 倍的话,则系统面临着CPU短缺的问题,或者是CPU的速 率过低,系统中有多数的进程在等待CPU,造成系统中进程运行过慢. 3.)如果空闲时间(cpu id)持续为 0 并且系统时间(cpu sy)是用户时间的两倍(cpu us)系统则面临着CPU资源 的短缺. 解决办法: ##### 当发生以上问题的时候请先调整应用程序对CPU的占用情况.使得应用程序能够更有效的使用CPU.同 时可以考虑增加更多的CPU. 关于CPU的使用情况还可以结合mpstat, ps aux top prstat –a等 等一些相应的命令来综合考虑关于具体的CPU的使用情况,和那些进程在占用大量的CPU时间.一般情 况下，应用程序的问题会比较大一些.比如一些SQL语句不合理等等都会造成这样的现象. 内存问题现象: 内存的瓶颈是由scan rate (sr)来决定的.scan rate是通过每秒的始终算法来进行页扫描的.如果scan rate(sr)连续的大于每秒 200 页则表示可能存在内存缺陷.同样的如果page项中的pi和po这两栏表示每 秒页面的调入的页数和每秒调出的页数.如果该值经常为非零值,也有可能存在内存的瓶颈,当然,如果个 别的时候不为 0 的话,属于正常的页面调度这个是虚拟内存的主要原理. 解决办法: 1.调节applications & servers使得对内存和cache的使用更加有效. 2.增加系统的内存. 3. Implement priority paging in s in pre solaris 8 versions by adding line "set priority paging=1" in /etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file. 关于内存的使用情况还可以结ps aux top prstat –a等等一些相应的命令来综合考虑关于具体的内 存的使用情况,和那些进程在占用大量的内存.一般情况下，如果内存的占用率比较高,但是,CPU的占用 很低的时候,可以考虑是有很多的应用程序占用了内存没有释放,但是,并没有占用CPU时间,可以考虑 应用程序,对于未占用CPU时间和一些后台的程序,释放内存的占用 #### 4.4 、案例 ##### 案例学习： ##### 1 ：在这个例子中,这个系统被充分利用 ``` # vmstat 1 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy wa id 3 0 206564 15092 80336 176080 0 0 0 0 718 26 81 19 0 0 2 0 206564 14772 80336 176120 0 0 0 0 758 23 96 4 0 0 1 0 206564 14208 80336 176136 0 0 0 0 820 20 96 4 0 0 1 0 206956 13884 79180 175964 0 412 0 2680 1008 80 93 7 0 0 2 0 207348 14448 78800 175576 0 412 0 412 763 70 84 16 0 0 2 0 207348 15756 78800 175424 0 0 0 0 874 25 89 11 0 0 1 0 207348 16368 78800 175596 0 0 0 0 940 24 86 14 0 0 1 0 207348 16600 78800 175604 0 0 0 0 929 27 95 3 0 2 3 0 207348 16976 78548 175876 0 0 0 2508 969 35 93 7 0 0 4 0 207348 16216 78548 175704 0 0 0 0 874 36 93 6 0 1 4 0 207348 16424 78548 175776 0 0 0 0 850 26 77 23 0 0 2 0 207348 17496 78556 175840 0 0 0 0 736 23 83 17 0 0 0 0 207348 17680 78556 175868 0 0 0 0 861 21 91 8 0 1 根据观察值,我们可以得到以下结论： 1.有大量的中断(in) 和较少的上下文切换(cs).这意味着一个单一的进程在产生对硬件设备的请求. 2.进一步显示某单个应用,user time(us)经常在85%或者更多.考虑到较少的上下文切换,这个应用应该 还在处理器中被处理. 3.运行队列还在可接受的性能范围内,其中有 2 个地方,是超出了允许限制. 2 ：在这个例子中,内核调度中的上下文切换处于饱和 # vmstat 1 ``` ``` procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy wa id 2 1 207740 98476 81344 180972 0 0 2496 0 900 2883 4 12 57 27 0 1 207740 96448 83304 180984 0 0 1968 328 810 2559 8 9 83 0 0 1 207740 94404 85348 180984 0 0 2044 0 829 2879 9 6 78 7 0 1 207740 92576 87176 180984 0 0 1828 0 689 2088 3 9 78 10 2 0 207740 91300 88452 180984 0 0 1276 0 565 2182 7 6 83 4 3 1 207740 90124 89628 180984 0 0 1176 0 551 2219 2 7 91 0 4 2 207740 89240 90512 180984 0 0 880 520 443 907 22 10 67 0 5 3 207740 88056 91680 180984 0 0 1168 0 628 1248 12 11 77 0 4 2 207740 86852 92880 180984 0 0 1200 0 654 1505 6 7 87 0 6 1 207740 85736 93996 180984 0 0 1116 0 526 1512 5 10 85 0 0 1 207740 84844 94888 180984 0 0 892 0 438 1556 6 4 90 0 根据观察值,我们可以得到以下结论： 1.上下文切换数目高于中断数目,说明kernel中相当数量的时间都开销在上下文切换线程. 2.大量的上下文切换将导致CPU 利用率分类不均衡.很明显实际上等待io 请求的百分比(wa)非常高, 以及user time百分比非常低(us). 3.因为CPU 都阻塞在IO请求上,所以运行队列里也有相当数目的可运行状态线程在等待执行. ``` #### 3 、 iostat 用途：报告中央处理器（CPU）统计信息和整个系统、适配器、tty 设备、磁盘和 CD-ROM 的输入／ 输出统计信息。 语法：iostat [ -c | -d ] [ -k ] [ -t | -m ] [ -V ] [ -x [ device ] ] [ interval [ count ] ] 描述：iostat 命令用来监视系统输入／输出设备负载，这通过观察与它们的平均传送速率相关的物 理磁盘的活动时间来实现。iostat 命令生成的报告可以用来更改系统配置来更好地平衡物理磁盘和适配 器之间的输入／输出负载。 参数：-c为汇报CPU的使用情况；-d为汇报磁盘的使用情况；-k表示每秒按kilobytes字节显示数据； -m表示每秒按M字节显示数据；-t为打印汇报的时间；-v表示打印出版本信息和用法；-x device指定 要统计的设备名称，默认为所有的设备；interval指每次统计间隔的时间；count指按照这个时间间隔统 计的次数。 iostat 结果解析 rrqm/s: 每秒进行 merge 的读操作数目。即 delta(rmerge)/s wrqm/s: 每秒进行 merge 的写操作数目。即 delta(wmerge)/s r/s: 每秒完成的读 I/O 设备次数。即 delta(rio)/s w/s: 每秒完成的写 I/O 设备次数。即 delta(wio)/s rsec/s: 每秒读扇区数。即 delta(rsect)/s wsec/s: 每秒写扇区数。即 delta(wsect)/s rkB/s: 每秒读K字节数。是 rsect/s 的一半，因为每扇区大小为 512 字节。 wkB/s: 每秒写K字节数。是 wsect/s 的一半。 avgrq-sz: 平均每次设备I/O操作的数据大小 (扇区)。即 delta(rsect+wsect)/delta(rio+wio) avgqu-sz: 平均I/O队列长度。即 delta(aveq)/s/1000 (因为aveq的单位为毫秒)。 await: 平均每次设备I/O操作的等待时间 (毫秒)。即 delta(ruse+wuse)/delta(rio+wio) svctm: 平均每次设备I/O操作的服务时间 (毫秒)。即 delta(use)/delta(rio+wio) %util: 一秒中有百分之多少的时间用于 I/O 操作，或者说一秒中有多少时间 I/O 队列是非空的。即 delta(use)/s/1000 (因为use的单位为毫秒) 如果 %util 接近 100%，说明产生的I/O请求太多，I/O系统已经满负荷，该磁盘可能存在瓶颈。 比较重要的参数 %util: 一秒中有百分之多少的时间用于 I/O 操作，或者说一秒中有多少时间 I/O 队列是非空的 svctm: 平均每次设备I/O操作的服务时间 await: 平均每次设备I/O操作的等待时间 avgqu-sz: 平均I/O队列长度 如果%util接近100%,表明i/o请求太多,i/o系统已经满负荷,磁盘可能存在瓶颈,一般%util大于70%,i/o压 力就比较大,读取速度有较多的wait.同时可以结合vmstat查看查看b参数(等待资源的进程数)和wa参数 (IO等待所占用的CPU时间的百分比,高过30%时IO压力高)。 要理解这些性能指标我们先看下图 ##### IO的执行过程的各个参数 上图的左边是iostat显示的各个性能指标，每个性能指标都会显示在一条虚线之上，这表明这个性 能指标是从虚线之上的那个读写阶段开始计量的，比如说图中的w/s从Linux IO scheduler开始穿过硬盘 控制器(CCIS/3ware)，这就表明w/s统计的是每秒钟从Linux IO scheduler通过硬盘控制器的写IO的数量。 结合上图对读IO操作的过程做一个说明，在从OS Buffer Cache传入到OS Kernel(Linux IO scheduler) 的读IO操作的个数实际上是rrqm/s+r/s，直到读IO请求到达OS Kernel层之后，有每秒钟有rrqm/s个 读IO操作被合并，最终转送给磁盘控制器的每秒钟读IO的个数为r/w；在进入到操作系统的设备层 (/dev/sda)之后，计数器开始对IO操作进行计时，最终的计算结果表现是await，这个值就是我们要的 IO响应时间了；svctm是在IO操作进入到磁盘控制器之后直到磁盘控制器返回结果所花费的时间，这 是一个实际IO操作所花的时间，当await与svctm相差很大的时候，我们就要注意磁盘的IO性能了； 而avgrq-sz是从OS Kernel往下传递请求时单个IO的大小，avgqu-sz则是在OS Kernel中IO请求队列 的 平均大小。 现在我们可以将iostat输出结果和我们上面讨论的指标挂钩了 设备IO操作:总IO(io)/s = r/s(读) +w/s(写) =1.46 + 25.28=26.74 平均每次设备I/O操作只需要0.36毫秒完成,现在却需要10.57毫秒完成，因为发出的请求太多(每秒26.74 个)，假如请求时同时发出的，可以这样计算平均等待时间: 平均等待时间=单个I/O服务器时间*(1+2+...+请求总数-1)/请求总数 每秒发出的I/0请求很多,但是平均队列就4,表示这些请求比较均匀,大部分处理还是比较及时 svctm 一般要小于 await (因为同时等待的请求的等待时间被重复计算了)，svctm 的大小一般和磁盘性 能有关，CPU/内存的负荷也会对其有影响，请求过多也会间接导致 svctm 的增加。await 的大小一般 取决于服务时间(svctm) 以及 I/O 队列的长度和 I/O 请求的发出模式。如果 svctm 比较接近 await， 说明 I/O 几乎没有等待时间；如果 await 远大于 svctm，说明 I/O 队列太长，应用得到的响应时间变 慢，如果响应时间超过了用户可以容许的范围，这时可以考虑更换更快的磁盘，调整内核 elevator 算 法，优化应用，或者升级 CPU。 队列长度(avgqu-sz)也可作为衡量系统 I/O 负荷的指标，但由于 avgqu-sz 是按照单位时间的平均值，所 以不能反映瞬间的 I/O 洪水。 **I/O** 系统 **vs.** 超市排队 举一个例子，我们在超市排队 checkout 时，怎么决定该去哪个交款台呢? 首当是看排的队人数， 5 个 人总比 20 人要快吧? 除了数人头，我们也常常看看前面人购买的东西多少，如果前面有个采购了一星 期食品的大妈，那么可以考虑换个队排了。还有就是收银员的速度了，如果碰上了连钱都点不清楚的新 手，那就有的 等了。另外，时机也很重要，可能 5 分钟前还人满为患的收款台，现在已是人去楼空，这时候交款可 是很爽啊，当然，前提是那过去的 5 分钟里所做的事情比排队要有意义 (不过我还没发现什么事情比 排队还无聊的)。 I/O 系统也和超市排队有很多类似之处: r/s+w/s 类似于交款人的总数 平均队列长度(avgqu-sz)类似于单位时间里平均排队人的个数 平均服务时间(svctm)类似于收银员的收款速度 平均等待时间(await)类似于平均每人的等待时间 平均I/O数据(avgrq-sz)类似于平均每人所买的东西多少 I/O 操作率 (％util)类似于收款台前有人排队的时间比例。 我们可以根据这些数据分析出 I/O 请求的模式，以及 I/O 的速度和响应时间。 一个例子 # iostat -x 1 avg-cpu: ％user ％nice ％sys ％idle 16.24 0.00 4.31 79.44 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm ％util /dev/cciss/c0d0 0.00 44.90 1.02 27.55 8.16 579.59 4.08 289.80 20.57 22.35 78.21 5.00 14.29 /dev/cciss/c0d0p1 0.00 44.90 1.02 27.55 8.16 579.59 4.08 289.80 20.57 22.35 78.21 5.00 14.29 /dev/cciss/c0d0p2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 上面的 iostat 输出表明秒有 28.57 次设备 I/O 操作: delta(io)/s = r/s +w/s = 1.02+27.55 = 28.57 (次/秒) 其中写操作占了主体 (w:r = 27:1)。 平均每次设备 I/O 操作只需要 5ms 就可以完成，但每个 I/O 请求却需要等上78ms，为什么? 因为发 出的 I/O 请求太多 (每秒钟约 29 个)，假设这些请求是同时发出的，那么平均等待时间可以这样计算: 平均等待时间 = 单个 I/O 服务时间 * ( 1 + 2 + ... + 请求总数-1) / 请求总数 应用到上面的例子: 平均等待时间 = 5ms * (1+2+...+28)/29 = 70ms，和iostat 给出的 78ms 的平均等待 时间很接近。这反过来表明 I/O 是同时发起的。每秒发出的 I/O 请求很多 (约 29 个)，平均队列却不 长 (只有 2 个 左右)，这表明这 29 个请求的到来并不均匀，大部分时间 I/O 是空闲的。一秒中有 14.29％ 的时间 I/O 队列中是有请求的，也就是说，85.71％ 的时间里I/O 系统无事可做，所有 29 个 I/O 请求都在 142 毫秒之内处理掉了。 delta(ruse+wuse)/delta(io) = await = 78.21 => delta(ruse+wuse)/s =78.21 * delta(io)/s = 78.21*28.57 = 2232.8， 表明每秒内的I/O请求总共需要等待2232.8ms。所以平均队列长度应为 2232.8ms/1000ms = 2.23，而 iostat 给出的平均队列长度 (avgqu-sz) 却为 22.35，为什么?! 因为 iostat 中有bug，avgqu-sz 值应为 2.23， 而不是 22.35。 #### 4 、 uptime uptime命令用于查看服务器运行了多长时间以及有多少个用户登录，快速获知服务器的负荷情况。 uptime的输出包含一项内容是load average，显示了最近 1 ， 5 ， 15 分钟的负荷情况。它的值代表等待 CPU处理的进程数，如果CPU没有时间处理这些进程，load average值会升高；反之则会降低。load average 的最佳值是 1 ，说明每个进程都可以马上处理并且没有CPU cycles被丢失。对于单CPU的机器， 1 或者 2 是可以接受的值；对于多路CPU的机器，load average值可能在 8 到 10 之间。也可以使用uptime命 令来判断网络性能。例如，某个网络应用性能很低，通过运行uptime查看服务器的负荷是否很高，如 果不是，那么问题应该是网络方面造成的。 以下是uptime的运行实例： 9:24am up 19:06, 1 user, load average: 0.00, 0.00, 0.00 也可以查看/proc/loadavg和/proc/uptime两个文件，注意不能编辑/proc中的文件，要用cat等命令来查 看，如： liyawei:~ # cat /proc/loadavg 0.0 0.00 0.00 1/55 5505 uptime命令用法十分简单：直接输入 # uptime 例： 18:02:41 up 41 days, 23:42, 1 user, load average: 0.00, 0.00, 0.00 ##### 1 可以被认为是最优的负载值。负载是会随着系统不同改变得。单CPU系统1-3和SMP系统6-10都是 ##### 可能接受的。 另外还有一个参数 -V ，是用来查询版本的。 (注意是大写的字母v) [linux @ localhost]$ uptime -V procps version 3.2.7 [linux @ localhost]$ uptime 显示结果为： 10:19:04 up 257 days, 18:56, 12 users, load average: 2.10, 2.10,2.09 显示内容说明： 10:19:04 //系统当前时间 up 257 days, 18:56 //主机已运行时间,时间越大，说明你的机器越稳定。 12 user //用户连接数，是总连接数而不是用户数 load average // 系统平均负载，统计最近 1 ， 5 ， 15 分钟的系统平均负载 那么什么是系统平均负载呢？ 系统平均负载是指在特定时间间隔内运行队列中的平均进程数。如 果每个CPU内核的当前活动进程数不大于 3 的话，那么系统的性能是良好的。如果每个CPU内核的任 务数大于 5 ，那么这台机器的性能有严重问题。如果你的linux主机是 1 个双核CPU的话，当Load Average 为 6 的时候说明机器已经被充分使用了。 #### 5 、 W w命令主要是查看当前登录的用户，这个命令相对来说比较简单。我们来看一下截图。 在上面这个截图里面呢，第一列user，代表登录的用户，第二列，tty代表用户登录的终端号，因为在 linux中并不是只有一个终端的，pts/2代表是图形界面的第二个终端（这仅是个人意见，网上的对pts 的看法可能有些争议）。第三列FROM代表登录的地方，如果是远程登录的，会显示ip地址，:0 表示的 是 display 0:0，意思就是主控制台的第一个虚拟终端。第四列login@代表登录的时间，第五列的IDLE 代表系统的闲置时间。最后一列what是代表正在运行的进程，因为我正在运行w命令，所以咋root 显示w。 #### 5 、 mpstat mpstat （RHEL5默认不安装） mpstat是MultiProcessor Statistics的缩写，是实时系统监控工具。其报告与CPU的一些统计信息， 这些信息存放在/proc/stat文件中。在多CPUs系统里，其不但能查看所有CPU的平均状况信息，而且 能够查看特定CPU的信息。下面只介绍mpstat与CPU相关的参数，mpstat的语法如下： mpstat [-P {|ALL}] [internal [count]] 参数的含义如下： -P {|ALL} 表示监控哪个CPU， cpu在[0,cpu个数-1]中取值 internal 相邻的两次采样的间隔时间 count 采样的次数，count只能和delay一起使用 当没有参数时，mpstat则显示系统启动以后所有信息的平均值。有interval时，第一行的信息自系 统启动以来的平均信息。 从第二行开始，输出为前一个interval时间段的平均信息。与CPU有关的输出的含义如下： [oracle@Test ~]$ mpstat -P ALL Linux 2.6.18-194.el5 (Test.linux.com) 2010年 06 月 22 日 09 时 18 分 18 秒 CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 09 时 18 分 18 秒 all 0.06 0.00 0.43 0.78 0.00 0.00 0.00 98.71 1069.35 09 时 18 分 18 秒 0 0.05 0.00 0.36 0.17 0.02 0.00 0.00 99.41 1032.01 09 时 18 分 18 秒 1 0.04 0.00 0.42 0.07 0.00 0.00 0.00 99.47 0.26 09 时 18 分 18 秒 2 0.11 0.00 0.28 0.08 0.00 0.00 0.00 99.52 0.00 09 时 18 分 18 秒 3 0.07 0.00 0.48 0.05 0.00 0.00 0.00 99.39 0.01 09 时 18 分 18 秒 4 0.08 0.00 0.19 5.63 0.00 0.02 0.00 94.08 24.51 09 时 18 分 18 秒 5 0.05 0.00 0.63 0.11 0.00 0.00 0.00 99.21 0.22 09 时 18 分 18 秒 6 0.07 0.00 0.45 0.10 0.00 0.01 0.00 99.36 12.33 09 时 18 分 18 秒 7 0.05 0.00 0.64 0.07 0.00 0.00 0.00 99.24 0.00 参数 解释 从/proc/stat获得数据CPU 处理器ID user 在internal时间段里，用户的CPU时间（%） ，不包含 nice值为负 进程 (usr/total)*100 nice 在internal时间段里，nice值为负进程的CPU时间（%） (nice/total)*100 system 在internal时间段里，核心时间（%） (system/total)*100 iowait 在internal时间段里，硬盘IO等待时间（%） (iowait/total)*100 irq 在internal时间段里 ，硬中断时间（%） (irq/total)*100 soft 在internal时间段里，软中断时间（%） (softirq/total)*100 idle 在internal时间段里，CPU除去等待磁盘IO操作外的因为任何原因而空闲的时间闲置时间（%） (idle/total)*100 intr/s 在internal时间段里，每秒CPU接收的中断的次数intr/total)*100 CPU总的工作时间=total_cur=user+system+nice+idle+iowait+irq+softirq total_pre=pre_user+ pre_system+ pre_nice+ pre_idle+ pre_iowait+ pre_irq+ pre_softirq user=user_cur – user_pre total=total_cur-total_pre 其中_cur 表示当前值，_pre表示interval时间前的值。上表中的所有值可取到两位小数点。 范例 1 ：average mode (粗略信息)当mpstat不带参数时，输出为从系统启动以来的平均值。 [work@builder linux-2.6.14]$ mpstat Linux 2.6.9-5.31AXsmp (builder.redflag-linux.com) 12/16/2005 09:38:46 AM CPU %user %nice %system %iowait %irq %soft %idle intr/s 09:38:48 AM all 23.28 0.00 1.75 0.50 0.00 0.00 74.47 1018.59 范例2: 每 2 秒产生了 2 个处理器的统计数据报告 下面的命令可以每 2 秒产生了 2 个处理器的统计数据报告，一共产生三个interval 的信息，然后再给出 这三个interval的平均信息。默认时，输出是按照CPU 号排序。第一个行给出了从系统引导以来的所有 活跃数据。接下来每行对应一个处理器的活跃状态。 [root@server yum_dir]# mpstat -P ALL 2 3 Linux 2.6.18-164.el5 (server.sys.com) 01/04/2010 09:34:20 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 09:34:22 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 1001.49 09:34:22 PM 0 0.00 0.00 0.50 0.00 0.00 0.00 0.00 99.50 1001.00 09:34:22 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 09:34:22 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 09:34:24 PM all 0.00 0.00 0.25 0.00 0.00 0.00 0.00 99.75 1005.00 09:34:24 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 1005.50 ##### 09:34:24 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 09:34:24 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 09:34:26 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 1001.49 09:34:26 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 1001.00 09:34:26 PM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 Average: CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s Average: all 0.00 0.00 0.08 0.00 0.00 0.00 0.00 99.92 1002.66 Average: 0 0.00 0.00 0.17 0.00 0.00 0.00 0.00 99.83 1002.49 Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 范例 3 ：比较带参数和不带参数的mpstat的结果。 在后台开一个2G的文件 # cat 1.img & 然后在另一个终端运行mpstat命令 [root@server ~]# cat 1.img & [1] 6934 [root@server ~]# mpstat Linux 2.6.18-164.el5 (server.sys.com) 01/04/2010 10:17:31 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:17:31 PM all 0.07 0.02 0.25 0.21 0.01 0.04 0.00 99.40 1004.57 [root@server ~]# mpstat Linux 2.6.18-164.el5 (server.sys.com) 01/04/2010 10:17:35 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:17:35 PM all 0.07 0.02 0.25 0.21 0.01 0.04 0.00 99.39 1004.73 [root@server ~]# mpstat Linux 2.6.18-164.el5 (server.sys.com) 01/04/2010 10:17:39 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:17:39 PM all 0.07 0.02 0.25 0.21 0.01 0.04 0.00 99.38 1004.96 [root@server ~]# mpstat Linux 2.6.18-164.el5 (server.sys.com) 01/04/2010 10:17:44 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:17:44 PM all 0.07 0.02 0.26 0.21 0.01 0.05 0.00 99.37 1005.20 [root@server ~]# mpstat 3 10 Linux 2.6.18-164.el5 (server.sys.com) 01/04/2010 10:17:55 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 10:17:58 PM all 13.12 0.00 20.93 0.00 1.83 9.80 0.00 54.32 2488.08 10:18:01 PM all 10.82 0.00 19.30 0.83 1.83 9.32 0.00 57.90 2449.83 10:18:04 PM all 10.95 0.00 20.40 0.17 1.99 8.62 0.00 57.88 2384.05 10:18:07 PM all 10.47 0.00 18.11 0.00 1.50 8.47 0.00 61.46 2416.00 10:18:10 PM all 11.81 0.00 22.63 0.00 1.83 11.98 0.00 51.75 2210.60 10:18:13 PM all 6.31 0.00 10.80 0.00 1.00 5.32 0.00 76.58 1795.33 10:18:19 PM all 1.75 0.00 3.16 0.75 0.25 1.25 0.00 92.85 1245.18 10:18:22 PM all 11.94 0.00 19.07 0.00 1.99 8.29 0.00 58.71 2630.46 10:18:25 PM all 11.65 0.00 19.30 0.50 2.00 9.15 0.00 57.40 2673.91 10:18:28 PM all 11.44 0.00 21.06 0.33 1.99 10.61 0.00 54.56 2369.87 Average: all 9.27 0.00 16.18 0.30 1.50 7.64 0.00 65.11 2173.54 [root@server ~]# 上两表显示出当要正确反映系统的情况，需要正确使用命令的参数。vmstat 和iostat 也需要注意这一 问题。 #### 6 、 pmap pmap命令可以显示进程的内存映射，使用这个命令可以找出造成内存瓶颈的原因。 # pmap -d PID 显示PID为 47394 进程的内存信息。 # pmap -d 47394 输出样例： 47394: /usr/bin/php-cgi Address Kbytes Mode Offset Device Mapping 0000000000400000 2584 r-x-- 0000000000000000 008:00002 php-cgi 0000000000886000 140 rw--- 0000000000286000 008:00002 php-cgi 00000000008a9000 52 rw--- 00000000008a9000 000:00000 [ anon ] 0000000000aa8000 76 rw--- 00000000002a8000 008:00002 php-cgi 000000000f678000 1980 rw--- 000000000f678000 000:00000 [ anon ] 000000314a600000 112 r-x-- 0000000000000000 008:00002 ld-2.5.so 000000314a81b000 4 r---- 000000000001b000 008:00002 ld-2.5.so 000000314a81c000 4 rw--- 000000000001c000 008:00002 ld-2.5.so 000000314aa00000 1328 r-x-- 0000000000000000 008:00002 libc-2.5.so 000000314ab4c000 2048 ----- 000000000014c000 008:00002 libc-2.5.so ..... ...... 00002af8d48fd000 4 rw--- 0000000000006000 008:00002 xsl.so 00002af8d490c000 40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so 00002af8d4916000 2044 ----- 00 0000000000a000 008:00002 libnss_files-2.5.so 00002af8d4b15000 4 r---- 0000000000009000 008:00002 libnss_files-2.5.so 00002af8d4b16000 4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so 00002af8d4b17000 768000 rw-s- 0000000000000000 000:00009 zero (deleted) 00007fffc95fe000 84 rw--- 00007ffffffea000 000:00000 [ stack ] ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ] mapped: 933712K writeable/private: 4304K shared: 768000K 最后一行非常重要： * mapped: 933712K 内存映射所占空间大小 * writeable/private: 4304K 私有地址空间大小 * shared: 768000K 共享地址空间大小 #### 7 、 sar sar是一个优秀的一般性能监视工具，它可以输出Linux所完成的几乎所有工作的数据。sar命令在 sysetat rpm中提供。示例中使用sysstat版本5.0.5，这是稳定的最新版本之一。关于版本和下载信息， 请访问sysstat主页http://perso.wanadoo.fr/sebastien.godard/。 sar可以显示CPU、运行队列、磁盘I/O、分页 （交换区）、内存、CPU中断、网络等性能数据。最重要 的sar功能是创建数据文件。每一个Linux系统都应该通过cron工作收集sar数据。该sar数据文件为 系统管理员提供历史性能信息。这个功能非常重要，它将sar和其他性能工具区分开。如果一个夜晚批 处理工作正常运行两次，直到下一个早上才会发现这种情况（除非被叫醒）。我们需要具备研究 12 小时 以前的性能数据的能力。sar数据收集器提供了这种能力。 sar 命令行的常用格式： sar [options] [-A] [-o file] t [n] 在命令行中，n 和t 两个参数组合起来定义采样间隔和次数，t为采样间隔，是必须有的参数，n 为采样次数，是可选的，默认值是 1 ，-o file表示将命令结果以二进制格式存放在文件中，file 在此处 不是关键字，是文件名。options 为命令行选项，sar命令的选项很多，下面只列出常用选项： - A：所有报告的总和。 - u：CPU利用率 - v：进程、I节点、文件和锁表状态。 - d：硬盘使用报告。 - r：没有使用的内存页面和硬盘块。 - g：串口I/O的情况。 - b：缓冲区使用情况。 - a：文件读写情况。 - c：系统调用情况。 - R：进程的活动情况。 - y：终端设备活动情况。 - w：系统交换活动。 #### 7.1 、 CPU 统计数据 sar -u输出显示CPU信息。-u选项是sar的默认选项。该输出以百分比显示CPU的使用情况。表3-2 解释该输出。 表3-2 sar -u字段 字 段 说 明 CPU CPU编号 %user CPU 花费在用户进程（如应用程序、Shell 脚本或与 该用户进行的交互）上的时间的百分比。 %nice (^) 运行正常进程所花的时间 %system 在内核模式（系统）中运行进程所花的时间, 也就是 CPU 用来执行核心任务的时间的百分比。 %iowait 没有进程在该CPU上执行时，处理器等待I/O完成的 时间 %idle 没有进程在该CPU上执行的时间 这些看起来应该比较熟悉，它和top报告中的CPU信息内容相同。以下显示输出格式： [oracle@Test ~]$ sar -u -o aixi 60 5 Linux 2.6.18-194.el5 (Test.linux.com) 06/22/10 13:41:25 CPU %user %nice %system %iowait %steal %idle 13:42:25 all 0.28 0.00 0.21 1.17 0.00 98.34 13:43:25 all 0.23 0.00 0.16 1.14 0.00 98.46 ``` 13:44:25 all 0.27 0.00 0.21 1.40 0.00 98.12 13:45:25 all 0.26 0.00 0.19 0.99 0.00 98.56 13:46:25 all 0.32 0.00 0.23 1.39 0.00 98.05 Average: all 0.27 0.00 0.20 1.22 0.00 98.31 ``` ``` 每 60 秒采样一次，连续采样 5 次，观察CPU 的使用情况，并将采样结果以二进制形式存入当前目 录下的文件aixi中 ``` ``` 在所有的显示中，我们应主要注意%wio和%idle，%wio的值过高，表示硬盘存在I/O瓶颈，%idle 值高，表示CPU较空闲，如果%idle值高但系统响应慢时，有可能是CPU等待分配内存，此时应加大内 存容量。%idle值如果持续低于 10 ，那么系统的CPU处理能力相对较低，表明系统中最需要解决的资源 是CPU。另外任何sar报告的第一列都是时间戳。 ``` ``` 如果要查看二进制文件aixi中的内容，则需键入如下sar命令： ``` ``` # sar -u -f aixi ``` ``` 可见，sar命令即可以实时采样，又可以对以往的采样结果进行查询。 ``` ``` 我们本来可以研究使用-f选项通过sadc创建的文件。这个sar语法显示sar -f/var/log/ sa/sa21的输出： ``` 在多CPU Linux系统中，sar命令也可以为每个CPU分解该信息，如以下sar -u -P ALL 5 5输出所示： #### 磁盘 I/O 统计数据 sar是一个研究磁盘I/O的优秀工具。以下是sar磁盘I/O输出的一个示例。 第一行-d显示磁盘I/O信息， 5 2 选项是间隔和迭代，就像sar数据收集器那样。表3-3列出了字段和说 明。 表3-3 sar -d字段 字 段 说 明 DEV 磁盘设备 tps 每秒传输数（或者每秒IO数） rd_sec/s 每秒 512 字节读取数 wr_sec/s 每秒 512 字节写入数 512 只是一个测量单位，不表示所有磁盘I/O均使用 512 字节块。DEV列是dev#-#格式的磁盘设备，其中 第一个#是设备主编号，第二个#是次编号或者连续编号。对于大于2.5的内核，sar使用次编号。例如， 在sar -d输出中看到的dev3-0和dev3-1。它们对应于/dev/hda和/dev/hdal。请看/dev中的以下各项： /dev/hda有主编号 3 和次编号 0 。hda1有主编号 3 和次编号 1 。 **3.2.4** 网络统计数据 sar提供四种不同的语法选项来显示网络信息。-n选项使用四个不同的开关：DEV、EDEV、SOCK和FULL。 DEV显示网络接口信息，EDEV显示关于网络错误的统计数据，SOCK显示套接字信息，FULL显示所有三 个开关。它们可以单独或者一起使用。表3-4显示通过-n DEV选项报告的字段。 表3-4 sar -n DEV字段 字 段 说 明 IFACE (^) LAN接口 rxpck/s (^) 每秒钟接收的数据包 txpck/s 每秒钟发送的数据包 rxbyt/s 每秒钟接收的字节数 txbyt/s 每秒钟发送的字节数 rxcmp/s 每秒钟接收的压缩数据包 txcmp/s 每秒钟发送的压缩数据包 rxmcst/s 每秒钟接收的多播数据包 以下是使用-n DEV选项的sar输出： 关于网络错误的信息可以用sar -n EDEV显示。表3-5列出了显示的字段。 表3-5 sar -n EDEV字段 字 段 说 明 IFACE LAN接口 rxerr/s 每秒钟接收的坏数据包 txerr/s 每秒钟发送的坏数据包 coll/s 每秒冲突数 rxdrop/s 因为缓冲充满，每秒钟丢弃的已接收数据包数 txdrop/s 因为缓冲充满，每秒钟丢弃的已发送数据包数 txcarr/s 发送数据包时，每秒载波错误数 rxfram/s 每秒接收数据包的帧对齐错误数 rxfifo/s 接收的数据包每秒FIFO过速的错误数 txfifo/s 发送的数据包每秒FIFO过速的错误数 SOCK参数显示IPCS套接字信息。表3-6列出显示的字段及其意义。 表3-6 sar -n SOCK字段 字 段 说 明 totsck 使用的套接字总数量 tcpsck 使用的TCP套接字数量 udpsck 使用的UDP套接字数量 rawsck 使用的raw套接字数量 ip-frag 使用的IP段数量 sar可以产生许多其他报告。我们有必要仔细阅读sar（ 1 ）手册页，查看是否有自己需要的其他报告 ``` 例二：使用命行sar -v t n ``` ``` 例如，每 30 秒采样一次，连续采样 5 次，观察核心表的状态，需键入如下命令： ``` ``` # sar -v 30 5 ``` ``` 屏幕显示： SCO_SV scosysv 3.2v5.0.5 i80386 10/01/2001 10:33:23 proc-sz ov inod-sz ov file-sz ov lock-sz (-v) 10:33:53 305/ 321 0 1337/2764 0 1561/1706 0 40/ 128 10:34:23 308/ 321 0 1340/2764 0 1587/1706 0 37/ 128 10:34:53 305/ 321 0 1332/2764 0 1565/1706 0 36/ 128 10:35:23 308/ 321 0 1338/2764 0 1592/1706 0 37/ 128 10:35:53 308/ 321 0 1335/2764 0 1591/1706 0 37/ 128 ``` ``` 显示内容包括： ``` ``` proc-sz：目前核心中正在使用或分配的进程表的表项数，由核心参数MAX-PROC控制。 ``` ``` inod-sz：目前核心中正在使用或分配的i节点表的表项数，由核心参数MAX-INODE控制。 ``` ``` file-sz： 目前核心中正在使用或分配的文件表的表项数，由核心参数MAX-FILE控制。 ``` ``` ov：溢出出现的次数。 ``` ``` Lock-sz：目前核心中正在使用或分配的记录加锁的表项数，由核心参数MAX-FLCKRE控制。 ``` ``` 显示格式为 ``` ``` 实际使用表项/可以使用的表项数 ``` ``` 显示内容表示，核心使用完全正常，三个表没有出现溢出现象，核心参数不需调整，如果出现溢出 时，要调整相应的核心参数，将对应的表项数加大。 ``` 例三：使用命行sar -d t n 例如，每 30 秒采样一次，连续采样 5 次，报告设备使用情况，需键入如下命令： # sar -d 30 5 屏幕显示： SCO_SV scosysv 3.2v5.0.5 i80386 10/01/2001 11:06:43 device %busy avque r+w/s blks/s avwait avserv (-d) 11:07:13 wd-0 1.47 2.75 4.67 14.73 5.50 3.14 11:07:43 wd-0 0.43 18.77 3.07 8.66 25.11 1.41 11:08:13 wd-0 0.77 2.78 2.77 7.26 4.94 2.77 11:08:43 wd-0 1.10 11.18 4.10 11.26 27.32 2.68 11:09:13 wd-0 1.97 21.78 5.86 34.06 69.66 3.35 Average wd-0 1.15 12.11 4.09 15.19 31.12 2.80 显示内容包括： device： sar命令正在监视的块设备的名字。 %busy： 设备忙时，传送请求所占时间的百分比。 avque： 队列站满时，未完成请求数量的平均值。 r+w/s： 每秒传送到设备或从设备传出的数据量。 blks/s： 每秒传送的块数，每块 512 字节。 avwait： 队列占满时传送请求等待队列空闲的平均时间。 avserv： 完成传送请求所需平均时间（毫秒）。 在显示的内容中，wd- 0是硬盘的名字，%busy的值比较小，说明用于处理传送请求的有效时间太少， 文件系统效率不高，一般来讲，%busy值高些，avque值低些，文件系统的效率比较高，如果%busy和 avque值相对比较高，说明硬盘传输速度太慢，需调整。 例四：使用命行sar -b t n 例如，每 30 秒采样一次，连续采样 5 次，报告缓冲区的使用情况，需键入如下命令： # sar -b 30 5 屏幕显示： SCO_SV scosysv 3.2v5.0.5 i80386 10/01/2001 14:54:59 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s (-b) 14:55:29 0 147 100 5 21 78 0 0 14:55:59 0 186 100 5 25 79 0 0 ##### 14:56:29 4 232 98 8 58 86 0 0 ##### 14:56:59 0 125 100 5 23 76 0 0 ##### 14:57:29 0 89 100 4 12 66 0 0 Average 1 156 99 5 28 80 0 0 显示内容包括： bread/s： 每秒从硬盘读入系统缓冲区buffer的物理块数。 lread/s： 平均每秒从系统buffer读出的逻辑块数。 %rcache： 在buffer cache中进行逻辑读的百分比。 bwrit/s： 平均每秒从系统buffer向磁盘所写的物理块数。 lwrit/s： 平均每秒写到系统buffer逻辑块数。 %wcache： 在buffer cache中进行逻辑读的百分比。 pread/s： 平均每秒请求物理读的次数。 pwrit/s： 平均每秒请求物理写的次数。 在显示的内容中，最重要的是%cache和%wcache两列，它们的值体现着buffer的使用效率，%rcache 的值小于 90 或者%wcache的值低于 65 ，应适当增加系统buffer的数量，buffer数量由核心参数NBUF 控制，使%rcache达到 90 左右，%wcache达到 80 左右。但buffer参数值的多少影响I/O效率，增加 buffer，应在较大内存的情况下，否则系统效率反而得不到提高。 例五：使用命行sar -g t n 例如，每 30 秒采样一次，连续采样 5 次，报告串口I/O的操作情况，需键入如下命令： # sar -g 30 5 屏幕显示： SCO_SV scosysv 3.2v5.0.5 i80386 11/22/2001 17:07:03 ovsiohw/s ovsiodma/s ovclist/s (-g) 17:07:33 0.00 0.00 0.00 17:08:03 0.00 0.00 0.00 17:08:33 0.00 0.00 0.00 17:09:03 0.00 0.00 0.00 17:09:33 0.00 0.00 0.00 Average 0.00 0.00 0.00 显示内容包括： ovsiohw/s：每秒在串口I/O硬件出现的溢出。 ``` ovsiodma/s：每秒在串口I/O的直接输入输出通道高速缓存出现的溢出。 ``` ovclist/s ：每秒字符队列出现的溢出。 在显示的内容中，每一列的值都是零，表明在采样时间内，系统中没有发生串口I/O溢出现象。 sar命令的用法很多，有时判断一个问题，需要几个sar命令结合起来使用，比如，怀疑CPU存在 瓶颈，可用sar -u 和sar -q来看，怀疑I/O存在瓶颈，可用sar -b、sar -u和sar -d来看。 -------------------------------------------------------------------------------- Sar -A 所有的报告总和 -a 文件读，写报告 -B 报告附加的buffer cache使用情况 -b buffer cache使用情况 -c 系统调用使用报告 -d 硬盘使用报告 -g 有关串口I/O情况 -h 关于buffer使用统计数字 -m IPC消息和信号灯活动 -n 命名cache -p 调页活动 -q 运行队列和交换队列的平均长度 -R 报告进程的活动 -r 没有使用的内存页面和硬盘块 -u CPU利用率 -v 进程，i节点，文件和锁表状态 -w 系统交换活动 -y TTY设备活动 -a 报告文件读，写报告 sar –a 5 5 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/07/2002 11:45:40 iget/s namei/s dirbk/s (-a) 11:45:45 6 2 2 11:45:50 91 20 28 11:45:55 159 20 18 11:46:00 157 21 19 11:46:05 177 30 35 Average 118 18 20 iget/s 每秒由i节点项定位的文件数量 namei/s 每秒文件系统路径查询的数量 dirbk/s 每秒所读目录块的数量 ##### ＊这些值越大，表明核心花在存取用户文件上的时间越多，它反映着一些程序和应用文件系统产生 的负荷。一般地，如果iget/s与namei/s的比值大于 5 ，并且namei/s的值大于 30 ，则说明文件系统 是低效的。这时需要检查文件系统的自由空间，看看是否自由空间过少。 -b 报告缓冲区（buffer cache）的使用情况 sar -b 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/07/2002 13:51:28 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s (-b) 13:51:30 382 1380 72 131 273 52 0 0 13:51:32 378 516 27 6 22 72 0 0 13:51:34 172 323 47 39 57 32 0 0 Average 310 739 58 58 117 50 0 0 bread/s 平均每秒从硬盘（或其它块设备）读入系统buffer的物理块数 lread/s 平均每秒从系统buffer读出的逻辑块数 %rcache 在buffer cache中进行逻辑读的百分比（即 100 ％ - bread/lreads） bwrit/s 平均每秒从系统buffer向磁盘（或其它块设备）所写的物理块数 lwrit/s 平均每秒写到系统buffer的逻辑块数 %wcache 在buffer cache中进行逻辑写的百分比（即 100 ％ - bwrit/lwrit）. pread/sgu 平均每秒请求进行物理读的次数 pwrit/s 平均每秒请求进行物理写的次数 ＊所显示的内容反映了目前与系统buffer有关的读，写活。在所报告的数字中，最重要的是%rcache 和%wcache（统称为cache命中率）两列，它们具体体现着系统buffer的效率。衡量cache效率的标准 是它的命中率值的大小。 ＊如果%rcache的值小于 90 或者%wcache的值低于 65 ，可能就需要增加系统buffer的数量。如果 在系统的应用中，系统的I/O活动十分频繁，并且在内存容量配置比较大时，可以增加buffer cache， 使%rcache达到 95 左右，%wcache达到 80 左右。 ＊系统buffer cache中，buffer的数量由核心参数NBUF控制。它是一个要调的参数。系统中buffer 数量的多少是影响系统I/O效率的瓶颈。要增加系统buffer数量，则要求应该有较大的内存配置。否 则一味增加buffer数量，势必减少用户进程在内存中的运行空间，这同样会导致系统效率下降。 -c 报告系统调用使用情况 sar -c 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/07/2002 17:02:42 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s (-c) 17:02:44 2262 169 141 0.00 0.00 131250 22159 17:02:46 1416 61 38 0.00 0.00 437279 6464 17:02:48 1825 43 25 0.00 0.00 109397 42331 Average 1834 91 68 0.00 0.00 225975 23651 scall/s 每秒使用系统调用的总数。一般地，当4~6个用户在系统上工作时，每秒大约 30 个左右。 sread/s 每秒进行读操作的系统调用数量。 swrit/s 每秒进行写操作的系统调用数量。 fork/s 每秒fork系统调用次数。当4~6个用户在系统上工作时，每秒大约0.5秒左右。 exec/s 每秒exec系统调用次数。 rchar/s 每秒由读操作的系统调用传送的字符（以字节为单位）。 wchar/s 每秒由写操作的系统调用传送的字符（以字节为单位）。 ＊如果scall/s持续地大于 300 ，则表明正在系统中运行的可能是效率很低的应用程序。在比较典 型的情况下，进行读操作的系统调用加上进行写操作的系统调用之和，约是scall的一半左右。 -d 报告硬盘使用情况 sar -d 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/07/2002 17:27:49 device %busy avque r+w/s blks/s avwait avserv (-d) 17:27:51 ida-0 6.93 1.00 13.86 259.41 0.00 5.00 ida-1 0.99 1.00 17.33 290.10 0.00 0.57 17:27:53 ida-0 75.50 1.00 54.00 157.00 0.00 13.98 ida-1 9.50 1.00 12.00 75.00 0.00 7.92 17:27:55 ida-0 7.46 1.00 46.77 213.93 0.00 1.60 ida-1 17.41 1.00 57.71 494.53 0.00 3.02 Average ida-0 29.85 1.00 38.14 210.28 0.00 7.83 ida-1 9.29 1.00 29.02 286.90 0.00 3.20 device 这是sar命令正在监视的块设备的名字。 %busy 设备忙时，运行传送请求所占用的时间。这个值以百分比表示。 avque 在指定的时间周期内，没有完成的请求数量的平均值。仅在队列被占满时取这个值。 r+w/s 每秒传送到设备或者从设备传送出的数据量。 blks/s 每秒传送的块数。每块 512 个字节。 avwait 传送请求等待队列空闲的平均时间（以毫秒为单位）。仅在队列被占满时取这个值。 avserv 完成传送请求所需平均时间（以毫秒为单位） ＊ida- 0和ida- 1是硬盘的设备名字。在显示的内容中，如果%busy的值比较小，说明用于处理传 送请求的有效时间太少，文件系统的效率不高。要使文件系统的效率得到优化，应使%busy的数值相对 高一些，而avque的值应该低一些。 -g 报告有关串口I/O情况 sar -g 3 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/13/2002 11:10:09 ovsiohw/s ovsiodma/s ovclist/s (-g) 11:10:12 0.00 0.00 0.00 11:10:15 0.00 0.00 0.00 11:10:18 0.00 0.00 0.00 Average 0.00 0.00 0.00 ovsiohw/s 每秒在串囗I/O硬件出现的溢出。 ovsiodma/s 每秒在串囗I/O的直接输入，输出信道高速缓存出现的溢出。 ovclist/s 每秒字符队列出现的溢出。 -m 报告进程间的通信活动（IPC消息和信号灯活动）情况 sar -m 4 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/13/2002 13:24:28 msg/s sema/s (-m) 13:24:32 2.24 9.95 13:24:36 2.24 21.70 13:24:40 2.00 36.66 Average 2.16 22.76 msg/s 每秒消息操作的次数（包括发送消息的接收信息）。 sema/s 每秒信号灯操作次数。 ＊信号灯和消息作为进程间通信的工具，如果在系统中运行的应用过程中没有使用它们，那么由 sar命令报告的msg 和sema的值都将等于0.00。如果使用了这些工具，并且其中或者msg/s大于 100 ， 或者sema/s大于 100 ，则表明这样的应用程序效率比较低。原因是在这样的应用程序中，大量的时间 花费在进程之间的沟通上，而对保证进程本身有效的运行时间必然产生不良的影响。 -n 报告命名缓冲区活动情况 sar -n 4 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/13/2002 13:37:31 c_hits cmisses (hit %) (-n) 13:37:35 1246 71 (94%) 13:37:39 1853 81 (95%) 13:37:43 969 56 (94%) Average 1356 69 (95%) c_hits cache命中的数量。 cmisses cache未命中的数量。 (hit %) 命中数量/(命中数理+未命中数量)。 ＊不难理解，(hit %)值越大越好，如果它低于 90 ％，则应该调整相应的核心参数。 -p 报告分页活动 sar -p 5 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/13/2002 13:45:26 vflt/s pflt/s pgfil/s rclm/s (-p) 13:45:31 36.25 50.20 0.00 0.00 13:45:36 32.14 58.48 0.00 0.00 ##### 13:45:41 79.80 58.40 0.00 0.00 Average 49.37 55.69 0.00 0.00 vflt/s 每秒进行页面故障地址转换的数量（由于有效的页面当前不在内存中）。 pflt/s 每秒来自由于保护错误出现的页面故障数量（由于对页面的非法存，取引起的页面故障）。 pgfil/s 每秒通过”页—入”满足vflt/s的数量。 rclm/s 每秒由系统恢复的有效页面的数量。有效页面被增加到自由页面队列上。 ＊如果vflt/s的值高于 100 ，可能预示着对于页面系统来说，应用程序的效率不高，也可能分页 参数需要调整，或者内存配置不太合适。 -q 报告进程队列（运行队列和交换队列的平均长度）情况 sar -q 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/13/2002 14:25:50 runq-sz %runocc swpq-sz %swpocc (-q) 14:25:52 4.0 50 14:25:54 9.0 100 14:25:56 9.0 100 Average 7.3 100 runq-sz 准备运行的进程运行队列。 %runocc 运行队列被占用的时间（百分比） swpq-sz 要被换出的进程交换队列。 %swpocc 交换队列被占用的时间（百分比）。 ＊如果%runocc大于 90 ，并且runq-sz的值大于 2 ，则表明CPU的负载较重。其直接后果，可能使 系统的响应速度降低。如果%swpocc大于 20 ，表明交换活动频繁，将严重导致系统效率下降。解决的办 法是加大内存或减少缓存区数量，从而减少交换及页—入,页—出活动。 -r 报告内存及交换区使用情况（没有使用的内存页面和硬盘块） sar -r 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/14/2002 10:14:19 freemem freeswp availrmem availsmem (-r) 10:14:22 279729 6673824 93160 1106876 10:14:24 279663 6673824 93160 1106876 10:14:26 279661 6673824 93160 1106873 Average 279684 6673824 93160 1106875 freemem 用户进程可以使用的内存页面数，4KB为一个页面。 freeswp 用于进程交换可以使用的硬盘盘块，512B为一个盘块。 -u CPU利用率 sar -u 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/14/2002 10:27:23 %usr %sys %wio %idle (-u) 10:27:25 2 3 8 88 10:27:27 3 3 5 89 10:27:29 0 0 0 100 Average 2 2 4 92 . %usr cpu处在用户模式下时间（百分比） %sys cpu处在系统模式下时间（百分比） %wio cpu等待输入，输出完成（时间百分比） %idle cpu空闲时间（百分比） ＊在显示的内容中，%usr和 %sys这两个值一般情况下对系统无特别影响，%wio的值不能太高， 如果%wio的值过高，则CPU花在等待输入，输出上的时间太多，这意味着硬盘存在I/O瓶颈。如果%idle 的值比较高，但系统响应并不快，那么这有可能是CPU花时间等待分配内存引起的。%idle的值可以较 深入帮助人们了解系统的性能，在这种情况上，%idle的值处于40~100之间，一旦它持续低于 30 ，则 表明进程竟争的主要资源不是内存而是CPU。 ＊在有大量用户运行的系统中，为了减少CPU的压力，应该使用智能多串卡，而不是非智能多串卡。 智能多串卡可以承担CPU的某些负担。 ＊此外，如果系统中有大型的作业运行，应该把它们合理调度，错开高峰，当系统相对空闲时再运 行。 -v 报告系统表的内容（进程，i节点，文件和锁表状态） sar -v 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/14/2002 10:56:46 proc-sz ov inod-sz ov file-sz ov lock-sz (-v) 10:56:48 449/ 500 0 994/4147 0 1313/2048 0 5/ 128 10:56:50 450/ 500 0 994/4147 0 1314/2048 0 5/ 128 10:56:52 450/ 500 0 994/4147 0 1314/2048 0 5/ 128 proc-sz 目前在核心中正在使用或分配的进程表的表项数 inod-sz 目前在核心中正在使用或分配的i节点表的表项数 file-sz 目前在核心中正在使用或分配的文件表的表项数 ov 溢出出现的次数 lock-sz 目前在核心中正在使用或分配的记录加锁的表项数 ＊除ov外，均涉及到unix的核心参数，它们分别受核心参数NPROC,NIMODE,NFILE和FLOCKREC 的控制。 ＊显示格式为： 实际使用表项/整个表可以使用的表项数 比如，proc-sz一列所显示的四个数字中，分母的 100 是系统中整个进程表的长度（可建立 100 个 表项），分子上的 24 ， 26 和 25 分别是采样的那一段时间所使用的进程表项。inod -sz，file-sz和lock-sz 三列数字的意义也相同。 三列ov的值分别对应进程表，i节点表和文件表，表明目前这三个表都没有出现溢出现象，当出 现溢出时，需要调整相应的核心参数，将对应表加大。 -w 系统交换活动 sar -w 2 3 SCO_SV scosvr 3.2v5.0.5 PentII(D)ISA 06/14/2002 11:22:05 swpin/s bswin/s swpot/s bswots pswch/s (-w) 11:22:07 0.00 0.0 0.00 0.0 330 11:22:09 0.00 0.0 0.00 0.0 892 11:22:11 0.00 0.0 0.00 0.0 1053 Average 0.00 0.0 0.00 0.0 757 swpin/s 每秒从硬盘交换区传送进入内存的次数。 bswin/s 每秒为换入而传送的块数。 swpot/s 每秒从内存传送到硬盘交换区的次数。 bswots 每秒为换出而传送的块数。 pswch/s 每秒进程交换的数量。 ＊swpin/s，bswin/s，swpot/s和bswots描述的是与硬盘交换区相关的交换活动。交换关系到系 统的效率。交换区在硬盘上对硬盘的读，写操作比内存读，写慢得多，因此，为了提高系统效率就应该 设法减少交换。通常的作法就是加大内存，使交换区中进行的交换活动为零，或接近为零。如果swpot/s 的值大于 1 ，预示可能需要增加内存或减少缓冲区（减少缓冲区能够释放一部分自由内存空间）。 #### 8 、 proc 文件系统 "proc文件系统是一个伪文件系统，它只存在内存当中，而不占用外存空间。它以文件系统的方式为 访问系统内核数据的操作提供接口。用户和应用程序可以通过proc得到系统的信息，并可以改变内核 的某些参数。" 这里将介绍如何从/proc文件系统中获取与防火墙相关的一些性能参数，以及如何通过/proc文件系统 修改内核的相关配置。 1 、从/proc文件系统获取相关的性能参数 cpu使用率：/proc/stat 内存使用情况： /proc/meminfo 网络负载信息：/proc/net/dev 相应的计算方法：(摘自：什么是proc文件系统) （ 1 ） 处理器使用率 （ 2 ） 内存使用率 （ 3 ） 流入流出数据包 （ 4 ） 整体网络负载 这些数据分别要从/proc/stat、/proc/net/dev、/proc/meminfo三个文件中提取。如里有问题或对要 提取的数据不太清楚，可以使用man proc来查看proc文件系统的联机手册。 （ 1 ） 处理器使用率 这里要从/proc/stat中提取四个数据：用户模式（user）、低优先级的用户模式（nice）、内 核 模 式（system） 以及空闲的处理器时间（idle）。它们均位于/proc/stat文件的第一行。CPU的利用率使用如下公式来 计算。 CPU利用率 = 100 *（user + nice + system）/（user + nice + system + idle） ##### （ 2 ） 内存使用率 这里需要从/proc/meminfo文件中提取两个数据，当前内存的使用量(cmem)以及内存总量(amem)。 内存使用百分比 = 100 * (cmem / umem) （ 3 ）网络利用率 为了得到网络利用率的相关数据，需要从/proc/net/dev文件中获得两个数据：从本机输出的数据包数， 流入本机的数据包数。它们都位于这个文件的第四行。 性能收集程序开始记录下这两个数据的初始值，以后每次获得这个值后均减去这个初始值即为从集群启 动开始从本节点通过的数据包。 利用上述数据计算出网络的平均负载，方法如下： 平均网络负载 = (输出的数据包+流入的数据包) / 2 2. 通过/proc文件系统调整相关的内核配置 允许ip转发 /proc/sys/net/ipv4/ip_forward 禁止ping/proc/sys/net/ipv4/icmp_echo_ignore_all 可以在命令行下直接往上述两个“文件”里头写入"1"来实现相关配置，如果写入"0"将取消相关配置。 不过在系统重启以后，这些配置将恢复默认设置，所以，如果想让这些修改生效，可以把下面的配置直 接写入/etc/profile文件，或者其他随系统启动而执行的程序文件中。 1.echo 1 > /proc/sys/net/ipv4/ip_forward 2.echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all 如果需要获取其他的性能参数，或者需要对内核进行更多的配置，可以参考proc文件系统介绍，也可 以直接通过man proc查看相关的信息。 ## 十二、 系统服务 #### 1 、系统服务分类，根据其使用的方法来分，可以被分为三类 a、由init控制的服务：基本都是系统级别的服务，运行级别这一章讲的就是这一类的服务 b、由System V启动脚本启动的服务：和我们打交道最多的一种服务，服务器基本都是这个类型的服务 c、由xinetd管理的服务 #### 2 、 System V 启动脚本启动的服务 ``` /etc/rc.d/init.d/目录下的内容如下：这些常用的服务器都是System v的服务，要控制System V 的服务，我们可以使用以下命令 #/etc/rc.d/init.d/script {start|stop|restart|reload|condrestart|status} stop：停止这个服务。 restart：先停止，再启动，也就是重新启动的意思。 reload：重新加载设定档，这个参数只有在服务已经启动的状况下才能使用。 condrestart：有条件的重新启动，这个服务必须是已经启动的，才会被重新启 动；如果这个服务尚未启动，则无须启动之。 status：察看目前服务的启动状态。 也可以使用service命令来执行脚本，例如 #service network {start|stop|restart|reload|condrestart|status} ``` System V的服务在不同级别下的默认开关可以不相同。我们还可以用两种方法来控制默认情况下， 开机是否开启某些服务,使用chkconfig和ntsysv（图形方式，默认只能定义当前级别，不过可以增加 参数来实现如# ntsysv –level 23 ） 来控制。 #chkconfig --list //查看系统 system v服务所有级别下的开关情况。 #chkconfig sshd on|off //更改 sshd服务2- 5级别的默认开关情况 #chkconfig sendmail off //所有级别关闭 sendmail服务 #chkconfig --level 23 sendmail off //在 2 、 3 级别关闭sendmail服务 #### 3 、 xinetd 服务管理 ``` xinetd服务的管理文件都放在 /etc/xinetd.d目录内，我们可以编辑这个目录内的服务文件来 开启和关闭服务。每个服务文件都有disable 这个行，如果把值改成yes就是禁用服务，如果是no， 那就是启动这个服务。修改成功后，要使修改生效，需要从新启动xinetd服务。 #service xinetd restart ``` ``` # vim chargen-dgram ``` #### 4 、常见服务列表 ##### 服务名 必需（是/否）用途描述 注解 ``` acon 否 语言支持 特别支持左手书写语言:阿拉伯语,波斯语和希伯莱 语 acpi 否 电源管理 手提电脑电池电扇监控器 acpid 否 监听精灵进程 此进程监听并分配内核中的acpi事件 ``` adsl 否 内部ADSL开关控制 只有你的计算机内部有互联网连接adsl 开关时才用到此服务 alsa 否 高级Linux声音构件 这个单独的声音系统实际包含在内核中 anacron 否 周期命令调度程序 一个任务调度工具 apmd 否 电源管理 手提电脑电源管理 apmiser 否 电源管理 另一手提电脑电池延长器 arpwatch 否 以太网IP地址配对监控器 用主机名监控并记录远程IP地址 atd 否 周期命令调度程序 一个任务调度工具 autofs 否 自动安装服务 几个命令服务文件系统自动安装之一.一些此 类服务专门针对发行配套软件,如果你使用的发行配套软件拥有自己的自动安装系统,不要用这一个. bluetooth 否 蓝牙技术核心 用于所有蓝牙服务 bootparamd 否 导入服务 以前导入无盘客户端/瘦客户端的方法.最新型的 方法为零配置系统(zeroconf system). canna 否 日语转换引擎 capi4linux 否 基本CAPI子系统 cpqarrayd 否 硬件服务 康柏独立冗余磁盘阵列(Raid Array)监控器 cpufreq 否 硬件服务 控查并配置CPU频率精灵程序模块 cpufreqd 否 硬件服务 此服务自 动衡量CPU频率来减少过热情况.在超频 时有用. crond 是 周期命令调度程序 一个任务调度工具 Cups-lpd 否 使旧式Lunux或商业Unix系统连接到打印主机上. 只有在允 许旧式系统访问打印机时才有用 cups 是 公共Unix打印系统 进行打印的必要功能 cvs 否 并发版本系统 用于管理多用户文档 devfsd 否 系统维护 此服务只清除动态桌面目录,除非你的系统经常崩 溃,否则不需要此服务. dhcpd 否 DHCP服务器 你的网络足够大,使用静态IP很麻烦吗?此项服 务对你的网络进行DHCP IP配置,方便网络应用. diald 否 拨号网络智能自动拨号器 此服务一经请求,即连接上网络. 你一旦输入电子邮件,点击发送,它就自动连接,发送电邮并断开. dkms 否 DKMS自安装导入 发行配套软件专用工具,用于OEM类型安装. 它允许管理员密码的最初导入设置以及常规应用的用户名密码,系统的最后配置. dm 是 显示管理器 X服务器的核心,使用图形用户界面(GUI)时必 需. dnbc 否 数字网络绑定Chrooter 这是一个简单的bash脚本,它将一 个BIND服务器放入一个chroot牢笼中.安装BIND,发布脚本并重启. Drakxtools-http 否 小型服务管理服务器 远程系统管理的发行配套软件专用工 具. dund 否 蓝牙拨号网络 fam 否 文件系统变更监控器 文件系统所有改变的记录器 finger 否 数据远程访问 此服务允许你远程访问用户登录日期,最后登 录日期与时间.用于不在办公室时监控雇员的工作习惯,主要的安全违反,因为你正有效地在线发布公 司机密数据. freshclam 是 ClamAV更新器 用于自动更新ClamAV gpm 是 鼠标 鼠标驱动器控制台模式 haldaemon 否 硬件监控系统 此服务监控硬件改变,为你改变新的或更改过 的硬件. harddrake 否 硬件服务 发行配套软件专用硬件探测与配置 heartbeat 否 高可用性服务 此服务旨在增加重要服务与服务器的优先级 hidd 否 蓝牙H.I.D.服务器 hplip 否 惠普Linux打印与成像 旧版惠普整成产品供应驱动器 hpoj 否 Pital?init,惠普办公喷墨打印机驱动器 惠普办公喷墨打印 机旧式驱动器.新式驱动器包含在打印机的打印驱动器内. httpd 否 Apache网络服务器 在系统上应用此服务有两个原因,一是 要用它作为网络服务器,二是用它作为网址开发器.如果没有此二项,则不必安装 Apache. hylafax?server 否 企业传真机?调制调解器服务 此服务仅用于 1 类与 2 类传真 机.如果你想用hylafax通过调制调解器发送传真,必须运行此服务.它并不是唯一有效的传真工具. ibod 否 按需ISDN MPPP带宽 与拨号网络一同使用,按需连接到网络. identd 否 TCP连接鉴定 imaps 否 安全IMAP服务器 imaps 否 IMAP服务器 iplog 否 用主机名或远程主机记录TCP,UDP,ICMP. 有用的网络监控 工具 ipop2 否 POP2邮件服务器 ipop3 否 POP3邮件服务器 ipsec 否 加密与验证通信 KLIPS为内核一半,PLUTO为用户空间一半. 在远程访问情况下十分有用. iptables 是 基于Packet过滤防火墙内核 所有优秀的Linux防火墙都基 于此项服务 ipvsadmin 否 Linux核心IP虚拟服务器 最早的Linux网络系统之一,已不 常用. irda 否 红外线设备界面 以前的无线设备支持 keytable 是 键盘映射 此服务明确告诉系统你正在使用哪种键盘 kheader 否 导入服务 此服务自动重建内核头导入 lads 否 登录异常探测系统 跟踪登录企图并警告入侵企图的工具 laptop mode 否 电源管理 减少电力耗费,延长手提电脑电池寿命的工具 leafnode 否 X? INETD NNTP服务 lisa 否 局域网信息服务器 lmsensors 否 硬件健康监控器 此服务要求系统主板支持并有合适的监控 系统,如CPU温度与电压监控器. mailman 否 GUN邮件列表管理器 常用的邮件列表工具,带Python编写的 管理网络界面.它允许列表成员发送邮件并回复邮件到同一个地址进行交流. 它还可用于向那些发送 请求的用户传送新闻时讯/产品更新. mandi 否 交互式防火墙 允许暂时无线访问系统的专用服务,将为当前 任务开放iptables防火墙,仅用于无线设备访问.在用户许可情况下才可使用,不能自动使用. mdadm 否 软阵列监控服务 这也是一个用于上述软件阵列栏的管理工 具 mdnsresponder 否 零配置DNS配置工具 messagebus 是 事件监控服务 此服务在必要时向所有用户发送广播信息,如 服务器将要重启. mon 是 系统监控精灵进程 许多系统服务要求此服务来运行 mysqld 否 MySQL服务器 如果你不需要这个数据库,不要打开它. named 否 绑定(BIND)服务器 这就是声名狼藉的名称服务器 netplugd 否 网卡精灵进程 此服务监控网络界面,根据信号关闭或启动它, 主要用于不经常连接的手提电脑. network 是 网络 此服务打开网卡,或为调制调解器供电. nfs 否 网络文件共享 此服务使用户访问NFS共享文件,为NFS系统 客户机所必需. nfsfs 否 网络文件共享服务器 只有在网络服务器上才被激活 nfslock 否 NFS文件锁定 只有在使用NFS网络/文件共享功能时,此服务 才被激活. nifd 否 Howl 客户端 此服务为零配置网络/系统提供ipv4链接本地 服务 nscd 否 密码与群查找服务 此服务用于减慢N.I.S/Y.P.nist,ldap 和hesiod之类的服务.专门为这些服务提供更长的中断时间. ntpd 否 NTP服务器的第 4 版 numlock 否 数字锁定键灯光控制 此服务保持数字锁定键的激活状态,打 开键盘上的数字键区. Oki4daemon 否 OKI4和兼容win打印机的兼容性精灵进程 只有在你有这些 打印机时才可用 pand 否 蓝牙个人区域网络 用于基于网络的家庭区域蓝牙技术 partmon 是 分区监控 此服务跟踪安装分区上的剩余空间.大多数文件系 统浏览器使用它来计算指定分区上的剩余空间. pcmcia 否 个人电脑内存卡国际协会 pg_autovacum 否 PostgreSQL维护 此服务自动运行PostgreSQL所需的空间 (vacuum)来减少磁盘空间,从数据库中拖动临时表格,并删除 PostgreSQL建立的临时文件. pop3s 否 安全POP3服务 POP3 SSL服务器 portmap 否 RPC支持 支持那些应用rpc的罕有的应用软件 postgresql 否 Postgresql数据库引擎 只有在运行或开发Postgresql数据 库驱动应用软件时才用到此服务 postfix 否 电子邮件服务器 与sendmail兼容的电子邮件服务器,比 sendmail更新,也变得比sendmail更通用. pptp 否 PPP断电服务 PPP频道断电服务,UPS打开时使用,以避免电 源返回系统时出现文件锁定问题. prelude 否 IDS 入侵探测系统 psacct 否 进程计算 活动进程追踪器,实际上是资源的浪费. rawdevices 是 分配raw设备,阻止其使用 DVDS,oracle DBMS等需要此服务 rsync 否 远程同步 使指定目录树上的文件远程同步的服务器,常用于 维护镜象地址,也在备份时用于保持公司文件为最新状态. saned 否 网络扫描仪 从网络上的任何工作站提供扫描仪访问 shorwall 是 防火墙 一个非常优秀的IPTables防火墙 smartd 否 自我监控服务 用于智能设备的OS访问,此服务允许Linux 告诉你是否设备将要变坏,但这要依靠设备的精确智能特性. smb 否 Samba网络服务 此服务提供samba服务,实现Windows网络 兼容性. snmpd 否 简单的网络管理协议 用于小型(家庭办公室)网络 sound 否 声音系统 此为Linux声音系统的核心,适用于桌面系统,在 服务器上则是资源的浪费. squid 否 高速缓存工具 用于高速缓存网络页面及DNS登录 ssh?xinetd 否 X?inetd OpenSSH服务器 OpenSSH的按需运行版本 sshd 否 OpenSSH服务器 如果你需要SSH访问你的系统时才开启此服 务,将不会使用x?inetd版本. subversion 否 并发版本系统 CVS的新型替代品 swat 否 Samba网络管理工具 基于Samba管理的网络 syslog 是 系统登录 一项必要的服务,控制整个系统上的所有登录. tmdns 否 多点传送DNS响应器 用于零配置环境 ultrabayed 否 ThinkPad工具 此服务为你的IBM ThindPad探测ultrabay, 并在适当情况下启动/关闭IDE接口. upsd 否 NUT精灵进程及驱动器 一个不间断地电源监控及报告工具, 此服务向中心地址报告,产生关于UPS统计的数据库. upsmon 否 UPS监控工具 此服务监控UPS的状况,在其运行低下时关系 系统. vncserver 否 虚拟网络计算服务器 在项目中应用VNC时非常有用 Webmin 否 远程管理工具 发行配套软件Agnostic远程管理工具.在机 器不能总是直接访问,如网络服务器集群时有用. winbind 否 Samba名称服务器 Samba网络运行所必需.此服务将用户与 群数据从windows网络映射到Linux工作站中. wine 否 Wine并非竞争者 此服务使MS Windows可在Linux上执 行,WINE是商业产品Crossover Office的限制版本. wlan 否 控制精灵进程 由于服务通常由init进程控制,此控制服务 不常用. xinetd 是 监控并控制其它服务器的服务器 这是一项必需的服务,它实 际上减少了服务器上CPU的负载.如果你需要SSH,ftp等但并不总是需要,x?inetd版本将在请求,甚 至是远程需求时启 动它们.此服务让它们生效,但如果它们一天/周只使用几次的话,又释放了时钟周 期. xfs 是 X字体服务器 你任何时间需要使用图形用户界面(GUI),就 需要此服务. ypbind 否 SUN的YP服务器名称服务器 此服务用于基于GLIBC的 NIS/YP网络服务 ## 十三、 环境管理 #### 1 、环境变量 在linux系统下，如果你下载并安装了应用程序，很有可能在键入它的名称时出现“command not found”的提示内容。如果每次都到安装目标文件夹内，找到可执行文件来进行操作就太繁琐了。这涉 及到环境变量PATH的设置问题，而PATH的设置也是在linux下定制环境变量的一个组成部分。 环境变量可以让子程序继续引用的原因，是因为： 1. 当启动一个 shell ，操作系统分配一记忆区块给 shell 使用，此区域之变量可以让子程序存 取； 2. 利用 export 功能，可以让变量的内容写到上述的记忆区块当中(环境变量)； 当加载另一个 shell 时 (亦即启动子程序，而离开原本的父程序了)，子 shell 3. 可以将父 shell 的环境变量所在的记忆区块导入自己的环境变量区块当中。 所以环境变量是和Shell紧密相关的，用户登录系统后就启动了一个Shell。对于Linux来说一般 是bash，但也可以重新设定或切换到其它的Shell（使用chsh命 令 ）。 根据发行版本的情况，bash有两个基本的系统级配置文件：/etc/bashrc和/etc/profile。这些配 置文件包含两组不同的变量：shell变量和环境变量。前者只是在特定的shell中固定（如bash）， 后 者在不同shell中固定。很明显，shell变量是局部的，而环境变量是全局的。环境变量是通过Shell 命令来设置的，设置好的环境变量又可以被所有当前用户所运行的程序所使用。对于bash这个Shell 程序来说，可以通过变量名来访问相应的环境变量，通过export来设置环境变量。 注：Linux的环境变量名称一般使用大写字母 #### 1.1 、 Linux 环境变量的种类 按环境变量的生存周期来划分，Linux的环境变量可分为两类： ○^1 永久的：需要修改配置文件，变量永久生效。^ ○^2 临时的：使用export命令行声明即可，变量在关闭shell时失效。^ #### 1.2 、设置变量的三种方法 (^) ○ 1 在/etc/profile文件中添加变量对所有用户生效（永久的） 用VI在文件/etc/profile文件中增加变量，该变量将会对Linux下所有用户有效，并且是“永久 生效”。 例如：编辑/etc/profile文件，添加CLASSPATH变量 # vi /etc/profile export CLASSPATH=./JAVA_HOME/lib;$JAVA_HOME/jre/lib 注 1 ：profile文件在系统启动时将被运行。大家可以在里面加入其他命令，但是一定要加正确，不然 的话系统会启动不起来的。 ○^2 在用户目录下的.bash_profile文件中增加变量对单一用户生效（永久的）^ 用VI在用户目录下的.bash_profile文件中增加变量，改变量仅会对当前用户有效，并且是“永 久的”。 例如：编辑guok用户目录（/home/guok）下的.bash_profile # vi /home/guok/.bash.profile ##### 添加如下内容： export CLASSPATH=./JAVA_HOME/lib;$JAVA_HOME/jre/lib 注 2 ：如果修改了/etc/profile，那么编辑结束后执行source profile 或 执行点命令 ./profile,PATH 的值就会立即生效了。这个方法的原理就是再执行一次/etc/profile shell脚本，注意如果用sh /etc/profile是不行的，因为sh是在子shell进程中执行的，即使PATH改变了也不会反应到当前环 境中，但是source是在当前 shell进程中执行的，所以我们能看到PATH的改变。 注 3 ：变量重复定义时，以后面的设置为先。 例如：在peofile文件默认对PATH变量都有设置PATH=￥￥￥￥￥￥￥，在以后可能在对PATH设 置，一般都加在profile文件的最后PATH=······（打个比方）。而系统之中认定的 PATH=·······￥￥￥￥￥￥￥￥￥，也就是说相同名字的环境变量，后写入的先起作用（通俗地讲）。 注 4 、特殊字符介绍。 例如在profile中有如下内容，通过以下内容说明特殊符号的用法。 export A=/q/jing:aaa/cc/ld export B=.:/liheng/wang export A=/cd/cdr:$A : 表示并列含义，例如A变量值有多个，用：符号进行分离。 . 表示你操作的当前目录。例如pap命令会查找B环境变量。 在/home键入B命令，系统首先在/home目录下（即当前路径）查找关于 B 的内容，如果没有在 /liheng/wang目录下查找关于B的内容。 $ 表示该变量本次定义之前的值，例如$A代表 /q/jing:aaa/cc/ld。也就是说 A=/cd/cdr:/q/jing:aaa/cc/ld 注 5 、常见的环境变量 PATH：决定了shell将到哪些目录中寻找命令或程序 HOME：当前用户主目录 MAIL：是指当前用户的邮件存放目录。 SHELL：是指当前用户用的是哪种Shell。 HISTSIZE：是指保存历史命令记录的条数。 LOGNAME：是指当前用户的登录名。 HOSTNAME：是指主机的名称，许多应用程序如果要用到主机名的话，通常是从这个环境变量中来取得的。 LANG/LANGUGE：是和语言相关的环境变量，使用多种语言的用户可以修改此环境变量。 PS1：是基本提示符，对于root用户是#，对于普通用户是$。 PS2：是附属提示符，默认是“>”。可以通过修改此环境变量来修改当前的命令符，比如下列命令 会将提示符修改成字符串“Hello,My NewPrompt :) ”。 # PS1=" Hello,My NewPrompt :) " ``` ○^3 直接运行export命令定义变量【只对当前shell（BASH）有效（临时的）】^ ``` 在shell的命令行下直接使用[export变量名=变量值]定义变量，该变量只在当前的shell（BASH）或 其子shell（BASH）下是有效的，shell关闭了，变量也就失效了，再打开新shell时就没有这个变量， 需要使用的话还需要重新定义。 #### 1.3 、环境变量设置命令 (^) ○1 echo $ <变量名> //显示 某个环境变量 ○2 env // environment (环境 ) 的简写，列出来所有的环境变量^ ``` ○3 set //显示所有本地定义的 Shell变量，这个命令除了会将环境变量列出来之外，其它我们 ``` 的自定义的变量，都会被列出来。因此，想要观察目前 shell 环境下的所有变量，就用 set 即可！ ``` ○4 export 命令^ 功能说明：设置或显示环境变量。 语 法：export [-fnp][变量名称]=[变量设置值] 补充说明：在shell中执行程序时，shell会提供一组环境变量。export可新增，修改或删除环 境变量，供后续执行的程序使用。export的效力仅及于该此登陆操作。 参 数： -f 代表[变量名称]中为函数名称。 -n 删除指定的变量。变量实际上并未删除，只是不会输出到后续指令的执行环境中。 -p 列出所有的shell赋予程序的环境变量。 一个变量创建时，它不会自动 地为在它之后创建的shell进程所知。而命令export可以向后面 的shell传递变量的值。当一个shell脚本调用并执行时，它不会自动得到原为脚本（调用者） 里定义的变量的访问权，除非这些变量已经被显式地设置为可用。export命令可以用于传递一 个或多个变量的值到任何后继脚本。 ``` (^) ○5 unset清除环境变量，如果未指定值，则该变量值将被设为NULL (^) ○6 readonly设置只读变量，只读变量设置后不能用unset清除，除非重启shell (^) ○7 declare 、typeset 这两个命令是完全一样的，他们允许指定变量的具体类型，在某些特定的语言 中, 这是一种指定类型的很弱的形式，declare 命令是在 Bash 版本 2 或之后的版本才被加入的，typeset 命令也可以工作在 ksh 脚本中。 [root@linux ~]# declare [-aixr] variable 参数： -a ：将后面的 variable 定义成为数组 (array) -i ：将后面接的 variable 定义成为整数数字 (integer) -x ：用法与 export 一样，就是将后面的 variable 变成环境变量； -r ：将一个 variable 的变量设定成为 readonly ，该变量不可被更改内容，也不能 unset 范例一：让变量 sum 进行 100+300+50 的加总结果 [root@linux ~]# sum=100+300+50 [root@linux ~]# echo $sum 100+300+50 <==咦！怎么没有帮我计算加总？因为这是文字型态的变量属性啊！ [root@linux ~]# declare -i sum=100+300+50 [root@linux ~]# echo $sum 450 范例二：将 sum 变成环境变量 [root@linux ~]# declare -x sum 范例三：让 sum 变成只读属性，不可更动！ [root@linux ~]# declare -r sum [root@linux ~]# sum=tesgting -bash: sum: readonly variable ## 十四、 网络管理 网卡在Linux操作系统中用 ethX,是由 0 开始的正整数，比如eth0、eth1...... ethX。而普通猫和 ADSL的接口是 pppX，比如ppp0等 #### 7.1 、 ifconfig ##### 1 、 关于网络接口及配置工具说明； ``` 在Linux操作系统中配置网络接口，一般是通过网络配置工具实现的，但最终目的还是通过网络配 ``` 置工具来达到修改与网络相关的配置文件而起作用的。由此说来，我们配置网络可以直接修改配置文件。 ``` 比如网络网络接口（网卡）的IP地址、子掩码、网关，在Slackware 中只需修改一个配置文件就 ``` 行了 /etc/rc.d/rc.inet1 ，而在 Redhat/Fedora 等或以Redhat/Fedora 为基础的发行版中，一般要 涉及到好几个文件，比如包括 /etc/sysconfig/network-scripts/ifcfg-eth0在内等。 ``` 了解Linux网络配置文件是极为重要的，我们通过工具修改了什么，是怎么生效的，只有了解网络 ``` 配置文件才能搞清楚。做个不恰当的比喻：Linux 系统是一个透明的盒子，至于盒子里装的是什么都是 一目了然的。而闭源操作系统，我们没有机会知道这些，更不知道他是怎么实现的。 ``` 对于复杂的网络模型，Linux操作系统是有极大的优势，可能在我们看看man 和help，修改修改 ``` 配置文件，在几分钟就可以搞定。但闭源图形界面的操作系统就没有这么幸运了，反复的点鼠标. ``` 点了几十次上百次也解决不了一个问题，这是极为常见的。 由于Linux操作系统存在很多的发行 ``` 和版本，大多发行版本都有自己的专用配置工具。主要是为了方便用户配置网络；但也有通用的配置工 具，比如 Linux ifconfig 、ifup、ifdown； ``` 2 关于网络硬件设备的驱动； ``` ``` 我在以前的文档中有写过，网络硬件，比如网卡（包括有线、无线），猫包括普通猫和ADSL猫等， ``` 都是需要内核支持的，所以我们首先得知道自己的网络设备是不是已经被硬内核支持了。如果不支持， 我们得找驱动（或通过内核编译）来支持它； 请参考： ``` 3 、Linux ifconfig 配置网络接口的工具介绍； ``` ``` Linux ifconfig 是一个用来查看、配置、启用或禁用网络接口的工具，这个工具极为常用的。比 ``` 如我们可以用这个工具来临时性的配置网卡的IP地址、掩码、广播地址、网关等。也可以把它写入一 个文件中（比如/etc/rc.d/rc.local)，这样系统引导后，会读取这个文件，为网卡设置IP地址； ``` 不过这样做目前看来没有太大的必要。主要是各个发行版本都有自己的配置工具，无论如何也能把 ``` 主机加入到网络中； 下面我们看看Linux ifconfig 用法； ``` 3.1 Linux ifconfig 查看网络接口状态； ``` ``` Linux ifconfig 如果不接任何参数，就会输出当前网络接口的情况； ``` 1. [root@localhost ~]# Linux ifconfig 2. eth0 Link encap:Ethernet HWaddr 00:C0:9F:94:78:0E 3. inet addr:192.168.1.88 Bcast:192.168.1.255 Mask:255.255.255.0 4. inet6 addr: fe80::2c0:9fff:fe94:780e/64 Scope:Link 5. UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 6. RX packets:850 errors:0 dropped:0 overruns:0 frame:0 7. TX packets:628 errors:0 dropped:0 overruns:0 carrier:0 8. collisions:0 txqueuelen:1000 9. RX bytes:369135 (360.4 KiB) TX bytes:75945 (74.1 KiB) 10. Interrupt:10 Base address:0x3000 11. 12. lo Link encap:Local Loopback 13. inet addr:127.0.0.1 Mask:255.0.0.0 14. inet6 addr: ::1/128 Scope:Host 15. UP LOOPBACK RUNNING MTU:16436 Metric:1 16. RX packets:57 errors:0 dropped:0 overruns:0 frame:0 17. TX packets:57 errors:0 dropped:0 overruns:0 carrier:0 18. collisions:0 txqueuelen:0 19. RX bytes:8121 (7.9 KiB) TX bytes:8121 (7.9 KiB) ``` 解说：eth0 表示第一块网卡， 其中 HWaddr 表示网卡的物理地址，我们可以看到目前这个网卡的 ``` 物理地址(MAC地址）是 00:C0:9F:94:78:0E ； inet addr 用来表示网卡的IP地址，此网卡的 IP地 址是 192.168.1.88， 广播地址， Bcast:192.168.1.255，掩码地址Mask:255.255.255.0 ``` lo 是表示主机的回坏地址，这个一般是用来测试一个网络程序，但又不想让局域网或外网的用户 ``` 能够查看，只能在此台主机上运行和查看所用的网络接口。比如我们把HTTPD服务器的指定到回坏地址， 在浏览器输入 127.0.0.1 就能看到你所架WEB网站了。但只是您能看得到，局域网的其它主机或用户 无从知道； ``` 如果我们想知道主机所有网络接口的情况，请用下面的命令； [root@localhost ~]# Linux ``` ifconfig -a 如果我们想查看某个端口，比如我们想查看eth0 的状态，就可以用下面的方法； [root@localhost ~]# Linux ifconfig eth0 ``` 3.2 Linux ifconfig 配置网络接口； ``` ``` Linux ifconfig 可以用来配置网络接口的IP地址、掩码、网关、物理地址等；值得一说的是用 ``` Linux ifconfig 为网卡指定IP地址，这只是用来调试网络用的，并不会更改系统关于网卡的配置文件。 ##### 如果您想把网络接口的IP地址固定下来，目前有三个方法：一是通过各个发行和版本专用的工具 来修改IP地址；二是直接修改网络接口的配置文件；三是修改特定的文件，加入Linux ifconfig 指 令来指定网卡的IP地址，比如在redhat或Fedora中，把Linux ifconfig 的语名写入 /etc/rc.d/rc.local文件中； ``` Linux ifconfig 配置网络端口的方法： Linux ifconfig 工具配置网络接口的方法是通过指令的 ``` 参数来达到目的的，我们只说最常用的参数； Linux ifconfig 网络端口 IP地址 hw MAC 地址 netmask 掩码地址 broadcast 广播地址 [up/down] ``` 实例一： ``` ``` 比如我们用Linux ifconfig 来调试 eth0网卡的地址 ``` 1. [root@localhost ~]# Linux ifconfig eth0 down 2. [root@localhost ~]# Linux ifconfig eth0 192.168.1.99 broadcast 192.168.1.255 netma ``` sk 255.255.255.0 ``` 3. [root@localhost ~]# Linux ifconfig eth0 up 4. [root@localhost ~]# Linux ifconfig eth0 5. eth0 Link encap:Ethernet HWaddr 00:11:00:00:11:11 6. inet addr:192.168.1.99 Bcast:192.168.1.255 Mask:255.255.255.0 7. UP BROADCAST MULTICAST MTU:1500 Metric:1 8. RX packets:0 errors:0 dropped:0 overruns:0 frame:0 9. TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 10. collisions:0 txqueuelen:1000 11. RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) 12. Interrupt:11 Base address:0x3400 ``` 注解： 上面的例子我们解说一下； ``` ``` 第一行：Linux ifconfig eth0 down 表示如果eth0是激活的，就把它DOWN掉。此命令等同于 ifdown ``` eth0； ``` 第二行：用Linux ifconfig 来配置 eth0的IP地址、广播地址和网络掩码； ``` ``` 第三行：用Linux ifconfig eth0 up 来激活eth0 ； 此命令等同于 ifup eth0 ``` ``` 第四行：用 Linux ifconfig eth0 来查看 eth0的状态； ``` ``` 当然您也可以用直接在指令IP地址、网络掩码、广播地址的同时，激活网卡；要加up参数；比如 ``` 下面的例子； [root@localhost ~]# Linux ifconfig eth0 192.168.1.99 broadcast 192.168.1.255 netmask 255.255.255.0 up ##### 实例二：在这个例子中，我们要学会设置网络IP地址的同时，学会设置网卡的物理地址（MAC地 ##### 址）； ``` 比如我们设置网卡eth1的IP地址、网络掩码、广播地址，物理地址并且激活它； [root@localhost ``` ~]# Linux ifconfig eth1 192.168.1.252 hw ether 00:11:00:00:11:11 netmask 255.255.255.0 broadcast 192.168.1.255 up或[root@localhost ~]# Linux ifconfig eth1 hw ether 00:11:00:00:11:22[root@localhost ~]# Linux ifconfig eth1 192.168.1.252 netmask 255.255.255.0 broadcast 192.168.1.255 up ``` 其中 hw 后面所接的是网络接口类型， ether表示乙太网， 同时也支持 ax25 、ARCnet、netrom ``` 等，详情请查看 man Linux ifconfig ； ``` 3.3 如何用Linux ifconfig 来配置虚拟网络接口； ``` ``` 有时我们为了满足不同的需要还需要配置虚拟网络接口，比如我们用不同的IP地址来架 运行多个 ``` HTTPD服务器，就要用到虚拟地址；这样就省却了同一个IP地址，如果开设两个的HTTPD服务器时， 要指定端口号。 ``` 虚拟网络接口指的是为一个网络接口指定多个IP地址，虚拟接口是这样的 eth0:0 、 eth0:1、 ``` eth0:2 ... .. eth1N。当然您为eth1 指定多个IP地址，也就是 eth1:0、eth1:1、eth1:2 ... ... 以此类推； ``` 其实用Linux ifconfig 为一个网卡配置多个IP地址，就用前面我们所说的Linux ifconfig的用 ``` 法，这个比较简单；看下面的例子； [root@localhost ~]# Linux ifconfig eth1:0 192.168.1.251 hw ether 00:11:00:00:11:33 netmask 255.255.255.0 broadcast 192.168.1.255 up或 [root@localhost ~]# Linux ifconfig eth1 hw ether 00:11:00:00:11:33[root@localhost ~]# Linux ifconfig eth1 192.168.1.251 netmask 255.255.255.0 broadcast 192.168.1.255 up ``` 注意：指定时，要为每个虚拟网卡指定不同的物理地址； ``` ``` 在 Redhat/Fedora 或与Redhat/Fedora类似的系统，您可以把配置网络IP地址、广播地址、掩码 ``` 地址、物理地址以及激活网络接口同时放在一个句子中，写入/etc/rc.d/rc.local中。比如下面的例 子； ``` Linux ifconfig eth1:0 192.168.1.250 hw ether 00: 11:00:00:11:44 netmask 255.255.255.0 ``` broadcast 192.168.1.255 up ``` Linux ifconfig eth1:1 192.168.1.249 hw ether 00:11:00:00:11:55 netmask 255.255.255.0 ``` broadcast 192.168.1.255 up ``` 解说：上面是为eth1的网络接口，设置了两个虚拟接口；每个接口都有自己的物理地址、IP地 ``` 址... ... ``` 3.4 如何用Linux ifconfig 来激活和终止网络接口的连接； ``` ``` 激活和终止网络接口的用 Linux ifconfig 命令，后面接网络接口，然后加上 down或up参数， ``` 就可以禁止或激活相应的网络接口了。当然也可以用专用工具ifup和ifdown 工具； 1. [root@localhost ~]# Linux ifconfig eth0 down 2. [root@localhost ~]# Linux ifconfig eth0 up 3. [root@localhost ~]# ifup eth0 4. [root@localhost ~]# ifdown eth0 ``` 对于激活其它类型的网络接口也是如此，比如 ppp0，wlan0等；不过只是对指定IP的网卡有效。 ``` 注意：对DHCP自动分配的IP，还得由各个发行版自带的网络工具来激活；当然得安装dhcp客户端； 这个您我们应该明白；比如Redhat/Fedora [root@localhost ~]# /etc/init.d/network start Slackware 发行版； [root@localhost ~]# /etc/rc.d/rc.inet1 ``` 4 、Debian、Slackware、Redhat/Fedora、SuSE等发行版专用网络接口配置工具； ``` ``` 由于 Linux ifconfig 用起来比较麻烦，而且是用来测试网络之用，但这个工具并不能修改网络接 ``` 口的相应配置文件。虽然也能通过把Linux ifconfig配置网络接口的语句写入类似/etc/rc.d/rc.local 文件中，但相对来说还是写入关于网络接口的配置文件中更为安全和可靠；但对于虚拟网络接口写入类 似/etc/rc.d/rc.local 中还是可以的； ``` 下面我们介绍一下各个发行版的网络接口配置工具； ``` ``` 4.1 Debian 网络接口配置文件和专用配置工具； ``` ``` 正在增加中； ``` ``` 4.2 Redhat/Fedora 网络接口的配置文件和网络接口专用配置工具； ``` ``` 在Redhat/Fedora 中，与乙太网卡相关的配置文件位于 /etc/sysconfig/network-scripts目录中， ``` 比如 ifcfg-eth0、ifcfg-eth1 .... .... ``` 4.21 Redhat/Fedora 或类似这样的系统，网卡的配置文件； ``` ``` ``` ``` 比如在Fedora 5.0中，ifcfg-eth 0 ； 如果您用DHCP服务器来自动获取IP的，一般情况下 ``` ifcfg-eth0的内容是类似下面这样的； 1. DEVICE=eth0 2. ONBOOT=yes 3. BOOTPROTO=dhcp 4. TYPE=Ethernet ##### 如果您是指定IP的，一般内容是类似下面的； 1. DEVICE=eth0 注：网络接口 2. ONBOOT=yes 注：开机引导时激活 3. BOOTPROTO=static 注：采用静态IP地址； 4. IPADDR=192 .168.1.238 注：IP地址 5. NETMASK=255.255.255.0 注：网络掩码； 6. GATEWAY=192.168.1.1 注：网关； ``` 下面的几个选项也可以利用； ``` 1. HOSTNAME=linxsir03 注：指定主机名； 2. DOMAIN=localdomain 注：指定域名； 3. HWADDR=00:00:11:22:00:aa 注： ``` 指定网卡硬件地址 (MAC地址）， 也可以省略，不过这在这里来更改MAC地址一般是不能生效的。 ``` 还是通过前面所说的Linux ifconfig的办法来更改吧； ``` 4.22 Redhat/Fedora 或类似系统， 配置网络的工具介绍 ； ``` ``` 在Redhat早期的版本中， 有linuxconf 、redhat-config-network 、netconfig 等工具； 在 ``` Redhat/Fedora 最新的版本有 system-config-network-tui （文本模式的) 、system-config-network （图形模式的），netconfig（文本模式的）。 ``` 这些工具都会直接修改Linux系统中关于网络接口的配置文件；这是 Linux ifconfig 所不能比的； ``` 其中 redhat-config-network 和system-config-network工具不仅仅是配置网卡的工具，还有配置 ISDN和普通猫、ADSL的工具、网络硬件的添加、主机名字的配置、DNS各客户端的配置等。其实是一 个工具组的集成； ``` 这些工具比较简单，以root权限运行命令就能调用，比如： ``` 1. [root@localhost ~]# /usr/sbin/system-config-network 2. [root@localhost ~]# system-config-network ``` 如果您设置了可执行命令的环境变量，不用加路径就可以运行，但前提是您得安装这个网络管理工 ``` 具； 不过值得一说的是netconfig 工具是一个在文本模式比较好的工具，推荐大家使用；理由是这个 工具在文本模式下，也有一个简单的图形界面；还有命令模式；功能强着呢； 1. [root@localhost ~]# netconfig -d eth0 注：配置eth0 2. [root@localhost ~]# netconfig -d eth1 注：配置eth1 ``` 4.23 Redhat/Fedora系统中的netconfig 特别介绍； ``` ``` netconfig这个工具，在Redhat/Fedora 或类似于它们的系统中都是存在的，这个工具比较强大。 ``` 所以特别介绍一下。但在Slackware中netconfig是TEXT模式下有一个图形模式，但不能象Linux ifconfig一样用命令来操作网卡接口； ``` netconfig 的用法如下： ``` 1. [root@localhost ~]# netconfig --help 注：帮助； 2. --bootproto=(dhcp|bootp|none) Boot protocol to use（ 3. --gateway=STRING Network gateway（指定网关） 4. --ip=STRING IP address（指定IP地址） 5. --nameserver=STRING Nameserver（指定DNS客户端） 6. --netmask=STRING Netmask（指定网络掩码） 7. --hostname=STRING Hostname（ 指定主机名） 8. --domain=STRING Domain name（指定域名） 9. -d, --device=STRING Network device （指定网络设备） 10. --nodns No DNS lookups （没有DNS查询） 11. --hwaddr=STRING Ethernet hardware address （指定网卡的物理地址） 12. --description=STRING Description of the device （描述性文字） 13. Help options: (帮助选项） 14. -?, --help Show this help message 15. --usage Display brief usage message ``` 实例一：设置网卡的DHCP模式自动获得IP [root@localhost ~]# netconfig -d eth0 ``` --bootproto=dhcp ``` 实例一：手动设置网卡的IP等 [root@localhost ~]# netconfig -d eth0 --ip=192.168.1.33 ``` --netmask=255.255.255.0 --gateway=192.168.1.1 ``` 4.3 Slackware网卡配置文件和配置工具； ``` ``` Slackware 有关网卡的配置文件是/etc/rc.d/rc.inet1.conf ， 这个文件包括乙太网接口的网卡 ``` 和无线网卡的配置。Slackware 还是比较纯净的，网络配置也较简单；在Slackware中也有netconfig 配置工具，也是text模式运行的，人机交互界面，这个设置比较简单； ``` Slackware 用netconfig 配置网卡完成后，其实质是修改了/etc/rc.d/rc.inet1.conf 文件。 ``` Slackware是源法原味的Linux系统，他的配置文件比较标准 ，所以我推荐您在生产型的系统，不妨 尝试一下Slackware ； ``` 配置好网卡后，我们还得运行下面的命令，才能激活网卡； [root@localhost ~]# ``` /etc/rc.d/rc.inet1 下面是一个例子，比如此机器有两个网卡eth0和eth1，eth0用DHCP获得IP地 址，eth1指定IP地址； 1. # Config information for eth0: 2. IPADDR[0]="" 3. NETMASK[0]="" 4. USE_DHCP[0]="yes" 注：在这里写上yes，表示用DHCP获得IP； 5. DHCP_HOSTNAME[0]="linuxsir01" 注：DNS服务器主机名，也可以用IP来指定DNS服务器； 6. # Config information for eth1: 注：网卡eth1的配置； 7. IPADDR[1]="192.168.1.33" 注：指定IP地址； 8. NETMASK[1]="255.255.255.0" 注：指定掩码； 9. USE_DHCP[1]="no" 注：不用DHCP获得IP； 10. DHCP_HOSTNAME[1]="" 11. # Config information for eth2: 12. IPADDR[2]="" 13. NETMASK[2]="" 14. USE_DHCP[2]="" 15. DHCP_HOSTNAME[2]="" 16. # Config information for eth3: 17. IPADDR[3]="" 18. NETMASK[3]="" 19. USE_DHCP[3]="" 20. DHCP_HOSTNAME[3]="" 21. # Default gateway IP address: 22. GATEWAY="192.168.1.1" 注：指定网关； ``` 4.4 SuSE或OpenSuSE网卡配置文件和配置工具； ``` ``` 正在更新之中； ``` ``` 5 、关于拔号工具的介绍； ``` ``` 有的弟兄可能需要ADSL猫和普通猫的拔号工具；现在我们分别介绍一下； ``` ``` 5.1 ADSL pppoe 拔号工具rp-pppoe； ``` ``` 如果您的ADSL不是路由的，如果是路由的，在路由路就能设置好自动拔号。只要把机器接上就能 ``` 用了，这个咱们不说了，路由器大家都会用；但如果您的ADSL不支持路由，或您想用您当前所用的主 机来做路由器；这就需要一个拔号软件； ``` 目前国内大多城市都用的是pppoe协议 ，所以我们有必要介绍 pppoe 拔号软件，在Linux中，这 ``` 个软件的名字是 rp- pppoe ； rp-pppoe 主页； [http://www.roaringpenguin.com/penguin/open_source_rp-pppoe.php](http://www.roaringpenguin.com/penguin/open_source_rp-pppoe.php) ``` 5.11 各大发行版自带的 rp-pppoe 的安装和使用； ``` ``` rp-pppoe 目前在各大发行版本都是存在的，比如Redhat/Fedora、红旗、Slackware、Debian、SuSE ``` 等系统，都是采用这个拔号软件，所以您大可不必为下载源码编译安装。只需要在各大发行版的安装盘 中就可以找得到；请用各大发行版自带的软件包管理工具来安装此软件包； ``` 如果您用的是各大发行版提供的rp- pppoe软件包 比如 RPM包的系统是用 rpm -ivh ``` rp-pppoe*.rpmSlackware 系统是用 installpkg rp-pppoe*.tgz 在Redhat/Fedora 中可以通过图形配 置工具来完成，redhat-config-network 命令，调用配置网络，要通过XDSL来添加拔号，比较简单； ``` 所有发行版通用的方法是adsl-setup 命令来配置ADSL； ``` 1. [root@localhost ~]# adsl-setup 注：配置pppoe拔号，请文档下面，都差不多； 2. [root@localhost ~]# adsl-start 注：启动拔号； 3. [root@localhost ~]# adsl-stop 注：断开连接； ``` 5.12 如果是源码包安装，我们要自己来编译安装； ``` ``` [root@localhost ~]# tar zxvf rp- pppoe-3.8.tar.gz ``` ``` [root@localhost ~]# cd rp-pppoe-3.8 ``` ``` [root@localhost rp-pppoe-3.8]# ./go ``` ``` Welcome to the Roaring Penguin PPPoE client setup. First, I will run ``` ``` some checks on your system to make sure the PPPoE client is installed ``` ``` properly... ``` ``` Looks good! Now, please enter some information: ``` ``` USER NAME ``` ``` >>> Enter your PPPoE user name (default bxxxnxnx@sympatico.ca): 在这里添写你的拔号用户 ``` 名；就是服务商提供的； ``` >>> Enter the Ethernet interface connected to the DSL modem ``` ``` For Solaris, this is likely to be something like /dev/hme0. ``` ``` For Linux, it will be ethn, where 'n' is a number. ``` ``` (default eth0): eth0 如果是乙太接口的ADSL，就要在这里写上接猫的那个网络接口号。此处是 ``` eth0； ``` Do you want the link to come up on demand, or stay up continuously? ``` ``` If you want it to come up on demand, enter the idle time in seconds ``` ``` after which the link should be dropped. If you want the link to ``` ``` stay up permanently, enter 'no' (two letters, lower-case.) ``` ``` NOTE: Demand-activated links do not interact well with dynamic IP ``` ``` addresses. You may have some problems with demand-activated links. ``` ``` >>> Enter the demand value (default no): 注：默认回车 ``` ``` >>> Enter the DNS information here: 202.96.134.133 注：在这里写上DNS服务器地址；可以 ``` 和提供商要，也可以用我写的这个； ``` Please enter the IP address of your ISP's secondary DNS server. ``` ``` If you just press enter, I will assume there is only one DNS server. ``` ``` >>> Enter the secondary DNS server address here: 202.96.128.143 这是第二个DNS服务器地 ``` 址； ``` >>> Please enter your PPPoE password:在这里输入用户的密码； ``` ``` >>> Please re-enter your PPPoE password: 确认密码； ``` ``` The firewall choices are: ``` ``` 0 - NONE: This script will not set any firewall rules. You are responsible ``` ``` for ensuring the security of your machine. You are STRONGLY ``` ``` recommended to use some kind of firewall rules. ``` ``` 1 - STANDALONE: Appropriate for a basic stand-alone web-surfing workstation ``` ``` 2 - MASQUERADE: Appropriate for a machine acting as an Internet gateway ``` ``` for a LAN ``` ``` >>> Choose a type of firewall (0-2): 2 注：在这里写上 2 ，可以共享上网的；当然还得加一 ``` 条防火墙规划 ； ``` Ethernet Interface: eth0 ``` ``` User name:dxxx ``` ``` Activate-on-demand: No ``` ``` Primary DNS: 202.96.134.133 ``` ``` Secondary DNS: 202.96.128.143 ``` ``` Firewalling: MASQUERADE ``` ``` >>> Accept these settings and adjust configuration files (y/n)? y 注：是不是保存配置； ``` ``` 关于共享上网，请参考：《ADSL共享上网的解决办法》 ``` ``` 5.2 普通猫的拔号工具介绍； ``` ``` 普通猫分为串口和PCI的，请查看 《关于网络设备概述 》 普通猫的拔号工具主要有kppp和wvdial； ``` 在Redhat/Fedora中，用system-config-network 或redhat-config-network 也能设置ppp拔号； 在 KDE桌面环境下，大家一般都用kppp拔号工具，点鼠标就可以完成； ``` wvdial工具是文本的，几乎在各大发行版都有。wvdial的配置文件是/etc/wvdial.conf 。如果您 ``` 的猫已经驱动好了，运行一下wvdialconf命令就生成了/etc/wvdial.conf了 。当然您得查看一下它 的内容； 1. [root@localhost ~]# wvdialconf 2. [root@localhost ~]# more /etc/wvdial.conf ``` 关于wvdial工具的使用，请查看《普通 56K猫拔号上网工具 wvdial 介绍》 ``` ``` 6 、无线网卡； ``` ``` 正在更新之中；由于我没有这样的网卡，是否有弟兄写一篇详尽一点的？在所有涉及无线网卡的文 ``` 档中，这块都是空白。缺的就是这个。 看来我是得弄一块无线网卡了。。。。。。 ``` 7 、DNS客户端配置文件/etc/resolv.conf； ``` ``` 本来不应该把DNS客户端配置文件放在这里来说，但由于新手弟兄上网时，虽然能拔号，但不能以 ``` 域名访问。究其原因是由于没有修改 /etc/resolv.conf 文件； ``` /etc/resolv.conf 里面存放的是各大通信公司DNS服务器列表；下面的三个地址可以用一用；当 ``` 然您可以打电话问你的服务商； 1. nameserver 202.96.134.133 2. nameserver 202.96.128.143 3. nameserver 202.96.68.38 ``` 本文写了常用的乙太网接口的配置，介绍了Linux ifconfig 、netconfig 等，我感觉最重要的还 ``` 是配置文件，新手弟兄还是仔细看看配置文件吧。当您用工具配置完成后，不妨查看一下相应配置文件 的变化。我认为这样的学习方式，能知其然，然后知所以然； #### 7.10 、 tcpdump ``` tcpdump是一个用于截取网络分组，并输出分组内容的工具。tcpdump凭借强大的功能和灵活的截 ``` 取策略，使其成为类UNIX系统下用于网络分析和问题排查的首选工具。 ``` tcpdump提供了源代码，公开了接口，因此具备很强的可扩展性，对于网络维护和入侵者都是非常 ``` 有用的工具。tcpdump存在于基本的Linux系统中，由于它需要将网络界面设置为混杂模式，普通用户 不能正常执行，但具备root权限的用户可以直接执行它来获取网络上的信息。因此系统中存在网络分 析工具主要不是对本机安全的威胁，而是对网络上的其他计算机的安全存在威胁。 7.1.1概述 顾名思义，tcpdump可以将网络中传送的数据包的“头”完全截获下来提供分析。它支持针对网络层、 协议、主机、网络或端口的过滤，并提供and、or、not等逻辑语句来帮助你去掉无用的信息。 引用 # tcpdump -vv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 11:53:21.444591 IP (tos 0x10, ttl 64, id 19324, offset 0, flags [DF], proto 6, length: 9 2) asptest.localdomain.ssh > 192.168.228.244.1858: P 3962132600:3962132652(52) ack 272652 5936 win 1266 asptest.localdomain.1077 > 192.168.228.153.domain: [bad udp cksum 166e!] 325+ PTR? 244.2 28.168.192.in-addr.arpa. (46) 11:53:21.446929 IP (tos 0x0, ttl 64, id 42911, offset 0, flags [DF], proto 17, length: 1 51) 192.168.228.153.domain > asptest.localdomain.1077: 325 NXDomain q: PTR? 244.228.168. 192.in-addr.arpa. 0/1/0 ns: 168.192.in-addr.arpa. (123) 11:53:21.447408 IP (tos 0x10, ttl 64, id 19328, offset 0, flags [DF], proto 6, length: 1 72) asptest.localdomain.ssh > 192.168.228.244.1858: P 168:300(132) ack 1 win 1266 347 packets captured 1474 packets received by filter 745 packets dropped by kernel 不带参数的tcpdump会收集网络中所有的信息包头，数据量巨大，必须过滤。 7.1.2、命令介绍 命令格式为：tcpdump [-nn] [-i 接口] [-w 储存档名] [- c 次数] [-Ae] [-qX] [-r 文件] [所欲捕获的数据内容] 参数： -nn，直接以 IP 及 Port Number 显示，而非主机名与服务名称。 -i，后面接要「监听」的网络接口，例如 eth0, lo, ppp0 等等的接口。 -w，如果你要将监听所得的数据包数据储存下来，用这个参数就对了。后面接文件名。 -c，监听的数据包数，如果没有这个参数， tcpdump 会持续不断的监听， 直到用户输入 [ctrl]-c 为止。 -A，数据包的内容以 ASCII 显示，通常用来捉取 WWW 的网页数据包资料。 -e，使用资料连接层 (OSI 第二层) 的 MAC 数据包数据来显示。 -q，仅列出较为简短的数据包信息，每一行的内容比较精简。 -X，可以列出十六进制 (hex) 以及 ASCII 的数据包内容，对于监听数据包内容很有用。 -r，从后面接的文件将数据包数据读出来。那个「文件」是已经存在的文件，并且这个「文件」是由 -w 所制 作出来的。所欲捕获的数据内容：我们可以专门针对某些通信协议或者是 IP 来源进行数据包 捕获。 那就可以简化输出的结果，并取得最有用的信息。常见的表示方法有。 'host foo', 'host 127.0.0.1' ：针对单台主机来进行数据包捕获。 'net 192.168' ：针对某个网段来进行数据包的捕获。 'src host 127.0.0.1' 'dst net 192.168'：同时加上来源(src)或目标(dst)限制。 'tcp port 21'：还可以针对通信协议检测，如tcp、udp、arp、ether 等。 除了这三种类型的关键字之外，其他重要的关键字如下：gateway, broadcast,less, greater,还有三种逻辑运算，取非运算是 'not ' '! ', 与运算是'and','&&';或运算 是'o r' ,'||'； 范例一：以 IP 与 Port Number 捉下 eth0 这个网卡上的数据包，持续 3 秒 [root@linux ~]# tcpdump -i eth0 -nn tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 01:33:40.41 IP 192.168.1.100.22 > 192.168.1.11.1190: P 116:232(116) ack 1 win 9648 01:33:40.41 IP 192.168.1.100.22 > 192.168.1.11.1190: P 232:364(132) ack 1 win 9648 <==按下 [ctrl]-c 之后结束 6680 packets captured <==捉取下来的数据包数量 14250 packets received by filter <==由过滤所得的总数据包数量 7512 packets dropped by kernel <==被核心所丢弃的数据包 至于那个在范例一所产生的输出中，我们可以大概区分为几个字段，现以范例一当中那行特殊字体行来 说明一下： · 01:33:40.41：这个是此数据包被捕获的时间，“时:分:秒”的单位。 · IP：通过的通信协议是IP。 · 192.168.1.100.22>：传送端是192.168.1.100这个IP，而传送的Port Number为 22 ，那个大于（>） 的符号指的是数据包的传输方向。 · 192.168.1.11.1190：接收端的IP是192.168.1.11，且该主机开启port 1190来接收。 · P 116:232(116)：这个数据包带有PUSH的数据传输标志，且传输的数据为整体数据的116~232 Byt e，所以这个数据包带有116 Bytes的数据量。 · ack 1 win 9648：ACK与Window size的相关资料。 最简单的说法，就是该数据包是由192.168.1.100传到192.168.1.11，通过的port是由 22 到 1190 ， 且带有116 Bytes的数据量，使用的是PUSH的标记，而不是SYN之类的主动联机标志。 接下来，在一个网络状态很忙的主机上面，你想要取得某台主机对你联机的数据包数据时，使用tcpdu mp配合管线命令与正则表达式也可以，不过，毕竟不好捕获。我们可以通过tcpdump的表达式功能， 就能够轻易地将所需要的数据独立的取出来。在上面的范例一当中，我们仅针对eth0做监听，所以整 个eth0接口上面的数据都会被显示到屏幕上，但这样不好分析，可以简化吗？例如，只取出port 21 的联机数据包，可以这样做： [root@linux ~]# tcpdump -i eth0 -nn port 21 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 01:54:37.96 IP 192.168.1.11.1240 > 192.168.1.100.21:. ack 1 win 65535 01:54:37.96 IP 192.168.1.100.21 > 192.168.1.11.1240:P 1:21(20) ack 1 win 5840 01:54:38.12 IP 192.168.1.11.1240 > 192.168.1.100.21:. ack 21 win 65515 01:54:42.79 IP 192.168.1.11.1240 > 192.168.1.100.21:P 1:17(16) ack 21 win 65515 01:54:42.79 IP 192.168.1.100.21 > 192.168.1.11.1240:. ack 17 win 5840 01:54:42.79 IP 192.168.1.100.21 > 192.168.1.11.1240: P 21:55(34) ack 17 win 5840 看！这样就仅取出port 21的信息，如果仔细看的话，你会发现数据包的传递都是双向的，Client端 发出请求而Server端则予以响应，所以，当然是有去有回了。而我们也就可以经过这个数据包的流向 来了解到数据包运动的过程了。例如： · 我们先在一个终端机窗口输入“tcpdump-i lo-nn”的监听。 · 再另开一个终端机窗口来对本机（127.0.0.1）登录“ssh localhost”，那么输出的结果会是如何？ [root@linux ~]# tcpdump -i lo -nn 1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 2 listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes 3 11:02:54.253777 IP 127.0.0.1.32936 > 127.0.0.1.22: S 933696132:933696132(0) win 32767 4 11:02:54.253831 IP 127.0.0.1.22 > 127.0.0.1.32936: S 920046702:920046702(0) ack 933696133 win 32767 5 11:02:54.253871 IP 127.0.0.1.32936 > 127.0.0.1.22:. ack 1 win 8192 6 11:02:54.272124 IP 127.0.0.1.22 > 127.0.0.1.32936: P 1:23(22) ack 1 win 8192 7 11:02:54.272375 IP 127.0.0.1.32936 > 127.0.0.1.22:. ack 23 win 8192 代码显示的头两行是tcpdump的基本说明，然后： · 第 3 行显示的是来自Client端带有SYN主动联机的数据包。 · 第 4 行显示的是来自Server端，除了响应Client端之外（ACK），还带有SYN主动联机的标志。 · 第 5 行则显示Client端响应Server确定联机建立（ACK）。 · 第 6 行以后则开始进入数据传输的步骤。 从第3~5行的流程来看，熟不熟悉啊？没错。那就是 3 次握手的基础流程，有趣吧。不过tcpdump之所 以被称为黑客软件之一远不止上面介绍的功能。上面介绍的功能可以用来作为我们主机的数据包联机与 传输的流程分析，这将有助于我们了解到数据包的运作，同时了解到主机的防火墙设置规则是否有需要 修订的地方。 还有更神奇的用法。当我们使用tcpdump在Router上面监听明文的传输数据时，例如FTP传输协议， 你觉得会发生什么问题呢？我们先在主机端执行“tcpdump -i lo port 21 -nn –X”，然后再以FTP 登录本机，并输入账号与密码，结果你就可以发现如下的状况： [root@linux ~]# tcpdump -i lo -nn -X 'port 21' 0x0000: 4500 0048 2a28 4000 4006 1286 7f00 0001 E..H*(@.@....... 0x0010: 7f00 0001 0015 80ab 8355 2149 835c d825 .........U!I.\.% 0x0020: 8018 2000 fe3c 0000 0101 080a 0e2e 0b67 .....<.........g 0x0030: 0e2e 0b61 3232 3020 2876 7346 5450 6420 ...a220.(vsFTPd. 0x0040: 322e 302e 3129 0d0a 2.0.1).. 0x0000: 4510 0041 d34b 4000 4006 6959 7f00 0001 E..A.K@.@.iY.... 0x0010: 7f00 0001 80ab 0015 835c d825 8355 215d .........\.%.U!] 0x0020: 8018 2000 fe35 0000 0101 080a 0e2e 1b37 .....5.........7 0x0030: 0e2e 0b67 5553 4552 2064 6d74 7361 690d ...gUSER.dmtsai. 0x0040: 0a. 0x0000: 4510 004a d34f 4000 4006 694c 7f00 0001 E..J.O@.@.iL.... 0x0010: 7f00 0001 80ab 0015 835c d832 8355 217f .........\.2.U!. 0x0020: 8018 2000 fe3e 0000 0101 080a 0e2e 3227 .....>........2' 0x0030: 0e2e 1b38 5041 5353 206d 7970 6173 7377 ...8PASS.mypassw 0x0040: 6f72 6469 7379 6f75 0d0a ordisyou.. 上面的输出结果已经被简化过了，你需要自行在你的输出结果中搜索相关的字符串才行。从上面输出结 果的特殊字体中，我们可以发现该FTP软件使用的是 vsFTPd，并且用户输入dmtsai这个账号名称，且 密码是mypasswordisyou。如果使用的是明文方式来传输你的网络数据呢？ 另外你得了解，为了让网络接口可以让tcpdump监听，所以执行tcpdump时网络接口会启动在“混杂模 式（promiscuous）”，所以你会在 /var/log/messages里面看到很多的警告信息，通知你说你的网卡被 设置成为混杂模式。别担心，那是正常的。至于更多的应用，请参考man tcpdump了。 例题：如何使用tcpdump监听来自eth0适配卡且通信协议为port 22，目标来源为192.168.1.100的 数据包资料？ 答：tcpdump -i eth0 -nn port 22 and src host 192.168.1.100 例题：如何使用tcpdump抓取访问eth0适配卡且访问端口为tcp 9080？ 答:tcpdump -i eth0 dst 172.168.70.35 and tcp port 9080 例题：如何使用tcpdump抓取与主机192.168.43.23或着与主机192.168.43.24通信报文，并且显示在 控制台上 答：tcpdump -X -s 1024 -i eth0 host \(192.168.43.23 or 192.168.43.24\) and host 172.16.7 0.35 ##### 注：必须指定网卡 7.1.3、tcpdump的表达式介绍 表达式是一个正则表达式，tcpdump利用它作为过滤报文的条件，如果一个报文满足表 达式的条件， 则这个报文将会被捕获。如果没有给出任何条件，则网络上所有的信息包 将会被截获。 在表达式中一般如下几种类型的关键字： 引用 第一种是关于类型的关键字，主要包括host，net，port，例如 host 210.27.48.2， 指明 210.27.48. 2 是一台主机，net 202.0.0.0指明202.0.0.0是一个网络地址，port 23 指明端口号是 23 。如果没有 指定类型，缺省的类型是host。 第二种是确定传输方向的关键字，主要包括src，dst，dst or src，dst and src， 这些关键字指明 了传输的方向。举例说明，src 210.27.48.2 ，指明ip包中源地址是 210.27.48.2 ， dst net 202. 0.0.0 指明目的网络地址是202.0.0.0。如果没有指明 方向关键字，则缺省是src or dst关键字。 第三种是协议的关键字，主要包括fddi，ip，arp，rarp，tcp，udp等类型。Fddi指明是在FDDI (分 布式光纤数据接口网络)上的特定的网络协议，实际上它是”ether”的别名，fddi和ether 具有类似 的源地址和目的地址，所以可以将fddi协议包当作ether的包进行处理和分析。 其他的几个关键字就 是指明了监听的包的协议内容。如果没有指定任何协议，则tcpdump 将会 监听所有协议的信息包。 除了这三种类型的关键字之外，其他重要的关键字如下：gateway， broadcast，less， greater， 还有三种逻辑运算，取非运算是 ‘not ' '! ‘， 与运算是’and’，’&&';或运算是’or’ ，’| |’； 这些关键字可以组合起来构成强大的组合条件来满足人们的需要。 四、输出结果介绍 下面我们介绍几种典型的tcpdump命令的输出信息 (1) 数据链路层头信息 使用命令： #tcpdump --e host ICE ICE 是一台装有linux的主机。它的MAC地址是 0 ： 90 ： 27 ： 58 ：AF：1A H219是一台装有Solaris的 SUN工作站。它的MAC地址是 8 ： 0 ： 20 ： 79 ：5B： 46 ； 上一条命令的输出结果如下所示： 引用 21:50:12.847509 eth0 < 8:0:20:79:5b:46 0:90:27:58:af:1a ip 60: h219.33357 > ICE. telne t 0:0(0) ack 22535 win 8760 (DF) 21 ： 50 ： 12 是显示的时间， 847509 是ID号，eth0 <表示从网络接口eth0接收该分组， eth0 >表示 从网络接口设备发送分组， 8:0:20:79:5b:46是主机H219的MAC地址， 它表明是从源地址H219发来 的分组. 0:90:27:58:af:1a是主机ICE的MAC地址， 表示该分组的目的地址是ICE。 ip 是表明该分 组是IP分组， 60 是分组的长度， h219.33357 > ICE. telnet 表明该分组是从主机H219的 33357 端 口发往主机ICE的 TELNET(23)端口。 ack 22535 表明对序列号是 222535 的包进行响应。 win 8760 表明发 送窗口的大小是 8760 。 ``` (2) ARP包的tcpdump输出信息 使用命令： #tcpdump arp ``` ##### 得到的输出结果是： ##### 引用 22:32:42.802509 eth0 > arp who-has route tell ICE (0:90:27:58:af:1a) 22:32:42.802902 eth0 < arp reply route is-at 0:90:27:12:10:66 (0:90:27:58:af:1a) 22:32:42是时间戳， 802509 是ID号， eth0 >表明从主机发出该分组，arp表明是ARP请求包， who -has route tell ICE表明是主机ICE请求主机route的MAC地址。 0:90:27:58:af:1a是主机 ICE的 MAC地址。 (3) TCP包的输出信息 用tcpdump捕获的TCP包的一般输出信息是： 引用 src > dst: flags data-seqno ack window urgent options src > dst:表明从源地址到目的地址， flags是TCP报文中的标志信息，S 是SYN标志， F (FIN)， P (PUSH) ， R (RST) "." (没有标记); data-seqno是报文中的数据 的顺序号， ack是下次期望的顺 序号， window是接收缓存的窗口大小， urgent表明 报文中是否有紧急指针。 Options是选项。 (4) UDP包的输出信息 用tcpdump捕获的UDP包的一般输出信息是： 引用 route.port1 > ICE.port2: udp lenth UDP十分简单，上面的输出行表明从主机route的port1端口发出的一个UDP报文 到主机ICE的port 2 端口，类型是UDP， 包的长度是lenth。 五、举例 (1) 想要截获所有210.27.48.1 的主机收到的和发出的所有的分组： #tcpdump host 210.27.48.1 (2) 想要截获主机210.27.48.1 和主机210.27.48.2或210.27.48.3的通信，使用命令（注意：括号 前的反斜杠是必须的）： #tcpdump host 210.27.48.1 and \(210.27.48.2 or 210.27.48.3 \) (3) 如果想要获取主机210.27.48.1除了和主机210.27.48.2之外所有主机通信的ip包，使用命令： #tcpdump ip host 210.27.48.1 and! 210.27.48.2 (4) 如果想要获取主机192.168.228.246接收或发出的ssh包，并且不转换主机名使用如下命令： #tcpdump -nn - n src host 192.168.228.246 and port 22 and tcp (5) 获取主机192.168.228.246接收或发出的ssh包，并把mac地址也一同显示： # tcpdump -e src host 192.168.228.246 and port 22 and tcp -n -nn (6) 过滤的是源主机为192.168.0.1与目的网络为192.168.0.0的报头： tcpdump src host 192.168.0.1 and dst net 192.168.0.0/24 (7) 过滤源主机物理地址为XXX的报头： tcpdump ether src 00:50:04:BA:9B and dst...... （为什么ether src后面没有host或者net？物理地址当然不可能有网络喽）。 (8) 过滤源主机192.168.0.1和目的端口不是telnet的报头，并导入到tes.t.txt文件中： Tcpdump src host 192.168.0.1 and dst port not telnet -l > test.txt ip icmp arp rarp 和 tcp、udp、icmp这些选项等都要放到第一个参数的位置，用来过滤数据报的类 型。 ## 十五、 配置文件 #### 1 、 配置文件介绍： 每个 Linux 程序都是一个可执行文件，它含有操作码列表，CPU 将执行这些操作码来完成特定的操作。 例如，ls 命令是由 /bin/ls 文件提供的，该文件含有机器指令的列表，在屏幕上显示当前目录中文件 的列表时需要使用这些机器指令。几乎每个程序的行为都可以通过修改其配置文件来按照您的偏好或需 要去定制。 Linux 中有没有一个标准的配置文件格式？ 一句话，没有。不熟悉 Linux 的用户（一定）会感到沮丧，因为每个配置文件看起来都象是一个要迎 接的新挑战。在 Linux 中，每个 程序员都可以自由选择他或她喜欢的配置文件格式。可以选择的格式 很多，从 /etc/shells 文件（它包含被一个换行符分开的 shell 的列表），到 Apache 的复杂的 /etc/httpd.conf 文件。 什么是系统配置文件？ 内核本身也可以看成是一个“程序”。为什么内核需要配置文件？内核需要了解系统中用户和组的列表， 进而管理文件权限（即根据权限判定特定用户（UNIX_USERS）是否可以打开某个文件）。注意，这些文 件不是明确地由程序读取的，而是由系统库所提供的一个函数读取，并被内核使用。例如，程序需要某 个用户的（加密过的）密码时不应该打开 /etc/passwd 文件。相反，程序应该调用系统库的 getpw() 函 数。这种函数也被称为系统调用。打开 /etc/passwd 文件和之后查找那个被请求的用户的密码都是由 内核（通过系统库）决定的。 除非另行指定，Red Hat Linux 系统中大多数配置文件都在 /etc 目录中。配置文件可以大致分为下面 几类： #### 2 、配置文件分类： ##### 访问文件 ``` /etc/host.conf 告诉网络域名服务器如何查找主机名。（通常是 /etc/hosts， 然后就是名称服务器；可通过 netconf 对其进行更改） ``` ``` /etc/hosts ``` ##### 包含（本地网络中）已知主机的一个列表。如果系统的 IP 不 ##### 是动态生成，就可以使用它。对于简单的主机名解析（点分表 ##### 示法），在请求 DNS 或 NIS 网络名称服务器之前， ``` /etc/hosts.conf 通常会告诉解析程序先查看这里。 /etc/hosts.allow 请参阅 hosts_access 的联机帮助页。至少由 tcpd 读取。 /etc/hosts.deny 请参阅 hosts_access 的联机帮助页。至少由 tcpd 读取。 ``` ##### 引导和登录／注销 ``` /etc/issue & /etc/issue.net ``` ``` 这些文件由 mingetty（和类似的程序）读取，用来向从终端（issue）或 通过 telnet 会 话（issue.net）连接的用户显示一个“welcome”字符串。 它们包括几行声明 Red Hat 版本号、名称和内核 ID 的信息。它们由 ``` ``` rc.local 使用。 /etc/redhat-release 包括一行声明 Red Hat 版本号和名称的信息。由 rc.local 使用。 ``` ``` /etc/rc.d/rc ``` ``` 通常在所有运行级别运行，级别作为参数传送。例如，要以图形（Graphics） 模式（X- Server）引导机器，请在命令行运行下面的命令： init 5 。运 行级别 5 表示以图形模式引导系统。 /etc/rc.d/rc.local 非正式的。可以从 rc、rc.sysinit 或 /etc/inittab 调用。 /etc/rc.d/rc.sysinit 通常是所有运行级别的第一个脚本。 ``` ``` /etc/rc.d/rc/rcX.d ``` ``` 从 rc 运行的脚本（ X 表示 1 到 5 之间的任意数字）。这些目录是特 定“运行级别”的目录。当系统启动时，它会识别要启动的运行级别，然 后调用该运行级别的特定目录中存在的所有启动脚本。例如，系统启动时 通常会在引导消息之后显示“entering run-level 3”的消息；这意味着 /etc/rc.d/rc3.d/ 目录中的所有初始化脚本都将被调用。 ``` ##### 文件系统 ##### 内核提供了一个接口，用来显示一些它的数据结构，这些数据结构对于决定诸如使用的中断、初始化 ##### 的设备和内存统计信息之类的系统参数可能很有用。这个接口是作为一个独立但虚拟的文件系统提供 的，称为 /proc 文件系统。很多系统实用程序都使用这个文件系统中存在的值来显示系统统计信息。 例如，/proc/modules 文件列举系统中当前加载的模块。lsmod 命令读取此信息，然后将其以人们可 以看懂的格式显示出来。下面表格中指定的 mtab 文件以同样的方式读取包含当前安装的文件系统的 /proc/mount 文件。 ``` /etc/mtab ``` ``` 这将随着 /proc/mount 文件的改变而不断改变。换句话说，文件系统被安装和 卸载时，改变会立即反映到此文件中。 1.文件格式 /etc/mtab的格式和/etc/fstab是一样的.但这个文件不能算是用户配置文件, 他是由系统维护的.和/etc/fstab的区别在于, fstab是系统启动时需挂载的文 件系统列表,而mtab是系统当前已挂载的文件系统列表,它由系统维护,在用户 执行了mount或者umount命令后自动更新.用户不应该对此文件作任何修改. 2.安全性 /etc/mtab的默认权限仍然是 644 3.相关命令 mount umount smbmount ``` ``` /etc/fstab ``` ##### 1.文件格式 ``` /etc/fstab记载了系统启动时自动挂载的文件系统。一行为一条记录。每条记 录有 6 个字段，字段间用空格或者tab键分开。这六个字段分别是：设备名称， 挂载点（除交换分区为swap外，都必须是一个存在的目录名），文件系统类型， mount选项，是否需要dump（ 1 表示需要， 0 表示不需要），在 reboot期间fsck 检查的顺序（激活文件系统设定为 1 ，其余文件系统设定为 2 ，若设定为 0 表示 该文件系统不需要被检查）。 在linux和windows共存时，也许想开机自动挂载windows分区，那么就可以 在这个文件里加上相应的记录。 ``` ##### 某些时候对硬盘分区作了调整以后，这里也需要做一些相应的修改。否则会出 ##### 现一些问题。 可用的mount选项： async 对该文件系统的所有I/O操作都异步执行 ro 该文件系统是只读的 rw 该文件系统是可读可写的 atime 更新每次存取inode的存取时间 auto 可以使用 -a 选项mount defaults 使用预设的选项：rw,suid,dev,exec,auto,nouser,async dev 解释在文件系统上的字符或区块设备 exec 允许执行二进制文件 noatime 不要在这个文件系统上更新存取时间 noauto 这个文件系统不能使用 -a 选项来mount nodev 不要解释在文件系统上的字符或区块设备 noexec 不允许在mounted文件系统上执行任何的二进制文件。这个选项对于具有包含 非它自己的二进制结构的文件系统服务器而言非常有用 nosuid 不允许setuid和setgid位发生作用。（这似乎很安全，但是在安装suidperl 后，同样不安全）。 nouser 限制一般非root用户mount文件系统 remount 尝试重新mount已经mounted的文件系统。这通常是用来改变文件系统的mount 标志，特别是让只读的文件系统变成可擦写的 suid 允许setuid和setgid位发生作用 sync 文件系统的所有I/O同步执行 user 允许一般非root用户mount文件系统。这个选项会应用noexec,nosuid,nodev 这三个选项（除非在命令行上有指定覆盖这些设定的选项）。 3.安全性 ``` /etc/fstab的默认权限是644,所有者和所有组均为root 2.相关命令 mount df 列举计算机当前“可以安装”的文件系统。这非常重要，因为计算机引导时将 运行 mount -a 命令，该命令负责安装 fstab 的倒数第二列中带有“1”标记 的每一个文件系统。 /etc/mtools.conf DOS 类型的文件系统上所有操作（创建目录、复制、格式化等等）的配置。 ``` 系统管理 ``` /etc/group ``` ##### 1.文件格式 ``` /etc/group存储了系统中所有用户的基本信息.它的格式和/etc/passwd的格 式基本类似,这里就说简单一点, /etc/group也是由一条条的记录组成.每条记录分 4 个字段.分别是组名,组口 令,组ID和该组包含用户列表.其中组口令不再使用（现在只是保留为 x）.最 后一个域是一个用逗号分隔的用户名列表,这个组的成员就是在这里列出的所 有用户. 2.安全性 /etc/group的默认权限是644,所有者和所有组均为root.注意经常检察. 3.相关命令 groupadd groupdel groupmod groups 包含有效的组名称和指定组中包括的用户。单一用户如果执行多个任务，可以 存在于多个组中。例如，如果一个“ 用户”是 “project 1”工程组的成员，同 时也是管理员，那么在 group 文件中他的条目看起来就会是这样的： user: * : group-id : project1 ``` ``` /etc/nologin ``` ``` 这是一个普通的文本文件.你可以在里面写上你喜欢的任何东西./etc/nologin 的作用在于,如果它存在,那么系统将拒绝任何非root用户的登录请求,并对其 它登录 用户显示此文件的内容 此文件常由系统在停机前自动生成.有时系统管理员也会手工生成它,用以禁止 其它用户登录,方便进行一些管理工作. ``` ``` etc/passwd ``` ##### 1.文件格式 ``` /etc/passwd存储了系统中所有用户的基本信息.可以说这是系统中最重要的一 个配置文件.对它作任何修改一定要小心谨慎.同时要经常检察这个文件,包括 它的内容和权限设置. 使用vi编辑程序打开此文件,可以看到这个文件由许多行记录组成.每一行记 录对应着一个用户.我们以第一行为例.第一行一般是root用户的记录,尽管这 不是必需的.实际上用户记录出现的顺序并没有任何的意义. 在我的系统中,/etc/passwd的第一行看起来是这样的: root:x:0:0:root:/root:/bin/bash 每一条记录都由 7 个字段组成,每个字段之间用冒号隔开.第一个字段是用户 名,示例中是root.第二个字段是用户口令,示例中是一个字符x,但这并不表示 ``` ``` root的口令是单个字符x,而是说用户口令被加密了,并且加密口令也没有放在 本文件中,而是放到了/etc/shadow(参考 /etc/shadow).假如删除这个x,那么 root的口令就清空了.第三个字段是用户的用户ID,即uid.第四个字段是用户 的组ID,即gid. 这里要注意,系统分辨两个用户是看他们的uid是否相同而不 是看他们的用户名是否相同.用户名不同但uid相同的两个用户实际上是同一 个用户.对组来说也有类似的规则.所以这两个字段大家一定要注意.第五个字 段是用户全称,没有什么实际用途,相当于注释,这里是root.第六个字段是用户 的主目录 (home),即登录系统后默认所处目录,这里是/root.最后一个字段是 用户的登录shell,可以是系统拥有的任何一个shell的完整路径,这里是 /bin/bash.注意,这个字段可以有一个特殊的值,即/sbin/nologin.如果把一个 用户的登录shell设置为 /sbin/nologin的话,系统将禁止此用户的本地登录. 请参阅“man passwd”。它包含一些用户帐号信息，包括密码（如果未被 shadow 程序加密过）。 2.安全性 /etc/passwd的默认权限为644,所有者和所有组均为root.切记,在任何情况下 都不要更改它. 3.相关命令 passwd useradd userdel adduser usermod users ``` /etc/rpmrc rpm 命令配置。所有的 rpm 命令行选项都可以在这个文件中一起设置，这样， 当任何 rpm 命令在该系统中运行时，所有的选项都会全局适用。 /etc/securetty ``` 包含设备名称，由 tty 行组成（每行一个名称，不包括前面的 /dev/），root 用户在这里被允许登录。 1.文件格式 这是一个设备文件的列表.文件名取相对于/dev的相对路径.如,/dev/tty1记 为tty1 root只有从这个列表中列出的设备上才可以登录系统. 例如: 代码: $cat /etc/securretty tty1 tty2 tty3 这里root被限定只能从/dev/tty1, /dev/tty2, /dev/tty3这三个设备上登录 系统 如果/etc/securretty不存在的话,那么root将可以从任何设备登录系统. 2.安全性 /etc/securetty的默认权限是600,所有者和所有组都是root ``` /etc/shadow 包含加密后的用户帐号密码信息，还可以包括密码时效信息。包括的字段有： ##### • 登录名 ##### • 加密后的密码 ##### • 从 1970 年 1 月 1 日到密码最后一次被更改的天数 ##### • 距密码可以更改之前的天数 ##### • 距密码必须更改之前的天数 ##### • 密码到期前用户被警告的天数 ##### • 密码到期后帐户被禁用的天数 ##### • 从 1970 年 1 月 1 日到帐号被禁用的天数 ##### 1.文件格式 /etc/shadow文件保存的是用户名,密码,用户账号设置相关信息。 例： root:$1$6UviCNvh$WTR0zPMek41KmzD0Z1DdV1:12264:3:4:5:6:12267: 第一段: root----- 用户注册名 第二段: $1$6UviCNvh$WTR0zPMek41KmzD0Z1DdV1 ----加密口令 第三段: 12264-----上次更动密码的日期,以 1970 年 1 月 1 日为1,1天加 1 第四段: 3---------密码将被允许修改之前的天数（ 0 表示“可在任何时间修 改”） 第五段: 4---------系统将强制用户修改为新密码之前的天数（ 1 表示“永远 都不能修改”） 第六段: 5---------密码过期之前，用户将被警告过期的天数（-1 表示 “没有 警告”） 第七段: 6---------密码过期之后，系统自动禁用帐户的天数（-1 表示 “永远 不会禁用”） 第八段: 12267-----该帐户被禁用的天数（-1 表示“该帐户被启用”）.以 1970 年 1 月 1 日为1,1月 2 日为 2 第九段 ------ 保留供将来使用 注:第 2 段中为*表示帐号不可登录,如密码前为 !! 或只有 !! 表示帐号被锁 2.安全性 ``` /etc/shadow的默认所有者和所有组均为root. ``` ``` 建议运行# chattr +i /etc/shadow来保护文件使其不被意外地删除或重写 ``` ``` 3.相关命令 ``` ``` passwd ``` ``` useradd ``` ``` userdel ``` ``` usermod ``` /ect/gshadow ##### 1.文件格式 ``` /ect/gshadow文件保存的是用户和组群设置的信息 例: root:!!::root,wa1 第一段：组名 第四段：该组包含用户列表 2.安全性 /etc/gshadow的默认所有者和所有组均为root. 建议运行# chattr +i /etc/shadow来保护文件使其不被意外地删除或重写 3.相关命令 groupadd groupdel groupmod groups ``` /etc/sysctl.conf ##### 1.文件格式 ``` /etc/sysctl.conf是sysctl程序的配置文件.sysctl可以在系统运行时更改内 核参数./etc/sysctl.conf中的配置将在系统起动时执行. 以 # 和 ; 开始的行是注释,将和空白行一起被忽略. 配置项的格式为: token = value token是一个键名,value是对应的键值.token和value前后的空格将被忽略 token不能是随意的字符串.他和/proc/sys下的文件有一一对应的关系: 假设foo是/proc/sys下的一个文件.删除foo的绝对路径前的 "/proc/sys" 这 一部分,然后把剩下部分中的 "/" 替换成 ".",得到的字符串就是foo所对应 的键名.例如: /proc/sys/net/ipv4/ip_forward对应的键 名为net.ipv4.ip_forward 应用举例: Redhat Linux 9默认是禁止ip转发的,而我们在做ip伪装时需要起用ip转发. 通常的做法是在iptables的规则之前加上一句: echo 1>/proc/sys/net/ipv4/ip_forward 实际上我们也可以在/etc/sysctl.conf中写上: ``` ``` net.ipv4.ip_forward = 1 这样系统就默认起用ip转发了.当然他不会立即生效.因为/etc/sysctl.conf 是在系统起动时读入的.想要立即生效的话,请使用sysctl命令. 2.安全性 /etc/sysctl.conf的默认权限是644,所有者和所有组均为root 3.See also sysctl(8) sysctl.conf(5) proc(5) procinfo(8) ``` ``` /etc/shells 包含系统可用的可能的“shell”的列表。 /etc/motd 每日消息；在管理员希望向 Linux 服务器的所有用户传达某个消息时使用。 ``` ##### 联网 ``` /etc/gated.conf gated 的配置。只能被 gated 守护进程所使用。 /etc/gated.version 包含 gated 守护进程的版本号。 /etc/gateway 由 routed 守护进程可选地使用。 ``` ``` /etc/networks ``` ##### 列举从机器所连接的网络可以访问的网络名和网络地址。 ##### 通过路由命令使用。允许使用网络名称。 ``` /etc/protocols ``` ##### 列举当前可用的协议。请参阅 NAG（网络管理员指南， ``` Network Administrators Guide）和联机帮助页。 C 接口 是 getprotoent。绝不能更改。 ``` ``` /etc/resolv.conf ``` ##### 在程序请求“解析”一个 IP 地址时告诉内核应该查询哪 ##### 个名称服务器。 ##### 1.文件格式 ``` /etc/resolv.conf是系统的DNS解析器配置文件，最常见 的用途是用来指定系统所使用的DNS服务器地址，您可以 最多指定MAXNS个 DNS服务器，MAXNS是一个常量，在 /usr/include/resolv.h中定义，一般为 3 。每 个DNS服务 器地址应该以点分十进制格式写在单独的行上，前面加上 关键字nameserver。例如： nameserver 173.26.100.99 nameserver 202.118.224.101 这里我们指定了两个DNS服务器，ip地址分别为 173.26.100.99和202.118.224.101。当系统需要进行DNS 解析时，优先使用列在前面的DNS Server，如果解析失败 则转而使用下一个DNS Server。 2.安全性 /etc/resolv.conf的默认权限为 0644 /etc/host.conf 1.文件格式 ``` ``` /etc/host.conf也是一个DNS解析器配置文件，但它最常 见的用途是用来指定解析器使用的方法。一般来说，DNS 解析可以使用两种方法，一是查询DNS服务器，二是使用 本地hosts主机表。/etc/host.conf可以用来指定优先使 用哪一种方法。可以使用order关键字来指定他们的优先 级.order后可跟一种或多种DNS查询方法，之间用逗号隔 开，其优先级依次降低。可用的DNS查询方法有： hosts,bind,nis，分别表示使用本地hosts主机表，DNS 服务器，NIS服务器来进行DNS查询。最常见的配置是： order bind,hosts 2.安全性 /etc/host.conf的默认权限为 0644 ``` ``` /etc/rpc ``` ##### 包含 RPC 指令／规则，这些指令／规则可以在 NFS 调用、 ##### 远程文件系统安装等中使用。 ``` /etc/exports 要导出的文件系统（NFS）和对它的权限。 ``` ``` /etc/services 将网络服务名转换为端口号／协议。由 inetd、telnet、 tcpdump 和一些其它程序读取。有一些 C 访问例程。 ``` ``` /etc/inetd.conf ``` ``` inetd 的配置文件。请参阅 inetd 联机帮助页。包含每个 网络服务的条目，inetd 必须为这些网络服务控制守护进 程或其它服务。注意，服务将会运行，但在 /etc/services 中将它们注释掉了，这样即使这些服务在运行也将不可用。 格式为： /etc/sendmail.cf 邮件程序 sendmail 的配置文件。比较隐晦，很难理解。 /etc/sysconfig/network 指出 NETWORKING=yes 或 no。至少由 rc.sysinit 读取。 /etc/sysconfig/network-scripts/if* Red Hat 网络配置脚本。 ``` ##### 系统命令 系统命令要独占地控制系统，并让一切正常工作。所有如 login（完成控制台用户身份验证阶段）或 bash（提供用户和计算机之间交互）之类的程序都是系统命令。因此，和它们有关的文件也特别重要。 这一类别中有下列令用户和管理员感兴趣的文件。 ``` /etc/lilo.conf ``` ##### 包含系统的缺省引导命令行参数，还有启动时使用的不同映象。您在 LILO ``` 引导提示的时候按 Tab 键就可以看到这个列表。 /etc/logrotate.conf 维护 /var/log 目录中的日志文件。 ``` ``` /etc/identd.conf ``` ``` identd 是一个服务器，它按照 RFC 1413 文档中指定的方式实现 TCP/IP 提议的标准 IDENT 用户身份识别协议。identd 的操作原理是查找特定 TCP/IP 连接并返回拥有此连接的进程的用户名。作为选择，它也可以返回 其它信息，而不是用户名。请参阅 identd 联机帮助页。 /etc/ld.so.conf “动态链接程序”（Dynamic Linker）的配置。 ``` ``` /etc/inittab ``` ##### 按年代来讲，这是 UNIX 中第一个配置文件。在一台 UNIX 机器打开之后启 ``` 动的第一个程序是 init，它知道该启动什么，这是由于 inittab 的存在。 ``` 在运行级别改变时，init 读取 inittab，然后控制主进程的启动。 1.文件格式 init进程将查看此文件来启动子进程,完成系统引导./etc/inittab描述了 一个进程是在系统引导时起动还是在系统引导完成后的某个情形下起动.他 也是由一行行的记录组成的.而以 # 开头的行是注释,将被忽略. 记录的格式是: id:runlevels:action:process id域是一个由 1 到 4 个字符组成的字符串,这个字符串必需是唯一的,即不 能有两条记录拥有相同的id域.id域是一个标志域,由它区分各条记录.注 意,对于gettys或者其他的login进程来说,id域必须是对应tty的tty后 缀,例如,对于tty1来说,id域应该是1.查看你的 /etc/inittab,会发现类 似下面这样的记录: ......... 1:2345:respawn:/sbin/mingetty tty1 2:2345:respawn:/sbin/mingetty tty2 3:2345:respawn:/sbin/mingetty tty3 ............. runlevels域是一个运行级的列表,可用的运行级有: 0 ---- 停机 1 ---- 单用户模式 2 ---- 不带NFS的多用户模式 3 ---- 完整的多用户模式 4 ---- 没有使用 5 ---- X11 6 ---- 重起系统 S ---- 单用户 s ---- 同S action域是一个预定义的动作,可用的action有: respawn 进程终止后立刻重新开始(如getty进程) wait 进程在进入指定的运行级后起动一次,然后init将等待它的终止 once 进程在进入指定的运行级后起动一次 boot 进程在系统引导时起动,runlevels域将被忽略 bootwait 进程在系统引导时起动,然后init将等待它的终止,runlevels域将被忽略 off 这个action不做任何事 ondemand 有一个特殊的运行级叫做ondemand runlevel,包括a,b和c.如果一个进程 被标记了ondemand runlevel,那么当要求切换到这个ondemand runlevel 时将会起动这个进程.但实际上的runlevel不会改变 initdefault 标记了initdefault这个action的记录项的runlevel域指定了系统引导完 成后进入的运行级 sysinit 在系统引导时起动这个进程.而且在所有的boot和bootwait项之前起 动.process域将被忽略 powerwait 在电力中断时起动这个进程.通常会由一个与连接到计算机的UPS系统对话 的进程通知init电力切断.init在继续之前将等待这个进程结束 powerfail 同powerwait类似,但是init不会等待这个进程结束 powerokwait 一旦init被通知电力已经恢复,将起动这个进程 powerfailnow 当init被告知UPS的电力亦将耗尽时起动这个进程 ctrlaltdel 当init接到SIGINT信号时起动这个进程.一般是按下了ctrl+alt+del这个 组合键 kbrequest 当一个特殊的键盘组合键被按下时起动这个进程 process域指定了将运行的进程,可以有参数.如果这个域以 + 开头,表明 init将为这个进程更新utmp/wtmp记录. 范例: id:3:initdefault: 系统引导完成后进入运行级 3 si::sysinit:/etc/rc.d/rc.sysinit 系统引导时运行/etc/rc.d/rc.sysinit l0:0:wait:/etc/rc.d/rc 0 系统进入运行级 0 时执行/etc/rc.d/rc 0,这里 0 是参数 ca::ctrlaltdel:/sbin/shutdown -t3 -r now 捕获到ctrl+alt+del时运行/sbin/shutdown -t3 - r now.如果想禁用 ctrl+alt+del这个组合键,直接删除或注释掉这行 pf::powerfail:/sbin/shutdown -f - h +2 "Power Failure; System Shutting Down" 电力中断时执行/sbin/shutdown -f - h +2 "Power Failure; System Shutting Down" 1:2345:respawn:/sbin/mingetty tty1 进入运行级2,3,4或 5 时执行respawn:/sbin/mingetty tty1.注意指定的 action是respawn,这也就是为什么我们在终端下logout后会立刻又出现一 个login提示符 x:5:respawn:/etc/X11/prefdm -nodaemon 进入运行级 5 时执行/etc/X11/prefdm -nodaemon,指定的action是respawm 2.安全性 /etc/inittab的权限是644,所有者和所有组均为root ##### 3.相关命令 ``` init telinit 更多内容请 man init man inittab ``` ``` /etc/termcap 一个数据库，包含所有可能的终端类型以及这些终端的性能。 ``` ##### 守护进程 ##### 守护进程是一种运行在非交互模式下的程序。一般来说，守护进程任务是和联网区域有关的：它们等 待连接，以便通过连接提供服务。Linux 可以使用从 Web 服务器到 ftp 服务器的很多守护进程。 ``` /etc/syslog.conf ``` ``` syslogd 是一种守护进程，它负责记录（写到磁盘）从其它程序发送到系统 的消息。这个服务尤其常被某些守护进程所使用，这些守护进程不会有另外 的方法来发出可能有问题存在的信号或向用户发送消息。 1.文件格式 /etc/syslog.conf是syslog守护程序的配置文件.syslog守护程序为记录 来自运行于系统之上的程序的消息提供了一种成熟的客户机 -服务器机制。 syslog 接收来自守护程序或程序的消息，根据优先级和类型将该消息分类， 然后根据由管理员可配置的规则将它写入日志。结果是一个健壮而统一的管 理日志的方法。 这个文件由一条条的规则组成.每条规则应该写在一行内.但是如果某行以 反斜线 \ 结尾的话,他的下个物理行将被认为与此行同属于一行.空白行和 以 # 开始的行将被忽略. 每条规则都是下面这种形式: facility.priority[;facility.priority .....] action facility和priority之间用一个英文句点分隔.他们的整体称为selector. 每条规则可以有多个selector,selector之间用分号隔开. 而selector和 action之间则用空格或者tab隔开. facility 指定 syslog 功能，主要包括以下这些： auth 由 pam_pwdb 报告的认证活动。 authpriv 包括特权信息如用户名在内的认证活动 cron 与 cron 和 at 有关的信息。 daemon 与 inetd 守护进程有关的信息。 kern 内核信息，首先通过 klogd 传递。 lpr 与打印服务有关的信息。 mail 与电子邮件有关的信息 mark syslog 内部功能用于生成时间戳 news 来自新闻服务器的信息 syslog 由 syslog 生成的信息 user 由用户程序生成的信息 uucp 由 uucp 生成的信息 local0----local7 与自定义程序使用，例如使用 local5 做为 ssh 功能 ``` * 通配符代表除了 ma rk 以外的所有功能 priority指定消息的优先级. 与每个功能对应的优先级是按一定顺序排列 的，emerg 是最高级，其次是 alert，依次类推。缺省时，在 /etc/syslog.conf 记录中指定的级别为该级别和更高级别。如果希望使用 确定的级别可以使用两个运算符号！(不等)和=。 user.=info 表示告知 syslog 接受所有在 info 级别上的 user 功能信息。 可用的 syslog 优先级如下: emerg 或 panic 该系统不可用 alert 需要立即被修改的条件 crit 阻止某些工具或子系统功能实现的错误条件 err 阻止工具或某些子系统部分功能实现的错误条件 warning 预警信息 notice 具有重要性的普通条件 info 提供信息的消息 debug 不包含函数条件或问题的其他信息 none 没有重要级，通常用于排错 * 所有级别，除了none action字段所表示的活动具有许多灵活性，特别是，可以使用名称管道的作 用是可以使 syslogd 生成后处理信息。 syslog 主要支持以下action file 指定文件的绝对路径,如: /var/log/messages. log信息将写到此文件中 terminal 或 printer 完全的串行或并行设备标志符,如/dev/console. log信息将送到此设备 @host 远程的日志服务器. log信息将送到此日志服务器 username 发送信息给指定用户 named pipe 指定使用 mkfifo 命令来创建的 FIFO 文件的绝对路径。 如果对此文件作了改动, 想要使改动生效，您需要向 syslog 守护程序通知 所做的更改。向它发送 SIGHUP 是个正确的办法，您可以用 killall 命令 轻松地做到这一点： # killall -HUP syslogd 2.安全性 您应该清楚如果 syslogd 写的日志文件还不存在的话，程序将创建它们。 无论您当前的 umask 如何设置，该文件将被创建为可被所有用户读取。如 果您关心安全性，那么您应该用 chmod 命令将该文件设置为仅 root 用户 可读写。此外，可以用适当的许可权配置 logrotate 程序（在下面描述） 以创建新的日志文件。syslog 守护程序始终会保留现有日志文件的当前属 性，因此一旦创建了文件，您就不需要担心它。 3.相关命令 logrotate ``` klogd syslogd dmesg ``` ``` /etc/httpd.conf ``` ``` Web 服务器 Apache 的配置文件。这个文件一般不在 /etc 中。它可能在 /usr/local/httpd/conf/ 或 /etc/httpd/conf/ 中，但是要确定它的位置， 您还需要检查特定的 Apache 安装信息。 /etc/conf.modules or /etc/modules.conf ``` ``` kerneld 的配置文件。有意思的是，kerneld 并不是“作为守护进程的”内 核。它其实是一种在需要时负责“快速”加载附加内核模块的守护进程。 ``` 用户程序 在 Linux（和一般的 UNIX）中，有无数的“用户”程序。最常见的一种用户程序配置文件是 /etc/lynx.cfg。这是著名的文本浏览器 lynx 的配置文件。通过这个文件，您可以定义代理服务器、 要使用的字符集等等。下面的代码样本展示了 lynx.cfg 文件的一部分，修改这部分代码可以改变 Linux 系统的代理服务器设置。缺省情况下，这些设置适用于在各自的 shell 中运行 lynx 的所有 用户，除非某个用户通过指定 --cfg = "mylynx.cfg" 重设了缺省的配置文件。 /etc/lynx.cfg 中的代理服务器设置 ``` .h1 proxy .h2 HTTP_PROXY .h2 HTTPS_PROXY .h2 FTP_PROXY .h2 GOPHER_PROXY .h2 NEWS_PROXY .h2 NNTP_PROXY # Lynx version 2.2 and beyond supports the use of proxy servers that can act as # firewall gateways and caching servers. They are preferable to the older # gateway servers. Each protocol used by Lynx can be mapped separately using # PROTOCOL_proxy environment variables (see Lynx Users Guide). If you have # not set them externally, you can set them at run time via this configuration file. # They will not override external settings. The no_proxy variable can be used # to inhibit proxying to selected regions of the Web (see below). Note that on # VMS these proxy variables are set as process logicals rather than symbols, to # preserve lowercasing, and will outlive the Lynx image. # .ex 15 http_proxy:http://proxy3.in.ibm.com:80/ ftp_proxy:http://proxy3.in.ibm.com:80/ #http_proxy:http://penguin.in.ibm.com:8080 #ftp_proxy:http://penguin.in.ibm.com:8080/ .h2 NO_PROXY # The no_proxy variable can be a comma-separated list of strings defining # no-proxy zones in the DNS domain name space. If a tail substring of the # domain-path for a host matches one of these strings, transactions with that ``` ``` # node will not be proxied. .ex no_proxy:demiurge.in.ibm.com, demiurge ``` ##### 更改配置文件 ##### 在更改配置文件时，如果程序不是由系统管理员或内核控制的，就要确保重新启动过使用该配置的程 ##### 序。普通用户通常没有启动或停止系统程序和／或守护进程的权限。 ##### 内核 更改内核中的配置文件会立即影响到系统。例如，更改 passwd 文件以增加用户将立即使该用户变为 可用。而且任何 Linux 系统的 /proc/sys 目录中都有一些内核可调参数。只有超级用户可以得到对 所有这些文件的写访问权力；其它用户只有只读访问权力。此目录中文件的分类的方式和 Linux 内 核源代码的分类方式一样。此目录中的每个文件都代表一个内核数据结构，这些数据结构可以被动态 地修改，从而改变系统性能。 注意：在更改其中任何文件的任何值之前，您应该确保自己全面了解该文件，以避免对系统造成不可 修复的损害。 /proc/sys/kernel/ 目录中的文件 ``` 文件名 描述 threads-max 内核可运行的最大任务数。 ``` ``` ctrl-alt-del ``` ##### 如果值为 1 ，那么顺序按下这几个键将“彻底地”重新引 ##### 导系统。 ``` sysrq 如果值为 1 ，Alt-SysRq 则为激活状态。 osrelease 显示操作系统的发行版版本号 ostype 显示操作系统的类型。 hostname 系统的主机名。 domainname 网络域，系统是该网络域的一部分。 ``` modprobe 指定 modprobe 是否应该在启动时自动运行并加载必需 的模块。 守护进程和系统程序 守护进程是永远运行在后台的程序，它默默地执行自己的任务。常见的守护进程有 in.ftpd（ftp 服 务器守护进程）、in.telnetd（telnet 服务器守护进程）和 syslogd（系统日志记录守护进程）。 有些守护进程在运行时会严密监视配置文件，在配置文件改变时就会自动重新加载它。但是大多数守 护进程并不会自动重新加载配置文件。我们需要以某种方式“告诉”这些守护进程配置文件已经被发 生了改变并应该重新加载。可以通过使用服务命令重新启动服务来达到这个目的（在 Red Hat Linux 系统上）。 例如，如果我们更改了网络配置，就需要发出： service network restart 。 注意：这些服务最常见的是 /etc/rc.d/init.d/* 目录中存在的脚本，在系统被引导时由 init 启动。 所以，您也可以执行如下操作来重新启动服务： /etc/rc.d/init.d/ start | stop | status start、stop 和 status 是这些脚本接受的输入值，用来执行操作。 用户程序 ##### 用户或系统程序在每次启动时都会读取其配置文件。尽管如此，请记住，有些系统程序在计算机打开 时情况不一样，它们的行为依赖于在 /etc/ 中的配置文件中读到的内容。所以，用户程序第一次启 动时将从 /etc/ 目录中存在的文件读取缺省配置。然后，用户可以通过使用 rc 和 .（点）文件来 定制程序，正如下面一节所示。 用户配置文件：.（点）文件和 rc 文件 我们已经看到怎样容易地配置程序。但是如果有的人不喜欢在 /etc/ 中配置程序的方式该怎么办呢？ “普通”用户不能简单地进入 /etc 然后更改配置文件；从文件系统的角度来看，配置文件的所有者 是 root 用户！这就是大多数用户程序都定义两个配置文件的原因：第一个是“系统”级别的，位于 /etc/；另一个属于用户“专用”，可以在他或她的主目录中找到。 例如，我在我的系统中安装了非常有用的 wget 实用程序。/etc/ 中有一个 /etc/wgetrc 文件。在 我的主目录中，有一个名为 .wgetrc 的文件，它描述了我定制的配置（只有在我，也就是用户运行 wget 命令时，才会加载这个配置文件）。其它用户在他们自己的主目录（/home/other）中也可以 有 .wgetrc 文件；当然，只有这些用户运行 wget 命令时，才会读取这个文件。换句话说，/etc/wgetrc 文件为 wget 提供了“缺省”值，而 /home/xxx/.wgetrc 文件列举了某个用户的“定制项”。重要 的是这只是“一般规则”，并非所有情况都如此。例如，一个象 pine 一样的程序，在 /etc/ 中并 没有任何文件，它只在用户主目录中有一个定制配置文件，名为 .pinerc。其它程序可能只有 /etc/ 中的缺省配置文件，而且可能不允许用户“定制”这些配置文件（/etc 目录中只有少数 config. 文 件是这种情况）。 通常使用的 rc 和 .（点）文件 文件名 描述 ``` ~/.bash_login ``` ``` 请参考“man bash”。如果 ~/.bash_profile 不存在，bash 则将 ~/.bash_login 作为 ~/.bash_profile 处理。 ``` ``` ~/.bash_logout 请参考“man bash”。在退出时由 bash 登录 shell 引用。 ``` ``` ~/.bash_profile 由 bash 登录 shell 引用^ /etc/profile 之后引用。 ~/.bash_history 先前执行的命令的列表。 ``` ``` ~/.bashrc ``` ``` 请参考“man bash”。由 bash 非登 录交互式 shell 引用（没有其它文 件）。除非设置了 BASH_ENV 或 ENV， 非交互式 shell 不引用任何文件。 ~/.emacs 启动时由 emac 读取。 ``` ``` ~/.forward ``` ##### 如果这里包含一个电子邮件地址，那 ##### 么所有发往 ~ 的所有者的邮件都会 ##### 被转发到这个电子邮件地址。 ``` ~/.fvwmrc ~/.fvwm2rc fvwm 和 fvwm2（基本的 X Window 管 理器）的配置文件。 ``` ``` ~/.hushlogin ``` ``` 请参考“man login”。引起“无提 示”登录（没有邮件通知、上次登录 信息或者 MOD 信息）。 ``` ~/.mail.rc 邮件程序的用户初始化文件。 ~/.ncftp/ ``` ncftp 程序的目录；包含书签、日志、 宏、首选项和跟踪信息。请参阅 man ncftp。ncftp 的目的是为因特网标准 文件传输协议（Internet standard File Transfer Protocol）提供一个 强大而灵活的接口。它旨在替换系统 所使用的标准的 ftp 程序。 ``` ~/.profile ``` 请参考“man bash”。如果 ~/.bash_profile 和 ~/.bash_login 文件不存在，bash 则将 ~/.profile 作为 ~/.bash_profile 处理，并被其 它继承 Bourn 的 shell 使用。 ``` ~/.pinerc Pine 配置 ~/.muttrc Mutt 配置 ~/.exrc ``` 这个文件可以控制 vi 的配置。 示例：set ai sm ruler 在此文件中写入上面一行会让 vi 设 置自动缩进、匹配括号、显示行号和 行-列这几个选项。 ``` ~/.vimrc 缺省的“Vim” 配置文件。和 .exrc 一样。 ~/.gtkrc GNOME 工具包（GNOME Toolkit）。 ~/.kderc KDE 配置。 ~/.netrc ftp 缺省登录名和密码。 ~/.rhosts ``` 由 r- 工具（如 rsh、rlogin 等等） 使用。因为冒充主机很容易，所以安 全性非常低。 ``` 1. 必须由用户（~/ 的所有者）或超 级用户拥有。 2. 列出一些主机，用户可以从这些主 机访问该帐号。 3. 如果是符号链接则被忽略。 ~/.rpmrc 请参阅“man rpm”。如 果 /etc/rpmrc 不存在则由 rpm 读取。 ~/.signature ##### 消息文本，将自动附加在从此帐号发 ##### 出的邮件末尾。 ~/.twmrc twm（ The Window Manager）的配置 文件。 ``` ~/.xinitrc ``` ``` 启动时由 X 读取（而不是由 xinit 脚本读取）。通常会启动一些程序。 示例：exec /usr/sbin/startkde 如果该文件中存在上面这行内容，那 么在从这个帐号发出 startx 命令 时，这一行就会启动“KDE 视窗管理 器”（KDE Window Manager）。 ``` ``` ~/.xmodmaprc ``` ``` 此文件被传送到 xmodmap 程序，而且 可以被命名为任何文件（例如 ~/.Xmodmap 和 ~/.keymap.km）。 ``` ``` ~/.xserverrc ``` ``` 如果 xinit 可以找到要执行的 X， xinit 就会将该文件作为 X 服务器 运行。 ~/News/Sent-Message-IDs gnus 的缺省邮件历史文件。 ``` ``` ~/.Xauthority 由 xdm 程序读和写，以处理权限。请 参阅 X、xdm 和 xauth 联机帮助页。 ``` ``` ~/.Xdefaults, ~/.Xdefaults-hostname ``` ``` 在主机 hostname 的启动过程中由 X 应用程序读取。如果找不到 -hostname 文件，则查找 .Xdefaults 文件。 ``` ``` ~/.Xmodmap 指向 .xmodmaprc；Red Hat 有使用这 个名称的 .xinitrc 文件。 ``` ``` ~/.Xresources ``` ``` 通常是传送到 xrdb 以加载 X 资源 数据库的文件的名称，旨在避免应用 程序需要读取一个很长 的 .Xdefaults 文件这样的情况。（有 些情况曾经使用了 ~/.Xres。） ~/mbox 用户的旧邮件。 ``` #### 3 、重要的配置文件列表： ##### 启动引导程序配置文件 ``` LILO /etc/lilo.conf GRUB /boot/grub/menu.lst 系统启动文件核脚本 主启动控制文件 /etc/inittab SysV启动脚本的位置 /etc/init.d、/etc/rc.d/init.d或/etc/rc.d SysV启动脚本链接的位置 /etc/init.d/rc?.d、/etc/rc.d/rc?.d或/etc/rc?.d 本地启动脚本 /etc/rc.d/rc.local、/etc/init.d/boot.local或/etc/rc.boot里的文件 网络配置文件 建立网络接口的脚本 /sbin/ifup 保存网络配置数据文件的目录 /etc/network、/etc/sysconfig/network和 /etc/sysconfig/network-scripts ``` 保存解析DNS服务的文件 /etc/resolv.conf DHCP客户端的配置文件 /etc/dhclient.conf 超级服务程序配置文件和目录 inetd配置文件 /etc/inetd.conf TCP Wrappers配置文件 /etc/hosts.allow和/etc/hosts.deny xinetd配置文件 /etc/xinetd.conf和/etc/xinetd.d目录里的文件 硬件配置 内核模块配置文件 /etc/modules.conf 硬件访问文件 Linux设备文件 /dev目录里 保存硬件和驱动程序数据的文件 /proc目录里 扫描仪配置文件 SANE主配置 /etc/sane.d/dll.conf 特定扫描仪的配置文件 /etc/sane.d目录里以扫描仪型号命名的文件 打印机配置文件 BSD LPD核LPRng的本地打印机主配置文件 /etc/printcap CUPS本地打印机主配置和远程访问受权文件 /etc/cups/cupsd.conf BSD LPD远程访问受权文件 /etc/hosts.lpd LPRng远程访问受权文件 /etc/lpd.perms 文件系统 文件系统表 /etc/fstab 软驱装配点 /floppy、/mnt/floppy或/media/floppy 光驱装配点 /cdrom、/mnt/cdrom或/media/cdrom shell配置文件 bash系统非登录配置文件 /etc/bashrc、/etc/bash.bashrc或/etc/bash.bashrc.local bash系统登录文件 /etc/profile和/etc/profile.d里的文件 bash用户非登录配置文件 ~/.bashrc bash用户登录配置文件 ~/.profile XFree86配置文件核目录 XFree86主配置文件 /etc/XF86config、/etc/X11/XF86Config或/etc/X11/XF86Config-4 字体服务程序配置文件 /etc/X11/fs/config Xft 1.x配置文件 /etcX11/XftConfig Xft 2.0配置文件 /etc/fonts/fonts.conf 字体目录 /usr/X11R6/lib/X11/fonts和/usr/share/fonts Web服务程序配置文件 Apache主配置文件 /etc/apache、/etc/httpd或/httpd/conf里的httpd.conf或httpd2.conf文 件 MIME类型文件 与Apache主配置文件在同一目录里的mime.types或apache-mime.types 文件服务程序配置文件 ProFTPd配置文件 /etc/proftpd.conf vsftpd配置文件 /etc/vsftpd.conf NFS服务程序的输出定义文件 /etc/exports NFS客户端装配的NFS输出 /etc/fstab Samba配置文件 /etc/samba/smb.conf ``` Samba用户配置文件 /etc/samba/smbpasswd 邮件服务程序配置文件 sendmail主配置文件 /etc/mail/sendmail.cf sendmail源配置文件 /etc/mail/sendmail.mc或/usr/share/sendmail/cf/cf/linux.smtp.mc或 其他文件 Postfix主配置文件 /etc/postfix/main.cf Exim主配置文件 /etc/exim/exim.cf Procmail配置文件 /etc/procmailrc或~/.procmailrc Fetchmail配置文件 ~/.fetchmailrc 远程登录配置文件 SSH服务程序配置文件 /etc/ssh/sshd_config SSH客户端配置文件 /etc/ssh/ssh_config XDM配置文件 /etc/X11/xdm目录下 GDM配置文件 /etc/X11/gdm目录下 VNC服务程序配置文件 /usr/X11R6/bin/vncserver启动脚本和~/.vnc目录里的文件 其他服务程序配置文件 DHCP服务程序配置文件 /etc/dhcpd.conf BIND服务程序配置文件 /etc/named.conf和/var/named/ NTP服务程序配置文件 /etc/ntp.conf ``` ## 十六、 计划任务 ##### 在很多时候为了自动化管理系统，我们都会用到计划任务，比如关机，管理，备份之类的操作，我 ##### 们都可以使用计划任务来完成，这样可以是管理员的工作量大大降低，而且可靠度更好。 linux系统支持一些能够自动执行任务的服务，我们称为计划任务。 LINUX有如下三种计划任务： at：指定一个时间执行一个任务 （适用一个或多个任务，执行一次后就不用） cron：根据一个时间表自动执行任务 （使用一个或多个任务，周期性执行） 系统级别的计划任务及其扩展anacron：在一个指定时间间隔错过后自动执行任务 #### 1 、 at: 安排一个任务在未来执行，需要一个 atd 的系统后台进程 检查 **atd** 进程是否启动 [root@centos61 桌面]# service atd status atd (pid 2274) 正在运行... [root@centos61 桌面]# chkconfig |grep atd atd 0:关闭 1:关闭 2:关闭 3:启用 4:启用 5:启用 6:关闭 如果未启动，可以使用如下命令： [root@centos61 桌面]# service atd start 正在启动 atd： [确定] [root@centos61 桌面]# chkconfig atd on 常用指令: at:安排延时任务 具体使用方法: 例1: #at now+2 minutes 回车 >输入要执行的命令 >ctrl+d 结束输入 [root@test ~]# at now+2 minutes at> wall Aixi at> job 2 at 2010-06-18 16:36 是ctrl+d中断输入，这个命令意思是发送一个广播内容是Hello Aixi.具体时间可以改,单位可 以改,可以用hours,months,years,weeks等. 例 2 我们还可以跟具体时间 [root@test ~]# at 16:39 dec 10 at> Hello Aixi at> ctrl+d结束输入 job 3 at 2010-12-10 16:39 意思是在今年的 12 月 10 日16:39运行这个命令.如果不加月和日,默认就是今天. Atq:查询当前的等待任务 用atq来查询，已经运行的任务，就消失了。这就是at计划任务的重点，只运行一次 atrm：删除等待任务 启动计划任务后，如果不想启动设定好的计划任务可以使用atrm命令删除。 格式：atrm 任务号 命令后面跟计划任务编号，如果不跟，就会删除这个用户所有的计划任务。 例 3 atrm 10 //删除计划任务 10 atq //查看计划任务是否删除 at将要运行的命令以文本形式写入/var/spool/at/目录内，等待atd服务的取用和执行。 还可以进入到/var/spool/at目录里把计划任务删除，计划任务的文件都保存在该目录里，可以用rm -f 文件名来删除(以文件的形式删除计划任务，因为计划任务是以文件形式保存在该目录中) 例4: #cd /var/spool/at //进入到/var/spool/at目录中 ls //显示目录中所有文件 rm -f a0000b0138b19c //删除计划任务 在通常情况下，超级用户都可以使用这个命令。对于其他用户来说，能否可以使用就取决于两个文 件:/etc/at.allow和/etc/at.deny。 at 命令是可以基于用户来控制的，我们可以明确指定哪些用户可以使用at计划任务，哪些用户不可以 使用at计划任务。 at的控制文件 /etc/at.allow /etc/at.deny 系统默认是有at.deny文件，如果某个用户名在这个文件里，他就不能使用at计划任务。如果有 at.allow文件，allow文件先行，检查了allow明确允许，就不会检查deny。 如果你要让哪个用户不能使用计划任务，就直接把他的用户名写进去就可以了，一排只能写一个。 #### 2 、 cron 服务 ( 参考网址： http://www.linuxsir.org/main/?q=node/209) 相对与at，cron的优点就是能够周期性的执行某个命令,at却只能执行一次，cron的后台进程名字 是crond ,cron也是system V的服务，所以我们可以service crond start|stop 来启动和关闭此服 务,也可以使用chkconfig或者ntsysv来选择cron服务的默认开启，这些命令在以前我们都讲过的 命令： #crontab -e 编辑当前用户的cron表 #crontab -l 查看当前用户的cron表 #crontab -r 删除当前用户的cron进程 #crontab -u 用户名 以某用户的身份来控制cron表 还有个重要的知识点，就是当用户的计划任务建立后是存放在var/spool/cron这个目录 当使用crontab -e编辑当前用户的cron表后，会出现一个vi文件，cron的格式是这样的。分成两 列，左边是时间，右边是运行的命令。时间是由 5 个部分组成。 例： * * * * * wall hello everyone 5 个星号分别代表：minute hour day-of- month month-of-year day-of- week ，而wall hello everyone 这是命令内容。上面的意识是每分每小时每天每月每周广播hello everyone。具体时间大 家可以自己定义。如果要每两分钟发送就用*/2代替第一个*。也可以是用具体时间来表示。 我们使用crontab -e编辑当前用户的cron表 这里的 5 个星号就代表的时间和日期： 第一个*星号代表个小时的第几分钟：minute 范围是从0-59 第二个*星号代表每天的第几个小时：hour 范围是从0-23 第三个*星号代表每月的第几个日：day -of -month 范围从1-31 第四个*星号代表没年的第几个月：month-of-year 范围从1-12 第五个*星号代表每周的星期几：day-of-week 范围从0-6，其中 0 表示星期日 用户名：也就是执行程序要通过哪个用户来执行，这个一般可以省略； 命令：执行的命令和参数。 时程表的格式如下 : f1 f2 f3 f4 f5 program 其中 f1 是表示分钟，f2 表示小时，f3 表示一个月份中的第几日，f4 表示月份，f5 表示一个星期 中的第几天。program 表 示要执行的程序。 当 f1 为 * 时表示每分钟都要执行 program，f2 为 * 时表示每小时都要执行程序，其余以此类推 当 f1 为 a -b 时表示从第 a 分钟到第 b 分钟这段时间内要执行，f2 为 a -b 时表示从第 a 到第 b 小时都要执行，其余以此类推 当 f1 为 */n 时表示每 n 分钟个时间间隔执行一次，f2 为 */n 表示每 n 小时个时间间隔执行一 次，其余以此类推 当 f1 为 a, b, c,... 时表示第 a, b, c,... 分钟要执行，f2 为 a, b, c,... 时表示第 a, b, c... 个小时要执行，其余以此类推 使用者也可以将所有的设定先存放在档案 file 中，用 cr ontab file 的方式来设定时程表。 例 1 ：如果我要表示 9 月 10 日 25 分执行ls var/spool/cron任务怎么表示？ 25 * 10 9 * ls var/spool/cron 由于我没表示小时，所以就只能里面为每小时 例 2 ：我要在每周日，每分钟执行wall Hello redking.blog.51cto.com这个命令，时间怎么表示？ */1 * * * 0 wall Hello redking.blog.51cto.com */ 表示每多少分钟执行一次 例 3 ： 每晚的21:30重启apache。 30 21 * * * /usr/local/etc/rc.d/lighttpd restart 例 4 ：每月 1 、 10 、 22 日的4 : 45重启apache 45 4 1,10,22 * * /usr/local/etc/rc.d/lighttpd restart 例 5 ：每周六、周日的1 : 10重启apache 10 1 * * 6,0 /usr/local/etc/rc.d/lighttpd restart 例 6 ：在每天18 : 00至23 : 00之间每隔 30 分钟重启apache 0,30 18-23 * * * /usr/local/etc/rc.d/lighttpd restart 例 7 ：每一小时重启apache * */1 * * * /usr/local/etc/rc.d/lighttpd restart 例 8 ：晚上 11 点到早上 7 点之间，每隔一小时重启apache * 23-7/1 * * * /usr/local/etc/rc.d/lighttpd restart 例 9 ：每月的 4 号与每周一到周三的 11 点重启apache 0 11 4 * mon-wed /usr/local/etc/rc.d/lighttpd restart 例 10 ：一月一号的 4 点重启apache 0 4 1 jan * /usr/local/etc/rc.d/lighttpd restart 例 11 ：在 12 月内, 每天的早上 6 点到 12 点中，每隔 3 个小时执行一次 /usr/bin/backup 0 6-12/3 * 12 * /usr/bin/backup 例 12 ：每月每天的午夜 0 点 20 分, 2 点 20 分, 4 点 20 分....执行 echo "haha" 20 0-23/2 * * * echo "haha" #### 3 、系统级别的计划任务及其扩展 anacrontab ##### 这个是系统设置好了，清理系统垃圾或者是自动执行某些脚本的系统任务，一般我们做了解就行了， 不要更改配置文件是/etc/conrtab SHELL：就是运行计划任务的解释器，默认是bash PATH：执行命令的环境变量 MAILTO：计划任务的出发者用户 HOME：家目录为/ run-parts是一个脚本，在/usr/bin/run-parts，作用是执行一个目录下的所有脚本/程序。 run-parts /etc/cron.hourly执行目录/etc/cron.hourly/之下的所有脚本/程序. run-parts下面就是运行的命令 vim /etc/crontab 与crontab -e写入的定时运行的区别? vim /etc/crontab：是系统级别定义的crontab，/etc/crontab的所有者和组都是root crontab -e ：是用户自定义的crontab，是所有的用户都可以写入的 ``` 两种方法记录的位置不一样，一个在/etc/ 另一个在/var/ 里面。都被cron服务调用 如果系统在以上说的时间没有开机怎么办？那么这个脚本不就是不能执行了？设计者早就想到了这个 问题，所以就有了cron服务的扩展,目的就是为了防止非 24 小时开机的计算机遗漏的守护任务,anacr ontab就是系统计划任务的扩展文件：在一个指定时间间隔错过后自动执行任务 格式是这样的： period delay job -identifier command period — 命令执行的频率（天数） delay — 延迟时间（分钟） job-identifier — 任务的描述，用在 anacron 的消息中，并作为作业时间戳文件的名称，只能包括 非空白的字符（除斜线外）。 command — 要执行的命令 ``` ``` 第一行的意思是：每天开机 65 分钟后就检查cron.daily文件是否被执行了，如果今天没有被执行就执 行他 第二行的意思是：每隔 7 天开机后 70 分钟检查cron.weekly文件是否被执行了,如果一周内没有被执行 就执行他 第三行的意思也差不多 下面说说关于cron服务的控制，和at差不多，就是/etc/cron.deny这个配置文件来控制，里面写入 要禁止使用cron用户的名字，一行一个就OK了 ``` ## 十七、 VI/VIM编辑器 ##### 常用快捷键： Ctrl+f 向下翻页 Ctrl+b 向上翻页 G 移动到文件最后一行 gg 移动到文件第一行 N+回车 N为数字，向下移到到N行 /关键字 向下寻找关键字 ?关键字 向上寻找关键字 # 从光标向后查找光标所在 关键词 * 从光标向前查找光标所在 关键词 n 向下重复上一次查找操作 N 与n相反，反向重复上一次查找操作 :n1,n2s/关键字1/关键字2/g 从第n1与n2行之间寻找关键字 1 ，并将关键字 1 替换为关键字 2 :1,$s/关键字1/关键字2/g 从第 1 行到最后一行寻找关键字 1 ，并将关键字 1 替换为关键字 2 :1,$s/关键字1/关键字2/gc 从第 1 行到最后一行寻找关键字 1 ，将关键字 1 替换为关键字 2 前会提示用 户确认是否替换 dd 删除整行 ndd n为数字，删除光标所在向下n行。 yy 复制光标所在行 nyy n为数字，复制光标所在向下n行 p,P 小p将复制的数据在光标下一行粘贴，大P将复制的数据在光标上一行粘贴 u 撤消前一个操作 Ctrl+r 重做上一个操作 . 将会重复上一个命令 i：在当前字符的左边插入 I：在当前行首插入 a：在当前字符的右边插入 A：在当前行尾插入 o：在当前行下面插入一个新行 O：在当前行上面插入一个新行 :w 保存数据 :wq 保存退出 :q! 不保存退出 :w文件名 相当于另存为 ## 十八、 压缩打包 linux下的压缩 命令有tar、gzip、gunzip、bzip2、bunzip2、 compress、uncompress、zip、unzip、rar、 unrar等等 ，压缩后的扩展名有.tar、.gz、. t a r. g z、. tgz、.bz2、.tar.bz2、.Z、.tar.Z、.zip、.rar 10种。 对应关系如下： 1 、*.tar 用 tar –xvf 解压 2 、*.gz 用 gzip -d或者gunzip 解压 3 、*. t a r. g z和*.tgz 用 tar –xzf 解压 4 、*.bz2 用 bzip2 -d或者用bunzip2 解压 5 、*.tar.bz2用tar –xjf 解压 6 、*.Z 用 uncompress 解压 7 、*.tar.Z 用tar –xZf 解压 8 、*.rar 用 unrar e解压 9 、*.zip 用 unzip 解压 在介绍压缩文件之前呢，首先要弄清两个概念：打包和压缩。打包是指将一大堆文件或目录什么的变成 一个总的文件，压缩则是将一个大的文件通过一些压 缩算法变成一个小文件。为什么要区分这两个概念呢? 其实这源于Linux中的很多压缩程序只能针对一个文件进行压缩，这样当你想要压缩一大堆文件时，你就得 先借助另它的工具将这一大堆文件先打成一个包，然后再就原来的压缩程序进行压缩。 #### Gzip/zcat #### Bzip2/bzcat #### Tar Linux下最常用的打包程序就是tar了，使用tar程序打出来的包我们常称为tar包，tar包文件的命令通 常都是以.tar结尾的。生成tar包后，就可以用其它的程序来进行压缩了，所以首先就来讲讲tar命令的基本 用法： tar命令的选项有很多(用man tar可以查看到)，但常用的就那么几个选项，下面来举例说明一下： **# tar -cf all.tar *.jpg** 这条命令是将所有.jpg的文件打成一个名为all.tar的包。-c是表示产生新的包，-f指 定包的文件名。 **# tar -rf all.tar *.gif** 这条命令是将所有.gif的文件增加到all.tar的包里面去。-r是表示增加文件的意思。 **# tar -uf all.tar logo.gif** 这条命令是更新原来tar包all.tar中logo.gif文件，-u是表示更新文件的意思。 **# tar -tf all.tar** 这条命令是列出all.tar包中所有文件，-t是列出文件的意思 **# tar -xf all.tar** 这条命令是解出all.tar包中所有文件，-t是解开的意思 以上就是tar的最基本的用法。为了方便用户在打包解包的同时可以压缩或解压文件，tar提供了一种特 殊的功能。这就是tar可以在打包或解包的同时调用其它的压缩程序，比如调用gzip、bzip2等。 **1) tar** 调用 **gzip** gzip是GNU组织开发的一个压缩程序，.gz结尾的文件就是gzip压缩的结果。与gzip相对的解压程序是 gunzip。tar中使用-z这个参数来调用gzip。下面来举例说明一下： **# tar -czf all.tar.gz *.jpg** 这条命令是将所有.jpg的文件打成一个tar包，并且将其用gzip压缩，生成一个 gzip压缩过的包，包名为all.tar.gz **# tar -xzf all.tar.gz** 这条命令是将上面产生的包解开。 **2) tar** 调用 **bzip2** bzip2是一个压缩能力更强的压缩程序，.bz2结尾的文件就是bzip2压缩的结果。与bzip2相对的解压程 序是bunzip2。tar中使用-j这个参数来调用gzip。下面来举例说明一下： **# tar -cjf all.tar.bz2 *.jpg** 这条命令是将所有.jpg的文件打成一个tar包，并且将其用bzip2压缩，生成一个 bzip2压缩过的包，包名为all.tar.bz2 **# tar -xjf all.tar.bz2** 这条命令是将上面产生的包解开。 下面对于tar系列的压缩文件作一个小结： 1)对于.tar结尾的文件 tar -xf all.tar 2)对于.gz结尾的文件 gzip -d all.gz gunzip all.gz 3)对于.tgz或. t a r. g z结尾的文件 tar -xzf all.tar.gz tar -xzf all.tgz 4)对于.bz2结尾的文件 bzip2 -d all.bz2 bunzip2 all.bz2 5)对于tar.bz2结尾的文件 tar -xjf all.tar.bz2 6)对于.Z结尾的文件 uncompress all.Z 7)对于.tar.Z结尾的文件 tar -xZf all.tar.z #### Cpio Unzip：解压zip Gnuzip：解压bz2 ## 十九、 性能优化 ##### 1 、 设置文件夹打开方式 ##### 2 、 设置屏幕保护时间 ##### 3 、 解除上网限制 ##### 4 、 ## 二十、 常见问题 #### 部分网站无法访问问题的解决 CentOS 5内核对TCP的读缓冲区大小有缺省设置，缺省为：net.ipv4.tcp_rmem = 4096 87380 4194304 解决办法就是将最后一个数字改小一点，具体操作就是在文件/etc/sysctl.conf中添加一行： net.ipv4.tcp_rmem = 4096 87380 174760 然后保存 重新启动网络service network restart,就OK了，如果还是部分网站上不去，可以检查/etc/sysctl.conf文件是 否和下面相同 net.ipv4.ip_local_port_range = 1024 65536 net.core.rmem_max= 174760 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 174760 net.ipv4.tcp_wmem=4096 65536 16777216 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_tw_recycle = 1 net.core.netdev_max_backlog = 30000 net.ipv4.tcp_no_metrics_save=1 net.core.somaxconn = 262144 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_orphans = 8000 net.ipv4.tcp_max_syn_backlog = 8000 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 net.ipv4.tcp_wmem=4096 65536 16777216 ：为自动调优定义每个 socket 使用的内存。第一个值 4096 是为socket 的发送缓冲区分配的最少字节数。第二个值 65536 是默认值（该值会被 wmem_default 覆 盖 ）， 缓冲区在系统负载不重的情况下可以增长到这个值。第三个值 16777216 是发送缓冲区空间的最大字节数（该 值会被wmem_max 覆盖） net.ipv4.tcp_rmem=4096 87380 174760：与 tcp_wmem 类似，不过它表示的是为自动调优所使用的接收 缓冲区的值。 net.core.rmem_max = 25165824 #定义最大的TCP/IP栈的接收窗口大小 net.core.rmem_default = 25165824 #定义默认的TCP/IP栈的接收窗口大小 net.core.wmem_max = 25165824 #定义最大的TCP/IP栈的发送窗口大小 net.core.wmem_default = 65536 #定义默认的TCP/IP栈的发送窗口大小 net.ipv4.tcp_sack =1 #启用有选择的应答（Selective Acknowledgment），这可以通过有 选择地应答乱序接收到的报文来提高性能（这样可以让发送者只发送丢失的报文段）；（对于广域网通信来说） 这个选项应该启用，但是这会增加对 CPU 的占用。 net.ipv4.tcp_window_scaling = 1 #启用RFC1323定义，支持超过64K窗口 net.ipv4.tcp_fack =1 #启用转发应答（Forward Acknowledgment），这可以进行有选择 应答（SACK）从而减少拥塞情况的发生；这个选项也应该启用。 net.ipv4.tcp_mem 24576 32768 49152 确定 TCP 栈应该如何反映内存使用；每个值的单位都是内存页 （通常是 4KB）。第一个值是内存使用的下限。第二个值是内存压力模式开始对缓冲区使用应用压力的上限。 第三个值是内存上限。在这个层次上可以将报文丢弃，从而减少对内存的使用。对于较大的 BDP 可以增大 这些值（但是要记住，其单位是内存页，而不是字节）。 #### Centos5 无法连接无线网络 系统－>管理->服务器设置->服务，将NetworkManager选项勾选，点击重启服务。然后就可以看到右上 角已经有了网络连接。 #### Linux 远程管理 Windows 程序 Rdesktop 详解 #rpm –q rdesktop //查找是否已经安装 #yum install rdesktop //使用yum安装 rdesktop 使用简单，windows也不和装什么服务端，是要把远程桌面共享打开就行了 具体使用方法要先打开终端，然后输入以下命令： **rdesktop -u yournape -p password -g 1024*720 192.168.0.2** rdesktop为使用远程桌面连接的命令； -u 用户名，yourname处为目标客户端的用户名； -p 客户端用户的密码； -g 指定使用屏幕大小-g 800*600+0+0这 个‘ ＋ 0 ’就是，就是你这个窗口的在你linux上出现的位置； 192.168.0.1 目标客户端的IP地址 实例： **[root@Centos5 ~]# rdesktop -u aixi -p d337448 -r clipboard:PRIMARYCLIPBOARD -r disk:centos=/root -r sound:local -z -a 16 10.26.11.72** $rdesktop 192.168.1.1 //打开了一个 8 位色彩的， $rdesktop -a 16 192.168.1.1 //这个是 16 位色彩的了，看起来好多了 $rdesktop -u administrator -p ****** -a 16 192.168.1.1 //都直接登陆了 $rdesktop -u administrator -p ****** -a 16 -r sound:local 192.168.1.1 加上-r sound:local可以把声 音也搞过来，-r 的作用挺多的可以重定向许多东西，看一下帮助就会收获不 少了。 -r comport:COM1=/dev/ttyS0 // 将串口 /dev/ttyS0 重定向为 COM1 -r comport:COM1=/dev/ttyS0,COM2=/dev/ttyS1 // 多个串口重定向 -r disk:floppy=/mnt/floppy // 将 /mnt/floppy 重定向为远程共享磁盘 'floppy' -r disk:floppy=/mnt/floppy,cdrom=/mnt/cdrom,root=/,c=/mnt/c // 多个磁盘重定向 -r clientname= // 为重定向的磁盘设置显示的客户端名称 -r lptport:LPT1=/dev/lp0 // 将并口 /dev/lp0 重定向为 LPT1 -r lptport:LPT1=/dev/lp0,LPT2=/dev/lp1 // 多个并口重定向 -r printer:mydeskjet // 打印机重定向 -r printer:mydeskjet="HP LaserJet IIIP" // 打印机重定向 -r sound:[local|off|remote] // 声音重定向 -r clipboard:PRIMARYCLIPBOARD : 这个一定要加上，要不然不能在主机Solaris和服务器Windows直接复 制粘贴文字了。贴中文也没有问题。 -r disk:sunway=/home/jianjian : 指定主机Solaris上的一个目录(/home/jianjian)映射到远程Windows上的 硬盘(盘符为sunway)，传送文件就不用再靠Samba或者FTP了。 -f ：全屏， 退出全屏：ctrl＋alt＋enter再次Ctrl+Alt+Enter即可再次进入全屏 -D：不显示标题栏，配合 -g 能更好地使用屏幕空间了； -K： 这个选项说明保持窗口管理器的按键组合绑定； -z：启动网络数据的压缩，减少带宽，局域网没什么作用； 提示：如果你的本地中文文件名在远程机器上显示为乱码的话，可能是你没有安装编码转化库，或者你 安装的编码转化库不能正确运行。 #### Linux 远程访问 Windows 共享目录 #mount –o username=用户名 –password=密码 //192.168.0.1/C$ /tmp/samba/ [root@Centos5 ~]# mount -o username=aixi,password=d337448 //10.26.11.72/d$ /root/aixi/ 说明：IP地址192.168.0.1为中文名文件所在的主机，文件位于C盘，该主机的用户名及密码为linux， /tmp/samba/为本地主机挂载目录。在浏览完成后，使用以下命令卸载。 #umount /tmp/samba/ #### 升级或安装程序后无法进入图形界面 ##### 报错如下： ``` Failed to start the X server (your graphical interface). lt is likely that it is not set up correctly. Would you like to view the X server output to diagnose the problem? 解决办法： #cat /var/log/Xorg.0.log | grep EE 查看报错日志 #sh NVDIA 重新安装显卡驱动 参考如下网址： http://www.linuxquestions.org/questions/linux-hardware-18/failed-to-start-the-x-server-your-graphical-user- interface-605516/ ``` #### Linux 自动登陆的设置方法 #### 方法一： ##### 1 、设置GDM ``` GDM是GNOME显示管理器，通过设置其配置文件/etc/gdm/custom.conf可以设置帐号自动登陆。 设置方法如下： 在/etc/gdm/custom.conf文件中添加以下内容 [daemon] AutomaticLogin=username AutomaticLoginEnable=True ``` ``` 其中，username是要自动登陆的用户名。 说明：username不能是root，也就说无法实现root的自动登陆。 2 、设置prefdm 其中，/etc/inittab文件的最后一行，该行命令的作用是启动X Windows，而/etc/X11/prefdm就是具体实 现启动X Windows的脚本。 在/etc/X11/prefdm中添加启动X Windows的命令，并退出。 /usr/bin/startx exit 1 说明： (1)这两行代码一定要在 [ -n "$preferred" ] && exec $preferred "$@" >/dev/null 2>&1 ``` ### 切换工作目录 ``` shell cd /home/hadoop ## 切换到用户主目录 cd ~ ## 切换到用户主目录 cd - ## 回退到上次所在的目录 cd ## 什么路径都不带，则回到用户的主目录 ``` ### 创建文件夹 ``` shell mkdir aaa ## 这是相对路径的写法 mkdir /data ## 这是绝对路径的写法 mkdir -p aaa/bbb/ccc ## 级联创建目录 ``` ### 删除文件夹 ```shell rmdir aaa ## 可以删除空目录 rm -r aaa ## 可以把aaa整个文件夹及其中的所有子节点全部删除 rm -rf aaa ## 强制删除aaa ``` ### 修改文件夹名称 ``` shell mv aaa boy mv #本质上是移动 mv install.log aaa/ #将当前目录下的install.log 移动到aaa文件夹中去 rename #可以用来批量更改文件名 [root@VM_0_14_centos aaa]# ll total 0 -rw-r--r--. 1 root root 0 Jul 28 17:33 1.txt -rw-r--r--. 1 root root 0 Jul 28 17:33 2.txt -rw-r--r--. 1 root root 0 Jul 28 17:33 3.txt [root@VM_0_14_centos aaa]# rename .txt .txt.bak * [root@VM_0_14_centos aaa]# ll total 0 -rw-r--r--. 1 root root 0 Jul 28 17:33 1.txt.bak -rw-r--r--. 1 root root 0 Jul 28 17:33 2.txt.bak -rw-r--r--. 1 root root 0 Jul 28 17:33 3.txt.bak ``` ## 文件操作 ### 创建文件 ```shell touch somefile.1 ## 创建一个空文件 echo "hi,boy" > somefile.2 ## 利用重定向“>”的功能，将一条指令的输出结果写入到一个文件中，会覆盖原文件内容，如果指定的文件不存在，则会创建出来 echo "hi baby" >> somefile.2 ## 将一条指令的输出结果追加到一个文件中，不会覆盖原文件内容 ``` ### vi文本编辑器 最基本用法 vi somefile.4 1. 首先会进入“一般模式”，此模式只接受各种快捷键，不能编辑文件内容 2. 按i键，就会从一般模式进入编辑模式，此模式下，敲入的都是文件内容 3. 编辑完成之后，按Esc键退出编辑模式，回到一般模式； 4. 再按：，进入“底行命令模式”，输入wq命令，回车即可 常用快捷键 一些有用的快捷键（在一般模式下使用）： * a 在光标后一位开始插入 * A 在该行的最后插入 * I 在该行的最前面插入 * gg 直接跳到文件的首行 * G 直接跳到文件的末行 * dd 删除一行 * 3dd 删除3行 * yy 复制一行 * 3yy 复制3行 * p 粘贴 * u undo * v 进入字符选择模式，选择完成后，按y复制，按p粘贴 * ctrl+v 进入块选择模式，选择完成后，按y复制，按p粘贴 * shift+v 进入行选择模式，选择完成后，按y复制，按p粘贴 查找并替换 1. 显示行号:set nu 2. 隐藏行号:set nonu 3. 查找关键字:/you ## 效果：查找文件中出现的you，并定位到第一个找到的地方，按n可以定位到下一个匹配位置（按N定位到上一个） 4. 替换操作:s/sad/bbb ## 查找光标所在行的第一个sad，替换为bbb.:%s/sad/bbb ##查找文件中所有sad，替换为bbb ### 拷贝/删除/移动 ``` shell cp somefile.1 /home/hadoop/ rm /home/hadoop/somefile.1 rm -f /home/hadoop/somefile.1 mv /home/hadoop/somefile.1 ../ ``` ### 查看文件内容 ``` shell cat somefile 一次性将文件内容全部输出（控制台） more somefile 可以翻页查看, 下翻一页(空格) 上翻一页（b） 退出（q） less somefile 可以翻页查看,下翻一页(空格) 上翻一页（b），上翻一行(↑) 下翻一行（↓） 可以搜索关键字（/keyword） 跳到文件末尾： G 跳到文件首行： gg 退出less ： q tail -10 install.log 查看文件尾部的10行 tail +10 install.log 查看文件 10-->末行 tail -f install.log 小f跟踪文件的唯一inode号，就算文件改名后，还是跟踪原来这个inode表示的文件 tail -F install.log 大F按照文件名来跟踪 head -10 install.log 查看文件头部的10 ``` ### 打包压缩 1. gzip压缩:gzip a.txt 2. 解压:gunzip a.txt.gz 或者 gzip -d a.txt.gz 3. bzip2压缩:bzip2 a 4. 解压:bunzip2 a.bz2 或者 bzip2 -d a.bz2 5. 打包：将指定文件或文件夹-> tar -cvf bak.tar ./aaa 将/etc/password追加文件到bak.tar中(r)-> tar -rvf bak.tar /etc/password 6. 解压:tar -xvf bak.tar 7. 打包并压缩:tar -zcvf a.tar.gz aaa/ 8. 解包并解压缩(重要的事情说三遍!!!):tar -zxvf a.tar.gz 解压到/usr/下->tar -zxvf a.tar.gz -C /usr 9. 查看压缩包内容:tar -ztvf a.tar.gz zip/unzip 10. 打包并压缩成bz2:tar -jcvf a.tar.bz2 11. 解压bz2:tar -jxvf a.tar.bz2 ## 查找命令 ### 常用查找命令的使用 1. 查找可执行的命令所在的路径：which ls 2. 查找可执行的命令和帮助的位置：whereis ls 3. 从某个文件夹开始查找文件：find / -name "hadooop*" 或者find / -name "hadooop*" -ls 4. 查找并删除：find / -name "hadooop*" -ok rm {} \;find / -name "hadooop*" -exec rm {} \; 5. 查找用户为hadoop的文件:find /usr -user hadoop -ls 6. 查找用户为hadoop的文件夹:find /home -user hadoop -type d -ls 7. 查找权限为777的文件:find / -perm -777 -type d -ls 8. 显示命令历史:history ### grep命令 1. 基本使用 查询包含hadoop的行 grep hadoop /etc/password 查询包含aaa的行 grep aaa ./*.txt 2. cut截取以:分割保留第七段: grep hadoop /etc/passwd | cut -d: -f7 3. 查询不包含hadoop的行:grep -v hadoop /etc/passwd 4. 正则表达包含hadoop:grep 'hadoop' /etc/passwd 5. 正则表达(点代表任意一个字符):grep 'h.*p' /etc/passwd 6. 正则表达以hadoop开头:grep '^hadoop' /etc/passwd 7. 正则表达以hadoop结尾:grep 'hadoop$' /etc/passwd ``` 规则： . : 任意一个字符 a* : 任意多个a(零个或多个a) a? : 零个或一个a a+ : 一个或多个a .* : 任意多个任意字符 \. : 转义. o\{2\} : o重复两次 ``` 8. 查找不是以#开头的行: grep -v '^#' a.txt | grep -v '^$' 9. 以h或r开头的:grep '^[hr]' /etc/passwd 10. 不是以h和r开头的:grep '^[^hr]' /etc/passwd 11. 不是以h到r开头的:grep '^[^h-r]' /etc/passwd ## 文件权限的操作 ### linux文件权限的描述格式解读 ``` drwxr-xr-x （也可以用二进制表示 111 101 101 --> 755） d：标识节点类型（d：文件夹 -：文件 l:链接） r：可读 w：可写 x：可执行 第一组rwx： ## 表示这个文件的拥有者对它的权限：可读可写可执行 第二组r-x： ## 表示这个文件的所属组用户对它的权限：可读，不可写，可执行 第三组r-x： ## 表示这个文件的其他用户（相对于上面两类用户）对它的权限：可读，不可写，可 ``` ### 修改文件权限 ``` shell chmod g-rw haha.dat ## 表示将haha.dat对所属组的rw权限取消 chmod o-rw haha.dat ## 表示将haha.dat对其他人的rw权限取消 chmod u+x haha.dat ## 表示将haha.dat对所属用户的权限增加x chmod a-x haha.dat ## 表示将haha.dat对所用户取消x权限 chmod 664 haha.dat ## 也可以用数字的方式来修改权限 就会修改成 rw-rw-r-- chmod -R 770 aaa/ ## 如果要将一个文件夹的所有内容权限统一修改，则可以-R参数 ``` ### 修改文件所有权 <只有root权限能执行> ``` shell chown angela aaa ## 改变所属用户 chown :angela aaa ## 改变所属组 chown angela:angela aaa/ ## 同时修改所属用户和所属组 ``` ## 基本的用户管理 ``` 添加一个用户： useradd spark passwd spark 根据提示设置密码； 即可 删除一个用户： userdel -r spark 加一个-r就表示把用户及用户的主目录都删除 ``` ### 添加用户 ``` 添加一个tom用户，设置它属于users组，并添加注释信息 分步完成：useradd tom usermod -g users tom usermod -c "hr tom" tom 一步完成：useradd -g users -c "hr tom" tom 设置tom用户的密码 passwd tom ``` ### 修改用户 ``` 修改tom用户的登陆名为tomcat usermod -l tomcat tom 将tomcat添加到sys和root组中 usermod -G sys,root tomcat 查看tomcat的组信息 groups tomcat ``` ### 用户组操作 ``` 添加一个叫america的组 groupadd america 将jerry添加到america组中 usermod -g america jerry 将tomcat用户从root组和sys组删除 gpasswd -d tomcat root gpasswd -d tomcat sys 将america组名修改为am groupmod -n am america ``` ### 为用户配置sudo权限 ``` 用root编辑 vi /etc/sudoers 在文件的如下位置，为hadoop添加一行即可 root ALL=(ALL) ALL hadoop ALL=(ALL) ALL 然后，hadoop用户就可以用sudo来执行系统级别的指令 [root@localhost ~]$ sudo useradd xiaoming ``` ## 系统管理操作 ### 挂载外部存储设备 可以挂载光盘、硬盘、磁带、光盘镜像文件等 1. 挂载光驱 mkdir /mnt/cdrom 创建一个目录，用来挂载 mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom/ 将设备/dev/cdrom挂载到 挂载点 ： /mnt/cdrom中 2. 挂载光盘镜像文件（.iso文件） mount -t iso9660 -o loop /home/hadoop/Centos-7.0.DVD.iso /mnt/centos 注：挂载的资源在重启后即失效，需要重新挂载。要想自动挂载，可以将挂载信息设置到/etc/fstab配置文件中，如下： /dev/cdrom/mnt/cdromiso9660 defaults 0 0 3. 卸载 umount umount /mnt/cdrom 4. 存储空间查看 df -h ### 统计文件或文件夹的大小 ``` du -sh /mnt/cdrom/packages df -h 查看磁盘的空间 ``` ### 系统服务管理 ``` shell service sshd status service sshd stop service sshd start service sshd restart ``` ### 系统启动级别管理 ``` shell vi /etc/inittab # Default runlevel. The runlevels used are: # 0 - halt (Do NOT set initdefault to this) # 1 - Single user mode # 2 - Multiuser, without NFS (The same as 3, if you do not have networking) # 3 - Full multiuser mode # 4 - unused # 5 - X11 # 6 - reboot (Do NOT set initdefault to this) # id:3:initdefault: ## 通常将默认启动级别设置为：3 ``` ### 进程管理 ``` top free ps -ef | grep ssh kill -9 ``` ## SSH免密登陆配置 ### SSH工作机制 1. 相关概念 SSH 为 Secure Shell（安全外壳协议） 的缩写。 很多ftp、pop和telnet在本质上都是不安全的，因为它们在网络上用明文传送口令和数据，别有用心的人非常容易就可以截获这些口令和数据。 而SSH就是专为远程登录会话和其他网络服务提供安全性的协议。 SSH是由客户端和服务端的软件组成的 服务端是一个守护进程(sshd)，他在后台运行并响应来自客户端的连接请求。 客户端包含ssh程序以及像scp（远程拷贝）、slogin（远程登陆）、sftp（安全文件传输）等其他的应用程序。 2. 认证机制 从客户端来看，SSH提供两种级别的安全验证。 第一种级别（基于口令的安全验证） 只要你知道自己帐号和口令，就可以登录到远程主机。 第二种级别（基于密钥的安全验证） 需要依靠密匙，也就是你必须为自己创建一对密匙，并把公用密匙放在需要访问的服务器上。如果你要连接到SSH服务器上， 客户端软件就会向服务器发出请求，请求用你的密匙进行安全验证。服务器收到请求之后，先在该服务器上你的主目录下寻找你的公用密匙， 然后把它和你发送过来的公用密匙进行比较。如果两个密匙一致，服务器就用公用密匙加密“质询”（challenge）并把它发送给客户端软件。 客户端软件收到“质询”之后就可以用你的私人密匙解密再把它发送给服务器。 ### 密钥登陆方式配置 假如 A 要登陆 B 在A上操作： 1. 首先生成密钥对:ssh-keygen (提示时，直接回车即可) 2. 再将A自己的公钥拷贝并追加到B的授权列表文件authorized_keys中: ssh-copy-id B ## 网络管理 ### 主机名配置 1. 查看主机名:hostname 2. 修改主机名(重启后无效):hostname hadoop 3. 修改主机名(重启后永久生效) :vi /ect/sysconfig/network ### IP地址配置 修改IP地址 1. 方式一：setup 用root输入setup命令，进入交互式修改界面 2. 方式二：修改配置文件 一般使用这种方法 (重启后永久生效) vi /etc/sysconfig/network-scripts/ifcfg-eth0 3. 方式三：ifconfig命令(重启后无效) ifconfig eth0 192.168.12.22修改IP地址 ### 网络服务管理 1. 后台服务管理 ``` shell service network status 查看指定服务的状态 service network stop 停止指定服务 service network start 启动指定服务 service network restart 重启指定服务 service --status-all 查看系统中所有的后台服务 ``` 2. 设置后台服务的自启配置 ``` shell chkconfig 查看所有服务器自启配置 chkconfig iptables off 关掉指定服务的自动启动 chkconfig iptables on 开启指定服务的自动启动 ``` ### 防火墙 #### iptables防火墙 1. 基本操作 * 查看防火墙状态 ``` service iptables status   ``` * 停止防火墙 ``` service iptables stop   ``` * 启动防火墙 ``` service iptables start   ``` * 重启防火墙 ``` service iptables restart   ``` * 永久关闭防火墙 ``` chkconfig iptables off   ``` * 永久关闭后重启 ``` chkconfig iptables on　　 ``` 2. 开启80端口 ``` vim /etc/sysconfig/iptables # 加入如下代码 -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT ``` #### firewall防火墙 1. 查看firewall服务状态 ``` systemctl status firewalld ``` 2. 查看firewall的状态 ``` firewall-cmd --state ``` 3. 开启、重启、关闭、firewalld.service服务 ``` # 开启 service firewalld start # 重启 service firewalld restart # 关闭 service firewalld stop ``` 4. 查看防火墙规则 ``` firewall-cmd --list-all ``` 5. 查询、开放、关闭端口 ``` # 查询端口是否开放 firewall-cmd --query-port=8080/tcp # 开放80端口 firewall-cmd --permanent --add-port=80/tcp # 移除端口 firewall-cmd --permanent --remove-port=8080/tcp #重启防火墙(修改配置后要重启防火墙) firewall-cmd --reload ``` 参数解释: * firwall-cmd：是Linux提供的操作firewall的一个工具； * --permanent：表示设置为持久； * --add-port：标识添加的端口； ================================================ FILE: docs/linux/ope.md ================================================ #逼格高又实用的 Linux 命令，开发、运维一定要懂！ ### 实用的 xargs 命令 在平时的使用中，我认为xargs这个命令还是较为重要和方便的。我们可以通过使用这个命令，将命令输出的结果作为参数传递给另一个命令。 比如说我们想找出某个路径下以 .conf 结尾的文件，并将这些文件进行分类，那么普通的做法就是先将以 .conf 结尾的文件先找出来，然后输出到一个文件中，接着cat这个文件，并使用file文件分类命令去对输出的文件进行分类。这个普通的方法还的确是略显麻烦，那么这个时候xargs命令就派上用场了。 例1：找出 / 目录下以.conf 结尾的文件，并进行文件分类 命令： ``` shell find / -name *.conf -type f -print | xargs file ``` 输出结果如下所示： ![ope](./../img/ope1.png) ### 命令或脚本后台运行 有时候我们进行一些操作的时候，不希望我们的操作在终端会话断了之后就跟着断了，特别是一些数据库导入导出操作，如果涉及到大数据量的操作，我们不可能保证我们的网络在我们的操作期间不出问题，所以后台运行脚本或者命令对我们来说是一大保障。 比如说我们想把数据库的导出操作后台运行，并且将命令的操作输出记录到文件，那么我们可以这么做： ``` nohup mysqldump -uroot -pxxxxx --all-databases > ./alldatabases.sql &（xxxxx是密码） ``` 当然如果你不想密码明文，你还可以这么做： ``` nohup mysqldump -uroot -p --all-databases > ./alldatabases.sql （后面不加&符号） ``` 执行了上述命令后，会提示叫你输入密码，输入密码后，该命令还在前台运行，但是我们的目的是后天运行该命令，这个时候你可以按下Ctrl+Z，然后在输入bg就可以达到第一个命令的效果，让该命令后台运行，同时也可以让密码隐蔽输入。 命令后台执行的结果会在命令执行的当前目录下留下一个 nohup.out 文件，查看这个文件就知道命令有没有执行报错等信息。 ### 找出当前系统内存使用量较高的进程 在很多运维的时候，我们发现内存耗用较为严重，那么怎么样才能找出内存消耗的进程排序呢？ 命令： ``` ps -aux | sort -rnk 4 | head -20 ``` ![ope](./../img/ope2.png) 输出的第4列就是内存的耗用百分比。最后一列就是相对应的进程。 ### 找出当前系统CPU使用量较高的进程 在很多运维的时候，我们发现CPU耗用较为严重，那么怎么样才能找出CPU消耗的进程排序呢？ 命令： ``` ps -aux | sort -rnk 3 | head -20 ``` ![ope](./../img/ope3.png) 输出的第3列为CPU的耗用百分比，最后一列就是对应的进程。 我想大家应该也发现了，sort 命令后的3、4其实就是代表着第3列进行排序、第4列进行排序。 ### 同时查看多个日志或数据文件 在日常工作中，我们查看日志文件的方式可能是使用tail命令在一个个的终端查看日志文件，一个终端就看一个日志文件。包括我在内也是，但是有时候也会觉得这种方式略显麻烦，其实有个工具叫做 multitail 可以在同一个终端同时查看多个日志文件。 首先安装 multitail： ``` # wget ftp://ftp.is.co.za/mirror/ftp.rpmforge.net/redhat/el6/en/x86_64/dag/RPMS/multitail-5.2.9-1.el6.rf.x86_64.rpm # yum -y localinstall multitail-5.2.9-1.el6.rf.x86_64.rpm ``` multitail 工具支持文本的高亮显示，内容过滤以及更多你可能需要的功能。 如下就来一个有用的例子： 此时我们既想查看secure的日志指定过滤关键字输出，又想查看实时的网络ping情况： 命令如下： ``` # multitail -e "Accepted" /var/log/secure -l "ping baidu.com" ``` ![ope](./../img/ope4.png) 是不是很方便？如果平时我们想查看两个日志之间的关联性，可以观察日志输出是否有触发等。如果分开两个终端可能来回进行切换有点浪费时间，这个multitail工具查看未尝不是一个好方法。 ### 持续 ping 并将结果记录到日志 这个时候你去ping几个包把结果丢出来，人家会反驳你，刚刚那段时间有问题而已，现在业务都恢复正常了，网络肯定正常啊，这个时候估计你要气死。 你要是再拿出zabbix等网络监控的数据，这个时候就不太妥当了，zabbix的采集数据间隔你不可能设置成1秒钟1次吧？小编就遇到过这样的问题，结果我通过以下的命令进行了ping监控采集。然后再有人让我背锅的时候，我把出问题时间段的ping数据库截取出来，大家公开谈，结果那次被我叼杠回去了，以后他们都不敢轻易甩锅了，这个感觉好啊。 命令： ``` ping api.jpush.cn | awk '{ print $0"\t" strftime("%Y-%m-%d %H:%M:%S",systime()) } ' >> /tmp/jiguang.log &` ``` 输出的结果会记录到/tmp/jiguang.log 中，每秒钟新增一条ping记录，如下： ![ope](./../img/ope5.png) ### ssh实现端口转发 可能很多的朋友都听说过ssh是linux下的远程登录安全协议，就是通俗的远程登录管理服务器。但是应该很少朋友会听说过ssh还可以做端口转发。其实ssh用来做端口转发的功能还是很强大的，下面就来做示范。 实例背景：我们公司是有堡垒机的，任何操作均需要在堡垒机上进行，有写开发人员需要访问ELasticSearch的head面板查看集群状态，但是我们并不想将ElasticSearch的9200端口映射出去，依然想通过堡垒机进行访问。 所以才会将通往堡垒机（192.168.1.15）的请求转发到服务器ElasticSearch（192.168.1.19）的9200上。 例子： 将发往本机（192.168.1.15）的9200端口访问转发到192.168.1.19的9200端口 ``` ssh -p 22 -C -f -N -g -L 9200:192.168.1.19:9200 ihavecar@192.168.1.19` ``` 记住：前提是先进行秘钥传输。 命令执行完后，访问192.168.1.15:9200端口则真实是访问192.168.1.19:9200端口。 ### grep命令进阶 grep命令是一种强大的文本搜索工具，它能使用正则表达式搜索文本，并把匹 配的行打印出来 grep搜索成功，则返回0，如果搜索不成功，则返回1，如果搜索的文件不存在，则返回2。 * grep的规则表达式（正则一定要转义） ``` shell ^ #锚定行的开始 如：'^grep'匹配所有以grep开头的行。 $ #锚定行的结束 如：'grep$'匹配所有以grep结尾的行。 . #匹配一个非换行符的字符 如：'gr.p'匹配gr后接一个任意字符，然后是p。 * #匹配零个或多个先前字符 如：'*grep'匹配所有一个或多个空格后紧跟grep的行。 .* #一起用代表任意字符。 [] #匹配一个指定范围内的字符，如'[Gg]rep'匹配Grep和grep。 [^] #匹配一个不在指定范围内的字符 \(..\) #标记匹配字符，如'\(love\)'，love被标记为1。 \< #锚定单词的开始，如:'\ #锚定单词的结束，如'grep\>'匹配包含以grep结尾的单词的行。 x\{m\} #重复字符x，m次，如：'0\{5\}'匹配包含5个o的行。 x\{m,\} #重复字符x,至少m次，如：'o\{5,\}'匹配至少有5个o的行。 x\{m,n\}#重复字符x，至少m次，不多于n次，如：'o\{5,10\}'匹配5--10个o的行。 \w #匹配文字和数字字符，也就是[A-Za-z0-9]， \W #\w的反置形式，匹配一个或多个非单词字符，如点号句号等。 \b #单词锁定符，如: '\bgrep\b'只匹配grep。 ``` * grep常见命令参数 ``` shell -n 打印行号 grep -n ".*" h.txt 所有打印行号 grep -n "root" h.txt 匹配的内容显示行号 -v 不包括 -E 表示过滤 多个参数 grep -Ev "sshd|network|crond|sysstat|" -o:仅打印你需要的东西，默认打印正行 grep -o "hello" h.txt -i:忽略大小写 grep -i "hello" h.txt -c: 用于统计文中出现的次数 --color=auto 过滤字段添加颜色 利用正则打印特定字符 \b：作为边界符，边界只包含特定字符的行 grep "\boldboy\b" /etc/passwd -->只过滤包含oldboy的行 ``` * Linux egrep命令详解 ``` shell egrep: == grep -E 用于显示文件中符合条件的字符 env|egrep "USER|MAIL|PWD|LOGNAME" 用的表达式不一样 ，egerp更加规范 egrep -o "oldboy|hello" h.txt -->仅仅输出 oldboy 和 hello ``` * 常用的命令展示 ``` shell #查找指定关键字个数 grep '\bboot\b' logs_bak.txt 【\b单词锁定符，只匹配boot】 #输出logs_bak.txt 文件中含有从logs.txt文件中读取出的关键词的内容行 cat logs_bak.txt cat logs.txt cat logs.txt | grep -nf logs_bak.txt #从多个文件中查找关键词 grep "omc" /etc/passwd /etc/shadow 【多文件查询时，会用冒号前添加文件名】 #打印IP信息 ifconfig eth0|grep -E "([0-9]{1,3}\.){3}" 【-E 表达式匹配，用小括号括起来表示一个整体】 #同时过滤多个关键字 cat /etc/passwd|grep -E "boy|omc" ==> cat /etc/passwd|egrep "omc|boy" 【用 | 划分多个关键字】 #显示当前目录下面以.txt 结尾的文件中的所有包含每个字符串至少有7个连续小写字符的字符串的行 grep '\w\{7\}' *.txt ==> grep '[a-z]\{7\}' *.txt 【注意特殊字符的转义】 ``` * 上下文的控制(了解) ``` shell # A 查询匹配内容的一行之外，后n行的显示 # B 查询匹配内容的一行之外，前n行的显示 # C 查询匹配内容的一行之外，显示上下n行 grep -n 'yum' -A 3 logs_bak.txt ``` ================================================ FILE: docs/micro/ddd.md ================================================ # 基于DDD的微服务设计和开发实战 你是否还在为微服务应该拆多小而争论不休？到底如何才能设计出收放自如的微服务？怎样才能保证业务领域模型与代码模型的一致性？或许本文能帮你找到答案。 本文是基于 DDD 的微服务设计和开发实战篇，通过借鉴领域驱动设计思想，指导微服务项目团队进行设计和开发（理论篇详见《当中台遇上 DDD，我们该如何设计微服务？》）。本文包括三部分内容：第一部分讲述领域驱动设计基本知识，包括：分层架构、服务视图、数据视图和领域事件发布和订阅等；第二部分讲述微服务设计方法、过程、模板、代码目录、设计原则等内容；最后部分以一个项目为例讲述基于 DDD 的微服务设计过程。 ## 目标 本文采用 DDD（领域驱动设计）作为微服务设计指导思想，通过事件风暴建立领域模型，合理划分领域逻辑和物理边界，建立领域对象及服务矩阵和服务架构图，定义符合 DDD 分层架构思想的代码结构模型，保证业务模型与代码模型的一致性。通过上述设计思想、方法和过程，指导团队按照 DDD 设计思想完成微服务设计和开发。 通过领域模型和 DDD 的分层思想，屏蔽外部变化对领域逻辑的影响，确保交付的软件产品是边界清晰的微服务，而不是内部边界依然混乱的小单体。在需求和设计变化时，可以轻松的完成微服务的开发、拆分和组合，确保微服务不易受外部变化的影响，并稳定运行。 ## 适用范围 本文适用于按照 DDD 设计方法进行微服务设计和开发的项目及相关人员。 ## DDD 分层架构视图 DDD 分层架构包括：展现层、应用层、领域层和基础层。 ![](./../img/micro/ddd1.png) DDD 分层架构各层职能如下： * 展现层 展现层负责向用户显示信息和解释用户指令。 * 应用层 应用层是很薄的一层，主要面向用户用例操作，协调和指挥领域对象来完成业务逻辑。应用层也是与其他系统的应用层进行交互的必要渠道。应用层服务尽量简单，它不包含业务规则或知识，只为下一层的领域对象协调任务，使它们互相协作。应用层还可进行安全认证、权限校验、分布式和持久化事务控制或向外部应用发送基于事件的消息等。 * 领域层 领域层是软件的核心所在，它实现全部业务逻辑并且通过各种校验手段保证业务正确性。它包含业务所涉及的领域对象（实体、值对象）、领域服务以及它们之间的关系。它负责表达业务概念、业务状态以及业务规则，具体表现形式就是领域模型。 * 基础层 基础层为各层提供通用的技术能力，包括：为应用层传递消息、提供 API 管理，为领域层提供数据库持久化机制等。它还能通过技术框架来支持各层之间的交互。 ## 服务视图 微服务内的服务视图 微服务内有 Facade 接口、应用服务、领域服务和基础服务，各层服务协同配合，为外部提供服务。 ![](./../img/micro/ddd2.png) ### 1、接口服务 接口服务位于用户接口层，用于处理用户发送的 Restful 请求和解析用户输入的配置文件等，并将信息传递给应用层。 ### 2、应用服务 应用服务位于应用层。用来表述应用和用户行为，负责服务的组合、编排和转发，负责处理业务用例的执行顺序以及结果的拼装。 应用层的服务包括应用服务和领域事件相关服务。 应用服务可对微服务内的领域服务以及微服务外的应用服务进行组合和编排，或者对基础层如文件、缓存等数据直接操作形成应用服务，对外提供粗粒度的服务。 领域事件服务包括两类：领域事件的发布和订阅。通过事件总线和消息队列实现异步数据传输，实现微服务之间的解耦。 ### 3、领域服务 领域服务位于领域层，为完成领域中跨实体或值对象的操作转换而封装的服务，领域服务以与实体和值对象相同的方式参与实施过程。 领域服务对同一个实体的一个或多个方法进行组合和封装，或对多个不同实体的操作进行组合或编排，对外暴露成领域服务。领域服务封装了核心的业务逻辑。实体自身的行为在实体类内部实现，向上封装成领域服务暴露。 为隐藏领域层的业务逻辑实现，所有领域方法和服务等均须通过领域服务对外暴露。 为实现微服务内聚合之间的解耦，原则上禁止跨聚合的领域服务调用和跨聚合的数据相互关联。 ### 4、基础服务 基础服务位于基础层。为各层提供资源服务（如数据库、缓存等），实现各层的解耦，降低外部资源变化对业务逻辑的影响。 基础服务主要为仓储服务，通过依赖反转的方式为各层提供基础资源服务，领域服务和应用服务调用仓储服务接口，利用仓储实现持久化数据对象或直接访问基础资源。 ## 微服务外的服务视图 ### 1. 前端应用与微服务 微服务中的应用服务通过用户接口层组装和数据转换后，发布在 API 网关，为前端应用提供数据展示服务。 ### 2. 微服务与外部应用 跨微服务数据处理时，对实时性要求高的场景，可选择直接调用应用服务的方式（新增和修改类型操作需关注事务一致性）。对实时性要求不高的场景，可选择异步化的领域事件驱动机制（最终数据一致性）。 ## 数据视图 DDD 分层架构中数据对象转换的过程如下图。 ![](./../img/micro/ddd3.png) 数据视图应用服务通过数据传输对象（DTO）完成外部数据交换。领域层通过领域对象（DO）作为领域实体和值对象的数据和行为载体。基础层利用持久化对象（PO）完成数据库的交换。 DTO 与 VO 通过 Restful 协议实现 JSON 格式和对象转换。 前端应用与应用层之间 DTO 与 DO 的转换发生在用户接口层。如微服务内应用服务需调用外部微服务的应用服务，则 DTO 的组装和 DTO 与 DO 的转换发生在应用层。 领域层 DO 与 PO 的转换发生在基础层。 ## 领域事件和事件总线 领域事件是领域模型中非常重要的部分，用来表示领域中发生的事件。一个领域事件将导致进一步的业务操作，有助于形成完整的业务闭环。领域事件主要用于解耦微服务，各个微服务之间不再是强一致性，而是基于事件的最终一致性。 ![](./../img/micro/ddd4.png) ### 微服务内的领域事件 微服务内的领域事件可以通过事件总线或利用应用服务实现不同聚合之间的业务协同。当微服务内发生领域事件时，由于大部分事件的集成发生在同一个线程内，不一定需要引入消息中间件。但一个事件如果同时更新多个聚合数据，按照 DDD“一个事务只更新一个聚合根”的原则，可以考虑引入消息中间件，通过异步化的方式，对微服务内不同的聚合根采用不同的事务。 ### 微服务之间的领域事件 微服务之间的数据交互方式通常有两种：应用服务调用和领域事件驱动机制。 领域事件驱动机制更多的用于不同微服务之间的集成，实现微服务之间的解耦。事件库（表）可以用于微服务之间的数据对账，在应用、网络等出现问题后，可以实现源和目的端的数据比对，在数据暂时不一致的情况下仍可根据这些数据完成后续业务处理流程，保证微服务之间数据的最终一致性。 应用服务调用方式通常应用于实时性要求高的业务场景，但一旦涉及到跨微服务的数据修改，将会增加分布式事务控制成本，影响系统性能，微服务之间的耦合度也会变高。 ### 事件总线 事件总线位于基础层，为应用层和领域层服务提供事件消息接收和分发等服务。其大致流程如下： 1、服务触发并发布事件。 2、事件总线事件分发。 * 如果是微服务内的订阅者（微服务内的其它聚合），则直接分发到指定订阅者。 * 如果是微服务外的订阅者，则事件消息先保存到事件库（表）并异步发送到消息中间件。 * 如果同时存在微服务内和外订阅者，则分发到内部订阅者，并将事件消息保存到事件库（表）并异步发送到消息中间件。为了保证事务的一致性，事件表可以共享业务数据库。也可以采用多个微服务共享事件库的方式。当业务操作和事件发布操作跨数据库时，须保证业务操作和事件发布操作数据的强一致性。 ### 事件数据持久化 事件数据的持久化存储可以有两种方案，在项目实施过程中根据具体场景选择最佳方案。 * 事件数据保存到微服务所在业务数据库的事件表中，利用本地事务保证业务操作和事件发布操作的强一致性。 * 事件数据保存到多个微服务共享的事件库中。需要注意的一点是：这时业务操作和事件发布操作会跨数据库操作，须保证事务的强一致性（如分布式事务机制）。 事件数据的持久化可以保证数据的完整性，基于这些数据可以完成跨微服务数据的一致性比对。 ## 微服务设计方法 ### 事件风暴 本阶段主要完成领域模型设计。 基于 DDD 的微服务设计通常采用事件风暴方法。通过事件风暴完成领域模型设计，划分出微服务逻辑边界和物理边界，定义领域模型中的领域对象，指导微服务设计和开发。事件风暴通常包括产品愿景、场景分析、领域建模、微服务设计和拆分等过程。本文不对事件风暴详细方法做深入描述，如感兴趣可查阅相关资料。 * 1、产品愿景 产品愿景是对产品的顶层价值设计，对产品目标用户、核心价值、差异化竞争点等信息达成一致，避免产品偏离方向。建议参与角色：业务需求方、产品经理和开发组长。 * 2、场景分析 场景分析是从用户视角出发，探索业务领域中的典型场景，产出领域中需要支撑的场景分类、用例操作以及不同子域之间的依赖关系，用以支撑领域建模。 建议参与角色：产品经理、需求分析人员、架构师、开发组长和测试组长。 * 3、领域建模 领域建模是通过对业务和问题域进行分析，建立领域模型，向上通过限界上下文指导微服务边界设计，向下通过聚合指导实体的对象设计。 建议参与角色：领域专家、产品经理、需求分析人员、架构师、开发组长和测试组长。 * 4、微服务拆分和设计 结合业务限界上下文与技术因素，对服务的粒度、分层、边界划分、依赖关系和集成关系进行梳理，完成微服务拆分和设计。 微服务设计应综合考虑业务职责单一、敏态与稳态业务分离、非功能性需求（如弹性伸缩要求、安全性等要求）、团队组织和沟通效率、软件包大小以及技术异构等因素。 建议参与角色：产品经理、需求分析人员、架构师、开发组长和测试组长。 ### 领域对象及服务矩阵和代码模型设计 本阶段完成领域对象及服务矩阵文档以及微服务代码模型设计。 * 1、领域对象及服务矩阵 根据事件风暴过程领域对象和关系，对产出的限界上下文、聚合、实体、值对象、仓储、事件、应用服务、领域服务等领域对象以及各对象之间的依赖关系进行梳理，确定各对象在分层架构中的位置和依赖关系，建立领域对象分层架构视图，为每个领域对象建立与代码模型对象的一一映射。 建议参与角色：架构师和开发组长。 * 2、微服务代码模型 根据领域对象在 DDD 分层架构中所在的层、领域类型、与代码对象的映射关系，定义领域对象在微服务代码模型中的包、类和方法名称等，设计微服务工程的代码层级和代码结构，明确各层间的调用关系。 建议参与角色：架构师和开发组长。 ### 领域对象及服务矩阵样例说明 领域对象及服务矩阵主要用来记录事件风暴和微服务设计过程中产出的领域对象属性，如：各领域对象在 DDD 分层架构中的位置、属性、依赖关系以及与代码对象的映射关系等。通过建立领域对象与代码对象的映射关系，可指导软件开发人员准确无误的按照设计文档完成微服务开发。 以下为领域对象及服务矩阵样例（部分数据，仅供参考）。 ![](./../img/micro/ddd5.png) 各栏说明如下： * 层： 定义领域对象位于 DDD 分层架构中的哪一层。如：接口层、应用层、领域层以及基础层等。 * 聚合： 在事件风暴过程中将关联紧密的实体和值对象等组合形成聚合。本栏说明聚合名称。 * 领域对象名称： 领域模型中领域对象的具体名称。如：“请假审批已通过”是类型为“事件”的领域对象；“请假单”是领域类型为“实体”的领域对象。 * 领域类型： 在领域模型中根据 DDD 知识域定义的领域对象的类型，如：限界上下文、聚合、聚合根（实体）、实体、值对象、事件、命令、应用服务、领域服务和仓储服务等。 依赖对象名称：根据业务对象依赖或分层调用依赖关系建立的领域对象的依赖关系（如服务调用依赖、关联对象聚合等）。本栏说明领域对象需依赖的其他领域对象，如上层服务在组合和编排过程中对下层服务的调用依赖、实体之间或者实体与值对象在聚合内的依赖等。 * 包名： 代码模型中的包名，本栏说明领域对象所在的软件包。 * 类名： 代码模型中的类名，本栏说明领域对象的类名。 * 方法名： 代码模型中的方法名，本栏说明领域对象实现或操作的方法名。 ## 微服务代码结构模型 微服务代码模型最终结果来源于领域对象及服务矩阵。在代码模型设计时须建立领域对象和代码对象的一一映射，保证业务模型与代码模型的一致性，即使不熟悉业务的开发人员或者不熟悉代码的业务人员也可以很快定位到代码位置。 ### 微服务代码总目录 基于 DDD 的代码模型包括 interfaces、application、domain 和 infrastructure 四个目录。 ![](./../img/micro/ddd6.png) * Interfaces（用户接口层）： 本目录主要存放用户接口层代码。前端应用通过本层向应用服务获取展现所需的数据。本层主要用于处理用户发送的 Restful 请求和解析用户输入的配置文件等，并将信息传递给 Application 层。主要代码形态是数据组装以及 Facade 接口等。 * Application（应用层）： 本目录主要存放应用层代码。应用服务代码基于微服务内的领域服务或微服务外的应用服务完成服务编排和组合。为用户接口层提供各种应用数据展现支持。主要代码形态是应用服务和领域事件等。 * Domain（领域层）： 本目录主要存放领域层代码。本层代码主要实现核心领域逻辑，其主要代码形态是实体类方法和领域服务等。 * Infrastructure（基础层）： 本目录存放基础层代码，为其它各层提供通用技术能力、三方软件包、配置和基础资源服务等。 ### 用户接口层代码模型 用户接口层代码模型目录包括：assembler、dto 和 facade。 ![](./../img/micro/ddd7.png) * Assembler：实现 DTO 与领域对象之间的相互转换和数据交换。理论上 Assembler 总是与 DTO 一同被使用。 * Dto：数据传输的载体，内部不存在任何业务逻辑，通过 DTO 把内部的领域对象与外界隔离。 * Facade：提供较粗粒度的调用接口，将用户请求委派给一个或多个应用服务进行处理。 ### 应用层代码模型 应用层代码模型目录包括：event 和 service。 ![](./../img/micro/ddd8.png) * Event（事件）：事件目录包括两个子目录：publish 和 subscribe。publish 目录主要存放微服务内领域事件发布相关代码。subscribe 目录主要存放微服务内聚合之间或外部微服务领域事件订阅处理相关代码。为了实现领域事件的统一管理，微服务内所有领域事件（包括应用层和领域层事件）的发布和订阅处理都统一放在应用层。 * Service（应用服务）：这里的服务是应用服务。应用服务对多个领域服务或外部应用服务进行封装、编排和组合，对外提供粗粒度的服务。 ### 领域层代码模型 微服务领域层包括一个或多个聚合代码包。标准的聚合代码模型包括：entity、repository 和 service 三个子目录。 ![](./../img/micro/ddd9.png) * Aggregate（聚合）：聚合代码包的根目录，实际项目中以实际业务属性的名称来命名。聚合定义了领域对象之间的关系和边界，实现领域模型的内聚。 * Entity（实体）：存放实体（含聚合根、实体和值对象）相关代码。同一实体所有相关的代码（含对同一实体类多个对象操作的方法，如对多个对象的 count 等）都放在一个实体类中。 * Service（领域服务）：存放对多个不同实体对象操作的领域服务代码。这部分代码以领域服务的形式存在，在设计时一个领域服务对应一个类。 * Repository（仓储）：存放聚合对应的查询或持久化领域对象的代码，通常包括仓储接口和仓储实现方法。为了方便聚合的拆分和组合，我们设定一个原则：一个聚合对应一个仓储。 * 特别说明：按照 DDD 分层原则，仓储实现本应属于基础层代码，但为了微服务代码拆分和重组的便利性，我们把聚合的仓储实现代码放到了领域层对应的聚合代码包内。如果需求或者设计发生变化导致聚合需要拆分或重新组合时，我们可以聚合代码包为单位，轻松实现微服务聚合的拆分和组合。 ### 基础层代码模型 基础层代码模型包括：config 和 util 两个子目录。 ![](./../img/micro/ddd10.png) * Config：主要存放配置相关代码。 * Util：主要存放平台、开发框架、消息、数据库、缓存、文件、总线、网关、第三方类库、通用算法等基础代码，可为不同的资源类别建立不同的子目录。 ### 微服务总目录结构 微服务总目录结构如下： ![](./../img/micro/ddd11.png) ### 微服务设计原则 微服务设计原则中如高内聚低耦合、复用、单一职责等原则在此就不赘述了，这里主要强调以下几条： * 第一条：“要领域驱动设计，而不是数据驱动设计，也不是界面驱动设计”。 微服务设计首先应建立领域模型，确定逻辑和物理边界后，然后才进行微服务边界拆分，而不是一上来就定义数据库表结构，也不是界面需要什么，就去调整领域逻辑代码。 领域模型和领域服务应具有高度通用性，通过接口层和应用层屏蔽外部变化对业务逻辑的影响，保证核心业务功能的稳定性。 * 第二条：“要边界清晰的微服务，而不是泥球小单体”。 微服务完成开发后其功能和代码也不是一成不变的。随着需求或设计变化，微服务内的代码也会分分合合。逻辑边界清晰的微服务，可快速实现微服务代码的拆分和组合。DDD 思想中的逻辑边界和分层设计也是为微服务各种可能的分分合合做准备的。 微服务内聚合与聚合之间的领域服务以及数据原则上禁止相互产生依赖。如有必要可通过上层的应用服务编排或者事件驱动机制实现聚合之间的解耦，以利于聚合之间的组合和拆分。 * 第三条：“要职能清晰的分层，而不是什么都放的大箩筐”。 分层架构中各层职能定位清晰，且都只能与其下方的层发生依赖，也就是说只能从外层调用内层服务，内层服务通过封装、组合或编排对外逐层暴露，服务粒度由细到粗。 应用层负责服务的编排和组合，领域层负责领域业务逻辑的实现，基础层为各层提供资源服务。 * 第四条：“要做自己能 hold 住的微服务，而不是过度拆分的微服务” 微服务的过度拆分必然会带来软件维护成本的上升，如：集成成本、运维成本以及监控和定位问题的成本。企业转型过程中很难短时间内提升这些能力，如果项目团队不具备这些能力，将很难 hold 住这些过细的微服务。而如果我们在微服务设计之初就已经定义好了微服务内的逻辑边界，项目初期我们可以尽可能少的拆分出过细的微服务，随着技术的积累和时间的推移，当我们具有这些能力后，由于微服务内有清晰的逻辑边界，这时就可以随时根据需要轻松的拆分或组合出新的微服务。 ## 不同场景的微服务设计 微服务的设计先从领域建模开始，领域模型是微服务设计的核心，微服务是领域建模的结果。在微服务设计之前，请先判断你的业务是否聚焦在领域和领域逻辑。 实际在做系统设计时我们可能面临各种不同的情形，如从传统单体拆分为多个微服务，也可能是一个全新领域的微服务设计（如创业中的应用），抑或是将一个单体中面临问题或性能瓶颈的模块拆分为微服务而其余功能仍为单体的情况。 下面分几类不同场景说明如何进行微服务和领域模型设计。 ### 新建系统的微服务设计 新建系统会遇到复杂和简单领域两种场景，两者的领域建模过程也会有所差别。 #### 1、简单领域的建模 对于简单的业务领域，一个领域可能就是一个小的子域。领域建模过程相对简单，根据事件风暴可以分解出事件、命令、实体、聚合和限界上下文等，根据领域模型和微服务拆分原则设计出微服务即可。 #### 2、复杂领域的建模 对于复杂的业务领域，领域可能还需要拆分为子域，甚至子域还会进一步拆分，如：保险领域可以拆分为承保、理赔、收付费和再保等子域，承保子域还可以再拆分为投保、保单管理等子子域。对于这种复杂的领域模型，是无法通过一个事件风暴完成领域建模的，即使能完成，其工程量也是非常浩大，效果也不一定好。 对于这种复杂的领域，我们可以分三阶段来完成领域模型和微服务设计。 * 拆分子域建立领域模型：根据业务特点考虑流程节点或功能模块等边界因素（微服务最终的拆分结果很多时候跟这些边界因素有一定的相关性），按领域逐级分解为大小合适的子域，针对子域进行事件风暴，记录领域对象、聚合和限界上下文，初步确定各级子域的领域模型。 * 领域模型微调：梳理领域内所有子域的领域模型，对各子域模型进行微调，这个过程重点考虑不同限界上下文内聚合的重新组合，同步需要考虑子域、限界上下文以及聚合之间的边界、服务以及事件之间的依赖关系，确定最终的领域模型。 * 微服务设计和拆分：根据领域模型的限界上下文和微服务的拆分原则，完成微服务的拆分和设计。 ### 单体遗留系统的微服务设计 如果一个单体遗留系统，只是将面临问题或性能瓶颈的模块拆分为微服务，而其余功能仍为单体。我们只需要将这些特定功能领域理解为一个简单的子领域，按照简单领域建模方式进行领域模型的设计即可。但在新微服务设计中需要考虑新老系统之间的服务协议，必要时引入防腐层。 ### 特别说明 虽然有些业务领域在事件风暴后发现无法建立领域模型，如数据处理或分析类场景，但本文所述的分层架构模型、服务之间规约和代码目录结构在微服务设计和开发中仍然是通用的。 ## 基于 DDD 的微服务设计和开发实例 为了更好的理解 DDD 的设计思想和过程，我们用一个场景简单但基本涵盖 DDD 设计思想的项目来说明微服务设计和开发过程。 ### 项目基本信息 项目主要目标是实现在线请假和考勤管理。基本功能包括：请假、考勤以及人员管理等。 * 请假：请假人填写请假单提交审批，根据请假人身份和请假天数进行校验，根据审批规则逐级递交审批，核批通过则完成审批。 * 考勤：根据考勤规则，剔除请假数据后，对员工考勤数据进行校验，输出考勤统计表。 * 人员管理：维护人员基本信息和上下级关系。 ...... ### 设计和实施步骤 #### 步骤一：事件风暴 由于项目目标基本明确，我们在事件风暴过程中裁剪了产品愿景，直接从用户旅程和场景分析开始。 * 1、场景分析：场景分析是一个发散的过程。根据不同角色的旅程和场景分析，尽可能全面的梳理从前端操作到后端业务逻辑发生的所有操作、命令、领域事件以及外部依赖关系等信息（如下图），如：请假人员会执行创建请假信息操作命令，审批人员会执行审批操作，请假审批通过后会产生领域事件，通知邮件系统反馈请假人员结果，并将请假数据发送到考勤以便核销等。在记录这些领域对象的同时，我们也会标记各对象在 DDD 中的层和对象类型等属性，如：应用服务、领域服务、事件和命令等类型。 ![](./../img/micro/ddd12.png) * 2、领域建模：领域建模是一个收敛的过程。这个收敛过程分三步：第一步根据场景分析中的操作集合定义领域实体；第二步根据领域实体业务关联性，定义聚合；第三步根据业务及语义边界等因素，定义限界上下文。 * 定义领域实体：在场景分析过程中梳理完操作、命令、领域事件以及外部依赖关系等领域对象后。分析这些操作应由什么实体发起或产生，从而定义领域实体对象，并将这些操作与实体进行关联。 在请假场景中，经分析需要有请假单实体对象，请假单实体有创建请假信息以及修改请假信息等操作。 * 定义聚合：将业务紧密相关的实体进行组合形成聚合，同时确定聚合中的聚合根、值对象和实体。经分析项目最终形成三个聚合：人员管理、请假和考勤。在请假聚合中有请假单、审批轨迹、审批规则等实体，其中请假单是聚合根，审批轨迹是请假单的值对象，审批规则是辅助实体。 * 聚合内须保证业务操作的事务性，高度内聚的实体对象可自包含完成本领域功能。聚合是可拆分为微服务的最小单元。在同一限界上下文内多个聚合可以组合为一个微服务。如有必要，也可以将某一个聚合独立为微服务。 * 定义限界上下文：根据领域及语义边界等因素确定限界上下文，将同一个语义环境下的一个或者多个聚合放在一个限界上下文内。由于人员管理与请假聚合两者业务关联紧密，共同完成人员请假功能，两者一起构成请假限界上下文，考勤聚合则单独形成考勤限界上下文。 * 3、微服务设计和拆分：理论上一个限界上下文可以设计为一个微服务，但还需要综合考虑多种外部因素，如：职责单一性、性能差异、版本发布频率、团队沟通效率和技术异构等要素。 由于本项目微服务设计受技术以及团队等因素影响相对较小，主要考虑职责单一性，因此根据限界上下文直接拆分为请假和考勤两个微服务。其中请假微服务包含人员和请假两个聚合，考勤微服务只包含考勤聚合。 #### 步骤二、领域对象及服务矩阵 将事件风暴中产出的领域对象按照各自所在的微服务进行分类，定义每个领域对象在微服务中的层、领域类型和依赖的领域对象等。 这个步骤最关键的工作是确定实体、方法、服务等领域对象在微服务分层架构中的位置以及各对象之间的依赖关系，形成服务矩阵（如下表）。这个过程也将在事件风暴数据的基础上，进一步细化领域对象以及它们之间关系，并补充事件风暴中可能遗漏的细节。 确定完各领域对象的属性后，按照代码模型设计各个领域对象在代码模型中的代码对象（包括代码对象所在的：包名、类名和方法名），建立领域对象与代码对象的一一映射关系。根据这种映射关系，相关人员可快速定位到业务逻辑所在的代码位置。 ![](./../img/micro/ddd13.png) #### 步骤三：领域模型及服务架构 根据领域模型中领域对象属性以及服务矩阵，画出领域对象及服务架构视图（如下图）。这个视图可以作为标准的 DDD 分层领域服务架构视图模型，应用在不同的领域模型中。这个模型可以清晰的体现微服务内实体、聚合之间的关系，各层服务之间的依赖关系以及应用层服务组合和编排的关系，微服务之间的服务调用以及事件驱动的前后处理逻辑关系。 在这个阶段，前端的设计也可以同步进行，在这里我们用到了微前端的设计理念，为请假和考勤微服务分别设计了请假和考勤微前端，基于微前端和微服务，形成从前端到后端的业务逻辑自包含组件。两个微前端之上有一个集成主页面，可根据页面流动态加载请假和考勤的微前端页面。 ![](./../img/micro/ddd14.png) #### 步骤四：代码模型设计 根据 DDD 的代码结构模型和各领域对象在所在的包、类和方法，定义出请假微服务的代码结构模型。应用层代码结构包括：应用服务以及事件发布相关代码（如下图）。 ![](./../img/micro/ddd15.png) 领域层代码结构包括一个或多个聚合的实体类以及领域服务相关代码（如下图）。在本项目中请假微服务领域层包含了请假和人员两个聚合。 ![](./../img/micro/ddd16.png) 领域模型中的一个聚合对应一个聚合代码包，如：人员和请假领域逻辑代码都放在各自的聚合代码包中，如随着业务发展，人员管理功能需要从请假微服务中拆分出来，我们只需要将人员聚合代码包稍加改造并独立部署即可快速发布为人员管理微服务。 #### 步骤五：详细设计 在完成领域模型和代码模型设计后，我们就可以开始详细设计了，详细设计主要结合具体的业务功能来开展，主要工作包括：系统界面、数据库表以及字段、服务参数规约及功能等。 #### 步骤六：代码开发 软件开发人员只需要按照设计文档和功能要求，找到业务功能对应的代码位置，完成代码开发和服务编排即可。 #### 步骤七：测试和发布 完成代码开发后，由开发人员编写单元测试用例，基于挡板模拟依赖对象完成跨服务的测试。单元测试完成后，在团队内可进一步完成微服务与相应微前端的集成和测试，形成请假和考勤两个业务组件。前端主页面完成请假和考勤微前端页面集成和页面流及组件基础数据配置，主页面可以按照页面流程动态加载请假和考勤微前端页面。最终部署的软件包包括：请假和考勤两个微服务，请假和考勤两个微前端，一个主页面共计五个。这五个部署包独立开发、独立运行和独立部署。 #### 技术组件说明 主页面和微前端采用：Vue(前端框架)，ElementUI(UI 框架 -PC)，VUX(UI 框架 - 移动端) 和 MPVUE(UI 框架 - 小程序) 等。微服务开发采用：Spring Cloud、Kafka、Redis 等。数据库采用：PostgreSQL。 ### 附录一：DDD 名词和术语 * Event Storming（事件风暴）：事件风暴是一项团队活动，旨在通过领域事件识别出聚合根，进而划分微服务的限界上下文。在活动中，团队先通过头脑风暴的形式罗列出领域中所有的领域事件，整合之后形成最终的领域事件集合，然后对于每一个事件，标注出导致该事件的命令（Command），再然后为每个事件标注出命令发起方的角色，命令可以是用户发起，也可以是第三方系统调用或者是定时器触发等。最后对事件进行分类整理出聚合根以及限界上下文。 * Entity（实体）：每个实体是唯一的，并且可以相当长的一段时间内持续地变化。我们可以对实体做多次修改，故一个实体对象可能和它先前的状态大不相同。但是，由于它们拥有相同的身份标识，他们依然是同一个实体。例如一件商品在电商商品上下文中是一个实体，通过商品中台唯一的商品 id 来标示这个实体。 * ValueObject（值对象）：值对象用于度量和描述事物，当你只关心某个对象的属性时，该对象便可作为一个值对象。实体与值对象的区别在于唯一的身份标识和可变性。当一个对象用于描述一个事物，但是又没有唯一标示，那么它就是一个值对象。例如商品中的商品类别，类别就没有一个唯一标识，通过图书、服装等这些值就能明确表示这个商品类别。 * Aggregate（聚合）：聚合是实体的升级，是由一组与生俱来就密切相关实体和值对象组合而成的，整个组合的最上层实体就是聚合。 * Bounded Context（限界上下文）：用来封装通用语言和领域对象，为领域提供上下文语境，保证在领域之内的一些术语、业务相关对象等（通用语言）有一个确切的含义，没有二义性。使团队所有成员能够明确地知道什么必须保持一致，什么必须独立开发。 ================================================ FILE: docs/micro/design.md ================================================ # 微服务架构的设计原则 ## 微服务架构的设计原则如下： 1. 高内聚、低耦合。 2. 无缝的 API 集成。 3. 为每一项服务分配唯一的资源标识。 4. 实时流量管理。 5. 最小化数据表，以优化加载。 6. 通过内/外部 API，执行持续监控。 7. 为每个微服务隔离数据的存储。这对于限制数据的访问和避免“服务的耦合”是非常有用的。 例如：基于用户的分类数据，我们可以实施命令查询的责任分离（Command Query Responsibility Segregation，CQRS）。 8. 去中心化。设计微服务架构的首要原则是：将单体结构分解成独立的多个实体，而这些实体就被称为微服务。 这些微服务能够独立于其他的系统功能提供服务，用户对它们采取的所有编辑、删除、或在其他地方的使用，都不会影响到本系统的整体性能。 9. 可扩展性。微服务的设计目标是：性能与效率。在现实世界中，解决大型系统的可扩展性问题，是任何微服务生态系统的性能体现。 虽然丰富的技术功能给大量的数据工作带来了多种数据片段，但是如果能恰当地实施、并使用各种应用程序控制器（Application Controllers），则会让微服务架构更具可扩展性。 10. 通过与 DevOps 的集成，实现持续交付。DevOps 的多技术互通与融合，比较适合于微服务架构。在设计微服务架构时，我们需要关注性能和系统效率的提升，这正好契合了 DevOps 的更快交付出方案的理念。 相对于传统的单体式设计，它更适合于部署性、可靠的和可扩展性的方案管理。 当然，相对于上述各项原则与优势，微服务架构也有着一定的局限性。不过好在我们拥有多种微服务的设计模式可供选择，来实现自己的系统设计目标。下面让我们来逐一进行讨论。 ## 针对有效协作的微服务设计模式 高效的微服务架构必须能够让多个微服务实现有效的协作和同步运行。 ### 聚合器微服务设计模式 由于会涉及到多种业务，我们有必要为最终用户截获输出、并将其予以合并。 对于用户来说，如果他们想自行合并数据，则需要具有对于系统的大量内部知识。 那么，我们在设计微服务架构时，为了打破这种单体性，就应当根据输出来进行资源的划分。因此，我们利用聚合器模式，来汇总这些数据。 这种方案可以通过两个主要组件，来呈现给最终用户。其中的一种是带有 API 网关的复合式微服务。它可以汇总数据，并将其转发给用户。 如果您需要在分解的系统中用到各种业务功能模块的话，复合式微服务应当成为您的首选。 ### 分支微服务设计模式 此模式扩展了聚合器的设计模式。在分支模式下，您可以通过两个独立的（更精确地说是：互斥的）微服务链，来同时处理请求和响应。 这种设计模式能够根据您的不同业务需求，为单个或多个服务链提供灵活性。 例如：针对某个电子商务网站或 Web 应用程序，我们可能按需接收来自不同微服务的多个数据源。 ### 后端为前端/API 网关 从每个运行服务处获取数据，是任何应用程序的首要任务。对于微服务架构而言，从独立的服务中提取数据，同样非常重要。 但是，仅通过一个用户界面（UI），从大量的微服务中获取用户手中的资源信息，并不是一件容易的事。 因此，就像企业里的服务台那样，我们可以在微服务架构中，用 API 网关来为所有的交互操作提供统一的入口。 另外，在安全方面，API 网关也有助于实现用户的授权、和为合适的用户提供相关的 API。 因此 API 网关作为单一的切入点，不仅能够充当代理服务器的作用，将各种请求路由到不同的微服务那里，还能汇总来自多个服务的输出结果，并发送给用户。 它可以处理多种协议请求，并按需进行转换（例如，实现 HTTPS 与 AMQP 之间的互转）。 ## 针对性能监控的微服务设计模式 性能监测是确保微服务架构成功的另一个重要方面。它有助于衡量系统的效率，并获悉拖慢系统的罪魁祸首。下列模式涉及到可观测性的范畴，能够保障微服务架构设计的鲁棒性。 ### 日志聚合 微服务是可以独立地、并行地支持多种其他服务的。而且，它的实例能够横跨各台机器。 同时，每个服务都会根据其执行情况生成一个日志入口。那么我们该如何跟踪这些大量的服务相关日志呢？ 这正是日志聚合模式的切入点。为了防止出现混乱的局面，我们应当设置一个可以对所有微服务实例进行日志聚合的主服务。而且，这种集中式日志应当具有搜索和监控的功能。 ### 综合监控（或称语义监控） 对微服务架构的监控，是一项繁琐、但又必要的任务。当有数百个服务同时运行时，要想在日志库里查明某个失败的根本原因，那就更难了。此时，综合监控可能会派上用场。 在您进行自动化测试时，综合监控能够帮助您将结果与生产环境定期进行映射和比较。一旦出现故障，用户将及时得到相关警告。 另外，语义监控还能帮助您实现如下两方面： 1. 监测自动化的测试案例。 2. 根据业务需求，检测生产环境中的故障。 同时，随着系统的负载和微服务数量的增加，对系统性能的持续监控，并跨服务发现潜在的问题，就显得非常重要了。 我们可以通过指标服务（metric service）来收集各类数据。指标服务可以是“推式”和“拉式”两种形式。 顾名思义，推式服务（如：AppDynamics）是将捕捉到的数据指标推送到服务端；而拉式服务（如：Prometheus）则负责从服务中拉取对应的数据指标。 ## API 健康检查 微服务架构的设计促进了各个服务之间的相互独立，并避免了系统中的任何延迟。 我们知道，API 在网络连接中能够起到基石作用。我们需要通过对 API 的定期健康检查，来提前发现各种障碍。 例如，您可能会经常观察到：某个微服务虽然处于启动和和运行状态，却没有能力去处理任何请求。 那么，以下便是一些导致 API 故障的因素： * 服务器的负载 * 用户的使用量 * 各种延迟 * 错误日志 * 下载量 为了应对上述因素，我们应该确保每一个服务在运行时，都配有一个健康检查的特定 API 端点。 例如：我们可以在每个服务的末尾，通过追加 HTTP/health 的参数，以返回各个服务实例、主机、连接和算法逻辑等方面的健康状况。 同时，服务注册中心需要定期调用健康检查的 API 端点，以执行相关的健康扫描任务。 而健康检查的内容则可以包括如下方面： * 针对特定应用的系统逻辑。 * 主机的状态。 * 连接到其他基础设施或任何服务实例的连接状态。 为业务能力而改变一切 在将单体架构分解成多个微服务的过程中，我们需要根据实际情况遵循不同的设计模式。 ## 针对业务能力的独特微服务 微服务的成功应该能够充分体现高内聚和低耦合的特点。因此各种服务需要在抽象出相似功能的基础上，保持低耦合的状态。那么，我们该如何将软件系统分解成为更小、更独立的逻辑单元呢？ 我们需要定义微服务的范围，从而支持特定的业务能力。例如，我们可以将组织的内部架构分为技术、营销、公关、销售、服务和运维等不同部门，这些不同的职能部门都可以被看作是各个微服务，而组织本身就是一套系统。 因此，为了保持效率和预估增长，我们需要按照业务能力对系统进行分解，基于各种能力所产生的价值，区分不同的业务领域。 ## 围绕相似业务能力的微服务 虽然我们可以按照业务能力分类出各种微服务，但是我们该如何处置那些服务中的通用类呢？在此，我们可以用需要干预的“神类（God Classes）”来分解这些类。 例如，在电子商务系统中，订单是一个共用类，一些诸如订单号、订单管理、订单退货、订单交付等服务都会用到它。因此针对该问题，我们引入了领域驱动设计（Domain-Driven Design，DDD）的微服务设计原则。 在领域驱动设计中，我们使用到了各个子域。这些子域模型应当被预定好功能的范围，即界限上下文（bounded context）。 这些界限上下文作为参数被用来创建微服务，从而克服了通用类的相关问题。 ## 刀砍藤蔓的模式 上面我们讨论了对全新单体架构概念的分解，下面我们来看看如何将现有的单体系统转换为微服务架构。在此，我们引入刀砍蔓藤的模式。 由于在 Web 应用中，经常会涉及到不同域里各个服务之间的往返调用，因此刀砍模式非常适用于对域的切分。 在该模式下，虽然系统会出现两个域共用一个相同 URI 的情况，但是一旦某个服务完成了转换，我们就会砍掉其对应的应用程序中的现有版本。而且，此过程会一直持续下去，直到单体系统不复存在为止。 ### 针对优化数据库存储的微服务设计模式 就微服务架构而言，其松耦合的特性造就了各个独立服务的部署与扩展能力。 但是由于不同的服务有着不同的存储需求，因此它们可能需要访问那些并非存储在本地的数据。 下面让我们来讨论一些根据不同的需求，所适合采用的主要数据库设计模式。 ## 基于服务的单独数据库 在领域驱动设计的原则中，基于服务的单独数据库，是将整个数据库分配给特定的微服务。 因此，我们需要按照服务来事先设计好独享的数据库。也就是说，任何其他的外部微服务都无法访问，其他未分配给自己的数据库里的数据，除非是通过微服务的 API 网关方式。 ## 基于服务的共享数据库 如果我们只是将上述独享数据库模式运用到，将单体架构分解成多个微服务的场景中，那么难免会碰到各种麻烦。 因此，我们可以在分解的过程中，对有限数量的服务采用基于服务的共享数据库模式。 而这个数量一般会被限制在 2～3 个，否则对系统的部署、自治性和扩展性有所影响。 ## 事件溯源式设计模式 当应用的当前状态发生变化时，我们如何才能确保架构能够按需变更，并根据这些变更实时地产生相应的事件呢？ 事件溯源（Event Sourcing）模式，能够根据每个业务实体的状态变化，顺次将新的事件追加到事件列表之中。 在系统中，诸如 Customer 这样的实体会产生许多事件，因此我们可以对实体的当时状态进行“截屏式”事件录入，以便进一步查询或通过自动化的状态调整，来优化负载。 ## 命令查询的责任分离（CQRS） 对于基于服务的数据库模式而言，由于访问被限制在了单个数据库之中，因此我们很难达到各种复杂的查询效果。那么我们该如何实现各种基于数据库系统的联合查询呢？ CQRS 模型将单个应用程序分为命令和查询两个部分： 1. 命令部分处理各种创建、更新和删除之类的请求。 2. 查询则用到了物化视图（materialized view），而这些视图是通过事件流来进行更新的。 同时，这些事件又是由事件溯源模式所产生，并标注了数据中的各种变化。 ### 针对无缝部署的微服务设计模式 在我们实施微服务时，难免会在服务的调用上碰到问题，因此我们可以采用横切（cross-cutting）的模式来简化工作。 ## 服务发现 由于采用了容器技术，IP 地址往往是被动态地分配的。这就意味着 IP 地址会随时发生改变，进而导致服务的中断。此外，用户需要记住每个服务的 URL，而这反而倒退成了紧耦合状态。 为了解决该问题，我们需要通过一个注册表，向用户的请求提供位置信息。服务实例在启动时，能够被注册到表中；而在关闭时，也能被注销。 此法有助于用户找出那些可用来查询的准确位置。此外，注册表通过健康检查，能够确保实例的可用性，进而提高系统的性能。 ## 蓝-绿部署 在微服务架构中，一个系统里往往有着多个微服务。如果我们因为部署、或更新版本而停止所有的服务的话，那么长时间的停机势必会影响整体的生产力。 因此，我们需要在设计微服务架构时，通过蓝-绿部署的模式，来避免该问题。 在这个模式中，我们同时有着蓝、绿两套相同且并行的环境。在任一时间点，只有一套环境（如蓝色系统）真实在线，并处理着真实的业务流量。 那么在需要进行新的部署时，我们将应用的最新版本上传到绿色系统中，并将真实对外的路由器切换到绿色系统上，以完成更新。 ## 结论 虽然您不一定会在自己的微服务系统中用到上述每一种设计模式，但是这些模式都有着它们独特的应用场景。 作为架构师，对于不同的微服务架构设计模式，您需要从应用的设计阶段到生产环境的维护阶段，持续进行各种评估、审计、测试和实践。相信它们在给您带来一致性标准的同时，也能提高您的应用的整体可靠性。 ================================================ FILE: docs/micro/distrimsg.md ================================================ # 分布式系统与消息的投递 消息是一个非常有趣的概念，它是由来源发出一个离散的通信单元，被发送给一个或者一群接受者，无论是单体服务还是分布式系统中都有消息的概念，只是这两种系统中传输消息的通道方法或者通道不同；单体服务中的消息往往可以通过 IO、进程间通信、方法调用的方式进行通信，而分布式系统中的远程调用就需要通过网络，使用 UDP 或者 TCP 等协议进行传输。 ![](./../img/dmsg1.png) 然而网络在计算机的世界中是最不可控的，如果我们通过网络请求调用其他服务的接口，可能就会由于种种原因没有将消息送达至目标的服务，对于当前服务我们并不能控制网络的传输，在很多时候也很难控制网络通信的质量，这也就是为什么『网络是稳定、可信赖的』分布式系统中常见的谬论之一。 * 通信渠道的不可靠是造成构建大规模分布式系统非常复杂并且困难的重要原因。 ### 网络请求 作为分布式系统之间各个节点的通信渠道，网络其实是非常不可靠通信方式，如果我们想要保证节点状态的一致性，这种通信方式的复杂性使得我们在进行跨服务调用时需要处理非常多的边界条件，在之前的文章 分布式系统 · 分布式事务的实现原理 中简单介绍过，网络通信可能会包含，成功、失败以及超时三种情况。 ![](./../img/dmsg2.png) 每一次网络请求其实都是一次信息的投递，由于当前的节点无法得知其他节点信息，只能通过网络请求的响应来得知这次信息投递的结果。 ### 成功与失败 虽然网络的情况比较不稳定，但是我们在大多数时候通过网络传输一些信息时，无论是返回的结果是成功还是失败，其实都能得到确定的结果： ![](./../img/dmsg3.png) 每一次确定的响应都需要这次请求在一个往返以及被调用节点中正确处理，流量既不能被中间代理丢包，也不能由于目标节点的错误导致无法发出响应，只有在同时满足了这两个条件的情况下，我们才能得到确定的响应结果。对于节点来说，这次请求返回成功还是失败都比较好处理，因为只要有确定的结果，网络请求这种通信方式与进程间通信或者方法调用这些更可靠的途径在处理上都没有太多的区别，但是在通信的过程中出现其他的问题时就比较棘手了。 ### 超时 在分布式系统中，不是任何的网络请求都能够得到确定的响应，如果网络请求在往返以及被调用节点处理的过程中出现了丢包或者节点错误，发出请求的节点就可能永远也无法得到这次请求的响应。 ![](./../img/dmsg4.png) 每一个节点在发出请求之后，都对这次请求如何路由以及被处理一无所知，所以节点需要设置一个合适的超时时间，如果请求没有在规定的时间内返回，就会认为当前请求已经超时，也就是网络请求失败了。 超时的网络请求是导致分布式系统难以处理的根本原因之一，在这种问题发生时节点并不知道目标节点是否收到了当前请求，对于幂等的网络请求还好，一旦请求可能会改变目标节点的状态就非常棘手了，因为我们并不能确定上一次网络请求是在哪一步失败的，如果是响应返回的过程中发生了故障，那么如果重试一些请求就会出现问题，可能会触发银行的两次转账，这是我们无论如何也无法接受的；总而言之，网络通信的不稳定迫使我们处理由于超时而出现的复杂问题，这也是在开发分布式系统时不得不考虑的。 ### 消息投递语义 在分布式系统中使用网络进行通信确实是一种不可靠的方式，消息的发送者只能知道掌控当前节点，所以没有办法保证传输渠道的可靠性，网络超时这种常见的通信错误极大地增加了分布式系统通信的复杂度，我们可以对网络提供的基本传输能力进行封装，保证数据通信的可靠性。 ![](./../img/dmsg5.png) 网络请求由于超时的问题，消息的发送者只能通过重试的方式对消息进行重发，但是这就可能会导致消息的重复发送与处理，然而如果超时后不重新发送消息也可能导致消息的丢失，所以如何在不可靠的通信方式中，保证消息不重不漏是非常关键的。 ![](./../img/dmsg6.png) 我们一般都会认为，消息的投递语义有三种，分别是最多一次（At-Most Once）、最少一次（At-Least Once）以及正好一次（Exactly Once），我们分别会介绍这三种消息投递语义究竟是如何工作的。 ### 最多一次 最多一次其实非常容易保证的，UDP 这种传输层的协议其实保证的就是最多一次消息投递，消息的发送者只会尝试发送该消息一次，并不会关心该消息是否得到了远程节点的响应。 ![](./../img/dmsg7.png) 无论该请求是否发送给了接受者，发送者都不会重新发送这条消息；这其实就是最最基本的消息投递语义，然而消息可能由于网络或者节点的故障出现丢失。 ### 最少一次 为了解决最多一次时的消息丢失问题，消息的发送者需要在网络出现超时重新发送相同的消息，也就是引入超时重试的机制，在发送者发出消息会监听消息的响应，如果超过了一定时间也没有得到响应就会重新发送该消息，直到得到确定的响应结果。 ![](./../img/dmsg8.png) 对于最少一次的投递语义，我们不仅需要引入超时重试机制，还需要关心每一次请求的响应，只有这样才能确保消息不会丢失，但是却可能会造成消息的重复，这就是最少一次在解决消息丢失后引入的新问题。 ### 正好一次 虽然最少一次解决了最多一次的消息丢失问题，但是由于重试却带来了另一个问题 - 消息重复，也就是接受者可能会多次收到同一条消息；从理论上来说，在分布式系统中想要解决消息重复的问题是不可能的，很多消息服务提供了正好一次的 QoS 其实是在接收端进行了去重。 ![](./../img/dmsg9.png) 消息去重需要生产者生产消息时加入去重的 key，消费者可以通过唯一的 key 来判断当前消息是否是重复消息，从消息发送者的角度来看，实现正好一次的投递是不可能的，但是从整体来看，我们可以通过唯一 key 或者重入幂等的方式对消息进行『去重』。 消息的重复是不可能避免的，除非我们允许消息的丢失，然而相比于丢失消息，重复发送消息其实是一种更能让人接受的处理方式，因为一旦消息丢失就无法找回，但是消息重复却可以通过其他方法来避免副作用。 ### 投递顺序 由于一些网络的问题，消息在投递时可能会出现顺序不一致性的情况，在网络条件非常不稳定时，我们就可能会遇到接收方处理消息的顺序和生产者投递的不一致；想要满足绝对的顺序投递，其实在生产者和消费者的单线程运行时是相对比较好解决的，但是在市面上比较主流的消息队列中，都不会对消息的顺序进行保证，在这种大前提下，消费者就需要对顺序不一致的消息进行处理，常见的两种方式就是使用序列号或者状态机。 ### 序列号 使用序列号保证投递顺序的方式其实与 TCP 协议中使用的 SEQ 非常相似，因为网络并不能保证所有数据包传输的顺序并且每个栈帧的传输大小有限，所以 TCP 协议在发送数据包时加入 SEQ，接受方可以通过 SEQ 将多个数据包拼接起来并交由上层协议进行处理。 ![](./../img/dmsg10.png) 在投递消息时加入序列号其实与 TCP 中的序列号非常类似，我们需要在数据之外增加消息的序列号，对于消费者就可以根据每一条消息附带的序列号选择如何处理顺序不一致的消息，对于不同的业务来说，常见的处理方式就是用阻塞的方式保证序列号的递增或者忽略部分『过期』的消息。 ### 状态机 使用序列号确实能够保证消息状态的一致，但是却需要在消息投递时额外增加字段，这样消费者才能在投递出现问题时进行处理，除了这种方式之外，我们也可以通过状态机的方式保证数据的一致性，每一个资源都有相应的状态迁移事件，这些事件其实就是一个个消息（或操作），它们能够修改资源的状态： ![](./../img/dmsg11.png) 在状态机中我们可以规定，状态的迁移方向，所有资源的状态只能按照我们规定好的线路进行改变，在这时只要对生产者投递的消息状态做一定的约束，例如：资源一旦 completed 就不会变成 failed，因为这两个状态都是业务逻辑中定义的最终状态，所以处于最终状态的资源都不会继续接受其他的消息。 假设我们有如下的两条消息 active 和 complete，它们分别会改变当前资源的状态，如果一个处于 pending 状态的资源先收到了 active 再收到 complete，那么状态就会从 pending 迁移到 active 再到 completed；但是如果资源先收到 complete 后收到 active，那么当前资源的状态会直接从 pending 跳跃到 completed，对于另一条消息就会直接忽略；从总体来看，虽然消息投递的顺序是乱序的，但是资源最终还是通过状态机达到了我们想要的正确状态，不会出现不一致的问题。 ### 协议 消息投递其实有非常多相关的应用，最常见的组件就是消息队列了，作为一种在各个 Web 项目中常用的组件，它提供了很多能力，包括消息的持久存储、不同的投递语义以及复杂的路由规则等等，能够显著地增加系统的可用性、起到比较比明显的削峰效果。 在这里将介绍几种比较常见的消息队列协议，我们将简单说明各个协议的作用以及它们的实现原理和关键特性，也会简单提及一些遵循这些协议实现的消息队列中间件。 ### AMQP 协议 AMQP 协议的全称是 Advanced Message Queuing Protocol，它是一个用于面向消息中间件的开放标准，协议中定义了队列、路由、可用性以及安全性等方面的内容。 ![](./../img/dmsg12.png) 该协议目前能够为通用的消息队列架构提供一系列的标准，将发布订阅、队列、事务以及流数据等功能抽象成了用于解决消息投递以及相关问题的标准，StormMQ、RabbitMQ 都是 AMQP 协议的一个实现。 在所有实现 AMQP 协议的消息中间中，RabbitMQ 其实是最出名的一个实现，在分布式系统中，它经常用于存储和转发消息，当生产者短时间内创建了大量的消息，就会通过消息中间件对消息转储，消费者会按照当前的资源对消息进行消费。 ![](./../img/dmsg13.png) RabbitMQ 在消息投递的过程中保证存储在 RabbitMQ 中的全部消息不会丢失、推送者和订阅者需要通过信号的方式确认消息的投递，它支持最多一次和最少一次的投递语义，当我们选择最少一次时，需要幂等或者重入机制保证消息重复不会出现问题。 ### MQTT 协议 另一个用于处理发布订阅功能的常见协议就是 MQTT 了，它建立在 TCP/IP 协议之上，能够在硬件性能底下或者网络状态糟糕的情况下完成发布与订阅的功能；与 AMQP 不同，MQTT 协议支持三种不同的服务质量级别（QoS），也就是投递语义，最多一次、最少一次和正好一次。 从理论上来看，在分布式系统中实现正好一次的投递语义是不可能的，这里实现的正好一次其实是协议层做了重试和去重机制，消费者在处理 MQTT 消息时就不需要关系消息是否重复这种问题了。 ### 总结 在分布式系统中想要保证消息的送达确实是一件比较复杂的事情，通信方式的不确定使得我们需要处理很多问题，我们既需要在网络错误或者超时时进行重试，还需要对一些请求支持重入和幂等，保证不会出现一致性的错误；这其实都是因为在分布式系统中，正好一次的消息投递语义是不存在的，消息要么可能会丢失，要么就可能会重复。 ================================================ FILE: docs/micro/fbs-lock.md ================================================ # 带你玩转分布式锁 大多数互联网系统都是分布式部署的，分布式部署确实能带来性能和效率上的提升，但为此，我们就需要多解决一个分布式环境下，数据一致性的问题。 当某个资源在多系统之间，具有共享性的时候，为了保证大家访问这个资源数据是一致的，那么就必须要求在同一时刻只能被一个客户端处理，不能并发的执行，否者就会出现同一时刻有人写有人读，大家访问到的数据就不一致了。 ## 一、我们为什么需要分布式锁？ 在单机时代，虽然不需要分布式锁，但也面临过类似的问题，只不过在单机的情况下，如果有多个线程要同时访问某个共享资源的时候，我们可以采用线程间加锁的机制，即当某个线程获取到这个资源后，就立即对这个资源进行加锁，当使用完资源之后，再解锁，其它线程就可以接着使用了。例如，在JAVA中，甚至专门提供了一些处理锁机制的一些API（synchronize/Lock等）。 但是到了分布式系统的时代，这种线程之间的锁机制，就没作用了，系统可能会有多份并且部署在不同的机器上，这些资源已经不是在线程之间共享了，而是属于进程之间共享的资源。 因此，为了解决这个问题，我们就必须引入「分布式锁」。 分布式锁，是指在分布式的部署环境下，通过锁机制来让多客户端互斥的对共享资源进行访问。 分布式锁要满足哪些要求呢？ * 排他性：在同一时间只会有一个客户端能获取到锁，其它客户端无法同时获取 * 避免死锁：这把锁在一段有限的时间之后，一定会被释放（正常释放或异常释放） * 高可用：获取或释放锁的机制必须高可用且性能佳 讲完了背景和理论，那我们接下来再看一下分布式锁的具体分类和实际运用。 ## 二、分布式锁的实现方式有哪些？ 目前主流的有三种，从实现的复杂度上来看，从上往下难度依次增加： * 基于数据库实现 * 基于Redis实现 * 基于ZooKeeper实现 无论哪种方式，其实都不完美，依旧要根据咱们业务的实际场景来选择。 ### 1 基于数据库实现： 基于数据库来做分布式锁的话，通常有两种做法： * 基于数据库的乐观锁 * 基于数据库的悲观锁 #### 我们先来看一下如何基于「乐观锁」来实现： 乐观锁机制其实就是在数据库表中引入一个版本号（version）字段来实现的。 当我们要从数据库中读取数据的时候，同时把这个version字段也读出来，如果要对读出来的数据进行更新后写回数据库，则需要将version加1，同时将新的数据与新的version更新到数据表中，且必须在更新的时候同时检查目前数据库里version值是不是之前的那个version，如果是，则正常更新。如果不是，则更新失败，说明在这个过程中有其它的进程去更新过数据了。 下面找图举例， ![fbs](./../img/fbs1.png) 如图，假设同一个账户，用户A和用户B都要去进行取款操作，账户的原始余额是2000，用户A要去取1500，用户B要去取1000，如果没有锁机制的话，在并发的情况下，可能会出现余额同时被扣1500和1000，导致最终余额的不正确甚至是负数。但如果这里用到乐观锁机制，当两个用户去数据库中读取余额的时候，除了读取到2000余额以外，还读取了当前的版本号version=1，等用户A或用户B去修改数据库余额的时候，无论谁先操作，都会将版本号加1，即version=2，那么另外一个用户去更新的时候就发现版本号不对，已经变成2了，不是当初读出来时候的1，那么本次更新失败，就得重新去读取最新的数据库余额。 通过上面这个例子可以看出来，使用「乐观锁」机制，必须得满足： 1. 锁服务要有递增的版本号version 2. 每次更新数据的时候都必须先判断版本号对不对，然后再写入新的版本号 #### 我们再来看一下如何基于「悲观锁」来实现： 悲观锁也叫作排它锁，在Mysql中是基于 for update 来实现加锁的，例如： ```java //锁定的方法-伪代码 public boolean lock(){ connection.setAutoCommit(false) for(){ result = select * from user where id = 100 for update; if(result){ //结果不为空， //则说明获取到了锁 return true; } //没有获取到锁，继续获取 sleep(1000); } return false; } //释放锁-伪代码 connection.commit(); ``` 上面的示例中，user表中，id是主键，通过 for update 操作，数据库在查询的时候就会给这条记录加上排它锁。 （需要注意的是，在InnoDB中只有字段加了索引的，才会是行级锁，否者是表级锁，所以这个id字段要加索引） 当这条记录加上排它锁之后，其它线程是无法操作这条记录的。 那么，这样的话，我们就可以认为获得了排它锁的这个线程是拥有了分布式锁，然后就可以执行我们想要做的业务逻辑，当逻辑完成之后，再调用上述释放锁的语句即可。 ### 2.基于Redis实现 基于Redis实现的锁机制，主要是依赖redis自身的原子操作，例如： ```Redis SET user_key user_value NX PX 100 ``` redis从2.6.12版本开始，SET命令才支持这些参数： NX：只在在键不存在时，才对键进行设置操作，SET key value NX 效果等同于 SETNX key value PX millisecond：设置键的过期时间为millisecond毫秒，当超过这个时间后，设置的键会自动失效 上述代码示例是指， 当redis中不存在user_key这个键的时候，才会去设置一个user_key键，并且给这个键的值设置为 user_value，且这个键的存活时间为100ms 为什么这个命令可以帮我们实现锁机制呢？ 因为这个命令是只有在某个key不存在的时候，才会执行成功。那么当多个进程同时并发的去设置同一个key的时候，就永远只会有一个进程成功。 当某个进程设置成功之后，就可以去执行业务逻辑了，等业务逻辑执行完毕之后，再去进行解锁。 解锁很简单，只需要删除这个key就可以了，不过删除之前需要判断，这个key对应的value是当初自己设置的那个。 另外，针对redis集群模式的分布式锁，可以采用redis的Redlock机制。 ### 3.基于ZooKeeper实现 其实基于ZooKeeper，就是使用它的临时有序节点来实现的分布式锁。 原理就是：当某客户端要进行逻辑的加锁时，就在zookeeper上的某个指定节点的目录下，去生成一个唯一的临时有序节点， 然后判断自己是否是这些有序节点中序号最小的一个，如果是，则算是获取了锁。如果不是，则说明没有获取到锁，那么就需要在序列中找到比自己小的那个节点，并对其调用exist()方法，对其注册事件监听，当监听到这个节点被删除了，那就再去判断一次自己当初创建的节点是否变成了序列中最小的。如果是，则获取锁，如果不是，则重复上述步骤。 当释放锁的时候，只需将这个临时节点删除即可。 ![fbs](./../img/fbs2.png) 如图，locker是一个持久节点，node_1/node_2/…/node_n 就是上面说的临时节点，由客户端client去创建的。 client_1/client_2/…/clien_n 都是想去获取锁的客户端。以client_1为例，它想去获取分布式锁，则需要跑到locker下面去创建临时节点（假如是node_1）创建完毕后，看一下自己的节点序号是否是locker下面最小的，如果是，则获取了锁。如果不是，则去找到比自己小的那个节点（假如是node_2），找到后，就监听node_2，直到node_2被删除，那么就开始再次判断自己的node_1是不是序列中最小的，如果是，则获取锁，如果还不是，则继续找一下一个节点。 以上，就讲完了为什么我们需要分布式锁这个技术，以及分布式锁中常见的三种机制，欢迎大家一起交流。 ================================================ FILE: docs/micro/kafka.md ================================================ # Kafka架构原理 >最终大家会掌握 Kafka 中最重要的概念，分别是 Broker、Producer、Consumer、Consumer Group、Topic、Partition、Replica、Leader、Follower，这是学会和理解 Kafka 的基础和必备内容。 ## 定义 Kafka 是一个分布式的基于发布/订阅模式的消息队列（Message Queue），主要应用与大数据实时处理领域。 ### 消息队列 Kafka 本质上是一个 MQ（Message Queue），使用消息队列的好处？（面试会问） * 解耦：允许我们独立的扩展或修改队列两边的处理过程。 * 可恢复性：即使一个处理消息的进程挂掉，加入队列中的消息仍然可以在系统恢复后被处理。 * 缓冲：有助于解决生产消息和消费消息的处理速度不一致的情况。 * 灵活性&峰值处理能力：不会因为突发的超负荷的请求而完全崩溃，消息队列能够使关键组件顶住突发的访问压力。 * 异步通信：消息队列允许用户把消息放入队列但不立即处理它。 ### 发布/订阅模式 ![kafka](../img/micro_kafka/kafka-1.png) 一对多，生产者将消息发布到 Topic 中，有多个消费者订阅该主题，发布到 Topic 的消息会被所有订阅者消费，被消费的数据不会立即从 Topic 清除。 ## 架构 ![kafka](../img/micro_kafka/kafka-2.png) Kafka 存储的消息来自任意多被称为 Producer 生产者的进程。数据从而可以被发布到不同的 Topic 主题下的不同 Partition 分区。 在一个分区内，这些消息被索引并连同时间戳存储在一起。其它被称为 Consumer 消费者的进程可以从分区订阅消息。 Kafka 运行在一个由一台或多台服务器组成的集群上，并且分区可以跨集群结点分布。 下面给出 Kafka 一些重要概念，让大家对 Kafka 有个整体的认识和感知，后面还会详细的解析每一个概念的作用以及更深入的原理： * Producer： 消息生产者，向 Kafka Broker 发消息的客户端。 * Consumer：消息消费者，从 Kafka Broker 取消息的客户端。 * Consumer Group：消费者组（CG），消费者组内每个消费者负责消费不同分区的数据，提高消费能力。一个分区只能由组内一个消费者消费，消费者组之间互不影响。所有的消费者都属于某个消费者组，即消费者组是逻辑上的一个订阅者。 * Broker：一台 Kafka 机器就是一个 Broker。一个集群由多个 Broker 组成。一个 Broker 可以容纳多个 Topic。 * Topic：可以理解为一个队列，Topic 将消息分类，生产者和消费者面向的是同一个 Topic。 * Partition：为了实现扩展性，提高并发能力，一个非常大的 Topic 可以分布到多个 Broker （即服务器）上，一个 Topic 可以分为多个 Partition，每个 Partition 是一个 有序的队列。 * Replica：副本，为实现备份的功能，保证集群中的某个节点发生故障时，该节点上的 Partition 数据不丢失，且 Kafka 仍然能够继续工作，Kafka 提供了副本机制，一个 Topic 的每个分区都有若干个副本，一个 Leader 和若干个 Follower。 * Leader：每个分区多个副本的“主”副本，生产者发送数据的对象，以及消费者消费数据的对象，都是 Leader。 * Follower：每个分区多个副本的“从”副本，实时从 Leader 中同步数据，保持和 Leader 数据的同步。Leader 发生故障时，某个 Follower 还会成为新的 Leader。 * Offset：消费者消费的位置信息，监控数据消费到什么位置，当消费者挂掉再重新恢复的时候，可以从消费位置继续消费。 * Zookeeper：Kafka 集群能够正常工作，需要依赖于 Zookeeper，Zookeeper 帮助 Kafka 存储和管理集群信息。 ## 工作流程 Kafka集群将 Record 流存储在称为 Topic 的类别中，每个记录由一个键、一个值和一个时间戳组成。 ![kafka](../img/micro_kafka/kafka-3.png) > Kafka 是一个分布式流平台，这到底是什么意思？ * 发布和订阅记录流，类似于消息队列或企业消息传递系统。 * 以容错的持久方式存储记录流。 * 处理记录流。 Kafka 中消息是以 Topic 进行分类的，生产者生产消息，消费者消费消息，面向的都是同一个 Topic。 Topic 是逻辑上的概念，而 Partition 是物理上的概念，每个 Partition 对应于一个 log 文件，该 log 文件中存储的就是 Producer 生产的数据。 Producer 生产的数据会不断追加到该 log 文件末端，且每条数据都有自己的 Offset。 消费者组中的每个消费者，都会实时记录自己消费到了哪个 Offset，以便出错恢复时，从上次的位置继续消费。 ## 存储机制 ![kafka](../img/micro_kafka/kafka-4.png) 由于生产者生产的消息会不断追加到 log 文件末尾，为防止 log 文件过大导致数据定位效率低下，Kafka 采取了分片和索引机制。 它将每个 Partition 分为多个 Segment，每个 Segment 对应两个文件：“.index” 索引文件和 “.log” 数据文件。 这些文件位于同一文件下，该文件夹的命名规则为：topic 名-分区号。例如，first 这个 topic 有三分分区，则其对应的文件夹为 first-0，first-1，first-2。 ``` java # ls /root/data/kafka/first-0 00000000000000009014.index 00000000000000009014.log 00000000000000009014.timeindex 00000000000000009014.snapshot leader-epoch-checkpoint ``` index 和 log 文件以当前 Segment 的第一条消息的 Offset 命名。下图为 index 文件和 log 文件的结构示意图： ![kafka](../img/micro_kafka/kafka-5.png) “.index” 文件存储大量的索引信息，“.log” 文件存储大量的数据，索引文件中的元数据指向对应数据文件中 Message 的物理偏移量。 ## 生产者 ### 分区策略 >分区原因： * 方便在集群中扩展，每个 Partition 可以通过调整以适应它所在的机器，而一个 Topic 又可以有多个 Partition 组成，因此可以以 Partition 为单位读写了。 * 可以提高并发，因此可以以 Partition 为单位读写了。 分区原则：我们需要将 Producer 发送的数据封装成一个 ProducerRecord 对象。 >该对象需要指定一些参数： * topic：string 类型，NotNull。 * partition：int 类型，可选。 * timestamp：long 类型，可选。 * key：string 类型，可选。 * value：string 类型，可选。 * headers：array 类型，Nullable。 ①指明 Partition 的情况下，直接将给定的 Value 作为 Partition 的值。 ②没有指明 Partition 但有 Key 的情况下，将 Key 的 Hash 值与分区数取余得到 Partition 值。 ③既没有 Partition 有没有 Key 的情况下，第一次调用时随机生成一个整数（后面每次调用都在这个整数上自增），将这个值与可用的分区数取余，得到 Partition 值，也就是常说的 Round-Robin 轮询算法。 ### 数据可靠性保证 为保证 Producer 发送的数据，能可靠地发送到指定的 Topic，Topic 的每个 Partition 收到 Producer 发送的数据后，都需要向 Producer 发送 ACK（ACKnowledge 确认收到）。 如果 Producer 收到 ACK，就会进行下一轮的发送，否则重新发送数据。 ![kafka](../img/micro_kafka/kafka-6.png) ①副本数据同步策略 何时发送 ACK？确保有 Follower 与 Leader 同步完成，Leader 再发送 ACK，这样才能保证 Leader 挂掉之后，能在 Follower 中选举出新的 Leader 而不丢数据。 多少个 Follower 同步完成后发送 ACK？全部 Follower 同步完成，再发送 ACK。 ![kafka](../img/micro_kafka/kafka-7.png) ②ISR 采用第二种方案，所有 Follower 完成同步，Producer 才能继续发送数据，设想有一个 Follower 因为某种原因出现故障，那 Leader 就要一直等到它完成同步。 这个问题怎么解决？Leader维护了一个动态的 in-sync replica set（ISR）：和 Leader 保持同步的 Follower 集合。 当 ISR 集合中的 Follower 完成数据的同步之后，Leader 就会给 Follower 发送 ACK。 如果 Follower 长时间未向 Leader 同步数据，则该 Follower 将被踢出 ISR 集合，该时间阈值由 replica.lag.time.max.ms 参数设定。Leader 发生故障后，就会从 ISR 中选举出新的 Leader。 ③ACK 应答机制 对于某些不太重要的数据，对数据的可靠性要求不是很高，能够容忍数据的少量丢失，所以没必要等 ISR 中的 Follower 全部接受成功。 所以 Kafka 为用户提供了三种可靠性级别，用户根据可靠性和延迟的要求进行权衡，选择以下的配置。 ![kafka](../img/micro_kafka/kafka-8.png) Ack 参数配置： * 0：Producer 不等待 Broker 的 ACK，这提供了最低延迟，Broker 一收到数据还没有写入磁盘就已经返回，当 Broker 故障时有可能丢失数据。 * 1：Producer 等待 Broker 的 ACK，Partition 的 Leader 落盘成功后返回 ACK，如果在 Follower 同步成功之前 Leader 故障，那么将会丢失数据。 * -1（all）：Producer 等待 Broker 的 ACK，Partition 的 Leader 和 Follower 全部落盘成功后才返回 ACK。但是在 Broker 发送 ACK 时，Leader 发生故障，则会造成数据重复。 ④故障处理细节 ![kafka](../img/micro_kafka/kafka-9.png) LEO：每个副本最大的 Offset。HW：消费者能见到的最大的 Offset，ISR 队列中最小的 LEO。 **Follower 故障**：Follower 发生故障后会被临时踢出 ISR 集合，待该 Follower 恢复后，Follower 会 读取本地磁盘记录的上次的 HW，并将 log 文件高于 HW 的部分截取掉，从 HW 开始向 Leader 进行同步数据操作。 等该 Follower 的 LEO 大于等于该 Partition 的 HW，即 Follower 追上 Leader 后，就可以重新加入 ISR 了。 **Leader 故障**：Leader 发生故障后，会从 ISR 中选出一个新的 Leader，之后，为保证多个副本之间的数据一致性，其余的 Follower 会先将各自的 log 文件高于 HW 的部分截掉，然后从新的 Leader 同步数据。 注意：这只能保证副本之间的数据一致性，并不能保证数据不丢失或者不重复。 ### Exactly Once 语义 将服务器的 ACK 级别设置为 -1，可以保证 Producer 到 Server 之间不会丢失数据，即 At Least Once 语义。 相对的，将服务器 ACK 级别设置为 0，可以保证生产者每条消息只会被发送一次，即 At Most Once 语义。 At Least Once 可以保证数据不丢失，但是不能保证数据不重复；相对的，At Most Once 可以保证数据不重复，但是不能保证数据不丢失。 但是，对于一些非常重要的信息，比如交易数据，下游数据消费者要求数据既不重复也不丢失，即 Exactly Once 语义。 0.11 版本的 Kafka，引入了幂等性：Producer 不论向 Server 发送多少重复数据，Server 端都只会持久化一条。 即： ``` At Least Once + 幂等性 = Exactly Once ``` 要启用幂等性，只需要将 Producer 的参数中 enable.idompotence 设置为 true 即可。 开启幂等性的 Producer 在初始化时会被分配一个 PID，发往同一 Partition 的消息会附带 Sequence Number。 而 Borker 端会对 做缓存，当具有相同主键的消息提交时，Broker 只会持久化一条。 但是 PID 重启后就会变化，同时不同的 Partition 也具有不同主键，所以幂等性无法保证跨分区会话的 Exactly Once。 ## 消费者 ### 消费方式 Consumer 采用 Pull（拉取）模式从 Broker 中读取数据。 Consumer 采用 Push（推送）模式，Broker 给 Consumer 推送消息的速率是由 Broker 决定的，很难适应消费速率不同的消费者。 它的目标是尽可能以最快速度传递消息，但是这样很容易造成 Consumer 来不及处理消息，典型的表现就是拒绝服务以及网络拥塞。 而 Pull 模式则可以根据 Consumer 的消费能力以适当的速率消费消息。Pull 模式不足之处是，如果 Kafka 没有数据，消费者可能会陷入循环中，一直返回空数据。 因为消费者从 Broker 主动拉取数据，需要维护一个长轮询，针对这一点， Kafka 的消费者在消费数据时会传入一个时长参数 timeout。 如果当前没有数据可供消费，Consumer 会等待一段时间之后再返回，这段时长即为 timeout。 ### 分区分配策略 一个 Consumer Group 中有多个 Consumer，一个 Topic 有多个 Partition，所以必然会涉及到 Partition 的分配问题，即确定哪个 Partition 由哪个 Consumer 来消费。 Kafka 有两种分配策略，一个是 RoundRobin，一个是 Range，默认为Range，当消费者组内消费者发生变化时，会触发分区分配策略（方法重新分配）。 ①RoundRobin ![kafka](../img/micro_kafka/kafka-10.png) RoundRobin 轮询方式将分区所有作为一个整体进行 Hash 排序，消费者组内分配分区个数最大差别为 1，是按照组来分的，可以解决多个消费者消费数据不均衡的问题。 但是，当消费者组内订阅不同主题时，可能造成消费混乱，如下图所示，Consumer0 订阅主题 A，Consumer1 订阅主题 B。 ![kafka](../img/micro_kafka/kafka-11.png) 将 A、B 主题的分区排序后分配给消费者组，TopicB 分区中的数据可能分配到 Consumer0 中。 ②Range ![kafka](../img/micro_kafka/kafka-12.png) Range 方式是按照主题来分的，不会产生轮询方式的消费混乱问题。 但是，如下图所示，Consumer0、Consumer1 同时订阅了主题 A 和 B，可能造成消息分配不对等问题，当消费者组内订阅的主题越多，分区分配可能越不均衡。 ![kafka](../img/micro_kafka/kafka-13.png) ### Offset 的维护 由于 Consumer 在消费过程中可能会出现断电宕机等故障，Consumer 恢复后，需要从故障前的位置继续消费。 所以 Consumer 需要实时记录自己消费到了哪个 Offset，以便故障恢复后继续消费。 Kafka 0.9 版本之前，Consumer 默认将 Offset 保存在 Zookeeper 中，从 0.9 版本开始，Consumer 默认将 Offset 保存在 Kafka 一个内置的 Topic 中，该 Topic 为 __consumer_offsets。 ## 总结 上面和大家一起深入探讨了 Kafka 的架构，比较偏重理论和基础，这是掌握 Kafka 的必要内容，接下来我会以代码和实例的方式，更新 Kafka 有关 API 以及事务、拦截器、监控等高级篇，让大家彻底理解并且会用 Kafka。 ================================================ FILE: docs/micro/redis_cluster.md ================================================ # Redis Cluster 原理 >Redis 缓存作为使用最多的缓存工具被各大厂商争相使用。通常我们会使用单体的 Redis 应用作为缓存服务，为了保证其高可用还会使用主从模式（Master-Slave），又或者是读写分离的设计。但是当缓存数据量增加以后，无法用单体服务器承载缓存服务时，就需要对缓存服务进行扩展。 将需要缓存的数据切分成不同的分区，将数据分区放到不同的服务器中，用分布式的缓存来承载高并发的缓存访问。恰好 Redis Cluster 方案刚好支持这部分功能。 今天就来一起看看 Redis Cluster 的核心原理和实践： * Redis Cluster 实现数据分区 * 分布式缓存节点之间的通讯 * 请求分布式缓存的路由 * 缓存节点的扩展和收缩 * 故障发现和恢复 ## Redis Cluster 实现数据分区 正如开篇中提到的，分布式数据库要解决的就是将整块数据，按照规则分配到多个缓存节点，解决的是单个缓存节点处理数量大的问题。 如果要将这些数据进行拆分，并且存放必须有一个算法。例如：哈希算法和哈希一致性算法，这些比较经典的算法。 Redis Cluster 则采用的是虚拟槽分区算法。其中提到了槽（Slot）的概念。这个槽是用来存放缓存信息的单位，在 Redis 中将存储空间分成了 16384 个槽，也就是说 Redis Cluster 槽的范围是 0 -16383（2^4 * 2^10）。 缓存信息通常是用 Key-Value 的方式来存放的，在存储信息的时候，集群会对 Key 进行 CRC16 校验并对 16384 取模（slot = CRC16(key)%16383）。 得到的结果就是 Key-Value 所放入的槽，从而实现自动分割数据到不同的节点上。然后再将这些槽分配到不同的缓存节点中保存。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-1.png) 如图 1 (Redis 集群中的数据分片)所示，假设有三个缓存节点分别是 1、2、3。Redis Cluster 将存放缓存数据的槽（Slot）分别放入这三个节点中： * 缓存节点 1 存放的是（0-5000）Slot 的数据。 * 缓存节点 2 存放的是（5001-10000）Slot 的数据。 * 缓存节点 3 存放的是（10000-16383）Slot 的数据。 此时 Redis Client 需要根据一个 Key 获取对应的 Value 的数据，首先通过 CRC16(key)%16383 计算出 Slot 的值，假设计算的结果是 5002。 将这个数据传送给 Redis Cluster，集群接受到以后会到一个对照表中查找这个 Slot=5002 属于那个缓存节点。 发现属于“缓存节点 2”，于是顺着红线的方向调用缓存节点 2 中存放的 Key-Value 的内容并且返回给 Redis Client。 ## 分布式缓存节点之间的通讯 如果说 Redis Cluster 的虚拟槽算法解决的是数据拆分和存放的问题，那么存放缓存数据的节点之间是如何通讯的，就是接下来我们要讨论的。 缓存节点中存放着缓存的数据，在 Redis Cluster 的分布式部署下，缓存节点会被分配到一台或者多台服务器上。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-2.png) 图 2：新上线的缓存节点 2 和缓存节点 1 进行通讯 缓存节点的数目也有可能根据缓存数据量和支持的并发进行扩展。如图 2 所示，假设 Redis Cluster 中存在“缓存节点 1”，此时由于业务扩展新增了“缓存节点 2”。 新加入的节点会通过 Gossip 协议向老节点，发出一个“Meet 消息”。收到消息以后“缓存节点 1”，会礼貌地回复一个“Pong 消息”。 此后“缓存节点 2”会定期发送给“缓存节点 1” 一个“Ping 消息”，同样的“缓存节点 1”每次都会回复“Pong 消息”。 上面这个例子说明了，在 Redis Cluster 中缓存节点之间是通过 Gossip 协议进行通讯的。 其实节点之间通讯的目的是为了维护节点之间的元数据信息。这个元数据就是每个节点包含哪些数据，是否出现故障。 节点之间通过 Gossip 协议不断相互交互这些信息，就好像一群人在一起八卦一样，没有多久每个节点就知道其他所有节点的情况了，这个情况就是节点的元数据。 整个传输过程大致分为以下几点： * Redis Cluster 的每个缓存节点都会开通一个独立的 TCP 通道，用于和其他节点通讯。 * 有一个节点定时任务，每隔一段时间会从系统中选出“发送节点”。这个“发送节点”按照一定频率，例如：每秒 5 次，随机向最久没有通讯的节点发起 Ping 消息。 * 接受到 Ping 消息的节点会使用 Pong 消息向“发送节点”进行回复。 不断重复上面行为，让所有节点保持通讯。他们之间通讯是通过 Gossip 协议实现的。 从类型上来说其分为了四种，分别是： * Meet 消息，用于通知新节点加入。就好像上面例子中提到的新节点上线会给老节点发送 Meet 消息，表示有“新成员”加入。 * Ping 消息，这个消息使用得最为频繁，该消息中封装了自身节点和其他节点的状态数据，有规律地发给其他节点。 * Pong 消息，在接受到 Meet 和 Ping 消息以后，也将自己的数据状态发给对方。同时也可以对集群中所有的节点发起广播，告知大家的自身状态。 * Fail 消息，如果一个节点下线或者挂掉了，会向集群中广播这个消息。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-3.png) 图 3：Gossip 协议结构 Gossip 协议的结构如图 3 所示，有其中 type 定义了消息的类型，例如：Meet、Ping、Pong、Fail 等消息。 另外有一个 myslots 的数组定义了节点负责的槽信息。每个节点发送 Gossip 协议给其他节点最重要的就是将该信息告诉其他节点。另外，消息体通过 clusterMsgData 对象传递消息征文。 ## 请求分布式缓存的路由 对内，分布式缓存的节点通过 Gossip 协议互相发送消息，为了保证节点之间了解对方的情况。 那么对外来说，一个 Redis 客户端如何通过分布式节点获取缓存数据，就是分布式缓存路由要解决的问题了。 上文提到了 Gossip 协议会将每个节点管理的槽信息发送给其他节点，其中用到了 unsigned char myslots[CLUSTER_SLOTS/8] 这样一个数组存放每个节点的槽信息。 myslots 属性是一个二进制位数组（bit array），其中 CLUSTER_SLOTS 为 16384。 这个数组的长度为 16384/8=2048 个字节，由于每个字节包含 8 个 bit 位（二进制位），所以共包含 16384 个 bit，也就是 16384 个二进制位。 每个节点用 bit 来标识自己是否拥有某个槽的数据。如图 4 所示，假设这个图表示节点 A 所管理槽的情况。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-4.png) 图 4：通过二进制数组存放槽信息 0、1、2 三个数组下标就表示 0、1、2 三个槽，如果对应的二进制值是 1，表示该节点负责存放 0、1、2 三个槽的数据。同理，后面的数组下标位 0 就表示该节点不负责存放对应槽的数据。 用二进制存放的优点是，判断的效率高，例如对于编号为 1 的槽，节点只要判断序列的第二位，时间复杂度为 O（1）。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-5.png) 图 5：接受节点把节点槽的对应信息保存在本地 如图 5 所示，当收到发送节点的节点槽信息以后，接受节点会将这些信息保存到本地的 clusterState 的结构中，其中 Slots 的数组就是存放每个槽对应哪些节点信息。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-6.png) 图 6：ClusterStatus 结构以及槽与节点的对应 如图 6 所示，ClusterState 中保存的 Slots 数组中每个下标对应一个槽，每个槽信息中对应一个 clusterNode 也就是缓存的节点。 这些节点会对应一个实际存在的 Redis 缓存服务，包括 IP 和 Port 的信息。 Redis Cluster 的通讯机制实际上保证了每个节点都有其他节点和槽数据的对应关系。 Redis 的客户端无论访问集群中的哪个节点都可以路由到对应的节点上，因为每个节点都有一份 ClusterState，它记录了所有槽和节点的对应关系。 下面来看看 Redis 客户端是如何通过路由来调用缓存节点的： ![redis-cluster](../img/micro_redis_cluster/redis-cluster-7.png) 图 7：MOVED 重定向请求 如图 7 所示，Redis 客户端通过 CRC16(key)%16383 计算出 Slot 的值，发现需要找“缓存节点 1”读/写数据，但是由于缓存数据迁移或者其他原因导致这个对应的 Slot 的数据被迁移到了“缓存节点 2”上面。 那么这个时候 Redis 客户端就无法从“缓存节点 1”中获取数据了。 但是由于“缓存节点 1”中保存了所有集群中缓存节点的信息，因此它知道这个 Slot 的数据在“缓存节点 2”中保存，因此向 Redis 客户端发送了一个 MOVED 的重定向请求。 这个请求告诉其应该访问的“缓存节点 2”的地址。Redis 客户端拿到这个地址，继续访问“缓存节点 2”并且拿到数据。 上面的例子说明了，数据 Slot 从“缓存节点 1”已经迁移到“缓存节点 2”了，那么客户端可以直接找“缓存节点 2”要数据。 那么如果两个缓存节点正在做节点的数据迁移，此时客户端请求会如何处理呢？ ![redis-cluster](../img/micro_redis_cluster/redis-cluster-8.png) 图 8：ASK 重定向请求 如图 8 所示，Redis 客户端向“缓存节点 1”发出请求，此时“缓存节点 1”正向“缓存节点 2”迁移数据，如果没有命中对应的 Slot，它会返回客户端一个 ASK 重定向请求并且告诉“缓存节点 2”的地址。 客户端向“缓存节点 2”发送 Asking 命令，询问需要的数据是否在“缓存节点 2”上，“缓存节点 2”接到消息以后返回数据是否存在的结果。 ## 缓存节点的扩展和收缩 作为分布式部署的缓存节点总会遇到缓存扩容和缓存故障的问题。这就会导致缓存节点的上线和下线的问题。 由于每个节点中保存着槽数据，因此当缓存节点数出现变动时，这些槽数据会根据对应的虚拟槽算法被迁移到其他的缓存节点上。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-9.png) 图 9：分布式缓存扩容 如图 9 所示，集群中本来存在“缓存节点 1”和“缓存节点 2”，此时“缓存节点 3”上线了并且加入到集群中。 此时根据虚拟槽的算法，“缓存节点 1”和“缓存节点 2”中对应槽的数据会应该新节点的加入被迁移到“缓存节点 3”上面。 针对节点扩容，新建立的节点需要运行在集群模式下，因此新建节点的配置最好与集群内其他节点配置保持一致。 新节点加入到集群的时候，作为孤儿节点是没有和其他节点进行通讯的。因此，其会采用 cluster meet 命令加入到集群中。 在集群中任意节点执行 cluster meet 命令让新节点加入进来。假设新节点是 192.168.1.1 5002，老节点是 192.168.1.1 5003，那么运行以下命令将新节点加入到集群中。 ``` 192.168.1.1 5003> cluster meet 192.168.1.1 5002 ``` 这个是由老节点发起的，有点老成员欢迎新成员加入的意思。新节点刚刚建立没有建立槽对应的数据，也就是说没有缓存任何数据。 如果这个节点是主节点，需要对其进行槽数据的扩容；如果这个节点是从节点，就需要同步主节点上的数据。总之就是要同步数据。 ![redis-cluster](../img/micro_redis_cluster/redis-cluster-10.png) 图 10：节点迁移槽数据的过程 如图 10 所示，由客户端发起节点之间的槽数据迁移，数据从源节点往目标节点迁移： * 客户端对目标节点发起准备导入槽数据的命令，让目标节点准备好导入槽数据。这里使用 cluster setslot {slot} importing {sourceNodeId} 命令。 * 之后对源节点发起送命令，让源节点准备迁出对应的槽数据。使用命令 cluster setslot {slot} importing {sourceNodeId}。 * 此时源节点准备迁移数据了，在迁移之前把要迁移的数据获取出来。通过命令 cluster getkeysinslot {slot} {count}。Count 表示迁移的 Slot 的个数。 * 然后在源节点上执行，migrate {targetIP} {targetPort} “” 0 {timeout} keys{keys} 命令，把获取的键通过流水线批量迁移到目标节点。 * 重复 3 和 4 两步不断将数据迁移到目标节点。目标节点获取迁移的数据。 * 完成数据迁移以后目标节点，通过 cluster setslot {slot} node {targetNodeId} 命令通知对应的槽被分配到目标节点，并且广播这个信息给全网的其他主节点，更新自身的槽节点对应表。 既然有缓存服务器的上线操作，那么也有下线的操作。下线操作正好和上线操作相反，将要下线缓存节点的槽数据分配到其他的缓存主节点中。 迁移的过程也与上线操作类似，不同的是下线的时候需要通知全网的其他节点忘记自己，此时通过命令 cluster forget{downNodeId} 通知其他的节点。 当节点收到 forget 命令以后会将这个下线节点放到仅用列表中，那么之后就不用再向这个节点发送 Gossip 的 Ping 消息了。 不过这个仅用表的超时时间是 60 秒，超过了这个时间，依旧还会对这个节点发起 Ping 消息。 不过可以使用 redis-trib.rb del-node{host:port} {donwNodeId} 命令帮助我们完成下线操作。 尤其是下线的节点是主节点的情况下，会安排对应的从节点接替主节点的位置。 ## 故障发现和恢复 前面在谈到缓存节点扩展和收缩是提到，缓存节点收缩时会有一个下线的动作。 有些时候是为了节约资源，或者是计划性的下线，但更多时候是节点出现了故障导致下线。 针对下线故障来说有两种下线的确定方式： **主观下线**：当节点 1 向节点 2 例行发送 Ping 消息的时候，如果节点 2 正常工作就会返回 Pong 消息，同时会记录节点 1 的相关信息。 同时接受到 Pong 消息以后节点 1 也会更新最近一次与节点 2 通讯的时间。 如果此时两个节点由于某种原因断开连接，过一段时间以后节点 1 还会主动连接节点 2，如果一直通讯失败，节点 1 中就无法更新与节点 2 最后通讯时间了。 此时节点 1 的定时任务检测到与节点 2 最好通讯的时间超过了 cluster-node-timeout 的时候，就会更新本地节点状态，把节点 2 更新为主观下线。 这里的 cluster-node-timeout 是节点挂掉被发现的超时时间，如果超过这个时间还没有获得节点返回的 Pong 消息就认为该节点挂掉了。 这里的主观下线指的是，节点 1 主观的认为节点 2 没有返回 Pong 消息，因此认为节点 2 下线。 只是节点 1 的主观认为，有可能是节点 1 与节点 2 之间的网络断开了，但是其他的节点依旧可以和节点 2 进行通讯，因此主观下线并不能代表某个节点真的下线了。 **客观下线**：由于 Redis Cluster 的节点不断地与集群内的节点进行通讯，下线信息也会通过 Gossip 消息传遍所有节点。 因此集群内的节点会不断收到下线报告，当半数以上持有槽的主节点标记了某个节点是主观下线时，便会触发客观下线的流程。 也就是说当集群内的半数以上的主节点，认为某个节点主观下线了，才会启动这个流程。 这个流程有一个前提，就是直针对主节点，如果是从节点就会忽略。也就是说集群中的节点每次接受到其他节点的主观下线是都会做以下的事情。 将主观下线的报告保存到本地的 ClusterNode 的结构中，并且针对主观下线报告的时效性进行检查，如果超过 cluster-node-timeout*2 的时间，就忽略这个报告。 否则就记录报告内容，并且比较被标记下线的主观节点的报告数量大于等于持有槽的主节点数量的时候，将其标记为客观下线。 同时向集群中广播一条 Fail 消息，通知所有的节点将故障节点标记为客观下线，这个消息指包含故障节点的 ID。 此后，群内所有的节点都会标记这个节点为客观下线，通知故障节点的从节点出发故障转移的流程，也就是故障的恢复。 说白了，客观下线就是整个集群中有一半的节点都认为某节点主观下线了，那么这个节点就被标记为客观下线了。 如果某个主节点被认为客观下线了，那么需要从它的从节点中选出一个节点替代主节点的位置。 此时下线主节点的所有从节点都担负着恢复义务，这些从节点会定时监测主节点是否下线。 一旦发现下线会走如下的恢复流程： ①资格检查，每个节点都会检查与主节点断开的时间。如果这个时间超过了 cluster-node-timeout*cluster-slave-validity-factor（从节点有效因子，默认为 10），那么就没有故障转移的资格。 也就是说这个从节点和主节点断开的太久了，很久没有同步主节点的数据了，不适合成为新的主节点，因为成为主节点以后其他的从节点回同步自己的数据。 ②触发选举，通过了上面资格的从节点都可以触发选举。但是出发选举是有先后顺序的，这里按照复制偏移量的大小来判断。 这个偏移量记录了执行命令的字节数。主服务器每次向从服务器传播 N 个字节时就会将自己的复制偏移量+N，从服务在接收到主服务器传送来的 N 个字节的命令时，就将自己的复制偏移量+N。 复制偏移量越大说明从节点延迟越低，也就是该从节点和主节点沟通更加频繁，该从节点上面的数据也会更新一些，因此复制偏移量大的从节点会率先发起选举。 ③发起选举，首先每个主节点会去更新配置纪元（clusterNode.configEpoch），这个值是不断增加的整数。 在节点进行 Ping/Pong 消息交互式也会更新这个值，它们都会将最大的值更新到自己的配置纪元中。 这个值记录了每个节点的版本和整个集群的版本。每当发生重要事情的时候，例如：出现新节点，从节点精选。都会增加全局的配置纪元并且赋给相关的主节点，用来记录这个事件。 说白了更新这个值目的是，保证所有主节点对这件“大事”保持一致。大家都统一成一个配置纪元（一个整数），表示大家都知道这个“大事”了。 更新完配置纪元以后，会想群内发起广播选举的消息（FAILOVER_AUTH_REQUEST）。并且保证每个从节点在一次配置纪元中只能发起一次选举。 ④投票选举，参与投票的只有主节点，从节点没有投票权，超过半数的主节点通过某一个节点成为新的主节点时投票完成。 如果在 cluster-node-timeout*2 的时间内从节点没有获得足够数量的票数，本次选举作废，进行第二轮选举。 这里每个候选的从节点会收到其他主节点投的票。在第2步领先的从节点通常此时会获得更多的票，因为它触发选举的时间更早一些。 获得票的机会更大，也是由于它和原主节点延迟少，理论上数据会更加新一点。 ⑤当满足投票条件的从节点被选出来以后，会触发替换主节点的操作。新的主节点别选出以后，删除原主节点负责的槽数据，把这些槽数据添加到自己节点上。 并且广播让其他的节点都知道这件事情，新的主节点诞生了。 ## 总结 本文通过 Redis Cluster 提供了分布式缓存的方案为出发点，针对此方案中缓存节点的分区方式进行了描述。 虚拟槽的分区算法，将整块数据分配到了不同的缓存节点，通过 Slot 和 Node 的对应关系让数据找到节点的位置。 对于分布式部署的节点，需要通过 Gossip 协议进行 Ping、Pong、Meet、Fail 的通讯，达到互通有无的目的。 当客户端调用缓存节点数据的时候通过 MOVED 和 ASKED 重定向请求找到正确的缓存节点。 并且介绍了在缓存扩容和收缩时需要注意的处理流程，以及数据迁移的方式。 最后，讲述如何发现故障（主观下线和客观下线）以及如何恢复故障（选举节点）的处理流程。 ================================================ FILE: docs/micro/spring-cloud-micro.md ================================================ # 基于SpringCloud分布式架构 ## 为什么要使用分布式架构 * Spring Cloud 专注于提供良好的开箱即用经验的典型用例和可扩展性机制覆盖 * 分布式/版本化配置 * 服务注册和发现 * 路由 * Service-to-Service 调用 * 负载均衡 * 断路器 * 分布式消息传递 这是分布式的优点，这样看起来可能比较抽象，举个例子来说，对于单体服务来说，如果我想更新订单中的某个功能，我是不是需要重启整个服务。 这个时候就会导致整个项目都处于不可用状态，或者在处理订单的时候由于程序代码写的有问题，导致死锁了，这个时候也会导致整个服务处于宕机专改，容错率很差。 但是分布式不同，如上图所示，订单服务、售后服务、用户服务都是独立的服务，如果需要更新订单服务或者订单服务发生死锁，受影响的只会是订单服务，售后服务与用户服务还是可以正常工作的，这就是分布式相对单体来说最大的优势之一。 ## 分布式基础组件 **Spring Cloud Config**：配置管理工具包，让你可以把配置放到远程服务器，集中化管理集群配置，目前支持本地存储、Git 以及 Subversion。 **Spring Cloud Bus**：事件、消息总线，用于在集群（例如，配置变化事件）中传播状态变化，可与 Spring Cloud Config 联合实现热部署。 **Eureka**：云端服务发现，一个基于 REST 的服务，用于定位服务，以实现云端中间层服务发现和故障转移。 **Hystrix**：熔断器，容错管理工具，旨在通过熔断机制控制服务和第三方库的节点,从而对延迟和故障提供更强大的容错能力。 **Zuul**：Zuul 是在云平台上提供动态路由,监控,弹性,安全等边缘服务的框架。Zuul 相当于是设备和 Netflix 流应用的 Web 网站后端所有请求的前门。 **Archaius**：配置管理 API，包含一系列配置管理 API，提供动态类型化属性、线程安全配置操作、轮询框架、回调机制等功能。 **Consul**：封装了 Consul 操作，Consul 是一个服务发现与配置工具，与 Docker 容器可以无缝集成。 **Spring Cloud for Cloud Foundry**：通过 Oauth2 协议绑定服务到 CloudFoundry，CloudFoundry 是 VMware 推出的开源 PaaS 云平台。 **Spring Cloud Sleuth**：日志收集工具包，封装了 Dapper 和 log-based 追踪以及 Zipkin 和 HTrace 操作，为 Spring Cloud 应用实现了一种分布式追踪解决方案。 **Spring Cloud Data Flow**：大数据操作工具，作为 Spring XD 的替代产品，它是一个混合计算模型，结合了流数据与批量数据的处理方式。 **Spring Cloud Security**：基于 Spring Security 的安全工具包，为你的应用程序添加安全控制。 **Spring Cloud Zookeeper**：操作 Zookeeper 的工具包，用于使用 Zookeeper 方式的服务发现和配置管理。 **Spring Cloud Stream**：数据流操作开发包，封装了与 Redis、Rabbit、Kafka 等发送接收消息。 **Spring Cloud CLI**：基于 Spring Boot CLI，可以让你以命令行方式快速建立云组件。 **Ribbon**：提供云端负载均衡，有多种负载均衡策略可供选择，可配合服务发现和断路器使用。 **Turbine**：Turbine 是聚合服务器发送事件流数据的一个工具，用来监控集群下 Hystrix 的 Metrics 情况。 **Feign**：Feign 是一种声明式、模板化的 HTTP 客户端。 **Spring Cloud Task**：提供云端计划任务管理、任务调度。 **Spring Cloud Connectors**：便于云端应用程序在各种 PaaS 平台连接到后端，如：数据库和消息代理服务。 **Spring Cloud Cluster**：提供 Leadership 选举，如：Zookeeper，Redis，Hazelcast，Consul 等常见状态模式的抽象和实现。 **Spring Cloud Starters**：Spring Boot 式的启动项目，为 Spring Cloud 提供开箱即用的依赖管理。 >我们常用的组件： * Spring Cloud Config * Spring Cloud Bus * Hystrix * Eureka * Zuul * Ribbon * Feign ### Eureka Eureka 属于 Spring Cloud Netflix 下的组件之一，主要负责服务的注册与发现，何为注册与发现？ 在刚刚我们分析的分布式中存在这一个问题，那就是订单服务与用户服务被独立了，那么他们怎么进行通信呢？比如在订单服务中获取用户的基础信息，这个时候我们需要怎么办？ 如果按照上面的架构图，直接去数据库获取就可以了，因为服务虽然独立了，但是数据库还是共享的，所以直接查询数据库就能得到结果，如果我们将数据库也拆分了呢？这个时候我们该怎么办呢？ 有人想到了，服务调用，服务调用是不是需要 IP 和端口才可以，那问题来了，对于订单服务来说，我怎么知道用户服务的 IP 和端口呢？在订单服务中写死吗？如果用户服务的端口发生改变了呢？ 这个时候 Eureka 就出来了，他就是为了解决服务的通信问题，每个服务都可以将自己的信息注册到 Eureka 中，比如 IP、端口、服务名等信息，这个时候如果订单服务想要获取用户服务的信息，只需要去 Eureka 中获取即可。 请看下图： ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-1.png) 这就是 Eureka 的主要功能，也是我们使用中的最值得注意的，他让服务之间的通信变得更加的简单灵活。 ### Spring Cloud Config Spring Cloud Config 为分布式系统中的外部配置提供服务器和客户端支持。使用 Config Server，您可以在所有环境中管理应用程序的外部属性。 客户端和服务器上的概念映射与 Spring Environment 和 PropertySource 抽象相同，因此它们与 Spring 应用程序非常契合，但可以与任何以任何语言运行的应用程序一起使用。 随着应用程序通过从开发人员到测试和生产的部署流程，您可以管理这些环境之间的配置，并确定应用程序具有迁移时需要运行的一切。 服务器存储后端的默认实现使用 Git，因此它轻松支持标签版本的配置环境，以及可以访问用于管理内容的各种工具。可以轻松添加替代实现，并使用 Spring 配置将其插入。 简单点来说集中来管理每个服务的配置文件，将配置文件与服务分离，这么多的目的是什么？ 举个简单的栗子，我们配置文件中肯定会存在数据库的连接信息，Redis 的连接信息，我们的环境是多样的，有开发环境、测试环境、预发布环境、生产环境。 每个环境对应的连接信息肯定是不相同的，难道每次发布的时候都要去修改一下服务中的配置文件？ 我能不能将这些变动较大的配置集中管理，不同环境的管理者分别对他们进行修改，就不需要再服务中做改动了，Config 就做到了。 ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-2.png) 这就是 Config 的大致架构，所有的配置文件都集中交给 Config 管理，拿 Config 怎么管理这些配置文件呢？ 你可以将每个环境的配置文件存放再一个位置，比如 Lgitlab、SVN、本地等等，Config 会根据根据你设置的位置读取配置文件进行管理，然后其他服务启动的时候直接到 Config 配置中心获取对应的配置文件即可。 这样开发人员只需要关注 -dev 的配置文件，测试人员只需要关注 -test 的配置文件，完全和服务解耦，你值得拥有。 ### Netflix Zuul（网关） 路由在微服务体系结构的一个组成部分。例如，/可以映射到您的 Web 应用程序，/api/users 映射到用户服务，并将 /api/shop 映射到商店服务。Zuul 是 Netflix 的基于 JVM 的路由器和服务器端负载均衡器。 >Netflix 使用 Zuul 进行以下操作： * 认证 -洞察 * 压力测试 * 金丝雀测试 * 动态路由 * 服务迁移 * 负载脱落 * 安全 * 静态响应处理 * 主动/主动流量管理 我们在日常开发过程中并不会使用那么多，基本上就是认证、动态路由、安全等等，我画了一张关于网关的架构图，请看： ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-3.png) 注意：Nginx 只能为我们做反向代理，不能做到权限认证，网关不但可以做到代理，也能做到权限认证、甚至还能做限流，所以我们要做分布式项目，少了他可不行。 ### Spring Cloud Bus ``` yml application.yml spring: datasource: username: root password: 123456 url: jdbc:mysql://localhost:3306/test driver-class-name: com.mysql.cj.jdbc.Driver ``` 比如上面这行配置大家都应该很熟悉，这是数据库的连接信息，如果它发生改变了怎么办呢？ 我们都知道，服务启动的时候会去 Config 配置中心拉取配置信息，但是启动完成之后修改了配置文件我们应该怎么办呢，重启服务器吗？ 我们可以通过 Spring Cloud Bus 来解决这个问题，Spring Cloud Bus 将轻量级消息代理链接到分布式系统的节点。然后可以将其用于广播状态更改（例如，配置更改）或其他管理指令。 我们可以通过 Spring Cloud Bus 来解决这个问题，Spring Cloud Bus 将轻量级消息代理链接到分布式系统的节点。然后可以将其用于广播状态更改（例如，配置更改）或其他管理指令。 这个需要我们有一点的 MQ 基础，不管是 RabbitMQ 还是 Kafka，都可以。 Bus 的基本原理就是：配置文件发生改变时，Config 会发出一个 MQ，告诉服务，配置文件发生改变了，并且还发出了改变的哪些信息，这个时候服务只需要根据 MQ 的信息做实时修改即可。 这是一个很简单的原理，理解起来可能也不会怎么难，画个图来理解一下： ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-4.png) 大致流程就是这样，核心就是通过 MQ 机制实现不重启服务也能做到配置文件的改动，这方便了运维工程师，不用每次修改配置文件的时候都要去重启一遍服务的烦恼。 ### Feign >Feign 是一个声明式的 Web 服务客户端。这使得 Web 服务客户端的写入更加方便 要使用 Feign 创建一个界面并对其进行注释。 它具有可插入注释支持，包括 Feign 注释和 JAX-RS 注释。Feign 还支持可插拔编码器和解码器。 Spring Cloud 增加了对 Spring MVC 注释的支持，并使用 Spring Web 中默认使用的 HttpMessageConverters。 Spring Cloud 集成 Ribbon 和 Eureka 以在使用 Feign 时提供负载均衡的 HTTP 客户端。 ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-5.png) Feign 基于 Rest 风格，简单易懂，他的底层是对 HttpClient 进行了一层封装，使用十分方便。 ### Netflix Hystrix（熔断） >Hystrix 支持回退的概念：当电路断开或出现错误时执行的默认代码路径。要为给定的 @FeignClient 启用回退，请将 Fallback 属性设置为实现回退的类名。 我们可以改造一下刚刚的调用架构： ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-6.png) 在这里我部署了一台备用服务器，当用户服务宕机了之后，订单服务进行远程调用的时候可以进入备用服务，这样就不会导致系统崩溃。 ### MQ（消息中间件） 我现在这里有一个需求，修改密码，修改密码需要发送短信验证码，发送短信在短信服务中，修改密码在用户服务中，这个时候就会出现服务调用。 而且我们知道，发送短信一般都是调用第三方的接口，那比如阿里的，既然牵扯到调用，那么就会存在很多不确定因素，比如网络问题。 假如，用户再点击发送短信验证码到时候用户服务调用短信服务，但是在短信服务中执行调用阿里的接口花费了很长的时间。 这个时候就会导致用户服务调短信服务超时，会返回给用户失败，但是，短信最后又发出去了，这种问题怎么解决呢？ 我们可以通过消息中间件来实现，使用异步讲给用户的反馈和发送短信分离，只要用户点了发送短信，直接返回成功，然后再启动发送验证码，60 秒重发一下，就算发送失败，用户还可以选择重新发送。 MQ 不但可以解耦服务，它还可以用来削峰，提高系统的性能，是一个不错的选择。 ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-7.png) ### 分布式事务 既然我们使用了分布式架构，那么有一点是我们必须要注意的，那就是事务问题。 如果一个服务的修改依赖另外一个服务的操作，这个时候如果操作不慎，就会导致可怕的后果。 举个例子，两个服务：钱包服务（用于充值提现）、交易钱包服务（用于交易），我现在想从钱包服务中转 1000 元到交易钱包服务中，我们应该如何保证他们数据的一致性呢？ 我这里有两种方案，第一种是通过 MQ 来保证一致性，另外一种就是通过分布式事务来确保一致性。 #### MQ 确保一致性 * 生成一个订单表，记录着转入转出的状态。 * 向 MQ 发送一条确认消息。 * 开启本地事务，执行转出操操作，并且提交事务。 交易钱包服务：接收 MQ 的消息，进行转入操作（此操作需要 Ack 确认机制的支持）。 系统中会一直定时扫描订单中状态，没有成功的就做补偿机制或者重试机制，这个不是唯一要求。 ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-8.png) 以上就是 MQ 确保分布式事务的大致思路，不是唯一，仅供参考。 #### Seata（分布式事务） Seata 有三个基本组成部分： * **事务协调器（TC）**：维护全局事务和分支事务的状态，驱动全局提交或回滚。 * **事务管理器 TM**：定义全局事务的范围：开始全局事务，提交或回滚全局事务。 * **资源管理器（RM）**：管理分支事务正在处理的资源，与 TC 进行对话以注册分支事务并报告分支事务的状态，并驱动分支事务的提交或回滚。 ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-9.png) Seata 管理的分布式事务的典型生命周期： * TM 要求 TC 开始一项新的全球交易。TC 生成代表全局事务的 XID。 * XID 通过微服务的调用链传播。 * RM 将本地事务注册为 XID 到 TC 的相应全局事务的分支。 * TM 要求 TC 提交或回滚 XID 的相应全局事务。 * TC 驱动 XID 对应的全局事务下的所有分支事务以完成分支提交或回滚。 ## 完整的分布式架构 完整的分布式架构如下图： ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-10.png) 这就是一套分布式基本的架构，请求从浏览器发出，经过 Nginx 反向代理到 Zuul 网关。 网关经过权限校验、然后分别转发到对应的服务中，每个服务都有自己独立的数据库，如果需要跨库查询的时候就需要用到分布式的远程调用（Feign）。 虽然这里我将服务拆分了，但是有一点需要注意的是网关，网关承载着所有的请求，如果请求过大会发生什么呢？ 服务宕机，所以一般情况下，网关都是集群部署，不止网关可以集群，其他的服务都可以做集群配置，比如：注册中心、Redis、MQ 等等都可以。 那我们将这个流程图再改良一下： ![spring-cloud-micro](../img/spring_cloud_micro/spring-cloud-micro-11.png) 现在这套架构就是比较程数的一套了，不管是性能还是稳定能，都是杠杠的，技术选择性的会也开得差不多了，最后技术总监做了一个总结。 ## 总结 ### 单体服务与分布式服务区别 区别|传统单体架构|分布式服务架构| --|--|-- 新功能开发|需要时间|容易开发和实现 部署|不经常且容易部署|经常发布，部署复杂 隔离性|故障影响范围大|故障影响范围小 架构设计|难度小|难度级数增大 系统性能|响应时间快，吞吐量小|响应时间慢，吞吐量大 系统运维|运维简单|运维复杂 新手上手|学习曲线大(业务逻辑)|学习曲线大(构架逻辑) 技术|技术单一且封闭|技术多样且开放 测试和查错|简单|复杂 系统扩展性|扩展性差|扩展性很好 系统管理|重点在于开放成本|重点在于服务治理和调度 ### 什么时候使用分布式/集群？ 总结如下几点： * 单机无法支持的时候。 * 想要更好的隔离性（功能与功能）。 * 想要更好用户体验的时候。 * 想要更好的扩展性。 * 想要更快的响应，更搞得吞吐量。 ### 使用分布式注意事项 虽然现在分布式技术已经十分成熟，但是里面的坑不是一点两点，比如：==如何保证分布式事务的一致性、如何保证服务调用的幂等性、如何保证消息的幂等性、如何设置熔断（服务的降级），如何保证服务的健壮性等等，==这些都是一直需要关注的问题，只有解决了这些问题，你的分布式架构才能真正的立于不败之地。 ### 关于组件停更消息 目前注册中心 Eureka、网关 Zuul，Feign 都相继停更了，停更不代表不能使用，只是除了 Bug 可能不会主动修复，所以这个时候我们可能就需要选择另外的组件了。 注册中心可以使用 Consul、Nacos，Zookeeper，网关则可以通过 Gateway 替换，OpenFeign 替换 Fiegn。 所以也没必要听到组件停更的消息就担心 Cloud 会不会凉，放心，它至少最近几年是不会凉的。 ================================================ FILE: docs/net/c_core_safety.md ================================================ # 让你的ASP.NET Core应用程序更安全 对于ASP.NET Core应用程序，除了提供认证和授权机制来保证服务的安全性，还需要考虑下面的一些安全因素： * CSRF * 强制HTTPS * 安全的HTTP Headers ### CSRF ASP.NET Core通过AntiForgeryToken来阻止CSRF攻击，一般来说，当用户做表单提交的时候，表单中隐藏的token也会被提交到服务端，与此同时cookie中包含的另一半token也被提交到服务端，服务端通过合并两份token来验证客户端的数据是否有效。 例如在ASP.NET Core中通过下面的方式渲染表单: ``` c#
``` 这样会生成下面的html，表单会包含一个隐藏的token ``` html
``` 服务端的代码通过在对应的action上标记ValidateAntiForgeryToken来验证客户端的请求 ``` C# public class ManageController { [HttpPost] [ValidateAntiForgeryToken] public IActionResult ChangePassword() { // ... return View(); } } ``` 是不是每个POST请求都需要添加这样的attribute呢？ ASP.NET Core中有Filter的概念，通过添加全局Filter就能帮我们达到这样的目的： ``` C# public class Startup { public void ConfigureServices(IServiceCollection services) { services.AddMvc(options => { options.Filters.Add(new AutoValidateAntiforgeryTokenAttribute()); }); } } ``` AutoValidateAntiforgeryTokenAttribute会自动忽略不需要做CSRF验证的请求类型，例如HttpGet请求。 ### 强制HTTPS 为了让服务更加安全，你还可以强制用户使用https，你可以通过配置API网关的方式达到这个目的，也可以使用ASP.NET Core自带的特性。 使用了RequireHttpsAttribute之后，http请求将会报错。 ```C# services.Configure(options => { options.Filters.Add(new RequireHttpsAttribute()); }); ``` 通过下面的方式强行跳转到https。 ``` C# public void Configure(IApplicationBuilder app, IHostingEnvironment env) { app.UseHttpsRedirection(); } ``` ### 让HTTP Header更加安全 通过https://securityheaders.com/来检查HTTP Header是不是安全，例如下面的扫描结果： ![safety](./../img/net/c_core_safety1.png) NWebsec是一个用来做安全相关的ASP.NET Core类库，针对ASP.NET Core中HTTP Header相关的修复，可以添加下面的Nuget包： NWebsec.AspNetCore.Middleware * Strict-Transport-Security：为了告诉浏览器所有的资源都必须使用https，你需要添加这个header: ``` C# app.UseHsts(hsts => hsts.MaxAge(365)); ``` * Redirect validation: 一旦启用了这个中间件，只能被Redirect到允许的站点, 否则会触发RedirectValidationException ``` C# app.UseRedirectValidation(opts => { opts.AllowSameHostRedirectsToHttps(); opts.AllowedDestinations("https://www.google.com/accounts/"); }); ``` * Referrer-Policy: 当用户点击了浏览器上的连接，请求报头中Referrer用来表示连接的来源，这个特性也可以用来做一些数据分析，通过Referrer-Policy可以控制是否显示Referrer: ``` C# app.UseReferrerPolicy(opts => opts.NoReferrer()); ``` * Content-Security-Policy:内容安全策略，这个http header可以让浏览器自动禁止外部注入恶意脚本，例如下面的策略将限制所有的脚本只能从同域加载： ``` C# 'Content-Security-Policy': 'script-src \'self\'' ``` * 下面的脚本引用将会引起浏览器报错： ``` js ``` * 使用NWebsec配置Content-Security-Policy: ``` C# app.UseCsp(options => options .DefaultSources(s => s.Self()) .ScriptSources(s => s.Self().CustomSources("scripts.nwebsec.com")) .ReportUris(r => r.Uris("/report"))); ``` * X-XSS-Protection: 防XSS攻击设置 ``` C# app.UseXXssProtection(options => options.EnabledWithBlockMode()); ``` * X-Content-Type-Options: 如果服务器发送响应头 X-Content-Type-Options: nosniff，则 script 和 styleSheet 元素会拒绝包含错误的 MIME 类型的响应。这是一种安全功能，有助于防止基于 MIME 类型混淆的攻击。 ``` C# app.UseXContentTypeOptions(); ``` ================================================ FILE: docs/net/c_core_study_route.md ================================================ # ASP.NET Core开发者指南 > 2019年[ASP.NET Core](https://docs.microsoft.com/zh-cn/aspnet/core/?view=aspnetcore-2.2)开发者指南: 你可以在下面找到一张图，该图展示了你可以选取的路径及你想学习的库，从而成为一名 ASP.NET Core 开发者。“作为 ASP.NET Core 开发者，我接下来应该学习什么？”，我把这张图作为建议给每个问过我这一问题的人。 ## 免责声明 > 该指南的目的是为了给读者心有个大概的轮廓。如果你对接下来要学习的内容感到困惑，这张路线图将指导你，而不是鼓励你选择时髦的东西。 > 你应该逐渐理解为什么一种工具比另一种工具更适合某些场景，并且记住时髦和新颖的东西并不总是意味着最适合这个工作。 ## 请给一个星星! :star: 如果你喜欢或正在使用这个项目进行学习或引用在你的解决方案中，请给它一个星星。谢谢! ## 路线图 ![路线图](./../img/net/aspnetcore-developer-roadmap.zh-Hans.png) ## 资源 1. 先决条件 - [C#](https://www.pluralsight.com/paths/csharp) - [Entity Framework](https://www.pluralsight.com/search?q=entity%20framework%20core) - [ASP.NET Core](https://www.pluralsight.com/search?q=asp.net%20core) - SQL基础知识 2. 通用开发技能 - 学习GIT, 在GitHub中创建开源项目 - 掌握HTTP(S)协议, 及其请求方法(GET, POST, PUT, PATCH, DELETE, OPTIONS) - 不要害怕使用 Google, [Google搜索技巧](http://www.powersearchingwithgoogle.com/) - 学习[dotnet CLI](https://docs.microsoft.com/zh-cn/dotnet/core/tools/?tabs=netcore2x) - 阅读一些关于算法和数据结构的书籍 3. 依赖注入 1. DI容器 - [Microsoft.Extensions.DependencyInjection](https://docs.microsoft.com/zh-cn/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-2.2) - [AutoFac](https://autofaccn.readthedocs.io/en/latest/integration/aspnetcore.html) - [Ninject](http://www.ninject.org/) - [StructureMap](https://github.com/structuremap/structuremap) - [Castle Windsor](https://github.com/castleproject/Windsor) 2. [生命周期](https://docs.microsoft.com/zh-cn/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-2.2#service-lifetimes) 3. [Scrutor](https://github.com/khellang/Scrutor) 4. 数据库 1. 关系数据库 1. [SQL Server](https://www.microsoft.com/zh-cn/sql-server/sql-server-2017) 2. [PostgreSQL](https://www.postgresql.org/) 3. [MariaDB](https://mariadb.org/) 4. [MySQL](https://www.mysql.com/) 2. 云数据库 - [CosmosDB](https://docs.microsoft.com/zh-cn/azure/cosmos-db) - [DynamoDB](https://aws.amazon.com/dynamodb/) 3. 搜索引擎 - [ElasticSearch](https://www.elastic.co/) - [Solr](http://lucene.apache.org/solr/) - [Sphinx](http://sphinxsearch.com/) 4. NoSQL - [MongoDB](https://docs.microsoft.com/zh-cn/aspnet/core/tutorials/first-mongo-app?view=aspnetcore-2.2&tabs=visual-studio) - [Redis](https://redis.io/) - [Apache Cassandra](http://cassandra.apache.org/) - [LiteDB](https://github.com/mbdavid/LiteDB) - [RavenDB](https://github.com/ravendb/ravendb) - [CouchDB](http://couchdb.apache.org/) 5. 缓存 1. 实体框架二级缓存 1. [EFSecondLevelCache.Core](https://github.com/VahidN/EFSecondLevelCache.Core) 2. [EntityFrameworkCore.Cacheable](https://github.com/SteffenMangold/EntityFrameworkCore.Cacheable) 2. [分布式缓存](https://docs.microsoft.com/en-us/aspnet/core/performance/caching/distributed?view=aspnetcore-2.2) 1. [Redis](https://redis.io/) 2. [Memcached](https://memcached.org/) 3. [内存缓存](https://docs.microsoft.com/en-us/aspnet/core/performance/caching/memory?view=aspnetcore-2.2) 6. 日志 1. 日志框架 - [Serilog](https://github.com/serilog/serilog) - [NLog](https://github.com/NLog/NLog) - [Elmah](https://elmah.github.io/) 2. 日志管理系统 - [Sentry.io](http://sentry.io) - [Loggly.com](https://loggly.com) - [Elmah.io](http://elmah.io) 7. 模板引擎 1. [Razor](https://docs.microsoft.com/zh-cn/aspnet/core/mvc/views/razor?view=aspnetcore-2.2) 2. [DotLiquid](https://github.com/dotliquid/dotliquid) 3. [Scriban](https://github.com/lunet-io/scriban) 4. [Fluid](https://github.com/sebastienros/fluid) 8. 实时通信 1. [SignalR](https://docs.microsoft.com/en-us/aspnet/core/signalr) 9. 对象映射 - [AutoMapper](https://github.com/AutoMapper/AutoMapper) - [Mapster](https://github.com/MapsterMapper/Mapster) - [AgileMapper](https://github.com/agileobjects/AgileMapper) - [ExpressMapper](http://expressmapper.org/) 10. API客户端 1. REST - [OData](https://blogs.msdn.microsoft.com/odatateam/2018/07/03/asp-net-core-odata-now-available/) - [Sieve](https://github.com/Biarity/Sieve) 2. GraphQL - [GraphQL-dotnet](https://github.com/graphql-dotnet/graphql-dotnet) 11. 最好掌握 - [MediatR](https://github.com/jbogard/MediatR) - [Fluent Validation](https://github.com/JeremySkinner/FluentValidation) - [Swashbuckle](https://github.com/domaindrivendev/Swashbuckle.AspNetCore) - [Benchmark.NET](https://github.com/dotnet/BenchmarkDotNet) - [Polly](https://github.com/App-vNext/Polly) - [NodaTime](https://github.com/nodatime/nodatime) - [GenFu](https://github.com/MisterJames/GenFu) 12. 测试 1. 单元测试 1. 测试框架 - [MSTest](https://docs.microsoft.com/zh-cn/dotnet/core/testing/unit-testing-with-mstest) - [NUnit](https://docs.microsoft.com/zh-cn/dotnet/core/testing/unit-testing-with-nunit) - [xUnit](https://docs.microsoft.com/zh-cn/dotnet/core/testing/unit-testing-with-dotnet-test) 2. 模拟工具 - [Moq](https://github.com/moq/moq4) - [NSubstitute](https://github.com/nsubstitute/NSubstitute) - [FakeItEasy](https://github.com/FakeItEasy/FakeItEasy) 3. 断言工具 - [FluentAssertion](https://github.com/fluentassertions/fluentassertions) - [Shouldly](https://github.com/shouldly/shouldly) 2. 行为测试 - [BDDfy](https://github.com/TestStack/TestStack.BDDfy) - [SpecFlow](https://github.com/techtalk/SpecFlow/tree/DotNetCore) - [LightBDD](https://github.com/LightBDD/LightBDD) 3. 集成测试 - [WebApplicationFactory](https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-2.2) - [TestServer](https://koukia.ca/integration-testing-in-asp-net-core-2-0-51d14ede3968) 4. 端到端测试 - [Selenium](https://www.automatetheplanet.com/webdriver-dotnetcore2/) - [Puppeteer-Sharp](https://github.com/kblok/puppeteer-sharp) 13. 任务调度 - [HangFire](https://github.com/HangfireIO/Hangfire) - [Coravel](https://github.com/jamesmh/coravel) - [Fluent Scheduler](https://github.com/fluentscheduler/FluentScheduler) 14. 微服务 1. 消息队列 - [RabbitMQ](https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html) - [Apache Kafka](https://github.com/confluentinc/confluent-kafka-dotnet) - [ActiveMQ](https://github.com/apache/activemq) - [Azure Service Bus](https://docs.microsoft.com/zh-cn/azure/service-bus-messaging/service-bus-messaging-overview) 2. 消息总线 - [MassTransit](https://github.com/MassTransit/MassTransit) - [NServiceBus](https://github.com/Particular/NServiceBus) - [CAP](https://github.com/dotnetcore/CAP) 15. SOLID原则 - [单一责任原则(SRP)](https://www.dotnetcurry.com/software-gardening/1148/solid-single-responsibility-principle) - [开放封闭原则(OCP)](https://www.dotnetcurry.com/software-gardening/1176/solid-open-closed-principle) - [里氏替换原则(LSP)](https://www.dotnetcurry.com/software-gardening/1235/liskov-substitution-principle-lsp-solid-patterns) - [依赖倒置原则(ISP)](https://www.dotnetcurry.com/software-gardening/1257/interface-segregation-principle-isp-solid-principle) - [接口分离原则(DIP)](https://www.dotnetcurry.com/software-gardening/1284/dependency-injection-solid-principles) 16. 设计模式 - [CQRS](https://docs.microsoft.com/zh-cn/azure/architecture/patterns/cqrs) - [装饰模式](https://www.dofactory.com/net/decorator-design-pattern) - [策略模式](https://www.dofactory.com/net/strategy-design-pattern) - [观察者模式](https://www.dofactory.com/net/observer-design-pattern) - [建造者模式](https://www.dofactory.com/net/builder-design-pattern) - [单例模式](https://www.dofactory.com/net/singleton-design-pattern) - [外观模式](https://www.dofactory.com/net/facade-design-pattern) - [中介者模式](https://www.dofactory.com/net/mediator-design-pattern) ## 总结 如果你认为该指南可以改进，请提交包含任何更新的 PR 并提交任何问题。此外，我将继续改进这个仓库，因此你可以 star 这个仓库以便于重新访问。 灵感来源 : [React Developer RoadMap](https://github.com/adam-golab/react-developer-roadmap) ## 贡献 该指南是使用[Draw.io](https://www.draw.io/)构建的。中文版项目文件为`aspnetcore-developer-roadmap.zh-Hans.xml`。要修改它, 请打开 draw.io, 点击 **Open Existing Diagram** 并选择项目中的 `xml` 文件。它将为你渲染生成路线图，更新它，上传和更新自述文件中的图像并创建一个 PR（导出为400%的png图片，然后使用[Compressor.io](https://compressor.io/compress)压缩）。 - 改进后提交 PR - 在Issues中讨论问题 - 推广项目 ================================================ FILE: docs/net/c_docker.md ================================================ # .NET Core 微服务架构-Docker部署 本文主要介绍通过Docker来部署通过.NET Core开发的微服务架构，部署的微服务主要包括统一网关（使用Ocelot开发）、统一认证（IdentityServer4）、应用服务（ASP.NET Core WebAPI）。 本文不针对微服务进行说明，后续会针对我对微服务的理解在做记录。 ### 一、Docker原理 对 Docker 最简单并且带有一定错误的认知就是 “Docker 是一种性能非常好的虚拟机”。 但是这是有一定错误的说法。Docker 相比于传统虚拟机的技术来说先进了不少，具体表现在 Docker 不是在宿主机上虚拟出一套硬件后再虚拟出一个操作系统，而是让 Docker 容器里面的进程直接运行在宿主机上（Docker 会做文件、网络等的隔离），这样一来 Docker 会 “体积更轻、跑的更快、同宿主机下可创建的个数更多”。 * Docker 中有三个核心概念：Image、Container、Repository。 * Image： 大家对 镜像 的概念不会陌生。但和 windows 的那种 iso 镜像相比，Docker 中的镜像是分层的，可复用的，而非简单的一堆文件迭在一起（类似于一个压缩包的源码和一个 git 仓库的区别）。 * Container： 容器的存在离不开镜像的支持，他是镜像运行时的一个载体（类似于实例和类的关系）。依托 Docker 的虚拟化技术，给容器创建了独立的端口、进程、文件等“空间”，Container 就是一个与宿机隔离 “容器”。容器可宿主机之间可以进行 port、volumes、network 等的通信。 Repository： Docker 的仓库和 git 的仓库比较相似，拥有仓库名、tag。在本地构建完镜像之后，即可通过仓库进行镜像的分发。 常用的 Docker hub有： https://hub.docker.com/（docker官方） https://cr.console.aliyun.com/ ### 二、Windows系统中Docker的安装 #### 1、启用Hyper-V 打开控制面板 - 程序和功能 - 启用或关闭Windows功能，勾选Hyper-V，然后点击确定即可，如图： ![cdoecker](./../img/net/c_docker1.png) #### 2、下载并安装Docker 下载地址：https://hub.docker.com/editions/community/docker-ce-desktop-windows，需要注册一个Docker账号才能下载 下载完成后直接安装 通过命令查看Docker版本号： ![cdoecker](./../img/net/c_docker2.png) #### 3、修改镜像地址 由于Docker官方镜像会非常慢，我是修改的阿里云的镜像 ![cdoecker](./../img/net/c_docker3.png) #### 4、测试 万能的HelloWorld，通过PowerShell运行 Docker run hello-world ![cdoecker](./../img/net/c_docker4.png) docker ps -a //查看运行的容器 ![cdoecker](./../img/net/c_docker5.png) 以上就是整个Windows中安装Docker环境的过程 ### 三、ubuntu中安装Docker环境 我是在AWS申请了一台免费的服务器，是用的ubuntu系统。如果需要申请AWS的服务器，可以通过AWS官网注册账号进行申请，需要填写信用卡账号，https://portal.aws.amazon.com/billing/signup#/start 本文ubuntu中安装Docker是直接参考官方教程：https://docs.docker.com/install/linux/docker-ce/ubuntu/ 安装的时候最好切换到root账号进行安装 ubuntu的远程是通过putty，具体怎么用可以百度 ### 四、发布并部署服务 #### 1、创建Dockerfile、发布应用程序 ![cdoecker](./../img/net/c_docker6.png) 这是我的工程目录，构建镜像是通过Dockerfile来构建的。 VS2017 是支持自动构建Dockerfile文件，工程右键-添加-Docker支持 下面是我的Dockerfile的内容： ``` C# FROM microsoft/aspnetcore:2.1 //基础镜像，这里是.net core的基础运行环境 WORKDIR /publish //创建工程目录 COPY . /publish //将当前目录拷贝到镜像中，注意 COPY . 后面又空格 EXPOSE 80 //容器对外暴露80端口 ENTRYPOINT ["dotnet", "ZY.Gateway.dll"] //启动容器内的服务 //这里相当于构建镜像的命令，是逐行执行 ``` 需要设置Dockerfile的编译为输出到发布目录 ![cdoecker](./../img/net/c_docker7.png) 以上都设置好以后，通过VS发布程序 ![cdoecker](./../img/net/c_docker8.png) 发布成功后，在工程目录的bin/release目录下找到我们的发布后的文件 ![cdoecker](./../img/net/c_docker9.png) 其他服务的发布跟以上发布类似 #### 2、上传到ubuntu中 通过WinScp工具进行上传，将三个服务都上传到服务器，WinScp怎么用，可以百度一下 ![cdoecker](./../img/net/c_docker10.png) #### 3、构建镜像 ``` C# docker build -t apigateway . //构建网关镜像 docker build -t identityserver -f /home/ubuntu/dockerapp/identityserver/publish/Dockerfile . //构建认证服务镜像 docker build -t testserver -f /home/ubuntu/dockerapp/testserver/publish/Dockerfile . //构建测试服务镜像 -t //镜像名称 -f //dockerfile文件路径 ``` ![cdoecker](./../img/net/c_docker11.png) docker images //查看镜像 ![cdoecker](./../img/net/c_docker12.png) #### 4、运行容器 镜像已经在前面构建完成，这一步会根据构建的镜像来运行容器，将我们的服务跑起来 ``` C# docker run -d -p 5000:80 --name apigateway_container apigateway docker run -d -p 6000:80 --name identityserver_container identityserver docker run -d -p 7000:80 --name testserver_container testserver //分别将网关服务，认证服务，测试服务容器跑起来 //-d 保持后台进程运行 -p 端口映射，{主机端口}:{容器端口} ``` 通过命令查看容器运行情况 docker ps -a ![cdoecker](./../img/net/c_docker13.png) 在配置网关服务的时候会涉及到容器与容器之间的网络访问，Docker在安装的时候会创建一个172.17.0.1的IP网关，可以通过172.17.0.1来做端口的转发。 通过命令可以查看docker0的网关 ![cdoecker](./../img/net/c_docker14.png) Api网关的路由转发配置 ![cdoecker](./../img/net/c_docker15.png) #### 5、调用服务 通过Postman来调用通过Docker运行的服务，通过API网关访问认证服务获取Token ![cdoecker](./../img/net/c_docker16.png) ### 总结 整个Docker的安装，服务的发布，镜像的打包，容器的运行就全部完成。 整个过程不是一气呵成的，踩了不少坑，在踩坑的过程中也学到和理解了很多东西。 [Docekr介绍使用](https://yeasy.gitbooks.io/docker_practice/content/) ================================================ FILE: docs/net/c_sharp.md ================================================ # C# 新特性 ## C# 6 ### 一、字符串插值 （String Interpolation） C# 6之前我们拼接字符串时需要这样 ```C# var Name = "Jack"; var results = "Hello" + Name; ``` 或者 ```C# var Name = "Jack"; var results = string.Format("Hello {0}", Name); ``` 但是C#6里我们就可以使用新的字符串插值特性 ```C# var Name = "Jack"; var results = $"Hello {Name}"; ``` 上面只是一个简单的例子，想想如果有多个值要替换的话，用C#6的这个新特性，代码就会大大减小，而且可读性比起之前大大增强 ```C# Person p = new Person {FirstName = "Jack", LastName = "Wang", Age = 100}; var results = string.Format("First Name: {0} LastName: {1} Age: { 2} ", p.FirstName, p.LastName, p.Age); ``` 有了字符串插值后: ```C# var results = $"First Name: {p.FirstName} LastName: {p.LastName} Age: {p.Age}"; ``` 字符串插值不光是可以插简单的字符串，还可以直接插入代码 ```C# Console.WriteLine($"Jack is saying { new Tools().SayHello() }"); var info = $"Your discount is {await GetDiscount()}"; ``` 那么如何处理多语言呢？我们可以使用 IFormattable下面的代码如何实现多语言？ ```C# Double remain = 2000.5; var results= $"your money is {remain:C}"; ``` 输出 your money is $2,000.50 使用IFormattable 多语言 ```C# class Program { static void Main(string[] args) { Double remain = 2000.5; var results= ChineseText($"your money is {remain:C}"); Console.WriteLine(results); Console.Read(); } public static string ChineseText(IFormattable formattable) { return formattable.ToString(null, new CultureInfo("zh-cn")); } } ``` 输出 your money is ￥2,000.50 ### 二、空操作符 ( ?. ) C# 6添加了一个 ?. 操作符，当一个对象或者属性职为空时直接返回null, 就不再继续执行后面的代码，在之前我们的代码里经常出现 NullException, 所以我们就需要加很多Null的判断，比如 ```C# if (user != null && user.Project != null && user.Project.Tasks != null && user.Project.Tasks.Count > 0) { Console.WriteLine(user.Project.Tasks.First().Name); } ``` 现在我们可以不用写 IF 直接写成如下这样 ```C# Console.WriteLine(user?.Project?.Tasks?.First()?.Name); ``` 这个?. 特性不光是可以用于取值，也可以用于方法调用，如果对象为空将不进行任何操作，下面的代码不会报错，也不会有任何输出。 ```C# class Program { static void Main(string[] args) { User user = null; user?.SayHello(); Console.Read(); } } public class User { public void SayHello() { Console.WriteLine("Ha Ha"); } } ``` 还可以用于数组的索引器 ```C# class Program { static void Main(string[] args) { User[] users = null; List listUsers = null; // Console.WriteLine(users[1]?.Name); // 报错 // Console.WriteLine(listUsers[1]?.Name); //报错 Console.WriteLine(users?[1].Name); // 正常 Console.WriteLine(listUsers?[1].Name); // 正常 Console.ReadLine(); } } ``` 注意： 上面的代码虽然可以让我们少些很多代码，而且也减少了空异常，但是我们却需要小心使用，因为有的时候我们确实是需要抛出空异常，那么使用这个特性反而隐藏了Bug ### 三、 NameOf 过去，我们有很多的地方需要些硬字符串，导致重构比较困难，而且一旦敲错字母很难察觉出来，比如 ```C# if (role == "admin") { } ``` WPF 也经常有这样的代码 ```C# public string Name { get { return name; } set { name= value; RaisePropertyChanged("Name"); } } ``` 现在有了C#6 NameOf后，我们可以这样 ```C# public string Name { get { return name; } set { name= value; RaisePropertyChanged(NameOf(Name)); } } static void Main(string[] args) { Console.WriteLine(nameof(User.Name)); // output: Name Console.WriteLine(nameof(System.Linq)); // output: Linq Console.WriteLine(nameof(List)); // output: List Console.ReadLine(); } ``` 注意： NameOf只会返回Member的字符串，如果前面有对象或者命名空间，NameOf只会返回 . 的最后一部分, 另外NameOf有很多情况是不支持的，比如方法，关键字，对象的实例以及字符串和表达式 ### 四、在Catch和Finally里使用Await 在之前的版本里，C#开发团队认为在Catch和Finally里使用Await是不可能，而现在他们在C#6里实现了它。 ```C# Resource res = null; try { res = await Resource.OpenAsync(); // You could always do this. } catch (ResourceException e) { await Resource.LogAsync(res, e); // Now you can do this … } finally { if (res != null) await res.CloseAsync(); // … and this. } ``` ### 五、表达式方法体 一句话的方法体可以直接写成箭头函数，而不再需要大括号 ```C# class Program { private static string SayHello() => "Hello World"; private static string JackSayHello() => $"Jack {SayHello()}"; static void Main(string[] args) { Console.WriteLine(SayHello()); Console.WriteLine(JackSayHello()); Console.ReadLine(); } } ``` ### 六、自动属性初始化器 之前我们需要赋初始化值，一般需要这样 ``` C# public class Person { public int Age { get; set; } public Person() { Age = 100; } } ``` 但是C# 6的新特性里我们这样赋值 ```C# public class Person { public int Age { get; set; } = 100; } ``` ### 七、只读自动属性 C# 1里我们可以这样实现只读属性 ``` C# public class Person { private int age=100; public int Age { get { return age; } } } ``` 但是当我们有自动属性时，我们没办法实行只读属性，因为自动属性不支持readonly关键字，所以我们只能缩小访问权限 ```C# public class Person { public int Age { get; private set; } } ``` 但是 C#6里我们可以实现readonly的自动属性了 ```C# public class Person { public int Age { get; } = 100; } ``` ### 八、异常过滤器 Exception Filter ```C# static void Main(string[] args) { try { throw new ArgumentException("Age"); } catch (ArgumentException argumentException) when( argumentException.Message.Equals("Name")) { throw new ArgumentException("Name Exception"); } catch (ArgumentException argumentException) when( argumentException.Message.Equals("Age")) { throw new Exception("not handle"); } catch (Exception e) { throw; } } ``` 在之前，一种异常只能被Catch一次，现在有了Filter后可以对相同的异常进行过滤，至于有什么用，那就是见仁见智了，我觉得上面的例子，定义两个具体的异常 NameArgumentException 和AgeArgumentException代码更易读。 ### 九、 Index 初始化器 这个主要是用在Dictionary上，至于有什么用，我目前没感觉到有一点用处，谁能知道很好的使用场景，欢迎补充: ```C# var names = new Dictionary { [1] = "Jack", [2] = "Alex", [3] = "Eric", [4] = "Jo" }; foreach (var item in names) { Console.WriteLine($"{item.Key} = {item.Value}"); } ``` ### 十、using 静态类的方法可以使用 static using 这个功能在我看来，同样是很没有用的功能，也为去掉前缀有的时候我们不知道这个是来自哪里的，而且如果有一个同名方法不知道具体用哪个，当然经证实是使用类本身的覆盖，但是容易搞混不是吗？ ```C# using System; using static System.Math; namespace CSharp6NewFeatures { class Program { static void Main(string[] args) { Console.WriteLine(Log10(5)+PI); } } } ``` ## C# 7 ### 数字字面量 现在可以在数字中加下划线，增加数字的可读性。编译器或忽略所有数字中的下划线 ``` C# int million = 1_000_000; ``` 虽然编译器允许在数字中任意位置添加任意个数的下划线，但显然，遵循管理，下划线应该每三位使用一次，而且，不可以将下划线放在数字的开头（_1000）或结尾（1000_） ### 改进的out关键字 C#7支持了out关键字的即插即用 ``` C# var a = 0; int.TryParse("345", out a); // 就地使用变量作为返回值 int.TryParse("345", int out b); ``` 允许以_（下划线）形式“舍弃”某个out参数，方便你忽略不关系的参数。例如下面的例子中，获得一个二维坐标的X可以重用获得二维坐标的X，Y方法，并舍弃掉Y： ``` C# struct Point { public int x; public int y; private void GetCoordinates(out int x, out int y) { x = this.x; y = this.y; } public void GetX() { // y被舍弃了，虽然GetCoordinates方法还是会传入2个变量，且执行y=this.y // 但它会在返回之后丢失 GetCoordinates(out int x, out _); WriteLine($"({x})"); } } ``` ### 模式匹配 模式匹配（Pattern matching）是C#7中引入的重要概念，它是之前is和case关键字的扩展。目前，C#拥有三种模式： * 常量模式：简单地判断某个变量是否等于一个常量（包括null） * 类型模式：简单地判断某个变量是否为一个类型的实例 * 变量模式：临时引入一个新的某个类型的变量（C#7新增） 下面的例子简单地演示了这三种模式： ``` C# class People { public int TotalMoney { get; set; } public People(int a) { TotalMoney = a; } } class Program { static void Main(string[] args) { var peopleList = new List() { new People(1), new People(1_000_000) }; foreach (var p in peopleList) { // 类型模式 if (p is People) WriteLine("是人"); // 常量模式 if (p.TotalMoney > 500_000) WriteLine("有钱"); // 变量模式 // 加入你需要先判断一个变量p是否为People，如果是，则再取它的TotalMoney字段 // 那么在之前的版本中必须要分开写 if (p is People) { var temp = (People)p; if (temp.TotalMoney > 500_000) WriteLine("有钱"); } // 变量模式允许你引入一个变量并立即使用它 if (p is People ppl && ppl.TotalMoney > 500_000) WriteLine("有钱"); } ReadKey(); } } ``` 可以看出，变量模式引入的临时变量ppl（称为模式变量）的作用域也是整个if语句体，它的类型是People类型 case关键字也得到了改进。现在，case后面也允许模式变量，还允许when子句，代码如下： ``` C# static void Main(string[] args) { var a = 13; switch (a) { // 现在i就是a // 由于现在case后面可以跟when子句的表达式，不同的case有机会相交 case int i when i % 2 == 1: WriteLine(i + " 是奇数"); break; // 只会匹配第一个case，所以这个分支无法到达 case int i when i > 10: WriteLine(i + " 大于10"); break; // 永远在最后被检查，即使它后面还有case子句 default: break; } ReadKey(); } ``` 上面的代码运行的结果是打印出13是奇数，我们可以看到，现在case功能非常强大，可以匹配更具体、跟他特定的范围。不过，多个case的范围重叠，编译器只会选择第一个匹配上的分支 ### 值类型元组 元组（Tuple）的概念早在C#4就提出来，它是一个任意类型变量的集合，并最多支持8个变量。在我们不打算手写一个类型或结构体来盛放一个变量集合时（例如，它是临时的且用完即弃），或者打算从一个方法中返回多个值，我们会考虑使用元组。不过相比C#7的元组，C#4的元组更像一个半成品，先看看C#4如何使用元组： ``` C# var beforeTuple = new Tuple(2, 3); var a = beforeTuple.Item1; ``` 通过上面的代码发现，C#4中元组最大的两个问题是： * Tuple类将其属性命名为Item1、Item2等，这些名称是无法改变的，只会让代码可读性变差 * Tuple是引用类型，使用任一Tuple类意味着在堆上分配对象，因此，会对性能造成负面影响 C#7引入的新元组（ValueTuple）解决了上面两个问题，它是一个结构体，并且你可以传入描述性名称（TupleElementNames属性）以便更容易地调用他们 ``` C# static void Main(string[] args) { // 未命名的元组，访问方式和之前的元组相同 var unnamed = ("one", "two"); var b = unnamed.Item1; // 带有命名的元组 var named = (first : "one", second : "two"); b = named.first; ReadKey(); } ``` 在背后，他们被编译器隐式地转化为： ``` C# ValueTuple unnamed = new ValueTuple() ("one", "two"); string b = unnamed.Item1; ValueTuple named = new ValueTuple() ("one", "two"); b = named.Item1; ``` 我们看到，编译器将带有命名元组的实名访问转换成对应的Item，转换是使用特性实现的 #### 元组的字段名称 可以在元组定义时传入变量。此时，元组的字段名称为变量名。如果没有指明字段名称，又传入了常量，则只能使用Item1、Item2等访问元组的成员 ``` C# static void Main(string[] args) { var localVariableOne = 5; var localVariableTwo = "some text"; // 显示实现的字段名称覆盖变量名 var tuple = (explicitFieldOne : localVariableOne, explicitFieldTwo : localVariableTwo); var a = tuple.explicitFieldOne; // 没有指定字段名称，又传入了变量名（需要C#7.1版本） var tuple2 = (localVariableOne, localVariableTwo); var b = tuple.localVariableOne; // 如果没有指明字段名称，又传入了常量，则只能使用Item1、Item2等访问元组的成员 var tuple3 = (5, "some text"); var c = tuple3.Item1; ReadKey(); } ``` 上面的代码给出了元组字段名称的优先级： * 首先是显示实现 * 其次是变量名（编译器自动推断的，需要C#7.1） * 最后是默认的Item1、Item2作为保留名称 另外，如果变量名或显示指定的描述名称是C#的关键字，则C#会改用ItemX作为字段名称（否则就会导致语法错误，例如将变量名为ToString的变量传入元组） ``` C# var ToString = "1"; var Item1 = 2; var tuple4 = (ToString, Item1); // ToString不能用作元组字段名称，强制改为Item1 var d = tuple4.Item1; // "1" // Item1不能用作元组字段名，强制改为Item2 var e = tuple4.Item2; // 2 ReadKey(); ``` ### 元组作为方法的参数和返回值 因为元组实际上是一个结构体，所以它当然可以作为方法的参数和返回值。因此，我们就有了可以返回多个变量的最简单、最优雅的方法（比使用out的可读性好很多）： ``` C# // 使用元组作方法的参数和返回值 (int, int) MultiplyAll(int multiplier, (int a, int b) members) { // 元组没有实现IEnumerator接口，不能foreach // foreach(var a in members) // 操作元组 return (members.a * multiplier, members.b * multiplier); } ``` 上面代码中的方法会将输入中的a和b都乘以multiplier，然后返回结构。由于元组是结构体，所以即使含有引用类型，其值类型的部分也会在栈上进行分配，相比C#4的元组，C#7中的元组有着更好的性能和更友好的访问方式 #### 相同类型元组的赋值 如果它们的基数（即成员数）相同，且每个元素的类型要么相同，要么可以实现隐式转换，则两个元组被看作相同的类型： ``` C# static void Main(string[] args) { var a = (first : "one", second : 1); WriteLine(a.GetType()); var b = (a : "hello", b : 2); WriteLine(b.GetType()); var c = (a : 3, b : "world"); WriteLine(c.GetType()); WriteLine(a.GetType() == b.GetType()); // True，两个元组基数和类型相同 WriteLine(a.GetType() == c.GetType()); // False，两个元组基数相同但类型不同 (string a, int b) d = a; // 属性first，second消失了，取而代之的是a和b WriteLine(d.a); // 定义了一个新的元组，成员为string和object类型 (string a, object b) e; // 由于int可以被隐式转换为object，所以可以这样赋值 e = a; ReadKey(); } ``` ### 解构 C#7允许你定义结构方法（Deconstructor），注意，它和C#诞生即存在的析构函数（Destructor）不同。解构函数和构造函数做的事情某种程度上是相对的——构造函数将若干个类型组合为一个大的类型，而结构方法将大类型拆散为一堆小类型，这些小类型可以是单个字段，也可以是元组。当类型成员很多而需要的部分通常较小时，解构方法会很有用，它可以防止类型传参时复制的高昂代价 #### 元组的解构 可以在括号内显示地声明每个字段的类型，为元组中的每个元素创建离散变量，也可以用var关键字 ``` C# static void Main(string[] args) { // 定义元组 (int count, double sum, double sumOfSquares) tuple = (1, 2, 3); // 使用方差的计算公式得到方差 var variance = tuple.sumOfSquares - tuple.sum * tuple.sum / tuple.count; // 将一个元组放在等号右边，将对应的变量值和类型放在等号左边，就会导致解构 (int count, double sum, double sumOfSquares) = (1, 2, 3); // 解构之后的方差计算，代码简洁美观 variance = sumOfSquares - sum * sum / count; // 也可以这样解构，这会导致编译器推断元组的类型为三个int var (a, b, c) = (1, 2, 3); ReadKey(); } ``` 上面的代码中，出现了两次解构方法的隐式调用：左边是一个没有元组变量名的元组（只有一些成员变量名），右边是元组的实例。解构方法所做的事情，就是将右边元组的实例中每个成员，逐个指派给左边元组的成员变量。例如： ``` C# (int count, double sum, double sumOfSquares) = (1, 2, 3); ``` 就会使得count，sum和sumOfSquares的值分别为1，2，3。如果没有这个功能，就需要定义3个变量，然后赋值3次，最终得到6行代码，大大提高了代码的可读性。 对于元组，C#提供了内置的解构支持，因此不需要手动写解构方法，如果需要对非元组类型进行解构，就需要定义自己的解构方法，显而易见，上面的解构通过如下的签名的函数完成： ``` C# public void Deconstruct(out int count, out double sum, out double sumOfSquares) ``` #### 解构其他类型 解构函数的名称必须为Deconstruct，下面的例子从一个较大的类型People中解构出我们想要的三项成员： ``` C# // 示例类型 public class People { public int ID; public string FirstName; public string MiddleName; public string LastName; public int Age; public string CompanyName; // 解构全名，包括姓、名字和中间名 public void Deconstruct(out string f, out string m, out string l) { f = FirstName; m = MiddleName; l = LastName; } } static void Main(string[] args) { var p = People(); p.FirstName = "Test"; var (fName, mName, lName) = p; WriteLine(fName); ReadKey(); } ``` 解构方法不能有返回值，且要解构的每个成员必须以out标识出来。如果编译器对一个类型的实例解构，却没发现对应的解构函数，就会发生编译时异常。如果在解构时发生隐式类型转换，则不会发生编译时异常，例如将上述的解构函数的输入参数类型都改为object类型，仍然可以完成解构，可以通过**重载解构函数对类型实现不同方式的解构** #### 忽略类型成员 为了少写代码，我们可以在解构时忽略类型成员。例如，我们如果只关系People的姓和名字，而不关心中间名，则不需要多写一个解构函数，而是利用现有的： ``` C# var (fName, _, lName) = p; ``` 通过使用下划线来忽略类型成员，此时仍然会调用带有三个参数的解构函数，但是p将会只有fName和lName两个成员元组也支持忽略类型成员的解构 #### 使用扩展方法进行解构 即使类型并非由自己定义，仍然可以通过解构扩展方法来解构类型，例如解构.NET自带的DateTime类型： ``` C# class Program { static void Main(string[] args) { var d = DateTime.Now; (string s, DayOfWeek dow) = d; WriteLine($"今天是 {s}, 是 {d}"); ReadKey(); } } public static class ReflectionExtensions { // 解构DateTime并获得想要的值 public static void Deconstruct(this DateTime dateTime, out string DateString, out DayOfWeek dayOfWeek) { DateString = dateTime.ToString("yyyy-MM-dd"); dayOfWeek = dateTime.DayOfWeek; } } ``` 如果类型提供了解构方法，你又在扩展方法中定义了与签名相同的解构方法，则编译器会优先选用类型提供的解构方法 ### 局部函数 局部函数（local functions）和匿名方法很像，当你有一个只会使用一次的函数（通常作为其他函数的辅助函数）时，可以使用局部函数或匿名方法。如下是一个利用局部函数和元组计算斐波那契数列的例子： ``` C# static void Main(string[] args) { WriteLine(Fibonacci(10)); ReadKey(); } public static int Fibonacci(int x) { if (x < 0) throw new ArgumentException("输入正整数", nameof(x)); return Fib(x).current; // 局部函数定义 (int current, int previous) Fib(int i) { if (i == 1) return (1, 0); var (p, pp) = Fib(i - 1); return (p + pp, p); } } ``` 局部函数是属于定义该函数的方法的，在上面的例子中，Fib函数只在Fibonacci方法中可用 * 局部函数只能在方法体中使用 * 不能在匿名方法中使用 * 只能用async和unsafe修饰局部函数，不能使用访问修饰符，默认是私有、静态的 * 局部函数和某普通方法签名相同，局部函数会将普通方法隐藏，局部函数所在的外部方法调用时，只会调用到局部函数 ### 更多的表达式体成员 C#6允许类型的定义中，字段后跟表达式作为默认值。C#7进一步允许了构造函数、getter、setter以及析构函数后跟表达式： ``` C# class CSharpSevenClass { int a; // get, set使用表达式 string b { get => b; set => b = "12345"; } // 构造函数 CSharpSevenClass(int x) => a = x; // 析构函数 ~CSharpSevenClass() => a = 0; } ``` 上面的代码演示了所有C#7中允许后跟表达式（但过去版本不允许）的类型实例成员 ## C# 8 ### 可空引用类型 从此，引用类型将会区分是否可分，可以从根源上解决 NullReferenceException。但是由于这个特性会打破兼容性，因此没有当作 error 来对待，而是使用 warning 折衷，而且开发人员需要手动 opt-in 才可以使用该特性（可以在项目层级或者文件层级进行设定）。 例如： ``` C# string s = null; // 产生警告: 对不可空引用类型赋值 null string? s = null; // Ok void M(string? s) { Console.WriteLine(s.Length); // 产生警告：可能为 null if (s != null) { Console.WriteLine(s.Length); // Ok } } ``` 至此，妈妈再也不用担心我的程序到处报 NullReferenceException 啦！ ### 异步流（Async streams） 考虑到大部分 Api 以及函数实现都有了对应的 async版本，而 IEnumerable和 IEnumerator还不能方便的使用 async/await就显得很麻烦了。 但是，现在引入了异步流，这些问题得到了解决。 我们通过新的 IAsyncEnumerable和 IAsyncEnumerator来实现这一点。同时，由于之前 foreach是基于IEnumerable和 IEnumerator实现的，因此引入了新的语法await foreach来扩展 foreach的适用性。 例如： ``` C# async Task GetBigResultAsync() { var result = await GetResultAsync(); if (result > 20) return result; else return -1; } async IAsyncEnumerable GetBigResultsAsync() { await foreach (var result in GetResultsAsync()) { if (result > 20) yield return result; } } ``` ### 范围和下标类型 C# 8.0 引入了 Index 类型，可用作数组下标，并且使用 ^ 操作符表示倒数。 不过要注意的是，倒数是从 1 开始的。 ``` C# Index i1 = 3; // 下标为 3 Index i2 = ^4; // 倒数第 4 个元素 int[] a = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }; Console.WriteLine($"{a[i1]}, {a[i2]}"); // "3, 6" ``` 除此之外，还引入了 “..” 操作符用来表示范围（注意是左闭右开区间）。 ``` C# var slice = a[i1..i2]; // { 3, 4, 5 } ``` 关于这个下标从 0 开始，倒数从 1 开始，范围左闭右开，笔者刚开始觉得很奇怪，但是发现 Python 等语言早已经做了这样的实践，并且效果不错。因此这次微软也采用了这种方式设计了 C# 8.0 的这个语法。 ### 接口的默认实现方法 从此接口中可以包含实现了： ``` C# interface ILogger { void Log(LogLevel level, string message); void Log(Exception ex) => Log(LogLevel.Error, ex.ToString()); // 这是一个默认实现重载 } class ConsoleLogger : ILogger { public void Log(LogLevel level, string message) { ... } // Log(Exception) 会得到执行的默认实现 } ``` 在上面的例子中，Log(Exception)将会得到执行的默认实现。 ### 模式匹配表达式和递归模式语句 现在可以这么写了（patterns 里可以包含 patterns） ``` C# IEnumerable GetEnrollees() { foreach (var p in People) { if (p is Student { Graduated: false, Name: string name }) yield return name; } } ``` Student { Graduated: false, Name: string name }检查 p 是否为 Graduated = false且 Name为 string的 Student，并且迭代返回 name。 可以这样写之后是不是很爽？ 更有： ``` C# var area = figure switch { Line _ => 0, Rectangle r => r.Width * r.Height, Circle c => c.Radius * 2.0 * Math.PI, _ => throw new UnknownFigureException(figure) }; ``` 典型的模式匹配语句，只不过没有用“match”关键字，而是沿用了 了“switch”关键字。 但是不得不说，一个字，爽！ ### 目标类型推导 以前我们写下面这种变量/成员声明的时候，大概最简单的写法就是： ``` C# var points = new [] { new Point(1, 4), new Point(2, 6) }; private List _myList = new List(); ``` 现在我们可以这么写啦： ``` C# Point[] ps = { new (1, 4), new (3,-2), new (9, 5) }; private List _myList = new (); ``` ## C# 9 ### 仅可初始化的属性 对象的初始化器非常了不起。它们为客户端创建对象提供了一种非常灵活且易于阅读的格式，而且特别适合嵌套对象的创建，我们可以通过嵌套对象一次性创建整个对象树。下面是一个简单的例子： ``` C# new Person { FirstName = "Scott", LastName = "Hunter" } ``` 对象初始化器还可以让程序员免于编写大量类型的构造样板代码，他们只需编写一些属性即可！ ``` C# public class Person { public string FirstName { get; set; } public string LastName { get; set; } } ``` 目前的一大限制是，属性必须是可变的，只有这样对象初始化器才能起作用，因为它们需要首先调用对象的构造函数（在这种情况下调用的是默认的无参构造函数），然后分配给属性设置器。 仅可初始化的属性可以解决这个问题！它们引入了init访问器。init访问器是set访问器的变体，它只能在对象初始化期间调用： ``` C# public class Person { public string FirstName { get; init; } public string LastName { get; init; } } ``` 在这种声明下，上述客户端代码仍然合法，但是后续如果你想为FirstName和LastName属性赋值就会出错。 ### 初始化访问器和只读字段 由于init访问器只能在初始化期间被调用，所以它们可以修改所在类的只读字段，就像构造函数一样。 ``` C# public class Person { private readonly string firstName; private readonly string lastName; public string FirstName { get => firstName; init => firstName = (value ?? throw new ArgumentNullException(nameof(FirstName))); } public string LastName { get => lastName; init => lastName = (value ?? throw new ArgumentNullException(nameof(LastName))); } } ``` ### 记录 如果你想保持某个属性不变，那么仅可初始化的属性非常有用。如果你希望整个对象都不可变，而且希望其行为宛如一个值，那么就应该考虑将其声明为记录： ``` C# public data class Person { public string FirstName { get; init; } public string LastName { get; init; } } ``` 上述类声明中的data关键字表明这是一个记录，因此它具备了其他一些类似于值的行为，后面我们将深入讨论。一般而言，我们更应该将记录视为“值”（数据），而非对象。它们不具备可变的封装状态。相反，你可以通过创建表示新状态的新记录来表示随着时间发生的变化。记录不是由标识确定，而是由其内容确定。 ### With表达式 处理不可变数据时，一种常见的模式是利用现有的值创建新值以表示新状态。例如，如果想修改某人的姓氏，那么我们会用一个新对象来表示，这个对象除了姓氏之外和旧对象完全一样。通常我们称该技术为非破坏性修改。记录代表的不是某段时间的某个人，而是给定时间点上这个人的状态。 为了帮助大家习惯这种编程风格，记录允许使用一种新的表达方式：with表达式： ``` C# var otherPerson = person with { LastName = "Hanselman" }; ``` with表达式使用对象初始化的语法来说明新对象与旧对象之间的区别。你可以指定多个属性。 记录隐式地定义了一个protected“复制构造函数”，这种构造函数利用现有的记录对象，将字段逐个复制到新的记录对象中： ``` C# protected Person(Person original) { /* copy all the fields */ } // generated ``` with表达式会调用复制构造函数，然后在其上应用对象初始化器，以相应地更改属性。 如果你不喜欢自动生成的复制构造函数，那么也可以自己定义，with表达式就会调用自定义的复制构造函数。 ### 基于值的相等 所有对象都会从object类继承一个虚的Equals(object)方法。在调用静态方法Object.Equals(object, object)且两个参数均不为null时，该Equals(object)就会被调用。 结构体可以重载这个方法，获得“基于值的相等性”，即递归调用Equals来比较结构的每个字段。记录也一样。 这意味着，如果两个记录对象的值一致，则二者相等，但两者不一定是同一对象。例如，如果我们再次修改前面那个人的姓氏： ``` C# var originalPerson = otherPerson with { LastName = "Hunter" }; ``` 现在，ReferenceEquals(person, originalPerson) = false（它们不是同一个对象），但Equals(person, originalPerson) = true （它们拥有相同的值）。 如果你不喜欢自动生成的Equals覆盖默认的逐字段比较的行为，则可以编写自己的Equals重载。你只需要确保你理解基于值的相等性在记录中的工作原理，尤其是在涉及继承的情况下，具体的内容我们稍后再做介绍。 除了基于值的Equals之外，还有一个基于值的GetHashCode()重载方法。 ### 数据成员 在绝大多数情况下，记录都是不可变的，它们的仅可初始化的属性是公开的，可以通过with表达式进行非破坏性修改。为了优化这种最常见的情况，我们改变了记录中类似于string FirstName这种成员声明的默认含义。在其他类和结构声明中，这种声明表示私有字段，但在记录中，这相当于公开的、仅可初始化的自动属性！因此，如下声明： ``` C# public data class Person { string FirstName; string LastName; } ``` 与之前提到过的下述声明完全相同： ``` C# public data class Person { public string FirstName { get; init; } public string LastName { get; init; } } ``` 我们认为这种方式可以让记录更加优美而清晰。如果你需要私有字段，则可以明确添加private修饰符： ``` C# private string firstName; ``` ### 位置记录 有时，用参数位置来声明记录会很有用，内容可以根据构造函数参数的位置来指定，并且可以通过位置解构来提取。 你完全可以在记录中指定自己的构造函数和析构函数： ```C# public data class Person { string FirstName; string LastName; public Person(string firstName, string lastName) => (FirstName, LastName) = (firstName, lastName); public void Deconstruct(out string firstName, out string lastName) => (firstName, lastName) = (FirstName, LastName); } ``` 但是，我们可以用更短的语法表达完全相同的内容（使用成员变量的大小写方式来命名参数）： ```C# public data class Person(string FirstName, string LastName); ``` 上述声明了仅可初始化的公开的自动属性以及构造函数和析构函数，因此你可以这样写： ``` C# var person = new Person("Scott", "Hunter"); // positional construction var (f, l) = person; // positional deconstruction ``` 如果你不喜欢生成的自动属性，则可以定义自己的同名属性，这样生成的构造函数和析构函数就会自动使用自己定义的属性。 ### 记录和修改 记录的语义是基于值的，因此在可变的状态中无法很好地使用。想象一下，如果我们将记录对象放入字典，那么就只能通过Equals和GethashCode找到了。但是，如果记录更改了状态，那么在判断相等时它代表的值也会发生改变！可能我们就找不到它了！在哈希表的实现中，这个性质甚至可能破坏数据结构，因为数据的存放位置是根据它“到达”哈希表时的哈希值决定的！ 而且，记录也可能有一些使用内部可变状态的高级方法，这些方法完全是合理的，例如缓存。但是可以考虑通过手工重载默认的行为来忽略这些状态。 ### with表达式和继承 众所周知，考虑继承时基于值的相等性和非破坏性修改是一个难题。下面我们在示例中添加一个继承的记录类Student： ``` C# public data class Person { string FirstName; string LastName; } public data class Student : Person { int ID; } ``` 在如下with表达式的示例中，我们实际创建一个Student，然后将其存储到Person变量中： ``` C# Person person = new Student { FirstName = "Scott", LastName = "Hunter", ID = GetNewId() }; otherPerson = person with { LastName = "Hanselman" }; ``` 在最后一行的with表达式中，编译器并不知道person实际上包含一个Student。而且，即使otherPerson不是Student对象，它也不是合法的副本，因为它包含了与第一个对象相同的ID属性。 C#解决了这个问题。记录有一个隐藏的虚方法，能够确保“克隆”整个对象。每个继承的记录类型都会通过重载这个方法来调用该类型的复制构造函数，而继承记录的复制构造函数会调用基类的复制构造函数。with表达式只需调用这个隐藏“clone”方法，然后在结果上应用对象初始化器即可。 ### 基于值的相等性与继承 与with表达式的支持类似，基于值的相等性也必须是“虚的”，即两个Student对象比较时需要比较所有字段，即使在比较时，能够静态地得知类型是基类，比如Person。这一点通过重写已经是虚方法的Equals方法可以轻松实现。 然而，相等性还有另外一个难题：如果需要比较两个不同类型的Person怎么办？我们不能简单地选择其中一个来决定是否相等：相等性应该是对称的，因此无论两个对象中的哪个首先出现，结果都应该相同。换句话说，二者之间必须就相等性达成一致！ 我们来举例说明这个问题： ``` C# Person person1 = new Person { FirstName = "Scott", LastName = "Hunter" }; Person person2 = new Student { FirstName = "Scott", LastName = "Hunter", ID = GetNewId() }; ``` 这两个对象彼此相等吗？person1可能会认为相等，因为person2拥有Person的所有字段，但person2可能会有不同的看法！我们需要确保二者都认同它们是不同的对象。 C#可以自动为你解决这个问题。具体的实现方式是：记录拥有一个名为EqualityContract的受保护虚属性。每个继承的记录都会重载这个属性，而且为了比较相等，两个对象必须具有相同的EqualityContract。 ### 顶级程序 使用C#编写一个简单的程序需要大量的样板代码： ``` C# using System; class Program { static void Main() { Console.WriteLine("Hello World!"); } } ``` 这不仅对初学者来说难度太高，而且代码混乱，缩进级别也太多。 在C# 9.0中，你只需编写顶层的主程序： ``` C# using System; Console.WriteLine("Hello World!"); ``` 任何语句都可以。程序必须位于using之后，文件中的任何类型或名称空间声明之前，而且只能在一个文件中，就像只有一个Main方法一样。 如果你想返回状态代码，则可以利用这种写法。如果你想await，那么也可以这么写。此外，如果你想访问命令行参数，则args可作为“魔术”参数使用。 局部函数是语句的一种形式，而且也可以在顶层程序中使用。在顶层语句之外的任何地方调用局部函数都会报错。 ### 改进后的模式匹配 C# 9.0中添加了几种新的模式。下面我们通过如下模式匹配教程的代码片段来看看这些新模式： ``` C# public static decimal CalculateToll(object vehicle) => vehicle switch { ... DeliveryTruck t when t.GrossWeightClass > 5000 => 10.00m + 5.00m, DeliveryTruck t when t.GrossWeightClass < 3000 => 10.00m - 2.00m, DeliveryTruck _ => 10.00m, _ => throw new ArgumentException("Not a known vehicle type", nameof(vehicle)) }; ``` ### 简单类型模式 当前，类型模式需要在类型匹配时声明一个标识符，即使该标识符是表示放弃的_也可以，如上面的DeliveryTruck _。而如今你可以像下面这样编写类型： ``` C# DeliveryTruck => 10.00m, ``` ### 关系模式 C# 9.0中引入了与关系运算符<、<=等相对应的模式。因此，你可以将上述模式的DeliveryTruck写成嵌套的switch表达式： ```C# DeliveryTruck t when t.GrossWeightClass switch { > 5000 => 10.00m + 5.00m, < 3000 => 10.00m - 2.00m, _ => 10.00m, }, ``` 这的 > 5000 和 < 3000是关系模式。 ### 逻辑模式 最后，你还可以将模式与逻辑运算符（and、or和not）组合在一起，它们以英文单词的形式出现，以避免与表达式中使用的运算符混淆。例如，上述嵌套的switch表达式可以按照升序写成下面这样： ``` C# DeliveryTruck t when t.GrossWeightClass switch { < 3000 => 10.00m - 2.00m, >= 3000 and <= 5000 => 10.00m, > 5000 => 10.00m + 5.00m, }, ``` 中间一行通过and将两个关系模式组合到一起，形成了表示间隔的模式。 not模式的常见用法也可应用于null常量模式，比如not null。例如，我们可以根据是否为null来拆分未知情况的处理方式： ``` C# not null => throw new ArgumentException($"Not a known vehicle type: {vehicle}", nameof(vehicle)), null => throw new ArgumentNullException(nameof(vehicle)) ``` 此外，如果if条件中包含is表达式，那么使用not也很方便，可以避免笨拙的双括号： ``` C# if (!(e is Customer)) { ... } ``` 你可以这样写： ``` C# if (e is not Customer) { ... } ``` ### 改进后的目标类型推断 “目标类型推断”指的是表达式从所在的上下文中获取类型。例如，null和lambda表达式始终是目标类型推断。 在C# 9.0中，有些以前不是目标类型推断的表达式也可以通过上下文来判断类型。 ### 支持目标类型推断的new表达式 C# 中的new表达式始终要求指定类型（隐式类型的数组表达式除外）。现在， 如果有明确的类型可以分配给表达式，则可以省去指定类型。 ``` C# Point p = new (3, 5); ``` ### 目标类型的??与?: 有时，条件判断表达式中??与?:的各个分支之间并不是很明显的同一种类型。现在这种情况会出错，但在C# 9.0中，如果两个分支都可以转换为目标类型，就没有问题： ``` C# Person person = student ?? customer; // Shared base type int? result = b ? 0 : null; // nullable value type ``` ### 支持协变的返回值 有时，我们需要表示出继承类中重载的某个方法的返回类型要比基类中的类型更具体。C# 9.0允许以下写法： ``` C# abstract class Animal { public abstract Food GetFood(); ... } class Tiger : Animal { public override Meat GetFood() => ...; } ``` 总结 上面80%我认为都是比较有用的新特性，后面的几个我觉得用处不大，当然如果找到合适的使用场景应该有用，欢迎大家补充。 最后，祝大家编程愉快。 ================================================ FILE: docs/net/c_sqlserver_nginx.md ================================================ # ASP.NET Core 实战：使用 Docker 容器化部署 ASP.NET Core + sqlserver + Nginx ### 一、前言 　　在之前的文章（ASP.NET Core 实战：Linux 小白的 .NET Core 部署之路）中，我介绍了如何在 Linux 环境中安装 .NET Core SDK / .NET Core Runtime、Nginx、sqlserver，以及如何将我们的 ASP.NET Core MVC 程序部署到 Linux 上，同时，使用 supervisor 守护程序守护我们的 .NET Core 程序。如果，你有看过那篇文章，并且和我一样是个 Linux 小白用户的话，可能第一感觉就是，把 .NET Core 项目部署在 IIS 上也挺好。 　　将 .NET Core 项目部署到 Linux 上如此复杂，就没有简单的部署方式吗？ 　　你好，有的，Docker 了解一下~~~ 　　PS：这里的示例代码还是采用之前的毕业设计项目，在这篇文章发布的时候，我已经在程序的仓库中添加了对于 Docker 的支持，你可以下载下来，自己尝试一下，毕竟，实践出真知。 　　 代码仓储：https://github.com/burningmyself/micro ### 二、Step by Step #### 1、安装 Docker & Docker Compose 　　在代码交付的过程中，偶尔会遇到这样的问题，在本地测试是好的，但是部署到测试环境、生产环境时就出这样那样的问题，同时，因为本地与测试环境、生产环境之间存在差异，我们可能无法在本地复现这些问题，那么，有没有一种工具可以很好的解决这一问题呢？随着历史的车轮不断前行，容器技术诞生了。 　　Docker，作为最近几年兴起的一种虚拟化容器技术，他可以将我们的运行程序与操作系统做一个隔离，例如这里我们需要运行 .NET Core 程序，我们不再需要关心底层的操作系统是什么，不需要在每台需要需要运行程序的机器上安装程序运行的各种依赖，我们可以通过程序打包成镜像的方式，将应用程序和该程序的依赖全部置于一个镜像文件中，这时，只要别的机器上有安装 Docker，就可以通过我们打包的这个镜像来运行这个程序。 #### 1.1、卸载 Docker 　　在安装 Docker 之前，我们应该确定当前的机器上是否已经安装好了 Docker，为了防止与现在安装的 Docker CE 发生冲突，这里我们先卸载掉以前版本的 Docker，如果你确定你的机器上并没有安装 Docker 的话此步可以跳过。 　　在 Linux 中可以使用 \ 加 Enter 在输入很长很长的语句时进行换行，这里和后面的命令都是采用这样的方式。 sudo yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine #### 1.2、添加 yum 源 　　在安装 Docker CE 的方式上，我是采用将 Docker CE 的源添加到 yum 源中，之后我们就可以直接使用 yum install 安装 Docker CE，整个的安装过程如下。 安装工具包从而可以让我们在 yum 中添加别的仓储源 ``` sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2 ``` 设置 docker ce 的稳定库地址 ``` sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo ``` 安装 docker ce ``` sudo yum install docker-ce docker-ce-cli containerd.io ``` 当我们安装好 Docker 之后，我们就可以使用 docker 命令验证我们是否在机器上成功安装了 Docker，同时，也可以使用 docker --version 命令查看我们安装的 Docker CE 版本。 ![cmn](./../img/net/c_sqlserver_nginx1.png) #### 1.3、设置开机自启 　　当 Docker 已经在我们的机器上安装完成后，我们就可以将 Docker 设置成机器的自启服务，这样，如果出现服务器重启的情况下，我们的 Docker 也可以随服务器的重启自动启动 Docker 服务。 启动 Docker 服务并允许开机自启 ``` sudo systemctl start docker ``` 查看当前 dokcer 的运行情况 ``` sudo systemctl status docker ``` #### 1.4、Hello World 　　就像我们在学习一门新的语言时，运行的第一句代码，几乎都是打印出 Hello World，而在 Docker Hub 中，也有这么一个镜像，在无数的 Docker 教程中，安装完 Docker 后，第一件事就是拉取这个镜像文件，“告诉” Docker，我来了。 　　Docker Hub 是存放镜像的仓库，里面包含了许多的镜像文件，因为服务器在国外的原因，下载的速度可能不理想，像国内的阿里云、腾讯云也有提供对于 Docker 镜像的加速器服务，你可以按需使用，当然，你也可以创建属于你的私有镜像仓库。 　　docker run 命令，它会在我们的本地镜像库中先寻找这个镜像，然后运行。如果在本地没有找到的话，则会自动使用 docker pull 从 Docker Hub 中寻找，能找到的话，则会自动下载到本地，然后运行，找不到的话，这条命令也就运行失败了。 ![cmn](./../img/net/c_sqlserver_nginx2.png) #### 1.5、安装 Docker Compose 　　在实际的项目开发中，我们可能会有多个应用镜像，例如在本篇文章的示例中，为了在 Docker 中运行我们的程序，我们需要三个镜像：应用程序自身镜像、sqlserver Server 镜像、以及 Nginx 镜像，为了将我们的程序启动起来，我们需要手敲各个容器的启动参数，环境变量，容器命名，指定不同容器的链接参数等等一系列的操作，又多又烦，可能某一步操作失败后程序就无法正常运行。而当我们使用了 Docker Compose 之后，我们就可以把这些命令一次性写在 docker-compose.yml 配置文件中，以后每次启动我们的应用程序时，只需要通过 docker compose 命令就可以自动帮我们完成这些操作。 从 github 下载 docker compose 二进制文件 ``` sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose ``` 对下载的二进制文件应用可执行权限 ``` sudo chmod +x /usr/local/bin/docker-compose ``` 查看 docker compose 版本 ``` docker-compose --version ``` #### 2、构建程序镜像 　　当我们在服务器上安装好 docker 和 docker compose 之后，就可以开始构建我们的程序镜像了。首先我们需要对我们的运行程序添加对于 Docker 的支持。你可以自己手动在 MVC 项目中添加 Dockerfile 文件，或是通过右键添加 Docker 支持。 ![cmn](./../img/net/c_sqlserver_nginx4.png) Dockerfile 就像一个执行的清单，它告诉 Docker，我们这个镜像在构建和运行时需要按照什么样的命令运行。打开 VS 为我们自动创建的 Dockerfile，可以看到清晰的分成了四块的内容。 ![cmn](./../img/net/c_sqlserver_nginx5.png) 我们知道，.NET Core 程序的运行需要依赖于 .NET Core Runtime（CoreCLR），因此，为了使我们的程序可以运行起来，我们需要从 hub 中拉取 runtime ，并在 此基础上构建我们的应用镜像。同时，为了避免因为基础的环境的不同造成对程序的影响，这里的 Runtime 需要同程序开发时的 .NET Core SDK 版本保持一致，所以这里我使用的是 .NET Core 3.0 Runtime。 　　一个镜像中包含了应用程序及其所有的依赖，与虚拟机不同的是，容器中的每个镜像最终是共享了宿主机的操作系统资源，容器作为用户空间中的独立进程运行在主机操作系统上。 ![cmn](./../img/net/c_sqlserver_nginx6.png) PS:图片版权归属于微软的技术文档，如有侵权，请联系我删除，源文件地址：什么是 Docker？ 　　镜像可以看成一个个小型的“虚拟主机”，这里我们在镜像中创建了一个 /app 路径作为我们程序在镜像中的工作目录，同时，将 80,443 端口暴露给 Docker，从而可以使我们在镜像外面通过端口访问到当前镜像中的运行的程序。 ``` FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim AS base WORKDIR /app EXPOSE 80 EXPOSE 443 ``` 　　因为我们的应用是一个微服架构的应用，最终的项目依赖于解决方案中的各个类库以及我们从 Nuget 中下载的各种第三方组件，在部署时，需要将这些组件打包成 dll 引用。所以，这里我们需要使用 .NET Core SDK 中包含的 .NET Core CLI 进行还原和构建。 　　就像在下面的代码中，我们在镜像的内部创建了一个 /src 的路径，将当前解决方案下的类库都复制到这个目录下，之后通过 dotnet restore 命令还原我们的主程序所依赖的各个组件。当我们还原好依赖的组件后，就可以使用 dotnet build 命令生成 Release版本的 dll 文件，同时输出到之前创建的 /app 路径下。 ``` FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster AS build WORKDIR /src COPY . . WORKDIR /src/templates/service/host/Base.IdentityServer RUN dotnet restore -nowarn:msb3202,nu1503 RUN dotnet build --no-restore -c Release -o /app ``` 上面一步可以看成我们在使用 VS 生成 Release 版本的解决方案，当生成没有出错之后，我们就可以进行程序的发布。 ``` FROM build AS publish RUN dotnet publish --no-restore -c Release -o /app ``` 　　当已经生成发布文件之后，按照我们平时部署在 Windows 上的过程，这时就可以通过 IIS 部署运行了，因此，构建我们应用镜像的最后一步就是通过 dotnet 命令执行我们的程序。 ``` FROM base AS final WORKDIR /app COPY --from=publish /app . ENTRYPOINT ["dotnet", "Base.IdentityServer.dll"] ``` 　　似乎到这一步构建程序镜像就结束了，按照这样流程做的话，就需要我们将整个的解决方案上传到服务器上了，可是，很多时候，我们仅仅是把我们在本地发布好的项目上传到服务器上，这与我们现在的构建流程具有很大的不同，所以这里我们来修改 Dockerfile 文件，从而符合我们的发布流程。 　　从上面分析 Dockerfile 的过程中不难看出，在服务器上构建镜像的第二步、第三步就是我们现在在开发环境中手动完成的部分，所以这里，我们只需要对这部分进行删除即可，修改后的 Dockerfile 如下。 ``` FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster WORKDIR /app COPY . /app EXPOSE 80 EXPOSE 443 ENTRYPOINT ["dotnet", "Base.IdentityServer.dll"] ``` 　　在修改后的 Dockerfile 中，可以看到，我们删去了 build 和 release 的过程，选择直接将我们 Dockerfile 路径下的文件拷贝到镜像中的 /app 路径下，然后直接执行 dotnet 命令，运行我们的程序。 　　为了确保 Dockerfile 与发布后的文件处于同一路径下，这里我们需要使用 VS 修改 Dockerfile 的属性值，确保会复制到输出的目录下，这里选择如果较新则复制即可。 ![cmn](./../img/net/c_sqlserver_nginx7.png) #### 3、编写 docker-compose.yml 　　当我们构建好应用的镜像，对于 Nginx 和 sqlserver 我们完全可以从 hub 中拉取下来，再执行一些配置即可。所以，我们现在就可以编写 docker compose 文件，来定义我们的应用镜像运行时需要包含的依赖以及每个镜像的启动顺序。 　　右键选中 MVC 项目，添加一个 docker-compose.yml 文件，同样的，需要修改该文件的属性，以便于该文件可以复制到输出目录下。注意，这里的文件名和上文的 Dockerfile 都是特定的，你不能做任何的修改。如果你的电脑上已经安装了 Docker for Windows，你也可以使用 VS，右键添加，选中容器业务流程协调程序支持自动对 docker compose 进行配置。 ![cmn](./../img/net/c_sqlserver_nginx8.png) 在 yml 文件中，我定义了三个镜像：AdminApiGateway.Host、Base.IdentityServer、Base.HttpApi.Host。三个镜像的定义中有许多相同的地方，都设置了自动重启（restart），以及都处于同一个桥接网络下（psu-net）从而达到镜像间的通信。 sqlserver 是 SqlServer 的镜像，我们通过环境变量 SA_PASSWORD 设置了 SqlServer 的数据库连接密码，并通过挂载卷的方式将镜像中的数据库文件持久化到我们的服务器本地路径中。同时，将镜像的 1433 端口映射到服务器的 1433 端口上。 　　AdminApiGateway.Host 则是我们的程序后台网关镜像，采用位于 /data/dotnet/AdminApiGateway/ 路径下的 Dockerfile 文件进行构建的，因为主程序的运行需要依赖于数据库，所以这里采用 depends_on 属性，使我们的应用镜像依赖于 sqlserver 镜像，即，在 sqlserver 启动后才会启动应用镜像。 　　nginx 则是我们的 nginx 镜像，这里将镜像中的 80 端口和 443 端口都映射到服务器 IP 上，因为我们需要配置 Nginx 从而监听我们的程序，所以通过挂载卷的方式，将本地的 nginx.conf 配置文件用配置映射到镜像中。同时，因为我们在构建应用镜像的 Dockerfile 文件时，对外暴露了 80,443 端口，所以这里就可以通过 links 属性进行监听（如果构建时未暴露端口，你可以在 docker compose 文件中通过 Expose 属性暴露镜像中的端口）。 　　Nginx 的配置文件如下，这里特别需要注意文件的格式，缩进，一点小错误都可能导致镜像无法正常运行。如果你和我一样将 nginx.conf 放到程序运行路径下的，别忘了修改文件的属性。 ``` user nginx; worker_processes 1; error_log /var/log/nginx/error_log.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; #gzip on; gzip on; #开启gzip gzip_disable "msie6"; #IE6不使用gzip gzip_vary on; #设置为on会在Header里增加 "Vary: Accept-Encoding" gzip_proxied any; #代理结果数据的压缩 gzip_comp_level 6; #gzip压缩比（1~9），越小压缩效果越差，但是越大处理越慢，所以一般取中间值 gzip_buffers 16 8k; #获取多少内存用于缓存压缩结果 gzip_http_version 1.1; #识别http协议的版本 gzip_min_length 1k; #设置允许压缩的页面最小字节数，超过1k的文件会被压缩 gzip_types application/javascript text/css; #对特定的MIME类型生效,js和css文件会被压缩 include /etc/nginx/conf.d/*.conf; server { #nginx同时开启http和https listen 80 default backlog=2048; listen 443 ssl; server_name ysf.djtlpay.com; ssl_certificate /ssl/1_ysf.djtlpay.com_bundle.crt; ssl_certificate_key /ssl/2_ysf.djtlpay.com.key; location / { proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Host $http_host; proxy_cache_bypass $http_upgrade; root /usr/share/nginx/html; index index.html index.htm; } } } ``` 一个完整的 docker compose 文件如下，包含了三个镜像以及一个桥接网络。 ``` version: '3.0' services: # 指定服务名称 sqlserver: # 指定服务使用的镜像 image: mcr.microsoft.com/mssql/server # 指定容器名称 container_name: sqlserver # 指定服务运行的端口 ports: - "1433" # 指定容器中需要挂载的文件 volumes: - /data/sqlserver:/var/opt/mssql # 挂断自动重新启动 restart: always environment: - TZ=Asia/Shanghai - SA_PASSWORD=mssql-MSSQL - ACCEPT_EULA=Y # 指定容器运行的用户为root user: root # 指定服务名称 redis: # 指定服务使用的镜像 image: redis # 指定容器名称 container_name: redis # 指定服务运行的端口 ports: - 6379:6379 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/redis:/data - /data/redis/redis.conf:/etc/redis.conf # 挂断自动重新启动 restart: always # 指定容器执行命令 command: redis-server /etc/redis.conf --requirepass xiujingredis. --appendonly yes # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 # 指定服务名称 mongo: # 指定服务使用的镜像 image: mongo # 指定容器名称 container_name: mongo # 指定服务运行的端口 ports: - 27017:27017 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime - /data/mongodb/db:/data/db - /data/mongodb/configdb:/data/configdb - /data/mongodb/initdb:/docker-entrypoint-initdb.d # 挂断自动重新启动 restart: always # 指定容器的环境变量 environment: - TZ=Asia/Shanghai # 设置容器时区与宿主机保持一致 - AUTH=yes - MONGO_INITDB_ROOT_USERNAME=admin - MONGO_INITDB_ROOT_PASSWORD=admin nginx: # 指定服务使用的镜像 image: nginx # 指定容器名称 container_name: nginx # 指定服务运行的端口 ports: - 80:80 - 443:443 # 指定容器中需要挂载的文件 volumes: - /etc/localtime:/etc/localtime # 挂断自动重新启动 restart: always AdminApiGateway.Host: image: 'volosoft/microservice-demo-public-website-gateway:${TAG:-latest}' build: context: ../ dockerfile: micro/gateways/AdminApiGateway.Host/Dockerfile depends_on: - sqlserver - redis - mongo Base.IdentityServer: image: 'Base.IdentityServer:${TAG:-latest}' build: context: ../ dockerfile: micro/modules/base/host/Base.IdentityServer/Dockerfile depends_on: - sqlserver - redis - mongo - AdminApiGateway.Host Base.HttpApi.Host: image: 'Base.HttpApi.Host:${TAG:-latest}' build: context: ../ dockerfile: micro/modules/base/host/Base.HttpApi.Host/Dockerfile depends_on: - sqlserver - redis - mongo - AdminApiGateway.Host - Base.IdentityServer ``` 这里需要注意，所有有用到镜像间的通信的地方，我们都需要使用镜像名进行指代，例如上面的 nginx 的配置文件中，我们需要将监听的地址改为镜像名称，以及，我们需要修改程序的数据库访问字符串的服务器地址 #### 4、发布部署程序 　　当我们构建好 docker compose 文件后就可以把整个文件上传到服务器上进行构建 docker 镜像了。这里我将所有的部署文件放在服务器的 /data/wwwroot/micro/ 路径下，这时我们就可以通过 docker compose 命令进行镜像构建。 　　定位到部署文件在的位置，我们可以直接使用下面的命令进行镜像的（重新）构建，启动，并链接一个服务相关的容器，整个过程都会在后台运行，如果你希望看到整个过程的话，你可以去掉 -d 参数。 执行镜像构建，启动 ``` docker-compose up -d ``` 　　当 up 命令执行完成后，我们就可以通过 ps 命令查看正在运行的容器，若有的容器并没有运行起来，则可以使用 logs 查看容器的运行日志从而进行排错。 查看所有正在运行的容器 ``` docker-compose ps ``` 显示容器运行日志 ``` docker-compose logs ``` ### 三、总结 　　 本章主要是介绍了如何通过 docker 容器，完整的部署一个可实际使用的 .NET Core 的单体应用，相比于之前通过 Linux 部署 .NET Core 应用，可以看到整个步骤少了很多，也简单很多。文中涉及到了一些 docker 的命令，如果你之前并没有接触过 docker 的话，可能需要你进一步的了解。当我们将程序打包成一个镜像之后，你完全可以将镜像上传到私有镜像仓库中，或是直接打包成镜像的压缩文件，这样，当需要切换部署环境时，只需要获取到这个镜像之后即可快速完成部署，相比之前，极大的方便了我们的工作。 ================================================ FILE: docs/php/kj.md ================================================ # PHP 在PHP开发中，选择合适的框架有助于加快软件开发，节约宝贵的项目时间，让开发者专注于功能的实现上。由于流行的框架经过了大量项目的检验，所以使用框架还有助于创建更加稳定和安全的应用程序。本文搜集了目前全球最流行的25款PHP框架，欢迎大家交流学习。 ## 1、Laravel Laravel是一个简单优雅的PHPWeb开发框架，可以将开发者从意大利面条式的代码中解放出来，通过简单、高雅、表达式语法开发出很棒的Web应用，Laravel拥有更富有表现力的语法、高质量的文档、丰富的扩展包，被称为“巨匠级PHP开发框架”。 ## 2、Phalcon Phalcon是一个开源的、全栈的、用C语言编写的PHP5框架，为开发者提供了网站及应用开发所需的大量高级工具，且Phalcon是松耦合的，开发者可以根据需要使用其他组件。Phalcon中的所有函数都以PHP类的方式呈现，开发者无需学习和使用C语言，且无需担心性能问题。 ## 3、Symfony2 Symfony2是一个开源的PHPWeb框架，有着开发速度快、性能高等特点。与其他框架相比，Symfony2的优势包括：支持DI（依赖注入）和IoC（控制反转）；扩展性强；文档和社区比较成熟。但是Symfony2的学习曲线也比较陡峭，没有经验的初学者往往需要一些练习才能掌握其特性。 ## 4、CodeIgniter CodeIgniter是一个简单快速的PHPMVC框架。CodeIgniter不需要大量代码（1.6.2版本仅为2.8MB，其中的1.3MB是可以删除的用户文档），也不会要求您插入类似于PEAR的庞大的库。它在PHP4和PHP5中表现同样良好，允许您创建可移植的应用程序。最后，您不必使用模板引擎来创建视图—只需沿用旧式的HTML和PHP即可。CodeIgniter入门非常容易，而且有很全面的中文版用户开发手册，在官方网站上可以了解到更多内容。 ## 5、Yii Yii是一个基于组件的高性能PHP框架，用于开发大型Web应用。Yii采用严格的OOP编写，并有着完善的库引用以及全面的教程。从MVC，DAO/ActiveRecord，widgets，caching，等级式RBAC，Web服务，到主题化，I18N和L10N，Yii提供了今日Web2.0应用开发所需要的几乎一切功能。事实上，Yii是最有效率的PHP框架之一。Yii是一个高性能的PHP5的web应用程序开发框架。通过一个简单的命令行工具yiic可以快速创建一个web应用程序的代码框架，开发者可以在生成的代码框架基础上添加业务逻辑，以快速完成应用程序的开发。 ## 6、Aura Aura为PHP5.4及以上版本提供独立的类库，它的代码非常干净，并且真正独立。这些包可以单独使用，也可以将它们合并到开发者自己的框架中。国内资料非常少。 ## 7、Cakephp CakePHP是一个运用了诸如ActiveRecord、AssociationDataMapping、FrontController和MVC等著名设计模式的快速开发框架。该项目主要目标是提供一个可以让各种层次的PHP开发人员快速地开发出健壮的Web应用，而又不失灵活性。CakePHP是一个基于PHP，免费且开源的迅速发展框架最开始从RubyOnRails框架里得到灵感。CakePHP拥有一个活跃的开发团队以及社区，使CakePHP本身更具备应有的价值。另外，使用CakePHP也意味着您的应用程序将更容易地测试以及更容易地被改良、更新。 ## 8、Zend 由最流行最专业的PHPIDE产品ZendStudio的开发商开发，ZendFramework(简写ZF)是开源的，主要用于Web应用程序的开发和服务，ZF采用MVC(Model–View-Controller)架构模式来分离应用程序中不同的部分方便程序的开发和维护。 ## 9、Kohana Kohana是一款纯PHP5的框架，基于MVC模式开发，它的特点就是高安全性，轻量级代码，容易使用。2009年9月发布了3.0版本，支持HMVC模式。 ## 10、FuelPHP FuelPHP是一个简单的、灵活的、社区驱动的PHP5.3web框架。它的诞生源自于很多开发社区对于现有开发框架的不满，FuelPHP非常便携，几乎所有的服务器都能够很好的支持，并且简洁强大的语法也能够让你很快喜欢上它。 ## 11、Slim Slim是一款PHP微框架，可以帮助你快速编写简单但功能强大的web应用和API。 ## 12、Typo3 Typo3内容管理系统，是基于PHP4/PHP5+MYsql的内容管理系统(框架)(CMS/CMF),兼容PHP4和PHP5.数据库系统除Mysql之外，也能运行于Oracle,MS-SQL,ODBC,LDAP等其它数据库系统，支持Typo3的服务器系统：Apache或者IIS架设的服务器。 ## 13、ThinkPHP ThinkPHP是为了简化企业级应用开发和敏捷WEB应用开发而诞生的。最早诞生于2006年初，2007年元旦正式更名为ThinkPHP，并且遵循Apache2开源协议发布。ThinkPHP从诞生以来一直秉承简洁实用的设计原则，在保持出色的性能和至简的代码的同时，也注重易用性。并且拥有众多原创功能和特性，在社区团队的积极参与下，在易用性、扩展性和性能方面不断优化和改进。ThinkPHP是一个快速、兼容而且简单的轻量级国产PHP开发框架，诞生于2006年初，原名FCS，2007年元旦正式更名为ThinkPHP，遵循Apache2开源协议发布，从Struts结构移植过来并做了改进和完善，同时也借鉴了国外很多优秀的框架和模式，使用面向对象的开发结构和MVC模式，融合了Struts的思想和TagLib（标签库）、RoR的ORM映射和ActiveRecord模式。 ## 14、Flight Flight是一个快速、简单、可扩展的微型PHP框架，允许用户快速的构建RestfulWeb应用程序，同样易于学习和使用，简单但是很强大！ ## 15、PHPixie PHPixie是一个轻量级的MVCPHP框架,设计用于快速开发,易于学习,并提供一个坚实的基础开发框架。 ## 16、Yaf Yaf，全称YetAnotherFramework，是一个C语言编写的PHP框架，是一个以PHP扩展形式提供的PHP开发框架,相比于一般的PHP框架,它更快，更轻便.它提供了Bootstrap,路由,分发,视图,插件,是一个全功能的PHP框架。最新版本为2014-10-22发布的2.3.3。 ## 17、Swoole Swoole是一种PHP高级Web开发框架，框架不是为了提升网站的性能，是为了提升网站的开发效率。最少的性能损耗，换取最大的开发效率。利用Swoole框架，开发一个复杂的Web功能，可以在很短的时间内完成。 ## 18、Nette Nette框架可以让您更有效地工作，专注于重要的内容，并使您的代码更加可读和结构化。 ## 19、Drupal Drupal是使用PHP语言编写的开源内容管理框架（CMF），它由内容管理系统（CMS）和PHP开发框架（Framework）共同构成。连续多年荣获全球最佳CMS大奖，是基于PHP语言最著名的WEB应用程序。截止2011年底，共有13,802位WEB专家参加了Drupal的开发工作；228个国家使用181种语言的729,791位网站设计工作者使用Drupal。著名案例包括：联合国、美国白宫、美国商务部、纽约时报、华纳、迪斯尼、联邦快递、索尼、美国哈佛大学、Ubuntu等。 ## 20、Workerman Workerman是一款纯PHP开发的开源高性能的PHPsocket服务器框架。被广泛的用于手机app、移动通讯，微信小程序，手游服务端、网络游戏、PHP聊天室、硬件通讯、智能家居、车联网、物联网等领域的开发。支持TCP长连接，支持Websocket、HTTP等协议，支持自定义协议。拥有异步Mysql、异步Redis、异步Http、异步消息队列等众多高性能组件。 ## 21、PHPDevShell PHPDevShell是一个开源（GNU/LGPL）的快速应用开发框架，用于开发不含Java的纯PHP。它有一个完整的GUI管理员后台界面。其主要目标在于开发插件一类的基于管理的应用，其中速度、安全、稳定性及弹性是最优先考虑的重点。其设计形成了一个简单的学习曲线，PHP开发者无需学习复杂的新术语。PHPDevShell的到来满足了开发者们对于一个轻量级但是功能完善，可以无限制的进行配置的GUI的需求。 ## 22、Akelos AkelosPHP框架是一个基于MVC设计模式的web应用开发平台。基于良好的使用习惯，使用它可以完成如下任务：◆方便的使用Ajax编写views；◆通过控制器管理请求（request）及响应（response）；◆管理国际化的应用；◆使用简单的协议与模型及数据库通信 你的Akelos应用可以在大多数共享主机服务供应方上运行，因为Akelos对服务器唯一的要求就是支持PHP。因此，AkelosPHP框架是理想的用于发布单独web应用的框架，因为它不需要非标准PHP配置便能运行。 ## 23、Prado PRADO团队由一些PRADO狂热者组成，这些成员开发并推动PRADO框架以及相关项目的进行。 PRADO的灵感起源于ApacheTapestry。从04年开始，PRADO成为SourceForge上的开源项目之一。这个项目目前进展到了3.x版本。 ## 24、ZooP ZoopPHP框架，意为Zoop面向对象的PHP框架。这是个稳定，可伸缩并可移植的框架。从诞生到现在的5年间，已经在不少产品开发中被使用。Zoop是一个快速，有效并干净的框架。它的伸缩性很好，你可以只安装你需要的功能。对代码并不很熟悉的开发者也可以通过Zoop快速的开发安全的web应用。熟练的开发者则可以更加将Zoop的弹性利用到极致。 ## 25、QPHP QPHP，意为快速PHP，它是一个与ASP.NET类似的MVC框架。基本上它是这样一个情况：◆整合了Java和C#的美感；◆除去了在其他PHP框架中使用的Perl形式的意义含糊的语言；◆大量基于OOP的概念 最后说一句，PHP是世界上最好的语言！ ================================================ FILE: docs/python/fabric.md ================================================ # 远程部署神器 Fabric，支持 Python3 如果你搜一圈 “Fabric “关键字，你会发现 90% 的资料都是过时的，因为现在 Fabric 支持 Python3，但是它又不兼容旧版 Fabric。所以，如果你按照那些教程去操作的话根本跑不通。 如果你还没用过 Fabric，那么这篇文章就是帮你快速上手 Fabric 的。不管你现在用不用，先了解了以后也用得着。 平时我们的开发流程是这样，经过几个月奋战，项目终于开发完了，测试也没问题了，我们就把代码提交到 GitHub 那样的托管平台，准备部署到正式环境。你小心翼翼地登录到正式服务器，进入到项目目录中，把代码从远程仓库拉下来，然后启动程序。后面每次有新功能发布或者哪怕只是修改了一个小小的 Bug 时，你都要执行重复的操作，登录服务器，切换到指定目录，拉取代码，重启服务。 其实这种操作非常繁琐，也没什么技术含量，还容易出问题，于是 Fabric 出场了。Fabric 是一个远程部署神器，它可以在本地执行远程服务器的命令。 怎么做？很简单，就几个步骤。 ### 安装 Fabric ``` pip install fabric --upgrade ``` 注意，如果你安装的是旧版的 Fabric，那么新版的 Fabric 是不兼容旧版的，目前 Fabric 有三个版本，Fabric1 就是以前的 Fabric，只支持 Python2，已不推荐使用，而 Fabric2 就是现在的 Fabric，同时支持 Python2 和 Python3， 也是官方强烈推荐的版本， 还有一个 Fabric3，这是网友从旧版的 Fabric1 克隆过来的非官方版本，但是兼容 Fabric1，也支持 Python2 和 Python3。 最新的 Fabric 不需要 fabfile.py 文件， 也不需要 fab 命令，而现在网络上几乎所有的教程、资料都还是基于 fabric1 写的，当你在看那些教程的时候，注意甄别。 而新版 Fabric 提供的 API 非常简单。 ### 运行命令 先看个例子，下面是一段部署脚本 ```python # deploy.py # 1. 创建一个远程连接 # 2. 进入指定目录 # 3. 在指定目录下面执行重启命令 from fabric import Connection def main(): # ip 我是随便填的 # 如果你的电脑配了ssh免密码登录，就不需要 connect_kwargs 来指定密码了。 c = Connection("root@232.231.231.22", connect_kwargs={"password": "youpassword"}) with c.cd('/var/www/youproject'): c.run("git pull origin master") c.run("/usr/bin/supervisorctl -c ../supervisor/supervisord.conf restart youproject") if __name__ == '__main__': main() ``` 执行 ``` python deploy.py ``` 执行完成后，最新代码就已经部署到正式环境并重启了服务，是不是非常方便，妈妈再也不要担心我在正式环境敲错命令删数据库跑路了。 Fabric 不仅支持 Linux，而且在 Windows 平台也能很好的运行，在中小型项目，它是非常不错的运维工具，有了 Frabic ，管理上百台服务器都不成问题。 ### 构建连接 ```python class Connection(Context): host = None user = None port = None ssh_config = None connect_timeout = None connect_kwargs = None ... ``` 构建 Connection 对象的方式有不同的方式，例如你可以将 host 写成 “root@192.168.101.1:22” ，也可以作为3个参数分开写。而 connect_kwargs 是字典对象，通常填服务器的登录密码或者密钥。 ### 上传文件 run 方法用于执行命令，cd 进入指定目录，put 方法用于上传文件， 例如： ```python from fabric import Connection c = Connection('web1') c.put('myfiles.tgz', '/opt/mydata') c.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz') ``` ### 多台服务器 如果是要在多台服务器运行命令，简单的办法就是使用迭代，挨个服务器执行命令： ```python # web1,web2,mac1 都是服务器的名字，你也可以用ip代替 >>> from fabric import Connection >>> for host in ('web1', 'web2', 'mac1'): >>> result = Connection(host).run('uname -s') ... print("{}: {}".format(host, result.stdout.strip())) ... web1: Linux web2: Linux mac1: Darwin ``` 或者使用 SerialGroup ```python from fabric import SerialGroup as Group pool = Group('web1', 'web2', 'web3', connect_kwargs={"password": "youpassword"} ) pool.put('myfiles.tgz', '/opt/mydata') pool.run('tar -C /opt/mydata -xzvf /opt/mydata/myfiles.tgz') ``` Group(*hosts, **kwargs) 参数说明： *hosts: 可以传入多个主机名或IP **kwargs 接收的参数可以和Connection一样，可以指定密码 本文完，你 get 了吗？ [更多参考请点击：](https://www.fabfile.org/)https://www.fabfile.org/ ================================================ FILE: docs/python/feature.md ================================================ # Python新特性 ## Python2.x与3​​.x版本区别 Python的3​​.0版本，常被称为Python 3000，或简称Py3k。相对于Python的早期版本，这是一个较大的升级。 为了不带入过多的累赘，Python 3.0在设计的时候没有考虑向下相容。 许多针对早期Python版本设计的程式都无法在Python 3.0上正常执行。 为了照顾现有程式，Python 2.6作为一个过渡版本，基本使用了Python 2.x的语法和库，同时考虑了向Python 3.0的迁移，允许使用部分Python 3.0的语法与函数。 新的Python程式建议使用Python 3.0版本的语法。 除非执行环境无法安装Python 3.0或者程式本身使用了不支援Python 3.0的第三方库。目前不支援Python 3.0的第三方库有Twisted, py2exe, PIL等。 大多数第三方库都正在努力地相容Python 3.0版本。即使无法立即使用Python 3.0，也建议编写相容Python 3.0版本的程式，然后使用Python 2.6, Python 2.7来执行。 Python 3.0的变化主要在以下几个方面: ### print 函数 print语句没有了，取而代之的是print()函数。 Python 2.6与Python 2.7部分地支持这种形式的print语法。在Python 2.6与Python 2.7里面，以下三种形式是等价的： ```python print "fish" print ("fish") #注意print后面有个空格 print("fish") #print()不能带有任何其它参数 ``` 然而，Python 2.6实际已经支持新的print()语法： ```python from __future__ import print_function print("fish", "panda", sep=', ') ``` ### Unicode Python 2 有 ASCII str() 类型，unicode() 是单独的，不是 byte 类型。 现在， 在 Python 3，我们最终有了 Unicode (utf-8) 字符串，以及一个字节类：byte 和 bytearrays。 由于 Python3.X 源码文件默认使用utf-8编码，这就使得以下代码是合法的： ```python >>> 中国 = 'china' >>>print(中国) china ``` Python 2.x ```python >>> str = "我爱北京天安门" >>> str '\xe6\x88\x91\xe7\x88\xb1\xe5\x8c\x97\xe4\xba\xac\xe5\xa4\xa9\xe5\xae\x89\xe9\x97\xa8' >>> str = u"我爱北京天安门" >>> str u'\u6211\u7231\u5317\u4eac\u5929\u5b89\u95e8' ``` Python 3.x ```python >>> str = "我爱北京天安门" >>> str '我爱北京天安门' ``` ### 除法运算 Python中的除法较其它语言显得非常高端，有套很复杂的规则。Python中的除法有两个运算符，/和// 首先来说/除法: 在python 2.x中/除法就跟我们熟悉的大多数语言，比如Java啊C啊差不多，整数相除的结果是一个整数，把小数部分完全忽略掉，浮点数除法会保留小数点的部分得到一个浮点数的结果。 在python 3.x中/除法不再这么做了，对于整数之间的相除，结果也会是浮点数。 Python 2.x: ```python >>> 1 / 2 0 >>> 1.0 / 2.0 0.5 ``` Python 3.x: ```python >>> 1/2 0.5 ``` 而对于//除法，这种除法叫做floor除法，会对除法的结果自动进行一个floor操作，在python 2.x和python 3.x中是一致的。 python 2.x: ```python >>> -1 // 2 -1 ``` python 3.x: ```python >>> -1 // 2 -1 ``` 注意的是并不是舍弃小数部分，而是执行 floor 操作，如果要截取整数部分，那么需要使用 math 模块的 trunc 函数 python 3.x: ```python >>> import math >>> math.trunc(1 / 2) 0 >>> math.trunc(-1 / 2) 0 ``` ### 异常 在 Python 3 中处理异常也轻微的改变了，在 Python 3 中我们现在使用 as 作为关键词。 捕获异常的语法由 except exc, var 改为 except exc as var。 使用语法except (exc1, exc2) as var可以同时捕获多种类别的异常。 Python 2.6已经支持这两种语法。 1. 在2.x时代，所有类型的对象都是可以被直接抛出的，在3.x时代，只有继承自BaseException的对象才可以被抛出。 2. 2.x raise语句使用逗号将抛出对象类型和参数分开，3.x取消了这种奇葩的写法，直接调用构造函数抛出对象即可。 在2.x时代，异常在代码中除了表示程序错误，还经常做一些普通控制结构应该做的事情，在3.x中可以看出，设计者让异常变的更加专一，只有在错误发生的情况才能去用异常捕获语句来处理。 ### xrange 在 Python 2 中 xrange() 创建迭代对象的用法是非常流行的。比如： for 循环或者是列表/集合/字典推导式。 这个表现十分像生成器（比如。"惰性求值"）。但是这个 xrange-iterable 是无穷的，意味着你可以无限遍历。 由于它的惰性求值，如果你不得仅仅不遍历它一次，xrange() 函数 比 range() 更快（比如 for 循环）。尽管如此，对比迭代一次，不建议你重复迭代多次，因为生成器每次都从头开始。 在 Python 3 中，range() 是像 xrange() 那样实现以至于一个专门的 xrange() 函数都不再存在（在 Python 3 中 xrange() 会抛出命名异常）。 ```python import timeit n = 10000 def test_range(n): return for i in range(n): pass def test_xrange(n): for i in xrange(n): pass ``` Python 2 ```python print 'Python', python_version() print '\ntiming range()' %timeit test_range(n) print '\n\ntiming xrange()' %timeit test_xrange(n) #Python 2.7.6 timing range() 1000 loops, best of 3: 433 µs per loop timing xrange() 1000 loops, best of 3: 350 µs per loop ``` Python 3 ```python print('Python', python_version()) print('\ntiming range()') %timeit test_range(n) #Python 3.4.1 timing range() 1000 loops, best of 3: 520 µs per loop ``` ```python print(xrange(10)) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 print(xrange(10)) NameError: name 'xrange' is not defined ``` ### 八进制字面量表示 八进制数必须写成0o777，原来的形式0777不能用了；二进制必须写成0b111。 新增了一个bin()函数用于将一个整数转换成二进制字串。 Python 2.6已经支持这两种语法。 在Python 3.x中，表示八进制字面量的方式只有一种，就是0o1000。 python 2.x ```python >>> 0o1000 512 >>> 01000 512 ``` python 3.x ```python >>> 01000 File "", line 1 01000 ^ SyntaxError: invalid token >>> 0o1000 512 ``` ### 不等运算符 Python 2.x中不等于有两种写法 != 和 <> Python 3.x中去掉了<>, 只有!=一种写法，还好，我从来没有使用<>的习惯 ### 去掉了repr表达式`` Python 2.x 中反引号``相当于repr函数的作用 Python 3.x 中去掉了``这种写法，只允许使用repr函数，这样做的目的是为了使代码看上去更清晰么？不过我感觉用repr的机会很少，一般只在debug的时候才用，多数时候还是用str函数来用字符串描述对象。 ```python def sendMail(from_: str, to: str, title: str, body: str) -> bool: pass ``` ### 多个模块被改名（根据PEP8） | 旧的名字 | 新的名字 | | ------------ | ------------- | | _winreg | winreg | | ConfigParser | configParaser | | copy_reg | copyreg | | Queue | queue | | SocketServer | socketserver | | repr | reprlib | StringIO模块现在被合并到新的io模组内。 new, md5, gopherlib等模块被删除。 Python 2.6已经支援新的io模组。 httplib, BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer, Cookie, cookielib被合并到http包内。 取消了exec语句，只剩下exec()函数。 Python 2.6已经支援exec()函数。 ### 数据类型 1. Py3.X去除了long类型，现在只有一种整型——int，但它的行为就像2.X版本的long 2. 新增了bytes类型，对应于2.X版本的八位串，定义一个bytes字面量的方法如下： ```python >>> b = b'china' >>> type(b) ``` str对象和bytes对象可以使用.encode() (str -> bytes) or .decode() (bytes -> str)方法相互转化。 ```python >>> s = b.decode() >>> s 'china' >>> b1 = s.encode() >>> b1 b'china' ``` 3. dict的.keys()、.items 和.values()方法返回迭代器，返回的值不再是list，而是view。所以dict.iterkeys(),dict.iteritems()和dict.itervalues()被去掉了。同时去掉的还有 dict.has_key()，用 in替代它吧 。 ### 打开文件 原： ```python file( ..... ) #或 open(.....) ``` 改为只能用 ```python open(.....) ``` ### 从键盘录入一个字符串 在python2.x中raw_input()和input( )，两个函数都存在，其中区别为： raw_input()---将所有输入作为字符串看待，返回字符串类型 input()-----只能接收"数字"的输入，在对待纯数字输入时具有自己的特性，它返回所输入的数字的类型（int, float ） 在python3.x中raw_input()和input( )进行了整合，去除了raw_input()，仅保留了input()函数，其接收任意任性输入，将所有输入默认为字符串处理，并返回字符串类型。 原: ```python raw_input( "提示信息" ) ``` 改为: ```python input( "提示信息" ) ``` ### map、filter 和 reduce 这三个函数号称是函数式编程的代表。在 Python3.x 和 Python2.x 中也有了很大的差异。 首先我们先简单的在 Python2.x 的交互下输入 map 和 filter,看到它们两者的类型是 built-in function(内置函数): ```python >>> map >>> filter >>> ``` 它们输出的结果类型都是列表: ```python >>> map(lambda x:x *2, [1,2,3]) [2, 4, 6] >>> filter(lambda x:x %2 ==0,range(10)) [0, 2, 4, 6, 8] >>> ``` 但是在Python 3.x中它们却不是这个样子了： ```python >>> map >>> map(print,[1,2,3]) >>> filter >>> filter(lambda x:x % 2 == 0, range(10)) >>> ``` 首先它们从函数变成了类，其次，它们的返回结果也从当初的列表成了一个可迭代的对象, 我们尝试用 next 函数来进行手工迭代: ```python >>> f =filter(lambda x:x %2 ==0, range(10)) >>> next(f) 0 >>> next(f) 2 >>> next(f) 4 >>> next(f) 6 >>> ``` 对于比较高端的 reduce 函数，它在 Python 3.x 中已经不属于 built-in 了，被挪到 functools 模块当中。 ## Python3.+新特性 新的语法特性： * 格式化字符串字面值 * 数字字符串中支持下划线 * 变量注释的语法 * 异步生成器 * 异步列表推导 * 用类处理数据时减少样板代码的数据类 * 一处可能无法向后兼容的变更涉及处理生成器中的异常 * 面向解释器的“开发模式 * 具有纳秒分辨率的时间对象 * 环境中默认使用UTF-8编码的UTF-8模式 * 触发调试器的一个新的内置函数 ### 新的模块 添加了一个安全模块secrets到标准库中 ### CPython实现改进 * 字典基于Raymond Hettinger的建议使用更紧凑的表示重新实现了，和PyPy的字典实现类似。结果是和3.5版本相比，3.6版本字典的内存使用减少了20%到25%。 * 使用新的协议，自定义类的创建被简化了。 * 类属性的定义顺序能够被保存。 * 在**kwargs中元素的顺序对应于传递给函数时的关键字参数的顺序。 * 添加了DTrace和SystemTap探测的支持。 * 新的PYTHONMALLOC环境变量可以用于调试解释器内存分配和访问错误。 ### 重大的标准库改进 * asyncio模块接收了新特性，重大的可用性和性能改进，然后修复了大量的BUG。从Python 3.6开始asyncio模块不再是临时的了，它已经被认为是稳定的了。 * 新的文件系统路径协议已实现，用于支持路径类对象。所有标准库函数在处理路径时已使用新的协议。 * datetime模块在本地时间消除歧义上获得了支持。 * typing模块接受了一些改进。 * tracemalloc模块经过重大改造，现在可以为ResourceWarnning提供更好的输出，也为内存分配错误提供更好的诊断。 ### 安全改进 * secrets模块已被添加，可以生成更安全的伪随机数。 * 在Linux上，os.urandom()现在会被锁住，直到系统的伪随机滴池被初始化增加安全。 * hashlib和ssl模块现在支持OpenSSL 1.1.0。 * hashlib模块现在支持BLAKE2、SHA-3和SHAKE摘要算法和scrypt()秘钥导出功能。 ### Windows改进 * Windows文件系统和控制台编码改为了UTF-8。 * python.exe和pythonw.exe现在支持长路径，详情请看[removing the MAX_PATH limitation](https://docs.python.org/3/using/windows.html#max-path)。 * 一个._pth文件可以被添加用于隔离模块，避免全路径搜索，详情请看文档。 ### Formatted字符串字面值 Formatted字符串是带有’f’字符前缀的字符串，可以很方便的格式化字符串。 ```python >>> name = "xiaoming" >>> f"He name is {name}" 'He name is xiaoming' >>> width = 10 >>> precision = 4 >>> value = decimal.Decimal("12.34567") >>> f"result: {value:{width}.{precision}}" 'result: 12.35' ``` ### 数字中支持下划线 数字中支持使用下划线，方便阅读，例如： ```python >>> 1_000_000_000_000_000 1000000000000000 >>> 0x_FF_FF_FF_FF 4294967295 ``` 字符串format方法也支持了’_’选项，当格式化为浮点数或整数时，以3位分隔，当格式化为’b’,’o’,’x’和’X’时，以4位分隔 ```python >>> '{:_}'.format(10000000) '10_000_000' >>> '{:_b}'.format(10000000) '1001_1000_1001_0110_1000_0000' ``` ### 变量注释语法 变量注释没有给变量带来特殊的意义，只是为了方便IDE做类型检查。 ```python >>> from typing import List,Dict >>> primes: List[int] = [] >>> stats: Dict[str, int] = {} ``` 上面代码中primes为变量名，List[int]为变量注释，用来说明primes列表是用来存放int类型数据的，但是这个不是强制性的，你使用append()方法添加一个str类型数据也是可以的，IDE会提示你添加的数据有误，但是运行时不会报错。 ### 异步生成器 在Python3.5中，await和yield不能再同一个函数中使用，但是Python3.6已经取消了这个限制，可以在同一个函数体中使用了 ```python async def ticker(delay, to): """Yield numbers from 0 to *to* every *delay* seconds.""" for i in range(to): yield i await asyncio.sleep(delay) ``` ### 异步列表推导 增加在list、set和dict的列表推导和生成表达式中使用async for。 如下面这段代码: ```python result = [] async for i in aiter(): if i % 2: result.append(i) ``` 使用异步推导式之后，可以简写成 ```python result = [i async for i in aiter() if i % 2] ``` 现在也支持在所有的推导式中使用await表达式 ``` result = [await fun() for fun in funcs] ``` ### Python数据类 众所周知，Python是处理结构化数据的一种快捷又方便的方法。Python提供了用来组织管理结构，并将常见行为与数据实例联系起来的类，但是拥有许多初始化器的类历来存在这个弊端：需要大量的样板代码为它们创建实例。比如说： ```python class User(): def __init__(self,name,user_id,just_joined=True): self.name=name self.id=user_id self.just_joined=just_joined ``` 为了使这实现自动化：为类创建实例，Python 3.7引入了一个新的模块dataclasses，如[pep-0557](https://www.python.org/dev/peps/pep-0557/)中所述。它提供了一个装饰器，能够以异常简单的方式重现上述行为： ```python @dataclass class User(): name:str user_id:int just_joined:bool=True ``` 因而生成的类运行起来如同普通的Python类。你还可以声明某些字段是“冻结”或不可变的，并且使创建属性的特殊方法（比如__hash__或__repr__）实现自动化（或手动覆盖）。 ### Python生成器异常处理 正如[PEP 479](https://www.python.org/dev/peps/pep-0479/)中概述，开发了一段时间的一处变更旨在让人们更容易调试Python生成器引发的StopIteration异常。以前，生成器遇到另一个问题时很容易引发StopIteration，而不是由于它用完了需要迭代的东西。这带来了一整批很难追踪的代码缺陷。 在Python 3.7中，生成器引发StopIteration异常后，StopIteration异常将被转换成RuntimeError异常，那样它不会悄悄一路影响应用程序的堆栈框架。这意味着如何处理生成器的行为方面不太敏锐的一些程序会在Python 3.7中抛出RuntimeError。而在Python 3.6中，这种行为生成一个弃用警告；在Python 3.7中，它生成一个完整的错误。 一个简易的方法是使用try/except代码段，在StopIteration传播到生成器的外面捕获它。更好的解决方案是重新考虑如何构建生成器――比如说，使用return语句来终止生成器，而不是手动引发StopIteration。想进一步了解如何在现有代码中补救这个问题，如何在新代码中防范该问题，请参阅[PEP 469](https://www.python.org/dev/peps/pep-0479/)。 ### Python开发模式 Python解释器新的命令行开关：-X让开发人员可以为解释器设置许多低级选项。在Python 3.7中，选项-X dev启用“开发模式”，这种运行时检查机制通常对性能有重大影响，但在调试过程中对开发人员很有用。 -X dev激活的选项包括： * asyncio模块的调试模式。这为异步操作提供了更详细的日志记录和异常处理，而异常操作可能很难调试或推理。 * 面向内存分配器的调试钩子。这对于编写CPython扩展件的那些人很有用。它能够实现更明确的运行时检查，了解CPython如何在内部分配内存和释放内存。 * 启用faulthandler模块，那样发生崩溃后，traceback始终转储出去。 ### 具有纳秒分辨率的Python时间函数 Python 3.7中一类新的时间函数返回纳秒精度的时间值。尽管Python是一种解释型语言，但是Python的核心开发人员维克多•斯廷纳（Victor Stinner）主张报告纳秒精度的时间。最主要的原因是，在处理转换其他程序（比如数据库）记录的时间值时，可以避免丢失精度。 新的时间函数使用后缀_ns。比如说，time.process_time()的纳秒版本是time.process_time_ns()。请注意，并非所有的时间函数都有对应的纳秒版本，因为其中一些时间函数并不得益于此。 ### Python UTF-8模式 Python一直支持UTF-8，以便轻松处理字符串和文本。但是周围环境中的语言环境（locale）有时仍是ASCII，而不是UTF-8，检测语言环境的机制并不总是很可靠。 Python 3.7添加了所谓的“UTF-8模式”，可通过-X命令行开关启用该模式，该模式假设UTF-8是环境提供的语言环境。在POSIX语言环境中，UTF-8模式默认情况下已被启用，但在其他位置默认情况下被禁用，以免破坏向后兼容。值得试一试在默认情况下开启UTF-8模式，但不应该在生产环境下启用它，除非你确信Python与周围环境的所有交互都使用UTF-8。 ### 内置breakpoint()函数 Python随带内置的调试器，不过它也可以连入到第三方调试工具，只要它们能与Python的内部调试API进行对话。不过，Python到目前为止缺少一种从Python应用程序里面以编程方式触发调试器的标准化方法。 Python 3.7添加了breakpoint()，这个内置函数使得函数被调用时，让执行切换到调试器。相应的调试器不一定是Python自己的pdb，可以是之前被设为首选调试器的任何调试器。以前，调试器不得不手动设置，然后调用，因而使代码更冗长。而有了breakpoint()，只需一个命令即可调用调试器，并且让设置调试器和调用调试器泾渭分明。 ### 其他新的Python 3.7功能 Python 3.7有另外的众多变更。下面是你在使用最新版本的Python时可能会遇到的其他一些功能： 面向线程本地存储支持的C-API [PEP 539](https://www.python.org/dev/peps/pep-0539/)中描述，线程特定存储（TSS）API取代了老式的线程本地存储（TLS）API。如果谁定制CPython或编写使用解释器的内部API的CPython扩展件，就要明白这一点。 模块属性访问定制 你在Python程序中创建模块时，现在可以针对该模块的实例定制属性访问的行为。为此，只需要在模块里面创建一个__getattr__方法，就跟为一个类创建方法那样。这样一来，就可以对诸如请求模块里面不存在的函数或方法之类的操作进行拦截、标记或代理。 Python importlib资源 importlib模块现在可用来读取“资源”，即随Python应用程序一并交付的二进制工件，比如数据文件。这样一来，开发人员可以通过importlib的抽象来访问那些文件，所以它们存储在系统上某个地方的.zip文件中还是存储在目录中并不重要。 底层优化 现在许多单独的操作更快速了： 由于新的操作码，方法调用起来最多快20%。（除非你在编写直接处理Python操作码的代码，否则不需要担心由此带来的影响。） 正则表达式中不区分大小写的匹配速度更快了，有时要快20倍。 源代码中的一些常量现在可以更高效地优化。 ### 更多特性 [Python Feature](https://github.com/leisurelicht/wtfpython-cn) ================================================ FILE: docs/python/str_joint.md ================================================ # Python 拼接字符串的 7 种方式 ## 1、来自C语言的%方式 ```python print('%s %s' % ('Hello', 'world')) >>> Hello world ``` %号格式化字符串的方式继承自古老的C语言，这在很多编程语言都有类似的实现。上例的%s是一个占位符，它仅代表一段字符串，并不是拼接的实际内容。实际的拼接内容在一个单独的%号后面，放在一个元组里。 类似的占位符还有：%d（代表一个整数）、%f（代表一个浮点数）、%x（代表一个16进制数），等等。%占位符既是这种拼接方式的特点，同时也是其限制，因为每种占位符都有特定意义，实际使用起来太麻烦了。 ## 2、format()拼接方式 ```python # 简洁版 s1 = 'Hello {}! My name is {}.'.format('World', 'Python猫') print(s1) >>>Hello World! My name is Python猫. # 对号入座版 s2 = 'Hello {0}! My name is {1}.'.format('World', 'Python猫') s3 = 'Hello {name1}! My name is {name2}.'.format(name1='World', name2='Python猫') print(s2) >>>Hello World! My name is Python猫. print(s3) >>>Hello World! My name is Python猫. ``` 这种方式使用花括号{}做占位符，在format方法中再转入实际的拼接值。容易看出，它实际上是对%号拼接方式的改进。这种方式在Python2.6中开始引入。 上例中，简洁版的花括号中无内容，缺点是容易弄错次序。对号入座版主要有两种，一种传入序列号，一种则使用key-value的方式。实战中，我们更推荐后一种，既不会数错次序，又更直观可读。 ## 3、() 类似元组方式 ```python s_tuple = ('Hello', ' ', 'world') s_like_tuple = ('Hello' ' ' 'world') print(s_tuple) >>>('Hello', ' ', 'world') print(s_like_tuple) >>>Hello world type(s_like_tuple) >>>str ``` 注意，上例中s_like_tuple并不是一个元组，因为元素间没有逗号分隔符，这些元素间可以用空格间隔，也可以不要空格。使用type()查看，发现它就是一个str类型。我没查到这是啥原因，猜测或许()括号中的内容是被Python优化处理了。 这种方式看起来很快捷，但是，括号()内要求元素是真实字符串，不能混用变量，所以不够灵活。 ```python # 多元素时，不支持有变量 str_1 = 'Hello' str_2 = (str_1 'world') >>> SyntaxError: invalid syntax str_3 = (str_1 str_1) >>> SyntaxError: invalid syntax # 但是下面写法不会报错 str_4 = (str_1) ``` 说实话，我不喜欢这种实现方式。浓浓的一股被面向对象思想毒害的臭味。 就不多说了。 5、常用的+号方式 ```python str_1 = 'Hello world！ ' str_2 = 'My name is Python猫.' print(str_1 + str_2) >>>Hello world！ My name is Python猫. print(str_1) >>>Hello world！ ``` 这种方式最常用、直观、易懂，是入门级的实现方式。但是，它也存在两处让人容易犯错的地方。 首先，新入门编程的同学容易犯错，他们不知道字符串是不可变类型，新的字符串会独占一块新的内存，而原来的字符串保持不变。上例中，拼接前有两段字符串，拼接后实际有三段字符串。 其次，一些有经验的老程序员也容易犯错，他们以为当拼接次数不超过3时，使用+号连接符就会比其它方式快（ps：不少Python教程都是如此建议），但这没有任何合理根据。 事实上，在拼接短的字面值时，由于CPython中的 常数折叠 （constant folding）功能，这些字面值会被转换成更短的形式，例如'a'+'b'+'c' 被转换成'abc'，'hello'+'world'也会被转换成'hello world'。这种转换是在编译期完成的，而到了运行期时就不会再发生任何拼接操作，因此会加快整体计算的速度。 常数折叠优化有一个限度，它要求拼接结果的长度不超过20。所以，当拼接的最终字符串长度不超过20时，+号操作符的方式，会比后面提到的join等方式快得多，这与+号的使用次数无关。 ## 6、join()拼接方式 ```python str_list = ['Hello', 'world'] str_join1 = ' '.join(str_list) str_join2 = '-'.join(str_list) print(str_join1) >>>Hello world print(str_join2) >>>Hello-world ``` str对象自带的join()方法，接受一个序列参数，可以实现拼接。拼接时，元素若不是字符串，需要先转换一下。可以看出，这种方法比较适用于连接序列对象中（例如列表）的元素，并设置统一的间隔符。 当拼接长度超过20时，这种方式基本上是首选。不过，它的缺点就是，不适合进行零散片段的、不处于序列集合的元素拼接。 ## 7、f-string方式 ```python name = 'world' myname = 'python_cat' words = f'Hello {name}. My name is {myname}.' print(words) >>> Hello world. My name is python_cat. ``` f-string方式出自PEP 498（Literal String Interpolation，字面字符串插值），从Python3.6版本引入。其特点是在字符串前加 f 标识，字符串中间则用花括号{}包裹其它字符串变量。 这种方式在可读性上秒杀format()方式，处理长字符串的拼接时，速度与join()方法相当。 尽管如此，这种方式与其它某些编程语言相比，还是欠优雅，因为它引入了一个 f 标识。而其它某些程序语言可以更简练，比如shell： ```shell name="world" myname="python_cat" words="Hello ${name}. My name is ${myname}." echo $words >>>Hello world. My name is python_cat. ``` 总结一下，我们前面说的“字符串拼接”，其实是从结果上理解。若从实现原理上划分的话，我们可以将这些方法划分出三种类型： 格式化类：%、format()、template 拼接类：+、()、join() 插值类：f-string 当要处理字符串列表等序列结构时，采用join()方式；拼接长度不超过20时，选用+号操作符方式；长度超过20的情况，高版本选用f-string，低版本时看情况使用format()或join()方式。 ================================================ FILE: docs/python/syntax_rule.md ================================================ # Python 语法技巧 Python 开发中有哪些高级技巧？这是知乎上一个问题，我总结了一些常见的技巧在这里，可能谈不上多高级，但掌握这些至少可以让你的代码看起来 Pythonic 一点。如果你还在按照类C语言的那套风格来写的话，在 code review 恐怕会要被吐槽了。 ## 列表推导式 ```python >>> chars = [ c for c in 'python' ] >>> chars ['p', 'y', 't', 'h', 'o', 'n'] ``` ## 字典推导式 ```python >>> dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5} >>> double_dict1 = {k:v*2 for (k,v) in dict1.items()} >>> double_dict1 {'a': 2, 'b': 4, 'c': 6, 'd': 8, 'e': 10} ``` ## 集合推导式 ```python >>> set1 = {1,2,3,4} >>> double_set = {i*2 for i in set1} >>> double_set {8, 2, 4, 6} ``` ## 合并字典 ```python >>> x = {'a':1,'b':2} >>> y = {'c':3, 'd':4} >>> z = {**x, **y} >>> z {'a': 1, 'b': 2, 'c': 3, 'd': 4} ``` ## 复制列表 ```python >>> nums = [1,2,3] >>> nums[::] [1, 2, 3] >>> copy_nums = nums[::] >>> copy_nums [1, 2, 3] ``` ## 反转列表 ```python >>> reverse_nums = nums[::-1] >>> reverse_nums [3, 2, 1] ``` PACKING / UNPACKING ## 变量交换 ```python >>> a,b = 1, 2 >>> a ,b = b,a >>> a 2 >>> b 1 ``` ## 高级拆包 ```python >>> a, *b = 1,2,3 >>> a 1 >>> b [2, 3] ``` 或者 ```python >>> a, *b, c = 1,2,3,4,5 >>> a 1 >>> b [2, 3, 4] >>> c 5 ``` ## 函数返回多个值（其实是自动packing成元组）然后unpacking赋值给4个变量 ```python >>> def f(): ... return 1, 2, 3, 4 ... >>> a, b, c, d = f() >>> a 1 >>> d 4 ``` ## 列表合并成字符串 ```python >>> " ".join(["I", "Love", "Python"]) 'I Love Python' ``` ## 链式比较 ```python >>> if a > 2 and a < 5: ... pass ... >>> if 2>> 0 0 1 1 2 2 ``` ## in 代替 or ```python >>> if x == 1 or x == 2 or x == 3: ... pass ... >>> if x in (1,2,3): ... pass ``` ## 字典代替多个if else ```python def fun(x): if x == 'a': return 1 elif x == 'b': return 2 else: return None def fun(x): return {"a": 1, "b": 2}.get(x) ``` ## 有下标索引的枚举 ```python >>> for i, e in enumerate(["a","b","c"]): ... print(i, e) ... 0 a 1 b 2 c ``` ## 生成器 注意区分列表推导式，生成器效率更高 ```python >>> g = (i**2 for i in range(5)) >>> g at 0x10881e518> >>> for i in g: ... print(i) ... 0 1 4 9 16 ``` ## 默认字典 defaultdict ```python >>> d = dict() >>> d['nums'] KeyError: 'nums' >>> >>> from collections import defaultdict >>> d = defaultdict(list) >>> d["nums"] [] ``` ## 字符串格式化 ```python >>> lang = 'python' >>> f'{lang} is most popular language in the world' 'python is most popular language in the world' ``` ## 列表中出现次数最多的元素 ```python >>> nums = [1,2,3,3] >>> max(set(nums), key=nums.count) 3 或者 from collections import Counter >>> Counter(nums).most_common()[0][0] 3 ``` ## 读写文件 ```python >>> with open("test.txt", "w") as f: ... f.writelines("hello") ``` ## 判断对象类型，可指定多个类型 ```python >>> isinstance(a, (int, str)) True ``` ## 类似的还有字符串的 startswith，endswith ```python >>> "http://foofish.net".startswith(('http','https')) True >>> "https://foofish.net".startswith(('http','https')) True ``` ## __str__ 与 __repr__ 区别 ```python >>> str(datetime.now()) '2018-11-20 00:31:54.839605' >>> repr(datetime.now()) 'datetime.datetime(2018, 11, 20, 0, 32, 0, 579521)' ``` 前者对人友好，可读性更强，后者对计算机友好，支持 obj == eval(repr(obj)) ## 使用装饰器 ```python def makebold(f): return lambda: "" + f() + "" def makeitalic(f): return lambda: "" + f() + "" @makebold @makeitalic def say(): return "Hello" >>> say() Hello ``` ## 不使用装饰器，可读性非常差 ```python def say(): return "Hello" >>> makebold(makeitalic(say))() Hello ``` ================================================ FILE: docs/sql/data_split.md ================================================ # 数据库之互联网常用分库分表方案 ## 一、数据库瓶颈 不管是IO瓶颈，还是CPU瓶颈，最终都会导致数据库的活跃连接数增加，进而逼近甚至达到数据库可承载活跃连接数的阈值。在业务Service来看就是，可用数据库连接少甚至无连接可用。接下来就可以想象了吧（并发量、吞吐量、崩溃）。 ### 1、IO瓶颈、 第一种：磁盘读IO瓶颈，热点数据太多，数据库缓存放不下，每次查询时会产生大量的IO，降低查询速度 -> 分库和垂直分表。 第二种：网络IO瓶颈，请求的数据太多，网络带宽不够 -> 分库。 ### 2、CPU瓶颈 第一种：SQL问题，如SQL中包含join，group by，order by，非索引字段条件查询等，增加CPU运算的操作 -> SQL优化，建立合适的索引，在业务Service层进行业务计算。 第二种：单表数据量太大，查询时扫描的行太多，SQL效率低，CPU率先出现瓶颈 -> 水平分表。 ## 二、分库分表 ### 1、水平分库 ![](../img/data_split/data_split.png) 1. 概念：以字段为依据，按照一定策略（hash、range等），将一个库中的数据拆分到多个库中。 2. 结果： * 每个库的结构都一样； * 每个库的数据都不一样，没有交集； * 所有库的并集是全量数据； 3. 场景：系统绝对并发量上来了，分表难以根本上解决问题，并且还没有明显的业务归属来垂直分库。 4. 分析：库多了，io和cpu的压力自然可以成倍缓解。 ###2、水平分表 ![](../img/data_split/data_split1.png) 1. 概念：以字段为依据，按照一定策略（hash、range等），将一个表中的数据拆分到多个表中。 2. 结果： * 每个表的结构都一样； * 每个表的数据都不一样，没有交集； * 所有表的并集是全量数据； 3. 场景：系统绝对并发量并没有上来，只是单表的数据量太多，影响了SQL效率，加重了CPU负担，以至于成为瓶颈。 4. 分析：表的数据量少了，单次SQL执行效率高，自然减轻了CPU的负担。 ### 3、垂直分库 ![](../img/data_split/data_split2.png) 1. 概念：以表为依据，按照业务归属不同，将不同的表拆分到不同的库中。 2. 结果： * 每个库的结构都不一样； * 每个库的数据也不一样，没有交集； * 所有库的并集是全量数据； 3. 场景：系统绝对并发量上来了，并且可以抽象出单独的业务模块。 4. 分析：到这一步，基本上就可以服务化了。例如，随着业务的发展一些公用的配置表、字典表等越来越多，这时可以将这些表拆到单独的库中，甚至可以服务化。再有，随着业务的发展孵化出了一套业务模式，这时可以将相关的表拆到单独的库中，甚至可以服务化。 ### 4、垂直分表 ![](../img/data_split/data_split3.png) 1. 概念：以字段为依据，按照字段的活跃性，将表中字段拆到不同的表（主表和扩展表）中。 2. 结果： * 每个表的结构都不一样； * 每个表的数据也不一样，一般来说，每个表的字段至少有一列交集，一般是主键，用于关联数据； * 所有表的并集是全量数据； 3. 场景：系统绝对并发量并没有上来，表的记录并不多，但是字段多，并且热点数据和非热点数据在一起，单行数据所需的存储空间较大。以至于数据库缓存的数据行减少，查询时会去读磁盘数据产生大量的随机读IO，产生IO瓶颈。 4. 分析：可以用列表页和详情页来帮助理解。垂直分表的拆分原则是将热点数据（可能会冗余经常一起查询的数据）放在一起作为主表，非热点数据放在一起作为扩展表。这样更多的热点数据就能被缓存下来，进而减少了随机读IO。拆了之后，要想获得全部数据就需要关联两个表来取数据。但记住，千万别用join，因为join不仅会增加CPU负担并且会讲两个表耦合在一起（必须在一个数据库实例上）。关联数据，应该在业务Service层做文章，分别获取主表和扩展表数据然后用关联字段关联得到全部数据。 ## 三、分库分表工具 1. sharding-sphere：jar，前身是sharding-jdbc； 2. TDDL：jar，Taobao Distribute Data Layer； 4. Mycat：中间件。 注：工具的利弊，请自行调研，官网和社区优先。 ## 四、分库分表步骤 根据容量（当前容量和增长量）评估分库或分表个数 -> 选key（均匀）-> 分表规则（hash或range等）-> 执行（一般双写）-> 扩容问题（尽量减少数据的移动）。 ## 五、分库分表问题 ### 1、非partition key的查询问题（水平分库分表，拆分策略为常用的hash法） 1. 端上除了partition key只有一个非partition key作为条件查询 * 映射法 ![](../img/data_split/data_split4.png) * 基因法 ![](../img/data_split/data_split5.png) 注：写入时，基因法生成user_id，如图。关于xbit基因，例如要分8张表，23=8，故x取3，即3bit基因。根据user_id查询时可直接取模路由到对应的分库或分表。根据user_name查询时，先通过user_name_code生成函数生成user_name_code再对其取模路由到对应的分库或分表。id生成常用snowflake算法。 2. 端上除了partition key不止一个非partition key作为条件查询 * 映射法 ![](../img/data_split/data_split6.png) * 冗余法 ![](../img/data_split/data_split7.png) 注：按照order_id或buyer_id查询时路由到db_o_buyer库中，按照seller_id查询时路由到db_o_seller库中。感觉有点本末倒置！有其他好的办法吗？改变技术栈呢？ 3. 后台除了partition key还有各种非partition key组合条件查询 * NoSQL法 ![](../img/data_split/data_split8.png) * 冗余法 ![](../img/data_split/data_split9.png) ### 2、非partition key跨库跨表分页查询问题（水平分库分表，拆分策略为常用的hash法） 注：用NoSQL法解决（ES等）。 ### 3、扩容问题（水平分库分表，拆分策略为常用的hash法） 1. 水平扩容库（升级从库法） ![](../img/data_split/data_split10.png) 注：扩容是成倍的。 2. 水平扩容表（双写迁移法） ![](../img/data_split/data_split11.png) 第一步：（同步双写）应用配置双写，部署； 第二步：（同步双写）将老库中的老数据复制到新库中； 第三步：（同步双写）以老库为准校对新库中的老数据； 第四步：（同步双写）应用去掉双写，部署； 注：双写是通用方案 ## 六、分库分表总结 1. 分库分表，首先得知道瓶颈在哪里，然后才能合理地拆分（分库还是分表？水平还是垂直？分几个？）。且不可为了分库分表而拆分。 2. 选key很重要，既要考虑到拆分均匀，也要考虑到非partition key的查询。 3. 只要能满足需求，拆分规则越简单越好。 ================================================ FILE: docs/sql/mybatis.md ================================================ # Mybatis使用心德 ## 什么是Mybatis？ 1. Mybatis是一个半ORM（对象关系映射）框架，它内部封装了JDBC，开发时只需要关注SQL语句本身，不需要花费精力去处理加载驱动、创建连接、创建statement等繁杂的过程。程序员直接编写原生态sql，可以严格控制sql执行性能，灵活度高。 2. MyBatis 可以使用 XML 或注解来配置和映射原生信息，将 POJO映射成数据库中的记录，避免了几乎所有的 JDBC 代码和手动设置参数以及获取结果集。 3. 通过xml 文件或注解的方式将要执行的各种 statement 配置起来，并通过java对象和 statement中sql的动态参数进行映射生成最终执行的sql语句，最后由mybatis框架执行sql并将结果映射为java对象并返回。（从执行sql到返回result的过程）。 ## Mybaits的优点： 1. 基于SQL语句编程，相当灵活，不会对应用程序或者数据库的现有设计造成任何影响，SQL写在XML里，解除sql与程序代码的耦合，便于统一管理；提供XML标签，支持编写动态SQL语句，并可重用。 2. 与JDBC相比，减少了50%以上的代码量，消除了JDBC大量冗余的代码，不需要手动开关连接； 3. 很好的与各种数据库兼容（因为MyBatis使用JDBC来连接数据库，所以只要JDBC支持的数据库MyBatis都支持）。 4. 能够与Spring很好的集成； 5. 提供映射标签，支持对象与数据库的ORM字段关系映射；提供对象关系映射标签，支持对象关系组件维护。 ## MyBatis框架的缺点： 1. SQL语句的编写工作量较大，尤其当字段多、关联表多时，对开发人员编写SQL语句的功底有一定要求。 2. SQL语句依赖于数据库，导致数据库移植性差，不能随意更换数据库。 ## MyBatis框架适用场合： 1. MyBatis专注于SQL本身，是一个足够灵活的DAO层解决方案。 2. 对性能的要求很高，或者需求变化较多的项目，如互联网项目，MyBatis将是不错的选择。 ## MyBatis与Hibernate有哪些不同？ 1. Mybatis和hibernate不同，它不完全是一个ORM框架，因为MyBatis需要程序员自己编写Sql语句。 2. Mybatis直接编写原生态sql，可以严格控制sql执行性能，灵活度高，非常适合对关系数据模型要求不高的软件开发，因为这类软件需求变化频繁，一但需求变化要求迅速输出成果。但是灵活的前提是mybatis无法做到数据库无关性，如果需要实现支持多种数据库的软件，则需要自定义多套sql映射文件，工作量大。 3. Hibernate对象/关系映射能力强，数据库无关性好，对于关系模型要求高的软件，如果用hibernate开发可以节省很多代码，提高效率。 ## #{}和${}的区别是什么？ 1. #{}是预编译处理，${}是字符串替换。 2. Mybatis在处理#{}时，会将sql中的#{}替换为?号，调用PreparedStatement的set方法来赋值； 3. Mybatis在处理${}时，就是把${}替换成变量的值。 4. 使用#{}可以有效的防止SQL注入，提高系统安全性。 ## 当实体类中的属性名和表中的字段名不一样 ，怎么办 ？ 第1种：通过在查询的sql语句中定义字段名的别名，让字段名的别名和实体类的属性名一致。 ``` xml ``` 第2种：通过 来映射字段名和实体类属性名的一一对应的关系。 ``` xml ``` ## 模糊查询like语句该怎么写? 第1种：在Java代码中添加sql通配符。 ``` xml string wildcardname = “%smi%”; list names = mapper.selectlike(wildcardname); ``` 第2种：在sql语句中拼接通配符，会引起sql注入 ``` xml string wildcardname = “smi”; list names = mapper.selectlike(wildcardname); ``` ## 通常一个Xml映射文件，都会写一个Dao接口与之对应，请问，这个Dao接口的工作原理是什么？Dao接口里的方法，参数不同时，方法能重载吗？ Dao接口即Mapper接口。接口的全限名，就是映射文件中的namespace的值；接口的方法名，就是映射文件中Mapper的Statement的id值；接口方法内的参数，就是传递给sql的参数。 Mapper接口是没有实现类的，当调用接口方法时，接口全限名+方法名拼接字符串作为key值，可唯一定位一个MapperStatement。在Mybatis中，每一个 ``` select * fromuser_user_t whereuser_name = #{0} anduser_area=#{1} ``` 2. 第二种：使用 @param 注解: ``` java public interface usermapper { user selectuser(@param(“username”) string username,@param(“hashedpassword”) string hashedpassword); } ``` 然后,就可以在xml像下面这样使用(推荐封装为一个map,作为单个参数传递给mapper): ``` xml ``` 3. 第三种：多个参数封装成map ``` java try { //映射文件的命名空间.SQL片段的ID，就可以调用对应的映射文件中的SQL //由于我们的参数超过了两个，而方法中只有一个Object参数收集，因此我们使用Map集合来装载我们的参数 Map < String, Object > map = new HashMap(); map.put("start", start); map.put("end", end); return sqlSession.selectList("StudentID.pagination", map); } catch (Exception e) { e.printStackTrace(); sqlSession.rollback(); throw e; } finally { MybatisUtil.closeSqlSession(); ``` ## Mybatis动态sql有什么用？执行原理？有哪些动态sql？ Mybatis动态sql可以在Xml映射文件内，以标签的形式编写动态sql，执行原理是根据表达式的值 完成逻辑判断并动态拼接sql的功能。 Mybatis提供了9种动态sql标签： trim|where|set|foreach|if|choose|when|otherwise|bind。 ## Xml映射文件中，除了常见的select|insert|updae|delete标签之外，还有哪些标签？ ```、、、、```，加上动态sql的9个标签，其中 ``````为sql片段标签，通过 ``````标签引入sql片段， ``````为不支持自增的主键生成策略标签。 ## Mybatis的Xml映射文件中，不同的Xml映射文件，id是否可以重复？ 不同的Xml映射文件，如果配置了namespace，那么id可以重复；如果没有配置namespace，那么id不能重复； 原因就是namespace+id是作为Map 的key使用的，如果没有namespace，就剩下id，那么，id重复会导致数据互相覆盖。有了namespace，自然id就可以重复，namespace不同，namespace+id自然也就不同。 ## 为什么说Mybatis是半自动ORM映射工具？它与全自动的区别在哪里？ Hibernate属于全自动ORM映射工具，使用Hibernate查询关联对象或者关联集合对象时，可以根据对象关系模型直接获取，所以它是全自动的。而Mybatis在查询关联对象或关联集合对象时，需要手动编写sql来完成，所以，称之为半自动ORM映射工具。 ## 一对一、一对多的关联查询 ？ ``` xml ``` ## MyBatis实现一对一有几种方式?具体怎么操作的？ 有联合查询和嵌套查询,联合查询是几个表联合查询,只查询一次, 通过在resultMap里面配置association节点配置一对一的类就可以完成； 嵌套查询是先查一个表，根据这个表里面的结果的 外键id，去再另外一个表里面查询数据,也是通过association配置，但另外一个表的查询通过select属性配置。 ## MyBatis实现一对多有几种方式,怎么操作的？ 有联合查询和嵌套查询。联合查询是几个表联合查询,只查询一次,通过在resultMap里面的collection节点配置一对多的类就可以完成；嵌套查询是先查一个表,根据这个表里面的 结果的外键id,去再另外一个表里面查询数据,也是通过配置collection,但另外一个表的查询通过select节点配置。 ## Mybatis是否支持延迟加载？如果支持，它的实现原理是什么？ Mybatis仅支持association关联对象和collection关联集合对象的延迟加载，association指的就是一对一，collection指的就是一对多查询。在Mybatis配置文件中，可以配置是否启用延迟加载lazyLoadingEnabled=true|false。 它的原理是，使用CGLIB创建目标对象的代理对象，当调用目标方法时，进入拦截器方法，比如调用a.getB().getName()，拦截器invoke()方法发现a.getB()是null值，那么就会单独发送事先保存好的查询关联B对象的sql，把B查询上来，然后调用a.setB(b)，于是a的对象b属性就有值了，接着完成a.getB().getName()方法的调用。这就是延迟加载的基本原理。 当然了，不光是Mybatis，几乎所有的包括Hibernate，支持延迟加载的原理都是一样的。 ## Mybatis的一级、二级缓存: 1）一级缓存: 基于 PerpetualCache 的 HashMap 本地缓存，其存储作用域为 Session，当 Session flush 或 close 之后，该 Session 中的所有 Cache 就将清空，默认打开一级缓存。 2）二级缓存与一级缓存其机制相同，默认也是采用 PerpetualCache，HashMap 存储，不同在于其存储作用域为 Mapper(Namespace)，并且可自定义存储源，如 Ehcache。默认不打开二级缓存，要开启二级缓存，使用二级缓存属性类需要实现Serializable序列化接口(可用来保存对象的状态),可在它的映射文件中配置 ； 3）对于缓存数据更新机制，当某一个作用域(一级缓存 Session/二级缓存Namespaces)的进行了C/U/D 操作后，默认该作用域下所有 select 中的缓存将被 clear。 ## 什么是MyBatis的接口绑定？有哪些实现方式？ 接口绑定，就是在MyBatis中任意定义接口,然后把接口里面的方法和SQL语句绑定, 我们直接调用接口方法就可以,这样比起原来了SqlSession提供的方法我们可以有更加灵活的选择和设置。 接口绑定有两种实现方式,一种是通过注解绑定，就是在接口的方法上面加上 @Select、@Update等注解，里面包含Sql语句来绑定；另外一种就是通过xml里面写SQL来绑定, 在这种情况下,要指定xml映射文件里面的namespace必须为接口的全路径名。当Sql语句比较简单时候,用注解绑定, 当SQL语句比较复杂时候,用xml绑定,一般用xml绑定的比较多。 ## 使用MyBatis的mapper接口调用时有哪些要求？ 1、Mapper接口方法名和mapper.xml中定义的每个sql的id相同； 2、Mapper接口方法的输入参数类型和mapper.xml中定义的每个sql 的parameterType的类型相同； 3、Mapper接口方法的输出参数类型和mapper.xml中定义的每个sql的resultType的类型相同； 4、Mapper.xml文件中的namespace即是mapper接口的类路径。 ## Mapper编写有哪几种方式？ 接口实现类继承SqlSessionDaoSupport：使用此种方法需要编写mapper接口，mapper接口实现类、mapper.xml文件。 1、在sqlMapConfig.xml中配置mapper.xml的位置 ``` xml ``` 2、定义mapper接口 3、实现类集成SqlSessionDaoSupportmapper方法中可以this.getSqlSession()进行数据增删改查。 4、spring 配置 ``` xml ``` 第二种：使用 org.mybatis.spring.mapper.MapperFactoryBean： 1、在sqlMapConfig.xml中配置mapper.xml的位置，如果mapper.xml和mappre接口的名称相同且在同一个目录，这里可以不用配置 ``` xml ``` 2、定义mapper接口： mapper.xml中的namespace为mapper接口的地址 mapper接口中的方法名和mapper.xml中的定义的statement的id保持一致 Spring中定义 ``` xml ``` 第三种：使用mapper扫描器： 1、mapper.xml文件编写： mapper.xml中的namespace为mapper接口的地址；mapper接口中的方法名和mapper.xml中的定义的statement的id保持一致；如果将mapper.xml和mapper接口的名称保持一致则不用在sqlMapConfig.xml中进行配置。 2、定义mapper接口： 注意mapper.xml的文件名和mapper的接口名称保持一致，且放在同一个目录 3、配置mapper扫描器： ``` xml ``` 4、使用扫描器后从spring容器中获取mapper的实现对象。 ## 简述Mybatis的插件运行原理，以及如何编写一个插件。 Mybatis仅可以编写针对ParameterHandler、ResultSetHandler、StatementHandler、Executor这4种接口的插件，Mybatis使用JDK的动态代理，为需要拦截的接口生成代理对象以实现接口方法拦截功能，每当执行这4种接口对象的方法时，就会进入拦截方法，具体就是InvocationHandler的invoke()方法，当然，只会拦截那些你指定需要拦截的方法。 编写插件：实现Mybatis的Interceptor接口并复写intercept()方法，然后在给插件编写注解，指定要拦截哪一个接口的哪些方法即可，记住，别忘了在配置文件中配置你编写的插件。 ================================================ FILE: docs/sql/mysql_backups.md ================================================ # 数据备份与恢复 ## 一、备份简介 ### 2.1 备份分类 按照不同的思考维度，通常将数据库的备份分为以下几类： **物理备份 与 逻辑备份** + 物理备份：备份的是完整的数据库目录和数据文件。采用该模式会进行大量的 IO 操作，但不含任何逻辑转换，因此备份和恢复速度通常都比较快。 + 逻辑备份：通过数据库结构和内容信息来进行备份。因为要执行逻辑转换，因此其速度较慢，并且在以文本格式保存时，其输出文件的大小大于物理备份。逻辑备份的还原的粒度可以从服务器级别（所有数据库）精确到具体表，但备份不会包括日志文件、配置文件等与数据库无关的内容。 **全量备份 与 增量备份** + 全量备份：备份服务器在给定时间点上的所有数据。 + 增量备份：备份在给定时间跨度内（从一个时间点到另一个时间点）对数据所做的更改。 **在线备份 与 离线备份** + 在线备份：数据库服务在运行状态下进行备份。此时其他客户端依旧可以连接到数据库，但为了保证数据的一致性，在备份期间可能会对数据进行加锁，此时客户端的访问依然会受限。 + 离线备份：在数据库服务停机状态下进行备份。此备份过程简单，但由于无法提供对外服务，通常会对业务造成比较大的影响。 ### 2.2 备份工具 MySQL 支持的备份工具有很多种，这里列出常用的三种： + **mysqldump**：这是 MySQL 自带的备份工具，其采用的备份方式是逻辑备份，支持全库备份、单库备份、单表备份。由于其采用的是逻辑备份，所以生成的备份文件比物理备份的大，且所需恢复时间也比较长。 + **mysqlpump**：这是 MySQL 5.7 之后新增的备份工具，在 mysqldump 的基础上进行了功能的扩展，支持多线程备份，支持对备份文件进行压缩，能够提高备份的速度和降低备份文件所需的储存空间。 + **Xtrabackup**：这是 Percona 公司开发的实时热备工具，能够在不停机的情况下进行快速可靠的热备份，并且备份期间不会间断数据库事务的处理。它支持数据的全备和增备，并且由于其采用的是物理备份的方式，所以恢复速度比较快。 ## 二、mysqldump ### 2.1 常用参数 mysqldump 的基本语法如下： ```shell # 备份数据库或数据库中的指定表 mysqldump [options] db_name [tbl_name ...] # 备份多个指定的数据库 mysqldump [options] --databases db_name ... # 备份当前数据库实例中的所有表 mysqldump [options] --all-databases ``` options 代表可选操作，常用的可选参数如下： + **--host=host_name， -h host_name** 指定服务器地址。 + **--user=user_name， -u user_name** 指定用户名。 + **--password[=password]， -p[password]** 指定密码。通常无需在命令行中明文指定，按照提示输入即可。 + **--default-character-set=charset_name** 导出文本使用的字符集，默认为 utf8。 + **--events， -E** 备份包含数据库中的事件。 + **--ignore-table=db_name.tbl_name** 不需要进行备份的表，必须使用数据库和表名来共同指定。也可以作用于视图。 + **--routines， -R** 备份包含数据库中的存储过程和自定义函数。 + **--triggers** 备份包含数据库中的触发器。 + **--where='where_condition'， -w 'where_condition'** 在对单表进行导出时候，可以指定过滤条件，例如指定用户名 `--where="user='jimf'"` 或用户范围 `-w"userid>1"` 。 + **--lock-all-tables， -x** 锁定所有数据库中的所有表，从而保证备份数据的一致性。此选项自动关闭 `--single-transaction` 和 `--lock-tables`。 + **--lock-tables， -l** 锁定当前数据库中所有表，能够保证当前数据库中表的一致性，但不能保证全局的一致性。 + **--single-transaction** 此选项会将事务隔离模式设置为 REPEATABLE READ 并开启一个事务，从而保证备份数据的一致性。主要用于事务表，如 InnoDB 表。 但是此时仍然不能在备份表上执行 ALTER TABLE， CREATE TABLE， DROP TABLE， RENAME TABLE， TRUNCATE TABLE 等操作，因为 REPEATABLE READ 并不能隔离这些操作。 另外需要注意的是 `--single-transaction` 选项与 `--lock-tables` 选项是互斥的，因为 LOCK TABLES 会导致任何正在挂起的事务被隐式提交。转储大表时，可以将 `--single-transaction` 选项与 `--quick` 选项组合使用 。 + **--quick， -q** 主要用于备份大表。它强制 mysqldump 一次只从服务器检索一行数据，避免一次检索所有行而导致缓存溢出。 + **--flush-logs, -F** 在开始备份前刷新 MySQL 的日志文件。此选项需要 RELOAD 权限。如果此选项与 `--all-databases` 配合使用，则会在每个数据库开始备份前都刷新一次日志。如果配合 `--lock-all-tables`，`--master-data` 或 `--single-transaction` 使用，则只会在锁定所有表或者开启事务时刷新一次。 + **--master-data[=*value*]** 可以通过配置此参数来控制生成的备份文件是否包含 CHANGE MASTER 语句，该语句中包含了当前时间点二进制日志的信息。该选项有两个可选值：1 和 2 ，设置为 1 时 CHANGE MASTER 语句正常生成，设置为 2 时以注释的方式生成。`--master-data` 选项还会自动关闭 `--lock-tables` 选项，而且如果你没有指定 `--single-transaction` 选项，那么它还会启用 `--lock-all-tables` 选项，在这种情况下，会在备份开始时短暂内获取全局读锁。 ### 2.2 全量备份 mysqldump 的全量备份与恢复的操作比较简单，示例如下： ```shell # 备份雇员库 mysqldump -uroot -p --databases employees > employees_bak.sql # 恢复雇员库 mysql -uroot -p < employees_bak.sql ``` 单表备份： ```shell # 备份雇员库中的职位表 mysqldump -uroot -p --single-transaction employees titles > titles_bak.sql # 恢复雇员库中的职位表 mysql> use employees; mysql> source /root/mysqldata/titles_bak.sql; ``` ### 2.3 增量备份 mysqldump 本身并不能直接进行增量备份，需要通过分析二进制日志的方式来完成。具体示例如下： #### 1. 基础全备 1.先执行一次全备作为基础，这里以单表备份为例，需要用到上文提到的 `--master-data` 参数，语句如下： ```shell mysqldump -uroot -p --master-data=2 --flush-logs employees titles > titles_bak.sql ``` 使用 more 命令查看备份文件，此时可以在文件开头看到 CHANGE MASTER 语句，语句中包含了二进制日志的名称和偏移量信息，具体如下： ```sql -- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=155; ``` #### 2. 增量恢复 对表内容进行任意修改，然后通过分析二进制日志文件来生成增量备份的脚本文件，示例如下： ```shell mysqlbinlog --start-position=155 \ --database=employees ${MYSQL_HOME}/data/mysql-bin.000004 > titles_inr_bak_01.sql ``` 需要注意的是，在实际生产环境中，可能在全量备份后与增量备份前的时间间隔里生成了多份二进制文件，此时需要对每一个二进制文件都执行相同的命令： ```shell mysqlbinlog --database=employees ${MYSQL_HOME}/data/mysql-bin.000005 > titles_inr_bak_02.sql mysqlbinlog --database=employees ${MYSQL_HOME}/data/mysql-bin.000006 > titles_inr_bak_03.sql ..... ``` 之后将全备脚本 ( titles_bak.sql )，以及所有的增备脚本 ( inr_01.sql，inr_02.sql .... ) 通过 source 命令导入即可，这样就完成了全量 + 增量的恢复。 ## 三、mysqlpump ### 3.1 功能优势 mysqlpump 在 mysqldump 的基础上进行了扩展增强，其主要的优点如下： - 能够并行处理数据库及其中的对象，从而可以加快备份进程； - 能够更好地控制数据库及数据库对象（表，存储过程，用户帐户等）； - 能够直接对备份文件进行压缩； - 备份时能够显示进度指标（估计值）； - 备份用户时生成的是 CREATE USER 与 GRANT 语句，而不是像 mysqldump 一样备份成数据，可以方便用户按需恢复。 ### 3.2 常用参数 mysqlpump 的使用和 mysqldump 基本一致，这里不再进行赘述。以下主要介绍部分新增的可选项，具体如下： + **--default-parallelism=N** 每个并行处理队列的默认线程数。默认值为 2。 + **--parallel-schemas=[N:]db_list** 用于并行备份多个数据库：db_list 是一个或多个以逗号分隔的数据库名称列表；N 为使用的线程数，如果没有设置，则使用 `--default-parallelism` 参数的值。 + **--users** 将用户信息备份为 CREATE USER 语句和 GRANT 语句 。如果想要只备份用户信息，则可以使用下面的命令： ```shell mysqlpump --exclude-databases=% --users ``` + **--compress-output=algorithm** 默认情况下，mysqlpump 不对备份文件进行压缩。可以使用该选项指定压缩格式，当前支持 LZ4 和 ZLIB 两种格式。需要注意的是压缩后的文件可以占用更少的存储空间，但是却不能直接用于备份恢复，需要先进行解压，具体如下： ```shell # 采用lz4算法进行压缩 mysqlpump --compress-output=LZ4 > dump.lz4 # 恢复前需要先进行解压 lz4_decompress input_file output_file # 采用ZLIB算法进行压缩 mysqlpump --compress-output=ZLIB > dump.zlib zlib_decompress input_file output_file ``` MySQL 发行版自带了上面两个压缩工具，不需要进行额外安装。以上就是 mysqlpump 新增的部分常用参数，完整参数可以参考官方文档：[mysqlpump — A Database Backup Program](https://dev.mysql.com/doc/refman/8.0/en/mysqlpump.html#option_mysqlpump_compress-output) ## 四、Xtrabackup ### 4.1 在线安装 Xtrabackup 可以直接使用 yum 命令进行安装，这里我的 MySQL 为 8.0 ，对应安装的 Xtrabackup 也为 8.0，命令如下： ```shell # 安装Percona yum 源 yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm # 安装 yum install percona-xtrabackup-80 ``` ### 4.2 全量备份 全量备份的具体步骤如下： #### 1. 创建备份 Xtrabackup 全量备份的基本语句如下，可以使用 target-dir 指明备份文件的存储位置，parallel 则是指明操作的并行度： ```shell xtrabackup --backup --user=root --password --parallel=3 --target-dir=/data/backups/ ``` 以上进行的是整个数据库实例的备份，如果需要备份指定数据库，则可以使用 --databases 进行指定。 另外一个容易出现的异常是：Xtrabackup 在进行备份时，默认会去 `/var/lib/mysql/mysql.sock` 文件里获取数据库的 socket 信息，如果你修改了数据库的 socket 配置，则需要使用 --socket 参数进行重新指定，否则会抛出找不到连接的异常。备份完整后需要立即执行的另外一个操作是 prepare （准备备份）。 #### 2. 准备备份 由于备份是将所有物理库表等文件复制到备份目录，而整个过程需要持续一段时间，此时备份的数据中就可能会包含尚未提交的事务或已经提交但尚未同步至数据文件中的事务，最终导致备份结果处于不一致状态。此时需要进行 prepare 操作来回滚未提交的事务及同步已经提交的事务至数据文件，从而使得整体达到一致性状态。命令如下： ```shell xtrabackup --prepare --target-dir=/data/backups/ ``` 需要特别注意的在该阶段不要随意中断 xtrabackup 进程，因为这可能会导致数据文件损坏，备份将无法使用。 #### 3. 恢复备份 由于 xtrabackup 执行的是物理备份，所以想要进行恢复，必须先要停止 MySQL 服务。同时这里我们可以删除 MySQL 的数据目录来模拟数据丢失的情况，之后使用以下命令将备份文件拷贝到 MySQL 的数据目录下： ```shell # 模拟数据异常丢失 rm -rf /usr/app/mysql-8.0.17/data/* # 将备份文件拷贝到 data 目录下 xtrabackup --copy-back --target-dir=/data/backups/ ``` copy-back 命令只需要指定备份文件的位置，不需要指定 MySQL 数据目录的位置，因为 Xtrabackup 会自动从 `/etc/my.cnf` 上获取 MySQL 的相关信息，包括数据目录的位置。如果不需要保留备份文件，可以直接使用 `--move-back` 命令，代表直接将备份文件移动到数据目录下。此时数据目录的所有者通常为执行命令的用户，需要更改为 mysql 用户，命令如下： ```shell chown -R mysql:mysql /usr/app/mysql-8.0.18/data ``` 再次启动即可完成备份恢复。 ### 4.3 增量备份 使用 Xtrabackup 进行增量备份时，每一次增量备份都需要以上一次的备份为基础，之后再将增量备份运用到第一次全备之上，从而完成备份。具体操作如下： #### 1. 创建备份 这里首先创建一个全备作为基础： ```shell xtrabackup --backup --user=root --password=xiujingmysql. --host=172.17.0.4 --port=13306 --datadir=/data/mysql/data --parallel-3 --target-dir=/data/backups ``` 之后修改库中任意数据，然后进行第一次增量备份，此时需要使用 `incremental-basedir` 指定基础目录为全备目录： ```shell xtrabackup --user=root --password --backup --target-dir=/data/backups/inc1 \ --incremental-basedir=/data/backups/base ``` 再修改库中任意数据，然后进行第二次增量备份，此时需要使用 `incremental-basedir` 指定基础目录为上一次增备目录： ```shell xtrabackup --user=root --password --backup --target-dir=/data/backups/inc2 \ --incremental-basedir=/data/backups/inc1 ``` #### 2. 准备备份 准备基础备份： ```shell xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base ``` 将第一次备份作用于全备数据： ```shell xtrabackup --prepare --apply-log-only --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc1 ``` 将第二次备份作用于全备数据： ```shell xtrabackup --prepare --target-dir=/data/backups/base \ --incremental-dir=/data/backups/inc2 ``` 在准备备份时候，除了最后一次增备外，其余的准备命令都需要加上 `--apply-log-only` 选项来阻止事务的回滚，因为备份时未提交的事务可能正在进行，并可能在下一次增量备份中提交，如果不进行阻止，那么增量备份将没有任何意义。 #### 3. 恢复备份 恢复备份和全量备份时相同，只需要最终准备好的全备数据复制到 MySQL 的数据目录下即可： ```shell xtrabackup --copy-back --target-dir=/data/backups/base # 必须修改文件权限，否则无法启动 chown -R mysql:mysql /usr/app/mysql-8.0.17/data ``` 此时增量备份就已经完成。需要说明的是：按照上面的情况，如果第二次备份之后发生了宕机，那么第二次备份后到宕机前的数据依然没法通过 Xtrabackup 进行恢复，此时就只能采用上面介绍的分析二进制日志的恢复方法。由此可以看出，无论是采用何种备份方式，二进制日志都是非常重要的，因此最好对其进行实时备份。 ## 五、二进制日志的备份 想要备份二进制日志文件，可以通过定时执行 cp 或 scp 等命令来实现，也可以通过 mysqlbinlog 自带的功能来实现远程备份，将远程服务器上的二进制日志文件复制到本机，命令如下： ```shell mysqlbinlog --read-from-remote-server --raw --stop-never \ --host=主机名 --port=3306 \ --user=用户名 --password=密码 初始复制时的日志文件名 ``` 需要注意的是这里的用户必须具有 replication slave 权限，因为上述命令本质上是模拟主从复制架构下，从节点通过 IO 线程不断去获取主节点的二进制日志，从而达到备份的目的。 ## 参考资料 + [Chapter 7 Backup and Recovery](https://dev.mysql.com/doc/refman/8.0/en/backup-and-recovery.html) + [mysqldump — A Database Backup Program](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) + [mysqlpump — A Database Backup Program](https://dev.mysql.com/doc/refman/8.0/en/mysqlpump.html) + [Percona XtraBackup - Documentation](https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html) ================================================ FILE: docs/sql/mysql_index.md ================================================ # MySQL索引优化 --- 本文主要讨论MySQL索引的部分知识。将会从MySQL索引基础、索引优化实战和数据库索引背后的数据结构三部分相关内容，下面一一展开。 ## 一、MySQL——索引基础 首先，我们将从索引基础开始介绍一下什么是索引，分析索引的几种类型，并探讨一下如何创建索引以及索引设计的基本原则。 此部分用于测试索引创建的pay_user表的结构如下： ![index](./../img/mysql/mysql-index-1.png) ### 1、什么是索引 “索引（在MySQL中也叫“键key”）是存储引擎快速找到记录的一种数据结构。” ——《高性能MySQL》 我们需要知道索引其实是一种数据结构，其功能是帮助我们快速匹配查找到需要的数据行，是数据库性能优化最常用的工具之一。其作用相当于超市里的导购员、书本里的目录。 ### 2、索引类型 可以使用SHOW INDEX FROM table_name;查看索引详情： ![index](./../img/mysql/mysql-index-2.png) * 主键索引 PRIMARY KEY 它是一种特殊的唯一索引，不允许有空值。一般是在建表的时候同时创建主键索引。注意：一个表只能有一个主键。 * 唯一索引 UNIQUE 唯一索引列的值必须唯一，但允许有空值。如果是组合索引，则列值的组合必须唯一。 可以通过ALTER TABLE table_name ADD UNIQUE [alias] (column);创建唯一索引： ![index](./../img/mysql/mysql-index-3.png) 可以通过ALTER TABLE table_name ADD UNIQUE [alias] (column1,column2);创建唯一组合索引： ![index](./../img/mysql/mysql-index-4.png) * 普通索引 INDEX 这是最基本的索引，它没有任何限制。 可以通过ALTER TABLE table_name ADD INDEX index_name (column);创建普通索引： ![index](./../img/mysql/mysql-index-5.png) * 组合索引 INDEX 即一个索引包含多个列，多用于避免回表查询。 可以通过ALTER TABLE table_name ADD INDEX index_name(column1,column2, column3);创建组合索引： ![index](./../img/mysql/mysql-index-6.png) 全文索引 FULLTEXT 也称全文检索，是目前搜索引擎使用的一种关键技术。 可以通过ALTER TABLE table_name ADD FULLTEXT (column);创建全文索引： ![index](./../img/mysql/mysql-index-7.png) 索引一经创建不能修改，如果要修改索引，只能删除重建。可以使用DROP INDEX index_name ON table_name;删除索引。 ### 3、索引设计的原则 1. 适合索引的列是出现在where子句中的列，或者连接子句中指定的列； 2. 基数较小的类，索引效果较差，没有必要在此列建立索引； 3. 使用短索引，如果对长字符串列进行索引，应该指定一个前缀长度，这样能够节省大量索引空间； 4. 不要过度索引。索引需要额外的磁盘空间，并降低写操作的性能。在修改表内容的时候，索引会进行更新甚至重构，索引列越多，这个时间就会越长。所以只保持需要的索引有利于查询即可。 ## 二、MySQL——索引优化实战 上面我们介绍了索引的基本内容，这部分我们介绍索引优化实战。在介绍索引优化实战之前，首先要介绍两个与索引相关的重要概念，这两个概念对于索引优化至关重要。 此部分用于测试的user表结构： ![index](./../img/mysql/mysql-index-8.png) ### 1、索引相关的重要概念 * 基数 单个列唯一键（distict_keys）的数量叫做基数。 SELECT COUNT(DISTINCT UserName),COUNT(DISTINCT LoginPassWord) FROM pay_user; ![index](./../img/mysql/mysql-index-9.png) pay_user表的总行数是5，UserName列的基数是10，说明LoginPassWord列里面有大量重复值，UserName列的基数等于总行数，说明UserName列没有重复值，相当于主键。 返回数据的比例： pay_user表中共有5条数据： ![index](./../img/mysql/mysql-index-11.png) 基数越大命中索引命中越高，一半取表数据30%一下走索引 * 回表 当对一个列创建索引之后，索引会包含该列的键值及键值对应行所在的rowid。通过索引中记录的rowid访问表中的数据就叫回表。回表次数太多会严重影响SQL性能，如果回表次数太多，就不应该走索引扫描，应该直接走全表扫描。 EXPLAIN命令结果中的Using Index意味着不会回表，通过索引就可以获得主要的数据。Using Where则意味着需要回表取数据。 ### 2、索引优化实战 有些时候虽然数据库有索引，但是并不被优化器选择使用。 我们可以通过SHOW STATUS LIKE 'Handler_read%';查看索引的使用情况： ![index](./../img/mysql/mysql-index-12.png) * Handler_read_key：如果索引正在工作，Handler_read_key的值将很高。 * Handler_read_rnd_next：数据文件中读取下一行的请求数，如果正在进行大量的表扫描，值将较高，则说明索引利用不理想。 #### 索引优化规则： * 如果MySQL估计使用索引比全表扫描还慢，则不会使用索引。 返回数据的比例是重要的指标，比例越低越容易命中索引。记住这个范围值——30%，后面所讲的内容都是建立在返回数据的比例在30%以内的基础上。 * 前导模糊查询不能命中索引。 UserName列创建普通索引： 前导模糊查询不能命中索引： EXPLAIN SELECT * FROM pay_user WHERE UserName LIKE '%s%'; 非前导模糊查询则可以使用索引，可优化为使用非前导模糊查询： EXPLAIN SELECT * FROM pay_user WHERE UserName LIKE 's%'; * 数据类型出现隐式转换的时候不会命中索引，特别是当列类型是字符串，一定要将字符常量值用引号引起来。 EXPLAIN SELECT * FROM pay_user WHERE UserName=1;--未使用索引 EXPLAIN SELECT * FROM user WHERE name='1';--使用索引 * 复合索引的情况下，查询条件不包含索引列最左边部分（不满足最左原则），不会命中符合索引。 注意，最左原则并不是说是查询条件的顺序 * union、in、or都能够命中索引，建议使用in。 查询的CPU消耗：or>in>union * 用or分割开的条件，如果or前的条件中列有索引，而后面的列中没有索引，那么涉及到的索引都不会被用到。 因为or后面的条件列中没有索引，那么后面的查询肯定要走全表扫描，在存在全表扫描的情况下，就没有必要多一次索引扫描增加IO访问。 * 负向条件查询不能使用索引，可以优化为in查询。 负向条件有：!=、<>、not in、not exists、not like等。 * 范围条件查询可以命中索引。范围条件有：<、<=、>、>=、between等。 范围列可以用到索引（联合索引必须是最左前缀），但是范围列后面的列无法用到索引，索引最多用于一个范围列，如果查询条件中有两个范围列则无法全用到索引： 如果是范围查询和等值查询同时存在，优先匹配等值查询列的索引： * 数据库执行计算不会命中索引 计算逻辑应该尽量放到业务层处理，节省数据库的CPU的同时最大限度的命中索引。 * 利用覆盖索引进行查询，避免回表。 被查询的列，数据能从索引中取得，而不用通过行定位符row-locator再到row上获取，即“被查询列要被所建的索引覆盖”，这能够加速查询速度。 当查询其他列时，就需要回表查询，这也是为什么要避免SELECT*的原因之一： * 建立索引的列，不允许为null。 单列索引不存null值，复合索引不存全为null的值，如果列允许为null，可能会得到“不符合预期”的结果集，所以，请使用not null约束以及默认值。 虽然IS NULL可以命中索引，但是NULL本身就不是一种好的数据库设计，应该使用NOT NULL约束以及默认值。 * 更新十分频繁的字段上不宜建立索引：因为更新操作会变更B+树，重建索引。这个过程是十分消耗数据库性能的。 * 区分度不大的字段上不宜建立索引：类似于性别这种区分度不大的字段，建立索引的意义不大。因为不能有效过滤数据，性能和全表扫描相当。另外返回数据的比例在30%以外的情况下，优化器不会选择使用索引。 * 业务上具有唯一特性的字段，即使是多个字段的组合，也必须建成唯一索引。虽然唯一索引会影响insert速度，但是对于查询的速度提升是非常明显的。另外，即使在应用层做了非常完善的校验控制，只要没有唯一索引，在并发的情况下，依然有脏数据产生。 * 多表关联时，要保证关联字段上一定有索引。 * 创建索引时避免以下错误观念：索引越多越好，认为一个查询就需要建一个索引；宁缺勿滥，认为索引会消耗空间、严重拖慢更新和新增速度；抵制唯一索引，认为业务的唯一性一律需要在应用层通过“先查后插”方式解决；过早优化，在不了解系统的情况下就开始优化。 ### 3、总结 对于自己编写的SQL查询语句，要尽量使用EXPLAIN命令分析一下，做一个对SQL性能有追求的程序员。衡量一个程序员是否靠谱，SQL能力是一个重要的指标。作为后端程序员，深以为然。 ## 三、数据库索引背后的数据结构 第一部分开头我们简单提到，索引是存储引擎快速找到记录的一种数据结构。进一步说，在数据库系统里，这种数据结构要满足特定查找算法，即这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。 ![index](./../img/mysql/mysql-index-13.png) ### 1、B-Tree B-Tree是一种平衡的多路查找(又称排序)树，在文件系统中和数据库系统中有所应用，主要用作文件的索引。其中的B就表示平衡(Balance) 。 ![index](./../img/mysql/mysql-index-14.png) ### B-Tree的特性 为了描述B-Tree，首先定义一条数据记录为一个二元组[key, data]，key为记录的键值，对于不同数据记录，key是互不相同的；data为数据记录除key外的数据。那么B-Tree是满足下列条件的数据结构： d为大于1的一个正整数，称为B-Tree的度： ![index](./../img/mysql/mysql-index-15.png) h为一个正整数，称为B-Tree的高度： ![index](./../img/mysql/mysql-index-16.png) key和指针互相间隔，节点两端是指针： ![index](./../img/mysql/mysql-index-17.png) 一个节点中的key从左到右非递减排列： ![index](./../img/mysql/mysql-index-18.png) 所有节点组成树结构。 每个指针要么为null，要么指向另外一个节点；每个非叶子节点由n-1个key和n个指针组成，其中d<=n<=2d： 每个叶子节点最少包含一个key和两个指针，最多包含2d-1个key和2d个指针，叶节点的指针均为null： 所有叶节点具有相同的深度，等于树高h。 如果某个指针在节点node最左边且不为null，则其指向节点的所有key小于key1，其中key1为node的第一个key的值： 如果某个指针在节点node最右边且不为null，则其指向节点的所有key大于keym，其中keym为node的最后一个key的值： 如果某个指针在节点node的左右相邻key分别是keyi和keyi+1且不为null，则其指向节点的所有key小于keyi+1且大于keyi： ### B-Tree查找数据 B-Tree是一个非常有效率的索引数据结构。这主要得益于B-Tree的度可以非常大，高度会变的非常小，只需要二分几次就可以找到数据。例如一个度为d的B-Tree，设其索引N个key，则其树高h的上限为logd((N+1)/2))，检索一个key，其查找节点个数的渐进复杂度为O(logdN)。 在B-Tree中按key检索数据的算法非常直观： * 首先从根节点进行二分查找，如果找到则返回对应节点的data； * 否则对相应区间的指针指向的节点递归进行查找，如果找到则返回对应节点的data； * 如果找不到，则重复上述“对相应区间的指针指向的节点递归进行查找”，直到找到节点或找到null指针，前者查找成功，后者查找失败。 ### 2、B+Tree B+Tree是B-Tree的一种变种。一般来说，B+Tree比B-Tree更适合实现外存储索引结构，具体原因与外存储器原理及计算机存取原理有关，将在以后讨论。 B+Tree的特性: 区别于B-Tree: 每个节点的指针上限为2d而不是2d+1； 内节点不存储data，只存储key；叶子节点不存储指针。 3、带有顺序访问指针的B+Tree 一般在数据库系统或者文件系统中，并不是直接使用B+Tree作为索引数据结构的，而是在B+Tree的基础上做了优化，增加了顺序访问指针，提升了区间查询的性能。 如上图所示，在B+Tree的每个叶子节点增加一个指向相邻叶子节点的指针，就形成了带有顺序访问指针的B+Tree。 例如要查询18到30之间的数据记录，只要先找到18，然后顺着顺序访问指针就可以访问到所有的数据节点。这样就提升了区间查询的性能。数据库的索引全扫描index和索引范围扫描range就是基于此实现的。 ================================================ FILE: docs/sql/mysql_log.md ================================================ # MySQL查询日志 MySQL中的日志包括：错误日志、二进制日志、通用查询日志、慢查询日志等等。这里主要介绍下比较常用的两个功能：通用查询日志和慢查询日志。 1. 通用查询日志：记录建立的客户端连接和执行的语句。 2. 慢查询日志：记录所有执行时间超过longquerytime秒的所有查询或者不使用索引的查询 ## 一、通用查询日志 在学习通用日志查询时，需要知道两个数据库中的常用命令： ```sql show variables like '%general%'; ``` 可以查看，当前的通用日志查询是否开启，如果general_log的值为ON则为开启，为OFF则为关闭（默认情况下是关闭的）。 ```sql show variables like '%log_output%'; ``` 查看当前慢查询日志输出的格式，可以是FILE（存储在数数据库的数据文件中的hostname.log），也可以是TABLE（存储在数据库中的mysql.general_log） 问题：如何开启MySQL通用查询日志，以及如何设置要输出的通用日志输出格式呢？ 开启通用日志查询： ```sql set global general_log=on; ``` 关闭通用日志查询： ```sql set global general_log=off; ``` 设置通用日志输出为表方式： ```sql set global log_output='TABLE'; ``` 设置通用日志输出为文件方式： ```sql set global log_output='FILE'; ``` 设置通用日志输出为表和文件方式： ```sql set global log_output='FILE,TABLE'; ``` （注意：上述命令只对当前生效，当MySQL重启失效，如果要永久生效，需要配置 my.cnf） my.cnf文件的配置如下： ```conf general_log=1 #为1表示开启通用日志查询，值为0表示关闭通用日志查询 log_output=FILE,TABLE #设置通用日志的输出格式为文件和表 ``` ## 二、慢查询日志 MySQL的慢查询日志是MySQL提供的一种日志记录，用来记录在MySQL中响应时间超过阈值的语句，具体指运行时间超过long_query_time值的SQL，则会被记录到慢查询日志中（日志可以写入文件或者数据库表，如果对性能要求高的话，建议写文件）。默认情况下，MySQL数据库是不开启慢查询日志的，long_query_time的默认值为10（即10秒，通常设置为1秒），即运行10秒以上的语句是慢查询语句。 一般来说，慢查询发生在大表（比如：一个表的数据量有几百万），且查询条件的字段没有建立索引，此时，要匹配查询条件的字段会进行全表扫描，耗时查过long_query_time，则为慢查询语句。 问题：如何查看当前慢查询日志的开启情况？ 在MySQL中输入命令： ```sql show variables like '%quer%'; ``` 主要掌握以下的几个参数： 1. slow_query_log 的值为ON为开启慢查询日志，OFF则为关闭慢查询日志。 2. slow_query_log_file 的值是记录的慢查询日志到文件中（注意：默认名为主机名.log，慢查询日志是否写入指定文件中，需要指定慢查询的输出日志格式为文件，相关命令为：show variables like '%log_output%'；去查看输出的格式）。 3. long_query_time 指定了慢查询的阈值，即如果执行语句的时间超过该阈值则为慢查询语句，默认值为10秒。 4. log_queries_not_using_indexes 如果值设置为ON，则会记录所有没有利用索引的查询（注意：如果只是将log_queries_not_using_indexes设置为ON，而将slow_query_log设置为OFF，此时该设置也不会生效，即该设置生效的前提是slow_query_log的值设置为ON），一般在性能调优的时候会暂时开启。 问题：设置MySQL慢查询的输出日志格式为文件还是表，或者两者都有？ 通过命令：show variables like '%log_output%'; 通过log_output的值可以查看到输出的格式，上面的值为TABLE。当然，我们也可以设置输出的格式为文本，或者同时记录文本和数据库表中，设置的命令如下： ```sql #慢查询日志输出到表中（即mysql.slow_log） set globallog_output='TABLE'; #慢查询日志仅输出到文本中(即：slow_query_log_file指定的文件) setglobal log_output='FILE'; #慢查询日志同时输出到文本和表中 setglobal log_output='FILE,TABLE'; ``` 关于慢查询日志的表中的数据个文本中的数据格式分析： ![1](./../img/sql_log1.png) 慢查询的日志记录到hostname.log文件中，格式如下： ![1](./../img/sql_log2.png) 可以看到，不管是表还是文件，都具体记录了：是那条语句导致慢查询（sql_text），该慢查询语句的查询时间（query_time），锁表时间（Lock_time），以及扫描过的行数（rows_examined）等信息。 问题：如何查询当前慢查询的语句的个数？ 在MySQL中有一个变量专门记录当前慢查询语句的个数： 输入命令：show global status like '%slow%'; ![1](./../img/sql_log3.png) （注意：上述所有命令，如果都是通过MySQL的shell将参数设置进去，如果重启MySQL，所有设置好的参数将失效，如果想要永久的生效，需要将配置参数写入my.cnf文件中）。 补充知识点：如何利用MySQL自带的慢查询日志分析工具mysqldumpslow分析日志？ ```shell perlmysqldumpslow –s c –t 10 slow-query.log ``` 具体参数设置如下： -s 表示按何种方式排序，c、t、l、r分别是按照记录次数、时间、查询时间、返回的记录数来排序，ac、at、al、ar，表示相应的倒叙； -t 表示top的意思，后面跟着的数据表示返回前面多少条； -g 后面可以写正则表达式匹配，大小写不敏感。 ![1](./../img/sql_log4.png) 上述中的参数含义如下： Count:414 语句出现了414次； Time=3.51s（1454） 执行最长时间为3.51s，累计总耗费时间1454s； Lock=0.0s（0） 等待锁最长时间为0s，累计等待锁耗费时间为0s； Rows=2194.9（9097604） 发送给客户端最多的行数为2194.9，累计发送给客户端的函数为90976404 （注意：mysqldumpslow脚本是用perl语言写的，具体mysqldumpslow的用法后期再讲） 问题：实际在学习过程中，如何得知设置的慢查询是有效的？ 很简单，我们可以手动产生一条慢查询语句，比如，如果我们的慢查询log_query_time的值设置为1，则我们可以执行如下语句： selectsleep(1); 该条语句即是慢查询语句，之后，便可以在相应的日志输出文件或表中去查看是否有该条语句。 ================================================ FILE: docs/sql/mysql_pxc.md ================================================ # Docker 搭建pxc集群 + haproxy + keepalived 高可用 ## docker基本指令： 1. 更新软件包 ```shell yum -y update ``` 2. 安装Docker虚拟机（centos 7） ```shell yum install -y docker ``` 3. 运行、重启、关闭Docker虚拟机 ```shell service docker start service docker stop ``` 3. 搜索镜像 ```shell docker search 镜像名称 ``` 4. 下载镜像 ```shell docker pull 镜像名称 ``` 5. 查看镜像 ```shell docker images ``` 6. 删除镜像 ```shell docker rmi 镜像名称 ``` 7. 运行容器 ```shell docker run 启动参数 镜像名称 ``` 8. 查看容器列表 ```shell docker ps -a ``` ## 安装PXC集群 1. 安装PXC镜像 ```shell docker pull percona/percona-xtradb-cluster ``` 2. 查看本地镜像 ```shell [root@VM_71_225_centos ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/hello-world latest e38bc07ac18e 2 months ago 1.85 kB docker.io/percona/percona-xtradb-cluster latest f1439de62087 3 months ago 413 MB docker.io/java latest d23bdf5b1b1b 17 months ago 643 MB ``` 3. docker.io/percona/percona-xtradb-cluster 太长，进行改名： ``` shell [root@VM_71_225_centos ~]# docker tag percona/percona-xtradb-cluster pxc [root@VM_71_225_centos ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/percona/percona-xtradb-cluster latest f1439de62087 3 months ago 413 MB pxc latest f1439de62087 3 months ago 413 MB ``` 4. 创建net1网段： ```shell docker network create --subnet=172.18.0.0/16 net1 ``` 5. 创建五个数据卷（pxc无法直接存取宿组机的数据，所以创建五个docker数据卷） ```shell docker volume create v1 docker volume create v2 docker volume create v3 docker volume create v4 docker volume create v5 ``` 6. 查看数据卷位置： ```shell [root@VM_71_225_centos code]# docker inspect v1 [ { "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/v1/_data", "Name": "v1", "Options": {}, "Scope": "local" } ] ``` 7. 创建5节点的PXC集群 创建第1个MySQL节点 ```shell docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=abc123456 -e CLUSTER_NAME=PXC -e XTRABACKUP_PASSWORD=abc123456 -v v1:/var/lib/mysql -v backup:/data --privileged --name=node1 --net=net1 --ip 172.18.0.2 pxc ``` 等待2分钟后，再创建第二个节点，等待第一个节点实例化完毕后，才能开启第二个节点实例，不然会瞬间停止 创建其他节点： 创建第2个MySQL节点 ```shell docker run -d -p 3307:3306 -e MYSQL_ROOT_PASSWORD=abc123456 -e CLUSTER_NAME=PXC -e XTRABACKUP_PASSWORD=abc123456 -e CLUSTER_JOIN=node1 -v v2:/var/lib/mysql -v backup:/data --privileged --name=node2 --net=net1 --ip 172.18.0.3 pxc ``` 创建第3个MySQL节点 ``` shell docker run -d -p 3308:3306 -e MYSQL_ROOT_PASSWORD=abc123456 -e CLUSTER_NAME=PXC -e XTRABACKUP_PASSWORD=abc123456 -e CLUSTER_JOIN=node1 -v v3:/var/lib/mysql --privileged --name=node3 --net=net1 --ip 172.18.0.4 pxc ``` 创建第4个MySQL节点 ``` shell docker run -d -p 3309:3306 -e MYSQL_ROOT_PASSWORD=abc123456 -e CLUSTER_NAME=PXC -e XTRABACKUP_PASSWORD=abc123456 -e CLUSTER_JOIN=node1 -v v4:/var/lib/mysql --privileged --name=node4 --net=net1 --ip 172.18.0.5 pxc ``` 创建第5个MySQL节点 ```shell docker run -d -p 3310:3306 -e MYSQL_ROOT_PASSWORD=abc123456 -e CLUSTER_NAME=PXC -e XTRABACKUP_PASSWORD=abc123456 -e CLUSTER_JOIN=node1 -v v5:/var/lib/mysql -v backup:/data --privileged --name=node5 --net=net1 --ip 172.18.0.6 pxc ``` ## 安装Haproxy进行高可用与负载均衡 1. 在mysql集群数据库进行创建该用户haproxy ```shell create user 'haproxy'@'%' identified by ''; ``` 2. 拉取haproxy ```shell docker pull haproxy ``` 3. 编写Haproxy配置文件 ```shell vi /usr/local/docker/haproxy/haproxy.cfg ``` 配置文件如下： ```conf global #工作目录 chroot /usr/local/etc/haproxy #日志文件，使用rsyslog服务中local5日志设备（/var/log/local5），等级info log 127.0.0.1 local5 info #守护进程运行 daemon ​ defaults log global mode http #日志格式 option httplog #日志中不记录负载均衡的心跳检测记录 option dontlognull #连接超时（毫秒） timeout connect 5000 #客户端超时（毫秒） timeout client 50000 #服务器超时（毫秒） timeout server 50000 ​ #监控界面 listen admin_stats #监控界面的访问的IP和端口 bind 0.0.0.0:8888 #访问协议 mode http #URI相对地址 stats uri /dbs #统计报告格式 stats realm Global\ statistics #登陆帐户信息 stats auth admin:abc123456 #数据库负载均衡 listen proxy-mysql #访问的IP和端口 bind 0.0.0.0:3306 #网络协议 mode tcp #负载均衡算法（轮询算法） #轮询算法：roundrobin #权重算法：static-rr #最少连接算法：leastconn #请求源IP算法：source balance roundrobin #日志格式 option tcplog #在MySQL中创建一个没有权限的haproxy用户，密码为空。Haproxy使用这个账户对MySQL数据库心跳检测 option mysql-check user haproxy server MySQL_1 172.18.0.2:3306 check weight 1 maxconn 2000 server MySQL_2 172.18.0.3:3306 check weight 1 maxconn 2000 server MySQL_3 172.18.0.4:3306 check weight 1 maxconn 2000 server MySQL_4 172.18.0.5:3306 check weight 1 maxconn 2000 server MySQL_5 172.18.0.6:3306 check weight 1 maxconn 2000 #使用keepalive检测死链 option tcpka ``` 4. 创建第1个Haproxy负载均衡服务器 ```shell docker run -it -d -p 4001:8888 -p 4002:3306 -v docker run -it -d -p 4001:8888 -p 4002:3306 -v /usr/local/docker/haproxy:/usr/local/etc/haproxy --name h1 --privileged --net=net1 --ip 172.18.0.7 haproxy ``` 5. 进入h1容器，启动Haproxy ```shell docker exec -it h1 bash haproxy -f /usr/local/etc/haproxy/haproxy.cfg ``` 6. 查看是否启动成功： 访问http://ip:4001/dbs ![pxc](./../img/pxc1.png) ## 安装keepalive实现双击热备 ================================================ FILE: docs/sql/mysql_use.md ================================================ # 埋在MySQL数据库应用中的17个关键问题！ MySQL的使用非常普遍，跟MySQL有关的话题也非常多，如性能优化、高可用性、强一致性、安全、备份、集群、横向扩展、纵向扩展、负载均衡、读写分离等。要想掌握其中的精髓，可得花费不少功力，虽然目前流行的MySQL替代方案有很多，可是从最小成本最容易维护的角度而言，MySQL还是首选。下面从应用场景的角度切入，对MySQL的技术点进行组织，写一份知识图谱，方便进行更深入的学习和总结。 如下图整理，我试着把MySQL的应用场景分为6种，每种场景下需要考虑的重点问题不一样，从而引出不同问题点下需要补齐的知识点，后续继续基于这些知识点进行学习和整理。 ![question](./../img/mysql_q_1.png) ## 一、单Master ![question](./../img/mysql_q_2.png) 单Master的情况是普遍存在的，对于很多个人站点、初创公司、小型内部系统，考虑到成本、更新频率、系统重要性等问题，系统只依赖一个单例数据库提供服务，基本上已经满足需求。这种场景下我觉得重点应该关注的话题有上图所示的四点。 其中最重要的环节是数据备份，如果是交易量非常低，并且具有非常明确的服务时间段特性的话，简单的mysqldump是可以胜任的。但是这是有缺陷的，数据还原之后注定从备份点到还原点之间的数据会丢失。然而在极多数的情况下，备份的工作是没法马虎的，如下列举的几点小细节。 ### 1）冷备： 停机，直接copy物理文件，InnoDB引擎（frm文件、共享表空间文件、独立表空间文件、重做日志文件、my.cnf）。 恢复：把文件copy到对应目录。 ### 2）热备： Ibbackup或者XtraBackup工具，记录重做日志文件检查点的LSN，copy共享表空间文件以及独立表空间文件（不产生任何阻塞），记录copy后重做日志文件检查点的LSN，copy备份是产生的重做日志。 恢复：恢复表空间文件，应用重做日志文件。 ### 3）温备： mysqldump，--single-transaction参数进行事务管理保证数据一致性。备份时不能用DDL语句。 恢复：直接执行文件，mysql –uroot –p <文件名.sql> 二进制半同步复制，主从服务器增量复制 恢复：mysqlbinlog ## 二、一主一从 ![question](./../img/mysql_q_3.png) 考虑一主一从的多数初衷是系统性能和系统高可用性问题，除了单Master场景中的备份工作需要做好以外，还有性能优化、读写分离、负载均衡三项重点工作需要考虑。其中性能优化的内容比较多，也是一块大主题，要从系统的服务指标作为依据采取相应的动作，多数系统要求的是3秒内完成请求，总体换算下来，数据库大概可以有1.5秒的总执行时间，能满足这个性能要求就是合理的优化方案。下学期以这样的优先级来分别整理内容：索引优化 -》 表设计优化 -》数据库配置优化 -》硬件优化。 读写分离和负载均衡的实现相对简单些，我目前维护的系统比较落后，没有做读写分离，因为是一套以报表类功能为主的系统，而负载均衡是依赖php代码来做的，从实际运维效果来看，不大理想，而且负载均衡的代码过分嵌入到业务逻辑代码中，给代码维护带来一定噪音。下学期计划对各种中间件进行实践和性能测试，到时候把一些测试数据分享出来。 ## 三、一主 n 从 ![quesiton](./../img/mysql_q_4.png) 一旦开始考虑一主多从的服务器架构，则证明你的系统对可用性、一致性、性能中一种或者多种的要求比较高。好多系统在开始搭建的时候都会往这个方向看齐，毕竟这样“看起来”系统会健壮很多。不过其实并不能单单依靠MySQL的配置和MySQL自带的中间件来解决可用性、一致性方面的问题。 ## 四、横向集群 ![quesion](./../img/mysql_q_5.png) 系统庞大到需要分库分表，其实是一件可喜可贺的事情，但是切记的是要前面提到性能优化工作做到极致之后才好考虑这些会增加系统复杂度的解决方案。横向集群主要是从业务特性的角度对系统进行切分，最彻底就是切分成了各个子系统，子系统之间通过一些数据同步的方案来把一些核心数据进行共享，以避免跨库调用跨库join。 然后是各种系统接口调用，把大事务拆成小事务，事务之间做好隔离和同步。上图中的三个问题在横向集群的架构体系中应属于很有特色的问题，在实际项目中其实是尽量去避免这些需求的存在的，不过如果确实需要了，也得有解决方案。下学期也将针对这些问题进行逐一整理，并测试一下一些号称支持这些功能的中间件。 ## 五、纵向集群 ![question](./../img/mysql_q_6.png) 横向集群的切分思路最终是切分子系统，而纵向集群最后遇到的最棘手的问题是扩缩容，我运维的一个系统是提前对数据做了256个切片，256切片中0~127切片和128~255切片分别存在两个一主两从的数据库集群中，系统运维了3年多，目前还没有扩容需求。设计初衷应该是考虑得到，假设有一天数据量非常大，可以把256个切片分4大片，分别存储到4个一主两从的集群中，从而实现扩容。 这个思路的确是可取的，只是我们的分库逻辑当前是php代码实现，也有一定程度上影响了业务代码的逻辑，运维起来有点心惊胆战，还是保持业务代码清爽比较好。 下学期将介绍一些实现了库路由功能的中间件的使用，也根据实际情况把想到的一些扩缩容方案实践一遍，敬请期待实操效果的分享。 ## 六、混合模式 与其说这部分内容讨论上面5种场景的混合，不如说这部分内容是做总结。上面的5种场景中，一共列举了17个问题点，这17个问题点基本上都是叠加式的，越往深入的框架去做就越需要考虑齐这17个问题点。17个问题点考虑全了，混合模式下的问题就不成问题了。 ================================================ FILE: docs/sql/mysql_yh.md ================================================ # MySQL 优化指南 ## 慢查询日志 开启撒网模式 开启了MySQL慢查询日志之后，MySQL会自动将执行时间超过指定秒数的SQL统统记录下来，这对于搜罗线上慢SQL有很大的帮助。 ``` sql SHOW VARIABLES LIKE 'slow%' ``` 以我刚安装的mysql5.7为例 查询结果是这样子的： ![sql](./../img/sql1.png "sql_show") 1. slow_launch_time：表示如果建立线程花费了比这个值更长的时间,slow_launch_threads 计数器将增加 2. slow_query_log：是否开启慢查询日志 ON开启，OFF关闭 默认没有开启 3. slow_query_log_file：日志保存路径 ``` sql SHOW VARIABLES LIKE 'long%' ``` ![sql](./../img/sql2.png) long_query_time：达到多少秒的sql就记录日志 客户端可以用set设置变量的方式让慢查询开启，但是个人不推荐，因为真实操作起来会有一些问题，比如说，重启MySQL后就失效了，或者是开启了慢查询，我又去改变量值，它就不生效了。 ## 编辑MySQL的配置文件： ```shell vim /etc/my.cnf ``` 加入如下三行： ``` config slow_query_log=ON slow_query_log_file=/var/lib/mysql/localhost-centos-slow.log long_query_time=3 ``` 我这里设置的是3秒 ## 重启MySQL ```shell systemctl restart mysqld; ``` ## 服务器开一个监控： ```shell tail -f /var/lib/mysql/localhost-centos-slow.log ``` ## 客户端走一条SQL： ```sql SELECT SLEEP(3) ``` 此时发现sql已经被记录到日志里了。（有时候不一定，我看到很多博客讲的是超过指定秒数，但我实验得出的结果是达到指定秒数） ## EXPLAIN 点对点分析 explain是一个神奇的命令，可以查看sql的具体的执行计划。 以一条联查sql为例： ```sql SELECT a.id,a.cn_name,a.role_id,r.name FROM tb_usr_admins a INNER JOIN tb_base_roles r ON r.id=a.role_id WHERE a.cn_name="接单人员" ``` 查询结果是： ![sql](./../img/sql3.png "sql") 加上explain命令来执行： ```sql EXPLAIN SELECT a.id,a.cn_name,a.role_id,r.name FROM tb_usr_admins a INNER JOIN tb_base_roles r ON r.id=a.role_id WHERE a.cn_name="接单人员" ``` 查询结果是： ![sql](./../img/sql4.png "sql") 这就是这条SQL的执行计划，下面来说明一下这个执行计划怎么看 id：代表优先级 id值越大，越先执行，id值相同，从上往下执行。（比如示例的这条sql的执行计划，就是先执行第一行，再执行第二行） ## select_type：表示select类型 取值如下 - simple 简单表 即不使用表连接或者子查询 - primary 包含union或者子查询的主查询 即外层的查询 - union UNION中的第二个或者后面的查询语句 - subquery 一般子查询中的子查询被标记为subquery，也就是位于select列表中的查询 - derived 派生表 该临时表是从子查询派生出来的 - 等等 ## type：表示MySQL在表中查找数据的方式，或者叫访问类型，以下对于type取值的说明 从上往下性能由最差到最好 1. all:全表扫描，MySQL遍历全表来找到匹配的行 2. index：索引全扫描，MySQL遍历挣个索引来查询匹配的行 3. range：索引范围扫描，常见于<、<=、>、>=、between等操作符 4. ref：使用非唯一索引或唯一索引的前缀扫描，返回匹配的单行数据 5. eq_ref：类似ref，区别就在于使用的索引是唯一索引，简单来说，就是多表连接中使用primary key或者unique index作为关联条件。 6. const/system：单表中最多有一个匹配行，查询起来非常迅速，常见于根据primary key或者唯一索引unique index进行的单表查询 7. null：mysql不用访问表或者索引，直接就能够得到查询的结果，例如select 1+2 as result。 ## possible_keys：表示查询时可能使用的索引 ## key：表示实际使用的索引 ## key_len：使用到索引字段的长度 ## rows：扫描数量 ## Extra：执行情况的说明和描述，包含不适合在其他列中显示但是对执行计划非常重要的额外信息，常用取值如下： - Using index：直接访问索引就取到了数据，高性能的表现。 - Using where：直接在主键索引上过滤数据，必带where子句，而且用不上索引 - Using index condition：先条件过滤索引，再查数据， - Using filesort：使用了外部文件排序 只要见到这个 就要优化掉 - Using temporary：创建了临时表来处理查询 只要见到这个 也要尽量优化掉 ## 优化争议无数的count() ## 统计列与统计行？ COUNT()是一个特殊的函数，有两种不同的作用，它可以统计某个列值的数量，也可以统计行数。 在统计列值的时候要求列值是非空的，也就是不统计null。 当我们统计行的时候，常见的是COUNT(*)，这种情况下，通配符*并不会像我们猜想的那样扩展成所有的列，实际上，它会忽略所有的列而直接统计所有的行数 ## 解密MyiSAM的‘快’ 这是一个容易产生误解的事情：MyiSAM的count()函数总是非常快。 不过它是有前提条件的，条件是没有任何where条件的count(*)才非常快，因为此时无须实际的去计算表的行数，mysql可以利用存储引擎的特性直接获得这个值，如果mysql知道某列不可能有null值，那么mysql内部会将count(列)表达式优化为count(*)。 当统计带有where条件的查询，那么mysql的count()和其他存储引擎就没有什么不同了。 ## COUNT(1)、COUNT(*)、COUNT(列) （先提前申明，本人是在innodb库里做的实验。） 1. count(1)和count(*)直接就是统计主键，他们两个的效率是一样的。如果删除主键，他们都走全表扫描。 2. 如果count(列)中的字段是索引的话，count(列)和count(*)一样快，否则count(列)走全表扫描。 ## 优化order by 语句 ## MySQL的排序方式 优化order by语句就不得不了解mysql的排序方式。 1. 第一种通过有序索引返回数据，这种方式的extra显示为Using Index,不需要额外的排序，操作效率较高。 2. 第二种是对返回的数据进行排序，也就是通常看到的Using filesort，filesort是通过相应的排序算法，将数据放在sort_buffer_size系统变量设置的内存排序区中进行排序，如果内存装载不下，它就会将磁盘上的数据进行分块，再对各个数据块进行排序，然后将各个块合并成有序的结果集。 ## filesort的优化 了解了MySQL排序的方式，优化目标就清晰了：尽量减少额外的排序，通过索引直接返回有序数据。where条件和order by使用相同的索引。 1. 创建合适的索引减少filesort的出现。 2. 查询时尽量只使用必要的字段，select 具体字段的名称，而不是select * 选择所有字段，这样可以减少排序区的使用，提高SQL性能。 ## 优化group by 语句 ## 为什么order by后面不能跟group by 事实上，MySQL在所有的group by 后面隐式的加了order by ，也就是说group by语句的结果会默认进行排序。 如果你要在order by后面加group by ，那结果执行的SQL是不是这样：select * from tb order by … group by … order by … ？ 这不是搞笑吗？ ## 禁止排序 既然知道问题了，那么就容易优化了，如果查询包括group by但又不关心结果集的顺序，而这种默认排序又导致了需要文件排序，则可以指定order by null 禁止排序。 例如： ```sql select * from tb group by name order by null; ``` ## 优化limit 分页 一个非常常见又非常头痛的场景：‘limit 1000,20’。 这时MySQL需要查询1020条记录然后只返回最后20条，前面的1000条都将被抛弃，这样的代价非常高。如果所有页面的访问频率都相同，那么这样的查询平均需要访问半个表的数据。 ## 第一种思路 在索引上分页 在索引上完成分页操作，最后根据主键关联回原表查询所需要的其他列的内容。 例如： ```sql SELECT * FROM tb_user LIMIT 1000,10 ``` 可以优化成这样： ```sql SELECT * FROM tb_user u INNER JOIN (SELECT id FROM tb_user LIMIT 1000,10) AS b ON b.id=u.id ``` ## 第二种思路 将limit转换成位置查询 这种思路需要加一个参数来辅助，标记分页的开始位置： ```sql SELECT * FROM tb_user WHERE id > 1000 LIMIT 10 ``` ## 优化子查询 子查询，也就是查询中有查询，常见的是where后面跟一个括号里面又是一条查询sql 尽可能的使用join关联查询来代替子查询。 当然 这不是绝对的，比如某些非常简单的子查询就比关联查询效率高，事实效果如何还要看执行计划。 只能说大部分的子查询都可以优化成Join关联查询。 ## 改变执行计划 ## 提高索引优先级 use index 可以让MySQL去参考指定的索引，但是无法强制MySQL去使用这个索引，当MySQL觉得这个索引效率太差，它宁愿去走全表扫描。。。 ```sql SELECT * FROM tb_user USE INDEX (user_name) ``` 注意：必须是索引，不能是普通字段，（亲测主键也不行）。 ## 忽略索引 ignore index 可以让MySQL忽略一个索引 SELECT * FROM tb_user IGNORE INDEX (user_name) WHERE user_name="张学友" ## 强制使用索引 使用了force index 之后 尽管效率非常低，MySQL也会照你的话去执行 ```sql SELECT * FROM tb_user FORCE INDEX (user_name) WHERE user_name="张学友" ``` ## 个人分享 查看执行计划时建议依次观察以下几个要点： 1. SQL内部的执行顺序。 2. 查看select的查询类型。 3. 实际有没有使用索引。 4. Extra描述信息 PS:一定要养成查看执行计划的习惯，这个习惯非常重要。 ================================================ FILE: docs/sql/mysql_yh17.md ================================================ # 项目中常用的19条MySQL优化 ## 一、EXPLAIN 做MySQL优化，我们要善用 EXPLAIN 查看SQL执行计划。 下面来个简单的示例，标注(1,2,3,4,5)我们要重点关注的数据 ![mysql](./../img/mysql_yh17_1.png) 1. type列，连接类型。一个好的sql语句至少要达到range级别。杜绝出现all级别 2. key列，使用到的索引名。如果没有选择索引，值是NULL。可以采取强制索引方式 3. key_len列，索引长度 4. rows列，扫描行数。该值是个预估值 5. extra列，详细说明。注意常见的不太友好的值有：Using filesort, Using temporary ## 二、SQL语句中IN包含的值不应过多 MySQL对于IN做了相应的优化，即将IN中的常量全部存储在一个数组里面，而且这个数组是排好序的。但是如果数值较多，产生的消耗也是比较大的。再例如：select id from table_name where num in(1,2,3) 对于连续的数值，能用 between 就不要用 in 了；再或者使用连接来替换。 ## 三、SELECT语句务必指明字段名称 SELECT *增加很多不必要的消耗（cpu、io、内存、网络带宽）；增加了使用覆盖索引的可能性；当表结构发生改变时，前断也需要更新。所以要求直接在select后面接上字段名。 ## 四、当只需要一条数据的时候，使用limit 1 这是为了使EXPLAIN中type列达到const类型 ## 五、如果排序字段没有用到索引，就尽量少排序 ## 六、如果限制条件中其他字段没有索引，尽量少用or or两边的字段中，如果有一个不是索引字段，而其他条件也不是索引字段，会造成该查询不走索引的情况。很多时候使用 union all 或者是union(必要的时候)的方式来代替“or”会得到更好的效果 ## 七、尽量用union all代替union union和union all的差异主要是前者需要将结果集合并后再进行唯一性过滤操作，这就会涉及到排序，增加大量的CPU运算，加大资源消耗及延迟。当然，union all的前提条件是两个结果集没有重复数据。 ## 八、不使用ORDER BY RAND() ``` sql select id from `table_name` order by rand() limit 1000; ``` 上面的sql语句，可优化为 ``` sql select id from `table_name` t1 join (select rand() * (select max(id) from `table_name`) as nid) t2 on t1.id > t2.nid limit 1000; ``` ## 九、区分in和exists， not in和not exists ``` sql select * from 表A where id in (select id from 表B) ``` 上面sql语句相当于 ``` sql select * from 表A where exists(select * from 表B where 表B.id=表A.id) ``` 区分in和exists主要是造成了驱动顺序的改变（这是性能变化的关键），如果是exists，那么以外层表为驱动表，先被访问，如果是IN，那么先执行子查询。所以IN适合于外表大而内表小的情况；EXISTS适合于外表小而内表大的情况。 关于not in和not exists，推荐使用not exists，不仅仅是效率问题，not in可能存在逻辑问题。如何高效的写出一个替代not exists的sql语句？ 原sql语句 ``` sql select colname … from A表 where a.id not in (select b.id from B表) ``` 高效的sql语句 ``` sql select colname … from A表 Left join B表 on where a.id = b.id where b.id is null ``` 取出的结果集如下图表示，A表不在B表中的数据 ![mysql](./../img/mysql_yh17_2.png) ## 十、使用合理的分页方式以提高分页的效率 ``` sql select id,name from table_name limit 866613, 20 ``` 使用上述sql语句做分页的时候，可能有人会发现，随着表数据量的增加，直接使用limit分页查询会越来越慢。 优化的方法如下：可以取前一页的最大行数的id，然后根据这个最大的id来限制下一页的起点。比如此列中，上一页最大的id是866612。sql可以采用如下的写法： ``` sql select id,name from table_name where id> 866612 limit 20 ``` ## 十一、分段查询 在一些用户选择页面中，可能一些用户选择的时间范围过大，造成查询缓慢。主要的原因是扫描行数过多。这个时候可以通过程序，分段进行查询，循环遍历，将结果合并处理进行展示。 如下图这个sql语句，扫描的行数成百万级以上的时候就可以使用分段查询 ![mysql](./../img/mysql_yh17_3.png) ## 十二、避免在 where 子句中对字段进行 null 值判断 对于null的判断会导致引擎放弃使用索引而进行全表扫描。 ## 十三、不建议使用%前缀模糊查询 例如LIKE “%name”或者LIKE “%name%”，这种查询会导致索引失效而进行全表扫描。但是可以使用LIKE “name%”。 **那如何查询%name%**? 如下图所示，虽然给secret字段添加了索引，但在explain结果果并没有使用 ![mysql](./../img/mysql_yh17_4.png) 那么如何解决这个问题呢，答案：**使用全文索引** 在我们查询中经常会用到select id,fnum,fdst from table_name where user_name like '%zhangsan%'; 。这样的语句，普通索引是无法满足查询需求的。庆幸的是在MySQL中，有全文索引来帮助我们。 创建全文索引的sql语法是： ``` sql ALTER TABLE `table_name` ADD FULLTEXT INDEX `idx_user_name` (`user_name`); ``` 使用全文索引的sql语句是： ``` sql select id,fnum,fdst from table_name where match(user_name) against('zhangsan' in boolean mode); ``` **注意：在需要创建全文索引之前，请联系DBA确定能否创建。同时需要注意的是查询语句的写法与普通索引的区别**. ## 十四、避免在where子句中对字段进行表达式操作 比如 ``` sql select user_id,user_project from table_name where age*2=36; ``` 中对字段就行了算术运算，这会造成引擎放弃使用索引，建议改成 ``` sql select user_id,user_project from table_name where age=36/2; ``` ## 十五、避免隐式类型转换 where 子句中出现 column 字段的类型和传入的参数类型不一致的时候发生的类型转换，建议先确定where中的参数类型 ![mysql](./../img/mysql_yh17_5.png) ## 十六、对于联合索引来说，要遵守最左前缀法则 举列来说索引含有字段id,name,school，可以直接用id字段，也可以id,name这样的顺序，但是name;school都无法使用这个索引。所以在创建联合索引的时候一定要注意索引字段顺序，常用的查询字段放在最前面 ## 十七、必要时可以使用force index来强制查询走某个索引 有的时候MySQL优化器采取它认为合适的索引来检索sql语句，但是可能它所采用的索引并不是我们想要的。这时就可以采用force index来强制优化器使用我们制定的索引。 ## 十八、注意范围查询语句 对于联合索引来说，如果存在范围查询，比如between,>,<等条件时，会造成后面的索引字段失效。 ## 十九、关于JOIN优化 ![mysql](./../img/mysql_yh17_6.png) * LEFT JOIN A表为驱动表 * INNER JOIN MySQL会自动找出那个数据少的表作用驱动表 * RIGHT JOIN B表为驱动表 注意：**MySQL中没有full join，可以用以下方式来解决** ``` sql select * from A left join B on B.name = A.name where B.name is null union all select * from B; ``` ### 尽量使用inner join，避免left join 参与联合查询的表至少为2张表，一般都存在大小之分。如果连接方式是inner join，在没有其他过滤条件的情况下MySQL会自动选择小表作为驱动表，但是left join在驱动表的选择上遵循的是左边驱动右边的原则，即left join左边的表名为驱动表。 ### 合理利用索引 被驱动表的索引字段作为on的限制字段。 ### 利用小表去驱动大表 ![mysql](./../img/mysql_yh17_7.png) 从原理图能够直观的看出如果能够减少驱动表的话，减少嵌套循环中的循环次数，以减少 IO总量及CPU运算的次数。 ### 巧用STRAIGHT_JOIN inner join是由mysql选择驱动表，但是有些特殊情况需要选择另个表作为驱动表，比如有group by、order by等「Using filesort」、「Using temporary」时。STRAIGHT_JOIN来强制连接顺序，在STRAIGHT_JOIN左边的表名就是驱动表，右边则是被驱动表。**在使用STRAIGHT_JOIN有个前提条件是该查询是内连接，也就是inner join。其他链接不推荐使用** ![mysql](./../img/mysql_yh17_8.png) 这个方式有时可能减少3倍的时间。 ================================================ FILE: docs/sql/sql_server_master.md ================================================ # SQL Server 2017搭建主从备份 ### 关于日志传输 和Oracle DG，Mysql主从一样，SQL Server也支持主从的高可用。进一步提高数据的安全性和业务的高可用。通过将主库上的日志传输到备用实例上，从而达到主备库数据的一致性。 优点 * 可以为一个主库添加多个备库，从而提高数据灾难性恢复的解决方法。 * 和其他数据库主从一样，从库可以提高只读访问(在还原数据期间)。 * 可以自定义数据延迟应用时间。这样好处就是如果主库错误修改了数据，而从库还没有应用修改的数据，那么就可以通过从库来把错误环境的数据还原回来。 日志传输过程 * 在主服务器实例中备份事务日志。 * 将事务日志文件复制到辅助服务器实例。 * 在辅助服务器实例中还原日志备份。 日志可传送到多个辅助服务器实例。 在这些情况下，将针对每个辅助服务器实例重复执行操作 2 和操作 3。 日志传送配置不会自动从主库故障转移到辅助服务器。 如果主数据库变为不可用，可手动切换到任意一个从库。 下图是由一个主库，三个从库组成的主从环境。 ![sqlserver](../img/sqlserver/sqlserver1.png) 注：SQL Server 2008 Enterprise 及更高版本支持备份压缩。 ### 前提条件 主数据库必须处于Full恢复模式或大容量日志恢复模式。 在配置日志传送之前，必须创建共享备份目录，以便备库可以访问事务日志备份。 这是对生成事务日志备份的目录的共享。 例如，如果将事务日志备份到目录 E:\log_backup，则可以对该目录创建共享。 ### 搭建主从 下面步骤是可选的，使用administrator管理员账户也是可以的。如果是使用administrator用户，则下文中关于sqladmin用户权限相关的，替换为administrator。 1. 主从两台分别创建sqladmin用户加入administrators组删除默认的users组，并且设置"密码永不过期"和"用户不能更改密码" 为sqladmin创建一个密码 win+R，输入lusrmgr.msc 2. 设置用户权限 右击用户，点击“属性”，将默认的USERS组删除，新增Administrators组。 ![sqlserver](../img/sqlserver/sqlserver2.png) 3. 设置共享目录 在主或者从服务器上创建日志存放目录E:\log_backup ![sqlserver](../img/sqlserver/sqlserver3.png) 在“高级共享”窗口中，勾选“共享此文件夹”选项，然后单机“权限”按钮对该共享文件夹的权限进行设置。需要让sqladmin用户具有完全控制该文件夹的权限，先将默认的“erverone”用户删除，然后添加sqladmin,administaor用户，并在“sqladmin,administaor”的权限中勾选“完全控制”，“更改”和“读取”项，然后单击两次“确定”按钮保存共享设置。 ![sqlserver](../img/sqlserver/sqlserver4.png) 在NTFS文件系统中，还需要设置用户对该目录的安全权限，如果安全权限不够，系统在写入备份文件的时候会报没有权限的错误。 可以在“安全”选项卡，单机“编辑”按钮，在“log_backup的权限”界面，单击“按钮”，添加sqladmin用户，然后在“sqladmin的权限”中选择“完全控制”权限，单机“确定”按钮保存权限信息。 4. 配置SQL Server启动模式 分别从主数据库服务器上和从数据库服务器上打开SQLServer配置管理器，将SQLServer服务和SQLServer代理服务的“登录身为”sqladmin用户且启动模式为：自动 ![sqlserver](../img/sqlserver/sqlserver5.png) 5. 配置日志传输 右击数据库服务器实例，选择“属性”选项，在弹出的“服务器属性”界面中，单机左侧的“安全性”，然后在右侧窗口中的“服务器身份验证”中选择“SQLServer和Windows身份验证模式”，并勾选“服务器代理账户”中的“启用服务器代理账户”选项。输入正确的“代理账号”和“密码”，单击“确定”按钮保存配置信息。。 ![sqlserver](../img/sqlserver/sqlserver6.png) 6. 在主数据库服务器中配置要同步的数据库AppsHK属性 ![sqlserver](../img/sqlserver/sqlserver7.png) 7. 设置完之后选择“事务日志传送”，勾选“将此数据库启用未日志传送配置中的主数据库”选项，单击“事务日志备份”中的“备份设置按钮”，打开“事务日志备份设置”界面。 ![sqlserver](../img/sqlserver/sqlserver8.png) ![sqlserver](../img/sqlserver/sqlserver9.png) 8. 主库上备份文件夹的网路路径可以在备库上进行测试，看备库能否访问 备库上进行操作： ![sqlserver](../img/sqlserver/sqlserver10.png) 9. 点击计划,在“作业计划属性”界面，确认“计划类型”为重复执行，为测试效果明显，设置为15秒执行一次作业计划。最后确认“持续时间”，根据自己需要设置，如果一直备份的话，可以设置为“无结束日期”。 ![sqlserver](../img/sqlserver/sqlserver11.png) 10. 设置完成，确认之后再次打开“事务日志备份设置”界面，则备份作业的作业名称后面变成“编辑作业”按钮，单击进去，将“所有者”修改为sqladmin。 如果没有sqladmin先添加 ![sqlserver](../img/sqlserver/sqlserver12.png) ![sqlserver](../img/sqlserver/sqlserver13.png) ![sqlserver](../img/sqlserver/sqlserver14.png) 11. 单击数据库属性界面的“辅助数据库”中的“添加”按钮，打开“辅助数据库设置”窗口。 ![sqlserver](../img/sqlserver/sqlserver15.png) ![sqlserver](../img/sqlserver/sqlserver16.png) ![sqlserver](../img/sqlserver/sqlserver17.png) 12. 设置完之后点击确定按钮，在数据库属性配置界面将配置好的脚本保存到本地，最后点击确定 ![sqlserver](../img/sqlserver/sqlserver18.png) 到此数据库主从配置完成，可以在主库进行操作，在从库上查看对应的数据是否同步，如果没有同步成功，可以在从库上查看同步任务状态，查看失败原因。 ![sqlserver](../img/sqlserver/sqlserver19.png) ================================================ FILE: docs/tool/cat-monitoring.md ================================================ # Cat分布式监控 ## Cat 是什么？ CAT（Central Application Tracking）是基于 Java 开发的实时应用监控平台，包括实时应用监控，业务监控。 CAT 作为服务端项目基础组件，提供了 Java, C/C++, Node.js, Python, Go 等多语言客户端，已经在美团点评的基础架构中间件框架（MVC 框架，RPC 框架，数据库框架，缓存框架等，消息队列，配置系统等）深度集成，为美团点评各业务线提供系统丰富的性能指标、健康状况、实时告警等。 CAT 很大的优势是它是一个实时系统，CAT 大部分系统是分钟级统计，但是从数据生成到服务端处理结束是秒级别，秒级定义是 48 分钟 40 秒，基本上看到 48 分钟 38 秒数据，整体报表的统计粒度是分钟级；第二个优势，监控数据是全量统计，客户端预计算；链路数据是采样计算。 [Github Cat](https://github.com/dianping/cat) ## Cat 功能亮点 * 实时处理：信息的价值会随时间锐减，尤其是事故处理过程中 * 全量数据：全量采集指标数据，便于深度分析故障案例 * 高可用：故障的还原与问题定位，需要高可用监控来支撑 * 故障容忍：故障不影响业务正常运转、对业务透明 * 高吞吐：海量监控数据的收集，需要高吞吐能力做保证 * 可扩展：支持分布式、跨 IDC 部署，横向扩展的监控系统 ## 为什么要用 Cat? **场景一:** 用户反馈 App 无法下单，用户反馈无法支付，用户反馈商品无法搜索等问题 场景一的问题在于当系统出现问题后，第一反馈的总是用户。我们需要做的是什么，是在出问题后研发第一时间知晓，而不是让用户来告诉我们出问题了。 Cat 可以出故障后提供秒级别的异常告警机制，不用再等用户来反馈问题了。 **场景二:** 出故障后如何快速定位问题 一般传统的方式当出现问题后，我们就会去服务器上看看服务是否还存活。如果存活就会看看日志是否有异常信息。 在 Cat 后台的首页，会展示各个系统的运行情况，如果有异常，则会大片飘红，非常明显。最直接的方式还是直接查看 Problem 报表，这里会为我们展示直接的异常信息，快速定位问题。 ![cat](../img/tool/cat-1.png) **场景三:** 用户反馈订单列表要 10 几秒才展示，用户反馈下单一直在转圈圈 场景三属于优化相关，对于研发来说，优化是一个长期的过程，没有最好只有更好。优化除了需要有对应的方案，最重要的是要对症下药。 所谓的对症下药也就是在优化之前，你得先知道哪里比较慢。RPC 调用慢？数据库查询慢？缓存更新慢? Cat 可以提供详细的性能数据，95 线，99 线等。更细粒度的就是可以看到某个请求或者某个业务方法的所有耗时逻辑，前提是你做了埋点操作。 ## Cat 报表 >Cat 目前有五种报表，每种都有特定的应用场景，下面我们来具体聊聊这些报表的作用。 **Transaction 报表** 适用于监控一段代码运行情况，比如：运行次数、QPS、错误次数、失败率、响应时间统计（平均影响时间、Tp 分位值）等等场景。 埋点方式： ``` java public void shopService() { Transaction transaction = Cat.newTransaction("ShopService", "Service"); try { service(); transaction.setStatus(Transaction.SUCCESS); } catch (Exception e) { transaction.setStatus(e); // catch 到异常，设置状态，代表此请求失败 Cat.logError(e); // 将异常上报到cat上 // 也可以选择向上抛出：throw e; } finally { transaction.complete(); } } ``` 可以在基础框架中对 Rpc, 数据库等框架进行埋点，这样就可以通过 Cat 来监控这些组件了。 业务中需要埋点也可以使用 Cat 的 Transaction，比如下单，支付等核心功能，通常我们对 URL 进行埋点就可以了，也就包含了具体的业务流程。 ![cat](../img/tool/cat-2.png) **Event 报表** 适用于监控一段代码运行次数，比如记录程序中一个事件记录了多少次，错误了多少次。Event 报表的整体结构与 Transaction 报表几乎一样，只缺少响应时间的统计。 埋点方式： ``` java Cat.logEvent("Func", "Func1"); ``` ![cat](../img/tool/cat-3.png) **Problem 报表** Problem 记录整个项目在运行过程中出现的问题，包括一些异常、错误、访问较长的行为。 如果有人反馈你的接口报 500 错误了，你进 Cat 后就直接可以去 Problem 报表了，错误信息就在 Problem 中。 ![cat](../img/tool/cat-4.png) Problem 报表不需要手动埋点，我们只需要在项目中集成日志的 LogAppender 就可以将所有 error 异常记录，下面的段落中会讲解如何整合 LogAppender。 **Heartbeat 报表** Heartbeat 报表是 CAT 客户端，以一分钟为周期，定期向服务端汇报当前运行时候的一些状态。 ![cat](../img/tool/cat-5.png) 系统指标有系统的负载信息，内存使用情况，磁盘使用情况等。 JVM 指标有 GC 相关信息，线程相关信息。 **Business 报表** Business 报表对应着业务指标，比如订单指标。与 Transaction、Event、Problem 不同，Business 更偏向于宏观上的指标，另外三者偏向于微观代码的执行情况。 这个报表我也没怎么用过，用的多的还是前面几个。 ![cat](../img/tool/cat-6.png) ## Cat 在 Kitty Cloud 中的应用 >Kitty Cloud 的基础组件是 Kitty，Kitty 里面对需要的一些框架都进行了一层包装，比如扩展，增加 Cat 埋点之类的功能。 **Cat 的集成** Kitty 中对 Cat 封装了一层，在使用的时候直接依赖 kitty-spring-cloud-starter-cat 即可整合 Cat 到项目中。 ``` com.cxytiandi kitty-spring-cloud-starter-cat Kitty Version ``` 然后在 application 配置文件中配置 Cat 的服务端地址信息，多个英文逗号分隔： ``` yml cat.servers=47.105.66.210 ``` 在项目的 resources 目录下创建 META-INF 目录，然后在 META-INF 中创建 app.properties 文件配置 app.name。此名称是在 Cat 后台显示的应用名 ``` yml app.name=kitty-cloud-comment-provider ``` 最后需要配置一下 Cat 的 LogAppender，这样应用在记录 error 级别的日志时，Cat 可以及时进行异常告警操作。 在 logback.xml 增加下面的配置： ``` yml ``` [更详细的内容请移步 Cat 的 Github 主页进行查看。]() **MVC 框架埋点** 基于 Spring Boot 做 Web 应用开发，我们最常用到的一个 Starter 包就是 spring-boot-starter-web。 如果你使用了 Kitty 来构建微服务的框架，那么就不再需要直接依赖 spring-boot-starter-web。而是需要依赖 Kitty 中的 kitty-spring-cloud-starter-web。 kitty-spring-cloud-starter-web 在 spring-boot-starter-web 的基础上进行了封装，会对请求的 Url 进行 Cat 埋点，会对一些通用信息进行接收透传，会对 RestTemplate 的调用进行 Cat 埋点。 在项目中依赖 kitty-spring-cloud-starter-web： ``` com.cxytiandi kitty-spring-cloud-starter-web Kitty Version ``` 启动项目，然后访问你的 REST API。可以在 Cat 的控制台看到 URL 的监控信息。 ![cat](../img/tool/cat-7.png) 点击 URL 进去可以看到具体的 URL 信息。 ![cat](../img/tool/cat-8.png) **Mybatis 埋点** Kitty 中 Mybatis 是用的 Mybatis Plus, 主要是对数据库相关操作的 SQL 进行了 Cat 埋点，可以很方便的查看 SQL 的耗时情况。 依赖 kitty-spring-cloud-starter-mybatis： ``` com.cxytiandi kitty-spring-cloud-starter-mybatis Kitty Version ``` [其他的使用方式还是跟 Mybatis Plus 一样，具体参考 Mybatis Plus 文档](https://mp.baomidou.com) 只要涉及到数据库的操作，都会在 Cat 中进行数据的展示。 ![cat](../img/tool/cat-9.png) 点击 SQL 进去还可以看到是哪个 Mapper 的操作。 ![cat](../img/tool/cat-10.png) 再进一步就可以看到具体的 SQL 语句和消耗的时间。 ![cat](../img/tool/cat-11.png) 有了这些数据，后端研发同学就可以对相关的 SQL 进行优化了。 **Redis 埋点**如果需要使用 Spring Data Redis 的话，直接集成 kitty-spring-cloud-starter-redis 就可以，kitty-spring-cloud-starter-redis 中对 Redis 的命令进行了埋点，可以在 Cat 上直观的查看对应的命令和消耗的时间。 添加对应的 Maven 依赖： ``` com.cxytiandi kitty-spring-cloud-starter-redis Kitty Version ``` 直接使用 StringRedisTemplate： ``` java @Autowired private StringRedisTemplate stringRedisTemplate; stringRedisTemplate.opsForValue().set("name", "yinjihuan"); ``` Cat 中可以看到 Redis 信息。 ![cat](../img/tool/cat-12.png) 点击 Redis 进去可以看到有哪些命令。 ![cat](../img/tool/cat-13.png) **MongoDB 埋点** Kitty 中对 Spring Data Mongodb 做了封装，只对 MongoTemplate 做了埋点。使用时需要依赖 kitty-spring-cloud-starter-mongodb。 ``` com.cxytiandi kitty-spring-cloud-starter-mongodb Kitty Version ``` 在发生 Mongo 的操作后，Cat 上就可以看到相关的数据了。 ![cat](../img/tool/cat-14.png) 点进去就可以看到是 MongoTemplate 的哪个方法发生了调用。 ![cat](../img/tool/cat-15.png) 再进一步就可以看到具体的 Mongo 参数和消耗的时间。 ![cat](../img/tool/cat-16.png) 还有 Dubbo, Feign，Jetcache，ElasticSearch 等框架的埋点就不细讲了，感兴趣的可以移步 Github 查看代码。 ## Cat 使用小技巧 **埋点工具类**如果要对业务方法进行监控，我们一般会用 Transaction 功能，将业务逻辑包含在 Transaction 里面，就能监控这个业务的耗时信息。 埋点的方式也是通过 Cat.newTransaction 来进行，具体可以参考上面 Transaction 介绍时给出的埋点示列。 像这种埋点的方式最好是有一个统一的工具类去做，将埋点的细节封装起来。 ``` java public class CatTransactionManager { public static T newTransaction(Supplier function, String type, String name, Map data) { Transaction transaction = Cat.newTransaction(type, name); if (data != null && !data.isEmpty()) { data.forEach(transaction::addData); } try { T result = function.get(); transaction.setStatus(Message.SUCCESS); return result; } catch (Exception e) { Cat.logError(e); if (e.getMessage() != null) { Cat.logEvent(type + "_" + name + "_Error", e.getMessage()); } transaction.setStatus(e); throw e; } finally { transaction.complete(); } } } ``` 工具类使用： ``` java public SearchResponse search(SearchRequest searchRequest, RequestOptions options) { Map catData = new HashMap<>(1); catData.put(ElasticSearchConstant.SEARCH_REQUEST, searchRequest.toString()); return CatTransactionManager.newTransaction(() -> { try { return restHighLevelClient.search(searchRequest, options); } catch (IOException e) { throw new RuntimeException(e); } }, ElasticSearchConstant.ES_CAT_TYPE, ElasticSearchConstant.SEARCH, catData); } ``` 通过使用工具类，不再需要每个监控的地方都是设置 Transaction 是否 complete，是否成功这些信息了。 **注解埋点** 为了让 Transaction 使用更方便，我们可以自定义注解来做这个事情。比如需要监控下单，支付等核心业务方法，那么就可以使用自定义的 Transaction 注解加在方法上，然后通过 AOP 去统一做监控。 定义注解： ``` java @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD, ElementType.TYPE}) public @interface CatTransaction { /** * 类型, 默认为Method * @return */ String type() default ""; /** * 名称, 默认为类名.方法名 * @return */ String name() default ""; /** * 是否保存参数信息到Cat * @return */ boolean isSaveParamToCat() default true; } ``` 定义切面： ``` java @Aspect public class CatTransactionAspect { @Around("@annotation(catTransaction)") public Object aroundAdvice(ProceedingJoinPoint joinpoint, CatTransaction catTransaction) throws Throwable { String type = catTransaction.type(); if (StringUtils.isEmpty(type)){ type = CatConstantsExt.METHOD; } String name = catTransaction.name(); if (StringUtils.isEmpty(name)){ name = joinpoint.getSignature().getDeclaringType().getSimpleName() + "." + joinpoint.getSignature().getName(); } Map data = new HashMap<>(1); if (catTransaction.isSaveParamToCat()) { Object[] args = joinpoint.getArgs(); if (args != null) { data.put("params", JsonUtils.toJson(args)); } } return CatTransactionManager.newTransaction(() -> { try { return joinpoint.proceed(); } catch (Throwable throwable) { throw new RuntimeException(throwable); } }, type, name, data); } } ``` 注解使用： ``` java @CatTransaction @Override public Page searchArticleIndex(ArticleIndexSearchParam param) { } ``` ## 你可能关心的几个问题 **Cat 能做链路跟踪吗？** Cat 主要是一个实时监控系统，并不是一个标准的全链路系统，主要是 Cat 的 logview 在异步线程等等一些场景下，不太合适，Cat 本身模型并不适合这个。Cat 的 Github 上有说明：在美团点评内部，有 mtrace 专门做全链路分析。 但是如果在 Mvc，远程调用等这些框架中做好了数据的无缝传输，Cat 也可以充当一个链路跟踪的系统，基本的场景足够了。 Cat 也可以构建远程消息树，可以看到请求经过了哪些服务，每个服务的耗时等信息。只不过服务之间的依赖关系图在 Cat 中没有。 下图请求从网关进行请求转发到 articles 上面，然后 articles 里面调用了 users 的接口。 ![cat](../img/tool/cat-17.png) **Cat 跟 Skywalking 哪个好用?** Skywalking 也是一款非常优秀的 APM 框架，我还没用过，不过看过一些文档，功能点挺全的 ，界面也挺好看。最大的优势是不用像 Cat 一样需要埋点，使用字节码增强的方式来对应用进行监控。 之所以列出这个小标题是因为如果大家还没有用的话肯定会纠结要选择落地哪个去做监控。我个人认为这两个都可以，可以自己先弄个简单的版本体验体验，结合你想要的功能点来评估落地哪个。 用 Cat 的话最好有一套基础框架，在基础框架中埋好点，这样才能在 Cat 中详细的显示各种信息来帮助我们快速定位问题和优化性能。 [感兴趣的 Star 下呗：https://github.com/yinjihuan/kitty-cloud](https://github.com/yinjihuan/kitty-cloud) 参考资料 [1 cat](https://github.com/dianping/cat) [2 mybatisPlus](https://mp.baomidou.com) [3 kitty-cloud](https://github.com/yinjihuan/kitty-cloud) ================================================ FILE: docs/tool/cicd.md ================================================ # 推荐 10 个好用的 CI/CD 工具 - 虽然云平台的到来让开发者免于安装和维护物理服务器，但测试和部署代码过程依旧需要人为完成，持续集成可以自动消除构建、测试和部署代码的大部分痛苦。如果希望最大限度提高效率，持续集成和交付工具是最好的选择。 实际上，所有开发者都可在软件开发中使用 CI/CD，但团队使用可以获得更大优势，尤其是大型团队，因为他们通常在处理相同的互锁代码块。持续集成最全面的实现是在测试之前构建代码，寻找未被发现的错误和不兼容问题，这些可能是由不同的团队成员写入代码时创建的，持续集成服务器可以同步所有程序员的工作，并帮助团队检测所有问题。 虽然 CI/CD 热度不退，但完全自动化的部署方式会让团队管理者感到不踏实，因此很多团队习惯在此过程中添加一些手动暂停，并增加问责制和其他保证代码在可控范围内的规则，这种混合方法被称为持续交付，因为它将代码提供给某一阶段或测试集群，并等待开发者最终推向生产。 如果在服务器机房中持续集成非常好，那么在云中可以实现更快交付和更高效率。在最好的情况下，云可以拆分任务并行运行。服务从大量硬件开始，然后在许多团队之间共享，只要所有人不同时推送代码，构建和测试将运行得很快。 或许，唯一令团队担心的问题就是失去控制。所有云服务都需要将代码交给第三方，这种选择可能让某些人感到不自由。即便所有的云服务都在努力强调安全性，但依旧让人担心。以下是在云中进行持续集成的 10 种不同选项，可以帮助开发者更好得适应这一过程。 ### CloudBees CloudBees 核心源自 Jenkins，这是著名的持续集成开源项目，添加了测试支持以及代码运行保证。该公司将所有实验插件清理干净，添加了一些自研实验插件，然后打磨正确以便在需要时如期工作。 CloudBees 雇用了 80％的 Jenkins 开发团队，为开源项目贡献代码，因此可以确信他们对这个平台有很好的理解。为了加快速度，CloudBees 还添加了大量并行化工具跟踪开发过程。 CloudBees 提供各种价位，从免费到入门套件和全年服务，并为需要该工具但不需要云计算的开发者提供 Jenkins 支持。 ### AWS CodePipeline 亚马逊用于持续集成和部署的工具 AWS CodePipeline 经过优化，可以将代码交付给 AWS 服务器，同时为代码和数据提供更复杂的路径。基本工具为主要编程语言（Java，Python，Node.js，Ruby，Go，Android，.Net Core for Linux）提供了一个很好的预配置构建环境，发送之前将结果转储到 S3 中并关闭服务器就可以开始运行。 CodeBuild 在 CodePipe 由 CodePipeline 触发时从 CodeCommit 中获取最新结果，并将其交给 CodeDeploy。如果需要配置大量 Code 代码，可以直接跳到 CodeStar，这提供了另一层面的自动化。在技术上并不需要支付任何代码层费用，亚马逊收取的费用主要是沿途使用的计算和存储资源。 ### Bitbucket Pipelines Atlassian 是流行的 Jira 和代码库 Bitbucket 背后的公司，决定创建 Bitbucket Pipelines（Bitbucket 云中的持续集成工具）以包括更多集成。构建机制和 Atlassian 其他工具之间的连接形式成为重点，因此这只作为 Bitbucket 中项目的一个选项，另一个选项指向部署。 如果开发者选择为主要语言（Java，JavaScript，Python，PHP，.Net 等）定义的模板，则只需单击几下即可构建和部署代码，除此之外的选项不存在。Atlassian 确实鼓励一个似乎是图表和 webhook 混合到其他服务的应用程序市场。在我写这篇文章时，图表上的顶级应用程序会将 Bitbucket 与 Jenkins 连接起来，大概是为了做一些无法在墙内快速完成的事情。 Pipelines 的主要优点是速度，Atlassian 已预先设计从代码到运行部署的大多数主要途径，只需花费几美元就可以使用。很难比较使用 Bitbucket 的成本，因为构建只需几分钟，就像大多数无服务器模型一样，团队通常会使用一组实例来处理 Jenkins 构建。 ### GitLab CI / CD Atlassian 最大的竞争对手之一是 GitLab。GitLab 的构建，测试和部署机制同样直接连接到其 Git 存储库，因此可以根据规则触发。该过程主要围绕 Docker 容器构建，可以大大简化围绕 Jenkins 构建必须完成的一些配置工作。 构建任务可以针对任何语言，但必须由 GitLab Runner 触发，GitLab Runner 是一个用 Go 编写的自动缩放工具，适用于大多数平台。这种灵活性意味着可以在其他计算机上触发任何任务，这对于精心设计的架构而言可能非常有用，这些架构不仅仅提供微服务。 定价与级别捆绑在一起以满足需求，最高级别可以获得最佳功能，比如安全仪表板和在共享机器集群上构建 50,000 分钟，部分流程中使用自己的计算机或在其他云中使用单独的实例是免费的。 ### CircleCI 许多持续集成工具专注于在 Linux 环境中构建代码。CircleCI 既可在 Linux 环境中构建和提供，也可以构建 Android 应用程序以及 Xcode（适用于 iOS，MacOS，tvOS 或 watchOS）。 CircleCI 使用 Docker，在其所有层次中为代码配置测试环境。构建从新容器开始，所有测试也是如此，Mac 工作在类似的虚拟机中，避免了配置中的一些问题。 定价主要集中在构建的 CPU 数量上。用户数和存储库数量上限为无穷大，但构建分钟数和容器数是计量的。第一个容器是免费的，可以在其中运行一个构建，如果想要更多并行性或更高吞吐量，需要收费。 ### Travis CI 如果构建需要在 Windows 机器上测试的代码，那么 Travis CI 可以提供一站式服务。该公司已经提供了一段时间的 MacOS 和 Linux 选项，刚刚推出 Windows 选项，让生成更多代码变得更加简单。 目前，Linux 代码支持 Ubuntu 基本版本，Mac 代码以 OS X，Xcode 和 JDK 的十几种组合之一运行。Windows 代码只支持一个版本的 Windows Server（1803）。Travis CI 提供 30 多种语言支持列表，并构建预先配置且可以运行的规则。 定价基于一次执行的并发任务数，但这些构建可以占用的分钟数没有正式限制。定制化工作没有免费选项，但开源项目永远免费，这可能是尝试 Travis CI 最简单的方法。 ### Azure Pipelines 虽然，Azure 可能没有太多提供 ENIAC 程序员，但确实为代码提供 Microsoft、Linux 和 MacOS 路径。该堆栈包含 Docker 容器和 Azure 硬件。如果更喜欢命令行，则可以使用 YAML 指定。 有免费并行任务试用，1800 分钟的构建时间。如果想要更多的并行性或更多构建时间则需要付费。 ### CodeShip 使用持续集成工具时，配置任务列表通常是最大的挑战。CodeShip 在两个服务级别采用两种不同的方法。基础版本计划包括大量自动化和预配置以及图形用户界面，用于设置任务的大致轮廓。高级版本允许配置和用于定义构建环境的 Docker 容器，可以选择将多少个构建专用于任务以及可能的配置。 基础版本提供免费套餐，包括一台构建机器，无限制项目和人员，但每月只有 100 个构建。如果超过 100 个项目，并希望在一个月内完成所有项目，则需要付费。一旦付费，就没有构建数量或者花费多少时间的上限，只需选择构建和测试即可处理任务。 ### Sauce Labs 以上大多数工具集中在编排从存储库到部署的代码流，Sauce Labs 则专注于测试。基于云的服务提供了各种各样的组合，以确保一切正常。如果想在 Windows 10 上运行的 Firefox 58 上进行测试，或者在 MacOS 上的 Firefox 56，只要选择 Java，Node，Ruby 或 PHP 等其中之一，测试脚本就可以用选择的语言编写，云将并行定位每个处理器。 Sauce Labs 专门将测试与其他 CI 工具或管道集成，因此可以在本地运行 Jenkins，然后将测试委托给 Sauce Labs。 ### Jenkins and Hudson 在云中启动持续集成最简单的方法之一是租用服务器实例并启动 Jenkins 或 Hudson。二者很久以前就开始用作测试 Java 代码的程序，当开发人员和 Oracle 之间出现争议时，其分成了两个阵营，开发人员需要认真查看开源许可。 虽然 Jenkins 和 Hudson 可能已经开始作为构建 Java 项目的工具，但其早已超越这个利基市场，并可处理几乎所有语言，有数千个插件来处理构建、打包、测试和部署。代码是开源的，因此使用时不需要额外付费，只需支付服务器费用和配置时间。 ================================================ FILE: docs/tool/git.md ================================================ # Git 常用命令速查手册 ![git](./../img/git1.png "git") ## 初始化仓库 ``` s git init ``` ## 设置远程仓库地址后再做push ''' s git remote add origin ''' ## 将文件添加到仓库 ``` s git add 文件名 # 将工作区的某个文件添加到暂存区 git add -u # 添加所有被tracked文件中被修改或删除的文件信息到暂存区，不处理untracked的文件 git add -A # 添加所有被tracked文件中被修改或删除的文件信息到暂存区，包括untracked的文件 git add . # 将当前工作区的所有文件都加入暂存区 git add -i # 进入交互界面模式，按需添加文件到缓存区 ``` ## 将暂存区文件提交到本地仓库 ``` s git commit -m "提交说明" # 将暂存区内容提交到本地仓库 git commit -a -m "提交说明" # 跳过缓存区操作，直接把工作区内容提交到本地仓库 ``` ## 查看仓库当前状态 ``` s git status ``` ## 比较文件异同 ``` s git diff # 工作区与暂存区的差异 git diff 分支名 #工作区与某分支的差异，远程分支这样写：remotes/origin/分支名 git diff HEAD # 工作区与HEAD指针指向的内容差异 git diff 提交id 文件路径 # 工作区某文件当前版本与历史版本的差异 git diff --stage # 工作区文件与上次提交的差异(1.6 版本前用 --cached) git diff 版本TAG # 查看从某个版本后都改动内容 git diff 分支A 分支B # 比较从分支A和分支B的差异(也支持比较两个TAG) git diff 分支A...分支B # 比较两分支在分开后各自的改动 # 另外：如果只想统计哪些文件被改动，多少行被改动，可以添加 --stat 参数 ``` ## 查看历史记录 ``` s git log # 查看所有commit记录(SHA-A校验和，作者名称，邮箱，提交时间，提交说明) git log -p -次数 # 查看最近多少次的提交记录 git log --stat # 简略显示每次提交的内容更改 git log --name-only # 仅显示已修改的文件清单 git log --name-status # 显示新增，修改，删除的文件清单 git log --oneline # 让提交记录以精简的一行输出 git log –graph –all --online # 图形展示分支的合并历史 git log --author=作者 # 查询作者的提交记录(和grep同时使用要加一个--all--match参数) git log --grep=过滤信息 # 列出提交信息中包含过滤信息的提交记录 git log -S查询内容 # 和--grep类似，S和查询内容间没有空格 git log fileName # 查看某文件的修改记录，找背锅专用 ``` ## 代码回滚 ``` s git reset HEAD^ # 恢复成上次提交的版本 git reset HEAD^^ # 恢复成上上次提交的版本，就是多个^，以此类推或用~次数 git reflog git reset --hard 版本号 --soft：只是改变HEAD指针指向，缓存区和工作区不变； --mixed：修改HEAD指针指向，暂存区内容丢失，工作区不变； --hard：修改HEAD指针指向，暂存区内容丢失，工作区恢复以前状态； ``` ## 同步远程仓库 ``` s git push -u origin master ``` ## 删除版本库文件 ``` s git rm 文件名 ``` ## 版本库里的版本替换工作区的版本 ``` s git checkout -- test.txt ``` ## 本地仓库内容推送到远程仓库 ``` s git remote add origin git@github.com:帐号名/仓库名.git ``` ## 将本地仓库内容推送到远程仓库 ''' s git add . #将当前目录所有文件添加到git暂存区 git commit -m "my first commit" #提交并备注提交信息 git push origin master #将本地提交推送到远程仓库 ''' ## 从远程仓库克隆项目到本地 ``` s git clone git@github.com:git帐号名/仓库名.git ``` ## 创建分支 ``` s git checkout -b dev -b表示创建并切换分支 上面一条命令相当于一面的二条： git branch dev //创建分支 git checkout dev //切换分支 ``` ## 查看分支 ``` s git branch ``` ## 合并分支 ``` s git merge dev //用于合并指定分支到当前分支 git merge --no-ff -m "merge with no-ff" dev //加上--no-ff参数就可以用普通模式合并，合并后的历史有分支，能看出来曾经做过合并 ``` ## 删除分支 ``` s git branch -d dev ``` ## 查看分支合并图 ``` s git log --graph --pretty=oneline --abbrev-commit ``` ## 查看远程库信息 ``` s git remote // -v 显示更详细的信息 ``` ## git相关配置 ``` s # 安装完Git后第一件要做的事，设置用户信息(global可换成local在单独项目生效)： git config --global user.name "用户名" # 设置用户名 git config --global user.email "用户邮箱" #设置邮箱 git config --global user.name # 查看用户名是否配置成功 git config --global user.email # 查看邮箱是否配置 # 其他查看配置相关 git config --global --list # 查看全局设置相关参数列表 git config --local --list # 查看本地设置相关参数列表 git config --system --list # 查看系统配置参数列表 git config --list # 查看所有Git的配置(全局+本地+系统) git config --global color.ui true //显示git相关颜色 ``` ## 撤消某次提交 ``` s git revert HEAD # 撤销最近的一个提交 git revert 版本号 # 撤销某次commit ``` ## 拉取远程分支到本地仓库 ``` s git checkout -b 本地分支 远程分支 # 会在本地新建分支，并自动切换到该分支 git fetch origin 远程分支:本地分支 # 会在本地新建分支，但不会自动切换，还需checkout git branch --set-upstream 本地分支 远程分支 # 建立本地分支与远程分支的链接 ``` ## 标签命令 ``` s git tag 标签 //打标签命令，默认为HEAD git tag //显示所有标签 git tag 标签 �版本号 //给某个commit版本添加标签 git show 标签 //显示某个标签的详细信息 ``` ## 同步远程仓库更新 ``` s git fetch origin master //从远程获取最新的到本地，首先从远程的origin的master主分支下载最新的版本到origin/master分支上，然后比较本地的master分支和origin/master分支的差别，最后进行合并。 git fetch比git pull更加安全 git pull origin master ``` ## 推送时选择强制推送 强制推送需要执行下面的命令(默认不推荐该行为)： ``` s git push origin master -f ``` ## git 提示授权失败解决 ``` s git config --system --unset credential.helper # 管理员权限执行命令 ``` ## git 授权永久有效 ``` s git config --global credential.helper 'store' ``` ## 程序员的那些迷之缩写 就像你可能不知道 现充 其实是 现实生活很充实的人生赢家 的缩写一样，我们经常看到 Github 上的码农们在 code review 时，把乱七八糟的缩写写得到处都是——娴熟的司机们都会使用缩写来达到提高逼格的效果——我们第一次看到时还是会出现一脸懵逼的状况，这里整理一下这些缩写都是什么含义，以后我们也可以欢快地装逼了。 1. PR: Pull Request. 拉取请求，给其他项目提交代码 2. LGTM: Looks Good To Me. 朕知道了 代码已经过 review，可以合并 3. SGTM: Sounds Good To Me. 和上面那句意思差不多，也是已经通过了 review 的意思 4. WIP: Work In Progress. 传说中提 PR 的最佳实践是，如果你有个改动很大的 PR，可以在写了一部分的情况下先提交，但是在标题里写上 WIP，以告诉项目维护者这个功能还未完成，方便维护者提前 review 部分提交的代码。 5. PTAL: Please Take A Look. 你来瞅瞅？用来提示别人来看一下 6. TBR: To Be Reviewed. 提示维护者进行 review 7. TL , DR: Too Long; Didn't Read. 太长懒得看。也有很多文档在做简略描述之前会写这么一句 8. TBD: To Be Done(or Defined/Discussed/Decided/Determined). 根据语境不同意义有所区别，但一般都是还没搞定的意思 9. PRD : Product Requirement Document. 产品需求文档 ## git常用命令归纳 |分支命令|说明| |:-----|:-----| |git branch| 列出所有本地分支机构。| |git branch -a| 列出远程和本地分支。| |git checkout -b branch_name| 创建一个本地分支并切换到该分支。 | |git checkout branch_name| 切换到现有分支。 | |git push origin branch_name| 将分支推送到远程。 | |git branch -m new_name| 重命名当前分支。 | |git branch -d branch_name| 删除本地分支。 | |git push origin :branch_name| 删除远程分支。| |日志命令|说明| |:-----|:-----| |git log --oneline| 单行显示提交历史记录。| |git log -2| 显示最近N次提交的提交历史记录。 | |git log -p -2| 用diff显示最近N次提交的提交历史记录。 | |git diff| 在工作树中显示所有本地文件更改。 | |git diff myfile| 显示对文件所做的更改。 | |git blame myfile| 显示谁更改了文件的内容和时间。 | |git remote show origin| 显示远程分支及其到本地的映射。| |清理命令|说明| |:-----|:-----| |git clean -f| 删除所有未跟踪的文件。 | |git clean -df| 删除所有未跟踪的文件和目录。 | |git checkout -- .| 撤消对所有文件的本地修改。 | |git reset HEAD myfile| 取消暂存文件。| |标签命令|说明| |:-----|:-----| |git tag| 列出所有标签。| |git tag -a tag_name -m "tag message"| 创建一个新标签。 | |git push --tags| 将所有标签推送到远程仓库。| |存放命令|说明| |:-----|:-----| |git stash save "stash name" && git stash| 将更改保存到存储中。 | |git stash list| 列出所有藏匿处。 | |git stash pop| 应用藏匿处。| ================================================ FILE: docs/tool/gitbook.md ================================================ # GitBook 使用教程 首先先献上 我的 [blog](https://burningmyself.github.io/blog/ "blog") 地址，可以在我的博客导航栏处找到，下面进行相关的介绍。 ## 背景 由于之前都把零散的知识都写在博客园上，要查找的时候不是很系统化，所以打算挪到[GitBook](https://www.gitbook.com/?t=2 "GitBook") 上来统一管理，而且[GitBook](https://www.gitbook.com/?t=2 "GitBook") 写完编译后可以生成静态页面发布到博客上，逼格满满的样子。 ## GitBook 简介 * [GitBook 官网](https://docs.gitbook.com/ "官网") * [GitBook 文档](https://docs.gitbook.com/ "文档") * [GitBook GitHub](https://github.com/GitbookIO/gitbook "GitHub") ## GitBook 准备工作 上面我推荐的是 Node.js + GitBook + Typora + Git，所以你还需要安装 Typora（一个很棒的支持 macOS、Windows、Linux 的 Markdown 编辑工具）和 Git 版本管理工具。戳下面： * [Node.js](https://nodejs.org/en/) * [Typora](https://typora.io/) * [Git](https://git-scm.com/) ### 安装 Node.js GitBook 是一个基于[Node.js](https://nodejs.org/en/ "Node.js") 的命令行工具，下载安装 Node.js，安装完成之后，你可以使用下面的命令来检验是否安装成功。 ```node $ node -v v10.5.0 ``` ### 安装 GitBook 输入下面的命令来安装 GitBook。 ```node $ npm install gitbook-cli -g ``` 安装完成之后，你可以使用下面的命令来检验是否安装成功 ```node $ gitbook -V CLI version: 2.3.2 Installing GitBook 3.2.3 ``` 更多详情请参照[GitBook 安装文档](https://docs.gitbook.com/ "文档") 来安装 GitBook。 ### 安装 Typora Typora 的安装很简单，难点在于需要翻墙才能下载（当然你也可以找我要）。Git 的安装也很简单，但要用好它需要不少时间，这里就不展开了（再讲下去你就要跑啦）。 ### 安装 GitBook 编辑器 去[GitBook 官网](http://downloads.editor.gitbook.com/download/ "官网") 下载 GitBook 编辑器；如果是 Mac 用户且安装过 brew cask 的话可以使用 brew cask install gitbook-editor 命令行来安装 GitBook 编辑器。 ## 先睹为快 GitBook 准备工作做好之后，我们进入一个你要写书的目录，输入如下命令。 ```node $ gitbook init warn: no summary file in this book info: create README.md info: create SUMMARY.md info: initialization is finished ``` 可以看到他会创建 README.md 和 SUMMARY.md 这两个文件，README.md 应该不陌生，就是说明文档，而 SUMMARY.md 其实就是书的章节目录，其默认内容如下所示： ``` # Summary * [Introduction](README.md) ``` 接下来，我们输入 $ gitbook serve 命令，然后在浏览器地址栏中输入 http://localhost:4000 便可预览书籍。 效果如下所示： ![效果图](./../img/gitbook1.png "效果图") 运行该命令后会在书籍的文件夹中生成一个 _book 文件夹, 里面的内容即为生成的 html 文件，我们可以使用下面命令来生成网页而不开启服务器。 ``` gitbook build ``` 下面我们来详细介绍下 GitBook 目录结构及相关文件。 ### Typora 来编辑这两个文件 ![效果图](./../img/gitbook4.png "效果图") 编辑 SUMMARY.md 文件，内容修改为： ``` # 目录 * [前言](README.md) * [第一章](Chapter1/README.md) * [第1节：衣](Chapter1/衣.md) * [第2节：食](Chapter1/食.md) * [第3节：住](Chapter1/住.md) * [第4节：行](Chapter1/行.md) * [第二章](Chapter2/README.md) * [第三章](Chapter3/README.md) * [第四章](Chapter4/README.md) ``` 然后我们回到命令行，在 mybook 文件夹中再次执行 gitbook init 命令。GitBook 会查找 SUMMARY.md 文件中描述的目录和文件，如果没有则会将其创建。 Typora 是所见即所得（实时渲染）的 Markdown 编辑器，这时候它是这样的： ![效果图](./../img/gitbook5.png "效果图") 接着我们执行 gitbook serve 来预览这本书籍，执行命令后会对 Markdown 格式的文档进行转换，默认转换为 html 格式，最后提示 “Serving book on http://localhost:4000”。嗯，打开浏览器看一下吧： ![效果图](./../img/gitbook6.png "效果图") ### gitbook 常用命令 当你写得差不多，你可以执行 gitbook build 命令构建书籍，默认将生成的静态网站输出到 _book 目录。实际上，这一步也包含在 gitbook serve 里面，因为它们是 HTML，所以 GitBook 通过 Node.js 给你提供服务了。 1. 当然，build 命令可以指定路径： ``` gitbook build [书籍路径] [输出路径] ``` 2. serve 命令也可以指定端口： ``` gitbook serve --port 2333 ``` 3. 你还可以生成 PDF 格式的电子书： ``` gitbook pdf ./ ./mybook.pdf ``` 4. 生成 epub 格式的电子书： ``` gitbook epub ./ ./mybook.epub ``` 5. 生成 mobi 格式的电子书： ``` gitbook mobi ./ ./mybook.mobi ``` 基本命令 ``` gitbook init //初始化目录文件 gitbook help //列出gitbook所有的命令 gitbook --help //输出gitbook-cli的帮助信息 gitbook build //生成静态网页 gitbook serve //生成静态网页并运行服务器 gitbook build --gitbook=2.0.1 //生成时指定gitbook的版本, 本地没有会先下载 gitbook ls //列出本地所有的gitbook版本 gitbook ls-remote //列出远程可用的gitbook版本 gitbook fetch 标签/版本号 //安装对应的gitbook版本 gitbook update //更新到gitbook的最新版本 gitbook uninstall 2.0.1 //卸载对应的gitbook版本 gitbook build --log=debug //指定log的级别 gitbook builid --debug //输出错误信息 ``` 如果生成不了，你可能还需要安装一些工具，比如 ebook-convert。或者在 Typora 中安装 Pandoc 进行导出。 除此之外，别忘了还可以用 Git 做版本管理呀！在 mybook 目录下执行 git init 初始化仓库，执行 git remote add 添加远程仓库（你得先在远端建好）。接着就可以愉快地 commit，push，pull … 啦！ 下面我们主要来讲讲 book.json 和 SUMMARY.md 文件。 ### book.json 该文件主要用来存放配置信息，我先放出我的配置文件。 ```conf { "title": "Blankj's Glory", "author": "Blankj", "description": "select * from learn", "language": "zh-hans", "gitbook": "3.2.3", "styles": { "website": "./styles/website.css" }, "structure": { "readme": "README.md" }, "links": { "sidebar": { "我的狗窝": "https://blankj.com" } }, "plugins": [ "-sharing", "splitter", "expandable-chapters-small", "anchors", "github", "github-buttons", "donate", "sharing-plus", "anchor-navigation-ex", "favicon" ], "pluginsConfig": { "github": { "url": "https://github.com/Blankj" }, "github-buttons": { "buttons": [{ "user": "Blankj", "repo": "glory", "type": "star", "size": "small", "count": true } ] }, "donate": { "alipay": "./source/images/donate.png", "title": "", "button": "赞赏", "alipayText": " " }, "sharing": { "douban": false, "facebook": false, "google": false, "hatenaBookmark": false, "instapaper": false, "line": false, "linkedin": false, "messenger": false, "pocket": false, "qq": false, "qzone": false, "stumbleupon": false, "twitter": false, "viber": false, "vk": false, "weibo": false, "whatsapp": false, "all": [ "google", "facebook", "weibo", "twitter", "qq", "qzone", "linkedin", "pocket" ] }, "anchor-navigation-ex": { "showLevel": false }, "favicon":{ "shortcut": "./source/images/favicon.jpg", "bookmark": "./source/images/favicon.jpg", "appleTouch": "./source/images/apple-touch-icon.jpg", "appleTouchMore": { "120x120": "./source/images/apple-touch-icon.jpg", "180x180": "./source/images/apple-touch-icon.jpg" } } } } ``` 相信很多节点自己也能猜到是什么意思，我还是简单介绍下吧。 * title:本书标题 * author:本书作者 * description:本书描述 * language:本书语言，中文设置 "zh-hans" 即可 * gitbook:指定使用的 GitBook 版本 * styles:自定义页面样式 * structure:指定 Readme、Summary、Glossary 和 Languages 对应的文件名 * links:在左侧导航栏添加链接信息 * plugins:配置使用的插件 * pluginsConfig:配置插件的属性 * SUMMARY.md:这个文件主要决定 GitBook 的章节目录，它通过 Markdown 中的列表语法来表示文件的父子关系，下面是一个简单的示例： ``` # Summary * [Introduction](README.md) * [Part I](part1/README.md) * [Writing is nice](part1/writing.md) * [GitBook is nice](part1/gitbook.md) * [Part II](part2/README.md) * [We love feedback](part2/feedback_please.md) * [Better tools for authors](part2/better_tools.md) ``` 这个配置对应的目录结构如下所示: ![如图](./../img/gitbook2.png "如图") 我们通过使用 标题 或者 水平分割线 将 GitBook 分为几个不同的部分，如下所示 ``` # Summary ### Part I * [Introduction](README.md) * [Writing is nice](part1/writing.md) * [GitBook is nice](part1/gitbook.md) ### Part II * [We love feedback](part2/feedback_please.md) * [Better tools for authors](part2/better_tools.md) --- * [Last part without title](part3/title.md) ``` 这个配置对应的目录结构如下所示: ![如图](./../img/gitbook3.png "如图") ### 插件 GitBook 有 [插件官网](https://plugins.gitbook.com/ "插件官网")，默认带有 5 个插件，highlight、search、sharing、font-settings、livereload，如果要去除自带的插件， 可以在插件名称前面加 -，比如： ``` "plugins": [ "-search" ] ``` 如果要配置使用的插件可以在 book.json 文件中加入即可，比如我们添加 [plugin-github](https://plugins.gitbook.com/plugin/github )，我们在 book.json 中加入配置如下即可： ``` { "plugins": [ "github" ], "pluginsConfig": { "github": { "url": "https://github.com/your/repo" } } } ``` 然后在终端输入 gitbook install ./ 即可。 ### 总结 1. 主要流程 安装node.js > 安装Gitbook > 安装Gitbook编辑器 > 安装calibre > 导出PDF 2. 安装 GitBook ``` npm install -g gitbook npm install -g gitbook-cli gitbook -V # 查看版本 ``` 3. 终端生成HTML [客户端编辑器下载：](https://www.gitbook.com/editor) ``` cd /Users/sunshine/GitBook/Library/Import/test/ gitbook build gitbook serve # 本地预览 _book ``` 4. 使用calibre插件生成PDF [插件下载：](https://calibre-ebook.com/download) ``` ln -s /Applications/calibre.app/Contents/MacOS/ebook-convert /usr/local/bin gitbook pdf . mypdf.pdf``` ``` 如果要指定插件的版本可以使用 plugin@0.3.1，因为一些插件可能不会随着 GitBook 版本的升级而升级。 ### 结语 这是我对 GitBook 使用的总结，希望能帮到今后需要的小伙伴 [相关资料](https://blog.csdn.net/axi295309066/article/details/61420694/) ================================================ FILE: docs/tool/gitcmr.md ================================================ # Git提交日志规范 在软件开发流程中，git或者其他的版本控制工具已经成为必不可少的一部分。Git每次提交代码都需要写commit message，否则不允许提交，一般来说commit message应该清晰明了,说明本次提交的目的，这对于之后bug定位，问题的分析都有很大的帮助。但是在日常开发中，大家的commit message都千奇百怪，fix issue，fix bug等等笼统不清晰的git message充斥在git history中，这就导致后续维护人员无法快速定位问题，有时甚至自己提交的代码都不知道是为什么，所以我们就需要一个规范来统一和管理commit message。 ## 规范梳理 目前比较知名的规范是Angular的commit message规范 ,同时也有很多相对应的IDE插件，比如Intellij IDEA的Git Commit Template 来帮助实现规范。现在以Angular的规范为基础介绍一套较为通用的git commit message规范。 ### Commit Message Format ```
``` 每一个commit message都应该包含header，body和footer。其中footer可以省略，但是header和body都不能为空。 ### Header Header分为三个部分type, scope, summary，其中type和summary为必填项，scope可以省略，格式如下: ``` (): ``` * Type: 用于说明git commit的类别，只允许使用下面的标识。 * + feat: 新功能（feature）。 * + fix: 修复bug，可以是QA发现的BUG，也可以是研发自己发现的BUG。 * + docs: 文档（documentation）。 * + style: 格式（不影响代码运行的变动）。 * + refactor: 重构（即不是新增功能，也不是修改bug的代码变动）。 * + perf: 优化相关，比如提升性能、体验。 * + test: 增加测试。 * + chore: 构建过程或辅助工具的变动。 * + revert: 回滚到上一个版本。 * Scope Scope用于说明 commit 影响的范围，比如Controller、DAO、View等等，视项目不同而不同。例如在Angular中可以是: * + animations * + bazel * + benchpress * + common * + compiler * + compiler-cli * + core * + elements 等等，如果其中包含了多个scope，可以用逗'*'隔'。 * Summary Summary是对commit的一个简短的描述，一般Git Commit Head总共不超过50个字符，所以summary必须精简。对于英文的commit summary，第一，要使用第一人称，现在时，比如change，不是changed也不是changes，第二，首字母无需大写，第三，句尾不要标点符号。中文除了时态，其他也一样。 根据上述规范git commit message header可以如下: ``` fix(Controller): request url map typo ``` ### Body 和Header中的summary一样。同时需要解释提交的动机，为什么需要更改，可以和之前的行为进行比较，来说明改动的影响等等。 是对本次commit的详细描述，可以分成多行： ``` More detailed explanatory text, if necessary. Wrap it to about 72 characters or so. Further paragraphs come after blank lines. - Bullet points are okay, too - Use a hanging indent ``` 有两个注意点： 1. 使用第一人称现在时 2. 说明代码变动的动机，以及与以前行为的对比 ### Footer Footer适用于当提交的改动包含了不可兼容变化或者弃用的变化，Footer部分以BREAKING CHANGE开头，后面是对变动的描述、以及变动理由和迁移方法，同时可以把相关Github issue，JIRA ticket或者其他文档链接填入其中。例子如下: ``` BREAKING CHANGE: Fixes # ``` ``` DEPRECATED: Closes # ``` * 关闭Issue 如果当前commit针对某个issue，那么可以在Footer部分关闭这个issue ``` Closes #123, #245, #992 ``` ### Revert 有一种特殊情况，如果当前commit用于撤销以前的commit，则必须以revert:开头，后面跟着被撤销的Commit的Header ```shell revert: feat(pencil): add 'graphiteWidth' option This reverts commit 667ecc1654a317a13331b17617d973392f415f02. ``` Body部分的格式是固定的，必须写成·This reverts commit · ### Commit message的使用 * 提供更多的Git History信息 当浏览Github Commit History页面时，只要看首行就可以知道某次的commit的目的，如果没有GUI工具，使用命令行工具使用 git log HEAD --pretty=format:%s 也能很方便清晰的浏览每次改动的信息。 * 快速查找Commit信息 用命令行工具，可以很方便的查找出，或者过滤出相关的commit git log HEAD --grep perf 例如上面的命令，就可以迅速的查处所有perf，性能修改相关的commit。 ### Commitizen 一个撰写合格的Commit message的工具 安装： ``` s npm install -g commitizen ``` 然后，在项目目录里，运行下面的命令，使其支持 Angular 的 Commit message 格式。 ``` commitizen init cz-conventional-changelog --save --save-exact ``` 以后凡是用到git commit的命令，一律改为git cz，这时，就会出现选项，用来生成符合格式的Commit message。 ### validate-commit-msg 用于检查Node项目的Commit message是否符合格式，需要手动安装 首先拷贝这个JS文件到代码库，命名为validate-commit-msg.js 然后把这个脚本加入Git的Hook，在package.json中使用ghooks，把这个脚本加为commit-msg时运行 ``` conf "config": { "ghooks": { "commit-msg": "./validate-commit-msg.js" } } ``` 然后每次Git commit时脚本就会自动检查Commit message是否合格，不合格就会报错 ### 简单的示例 ``` feat: 新增分析师清理功能 分析师清理功能，包括 1. 查询分析师 2. 分时段清理 ``` 不包含正文的提交说明 ``` docs: correct spelling of CHANGELOG ``` 包含作用域的提交说明 ``` feat(lang): added polish language ``` 为fix编写的提交说明，包含可选的issue编号 ``` fix: minor typos in code see the issue for details on the typos fixed fixes issue #12 ``` # Git Commit Message Rules ## `GCM001` 首行作为「总结行」，简明扼要的总结提交内容 无论是 Kernel 领袖 Torvalds，还是 Vim 骨灰级玩家 [Tim Pope](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)，都提出了这一点要求。其中，Tim Pope 建议，该行信息长度应该控制在50个字符以内。 即使 Torvalds 并没有限制该行信息的长度，但是我们还是要对该长度进行控制。在`GCM005`中列举了若干需要用到总结行的命令和工具。在这些命令或工具中，如果「总结行」长度过长，那么显示效果和阅读体验将会大打折扣。 ## `GCM002` 「总结行」行文使用「祈使句」 如果某次提交，主要是用于修复某个 Bug，那么使用英文写日志， 就应该表述为 「Fix Bug」，而不是 「Fixing Bug」或「Fixed Bug」。如果使用中文，则应写作「修复 Bug」，避免写作「修复了 Bug」或者 「已修复 Bug」等形式。 统一使用祈使句，一方面，可以统一形式；另外祈使句的 Git 日志，也能够与 `git merge` 和 `get revert` 自动生成的 Git 日志的形式相吻合。 在 GitLab 中，「总结行」中使用`#XXX`来引用 Issue。其中`XXX`就是 Issue ID。如果这个提交和这个 Issue 是位于同一个项目中的，那么 GitLab 就会自动为你创建一个指向 Issue 的链接。 在GitHub中，你甚至可以在「总结行」中，[使用关键词关闭一个Issue](https://help.github.com/articles/closing-issues-using-keywords/)。例如「总结行」中包含「Fix #123」，那么这个提交一旦被合并至主分支，Issue123 将会被 Close。 ## `GCM003` 不在「总结行」结尾使用标点符号 「总结行」的作用，类似于标题。因此，不需要在总结行末尾添加标点符号。 ## `GCM004` 使用「正文」来描述提交细节信息 提交的「正文」，应当回答以下几个问题： 1. `WHY` 这次提交是为了解决什么样的问题？ * 项目管理者，可能需要通过这个问题的回答，来判断究竟是否要将本次提交合并至 master 分支； * 执行 Code Review 时，Reviewers 可以通过这个问题的回答，更加迅速地了解本次提交的内容，以及剔除不相关的提交内容； * 当我们为了快速排查一次引入问题的提交时，这个信息也能够为我们提供很多有价值的参考。 2. `HOW` 这次提交如何解决刚才所提到的问题？ 在更高的层面上，描述一下我们为了解决所提到的问题，采取了哪些策略和算法。例如，「通过在XXX过程中引入一个XXX对应于XXX关系的哈希表，提高对XXX的索引性能，进一步解决XXX方面执行效率的瓶颈。」就是一个很不错的描述。 当然，如果你所采取的做法非常明显，比如这次提交仅仅是为了修正不规范的代码风格，或者修正一个拼写错误，你大可省略这一段的描述。 当然，如果目前的做法有哪些地方不太妥当，或者只是临时的解决办法（Walk Across Solution），需要后续更正，这些内容也可以在本段中进行描述说明。 3. `OTHERS` 这次提交还包含其他哪些修改？ 提交日志是否需要包含这段内容，取决于你的团队对单次提交所包含的修改内容多少的容忍程度。一般来讲，Git 鼓励我们只做「原子提交」，即每次提交仅仅只涉及一个内容，只解决一个问题。如果单次提交确实涉及到其他的修改项，可以在本段中列出。但是本着 「原子提交」的原则，其他的修改项不宜过多，尽量控制在 2 个以内。 此外，「正文」在行文中，可以使用列表的形式对所阐述的信息分条阐述。列表标记可以使用 `*` 或 `-` 符号。这一点和 `Markdown` 格式非常类似。 ## `GCM005` 「总结行」与「正文」之间使用「空行分隔」 「空行分隔」用于区别「总结行」与「正文」部分，某些命令或工具，都会根据「空行分隔」对提交日志进行解析： * `git format-patch`和`git send-email`命令中，「总结行」将作为邮件标题使用，「正文」将作为邮件正文使用； * `git log --oneline`，`git shortlog`命令将使用「总结行」作为提交简要描述，如果没有「空行分隔」，提交简要描述中，将会包含「正文」部分； * 使用`git rebase -i`后，自动启动的编辑器中，会使用「总结行」作为每个提交的简要描述； * 如果 Git 设置了`merge.summary`选项，那么当执行`merge`操作时，Git 将会汇总来自所有被 merge 的提交的「总结行」，作为本次 merge 提交的「总结行」； * `gitk`工具有专门的一栏用于显示「总结行」； * `GitLab`和`GitHub`在他们的用户界面总，都专门为「总结行」做了显示设计； ## `GCM006` 「正文」中的不同段落使用空行分隔 该原则与 `Markdown` 的段落分隔方法相似。 ## `GCM007` 「正文」每一行的长度控制在 72 个字符以内 在 Git 的设计思路中，提交日志的排版，换行等工作，需要日志的编写者来负责处理的。因为只有日志提交者，才知道究竟哪些地方需要换行，哪些地方不能换行。[Torvalds对此也有解释](https://github.com/torvalds/linux/pull/17#issuecomment-5660604)： * `git log`命令在显示提交日志时，并不负责对日志进行分页换行处理，所以如何断行机比较美观，需要日志编写者来考虑处理； * 某些信息，例如编译器的编译输出等，是不能包含换行的。 **为何日志的长度需要限制在 72 个字符呢？** 1. 默认情况下，`git log`命令会调用`less -S`对提交日志进行分页显示。因此，对于常用的宽度为80字符的终端来讲，如果你的日志中一行的长度超过80，那么长度超过80的的部分将会显示在终端之外，阅读起来将会很不方便。在宽度为80字符的终端中，为了更好的显示日志内容，80个字符减去左边可能会存在的4字符缩进，以及右边为了保证左右对称的4字符宽度，就得到了 72 字符的日志长度限制。 2. 使用`git format-patch --stdout`命令时，Git 会将提交转换成多个邮件序列。对于纯文本格式的邮件，为了保证邮件阅读体验，一般要保证在考虑了历史邮件嵌套的情况下，在80个字符的终端中，邮件仍然可以较美观的显示，因此，对于这个需求，72 字符的宽度限制，仍然是一个不错的选择。 ## `GCM008` 确保你的提交是「原子提交」 「原子提交」指的是 每个提交**仅包含一个修改**，且**必须包含该修改所有的相关内容**。因此，你不可以： * 提交中混合与提交目的不相关的提交内容； * 提交不完整的内容，甚至导致代码不能正常编译的内容； * 在一个较大的修改提交中「隐藏」一些很小的代码改动。 使用`git add -p`命令，可以帮助你较方便的将你的提交整理成一个个小的的「原子提交」。 ================================================ FILE: docs/tool/gitflow.md ================================================ # Git基本命令和GitFlow工作流 --- ### git 团队协作的一些命令 1. 开分支 ``` git branch 新分支名 例如，在master分支下，新开一个开发分支： git branch dev ``` 2. 切换到新分支 ``` git checkout 分支名 例如，在master分支下，切换到新开的dev： git checkout dev ``` 3. 开分支和切换分支合并到一个命令 ``` git checkout -b 新分支名 例如，新开一个开发分支，并立即切换到该分支： git checkout -b dev ``` 4. 切换回原分支 ``` git checkout 原分支名 例如，切换回master git checkout master 注意：当前分支有修改，还未commit的时候，会切换失败，应当先commit，但可以不用push ``` 5. 合并分支 ``` git merge 需要合并的分支名 例如，刚刚已经切换回master，现在需要合并dev的内容： git merge dev 建议在GitLab(或者其他git系统)上面创建merge request的形式来进行分支的合并和代码审核。 ``` 6. 查看本地分支列表 ``` git branch -a 前面带remotes/origin 的，是远程分支 ``` 7. 查看远程分支列表 ``` git branch -r ``` 8. 向远程提交本地新开的分支 ``` git push origin 新分支名 例如，刚刚在master下新开的dev分支： git push origin dev ``` 9. 删除远程分支 ``` git push origin :远程分支名 例如，删除刚刚提交到远程的dev分支： git push origin :dev ``` 10. 删除本地分支 ``` git branch 分支名称 -d 例如，在master分支下，删除新开的dev分支： git branch dev -d 注意：如果dev的更改，push到远程，在GitLab(或者其他git系统)上面进行了merge操作，但是本地master没有pull最新的代码，会删除不成功，可以先git pull origin master，或者强制删除 git branch dev -D ``` 11. 更新分支列表信息 ``` git fetch -p ``` 12. TortoiseGit(乌龟git) 不可否认，在windows下，这个是个不错的工具。不管你是命令行新手还是重度使用者，我觉得都可以尝试一下。 ### Git工作流指南：Gitflow工作流 在你开始阅读之前，请记住：流程应被视作为指导方针，而非“铁律”。我们只是想告诉你可能的做法。因此，如果有必要的话，你可以组合使用不同的流程 ![gitflow](../img/gitflow/git-workflows-gitflow.png) Gitflow工作流定义了一个围绕项目发布的严格分支模型。虽然比功能分支工作流复杂几分，但提供了用于一个健壮的用于管理大型项目的框架。 Gitflow工作流没有用超出功能分支工作流的概念和命令，而是为不同的分支分配一个很明确的角色，并定义分支之间如何和什么时候进行交互。除了使用功能分支，在做准备、维护和记录发布也使用各自的分支。当然你可以用上功能分支工作流所有的好处：Pull Requests、隔离实验性开发和更高效的协作。 #### 工作方式 Gitflow工作流仍然用中央仓库作为所有开发者的交互中心。和其它的工作流一样，开发者在本地工作并push分支到要中央仓库中。 #### 历史分支 相对使用仅有的一个master分支，Gitflow工作流使用2个分支来记录项目的历史。master分支存储了正式发布的历史，而develop分支作为功能的集成分支。这样也方便master分支上的所有提交分配一个版本号。 ![gitflow](../img/gitflow/git-workflow-release-cycle-1historical.png) 剩下要说明的问题围绕着这2个分支的区别展开。 #### 功能分支 每个新功能位于一个自己的分支，这样可以push到中央仓库以备份和协作。但功能分支不是从master分支上拉出新分支，而是使用develop分支作为父分支。当新功能完成时，合并回develop分支。新功能提交应该从不直接与master分支交互。 ![gitflow](../img/gitflow/git-workflow-release-cycle-2feature.png) 注意，从各种含义和目的上来看，功能分支加上develop分支就是功能分支工作流的用法。但Gitflow工作流没有在这里止步。 #### 发布分支 ![gitflow](../img/gitflow/git-workflow-release-cycle-3release.png) 一旦develop分支上有了做一次发布（或者说快到了既定的发布日）的足够功能，就从develop分支上fork一个发布分支。新建的分支用于开始发布循环，所以从这个时间点开始之后新的功能不能再加到这个分支上 —— 这个分支只应该做Bug修复、文档生成和其它面向发布任务。一旦对外发布的工作都完成了，发布分支合并到master分支并分配一个版本号打好Tag。另外，这些从新建发布分支以来的做的修改要合并回develop分支。 使用一个用于发布准备的专门分支，使得一个团队可以在完善当前的发布版本的同时，另一个团队可以继续开发下个版本的功能。 这也打造定义良好的开发阶段（比如，可以很轻松地说，『这周我们要做准备发布版本4.0』，并且在仓库的目录结构中可以实际看到）。 #### 常用的分支约定： ``` 用于新建发布分支的分支: develop 用于合并的分支: master 分支命名: release-* 或 release/* ``` #### 维护分支 ![gitflow](../img/gitflow/git-workflow-release-cycle-4maintenance.png) 维护分支或说是热修复（hotfix）分支用于生成快速给产品发布版本（production releases）打补丁，这是唯一可以直接从master分支fork出来的分支。修复完成，修改应该马上合并回master分支和develop分支（当前的发布分支），master分支应该用新的版本号打好Tag。 为Bug修复使用专门分支，让团队可以处理掉问题而不用打断其它工作或是等待下一个发布循环。你可以把维护分支想成是一个直接在master分支上处理的临时发布。 #### 示例 下面的示例演示本工作流如何用于管理单个发布循环。假设你已经创建了一个中央仓库。 ##### 创建开发分支 ![gitflow](../img/gitflow/git-workflow-release-cycle-5createdev.png) 第一步为master分支配套一个develop分支。简单来做可以本地创建一个空的develop分支，push到服务器上： ``` git branch develop git push -u origin develop ``` 以后这个分支将会包含了项目的全部历史，而master分支将只包含了部分历史。其它开发者这时应该克隆中央仓库，建好develop分支的跟踪分支： ``` git clone ssh://user@host/path/to/repo.git git checkout -b develop origin/develop ``` 现在每个开发都有了这些历史分支的本地拷贝。 ##### 工程师A和工程师B开始开发新功能 ![gitflow](../img/gitflow/git-workflow-release-cycle-6maryjohnbeginnew.png) 这个示例中，工程师A和工程师B开始各自的功能开发。他们需要为各自的功能创建相应的分支。新分支不是基于master分支，而是应该基于develop分支： ``` git checkout -b some-feature develop ``` 他们用老套路添加提交到各自功能分支上：编辑、暂存、提交： ``` git status git add git commit ``` ##### 工程师A完成功能开发 ![gitflow](../img/gitflow/git-workflow-release-cycle-7maryfinishes.png) 添加了提交后，工程师A觉得她的功能OK了。如果团队使用Pull Requests，这时候可以发起一个用于合并到develop分支。否则她可以直接合并到她本地的develop分支后push到中央仓库： ``` git pull origin develop git checkout develop git merge some-feature git push git branch -d some-feature ``` 第一条命令在合并功能前确保develop分支是最新的。注意，功能决不应该直接合并到master分支。冲突解决方法和集中式工作流一样。 ##### 工程师A开始准备发布 ![gitflow](../img/gitflow/git-workflow-release-cycle-8maryprepsrelease.png) 这个时候工程师B正在实现他的功能，工程师A开始准备她的第一个项目正式发布。像功能开发一样，她用一个新的分支来做发布准备。这一步也确定了发布的版本号： ``` git checkout -b release-0.1 develop ``` 这个分支是清理发布、执行所有测试、更新文档和其它为下个发布做准备操作的地方，像是一个专门用于改善发布的功能分支。 只要工程师A创建这个分支并push到中央仓库，这个发布就是功能冻结的。任何不在develop分支中的新功能都推到下个发布循环中。 ##### 工程师A完成发布 ![gitflow](../img/gitflow/git-workflow-release-cycle-9maryfinishes.png) 一旦准备好了对外发布，工程师A合并修改到master分支和develop分支上，删除发布分支。合并回develop分支很重要，因为在发布分支中已经提交的更新需要在后面的新功能中也要是可用的。另外，如果工程师A的团队要求Code Review，这是一个发起Pull Request的理想时机。 ``` git checkout master git merge release-0.1 git push git checkout develop git merge release-0.1 git push git branch -d release-0.1 ``` 发布分支是作为功能开发（develop分支）和对外发布（master分支）间的缓冲。只要有合并到master分支，就应该打好Tag以方便跟踪。 ``` git tag -a 0.1 -m "Initial public release" master git push --tags ``` Git有提供各种勾子（hook），即仓库有事件发生时触发执行的脚本。可以配置一个勾子，在你push中央仓库的master分支时，自动构建好对外发布。 ##### 最终用户发现Bug ![gitflow](../img/gitflow/git-workflow-gitflow-enduserbug.png) 对外发布后，工程师A回去和工程师B一起做下个发布的新功能开发，直到有最终用户开了一个Ticket抱怨当前版本的一个Bug。为了处理Bug，工程师A（或工程师B）从master分支上拉出了一个维护分支，提交修改以解决问题，然后直接合并回master分支： ``` git checkout -b issue-#001 master ``` Fix the bug ``` git checkout master git merge issue-#001 git push ``` 就像发布分支，维护分支中新加这些重要修改需要包含到develop分支中，所以工程师A要执行一个合并操作。然后就可以安全地删除这个分支了： ``` git checkout develop git merge issue-#001 git push git branch -d issue-#001 ``` ### 最后 Git是一个不错的版本管理工具，但也仅仅是工具。记住，这里演示的工作流只是可能用法的例子，而不是在实际工作中使用Git不可违逆的条例。所以不要畏惧按自己需要对工作流的用法做取舍。不变的目标就是让Git为你所用。 ### 参考资料 1. [gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow/) 2. [gitflow](https://nvie.com/posts/a-successful-git-branching-model/) ================================================ FILE: docs/tool/gitquestion.md ================================================ # Git 使用过程中遇到的问题以及解决办法 git 是项目当中最常用的代码管理库，熟练的使用git不是万能的，但不能熟练的使用git是万万不能的，归纳了一下真正开始在多人协作的代码库中提交自己的代码时遇到的问题。 ## git fetch 失效的问题 在项目工程中，主要使用的是两个分支，一个是主分支master，一个是开发分支develop，我们一般在develop中进行开发，master分支中用来存重大版本的代码。当需要获取最新的代码时，使用git fetch 或者 $ git fetch origin develop:develop 命令从远程develop分支上拉取最新的代码。 但有时候这个命令会失效，拉取不到最新的代码，出现这样的错误提示 ```git fatal: Refusing to fetch into current branch refs/heads/develop of non-bare repository fatal: The remote end hung up unexpectedly ``` 这种时候，先切换到master分支，然后再从master分支fetch分支develop上的代码，就能够成功了。 ```git $ git fetch origin develop:develop fatal: Refusing to fetch into current branch refs/heads/develop of non-bare repository fatal: The remote end hung up unexpectedly $ git checkout master Switched to branch 'master' Your branch is up-to-date with 'origin/master'. $ git fetch origin develop:develop From 172.20.10.91:developers/android_do_as 5ee9941..ff421cf develop -> develop ``` 究其原因，就和fatal的提示一样，在非bare的git仓库中，如果你要同步的本地跟踪分支是当前分支，就会出现拒绝fetch的情况。也就是说不可以在非bare的git仓库中通过fetch快进你的当前分支与远程同步。 ## git 错误提交或者错误的合并了解决方案 第一次在代码库中提交代码，心情比较激动，直接本地多次提交之后，就和远程分支给merge了。命令看起来是这么用的，但这样就会在代码线上弄出一条新线，而不是一条线，多么丑啊。还好没有push到远程去，所以就要看看如何解决，把它弄成一条线了。 当已经错误的提交，或者是错误的合并已经产生了，首先，要想办法回到过去，我多想回到过去，再回到你的身边。 ```git $ git reflog ff421cf HEAD@{0}: checkout: moving from master to develop efaaa61 HEAD@{1}: checkout: moving from develop to master ``` 首先用git reflog 命令，看看最近自己做过什么，哪里是自己想回去的地方。 ```git $ git reset --hard 72b075e $ git clean --f ``` 然后再使用 reset命令，复制自己想要去的地方的哈希码，穿越时光回到过去。顺便 clean 一下，保持清洁。 这样你就能去到任何你想去的地方，so happy。 但如何把多个提交合并成一个提交呢？为了保持代码树的干净漂亮，在本地的多次提交保存，弄成一次提交再推到远程去可能会更加好一点，所以可以是用 rebase 命令进行衍合。 在自己多次提交的本地分支上进行衍合一般不会出现冲突，找准本地分支的第一次提交的哈希码值，或者数清楚自己提交了几次用rebase命令就可以将多次提交合并成一次提交 ```git $ git rebase -i head~11 ``` 使用这个命令之后，会进入到vim模式，将自己需要的提交信息设置为p，其他的设置为s，最后使用：wq退出保存即可。 ```git # Rebase 6a2eb6d..f7d0bd7 onto 6a2eb6d # # Commands: # p, pick = use commit # r, reword = use commit, but edit the commit message # e, edit = use commit, but stop for amending # s, squash = use commit, but meld into previous commit # f, fixup = like "squash", but discard this commit's log message # x, exec = run command (the rest of the line) using shell # # These lines can be re-ordered; they are executed from top to bottom. # # If you remove a line here THAT COMMIT WILL BE LOST. # # However, if you remove everything, the rebase will be aborted. # # Note that empty commits are commented out ``` 这是对命令的提示。 但最好还是不要面对需要这样使用rebase的情景，应该好好使用commit 命令的参数。比如： ```git git commit --amend ``` 使用这个命令，会将本次提交写到上次的提交中去，就不会产生多条的提交信息。 ## git rebase 中的冲突处理 在合并代码的过程中，可以使用merge，也可以使用rebase，如果使用merge，合并之后推上远程就会有两条线，但使用rebase就只会产生一条线，变得简洁而优雅，所以在合并代码的过程中建议尽量使用rebase命令。 在合并代码中可能会遇到冲突，当提示有冲突时，我们可以使用外部的图形化冲突处理工具来处理冲突。这里使用的是KDiff3来处理冲突。 1. 安装Kdiff3 软件。（最好使用默认路径） 2. 添加kdiff3到git mergetool里。 git config --global merge.tool kdiff3 3. 添加kdiff3路径到 git global config里。git config --global mergetool.kdiff3.path "C:\Program Files\KDiff3\kdiff3.exe" 4. 以后merge发生冲突时：git mergetool 来做图形化merge。进入编辑冲突。 带问号的才是冲突。当两者都需要保存时，右击标记处，选完B后，再次右击，选择C。A是最原始的代码，B是自己的，C是别人的。 在这里需要注意的是，首先要设置好kDiff3的默认编码，和自己的工程编码一样，要不解决完冲突之后还要解决乱码问题。。。。 ================================================ FILE: docs/tool/gitstudy.md ================================================ # 30 分钟让你掌握 Git 的黑魔法 ## 担忧 很多人怕使用 git，我个人觉得主要可能是两部分的原因： * 没接触过：平时接触的代码还托管在 SVN 或 CVS 等工具上。 * 不太熟悉：可能对 git 的使用还不太熟悉和全面，导致在使用 git 时步步为营。 >Never Be Afraid To Try Something New. 代码对于开发者是劳作成果的结晶，对于公司而言是核心资产，有一些担忧也是正常的。但 git 也并没有我们想象中的那么复杂，让我们每次使用都心有余悸，其实我们只需要稍微花一点时间尝试多多了解它，在很多时候你会发现，非但 git 不会让你产生担忧，反而会让自己的交付过程更加高效。 ## Version Control 谈及 git 就不得不提到版本控制，我们不妨先来看下版本控制是做什么的，这将有助于后续对 git 的理解。 当你在工作中面对的是一些经常变化的文档、代码等交付物的时候，考虑如何去追踪和记录这些 changes 就变得非常重要，原因可能是： * 对于频繁改动和改进的交付物，非常有必要去记录下每次变更的内容，每次记录的内容汇成了一段修改的历史，有了历史我们才知道我们曾经做了什么。 * 记录的历史中必须要包含一些重要的信息，这样追溯才变得有意义，比如： - Who: 是谁执行的变更？ - When: 什么时候做出的变更？ - What: 这次变更做了什么事情？ * 最好可以支持撤销变更，不让某一个提交的严重问题，去污染整个提交历史。 版本控制系统（VCS: Version Control System），正会为你提供这种记录和追溯变更的能力。 ![git](./../img/gitstudy1.png) 大多数的 VCS 支持在多个使用者之间共享变更的提交历史，这从实质上让团队协同变为了可能，简单说来就是： * 你可以看到我的变更提交。 * 我也可以看到你的变更提交。 * 如果双方都进行了变更提交，也可以以某种方式方法进行比对和合并，最终作出统一的变更版本。 VCS 历经多年的发展，目前业界中有许多 VCS 工具可供我们选择。在本文中，我们将会针对目前最流行的 git 来介绍。 ## git是黑魔法么？ 刚接触 git 时，git 确实有让人觉得有点像黑魔法一样神秘，但是又有哪个技术不是这样呢？当我们了解其基本的数据结构后，会发现 git 从使用角度来讲其实并不复杂，我们甚至可以更进一步的学习 git 的一些先进的软件设计理论，从中获益。首先，让我们先从 commit 说起。 ### git commit object * 提交对象(git commit object): 每一个提交在 git 中都通过 git commit object 存储，对象具有一个全局唯一的名称，叫做 revision hash。它的名字是由 SHA-1 算法生成，形如 "998622294a6c520db718867354bf98348ae3c7e2"，我们通常会取其缩写方便使用，如 "9986222"。 * 对象构成: commit 对象包含了 author + commit message 的基本信息。 * 对象存储：git commit object 保存一次变更提交内的所有变更内容，而不是增量变化的数据 delta (很多人都理解错了这一点)，所以 git 对于每次改动存储的都是全部状态的数据。 * 大对象存储：因对于大文件的修改和存储，同样也是存储全部状态的数据，所以可能会影响 git 使用时的性能(glfs 可以改进这一点）。 * 提交树: 多个 commit 对象会组成一个提交树，它让我们可以轻松的追溯 commit 的历史，也能对比树上 commit 与 commit 之间的变更差异。 ### git commit 练习 让我们通过实战来帮助理解，第一步我们来初始化一个 repository（git仓库），默认初始化之后仓库是空的，其中既没有保存任何文本内容也没有附带任何提交： ```s $ git init hackers $ cd hackers $ git status ``` 第二步，让我们来看下执行过后 git 给出的输出内容，它会指引我们进行进一步的了解： ```s ➜ hackers git:(master) git status On branch master No commits yet nothing to commit (create/copy files anduse "git add" to track) ``` >output 1: On branch master * 对于刚刚创建空仓库来说，master 是我们的默认分支，一个 git 仓库下可以有很多分支(branches)，具体某一个分支的命名可以完全由你自己决定，通常会起便于理解的名字，如果用 hash 号的话肯定不是一个好主意。 * branches 是一种引用(ref)，他们指向了一个确定的 commit hash 号，这样我们就可以明确我们的分支当前的内容。 * 除了 branches 引用以外，还有一种引用叫做 tags，相信大家也不会陌生。 * master 通常被我们更加熟知，因为大多数的分支开发模式都是用 master 来指向“最新”的 commit。 * On branch master 代表着我们当前是在 master 分支下操作，所以每次当我们在提交新的 commit 时，git 会自动将 master 指向我们新的 commit，当工作在其他分支上时，同理。 * 有一个很特殊的 ref 名称叫做"HEAD"，它指向我们当前正在操作的 branches 或 tags (正常工作时)，其命名上非常容易理解，表示当前的引用状态。 * 通过 git branch(或 gittag) 命令你可以灵活的操作和修改 branches 或 tags。 >output 2: No commits yet 对于空仓库来说，目前我们还没有进行任意的提交。 >nothing to commit (create/copy files anduse "git add" to track) output 中提示我们需要使用 git add 命令，说到这里就必须要提到暂存或索引(stage)，那么如何去理解暂存呢？ 一个文件从改动到提交到 git 仓库，需要经历三个状态： * 工作区：工作区指的是我们本地工作的目录，比如我们可以在刚才创建的 hackers 目录下新增一个readme文件，readme 文件这时只是本地文件系统上的修改，还未存储到 git。 * 暂存(索引)区: 暂存实际上是将我们本地文件系统的改动转化为 git 的对象存储的过程。 * 仓库：git commit 后将提交对象存储到 git 仓库。 ![git](./../img/gitstudy2.png) git add 的帮助文档中很详细的解释了暂存这一过程： >DESCRIPTION This command updates the index using thecurrent content found in the working tree, to prepare the content stagedfor the next commit. （ git add 命令将更新暂存区，为接下来的提交做准备） It typically adds the current content ofexisting paths as a whole, but with some options it can also be used toadd content with only part of the changes made to the working tree filesapplied, or remove paths that do not exist in the working tree anymore. The "index" holds a snapshot ofthe content of the working tree, and it is this snapshot that is taken as thecontents of the next commit. （暂存区的 index 保存的是改动的完整文件和目录的快照(非delta)） Thus after making any changes to theworking tree, and before running the commit command, you must use the addcommand to add any new or modified files to the index. （暂存是我们将改动提交到 git 仓库之前必须经历的状态） 对 git 暂存有一定了解后，其相关操作的使用其实也非常简单，简要的说明如下： 1. 暂存区操作 * 通过 git add 命令将改动暂存； * 可以使用 git add -p 来依次暂存每一个文件改动，过程中我们可以灵活选择文件中的变更内容，从而决定哪些改动暂存； * 如果 git add 不会暂存被 ignore 的文件改动； * 通过 git rm 命令，我们可以删除文件的同时将其从暂存区中剔除； 2. 暂存区修正 * 通过 git reset 命令进行修正，可以先将暂存区的内容清空，在使用 git add -p 命令对改动 review 和暂存； * 这个过程不会对你的文件进行任何修改操作，只是 git 会认为目前没有改动需要被提交； * 如果我们想分阶段(or 分文件)进行 reset，可以使用 git reset FILE or git reset -p 命令； 3. 暂存区状态 * 可以用 git diff --staged 依次检查暂存区内每一个文件的修改； * 用 git diff 查看剩余的还未暂存内容的修改； 4. Just Commit！ * 当你对需要修改的内容和范围满意时，你就可以将暂存区的内容进行 commit 了，命令为 git commit； * 如果你觉得需要把所有当前工作空间的修改全部 commit，可以执行 git commit -a ，这相当于先执行 git add 后执行 git commit，将暂存和提交的指令合二为一，这对于一些开发者来说是很高效的，但是如果提交过大这样做通常不合适； * 我们建议一个提交中只做一件事，这在符合单一职责的同时，也可以让我们明确的知道每一个 commit 中做了一件什么事情而不是多个事情。所以通常我们的使用习惯都是执行 git add -p 来 review 我们将要暂存内容是否合理？是否需要更细的拆分提交？这些优秀的工程实践，将会让代码库中的 commits 更加优雅☕️； ok，我们已经在不知不觉中了解了很多内容，我们来回顾下，它们包括了： * commit 包含的信息是什么？ * commit 是如何表示的？ * 暂存区是什么？如何全部添加、一次添加、删除、查询和修正？ * 如何将暂存区的改动内容 commit？ * 不要做大提交，一个提交只做一件事； 附带的，在了解 commit 过程中我们知道了从本地改动到提交到 git 仓库，经历的几个关键的状态： * 工作区(Working Directory) * 暂存区(Index) * Git 仓库(Git Repo) 下图为上述过程中各个状态的转换过程： * 本地改动文件时，此时还仅仅是工作区内的改动； * 当执行 git add 之后，工作区内的改动被索引在暂存区； * 当执行 git commit 之后，暂存区的内容对象将会存储在 git 仓库中，并执行更新 HEAD 指向等后续操作，这样就完成了引用与提交、提交与改动快照的一一对应了。 ![git](./../img/gitstudy3.png) 正是因为 git 本身对于这几个区域(状态)的设计，为 git 在本地开发过程带来了灵活的管理空间。我们可以根据自己的情况，自由的选择哪些改动暂存、哪些暂存的改动可以 commit、commit 可以关联到那个引用，从而进一步与其他人进行协同。 ### 提交之后 我们已经有了一个 commit，现在我们可以围绕 commit 做更多有趣的事情： * 查看 commit 历史: git log(or git log --oneline)。 * 在 commit 中查看改动的 diff:git log -p。 * 查看 ref 与提交的关联关系，如当前 master 指向的 commit: git show master。 * 检出覆盖: git checkout NAME（如果NAME是一个具体的提交哈希值时，git 会认为状态是“detached(分离的)”，因为 gitcheckout 过程中重要的一步是将 HEAD 指向那个分支的最后一次 commit。所以如果这样做，将意味着没有分支在引用此提交，所以若我们这时候进行提交的话，没有人会知道它们的存在）。 * 使用 git revert NAME 来对 commit 进行反转操作。 * 使用 git diff NAME.. 将旧版本与当前版本进行比较，查看 diff。 * 使用 git log NAME, 查看指定区间的提交。 * 使用 git reset NAME 进行提交重置操作。 * 使用 git reset --hard NAME：将所有文件的状态强制重置为 NAME 的状态，使用上需要小心。 ### 引用基本操作 引用(refs)包含两种分别是 branches 和 tags， 我们接下来简单介绍下相关操作： * git branch b 命令可以让我们创建一个名称为 b 的分支； * 当我们创建了一个 b 分支后，这也相当于意味着 b 的指向就是 HEAD 对应的 commit； * 我们可以先在 b 分支上创建一个新的 commitA ，然后假如切回 master 分支上，这时再提交了一个新的 commitB，那么 master 和 HEAD 将会指向了新的commit __B，而 b 分支指向的还是原来的 commit A； * git checkout b 可以切换到 b 分支上,切换后新的提交都会在 b 分支上，理所应当； * git checkout master 切换回 master 后，b分支的提交也不会带回 master 上，分支隔离； 分支上提交隔离的设计，可以让我们非常轻松的切换我们的修改，非常方便的做各类测试。 tags 的名称不会改变，而且它们有自己的描述信息(比如可以作为 release note 以及标记发布的版本号等)。 ### 做好你的提交 可能很多人的提交历史是长这个样子的： ``` s commit 14: add feature x – maybe even witha commit message about x! commit 13: forgot to add file commit 12: fix bug commit 11: typo commit 10: typo2 commit 9: actually fix commit 8: actually actually fix commit 7: tests pass commit 6: fix example code commit 5: typo commit 4: x commit 3: x commit 2: x commit 1: x ``` 单就 git 而言，这看上去是没有问题而且合法的，但对于那些对你修改感兴趣的人（很可能是未来的你!），这样的提交在信息在追溯历史时可能并没有多大帮助。但是如果你的提交已经长成这个样子，我们该怎么办？ 没关系，git 有办法可以弥补这一些： ### git commit --amend 我们可以将新的改动提交到当前最近的提交上，比如你发现少改了什么，但是又不想多出一个提交时会很有用。 如果我们认为我们的提交信息写的并不好，我要修改修改，这也是一种办法，但是并不是最好的办法。 这个操作会更改先前的提交，并为其提供新的hash值。 ### git rebase -i HEAD~13 这个命令非常强大，可以说是 git 提交管理的神器，此命令含义是我们可以针对之前的 13 次的提交在 VI 环境中进行重新修改设计： * 操作选项 p 意味着保持原样什么都不做，我们可以通过 vim 中编辑提交的顺序，使其在提交树上生效； * 操作选项 r: 我们可以修改提交信息，这种方式比 commit --amend 要好的多，因为不会新生成一个 commit； * 操作选项 e: 我们可以修改 commit，比如新增或者删除某些文件改动； * 操作选项 s: 我们可以将这个提交与其上一次的提交进行合并，并重新编辑提交信息； * 操作选项 f: f 代表着"fixup"。例如我们如果想针对之前一个老的提交进行 fixup，又不想做一次新的提交破坏提交树的历史的逻辑含义，可以采用这种方式，这种处理方式非常优雅； ### 关于git 版本控制的一个常见功能是允许多个人对一组文件进行更改，而不会互相影响。或者更确切地说，为了确保如果他们不会踩到彼此的脚趾，不会在提交代码到服务端时偷偷的覆盖彼此的变化。 在 git 中我们如何保证这一点呢？ git 与svn 不同，git 不存在本地文件 lock 的情况，这是一种避免出现写作问题的方式，但是并不方便，而 git 与 svn 最大的不同在于它是一个分布式 VCS，这意味着： * 每个人都有整个存储库的本地副本（其中不仅包含了自己的，也包含了其他人的提交到仓库的所有内容）。 * 一些 VCS 是集中式的（例如，svn）：服务器具有所有提交，而客户端只有他们“已检出”的文件。所以基本上在本地我们只有当前文件，每次涉及本地不存在的文件操作时，都需要访问服务端进行进一步交互。 * 每一个本地副本都可以当作服务端对外提供 git 服务。 * 我们可以用 git push 推送本地内容到任意我们有权限的 git 远端仓库。 * 不管是集团的 force、github、gitlab 等工具，其实本质上都是提供的 git 仓库存储的相关服务，在这一点上其实并没有特别之处，针对 git 本身和其协议上是透明的。 ![git](./../img/gitstudy4.png) ![git](./../img/gitstudy5.png) ### git冲突解决 冲突的产生几乎是不可避免的，当冲突产生时你需要将一个分支中的更改与另一个分支中的更改合并，对应 git 的命令为 git merge NAME ，一般过程如下： * 找到 HEAD 和 NAME 的一个共同祖先(common base)； * 尝试将这些 NAME 到共同祖先之间的修改合并到 HEAD 上； * 新创建一个 merge commit 对象，包含所有的这些变更内容； * HEAD 指向这个新的 mergecommit； git 将会保证这个过程改动不会丢失，另外一个命令你可能会比较熟悉，那就是 git pull 命令，git pull 命令实际上包含了 git merge 的过程，具体过程为： * git fetch REMOTE * git merge REMOTE/BRANCH * 和 git push一样，有的时候需要先设置 "tracking"(-u) ，这样可以将本地和远程的分支一一对应。 如果每次 merge 都如此顺利，那肯定是非常完美的，但有时候你会发现在合并时产生了冲突文件，这时候也不用担心，如何处理冲突的简要介绍如下： * 冲突只是因为 git 不清楚你最终要合并后的文本是什么样子，这是很正常的情况； * 产生冲突时，git 会中断合并操作，并指导你解决好所有的冲突文件； * 打开你的冲突文件，找到 <<<<<<< ，这是你需要开始处理冲突的地方，然后找到=======，等号上面的内容是 HEAD 到共同祖先之间的改动，等号下面是 NAME 到共同祖先之间的改动。用 git mergetool 通常是比较好的选择，当然现在大多数 IDE 都集成了不错的冲突解决工具； * 当你把冲突全部解决完毕，请用 git add . 来暂存这些改动； * 最后进行 git commit，如果你想放弃当前修改重新解决可以使用 git merge --abort ，非常方便； * 当你完成了以上这些艰巨的任务，最后 git push 吧！ ### push失败？ 排除掉远端的 git 服务存在问题以外，我们 push 失败的大多数原因都是因为我们在工作的内容其他人也在工作的关系。 Git 是这样判断的： 1. 会判断 REMOTE 的当前 commit 是不是你当前正在 pushing commit 的祖先。 2. 如果是的话，代表你的提交是相对比较新的，push 是可以成功的(fast-forwarding)。 3. 否则 push 失败并提示你其他人已经在你 push 之前执行更新(push is rejected)。 当发生push is rejected 后我们的几个处理方法如下: * 使用 git pull 合并远程的最新更改（git pull相当于 git fetch + git merge） ； * 使用 --force 强制推送本地变化到远端饮用进行覆盖，需要注意的是 这种覆盖操作可能会丢失其他人的提交内容； * 可以使用 --force-with-lease 参数，这样只有远端的 ref 自上次从 fetch 后没有改变时才会强制进行更改，否则 reject the push，这样的操作更安全，是一种非常推荐使用的方式； * 如果 rebase 操作了本地的一些提交，而这些提交之前已经 push 过了的话，你可能需要进行 force push 了，可以想象看为什么？ 本文只是选取部分 Git 基本命令进行介绍，目的是抛砖引玉，让大家对 git 有一个基本的认识。当我们深入挖掘 Git 时，你会发现它本身有着如此多优秀的设计理念，值得我们学习和探究。 **不要让 Git 成为你认知领域的黑魔法，而是让 Git 成为你掌握的魔法。** [参考英文](https://hacker-tools.github.io/version-control/) ================================================ FILE: docs/tool/gitusual.md ================================================ # 10 大 Git 命令 动画展示 >git merge、git rebase、git reset、git revert、git fetch、git pull、git reflog 你知道这些 git 命令执行的究竟是什么任务吗？如果你还有些分不清楚，那千万不能错过这篇文章。 ## 合并 拥有多个分支是很方便的，这样可以将不同的新修改互相隔离开，而且还能确保你不会意外地向生产代码推送未经许可或破损的代码修改。但一旦这些修改得到了批准许可，我们就需要将其部署到我们的生产分支中！ 可将一个分支的修改融入到另一个分支的一种方式是执行 git merge。Git 可执行两种类型的合并：fast-forward 和 no-fast-forward。现在你可能分不清，但我们马上就来看看它们的差异所在。 ### Fast-forward (—ff) 在当前分支相比于我们要合并的分支没有额外的提交（commit）时，可以执行 fast-forward 合并。Git 很懒，首先会尝试执行最简单的选项：fast-forward！这类合并不会创建新的提交，而是会将我们正在合并的分支上的提交直接合并到当前分支。 ![gitusual](../img/gitusual/gitusual-1.gif) 完美！现在，我们在 dev 分支上所做的所有改变都合并到了 master 分支上。那么 no-fast-forward 又是什么意思呢？ ### No-fast-foward (—no-ff) 如果你的当前分支相比于你想要合并的分支没有任何提交，那当然很好，但很遗憾现实情况很少如此！如果我们在当前分支上提交我们想要合并的分支不具备的改变，那么 git 将会执行 no-fast-forward 合并。 使用 no-fast-forward 合并时，Git 会在当前活动分支上创建新的 merging commit。这个提交的父提交（parent commit）即指向这个活动分支，也指向我们想要合并的分支！ ![gitusual](../img/gitusual/gitusual-2.gif) 没什么大不了的，完美的合并！现在，我们在 dev 分支上所做的所有改变都合并到了 master 分支上。 ## 合并冲突 尽管 Git 能够很好地决定如何合并分支以及如何向文件添加修改，但它并不总是能完全自己做决定。当我们想要合并的两个分支的同一文件中的同一行代码上有不同的修改，或者一个分支删除了一个文件而另一个分支修改了这个文件时，Git 就不知道如何取舍了。 在这样的情况下，Git 会询问你想要保留哪种选择？假设在这两个分支中，我们都编辑了 README.md 的第一行。 ![gitusual](../img/gitusual/gitusual-3.gif) 如果我们想把 dev 合并到 master，就会出现一个合并冲突：你想要标题是 Hello! 还是 Hey!？ 当尝试合并这些分支时，Git 会向你展示冲突出现的位置。我们可以手动移除我们不想保留的修改，保存这些修改，再次添加这个已修改的文件，然后提交这些修改。 ![gitusual](../img/gitusual/gitusual-4.gif) 完成！尽管合并冲突往往很让人厌烦，但这是合理的：Git 不应该瞎猜我们想要保留哪些修改。 ## 变基（Rebasing） 我们刚看到可通过执行 git merge 将一个分支的修改应用到另一个分支。另一种可将一个分支的修改融入到另一个分支的方式是执行 git rebase。 git rebase 会将当前分支的提交复制到指定的分支之上。 ![gitusual](../img/gitusual/gitusual-5.gif) 完美，现在我们在 dev 分支上获取了 master 分支上的所有修改。 变基与合并有一个重大的区别：Git 不会尝试确定要保留或不保留哪些文件。我们执行 rebase 的分支总是含有我们想要保留的最新近的修改！这样我们不会遇到任何合并冲突，而且可以保留一个漂亮的、线性的 Git 历史记录。 上面这个例子展示了在 master 分支上的变基。但是，在更大型的项目中，你通常不需要这样的操作。git rebase 在为复制的提交创建新的 hash 时会修改项目的历史记录。 如果你在开发一个 feature 分支并且 master 分支已经更新过，那么变基就很好用。你可以在你的分支上获取所有更新，这能防止未来出现合并冲突。 ## 交互式变基（Interactive Rebase） 在为提交执行变基之前，我们可以修改它们！我们可以使用交互式变基来完成这一任务。交互式变基在你当前开发的分支上以及想要修改某些提交时会很有用。 在我们正在 rebase 的提交上，我们可以执行以下 6 个动作： * reword：修改提交信息； * edit：修改此提交； * squash：将提交融合到前一个提交中； * fixup：将提交融合到前一个提交中，不保留该提交的日志消息； * exec：在每个提交上运行我们想要 rebase 的命令； * drop：移除该提交。 很棒！这样我们就能完全控制我们的提交了。如果你想要移除一个提交，只需 drop 即可。 ![gitusual](../img/gitusual/gitusual-6.gif) 如果你想把多个提交融合到一起以便得到清晰的提交历史，那也没有问题！ ![gitusual](../img/gitusual/gitusual-7.gif) 交互式变基能为你在 rebase 时提供大量控制，甚至可以控制当前的活动分支。 ## 重置（Resetting） 当我们不想要之前提交的修改时，就会用到这个命令。也许这是一个 WIP 提交或者可能是引入了 bug 的提交，这时候就要执行 git reset。 git reset 能让我们不再使用当前台面上的文件，让我们可以控制 HEAD 应该指向的位置。 ## 软重置 软重置会将 HEAD 移至指定的提交（或与 HEAD 相比的提交的索引），而不会移除该提交之后加入的修改！ 假设我们不想保留添加了一个 style.css 文件的提交 9e78i，而且我们也不想保留添加了一个 index.js 文件的提交 035cc。但是，我们确实又想要保留新添加的 style.css 和 index.js 文件！这是软重置的一个完美用例。 ![gitusual](../img/gitusual/gitusual-8.gif) 输入 git status 后，你会看到我们仍然可以访问在之前的提交上做过的所有修改。这很好，这意味着我们可以修复这些文件的内容，之后再重新提交它们！ ## 硬重置 有时候我们并不想保留特定提交引入的修改。不同于软重置，我们应该再也无需访问它们。Git 应该直接将整体状态直接重置到特定提交之前的状态：这甚至包括你在工作目录中和暂存文件上的修改。 ![gitusual](../img/gitusual/gitusual-9.gif) Git 丢弃了 9e78i 和 035cc 引入的修改，并将状态重置到了 ec5be 的状态。 ## 还原（Reverting） 另一种撤销修改的方法是执行 git revert。通过对特定的提交执行还原操作，我们会创建一个包含已还原修改的新提交。 假设 ec5be 添加了一个 index.js 文件。但之后我们发现其实我们再也不需要由这个提交引入的修改了。那就还原 ec5be 提交吧！ ![gitusual](../img/gitusual/gitusual-10.gif) 完美！提交 9e78i 还原了由提交 ec5be 引入的修改。在撤销特定的提交时，git revert 非常有用，同时也不会修改分支的历史。 ## 拣选（Cherry-picking） 当一个特定分支包含我们的活动分支需要的某个提交时，我们对那个提交执行 cherry-pick！对一个提交执行 cherry-pick 时，我们会在活动分支上创建一个新的提交，其中包含由拣选出来的提交所引入的修改。 假设 dev 分支上的提交 76d12 为 index.js 文件添加了一项修改，而我们希望将其整合到 master 分支中。我们并不想要整个 dev 分支，而只需要这个提交！ ![gitusual](../img/gitusual/gitusual-11.gif) 现在 master 分支包含 76d12 引入的修改了。 ## 取回（Fetching） 如果你有一个远程 Git 分支，比如在 GitHub 上的分支，当远程分支上包含当前分支没有的提交时，可以使用取回。比如当合并了另一个分支或你的同事推送了一个快速修复时。 通过在这个远程分支上执行 git fetch，我们就可在本地获取这些修改。这不会以任何方式影响你的本地分支：fetch 只是单纯地下载新的数据而已。 ![gitusual](../img/gitusual/gitusual-12.gif) 现在我们可以看到自上次推送以来的所有修改了。这些新数据也已经在本地了，我们可以决定用这些新数据做什么了。 ## 拉取（Pulling） 尽管 git fetch 可用于获取某个分支的远程信息，但我们也可以执行 git pull。git pull 实际上是两个命令合成了一个：git fetch 和 git merge。当我们从来源拉取修改时，我们首先是像 git fetch 那样取回所有数据，然后最新的修改会自动合并到本地分支中。 ![gitusual](../img/gitusual/gitusual-13.gif) 很好，我们现在与远程分支完美同步了，并且也有了所有最新的修改！ ## Reflog 每个人都会犯错，但犯错其实没啥！有时候你可能感觉你把 git repo 完全搞坏了，让你想完全删了了事。 git reflog 是一个非常有用的命令，可以展示已经执行过的所有动作的日志。包括合并、重置、还原，基本上包含你对你的分支所做的任何修改。 ![gitusual](../img/gitusual/gitusual-14.gif) 如果你犯了错，你可以根据 reflog 提供的信息通过重置 HEAD 来轻松地重做！ 假设我们实际上并不需要合并原有分支。当我们执行 git reflog 命令时，我们可以看到这个 repo 的状态在合并前位于 HEAD@{1}。那我们就执行一次 git reset，将 HEAD 重新指向在 HEAD@{1} 的位置。 ![gitusual](../img/gitusual/gitusual-15.gif) 我们可以看到最新的动作已被推送给 reflog。 ================================================ FILE: docs/tool/markdown.md ================================================ ## 主要内容 > #### Markdown*是什么*？ > #### *谁*创造了它？ > #### *为什么*要使用它？ > #### *怎么*使用？ > #### *谁*在用？ > #### 尝试一下 ## 正文 ### 1. Markdown*是什么*？ **Markdown**是一种轻量级**标记语言**，它以纯文本形式(*易读、易写、易更改*)编写文档，并最终以HTML格式发布。 **Markdown**也可以理解为将以MARKDOWN语法编写的语言转换成HTML内容的工具。 ### 2. *谁*创造了它？ 它由[**Aaron Swartz**](http://www.aaronsw.com/)和**John Gruber**共同设计，**Aaron Swartz**就是那位于去年（*2013年1月11日*）自杀,有着**开挂**一般人生经历的程序员。维基百科对他的[介绍](http://zh.wikipedia.org/wiki/%E4%BA%9A%E4%BC%A6%C2%B7%E6%96%AF%E6%B2%83%E8%8C%A8)是：**软件工程师、作家、政治组织者、互联网活动家、维基百科人**。 他有着足以让你跪拜的人生经历： + **14岁**参与RSS 1.0规格标准的制订。 + **2004**年入读**斯坦福**，之后退学。 + **2005**年创建[Infogami](http://infogami.org/)，之后与[Reddit](http://www.reddit.com/)合并成为其合伙人。 + **2010**年创立求进会（Demand Progress），积极参与禁止网络盗版法案（SOPA）活动，最终该提案被撤回。 + **2011**年7月19日，因被控从MIT和JSTOR下载480万篇学术论文并以免费形式上传于网络被捕。 + **2013**年1月自杀身亡。 ![Aaron Swartz](https://github.com/younghz/Markdown/raw/master/resource/Aaron_Swartz.jpg) 天才都有早逝的归途。 ### 3. *为什么*要使用它？ + 它是易读（看起来舒服）、易写（语法简单）、易更改**纯文本**。处处体现着**极简主义**的影子。 + 兼容HTML，可以转换为HTML格式发布。 + 跨平台使用。 + 越来越多的网站支持Markdown。 + 更方便清晰地组织你的电子邮件。（Markdown-here, Airmail） + 摆脱Word（我不是认真的）。 ### 4. *怎么*使用？ 如果不算**扩展**，Markdown的语法绝对**简单**到让你爱不释手。 Markdown语法主要分为如下几大部分： **标题**，**段落**，**区块引用**，**代码区块**，**强调**，**列表**，**分割线**，**链接**，**图片**，**反斜杠 `\`**，**符号'`'**。 #### 4.1 标题 两种形式： 1）使用`=`和`-`标记一级和二级标题。 > 一级标题 > `=========` > 二级标题 > `---------` 效果： > 一级标题 > ========= > 二级标题 > --------- 2）使用`#`，可表示1-6级标题。 > \# 一级标题 > \## 二级标题 > \### 三级标题 > \#### 四级标题 > \##### 五级标题 > \###### 六级标题 效果： > # 一级标题 > ## 二级标题 > ### 三级标题 > #### 四级标题 > ##### 五级标题 > ###### 六级标题 #### 4.2 段落 段落的前后要有空行，所谓的空行是指没有文字内容。若想在段内强制换行的方式是使用**两个以上**空格加上回车（引用中换行省略回车）。 #### 4.3 区块引用 在段落的每行或者只在第一行使用符号`>`,还可使用多个嵌套引用，如： > \> 区块引用 > \>> 嵌套引用 效果： > 区块引用 >> 嵌套引用 #### 4.4 代码区块 代码区块的建立是在每行加上4个空格或者一个制表符（如同写代码一样）。如 普通段落： void main() { printf("Hello, Markdown."); } 代码区块： void main() { printf("Hello, Markdown."); } **注意**:需要和普通段落之间存在空行。 #### 4.5 强调 在强调内容两侧分别加上`*`或者`_`，如： > \*斜体\*，\_斜体\_ > \*\*粗体\*\*，\_\_粗体\_\_ 效果： > *斜体*，_斜体_ > **粗体**，__粗体__ #### 4.6 列表 使用`·`、`+`、或`-`标记无序列表，如： > \-（+\*） 第一项 > \-（+\*） 第二项 > \- （+\*）第三项 **注意**：标记后面最少有一个_空格_或_制表符_。若不在引用区块中，必须和前方段落之间存在空行。 效果： > + 第一项 > + 第二项 > + 第三项 有序列表的标记方式是将上述的符号换成数字,并辅以`.`，如： > 1 . 第一项 > 2 . 第二项 > 3 . 第三项 效果： > 1. 第一项 > 2. 第二项 > 3. 第三项 #### 4.7 分割线 分割线最常使用就是三个或以上`*`，还可以使用`-`和`_`。 #### 4.8 链接 链接可以由两种形式生成：**行内式**和**参考式**。 **行内式**： > \[burningmyself的Markdown库\]\(https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown"\)。 效果： > [burningmyself的Markdown库](https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown")。 **参考式**： > \[burningmyself的Markdown库1\]\[1\] > \[burningmyself库2\]\[2\] > \[1\]:https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown" > \[2\]:https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown" 效果： > [burningmyself的Markdown库1][1] > [burningmyself的Markdown库2][2] [1]: https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown" [2]: https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown" **注意**：上述的`[1]:https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown"`不出现在区块中。 #### 4.9 图片 添加图片的形式和链接相似，只需在链接的基础上前方加一个`！`。 #### 4.10 反斜杠`\` 相当于**反转义**作用。使符号成为普通符号。 #### 4.11 符号'`' 起到标记作用。如： >\`ctrl+a\` 效果： >`ctrl+a` #### 5. *谁*在用？ Markdown的使用者： + GitHub + 简书 + Stack Overflow + Apollo + Moodle + Reddit + 等等 #### 6. 尝试一下 + **Chrome**下的插件诸如`stackedit`与`markdown-here`等非常方便，也不用担心平台受限。 + **在线**的dillinger.io评价也不错 + **Windowns**下的MarkdownPad也用过，不过免费版的体验不是很好。 + **Mac**下的Mou是国人贡献的，口碑很好。 + **Linux**下的ReText不错。 **当然，最终境界永远都是笔下是语法，心中格式化 :)。** **** **注意**：不同的Markdown解释器或工具对相应语法（扩展语法）的解释效果不尽相同，具体可参见工具的使用说明。 虽然有人想出面搞一个所谓的标准化的Markdown，[没想到还惹怒了健在的创始人John Gruber] (http://blog.codinghorror.com/standard-markdown-is-now-common-markdown/ )。 **** 以上基本是所有traditonal markdown的语法。 ### 其它： 列表的使用(非traditonal markdown)： 用`|`表示表格纵向边界，表头和表内容用`-`隔开，并可用`:`进行对齐设置，两边都有`:`则表示居中，若不加`:`则默认左对齐。 |代码库 |链接 | |:------------------------------------:|------------------------------------| |MarkDown |[https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md](https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown")| |MarkDownCopy |[https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md](https://github.com/burningmyself/blog/tree/master/docs/tool/markdown.md "Markdown")| 关于其它扩展语法可参见具体工具的使用说明。 ================================================ FILE: docs/tool/minio.md ================================================ # MinIO 搭建使用 ## MinIO简介 MinIO 是一款基于Go语言的高性能对象存储服务，在Github上已有19K+Star。它采用了Apache License v2.0开源协议，非常适合于存储大容量非结构化的数据，例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等。 本文将使用 MinIO 来自建一个对象存储服务用于存储图片。 ## 安装及部署 > MinIO的安装方式有很多，这里我们使用它在Docker环境下的安装方式。 * 下载MinIO的Docker镜像： ``` shell docker pull minio/minio ``` * 在Docker容器中运行MinIO，这里我们将MiniIO的数据和配置文件夹挂在到宿主机上： ``` shell docker run -p 9000:9000 --name minio \ --restart=always \ -v /etc/localtime:/etc/localtime \ -v /data/minio/data:/data \ -v /data/minio/config:/root/.minio \ -d minio/minio server /data ``` * 运行成功后，访问该地址来登录并使用MinIO，默认Access Key和Secret都是minioadmin：http://localhost:9000 ![minio](./../img/minio1.png) ## 上传文件及使用 >通过使用MinIO的网页端即可完成文件的上传下载功能，下面我们以图片上传下载为例来演示下该功能。 * 在存储文件之前，我们需要新建一个存储桶： ![minio](./../img/minio2.png) * 存储桶创建完成后，通过上传按钮可以上传文件，这里我们上传一张图片： ![minio](./../img/minio3.png) * 图片上传完成后，我们可以通过拷贝链接按钮来获取图片访问路径，但是这只是个临时的访问路径： ![minio](./../img/minio4.png) * 要想获取一个永久的访问路径，需要修改存储桶的访问策略，我们可以点击存储桶右上角的编辑策略按钮来修改访问策略； ![minio](./../img/minio5.png) * 这里有三种访问策略可以选择，一种只读、一种只写、一种可读可写，这里我们选择只读即可，但是需要注意的是，访问前缀需要设置为*.*，否则会无法访问； ![minio](./../img/minio6.png) * 设置完成后，我们只需要通过拷贝链接中的前一串路径即可永久访问该文件； ## MinIO客户端的使用 >虽然MinIO的网页端管理已经很方便了，但是官网还是给我们提供了基于命令行的客户端MinIO Client(简称mc)，下面我们来讲讲它的使用方法。 ### 常用命令 > 下面我们先来熟悉下mc的命令，这些命令和Linux中的命令有很多相似之处。 | 命令 | 作用 | | ------- | --------------------------------------------- | | ls | 列出文件和文件夹 | | mb | 创建一个存储桶或一个文件夹 | | cat | 显示文件和对象内容 | | pipe | 将一个STDIN重定向到一个对象或者文件或者STDOUT | | share | 生成用于共享的URL | | cp | 拷贝文件和对象 | | mirror | 给存储桶和文件夹做镜像 | | find | 基于参数查找文件 | | diff | 对两个文件夹或者存储桶比较差异 | | rm | 删除文件和对象 | | events | 管理对象通知 | | watch | 监听文件和对象的事件 | | policy | 管理访问策略 | | session | 为cp命令管理保存的会话 | | config | 管理mc配置文件 | | update | 检查软件更新 | | version | 输出版本信息 | ### 安装及配置 > 由于MinIO服务端中并没有自带客户端，所以我们需要安装配置完客户端后才能使用，这里以Docker环境下的安装为例。 - 下载MinIO Client 的Docker镜像： ```bash docker pull minio/mc ``` - 在Docker容器中运行mc： ```bash docker run -it --entrypoint=/bin/sh minio/mc ``` - 运行完成后我们需要进行配置，将我们自己的MinIO服务配置到客户端上去，配置的格式如下： ```bash mc config host add ``` - 对于我们的MinIO服务可以这样配置： ```bash mc config host add minio http://localhost:9000 minioadmin minioadmin S3v4 ``` ### 常用操作 - 查看存储桶和查看存储桶中存在的文件： ```bash # 查看存储桶 mc ls minio # 查看存储桶中存在的文件 mc ls minio/blog ``` ![minio](./../img/minio7.png) - 创建一个名为`test`的存储桶： ```bash mc mb minio/test ``` - 共享`avatar.png`文件的下载路径： ```bash mc share download minio/blog/avatar.png ``` - 查找`blog`存储桶中的png文件： ```bash mc find minio/blog --name "*.png" ``` - 设置`test`存储桶的访问权限为`只读`： ```bash # 目前可以设置这四种权限：none, download, upload, public mc policy set download minio/test/ # 查看存储桶当前权限 mc policy list minio/test/ ``` ## 参考资料 详细了解MinIO可以参考官方文档：https://docs.min.io/cn/minio-quickstart-guide.html ================================================ FILE: docs/tool/mkdocs.md ================================================ # mkdocs简单使用 [官网](http://www.mkdocs.org/) ## 一、安装 ``` python # 查看 python 版本 python --version # Python 2.7.2 # 查看 pip 版本 pip --version # pip 1.5.2 # 更新 pip pip install --upgrade pip # 安装 mkdocs pip install mkdocs pip install --upgrade mkdocs # 查看 mkdocs 版本 mkdocs --version # mkdocs, version 0.17.3 # 查看帮助 mkdocs --help # 查看具体命令的帮助 mkdocs build --help # get started mkdocs new my-project cd my-project # 实时测试 mkdocs serve -a 0.0.0.0:8000 # 打包成静态文件 mkdocs build --clean # 打包到特定文件夹 mkdocs build --clean -d ./myFiles ``` ## 二、配置文件 默认使用的配置文件是 ./mkdocs.yml 使用的目录是 ./docs/ ./mkdocs.yml 配置文件的一般内容是: ``` yml site_name: webSite_Title pages: - index.md - about.md theme: readthedocs ``` 如果不配置 pages , 那么 mkdocs 会自动遍历检索 ./docs 目录中的所有 md 文档，并生成 pages 内容,但是是按照字母顺序排序的，如果需要，还是手动设置 pages 内容 pages 使用 HOME 作为 index.md 的标题 , About 作为 about.md 的标题 并且使用 index.md 作为显示首页, pages 中没有配置 index.md 首页会无法显示 所以一般的配置都是 ``` yml pages: - Home: index.md - User Guide: - 'Writing your docs': 'user-guide/writing-your-docs.md' - 'Styling your docs': 'user-guide/styling-your-docs.md' - About: - 'License': 'about/license.md' - 'Release Notes': 'about/release-notes.md' ``` mkdocs 包含了两个内置的 theme: mkdocs 和 readthedocs mkdocs 是默认的theme 当然你也可以创建自己的theme,或使用第三方theme，例如 ``` python pip install windmill pip install --upgrade windmill # 然后指定 theme: windmill ``` docs_dir 可以让你指定 docs/ 目录的位置 site_dir 可以让你指定 build后的输出目录位置 dev_addr 可以让你指定 serve 时监听的 IP:port 还有 plugins ## 三、简单配置 ``` python #!/bin/bash # ================ 基础功能区 ================ scriptDir=$(dirname $(readlink -f $0)) scriptName=$(basename $0) # ================ 函数定义区 ================ # 获取git项目名称映射的中文名称,retCode=155表示映射到了 function getMapNameForGitProjectName { # 第一个参数是 git项目名称 local gitProjectName="$1" # 获取git项目名称为 xxx.git local tmpBaseName=$(basename ${gitProjectName}) # 去掉 .git local tmpOriProjName=${tmpBaseName%.*} # 声明使用的映射文件 local tmpMappingFile="${scriptDir}/mapForProject.txt" if (! [ -e ${tmpMappingFile} ]);then echo ${tmpOriProjName} return 0 fi # 查找 git项目名称:yyy 中冒号后面的部分 local tmpMappedName=$(grep -oP "(?<=${tmpOriProjName}:).*$" ${tmpMappingFile}) if ([ -n "${tmpMappedName}" ]);then echo ${tmpMappedName} return 155 else echo ${tmpOriProjName} fi } # 遍历 映射文件, 匹配nginx根目录中的文件夹,生成连接 function doAddLinkForProjectInMap { # 第一个参数是 nginx根目录 local nginxRootDir="$1" # 第二个参数是 index.html ，表示要追加内容的文件 local indexFile="$2" if ([ -z "${nginxRootDir}" ]);then echo "为doAddLinkForProjectInMap函数提供的参数错误" return 1 fi if ([ -z "${indexFile}" ]);then echo "为doAddLinkForProjectInMap函数提供的参数错误" return 1 fi # 声明使用的映射文件 local tmpMappingFile="${scriptDir}/mapForProject.txt" if (! [ -e ${tmpMappingFile} ]);then echo "映射文件不存在" return 1 fi # 追加 ul 到 indexFile cat >> ${indexFile} < EOF for line in `cat ${tmpMappingFile}`;do # 获取每一行冒号前面的部分 tmpDirName=${line%%:*} # 匹配 $string的后缀 的最长匹配,然后删除,支持正则表达式 tmpMapName=${line#*:} # 匹配 $string的前缀 的最短匹配,然后删除,支持正则表达式 if ([ -z "${tmpDirName}" ]);then continue fi if ([ -e "${nginxRootDir}/${tmpDirName}" ]);then # 如果存在该文件夹 echo "
  • ${tmpMapName}
    • " >> ${indexFile} fi done # 追加 /ul cat >> ${nginxDir}/index.html < EOF echo "按顺序追加映射文件内的文件夹完成" return 0 } # 追加文件夹内容到 mkdocs.yml function appendPagesContent { local dstDir=$1; local tmpDir=$2; local indent="$3"; local tmpFile; local tmpRelativeFilePath; local tmpBaseName; local tmpBaseNameWithoutExtension; local exitCode=0; # 返回 155 表示成功执行,且插入了内容,返回0表示成功执行 # 先增加 index tmpFile="${tmpDir}/index.md" if ([ -f ${tmpFile} ]);then tmpRelativeFilePath=${tmpFile#${dstDir}/docs/} tmpBaseName=$(basename ${tmpFile}) echo "${indent}- Home: ${tmpRelativeFilePath}" >> ${dstDir}/mkdocs.yml # 成功插入了内容,设置exitCode exitCode=155; fi # 将文件夹下面的md文档和文件夹加入到 mkdocs.yml for tmpFile in ${tmpDir}/*;do tmpRelativeFilePath=${tmpFile#${dstDir}/docs/} tmpBaseName=$(basename ${tmpFile}) tmpBaseNameWithoutExtension=${tmpBaseName%.*} if ([ -f "${tmpFile}" ] && [[ $(echo ${tmpFile} | tr 'A-Z' 'a-z') == *.md ]]);then if ([ ${tmpBaseNameWithoutExtension} = index ]);then continue; else echo "${indent}- ${tmpBaseNameWithoutExtension}: ${tmpRelativeFilePath}" >> ${dstDir}/mkdocs.yml # 成功插入了内容,设置exitCode exitCode=155; fi fi if ([ -d "${tmpFile}" ]);then echo "${indent}- ${tmpBaseNameWithoutExtension}:" >> ${dstDir}/mkdocs.yml appendPagesContent ${dstDir} ${tmpFile} "${indent} " local tmpReturnCode=$? if ([ ${tmpReturnCode} -ne 155 ]);then # 没有追加任何内容 # 删除已追加的最后一行 sed -i '$d' ${dstDir}/mkdocs.yml else # 成功插入了内容,设置exitCode exitCode=155; fi fi done # 将文件夹下面的非md文档放到一个html文件中展示 # 生成一个临时文件 tmpOtherFileMd=`mktemp` for tmpFile in ${tmpDir}/*;do tmpRelativeFilePath=${tmpFile#${dstDir}/docs/} tmpBaseName=$(basename ${tmpFile}) if ([ -f "${tmpFile}" ] \ && ! [[ $(echo ${tmpFile} | tr 'A-Z' 'a-z') == *.md ]] \ && ! [[ $(echo ${tmpFile} | tr 'A-Z' 'a-z') == *.png ]] \ && ! [[ $(echo ${tmpFile} | tr 'A-Z' 'a-z') == *.jpg ]] \ );then echo "###### [${tmpBaseName}](./${tmpBaseName})" >> ${tmpOtherFileMd} fi done if ([ -n "$(cat ${tmpOtherFileMd})" ]);then # 将临时生成的md文件拷贝到目标文件夹 tmpFile="${tmpDir}/其他文档(自动生成).md" tmpRelativeFilePath=${tmpFile#${dstDir}/docs/} tmpBaseName=$(basename ${tmpFile}) tmpBaseNameWithoutExtension=${tmpBaseName%.*} mv ${tmpOtherFileMd} "${tmpFile}" # 将md文件添加到mkdocs echo "${indent}- ${tmpBaseNameWithoutExtension}: ${tmpRelativeFilePath}" >> ${dstDir}/mkdocs.yml # 成功插入了内容,设置exitCode exitCode=155; fi # 返回指定状态码 return ${exitCode}; } # ================ 默认参数定义区 ================ gitusername='globalreader' gitpasswd='12345qwert' branch='master' # 定义使用帮助说明 helpMsg=" 所有参数不区分大小写 --gituri docs项目git地址 " # ================ 处理选项 ================ while [ -n "$1" ];do tmpOpt=$(echo "$1"|tr A-Z a-z) # 判断参数是否是 --help , 是则显示帮助并退出 if ([ $tmpOpt = "--help" ]);then echo "${helpMsg}" exit 0 fi # 判断参数是否是 --bool开头,如果是则将变量置为1 并 shift 1 if ([[ $tmpOpt = --bool* ]]);then tmpOpt=${tmpOpt#--} echo "检测到参数 ${tmpOpt}" export "${tmpOpt}"="1" shift 1 continue fi # 判断参数是否是 --开头,如果是则将变量置为下一个参数 并 shift 2 if ([[ $tmpOpt = --* ]]);then tmpOpt=${tmpOpt#--} echo "检测到参数 ${tmpOpt} = $2" export "${tmpOpt}"="$2" shift 2 if ([ $? -ne 0 ]);then shift 1 fi continue fi shift 1 done # ================ 检查参数 ================ echo "检查参数" echo "检查gituri" if ([ -z "${gituri}" ]);then echo echo "请指定 gituri" echo "具体帮助如下:" echo "${helpMsg}" echo exit 1 fi # ================ 开始执行 ================ # ================ git 检出代码 ================ echo "使用git检出代码" # 生成一个临时文件夹 用于存放 检出的代码 tmpMkdocsDir=`mktemp -d` tmpGitDir="${tmpMkdocsDir}/docs" # 检出代码 echo "git clone --branch ${branch} --single-branch ${gituri} ${tmpGitDir} 1>/dev/null" # 把gituri加上用户名和密码 tmpHostAndPort=$(echo "${gituri}" | grep -oP '(?<=://)[^/]+?\.[^/]+(?=/?)|^[^/]+?\.[^/]+(?=/?)') gitrealuri=${gituri/${tmpHostAndPort}/${gitusername}:${gitpasswd}@${tmpHostAndPort}} git clone --branch ${branch} --single-branch ${gitrealuri} ${tmpGitDir} 1>/dev/null if ([ $? -ne 0 ]);then echo echo "git clone failed!" rm -rf ${tmpMkdocsDir} echo exit 1 fi # 加入 site_name 到 mkdocs.yml tmpBaseName=$(basename ${gituri}) mySiteName=${tmpBaseName%.*} cat > ${tmpMkdocsDir}/mkdocs.yml << EOF site_name: $(getMapNameForGitProjectName ${mySiteName}) pages: EOF # 使用README作为index.md if (! [ -f ${tmpGitDir}/README.md ]);then echo "Back
    " > ${tmpGitDir}/index.md else sed -i '1i\ Back
    ' ${tmpGitDir}/README.md mv -f ${tmpGitDir}/README.md ${tmpGitDir}/index.md fi # 生成 mkdocs.yml appendPagesContent ${tmpMkdocsDir} ${tmpGitDir} "" # 加入 theme 到 mkdocs.yml cat >> ${tmpMkdocsDir}/mkdocs.yml << EOF theme: windmill EOF # 运行 mkdocs build, 该命令会生成一个 mySiteName 目录 cd ${tmpMkdocsDir} echo echo "cat ${tmpMkdocsDir}/mkdocs.yml" cat ${tmpMkdocsDir}/mkdocs.yml echo echo mkdocs build -d ${tmpMkdocsDir}/${mySiteName} if ([ $? -ne 0 ]);then echo echo "mkdocs build failed!" rm -rf ${tmpMkdocsDir} echo exit 1 fi # 将 build 后的文件拷贝到指定位置 nginxDir=/usr/share/nginx/html/docs rm -rf ${nginxDir}/${mySiteName} cp -r ${tmpMkdocsDir}/${mySiteName} ${nginxDir}/${mySiteName} # 生成 index.html cat > ${nginxDir}/index.html < MyDocs
      EOF # 遍历nginx根目录下的文件夹，获取其映射后的名称 加入到index.html中 # 先按顺序加入文档连接 doAddLinkForProjectInMap "${nginxDir}" "${nginxDir}/index.html" # 继续遍历nginx根目录下加入未映射到的文件夹 # 追加 ul cat >> ${nginxDir}/index.html < EOF for file in ${nginxDir}/*;do if ([ $(basename ${file}) != "index.html" ]);then tmpMappedName=$(getMapNameForGitProjectName ${file}); tmpRetCode=$? # 存在映射的文档之前已经加过了,所以这时只添加未映射的文档 # tmpRetCode = 155 表示获取到了映射名称 if ([ ${tmpRetCode} -ne 155 ]);then echo "
    • ${tmpMappedName}
      • " >> ${nginxDir}/index.html fi fi done # 追加 /ul cat >> ${nginxDir}/index.html < EOF # 追加 /body cat >> ${nginxDir}/index.html < EOF # 结束后删除文件 rm -rf ${tmpMkdocsDir} echo "成功使用mkdocs部署文档项目${gituri}" echo ``` ## 中文搜索问题 ``` python # 安装分词工具 pip install jieba # 修改 /usr/lib/python2.7/site-packages/mkdocs/contrib/legacy_search/search_index.py def generate_search_index(self): """python to json conversion""" page_dicts = { 'docs': self._entries, } # 以下为新增部分 for doc in page_dicts['docs']: # 调用jieba的cut接口生成分词库，过滤重复词，过滤空格 tokens = list(set([token.lower() for token in jieba.cut_for_search(doc['title'].replace('\n', ''), True)])) if '' in tokens: tokens.remove('') doc['title_tokens'] = tokens tokens = list(set([token.lower() for token in jieba.cut_for_search(doc['text'].replace('\n', ''), True)])) if '' in tokens: tokens.remove('') doc['text_tokens'] = tokens # 新增部分结束 return json.dumps(page_dicts, sort_keys=True, indent=4) # 修改 /usr/lib/python2.7/site-packages/mkdocs/contrib/legacy_search/templates/search/lunr.min.js # 这个是压缩过的, 是不是可以下载一个未压缩版的, 应该更好修改 # 有两个地方要修改： # lunr.Index.prototype.add: 获取分词数据的方式，不在从内部的分词接口计算分词，直接从文档的词库加载 # lunr.trimmer: 过滤空白字符的接口，修改匹配方式，把原来只匹配字母、数字改成匹配所有非空字符 lunr.Index.prototype.add = function (doc, emitEvent) { var docTokens = {}, allDocumentTokens = new lunr.SortedSet, docRef = doc[this._ref], emitEvent = emitEvent === undefined ? true : emitEvent this._fields.forEach(function (field) { // 删掉内部接口计算分词 // var fieldTokens = this.pipeline.run(this.tokenizerFn(doc[field.name])) // 直接从文档词库加载 var fieldTokens = doc[field.name + '_tokens'] docTokens[field.name] = fieldTokens for (var i = 0; i < fieldTokens.length; i++) { var token = fieldTokens[i] allDocumentTokens.add(token) this.corpusTokens.add(token) } }, this) lunr.trimmer = function (token) { var result = token.replace(/^\s+/, '').replace(/\s+$/, '') // \W -> \s return result === '' ? undefined : result } ``` [官网文档]: https://squidfunk.github.io/mkdocs-material/ ================================================ FILE: docs/web/ali_js_style.md ================================================ # Airbnb JavaScript Style Guide() { **用更合理的方式写 JavaScript** [![Downloads](https://img.shields.io/npm/dm/eslint-config-airbnb.svg)](https://www.npmjs.com/package/eslint-config-airbnb) [![Downloads](https://img.shields.io/npm/dm/eslint-config-airbnb-base.svg)](https://www.npmjs.com/package/eslint-config-airbnb-base) [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/airbnb/javascript?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) ES5 的编码规范请查看[版本一](https://github.com/sivan/javascript-style-guide/blob/master/es5/README.md)，[版本二](https://github.com/adamlu/javascript-style-guide)。 翻译自 [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript) 。 ## 目录 1. [类型](#types) 1. [引用](#references) 1. [对象](#objects) 1. [数组](#arrays) 1. [解构](#destructuring) 1. [字符串](#strings) 1. [函数](#functions) 1. [箭头函数](#arrow-functions) 1. [构造函数](#constructors) 1. [模块](#modules) 1. [迭代器和生成器](#iterators-and-generators) 1. [属性](#properties) 1. [变量](#variables) 1. [提升](#hoisting) 1. [比较运算符和等号](#comparison-operators--equality) 1. [代码块](#blocks) 1. [注释](#comments) 1. [空白](#whitespace) 1. [逗号](#commas) 1. [分号](#semicolons) 1. [类型转换](#type-casting--coercion) 1. [命名规则](#naming-conventions) 1. [存取器](#accessors) 1. [事件](#events) 1. [jQuery](#jquery) 1. [ECMAScript 5 兼容性](#ecmascript-5-compatibility) 1. [ECMAScript 6 编码规范](#ecmascript-6-styles) 1. [测试](#testing) 1. [性能](#performance) 1. [相关资源](#resources) 1. [使用情况](#in-the-wild) 1. [其他翻译](#translation) 1. [JavaScript 编码规范说明](#the-javascript-style-guide-guide) 1. [讨论 JavaScript](#chat-with-us-about-javascript) 1. [贡献者](#contributors) 1. [许可协议](#license) ## 类型 - [1.1](#1.1) **基本类型**: 直接存取基本类型。 + `字符串` + `数值` + `布尔类型` + `null` + `undefined` ```javascript const foo = 1; let bar = foo; bar = 9; console.log(foo, bar); // => 1, 9 ``` - [1.2](#1.2) **复杂类型**: 通过引用的方式存取复杂类型。 + `对象` + `数组` + `函数` ```javascript const foo = [1, 2]; const bar = foo; bar[0] = 9; console.log(foo[0], bar[0]); // => 9, 9 ``` **[⬆ 返回目录](#table-of-contents)** ## 引用 - [2.1](#2.1) 对所有的引用使用 `const` ；不要使用 `var`。 > 为什么？这能确保你无法对引用重新赋值，也不会导致出现 bug 或难以理解。 ```javascript // bad var a = 1; var b = 2; // good const a = 1; const b = 2; ``` - [2.2](#2.2) 如果你一定需要可变动的引用，使用 `let` 代替 `var`。 > 为什么？因为 `let` 是块级作用域，而 `var` 是函数作用域。 ```javascript // bad var count = 1; if (true) { count += 1; } // good, use the let. let count = 1; if (true) { count += 1; } ``` - [2.3](#2.3) 注意 `let` 和 `const` 都是块级作用域。 ```javascript // const 和 let 只存在于它们被定义的区块内。 { let a = 1; const b = 1; } console.log(a); // ReferenceError console.log(b); // ReferenceError ``` **[⬆ 返回目录](#table-of-contents)** ## 对象 - [3.1](#3.1) 使用字面值创建对象。 ```javascript // bad const item = new Object(); // good const item = {}; ``` - [3.2](#3.2) 如果你的代码在浏览器环境下执行，别使用 [保留字](http://es5.github.io/#x7.6.1) 作为键值。这样的话在 IE8 不会运行。 [更多信息](https://github.com/airbnb/javascript/issues/61)。 但在 ES6 模块和服务器端中使用没有问题。 ```javascript // bad const superman = { default: { clark: 'kent' }, private: true, }; // good const superman = { defaults: { clark: 'kent' }, hidden: true, }; ``` - [3.3](#3.3) 使用同义词替换需要使用的保留字。 ```javascript // bad const superman = { class: 'alien', }; // bad const superman = { klass: 'alien', }; // good const superman = { type: 'alien', }; ``` - [3.4](#3.4) 创建有动态属性名的对象时，使用可被计算的属性名称。 > 为什么？因为这样可以让你在一个地方定义所有的对象属性。 ```javascript function getKey(k) { return `a key named ${k}`; } // bad const obj = { id: 5, name: 'San Francisco', }; obj[getKey('enabled')] = true; // good const obj = { id: 5, name: 'San Francisco', [getKey('enabled')]: true, }; ``` - [3.5](#3.5) 使用对象方法的简写。 ```javascript // bad const atom = { value: 1, addValue: function (value) { return atom.value + value; }, }; // good const atom = { value: 1, addValue(value) { return atom.value + value; }, }; ``` - [3.6](#3.6) 使用对象属性值的简写。 > 为什么？因为这样更短更有描述性。 ```javascript const lukeSkywalker = 'Luke Skywalker'; // bad const obj = { lukeSkywalker: lukeSkywalker, }; // good const obj = { lukeSkywalker, }; ``` - [3.7](#3.7) 在对象属性声明前把简写的属性分组。 > 为什么？因为这样能清楚地看出哪些属性使用了简写。 ```javascript const anakinSkywalker = 'Anakin Skywalker'; const lukeSkywalker = 'Luke Skywalker'; // bad const obj = { episodeOne: 1, twoJedisWalkIntoACantina: 2, lukeSkywalker, episodeThree: 3, mayTheFourth: 4, anakinSkywalker, }; // good const obj = { lukeSkywalker, anakinSkywalker, episodeOne: 1, twoJedisWalkIntoACantina: 2, episodeThree: 3, mayTheFourth: 4, }; ``` **[⬆ 返回目录](#table-of-contents)** ## 数组 - [4.1](#4.1) 使用字面值创建数组。 ```javascript // bad const items = new Array(); // good const items = []; ``` - [4.2](#4.2) 向数组添加元素时使用 Arrary#push 替代直接赋值。 ```javascript const someStack = []; // bad someStack[someStack.length] = 'abracadabra'; // good someStack.push('abracadabra'); ``` - [4.3](#4.3) 使用拓展运算符 `...` 复制数组。 ```javascript // bad const len = items.length; const itemsCopy = []; let i; for (i = 0; i < len; i++) { itemsCopy[i] = items[i]; } // good const itemsCopy = [...items]; ``` - [4.4](#4.4) 使用 Array#from 把一个类数组对象转换成数组。 ```javascript const foo = document.querySelectorAll('.foo'); const nodes = Array.from(foo); ``` **[⬆ 返回目录](#table-of-contents)** ## 解构 - [5.1](#5.1) 使用解构存取和使用多属性对象。 > 为什么？因为解构能减少临时引用属性。 ```javascript // bad function getFullName(user) { const firstName = user.firstName; const lastName = user.lastName; return `${firstName} ${lastName}`; } // good function getFullName(obj) { const { firstName, lastName } = obj; return `${firstName} ${lastName}`; } // best function getFullName({ firstName, lastName }) { return `${firstName} ${lastName}`; } ``` - [5.2](#5.2) 对数组使用解构赋值。 ```javascript const arr = [1, 2, 3, 4]; // bad const first = arr[0]; const second = arr[1]; // good const [first, second] = arr; ``` - [5.3](#5.3) 需要回传多个值时，使用对象解构，而不是数组解构。 > 为什么？增加属性或者改变排序不会改变调用时的位置。 ```javascript // bad function processInput(input) { // then a miracle occurs return [left, right, top, bottom]; } // 调用时需要考虑回调数据的顺序。 const [left, __, top] = processInput(input); // good function processInput(input) { // then a miracle occurs return { left, right, top, bottom }; } // 调用时只选择需要的数据 const { left, right } = processInput(input); ``` **[⬆ 返回目录](#table-of-contents)** ## Strings - [6.1](#6.1) 字符串使用单引号 `''` 。 ```javascript // bad const name = "Capt. Janeway"; // good const name = 'Capt. Janeway'; ``` - [6.2](#6.2) 字符串超过 80 个字节应该使用字符串连接号换行。 - [6.3](#6.3) 注：过度使用字串连接符号可能会对性能造成影响。[jsPerf](http://jsperf.com/ya-string-concat) 和 [讨论](https://github.com/airbnb/javascript/issues/40). ```javascript // bad const errorMessage = 'This is a super long error that was thrown because of Batman. When you stop to think about how Batman had anything to do with this, you would get nowhere fast.'; // bad const errorMessage = 'This is a super long error that was thrown because \ of Batman. When you stop to think about how Batman had anything to do \ with this, you would get nowhere \ fast.'; // good const errorMessage = 'This is a super long error that was thrown because ' + 'of Batman. When you stop to think about how Batman had anything to do ' + 'with this, you would get nowhere fast.'; ``` - [6.4](#6.4) 程序化生成字符串时，使用模板字符串代替字符串连接。 > 为什么？模板字符串更为简洁，更具可读性。 ```javascript // bad function sayHi(name) { return 'How are you, ' + name + '?'; } // bad function sayHi(name) { return ['How are you, ', name, '?'].join(); } // good function sayHi(name) { return `How are you, ${name}?`; } ``` **[⬆ 返回目录](#table-of-contents)** ## 函数 - [7.1](#7.1) 使用函数声明代替函数表达式。 > 为什么？因为函数声明是可命名的，所以他们在调用栈中更容易被识别。此外，函数声明会把整个函数提升（hoisted），而函数表达式只会把函数的引用变量名提升。这条规则使得[箭头函数](#arrow-functions)可以取代函数表达式。 ```javascript // bad const foo = function () { }; // good function foo() { } ``` - [7.2](#7.2) 函数表达式: ```javascript // 立即调用的函数表达式 (IIFE) (() => { console.log('Welcome to the Internet. Please follow me.'); })(); ``` - [7.3](#7.3) 永远不要在一个非函数代码块（`if`、`while` 等）中声明一个函数，把那个函数赋给一个变量。浏览器允许你这么做，但它们的解析表现不一致。 - [7.4](#7.4) **注意:** ECMA-262 把 `block` 定义为一组语句。函数声明不是语句。[阅读 ECMA-262 关于这个问题的说明](http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf#page=97)。 ```javascript // bad if (currentUser) { function test() { console.log('Nope.'); } } // good let test; if (currentUser) { test = () => { console.log('Yup.'); }; } ``` - [7.5](#7.5) 永远不要把参数命名为 `arguments`。这将取代原来函数作用域内的 `arguments` 对象。 ```javascript // bad function nope(name, options, arguments) { // ...stuff... } // good function yup(name, options, args) { // ...stuff... } ``` - [7.6](#7.6) 不要使用 `arguments`。可以选择 rest 语法 `...` 替代。 > 为什么？使用 `...` 能明确你要传入的参数。另外 rest 参数是一个真正的数组，而 `arguments` 是一个类数组。 ```javascript // bad function concatenateAll() { const args = Array.prototype.slice.call(arguments); return args.join(''); } // good function concatenateAll(...args) { return args.join(''); } ``` - [7.7](#7.7) 直接给函数的参数指定默认值，不要使用一个变化的函数参数。 ```javascript // really bad function handleThings(opts) { // 不！我们不应该改变函数参数。 // 更加糟糕: 如果参数 opts 是 false 的话，它就会被设定为一个对象。 // 但这样的写法会造成一些 Bugs。 //（译注：例如当 opts 被赋值为空字符串，opts 仍然会被下一行代码设定为一个空对象。） opts = opts || {}; // ... } // still bad function handleThings(opts) { if (opts === void 0) { opts = {}; } // ... } // good function handleThings(opts = {}) { // ... } ``` - [7.8](#7.8) 直接给函数参数赋值时需要避免副作用。 > 为什么？因为这样的写法让人感到很困惑。 ```javascript var b = 1; // bad function count(a = b++) { console.log(a); } count(); // 1 count(); // 2 count(3); // 3 count(); // 3 ``` **[⬆ 返回目录](#table-of-contents)** ## 箭头函数 - [8.1](#8.1) 当你必须使用函数表达式（或传递一个匿名函数）时，使用箭头函数符号。 > 为什么？因为箭头函数创造了新的一个 `this` 执行环境（译注：参考 [Arrow functions - JavaScript | MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions) 和 [ES6 arrow functions, syntax and lexical scoping](http://toddmotto.com/es6-arrow-functions-syntaxes-and-lexical-scoping/)），通常情况下都能满足你的需求，而且这样的写法更为简洁。 > 为什么不？如果你有一个相当复杂的函数，你或许可以把逻辑部分转移到一个函数声明上。 ```javascript // bad [1, 2, 3].map(function (x) { const y = x + 1; return x * y; }); // good [1, 2, 3].map((x) => { const y = x + 1; return x * y; }); ``` - [8.2](#8.2) 如果一个函数适合用一行写出并且只有一个参数，那就把花括号、圆括号和 `return` 都省略掉。如果不是，那就不要省略。 > 为什么？语法糖。在链式调用中可读性很高。 > 为什么不？当你打算回传一个对象的时候。 ```javascript // good [1, 2, 3].map(x => x * x); // good [1, 2, 3].reduce((total, n) => { return total + n; }, 0); ``` **[⬆ 返回目录](#table-of-contents)** ## 构造器 - [9.1](#9.1) 总是使用 `class`。避免直接操作 `prototype` 。 > 为什么? 因为 `class` 语法更为简洁更易读。 ```javascript // bad function Queue(contents = []) { this._queue = [...contents]; } Queue.prototype.pop = function() { const value = this._queue[0]; this._queue.splice(0, 1); return value; } // good class Queue { constructor(contents = []) { this._queue = [...contents]; } pop() { const value = this._queue[0]; this._queue.splice(0, 1); return value; } } ``` - [9.2](#9.2) 使用 `extends` 继承。 > 为什么？因为 `extends` 是一个内建的原型继承方法并且不会破坏 `instanceof`。 ```javascript // bad const inherits = require('inherits'); function PeekableQueue(contents) { Queue.apply(this, contents); } inherits(PeekableQueue, Queue); PeekableQueue.prototype.peek = function() { return this._queue[0]; } // good class PeekableQueue extends Queue { peek() { return this._queue[0]; } } ``` - [9.3](#9.3) 方法可以返回 `this` 来帮助链式调用。 ```javascript // bad Jedi.prototype.jump = function() { this.jumping = true; return true; }; Jedi.prototype.setHeight = function(height) { this.height = height; }; const luke = new Jedi(); luke.jump(); // => true luke.setHeight(20); // => undefined // good class Jedi { jump() { this.jumping = true; return this; } setHeight(height) { this.height = height; return this; } } const luke = new Jedi(); luke.jump() .setHeight(20); ``` - [9.4](#9.4) 可以写一个自定义的 `toString()` 方法，但要确保它能正常运行并且不会引起副作用。 ```javascript class Jedi { constructor(options = {}) { this.name = options.name || 'no name'; } getName() { return this.name; } toString() { return `Jedi - ${this.getName()}`; } } ``` **[⬆ 返回目录](#table-of-contents)** ## 模块 - [10.1](#10.1) 总是使用模组 (`import`/`export`) 而不是其他非标准模块系统。你可以编译为你喜欢的模块系统。 > 为什么？模块就是未来，让我们开始迈向未来吧。 ```javascript // bad const AirbnbStyleGuide = require('./AirbnbStyleGuide'); module.exports = AirbnbStyleGuide.es6; // ok import AirbnbStyleGuide from './AirbnbStyleGuide'; export default AirbnbStyleGuide.es6; // best import { es6 } from './AirbnbStyleGuide'; export default es6; ``` - [10.2](#10.2) 不要使用通配符 import。 > 为什么？这样能确保你只有一个默认 export。 ```javascript // bad import * as AirbnbStyleGuide from './AirbnbStyleGuide'; // good import AirbnbStyleGuide from './AirbnbStyleGuide'; ``` - [10.3](#10.3) 不要从 import 中直接 export。 > 为什么？虽然一行代码简洁明了，但让 import 和 export 各司其职让事情能保持一致。 ```javascript // bad // filename es6.js export { es6 as default } from './airbnbStyleGuide'; // good // filename es6.js import { es6 } from './AirbnbStyleGuide'; export default es6; ``` **[⬆ 返回目录](#table-of-contents)** ## Iterators and Generators - [11.1](#11.1) 不要使用 iterators。使用高阶函数例如 `map()` 和 `reduce()` 替代 `for-of`。 > 为什么？这加强了我们不变的规则。处理纯函数的回调值更易读，这比它带来的副作用更重要。 ```javascript const numbers = [1, 2, 3, 4, 5]; // bad let sum = 0; for (let num of numbers) { sum += num; } sum === 15; // good let sum = 0; numbers.forEach((num) => sum += num); sum === 15; // best (use the functional force) const sum = numbers.reduce((total, num) => total + num, 0); sum === 15; ``` - [11.2](#11.2) 现在还不要使用 generators。 > 为什么？因为它们现在还没法很好地编译到 ES5。 (译者注：目前(2016/03) Chrome 和 Node.js 的稳定版本都已支持 generators) **[⬆ 返回目录](#table-of-contents)** ## 属性 - [12.1](#12.1) 使用 `.` 来访问对象的属性。 ```javascript const luke = { jedi: true, age: 28, }; // bad const isJedi = luke['jedi']; // good const isJedi = luke.jedi; ``` - [12.2](#12.2) 当通过变量访问属性时使用中括号 `[]`。 ```javascript const luke = { jedi: true, age: 28, }; function getProp(prop) { return luke[prop]; } const isJedi = getProp('jedi'); ``` **[⬆ 返回目录](#table-of-contents)** ## 变量 - [13.1](#13.1) 一直使用 `const` 来声明变量，如果不这样做就会产生全局变量。我们需要避免全局命名空间的污染。[地球队长](http://www.wikiwand.com/en/Captain_Planet)已经警告过我们了。（译注：全局，global 亦有全球的意思。地球队长的责任是保卫地球环境，所以他警告我们不要造成「全球」污染。） ```javascript // bad superPower = new SuperPower(); // good const superPower = new SuperPower(); ``` - [13.2](#13.2) 使用 `const` 声明每一个变量。 > 为什么？增加新变量将变的更加容易，而且你永远不用再担心调换错 `;` 跟 `,`。 ```javascript // bad const items = getItems(), goSportsTeam = true, dragonball = 'z'; // bad // (compare to above, and try to spot the mistake) const items = getItems(), goSportsTeam = true; dragonball = 'z'; // good const items = getItems(); const goSportsTeam = true; const dragonball = 'z'; ``` - [13.3](#13.3) 将所有的 `const` 和 `let` 分组 > 为什么？当你需要把已赋值变量赋值给未赋值变量时非常有用。 ```javascript // bad let i, len, dragonball, items = getItems(), goSportsTeam = true; // bad let i; const items = getItems(); let dragonball; const goSportsTeam = true; let len; // good const goSportsTeam = true; const items = getItems(); let dragonball; let i; let length; ``` - [13.4](#13.4) 在你需要的地方给变量赋值，但请把它们放在一个合理的位置。 > 为什么？`let` 和 `const` 是块级作用域而不是函数作用域。 ```javascript // good function() { test(); console.log('doing stuff..'); //..other stuff.. const name = getName(); if (name === 'test') { return false; } return name; } // bad - unnecessary function call function(hasName) { const name = getName(); if (!hasName) { return false; } this.setFirstName(name); return true; } // good function(hasName) { if (!hasName) { return false; } const name = getName(); this.setFirstName(name); return true; } ``` **[⬆ 返回目录](#table-of-contents)** ## Hoisting - [14.1](#14.1) `var` 声明会被提升至该作用域的顶部，但它们赋值不会提升。`let` 和 `const` 被赋予了一种称为「[暂时性死区（Temporal Dead Zones, TDZ）](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/let#Temporal_dead_zone_and_errors_with_let)」的概念。这对于了解为什么 [type of 不再安全](http://es-discourse.com/t/why-typeof-is-no-longer-safe/15)相当重要。 ```javascript // 我们知道这样运行不了 // （假设 notDefined 不是全局变量） function example() { console.log(notDefined); // => throws a ReferenceError } // 由于变量提升的原因， // 在引用变量后再声明变量是可以运行的。 // 注：变量的赋值 `true` 不会被提升。 function example() { console.log(declaredButNotAssigned); // => undefined var declaredButNotAssigned = true; } // 编译器会把函数声明提升到作用域的顶层， // 这意味着我们的例子可以改写成这样： function example() { let declaredButNotAssigned; console.log(declaredButNotAssigned); // => undefined declaredButNotAssigned = true; } // 使用 const 和 let function example() { console.log(declaredButNotAssigned); // => throws a ReferenceError console.log(typeof declaredButNotAssigned); // => throws a ReferenceError const declaredButNotAssigned = true; } ``` - [14.2](#14.2) 匿名函数表达式的变量名会被提升，但函数内容并不会。 ```javascript function example() { console.log(anonymous); // => undefined anonymous(); // => TypeError anonymous is not a function var anonymous = function() { console.log('anonymous function expression'); }; } ``` - [14.3](#14.3) 命名的函数表达式的变量名会被提升，但函数名和函数函数内容并不会。 ```javascript function example() { console.log(named); // => undefined named(); // => TypeError named is not a function superPower(); // => ReferenceError superPower is not defined var named = function superPower() { console.log('Flying'); }; } // the same is true when the function name // is the same as the variable name. function example() { console.log(named); // => undefined named(); // => TypeError named is not a function var named = function named() { console.log('named'); } } ``` - [14.4](#14.4) 函数声明的名称和函数体都会被提升。 ```javascript function example() { superPower(); // => Flying function superPower() { console.log('Flying'); } } ``` - 想了解更多信息，参考 [Ben Cherry](http://www.adequatelygood.com/) 的 [JavaScript Scoping & Hoisting](http://www.adequatelygood.com/2010/2/JavaScript-Scoping-and-Hoisting)。 **[⬆ 返回目录](#table-of-contents)** ## 比较运算符和等号 - [15.1](#15.1) 优先使用 `===` 和 `!==` 而不是 `==` 和 `!=`. - [15.2](#15.2) 条件表达式例如 `if` 语句通过抽象方法 `ToBoolean` 强制计算它们的表达式并且总是遵守下面的规则： + **对象** 被计算为 **true** + **Undefined** 被计算为 **false** + **Null** 被计算为 **false** + **布尔值** 被计算为 **布尔的值** + **数字** 如果是 **+0、-0、或 NaN** 被计算为 **false**, 否则为 **true** + **字符串** 如果是空字符串 `''` 被计算为 **false**，否则为 **true** ```javascript if ([0]) { // true // An array is an object, objects evaluate to true } ``` - [15.3](#15.3) 使用简写。 ```javascript // bad if (name !== '') { // ...stuff... } // good if (name) { // ...stuff... } // bad if (collection.length > 0) { // ...stuff... } // good if (collection.length) { // ...stuff... } ``` - [15.4](#15.4) 想了解更多信息，参考 Angus Croll 的 [Truth Equality and JavaScript](http://javascriptweblog.wordpress.com/2011/02/07/truth-equality-and-javascript/#more-2108)。 **[⬆ 返回目录](#table-of-contents)** ## 代码块 - [16.1](#16.1) 使用大括号包裹所有的多行代码块。 ```javascript // bad if (test) return false; // good if (test) return false; // good if (test) { return false; } // bad function() { return false; } // good function() { return false; } ``` - [16.2](#16.2) 如果通过 `if` 和 `else` 使用多行代码块，把 `else` 放在 `if` 代码块关闭括号的同一行。 ```javascript // bad if (test) { thing1(); thing2(); } else { thing3(); } // good if (test) { thing1(); thing2(); } else { thing3(); } ``` **[⬆ 返回目录](#table-of-contents)** ## 注释 - [17.1](#17.1) 使用 `/** ... */` 作为多行注释。包含描述、指定所有参数和返回值的类型和值。 ```javascript // bad // make() returns a new element // based on the passed in tag name // // @param {String} tag // @return {Element} element function make(tag) { // ...stuff... return element; } // good /** * make() returns a new element * based on the passed in tag name * * @param {String} tag * @return {Element} element */ function make(tag) { // ...stuff... return element; } ``` - [17.2](#17.2) 使用 `//` 作为单行注释。在评论对象上面另起一行使用单行注释。在注释前插入空行。 ```javascript // bad const active = true; // is current tab // good // is current tab const active = true; // bad function getType() { console.log('fetching type...'); // set the default type to 'no type' const type = this._type || 'no type'; return type; } // good function getType() { console.log('fetching type...'); // set the default type to 'no type' const type = this._type || 'no type'; return type; } ``` - [17.3](#17.3) 给注释增加 `FIXME` 或 `TODO` 的前缀可以帮助其他开发者快速了解这是一个需要复查的问题，或是给需要实现的功能提供一个解决方式。这将有别于常见的注释，因为它们是可操作的。使用 `FIXME -- need to figure this out` 或者 `TODO -- need to implement`。 - [17.4](#17.4) 使用 `// FIXME`: 标注问题。 ```javascript class Calculator { constructor() { // FIXME: shouldn't use a global here total = 0; } } ``` - [17.5](#17.5) 使用 `// TODO`: 标注问题的解决方式。 ```javascript class Calculator { constructor() { // TODO: total should be configurable by an options param this.total = 0; } } ``` **[⬆ 返回目录](#table-of-contents)** ## 空白 - [18.1](#18.1) 使用 2 个空格作为缩进。 ```javascript // bad function() { ∙∙∙∙const name; } // bad function() { ∙const name; } // good function() { ∙∙const name; } ``` - [18.2](#18.2) 在花括号前放一个空格。 ```javascript // bad function test(){ console.log('test'); } // good function test() { console.log('test'); } // bad dog.set('attr',{ age: '1 year', breed: 'Bernese Mountain Dog', }); // good dog.set('attr', { age: '1 year', breed: 'Bernese Mountain Dog', }); ``` - [18.3](#18.3) 在控制语句（`if`、`while` 等）的小括号前放一个空格。在函数调用及声明中，不在函数的参数列表前加空格。 ```javascript // bad if(isJedi) { fight (); } // good if (isJedi) { fight(); } // bad function fight () { console.log ('Swooosh!'); } // good function fight() { console.log('Swooosh!'); } ``` - [18.4](#18.4) 使用空格把运算符隔开。 ```javascript // bad const x=y+5; // good const x = y + 5; ``` - [18.5](#18.5) 在文件末尾插入一个空行。 ```javascript // bad (function(global) { // ...stuff... })(this); ``` ```javascript // bad (function(global) { // ...stuff... })(this);↵ ↵ ``` ```javascript // good (function(global) { // ...stuff... })(this);↵ ``` - [18.5](#18.5) 在使用长方法链时进行缩进。使用前面的点 `.` 强调这是方法调用而不是新语句。 ```javascript // bad $('#items').find('.selected').highlight().end().find('.open').updateCount(); // bad $('#items'). find('.selected'). highlight(). end(). find('.open'). updateCount(); // good $('#items') .find('.selected') .highlight() .end() .find('.open') .updateCount(); // bad const leds = stage.selectAll('.led').data(data).enter().append('svg:svg').class('led', true) .attr('width', (radius + margin) * 2).append('svg:g') .attr('transform', 'translate(' + (radius + margin) + ',' + (radius + margin) + ')') .call(tron.led); // good const leds = stage.selectAll('.led') .data(data) .enter().append('svg:svg') .classed('led', true) .attr('width', (radius + margin) * 2) .append('svg:g') .attr('transform', 'translate(' + (radius + margin) + ',' + (radius + margin) + ')') .call(tron.led); ``` - [18.6](#18.6) 在块末和新语句前插入空行。 ```javascript // bad if (foo) { return bar; } return baz; // good if (foo) { return bar; } return baz; // bad const obj = { foo() { }, bar() { }, }; return obj; // good const obj = { foo() { }, bar() { }, }; return obj; ``` **[⬆ 返回目录](#table-of-contents)** ## 逗号 - [19.1](#19.1) 行首逗号：**不需要**。 ```javascript // bad const story = [ once , upon , aTime ]; // good const story = [ once, upon, aTime, ]; // bad const hero = { firstName: 'Ada' , lastName: 'Lovelace' , birthYear: 1815 , superPower: 'computers' }; // good const hero = { firstName: 'Ada', lastName: 'Lovelace', birthYear: 1815, superPower: 'computers', }; ``` - [19.2](#19.2) 增加结尾的逗号: **需要**。 > 为什么? 这会让 git diffs 更干净。另外，像 babel 这样的转译器会移除结尾多余的逗号，也就是说你不必担心老旧浏览器的[尾逗号问题](https://github.com/sivan/javascript-style-guide/blob/master/es5/README.md)。 ```javascript // bad - git diff without trailing comma const hero = { firstName: 'Florence', - lastName: 'Nightingale' + lastName: 'Nightingale', + inventorOf: ['coxcomb graph', 'modern nursing'] } // good - git diff with trailing comma const hero = { firstName: 'Florence', lastName: 'Nightingale', + inventorOf: ['coxcomb chart', 'modern nursing'], } // bad const hero = { firstName: 'Dana', lastName: 'Scully' }; const heroes = [ 'Batman', 'Superman' ]; // good const hero = { firstName: 'Dana', lastName: 'Scully', }; const heroes = [ 'Batman', 'Superman', ]; ``` **[⬆ 返回目录](#table-of-contents)** ## 分号 - [20.1](#20.1) **使用分号** ```javascript // bad (function() { const name = 'Skywalker' return name })() // good (() => { const name = 'Skywalker'; return name; })(); // good (防止函数在两个 IIFE 合并时被当成一个参数) ;(() => { const name = 'Skywalker'; return name; })(); ``` [Read more](http://stackoverflow.com/a/7365214/1712802). **[⬆ 返回目录](#table-of-contents)** ## 类型转换 - [21.1](#21.1) 在语句开始时执行类型转换。 - [21.2](#21.2) 字符串： ```javascript // => this.reviewScore = 9; // bad const totalScore = this.reviewScore + ''; // good const totalScore = String(this.reviewScore); ``` - [21.3](#21.3) 对数字使用 `parseInt` 转换，并带上类型转换的基数。 ```javascript const inputValue = '4'; // bad const val = new Number(inputValue); // bad const val = +inputValue; // bad const val = inputValue >> 0; // bad const val = parseInt(inputValue); // good const val = Number(inputValue); // good const val = parseInt(inputValue, 10); ``` - [21.4](#21.4) 如果因为某些原因 parseInt 成为你所做的事的瓶颈而需要使用位操作解决[性能问题](http://jsperf.com/coercion-vs-casting/3)时，留个注释说清楚原因和你的目的。 ```javascript // good /** * 使用 parseInt 导致我的程序变慢， * 改成使用位操作转换数字快多了。 */ const val = inputValue >> 0; ``` - [21.5](#21.5) **注:** 小心使用位操作运算符。数字会被当成 [64 位值](http://es5.github.io/#x4.3.19)，但是位操作运算符总是返回 32 位的整数（[参考](http://es5.github.io/#x11.7)）。位操作处理大于 32 位的整数值时还会导致意料之外的行为。[关于这个问题的讨论](https://github.com/airbnb/javascript/issues/109)。最大的 32 位整数是 2,147,483,647： ```javascript 2147483647 >> 0 //=> 2147483647 2147483648 >> 0 //=> -2147483648 2147483649 >> 0 //=> -2147483647 ``` - [21.6](#21.6) 布尔: ```javascript const age = 0; // bad const hasAge = new Boolean(age); // good const hasAge = Boolean(age); // good const hasAge = !!age; ``` **[⬆ 返回目录](#table-of-contents)** ## 命名规则 - [22.1](#22.1) 避免单字母命名。命名应具备描述性。 ```javascript // bad function q() { // ...stuff... } // good function query() { // ..stuff.. } ``` - [22.2](#22.2) 使用驼峰式命名对象、函数和实例。 ```javascript // bad const OBJEcttsssss = {}; const this_is_my_object = {}; function c() {} // good const thisIsMyObject = {}; function thisIsMyFunction() {} ``` - [22.3](#22.3) 使用帕斯卡式命名构造函数或类。 ```javascript // bad function user(options) { this.name = options.name; } const bad = new user({ name: 'nope', }); // good class User { constructor(options) { this.name = options.name; } } const good = new User({ name: 'yup', }); ``` - [22.4](#22.4) 不要使用下划线 `_` 结尾或开头来命名属性和方法。 ```javascript // bad this.__firstName__ = 'Panda'; this.firstName_ = 'Panda'; this._firstName = 'Panda'; // good this.firstName = 'Panda'; ``` - [22.5](#22.5) 别保存 `this` 的引用。使用箭头函数或 Function#bind。 ```javascript // bad function foo() { const self = this; return function() { console.log(self); }; } // bad function foo() { const that = this; return function() { console.log(that); }; } // good function foo() { return () => { console.log(this); }; } ``` - [22.6](#22.6) 如果你的文件只输出一个类，那你的文件名必须和类名完全保持一致。 ```javascript // file contents class CheckBox { // ... } export default CheckBox; // in some other file // bad import CheckBox from './checkBox'; // bad import CheckBox from './check_box'; // good import CheckBox from './CheckBox'; ``` - [22.7](#22.7) 当你导出默认的函数时使用驼峰式命名。你的文件名必须和函数名完全保持一致。 ```javascript function makeStyleGuide() { } export default makeStyleGuide; ``` - [22.8](#22.8) 当你导出单例、函数库、空对象时使用帕斯卡式命名。 ```javascript const AirbnbStyleGuide = { es6: { } }; export default AirbnbStyleGuide; ``` **[⬆ 返回目录](#table-of-contents)** ## 存取器 - [23.1](#23.1) 属性的存取函数不是必须的。 - [23.2](#23.2) 如果你需要存取函数时使用 `getVal()` 和 `setVal('hello')`。 ```javascript // bad dragon.age(); // good dragon.getAge(); // bad dragon.age(25); // good dragon.setAge(25); ``` - [23.3](#23.3) 如果属性是布尔值，使用 `isVal()` 或 `hasVal()`。 ```javascript // bad if (!dragon.age()) { return false; } // good if (!dragon.hasAge()) { return false; } ``` - [23.4](#23.4) 创建 `get()` 和 `set()` 函数是可以的，但要保持一致。 ```javascript class Jedi { constructor(options = {}) { const lightsaber = options.lightsaber || 'blue'; this.set('lightsaber', lightsaber); } set(key, val) { this[key] = val; } get(key) { return this[key]; } } ``` **[⬆ 返回目录](#table-of-contents)** ## 事件 - [24.1](#24.1) 当给事件附加数据时（无论是 DOM 事件还是私有事件），传入一个哈希而不是原始值。这样可以让后面的贡献者增加更多数据到事件数据而无需找出并更新事件的每一个处理器。例如，不好的写法： ```javascript // bad $(this).trigger('listingUpdated', listing.id); ... $(this).on('listingUpdated', function(e, listingId) { // do something with listingId }); ``` 更好的写法： ```javascript // good $(this).trigger('listingUpdated', { listingId : listing.id }); ... $(this).on('listingUpdated', function(e, data) { // do something with data.listingId }); ``` **[⬆ 返回目录](#table-of-contents)** ## jQuery - [25.1](#25.1) 使用 `$` 作为存储 jQuery 对象的变量名前缀。 ```javascript // bad const sidebar = $('.sidebar'); // good const $sidebar = $('.sidebar'); ``` - [25.2](#25.2) 缓存 jQuery 查询。 ```javascript // bad function setSidebar() { $('.sidebar').hide(); // ...stuff... $('.sidebar').css({ 'background-color': 'pink' }); } // good function setSidebar() { const $sidebar = $('.sidebar'); $sidebar.hide(); // ...stuff... $sidebar.css({ 'background-color': 'pink' }); } ``` - [25.3](#25.3) 对 DOM 查询使用层叠 `$('.sidebar ul')` 或 父元素 > 子元素 `$('.sidebar > ul')`。 [jsPerf](http://jsperf.com/jquery-find-vs-context-sel/16) - [25.4](#25.4) 对有作用域的 jQuery 对象查询使用 `find`。 ```javascript // bad $('ul', '.sidebar').hide(); // bad $('.sidebar').find('ul').hide(); // good $('.sidebar ul').hide(); // good $('.sidebar > ul').hide(); // good $sidebar.find('ul').hide(); ``` **[⬆ 返回目录](#table-of-contents)** ## ECMAScript 5 兼容性 - [26.1](#26.1) 参考 [Kangax](https://twitter.com/kangax/) 的 ES5 [兼容性](http://kangax.github.com/es5-compat-table/)。 **[⬆ 返回目录](#table-of-contents)** ## ECMAScript 6 规范 - [27.1](#27.1) 以下是链接到 ES6 各个特性的列表。 1. [箭头函数](#arrow-functions) 1. [类](#constructors) 1. [对象方法简写](#es6-object-shorthand) 1. [对象属性简写](#es6-object-concise) 1. [对象中的可计算属性](#es6-computed-properties) 1. [模板字符串](#es6-template-literals) 1. [解构](#destructuring) 1. [默认参数](#es6-default-parameters) 1. [Rest](#es6-rest) 1. [数组 Spreads](#es6-array-spreads) 1. [Let 及 Const](#references) 1. [迭代器和生成器](#iterators-and-generators) 1. [模块](#modules) **[⬆ 返回目录](#table-of-contents)** ## 测试 - [28.1](#28.1) **Yup.** ```javascript function() { return true; } ``` **[⬆ 返回目录](#table-of-contents)** ## 性能 - [On Layout & Web Performance](http://kellegous.com/j/2013/01/26/layout-performance/) - [String vs Array Concat](http://jsperf.com/string-vs-array-concat/2) - [Try/Catch Cost In a Loop](http://jsperf.com/try-catch-in-loop-cost) - [Bang Function](http://jsperf.com/bang-function) - [jQuery Find vs Context, Selector](http://jsperf.com/jquery-find-vs-context-sel/13) - [innerHTML vs textContent for script text](http://jsperf.com/innerhtml-vs-textcontent-for-script-text) - [Long String Concatenation](http://jsperf.com/ya-string-concat) - [Are Javascript functions like `map()`, `reduce()`, and `filter()` optimized for traversing arrays?](https://www.quora.com/JavaScript-programming-language-Are-Javascript-functions-like-map-reduce-and-filter-already-optimized-for-traversing-array/answer/Quildreen-Motta) - 等等... **[⬆ 返回目录](#table-of-contents)** ## 相关资源（英文） **了解 ES6** - [ECMA 2015 (ES6) 规范草案](https://people.mozilla.org/~jorendorff/es6-draft.html) - [ExploringJS](http://exploringjs.com/) - [ES6 兼容性表](https://kangax.github.io/compat-table/es6/) - [ES6 特性全面概况](http://es6-features.org/) **看看这个** - [Annotated ECMAScript 5.1](http://es5.github.com/) **工具** - 代码风格检查器（Lint） + [ESlint](http://eslint.org/) - [Airbnb Style .eslintrc](https://github.com/airbnb/javascript/blob/master/linters/.eslintrc) + [JSHint](http://www.jshint.com/) - [Airbnb Style .jshintrc](https://github.com/airbnb/javascript/blob/master/linters/jshintrc) + [JSCS](https://github.com/jscs-dev/node-jscs) - [Airbnb Style Preset](https://github.com/jscs-dev/node-jscs/blob/master/presets/airbnb.json) **其他风格指南** - [Google JavaScript Style Guide](http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml) - [jQuery Core Style Guidelines](http://docs.jquery.com/JQuery_Core_Style_Guidelines) - [Principles of Writing Consistent, Idiomatic JavaScript](https://github.com/rwldrn/idiomatic.js/) **其他风格** - [Naming this in nested functions](https://gist.github.com/4135065) - Christian Johansen - [Conditional Callbacks](https://github.com/airbnb/javascript/issues/52) - Ross Allen - [Popular JavaScript Coding Conventions on Github](http://sideeffect.kr/popularconvention/#javascript) - JeongHoon Byun - [Multiple var statements in JavaScript, not superfluous](http://benalman.com/news/2012/05/multiple-var-statements-javascript/) - Ben Alman **拓展阅读** - [Understanding JavaScript Closures](http://javascriptweblog.wordpress.com/2010/10/25/understanding-javascript-closures/) - Angus Croll - [Basic JavaScript for the impatient programmer](http://www.2ality.com/2013/06/basic-javascript.html) - Dr. Axel Rauschmayer - [You Might Not Need jQuery](http://youmightnotneedjquery.com/) - Zack Bloom & Adam Schwartz - [ES6 Features](https://github.com/lukehoban/es6features) - Luke Hoban - [Frontend Guidelines](https://github.com/bendc/frontend-guidelines) - Benjamin De Cock **书籍** - [JavaScript: The Good Parts](http://www.amazon.com/JavaScript-Good-Parts-Douglas-Crockford/dp/0596517742) - Douglas Crockford - [JavaScript Patterns](http://www.amazon.com/JavaScript-Patterns-Stoyan-Stefanov/dp/0596806752) - Stoyan Stefanov - [Pro JavaScript Design Patterns](http://www.amazon.com/JavaScript-Design-Patterns-Recipes-Problem-Solution/dp/159059908X) - Ross Harmes and Dustin Diaz - [High Performance Web Sites: Essential Knowledge for Front-End Engineers](http://www.amazon.com/High-Performance-Web-Sites-Essential/dp/0596529309) - Steve Souders - [Maintainable JavaScript](http://www.amazon.com/Maintainable-JavaScript-Nicholas-C-Zakas/dp/1449327680) - Nicholas C. Zakas - [JavaScript Web Applications](http://www.amazon.com/JavaScript-Web-Applications-Alex-MacCaw/dp/144930351X) - Alex MacCaw - [Pro JavaScript Techniques](http://www.amazon.com/Pro-JavaScript-Techniques-John-Resig/dp/1590597273) - John Resig - [Smashing Node.js: JavaScript Everywhere](http://www.amazon.com/Smashing-Node-js-JavaScript-Everywhere-Magazine/dp/1119962595) - Guillermo Rauch - [Secrets of the JavaScript Ninja](http://www.amazon.com/Secrets-JavaScript-Ninja-John-Resig/dp/193398869X) - John Resig and Bear Bibeault - [Human JavaScript](http://humanjavascript.com/) - Henrik Joreteg - [Superhero.js](http://superherojs.com/) - Kim Joar Bekkelund, Mads Mobæk, & Olav Bjorkoy - [JSBooks](http://jsbooks.revolunet.com/) - Julien Bouquillon - [Third Party JavaScript](http://manning.com/vinegar/) - Ben Vinegar and Anton Kovalyov - [Effective JavaScript: 68 Specific Ways to Harness the Power of JavaScript](http://amzn.com/0321812182) - David Herman - [Eloquent JavaScript](http://eloquentjavascript.net/) - Marijn Haverbeke **博客** - [DailyJS](http://dailyjs.com/) - [JavaScript Weekly](http://javascriptweekly.com/) - [JavaScript, JavaScript...](http://javascriptweblog.wordpress.com/) - [Bocoup Weblog](http://weblog.bocoup.com/) - [Adequately Good](http://www.adequatelygood.com/) - [NCZOnline](http://www.nczonline.net/) - [Perfection Kills](http://perfectionkills.com/) - [Ben Alman](http://benalman.com/) - [Dmitry Baranovskiy](http://dmitry.baranovskiy.com/) - [Dustin Diaz](http://dustindiaz.com/) - [nettuts](http://net.tutsplus.com/?s=javascript) **播客** - [JavaScript Jabber](http://devchat.tv/js-jabber/) **[⬆ 返回目录](#table-of-contents)** ## 使用情况 下列组织应用这份风格指南。 - **3blades**: [3Blades/javascript](https://github.com/3blades/javascript) - **4Catalyzer**: [4Catalyzer/javascript](https://github.com/4Catalyzer/javascript) - **Aan Zee**: [AanZee/javascript](https://github.com/AanZee/javascript) - **Adult Swim**: [adult-swim/javascript](https://github.com/adult-swim/javascript) - **Airbnb**: [airbnb/javascript](https://github.com/airbnb/javascript) - **AltSchool**: [AltSchool/javascript](https://github.com/AltSchool/javascript) - **Apartmint**: [apartmint/javascript](https://github.com/apartmint/javascript) - **Ascribe**: [ascribe/javascript](https://github.com/ascribe/javascript) - **Avalara**: [avalara/javascript](https://github.com/avalara/javascript) - **Avant**: [avantcredit/javascript](https://github.com/avantcredit/javascript) - **Axept**: [axept/javascript](https://github.com/axept/javascript) - **BashPros**: [BashPros/javascript](https://github.com/BashPros/javascript) - **Billabong**: [billabong/javascript](https://github.com/billabong/javascript) - **Bisk**: [bisk/javascript](https://github.com/Bisk/javascript/) - **Bonhomme**: [bonhommeparis/javascript](https://github.com/bonhommeparis/javascript) - **Brainshark**: [brainshark/javascript](https://github.com/brainshark/javascript) - **CaseNine**: [CaseNine/javascript](https://github.com/CaseNine/javascript) - **Chartboost**: [ChartBoost/javascript-style-guide](https://github.com/ChartBoost/javascript-style-guide) - **ComparaOnline**: [comparaonline/javascript](https://github.com/comparaonline/javascript-style-guide) - **Compass Learning**: [compasslearning/javascript-style-guide](https://github.com/compasslearning/javascript-style-guide) - **DailyMotion**: [dailymotion/javascript](https://github.com/dailymotion/javascript) - **DoSomething**: [DoSomething/eslint-config](https://github.com/DoSomething/eslint-config) - **Digitpaint** [digitpaint/javascript](https://github.com/digitpaint/javascript) - **Ecosia**: [ecosia/javascript](https://github.com/ecosia/javascript) - **Evernote**: [evernote/javascript-style-guide](https://github.com/evernote/javascript-style-guide) - **Evolution Gaming**: [evolution-gaming/javascript](https://github.com/evolution-gaming/javascript) - **EvozonJs**: [evozonjs/javascript](https://github.com/evozonjs/javascript) - **ExactTarget**: [ExactTarget/javascript](https://github.com/ExactTarget/javascript) - **Expensify** [Expensify/Style-Guide](https://github.com/Expensify/Style-Guide/blob/master/javascript.md) - **Flexberry**: [Flexberry/javascript-style-guide](https://github.com/Flexberry/javascript-style-guide) - **Gawker Media**: [gawkermedia/javascript](https://github.com/gawkermedia/javascript) - **General Electric**: [GeneralElectric/javascript](https://github.com/GeneralElectric/javascript) - **Generation Tux**: [GenerationTux/javascript](https://github.com/generationtux/styleguide) - **GoodData**: [gooddata/gdc-js-style](https://github.com/gooddata/gdc-js-style) - **Grooveshark**: [grooveshark/javascript](https://github.com/grooveshark/javascript) - **Honey**: [honeyscience/javascript](https://github.com/honeyscience/javascript) - **How About We**: [howaboutwe/javascript](https://github.com/howaboutwe/javascript-style-guide) - **Huballin**: [huballin/javascript](https://github.com/huballin/javascript) - **HubSpot**: [HubSpot/javascript](https://github.com/HubSpot/javascript) - **Hyper**: [hyperoslo/javascript-playbook](https://github.com/hyperoslo/javascript-playbook/blob/master/style.md) - **InterCity Group**: [intercitygroup/javascript-style-guide](https://github.com/intercitygroup/javascript-style-guide) - **Jam3**: [Jam3/Javascript-Code-Conventions](https://github.com/Jam3/Javascript-Code-Conventions) - **JeopardyBot**: [kesne/jeopardy-bot](https://github.com/kesne/jeopardy-bot/blob/master/STYLEGUIDE.md) - **JSSolutions**: [JSSolutions/javascript](https://github.com/JSSolutions/javascript) - **KickorStick**: [kickorstick/javascript](https://github.com/kickorstick/javascript) - **Kinetica Solutions**: [kinetica/javascript](https://github.com/kinetica/Javascript-style-guide) - **Lonely Planet**: [lonelyplanet/javascript](https://github.com/lonelyplanet/javascript) - **M2GEN**: [M2GEN/javascript](https://github.com/M2GEN/javascript) - **Mighty Spring**: [mightyspring/javascript](https://github.com/mightyspring/javascript) - **MinnPost**: [MinnPost/javascript](https://github.com/MinnPost/javascript) - **MitocGroup**: [MitocGroup/javascript](https://github.com/MitocGroup/javascript) - **ModCloth**: [modcloth/javascript](https://github.com/modcloth/javascript) - **Money Advice Service**: [moneyadviceservice/javascript](https://github.com/moneyadviceservice/javascript) - **Muber**: [muber/javascript](https://github.com/muber/javascript) - **National Geographic**: [natgeo/javascript](https://github.com/natgeo/javascript) - **Nimbl3**: [nimbl3/javascript](https://github.com/nimbl3/javascript) - **Nulogy**: [nulogy/javascript](https://github.com/nulogy/javascript) - **Orange Hill Development**: [orangehill/javascript](https://github.com/orangehill/javascript) - **Orion Health**: [orionhealth/javascript](https://github.com/orionhealth/javascript) - **OutBoxSoft**: [OutBoxSoft/javascript](https://github.com/OutBoxSoft/javascript) - **Peerby**: [Peerby/javascript](https://github.com/Peerby/javascript) - **Razorfish**: [razorfish/javascript-style-guide](https://github.com/razorfish/javascript-style-guide) - **reddit**: [reddit/styleguide/javascript](https://github.com/reddit/styleguide/tree/master/javascript) - **React**: [facebook.github.io/react/contributing/how-to-contribute.html#style-guide](https://facebook.github.io/react/contributing/how-to-contribute.html#style-guide) - **REI**: [reidev/js-style-guide](https://github.com/rei/code-style-guides/blob/master/docs/javascript.md) - **Ripple**: [ripple/javascript-style-guide](https://github.com/ripple/javascript-style-guide) - **SeekingAlpha**: [seekingalpha/javascript-style-guide](https://github.com/seekingalpha/javascript-style-guide) - **Shutterfly**: [shutterfly/javascript](https://github.com/shutterfly/javascript) - **Sourcetoad**: [sourcetoad/javascript](https://github.com/sourcetoad/javascript) - **Springload**: [springload/javascript](https://github.com/springload/javascript) - **StratoDem Analytics**: [stratodem/javascript](https://github.com/stratodem/javascript) - **SteelKiwi Development**: [steelkiwi/javascript](https://github.com/steelkiwi/javascript) - **StudentSphere**: [studentsphere/javascript](https://github.com/studentsphere/guide-javascript) - **SwoopApp**: [swoopapp/javascript](https://github.com/swoopapp/javascript) - **SysGarage**: [sysgarage/javascript-style-guide](https://github.com/sysgarage/javascript-style-guide) - **Syzygy Warsaw**: [syzygypl/javascript](https://github.com/syzygypl/javascript) - **Target**: [target/javascript](https://github.com/target/javascript) - **TheLadders**: [TheLadders/javascript](https://github.com/TheLadders/javascript) - **The Nerdery**: [thenerdery/javascript-standards](https://github.com/thenerdery/javascript-standards) - **T4R Technology**: [T4R-Technology/javascript](https://github.com/T4R-Technology/javascript) - **VoxFeed**: [VoxFeed/javascript-style-guide](https://github.com/VoxFeed/javascript-style-guide) - **WeBox Studio**: [weboxstudio/javascript](https://github.com/weboxstudio/javascript) - **Weggo**: [Weggo/javascript](https://github.com/Weggo/javascript) - **Zillow**: [zillow/javascript](https://github.com/zillow/javascript) - **ZocDoc**: [ZocDoc/javascript](https://github.com/ZocDoc/javascript) **[⬆ 返回目录](#table-of-contents)** ## 翻译 这份风格指南也有其他语言的译本： - ![br](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Brazil.png) **Brazilian Portuguese**: [armoucar/javascript-style-guide](https://github.com/armoucar/javascript-style-guide) - ![bg](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Bulgaria.png) **Bulgarian**: [borislavvv/javascript](https://github.com/borislavvv/javascript) - ![ca](https://raw.githubusercontent.com/fpmweb/javascript-style-guide/master/img/catala.png) **Catalan**: [fpmweb/javascript-style-guide](https://github.com/fpmweb/javascript-style-guide) - ![tw](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Taiwan.png) **Chinese(Traditional)**: [jigsawye/javascript](https://github.com/jigsawye/javascript) - ![cn](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/China.png) **Chinese(Simplified)**: [yuche/javascript](https://github.com/yuche/javascript) - ![fr](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/France.png) **French**: [nmussy/javascript-style-guide](https://github.com/nmussy/javascript-style-guide) - ![de](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Germany.png) **German**: [timofurrer/javascript-style-guide](https://github.com/timofurrer/javascript-style-guide) - ![it](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Italy.png) **Italian**: [sinkswim/javascript-style-guide](https://github.com/sinkswim/javascript-style-guide) - ![jp](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Japan.png) **Japanese**: [mitsuruog/javacript-style-guide](https://github.com/mitsuruog/javacript-style-guide) - ![kr](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/South-Korea.png) **Korean**: [tipjs/javascript-style-guide](https://github.com/tipjs/javascript-style-guide) - ![pl](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Poland.png) **Polish**: [mjurczyk/javascript](https://github.com/mjurczyk/javascript) - ![ru](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Russia.png) **Russian**: [uprock/javascript](https://github.com/uprock/javascript) - ![es](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Spain.png) **Spanish**: [paolocarrasco/javascript-style-guide](https://github.com/paolocarrasco/javascript-style-guide) - ![th](https://raw.githubusercontent.com/gosquared/flags/master/flags/flags/shiny/24/Thailand.png) **Thai**: [lvarayut/javascript-style-guide](https://github.com/lvarayut/javascript-style-guide) ## JavaScript 编码规范说明 - [Reference](https://github.com/airbnb/javascript/wiki/The-JavaScript-Style-Guide-Guide) ## 讨论 JavaScript - 欢迎到 [gitter](https://gitter.im/airbnb/javascript) 与我们聊天（英文）。 ## 贡献者 - [查看原始项目贡献者](https://github.com/airbnb/javascript/graphs/contributors) - [查看简中翻译贡献者](https://github.com/yuche/javascript/graphs/contributors) ## 许可协议 (The MIT License) Copyright (c) 2014 Airbnb Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. **[⬆ 返回目录](#table-of-contents)** ## 修订 我们鼓励您派生本指南和更改规则以适应您的团队需求。您可以在下方列出对本风格指南的修改，以便定期更新本指南而无需处理合并冲突。 # }; ================================================ FILE: docs/web/es6.md ================================================ # JavaScript ES6 规范 ## ES6 简介 ECMAScript 6 简称 ES6，是 JavaScript 语言的下一代标准，已经在2015年6月正式发布了。它的目标是使得 JavaScript 语言可以用来编写复杂的大型应用程序，成为企业级开发语言。 ECMAScript 和 JavaScript 的关系：前者是后者的语法规格，后者是前者的一种实现 [Babel](http://babeljs.io/)：将ES6代码转为ES5代码 ## 新特性 ### let、const let 定义的变量不会被变量提升，const 定义的常量不能被修改，let 和 const 都是块级作用域 ES6前，js 是没有块级作用域 {} 的概念的。（有函数作用域、全局作用域、eval作用域） ES6后，let 和 const 的出现，js 也有了块级作用域的概念，前端的知识是日新月异的~ 变量提升：在ES6以前，var关键字声明变量。无论声明在何处，都会被视为声明在函数的最顶部；不在函数内即在全局作用域的最顶部。这样就会引起一些误解。例如： ```js console.log(a); // undefined var a = 'hello'; # 上面的代码相当于 var a; console.log(a); a = 'hello'; # 而 let 就不会被变量提升 console.log(a); // a is not defined let a = 'hello'; ``` const 定义的常量不能被修改 ```js var name = "bai"; name = "ming"; console.log(name); // ming const name = "bai"; name = "ming"; // Assignment to constant variable. console.log(name); ``` ### import、export import导入模块、export导出模块 ```js // 全部导入 import people from './example' // 将整个模块当作单一对象进行导入，该模块的所有导出都会作为对象的属性存在 import * as example from "./example.js" console.log(example.name) console.log(example.getName()) // 导入部分，引入非 default 时，使用花括号 import {name, age} from './example' // 导出默认, 有且只有一个默认 export default App // 部分导出 export class App extend Component {}; ``` ### class、extends、super ES5中最令人头疼的的几个部分：原型、构造函数，继承，有了ES6我们不再烦恼！ ES6引入了Class（类）这个概念。 ```js class Animal { constructor() { this.type = 'animal'; } says(say) { console.log(this.type + ' says ' + say); } } let animal = new Animal(); animal.says('hello'); //animal says hello class Cat extends Animal { constructor() { super(); this.type = 'cat'; } } let cat = new Cat(); cat.says('hello'); //cat says hello ``` 上面代码首先用class定义了一个“类”，可以看到里面有一个constructor方法，这就是构造方法，而this关键字则代表实例对象。简单地说，constructor内定义的方法和属性是实例对象自己的，而constructor外定义的方法和属性则是所有实力对象可以共享的。 Class之间可以通过extends关键字实现继承，这比ES5的通过修改原型链实现继承，要清晰和方便很多。上面定义了一个Cat类，该类通过extends关键字，继承了Animal类的所有属性和方法。 super关键字，它指代父类的实例（即父类的this对象）。子类必须在constructor方法中调用super方法，否则新建实例时会报错。这是因为子类没有自己的this对象，而是继承父类的this对象，然后对其进行加工。如果不调用super方法，子类就得不到this对象。 ES6的继承机制，实质是先创造父类的实例对象this（所以必须先调用super方法），然后再用子类的构造函数修改this。 ```js // ES5 var Shape = function(id, x, y) { this.id = id, this.move(x, y); }; Shape.prototype.move = function(x, y) { this.x = x; this.y = y; }; var Rectangle = function id(ix, x, y, width, height) { Shape.call(this, id, x, y); this.width = width; this.height = height; }; Rectangle.prototype = Object.create(Shape.prototype); Rectangle.prototype.constructor = Rectangle; var Circle = function(id, x, y, radius) { Shape.call(this, id, x, y); this.radius = radius; }; Circle.prototype = Object.create(Shape.prototype); Circle.prototype.constructor = Circle; // ES6 class Shape { constructor(id, x, y) { this.id = id this.move(x, y); } move(x, y) { this.x = x this.y = y; } } class Rectangle extends Shape { constructor(id, x, y, width, height) { super(id, x, y) this.width = width this.height = height; } } class Circle extends Shape { constructor(id, x, y, radius) { super(id, x, y) this.radius = radius; } } ``` ### arrow functions （箭头函数） 函数的快捷写法。不需要 function 关键字来创建函数，省略 return 关键字，继承当前上下文的 this 关键字 ```js // ES5 var arr1 = [1, 2, 3]; var newArr1 = arr1.map(function(x) { return x + 1; }); // ES6 let arr2 = [1, 2, 3]; let newArr2 = arr2.map((x) => { x + 1 }); ``` 箭头函数小细节：当你的函数有且仅有一个参数的时候，是可以省略掉括号的；当你函数中有且仅有一个表达式的时候可以省略{} ```js let arr2 = [1, 2, 3]; let newArr2 = arr2.map(x => x + 1); ``` JavaScript语言的this对象一直是一个令人头痛的问题，运行上面的代码会报错，这是因为setTimeout中的this指向的是全局对象。 ```js class Animal { constructor() { this.type = 'animal'; } says(say) { setTimeout(function() { console.log(this.type + ' says ' + say); }, 1000); } } var animal = new Animal(); animal.says('hi'); //undefined says hi ``` 解决办法： ```js // 传统方法1: 将this传给self,再用self来指代this says(say) { var self = this; setTimeout(function() { console.log(self.type + ' says ' + say); }, 1000); } // 传统方法2: 用bind(this),即 says(say) { setTimeout(function() { console.log(this.type + ' says ' + say); }.bind(this), 1000); } // ES6: 箭头函数 // 当我们使用箭头函数时，函数体内的this对象，就是定义时所在的对象 says(say) { setTimeout(() => { console.log(this.type + ' says ' + say); }, 1000); } ``` ### template string （模板字符串） 解决了 ES5 在字符串功能上的痛点。 第一个用途：字符串拼接。将表达式嵌入字符串中进行拼接，用 ` 和${}`来界定。 ```js // es5 var name1 = "bai"; console.log('hello' + name1); // es6 const name2 = "ming"; console.log(`hello${name2}`); ``` 第二个用途：在ES5时我们通过反斜杠来做多行字符串拼接。ES6反引号 `` 直接搞定。 ```js // es5 var msg = "Hi \ man!"; // es6 const template = `
      hello world
      `; ``` 另外：includes repeat ```js // includes：判断是否包含然后直接返回布尔值 let str = 'hahah'; console.log(str.includes('y')); // false // repeat: 获取字符串重复n次 let s = 'he'; console.log(s.repeat(3)); // 'hehehe' ``` ### destructuring （解构） 简化数组和对象中信息的提取。 ES6前，我们一个一个获取对象信息； ES6后，解构能让我们从对象或者数组里取出数据存为变量 ```js // ES5 var people1 = { name: 'bai', age: 20, color: ['red', 'blue'] }; var myName = people1.name; var myAge = people1.age; var myColor = people1.color[0]; console.log(myName + '----' + myAge + '----' + myColor); // ES6 let people2 = { name: 'ming', age: 20, color: ['red', 'blue'] } let { name, age } = people2; let [first, second] = people2.color; console.log(`${name}----${age}----${first}`); ``` ### default 函数默认参数 ```js // ES5 给函数定义参数默认值 function foo(num) { num = num || 200; return num; } // ES6 function foo(num = 200) { return num; } ``` ### rest arguments （rest参数） 解决了 es5 复杂的 arguments 问题 ```js function foo(x, y, ...rest) { return ((x + y) * rest.length); } foo(1, 2, 'hello', true, 7); // 9 ``` ### Spread Operator （展开运算符） 第一个用途：组装数组 ```js let color = ['red', 'yellow']; let colorful = [...color, 'green', 'blue']; console.log(colorful); // ["red", "yellow", "green", "blue"] ``` 第二个用途：获取数组除了某几项的其他项 ```js let num = [1, 3, 5, 7, 9]; let [first, second, ...rest] = num; console.log(rest); // [5, 7, 9] ``` ### 对象 对象初始化简写 ```js // ES5 function people(name, age) { return { name: name, age: age }; } // ES6 function people(name, age) { return { name, age }; } ``` 对象字面量简写（省略冒号与 function 关键字） ```js // ES5 var people1 = { name: 'bai', getName: function () { console.log(this.name); } }; // ES6 let people2 = { name: 'bai', getName () { console.log(this.name); } }; ``` 另外：Object.assign() ES6 对象提供了Object.assign()这个方法来实现浅复制。Object.assign()可以把任意多个源对象自身可枚举的属性拷贝给目标对象，然后返回目标对象。第一参数即为目标对象。在实际项目中，我们为了不改变源对象。一般会把目标对象传为{} ```js const obj = Object.assign({}, objA, objB) // 给对象添加属性 this.seller = Object.assign({}, this.seller, response.data) ``` ### Promise 用同步的方式去写异步代码 ```js // 发起异步请求 fetch('/api/todos') .then(res => res.json()) .then(data => ({ data })) .catch(err => ({ err })); ``` ### Generators 生成器（ generator）是能返回一个迭代器的函数。 生成器函数也是一种函数，最直观的表现就是比普通的function多了个星号*，在其函数体内可以使用yield关键字,有意思的是函数会在每个yield后暂停。 这里生活中有一个比较形象的例子。咱们到银行办理业务时候都得向大厅的机器取一张排队号。你拿到你的排队号，机器并不会自动为你再出下一张票。也就是说取票机“暂停”住了，直到下一个人再次唤起才会继续吐票。 迭代器：当你调用一个generator时，它将返回一个迭代器对象。这个迭代器对象拥有一个叫做next的方法来帮助你重启generator函数并得到下一个值。next方法不仅返回值，它返回的对象具有两个属性：done和value。value是你获得的值，done用来表明你的generator是否已经停止提供值。继续用刚刚取票的例子，每张排队号就是这里的value，打印票的纸是否用完就这是这里的done。 ```js // 生成器 function *createIterator() { yield 1; yield 2; yield 3; } // 生成器能像正规函数那样被调用，但会返回一个迭代器 let iterator = createIterator(); console.log(iterator.next().value); // 1 console.log(iterator.next().value); // 2 console.log(iterator.next().value); // 3 ``` 迭代器对异步编程作用很大，异步调用对于我们来说是很困难的事，我们的函数并不会等待异步调用完再执行，你可能会想到用回调函数，（当然还有其他方案比如Promise比如Async/await）。 生成器可以让我们的代码进行等待。就不用嵌套的回调函数。使用generator可以确保当异步调用在我们的generator函数运行一下行代码之前完成时暂停函数的执行。 那么问题来了，咱们也不能手动一直调用next()方法，你需要一个能够调用生成器并启动迭代器的方法。就像这样子的： ```js function run(taskDef) { // taskDef 即一个生成器函数 // 创建迭代器，让它在别处可用 let task = taskDef(); // 启动任务 let result = task.next(); // 递归使用函数来保持对 next() 的调用 function step() { // 如果还有更多要做的 if (!result.done) { result = task.next(); step(); } } // 开始处理过程 step(); } ``` ## 总结 以上就是 ES6 最常用的一些语法，可以说这20%的语法，在ES6的日常使用中占了80% [更多ES6语法点击这里](http://es6.ruanyifeng.com/) [《JavaScript 语言入门教程》](https://wangdoc.com/) ================================================ FILE: docs/web/javascript.md ================================================ # JavaScript 基础 ## 一、概念简介 JavaScript 是一种专为与网页交互而设计的脚本语言，由以下三个部分组成： - **ECMAScript**：由 ECMA-262 定义，提供核心语言功能； - **文档对象模型 (DOM)**：提供访问和操作网页内容的方法和接口； - **浏览器对象模型 (BOM)**：提供与浏览器交互的方法和接口。 ECMAScript 提供了语言的核心功能，它定义了以下七种数据类型： - **六种基本数据类型**：`Undefined`，`Null`，`Boolean`，`Number`，`String`，`Symbol` ( ES 6新增 )； - **一种引用数据类型**：统称为 Object 类型；具体又细分为 `Object`，`Array`，`Date`，`RegExp`，`Function` 等类型。另外和 Java 语言类似，对于布尔，数值，字符串等基本类型，分别存在其对应的包装类型 Boolean，Number，String，但通常我们并不会使用到这些包装类型，只需要使用其基本类型即可。 ## 二、基本类型 ### 2.1 数值类型 **1. 进制数值** ECMAScript 中的 Number 支持以下三种常用进制： + **十进制**：正常数值就是十进制； + **八进制**：八进制字面值的第一位必须是零（0），然后是八进制数字序列（0～7）； + **十六进制**：十六进制字面值的前两位必须是 0x，后跟任意的十六进制数字（0～9 及 A～F）。 ```javascript console.log(56); // 56 console.log(070); // 56 console.log(0x38); // 56 ``` **2. 浮点数值** ECMAScript 的数值类型同样支持浮点数，但是由于保存浮点数值需要的内存空间是保存整数值的两倍，因此 ECMAScript 会尽量将浮点数值转换为整数值存储： ```javascript var a = 10.0; console.log(a); // 10 ``` 和其他语言类似，浮点数中的数值也是不精准的，示例如下： ```javascript var a = 0.1; var b = 0.2; a + b ; // 0.30000000000000004 a+b === 0.3 ; // false ``` 如果想要对浮点数进行精确计算，可以使用 [decimal.js](https://github.com/MikeMcl/decimal.js) ，[ math.js](https://github.com/josdejong/mathjs) 等第三方库。 **3. 科学计数法** ECMAScript 支持使用科学计数法来表达数值： ```javascript 8e-2 // 0.08 8e2 // 800 ``` **4. parseInt() \ parseFloat()** parseInt 可以用于解析字符串并返回整数，parseFloat 用于解析字符串并返回浮点数： ```javascript parseInt("56"); // 56 parseInt("0x38", 16); // 56 支持使用第二个参数来表示转换的进制 parseInt("56.6"); // 56 parseFloat("12.2"); // 12.2 parseInt("blue"); // NaN NaN用于表示一个本来要返回数值的操作却未返回数值的情况 ``` **5. toFixed()** toFixed 用于保留指定位数的小数，但需要注意的是其四舍五入的行为是不确定的： ```javascript 1.35.toFixed(1) // 1.4 正确 1.335.toFixed(2) // 1.33 错误 1.3335.toFixed(3) // 1.333 错误 1.33335.toFixed(4) // 1.3334 正确 1.333335.toFixed(5) // 1.33333 错误 1.3333335.toFixed(6) // 1.333333 错误 ``` 想要解决这个问题，需要重写 toFixed 方法并通过判断最后一位是否大于或等于5来决定是否需要进位，具体代码如下： ```javascript // toFixed兼容方法 Number.prototype.toFixed = function(len){ if(len>20 || len<0){ throw new RangeError('toFixed() digits argument must be between 0 and 20'); } // .123转为0.123 var number = Number(this); if (isNaN(number) || number >= Math.pow(10, 21)) { return number.toString(); } if (typeof (len) == 'undefined' || len == 0) { return (Math.round(number)).toString(); } var result = number.toString(), numberArr = result.split('.'); if(numberArr.length<2){ //整数的情况 return padNum(result); } var intNum = numberArr[0], //整数部分 deciNum = numberArr[1],//小数部分 lastNum = deciNum.substr(len, 1);//最后一个数字 if(deciNum.length == len){ //需要截取的长度等于当前长度 return result; } if(deciNum.length < len){ //需要截取的长度大于当前长度 1.3.toFixed(2) return padNum(result) } //需要截取的长度小于当前长度，需要判断最后一位数字 result = intNum + '.' + deciNum.substr(0, len); if(parseInt(lastNum, 10)>=5){ //最后一位数字大于5，要进位 var times = Math.pow(10, len); //需要放大的倍数 var changedInt = Number(result.replace('.',''));//截取后转为整数 changedInt++;//整数进位 changedInt /= times;//整数转为小数，注：有可能还是整数 result = padNum(changedInt+''); } return result; //对数字末尾加0 function padNum(num){ var dotPos = num.indexOf('.'); if(dotPos === -1){ //整数的情况 num += '.'; for(var i = 0;i 参考自：[*js中小数四舍五入和浮点数的研究*](http://caibaojian.com/js-tofixed.html) ### 2.2 字符类型 **1. 字符串表示** ECMAScript 支持使用双引号 ` " ` 或单引号 ` ' ` 来表示字符串，并且 ECMAScript 中的字符串是不可变的，也就是说，字符串一旦创建，它们的值就不能改变。要改变某个变量保存的字符串，首先要销毁原来的字符串，然后再用另一个包含新值的字符串填充该变量，示例如下： ```javascript var lang = "Java"; /*程序会创建一个能容纳 10 个字符的新字符串用于填充"Java"和"Script"，之后再销毁原有的字符串"Java"和"Script"*/ lang = lang + "Script"; ``` **2. 转换为字符串** 要把一个值转换为一个字符串有两种方式： + 使用对象方法 **toString()** ：大多数对象都具有这个方法，但需要注意的是 null 和 undefined 没有； + 使用转型函数 **String()** ：使用该转型函数时，如果传入的值有 toString() 方法，则调用该方法并返回相应的结果；如果传入的值是 null，则返回 "null" ；如果传入值是 undefined，则返回 "undefined" 。 示例如下： ```javascript var a = null; a.toString() // Uncaught TypeError: Cannot read property 'toString' of null String(a) // "null" ``` **3. 常用的字符串操作** + **concat()** ：用于拼接一个或多个字符串； + **slice()** ：用于截取字符串，接收两个参数，分别代表截取的开始位置和结束位置； + **substring()**：用于截取字符串，接收两个参数，分别代表截取的开始位置和结束位置； + **substr()** ：用于截取字符串，接收两个参数，分别代表截取的开始位置和截取的长度； + **indexOf() \ lastIndexOf()** ：均接收两个参数，分别代表待查找的字符串和查找的开始位置； + **trim()** ：用于去除字符串前后的空格。 slice，substring，substr 等方法在传入正数参数时，其行为比较好预期，但传递参数是负数时，则具体的行为表现如下: + **slice()** ：会将传入的负值与字符串的长度相加； + **substring()** ：方法会把所有负值参数都转换为 0 ； + **substr()** ：会将第一个负值参数加上字符串的长度，如果传递了第二个参数且为负数时候，会将其转换为 0 。 ```javascript var stringValue = "hello world"; // 只接收一个参数时 alert(stringValue.slice(3)); // "lo world" alert(stringValue.substring(3)); // "lo world" alert(stringValue.substr(3)); // "lo world" // 接收两个参数时候 alert(stringValue.slice(3, 7)); // "lo w" alert(stringValue.substring(3,7)); // "lo w" alert(stringValue.substr(3, 7)); // "lo worl" // 当第一个参数为负值时 alert(stringValue.slice(-3)); // "rld" 按照规则等价于： slice(8) alert(stringValue.substring(-3)); // "hello world" 按照规则等价于： substring(0) alert(stringValue.substr(-3)); // "rld" 按照规则等价于： substr(8) // 当第二个参数为负值时 alert(stringValue.slice(3, -4)); // "lo w" 按照规则等价于： slice(3,7) alert(stringValue.substring(3, -4)); // "hel" 按照规则等价于： substring(3,0) alert(stringValue.substr(3, -4)); // ""（空字符串） 按照规则等价于： substr(3,0) ``` ### 2.3 基本类型检测 JavaScript 是一种弱类型的语言，在声明变量时候可以不必指明其具体类型，而是由程序进行推断。如果想要知道变量具体属于哪一个基础类型，可以使用 **typeof** 关键字，它的返回情况如下： - **undefined**：如果对应的值未定义； - **boolean**：如果对应的值是布尔值； - **string**：如果对应的值是字符串； - **number**：如果对应的值是数值； - **object**：如果对应的值是对象或 null； - **function**：如果对应的值是函数则返回 function。 函数在本质上也是对象，但是由于其一等公民的特殊地位，所以将其和其他普通对象进行区分是很有必要的，因此 typeof 对其检测时会返回 function ，而不是 object 。 ## 三、引用类型 ### 3.1 Object 类型 创建 Object 实例有以下两种方式： + 使用 new 操作符后跟着 Object 构造函数； + 使用对象字面量的方式。 ```javascript // 1. 使用new操作符 var user = new Object(); user.name = "heibaiying"; user.age = 30; // 2. 使用对象字面量 var user = { name: "heibaiying", age: 30 }; ``` ### 3.2 Array 类型 创建数组也有两种方式，基于构造函数的方式和基于对象字面量的方式： ```javascript // 1.基于构造函数的方式 var colors = new Array(); var colors = new Array(20); var colors = new Array("red", "blue", "green"); // 2.基于对象字面量的方式 var names = []; var colors = ["red", "blue", "green"]; ``` 数组的长度保存在其 length 属性中，和其他语言中的 length 属性不同，这个值是不是只读的，可以用其进行数组的截断操作或添加新的数据项，示例如下： ```javascript var colors = ["red", "blue", "green"]; colors.length = 2; // ["red", "blue"] colors[colors.length] = "green"; // ["red", "blue", "green"] colors[10] = "black"; // ["red", "blue", "green", empty × 7, "black"] ``` 数组的其他常用方法如下： **1. 检测数组** ```javascript colors instanceof Array Array.isArray(colors) ``` **2. 转换方法** ```java var colors = ["red", "blue", "green"]; colors.valueOf(); // [ 'red', 'blue', 'green' ] colors; // [ 'red', 'blue', 'green' ] colors.toString(); // red,blue,green colors.join("|"); // red|blue|green ``` **3. 栈方法** ECMAScript 的数组提供了类似栈的特性，能够实现后进先出： ```javascript var colors = ["red", "blue", "green"]; colors.push("black"); // ["red", "blue", "green", "black"] colors.pop() // "black" colors // ["red", "blue", "green"] ``` **4. 队列方法** ECMAScript 的数组提供了类似栈的特性，能够实现先进先出： ```javascript colors.push("black","yellow"); // ["red", "blue", "green", "black", "yellow"] colors.shift() // "red" colors // ["blue", "green", "black", "yellow"] ``` **5. 重排序方法** ```javascript var values = [1, 2, 3, 4, 5]; values.reverse(); values // [5, 4, 3, 2, 1] // 支持传入排序函数进行自定义排序 function compare(value1, value2) { if (value1 < value2) { return -1; } else if (value1 > value2) { return 1; } else { return 0; } } values.sort(compare) values // [1, 2, 3, 4, 5] ``` **6. 操作方法** **concat()** 用于拼接并返回新的数组： ```javascript var colors = ["red", "green", "blue"]; var colors2 = colors.concat("yellow", ["black", "brown"]); colors // ["red", "green", "blue"] colors2 // ["red", "green", "blue", "yellow", "black", "brown"] ``` **slice()** 用于截取数组并返回新的数组，它接收两个参数，分别代表截取的开始位置和结束位置，它是一个前开后闭的区间： ```javascript var colors = ["red", "green", "blue", "yellow", "purple"]; var colors2 = colors.slice(1); // ["green", "blue", "yellow", "purple"] var colors3 = colors.slice(0,2); // ["red", "green"] ``` **splice()** 用于删除并在删除位置新增数据项，它接收任意个参数，其中第一个参数为删除的开始位置，第二个参数为删除多少个数据项，之后可以接任意个参数，用于表示待插入的数据项： ```javascript var colors = ["red", "green", "blue", "yellow"]; colors.splice(1,2) // 返回删除的数据项：["green", "blue"] colors // ["red", "yellow"] colors.splice(1,0,"black","green") // [] colors // ["red", "black", "green", "yellow"] ``` **7. 位置方法** **indexOf()** 和 **lastIndexOf()** 用于查找指定元素的 Index ，它们都接收两个参数：待查找项和查找的起点位置： ```shell var colors = ["red", "green", "blue", "yellow", "green", "blue"]; colors.indexOf("green"); // 1 colors.indexOf("green", 3); // 4 colors.lastIndexOf("green"); // 4 colors.lastIndexOf("green", 3); // 1 ``` **8. 迭代方法** ECMAScript 5 提供了五个迭代方法： - **every()**：判断数组中的每个元素是否满足指定条件，如果全部满足则返回 true，否则返回 flase； - **some()**：判断数组中的每个元素是否满足指定条件，只要有一个满足则返回 true，否则返回 flase； - **filter()**：过滤并返回符合条件的元素组成的数组。 - **forEach()**：对数组中的每一项运行给定函数。 - **map()**：对数组中的每一项运行给定函数，并返回每次函数调用结果所组成的数组。 ```javascript var numbers = [1, 2, 3, 4, 5, 4, 3, 2, 1]; numbers.every(function (value, index, array) { return value > 3; }); // false numbers.some(function (value, index, array) { return value > 3; }); // true numbers.filter(function (value, index, array) { return value > 3; }); // [4, 5, 4] numbers.forEach(function (value, index, array) { console.log(value); }); numbers.map(function (value, index, array) { return value * 10; }); // [10, 20, 30, 40, 50, 40, 30, 20, 10] ``` **9. 归并方法** ECMAScript 5 提供了两个归并数组的方法： **reduce()** 和 **reduceRight()** 。 它们都接收四个参数：前一个值、当前值、当前项的索引 和 数组本身，使用示例如下： ```javascript var values = [1, 2, 3, 4, 5]; var sum01 = values.reduce(function (prev, cur, index, array) { return prev + cur; }); // 15 var sum02 = values.reduceRight(function (prev, cur, index, array) { return prev + cur; }); // 15 ``` ### 3.3 Date 类型 创建一个日期对象，可以使用 new 操作符和 Date 构造函数： ```java var now = new Date(); now.toLocaleString() // 2019-9-14 9:53:59 AM var date = new Date(2018, 7, 8, 8, 30, 20); date.toLocaleString(); // 2018-8-8 8:30:20 AM ``` 如果你只想知道当前时间的毫秒数，可以直接使用 Date 对象的静态方法： ```javascript Date.now() 1568426130593 ``` **1. 格式转换** - **toLocaleString()** ：按照浏览器所在时区返回相应的日期格式； - **toString()** ：返回日期时间数据和的时区数据； - **valueOf()** ：返回日期的时间戳格式。 ```javascript var date = new Date(2018, 7, 8, 8, 30, 20); console.log(date.toLocaleString()); // 2018-8-8 8:30:20 AM console.log(date.toString()); // Wed Aug 08 2018 08:30:20 GMT+0800 (GMT+08:00) console.log(date.valueOf()); // 1533688220000 ``` 由于 **valueOf()** 返回的是日期的时间戳格式，所以对于 date 对象，可以直接使用比较运算符来比较其大小： ```javascript var date01 = new Date(2018, 7, 8, 8, 30, 20); var date02 = new Date(2016, 7, 8, 8, 30, 20); console.log(date01 > date02); // true console.log(date01 < date02); // flase ``` **2. 常用方法** - **getTime() \ setTime(毫秒)** ：返回和设置整个日期的所代表的毫秒数；与 valueOf() 方法返回的值相同； - **getFullYear() \ setFullYear(年)** ：返回和设置4位数的年份； - **getMonth() \ setMonth(月)** ：返回和设置月份，其中 0 表示一月， 11 表示十二月； - **getDate() \ setDate(日)** ：返回和设置月份中的天数（1到31）； - **getDay()** ：返回和设置星期几 ( 其中0表示星期日， 6表示星期六）； - **getHours() \ setHours(时)** ：返回和设置小时数（0到23）； - **getMinutes() \ setMinutes(分)** ：返回和设置日期中的分钟数（0到59）； - **getSeconds() \ setSeconds(秒)** ：返回和设置日期中的秒数（0到59）； - **getMilliseconds() \ setMilliseconds(毫秒)** ：返回和设置日期中的毫秒数。 ### 3.4 Funcation 类型 **1. 函数参数** ECMAScript 使用 function 关键字来声明函数，但和其他语言不同的是，ECMAScript 中函数对于参数的限制是非常宽松的，例如你在定义函数时定义了两个参数，但在调用时可以只传递一个参数、也可以传三个参数，甚至不传递，示例如下： ```java function test(first, second) { console.log("first:" + first + ",second:" + second); } test(1) // first:1,second:undefined test(1,2) // first:1,second:2 test(1,2,3) // first:1,second:2 ``` 之所以能实现这样的效果，是因为 ECMAScript 在函数内部使用了一个数组 arguments 来维护所有参数，函数接收到的始终都是这个数组，而在实际使用时指向的也是这个数组中的具体元素，所以以上的函数等价于下面的函数： ```javascript function test(first, second) { console.log("first:" + arguments[0] + ",second:" + arguments[1]); } ``` **2. 改变函数作用域** 在 ECMAScript 5 中，每个函数都包含两个非继承而来的方法：**apply()** 和 **call()** ，它们都用于在特定的作用域中调用函数。简单来说，可以用这两个方法来改变函数的实际调用对象，从而改变 this 的值，因为 this 总是指向当前函数的实际调用对象： ```javascript window.color = "red"; var o = { color: "blue" }; function sayColor(){ console.log(this.color); } sayColor(); // red sayColor.call(this); // red sayColor.call(window); // red sayColor.call(o); // blue 此时this指向的是函数调用对象，即 o ``` **apply()** 和 **call()** 的第一个参数都是指代函数的调用对象，它们的区别主要在于第二个参数：**apply()** 支持使用数组或 arguments 给调用函数传值，而 **call()** 给调用函数传值时，必须逐个列举： ```javascript function sum(num1, num2) { return num1 + num2; } function callSum1(num1, num2) { return sum.apply(this, arguments); } function callSum2(num1, num2) { return sum.apply(this, [num1, num2]); } function callSum3(num1, num2) { return sum.call(this, num1, num2); } callSum1(10, 10); callSum2(10, 10); callSum3(10, 10); ``` **3. 绑定函数作用域** 如果想要将函数绑定在某个特定的作用域上，可以使用 **bind()** 函数： ```javascript window.color = "red"; var o = { color: "blue" }; function sayColor(){ console.log(this.color); } var objectSayColor = sayColor.bind(o); objectSayColor(); // blue 此时即便是在全局作用域中调用这个函数，其作用域仍然被永远绑定在 o 对象上 ``` ### 3.5 引用类型检测 想要检测某个对象是否属于某个引用类型，可以使用 **instanceof** 关键字： ```shell var date = new Date(); date instanceof Date // true ``` ## 四、内置对象 ### 4.1 Global 对象 ECMAScript 中内置了一个全局对象 Global ，任何不属于任何其他对象的属性和方法，最终都是它的属性和方法。 ES 通过该内置对象，提供了一些可以直接调用的全局方法，常用的如下： + **isNaN()**：用于确定一个值是否为 NaN； + **isFinite()**：用于判断被传入的参数值是否为一个有限数值； + **parseInt() \ parseFloat()**：解析并返回一个整数 \ 浮点数； + **encodeURI()**：对 URI 进行编码，但不会对本身属于 URI 的特殊字符进行编码，例如冒号、正斜杠、问号和井号； + **encodeURIComponent()**：对 URI 进行编码，会对任何非标准字符进行编码。 ### 4.2 window 对象 ECMAScript 并没有提供任何直接访问 Global 对象的方法，但是浏览器却基于 Global 扩展实现了 window 对象，可以直接在浏览器环境下使用： ```javascript window.isFinite(12) // true a = 12; window.a // 12 ``` ## 五、作用域与闭包 ### 5.1 作用域 在 ECMAScript 6 之前，只存在两种作用域，即：全局作用域 和 函数作用域，不存在块级作用域。这意味着在除了函数外的任何代码块中使用 var 关键字声明的变量都会被提升为全局变量，示例如下： ```javascript function test() { var age =12; } age // age is not defined if (true) { var name = "heibaiying"; } name // heibaiying ``` 这种情况同样适用与 for 循环代码块： ```javascript for (var i = 0; i < 10; i++) {} console.log(i); // 10 ``` ### 5.2 作用域链 由于函数作用域的存在，函数内的变量不能被外部访问，但是函数内的变量可以被其内部的函数访问，并且函数也可以访问其父级作用域上的变量，从而形成一条从其自身作用域到全局作用域的链条，示例如下： ```javascript var global = "global"; var outer = "outer global"; (function outer() { var outer = "outer"; function inner() { console.log(global, outer); } inner() })(); // 输出：global outer ``` ### 5.3 闭包 由于函数作用域的存在，函数内的变量不能被外部访问，这可以保证变量的私有性。但如果你想允许外部对内部变量进行特定操作，可以通过闭包来实现。闭包是指有权访问另一个函数作用域中的变量的函数。示例如下： ```java var contain = function () { var arr = []; return { push: function () { arr.push(...arguments); }, get: function () { return arr; } } }; var ctn = contain(); ctn.push(1, 2, 3, 4); ctn.get(); // [ 1, 2, 3, 4 ] ``` ## 六、对象设计 ECMAScript 中的对象都有两种基本属性：数据属性和访问器属性。 ### 6.1 数据属性 数据属性有以下 4 个描述其行为的特性： + **Enumerable**：表示该属性能否通过 for-in 循环返回；对于直接在对象上定义的属性， 该值默认为 true。 + **Writable**：表示能否修改属性的值；对于直接在对象上定义的属性， 该值默认为 true。 + **Value**：对应属性的数据值。默认值为 undefined。 + **Configurable**：表示能否对属性进行删除，修改等配置操作，对于直接在对象上定义的属性， 该值默认为 true。需要注意的是一旦将该属性的值设置为 false，就不能再将其设置为 true 。即一旦设置为不可配置，就不能再修改为可配置。因为你已经修改为不可配置，此时任何配置操作都无效了，自然修改 Configurable 属性的操作也无效。 ```javascript var person = {age: 12}; Object.defineProperty(person, "name", { Enumerable: false, writable: false, value: "heibai" }); console.log(person.name); // heibai person.name = "ying"; console.log(person.name); // ying for (var key in person) { console.log("key:" + key + ",value:" + person[key]) /// key:age,value:12 } ``` ### 6.2 访问器属性 访问器属性也有以下 4 个描述其行为的特性： + **Configurable**：表示能否对属性进行删除，修改等配置操作；对于直接在对象上定义的属性， 该值默认为 true。 + **Enumerable**：表示该属性能否通过 for-in 循环返回；对于直接在对象上定义的属性， 该值默认为 true。 + **Get**：在读取属性时调用的函数。默认值为 undefined。 + **Set**：在写入属性时调用的函数。默认值为 undefined。 ```javascript var student = { _age: null, birthday: new Date(2012,7,2) }; Object.defineProperty(student, "age", { get: function () { if (this._age == null) { // 如果年龄不存在就根据出生日期计算 return new Date().getFullYear() - this.birthday.getFullYear() }else { return this._age; } }, set: function (newValue) { if (newValue < 0) { console.log("年龄不能设置为负数"); } else { this._age = newValue; } } }); student.age = -1; // 输出：年龄不能设置为负数 console.log(student.age); // 输出：7 student.age = 12; console.log(student.age); // 输出 12 ``` ### 6.3 读取属性 想要获取一个对象的数据属性和访问器属性，可以使用 **Object.getOwnPropertyDescriptor()** 方法，类似于其他语言中的反射机制。这个方法接收两个参数：属性所在的对象和要读取属性名称。沿用上面的例子，示例如下： ```javascript var descriptor = Object.getOwnPropertyDescriptor(student, "age"); console.log(descriptor.get); // 输出 [Function: get] console.log(descriptor.enumerable); // 输出 false ``` ### 6.4 创建对象 在 ECMAScript 中，对象就是一种特殊的函数，想要声明一个对象，可以结合使用构造器模式和原型模式：基本属性可以通过构造器传入；但方法声明需要定义在原型属性上，如果直接定义在构造器上，每个对象实例都会创建该方法函数，即每个对象实例调用的都是自己重复声明的方法函数，示例如下： ```javascript function Person(name, age, job) { this.name = name; this.age = age; this.job = job; this.friends = ["hei", "bai"]; // 方法应该声明在原型属性上，而不是这里 this.sayAge = function () { console.log(this.age) } } Person.prototype = { constructor: Person, // 方法应该声明在这 sayName: function () { alert(this.name); } } var person1 = new Person("user01", 29, "Software Engineer"); var person2 = new Person("user02", 27, "Doctor"); person1.friends.push("ying"); console.log(person1.friends); // [ 'hei', 'bai', 'ying' ] console.log(person2.friends); // [ 'hei', 'bai'] console.log(person1 instanceof Person); // true console.log(person1.constructor === Person); // true console.log(person1.sayName === person2.sayName); // true console.log(person1.sayAge===person2.sayAge); // false ``` ## 参考资料 1. 尼古拉斯·泽卡斯 . JavaScript高级程序设计（第3版）. 人民邮电出版社 . 2012-3-29 2. [JS中小数四舍五入和浮点数的研究](http://caibaojian.com/js-tofixed.html) ================================================ FILE: docs/web/js_tool_method.md ================================================ # JavaScript 工具函数大全 ## 数组 1. all：布尔全等判断 ``` js const all = (arr, fn = Boolean) => arr.every(fn); all([4, 2, 3], x => x > 1); // true all([1, 2, 3]); // true ``` 2. allEqual：检查数组各项相等 ``` js const allEqual = arr => arr.every(val => val === arr[0]); allEqual([1, 2, 3, 4, 5, 6]); // false allEqual([1, 1, 1, 1]); // true ``` 3. approximatelyEqual：约等于 ``` js const approximatelyEqual = (v1, v2, epsilon = 0.001) => Math.abs(v1 - v2) < epsilon; approximatelyEqual(Math.PI / 2.0, 1.5708); // true ``` 4. arrayToCSV：数组转CSV格式（带空格的字符串） ``` js const arrayToCSV = (arr, delimiter = ',') => arr.map(v => v.map(x => `"${x}"`).join(delimiter)).join('\n'); arrayToCSV([['a', 'b'], ['c', 'd']]); // '"a","b"\n"c","d"' arrayToCSV([['a', 'b'], ['c', 'd']], ';'); // '"a";"b"\n"c";"d"' ``` 5. arrayToHtmlList：数组转li列表 ``` js const arrayToHtmlList = (arr, listID) => (el => ( (el = document.querySelector('#' + listID)), (el.innerHTML += arr.map(item => `
    • ${item}
      • `).join('')) ))(); arrayToHtmlList(['item 1', 'item 2'], 'myListID'); ``` 6. average：平均数 ``` js const average = (...nums) => nums.reduce((acc, val) => acc + val, 0) / nums.length; average(...[1, 2, 3]); // 2 average(1, 2, 3); // 2 ``` 7. averageBy：数组对象属性平均数 此代码段将获取数组对象属性的平均值 ``` js const averageBy = (arr, fn) => arr.map(typeof fn === 'function' ? fn : val => val[fn]).reduce((acc, val) => acc + val, 0) / arr.length; averageBy([{ n: 4 }, { n: 2 }, { n: 8 }, { n: 6 }], o => o.n); // 5 averageBy([{ n: 4 }, { n: 2 }, { n: 8 }, { n: 6 }], 'n'); // 5 ``` 8. bifurcate：拆分断言后的数组 可以根据每个元素返回的值，使用reduce()和push() 将元素添加到第二次参数fn中 。 ``` js const bifurcate = (arr, filter) => arr.reduce((acc, val, i) => (acc[filter[i] ? 0 : 1].push(val), acc), [[], []]); bifurcate(['beep', 'boop', 'foo', 'bar'], [true, true, false, true]); // [ ['beep', 'boop', 'bar'], ['foo'] ] ``` 9. castArray：其它类型转数组 ``` js const castArray = val => (Array.isArray(val) ? val : [val]); castArray('foo'); // ['foo'] castArray([1]); // [1] castArray(1); // [1] ``` 10. compact：去除数组中的无效/无用值 ``` js const compact = arr => arr.filter(Boolean); compact([0, 1, false, 2, '', 3, 'a', 'e' * 23, NaN, 's', 34]); // [ 1, 2, 3, 'a', 's', 34 ] ``` 11. countOccurrences：检测数值出现次数 ``` js const countOccurrences = (arr, val) => arr.reduce((a, v) => (v === val ? a + 1 : a), 0); countOccurrences([1, 1, 2, 1, 2, 3], 1); // 3 ``` 12. deepFlatten：递归扁平化数组 ``` js const deepFlatten = arr => [].concat(...arr.map(v => (Array.isArray(v) ? deepFlatten(v) : v))); deepFlatten([1, [2], [[3], 4], 5]); // [1,2,3,4,5] ``` 13. difference：寻找差异（并返回第一个数组独有的） 此代码段查找两个数组之间的差异，并返回第一个数组独有的。 ``` js const difference = (a, b) => { const s = new Set(b); return a.filter(x => !s.has(x)); }; difference([1, 2, 3], [1, 2, 4]); // [3] ``` 14. differenceBy：先执行再寻找差异 在将给定函数应用于两个列表的每个元素之后，此方法返回两个数组之间的差异。 ``` js const differenceBy = (a, b, fn) => { const s = new Set(b.map(fn)); return a.filter(x => !s.has(fn(x))); }; differenceBy([2.1, 1.2], [2.3, 3.4], Math.floor); // [1.2] differenceBy([{ x: 2 }, { x: 1 }], [{ x: 1 }], v => v.x); // [ { x: 2 } ] ``` 15. dropWhile：删除不符合条件的值 此代码段从数组顶部开始删除元素，直到传递的函数返回为true。 ``` js const dropWhile = (arr, func) => { while (arr.length > 0 && !func(arr[0])) arr = arr.slice(1); return arr; }; dropWhile([1, 2, 3, 4], n => n >= 3); // [3,4] ``` 16. flatten：指定深度扁平化数组 此代码段第二参数可指定深度。 ``` js const flatten = (arr, depth = 1) => arr.reduce((a, v) => a.concat(depth > 1 && Array.isArray(v) ? flatten(v, depth - 1) : v), []); flatten([1, [2], 3, 4]); // [1, 2, 3, 4] flatten([1, [2, [3, [4, 5], 6], 7], 8], 2); // [1, 2, 3, [4, 5], 6, 7, 8] ``` 17. indexOfAll：返回数组中某值的所有索引 此代码段可用于获取数组中某个值的所有索引，如果此值中未包含该值，则返回一个空数组。 ``` js const indexOfAll = (arr, val) => arr.reduce((acc, el, i) => (el === val ? [...acc, i] : acc), []); indexOfAll([1, 2, 3, 1, 2, 3], 1); // [0,3] indexOfAll([1, 2, 3], 4); // [] ``` 18. intersection：两数组的交集 ``` js const intersection = (a, b) => { const s = new Set(b); return a.filter(x => s.has(x)); }; intersection([1, 2, 3], [4, 3, 2]); // [2, 3] ``` 19. intersectionWith：两数组都符合条件的交集 此片段可用于在对两个数组的每个元素执行了函数之后，返回两个数组中存在的元素列表。 ``` js const intersectionBy = (a, b, fn) => { const s = new Set(b.map(fn)); return a.filter(x => s.has(fn(x))); }; intersectionBy([2.1, 1.2], [2.3, 3.4], Math.floor); // [2.1] ``` 20. intersectionWith：先比较后返回交集 ``` js const intersectionWith = (a, b, comp) => a.filter(x => b.findIndex(y => comp(x, y)) !== -1); intersectionWith([1, 1.2, 1.5, 3, 0], [1.9, 3, 0, 3.9], (a, b) => Math.round(a) === Math.round(b)); // [1.5, 3, 0] ``` 21. minN：返回指定长度的升序数组 ``` js const minN = (arr, n = 1) => [...arr].sort((a, b) => a - b).slice(0, n); minN([1, 2, 3]); // [1] minN([1, 2, 3], 2); // [1,2] ``` 22. negate：根据条件反向筛选 ``` js const negate = func => (...args) => !func(...args); [1, 2, 3, 4, 5, 6].filter(negate(n => n % 2 === 0)); // [ 1, 3, 5 ] ``` 23. randomIntArrayInRange：生成两数之间指定长度的随机数组 ``` js const randomIntArrayInRange = (min, max, n = 1) => Array.from({ length: n }, () => Math.floor(Math.random() * (max - min + 1)) + min); randomIntArrayInRange(12, 35, 10); // [ 34, 14, 27, 17, 30, 27, 20, 26, 21, 14 ] ``` 24. sample：在指定数组中获取随机数 ``` js const sample = arr => arr[Math.floor(Math.random() * arr.length)]; sample([3, 7, 9, 11]); // 9 ``` 25. sampleSize：在指定数组中获取指定长度的随机数 此代码段可用于从数组中获取指定长度的随机数，直至穷尽数组。 使用Fisher-Yates算法对数组中的元素进行随机选择。 ``` js const sampleSize = ([...arr], n = 1) => { let m = arr.length; while (m) { const i = Math.floor(Math.random() * m--); [arr[m], arr[i]] = [arr[i], arr[m]]; } return arr.slice(0, n); }; sampleSize([1, 2, 3], 2); // [3,1] sampleSize([1, 2, 3], 4); // [2,3,1] ``` 26. shuffle：“洗牌” 数组 此代码段使用Fisher-Yates算法随机排序数组的元素。 ``` js const shuffle = ([...arr]) => { let m = arr.length; while (m) { const i = Math.floor(Math.random() * m--); [arr[m], arr[i]] = [arr[i], arr[m]]; } return arr; }; const foo = [1, 2, 3]; shuffle(foo); // [2, 3, 1], foo = [1, 2, 3] ``` 27. nest：根据parent_id生成树结构（阿里一面真题） 根据每项的parent_id，生成具体树形结构的对象。 ``` js const nest = (items, id = null, link = 'parent_id') => items .filter(item => item[link] === id) .map(item => ({ ...item, children: nest(items, item.id) })); ``` 用法： ``` js const comments = [ { id: 1, parent_id: null }, { id: 2, parent_id: 1 }, { id: 3, parent_id: 1 }, { id: 4, parent_id: 2 }, { id: 5, parent_id: 4 } ]; const nestedComments = nest(comments); // [{ id: 1, parent_id: null, children: [...] }] ``` ## 函数 1. attempt：捕获函数运行异常 该代码段执行一个函数，返回结果或捕获的错误对象。 ``` js onst attempt = (fn, ...args) => { try { return fn(...args); } catch (e) { return e instanceof Error ? e : new Error(e); } }; var elements = attempt(function(selector) { return document.querySelectorAll(selector); }, '>_>'); if (elements instanceof Error) elements = []; // elements = [] ``` 2. defer：推迟执行 ``` js const defer = (fn, ...args) => setTimeout(fn, 1, ...args); defer(console.log, 'a'), console.log('b'); // logs 'b' then 'a' ``` 3. runPromisesInSeries：运行多个Promises ``` js const runPromisesInSeries = ps => ps.reduce((p, next) => p.then(next), Promise.resolve()); const delay = d => new Promise(r => setTimeout(r, d)); runPromisesInSeries([() => delay(1000), () => delay(2000)]); //依次执行每个Promises ，总共需要3秒钟才能完成 ``` 4. timeTaken：计算函数执行时间 ``` js const timeTaken = callback => { console.time('timeTaken'); const r = callback(); console.timeEnd('timeTaken'); return r; }; timeTaken(() => Math.pow(2, 10)); // 1024, (logged): timeTaken: 0.02099609375ms ``` 5. createEventHub：简单的发布/订阅模式 创建一个发布/订阅（发布-订阅）事件集线，有emit，on和off方法。 * 使用Object.create(null)创建一个空的hub对象。 * emit，根据event参数解析处理程序数组，然后.forEach()通过传入数据作为参数来运行每个处理程序。 * on，为事件创建一个数组（若不存在则为空数组），然后.push()将处理程序添加到该数组。 * off，用.findIndex()在事件数组中查找处理程序的索引，并使用.splice()删除。 ``` js const createEventHub = () => ({ hub: Object.create(null), emit(event, data) { (this.hub[event] || []).forEach(handler => handler(data)); }, on(event, handler) { if (!this.hub[event]) this.hub[event] = []; this.hub[event].push(handler); }, off(event, handler) { const i = (this.hub[event] || []).findIndex(h => h === handler); if (i > -1) this.hub[event].splice(i, 1); if (this.hub[event].length === 0) delete this.hub[event]; } }); ``` 用法： ``` js const handler = data => console.log(data); const hub = createEventHub(); let increment = 0; // 订阅，监听不同事件 hub.on('message', handler); hub.on('message', () => console.log('Message event fired')); hub.on('increment', () => increment++); // 发布：发出事件以调用所有订阅给它们的处理程序，并将数据作为参数传递给它们 hub.emit('message', 'hello world'); // 打印 'hello world' 和 'Message event fired' hub.emit('message', { hello: 'world' }); // 打印 对象 和 'Message event fired' hub.emit('increment'); // increment = 1 // 停止订阅 hub.off('message', handler); ``` 6. memoize：缓存函数 通过实例化一个Map对象来创建一个空的缓存。 通过检查输入值的函数输出是否已缓存，返回存储一个参数的函数，该参数将被提供给已记忆的函数；如果没有，则存储并返回它。 ``` js const memoize = fn => { const cache = new Map(); const cached = function(val) { return cache.has(val) ? cache.get(val) : cache.set(val, fn.call(this, val)) && cache.get(val); }; cached.cache = cache; return cached; }; ``` Ps: 这个版本可能不是很清晰，还有Vue源码版的： ``` js /** * Create a cached version of a pure function. */ export function cached (fn: F): F { const cache = Object.create(null) return (function cachedFn (str: string) { const hit = cache[str] return hit || (cache[str] = fn(str)) }: any) } ``` 7. once：只调用一次的函数 ``` js const once = fn => { let called = false return function () { if (!called) { called = true fn.apply(this, arguments) } } }; ``` 8. flattenObject：以键的路径扁平化对象 使用递归。 * 利用Object.keys(obj)联合Array.prototype.reduce()，以每片叶子节点转换为扁平的路径节点。 * 如果键的值是一个对象，则函数使用调用适当的自身prefix以创建路径Object.assign()。 * 否则，它将适当的前缀键值对添加到累加器对象。 * prefix除非您希望每个键都有一个前缀，否则应始终省略第二个参数。 ``` js const flattenObject = (obj, prefix = '') => Object.keys(obj).reduce((acc, k) => { const pre = prefix.length ? prefix + '.' : ''; if (typeof obj[k] === 'object') Object.assign(acc, flattenObject(obj[k], pre + k)); else acc[pre + k] = obj[k]; return acc; }, {}); flattenObject({ a: { b: { c: 1 } }, d: 1 }); // { 'a.b.c': 1, d: 1 } ``` 9. unflattenObject：以键的路径展开对象 与上面的相反，展开对象。 ``` js const unflattenObject = obj => Object.keys(obj).reduce((acc, k) => { if (k.indexOf('.') !== -1) { const keys = k.split('.'); Object.assign( acc, JSON.parse( '{' + keys.map((v, i) => (i !== keys.length - 1 ? `"${v}":{` : `"${v}":`)).join('') + obj[k] + '}'.repeat(keys.length) ) ); } else acc[k] = obj[k]; return acc; }, {}); unflattenObject({ 'a.b.c': 1, d: 1 }); // { a: { b: { c: 1 } }, d: 1 } ``` 这个的用途，在做Tree组件或复杂表单时取值非常舒服。 ## 字符串 1. byteSize：返回字符串的字节长度 ``` js const byteSize = str => new Blob([str]).size; byteSize('😀'); // 4 byteSize('Hello World'); // 11 ``` 2. capitalize：首字母大写 ``` js const capitalize = ([first, ...rest]) => first.toUpperCase() + rest.join(''); capitalize('fooBar'); // 'FooBar' capitalize('fooBar', true); // 'Foobar' ``` 3. capitalizeEveryWord：每个单词首字母大写 ``` js const capitalizeEveryWord = str => str.replace(/\b[a-z]/g, char => char.toUpperCase()); capitalizeEveryWord('hello world!'); // 'Hello World!' ``` 4. decapitalize：首字母小写 ``` js const decapitalize = ([first, ...rest]) => first.toLowerCase() + rest.join('') decapitalize('FooBar'); // 'fooBar' decapitalize('FooBar'); // 'fooBar' ``` 5. luhnCheck：银行卡号码校验（luhn算法） Luhn算法的实现，用于验证各种标识号，例如信用卡号，IMEI号，国家提供商标识号等。 与String.prototype.split('')结合使用，以获取数字数组。获得最后一个数字。实施luhn算法。如果被整除，则返回，否则返回。 ``` js const luhnCheck = num => { let arr = (num + '') .split('') .reverse() .map(x => parseInt(x)); let lastDigit = arr.splice(0, 1)[0]; let sum = arr.reduce((acc, val, i) => (i % 2 !== 0 ? acc + val : acc + ((val * 2) % 9) || 9), 0); sum += lastDigit; return sum % 10 === 0; }; ``` 用例: ``` js luhnCheck('4485275742308327'); // true luhnCheck(6011329933655299); // false luhnCheck(123456789); // false ``` 补充：银行卡号码的校验规则： 银行卡号码的校验采用Luhn算法，校验过程大致如下： * 从右到左给卡号字符串编号，最右边第一位是1，最右边第二位是2，最右边第三位是3…. * 从右向左遍历，对每一位字符t执行第三个步骤，并将每一位的计算结果相加得到一个数s。 * 对每一位的计算规则：如果这一位是奇数位，则返回t本身，如果是偶数位，则先将t乘以2得到一个数n，如果n是一位数（小于10），直接返回n，否则将n的个位数和十位数相加返回。 * 如果s能够整除10，则此号码有效，否则号码无效。 因为最终的结果会对10取余来判断是否能够整除10，所以又叫做模10算法。 当然，还是库比较香: bankcardinfo 6. splitLines：将多行字符串拆分为行数组。 使用String.prototype.split()和正则表达式匹配换行符并创建一个数组。 ``` js const splitLines = str => str.split(/\r?\n/); splitLines('This\nis a\nmultiline\nstring.\n'); // ['This', 'is a', 'multiline', 'string.' , ''] ``` 7. stripHTMLTags：删除字符串中的HTMl标签 从字符串中删除HTML / XML标签。 使用正则表达式从字符串中删除HTML / XML 标记。 ``` js const stripHTMLTags = str => str.replace(/<[^>]*>/g, ''); stripHTMLTags('

      lorem ipsum

      '); // 'lorem ipsum' ``` ## 对象 1. dayOfYear：当前日期天数 ``` js const dayOfYear = date => Math.floor((date - new Date(date.getFullYear(), 0, 0)) / 1000 / 60 / 60 / 24); dayOfYear(new Date()); // 285 ``` 2. forOwn：迭代属性并执行回调 ``` js const forOwn = (obj, fn) => Object.keys(obj).forEach(key => fn(obj[key], key, obj)); forOwn({ foo: 'bar', a: 1 }, v => console.log(v)); // 'bar', 1 ``` 3. Get Time From Date：返回当前24小时制时间的字符串 ``` js const getColonTimeFromDate = date => date.toTimeString().slice(0, 8); getColonTimeFromDate(new Date()); // "08:38:00" ``` 4. Get Days Between Dates：返回日期间的天数 ``` js const getDaysDiffBetweenDates = (dateInitial, dateFinal) => (dateFinal - dateInitial) / (1000 * 3600 * 24); getDaysDiffBetweenDates(new Date('2019-01-01'), new Date('2019-10-14')); // 286 ``` 5. is：检查值是否为特定类型。 ``` js const is = (type, val) => ![, null].includes(val) && val.constructor === type; is(Array, [1]); // true is(ArrayBuffer, new ArrayBuffer()); // true is(Map, new Map()); // true is(RegExp, /./g); // true is(Set, new Set()); // true is(WeakMap, new WeakMap()); // true is(WeakSet, new WeakSet()); // true is(String, ''); // true is(String, new String('')); // true is(Number, 1); // true is(Number, new Number(1)); // true is(Boolean, true); // true is(Boolean, new Boolean(true)); // true ``` 6. isAfterDate：检查是否在某日期后 ``` js const isAfterDate = (dateA, dateB) => dateA > dateB; isAfterDate(new Date(2010, 10, 21), new Date(2010, 10, 20)); // true ``` 7. isBeforeDate：检查是否在某日期前 ``` js const isBeforeDate = (dateA, dateB) => dateA < dateB; isBeforeDate(new Date(2010, 10, 20), new Date(2010, 10, 21)); // true ``` 8. tomorrow：获取明天的字符串格式时间 ``` js const tomorrow = () => { let t = new Date(); t.setDate(t.getDate() + 1); return t.toISOString().split('T')[0]; }; tomorrow(); // 2019-10-15 (如果明天是2019-10-15) ``` 9. equals：全等判断 在两个变量之间进行深度比较以确定它们是否全等。 此代码段精简的核心在于Array.prototype.every()的使用。 ``` js const equals = (a, b) => { if (a === b) return true; if (a instanceof Date && b instanceof Date) return a.getTime() === b.getTime(); if (!a || !b || (typeof a !== 'object' && typeof b !== 'object')) return a === b; if (a.prototype !== b.prototype) return false; let keys = Object.keys(a); if (keys.length !== Object.keys(b).length) return false; return keys.every(k => equals(a[k], b[k])); }; ``` 用法： ``` js equals({ a: [2, { e: 3 }], b: [4], c: 'foo' }, { a: [2, { e: 3 }], b: [4], c: 'foo' }); // true ``` ## 数字 1. randomIntegerInRange：生成指定范围的随机整数 ``` js const randomIntegerInRange = (min, max) => Math.floor(Math.random() * (max - min + 1)) + min; randomIntegerInRange(0, 5); // 3 ``` 2. randomNumberInRange：生成指定范围的随机小数 ``` js const randomNumberInRange = (min, max) => Math.random() * (max - min) + min; randomNumberInRange(2, 10); // 6.0211363285087005 ``` 3. round：四舍五入到指定位数 ``` js const round = (n, decimals = 0) => Number(`${Math.round(`${n}e${decimals}`)}e-${decimals}`); round(1.005, 2); // 1.01 ``` 4. sum：计算数组或多个数字的总和 ``` js const sum = (...arr) => [...arr].reduce((acc, val) => acc + val, 0); sum(1, 2, 3, 4); // 10 sum(...[1, 2, 3, 4]); // 10 ``` 5. toCurrency：简单的货币单位转换 ``` js const toCurrency = (n, curr, LanguageFormat = undefined) => Intl.NumberFormat(LanguageFormat, { style: 'currency', currency: curr }).format(n); toCurrency(123456.789, 'EUR'); // €123,456.79 toCurrency(123456.789, 'USD', 'en-us'); // $123,456.79 toCurrency(123456.789, 'USD', 'fa'); // ۱۲۳٬۴۵۶٫۷۹ toCurrency(322342436423.2435, 'JPY'); // ¥322,342,436,423 ``` ## 浏览器操作及其它 1. bottomVisible：检查页面底部是否可见 ``` js const bottomVisible = () => document.documentElement.clientHeight + window.scrollY >= (document.documentElement.scrollHeight || document.documentElement.clientHeight); bottomVisible(); // true ``` 2. Create Directory：检查创建目录 此代码段调用fs模块的existsSync()检查目录是否存在，如果不存在，则mkdirSync()创建该目录。 ``` js const fs = require('fs'); const createDirIfNotExists = dir => (!fs.existsSync(dir) ? fs.mkdirSync(dir) : undefined); createDirIfNotExists('test'); ``` 3. currentURL：返回当前链接url ``` js const currentURL = () => window.location.href; currentURL(); // 'https://juejin.im' ``` 4. distance：返回两点间的距离 该代码段通过计算欧几里得距离来返回两点之间的距离。 ``` js const distance = (x0, y0, x1, y1) => Math.hypot(x1 - x0, y1 - y0); distance(1, 1, 2, 3); // 2.23606797749979 ``` 5. elementContains：检查是否包含子元素 此代码段检查父元素是否包含子元素。 ``` js const elementContains = (parent, child) => parent !== child && parent.contains(child); elementContains(document.querySelector('head'), document.querySelector('title')); // true elementContains(document.querySelector('body'), document.querySelector('body')); // false ``` 6. getStyle：返回指定元素的生效样式 ``` js const getStyle = (el, ruleName) => getComputedStyle(el)[ruleName]; getStyle(document.querySelector('p'), 'font-size'); // '16px' ``` 7. getType：返回值或变量的类型名 ``` js const getType = v => v === undefined ? 'undefined' : v === null ? 'null' : v.constructor.name.toLowerCase(); getType(new Set([1, 2, 3])); // 'set' getType([1, 2, 3]); // 'array' ``` 8. hasClass：校验指定元素的类名 ``` js const hasClass = (el, className) => el.classList.contains(className); hasClass(document.querySelector('p.special'), 'special'); // true ``` 9. hide：隐藏所有的指定标签 ``` js const hide = (...el) => [...el].forEach(e => (e.style.display = 'none')); hide(document.querySelectorAll('img')); // 隐藏所有标签 ``` 10. httpsRedirect：HTTP 跳转 HTTPS ``` js const httpsRedirect = () => { if (location.protocol !== 'https:') location.replace('https://' + location.href.split('//')[1]); }; httpsRedirect(); // 若在`http://www.baidu.com`, 则跳转到`https://www.baidu.com` ``` 11. insertAfter：在指定元素之后插入新元素 ``` js const insertAfter = (el, htmlString) => el.insertAdjacentHTML('afterend', htmlString); //
      ...

      after

      insertAfter(document.getElementById('myId'), '

      after

      '); ``` 12. insertBefore：在指定元素之前插入新元素 ``` js const insertBefore = (el, htmlString) => el.insertAdjacentHTML('beforebegin', htmlString); insertBefore(document.getElementById('myId'), '

      before

      '); //

      before

      ...
      ``` 13. isBrowser：检查是否为浏览器环境 此代码段可用于确定当前运行时环境是否为浏览器。这有助于避免在服务器（节点）上运行前端模块时出错。 ``` js const isBrowser = () => ![typeof window, typeof document].includes('undefined'); isBrowser(); // true (browser) isBrowser(); // false (Node) ``` 14. isBrowserTab：检查当前标签页是否活动 ``` js const isBrowserTabFocused = () => !document.hidden; isBrowserTabFocused(); // true ``` 15. nodeListToArray：转换nodeList为数组 ``` js const nodeListToArray = nodeList => [...nodeList]; nodeListToArray(document.childNodes); // [ , html ] ``` 16. Random Hexadecimal Color Code：随机十六进制颜色 ``` js const randomHexColorCode = () => { let n = (Math.random() * 0xfffff * 1000000).toString(16); return '#' + n.slice(0, 6); }; randomHexColorCode(); // "#e34155" ``` 17. scrollToTop：平滑滚动至顶部 ``` js const scrollToTop = () => { const c = document.documentElement.scrollTop || document.body.scrollTop; if (c > 0) { window.requestAnimationFrame(scrollToTop); window.scrollTo(0, c - c / 8); } }; scrollToTop(); ``` 18. smoothScroll：滚动到指定元素区域 该代码段可将指定元素平滑滚动到浏览器窗口的可见区域。 ``` js const smoothScroll = element => document.querySelector(element).scrollIntoView({ behavior: 'smooth' }); smoothScroll('#fooBar'); smoothScroll('.fooBar'); ``` 19. detectDeviceType：检测移动/PC设备 ``` js const detectDeviceType = () => /Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(navigator.userAgent) ? 'Mobile' : 'Desktop'; ``` 20. getScrollPosition：返回当前的滚动位置 默认参数为window ，pageXOffset(pageYOffset)为第一选择，没有则用scrollLeft(scrollTop) ``` js const getScrollPosition = (el = window) => ({ x: el.pageXOffset !== undefined ? el.pageXOffset : el.scrollLeft, y: el.pageYOffset !== undefined ? el.pageYOffset : el.scrollTop }); getScrollPosition(); // {x: 0, y: 200} ``` 21. size：获取不同类型变量的字节长度 这个的实现非常巧妙，利用Blob类文件对象的特性，获取对象的长度。 另外，多重三元运算符，是真香。 ``` js const size = val => Array.isArray(val) ? val.length : val && typeof val === 'object' ? val.size || val.length || Object.keys(val).length : typeof val === 'string' ? new Blob([val]).size : 0; size([1, 2, 3, 4, 5]); // 5 size('size'); // 4 size({ one: 1, two: 2, three: 3 }); // 3 ``` 22. escapeHTML：转义HTML 当然是用来防XSS攻击啦。 ``` js const escapeHTML = str => str.replace( /[&<>'"]/g, tag => ({ '&': '&', '<': '<', '>': '>', "'": ''', '"': '"' }[tag] || tag) ); escapeHTML('Me & you'); // '<a href="#">Me & you</a>' ``` ================================================ FILE: docs/web/node.js.md ================================================ ## Node 介绍 - Node 中的 JavaScript + EcmaScript * 变量 * 方法 * 数据类型 * 内置对象 * Array * Object * Date * Math + 模块系统 * 在 Node 中没有全局作用域的概念 * 在 Node 中，只能通过 require 方法来加载执行多个 JavaScript 脚本文件 * require 加载只能是执行其中的代码，文件与文件之间由于是模块作用域，所以不会有污染的问题 - 模块完全是封闭的 - 外部无法访问内部 - 内部也无法访问外部 * 模块作用域固然带来了一些好处，可以加载执行多个文件，可以完全避免变量命名冲突污染的问题 * 但是某些情况下，模块与模块是需要进行通信的 * 在每个模块中，都提供了一个对象：`exports` * 该对象默认是一个空对象 * 你要做的就是把需要被外部访问使用的成员手动的挂载到 `exports` 接口对象中 * 然后谁来 `require` 这个模块，谁就可以得到模块内部的 `exports` 接口对象 * 还有其它的一些规则，具体后面讲，以及如何在项目中去使用这种编程方式，会通过后面的案例来处理 + 核心模块 * 核心模块是由 Node 提供的一个个的具名的模块，它们都有自己特殊的名称标识，例如 - fs 文件操作模块 - http 网络服务构建模块 - os 操作系统信息模块 - path 路径处理模块 - 。。。。 * 所有核心模块在使用的时候都必须手动的先使用 `require` 方法来加载，然后才可以使用，例如： - `var fs = require('fs')` - http + require + 端口号 * ip 地址定位计算机 * 端口号定位具体的应用程序 + Content-Type * 服务器最好把每次响应的数据是什么内容类型都告诉客户端，而且要正确的告诉 * 不同的资源对应的 Content-Type 是不一样，具体参照：http://tool.oschina.net/commons * 对于文本类型的数据，最好都加上编码，目的是为了防止中文解析乱码问题 + 通过网络发送文件 * 发送的并不是文件，本质上来讲发送是文件的内容 * 当浏览器收到服务器响应内容之后，就会根据你的 Content-Type 进行对应的解析处理 - 模块系统 - Node 中的其它的核心模块 - 做一个小管理系统： + CRUD - Express Web 开发框架 + `npm install express` ## 文档 * [官方文档](https://nodejs.org/en/docs/) * [菜鸟文档](http://www.runoob.com/nodejs/nodejs-tutorial.html) * [廖雪峰文档](https://www.liaoxuefeng.com/wiki/001434446689867b27157e896e74d51a89c25cc8b43bdb3000/001434501245426ad4b91f2b880464ba876a8e3043fc8ef000) ================================================ FILE: docs/web/react.md ================================================ # React 开发者指南 > 该指南将助你在 2019 成为一名 React 开发者 你可以在下面找到一张图，该图展示了你可以选取的路径及你想学习的库，从而成为一名 React 开发者。“作为 React 开发者，我接下来应该学习什么？”，我把这张图作为建议给每个问过我这一问题的人。 ## 免责声明 > 该指南的目的是为了给你心有个大概的轮廓。如果你对接下来要学习的内容感到困惑，指南将指导你而不是鼓励你选择时髦和新颖的东西。 > 你应该逐渐理解为什么一种工具比另一种工具更适合某些情况，并且记住时髦和新颖的东西并不总是意味着最适合这个工作。 ## Roadmap ![Roadmap](./../img/roadmap-cn.png) ## 资源 1. 基础 1. HTML - 学习 HTML 基础知识 - 做几个页面来练习 2. CSS - 学习 CSS 基础知识 - 完成上一步的样式页面 - 使用 grid 布局和 flexbox 布局构建页面 3. JS 基础 - 熟悉语法 - 学习 DOM 的基本操作 - 学习 JS 的典型机制（状态提升，事件冒泡，原型） - 实现一些 AJAX（XHR）调用 - 学习新特性 (ECMA Script 6+) - 另外，熟悉 jQuery 库 2. 常用开发技能 1. 学习 GIT 的使用, 在 GitHub 上创建一些仓库, 并和其他人分享你的代码 2. 掌握 HTTP(S) 协议, 及其请求方法 (GET, POST, PUT, PATCH, DELETE, OPTIONS) 3. 不要害怕使用 Google, [使用 Google 进行强力搜索](http://www.powersearchingwithgoogle.com/) 4. 熟悉终端，并配置你的 shell (bash, zsh, fish) 5. 阅读一些关于算法和数据结构的书籍 6. 阅读一些关于设计模式的书籍 3. 在[官网](https://reactjs.org/tutorial/tutorial.html)上学习 React 或者完成一些[课程](https://egghead.io/courses/the-beginner-s-guide-to-react) 4. 熟悉你将用到的工具 1. 包管理器 - [npm](https://www.npmjs.com/) - [yarn](https://yarnpkg.com/lang/en/) - [pnpm](https://pnpm.js.org/) 2. 任务运行器 - [npm 脚本](https://docs.npmjs.com/misc/scripts) - [gulp](https://gulpjs.com/) - [Webpack](https://webpack.js.org/) - [Rollup](https://rollupjs.org/guide/en) - [Parcel](https://parceljs.org/) 5. 样式 1. CSS 预处理器 - [Sass/CSS](https://sass-lang.com/) - [PostCSS](https://postcss.org/) - [Less](http://lesscss.org/) - [Stylus](http://stylus-lang.com/) 2. CSS 框架 - [Bootstrap](https://getbootstrap.com/) - [Materialize](https://materializecss.com/), [Material UI](https://material-ui.com/), [Material Design Lite](https://getmdl.io/) - [Bulma](https://bulma.io/) - [Semantic UI](https://semantic-ui.com/) 3. CSS 架构 - [BEM](http://getbem.com/) - [CSS Modules](https://github.com/css-modules/css-modules) - [Atomic](https://acss.io/) - [OOCSS](https://github.com/stubbornella/oocss/wiki) - [SMACSS](https://smacss.com/) - [SUITCSS](https://suitcss.github.io/) 4. JS 编写 CSS - [Styled Components](https://www.styled-components.com/) - [Radium](https://formidable.com/open-source/radium/) - [Emotion](https://emotion.sh/) - [JSS](http://cssinjs.org/) - [Aphrodite](https://github.com/Khan/aphrodite) 6. 状态管理 1. [组件状态](https://reactjs.org/docs/faq-state.html)/[上下文 API](https://reactjs.org/docs/context.html) 2. [Redux](https://redux.js.org/) 1. 异步操作 (Side Effects) - [Redux Thunk](https://github.com/reduxjs/redux-thunk) - [Redux Better Promise](https://github.com/Lukasz-pluszczewski/redux-better-promise) - [Redux Saga](https://redux-saga.js.org/) - [Redux Observable](https://redux-observable.js.org) 2. 助手 - [Rematch](https://rematch.gitbooks.io/rematch/) - [Reselect](https://github.com/reduxjs/reselect) 3. 数据持久化 - [Redux Persist](https://github.com/rt2zz/redux-persist) - [Redux Phoenix](https://github.com/adam-golab/redux-phoenix) 4. [Redux Form](https://redux-form.com) 3. [MobX](https://mobx.js.org/) 7. 类型检查器 - [PropTypes](https://reactjs.org/docs/typechecking-with-proptypes.html) - [TypeScript](https://www.typescriptlang.org/) - [Flow](https://flow.org/en/) 8. 表单助手 - [Redux Form](https://redux-form.com) - [Formik](https://github.com/jaredpalmer/formik) - [Formsy](https://github.com/formsy/formsy-react) - [Final Form](https://github.com/final-form/final-form) 9. 路由 - [React-Router](https://reacttraining.com/react-router/) - [Router5](https://router5.js.org/) - [Redux-First Router](https://github.com/faceyspacey/redux-first-router) - [Reach Router](https://reach.tech/router/) 10. API 客户端 1. REST - [Fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) - [SuperAgent](https://visionmedia.github.io/superagent/) - [axios](https://github.com/axios/axios) 2. GraphQL - [Apollo](https://www.apollographql.com/docs/react/) - [Relay](https://facebook.github.io/relay/) - [urql](https://github.com/FormidableLabs/urql) 11. 实用工具库 - [Lodash](https://lodash.com/) - [Moment](https://momentjs.com/) - [classnames](https://github.com/JedWatson/classnames) - [Numeral](http://numeraljs.com/) - [RxJS](http://reactivex.io/) - [ImmutableJS](https://facebook.github.io/immutable-js/) - [Ramda](https://ramdajs.com/) 12. 测试 1. 单元（Unit）测试 - [Jest](https://facebook.github.io/jest/) - [Enzyme](http://airbnb.io/enzyme/) - [Sinon](http://sinonjs.org/) - [Mocha](https://mochajs.org/) - [Chai](http://www.chaijs.com/) - [AVA](https://github.com/avajs/ava) - [Tape](https://github.com/substack/tape) 2. 端到端（E2E）测试 - [Selenium](https://www.seleniumhq.org/), [Webdriver](http://webdriver.io/) - [Cypress](https://cypress.io/) - [Puppeteer](https://pptr.dev/) - [Cucumber.js](https://github.com/cucumber/cucumber-js) - [Nightwatch.js](http://nightwatchjs.org/) 3. 集成测试 - [Karma](https://karma-runner.github.io/) 13. 国际化（i18n) - [React Intl](https://github.com/yahoo/react-intl) - [React i18next](https://react.i18next.com/) 14. 服务端渲染（SSR) - [Next.js](https://nextjs.org/) - [After.js](https://github.com/jaredpalmer/after.js) - [Rogue](https://github.com/alidcastano/rogue.js) 15. 静态网站生成器 - [Gatsby](https://www.gatsbyjs.org/) 16. 后端集成框架 - [React on Rails](https://shakacode.gitbooks.io/react-on-rails/content/) 17. 移动端 - [React Native](https://facebook.github.io/react-native/) - [Cordova](https://cordova.apache.org/)/[Phonegap](https://phonegap.com/) 18. 桌面端 - [Proton Native](https://proton-native.js.org/) - [Electron](https://electronjs.org/) - [React Native Windows](https://github.com/Microsoft/react-native-windows) 19. 虚拟现实（VR) - [React 360](https://facebook.github.io/react-360/) ## 总结 如果你认为指南可以改进，请提交包含任何更新的 PR 并提交任何问题。此外，我将继续改进这个仓库，因此你可以 star 这个仓库以便于重新访问。 ## 贡献 该指南是使用[Draw.io](https://www.draw.io/)构建的。项目文件可以在 `/src` 目录中找到。要修改它, 请打开 draw.io, 点击 **Open Existing Diagram** 并选择项目中的 `xml` 文件。它将为你打开指南，更新它，上传和更新自述文件中的图像并创建一个 PR（导出为 png）。 - 改进后提交 PR - 讨论问题中的想法 - 传播消息 ================================================ FILE: docs/web/react_interview.md ================================================ # React 面试问题 >如果你是一位有理想的前端开发人员，并且正在准备面试，那么这篇文章就是为你准备的。本文收集了 React 面试中最常见的 50 大问题，这是一份理想的指南，让你为 React 相关的面试做好充分的准备工作。首先我们快速了解一下 React 在市场上的需求和现状，然后再开始讨论 React 面试问题。 JavaScript 工具的市场地位正在缓慢而稳定地上升当中，而对 React 认证的需求正在飞速增长。选择正确的技术来开发应用程序或网站变得愈加艰难。React 被认为是 Javascript 语言中增长最快的框架。 虚拟 DOM 和可复用部件等独特特性吸引了前端开发人员的注意。尽管成熟的框架（如 Angular、Meteor 和 Vue 等）在 MVC（模型 - 视图 - 控制器）中只是一个“视图”库，但它们都有很强的竞争力。下图显示了常见 JS 框架的趋势： ![react-top](./../img/react-top.png) 以下是面试官最有可能提出的 50 个面试问题和答案。 ## React 面试问题——常规知识 1. 真实 DOM 和虚拟 DOM 的区别 真实 DOM|虚拟 DOM --|:--| 1.更新较慢| 1.更新较快 2.可以直接更新 HTML|2.不能直接更新 HTML 3.元素更新时创建一个新 DOM|3.元素更新时更新 JSX 4.DOM 操作开销较大|4.DOM 操作非常容易 5.内存浪费严重|5.没有内存浪费 2. 什么是 React？ * React 是 2011 年由 Facebook 开发的前端 JavaScript 库。 * 它遵循基于组件的方法，这种方法可以用来构建可复用的 UI 组件。 * 它用于复杂的交互式 Web 端和移动端用户界面开发。 * 尽管它在 2015 年才开源，但得到了一家巨头的支持。 3. React 的特点是什么？ * 轻量级 DOM，以获得更好的性能。 * 在 React 中，一切都被视为组件。 * React 使用 JSX（JavaScript eXtension），使我们可以编写类似于 HTML 的 JavaScript。 * React 不是直接运行在浏览器的文档对象模型（DOM）上，而是运行在虚拟 DOM 上。 * ReactJS 遵循单向数据流或单向数据绑定。 4. 列出 React 的一些主要优势。 * 可以提高应用程序的性能。 * 可以方便地用在客户端和服务端。 * 由于有了 JSX，代码的可读性提高了。 * 使用 React 后，编写 UI 测试用例变得非常容易。 5. React 有哪些局限？ * React 只是一个库，而不是一个成熟的框架。 * 它的库很大，需要花费一些时间来理解。 * 新手程序员可能很难入门。 * 由于它使用了内联模板和 JSX，编码也比较复杂。 6. 什么是 JSX？ JSX 是 JavaScript XML 的简写。这是 React 使用的一种文件类型，具备 JavaScript 的表现力，并使用 HTML 作为模板语法。这样一来 HTML 文件理解起来就非常简单。这种文件可以创造稳健的应用程序并提高其效率。下面是一个 JSX 实例： ``` jsx render(){ return(

      Hello World from Codersera!!

      ); } ``` 7. 你对虚拟 DOM 有什么了解？解释其工作机制。 虚拟 DOM 是轻量级的 JavaScript 对象，一开始只是真实 DOM 的一个副本。它是一个节点树，将组件列为对象及其属性和内容的列表。React 的渲染功能从 React 的各个部分生成一个节点树。然后，它会根据由不同用户或系统行为引起的信息模型突变来更新此树。 虚拟 DOM 的工作机制只有简单的三步组成。 1. 每当任何基础信息更改时，整个 UI 就会以虚拟 DOM 的表示形式重新渲染。 ![ui-dom](./../img/ui-dom-1.png) 2. 然后计算先前的 DOM 表示和新的 DOM 表示之间的区别。 ![ui-dom](./../img/ui-dom-2.png) 3. 计算完成后，只有实际更改的内容才会更新到真实 DOM。 ![ui-dom](./../img/ui-dom-3.png) 8. 为什么浏览器无法读取 JSX？ React 使用 JSX（JavaScript eXtension），我们可以用它编写类似于 HTML 的 JavaScript。但由于 JSX 不是合法的 JavaScript，因此浏览器无法直接读取它。如果 JavaScript 文件包含 JSX，则必须将其转换。你需要一个转换器将 JSX 转换为浏览器可以理解的常规 Javascript。目前最常用的转换器是 Babel。 9. 与 ES5 相比，React 的 ES6 语法有何不同？ ES5 和 ES6 的语法区别如下： ```javascript // ES5 var React = require('react'); // ES6 import React from 'react'; ******** export vs exports ********* // ES5 module.exports = Component; // ES6 export default Component; ****** function ***** // ES5 var MyComponent = React.createClass({ render: function() { return

      Hello CoderSera!

      }, }); // ES6 class MyComponent extends React.Component { render() { return

      Hello CoderSera!

      } } ******* props ****** // ES5 var App = React.createClass({ propTypes: { name: React.PropTypes.string }, render: function() { return

      Hello, { this.props.name }! < /h3> }, }); // ES6 class App extends React.Component { render() { return

      Hello, { this.props.name }!

      } } ****** state ***** // ES5 var App = React.createClass({ getInitialState: function() { return { name: 'world' }; } render: function() { return

      Hello, { this.state.name }! < /h3>; }, }); // ES6 class App extends React.Component { constructor() { super(); this.state = { name: 'world' }; } render() { return

      Hello, { this.state.name }! < /h3> } render() { return;

      Hello, { this.state.name }! < /h3> } ``` 10. React 和 Angular 有何不同？ React 对比 Angular|React|Angular --|:--|:--| 架构|使用虚拟 DOM|使用真实 DOM 渲染|服务端渲染|客户端渲染 DOM|使用虚拟 DOM|使用真实 DOM 数据绑定|单向数据绑定|双向数据绑定 调试|编译时调试|运行时调试 开发者|Facebook|谷歌 11. 如何理解“在 React 中，一切都是组件”。 React 应用程序的 UI 构建块都是组件。这些部分将整个 UI 划分为许多可自治和可复用的微小部分。然后独立的某个部分发生变化就不会影响 UI 的其余部分。 12. 解释 React 中 render() 的目的。 它被视为普通函数，但 render() 函数必须返回某些值，无论值是否为空。调用组件文件时默认会调用 render() 方法，因为组件需要显示 HTML 标记，或者我们可以说 JSX 语法。每个 React 组件必须有一个 render() 函数，它返回单个 React 元素，该元素代表原生 DOM 组件。如果需要渲染多个 HTML 元素，则必须将它们分组在一个封闭的标签内，如form、group和div等。此函数必须保持纯净，就是说它在每次调用时必须返回相同的结果。 ``` jsx import React, { Component } from 'react'; class App extends Component { render() { return (

      hello CoderSera

      ) } } export default App; ``` 13. 什么是 Hooks？ Hooks 是一项新功能，使你无需编写类即可使用状态等 React 功能。来看一个 useState hook 示例。 ```jsx import {useState} from 'react';

      function Example() {
      // Declare a new ``` 14. 什么是 props？ * Props 用于将数据从父级传递到子级或由组件本身传递。它们是不可变的，因此不会更改。 * Props 是不可变的。因为它们是基于纯函数的概念开发的。在纯函数中，我们无法更改参数数据。因此在 ReactJS 中也无法更改 props 的数据。 15. React 中的状态是什么，如何使用？ 组件可以通过状态来跟踪其执行的任何渲染之间的信息。 状态用于可变数据或将要更改的数据。这对于用户输入尤其方便。以搜索栏为例，用户输入数据时他们看到的内容也会更新。 16. 状态和 props 的区别。 条件|状态|Props --|:--|:--| 从父组件接收初始值|是|是 父组件可以更改值|否|是 在组件内设置默认值|是|是 在组件内更改|是|否 为子组件设置初始值|是|是 在子组件内更改|否|是 17. 如何更新组件的状态？ 可以使用 this.setState() 更新组件的状态。 ``` jsx class MyComponent extends React.Component { constructor() { super(); this.state = { name: 'Maxx', id: '101' } } render() { setTimeout(()=>{this.setState({name:'Jaeha', id:'222'})},2000) return (

      Hello {this.state.name}

      Your Id is {this.state.id}

      ); } } ReactDOM.render( , document.getElementById('content') ); ``` 18.React 中的箭头函数是什么？如何使用？ **粗箭头**=> 用于定义匿名函数，这通常是将参数传递给回调函数的最简单方法。但是你需要在使用它时优化性能。注意：每次渲染组件时，在 render 方法中使用箭头函数都会创建一个新函数，这可能会影响性能。 ```jsx //General way render() { return( ); } //With Arrow Function render() { return( this.handleOnChange(e) } /> ); } ``` 19. 有状态和无状态组件的区别。 有状态组件|无状态组件 --|:--| 在内存中存储组件状态更改的信息|计算组件的内部状态 有权更改状态|无权更改状态 包含过去、现在和可能的未来状态更改的信息|没有包含关于状态更改的信息 无状态组件通知它们关于状态更改的需求，然后它们将 props 传递给前者|它们从有状态组件接收 props，将其视为回调函数 20. React 组件的生命周期有哪些阶段？ React 组件的生命周期分为三个不同阶段： * **初始化**：在初始化阶段，我们为 this.props 和 this.state 定义默认值和初始值。 * **挂载**：挂载是将组件插入 DOM 时发生的过程 * **更新**：这个阶段中，每当状态更改或组件收到新的 prop 时，组件都会更新 * **卸载**：这是从 DOM 卸载组件的阶段。 21. 详细解释 React 组件的生命周期技术。 一些最重要的生命周期方法包括： * componentWillMount()——在初始渲染发生之前，即在 React 将组件插入 DOM 之前立即调用一次。请务必注意，在此方法中调用 this.setState() 不会触发重新渲染。 * componentDidMount()——在渲染函数之后触发此方法。现在可以访问更新的 DOM，这意味着该方法是初始化其他需要访问 DOM 的 Javascript 库以及数据提取操作的最佳选择。 * componentWillReceiveProps()——componentWillReceiveProps() 在组件接收新 props 时调用。在调用 render() 方法之前，我们可以用这个方法对 prop 过渡做出反应。在此函数中调用 this.setState() 不会触发额外的重新渲染，我们可以通过 this.props 访问旧的 props。 * shouldComponentUpdate()——我们可以用它来决定下一个组件的状态是否应触发重新渲染。此方法返回一个布尔值，默认为 true。但是我们可以返回 false，并且不会调用以下方法： * componentWillUpdate()——当接收到新的 props 或状态时，在渲染（更新）之前立即调用此方法。我们可以用它在更新之前做准备，但是不允许使用 this.setState()。 * componentDidUpdate()——React 更新 DOM 后立即调用此方法。我们可以使用此方法与更新后的 DOM 交互，或执行任何渲染后操作。 * componentWillUnmount()——从 DOM 卸载组件之前立即调用此方法。我们可以用它执行可能需要的任何清理工作。 22. React 中的事件是什么？ 在 React 中，事件是对特定动作（如鼠标悬停、鼠标单击和按键等）触发的反应。处理这些事件类似于处理 DOM 元素上的事件。但是在语法上存在一些差异，例如： * 事件命名使用驼峰式大小写，而不是仅使用小写字母。 * 事件作为函数而不是字符串传递。 事件参数包含一组特定于事件的属性。每个事件类型都包含它自己的属性和行为，这些属性和行为只能通过它的事件处理程序访问。 23. 如何在 React 中创建事件？ ```jsx class Display extends React.Component({ show(evt) { // code }, render() { // Render the div with an onClick prop (value is a function) return (
      Click Me!
      ); } }); ``` 24. 什么是 React 中的合成事件？ 合成事件是围绕浏览器原生事件充当跨浏览器包装器的对象。它们将不同的浏览器行为合并为一个 API。这样做是为了确保在各个浏览器的事件中显示一致的特征。 25. 如何理解 React 中的引用？ Ref 是 React 引用的简写。它是一个属性，帮助存储对特定元素或组件的引用，由组件渲染的配置函数返回。它用于返回对渲染返回的特定元素或组件的引用。当我们需要 DOM 测量或向组件添加方法时，它们会派上用场。 ``` jsx class ReferenceDemo extends React.Component{ display() { const name = this.inputDemo.value; document.getElementById('disp').innerHTML = name; } render() { return(
      Name: this.inputDemo = input} />

      Hello !!!

      ); } } ``` 26. 列出一些应该使用引用的情况？ 以下是应使用 ref 的情况： * 当你需要管理焦点、选择文本或媒体播放时。 * 触发命令式动画。 * 与第三方 DOM 库集成。 27. 如何模块化 React 代码？ 可以使用 export 和 import 属性来模块化软件。它们有助于在不同的文档中单独编写组件。 ```jsx //ChildComponent.jsx export default class ChildComponent extends React.Component { render() { return(

      This is a child component

      ) } } //ParentComponent.jsx import ChildComponent from './childcomponent.js'; class ParentComponent extends React.Component { render() { return(
      ); } } ``` 28. 在 React 中如何创建表单？ React 提供了一种有状态的，响应式的方法来构建表单。与其他 DOM 元素不同，HTML 表单元素在 React 中的工作机制有所不同。例如，表单数据通常由组件而不是 DOM 处理，并且通常使用受控组件来实现。 区别在于可以使用回调函数来处理表单事件，然后使用容器的状态存储表单数据。这使你的组件可以更好地控制表单控制元素和表单数据。 回调函数是在发生事件（包括更改表单控制值或表单提交）时触发的。 ``` jsx handleSubmit(event) { alert('A name was submitted: ' + this.state.value); event.preventDefault(); } render() { return (
      ); } ``` 29. 如何理解受控和非受控组件？ 受控组件|非受控组件 --|:--| 它们不维护自己的状态|它们维护自己的状态 数据由父组件控制|数据由 DOM 控制 它们通过 props 获得当前值，然后通过回调通知更改|使用引用来获得它们的当前值 30. 高阶组件（HOC）是什么意思？ React 中的高阶组件是一种在组件之间共享通用功能而无需重复代码的模式。 高阶组件实际上不是组件，它是一个接受组件并返回新组件的函数。它将一个组件转换为另一个组件，并添加其他数据或功能 31. HOC 可以做什么？ HOC 可用于许多任务，例如： * 代码复用，逻辑和引导抽象。 * 渲染劫持。 * 状态抽象和控制。 * Props 操控。 32. 什么是纯组件？ React 并没有在我们的组件中编写 shouldComponent 方法，而是引入了一个带有内置 shouldComponentUpdate 实现的新组件，它是 React.PureComponent 组件。 React.PureComponent 通过浅层 prop 和状态比较来实现它。在某些情况下，你可以使用 React.PureComponent 来提高性能。 33. React 中键有什么用途？ 键可帮助 React 识别哪些项目已更改、添加或删除。应该为数组内的元素提供键，以赋予元素稳定的身份。键必须是唯一的。 当使用动态创建的组件或用户更改列表时，React 键非常有用。设置键值后，更改后的组件就能保持唯一标识。 34. MVC 框架的主要问题有哪些？ 以下是 MVC 框架的一些主要问题： * MVC 不能解决代码复杂性问题。它也不能解决代码复用或灵活性问题。 * 它不保证解耦代码。 35. 你对 Flux 有什么了解？ Flux 是 Facebook 内部与 React 搭配使用的架构。它不是框架或库。只是一种新型的体系结构，是对 React 和单向数据流概念的补充： Flux 的各个组成部分如下： * 动作（Actions）——帮助数据传递到调度器的辅助方法。 * 调度器（Dispatcher）——接收动作并将负载广播到已注册的回调。 * 存储（Stores）——具有已注册到调度器的回调的应用程序状态和逻辑的容器。 * 控制器视图（Controller Views）——从组件中获取状态并通过 props 传递给子组件的 React 组件。 !['ui-dom-4'](./../img/ui-dom-4.png) 36. 什么是 Redux？ Redux 是一种状态管理工具。尽管它主要与 React 搭配使用，但也可以与其他任何 JavaScript 框架或库搭配。 Redux 允许你在一个称为存储（Store）的对象中管理整个应用程序状态。 对存储的更新将触发与存储的已更新部分连接的组件的重新渲染。当我们想要更新某些东西时，我们称之为动作（Action）。我们还创建函数来处理这些动作并返回更新的存储。这些函数称为 Reducer。 37. Redux 遵循的三大原则是什么？ * 单一可信来源：整个应用程序的状态存储在单个存储区中的对象 / 状态树中。单一状态树使我们能更容易地跟踪历史更改，更方便地调试或检查应用程序。 * 状态是只读的：更改状态的唯一方法是触发动作。动作是描述更改的普通 JS 对象。就像状态是数据的最小表示一样，动作是数据更改的最小表示。 * 使用纯函数更改：为了确认动作是如何转换状态树的，你需要纯函数。纯函数是返回值仅取决于其参数值的函数。 38. 如何理解“单一可信源”？ 单一可信源（SSOT）是构造信息模型和相关数据模式的实践，其中每个数据元素都只能在一个地方掌握（或编辑） Redux 使用“存储”将应用程序的整个状态存储在一个位置。因此，组件的所有状态都存储在存储中，并且存储本身会接收更新。单一状态树使我们能更容易地跟踪历史更改，更方便地调试或检查应用程序。 39. 列出 Redux 的组件。 Redux 由以下组件组成： * 动作——这是一个描述发生了什么的对象。 * Reducer——确定状态如何变化的地方。 * 存储——整个应用程序的状态 / 对象树保存在存储中。 * 视图——仅显示存储提供的数据。 40. 数据在 Redux 中是如何流动的？ !['ui-dom-5'](./../img/ui-dom-5.png) 41. 在 Redux 中如何定义动作？ React 中的动作必须具有 type 属性，该属性指示正在执行的 ACTION 的类型。必须将它们定义为字符串常量，你也可以为其添加更多属性。在 Redux 中使用称为“动作创建者”的函数来创建动作。以下是动作和动作创建者的示例： ``` jsx function addTodo(text) { return { type: ADD_TODO, text } } ``` 42. 说明 Reducer 的作用。 Reducer 是用于指示 ACTION 反应中应用程序状态变化的简单功能。它接收先前的状态和动作，然后返回新的状态。它根据动作类型确定需要哪种更新，然后返回新值。如果没有要完成的工作，它将按原样返回先前状态。 43. 在 Redux 中存储的用途是什么？ 存储是一个 JavaScript 对象，可以保存应用程序的状态，并提供一些辅助方法来访问状态、调度动作并记录侦听器。应用程序的整个状态 / 对象树存储在单个存储中。因此 Redux 非常容易理解且可预测。我们可以将中间件转移到存储，以管理数据处理任务，并维护更改存储状态的各种活动的日志。通过 Reducer，所有活动都返回新的状态。 44. Redux 与 Flux 有何不同？ Flux|Redux --|:--| 存储包括状态和更改逻辑|存储和更改逻辑是分离的 有多个存储|只有一个存储 所有存储不互通，是平行的|带有分层 Reducer 的单个存储 有单个调度器|没有调度器的概念 React 组件订阅到存储|容器组件是有联系的 状态是可变的|状态是不可变的 45. Redux 有哪些优势？ Redux 的优点如下： * 结果的可预测性——由于总是有单一可信源，比如存储，因此当前状态与动作及应用程序的其他部分同步时不会出现混乱。 * 可维护性——代码易于维护，具有可预测的结果和严格的结构。 * 服务端渲染——你只需将在服务器上创建的存储传递给客户端即可。这对于初始渲染非常有用，并优化了应用程序性能，提供了更好的用户体验。 * 开发人员工具——从动作到状态更改，开发人员可以利用这些工具实时跟踪应用程序中发生的所有事情。 * 社区和生态系统——Redux 背后拥有巨大的社区，用起来更加便利。大批优秀的开发者为库的发展做出了贡献，并开发了很多应用程序。 * 易于测试——Redux 的代码主要是较小的、纯净的和孤立的函数。这使代码可测试且独立。 * 组织——Redux 精确地规定了代码的组织方式，这使得团队合作时代码更加一致，更容易理解。 46. 什么是 React Router？ React Router 是建立在 React 之上的功能强大的路由库。它使 URL 与网页上显示的数据保持同步。它保持标准化的结构和行为，可用于开发单页 Web 应用程序。React Router 有一个简单的 API。React Router 提供了一种方法，**只会显示你的应用中路由匹配你的定义的那些组件**。 47. 为什么在 React Router v4 中使用 switch 关键字？ 在 Switch 组件内，**Route**和**Redirect**组件嵌套在内部。从 Switch 顶部的 Route/Redirect 组件开始到底部的 Route/Redirect，根据浏览器中当前的 URL 是否与 Route/Redirect 组件的 prop/ 路径匹配，将每个组件评估为 true 或 false。 Switch 只会渲染第一个匹配的子级。当我们嵌套了下面这样的路由时真的很方便： ```jsx ``` 48. 为什么我们在 React 中需要一个路由器？ 路由器用于定义多个路由，并且当用户键入特定的 URL 时，如果该 URL 与路由器内部定义的任何“路由”的路径匹配，则该用户将被重定向到该路由。因此我们需要在应用程序中添加一个路由器库，以允许创建多个路由，每个路由都为我们指向一个独特的视图。 从 React Router 包导入的组件有两个属性，一个是将用户引导到指定路径的 path，另一个是用于定义所述路径中内容的 component。 ```jsx ``` 49. 列出 React Router 的优点。 几个优点是： 1. 就像 React 基于组件的理念一样，在 React Router v4 中 API 是“完全组件化的”。路由器可以可视化为单个根组件（），其中包含特定的子路由（）。 2. 无需手动设置历史值：在 React Router v4 中，我们要做的就是将路由包装在组件中。 3. 包是拆分的：三个包分别用于 Web、Native 和 Core。这使我们的应用更加紧凑。它们的编码样式类似，所以很容易来回切换。 50. React Router 与传统路由有何不同？ ~|传统路由|React 路由 --|:--|:--| 参与的页面|每个视图对应一个新页面|只涉及单个 HTML 页面 URL 更改|向服务器发送一个 HTTP 请求并接收对应的 HTML 页面|只有历史属性被更改 体验|用户其实是在每个视图的不同页面间切换|用户以为自己正在不同的页面间切换 [原文链接：](https://codersera.com/blog/top-50-react-questions-you-need-to-prepare-for-the-interview-in-2019/ )https://codersera.com/blog/top-50-react-questions-you-need-to-prepare-for-the-interview-in-2019/ ================================================ FILE: docs/web/vue_cp_react.md ================================================ # 前端框架用vue还是react？清晰对比两者差异 ## 前言 近两年前端技术层出不穷，目前市面上已经有了很多供前端人员使用的开发框架，转眼19年已过大半，前端框架领域日趋成熟，实现了三足鼎立的局面，截止到10月22日，Angular，react和vue数据统计如下图所示: ![vue-react-1](./../img/vue-react-1.png) 最近在学习使用框架的时候，分别使用vue和react开发了两个移动端产品，对这两个框架的学习曲线有了一些感悟，这两个都是现在比较热门的js框架，它俩在使用方式上和学习复杂度上还是有很大区别的，这里简单总结下两者的差异。 主要从以下几个方面入手方面展开： * 框架的诞生 * 设计思想 * 编写语法 * 脚手架构建工具 * 数据绑定 * 虚拟DOM * 指令 * 性能优化 * 原生渲染native * ssr服务端渲染 * 生命周期函数 * 销毁组件 * 状态集管理工具 ## 诞生 ### vue vue由尤雨溪开发，由独立团队维护，现在大部分的子项目都交给团队成员打理，Vue核心库依然主要由尤雨溪亲自维护。vue近几年来特别的受关注，三年前的时候angularJS霸占前端JS框架市场很长时间，接着react框架横空出世，因为它有一个特性是虚拟DOM，从性能上碾轧angularJS，这个时候，vue1.0悄悄的问世了，它的优雅，轻便也吸引了一部分用户，开始受到关注，16年中旬，VUE2.0问世，不管从性能上，还是从成本上都隐隐超过了react，火的一塌糊涂，这个时候，angular开发团队也开发了angular2.0版本，并且更名为angular，吸收了react、vue的优点，加上angular本身的特点，也吸引到很多用户，目前已经迭代到8.0了。友情提示注意下vue的诞生时间，如果正好有小伙伴在面试，被问到你是从什么时候开始接触并且使用vue的，你要是回答用了5、6年了那场面就十分尴尬了。 ### react 起初facebook在建设instagram（图片分享）的时候，因为牵扯到一个东西叫数据流，那为了处理数据流并且还要考虑好性能方面的问题，Facebook开始对市场上的各种前端MVC框架去进行一个研究，然而并没有看上眼的，于是Facebook觉得，还是自己开发一个才是最棒的，那么他们决定抛开很多所谓的“最佳实践”，重新思考前端界面的构建方式，他们就自己开发了一套，果然大牛创造力还是很强大的。 React 起源于 Facebook 的内部项目，因为该公司对市场上所有 JavaScript MVC 框架，都不满意，就决定自己写一套，用来架设Instagram 的网站。做出来以后，发现这套东西很好用，就在2013年5月开源了。 ## 设计思想 ### vue vue的官网中说它是一款渐进式框架，采用自底向上增量开发的设计。这里我们需要明确一个概念，什么是渐进式框架。在声明式渲染（视图模板引擎）的基础上，我们可以通过添加组件系统（components）、客户端路由（vue-router）、大规模状态管理（vuex）来构建一个完整的框架。Vue从设计角度来讲，虽然能够涵盖所有这些内容，但是你并不需要一上手就把所有东西全用上，因为没有必要。无论从学习角度，还是实际情况，这都是可选的。声明式渲染和组建系统是Vue的核心库所包含内容，而客户端路由、状态管理、构建工具都有专门解决方案。这些解决方案相互独立，你可以在核心的基础上任意选用其他的部件，不一定要全部整合在一起。可以看到，所说的“渐进式”，其实就是Vue的使用方式，同时也体现了Vue的设计的理念。 ### react react主张函数式编程，所以推崇纯组件，数据不可变，单向数据流，当然需要双向的地方也可以手动实现，比如借助 onChange 和 setState 来实现一个双向的数据流。而vue是基于可变数据的，支持双向绑定，它提供了v-model这样的指令来实现文本框的数据流双向绑定。 ## 编写语法 ### vue vue推荐的做法是webpack+vue-loader的单文件组件格式，vue保留了html、css、js分离的写法，使得现有的前端开发者在开发的时候能保持原有的习惯，更接近常用的web开发方式，模板就是普通的html，数据绑定使用mustache风格，样式直接使用css。其中**style**标签还提供了一个可选的scoped属性，它会为组件内 CSS 指定作用域，用它来控制仅对当前组件有效还是全局生效。 模板和JSX是各有利弊的东西。模板更贴近我们的HTML，可以让我们更直观地思考语义结构，更好地结合CSS的书写。 同时vue也支持JSX语法，因为是真正的JavaScript，拥有这个语言本身的所有的能力，可以进行复杂的逻辑判断，进行选择性的返回最终要返回的DOM结构，能够实现一些在模板的语法限制下，很难做到的一些事情。 ### react 用过react的开发者可能知道，react是没有模板的，直接就是一个渲染函数，它中间返回的就是一个虚拟DOM树，React推荐的做法是 JSX + inline style, 也就是把HTML和CSS全都写进JavaScript了,即'all in js'。JSX实际就是一套使用XML语法，用于让我们更简单地去描述树状结构的语法糖。在react中，所有的组件的渲染功能都依靠JSX。你可以在render()中编写类似XML的语法，它最终会被编译成原生JavaScript。不仅仅是 HTML 可以用 JSX 来表达，现在的潮流也越来越多地将 CSS 也纳入到 JavaScript 中来处理。JSX是基于 JS 之上的一套额外语法，学习使用起来有一定的成本。 ## 构建工具 ### vue vue提供了CLI 脚手架，可以帮助你非常容易地构建项目。全局安装之后，我们就可以用 vue create命令创建一个新的项目，vue 的 CLI 跟其他 CLI不同之处在于，有多个可选模板，有简单的也有复杂的，可以让用户自定义选择需要安装的模块，还可以将你的选择保存成模板，便于后续使用。 极简的配置，更快的安装，可以更快的上手。它也有一个更完整的模板，包括单元测试在内的各种内容都涵盖，但是，它的复杂度也更高，这又涉及到根据用例来选择恰当复杂度的问题。 ### react React 在这方面也提供了 create-react-app，但是现在还存在一些局限性： * 它不允许在项目生成时进行任何配置，而 Vue CLI 运行于可升级的运行时依赖之上，该运行时可以通过插件进行扩展。 * 它只提供一个构建单页面应用的默认选项，而 Vue 提供了各种用途的模板。 * 它不能用用户自建的预设配置构建项目，这对企业环境下预先建立约定是特别有用的。 而要注意的是这些限制是故意设计的，这有它的优势。例如，如果你的项目需求非常简单，你就不需要自定义生成过程。你能把它作为一个依赖来更新。 ## 数据绑定 ### vue vue是实现了双向数据绑定的mvvm框架，当视图改变更新模型层，当模型层改变更新视图层。在vue中，使用了双向绑定技术，就是View的变化能实时让Model发生变化，而Model的变化也能实时更新到View。 Vue采用数据劫持&发布-订阅模式的方式，vue在创建vm的时候，会将数据配置在实例当中，然后通过Object.defineProperty对数据进行操作，为数据动态添加了getter与setter方法，当获取数据的时候会触发对应的getter方法，当设置数据的时候会触发对应的setter方法，从而进一步触发vm的watcher方法，然后数据更改，vm则会进一步触发视图更新操作。 ### react react是单向数据流，react中属性是不允许更改的，状态是允许更改的。react中组件不允许通过this.state这种方式直接更改组件的状态。自身设置的状态，可以通过setState来进行更改。在setState中，传入一个对象，就会将组件的状态中键值对的部分更改，还可以传入一个函数，这个回调函数必须向上面方式一样的一个对象函数可以接受prevState和props。通过调用this.setState去更新this.state,不能直接操作this.state，请把它当成不可变的。 调用setState更新this.state，它不是马上就会生效的，它是异步的。所以不要认为调用完setState后可以立马获取到最新的值。多个顺序执行的setState不是同步的一个接着一个的执行，会加入一个异步队列，然后最后一起执行，即批处理。 setState是异步的，导致获取dom可能拿的还是之前的内容，所以我们需要在setState第二个参数（回调函数）中获取更新后的新的内容。 ## diff算法 ### vue vue中diff算法实现流程 在内存中构建虚拟dom树 将内存中虚拟dom树渲染成真实dom结构 数据改变的时候，将之前的虚拟dom树结合新的数据生成新的虚拟dom树 将此次生成好的虚拟dom树和上一次的虚拟dom树进行一次比对（diff算法进行比对），来更新只需要被替换的DOM，而不是全部重绘。在Diff算法中，只平层的比较前后两棵DOM树的节点，没有进行深度的遍历。 会将对比出来的差异进行重新渲染 ### react react中diff算法实现流程 DOM结构发生改变-----直接卸载并重新create DOM结构一样-----不会卸载,但是会update变化的内容 所有同一层级的子节点.他们都可以通过key来区分-----同时遵循1.2两点 （其实这个key的存在与否只会影响diff算法的复杂度,换言之,你不加key的情况下,diff算法就会以暴力的方式去根据一二的策略更新,但是你加了key,diff算法会引入一些另外的操作） React会逐个对节点进行更新，转换到目标节点。而最后插入新的节点，涉及到的DOM操作非常多。diff总共就是移动、删除、增加三个操作，而如果给每个节点唯一的标识（key），那么React优先采用移动的方式，能够找到正确的位置去插入新的节点。 vue会跟踪每一个组件的依赖关系，不需要重新渲染整个组件树。而对于React而言,每当应用的状态被改变时,全部组件都会重新渲染,所以react中会需要shouldComponentUpdate这个生命周期函数方法来进行控制。 ## 指令 指令 (Directives) 是带有 v- 前缀的特殊特性，指令的职责是，当表达式的值改变时，将其产生的连带影响，响应式地作用于 DOM。 ### vue vue中提供很多内部指令供我们使用，它可以让我们进行一些模板的操作，例如有时候，我们的data中的存放的数据不是个简单的数字或者字符串，而是数组Array类型，这个时候，我们要把数组的元素展示在视图上，就需要用到vue提供的 v-for 指令，来实现列表的渲染。 ### react 因为react中没有v-for指令，所以循环渲染的时候需要用到map()方法来渲染视图，并且将符合条件的元素放入一个新数组返回。 ## 性能优化 ### vue vue中的每个组件内部自动实现了 shouldComponentUpdate的优化，在vue里面由于依赖追踪系统的存在，当任意数据变动的时，Vue的每一个组件都精确地知道自己是否需要重绘，所以并不需要手动优化。用vue渲染这些组件的时候，数据变了，对应的组件基本上去除了手动优化的必要性。而在react中我们需要手动去优化其性能，但是当数据特别多的时候vue中的watcher也会特别多，从而造成页面卡顿，所以一般数据比较多的大型项目会倾向于使用react。在react官网中，官方也建议我们使用React来构建快速响应的大型 Web 应用程序。 ### react 当props或state发生改变的时候会触发 shouldComponentUpdate生命周期函数，它是用来控制组件是否被重新渲染的，如果它返回true，则执行render函数，更新组件；如果它返回false，则不会触发重新渲染的过程。 有的时候我们希望它在更新之前，和之前的状态进行一个对比，这个时候我们就需要重写 shouldComponentUpdate来避免不必要的dom操作，对比当前的props或state和更新之后的nextProps或nextState，返回true时 ，组件更新；返回false，则不会更新，节省性能。 ``` js shouldComponentUpdate(nextProps, nextState) { if (this.props.a !== nextProps.a) { return true; } if (this.state.b !== nextState.b) { return true; } return false; } ``` 我们也可以创建一个继承React.PureComponent的React组件，它自带 shouldComponentUpdate，可以对props进行浅比较，发现更新之后的props与当前的props一样，就不会进行render了。 classTestextendsReact.PureComponent{constructor(props){super(props);}render(){return
      hello...{this.props.a}
      }} 由于React.PureComponent进行的是浅比较，也就是说它只会对比原对象的值是否相同，当我们的props或state为数组或者对象这种引用类型的时候，我们修改它的数值，由于数据引用指针没有发生改变，所以组件也是不会重新渲染的。这个时候我们就需要进行深拷贝，创建一个新的对象或数组，将原对象的各项属性的"值"（数组的所有元素）拷贝过来，是"值"而不仅仅是"引用地址"。我们可以使用slice()方法： ``` js ew_state.todos = new_state.todos.slice(); ``` 或者引入immutable库来实现数据不可变。 ## 原生渲染native native指的是使用原生API来开发App，比如ios使用OC语言，android使用java。 ### vue vue和Weex进行官方合作，weex是阿里巴巴发起的跨平台用户界面开发框架，它的思想是多个平台，只写一套代码，weex允许你使用 vue 语法开发不仅仅可以运行在浏览器端，还能被用于开发 iOS 和 Android 上的原生应用的组件。即只需要编写一份代码，即可运行在Web、iOS、Android上。 weex相对来说上手比较简单，安装vue-cli之后就可以使用，学习门槛低，但是它的社区目前还处于成长期，react native的社区非常成熟活跃，有非常丰富的组件可供扩展。 ### react react native是Facebook在2015年3月在F8开发者大会上开源的跨平台UI框架，需针对iOS、Android不同编写2份代码，使用react native需要按照文档安装配置很多依赖的工具，相对比较麻烦。weex的思想是多个平台，只写一套代码，而react-native的思想是多个平台可以写多套代码，但其使用的是同一套语言框架。 weex的目标在于抹平各个平台的差异性，从而简化应用开发。而react-native承认了各个平台之间的差异，退而求其次，在语言和框架层面对平台进行抽象，从方法论的角度去解决多平台开发的问题。 ## ssr服务端渲染 服务端渲染核心在于方便seo优化，后端先调用数据库，获得数据之后，将数据和页面元素进行拼装，组合成完整的html页面，再直接返回给浏览器，以便用户浏览。 ### vue 2016 年 10 月 25 日，zeit.co背后的团队对外发布了 Next.js，一个 React 的服务端渲染应用框架。几小时后，与 Next.js 异曲同工，一个基于 Vue.js 的服务端渲染应用框架应运而生，我们称之为：Nuxt.js。 服务端渲染支持流式渲染，因为HTTP请求也是流式，Vue 的服务端渲染结果可以直接 pipe 到返回的请求里面。这样一来，就可以更早地在浏览器中呈现给用户内容，通过合理的缓存策略，可以有效地提升服务端渲染的性能。 * 基于 Vue.js * 自动代码分层 * 服务端渲染 * 强大的路由功能，支持异步数据 * 静态文件服务 * ES2015+ 语法支持 * 打包和压缩 JS 和 CSS * HTML 头部标签管理 * 本地开发支持热加载 * 集成 ESLint * 支持各种样式预处理器： SASS、LESS、 Stylus 等等 * 支持 HTTP/2 推送 ### react Next是一个React框架，允许使用React构建SSR和静态web应用 * 服务器渲染，获取数据非常简单 * 无需学习新框架，支持静态导出。 * 支持CSS-in-JS库 * 自动代码拆分，加快页面加载速度，不加载不必要的代码 * 基于Webpack的开发环境，支持模块热更新（HMR） * 支持Babel和Webpack自定义配置服务器、路由和next插件。 * 能够部署在任何能运行node的平台 * 内置页面搜索引擎优化（SEO）处理 * 在生产环境下，打包文件体积更小，运行速度更快 ## 生命周期 ### vue 【初始化阶段（4个）】 （1）beforeCreate 此钩子函数不能获取到数据，dom元素也没有渲染出来，此钩子函数不会用来做什么事情。 （2）created 此钩子函数，数据已经挂载了，但是dom节点还是没有渲染出来，在这个钩子函数里面，如果同步更改数据的话，不会影响运行中钩子函数的执行。可以用来发送ajax请求，也可以做一些初始化事件的相关操作。 （3）beforeMount 代表dom节点马上要被渲染出来了，但是还没有真正的渲染出来，此钩子函数跟created钩子函数基本一样，也可以做一些初始化数据的配置。 （4）mounted 是生命周期初始化阶段的最后一个钩子函数，数据已经挂载完毕了，真实dom也可以获取到了。 【运行中阶段（2个）】 （5）beforeUpdate 运行中钩子函数beforeUpdate默认是不会执行的，当数据更改的时候，才会执行。数据更新的时候，先调用beforeUpdate，然后数据更新引发视图渲染完成之后，再会执行updated。运行时beforeUpdate这个钩子函数获取的数据还是更新之前的数据（获取的是更新前的dom内容），在这个钩子函数里面，千万不能对数据进行更改，会造成死循环。 （6）updated 这个钩子函数获取的数据是更新后的数据，生成新的虚拟dom，跟上一次的虚拟dom结构进行比较，比较出来差异（diff算法）后再渲染真实dom，当数据引发dom重新渲染的时候，在updated钩子函数里面就可以获取最新的真实dom了。 【销毁阶段（2个）】 （7）beforeDestroy 切换路由的时候，组件就会被销毁了，销毁之前执行beforeDestroy。在这个钩子函数里面，我们可以做一些善后的操作，例如可以清空一下全局的定时器（created钩子函数绑定的初始化阶段的事件）、清除事件绑定。 （8）destoryed 组件销毁后执行destroyed，销毁后组件的双向数据绑定、事件监听watcher相关的都被移除掉了，但是组件的真实dom结构还是存在在页面中的。 添加keep-alive标签后会增加active和deactive这两个生命周期函数，初始化操作放在actived里面，一旦切换组件，因为组件没有被销毁，所以它不会执行销毁阶段的钩子函数，所以移除操作需要放在deactived里面，在里面进行一些善后操作，这个时候created钩子函数只会执行一次，销毁的钩子函数一直没有执行。 ![vue-react-2](./../img/vue-react-2.png) ### react 【初始化阶段（5个）】： （1）getDefaultProps：实例化组件之后，组件的getDefaultProps钩子函数会执行 这个钩子函数的目的是为组件的实例挂载默认的属性 这个钩子函数只会执行一次，也就是说，只在第一次实例化的时候执行，创建出所有实例共享的默认属性，后面再实例化的时候，不会执行getDefaultProps，直接使用已有的共享的默认属性 理论上来说，写成函数返回对象的方式，是为了防止实例共享，但是react专门为了让实例共享，只能让这个函数只执行一次 组件间共享默认属性会减少内存空间的浪费，而且也不需要担心某一个实例更改属性后其他的实例也会更改的问题，因为组件不能自己更改属性，而且默认属性的优先级低。 （2）getInitialState：为实例挂载初始状态，且每次实例化都会执行，也就是说，每一个组件实例都拥有自己独立的状态。 （3）componentWillMount：执行componentWillMount，相当于Vue里的created+beforeMount，这里是在渲染之前最后一次更改数据的机会，在这里更改的话是不会触发render的重新执行。 （4）render：渲染dom render()方法必须是一个纯函数，他不应该改变 state，也不能直接和浏览器进行交互，应该将事件放在其他生命周期函数中。 如果 shouldComponentUpdate()返回 false， render()不会被调用。 （5）componentDidMount：相当于Vue里的mounted,多用于操作真实dom 【运行中阶段（5个）】 当组件mount到页面中之后，就进入了运行中阶段，在这里有5个钩子函数，但是这5个函数只有在数据（属性、状态）发送改变的时候才会执行 （1）componentWillReceiveProps(nextProps,nextState) 当父组件给子组件传入的属性改变的时候，子组件的这个函数才会执行。初始化props时候不会主动执行 当执行的时候，函数接收的参数是子组件接收到的新参数，这个时候，新参数还没有同步到this.props上,多用于判断新属性和原有属性的变化后更改组件的状态。 （2）接下来就会执行shouldComponentUpdate(nextProps,nextState),这个函数的作用：当属性或状态发生改变后控制组件是否要更新，提高性能,返回true就更新，否则不更新，默认返回true。 接收nextProp、nextState，根据根据新属性状态和原属性状态作出对比、判断后控制是否更新 如果 shouldComponentUpdate()返回 false， componentWillUpdate, render和 componentDidUpdate不会被调用。 （3）componentWillUpdate,在这里，组件马上就要重新render了，多做一些准备工作，千万千万，不要在这里修改状态，否则会死循环 相当于Vue中的beforeUpdate （4）render，重新渲染dom （5）componentDidUpdate，在这里，新的dom结构已经诞生了,相当于Vue里的updated 【销毁阶段】 当组件被销毁之前的一刹那，会触发componentWillUnmount，临死前的挣扎 相当于Vue里的beforeDestroy，所以说一般会做一些善后的事情，例如使定时器无效，取消网络请求或清理在 componentDidMount中创建的任何监听。 ![vue-react-3](./../img/vue-react-3.png) ## 销毁组件 ### vue vue在调用$destroy方法的时候就会执行beforeDestroy生命周期函数，然后组件被销毁，这个时候组件的dom结构还存在于页面结构中，也就说如果想要对残留的dom结构进行处理必须在destroyed生命周期函数中处理。 ### react react执行完componentWillUnmount之后把事件、数据、dom都全部处理掉了，也就是说当父组件从渲染这个子组件变成不渲染这个子组件的时候，子组件相当于被销毁，所以根本不需要其他的钩子函数了。react销毁组件的时候，会将组件的dom结构也移除，vue则不然，在调用destory方法销毁组件的时候，组件的dom结构还是存在于页面中的，this.$destory组件结构还是存在的，只是移除了事件监听，所以这就是为什么vue中有destroyed，而react却没有componentDidUnmount。 ## 状态集管理工具 ### vue vuex是一个专门为vue构建的状态集管理工具，vue和react都是基于组件化开发的，项目中包含很多的组件，组件都会有组件嵌套，想让组件中的数据被其他组件也可以访问到就需要使用到Vuex。 vuex的流程 将需要共享的状态挂载到state上：this.$store.state来调用 创建store，将状态挂载到state上，在根实例里面配置store，之后我们在组件中就可以通过this.$store.state来使用state中管理的数据，但是这样使用时，当state的数据更改的时候，vue组件并不会重新渲染，所以我们要通过计算属性computed来使用，但是当我们使用多个数据的时候这种写法比较麻烦，vuex提供了mapState辅助函数，帮助我们在组件中获取并使用vuex的store中保存的状态。 我们通过getters来创建状态：通过this.$store.getters来调用 可以根据某一个状态派生出一个新状态，vuex也提供了mapGetters辅助函数来帮助我们在组件中使用getters里的状态。 使用mutations来更改state：通过this.$store.commit来调用 我们不能直接在组件中更改state，而是需要使用mutations来更改，mutations也是一个纯对象，里面包含很多更改state的方法，这些方法的形参接收到state，在函数体里更改，这时，组件用到的数据也会更改，实现响应式。vuex提供了mapMutations方法来帮助我们在组件中调用mutations 的方法。 使用actions来处理异步操作：this.$store.dispatch来调用 Actions类似于mutations，不同在于：Actions提交的是mutations，而不是直接变更状态。Actions可以包含任意异步操作。也就是说，如果有这样的需求：在一个异步操作处理之后，更改状态，我们在组件中应该先调用actions，来进行异步动作，然后由actions调用mutations来更改数据。在组件中通过this.$store.dispatch方法调用actions的方法，当然也可以使用mapMutations来辅助使用。 ### react 2015年Redux出现，将 Flux 与函数式编程结合一起，很短时间内就成为了最热门的前端架构。它的出现主要是为解决react中组件之间的通信问题。建议把数据放入到redux中管理，目的就是方便数据统一，好管理。项目一旦出现问题，可以直接定位问题点。组件扩展的时候，后续涉及到传递的问题。本来的话，组件使用自己的数据，但是后来公用组件，还需要考虑如何值传递，在redux中可以存储至少5G以上的数据。 redux的流程 ![vue-react-4](./../img/vue-react-4.png) 1. 创建store： 从redux工具中取出createStore去生成一个store。 2. 创建一个reducer，然后将其传入到createStore中辅助store的创建。 reducer是一个纯函数，接收当前状态和action，返回一个状态，返回什么，store的状态就3是什么，需要注意的是，不能直接操作当前状态，而是需要返回一个新的状态。 想要给store创建默认状态其实就是给reducer一个参数创建默认值。 4. 组件通过调用store.getState方法来使用store中的state，挂载在了自己的状态上。 5. 组件产生用户操作，调用actionCreator的方法创建一个action，利用store.dispatch方法传递给reducer 6. reducer对action上的标示性信息做出判断后对新状态进行处理，然后返回新状态，这个时候store的数据就会发生改变， reducer返回什么状态，store.getState就可以获取什么状态。 7. 我们可以在组件中，利用store.subscribe方法去订阅数据的变化，也就是可以传入一个函数，当数据变化的时候，传入的函数会执行，在这个函数中让组件去获取最新的状态。 ## 小结 vue和react的核心都是专注于轻量级的视图层，虽然只是解决一个很小的问题，但是它们庞大的生态圈提供了丰富的配套工具，一开始它并不会给你提供全套的配置方案，将所有的功能都一次性给你打包好，它只会给你提供一些简单的核心功能，当你需要做一个更复杂的应用时，再增添相应的工具。例如做一个单页应用的时候才需要用路由；做一个相当庞大的应用，涉及到多组件状态共享以及多个开发者共同协作时，才可能需要大规模状态管理方案。 框架的存在就是为了帮助我们应对不同的项目复杂度，当我们面对一个大型、复杂的开发项目时，使用太简陋的工具会极大的降低开发人员的生产力，影响工作效率，框架的诞生就是在这些工程中提取一些重复的并且已经受过验证的模式，抽象到一个已经帮你设计好的API封装当中，帮助我们去应对不同复杂度的问题。所以在开发的过程中，选择一个合适的框架就会事半功倍。但是，框架本身也有复杂度，有些框架会让人一时不知如何上手。当你接到一个并不复杂的需求，却使用了很复杂的框架，那么就相当于杀鸡用牛刀，会遇到工具复杂度所带来的副作用，不仅会失去工具本身所带来优势，还会增加各种问题，例如学习成本、上手成本，以及实际开发效率等。 所以并不是说做得少的框架就不如做的做的框架，每个框架都有各自的优势和劣势，并不能找到完全符合需求的框架，最重要的适合当前项目，目前两大框架的生态圈一片繁荣，react社区是当前最活跃的，最快的时候三天更新一个版本，一个问题可能存在几十种不同的解决方案，这就需要我们前端人员去在不同的功能之间做取舍，以后前端框架的发展方向应该是小而精、灵活以及开放的，核心功能+生态附加库可以帮我们更加灵活的构建项目，为了跟上前进的脚步，就需要不停的吸收最新的内容，这也是从事前端开发领域的一大乐趣，希望大家都能在学习中获得长足的进步。 ================================================ FILE: mkdocs.yml ================================================ site_name: BurningMyself site_description: .net,java,php,python,docker,web site_author: BurningMyself site_url: https://burningmyself.github.io # site_favicon: favicon.ico copyright: BurningMyself 蜀ICP备15033200号-1 site_url: https://burningmyself.gitee.io repo_name: 'burningmyself.github.io' repo_url: 'https://github.com/burningmyself/burningmyself.github.io' theme: name: material highlightjs: true favicon: favicon.ico language: 'zh' feature: tabs: false hljs_languages: - yaml - rust shortcuts: help: 191 # ? next: 78 # n previous: 80 # p search: 83 # s nav: - 介绍: index.md - 技术博文: - .NET: - C#新特性语法: net/c_sharp.md - .Net Core Docker 部署: net/c_docker.md - Docker 容器化部署 ASP.NET Core: net/c_sqlserver_nginx.md - 让你的ASP.NET Core应用程序更安全: net/c_core_safety.md - ASP.NET Core开发者指南: net/c_core_study_route.md - Java: - Java特性: java/feature.md - Java类加载: java/load-class.md - Orm的优缺点: java/orm.md - BIO与NIO: java/bio-nio.md - RocketMq下载与安装: java/rocketmq/rmq-1.md - Spring常用注解: java/springAnnotation.md - Java 虚拟机: java/javavm.md - Spring 常用设计: java/springdesign.md - Spring Cloud: java/spring-cloud.md - Java simple: java/java-simple.md - Java Validator: java/java-validator.md - Java String: java/java-string.md - Java Utils: java/java-utils.md - Python: - Python特性: python/feature.md - Python字符拼接: python/str_joint.md - Python语法技巧: python/syntax_rule.md - 远程部署神器 Fabric: python/fabric.md - Go: - Go基础语法: go/go_base.md - PHP: - 框架: php/kj.md - Web: - es6语法: web/es6.md - 阿里js样式规范: web/ali_js_style.md - node.js文档: web/node.js.md - React 开发者指南: web/react.md - Dart语法学习: dart/syntax.md - React 面试: web/react_interview.md - js 工具函数: web/js_tool_method.md - vue与react比较: web/vue_cp_react.md - JavaScript 基础: web/javascript.md - SQL: - MySQL优化: sql/mysql_yh.md - MySQL优化: sql/mysql_yh17.md - MySQL日志: sql/mysql_log.md - MySQL索引: sql/mysql_index.md - MySQL PXC集群: sql/mysql_pxc.md - MySQL 数据库应用: sql/mysql_use.md - MySql 备份: sql/mysql_backups.md - Sql Server 主从备份: sql/sql_server_master.md - 数据库之互联网常用分库分表方案: sql/data_split.md - Mybatis使用心德: sql/mybatis.md - Linux: - Linux学习笔记: linux/linux.md - 常用命令: linux/often.md - 实用的Linux 命令: linux/ope.md - Docker: - Docker全科: docker/docker.md - Docker 核心技术: docker/docker_core.md - docker和docker-compose配置常用环境: docker/docker-compose.md - Docker部署Jenkins: docker/docker-jenkins.md - Docker部署Phabricator: docker/docker-phabricator.md - Kubernetes: - Kubernetes架构原则和对象设计: kubernetes/k8s_fw_rule_and_object_design.md - Kubernetes控制平面组件etcd: kubernetes/k8s_etcd.md - 深入理解Kube-APIServer: kubernetes/k8s_kube_APIServer.md - kubernetes控制平面组件: kubernetes/k8s_controller_manager.md - Tool: - Git: tool/git.md - Git动画展示: tool/gitusual.md - Git问题处理: tool/gitquestion.md - GitFlow: tool/gitflow.md - GitBook: tool/gitbook.md - Git提交日志规范: tool/gitcmr.md - Markdown: tool/markdown.md - CI/CD: tool/cicd.md - mkdocs简单使用: tool/mkdocs.md - Git的黑魔法: tool/gitstudy.md - MinIO 搭建使用: tool/minio.md - Cat分布式监控: tool/cat-monitoring.md - 云架构: - 云原生: cloud/native.md - 虚拟化技术: cloud/virtual.md - 云计算: cloud/compute.md - 应用部署容器化演进: cloud/apprelease.md - 容器技术所涉及Linux内核关键技术: cloud/containerlinux.md - Docker: - 容器管理工具Docker生态架构及部署: cloud/docker/docker_native.md - 使用容器运行Nginx应用及Docker命令: cloud/docker/docker_nginx.md - Docker容器镜像: cloud/docker/docker_image.md - Docker容器镜像加速器及本地容器镜像仓库: cloud/docker/docker_image_fast.md - Docker容器化部署企业级应用集群: cloud/docker/docker_container_enterprice.md - Dockerfile精讲及新型容器镜像构建技术: cloud/docker/docker_file.md - Docker容器网络与通信原理深度解析: cloud/docker/docker_network.md - Docker容器数据持久化存储机制: cloud/docker/docker_date.md - Docker容器服务编排利器Docker Compose应用实战: cloud/docker/docker_compose.md - Docker主机集群化方案 Docker Swarm: cloud/docker/docker_swarm.md - 基于Docker容器DevOps应用方案 企业业务代码发布系统: cloud/docker/docker_devops.md - 轻量级或工业级容器管理工具: cloud/docker/docker_container.md - Kubernetes: - kubeadm极速部署Kubernetes: cloud/kubernetes.md - kubernetes介绍与集群架构: cloud/kubernetes/kubernetes_introduce.md - Kubernetes集群部署方式说明: cloud/kubernetes/kubernetes_way.md - kubeadm部署单Master节点kubernetes集群: cloud/kubernetes/kubernetes_master.md - kubeadm部署高可用kubernetes集群: cloud/kubernetes/kubernetes_hight.md - 使用RKE构建企业生产级Kubernetes集群: cloud/kubernetes/kubernetes_rke.md - Kubernetes高可用集群二进制部署（Runtime Docker）: cloud/kubernetes/kubernetes_hight_bin1.md - Kubernetes高可用集群二进制部署（Runtime Containerd）: cloud/kubernetes/kubernetes_hight_bin2.md - Kubernetes集群UI及主机资源监控: cloud/kubernetes/kubernetes_ui.md - 使用sealos部署kubernetes集群并实现集群管理: cloud/kubernetes/kubernetes_sealos.md - kubernetes集群命令语法: cloud/kubernetes/kubernetes_cluster.md - Kubernetes核心概念: cloud/kubernetes/kubernetes_core.md - Kubernetes集群 服务暴露 Nginx Ingress Controller: cloud/kubernetes/kubernetes_nginx_ingress_controller.md - Kubernetes集群 服务暴露 Traefik: cloud/kubernetes/kubernetes_traefik.md - Kubernetes配置与密钥管理 ConfigMap&Secret: cloud/kubernetes/kubernetes_configMap_secret.md - Kubernetes集群使用容器镜像仓库Harbor: cloud/kubernetes/kubernetes_harbor.md - Kubernetes集群安全管理: cloud/kubernetes/kubernetes_safety.md - kubernetes持久化存储卷: cloud/kubernetes/kubernetes_storage_volume.md - kubernetes存储解决方案Ceph: cloud/kubernetes/kubernetes_storage_ceph.md - Kubernetes集群公共服务: cloud/kubernetes/kubernetes_cluster_serve.md - kubernetes集群java项目上云部署: cloud/kubernetes/kubernetes_deploy_java.md - kubernetes集群Python项目上云部署: cloud/kubernetes/kubernetes_deploy_python.md - Kubernetes集群golang项目上云部署: cloud/kubernetes/kubernetes_deploy_golang.md - helm部署prometheus监控系统及应用: cloud/kubernetes/kubernetes_helm_prometheus.md - kubernetes日志收集方案ELK: cloud/kubernetes/kubernetes_logs_collect.md - 企业级中间件上云部署zookeeper: cloud/kubernetes/kubernetes_zookeeper.md - kubernetes云原生中间件上云部署kafka: cloud/kubernetes/kubernetes_kafka.md - rocketmq部署: cloud/kubernetes/kubernetes_rokectmq.md - Kubernetes集群包管理解决方案 Helm: cloud/kubernetes/kubernetes_helm.md - Kubernetes原生配置管理利器 kustomize: cloud/kubernetes/kubernetes_kustomize.md - kubernetes集群网络解决方案 flannel: cloud/kubernetes/kubernetes_flannel.md - kubernetes集群网络解决方案 calico: cloud/kubernetes/kubernetes_calico.md - kubernetes集群 underlay 网络方案 hybridnet: cloud/kubernetes/kubernetes_hybridnet.md - kubernetes版本双栈协议（IPv4&IPv6）集群部署: cloud/kubernetes/kubernetes_ipv4_and_ipv6.md - Rancher容器云管理平台: cloud/kubernetes/kubernetes_rancher.md - 使用kubeconfig管理多集群方法: cloud/kubernetes/kubernetes_kubeconfig.md - karmada实现k8s集群联邦: cloud/kubernetes/kubernetes_karmada.md - 安装kubesphere使用: cloud/kubernetes/kubernetes_kubesphere.md - 阿里云容器服务ACK: cloud/kubernetes/kubernetes_ack.md - 基于kubernetes集群构建大中型企业CICD应用平台: cloud/kubernetes/kubernetes_devops.md - 基于KubeSphere实现DevOps: cloud/kubernetes/kubernetes_kubesphere_devops.md - 云原生多云持续交付GitOps: cloud/kubernetes/kubernetes_gitops.md - kubernetes集群备份与恢复管理利器Velero: cloud/kubernetes/kubernetes_velero.md - kubernetes集群舰队管理Kurator: cloud/kubernetes/kubernetes_kurator.md - Serverless之OpenFaaS函数即服务: cloud/kubernetes/kubernetes_openfass.md - Flink基于Kubernetes部署: cloud/kubernetes/kubernetes_flink.md - 大数据HDFS分布式文件系统搭建: cloud/kubernetes/kubernetes_hdfs.md - Spark与Kubernetes整合: cloud/kubernetes/kubernetes_spark.md - 源监控: cloud/kubernetes/kubernetes_ui.md - 微服务: - 分布式锁: micro/fbs-lock.md - 微服务设计: micro/design.md - 分布式系统与消息的投递: micro/distrimsg.md - 基于DDD的微服务设计和开发实战: micro/ddd.md - Kafka架构原理: micro/kafka.md - Redis Cluster原理: micro/redis_cluster.md - 分布式架构: micro/spring-cloud-micro.md - 经验分析: - DevOps: exp/devops.md - Micro-Service: exp/micro-service.md - Raft算法和Gossip协议: exp/raft-gossip.md - 集群和负载均衡: exp/cl.md - 人工智能: exp/ai.md - 代码原则: exp/code-principle.md - 如何成为技术大牛: exp/techbig.md - 程序员如何技术成长: exp/learnweetout.md - 产品与技术: exp/pt.md - 深度学习框架: exp/four_deep_learning.md - 在阿里做了5年技术Leader，我总结出这些套路！: exp/tl.md - CTO 技能图谱 : exp/cto.md - 构架: - 构架拆分: framework/split.md - 分布式、高并发、多线程: framework/fgb.md - 如何理解敏捷开发: framework/agility.md - 走向架构师必备的技能: framework/fwork.md - 数据中台的思考与总结: framework/data_middle.md - 必学的 10 大算法: framework/algorithm-ten.md - 情感: - 情商: emotion/eq.md - 工作心得: emotion/workheard.md - 看书: emotion/lookbook.md - 自律: emotion/selfdiscipline.md - 为什么活着失败: emotion/livefail.md - 情绪管理: emotion/emotion.md - 所有的失去，都会以另一种方式归来: emotion/losecome.md - 人生有这三种好心态: emotion/threeheart.md - 余生，学会一个人走，不管有没有人陪: emotion/onepath.md - 往后余生还很精彩，别被熬夜拖垮了: emotion/twopath.md - 心态好的人，一辈子都好: emotion/lifetime.md markdown_extensions: - admonition - codehilite: guess_lang: false linenums: false - toc: permalink: true baselevel: 1 separator: "_" - footnotes - meta - def_list - pymdownx.arithmatex - pymdownx.betterem: smart_enable: all - pymdownx.caret - pymdownx.critic - pymdownx.details - pymdownx.emoji: emoji_generator: !!python/name:pymdownx.emoji.to_png #emoji_generator: !!python/name:pymdownx.emoji.to_svg #emoji_generator: !!python/name:pymdownx.emoji.to_png_sprite #emoji_generator: !!python/name:pymdownx.emoji.to_svg_sprite #emoji_generator: !!python/name:pymdownx.emoji.to_awesome #emoji_generator: !!python/name:pymdownx.emoji.to_alt - pymdownx.inlinehilite - pymdownx.magiclink - pymdownx.mark - pymdownx.smartsymbols - pymdownx.superfences - pymdownx.tasklist - pymdownx.tilde extra_javascript: - 'js/extra.js' - 'js/baidu-tongji.js' - 'https://www.googletagmanager.com/gtag/js?id=UA-155084439-1' - 'https://www.googletagmanager.com/gtag/js?id=UA-155132293-1' - 'js/google.js' - 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-MML-AM_CHTML' extra: social: - icon: fontawesome/brands/github link: https://github.com/burningmyself - icon: fontawesome/brands/google link: https://gitee.com/burningmyself

    For online documentation and support please refer to nginx.org.
    Commercial support is available at nginx.com.