[
{
"path": ".github/ISSUE_TEMPLATE/bug-report.yaml",
"content": "name: Bug Report 问题提交\ndescription: Report a bug encountered while using kubeasz 项目使用问题提交\nlabels: kind/bug\nbody:\n - type: textarea\n id: problem\n attributes:\n label: What happened? 发生了什么问题？\n description: |\n Please provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner.\n 操作命令，输出日志等，请尽可能提供详细信息，否则可能导致您的问题无法及时得到跟踪和解决。\n validations:\n required: true\n\n - type: textarea\n id: expected\n attributes:\n label: What did you expect to happen? 期望的结果是什么？\n validations:\n required: true\n\n - type: textarea\n id: repro\n attributes:\n label: How can we reproduce it (as minimally and precisely as possible)? 尽可能最小化、精确地描述如何复现问题\n validations:\n required: true\n\n - type: textarea\n id: additional\n attributes:\n label: Anything else we need to know? 其他需要说明的情况\n\n - type: textarea\n id: kubeVersion\n attributes:\n label: Kubernetes version k8s 版本\n value: |\n \n\n \n validations:\n required: true\n\n - type: textarea\n id: kubeaszVersion\n attributes:\n label: Kubeasz version \n value: |\n \n\n \n validations:\n required: true\n\n - type: textarea\n id: osVersion\n attributes:\n label: OS version 操作系统版本\n value: |\n \n\n ```console\n # On Linux:\n $ cat /etc/os-release\n # paste output here\n $ uname -a\n # paste output here\n ```\n\n \n validations:\n required: true\n\n - type: textarea\n id: plugins\n attributes:\n label: Related plugins (CNI, CSI, ...) and versions (if applicable) 其他网络插件等需要说明的情况\n value: |\n \n\n \n"
},
{
"path": ".github/ISSUE_TEMPLATE/enhancement.yaml",
"content": "name: Enhancement Tracking Issue\ndescription: Provide supporting details for a feature in development\nlabels: kind/feature\nbody:\n - type: textarea\n id: feature\n attributes:\n label: What would you like to be added?\n description: |\n Feature requests are unlikely to make progress as issues. \n A proposal that works through the design along with the implications of the change can be opened as a KEP.\n validations:\n required: true\n\n - type: textarea\n id: rationale\n attributes:\n label: Why is this needed?\n validations:\n required: true\n"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"content": "\n\n#### What type of PR is this?\n\n\n\n#### What this PR does / why we need it:\n\n#### Which issue(s) this PR fixes:\n\nFixes #\n\n#### Special notes for your reviewer:\n\n#### Does this PR introduce a user-facing change?\n\n```release-note\n\n```\n\n#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:\n\n\n```docs\n\n```\n"
},
{
"path": ".github/workflows/mirror.yml",
"content": "name: Mirroring\n\non:\n push:\n #branches:\n # - 'master'\n tags:\n - '*.*.*'\n\njobs:\n to_gitee:\n runs-on: ubuntu-latest\n steps: # <-- must use actions/checkout before mirroring!\n - uses: actions/checkout@v2\n with:\n fetch-depth: 0\n - uses: pixta-dev/repository-mirroring-action@v1\n with:\n target_repo_url:\n git@gitee.com:easzlab/kubeasz.git\n ssh_private_key:\n ${{ secrets.SYNCGITEE }} # 密钥 (secret)\n"
},
{
"path": ".github/workflows/stale.yml",
"content": "name: Close inactive issues\non:\n schedule:\n - cron: \"1 21 * * *\"\n\njobs:\n close-issues:\n runs-on: ubuntu-latest\n permissions:\n issues: write\n pull-requests: write\n steps:\n - uses: actions/stale@v5\n with:\n operations-per-run: 50\n days-before-issue-stale: 30\n days-before-issue-close: 7\n stale-issue-label: \"stale\"\n stale-issue-message: \"This issue is stale because it has been open for 30 days with no activity.\"\n close-issue-message: \"This issue was closed because it has been inactive for 14 days since being marked as stale.\"\n days-before-pr-stale: -1\n days-before-pr-close: -1\n repo-token: ${{ secrets.GITHUB_TOKEN }}\n"
},
{
"path": ".gitignore",
"content": "# download directory\ndown/*\n\n# binaries directory\nbin/*\n\n# k8s storage manifests \nmanifests/storage/*\n!manifests/storage/test.yaml\n!manifests/storage/local-storage/\n\n# role based variable settings, exclude roles/os-harden/vars/\n#/roles/*/vars/*\n#!/roles/os-harden/vars/\n\n# cluster instances\nclusters/\n\n#\n*.crt\n*.key\n*.pem\n"
},
{
"path": "README.md",
"content": "\n \n \n Kubernetes \n 1.23 \n 1.24-1.28 \n 1.29 \n 1.30 \n 1.31 \n 1.32 \n 1.33 \n 1.34 \n \n \n \n \n kubeasz \n 3.2.0 \n 3.6.2 \n 3.6.3 \n 3.6.4 \n 3.6.5 \n 3.6.6 \n 3.6.7 \n 3.6.8 \n \n \n
\n\n## 支持系统\n\n- **Alibaba Linux** 2.1903, 3.2104\n- **Alma Linux** 8, 9\n- **Anolis OS** 8.x RHCK, 8.x ANCK\n- **CentOS/RHEL** 7, 8, 9\n- **Debian** 10, 11([notes](docs/setup/multi_os.md#Debian))\n- **Fedora** 34, 35, 36, 37\n- **Kylin Linux Advanced Server V10** 麒麟V10 Tercel, Lance, Halberd\n- **openEuler** 22.03 LTS, 24.03 LTS([notes](docs/setup/multi_os.md#openEuler))\n- **openSUSE** Leap 15.x([notes](docs/setup/multi_os.md#openSUSE))\n- **Rocky Linux** 8, 9\n- **Ubuntu** 16.04, 18.04, 20.04, 22.04, 24.04\n\n能够支持大部分使用systemd的linux发行版，如果安装有问题先请查看[文档](docs/setup/multi_os.md)；如果某个能够支持安装的系统没有在列表中，欢迎提PR 告知。\n\n## 快速指南\n\n单机快速体验k8s集群的测试环境--[AllinOne部署](docs/setup/quickStart.md)\n\n## 安装指南\n\n\n\n## 使用指南\n\n\n\n## 沟通交流\n\n- 微信：k8s&kubeasz实践, 搜索微信号`badtobone`, 请按格式备注（${城市}-${github用户名}）, 验证后加入群聊。\n- 推荐阅读\n - [kubernetes架构师课程](https://www.toutiao.com/c/user/token/MS4wLjABAAAA0YFomuMNm87NNysXeUsQdI0Tt3gOgz8WG_0B3MzxsmI/?tab=article)\n - [kubernetes-the-hard-way](https://github.com/kelseyhightower/kubernetes-the-hard-way)\n - [feisky-Kubernetes 指南](https://github.com/feiskyer/kubernetes-handbook/blob/master/SUMMARY.md)\n - [opsnull 安装教程](https://github.com/opsnull/follow-me-install-kubernetes-cluster)\n\n## 贡献&致谢\n\n欢迎提[Issues](https://github.com/easzlab/kubeasz/issues)和[PRs](docs/mixes/HowToContribute.md)参与维护项目！感谢您的关注与支持！\n- [如何 PR](docs/mixes/HowToContribute.md)\n- [如何捐赠](docs/mixes/donate.md)\n\nCopyright 2017 gjmzj (jmgaozz@163.com) Apache License 2.0, 详情见 [LICENSE](docs/mixes/LICENSE) 文件。\n"
},
{
"path": "ansible.cfg",
"content": "# Example config file for ansible -- https://ansible.com/\n# =======================================================\n\n# Nearly all parameters can be overridden in ansible-playbook\n# or with command line flags. Ansible will read ANSIBLE_CONFIG,\n# ansible.cfg in the current working directory, .ansible.cfg in\n# the home directory, or /etc/ansible/ansible.cfg, whichever it\n# finds first\n\n# For a full list of available options, run ansible-config list or see the\n# documentation: https://docs.ansible.com/ansible/latest/reference_appendices/config.html.\n\n[defaults]\n#inventory = /etc/ansible/hosts\n#library = ~/.ansible/plugins/modules:/usr/share/ansible/plugins/modules\n#module_utils = ~/.ansible/plugins/module_utils:/usr/share/ansible/plugins/module_utils\n#remote_tmp = ~/.ansible/tmp\n#local_tmp = ~/.ansible/tmp\n#forks = 5\n#poll_interval = 0.001\n#ask_pass = False\n#transport = smart\n\n# Plays will gather facts by default, which contain information about\n# the remote system.\n#\n# smart - gather by default, but don't regather if already gathered\n# implicit - gather by default, turn off with gather_facts: False\n# explicit - do not gather by default, must say gather_facts: True\ngathering = smart\n\n# This only affects the gathering done by a play's gather_facts directive,\n# by default gathering retrieves all facts subsets\n# all - gather all subsets\n# network - gather min and network facts\n# hardware - gather hardware facts (longest facts to retrieve)\n# virtual - gather min and virtual facts\n# facter - import facts from facter\n# ohai - import facts from ohai\n# You can combine them using comma (ex: network,virtual)\n# You can negate them using ! (ex: !hardware,!facter,!ohai)\n# A minimal set of facts is always gathered.\n#\n#gather_subset = all\n\n# some hardware related facts are collected\n# with a maximum timeout of 10 seconds. This\n# option lets you increase or decrease that\n# timeout to something more suitable for the\n# environment.\n#\ngather_timeout = 7\n\n# Ansible facts are available inside the ansible_facts.* dictionary\n# namespace. This setting maintains the behaviour which was the default prior\n# to 2.5, duplicating these variables into the main namespace, each with a\n# prefix of 'ansible_'.\n# This variable is set to True by default for backwards compatibility. It\n# will be changed to a default of 'False' in a future release.\n#\n#inject_facts_as_vars = True\n\n# Paths to search for collections, colon separated\n# collections_paths = ~/.ansible/collections:/usr/share/ansible/collections\n\n# Paths to search for roles, colon separated\nroles_path = /etc/kubeasz/roles \n\n# Host key checking is enabled by default\nhost_key_checking = False\n\n# You can only have one 'stdout' callback type enabled at a time. The default\n# is 'default'. The 'yaml' or 'debug' stdout callback plugins are easier to read.\n#\n#stdout_callback = default\n#stdout_callback = yaml\n#stdout_callback = debug\n\n\n# Ansible ships with some plugins that require whitelisting,\n# this is done to avoid running all of a type by default.\n# These setting lists those that you want enabled for your system.\n# Custom plugins should not need this unless plugin author disables them\n# by default.\n#\n# Enable callback plugins, they can output to stdout but cannot be 'stdout' type.\n#callback_whitelist = timer, mail\n\n# Determine whether includes in tasks and handlers are \"static\" by\n# default. As of 2.0, includes are dynamic by default. Setting these\n# values to True will make includes behave more like they did in the\n# 1.x versions.\n#\n#task_includes_static = False\n#handler_includes_static = False\n\n# Controls if a missing handler for a notification event is an error or a warning\n#error_on_missing_handler = True\n\n# Default timeout for connection plugins\n#timeout = 10\n\n# Default user to use for playbooks if user is not specified\n# Uses the connection plugin's default, normally the user currently executing Ansible,\n# unless a different user is specified here.\n#\n#remote_user = root\n\n# Logging is off by default unless this path is defined.\n#log_path = /var/log/ansible.log\n\n# Default module to use when running ad-hoc commands\n#module_name = command\n\n# Use this shell for commands executed under sudo.\n# you may need to change this to /bin/bash in rare instances\n# if sudo is constrained.\n#\n#executable = /bin/sh\n\n# By default, variables from roles will be visible in the global variable\n# scope. To prevent this, set the following option to True, and only\n# tasks and handlers within the role will see the variables there\n#\nprivate_role_vars = True\n\n# List any Jinja2 extensions to enable here.\n#jinja2_extensions = jinja2.ext.do,jinja2.ext.i18n\n\n# If set, always use this private key file for authentication, same as\n# if passing --private-key to ansible or ansible-playbook\n#\n#private_key_file = /path/to/file\n\n# If set, configures the path to the Vault password file as an alternative to\n# specifying --vault-password-file on the command line. This can also be\n# an executable script that returns the vault password to stdout.\n#\n#vault_password_file = /path/to/vault_password_file\n\n# Format of string {{ ansible_managed }} available within Jinja2\n# templates indicates to users editing templates files will be replaced.\n# replacing {file}, {host} and {uid} and strftime codes with proper values.\n#\n#ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}\n\n# {file}, {host}, {uid}, and the timestamp can all interfere with idempotence\n# in some situations so the default is a static string:\n#\n#ansible_managed = Ansible managed\n\n# By default, ansible-playbook will display \"Skipping [host]\" if it determines a task\n# should not be run on a host. Set this to \"False\" if you don't want to see these \"Skipping\"\n# messages. NOTE: the task header will still be shown regardless of whether or not the\n# task is skipped.\n#\ndisplay_skipped_hosts = False\n\n# By default, if a task in a playbook does not include a name: field then\n# ansible-playbook will construct a header that includes the task's action but\n# not the task's args. This is a security feature because ansible cannot know\n# if the *module* considers an argument to be no_log at the time that the\n# header is printed. If your environment doesn't have a problem securing\n# stdout from ansible-playbook (or you have manually specified no_log in your\n# playbook on all of the tasks where you have secret information) then you can\n# safely set this to True to get more informative messages.\n#\ndisplay_args_to_stdout = False \n\n# Ansible will raise errors when attempting to dereference\n# Jinja2 variables that are not set in templates or action lines. Uncomment this line\n# to change this behavior.\n#\nerror_on_undefined_vars = True\n\n# Ansible may display warnings based on the configuration of the\n# system running ansible itself. This may include warnings about 3rd party packages or\n# other conditions that should be resolved if possible.\n# To disable these warnings, set the following value to False:\n#\nsystem_warnings = False\n\n# Ansible may display deprecation warnings for language\n# features that should no longer be used and will be removed in future versions.\n# To disable these warnings, set the following value to False:\n#\ndeprecation_warnings = False\n\n# Ansible can optionally warn when usage of the shell and\n# command module appear to be simplified by using a default Ansible module\n# instead. These warnings can be silenced by adjusting the following\n# setting or adding warn=yes or warn=no to the end of the command line\n# parameter string. This will for example suggest using the git module\n# instead of shelling out to the git command.\n#\n#command_warnings = False\n\n\n# set plugin path directories here, separate with colons\n#action_plugins = /usr/share/ansible/plugins/action\n#become_plugins = /usr/share/ansible/plugins/become\n#cache_plugins = /usr/share/ansible/plugins/cache\n#callback_plugins = /usr/share/ansible/plugins/callback\n#connection_plugins = /usr/share/ansible/plugins/connection\n#lookup_plugins = /usr/share/ansible/plugins/lookup\n#inventory_plugins = /usr/share/ansible/plugins/inventory\n#vars_plugins = /usr/share/ansible/plugins/vars\n#filter_plugins = /usr/share/ansible/plugins/filter\n#test_plugins = /usr/share/ansible/plugins/test\n#terminal_plugins = /usr/share/ansible/plugins/terminal\n#strategy_plugins = /usr/share/ansible/plugins/strategy\n\n\n# Ansible will use the 'linear' strategy but you may want to try another one.\n#strategy = linear\n\n# By default, callbacks are not loaded for /bin/ansible. Enable this if you\n# want, for example, a notification or logging callback to also apply to\n# /bin/ansible runs\n#\n#bin_ansible_callbacks = False\n\n\n# Don't like cows? that's unfortunate.\n# set to 1 if you don't want cowsay support or export ANSIBLE_NOCOWS=1\n#nocows = 1\n\n# Set which cowsay stencil you'd like to use by default. When set to 'random',\n# a random stencil will be selected for each task. The selection will be filtered\n# against the `cow_whitelist` option below.\n#\n#cow_selection = default\n#cow_selection = random\n\n# When using the 'random' option for cowsay, stencils will be restricted to this list.\n# it should be formatted as a comma-separated list with no spaces between names.\n# NOTE: line continuations here are for formatting purposes only, as the INI parser\n# in python does not support them.\n#\n#cow_whitelist=bud-frogs,bunny,cheese,daemon,default,dragon,elephant-in-snake,elephant,eyes,\\\n# hellokitty,kitty,luke-koala,meow,milk,moofasa,moose,ren,sheep,small,stegosaurus,\\\n# stimpy,supermilker,three-eyes,turkey,turtle,tux,udder,vader-koala,vader,www\n\n# Don't like colors either?\n# set to 1 if you don't want colors, or export ANSIBLE_NOCOLOR=1\n#\n#nocolor = 1\n\n# If set to a persistent type (not 'memory', for example 'redis') fact values\n# from previous runs in Ansible will be stored. This may be useful when\n# wanting to use, for example, IP information from one group of servers\n# without having to talk to them in the same playbook run to get their\n# current IP information.\n#\n#fact_caching = memory\n\n# This option tells Ansible where to cache facts. The value is plugin dependent.\n# For the jsonfile plugin, it should be a path to a local directory.\n# For the redis plugin, the value is a host:port:database triplet: fact_caching_connection = localhost:6379:0\n#\n#fact_caching_connection=/tmp\n\n# retry files\n# When a playbook fails a .retry file can be created that will be placed in ~/\n# You can enable this feature by setting retry_files_enabled to True\n# and you can change the location of the files by setting retry_files_save_path\n#\nretry_files_enabled = False\n#retry_files_save_path = ~/.ansible-retry\n\n# prevents logging of task data, off by default\n#no_log = False\n\n# prevents logging of tasks, but only on the targets, data is still logged on the master/controller\n#no_target_syslog = False\n\n# Controls whether Ansible will raise an error or warning if a task has no\n# choice but to create world readable temporary files to execute a module on\n# the remote machine. This option is False by default for security. Users may\n# turn this on to have behaviour more like Ansible prior to 2.1.x. See\n# https://docs.ansible.com/ansible/latest/user_guide/become.html#becoming-an-unprivileged-user\n# for more secure ways to fix this than enabling this option.\n#\n#allow_world_readable_tmpfiles = False\n\n# Controls what compression method is used for new-style ansible modules when\n# they are sent to the remote system. The compression types depend on having\n# support compiled into both the controller's python and the client's python.\n# The names should match with the python Zipfile compression types:\n# * ZIP_STORED (no compression. available everywhere)\n# * ZIP_DEFLATED (uses zlib, the default)\n# These values may be set per host via the ansible_module_compression inventory variable.\n#\n#module_compression = 'ZIP_DEFLATED'\n\n# This controls the cutoff point (in bytes) on --diff for files\n# set to 0 for unlimited (RAM may suffer!).\n#\n#max_diff_size = 104448\n\n# Controls showing custom stats at the end, off by default\n#show_custom_stats = False\n\n# Controls which files to ignore when using a directory as inventory with\n# possibly multiple sources (both static and dynamic)\n#\n#inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo\n\n# This family of modules use an alternative execution path optimized for network appliances\n# only update this setting if you know how this works, otherwise it can break module execution\n#\n#network_group_modules=eos, nxos, ios, iosxr, junos, vyos\n\n# When enabled, this option allows lookups (via variables like {{lookup('foo')}} or when used as\n# a loop with `with_foo`) to return data that is not marked \"unsafe\". This means the data may contain\n# jinja2 templating language which will be run through the templating engine.\n# ENABLING THIS COULD BE A SECURITY RISK\n#\n#allow_unsafe_lookups = False\n\n# set default errors for all plays\n#any_errors_fatal = False\n\n\n[inventory]\n# List of enabled inventory plugins and the order in which they are used.\n#enable_plugins = host_list, script, auto, yaml, ini, toml\n\n# Ignore these extensions when parsing a directory as inventory source\n#ignore_extensions = .pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, ~, .orig, .ini, .cfg, .retry\n\n# ignore files matching these patterns when parsing a directory as inventory source\n#ignore_patterns=\n\n# If 'True' unparsed inventory sources become fatal errors, otherwise they are warnings.\n#unparsed_is_failed = False\n\n\n[privilege_escalation]\n#become = False\n#become_method = sudo\n#become_ask_pass = False\n\n\n## Connection Plugins ##\n\n# Settings for each connection plugin go under a section titled '[[plugin_name]_connection]'\n# To view available connection plugins, run ansible-doc -t connection -l\n# To view available options for a connection plugin, run ansible-doc -t connection [plugin_name]\n# https://docs.ansible.com/ansible/latest/plugins/connection.html\n\n[paramiko_connection]\n# uncomment this line to cause the paramiko connection plugin to not record new host\n# keys encountered. Increases performance on new host additions. Setting works independently of the\n# host key checking setting above.\n#record_host_keys=False\n\n# by default, Ansible requests a pseudo-terminal for commands executed under sudo. Uncomment this\n# line to disable this behaviour.\n#pty = False\n\n# paramiko will default to looking for SSH keys initially when trying to\n# authenticate to remote devices. This is a problem for some network devices\n# that close the connection after a key failure. Uncomment this line to\n# disable the Paramiko look for keys function\n#look_for_keys = False\n\n# When using persistent connections with Paramiko, the connection runs in a\n# background process. If the host doesn't already have a valid SSH key, by\n# default Ansible will prompt to add the host key. This will cause connections\n# running in background processes to fail. Uncomment this line to have\n# Paramiko automatically add host keys.\n#host_key_auto_add = True\n\n\n[ssh_connection]\n# ssh arguments to use\n# Leaving off ControlPersist will result in poor performance, so use\n# paramiko on older platforms rather than removing it, -C controls compression use\n#ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s\n\n# The base directory for the ControlPath sockets.\n# This is the \"%(directory)s\" in the control_path option\n#\n# Example:\n# control_path_dir = /tmp/.ansible/cp\ncontrol_path_dir = /tmp\n\n# The path to use for the ControlPath sockets. This defaults to a hashed string of the hostname,\n# port and username (empty string in the config). The hash mitigates a common problem users\n# found with long hostnames and the conventional %(directory)s/ansible-ssh-%%h-%%p-%%r format.\n# In those cases, a \"too long for Unix domain socket\" ssh error would occur.\n#\n# Example:\n# control_path = %(directory)s/%%C\ncontrol_path = /tmp/ansible-ssh-%%h-%%p-%%r\n\n# Enabling pipelining reduces the number of SSH operations required to\n# execute a module on the remote server. This can result in a significant\n# performance improvement when enabled, however when using \"sudo:\" you must\n# first disable 'requiretty' in /etc/sudoers\n#\n# By default, this option is disabled to preserve compatibility with\n# sudoers configurations that have requiretty (the default on many distros).\n#\npipelining = True \n\n# Control the mechanism for transferring files (old)\n# * smart = try sftp and then try scp [default]\n# * True = use scp only\n# * False = use sftp only\n#scp_if_ssh = smart\n\n# Control the mechanism for transferring files (new)\n# If set, this will override the scp_if_ssh option\n# * sftp = use sftp to transfer files\n# * scp = use scp to transfer files\n# * piped = use 'dd' over SSH to transfer files\n# * smart = try sftp, scp, and piped, in that order [default]\n#transfer_method = smart\n\n# If False, sftp will not use batch mode to transfer files. This may cause some\n# types of file transfer failures impossible to catch however, and should\n# only be disabled if your sftp version has problems with batch mode\nsftp_batch_mode = True\n\n# The -tt argument is passed to ssh when pipelining is not enabled because sudo\n# requires a tty by default.\n#usetty = True\n\n# Number of times to retry an SSH connection to a host, in case of UNREACHABLE.\n# For each retry attempt, there is an exponential backoff,\n# so after the first attempt there is 1s wait, then 2s, 4s etc. up to 30s (max).\n#retries = 3\n\n\n[persistent_connection]\n# Configures the persistent connection timeout value in seconds. This value is\n# how long the persistent connection will remain idle before it is destroyed.\n# If the connection doesn't receive a request before the timeout value\n# expires, the connection is shutdown. The default value is 30 seconds.\n#connect_timeout = 30\n\n# The command timeout value defines the amount of time to wait for a command\n# or RPC call before timing out. The value for the command timeout must\n# be less than the value of the persistent connection idle timeout (connect_timeout)\n# The default value is 30 second.\n#command_timeout = 30\n\n\n## Become Plugins ##\n\n# Settings for become plugins go under a section named '[[plugin_name]_become_plugin]'\n# To view available become plugins, run ansible-doc -t become -l\n# To view available options for a specific plugin, run ansible-doc -t become [plugin_name]\n# https://docs.ansible.com/ansible/latest/plugins/become.html\n\n[sudo_become_plugin]\n#flags = -H -S -n\n#user = root\n\n\n[selinux]\n# file systems that require special treatment when dealing with security context\n# the default behaviour that copies the existing context or uses the user default\n# needs to be changed to use the file system dependent context.\n#special_context_filesystems=fuse,nfs,vboxsf,ramfs,9p,vfat\n\n# Set this to True to allow libvirt_lxc connections to work without SELinux.\n#libvirt_lxc_noseclabel = False\n\n\n[colors]\n#highlight = white\n#verbose = blue\n#warn = bright purple\n#error = red\n#debug = dark gray\n#deprecate = purple\n#skip = cyan\n#unreachable = red\n#ok = green\n#changed = yellow\n#diff_add = green\n#diff_remove = red\n#diff_lines = cyan\n\n\n[diff]\n# Always print diff when running ( same as always running with -D/--diff )\n#always = False\n\n# Set how many context lines to show in diff\n#context = 3\n\n[galaxy]\n# Controls whether the display wheel is shown or not\n#display_progress=\n\n# Validate TLS certificates for Galaxy server\n#ignore_certs = False\n\n# Role or collection skeleton directory to use as a template for\n# the init action in ansible-galaxy command\n#role_skeleton=\n\n# Patterns of files to ignore inside a Galaxy role or collection\n# skeleton directory\n#role_skeleton_ignore=\"^.git$\", \"^.*/.git_keep$\"\n\n# Galaxy Server URL\n#server=https://galaxy.ansible.com\n\n# A list of Galaxy servers to use when installing a collection.\n#server_list=automation_hub, release_galaxy\n\n# Server specific details which are mentioned in server_list\n#[galaxy_server.automation_hub]\n#url=https://cloud.redhat.com/api/automation-hub/\n#auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token\n#token=my_ah_token\n#\n#[galaxy_server.release_galaxy]\n#url=https://galaxy.ansible.com/\n#token=my_token\n"
},
{
"path": "docs/blog/seperated_containerd_services_for_docker_and_k8s.md",
"content": "# 为docker和k8s创建独立的containerd进程\n\n## 背景\n\n公司有一台带GPU显卡的服务器，为了通过vllm推理镜像运行像“qwen3-32b”这样的大语言模型，已经配置了带`nvidia-container-runtime`运行时的docker服务；现在需要该服务器加入到k8s集群；因为docker和k8s都需要containerd服务；为了避免冲突，需要两个完全独立的 containerd 实例。\n\n## 方案\n\n必须隔离的 6 个关键点\n\n\n| 项 | Docker containerd | K8s containerd |\n| :--- | :--- | :--- |\n| binary | Docker 自带 | 系统安装|\n| systemd unit | docker.service\t| containerd-k8s.service |\n| socket | /var/run/docker/containerd.sock | /run/containerd-k8s/containerd.sock |\n| root | /var/lib/docker/containerd | /var/lib/containerd-k8s |\n| state | /run/docker/containerd | /run/containerd-k8s |\n| config | Docker 管理 | /etc/containerd-k8s/config.toml |\n\n## 基于kubeasz 安装步骤\n\n`kubeasz` 3.6.9 版本以上支持快速配置自定义的`containerd`服务; 在正常安装之前，首先修改 example/config.yml 配置文件参考如下：\n\n```\n# [containerd] root 存储目录，默认：/var/lib/containerd\nCONTAINERD_ROOT_DIR: \"/var/lib/k8scontainerd\"\n\n# [containerd] state 存储目录，默认：/run/containerd\nCONTAINERD_STATE_DIR: \"/run/k8scontainerd\"\n\n# [containerd] config 目录，默认：/etc/containerd\nCONTAINERD_CONFIG_DIR: \"/etc/k8scontainerd\"\n\n# [containerd] systemd service 名称，默认：containerd.service\nCONTAINERD_SERVICE_NAME: \"k8scontainerd.service\"\n```\n\n然后按照正常的安装流程即可。\n\n## 验证\n\n```\nps -ef | grep containerd\n\n# 可以看到两个不同的进程\n\n/opt/kube/bin/containerd-bin/containerd --log-level warn --config /etc/k8scontainerd/config.toml\n/usr/bin/containerd --config /var/run/docker/containerd/containerd.toml\n```\n\nThat's it. Have Fun!\n"
},
{
"path": "docs/deprecated/efk.md",
"content": "### 第一部分：EFK\n\n本文档已过期(deprecated)\n\n`EFK` 插件是`k8s`项目的一个日志解决方案，它包括三个组件：[Elasticsearch](), [Fluentd](), [Kibana]()；Elasticsearch 是日志存储和日志搜索引擎，Fluentd 负责把`k8s`集群的日志发送给 Elasticsearch, Kibana 则是可视化界面查看和检索存储在 ES 中的数据。\n- 建议在熟悉本文档内容后使用[Log-Pilot + ES + Kibana 日志方案](log-pilot.md)\n\n### 准备 \n\n参考官方[部署文档](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch)的基础上使用本项目`manifests/efk/`部署，以下为几点主要的修改：\n\n+ 修改 fluentd-es-configmap.yaml 中的部分 journald 日志源（增加集群组件服务日志搜集）\n+ 修改官方docker镜像，方便国内下载加速\n+ 修改 es-statefulset.yaml 支持日志存储持久化等\n+ 增加自动清理日志，见后文`第四部分`\n\n### 安装\n\n``` bash\n$ kubectl apply -f /etc/kubeasz/manifests/efk/\n$ kubectl apply -f /etc/kubeasz/manifests/efk/es-without-pv/\n```\n\n### 验证\n\n``` bash\nkubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'\nelasticsearch-logging-0 1/1 Running 0 19h\nelasticsearch-logging-1 1/1 Running 0 19h\nfluentd-es-v2.0.2-6c95c 1/1 Running 0 17h\nfluentd-es-v2.0.2-f2xh8 1/1 Running 0 8h\nfluentd-es-v2.0.2-pv5q5 1/1 Running 0 8h\nkibana-logging-d5cffd7c6-9lz2p 1/1 Running 0 1m\n```\nkibana Pod 第一次启动时会用较长时间(10-20分钟)来优化和 Cache 状态页面，可以查看 Pod 的日志观察进度，如下等待 `Ready` 状态\n\n``` bash\n$ kubectl logs -n kube-system kibana-logging-d5cffd7c6-9lz2p -f\n...\n{\"type\":\"log\",\"@timestamp\":\"2018-03-13T07:33:00Z\",\"tags\":[\"listening\",\"info\"],\"pid\":1,\"message\":\"Server running at http://0:5601\"}\n{\"type\":\"log\",\"@timestamp\":\"2018-03-13T07:33:00Z\",\"tags\":[\"status\",\"ui settings\",\"info\"],\"pid\":1,\"state\":\"green\",\"message\":\"Status changed from uninitialized to green - Ready\",\"prevState\":\"uninitialized\",\"prevMsg\":\"uninitialized\"}\n```\n\n### 访问 Kibana\n\n推荐使用`kube-apiserver`方式访问（可以使用证书和rbac等方式进行认证授权），获取访问 URL\n\n- 使用证书登录(生成kubecfg.p12，并将证书下载到本地安装)：\n```bash\ngrep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d > kubecfg.crt\ngrep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d > kubecfg.key\nopenssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name \"kubernetes-client\"\n```\n\n``` bash\n$ kubectl cluster-info | grep Kibana\nKibana is running at https://192.168.1.10:8443/api/v1/namespaces/kube-system/services/kibana-logging/proxy\n```\n浏览器访问 URL：`https://192.168.1.10:8443/api/v1/namespaces/kube-system/services/kibana-logging/proxy`，然后使用`basic-auth`或者`证书` 的方式认证后即可，关于认证可以参考[dashboard文档](dashboard.md)\n\n首次登录需要在`Management` - `Index Patterns` 创建 `index pattern`，可以使用默认的 logstash-* pattern，点击下一步；在 Time Filter field name 下拉框选择 @timestamp; 点击创建Index Pattern后，稍等几分钟就可以在 Discover 菜单看到 ElasticSearch logging 中汇聚的日志；\n\n### 第二部分：日志持久化之静态PV\n日志数据是存放于 `Elasticsearch POD`中，但是默认情况下它使用的是`emptyDir`存储类型，所以当 `POD`被删除或重新调度时，日志数据也就丢失了。以下讲解使用`NFS` 服务器手动（静态）创建`PV` 持久化保存日志数据的例子。\n\n#### 配置 NFS\n\n+ 准备一个nfs服务器，如果没有可以参考[nfs-server](nfs-server.md)创建。 \n+ 配置nfs服务器的共享目录，即修改`/etc/exports`（根据实际网段替换`192.168.1.*`），修改后重启`systemctl restart nfs-server`。\n\n``` bash\n/share 192.168.1.*(rw,sync,insecure,no_subtree_check,no_root_squash)\n/share/es0 192.168.1.*(rw,sync,insecure,no_subtree_check,no_root_squash)\n/share/es1 192.168.1.*(rw,sync,insecure,no_subtree_check,no_root_squash)\n/share/es2 192.168.1.*(rw,sync,insecure,no_subtree_check,no_root_squash)\n```\n\n#### 使用静态 PV安装 EFK\n\n- 请按实际日志容量需求修改 `es-static-pv/es-statefulset.yaml` 文件中 volumeClaimTemplates 设置的 storage: 4Gi 大小\n- 请根据实际nfs服务器地址、共享目录、容量大小修改 `es-static-pv/es-pv*.yaml` 文件中对应的设置\n\n``` bash\n# 如果之前已经安装了默认的EFK，请用以下两个命令先删除它\n$ kubectl delete -f /etc/kubeasz/manifests/efk/\n$ kubectl delete -f /etc/kubeasz/manifests/efk/es-without-pv/\n\n# 安装静态PV 的 EFK\n$ kubectl apply -f /etc/kubeasz/manifests/efk/\n$ kubectl apply -f /etc/kubeasz/manifests/efk/es-static-pv/\n```\n+ 目录`es-static-pv` 下首先是利用 NFS服务预定义了三个 PV资源，然后在 `es-statefulset.yaml`定义中使用 `volumeClaimTemplates` 去匹配使用预定义的 PV资源；注意 PV参数：`accessModes` `storageClassName` `storage`容量大小必须两边匹配。 \n\n#### 验证安装\n\n+ 1.集群中查看 `pod` `pv` `pvc` 等资源\n\n``` bash\n$ kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'\nelasticsearch-logging-0 1/1 Running 0 10m\nelasticsearch-logging-1 1/1 Running 0 10m\nfluentd-es-v2.0.2-6c95c 1/1 Running 0 10m\nfluentd-es-v2.0.2-f2xh8 1/1 Running 0 10m\nfluentd-es-v2.0.2-pv5q5 1/1 Running 0 10m\nkibana-logging-d5cffd7c6-9lz2p 1/1 Running 0 10m\n\n$ kubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npv-es-0 4Gi RWX Recycle Bound kube-system/elasticsearch-logging-elasticsearch-logging-0 es-storage-class 1m\npv-es-1 4Gi RWX Recycle Bound kube-system/elasticsearch-logging-elasticsearch-logging-1 es-storage-class 1m\npv-es-2 4Gi RWX Recycle Available es-storage-class 1m\n\n$ kubectl get pvc --all-namespaces\nNAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\nkube-system elasticsearch-logging-elasticsearch-logging-0 Bound pv-es-0 4Gi RWX es-storage-class 2m\nkube-system elasticsearch-logging-elasticsearch-logging-1 Bound pv-es-1 4Gi RWX es-storage-class 1m\n```\n\n+ 2.网页访问 `kibana`查看具体的日志，如上须等待（约15分钟） `kibana Pod`优化和 Cache 状态页面，达到 `Ready` 状态。\n\n+ 3.登录 NFS Server 查看对应目录和内部数据\n\n``` bash\n$ ls /share\nes0 es1 es2\n```\n\n### 第三部分：日志持久化之动态PV\n`PV` 作为集群的存储资源，`StatefulSet` 依靠它实现 POD的状态数据持久化，但是当 `StatefulSet`动态伸缩时，它的 `PVC`请求也会变化，如果每次都需要管理员手动去创建对应的 `PV`资源，那就很不方便；因此 K8S还提供了 `provisioner`来动态创建 `PV`，不仅节省了管理员的时间，还可以根据不同的 `StorageClasses`封装不同类型的存储供 PVC 选用。\n\n+ 此功能需要 `API-SERVER` 参数 `--admission-control`字符串设置中包含 `DefaultStorageClass`，本项目中已经开启。\n+ `provisioner`指定 Volume 插件的类型，包括内置插件（如 kubernetes.io/glusterfs）和外部插件（如 external-storage 提供的 ceph.com/cephfs，nfs-client等），以下讲解使用 `nfs-client-provisioner`来动态创建 `PV`来持久化保存 `EFK`的日志数据。\n\n#### 配置 NFS（同上）\n\n确保 `/etc/exports` 配置如下共享目录，并确保 `/share`目录可读可写权限，否则可能因为权限问题无法动态生成 PV的对应目录。（根据实际情况替换IP段`192.168.1.*`）\n``` bash\n/share 192.168.1.*(rw,sync,insecure,no_subtree_check,no_root_squash)\n```\n\n#### 使用动态 PV安装 EFK\n\n- 首先根据[集群存储](../setup/08-cluster-storage.md)创建nfs-client-provisioner\n- 然后按实际需求修改 `es-dynamic-pv/es-statefulset.yaml` 文件中 volumeClaimTemplates 设置的 storage: 4Gi 大小 \n\n``` bash\n# 如果之前已经安装了默认的EFK或者静态PV EFK，请用以下命令先删除它\n$ kubectl delete -f /etc/kubeasz/manifests/efk/\n$ kubectl delete -f /etc/kubeasz/manifests/efk/es-without-pv/\n$ kubectl delete -f /etc/kubeasz/manifests/efk/es-static-pv/\n\n# 安装动态PV 的 EFK\n$ kubectl apply -f /etc/kubeasz/manifests/efk/\n$ kubectl apply -f /etc/kubeasz/manifests/efk/es-dynamic-pv/\n```\n+ 首先 `nfs-client-provisioner.yaml` 创建一个工作 POD，它监听集群的 PVC请求，并当 PVC请求来到时调用 `nfs-client` 去请求 `nfs-server`的存储资源，成功后即动态生成对应的 PV资源。\n+ `nfs-dynamic-storageclass.yaml` 定义 NFS存储类型的类型名 `nfs-dynamic-class`，然后在 `es-statefulset.yaml`中必须使用这个类型名才能动态请求到资源。\n\n#### 验证安装\n\n+ 1.集群中查看 `pod` `pv` `pvc` 等资源\n\n``` bash\n$ kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'\nelasticsearch-logging-0 1/1 Running 0 10m\nelasticsearch-logging-1 1/1 Running 0 10m\nfluentd-es-v2.0.2-6c95c 1/1 Running 0 10m\nfluentd-es-v2.0.2-f2xh8 1/1 Running 0 10m\nfluentd-es-v2.0.2-pv5q5 1/1 Running 0 10m\nkibana-logging-d5cffd7c6-9lz2p 1/1 Running 0 10m\n\n$ kubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npvc-50644f36-358b-11e8-9edd-525400cecc16 4Gi RWX Delete Bound kube-system/elasticsearch-logging-elasticsearch-logging-0 nfs-dynamic-class 10m\npvc-5b105ee6-358b-11e8-9edd-525400cecc16 4Gi RWX Delete Bound kube-system/elasticsearch-logging-elasticsearch-logging-1 nfs-dynamic-class 10m\n\n$ kubectl get pvc --all-namespaces\nNAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\nkube-system elasticsearch-logging-elasticsearch-logging-0 Bound pvc-50644f36-358b-11e8-9edd-525400cecc16 4Gi RWX nfs-dynamic-class 10m\nkube-system elasticsearch-logging-elasticsearch-logging-1 Bound pvc-5b105ee6-358b-11e8-9edd-525400cecc16 4Gi RWX nfs-dynamic-class 10m\n```\n\n+ 2.网页访问 `kibana`查看具体的日志，如上须等待（约15分钟） `kibana Pod`优化和 Cache 状态页面，达到 `Ready` 状态。\n\n+ 3.登录 NFS Server 查看对应目录和内部数据\n\n``` bash\n$ ls /share # 可以看到类似如下的目录生成\nkube-system-elasticsearch-logging-elasticsearch-logging-0-pvc-50644f36-358b-11e8-9edd-525400cecc16\nkube-system-elasticsearch-logging-elasticsearch-logging-1-pvc-5b105ee6-358b-11e8-9edd-525400cecc16\n```\n\n### 第四部分：日志自动清理\n\n我们知道日志都存储在elastic集群中，且日志每天被分割成一个index，例如：\n\n```\n/ # curl elasticsearch-logging:9200/_cat/indices?v\nhealth status index uuid pri rep docs.count docs.deleted store.size pri.store.size\ngreen open logstash-2019.04.29 ejMBlRcJQvqK76xIerenYg 5 1 69864 0 65.9mb 32.9mb\ngreen open logstash-2019.04.28 hacNCuQVTQCUL62Sl8avOA 5 1 17558 0 21.3mb 10.6mb\ngreen open .kibana_1 MVjF8lQeRDeKfoZcDhA93A 1 1 2 0 30.1kb 15kb\ngreen open logstash-2019.05.05 m2aD8X9RQ3u48DvVq18x_Q 5 1 31218 0 34.4mb 17.2mb\ngreen open logstash-2019.05.01 66OjwM5wT--DZaVfzUdXYQ 5 1 50610 0 54.6mb 27.1mb\ngreen open logstash-2019.04.30 L3AH165jT6izjHHa5L5g0w 5 1 56401 0 55.5mb 27.8mb\n...\n```\n\n因此 EFK 中的日志自动清理，只要定时去删除 es 中的 index 即可，如下命令\n\n```\n$ curl -X DELETE elasticsearch-logging:9200/logstash-xxxx.xx.xx\n```\n\n基于 alpine:3.8 创建镜像`es-index-rotator` [查看Dockerfile](../../dockerfiles/es-index-rotator/Dockerfile)，然后创建一个cronjob去完成清理任务\n\n```\n$ kubectl apply -f /etc/kubeasz/manifests/efk/es-index-rotator/\n```\n\n#### 验证日志清理\n\n- 查看 cronjob\n\n```\n$ kubectl get cronjob -n kube-system \nNAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE\nes-index-rotator 3 1 */1 * * False 0 19h 20h\n```\n- 查看日志清理情况\n\n```\n$ kubectl get pod -n kube-system |grep es-index-rotator\nes-index-rotator-1557507780-7xb89 0/1 Completed 0 19h\n\n# 查看日志，可以了解日志清理情况\n$ kubectl logs -n kube-system es-index-rotator-1557507780-7xb89 es-index-rotator \n```\nHAVE FUN!\n\n### 参考\n\n1. [EFK 配置](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch)\n1. [nfs-client-provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client)\n1. [persistent-volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims)\n1. [storage-classes](https://kubernetes.io/docs/concepts/storage/storage-classes/)\n\n\n"
},
{
"path": "docs/deprecated/gitlab/app.yaml.md",
"content": "## 3.3 K8S 应用部署模板 app.yaml\n\n以下示例配置仅做参考，描述一个简单 java spring boot项目的 k8s 部署文件模板；在实际部署前，CI/CD流程中会对变量做替换。详见 [gitlab-ci.yml文件](gitlab-ci.yml.md)。\n\n``` bash\ncat > .ci/app.yaml << EOF\n---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: PROJECT_NS\n---\napiVersion: v1\nkind: Secret\nmetadata:\n name: harborkey1\n namespace: PROJECT_NS\ndata:\n #待替换的变量DOCKER_KEY，参考 docs/guide/harbor.md#k8s%E4%B8%AD%E4%BD%BF%E7%94%A8harbor\n .dockerconfigjson: DOCKER_KEY\ntype: kubernetes.io/dockerconfigjson\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: APP_NAME\n namespace: PROJECT_NS\nspec:\n replicas: APP_REP\n template:\n metadata:\n labels:\n run: APP_NAME\n spec:\n containers:\n - name: APP_NAME\n image: ProjectImage\n env:\n # 设置java的时区\n - name: TZ\n value: \"Asia/Shanghai\"\n resources:\n limits:\n cpu: 500m\n memory: 1600Mi\n requests:\n cpu: 200m\n memory: 800Mi\n ports:\n - containerPort: 8080\n imagePullSecrets:\n - name: harborkey1\n\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n run: APP_NAME\n name: APP_NAME\n namespace: PROJECT_NS\nspec:\n ports:\n - port: 80\n protocol: TCP\n targetPort: 8080\n selector:\n run: APP_NAME\n sessionAffinity: None\n\n---\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: APP_NAME-ingress\n namespace: PROJECT_NS\nspec:\n rules:\n - host: AppDomain\n http:\n paths:\n - path: /AppPath\n backend:\n serviceName: APP_NAME\n servicePort: 80\nEOF\n```\n\n"
},
{
"path": "docs/deprecated/gitlab/config.sh.md",
"content": "## 3.2 环境配置替换 config.sh \n\n首先应用开发人员需要整理在不同环境（测试环境/生产环境）的配置参数，并在源代码中约定好替换的名称（如db_host, db_usr）；然后用户必须在项目gitlab web界面（“Settings”>\"CI/CD\">\"Variables\"）配置变量；最后根据gitlab-ci.yml文件定义CI/CD执行的需要，编写如下简单变量替换shell脚本；该shell脚本分别在测试环境打包阶段（beta-build）和生产环境打包阶段（prod-build）阶段运行。\n\n以下脚本仅作示例，实际应根据项目需要增加/修改需替换变量名称与对应源代码中的配置文件\n\n``` bash\ncat > .ci/config.sh << EOF\n#!/bin/bash\n\n#set -o verbose\n#set -o xtrace\n\nbeta_config() {\nsed -i \\\n -e \"s/db_host/$BETA_DB_HOST/g\" \\\n -e \"s/db_usr/$BETA_DB_USR/g\" \\\n -e \"s/db_pwd/$BETA_DB_PWD/g\" \\\n example-web/src/main/resources/config/datasource.properties # 项目源码的配置文件\nsed -i \\\n -e \"s/redis_host/$BETA_REDIS_HOST/g\" \\\n -e \"s/redis_port/$BETA_REDIS_PORT/g\" \\\n -e \"s/redis_pwd/$BETA_REDIS_PWD/g\" \\\n example-web/src/main/resources/config/redis.properties # 项目源码的配置文件\n}\n\nprod_config() {\nsed -i \\\n -e \"s/db_host/$PROD_DB_HOST/g\" \\\n -e \"s/db_usr/$PROD_DB_USR/g\" \\\n -e \"s/db_pwd/$PROD_DB_PWD/g\" \\\n example-web/src/main/resources/config/datasource.properties\nsed -i \\\n -e \"s/redis_host/$PROD_REDIS_HOST/g\" \\\n -e \"s/redis_port/$PROD_REDIS_PORT/g\" \\\n -e \"s/redis_pwd/$PROD_REDIS_PWD/g\" \\\n example-web/src/main/resources/config/redis.properties\n}\n\nif [[ \"$CI_JOB_STAGE\" == \"beta-build\" ]];then\n\tbeta_config\nelif [[ \"$CI_JOB_STAGE\" == \"prod-build\" ]];then\n\tprod_config\nelse\n\techo \"error: undefined CI_JOB_STAGE!\"\nfi\nEOF\n```\n\n"
},
{
"path": "docs/deprecated/gitlab/gitlab-ci.yml.md",
"content": "## 3.1 配置 gitlab-ci.yml\n\n示例应用搭建 CI/CD 流水线的背景需求\n\n- 应用测试环境部署在本地k8s平台，生产环境部署在阿里云上k8s平台\n- 应用的多个feature分支可以并行测试\n- 对于即将发布的release分支，本地提供封版测试环境，阿里云上提供UAT测试环境\n\n以下示例配置为个人经验总结，仅供参考，可以根据自己的理解和项目需要不断优化完善；总体来说 gitlab-ci.yml 配置很丰富，基本上能够满足各种个性化的CI/CD流程需要。\n\n``` bash\n$ cat > .ci/gitlab-ci.yml << EOF\nvariables: ### 定义全局变量 http://gitlab.test.com/help/ci/variables/README.md\n PROJECT_NS: '$CI_PROJECT_NAMESPACE-$CI_JOB_STAGE' # 定义项目命名空间，对应k8s的namespace\n APP_NAME: '$CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG' # 使用项目名和git提交信息作为应用名\n IMAGE_NAME: '$CI_PROJECT_NAMESPACE-$CI_PROJECT_NAME:$CI_PIPELINE_ID' # 定义镜像名称 \n\nstages: ### 定义ci各阶段\n - beta-build # beta环境编译打包\n - beta-deploy # beta环境部署\n - beta-feature-delete # beta环境feature分支手动删除\n - prod-build # prod环境编译打包\n - prod-uat-deploy # prod-uat环境部署\n - prod-deploy # prod环境部署\n - prod-rollback # prod回滚\n\njob_beta_build:\n stage: beta-build # beta环境编译打包\n tags:\n - build-shell # 定义带`build-shell`标签的runner可以运行该job\n only: # 定义只在如下分支或者tag运行该job \n - master\n - develop\n - /^feature.*$/\n - release\n #when: manual # 调试阶段可以先手动，后续可以注释掉以自动运行\n script: ### runner上运行的脚本\n - bash .ci/config.sh # 不同环境配置替换，后文详解 config.sh\n - mvn clean install -Dmaven.test.skip=true -U # mvn 编译，可以去runner 虚机上手动执行编译测试\n - mv example-web/target/*.jar dockerfiles/ # 把mvn生成的xxx.jar移动到dockerfiles目录下\n - export IMAGE=`echo $IMAGE_NAME | sed 's/\\//-/g'` # 转换镜像名，例：mygroup/java/example:172 >> mygroup-java-example:172\n - cd dockerfiles && docker build -t $BETA_HARBOR/example/$IMAGE . # 创建 docker 镜像\n - docker login -u $BETA_HARBOR_USR -p $BETA_HARBOR_PWD $BETA_HARBOR # 登录到内部镜像仓库 harbor，并推送\n - docker push $BETA_HARBOR/example/$IMAGE \n - docker logout $BETA_HARBOR\n\njob_push_beta: ### 推送到beta环境，可以推送不同分支 develop, feature-1, ...>\n stage: beta-deploy # 可以做到多分支同时测试，甚至最后的release分支也要在beta封版测试\n tags:\n - beta-shell # 定义带`beta-shell`标签的runner可以运行该job\n only:\n - master\n - develop\n - /^feature.*$/\n - release\n when: manual # 调试阶段可以先手动，后续可以注释掉以自动运行\n variables:\n BETA_EXP_Domain: '$CI_COMMIT_REF_SLUG.example.test.com' # job内部变量，指定该应用在beta环境的 ingress 域名\n script:\n - export IMAGE=`echo $IMAGE_NAME | sed 's/\\//-/g'` # 转换 $IMAGE_NAME 中可能的 / 字符\n - export PROJECT_NS=`echo $PROJECT_NS | sed 's/\\//-/g'` # 转换命名空间中可能有的 / 字符\n # 替换beta环境的参数配置\n - sed -i \"s/PROJECT_NS/$PROJECT_NS/g\" .ci/app.yaml ### app.yaml 即k8s的部署模板文件，详见后面 app.yaml.md 文档，注意这里的变量有的来自>\n - sed -i \"s/APP_NAME/$APP_NAME/g\" .ci/app.yaml # gitlab 系统变量, 有的是在项目 CI/CD 设置里面用户定义的变量\n - sed -i \"s/APP_REP/$BETA_APP_REP/g\" .ci/app.yaml\n - sed -i \"s/AppDomain/$BETA_EXP_Domain/g\" .ci/app.yaml\n - sed -i \"s/ProjectImage/$BETA_HARBOR\\/example\\/$IMAGE/g\" .ci/app.yaml\n - sed -i \"s/DOCKER_KEY/$BETA_KEY/g\" .ci/app.yaml # DOCKER_KEY 为k8s平台能从镜像仓库pull所需的认证信息，详见harbor文档\n #\n - mkdir -p /opt/kube/$PROJECT_NS/$APP_NAME # 在runner：beta-shell虚机本地创建应用配置目录，调试检查用\n - cp -f .ci/app.yaml /opt/kube/$PROJECT_NS/$APP_NAME\n - kubectl --kubeconfig=/etc/.beta/config apply -f .ci/app.yaml # 部署应用（runner虚机上预先配置了kubectl权限执行测试k8s平台）\n\njob_delete_beta: ### 多测试环境并行部署在beta k8s平台，feature分支测试完毕后删除代码分支，\n stage: beta-feature-delete # 同时需要删除该分支在k8s平台上的部署，可以由开发人员自行执行该job删除\n tags:\n - beta-shell\n only:\n - /^feature.*$/\n when: manual\n script:\n - export PROJECT_NS=`echo $PROJECT_NS | sed 's/\\//-/g'`\n - kubectl --kubeconfig=/etc/.beta/config delete deploy,svc,ing $APP_NAME -n $PROJECT_NS\n\njob_prod_build: ### prod环境编译打包，这里prod环境我们使用阿里云上的K8S\n stage: prod-build # 阿里云k8s平台上运行的uat环境和正式环境都使用本次打包镜像\n tags:\n - build-shell\n only: # 仅master和release分支可以执行该job\n - master\n - release\n #when: manual\n script:\n - bash .ci/config.sh # config.sh 会执行替换生产环境的变量\n - mvn clean install -Dmaven.test.skip=true -U # mvn 编译，可以去runner 虚机上手动执行编译测试\n - mv example-web/target/*.jar dockerfiles/ # 把mvn生成的xxx.jar移动到dockerfiles目录下\n - export IMAGE=`echo $IMAGE_NAME | sed 's/\\//-/g'`\n - cd dockerfiles && docker build -t $PROD_HARBOR/example/$IMAGE .\n - docker login -u $PROD_HARBOR_USR -p $PROD_HARBOR_PWD $PROD_HARBOR\n - docker push $PROD_HARBOR/example/$IMAGE\n - docker logout $PROD_HARBOR\n\njob_push_prod_uat: ### 部署至阿里云uat环境\n stage: prod-uat-deploy\n tags:\n - prod-shell\n when: manual\n only: # 仅master和release分支可以执行该job\n - master\n - release\n variables:\n PROD_EXP_Domain: 'example-uat.xxxx.com' # job内部变量，指定该应用在uat环境的 ingress 域名\n script:\n - export IMAGE=`echo $IMAGE_NAME | sed 's/\\//-/g'`\n - export PROJECT_NS=`echo $PROJECT_NS | sed 's/\\//-/g'`\n # 替换prod环境的参数配置\n - sed -i \"s/PROJECT_NS/$PROJECT_NS/g\" .ci/app.yaml\n - sed -i \"s/APP_NAME/$CI_PROJECT_NAME/g\" .ci/app.yaml\n - sed -i \"s/APP_REP/1/g\" .ci/app.yaml\n - sed -i \"s/AppDomain/$PROD_EXP_Domain/g\" .ci/app.yaml\n - sed -i \"s/ProjectImage/$PROD_HARBOR\\/example\\/$IMAGE/g\" .ci/app.yaml\n - sed -i \"s/DOCKER_KEY/$PROD_KEY/g\" .ci/app.yaml\n #\n - mkdir -p /opt/kube/$PROJECT_NS/$APP_NAME\n - cp -f .ci/app.yaml /opt/kube/$PROJECT_NS/$APP_NAME\n - kubectl --kubeconfig=/etc/.aliyun/config apply -f .ci/app.yaml\n\njob_push_prod_release: ### 部署至阿里云正式环境\n stage: prod-deploy\n tags:\n - prod-shell\n when: manual\n only: # 仅master和release分支可以执行该job\n - master\n - release\n variables:\n PROD_EXP_Domain: 'example.xxxx.com' # 指定该应用在阿里云正式环境的 ingress 域名\n script:\n - export IMAGE=`echo $IMAGE_NAME | sed 's/\\//-/g'`\n - export PROJECT_NS=`echo $PROJECT_NS | sed 's/\\//-/g'`\n # 替换prod环境的参数配置\n - sed -i \"s/PROJECT_NS/$PROJECT_NS/g\" .ci/app.yaml\n - sed -i \"s/APP_NAME/$CI_PROJECT_NAME/g\" .ci/app.yaml\n - sed -i \"s/APP_REP/$PROD_APP_REP/g\" .ci/app.yaml\n - sed -i \"s/AppDomain/$PROD_EXP_HOST/g\" .ci/app.yaml\n - sed -i \"s/ProjectImage/$PROD_HARBOR\\/example\\/$IMAGE/g\" .ci/app.yaml\n - sed -i \"s/DOCKER_KEY/$PROD_KEY/g\" .ci/app.yaml\n #\n - mkdir -p /opt/kube/$PROJECT_NS/$APP_NAME\n - cp -f .ci/app.yaml /opt/kube/$PROJECT_NS/$APP_NAME\n - kubectl --kubeconfig=/etc/.aliyun/config apply -f .ci/app.yaml\n\n1/3 rollback: ### 定义生产环境回退job \n stage: prod-rollback\n tags:\n - prod-shell\n when: manual\n only:\n - master\n - /^release.*$/\n variables:\n PROJECT_NS: '$CI_PROJECT_NAMESPACE-prod-deploy' # 定义job内变量覆盖全局变量设置\n script:\n - kubectl --kubeconfig=/etc/.aliyun/config -n $PROJECT_NS rollout undo deployment $CI_PROJECT_NAME --to-revision=1\n\n2/3 rollback:\n stage: prod-rollback\n tags:\n - prod-shell\n when: manual\n only:\n - master\n - /^release.*$/\n variables:\n PROJECT_NS: '$CI_PROJECT_NAMESPACE-prod-deploy' # 定义job内变量覆盖全局变量设置\n script:\n - kubectl --kubeconfig=/etc/.aliyun/config -n $PROJECT_NS rollout undo deployment $CI_PROJECT_NAME --to-revision=2\n\n3/3 rollback:\n stage: prod-rollback\n tags:\n - prod-shell\n when: manual\n only:\n - master\n - /^release.*$/\n variables:\n PROJECT_NS: '$CI_PROJECT_NAMESPACE-prod-deploy' # 定义job内变量覆盖全局变量设置\n script:\n - kubectl --kubeconfig=/etc/.aliyun/config -n $PROJECT_NS rollout undo deployment $CI_PROJECT_NAME --to-revision=3\nEOF\n```\n\n恭喜终于看完 gitlab-ci.yml 文件，怎么样，是不是一千个人可以写出一万个 CI/CD 流程 :)\n"
},
{
"path": "docs/deprecated/gitlab/gitlab-install.md",
"content": "# 安装 gitlab [Deprecated]\n\ngitlab 是深受企业用户喜爱的基于 git 的代码管理系统。安装 gitlab 最理想的方式是利用 gitlab charts 部署到 k8s 集群上，但此方式还未成熟，期待后续推出更成熟稳定版本；本文使用 Docker 方式安装 gitlab:\n\n- 环境：Ubuntu 16.04，虚机内存/CPU/存储请根据实际使用情况配置，一般`4C/8G/200G`足够\n- 安装 docker: 18.06.1-ce\n\n## 准备启动脚本\n\n``` bash\n$ cat > gitlab-setup.sh << EOF\n#!/bin/bash\n# 注意：设置 gitlab_shell_ssh_port 是为了后续可以使用 SSH 方式访问你的项目\ndocker run --detach \\\\\n --hostname gitlab.test.com \\\\\n --env GITLAB_OMNIBUS_CONFIG=\"external_url 'http://gitlab.test.com/'; gitlab_rails['gitlab_shell_ssh_port'] = 6022;\" \\\\\n --publish 443:443 --publish 80:80 --publish 6022:22 \\\\\n --name gitlab \\\\\n --restart always \\\\\n --volume /srv/gitlab/config:/etc/gitlab \\\\\n --volume /srv/gitlab/logs:/var/log/gitlab \\\\\n --volume /srv/gitlab/data:/var/opt/gitlab \\\\\n docker.mirrors.ustc.edu.cn/gitlab/gitlab-ce:11.2.2-ce.0\nEOF\n```\n执行启动脚本：`sh gitlab-setup.sh` 执行成功后，等待数分钟可以看到\n\n```\n$ docker ps -a\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n4f9d5f97f494 docker.mirrors.ustc.edu.cn/gitlab/gitlab-ce:11.2.2-ce.0 \"/assets/wrapper\" 9 minutes ago Up 9 minutes (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:6022->22/tcp gitlab\n```\n## 配置 gitlab\n\n```\n$ docker exec -it gitlab vi /etc/gitlab/gitlab.rb\n```\n请阅读后修改（因为前面docker run 已经指定了必要参数，可以不修改，后续有需要再修改），修改保存以后需要重启容器\n\n```\n$ docker restart gitlab\n```\n## 首次访问 gitlab\n\n使用域名`gitlab.test.com`或者该主机 IP 首次登录时会要求设置 root 用户的密码，完成后就可以用 root 和新设密码登录；然后按需创建 Group, User, Projects等，还有相关配置。\n\n## 备份数据\n\n无论是企业、组织、个人都十分重视代码资产，之前我们的 gitlab 安装是单机版的，虽然可以有硬盘 raid 等保护，还有是丢失 gitlab 数据和配置的风险，因此我们有必要再做一些备份操作。这里利用 crontab 定期执行 rsync 命令备份到其他服务器。\n\n``` bash\n# 创建备份脚本\ncat > /root/gitlab-backup.sh << EOF\n#!/bin/bash\n# 请事先配置 gitlab 服务器到备份服务器的免密码 ssh 登录\nrsync -av --delete /srv/gitlab/config '-e ssh -l root' 192.168.1.xx:/backup_gitlab/config\nrsync -av --delete /srv/gitlab/data '-e ssh -l root' 192.168.1.xx:/backup_gitlab/data\nEOF\n\n# 创建并应用 crontab\ncat > /etc/cron.d/gitlab-backup << EOF\n## 每3个小时同步备份一次，具体根据需要修改\n11 */3 * * * root bash /root/gitlab-backup.sh > /root/gitlab/sync.log 2>&1\nEOF\n```\n如果 gitlab 服务器真的出现不可恢复的故障，丢失数据，那么至少保留有3小时前的备份，利用备份的文件，同样再用 docker 挂载 volume的方式运行，这样就可以恢复原 gitlab 服务运行。\n\n## 升级 gitlab\n\n因为前面使用了 docker 方式安装，因此 gitlab 升级很方便。\n\n- 升级前停止/删除容器：`$ docker stop gitlab && docker rm gitlab`\n- 如上节执行备份数据\n- 修改 gitlab-setup.sh 指定新的版本，执行该脚本\n\n## 参考\n\n- 1.[Install GitLab with Docker](https://docs.gitlab.com/omnibus/docker/)\n"
},
{
"path": "docs/deprecated/gitlab/gitlab-runner.md",
"content": "## 安装 Gitlab Runner\n\nGitlab Runner 安装方式有很多，可以参考官网文档 https://docs.gitlab.com/runner/install/; 这里为了方便直接在 Ubuntu1604 上 apt方式安装了。\n\n``` bash\n$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash\n$ apt-get install gitlab-runner\n```\n\n安装完成后就可以看到服务运行状态：`systemctl status gitlab-runner`，因为示例的java spring boot 项目需要，该虚机上要同时安装和配置 mvn 和 docker 环境。\n\n注意：需要通过 gitlab-runner shell 执行docker镜像打包等命令，因此要修改下 gitlab-runner 服务运行用户：\"--user\" \"gitlab-runner\" 改成 \"--user\" \"root\"\n\n``` bash\nvi /etc/systemd/system/gitlab-runner.service\n[Unit]\nDescription=GitLab Runner\nAfter=syslog.target network.target\nConditionFileIsExecutable=/usr/lib/gitlab-runner/gitlab-runner\n\n[Service]\nStartLimitInterval=5\nStartLimitBurst=10\nExecStart=/usr/lib/gitlab-runner/gitlab-runner \"run\" \"--working-directory\" \"/home/gitlab-runner\" \"--config\" \"/etc/gitlab-runner/config.toml\" \"--service\" \"gitlab-runner\" \"--syslog\" \"--user\" \"gitlab-runner\"\n\nRestart=always\nRestartSec=120\n\n[Install]\nWantedBy=multi-user.target\n```\n\n以上配置改完保存后执行服务重启：\n\n``` bash\n$ systemctl daemon-reload\n$ systemctl restart gitlab-runner\n```\n\n### 注册 Runner\n\n运行命令`gitlab-runner register`后进入交互式界面，按照提示注册，关注下面注释内容。\n\n``` bash\n$ gitlab-runner register\nRuntime platform arch=amd64 os=linux pid=3269 revision=8bb608ff version=11.7.0\nRunning in system-mode. \n \nPlease enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):\nhttp://gitlab.test.com/ ### 这里输入gitlab URL\nPlease enter the gitlab-ci token for this runner:\ntzfBWCX-tQxxo1TCcoeJ ### 这里输入项目的token\nPlease enter the gitlab-ci description for this runner:\n[k8s403]: build-shell ### 命名此runner\nPlease enter the gitlab-ci tags for this runner (comma separated):\nbuild-shell ### 重要：指定runner tag，在gitlab-ci.yml文件中定义该tag才能执行 mvn编译/docker打包的任务\nRegistering runner... succeeded runner=tzfBWCX-\nPlease enter the executor: docker-ssh, parallels, shell, ssh, virtualbox, kubernetes, docker, docker+machine, docker-ssh+machine:\nshell ### 作为入门，在虚机上运行shell命令方式，方便调试\nRunner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!\n```\n另外根据示例项目的ci/cd流程，还需要注册标签 tag 为 `beta-shell` 和 `prod-shell` 的两个 Runner; 注意这两个runner所在虚机需要分别配置测试k8s和生产k8s的 kubeconfig 配置，这样 Runner 才能通过 shell 脚本执行 kubectl apply 命令部署应用。三个 Runner 注册成功后可以看到如图：\n\n\n\n"
},
{
"path": "docs/deprecated/gitlab/readme.md",
"content": "# Gitlab CI/CD 基础\n\nGitlab-ci 兼容 travis ci 格式，也是最流行的 CI 工具之一；本文讲解利用 gitlab, gitlab-runner, docker, harbor, kubernetes 等流行开源工具搭建一个自动化CI/CD流水线；示例配置以简单实用为原则，暂时没有选用 dind（docker in dockers）打包、gitlab Auto DevOps 等方式。一个最简单的流水线如下：\n\n- 代码提交 --> 镜像构建 --> 部署测试 --> 部署生产\n\n## 0.前提条件\n\n- 正常运行的 gitlab，[安装 gitlab 文档](gitlab-install.md)\n- 正常运行的容器仓库，[安装 Harbor 文档](../harbor.md)\n- 正常运行的 k8s，可以本地自建 k8s 集群，也可以使用公有云 k8s 集群\n- 若干虚机运行 gitlab-runner: 运行自动化流水线任务 pipeline job\n- 了解代码管理流程 gitflow 等\n\n## 1.准备测试项目代码\n\n假设你要开发一个 spring boot 项目；先登录你的 gitlab 账号，创建项目，上传你的代码；项目根目录看起来如下：\n\n```\n-rw-r--r-- 1 root root 44 Jan 2 16:38 eclipse.bat\ndrwxr-xr-x 8 root root 4096 Jan 7 15:29 .git/\n-rw-r--r-- 1 root root 276 Jan 7 08:44 .gitignore\ndrwxr-xr-x 3 root root 4096 Jan 7 08:44 example-api/\ndrwxr-xr-x 3 root root 4096 Jan 7 08:44 example-biz/\ndrwxr-xr-x 3 root root 4096 Jan 2 16:38 example-dal/\ndrwxr-xr-x 3 root root 4096 Jan 2 16:38 example-web/\n-rw-r--r-- 1 root root 54 Jan 2 16:38 install.bat\n-rw-r--r-- 1 root root 10419 Jan 2 16:38 pom.xml\n```\n传统做法是在本地配置好相关环境后使用 mvn 编译生成jar包，然后测试运行jar；这里我们要把应用打包成 docker 镜像，并创建 CI/CD 流水线：如下示例，在项目根目录新增创建2个文件夹及相关文件\n\n``` bash\ndockerfiles ### 新增文件夹用来 docker 镜像打包\n└── Dockerfile # 定义 docker 镜像\n.ci ### 新增文件夹用来存放 CI/CD 相关内容\n├── app.yaml # k8s 平台的应用部署文件\n├── config.sh # 配置替换脚本\n└── gitlab-ci.yml # gitlab-ci 的主配置文件\n```\n\n## 2.准备 docker 镜像描述文件 Dockerfile\n\n我们把 Dockerfile 放在独立目录下，java spring boot 应用可以这样写：\n\n``` bash\ncat > dockerfiles/Dockerfile << EOF\nFROM openjdk:8-jdk-alpine\nVOLUME /tmp\nCOPY *.jar app.jar # 这里 *.jar 包就是后续在cicd pipeline 过程中 mvn 生成的jar包移动到此目录\nENTRYPOINT [\"java\",\"-Djava.security.egd=file:/dev/./urandom\",\"-jar\",\"/app.jar\"]\nEOF\n```\n\n## 3.准备 CI/CD 相关脚本和文件\n\n装完 gitlab 后使用浏览器登录gitlab，很容易找到帮助文档，里面有介绍gitlab-ci的内容（文档权威、详细！请多多阅读~ 随着CI/CD流程的深入，部分内容也可以回来查阅），先看如下文档（假设你本地gitlab使用域名`gitlab.test.com`）\n\n- 文档首页 http://gitlab.test.com/help\n- gitlab-ci 基本概念 http://gitlab.test.com/help/ci/README.md\n- variables 变量 http://gitlab.test.com/help/ci/variables/README.md\n\n目录`.ci`下面的三个文件`app.yaml`, `config.sh`, `gitlab-ci.yml`是互相关联的；gitlab-ci.yml 文件中会调用到另外两个文件；文件之间又通过一些变量定义联系，流程中用到的变量大致可以分为三种：\n\n- 第一种是gitlab自身预定义变量（比如项目名: CI_PROJECT_NAME，流水线ID: CI_PIPELINE_ID）；无需更改；\n- 第二种是在gitlab-ci.yml文件中定义的变量，一般是少量的自定义变量；按需少量改动；\n- 第三种是用户可以在项目web界面配置的变量：“Settings”>\"CI/CD\">\"Variables\"，本示例项目用到该类型变量举例：\n\n|变量|值|注解|\n|:-|:-|:-|\n|BETA_APP_REP|1|beta环境应用副本数|\n|BETA_DB_HOST|1.1.1.1:3306|beta环境应用连接数据库主机|\n|BETA_DB_PWD|xxxx|beta环境数据库连接密码|\n|BETA_DB_USR|xxxx|beta环境数据库连接用户|\n|BETA_REDIS_HOST|1.1.1.2|beta环境redis主机|\n|BETA_REDIS_PORT|6379|beta环境redis端口|\n|BETA_REDIS_PWD|xxxx|beta环境redis密码|\n|BETA_HARBOR|1.1.1.3|beta环境镜像仓库地址|\n|BETA_HARBOR_PWD|xxxx|beta环境镜像仓库密码|\n|BETA_HARBOR_USR|xxxx|beta环境镜像仓库用户|\n|PROD_APP_REP|2|prod环境应用副本数|\n|PROD_DB_HOST|2.2.2.1:3306|prod环境应用连接数据库主机|\n|PROD_DB_PWD|xxxx|prod环境数据库连接密码|\n|PROD_DB_USR|xxxx|prod环境数据库连接用户|\n|PROD_REDIS_HOST|2.2.2.2|prod环境redis主机|\n|PROD_REDIS_PORT|6379|prod环境redis端口|\n|PROD_REDIS_PWD|xxxx|prod环境redis密码|\n|PROD_HARBOR|2.2.2.3|prod环境镜像仓库地址|\n|PROD_HARBOR_PWD|xxxx|prod环境镜像仓库密码|\n|PROD_HARBOR_USR|xxxx|prod环境镜像仓库用户|\n|...|...|根据项目需要自行添加设置|\n\n掌握了以上基础知识，可以开始以下三个任务：\n\n- 3.1[配置 gitlab-ci.yml](gitlab-ci.yml.md), 整个CI/CD的主配置文件，定义所有的CI/CD阶段和每个阶段的任务\n- 3.2[配置 config.sh](config.sh.md)，根据不同分支/环境替换不同的应用程序变量（对应上述第三种变量）\n- 3.3[配置 app.yaml](app.yaml.md)，K8S应用部署简单模板，替换完成后可以部署到测试/生产的K8S平台上\n\n## 4.为项目配置 CI/CD 及创建 RUNNER\n\n使用浏览器访问gitlab，登录后，在项目页面进行配置，如图：\n\n\n\n- 在 General pipelines 中配置 Custom CI config path 为 .ci/gitlab-ci.yml\n- 在 Variables 中配置需要用到的变量\n- 在 Runners 中配置注册 gitlab-runner 实例（runner 就是用来自动执行ci job的），点进去后如图：\n\n\n\n- 作为入门，先来手动创建 specific Runner，后续同样可以创建 Group Runners/Shared Runners，使用起来更方便；本文档暂不涉及在 kubernetes 自动创建 Runner\n - 按照官网文档安装 Gitlab Runner，参考[文档](gitlab-runner.md)\n - 记下 gitlab URL, 项目 token，注册 Runner 时要用到\n - 在 Gitlab Runner 注册本项目\n\n## 5.提交代码测试 CI/CD Pipelines\n\n终于经过 1~4 步骤把示例项目的CI/CD 流水线创建出来了，是时候试试提交代码测试下成果了；别担心，初次 CI/CD job执行一般都会失败的:) 好在现在你已经基本了解了所有CI/CD流程与配置，失败了就查看错误日志一一排除。另外因为采用虚机安装 Runner 执行 shell 脚本的方式执行 ci job，我们始终可以登录虚机以手动执行 shell 脚本的方式调试，这对于初学来说很有帮助。查看 CI/CD 执行情况如图：\n\n\n\n## 6.gitlab-ci 安全实践\n\n现在为止 CICD Pipelines 已经可以跑通了，甚至稍微修改下 gitlab-ci.yml 配置，项目代码每一次提交后可以自动执行`编译`、`打包`、`部署测试`、`部署生产`等等工作；也许你还没来得及慢慢体会这顺畅的感觉，赶紧先踩个刹车，控制下车速；因为现在你需要考虑 gitlab-ci 的安全配置了，这很重要！\n\n首先 gitlab 项目的基本安全就是项目成员控制，访问项目的权限分为：所有者（Owner），维护者（Maintainer），开发者（Developer），报告者（Reporter），访客（Guest）；详细的权限介绍请查阅官方文档，这里简单地介绍两类权限：所有者和维护者属于`特权用户`，开发者属于`普通用户`，他们应该具有如下权限区分：\n\n- 特权用户对整个项目负责，包括项目代码开发、配置管理、CI流程、测试环境、生产环境等\n- 特权用户可以提交代码到所有分支包括 master/release 分支，执行所有 ci job\n- 普通用户只负责对应项目模块代码开发、不接触程序配置、只能访问测试环境\n- 普通用户只能提交代码到 develop/feature 分支，只能执行这两个分支的 ci job\n\n以下的安全实践配置作为个人经验分享，仅作参考；如果你的项目需要更高的安全性，请阅读 gitlab-ci 官方相关文档，尝试找到属于自己的最佳实践。\n\n- 正确设置项目成员（Settings > Members），严格限制项目维护者（Maintainer）人数，大部分应该作为开发者（Developer）提交代码\n- 配置项目受保护分支/受保护标签，一般把master/release分支设置成受保护分支，限制只有维护者才能在保护分支commit和merge，从而限制只有维护者才能执行部署生产的 ci job，http://gitlab.test.com/help/user/project/protected_branches.md\n- 配置受保护的变量，受保护的变量只在受保护分支和受保护tag的pipeline中可见，防止生产环境配置参数泄露，http://gitlab.test.com/help/ci/variables/README#protected-variables\n- 配置受保护的Runner，只能执行受保护分支上的 ci jobs\n- CICD Pipelines 中发布生产的任务请设置手动执行，同样生产的回退任务设置手动执行\n\n"
},
{
"path": "docs/deprecated/jenkins.md",
"content": "# Jenkins CI/CD\n\n**此文档已过期，仅留档**\n\n## 前言\n本文档介绍如何快速通过K8s集群实现Jenkins 动态Slave CI/CD流程。\n\n## 开始之前\n在开始之前需要准备以下环境：\n- k8s dns组件 \n参考文档：[kubedns](kubedns.md)\n- helm \n为了简化部署，通过helm来安装Jenkins，可参考文档：[helm](helm.md)\n- 持久化存储 \n这里使用**NFS**演示，参考文档：[cluster-storage](../setup/08-cluster-storage.md)。\n如果k8s集群是部署在公有云，也可使用厂商的NAS等存储方案，项目中已集成支持阿里云NAS，其他的方案参考相关厂商文档\n\n- Ingress Controller(nginx-ingress/traefik) \n默认是通过Ingress访问Jenkins，因此需要安装一种`Ingress Controller`。参考文档：[ingress](ingress.md)\n- Gitlab 代码管理仓库 \n用于提交代码后自动触发CI, 目前项目中还没有相关内容，可[参考官网](https://about.gitlab.com/installation/)进行安装。\n\n## 安装Jenkins\n执行以下命令快速安装：\n```\nhelm install manifests/jenkins/ --name jenkins\n```\n如果通过/etc/kubeasz/roles/helm/helm.yml安装的helm，安装过程会出现如下错误\n\n``` bash\nE0703 08:40:22.376225 19888 portforward.go:331] an error occurred forwarding 41655 -> 44134: error forwarding port 44134 to pod 5098414beaaa07140a4ba3240690b1ce989ece01e5db33db65eec83bd64bdedf, uid : exit status 1: 2018/07/03 08:40:22 socat[19991] E write(5, 0x1aec120, 3424): Connection reset by peer\nError: transport is closing\n```\n请执行以下命令快速安装进行修复：\n```\nhelm install --tls manifests/jenkins/ --name jenkins\n```\n\n由于初始化过程中，默认安装指定的插件，所以启动较慢，大概5-10分钟左右就可以启动完成了。 \n\n部分默认配置说明：\n**注**：以下配置都定义在`manifests/jenkins/values.yaml`文件中。\n\n \n 字段 说明 默认值 \n \n InstallPlugins \n 初始化安装的插件 \n \n \n kubernetes:1.6.3 \n workflow-aggregator:2.5 \n workflow-job:2.21 \n credentials-binding:1.16 \n git:3.9.0 \n gitlab:1.5.6 \n \n \n \n \n HostName \n Ingress访问入口 \n jenkins.local.com \n \n \n AdminPassword \n admin登录密码 \n admin \n \n \n UpdateCenter \n 插件下载镜像地址 \n https://mirrors.tuna.tsinghua.edu.cn/jenkins \n \n \n StorageClass \n 持久化存储SC \n nfs-dynamic-class \n \n
\n\n\n## 配置Kubernetes plugin\n登录Jenkins，点击左边导航`系统管理`——>`系统设置`，拖动到最下面可以看到`云——>Kubernetes`配置，默认配置有以下字段： \n\n- Name：配置名称，后面运行测试的时候会用到，用于区别多个Kubernetes配置，默认为：kubernetes\n- Kubernetes URL：集群访问url，可通过`kubectl cluster-info`查看，如果集群有部署**DNS**插件, 也可以直接填服务名称(自动解析)，默认使用服务名称：https://kubernetes\n- Jenkins URL：Jenkins访问地址，默认使用服务名称+端口号\n\n在Jenkins初始化时，默认都已经配置好了，可以直接新建项目测试了。\n\n## 简单测试\n点击左边：新建任务——>流水线(Pipeline)\n任务名称可以随便起，这里为：k8s-test\n配置——>流水线，选择`Pipeline script`\n以下为测试脚本内容：\n```\npodTemplate(label: 'jenkins-slave', cloud: 'kubernetes')\n{\n node ('jenkins-slave') {\n stage('test') {\n echo \"hello, world\"\n sleep 60\n }\n }\n}\n```\n\n- cloud：插件配置中的Name\n- label：插件配置中的Images——>Kubernetes Pod Tempalte——>Labels\n- node：与label一致即可\n\n保存配置，点击立即构建，查看控制台输出，出现以下内容就表示运行成功了：\n```\nAgent default-lsths is provisioned from template Kubernetes Pod Template\nAgent specification [Kubernetes Pod Template] (jenkins-slave): \n* [jnlp] jenkins/jnlp-slave:alpine(resourceRequestCpu: 200m, resourceRequestMemory: 256Mi, resourceLimitCpu: 200m, resourceLimitMemory: 256Mi)\n\nRunning on default-lsths in /home/jenkins/workspace/k8s-test\n[Pipeline] {\n[Pipeline] stage\n[Pipeline] { (test)\n[Pipeline] echo\nhello, world\n[Pipeline] sleep\nSleeping for 1 min 0 sec\n[Pipeline] }\n[Pipeline] // stage\n[Pipeline] }\n[Pipeline] // node\n[Pipeline] }\n[Pipeline] // podTemplate\n[Pipeline] End of Pipeline\nFinished: SUCCESS\n```\n\n\n## 配置自动触发CI\n\n- 配置Gitlab项目 \n在`Gitlab`中创建一个测试项目，将上面测试的脚本内容写入到一个`Jenkinsfile`文件中，然后上传到该测试项目根路径下。\n\n- 配置Jenkins项目 \n点击项目`配置`——>`构建触发器`——>勾选`Build when a change is pushed to GitLab. GitLab webhook URL:http://jenkins.local.com/project/k8s-test`——>保存配置\n\n- 配置Webhook \n进入Gitlab测试项目的`Settings——>Integrations`，一般只需要填写`URL`即可，其他的可根据需求环境配置\n默认Jenkins配置不允许匿名用户触发构建，因此还需要添加用户和token。 \nURL的格式为： \n`http://[UserID]:[API Token]@jenkins.local.com/project/[ProjectName]`\n\nJenkins 用户ID Token查看：\n点击右上角的`用户名——>设置——>API Token(点击Show API Token...)`\n\n最终Webhook中的URL类似：\nhttp://admin:a910b1492e39e9dd1ea48ea7f7638aaf@jenkins.local.com/project/k8s-test\n\n后面只需要我们一提交代码到Git仓库，就会自动触发Jenkins进行构建了。\n\n## 项目应用\n这里我们以一个简单的Java项目为例，实战演示如何进行CI/CD。\n基本环境配置上面已经说过了，这里就不多介绍。 \n示例项目：https://github.com/lusyoe/springboot-k8s-example\n\n结构说明：\n- 镜像构建文件：`Dockerfile`\n- k8s应用配置：`k8s-example.yaml`\n- 项目源码：`src`\n- Jenkins构建文件：`jenkins/Jenkinsfile`\n\n构建流程说明：\n- 通过Jenkins kubernetes插件，定义构建过程中所需的3个docker容器：maven、docker、kubectl (这3个容器都在一个pod中)\n- 挂载docker.sock和kubeconfig文件\n- 首先使用`maven`容器，检出代码，执行项目构建\n- 使用`docker`容器，构建镜像，推送到镜像参考\n- 使用`kubectl`容器，部署`k8s-example`应用(这里后面也可以使用helm)\n\n访问： \n项目通过Ingress访问`k8s-example.com`，出现`hello, world`,就表示服务部署成功了。\n"
},
{
"path": "docs/deprecated/kuboard.md",
"content": "# 安装 Kuboard\n\n## Kuboard 介绍\n\nKuboard 是一款免费的 Kubernetes 管理工具，提供了丰富的功能：\n\n* Kubernetes 多集群管理\n* Kubernetes 基本管理功能\n * 节点管理\n * 名称空间管理\n * 存储类/存储卷管理\n * 控制器（Deployment/StatefulSet/DaemonSet/CronJob/Job/ReplicaSet）管理\n * Service/Ingress 管理\n * ConfigMap/Secret 管理\n * CustomerResourceDefinition 管理\n* Kubernetes 问题诊断\n * Top Nodes / Top Pods\n * 事件列表及通知\n * 容器日志及终端\n * KuboardProxy (kubectl proxy 的在线版本)\n * PortForward (kubectl port-forward 的快捷版本)\n * 复制文件 （kubectl cp 的在线版本）\n* 认证与授权\n * Github/GitLab 单点登录\n * KeyCloak 认证\n * LDAP 认证\n * 完整的 RBAC 权限管理\n* Kuboard 特色功能\n * Kuboard 官方套件\n * Grafana+Prometheus 资源监控\n * Grafana+Loki+Promtail 日志聚合\n * Kuboard 自定义名称空间布局\n * Kuboard 中英文语言包\n\n\n \n \n
\n\n\n \n\n点击这里可以查看 [Kuboard 的安装文档](https://kuboard.cn/install/v3/install.html)\n\n## 在线演示\n\n\n 在线演示环境中，您具备 只读 权限，只能体验 Kuboard 的一部分功能。
\n\n\n## 特点介绍\n\n相较于 Kubernetes Dashboard 等其他 Kubernetes 管理界面，Kuboard 的主要特点有：\n\n* 多种认证方式\n\n Kuboard 可以使用内建用户库、gitlab / github 单点登录或者 LDAP 用户库进行认证，避免管理员将 ServiceAccount 的 Token 分发给普通用户而造成的麻烦。使用内建用户库时，管理员可以配置用户的密码策略、密码过期时间等安全设置。\n\n \n\n* 多集群管理\n\n 管理员可以将多个 Kubernetes 集群导入到 Kuboard 中，并且通过权限控制，将不同集群/名称空间的权限分配给指定的用户或用户组。\n\n \n\n* 微服务分层展示\n\n 在 Kuboard 的名称空间概要页中，以经典的微服务分层方式将工作负载划分到不同的分层，更加直观地展示微服务架构的结构，并且可以为每一个名称空间自定义名称空间布局。\n\n \n\n* 工作负载的直观展示\n\n Kuboard 中将 Deployment 的历史版本、所属的 Pod 列表、Pod 的关联事件、容器信息合理地组织在同一个页面中，可以帮助用户最快速的诊断问题和执行各种相关操作。\n\n \n\n* 工作负载编辑\n\n Kuboard 提供了图形化的工作负载编辑界面，用户无需陷入繁琐的 YAML 文件细节中，即可轻松完成对容器的编排任务。支持的 Kubernetes 对象类型包括：Node、Namespace、Deployment、StatefulSet、DaemonSet、Secret、ConfigMap、Service、Ingress、StorageClass、PersistentVolumeClaim、LimitRange、ResourceQuota、ServiceAccount、Role、RoleBinding、ClusterRole、ClusterRoleBinding、CustomResourceDefinition、CustomResource 等各类常用 Kubernetes 对象，\n\n \n\n* 存储类型支持\n\n 在 Kuboard 中，可以方便地对接 NFS、CephFS 等常用存储类型，并且支持对 CephFS 类型的存储卷声明执行扩容和快照操作。\n\n \n\n* 丰富的互操作性\n\n 可以提供许多通常只在 `kubectl` 命令行界面中才提供的互操作手段，例如：\n\n * Top Nodes / Top Pods\n * 容器的日志、终端\n * 容器的文件浏览器（支持从容器中下载文件、上传文件到容器）\n * KuboardProxy（在浏览器中就可以提供 `kubectl proxy` 的功能）\n\n \n\n* 套件扩展\n\n Kuboard 提供了必要的套件库，使得用户可以根据自己的需要扩展集群的管理能力。当前提供的套件有：\n\n * 资源层监控套件，基于 Prometheus / Grafana 提供 K8S 集群的监控能力，可以监控集群、节点、工作负载、容器组等各个级别对象的 CPU、内存、网络、磁盘等资源的使用情况；\n * 日志聚合套件，基于 Grafana / Loki / Promtail 实现日志聚合；\n * 存储卷浏览器，查看和操作存储卷中的内容；\n\n \n\n\n\n访问 Kuboard 网站 https://kuboard.cn 可以加入 Kuboard 社群，并获得帮助。\n"
},
{
"path": "docs/deprecated/practice/dockerize_system_service.md",
"content": "# 容器化系统服务\n\n## 容器化 haproxy\n\n本例使用 [docker hub 官方](https://github.com/docker-library/haproxy) 维护的 haproxy 镜像；haproxy 配置举例如下\n\n```\nglobal\n log stdout format raw local1 notice\n nbproc 1\n\ndefaults\n log global\n timeout connect 5s\n timeout client 10m\n timeout server 10m\n\nlisten apiservers\n bind 0.0.0.0:6443\n mode tcp\n option tcplog\n option dontlognull\n option dontlog-normal\n balance roundrobin \n server 192.168.1.1 192.168.1.1:6443 check inter 10s fall 2 rise 2 weight 1\n server 192.168.1.2 192.168.1.2:6443 check inter 10s fall 2 rise 2 weight 1\n```\n\n在 systemd 系统上编写服务文件如下 /etc/systemd/system/haproxy.service\n\n```\n[Unit]\nDescription=haproxy\nDocumentation=https://github.com/docker-library/haproxy\nAfter=docker.service\nRequires=docker.service\n\n[Service]\nUser=root\nExecStart=/bin/docker run \\\n --name haproxy \\\n --publish 6443:6443 \\\n --volume /etc/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg \\\n docker.io/library/haproxy:1.9.8-alpine\nExecStop=/bin/docker rm -f haproxy\nExecReload=/bin/docker kill -s HUP haproxy\nRestart=always\nRestartSec=10\nDelegate=yes\nLimitNOFILE=50000\nLimitNPROC=50000\n\n[Install]\nWantedBy=multi-user.target\n```\n\n## 容器化 chrony\n\n- chrony 服务器端配置（假设chrony服务器端192.168.1.1）\n\n```\n$ cat /etc/chrony.conf\n# Use public servers from the pool.ntp.org project.\nserver ntp1.aliyun.com iburst\nserver ntp2.aliyun.com iburst\npool pool.ntp.org iburst\n\n# Ignor source level\nstratumweight 0\n\n# Record the rate at which the system clock gains/losses time.\ndriftfile /var/lib/chrony/drift\n\n# Allow the system clock to be stepped in the first five updates\n# if its offset is larger than 1 second.\nmakestep 1 5\n\n# Enable kernel synchronization of the real-time clock (RTC).\nrtcsync\n\n# Allow NTP client access from local network.\nallow 0.0.0.0/0\n\n# Serve time even if not synchronized to a time source.\nlocal stratum 10\n\n# Select which information is logged.\n#log measurements statistics tracking\n\n#\nnoclientlog\n```\n- chrony 客户端配置\n\n```\n$ cat /etc/chrony.conf\n# Use local chrony server.\nserver 192.168.1.1 iburst\n\n# Record the rate at which the system clock gains/losses time.\ndriftfile /var/lib/chrony/drift\n\n# Allow the system clock to be stepped in the first five updates\n# if its offset is larger than 1 second.\nmakestep 1 5\n\n# Enable kernel synchronization of the real-time clock (RTC).\nrtcsync\n\n# Select which information is logged.\n#log measurements statistics tracking\n```\n\n- 在 systemd 系统上编写服务文件如下 /etc/systemd/system/chrony.service\n\n```\n[Unit]\nDescription=chrony\nDocumentation=https://github.com/kubeasz/dockerfiles/chrony\nAfter=docker.service\nRequires=docker.service\n\n[Service]\nUser=root\nExecStart=/opt/kube/bin/docker run \\\n --cap-add SYS_TIME \\\n --name chrony \\\n --network host \\\n --volume /etc/chrony.conf:/etc/chrony/chrony.conf \\\n --volume /var/lib/chrony:/var/lib/chrony \\\n easzlab/chrony:0.1.0\nExecStartPost=/sbin/iptables -t raw -A PREROUTING -p udp -m udp --dport 123 -j NOTRACK\nExecStartPost=/sbin/iptables -t raw -A OUTPUT -p udp -m udp --sport 123 -j NOTRACK\nExecStop=/opt/kube/bin/docker rm -f chrony\nRestart=always\nRestartSec=10\nDelegate=yes\n\n[Install]\nWantedBy=multi-user.target\n```\n"
},
{
"path": "docs/deprecated/practice/es_cluster.md",
"content": "# Elasticsearch 部署实践\n\n`Elasticsearch`是目前全文搜索引擎的首选，它可以快速地储存、搜索和分析海量数据；也可以看成是真正分布式的高效数据库集群；`Elastic`的底层是开源库`Lucene`；封装并提供了`REST API`的操作接口。\n\n## 单节点 docker 测试安装 \n \n``` bash\ncat > es-start.sh << EOF\n#!/bin/bash\n\nsysctl -w vm.max_map_count=262144\n\ndocker run --detach \\\n --name es01 \\\n -p 9200:9200 -p 9300:9300 \\\n -e \"discovery.type=single-node\" \\\n -e \"bootstrap.memory_lock=true\" --ulimit memlock=-1:-1 \\\n --ulimit nofile=65536:65536 \\\n --volume /srv/elasticsearch/data:/usr/share/elasticsearch/data \\\n --volume /srv/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \\\n jmgao1983/elasticsearch:6.4.0\nEOF\n```\n\n执行`sh es-start.sh`后，就在本地运行了。\n\n- 验证 docker 镜像运行情况 \n\n``` bash\nroot@docker-ts:~# docker ps -a\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n171f3fecb596 jmgao1983/elasticsearch:6.4.0 \"/usr/local/bin/do...\" 2 hours ago Up 2 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp es01\n```\n\n- 验证 es 健康检查 \n\n``` bash\nroot@docker-ts:~# curl http://127.0.0.1:9200/_cat/health\nepoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent\n1535523956 06:25:56 docker-es green 1 1 0 0 0 0 0 0 - 100.0%\n```\n\n## 在 k8s 上部署 Elasticsearch 集群\n\n在生产环境下，Elasticsearch 集群由不同的角色节点组成：\n\n- master 节点：参与主节点选举，不存储数据；建议3个以上，维护整个集群的稳定可靠状态\n- data 节点：不参与选主，负责存储数据；主要消耗磁盘，内存\n- client 节点：不参与选主，不存储数据；负责处理用户请求，实现请求转发，负载均衡等功能\n\n这里使用`helm chart`来部署 (https://github.com/helm/charts/tree/master/incubator/elasticsearch)\n\n- 1.安装 helm: 以本项目[安全安装helm](../guide/helm.md)为例\n- 2.准备 PV: 以本项目[K8S 集群存储](../setup/08-cluster-storage.md)创建`nfs`动态 PV 为例\n- 3.安装 elasticsearch chart \n\n``` bash\n$ cd /etc/kubeasz/manifests/es-cluster\n# 如果你的helm安装没有启用tls证书，请忽略以下--tls参数\n$ helm install --tls --name es-cluster --namespace elastic -f es-values.yaml elasticsearch\n```\n\n- 4.验证 es 集群 \n\n``` bash\n# 验证k8s上 es集群状态\n$ kubectl get pod,svc -n elastic \nNAME READY STATUS RESTARTS AGE\npod/es-cluster-elasticsearch-client-778df74c8f-7fj4k 1/1 Running 0 2m17s\npod/es-cluster-elasticsearch-client-778df74c8f-skh8l 1/1 Running 0 2m3s\npod/es-cluster-elasticsearch-data-0 1/1 Running 0 25m\npod/es-cluster-elasticsearch-data-1 1/1 Running 0 11m\npod/es-cluster-elasticsearch-master-0 1/1 Running 0 25m\npod/es-cluster-elasticsearch-master-1 1/1 Running 0 12m\npod/es-cluster-elasticsearch-master-2 1/1 Running 0 10m\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/es-cluster-elasticsearch-client NodePort 10.68.157.105 9200:29200/TCP,9300:29300/TCP 25m\nservice/es-cluster-elasticsearch-discovery ClusterIP None 9300/TCP 25m\n\n# 验证 es集群本身状态\n$ curl $NODE_IP:29200/_cat/health\n1539335131 09:05:31 es-on-k8s green 7 2 0 0 0 0 0 0 - 100.0%\n\n$ curl $NODE_IP:29200/_cat/indices?v\nhealth status index uuid pri rep docs.count docs.deleted store.size pri.store.size\nroot@k8s401:/etc/kubeasz# curl 10.100.97.41:29200/_cat/nodes?\n172.31.2.4 27 80 5 0.09 0.11 0.21 mi - es-cluster-elasticsearch-master-0\n172.31.1.7 30 97 3 0.39 0.29 0.27 i - es-cluster-elasticsearch-client-778df74c8f-skh8l\n172.31.3.7 20 97 3 0.11 0.17 0.18 i - es-cluster-elasticsearch-client-778df74c8f-7fj4k\n172.31.1.5 8 97 5 0.39 0.29 0.27 di - es-cluster-elasticsearch-data-0\n172.31.2.5 8 80 3 0.09 0.11 0.21 di - es-cluster-elasticsearch-data-1\n172.31.1.6 18 97 4 0.39 0.29 0.27 mi - es-cluster-elasticsearch-master-2\n172.31.3.6 20 97 4 0.11 0.17 0.18 mi * es-cluster-elasticsearch-master-1\n```\n\n### es 性能压测\n\n如上已使用 chart 在 k8s上部署了 **7** 节点的 elasticsearch 集群；各位应该十分好奇性能怎么样；官方提供了压测工具[esrally](https://github.com/elastic/rally)可以方便的进行性能压测，这里省略安装和测试过程；压测机上执行： \n`esrally --track=http_logs --target-hosts=\"$NODE_IP:29200\" --pipeline=benchmark-only --report-file=report.md` \n压测过程需要1-2个小时，部分压测结果如下： \n\n``` bash\n------------------------------------------------------\n _______ __ _____\n / ____(_)___ ____ _/ / / ___/_________ ________\n / /_ / / __ \\/ __ `/ / \\__ \\/ ___/ __ \\/ ___/ _ \\\n / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/\n/_/ /_/_/ /_/\\__,_/_/ /____/\\___/\\____/_/ \\___/\n------------------------------------------------------\n\n| Lap | Metric | Task | Value | Unit |\n|------:|-------------------------------------:|-------------:|------------:|--------:|\n...\n| All | Min Throughput | index-append | 16903.2 | docs/s |\n| All | Median Throughput | index-append | 17624.4 | docs/s |\n| All | Max Throughput | index-append | 19382.8 | docs/s |\n| All | 50th percentile latency | index-append | 1865.74 | ms |\n| All | 90th percentile latency | index-append | 3708.04 | ms |\n| All | 99th percentile latency | index-append | 6379.49 | ms |\n| All | 99.9th percentile latency | index-append | 8389.74 | ms |\n| All | 99.99th percentile latency | index-append | 9612.84 | ms |\n| All | 100th percentile latency | index-append | 9861.02 | ms |\n| All | 50th percentile service time | index-append | 1865.74 | ms |\n| All | 90th percentile service time | index-append | 3708.04 | ms |\n| All | 99th percentile service time | index-append | 6379.49 | ms |\n| All | 99.9th percentile service time | index-append | 8389.74 | ms |\n| All | 99.99th percentile service time | index-append | 9612.84 | ms |\n| All | 100th percentile service time | index-append | 9861.02 | ms |\n| All | error rate | index-append | 0 | % |\n| All | Min Throughput | default | 0.66 | ops/s |\n| All | Median Throughput | default | 0.66 | ops/s |\n| All | Max Throughput | default | 0.66 | ops/s |\n| All | 50th percentile latency | default | 770131 | ms |\n| All | 90th percentile latency | default | 825511 | ms |\n| All | 99th percentile latency | default | 838030 | ms |\n| All | 100th percentile latency | default | 839382 | ms |\n| All | 50th percentile service time | default | 1539.4 | ms |\n| All | 90th percentile service time | default | 1635.39 | ms |\n| All | 99th percentile service time | default | 1728.02 | ms |\n| All | 100th percentile service time | default | 1736.2 | ms |\n| All | error rate | default | 0 | % |\n...\n``` \n\n从测试结果看：集群的吞吐可以（k8s es-client pod还可以扩展）；延迟略高一些（因为使用了nfs共享存储）；整体效果不错。\n\n### 中文分词安装\n\n安装 ik 插件即可，可以自定义已安装ik插件的es docker镜像：创建如下 Dockerfile \n\n``` bash\nFROM jmgao1983/elasticsearch:6.4.0\n\nRUN /usr/share/elasticsearch/bin/elasticsearch-plugin install \\\n --batch https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.4.0/elasticsearch-analysis-ik-6.4.0.zip \\\n && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime\n```\n\n### 参考阅读\n\n1. [Elasticsearch 入门教程](http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html)\n2. [Elasticsearch 压测方案之 esrally 简介](https://segmentfault.com/a/1190000011174694)\n"
},
{
"path": "docs/deprecated/practice/go_web_app/Dockerfile",
"content": "# a demon for containerize golang web apps\n#\n# @author:\n# @repo: \n# @ref: \n\n# stage 1: build src code to binary\nFROM golang:1.13-alpine3.10 as builder\n\nCOPY *.go /app/\n\nRUN cd /app && go build -o hellogo .\n\n# stage 2: use alpine as base image\nFROM alpine:3.10\n\nRUN apk update && \\\n apk --no-cache add tzdata ca-certificates && \\\n cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \\\n apk del tzdata && \\\n rm -rf /var/cache/apk/*\n\nCOPY --from=builder /app/hellogo /hellogo\n\nCMD [\"/hellogo\"] \n"
},
{
"path": "docs/deprecated/practice/go_web_app/Dockerfile-more",
"content": "# build stage\nFROM golang:1.13 as builder\n\n# ENV GOPROXY=https://goproxy.cn\n# 设置 GOPROXY 是为编译时能够通过代理下载Qiang外的包\n# 设置 GOPRIVATE 是为编译时下载本地gitlab上的包时候不使用代理\nENV GOPROXY=https://goproxy.io\nENV GOPRIVATE=gitlab.yourdomain.com/*\n\nWORKDIR /root\n\nCOPY ./ .\n\n# 本地 gitlab 上的项目非公开，编译时需要用 ssh key 的方式下载本地 gitlab 包\n# 提前把 ssh key 中的公钥上传到gitlab 个人profile中的 SSH KEY 中\n# 在 docker build 时通过命令行参数用--build-arg 'SSH_PKEY=${KEY_TXT}' 传入\n# 在 CICD 流水线中，${KEY_TXT} 可以是jenkins中的secret-text参数，也可以是gitlab-ci中的secret variables\nARG SSH_PKEY\n\n# 设置 git config 是为了拉区项目时使用ssh方式 git@gitlab.yourdomain.com:xxx/yyy.git\n# \n\nRUN git config --global url.\"git@gitlab.yourdomain.com:\".insteadof \"https://gitlab.yourdomain.com/\" && \\\n mkdir -p /root/.ssh && \\\n echo \"-----BEGIN RSA PRIVATE KEY-----\" > /root/.ssh/id_rsa && \\\n echo \"${SSH_PKEY}\" >> /root/.ssh/id_rsa && \\\n echo \"-----END RSA PRIVATE KEY-----\" >> /root/.ssh/id_rsa && \\\n sed -i \"2s/ /\\\\n/g\" /root/.ssh/id_rsa && \\\n echo \"StrictHostKeyChecking no\" > /root/.ssh/config && \\\n chmod 600 /root/.ssh/id_rsa\n\nRUN go mod tidy && \\\n go mod download\n\nRUN CGO_ENABLED=0 GOOS=linux go build -installsuffix cgo -o main cmd/main.go\n\n# final stage\nFROM alpine:3.10\n\nWORKDIR /home/admin/bin\n\nCOPY --from=builder /root/main .\n\nCMD [\"./main\"]\n"
},
{
"path": "docs/deprecated/practice/go_web_app/hellogo.go",
"content": "package main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"math/rand\"\n\t\"net/http\"\n\t\"time\"\n)\n\nvar appVersion = \"1.2\" //Default/fallback version\nvar instanceNum int\n\nfunc getFrontpage(w http.ResponseWriter, r *http.Request) {\n\tt := time.Now()\n\tfmt.Fprintf(w, \"Hello, Go! I'm instance %d running version %s at %s\\n\", instanceNum, appVersion, t.Format(\"2019-01-02 15:04:05\"))\n}\n\nfunc health(w http.ResponseWriter, r *http.Request) {\n\tw.WriteHeader(http.StatusOK)\n}\n\nfunc getVersion(w http.ResponseWriter, r *http.Request) {\n\tfmt.Fprintf(w, \"%s\\n\", appVersion)\n}\n\nfunc main() {\n\trand.Seed(time.Now().UTC().UnixNano())\n\tinstanceNum = rand.Intn(1000)\n\thttp.HandleFunc(\"/\", getFrontpage)\n\thttp.HandleFunc(\"/health\", health)\n\thttp.HandleFunc(\"/version\", getVersion)\n\tlog.Fatal(http.ListenAndServe(\":3000\", nil))\n}\n"
},
{
"path": "docs/deprecated/practice/go_web_app/hellogo.yaml",
"content": "---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: hellogo-deploy\nspec:\n replicas: 3\n minReadySeconds: 5 # Wait 5 seconds after each new pod comes up before marked as \"ready\"\n strategy:\n type: RollingUpdate # describe how we do rolling updates\n rollingUpdate:\n maxUnavailable: 1 # When updating take one pod down at a time\n maxSurge: 1\n selector:\n matchLabels:\n name: hellogo-app\n template:\n metadata:\n labels:\n name: hellogo-app\n spec:\n containers:\n - name: hellogo\n image: hellogo:v1.0\n imagePullPolicy: IfNotPresent\n resources:\n requests:\n memory: \"32Mi\"\n cpu: \"50m\"\n limits:\n memory: \"64Mi\"\n cpu: \"100m\"\n ports:\n - containerPort: 3000\n\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: hellogo-svc\nspec:\n type: NodePort\n ports:\n - name: http\n port: 80\n targetPort: 3000\n nodePort: 30000\n selector:\n name: hellogo-app\n"
},
{
"path": "docs/deprecated/practice/go_web_app/readme.md",
"content": "# 容器化 GO 应用\n\nGolang 作为服务器端新兴热门语言同时也是容器技术的主要编写语言备受关注；它简洁、有趣、并行、安全等特点让 GO 应用容器化相对省心；一般来说做下时间本地化、安装信任根证书，然后把编译生成的二进制拷贝进去即可。\n\n## 一个演示 GO WEB 应用\n\n[hellogo 代码](hellogo.go)\n\n## Dockerfile\n\n作为演示项目的Dockerfile比较简单，请看 [Dockerfile 文件](Dockerfile)\n\n- 采用 docker 多阶段编译，使生成的目标镜像最小\n- 使用 alpine 基础镜像\n- 安装 tzdata 做时间本地化\n- 安装信任根证书\n\n一个真实复杂go项目的Dockerfile可能如这个例子：[复杂 Dockerfile](Dockerfile-more)\n\n## 制作镜像\n\n在 Dockerfile 文件所在目录，执行\n\n```\ndocker build -t hellogo:v1.0 .\n```\n\n## 本地测试应用\n\n- 1.单机运行 hellogo 容器应用 \n\n```\ndocker run -d --name hello -p3000:3000 hellogo:v1.0\n```\n\n- 2.验证测试\n\n``` bash\n# 查看本地监听端口\n$ ss -ntl|grep 3000\nLISTEN 0 128 *:3000 *:*\n\n# 查看应用状态\n$ curl localhost:3000\nHello, Go! I'm instance 987 running version 1.2 at 13109-10-13 08:39:11\n\n$ curl localhost:3000/health -i\nHTTP/1.1 200 OK\nDate: Sun, 13 Oct 2019 00:39:15 GMT\nContent-Length: 0\n\n$ curl localhost:3000/version\n1.2\n```\n\n## 在 k8s 上运行演示应用\n\n- 可以参考项目`github.com/easzlab/kubeasz` 快速搭建一个本地 k8s 测试环境\n\n- 1.编写基于k8s的应用编排文件 [hellogo.yaml](hellogo.yaml)\n - 设置应用副本数`replicas: 3`\n - 预设新副本启动延迟5秒`minReadySeconds: 5`\n - 设置滚动更新策略\n - 设置资源使用限制，安装实际情况修改\n - 设置服务对外暴露方式 NodePort，根据实际情况修改端口，或者使用 ingress 方式\n\n- 2.在 k8s 上运行应用\n\n``` bash\n# 运行\n$ kubectl apply -f hellogo.yaml\n\n# 验证\n$ kubectl get pod\nNAME READY STATUS RESTARTS AGE\nhellogo-deploy-854dcd85c-2zm9l 1/1 Running 0 12m\nhellogo-deploy-854dcd85c-7nfk5 1/1 Running 0 12m\nhellogo-deploy-854dcd85c-ns7fp 1/1 Running 0 12m\n\n$kubectl get deploy\nNAME READY UP-TO-DATE AVAILABLE AGE\nhellogo-deploy 3/3 3 3 13m\n\n$kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nhellogo-svc NodePort 10.68.194.109 80:30000/TCP 13m\n\n# 使用curl测试应用三副本状态（用curl多次访问看到三个不同`instance id`）\n$ curl http://192.168.111.3:30000\nHello, Go! I'm instance 629 running version 1.2 at 13109-10-13 09:06:25\n\n$ curl http://192.168.111.3:30000\nHello, Go! I'm instance 722 running version 1.2 at 13109-10-13 09:06:27\n\n$curl http://192.168.111.3:30000\nHello, Go! I'm instance 799 running version 1.2 at 13109-10-13 09:06:28\n```\n"
},
{
"path": "docs/deprecated/practice/java_war_app.md",
"content": "# JAVA WAR 应用迁移 K8S 实践\n\n初步思路是这样：应用代码与应用配置分离，应用代码打包成 docker 镜像存于内部 harbor 仓库，应用配置使用 configmap 挂载，这样不同的环境只需要修改 configmap 即可部署。\n\n- 使用 maven 把 java 应用代码打包成 xxx.war\n- 基于 tomcat 镜像和 xxx.war 做成应用 docker 镜像\n- 编写 k8s deployment 文件，在 pod 指定上述应用镜像，同时把应用配置做成 configmap 挂载到 pod 里\n\n经过多次尝试部署发现问题：configmap配置是可以挂载上去，但是会把目录下其他的文件删掉，而且tomcat 目录 webapps/xxxxx/下其他目录也消失了。原来是因为 tomcat 容器完全启动完成后才会解压 war包，而 configmap 配置文件是一开始就挂载上去了，导致失败。\n\n- 调整应用镜像打包过程：xxx.war 先解压后再进行应用镜像打包\n\n## 应用 gitlab CI/CD 集成\n\n- 在内部gitlab创建项目，上传应用java代码，同时在项目根目录下新加如下目录和文件，配置相应的 gitlab-runner 和 环境变量参数\n\n``` bash\n├── .app.yaml\t\t# k8s deployment 部署模板文件 \n├── config.yaml\t\t# k8s configmap 配置模板文件\n├── dockerfiles\n│ └── Dockerfile\t# Dockerfile 文件\n├── .gitlab-ci.yml\t# gitlab ci 配置文件\n└── .ns.yaml\t\t# k8s namespace 和 imagePullSecrets的配置文件\n```\n### gitlab-ci 文件摘要\n\n``` bash\nvariables:\n PROJECT_NS: '$CI_PROJECT_NAMESPACE-$CI_JOB_STAGE'\n APP_NAME: '$CI_PROJECT_NAME-$CI_COMMIT_REF_SLUG'\n\nstages:\n - package\n - beta\n\njob_package:\n stage: package\n tags:\n - package-shell\n only:\n - master\n - /^feature-.*$/\n script:\n - mvn clean install -Dmaven.test.skip=true\n - unzip target/xxxx.war -d dockerfiles/project\n - cd dockerfiles && docker build -t harbor.test.lo/project/$CI_PROJECT_NAME:$CI_PIPELINE_ID .\n - docker login -u $HARBOR_USR -p $HARBOR_PWD harbor.test.lo\n - docker push harbor.test.lo/project/$CI_PROJECT_NAME:$CI_PIPELINE_ID\n - docker logout harbor.test.lo\n\njob_push_beta:\n stage: beta\n tags:\n - beta-shell\n only:\n - master\n - /^feature-.*$/\n when: manual\n script:\n # 替换beta环境的参数配置\n - sed -i \"s/PROJECT_NS/$PROJECT_NS/g\" config.yaml .app.yaml .ns.yaml\n - sed -i \"s/TemplateProject/$APP_NAME/g\" config.yaml .app.yaml\n - sed -i \"s/DB_HOST/$BETA_DB_HOST/g\" config.yaml\n - sed -i \"s/DB_PWD/$BETA_DB_PWD/g\" config.yaml\n - sed -i \"s/APP_REP/$BETA_APP_REP/g\" .app.yaml\n - sed -i \"s/ProjectImage/$CI_PROJECT_NAME:$CI_PIPELINE_ID/g\" .app.yaml\n #\n - mkdir -p /opt/kube/$PROJECT_NS/$APP_NAME\n - cp -f .ns.yaml config.yaml .app.yaml /opt/kube/$PROJECT_NS/$APP_NAME\n - kubectl --kubeconfig=/etc/.beta/config apply -f .ns.yaml\n - kubectl --kubeconfig=/etc/.beta/config apply -f config.yaml\n - kubectl --kubeconfig=/etc/.beta/config apply -f .app.yaml\n\n# 生产部署与beta环境类同，这里省略\n```\n\n### Dockerfile 编写\n\n```\nFROM tomcat:8.5.33-jre8-alpine\n\nCOPY . /usr/local/tomcat/webapps/\n\n# 设置tomcat日志使用的时区\nRUN sed -i 's/^JAVA_OPTS=.*webresources\\\"$/JAVA_OPTS=\\\"$JAVA_OPTS -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Duser.timezone=GMT+08\\\"/g' /usr/local/tomcat/bin/catalina.sh\n```\n\n### k8s deployment 配置举例\n\n```\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: TemplateProject\n namespace: PROJECT_NS\nspec:\n replicas: APP_REP\n template:\n metadata:\n labels:\n run: TemplateProject\n spec:\n containers:\n - name: TemplateProject\n image: harbor.test.lo/project/ProjectImage\n imagePullPolicy: IfNotPresent\n ports:\n - containerPort: 8080\n volumeMounts:\n - name: db-config\n mountPath: \"/usr/local/tomcat/webapps/project/xxxx/yyyy/config/datasource.properties\"\n subPath: datasource.properties\n imagePullSecrets:\n - name: projectkey1\n volumes:\n - name: db-config\n configMap:\n name: TemplateProject-config\n defaultMode: 0640\n items:\n - path: datasource.properties\n key: datasource.properties\n\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n run: TemplateProject\n name: TemplateProject\n namespace: PROJECT_NS\nspec:\n ports:\n - port: 80\n protocol: TCP\n targetPort: 8080\n selector:\n run: TemplateProject\n sessionAffinity: None\n```\n\n### k8s configmap 配置举例\n\n```\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: TemplateProject-config\n namespace: PROJECT_NS\ndata:\n datasource.properties: |\n dataSource.maxIdle = 5\n dataSource.maxActive = 41\n dataSource.driverClassName = com.mysql.jdbc.Driver\n dataSource.url = jdbc:mysql://DB_HOST:8066/project?useUnicode=true&characterEncoding=utf-8\n dataSource.username = username\n dataSource.password = DB_PWD\n```\n"
},
{
"path": "docs/deprecated/practice/mariadb_cluster.md",
"content": "# Mariadb 数据库集群\n\nMariadb 是从 MySQL 衍生出来的开源关系型数据库，目前兼容 mysql 5.7 版本；它也非常流行，拥有 Google Facebook 等重要企业用户。本文档介绍使用 helm charts 方式安装 mariadb cluster，仅供实践交流使用。\n\n## 前提条件\n\n- 已部署 k8s 集群，参考[这里](../setup/quickStart.md)\n- 已部署 helm，参考[这里](../guide/helm.md)\n- 集群提供持久性存储，参考[这里](../setup/08-cluster-storage.md)\n\n\n## mariadb charts 配置修改\n\n按照惯例，直接把 chart 下载到本地，然后把配置复制 values.yaml 出来进行修改，这样方便以后整体更新 chart，安装实际使用需要修改配置文件\n\n``` bash\n$ cd /etc/kubeasz/manifests/mariadb-cluster\n# 编辑 my-values.yaml 修改以下部分\n\nservice:\n type: NodePort # 方便集群外部访问\n port: 3306\n nodePort:\n master: 33306 # 设置主库的nodePort\n slave: 33307 # 设置从库的nodePort\n\nrootUser: # 设置 root 密码\n password: test.c0m\n forcePassword: true\n\ndb: # 设置初始测试数据库\n user: hello\n password: hello\n name: hello\n forcePassword: true\n\nreplication: # 设置主从复制\n enabled: true\n user: replicator\n password: R4%forep11CAT0r\n forcePassword: true\n\nmaster:\n affinity: {}\n antiAffinity: soft\n tolerations: []\n persistence:\n enabled: true # 启用持久化存储\n mountPath: /bitnami/mariadb\n storageClass: \"nfs-db\" # 设置使用 nfs-db 存储类\n annotations: {}\n accessModes:\n - ReadWriteOnce\n size: 5Gi # 设置存储容量 \n\nslave:\n replicas: 1\n affinity: {}\n antiAffinity: soft\n tolerations: []\n persistence:\n enabled: false # 从库这里没有启用持久性存储\n```\n\n## 安装\n\n使用 helm 安装\n\n``` bash\n$ cd /etc/kubeasz/manifests/mariadb-cluster\n$ helm install --name mariadb --namespace default -f my-values.yaml ./mariadb\n```\n\n## 验证\n\n``` bash\n$ kubectl get pod,svc | grep mariadb\npod/mariadb-mariadb-master-0 1/1 Running 0 27m\npod/mariadb-mariadb-slave-0 1/1 Running 0 29m\n\nservice/mariadb NodePort 10.68.170.168 3306:33306/TCP 29m\nservice/mariadb-mariadb-slave NodePort 10.68.151.95 3306:33307/TCP 29m\n```\n\n"
},
{
"path": "docs/guide/argocd.md",
"content": "# argocd 安装\n\n用 GitOps 方式把 Kubernetes 声明式配置“自动、可观测、可回滚”地同步到集群的控制器；它是 Kubernetes 世界里 GitOps 的事实标准。\n\n## 初始安装\n- 建议使用helm chart 方式基础安装；后续用声明式方式配置cluster、project、repository等\n\n## 服务暴露\n- 建议使用ingress方式\n- 备用：kubectl patch svc argocd-server -n argocd -p '{\"spec\": {\"type\": \"NodePort\"}}'\n\n## 密码登录\n- 获取初始化密码 `argocd admin initial-password -n argocd`\n- 登录 `argocd login {nodeIP}:{nodePort}`\n- 更新密码 `argocd account update-password`\n- 重置遗忘密码 \n\n```\nkubectl -n argocd patch secret argocd-secret -p '{\"data\": {\"admin.password\": null, \"admin.passwordMtime\": null}}'\nkubectl -n argocd delete pods -l app.kubernetes.io/name=argocd-server\n```\n\n## SSO 登录\n- 参考文档：https://help.aliyun.com/zh/ram/obtain-user-information-through-oidc\n- 阿里云控制台-RAM访问控制-集成管理-OAuth应用：创建应用 https://ram.console.aliyun.com/applications/create\n - OAuth 协议版本：2.0\n - 应用类型：Web应用\n - 回调地址：填写 https://${argocd-server-domain}/api/dex/callback\n - OAuth 范围：openid(必选), aliuid(可选), profile(可选)\n\n- OAuth应用创建后，准备以下参数\n - \"应用 ID\" --> dex.config: connectors oidc.config.clientID\n - 创建应用密码 --> dex.config: connectors oidc.config.clientSecret\n\n- 配置argocd-cm\n\n```\n dex.config: |\n connectors:\n - type: oidc\n id: aliyun\n name: aliyun\n config:\n issuer: https://oauth.aliyun.com\n clientID: \"406************\"\n clientSecret: E8G***************************************************b6\n scopes:\n - profile\n - openid\n - aliuid\n getUserInfo: true\n userIDKey: uid\n userNameKey: uid\n claimMapping:\n preferred_username: name\n email: uid\n```\n\n- 配置argocd-rbac-cm\n\n```\ndata:\n policy.csv: |\n # 设置普通用户app-dev 只读权限\n p, role:app-dev, projects, get, *, allow\n p, role:app-dev, applications, get, *, allow\n p, role:app-dev, logs, get, *, allow\n p, role:app-dev, exec, create, */*, allow\n\n # 设置测试项目，所有权限\n p, role:app-dev, applications, *, test-project/*, allow\n \n # 阿里云子账号 ID：2***********84\n g, \"2***********84\", role:admin\n g, \"2***********27\", role:app-dev\n\n policy.default: role:''\n scopes: '[name]'\n```\n\n## 支持 application in any namespace\n\n- 配置 argocd-cm\n\n```\ndata:\n # 设置argocd 资源标记方式，使用annotation，禁用labelKey\n # application.instanceLabelKey: argocd.argoproj.io/instance\n application.resourceTrackingMethod: annotation\n```\n\n- 配置 argocd-cmd-params-cm\n\n```\ndata:\n #application.namespaces: app-team-one, app-team-two\n application.namespaces: '*'\n applicationsetcontroller.allowed.scm.providers: '*'\n applicationsetcontroller.namespaces: '*'\n```\n\n然后重启 argocd-server 和 argocd-application-controller\n\n## 其他设置\n\n- argocd 部署应用 ingress 资源一直Progressing，参考：https://github.com/argoproj/argo-cd/issues/14607\n\n```\n# 修改argocd-cm configmap，重启argocd-application-controller\ndata:\n resource.customizations: |\n networking.k8s.io/Ingress:\n health.lua: |\n hs = {}\n hs.status = \"Healthy\"\n hs.message = \"Skip health check for Ingress\"\n return hs\n```\n"
},
{
"path": "docs/guide/chrony.md",
"content": "# chrony 时间同步\n\n在安装k8s集群前需确保各节点时间同步；`chrony` 是一个优秀的 `NTP` 实现，性能比ntp好，且配置管理方便；它既可作时间服务器服务端，也可作客户端。\n\n- `OpenStack` 社区也推荐使用 `chrony`实现各节点之间的时间同步\n\n## 安装配置介绍\n\n项目中选定一个节点(`groups.chrony[0]`)作为集群内部其他节点的时间同步源，而这个节点本身从公网源同步；当然如果整个集群都无法访问公网，那么请手动校准这个节点的时间后，仍旧可以作为内部集群的时间源服务器。\n\n- 配置 chrony server，详见roles/chrony/templates/server.conf.j2 \n\n- 配置 chrony client，详见roles/chrony/templates/client.conf.j2\n\n## `kubeasz` 集成安装\n\n- 修改 clusters/${cluster_name}/hosts 文件，在 `chrony`组中加入选中的节点ip\n- [可选] 修改 clusters/${cluster_name}/config.yml 中的相关配置\n-执行命令安装 `ezctl setup ${cluster_name} 01`\n\n## 验证安装\n\n- 检查chronyd服务状态 `systemctl status chronyd`\n- 检查chronyd时间同步日志 `/var/log/chrony`\n\n## 验证时间同步状态完成\n\nchrony 服务启动后，chrony server 会与配置的公网参考时间服务器进行同步；server 同步完成后，chrony client 会与 server 进行时间同步；一般来说整个集群达到时间同步需要几十分钟。可以用如下命令检查，初始时 **NTP synchronized: no**，同步完成后 **NTP synchronized: yes**\n\n``` bash\n$ ansible -i clusters/${cluster_name}/hosts all -m shell -a 'timedatectl'\n192.168.1.1 | SUCCESS | rc=0 >>\n Local time: Sat 2019-01-26 11:51:51 HKT\n Universal time: Sat 2019-01-26 03:51:51 UTC\n RTC time: Sat 2019-01-26 03:51:52\n Time zone: Asia/Hong_Kong (HKT, +0800)\n Network time on: yes\nNTP synchronized: yes\n RTC in local TZ: no\n\n192.168.1.4 | SUCCESS | rc=0 >>\n Local time: Sat 2019-01-26 11:51:51 HKT\n Universal time: Sat 2019-01-26 03:51:51 UTC\n RTC time: Sat 2019-01-26 03:51:52\n Time zone: Asia/Hong_Kong (HKT, +0800)\n Network time on: yes\nNTP synchronized: yes\n RTC in local TZ: no\n\n192.168.1.2 | SUCCESS | rc=0 >>\n Local time: Sat 2019-01-26 11:51:51 HKT\n Universal time: Sat 2019-01-26 03:51:51 UTC\n RTC time: Sat 2019-01-26 03:51:52\n Time zone: Asia/Hong_Kong (HKT, +0800)\n Network time on: yes\nNTP synchronized: yes\n RTC in local TZ: no\n\n192.168.1.3 | SUCCESS | rc=0 >>\n Local time: Sat 2019-01-26 11:51:51 HKT\n Universal time: Sat 2019-01-26 03:51:51 UTC\n RTC time: Sat 2019-01-26 03:51:52\n Time zone: Asia/Hong_Kong (HKT, +0800)\n Network time on: yes\nNTP synchronized: yes\n RTC in local TZ: no\n```\n"
},
{
"path": "docs/guide/dashboard.1.6.3.md",
"content": "## dashboard\n\n本文档基于 dashboard 1.6.3版本，从 1.7.x 版本以后，dashboard 默认开启自带的登录验证界面，登录流程差异详见[新版本](dashboard.md)。\n\n+ 注意：实际测试k8s版本<=1.9.1支持dashboard 1.6.3, 建议k8s 1.9 以后使用 dashboard 新版本。\n\n### 部署\n\n``` bash\n# 部署dashboard 主yaml配置文件\n$ kubectl create -f /etc/kubeasz/manifests/dashboard/1.6.3/kubernetes-dashboard.yaml\n# 部署基本密码认证配置[可选]，密码文件位于 /etc/kubernetes/ssl/basic-auth.csv\n$ kubectl create -f /etc/kubeasz/manifests/dashboard/1.6.3/ui-admin-rbac.yaml\n$ kubectl create -f /etc/kubeasz/manifests/dashboard/1.6.3/ui-read-rbac.yaml\n```\n\n请在另外窗口打开 [kubernetes-dashboard.yaml](../../manifests/dashboard/1.6.3/kubernetes-dashboard.yaml)\n\n+ 由于 kube-apiserver 启用了 RBAC授权，dashboard使用的 ServiceAccount `kubernetes-dashboard` 必须有相应的权限去访问apiserver(在新版本1.8.0中，该访问权限已按最小化方式授权)，在1.6.3 版本，先粗放一点，把`kubernetes-dashboard` 与 集群角色 `cluster-admin` 绑定，这样dashboard就拥有了所有访问apiserver的权限。\n+ 开发测试环境为了方便配置dashboard-service时候，指定 `NodePort`方式暴露服务，这样集群外部可以使用 `http://NodeIP:NodePort` 方式直接访问 dashboard，生产环境建议关闭该访问途径。\n\n### 验证\n\n``` bash\n# 查看pod 运行状态\nkubectl get pod -n kube-system | grep dashboard\nkubernetes-dashboard-86bd8778bf-w4974 1/1 Running 0 12h\n# 查看dashboard service\nkubectl get svc -n kube-system|grep dashboard\nkubernetes-dashboard NodePort 10.68.7.67 80:5452/TCP\t12h\n# 查看集群服务\nkubectl cluster-info|grep dashboard\nkubernetes-dashboard is running at https://192.168.1.10:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy\n# 查看pod 运行日志，关注有没有错误\nkubectl logs kubernetes-dashboard-86bd8778bf-w4974 -n kube-system\n```\n\n### 访问\n\n因为dashboard 作为k8s 原生UI，能够展示各种资源信息，甚至可以有修改、增加、删除权限，所以有必要对访问进行认证和控制，本项目预置部署的集群有以下安全设置：详见 [apiserver配置模板](../../roles/kube-master/templates/kube-apiserver.service.j2)\n\n+ 启用 `TLS认证` `RBAC授权`等安全特性\n+ 关闭 apiserver非安全端口8080的外部访问`--insecure-bind-address=127.0.0.1`\n+ 关闭匿名认证`--anonymous-auth=false`\n+ 补充启用基本密码认证 `--token-auth-file=/etc/kubernetes/ssl/basic-auth.csv`，[密码文件模板](../../roles/kube-master/templates/basic-auth.csv.j2)中按照每行(密码,用户名,序号)的格式，可以定义多个用户\n\n#### 1. 临时访问：使用 `http://NodeIP:NodePort` 方式直接访问 dashboard，生产环境建议关闭该途径\n\n#### 2. 用户+密码访问：安全性比证书方式差点，务必保管好密码文件`basic-auth.csv`\n\n- 这里演示两种权限，使用admin 登录dashboard拥有所有权限，使用readonly 登录后仅查看权限，首先在 master节点文件 `/etc/kubernetes/ssl/basic-auth.csv` 确认用户名和密码，如果要增加或者修改用户，修改保存该文件后记得逐个重启你的master 节点\n- 为了演示用户密码访问，如果你已经完成证书访问方式，你可以在浏览器删除证书，或者访问时候浏览器询问你证书时不选证书\n- 2.1 设置用户admin 的RBAC 权限，如下运行配置文件 `kubectl create -f ui-admin-rbac.yaml`\n\n``` bash\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: ui-admin\nrules:\n- apiGroups:\n - \"\"\n resources:\n - services\n - services/proxy\n verbs:\n - '*'\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n name: ui-admin-binding\n namespace: kube-system\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: ui-admin\nsubjects:\n- apiGroup: rbac.authorization.k8s.io\n kind: User\n name: admin\n``` \n- 2.2 设置用户readonly 的RBAC 权限，如下运行配置文件 `kubectl create -f ui-read-rbac.yaml`\n\n``` bash\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: ui-read\nrules:\n- apiGroups:\n - \"\"\n resources:\n - services\n - services/proxy\n verbs:\n - get\n - list\n - watch\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n name: ui-read-binding\n namespace: kube-system\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: ui-read\nsubjects:\n- apiGroup: rbac.authorization.k8s.io\n kind: User\n name: readonly\n```\n- 2.3 访问 `https://x.x.x.x:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy` 使用 admin登录拥有所有权限，比如删除某个部署；使用 readonly登录只有查看权限，尝试删除某个部署会提示错误 `forbidden: User \\\"readonly\\\" cannot delete services/proxy in the namespace \\\"kube-system\\\"`\n\n#### 3. 证书访问：最安全的方式，配置较复杂\n- 使用集群CA 生成客户端证书，可以根据需要生成权限不同的证书，这里为了演示直接使用 kubectl使用的证书和key(在03.kubectl.yml阶段生成)，该证书拥有所有权限\n- 指定格式导出该证书，进入`/etc/kubernetes/ssl`目录，使用命令`openssl pkcs12 -export -in admin.pem -inkey admin-key.pem -out kube-admin.p12` 提示输入证书密码和确认密码，可以用密码再增加一层保护，也可以直接回车跳过，完成后目录下多了 `kube-admin.p12`文件，将它分发给授权的用户\n- 用户将 `kube-admin.p12` 双击导入证书即可，`IE` 和`Chrome` 中输入`https://x.x.x.x:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy` 或者 `https://x.x.x.x:6443/ui` 即可访问。补充：最新firefox需要在浏览器中单独导入 [选项] - [隐私与安全] - [证书/查看证书] - [您的证书] 页面点击 [导入] 该证书\n\n### 小结\n\n+ dashboard 版本 1.6.3 访问控制实现较复杂，文档中给出的例子也有助于你理解 RBAC的灵活控制能力，当然最好去[官方文档](https://kubernetes.io/docs/admin/authorization/rbac/)学习一下，这块篇幅不长\n+ 由于还未部署 Heapster 插件，当前 dashboard 不能展示 Pod、Nodes 的 CPU、内存等 metric 图形，后续部署 heapster后自然能够看到\n+ 本文中的权限设置仅供演示用，生产环境请在此基础上修改成适合你安全需求的方式\n\n"
},
{
"path": "docs/guide/dashboard.2.x.md",
"content": "## dashboard\n\n本文档基于 dashboard 2.2 版本，k8s 1.22 版本，因 dashboard 1.7 以后默认开启了自带的登录验证机制，因此不同版本登录有差异：\n\n- 旧版（<= 1.6）建议通过apiserver访问，直接通过apiserver 认证授权机制去控制 dashboard权限，详见[旧版文档](dashboard.1.6.3.md)\n- 新版（>= 1.7）可以使用自带的登录界面，使用不同Service Account Tokens 去控制访问 dashboard的权限\n\n### 部署\n\n参考 https://github.com/kubernetes/dashboard\n\n+ 增加了通过`api-server`方式访问dashboard\n+ 增加了`NodePort`方式暴露服务，这样集群外部可以使用 `https://NodeIP:NodePort` (注意是https不是http，区别于1.6.3版本) 直接访问 dashboard。\n\n安装部署\n\n``` bash\n# ezctl 集成部署组件，xxxx 代表集群部署名\n# dashboard 部署文件位于 /etc/kubeasz/clusters/xxxx/yml/dashboard/ 目录\n./ezctl setup xxxx 07\n```\n\n### 验证部署\n\n``` bash\n# 查看pod 运行状态\nkubectl get pod -n kube-system | grep dashboard\ndashboard-metrics-scraper-856586f554-l6bf4 1/1 Running 0 35m\nkubernetes-dashboard-698d4c759b-67gzg 1/1 Running 0 35m\n\n# 查看dashboard service\nkubectl get svc -n kube-system|grep dashboard\nkubernetes-dashboard NodePort 10.68.219.38 443:24108/TCP 53s\n\n# 查看pod 运行日志\nkubectl logs -n kube-system kubernetes-dashboard-698d4c759b-67gzg\n```\n\n### 登陆\n\n因为dashboard 作为k8s 原生UI，能够展示各种资源信息，甚至可以有修改、增加、删除权限，所以有必要对访问进行认证和控制，为演示方便这里使用 `https://NodeIP:NodePort` 方式访问 dashboard，支持两种登录方式：Kubeconfig、令牌(Token)\n\n**注意：** 使用chrome浏览器访问 `https://NodeIP:NodePort` 可能提示安全风险无法访问，可以换firefox浏览器设置安全例外，继续访问。 \n\n- Token令牌方式登录（admin）\n\n选择 Token 方式登录，复制下面输出的admin token 字段到输入框\n\n``` bash\n# 获取 Bearer Token，找到输出中 ‘token:’ 开头的后面部分\n$ kubectl describe -n kube-system secrets admin-user \n```\n\n- Token令牌方式登录（只读）\n\n选择 Token 方式登录，复制下面输出的read token 字段到输入框\n\n``` bash\n# 获取 Bearer Token，找到输出中 ‘token:’ 开头的后面部分\n$ kubectl describe -n kube-system secrets dashboard-read-user \n```\n\n- Kubeconfig登录（admin）\nAdmin kubeconfig文件默认位置：`/root/.kube/config`，该文件中默认没有token字段，使用Kubeconfig方式登录，还需要将token追加到该文件中，完整的文件格式如下：\n```\napiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: LS0tLS1CRUdxxxxxxxxxxxxxx\n server: https://192.168.1.2:6443\n name: kubernetes\ncontexts:\n- context:\n cluster: kubernetes\n user: admin\n name: kubernetes\ncurrent-context: kubernetes\nkind: Config\npreferences: {}\nusers:\n- name: admin\n user:\n client-certificate-data: LS0tLS1CRUdJTiBDRxxxxxxxxxxx\n client-key-data: LS0tLS1CRUdJTxxxxxxxxxxxxxx\n token: eyJhbGcixxxxxxxxxxxxxxxx\n```\n\n- Kubeconfig登录（只读）\n首先[创建只读权限 kubeconfig文件](../op/kcfg-adm.md)，然后类似追加只读token到该文件，略。\n\n### 参考\n\n- 1.[Dashboard docs](https://github.com/kubernetes/dashboard/blob/master/docs/README.md)\n- 2.[a-read-only-kubernetes-dashboard](https://blog.cowger.us/2018/07/03/a-read-only-kubernetes-dashboard.html)\n"
},
{
"path": "docs/guide/dashboard.md",
"content": "## dashboard\n\n本文档基于 dashboard 7.12.0 版本，k8s 1.32 版本，dashboard 7.0.0 以后引入大量不兼容变化。\n\n### 部署\n\n假设已经使用kubeasz 部署k8s集群完成；新版dashboard 部署如下：（以单机集群为例，其他情况请修改集群名称'default'为实际的名称）\n\n``` bash\n# 1. 修改 clusters/default/config.yml 文件，设置 dashboard_install: \"yes\"\n\n# 2. 下载dashboard 需要的镜像\n./ezdown -X dashboard\n\n# 3. 执行安装，配置文件位于 clusters/default/yml/dashboard/ 目录\ndk ezctl setup default 07\n```\n\n+ 增加`NodePort`方式暴露服务，这样集群外部可以使用 `https://NodeIP:NodePort` (注意是https不是http) 直接访问 dashboard。\n\n### 验证\n\n``` bash\n# 查看pod 运行状态\nkubectl get pod -n kube-system |grep kubernetes-dashboard\nkubernetes-dashboard-api-6d77cb7964-4tklq 1/1 Running 0 17h\nkubernetes-dashboard-auth-5fbd64f659-f9dst 1/1 Running 0 17h\nkubernetes-dashboard-kong-6dcdbf5dfd-829h4 1/1 Running 0 17h\nkubernetes-dashboard-metrics-scraper-7757c48476-4lcrq 1/1 Running 0 17h\nkubernetes-dashboard-web-5f9f47979-7khrk 1/1 Running 0 17h\n\n# 查看service\nkubectl get svc -n kube-system |grep kong\nkubernetes-dashboard-kong-proxy NodePort 10.68.148.170 443:31544/TCP 17h\n```\n\n### 登陆\n\n因为dashboard 作为k8s 原生UI，能够展示各种资源信息，甚至可以有修改、增加、删除权限，所以有必要对访问进行认证和控制，为演示方便这里使用 `https://NodeIP:NodePort` 方式访问 dashboard，目前支持登录方式：令牌(Token)\n\n**注意：** 使用chrome浏览器访问 `https://NodeIP:NodePort` 可能提示安全风险无法访问，可以换firefox浏览器设置安全例外，继续访问。\n\n- Token令牌方式登录（admin）\n\n选择 Token 方式登录，复制下面输出的admin token 字段到输入框\n\n``` bash\n# 获取 Bearer Token，找到输出中 ‘token:’ 开头的后面部分\n$ kubectl describe -n kube-system secrets admin-user \n```\n\n- Token令牌方式登录（只读）\n\n选择 Token 方式登录，复制下面输出的read token 字段到输入框\n\n``` bash\n# 获取 Bearer Token，找到输出中 ‘token:’ 开头的后面部分\n$ kubectl describe -n kube-system secrets dashboard-read-user \n```\n\n### 参考\n\n- [旧版文档 dashboard 1.6.3](dashboard.1.6.3.md)\n- [旧版文档 dashboard 2.x](dashboard.2.x.md)\n- https://github.com/kubernetes/dashboard\n"
},
{
"path": "docs/guide/harbor.md",
"content": "# harbor 镜像仓库\n\nHabor是由VMWare中国团队开源的企业级容器镜像仓库。特性包括：友好的用户界面，基于角色的访问控制，水平扩展，同步复制，AD/LDAP集成以及审计日志等。本文档仅说明单机安装harbor 服务。\n\n- 目录\n - 安装步骤\n - 安装讲解\n - 配置docker/containerd信任harbor证书\n - 在k8s集群使用harbor\n - 管理维护\n\n### 安装步骤\n\n1. 下载离线安装包，成功后在/etc/kubeasz/down/目录下有离线包harbor-offline-installer-$HARBOR_VER.tgz\n\n```\nezdown -D\nezdown -R\n```\n\n2. 利用ezctl [文档](../setup/ezctl.md) 创建一个新的集群，已有集群修改同样的文件\n\n```\n#clusters/xxx/hosts 中修改如下，配置harbor组下机器，设置NEW_INSTALL=true\n...\n# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one\n[harbor]\n192.168.1.8 NEW_INSTALL=true\n...\n\n#clusters/xxx/config.yml 中修改如下，按需修改HARBOR_DOMAIN/HARBOR_TLS_PORT 等配置项，举例如下\n############################\n# role:harbor\n############################\n# harbor version，完整版本号\nHARBOR_VER: \"v2.10.2\"\nHARBOR_DOMAIN: \"harbor.yourdomain.com\"\nHARBOR_PATH: /var/data\nHARBOR_TLS_PORT: 8443\nHARBOR_REGISTRY: \"{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}\"\n\n# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'\nHARBOR_SELF_SIGNED_CERT: true\n\n# install component\nHARBOR_WITH_TRIVY: false\n```\n\n3. 配置完成后，执行 `./ezctl setup xxx harbor`，完成harbor安装和docker 客户端配置\n\n- 安装验证\n\n1. 在harbor节点使用`docker ps -a` 查看harbor容器组件运行情况\n2. 浏览器访问地址（忽略证书报错） `https://${HARBOR_DOMAIN}:${HARBOR_TLS_PORT}`，管理员账号是 admin ，密码见harbor.yml文件 harbor_admin_password 对应值（默认密码 Harbor12345 已被随机生成的16位随机密码替换，不然存在安全隐患)\n\n### 安装讲解\n\n根据`playbooks/11.harbor.yml`文件，harbor节点需要以下步骤：\n\n- role `os-harden` 系统安全加固（可选）\n- role `chrony` 时间同步服务（可选）\n- role `prepare` 基础系统环境准备\n- role `docker` 安装docker\n- role `harbor` 安装harbor\n- 注意：`kube_node`节点在harbor部署完之后，需要配置harbor的证书（详见下节配置docker/containerd信任harbor证书），并可以在hosts里面添加harbor的域名解析，如果你的环境中有dns服务器，可以跳过hosts文件设置\n\n1. 下载docker-compose可执行文件到$PATH目录\n1. 自注册变量result判断是否已经安装harbor，避免重复安装问题\n1. 解压harbor离线安装包到指定目录\n1. 导入harbor所需 docker images\n1. 创建harbor证书和私钥(复用集群的CA证书)\n1. 修改harbor.yml配置文件\n1. 启动harbor安装脚本\n\n### 在k8s集群使用harbor\n\nadmin用户web登录后可以方便的创建项目，并指定项目属性(公开或者私有)；然后创建用户，并在项目`成员`选项中选择用户和权限；\n\n#### 镜像上传\n\n使用docker客户端登录`{{ HARBOR_REGISTRY }}`，然后把镜像tag成 `{{ HARBOR_REGISTRY }}/$项目名/$镜像名:$TAG` 之后，即可使用docker push 上传\n\n``` bash\ndocker login harbor.test.com\nUsername: \nPassword:\nLogin Succeeded\ndocker tag busybox:latest harbor.test.com/library/busybox:latest\ndocker push harbor.test.com/library/busybox:latest\nThe push refers to a repository [harbor.test.com/library/busybox]\n0271b8eebde3: Pushed \nlatest: digest: sha256:91ef6c1c52b166be02645b8efee30d1ee65362024f7da41c404681561734c465 size: 527\n```\n#### k8s中使用harbor\n\n1. 如果镜像保存在harbor中的公开项目中，那么只需要在yaml文件中简单指定harbor私有镜像即可，例如\n\n``` bash\napiVersion: v1\nkind: Pod\nmetadata:\n name: test-busybox\nspec:\n containers:\n - name: test-busybox\n image: harbor.test.com/xxx/busybox:latest\n imagePullPolicy: Always\n```\n\n2. 如果镜像保存在harbor中的私有项目中，那么yaml文件中使用该私有项目的镜像需要指定`imagePullSecrets`，例如\n\n``` bash\napiVersion: v1\nkind: Pod\nmetadata:\n name: test-busybox\nspec:\n containers:\n - name: test-busybox\n image: harbor.test.com/xxx/busybox:latest\n imagePullPolicy: Always\n imagePullSecrets:\n - name: harborkey1\n```\n其中 `harborKey1`可以用以下两种方式生成：\n\n+ 1.使用 `kubectl create secret docker-registry harborkey1 --docker-server=harbor.test.com --docker-username=admin --docker-password=Harbor12345 --docker-email=team@test.com`\n+ 2.使用yaml配置文件生成 \n\n``` bash\n//harborkey1.yaml\napiVersion: v1\nkind: Secret\nmetadata:\n name: harborkey1\n namespace: default\ndata:\n .dockerconfigjson: {base64 -w 0 ~/.docker/config.json}\ntype: kubernetes.io/dockerconfigjson\n```\n前面docker login会在~/.docker下面创建一个config.json文件保存鉴权串，这里secret yaml的.dockerconfigjson后面的数据就是那个json文件的base64编码输出（-w 0让base64输出在单行上，避免折行）\n\n### 管理维护\n\n+ 日志目录 `/var/log/harbor`\n+ 数据目录 `/var/data` ，其中最主要是 `/var/data/database` 和 `/var/data/registry` 目录，如果你要彻底重新安装harbor，删除这两个目录即可\n\n先进入harbor安装目录 `cd /var/data/harbor`，常规操作如下：\n\n1. 暂停harbor `docker-compose stop` : docker容器stop，并不删除容器\n2. 恢复harbor `docker-compose start` : 恢复docker容器运行\n3. 停止harbor `docker-compose down -v` : 停止并删除docker容器\n4. 启动harbor `docker-compose up -d` : 启动所有docker容器\n\n修改harbor的运行配置，需要如下步骤：\n\n``` bash\n# 停止 harbor\n docker-compose down -v\n# 修改配置\n vim harbor.yml\n# 执行./prepare已更新配置到docker-compose.yml文件\n ./prepare\n# 启动 harbor\n docker-compose up -d\n```\n"
},
{
"path": "docs/guide/helm.md",
"content": "# Helm\n\n`Helm`致力于成为k8s集群的应用包管理工具，希望像linux 系统的`RPM` `DPKG`那样成功；确实在k8s上部署复杂一点的应用很麻烦，需要管理很多yaml文件（configmap,controller,service,rbac,pv,pvc等等），而helm能够整齐管理这些文档：版本控制，参数化安装，方便的打包与分享等。 \n- 建议积累一定k8s经验以后再去使用helm；对于初学者来说手工去配置那些yaml文件对于快速学习k8s的设计理念和运行原理非常有帮助，而不是直接去使用helm，面对又一层封装与复杂度。\n- 本文基于helm 3（建议版本），helm 2 文档[请看这里](helm2.md)\n\n## 安装 helm\n\n在官方repo下载[release版本](https://github.com/helm/helm/releases)中自带的二进制文件即可（以Linux amd64为例）\n\n```\nwget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz\nmv ./linux-amd64/helm /usr/bin\n```\n\n- 启用官方 charts 仓库\n\n```\nhelm repo add stable https://kubernetes-charts.storage.googleapis.com/\n```\n国内镜像\n```\n helm repo add stable http://mirror.azure.cn/kubernetes/charts\n```\n\n\n## 使用 helm 安装应用\n\nhelm3 安装命令与 helm2 稍有变化，个人习惯先下载对应charts到本地然后按照固定目录格式安装，以创建一个redis集群举例：\n\n- 创建 redis-cluster 目录\n``` bash\nmkdir -p /opt/charts/redis-cluster\ncd /opt/charts/redis-cluster\n```\n\n- 下载最新stalbe/redis-ha\n```\nhelm repo update\nhelm pull stable/redis-ha\n```\n\n- 解压 charts，复制 values.yaml设置\n```\ntar zxvf redis-ha-*.tgz\ncp redis-ha/values.yaml .\n```\n\n- 创建 start.sh 脚本记录启动命令\n```\ncat > start.sh << EOF\n#!/bin/sh\nset -x\n\nROOT=$(cd `dirname $0`; pwd)\ncd $ROOT\n\nhelm install redis \\\n\t--create-namespace \\\n\t--namespace dependency \\\n\t-f ./values.yaml \\\n\t./redis-ha\nEOF\n```\n\n- 查看当前目录结构如下\n```\ntree .\n.\n├── redis-ha\t\t# redis-ha 原始charts目录\n├── start.sh\t\t# 启动命名脚本\n└── values.yaml\t\t# 个性化参数配置\n```\n\n- 修改当前目录的 values.yaml 为你的个性化配置\n``` bash\n#举例values.yaml 配置如下，没有启用PV\n#cat values.yaml\nimage:\n repository: redis\n tag: 5.0.6-alpine\n\nreplicas: 2\n\n## Redis specific configuration options\nredis:\n port: 6379\n masterGroupName: \"mymaster\" # must match ^[\\\\w-\\\\.]+$) and can be templated\n config:\n ## For all available options see http://download.redis.io/redis-stable/redis.conf\n min-replicas-to-write: 1\n min-replicas-max-lag: 5 # Value in seconds\n maxmemory: \"4g\" # Max memory to use for each redis instance. Default is unlimited.\n maxmemory-policy: \"allkeys-lru\" # Max memory policy to use for each redis instance. Default is volatile-lru.\n repl-diskless-sync: \"yes\"\n rdbcompression: \"yes\"\n rdbchecksum: \"yes\"\n\n resources:\n requests:\n memory: 200Mi\n cpu: 100m\n limits:\n memory: 4000Mi\n\n## Sentinel specific configuration options\nsentinel:\n port: 26379\n quorum: 1\n\n resources:\n requests:\n memory: 200Mi\n cpu: 100m\n limits:\n memory: 200Mi\n\nhardAntiAffinity: true\n\n## Configures redis with AUTH (requirepass & masterauth conf params)\nauth: false\n\npersistentVolume:\n enabled: false\n\nhostPath:\n path: \"/data/mcs-redis/{{ .Release.Name }}\"\n```\n\n- 执行安装\n```\nbash ./start.sh\n```\n\n- 查看安装\n```\nhelm ls -A\nNAME \tNAMESPACE \tREVISION\tUPDATED \tSTATUS \tCHART \tAPP VERSION\nredis\tdependency\t1 \t2020-05-28 20:57:31.166002853 +0800 CST\tdeployed\tredis-ha-4.4.4\t5.0.6\n\n# 查看k8s上资源\nkubectl get pod,svc -n dependency\nNAME READY STATUS RESTARTS AGE\npod/redis-redis-ha-server-0 2/2 Running 0 119s\npod/redis-redis-ha-server-1 2/2 Running 0 104s\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/redis-redis-ha ClusterIP None 6379/TCP,26379/TCP 119s\nservice/redis-redis-ha-announce-0 ClusterIP 10.68.41.65 6379/TCP,26379/TCP 119s\nservice/redis-redis-ha-announce-1 ClusterIP 10.68.64.49 6379/TCP,26379/TCP 119s\n```\n\n"
},
{
"path": "docs/guide/hpa.md",
"content": "## Horizontal Pod Autoscaling\n\n自动水平伸缩，是指运行在k8s上的应用负载(POD)，可以根据资源使用率进行自动扩容、缩容；我们知道应用的资源使用率通常都有高峰和低谷，所以k8s的`HPA`特性应运而生；它也是最能体现区别于传统运维的优势之一，不仅能够弹性伸缩，而且完全自动化！\n\n根据 CPU 使用率或自定义 metrics 自动扩展 Pod 数量（支持 replication controller、deployment）；k8s1.6版本之前是通过kubelet来获取监控指标，1.6版本之后是通过api server、heapster或者kube-aggregator来获取监控指标。\n\n### Metrics支持\n\n根据不同版本的API中，HPA autoscale时靠以下指标来判断资源使用率：\n- autoscaling/v1: CPU\n- autoscaling/v2alpha1\n - 内存\n - 自定义metrics\n - 多metrics组合: 根据每个metric的值计算出scale的值，并将最大的那个值作为扩容的最终结果\n\n### 基础示例\n\n本实验环境基于k8s 1.8 和 1.9，仅使用`autoscaling/v1` 版本API，**注意确保**`k8s` 集群插件`kubedns` 和 `heapster` 工作正常。\n\n``` bash\n# 创建deploy和service\n$ kubectl run php-apache --image=pilchard/hpa-example --requests=cpu=200m --expose --port=80\n\n# 创建autoscaler\n$ kubectl autoscale deploy php-apache --cpu-percent=50 --min=1 --max=10\n\n# 等待3~5分钟查看hpa状态\n$ kubectl get hpa php-apache\nNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE\nphp-apache Deployment/php-apache 0% / 50% 1 10 1 3m\n\n# 增加负载\n$ kubectl run --rm -it load-generator --image=busybox /bin/sh\nHit enter for command prompt\n$ while true; do wget -q -O- http://php-apache; done;\n\n# 等待约5分钟查看hpa显示负载增加，且副本数目增加为4\n$ kubectl get hpa php-apache\nNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE\nphp-apache Deployment/php-apache 430% / 50% 1 10 4 4m\n\n# 注意k8s为了避免频繁增删pod，对副本的增加速度有限制\n# 实验过程可以看到副本数目从1到4到8到10，大概都需要4~5分钟的缓冲期\n$ kubectl get hpa php-apache\nNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE\nphp-apache Deployment/php-apache 86% / 50% 1 10 8 9m\n$ kubectl get hpa php-apache\nNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE\nphp-apache Deployment/php-apache 52% / 50% 1 10 10 12m\n\n# 清除负载，CTRL+C 结束上述循环程序，稍后副本数目变回1\n$ kubectl get hpa php-apache\nNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE\nphp-apache Deployment/php-apache 0% / 50% 1 10 1 17m\n```\n\n"
},
{
"path": "docs/guide/index.md",
"content": "## 使用指南\n\n### 附加组件安装\n\n- 安装 [kubedns](kubedns.md)\n- 安装 [dashboard](dashboard.md)\n- 安装 [metrics-server](metrics-server.md)\n- 安装 [prometheus](prometheus.md)\n- 安装 [kubeapps](kubeapps.md)\n- 安装 [ingress](ingress.md)\n- 安装 [helm](helm.md)\n- 安装 [efk](efk.md)\n- 安装 [harbor](harbor.md)\n- 安装 [metallb](metallb.md)\n\n### 基础特性演示\n\n- 自动水平伸缩 [Horizontal Pod Autoscaling](hpa.md)\n- 网络安全策略 [Network Policy](networkpolicy.md)\n- 滚动更新 [rollingupdate](rollingupdateWithZeroDowntime.md)\n\n\n"
},
{
"path": "docs/guide/ingress-tls.md",
"content": "# 使用 traefik 配置 https ingress\n\n本文档已过期，安装最新版本，请参考相关官方文档。\n\n本文档基于 traefik 配置 https ingress 规则，请先阅读[配置基本 ingress](ingress.md)。与基本 ingress-controller 相比，需要额外配置 https tls 证书，主要步骤如下：\n\n## 1.准备 tls 证书\n\n可以使用Let's Encrypt签发的免费证书，这里为了测试方便使用自签证书 (tls.key/tls.crt)，注意CN 配置为 ingress 的域名：\n\n``` bash\n$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj \"/CN=hello.test.com\"\n```\n\n## 2.在 kube-system 命名空间创建 secret: traefik-cert，以便后面 traefik-controller 挂载该证书\n\n``` bash\n$ kubectl -n kube-system create secret tls traefik-cert --key=tls.key --cert=tls.crt\n```\n\n## 3.创建 traefik-controller，增加 traefik.toml 配置文件及https 端口暴露等，详见该 yaml 文件\n\n``` bash\n$ kubectl apply -f /etc/kubeasz/manifests/ingress/traefik/tls/traefik-controller.yaml\n```\n\n## 4.创建 https ingress 例子\n\n``` bash\n# 创建示例应用\n$ kubectl run test-hello --image=nginx:alpine --port=80 --expose\n# hello-tls-ingress 示例\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: hello-tls-ingress\n annotations:\n kubernetes.io/ingress.class: traefik\nspec:\n rules:\n - host: hello.test.com\n http:\n paths:\n - backend:\n serviceName: test-hello\n servicePort: 80\n tls:\n - secretName: traefik-cert\n# 创建https ingress\n$ kubectl apply -f /etc/kubeasz/manifests/ingress/traefik/tls/hello-tls.ing.yaml\n# 注意根据hello示例，需要在default命名空间创建对应的secret: traefik-cert\n$ kubectl create secret tls traefik-cert --key=tls.key --cert=tls.crt\n```\n\n## 5.验证 https 访问\n\n验证 traefik-ingress svc\n\n``` bash\n$ kubectl get svc -n kube-system traefik-ingress-service \nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ntraefik-ingress-service NodePort 10.68.250.253 80:23456/TCP,443:23457/TCP,8080:35941/TCP 66m\n```\n\n可以看到项目默认使用nodePort 23456暴露traefik 80端口，nodePort 23457暴露 traefik 443端口，因此在客户端 hosts 增加记录 `$Node_IP hello.test.com`之后，可以在浏览器验证访问如下：\n\n``` bash\nhttps://hello.test.com:23457\n```\n\n如果你已经配置了[转发 ingress nodePort](../op/loadballance_ingress_nodeport.md)，那么增加对应 hosts记录后，可以验证访问 `https://hello.test.com`\n\n## 配置 dashboard ingress\n\n前提1：k8s 集群的dashboard 已安装\n\n```\n$ kubectl get svc -n kube-system | grep dashboard\nkubernetes-dashboard NodePort 10.68.211.168 443:39308/TCP\t3d11h\n```\n前提2：`/etc/kubeasz/manifests/ingress/traefik/tls/traefik-controller.yaml`的配置文件`traefik.toml`开启了`insecureSkipVerify = true`\n\n配置 dashboard ingress：`kubectl apply -f /etc/kubeasz/manifests/ingress/traefik/tls/k8s-dashboard.ing.yaml` 内容如下：\n\n```\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: kubernetes-dashboard\n namespace: kube-system\n annotations:\n traefik.ingress.kubernetes.io/redirect-entry-point: https\nspec:\n rules:\n - host: dashboard.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: kubernetes-dashboard\n servicePort: 443\n```\n- 注意annotations 配置了 http 跳转 https 功能\n- 注意后端服务是443端口\n\n## 参考\n\n- [Add a TLS Certificate to the Ingress](https://docs.traefik.io/user-guide/kubernetes/#add-a-tls-certificate-to-the-ingress)\n"
},
{
"path": "docs/guide/ingress.md",
"content": "## Ingress简介\n\n本文档已过期，安装最新版本，请参考相关官方文档。\n\ningress就是从外部访问k8s集群的入口，将用户的URL请求转发到不同的service上。ingress相当于nginx反向代理服务器，它包括的规则定义就是URL的路由信息；它的实现需要部署`Ingress controller`(比如 [traefik](https://github.com/containous/traefik) [ingress-nginx](https://github.com/kubernetes/ingress-nginx) 等)，`Ingress controller`通过apiserver监听ingress和service的变化，并根据规则配置负载均衡并提供访问入口，达到服务发现的作用。\n\n- 未配置ingress：\n\n集群外部 -> NodePort -> K8S Service\n\n- 配置ingress:\n\n集群外部 -> Ingress -> K8S Service\n\n- **注意：ingress 本身也需要部署`Ingress controller`时使用以下几种方式让外部访问**\n - 使用`NodePort`方式\n - 使用`hostPort`方式\n - 使用LoadBalancer地址方式\n\n- 以下讲解基于`Traefik`，如果想要了解`ingress-nginx`的原理与实践，推荐阅读博客[烂泥行天下](https://www.ilanni.com/?p=14501)的相关文章\n\n### 部署 Traefik\n\nTraefik 提供了一个简单好用 `Ingress controller`，下文侧重讲解 ingress部署和测试例子。请查看yaml配置 [traefik-ingress.yaml](../../manifests/ingress/traefik/traefik-ingress.yaml)，参考[traefik 官方k8s例子](https://github.com/containous/traefik/tree/master/examples/k8s)\n\n#### 安装 traefik ingress-controller\n\n``` bash\nkubectl create -f /etc/kubeasz/manifests/ingress/traefik/traefik-ingress.yaml\n```\n+ 注意需要配置 `RBAC`授权\n+ 注意`trafik pod`中 `80`端口为 traefik ingress-controller的服务端口，`8080`端口为 traefik 的管理WEB界面；为后续配置方便指定`80` 端口暴露`NodePort`端口为 `23456`(对应于在hosts配置中`NODE_PORT_RANGE`范围内可用端口)\n\n#### 验证 traefik ingress-controller\n\n``` bash\n# kubectl get deploy -n kube-system traefik-ingress-controller\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\ntraefik-ingress-controller 1 1 1 1 4m\n\n# kubectl get svc -n kube-system traefik-ingress-service\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ntraefik-ingress-service NodePort 10.68.69.170 80:23456/TCP,8080:34815/TCP 4m\n```\n+ 可以看到`traefik-ingress-service` 服务端口`80`暴露的nodePort确实为`23456`\n\n#### 测试 ingress\n\n+ 首先创建测试用K8S应用，并且该应用服务不用nodePort暴露，而是用ingress方式让外部访问\n\n``` bash\nkubectl run test-hello --image=nginx:alpine --expose --port=80\n##\n# kubectl get deploy test-hello\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\ntest-hello 1 1 1 1 56s\n# kubectl get svc test-hello\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ntest-hello ClusterIP 10.68.124.115 80/TCP 1m\n```\n+ 然后为这个应用创建 ingress，`kubectl create -f /etc/kubeasz/manifests/ingress/test-hello.ing.yaml`\n\n``` bash\n# test-hello.ing.yaml内容\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: test-hello\nspec:\n rules:\n - host: hello.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: test-hello\n servicePort: 80\n```\n+ 集群内部尝试访问: `curl -H Host:hello.test.com 10.68.69.170(traefik-ingress-service的服务地址)` 能够看到欢迎页面 `Welcome to nginx!`；\n+ 在集群外部尝试访问(假定集群一个NodeIP为 192.168.1.1): `curl -H Host:hello.test.com 192.168.1.1:23456`，也能够看到欢迎页面 `Welcome to nginx!`，说明ingress测试成功\n\n#### 为 traefik WEB 管理页面创建 ingress 规则 \n\n`kubectl create -f /etc/kubeasz/manifests/ingress/traefik/traefik-ui.ing.yaml`\n\n``` bash\n# traefik-ui.ing.yaml内容\n---\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: traefik-web-ui\n namespace: kube-system\nspec:\n rules:\n - host: traefik-ui.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: traefik-ingress-service\n servicePort: 8080\n```\n\n+ 在集群外部可以使用 `curl -H Host:traefik-ui.test.com 192.168.1.1:23456` 尝试访问WEB管理页面，返回 `Found .`说明 traefik-ui的ingress配置生效了。\n\n+ 在客户端主机也可以通过修改本机 `hosts` 文件，如上例子，增加两条记录：\n\n``` text\n192.168.1.1\thello.test.com\n192.168.1.1\ttraefik-ui.test.com\n```\n打开浏览器输入域名 `http://hello.test.com:23456` 和 `http://traefik-ui.test.com:23456` 就可以访问k8s的应用服务了。\n\n### 可选1: 使用`LoadBalancer`服务类型来暴露ingress，自有环境（非公有云）可以参考[metallb文档](metallb.md)\n\n``` bash\n# 修改traefik-ingress 使用 LoadBalancer服务\n$ sed -i 's/NodePort$/LoadBalancer/g' /etc/kubeasz/manifests/ingress/traefik/traefik-ingress.yaml\n# 创建traefik-ingress\n$ kubectl apply -f /etc/kubeasz/manifests/ingress/traefik/traefik-ingress.yaml\n# 验证\n$ kubectl get svc --all-namespaces |grep traefik\nkube-system traefik-ingress-service LoadBalancer 10.68.163.243 192.168.1.241 80:23456/TCP,8080:37088/TCP 1m\n```\n这时可以修改客户端本机 `hosts`文件：(如上例192.168.1.241)\n\n``` text\n192.168.1.241 hello.test.com\n192.168.1.241 traefik-ui.test.com\n```\n打开浏览器输入域名 `http://hello.test.com` 和 `http://traefik-ui.test.com`可以正常访问。\n\n### 可选2: 部署`ingress-service`的负载均衡\n\n- 利用 nginx/haproxy 等集群，可以做代理转发以去掉 `23456`这个端口。如果你的集群根据本项目部署了高可用方案，那么可以利用`LB` 节点haproxy 来做，当然如果生产环境K8S应用已经部署非常多，建议还是使用独立的 `nginx/haproxy`集群。\n\n具体参考[配置转发 ingress nodePort](../op/loadballance_ingress_nodeport.md)，如上配置访问集群`MASTER_IP`的`80`端口时，由haproxy代理转发到实际的node节点暴露的nodePort端口上了。这时可以修改客户端本机 `hosts`文件如下：(假定 MASTER_IP=192.168.1.10)\n\n``` text\n192.168.1.10 hello.test.com\n192.168.1.10 traefik-ui.test.com\n```\n打开浏览器输入域名 `http://hello.test.com` 和 `http://traefik-ui.test.com`可以正常访问。\n\n## 下一步[配置https ingress](ingress-tls.md)\n"
},
{
"path": "docs/guide/ipvs.md",
"content": "# IPVS 服务负载均衡\n\nkube-proxy 组件监听 API server 中 service 和 endpoint 的变化情况，从而为 k8s 集群内部的 service 提供动态负载均衡。在v1.10之前主要通过 iptables来实现，是稳定、推荐的方式，但是当服务多的时候会产生太多的 iptables 规则，大规模情况下有明显的性能问题；在v1.11 GA的 ipvs高性能负载模式，采用增量式更新，并可以保证 service 更新期间连接的保持。\n\n- NOTE: k8s v1.11.0 CentOS7下使用ipvs模式会有问题（见 kubernetes/kubernetes#65461），测试 k8s v1.10.2 CentOS7 可以。\n\n## 启用 ipvs\n\n建议 k8s 版本1.13 及以后启用 ipvs，只要在 kube-proxy 启动参数（或者配置文件中）中增加 `--proxy-mode=ipvs`:\n\n``` bash\n[Unit]\nDescription=Kubernetes Kube-Proxy Server\nAfter=network.target\n\n[Service]\nWorkingDirectory=/var/lib/kube-proxy\nExecStart={{ bin_dir }}/kube-proxy \\\n --bind-address={{ NODE_IP }} \\\n --hostname-override={{ NODE_IP }} \\\n --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\\n --logtostderr=true \\\n --proxy-mode=ipvs\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\n```\n"
},
{
"path": "docs/guide/istio.md",
"content": "---\ntitle: \"Istio 1.1.7 安装 \"\ndate: 2019-05-19T19:44:00+08:00\n---\n\n#### Service Mesh(服务网格) \n\n--- \nKubernetes 已经给我们带来了诸多的好处。但是仍有些需求比如 A/B 测试、金丝雀发布、限流、访问控制,端到端认证等需要运维人员进一步去解决。\n\nIstio 是完全开源的服务网格,提供了一套完整的解决方案，可以透明地分层到现有的分布式应用程序上。对开发人员几乎无感的同时获得超能力。\n\n如果想要现有的服务支持 Istio，只需要在当前的环境中部署一个特殊的 sidecar 代理，即可。\n\n##### 前提 \n\n---- \n\n- 安装 Kubernetes 集群 1.9+ \n- [安装 Helm](./helm.md) \n\n##### 准备\n\n---- \n\n进入 [Istio release](https://github.com/istio/istio/releases) 页面下载最新版安装包并解压到当前目录,\n\n```sh\ncurl -L https://git.io/getLatestIstio | sh -\n\n\nll istio-1.1.7/\ntotal 40\ndrwxr-xr-x 2 root root 4096 May 15 08:59 bin\ndrwxr-xr-x 6 root root 4096 May 15 08:59 install\n-rw-r--r-- 1 root root 602 May 15 08:59 istio.VERSION\n-rw-r--r-- 1 root root 11343 May 15 08:59 LICENSE\n-rw-r--r-- 1 root root 5921 May 15 08:59 README.md\ndrwxr-xr-x 15 root root 4096 May 15 08:59 samples\ndrwxr-xr-x 7 root root 4096 May 15 08:59 tools\n```\n- install Kubernetes 安装所需的 .yaml 文件\n- samples Task中的示例应用\n- bin/istioctl 客户端工具\n- istio.VERSION 配置文件\n\n#### 安装 \n---- \n\n注意事项\n\n- Node 节点内存不能低于 4G，否则相关容器可能启动失败 \n- Istio 默认使用‘负载均衡器’服务对象类型。对于裸机安装没有负载均衡器的情况下，安装需指定‘NodePort’类型。\n\n\n##### 方案1：使用 Helm template 进行安装\n\n```bash\ncd /usr/local/src/istio-1.1.7\n\nkubectl create namespace istio-system\n\n# 安装 istio-init chart，来启动 Istio CRD 的安装过程\nhelm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system --set gateways.istio-ingressgateway.type=NodePort --set gateways.istio-egressgateway.type=NodePort | kubectl apply -f -\n\n# 稍等一会儿执行\n# 输出 23 或者 28 （若开启了 cert-manager）\nkubectl get crds | grep 'istio.io\\|certmanager.k8s.io' | wc -l\n\n# 部署与你选择的配置文件相对应的 Istio 的核心组件\n# 不同配置说明 https://istio.io/zh/docs/setup/kubernetes/additional-setup/config-profiles/\n\n# 选择 default 配置\nhelm template install/kubernetes/helm/istio --name istio --namespace istio-system \\\n --set gateways.istio-ingressgateway.type=NodePort \\\n --set gateways.istio-egressgateway.type=NodePort | kubectl apply -f -\n\n# 或选择 demo 配置\nhelm template install/kubernetes/helm/istio --name istio --namespace istio-system \\\n --set gateways.istio-ingressgateway.type=NodePort \\\n --set gateways.istio-egressgateway.type=NodePort \\\n --values install/kubernetes/helm/istio/values-istio-demo.yaml | kubectl apply -f -\n```\n\n##### 方案2：在 Helm 和 Tiller 的环境中使用 helm install 命令进行安装\n\n见[官方文档](https://istio.io/zh/docs/setup/kubernetes/install/helm/#%E6%96%B9%E6%A1%88-2-%E5%9C%A8-helm-%E5%92%8C-tiller-%E7%9A%84%E7%8E%AF%E5%A2%83%E4%B8%AD%E4%BD%BF%E7%94%A8-helm-install-%E5%91%BD%E4%BB%A4%E8%BF%9B%E8%A1%8C%E5%AE%89%E8%A3%85)\n\n\n##### 验证\n```bash\nkubectl get pod -n istio-system\n\n# default 配置时\nNAME READY STATUS RESTARTS AGE\nistio-citadel-899dfb67c-5hlsc 1/1 Running 0 49s\nistio-cleanup-secrets-1.1.7-nkdxt 0/1 Completed 0 50s\nistio-galley-555dd7c7d7-rpfln 1/1 Running 0 49s\nistio-ingressgateway-5b547dfb7b-ctm5l 1/1 Running 0 49s\nistio-init-crd-10-l9xcj 0/1 Completed 0 66s\nistio-init-crd-11-nqvml 0/1 Completed 0 66s\nistio-pilot-9f5c75ddf-n5s6p 2/2 Running 0 49s\nistio-policy-bd45d757d-6qcdg 2/2 Running 1 49s\nistio-security-post-install-1.1.7-nbwwv 0/1 Completed 0 50s\nistio-sidecar-injector-998dd6cbb-n2hdm 1/1 Running 0 49s\nistio-telemetry-656df5b64-k8vkf 2/2 Running 1 49s\nprometheus-7f87866f5f-t97wc 1/1 Running 0 49s\n\n# demo 配置时\ngrafana-749c78bcc5-fbzmn 1/1 Running 0 101s\nistio-citadel-899dfb67c-8shx2 1/1 Running 0 100s\nistio-cleanup-secrets-1.1.7-jbhsl 0/1 Completed 0 102s\nistio-egressgateway-748d5fd794-x5bjt 1/1 Running 0 101s\nistio-galley-555dd7c7d7-86r2b 1/1 Running 0 101s\nistio-grafana-post-install-1.1.7-kq7b4 0/1 Completed 0 103s\nistio-ingressgateway-55dd86767f-jd9m4 1/1 Running 0 101s\nistio-init-crd-10-l9xcj 0/1 Completed 0 16m\nistio-init-crd-11-nqvml 0/1 Completed 0 16m\nistio-pilot-6964dd4957-7bzdq 2/2 Running 0 101s\nistio-policy-689687bd77-ncw2n 2/2 Running 1 101s\nistio-security-post-install-1.1.7-t2kwh 0/1 Completed 0 102s\nistio-sidecar-injector-998dd6cbb-7mwkh 1/1 Running 0 100s\nistio-telemetry-8564679887-59c8z 2/2 Running 1 101s\nistio-tracing-595796cf54-jn49s 1/1 Running 0 100s\nkiali-5df77dc9b6-psjs4 1/1 Running 0 101s\nprometheus-7f87866f5f-hrbgt 1/1 Running 0 100s\n\n```\n\n```bash\nkubectl get svc -n istio-system\n\n# default 配置时\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nistio-citadel ClusterIP 10.68.236.249 8060/TCP,15014/TCP 75s\nistio-galley ClusterIP 10.68.105.102 443/TCP,15014/TCP,9901/TCP 75s\nistio-ingressgateway NodePort 10.68.181.46 15020:32761/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:33185/TCP,15030:20745/TCP,15031:36208/TCP,15032:34095/TCP,15443:36244/TCP 75s\nistio-pilot ClusterIP 10.68.252.143 15010/TCP,15011/TCP,8080/TCP,15014/TCP 75s\nistio-policy ClusterIP 10.68.40.51 9091/TCP,15004/TCP,15014/TCP 75s\nistio-sidecar-injector ClusterIP 10.68.55.134 443/TCP 74s\nistio-telemetry ClusterIP 10.68.16.11 9091/TCP,15004/TCP,15014/TCP,42422/TCP 75s\nprometheus ClusterIP 10.68.65.238 9090/TCP 75s\n\n# demo 配置时\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ngrafana ClusterIP 10.68.65.248 3000/TCP 2m27s\nistio-citadel ClusterIP 10.68.72.100 8060/TCP,15014/TCP 2m26s\nistio-egressgateway NodePort 10.68.21.24 80:26775/TCP,443:28249/TCP,15443:38494/TCP 2m27s\nistio-galley ClusterIP 10.68.73.9 443/TCP,15014/TCP,9901/TCP 2m27s\nistio-ingressgateway NodePort 10.68.122.190 15020:39248/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:33522/TCP,15030:26010/TCP,15031:27064/TCP,15032:32158/TCP,15443:30848/TCP 2m27s\nistio-pilot ClusterIP 10.68.116.5 15010/TCP,15011/TCP,8080/TCP,15014/TCP 2m26s\nistio-policy ClusterIP 10.68.239.246 9091/TCP,15004/TCP,15014/TCP 2m27s\nistio-sidecar-injector ClusterIP 10.68.93.151 443/TCP 2m26s\nistio-telemetry ClusterIP 10.68.117.254 9091/TCP,15004/TCP,15014/TCP,42422/TCP 2m26s\njaeger-agent ClusterIP None 5775/UDP,6831/UDP,6832/UDP 2m25s\njaeger-collector ClusterIP 10.68.103.8 14267/TCP,14268/TCP 2m26s\njaeger-query ClusterIP 10.68.73.252 16686/TCP 2m26s\nkiali ClusterIP 10.68.214.228 20001/TCP 2m27s\nprometheus ClusterIP 10.68.203.209 9090/TCP 2m26s\ntracing ClusterIP 10.68.113.236 80/TCP 2m25s\nzipkin ClusterIP 10.68.96.189 9411/TCP 2m25s\n```\n\n##### Sidecar 的自动注入\n\n注意事项\n\n需要在kube-apiserver 启动 admission-control 参数中加入 MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook并确保正确的顺序,如果是多master安装，确保每个kube-apiserver都要进行修改。\n\n\n##### 部署应用验证\n\nistio 的samples目录中有很多示例。我们现在使用samples/sleep/sleep.yaml 来验证刚刚开启的Sidecar自动注入功能。\n\n进入目录 istio-1.1.7/ 部署一个新的应用\n\n```bash\ncd istio-1.1.7/\nkubectl apply -f samples/sleep/sleep.yaml\n\nkubectl get pod \nNAME READY STATUS RESTARTS AGE\nsleep-7549f66447-wv8cl 1/1 Running 0 1m\n```\n\n一切都是熟悉的味道。下面给 default 命名空间设置标签：istio-injection=enabled，这样就会在pod 创建时触发 Sidecar 的注入过程。从此default 名称空间拥有了超能力.\n\n```bash\nkubectl label namespace default istio-injection=enabled\nkubectl get namespace -L istio-injection\nNAME STATUS AGE ISTIO-INJECTION\ndefault Active 1h enabled\nistio-system Active 3d22h \nkube-public Active 4d2h \nkube-system Active 4d2h\n```\n接下来删除上面创建的pod，观察下有什么变化。\n\n```bash\nkubectl delete pod sleep-7549f66447-wv8cl\npod \"sleep-7549f66447-wv8cl\" deleted\n\nkubectl get pod \nNAME READY STATUS RESTARTS AGE\nsleep-7549f66447-x4td6 2/2 Running 0 37s\n```\n\n刚刚的pod里面现在已经拥有两个容器，进入pod一探究竟。\n```bash\n kubectl describe pod sleep-7549f66447-x4td6\n\n ....\n Containers:\n sleep:\n Container ID: docker://\n Image: pstauffer/curl\n .... \n \n istio-proxy:\n Container ID: docker://\n Image: docker.io/istio/proxyv2:1.1.7\n ....\n \n```\n多出了一个 `istio-proxy` 容器及其对应的存储卷\n\n\n#### 卸载istio \n\n---\n\n```bash\n# 采用 default 配置安装\nhelm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl delete -f -\n# 采用 demo 配置安装\nhelm template install/kubernetes/helm/istio --name istio --namespace istio-system \\\n --values install/kubernetes/helm/istio/values-istio-demo.yaml | kubectl delete -f -\n\nkubectl delete namespace istio-system\n```\n\n\n#### 资源\n- [官方安装文档](https://istio.io/zh/docs/setup/kubernetes/install/helm/)\n"
},
{
"path": "docs/guide/kernel_upgrade.md",
"content": "# Linux Kernel 升级\n\nk8s,docker,cilium等很多功能、特性需要较新的linux内核支持，所以有必要在集群部署前对内核进行升级；CentOS7 和 Ubuntu16.04可以很方便的完成内核升级。\n\n## CentOS7\n\n红帽企业版 Linux 仓库网站 https://www.elrepo.org，主要提供各种硬件驱动（显卡、网卡、声卡等）和内核升级相关资源；兼容 CentOS7 内核升级。如下按照网站提示载入elrepo公钥及最新elrepo版本，然后按步骤升级内核（以安装长期支持版本 kernel-lt 为例）\n\n``` bash\n#安装所需软件包\nyum install -y perl wget\n\n#下载所需内核版本的 RPM 包，更多版本可以从中寻找（http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/）\nwget http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-lt-5.4.278-1.el7.elrepo.x86_64.rpm\nwget http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-lt-devel-5.4.278-1.el7.elrepo.x86_64.rpm\nwget http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-lt-headers-5.4.278-1.el7.elrepo.x86_64.rpm\nwget http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-lt-tools-5.4.278-1.el7.elrepo.x86_64.rpm\nwget http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-lt-tools-libs-5.4.278-1.el7.elrepo.x86_64.rpm\n\n# 卸载旧版工具（安装kernel-lt-tools会和旧版本的kernel-tools导致冲突，需要卸载旧版本的）\nyum remove kernel-tools kernel-tools-libs -y\n\n#安装下载的 RPM 包\nrpm -ivh kernel-lt-tools-libs-5.4.278-1.el7.elrepo.x86_64.rpm\nrpm -ivh kernel-lt-tools-5.4.278-1.el7.elrepo.x86_64.rpm \nrpm -ivh kernel-lt-5.4.278-1.el7.elrepo.x86_64.rpm\nrpm -ivh kernel-lt-devel-5.4.278-1.el7.elrepo.x86_64.rpm \n\n#验证安装，可以看到新版本的和旧版本的\nrpm -qa | grep kernel\nkernel-lt-5.4.278-1.el7.elrepo.x86_64\nkernel-lt-tools-libs-5.4.278-1.el7.elrepo.x86_64\nkernel-3.10.0-1160.71.1.el7.x86_64\nkernel-lt-devel-5.4.278-1.el7.elrepo.x86_64\nkernel-lt-tools-5.4.278-1.el7.elrepo.x86_64\n\n#查看默认启动顺序\nawk -F\\' '$1==\"menuentry \" {print i++ \" : \" $2}' /etc/grub2.cfg\n0 : CentOS Linux (5.4.278-1.el7.elrepo.x86_64) 7 (Core)\n1 : CentOS Linux (3.10.0-1160.71.1.el7.x86_64) 7 (Core)\n2 : CentOS Linux (0-rescue-0b208d4cc51848998d32430e022d3040) 7 (Core)\n#设置默认启动内核顺序\ngrub2-set-default 0 \n#重启\nreboot\n#重启后进行检查是否成功切换到新内核\nuname -r\n5.4.278-1.el7.elrepo.x86_64\n```\n\n## Ubuntu16.04\n\n``` bash\n打开 http://kernel.ubuntu.com/~kernel-ppa/mainline/ 并选择列表中选择你需要的版本（以4.16.3为例）。\n接下来，根据你的系统架构下载 如下.deb 文件：\nBuild for amd64 succeeded (see BUILD.LOG.amd64):\n linux-headers-4.16.3-041603_4.16.3-041603.201804190730_all.deb\n linux-headers-4.16.3-041603-generic_4.16.3-041603.201804190730_amd64.deb\n linux-image-4.16.3-041603-generic_4.16.3-041603.201804190730_amd64.deb\n#安装后重启即可\n$ sudo dpkg -i *.deb\n```\n"
},
{
"path": "docs/guide/kubedns.md",
"content": "# 集群 DNS\n\nDNS 是 k8s 集群首要部署的组件，它为集群中的其他 pods 提供域名解析服务；主要可以解析 `集群服务名 SVC` 和 `Pod hostname`；目前建议部署 `coredns`。\n\nNodeLocal DNSCache在集群的上运行一个dnsCache daemonset来提高clusterDNS性能和可靠性。在K8S集群上的一些测试表明：相比于纯coredns方案，nodelocaldns + coredns方案能够大幅降低DNS查询timeout的频次，提升服务稳定性。参考官方文档：https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/\n\n### 部署 dns\n\n配置文件参考 `https://github.com/kubernetes/kubernetes` 项目目录 `kubernetes/cluster/addons/dns`\n\n+ 安装\n\n目前 kubeasz 已经自动集成安装 coredns 和 nodelocaldns 组件，配置模板位于`roles/cluster-addon/templates/`目录。\n\n``` bash\n# 默认已经集成安装，假设集群名为xxxx\nezctl setup xxxx all\n\n# 如果需要分步安装\nezctl setup xxxx 07\n\n# 如果需要手动安装\nkubectl apply -f /etc/kubeasz/clusters/xxxx/yml/coredns.yaml\nkubectl apply -f /etc/kubeasz/clusters/xxxx/yml/nodelocaldns.yaml\n```\n\n### 验证 dns服务\n\n新建一个测试nginx服务\n\n`kubectl run nginx --image=nginx --expose --port=80`\n\n确认nginx服务\n\n``` bash\nkubectl get pod|grep nginx\nnginx-7cbc4b4d9c-fl46v 1/1 Running 0 1m\nkubectl get svc|grep nginx\nnginx ClusterIP 10.68.33.167 80/TCP 1m\n```\n\n测试pod alpine\n\n``` bash\nkubectl run test --rm -it --image=alpine /bin/sh\nIf you don't see a command prompt, try pressing enter.\n\n/ # cat /etc/resolv.conf\nnameserver 10.68.0.2\nsearch default.svc.cluster.local. svc.cluster.local. cluster.local.\noptions ndots:5\n\n# 测试集群内部服务解析\n/ # nslookup nginx.default.svc.cluster.local\nServer: 10.68.0.2\nAddress 1: 10.68.0.2 kube-dns.kube-system.svc.cluster.local\n\nName: nginx\nAddress 1: 10.68.33.167 nginx.default.svc.cluster.local\n\n/ # nslookup kubernetes.default.svc.cluster.local\nServer: 10.68.0.2\nAddress 1: 10.68.0.2 kube-dns.kube-system.svc.cluster.local\n\nName: kubernetes\nAddress 1: 10.68.0.1 kubernetes.default.svc.cluster.local\n\n# 测试外部域名的解析，默认集成node的dns解析\n/ # nslookup www.baidu.com\nServer: 10.68.0.2\nAddress 1: 10.68.0.2 kube-dns.kube-system.svc.cluster.local\n\nName: www.baidu.com\nAddress 1: 180.97.33.108\nAddress 2: 180.97.33.107\n/ #\n```\n\n- Note1: 如果你使用`calico`网络组件，安装完集群后，直接安装dns组件，可能会出现如下BUG，分析是因为calico分配pod地址时候会从网段的第一个地址（网络地址）开始，详见提交的 [ISSUE #1710](https://github.com/projectcalico/calico/issues/1710)，临时解决办法为手动删除POD，重新创建后获取后面的IP地址\n\n```\n# BUG出现现象\n$ kubectl get pod --all-namespaces -o wide\nNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE\ndefault busy-5cc98488d4-s894w 1/1 Running 0 28m 172.20.24.193 192.168.97.24\nkube-system calico-kube-controllers-6597d9c664-nq9hn 1/1 Running 0 1h 192.168.97.24 192.168.97.24\nkube-system calico-node-f8gnf 2/2 Running 0 1h 192.168.97.24 192.168.97.24\nkube-system kube-dns-69bf9d5cc9-c68mw 0/3 CrashLoopBackOff 27 31m 172.20.24.192 192.168.97.24\n\n# 解决办法，删除pod，自动重建\n$ kubectl delete pod -n kube-system kube-dns-69bf9d5cc9-c68mw\n```\n\n- Note2: 使用``` kubectl run test -it --rm --image=busybox /bin/sh``` 进行解析测试可能会失败, busybox内的nslookup程序有bug, 详见 https://github.com/kubernetes/dns/issues/109\n"
},
{
"path": "docs/guide/kubesphere.md",
"content": "# 在 Kubernetes 安装 KubeSphere 容器平台\n\n## 什么是 KubeSphere\n\n[KubeSphere](https://github.com/kubesphere/kubesphere) 是在 [Kubernetes](https://kubernetes.io) 之上构建的面向云原生应用的**开源容器平台**，支持多云与多集群管理，提供全栈的 IT 自动化运维能力，简化企业的 DevOps 工作流。它的架构可以非常方便地使第三方应用与云原生生态组件进行即插即用 (plug-and-play) 的集成。\n\nKubeSphere 作为一个**全栈的多租户容器平台**，不仅支持**安装和纳管原生 Kubernetes**，还设计了一套完整的管理界面，方便开发者与运维人员在一个**统一的平台**中安装与管理最常用的云原生工具，**从业务视角提供一致的用户体验来降低复杂性**。目前最新的 3.0 版本提供以下功能：\n\n|功能 |介绍 |\n| --- | ---|\n| Kubernetes 集群搭建与运维 | 支持在线 & 离线安装、升级与扩容 Kubernetes 集群，支持安装 “云原生全家桶” |\n| Kubernetes 资源可视化管理 | 比 Kubernetes 原生 Dashboard 功能更丰富的控制面板，支持向导式创建与管理 Kubernetes 资源 |\n| 基于 Jenkins 的 DevOps 系统 | 支持图形化与脚本两种方式构建 CI/CD 流水线，内置 Source/Binary to Image 等 CD 工具 |\n| 应用商店与应用生命周期管理 | 内置 Redis、MySQL 等十五个常用应用，基于 Helm 提供应用上传、审核、发布、部署、下架等操作 |\n| 基于 Istio 的微服务治理 (Service Mesh) | 提供可视化无代码侵入的**灰度发布、熔断机制、流量治理与流量拓扑、分布式链路追踪** |\n| 多租户管理 | 提供基于角色的细粒度多租户统一认证，支持**对接企业 LDAP/AD**，提供多层级的权限管理 |\n| 丰富的可观察性功能 | UI 提供集群/工作负载/Pod/容器等多维度的监控、事件/日志查询、告警与通知管理 |\n| 存储管理 | 支持对接 Ceph、GlusterFS、NFS，支持可视化管理 PVC、PV、StorageClass |\n| 网络管理 | 支持 Calico、Flannel，提供 Porter LB 帮助暴露物理环境 Kubernetes 集群的 LoadBalancer 服务 |\n| GPU support | 集群支持添加 GPU 与 vGPU，可运行 TensorFlow 等 ML 框架 |\n\n\n## 在 Kubernetes 与 Kubeasz 之上安装 KubeSphere\n\n作为一个轻量化容器平台，KubeSphere 可以安装在任何私有或托管的 Kubernetes、虚拟机、裸机、本地环境、公有云、混合云之上，并且所有功能组件都是可插拔的。当使用 Kubeasz 完成 Kubernetes 集群的安装后，可参考以下步骤在 Kubernetes 上安装 KubeSphere。\n\n**前提条件**\n\n> - Kubernetes 版本必须是：1.15.x、1.16.x、1.17.x 或 1.18.x；\n> - 您的机器满足最低硬件要求：CPU > 1 Core，可用内存 > 2 G；\n> - 安装之前，Kubernetes 集群已配置**默认**存储类型 (StorageClass)；\n> - 当使用 `--cluster-signing-cert-file` 和 `--cluster-signing-key-file` 参数启动时，在 `kube-apiserver` 中会激活 CSR 签名功能。请参见 [RKE 安装问题](https://github.com/kubesphere/kubesphere/issues/1925#issuecomment-591698309)；\n> - 有关在 Kubernetes 上安装 KubeSphere 的准备工作，请参见[准备工作](https://kubesphere.io/zh/docs/installing-on-kubernetes/introduction/prerequisites/)。\n>\n\n1. 若待安装的环境满足以上条件，则可以执行以下命令部署 KubeSphere：\n\n ```yaml\n kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/kubesphere-installer.yaml\n \n kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/cluster-configuration.yaml\n ```\n\n2. 等待安装成功（取决于您的网络状况，约十几至二十几分钟不等），运行以下命令查看安装日志：\n\n ```bash\n kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f\n ```\n\n \n\n3. 使用 `kubectl get pod --all-namespaces` 查看所有 Pod 是否在 KubeSphere 的相关命名空间中正常运行。如果是，请通过以下命令检查控制台的端口（默认为 `30880`）：\n\n ```bash\n kubectl get svc/ks-console -n kubesphere-system\n ```\n\n4. 请确保在安全组中打开了端口 `30880`，并通过 NodePort `(IP:30880)` 使用默认帐户和密码 `(admin/P@88w0rd)` 访问 Web 控制台。\n\n5. 登录控制台后，您可以在**服务组件**中查看不同组件的状态。如果要使用相关服务，可能需要等待某些组件启动并运行。\n\n**Tips**：若要在 KubeSphere 中启用其他组件，请参见[启用可插拔组件](https://kubesphere.io/zh/docs/pluggable-components/)。开启安装前确认您的机器资源已符合[资源最低要求](https://kubesphere.io/zh/docs/pluggable-components/overview/)。\n\n## 延伸阅读\n\n- [安装 Kubeasz 与 KubeSphere](https://kubesphere.com.cn/forum/d/716-play-with-kubesphere-and-kubeasz)\n- [在 Linux 完整安装 KubeSphere 与 Kubernetes](https://kubesphere.io/zh/docs/installing-on-linux/introduction/intro/)\n- [KubeSphere 官网](https://kubesphere.io/zh/)\n- [常见问题](https://kubesphere.io/zh/docs/faq/)\n\n\n"
},
{
"path": "docs/guide/log-pilot.md",
"content": "# Log-Pilot Elasticsearch Kibana 日志解决方案\n\n该方案是社区方案`EFK`的升级版，它支持两种搜集形式，对应容器标准输出日志和容器内的日志文件；个人使用了一把，在原有`EFK`经验的基础上非常简单、方便，值得推荐；更多的关于`log-pilot`的介绍详见链接：\n\n- github 项目地址: https://github.com/AliyunContainerService/log-pilot\n- 阿里云介绍文档: https://help.aliyun.com/document_detail/86552.html\n- 介绍文档2: https://yq.aliyun.com/articles/674327\n\n## 安装步骤\n\n- 1.安装 ES 集群，同[EFK](efk.md)文档\n\n- 2.安装 Kibana，同[EFK](efk.md)文档\n\n- 3.安装 Log-Pilot\n\n``` bash\nkubectl apply -f /etc/kubeasz/manifests/efk/log-pilot/log-pilot-filebeat.yaml\n```\n\n- 4.创建示例应用，采集日志\n\n``` bash\n$ cat > tomcat.yaml << EOF\napiVersion: v1\nkind: Pod\nmetadata:\n name: tomcat\nspec:\n containers:\n - name: tomcat\n image: \"tomcat:7.0\"\n env:\n # 1、stdout为约定关键字，表示采集标准输出日志\n # 2、配置标准输出日志采集到ES的catalina索引下\n - name: aliyun_logs_catalina\n value: \"stdout\"\n # 1、配置采集容器内文件日志，支持通配符\n # 2、配置该日志采集到ES的access索引下\n - name: aliyun_logs_access\n value: \"/usr/local/tomcat/logs/catalina.*.log\"\n volumeMounts:\n - name: tomcat-log\n mountPath: /usr/local/tomcat/logs\n volumes:\n # 容器内文件日志路径需要配置emptyDir\n - name: tomcat-log\n emptyDir: {}\nEOF\n\n$ kubectl apply -f tomcat.yaml \n```\n\n- 5.在 kibana 创建 Index Pattern，验证日志已搜集，如上示例应用，应创建如下 index pattern\n - catalina-*\n - access-*\n"
},
{
"path": "docs/guide/lvm.md",
"content": "# LVM 操作\n\n以下是使用 parted 对 /dev/sdb 进行分区并配置 LVM 的完整操作流程，包含详细解释和注意事项。\n\n## 1. 磁盘分区 (使用 parted)\n\n``` bash\nsudo parted /dev/sdb\n# 在交互界面执行：\n(parted) mklabel gpt # 创建 GPT 分区表（兼容大容量磁盘）\n(parted) mkpart lvm 0% 100% # 创建占用整个磁盘的 LVM 分区\n(parted) set 1 lvm on # 设置分区类型为 LVM\n(parted) print # 验证分区信息\n(parted) quit # 退出\n# 验证分区结果：应看到 sdb1 分区\nlsblk /dev/sdb\n```\n\n## 2. 创建物理卷 (PV)\n\n``` bash\nsudo pvcreate /dev/sdb1 # 将分区初始化为物理卷\nsudo pvs # 查看已创建的物理卷\n# 输出示例：\n PV VG Fmt Attr PSize PFree\n /dev/sdb1 lvm2 --- 100.00g 100.00g\n```\n\n## 3. 创建卷组 (VG)\n\n``` bash\nsudo vgcreate vg_data /dev/sdb1 # 创建名为 vg_data 的卷组\nsudo vgs # 查看卷组信息\n# 参数说明：\n● vg_data：自定义卷组名称\n● -s 4M：可指定 PE 大小（默认 4MB）\n```\n\n## 4. 创建逻辑卷 (LV)\n\n```\nsudo lvcreate -n lv_app -L 50G vg_data # 创建 50G 的逻辑卷\nsudo lvs # 查看逻辑卷\n可选参数：\n● -l 100%FREE：使用全部剩余空间\n● -i 3 -I 4：创建条带化卷（需多磁盘）\n```\n\n## 5. 创建文件系统\n\n```\nsudo mkfs.xfs /dev/vg_data/lv_app # 创建 XFS 文件系统\n# 或使用 ext4：\n# sudo mkfs.ext4 /dev/vg_data/lv_app\n格式选择建议：\n● XFS：适合大文件/高并发\n● ext4：兼容性好\n```\n\n## 6. 挂载文件系统\n\n```\nsudo mkdir /data \t \t\t# 创建挂载点\nsudo mount /dev/vg_data/lv_app /data \t\t# 临时挂载\ndf -hT /data \t\t# 验证挂载\n# 持久化挂载：\necho '/dev/mapper/vg_data-lv_app /data xfs defaults 0 0' | sudo tee -a /etc/fstab\nsudo mount -a # 测试 fstab 配置\n```\n- 建议使用uuid方式挂载\n```\n# 查看uuid\nsudo blkid /dev/mapper/vg_data-lv_app\n/dev/mapper/vg_data-lv_app: UUID=\"b8520e35-3a01-4ec7-b31a-3371f31c4de7\" BLOCK_SIZE=\"4096\" TYPE=\"xfs\"\n#\necho 'UUID=\"b8520e35-3a01-4ec7-b31a-3371f31c4de7\" /data xfs defaults 0 0' | sudo tee -a /etc/fstab\n```\n\n## 完整操作流程图\ngraph TD A[磁盘/dev/sdb] --> B[parted创建GPT分区] B --> C[pvcreate创建物理卷] C --> D[vgcreate创建卷组] D --> E[lvcreate创建逻辑卷] E --> F[mkfs创建文件系统] F --> G[mount挂载使用]\nmermaidgraph TD\n A[磁盘/dev/sdb] --> B[parted创建GPT分区]\n B --> C[pvcreate创建物理卷]\n C --> D[vgcreate创建卷组]\n D --> E[lvcreate创建逻辑卷]\n E --> F[mkfs创建文件系统]\n F --> G[mount挂载使用]\n\n关键命令速查表\n\n| 操作 | 命令 |\n|:---|:---|\n|查看块设备|lsblk|\n|验证分区表|parted /dev/sdb print|\n|扩展逻辑卷|lvextend -L +10G /dev/vg_data/lv_app|\n|扩展文件系统 (XFS)|xfs_growfs /data|\n|扩展文件系统 (ext4)|resize2fs /dev/vg_data/lv_app|\n|删除卷组|vgremove vg_data|\n\n注意事项\n● 数据备份：操作前确认磁盘无重要数据\n● 容量对齐：生产环境建议保持 1MB 对齐（parted 使用 % 单位自动对齐）\n● 在线扩展：XFS 支持在线扩容，但不支持缩小\n● RAID 整合：可在 LVM 层整合多个 PV 实现软 RAID\n● 快照功能：使用 lvcreate -s 创建快照卷实现备份\n通过以上步骤，您已成功将原始磁盘配置为可弹性管理的存储空间。后续可通过 LVM 的动态调整特性，实现无需卸载的存储扩容。\n"
},
{
"path": "docs/guide/metallb.md",
"content": "# metallb 网络负载均衡\n\n本文档已过期，以下内容仅做介绍，安装请参考最新官方文档\n\n`Metallb`是在自有硬件上（非公有云）实现 `Kubernetes Load-balancer`的工具，由`google`团队开源，值得推荐！项目[github主页](https://github.com/google/metallb)。\n\n## metallb 简介\n\n这里简单介绍下它的实现原理，具体可以参考[metallb官网](https://metallb.universe.tf/)，文档非常简洁、清晰。目前有如下的使用限制：\n\n- `Kubernetes v1.9.0`版本以上，暂不支持`ipvs`模式\n- 支持网络组件 (flannel/weave/romana), calico 部分支持\n- `layer2`和`bgp`两种模式，其中`bgp`模式需要外部网络设备支持`bgp`协议\n\n`metallb`主要实现了两个功能：地址分配和对外宣告\n\n- 地址分配：需要向网络管理员申请一段ip地址，如果是layer2模式需要这段地址与node节点地址同个网段（同一个二层）；如果是bgp模式没有这个限制。\n- 对外宣告：layer2模式使用arp协议，利用节点的mac额外宣告一个loadbalancer的ip（同mac多ip）；bgp模式下节点利用bgp协议与外部网络设备建立邻居，宣告loadbalancer的地址段给外部网络。\n\n"
},
{
"path": "docs/guide/metrics-server.md",
"content": "# Metrics Server\n\n从 v1.8 开始，资源使用情况的度量（如容器的 CPU 和内存使用）可以通过 Metrics API 获取；前提是集群中要部署 Metrics Server，它从Kubelet 公开的Summary API采集指标信息，关于更多的背景介绍请参考如下文档： \n- Metrics Server[设计提案](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md)\n\n大致是说它符合k8s的监控架构设计，受heapster项目启发，并且比heapster优势在于：访问不需要apiserver的代理机制，提供认证和授权等；很多集群内组件依赖它（HPA,scheduler,kubectl top），因此它应该在集群中默认运行；部分k8s集群的安装工具已经默认集成了Metrics Server的安装，以下概述下它的安装：\n\n- 1.metric-server是扩展的apiserver，依赖于[kube-aggregator](https://github.com/kubernetes/kube-aggregator)，因此需要在apiserver中开启相关参数。\n- 2.需要在集群中运行deployment处理请求\n\n从kubeasz 0.1.0 开始，metrics-server已经默认集成安装，请查看`/etc/kubeasz/clusters/xxxx/config.yml`中的设置\n- 参考：https://github.com/kubernetes-sigs/metrics-server\n\n## 前提\n\n- 1.设置apiserver相关[参数](../../roles/kube-master/templates/kube-apiserver.service.j2)\n``` bash\n... # 省略\n --requestheader-client-ca-file={{ ca_dir }}/ca.pem \\\n --requestheader-allowed-names=aggregator \\\n --requestheader-extra-headers-prefix=X-Remote-Extra- \\\n --requestheader-group-headers=X-Remote-Group \\\n --requestheader-username-headers=X-Remote-User \\\n --proxy-client-cert-file={{ ca_dir }}/aggregator-proxy.pem \\\n --proxy-client-key-file={{ ca_dir }}/aggregator-proxy-key.pem \\\n --enable-aggregator-routing=true \\\n```\n- 2.生成[aggregator proxy相关证书](../../roles/kube-master/tasks/main.yml)\n\n参考1：https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/ \n参考2：https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/\n\n## 安装\n\n``` bash\n# 默认已经集成安装，假设集群名为xxxx\nezctl setup xxxx all\n\n# 如果需要分步安装\nezctl setup xxxx 07\n\n# 如果需要手动安装\nkubectl apply -f /etc/kubeasz/clusters/xxxx/yml/metrics-server.yaml\n```\n\n## 验证\n\n- 查看生成的新api：v1beta1.metrics.k8s.io\n``` bash\n$ kubectl get apiservice|grep metrics\nv1beta1.metrics.k8s.io 1d\n```\n\n- 查看kubectl top命令（无需额外安装heapster）\n``` bash\n$ kubectl top node\nNAME CPU(cores) CPU% MEMORY(bytes) MEMORY% \n192.168.1.1 116m 2% 2342Mi 60% \n192.168.1.2 79m 1% 1824Mi 47% \n192.168.1.3 82m 2% 1897Mi 49% \n$ kubectl top pod --all-namespaces \t# 输出略\n```\n\n- 验证基于metrics-server实现的基础hpa自动缩放，请参考[hpa.md](hpa.md)\n"
},
{
"path": "docs/guide/networkpolicy.md",
"content": "## Network Policy\n\n`Network Policy`提供了基于策略的网络控制，用于隔离应用并减少攻击面。它使用标签选择器模拟传统的分段网络，并通过策略控制它们之间的流量以及来自外部的流量；目前基于`linux iptables`实现，使用类似`nf_conntrack`检查记录网络流量`session`从而决定流量是否阻断；因此它是`状态检测防火墙`。\n\n- 网络插件要支持 Network Policy，如 Calico、Romana、Weave Net\n\n### 简单示例\n\n实验环境：k8s v1.9, calico 2.6.5\n\n首先部署测试用nginx服务\n\n``` bash\n$ kubectl run nginx --image=nginx --replicas=3 --port=80 --expose\n# 验证测试nginx服务\n$ kubectl get pod -o wide \nNAME READY STATUS RESTARTS AGE IP NODE\nnginx-7587c6fdb6-p2fpz 1/1 Running 0 55m 172.20.125.2 10.0.96.7\nnginx-7587c6fdb6-pbw7c 1/1 Running 0 55m 172.20.124.2 10.0.96.6\nnginx-7587c6fdb6-v48db 1/1 Running 0 55m 172.20.121.195 10.0.96.4\n$ kubectl get svc nginx\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nnginx ClusterIP 10.68.7.183 80/TCP 1h\n```\n默认情况下，其他pod可以访问nginx服务\n\n``` bash\n$ kubectl run busy1 --rm -it --image=busybox /bin/sh\nIf you don't see a command prompt, try pressing enter.\n/ # wget --spider --timeout=1 nginx\nConnecting to nginx (10.68.7.183:80)\n```\n创建`DefaultDeny Network Policy`后，其他Pod（包括namespace外部）不能访问nginx\n\n``` bash\n$ cat > default-deny.yaml << EOF\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n name: default-deny\nspec:\n podSelector: {}\n policyTypes:\n - Ingress\nEOF\n$ kubectl create -f default-deny.yaml\nnetworkpolicy \"default-deny\" created\n$ kubectl run busy1 --rm -it --image=busybox /bin/sh\nIf you don't see a command prompt, try pressing enter.\n/ # wget --spider --timeout=1 nginx\nConnecting to nginx (10.68.7.183:80)\nwget: download timed out\n```\n创建一个允许带有access=true的Pod访问nginx的网络策略\n\n``` bash\n$ cat > nginx-policy.yaml << EOF\nkind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n name: access-nginx\nspec:\n podSelector:\n matchLabels:\n run: nginx\n ingress:\n - from:\n - podSelector:\n matchLabels:\n access: \"true\"\nEOF\n$ kubectl create -f nginx-policy.yaml\nnetworkpolicy \"access-nginx\" created\n\n# 不带access=true标签的Pod还是无法访问nginx服务\n$ kubectl run busy1 --rm -it --image=busybox /bin/sh\nIf you don't see a command prompt, try pressing enter.\n/ # wget --spider --timeout=1 nginx\nConnecting to nginx (10.68.7.183:80)\nwget: download timed out\n\n# 而带有access=true标签的Pod可以访问nginx服务\n$ kubectl run busy2 --rm -it --labels=\"access=true\" --image=busybox /bin/sh\nIf you don't see a command prompt, try pressing enter.\n/ # wget --spider --timeout=1 nginx\nConnecting to nginx (10.68.7.183:80)\n```\n\n### 示例策略解读\n\n``` bash\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n name: test-network-policy\n namespace: default\nspec:\n podSelector:\n matchLabels:\n role: db\n policyTypes:\n - Ingress\n - Egress\n ingress:\n - from:\n - ipBlock:\n cidr: 172.17.0.0/16\n except:\n - 172.17.1.0/24\n - namespaceSelector:\n matchLabels:\n project: myproject\n - podSelector:\n matchLabels:\n role: frontend\n ports:\n - protocol: TCP\n port: 6379\n egress:\n - to:\n - ipBlock:\n cidr: 10.0.0.0/24\n ports:\n - protocol: TCP\n port: 5978\n```\n- 策略作用的对象Pods：default命名空间下带有`role=db`标签的Pod\n - 内向流量策略\n - 允许属于`172.17.0.0/16`网段但不属于`172.17.1.0/24`的源地址访问该对象Pods的TCP 6379端口\n - 允许带有project=myprojects标签的namespace中所有Pod访问该对象Pods的TCP 6379端口\n - 允许default命名空间下带有role=frontend标签的Pod访问该对象Pods的TCP 6379端口\n - 拒绝其他所有主动访问该对象Pods的网络流量\n - 外向流量策略\n - 允许该对象Pods主动访问目的地址属于`10.0.0.0/24`网段且目的端口为TCP 5978的流量\n - 拒绝该对象Pods其他所有主动外向网络流量\n\n### 使用场景\n\n参考阅读[ahmetb/kubernetes-network-policy-recipes](https://github.com/ahmetb/kubernetes-network-policy-recipes) 该项目举例一些使用NetworkPolicy的场景，并有形象的配图\n\n#### 拒绝其他namespaces访问服务\n\n\n\n+ 场景1：你的k8s集群应用按照namespaces区分生产、测试环境，你要确保生产环境不会受到测试环境错误访问影响\n+ 场景2：你的k8s集群有多租户应用采用namespaces区分的，你要确保多租户之间的应用隔离\n\n在你需要隔离的命名空间创建如下策略:\n\n``` bash\nkind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n namespace: your-ns\n name: deny-other-namespaces\nspec:\n podSelector:\n matchLabels:\n ingress:\n - from:\n - podSelector: {}\n```\n\n#### 允许外部访问服务\n\n+ 场景：暴露特定Pod的特定端口给外部访问\n\n\n\n``` bash\n# 创建示例应用待暴露服务\n$ kubectl run web --image=nginx --labels=app=web --port 80 --expose\n\n# 创建网络策略\nkind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n name: web-allow-external\nspec:\n podSelector:\n matchLabels:\n app: web\n ingress:\n - from: []\n ports:\n - protocol: TCP\n port: 80\n```\n"
},
{
"path": "docs/guide/nfs-server.md",
"content": "## 创建 NFS 服务器\n\nNFS 允许系统将其目录和文件共享给网络上的其他系统。通过 NFS，用户和应用程序可以访问远程系统上的文件，就象它们是本地文件一样。\n\n### 安装\nUbuntu 16.04 键入以下命令安装 NFS 服务器：\n\n``` bash\napt install nfs-kernel-server\n```\n\n### 配置\n编辑`/etc/exports`文件添加需要共享目录，每个目录的设置独占一行，编写格式如下：\n\n`NFS共享目录路径 客户机IP或者名称(参数1,参数2,...,参数n)`\n\n例如：\n\n``` bash\n/home *(ro,sync,insecure,no_root_squash)\n/share 192.168.1.0/24(rw,sync,insecure,no_subtree_check,no_root_squash)\n```\n| 参数 | 说明 |\n| :- | :- |\n| ro | 只读访问 |\n| rw | 读写访问 |\n| sync | 所有数据在请求时写入共享 |\n| async | nfs在写入数据前可以响应请求 |\n| secure | nfs通过1024以下的安全TCP/IP端口发送 |\n| insecure | nfs通过1024以上的端口发送 |\n| wdelay | 如果多个用户要写入nfs目录，则归组写入（默认） |\n| no_wdelay | 如果多个用户要写入nfs目录，则立即写入，当使用async时，无需此设置 |\n| hide | 在nfs共享目录中不共享其子目录 |\n| no_hide | 共享nfs目录的子目录 |\n| subtree_check | 如果共享/usr/bin之类的子目录时，强制nfs检查父目录的权限（默认） |\n| no_subtree_check | 不检查父目录权限 |\n| all_squash | 共享文件的UID和GID映射匿名用户anonymous，适合公用目录 |\n| no_all_squash | 保留共享文件的UID和GID（默认） |\n| root_squash | root用户的所有请求映射成如anonymous用户一样的权限（默认） |\n| no_root_squash | root用户具有根目录的完全管理访问权限 |\n| anonuid=xxx | 指定nfs服务器/etc/passwd文件中匿名用户的UID |\n| anongid=xxx | 指定nfs服务器/etc/passwd文件中匿名用户的GID |\n\n+ 注1：尽量指定主机名或IP或IP段最小化授权可以访问NFS 挂载的资源的客户端；注意如果在k8s集群中配合nfs-client-provisioner使用的话，这里需要指定pod的IP段，否则nfs-client-provisioner pod无法启动，报错 mount.nfs: access denied by server while mounting\n+ 注2：经测试参数insecure必须要加，否则客户端挂载出错mount.nfs: access denied by server while mounting\n\n### 启动\n\n配置完成后，您可以在终端提示符后运行以下命令来启动 NFS 服务器：\n\n``` bash\nsystemctl start nfs-kernel-server.service\n```\n\n### 客户端挂载\n\nUbuntu 16.04，首先需要安装 `nfs-common` 包\n\n``` bash\napt install nfs-common\n```\nCentOS 7, 需要安装 `nfs-utils` 包\n\n``` bash\nyum install nfs-utils\n```\n\n使用 mount 命令来挂载其他机器共享的 NFS 目录。可以在终端提示符后输入以下类似的命令：\n\n``` bash\nmount example.hostname.com:/ubuntu /local/ubuntu\n```\n挂载点 /local/ubuntu 目录必须已经存在。而且在 /local/ubuntu 目录中没有文件或子目录。\n\n另一个挂载NFS 共享的方式就是在 /etc/fstab 文件中添加一行。该行必须指明 NFS 服务器的主机名、服务器输出的目录名以及挂载 NFS 共享的本机目录。\n\n以下是在 /etc/fstab 中的常用语法：\n\n``` bash\nexample.hostname.com:/ubuntu /local/ubuntu nfs rsize=8192,wsize=8192,timeo=14,intr\n```\n"
},
{
"path": "docs/guide/prometheus.md",
"content": "# Prometheus\n`prometheus`已经成为k8s集群上默认的监控解决方案，它的监控理念、数据结构设计其实相当精简，包括其非常灵活的查询语言；但是对于初学者来说，想要在k8s集群中实践搭建一套相对可用的部署却比较麻烦。本项目3.x采用的helm chart方式部署，使用的charts地址: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack\n\n## 安装\n\nkubeasz 集成安装\n\n- 1.修改 /etc/kubeasz/clusters/xxxx/config.yml 中配置项 prom_install: \"yes\"\n- 2.下载镜像 /etc/kubeasz/ezdown -X prometheus\n- 3.安装 /etc/kubeasz/ezctl setup xxxx 07\n\n生成的charts自定义配置在/etc/kubeasz/clusters/xxxx/yml/prom-values.yaml\n\n注1：如果需要修改配置，修改roles/cluster-addon/templates/prometheus/values.yaml.j2 后重新执行安装命令\n\n注2：如果集群节点有增减，重新执行安装命令\n\n注3：涉及到很多相关镜像下载比较慢，另外部分k8s.gcr.io的镜像已经替换成easzlab的mirror镜像地址\n\n## 验证安装\n\n``` bash \n# 查看相关pod和svc\n$ kubectl get pod,svc -n monitor\nNAME READY STATUS RESTARTS AGE\npod/alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 160m\npod/prometheus-grafana-69f88948bc-7hnbp 3/3 Running 0 160m\npod/prometheus-kube-prometheus-operator-f8f4758cb-bm6gs 1/1 Running 0 160m\npod/prometheus-kube-state-metrics-74b8f49c6c-f9wgg 1/1 Running 0 160m\npod/prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 160m\npod/prometheus-prometheus-node-exporter-6nfb4 1/1 Running 0 160m\npod/prometheus-prometheus-node-exporter-q4qq2 1/1 Running 0 160m\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 160m\nservice/prometheus-grafana NodePort 10.68.253.23 80:30903/TCP 160m\nservice/prometheus-kube-prometheus-alertmanager NodePort 10.68.125.191 9093:30902/TCP 160m\nservice/prometheus-kube-prometheus-operator NodePort 10.68.161.218 443:30900/TCP 160m\nservice/prometheus-kube-prometheus-prometheus NodePort 10.68.64.217 9090:30901/TCP 160m\nservice/prometheus-kube-state-metrics ClusterIP 10.68.111.106 8080/TCP 160m\nservice/prometheus-operated ClusterIP None 9090/TCP 160m\nservice/prometheus-prometheus-node-exporter ClusterIP 10.68.252.83 9100/TCP 160m\n```\n\n- 访问prometheus的web界面：`http://$NodeIP:30901`\n- 访问alertmanager的web界面：`http://$NodeIP:30902`\n- 访问grafana的web界面：`http://$NodeIP:30903` (默认用户密码 admin:Admin1234!)\n\n## 其他操作\n\n-- 以下内容没有更新测试\n\n### [可选] 配置钉钉告警\n\n- 创建钉钉群，获取群机器人 webhook 地址\n\n使用钉钉创建群聊以后可以方便设置群机器人，【群设置】-【群机器人】-【添加】-【自定义】-【添加】，然后按提示操作即可，参考 https://open.dingtalk.com/document/group/custom-robot-access\n\n上述配置好群机器人，获得这个机器人对应的Webhook地址，记录下来，后续配置钉钉告警插件要用，格式如下\n\n```\nhttps://oapi.dingtalk.com/robot/send?access_token=xxxxxxxx\n```\n\n- 创建钉钉告警插件，参考:\n - https://github.com/timonwong/prometheus-webhook-dingtalk\n - http://theo.im/blog/2017/10/16/release-prometheus-alertmanager-webhook-for-dingtalk/\n\n``` bash\n# 编辑修改文件中 access_token=xxxxxx 为上一步你获得的机器人认证 token\n$ vi /etc/kubeasz/roles/cluster-addon/templates/prometheus/dingtalk-webhook.yaml\n# 运行插件\n$ kubectl apply -f /etc/kubeasz/roles/cluster-addon/templates/prometheus/dingtalk-webhook.yaml\n```\n\n- 修改 alertsmanager 告警配置，重新运行安装命令/etc/kubeasz/ezctl setup xxxx 07，成功后如上节测试告警发送\n\n``` bash\n# 修改 alertsmanager 告警配置\n$ vi /etc/kubeasz/roles/cluster-addon/templates/prometheus/values.yaml.j2 \n# 增加 receiver dingtalk，然后在 route 配置使用 receiver: dingtalk\n receivers:\n - name: dingtalk\n webhook_configs:\n - send_resolved: false\n url: http://webhook-dingtalk.monitor.svc.cluster.local:8060/dingtalk/webhook1/send\n# ...\n```\n"
},
{
"path": "docs/guide/rollingupdateWithZeroDowntime.md",
"content": "## 1、前言\n在当下微服务架构盛行的时代，用户希望应用程序时时刻刻都是可用，为了满足不断变化的新业务，需要不断升级更新应用程序，有时可能需要频繁的发布版本。实现\"零停机\"、“零感知”的持续集成(Continuous Integration)和持续交付/部署(Continuous Delivery)应用程序，一直都是软件升级换代不得不面对的一个难题和痛点，也是一种追求的理想方式，也是DevOps诞生的目的。\n## 2、滚动发布\n把一次完整的发布过程，合理地分成多个批次，每次发布一个批次，**成功后**，再发布下一个批次，最终完成所有批次的发布。在整个滚动过程期间，保证始终有可用的副本在运行，从而平滑的发布新版本，实现**零停机(without an outage)**、用户**零感知**，是一种非常主流的发布方式。由于其自动化程度比较高，通常需要复杂的发布工具支撑，而k8s可以完美的胜任这个任务。 \n## 3、k8s滚动更新机制\n**k8s创建副本应用程序的最佳方法就是部署(Deployment)，部署自动创建副本集(ReplicaSet)，副本集可以精确地控制每次替换的Pod数量，从而可以很好的实现滚动更新**。具体来说，k8s每次使用一个新的副本控制器(replication controller)来替换已存在的副本控制器，从而始终使用一个新的Pod模板来替换旧的pod模板。\n>大致步骤如下：\n>1. 创建一个新的replication controller。\n>2. 增加或减少pod副本数量，直到满足当前批次期望的数量。\n>3. 删除旧的replication controller。\n\n## 4、演示\n>使用kubectl更新一个已部署的应用程序，并模拟回滚。为了方便分析，将应用程序的pod副本数量设置为10。\n``` bash\n$ kubectl run busy --image=busybox:1.28.4 sleep 36000000 --replicas=10\n```\n### 4.1. 发布微服务\n- 当前服务状态查看\n``` bash\n# 查看部署列表\nroot@kube-aio:~# kubectl get deploy busy\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nbusy 10 10 10 10 5m\n\n# 查看正在运行的pod\nroot@kube-aio:~# kubectl get pod | grep busy\nbusy-794c95f5d7-56b6w 1/1 Running 0 5m\nbusy-794c95f5d7-8ddjr 1/1 Running 0 5m\nbusy-794c95f5d7-8zm8r 1/1 Running 0 5m\nbusy-794c95f5d7-9hjhp 1/1 Running 0 5m\nbusy-794c95f5d7-df2r2 1/1 Running 0 5m\nbusy-794c95f5d7-fsn94 1/1 Running 0 5m\nbusy-794c95f5d7-k4w8r 1/1 Running 0 5m\nbusy-794c95f5d7-lsmgb 1/1 Running 0 5m\nbusy-794c95f5d7-rg8kw 1/1 Running 0 5m\nbusy-794c95f5d7-xpxxt 1/1 Running 0 5m\n\n# 通过pod描述，查看应用程序的当前映像版本\nroot@kube-aio:~# kubectl describe pod busy-794c95f5d7-56b6w |grep Image\n Image: busybox:1.28.4\n Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47\n```\n- 升级镜像版本到1.29\n - 为了更清晰看到更新过程，可另开一个窗口使用`$ watch kubectl get deployment busy`实时查看变化\n``` bash\n$ kubectl set image deployments/busy busy=busybox:1.29\n```\n### 4.2. 验证发布\n``` bash\n# 检查rollout状态\nroot@kube-aio:~# kubectl rollout status deployments/busy\ndeployment \"busy\" successfully rolled out\n\n# 检查pod详情\nroot@kube-aio:~# kubectl describe pod busy-665cdb7b-44jnt |grep Image\n Image: busybox:1.29\n Image ID: docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd\n```\n从上面可以看到，镜像已经升级到1.29版本\n### 4.3. 回滚发布\n``` bash\n# 回滚发布\nroot@kube-aio:~# kubectl rollout undo deployments/busy\ndeployment.apps \"busy\" \n\n# 回滚完成\nroot@kube-aio:~# kubectl rollout status deployments/busy\ndeployment \"busy\" successfully rolled out\n\n# 镜像又回退到1.28.4 版本\nroot@kube-aio:~# kubectl describe pod busy-794c95f5d7-4x9bn |grep Image\n Image: busybox:1.28.4\n Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47\n```\n\n到目前为止，整个滚动发布工作就圆满完成了！！！\n**那么如果我们想回滚到指定版本呢？答案是k8s完美支持，并且还可以通过资源文件进行配置保留的历史版次量**。由于篇幅有限，感兴趣的朋友，可以自己下去实战，回滚命令如下：\n```javascript\nkubectl rollout undo deployment/busy --to-revision=<版次>\n```\n## 5、原理\nk8s精确地控制着整个发布过程，分批次有序地进行着滚动更新，直到把所有旧的副本全部更新到新版本。实际上，k8s是通过两个参数来精确地控制着每次滚动的pod数量：\n\n>* **`maxSurge` 滚动更新过程中运行操作期望副本数的最大pod数，可以为绝对数值(eg：5)，但不能为0；也可以为百分数(eg：10%)。**\n>* **`maxUnavailable` 滚动更新过程中不可用的最大pod数，可以为绝对数值(eg：5)，但不能为0；也可以为百分数(eg：10%)。**\n\n如果未指定这两个可选参数，则k8s会使用默认配置： \n``` bash\nroot@kube-aio:~# kubectl get deploy busy -o yaml\napiVersion: apps/v1 \nkind: Deployment\nmetadata:\n annotations:\n deployment.kubernetes.io/revision: \"3\"\n creationTimestamp: 2018-08-19T02:42:56Z\n generation: 3\n labels:\n run: busy\n name: busy\n namespace: default\n resourceVersion: \"199461\"\n uid: 93fde307-a359-11e8-a93b-525400c61543\nspec:\n progressDeadlineSeconds: 600\n replicas: 10\n revisionHistoryLimit: 10\n selector:\n matchLabels:\n run: busy\n strategy:\n rollingUpdate:\n maxSurge: 1\t# 滚动更新中最多超过预期值的 pod数\n maxUnavailable: 1\t# 滚动更新中最多不可用的 pod数\n type: RollingUpdate\n...\n```\n### 5.1. 浅析部署概况\n``` bash\n# 初始状态\nroot@kube-aio:~# kubectl get deploy busy\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nbusy 10 10 10 10 1h\n\n# 再做一遍回退\nroot@kube-aio:~# kubectl rollout undo deploy busy\ndeployment.apps \"busy\" \n\n# 更新过程1\nroot@kube-aio:~# kubectl get deploy busy\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nbusy 10 11 2 9 1h\n\n# 更新过程2\nroot@kube-aio:~# kubectl get deploy busy\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nbusy 10 11 4 9 1h\n\n# 更新过程3\nroot@kube-aio:~# kubectl get deploy busy\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nbusy 10 11 6 9 1h\n\n# 更新结束\nroot@kube-aio:~# kubectl get deploy busy\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nbusy 10 10 10 10 1h\n```\n>* `DESIRED` 最终期望处于READY状态的副本数 \n>* `CURRENT` 当前的副本总数 \n>* `UP-TO-DATE` 当前完成更新的副本数 \n>* `AVAILABLE` 当前可用的副本数 \n\n当前的副本总数：10(DESIRED) + 1(maxSurge) = 11，所以CURRENT为11。\n当前可用的副本数：10(DESIRED) - 1(maxUnavailable) = 9，所以AVAILABLE为9。\n\n### 5.2. 浅析部署详情\n\n``` bash\nroot@kube-aio:~# kubectl describe deploy busy\nName: busy\nNamespace: default\nCreationTimestamp: Sun, 19 Aug 2018 12:27:19 +0800\nLabels: run=busy\nAnnotations: deployment.kubernetes.io/revision=2\nSelector: run=busy\nReplicas: 10 desired | 10 updated | 10 total | 10 available | 0 unavailable\nStrategyType: RollingUpdate\nMinReadySeconds: 0\nRollingUpdateStrategy: 1 max unavailable, 1 max surge\nPod Template:\n Labels: run=busy\n Containers:\n busy:\n Image: busybox:1.29\n Port: \n Host Port: \n Args:\n sleep\n 3600000\n Environment: \n Mounts: \n Volumes: \nConditions:\n Type Status Reason\n ---- ------ ------\n Available True MinimumReplicasAvailable\n Progressing True NewReplicaSetAvailable\nOldReplicaSets: \nNewReplicaSet: busy-84cb46955d (10/10 replicas created)\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal ScalingReplicaSet 1m deployment-controller Scaled up replica set busy-9669c8599 to 10\n Normal ScalingReplicaSet 46s deployment-controller Scaled up replica set busy-84cb46955d to 1\n Normal ScalingReplicaSet 46s deployment-controller Scaled down replica set busy-9669c8599 to 9\n Normal ScalingReplicaSet 46s deployment-controller Scaled up replica set busy-84cb46955d to 2\n Normal ScalingReplicaSet 43s deployment-controller Scaled down replica set busy-9669c8599 to 8\n Normal ScalingReplicaSet 43s deployment-controller Scaled up replica set busy-84cb46955d to 3\n Normal ScalingReplicaSet 43s deployment-controller Scaled down replica set busy-9669c8599 to 7\n Normal ScalingReplicaSet 43s deployment-controller Scaled up replica set busy-84cb46955d to 4\n Normal ScalingReplicaSet 40s deployment-controller Scaled down replica set busy-9669c8599 to 6\n Normal ScalingReplicaSet 28s (x12 over 40s) deployment-controller (combined from similar events): Scaled down replica set busy-9669c8599 to 0\n```\n整个滚动过程是通过控制两个副本集来完成的，新的副本集：busy-84cb46955d；旧的副本集：busy-9669c8599 。\n理想状态下的滚动过程：\n>1. 创建新副本集，并为其分配1个新版本的pod。\n>2. 通知旧副本集，销毁1个旧版本的pod。\n>3. 当旧副本销毁成功后，通知新副本集，再新增1个新版本的pod；当新副本创建成功后，通知旧副本再减少1个pod。\n>只要销毁成功，新副本集就会创造新的pod，一直循环，直到旧的副本集pod数量为0。\n### 5.4 总结\n**`无论理想还是不理想，k8s最终都会使应用程序全部更新到期望状态，都会始终保持最大的副本总数和可用副本总数的不变性！！！`**\n\n[阅读原文](http://www.cnblogs.com/justmine/p/8688828.html)\n\n"
},
{
"path": "docs/mixes/DoneList.md",
"content": "## 前言\n\n`kubeasz`项目开始于`2017.11`，半年多时间以来，从最开始单一的ansible部署脚本朝着提供部署高可用 K8S集群的完整解决方案的目标不断前进，接下去项目的发展需要各位的共同参与和贡献，希望越做越好，为国内k8s学习、实践者提供更多帮助。 \n\n### 项目已完成部分 \n\n\n \n 类型 \n 描述 \n 备注 \n \n \n 集群部署 \n 服务器基础安全加固与参数优化 \n 已完成 \n \n \n 基础服务 \n 集群监控告警-prometheus \n 已完成基础，待优化 \n \n \n 应用服务 \n jenkins集成 \n 已完成 \n \n \n 集群部署 \n kube-router网络插件 \n 已完成 \n \n \n 基础服务 \n metrics server \n 已完成 \n \n \n 集群部署 \n ipvs代理模式跟进 \n 已完成 \n \n \n 集群部署 \n cilium网络插件 \n 已完成 \n \n \n 集群部署 \n 集群内时间同步-Chrony \n 已完成 \n \n
\n\n\n"
},
{
"path": "docs/mixes/HowToContribute.md",
"content": "# 为项目`kubeasz`提交`pull request`\n\n首先请核对下本地git config配置的用户名和邮箱与你github上的注册用户和邮箱一致，否则即使`pull request`被接受，贡献者列表中也看不到自己的名字，设置命令：\n\n``` bash\n$ git config --global user.email \"you@example.com\"\n$ git config --global user.name \"Your Name\"\n```\n\n- 1.登录github，在本项目页面点击`fork`到自己仓库\n- 2.clone 自己的仓库到本地：`git clone https://github.com/xxx/kubeasz.git`\n- 3.在 master 分支添加原始仓库为上游分支：`git remote add upstream https://github.com/easzlab/kubeasz.git`\n- 4.在本地新建开发分支：`git checkout -b dev`\n- 5.在开发分支修改代码并提交：`git add .`, `git commit -am 'xx变更说明'`\n- 6.切换至 master 分支，同步原始仓库：`git checkout master`， `git pull upstream master`\n- 7.切换至 dev 分支，合并本地 master 分支（已经和原始仓库同步），可能需要解冲突：`git checkout dev`, `git merge master`\n- 8.提交本地 dev 分支到自己的远程 dev 仓库：`git push origin dev`\n- 9.在github自己仓库页面，点击`Compare & pull request`给原始仓库发 pull request 请求\n- a.等待原作者回复（接受/拒绝）\n"
},
{
"path": "docs/mixes/LICENSE",
"content": "Apache License\nVersion 2.0, January 2004\nhttp://www.apache.org/licenses/\n\nTERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n1. Definitions.\n\n\"License\" shall mean the terms and conditions for use, reproduction, and\ndistribution as defined by Sections 1 through 9 of this document.\n\n\"Licensor\" shall mean the copyright owner or entity authorized by the copyright\nowner that is granting the License.\n\n\"Legal Entity\" shall mean the union of the acting entity and all other entities\nthat control, are controlled by, or are under common control with that entity.\nFor the purposes of this definition, \"control\" means (i) the power, direct or\nindirect, to cause the direction or management of such entity, whether by\ncontract or otherwise, or (ii) ownership of fifty percent (50%) or more of the\noutstanding shares, or (iii) beneficial ownership of such entity.\n\n\"You\" (or \"Your\") shall mean an individual or Legal Entity exercising\npermissions granted by this License.\n\n\"Source\" form shall mean the preferred form for making modifications, including\nbut not limited to software source code, documentation source, and configuration\nfiles.\n\n\"Object\" form shall mean any form resulting from mechanical transformation or\ntranslation of a Source form, including but not limited to compiled object code,\ngenerated documentation, and conversions to other media types.\n\n\"Work\" shall mean the work of authorship, whether in Source or Object form, made\navailable under the License, as indicated by a copyright notice that is included\nin or attached to the work (an example is provided in the Appendix below).\n\n\"Derivative Works\" shall mean any work, whether in Source or Object form, that\nis based on (or derived from) the Work and for which the editorial revisions,\nannotations, elaborations, or other modifications represent, as a whole, an\noriginal work of authorship. For the purposes of this License, Derivative Works\nshall not include works that remain separable from, or merely link (or bind by\nname) to the interfaces of, the Work and Derivative Works thereof.\n\n\"Contribution\" shall mean any work of authorship, including the original version\nof the Work and any modifications or additions to that Work or Derivative Works\nthereof, that is intentionally submitted to Licensor for inclusion in the Work\nby the copyright owner or by an individual or Legal Entity authorized to submit\non behalf of the copyright owner. For the purposes of this definition,\n\"submitted\" means any form of electronic, verbal, or written communication sent\nto the Licensor or its representatives, including but not limited to\ncommunication on electronic mailing lists, source code control systems, and\nissue tracking systems that are managed by, or on behalf of, the Licensor for\nthe purpose of discussing and improving the Work, but excluding communication\nthat is conspicuously marked or otherwise designated in writing by the copyright\nowner as \"Not a Contribution.\"\n\n\"Contributor\" shall mean Licensor and any individual or Legal Entity on behalf\nof whom a Contribution has been received by Licensor and subsequently\nincorporated within the Work.\n\n2. Grant of Copyright License.\n\nSubject to the terms and conditions of this License, each Contributor hereby\ngrants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free,\nirrevocable copyright license to reproduce, prepare Derivative Works of,\npublicly display, publicly perform, sublicense, and distribute the Work and such\nDerivative Works in Source or Object form.\n\n3. Grant of Patent License.\n\nSubject to the terms and conditions of this License, each Contributor hereby\ngrants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free,\nirrevocable (except as stated in this section) patent license to make, have\nmade, use, offer to sell, sell, import, and otherwise transfer the Work, where\nsuch license applies only to those patent claims licensable by such Contributor\nthat are necessarily infringed by their Contribution(s) alone or by combination\nof their Contribution(s) with the Work to which such Contribution(s) was\nsubmitted. If You institute patent litigation against any entity (including a\ncross-claim or counterclaim in a lawsuit) alleging that the Work or a\nContribution incorporated within the Work constitutes direct or contributory\npatent infringement, then any patent licenses granted to You under this License\nfor that Work shall terminate as of the date such litigation is filed.\n\n4. Redistribution.\n\nYou may reproduce and distribute copies of the Work or Derivative Works thereof\nin any medium, with or without modifications, and in Source or Object form,\nprovided that You meet the following conditions:\n\nYou must give any other recipients of the Work or Derivative Works a copy of\nthis License; and\nYou must cause any modified files to carry prominent notices stating that You\nchanged the files; and\nYou must retain, in the Source form of any Derivative Works that You distribute,\nall copyright, patent, trademark, and attribution notices from the Source form\nof the Work, excluding those notices that do not pertain to any part of the\nDerivative Works; and\nIf the Work includes a \"NOTICE\" text file as part of its distribution, then any\nDerivative Works that You distribute must include a readable copy of the\nattribution notices contained within such NOTICE file, excluding those notices\nthat do not pertain to any part of the Derivative Works, in at least one of the\nfollowing places: within a NOTICE text file distributed as part of the\nDerivative Works; within the Source form or documentation, if provided along\nwith the Derivative Works; or, within a display generated by the Derivative\nWorks, if and wherever such third-party notices normally appear. The contents of\nthe NOTICE file are for informational purposes only and do not modify the\nLicense. You may add Your own attribution notices within Derivative Works that\nYou distribute, alongside or as an addendum to the NOTICE text from the Work,\nprovided that such additional attribution notices cannot be construed as\nmodifying the License.\nYou may add Your own copyright statement to Your modifications and may provide\nadditional or different license terms and conditions for use, reproduction, or\ndistribution of Your modifications, or for any such Derivative Works as a whole,\nprovided Your use, reproduction, and distribution of the Work otherwise complies\nwith the conditions stated in this License.\n\n5. Submission of Contributions.\n\nUnless You explicitly state otherwise, any Contribution intentionally submitted\nfor inclusion in the Work by You to the Licensor shall be under the terms and\nconditions of this License, without any additional terms or conditions.\nNotwithstanding the above, nothing herein shall supersede or modify the terms of\nany separate license agreement you may have executed with Licensor regarding\nsuch Contributions.\n\n6. Trademarks.\n\nThis License does not grant permission to use the trade names, trademarks,\nservice marks, or product names of the Licensor, except as required for\nreasonable and customary use in describing the origin of the Work and\nreproducing the content of the NOTICE file.\n\n7. Disclaimer of Warranty.\n\nUnless required by applicable law or agreed to in writing, Licensor provides the\nWork (and each Contributor provides its Contributions) on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,\nincluding, without limitation, any warranties or conditions of TITLE,\nNON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are\nsolely responsible for determining the appropriateness of using or\nredistributing the Work and assume any risks associated with Your exercise of\npermissions under this License.\n\n8. Limitation of Liability.\n\nIn no event and under no legal theory, whether in tort (including negligence),\ncontract, or otherwise, unless required by applicable law (such as deliberate\nand grossly negligent acts) or agreed to in writing, shall any Contributor be\nliable to You for damages, including any direct, indirect, special, incidental,\nor consequential damages of any character arising as a result of this License or\nout of the use or inability to use the Work (including but not limited to\ndamages for loss of goodwill, work stoppage, computer failure or malfunction, or\nany and all other commercial damages or losses), even if such Contributor has\nbeen advised of the possibility of such damages.\n\n9. Accepting Warranty or Additional Liability.\n\nWhile redistributing the Work or Derivative Works thereof, You may choose to\noffer, and charge a fee for, acceptance of support, warranty, indemnity, or\nother liability obligations and/or rights consistent with this License. However,\nin accepting such obligations, You may act only on Your own behalf and on Your\nsole responsibility, not on behalf of any other Contributor, and only if You\nagree to indemnify, defend, and hold each Contributor harmless for any liability\nincurred by, or claims asserted against, such Contributor by reason of your\naccepting any such warranty or additional liability.\n\nEND OF TERMS AND CONDITIONS\n\nAPPENDIX: How to apply the Apache License to your work\n\nTo apply the Apache License to your work, attach the following boilerplate\nnotice, with the fields enclosed by brackets \"{}\" replaced with your own\nidentifying information. (Don't include the brackets!) The text should be\nenclosed in the appropriate comment syntax for the file format. We also\nrecommend that a file or class name and description of purpose be included on\nthe same \"printed page\" as the copyright notice for easier identification within\nthird-party archives.\n\n Copyright 2017 jmgao\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License."
},
{
"path": "docs/mixes/conformance.md",
"content": "# 关于K8S集群一致性认证\n\nCNCF 一致性认证项目(https://github.com/cncf/k8s-conformance) 可以很方便帮助k8s搭建者和用户确认集群各项功能符合预期，既符合k8s设计标准。\n\n# kubeasz 通过一致性测试\n\n自kubeasz 3.0.0 版本，k8s v1.20.2开始，正式通过cncf一致性认证，成为cncf 官方认证安装工具；后续k8s主要版本发布或者kubeasz有大版本更新，会优先确保通过集群一致性认证。\n\n- v1.34 [进行中]()\n- v1.33 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.33/kubeasz)\n- v1.32 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.32/kubeasz)\n- v1.31 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.31/kubeasz)\n- v1.30 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.30/kubeasz)\n- v1.29 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.29/kubeasz)\n- v1.28 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.28/kubeasz)\n- v1.27 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.27/kubeasz)\n- v1.26 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.26/kubeasz)\n- v1.25 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.25/kubeasz)\n- v1.24 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.24/kubeasz)\n- v1.23 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.23/kubeasz)\n- v1.22 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.22/kubeasz)\n- v1.21 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.21/kubeasz)\n- v1.20 [已认证](https://github.com/cncf/k8s-conformance/tree/master/v1.20/kubeasz)\n\n\n## Conformance Test\n\n按照测试文档，注意以下几点：\n\n1.解决qiang的问题，可以临时去国外公有云创建集群，然后运行测试项目。\n\n2.集群要保障资源，建议3个节点\n\n3.网络组件选择calico，其他组件可能有bug导致特定测试项失败\n\n\n# 附：测试流程\n\n## Node Provisioning\n\nProvision 3 nodes for your cluster (OS: Ubuntu 20.04)\n\n1 master node (4c16g)\n\n2 worker node (4c16g)\n\nfor a High-Availability Kubernetes Cluster, read [more](https://github.com/easzlab/kubeasz/blob/master/docs/setup/00-planning_and_overall_intro.md)\n\n## Install the cluster\n\n(1) Download 'kubeasz' code, the binaries and offline images\n\n```\nexport release=3.2.0\ncurl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown\nchmod +x ./ezdown\n./ezdown -D -m standard\n```\n\n(2) install an all-in-one cluster\n\n```\n./ezdown -S\nsource ~/.bashrc\ndk ezctl start-aio\n```\n\n(3) Add two worker nodes\n\n```\nssh-copy-id ${worker1_ip}\ndk ezctl add-node default ${worker1_ip}\n\nssh-copy-id ${worker2_ip}\ndk ezctl add-node default ${worker2_ip}\n```\n\n## Run Conformance Test\n\nThe standard tool for running these tests is\n[Sonobuoy](https://github.com/heptio/sonobuoy). Sonobuoy is\nregularly built and kept up to date to execute against all\ncurrently supported versions of kubernetes.\n\nDownload a [binary release](https://github.com/heptio/sonobuoy/releases) of the CLI\n\nDeploy a Sonobuoy pod to your cluster with:\n\n```\n$ sonobuoy run --plugin-env=e2e.E2E_EXTRA_ARGS=\"--ginkgo.v\" --mode=certified-conformance \n```\n\n**NOTE:** You can run the command synchronously by adding the flag `--wait` but be aware that running the Conformance tests can take an hour or more.\n\nView actively running pods:\n\n```\n$ sonobuoy status\n```\n\nTo inspect the logs:\n\n```\n$ sonobuoy logs\n```\n\nOnce `sonobuoy status` shows the run as `completed`, copy the output directory from the main Sonobuoy pod to a local directory:\n\n```\n$ outfile=$(sonobuoy retrieve)\n```\n\nThis copies a single `.tar.gz` snapshot from the Sonobuoy pod into your local\n`.` directory. Extract the contents into `./results` with:\n\n```\nmkdir ./results; tar xzf $outfile -C ./results\n```\n\n**NOTE:** The two files required for submission are located in the tarball under **plugins/e2e/results/{e2e.log,junit.xml}**.\n\nTo clean up Kubernetes objects created by Sonobuoy, run:\n\n```\nsonobuoy delete\n```\n"
},
{
"path": "docs/mixes/donate.md",
"content": "# 捐赠\n\n如果觉得本项目对您有帮助，请小小鼓励下项目作者，谢谢！\n\n支付宝码(左)和微信钱包码(右)\n\n \n\n"
},
{
"path": "docs/op/ch_apiserver_cert.md",
"content": "# 修改 APISERVER（MASTER）证书\n\n`kubeasz` 创建集群后，APISERVER（MASTER）证书默认 CN 包含如下`域名`和`IP`：参见`roles/kube-master/templates/kubernetes-csr.json.j2`\n\n```\n \"hosts\": [\n \"127.0.0.1\",\n{% if groups['ex_lb']|length > 0 %}\n \"{{ hostvars[groups['ex_lb'][0]]['EX_APISERVER_VIP'] }}\",\n{% endif %}\n{% for host in groups['kube_master'] %}\n \"{{ host }}\",\n{% endfor %}\n \"{{ CLUSTER_KUBERNETES_SVC_IP }}\",\n{% for host in MASTER_CERT_HOSTS %}\n \"{{ host }}\",\n{% endfor %}\n \"kubernetes\",\n \"kubernetes.default\",\n \"kubernetes.default.svc\",\n \"kubernetes.default.svc.cluster\",\n \"kubernetes.default.svc.cluster.local\"\n ],\n```\n\n有的时候（比如apiserver地址通过边界防火墙的NAT转换成公网IP访问，或者需要添加公网域名访问）我们需要在 APISERVER（MASTER）证书中添加一些`域名`或者`IP`，可以方便操作如下：\n\n## 1.修改配置文件`/etc/kubeasz/clusters/${集群名}/config.yaml`\n\n``` bash\n# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）\nMASTER_CERT_HOSTS:\n - \"10.1.1.1\"\n - \"k8s.test.io\"\n #- \"www.test.com\"\n```\n\n## 2.执行新证书生成并重启apiserver\n\n``` bash\n$ ezctl setup ${集群名} 04 -t change_cert,restart_master \n```\n"
},
{
"path": "docs/op/cluster_restore.md",
"content": "# K8S 集群备份与恢复\n\n虽然 K8S 集群可以配置成多主多节点的高可用的部署，还是有必要了解下集群的备份和容灾恢复能力；在高可用k8s集群中 etcd集群保存了整个集群的状态，因此这里的备份与恢复重点就是：\n\n- 从运行的etcd集群备份数据到磁盘文件\n- 从etcd备份文件恢复数据，从而使集群恢复到备份时状态\n\n## 备份与恢复操作说明\n\n- 1.首先搭建一个测试集群，部署几个测试deployment，验证集群各项正常后，进行一次备份(假设集群名为k8s-01)：\n\n``` bash\n$ ezctl backup k8s-01\n# 或者如下手动执行ansible命令\n# ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/94.backup.yml\n```\n\n执行完毕可以在部署主机的备份目录下检查备份情况，示例如下：\n\n```\n/etc/kubeasz/clusters/k8s-01/backup/\n├── snapshot_202106201205.db\n├── snapshot_202106211406.db\n└── snapshot.db\n```\n其中，snapshot.db始终为最近一次备份文件\n\n- 2.模拟误删除操作（略）\n\n- 3.恢复集群及验证\n\n可以在 `roles/cluster-restore/defaults/main.yml` 文件中配置需要恢复的 etcd备份版本（从上述备份目录中选取），默认使用最近一次备份；执行恢复后，需要一定时间等待 pod/svc 等资源恢复重建。\n\n``` bash\n$ ezctl restore k8s-01\n# 或者如下手动执行ansible命令\n# ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/95.restore.yml\n```\n如果集群主要组件（master/etcd/node）等出现不可恢复问题，可以尝试使用如下步骤 [清理]() --> [创建]() --> [恢复]()\n\n``` bash\n$ ezctl clean k8s-01\n# 或者如下手动执行ansible命令\n# ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/99.clean.yml\n$ ezctl setup k8s-01 01\n$ ezctl setup k8s-01 02\n$ ezctl setup k8s-01 03\n$ ezctl setup k8s-01 04\n$ ezctl setup k8s-01 05\n...\n$ ezctl restore k8s-01\n# ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml playbooks/95.restore.yml\n```\n\n## 参考\n\n- https://etcd.io/docs/v3.4/op-guide/recovery/\n"
},
{
"path": "docs/op/force_ch_certs.md",
"content": "# 强制更新CA和所有证书\n\n- WARNNING: 此命令使用需要小心谨慎，确保了解功能背景和可能的结果；执行后，它会重新创建集群CA证书以及由它颁发的所有其他证书；一般适合于集群admin.conf不小心泄露，为了避免集群被非法访问，重新创建CA，从而使已泄漏的admin.conf失效。\n\n- 如果需要分发受限的kubeconfig，强烈建议使用[自定义权限和期限的kubeconfig](kcfg-adm.md)\n\n## 使用帮助\n\n确认需要强制更新后，在ansible 控制节点使用如下命令：(xxx 表示需要操作的集群名)\n\n``` bash\ndocker exec -it kubeasz ezctl kca-renew xxx\n# 或者使用 dk ezctl kca-renew xxx\n```\n\n上述命令执行后，按序进行以下的操作：详见`playbooks/96.update-certs.yml`\n\n- 重新生成CA证书，以及各种kubeconfig\n- 签发新etcd证书，并使用新证书重启etcd服务\n- 签发新kube-apiserver 证书，并重启kube-apiserver/kube-controller-manager/kube-scheduler 服务\n- 签发新kubelet 证书，并重启kubelet/kube-proxy 服务\n- 重启网络组件pod\n- 重启其他集群组件pod\n\n- **特别注意：** 如果集群中运行的业务负载pod需要访问apiserver，需要重启这些pod\n\n## 检查验证\n\n更新完毕，注意检查集群组件日志和容器pod日志，确认集群处于正常状态\n\n- 集群组件日志：使用journalctl -u xxxx.service -f 依次检查 etcd.service/kube-apiserver.service/kube-controller-manager.service/kube-scheduler.service/kubelet.service/kube-proxy.service\n- 容器pod日志：使用 kubectl logs 方式检查容器日志\n"
},
{
"path": "docs/op/kcfg-adm.md",
"content": "# 管理客户端kubeconfig \n\n默认 k8s集群安装成功后生成客户端kubeconfig，它拥有集群管理的所有权限（不要将这个admin权限、50年期限的kubeconfig流露出去）；而我们经常需要将限定权限、限定期限的kubeconfig 分发给普通用户；利用cfssl签发自定义用户证书和k8s灵活的rbac权限绑定机制，ezctl 工具封装了这个功能。\n\n## 使用帮助\n\n```\nezctl help kcfg-adm\nUsage: ezctl kcfg-adm \navailable :\n -A to add a client kubeconfig with a newly created user\n -D to delete a client kubeconfig with the existed user\n -L to list all of the users\n -e to set expiry of the user certs in hours (ex. 24h, 8h, 240h)\n -t to set a user-type (admin or view)\n -u to set a user-name prefix\n\nexamples: ./ezctl kcfg-adm test-k8s -L\n ./ezctl kcfg-adm default -A -e 240h -t admin -u jack\n ./ezctl kcfg-adm default -D -u jim-202101162141\n```\n\n- 可以设置过期时间\n- 可以设置权限：管理员权限（admin）和只读权限（view）\n\n## 使用举例\n\n- 1.查看集群k8s-01当前自定义kubeconfig\n\n```\nezctl kcfg-adm k8s-01 -L\n2021-01-24 16:32:43 INFO list-kcfg k8s-01\n2021-01-24 16:32:43 INFO list-kcfg in cluster:k8s-01\n\nUSER TYPE EXPIRY(+8h if in Asia/Shanghai)\n---------------------------------------------------------------------------------\n\n2021-01-24 16:32:43 INFO list-kcfg k8s-01 success\n```\n初始情况下列表为空\n\n- 2.增加集群k8s-01一个自定义用户kubeconfig，用户名user01，期限24h，只读权限\n\n```\nezctl kcfg-adm k8s-01 -A -u user01 -e 24h -t view\n2021-01-24 17:32:33 INFO add-kcfg k8s-01\n2021-01-24 17:32:33 INFO add-kcfg in cluster:k8s-01 with user:user01-202101241732\n\nPLAY [localhost] *****************************************************************************************************\n\n...（此处省略输出） \n\nTASK [deploy : debug] ************************************************************************************************\nok: [localhost] => {\n \"msg\": \"查看user01-202101241732自定义kubeconfig：/etc/kubeasz/clusters/k8s-01/ssl/users/user01-202101241732.kubeconfig\"\n}\n\nPLAY RECAP ***********************************************************************************************************\nlocalhost : ok=12 changed=10 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0\n\n2021-01-24 17:32:41 INFO add-kcfg k8s-01 success\n```\n生成的kubeconfig位于 /etc/kubeasz/clusters/k8s-01/ssl/users/user01-202101241732.kubeconfig\n\n- 3.再增加一个用户user02，期限240h，admin权限\n\n```\nezctl kcfg-adm k8s-01 -A -u user02 -e 240h -t admin\n2021-01-24 18:38:47 INFO add-kcfg k8s-01\n2021-01-24 18:38:47 INFO add-kcfg in cluster:k8s-01 with user:user02-202101241838\n\nPLAY [localhost] *****************************************************************************************************\n\n...（此处省略输出）\n\nTASK [deploy : debug] ************************************************************************************************\nok: [localhost] => {\n \"msg\": \"查看user02-202101241838自定义kubeconfig：/etc/kubeasz/clusters/k8s-01/ssl/users/user02-202101241838.kubeconfig\"\n}\n\nPLAY RECAP ***********************************************************************************************************\nlocalhost : ok=12 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0\n\n2021-01-24 18:38:55 INFO add-kcfg k8s-01 success\n```\n\n- 4.再次查看集群k8s-01当前自定义kubeconfig\n\n```\nezctl kcfg-adm k8s-01 -L\n2021-01-24 18:40:30 INFO list-kcfg k8s-01\n2021-01-24 18:40:30 INFO list-kcfg in cluster:k8s-01\n\nUSER TYPE EXPIRY(+8h if in Asia/Shanghai)\n---------------------------------------------------------------------------------\nuser02-202101241838 cluster-admin 2021-02-03T10:34:00Z\nuser01-202101241732 view 2021-01-25T09:28:00Z\n\n2021-01-24 18:40:31 INFO list-kcfg k8s-01 success\n```\n\n- 5.删除user01-202101241732 权限\n\n``` bash\nezctl kcfg-adm k8s-01 -D -u user01-202101241732\n2021-01-24 21:41:50 INFO del-kcfg k8s-01\n2021-01-24 21:41:50 INFO del-kcfg in cluster:k8s-01 with user:user01-202101241732\nclusterrolebinding.rbac.authorization.k8s.io \"crb-user01-202101241732\" deleted\n2021-01-24 21:41:50 INFO del-kcfg k8s-01 success\n\nezctl kcfg-adm k8s-01 -L\n2021-01-24 21:42:02 INFO list-kcfg k8s-01\n2021-01-24 21:42:02 INFO list-kcfg in cluster:k8s-01\n\nUSER TYPE EXPIRY(+8h if in Asia/Shanghai)\n---------------------------------------------------------------------------------\nuser02-202101241838 cluster-admin 2021-02-03T10:34:00Z\n\n2021-01-24 21:42:02 INFO list-kcfg k8s-01 success\n```\n"
},
{
"path": "docs/op/loadballance_ingress_nodeport.md",
"content": "# 配置负载转发 ingress nodeport\n\n向集群外暴露 ingress-controller 本身的服务端口（80/443/8080）一般有以下三种方法：\n\n- 1.部署ingress-controller时使用`hostNetwork: true`，这样就可以直接使用上述端口，可能与host已listen端口冲突\n- 2.部署ingress-controller时使用`LoadBalancer`类型服务，需要集群支持`LoadBalancer`\n- 3.部署ingress-controller时使用`nodePort`类型服务，然后在集群外使用 haproxy/f5 等配置 virtual server 集群\n\n本文档讲解使用 haproxy 配置 ingress的 VS 集群，前提是配置了自建`ex_lb`节点\n\n## 1.配置 ex_lb 参数开启转发 ingress nodeport\n\n``` bash\n# 编辑 roles/ex-lb/defaults/main.yml，配置如下变量\nINGRESS_NODEPORT_LB: \"yes\"\nINGRESS_TLS_NODEPORT_LB: \"yes\"\n```\n\n## 2.重新配置启动LB节点服务\n\n``` bash\n$ ezctl setup ${集群名} ex-lb \n```\n\n## 3.验证 ex_lb 节点的 haproxy 服务配置 `/etc/haproxy/haproxy.cfg` 包含如下配置\n\n``` bash\n... 前文省略\nlisten kube_master\n bind 0.0.0.0:8443\n mode tcp\n option tcplog\n balance roundrobin\n server 192.168.1.1 192.168.1.1:6443 check inter 2000 fall 2 rise 2 weight 1\n server 192.168.1.2 192.168.1.2:6443 check inter 2000 fall 2 rise 2 weight 1\n\nlisten ingress-node\n bind 0.0.0.0:80\n mode tcp\n option tcplog\n balance roundrobin\n server 192.168.1.3 192.168.1.3:23456 check inter 2000 fall 2 rise 2 weight 1\n server 192.168.1.4 192.168.1.4:23456 check inter 2000 fall 2 rise 2 weight 1\n\nlisten ingress-node-tls\n bind 0.0.0.0:443\n mode tcp\n option tcplog\n balance roundrobin\n server 192.168.1.3 192.168.1.3:23457 check inter 2000 fall 2 rise 2 weight 1\n server 192.168.1.4 192.168.1.4:23457 check inter 2000 fall 2 rise 2 weight 1\n```\n\n验证成功后，我们可以方便的去做[配置ingress](../guide/ingress.md)和[配置https ingress](../guide/ingress-tls.md)实验了。\n"
},
{
"path": "docs/op/op-etcd.md",
"content": "# 管理 etcd 集群\n\nEtcd 集群支持在线改变集群成员节点，可以增加、修改、删除成员节点；不过改变成员数量仍旧需要满足集群成员多数同意原则（quorum），另外请记住集群成员数量变化的影响：\n\n- 注意：如果etcd 集群有故障节点，务必先删除故障节点，然后添加新节点，[参考FAQ](https://etcd.io/docs/v3.4.0/faq/)\n- 增加 etcd 集群节点, 提高集群稳定性\n- 增加 etcd 集群节点, 提高集群读性能（所有节点数据一致，客户端可以从任意节点读取数据）\n- 增加 etcd 集群节点, 降低集群写性能（所有节点数据一致，每一次写入会需要所有节点数据同步）\n\n## 备份 etcd 数据\n\n1. 手动在任意正常 etcd 节点上执行备份：\n\n``` bash\n# snapshot备份\n$ ETCDCTL_API=3 etcdctl snapshot save backup.db\n# 查看备份\n$ ETCDCTL_API=3 etcdctl --write-out=table snapshot status backup.db\n```\n\n2. 使用 kubeasz 备份\n_cluster_name_ 为 k8s-01\n\n``` bash \nezctl backup k8s-01\n```\n\n使用 crontab 定时备份示例(使用 容器化的 kubeasz，每日01:01 备份)\n```\n1 1 * * * /usr/bin/docker exec -i kubeasz ezctl backup k8s-01\n```\n\n备份文件在 \n\n```\n{{ base_dir }}/clusters/k8s-01/backup\n```\n\n## etcd 集群节点操作\n\n执行如下 (假设待操作节点为 192.168.1.11，集群名称test-k8s)：\n\n- 增加 etcd 节点：\n\n``` bash\n# ssh 免密码登录\n$ ssh-copy-id 192.168.1.11\n\n# 新增节点\n$ ezctl add-etcd test-k8s 192.168.1.11\n```\n\n- 删除 etcd 节点：`$ ezctl del-etcd test-k8s 192.168.1.11`\n\n具体操作流程参考 ezctl中 add-etcd/del-etcd 相关函数和playbooks/ 目录的操作剧本\n\n### 验证 etcd 集群\n\n``` bash\n# 登录任意etcd节点验证etcd集群状态\n$ export ETCDCTL_API=3 \n$ etcdctl member list\n\n# 验证所有etcd节点服务状态和日志\n$ systemctl status etcd\n$ journalctl -u etcd -f\n```\n\n## 参考\n\n- 官方文档 https://etcd.io/docs/v3.5/op-guide/runtime-configuration/ \n"
},
{
"path": "docs/op/op-index.md",
"content": "# 集群运维管理指南 operation guide\n\n- [管理 NODE 节点](op-node.md)\n- [管理 MASTER 节点](op-master.md)\n- [管理 ETCD 节点](op-etcd.md)\n- [升级 K8S 版本](upgrade.md)\n- [集群备份与恢复](cluster_restore.md)\n- [管理分发用户 kubeconfig](kcfg-adm.md)\n- [修改 APISERVER 证书](ch_apiserver_cert.md)\n- [强制更新CA和所有证书](force_ch_certs.md)\n- [配置负载转发 ingress nodeport](loadballance_ingress_nodeport.md)\n"
},
{
"path": "docs/op/op-master.md",
"content": "# 管理 kube_master 节点\n\n## 1.增加 kube_master 节点\n\n新增`kube_master`节点大致流程为：(参考ezctl 中add-master函数和playbooks/23.addmaster.yml)\n- [可选]新节点安装 chrony 时间同步\n- 新节点预处理 prepare\n- 新节点安装 container runtime \n- 新节点安装 kube_master 服务\n- 新节点安装 kube_node 服务\n- 新节点安装网络插件相关\n- 禁止业务 pod调度到新master节点\n- 更新 node 节点 haproxy 负载均衡并重启\n\n### 操作步骤\n\n执行如下 (假设待增加节点为 192.168.1.11, 集群名称test-k8s)：\n\n``` bash\n# ssh 免密码登录\n$ ssh-copy-id 192.168.1.11\n\n# 新增节点\n$ ezctl add-master test-k8s 192.168.1.11\n\n# 同理，重复上面步骤再新增节点并自定义nodename\n$ ezctl add-master test-k8s 192.168.1.12 k8s_nodename=master-03\n```\n\n### 验证\n\n``` bash\n# 在新节点master 服务状态\n$ systemctl status kube-apiserver \n$ systemctl status kube-controller-manager\n$ systemctl status kube-scheduler\n\n# 查看新master的服务日志\n$ journalctl -u kube-apiserver -f\n\n# 查看集群节点，可以看到新 master节点 Ready, 并且禁止了POD 调度功能\n$ kubectl get node\nNAME STATUS ROLES AGE VERSION\n192.168.1.1 Ready,SchedulingDisabled 3h v1.9.3\n192.168.1.2 Ready,SchedulingDisabled 3h v1.9.3\n192.168.1.3 Ready 3h v1.9.3\n192.168.1.4 Ready 3h v1.9.3\n192.168.1.11 Ready,SchedulingDisabled 2h v1.9.3\t# 新增 master节点\n```\n\n## 2.删除 kube_master 节点\n\n\n删除`kube_master`节点大致流程为：(参考ezctl 中del-master函数和playbooks/33.delmaster.yml)\n- 检测是否可以删除\n- 迁移节点 pod\n- 删除 master 相关服务及文件\n- 删除 node 相关服务及文件\n- 从集群删除 node 节点\n- 从 ansible hosts 移除节点\n- 在 ansible 控制端更新 kubeconfig\n- 更新 node 节点 haproxy 配置\n\n### 操作步骤\n\n``` bash\n$ ezctl del-master test-k8s 192.168.1.11 # 假设待删除节点 192.168.1.11\n```\n\n### 验证\n\n略\n\n"
},
{
"path": "docs/op/op-node.md",
"content": "# 管理 node 节点\n\n目录\n- 1.增加 kube_node 节点\n- 2.增加非标准ssh端口节点\n- 3.删除 kube_node 节点\n\n## 1.增加 kube_node 节点\n\n新增`kube_node`节点大致流程为：(参考ezctl 里面add-node函数 和 playbooks/22.addnode.yml)\n- [可选]新节点安装 chrony 时间同步\n- 新节点预处理 prepare\n- 新节点安装 container runtime\n- 新节点安装 kube_node 服务\n- 新节点安装网络插件相关\n\n### 操作步骤\n\n执行如下 (假设待增加节点为 192.168.1.11，k8s集群名为 test-k8s)：\n\n``` bash\n# ssh 免密码登录\n$ ssh-copy-id 192.168.1.11\n\n# 新增节点\n$ ezctl add-node test-k8s 192.168.1.11\n\n# 同理，重复上面步骤再新增节点并自定义nodename\n$ ezctl add-node test-k8s 192.168.1.12 k8s_nodename=worker-03\n```\n\n### 验证\n\n``` bash\n# 验证新节点状态\n$ kubectl get node\n\n# 验证新节点的网络插件calico 或flannel 的Pod 状态\n$ kubectl get pod -n kube-system\n\n# 验证新建pod能否调度到新节点，略\n```\n\n## 2.增加非标准ssh端口节点\n\n假设待添加节点192.168.2.1，ssh 端口 10022；然后执行 \n\n``` bash\n$ ssh-copy-id -p 10022 192.168.2.1\n$ ezctl add-node test-k8s 192.168.2.1 ansible_ssh_port=10022\n```\n\n- 注意：如果在添加节点时需要设置其他个性化变量，可以同理在后面不断添加\n\n\n## 3.删除 kube_node 节点\n\n删除 node 节点流程：(参考ezctl 里面del-node函数 和 playbooks/32.delnode.yml)\n- 检测是否可以删除\n- 迁移节点上的 pod\n- 删除 node 相关服务及文件\n- 从集群删除 node\n\n### 操作步骤\n\n``` bash\n$ ezctl del-node test-k8s 192.168.1.11 # 假设待删除节点为 192.168.1.11\n```\n\n### 验证\n\n略\n"
},
{
"path": "docs/op/upgrade.md",
"content": "## k8s 集群升级\n\n集群升级存在一定风险，请谨慎操作。 \n\n- 支持k8s相同大版本基础上升级任意小版本，比如当前安装集群为1.25.0，你可以方便的升级到任何1.25.x版本\n- 不建议跨大版本升级，一般大版本更新时k8s api有一些变动\n\n### 备份etcd数据\n\n- 自动备份\n\n`kubeasz`项目也可以如下方便执行备份（假设集群名为k8s-01），详情阅读文档[备份恢复](cluster_restore.md)\n\n```\ndk ezctl backup k8s-01\n```\n\n- 手动备份 etcd数据，在任意 etcd节点上执行：\n\n``` bash\n# snapshot备份\n$ ETCDCTL_API=3 etcdctl snapshot save backup.db\n# 查看备份\n$ ETCDCTL_API=3 etcdctl --write-out=table snapshot status backup.db\n```\n\n### k8s 升级小版本\n\n快速升级`k8s`小版本，比较常见如`Bug修复` `特性发布`时使用。\n\n- 首先去官网release下载待升级的k8s版本，例如`https://dl.k8s.io/v1.25.4/kubernetes-server-linux-amd64.tar.gz`\n- 解压下载的tar.gz文件，找到如下`kube*`开头的二进制，复制替换kubeasz控制端目录`/etc/kubeasz/bin`对应文件\n - kube-apiserver\n - kube-controller-manager\n - kubectl\n - kubelet\n - kube-proxy\n - kube-scheduler\n\n- 切换当前所在集群为升级集群, 在kubeasz控制端执行`dk ezctl checkout k8s-01`\n- 在kubeasz控制端执行`dk ezctl upgrade k8s-01` 即可完成k8s 升级，不会中断业务应用\n\n\n### 其他升级说明\n\n其他升级是指升级k8s组件包括：`etcd版本` `docker版本`，一般不需要用到，不建议升级，以下仅作说明。\n\n- 1.下载所有组件相关新的二进制解压并替换 `/etc/kubeasz/bin/` 目录下文件\n\n- 2.升级 etcd: `ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml -t upgrade_etcd playbooks/02.etcd.yml`\n\n- 3.升级 docker （建议使用k8s官方支持的docker稳定版本）\n - 如果可以接受短暂业务中断，执行 `ansible-playbook -t upgrade_docker 03.docker.yml`\n - 如果要求零中断升级，执行 `ansible-playbook -i clusters/k8s-01/hosts -e @clusters/k8s-01/config.yml -t download_docker playbooks/03.runtime.yml`，然后手动执行如下\n - 待升级节点，先应用`kubectl cordon`和`kubectl drain`命令迁移业务pod\n - 待升级节点执行 `systemctl restart docker`\n - 恢复节点可调度 `kubectl uncordon`\n"
},
{
"path": "docs/release-notes/kubeasz-3.6.0.md",
"content": "## kubeasz 3.6.0 (Beginning of Summer)\n\n微雨过，小荷翻。榴花开欲然。kubeasz 3.6.0 发布：支持k8s v1.27版本，支持更多操作系统安装，以及组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.27.1\n- cilium: v1.13.2\n- flannel: v0.21.4\n- harbor: v2.6.4\n- metrics-server: v0.6.3\n- k8s-dns-node-cache: 1.22.20\n- kube-prometheus-stack: 45.23.0\n\n### 调整项目分支更新规则\n\nk8s大版本对应kubeasz特定的大版本号，详见README.md 中版本对照表，当前积极更新的分支如下：\n\n- master：默认保持与最新分支同步，当前与v3.6同步\n- v3.6：对应k8s v1.27 版本，持续保持更新\n- v3.5：对应k8s v1.26 版本，主要使用cherry-pick方式合并后续版本中的重要commit\n- v3.4：对应k8s v1.25 版本，主要使用cherry-pick方式合并后续版本中的重要commit\n- v3.3：对应k8s v1.24 版本，主要使用cherry-pick方式合并后续版本中的重要commit\n\n### 支持更多操作系统安装\n\n本次增加测试支持大部分使用systemd的linux发行版，如果安装有问题先请查看(docs/setup/multi_os.md)；如果某个能够支持安装的系统没有在列表中，欢迎提PR 告知。\n\n- **Alibaba Linux** 2.1903, 3.2104([notes](docs/setup/multi_os.md#Alibaba))\n- **Alma Linux** 8, 9\n- **Anolis OS** 8.x RHCK, 8.x ANCK([notes](docs/setup/multi_os.md#Anolis))\n- **CentOS/RHEL** 7, 8, 9\n- **Debian** 10, 11([notes](docs/setup/multi_os.md#Debian))\n- **Fedora** 34, 35, 36, 37\n- **openSUSE** Leap 15.x([notes](docs/setup/multi_os.md#openSUSE))\n- **Rocky Linux** 8, 9\n- **Ubuntu** 16.04, 18.04, 20.04, 22.04\n\n### 重要更新\n\n- 重写`ezdown`脚本支持下载多系统软件包部分\n- 重写`role:prepare`支持离线安装多系统软件包部分\n- 简化harbor安装后集成使用，目前在containerd容器运行时中额外配置允许insecure仓库方式\n- 修复pod挂载 hostpath volume，删除pod会卡住问题 (#1259) by itswl\n- 增加设置limits for pids #1265 by AsonZhang\n\n### 其他\n\n- 增加项目`ISSUE`模版 \n- 修复chronyd 服务可能出现 enable失败问题 (#1254) by Roach57\n- 增加ezctl setup脚本执行时打印版本信息\n"
},
{
"path": "docs/release-notes/kubeasz-3.6.1.md",
"content": "## kubeasz 3.6.1\n\nkubeasz 3.6.1 发布：支持k8s v1.27版本，组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.27.2\n- calico: v3.24.6\n- kube-ovn: v1.11.5\n- kube-router: v1.5.4\n\n### 增加应用部署插件 kubeapps\n\nKubeapps 是一个基于 Web 的应用程序，它可以在 Kubernetes 集群上进行一站式安装，并使用户能够部署、管理和升级应用\n程序。https://github.com/easzlab/kubeasz/blob/master/docs/guide/kubeapps.md\n\n### 重要更新\n\n- 重写`ezdown`脚本支持下载额外的应用容器镜像\n- 增加`local-path-provisioner`本地文件目录提供者\n- 设置允许kubelet并行拉取容器镜像\n\n### 其他\n\n- 增加kubectl-node-shell 脚本\n- 修复ansible connect local 是 python 解析器不确定问题\n- 修复typo #1273\n- 部分文档更新\n"
},
{
"path": "docs/release-notes/kubeasz-3.6.2.md",
"content": "## kubeasz 3.6.2\n\nkubeasz 3.6.2 发布：支持k8s v1.28版本，组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.28.1\n- etcd: v3.5.9\n- containerd: 1.6.23\n- runc: v1.1.9\n- cni: v1.3.0\n- coredns: 1.11.1\n- cilium: 1.13.6\n- flannel: v0.22.2\n\n### 修改kubeasz支持k8s版本对应规则 \n\n原有模式每个k8s大版本都有推荐对应的kubeasz版本，这样做会导致kubeasz版本碎片化，追踪问题很麻烦，而且也影响普通用户安装体验。从kubeasz 3.6.2版本开始，默认最新版本kubeasz兼容支持安装最新的三个k8s大版本。具体安装说明如下：\n\n(如果/etc/kubeasz/bin 目录下已经有kube* 文件，需要先删除 rm -f /etc/kubeasz/bin/kube*)\n\n- 安装 k8s v1.28: 使用 kubeasz 3.6.2，执行./ezdown -D 默认下载即可\n- 安装 k8s v1.27: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.27.5 下载\n- 安装 k8s v1.26: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.26.8 下载\n- 安装 k8s v1.25: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.25.13 下载\n- 安装 k8s v1.24: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.24.17 下载\n\n\n### 重要更新\n\n- 增加支持containerd 可配置trusted insecure registries \n- 修复calico rr 模式的节点设置 #1308\n- 修复自定义节点名称设置 /etc/hosts方案\n- fix: kubelet failed when enabling kubeReserved or systemReserved\n\n### 其他\n\n- 修复：disable selinux on deploy host\n- helm部署redis-ha添加国内可访问镜像 by heyanyanchina123\n- 修复多集群管理时, 若当前ezctl配置不是升级集群,会导致升级失败 by learn0208\n- add ipvs配置打开strictARP #1298\n- revert for supporting k8s version <= 1.26\n- add kubetail, by WeiLai\n- update manifests:es-cluster/mysql-cluster\n"
},
{
"path": "docs/release-notes/kubeasz-3.6.3.md",
"content": "## kubeasz 3.6.3\n\nkubeasz 3.6.3 发布：支持k8s v1.29版本，组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.29.0\n- etcd: v3.5.10\n- containerd: 1.6.26\n- runc: v1.1.10\n- calico: v3.26.4\n- cilium: 1.14.5\n\n### 修改kubeasz支持k8s版本对应规则 \n\n原有模式每个k8s大版本都有推荐对应的kubeasz版本，这样做会导致kubeasz版本碎片化，追踪问题很麻烦，而且也影响普通用户安装体验。从kubeasz 3.6.2版本开始，默认最新版本kubeasz兼容支持安装最新的三个k8s大版本。具体安装说明如下：\n\n(如果/etc/kubeasz/bin 目录下已经有kube* 文件，需要先删除 rm -f /etc/kubeasz/bin/kube*)\n\n- 安装 k8s v1.29: 使用 kubeasz 3.6.3，执行./ezdown -D 默认下载即可\n- 安装 k8s v1.28: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.28.5 下载\n- 安装 k8s v1.27: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.27.9 下载\n- 安装 k8s v1.26: 使用 kubeasz 3.6.2，执行./ezdown -D -k v1.26.12 下载\n\n### 重要更新\n\n- deprecated role: os-harden，因为扩大支持更多linux发行版，系统加固方式无法在各种系统上充分测试，感谢 #1338 issue 反馈问题 \n- adjust docker setup scripts\n- update harbor v2.8.4 and fix harbor setup\n- fix nodelocaldns yaml\n\n### 其他\n\n- docs update: add argocd guide \n- docs: fix the quickStart.md url in network-plugin\n"
},
{
"path": "docs/release-notes/kubeasz-3.6.4.md",
"content": "## kubeasz 3.6.4\n\nkubeasz 3.6.4 发布：支持k8s v1.30版本，组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.30.1\n- etcd: v3.5.12\n- containerd: 1.7.17\n- runc: v1.1.12\n- calico: v3.26.4\n- cilium: 1.15.5\n- cni: v1.4.1\n- harbor: v2.10.2\n- metrics-server: v0.7.1\n\n### 重要更新\n\n- 安全更新：to solve CVE-2024-21626: update containerd, runc\n- 安装流程：role 'prepare' 阶段增加设置hostname，这样当网络组件为calico时不会因为主机名相同而出错；同时在example/config.yml 中增加配置开关`ENABLE_SETTING_HOSTNAME`\n- 操作系统：增加测试支持 Ubuntu 2404\n - 已知在ubuntu 2404上使用网络插件calico v3.26.4不兼容，提示：ipset v7.11: Kernel and userspace incompatible\n - 使用cilium 组件没有问题\n\n### 其他\n\n- 21376465de7f44d1ec997bde096afc7404ce45c5 fix: cilium ui images settings\n- c40548e0e33cab3c4e5742aacce11101ac0c7366 #1343, 恢复podPidsLimit=-1默认设置\n- \n"
},
{
"path": "docs/release-notes/kubeasz-3.6.5.md",
"content": "## kubeasz 3.6.5\n\nkubeasz 3.6.5 发布：支持k8s v1.31 版本，组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.31.2\n- etcd: v3.5.16\n- containerd: 1.7.23\n- runc: v1.1.15\n- calico: v3.28.2\n- coredns: 1.11.3\n- dnsnodecache: 1.23.1\n- cilium: 1.16.3\n- flannel: v0.26.0\n- cni: v1.6.0\n- harbor: v2.11.1\n- metrics-server: v0.7.2\n- pause: 3.10\n\n### 更新\n\n- 修正centos9 下prepare脚本运行的问题 #1397 By GitHubAwan\n- style: trim trailing whitespace & add logger source line number #1413 By kelein\n- 操作系统：增加测试支持 Ubuntu 2404\n - 修复在ubuntu 2404上使用网络插件calico ipSet兼容性问题（calico v3.28.2）\n\n### 其他\n\n- 修复calico hostname 设置\n- 更新部分文档\n- \n"
},
{
"path": "docs/release-notes/kubeasz-3.6.6.md",
"content": "## kubeasz 3.6.6\n\nkubeasz 3.6.6 发布：支持k8s v1.32 版本，组件更新和一些bugfix。\n\n### 版本更新\n\n- k8s: v1.32.3\n- etcd: v3.5.20\n- containerd: 2.0.4\n- runc: v1.2.6\n- calico: v3.28.3\n- coredns: 1.11.4\n- cni: v1.6.2\n- harbor: v2.12.2\n\n### 更新\n\n- 更新国内docker镜像仓库加速设置，解决ezdown脚本无法下载镜像问题；同步更新containerd 镜像仓库加速设置\n- 主要组件大版本更新：containerd 从 1.7.x 更新大版本 2.0.x，更新主要配置文件；runc 从 1.1.x 更新大版本 1.2.x\n- 安装逻辑更新：新增节点不再重复执行网络插件安装，避免部分网络插件自动重启业务pod，by gogeof\n- 安装逻辑更新：每次执行脚本 containerd 都会被重新安装，不管原先是否已经运行\n- 优化更新 ezctl 脚本从 ezdown 加载变量方式，by RadPaperDinosaur\n\n\n### 其他\n\n- 修复 CLUSTER_DNS_SVC_IP & CLUSTER_KUBERNETES_SVC_IP 地址生成规则，by yunpiao\n- 更新conformance文档\n- \n"
},
{
"path": "docs/release-notes/kubeasz-3.6.7.md",
"content": "## kubeasz 3.6.7\n\nkubeasz 3.6.7 发布：支持k8s v1.33 版本，组件更新和bugfix。\n\n### 版本更新\n\n- k8s: v1.33.1\n- etcd: v3.5.21\n- containerd: 2.1.1\n- runc: v1.2.6\n- calico: v3.28.4\n- cilium: 1.17.4\n- coredns: 1.12.1\n- cni: v1.7.1\n- dnsNodeCache: 1.25.0\n- harbor: v2.12.4\n- local-path-provisioner: v0.0.31\n- dashboard 7.12.0\n\n### 更新\n\n- 增加可选组件`kubeblocks`集成，增加多种数据库高可用方案\n- 重写脚本ezdown中关于镜像下载保存部分，清理冗余，增加错误错误处理\n- 修复添加/删除master节点时/etc/hosts问题 #1464\n- 修复使用静态编译的containerd二进制，并设置日志为warn级别，避免当容器使用exec类健康检查时产生过多日志\n- 修复./ezdown -D 偶发403报错 #1470\n- 修复cilium 组件原cilium_connectivity_check脚本执行条件\n\n### 文档更新\n\n- 更新一致性认证文档 conformance.md\n"
},
{
"path": "docs/release-notes/kubeasz-3.6.8.md",
"content": "## kubeasz 3.6.8\n\nkubeasz 3.6.8 发布：支持k8s v1.34 版本，组件更新和bugfix。\n\n### 版本更新\n\n- k8s: v1.34.1\n- etcd: v3.6.4\n- containerd: 2.1.4\n- runc: v1.3.1\n- coredns: 1.12.4\n- cni: v1.8.0\n- dnsNodeCache: 1.26.4\n- metrics: v0.8.0\n- flannel: v0.27.3\n- kubeblocks: 1.0.0\n- kube-prometheus-stack: 75.7.0\n\n### 重要更新\n\n- 调整系统内核设置 commit f9bdbeb4e3bd6b98a03a900d3e50ef29da6a590f, #1478\n- 新增支持 openEuler 22.03 LTS, 24.03 LTS\n- 优化节点只需运行一次 prepare task\n- 增加可选开启集群审计功能\n- 修复 calico mtu 设置 #1444\n- 修复 calico vxlan overlay 设置 #1492 \n- 更新 containerd 配置容器镜像仓库方式\n\n### 文档更新\n\n- 实验性混合架构部署文档 https://github.com/easzlab/kubeasz/blob/master/docs/setup/mix_arch.md\n- updat kernel_upgrade.md for centos7 by Zlanghu #1483\n\n感谢新增贡献者：\n\nvistamin #1444\nnewfzk #1477\nlearn0208 #1478\nZlanghu #1483\nnewfzk #1492\nTOT-JIN #1495\n"
},
{
"path": "docs/setup/00-planning_and_overall_intro.md",
"content": "## 00-集群规划和基础参数设定\n\n### HA architecture\n\n 2d v1.9.0\n192.168.1.43 Ready 2d v1.9.0\n192.168.1.44 Ready 2d v1.9.0\n```\n\n\n[后一篇](06-install_network_plugin.md)\n"
},
{
"path": "docs/setup/06-install_network_plugin.md",
"content": "## 06-安装网络组件\n\n首先回顾下K8S网络设计原则，在配置集群网络插件或者实践K8S 应用/服务部署请牢记这些原则：\n\n- 1.每个Pod都拥有一个独立IP地址，Pod内所有容器共享一个网络命名空间\n- 2.集群内所有Pod都在一个直接连通的扁平网络中，可通过IP直接访问\n - 所有容器之间无需NAT就可以直接互相访问\n - 所有Node和所有容器之间无需NAT就可以直接互相访问\n - 容器自己看到的IP跟其他容器看到的一样\n- 3.Service cluster IP只可在集群内部访问，外部请求需要通过NodePort、LoadBalance或者Ingress来访问\n\n`Container Network Interface (CNI)`是目前CNCF主推的网络模型，它由两部分组成：\n\n- CNI Plugin负责给容器配置网络，它包括两个基本的接口\n - 配置网络: AddNetwork(net *NetworkConfig, rt *RuntimeConf) (types.Result, error)\n - 清理网络: DelNetwork(net *NetworkConfig, rt *RuntimeConf) error\n- IPAM Plugin负责给容器分配IP地址\n\nKubernetes Pod的网络是这样创建的：\n- 0. 每个Pod除了创建时指定的容器外，都有一个kubelet启动时指定的`基础容器`，即`pause`容器 \n- 1. kubelet创建`基础容器`生成network namespace\n- 2. kubelet调用网络CNI driver，由它根据配置调用具体的CNI 插件\n- 3. CNI 插件给`基础容器`配置网络\n- 4. Pod 中其他的容器共享使用`基础容器`的网络\n\n本项目基于CNI driver 调用各种网络插件来配置kubernetes的网络，常用CNI插件有 `flannel` `calico` `cilium`等等，这些插件各有优势，也在互相借鉴学习优点，比如：在所有node节点都在一个二层网络时候，flannel提供hostgw实现，避免vxlan实现的udp封装开销，估计是目前最高效的；calico也针对L3 Fabric，推出了IPinIP的选项，利用了GRE隧道封装；因此这些插件都能适合很多实际应用场景。\n\n项目当前内置支持的网络插件有：`calico` `cilium` `flannel` `kube-ovn` `kube-router`\n\n### 安装讲解\n\n- [安装calico](network-plugin/calico.md)\n- [安装cilium](network-plugin/cilium.md) \n- [安装flannel](network-plugin/flannel.md)\n- [安装kube-ovn](network-plugin/kube-ovn.md) 暂未更新\n- [安装kube-router](network-plugin/kube-router.md) 暂未更新\n\n### 参考\n- [kubernetes.io networking docs](https://kubernetes.io/docs/concepts/cluster-administration/networking/) \n- [feiskyer-kubernetes指南网络章节](https://github.com/feiskyer/kubernetes-handbook/blob/master/zh/network/network.md)\n\n\n[后一篇](07-install_cluster_addon.md)\n"
},
{
"path": "docs/setup/07-install_cluster_addon.md",
"content": "# 07-安装集群主要插件\n\n目前挑选一些常用、必要的插件自动集成到安装脚本之中: \n\n## 集群默认安装\n\n- [coredns](../guide/kubedns.md)\n- [nodelocaldns](../guide/kubedns.md)\n- [metrics-server](../guide/metrics-server.md)\n- [dashboard](../guide/dashboard.md)\n\nkubeasz 默认安装上述基础插件，并支持离线方式安装(./ezdown -D 命令会自动下载组件镜像，并推送到本地镜像仓库easzlab.io.local:5000)\n\n## 集群可选安装\n\n- [prometheus](../guide/prometheus.md)\n- [network_check](network-plugin/network-check.md)\n- [nfs_provisioner]()\n\nkubeasz 默认不安装上述插件，可以在配置文件(clusters/xxx/config.yml)中开启，支持离线方式安装(./ezdown -X 会额外下载这些组件镜像，并推送到本地镜像仓库easzlab.io.local:5000)\n\n## 安装脚本\n\n详见`roles/cluster-addon/` 目录\n\n- 1.根据hosts文件中配置的`CLUSTER_DNS_SVC_IP` `CLUSTER_DNS_DOMAIN`等参数生成kubedns.yaml和coredns.yaml文件\n- 2.注册变量pod_info，pod_info用来判断现有集群是否已经运行各种插件\n- 3.根据pod_info和`配置开关`逐个进行/跳过插件安装\n\n## 下一步\n\n- [创建ex_lb节点组](ex-lb.md), 向集群外提供高可用apiserver\n- [创建集群持久化存储](08-cluster-storage.md)\n"
},
{
"path": "docs/setup/08-cluster-storage.md",
"content": "# K8S 集群存储 \n\n## 前言\n在kubernetes(k8s)中对于存储的资源抽象了两个概念，分别是PersistentVolume(PV)、PersistentVolumeClaim(PVC)。\n- PV是集群中的资源\n- PVC是对这些资源的请求。\n\n如上面所说PV和PVC都只是抽象的概念，在k8s中是通过插件的方式提供具体的存储实现。目前包含有NFS、iSCSI和云提供商指定的存储系统，更多的存储实现[参考官方文档](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)。\n\n以下介绍两种`provisioner`, 可以提供静态或者动态的PV\n\n- [nfs-provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner): NFS存储目录供应者\n- [local-path-provisioner](https://github.com/rancher/local-path-provisioner): 本地存储目录供应者\n\n## NFS存储目录供应者\n\n首先我们需要一个NFS服务器，用于提供底层存储。通过文档[nfs-server](../guide/nfs-server.md)，我们可以创建一个NFS服务器。\n\n### 静态 PV\n- 创建静态 pv，指定容量，访问模式，回收策略，存储类等\n\n``` bash\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n name: pv-es-0\nspec:\n capacity:\n storage: 4Gi\n accessModes:\n - ReadWriteMany\n volumeMode: Filesystem\n persistentVolumeReclaimPolicy: Recycle\n storageClassName: \"es-storage-class\"\n nfs:\n # 根据实际共享目录修改\n path: /share/es0\n # 根据实际 nfs服务器地址修改\n server: 192.168.1.208\n```\n- 创建 pvc即可绑定使用上述 pv了，具体请看后文 test pod例子\n\n### 创建动态PV\n\n在一个工作k8s 集群中，`PVC`请求会很多，如果每次都需要管理员手动去创建对应的 `PV`资源，那就很不方便；因此 K8S还提供了多种 `provisioner`来动态创建 `PV`，不仅节省了管理员的时间，还可以根据`StorageClasses`封装不同类型的存储供 PVC 选用。\n\n项目中以nfs-client-provisioner为例 https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner\n\n- 1.编辑集群配置文件：clusters/${集群名}/config.yml\n\n``` bash\n... 省略\n# 在role:cluster-addon 中启用nfs-provisioner 安装\nnfs_provisioner_install: \"yes\"\t\t\t# 修改为yes\nnfs_provisioner_namespace: \"kube-system\"\nnfs_provisioner_ver: \"v4.0.1\"\nnfs_storage_class: \"managed-nfs-storage\"\t\nnfs_server: \"192.168.31.244\"\t\t\t# 修改为实际nfs server地址\nnfs_path: \"/data/nfs\"\t\t\t\t# 修改为实际的nfs共享目录\n\n```\n\n- 2.创建 nfs provisioner\n\n``` bash\n$ dk ezctl setup ${集群名} 07 \n\n# 执行成功后验证\n$ kubectl get pod --all-namespaces |grep nfs-client\nkube-system nfs-client-provisioner-84ff87c669-ksw95 1/1 Running 0 21m\n```\n\n- 3.验证使用动态 PV\n\n在目录clusters/${集群名}/yml/nfs-provisioner/ 有个测试例子\n\n``` bash\n$ kubectl apply -f /etc/kubeasz/clusters/hello/yml/nfs-provisioner/test-pod.yaml\n\n# 验证测试pod\nkubectl get pod\nNAME READY STATUS RESTARTS AGE\ntest-pod 0/1 Completed 0 6h36m\n\n# 验证自动创建的pv 资源，\nkubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npvc-44d34a50-e00b-4f6c-8005-40f5cc54af18 2Mi RWX Delete Bound default/test-claim managed-nfs-storage 6h36m\n\n# 验证PVC已经绑定成功：STATUS字段为 Bound\nkubectl get pvc\nNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\ntest-claim Bound pvc-44d34a50-e00b-4f6c-8005-40f5cc54af18 2Mi RWX managed-nfs-storage 6h37m\n```\n\n另外，Pod启动完成后，在挂载的目录中创建一个`SUCCESS`文件。我们可以到NFS服务器去看下：\n\n```\n.\n└── default-test-claim-pvc-44d34a50-e00b-4f6c-8005-40f5cc54af18\n └── SUCCESS\n```\n如上，可以发现挂载的时候，nfs-client根据PVC自动创建了一个目录，我们Pod中挂载的`/mnt`，实际引用的就是该目录，而我们在`/mnt`下创建的`SUCCESS`文件，也自动写入到了这里。\n\n后面当我们需要为上层应用提供持久化存储时，只需要提供`StorageClass`即可。很多应用都会根据`StorageClass`来创建他们的所需的PVC, 最后再把PVC挂载到他们的Deployment或StatefulSet中使用，比如：efk、jenkins等\n\n## 本地存储目录供应者\n\n当应用对于磁盘I/O性能要求高，比较适合本地文件目录存储，特别地可以本地挂载SSD磁盘（注意本地磁盘需要配置raid冗余策略）。Local Path Provisioner 可以方便地在k8s集群中使用本地文件目录存储。\n\n在kubeasz项目中集成安装\n\n- 1.编辑集群配置文件：clusters/${集群名}/config.yml\n\n``` bash\n... 省略\nlocal_path_provisioner_install: \"yes\" # 修改为yes\n# 设置默认本地存储路径\nlocal_path_provisioner_dir: \"/opt/local-path-provisioner\"\n```\n\n- 2.创建 local path provisioner\n\n``` bash\n$ dk ezctl setup ${集群名} 07\n\n# 执行成功后验证\n$ kubectl get pod --all-namespaces |grep provisioner\n```\n\n- 3.验证使用（略）\n"
},
{
"path": "docs/setup/config_guide.md",
"content": "# 个性化集群参数配置\n\n`kubeasz`创建集群主要在以下两个地方进行配置：(假设集群名xxxx)\n\n- clusters/xxxx/hosts 文件（模板在example/hosts.multi-node）：集群主要节点定义和主要参数配置、全局变量\n- clusters/xxxx/config.yml（模板在examples/config.yml）：其他参数配置或者部分组件附加参数\n\n## clusters/xxxx/hosts (ansible hosts)\n\n如[集群规划与安装概览](00-planning_and_overall_intro.md)中介绍，主要包括集群节点定义和集群范围的主要参数配置\n\n- 尽量保持配置简单灵活\n- 尽量保持配置项稳定\n\n常用设置项：\n\n- 修改容器运行时: CONTAINER_RUNTIME=\"containerd\"\n- 修改集群网络插件：CLUSTER_NETWORK=\"calico\"\n- 修改容器网络地址：CLUSTER_CIDR=\"192.168.0.0/16\"\n- 修改NodePort范围：NODE_PORT_RANGE=\"30000-32767\"\n\n## clusters/xxxx/config.yml\n\n主要包括集群某个具体组件的个性化配置，具体组件的配置项可能会不断增加；可以在不做任何配置更改情况下使用默认值创建集群\n\n根据实际需要配置 k8s 集群，常用举例\n\n- 配置使用离线安装系统包：INSTALL_SOURCE: \"offline\" （需要ezdown -P 下载离线系统软件）\n- 配置CA证书以及其签发证书的有效期\n- 配置 apiserver 支持公网域名：MASTER_CERT_HOSTS\n- 配置 cluster-addon 组件安装\n- ...\n"
},
{
"path": "docs/setup/ex-lb.md",
"content": "## EX-LB 负载均衡部署\n\n根据[HA 2x架构](00-planning_and_overall_intro.md)，k8s集群自身高可用已经不依赖于外部 lb 服务；但是有时我们要从外部访问 apiserver（比如 CI 流程），就需要 ex_lb 来请求多个 apiserver；\n\n还有一种情况是需要[负载转发到ingress服务](../op/loadballance_ingress_nodeport.md)，也需要部署ex_lb；\n\n**注意：当遇到公有云环境无法自建 ex_lb 服务时，可以配置对应的云负载均衡服务**\n\n### ex_lb 服务组件\n\n更新：kubeasz 3.0.2 重写了ex-lb服务安装，利用最小化依赖编译安装的二进制文件，不依赖于linux发行版；优点是可以统一版本和简化离线安装部署，并且理论上能够支持更多linux发行版\n\n\nex_lb 服务由 keepalived 和 l4lb 组成：\n- l4lb：是一个精简版（仅支持四层转发）的nginx编译二进制版本\n- keepalived：利用主备节点vrrp协议通信和虚拟地址，消除l4lb的单点故障；keepalived保持存活，它是基于VRRP协议保证所谓的高可用或热备的，这里用来预防l4lb的单点故障。\n\nkeepalived与l4lb配合，实现master的高可用过程如下：\n\n+ 1.keepalived利用vrrp协议生成一个虚拟地址(VIP)，正常情况下VIP存活在keepalive的主节点，当主节点故障时，VIP能够漂移到keepalived的备节点，保障VIP地址高可用性。\n+ 2.在keepalived的主备节点都配置相同l4lb负载配置，并且监听客户端请求在VIP的地址上，保障随时都有一个l4lb负载均衡在正常工作。并且keepalived启用对l4lb进程的存活检测，一旦主节点l4lb进程故障，VIP也能切换到备节点，从而让备节点的l4lb进行负载工作。\n+ 3.在l4lb的配置中配置多个后端真实kube-apiserver的endpoints，并启用存活监测后端kube-apiserver，如果一个kube-apiserver故障，l4lb会将其剔除负载池。\n\n#### 安装l4lb\n\n#### 配置l4lb (roles/ex-lb/templates/l4lb.conf.j2)\n\n配置由全局配置和三个upstream servers配置组成：\n- apiservers 用于转发至多个apiserver\n- ingress-nodes 用于转发至node节点的ingress http服务，[参阅](../op/loadballance_ingress_nodeport.md)\n- ingress-tls-nodes 用于转发至node节点的ingress https服务\n\n#### 安装keepalived\n\n#### 配置keepalived主节点 [keepalived-master.conf.j2](../../roles/ex-lb/templates/keepalived-master.conf.j2)\n\n``` bash\nglobal_defs {\n}\n\nvrrp_track_process check-l4lb {\n process l4lb\n weight -60\n delay 3\n}\n\nvrrp_instance VI-01 {\n state MASTER\n priority 120\n unicast_src_ip {{ inventory_hostname }}\n unicast_peer {\n{% for h in groups['ex_lb'] %}{% if h != inventory_hostname %}\n {{ h }}\n{% endif %}{% endfor %}\n }\n dont_track_primary\n interface {{ LB_IF }}\n virtual_router_id {{ ROUTER_ID }}\n advert_int 3\n track_process {\n check-l4lb\n }\n virtual_ipaddress {\n {{ EX_APISERVER_VIP }}\n }\n}\n```\n+ vrrp_track_process 定义了监测l4lb进程是否存活，如果进程不存在，根据`weight -60`设置将主节点优先级降低60，这样原先备节点将变成主节点。\n+ vrrp_instance 定义了vrrp组，包括优先级、使用端口、router_id、心跳频率、检测脚本、虚拟地址VIP等\n+ 特别注意 `virtual_router_id` 标识了一个 VRRP组，在同网段下必须唯一，否则出现 `Keepalived_vrrp: bogus VRRP packet received on eth0 !!!`类似报错\n+ 配置 vrrp 协议通过单播发送\n\n#### 配置keepalived备节点 [keepalived-backup.conf.j2](../../roles/ex-lb/templates/keepalived-backup.conf.j2)\n\n+ 备节点的配置类似主节点，除了优先级和检测脚本，其他如 `virtual_router_id` `advert_int` `virtual_ipaddress`必须与主节点一致\n\n### 启动 keepalived 和 l4lb 后验证\n\n+ lb 节点验证\n\n``` bash\nsystemctl status l4lb \t# 检查进程状态\njournalctl -u l4lb\t\t# 检查进程日志是否有报错信息\nsystemctl status keepalived \t# 检查进程状态\njournalctl -u keepalived\t# 检查进程日志是否有报错信息\n```\n+ 在 keepalived 主节点\n\n``` bash\nip a\t\t\t\t# 检查 master的 VIP地址是否存在\n```\n### keepalived 主备切换演练\n\n1. 尝试关闭 keepalived主节点上的 l4lb进程，然后在keepalived 备节点上查看 master的 VIP地址是否能够漂移过来，并依次检查上一步中的验证项。\n1. 尝试直接关闭 keepalived 主节点系统，检查各验证项。\n\n"
},
{
"path": "docs/setup/ezctl.md",
"content": "# ezctl 命令行介绍\n\n## 为什么使用 ezctl\n\nkubeasz 项目使用ezctl 方便地创建和管理多个k8s 集群，ezctl 使用shell 脚本封装ansible-playbook 执行命令，它十分轻量、简单和易于扩展。\n\n### 使用帮助\n\n随时运行 ezctl 获取命令行提示信息，如下\n\n```\nUsage: ezctl COMMAND [args]\n-------------------------------------------------------------------------------------\nCluster setups:\n list\t\t to list all of the managed clusters\n checkout to switch default kubeconfig of the cluster\n new to start a new k8s deploy with name 'cluster'\n setup to setup a cluster, also supporting a step-by-step way\n start to start all of the k8s services stopped by 'ezctl stop'\n stop to stop all of the k8s services temporarily\n upgrade to upgrade the k8s cluster\n destroy to destroy the k8s cluster\n backup to backup the cluster state (etcd snapshot)\n restore to restore the cluster state from backups\n start-aio\t\t to quickly setup an all-in-one cluster with 'default' settings\n\nCluster ops:\n add-etcd to add a etcd-node to the etcd cluster\n add-master to add a master node to the k8s cluster\n add-node to add a work node to the k8s cluster\n del-etcd to delete a etcd-node from the etcd cluster\n del-master to delete a master node from the k8s cluster\n del-node to delete a work node from the k8s cluster\n\nExtra operation:\n kcfg-adm to manage client kubeconfig of the k8s cluster\n\nUse \"ezctl help \" for more information about a given command.\n```\n\n- 命令集 1：集群安装相关操作\n - 显示当前所有管理的集群\n - 切换默认集群\n - 创建新集群配置\n - 安装新集群\n - 启动临时停止的集群\n - 临时停止某个集群（包括集群内运行的pod）\n - 升级集群k8s组件版本\n - 删除集群\n - 备份集群（仅etcd数据，不包括pv数据和业务应用数据）\n - 从备份中恢复集群\n - 创建单机集群（类似 minikube）\n- 命令集 2：集群节点操作\n - 增加 etcd 节点\n - 增加主节点\n - 增加工作节点\n - 删除 etcd 节点\n - 删除主节点\n - 删除工作节点\n- 命令集3：额外操作\n - 管理客户端kubeconfig\n\n#### 举例创建、安装新集群流程\n\n- 1.首先创建集群配置实例 \n\n``` bash\n~# ezctl new k8s-01\n2021-01-19 10:48:23 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-01\n2021-01-19 10:48:23 DEBUG set version of common plugins\n2021-01-19 10:48:23 DEBUG cluster k8s-01: files successfully created.\n2021-01-19 10:48:23 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-01/hosts'\n2021-01-19 10:48:23 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-01/config.yml'\n```\n然后根据提示配置'/etc/kubeasz/clusters/k8s-01/hosts' 和 '/etc/kubeasz/clusters/k8s-01/config.yml'；为方便测试我们在hosts里面设置单节点集群（etcd/kube_master/kube_node配置同一个节点，注意节点需先设置ssh免密码登陆）, config.yml 使用默认配置即可。\n\n- 2.然后开始安装集群\n\n``` bash\n# 一键安装\nezctl setup k8s-01 all\n\n# 或者分步安装，具体使用 ezctl help setup 查看分步安装帮助信息\n# ezctl setup k8s-01 01\n# ezctl setup k8s-01 02\n# ezctl setup k8s-01 03\n# ezctl setup k8s-01 04\n... \n```\n\n- 3.重复步骤1，2可以创建、管理多个k8s集群（建议ezctl使用独立的部署节点）\n\nezctl 创建管理的多集群拓扑如下\n\n```\n+----------------+ +-----------------+\n|ezctl 1.1.1.1 | |cluster-aio: |\n+--+---+---+-----+ | |\n | | | |master 4.4.4.4 |\n | | +-------------------->+etcd 4.4.4.4 |\n | | |node 4.4.4.4 |\n | +--------------+ +-----------------+\n | |\n v v\n+--+------------+ +---+----------------------------+\n| cluster-1: | | cluster-2: |\n| | | |\n| master 2.2.2.1| | master 3.3.3.1/3.3.3.2 |\n| etcd 2.2.2.2| | etcd 3.3.3.1/3.3.3.2/3.3.3.3 |\n| node 2.2.2.3| | node 3.3.3.4/3.3.3.5/3.3.3.6 |\n+---------------+ +--------------------------------+\n```\n\nThat's it! 赶紧动手测试吧，欢迎通过 Issues 和 PRs 反馈您的意见和建议！\n"
},
{
"path": "docs/setup/kubeasz_on_public_cloud.md",
"content": "# 公有云上部署 kubeasz\n\n在公有云上使用`kubeasz`部署`k8s`集群需要注意以下几个常见问题。\n\n### 安全组\n\n注意虚机的安全组规则配置，一般集群内部节点之间端口全部放开即可；\n\n### 网络组件\n\n一般公有云对网络限制较多，跨节点 pod 通讯需要使用 OVERLAY 添加报头；默认配置详见example/config.yml\n\n- flannel 使用 vxlan 模式：`FLANNEL_BACKEND: \"vxlan\"`\n- calico 开启 ipinip：`CALICO_IPV4POOL_IPIP: \"Always\"`\n- kube-router 开启 ipinip：`OVERLAY_TYPE: \"full\"`\n\n### 节点公网访问\n\n可以在安装时每个节点绑定`弹性公网地址`(EIP)，装完集群解绑；也可以开通NAT网关，或者利用iptables自建上网网关等方式\n\n### 负载均衡\n\n一般云厂商会限制使用`keepalived+haproxy`自建负载均衡，你可以根据云厂商文档使用云负载均衡（内网）四层TCP负载模式；\n\n- kubeasz 2x 版本已无需依赖外部负载均衡实现apiserver的高可用，详见 [2x架构](https://github.com/easzlab/kubeasz/blob/dev2/docs/setup/00-planning_and_overall_intro.md#ha-architecture)\n- kubeasz 1x 及以前版本需要负载均衡实现apiserver高可用，详见 [1x架构](https://github.com/easzlab/kubeasz/blob/dev1/docs/setup/00-planning_and_overall_intro.md#ha-architecture)\n\n### 时间同步\n\n一般云厂商提供的虚机都已默认安装时间同步服务，无需自行安装。 \n\n### 访问 APISERVER\n\n在公有云上安装完集群后，需要在公网访问集群 apiserver，而我们在安装前可能没有规划公网IP或者公网域名；而 apiserver 肯定需要 https 方式访问，在证书创建时需要加入公网ip/域名；可以参考这里[修改 APISERVER（MASTER）证书](../op/ch_apiserver_cert.md)\n\n## 在公有云上部署多主高可用集群\n\n处理好以上讨论的常见问题后，在公有云上使用 kubeasz 安装集群与自有环境没有差异。\n\n- 使用 kubeasz 2x 版本安装单节点、单主多节点、多主多节点 k8s 集群，云上云下的预期安装体验完全一致\n"
},
{
"path": "docs/setup/mix_arch.md",
"content": "# 混合架构集群部署\n\n混合架构集群本文是指集群中既有linux amd64架构机器，也有linux arm64架构机器；这里只记录一个简单的操作说明，实际操作注意风险。\n\n## 部署思路\n\n1. 先选定一台amd64架构的机器做“amd64部署机”，使用它先部署amd64架构的集群\n2. 选一台arm64架构的机器做“arm64部署机”，复制amd64部署机的/etc/kubeasz目录文件（除去目录中的bin、down子目录），然后重新下载arm64架构的二进制和镜像，然后添加arm64节点到原有集群即可\n\n## 操作步骤\n\n1. 假设已经正常部署了amd64架构的三节点集群\n2. 在“amd64部署机” 目录 /etc/kubeasz 中移除子目录 bin 和 down，然后把整体/etc/kubeasz 目录复制到“arm64部署机”\n\n```\n# 登录amd64部署机\ncd /etc/kubeasz; mv bin down /tmp/; scp -r /etc/kubeasz root@{_ip_arm64}:/etc/\n# 复制完成后找回 bin 和 down 子目录\nmv /tmp/bin /etc/kubeasz/; mv /tmp/down /etc/kubeasz/\n```\n3. 登录“arm64部署机”，执行下载，其他准备工作\n\n```\ncd /etc/kubeasz\n# 下载基础部分\n./ezdown -D\n# 下载额外部分（如有）\n./ezdown -X ...\n# 运行部署容器\n./ezdown -S\n# 配置机器ssh免密码登录，集群所有节点都免密，包括待新增arm64节点\nssh-copy-id xx.xx.xx.xx\nssh-copy-id ...\n# 复制kubeconfig\nmkdir /root/.kube/; cp clusters/default/kubectl.kubeconfig /root/.kube/config\n```\n4. 添加arm64新节点到集群\n\n```\nsource ~/.bashrc\n# 添加新节点 x.x.x.x\ndk ezctl add-node default x.x.x.x\n```\n5. 验证\n\n```\n$ kubectl get node -owide\nNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\nk8s-x.x.x-19 Ready master 5d8h v1.33.1 x.x.x.19 Ubuntu 20.04.4 LTS 5.4.0-122-generic containerd://2.1.1\nk8s-x.x.x-90 Ready node 5d8h v1.33.1 x.x.x.90 Ubuntu 22.04.5 LTS 5.15.0-134-generic containerd://2.1.1\nk8s-x.x.x-91 Ready node 5d8h v1.33.1 x.x.x.91 Ubuntu 22.04.5 LTS 5.15.0-134-generic containerd://2.1.1\nk8s-x.x.x-93 Ready node 79s v1.33.1 x.x.x.93 Ubuntu 22.04.5 LTS 5.15.0-140-generic containerd://2.1.1\n\n$ kubectl describe node|grep beta.kubernetes.io/arch\nLabels: beta.kubernetes.io/arch=amd64\nLabels: beta.kubernetes.io/arch=amd64\nLabels: beta.kubernetes.io/arch=amd64\nLabels: beta.kubernetes.io/arch=arm64\n```\n\n## 小结\n通过以上步骤，成功实现了在amd64集群中添加arm64节点；充分展示kubeasz 项目部署集群的灵活性和可配置性；部署过程中ansible执行的过程性输出内容，以近乎白盒的方式展示每一个细节；假如出错有详细的说明，帮助定位，并且随时可以修改执行脚本，安装的幂等性保证随时可以重新安装以修复错误。`Hack it, and have fun!`\n\n"
},
{
"path": "docs/setup/multi_os.md",
"content": "# 操作系统说明\n\n目前发现部分使用新内核的linux发行版，k8s 安装使用 cgroup v2版本时，有时候安装会失败，需要删除/清理集群后重新安装。已报告可能发生于 Alma Linux 9, Rocky Linux 9, Fedora 37；建议如下步骤处理：\n\n- 1.确认系统使用的cgroup v2版本\n```\nstat -fc %T /sys/fs/cgroup/ \ncgroup2fs\n```\n- 2.初次安装时kubelet可能启动失败，日志报错类似：err=\"openat2 /sys/fs/cgroup/kubepods.slice/cpu.weight: no such file or directory\"\n\n- 3.建议删除集群然后重新安装，一般能够成功\n```\n# 删除集群\ndk ezctl destroy xxxx\n\n# 重启\nreboot\n\n# 启动后重新安装\ndk ezctl setup xxxx all\n```\n\n## Debian\n\n- Debian 11：默认可能没有安装iptables，使用kubeasz 安装前需要执行：\n\n``` bash \napt update\n\napt install iptables -y\n```\n\n## openEuler\n\n- openEuler 24.03 需要安装iptables\n\n``` bash\nyum install iptables -y\n```\n\n## openSUSE\n\n- openSUSE Leap 15.4：需要安装iptables\n\n``` bash\nzypper install iptables\nln -s /usr/sbin/iptables /sbin/iptables\n```\n"
},
{
"path": "docs/setup/multi_platform.md",
"content": "# 多架构支持\n\nkubeasz 3.4.1 以后支持多CPU架构，当前已支持linux amd64和linux arm64，更多架构支持根据后续需求来计划。\n\n## 使用方式\n\nkubeasz 多架构安装逻辑：根据部署机器（执行ezdown/ezctl命令的机器）的架构，会自动判断下载对应amd64/arm64的二进制文件和容器镜像，然后推送安装到整个集群。\n\n- 暂不支持自动部署混合架构集群，如有需要可以按[说明文档](mix_arch.md)手动操作。\n- harbor目前仅支持amd64安装\n\n## 架构支持备忘\n\n#### k8s核心组件本身提供多架构的二进制文件/容器镜像下载，项目调整了下载二进制文件的容器dockerfile\n\n- https://github.com/easzlab/dockerfile-kubeasz-k8s-bin\n\n#### kubeasz其他用到的二进制或镜像，重新调整了容器创建dockerfile\n\n- https://github.com/easzlab/dockerfile-kubeasz-ext-bin\n- https://github.com/easzlab/dockerfile-kubeasz-ext-build\n- https://github.com/easzlab/dockerfile-kubeasz-sys-pkg\n- https://github.com/easzlab/dockerfile-kubeasz-mirrored-images\n- https://github.com/easzlab/dockerfile-kubeasz\n- https://github.com/easzlab/dockerfile-ansible\n\n#### 其他组件(coredns/network plugin/dashboard/metrics-server等)一般都提供多架构的容器镜像，可以直接下载拉取\n\n\n"
},
{
"path": "docs/setup/network-plugin/calico-bgp-rr.md",
"content": "# calico 配置 BGP Route Reflectors\n\n`Calico`作为`k8s`的一个流行网络插件，它依赖`BGP`路由协议实现集群节点上的`POD`路由互通；而路由互通的前提是节点间建立 BGP Peer 连接。BGP 路由反射器（Route Reflectors，简称 RR）可以简化集群BGP Peer的连接方式，它是解决BGP扩展性问题的有效方式；具体来说：\n\n- 没有 RR 时，所有节点之间需要两两建立连接（IBGP全互联），节点数量增加将导致连接数剧增、资源占用剧增\n- 引入 RR 后，其他 BGP 路由器只需要与它建立连接并交换路由信息，节点数量增加连接数只是线性增加，节省系统资源\n\ncalico-node 版本 v3.3 开始支持内建路由反射器，非常方便，因此使用 calico 作为网络插件可以支持大规模节点数的`K8S`集群。\n\n- 建议集群节点数大于50时，应用BGP Route Reflectors 特性\n\n## 前提条件\n\nk8s 集群使用calico网络插件部署成功。本实验环境为按照kubeasz安装的2主2从集群，calico 版本 v3.19.4。\n\n```\n$ kubectl get node\nNAME STATUS ROLES AGE VERSION\n192.168.1.1 Ready,SchedulingDisabled master 178m v1.13.1\n192.168.1.2 Ready,SchedulingDisabled master 178m v1.13.1\n192.168.1.3 Ready node 178m v1.13.1\n192.168.1.4 Ready node 178m v1.13.1\n$ kubectl get pod -n kube-system -o wide | grep calico\ncalico-kube-controllers-77487546bd-jqrlc 1/1 Running 0 179m 192.168.1.3 192.168.1.3 \ncalico-node-67t5m 2/2 Running 0 179m 192.168.1.1 192.168.1.1 \ncalico-node-drmhq 2/2 Running 0 179m 192.168.1.2 192.168.1.2 \ncalico-node-rjtkv 2/2 Running 0 179m 192.168.1.4 192.168.1.4 \ncalico-node-xtspl 2/2 Running 0 179m 192.168.1.3 192.168.1.3 \n```\n查看当前集群中BGP连接情况：可以看到集群中4个节点两两建立了 BGP 连接\n\n```\n$ dk ansible -i /etc/kubeasz/clusters/xxx/hosts all -m shell -a '/opt/kube/bin/calicoctl node status'\n192.168.1.3 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-------------------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-------------------+-------+----------+-------------+\n| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |\n| 192.168.1.2 | node-to-node mesh | up | 03:08:18 | Established |\n| 192.168.1.4 | node-to-node mesh | up | 03:08:19 | Established |\n+--------------+-------------------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n\n192.168.1.2 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-------------------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-------------------+-------+----------+-------------+\n| 192.168.1.4 | node-to-node mesh | up | 03:08:17 | Established |\n| 192.168.1.3 | node-to-node mesh | up | 03:08:18 | Established |\n| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |\n+--------------+-------------------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n\n192.168.1.1 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-------------------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-------------------+-------+----------+-------------+\n| 192.168.1.2 | node-to-node mesh | up | 03:08:21 | Established |\n| 192.168.1.3 | node-to-node mesh | up | 03:08:21 | Established |\n| 192.168.1.4 | node-to-node mesh | up | 03:08:21 | Established |\n+--------------+-------------------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n\n192.168.1.4 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-------------------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-------------------+-------+----------+-------------+\n| 192.168.1.2 | node-to-node mesh | up | 03:08:17 | Established |\n| 192.168.1.3 | node-to-node mesh | up | 03:08:19 | Established |\n| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |\n+--------------+-------------------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n```\n\n## kubeasz 自动安装启用 route reflector\n\n- 修改`/etc/kubeasz/clusters/xxx/config.yml`文件，设置配置项`CALICO_RR_ENABLED: true` \n- 重新执行网络安装 `dk ezctl setup xxx 07`\n\n执行完成，检查bgp连接验证即可。\n\n### 附：手动安装route reflector 过程讲解\n\n- 选择并配置 Route Reflector 节点\n\n首先查看当前集群中的节点：\n\n```\n$ calicoctl get node -o wide\nNAME ASN IPV4 IPV6\nk8s401 (64512) 192.168.1.1/24\nk8s402 (64512) 192.168.1.2/24\nk8s403 (64512) 192.168.1.3/24\nk8s404 (64512) 192.168.1.4/24\n```\n\n可以在集群中选择1个或多个节点作为 rr 节点，这里先选择节点：k8s401\n\n``` bash\n#配置routeReflectorClusterID\ncalicoctl patch node k8s401 -p '{\"spec\": {\"bgp\": {\"routeReflectorClusterID\": \"244.0.0.1\"}}}'\n\n#配置node label\ncalicoctl patch node k8s401 -p '{\"metadata\": {\"labels\": {\"route-reflector\": \"true\"}}}'\n```\n\n- 配置 BGP node 与 Route Reflector 的连接建立规则\n\n``` bash\n$ cat << EOF | calicoctl create -f -\nkind: BGPPeer\napiVersion: projectcalico.org/v3\nmetadata:\n name: peer-with-route-reflectors\nspec:\n nodeSelector: all()\n peerSelector: route-reflector == 'true'\nEOF\n```\n\n- 配置全局禁用全连接（BGP full mesh）\n\n```\n$ cat << EOF | calicoctl create -f -\napiVersion: projectcalico.org/v3\nkind: BGPConfiguration\nmetadata:\n name: default\nspec:\n logSeverityScreen: Info\n nodeToNodeMeshEnabled: false\n asNumber: 64512\nEOF\n```\n\n- 验证增加 rr 之后的bgp 连接情况\n\n``` \n$ dk ansible -i /etc/kubeasz/clusters/xxx/hosts all -m shell -a '/opt/kube/bin/calicoctl node status'\n192.168.1.4 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-----------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-----------+-------+----------+-------------+\n| 192.168.1.1 | node specific | up | 11:02:55 | Established |\n+--------------+-----------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n\n192.168.1.3 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-----------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-----------+-------+----------+-------------+\n| 192.168.1.1 | node specific | up | 11:02:55 | Established |\n+--------------+-----------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n\n192.168.1.1 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+---------------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+---------------+-------+----------+-------------+\n| 192.168.1.2 | node specific | up | 11:02:55 | Established |\n| 192.168.1.3 | node specific | up | 11:02:55 | Established |\n| 192.168.1.4 | node specific | up | 11:02:55 | Established |\n+--------------+---------------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n\n192.168.1.2 | SUCCESS | rc=0 >>\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-----------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-----------+-------+----------+-------------+\n| 192.168.1.1 | node specific | up | 11:02:55 | Established |\n+--------------+-----------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.\n```\n可以看到所有其他节点都与所选rr节点建立bgp连接。\n\n- 再增加一个 rr 节点(略)\n\n步骤同上，添加成功后可以看到所有其他节点都与两个rr节点建立bgp连接，两个rr节点之间也建立bgp连接。对于节点数较多的`K8S`集群建议配置2-3个 RR 节点。\n\n## 参考文档\n\n- 1.[Calico bgp 配置指南](https://projectcalico.docs.tigera.io/reference/resources/bgpconfig)\n- 2.[BGP路由反射器基础](https://www.sohu.com/a/140033025_761420)\n"
},
{
"path": "docs/setup/network-plugin/calico.md",
"content": "## 06-安装calico网络组件.md\n\ncalico 是k8s社区最流行的网络插件之一，也是k8s-conformance test 默认使用的网络插件，功能丰富，支持network policy；是当前kubeasz项目的默认网络插件。\n\n如果需要安装calico，请在`clusters/xxxx/hosts`文件中设置变量 `CLUSTER_NETWORK=\"calico\"`，参考[这里](../config_guide.md)\n\n``` bash\nroles/calico/\n├── tasks\n│ └── main.yml\n├── templates\n│ ├── calico-csr.json.j2\n│ ├── calicoctl.cfg.j2\n│ ├── calico-v3.15.yaml.j2\n│ ├── calico-v3.19.yaml.j2\n│ └── calico-v3.8.yaml.j2\n└── vars\n └── main.yml\n```\n请在另外窗口打开`roles/calico/tasks/main.yml`文件，对照看以下讲解内容。\n\n### 创建calico 证书申请\n\n``` bash\n{\n \"CN\": \"calico\",\n \"hosts\": [],\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"k8s\",\n \"OU\": \"System\"\n }\n ]\n}\n```\ncalico 使用客户端证书，所以hosts字段可以为空；后续可以看到calico证书用在四个地方：\n\n- calico/node 这个docker 容器运行时访问 etcd 使用证书\n- cni 配置文件中，cni 插件需要访问 etcd 使用证书\n- calicoctl 操作集群网络时访问 etcd 使用证书\n- calico/kube-controllers 同步集群网络策略时访问 etcd 使用证书\n\n### 创建 calico DaemonSet yaml文件和rbac 文件\n\n请对照 roles/calico/templates/calico.yaml.j2文件注释和以下注意内容\n\n+ 详细配置参数请参考[calico官方文档](https://projectcalico.docs.tigera.io/reference/node/configuration)\n+ 配置ETCD_ENDPOINTS 、CA、证书等，所有{{ }}变量与ansible hosts文件中设置对应\n+ 配置集群POD网络 CALICO_IPV4POOL_CIDR={{ CLUSTER_CIDR }}\n+ 配置FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT 默认允许Pod到Node的网络流量，更多[felix配置选项](https://projectcalico.docs.tigera.io/reference/felix/configuration)\n\n### 安装calico 网络\n\n+ 安装前检查主机名不能有大写字母，只能由`小写字母` `-` `.` 组成 (name must consist of lower case alphanumeric characters, '-' or '.' (regex: [a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*))(calico-node v3.0.6以上已经解决主机大写字母问题)\n+ **安装前必须确保各节点主机名不重复** ，calico node name 由节点主机名决定，如果重复，那么重复节点在etcd中只存储一份配置，BGP 邻居也不会建立。\n+ 安装之前必须确保`kube_master`和`kube_node`节点已经成功部署\n+ 轮询等待calico 网络插件安装完成，删除之前kube_node安装时默认cni网络配置\n\n### [可选]配置calicoctl工具 [calicoctl.cfg.j2](roles/calico/templates/calicoctl.cfg.j2)\n\n``` bash\napiVersion: projectcalico.org/v3\nkind: CalicoAPIConfig\nmetadata:\nspec:\n datastoreType: \"etcdv3\"\n etcdEndpoints: {{ ETCD_ENDPOINTS }}\n etcdKeyFile: /etc/calico/ssl/calico-key.pem\n etcdCertFile: /etc/calico/ssl/calico.pem\n etcdCACertFile: {{ ca_dir }}/ca.pem\n```\n\n### 验证calico网络\n\n执行calico安装成功后可以验证如下：(需要等待镜像下载完成，有时候即便上一步已经配置了docker国内加速，还是可能比较慢，请确认以下容器运行起来以后，再执行后续验证步骤)\n\n``` bash\nkubectl get pod --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\nkube-system calico-kube-controllers-5c6b98d9df-xj2n4 1/1 Running 0 1m\nkube-system calico-node-4hr52 2/2 Running 0 1m\nkube-system calico-node-8ctc2 2/2 Running 0 1m\nkube-system calico-node-9t8md 2/2 Running 0 1m\n```\n\n**查看网卡和路由信息**\n\n先在集群创建几个测试pod: `kubectl run test --image=busybox --replicas=3 sleep 30000`\n\n``` bash\n# 查看网卡信息\nip a\n```\n\n+ 可以看到包含类似cali1cxxx的网卡，是calico为测试pod生成的\n+ tunl0网卡现在不用管，是默认生成的，当开启IPIP 特性时使用的隧道\n\n``` bash\n# 查看路由\nroute -n\nKernel IP routing table\nDestination Gateway Genmask Flags Metric Ref Use Iface\n0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 ens3\n192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens3\n172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0\n172.20.3.64 192.168.1.34 255.255.255.192 UG 0 0 0 ens3\n172.20.33.128 0.0.0.0 255.255.255.192 U 0 0 0 *\n172.20.33.129 0.0.0.0 255.255.255.255 UH 0 0 0 caliccc295a6d4f\n172.20.104.0 192.168.1.35 255.255.255.192 UG 0 0 0 ens3\n172.20.166.128 192.168.1.63 255.255.255.192 UG 0 0 0 ens3\n```\n\n**查看所有calico节点状态**\n\n``` bash\ncalicoctl node status\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-------------------+-------+----------+-------------+\n| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |\n+--------------+-------------------+-------+----------+-------------+\n| 192.168.1.34 | node-to-node mesh | up | 12:34:00 | Established |\n| 192.168.1.35 | node-to-node mesh | up | 12:34:00 | Established |\n| 192.168.1.63 | node-to-node mesh | up | 12:34:01 | Established |\n+--------------+-------------------+-------+----------+-------------+\n```\n\n**BGP 协议是通过TCP 连接来建立邻居的，因此可以用netstat 命令验证 BGP Peer**\n\n``` bash\nnetstat -antlp|grep ESTABLISHED|grep 179\ntcp 0 0 192.168.1.66:179 192.168.1.35:41316 ESTABLISHED 28479/bird \ntcp 0 0 192.168.1.66:179 192.168.1.34:40243 ESTABLISHED 28479/bird \ntcp 0 0 192.168.1.66:179 192.168.1.63:48979 ESTABLISHED 28479/bird\n```\n\n**查看etcd中calico相关信息**\n\n因为这里calico网络使用etcd存储数据，所以可以在etcd集群中查看数据\n\n+ calico 3.x 版本默认使用 etcd v3存储，**登录集群的一个etcd 节点**，查看命令：\n\n``` bash\n# 查看所有calico相关数据\nETCDCTL_API=3 etcdctl --endpoints=\"http://127.0.0.1:2379\" get --prefix /calico\n# 查看 calico网络为各节点分配的网段\nETCDCTL_API=3 etcdctl --endpoints=\"http://127.0.0.1:2379\" get --prefix /calico/ipam/v2/host\n```\n\n\n## 下一步：[设置 BGP Route Reflector](calico-bgp-rr.md)\n"
},
{
"path": "docs/setup/network-plugin/cilium-example.md",
"content": "## 开始使用 cilium\n\n以下为简要翻译 `cilium doc`上的一个应用示例[原文](https://docs.cilium.io/en/stable/gettingstarted/http/)，部署在单节点k8s 环境的实践。\n\n### 部署示例应用\n\n官方文档用几个`pod/svc` 抽象一个有趣的应用场景（星战迷）：星战中帝国方建造了被称为“终极武器”的“死星”，它是一个卫星大小的战斗空间站，它的核心是使用凯伯晶体（Kyber Crystal）的超级激光炮，剧中它的首秀就以完全火力摧毁了“杰达圣城”（Jedha）。下面将用运行于 k8s上的 pod/svc/cilium 等模拟“死星“的一个“飞船登陆”系统安全策略设计。\n\n- deploy/deathstar：作为控制整个“死星”的飞船登陆管理系统，它暴露一个SVC，提供HTTP REST 接口给飞船请求登陆使用；\n- pod/tiefighter：作为“帝国”方的常规战斗飞船，它会调用上述 HTTP 接口，请求登陆“死星”；\n- pod/xwing：作为“盟军”方的飞行舰，它也尝试调用 HTTP 接口，请求登陆“死星”；\n\n 80/TCP 4h\nservice/kubernetes ClusterIP 10.68.0.1 443/TCP 5h\n```\n每个 POD 在 `cilium` 中都表示为 `Endpoint`，初始每个 `Endpoint` 的”进出安全策略“状态均为 `Disabled`，如下：(已省略部分无关 POD 信息)\n\n``` bash\n$ kubectl exec -n kube-system cilium-6t5vx -- cilium endpoint list\nENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS \n ENFORCEMENT ENFORCEMENT \n643 Disabled Disabled 31371 k8s:class=deathstar f00d::ac14:0:0:283 172.20.0.246 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=empire \n1011 Disabled Disabled 31371 k8s:class=deathstar f00d::ac14:0:0:3f3 172.20.0.63 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=empire \n32030 Disabled Disabled 5350 k8s:class=tiefighter f00d::ac14:0:0:7d1e 172.20.0.201 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=empire \n45943 Disabled Disabled 14309 k8s:class=xwing f00d::ac14:0:0:b377 172.20.0.189 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=alliance \n52035 Disabled Disabled 4 reserved:health f00d::ac14:0:0:cb43 172.20.0.92 ready \n```\n\n### 检查初始状态\n\n当然“死星”应该只允许“帝国”的飞船着陆，因为没有应用任何策略，所以初始状态下“帝国”和“联盟”的飞船都可以登陆，如下测试：\n\n``` bash\n$ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing\nShip landed # 成功着陆\n$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing\nShip landed # 成功着陆\n```\n\n### 应用 L3/L4 策略\n\n现在我们应用策略，仅让带有标签 `org=empire`的飞船登陆“死星”；那么带有标签 `org=alliance`的“联盟”飞船将禁止登陆；这个就是我们熟悉的传统L3/L4 防火墙策略，并跟踪连接（会话）状态；\n\n \nkube-ovn kube-ovn-cni-7dwmx 1/1 Running 2 35h 192.168.1.42 192.168.1.42 \nkube-ovn kube-ovn-cni-lhlvl 1/1 Running 2 35h 192.168.1.41 192.168.1.41 \nkube-ovn kube-ovn-controller-57955db7b4-6x6hd 1/1 Running 0 35h 192.168.1.43 192.168.1.43 \nkube-ovn kube-ovn-controller-57955db7b4-chvz4 1/1 Running 0 35h 192.168.1.42 192.168.1.42 \nkube-ovn ovn-central-bb8747d77-tr5nz 1/1 Running 0 35h 192.168.1.41 192.168.1.41 \nkube-ovn ovs-ovn-2qhhr 1/1 Running 0 35h 192.168.1.41 192.168.1.41 \nkube-ovn ovs-ovn-np8rn 1/1 Running 0 35h 192.168.1.43 192.168.1.43 \nkube-ovn ovs-ovn-pkjw4 1/1 Running 0 35h 192.168.1.42 192.168.1.42 \nkube-system coredns-55f46dd959-76qb5 1/1 Running 0 35h 10.16.0.12 192.168.1.42 \nkube-system coredns-55f46dd959-wn8kw 1/1 Running 0 35h 10.16.0.11 192.168.1.43 \nkube-system heapster-fdb7596d6-xmmrx 1/1 Running 0 35h 10.16.0.15 192.168.1.42 \nkube-system kubernetes-dashboard-68ddcc97fc-dwzbf 1/1 Running 0 35h 10.16.0.14 192.168.1.42 \nkube-system metrics-server-6c898b5b8b-zvct2 1/1 Running 0 35h 10.16.0.13 192.168.1.43 \n```\n\n直观上 kube-ovn 与传统 k8s 网络（flannel/calico等）比较最大的不同是 pod 子网的分配：\n\n- 传统网络插件下，集群中 pod 一般是不同 node 节点分配不同的子网；然后通过 overlay 等技术打通不同 node 节点的 pod 子网；\n- kube-ovn 中 pod 网络根据其所在的 namespace 而定； namespace 在创建时可以根据 annotation 来配置它的子网/网关等参数；默认使用 10.16.0.0/16 的子网；\n\n### 测试 namespace 子网分配\n\n新建一个 subnet 并绑定 namespace 测试分配一个新的 pod 子网\n\n```\n# 创建一个 namespace: test-ns\n$ cat > test-ns.yaml << EOF\napiVersion: v1\nkind: Namespace\nmetadata:\n annotations:\n name: test-ns\nEOF\n$ kubectl apply -f test-ns.yaml\n\n# 创建一个 subnet: test-subnet 并绑定 namespace test-ns\n$ cat > test-subnet.yaml << EOF\napiVersion: kubeovn.io/v1\nkind: Subnet\nmetadata:\n name: test-subnet\nspec:\n protocol: IPv4\n default: false\n namespaces:\n - test-ns\n cidrBlock: 10.17.0.0/24\n gateway: 10.17.0.1\n excludeIps:\n - 10.17.0.1..10.17.0.10\nEOF\n$ kubectl apply -f test-subnet.yaml\n\n# 在 test-ns 中创建 nginx 部署\n$ kubectl run -n test-ns nginx --image=nginx --replicas=2 --port=80 --expose\n\n# 在 default 中创建 busy 客户端\n$ kubectl run busy --image=busybox sleep 360000\n```\n\n创建成功后，查看 pod 地址的分配，可以看到确实 test-ns 中 pod 使用新的子网，而 default 中 pod 使用了默认子网，并验证 pod 之间的联通性（默认可通）\n\n```\n$ kubectl get pod --all-namespaces -o wide\nNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\ndefault busy-6c55ccddc5-qrm5j 1/1 Running 0 31h 10.16.0.16 192.168.1.43 \nkube-ovn kube-ovn-cni-5php2 1/1 Running 2 35h 192.168.1.43 192.168.1.43 \nkube-ovn kube-ovn-cni-7dwmx 1/1 Running 2 35h 192.168.1.42 192.168.1.42 \nkube-ovn kube-ovn-cni-lhlvl 1/1 Running 2 35h 192.168.1.41 192.168.1.41 \nkube-ovn kube-ovn-controller-57955db7b4-6x6hd 1/1 Running 0 35h 192.168.1.43 192.168.1.43 \nkube-ovn kube-ovn-controller-57955db7b4-chvz4 1/1 Running 0 35h 192.168.1.42 192.168.1.42 \nkube-ovn ovn-central-bb8747d77-tr5nz 1/1 Running 0 35h 192.168.1.41 192.168.1.41 \nkube-ovn ovs-ovn-2qhhr 1/1 Running 0 35h 192.168.1.41 192.168.1.41 \nkube-ovn ovs-ovn-np8rn 1/1 Running 0 35h 192.168.1.43 192.168.1.43 \nkube-ovn ovs-ovn-pkjw4 1/1 Running 0 35h 192.168.1.42 192.168.1.42 \nkube-system coredns-55f46dd959-76qb5 1/1 Running 0 35h 10.16.0.12 192.168.1.42 \nkube-system coredns-55f46dd959-wn8kw 1/1 Running 0 35h 10.16.0.11 192.168.1.43 \nkube-system heapster-fdb7596d6-xmmrx 1/1 Running 0 35h 10.16.0.15 192.168.1.42 \nkube-system kubernetes-dashboard-68ddcc97fc-dwzbf 1/1 Running 0 35h 10.16.0.14 192.168.1.42 \nkube-system metrics-server-6c898b5b8b-zvct2 1/1 Running 0 35h 10.16.0.13 192.168.1.43 \ntest-ns nginx-755464dd6c-s6flj 1/1 Running 0 31h 10.17.0.12 192.168.1.42 \ntest-ns nginx-755464dd6c-zct56 1/1 Running 0 31h 10.17.0.11 192.168.1.43 \n```\n\n- 更多的测试（pod网络QOS限速，namespace网络隔离等）请参考 kube-ovn 项目说明文档\n\n### 延伸阅读\n\n- [kube-ovn 官方文档](https://github.com/alauda/kube-ovn/tree/master/docs)\n- [从 Bridge 到 OVS，探索虚拟交换机](https://www.cnblogs.com/bakari/p/8097439.html)\n"
},
{
"path": "docs/setup/network-plugin/kube-router.md",
"content": "# kube-router 网络组件\n\n(以下文档暂未更新，以插件官网文档为准)\n\nkube-router是一个简单、高效的网络插件，它提供一揽子解决方案： \n- 基于GoBGP 提供Pod 网络互联（Routing）\n- 使用ipsets优化的iptables 提供网络策略支持（Firewall/NetworkPolicy）\n- 基于IPVS/LVS 提供高性能服务代理（Service Proxy）(注：由于 k8s 新版本中 ipvs 已可用，因此这里不选择启用kube-router基于ipvs的service proxy)\n\n更多介绍请前往`https://github.com/cloudnativelabs/kube-router`\n\n## 配置\n\n本项目提供多种网络插件可选，如果需要安装kube-router，请在/etc/kubeasz/hosts文件中设置变量 `CLUSTER_NETWORK=\"kube-router\"`，更多设置请查看`roles/kube-router/defaults/main.yml`\n\n- kube-router需要在所有master节点和node节点安装\n\n## 安装\n\n- 单步安装已经集成：`ansible-playbook 90.setup.yml`\n- 分步安装请执行：`ansible-playbook 06.network.yml`\n\n## 验证\n\n- 1.pod间网络联通性：略\n\n- 2.host路由表\n\n``` bash\n# master上路由\nroot@master1:~$ ip route\n...\n172.20.1.0/24 via 192.168.1.2 dev ens3 proto 17 \n172.20.2.0/24 via 192.168.1.3 dev ens3 proto 17 \n...\n\n# node3上路由\nroot@node3:~$ ip route\n... \n172.20.0.0/24 via 192.168.1.1 dev ens3 proto 17 \n172.20.1.0/24 via 192.168.1.2 dev ens3 proto 17 \n172.20.2.0/24 dev kube-bridge proto kernel scope link src 172.20.2.1 \n...\n```\n\n- 3.bgp连接状态\n\n``` bash\n# master上\nroot@master1:~$ netstat -antlp|grep router|grep LISH|grep 179\ntcp 0 0 192.168.1.1:179 192.168.1.3:58366 ESTABLISHED 26062/kube-router\ntcp 0 0 192.168.1.1:42537 192.168.1.2:179 ESTABLISHED 26062/kube-router\n\n# node3上\nroot@node3:~$ netstat -antlp|grep router|grep LISH|grep 179\ntcp 0 0 192.168.1.3:58366 192.168.1.1:179 ESTABLISHED 18897/kube-router\ntcp 0 0 192.168.1.3:179 192.168.1.2:43928 ESTABLISHED 18897/kube-router\n\n```\n\n- 4.NetworkPolicy有效性，验证参照[这里](../../guide/networkpolicy.md)\n\n- 5.ipset列表查看\n\n``` bash\n$ ipset list\n...\nName: kube-router-pod-subnets\nType: hash:net\nRevision: 6\nHeader: family inet hashsize 1024 maxelem 65536 timeout 0\nSize in memory: 672\nReferences: 2\nMembers:\n172.20.1.0/24 timeout 0\n172.20.2.0/24 timeout 0\n172.20.0.0/24 timeout 0\n\nName: kube-router-node-ips\nType: hash:ip\nRevision: 4\nHeader: family inet hashsize 1024 maxelem 65536 timeout 0\nSize in memory: 416\nReferences: 1\nMembers:\n192.168.1.1 timeout 0\n192.168.1.2 timeout 0\n192.168.1.3 timeout 0\n...\n```\n"
},
{
"path": "docs/setup/network-plugin/network-check.md",
"content": "# network-check\n\n网络测试组件，根据cilium connectivity-check 脚本修改而来；利用cronjob 定期检测集群各节点、容器、serviceip、nodeport等之间的网络联通性；可以方便的判断当前集群网络是否正常。\n\n目前检测如下：\n\n``` bash\nkubectl get cronjobs.batch -n network-test\nNAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE\ntest01-pod-to-container */5 * * * * False 0 3m19s 6d3h\ntest02-pod-to-node-nodeport */5 * * * * False 0 3m19s 6d3h\ntest03-pod-to-multi-node-clusterip */5 * * * * False 1 6d3h 6d3h\ntest04-pod-to-multi-node-headless */5 * * * * False 1 6d3h 6d3h\ntest05-pod-to-multi-node-nodeport */5 * * * * False 1 6d3h 6d3h\ntest06-pod-to-external-1111 */5 * * * * False 0 3m19s 6d3h\ntest07-pod-to-external-fqdn-baidu */5 * * * * False 0 3m19s 6d3h\ntest08-host-to-multi-node-clusterip */5 * * * * False 1 6d3h 6d3h\ntest09-host-to-multi-node-headless */5 * * * * False 1 6d3h 6d3h\n```\n\n+ 带`multi-node`的测试需要多节点集群才能运行，如果单节点集群，测试pod会处于`Pending`状态\n+ 带`external`的测试需要节点能够访问互联网，否则测试会失败\n\n## 启用网络检测\n\n- 下载额外容器镜像 `./ezdown -X network-check`\n\n- 配置集群，在配置文件`/etc/kubeasz/clusters/xxx/config.yml` (xxx为集群名) 修改如下选项\n\n```\n# network-check 自动安装\nnetwork_check_enabled: true\nnetwork_check_schedule: \"*/5 * * * *\" # 检测频率，默认5分钟执行一次\n```\n\n- 安装网络检测插件 `docker exec -it kubeasz ezctl setup xxx 07`\n\n## 检查测试结果\n\n大约等待5分钟左右，查看运行结果，如果pod 状态为`Completed` 表示检测正常通过。\n\n```\nkubectl get pod -n network-test\nNAME READY STATUS RESTARTS AGE\necho-server-58d7bb7f6-77ps6 1/1 Running 0 6d4h\necho-server-host-cc87c966d-bk57t 1/1 Running 0 6d4h\ntest01-pod-to-container-27606775-q6xlb 0/1 Completed 0 3m10s\ntest02-pod-to-node-nodeport-27606775-x2v5d 0/1 Completed 0 3m10s\ntest03-pod-to-multi-node-clusterip-27597895-cbq8d 0/1 Pending 0 6d4h\ntest04-pod-to-multi-node-headless-27597895-qzsgz 0/1 Pending 0 6d4h\ntest05-pod-to-multi-node-nodeport-27597895-kb5r7 0/1 Pending 0 6d4h\ntest06-pod-to-external-1111-27606775-p6v8s 0/1 Completed 0 3m10s\ntest07-pod-to-external-fqdn-baidu-27606775-qdfwd 0/1 Completed 0 3m10s\ntest08-host-to-multi-node-clusterip-27597895-qsgn9 0/1 Pending 0 6d4h\ntest09-host-to-multi-node-headless-27597895-hpkt5 0/1 Pending 0 6d4h\n```\n\n+ pod 状态为`Completed` 表示检测正常通过\n+ pod 状态为`Pending` 表示该检测需要多节点的k8s集群才会运行\n\n## 禁用网络检测\n\n如果集群已经开启网络检测，检测结果符合预期，并且不想继续循环检测时，只要删除对应namespace即可\n\n```\nkubectl delete ns network-test\n```\n"
},
{
"path": "docs/setup/offline_install.md",
"content": "# 离线安装集群\n\n使用kubeasz 离线安装 k8s集群需要下载四个部分：\n\n- kubeasz 项目代码\n- 二进制文件（k8s、etcd、containerd等组件）\n- 容器镜像文件（calico、coredns、metrics-server等容器镜像）\n- 系统软件安装包（ipset、libseccomp2等，仅无法使用本地yum/apt源时需要）\n\n## 离线文件准备\n\n在一台能够访问互联网的服务器上执行：\n\n- 下载工具脚本ezdown，举例使用kubeasz版本3.6.0\n\n``` bash\nexport release=3.6.0\nwget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown\nchmod +x ./ezdown\n```\n\n- 使用工具脚本下载（更多关于ezdown的参数，运行./ezdown 查看）\n\n下载kubeasz代码、二进制、默认容器镜像\n\n``` bash\n# 国内环境\n./ezdown -D\n```\n\n[可选]如果需要更多组件，请下载额外容器镜像（cilium,flannel,prometheus等）\n\n``` bash\n./ezdown -X flannel\n./ezdown -X prometheus\n...\n```\n\n下载离线系统包 (适用于无法使用yum/apt仓库情形)\n\n``` bash\n# 如果操作系统是ubuntu 22.04\n./ezdown -P ubuntu_22\n```\n\n上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录`/etc/kubeasz`\n\n- `/etc/kubeasz` 包含 kubeasz 版本为 ${release} 的发布代码\n- `/etc/kubeasz/bin` 包含 k8s/etcd/docker/cni 等二进制文件\n- `/etc/kubeasz/down` 包含集群安装时需要的离线容器镜像\n- `/etc/kubeasz/down/packages` 包含集群安装时需要的系统基础软件\n\n## 离线安装\n\n上述下载完成后，把`/etc/kubeasz`整个目录复制到目标离线服务器相同目录，然后在离线服务器/etc/kubeasz目录下执行：\n\n- 离线安装 docker，检查本地文件，正常会提示所有文件已经下载完成，并上传到本地私有镜像仓库\n\n```\n./ezdown -D\n./ezdown -X flannel\n./ezdown -X prometheus\n...\n```\n\n- 启动 kubeasz 容器\n\n```\n./ezdown -S\n```\n\n- 设置参数允许离线安装系统软件包\n\n```\nsed -i 's/^INSTALL_SOURCE.*$/INSTALL_SOURCE: \"offline\"/g' /etc/kubeasz/example/config.yml \n```\n\n- 举例安装单节点集群，参考 https://github.com/easzlab/kubeasz/blob/master/docs/setup/quickStart.md\n\n``` bash\nsource ~/.bashrc\ndk ezctl start-aio\n# 或者执行 docker exec -it kubeasz ezctl start-aio\n```\n\n- 多节点集群，进入kubeasz 容器内 `docker exec -it kubeasz bash`，参考https://github.com/easzlab/kubeasz/blob/master/docs/setup/00-planning_and_overall_intro.md 进行集群规划和设置后使用./ezctl 命令安装\n\n"
},
{
"path": "docs/setup/quickStart.md",
"content": "## 快速指南\n\n本文档适用于kubeasz 3.3.1以上版本，部署单节点集群(aio)，作为快速体验k8s集群的测试环境。\n\n### 1.基础系统配置\n\n- 准备一台虚机配置内存4G/硬盘30G以上\n- 最小化安装`Ubuntu 22.04 server`\n- 配置基础网络、更新源、SSH登录等\n\n**注意:** 确保在干净的系统上开始安装，不能使用曾经装过kubeadm或其他k8s发行版的环境\n\n### 2.下载文件\n\n- 下载工具脚本ezdown，举例使用kubeasz版本3.6.7\n\n``` bash\nexport release=3.6.7\nwget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown\nchmod +x ./ezdown\n```\n\n- 使用工具脚本下载（更多关于ezdown的参数，运行./ezdown 查看）\n\n下载kubeasz代码、二进制、默认容器镜像\n\n``` bash\n# 国内环境\n./ezdown -D\n# 海外环境\n#./ezdown -D -m standard\n```\n\n【可选】下载额外容器镜像（cilium,flannel,prometheus等）\n\n``` bash\n# 按需下载\n./ezdown -X dashboard\n./ezdown -X prometheus\n...\n```\n\n【可选】下载离线系统包 (适用于无法使用yum/apt仓库情形)\n\n``` bash\n./ezdown -P\n```\n\n上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录`/etc/kubeasz`\n\n- `/etc/kubeasz` 包含 kubeasz 版本为 ${release} 的发布代码\n- `/etc/kubeasz/bin` 包含 k8s/etcd/docker/cni 等二进制文件\n- `/etc/kubeasz/down` 包含集群安装时需要的离线容器镜像\n- `/etc/kubeasz/down/packages` 包含集群安装时需要的系统基础软件\n\n### 3.安装集群\n\n- 容器化运行 kubeasz\n\n```\n./ezdown -S\n```\n\n- 使用默认配置安装 aio 集群\n\n```\ndocker exec -it kubeasz ezctl start-aio\n# 如果安装失败，查看日志排除后，使用如下命令重新安装aio集群\n# docker exec -it kubeasz ezctl setup default all\n```\n\n### 4.验证安装\n\n``` bash\n$ source ~/.bashrc\n$ kubectl version # 验证集群版本 \n$ kubectl get node # 验证节点就绪 (Ready) 状态\n$ kubectl get pod -A # 验证集群pod状态，默认已安装网络插件、coredns、metrics-server等\n$ kubectl get svc -A # 验证集群服务状态\n```\n\n- 登录 `dashboard`可以查看和管理集群，更多内容请查阅[dashboard文档](../guide/dashboard.md)\n\n### 5.清理\n\n以上步骤创建的K8S开发测试环境请尽情折腾，碰到错误尽量通过查看日志、上网搜索、提交`issues`等方式解决；当然你也可以清理集群后重新创建。\n\n在宿主机上，按照如下步骤清理\n\n- 清理集群 `docker exec -it kubeasz ezctl destroy default`\n- 重启节点，以确保清理残留的虚拟网卡、路由等信息\n"
},
{
"path": "example/config.yml",
"content": "############################\n# prepare\n############################\n# 可选离线安装系统软件包 (offline|online)\nINSTALL_SOURCE: \"online\"\n\n# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening\n# (deprecated) 未更新上游项目，未验证最新k8s集群安装，不建议启用\nOS_HARDEN: false\n\n\n############################\n# role:deploy\n############################\n# default: ca will expire in 100 years\n# default: certs issued by the ca will expire in 50 years\nCA_EXPIRY: \"876000h\"\nCERT_EXPIRY: \"438000h\"\n\n# force to recreate CA and other certs, not suggested to set 'true'\nCHANGE_CA: false\n\n# kubeconfig 配置参数\nCLUSTER_NAME: \"cluster1\"\nCONTEXT_NAME: \"context-{{ CLUSTER_NAME }}\"\n\n# k8s version\nK8S_VER: \"__k8s_ver__\"\n\n# set unique 'k8s_nodename' for each node, if not set(default:'') ip add will be used\n# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',\n# and must start and end with an alphanumeric character (e.g. 'example.com'),\n# regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'\nK8S_NODENAME: \"{%- if k8s_nodename != '' -%} \\\n {{ k8s_nodename|replace('_', '-')|lower }} \\\n {%- else -%} \\\n k8s-{{ inventory_hostname|replace('.', '-') }} \\\n {%- endif -%}\"\n\n# use 'K8S_NODENAME' to set hostname\nENABLE_SETTING_HOSTNAME: true\n\n\n############################\n# role:etcd\n############################\n# 设置不同的wal目录，可以避免磁盘io竞争，提高性能\nETCD_DATA_DIR: \"/var/lib/etcd\"\nETCD_WAL_DIR: \"\"\n\n\n############################\n# role:runtime [containerd,docker]\n############################\n# [.]启用拉取加速镜像仓库\nENABLE_MIRROR_REGISTRY: true\n\n# [.]添加信任的私有仓库\n# 必须按照如下示例格式，协议头'http://'和'https://'不能省略\nINSECURE_REG:\n - \"http://easzlab.io.local:5000\"\n - \"https://reg.yourcompany.com\"\n\n# [.]基础容器镜像\nSANDBOX_IMAGE: \"easzlab.io.local:5000/easzlab/pause:__pause__\"\n\n# [containerd] root 存储目录，默认：/var/lib/containerd\nCONTAINERD_ROOT_DIR: \"/var/lib/containerd\"\n\n# [containerd] state 存储目录，默认：/run/containerd\nCONTAINERD_STATE_DIR: \"/run/containerd\"\n\n# [containerd] config 目录，默认：/etc/containerd\nCONTAINERD_CONFIG_DIR: \"/etc/containerd\"\n\n# [containerd] systemd service 名称，默认：containerd.service\nCONTAINERD_SERVICE_NAME: \"containerd.service\"\n\n# [docker]容器存储目录\nDOCKER_STORAGE_DIR: \"/var/lib/docker\"\n\n# [docker]开启Restful API\nDOCKER_ENABLE_REMOTE_API: false\n\n\n############################\n# role:kube-master\n############################\n# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）\nMASTER_CERT_HOSTS:\n - \"10.1.1.1\"\n - \"k8s.easzlab.io\"\n #- \"www.test.com\"\n\n# node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）\n# 如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段\n# https://github.com/coreos/flannel/issues/847\nNODE_CIDR_LEN: 24\n\n# 是否启用集群audit功能\nENABLE_CLUSTER_AUDIT: false\n\n############################\n# role:kube-node\n############################\n# Kubelet 根目录\nKUBELET_ROOT_DIR: \"/var/lib/kubelet\"\n\n# node节点最大pod 数\nMAX_PODS: 110\n\n# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量\n# 数值设置详见templates/kubelet-config.yaml.j2\nKUBE_RESERVED_ENABLED: \"no\"\n\n# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；\n# 并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2\n# 系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留\n# 另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存\nSYS_RESERVED_ENABLED: \"no\"\n\n\n############################\n# role:network [flannel,calico,cilium,kube-ovn,kube-router]\n############################\n# ------------------------------------------- flannel\n# [flannel]设置flannel 后端\"host-gw\",\"vxlan\"等\nFLANNEL_BACKEND: \"vxlan\"\nDIRECT_ROUTING: false\n\n# [flannel] \nflannel_ver: \"__flannel__\"\n\n# ------------------------------------------- calico\n# 模式可选项有: [Always, CrossSubnet, Never],跨子网可以配置为Always与CrossSubnet\n# CrossSubnet为隧道+BGP路由混合模式可以提升网络性能，同子网配置为Never即可.\n# 公有云建议使用always比较省事，其他的话需要修改各自公有云的网络配置，具体可以参考各个公有云说明\nCALICO_ENABLE_OVERLAY: \"Always\"\n\n# [calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现\nIP_AUTODETECTION_METHOD: \"can-reach={{ groups['kube_master'][0] }}\"\n\n# [calico]设置calico 网络 backend: bird, vxlan, none\n# 少数公有云（Azure）或者私有云不支持IPinIP封包，可以使用 vxlan 模式\nCALICO_NETWORKING_BACKEND: \"bird\"\n\n# [calico]设置calico 是否使用route reflectors\n# 如果集群规模超过50个节点，建议启用该特性\nCALICO_RR_ENABLED: false\n\n# CALICO_RR_NODES 配置route reflectors的节点，如果未设置默认使用集群master节点 \n# CALICO_RR_NODES: [\"192.168.1.1\", \"192.168.1.2\"]\nCALICO_RR_NODES: []\n\n# [calico]更新支持calico 版本: [\"3.19\", \"3.23\"]\ncalico_ver: \"__calico__\"\n\n# [calico]calico 主版本\ncalico_ver_main: \"{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}\"\n\n# ------------------------------------------- cilium\n# [cilium]镜像版本\ncilium_ver: \"__cilium__\"\ncilium_connectivity_check: false\ncilium_hubble_enabled: false\ncilium_hubble_ui_enabled: false\n\n# ------------------------------------------- kube-ovn\n# [kube-ovn]离线镜像tar包\nkube_ovn_ver: \"__kube_ovn__\"\n\n# ------------------------------------------- kube-router\n# [kube-router]公有云上存在限制，一般需要始终开启 ipinip；自有环境可以设置为 \"subnet\"\nOVERLAY_TYPE: \"full\"\n\n# [kube-router]NetworkPolicy 支持开关\nFIREWALL_ENABLE: true\n\n# [kube-router]kube-router 镜像版本\nkube_router_ver: \"__kube_router__\"\n\n\n############################\n# role:cluster-addon\n############################\n# coredns 自动安装\ndns_install: \"yes\"\ncorednsVer: \"__coredns__\"\nENABLE_LOCAL_DNS_CACHE: true\ndnsNodeCacheVer: \"__dns_node_cache__\"\n# 设置 local dns cache 地址\nLOCAL_DNS_CACHE: \"169.254.20.10\"\n\n# metric server 自动安装\nmetricsserver_install: \"yes\"\nmetricsVer: \"__metrics__\"\n\n# dashboard 自动安装\ndashboard_install: \"no\"\ndashboardVer: \"__dashboard__\"\n\n# local-storage (local-path-provisioner) 自动安装\nlocal_path_provisioner_install: \"no\"\nlocal_path_provisioner_ver: \"__local_path_provisioner__\"\nlocal_path_storage_class: \"local-path\"\n# 设置默认本地存储路径\nlocal_path_provisioner_dir: \"/opt/local-path-provisioner\"\n\n# nfs-provisioner 自动安装\nnfs_provisioner_install: \"no\"\nnfs_provisioner_namespace: \"kube-system\"\nnfs_provisioner_ver: \"__nfs_provisioner__\"\nnfs_storage_class: \"managed-nfs-storage\"\nnfs_server: \"192.168.1.10\"\nnfs_path: \"/data/nfs\"\n\n# openebs 自动安装\nopenebs_install: \"no\"\nopenebs_ver: \"__openebs_ver__\"\nopenebs_namespace: \"openebs\"\nopenebs_hostpath: \"/var/openebs/local\"\nopenebs_hostpath_storage_class: \"openebs-hostpath\"\nopenebs_lvm_storage_class: \"openebs-lvmpv\"\nopenebs_lvm_vg: \"vg_k8s\"\n\n# prometheus 自动安装\nprom_install: \"no\"\nprom_namespace: \"monitor\"\nprom_storage_class: \"\"\nprom_chart_ver: \"__prom_chart__\"\n\n# minio 自动安装\nminio_install: \"no\"\nminio_namespace: \"minio\"\nminio_storage_class: \"{{ openebs_lvm_storage_class }}\"\nminio_chart_ver: \"__minio_chart__\"\nminio_root_user: \"3aea61ca94177dx\"\nminio_root_password: \"0f3b19e46dd3aea61ca94177d\"\n# 单机版=1，集群版=4以上\nminio_pool_servers: 4\nminio_pool_size: 10Gi\n# 是否启用tls证书，如果未启用则使用http协议\nminio_tls_enabled: false\n# 是否使用权威证书，如果使用需要提前把证书放到目录 roles/cluster-addon/templates/minio/; 并且要求\n# 证书和私钥的名称分别为server.crt和server.key\nminio_with_global_cert: false\n\n# nacos 自动安装\nnacos_install: \"no\"\nnacos_namespace: \"nacos\"\nnacos_mysql_host: \"semisync-mysql-cluster-mysql\"\nnacos_mysql_db: \"nacos\"\nnacos_mysql_port: \"3306\"\nnacos_mysql_user: \"__dbuser__\"\nnacos_mysql_password: \"__yourpassword__\"\nnacos_storage_class: \"{{ openebs_lvm_storage_class }}\"\n\n# rocketmq 自动安装\nrocketmq_install: \"no\"\nrocketmq_namespace: \"rocketmq\"\nrocketmq_storage_class: \"{{ openebs_lvm_storage_class }}\"\n\n# network-check 自动安装\nnetwork_check_enabled: false \nnetwork_check_schedule: \"*/5 * * * *\"\n\n# kubeblocks 自动安装\nkubeblocks_ver: \"__kubeblocks_ver__\"\nkubeblocks_install: \"no\"\n\n# ingress-nginx 自动安装\n# ingress-nginx 只会部署到node with 标签：ingress-controller/provider=ingress-nginx\ningress_nginx_install: \"no\"\ningress_nginx_namespace: \"ingress-nginx\"\ningress_nginx_ver: \"__ingress_nginx_ver__\"\ningress_nginx_metrics_enabled: true # 需要先部署prometheus\n\n# argocd 自动安装\nargocd_install: \"no\"\n\n\n############################\n# role:harbor\n############################\n# harbor version，完整版本号\nHARBOR_VER: \"__harbor__\"\nHARBOR_DOMAIN: \"harbor.easzlab.io.local\"\nHARBOR_PATH: /var/data\nHARBOR_TLS_PORT: 8443\nHARBOR_REGISTRY: \"{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}\"\n\n# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'\nHARBOR_SELF_SIGNED_CERT: true\n\n# install extra component\nHARBOR_WITH_TRIVY: false\n"
},
{
"path": "example/hosts.allinone",
"content": "# 'etcd' cluster should have odd member(s) (1,3,5,...)\n[etcd]\n192.168.1.1\n\n# master node(s), set unique 'k8s_nodename' for each node\n# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',\n# and must start and end with an alphanumeric character\n[kube_master]\n192.168.1.1\n\n# work node(s), set unique 'k8s_nodename' for each node\n# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',\n# and must start and end with an alphanumeric character\n[kube_node]\n192.168.1.1 k8s_nodename=''\n\n# [optional] harbor server, a private docker registry\n# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one\n[harbor]\n#192.168.1.8 NEW_INSTALL=false\n\n# [optional] loadbalance for accessing k8s from outside\n[ex_lb]\n#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443\n#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443\n\n# [optional] ntp server for the cluster\n[chrony]\n#192.168.1.1\n\n[all:vars]\n# --------- Main Variables ---------------\n# Secure port for apiservers\nSECURE_PORT=\"6443\"\n\n# Cluster container-runtime supported: docker, containerd\n# if k8s version >= 1.24, docker is not supported\nCONTAINER_RUNTIME=\"containerd\"\n\n# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn\nCLUSTER_NETWORK=\"calico\"\n\n# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'\nPROXY_MODE=\"ipvs\"\n\n# K8S Service CIDR, not overlap with node(host) networking\nSERVICE_CIDR=\"10.68.0.0/16\"\n\n# Cluster CIDR (Pod CIDR), not overlap with node(host) networking\nCLUSTER_CIDR=\"172.20.0.0/16\"\n\n# NodePort Range\nNODE_PORT_RANGE=\"30000-32767\"\n\n# Cluster DNS Domain\nCLUSTER_DNS_DOMAIN=\"cluster.local\"\n\n# -------- Additional Variables (don't change the default value right now)---\n# Binaries Directory\nbin_dir=\"/opt/kube/bin\"\n\n# Deploy Directory (kubeasz workspace)\nbase_dir=\"/etc/kubeasz\"\n\n# Directory for a specific cluster\ncluster_dir=\"{{ base_dir }}/clusters/_cluster_name_\"\n\n# CA and other components cert/key Directory\nca_dir=\"/etc/kubernetes/ssl\"\n\n# Default 'k8s_nodename' is empty\nk8s_nodename=''\n\n# Default python interpreter\nansible_python_interpreter=/usr/bin/python3\n"
},
{
"path": "example/hosts.multi-node",
"content": "# 'etcd' cluster should have odd member(s) (1,3,5,...)\n[etcd]\n192.168.1.1\n192.168.1.2\n192.168.1.3\n\n# master node(s), set unique 'k8s_nodename' for each node\n# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',\n# and must start and end with an alphanumeric character\n[kube_master]\n192.168.1.1 k8s_nodename='master-01'\n192.168.1.2 k8s_nodename='master-02'\n192.168.1.3 k8s_nodename='master-03'\n\n# work node(s), set unique 'k8s_nodename' for each node\n# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',\n# and must start and end with an alphanumeric character\n[kube_node]\n192.168.1.4 k8s_nodename='worker-01'\n192.168.1.5 k8s_nodename='worker-02'\n\n# [optional] harbor server, a private docker registry\n# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one\n[harbor]\n#192.168.1.8 NEW_INSTALL=false\n\n# [optional] loadbalance for accessing k8s from outside\n[ex_lb]\n#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443\n#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443\n\n# [optional] ntp server for the cluster\n[chrony]\n#192.168.1.1\n\n[all:vars]\n# --------- Main Variables ---------------\n# Secure port for apiservers\nSECURE_PORT=\"6443\"\n\n# Cluster container-runtime supported: docker, containerd\n# if k8s version >= 1.24, docker is not supported\nCONTAINER_RUNTIME=\"containerd\"\n\n# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn\nCLUSTER_NETWORK=\"calico\"\n\n# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'\nPROXY_MODE=\"ipvs\"\n\n# K8S Service CIDR, not overlap with node(host) networking\nSERVICE_CIDR=\"10.68.0.0/16\"\n\n# Cluster CIDR (Pod CIDR), not overlap with node(host) networking\nCLUSTER_CIDR=\"172.20.0.0/16\"\n\n# NodePort Range\nNODE_PORT_RANGE=\"30000-32767\"\n\n# Cluster DNS Domain\nCLUSTER_DNS_DOMAIN=\"cluster.local\"\n\n# -------- Additional Variables (don't change the default value right now) ---\n# Binaries Directory\nbin_dir=\"/opt/kube/bin\"\n\n# Deploy Directory (kubeasz workspace)\nbase_dir=\"/etc/kubeasz\"\n\n# Directory for a specific cluster\ncluster_dir=\"{{ base_dir }}/clusters/_cluster_name_\"\n\n# CA and other components cert/key Directory\nca_dir=\"/etc/kubernetes/ssl\"\n\n# Default 'k8s_nodename' is empty\nk8s_nodename=''\n\n# Default python interpreter\nansible_python_interpreter=/usr/bin/python3\n"
},
{
"path": "ezctl",
"content": "#!/bin/bash\n# Create & manage k8s clusters\n# shellcheck disable=SC2155\n\nset -o nounset\nset -o errexit\n#set -o xtrace\n\nfunction usage() {\n echo -e \"\\033[33mUsage:\\033[0m ezctl COMMAND [args]\"\n cat < to switch default kubeconfig of the cluster\n new to start a new k8s deploy with name 'cluster'\n setup to setup a cluster, also supporting a step-by-step way\n start to start all of the k8s services stopped by 'ezctl stop'\n stop to stop all of the k8s services temporarily\n upgrade to upgrade the k8s cluster\n destroy to destroy the k8s cluster\n backup to backup the cluster state (etcd snapshot)\n restore to restore the cluster state from backups\n start-aio\t\t to quickly setup an all-in-one cluster with default settings\n\nCluster ops:\n add-etcd to add a etcd-node to the etcd cluster\n add-master to add a master node to the k8s cluster\n add-node to add a work node to the k8s cluster\n del-etcd to delete a etcd-node from the etcd cluster\n del-master to delete a master node from the k8s cluster\n del-node to delete a work node from the k8s cluster\n\nExtra operation:\n kca-renew to force renew CA certs and all the other certs (with caution)\n kcfg-adm to manage client kubeconfig of the k8s cluster\n\nUse \"ezctl help \" for more information about a given command.\nEOF\n}\n\nfunction logger() {\n TIMESTAMP=$(date +'%Y-%m-%d %H:%M:%S')\n local FNAME=$(basename \"${BASH_SOURCE[1]}\")\n local SOURCE=\"\\033[36m[$FNAME:${BASH_LINENO[0]}]\\033[0m\"\n case \"$1\" in\n debug)\n echo -e \"\\033[36m$TIMESTAMP\\033[0m $SOURCE \\033[36mDEBUG $2\\033[0m\"\n ;;\n info)\n echo -e \"\\033[36m$TIMESTAMP\\033[0m $SOURCE \\033[32mINFO $2\\033[0m\"\n ;;\n warn)\n echo -e \"\\033[36m$TIMESTAMP\\033[0m $SOURCE \\033[33mWARN $2\\033[0m\"\n ;;\n error)\n echo -e \"\\033[36m$TIMESTAMP\\033[0m $SOURCE \\033[31mERROR $2\\033[0m\"\n ;;\n *) ;;\n esac\n}\n\nfunction help-info() {\n case \"$1\" in\n (setup)\n\t usage-setup\n ;;\n (add-etcd)\n echo -e \"read more > 'https://github.com/easzlab/kubeasz/blob/master/docs/op/op-etcd.md'\"\n ;;\n (add-master)\n echo -e \"read more > 'https://github.com/easzlab/kubeasz/blob/master/docs/op/op-master.md'\"\n ;;\n (add-node)\n echo -e \"read more > 'https://github.com/easzlab/kubeasz/blob/master/docs/op/op-node.md'\"\n ;;\n (del-etcd)\n echo -e \"read more > 'https://github.com/easzlab/kubeasz/blob/master/docs/op/op-etcd.md'\"\n ;;\n (del-master)\n echo -e \"read more > 'https://github.com/easzlab/kubeasz/blob/master/docs/op/op-master.md'\"\n ;;\n (del-node)\n echo -e \"read more > 'https://github.com/easzlab/kubeasz/blob/master/docs/op/op-node.md'\"\n ;;\n (kca-renew)\n\t echo -e \"WARNNING: this command should be used with caution\"\n\t echo -e \"force to recreate CA certs and all of the others certs used in the cluster\"\n\t echo -e \"it should be used only when the admin.conf leaked\"\n ;;\n (kcfg-adm)\n\t usage-kcfg-adm\n ;;\n (*)\n echo -e \"todo: help info $1\"\n ;;\n esac\n}\n\nfunction usage-kcfg-adm(){\n echo -e \"\\033[33mUsage:\\033[0m ezctl kcfg-adm \"\n cat <:\n -A to add a client kubeconfig with a newly created user\n -D to delete a client kubeconfig with the existed user\n -L to list all of the users\n -e to set expiry of the user certs in hours (ex. 24h, 8h, 240h)\n -t to set a user-type (admin or view)\n -u to set a user-name prefix\n\nexamples: ./ezctl kcfg-adm test-k8s -L\n ./ezctl kcfg-adm default -A -e 240h -t admin -u jack\n ./ezctl kcfg-adm default -D -u jim-202101162141\nEOF\n}\n\nfunction usage-setup(){\n echo -e \"\\033[33mUsage:\\033[0m ezctl setup \"\n cat < /dev/null 2>&1 || { logger debug \"disable registry mirrors\"; registryMirror=false; }\n\n sed -i -e \"s/__k8s_ver__/$k8sVer/g\" \\\n -e \"s/__flannel__/$flannelVer/g\" \\\n\t -e \"s/__calico__/$calicoVer/g\" \\\n\t -e \"s/__cilium__/$ciliumVer/g\" \\\n\t -e \"s/__kube_ovn__/$kubeOvnVer/g\" \\\n\t -e \"s/__kube_router__/$kubeRouterVer/g\" \\\n\t -e \"s/__coredns__/$corednsVer/g\" \\\n\t -e \"s/__pause__/$pauseVer/g\" \\\n\t -e \"s/__dns_node_cache__/$dnsNodeCacheVer/g\" \\\n\t -e \"s/__dashboard__/$dashboardVer/g\" \\\n\t -e \"s/__local_path_provisioner__/$localpathProvisionerVer/g\" \\\n\t -e \"s/__nfs_provisioner__/$nfsProvisionerVer/g\" \\\n\t -e \"s/__openebs_ver__/$openebsVer/g\" \\\n\t -e \"s/__prom_chart__/$promChartVer/g\" \\\n\t -e \"s/__minio_chart__/$minioOperatorVer/g\" \\\n\t -e \"s/__kubeblocks_ver__/$kubeblocksVer/g\" \\\n\t -e \"s/__ingress_nginx_ver__/$ingressNginxVer/g\" \\\n\t -e \"s/__harbor__/$HARBOR_VER/g\" \\\n\t -e \"s/^ENABLE_MIRROR_REGISTRY.*$/ENABLE_MIRROR_REGISTRY: $registryMirror/g\" \\\n\t -e \"s/__metrics__/$metricsVer/g\" \"clusters/$1/config.yml\"\n\n\n logger debug \"cluster $1: files successfully created.\"\n logger info \"next steps 1: to config '$BASE/clusters/$1/hosts'\"\n logger info \"next steps 2: to config '$BASE/clusters/$1/config.yml'\"\n}\n\nfunction setup() {\n [[ -d \"clusters/$1\" ]] || { logger error \"invalid config, run 'ezctl new $1' first\"; return 1; }\n [[ -f \"bin/kube-apiserver\" ]] || { logger error \"no binaries founded, run 'ezdown -D' fist\"; return 1; }\n\n # for extending usage\n EXTRA_ARGS=$(echo \"$*\"|sed \"s/$1 $2//g\"|sed \"s/^ *//g\")\n\n PLAY_BOOK=\"dummy.yml\"\n case \"$2\" in\n (01|prepare)\n PLAY_BOOK=\"01.prepare.yml\"\n ;;\n (02|etcd)\n PLAY_BOOK=\"02.etcd.yml\"\n ;;\n (03|container-runtime)\n PLAY_BOOK=\"03.runtime.yml\"\n ;;\n (04|kube-master)\n PLAY_BOOK=\"04.kube-master.yml\"\n ;;\n (05|kube-node)\n PLAY_BOOK=\"05.kube-node.yml\"\n ;;\n (06|network)\n PLAY_BOOK=\"06.network.yml\"\n ;;\n (07|cluster-addon)\n PLAY_BOOK=\"07.cluster-addon.yml\"\n ;;\n (90|all)\n PLAY_BOOK=\"90.setup.yml\"\n ;;\n (10|ex-lb)\n PLAY_BOOK=\"10.ex-lb.yml\"\n ;;\n (11|harbor)\n PLAY_BOOK=\"11.harbor.yml\"\n ;;\n (*)\n usage-setup\n exit 1\n ;;\n esac\n\n COMMAND=\"ansible-playbook -i clusters/$1/hosts -e @clusters/$1/config.yml $EXTRA_ARGS playbooks/$PLAY_BOOK\"\n echo \"$COMMAND\"\n\n k8s_ver=$(bin/kube-apiserver --version|cut -d' ' -f2)\n etcd_ver=v$(bin/etcd --version|grep 'etcd Version'|cut -d' ' -f3)\n network_cni=$(grep CLUSTER_NETWORK \"clusters/$1/hosts\"|cut -d'\"' -f2|sed 's/-//g')\n network_cni_ver=$(grep -i \"${network_cni}Ver\" ezdown|cut -d'=' -f2|head -n1)\n\n cat < /dev/null 2>&1 || { logger error \"md5sum not found\"; return 1; }\n\n CLUSTERS=$(cd clusters && echo -- *)\n CFG_MD5=$(sed '/server/d' ~/.kube/config|md5sum|cut -d' ' -f1)\n cd \"$BASE\"\n\n logger info \"list of managed clusters:\"\n i=1; for c in $CLUSTERS;\n do\n if [[ -f \"clusters/$c/kubectl.kubeconfig\" ]];then\n c_md5=$(sed '/server/d' \"clusters/$c/kubectl.kubeconfig\"|md5sum|cut -d' ' -f1)\n if [[ \"$c_md5\" = \"$CFG_MD5\" ]];then\n echo -e \"==> cluster $i:\\t$c (\\033[32mcurrent\\033[0m)\"\n else\n echo -e \"==> cluster $i:\\t$c\"\n fi\n ((i++))\n fi\n done\n}\n\n\nfunction checkout() {\n [[ -d \"clusters/$1\" ]] || { logger error \"invalid config, run 'ezctl new $1' first\"; return 1; }\n [[ -f \"clusters/$1/kubectl.kubeconfig\" ]] || { logger error \"invalid kubeconfig, run 'ezctl setup $1' first\"; return 1; }\n logger info \"set default kubeconfig: cluster $1 (\\033[32mcurrent\\033[0m)\"\n /bin/cp -f \"clusters/$1/kubectl.kubeconfig\" ~/.kube/config\n}\n\n### in-cluster operation functions ##############################\n\nfunction add-node() {\n # check new node's address regexp\n [[ $2 =~ ^(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})(\\.(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})){3}$ ]] || { logger error \"Invalid ip add:$2\"; return 1; }\n\n # check if the new node already exsited\n sed -n '/^\\[kube_master/,/^\\[harbor/p' \"$BASE/clusters/$1/hosts\"|grep -E \"^$2$|^$2 \" && { logger error \"node $2 already existed in $BASE/clusters/$1/hosts\"; return 2; }\n\n logger info \"add $2 into 'kube_node' group\"\n NODE_INFO=\"${@:2}\"\n sed -i \"/\\[kube_node/a $NODE_INFO\" \"$BASE/clusters/$1/hosts\"\n\n logger info \"start to add a work node:$2 into cluster:$1\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/22.addnode.yml\" -e \"NODE_TO_ADD=$2\" -e \"@clusters/$1/config.yml\"\n}\n\nfunction add-master() {\n # check new master's address regexp\n [[ $2 =~ ^(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})(\\.(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})){3}$ ]] || { logger error \"Invalid ip add:$2\"; return 1; }\n\n # check if the new master already exsited\n sed -n '/^\\[kube_master/,/^\\[kube_node/p' \"$BASE/clusters/$1/hosts\"|grep -E \"^$2$|^$2 \" && { logger error \"master $2 already existed!\"; return 2; }\n\n logger info \"add $2 into 'kube_master' group\"\n MASTER_INFO=\"${@:2}\"\n sed -i \"/\\[kube_master/a $MASTER_INFO\" \"$BASE/clusters/$1/hosts\"\n\n logger info \"start to add a master node:$2 into cluster:$1\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/23.addmaster.yml\" -e \"NODE_TO_ADD=$2\" -e \"@clusters/$1/config.yml\"\n\n logger info \"re-setting /etc/hosts for all nodes\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/90.setup.yml\" -t set_hosts -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfigure and restart 'kube-lb' service\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/90.setup.yml\" -t restart_kube-lb -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfigure and restart 'ex-lb' service\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/10.ex-lb.yml\" -t restart_lb -e \"@clusters/$1/config.yml\"\n}\n\nfunction add-etcd() {\n # check new node's address regexp\n [[ $2 =~ ^(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})(\\.(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})){3}$ ]] || { logger error \"Invalid ip add:$2\"; return 1; }\n\n # check if the new node already exsited\n sed -n '/^\\[etcd/,/^\\[kube_master/p' \"$BASE/clusters/$1/hosts\"|grep -E \"^$2$|^$2 \" && { logger error \"etcd $2 already existed!\"; return 2; }\n\n logger info \"add $2 into 'etcd' group\"\n ETCD_INFO=\"${@:2}\"\n sed -i \"/\\[etcd/a $ETCD_INFO\" \"$BASE/clusters/$1/hosts\"\n\n logger info \"start to add a etcd node:$2 into cluster:$1\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/21.addetcd.yml\" -e \"NODE_TO_ADD=$2\" -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfig &restart the etcd cluster\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/02.etcd.yml\" -t restart_etcd -e \"@clusters/$1/config.yml\"\n\n logger info \"restart apiservers to use the new etcd cluster\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/04.kube-master.yml\" -t restart_master -e \"@clusters/$1/config.yml\"\n}\n\nfunction del-etcd() {\n # check node's address regexp\n [[ $2 =~ ^(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})(\\.(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})){3}$ ]] || { logger error \"Invalid ip add:$2\"; return 1; }\n\n # check if the deleting node exsited\n sed -n '/^\\[etcd/,/^\\[kube_master/p' \"$BASE/clusters/$1/hosts\"|grep -E \"^$2$|^$2 \" || { logger error \"etcd $2 not existed!\"; return 2; }\n\n logger warn \"start to delete the etcd node:$2 from cluster:$1\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/31.deletcd.yml\" -e \"ETCD_TO_DEL=$2\" -e \"CLUSTER=$1\" -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfig &restart the etcd cluster\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/02.etcd.yml\" -t restart_etcd -e \"@clusters/$1/config.yml\"\n\n logger info \"restart apiservers to use the new etcd cluster\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/04.kube-master.yml\" -t restart_master -e \"@clusters/$1/config.yml\"\n}\n\nfunction del-node() {\n # check node's address regexp\n [[ $2 =~ ^(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})(\\.(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})){3}$ ]] || { logger error \"Invalid ip add:$2\"; return 2; }\n\n # check if the deleting node exsited\n sed -n '/^\\[kube_master/,/^\\[harbor/p' \"$BASE/clusters/$1/hosts\"|grep -E \"^$2$|^$2 \" || { logger error \"node $2 not existed in $BASE/clusters/$1/hosts\"; return 2; }\n\n logger warn \"start to delete the node:$2 from cluster:$1\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/32.delnode.yml\" -e \"NODE_TO_DEL=$2\" -e \"CLUSTER=$1\" -e \"@clusters/$1/config.yml\"\n}\n\nfunction del-master() {\n # check node's address regexp\n [[ $2 =~ ^(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})(\\.(2(5[0-5]{1}|[0-4][0-9]{1})|[0-1]?[0-9]{1,2})){3}$ ]] || { logger error \"Invalid ip add:$2\"; return 2; }\n\n # check if the deleting master exsited\n sed -n '/^\\[kube_master/,/^\\[kube_node/p' \"$BASE/clusters/$1/hosts\"|grep -E \"^$2$|^$2 \" || { logger error \"master $2 not existed!\"; return 2; }\n\n NODE_NAME=$(bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" get node -owide|grep \" $2 \"|awk '{print $1}')\n\n logger warn \"start to delete the master:$2 from cluster:$1\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/33.delmaster.yml\" -e \"NODE_TO_DEL=$2\" -e \"CLUSTER=$1\" -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfig kubeconfig in ansible manage node\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/roles/deploy/deploy.yml\" -t create_kctl_cfg -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfigure and restart 'kube-lb' service\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/90.setup.yml\" -t restart_kube-lb -e \"@clusters/$1/config.yml\"\n\n logger info \"reconfigure and restart 'ex-lb' service\"\n ansible-playbook -i \"$BASE/clusters/$1/hosts\" \"$BASE/playbooks/10.ex-lb.yml\" -t restart_lb -e \"@clusters/$1/config.yml\"\n\n logger info \"delete the master-node: $NODE_NAME\"\n bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" delete node \"$NODE_NAME\"\n}\n\n\nfunction start-aio(){\n set +u\n # Check ENV 'HOST_IP', exists if the CMD 'ezctl' running in a docker container\n if [[ -z $HOST_IP ]];then\n # ezctl runs in a host machine, get host's ip\n HOST_IF=$(ip route|grep default|head -n1|cut -d' ' -f5)\n HOST_IP=$(ip a|grep \"$HOST_IF$\"|head -n1|awk '{print $2}'|cut -d'/' -f1)\n fi\n set -u\n logger info \"get local host ipadd: $HOST_IP\"\n\n # allow ssh login using key locally\n if [[ ! -e /root/.ssh/id_rsa ]]; then\n logger debug \"generate ssh key pair\"\n ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa > /dev/null\n cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys\n ssh-keyscan -t ecdsa -H \"$HOST_IP\" >> /root/.ssh/known_hosts\n fi\n\n new default\n /bin/cp -f example/hosts.allinone \"clusters/default/hosts\"\n sed -i \"s/_cluster_name_/default/g\" \"clusters/default/hosts\"\n sed -i \"s/192.168.1.1/$HOST_IP/g\" \"clusters/default/hosts\"\n\n setup default all\n}\n\n### Extra functions #############################################\nfunction renew-ca() {\n [[ -d \"clusters/$1\" ]] || { logger error \"invalid cluster, run 'ezctl new $1' first\"; return 1; }\n\n logger warn \"WARNNING: this script should be used with greate caution\"\n logger warn \"WARNNING: it will recreate CA certs and all of the others certs used in the cluster\"\n\n COMMAND=\"ansible-playbook -i clusters/$1/hosts -e @clusters/$1/config.yml -e CHANGE_CA=true playbooks/96.update-certs.yml -t force_change_certs\"\n echo \"$COMMAND\"\n logger info \"cluster:$1 process begins in 5s, press any key to abort:\\n\"\n ! (read -r -t5 -n1) || { logger warn \"process abort\"; return 1; }\n\n ${COMMAND} || return 1\n}\n\n\nEXPIRY=\"4800h\" # default cert will expire in 200 days\nUSER_TYPE=\"admin\" # admin/view, admin=clusterrole:cluster-admin view=clusterrole:view\nUSER_NAME=\"user\"\nfunction kcfg-adm() {\n OPTIND=2\n ACTION=\"\"\n while getopts \"ADLe:t:u:\" OPTION; do\n case $OPTION in\n A)\n ACTION=\"add-kcfg $1\"\n ;;\n D)\n ACTION=\"del-kcfg $1\"\n ;;\n L)\n ACTION=\"list-kcfg $1\"\n ;;\n e)\n EXPIRY=\"$OPTARG\"\n\t [[ $OPTARG =~ ^[1-9][0-9]*h$ ]] || { logger error \"'-e' must be set like '2h, 5h, 50000h, ...'\"; exit 1; }\n ;;\n t)\n USER_TYPE=\"$OPTARG\"\n\t [[ $OPTARG =~ ^(admin|view)$ ]] || { logger error \"'-t' can only be set as 'admin' or 'view'\"; exit 1; }\n ;;\n u)\n USER_NAME=\"$OPTARG\"\n ;;\n ?)\n help-info kcfg-adm\n return 1\n ;;\n esac\n done\n\n [[ \"$ACTION\" == \"\" ]] && { logger error \"illegal option\"; help-info kcfg-adm; exit 1; }\n\n logger info \"$ACTION\"\n ${ACTION} || { logger error \"$ACTION fail\"; return 1; }\n logger info \"$ACTION success\"\n}\n\nfunction add-kcfg(){\n USER_NAME=\"$USER_NAME\"-$(date +'%Y%m%d%H%M')\n logger info \"add-kcfg in cluster:$1 with user:$USER_NAME\"\n ansible-playbook -i \"clusters/$1/hosts\" -e \"@clusters/$1/config.yml\" -e \"CUSTOM_EXPIRY=$EXPIRY\" \\\n -e \"USER_TYPE=$USER_TYPE\" -e \"USER_NAME=$USER_NAME\" -e \"ADD_KCFG=true\" \\\n -t add-kcfg \"roles/deploy/deploy.yml\"\n}\n\nfunction del-kcfg(){\n logger info \"del-kcfg in cluster:$1 with user:$USER_NAME\"\n CRB=$(bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" get clusterrolebindings -ojsonpath=\"{.items[?(@.subjects[0].name == '$USER_NAME')].metadata.name}\") && \\\n bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" delete clusterrolebindings \"$CRB\" && \\\n /bin/rm -f \"clusters/$1/ssl/users/$USER_NAME\"*\n}\n\nfunction list-kcfg(){\n logger info \"list-kcfg in cluster:$1\"\n ADMINS=$(bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" get clusterrolebindings -ojsonpath='{.items[?(@.roleRef.name == \"cluster-admin\")].subjects[*].name}')\n VIEWS=$(bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" get clusterrolebindings -ojsonpath='{.items[?(@.roleRef.name == \"view\")].subjects[*].name}')\n ALL=$(bin/kubectl --kubeconfig=\"clusters/$1/kubectl.kubeconfig\" get clusterrolebindings -ojsonpath='{.items[*].subjects[*].name}')\n\n printf \"\\n%-30s %-15s %-20s\\n\" USER TYPE \"EXPIRY(+8h if in Asia/Shanghai)\"\n echo \"---------------------------------------------------------------------------------\"\n\n for u in $ADMINS; do\n if [[ $u =~ ^.*-[0-9]{12}$ ]];then\n t=$(bin/cfssl-certinfo -cert \"clusters/$1/ssl/users/$u.pem\"|grep not_after|awk '{print $2}'|sed 's/\"//g'|sed 's/,//g')\n printf \"%-30s %-15s %-20s\\n\" \"$u\" cluster-admin \"$t\"\n fi\n done;\n\n for u in $VIEWS; do\n if [[ $u =~ ^.*-[0-9]{12}$ ]];then\n t=$(bin/cfssl-certinfo -cert \"clusters/$1/ssl/users/$u.pem\"|grep not_after|awk '{print $2}'|sed 's/\"//g'|sed 's/,//g')\n printf \"%-30s %-15s %-20s\\n\" \"$u\" view \"$t\"\n fi\n done;\n\n for u in $ALL; do\n if [[ $u =~ ^.*-[0-9]{12}$ ]];then\n [[ $ADMINS == *$u* ]] || [[ $VIEWS == *$u* ]] || {\n t=$(bin/cfssl-certinfo -cert \"clusters/$1/ssl/users/$u.pem\"|grep not_after|awk '{print $2}'|sed 's/\"//g'|sed 's/,//g')\n printf \"%-30s %-15s %-20s\\n\" \"$u\" unknown \"$t\"\n }\n fi\n done;\n echo \"\"\n}\n\n\n### Main Lines ##################################################\nfunction main() {\n BASE=\"/etc/kubeasz\"\n [[ -d \"$BASE\" ]] || { logger error \"invalid dir:$BASE, try: 'ezdown -D'\"; exit 1; }\n cd \"$BASE\"\n\n # check bash shell\n readlink /proc/$$/exe|grep -q \"bash\" || { logger error \"you should use bash shell only\"; exit 1; }\n\n # check 'ansible' executable\n which ansible > /dev/null 2>&1 || { logger error \"need 'ansible', try: 'pip install ansible'\"; usage; exit 1; }\n\n [ \"$#\" -gt 0 ] || { usage >&2; exit 2; }\n\n case \"$1\" in\n ### in-cluster operations #####################\n (add-etcd)\n [ \"$#\" -gt 2 ] || { usage >&2; exit 2; }\n add-etcd \"${@:2}\"\n ;;\n (add-master)\n [ \"$#\" -gt 2 ] || { usage >&2; exit 2; }\n add-master \"${@:2}\"\n ;;\n (add-node)\n [ \"$#\" -gt 2 ] || { usage >&2; exit 2; }\n add-node \"${@:2}\"\n ;;\n (del-etcd)\n [ \"$#\" -eq 3 ] || { usage >&2; exit 2; }\n del-etcd \"$2\" \"$3\"\n ;;\n (del-master)\n [ \"$#\" -eq 3 ] || { usage >&2; exit 2; }\n del-master \"$2\" \"$3\"\n ;;\n (del-node)\n [ \"$#\" -eq 3 ] || { usage >&2; exit 2; }\n del-node \"$2\" \"$3\"\n ;;\n ### cluster-wide operations #######################\n (checkout)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n checkout \"$2\"\n ;;\n (list)\n [ \"$#\" -eq 1 ] || { usage >&2; exit 2; }\n list\n ;;\n (new)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n new \"$2\"\n ;;\n (setup)\n [ \"$#\" -ge 3 ] || { usage-setup >&2; exit 2; }\n setup \"${@:2}\"\n ;;\n (start)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n cmd \"$2\" start\n ;;\n (stop)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n cmd \"$2\" stop\n ;;\n (upgrade)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n cmd \"$2\" upgrade\n ;;\n (backup)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n cmd \"$2\" backup\n ;;\n (restore)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n cmd \"$2\" restore\n ;;\n (destroy)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n cmd \"$2\" destroy\n ;;\n (start-aio)\n [ \"$#\" -eq 1 ] || { usage >&2; exit 2; }\n start-aio\n ;;\n ### extra operations ##############################\n (kca-renew)\n [ \"$#\" -eq 2 ] || { usage >&2; exit 2; }\n renew-ca \"$2\"\n ;;\n (kcfg-adm)\n [ \"$#\" -gt 2 ] || { usage-kcfg-adm >&2; exit 2; }\n kcfg-adm \"${@:2}\"\n ;;\n (help)\n [ \"$#\" -gt 1 ] || { usage >&2; exit 2; }\n help-info \"$2\"\n exit 0\n ;;\n (*)\n usage\n exit 0\n ;;\n esac\n }\n\nmain \"$@\"\n"
},
{
"path": "ezdown",
"content": "#!/bin/bash\n#--------------------------------------------------\n# This script is used for:\n# 1. to download the scripts/binaries/images needed for installing a k8s cluster with kubeasz\n# 2. to run kubeasz in a container (recommended)\n# @author: gjmzj\n# @usage: ./ezdown\n# @repo: https://github.com/easzlab/kubeasz\n#--------------------------------------------------\n# shellcheck disable=SC2155\nset -o nounset\nset -o errexit\nset -o pipefail\n#set -o xtrace\n\n# default settings, can be overridden by cmd line options, see usage\nDOCKER_VER=28.5.2\nKUBEASZ_VER=3.6.8\nK8S_BIN_VER=v1.34.3\n# https://github.com/easzlab/dockerfile-kubeasz-ext-bin\nEXT_BIN_VER=1.13.3\n# https://github.com/easzlab/dockerfile-kubeasz-sys-pkg\nSYS_PKG_VER=1.0.4\nHARBOR_VER=v2.12.4\nREGISTRY_MIRROR=docker.1ms.run\n\n# images downloaded by default(with 'ezdown -D')\n# https://github.com/projectcalico/calico\ncalicoVer=v3.28.4\n# https://github.com/coredns/coredns\ncorednsVer=1.12.4\n# https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/\ndnsNodeCacheVer=1.26.4\n# https://github.com/kubernetes-sigs/metrics-server\nmetricsVer=v0.8.0\npauseVer=3.10\n\n# images not downloaded by default(only download with 'ezdown -X ***')\n# https://github.com/cilium/cilium\n# https://docs.cilium.io/en/stable/installation/k8s-install-helm/\nciliumVer=1.17.4\n# https://github.com/flannel-io/flannel\nflannelVer=v0.27.3\n# https://github.com/cloudnativelabs/kube-router\nkubeRouterVer=v1.5.4\n# https://github.com/kubeovn/kube-ovn\nkubeOvnVer=v1.11.5\n# https://github.com/kubernetes/dashboard\ndashboardVer=7.14.0\n# https://github.com/rancher/local-path-provisioner\nlocalpathProvisionerVer=v0.0.31\n# https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner\nnfsProvisionerVer=v4.0.2\n#https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack\npromChartVer=75.7.0\n#https://kubeblocks.io/docs/release-1_0/user_docs/overview/introduction\nkubeblocksVer=1.0.1\n#https://min.io/docs/minio/kubernetes/upstream/operations/install-deploy-manage/deploy-operator-helm.html\nminioOperatorVer=7.1.1\n# https://openebs.io/docs/quickstart-guide/installation\nopenebsVer=4.3.2\n# https://kubernetes.github.io/ingress-nginx/deploy/\ningressNginxVer=4.13.0\n\nfunction usage() {\n echo -e \"\\033[33mUsage:\\033[0m ezdown [options] [args]\"\n cat < download system packages of the OS (ubuntu_22,debian_11,...)\n -R download Registry(harbor) offline installer\n -S start kubeasz in a container\n -X download extra images\n -d set docker-ce version, default \"$DOCKER_VER\"\n -e set kubeasz-ext-bin version, default \"$EXT_BIN_VER\"\n -k set kubeasz-k8s-bin version, default \"$K8S_BIN_VER\"\n -m set docker registry mirrors, default \"docker.1ms.run\"\n -z set kubeasz version, default \"$KUBEASZ_VER\"\nEOF\n}\n\nfunction usage-down-sys-pkg(){\n echo -e \"\\033[33mUsage:\\033[0m ezdown -P \"\n cat <\"\n cat < /etc/systemd/system/docker.service << EOF\n[Unit]\nDescription=Docker Application Container Engine\n[Service]\nEnvironment=\"PATH=/opt/kube/bin/docker-bin:/bin:/sbin:/usr/bin:/usr/sbin\"\nExecStartPre=/bin/sh -c 'groupadd docker > /dev/null 2>&1 || echo \"\"'\nExecStart=/opt/kube/bin/docker-bin/dockerd\n#ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT\nExecReload=/bin/kill -s HUP \\$MAINPID\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=infinity\nLimitNPROC=infinity\nLimitCORE=infinity\nDelegate=yes\nKillMode=process\n[Install]\nWantedBy=multi-user.target\nEOF\n\n # configuration for dockerd\n mkdir -p /etc/docker\n DOCKER_VER_MAIN=$(echo \"$DOCKER_VER\"|cut -d. -f1)\n CGROUP_DRIVER=\"cgroupfs\"\n ((DOCKER_VER_MAIN>=20)) && CGROUP_DRIVER=\"systemd\"\n logger debug \"generate docker config: /etc/docker/daemon.json\"\n if [ -n \"$REGISTRY_MIRROR\" ];then\n logger debug \"prepare register mirror\"\n cat > /etc/docker/daemon.json << EOF\n{\n \"exec-opts\": [\"native.cgroupdriver=$CGROUP_DRIVER\"],\n \"registry-mirrors\": [\n \"https://docker.1ms.run\",\n \"https://hub1.nat.tf\",\n \"https://docker.1panel.live\",\n \"https://hub.rat.dev\",\n \"https://docker.amingg.com\"\n ],\n \"insecure-registries\": [\"easzlab.io.local:5000\"],\n \"max-concurrent-downloads\": 10,\n \"log-driver\": \"json-file\",\n \"log-level\": \"warn\",\n \"log-opts\": {\n \"max-size\": \"10m\",\n \"max-file\": \"3\"\n },\n \"data-root\": \"/var/lib/docker\"\n}\nEOF\n else\n logger debug \"standard config without registry mirrors\"\n cat > /etc/docker/daemon.json << EOF\n{\n \"exec-opts\": [\"native.cgroupdriver=$CGROUP_DRIVER\"],\n \"insecure-registries\": [\"easzlab.io.local:5000\"],\n \"max-concurrent-downloads\": 10,\n \"log-driver\": \"json-file\",\n \"log-level\": \"warn\",\n \"log-opts\": {\n \"max-size\": \"10m\",\n \"max-file\": \"3\"\n },\n \"data-root\": \"/var/lib/docker\"\n}\nEOF\n fi\n\n if [[ -f /etc/selinux/config ]]; then\n logger debug \"turn off selinux\"\n getenforce|grep Disabled || setenforce 0\n sed -i 's/^SELINUX=.*$/SELINUX=disabled/g' /etc/selinux/config\n fi\n\n logger debug \"enable and start docker\"\n systemctl enable docker\n systemctl daemon-reload && systemctl restart docker && sleep 3\n}\n\nfunction get_kubeasz() {\n # check if kubeasz already existed\n [[ -d \"$BASE/roles/kube-node\" ]] && { logger warn \"kubeasz already existed\"; return 0; }\n\n if [[ ! -f \"$imageDir/kubeasz_$KUBEASZ_VER.tar\" ]];then\n logger info \"downloading kubeasz: $KUBEASZ_VER\"\n docker pull \"easzlab/kubeasz:$KUBEASZ_VER\" && \\\n docker save -o \"$imageDir/kubeasz_$KUBEASZ_VER.tar\" \"easzlab/kubeasz:$KUBEASZ_VER\" || \\\n { logger error \"download failed!\"; return 1; }\n else\n docker load -i \"$imageDir/kubeasz_$KUBEASZ_VER.tar\"\n fi\n\n docker ps -a |grep -q temp_easz && { logger debug \"remove existing container\"; docker rm -f temp_easz; }\n logger debug \" run a temporary container\"\n docker run -d --name temp_easz easzlab/kubeasz:${KUBEASZ_VER} || { logger error \"failed.\"; exit 1; }\n\n [[ -d \"$BASE/down\" ]] && /bin/mv -f \"$BASE/down\" /tmp\n [[ -d \"$BASE/bin\" ]] && /bin/mv -f \"$BASE/bin\" /tmp\n\n rm -rf \"$BASE\" && \\\n logger debug \"cp kubeasz code from the temporary container\" && \\\n docker cp \"temp_easz:$BASE\" \"$BASE\" && \\\n logger debug \"stop&remove temporary container\" && \\\n docker rm -f temp_easz\n\n mkdir -p \"$BASE/bin\" \"$BASE/down\"\n [[ -d \"/tmp/down\" ]] && /bin/mv -f /tmp/down/* \"$BASE/down\"\n [[ -d \"/tmp/bin\" ]] && /bin/mv -f /tmp/bin/* \"$BASE/bin\"\n return 0\n}\n\nfunction get_k8s_bin() {\n [[ -f \"$BASE/bin/kubelet\" ]] && { logger warn \"kubernetes binaries existed\"; return 0; }\n\n logger info \"downloading kubernetes: $K8S_BIN_VER binaries\"\n docker run --rm -v \"$BASE/bin\":/tmp/out easzlab/kubeasz-k8s-bin:\"$K8S_BIN_VER\" \\\n sh -c \"cp -f /k8s/* /tmp/out/\"\n}\n\nfunction get_ext_bin() {\n [[ -f \"$BASE/bin/etcdctl\" ]] && { logger warn \"extra binaries existed\"; return 0; }\n\n logger info \"downloading extral binaries kubeasz-ext-bin:$EXT_BIN_VER\"\n docker run --rm -v \"$BASE/bin\":/tmp/out \"easzlab/kubeasz-ext-bin:$EXT_BIN_VER\" \\\n sh -c \"cp -rf /extra/* /tmp/out/\"\n}\n\nfunction get_sys_pkg() {\n [[ -f \"$BASE/down/packages/$1.tgz\" ]] && { logger warn \"system packages for $1 existed\"; return 0; }\n\n docker run --rm -v \"$BASE/down\":/tmp/out \"easzlab/kubeasz-sys-pkg:$SYS_PKG_VER\" \\\n\t sh -c \"cp -r /packages /tmp/out/\"\n}\n\nfunction get_harbor_offline_pkg() {\n [[ -f \"$BASE/down/harbor-offline-installer-$HARBOR_VER.tgz\" ]] && { logger warn \"harbor-offline existed\"; return 0; }\n\n if [[ \"$ARCH\" == aarch64 ]];then\n docker run --rm -v \"$BASE/down\":/tmp/out \"easzlab/harbor-offline:${HARBOR_VER}-aarch64\" \\\n sh -c \"cp /harbor-offline-installer-$HARBOR_VER.tgz /tmp/out/\"\n else\n docker run --rm -v \"$BASE/down\":/tmp/out \"easzlab/harbor-offline:$HARBOR_VER\" \\\n sh -c \"cp /harbor-offline-installer-$HARBOR_VER.tgz /tmp/out/\"\n fi\n}\n\nfunction get_default_images() {\n logger info \"download default images, then upload to the local registry\"\n\n IMAGES=(\\\n \"calico/cni:$calicoVer\" \\\n \"calico/kube-controllers:$calicoVer\" \\\n \"calico/node:$calicoVer\" \\\n \"coredns/coredns:$corednsVer\" \\\n \"easzlab/k8s-dns-node-cache:$dnsNodeCacheVer\" \\\n \"easzlab/metrics-server:$metricsVer\" \\\n \"easzlab/pause:$pauseVer\" \\\n )\n down_and_save_images\n}\n\nfunction get_extra_images() {\n logger info \"download images for $1, then upload to the local registry\"\n\n case \"$1\" in\n argocd)\n IMAGES=(\\\n \"quay.io/argoproj/argocd:v3.2.5\" \\\n \"ghcr.io/dexidp/dex:v2.44.0\" \\\n \"ecr-public.aws.com/docker/library/redis:8.2.2-alpine\" \\\n )\n down_and_save_images argocd\n ;;\n\n cilium)\n IMAGES=(\\\n \"cilium/cilium:v$ciliumVer\" \\\n \"cilium/operator-generic:v$ciliumVer\" \\\n \"cilium/hubble-relay:v$ciliumVer\" \\\n \"cilium/hubble-ui-backend:v0.13.2\" \\\n \"cilium/hubble-ui:v0.13.2\" \\\n )\n down_and_save_images cilium\n ;;\n\n flannel)\n IMAGES=(\\\n \"ghcr.io/flannel-io/flannel:v0.27.3\" \\\n \"ghcr.io/flannel-io/flannel-cni-plugin:v1.7.1-flannel1\" \\\n )\n down_and_save_images flannel\n ;;\n\n ingress-nginx)\n IMAGES=(\\\n \"easzlab/ingress-nginx-controller:v1.13.0\" \\\n \"easzlab/kube-webhook-certgen:v1.6.0\" \\\n )\n down_and_save_images\n ;;\n\n dashboard)\n IMAGES=(\\\n \"kubernetesui/dashboard-api:1.14.0\" \\\n \"kubernetesui/dashboard-auth:1.4.0\" \\\n \"kubernetesui/dashboard-metrics-scraper:1.2.2\" \\\n \"kubernetesui/dashboard-web:1.7.0\" \\\n \"kong:3.9\" \\\n )\n down_and_save_images kubernetesui\n ;;\n\n kubeblocks)\n IMAGES=(\\\n \"easzlab/snapshot-controller:v8.3.0\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-charts:${kubeblocksVer}\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks:${kubeblocksVer}\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:1.0.0\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:${kubeblocksVer}\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-dataprotection:${kubeblocksVer}\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/datasafed:0.2.1\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-mysql)\n IMAGES=(\\\n \"apecloud/mysql_audit_log:8.0.33\" \\\n \"apecloud/xtrabackup:8.0\" \\\n \"apecloud/jemalloc:5.3.0\" \\\n \"apecloud/syncer:0.5.0\" \\\n \"apecloud/mysql:8.0.39\" \\\n \"apecloud/mysqld-exporter:0.15.1\" \\\n \"apecloud/proxysql:2.4.4\" \\\n \"apecloud/percona-xtrabackup:8.0\" \\\n \"apecloud/wal-g-mysql:2.0.1-1-ubuntu\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-pg)\n IMAGES=(\\\n \"apecloud/spilo:16.4.0\" \\\n\t \"apecloud/spilo-init:0.1\" \\\n \"apecloud/dbctl:0.2.0\" \\\n \"apecloud/pgbouncer:1.19.0\" \\\n \"apecloud/postgres-exporter:v0.15.0\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-redis)\n IMAGES=(\\\n \"apecloud/dbctl:0.1.8\" \\\n \"apecloud/agamotto:0.1.2-beta.1\" \\\n \"apecloud/redis:8.2.1\" \\\n \"apecloud/redis-stack-server:7.2.0-v14\" \\\n \"apecloud/redis-stack-server:7.2.0-v18\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-mongodb)\n IMAGES=(\\\n \"apecloud/syncer:0.3.7\" \\\n \"apecloud/mongo:5.0.30\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-elasticsearch)\n IMAGES=(\\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch-plugins:0.1.0\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch:8.8.2\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch-agent:0.1.0\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/elasticsearch-exporter:v1.7.0\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/curl-jq:0.1.0\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-clickhouse)\n IMAGES=(\\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/busybox:1.36\" \\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/clickhouse:24.8.3-debian-12-r1\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kb-addon-minio)\n IMAGES=(\\\n \"apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/minio:RELEASE.2024-06-29T01-20-47Z\" \\\n )\n down_and_save_images apecloud\n ;;\n\n kube-ovn)\n IMAGES=(\\\n \"kubeovn/kube-ovn:$kubeOvnVer\" \\\n )\n down_and_save_images kubeovn\n ;;\n\n kube-router)\n IMAGES=(\\\n \"cloudnativelabs/kube-router:$kubeRouterVer\" \\\n )\n down_and_save_images cloudnativelabs\n ;;\n\n local-path-provisioner)\n IMAGES=(\\\n \"rancher/local-path-provisioner:$localpathProvisionerVer\" \\\n )\n down_and_save_images rancher\n ;;\n\n minio)\n IMAGES=(\\\n \"quay.io/minio/operator:v${minioOperatorVer}\" \\\n \"quay.io/minio/operator-sidecar:v7.0.1\" \\\n \"quay.io/minio/minio:RELEASE.2025-04-08T15-41-24Z\" \\\n )\n down_and_save_images minio\n ;;\n\n nacos)\n IMAGES=(\\\n \"nacos/nacos-server:v2.4.3\" \\\n \"nacos/nacos-peer-finder-plugin:1.1\" \\\n )\n down_and_save_images nacos\n ;;\n\n network-check)\n IMAGES=(\\\n \"easzlab/json-mock:v1.3.0\" \\\n \"easzlab/alpine-curl:v7.85.0\" \\\n )\n down_and_save_images\n ;;\n\n nfs-provisioner)\n IMAGES=(\\\n \"easzlab/nfs-subdir-external-provisioner:$nfsProvisionerVer\" \\\n )\n down_and_save_images\n ;;\n\n openebs)\n IMAGES=(\\\n \"bitnami/kubectl:1.25.15\" \\\n \"openebs/provisioner-localpv:4.3.0\" \\\n \"openebs/linux-utils:4.2.0\" \\\n \"openebs/lvm-driver:1.7.0\" \\\n \"easzlab/csi-node-driver-registrar:v2.13.0\" \\\n \"easzlab/csi-resizer:v1.11.2\" \\\n \"easzlab/csi-snapshotter:v7.0.0\" \\\n \"easzlab/csi-provisioner:v5.2.0\" \\\n \"easzlab/snapshot-controller:v7.0.0\" \\\n )\n down_and_save_images openebs\n ;;\n\n rocketmq)\n IMAGES=(\\\n \"apache/rocketmq-operator:latest\" \\\n \"apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0\" \\\n \"apacherocketmq/rocketmq-nameserver:4.5.0-alpine-operator-0.3.0\" \\\n \"apacherocketmq/rocketmq-console:2.0.0\" \\\n )\n down_and_save_images rocketmq\n ;;\n\n prometheus)\n IMAGES=(\\\n \"easzlab/kube-state-metrics:v2.16.0\" \\\n \"easzlab/kube-webhook-certgen:v1.6.0\" \\\n )\n down_and_save_images\n IMAGES=(\\\n \"grafana/grafana:12.0.2\" \\\n \"quay.io/kiwigrid/k8s-sidecar:1.30.5\" \\\n \"quay.io/prometheus-operator/prometheus-config-reloader:v0.83.0\" \\\n \"quay.io/prometheus-operator/prometheus-operator:v0.83.0\" \\\n \"quay.io/prometheus/alertmanager:v0.28.1\" \\\n \"quay.io/prometheus/node-exporter:v1.9.1\" \\\n \"quay.io/prometheus/prometheus:v3.4.2\" \\\n )\n down_and_save_images prometheus\n ;;\n\n *)\n logger error \"invalid option: $1\"\n usage-down-ext-img\n exit 1\n ;;\n esac\n}\n\n# 优先下载原始镜像；如果失败，尝试用加速地址下载\nfunction down_and_save_images(){\n if [ \"$#\" -eq 1 ];then\n down_and_save_images_orig $1 || down_and_save_images_with_mirror $1\n else\n down_and_save_images_orig || down_and_save_images_with_mirror\n fi\n}\n\n# 参数扩展说明：\n# ${var%%pattern} - 从**右边**删除**最长匹配**的 pattern 后缀\n# ${var%pattern} - 从**右边**删除**最短匹配**的 pattern 后缀\n# ${var##pattern} - 从**左边**删除**最长匹配**的 pattern 前缀\n# ${var#pattern} - 从**左边**删除**最短匹配**的 pattern 前缀\nfunction down_and_save_images_orig(){\n NS=\"easzlab\"\n [ \"$#\" -eq 1 ] && NS=\"$1\"\n for item in \"${IMAGES[@]}\"; do\n image_part=\"${item##*/}\"\n image_name=\"${image_part%:*}\"\n image_tag=\"${image_part##*:}\"\n image_file=\"$imageDir/${image_name}_${image_tag}.tar\"\n if [[ ! -f \"$image_file\" ]];then\n docker pull \"$item\" && \\\n docker save -o \"$image_file\" \"$item\" || \\\n { logger error \"download $item failed!\"; return 1; }\n else\n docker load -i \"$image_file\"\n fi\n docker tag \"$item\" \"easzlab.io.local:5000/${NS}/${image_part}\"\n docker push \"easzlab.io.local:5000/${NS}/${image_part}\" || \\\n { logger error \"push easzlab.io.local:5000/${NS}/${image_part} failed!\"; return 1; }\n done\n}\n\n# 尝试使用加速地址下载，比如：alpine:latest 替换成 $REGISTRY_MIRROR/library/alpine:latest 下载\nfunction down_and_save_images_with_mirror(){\n [[ \"$REGISTRY_MIRROR\" == \"\" ]] && { logger error \"no registry mirror set\"; return 1; }\n NS=\"easzlab\"\n [ \"$#\" -eq 1 ] && NS=\"$1\"\n for item in \"${IMAGES[@]}\"; do\n image_part=\"${item##*/}\"\n image_name=\"${image_part%:*}\"\n image_tag=\"${image_part##*:}\"\n image_file=\"$imageDir/${image_name}_${image_tag}.tar\"\n\n item=$(normalize_image \"$item\")\n registry=\"${item%%/*}\"\n repository=\"${item#*/}\"\n\n [[ \"$registry\" == \"docker.io\" ]] && item=\"${REGISTRY_MIRROR}/${repository}\"\n\n if [[ ! -f \"$image_file\" ]];then\n docker pull \"$item\" && \\\n docker save -o \"$image_file\" \"$item\" || \\\n { logger error \"download $item failed!\"; return 1; }\n else\n docker load -i \"$image_file\"\n fi\n docker tag \"$item\" \"easzlab.io.local:5000/${NS}/${image_part}\"\n docker push \"easzlab.io.local:5000/${NS}/${image_part}\" || \\\n { logger error \"push easzlab.io.local:5000/${NS}/${image_part} failed!\"; return 1; }\n done\n}\n\n# 将镜像名称转换为标准格式: ${registry}/${repository}:${tag}\n# 标准格式规则：\n# 1. 如果没有 registry，默认使用 docker.io\n# 2. 如果没有 tag，默认使用 latest\n# 3. 如果 repository 不包含 /，且 registry 是 docker.io，则添加 library/ 前缀\nfunction normalize_image() {\n local image=\"$1\"\n local registry=\"\"\n local repository=\"\"\n local tag=\"\"\n\n # 提取 tag（如果存在）\n if [[ \"$image\" == *\":\"* ]]; then\n tag=\"${image##*:}\"\n image=\"${image%:*}\"\n else\n tag=\"latest\"\n fi\n\n # 提取 registry 和 repository\n # 判断是否包含 registry（包含域名特征：包含点号或端口号）\n if [[ \"$image\" == *\".\"* ]] || [[ \"$image\" == *\":\"* ]]; then\n # 包含 registry\n registry=\"${image%%/*}\"\n repository=\"${image#*/}\"\n else\n # 不包含 registry，使用默认的 docker.io\n registry=\"docker.io\"\n repository=\"$image\"\n fi\n\n # 如果 repository 不包含 /，且 registry 是 docker.io，则添加 library/ 前缀\n if [[ \"$registry\" == \"docker.io\" ]] && [[ \"$repository\" != *\"/\"* ]]; then\n repository=\"library/$repository\"\n fi\n\n # 输出标准格式\n echo \"${registry}/${repository}:${tag}\"\n}\n\n\nfunction download_all() {\n mkdir -p /opt/kube/bin \"$BASE/down\" \"$BASE/bin\"\n download_docker && \\\n install_docker && \\\n get_kubeasz && \\\n get_k8s_bin && \\\n get_ext_bin && \\\n start_local_registry && \\\n get_default_images\n}\n\nfunction start_local_registry() {\n if [[ ! -f \"$imageDir/registry-2.tar\" ]];then\n docker pull \"registry:2\" && \\\n docker save -o \"$imageDir/registry-2.tar\" \"registry:2\"\n fi\n\n docker ps -a --format=\"{{ .Names }}\"|grep local_registry > /dev/null 2>&1 && \\\n { logger warn \"local_registry is already running\"; return 0; }\n\n logger info \"start local registry ...\"\n docker load -i \"$imageDir/registry-2.tar\" > /dev/null\n mkdir -p /opt/kube/registry\n docker run -d \\\n --name local_registry \\\n --network host \\\n --restart always \\\n --volume /opt/kube/registry:/var/lib/registry \\\n registry:2\n\n sed -i \"/easzlab.io.local/d\" /etc/hosts\n echo \"127.0.0.1 easzlab.io.local\" >> /etc/hosts\n}\n\n\nfunction start_kubeasz_docker() {\n # create cmd alias in /root/.bashrc\n sed -i '/docker exec/d' /root/.bashrc\n echo \"alias dk='docker exec -it kubeasz' # generated by kubeasz\" >> /root/.bashrc\n\n [[ -d \"$BASE/roles/kube-node\" ]] || { logger error \"not initialized. try 'ezdown -D' first.\"; exit 1; }\n docker ps -a --format=\"{{ .Names }}\"|grep kubeasz > /dev/null 2>&1 && \\\n docker rm -f kubeasz > /dev/null\n\n if [[ ! -f \"$imageDir/kubeasz_$KUBEASZ_VER.tar\" ]];then\n logger info \"downloading kubeasz: $KUBEASZ_VER\"\n docker pull \"easzlab/kubeasz:$KUBEASZ_VER\" && \\\n docker save -o \"$imageDir/kubeasz_$KUBEASZ_VER.tar\" \"easzlab/kubeasz:$KUBEASZ_VER\"\n else\n docker load -i \"$imageDir/kubeasz_$KUBEASZ_VER.tar\"\n fi\n\n logger info \"try to run kubeasz in a container\"\n # get host's IP\n host_if=$(ip route|grep default|head -n1|cut -d' ' -f5)\n host_ip=$(ip a|grep \"$host_if$\"|head -n1|awk '{print $2}'|cut -d'/' -f1)\n logger debug \"get host IP: $host_ip\"\n\n # allow ssh login using key locally\n if [[ ! -e /root/.ssh/id_rsa ]]; then\n logger debug \"generate ssh key pair\"\n ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa > /dev/null\n cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys\n ssh-keyscan -t ecdsa -H \"$host_ip\" >> /root/.ssh/known_hosts\n fi\n\n # run kubeasz docker container\n docker run --detach \\\n --env HOST_IP=\"$host_ip\" \\\n --name kubeasz \\\n --network host \\\n --restart always \\\n --volume \"$BASE\":\"$BASE\" \\\n --volume /root/.kube:/root/.kube \\\n --volume /root/.ssh:/root/.ssh \\\n --volume /etc/docker:/etc/docker \\\n easzlab/kubeasz:${KUBEASZ_VER}\n}\n\n\n### Main Lines ##################################################\nfunction main() {\n BASE=\"/etc/kubeasz\"\n IMAGES=()\n imageDir=\"$BASE/down\"\n\n # check if use bash shell\n # readlink /proc/$$/exe|grep -q \"bash\" || { logger error \"you should use bash shell, not sh\"; exit 1; }\n # check if use with root\n # [[ \"$EUID\" -ne 0 ]] && { logger error \"you should run this script as root\"; exit 1; }\n\n # get architecture\n ARCH=$(uname -m)\n\n [[ \"$#\" -eq 0 ]] && { usage >&2; exit 1; }\n\n ACTION=\"\"\n while getopts \"CDP:RSX:d:e:k:m:z:\" OPTION; do\n case \"$OPTION\" in\n D)\n ACTION=\"download_all\"\n ;;\n P)\n [[ $OPTARG =~ (ubuntu_[0-9]+|centos_[0-9]+|debian_[0-9]+|fedora_[0-9]+|almalinux_[0-9]+|opensuse_leap_[0-9]+|rocky_[0-9]+) ]] || \\\n { usage-down-sys-pkg; exit 1; }\n SYS_PKG_VER=\"${SYS_PKG_VER}_$OPTARG\"\n ACTION=\"get_sys_pkg $OPTARG\"\n ;;\n R)\n ACTION=\"get_harbor_offline_pkg\"\n ;;\n S)\n ACTION=\"start_kubeasz_docker\"\n ;;\n X)\n ACTION=\"get_extra_images $OPTARG\"\n ;;\n d)\n DOCKER_VER=\"$OPTARG\"\n ;;\n e)\n EXT_BIN_VER=\"$OPTARG\"\n ;;\n k)\n K8S_BIN_VER=\"$OPTARG\"\n ;;\n m)\n REGISTRY_MIRROR=\"$OPTARG\"\n ;;\n z)\n KUBEASZ_VER=\"$OPTARG\"\n ;;\n ?)\n usage\n exit 1\n ;;\n esac\n done\n\n [[ \"$ACTION\" == \"\" ]] && { logger error \"illegal option\"; usage; exit 1; }\n\n # excute cmd \"$ACTION\"\n logger info \"Action begin: $ACTION\"\n ${ACTION} || { logger error \"Action failed: $ACTION\"; return 1; }\n logger info \"Action successed: $ACTION\"\n}\n\nmain \"$@\"\n"
},
{
"path": "manifests/deprecated/efk/es-dynamic-pv/es-statefulset.yaml",
"content": "# RBAC authn and authz\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: elasticsearch-logging\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nrules:\n- apiGroups:\n - \"\"\n resources:\n - \"services\"\n - \"namespaces\"\n - \"endpoints\"\n verbs:\n - \"get\"\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n namespace: kube-system\n name: elasticsearch-logging\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nsubjects:\n- kind: ServiceAccount\n name: elasticsearch-logging\n namespace: kube-system\n apiGroup: \"\"\nroleRef:\n kind: ClusterRole\n name: elasticsearch-logging\n apiGroup: \"\"\n---\n# Elasticsearch deployment itself\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nspec:\n serviceName: elasticsearch-logging\n replicas: 2\n selector:\n matchLabels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n template:\n metadata:\n labels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n kubernetes.io/cluster-service: \"true\"\n spec:\n serviceAccountName: elasticsearch-logging\n containers:\n #- image: gcr.io/fluentd-elasticsearch/elasticsearch:v6.6.1\n - image: easzlab/elasticsearch:v6.6.1\n name: elasticsearch-logging\n resources:\n # need more cpu upon initialization, therefore burstable class\n limits:\n cpu: 1000m\n requests:\n cpu: 100m\n ports:\n - containerPort: 9200\n name: db\n protocol: TCP\n - containerPort: 9300\n name: transport\n protocol: TCP\n volumeMounts:\n - name: elasticsearch-logging\n mountPath: /data\n env:\n - name: \"NAMESPACE\"\n valueFrom:\n fieldRef:\n fieldPath: metadata.namespace\n # Elasticsearch requires vm.max_map_count to be at least 262144.\n # If your OS already sets up this number to a higher value, feel free\n # to remove this init container.\n initContainers:\n - image: alpine:3.6\n command: [\"/sbin/sysctl\", \"-w\", \"vm.max_map_count=262144\"]\n name: elasticsearch-logging-init\n securityContext:\n privileged: true\n volumeClaimTemplates:\n - metadata:\n name: elasticsearch-logging\n spec:\n accessModes: [ \"ReadWriteMany\" ]\n storageClassName: \"nfs-dynamic-class\"\n resources:\n requests:\n storage: 4Gi\n"
},
{
"path": "manifests/deprecated/efk/es-index-rotator/rotator.yaml",
"content": "apiVersion: batch/v1beta1\nkind: CronJob\nmetadata:\n name: es-index-rotator\n namespace: kube-system\nspec:\n # 每天1点3分执行\n schedule: \"3 1 */1 * *\"\n jobTemplate:\n spec:\n template:\n spec:\n containers:\n - name: es-index-rotator\n image: easzlab/es-index-rotator:0.2.1\n # 保留最近10天日志\n command:\n - /bin/rotate.sh\n - \"10\"\n - \"logstash\" # fluented 默认创建的index形如'logstash-2020.01.01'\n restartPolicy: OnFailure\n concurrencyPolicy: Forbid\n successfulJobsHistoryLimit: 2\n failedJobsHistoryLimit: 1\n"
},
{
"path": "manifests/deprecated/efk/es-service.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\n kubernetes.io/name: \"Elasticsearch\"\nspec:\n ports:\n - port: 9200\n protocol: TCP\n targetPort: db\n clusterIP: None\n selector:\n k8s-app: elasticsearch-logging\n"
},
{
"path": "manifests/deprecated/efk/es-static-pv/es-pv0.yaml",
"content": "apiVersion: v1\nkind: PersistentVolume\nmetadata:\n name: pv-es-0\nspec:\n capacity:\n storage: 4Gi\n accessModes:\n - ReadWriteMany\n volumeMode: Filesystem\n persistentVolumeReclaimPolicy: Recycle\n storageClassName: \"es-storage-class\"\n nfs:\n # 根据实际共享目录修改\n path: /share/es0\n # 根据实际 nfs服务器地址修改\n server: 192.168.1.208\n"
},
{
"path": "manifests/deprecated/efk/es-static-pv/es-pv1.yaml",
"content": "apiVersion: v1\nkind: PersistentVolume\nmetadata:\n name: pv-es-1\nspec:\n capacity:\n storage: 4Gi\n accessModes:\n - ReadWriteMany\n volumeMode: Filesystem\n persistentVolumeReclaimPolicy: Recycle\n storageClassName: \"es-storage-class\"\n nfs:\n # 根据实际共享目录修改\n path: /share/es1\n # 根据实际 nfs服务器地址修改\n server: 192.168.1.208\n"
},
{
"path": "manifests/deprecated/efk/es-static-pv/es-pv2.yaml",
"content": "apiVersion: v1\nkind: PersistentVolume\nmetadata:\n name: pv-es-2\nspec:\n capacity:\n storage: 4Gi\n accessModes:\n - ReadWriteMany\n volumeMode: Filesystem\n persistentVolumeReclaimPolicy: Recycle\n storageClassName: \"es-storage-class\"\n nfs:\n # 根据实际共享目录修改\n path: /share/es2\n # 根据实际 nfs服务器地址修改\n server: 192.168.1.208\n"
},
{
"path": "manifests/deprecated/efk/es-static-pv/es-statefulset.yaml",
"content": "# RBAC authn and authz\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: elasticsearch-logging\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nrules:\n- apiGroups:\n - \"\"\n resources:\n - \"services\"\n - \"namespaces\"\n - \"endpoints\"\n verbs:\n - \"get\"\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n namespace: kube-system\n name: elasticsearch-logging\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nsubjects:\n- kind: ServiceAccount\n name: elasticsearch-logging\n namespace: kube-system\n apiGroup: \"\"\nroleRef:\n kind: ClusterRole\n name: elasticsearch-logging\n apiGroup: \"\"\n---\n# Elasticsearch deployment itself\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nspec:\n serviceName: elasticsearch-logging\n replicas: 2\n selector:\n matchLabels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n template:\n metadata:\n labels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n kubernetes.io/cluster-service: \"true\"\n spec:\n serviceAccountName: elasticsearch-logging\n containers:\n #- image: gcr.io/fluentd-elasticsearch/elasticsearch:v6.6.1\n - image: easzlab/elasticsearch:v6.6.1\n name: elasticsearch-logging\n resources:\n # need more cpu upon initialization, therefore burstable class\n limits:\n cpu: 1000m\n requests:\n cpu: 100m\n ports:\n - containerPort: 9200\n name: db\n protocol: TCP\n - containerPort: 9300\n name: transport\n protocol: TCP\n volumeMounts:\n - name: elasticsearch-logging\n mountPath: /data\n env:\n - name: \"NAMESPACE\"\n valueFrom:\n fieldRef:\n fieldPath: metadata.namespace\n # Elasticsearch requires vm.max_map_count to be at least 262144.\n # If your OS already sets up this number to a higher value, feel free\n # to remove this init container.\n initContainers:\n - image: alpine:3.6\n command: [\"/sbin/sysctl\", \"-w\", \"vm.max_map_count=262144\"]\n name: elasticsearch-logging-init\n securityContext:\n privileged: true\n volumeClaimTemplates:\n - metadata:\n name: elasticsearch-logging\n spec:\n accessModes: [ \"ReadWriteMany\" ]\n storageClassName: \"es-storage-class\"\n resources:\n requests:\n storage: 4Gi\n"
},
{
"path": "manifests/deprecated/efk/es-without-pv/es-statefulset.yaml",
"content": "# RBAC authn and authz\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: elasticsearch-logging\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nrules:\n- apiGroups:\n - \"\"\n resources:\n - \"services\"\n - \"namespaces\"\n - \"endpoints\"\n verbs:\n - \"get\"\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n namespace: kube-system\n name: elasticsearch-logging\n labels:\n k8s-app: elasticsearch-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nsubjects:\n- kind: ServiceAccount\n name: elasticsearch-logging\n namespace: kube-system\n apiGroup: \"\"\nroleRef:\n kind: ClusterRole\n name: elasticsearch-logging\n apiGroup: \"\"\n---\n# Elasticsearch deployment itself\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: elasticsearch-logging\n namespace: kube-system\n labels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nspec:\n serviceName: elasticsearch-logging\n replicas: 2\n selector:\n matchLabels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n template:\n metadata:\n labels:\n k8s-app: elasticsearch-logging\n version: v6.6.1\n kubernetes.io/cluster-service: \"true\"\n spec:\n serviceAccountName: elasticsearch-logging\n containers:\n #- image: gcr.io/fluentd-elasticsearch/elasticsearch:v6.6.1\n - image: easzlab/elasticsearch:v6.6.1\n name: elasticsearch-logging\n resources:\n # need more cpu upon initialization, therefore burstable class\n limits:\n cpu: 1000m\n requests:\n cpu: 100m\n ports:\n - containerPort: 9200\n name: db\n protocol: TCP\n - containerPort: 9300\n name: transport\n protocol: TCP\n volumeMounts:\n - name: elasticsearch-logging\n mountPath: /data\n env:\n - name: \"NAMESPACE\"\n valueFrom:\n fieldRef:\n fieldPath: metadata.namespace\n volumes:\n - name: elasticsearch-logging\n emptyDir: {}\n # Elasticsearch requires vm.max_map_count to be at least 262144.\n # If your OS already sets up this number to a higher value, feel free\n # to remove this init container.\n initContainers:\n - image: alpine:3.6\n command: [\"/sbin/sysctl\", \"-w\", \"vm.max_map_count=262144\"]\n name: elasticsearch-logging-init\n securityContext:\n privileged: true\n"
},
{
"path": "manifests/deprecated/efk/fluentd-es-configmap.yaml",
"content": "kind: ConfigMap\napiVersion: v1\nmetadata:\n name: fluentd-es-config-v0.2.0\n namespace: kube-system\n labels:\n addonmanager.kubernetes.io/mode: Reconcile\ndata:\n system.conf: |-\n \n root_dir /tmp/fluentd-buffers/\n \n\n containers.input.conf: |-\n # This configuration file for Fluentd / td-agent is used\n # to watch changes to Docker log files. The kubelet creates symlinks that\n # capture the pod name, namespace, container name & Docker container ID\n # to the docker logs for pods in the /var/log/containers directory on the host.\n # If running this fluentd configuration in a Docker container, the /var/log\n # directory should be mounted in the container.\n #\n # These logs are then submitted to Elasticsearch which assumes the\n # installation of the fluent-plugin-elasticsearch & the\n # fluent-plugin-kubernetes_metadata_filter plugins.\n # See https://github.com/uken/fluent-plugin-elasticsearch &\n # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for\n # more information about the plugins.\n #\n # Example\n # =======\n # A line in the Docker log file might look like this JSON:\n #\n # {\"log\":\"2014/09/25 21:15:03 Got request with path wombat\\n\",\n # \"stream\":\"stderr\",\n # \"time\":\"2014-09-25T21:15:03.499185026Z\"}\n #\n # The time_format specification below makes sure we properly\n # parse the time format produced by Docker. This will be\n # submitted to Elasticsearch and should appear like:\n # $ curl 'http://elasticsearch-logging:9200/_search?pretty'\n # ...\n # {\n # \"_index\" : \"logstash-2014.09.25\",\n # \"_type\" : \"fluentd\",\n # \"_id\" : \"VBrbor2QTuGpsQyTCdfzqA\",\n # \"_score\" : 1.0,\n # \"_source\":{\"log\":\"2014/09/25 22:45:50 Got request with path wombat\\n\",\n # \"stream\":\"stderr\",\"tag\":\"docker.container.all\",\n # \"@timestamp\":\"2014-09-25T22:45:50+00:00\"}\n # },\n # ...\n #\n # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log\n # record & add labels to the log record if properly configured. This enables users\n # to filter & search logs on any metadata.\n # For example a Docker container's logs might be in the directory:\n #\n # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b\n #\n # and in the file:\n #\n # 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log\n #\n # where 997599971ee6... is the Docker ID of the running container.\n # The Kubernetes kubelet makes a symbolic link to this file on the host machine\n # in the /var/log/containers directory which includes the pod name and the Kubernetes\n # container name:\n #\n # synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log\n # ->\n # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log\n #\n # The /var/log directory on the host is mapped to the /var/log directory in the container\n # running this instance of Fluentd and we end up collecting the file:\n #\n # /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log\n #\n # This results in the tag:\n #\n # var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log\n #\n # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name\n # which are added to the log message as a kubernetes field object & the Docker container ID\n # is also added under the docker field object.\n # The final tag is:\n #\n # kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log\n #\n # And the final log record look like:\n #\n # {\n # \"log\":\"2014/09/25 21:15:03 Got request with path wombat\\n\",\n # \"stream\":\"stderr\",\n # \"time\":\"2014-09-25T21:15:03.499185026Z\",\n # \"kubernetes\": {\n # \"namespace\": \"default\",\n # \"pod_name\": \"synthetic-logger-0.25lps-pod\",\n # \"container_name\": \"synth-lgr\"\n # },\n # \"docker\": {\n # \"container_id\": \"997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b\"\n # }\n # }\n #\n # This makes it easier for users to search for logs by pod name or by\n # the name of the Kubernetes container regardless of how many times the\n # Kubernetes pod has been restarted (resulting in a several Docker container IDs).\n\n # Json Log Example:\n # {\"log\":\"[info:2016-02-16T16:04:05.930-08:00] Some log text here\\n\",\"stream\":\"stdout\",\"time\":\"2016-02-17T00:04:05.931087621Z\"}\n # CRI Log Example:\n # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here\n \n @id fluentd-containers.log\n @type tail\n path /var/log/containers/*.log\n pos_file /var/log/es-containers.log.pos\n tag raw.kubernetes.*\n read_from_head true\n \n @type multi_format\n \n format json\n time_key time\n time_format %Y-%m-%dT%H:%M:%S.%NZ\n \n \n format /^(?.+) (?stdout|stderr) [^ ]* (?.*)$/\n time_format %Y-%m-%dT%H:%M:%S.%N%:z\n \n \n \n\n # Detect exceptions in the log output and forward them as one log entry.\n \n @id raw.kubernetes\n @type detect_exceptions\n remove_tag_prefix raw\n message log\n stream stream\n multiline_flush_interval 5\n max_bytes 500000\n max_lines 1000\n \n\n # Concatenate multi-line logs\n \n @id filter_concat\n @type concat\n key log\n use_first_timestamp true\n multiline_end_regexp /\\n$/\n separator \"\"\n \n\n # Enriches records with Kubernetes metadata\n \n @id filter_kubernetes_metadata\n @type kubernetes_metadata\n \n\n # Fixes json fields in Elasticsearch\n \n @id filter_parser\n @type parser\n key_name log\n reserve_data true\n remove_key_name_field true\n \n @type multi_format\n \n format json\n \n \n format none\n \n \n \n\n system.input.conf: |-\n\n # Logs from systemd-journal for interesting services.\n # TODO(random-liu): Remove this after cri container runtime rolls out.\n \n @id journald-docker\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"docker.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-docker.pos\n \n read_from_head true\n tag docker\n \n\n \n @id journald-container-runtime\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"{{ fluentd_container_runtime_service }}.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-container-runtime.pos\n \n read_from_head true\n tag container-runtime\n \n\n \n @id journald-etcd\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"etcd.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-etcd.pos\n \n read_from_head true\n tag etcd\n \n\n \n @id journald-kube-apiserver\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"kube-apiserver.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-kube-apiserver.pos\n \n read_from_head true\n tag kube-apiserver\n \n\n \n @id journald-kube-controller-manager\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"kube-controller-manager.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-kube-controller-manager.pos\n \n read_from_head true\n tag kube-controller-manager\n \n\n \n @id journald-kube-scheduler\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"kube-scheduler.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-kube-scheduler.pos\n \n read_from_head true\n tag kube-scheduler\n \n\n \n @id journald-kubelet\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"kubelet.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-kubelet.pos\n \n read_from_head true\n tag kubelet\n \n\n \n @id journald-kube-proxy\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"kube-proxy.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-kube-proxy.pos\n \n read_from_head true\n tag kube-proxy\n \n\n \n @id journald-node-problem-detector\n @type systemd\n matches [{ \"_SYSTEMD_UNIT\": \"node-problem-detector.service\" }]\n \n @type local\n persistent true\n path /var/log/journald-node-problem-detector.pos\n \n read_from_head true\n tag node-problem-detector\n \n\n \n @id kernel\n @type systemd\n matches [{ \"_TRANSPORT\": \"kernel\" }]\n \n @type local\n persistent true\n path /var/log/kernel.pos\n \n \n fields_strip_underscores true\n fields_lowercase true\n \n read_from_head true\n tag kernel\n \n\n forward.input.conf: |-\n # Takes the messages sent over TCP\n \n @id forward\n @type forward\n \n\n monitoring.conf: |-\n # Prometheus Exporter Plugin\n # input plugin that exports metrics\n \n @id prometheus\n @type prometheus\n \n\n \n @id monitor_agent\n @type monitor_agent\n \n\n # input plugin that collects metrics from MonitorAgent\n \n @id prometheus_monitor\n @type prometheus_monitor\n \n host ${hostname}\n \n \n\n # input plugin that collects metrics for output plugin\n \n @id prometheus_output_monitor\n @type prometheus_output_monitor\n \n host ${hostname}\n \n \n\n # input plugin that collects metrics for in_tail plugin\n \n @id prometheus_tail_monitor\n @type prometheus_tail_monitor\n \n host ${hostname}\n \n \n\n output.conf: |-\n \n @id elasticsearch\n @type elasticsearch\n @log_level info\n type_name _doc\n include_tag_key true\n host elasticsearch-logging\n port 9200\n logstash_format true\n \n @type file\n path /var/log/fluentd-buffers/kubernetes.system.buffer\n flush_mode interval\n retry_type exponential_backoff\n flush_thread_count 2\n flush_interval 5s\n retry_forever\n retry_max_interval 30\n chunk_limit_size 2M\n queue_limit_length 8\n overflow_action block\n \n \n"
},
{
"path": "manifests/deprecated/efk/fluentd-es-ds.yaml",
"content": "apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: fluentd-es\n namespace: kube-system\n labels:\n k8s-app: fluentd-es\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: fluentd-es\n labels:\n k8s-app: fluentd-es\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nrules:\n- apiGroups:\n - \"\"\n resources:\n - \"namespaces\"\n - \"pods\"\n verbs:\n - \"get\"\n - \"watch\"\n - \"list\"\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: fluentd-es\n labels:\n k8s-app: fluentd-es\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nsubjects:\n- kind: ServiceAccount\n name: fluentd-es\n namespace: kube-system\n apiGroup: \"\"\nroleRef:\n kind: ClusterRole\n name: fluentd-es\n apiGroup: \"\"\n---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: fluentd-es-v2.4.0\n namespace: kube-system\n labels:\n k8s-app: fluentd-es\n version: v2.4.0\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nspec:\n selector:\n matchLabels:\n k8s-app: fluentd-es\n version: v2.4.0\n template:\n metadata:\n labels:\n k8s-app: fluentd-es\n kubernetes.io/cluster-service: \"true\"\n version: v2.4.0\n # This annotation ensures that fluentd does not get evicted if the node\n # supports critical pod annotation based priority scheme.\n # Note that this does not guarantee admission on the nodes (#40573).\n annotations:\n seccomp.security.alpha.kubernetes.io/pod: 'docker/default'\n spec:\n priorityClassName: system-node-critical\n serviceAccountName: fluentd-es\n containers:\n - name: fluentd-es\n #image: k8s.gcr.io/fluentd-elasticsearch:v2.4.0\n image: mirrorgooglecontainers/fluentd-elasticsearch:v2.4.0\n env:\n - name: FLUENTD_ARGS\n value: --no-supervisor -q\n resources:\n limits:\n memory: 500Mi\n requests:\n cpu: 100m\n memory: 200Mi\n volumeMounts:\n - name: varlog\n mountPath: /var/log\n - name: varlibdockercontainers\n mountPath: /var/lib/docker/containers\n readOnly: true\n - name: config-volume\n mountPath: /etc/fluent/config.d\n #nodeSelector:\n #beta.kubernetes.io/fluentd-ds-ready: \"true\"\n terminationGracePeriodSeconds: 30\n volumes:\n - name: varlog\n hostPath:\n path: /var/log\n - name: varlibdockercontainers\n hostPath:\n path: /var/lib/docker/containers\n - name: config-volume\n configMap:\n name: fluentd-es-config-v0.2.0\n"
},
{
"path": "manifests/deprecated/efk/kibana-deployment.yaml",
"content": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: kibana-logging\n namespace: kube-system\n labels:\n k8s-app: kibana-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\nspec:\n replicas: 1\n selector:\n matchLabels:\n k8s-app: kibana-logging\n template:\n metadata:\n labels:\n k8s-app: kibana-logging\n annotations:\n seccomp.security.alpha.kubernetes.io/pod: 'docker/default'\n spec:\n containers:\n - name: kibana-logging\n #image: docker.elastic.co/kibana/kibana-oss:6.6.1\n image: easzlab/kibana-oss:6.6.1\n resources:\n # need more cpu upon initialization, therefore burstable class\n limits:\n cpu: 1000m\n requests:\n cpu: 100m\n env:\n - name: ELASTICSEARCH_URL\n value: http://elasticsearch-logging:9200\n # if kibana service is exposed by nodePort, use lines commited out instead\n #- name: SERVER_BASEPATH\n # value: \"\"\n - name: SERVER_BASEPATH\n value: /api/v1/namespaces/kube-system/services/kibana-logging/proxy\n ports:\n - containerPort: 5601\n name: ui\n protocol: TCP\n"
},
{
"path": "manifests/deprecated/efk/kibana-service.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: kibana-logging\n namespace: kube-system\n labels:\n k8s-app: kibana-logging\n kubernetes.io/cluster-service: \"true\"\n addonmanager.kubernetes.io/mode: Reconcile\n kubernetes.io/name: \"Kibana\"\nspec:\n ports:\n - port: 5601\n protocol: TCP\n targetPort: ui\n selector:\n k8s-app: kibana-logging\n #type: NodePort\n"
},
{
"path": "manifests/deprecated/efk/log-pilot/log-pilot-filebeat.yaml",
"content": "apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: log-pilot\n labels:\n app: log-pilot\n namespace: kube-system\nspec:\n selector:\n matchLabels:\n app: log-pilot\n updateStrategy:\n type: RollingUpdate\n template:\n metadata:\n labels:\n app: log-pilot\n spec:\n # 是否允许部署到Master节点上\n #tolerations:\n #- key: node-role.kubernetes.io/master\n # effect: NoSchedule\n # priorityClassName: system-cluster-critical\n containers:\n - name: log-pilot\n # 版本请参考https://github.com/AliyunContainerService/log-pilot/releases\n image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.7-filebeat\n resources:\n limits:\n memory: 500Mi\n requests:\n cpu: 200m\n memory: 200Mi\n env:\n - name: \"NODE_NAME\"\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n - name: \"LOGGING_OUTPUT\"\n value: \"elasticsearch\"\n # 请确保集群到ES网络可达\n - name: \"ELASTICSEARCH_HOSTS\"\n value: \"elasticsearch-logging:9200\"\n # 配置ES访问权限\n - name: \"ELASTICSEARCH_USER\"\n value: \"\"\n - name: \"ELASTICSEARCH_PASSWORD\"\n value: \"\"\n volumeMounts:\n - name: sock\n mountPath: /var/run/docker.sock\n - name: root\n mountPath: /host\n readOnly: true\n - name: varlib\n mountPath: /var/lib/filebeat\n - name: varlog\n mountPath: /var/log/filebeat\n - name: localtime\n mountPath: /etc/localtime\n readOnly: true\n livenessProbe:\n failureThreshold: 3\n exec:\n command:\n - /pilot/healthz\n initialDelaySeconds: 10\n periodSeconds: 10\n successThreshold: 1\n timeoutSeconds: 2\n securityContext:\n capabilities:\n add:\n - SYS_ADMIN\n terminationGracePeriodSeconds: 30\n imagePullSecrets:\n - name: ydy-test-key\n volumes:\n - name: sock\n hostPath:\n path: /var/run/docker.sock\n - name: root\n hostPath:\n path: /\n - name: varlib\n hostPath:\n path: /var/lib/filebeat\n type: DirectoryOrCreate\n - name: varlog\n hostPath:\n path: /var/log/filebeat\n type: DirectoryOrCreate\n - name: localtime\n hostPath:\n path: /etc/localtime\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/.helmignore",
"content": ".git\n# OWNERS file for Kubernetes\nOWNERS"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/Chart.yaml",
"content": "name: elasticsearch\nhome: https://www.elastic.co/products/elasticsearch\nversion: 1.7.2\nappVersion: 6.4.0\ndescription: Flexible and powerful open source, distributed real-time search and analytics\n engine.\nicon: https://static-www.elastic.co/assets/blteb1c97719574938d/logo-elastic-elasticsearch-lt.svg\nsources:\n- https://www.elastic.co/products/elasticsearch\n- https://github.com/jetstack/elasticsearch-pet\n- https://github.com/giantswarm/kubernetes-elastic-stack\n- https://github.com/GoogleCloudPlatform/elasticsearch-docker\n- https://github.com/clockworksoul/helm-elasticsearch\n- https://github.com/pires/kubernetes-elasticsearch-cluster\nmaintainers:\n- name: simonswine\n email: christian@jetstack.io\n- name: icereval\n email: michael.haselton@gmail.com\n- name: rendhalver\n email: pete.brown@powerhrg.com\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/OWNERS",
"content": "approvers:\n- simonswine\n- icereval\n- rendhalver\nreviewers:\n- simonswine\n- icereval\n- rendhalver\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/README.md",
"content": "# Elasticsearch Helm Chart\n\nThis chart uses a standard Docker image of Elasticsearch (docker.elastic.co/elasticsearch/elasticsearch-oss) and uses a service pointing to the master's transport port for service discovery.\nElasticsearch does not communicate with the Kubernetes API, hence no need for RBAC permissions.\n\n## Warning for previous users\nIf you are currently using an earlier version of this Chart you will need to redeploy your Elasticsearch clusters. The discovery method used here is incompatible with using RBAC.\nIf you are upgrading to Elasticsearch 6 from the 5.5 version used in this chart before, please note that your cluster needs to do a full cluster restart.\nThe simplest way to do that is to delete the installation (keep the PVs) and install this chart again with the new version.\nIf you want to avoid doing that upgrade to Elasticsearch 5.6 first before moving on to Elasticsearch 6.0.\n\n## Prerequisites Details\n\n* Kubernetes 1.6+\n* PV dynamic provisioning support on the underlying infrastructure\n\n## StatefulSets Details\n* https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/\n\n## StatefulSets Caveats\n* https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations\n\n## Todo\n\n* Implement TLS/Auth/Security\n* Smarter upscaling/downscaling\n* Solution for memory locking\n\n## Chart Details\nThis chart will do the following:\n\n* Implemented a dynamically scalable elasticsearch cluster using Kubernetes StatefulSets/Deployments\n* Multi-role deployment: master, client (coordinating) and data nodes\n* Statefulset Supports scaling down without degrading the cluster\n\n## Installing the Chart\n\nTo install the chart with the release name `my-release`:\n\n```bash\n$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator\n$ helm install --name my-release incubator/elasticsearch\n```\n\n## Deleting the Charts\n\nDelete the Helm deployment as normal\n\n```\n$ helm delete my-release\n```\n\nDeletion of the StatefulSet doesn't cascade to deleting associated PVCs. To delete them:\n\n```\n$ kubectl delete pvc -l release=my-release,component=data\n```\n\n## Configuration\n\nThe following table lists the configurable parameters of the elasticsearch chart and their default values.\n\n| Parameter | Description | Default |\n| ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------ |\n| `appVersion` | Application Version (Elasticsearch) | `6.4.0` |\n| `image.repository` | Container image name | `docker.elastic.co/elasticsearch/elasticsearch-oss` |\n| `image.tag` | Container image tag | `6.4.0` |\n| `image.pullPolicy` | Container pull policy | `Always` |\n| `cluster.name` | Cluster name | `elasticsearch` |\n| `cluster.xpackEnable` | Writes the X-Pack configuration options to the configuration file | `false` |\n| `cluster.config` | Additional cluster config appended | `{}` |\n| `cluster.keystoreSecret` | Name of secret holding secure config options in an es keystore | `nil` |\n| `cluster.env` | Cluster environment variables | `{MINIMUM_MASTER_NODES: \"2\"}` |\n| `client.name` | Client component name | `client` |\n| `client.replicas` | Client node replicas (deployment) | `2` |\n| `client.resources` | Client node resources requests & limits | `{} - cpu limit must be an integer` |\n| `client.priorityClassName` | Client priorityClass | `nil` |\n| `client.heapSize` | Client node heap size | `512m` |\n| `client.podAnnotations` | Client Deployment annotations | `{}` |\n| `client.nodeSelector` | Node labels for client pod assignment | `{}` |\n| `client.tolerations` | Client tolerations | `[]` |\n| `client.serviceAnnotations` | Client Service annotations | `{}` |\n| `client.serviceType` | Client service type | `ClusterIP` |\n| `client.loadBalancerIP` | Client loadBalancerIP | `{}` |\n| `client.loadBalancerSourceRanges` | Client loadBalancerSourceRanges | `{}` |\n| `master.exposeHttp` | Expose http port 9200 on master Pods for monitoring, etc | `false` |\n| `master.name` | Master component name | `master` |\n| `master.replicas` | Master node replicas (deployment) | `2` |\n| `master.resources` | Master node resources requests & limits | `{} - cpu limit must be an integer` |\n| `master.priorityClassName` | Master priorityClass | `nil` |\n| `master.podAnnotations` | Master Deployment annotations | `{}` |\n| `master.nodeSelector` | Node labels for master pod assignment | `{}` |\n| `master.tolerations` | Master tolerations | `[]` |\n| `master.heapSize` | Master node heap size | `512m` |\n| `master.name` | Master component name | `master` |\n| `master.persistence.enabled` | Master persistent enabled/disabled | `true` |\n| `master.persistence.name` | Master statefulset PVC template name | `data` |\n| `master.persistence.size` | Master persistent volume size | `4Gi` |\n| `master.persistence.storageClass` | Master persistent volume Class | `nil` |\n| `master.persistence.accessMode` | Master persistent Access Mode | `ReadWriteOnce` |\n| `data.exposeHttp` | Expose http port 9200 on data Pods for monitoring, etc | `false` |\n| `data.replicas` | Data node replicas (statefulset) | `2` |\n| `data.resources` | Data node resources requests & limits | `{} - cpu limit must be an integer` |\n| `data.priorityClassName` | Data priorityClass | `nil` |\n| `data.heapSize` | Data node heap size | `1536m` |\n| `data.persistence.enabled` | Data persistent enabled/disabled | `true` |\n| `data.persistence.name` | Data statefulset PVC template name | `data` |\n| `data.persistence.size` | Data persistent volume size | `30Gi` |\n| `data.persistence.storageClass` | Data persistent volume Class | `nil` |\n| `data.persistence.accessMode` | Data persistent Access Mode | `ReadWriteOnce` |\n| `data.podAnnotations` | Data StatefulSet annotations | `{}` |\n| `data.nodeSelector` | Node labels for data pod assignment | `{}` |\n| `data.tolerations` | Data tolerations | `[]` |\n| `data.terminationGracePeriodSeconds` | Data termination grace period (seconds) | `3600` |\n| `data.antiAffinity` | Data anti-affinity policy | `soft` |\n\nSpecify each parameter using the `--set key=value[,key=value]` argument to `helm install`.\n\nIn terms of Memory resources you should make sure that you follow that equation:\n\n- `${role}HeapSize < ${role}MemoryRequests < ${role}MemoryLimits`\n\nThe YAML value of cluster.config is appended to elasticsearch.yml file for additional customization (\"script.inline: on\" for example to allow inline scripting)\n\n# Deep dive\n\n## Application Version\n\nThis chart aims to support Elasticsearch v2 and v5 deployments by specifying the `values.yaml` parameter `appVersion`.\n\n### Version Specific Features\n\n* Memory Locking *(variable renamed)*\n* Ingest Node *(v5)*\n* X-Pack Plugin *(v5)*\n\nUpgrade paths & more info: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html\n\n## Mlocking\n\nThis is a limitation in kubernetes right now. There is no way to raise the\nlimits of lockable memory, so that these memory areas won't be swapped. This\nwould degrade performance heavily. The issue is tracked in\n[kubernetes/#3595](https://github.com/kubernetes/kubernetes/issues/3595).\n\n```\n[WARN ][bootstrap] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory\n[WARN ][bootstrap] This can result in part of the JVM being swapped out.\n[WARN ][bootstrap] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536\n```\n\n## Minimum Master Nodes\n> The minimum_master_nodes setting is extremely important to the stability of your cluster. This setting helps prevent split brains, the existence of two masters in a single cluster.\n\n>When you have a split brain, your cluster is at danger of losing data. Because the master is considered the supreme ruler of the cluster, it decides when new indices can be created, how shards are moved, and so forth. If you have two masters, data integrity becomes perilous, since you have two nodes that think they are in charge.\n\n>This setting tells Elasticsearch to not elect a master unless there are enough master-eligible nodes available. Only then will an election take place.\n\n>This setting should always be configured to a quorum (majority) of your master-eligible nodes. A quorum is (number of master-eligible nodes / 2) + 1\n\nMore info: https://www.elastic.co/guide/en/elasticsearch/guide/1.x/_important_configuration_changes.html#_minimum_master_nodes\n\n# Client and Coordinating Nodes\n\nElasticsearch v5 terminology has updated, and now refers to a `Client Node` as a `Coordinating Node`.\n\nMore info: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-node.html#coordinating-node\n\n## Select right storage class for SSD volumes\n\n### GCE + Kubernetes 1.5\n\nCreate StorageClass for SSD-PD\n\n```\n$ kubectl create -f - < >(tee -a \"/var/log/elasticsearch-hooks.log\")\n NODE_NAME=${HOSTNAME}\n echo \"Prepare to migrate data of the node ${NODE_NAME}\"\n echo \"Move all data from node ${NODE_NAME}\"\n curl -s -XPUT -H 'Content-Type: application/json' '{{ template \"elasticsearch.client.fullname\" . }}:9200/_cluster/settings' -d \"{\n \\\"transient\\\" :{\n \\\"cluster.routing.allocation.exclude._name\\\" : \\\"${NODE_NAME}\\\"\n }\n }\"\n echo \"\"\n\n while true ; do\n echo -e \"Wait for node ${NODE_NAME} to become empty\"\n SHARDS_ALLOCATION=$(curl -s -XGET 'http://{{ template \"elasticsearch.client.fullname\" . }}:9200/_cat/shards')\n if ! echo \"${SHARDS_ALLOCATION}\" | grep -E \"${NODE_NAME}\"; then\n break\n fi\n sleep 1\n done\n echo \"Node ${NODE_NAME} is ready to shutdown\"\n post-start-hook.sh: |-\n #!/bin/bash\n exec &> >(tee -a \"/var/log/elasticsearch-hooks.log\")\n NODE_NAME=${HOSTNAME}\n CLUSTER_SETTINGS=$(curl -s -XGET \"http://{{ template \"elasticsearch.client.fullname\" . }}:9200/_cluster/settings\")\n if echo \"${CLUSTER_SETTINGS}\" | grep -E \"${NODE_NAME}\"; then\n echo \"Activate node ${NODE_NAME}\"\n curl -s -XPUT -H 'Content-Type: application/json' \"http://{{ template \"elasticsearch.client.fullname\" . }}:9200/_cluster/settings\" -d \"{\n \\\"transient\\\" :{\n \\\"cluster.routing.allocation.exclude._name\\\" : null\n }\n }\"\n fi\n echo \"Node ${NODE_NAME} is ready to be used\"\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/templates/data-pdb.yaml",
"content": "{{- if .Values.data.podDisruptionBudget.enabled }}\napiVersion: policy/v1beta1\nkind: PodDisruptionBudget\nmetadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n component: \"{{ .Values.data.name }}\"\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n name: {{ template \"elasticsearch.data.fullname\" . }}\nspec:\n{{- if .Values.data.podDisruptionBudget.minAvailable }}\n minAvailable: {{ .Values.data.podDisruptionBudget.minAvailable }}\n{{- end }}\n{{- if .Values.data.podDisruptionBudget.maxUnavailable }}\n maxUnavailable: {{ .Values.data.podDisruptionBudget.maxUnavailable }}\n{{- end }}\n selector:\n matchLabels:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.data.name }}\"\n release: {{ .Release.Name }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/templates/data-statefulset.yaml",
"content": "apiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n component: \"{{ .Values.data.name }}\"\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n name: {{ template \"elasticsearch.data.fullname\" . }}\nspec:\n serviceName: {{ template \"elasticsearch.data.fullname\" . }}\n replicas: {{ .Values.data.replicas }}\n selector:\n matchLabels:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.data.name }}\"\n template:\n metadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.data.name }}\"\n release: {{ .Release.Name }}\n {{- if .Values.data.podAnnotations }}\n annotations:\n{{ toYaml .Values.data.podAnnotations | indent 8 }}\n {{- end }}\n spec:\n{{- if .Values.data.priorityClassName }}\n priorityClassName: \"{{ .Values.data.priorityClassName }}\"\n{{- end }}\n securityContext:\n fsGroup: 1000\n {{- if eq .Values.data.antiAffinity \"hard\" }}\n affinity:\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - topologyKey: \"kubernetes.io/hostname\"\n labelSelector:\n matchLabels:\n app: \"{{ template \"elasticsearch.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n component: \"{{ .Values.data.name }}\"\n {{- else if eq .Values.data.antiAffinity \"soft\" }}\n affinity:\n podAntiAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 1\n podAffinityTerm:\n topologyKey: kubernetes.io/hostname\n labelSelector:\n matchLabels:\n app: \"{{ template \"elasticsearch.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n component: \"{{ .Values.data.name }}\"\n {{- end }}\n{{- if .Values.data.nodeSelector }}\n nodeSelector:\n{{ toYaml .Values.data.nodeSelector | indent 8 }}\n{{- end }}\n{{- if .Values.data.tolerations }}\n tolerations:\n{{ toYaml .Values.data.tolerations | indent 8 }}\n{{- end }}\n initContainers:\n # see https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html\n # and https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#mlockall\n - name: \"sysctl\"\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\" \n imagePullPolicy: \"Always\"\n command: [\"sysctl\", \"-w\", \"vm.max_map_count=262144\"]\n securityContext:\n privileged: true\n - name: \"chown\"\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\"\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n command:\n - /bin/bash\n - -c\n - chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data &&\n chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/logs\n securityContext:\n runAsUser: 0\n volumeMounts:\n - mountPath: /usr/share/elasticsearch/data\n name: data\n containers:\n - name: elasticsearch\n env:\n - name: DISCOVERY_SERVICE\n value: {{ template \"elasticsearch.fullname\" . }}-discovery\n - name: NODE_MASTER\n value: \"false\"\n - name: PROCESSORS\n valueFrom:\n resourceFieldRef:\n resource: limits.cpu\n - name: ES_JAVA_OPTS\n value: \"-Djava.net.preferIPv4Stack=true -Xms{{ .Values.data.heapSize }} -Xmx{{ .Values.data.heapSize }}\"\n {{- range $key, $value := .Values.cluster.env }}\n - name: {{ $key }}\n value: {{ $value | quote }}\n {{- end }}\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\"\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n ports:\n - containerPort: 9300\n name: transport\n{{ if .Values.data.exposeHttp }}\n - containerPort: 9200\n name: http\n{{ end }}\n resources:\n{{ toYaml .Values.data.resources | indent 12 }}\n readinessProbe:\n httpGet:\n path: /_cluster/health?local=true\n port: 9200\n initialDelaySeconds: 5\n volumeMounts:\n - mountPath: /usr/share/elasticsearch/data\n name: data\n - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml\n name: config\n subPath: elasticsearch.yml\n{{- if hasPrefix \"2.\" .Values.image.tag }}\n - mountPath: /usr/share/elasticsearch/config/logging.yml\n name: config\n subPath: logging.yml\n{{- end }}\n{{- if hasPrefix \"5.\" .Values.image.tag }}\n - mountPath: /usr/share/elasticsearch/config/log4j2.properties\n name: config\n subPath: log4j2.properties\n{{- end }}\n - name: config\n mountPath: /pre-stop-hook.sh\n subPath: pre-stop-hook.sh\n - name: config\n mountPath: /post-start-hook.sh\n subPath: post-start-hook.sh\n{{- if .Values.cluster.keystoreSecret }}\n - name: keystore\n mountPath: \"/usr/share/elasticsearch/config/elasticsearch.keystore\"\n subPath: elasticsearch.keystore\n readOnly: true\n{{- end }}\n lifecycle:\n preStop:\n exec:\n command: [\"/bin/bash\",\"/pre-stop-hook.sh\"]\n postStart:\n exec:\n command: [\"/bin/bash\",\"/post-start-hook.sh\"]\n terminationGracePeriodSeconds: {{ .Values.data.terminationGracePeriodSeconds }}\n{{- if .Values.image.pullSecrets }}\n imagePullSecrets:\n {{- range $pullSecret := .Values.image.pullSecrets }}\n - name: {{ $pullSecret }}\n {{- end }}\n{{- end }}\n volumes:\n - name: config\n configMap:\n name: {{ template \"elasticsearch.fullname\" . }}\n{{- if .Values.cluster.keystoreSecret }}\n - name: keystore\n secret:\n secretName: {{ .Values.cluster.keystoreSecret }}\n{{- end }}\n {{- if not .Values.data.persistence.enabled }}\n - name: data\n emptyDir: {}\n {{- end }}\n updateStrategy:\n type: {{ .Values.data.updateStrategy.type }}\n {{- if .Values.data.persistence.enabled }}\n volumeClaimTemplates:\n - metadata:\n name: {{ .Values.data.persistence.name }}\n spec:\n accessModes:\n - {{ .Values.data.persistence.accessMode | quote }}\n {{- if .Values.data.persistence.storageClass }}\n {{- if (eq \"-\" .Values.data.persistence.storageClass) }}\n storageClassName: \"\"\n {{- else }}\n storageClassName: \"{{ .Values.data.persistence.storageClass }}\"\n {{- end }}\n {{- end }}\n resources:\n requests:\n storage: \"{{ .Values.data.persistence.size }}\"\n {{- end }}\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/templates/master-pdb.yaml",
"content": "{{- if .Values.master.podDisruptionBudget.enabled }}\napiVersion: policy/v1beta1\nkind: PodDisruptionBudget\nmetadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n component: \"{{ .Values.master.name }}\"\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n name: {{ template \"elasticsearch.master.fullname\" . }}\nspec:\n{{- if .Values.master.podDisruptionBudget.minAvailable }}\n minAvailable: {{ .Values.master.podDisruptionBudget.minAvailable }}\n{{- end }}\n{{- if .Values.master.podDisruptionBudget.maxUnavailable }}\n maxUnavailable: {{ .Values.master.podDisruptionBudget.maxUnavailable }}\n{{- end }}\n selector:\n matchLabels:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.master.name }}\"\n release: {{ .Release.Name }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/templates/master-statefulset.yaml",
"content": "apiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n component: \"{{ .Values.master.name }}\"\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n name: {{ template \"elasticsearch.master.fullname\" . }}\nspec:\n serviceName: {{ template \"elasticsearch.master.fullname\" . }}\n replicas: {{ .Values.master.replicas }}\n selector:\n matchLabels:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.master.name }}\"\n template:\n metadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.master.name }}\"\n release: {{ .Release.Name }}\n {{- if .Values.master.podAnnotations }}\n annotations:\n{{ toYaml .Values.master.podAnnotations | indent 8 }}\n {{- end }}\n spec:\n{{- if .Values.master.priorityClassName }}\n priorityClassName: \"{{ .Values.master.priorityClassName }}\"\n{{- end }}\n securityContext:\n fsGroup: 1000\n {{- if eq .Values.master.antiAffinity \"hard\" }}\n affinity:\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - topologyKey: \"kubernetes.io/hostname\"\n labelSelector:\n matchLabels:\n app: \"{{ template \"elasticsearch.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n component: \"{{ .Values.master.name }}\"\n {{- else if eq .Values.master.antiAffinity \"soft\" }}\n affinity:\n podAntiAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 1\n podAffinityTerm:\n topologyKey: kubernetes.io/hostname\n labelSelector:\n matchLabels:\n app: \"{{ template \"elasticsearch.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n component: \"{{ .Values.master.name }}\"\n {{- end }}\n{{- if .Values.master.nodeSelector }}\n nodeSelector:\n{{ toYaml .Values.master.nodeSelector | indent 8 }}\n{{- end }}\n{{- if .Values.master.tolerations }}\n tolerations:\n{{ toYaml .Values.master.tolerations | indent 8 }}\n{{- end }}\n initContainers:\n # see https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html\n # and https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#mlockall\n - name: \"sysctl\"\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\" \n imagePullPolicy: \"Always\"\n command: [\"sysctl\", \"-w\", \"vm.max_map_count=262144\"]\n securityContext:\n privileged: true\n - name: \"chown\"\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\"\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n command:\n - /bin/bash\n - -c\n - chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data &&\n chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/logs\n securityContext:\n runAsUser: 0\n volumeMounts:\n - mountPath: /usr/share/elasticsearch/data\n name: data\n containers:\n - name: elasticsearch\n env:\n - name: NODE_DATA\n value: \"false\"\n{{- if hasPrefix \"5.\" .Values.appVersion }}\n - name: NODE_INGEST\n value: \"false\"\n{{- end }}\n - name: DISCOVERY_SERVICE\n value: {{ template \"elasticsearch.fullname\" . }}-discovery\n - name: PROCESSORS\n valueFrom:\n resourceFieldRef:\n resource: limits.cpu\n - name: ES_JAVA_OPTS\n value: \"-Djava.net.preferIPv4Stack=true -Xms{{ .Values.master.heapSize }} -Xmx{{ .Values.master.heapSize }}\"\n {{- range $key, $value := .Values.cluster.env }}\n - name: {{ $key }}\n value: {{ $value | quote }}\n {{- end }}\n resources:\n{{ toYaml .Values.master.resources | indent 12 }}\n readinessProbe:\n httpGet:\n path: /_cluster/health?local=true\n port: 9200\n initialDelaySeconds: 5\n image: \"{{ .Values.image.repository }}:{{ .Values.image.tag }}\"\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n ports:\n - containerPort: 9300\n name: transport\n{{ if .Values.master.exposeHttp }}\n - containerPort: 9200\n name: http\n{{ end }}\n volumeMounts:\n - mountPath: /usr/share/elasticsearch/data\n name: data\n - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml\n name: config\n subPath: elasticsearch.yml\n{{- if hasPrefix \"2.\" .Values.image.tag }}\n - mountPath: /usr/share/elasticsearch/config/logging.yml\n name: config\n subPath: logging.yml\n{{- end }}\n{{- if hasPrefix \"5.\" .Values.image.tag }}\n - mountPath: /usr/share/elasticsearch/config/log4j2.properties\n name: config\n subPath: log4j2.properties\n{{- end }}\n{{- if .Values.cluster.keystoreSecret }}\n - name: keystore\n mountPath: \"/usr/share/elasticsearch/config/elasticsearch.keystore\"\n subPath: elasticsearch.keystore\n readOnly: true\n{{- end }}\n{{- if .Values.image.pullSecrets }}\n imagePullSecrets:\n {{- range $pullSecret := .Values.image.pullSecrets }}\n - name: {{ $pullSecret }}\n {{- end }}\n{{- end }}\n volumes:\n - name: config\n configMap:\n name: {{ template \"elasticsearch.fullname\" . }}\n{{- if .Values.cluster.keystoreSecret }}\n - name: keystore\n secret:\n secretName: {{ .Values.cluster.keystoreSecret }}\n{{- end }}\n {{- if not .Values.master.persistence.enabled }}\n - name: data\n emptyDir: {}\n {{- end }}\n updateStrategy:\n type: {{ .Values.master.updateStrategy.type }}\n {{- if .Values.master.persistence.enabled }}\n volumeClaimTemplates:\n - metadata:\n name: {{ .Values.master.persistence.name }}\n spec:\n accessModes:\n - {{ .Values.master.persistence.accessMode | quote }}\n {{- if .Values.master.persistence.storageClass }}\n {{- if (eq \"-\" .Values.master.persistence.storageClass) }}\n storageClassName: \"\"\n {{- else }}\n storageClassName: \"{{ .Values.master.persistence.storageClass }}\"\n {{- end }}\n {{- end }}\n resources:\n requests:\n storage: \"{{ .Values.master.persistence.size }}\"\n {{ end }}\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/templates/master-svc.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: {{ template \"elasticsearch.name\" . }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n component: \"{{ .Values.master.name }}\"\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n name: {{ template \"elasticsearch.fullname\" . }}-discovery\nspec:\n clusterIP: None\n ports:\n - port: 9300\n targetPort: transport\n selector:\n app: {{ template \"elasticsearch.name\" . }}\n component: \"{{ .Values.master.name }}\"\n release: {{ .Release.Name }}\n"
},
{
"path": "manifests/deprecated/es-cluster/elasticsearch/values.yaml",
"content": "# Default values for elasticsearch.\n# This is a YAML-formatted file.\n# Declare variables to be passed into your templates.\nappVersion: \"6.4.0\"\n\nimage:\n repository: \"docker.elastic.co/elasticsearch/elasticsearch-oss\"\n tag: \"6.4.0\"\n pullPolicy: \"IfNotPresent\"\n # If specified, use these secrets to access the image\n # pullSecrets:\n # - registry-secret\n\ncluster:\n name: \"elasticsearch\"\n # If you want X-Pack installed, switch to an image that includes it, enable this option and toggle the features you want\n # enabled in the environment variables outlined in the README\n xpackEnable: false\n # Some settings must be placed in a keystore, so they need to be mounted in from a secret.\n # Use this setting to specify the name of the secret\n # keystoreSecret: eskeystore\n config: {}\n env:\n # IMPORTANT: https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#minimum_master_nodes\n # To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible\n # node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.\n MINIMUM_MASTER_NODES: \"2\"\n\nclient:\n name: client\n replicas: 2\n serviceType: ClusterIP\n loadBalancerIP: {}\n loadBalancerSourceRanges: {}\n## (dict) If specified, apply these annotations to the client service\n# serviceAnnotations:\n# example: client-svc-foo\n heapSize: \"512m\"\n antiAffinity: \"soft\"\n nodeSelector: {}\n tolerations: []\n resources:\n limits:\n cpu: \"1\"\n # memory: \"1024Mi\"\n requests:\n cpu: \"25m\"\n memory: \"512Mi\"\n priorityClassName: \"\"\n ## (dict) If specified, apply these annotations to each client Pod\n # podAnnotations:\n # example: client-foo\n podDisruptionBudget:\n enabled: false\n minAvailable: 1\n # maxUnavailable: 1\n\nmaster:\n name: master\n exposeHttp: false\n replicas: 3\n heapSize: \"512m\"\n persistence:\n enabled: true\n accessMode: ReadWriteOnce\n name: data\n size: \"4Gi\"\n # storageClass: \"ssd\"\n antiAffinity: \"soft\"\n nodeSelector: {}\n tolerations: []\n resources:\n limits:\n cpu: \"1\"\n # memory: \"1024Mi\"\n requests:\n cpu: \"25m\"\n memory: \"512Mi\"\n priorityClassName: \"\"\n ## (dict) If specified, apply these annotations to each master Pod\n # podAnnotations:\n # example: master-foo\n podDisruptionBudget:\n enabled: false\n minAvailable: 2 # Same as `cluster.env.MINIMUM_MASTER_NODES`\n # maxUnavailable: 1\n updateStrategy:\n type: OnDelete\n\ndata:\n name: data\n exposeHttp: false\n replicas: 2\n heapSize: \"1536m\"\n persistence:\n enabled: true\n accessMode: ReadWriteOnce\n name: data\n size: \"30Gi\"\n # storageClass: \"ssd\"\n terminationGracePeriodSeconds: 3600\n antiAffinity: \"soft\"\n nodeSelector: {}\n tolerations: []\n resources:\n limits:\n cpu: \"1\"\n # memory: \"2048Mi\"\n requests:\n cpu: \"25m\"\n memory: \"1536Mi\"\n priorityClassName: \"\"\n ## (dict) If specified, apply these annotations to each data Pod\n # podAnnotations:\n # example: data-foo\n podDisruptionBudget:\n enabled: false\n # minAvailable: 1\n maxUnavailable: 1\n updateStrategy:\n type: OnDelete\n"
},
{
"path": "manifests/deprecated/es-cluster/es-values.yaml",
"content": "image:\n repository: \"jmgao1983/elasticsearch\"\n\ncluster:\n name: \"es-on-k8s\"\n env:\n MINIMUM_MASTER_NODES: \"2\"\n\nclient:\n serviceType: NodePort\n\nmaster:\n name: master\n replicas: 3\n heapSize: \"512m\"\n persistence:\n enabled: true\n accessMode: ReadWriteOnce\n name: data\n size: \"4Gi\"\n storageClass: \"nfs-es\"\n\ndata:\n name: data\n replicas: 2\n heapSize: \"1536m\"\n persistence:\n enabled: true\n accessMode: ReadWriteOnce\n name: data\n size: \"40Gi\"\n storageClass: \"nfs-es\"\n terminationGracePeriodSeconds: 3600\n resources:\n limits:\n cpu: \"1\"\n # memory: \"2048Mi\"\n requests:\n cpu: \"25m\"\n memory: \"1536Mi\"\n podDisruptionBudget:\n enabled: false\n # minAvailable: 1\n maxUnavailable: 1\n"
},
{
"path": "manifests/deprecated/ingress/nginx-ingress/nginx-ingress-svc.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: ingress-nginx\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nspec:\n type: NodePort\n ports:\n - name: http\n port: 80\n targetPort: 80\n protocol: TCP\n # 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围\n # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露给外部的访问\n nodePort: 23456\n - name: https\n port: 443\n targetPort: 443\n protocol: TCP\n # 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围\n # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露https\n nodePort: 23457\n - name: test-mysql\n port: 3306\n targetPort: 3306\n protocol: TCP\n nodePort: 23306\n - name: test-mysql-read\n port: 3307\n targetPort: 3307\n protocol: TCP\n nodePort: 23307\n - name: test-dns\n port: 53\n targetPort: 53\n protocol: UDP\n nodePort: 20053\n selector:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n\n"
},
{
"path": "manifests/deprecated/ingress/nginx-ingress/nginx-ingress.yaml",
"content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: ingress-nginx\n\n---\n\nkind: ConfigMap\napiVersion: v1\nmetadata:\n name: nginx-configuration\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n\n---\nkind: ConfigMap\napiVersion: v1\nmetadata:\n name: tcp-services\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n\n---\nkind: ConfigMap\napiVersion: v1\nmetadata:\n name: udp-services\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: nginx-ingress-serviceaccount\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n name: nginx-ingress-clusterrole\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nrules:\n - apiGroups:\n - \"\"\n resources:\n - configmaps\n - endpoints\n - nodes\n - pods\n - secrets\n verbs:\n - list\n - watch\n - apiGroups:\n - \"\"\n resources:\n - nodes\n verbs:\n - get\n - apiGroups:\n - \"\"\n resources:\n - services\n verbs:\n - get\n - list\n - watch\n - apiGroups:\n - \"extensions\"\n resources:\n - ingresses\n verbs:\n - get\n - list\n - watch\n - apiGroups:\n - \"\"\n resources:\n - events\n verbs:\n - create\n - patch\n - apiGroups:\n - \"extensions\"\n resources:\n - ingresses/status\n verbs:\n - update\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n name: nginx-ingress-role\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nrules:\n - apiGroups:\n - \"\"\n resources:\n - configmaps\n - pods\n - secrets\n - namespaces\n verbs:\n - get\n - apiGroups:\n - \"\"\n resources:\n - configmaps\n resourceNames:\n # Defaults to \"-\"\n # Here: \"-\"\n # This has to be adapted if you change either parameter\n # when launching the nginx-ingress-controller.\n - \"ingress-controller-leader-nginx\"\n verbs:\n - get\n - update\n - apiGroups:\n - \"\"\n resources:\n - configmaps\n verbs:\n - create\n - apiGroups:\n - \"\"\n resources:\n - endpoints\n verbs:\n - get\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n name: nginx-ingress-role-nisa-binding\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: Role\n name: nginx-ingress-role\nsubjects:\n - kind: ServiceAccount\n name: nginx-ingress-serviceaccount\n namespace: ingress-nginx\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: nginx-ingress-clusterrole-nisa-binding\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: nginx-ingress-clusterrole\nsubjects:\n - kind: ServiceAccount\n name: nginx-ingress-serviceaccount\n namespace: ingress-nginx\n\n---\n\napiVersion: apps/v1 \nkind: Deployment\nmetadata:\n name: nginx-ingress-controller\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nspec:\n replicas: 1\n selector:\n matchLabels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n template:\n metadata:\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n annotations:\n prometheus.io/port: \"10254\"\n prometheus.io/scrape: \"true\"\n spec:\n serviceAccountName: nginx-ingress-serviceaccount\n containers:\n - name: nginx-ingress-controller\n #image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0\n #使用以下镜像，方便国内下载加速\n image: jmgao1983/nginx-ingress-controller:0.21.0\n args:\n - /nginx-ingress-controller\n - --configmap=$(POD_NAMESPACE)/nginx-configuration\n - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services\n - --udp-services-configmap=$(POD_NAMESPACE)/udp-services\n - --publish-service=$(POD_NAMESPACE)/ingress-nginx\n - --annotations-prefix=nginx.ingress.kubernetes.io\n securityContext:\n capabilities:\n drop:\n - ALL\n add:\n - NET_BIND_SERVICE\n # www-data -> 33\n runAsUser: 33\n env:\n - name: POD_NAME\n valueFrom:\n fieldRef:\n fieldPath: metadata.name\n - name: POD_NAMESPACE\n valueFrom:\n fieldRef:\n fieldPath: metadata.namespace\n ports:\n - name: http\n containerPort: 80\n - name: https\n containerPort: 443\n # hostPort可以直接使用node节点的网络端口暴露服务\n #- name: mysql\n # containerPort: 3306\n # hostPort: 3306\n #- name: dns\n # containerPort: 53\n # hostPort: 53\n # protocol: UDP\n livenessProbe:\n failureThreshold: 3\n httpGet:\n path: /healthz\n port: 10254\n scheme: HTTP\n initialDelaySeconds: 10\n periodSeconds: 10\n successThreshold: 1\n timeoutSeconds: 1\n readinessProbe:\n failureThreshold: 3\n httpGet:\n path: /healthz\n port: 10254\n scheme: HTTP\n periodSeconds: 10\n successThreshold: 1\n timeoutSeconds: 1\n\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: ingress-nginx\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\nspec:\n type: NodePort\n ports:\n - name: http\n port: 80\n targetPort: 80\n protocol: TCP\n # 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围\n # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露给外部的访问\n nodePort: 23456\n - name: https\n port: 443\n targetPort: 443\n protocol: TCP\n # 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围\n # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露https\n nodePort: 23457\n selector:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\n\n---\n\n"
},
{
"path": "manifests/deprecated/ingress/nginx-ingress/tcp-services-configmap.yaml",
"content": "kind: ConfigMap\napiVersion: v1\nmetadata:\n name: tcp-services\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\ndata:\n 3306: \"mariadb/mydb-mariadb:3306\"\n 3307: \"mariadb/mydb-mariadb-slave:3306\"\n\n"
},
{
"path": "manifests/deprecated/ingress/nginx-ingress/udp-services-configmap.yaml",
"content": "kind: ConfigMap\napiVersion: v1\nmetadata:\n name: udp-services\n namespace: ingress-nginx\n labels:\n app.kubernetes.io/name: ingress-nginx\n app.kubernetes.io/part-of: ingress-nginx\ndata:\n 53: \"kube-system/kube-dns:53\"\n\n"
},
{
"path": "manifests/deprecated/ingress/test-hello.ing.yaml",
"content": "# kubectl run test-hello --image=nginx --expose --port=80\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: test-hello\nspec:\n rules:\n - host: hello.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: test-hello\n servicePort: 80\n"
},
{
"path": "manifests/deprecated/ingress/traefik/tls/hello-tls.ing.yaml",
"content": "apiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: hello-tls-ingress\n annotations:\n kubernetes.io/ingress.class: traefik\nspec:\n rules:\n - host: hello.test.com\n http:\n paths:\n - backend:\n serviceName: test-hello\n servicePort: 80\n tls:\n - secretName: traefik-cert\n"
},
{
"path": "manifests/deprecated/ingress/traefik/tls/k8s-dashboard.ing.yaml",
"content": "apiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: kubernetes-dashboard\n namespace: kube-system\n annotations:\n traefik.ingress.kubernetes.io/redirect-entry-point: https\nspec:\n rules:\n - host: dashboard.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: kubernetes-dashboard\n servicePort: 443\n\n"
},
{
"path": "manifests/deprecated/ingress/traefik/tls/traefik-controller.yaml",
"content": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: traefik-conf\n namespace: kube-system\ndata:\n traefik.toml: |\n # 设置insecureSkipVerify = true，可以配置backend为443(比如dashboard)的ingress规则\n insecureSkipVerify = true\n defaultEntryPoints = [\"http\", \"https\"]\n [entryPoints]\n [entryPoints.http]\n address = \":80\"\n ### 配置http 强制跳转 https\n #[entryPoints.http.redirect]\n # entryPoint = \"https\"\n ### 配置只信任trustedIPs传递过来X-Forwarded-*，默认全部信任；为了防止客户端地址伪造，需开启这个\n #[entryPoints.http.forwardedHeaders]\n # trustedIPs = [\"10.1.0.0/16\", \"172.20.0.0/16\", \"192.168.1.3\"]\n [entryPoints.https]\n address = \":443\"\n [entryPoints.https.tls]\n [[entryPoints.https.tls.certificates]]\n CertFile = \"/ssl/tls.crt\"\n KeyFile = \"/ssl/tls.key\"\n---\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: traefik-ingress-controller\n namespace: kube-system\n labels:\n k8s-app: traefik-ingress-lb\nspec:\n replicas: 1\n selector:\n matchLabels:\n k8s-app: traefik-ingress-lb\n template:\n metadata:\n labels:\n k8s-app: traefik-ingress-lb\n name: traefik-ingress-lb\n spec:\n serviceAccountName: traefik-ingress-controller\n terminationGracePeriodSeconds: 60\n volumes:\n - name: ssl\n secret:\n secretName: traefik-cert\n - name: config\n configMap:\n name: traefik-conf\n #nodeSelector:\n # node-role.kubernetes.io/traefik: \"true\"\n containers:\n - image: traefik:v1.7.20\n imagePullPolicy: IfNotPresent\n name: traefik-ingress-lb\n volumeMounts:\n - mountPath: \"/ssl\"\n name: \"ssl\"\n - mountPath: \"/config\"\n name: \"config\"\n resources:\n limits:\n cpu: 1000m\n memory: 800Mi\n requests:\n cpu: 500m\n memory: 600Mi\n args:\n - --configfile=/config/traefik.toml\n - --api\n - --kubernetes\n - --logLevel=INFO\n securityContext:\n capabilities:\n drop:\n - ALL\n add:\n - NET_BIND_SERVICE\n ports:\n - name: http\n containerPort: 80\n hostPort: 80\n - name: https\n containerPort: 443\n hostPort: 443\n---\nkind: Service\napiVersion: v1\nmetadata:\n name: traefik-ingress-service\n namespace: kube-system\nspec:\n selector:\n k8s-app: traefik-ingress-lb\n ports:\n - protocol: TCP\n # 该端口为 traefik ingress-controller的服务端口\n port: 80\n # 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围\n # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露给外部的访问\n nodePort: 23456\n name: http\n - protocol: TCP\n # \n port: 443\n nodePort: 23457\n name: https\n - protocol: TCP\n # 该端口为 traefik 的管理WEB界面\n port: 8080\n name: admin\n type: NodePort\n---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: traefik-ingress-controller\nrules:\n - apiGroups:\n - \"\"\n resources:\n - pods\n - services\n - endpoints\n - secrets\n verbs:\n - get\n - list\n - watch\n - apiGroups:\n - extensions\n resources:\n - ingresses\n verbs:\n - get\n - list\n - watch\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: traefik-ingress-controller\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: traefik-ingress-controller\nsubjects:\n- kind: ServiceAccount\n name: traefik-ingress-controller\n namespace: kube-system\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: traefik-ingress-controller\n namespace: kube-system\n"
},
{
"path": "manifests/deprecated/ingress/traefik/traefik-ingress.yaml",
"content": "---\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: traefik-ingress-controller\nrules:\n - apiGroups:\n - \"\"\n resources:\n - pods\n - services\n - endpoints\n - secrets\n verbs:\n - get\n - list\n - watch\n - apiGroups:\n - extensions\n resources:\n - ingresses\n verbs:\n - get\n - list\n - watch\n---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: traefik-ingress-controller\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: traefik-ingress-controller\nsubjects:\n- kind: ServiceAccount\n name: traefik-ingress-controller\n namespace: kube-system\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: traefik-ingress-controller\n namespace: kube-system\n---\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: traefik-ingress-controller\n namespace: kube-system\n labels:\n k8s-app: traefik-ingress-lb\nspec:\n replicas: 1\n selector:\n matchLabels:\n k8s-app: traefik-ingress-lb\n template:\n metadata:\n labels:\n k8s-app: traefik-ingress-lb\n name: traefik-ingress-lb\n spec:\n serviceAccountName: traefik-ingress-controller\n terminationGracePeriodSeconds: 60\n containers:\n - image: traefik:v1.7.20\n imagePullPolicy: IfNotPresent\n name: traefik-ingress-lb\n args:\n - --api\n - --kubernetes\n - --logLevel=INFO\n---\nkind: Service\napiVersion: v1\nmetadata:\n name: traefik-ingress-service\n namespace: kube-system\nspec:\n selector:\n k8s-app: traefik-ingress-lb\n ports:\n - protocol: TCP\n # 该端口为 traefik ingress-controller的服务端口\n port: 80\n # 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围\n # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露给外部的访问\n nodePort: 23456\n name: web\n - protocol: TCP\n # 该端口为 traefik 的管理WEB界面\n port: 8080\n name: admin\n type: NodePort\n"
},
{
"path": "manifests/deprecated/ingress/traefik/traefik-ui.ing.yaml",
"content": "---\napiVersion: networking.k8s.io/v1beta1\nkind: Ingress\nmetadata:\n name: traefik-web-ui\n namespace: kube-system\nspec:\n rules:\n - host: traefik-ui.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: traefik-ingress-service\n servicePort: 8080\n"
},
{
"path": "manifests/deprecated/ingress/whoami.ing.yaml",
"content": "# kubectl run whoami --image=emilevauge/whoami --port=80 --expose\napiVersion: networking.k8s.io/v1beta1 \nkind: Ingress\nmetadata:\n name: test-whoami\nspec:\n rules:\n - host: who.test.com\n http:\n paths:\n - path: /\n backend:\n serviceName: whoami\n servicePort: 80\n\n"
},
{
"path": "manifests/deprecated/ingress/whoami.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: whoami\n labels:\n app: whoami \nspec:\n ports:\n - name: web\n port: 80\n targetPort: 80\n selector:\n app: whoami\n sessionAffinity: None\n #type: NodePort\n\n---\napiVersion: apps/v1\nkind: Deployment \nmetadata:\n name: whoami\nspec: \n replicas: 2\n selector:\n matchLabels:\n app: whoami\n template:\n metadata:\n labels:\n app: whoami\n spec:\n containers:\n - name: whoami \n image: emilevauge/whoami\n ports:\n - containerPort: 80\n"
},
{
"path": "manifests/deprecated/jenkins/.helmignore",
"content": "# Patterns to ignore when building packages.\n# This supports shell glob matching, relative path matching, and\n# negation (prefixed with !). Only one pattern per line.\n.DS_Store\n# Common VCS dirs\n.git/\n.gitignore\n.bzr/\n.bzrignore\n.hg/\n.hgignore\n.svn/\n# Common backup files\n*.swp\n*.bak\n*.tmp\n*~\n# Various IDEs\n.project\n.idea/\n*.tmproj\n"
},
{
"path": "manifests/deprecated/jenkins/Chart.yaml",
"content": "name: jenkins\nhome: https://jenkins.io/\nversion: 0.16.6\nappVersion: 2.121.1\ndescription: Open source continuous integration server. It supports multiple SCM tools\n including CVS, Subversion and Git. It can execute Apache Ant and Apache Maven-based\n projects as well as arbitrary scripts.\nsources:\n- https://github.com/jenkinsci/jenkins\n- https://github.com/jenkinsci/docker-jnlp-slave\nmaintainers:\n- name: lachie83\n email: lachlan.evenson@microsoft.com\n- name: viglesiasce\n email: viglesias@google.com\n- name: lusyoe\n email: lusyoe@163.com\nicon: https://wiki.jenkins-ci.org/download/attachments/2916393/logo.png\n"
},
{
"path": "manifests/deprecated/jenkins/OWNERS",
"content": "approvers:\n- lachie83\n- viglesiasce\nreviewers:\n- lachie83\n- viglesiasce\n"
},
{
"path": "manifests/deprecated/jenkins/README.md",
"content": "# Jenkins Helm Chart\n\nJenkins master and slave cluster utilizing the Jenkins Kubernetes plugin\n\n* https://wiki.jenkins-ci.org/display/JENKINS/Kubernetes+Plugin\n\nInspired by the awesome work of Carlos Sanchez \n\n## Chart Details\n\nThis chart will do the following:\n\n* 1 x Jenkins Master with port 8080 exposed on an external LoadBalancer\n* All using Kubernetes Deployments\n\n## Installing the Chart\n\nTo install the chart with the release name `my-release`:\n\n```bash\n$ helm install --name my-release stable/jenkins\n```\n\n## Configuration\n\nThe following tables list the configurable parameters of the Jenkins chart and their default values.\n\n### Jenkins Master\n| Parameter | Description | Default |\n| --------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------- |\n| `nameOverride` | Override the resource name prefix | `jenkins` |\n| `fullnameOverride` | Override the full resource names | `jenkins-{release-name}` (or `jenkins` if release-name is `jenkins`) |\n| `Master.Name` | Jenkins master name | `jenkins-master` |\n| `Master.Image` | Master image name | `jenkinsci/jenkins` |\n| `Master.ImageTag` | Master image tag | `lts` |\n| `Master.ImagePullPolicy` | Master image pull policy | `Always` |\n| `Master.ImagePullSecret` | Master image pull secret | Not set |\n| `Master.Component` | k8s selector key | `jenkins-master` |\n| `Master.UseSecurity` | Use basic security | `true` |\n| `Master.AdminUser` | Admin username (and password) created as a secret if useSecurity is true | `admin` |\n| `Master.resources` | Resources allocation (Requests and Limits) | `{requests: {cpu: 50m, memory: 256Mi}, limits: {cpu: 2000m, memory: 2048Mi}}`|\n| `Master.InitContainerEnv` | Environment variables for Init Container | Not set |\n| `Master.ContainerEnv` | Environment variables for Jenkins Container | Not set |\n| `Master.UsePodSecurityContext` | Enable pod security context (must be `true` if `RunAsUser` or `FsGroup` are set) | `true` |\n| `Master.RunAsUser` | uid that jenkins runs with | `0` |\n| `Master.FsGroup` | uid that will be used for persistent volume | `0` |\n| `Master.ServiceAnnotations` | Service annotations | `{}` |\n| `Master.ServiceType` | k8s service type | `LoadBalancer` |\n| `Master.ServicePort` | k8s service port | `8080` |\n| `Master.NodePort` | k8s node port | Not set |\n| `Master.HealthProbes` | Enable k8s liveness and readiness probes | `true` |\n| `Master.HealthProbesLivenessTimeout` | Set the timeout for the liveness probe | `120` |\n| `Master.HealthProbesReadinessTimeout` | Set the timeout for the readiness probe | `60` |\n| `Master.HealthProbeLivenessFailureThreshold` | Set the failure threshold for the liveness probe | `12` |\n| `Master.ContainerPort` | Master listening port | `8080` |\n| `Master.SlaveListenerPort` | Listening port for agents | `50000` |\n| `Master.DisabledAgentProtocols` | Disabled agent protocols | `JNLP-connect JNLP2-connect` |\n| `Master.CSRF.DefaultCrumbIssuer.Enabled` | Enable the default CSRF Crumb issuer | `true` |\n| `Master.CSRF.DefaultCrumbIssuer.ProxyCompatability` | Enable proxy compatibility | `true` |\n| `Master.CLI` | Enable CLI over remoting | `false` |\n| `Master.LoadBalancerSourceRanges` | Allowed inbound IP addresses | `0.0.0.0/0` |\n| `Master.LoadBalancerIP` | Optional fixed external IP | Not set |\n| `Master.JMXPort` | Open a port, for JMX stats | Not set |\n| `Master.CustomConfigMap` | Use a custom ConfigMap | `false` |\n| `Master.Ingress.Annotations` | Ingress annotations | `{}` |\n| `Master.Ingress.TLS` | Ingress TLS configuration | `[]` |\n| `Master.InitScripts` | List of Jenkins init scripts | Not set |\n| `Master.CredentialsXmlSecret` | Kubernetes secret that contains a 'credentials.xml' file | Not set |\n| `Master.SecretsFilesSecret` | Kubernetes secret that contains 'secrets' files | Not set |\n| `Master.Jobs` | Jenkins XML job configs | Not set |\n| `Master.InstallPlugins` | List of Jenkins plugins to install | `kubernetes:0.11 workflow-aggregator:2.5 credentials-binding:1.11 git:3.2.0` |\n| `Master.ScriptApproval` | List of groovy functions to approve | Not set |\n| `Master.NodeSelector` | Node labels for pod assignment | `{}` |\n| `Master.Affinity` | Affinity settings | `{}` |\n| `Master.Tolerations` | Toleration labels for pod assignment | `{}` |\n| `Master.PodAnnotations` | Annotations for master pod | `{}` |\n| `NetworkPolicy.Enabled` | Enable creation of NetworkPolicy resources. | `false` |\n| `NetworkPolicy.ApiVersion` | NetworkPolicy ApiVersion | `extensions/v1beta1` |\n| `rbac.install` | Create service account and ClusterRoleBinding for Kubernetes plugin | `false` |\n| `rbac.apiVersion` | RBAC API version | `v1beta1` |\n| `rbac.roleRef` | Cluster role name to bind to | `cluster-admin` |\n\n### Jenkins Agent\n\n| Parameter | Description | Default |\n| ----------------------- | ----------------------------------------------- | ---------------------- |\n| `Agent.AlwaysPullImage` | Always pull agent container image before build | `false` |\n| `Agent.Enabled` | Enable Kubernetes plugin jnlp-agent podTemplate | `true` |\n| `Agent.Image` | Agent image name | `jenkinsci/jnlp-slave` |\n| `Agent.ImagePullSecret` | Agent image pull secret | Not set |\n| `Agent.ImageTag` | Agent image tag | `2.62` |\n| `Agent.Privileged` | Agent privileged container | `false` |\n| `Agent.resources` | Resources allocation (Requests and Limits) | `{requests: {cpu: 200m, memory: 256Mi}, limits: {cpu: 200m, memory: 256Mi}}`|\n| `Agent.volumes` | Additional volumes | `nil` |\n\nSpecify each parameter using the `--set key=value[,key=value]` argument to `helm install`.\n\nAlternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,\n\n```bash\n$ helm install --name my-release -f values.yaml stable/jenkins\n```\n\n> **Tip**: You can use the default [values.yaml](values.yaml)\n\n## Mounting volumes into your Agent pods\n\nYour Jenkins Agents will run as pods, and it's possible to inject volumes where needed:\n\n```yaml\nAgent:\n volumes:\n - type: Secret\n secretName: jenkins-mysecrets\n mountPath: /var/run/secrets/jenkins-mysecrets\n```\n\nThe supported volume types are: `ConfigMap`, `EmptyDir`, `HostPath`, `Nfs`, `Pod`, `Secret`. Each type supports a different set of configurable attributes, defined by [the corresponding Java class](https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes).\n\n## NetworkPolicy\n\nTo make use of the NetworkPolicy resources created by default,\ninstall [a networking plugin that implements the Kubernetes\nNetworkPolicy spec](https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy#before-you-begin).\n\nFor Kubernetes v1.5 & v1.6, you must also turn on NetworkPolicy by setting\nthe DefaultDeny namespace annotation. Note: this will enforce policy for _all_ pods in the namespace:\n\n kubectl annotate namespace default \"net.beta.kubernetes.io/network-policy={\\\"ingress\\\":{\\\"isolation\\\":\\\"DefaultDeny\\\"}}\"\n\nInstall helm chart with network policy enabled:\n\n $ helm install stable/jenkins --set NetworkPolicy.Enabled=true\n\n## Persistence\n\nThe Jenkins image stores persistence under `/var/jenkins_home` path of the container. A dynamically managed Persistent Volume\nClaim is used to keep the data across deployments, by default. This is known to work in GCE, AWS, and minikube. Alternatively,\na previously configured Persistent Volume Claim can be used.\n\nIt is possible to mount several volumes using `Persistence.volumes` and `Persistence.mounts` parameters.\n\n### Persistence Values\n\n| Parameter | Description | Default |\n| --------------------------- | ------------------------------- | --------------- |\n| `Persistence.Enabled` | Enable the use of a Jenkins PVC | `true` |\n| `Persistence.ExistingClaim` | Provide the name of a PVC | `nil` |\n| `Persistence.AccessMode` | The PVC access mode | `ReadWriteOnce` |\n| `Persistence.Size` | The size of the PVC | `8Gi` |\n| `Persistence.volumes` | Additional volumes | `nil` |\n| `Persistence.mounts` | Additional mounts | `nil` |\n| `Persistence.StorageClass` | The PV Provisioner | `nfs-dynamic-class`|\n\n#### Existing PersistentVolumeClaim\n\n1. Create the PersistentVolume\n1. Create the PersistentVolumeClaim\n1. Install the chart\n\n```bash\n$ helm install --name my-release --set Persistence.ExistingClaim=PVC_NAME stable/jenkins\n```\n\n## Custom ConfigMap\n\nWhen creating a new parent chart with this chart as a dependency, the `CustomConfigMap` parameter can be used to override the default config.xml provided.\nIt also allows for providing additional xml configuration files that will be copied into `/var/jenkins_home`. In the parent chart's values.yaml,\nset the `jenkins.Master.CustomConfigMap` value to true like so\n\n```yaml\njenkins:\n Master:\n CustomConfigMap: true\n```\n\nand provide the file `templates/config.tpl` in your parent chart for your use case. You can start by copying the contents of `config.yaml` from this chart into your parent charts `templates/config.tpl` as a basis for customization. Finally, you'll need to wrap the contents of `templates/config.tpl` like so:\n\n```yaml\n{{- define \"override_config_map\" }}\n \n{{ end }}\n```\n\n## RBAC\n\nIf running upon a cluster with RBAC enabled you will need to do the following:\n\n* `helm install stable/jenkins --set rbac.install=true`\n* Create a Jenkins credential of type Kubernetes service account with service account name provided in the `helm status` output.\n* Under configure Jenkins -- Update the credentials config in the cloud section to use the service account credential you created in the step above.\n\n## Run Jenkins as non root user\n\nThe default settings of this helm chart let Jenkins run as root user with uid `0`.\nDue to security reasons you may want to run Jenkins as a non root user.\nFortunately the default jenkins docker image `jenkins/jenkins` contains a user `jenkins` with uid `1000` that can be used for this purpose.\n\nSimply use the following settings to run Jenkins as `jenkins` user with uid `1000`.\n\n```yaml\njenkins:\n Master:\n RunAsUser: 1000\n FsGroup: 1000\n```\n\nDocs taken from https://github.com/jenkinsci/docker/blob/master/Dockerfile:\n_Jenkins is run with user `jenkins`, uid = 1000. If you bind mount a volume from the host or a data container,ensure you use the same uid_\n\n## Running behind a forward proxy\n\nThe master pod uses an Init Container to install plugins etc. If you are behind a corporate proxy it may be useful to set `Master.InitContainerEnv` to add environment variables such as `http_proxy`, so that these can be downloaded.\n\nAdditionally, you may want to add env vars for the Jenkins container, and the JVM (`Master.JavaOpts`).\n\n```yaml\nMaster:\n InitContainerEnv:\n - name: http_proxy\n value: \"http://192.168.64.1:3128\"\n - name: https_proxy\n value: \"http://192.168.64.1:3128\"\n - name: no_proxy\n value: \"\"\n ContainerEnv:\n - name: http_proxy\n value: \"http://192.168.64.1:3128\"\n - name: https_proxy\n value: \"http://192.168.64.1:3128\"\n JavaOpts: >-\n -Dhttp.proxyHost=192.168.64.1\n -Dhttp.proxyPort=3128\n -Dhttps.proxyHost=192.168.64.1\n -Dhttps.proxyPort=3128\n```\n"
},
{
"path": "manifests/deprecated/jenkins/templates/NOTES.txt",
"content": "1. Get your '{{ .Values.Master.AdminUser }}' user password by running:\n printf $(kubectl get secret --namespace {{ .Release.Namespace }} {{ template \"jenkins.fullname\" . }} -o jsonpath=\"{.data.jenkins-admin-password}\" | base64 --decode);echo\n\n{{- if .Values.Master.HostName }}\n\n2. Visit http://{{ .Values.Master.HostName }}\n{{- else }}\n2. Get the Jenkins URL to visit by running these commands in the same shell:\n{{- if contains \"NodePort\" .Values.Master.ServiceType }}\n export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath=\"{.spec.ports[0].nodePort}\" services {{ template \"jenkins.fullname\" . }})\n export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath=\"{.items[0].status.addresses[0].address}\")\n echo http://$NODE_IP:$NODE_PORT/login\n\n{{- else if contains \"LoadBalancer\" .Values.Master.ServiceType }}\n NOTE: It may take a few minutes for the LoadBalancer IP to be available.\n You can watch the status of by running 'kubectl get svc --namespace {{ .Release.Namespace }} -w {{ template \"jenkins.fullname\" . }}'\n export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template \"jenkins.fullname\" . }} --template \"{{ \"{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}\" }}\")\n echo http://$SERVICE_IP:{{ .Values.Master.ServicePort }}/login\n\n{{- else if contains \"ClusterIP\" .Values.Master.ServiceType }}\n export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l \"component={{ template \"jenkins.fullname\" . }}-master\" -o jsonpath=\"{.items[0].metadata.name}\")\n echo http://127.0.0.1:{{ .Values.Master.ServicePort }}\n kubectl port-forward $POD_NAME {{ .Values.Master.ServicePort }}:{{ .Values.Master.ServicePort }}\n\n{{- end }}\n{{- end }}\n\n3. Login with the password from step 1 and the username: {{ .Values.Master.AdminUser }}\n\nFor more information on running Jenkins on Kubernetes, visit:\nhttps://cloud.google.com/solutions/jenkins-on-container-engine\n\n{{- if .Values.Persistence.Enabled }}\n{{- else }}\n#################################################################################\n###### WARNING: Persistence is disabled!!! You will lose your data when #####\n###### the Jenkins pod is terminated. #####\n#################################################################################\n{{- end }}\n\n{{- if .Values.rbac.install }}\nConfigure the Kubernetes plugin in Jenkins to use the following Service Account name {{ template \"jenkins.fullname\" . }} using the following steps:\n Create a Jenkins credential of type Kubernetes service account with service account name {{ template \"jenkins.fullname\" . }}\n Under configure Jenkins -- Update the credentials config in the cloud section to use the service account credential you created in the step above.\n{{- end }}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/_helpers.tpl",
"content": "{{/* vim: set filetype=mustache: */}}\n{{/*\nExpand the name of the chart.\n*/}}\n{{- define \"jenkins.name\" -}}\n{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n\n{{/*\nCreate a default fully qualified app name.\nWe truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).\nIf release name contains chart name it will be used as a full name.\n*/}}\n{{- define \"jenkins.fullname\" -}}\n{{- if .Values.fullnameOverride -}}\n{{- .Values.fullnameOverride | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- $name := default .Chart.Name .Values.nameOverride -}}\n{{- if contains $name .Release.Name -}}\n{{- .Release.Name | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- printf \"%s-%s\" .Release.Name $name | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n{{- end -}}\n{{- end -}}\n\n{{- define \"jenkins.kubernetes-version\" -}}\n {{- range .Values.Master.InstallPlugins -}}\n {{ if hasPrefix \"kubernetes:\" . }}\n {{- $split := splitList \":\" . }}\n {{- printf \"%s\" (index $split 1 ) -}}\n {{- end -}}\n {{- end -}}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/config.yaml",
"content": "{{- if not .Values.Master.CustomConfigMap }}\n\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"jenkins.fullname\" . }}\ndata:\n config.xml: |-\n \n \n {{ .Values.Master.ImageTag }} \n 0 \n NORMAL \n {{ .Values.Master.UseSecurity }} \n \n true \n \n \n false \n \n ${JENKINS_HOME}/workspace/${ITEM_FULLNAME} \n ${ITEM_ROOTDIR}/builds \n \n \n \n \n \n kubernetes \n \n{{- if .Values.Agent.Enabled }}\n \n default \n 2147483647 \n 0 \n {{ .Values.Agent.Component }} \n \n {{- $local := dict \"first\" true }}\n {{- range $key, $value := .Values.Agent.NodeSelector }}\n {{- if not $local.first }},{{- end }}\n {{- $key }}={{ $value }}\n {{- $_ := set $local \"first\" false }}\n {{- end }} \n NORMAL \n \n{{- range $index, $volume := .Values.Agent.volumes }}\n \n{{- range $key, $value := $volume }}{{- if not (eq $key \"type\") }}\n <{{ $key }}>{{ $value }}\n{{- end }}{{- end }}\n \n{{- end }}\n \n \n \n jnlp \n {{ .Values.Agent.Image }}:{{ .Values.Agent.ImageTag }} \n{{- if .Values.Agent.Privileged }}\n true \n{{- else }}\n false \n{{- end }}\n {{ .Values.Agent.AlwaysPullImage }} \n /home/jenkins \n ${computer.jnlpmac} ${computer.name} \n false \n # Resources configuration is a little hacky. This was to prevent breaking\n # changes, and should be cleanned up in the future once everybody had\n # enough time to migrate.\n {{.Values.Agent.Cpu | default .Values.Agent.resources.requests.cpu}} \n {{.Values.Agent.Memory | default .Values.Agent.resources.requests.memory}} \n {{.Values.Agent.Cpu | default .Values.Agent.resources.limits.cpu}} \n {{.Values.Agent.Memory | default .Values.Agent.resources.limits.memory}} \n \n \n \n \n {{ .Values.Agent.ImagePullSecret }} \n \n \n{{- else }}\n \n{{- end -}}\n \n https://kubernetes \n false \n {{ .Release.Namespace }} \n http://{{ template \"jenkins.fullname\" . }}:{{.Values.Master.ServicePort}}{{ default \"\" .Values.Master.JenkinsUriPrefix }} \n {{ template \"jenkins.fullname\" . }}-agent:50000 \n 10 \n 5 \n 0 \n 0 \n \n \n 5 \n 0 \n \n \n \n All \n false \n false \n \n \n \n All \n 50000 \n \n{{- range .Values.Master.DisabledAgentProtocols }}\n {{ . }} \n{{- end }}\n \n \n{{- if .Values.Master.CSRF.DefaultCrumbIssuer.ProxyCompatability }}\n true \n{{- end }}\n \n{{- end }}\n true \n \n{{- if .Values.Master.ScriptApproval }}\n scriptapproval.xml: |-\n \n \n \n{{- range $key, $val := .Values.Master.ScriptApproval }}\n {{ $val }} \n{{- end }}\n \n \n{{- end }}\n jenkins.CLI.xml: |-\n \n \n{{- if .Values.Master.CLI }}\n true \n{{- else }}\n false \n{{- end }}\n \n hudson.model.UpdateCenter.xml: |-\n \n \n \n default \n{{- if .Values.Master.UpdateCenter }}\n {{ .Values.Master.UpdateCenter }} \n{{- else }}\n https://updates.jenkins.io/update-center.json \n{{- end }}\n \n \n apply_config.sh: |-\n mkdir -p /usr/share/jenkins/ref/secrets/;\n echo \"false\" > /usr/share/jenkins/ref/secrets/slave-to-master-security-kill-switch;\n cp -n /var/jenkins_config/config.xml /var/jenkins_home;\n cp -n /var/jenkins_config/jenkins.CLI.xml /var/jenkins_home;\n cp -n /var/jenkins_config/hudson.model.UpdateCenter.xml /var/jenkins_home;\n{{- if .Values.Master.InstallPlugins }}\n # Install missing plugins\n cp /var/jenkins_config/plugins.txt /var/jenkins_home;\n rm -rf /usr/share/jenkins/ref/plugins/*.lock\n /usr/local/bin/install-plugins.sh `echo $(cat /var/jenkins_home/plugins.txt)`;\n # Copy plugins to shared volume\n cp -n /usr/share/jenkins/ref/plugins/* /var/jenkins_plugins;\n{{- end }}\n{{- if .Values.Master.ScriptApproval }}\n cp -n /var/jenkins_config/scriptapproval.xml /var/jenkins_home/scriptApproval.xml;\n{{- end }}\n{{- if .Values.Master.InitScripts }}\n mkdir -p /var/jenkins_home/init.groovy.d/;\n cp -n /var/jenkins_config/*.groovy /var/jenkins_home/init.groovy.d/\n{{- end }}\n{{- if .Values.Master.CredentialsXmlSecret }}\n cp -n /var/jenkins_credentials/credentials.xml /var/jenkins_home;\n{{- end }}\n{{- if .Values.Master.SecretsFilesSecret }}\n cp -n /var/jenkins_secrets/* /usr/share/jenkins/ref/secrets;\n{{- end }}\n{{- if .Values.Master.Jobs }}\n for job in $(ls /var/jenkins_jobs); do\n mkdir -p /var/jenkins_home/jobs/$job\n cp -n /var/jenkins_jobs/$job /var/jenkins_home/jobs/$job/config.xml\n done\n{{- end }}\n{{- range $key, $val := .Values.Master.InitScripts }}\n init{{ $key }}.groovy: |-\n{{ $val | indent 4 }}\n{{- end }}\n plugins.txt: |-\n{{- if .Values.Master.InstallPlugins }}\n{{- range $index, $val := .Values.Master.InstallPlugins }}\n{{ $val | indent 4 }}\n{{- end }}\n{{- end }}\n{{ else }}\n{{ include \"override_config_map\" . }}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/home-pvc.yaml",
"content": "{{- if and .Values.Persistence.Enabled (not .Values.Persistence.ExistingClaim) -}}\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n{{- if .Values.Persistence.Annotations }}\n annotations:\n{{ toYaml .Values.Persistence.Annotations | indent 4 }}\n{{- end }}\n name: {{ template \"jenkins.fullname\" . }}\n labels:\n app: {{ template \"jenkins.fullname\" . }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n release: \"{{ .Release.Name }}\"\n heritage: \"{{ .Release.Service }}\"\nspec:\n accessModes:\n - {{ .Values.Persistence.AccessMode | quote }}\n resources:\n requests:\n storage: {{ .Values.Persistence.Size | quote }}\n{{- if .Values.Persistence.StorageClass }}\n{{- if (eq \"-\" .Values.Persistence.StorageClass) }}\n storageClassName: \"\"\n{{- else }}\n storageClassName: \"{{ .Values.Persistence.StorageClass }}\"\n{{- end }}\n{{- end }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jenkins-agent-svc.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: {{ template \"jenkins.fullname\" . }}-agent\n labels:\n app: {{ template \"jenkins.fullname\" . }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n component: \"{{ .Release.Name }}-{{ .Values.Master.Component }}\"\n{{- if .Values.Master.SlaveListenerServiceAnnotations }}\n annotations:\n{{ toYaml .Values.Master.SlaveListenerServiceAnnotations | indent 4 }}\n{{- end }}\nspec:\n ports:\n - port: {{ .Values.Master.SlaveListenerPort }}\n targetPort: {{ .Values.Master.SlaveListenerPort }}\n name: slavelistener\n selector:\n component: \"{{ .Release.Name }}-{{ .Values.Master.Component }}\"\n type: {{ .Values.Master.SlaveListenerServiceType }}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jenkins-master-deployment.yaml",
"content": "apiVersion: apps/v1 \nkind: Deployment\nmetadata:\n name: {{ template \"jenkins.fullname\" . }}\n labels:\n heritage: {{ .Release.Service | quote }}\n release: {{ .Release.Name | quote }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n component: \"{{ .Release.Name }}-{{ .Values.Master.Name }}\"\nspec:\n replicas: 1\n strategy:\n type: RollingUpdate\n selector:\n matchLabels:\n component: \"{{ .Release.Name }}-{{ .Values.Master.Component }}\"\n template:\n metadata:\n labels:\n app: {{ template \"jenkins.fullname\" . }}\n heritage: {{ .Release.Service | quote }}\n release: {{ .Release.Name | quote }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n component: \"{{ .Release.Name }}-{{ .Values.Master.Component }}\"\n annotations:\n checksum/config: {{ include (print $.Template.BasePath \"/config.yaml\") . | sha256sum }}\n {{- if .Values.Master.PodAnnotations }}\n{{ toYaml .Values.Master.PodAnnotations | indent 8 }}\n {{- end }}\n spec:\n {{- if .Values.Master.NodeSelector }}\n nodeSelector:\n{{ toYaml .Values.Master.NodeSelector | indent 8 }}\n {{- end }}\n {{- if .Values.Master.Tolerations }}\n tolerations:\n{{ toYaml .Values.Master.Tolerations | indent 8 }}\n {{- end }}\n {{- if .Values.Master.Affinity }}\n affinity:\n{{ toYaml .Values.Master.Affinity | indent 8 }}\n {{- end }}\n{{- if .Values.Master.UsePodSecurityContext }}\n securityContext:\n runAsUser: {{ default 0 .Values.Master.RunAsUser }}\n{{- if and (.Values.Master.RunAsUser) (.Values.Master.FsGroup) }}\n{{- if not (eq .Values.Master.RunAsUser 0.0) }}\n fsGroup: {{ .Values.Master.FsGroup }}\n{{- end }}\n{{- end }}\n{{- end }}\n serviceAccountName: {{ if .Values.rbac.install }}{{ template \"jenkins.fullname\" . }}{{ else }}\"{{ .Values.rbac.serviceAccountName }}\"{{ end }}\n initContainers:\n - name: \"copy-default-config\"\n image: \"{{ .Values.Master.Image }}:{{ .Values.Master.ImageTag }}\"\n imagePullPolicy: \"{{ .Values.Master.ImagePullPolicy }}\"\n command: [ \"sh\", \"/var/jenkins_config/apply_config.sh\" ]\n {{- if .Values.Master.InitContainerEnv }}\n env:\n{{ toYaml .Values.Master.InitContainerEnv | indent 12 }}\n {{- end }}\n volumeMounts:\n -\n mountPath: /var/jenkins_home\n name: jenkins-home\n -\n mountPath: /var/jenkins_config\n name: jenkins-config\n {{- if .Values.Master.CredentialsXmlSecret }}\n -\n mountPath: /var/jenkins_credentials\n name: jenkins-credentials\n readOnly: true\n {{- end }}\n {{- if .Values.Master.SecretsFilesSecret }}\n -\n mountPath: /var/jenkins_secrets\n name: jenkins-secrets\n readOnly: true\n {{- end }}\n {{- if .Values.Master.Jobs }}\n -\n mountPath: /var/jenkins_jobs\n name: jenkins-jobs\n readOnly: true\n {{- end }}\n {{- if .Values.Master.InstallPlugins }}\n -\n mountPath: /var/jenkins_plugins\n name: plugin-dir\n {{- end }}\n -\n mountPath: /usr/share/jenkins/ref/secrets/\n name: secrets-dir\n containers:\n - name: {{ template \"jenkins.fullname\" . }}\n image: \"{{ .Values.Master.Image }}:{{ .Values.Master.ImageTag }}\"\n imagePullPolicy: \"{{ .Values.Master.ImagePullPolicy }}\"\n {{- if .Values.Master.UseSecurity }}\n args: [ \"--argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)\", \"--argumentsRealm.roles.$(ADMIN_USER)=admin\"]\n {{- end }}\n env:\n - name: JAVA_OPTS\n value: \"{{ default \"\" .Values.Master.JavaOpts}}\"\n - name: JENKINS_OPTS\n value: \"{{ if .Values.Master.JenkinsUriPrefix }}--prefix={{ .Values.Master.JenkinsUriPrefix }} {{ end }}{{ default \"\" .Values.Master.JenkinsOpts}}\"\n {{- if .Values.Master.UseSecurity }}\n - name: ADMIN_PASSWORD\n valueFrom:\n secretKeyRef:\n name: {{ template \"jenkins.fullname\" . }}\n key: jenkins-admin-password\n - name: ADMIN_USER\n valueFrom:\n secretKeyRef:\n name: {{ template \"jenkins.fullname\" . }}\n key: jenkins-admin-user\n {{- end }}\n {{- if .Values.Master.ContainerEnv }}\n{{ toYaml .Values.Master.ContainerEnv | indent 12 }}\n {{- end }}\n ports:\n - containerPort: {{ .Values.Master.ContainerPort }}\n name: http\n - containerPort: {{ .Values.Master.SlaveListenerPort }}\n name: slavelistener\n {{- if .Values.Master.JMXPort }}\n - containerPort: {{ .Values.Master.JMXPort }}\n name: jmx\n {{- end }}\n{{- if .Values.Master.HealthProbes }}\n livenessProbe:\n httpGet:\n path: /login\n port: http\n initialDelaySeconds: {{ .Values.Master.HealthProbesLivenessTimeout }}\n timeoutSeconds: 5\n failureThreshold: {{ .Values.Master.HealthProbeLivenessFailureThreshold }}\n readinessProbe:\n httpGet:\n path: /login\n port: http\n initialDelaySeconds: {{ .Values.Master.HealthProbesReadinessTimeout }}\n{{- end }}\n resources:\n{{ if or .Values.Master.Cpu .Values.Master.Memory }}\n requests:\n cpu: \"{{ .Values.Master.Cpu }}\"\n memory: \"{{ .Values.Master.Memory }}\"\n{{ else }}\n{{ toYaml .Values.Master.resources | indent 12 }}\n{{ end }}\n volumeMounts:\n{{- if .Values.Persistence.mounts }}\n{{ toYaml .Values.Persistence.mounts | indent 12 }}\n{{- end }}\n -\n mountPath: /var/jenkins_home\n name: jenkins-home\n readOnly: false\n -\n mountPath: /var/jenkins_config\n name: jenkins-config\n readOnly: true\n {{- if .Values.Master.CredentialsXmlSecret }}\n -\n mountPath: /var/jenkins_credentials\n name: jenkins-credentials\n readOnly: true\n {{- end }}\n {{- if .Values.Master.SecretsFilesSecret }}\n -\n mountPath: /var/jenkins_secrets\n name: jenkins-secrets\n readOnly: true\n {{- end }}\n {{- if .Values.Master.Jobs }}\n -\n mountPath: /var/jenkins_jobs\n name: jenkins-jobs\n readOnly: true\n {{- end }}\n {{- if .Values.Master.InstallPlugins }}\n -\n mountPath: /usr/share/jenkins/ref/plugins/\n name: plugin-dir\n readOnly: false\n {{- end }}\n -\n mountPath: /usr/share/jenkins/ref/secrets/\n name: secrets-dir\n readOnly: false\n volumes:\n{{- if .Values.Persistence.volumes }}\n{{ toYaml .Values.Persistence.volumes | indent 6 }}\n{{- end }}\n - name: jenkins-config\n configMap:\n name: {{ template \"jenkins.fullname\" . }}\n {{- if .Values.Master.CredentialsXmlSecret }}\n - name: jenkins-credentials\n secret:\n secretName: {{ .Values.Master.CredentialsXmlSecret }}\n {{- end }}\n {{- if .Values.Master.SecretsFilesSecret }}\n - name: jenkins-secrets\n secret:\n secretName: {{ .Values.Master.SecretsFilesSecret }}\n {{- end }}\n {{- if .Values.Master.Jobs }}\n - name: jenkins-jobs\n configMap:\n name: {{ template \"jenkins.fullname\" . }}-jobs\n {{- end }}\n {{- if .Values.Master.InstallPlugins }}\n - name: plugin-dir\n emptyDir: {}\n {{- end }}\n - name: secrets-dir\n emptyDir: {}\n - name: jenkins-home\n {{- if .Values.Persistence.Enabled }}\n persistentVolumeClaim:\n claimName: {{ .Values.Persistence.ExistingClaim | default (include \"jenkins.fullname\" .) }}\n {{- else }}\n emptyDir: {}\n {{- end -}}\n{{- if .Values.Master.ImagePullSecret }}\n imagePullSecrets:\n - name: {{ .Values.Master.ImagePullSecret }}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jenkins-master-ingress.yaml",
"content": "{{- if .Values.Master.HostName }}\napiVersion: {{ .Values.Master.Ingress.ApiVersion }}\nkind: Ingress\nmetadata:\n{{- if .Values.Master.Ingress.Annotations }}\n annotations:\n{{ toYaml .Values.Master.Ingress.Annotations | indent 4 }}\n{{- end }}\n name: {{ template \"jenkins.fullname\" . }}\nspec:\n rules:\n - host: {{ .Values.Master.HostName | quote }}\n http:\n paths:\n - backend:\n serviceName: {{ template \"jenkins.fullname\" . }}\n servicePort: {{ .Values.Master.ServicePort }}\n{{- if .Values.Master.Ingress.TLS }}\n tls:\n{{ toYaml .Values.Master.Ingress.TLS | indent 4 }}\n{{- end -}}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jenkins-master-networkpolicy.yaml",
"content": "{{- if .Values.NetworkPolicy.Enabled }}\nkind: NetworkPolicy\napiVersion: {{ .Values.NetworkPolicy.ApiVersion }}\nmetadata:\n name: \"{{ .Release.Name }}-{{ .Values.Master.Component }}\"\nspec:\n podSelector:\n matchLabels:\n component: \"{{ .Release.Name }}-{{ .Values.Master.Component }}\"\n ingress:\n # Allow web access to the UI\n - ports:\n - port: {{ .Values.Master.ContainerPort }}\n # Allow inbound connections from slave\n - from:\n - podSelector:\n matchLabels:\n \"jenkins/{{ .Release.Name }}-{{ .Values.Agent.Component }}\": \"true\"\n ports:\n - port: {{ .Values.Master.SlaveListenerPort }}\n{{- if .Values.Agent.Enabled }}\n---\nkind: NetworkPolicy\napiVersion: {{ .Values.NetworkPolicy.ApiVersion }}\nmetadata:\n name: \"{{ .Release.Name }}-{{ .Values.Agent.Component }}\"\nspec:\n podSelector:\n matchLabels:\n # DefaultDeny\n \"jenkins/{{ .Release.Name }}-{{ .Values.Agent.Component }}\": \"true\"\n{{- end }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jenkins-master-svc.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: {{template \"jenkins.fullname\" . }}\n labels:\n app: {{ template \"jenkins.fullname\" . }}\n heritage: {{.Release.Service | quote }}\n release: {{.Release.Name | quote }}\n chart: \"{{.Chart.Name}}-{{.Chart.Version}}\"\n component: \"{{.Release.Name}}-{{.Values.Master.Component}}\"\n{{- if .Values.Master.ServiceAnnotations }}\n annotations:\n{{ toYaml .Values.Master.ServiceAnnotations | indent 4 }}\n{{- end }}\nspec:\n ports:\n - port: {{.Values.Master.ServicePort}}\n name: http\n targetPort: {{.Values.Master.ContainerPort}}\n {{if (and (eq .Values.Master.ServiceType \"NodePort\") (not (empty .Values.Master.NodePort)))}}\n nodePort: {{.Values.Master.NodePort}}\n {{end}}\n selector:\n component: \"{{.Release.Name}}-{{.Values.Master.Component}}\"\n type: {{.Values.Master.ServiceType}}\n {{if eq .Values.Master.ServiceType \"LoadBalancer\"}}\n loadBalancerSourceRanges: {{.Values.Master.LoadBalancerSourceRanges}}\n {{if .Values.Master.LoadBalancerIP}}\n loadBalancerIP: {{.Values.Master.LoadBalancerIP}}\n {{end}}\n {{end}}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jenkins-test.yaml",
"content": "apiVersion: v1\nkind: Pod\nmetadata:\n name: \"{{ .Release.Name }}-ui-test-{{ randAlphaNum 5 | lower }}\"\n annotations:\n \"helm.sh/hook\": test-success\nspec:\n {{- if .Values.Master.NodeSelector }}\n nodeSelector:\n{{ toYaml .Values.Master.NodeSelector | indent 4 }}\n {{- end }}\n {{- if .Values.Master.Tolerations }}\n tolerations:\n{{ toYaml .Values.Master.Tolerations | indent 4 }}\n {{- end }}\n initContainers:\n - name: \"test-framework\"\n image: \"dduportal/bats:0.4.0\"\n command:\n - \"bash\"\n - \"-c\"\n - |\n set -ex\n # copy bats to tools dir\n cp -R /usr/local/libexec/ /tools/bats/\n volumeMounts:\n - mountPath: /tools\n name: tools\n containers:\n - name: {{ .Release.Name }}-ui-test\n image: {{ .Values.Master.Image }}:{{ .Values.Master.ImageTag }}\n command: [\"/tools/bats/bats\", \"-t\", \"/tests/run.sh\"]\n volumeMounts:\n - mountPath: /tests\n name: tests\n readOnly: true\n - mountPath: /tools\n name: tools\n volumes:\n - name: tests\n configMap:\n name: {{ template \"jenkins.fullname\" . }}-tests\n - name: tools\n emptyDir: {}\n restartPolicy: Never\n"
},
{
"path": "manifests/deprecated/jenkins/templates/jobs.yaml",
"content": "{{- if .Values.Master.Jobs }}\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"jenkins.fullname\" . }}-jobs\ndata:\n{{ .Values.Master.Jobs | indent 2 }}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/jenkins/templates/rbac.yaml",
"content": "{{ if .Values.rbac.install }}\n{{- $serviceName := include \"jenkins.fullname\" . -}}\napiVersion: rbac.authorization.k8s.io/{{ required \"A valid .Values.rbac.apiVersion entry required!\" .Values.rbac.apiVersion }}\nkind: ClusterRoleBinding\nmetadata:\n name: {{ $serviceName }}-role-binding\n labels:\n app: {{ $serviceName }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n release: \"{{ .Release.Name }}\"\n heritage: \"{{ .Release.Service }}\"\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: {{ .Values.rbac.roleRef }}\nsubjects:\n- kind: ServiceAccount\n name: {{ $serviceName }}\n namespace: {{ .Release.Namespace }}\n{{ end }}"
},
{
"path": "manifests/deprecated/jenkins/templates/secret.yaml",
"content": "{{- if .Values.Master.UseSecurity }}\napiVersion: v1\nkind: Secret\nmetadata:\n name: {{ template \"jenkins.fullname\" . }}\n labels:\n app: {{ template \"jenkins.fullname\" . }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n release: \"{{ .Release.Name }}\"\n heritage: \"{{ .Release.Service }}\"\ntype: Opaque\ndata:\n {{ if .Values.Master.AdminPassword }}\n jenkins-admin-password: {{ .Values.Master.AdminPassword | b64enc | quote }}\n {{ else }}\n jenkins-admin-password: {{ randAlphaNum 10 | b64enc | quote }}\n {{ end }}\n jenkins-admin-user: {{ .Values.Master.AdminUser | b64enc | quote }}\n{{- end }}"
},
{
"path": "manifests/deprecated/jenkins/templates/service-account.yaml",
"content": "{{ if .Values.rbac.install }}\n{{- $serviceName := include \"jenkins.fullname\" . -}}\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: {{ $serviceName }}\n labels:\n app: {{ $serviceName }}\n chart: \"{{ .Chart.Name }}-{{ .Chart.Version }}\"\n release: \"{{ .Release.Name }}\"\n heritage: \"{{ .Release.Service }}\"\n{{ end }}"
},
{
"path": "manifests/deprecated/jenkins/templates/test-config.yaml",
"content": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"jenkins.fullname\" . }}-tests\ndata:\n run.sh: |-\n @test \"Testing Jenkins UI is accessible\" {\n curl --retry 48 --retry-delay 10 {{ template \"jenkins.fullname\" . }}:{{ .Values.Master.ServicePort }}{{ default \"\" .Values.Master.JenkinsUriPrefix }}/login\n }\n"
},
{
"path": "manifests/deprecated/jenkins/values.yaml",
"content": "# Default values for jenkins.\n# This is a YAML-formatted file.\n# Declare name/value pairs to be passed into your templates.\n# name: value\n\n## Overrides for generated resource names\n# See templates/_helpers.tpl\n# nameOverride:\n# fullnameOverride:\n\nMaster:\n Name: jenkins-master\n Image: \"jenkins/jenkins\"\n ImageTag: \"2.138.2-alpine\"\n ImagePullPolicy: \"IfNotPresent\"\n# ImagePullSecret: jenkins\n Component: \"jenkins-master\"\n UseSecurity: true\n AdminUser: admin\n AdminPassword: admin\n resources:\n requests:\n cpu: \"50m\"\n memory: \"256Mi\"\n limits:\n cpu: \"2000m\"\n memory: \"2048Mi\"\n # Environment variables that get added to the init container (useful for e.g. http_proxy)\n # InitContainerEnv:\n # - name: http_proxy\n # value: \"http://192.168.64.1:3128\"\n # ContainerEnv:\n # - name: http_proxy\n # value: \"http://192.168.64.1:3128\"\n # Set min/max heap here if needed with:\n # JavaOpts: \"-Xms512m -Xmx512m\"\n # JenkinsOpts: \"\"\n # JenkinsUriPrefix: \"/jenkins\"\n\n # Enable pod security context (must be `true` if RunAsUser or FsGroup are set)\n # UsePodSecurityContext: true\n\n # Set RunAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.\n # When setting RunAsUser to a different value than 0 also set FsGroup to the same value:\n # RunAsUser: \n # FsGroup: \n ServicePort: 8080\n # For minikube, set this to NodePort, elsewhere use LoadBalancer\n # Use ClusterIP if your setup includes ingress controller\n ServiceType: ClusterIP\n # Master Service annotations\n ServiceAnnotations: {}\n # service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https\n # Used to create Ingress record (should used with ServiceType: ClusterIP)\n HostName: jenkins.local.com\n # NodePort: \n -Djava.awt.headless=true\n -Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai\n -Dfile.encoding=UTF-8\n # -Dcom.sun.management.jmxremote.port=4000\n # -Dcom.sun.management.jmxremote.authenticate=false\n # -Dcom.sun.management.jmxremote.ssl=false\n # JMXPort: 4000\n\n # 插件镜像地址\n UpdateCenter: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/current/update-center.json\n\n # List of plugins to be install during Jenkins master start\n InstallPlugins:\n - kubernetes:1.13.5\n - workflow-aggregator:2.5\n - workflow-job:2.25\n - credentials-binding:1.17\n - git:3.9.1\n - gitlab:1.5.10\n # Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval\n # ScriptApproval:\n # - \"method groovy.json.JsonSlurperClassic parseText java.lang.String\"\n # - \"new groovy.json.JsonSlurperClassic\"\n # List of groovy init scripts to be executed during Jenkins master start\n InitScripts:\n # - |\n # print 'adding global pipeline libraries, register properties, bootstrap jobs...'\n # Kubernetes secret that contains a 'credentials.xml' for Jenkins\n # CredentialsXmlSecret: jenkins-credentials\n # Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,\n # useful to manage encryption keys used for credentials.xml for instance (such as\n # master.key and hudson.util.Secret)\n # SecretsFilesSecret: jenkins-secrets\n # Jenkins XML job configs to provision\n # Jobs: |-\n # test: |-\n # <>\n CustomConfigMap: false\n # Node labels and tolerations for pod assignment\n # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector\n # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature\n NodeSelector: {}\n\n Tolerations: {}\n PodAnnotations: {}\n\n Ingress:\n ApiVersion: networking.k8s.io/v1beta1 \n Annotations:\n # kubernetes.io/ingress.class: nginx\n # kubernetes.io/tls-acme: \"true\"\n\n TLS:\n # - secretName: jenkins.cluster.local\n # hosts:\n # - jenkins.cluster.local\n\nAgent:\n Enabled: true\n Image: jenkinsci/jnlp-slave\n ImageTag: alpine\n# ImagePullSecret: jenkins\n Component: \"jenkins-slave\"\n Privileged: false\n resources:\n requests:\n cpu: \"200m\"\n memory: \"256Mi\"\n limits:\n cpu: \"200m\"\n memory: \"256Mi\"\n # You may want to change this to true while testing a new image\n AlwaysPullImage: false\n # You can define the volumes that you want to mount for this container\n # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret\n # Configure the attributes as they appear in the corresponding Java class for that type\n # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes\n volumes:\n # - type: Secret\n # secretName: mysecret\n # mountPath: /var/myapp/mysecret\n NodeSelector: {}\n # Key Value selectors. Ex:\n # jenkins-agent: v1\n\nPersistence:\n Enabled: true\n ## A manually managed Persistent Volume and Claim\n ## Requires Persistence.Enabled: true\n ## If defined, PVC must be created manually before volume will be bound\n # ExistingClaim:\n\n ## jenkins data Persistent Volume Storage Class\n ## If defined, storageClassName: \n ## If set to \"-\", storageClassName: \"\", which disables dynamic provisioning\n ## If undefined (the default) or set to null, no storageClassName spec is\n ## set, choosing the default provisioner. (gp2 on AWS, standard on\n ## GKE, AWS & OpenStack)\n ##\n StorageClass: \"nfs-dynamic-class\"\n\n Annotations: {}\n AccessMode: ReadWriteOnce\n Size: 8Gi\n volumes:\n # - name: nothing\n # emptyDir: {}\n mounts:\n # - mountPath: /var/nothing\n # name: nothing\n # readOnly: true\n\nNetworkPolicy:\n # Enable creation of NetworkPolicy resources.\n Enabled: false\n # For Kubernetes v1.7, use 'networking.k8s.io/v1'\n ApiVersion: networking.k8s.io/v1 \n\n## Install Default RBAC roles and bindings\nrbac:\n install: true\n serviceAccountName: default\n # RBAC api version (currently either v1beta1 or v1alpha1 or v1)\n apiVersion: v1\n # Cluster role reference\n roleRef: cluster-admin\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/.helmignore",
"content": ".git\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/Chart.yaml",
"content": "name: mariadb\nversion: 5.5.0\nappVersion: 10.1.37\ndescription: Fast, reliable, scalable, and easy to use open-source relational database system. MariaDB Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software. Highly available MariaDB cluster.\nkeywords:\n- mariadb\n- mysql\n- database\n- sql\n- prometheus\nhome: https://mariadb.org\nicon: https://bitnami.com/assets/stacks/mariadb/img/mariadb-stack-220x234.png\nsources:\n- https://github.com/bitnami/bitnami-docker-mariadb\n- https://github.com/prometheus/mysqld_exporter\nmaintainers:\n- name: Bitnami\n email: containers@bitnami.com\nengine: gotpl\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/OWNERS",
"content": "approvers:\n- prydonius\n- tompizmor\n- sameersbn\n- carrodher\n- juan131\nreviewers:\n- prydonius\n- tompizmor\n- sameersbn\n- carrodher\n- juan131\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/README.md",
"content": "# MariaDB\n\n[MariaDB](https://mariadb.org) is one of the most popular database servers in the world. It’s made by the original developers of MySQL and guaranteed to stay open source. Notable users include Wikipedia, Facebook and Google.\n\nMariaDB is developed as open source software and as a relational database it provides an SQL interface for accessing data. The latest versions of MariaDB also include GIS and JSON features.\n\n## TL;DR\n\n```bash\n$ helm install stable/mariadb\n```\n\n## Introduction\n\nThis chart bootstraps a [MariaDB](https://github.com/bitnami/bitnami-docker-mariadb) replication cluster deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.\n\nBitnami charts can be used with [Kubeapps](https://kubeapps.com/) for deployment and management of Helm Charts in clusters.\n\n## Prerequisites\n\n- Kubernetes 1.10+\n- PV provisioner support in the underlying infrastructure\n\n## Installing the Chart\n\nTo install the chart with the release name `my-release`:\n\n```bash\n$ helm install --name my-release stable/mariadb\n```\n\nThe command deploys MariaDB on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.\n\n> **Tip**: List all releases using `helm list`\n\n## Uninstalling the Chart\n\nTo uninstall/delete the `my-release` deployment:\n\n```bash\n$ helm delete my-release\n```\n\nThe command removes all the Kubernetes components associated with the chart and deletes the release.\n\n## Configuration\n\nThe following table lists the configurable parameters of the MariaDB chart and their default values.\n\n| Parameter | Description | Default |\n|-------------------------------------------|-----------------------------------------------------|-------------------------------------------------------------------|\n| `global.imageRegistry` | Global Docker image registry | `nil` |\n| `image.registry` | MariaDB image registry | `docker.io` |\n| `image.repository` | MariaDB Image name | `bitnami/mariadb` |\n| `image.tag` | MariaDB Image tag | `{VERSION}` |\n| `image.pullPolicy` | MariaDB image pull policy | `Always` if `imageTag` is `latest`, else `IfNotPresent` |\n| `image.pullSecrets` | Specify docker-registry secret names as an array | `[]` (does not add image pull secrets to deployed pods) |\n| `image.debug` | Specify if debug logs should be enabled | `false` |\n| `service.type` | Kubernetes service type | `ClusterIP` |\n| `service.clusterIp` | Specific cluster IP when service type is cluster IP. Use None for headless service | `nil` |\n| `service.port` | MySQL service port | `3306` |\n| `serviceAccount.create` | Specifies whether a ServiceAccount should be created | `false` |\n| `serviceAccount.name` | The name of the ServiceAccount to create | Generated using the mariadb.fullname template |\n| `securityContext.enabled` | Enable security context | `true` |\n| `securityContext.fsGroup` | Group ID for the container | `1001` |\n| `securityContext.runAsUser` | User ID for the container | `1001` |\n| `existingSecret` | Use Existing secret for Password details (`rootUser.password`, `db.password`, `replication.password` will be ignored and picked up from this secret) | |\n| `rootUser.password` | Password for the `root` user. Ignored if existing secret is provided. | _random 10 character alphanumeric string_ |\n| `rootUser.forcePassword` | Force users to specify a password | `false` |\n| `db.user` | Username of new user to create | `nil` |\n| `db.password` | Password for the new user. Ignored if existing secret is provided. | _random 10 character alphanumeric string if `db.user` is defined_ |\n| `db.name` | Name for new database to create | `my_database` |\n| `replication.enabled` | MariaDB replication enabled | `true` |\n| `replication.user` |MariaDB replication user | `replicator` |\n| `replication.password` | MariaDB replication user password. Ignored if existing secret is provided. | _random 10 character alphanumeric string_ |\n| `initdbScripts` | List of initdb scripts | `nil` |\n| `initdbScriptsConfigMap` | ConfigMap with the initdb scripts (Note: Overrides `initdbScripts`) | `nil` |\n| `master.annotations[].key` | key for the the annotation list item | `nil` |\n| `master.annotations[].value` | value for the the annotation list item | `nil` |\n| `master.affinity` | Master affinity (in addition to master.antiAffinity when set) | `{}` |\n| `master.antiAffinity` | Master pod anti-affinity policy | `soft` |\n| `master.tolerations` | List of node taints to tolerate (master) | `[]` |\n| `master.persistence.enabled` | Enable persistence using PVC | `true` |\n| `master.persistence.existingClaim` | Provide an existing `PersistentVolumeClaim` | `nil` |\n| `master.persistence.mountPath` | Path to mount the volume at | `/bitnami/mariadb` |\n| `master.persistence.annotations` | Persistent Volume Claim annotations | `{}` |\n| `master.persistence.storageClass` | Persistent Volume Storage Class | `` |\n| `master.persistence.accessModes` | Persistent Volume Access Modes | `[ReadWriteOnce]` |\n| `master.persistence.size` | Persistent Volume Size | `8Gi` |\n| `master.extraInitContainers` | Additional init containers as a string to be passed to the `tpl` function (master) | |\n| `master.config` | Config file for the MariaDB Master server | `_default values in the values.yaml file_` |\n| `master.resources` | CPU/Memory resource requests/limits for master node | `{}` |\n| `master.livenessProbe.enabled` | Turn on and off liveness probe (master) | `true` |\n| `master.livenessProbe.initialDelaySeconds`| Delay before liveness probe is initiated (master) | `120` |\n| `master.livenessProbe.periodSeconds` | How often to perform the probe (master) | `10` |\n| `master.livenessProbe.timeoutSeconds` | When the probe times out (master) | `1` |\n| `master.livenessProbe.successThreshold` | Minimum consecutive successes for the probe (master)| `1` |\n| `master.livenessProbe.failureThreshold` | Minimum consecutive failures for the probe (master) | `3` |\n| `master.readinessProbe.enabled` | Turn on and off readiness probe (master) | `true` |\n| `master.readinessProbe.initialDelaySeconds`| Delay before readiness probe is initiated (master) | `30` |\n| `master.readinessProbe.periodSeconds` | How often to perform the probe (master) | `10` |\n| `master.readinessProbe.timeoutSeconds` | When the probe times out (master) | `1` |\n| `master.readinessProbe.successThreshold` | Minimum consecutive successes for the probe (master)| `1` |\n| `master.readinessProbe.failureThreshold` | Minimum consecutive failures for the probe (master) | `3` |\n| `slave.replicas` | Desired number of slave replicas | `1` |\n| `slave.annotations[].key` | key for the the annotation list item | `nil` |\n| `slave.annotations[].value` | value for the the annotation list item | `nil` |\n| `slave.affinity` | Slave affinity (in addition to slave.antiAffinity when set) | `{}` |\n| `slave.antiAffinity` | Slave pod anti-affinity policy | `soft` |\n| `slave.tolerations` | List of node taints to tolerate for (slave) | `[]` |\n| `slave.persistence.enabled` | Enable persistence using a `PersistentVolumeClaim` | `true` |\n| `slave.persistence.annotations` | Persistent Volume Claim annotations | `{}` |\n| `slave.persistence.storageClass` | Persistent Volume Storage Class | `` |\n| `slave.persistence.accessModes` | Persistent Volume Access Modes | `[ReadWriteOnce]` |\n| `slave.persistence.size` | Persistent Volume Size | `8Gi` |\n| `slave.extraInitContainers` | Additional init containers as a string to be passed to the `tpl` function (slave) | |\n| `slave.config` | Config file for the MariaDB Slave replicas | `_default values in the values.yaml file_` |\n| `slave.resources` | CPU/Memory resource requests/limits for slave node | `{}` |\n| `slave.livenessProbe.enabled` | Turn on and off liveness probe (slave) | `true` |\n| `slave.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated (slave) | `120` |\n| `slave.livenessProbe.periodSeconds` | How often to perform the probe (slave) | `10` |\n| `slave.livenessProbe.timeoutSeconds` | When the probe times out (slave) | `1` |\n| `slave.livenessProbe.successThreshold` | Minimum consecutive successes for the probe (slave) | `1` |\n| `slave.livenessProbe.failureThreshold` | Minimum consecutive failures for the probe (slave) | `3` |\n| `slave.readinessProbe.enabled` | Turn on and off readiness probe (slave) | `true` |\n| `slave.readinessProbe.initialDelaySeconds`| Delay before readiness probe is initiated (slave) | `45` |\n| `slave.readinessProbe.periodSeconds` | How often to perform the probe (slave) | `10` |\n| `slave.readinessProbe.timeoutSeconds` | When the probe times out (slave) | `1` |\n| `slave.readinessProbe.successThreshold` | Minimum consecutive successes for the probe (slave) | `1` |\n| `slave.readinessProbe.failureThreshold` | Minimum consecutive failures for the probe (slave) | `3` |\n| `metrics.enabled` | Start a side-car prometheus exporter | `false` |\n| `metrics.image.registry` | Exporter image registry | `docker.io` |\n| `metrics.image.repository` | Exporter image name | `prom/mysqld-exporter` |\n| `metrics.image.tag` | Exporter image tag | `v0.10.0` |\n| `metrics.image.pullPolicy` | Exporter image pull policy | `IfNotPresent` |\n| `metrics.resources` | Exporter resource requests/limit | `nil` |\n\nThe above parameters map to the env variables defined in [bitnami/mariadb](http://github.com/bitnami/bitnami-docker-mariadb). For more information please refer to the [bitnami/mariadb](http://github.com/bitnami/bitnami-docker-mariadb) image documentation.\n\nSpecify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,\n\n```bash\n$ helm install --name my-release \\\n --set root.password=secretpassword,user.database=app_database \\\n stable/mariadb\n```\n\nThe above command sets the MariaDB `root` account password to `secretpassword`. Additionally it creates a database named `my_database`.\n\nAlternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,\n\n```bash\n$ helm install --name my-release -f values.yaml stable/mariadb\n```\n\n> **Tip**: You can use the default [values.yaml](values.yaml)\n\n## Initialize a fresh instance\n\nThe [Bitnami MariaDB](https://github.com/bitnami/bitnami-docker-mariadb) image allows you to use your custom scripts to initialize a fresh instance. In order to execute the scripts, they must be located inside the chart folder `files/docker-entrypoint-initdb.d` so they can be consumed as a ConfigMap.\n\nAlternatively, you can specify custom scripts using the `initdbScripts` parameter as dict.\n\nIn addition to these options, you can also set an external ConfigMap with all the initialization scripts. This is done by setting the `initdbScriptsConfigMap` parameter. Note that this will override the two previous options.\n\nThe allowed extensions are `.sh`, `.sql` and `.sql.gz`.\n\n## Persistence\n\nThe [Bitnami MariaDB](https://github.com/bitnami/bitnami-docker-mariadb) image stores the MariaDB data and configurations at the `/bitnami/mariadb` path of the container.\n\nThe chart mounts a [Persistent Volume](kubernetes.io/docs/user-guide/persistent-volumes/) volume at this location. The volume is created using dynamic volume provisioning, by default. An existing PersistentVolumeClaim can be defined.\n\n## Extra Init Containers\n\nThe feature allows for specifying a template string for a initContainer in the master/slave pod. Usecases include situations when you need some pre-run setup. For example, in IKS (IBM Cloud Kubernetes Service), non-root users do not have write permission on the volume mount path for NFS-powered file storage. So, you could use a initcontainer to `chown` the mount. See a example below, where we add an initContainer on the master pod that reports to an external resource that the db is going to starting.\n`values.yaml`\n```yaml\nmaster:\n extraInitContainers: |\n - name: initcontainer\n image: alpine:latest\n command: [\"/bin/sh\", \"-c\"]\n args:\n - curl http://api-service.local/db/starting;\n```\n\n## Upgrading\n\nIt's necessary to set the `rootUser.password` parameter when upgrading for readiness/liveness probes to work properly. When you install this chart for the first time, some notes will be displayed providing the credentials you must use under the 'Administrator credentials' section. Please note down the password and run the command below to upgrade your chart:\n\n```bash\n$ helm upgrade my-release stable/mariadb --set rootUser.password=[ROOT_PASSWORD]\n```\n\n| Note: you need to substitute the placeholder _[ROOT_PASSWORD]_ with the value obtained in the installation notes.\n\n### To 5.0.0\n\nBackwards compatibility is not guaranteed unless you modify the labels used on the chart's deployments.\nUse the workaround below to upgrade from versions previous to 5.0.0. The following example assumes that the release name is mariadb:\n\n```console\n$ kubectl delete statefulset opencart-mariadb --cascade=false\n```\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/files/docker-entrypoint-initdb.d/README.md",
"content": "You can copy here your custom .sh, .sql or .sql.gz file so they are executed during the first boot of the image.\n\nMore info in the [bitnami-docker-mariadb](https://github.com/bitnami/bitnami-docker-mariadb#initializing-a-new-instance) repository."
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/NOTES.txt",
"content": "\nPlease be patient while the chart is being deployed\n\nTip:\n\n Watch the deployment status using the command: kubectl get pods -w --namespace {{ .Release.Namespace }} -l release={{ .Release.Name }}\n\nServices:\n\n echo Master: {{ template \"mariadb.fullname\" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.service.port }}\n{{- if .Values.replication.enabled }}\n echo Slave: {{ template \"slave.fullname\" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.service.port }}\n{{- end }}\n\nAdministrator credentials:\n\n Username: root\n Password : $(kubectl get secret --namespace {{ .Release.Namespace }} {{ template \"mariadb.fullname\" . }} -o jsonpath=\"{.data.mariadb-root-password}\" | base64 --decode)\n\nTo connect to your database:\n\n 1. Run a pod that you can use as a client:\n\n kubectl run {{ template \"mariadb.fullname\" . }}-client --rm --tty -i --restart='Never' --image {{ template \"mariadb.image\" . }} --namespace {{ .Release.Namespace }} --command -- bash\n\n 2. To connect to master service (read/write):\n\n mysql -h {{ template \"mariadb.fullname\" . }}.{{ .Release.Namespace }}.svc.cluster.local -uroot -p {{ .Values.db.name }}\n\n{{- if .Values.replication.enabled }}\n\n 3. To connect to slave service (read-only):\n\n mysql -h {{ template \"slave.fullname\" . }}.{{ .Release.Namespace }}.svc.cluster.local -uroot -p {{ .Values.db.name }}\n{{- end }}\n\nTo upgrade this helm chart:\n\n 1. Obtain the password as described on the 'Administrator credentials' section and set the 'rootUser.password' parameter as shown below:\n\n ROOT_PASSWORD=$(kubectl get secret --namespace {{ .Release.Namespace }} {{ template \"mariadb.fullname\" . }} -o jsonpath=\"{.data.mariadb-root-password}\" | base64 --decode)\n helm upgrade {{ .Release.Name }} stable/mariadb --set rootUser.password=$ROOT_PASSWORD\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/_helpers.tpl",
"content": "{{/* vim: set filetype=mustache: */}}\n{{/*\nExpand the name of the chart.\n*/}}\n{{- define \"mariadb.name\" -}}\n{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n\n{{/*\nCreate a default fully qualified app name.\nWe truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).\nIf release name contains chart name it will be used as a full name.\n*/}}\n{{- define \"mariadb.fullname\" -}}\n{{- if .Values.fullnameOverride -}}\n{{- .Values.fullnameOverride | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- $name := default .Chart.Name .Values.nameOverride -}}\n{{- if contains $name .Release.Name -}}\n{{- printf .Release.Name | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- printf \"%s-%s\" .Release.Name $name | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n{{- end -}}\n{{- end -}}\n\n{{- define \"master.fullname\" -}}\n{{- if .Values.replication.enabled -}}\n{{- printf \"%s-%s\" .Release.Name \"mariadb-master\" | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- printf \"%s-%s\" .Release.Name \"mariadb\" | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n{{- end -}}\n\n{{- define \"slave.fullname\" -}}\n{{- printf \"%s-%s\" .Release.Name \"mariadb-slave\" | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n\n{{- define \"mariadb.chart\" -}}\n{{- printf \"%s-%s\" .Chart.Name .Chart.Version | replace \"+\" \"_\" | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n\n{{/*\nReturn the proper MariaDB image name\n*/}}\n{{- define \"mariadb.image\" -}}\n{{- $registryName := .Values.image.registry -}}\n{{- $repositoryName := .Values.image.repository -}}\n{{- $tag := .Values.image.tag | toString -}}\n{{/*\nHelm 2.11 supports the assignment of a value to a variable defined in a different scope,\nbut Helm 2.9 and 2.10 doesn't support it, so we need to implement this if-else logic.\nAlso, we can't use a single if because lazy evaluation is not an option\n*/}}\n{{- if .Values.global }}\n {{- if .Values.global.imageRegistry }}\n {{- printf \"%s/%s:%s\" .Values.global.imageRegistry $repositoryName $tag -}}\n {{- else -}}\n {{- printf \"%s/%s:%s\" $registryName $repositoryName $tag -}}\n {{- end -}}\n{{- else -}}\n {{- printf \"%s/%s:%s\" $registryName $repositoryName $tag -}}\n{{- end -}}\n{{- end -}}\n\n{{/*\nReturn the proper metrics image name\n*/}}\n{{- define \"metrics.image\" -}}\n{{- $registryName := .Values.metrics.image.registry -}}\n{{- $repositoryName := .Values.metrics.image.repository -}}\n{{- $tag := .Values.metrics.image.tag | toString -}}\n{{- printf \"%s/%s:%s\" $registryName $repositoryName $tag -}}\n{{- end -}}\n\n{{ template \"mariadb.initdbScriptsCM\" . }}\n{{/*\nGet the initialization scripts ConfigMap name.\n*/}}\n{{- define \"mariadb.initdbScriptsCM\" -}}\n{{- if .Values.initdbScriptsConfigMap -}}\n{{- printf \"%s\" .Values.initdbScriptsConfigMap -}}\n{{- else -}}\n{{- printf \"%s-init-scripts\" (include \"mariadb.fullname\" .) -}}\n{{- end -}}\n{{- end -}}\n\n{{/*\nCreate the name of the service account to use\n*/}}\n{{- define \"mariadb.serviceAccountName\" -}}\n{{- if .Values.serviceAccount.create -}}\n {{ default (include \"mariadb.fullname\" .) .Values.serviceAccount.name }}\n{{- else -}}\n {{ default \"default\" .Values.serviceAccount.name }}\n{{- end -}}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/initialization-configmap.yaml",
"content": "{{- if and (or (.Files.Glob \"files/docker-entrypoint-initdb.d/*.{sh,sql,sql.gz}\") .Values.initdbScripts) (not .Values.initdbScriptsConfigMap) }}\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"master.fullname\" . }}-init-scripts\n labels:\n app: {{ template \"mariadb.name\" . }}\n chart: {{ template \"mariadb.chart\" . }}\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\n component: \"master\"\n{{- if and (.Files.Glob \"files/docker-entrypoint-initdb.d/*.sql.gz\") (not .Values.initdbScriptsConfigMap) }}\nbinaryData:\n{{- $root := . }}\n{{- range $path, $bytes := .Files.Glob \"files/docker-entrypoint-initdb.d/*.sql.gz\" }}\n {{ base $path }}: {{ $root.Files.Get $path | b64enc | quote }}\n{{- end }}\n{{- end }}\ndata:\n{{- if and (.Files.Glob \"files/docker-entrypoint-initdb.d/*.{sh,sql}\") (not .Values.initdbScriptsConfigMap) }}\n{{ (.Files.Glob \"files/docker-entrypoint-initdb.d/*.{sh,sql}\").AsConfig | indent 2 }}\n{{- end }}\n{{- with .Values.initdbScripts }}\n{{ toYaml . | indent 2 }}\n{{- end }}\n{{ end }}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/master-configmap.yaml",
"content": "{{- if .Values.master.config }}\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"master.fullname\" . }}\n labels:\n app: {{ template \"mariadb.name\" . }}\n component: \"master\"\n chart: {{ template \"mariadb.chart\" . }}\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\ndata:\n my.cnf: |-\n{{ .Values.master.config | indent 4 }}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/master-statefulset.yaml",
"content": "apiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: {{ template \"master.fullname\" . }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n chart: {{ template \"mariadb.chart\" . }}\n component: \"master\"\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\nspec:\n selector:\n matchLabels:\n release: \"{{ .Release.Name }}\"\n component: \"master\"\n app: {{ template \"mariadb.name\" . }}\n serviceName: \"{{ template \"master.fullname\" . }}\"\n replicas: 1\n updateStrategy:\n type: RollingUpdate\n template:\n metadata:\n {{- if .Values.master.annotations }}\n annotations:\n {{- range .Values.master.annotations }}\n {{ .key }}: '{{ .value }}'\n {{- end }}\n {{- end }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"master\"\n release: \"{{ .Release.Name }}\"\n chart: {{ template \"mariadb.chart\" . }}\n spec:\n serviceAccountName: \"{{ template \"mariadb.serviceAccountName\" . }}\"\n {{- if .Values.securityContext.enabled }}\n securityContext:\n fsGroup: {{ .Values.securityContext.fsGroup }}\n runAsUser: {{ .Values.securityContext.runAsUser }}\n {{- end }}\n {{- if eq .Values.master.antiAffinity \"hard\" }}\n affinity:\n {{- with .Values.master.affinity }}\n{{ toYaml . | indent 8 }}\n {{- end }}\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - topologyKey: \"kubernetes.io/hostname\"\n labelSelector:\n matchLabels:\n app: \"{{ template \"mariadb.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n {{- else if eq .Values.master.antiAffinity \"soft\" }}\n affinity:\n {{- with .Values.master.affinity }}\n{{ toYaml . | indent 8 }}\n {{- end }}\n podAntiAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 1\n podAffinityTerm:\n topologyKey: kubernetes.io/hostname\n labelSelector:\n matchLabels:\n app: \"{{ template \"mariadb.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n {{- else}}\n {{- with .Values.master.affinity }}\n affinity:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- end }}\n {{- with .Values.master.tolerations }}\n tolerations:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- if .Values.image.pullSecrets }}\n imagePullSecrets:\n {{- range .Values.image.pullSecrets }}\n - name: {{ . }}\n {{- end}}\n {{- end }}\n {{- if .Values.master.extraInitContainers }}\n initContainers:\n{{ tpl .Values.master.extraInitContainers . | indent 6}}\n {{- end }}\n containers:\n - name: \"mariadb\"\n image: {{ template \"mariadb.image\" . }}\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n env:\n {{- if .Values.image.debug}}\n - name: BITNAMI_DEBUG\n value: \"true\"\n {{- end }}\n - name: MARIADB_ROOT_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-root-password\n {{- if .Values.db.user }}\n - name: MARIADB_USER\n value: \"{{ .Values.db.user }}\"\n - name: MARIADB_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-password\n {{- end }}\n - name: MARIADB_DATABASE\n value: \"{{ .Values.db.name }}\"\n {{- if .Values.replication.enabled }}\n - name: MARIADB_REPLICATION_MODE\n value: \"master\"\n - name: MARIADB_REPLICATION_USER\n value: \"{{ .Values.replication.user }}\"\n - name: MARIADB_REPLICATION_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-replication-password\n {{- end }}\n ports:\n - name: mysql\n containerPort: 3306\n {{- if .Values.master.livenessProbe.enabled }}\n livenessProbe:\n exec:\n command: [\"sh\", \"-c\", \"exec mysqladmin status -uroot -p$MARIADB_ROOT_PASSWORD\"]\n initialDelaySeconds: {{ .Values.master.livenessProbe.initialDelaySeconds }}\n periodSeconds: {{ .Values.master.livenessProbe.periodSeconds }}\n timeoutSeconds: {{ .Values.master.livenessProbe.timeoutSeconds }}\n successThreshold: {{ .Values.master.livenessProbe.successThreshold }}\n failureThreshold: {{ .Values.master.livenessProbe.failureThreshold }}\n {{- end }}\n {{- if .Values.master.readinessProbe.enabled }}\n readinessProbe:\n exec:\n command: [\"sh\", \"-c\", \"exec mysqladmin status -uroot -p$MARIADB_ROOT_PASSWORD\"]\n initialDelaySeconds: {{ .Values.master.readinessProbe.initialDelaySeconds }}\n periodSeconds: {{ .Values.master.readinessProbe.periodSeconds }}\n timeoutSeconds: {{ .Values.master.readinessProbe.timeoutSeconds }}\n successThreshold: {{ .Values.master.readinessProbe.successThreshold }}\n failureThreshold: {{ .Values.master.readinessProbe.failureThreshold }}\n {{- end }}\n resources:\n{{ toYaml .Values.master.resources | indent 10 }}\n volumeMounts:\n - name: data\n mountPath: {{ .Values.master.persistence.mountPath }}\n {{- if or (.Files.Glob \"files/docker-entrypoint-initdb.d/*.{sh,sql,sql.gz}\") .Values.initdbScriptsConfigMap .Values.initdbScripts }}\n - name: custom-init-scripts\n mountPath: /docker-entrypoint-initdb.d\n {{- end }}\n {{- if .Values.master.config }}\n - name: config\n mountPath: /opt/bitnami/mariadb/conf/my.cnf\n subPath: my.cnf\n {{- end }}\n{{- if .Values.metrics.enabled }}\n - name: metrics\n image: {{ template \"metrics.image\" . }}\n imagePullPolicy: {{ .Values.metrics.image.pullPolicy | quote }}\n env:\n - name: MARIADB_ROOT_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-root-password\n command: [ 'sh', '-c', 'DATA_SOURCE_NAME=\"root:$MARIADB_ROOT_PASSWORD@(localhost:3306)/\" /bin/mysqld_exporter' ]\n ports:\n - name: metrics\n containerPort: 9104\n livenessProbe:\n httpGet:\n path: /metrics\n port: metrics\n initialDelaySeconds: 15\n timeoutSeconds: 5\n readinessProbe:\n httpGet:\n path: /metrics\n port: metrics\n initialDelaySeconds: 5\n timeoutSeconds: 1\n resources:\n{{ toYaml .Values.metrics.resources | indent 10 }}\n{{- end }}\n volumes:\n {{- if .Values.master.config }}\n - name: config\n configMap:\n name: {{ template \"master.fullname\" . }}\n {{- end }}\n {{- if or (.Files.Glob \"files/docker-entrypoint-initdb.d/*.{sh,sql,sql.gz}\") .Values.initdbScriptsConfigMap .Values.initdbScripts }}\n - name: custom-init-scripts\n configMap:\n name: {{ template \"mariadb.initdbScriptsCM\" . }}\n {{- end }}\n{{- if and .Values.master.persistence.enabled .Values.master.persistence.existingClaim }}\n - name: data\n persistentVolumeClaim:\n claimName: {{ .Values.master.persistence.existingClaim }}\n{{- else if not .Values.master.persistence.enabled }}\n - name: data\n emptyDir: {}\n{{- else if and .Values.master.persistence.enabled (not .Values.master.persistence.existingClaim) }}\n volumeClaimTemplates:\n - metadata:\n name: data\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"master\"\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\n spec:\n accessModes:\n {{- range .Values.master.persistence.accessModes }}\n - {{ . | quote }}\n {{- end }}\n resources:\n requests:\n storage: {{ .Values.master.persistence.size | quote }}\n {{- if .Values.master.persistence.storageClass }}\n {{- if (eq \"-\" .Values.master.persistence.storageClass) }}\n storageClassName: \"\"\n {{- else }}\n storageClassName: {{ .Values.master.persistence.storageClass | quote }}\n {{- end }}\n {{- end }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/master-svc.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: {{ template \"mariadb.fullname\" . }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"master\"\n chart: {{ template \"mariadb.chart\" . }}\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\n{{- if .Values.metrics.enabled }}\n annotations:\n{{ toYaml .Values.metrics.annotations | indent 4 }}\n{{- end }}\nspec:\n type: {{ .Values.service.type }}\n {{- if eq .Values.service.type \"ClusterIP\" }}\n {{- if .Values.service.clusterIp }}\n clusterIP: {{ .Values.service.clusterIp }}\n {{- end }}\n {{- end }}\n ports:\n - name: mysql\n port: {{ .Values.service.port }}\n targetPort: mysql\n{{- if eq .Values.service.type \"NodePort\" }}\n{{- if .Values.service.nodePort }}\n{{- if .Values.service.nodePort.master }}\n nodePort: {{ .Values.service.nodePort.master }}\n{{- end }}\n{{- end }}\n{{- end }}\n{{- if .Values.metrics.enabled }}\n - name: metrics\n port: 9104\n targetPort: metrics\n{{- end }}\n selector:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"master\"\n release: \"{{ .Release.Name }}\"\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/secrets.yaml",
"content": "{{- if (not .Values.existingSecret) -}}\napiVersion: v1\nkind: Secret\nmetadata:\n name: {{ template \"mariadb.fullname\" . }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n chart: {{ template \"mariadb.chart\" . }}\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\ntype: Opaque\ndata:\n {{- if .Values.rootUser.password }}\n mariadb-root-password: \"{{ .Values.rootUser.password | b64enc }}\"\n {{- else if (not .Values.rootUser.forcePassword) }}\n mariadb-root-password: \"{{ randAlphaNum 10 | b64enc }}\"\n {{ else }}\n mariadb-root-password: {{ required \"A MariaDB Root Password is required!\" .Values.rootUser.password }}\n {{- end }}\n {{- if .Values.db.user }}\n {{- if .Values.db.password }}\n mariadb-password: \"{{ .Values.db.password | b64enc }}\"\n {{- else if (not .Values.db.forcePassword) }}\n mariadb-password: \"{{ randAlphaNum 10 | b64enc }}\"\n {{- else }}\n mariadb-password: {{ required \"A MariaDB Database Password is required!\" .Values.db.password }}\n {{- end }}\n {{- end }}\n {{- if .Values.replication.enabled }}\n {{- if .Values.replication.password }}\n mariadb-replication-password: \"{{ .Values.replication.password | b64enc }}\"\n {{- else if (not .Values.replication.forcePassword) }}\n mariadb-replication-password: \"{{ randAlphaNum 10 | b64enc }}\"\n {{- else }}\n mariadb-replication-password: {{ required \"A MariaDB Replication Password is required!\" .Values.replication.password }}\n {{- end }}\n {{- end }}\n{{- end }}"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/slave-configmap.yaml",
"content": "{{- if and .Values.replication.enabled .Values.slave.config }}\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"slave.fullname\" . }}\n labels:\n app: {{ template \"mariadb.name\" . }}\n component: \"slave\"\n chart: {{ template \"mariadb.chart\" . }}\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\ndata:\n my.cnf: |-\n{{ .Values.slave.config | indent 4 }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/slave-statefulset.yaml",
"content": "{{- if .Values.replication.enabled }}\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: {{ template \"slave.fullname\" . }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n chart: {{ template \"mariadb.chart\" . }}\n component: \"slave\"\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\nspec:\n selector:\n matchLabels:\n release: \"{{ .Release.Name }}\"\n component: \"slave\"\n app: {{ template \"mariadb.name\" . }}\n serviceName: \"{{ template \"slave.fullname\" . }}\"\n replicas: {{ .Values.slave.replicas }}\n updateStrategy:\n type: RollingUpdate\n template:\n metadata:\n {{- if .Values.slave.annotations }}\n annotations:\n {{- range .Values.slave.annotations }}\n {{ .key }}: '{{ .value }}'\n {{- end }}\n {{- end }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"slave\"\n release: \"{{ .Release.Name }}\"\n chart: {{ template \"mariadb.chart\" . }}\n spec:\n serviceAccountName: \"{{ template \"mariadb.serviceAccountName\" . }}\"\n {{- if .Values.securityContext.enabled }}\n securityContext:\n fsGroup: {{ .Values.securityContext.fsGroup }}\n runAsUser: {{ .Values.securityContext.runAsUser }}\n {{- end }}\n {{- if eq .Values.slave.antiAffinity \"hard\" }}\n affinity:\n {{- with .Values.slave.affinity }}\n{{ toYaml . | indent 8 }}\n {{- end }}\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - topologyKey: \"kubernetes.io/hostname\"\n labelSelector:\n matchLabels:\n app: \"{{ template \"mariadb.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n {{- else if eq .Values.slave.antiAffinity \"soft\" }}\n affinity:\n {{- with .Values.slave.affinity }}\n{{ toYaml . | indent 8 }}\n {{- end }}\n podAntiAffinity:\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 1\n podAffinityTerm:\n topologyKey: kubernetes.io/hostname\n labelSelector:\n matchLabels:\n app: \"{{ template \"mariadb.name\" . }}\"\n release: \"{{ .Release.Name }}\"\n {{- else}}\n {{- with .Values.slave.affinity }}\n affinity:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- end }}\n {{- with .Values.slave.tolerations }}\n tolerations:\n{{ toYaml . | indent 8 }}\n {{- end }}\n {{- if .Values.image.pullSecrets }}\n imagePullSecrets:\n {{- range .Values.image.pullSecrets }}\n - name: {{ . }}\n {{- end}}\n {{- end }}\n {{- if .Values.master.extraInitContainers }}\n initContainers:\n{{ tpl .Values.master.extraInitContainers . | indent 6}}\n {{- end }}\n containers:\n - name: \"mariadb\"\n image: {{ template \"mariadb.image\" . }}\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n env:\n {{- if .Values.image.debug}}\n - name: BITNAMI_DEBUG\n value: \"true\"\n {{- end }}\n - name: MARIADB_REPLICATION_MODE\n value: \"slave\"\n - name: MARIADB_MASTER_HOST\n value: {{ template \"mariadb.fullname\" . }}\n - name: MARIADB_MASTER_PORT_NUMBER\n value: \"3306\"\n - name: MARIADB_MASTER_ROOT_USER\n value: \"root\"\n - name: MARIADB_MASTER_ROOT_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-root-password\n - name: MARIADB_REPLICATION_USER\n value: \"{{ .Values.replication.user }}\"\n - name: MARIADB_REPLICATION_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-replication-password\n ports:\n - name: mysql\n containerPort: 3306\n {{- if .Values.slave.livenessProbe.enabled }}\n livenessProbe:\n exec:\n command: [\"sh\", \"-c\", \"exec mysqladmin status -uroot -p$MARIADB_MASTER_ROOT_PASSWORD\"]\n initialDelaySeconds: {{ .Values.slave.livenessProbe.initialDelaySeconds }}\n periodSeconds: {{ .Values.slave.livenessProbe.periodSeconds }}\n timeoutSeconds: {{ .Values.slave.livenessProbe.timeoutSeconds }}\n successThreshold: {{ .Values.slave.livenessProbe.successThreshold }}\n failureThreshold: {{ .Values.slave.livenessProbe.failureThreshold }}\n {{- end }}\n {{- if .Values.slave.readinessProbe.enabled }}\n readinessProbe:\n exec:\n command: [\"sh\", \"-c\", \"exec mysqladmin status -uroot -p$MARIADB_MASTER_ROOT_PASSWORD\"]\n initialDelaySeconds: {{ .Values.slave.readinessProbe.initialDelaySeconds }}\n periodSeconds: {{ .Values.slave.readinessProbe.periodSeconds }}\n timeoutSeconds: {{ .Values.slave.readinessProbe.timeoutSeconds }}\n successThreshold: {{ .Values.slave.readinessProbe.successThreshold }}\n failureThreshold: {{ .Values.slave.readinessProbe.failureThreshold }}\n {{- end }}\n resources:\n{{ toYaml .Values.slave.resources | indent 10 }}\n volumeMounts:\n - name: data\n mountPath: /bitnami/mariadb\n{{- if .Values.slave.config }}\n - name: config\n mountPath: /opt/bitnami/mariadb/conf/my.cnf\n subPath: my.cnf\n{{- end }}\n{{- if .Values.metrics.enabled }}\n - name: metrics\n image: {{ template \"metrics.image\" . }}\n imagePullPolicy: {{ .Values.metrics.image.pullPolicy | quote }}\n env:\n - name: MARIADB_MASTER_ROOT_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-root-password\n command: [ 'sh', '-c', 'DATA_SOURCE_NAME=\"root:$MARIADB_MASTER_ROOT_PASSWORD@(localhost:3306)/\" /bin/mysqld_exporter' ]\n ports:\n - name: metrics\n containerPort: 9104\n livenessProbe:\n httpGet:\n path: /metrics\n port: metrics\n initialDelaySeconds: 15\n timeoutSeconds: 5\n readinessProbe:\n httpGet:\n path: /metrics\n port: metrics\n initialDelaySeconds: 5\n timeoutSeconds: 1\n resources:\n{{ toYaml .Values.metrics.resources | indent 10 }}\n{{- end }}\n volumes:\n {{- if .Values.slave.config }}\n - name: config\n configMap:\n name: {{ template \"slave.fullname\" . }}\n {{- end }}\n{{- if .Values.slave.persistence.enabled }}\n volumeClaimTemplates:\n - metadata:\n name: data\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"slave\"\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\n spec:\n accessModes:\n {{- range .Values.slave.persistence.accessModes }}\n - {{ . | quote }}\n {{- end }}\n resources:\n requests:\n storage: {{ .Values.slave.persistence.size | quote }}\n {{- if .Values.slave.persistence.storageClass }}\n {{- if (eq \"-\" .Values.slave.persistence.storageClass) }}\n storageClassName: \"\"\n {{- else }}\n storageClassName: {{ .Values.slave.persistence.storageClass | quote }}\n {{- end }}\n {{- end }}\n{{- else }}\n - name: \"data\"\n emptyDir: {}\n{{- end }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/slave-svc.yaml",
"content": "{{- if .Values.replication.enabled }}\napiVersion: v1\nkind: Service\nmetadata:\n name: {{ template \"slave.fullname\" . }}\n labels:\n app: \"{{ template \"mariadb.name\" . }}\"\n chart: {{ template \"mariadb.chart\" . }}\n component: \"slave\"\n release: {{ .Release.Name | quote }}\n heritage: {{ .Release.Service | quote }}\n{{- if .Values.metrics.enabled }}\n annotations:\n{{ toYaml .Values.metrics.annotations | indent 4 }}\n{{- end }}\nspec:\n type: {{ .Values.service.type }}\n {{- if eq .Values.service.type \"ClusterIP\" }}\n {{- if .Values.service.clusterIp }}\n clusterIP: {{ .Values.service.clusterIp }}\n {{- end }}\n {{- end }}\n ports:\n - name: mysql\n port: {{ .Values.service.port }}\n targetPort: mysql\n{{- if (eq .Values.service.type \"NodePort\") }}\n{{- if .Values.service.nodePort }}\n{{- if .Values.service.nodePort.slave }}\n nodePort: {{ .Values.service.nodePort.slave }}\n{{- end }}\n{{- end }}\n{{- end }}\n{{- if .Values.metrics.enabled }}\n - name: metrics\n port: 9104\n targetPort: metrics\n{{- end }}\n selector:\n app: \"{{ template \"mariadb.name\" . }}\"\n component: \"slave\"\n release: \"{{ .Release.Name }}\"\n{{- end }}\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/test-runner.yaml",
"content": "apiVersion: v1\nkind: Pod\nmetadata:\n name: \"{{ template \"mariadb.fullname\" . }}-test-{{ randAlphaNum 5 | lower }}\"\n annotations:\n \"helm.sh/hook\": test-success\nspec:\n initContainers:\n - name: \"test-framework\"\n image: \"dduportal/bats:0.4.0\"\n command:\n - \"bash\"\n - \"-c\"\n - |\n set -ex\n # copy bats to tools dir\n cp -R /usr/local/libexec/ /tools/bats/\n volumeMounts:\n - mountPath: /tools\n name: tools\n containers:\n - name: mariadb-test\n image: {{ template \"mariadb.image\" . }}\n imagePullPolicy: {{ .Values.image.pullPolicy | quote }}\n command: [\"/tools/bats/bats\", \"-t\", \"/tests/run.sh\"]\n env:\n - name: MARIADB_ROOT_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"mariadb.fullname\" . }}\n {{- end }}\n key: mariadb-root-password\n volumeMounts:\n - mountPath: /tests\n name: tests\n readOnly: true\n - mountPath: /tools\n name: tools\n volumes:\n - name: tests\n configMap:\n name: {{ template \"mariadb.fullname\" . }}-tests\n - name: tools\n emptyDir: {}\n restartPolicy: Never\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/templates/tests.yaml",
"content": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"mariadb.fullname\" . }}-tests\ndata:\n run.sh: |-\n @test \"Testing MariaDB is accessible\" {\n mysql -h {{ template \"mariadb.fullname\" . }} -uroot -p$MARIADB_ROOT_PASSWORD -e 'show databases;'\n }\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/values-production.yaml",
"content": "## Global Docker image registry\n## Please, note that this will override the image registry for all the images, including dependencies, configured to use the global value\n##\n# global:\n# imageRegistry:\n\n## Bitnami MariaDB image\n## ref: https://hub.docker.com/r/bitnami/mariadb/tags/\n##\nimage:\n registry: docker.io\n repository: bitnami/mariadb\n tag: 10.1.37\n ## Specify a imagePullPolicy\n ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'\n ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images\n ##\n pullPolicy: IfNotPresent\n ## Optionally specify an array of imagePullSecrets.\n ## Secrets must be manually created in the namespace.\n ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/\n ##\n # pullSecrets:\n # - myRegistrKeySecretName\n\n ## Set to true if you would like to see extra information on logs\n ## It turns BASH and NAMI debugging in minideb\n ## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging\n debug: false\n\nservice:\n ## Kubernetes service type, ClusterIP and NodePort are supported at present\n type: ClusterIP\n # clusterIp: None\n port: 3306\n ## Specify the nodePort value for the LoadBalancer and NodePort service types.\n ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport\n ##\n # nodePort:\n # master: 30001\n # slave: 30002\n\n## Pods Service Account\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\nserviceAccount:\n ## Specifies whether a ServiceAccount should be created\n ##\n create: false\n ## The name of the ServiceAccount to use.\n ## If not set and create is true, a name is generated using the mariadb.fullname template\n # name:\n\n## Pod Security Context\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/\n##\nsecurityContext:\n enabled: true\n fsGroup: 1001\n runAsUser: 1001\n\n# # Use existing secret (ignores root, db and replication passwords)\n# existingSecret:\n\nrootUser:\n ## MariaDB admin password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-the-root-password-on-first-run\n ##\n password:\n ##\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: true\n\ndb:\n ## MariaDB username and password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#creating-a-database-user-on-first-run\n ##\n user:\n password:\n ## Password is ignored if existingSecret is specified.\n ## Database to create\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#creating-a-database-on-first-run\n ##\n name: my_database\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: true\n\nreplication:\n ## Enable replication. This enables the creation of replicas of MariaDB. If false, only a\n ## master deployment would be created\n enabled: true\n ##\n ## MariaDB replication user\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-up-a-replication-cluster\n ##\n user: replicator\n ## MariaDB replication user password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-up-a-replication-cluster\n ##\n password:\n ## Password is ignored if existingSecret is specified.\n ##\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: true\n\n## initdb scripts\n## Specify dictionnary of scripts to be run at first boot\n## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory\n##\n# initdbScripts:\n# my_init_script.sh: |\n# #!/bin/sh\n# echo \"Do something.\"\n#\n## ConfigMap with scripts to be run at first boot\n## Note: This will override initdbScripts\n# initdbScriptsConfigMap:\n\nmaster:\n ## Mariadb Master additional pod annotations\n ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/\n # annotations:\n # - key: key1\n # value: value1\n\n ## Affinity for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n ##\n affinity: {}\n\n ## Kept for backwards compatibility. You can now disable it by removing it.\n ## if you wish to set it through master.affinity.podAntiAffinity instead.\n ##\n antiAffinity: soft\n\n ## Tolerations for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/\n ##\n tolerations: []\n\n ## Enable persistence using Persistent Volume Claims\n ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/\n ##\n persistence:\n ## If true, use a Persistent Volume Claim, If false, use emptyDir\n ##\n enabled: true\n # Enable persistence using an existing PVC\n # existingClaim:\n mountPath: /bitnami/mariadb\n ## Persistent Volume Storage Class\n ## If defined, storageClassName: \n ## If set to \"-\", storageClassName: \"\", which disables dynamic provisioning\n ## If undefined (the default) or set to null, no storageClassName spec is\n ## set, choosing the default provisioner. (gp2 on AWS, standard on\n ## GKE, AWS & OpenStack)\n ##\n # storageClass: \"-\"\n ## Persistent Volume Claim annotations\n ##\n annotations: {}\n ## Persistent Volume Access Mode\n ##\n accessModes:\n - ReadWriteOnce\n ## Persistent Volume size\n ##\n size: 8Gi\n ##\n extraInitContainers: |\n # - name: do-something\n # image: busybox\n # command: ['do', 'something']\n\n ## Configure MySQL with a custom my.cnf file\n ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file\n ##\n config: |-\n [mysqld]\n skip-name-resolve\n explicit_defaults_for_timestamp\n basedir=/opt/bitnami/mariadb\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n tmpdir=/opt/bitnami/mariadb/tmp\n max_allowed_packet=16M\n bind-address=0.0.0.0\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n log-error=/opt/bitnami/mariadb/logs/mysqld.log\n character-set-server=UTF8\n collation-server=utf8_general_ci\n\n [client]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n default-character-set=UTF8\n\n [manager]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n\n ## Configure master resource requests and limits\n ## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n ##\n resources: {}\n livenessProbe:\n enabled: true\n ##\n ## Initializing the database could take some time\n initialDelaySeconds: 120\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n readinessProbe:\n enabled: true\n initialDelaySeconds: 15\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n\nslave:\n replicas: 2\n\n\n ## Mariadb Slave additional pod annotations\n ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/\n # annotations:\n # - key: key1\n # value: value1\n\n ## Affinity for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n ##\n affinity: {}\n\n ## Kept for backwards compatibility. You can now disable it by removing it.\n ## if you wish to set it through slave.affinity.podAntiAffinity instead.\n ##\n antiAffinity: soft\n\n ## Tolerations for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/\n ##\n tolerations: []\n\n persistence:\n ## If true, use a Persistent Volume Claim, If false, use emptyDir\n ##\n enabled: true\n # storageClass: \"-\"\n annotations:\n accessModes:\n - ReadWriteOnce\n ## Persistent Volume size\n ##\n size: 8Gi\n ##\n extraInitContainers: |\n # - name: do-something\n # image: busybox\n # command: ['do', 'something']\n\n ## Configure MySQL slave with a custom my.cnf file\n ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file\n ##\n config: |-\n [mysqld]\n skip-name-resolve\n explicit_defaults_for_timestamp\n basedir=/opt/bitnami/mariadb\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n tmpdir=/opt/bitnami/mariadb/tmp\n max_allowed_packet=16M\n bind-address=0.0.0.0\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n log-error=/opt/bitnami/mariadb/logs/mysqld.log\n character-set-server=UTF8\n collation-server=utf8_general_ci\n\n [client]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n default-character-set=UTF8\n\n [manager]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n\n ##\n ## Configure slave resource requests and limits\n ## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n ##\n resources: {}\n livenessProbe:\n enabled: true\n ##\n ## Initializing the database could take some time\n initialDelaySeconds: 120\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n readinessProbe:\n enabled: true\n initialDelaySeconds: 15\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n\nmetrics:\n enabled: true\n image:\n registry: docker.io\n repository: prom/mysqld-exporter\n tag: v0.10.0\n pullPolicy: IfNotPresent\n resources: {}\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/port: \"9104\"\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/mariadb/values.yaml",
"content": "## Global Docker image registry\n## Please, note that this will override the image registry for all the images, including dependencies, configured to use the global value\n##\n# global:\n# imageRegistry:\n\n## Bitnami MariaDB image\n## ref: https://hub.docker.com/r/bitnami/mariadb/tags/\n##\nimage:\n registry: docker.io\n repository: bitnami/mariadb\n tag: 10.1.37\n ## Specify a imagePullPolicy\n ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'\n ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images\n ##\n pullPolicy: IfNotPresent\n ## Optionally specify an array of imagePullSecrets.\n ## Secrets must be manually created in the namespace.\n ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/\n ##\n # pullSecrets:\n # - myRegistrKeySecretName\n\n ## Set to true if you would like to see extra information on logs\n ## It turns BASH and NAMI debugging in minideb\n ## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging\n debug: false\n\nservice:\n ## Kubernetes service type, ClusterIP and NodePort are supported at present\n type: ClusterIP\n # clusterIp: None\n port: 3306\n ## Specify the nodePort value for the LoadBalancer and NodePort service types.\n ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport\n ##\n # nodePort:\n # master: 30001\n # slave: 30002\n\n## Pods Service Account\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\nserviceAccount:\n ## Specifies whether a ServiceAccount should be created\n ##\n create: false\n ## The name of the ServiceAccount to use.\n ## If not set and create is true, a name is generated using the mariadb.fullname template\n # name:\n\n## Pod Security Context\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/\n##\nsecurityContext:\n enabled: true\n fsGroup: 1001\n runAsUser: 1001\n\n# # Use existing secret (ignores root, db and replication passwords)\n# existingSecret:\n\nrootUser:\n ## MariaDB admin password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-the-root-password-on-first-run\n ##\n password:\n ##\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: false\n\ndb:\n ## MariaDB username and password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#creating-a-database-user-on-first-run\n ##\n user:\n password:\n ## Password is ignored if existingSecret is specified.\n ## Database to create\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#creating-a-database-on-first-run\n ##\n name: my_database\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: false\n\nreplication:\n ## Enable replication. This enables the creation of replicas of MariaDB. If false, only a\n ## master deployment would be created\n enabled: true\n ##\n ## MariaDB replication user\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-up-a-replication-cluster\n ##\n user: replicator\n ## MariaDB replication user password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-up-a-replication-cluster\n ##\n password:\n ## Password is ignored if existingSecret is specified.\n ##\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: false\n\n## initdb scripts\n## Specify dictionnary of scripts to be run at first boot\n## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory\n##\n# initdbScripts:\n# my_init_script.sh: |\n# #!/bin/sh\n# echo \"Do something.\"\n#\n## ConfigMap with scripts to be run at first boot\n## Note: This will override initdbScripts\n# initdbScriptsConfigMap:\n\nmaster:\n ## Mariadb Master additional pod annotations\n ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/\n # annotations:\n # - key: key1\n # value: value1\n\n ## Affinity for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n ##\n affinity: {}\n\n ## Kept for backwards compatibility. You can now disable it by removing it.\n ## if you wish to set it through master.affinity.podAntiAffinity instead.\n ##\n antiAffinity: soft\n\n ## Tolerations for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/\n ##\n tolerations: []\n\n ## Enable persistence using Persistent Volume Claims\n ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/\n ##\n persistence:\n ## If true, use a Persistent Volume Claim, If false, use emptyDir\n ##\n enabled: true\n # Enable persistence using an existing PVC\n # existingClaim:\n mountPath: /bitnami/mariadb\n ## Persistent Volume Storage Class\n ## If defined, storageClassName: \n ## If set to \"-\", storageClassName: \"\", which disables dynamic provisioning\n ## If undefined (the default) or set to null, no storageClassName spec is\n ## set, choosing the default provisioner. (gp2 on AWS, standard on\n ## GKE, AWS & OpenStack)\n ##\n # storageClass: \"-\"\n ## Persistent Volume Claim annotations\n ##\n annotations: {}\n ## Persistent Volume Access Mode\n ##\n accessModes:\n - ReadWriteOnce\n ## Persistent Volume size\n ##\n size: 8Gi\n ##\n extraInitContainers: |\n # - name: do-something\n # image: busybox\n # command: ['do', 'something']\n\n ## Configure MySQL with a custom my.cnf file\n ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file\n ##\n config: |-\n [mysqld]\n skip-name-resolve\n explicit_defaults_for_timestamp\n basedir=/opt/bitnami/mariadb\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n tmpdir=/opt/bitnami/mariadb/tmp\n max_allowed_packet=16M\n bind-address=0.0.0.0\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n log-error=/opt/bitnami/mariadb/logs/mysqld.log\n character-set-server=UTF8\n collation-server=utf8_general_ci\n\n [client]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n default-character-set=UTF8\n\n [manager]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n\n ## Configure master resource requests and limits\n ## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n ##\n resources: {}\n livenessProbe:\n enabled: true\n ##\n ## Initializing the database could take some time\n initialDelaySeconds: 120\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n readinessProbe:\n enabled: true\n initialDelaySeconds: 30\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n\nslave:\n replicas: 1\n\n ## Mariadb Slave additional pod annotations\n ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/\n # annotations:\n # - key: key1\n # value: value1\n\n ## Affinity for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n ##\n affinity: {}\n\n ## Kept for backwards compatibility. You can now disable it by removing it.\n ## if you wish to set it through slave.affinity.podAntiAffinity instead.\n ##\n antiAffinity: soft\n\n ## Tolerations for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/\n ##\n tolerations: []\n\n persistence:\n ## If true, use a Persistent Volume Claim, If false, use emptyDir\n ##\n enabled: true\n # storageClass: \"-\"\n annotations:\n accessModes:\n - ReadWriteOnce\n ## Persistent Volume size\n ##\n size: 8Gi\n ##\n extraInitContainers: |\n # - name: do-something\n # image: busybox\n # command: ['do', 'something']\n\n ## Configure MySQL slave with a custom my.cnf file\n ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file\n ##\n config: |-\n [mysqld]\n skip-name-resolve\n explicit_defaults_for_timestamp\n basedir=/opt/bitnami/mariadb\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n tmpdir=/opt/bitnami/mariadb/tmp\n max_allowed_packet=16M\n bind-address=0.0.0.0\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n log-error=/opt/bitnami/mariadb/logs/mysqld.log\n character-set-server=UTF8\n collation-server=utf8_general_ci\n\n [client]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n default-character-set=UTF8\n\n [manager]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n\n ##\n ## Configure slave resource requests and limits\n ## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n ##\n resources: {}\n livenessProbe:\n enabled: true\n ##\n ## Initializing the database could take some time\n initialDelaySeconds: 120\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n readinessProbe:\n enabled: true\n initialDelaySeconds: 45\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n\nmetrics:\n enabled: false\n image:\n registry: docker.io\n repository: prom/mysqld-exporter\n tag: v0.10.0\n pullPolicy: IfNotPresent\n resources: {}\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/port: \"9104\"\n"
},
{
"path": "manifests/deprecated/mariadb-cluster/my-values.yaml",
"content": "## Global Docker image registry\n## Please, note that this will override the image registry for all the images, including dependencies, configured to use the global value\n##\n# global:\n# imageRegistry:\n\n## Bitnami MariaDB image\n## ref: https://hub.docker.com/r/bitnami/mariadb/tags/\n##\nimage:\n registry: docker.io\n repository: bitnami/mariadb\n tag: 10.1.37\n ## Specify a imagePullPolicy\n ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'\n ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images\n ##\n pullPolicy: IfNotPresent\n ## Optionally specify an array of imagePullSecrets.\n ## Secrets must be manually created in the namespace.\n ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/\n ##\n # pullSecrets:\n # - myRegistrKeySecretName\n\n ## Set to true if you would like to see extra information on logs\n ## It turns BASH and NAMI debugging in minideb\n ## ref: https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging\n debug: false\n\nservice:\n ## Kubernetes service type, ClusterIP and NodePort are supported at present\n type: NodePort\n # clusterIp: None\n port: 3306\n ## Specify the nodePort value for the LoadBalancer and NodePort service types.\n ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport\n ##\n nodePort:\n master: 33306\n slave: 33307\n\n## Pods Service Account\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\nserviceAccount:\n ## Specifies whether a ServiceAccount should be created\n ##\n create: false\n ## The name of the ServiceAccount to use.\n ## If not set and create is true, a name is generated using the mariadb.fullname template\n # name:\n\n## Pod Security Context\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/\n##\nsecurityContext:\n enabled: true\n fsGroup: 1001\n runAsUser: 1001\n\n# # Use existing secret (ignores root, db and replication passwords)\n# existingSecret:\n\nrootUser:\n ## MariaDB admin password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-the-root-password-on-first-run\n ##\n password: test.c0m\n ##\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: true\n\ndb:\n ## MariaDB username and password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#creating-a-database-user-on-first-run\n ##\n user: hello\n password: hello\n ## Password is ignored if existingSecret is specified.\n ## Database to create\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#creating-a-database-on-first-run\n ##\n name: hello\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: true\n\nreplication:\n ## Enable replication. This enables the creation of replicas of MariaDB. If false, only a\n ## master deployment would be created\n enabled: true\n ##\n ## MariaDB replication user\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-up-a-replication-cluster\n ##\n user: replicator\n ## MariaDB replication user password\n ## ref: https://github.com/bitnami/bitnami-docker-mariadb#setting-up-a-replication-cluster\n ##\n password: R4%forep11CAT0r\n ## Password is ignored if existingSecret is specified.\n ##\n ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.\n ## If it is not force, a random password will be generated.\n forcePassword: true\n\n## initdb scripts\n## Specify dictionnary of scripts to be run at first boot\n## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory\n##\n# initdbScripts:\n# my_init_script.sh: |\n# #!/bin/sh\n# echo \"Do something.\"\n#\n## ConfigMap with scripts to be run at first boot\n## Note: This will override initdbScripts\n# initdbScriptsConfigMap:\n\nmaster:\n ## Mariadb Master additional pod annotations\n ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/\n # annotations:\n # - key: key1\n # value: value1\n\n ## Affinity for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n ##\n affinity: {}\n\n ## Kept for backwards compatibility. You can now disable it by removing it.\n ## if you wish to set it through master.affinity.podAntiAffinity instead.\n ##\n antiAffinity: soft\n\n ## Tolerations for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/\n ##\n tolerations: []\n\n ## Enable persistence using Persistent Volume Claims\n ## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/\n ##\n persistence:\n ## If true, use a Persistent Volume Claim, If false, use emptyDir\n ##\n enabled: true\n # Enable persistence using an existing PVC\n # existingClaim:\n mountPath: /bitnami/mariadb\n ## Persistent Volume Storage Class\n ## If defined, storageClassName: \n ## If set to \"-\", storageClassName: \"\", which disables dynamic provisioning\n ## If undefined (the default) or set to null, no storageClassName spec is\n ## set, choosing the default provisioner. (gp2 on AWS, standard on\n ## GKE, AWS & OpenStack)\n ##\n storageClass: \"nfs-db\"\n ## Persistent Volume Claim annotations\n ##\n annotations: {}\n ## Persistent Volume Access Mode\n ##\n accessModes:\n - ReadWriteOnce\n ## Persistent Volume size\n ##\n size: 5Gi\n ##\n extraInitContainers: |\n # - name: do-something\n # image: busybox\n # command: ['do', 'something']\n\n ## Configure MySQL with a custom my.cnf file\n ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file\n ##\n config: |-\n [mysqld]\n skip-name-resolve\n explicit_defaults_for_timestamp\n basedir=/opt/bitnami/mariadb\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n tmpdir=/opt/bitnami/mariadb/tmp\n bind-address=0.0.0.0\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n log-error=/opt/bitnami/mariadb/logs/mysqld.log\n character-set-server=UTF8\n collation-server=utf8_general_ci\n # optimize\n max_allowed_packet = 1024M\n table_open_cache = 512\n sort_buffer_size = 2M\n read_buffer_size = 2M\n read_rnd_buffer_size = 8M\n thread_cache_size = 8\n query_cache_size = 32M\n max_heap_table_size=1024M\n tmp_table_size=1024M\n max_connections=65535\n max_connect_errors=65535\n wait_timeout=172800\n interactive_timeout=172800\n connect_timeout=30\n # log settings\n expire_logs_days=3\n\n [client]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n default-character-set=UTF8\n\n [manager]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n\n ## Configure master resource requests and limits\n ## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n ##\n resources: {}\n livenessProbe:\n enabled: true\n ##\n ## Initializing the database could take some time\n initialDelaySeconds: 120\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n readinessProbe:\n enabled: true\n initialDelaySeconds: 15\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n\nslave:\n replicas: 1 \n\n\n ## Mariadb Slave additional pod annotations\n ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/\n # annotations:\n # - key: key1\n # value: value1\n\n ## Affinity for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n ##\n affinity: {}\n\n ## Kept for backwards compatibility. You can now disable it by removing it.\n ## if you wish to set it through slave.affinity.podAntiAffinity instead.\n ##\n antiAffinity: soft\n\n ## Tolerations for pod assignment\n ## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/\n ##\n tolerations: []\n\n persistence:\n ## If true, use a Persistent Volume Claim, If false, use emptyDir\n ##\n enabled: false\n # storageClass: \"-\"\n annotations:\n accessModes:\n - ReadWriteOnce\n ## Persistent Volume size\n ##\n size: 5Gi\n ##\n extraInitContainers: |\n # - name: do-something\n # image: busybox\n # command: ['do', 'something']\n\n ## Configure MySQL slave with a custom my.cnf file\n ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file\n ##\n config: |-\n [mysqld]\n skip-name-resolve\n explicit_defaults_for_timestamp\n basedir=/opt/bitnami/mariadb\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n tmpdir=/opt/bitnami/mariadb/tmp\n bind-address=0.0.0.0\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n log-error=/opt/bitnami/mariadb/logs/mysqld.log\n character-set-server=UTF8\n collation-server=utf8_general_ci\n # optimize\n max_allowed_packet = 1024M\n table_open_cache = 512\n sort_buffer_size = 2M\n read_buffer_size = 2M\n read_rnd_buffer_size = 8M\n thread_cache_size = 8\n query_cache_size = 32M\n max_heap_table_size=1024M\n tmp_table_size=1024M\n max_connections=65535\n max_connect_errors=65535\n wait_timeout=172800\n interactive_timeout=172800\n connect_timeout=30\n # log settings\n expire_logs_days=3\n\n [client]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n default-character-set=UTF8\n\n [manager]\n port=3306\n socket=/opt/bitnami/mariadb/tmp/mysql.sock\n pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid\n\n ##\n ## Configure slave resource requests and limits\n ## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n ##\n resources: {}\n livenessProbe:\n enabled: true\n ##\n ## Initializing the database could take some time\n initialDelaySeconds: 120\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n readinessProbe:\n enabled: true\n initialDelaySeconds: 15\n ##\n ## Default Kubernetes values\n periodSeconds: 10\n timeoutSeconds: 1\n successThreshold: 1\n failureThreshold: 3\n\nmetrics:\n enabled: false\n image:\n registry: docker.io\n repository: prom/mysqld-exporter\n tag: v0.10.0\n pullPolicy: IfNotPresent\n resources: {}\n annotations:\n prometheus.io/scrape: \"true\"\n prometheus.io/port: \"9104\"\n"
},
{
"path": "manifests/deprecated/mysql-cluster/mysql-configmap.yaml",
"content": "# https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: mysql\n labels:\n app: mysql\n app.kubernetes.io/name: mysql\ndata:\n primary.cnf: |\n # Apply this config only on the primary.\n [mysqld]\n log-bin\n replica.cnf: |\n # Apply this config only on replicas.\n [mysqld]\n super-read-only\n"
},
{
"path": "manifests/deprecated/mysql-cluster/mysql-services.yaml",
"content": "# https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/\n# Headless service for stable DNS entries of StatefulSet members.\napiVersion: v1\nkind: Service\nmetadata:\n name: mysql\n labels:\n app: mysql\n app.kubernetes.io/name: mysql\nspec:\n ports:\n - name: mysql\n port: 3306\n clusterIP: None\n selector:\n app: mysql\n---\n# Client service for connecting to any MySQL instance for reads.\n# For writes, you must instead connect to the primary: mysql-0.mysql.\napiVersion: v1\nkind: Service\nmetadata:\n name: mysql-read\n labels:\n app: mysql\n app.kubernetes.io/name: mysql\n readonly: \"true\"\nspec:\n ports:\n - name: mysql\n port: 3306\n selector:\n app: mysql\n"
},
{
"path": "manifests/deprecated/mysql-cluster/mysql-statefulset.yaml",
"content": "# https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: mysql\nspec:\n selector:\n matchLabels:\n app: mysql\n app.kubernetes.io/name: mysql\n serviceName: mysql\n replicas: 2\n template:\n metadata:\n labels:\n app: mysql\n app.kubernetes.io/name: mysql\n spec:\n initContainers:\n - name: init-mysql\n image: mysql:5.7\n command:\n - bash\n - \"-c\"\n - |\n set -ex\n # Generate mysql server-id from pod ordinal index.\n [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1\n ordinal=${BASH_REMATCH[1]}\n echo [mysqld] > /mnt/conf.d/server-id.cnf\n # Add an offset to avoid reserved server-id=0 value.\n echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf\n # Copy appropriate conf.d files from config-map to emptyDir.\n if [[ $ordinal -eq 0 ]]; then\n cp /mnt/config-map/primary.cnf /mnt/conf.d/\n else\n cp /mnt/config-map/replica.cnf /mnt/conf.d/\n fi\n volumeMounts:\n - name: conf\n mountPath: /mnt/conf.d\n - name: config-map\n mountPath: /mnt/config-map\n - name: clone-mysql\n #image: gcr.io/google-samples/xtrabackup:1.0\n image: jmgao1983/xtrabackup:1.0\n command:\n - bash\n - \"-c\"\n - |\n set -ex\n # Skip the clone if data already exists.\n [[ -d /var/lib/mysql/mysql ]] && exit 0\n # Skip the clone on primary (ordinal index 0).\n [[ `hostname` =~ -([0-9]+)$ ]] || exit 1\n ordinal=${BASH_REMATCH[1]}\n [[ $ordinal -eq 0 ]] && exit 0\n # Clone data from previous peer.\n ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql\n # Prepare the backup.\n xtrabackup --prepare --target-dir=/var/lib/mysql\n volumeMounts:\n - name: data\n mountPath: /var/lib/mysql\n subPath: mysql\n - name: conf\n mountPath: /etc/mysql/conf.d\n containers:\n - name: mysql\n image: mysql:5.7\n env:\n - name: MYSQL_ALLOW_EMPTY_PASSWORD\n value: \"1\"\n ports:\n - name: mysql\n containerPort: 3306\n volumeMounts:\n - name: data\n mountPath: /var/lib/mysql\n subPath: mysql\n - name: conf\n mountPath: /etc/mysql/conf.d\n resources:\n requests:\n cpu: 500m\n memory: 1Gi\n livenessProbe:\n exec:\n command: [\"mysqladmin\", \"ping\"]\n initialDelaySeconds: 30\n periodSeconds: 10\n timeoutSeconds: 5\n readinessProbe:\n exec:\n # Check we can execute queries over TCP (skip-networking is off).\n command: [\"mysql\", \"-h\", \"127.0.0.1\", \"-e\", \"SELECT 1\"]\n initialDelaySeconds: 5\n periodSeconds: 2\n timeoutSeconds: 1\n - name: xtrabackup\n #image: gcr.io/google-samples/xtrabackup:1.0\n image: jmgao1983/xtrabackup:1.0\n ports:\n - name: xtrabackup\n containerPort: 3307\n command:\n - bash\n - \"-c\"\n - |\n set -ex\n cd /var/lib/mysql\n\n # Determine binlog position of cloned data, if any.\n if [[ -f xtrabackup_slave_info && \"x$( change_master_to.sql.in\n # Ignore xtrabackup_binlog_info in this case (it's useless).\n rm -f xtrabackup_slave_info xtrabackup_binlog_info\n elif [[ -f xtrabackup_binlog_info ]]; then\n # We're cloning directly from primary. Parse binlog position.\n [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1\n rm -f xtrabackup_binlog_info xtrabackup_slave_info\n echo \"CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\\\n MASTER_LOG_POS=${BASH_REMATCH[2]}\" > change_master_to.sql.in\n fi\n\n # Check if we need to complete a clone by starting replication.\n if [[ -f change_master_to.sql.in ]]; then\n echo \"Waiting for mysqld to be ready (accepting connections)\"\n until mysql -h 127.0.0.1 -e \"SELECT 1\"; do sleep 1; done\n\n echo \"Initializing replication from clone position\"\n mysql -h 127.0.0.1 \\\n -e \"$(=2.0.1 to >=3.0.0 of this chart, `Role`, `RoleBinding`, and `ServiceAccount` resources should be deleted manually.\n\n### Upgrading the chart from 3.x to 4.x\n\nStarting from version `4.x` HAProxy sidecar prometheus-exporter removed and replaced by the embedded [HAProxy metrics endpoint](https://github.com/haproxy/haproxy/tree/master/contrib/prometheus-exporter), as a result when upgrading from version 3.x to 4.x section `haproxy.exporter` should be removed and the `haproxy.metrics` need to be configured for fit your needs.\n\n## Installing the Chart\n\nTo install the chart\n\n```bash\n$ helm install stable/redis-ha\n```\n\nThe command deploys Redis on the Kubernetes cluster in the default configuration. By default this chart install one master pod containing redis master container and sentinel container along with 2 redis slave pods each containing their own sentinel sidecars. The [configuration](#configuration) section lists the parameters that can be configured during installation.\n\n> **Tip**: List all releases using `helm list`\n\n## Uninstalling the Chart\n\nTo uninstall/delete the deployment:\n\n```bash\n$ helm delete \n```\n\nThe command removes all the Kubernetes components associated with the chart and deletes the release.\n\n## Configuration\n\nThe following table lists the configurable parameters of the Redis chart and their default values.\n\n| Parameter | Description | Default |\n|:--------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------|\n| `image` | Redis image | `redis` |\n| `imagePullSecrets` | Reference to one or more secrets to be used when pulling redis images | [] |\n| `tag` | Redis tag | `5.0.6-alpine` |\n| `replicas` | Number of redis master/slave pods | `3` |\n| `serviceAccount.create` | Specifies whether a ServiceAccount should be created | `true` |\n| `serviceAccount.name` | The name of the ServiceAccount to create | Generated using the redis-ha.fullname template |\n| `rbac.create` | Create and use RBAC resources | `true` |\n| `redis.port` | Port to access the redis service | `6379` |\n| `redis.masterGroupName` | Redis convention for naming the cluster group: must match `^[\\\\w-\\\\.]+$` and can be templated | `mymaster` |\n| `redis.config` | Any valid redis config options in this section will be applied to each server (see below) | see values.yaml |\n| `redis.customConfig` | Allows for custom redis.conf files to be applied. If this is used then `redis.config` is ignored | `` |\n| `redis.resources` | CPU/Memory for master/slave nodes resource requests/limits | `{}` |\n| `sentinel.port` | Port to access the sentinel service | `26379` |\n| `sentinel.quorum` | Minimum number of servers necessary to maintain quorum | `2` |\n| `sentinel.config` | Valid sentinel config options in this section will be applied as config options to each sentinel (see below) | see values.yaml |\n| `sentinel.customConfig` | Allows for custom sentinel.conf files to be applied. If this is used then `sentinel.config` is ignored | `` |\n| `sentinel.resources` | CPU/Memory for sentinel node resource requests/limits | `{}` |\n| `init.resources` | CPU/Memory for init Container node resource requests/limits | `{}` |\n| `auth` | Enables or disables redis AUTH (Requires `redisPassword` to be set) | `false` |\n| `redisPassword` | A password that configures a `requirepass` and `masterauth` in the conf parameters (Requires `auth: enabled`) | `` |\n| `authKey` | The key holding the redis password in an existing secret. | `auth` |\n| `existingSecret` | An existing secret containing a key defined by `authKey` that configures `requirepass` and `masterauth` in the conf parameters (Requires `auth: enabled`, cannot be used in conjunction with `.Values.redisPassword`) | `` |\n| `nodeSelector` | Node labels for pod assignment | `{}` |\n| `tolerations` | Toleration labels for pod assignment | `[]` |\n| `hardAntiAffinity` | Whether the Redis server pods should be forced to run on separate nodes. | `true` |\n| `additionalAffinities` | Additional affinities to add to the Redis server pods. | `{}` |\n| `securityContext` | Security context to be added to the Redis server pods. | `{runAsUser: 1000, fsGroup: 1000, runAsNonRoot: true}` |\n| `affinity` | Override all other affinity settings with a string. | `\"\"` |\n| `persistentVolume.size` | Size for the volume | 10Gi |\n| `persistentVolume.annotations` | Annotations for the volume | `{}` |\n| `persistentVolume.reclaimPolicy` | Method used to reclaim an obsoleted volume. `Delete` or `Retain` | `\"\"` |\n| `emptyDir` | Configuration of `emptyDir`, used only if persistentVolume is disabled and no hostPath specified | `{}` |\n| `exporter.enabled` | If `true`, the prometheus exporter sidecar is enabled | `false` |\n| `exporter.image` | Exporter image | `oliver006/redis_exporter` |\n| `exporter.tag` | Exporter tag | `v0.31.0` |\n| `exporter.port` | Exporter port | `9121` |\n| `exporter.annotations` | Prometheus scrape annotations | `{prometheus.io/path: /metrics, prometheus.io/port: \"9121\", prometheus.io/scrape: \"true\"}` |\n| `exporter.extraArgs` | Additional args for the exporter | `{}` |\n| `exporter.script` | A custom custom Lua script that will be mounted to exporter for collection of custom metrics. Creates a ConfigMap and sets env var `REDIS_EXPORTER_SCRIPT`. | |\n| `exporter.serviceMonitor.enabled` | Use servicemonitor from prometheus operator | `false` |\n| `exporter.serviceMonitor.namespace` | Namespace the service monitor is created in | `default` |\n| `exporter.serviceMonitor.interval` | Scrape interval, If not set, the Prometheus default scrape interval is used | `nil` |\n| `exporter.serviceMonitor.telemetryPath` | Path to redis-exporter telemetry-path | `/metrics` |\n| `exporter.serviceMonitor.labels` | Labels for the servicemonitor passed to Prometheus Operator | `{}` |\n| `exporter.serviceMonitor.timeout` | How long until a scrape request times out. If not set, the Prometheus default scape timeout is used | `nil` |\n| `haproxy.enabled` | Enabled HAProxy LoadBalancing/Proxy | `false` |\n| `haproxy.replicas` | Number of HAProxy instances | `3` |\n| `haproxy.image.repository`| HAProxy Image Repository | `haproxy` |\n| `haproxy.image.tag` | HAProxy Image Tag | `2.0.1` |\n| `haproxy.image.pullPolicy`| HAProxy Image PullPolicy | `IfNotPresent` |\n| `haproxy.imagePullSecrets`| Reference to one or more secrets to be used when pulling haproxy images | [] |\n| `haproxy.annotations` | HAProxy template annotations | `{}` |\n| `haproxy.customConfig` | Allows for custom config-haproxy.cfg file to be applied. If this is used then default config will be overwriten | `` |\n| `haproxy.extraConfig` | Allows to place any additional configuration section to add to the default config-haproxy.cfg | `` |\n| `haproxy.resources` | HAProxy resources | `{}` |\n| `haproxy.emptyDir` | Configuration of `emptyDir` | `{}` |\n| `haproxy.service.type` | HAProxy service type \"ClusterIP\", \"LoadBalancer\" or \"NodePort\" | `ClusterIP` |\n| `haproxy.service.nodePort` | HAProxy service nodePort value (haproxy.service.type must be NodePort) | not set |\n| `haproxy.service.annotations` | HAProxy service annotations | `{}` |\n| `haproxy.stickyBalancing` | HAProxy sticky load balancing to Redis nodes. Helps with connections shutdown. | `false` |\n| `haproxy.hapreadport.enable` | Enable a read only port for redis slaves | `false` |\n| `haproxy.hapreadport.port` | Haproxy port for read only redis slaves | `6380` |\n| `haproxy.metrics.enabled` | HAProxy enable prometheus metric scraping | `false` |\n| `haproxy.metrics.port` | HAProxy prometheus metrics scraping port | `9101` |\n| `haproxy.metrics.portName` | HAProxy metrics scraping port name | `exporter-port` |\n| `haproxy.metrics.scrapePath` | HAProxy prometheus metrics scraping port | `/metrics` |\n| `haproxy.metrics.serviceMonitor.enabled` | Use servicemonitor from prometheus operator for HAProxy metrics | `false` |\n| `haproxy.metrics.serviceMonitor.namespace` | Namespace the service monitor for HAProxy metrics is created in | `default` |\n| `haproxy.metrics.serviceMonitor.interval` | Scrape interval, If not set, the Prometheus default scrape interval is used | `nil` |\n| `haproxy.metrics.serviceMonitor.telemetryPath` | Path to HAProxy metrics telemetry-path | `/metrics` |\n| `haproxy.metrics.serviceMonitor.labels` | Labels for the HAProxy metrics servicemonitor passed to Prometheus Operator | `{}` |\n| `haproxy.metrics.serviceMonitor.timeout` | How long until a scrape request times out. If not set, the Prometheus default scape timeout is used | `nil` |\n| `haproxy.init.resources` | Extra init resources | `{}` |\n| `haproxy.timeout.connect` | haproxy.cfg `timeout connect` setting | `4s` |\n| `haproxy.timeout.server` | haproxy.cfg `timeout server` setting | `30s` |\n| `haproxy.timeout.client` | haproxy.cfg `timeout client` setting | `30s` |\n| `haproxy.timeout.check` | haproxy.cfg `timeout check` setting | `2s` |\n| `haproxy.priorityClassName` | priorityClassName for `haproxy` deployment | not set |\n| `haproxy.securityContext` | Security context to be added to the HAProxy deployment. | `{runAsUser: 1000, fsGroup: 1000, runAsNonRoot: true}` |\n| `haproxy.hardAntiAffinity` | Whether the haproxy pods should be forced to run on separate nodes. | `true` |\n| `haproxy.affinity` | Override all other haproxy affinity settings with a string. | `\"\"` |\n| `haproxy.additionalAffinities` | Additional affinities to add to the haproxy server pods. | `{}` |\n| `podDisruptionBudget` | Pod Disruption Budget rules | `{}` |\n| `priorityClassName` | priorityClassName for `redis-ha-statefulset` | not set |\n| `hostPath.path` | Use this path on the host for data storage | not set |\n| `hostPath.chown` | Run an init-container as root to set ownership on the hostPath | `true` |\n| `sysctlImage.enabled` | Enable an init container to modify Kernel settings | `false` |\n| `sysctlImage.command` | sysctlImage command to execute | [] |\n| `sysctlImage.registry` | sysctlImage Init container registry | `docker.io` |\n| `sysctlImage.repository` | sysctlImage Init container name | `busybox` |\n| `sysctlImage.tag` | sysctlImage Init container tag | `1.31.1` |\n| `sysctlImage.pullPolicy` | sysctlImage Init container pull policy | `Always` |\n| `sysctlImage.mountHostSys`| Mount the host `/sys` folder to `/host-sys` | `false` |\n| `sysctlImage.resources` | sysctlImage resources | `{}` |\n| `schedulerName` | Alternate scheduler name | `nil` |\n\nSpecify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,\n\n```bash\n$ helm install \\\n --set image=redis \\\n --set tag=5.0.5-alpine \\\n stable/redis-ha\n```\n\nThe above command sets the Redis server within `default` namespace.\n\nAlternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,\n\n```bash\n$ helm install -f values.yaml stable/redis-ha\n```\n\n> **Tip**: You can use the default [values.yaml](values.yaml)\n\n## Custom Redis and Sentinel config options\n\nThis chart allows for most redis or sentinel config options to be passed as a key value pair through the `values.yaml` under `redis.config` and `sentinel.config`. See links below for all available options.\n\n[Example redis.conf](http://download.redis.io/redis-stable/redis.conf)\n[Example sentinel.conf](http://download.redis.io/redis-stable/sentinel.conf)\n\nFor example `repl-timeout 60` would be added to the `redis.config` section of the `values.yaml` as:\n\n```yml\n repl-timeout: \"60\"\n```\n\nNote:\n\n1. Some config options should be renamed by redis version，e.g.:\n\n ```\n # In redis 5.x，see https://raw.githubusercontent.com/antirez/redis/5.0/redis.conf\n min-replicas-to-write: 1\n min-replicas-max-lag: 5\n\n # In redis 4.x and redis 3.x，see https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf and https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf\n min-slaves-to-write 1\n min-slaves-max-lag 5\n ```\n\nSentinel options supported must be in the the `sentinel ` format. For example, `sentinel down-after-milliseconds 30000` would be added to the `sentinel.config` section of the `values.yaml` as:\n\n```yml\n down-after-milliseconds: 30000\n```\n\nIf more control is needed from either the redis or sentinel config then an entire config can be defined under `redis.customConfig` or `sentinel.customConfig`. Please note that these values will override any configuration options under their respective section. For example, if you define `sentinel.customConfig` then the `sentinel.config` is ignored.\n\n## Host Kernel Settings\nRedis may require some changes in the kernel of the host machine to work as expected, in particular increasing the `somaxconn` value and disabling transparent huge pages.\nTo do so, you can set up a privileged initContainer with the `sysctlImage` config values, for example:\n```\nsysctlImage:\n enabled: true\n mountHostSys: true\n command:\n - /bin/sh\n - -xc\n - |-\n sysctl -w net.core.somaxconn=10000\n echo never > /host-sys/kernel/mm/transparent_hugepage/enabled\n```\n\n## HAProxy startup\n\nWhen HAProxy is enabled, it will attempt to connect to each announce-service of each redis replica instance in its init container before starting.\nIt will fail if announce-service IP is not available fast enough (10 seconds max by announce-service).\nA such case could happen if the orchestator is pending the nomination of redis pods.\nRisk is limited because announce-service is using `publishNotReadyAddresses: true`, although, in such case, HAProxy pod will be rescheduled afterward by the orchestrator.\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/ci/haproxy-enabled-values.yaml",
"content": "---\n## Enable HAProxy to manage Load Balancing\nhaproxy:\n enabled: true\n annotations:\n any.domain/key: \"value\"\n serviceAccount:\n create: true\n metrics:\n enabled: true\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/NOTES.txt",
"content": "Redis can be accessed via port {{ .Values.redis.port }} and Sentinel can be accessed via port {{ .Values.sentinel.port }} on the following DNS name from within your cluster:\n{{ template \"redis-ha.fullname\" . }}.{{ .Release.Namespace }}.svc.cluster.local\n\nTo connect to your Redis server:\n\n{{- if .Values.auth }}\n1. To retrieve the redis password:\n echo $(kubectl get secret {{ template \"redis-ha.fullname\" . }} -o \"jsonpath={.data['auth']}\" | base64 --decode)\n\n2. Connect to the Redis master pod that you can use as a client. By default the {{ template \"redis-ha.fullname\" . }}-server-0 pod is configured as the master:\n\n kubectl exec -it {{ template \"redis-ha.fullname\" . }}-server-0 sh -n {{ .Release.Namespace }}\n\n3. Connect using the Redis CLI (inside container):\n\n redis-cli -a \n{{- else }}\n1. Run a Redis pod that you can use as a client:\n\n kubectl exec -it {{ template \"redis-ha.fullname\" . }}-server-0 sh -n {{ .Release.Namespace }}\n\n2. Connect using the Redis CLI:\n\n redis-cli -h {{ template \"redis-ha.fullname\" . }}.{{ .Release.Namespace }}.svc.cluster.local\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/_configs.tpl",
"content": "{{/* vim: set filetype=mustache: */}}\n\n{{- define \"config-redis.conf\" }}\n{{- if .Values.redis.customConfig }}\n{{ tpl .Values.redis.customConfig . | indent 4 }}\n{{- else }}\n dir \"/data\"\n port {{ .Values.redis.port }}\n {{- range $key, $value := .Values.redis.config }}\n {{ $key }} {{ $value }}\n {{- end }}\n{{- if .Values.auth }}\n requirepass replace-default-auth\n masterauth replace-default-auth\n{{- end }}\n{{- end }}\n{{- end }}\n\n{{- define \"config-sentinel.conf\" }}\n{{- if .Values.sentinel.customConfig }}\n{{ tpl .Values.sentinel.customConfig . | indent 4 }}\n{{- else }}\n dir \"/data\"\n {{- range $key, $value := .Values.sentinel.config }}\n {{- if eq \"maxclients\" $key }}\n {{ $key }} {{ $value }}\n {{- else }}\n sentinel {{ $key }} {{ template \"redis-ha.masterGroupName\" $ }} {{ $value }}\n {{- end }}\n {{- end }}\n{{- if .Values.auth }}\n sentinel auth-pass {{ template \"redis-ha.masterGroupName\" . }} replace-default-auth\n{{- end }}\n{{- end }}\n{{- end }}\n\n{{- define \"config-init.sh\" }}\n HOSTNAME=\"$(hostname)\"\n INDEX=\"${HOSTNAME##*-}\"\n MASTER=\"$(redis-cli -h {{ template \"redis-ha.fullname\" . }} -p {{ .Values.sentinel.port }} sentinel get-master-addr-by-name {{ template \"redis-ha.masterGroupName\" . }} | grep -E '[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}')\"\n MASTER_GROUP=\"{{ template \"redis-ha.masterGroupName\" . }}\"\n QUORUM=\"{{ .Values.sentinel.quorum }}\"\n REDIS_CONF=/data/conf/redis.conf\n REDIS_PORT={{ .Values.redis.port }}\n SENTINEL_CONF=/data/conf/sentinel.conf\n SENTINEL_PORT={{ .Values.sentinel.port }}\n SERVICE={{ template \"redis-ha.fullname\" . }}\n set -eu\n\n sentinel_update() {\n echo \"Updating sentinel config with master $MASTER\"\n eval MY_SENTINEL_ID=\"\\${SENTINEL_ID_$INDEX}\"\n sed -i \"1s/^/sentinel myid $MY_SENTINEL_ID\\\\n/\" \"$SENTINEL_CONF\"\n sed -i \"2s/^/sentinel monitor $MASTER_GROUP $1 $REDIS_PORT $QUORUM \\\\n/\" \"$SENTINEL_CONF\"\n echo \"sentinel announce-ip $ANNOUNCE_IP\" >> $SENTINEL_CONF\n echo \"sentinel announce-port $SENTINEL_PORT\" >> $SENTINEL_CONF\n }\n\n redis_update() {\n echo \"Updating redis config\"\n echo \"slaveof $1 $REDIS_PORT\" >> \"$REDIS_CONF\"\n echo \"slave-announce-ip $ANNOUNCE_IP\" >> $REDIS_CONF\n echo \"slave-announce-port $REDIS_PORT\" >> $REDIS_CONF\n }\n\n copy_config() {\n cp /readonly-config/redis.conf \"$REDIS_CONF\"\n cp /readonly-config/sentinel.conf \"$SENTINEL_CONF\"\n }\n\n setup_defaults() {\n echo \"Setting up defaults\"\n if [ \"$INDEX\" = \"0\" ]; then\n echo \"Setting this pod as the default master\"\n redis_update \"$ANNOUNCE_IP\"\n sentinel_update \"$ANNOUNCE_IP\"\n sed -i \"s/^.*slaveof.*//\" \"$REDIS_CONF\"\n else\n DEFAULT_MASTER=\"$(getent hosts \"$SERVICE-announce-0\" | awk '{ print $1 }')\"\n if [ -z \"$DEFAULT_MASTER\" ]; then\n echo \"Unable to resolve host\"\n exit 1\n fi\n echo \"Setting default slave config..\"\n redis_update \"$DEFAULT_MASTER\"\n sentinel_update \"$DEFAULT_MASTER\"\n fi\n }\n\n find_master() {\n echo \"Attempting to find master\"\n if [ \"$(redis-cli -h \"$MASTER\"{{ if .Values.auth }} -a \"$AUTH\"{{ end }} ping)\" != \"PONG\" ]; then\n echo \"Can't ping master, attempting to force failover\"\n if redis-cli -h \"$SERVICE\" -p \"$SENTINEL_PORT\" sentinel failover \"$MASTER_GROUP\" | grep -q 'NOGOODSLAVE' ; then\n setup_defaults\n return 0\n fi\n sleep 10\n MASTER=\"$(redis-cli -h $SERVICE -p $SENTINEL_PORT sentinel get-master-addr-by-name $MASTER_GROUP | grep -E '[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}')\"\n if [ \"$MASTER\" ]; then\n sentinel_update \"$MASTER\"\n redis_update \"$MASTER\"\n else\n echo \"Could not failover, exiting...\"\n exit 1\n fi\n else\n echo \"Found reachable master, updating config\"\n sentinel_update \"$MASTER\"\n redis_update \"$MASTER\"\n fi\n }\n\n mkdir -p /data/conf/\n\n echo \"Initializing config..\"\n copy_config\n\n ANNOUNCE_IP=$(getent hosts \"$SERVICE-announce-$INDEX\" | awk '{ print $1 }')\n if [ -z \"$ANNOUNCE_IP\" ]; then\n \"Could not resolve the announce ip for this pod\"\n exit 1\n elif [ \"$MASTER\" ]; then\n find_master\n else\n setup_defaults\n fi\n\n if [ \"${AUTH:-}\" ]; then\n echo \"Setting auth values\"\n ESCAPED_AUTH=$(echo \"$AUTH\" | sed -e 's/[\\/&]/\\\\&/g');\n sed -i \"s/replace-default-auth/${ESCAPED_AUTH}/\" \"$REDIS_CONF\" \"$SENTINEL_CONF\"\n fi\n\n echo \"Ready...\"\n{{- end }}\n\n{{- define \"config-haproxy.cfg\" }}\n{{- if .Values.haproxy.customConfig }}\n{{ .Values.haproxy.customConfig | indent 4}}\n{{- else }}\n defaults REDIS\n mode tcp\n timeout connect {{ .Values.haproxy.timeout.connect }}\n timeout server {{ .Values.haproxy.timeout.server }}\n timeout client {{ .Values.haproxy.timeout.client }}\n timeout check {{ .Values.haproxy.timeout.check }}\n\n listen health_check_http_url\n bind :8888\n mode http\n monitor-uri /healthz\n option dontlognull\n\n {{- $root := . }}\n {{- $fullName := include \"redis-ha.fullname\" . }}\n {{- $replicas := int (toString .Values.replicas) }}\n {{- $masterGroupName := include \"redis-ha.masterGroupName\" . }}\n {{- range $i := until $replicas }}\n # Check Sentinel and whether they are nominated master\n backend check_if_redis_is_master_{{ $i }}\n mode tcp\n option tcp-check\n tcp-check connect\n {{- if $root.auth }}\n tcp-check send AUTH\\ {{ $root.redisPassword }}\\r\\n\n tcp-check expect string +OK\n {{- end }}\n tcp-check send PING\\r\\n\n tcp-check expect string +PONG\n tcp-check send SENTINEL\\ get-master-addr-by-name\\ {{ $masterGroupName }}\\r\\n\n tcp-check expect string REPLACE_ANNOUNCE{{ $i }}\n tcp-check send QUIT\\r\\n\n tcp-check expect string +OK\n {{- range $i := until $replicas }}\n server R{{ $i }} {{ $fullName }}-announce-{{ $i }}:26379 check inter 1s\n {{- end }}\n {{- end }}\n\n # decide redis backend to use\n #master\n frontend ft_redis_master\n bind *:{{ $root.Values.redis.port }}\n use_backend bk_redis_master\n {{- if .Values.haproxy.readOnly.enabled }}\n #slave\n frontend ft_redis_slave\n bind *:{{ .Values.haproxy.readOnly.port }}\n use_backend bk_redis_slave\n {{- end }}\n # Check all redis servers to see if they think they are master\n backend bk_redis_master\n {{- if .Values.haproxy.stickyBalancing }}\n balance source\n hash-type consistent\n {{- end }}\n mode tcp\n option tcp-check\n tcp-check connect\n {{- if .Values.auth }}\n tcp-check send AUTH\\ REPLACE_AUTH_SECRET\\r\\n\n tcp-check expect string +OK\n {{- end }}\n tcp-check send PING\\r\\n\n tcp-check expect string +PONG\n tcp-check send info\\ replication\\r\\n\n tcp-check expect string role:master\n tcp-check send QUIT\\r\\n\n tcp-check expect string +OK\n {{- range $i := until $replicas }}\n use-server R{{ $i }} if { srv_is_up(R{{ $i }}) } { nbsrv(check_if_redis_is_master_{{ $i }}) ge 2 }\n server R{{ $i }} {{ $fullName }}-announce-{{ $i }}:{{ $root.Values.redis.port }} check inter 1s fall 1 rise 1\n {{- end }}\n {{- if .Values.haproxy.readOnly.enabled }}\n backend bk_redis_slave\n {{- if .Values.haproxy.stickyBalancing }}\n balance source\n hash-type consistent\n {{- end }}\n mode tcp\n option tcp-check\n tcp-check connect\n {{- if .Values.auth }}\n tcp-check send AUTH\\ REPLACE_AUTH_SECRET\\r\\n\n tcp-check expect string +OK\n {{- end }}\n tcp-check send PING\\r\\n\n tcp-check expect string +PONG\n tcp-check send info\\ replication\\r\\n\n tcp-check expect string role:slave\n tcp-check send QUIT\\r\\n\n tcp-check expect string +OK\n {{- range $i := until $replicas }}\n server R{{ $i }} {{ $fullName }}-announce-{{ $i }}:{{ $root.Values.redis.port }} check inter 1s fall 1 rise 1\n {{- end }}\n {{- end }}\n {{- if .Values.haproxy.metrics.enabled }}\n frontend metrics\n mode http\n bind *:{{ .Values.haproxy.metrics.port }}\n option http-use-htx\n http-request use-service prometheus-exporter if { path {{ .Values.haproxy.metrics.scrapePath }} }\n {{- end }}\n{{- if .Values.haproxy.extraConfig }}\n # Additional configuration\n{{ .Values.haproxy.extraConfig | indent 4 }}\n{{- end }}\n{{- end }}\n{{- end }}\n\n\n{{- define \"config-haproxy_init.sh\" }}\n HAPROXY_CONF=/data/haproxy.cfg\n cp /readonly/haproxy.cfg \"$HAPROXY_CONF\"\n {{- $fullName := include \"redis-ha.fullname\" . }}\n {{- $replicas := int (toString .Values.replicas) }}\n {{- range $i := until $replicas }}\n for loop in $(seq 1 10); do\n getent hosts {{ $fullName }}-announce-{{ $i }} && break\n echo \"Waiting for service {{ $fullName }}-announce-{{ $i }} to be ready ($loop) ...\" && sleep 1\n done\n ANNOUNCE_IP{{ $i }}=$(getent hosts \"{{ $fullName }}-announce-{{ $i }}\" | awk '{ print $1 }')\n if [ -z \"$ANNOUNCE_IP{{ $i }}\" ]; then\n echo \"Could not resolve the announce ip for {{ $fullName }}-announce-{{ $i }}\"\n exit 1\n fi\n sed -i \"s/REPLACE_ANNOUNCE{{ $i }}/$ANNOUNCE_IP{{ $i }}/\" \"$HAPROXY_CONF\"\n\n if [ \"${AUTH:-}\" ]; then\n echo \"Setting auth values\"\n ESCAPED_AUTH=$(echo \"$AUTH\" | sed -e 's/[\\/&]/\\\\&/g');\n sed -i \"s/REPLACE_AUTH_SECRET/${ESCAPED_AUTH}/\" \"$HAPROXY_CONF\"\n fi\n {{- end }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/_helpers.tpl",
"content": "{{/* vim: set filetype=mustache: */}}\n\n{{/*\nCreate a default fully qualified app name.\nWe truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).\n*/}}\n{{- define \"redis-ha.name\" -}}\n{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n\n{{/*\nCreate a default fully qualified app name.\nWe truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).\n*/}}\n{{- define \"redis-ha.fullname\" -}}\n{{- if .Values.fullnameOverride -}}\n{{- .Values.fullnameOverride | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- $name := default .Chart.Name .Values.nameOverride -}}\n{{- if contains $name .Release.Name -}}\n{{- .Release.Name | trunc 63 | trimSuffix \"-\" -}}\n{{- else -}}\n{{- printf \"%s-%s\" .Release.Name $name | trunc 63 | trimSuffix \"-\" -}}\n{{- end -}}\n{{- end -}}\n{{- end -}}\n\n\n{{/*\nReturn sysctl image\n*/}}\n{{- define \"redis.sysctl.image\" -}}\n{{- $registryName := default \"docker.io\" .Values.sysctlImage.registry -}}\n{{- $tag := default \"latest\" .Values.sysctlImage.tag | toString -}}\n{{- printf \"%s/%s:%s\" $registryName .Values.sysctlImage.repository $tag -}}\n{{- end -}}\n\n{{- /*\nCredit: @technosophos\nhttps://github.com/technosophos/common-chart/\nlabels.standard prints the standard Helm labels.\nThe standard labels are frequently used in metadata.\n*/ -}}\n{{- define \"labels.standard\" -}}\napp: {{ template \"redis-ha.name\" . }}\nheritage: {{ .Release.Service | quote }}\nrelease: {{ .Release.Name | quote }}\nchart: {{ template \"chartref\" . }}\n{{- end -}}\n\n{{- /*\nCredit: @technosophos\nhttps://github.com/technosophos/common-chart/\nchartref prints a chart name and version.\nIt does minimal escaping for use in Kubernetes labels.\nExample output:\n zookeeper-1.2.3\n wordpress-3.2.1_20170219\n*/ -}}\n{{- define \"chartref\" -}}\n {{- replace \"+\" \"_\" .Chart.Version | printf \"%s-%s\" .Chart.Name -}}\n{{- end -}}\n\n{{/*\nCreate the name of the service account to use\n*/}}\n{{- define \"redis-ha.serviceAccountName\" -}}\n{{- if .Values.serviceAccount.create -}}\n {{ default (include \"redis-ha.fullname\" .) .Values.serviceAccount.name }}\n{{- else -}}\n {{ default \"default\" .Values.serviceAccount.name }}\n{{- end -}}\n{{- end -}}\n\n{{- define \"redis-ha.masterGroupName\" -}}\n{{- $masterGroupName := tpl ( .Values.redis.masterGroupName | default \"\") . -}}\n{{- $validMasterGroupName := regexMatch \"^[\\\\w-\\\\.]+$\" $masterGroupName -}}\n{{- if $validMasterGroupName -}}\n{{ $masterGroupName }}\n{{- else -}}\n{{ required \"A valid .Values.redis.masterGroupName entry is required (matching ^[\\\\w-\\\\.]+$)\" \"\"}}\n{{- end -}}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-auth-secret.yaml",
"content": "{{- if and .Values.auth (not .Values.existingSecret) -}}\napiVersion: v1\nkind: Secret\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}\n namespace: {{ .Release.Namespace }}\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\ntype: Opaque\ndata:\n {{ .Values.authKey }}: {{ .Values.redisPassword | b64enc | quote }}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-announce-service.yaml",
"content": "{{- $fullName := include \"redis-ha.fullname\" . }}\n{{- $namespace := .Release.Namespace -}}\n{{- $replicas := int (toString .Values.replicas) }}\n{{- $root := . }}\n{{- range $i := until $replicas }}\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: {{ $fullName }}-announce-{{ $i }}\n namespace: {{ $namespace }}\n labels:\n{{ include \"labels.standard\" $root | indent 4 }}\n annotations:\n service.alpha.kubernetes.io/tolerate-unready-endpoints: \"true\"\n {{- if $root.Values.serviceAnnotations }}\n{{ toYaml $root.Values.serviceAnnotations | indent 4 }}\n {{- end }}\nspec:\n publishNotReadyAddresses: true\n type: ClusterIP\n ports:\n - name: server\n port: {{ $root.Values.redis.port }}\n protocol: TCP\n targetPort: redis\n - name: sentinel\n port: {{ $root.Values.sentinel.port }}\n protocol: TCP\n targetPort: sentinel\n {{- if $root.Values.exporter.enabled }}\n - name: exporter\n port: {{ $root.Values.exporter.port }}\n protocol: TCP\n targetPort: exporter-port\n {{- end }}\n selector:\n release: {{ $root.Release.Name }}\n app: {{ include \"redis-ha.name\" $root }}\n \"statefulset.kubernetes.io/pod-name\": {{ $fullName }}-server-{{ $i }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-configmap.yaml",
"content": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-configmap\n namespace: {{ .Release.Namespace }}\n labels:\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n app: {{ template \"redis-ha.fullname\" . }}\ndata:\n redis.conf: |\n{{- include \"config-redis.conf\" . }}\n\n sentinel.conf: |\n{{- include \"config-sentinel.conf\" . }}\n\n init.sh: |\n{{- include \"config-init.sh\" . }}\n{{ if .Values.haproxy.enabled }}\n haproxy.cfg: |-\n{{- include \"config-haproxy.cfg\" . }}\n{{- end }}\n haproxy_init.sh: |\n{{- include \"config-haproxy_init.sh\" . }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-exporter-script-configmap.yaml",
"content": "{{- if .Values.exporter.script }}\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-exporter-script-configmap\n namespace: {{ .Release.Namespace }}\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\ndata:\n script: {{ toYaml .Values.exporter.script | indent 2 }}\n{{- end }}"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-pdb.yaml",
"content": "{{- if .Values.podDisruptionBudget -}}\napiVersion: policy/v1beta1\nkind: PodDisruptionBudget\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-pdb\n namespace: {{ .Release.Namespace }}\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\nspec:\n selector:\n matchLabels:\n release: {{ .Release.Name }}\n app: {{ template \"redis-ha.name\" . }}\n{{ toYaml .Values.podDisruptionBudget | indent 2 }}\n{{- end -}}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-role.yaml",
"content": "{{- if and .Values.serviceAccount.create .Values.rbac.create }}\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}\n namespace: {{ .Release.Namespace }}\n labels:\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n app: {{ template \"redis-ha.fullname\" . }}\nrules:\n- apiGroups:\n - \"\"\n resources:\n - endpoints\n verbs:\n - get\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-rolebinding.yaml",
"content": "{{- if and .Values.serviceAccount.create .Values.rbac.create }}\nkind: RoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}\n namespace: {{ .Release.Namespace }}\n labels:\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n app: {{ template \"redis-ha.fullname\" . }}\nsubjects:\n- kind: ServiceAccount\n name: {{ template \"redis-ha.serviceAccountName\" . }}\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: Role\n name: {{ template \"redis-ha.fullname\" . }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-service.yaml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}\n namespace: {{ .Release.Namespace }}\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\n{{- if and ( .Values.exporter.enabled ) ( .Values.exporter.serviceMonitor.enabled ) }}\n servicemonitor: enabled\n{{- end }}\n annotations:\n {{- if .Values.serviceAnnotations }}\n{{ toYaml .Values.serviceAnnotations | indent 4 }}\n {{- end }}\nspec:\n type: ClusterIP\n clusterIP: None\n ports:\n - name: server\n port: {{ .Values.redis.port }}\n protocol: TCP\n targetPort: redis\n - name: sentinel\n port: {{ .Values.sentinel.port }}\n protocol: TCP\n targetPort: sentinel\n{{- if .Values.exporter.enabled }}\n - name: exporter-port\n port: {{ .Values.exporter.port }}\n protocol: TCP\n targetPort: exporter-port\n{{- end }}\n selector:\n release: {{ .Release.Name }}\n app: {{ template \"redis-ha.name\" . }}"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-serviceaccount.yaml",
"content": "{{- if .Values.serviceAccount.create }}\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: {{ template \"redis-ha.serviceAccountName\" . }}\n namespace: {{ .Release.Namespace }}\n labels:\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n app: {{ template \"redis-ha.fullname\" . }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-servicemonitor.yaml",
"content": "{{- if and ( .Capabilities.APIVersions.Has \"monitoring.coreos.com/v1\" ) ( .Values.exporter.serviceMonitor.enabled ) ( .Values.exporter.enabled ) }}\napiVersion: monitoring.coreos.com/v1\nkind: ServiceMonitor\nmetadata:\n{{- if .Values.exporter.serviceMonitor.labels }}\n labels:\n{{ toYaml .Values.exporter.serviceMonitor.labels | indent 4}}\n{{- end }}\n name: {{ template \"redis-ha.fullname\" . }}\n namespace: {{ .Release.Namespace }}\n{{- if .Values.exporter.serviceMonitor.namespace }}\n namespace: {{ .Values.exporter.serviceMonitor.namespace }}\n{{- end }}\nspec:\n endpoints:\n - targetPort: {{ .Values.exporter.port }}\n{{- if .Values.exporter.serviceMonitor.interval }}\n interval: {{ .Values.exporter.serviceMonitor.interval }}\n{{- end }}\n{{- if .Values.exporter.serviceMonitor.telemetryPath }}\n path: {{ .Values.exporter.serviceMonitor.telemetryPath }}\n{{- end }}\n{{- if .Values.exporter.serviceMonitor.timeout }}\n scrapeTimeout: {{ .Values.exporter.serviceMonitor.timeout }}\n{{- end }}\n jobLabel: {{ template \"redis-ha.fullname\" . }}\n namespaceSelector:\n matchNames:\n - {{ .Release.Namespace }}\n selector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}\n release: {{ .Release.Name }}\n servicemonitor: enabled\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-ha-statefulset.yaml",
"content": "apiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-server\n namespace: {{ .Release.Namespace }}\n labels:\n {{ template \"redis-ha.fullname\" . }}: replica\n{{ include \"labels.standard\" . | indent 4 }}\nspec:\n selector:\n matchLabels:\n release: {{ .Release.Name }}\n app: {{ template \"redis-ha.name\" . }}\n serviceName: {{ template \"redis-ha.fullname\" . }}\n replicas: {{ .Values.replicas }}\n podManagementPolicy: OrderedReady\n updateStrategy:\n type: RollingUpdate\n template:\n metadata:\n annotations:\n checksum/init-config: {{ print (include \"config-redis.conf\" .) (include \"config-init.sh\" .) | sha256sum }}\n {{- if .Values.podAnnotations }}\n{{ toYaml .Values.podAnnotations | indent 8 }}\n {{- end }}\n {{- if .Values.exporter.enabled }}\n prometheus.io/port: \"{{ .Values.exporter.port }}\"\n prometheus.io/scrape: \"true\"\n prometheus.io/path: {{ .Values.exporter.scrapePath }}\n {{- end }}\n labels:\n release: {{ .Release.Name }}\n app: {{ template \"redis-ha.name\" . }}\n {{ template \"redis-ha.fullname\" . }}: replica\n {{- range $key, $value := .Values.labels }}\n {{ $key }}: {{ $value }}\n {{- end }}\n spec:\n {{- if .Values.schedulerName }}\n schedulerName: \"{{ .Values.schedulerName }}\"\n {{- end }}\n {{- if .Values.nodeSelector }}\n nodeSelector:\n{{ toYaml .Values.nodeSelector | indent 8 }}\n {{- end }}\n {{- if .Values.tolerations }}\n tolerations:\n{{ toYaml .Values.tolerations | indent 8 }}\n {{- end }}\n affinity:\n {{- if .Values.affinity }}\n {{- with .Values.affinity }}\n{{ tpl . $ | indent 8 }}\n {{- end }}\n {{- else }}\n {{- if .Values.additionalAffinities }}\n{{ toYaml .Values.additionalAffinities | indent 8 }}\n {{- end }}\n podAntiAffinity:\n {{- if .Values.hardAntiAffinity }}\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}\n release: {{ .Release.Name }}\n {{ template \"redis-ha.fullname\" . }}: replica\n topologyKey: kubernetes.io/hostname\n {{- else }}\n preferredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}\n release: {{ .Release.Name }}\n {{ template \"redis-ha.fullname\" . }}: replica\n topologyKey: kubernetes.io/hostname\n {{- end }}\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 100\n podAffinityTerm:\n labelSelector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}\n release: {{ .Release.Name }}\n {{ template \"redis-ha.fullname\" . }}: replica\n topologyKey: failure-domain.beta.kubernetes.io/zone\n {{- end }}\n {{- if .Values.imagePullSecrets }}\n imagePullSecrets: {{ toYaml .Values.imagePullSecrets | nindent 8 }}\n {{- end }}\n securityContext:\n{{ toYaml .Values.securityContext | indent 8 }}\n serviceAccountName: {{ template \"redis-ha.serviceAccountName\" . }}\n initContainers:\n {{- if .Values.sysctlImage.enabled }}\n - name: init-sysctl\n image: {{ template \"redis.sysctl.image\" . }}\n imagePullPolicy: {{ .Values.sysctlImage.pullPolicy }}\n resources:\n{{ toYaml .Values.sysctlImage.resources | indent 10 }}\n {{- if .Values.sysctlImage.mountHostSys }}\n volumeMounts:\n - name: host-sys\n mountPath: /host-sys\n {{- end }}\n command:\n{{ toYaml .Values.sysctlImage.command | indent 10 }}\n securityContext:\n runAsNonRoot: false\n privileged: true\n runAsUser: 0\n {{- end }}\n{{- if and .Values.hostPath.path .Values.hostPath.chown }}\n - name: hostpath-chown\n image: {{ .Values.image.repository }}:{{ .Values.image.tag }}\n securityContext:\n runAsNonRoot: false\n runAsUser: 0\n command:\n - chown\n - \"{{ .Values.securityContext.runAsUser }}\"\n - /data\n volumeMounts:\n - name: data\n mountPath: /data\n{{- end }}\n - name: config-init\n image: {{ .Values.image.repository }}:{{ .Values.image.tag }}\n imagePullPolicy: {{ .Values.image.pullPolicy }}\n resources:\n{{ toYaml .Values.init.resources | indent 10 }}\n command:\n - sh\n args:\n - /readonly-config/init.sh\n env:\n{{- $replicas := int (toString .Values.replicas) -}}\n{{- range $i := until $replicas }}\n - name: SENTINEL_ID_{{ $i }}\n value: {{ printf \"%s\\n%s\\nindex: %d\" (include \"redis-ha.name\" $) ($.Release.Name) $i | sha1sum }}\n{{ end -}}\n{{- if .Values.auth }}\n - name: AUTH\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"redis-ha.fullname\" . }}\n {{- end }}\n key: {{ .Values.authKey }}\n{{- end }}\n volumeMounts:\n - name: config\n mountPath: /readonly-config\n readOnly: true\n - name: data\n mountPath: /data\n containers:\n - name: redis\n image: {{ .Values.image.repository }}:{{ .Values.image.tag }}\n imagePullPolicy: {{ .Values.image.pullPolicy }}\n command:\n - redis-server\n args:\n - /data/conf/redis.conf\n {{- if .Values.auth }}\n env:\n - name: AUTH\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"redis-ha.fullname\" . }}\n {{- end }}\n key: {{ .Values.authKey }}\n {{- end }}\n livenessProbe:\n tcpSocket:\n port: {{ .Values.redis.port }}\n initialDelaySeconds: 15\n resources:\n{{ toYaml .Values.redis.resources | indent 10 }}\n ports:\n - name: redis\n containerPort: {{ .Values.redis.port }}\n volumeMounts:\n - mountPath: /data\n name: data\n - name: sentinel\n image: {{ .Values.image.repository }}:{{ .Values.image.tag }}\n imagePullPolicy: {{ .Values.image.pullPolicy }}\n command:\n - redis-sentinel\n args:\n - /data/conf/sentinel.conf\n{{- if .Values.auth }}\n env:\n - name: AUTH\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"redis-ha.fullname\" . }}\n {{- end }}\n key: {{ .Values.authKey }}\n{{- end }}\n livenessProbe:\n tcpSocket:\n port: {{ .Values.sentinel.port }}\n initialDelaySeconds: 15\n resources:\n{{ toYaml .Values.sentinel.resources | indent 10 }}\n ports:\n - name: sentinel\n containerPort: {{ .Values.sentinel.port }}\n volumeMounts:\n - mountPath: /data\n name: data\n{{- if .Values.exporter.enabled }}\n - name: redis-exporter\n image: \"{{ .Values.exporter.image }}:{{ .Values.exporter.tag }}\"\n imagePullPolicy: {{ .Values.exporter.pullPolicy }}\n args:\n {{- range $key, $value := .Values.exporter.extraArgs }}\n - --{{ $key }}={{ $value }}\n {{- end }}\n env:\n - name: REDIS_ADDR\n value: redis://localhost:{{ .Values.redis.port }}\n {{- if .Values.auth }}\n - name: REDIS_PASSWORD\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"redis-ha.fullname\" . }}\n {{- end }}\n key: {{ .Values.authKey }}\n {{- end }}\n {{- if .Values.exporter.script }}\n - name: REDIS_EXPORTER_SCRIPT\n value: /script/script.lua\n {{- end }}\n livenessProbe:\n httpGet:\n path: {{ .Values.exporter.scrapePath }}\n port: {{ .Values.exporter.port }}\n initialDelaySeconds: 15\n timeoutSeconds: 1\n periodSeconds: 15\n resources:\n{{ toYaml .Values.exporter.resources | indent 10 }}\n ports:\n - name: exporter-port\n containerPort: {{ .Values.exporter.port }}\n {{- if .Values.exporter.script }}\n volumeMounts:\n - mountPath: /script\n name: script-mount\n {{- end }}\n{{- end }}\n{{- if .Values.priorityClassName }}\n priorityClassName: {{ .Values.priorityClassName }}\n{{- end }}\n volumes:\n - name: config\n configMap:\n name: {{ template \"redis-ha.fullname\" . }}-configmap\n {{- if .Values.sysctlImage.mountHostSys }}\n - name: host-sys\n hostPath:\n path: /sys\n {{- end }}\n {{- if .Values.exporter.script }}\n - name: script-mount\n configMap:\n name: {{ template \"redis-ha.fullname\" . }}-exporter-script-configmap\n items:\n - key: script\n path: script.lua\n {{- end }}\n{{- if .Values.persistentVolume.enabled }}\n volumeClaimTemplates:\n - metadata:\n name: data\n annotations:\n {{- range $key, $value := .Values.persistentVolume.annotations }}\n {{ $key }}: {{ $value }}\n {{- end }}\n spec:\n accessModes:\n {{- range .Values.persistentVolume.accessModes }}\n - {{ . | quote }}\n {{- end }}\n resources:\n requests:\n storage: {{ .Values.persistentVolume.size | quote }}\n {{- if .Values.persistentVolume.storageClass }}\n {{- if (eq \"-\" .Values.persistentVolume.storageClass) }}\n storageClassName: \"\"\n {{- else }}\n storageClassName: \"{{ .Values.persistentVolume.storageClass }}\"\n {{- end }}\n {{- end }}\n {{- if .Values.persistentVolume.reclaimPolicy }}\n persistentVolumeReclaimPolicy: \"{{ .Values.persistentVolume.reclaimPolicy }}\"\n {{- end }}\n{{- else if .Values.hostPath.path }}\n - name: data\n hostPath:\n path: {{ tpl .Values.hostPath.path .}}\n{{- else }}\n - name: data\n emptyDir:\n{{ toYaml .Values.emptyDir | indent 10 }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-haproxy-deployment.yaml",
"content": "{{- if .Values.haproxy.enabled }}\nkind: Deployment\napiVersion: apps/v1\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-haproxy\n namespace: {{ .Release.Namespace }}\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\nspec:\n strategy:\n type: RollingUpdate\n revisionHistoryLimit: 1\n replicas: {{ .Values.haproxy.replicas }}\n selector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}-haproxy\n release: {{ .Release.Name }}\n template:\n metadata:\n name: {{ template \"redis-ha.fullname\" . }}-haproxy\n labels:\n app: {{ template \"redis-ha.name\" . }}-haproxy\n release: {{ .Release.Name }}\n annotations:\n {{- if .Values.haproxy.metrics.enabled }}\n prometheus.io/port: \"{{ .Values.haproxy.metrics.port }}\"\n prometheus.io/scrape: \"true\"\n prometheus.io/path: \"{{ .Values.haproxy.metrics.scrapePath }}\"\n {{- end }}\n checksum/config: {{ print (include \"config-haproxy.cfg\" .) (include \"config-haproxy_init.sh\" .) | sha256sum }}\n {{- if .Values.haproxy.annotations }}\n{{ toYaml .Values.haproxy.annotations | indent 8 }}\n {{- end }}\n spec:\n # Needed when using unmodified rbac-setup.yml\n {{ if .Values.haproxy.serviceAccount.create }}\n serviceAccountName: {{ template \"redis-ha.serviceAccountName\" . }}-haproxy\n {{ end }}\n nodeSelector:\n{{ toYaml .Values.nodeSelector | indent 8 }}\n tolerations:\n{{ toYaml .Values.tolerations | indent 8 }}\n affinity:\n {{- if .Values.haproxy.affinity }}\n {{- with .Values.haproxy.affinity }}\n{{ tpl . $ | indent 8 }}\n {{- end }}\n {{- else }}\n {{- if .Values.haproxy.additionalAffinities }}\n{{ toYaml .Values.haproxy.additionalAffinities | indent 8 }}\n {{- end }}\n podAntiAffinity:\n {{- if .Values.haproxy.hardAntiAffinity }}\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}-haproxy\n release: {{ .Release.Name }}\n topologyKey: kubernetes.io/hostname\n {{- else }}\n preferredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}-haproxy\n release: {{ .Release.Name }}\n topologyKey: kubernetes.io/hostname\n {{- end }}\n preferredDuringSchedulingIgnoredDuringExecution:\n - weight: 100\n podAffinityTerm:\n labelSelector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}-haproxy\n release: {{ .Release.Name }}\n topologyKey: failure-domain.beta.kubernetes.io/zone\n {{- end }}\n initContainers:\n - name: config-init\n image: {{ .Values.haproxy.image.repository }}:{{ .Values.haproxy.image.tag }}\n imagePullPolicy: {{ .Values.haproxy.image.pullPolicy }}\n resources:\n{{ toYaml .Values.haproxy.init.resources | indent 10 }}\n command:\n - sh\n args:\n - /readonly/haproxy_init.sh\n{{- if .Values.auth }}\n env:\n - name: AUTH\n valueFrom:\n secretKeyRef:\n {{- if .Values.existingSecret }}\n name: {{ .Values.existingSecret }}\n {{- else }}\n name: {{ template \"redis-ha.fullname\" . }}\n {{- end }}\n key: {{ .Values.authKey }}\n{{- end }}\n volumeMounts:\n - name: config-volume\n mountPath: /readonly\n readOnly: true\n - name: data\n mountPath: /data\n {{- if .Values.haproxy.imagePullSecrets }}\n imagePullSecrets: {{ toYaml .Values.haproxy.imagePullSecrets | nindent 8 }}\n {{- end }}\n securityContext:\n{{ toYaml .Values.haproxy.securityContext | indent 8 }}\n containers:\n - name: haproxy\n image: {{ .Values.haproxy.image.repository }}:{{ .Values.haproxy.image.tag }}\n imagePullPolicy: {{ .Values.haproxy.image.pullPolicy }}\n livenessProbe:\n httpGet:\n path: /healthz\n port: 8888\n initialDelaySeconds: 5\n periodSeconds: 3\n ports:\n - name: redis\n containerPort: {{ default \"6379\" .Values.redis.port }}\n {{- if .Values.haproxy.readOnly.enabled }}\n - name: readonlyport\n containerPort: {{ default \"6380\" .Values.haproxy.readOnly.port }}\n {{- end }}\n {{- if .Values.haproxy.metrics.enabled }}\n - name: metrics-port\n containerPort: {{ default \"9101\" .Values.haproxy.metrics.port }}\n {{- end }}\n resources:\n{{ toYaml .Values.haproxy.resources | indent 10 }}\n volumeMounts:\n - name: data\n mountPath: /usr/local/etc/haproxy\n - name: shared-socket\n mountPath: /run/haproxy\n{{- if .Values.haproxy.priorityClassName }}\n priorityClassName: {{ .Values.haproxy.priorityClassName }}\n{{- end }}\n volumes:\n - name: config-volume\n configMap:\n name: {{ template \"redis-ha.fullname\" . }}-configmap\n - name: shared-socket\n emptyDir: \n{{ toYaml .Values.haproxy.emptyDir | indent 10 }}\n - name: data\n emptyDir: \n{{ toYaml .Values.haproxy.emptyDir | indent 10 }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-haproxy-service.yaml",
"content": "{{- if .Values.haproxy.enabled }}\napiVersion: v1\nkind: Service\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-haproxy\n namespace: {{ .Release.Namespace }}\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\n component: {{ template \"redis-ha.fullname\" . }}-haproxy\n annotations:\n {{- if .Values.haproxy.service.annotations }}\n{{ toYaml .Values.haproxy.service.annotations | indent 4 }}\n {{- end }}\nspec:\n type: {{ default \"ClusterIP\" .Values.haproxy.service.type }}\n {{- if and (eq .Values.haproxy.service.type \"LoadBalancer\") .Values.haproxy.service.loadBalancerIP }}\n loadBalancerIP: {{ .Values.haproxy.service.loadBalancerIP }}\n {{- end }}\n ports:\n - name: haproxy\n port: {{ .Values.redis.port }}\n protocol: TCP\n targetPort: redis\n {{- if and (eq .Values.haproxy.service.type \"NodePort\") .Values.haproxy.service.nodePort }}\n nodePort: {{ .Values.haproxy.service.nodePort }}\n {{- end }}\n{{- if .Values.haproxy.readOnly.enabled }}\n - name: haproxyreadonly\n port: {{ .Values.haproxy.readOnly.port }}\n protocol: TCP\n targetPort: {{ .Values.haproxy.readOnly.port }}\n{{- end }}\n{{- if .Values.haproxy.metrics.enabled }}\n - name: {{ .Values.haproxy.metrics.portName }}\n port: {{ .Values.haproxy.metrics.port }}\n protocol: TCP\n targetPort: metrics-port\n{{- end }}\n selector:\n release: {{ .Release.Name }}\n app: {{ template \"redis-ha.name\" . }}-haproxy\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-haproxy-serviceaccount.yaml",
"content": "{{- if and .Values.haproxy.serviceAccount.create .Values.haproxy.enabled }}\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: {{ template \"redis-ha.serviceAccountName\" . }}-haproxy\n namespace: {{ .Release.Namespace }}\n labels:\n heritage: {{ .Release.Service }}\n release: {{ .Release.Name }}\n chart: {{ .Chart.Name }}-{{ .Chart.Version }}\n app: {{ template \"redis-ha.fullname\" . }}\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/redis-haproxy-servicemonitor.yaml",
"content": "{{- if and ( .Capabilities.APIVersions.Has \"monitoring.coreos.com/v1\" ) ( .Values.haproxy.metrics.serviceMonitor.enabled ) ( .Values.haproxy.metrics.enabled ) }}\napiVersion: monitoring.coreos.com/v1\nkind: ServiceMonitor\nmetadata:\n{{- with .Values.haproxy.metrics.serviceMonitor.labels }}\n labels: {{ toYaml . | nindent 4}}\n{{- end }}\n name: {{ template \"redis-ha.fullname\" . }}-haproxy\n namespace: {{ .Release.Namespace }}\n{{- if .Values.haproxy.metrics.serviceMonitor.namespace }}\n namespace: {{ .Values.haproxy.metrics.serviceMonitor.namespace }}\n{{- end }}\nspec:\n endpoints:\n - targetPort: {{ .Values.haproxy.metrics.port }}\n{{- if .Values.haproxy.metrics.serviceMonitor.interval }}\n interval: {{ .Values.haproxy.metrics.serviceMonitor.interval }}\n{{- end }}\n{{- if .Values.haproxy.metrics.serviceMonitor.telemetryPath }}\n path: {{ .Values.haproxy.metrics.serviceMonitor.telemetryPath }}\n{{- end }}\n{{- if .Values.haproxy.metrics.serviceMonitor.timeout }}\n scrapeTimeout: {{ .Values.haproxy.metrics.serviceMonitor.timeout }}\n{{- end }}\n jobLabel: {{ template \"redis-ha.fullname\" . }}-haproxy\n namespaceSelector:\n matchNames:\n - {{ .Release.Namespace }}\n selector:\n matchLabels:\n app: {{ template \"redis-ha.name\" . }}\n release: {{ .Release.Name }}\n component: {{ template \"redis-ha.fullname\" . }}-haproxy\n{{- end }}\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/tests/test-redis-ha-configmap.yaml",
"content": "apiVersion: v1\nkind: Pod\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-configmap-test\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\n annotations:\n \"helm.sh/hook\": test-success\nspec:\n containers:\n - name: check-init\n image: koalaman/shellcheck:v0.5.0\n args:\n - --shell=sh\n - /readonly-config/init.sh\n volumeMounts:\n - name: config\n mountPath: /readonly-config\n readOnly: true\n {{- if .Values.imagePullSecrets }}\n imagePullSecrets: {{ toYaml .Values.imagePullSecrets | nindent 4 }}\n {{- end }}\n restartPolicy: Never\n volumes:\n - name: config\n configMap:\n name: {{ template \"redis-ha.fullname\" . }}-configmap\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/templates/tests/test-redis-ha-pod.yaml",
"content": "apiVersion: v1\nkind: Pod\nmetadata:\n name: {{ template \"redis-ha.fullname\" . }}-service-test\n labels:\n{{ include \"labels.standard\" . | indent 4 }}\n annotations:\n \"helm.sh/hook\": test-success\nspec:\n containers:\n - name: \"{{ .Release.Name }}-service-test\"\n image: {{ .Values.image.repository }}:{{ .Values.image.tag }}\n command:\n - sh\n - -c\n - redis-cli -h {{ template \"redis-ha.fullname\" . }} -p {{ .Values.redis.port }} info server\n {{- if .Values.imagePullSecrets }}\n imagePullSecrets: {{ toYaml .Values.imagePullSecrets | nindent 4 }}\n {{- end }}\n restartPolicy: Never\n"
},
{
"path": "manifests/deprecated/redis-cluster/redis-ha/values.yaml",
"content": "## Configure resource requests and limits\n## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n##\nimage:\n repository: redis\n tag: 5.0.6-alpine\n pullPolicy: IfNotPresent\n\n## Reference to one or more secrets to be used when pulling images\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/\n## This imagePullSecrets is only for redis images\n##\nimagePullSecrets: []\n# - name: \"image-pull-secret\"\n\n## replicas number for each component\nreplicas: 3\n\n## Kubernetes priorityClass name for the redis-ha-server pod\n# priorityClassName: \"\"\n\n## Custom labels for the redis pod\nlabels: {}\n\n## Pods Service Account\n## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\nserviceAccount:\n ## Specifies whether a ServiceAccount should be created\n ##\n create: true\n ## The name of the ServiceAccount to use.\n ## If not set and create is true, a name is generated using the redis-ha.fullname template\n # name:\n\n## Enables a HA Proxy for better LoadBalancing / Sentinel Master support. Automatically proxies to Redis master.\n## Recommend for externally exposed Redis clusters.\n## ref: https://cbonte.github.io/haproxy-dconv/1.9/intro.html\nhaproxy:\n enabled: false\n # Enable if you want a dedicated port in haproxy for redis-slaves\n readOnly:\n enabled: false\n port: 6380\n replicas: 3\n image:\n repository: haproxy\n tag: 2.0.4\n pullPolicy: IfNotPresent\n\n ## Reference to one or more secrets to be used when pulling images\n ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/\n ##\n imagePullSecrets: []\n # - name: \"image-pull-secret\"\n\n annotations: {}\n resources: {}\n emptyDir: {}\n ## Enable sticky sessions to Redis nodes via HAProxy\n ## Very useful for long-living connections as in case of Sentry for example\n stickyBalancing: false\n ## Kubernetes priorityClass name for the haproxy pod\n # priorityClassName: \"\"\n ## Service type for HAProxy\n ##\n service:\n type: ClusterIP\n loadBalancerIP:\n annotations: {}\n serviceAccount:\n create: true\n ## Official HAProxy embedded prometheus metrics settings.\n ## Ref: https://github.com/haproxy/haproxy/tree/master/contrib/prometheus-exporter\n ##\n metrics:\n enabled: false\n # prometheus port & scrape path\n port: 9101\n portName: exporter-port\n scrapePath: /metrics\n\n serviceMonitor:\n # When set true then use a ServiceMonitor to configure scraping\n enabled: false\n # Set the namespace the ServiceMonitor should be deployed\n # namespace: monitoring\n # Set how frequently Prometheus should scrape\n # interval: 30s\n # Set path to redis-exporter telemtery-path\n # telemetryPath: /metrics\n # Set labels for the ServiceMonitor, use this to define your scrape label for Prometheus Operator\n # labels: {}\n # Set timeout for scrape\n # timeout: 10s\n init:\n resources: {}\n timeout:\n connect: 4s\n server: 30s\n client: 30s\n check: 2s\n securityContext:\n runAsUser: 1000\n fsGroup: 1000\n runAsNonRoot: true\n\n ## Whether the haproxy pods should be forced to run on separate nodes.\n hardAntiAffinity: true\n\n ## Additional affinities to add to the haproxy pods.\n additionalAffinities: {}\n\n ## Override all other affinity settings for the haproxy pods with a string.\n affinity: |\n\n ## Custom config-haproxy.cfg files used to override default settings. If this file is\n ## specified then the config-haproxy.cfg above will be ignored.\n # customConfig: |-\n # Define configuration here\n ## Place any additional configuration section to add to the default config-haproxy.cfg\n # extraConfig: |-\n # Define configuration here\n\n\n## Role Based Access\n## Ref: https://kubernetes.io/docs/admin/authorization/rbac/\n##\nrbac:\n create: true\n\nsysctlImage:\n enabled: false\n command: []\n registry: docker.io\n repository: busybox\n tag: 1.31.1\n pullPolicy: Always\n mountHostSys: false\n resources: {}\n\n## Use an alternate scheduler, e.g. \"stork\".\n## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/\n##\n# schedulerName:\n\n## Redis specific configuration options\nredis:\n port: 6379\n masterGroupName: \"mymaster\" # must match ^[\\\\w-\\\\.]+$) and can be templated\n config:\n ## Additional redis conf options can be added below\n ## For all available options see http://download.redis.io/redis-stable/redis.conf\n min-replicas-to-write: 1\n min-replicas-max-lag: 5 # Value in seconds\n maxmemory: \"0\" # Max memory to use for each redis instance. Default is unlimited.\n maxmemory-policy: \"volatile-lru\" # Max memory policy to use for each redis instance. Default is volatile-lru.\n # Determines if scheduled RDB backups are created. Default is false.\n # Please note that local (on-disk) RDBs will still be created when re-syncing with a new slave. The only way to prevent this is to enable diskless replication.\n save: \"900 1\"\n # When enabled, directly sends the RDB over the wire to slaves, without using the disk as intermediate storage. Default is false.\n repl-diskless-sync: \"yes\"\n rdbcompression: \"yes\"\n rdbchecksum: \"yes\"\n\n\n ## Custom redis.conf files used to override default settings. If this file is\n ## specified then the redis.config above will be ignored.\n # customConfig: |-\n # Define configuration here\n\n resources: {}\n # requests:\n # memory: 200Mi\n # cpu: 100m\n # limits:\n # memory: 700Mi\n\n## Sentinel specific configuration options\nsentinel:\n port: 26379\n quorum: 2\n config:\n ## Additional sentinel conf options can be added below. Only options that\n ## are expressed in the format simialar to 'sentinel xxx mymaster xxx' will\n ## be properly templated expect maxclients option.\n ## For available options see http://download.redis.io/redis-stable/sentinel.conf\n down-after-milliseconds: 10000\n ## Failover timeout value in milliseconds\n failover-timeout: 180000\n parallel-syncs: 5\n maxclients: 10000\n\n ## Custom sentinel.conf files used to override default settings. If this file is\n ## specified then the sentinel.config above will be ignored.\n # customConfig: |-\n # Define configuration here\n\n resources: {}\n # requests:\n # memory: 200Mi\n # cpu: 100m\n # limits:\n # memory: 200Mi\n\nsecurityContext:\n runAsUser: 1000\n fsGroup: 1000\n runAsNonRoot: true\n\n## Node labels, affinity, and tolerations for pod assignment\n## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector\n## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature\n## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\nnodeSelector: {}\n\n## Whether the Redis server pods should be forced to run on separate nodes.\n## This is accomplished by setting their AntiAffinity with requiredDuringSchedulingIgnoredDuringExecution as opposed to preferred.\n## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature\n##\nhardAntiAffinity: true\n\n## Additional affinities to add to the Redis server pods.\n##\n## Example:\n## nodeAffinity:\n## preferredDuringSchedulingIgnoredDuringExecution:\n## - weight: 50\n## preference:\n## matchExpressions:\n## - key: spot\n## operator: NotIn\n## values:\n## - \"true\"\n##\n## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity\n##\nadditionalAffinities: {}\n\n## Override all other affinity settings for the Redis server pods with a string.\n##\n## Example:\n## affinity: |\n## podAntiAffinity:\n## requiredDuringSchedulingIgnoredDuringExecution:\n## - labelSelector:\n## matchLabels:\n## app: {{ template \"redis-ha.name\" . }}\n## release: {{ .Release.Name }}\n## topologyKey: kubernetes.io/hostname\n## preferredDuringSchedulingIgnoredDuringExecution:\n## - weight: 100\n## podAffinityTerm:\n## labelSelector:\n## matchLabels:\n## app: {{ template \"redis-ha.name\" . }}\n## release: {{ .Release.Name }}\n## topologyKey: failure-domain.beta.kubernetes.io/zone\n##\naffinity: |\n\n# Prometheus exporter specific configuration options\nexporter:\n enabled: false\n image: oliver006/redis_exporter\n tag: v1.3.2\n pullPolicy: IfNotPresent\n\n # prometheus port & scrape path\n port: 9121\n scrapePath: /metrics\n\n # cpu/memory resource limits/requests\n resources: {}\n\n # Additional args for redis exporter\n extraArgs: {}\n\n # Used to mount a LUA-Script via config map and use it for metrics-collection\n # script: |\n # -- Example script copied from: https://github.com/oliver006/redis_exporter/blob/master/contrib/sample_collect_script.lua\n # -- Example collect script for -script option\n # -- This returns a Lua table with alternating keys and values.\n # -- Both keys and values must be strings, similar to a HGETALL result.\n # -- More info about Redis Lua scripting: https://redis.io/commands/eval\n #\n # local result = {}\n #\n # -- Add all keys and values from some hash in db 5\n # redis.call(\"SELECT\", 5)\n # local r = redis.call(\"HGETALL\", \"some-hash-with-stats\")\n # if r ~= nil then\n # for _,v in ipairs(r) do\n # table.insert(result, v) -- alternating keys and values\n # end\n # end\n #\n # -- Set foo to 42\n # table.insert(result, \"foo\")\n # table.insert(result, \"42\") -- note the string, use tostring() if needed\n #\n # return result\n\n serviceMonitor:\n # When set true then use a ServiceMonitor to configure scraping\n enabled: false\n # Set the namespace the ServiceMonitor should be deployed\n # namespace: monitoring\n # Set how frequently Prometheus should scrape\n # interval: 30s\n # Set path to redis-exporter telemtery-path\n # telemetryPath: /metrics\n # Set labels for the ServiceMonitor, use this to define your scrape label for Prometheus Operator\n # labels: {}\n # Set timeout for scrape\n # timeout: 10s\n\npodDisruptionBudget: {}\n # maxUnavailable: 1\n # minAvailable: 1\n\n## Configures redis with AUTH (requirepass & masterauth conf params)\nauth: false\n# redisPassword:\n\n## Use existing secret containing key `authKey` (ignores redisPassword)\n# existingSecret:\n\n## Defines the key holding the redis password in existing secret.\nauthKey: auth\n\npersistentVolume:\n enabled: true\n ## redis-ha data Persistent Volume Storage Class\n ## If defined, storageClassName: \n ## If set to \"-\", storageClassName: \"\", which disables dynamic provisioning\n ## If undefined (the default) or set to null, no storageClassName spec is\n ## set, choosing the default provisioner. (gp2 on AWS, standard on\n ## GKE, AWS & OpenStack)\n ##\n # storageClass: \"-\"\n accessModes:\n - ReadWriteOnce\n size: 10Gi\n annotations: {}\n # reclaimPolicy per https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming\n reclaimPolicy: \"\"\ninit:\n resources: {}\n\n# To use a hostPath for data, set persistentVolume.enabled to false\n# and define hostPath.path.\n# Warning: this might overwrite existing folders on the host system!\nhostPath:\n ## path is evaluated as template so placeholders are replaced\n # path: \"/data/{{ .Release.Name }}\"\n\n # if chown is true, an init-container with root permissions is launched to\n # change the owner of the hostPath folder to the user defined in the\n # security context\n chown: true\n\nemptyDir: {}\n"
},
{
"path": "manifests/deprecated/redis-cluster/start.sh",
"content": "#!/bin/sh\nset -x\n\nROOT=$(cd `dirname $0`; pwd)\ncd $ROOT\n\nhelm install redis \\\n\t--create-namespace \\\n\t--namespace dependency \\\n\t-f ./values.yaml \\\n\t./redis-ha\n"
},
{
"path": "manifests/deprecated/redis-cluster/values.yaml",
"content": "image:\n repository: redis\n tag: 5.0.6-alpine\n\nreplicas: 2\n\n## Redis specific configuration options\nredis:\n port: 6379\n masterGroupName: \"mymaster\" # must match ^[\\\\w-\\\\.]+$) and can be templated\n config:\n ## For all available options see http://download.redis.io/redis-stable/redis.conf\n min-replicas-to-write: 1\n min-replicas-max-lag: 5 # Value in seconds\n maxmemory: \"4g\" # Max memory to use for each redis instance. Default is unlimited.\n maxmemory-policy: \"allkeys-lru\" # Max memory policy to use for each redis instance. Default is volatile-lru.\n repl-diskless-sync: \"yes\"\n rdbcompression: \"yes\"\n rdbchecksum: \"yes\"\n\n resources:\n requests:\n memory: 200Mi\n cpu: 100m\n limits:\n memory: 4000Mi\n\n## Sentinel specific configuration options\nsentinel:\n port: 26379\n quorum: 1\n\n resources:\n requests:\n memory: 200Mi\n cpu: 100m\n limits:\n memory: 200Mi\n\nhardAntiAffinity: true\n\n## Configures redis with AUTH (requirepass & masterauth conf params)\nauth: false\n\npersistentVolume:\n enabled: false\n\nhostPath:\n path: \"/data/mcs-redis/{{ .Release.Name }}\"\n"
},
{
"path": "manifests/deprecated/storage/local-storage/example-sts.yml",
"content": "apiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n name: local-test\nspec:\n serviceName: \"\"\n replicas: 2\n selector:\n matchLabels:\n app: local-test\n template:\n metadata:\n labels:\n app: local-test\n spec:\n containers:\n - name: test-container\n image: busybox\n command:\n - \"/bin/sh\"\n args:\n - \"-c\"\n - \"sleep 100000\"\n volumeMounts:\n - name: local-vol\n mountPath: /usr/test-pod\n volumeClaimTemplates:\n - metadata:\n name: local-vol\n spec:\n accessModes: [ \"ReadWriteOnce\" ]\n storageClassName: \"local-storage\"\n resources:\n requests:\n storage: 5Gi\n"
},
{
"path": "manifests/deprecated/storage/local-storage/local-pv1.yml",
"content": "apiVersion: v1\nkind: PersistentVolume\nmetadata:\n name: local-pv1\nspec:\n capacity:\n storage: 5Gi\n volumeMode: Filesystem\n accessModes:\n - ReadWriteOnce\n persistentVolumeReclaimPolicy: Delete\n storageClassName: local-storage\n local:\n path: /mnt/disks/vol1\n nodeAffinity:\n required:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/hostname\n operator: In\n values:\n - 192.168.1.2\n - 192.168.1.3\n"
},
{
"path": "manifests/deprecated/storage/local-storage/local-pv2.yml",
"content": "apiVersion: v1\nkind: PersistentVolume\nmetadata:\n name: local-pv2\nspec:\n capacity:\n storage: 5Gi\n volumeMode: Filesystem\n accessModes:\n - ReadWriteOnce\n persistentVolumeReclaimPolicy: Delete\n storageClassName: local-storage\n local:\n path: /mnt/disks/vol2\n nodeAffinity:\n required:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/hostname\n operator: In\n values:\n - 192.168.1.4\n"
},
{
"path": "manifests/deprecated/storage/local-storage/local-storage-class.yml",
"content": "kind: StorageClass\napiVersion: storage.k8s.io/v1\nmetadata:\n name: local-storage\nprovisioner: kubernetes.io/no-provisioner\nvolumeBindingMode: WaitForFirstConsumer\n"
},
{
"path": "manifests/deprecated/storage/test.yaml",
"content": "kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: test-claim\nspec:\n storageClassName: nfs-dynamic-class\n accessModes:\n - ReadWriteMany\n resources:\n requests:\n storage: 1Mi\n\n---\nkind: Pod\napiVersion: v1\nmetadata:\n name: test\nspec:\n containers:\n - name: test\n image: busybox:1.28.4\n imagePullPolicy: IfNotPresent\n command:\n - \"/bin/sh\"\n args:\n - \"-c\"\n - \"echo 'hello k8s' > /mnt/SUCCESS && sleep 36000 || exit 1\"\n volumeMounts:\n - name: nfs-pvc\n mountPath: \"/mnt\"\n restartPolicy: \"Never\"\n volumes:\n - name: nfs-pvc\n persistentVolumeClaim:\n claimName: test-claim\n"
},
{
"path": "playbooks/01.prepare.yml",
"content": "# [optional] to synchronize system time of nodes with 'chrony' \n- hosts:\n - kube_master\n - kube_node\n - etcd\n - ex_lb\n - chrony\n roles:\n - { role: os-harden, when: \"OS_HARDEN|bool\" }\n - { role: chrony, when: \"groups['chrony']|length > 0\" }\n\n# to create CA, kubeconfig, kube-proxy.kubeconfig etc.\n- hosts: localhost\n roles:\n - deploy\n\n# prepare tasks for all nodes\n- hosts:\n - kube_master\n - kube_node\n - etcd\n roles:\n - prepare\n"
},
{
"path": "playbooks/02.etcd.yml",
"content": "# to install etcd cluster\n- hosts: etcd\n roles:\n - etcd\n"
},
{
"path": "playbooks/03.runtime.yml",
"content": "# to install a container runtime\n- hosts:\n - kube_master\n - kube_node\n roles:\n - { role: docker, when: \"CONTAINER_RUNTIME == 'docker'\" }\n - { role: containerd, when: \"CONTAINER_RUNTIME == 'containerd'\" }\n"
},
{
"path": "playbooks/04.kube-master.yml",
"content": "# to set up 'kube_master' nodes\n- hosts: kube_master\n roles:\n - kube-lb\n - kube-master\n - kube-node\n"
},
{
"path": "playbooks/05.kube-node.yml",
"content": "# to set up 'kube_node' nodes\n- hosts: kube_node\n roles:\n - { role: kube-lb, when: \"inventory_hostname not in groups['kube_master']\" }\n - { role: kube-node, when: \"inventory_hostname not in groups['kube_master']\" }\n"
},
{
"path": "playbooks/06.network.yml",
"content": "# to install network plugin, only one can be choosen \n- hosts:\n - kube_master\n - kube_node\n roles:\n - { role: calico, when: \"CLUSTER_NETWORK == 'calico'\" }\n - { role: cilium, when: \"CLUSTER_NETWORK == 'cilium'\" }\n - { role: flannel, when: \"CLUSTER_NETWORK == 'flannel'\" }\n - { role: kube-router, when: \"CLUSTER_NETWORK == 'kube-router'\" }\n - { role: kube-ovn, when: \"CLUSTER_NETWORK == 'kube-ovn'\" }\n"
},
{
"path": "playbooks/07.cluster-addon.yml",
"content": "# to install clust-addons\n- hosts: localhost\n roles:\n - cluster-addon\n"
},
{
"path": "playbooks/10.ex-lb.yml",
"content": "- hosts: ex_lb\n roles:\n - ex-lb\n"
},
{
"path": "playbooks/11.harbor.yml",
"content": "# [optional] to set up a HARBOR, and to integrate the HARBOR with k8s cluster\n# read the guide: 'guide/harbor.md'\n\n### --- install harbor ---\n- hosts: harbor\n roles:\n - { role: os-harden, when: \"NEW_INSTALL|bool and OS_HARDEN|bool\" }\n - { role: chrony, when: \"NEW_INSTALL|bool and groups['chrony']|length > 0\" }\n - { role: prepare, when: \"NEW_INSTALL|bool\" }\n - { role: docker, when: \"NEW_INSTALL|bool\" }\n - { role: harbor, when: \"NEW_INSTALL|bool\" }\n\n### --- config k8s nodes to work with harbor ---\n- hosts: \n - kube_master\n - kube_node\n tasks:\n # [optional] if you have a DNS server, add an 'A record' instead\n - name: Adding an '/etc/hosts' entry for the HARBOR DOMAIN\n lineinfile:\n dest: /etc/hosts\n state: present\n regexp: '{{ HARBOR_DOMAIN }}'\n line: \"{{ groups['harbor'][0] }} {{ HARBOR_DOMAIN }}\"\n when: \"hostvars[groups.harbor[0]]['HARBOR_DOMAIN'] != ''\"\n"
},
{
"path": "playbooks/21.addetcd.yml",
"content": "# add new-etcd node, one at a time\n- hosts: \"{{ NODE_TO_ADD }}\"\n tasks:\n # step1: find a healthy member in the etcd cluster\n - block:\n - name: set NODE_IPS of the etcd cluster\n set_fact: NODE_IPS=\"{% for host in groups['etcd'] %}{{ host }} {% endfor %}\"\n\n - name: get etcd cluster status\n shell: 'for ip in {{ NODE_IPS }};do \\\n ETCDCTL_API=3 {{ base_dir }}/bin/etcdctl \\\n --endpoints=https://\"$ip\":2379 \\\n --cacert={{ cluster_dir }}/ssl/ca.pem \\\n --cert={{ cluster_dir }}/ssl/etcd.pem \\\n --key={{ cluster_dir }}/ssl/etcd-key.pem \\\n endpoint health; \\\n done'\n register: ETCD_CLUSTER_STATUS\n\n - debug: var=\"ETCD_CLUSTER_STATUS\"\n\n - name: get a running ectd node\n shell: 'echo -e \"{{ ETCD_CLUSTER_STATUS.stdout }}\" \\\n \"{{ ETCD_CLUSTER_STATUS.stderr }}\" \\\n |grep \"is healthy\"|sed -n \"1p\"|cut -d: -f2|cut -d/ -f3'\n register: RUNNING_NODE\n\n - debug: var=\"RUNNING_NODE.stdout\"\n connection: local\n\n # step2: add a new member in the etcd cluster\n - name: add a new etcd member\n shell: \"ETCDCTL_API=3 {{ bin_dir }}/etcdctl member add etcd-{{ NODE_TO_ADD }} \\\n --peer-urls=https://{{ NODE_TO_ADD }}:2380\"\n delegate_to: \"{{ RUNNING_NODE.stdout }}\"\n\n# step3: start the new-etcd node\n- hosts: \"{{ NODE_TO_ADD }}\"\n vars:\n CLUSTER_STATE: existing\n roles:\n - { role: os-harden, when: \"OS_HARDEN|bool\" }\n - { role: chrony, when: \"groups['chrony']|length > 0\" }\n - prepare\n - etcd\n"
},
{
"path": "playbooks/22.addnode.yml",
"content": "# Note: this playbook can not run independently\n\n- hosts: \"{{ NODE_TO_ADD }}\" \n roles:\n - { role: os-harden, when: \"OS_HARDEN|bool\" }\n - { role: chrony, when: \"groups['chrony']|length > 0\" } \n - prepare\n - { role: docker, when: \"CONTAINER_RUNTIME == 'docker'\" }\n - { role: containerd, when: \"CONTAINER_RUNTIME == 'containerd'\" }\n - kube-lb\n - kube-node\n - { role: calico, when: \"CLUSTER_NETWORK == 'calico'\" }\n - { role: cilium, when: \"CLUSTER_NETWORK == 'cilium'\" }\n - { role: flannel, when: \"CLUSTER_NETWORK == 'flannel'\" }\n - { role: kube-router, when: \"CLUSTER_NETWORK == 'kube-router'\" }\n"
},
{
"path": "playbooks/23.addmaster.yml",
"content": "# Note: this playbook cann't run independently\n\n- hosts: \"{{ NODE_TO_ADD }}\" \n roles:\n - { role: os-harden, when: \"OS_HARDEN|bool\" }\n - { role: chrony, when: \"groups['chrony']|length > 0\" } \n - prepare\n - { role: docker, when: \"CONTAINER_RUNTIME == 'docker'\" }\n - { role: containerd, when: \"CONTAINER_RUNTIME == 'containerd'\" }\n - kube-lb\n - kube-master\n - kube-node\n - { role: calico, when: \"CLUSTER_NETWORK == 'calico'\" }\n - { role: cilium, when: \"CLUSTER_NETWORK == 'cilium'\" }\n - { role: flannel, when: \"CLUSTER_NETWORK == 'flannel'\" }\n - { role: kube-router, when: \"CLUSTER_NETWORK == 'kube-router'\" }\n - { role: kube-ovn, when: \"CLUSTER_NETWORK == 'kube-ovn'\" }\n"
},
{
"path": "playbooks/31.deletcd.yml",
"content": "# WARNNING: this playbook will clean the etcd {{ ETCD_TO_DEL }} \n\n- hosts: localhost \n vars_prompt:\n - name: \"ETCD_TO_DEL\"\n prompt: \"which etcd node is about to be deleted?(e.g 192.168.1.1)\"\n private: no\n confirm: yes\n tasks:\n # step0: run prechecks\n - fail: msg=\"{{ ETCD_TO_DEL }} is NOT a member of etcd cluster!\"\n when: \"ETCD_TO_DEL not in groups['etcd']\" \n\n - fail: msg=\"you CAN NOT delete the last member of etcd cluster!\"\n when: \"groups['etcd']|length < 2\" \n\n - block:\n # step1: find a healthy member in the etcd cluster\n - name: set NODE_IPS of the etcd cluster\n set_fact: NODE_IPS=\"{% for host in groups['etcd'] %}{{ host }} {% endfor %}\"\n\n - name: get etcd cluster status\n shell: 'for ip in {{ NODE_IPS }};do \\\n ETCDCTL_API=3 {{ base_dir }}/bin/etcdctl \\\n --endpoints=https://\"$ip\":2379 \\\n --cacert={{ cluster_dir }}/ssl/ca.pem \\\n --cert={{ cluster_dir }}/ssl/etcd.pem \\\n --key={{ cluster_dir }}/ssl/etcd-key.pem \\\n endpoint health; \\\n done'\n register: ETCD_CLUSTER_STATUS\n ignore_errors: true\n\n - debug: var=\"ETCD_CLUSTER_STATUS\"\n\n - name: get a running ectd node\n shell: 'echo -e \"{{ ETCD_CLUSTER_STATUS.stdout }}\" \\\n \"{{ ETCD_CLUSTER_STATUS.stderr }}\" \\\n |grep \"is healthy\"|sed -n \"1p\"|cut -d: -f2|cut -d/ -f3'\n register: RUNNING_NODE\n\n - debug: var=\"RUNNING_NODE.stdout\"\n\n # step2: remove jobs run on the healthy member\n - name: get ID of etcd node to delete\n shell: \"ETCDCTL_API=3 {{ bin_dir }}/etcdctl member list \\\n |grep {{ ETCD_TO_DEL }}:2380|cut -d',' -f1\"\n register: ETCD_ID\n delegate_to: \"{{ RUNNING_NODE.stdout }}\"\n \n - name: get NAME of etcd node to delete\n shell: \"ETCDCTL_API=3 {{ bin_dir }}/etcdctl member list \\\n |grep {{ ETCD_TO_DEL }}:2380|cut -d' ' -f3|cut -d',' -f1\"\n register: ETCD_NAME\n delegate_to: \"{{ RUNNING_NODE.stdout }}\"\n \n - debug: var=\"ETCD_NAME.stdout\"\n\n - name: delete a etcd member\n shell: \"ETCDCTL_API=3 {{ bin_dir }}/etcdctl member remove {{ ETCD_ID.stdout }}\"\n delegate_to: \"{{ RUNNING_NODE.stdout }}\"\n when: \"ETCD_ID.stdout != ''\"\n \n - name: clean etcd {{ ETCD_TO_DEL }}\n shell: \"cd {{ base_dir }} && ansible-playbook -i clusters/{{ CLUSTER }}/hosts \\\n roles/clean/clean_node.yml \\\n -e NODE_TO_CLEAN={{ ETCD_TO_DEL }} \\\n -e DEL_ETCD=yes >> /tmp/ansible-`date +'%Y%m%d%H%M%S'`.log 2>&1 \\\n || echo 'data not cleaned on {{ ETCD_TO_DEL }}'\"\n register: CLEAN_STATUS\n\n - debug: var=\"CLEAN_STATUS\"\n\n # lineinfile is inadequate to delete lines between some specific line range\n - name: remove the etcd's node entry in hosts\n shell: 'sed -i \"/^\\[etcd/,/^\\[kube_master/ {/^{{ ETCD_TO_DEL }}$/d}\" {{ base_dir }}/clusters/{{ CLUSTER }}/hosts'\n\n # lineinfile is inadequate to delete lines between some specific line range\n - name: remove the etcd's node entry in hosts\n shell: 'sed -i \"/^\\[etcd/,/^\\[kube_master/ {/^{{ ETCD_TO_DEL }} /d}\" {{ base_dir }}/clusters/{{ CLUSTER }}/hosts'\n when: \"groups['etcd']|length > 1 and ETCD_TO_DEL in groups['etcd']\" \n"
},
{
"path": "playbooks/32.delnode.yml",
"content": "# WARNNING: this playbook will clean the node {{ NODE_TO_DEL }}\n\n- hosts: localhost \n tasks:\n - fail: msg=\"you CAN NOT delete the last member of kube_master!\"\n when: \"groups['kube_master']|length < 2 and NODE_TO_DEL in groups['kube_master']\"\n\n - name: 注册变量 K8S 主版本\n shell: echo {{ K8S_VER }}|awk -F. '{print $1\".\"$2}'\n register: K8S_VER_MAIN\n\n - name: 设置kubectl drain 参数\n set_fact: DRAIN_OPT=\"--delete-emptydir-data --ignore-daemonsets --force\"\n when: \"K8S_VER_MAIN.stdout|float > 1.19\"\n\n - name: 设置kubectl drain 参数\n set_fact: DRAIN_OPT=\"--delete-local-data --ignore-daemonsets --force\"\n when: \"K8S_VER_MAIN.stdout|float < 1.20\"\n\n - name: debug info\n debug: var=\"DRAIN_OPT\"\n\n - name: get the node name to delete\n shell: \"{{ base_dir }}/bin/kubectl get node -owide|grep ' {{ NODE_TO_DEL }} '|awk '{print $1}'\"\n register: NODE_NAME\n\n - debug: var=\"NODE_NAME.stdout\"\n\n - name: run kubectl drain @{{ NODE_NAME.stdout }}\n shell: \"{{ base_dir }}/bin/kubectl drain {{ NODE_NAME.stdout }} {{ DRAIN_OPT }}\"\n #ignore_errors: true\n\n - name: clean node {{ NODE_TO_DEL }}\n shell: \"cd {{ base_dir }} && ansible-playbook -i clusters/{{ CLUSTER }}/hosts \\\n roles/clean/clean_node.yml \\\n -e NODE_TO_CLEAN={{ NODE_TO_DEL }} \\\n -e DEL_NODE=yes \\\n -e DEL_LB=yes >> /tmp/ansible-`date +'%Y%m%d%H%M%S'`.log 2>&1 \\\n || echo 'data not cleaned on {{ NODE_TO_DEL }}'\"\n register: CLEAN_STATUS\n\n - debug: var=\"CLEAN_STATUS\"\n\n - name: run kubectl delete node {{ NODE_NAME.stdout }}\n shell: \"{{ base_dir }}/bin/kubectl delete node {{ NODE_NAME.stdout }}\"\n ignore_errors: true\n\n # lineinfile is inadequate to delete lines between some specific line range\n - name: remove the node's entry in hosts\n shell: 'sed -i \"/^\\[kube_node/,/^\\[harbor/ {/^{{ NODE_TO_DEL }}$/d}\" {{ base_dir }}/clusters/{{ CLUSTER }}/hosts'\n\n # lineinfile is inadequate to delete lines between some specific line range\n - name: remove the node's entry in hosts\n shell: 'sed -i \"/^\\[kube_node/,/^\\[harbor/ {/^{{ NODE_TO_DEL }} /d}\" {{ base_dir }}/clusters/{{ CLUSTER }}/hosts'\n"
},
{
"path": "playbooks/33.delmaster.yml",
"content": "# WARNNING: this playbook will clean the kube_master node {{ NODE_TO_DEL }}\n\n- hosts: localhost\n tasks:\n - fail: msg=\"you CAN NOT delete the last member of kube_master!\"\n when: \"groups['kube_master']|length < 2 and NODE_TO_DEL in groups['kube_master']\"\n\n - name: 注册变量 K8S 主版本\n shell: echo {{ K8S_VER }}|awk -F. '{print $1\".\"$2}'\n register: K8S_VER_MAIN\n\n - name: 设置kubectl drain 参数\n set_fact: DRAIN_OPT=\"--delete-emptydir-data --ignore-daemonsets --force\"\n when: \"K8S_VER_MAIN.stdout|float > 1.19\"\n\n - name: 设置kubectl drain 参数\n set_fact: DRAIN_OPT=\"--delete-local-data --ignore-daemonsets --force\"\n when: \"K8S_VER_MAIN.stdout|float < 1.20\"\n\n - name: debug info\n debug: var=\"DRAIN_OPT\"\n\n - name: get the node name to delete\n shell: \"{{ base_dir }}/bin/kubectl get node -owide|grep ' {{ NODE_TO_DEL }} '|awk '{print $1}'\"\n register: NODE_NAME\n\n - debug: var=\"NODE_NAME.stdout\"\n\n - name: run kubectl drain @{{ NODE_NAME.stdout }}\n shell: \"{{ base_dir }}/bin/kubectl drain {{ NODE_NAME.stdout }} {{ DRAIN_OPT }}\"\n #ignore_errors: true\n\n - name: clean node {{ NODE_TO_DEL }}\n shell: \"cd {{ base_dir }} && ansible-playbook -i clusters/{{ CLUSTER }}/hosts \\\n roles/clean/clean_node.yml \\\n -e NODE_TO_CLEAN={{ NODE_TO_DEL }} \\\n -e DEL_MASTER=yes \\\n -e DEL_NODE=yes \\\n -e DEL_LB=yes >> /tmp/ansible-`date +'%Y%m%d%H%M%S'`.log 2>&1 \\\n || echo 'data not cleaned on {{ NODE_TO_DEL }}'\"\n register: CLEAN_STATUS\n\n - debug: var=\"CLEAN_STATUS\"\n\n # lineinfile is inadequate to delete lines between some specific line range\n - name: remove the master's entry in hosts\n shell: 'sed -i \"/^\\[kube_master/,/^\\[harbor/ {/^{{ NODE_TO_DEL }}$/d}\" {{ base_dir }}/clusters/{{ CLUSTER }}/hosts'\n\n # lineinfile is inadequate to delete lines between some specific line range\n - name: remove the master's entry in hosts\n shell: 'sed -i \"/^\\[kube_master/,/^\\[harbor/ {/^{{ NODE_TO_DEL }} /d}\" {{ base_dir }}/clusters/{{ CLUSTER }}/hosts'\n"
},
{
"path": "playbooks/90.setup.yml",
"content": "# [optional] to synchronize time of nodes with 'chrony'\n- hosts:\n - kube_master\n - kube_node\n - etcd\n - ex_lb\n - chrony\n roles:\n - { role: os-harden, when: \"OS_HARDEN|bool\" }\n - { role: chrony, when: \"groups['chrony']|length > 0\" }\n\n# to create CA, kubeconfig, kube-proxy.kubeconfig etc.\n- hosts: localhost\n roles:\n - deploy\n\n# prepare tasks for all nodes\n- hosts:\n - kube_master\n - kube_node\n - etcd\n roles:\n - prepare\n\n# to install etcd cluster\n- hosts: etcd\n roles:\n - etcd\n\n# to install container runtime\n- hosts:\n - kube_master\n - kube_node\n roles:\n - { role: docker, when: \"CONTAINER_RUNTIME == 'docker'\" }\n - { role: containerd, when: \"CONTAINER_RUNTIME == 'containerd'\" }\n\n# to set up 'kube_master' nodes\n- hosts: kube_master\n roles:\n - kube-lb\n - kube-master\n - kube-node\n\n# to set up 'kube_node' nodes\n- hosts: kube_node\n roles:\n - { role: kube-lb, when: \"inventory_hostname not in groups['kube_master']\" }\n - { role: kube-node, when: \"inventory_hostname not in groups['kube_master']\" }\n\n# to install network plugin, only one can be choosen\n- hosts:\n - kube_master\n - kube_node\n roles:\n - { role: calico, when: \"CLUSTER_NETWORK == 'calico'\" }\n - { role: cilium, when: \"CLUSTER_NETWORK == 'cilium'\" }\n - { role: flannel, when: \"CLUSTER_NETWORK == 'flannel'\" }\n - { role: kube-router, when: \"CLUSTER_NETWORK == 'kube-router'\" }\n - { role: kube-ovn, when: \"CLUSTER_NETWORK == 'kube-ovn'\" }\n\n# to install cluster-addons\n- hosts: localhost\n roles:\n - cluster-addon\n"
},
{
"path": "playbooks/91.start.yml",
"content": "- hosts: etcd\n tasks:\n - name: starting etcd cluster\n service: name=etcd state=started enabled=yes\n\n- hosts:\n - kube_master\n - kube_node\n tasks:\n - name: starting kube-lb\n service: name=kube-lb state=started enabled=yes\n\n- hosts: kube_master\n tasks:\n - name: starting kube_master services\n service: name={{ item }} state=started enabled=yes\n with_items:\n - kube-apiserver\n - kube-controller-manager\n - kube-scheduler\n\n- hosts:\n - kube_master\n - kube_node\n tasks:\n - name: starting docker\n service: name=docker state=started enabled=yes\n when: \"CONTAINER_RUNTIME == 'docker'\"\n\n - name: starting containerd\n service: name=containerd state=started enabled=yes\n when: \"CONTAINER_RUNTIME == 'containerd'\"\n\n - name: starting kube_node services\n service: name={{ item }} state=started enabled=yes\n with_items:\n - kubelet\n - kube-proxy\n\n- hosts: ex_lb\n tasks:\n - name: starting external loadbalance\n service: name={{ item }} state=started enabled=yes\n with_items:\n - l4lb\n - keepalived\n"
},
{
"path": "playbooks/92.stop.yml",
"content": "- hosts: kube_master\n tasks:\n - name: stopping kube_master services\n service: name={{ item }} state=stopped enabled=no\n with_items:\n - kube-apiserver\n - kube-controller-manager\n - kube-scheduler\n\n- hosts: etcd\n tasks:\n - name: stopping etcd cluster\n service: name=etcd state=stopped enabled=no\n\n- hosts: ex_lb\n tasks:\n - name: stopping external loadbalance\n service: name={{ item }} state=stopped enabled=no\n with_items:\n - l4lb\n - keepalived\n\n- hosts:\n - kube_master\n - kube_node\n tasks:\n - name: stopping kube_node services\n service: name={{ item }} state=stopped enabled=no\n with_items:\n - kube-lb\n - kubelet\n - kube-proxy\n\n - name: stopping docker\n service: name=docker state=stopped enabled=no\n when: \"CONTAINER_RUNTIME == 'docker'\"\n\n - name: stopping containerd\n service: name=containerd state=stopped enabled=no\n when: \"CONTAINER_RUNTIME == 'containerd'\"\n"
},
{
"path": "playbooks/93.upgrade.yml",
"content": "# WARNING: Upgrade the k8s cluster can be risky. Make sure you know what you are doing.\n# Read the guide: 'op/upgrade.md' .\n# Usage: ezctl upgrade \n\n# check k8s version\n- hosts: kube_master\n tasks:\n - name: get running k8s version\n shell: \"{{ bin_dir }}/kube-apiserver --version\"\n register: RUNNING_VER\n run_once: true\n\n - name: print running version\n debug: var=\"RUNNING_VER.stdout\"\n run_once: true\n\n - name: get update version\n shell: \"{{ base_dir }}/bin/kube-apiserver --version\"\n register: UPDATE_VER\n run_once: true\n connection: local\n\n - name: print update version\n debug: var=\"UPDATE_VER.stdout\"\n run_once: true\n\n - name: check version\n fail: msg=\"running version is the same as the update version, UPDATE ABORT.\"\n when: \"RUNNING_VER.stdout == UPDATE_VER.stdout\"\n\n# update masters\n- hosts: \n - kube_master\n roles:\n - kube-master\n - kube-node\n\n# update nodes\n- hosts: \n - kube_node\n roles:\n - { role: kube-node, when: \"inventory_hostname not in groups['kube_master']\" }\n"
},
{
"path": "playbooks/94.backup.yml",
"content": "# cluster-backup playbook\n# read the guide: 'op/cluster_restore.md'\n\n- hosts:\n - localhost\n tasks:\n # step1: find a healthy member in the etcd cluster\n - name: set NODE_IPS of the etcd cluster\n set_fact: NODE_IPS=\"{% for host in groups['etcd'] %}{{ host }} {% endfor %}\"\n\n - name: get etcd cluster status\n shell: 'for ip in {{ NODE_IPS }};do \\\n ETCDCTL_API=3 {{ base_dir }}/bin/etcdctl \\\n --endpoints=https://\"$ip\":2379 \\\n --cacert={{ cluster_dir }}/ssl/ca.pem \\\n --cert={{ cluster_dir }}/ssl/etcd.pem \\\n --key={{ cluster_dir }}/ssl/etcd-key.pem \\\n endpoint health; \\\n done'\n register: ETCD_CLUSTER_STATUS\n ignore_errors: true\n\n - debug: var=\"ETCD_CLUSTER_STATUS\"\n\n - name: get a running ectd node\n shell: 'echo -e \"{{ ETCD_CLUSTER_STATUS.stdout }}\" \\\n \"{{ ETCD_CLUSTER_STATUS.stderr }}\" \\\n |grep \"is healthy\"|sed -n \"1p\"|cut -d: -f2|cut -d/ -f3'\n register: RUNNING_NODE\n\n - debug: var=\"RUNNING_NODE.stdout\"\n\n - name: get current time\n shell: \"date +'%Y%m%d%H%M'\"\n register: timestamp\n\n # step2: backup data to the ansible node \n - name: make a backup on the etcd node\n shell: \"mkdir -p {{ cluster_dir }}/backup && cd {{ cluster_dir }}/backup && \\\n ETCDCTL_API=3 {{ base_dir }}/bin/etcdctl \\\n --endpoints=https://{{ RUNNING_NODE.stdout }}:2379 \\\n --cacert={{ cluster_dir }}/ssl/ca.pem \\\n --cert={{ cluster_dir }}/ssl/etcd.pem \\\n --key={{ cluster_dir }}/ssl/etcd-key.pem \\\n snapshot save snapshot_{{ timestamp.stdout }}.db\"\n\n - name: update the latest backup\n shell: 'cd {{ cluster_dir }}/backup/ && /bin/cp -f snapshot_{{ timestamp.stdout }}.db snapshot.db'\n"
},
{
"path": "playbooks/95.restore.yml",
"content": "# cluster-restore playbook\n# read the guide: 'op/cluster_restore.md'\n# https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#restoring-an-etcd-cluster\n\n- hosts: kube_master\n tasks:\n - name: stopping kube_master services\n service: name={{ item }} state=stopped\n with_items:\n - kube-apiserver\n - kube-controller-manager\n - kube-scheduler\n\n- hosts:\n - kube_master\n - kube_node\n tasks:\n - name: stopping kube_node services\n service: name={{ item }} state=stopped\n with_items:\n - kubelet\n - kube-proxy\n\n- hosts: etcd\n roles:\n - cluster-restore\n\n- hosts: kube_master\n tasks:\n - name: starting kube_master services\n service: name={{ item }} state=started enabled=yes\n with_items:\n - kube-apiserver\n - kube-controller-manager\n - kube-scheduler\n\n- hosts:\n - kube_master\n - kube_node\n tasks:\n - name: starting kube_node services\n service: name={{ item }} state=started enabled=yes\n with_items:\n - kubelet\n - kube-proxy\n"
},
{
"path": "playbooks/96.update-certs.yml",
"content": "# Note: this scripts should be used with caution.\n# Force to recreate CA certs and all of the others certs used in the cluster. \n# It should be used when the admin.conf leaked, and a new one will be created in place of the leaked one.\n\n# backup old certs\n- hosts: localhost\n tasks:\n - name: backup old certs\n shell: \"cd {{ cluster_dir }} && \\\n cp -r ssl ssl-$(date +'%Y%m%d%H%M')\"\n tags: force_change_certs\n\n# to create CA, kubeconfig, kube-proxy.kubeconfig etc.\n# need to set 'CHANGE_CA=true'\n- hosts: localhost\n roles:\n - deploy\n\n# to install etcd cluster\n# to run with '-t force_change_certs'\n- hosts: etcd\n roles:\n - etcd\n\n# to set up 'kube_master' nodes\n# to run with '-t force_change_certs'\n- hosts: kube_master\n roles:\n - kube-master\n\n# to set up 'kube_node' nodes\n# to run with '-t force_change_certs'\n- hosts:\n - kube_master\n - kube_node\n roles:\n - kube-node \n\n# to install network plugin, only one can be choosen\n# to run with '-t force_change_certs'\n- hosts:\n - kube_master\n - kube_node\n roles:\n - { role: calico, when: \"CLUSTER_NETWORK == 'calico'\" }\n - { role: cilium, when: \"CLUSTER_NETWORK == 'cilium'\" }\n - { role: flannel, when: \"CLUSTER_NETWORK == 'flannel'\" }\n - { role: kube-router, when: \"CLUSTER_NETWORK == 'kube-router'\" }\n - { role: kube-ovn, when: \"CLUSTER_NETWORK == 'kube-ovn'\" }\n\n# to install cluster-addons\n- hosts: localhost\n roles:\n - cluster-addon\n"
},
{
"path": "playbooks/99.clean.yml",
"content": "# WARNING: This playbook will erase the entire k8s-cluster, include PODs, ETCD data etc.\n# Make sure you know what you are doing.\n\n- hosts:\n - kube_master\n - kube_node\n - ex_lb\n - etcd\n vars:\n DEL_MASTER: \"yes\"\n DEL_NODE: \"yes\"\n DEL_ETCD: \"yes\"\n DEL_LB: \"yes\"\n DEL_CHRONY: \"yes\"\n DEL_ENV: \"yes\"\n roles:\n - clean\n"
},
{
"path": "roles/calico/tasks/calico-rr.yml",
"content": "- block:\n - name: 选择rr节点(master节点)\n set_fact: NODE_IPS=\"{% for host in groups['kube_master'] %}{{ host }} {% endfor %}\"\n when: \"CALICO_RR_NODES|length == 0\"\n\n - name: 选择rr节点\n set_fact: NODE_IPS=\"{% for host in CALICO_RR_NODES %}{{ host }} {% endfor %}\"\n when: \"CALICO_RR_NODES|length > 0\"\n\n - name: 显示rr节点\n debug: var=\"NODE_IPS\"\n\n - name: 配置routeReflectorClusterID\n shell: 'for ip in {{ NODE_IPS }};do \\\n node_name=$({{ bin_dir }}/calicoctl get node -owide|grep \" $ip/\"|cut -d\" \" -f1) && \\\n {{ bin_dir }}/calicoctl patch node \"$node_name\" \\\n -p \"{\\\"spec\\\": {\\\"bgp\\\": {\\\"routeReflectorClusterID\\\": \\\"244.0.0.1\\\"}}}\"; \\\n done'\n\n - name: node label\n shell: 'for ip in {{ NODE_IPS }};do \\\n node_name=$({{ bin_dir }}/calicoctl get node -owide|grep \" $ip/\"|cut -d\" \" -f1) && \\\n {{ base_dir }}/bin/kubectl label node \"$node_name\" route-reflector=true --overwrite;\n done'\n connection: local\n\n - name: 配置 calico bgp yaml文件\n template: src={{ item }}.j2 dest=/etc/calico/{{ item }}\n with_items:\n - \"bgp-default.yaml\"\n - \"bgp-rr.yaml\"\n\n - name: 应用 calico bgp 配置\n shell: \"{{ bin_dir }}/calicoctl apply -f /etc/calico/bgp-rr.yaml && \\\n sleep 5 && \\\n {{ bin_dir }}/calicoctl apply -f /etc/calico/bgp-default.yaml && sleep 2\"\n run_once: true\n\n- name: 查看bgp连接\n shell: \"{{ bin_dir }}/calicoctl node status\"\n register: bgp_status\n\n- debug: var=\"bgp_status.stdout_lines\"\n"
},
{
"path": "roles/calico/tasks/main.yml",
"content": "- block:\n - name: 创建calico 证书请求\n template: src=calico-csr.json.j2 dest={{ cluster_dir }}/ssl/calico-csr.json\n\n - name: 创建 calico证书和私钥\n shell: \"cd {{ cluster_dir }}/ssl && {{ base_dir }}/bin/cfssl gencert \\\n -ca=ca.pem \\\n -ca-key=ca-key.pem \\\n -config=ca-config.json \\\n -profile=kubernetes calico-csr.json|{{ base_dir }}/bin/cfssljson -bare calico\"\n\n - name: 删除旧 calico-etcd-secrets\n shell: \"{{ base_dir }}/bin/kubectl -n kube-system delete secrets calico-etcd-secrets || echo NotFound\"\n\n - name: 创建 calico-etcd-secrets\n shell: \"cd {{ cluster_dir }}/ssl && \\\n {{ base_dir }}/bin/kubectl create secret generic -n kube-system calico-etcd-secrets \\\n --from-file=etcd-ca=ca.pem \\\n --from-file=etcd-key=calico-key.pem \\\n --from-file=etcd-cert=calico.pem\"\n\n - name: 配置 calico DaemonSet yaml文件\n template: src=calico-{{ calico_ver_main }}.yaml.j2 dest={{ cluster_dir }}/yml/calico.yaml\n\n - name: 删除 calico网络\n shell: \"{{ base_dir }}/bin/kubectl delete -f {{ cluster_dir }}/yml/calico.yaml || echo NotFound; sleep 3\"\n when: 'CHANGE_CA|bool'\n\n - name: 运行 calico网络\n shell: \"{{ base_dir }}/bin/kubectl apply -f {{ cluster_dir }}/yml/calico.yaml\"\n run_once: true\n connection: local\n tags: force_change_certs\n\n- name: 在节点创建相关目录\n file: name={{ item }} state=directory\n with_items:\n - /etc/calico/ssl\n\n- name: 分发calico证书相关\n copy: src={{ cluster_dir }}/ssl/{{ item }} dest=/etc/calico/ssl/{{ item }}\n with_items:\n - ca.pem\n - calico.pem\n - calico-key.pem\n tags: force_change_certs\n\n- name: 删除默认cni配置\n file: path=/etc/cni/net.d/10-default.conf state=absent\n\n- name: 下载calicoctl 客户端\n copy: src={{ base_dir }}/bin/{{ item }} dest={{ bin_dir }}/{{ item }} mode=0755\n with_items:\n #- calico\n - calicoctl\n ignore_errors: true\n\n- name: 准备 calicoctl配置文件\n template: src=calicoctl.cfg.j2 dest=/etc/calico/calicoctl.cfg\n\n- name: 轮询等待calico-node 运行\n shell: \"{{ base_dir }}/bin/kubectl get pod -n kube-system -o wide|grep 'calico-node'|grep ' {{ K8S_NODENAME }} '|awk '{print $3}'\"\n register: pod_status\n until: pod_status.stdout == \"Running\"\n retries: 15\n delay: 15\n ignore_errors: true\n connection: local\n tags: force_change_certs\n\n- import_tasks: calico-rr.yml\n when: 'CALICO_RR_ENABLED|bool'\n tags: force_change_certs\n"
},
{
"path": "roles/calico/templates/bgp-default.yaml.j2",
"content": "apiVersion: projectcalico.org/v3\nkind: BGPConfiguration\nmetadata:\n name: default\nspec:\n logSeverityScreen: Info\n nodeToNodeMeshEnabled: false\n asNumber: {{ CALICO_AS_NUMBER }}\n"
},
{
"path": "roles/calico/templates/bgp-rr.yaml.j2",
"content": "kind: BGPPeer\napiVersion: projectcalico.org/v3\nmetadata:\n name: peer-with-route-reflectors\nspec:\n nodeSelector: all()\n peerSelector: route-reflector == 'true'\n"
},
{
"path": "roles/calico/templates/calico-csr.json.j2",
"content": "{\n \"CN\": \"calico\",\n \"hosts\": [],\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"k8s\",\n \"OU\": \"System\"\n }\n ]\n}\n"
},
{
"path": "roles/calico/templates/calico-v3.24.yaml.j2",
"content": "---\n# download Release archive (https://github.com/projectcalico/calico/releases/download/v3.24.5/release-v3.24.5.tgz)\n# Release notes: https://projectcalico.docs.tigera.io/archive/v3.24/release-notes/\n\n# Datastore: etcd, using Typha is redundant and not recommended.\n# Kubeasz uses cmd-line-way( kubectl create) to create etcd-secrets, see more in 'roles/calico/tasks/main.yml'\n\n# source from: release-v3.24.5/manifests/calico-etcd.yaml\n\n# Source: calico/templates/calico-kube-controllers.yaml\n# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict\n\napiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\nspec:\n maxUnavailable: 1\n selector:\n matchLabels:\n k8s-app: calico-kube-controllers\n---\n# Source: calico/templates/calico-kube-controllers.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-node\n namespace: kube-system\n---\n# Source: calico/templates/calico-config.yaml\n# This ConfigMap is used to configure a self-hosted Calico installation.\nkind: ConfigMap\napiVersion: v1\nmetadata:\n name: calico-config\n namespace: kube-system\ndata:\n # Configure this with the location of your etcd cluster.\n etcd_endpoints: \"{{ ETCD_ENDPOINTS }}\"\n # If you're using TLS enabled etcd uncomment the following.\n # You must also populate the Secret below with these files.\n etcd_ca: \"/calico-secrets/etcd-ca\"\n etcd_cert: \"/calico-secrets/etcd-cert\"\n etcd_key: \"/calico-secrets/etcd-key\"\n # Typha is disabled.\n typha_service_name: \"none\"\n # Configure the backend to use.\n calico_backend: \"{{ CALICO_NETWORKING_BACKEND }}\"\n\n # Configure the MTU to use for workload interfaces and tunnels.\n # By default, MTU is auto-detected, and explicitly setting this field should not be required.\n # You can override auto-detection by providing a non-zero value.\n veth_mtu: \"0\"\n\n # The CNI network configuration to install on each node. The special\n # values in this config will be automatically populated.\n cni_network_config: |-\n {\n \"name\": \"k8s-pod-network\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"calico\",\n \"log_level\": \"info\",\n \"log_file_path\": \"/var/log/calico/cni/cni.log\",\n \"etcd_endpoints\": \"{{ ETCD_ENDPOINTS }}\",\n \"etcd_key_file\": \"/etc/calico/ssl/calico-key.pem\",\n \"etcd_cert_file\": \"/etc/calico/ssl/calico.pem\",\n \"etcd_ca_cert_file\": \"{{ ca_dir }}/ca.pem\",\n \"mtu\": __CNI_MTU__,\n \"ipam\": {\n \"type\": \"calico-ipam\"\n },\n \"policy\": {\n \"type\": \"k8s\"\n },\n \"kubernetes\": {\n \"kubeconfig\": \"/etc/cni/net.d/calico-kubeconfig\"\n }\n },\n {\n \"type\": \"portmap\",\n \"snat\": true,\n \"capabilities\": {\"portMappings\": true}\n },\n {\n \"type\": \"bandwidth\",\n \"capabilities\": {\"bandwidth\": true}\n }\n ]\n }\n---\n# Source: calico/templates/calico-kube-controllers-rbac.yaml\n# Include a clusterrole for the kube-controllers component,\n# and bind it to the calico-kube-controllers serviceaccount.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-kube-controllers\nrules:\n # Pods are monitored for changing labels.\n # The node controller monitors Kubernetes nodes.\n # Namespace and serviceaccount labels are used for policy.\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n - serviceaccounts\n verbs:\n - watch\n - list\n - get\n # Watch for changes to Kubernetes NetworkPolicies.\n - apiGroups: [\"networking.k8s.io\"]\n resources:\n - networkpolicies\n verbs:\n - watch\n - list\n---\n# Source: calico/templates/calico-node-rbac.yaml\n# Include a clusterrole for the calico-node DaemonSet,\n# and bind it to the calico-node serviceaccount.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-node\nrules:\n # Used for creating service account tokens to be used by the CNI plugin\n - apiGroups: [\"\"]\n resources:\n - serviceaccounts/token\n resourceNames:\n - calico-node\n verbs:\n - create\n # The CNI plugin needs to get pods, nodes, and namespaces.\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n verbs:\n - get\n # EndpointSlices are used for Service-based network policy rule\n # enforcement.\n - apiGroups: [\"discovery.k8s.io\"]\n resources:\n - endpointslices\n verbs:\n - watch\n - list\n - apiGroups: [\"\"]\n resources:\n - endpoints\n - services\n verbs:\n # Used to discover service IPs for advertisement.\n - watch\n - list\n # Pod CIDR auto-detection on kubeadm needs access to config maps.\n - apiGroups: [\"\"]\n resources:\n - configmaps\n verbs:\n - get\n - apiGroups: [\"\"]\n resources:\n - nodes/status\n verbs:\n # Needed for clearing NodeNetworkUnavailable flag.\n - patch\n---\n# Source: calico/templates/calico-kube-controllers-rbac.yaml\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-kube-controllers\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-kube-controllers\nsubjects:\n- kind: ServiceAccount\n name: calico-kube-controllers\n namespace: kube-system\n---\n# Source: calico/templates/calico-node-rbac.yaml\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: calico-node\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-node\nsubjects:\n- kind: ServiceAccount\n name: calico-node\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\n# This manifest installs the calico-node container, as well\n# as the CNI plugins and network config on\n# each master and worker node in a Kubernetes cluster.\nkind: DaemonSet\napiVersion: apps/v1\nmetadata:\n name: calico-node\n namespace: kube-system\n labels:\n k8s-app: calico-node\nspec:\n selector:\n matchLabels:\n k8s-app: calico-node\n updateStrategy:\n type: RollingUpdate\n rollingUpdate:\n maxUnavailable: 1\n template:\n metadata:\n labels:\n k8s-app: calico-node\n spec:\n nodeSelector:\n kubernetes.io/os: linux\n hostNetwork: true\n tolerations:\n # Make sure calico-node gets scheduled on all nodes.\n - effect: NoSchedule\n operator: Exists\n # Mark the pod as a critical add-on for rescheduling.\n - key: CriticalAddonsOnly\n operator: Exists\n - effect: NoExecute\n operator: Exists\n serviceAccountName: calico-node\n # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a \"force\n # deletion\": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.\n terminationGracePeriodSeconds: 0\n priorityClassName: system-node-critical\n initContainers:\n # This container installs the CNI binaries\n # and CNI network config file on each node.\n - name: install-cni\n image: easzlab.io.local:5000/easzlab/cni:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n command: [\"/opt/cni/bin/install\"]\n envFrom:\n - configMapRef:\n # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.\n name: kubernetes-services-endpoint\n optional: true\n env:\n # Name of the CNI config file to create.\n - name: CNI_CONF_NAME\n value: \"10-calico.conflist\"\n # The CNI network config to install on each node.\n - name: CNI_NETWORK_CONFIG\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: cni_network_config\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # CNI MTU Config variable\n - name: CNI_MTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Prevents the container from sleeping forever.\n - name: SLEEP\n value: \"false\"\n volumeMounts:\n - mountPath: /host/opt/cni/bin\n name: cni-bin-dir\n - mountPath: /host/etc/cni/net.d\n name: cni-net-dir\n - mountPath: /calico-secrets\n name: etcd-certs\n securityContext:\n privileged: true\n # This init container mounts the necessary filesystems needed by the BPF data plane\n # i.e. bpf at /sys/fs/bpf and cgroup2 at /run/calico/cgroup. Calico-node initialisation is executed\n # in best effort fashion, i.e. no failure for errors, to not disrupt pod creation in iptable mode.\n - name: \"mount-bpffs\"\n image: easzlab.io.local:5000/easzlab/node:{{ calico_ver }} \n imagePullPolicy: IfNotPresent\n command: [\"calico-node\", \"-init\", \"-best-effort\"]\n volumeMounts:\n - mountPath: /sys/fs\n name: sys-fs\n # Bidirectional is required to ensure that the new mount we make at /sys/fs/bpf propagates to the host\n # so that it outlives the init container.\n mountPropagation: Bidirectional\n - mountPath: /var/run/calico\n name: var-run-calico\n # Bidirectional is required to ensure that the new mount we make at /run/calico/cgroup propagates to the host\n # so that it outlives the init container.\n mountPropagation: Bidirectional\n # Mount /proc/ from host which usually is an init program at /nodeproc. It's needed by mountns binary,\n # executed by calico-node, to mount root cgroup2 fs at /run/calico/cgroup to attach CTLB programs correctly.\n - mountPath: /nodeproc\n name: nodeproc\n readOnly: true\n securityContext:\n privileged: true\n containers:\n # Runs calico-node container on each Kubernetes node. This\n # container programs network policy and routes on each\n # host.\n - name: calico-node\n image: easzlab.io.local:5000/easzlab/node:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n envFrom:\n - configMapRef:\n # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.\n name: kubernetes-services-endpoint\n optional: true\n env:\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # Location of the CA certificate for etcd.\n - name: ETCD_CA_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_ca\n # Location of the client key for etcd.\n - name: ETCD_KEY_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_key\n # Location of the client certificate for etcd.\n - name: ETCD_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_cert\n # Set noderef for node controller.\n - name: CALICO_K8S_NODE_REF\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n # Choose the backend to use.\n - name: CALICO_NETWORKING_BACKEND\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: calico_backend\n # Cluster type to identify the deployment type\n - name: CLUSTER_TYPE\n value: \"k8s,bgp\"\n # Auto-detect the BGP IP address.\n - name: IP\n value: \"autodetect\"\n - name: IP_AUTODETECTION_METHOD\n value: \"{{ IP_AUTODETECTION_METHOD }}\"\n # Enable IPIP\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - name: CALICO_IPV4POOL_IPIP\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n{% endif %}\n # Enable or Disable VXLAN on the default IP pool.\n{% if CALICO_NETWORKING_BACKEND == \"vxlan\" %}\n - name: CALICO_IPV4POOL_VXLAN\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n - name: CALICO_IPV6POOL_VXLAN\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n{% endif %}\n # Set MTU for tunnel device used if ipip is enabled\n - name: FELIX_IPINIPMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Set MTU for the VXLAN tunnel device.\n - name: FELIX_VXLANMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Set MTU for the Wireguard tunnel device.\n - name: FELIX_WIREGUARDMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # The default IPv4 pool to create on startup if none exists. Pod IPs will be\n # chosen from this range. Changing this value after installation will have\n # no effect. This should fall within `--cluster-cidr`.\n - name: CALICO_IPV4POOL_CIDR\n value: \"{{ CLUSTER_CIDR }}\"\n # Disable file logging so `kubectl logs` works.\n - name: CALICO_DISABLE_FILE_LOGGING\n value: \"true\"\n # Set Felix endpoint to host default action to ACCEPT.\n - name: FELIX_DEFAULTENDPOINTTOHOSTACTION\n value: \"ACCEPT\"\n # Disable IPv6 on Kubernetes.\n - name: FELIX_IPV6SUPPORT\n value: \"false\"\n - name: FELIX_HEALTHENABLED\n value: \"true\"\n # Set Kubernetes NodePorts: If services do use NodePorts outside Calico’s expected range,\n # Calico will treat traffic to those ports as host traffic instead of pod traffic.\n - name: FELIX_KUBENODEPORTRANGES\n value: \"{{ NODE_PORT_RANGE.split('-')[0] }}:{{ NODE_PORT_RANGE.split('-')[1] }}\"\n - name: FELIX_PROMETHEUSMETRICSENABLED\n value: \"false\"\n securityContext:\n privileged: true\n resources:\n requests:\n cpu: 250m\n lifecycle:\n preStop:\n exec:\n command:\n - /bin/calico-node\n - -shutdown\n livenessProbe:\n exec:\n command:\n - /bin/calico-node\n - -felix-live\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - -bird-live\n{% endif %}\n periodSeconds: 10\n initialDelaySeconds: 10\n failureThreshold: 6\n timeoutSeconds: 10\n readinessProbe:\n exec:\n command:\n - /bin/calico-node\n - -felix-ready\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - -bird-ready\n{% endif %}\n periodSeconds: 10\n timeoutSeconds: 10\n volumeMounts:\n # For maintaining CNI plugin API credentials.\n - mountPath: /host/etc/cni/net.d\n name: cni-net-dir\n readOnly: false\n - mountPath: /lib/modules\n name: lib-modules\n readOnly: true\n - mountPath: /run/xtables.lock\n name: xtables-lock\n readOnly: false\n - mountPath: /var/run/calico\n name: var-run-calico\n readOnly: false\n - mountPath: /var/lib/calico\n name: var-lib-calico\n readOnly: false\n - mountPath: /calico-secrets\n name: etcd-certs\n - name: policysync\n mountPath: /var/run/nodeagent\n # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the\n # parent directory.\n - name: bpffs\n mountPath: /sys/fs/bpf\n - name: cni-log-dir\n mountPath: /var/log/calico/cni\n readOnly: true\n volumes:\n # Used by calico-node.\n - name: lib-modules\n hostPath:\n path: /lib/modules\n - name: var-run-calico\n hostPath:\n path: /var/run/calico\n - name: var-lib-calico\n hostPath:\n path: /var/lib/calico\n - name: xtables-lock\n hostPath:\n path: /run/xtables.lock\n type: FileOrCreate\n - name: sys-fs\n hostPath:\n path: /sys/fs/\n type: DirectoryOrCreate\n - name: bpffs\n hostPath:\n path: /sys/fs/bpf\n type: Directory\n # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.\n - name: nodeproc\n hostPath:\n path: /proc\n # Used to install CNI.\n - name: cni-bin-dir\n hostPath:\n path: /opt/cni/bin \n - name: cni-net-dir\n hostPath:\n path: /etc/cni/net.d\n # Used to access CNI logs.\n - name: cni-log-dir\n hostPath:\n path: /var/log/calico/cni\n # Mount in the etcd TLS secrets with mode 400.\n # See https://kubernetes.io/docs/concepts/configuration/secret/\n - name: etcd-certs\n secret:\n secretName: calico-etcd-secrets\n defaultMode: 0400\n # Used to create per-pod Unix Domain Sockets\n - name: policysync\n hostPath:\n type: DirectoryOrCreate\n path: /var/run/nodeagent\n---\n# Source: calico/templates/calico-kube-controllers.yaml\n# See https://github.com/projectcalico/kube-controllers\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\nspec:\n # The controllers can only have a single active instance.\n replicas: 1\n selector:\n matchLabels:\n k8s-app: calico-kube-controllers\n strategy:\n type: Recreate\n template:\n metadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\n spec:\n nodeSelector:\n kubernetes.io/os: linux\n tolerations:\n # Mark the pod as a critical add-on for rescheduling.\n - key: CriticalAddonsOnly\n operator: Exists\n - key: node-role.kubernetes.io/master\n effect: NoSchedule\n - key: node-role.kubernetes.io/control-plane\n effect: NoSchedule\n serviceAccountName: calico-kube-controllers\n priorityClassName: system-cluster-critical\n # The controllers must run in the host network namespace so that\n # it isn't governed by policy that would prevent it from working.\n hostNetwork: true\n containers:\n - name: calico-kube-controllers\n image: easzlab.io.local:5000/easzlab/kube-controllers:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n env:\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # Location of the CA certificate for etcd.\n - name: ETCD_CA_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_ca\n # Location of the client key for etcd.\n - name: ETCD_KEY_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_key\n # Location of the client certificate for etcd.\n - name: ETCD_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_cert\n # Choose which controllers to run.\n - name: ENABLED_CONTROLLERS\n value: policy,namespace,serviceaccount,workloadendpoint,node\n volumeMounts:\n # Mount in the etcd TLS secrets.\n - mountPath: /calico-secrets\n name: etcd-certs\n livenessProbe:\n exec:\n command:\n - /usr/bin/check-status\n - -l\n periodSeconds: 10\n initialDelaySeconds: 10\n failureThreshold: 6\n timeoutSeconds: 10\n readinessProbe:\n exec:\n command:\n - /usr/bin/check-status\n - -r\n periodSeconds: 10\n volumes:\n # Mount in the etcd TLS secrets with mode 400.\n # See https://kubernetes.io/docs/concepts/configuration/secret/\n - name: etcd-certs\n secret:\n secretName: calico-etcd-secrets\n defaultMode: 0440\n"
},
{
"path": "roles/calico/templates/calico-v3.26.yaml.j2",
"content": "---\n# download Release archive: https://github.com/projectcalico/calico/releases/download/v3.26.4/release-v3.26.4.tgz\n# Release notes: https://projectcalico.docs.tigera.io/archive/v3.26/release-notes/\n\n# Datastore: etcd, using Typha is redundant and not recommended.\n# Kubeasz uses command ( kubectl create) to create etcd-secrets, see more in 'roles/calico/tasks/main.yml'\n\n# source from: release-v3.26.4/manifests/calico-etcd.yaml\n\n# Source: calico/templates/calico-kube-controllers.yaml\n# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict\n\napiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\nspec:\n maxUnavailable: 1\n selector:\n matchLabels:\n k8s-app: calico-kube-controllers\n---\n# Source: calico/templates/calico-kube-controllers.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-node\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-cni-plugin\n namespace: kube-system\n---\n# Source: calico/templates/calico-config.yaml\n# This ConfigMap is used to configure a self-hosted Calico installation.\nkind: ConfigMap\napiVersion: v1\nmetadata:\n name: calico-config\n namespace: kube-system\ndata:\n # Configure this with the location of your etcd cluster.\n etcd_endpoints: \"{{ ETCD_ENDPOINTS }}\"\n # If you're using TLS enabled etcd uncomment the following.\n # You must also populate the Secret below with these files.\n etcd_ca: \"/calico-secrets/etcd-ca\"\n etcd_cert: \"/calico-secrets/etcd-cert\"\n etcd_key: \"/calico-secrets/etcd-key\"\n # Typha is disabled.\n typha_service_name: \"none\"\n # Configure the backend to use.\n calico_backend: \"{{ CALICO_NETWORKING_BACKEND }}\"\n\n # Configure the MTU to use for workload interfaces and tunnels.\n # By default, MTU is auto-detected, and explicitly setting this field should not be required.\n # You can override auto-detection by providing a non-zero value.\n veth_mtu: \"0\"\n\n # The CNI network configuration to install on each node. The special\n # values in this config will be automatically populated.\n cni_network_config: |-\n {\n \"name\": \"k8s-pod-network\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"calico\",\n \"log_level\": \"info\",\n \"log_file_path\": \"/var/log/calico/cni/cni.log\",\n \"etcd_endpoints\": \"{{ ETCD_ENDPOINTS }}\",\n \"etcd_key_file\": \"/etc/calico/ssl/calico-key.pem\",\n \"etcd_cert_file\": \"/etc/calico/ssl/calico.pem\",\n \"etcd_ca_cert_file\": \"{{ ca_dir }}/ca.pem\",\n \"mtu\": __CNI_MTU__,\n \"ipam\": {\n \"type\": \"calico-ipam\"\n },\n \"policy\": {\n \"type\": \"k8s\"\n },\n \"kubernetes\": {\n \"kubeconfig\": \"/etc/cni/net.d/calico-kubeconfig\"\n }\n },\n {\n \"type\": \"portmap\",\n \"snat\": true,\n \"capabilities\": {\"portMappings\": true}\n },\n {\n \"type\": \"bandwidth\",\n \"capabilities\": {\"bandwidth\": true}\n }\n ]\n }\n---\n# Source: calico/templates/calico-kube-controllers-rbac.yaml\n# Include a clusterrole for the kube-controllers component,\n# and bind it to the calico-kube-controllers serviceaccount.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-kube-controllers\nrules:\n # Pods are monitored for changing labels.\n # The node controller monitors Kubernetes nodes.\n # Namespace and serviceaccount labels are used for policy.\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n - serviceaccounts\n verbs:\n - watch\n - list\n - get\n # Watch for changes to Kubernetes NetworkPolicies.\n - apiGroups: [\"networking.k8s.io\"]\n resources:\n - networkpolicies\n verbs:\n - watch\n - list\n---\n# Source: calico/templates/calico-node-rbac.yaml\n# Include a clusterrole for the calico-node DaemonSet,\n# and bind it to the calico-node serviceaccount.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-node\nrules:\n # Used for creating service account tokens to be used by the CNI plugin\n - apiGroups: [\"\"]\n resources:\n - serviceaccounts/token\n resourceNames:\n - calico-cni-plugin\n verbs:\n - create\n # The CNI plugin needs to get pods, nodes, and namespaces.\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n verbs:\n - get\n # EndpointSlices are used for Service-based network policy rule\n # enforcement.\n - apiGroups: [\"discovery.k8s.io\"]\n resources:\n - endpointslices\n verbs:\n - watch\n - list\n - apiGroups: [\"\"]\n resources:\n - endpoints\n - services\n verbs:\n # Used to discover service IPs for advertisement.\n - watch\n - list\n # Pod CIDR auto-detection on kubeadm needs access to config maps.\n - apiGroups: [\"\"]\n resources:\n - configmaps\n verbs:\n - get\n - apiGroups: [\"\"]\n resources:\n - nodes/status\n verbs:\n # Needed for clearing NodeNetworkUnavailable flag.\n - patch\n---\n# Source: calico/templates/calico-node-rbac.yaml\n# CNI cluster role\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-cni-plugin\nrules:\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n verbs:\n - get\n---\n# Source: calico/templates/calico-kube-controllers-rbac.yaml\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-kube-controllers\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-kube-controllers\nsubjects:\n- kind: ServiceAccount\n name: calico-kube-controllers\n namespace: kube-system\n---\n# Source: calico/templates/calico-node-rbac.yaml\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: calico-node\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-node\nsubjects:\n- kind: ServiceAccount\n name: calico-node\n namespace: kube-system\n---\n# Source: calico/templates/calico-node-rbac.yaml\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: calico-cni-plugin\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-cni-plugin\nsubjects:\n- kind: ServiceAccount\n name: calico-cni-plugin\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\n# This manifest installs the calico-node container, as well\n# as the CNI plugins and network config on\n# each master and worker node in a Kubernetes cluster.\nkind: DaemonSet\napiVersion: apps/v1\nmetadata:\n name: calico-node\n namespace: kube-system\n labels:\n k8s-app: calico-node\nspec:\n selector:\n matchLabels:\n k8s-app: calico-node\n updateStrategy:\n type: RollingUpdate\n rollingUpdate:\n maxUnavailable: 1\n template:\n metadata:\n labels:\n k8s-app: calico-node\n spec:\n nodeSelector:\n kubernetes.io/os: linux\n hostNetwork: true\n tolerations:\n # Make sure calico-node gets scheduled on all nodes.\n - effect: NoSchedule\n operator: Exists\n # Mark the pod as a critical add-on for rescheduling.\n - key: CriticalAddonsOnly\n operator: Exists\n - effect: NoExecute\n operator: Exists\n serviceAccountName: calico-node\n # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a \"force\n # deletion\": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.\n terminationGracePeriodSeconds: 0\n priorityClassName: system-node-critical\n initContainers:\n # This container installs the CNI binaries\n # and CNI network config file on each node.\n - name: install-cni\n image: easzlab.io.local:5000/easzlab/cni:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n command: [\"/opt/cni/bin/install\"]\n envFrom:\n - configMapRef:\n # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.\n name: kubernetes-services-endpoint\n optional: true\n env:\n # Name of the CNI config file to create.\n - name: CNI_CONF_NAME\n value: \"10-calico.conflist\"\n # The CNI network config to install on each node.\n - name: CNI_NETWORK_CONFIG\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: cni_network_config\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # CNI MTU Config variable\n - name: CNI_MTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Prevents the container from sleeping forever.\n - name: SLEEP\n value: \"false\"\n volumeMounts:\n - mountPath: /host/opt/cni/bin\n name: cni-bin-dir\n - mountPath: /host/etc/cni/net.d\n name: cni-net-dir\n - mountPath: /calico-secrets\n name: etcd-certs\n securityContext:\n privileged: true\n # This init container mounts the necessary filesystems needed by the BPF data plane\n # i.e. bpf at /sys/fs/bpf and cgroup2 at /run/calico/cgroup. Calico-node initialisation is executed\n # in best effort fashion, i.e. no failure for errors, to not disrupt pod creation in iptable mode.\n - name: \"mount-bpffs\"\n image: easzlab.io.local:5000/easzlab/node:{{ calico_ver }} \n imagePullPolicy: IfNotPresent\n command: [\"calico-node\", \"-init\", \"-best-effort\"]\n volumeMounts:\n - mountPath: /sys/fs\n name: sys-fs\n # Bidirectional is required to ensure that the new mount we make at /sys/fs/bpf propagates to the host\n # so that it outlives the init container.\n mountPropagation: Bidirectional\n - mountPath: /var/run/calico\n name: var-run-calico\n # Bidirectional is required to ensure that the new mount we make at /run/calico/cgroup propagates to the host\n # so that it outlives the init container.\n mountPropagation: Bidirectional\n # Mount /proc/ from host which usually is an init program at /nodeproc. It's needed by mountns binary,\n # executed by calico-node, to mount root cgroup2 fs at /run/calico/cgroup to attach CTLB programs correctly.\n - mountPath: /nodeproc\n name: nodeproc\n readOnly: true\n securityContext:\n privileged: true\n containers:\n # Runs calico-node container on each Kubernetes node. This\n # container programs network policy and routes on each\n # host.\n - name: calico-node\n image: easzlab.io.local:5000/easzlab/node:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n envFrom:\n - configMapRef:\n # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.\n name: kubernetes-services-endpoint\n optional: true\n env:\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # Location of the CA certificate for etcd.\n - name: ETCD_CA_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_ca\n # Location of the client key for etcd.\n - name: ETCD_KEY_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_key\n # Location of the client certificate for etcd.\n - name: ETCD_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_cert\n # Set noderef for node controller.\n - name: CALICO_K8S_NODE_REF\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n # Choose the backend to use.\n - name: CALICO_NETWORKING_BACKEND\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: calico_backend\n # Cluster type to identify the deployment type\n - name: CLUSTER_TYPE\n value: \"k8s,bgp\"\n # Auto-detect the BGP IP address.\n - name: IP\n value: \"autodetect\"\n - name: IP_AUTODETECTION_METHOD\n value: \"{{ IP_AUTODETECTION_METHOD }}\"\n # Enable IPIP\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - name: CALICO_IPV4POOL_IPIP\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n{% endif %}\n # Enable or Disable VXLAN on the default IP pool.\n{% if CALICO_NETWORKING_BACKEND == \"vxlan\" %}\n - name: CALICO_IPV4POOL_VXLAN\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n - name: CALICO_IPV6POOL_VXLAN\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n{% endif %}\n # Set MTU for tunnel device used if ipip is enabled\n - name: FELIX_IPINIPMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Set MTU for the VXLAN tunnel device.\n - name: FELIX_VXLANMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Set MTU for the Wireguard tunnel device.\n - name: FELIX_WIREGUARDMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # The default IPv4 pool to create on startup if none exists. Pod IPs will be\n # chosen from this range. Changing this value after installation will have\n # no effect. This should fall within `--cluster-cidr`.\n - name: CALICO_IPV4POOL_CIDR\n value: \"{{ CLUSTER_CIDR }}\"\n # Disable file logging so `kubectl logs` works.\n - name: CALICO_DISABLE_FILE_LOGGING\n value: \"true\"\n # Set Felix endpoint to host default action to ACCEPT.\n - name: FELIX_DEFAULTENDPOINTTOHOSTACTION\n value: \"ACCEPT\"\n # Disable IPv6 on Kubernetes.\n - name: FELIX_IPV6SUPPORT\n value: \"false\"\n - name: FELIX_HEALTHENABLED\n value: \"true\"\n # Set Kubernetes NodePorts: If services do use NodePorts outside Calico’s expected range,\n # Calico will treat traffic to those ports as host traffic instead of pod traffic.\n - name: FELIX_KUBENODEPORTRANGES\n value: \"{{ NODE_PORT_RANGE.split('-')[0] }}:{{ NODE_PORT_RANGE.split('-')[1] }}\"\n - name: FELIX_PROMETHEUSMETRICSENABLED\n value: \"false\"\n securityContext:\n privileged: true\n resources:\n requests:\n cpu: 250m\n lifecycle:\n preStop:\n exec:\n command:\n - /bin/calico-node\n - -shutdown\n livenessProbe:\n exec:\n command:\n - /bin/calico-node\n - -felix-live\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - -bird-live\n{% endif %}\n periodSeconds: 10\n initialDelaySeconds: 10\n failureThreshold: 6\n timeoutSeconds: 10\n readinessProbe:\n exec:\n command:\n - /bin/calico-node\n - -felix-ready\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - -bird-ready\n{% endif %}\n periodSeconds: 10\n timeoutSeconds: 10\n volumeMounts:\n # For maintaining CNI plugin API credentials.\n - mountPath: /host/etc/cni/net.d\n name: cni-net-dir\n readOnly: false\n - mountPath: /lib/modules\n name: lib-modules\n readOnly: true\n - mountPath: /run/xtables.lock\n name: xtables-lock\n readOnly: false\n - mountPath: /var/run/calico\n name: var-run-calico\n readOnly: false\n - mountPath: /var/lib/calico\n name: var-lib-calico\n readOnly: false\n - mountPath: /calico-secrets\n name: etcd-certs\n - name: policysync\n mountPath: /var/run/nodeagent\n # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the\n # parent directory.\n - name: bpffs\n mountPath: /sys/fs/bpf\n - name: cni-log-dir\n mountPath: /var/log/calico/cni\n readOnly: true\n volumes:\n # Used by calico-node.\n - name: lib-modules\n hostPath:\n path: /lib/modules\n - name: var-run-calico\n hostPath:\n path: /var/run/calico\n - name: var-lib-calico\n hostPath:\n path: /var/lib/calico\n - name: xtables-lock\n hostPath:\n path: /run/xtables.lock\n type: FileOrCreate\n - name: sys-fs\n hostPath:\n path: /sys/fs/\n type: DirectoryOrCreate\n - name: bpffs\n hostPath:\n path: /sys/fs/bpf\n type: Directory\n # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.\n - name: nodeproc\n hostPath:\n path: /proc\n # Used to install CNI.\n - name: cni-bin-dir\n hostPath:\n path: /opt/cni/bin\n - name: cni-net-dir\n hostPath:\n path: /etc/cni/net.d\n # Used to access CNI logs.\n - name: cni-log-dir\n hostPath:\n path: /var/log/calico/cni\n # Mount in the etcd TLS secrets with mode 400.\n # See https://kubernetes.io/docs/concepts/configuration/secret/\n - name: etcd-certs\n secret:\n secretName: calico-etcd-secrets\n defaultMode: 0400\n # Used to create per-pod Unix Domain Sockets\n - name: policysync\n hostPath:\n type: DirectoryOrCreate\n path: /var/run/nodeagent\n---\n# Source: calico/templates/calico-kube-controllers.yaml\n# See https://github.com/projectcalico/kube-controllers\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\nspec:\n # The controllers can only have a single active instance.\n replicas: 1\n selector:\n matchLabels:\n k8s-app: calico-kube-controllers\n strategy:\n type: Recreate\n template:\n metadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\n spec:\n nodeSelector:\n kubernetes.io/os: linux\n tolerations:\n # Mark the pod as a critical add-on for rescheduling.\n - key: CriticalAddonsOnly\n operator: Exists\n - key: node-role.kubernetes.io/master\n effect: NoSchedule\n - key: node-role.kubernetes.io/control-plane\n effect: NoSchedule\n serviceAccountName: calico-kube-controllers\n priorityClassName: system-cluster-critical\n # The controllers must run in the host network namespace so that\n # it isn't governed by policy that would prevent it from working.\n hostNetwork: true\n containers:\n - name: calico-kube-controllers\n image: easzlab.io.local:5000/easzlab/kube-controllers:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n env:\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # Location of the CA certificate for etcd.\n - name: ETCD_CA_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_ca\n # Location of the client key for etcd.\n - name: ETCD_KEY_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_key\n # Location of the client certificate for etcd.\n - name: ETCD_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_cert\n # Choose which controllers to run.\n - name: ENABLED_CONTROLLERS\n value: policy,namespace,serviceaccount,workloadendpoint,node\n volumeMounts:\n # Mount in the etcd TLS secrets.\n - mountPath: /calico-secrets\n name: etcd-certs\n livenessProbe:\n exec:\n command:\n - /usr/bin/check-status\n - -l\n periodSeconds: 10\n initialDelaySeconds: 10\n failureThreshold: 6\n timeoutSeconds: 10\n readinessProbe:\n exec:\n command:\n - /usr/bin/check-status\n - -r\n periodSeconds: 10\n volumes:\n # Mount in the etcd TLS secrets with mode 400.\n # See https://kubernetes.io/docs/concepts/configuration/secret/\n - name: etcd-certs\n secret:\n secretName: calico-etcd-secrets\n defaultMode: 0440\n"
},
{
"path": "roles/calico/templates/calico-v3.28.yaml.j2",
"content": "---\n# download Release archive: https://github.com/projectcalico/calico/releases/download/v3.28.2/release-v3.28.2.tgz\n# Release notes: https://projectcalico.docs.tigera.io/archive/v3.28/release-notes/\n\n# Datastore: etcd, using Typha is redundant and not recommended.\n# Kubeasz uses command ( kubectl create) to create etcd-secrets, see more in 'roles/calico/tasks/main.yml'\n\n# source from: release-v3.28.2/manifests/calico-etcd.yaml\n\n# Source: calico/templates/calico-kube-controllers.yaml\n# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict\n\napiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\nspec:\n maxUnavailable: 1\n selector:\n matchLabels:\n k8s-app: calico-kube-controllers\n---\n# Source: calico/templates/calico-kube-controllers.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-node\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: calico-cni-plugin\n namespace: kube-system\n---\n# Source: calico/templates/calico-config.yaml\n# This ConfigMap is used to configure a self-hosted Calico installation.\nkind: ConfigMap\napiVersion: v1\nmetadata:\n name: calico-config\n namespace: kube-system\ndata:\n # Configure this with the location of your etcd cluster.\n etcd_endpoints: \"{{ ETCD_ENDPOINTS }}\"\n # If you're using TLS enabled etcd uncomment the following.\n # You must also populate the Secret below with these files.\n etcd_ca: \"/calico-secrets/etcd-ca\"\n etcd_cert: \"/calico-secrets/etcd-cert\"\n etcd_key: \"/calico-secrets/etcd-key\"\n # Typha is disabled.\n typha_service_name: \"none\"\n # Configure the backend to use.\n calico_backend: \"{{ CALICO_NETWORKING_BACKEND }}\"\n\n # Configure the MTU to use for workload interfaces and tunnels.\n # By default, MTU is auto-detected, and explicitly setting this field should not be required.\n # You can override auto-detection by providing a non-zero value.\n veth_mtu: \"0\"\n\n # The CNI network configuration to install on each node. The special\n # values in this config will be automatically populated.\n cni_network_config: |-\n {\n \"name\": \"k8s-pod-network\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"calico\",\n \"log_level\": \"info\",\n \"log_file_path\": \"/var/log/calico/cni/cni.log\",\n \"etcd_endpoints\": \"{{ ETCD_ENDPOINTS }}\",\n \"etcd_key_file\": \"/etc/calico/ssl/calico-key.pem\",\n \"etcd_cert_file\": \"/etc/calico/ssl/calico.pem\",\n \"etcd_ca_cert_file\": \"{{ ca_dir }}/ca.pem\",\n \"mtu\": __CNI_MTU__,\n \"ipam\": {\n \"type\": \"calico-ipam\"\n },\n \"policy\": {\n \"type\": \"k8s\"\n },\n \"kubernetes\": {\n \"kubeconfig\": \"/etc/cni/net.d/calico-kubeconfig\"\n }\n },\n {\n \"type\": \"portmap\",\n \"snat\": true,\n \"capabilities\": {\"portMappings\": true}\n },\n {\n \"type\": \"bandwidth\",\n \"capabilities\": {\"bandwidth\": true}\n }\n ]\n }\n---\n# Source: calico/templates/calico-kube-controllers-rbac.yaml\n# Include a clusterrole for the kube-controllers component,\n# and bind it to the calico-kube-controllers serviceaccount.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-kube-controllers\nrules:\n # Pods are monitored for changing labels.\n # The node controller monitors Kubernetes nodes.\n # Namespace and serviceaccount labels are used for policy.\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n - serviceaccounts\n verbs:\n - watch\n - list\n - get\n # Watch for changes to Kubernetes NetworkPolicies.\n - apiGroups: [\"networking.k8s.io\"]\n resources:\n - networkpolicies\n verbs:\n - watch\n - list\n---\n# Source: calico/templates/calico-node-rbac.yaml\n# Include a clusterrole for the calico-node DaemonSet,\n# and bind it to the calico-node serviceaccount.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-node\nrules:\n # Used for creating service account tokens to be used by the CNI plugin\n - apiGroups: [\"\"]\n resources:\n - serviceaccounts/token\n resourceNames:\n - calico-cni-plugin\n verbs:\n - create\n # The CNI plugin needs to get pods, nodes, and namespaces.\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n verbs:\n - get\n # EndpointSlices are used for Service-based network policy rule\n # enforcement.\n - apiGroups: [\"discovery.k8s.io\"]\n resources:\n - endpointslices\n verbs:\n - watch\n - list\n - apiGroups: [\"\"]\n resources:\n - endpoints\n - services\n verbs:\n # Used to discover service IPs for advertisement.\n - watch\n - list\n # Pod CIDR auto-detection on kubeadm needs access to config maps.\n - apiGroups: [\"\"]\n resources:\n - configmaps\n verbs:\n - get\n - apiGroups: [\"\"]\n resources:\n - nodes/status\n verbs:\n # Needed for clearing NodeNetworkUnavailable flag.\n - patch\n---\n# Source: calico/templates/calico-node-rbac.yaml\n# CNI cluster role\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-cni-plugin\nrules:\n - apiGroups: [\"\"]\n resources:\n - pods\n - nodes\n - namespaces\n verbs:\n - get\n---\n# Source: calico/templates/calico-kube-controllers-rbac.yaml\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n name: calico-kube-controllers\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-kube-controllers\nsubjects:\n- kind: ServiceAccount\n name: calico-kube-controllers\n namespace: kube-system\n---\n# Source: calico/templates/calico-node-rbac.yaml\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: calico-node\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-node\nsubjects:\n- kind: ServiceAccount\n name: calico-node\n namespace: kube-system\n---\n# Source: calico/templates/calico-node-rbac.yaml\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: calico-cni-plugin\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: calico-cni-plugin\nsubjects:\n- kind: ServiceAccount\n name: calico-cni-plugin\n namespace: kube-system\n---\n# Source: calico/templates/calico-node.yaml\n# This manifest installs the calico-node container, as well\n# as the CNI plugins and network config on\n# each master and worker node in a Kubernetes cluster.\nkind: DaemonSet\napiVersion: apps/v1\nmetadata:\n name: calico-node\n namespace: kube-system\n labels:\n k8s-app: calico-node\nspec:\n selector:\n matchLabels:\n k8s-app: calico-node\n updateStrategy:\n type: RollingUpdate\n rollingUpdate:\n maxUnavailable: 1\n template:\n metadata:\n labels:\n k8s-app: calico-node\n spec:\n nodeSelector:\n kubernetes.io/os: linux\n hostNetwork: true\n tolerations:\n # Make sure calico-node gets scheduled on all nodes.\n - effect: NoSchedule\n operator: Exists\n # Mark the pod as a critical add-on for rescheduling.\n - key: CriticalAddonsOnly\n operator: Exists\n - effect: NoExecute\n operator: Exists\n serviceAccountName: calico-node\n # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a \"force\n # deletion\": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.\n terminationGracePeriodSeconds: 0\n priorityClassName: system-node-critical\n initContainers:\n # This container installs the CNI binaries\n # and CNI network config file on each node.\n - name: install-cni\n image: easzlab.io.local:5000/easzlab/cni:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n command: [\"/opt/cni/bin/install\"]\n envFrom:\n - configMapRef:\n # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.\n name: kubernetes-services-endpoint\n optional: true\n env:\n # Name of the CNI config file to create.\n - name: CNI_CONF_NAME\n value: \"10-calico.conflist\"\n # The CNI network config to install on each node.\n - name: CNI_NETWORK_CONFIG\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: cni_network_config\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # CNI MTU Config variable\n - name: CNI_MTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Prevents the container from sleeping forever.\n - name: SLEEP\n value: \"false\"\n volumeMounts:\n - mountPath: /host/opt/cni/bin\n name: cni-bin-dir\n - mountPath: /host/etc/cni/net.d\n name: cni-net-dir\n - mountPath: /calico-secrets\n name: etcd-certs\n securityContext:\n privileged: true\n # This init container mounts the necessary filesystems needed by the BPF data plane\n # i.e. bpf at /sys/fs/bpf and cgroup2 at /run/calico/cgroup. Calico-node initialisation is executed\n # in best effort fashion, i.e. no failure for errors, to not disrupt pod creation in iptable mode.\n - name: \"mount-bpffs\"\n image: easzlab.io.local:5000/easzlab/node:{{ calico_ver }} \n imagePullPolicy: IfNotPresent\n command: [\"calico-node\", \"-init\", \"-best-effort\"]\n volumeMounts:\n - mountPath: /sys/fs\n name: sys-fs\n # Bidirectional is required to ensure that the new mount we make at /sys/fs/bpf propagates to the host\n # so that it outlives the init container.\n mountPropagation: Bidirectional\n - mountPath: /var/run/calico\n name: var-run-calico\n # Bidirectional is required to ensure that the new mount we make at /run/calico/cgroup propagates to the host\n # so that it outlives the init container.\n mountPropagation: Bidirectional\n # Mount /proc/ from host which usually is an init program at /nodeproc. It's needed by mountns binary,\n # executed by calico-node, to mount root cgroup2 fs at /run/calico/cgroup to attach CTLB programs correctly.\n - mountPath: /nodeproc\n name: nodeproc\n readOnly: true\n securityContext:\n privileged: true\n containers:\n # Runs calico-node container on each Kubernetes node. This\n # container programs network policy and routes on each\n # host.\n - name: calico-node\n image: easzlab.io.local:5000/easzlab/node:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n envFrom:\n - configMapRef:\n # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.\n name: kubernetes-services-endpoint\n optional: true\n env:\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # Location of the CA certificate for etcd.\n - name: ETCD_CA_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_ca\n # Location of the client key for etcd.\n - name: ETCD_KEY_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_key\n # Location of the client certificate for etcd.\n - name: ETCD_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_cert\n # Set noderef for node controller.\n - name: CALICO_K8S_NODE_REF\n valueFrom:\n fieldRef:\n fieldPath: spec.nodeName\n # Choose the backend to use.\n - name: CALICO_NETWORKING_BACKEND\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: calico_backend\n # Cluster type to identify the deployment type\n - name: CLUSTER_TYPE\n value: \"k8s,bgp\"\n # Auto-detect the BGP IP address.\n - name: IP\n value: \"autodetect\"\n - name: IP_AUTODETECTION_METHOD\n value: \"{{ IP_AUTODETECTION_METHOD }}\"\n # Enable IPIP\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - name: CALICO_IPV4POOL_IPIP\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n{% endif %}\n # Enable or Disable VXLAN on the default IP pool.\n{% if CALICO_NETWORKING_BACKEND == \"vxlan\" %}\n - name: CALICO_IPV4POOL_VXLAN\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n # Enable or Disable VXLAN on the default IPv6 IP pool.\n - name: CALICO_IPV6POOL_VXLAN\n value: \"{{ CALICO_ENABLE_OVERLAY }}\"\n{% endif %}\n # Set MTU for tunnel device used if ipip is enabled\n - name: FELIX_IPINIPMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Set MTU for the VXLAN tunnel device.\n - name: FELIX_VXLANMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # Set MTU for the Wireguard tunnel device.\n - name: FELIX_WIREGUARDMTU\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: veth_mtu\n # The default IPv4 pool to create on startup if none exists. Pod IPs will be\n # chosen from this range. Changing this value after installation will have\n # no effect. This should fall within `--cluster-cidr`.\n - name: CALICO_IPV4POOL_CIDR\n value: \"{{ CLUSTER_CIDR }}\"\n # Disable file logging so `kubectl logs` works.\n - name: CALICO_DISABLE_FILE_LOGGING\n value: \"true\"\n # Set Felix endpoint to host default action to ACCEPT.\n - name: FELIX_DEFAULTENDPOINTTOHOSTACTION\n value: \"ACCEPT\"\n # Disable IPv6 on Kubernetes.\n - name: FELIX_IPV6SUPPORT\n value: \"false\"\n - name: FELIX_HEALTHENABLED\n value: \"true\"\n # Set Kubernetes NodePorts: If services do use NodePorts outside Calico’s expected range,\n # Calico will treat traffic to those ports as host traffic instead of pod traffic.\n - name: FELIX_KUBENODEPORTRANGES\n value: \"{{ NODE_PORT_RANGE.split('-')[0] }}:{{ NODE_PORT_RANGE.split('-')[1] }}\"\n - name: FELIX_PROMETHEUSMETRICSENABLED\n value: \"false\"\n securityContext:\n privileged: true\n resources:\n requests:\n cpu: 250m\n lifecycle:\n preStop:\n exec:\n command:\n - /bin/calico-node\n - -shutdown\n livenessProbe:\n exec:\n command:\n - /bin/calico-node\n - -felix-live\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - -bird-live\n{% endif %}\n periodSeconds: 10\n initialDelaySeconds: 10\n failureThreshold: 6\n timeoutSeconds: 10\n readinessProbe:\n exec:\n command:\n - /bin/calico-node\n - -felix-ready\n{% if CALICO_NETWORKING_BACKEND == \"bird\" %}\n - -bird-ready\n{% endif %}\n periodSeconds: 10\n timeoutSeconds: 10\n volumeMounts:\n # For maintaining CNI plugin API credentials.\n - mountPath: /host/etc/cni/net.d\n name: cni-net-dir\n readOnly: false\n - mountPath: /lib/modules\n name: lib-modules\n readOnly: true\n - mountPath: /run/xtables.lock\n name: xtables-lock\n readOnly: false\n - mountPath: /var/run/calico\n name: var-run-calico\n readOnly: false\n - mountPath: /var/lib/calico\n name: var-lib-calico\n readOnly: false\n - mountPath: /calico-secrets\n name: etcd-certs\n - name: policysync\n mountPath: /var/run/nodeagent\n # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the\n # parent directory.\n - name: bpffs\n mountPath: /sys/fs/bpf\n - name: cni-log-dir\n mountPath: /var/log/calico/cni\n readOnly: true\n volumes:\n # Used by calico-node.\n - name: lib-modules\n hostPath:\n path: /lib/modules\n - name: var-run-calico\n hostPath:\n path: /var/run/calico\n type: DirectoryOrCreate\n - name: var-lib-calico\n hostPath:\n path: /var/lib/calico\n type: DirectoryOrCreate\n - name: xtables-lock\n hostPath:\n path: /run/xtables.lock\n type: FileOrCreate\n - name: sys-fs\n hostPath:\n path: /sys/fs/\n type: DirectoryOrCreate\n - name: bpffs\n hostPath:\n path: /sys/fs/bpf\n type: Directory\n # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.\n - name: nodeproc\n hostPath:\n path: /proc\n # Used to install CNI.\n - name: cni-bin-dir\n hostPath:\n path: /opt/cni/bin\n type: DirectoryOrCreate\n - name: cni-net-dir\n hostPath:\n path: /etc/cni/net.d\n # Used to access CNI logs.\n - name: cni-log-dir\n hostPath:\n path: /var/log/calico/cni\n # Mount in the etcd TLS secrets with mode 400.\n # See https://kubernetes.io/docs/concepts/configuration/secret/\n - name: etcd-certs\n secret:\n secretName: calico-etcd-secrets\n defaultMode: 0400\n # Used to create per-pod Unix Domain Sockets\n - name: policysync\n hostPath:\n type: DirectoryOrCreate\n path: /var/run/nodeagent\n---\n# Source: calico/templates/calico-kube-controllers.yaml\n# See https://github.com/projectcalico/kube-controllers\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\nspec:\n # The controllers can only have a single active instance.\n replicas: 1\n selector:\n matchLabels:\n k8s-app: calico-kube-controllers\n strategy:\n type: Recreate\n template:\n metadata:\n name: calico-kube-controllers\n namespace: kube-system\n labels:\n k8s-app: calico-kube-controllers\n spec:\n nodeSelector:\n kubernetes.io/os: linux\n tolerations:\n # Mark the pod as a critical add-on for rescheduling.\n - key: CriticalAddonsOnly\n operator: Exists\n - key: node-role.kubernetes.io/master\n effect: NoSchedule\n - key: node-role.kubernetes.io/control-plane\n effect: NoSchedule\n serviceAccountName: calico-kube-controllers\n priorityClassName: system-cluster-critical\n # The controllers must run in the host network namespace so that\n # it isn't governed by policy that would prevent it from working.\n hostNetwork: true\n containers:\n - name: calico-kube-controllers\n image: easzlab.io.local:5000/easzlab/kube-controllers:{{ calico_ver }}\n imagePullPolicy: IfNotPresent\n env:\n # The location of the etcd cluster.\n - name: ETCD_ENDPOINTS\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_endpoints\n # Location of the CA certificate for etcd.\n - name: ETCD_CA_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_ca\n # Location of the client key for etcd.\n - name: ETCD_KEY_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_key\n # Location of the client certificate for etcd.\n - name: ETCD_CERT_FILE\n valueFrom:\n configMapKeyRef:\n name: calico-config\n key: etcd_cert\n # Choose which controllers to run.\n - name: ENABLED_CONTROLLERS\n value: policy,namespace,serviceaccount,workloadendpoint,node\n volumeMounts:\n # Mount in the etcd TLS secrets.\n - mountPath: /calico-secrets\n name: etcd-certs\n livenessProbe:\n exec:\n command:\n - /usr/bin/check-status\n - -l\n periodSeconds: 10\n initialDelaySeconds: 10\n failureThreshold: 6\n timeoutSeconds: 10\n readinessProbe:\n exec:\n command:\n - /usr/bin/check-status\n - -r\n periodSeconds: 10\n volumes:\n # Mount in the etcd TLS secrets with mode 400.\n # See https://kubernetes.io/docs/concepts/configuration/secret/\n - name: etcd-certs\n secret:\n secretName: calico-etcd-secrets\n defaultMode: 0440\n"
},
{
"path": "roles/calico/templates/calicoctl.cfg.j2",
"content": "apiVersion: projectcalico.org/v3\nkind: CalicoAPIConfig\nmetadata:\nspec:\n datastoreType: \"etcdv3\"\n etcdEndpoints: {{ ETCD_ENDPOINTS }}\n etcdKeyFile: /etc/calico/ssl/calico-key.pem\n etcdCertFile: /etc/calico/ssl/calico.pem\n etcdCACertFile: {{ ca_dir }}/ca.pem \n"
},
{
"path": "roles/calico/vars/main.yml",
"content": "# etcd 集群服务地址列表, 根据etcd组成员自动生成\nTMP_ENDPOINTS: \"{% for h in groups['etcd'] %}https://{{ h }}:2379,{% endfor %}\"\nETCD_ENDPOINTS: \"{{ TMP_ENDPOINTS.rstrip(',') }}\"\n\n# calico AS number\nCALICO_AS_NUMBER: 64512\n"
},
{
"path": "roles/chrony/chrony.yml",
"content": "- hosts:\n - kube_master\n - kube_node\n - etcd\n - ex_lb\n - chrony\n roles:\n - { role: chrony, when: \"groups['chrony']|length > 0\" }\n"
},
{
"path": "roles/chrony/defaults/main.yml",
"content": "# 设置时间源服务器【重要：集群内机器时间必须同步】\nntp_servers:\n - \"ntp1.aliyun.com\"\n - \"time1.cloud.tencent.com\"\n - \"0.cn.pool.ntp.org\"\n\n# 设置允许内部时间同步的网络段，比如\"10.0.0.0/8\"，默认全部允许\nlocal_network: \"0.0.0.0/0\"\n"
},
{
"path": "roles/chrony/tasks/main.yml",
"content": "- name: prepare some dirs\n file: name={{ item }} state=directory\n with_items:\n - \"/etc/chrony\"\n - \"/var/lib/chrony\"\n - \"/var/log/chrony\"\n\n- name: 卸载 ntp\n package: name=ntp state=absent\n ignore_errors: true\n\n- name: 下载二进制文件chronyd\n copy: src={{ base_dir }}/bin/chronyd dest=/usr/sbin/chronyd mode=0755\n\n- name: 创建chronyd的systemd unit文件\n template: src=chronyd.service.j2 dest=/etc/systemd/system/chronyd.service\n\n- name: 配置 chrony server\n template: src=server.conf.j2 dest=/etc/chrony/chrony.conf\n when: 'inventory_hostname == groups.chrony[0]'\n\n- name: 配置 chrony client\n template: src=client.conf.j2 dest=/etc/chrony/chrony.conf\n when: 'inventory_hostname != groups.chrony[0]'\n\n- name: 开机启用chronyd服务\n shell: systemctl disable chronyd && systemctl enable chronyd\n ignore_errors: true\n\n- name: 开启chronyd服务\n shell: systemctl daemon-reload && systemctl restart chronyd\n ignore_errors: true\n tags: restart_chronyd\n\n- name: 以轮询的方式等待chronyd服务启动\n shell: \"systemctl is-active chronyd.service\"\n register: svc_status\n until: '\"active\" in svc_status.stdout'\n retries: 3\n delay: 3\n tags: restart_chronyd\n"
},
{
"path": "roles/chrony/templates/chronyd.service.j2",
"content": "[Unit]\nDescription=chrony, an NTP client/server\nDocumentation=https://chrony.tuxfamily.org/documentation.html\nConflicts=systemd-timesyncd.service openntpd.service ntpd.service ntp.service ntpsec.service\nAfter=network.target\nConditionCapability=CAP_SYS_TIME\n\n[Service]\n# sysctl net.netfilter.nf_conntrack_count\nType=forking\nPIDFile=/var/run/chrony/chronyd.pid\nExecStart=/usr/sbin/chronyd -f /etc/chrony/chrony.conf\nExecStartPost=/sbin/iptables -t raw -A PREROUTING -p udp -m udp --dport 123 -j NOTRACK\nExecStartPost=/sbin/iptables -t raw -A OUTPUT -p udp -m udp --sport 123 -j NOTRACK\nPrivateTmp=yes\nProtectHome=yes\nProtectSystem=full\n\n[Install]\nWantedBy=multi-user.target\n"
},
{
"path": "roles/chrony/templates/client.conf.j2",
"content": "# Use local server \nserver {{ groups['chrony'][0] }} iburst\n\n# Record the rate at which the system clock gains/losses time.\ndriftfile /var/lib/chrony/drift\n\n# Allow the system clock to be stepped in the first three updates\n# if its offset is larger than 1 second.\nmakestep 1.0 3\n\n# This directive enables kernel synchronisation (every 11 minutes) of the\n# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.\nrtcsync\n\n# Specify directory for dumping measurements.\ndumpdir /var/lib/chrony\n\n# This directive designates subnets (or nodes) from which NTP clients are allowed\n# to access to 'chronyd'.\nallow {{ local_network }}\n\n# Stop bad estimates upsetting machine clock.\nmaxupdateskew 100.0\n\n# Ignor source level\nstratumweight 0\n\n# Comment this line out to turn off logging.\n#log tracking measurements statistics\nlogdir /var/log/chrony\nlog statistics measurements tracking\nnoclientlog\n"
},
{
"path": "roles/chrony/templates/server.conf.j2",
"content": "# Use public servers from the pool.ntp.org project.\n{% for HOST in ntp_servers %}\nserver {{ HOST }} iburst\n{% endfor %}\npool pool.ntp.org iburst\npool 2.debian.pool.ntp.org iburst\n\n# Record the rate at which the system clock gains/losses time.\ndriftfile /var/lib/chrony/drift\n\n# Allow the system clock to be stepped in the first three updates\n# if its offset is larger than 1 second.\nmakestep 1.0 3\n\n# This directive enables kernel synchronisation (every 11 minutes) of the\n# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.\nrtcsync\n\n# Specify directory for dumping measurements.\ndumpdir /var/lib/chrony\n\n# This directive lets 'chronyd' to serve time even if unsynchronised to any NTP server.\nlocal stratum 10\n\n# This directive designates subnets (or nodes) from which NTP clients are allowed\n# to access to 'chronyd'.\nallow {{ local_network }}\n\n# Stop bad estimates upsetting machine clock.\nmaxupdateskew 100.0\n\n# Ignor source level\nstratumweight 0\n\n# Comment this line out to turn off logging.\n#log tracking measurements statistics\nlogdir /var/log/chrony\nlog statistics measurements tracking\nnoclientlog\n"
},
{
"path": "roles/cilium/cilium.yml",
"content": "- hosts:\n - kube_master\n - kube_node\n roles:\n - cilium\n"
},
{
"path": "roles/cilium/files/star_war_example/http-sw-app.yaml",
"content": "---\napiVersion: v1\nkind: Service\nmetadata:\n name: deathstar\nspec:\n type: ClusterIP\n ports:\n - port: 80\n selector:\n org: empire\n class: deathstar\n---\napiVersion: apps/v1 \nkind: Deployment\nmetadata:\n name: deathstar\nspec:\n replicas: 2\n selector:\n matchLabels:\n org: empire\n class: deathstar\n template:\n metadata:\n labels:\n org: empire\n class: deathstar\n spec:\n containers:\n - name: deathstar\n image: docker.io/cilium/starwars\n---\napiVersion: v1\nkind: Pod\nmetadata:\n name: tiefighter\n labels:\n org: empire\n class: tiefighter\nspec:\n containers:\n - name: spaceship\n image: docker.io/tgraf/netperf\n---\napiVersion: v1\nkind: Pod\nmetadata:\n name: xwing\n labels:\n org: alliance\n class: xwing\nspec:\n containers:\n - name: spaceship\n image: docker.io/tgraf/netperf\n"
},
{
"path": "roles/cilium/files/star_war_example/sw_l3_l4_l7_policy.yaml",
"content": "apiVersion: \"cilium.io/v2\"\nkind: CiliumNetworkPolicy\ndescription: \"L7 policy to restrict access to specific HTTP call\"\nmetadata:\n name: \"rule1\"\nspec:\n endpointSelector:\n matchLabels:\n org: empire\n class: deathstar\n ingress:\n - fromEndpoints:\n - matchLabels:\n org: empire\n toPorts:\n - ports:\n - port: \"80\"\n protocol: TCP\n rules:\n http:\n - method: \"POST\"\n path: \"/v1/request-landing\"\n"
},
{
"path": "roles/cilium/files/star_war_example/sw_l3_l4_policy.yaml",
"content": "apiVersion: \"cilium.io/v2\"\nkind: CiliumNetworkPolicy\ndescription: \"L3-L4 policy to restrict deathstar access to empire ships only\"\nmetadata:\n name: \"rule1\"\nspec:\n endpointSelector:\n matchLabels:\n org: empire\n class: deathstar\n ingress:\n - fromEndpoints:\n - matchLabels:\n org: empire\n toPorts:\n - ports:\n - port: \"80\"\n protocol: TCP\n"
},
{
"path": "roles/cilium/tasks/main.yml",
"content": "# https://docs.cilium.io/en/stable/installation/k8s-install-helm/#k8s-install-helm\n- name: 转换内核版本为浮点数\n set_fact:\n KERNEL_VER: \"{{ ansible_kernel.split('-')[0].split('.')[0]|int + ansible_kernel.split('-')[0].split('.')[1]|int/100 }}\"\n\n- name: 检查内核版本>4.9\n fail: msg=\"kernel {{ ansible_kernel }} is too old for cilium installing\"\n when: \"KERNEL_VER|float <= 4.09\"\n\n- block:\n - name: 创建 cilium chart 个性化设置\n template: src=values.yaml.j2 dest={{ cluster_dir }}/yml/cilium-values.yaml\n\n - name: helm 删除 cilium {{ cilium_ver }}\n shell: \"{{ base_dir }}/bin/helm delete cilium -n kube-system || echo true; sleep 3\"\n tags: force_change_certs\n when: 'CHANGE_CA|bool'\n\n - name: helm 创建 cilium {{ cilium_ver }}\n shell: \"{{ base_dir }}/bin/helm upgrade cilium --install \\\n -n kube-system -f {{ cluster_dir }}/yml/cilium-values.yaml \\\n {{ base_dir }}/roles/cilium/files/cilium-{{ cilium_ver }}.tgz\"\n tags: force_change_certs\n run_once: true\n connection: local \n\n- name: 下载client工具\n copy: src={{ base_dir }}/bin/{{ item }} dest={{ bin_dir }}/{{ item }} mode=0755\n with_items:\n - cilium\n - hubble\n\n# 删除原有cni配置\n- name: 删除默认cni配置\n file: path=/etc/cni/net.d/10-default.conf state=absent\n\n# 等待网络插件部署成功，视下载镜像速度而定\n- name: 轮询等待cilium-node 运行\n shell: \"{{ base_dir }}/bin/kubectl get pod -n kube-system -owide -lk8s-app=cilium|grep ' {{ K8S_NODENAME }} '|awk '{print $3}'\"\n register: pod_status\n until: pod_status.stdout == \"Running\"\n retries: 15\n delay: 8\n ignore_errors: true\n connection: local\n tags: force_change_certs\n\n# hubble-relay 可能需要重启一下\n- name: 重启hubble-relay pod\n shell: \"{{ base_dir }}/bin/kubectl -n kube-system scale deploy hubble-relay --replicas=0 && sleep 5 && \\\n {{ base_dir }}/bin/kubectl -n kube-system scale deploy hubble-relay --replicas=1\"\n run_once: true\n connection: local\n when: \"cilium_hubble_enabled|bool\"\n tags: force_change_certs\n"
},
{
"path": "roles/cilium/templates/values.yaml.j2",
"content": "image:\n repository: \"easzlab.io.local:5000/cilium/cilium\"\n tag: \"v{{ cilium_ver }}\"\n useDigest: false\n\n# -- Additional agent container arguments.\n{% if ENABLE_LOCAL_DNS_CACHE %}\nextraArgs:\n - --exclude-local-address=\"{{ LOCAL_DNS_CACHE }}/32\"\n{% endif %}\n\nresources:\n limits:\n cpu: 4000m\n memory: 4Gi\n requests:\n cpu: 100m\n memory: 512Mi\n\n{% if cilium_hubble_enabled %}\nhubble:\n enabled: true\n relay:\n enabled: true\n image:\n repository: \"easzlab.io.local:5000/cilium/hubble-relay\"\n tag: \"v{{ cilium_ver }}\"\n useDigest: false\n ui:\n{% if cilium_hubble_ui_enabled %}\n enabled: true\n{% else %}\n enabled: false\n{% endif %}\n backend:\n image:\n repository: \"easzlab.io.local:5000/cilium/hubble-ui-backend\"\n tag: \"v0.13.2\"\n useDigest: false\n frontend:\n image:\n repository: \"easzlab.io.local:5000/cilium/hubble-ui\"\n tag: \"v0.13.2\"\n useDigest: false\n{% endif %}\n\nidentityAllocationMode: \"crd\"\n\nipam:\n # ref: https://docs.cilium.io/en/stable/concepts/networking/ipam/\n mode: \"cluster-pool\"\n operator:\n clusterPoolIPv4PodCIDRList: [\"{{ CLUSTER_CIDR }}\"]\n clusterPoolIPv4MaskSize: 24\n\n# -- Enable Layer 7 network policy.\nl7Proxy: true\n\n# -- Configure TLS configuration in the agent.\ntls:\n enabled: true\n secretsBackend: local\n\netcd:\n enabled: false\n\noperator:\n enabled: true\n image:\n repository: \"easzlab.io.local:5000/cilium/operator\"\n tag: \"v{{ cilium_ver }}\"\n useDigest: false\n replicas: 1\n resources:\n limits:\n cpu: 1000m\n memory: 1Gi\n requests:\n cpu: 100m\n memory: 128Mi\n\npreflight:\n enabled: false\n\nclustermesh:\n useAPIServer: false\n\nenvoy:\n enabled: false\n"
},
{
"path": "roles/clean/clean_node.yml",
"content": "- hosts: \"{{ NODE_TO_CLEAN }}\"\n roles:\n - clean\n"
},
{
"path": "roles/clean/defaults/main.yml",
"content": "# 是否删除 kube_master 相关服务\nDEL_MASTER: \"no\"\n\n# 是否删除 kube_node 相关服务\nDEL_NODE: \"no\"\n\n# 是否删除 etc 相关服务\nDEL_ETCD: \"no\"\n\n# 是否删除 lb 相关服务\nDEL_LB: \"no\"\n\n# 是否删除 chrony 相关服务\nDEL_CHRONY: \"no\"\n\n# 是否删除 kubeasz 环境变量\nDEL_ENV: \"yes\"\n"
},
{
"path": "roles/clean/tasks/clean_chrony.yml",
"content": "- block:\n - name: stop and disable chronyd\n service: name=chronyd state=stopped enabled=no\n ignore_errors: true\n\n - name: remove files and dirs\n file: name={{ item }} state=absent\n with_items:\n - \"/etc/chrony\"\n - \"/var/lib/chrony\"\n - \"/var/log/chrony\"\n - \"/var/run/chrony\"\n - \"/etc/systemd/system/chronyd.service\"\n ignore_errors: true\n when: \"groups['chrony']|length > 0\"\n"
},
{
"path": "roles/clean/tasks/clean_etcd.yml",
"content": "# to clean 'etcd' nodes\n- block:\n - name: stop and disable etcd service\n service:\n name: etcd\n state: stopped\n enabled: no\n ignore_errors: true\n\n - name: remove files and dirs\n file: name={{ item }} state=absent\n with_items:\n - \"{{ ETCD_DATA_DIR }}\"\n - \"{{ ETCD_WAL_DIR }}\"\n - \"/backup/k8s\"\n - \"/etc/systemd/system/etcd.service\"\n ignore_errors: true\n when: \"inventory_hostname in groups['etcd']\"\n"
},
{
"path": "roles/clean/tasks/clean_lb.yml",
"content": "# to clean 'lb' service\n- block:\n - name: get service info\n shell: 'systemctl list-units --type=service |grep -E \"l4lb|keepalived|ssh\"'\n register: service_info\n\n - name: remove service l4lb\n service: name=l4lb state=stopped enabled=no\n when: '\"l4lb\" in service_info.stdout'\n ignore_errors: true\n\n - name: remove service keepalived\n service: name=keepalived state=stopped enabled=no\n when: '\"keepalived\" in service_info.stdout'\n ignore_errors: true\n\n - name: remove files and dirs\n file: name={{ item }} state=absent\n with_items:\n - \"/etc/l4lb\"\n - \"/etc/keepalived\"\n - \"/etc/systemd/system/l4lb.service\"\n - \"/etc/systemd/system/keepalived.service\"\n - \"/usr/local/sbin/keepalived\"\n ignore_errors: true\n when: \"inventory_hostname in groups['ex_lb']\"\n"
},
{
"path": "roles/clean/tasks/clean_master.yml",
"content": "# to clean 'kube_master' nodes\n- name: stop and disable kube_master service\n service: name={{ item }} state=stopped enabled=no\n with_items:\n - kube-apiserver\n - kube-controller-manager\n - kube-scheduler\n ignore_errors: true\n when: \"inventory_hostname in groups['kube_master']\"\n\n- name: remove files and dirs of 'kube_master' nodes\n file: name={{ item }} state=absent\n with_items:\n - \"/var/run/kubernetes\"\n - \"/etc/systemd/system/kube-apiserver.service\"\n - \"/etc/systemd/system/kube-controller-manager.service\"\n - \"/etc/systemd/system/kube-scheduler.service\"\n ignore_errors: true\n when: \"inventory_hostname in groups['kube_master']\"\n"
},
{
"path": "roles/clean/tasks/clean_node.yml",
"content": "# to clean 'kube_node' nodes\n- block:\n - name: stop and disable kube_node service\n service: name={{ item }} state=stopped enabled=no\n with_items:\n - kube-lb\n - kubelet\n - kube-proxy\n ignore_errors: true\n\n - name: umount kubelet filesystems\n shell: \"mount | grep '/var/lib/kubelet'| awk '{print $3}'|xargs umount || exit 0\"\n ignore_errors: true\n\n - name: remove files and dirs of 'kube_node' nodes\n file: name={{ item }} state=absent\n with_items:\n - \"/var/lib/kubelet/\"\n - \"/var/lib/kube-proxy/\"\n - \"/etc/systemd/system/kube-lb.service\"\n - \"/etc/systemd/system/kubelet.service\"\n - \"/etc/systemd/system/kube-proxy.service\"\n - \"/etc/kube-lb/\"\n - \"/etc/kubernetes/\"\n - \"/opt/kubeasz_prepare_tasks\"\n - \"/root/.kube/config\"\n ignore_errors: true\n\n# to clean container runtime and networking\n - block:\n - name: to check if container 'kubeasz' is running\n shell: 'docker ps|grep kubeasz || echo \"NOT FOUND\"'\n register: install_info\n\n - block:\n - name: stop and disable docker service\n service:\n name: docker\n state: stopped\n enabled: no\n ignore_errors: true\n \n # as k8s-network-plugins use host-network, '/var/run/docker/netns/default' must be umounted\n - name: umount docker filesystem-1\n mount: path=/var/run/docker/netns/default state=unmounted\n \n - name: umount docker filesystem-2\n mount: path=/var/lib/docker/overlay state=unmounted\n \n - name: umount docker filesystem-3\n shell: \"echo /var/lib/docker/overlay2/*/merged|xargs umount || exit 0\"\n ignore_errors: true\n\n - name: umount docker filesystem-4\n shell: \"echo /var/lib/docker/containers/*/mounts/shm|xargs umount || exit 0\"\n ignore_errors: true\n\n - name: umount docker filesystem-5\n shell: \"echo /var/run/docker/netns/*|xargs umount || exit 0\"\n ignore_errors: true\n\n - name: remove files and dirs\n file: name={{ item }} state=absent\n with_items:\n - \"/var/lib/docker/\"\n - \"/var/lib/dockershim/\"\n - \"/var/run/docker/\"\n - \"/etc/docker/\"\n - \"/etc/systemd/system/docker.service\"\n - \"/etc/systemd/system/docker.service.requires/\"\n - \"/etc/systemd/system/docker.service.d/\"\n - \"/etc/bash_completion.d/docker\"\n - \"/usr/bin/docker\"\n when: \"'kubeasz' not in install_info.stdout\"\n ignore_errors: true\n when: CONTAINER_RUNTIME == 'docker'\n\n - block:\n - name: stop and disable {{ CONTAINERD_SERVICE_NAME }}\n service:\n name: containerd\n state: stopped\n enabled: no\n ignore_errors: true\n\n - name: umount containerd filesystems\n shell: \"mount | grep 'containerd/io.containerd'| awk '{print $3}'|xargs umount || exit 0\"\n ignore_errors: true\n\n - name: remove files and dirs\n file: name={{ item }} state=absent\n with_items:\n - \"{{ CONTAINERD_CONFIG_DIR }}\"\n - \"/etc/crictl.yaml\"\n - \"/etc/systemd/system/{{ CONTAINERD_SERVICE_NAME }}\"\n - \"/opt/containerd/\"\n - \"{{ CONTAINERD_ROOT_DIR }}\"\n - \"{{ CONTAINERD_STATE_DIR }}\"\n ignore_errors: true\n when: CONTAINER_RUNTIME == 'containerd'\n\n - name: remove files and dirs2\n file: name={{ item }} state=absent\n with_items:\n - \"/etc/cni/\"\n - \"/run/flannel/\"\n - \"/etc/calico/\"\n - \"/var/lib/calico/\"\n - \"/var/log/calico/\"\n - \"/etc/cilium/\"\n - \"/sys/fs/bpf/tc/\"\n - \"/var/lib/cni/\"\n - \"/var/lib/kube-router/\"\n - \"/var/run/openvswitch/\"\n - \"/var/run/ovn/\"\n - \"/etc/origin/openvswitch/\"\n - \"/etc/origin/ovn/\"\n - \"/var/log/openvswitch/\"\n - \"/var/log/ovn/\"\n - \"/var/log/kube-ovn/\"\n ignore_errors: true\n\n when: \"inventory_hostname in groups['kube_master'] or inventory_hostname in groups['kube_node']\"\n"
},
{
"path": "roles/clean/tasks/main.yml",
"content": "# \n- import_tasks: clean_etcd.yml\n when: 'DEL_ETCD == \"yes\"'\n\n- import_tasks: clean_master.yml\n when: 'DEL_MASTER == \"yes\"'\n\n- import_tasks: clean_node.yml\n when: 'DEL_NODE == \"yes\"'\n\n- import_tasks: clean_lb.yml\n when: 'DEL_LB == \"yes\"'\n\n- import_tasks: clean_chrony.yml\n when: 'DEL_CHRONY == \"yes\"'\n\n- name: clean 'ENV PATH'\n lineinfile:\n dest: ~/.bashrc\n state: absent\n regexp: '{{ item }}'\n with_items:\n - 'kubeasz'\n - 'helm completion'\n - 'kubectl completion'\n - 'crictl completion'\n - 'HELM_TLS_ENABLE'\n when: 'DEL_ENV == \"yes\"'\n\n- name: 删除 k8s_nodename 在节点的 /etc/hosts 地址解析\n blockinfile:\n path: /etc/hosts\n state: absent\n marker: \"### {mark} KUBEASZ MANAGED BLOCK\"\n\n #- name: remove binaries\n # file: name={{ item }} state=absent\n # with_items:\n # - \"/opt/kube/bin\"\n # when: 'DEL_ETCD == \"yes\" and DEL_NODE == \"yes\" and DEL_MASTER == \"yes\"'\n\n- name: 重启提示 WARNNING\n debug:\n msg: \"[重要]: 请重启节点以确保清除系统残留的虚拟网卡、路由信息、iptalbes|ipvs规则等 \\\n [IMPORTANT]: please reboot nodes, makesure to clean out net interfaces, routes and iptables/ipvs rules\"\n when: 'DEL_ETCD == \"yes\" and DEL_NODE == \"yes\" and DEL_MASTER == \"yes\"'\n"
},
{
"path": "roles/cluster-addon/files/kubeblocks_crds.yaml",
"content": "---\napiVersion: apiextensions.k8s.io/v1\nkind: CustomResourceDefinition\nmetadata:\n annotations:\n controller-gen.kubebuilder.io/version: v0.14.0\n labels:\n app.kubernetes.io/name: kubeblocks\n name: clusterdefinitions.apps.kubeblocks.io\nspec:\n group: apps.kubeblocks.io\n names:\n categories:\n - kubeblocks\n kind: ClusterDefinition\n listKind: ClusterDefinitionList\n plural: clusterdefinitions\n shortNames:\n - cd\n singular: clusterdefinition\n scope: Cluster\n versions:\n - additionalPrinterColumns:\n - description: topologies\n jsonPath: .status.topologies\n name: Topologies\n type: string\n - description: status phase\n jsonPath: .status.phase\n name: STATUS\n type: string\n - jsonPath: .metadata.creationTimestamp\n name: AGE\n type: date\n name: v1\n schema:\n openAPIV3Schema:\n description: |-\n ClusterDefinition defines the topology for databases or storage systems,\n offering a variety of topological configurations to meet diverse deployment needs and scenarios.\n\n\n It includes a list of Components and/or Shardings, each linked to a ComponentDefinition or a ShardingDefinition,\n which enhances reusability and reduce redundancy.\n For example, widely used components such as etcd and Zookeeper can be defined once and reused across multiple ClusterDefinitions,\n simplifying the setup of new systems.\n\n\n Additionally, ClusterDefinition also specifies the sequence of startup, upgrade, and shutdown between Components and/or Shardings,\n ensuring a controlled and predictable management of cluster lifecycles.\n properties:\n apiVersion:\n description: |-\n APIVersion defines the versioned schema of this representation of an object.\n Servers should convert recognized schemas to the latest internal value, and\n may reject unrecognized values.\n More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\n type: string\n kind:\n description: |-\n Kind is a string value representing the REST resource this object represents.\n Servers may infer this from the endpoint the client submits requests to.\n Cannot be updated.\n In CamelCase.\n More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\n type: string\n metadata:\n type: object\n spec:\n description: ClusterDefinitionSpec defines the desired state of ClusterDefinition.\n properties:\n topologies:\n description: Topologies defines all possible topologies within the\n cluster.\n items:\n description: ClusterTopology represents the definition for a specific\n cluster topology.\n properties:\n components:\n description: Components specifies the components in the topology.\n items:\n description: ClusterTopologyComponent defines a Component\n within a ClusterTopology.\n properties:\n compDef:\n description: \"Specifies the exact name, name prefix, or\n regular expression pattern for matching the name of\n the ComponentDefinition\\ncustom resource (CR) that defines\n the Component's characteristics and behavior.\\n\\n\\nThe\n system selects the ComponentDefinition CR with the latest\n version that matches the pattern.\\nThis approach allows:\\n\\n\\n1.\n Precise selection by providing the exact name of a ComponentDefinition\n CR.\\n2. Flexible and automatic selection of the most\n up-to-date ComponentDefinition CR\\n\\t by specifying\n a name prefix or regular expression pattern.\\n\\n\\nCannot\n be updated once set.\"\n maxLength: 64\n type: string\n name:\n description: |-\n Defines the unique identifier of the component within the cluster topology.\n\n\n It follows IANA Service naming rules and is used as part of the Service's DNS name.\n The name must start with a lowercase letter, can contain lowercase letters, numbers,\n and hyphens, and must end with a lowercase letter or number.\n\n\n If the @template field is set to true, the name will be used as a prefix to match the specific components dynamically created.\n\n\n Cannot be updated once set.\n maxLength: 16\n pattern: ^[a-z]([a-z0-9\\-]*[a-z0-9])?$\n type: string\n template:\n description: |-\n Specifies whether the topology component will be considered as a template for instantiating components upon user requests dynamically.\n\n\n Cannot be updated once set.\n type: boolean\n required:\n - compDef\n - name\n type: object\n maxItems: 128\n type: array\n default:\n description: |-\n Default indicates whether this topology serves as the default configuration.\n When set to true, this topology is automatically used unless another is explicitly specified.\n type: boolean\n name:\n description: |-\n Name is the unique identifier for the cluster topology.\n Cannot be updated.\n maxLength: 32\n type: string\n orders:\n description: |-\n Specifies the sequence in which components within a cluster topology are\n started, stopped, and upgraded.\n This ordering is crucial for maintaining the correct dependencies and operational flow across components.\n properties:\n provision:\n description: |-\n Specifies the order for creating and initializing entities.\n This is designed for entities that depend on one another. Entities without dependencies can be grouped together.\n\n\n Entities that can be provisioned independently or have no dependencies can be listed together in the same stage,\n separated by commas.\n items:\n type: string\n type: array\n terminate:\n description: |-\n Outlines the order for stopping and deleting entities.\n This sequence is designed for entities that require a graceful shutdown or have interdependencies.\n\n\n Entities that can be terminated independently or have no dependencies can be listed together in the same stage,\n separated by commas.\n items:\n type: string\n type: array\n update:\n description: |-\n Update determines the order for updating entities' specifications, such as image upgrades or resource scaling.\n This sequence is designed for entities that have dependencies or require specific update procedures.\n\n\n Entities that can be updated independently or have no dependencies can be listed together in the same stage,\n separated by commas.\n items:\n type: string\n type: array\n type: object\n shardings:\n description: Shardings specifies the shardings in the topology.\n items:\n description: ClusterTopologySharding defines a sharding within\n a ClusterTopology.\n properties:\n name:\n description: |-\n Defines the unique identifier of the sharding within the cluster topology.\n It follows IANA Service naming rules and is used as part of the Service's DNS name.\n The name must start with a lowercase letter, can contain lowercase letters, numbers,\n and hyphens, and must end with a lowercase letter or number.\n\n\n Cannot be updated once set.\n maxLength: 16\n pattern: ^[a-z]([a-z0-9\\-]*[a-z0-9])?$\n type: string\n shardingDef:\n description: |-\n Specifies the sharding definition that defines the characteristics and behavior of the sharding.\n\n\n The system selects the ShardingDefinition CR with the latest version that matches the pattern.\n This approach allows:\n\n\n 1. Precise selection by providing the exact name of a ShardingDefinition CR.\n 2. Flexible and automatic selection of the most up-to-date ShardingDefinition CR\n by specifying a regular expression pattern.\n\n\n Once set, this field cannot be updated.\n maxLength: 64\n type: string\n required:\n - name\n - shardingDef\n type: object\n maxItems: 128\n type: array\n required:\n - name\n type: object\n maxItems: 128\n minItems: 1\n type: array\n type: object\n status:\n description: ClusterDefinitionStatus defines the observed state of ClusterDefinition\n properties:\n message:\n description: Provides additional information about the current phase.\n type: string\n observedGeneration:\n description: Represents the most recent generation observed for this\n ClusterDefinition.\n format: int64\n type: integer\n phase:\n description: |-\n Specifies the current phase of the ClusterDefinition. Valid values are `empty`, `Available`, `Unavailable`.\n When `Available`, the ClusterDefinition is ready and can be referenced by related objects.\n enum:\n - Available\n - Unavailable\n type: string\n topologies:\n description: Topologies this ClusterDefinition supported.\n type: string\n type: object\n type: object\n served: true\n storage: true\n subresources:\n status: {}\n - additionalPrinterColumns:\n - description: topologies\n jsonPath: .status.topologies\n name: Topologies\n type: string\n - description: service references\n jsonPath: .status.serviceRefs\n name: ServiceRefs\n type: string\n - description: status phase\n jsonPath: .status.phase\n name: STATUS\n type: string\n - jsonPath: .metadata.creationTimestamp\n name: AGE\n type: date\n name: v1alpha1\n schema:\n openAPIV3Schema:\n description: |-\n ClusterDefinition defines the topology for databases or storage systems,\n offering a variety of topological configurations to meet diverse deployment needs and scenarios.\n\n\n It includes a list of Components, each linked to a ComponentDefinition, which enhances reusability and reduce redundancy.\n For example, widely used components such as etcd and Zookeeper can be defined once and reused across multiple ClusterDefinitions,\n simplifying the setup of new systems.\n\n\n Additionally, ClusterDefinition also specifies the sequence of startup, upgrade, and shutdown for Components,\n ensuring a controlled and predictable management of component lifecycles.\n properties:\n apiVersion:\n description: |-\n APIVersion defines the versioned schema of this representation of an object.\n Servers should convert recognized schemas to the latest internal value, and\n may reject unrecognized values.\n More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\n type: string\n kind:\n description: |-\n Kind is a string value representing the REST resource this object represents.\n Servers may infer this from the endpoint the client submits requests to.\n Cannot be updated.\n In CamelCase.\n More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\n type: string\n metadata:\n type: object\n spec:\n description: ClusterDefinitionSpec defines the desired state of ClusterDefinition.\n properties:\n componentDefs:\n description: |-\n Provides the definitions for the cluster components.\n\n\n Deprecated since v0.9.\n Components should now be individually defined using ComponentDefinition and\n collectively referenced via `topology.components`.\n This field is maintained for backward compatibility and its use is discouraged.\n Existing usage should be updated to the current preferred approach to avoid compatibility issues in future releases.\n items:\n description: |-\n ClusterComponentDefinition defines a Component within a ClusterDefinition but is deprecated and\n has been replaced by ComponentDefinition.\n\n\n Deprecated: Use ComponentDefinition instead. This type is deprecated as of version 0.8.\n properties:\n characterType:\n description: Defines well-known database component name, such\n as mongos(mongodb), proxy(redis), mariadb(mysql).\n type: string\n componentDefRef:\n description: |-\n Used to inject values from other components into the current component. Values will be saved and updated in a\n configmap and mounted to the current component.\n items:\n description: |-\n ComponentDefRef is used to select the component and its fields to be referenced.\n\n\n Deprecated since v0.8.\n properties:\n componentDefName:\n description: The name of the componentDef to be selected.\n type: string\n componentRefEnv:\n description: The values that are to be injected as environment\n variables into each component.\n items:\n description: |-\n ComponentRefEnv specifies name and value of an env.\n\n\n Deprecated since v0.8.\n properties:\n name:\n description: The name of the env, it must be a C\n identifier.\n pattern: ^[A-Za-z_][A-Za-z0-9_]*$\n type: string\n value:\n description: The value of the env.\n type: string\n valueFrom:\n description: The source from which the value of\n the env.\n properties:\n fieldPath:\n description: |-\n The jsonpath of the source to select when the Type is `FieldRef`.\n Two objects are registered in the jsonpath: `componentDef` and `components`:\n\n\n - `componentDef` is the component definition object specified in `componentRef.componentDefName`.\n - `components` are the component list objects referring to the component definition object.\n type: string\n format:\n default: =\"$POD_FQDN\"\n description: |-\n Defines the format of each headless service address.\n Three builtin variables can be used as placeholders: `$POD_ORDINAL`, `$POD_FQDN`, `$POD_NAME`\n\n\n - `$POD_ORDINAL` represents the ordinal of the pod.\n - `$POD_FQDN` represents the fully qualified domain name of the pod.\n - `$POD_NAME` represents the name of the pod.\n type: string\n joinWith:\n default: ','\n description: The string used to join the values\n of headless service addresses.\n type: string\n type:\n allOf:\n - enum:\n - FieldRef\n - ServiceRef\n - HeadlessServiceRef\n - enum:\n - FieldRef\n - ServiceRef\n - HeadlessServiceRef\n description: 'Specifies the source to select.\n It can be one of three types: `FieldRef`,\n `ServiceRef`, `HeadlessServiceRef`.'\n type: string\n required:\n - type\n type: object\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n failurePolicy:\n allOf:\n - enum:\n - Ignore\n - Fail\n - enum:\n - Ignore\n - Fail\n description: Defines the policy to be followed in case\n of a failure in finding the component.\n type: string\n required:\n - componentDefName\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - componentDefName\n x-kubernetes-list-type: map\n configSpecs:\n description: Defines the template of configurations.\n items:\n properties:\n asEnvFrom:\n description: |-\n Specifies the containers to inject the ConfigMap parameters as environment variables.\n\n\n This is useful when application images accept parameters through environment variables and\n generate the final configuration file in the startup script based on these variables.\n\n\n This field allows users to specify a list of container names, and KubeBlocks will inject the environment\n variables converted from the ConfigMap into these designated containers. This provides a flexible way to\n pass the configuration items from the ConfigMap to the container without modifying the image.\n\n\n Deprecated: `asEnvFrom` has been deprecated since 0.9.0 and will be removed in 0.10.0.\n Use `injectEnvTo` instead.\n items:\n type: string\n type: array\n x-kubernetes-list-type: set\n asSecret:\n description: Whether to store the final rendered parameters\n as a secret.\n type: boolean\n constraintRef:\n description: Specifies the name of the referenced configuration\n constraints object.\n maxLength: 63\n pattern: ^[a-z0-9]([a-z0-9\\.\\-]*[a-z0-9])?$\n type: string\n defaultMode:\n description: |-\n The operator attempts to set default file permissions for scripts (0555) and configurations (0444).\n However, certain database engines may require different file permissions.\n You can specify the desired file permissions here.\n\n\n Must be specified as an octal value between 0000 and 0777 (inclusive),\n or as a decimal value between 0 and 511 (inclusive).\n YAML supports both octal and decimal values for file permissions.\n\n\n Please note that this setting only affects the permissions of the files themselves.\n Directories within the specified path are not impacted by this setting.\n It's important to be aware that this setting might conflict with other options\n that influence the file mode, such as fsGroup.\n In such cases, the resulting file mode may have additional bits set.\n Refers to documents of k8s.ConfigMapVolumeSource.defaultMode for more information.\n format: int32\n type: integer\n injectEnvTo:\n description: |-\n Specifies the containers to inject the ConfigMap parameters as environment variables.\n\n\n This is useful when application images accept parameters through environment variables and\n generate the final configuration file in the startup script based on these variables.\n\n\n This field allows users to specify a list of container names, and KubeBlocks will inject the environment\n variables converted from the ConfigMap into these designated containers. This provides a flexible way to\n pass the configuration items from the ConfigMap to the container without modifying the image.\n items:\n type: string\n type: array\n x-kubernetes-list-type: set\n keys:\n description: |-\n Specifies the configuration files within the ConfigMap that support dynamic updates.\n\n\n A configuration template (provided in the form of a ConfigMap) may contain templates for multiple\n configuration files.\n Each configuration file corresponds to a key in the ConfigMap.\n Some of these configuration files may support dynamic modification and reloading without requiring\n a pod restart.\n\n\n If empty or omitted, all configuration files in the ConfigMap are assumed to support dynamic updates,\n and ConfigConstraint applies to all keys.\n items:\n type: string\n type: array\n x-kubernetes-list-type: set\n legacyRenderedConfigSpec:\n description: |-\n Specifies the secondary rendered config spec for pod-specific customization.\n\n\n The template is rendered inside the pod (by the \"config-manager\" sidecar container) and merged with the main\n template's render result to generate the final configuration file.\n\n\n This field is intended to handle scenarios where different pods within the same Component have\n varying configurations. It allows for pod-specific customization of the configuration.\n\n\n Note: This field will be deprecated in future versions, and the functionality will be moved to\n `cluster.spec.componentSpecs[*].instances[*]`.\n properties:\n namespace:\n default: default\n description: |-\n Specifies the namespace of the referenced configuration template ConfigMap object.\n An empty namespace is equivalent to the \"default\" namespace.\n pattern: ^[a-z0-9]([a-z0-9\\-]*[a-z0-9])?$\n type: string\n policy:\n default: none\n description: Defines the strategy for merging externally\n imported templates into component templates.\n enum:\n - patch\n - replace\n - none\n type: string\n templateRef:\n description: Specifies the name of the referenced\n configuration template ConfigMap object.\n pattern: ^[a-z0-9]([a-z0-9\\.\\-]*[a-z0-9])?$\n type: string\n required:\n - templateRef\n type: object\n name:\n description: Specifies the name of the configuration template.\n maxLength: 63\n pattern: ^[a-z0-9]([a-z0-9\\.\\-]*[a-z0-9])?$\n type: string\n namespace:\n default: default\n description: |-\n Specifies the namespace of the referenced configuration template ConfigMap object.\n An empty namespace is equivalent to the \"default\" namespace.\n maxLength: 63\n pattern: ^[a-z0-9]([a-z0-9\\-]*[a-z0-9])?$\n type: string\n reRenderResourceTypes:\n description: |-\n Specifies whether the configuration needs to be re-rendered after v-scale or h-scale operations to reflect changes.\n\n\n In some scenarios, the configuration may need to be updated to reflect the changes in resource allocation\n or cluster topology. Examples:\n\n\n - Redis: adjust maxmemory after v-scale operation.\n - MySQL: increase max connections after v-scale operation.\n - Zookeeper: update zoo.cfg with new node addresses after h-scale operation.\n items:\n description: RerenderResourceType defines the resource\n requirements for a component.\n enum:\n - vscale\n - hscale\n - tls\n - shardingHScale\n type: string\n type: array\n x-kubernetes-list-type: set\n templateRef:\n description: Specifies the name of the referenced configuration\n template ConfigMap object.\n maxLength: 63\n pattern: ^[a-z0-9]([a-z0-9\\.\\-]*[a-z0-9])?$\n type: string\n volumeName:\n description: |-\n Refers to the volume name of PodTemplate. The configuration file produced through the configuration\n template will be mounted to the corresponding volume. Must be a DNS_LABEL name.\n The volume name must be defined in podSpec.containers[*].volumeMounts.\n maxLength: 63\n pattern: ^[a-z]([a-z0-9\\-]*[a-z0-9])?$\n type: string\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n consensusSpec:\n description: Defines spec for `Consensus` workloads. It's required\n if the workload type is `Consensus`.\n properties:\n followers:\n description: Members of the consensus set that have voting\n rights but are not the leader.\n items:\n description: ConsensusMember is deprecated since v0.7.\n properties:\n accessMode:\n default: ReadWrite\n description: Specifies the services that this member\n is capable of providing.\n enum:\n - None\n - Readonly\n - ReadWrite\n type: string\n name:\n default: leader\n description: Specifies the name of the consensus member.\n type: string\n replicas:\n default: 0\n description: |-\n Indicates the number of Pods that perform this role.\n The default is 1 for `Leader`, 0 for `Learner`, others for `Followers`.\n format: int32\n minimum: 0\n type: integer\n required:\n - accessMode\n - name\n type: object\n type: array\n leader:\n description: Represents a single leader in the consensus\n set.\n properties:\n accessMode:\n default: ReadWrite\n description: Specifies the services that this member\n is capable of providing.\n enum:\n - None\n - Readonly\n - ReadWrite\n type: string\n name:\n default: leader\n description: Specifies the name of the consensus member.\n type: string\n replicas:\n default: 0\n description: |-\n Indicates the number of Pods that perform this role.\n The default is 1 for `Leader`, 0 for `Learner`, others for `Followers`.\n format: int32\n minimum: 0\n type: integer\n required:\n - accessMode\n - name\n type: object\n learner:\n description: Represents a member of the consensus set that\n does not have voting rights.\n properties:\n accessMode:\n default: ReadWrite\n description: Specifies the services that this member\n is capable of providing.\n enum:\n - None\n - Readonly\n - ReadWrite\n type: string\n name:\n default: leader\n description: Specifies the name of the consensus member.\n type: string\n replicas:\n default: 0\n description: |-\n Indicates the number of Pods that perform this role.\n The default is 1 for `Leader`, 0 for `Learner`, others for `Followers`.\n format: int32\n minimum: 0\n type: integer\n required:\n - accessMode\n - name\n type: object\n llPodManagementPolicy:\n description: |-\n Controls the creation of pods during initial scale up, replacement of pods on nodes, and scaling down.\n\n\n - `OrderedReady`: Creates pods in increasing order (pod-0, then pod-1, etc). The controller waits until each pod\n is ready before continuing. Pods are removed in reverse order when scaling down.\n - `Parallel`: Creates pods in parallel to match the desired scale without waiting. All pods are deleted at once\n when scaling down.\n type: string\n llUpdateStrategy:\n description: |-\n Specifies the low-level StatefulSetUpdateStrategy to be used when updating Pods in the StatefulSet upon a\n revision to the Template.\n `UpdateStrategy` will be ignored if this is provided.\n properties:\n rollingUpdate:\n description: RollingUpdate is used to communicate parameters\n when Type is RollingUpdateStatefulSetStrategyType.\n properties:\n maxUnavailable:\n anyOf:\n - type: integer\n - type: string\n description: |-\n The maximum number of pods that can be unavailable during the update.\n Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%).\n Absolute number is calculated from percentage by rounding up. This can not be 0.\n Defaults to 1. This field is alpha-level and is only honored by servers that enable the\n MaxUnavailableStatefulSet feature. The field applies to all pods in the range 0 to\n Replicas-1. That means if there is any unavailable pod in the range 0 to Replicas-1, it\n will be counted towards MaxUnavailable.\n x-kubernetes-int-or-string: true\n partition:\n description: |-\n Partition indicates the ordinal at which the StatefulSet should be partitioned\n for updates. During a rolling update, all pods from ordinal Replicas-1 to\n Partition are updated. All pods from ordinal Partition-1 to 0 remain untouched.\n This is helpful in being able to do a canary based deployment. The default value is 0.\n format: int32\n type: integer\n type: object\n type:\n description: |-\n Type indicates the type of the StatefulSetUpdateStrategy.\n Default is RollingUpdate.\n type: string\n type: object\n updateStrategy:\n default: Serial\n description: |-\n Specifies the strategy for updating Pods.\n For workloadType=`Consensus`, the update strategy can be one of the following:\n\n\n - `Serial`: Updates Members sequentially to minimize component downtime.\n - `BestEffortParallel`: Updates Members in parallel to minimize component write downtime. Majority remains online\n at all times.\n - `Parallel`: Forces parallel updates.\n enum:\n - Serial\n - BestEffortParallel\n - Parallel\n type: string\n required:\n - leader\n type: object\n customLabelSpecs:\n description: Used for custom label tags which you want to add\n to the component resources.\n items:\n description: CustomLabelSpec is deprecated since v0.8.\n properties:\n key:\n description: The key of the label.\n type: string\n resources:\n description: The resources that will be patched with the\n label.\n items:\n description: GVKResource is deprecated since v0.8.\n properties:\n gvk:\n description: |-\n Represents the GVK of a resource, such as \"v1/Pod\", \"apps/v1/StatefulSet\", etc.\n When a resource matching this is found by the selector, a custom label will be added if it doesn't already exist,\n or updated if it does.\n type: string\n selector:\n additionalProperties:\n type: string\n description: A label query used to filter a set\n of resources.\n type: object\n required:\n - gvk\n type: object\n type: array\n value:\n description: The value of the label.\n type: string\n required:\n - key\n - value\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - key\n x-kubernetes-list-type: map\n description:\n description: Description of the component definition.\n type: string\n exporter:\n description: Defines the metrics exporter.\n properties:\n containerName:\n description: Specifies the name of the built-in metrics\n exporter container.\n type: string\n scrapePath:\n description: |-\n Specifies the http/https url path to scrape for metrics.\n If empty, Prometheus uses the default value (e.g. `/metrics`).\n type: string\n scrapePort:\n description: Specifies the port name to scrape for metrics.\n type: string\n scrapeScheme:\n description: |-\n Specifies the schema to use for scraping.\n `http` and `https` are the expected values unless you rewrite the `__scheme__` label via relabeling.\n If empty, Prometheus uses the default value `http`.\n enum:\n - http\n - https\n type: string\n type: object\n horizontalScalePolicy:\n description: Defines the behavior of horizontal scale.\n properties:\n backupPolicyTemplateName:\n description: Refers to the backup policy template.\n type: string\n type:\n default: None\n description: |-\n Determines the data synchronization method when a component scales out.\n The policy can be one of the following: {None, CloneVolume}. The default policy is `None`.\n\n\n - `None`: This is the default policy. It creates an empty volume without data cloning.\n - `CloneVolume`: This policy clones data to newly scaled pods. It first tries to use a volume snapshot.\n If volume snapshot is not enabled, it will attempt to use a backup tool. If neither method works, it will report an error.\n - `Snapshot`: This policy is deprecated and is an alias for CloneVolume.\n enum:\n - None\n - CloneVolume\n - Snapshot\n type: string\n volumeMountsName:\n description: |-\n Specifies the volumeMount of the container to backup.\n This only works if Type is not None. If not specified, the first volumeMount will be selected.\n type: string\n type: object\n logConfigs:\n description: Specify the logging files which can be observed\n and configured by cluster users.\n items:\n properties:\n filePathPattern:\n description: |-\n Specifies the paths or patterns identifying where the log files are stored.\n This field allows the system to locate and manage log files effectively.\n\n\n Examples:\n\n\n - /home/postgres/pgdata/pgroot/data/log/postgresql-*\n - /data/mysql/log/mysqld-error.log\n maxLength: 4096\n type: string\n name:\n description: |-\n Specifies a descriptive label for the log type, such as 'slow' for a MySQL slow log file.\n It provides a clear identification of the log's purpose and content.\n maxLength: 128\n type: string\n required:\n - filePathPattern\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n monitor:\n description: |-\n Deprecated since v0.9\n monitor is monitoring config which provided by provider.\n properties:\n builtIn:\n default: false\n description: |-\n builtIn is a switch to enable KubeBlocks builtIn monitoring.\n If BuiltIn is set to true, monitor metrics will be scraped automatically.\n If BuiltIn is set to false, the provider should set ExporterConfig and Sidecar container own.\n type: boolean\n exporterConfig:\n description: |-\n exporterConfig provided by provider, which specify necessary information to Time Series Database.\n exporterConfig is valid when builtIn is false.\n properties:\n scrapePath:\n default: /metrics\n description: scrapePath is exporter url path for Time\n Series Database to scrape metrics.\n maxLength: 128\n type: string\n scrapePort:\n anyOf:\n - type: integer\n - type: string\n description: scrapePort is exporter port for Time Series\n Database to scrape metrics.\n x-kubernetes-int-or-string: true\n required:\n - scrapePort\n type: object\n type: object\n name:\n description: |-\n This name could be used as default name of `cluster.spec.componentSpecs.name`, and needs to conform with same\n validation rules as `cluster.spec.componentSpecs.name`, currently complying with IANA Service Naming rule.\n This name will apply to cluster objects as the value of label \"apps.kubeblocks.io/component-name\".\n maxLength: 22\n pattern: ^[a-z]([a-z0-9\\-]*[a-z0-9])?$\n type: string\n podSpec:\n description: Defines the pod spec template of component.\n properties:\n activeDeadlineSeconds:\n description: |-\n Optional duration in seconds the pod may be active on the node relative to\n StartTime before the system will actively try to mark it failed and kill associated containers.\n Value must be a positive integer.\n format: int64\n type: integer\n affinity:\n description: If specified, the pod's scheduling constraints\n properties:\n nodeAffinity:\n description: Describes node affinity scheduling rules\n for the pod.\n properties:\n preferredDuringSchedulingIgnoredDuringExecution:\n description: |-\n The scheduler will prefer to schedule pods to nodes that satisfy\n the affinity expressions specified by this field, but it may choose\n a node that violates one or more of the expressions. The node that is\n most preferred is the one with the greatest sum of weights, i.e.\n for each node that meets all of the scheduling requirements (resource\n request, requiredDuringScheduling affinity expressions, etc.),\n compute a sum by iterating through the elements of this field and adding\n \"weight\" to the sum if the node matches the corresponding matchExpressions; the\n node(s) with the highest sum are the most preferred.\n items:\n description: |-\n An empty preferred scheduling term matches all objects with implicit weight 0\n (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op).\n properties:\n preference:\n description: A node selector term, associated\n with the corresponding weight.\n properties:\n matchExpressions:\n description: A list of node selector requirements\n by node's labels.\n items:\n description: |-\n A node selector requirement is a selector that contains values, a key, and an operator\n that relates the key and values.\n properties:\n key:\n description: The label key that\n the selector applies to.\n type: string\n operator:\n description: |-\n Represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.\n type: string\n values:\n description: |-\n An array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. If the operator is Gt or Lt, the values\n array must have a single element, which will be interpreted as an integer.\n This array is replaced during a strategic merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchFields:\n description: A list of node selector requirements\n by node's fields.\n items:\n description: |-\n A node selector requirement is a selector that contains values, a key, and an operator\n that relates the key and values.\n properties:\n key:\n description: The label key that\n the selector applies to.\n type: string\n operator:\n description: |-\n Represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.\n type: string\n values:\n description: |-\n An array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. If the operator is Gt or Lt, the values\n array must have a single element, which will be interpreted as an integer.\n This array is replaced during a strategic merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n type: object\n x-kubernetes-map-type: atomic\n weight:\n description: Weight associated with matching\n the corresponding nodeSelectorTerm, in the\n range 1-100.\n format: int32\n type: integer\n required:\n - preference\n - weight\n type: object\n type: array\n requiredDuringSchedulingIgnoredDuringExecution:\n description: |-\n If the affinity requirements specified by this field are not met at\n scheduling time, the pod will not be scheduled onto the node.\n If the affinity requirements specified by this field cease to be met\n at some point during pod execution (e.g. due to an update), the system\n may or may not try to eventually evict the pod from its node.\n properties:\n nodeSelectorTerms:\n description: Required. A list of node selector\n terms. The terms are ORed.\n items:\n description: |-\n A null or empty node selector term matches no objects. The requirements of\n them are ANDed.\n The TopologySelectorTerm type implements a subset of the NodeSelectorTerm.\n properties:\n matchExpressions:\n description: A list of node selector requirements\n by node's labels.\n items:\n description: |-\n A node selector requirement is a selector that contains values, a key, and an operator\n that relates the key and values.\n properties:\n key:\n description: The label key that\n the selector applies to.\n type: string\n operator:\n description: |-\n Represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.\n type: string\n values:\n description: |-\n An array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. If the operator is Gt or Lt, the values\n array must have a single element, which will be interpreted as an integer.\n This array is replaced during a strategic merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchFields:\n description: A list of node selector requirements\n by node's fields.\n items:\n description: |-\n A node selector requirement is a selector that contains values, a key, and an operator\n that relates the key and values.\n properties:\n key:\n description: The label key that\n the selector applies to.\n type: string\n operator:\n description: |-\n Represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.\n type: string\n values:\n description: |-\n An array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. If the operator is Gt or Lt, the values\n array must have a single element, which will be interpreted as an integer.\n This array is replaced during a strategic merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n type: object\n x-kubernetes-map-type: atomic\n type: array\n required:\n - nodeSelectorTerms\n type: object\n x-kubernetes-map-type: atomic\n type: object\n podAffinity:\n description: Describes pod affinity scheduling rules\n (e.g. co-locate this pod in the same node, zone, etc.\n as some other pod(s)).\n properties:\n preferredDuringSchedulingIgnoredDuringExecution:\n description: |-\n The scheduler will prefer to schedule pods to nodes that satisfy\n the affinity expressions specified by this field, but it may choose\n a node that violates one or more of the expressions. The node that is\n most preferred is the one with the greatest sum of weights, i.e.\n for each node that meets all of the scheduling requirements (resource\n request, requiredDuringScheduling affinity expressions, etc.),\n compute a sum by iterating through the elements of this field and adding\n \"weight\" to the sum if the node has pods which matches the corresponding podAffinityTerm; the\n node(s) with the highest sum are the most preferred.\n items:\n description: The weights of all of the matched\n WeightedPodAffinityTerm fields are added per-node\n to find the most preferred node(s)\n properties:\n podAffinityTerm:\n description: Required. A pod affinity term,\n associated with the corresponding weight.\n properties:\n labelSelector:\n description: |-\n A label query over a set of resources, in this case pods.\n If it's null, this PodAffinityTerm matches with no Pods.\n properties:\n matchExpressions:\n description: matchExpressions is a\n list of label selector requirements.\n The requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label\n key that the selector applies\n to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n matchLabelKeys:\n description: |-\n MatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MatchLabelKeys and LabelSelector.\n Also, MatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n mismatchLabelKeys:\n description: |-\n MismatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector.\n Also, MismatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n namespaceSelector:\n description: |-\n A label query over the set of namespaces that the term applies to.\n The term is applied to the union of the namespaces selected by this field\n and the ones listed in the namespaces field.\n null selector and null or empty namespaces list means \"this pod's namespace\".\n An empty selector ({}) matches all namespaces.\n properties:\n matchExpressions:\n description: matchExpressions is a\n list of label selector requirements.\n The requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label\n key that the selector applies\n to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n namespaces:\n description: |-\n namespaces specifies a static list of namespace names that the term applies to.\n The term is applied to the union of the namespaces listed in this field\n and the ones selected by namespaceSelector.\n null or empty namespaces list and null namespaceSelector means \"this pod's namespace\".\n items:\n type: string\n type: array\n topologyKey:\n description: |-\n This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching\n the labelSelector in the specified namespaces, where co-located is defined as running on a node\n whose value of the label with key topologyKey matches that of any node on which any of the\n selected pods is running.\n Empty topologyKey is not allowed.\n type: string\n required:\n - topologyKey\n type: object\n weight:\n description: |-\n weight associated with matching the corresponding podAffinityTerm,\n in the range 1-100.\n format: int32\n type: integer\n required:\n - podAffinityTerm\n - weight\n type: object\n type: array\n requiredDuringSchedulingIgnoredDuringExecution:\n description: |-\n If the affinity requirements specified by this field are not met at\n scheduling time, the pod will not be scheduled onto the node.\n If the affinity requirements specified by this field cease to be met\n at some point during pod execution (e.g. due to a pod label update), the\n system may or may not try to eventually evict the pod from its node.\n When there are multiple elements, the lists of nodes corresponding to each\n podAffinityTerm are intersected, i.e. all terms must be satisfied.\n items:\n description: |-\n Defines a set of pods (namely those matching the labelSelector\n relative to the given namespace(s)) that this pod should be\n co-located (affinity) or not co-located (anti-affinity) with,\n where co-located is defined as running on a node whose value of\n the label with key matches that of any node on which\n a pod of the set of pods is running\n properties:\n labelSelector:\n description: |-\n A label query over a set of resources, in this case pods.\n If it's null, this PodAffinityTerm matches with no Pods.\n properties:\n matchExpressions:\n description: matchExpressions is a list\n of label selector requirements. The\n requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label key\n that the selector applies to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n matchLabelKeys:\n description: |-\n MatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MatchLabelKeys and LabelSelector.\n Also, MatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n mismatchLabelKeys:\n description: |-\n MismatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector.\n Also, MismatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n namespaceSelector:\n description: |-\n A label query over the set of namespaces that the term applies to.\n The term is applied to the union of the namespaces selected by this field\n and the ones listed in the namespaces field.\n null selector and null or empty namespaces list means \"this pod's namespace\".\n An empty selector ({}) matches all namespaces.\n properties:\n matchExpressions:\n description: matchExpressions is a list\n of label selector requirements. The\n requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label key\n that the selector applies to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n namespaces:\n description: |-\n namespaces specifies a static list of namespace names that the term applies to.\n The term is applied to the union of the namespaces listed in this field\n and the ones selected by namespaceSelector.\n null or empty namespaces list and null namespaceSelector means \"this pod's namespace\".\n items:\n type: string\n type: array\n topologyKey:\n description: |-\n This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching\n the labelSelector in the specified namespaces, where co-located is defined as running on a node\n whose value of the label with key topologyKey matches that of any node on which any of the\n selected pods is running.\n Empty topologyKey is not allowed.\n type: string\n required:\n - topologyKey\n type: object\n type: array\n type: object\n podAntiAffinity:\n description: Describes pod anti-affinity scheduling\n rules (e.g. avoid putting this pod in the same node,\n zone, etc. as some other pod(s)).\n properties:\n preferredDuringSchedulingIgnoredDuringExecution:\n description: |-\n The scheduler will prefer to schedule pods to nodes that satisfy\n the anti-affinity expressions specified by this field, but it may choose\n a node that violates one or more of the expressions. The node that is\n most preferred is the one with the greatest sum of weights, i.e.\n for each node that meets all of the scheduling requirements (resource\n request, requiredDuringScheduling anti-affinity expressions, etc.),\n compute a sum by iterating through the elements of this field and adding\n \"weight\" to the sum if the node has pods which matches the corresponding podAffinityTerm; the\n node(s) with the highest sum are the most preferred.\n items:\n description: The weights of all of the matched\n WeightedPodAffinityTerm fields are added per-node\n to find the most preferred node(s)\n properties:\n podAffinityTerm:\n description: Required. A pod affinity term,\n associated with the corresponding weight.\n properties:\n labelSelector:\n description: |-\n A label query over a set of resources, in this case pods.\n If it's null, this PodAffinityTerm matches with no Pods.\n properties:\n matchExpressions:\n description: matchExpressions is a\n list of label selector requirements.\n The requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label\n key that the selector applies\n to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n matchLabelKeys:\n description: |-\n MatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MatchLabelKeys and LabelSelector.\n Also, MatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n mismatchLabelKeys:\n description: |-\n MismatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector.\n Also, MismatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n namespaceSelector:\n description: |-\n A label query over the set of namespaces that the term applies to.\n The term is applied to the union of the namespaces selected by this field\n and the ones listed in the namespaces field.\n null selector and null or empty namespaces list means \"this pod's namespace\".\n An empty selector ({}) matches all namespaces.\n properties:\n matchExpressions:\n description: matchExpressions is a\n list of label selector requirements.\n The requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label\n key that the selector applies\n to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n namespaces:\n description: |-\n namespaces specifies a static list of namespace names that the term applies to.\n The term is applied to the union of the namespaces listed in this field\n and the ones selected by namespaceSelector.\n null or empty namespaces list and null namespaceSelector means \"this pod's namespace\".\n items:\n type: string\n type: array\n topologyKey:\n description: |-\n This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching\n the labelSelector in the specified namespaces, where co-located is defined as running on a node\n whose value of the label with key topologyKey matches that of any node on which any of the\n selected pods is running.\n Empty topologyKey is not allowed.\n type: string\n required:\n - topologyKey\n type: object\n weight:\n description: |-\n weight associated with matching the corresponding podAffinityTerm,\n in the range 1-100.\n format: int32\n type: integer\n required:\n - podAffinityTerm\n - weight\n type: object\n type: array\n requiredDuringSchedulingIgnoredDuringExecution:\n description: |-\n If the anti-affinity requirements specified by this field are not met at\n scheduling time, the pod will not be scheduled onto the node.\n If the anti-affinity requirements specified by this field cease to be met\n at some point during pod execution (e.g. due to a pod label update), the\n system may or may not try to eventually evict the pod from its node.\n When there are multiple elements, the lists of nodes corresponding to each\n podAffinityTerm are intersected, i.e. all terms must be satisfied.\n items:\n description: |-\n Defines a set of pods (namely those matching the labelSelector\n relative to the given namespace(s)) that this pod should be\n co-located (affinity) or not co-located (anti-affinity) with,\n where co-located is defined as running on a node whose value of\n the label with key matches that of any node on which\n a pod of the set of pods is running\n properties:\n labelSelector:\n description: |-\n A label query over a set of resources, in this case pods.\n If it's null, this PodAffinityTerm matches with no Pods.\n properties:\n matchExpressions:\n description: matchExpressions is a list\n of label selector requirements. The\n requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label key\n that the selector applies to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n matchLabelKeys:\n description: |-\n MatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key in (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MatchLabelKeys and LabelSelector.\n Also, MatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n mismatchLabelKeys:\n description: |-\n MismatchLabelKeys is a set of pod label keys to select which pods will\n be taken into consideration. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are merged with `LabelSelector` as `key notin (value)`\n to select the group of existing pods which pods will be taken into consideration\n for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming\n pod labels will be ignored. The default value is empty.\n The same key is forbidden to exist in both MismatchLabelKeys and LabelSelector.\n Also, MismatchLabelKeys cannot be set when LabelSelector isn't set.\n This is an alpha field and requires enabling MatchLabelKeysInPodAffinity feature gate.\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n namespaceSelector:\n description: |-\n A label query over the set of namespaces that the term applies to.\n The term is applied to the union of the namespaces selected by this field\n and the ones listed in the namespaces field.\n null selector and null or empty namespaces list means \"this pod's namespace\".\n An empty selector ({}) matches all namespaces.\n properties:\n matchExpressions:\n description: matchExpressions is a list\n of label selector requirements. The\n requirements are ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label key\n that the selector applies to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n namespaces:\n description: |-\n namespaces specifies a static list of namespace names that the term applies to.\n The term is applied to the union of the namespaces listed in this field\n and the ones selected by namespaceSelector.\n null or empty namespaces list and null namespaceSelector means \"this pod's namespace\".\n items:\n type: string\n type: array\n topologyKey:\n description: |-\n This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching\n the labelSelector in the specified namespaces, where co-located is defined as running on a node\n whose value of the label with key topologyKey matches that of any node on which any of the\n selected pods is running.\n Empty topologyKey is not allowed.\n type: string\n required:\n - topologyKey\n type: object\n type: array\n type: object\n type: object\n automountServiceAccountToken:\n description: AutomountServiceAccountToken indicates whether\n a service account token should be automatically mounted.\n type: boolean\n containers:\n description: |-\n List of containers belonging to the pod.\n Containers cannot currently be added or removed.\n There must be at least one container in a Pod.\n Cannot be updated.\n items:\n description: A single application container that you want\n to run within a pod.\n properties:\n args:\n description: |-\n Arguments to the entrypoint.\n The container image's CMD is used if this is not provided.\n Variable references $(VAR_NAME) are expanded using the container's environment. If a variable\n cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will\n produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless\n of whether the variable exists or not. Cannot be updated.\n More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell\n items:\n type: string\n type: array\n command:\n description: |-\n Entrypoint array. Not executed within a shell.\n The container image's ENTRYPOINT is used if this is not provided.\n Variable references $(VAR_NAME) are expanded using the container's environment. If a variable\n cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will\n produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless\n of whether the variable exists or not. Cannot be updated.\n More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell\n items:\n type: string\n type: array\n env:\n description: |-\n List of environment variables to set in the container.\n Cannot be updated.\n items:\n description: EnvVar represents an environment variable\n present in a Container.\n properties:\n name:\n description: Name of the environment variable.\n Must be a C_IDENTIFIER.\n type: string\n value:\n description: |-\n Variable references $(VAR_NAME) are expanded\n using the previously defined environment variables in the container and\n any service environment variables. If a variable cannot be resolved,\n the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e.\n \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\".\n Escaped references will never be expanded, regardless of whether the variable\n exists or not.\n Defaults to \"\".\n type: string\n valueFrom:\n description: Source for the environment variable's\n value. Cannot be used if value is not empty.\n properties:\n configMapKeyRef:\n description: Selects a key of a ConfigMap.\n properties:\n key:\n description: The key to select.\n type: string\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the ConfigMap\n or its key must be defined\n type: boolean\n required:\n - key\n type: object\n x-kubernetes-map-type: atomic\n fieldRef:\n description: |-\n Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`,\n spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.\n properties:\n apiVersion:\n description: Version of the schema the\n FieldPath is written in terms of,\n defaults to \"v1\".\n type: string\n fieldPath:\n description: Path of the field to select\n in the specified API version.\n type: string\n required:\n - fieldPath\n type: object\n x-kubernetes-map-type: atomic\n resourceFieldRef:\n description: |-\n Selects a resource of the container: only resources limits and requests\n (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.\n properties:\n containerName:\n description: 'Container name: required\n for volumes, optional for env vars'\n type: string\n divisor:\n anyOf:\n - type: integer\n - type: string\n description: Specifies the output format\n of the exposed resources, defaults\n to \"1\"\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n resource:\n description: 'Required: resource to\n select'\n type: string\n required:\n - resource\n type: object\n x-kubernetes-map-type: atomic\n secretKeyRef:\n description: Selects a key of a secret in\n the pod's namespace\n properties:\n key:\n description: The key of the secret to\n select from. Must be a valid secret\n key.\n type: string\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the Secret\n or its key must be defined\n type: boolean\n required:\n - key\n type: object\n x-kubernetes-map-type: atomic\n type: object\n required:\n - name\n type: object\n type: array\n envFrom:\n description: |-\n List of sources to populate environment variables in the container.\n The keys defined within a source must be a C_IDENTIFIER. All invalid keys\n will be reported as an event when the container is starting. When a key exists in multiple\n sources, the value associated with the last source will take precedence.\n Values defined by an Env with a duplicate key will take precedence.\n Cannot be updated.\n items:\n description: EnvFromSource represents the source\n of a set of ConfigMaps\n properties:\n configMapRef:\n description: The ConfigMap to select from\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the ConfigMap\n must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n prefix:\n description: An optional identifier to prepend\n to each key in the ConfigMap. Must be a C_IDENTIFIER.\n type: string\n secretRef:\n description: The Secret to select from\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the Secret\n must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n type: object\n type: array\n image:\n description: |-\n Container image name.\n More info: https://kubernetes.io/docs/concepts/containers/images\n This field is optional to allow higher level config management to default or override\n container images in workload controllers like Deployments and StatefulSets.\n type: string\n imagePullPolicy:\n description: |-\n Image pull policy.\n One of Always, Never, IfNotPresent.\n Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n type: string\n lifecycle:\n description: |-\n Actions that the management system should take in response to container lifecycle events.\n Cannot be updated.\n properties:\n postStart:\n description: |-\n PostStart is called immediately after a container is created. If the handler fails,\n the container is terminated and restarted according to its restart policy.\n Other management of the container blocks until the hook completes.\n More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks\n properties:\n exec:\n description: Exec specifies the action to\n take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in\n the request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a\n custom header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP\n server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n sleep:\n description: Sleep represents the duration\n that the container should sleep before being\n terminated.\n properties:\n seconds:\n description: Seconds is the number of\n seconds to sleep.\n format: int64\n type: integer\n required:\n - seconds\n type: object\n tcpSocket:\n description: |-\n Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept\n for the backward compatibility. There are no validation of this field and\n lifecycle hooks will fail in runtime when tcp handler is specified.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n type: object\n preStop:\n description: |-\n PreStop is called immediately before a container is terminated due to an\n API request or management event such as liveness/startup probe failure,\n preemption, resource contention, etc. The handler is not called if the\n container crashes or exits. The Pod's termination grace period countdown begins before the\n PreStop hook is executed. Regardless of the outcome of the handler, the\n container will eventually terminate within the Pod's termination grace\n period (unless delayed by finalizers). Other management of the container blocks until the hook completes\n or until the termination grace period is reached.\n More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks\n properties:\n exec:\n description: Exec specifies the action to\n take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in\n the request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a\n custom header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP\n server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n sleep:\n description: Sleep represents the duration\n that the container should sleep before being\n terminated.\n properties:\n seconds:\n description: Seconds is the number of\n seconds to sleep.\n format: int64\n type: integer\n required:\n - seconds\n type: object\n tcpSocket:\n description: |-\n Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept\n for the backward compatibility. There are no validation of this field and\n lifecycle hooks will fail in runtime when tcp handler is specified.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n type: object\n type: object\n livenessProbe:\n description: |-\n Periodic probe of container liveness.\n Container will be restarted if the probe fails.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n name:\n description: |-\n Name of the container specified as a DNS_LABEL.\n Each container in a pod must have a unique name (DNS_LABEL).\n Cannot be updated.\n type: string\n ports:\n description: |-\n List of ports to expose from the container. Not specifying a port here\n DOES NOT prevent that port from being exposed. Any port which is\n listening on the default \"0.0.0.0\" address inside a container will be\n accessible from the network.\n Modifying this array with strategic merge patch may corrupt the data.\n For more information See https://github.com/kubernetes/kubernetes/issues/108255.\n Cannot be updated.\n items:\n description: ContainerPort represents a network\n port in a single container.\n properties:\n containerPort:\n description: |-\n Number of port to expose on the pod's IP address.\n This must be a valid port number, 0 < x < 65536.\n format: int32\n type: integer\n hostIP:\n description: What host IP to bind the external\n port to.\n type: string\n hostPort:\n description: |-\n Number of port to expose on the host.\n If specified, this must be a valid port number, 0 < x < 65536.\n If HostNetwork is specified, this must match ContainerPort.\n Most containers do not need this.\n format: int32\n type: integer\n name:\n description: |-\n If specified, this must be an IANA_SVC_NAME and unique within the pod. Each\n named port in a pod must have a unique name. Name for the port that can be\n referred to by services.\n type: string\n protocol:\n default: TCP\n description: |-\n Protocol for port. Must be UDP, TCP, or SCTP.\n Defaults to \"TCP\".\n type: string\n required:\n - containerPort\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - containerPort\n - protocol\n x-kubernetes-list-type: map\n readinessProbe:\n description: |-\n Periodic probe of container service readiness.\n Container will be removed from service endpoints if the probe fails.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n resizePolicy:\n description: Resources resize policy for the container.\n items:\n description: ContainerResizePolicy represents resource\n resize policy for the container.\n properties:\n resourceName:\n description: |-\n Name of the resource to which this resource resize policy applies.\n Supported values: cpu, memory.\n type: string\n restartPolicy:\n description: |-\n Restart policy to apply when specified resource is resized.\n If not specified, it defaults to NotRequired.\n type: string\n required:\n - resourceName\n - restartPolicy\n type: object\n type: array\n x-kubernetes-list-type: atomic\n resources:\n description: |-\n Compute Resources required by this container.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n properties:\n claims:\n description: |-\n Claims lists the names of resources, defined in spec.resourceClaims,\n that are used by this container.\n\n\n This is an alpha field and requires enabling the\n DynamicResourceAllocation feature gate.\n\n\n This field is immutable. It can only be set for containers.\n items:\n description: ResourceClaim references one entry\n in PodSpec.ResourceClaims.\n properties:\n name:\n description: |-\n Name must match the name of one entry in pod.spec.resourceClaims of\n the Pod where this field is used. It makes that resource available\n inside a container.\n type: string\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n limits:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Limits describes the maximum amount of compute resources allowed.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n type: object\n requests:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Requests describes the minimum amount of compute resources required.\n If Requests is omitted for a container, it defaults to Limits if that is explicitly specified,\n otherwise to an implementation-defined value. Requests cannot exceed Limits.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n type: object\n type: object\n restartPolicy:\n description: |-\n RestartPolicy defines the restart behavior of individual containers in a pod.\n This field may only be set for init containers, and the only allowed value is \"Always\".\n For non-init containers or when this field is not specified,\n the restart behavior is defined by the Pod's restart policy and the container type.\n Setting the RestartPolicy as \"Always\" for the init container will have the following effect:\n this init container will be continually restarted on\n exit until all regular containers have terminated. Once all regular\n containers have completed, all init containers with restartPolicy \"Always\"\n will be shut down. This lifecycle differs from normal init containers and\n is often referred to as a \"sidecar\" container. Although this init\n container still starts in the init container sequence, it does not wait\n for the container to complete before proceeding to the next init\n container. Instead, the next init container starts immediately after this\n init container is started, or after any startupProbe has successfully\n completed.\n type: string\n securityContext:\n description: |-\n SecurityContext defines the security options the container should be run with.\n If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.\n More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/\n properties:\n allowPrivilegeEscalation:\n description: |-\n AllowPrivilegeEscalation controls whether a process can gain more\n privileges than its parent process. This bool directly controls if\n the no_new_privs flag will be set on the container process.\n AllowPrivilegeEscalation is true always when the container is:\n 1) run as Privileged\n 2) has CAP_SYS_ADMIN\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n capabilities:\n description: |-\n The capabilities to add/drop when running containers.\n Defaults to the default set of capabilities granted by the container runtime.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n add:\n description: Added capabilities\n items:\n description: Capability represent POSIX\n capabilities type\n type: string\n type: array\n drop:\n description: Removed capabilities\n items:\n description: Capability represent POSIX\n capabilities type\n type: string\n type: array\n type: object\n privileged:\n description: |-\n Run container in privileged mode.\n Processes in privileged containers are essentially equivalent to root on the host.\n Defaults to false.\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n procMount:\n description: |-\n procMount denotes the type of proc mount to use for the containers.\n The default is DefaultProcMount which uses the container runtime defaults for\n readonly paths and masked paths.\n This requires the ProcMountType feature flag to be enabled.\n Note that this field cannot be set when spec.os.name is windows.\n type: string\n readOnlyRootFilesystem:\n description: |-\n Whether this container has a read-only root filesystem.\n Default is false.\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n runAsGroup:\n description: |-\n The GID to run the entrypoint of the container process.\n Uses runtime default if unset.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n runAsNonRoot:\n description: |-\n Indicates that the container must run as a non-root user.\n If true, the Kubelet will validate the image at runtime to ensure that it\n does not run as UID 0 (root) and fail to start the container if it does.\n If unset or false, no such validation will be performed.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: boolean\n runAsUser:\n description: |-\n The UID to run the entrypoint of the container process.\n Defaults to user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n seLinuxOptions:\n description: |-\n The SELinux context to be applied to the container.\n If unspecified, the container runtime will allocate a random SELinux context for each\n container. May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n level:\n description: Level is SELinux level label\n that applies to the container.\n type: string\n role:\n description: Role is a SELinux role label\n that applies to the container.\n type: string\n type:\n description: Type is a SELinux type label\n that applies to the container.\n type: string\n user:\n description: User is a SELinux user label\n that applies to the container.\n type: string\n type: object\n seccompProfile:\n description: |-\n The seccomp options to use by this container. If seccomp options are\n provided at both the pod & container level, the container options\n override the pod options.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n localhostProfile:\n description: |-\n localhostProfile indicates a profile defined in a file on the node should be used.\n The profile must be preconfigured on the node to work.\n Must be a descending path, relative to the kubelet's configured seccomp profile location.\n Must be set if type is \"Localhost\". Must NOT be set for any other type.\n type: string\n type:\n description: |-\n type indicates which kind of seccomp profile will be applied.\n Valid options are:\n\n\n Localhost - a profile defined in a file on the node should be used.\n RuntimeDefault - the container runtime default profile should be used.\n Unconfined - no profile should be applied.\n type: string\n required:\n - type\n type: object\n windowsOptions:\n description: |-\n The Windows specific settings applied to all containers.\n If unspecified, the options from the PodSecurityContext will be used.\n If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is linux.\n properties:\n gmsaCredentialSpec:\n description: |-\n GMSACredentialSpec is where the GMSA admission webhook\n (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the\n GMSA credential spec named by the GMSACredentialSpecName field.\n type: string\n gmsaCredentialSpecName:\n description: GMSACredentialSpecName is the\n name of the GMSA credential spec to use.\n type: string\n hostProcess:\n description: |-\n HostProcess determines if a container should be run as a 'Host Process' container.\n All of a Pod's containers must have the same effective HostProcess value\n (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers).\n In addition, if HostProcess is true then HostNetwork must also be set to true.\n type: boolean\n runAsUserName:\n description: |-\n The UserName in Windows to run the entrypoint of the container process.\n Defaults to the user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: string\n type: object\n type: object\n startupProbe:\n description: |-\n StartupProbe indicates that the Pod has successfully initialized.\n If specified, no other probes are executed until this completes successfully.\n If this probe fails, the Pod will be restarted, just as if the livenessProbe failed.\n This can be used to provide different probe parameters at the beginning of a Pod's lifecycle,\n when it might take a long time to load data or warm a cache, than during steady-state operation.\n This cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n stdin:\n description: |-\n Whether this container should allocate a buffer for stdin in the container runtime. If this\n is not set, reads from stdin in the container will always result in EOF.\n Default is false.\n type: boolean\n stdinOnce:\n description: |-\n Whether the container runtime should close the stdin channel after it has been opened by\n a single attach. When stdin is true the stdin stream will remain open across multiple attach\n sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the\n first client attaches to stdin, and then remains open and accepts data until the client disconnects,\n at which time stdin is closed and remains closed until the container is restarted. If this\n flag is false, a container processes that reads from stdin will never receive an EOF.\n Default is false\n type: boolean\n terminationMessagePath:\n description: |-\n Optional: Path at which the file to which the container's termination message\n will be written is mounted into the container's filesystem.\n Message written is intended to be brief final status, such as an assertion failure message.\n Will be truncated by the node if greater than 4096 bytes. The total message length across\n all containers will be limited to 12kb.\n Defaults to /dev/termination-log.\n Cannot be updated.\n type: string\n terminationMessagePolicy:\n description: |-\n Indicate how the termination message should be populated. File will use the contents of\n terminationMessagePath to populate the container status message on both success and failure.\n FallbackToLogsOnError will use the last chunk of container log output if the termination\n message file is empty and the container exited with an error.\n The log output is limited to 2048 bytes or 80 lines, whichever is smaller.\n Defaults to File.\n Cannot be updated.\n type: string\n tty:\n description: |-\n Whether this container should allocate a TTY for itself, also requires 'stdin' to be true.\n Default is false.\n type: boolean\n volumeDevices:\n description: volumeDevices is the list of block devices\n to be used by the container.\n items:\n description: volumeDevice describes a mapping of\n a raw block device within a container.\n properties:\n devicePath:\n description: devicePath is the path inside of\n the container that the device will be mapped\n to.\n type: string\n name:\n description: name must match the name of a persistentVolumeClaim\n in the pod\n type: string\n required:\n - devicePath\n - name\n type: object\n type: array\n volumeMounts:\n description: |-\n Pod volumes to mount into the container's filesystem.\n Cannot be updated.\n items:\n description: VolumeMount describes a mounting of\n a Volume within a container.\n properties:\n mountPath:\n description: |-\n Path within the container at which the volume should be mounted. Must\n not contain ':'.\n type: string\n mountPropagation:\n description: |-\n mountPropagation determines how mounts are propagated from the host\n to container and the other way around.\n When not set, MountPropagationNone is used.\n This field is beta in 1.10.\n type: string\n name:\n description: This must match the Name of a Volume.\n type: string\n readOnly:\n description: |-\n Mounted read-only if true, read-write otherwise (false or unspecified).\n Defaults to false.\n type: boolean\n subPath:\n description: |-\n Path within the volume from which the container's volume should be mounted.\n Defaults to \"\" (volume's root).\n type: string\n subPathExpr:\n description: |-\n Expanded path within the volume from which the container's volume should be mounted.\n Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment.\n Defaults to \"\" (volume's root).\n SubPathExpr and SubPath are mutually exclusive.\n type: string\n required:\n - mountPath\n - name\n type: object\n type: array\n workingDir:\n description: |-\n Container's working directory.\n If not specified, the container runtime's default will be used, which\n might be configured in the container image.\n Cannot be updated.\n type: string\n required:\n - name\n type: object\n type: array\n dnsConfig:\n description: |-\n Specifies the DNS parameters of a pod.\n Parameters specified here will be merged to the generated DNS\n configuration based on DNSPolicy.\n properties:\n nameservers:\n description: |-\n A list of DNS name server IP addresses.\n This will be appended to the base nameservers generated from DNSPolicy.\n Duplicated nameservers will be removed.\n items:\n type: string\n type: array\n options:\n description: |-\n A list of DNS resolver options.\n This will be merged with the base options generated from DNSPolicy.\n Duplicated entries will be removed. Resolution options given in Options\n will override those that appear in the base DNSPolicy.\n items:\n description: PodDNSConfigOption defines DNS resolver\n options of a pod.\n properties:\n name:\n description: Required.\n type: string\n value:\n type: string\n type: object\n type: array\n searches:\n description: |-\n A list of DNS search domains for host-name lookup.\n This will be appended to the base search paths generated from DNSPolicy.\n Duplicated search paths will be removed.\n items:\n type: string\n type: array\n type: object\n dnsPolicy:\n description: |-\n Set DNS policy for the pod.\n Defaults to \"ClusterFirst\".\n Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.\n DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.\n To have DNS options set along with hostNetwork, you have to specify DNS policy\n explicitly to 'ClusterFirstWithHostNet'.\n type: string\n enableServiceLinks:\n description: |-\n EnableServiceLinks indicates whether information about services should be injected into pod's\n environment variables, matching the syntax of Docker links.\n Optional: Defaults to true.\n type: boolean\n ephemeralContainers:\n description: |-\n List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing\n pod to perform user-initiated actions such as debugging. This list cannot be specified when\n creating a pod, and it cannot be modified by updating the pod spec. In order to add an\n ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource.\n items:\n description: |-\n An EphemeralContainer is a temporary container that you may add to an existing Pod for\n user-initiated activities such as debugging. Ephemeral containers have no resource or\n scheduling guarantees, and they will not be restarted when they exit or when a Pod is\n removed or restarted. The kubelet may evict a Pod if an ephemeral container causes the\n Pod to exceed its resource allocation.\n\n\n To add an ephemeral container, use the ephemeralcontainers subresource of an existing\n Pod. Ephemeral containers may not be removed or restarted.\n properties:\n args:\n description: |-\n Arguments to the entrypoint.\n The image's CMD is used if this is not provided.\n Variable references $(VAR_NAME) are expanded using the container's environment. If a variable\n cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will\n produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless\n of whether the variable exists or not. Cannot be updated.\n More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell\n items:\n type: string\n type: array\n command:\n description: |-\n Entrypoint array. Not executed within a shell.\n The image's ENTRYPOINT is used if this is not provided.\n Variable references $(VAR_NAME) are expanded using the container's environment. If a variable\n cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will\n produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless\n of whether the variable exists or not. Cannot be updated.\n More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell\n items:\n type: string\n type: array\n env:\n description: |-\n List of environment variables to set in the container.\n Cannot be updated.\n items:\n description: EnvVar represents an environment variable\n present in a Container.\n properties:\n name:\n description: Name of the environment variable.\n Must be a C_IDENTIFIER.\n type: string\n value:\n description: |-\n Variable references $(VAR_NAME) are expanded\n using the previously defined environment variables in the container and\n any service environment variables. If a variable cannot be resolved,\n the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e.\n \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\".\n Escaped references will never be expanded, regardless of whether the variable\n exists or not.\n Defaults to \"\".\n type: string\n valueFrom:\n description: Source for the environment variable's\n value. Cannot be used if value is not empty.\n properties:\n configMapKeyRef:\n description: Selects a key of a ConfigMap.\n properties:\n key:\n description: The key to select.\n type: string\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the ConfigMap\n or its key must be defined\n type: boolean\n required:\n - key\n type: object\n x-kubernetes-map-type: atomic\n fieldRef:\n description: |-\n Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`,\n spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.\n properties:\n apiVersion:\n description: Version of the schema the\n FieldPath is written in terms of,\n defaults to \"v1\".\n type: string\n fieldPath:\n description: Path of the field to select\n in the specified API version.\n type: string\n required:\n - fieldPath\n type: object\n x-kubernetes-map-type: atomic\n resourceFieldRef:\n description: |-\n Selects a resource of the container: only resources limits and requests\n (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.\n properties:\n containerName:\n description: 'Container name: required\n for volumes, optional for env vars'\n type: string\n divisor:\n anyOf:\n - type: integer\n - type: string\n description: Specifies the output format\n of the exposed resources, defaults\n to \"1\"\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n resource:\n description: 'Required: resource to\n select'\n type: string\n required:\n - resource\n type: object\n x-kubernetes-map-type: atomic\n secretKeyRef:\n description: Selects a key of a secret in\n the pod's namespace\n properties:\n key:\n description: The key of the secret to\n select from. Must be a valid secret\n key.\n type: string\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the Secret\n or its key must be defined\n type: boolean\n required:\n - key\n type: object\n x-kubernetes-map-type: atomic\n type: object\n required:\n - name\n type: object\n type: array\n envFrom:\n description: |-\n List of sources to populate environment variables in the container.\n The keys defined within a source must be a C_IDENTIFIER. All invalid keys\n will be reported as an event when the container is starting. When a key exists in multiple\n sources, the value associated with the last source will take precedence.\n Values defined by an Env with a duplicate key will take precedence.\n Cannot be updated.\n items:\n description: EnvFromSource represents the source\n of a set of ConfigMaps\n properties:\n configMapRef:\n description: The ConfigMap to select from\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the ConfigMap\n must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n prefix:\n description: An optional identifier to prepend\n to each key in the ConfigMap. Must be a C_IDENTIFIER.\n type: string\n secretRef:\n description: The Secret to select from\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the Secret\n must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n type: object\n type: array\n image:\n description: |-\n Container image name.\n More info: https://kubernetes.io/docs/concepts/containers/images\n type: string\n imagePullPolicy:\n description: |-\n Image pull policy.\n One of Always, Never, IfNotPresent.\n Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n type: string\n lifecycle:\n description: Lifecycle is not allowed for ephemeral\n containers.\n properties:\n postStart:\n description: |-\n PostStart is called immediately after a container is created. If the handler fails,\n the container is terminated and restarted according to its restart policy.\n Other management of the container blocks until the hook completes.\n More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks\n properties:\n exec:\n description: Exec specifies the action to\n take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in\n the request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a\n custom header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP\n server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n sleep:\n description: Sleep represents the duration\n that the container should sleep before being\n terminated.\n properties:\n seconds:\n description: Seconds is the number of\n seconds to sleep.\n format: int64\n type: integer\n required:\n - seconds\n type: object\n tcpSocket:\n description: |-\n Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept\n for the backward compatibility. There are no validation of this field and\n lifecycle hooks will fail in runtime when tcp handler is specified.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n type: object\n preStop:\n description: |-\n PreStop is called immediately before a container is terminated due to an\n API request or management event such as liveness/startup probe failure,\n preemption, resource contention, etc. The handler is not called if the\n container crashes or exits. The Pod's termination grace period countdown begins before the\n PreStop hook is executed. Regardless of the outcome of the handler, the\n container will eventually terminate within the Pod's termination grace\n period (unless delayed by finalizers). Other management of the container blocks until the hook completes\n or until the termination grace period is reached.\n More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks\n properties:\n exec:\n description: Exec specifies the action to\n take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in\n the request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a\n custom header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP\n server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n sleep:\n description: Sleep represents the duration\n that the container should sleep before being\n terminated.\n properties:\n seconds:\n description: Seconds is the number of\n seconds to sleep.\n format: int64\n type: integer\n required:\n - seconds\n type: object\n tcpSocket:\n description: |-\n Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept\n for the backward compatibility. There are no validation of this field and\n lifecycle hooks will fail in runtime when tcp handler is specified.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n type: object\n type: object\n livenessProbe:\n description: Probes are not allowed for ephemeral\n containers.\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n name:\n description: |-\n Name of the ephemeral container specified as a DNS_LABEL.\n This name must be unique among all containers, init containers and ephemeral containers.\n type: string\n ports:\n description: Ports are not allowed for ephemeral containers.\n items:\n description: ContainerPort represents a network\n port in a single container.\n properties:\n containerPort:\n description: |-\n Number of port to expose on the pod's IP address.\n This must be a valid port number, 0 < x < 65536.\n format: int32\n type: integer\n hostIP:\n description: What host IP to bind the external\n port to.\n type: string\n hostPort:\n description: |-\n Number of port to expose on the host.\n If specified, this must be a valid port number, 0 < x < 65536.\n If HostNetwork is specified, this must match ContainerPort.\n Most containers do not need this.\n format: int32\n type: integer\n name:\n description: |-\n If specified, this must be an IANA_SVC_NAME and unique within the pod. Each\n named port in a pod must have a unique name. Name for the port that can be\n referred to by services.\n type: string\n protocol:\n default: TCP\n description: |-\n Protocol for port. Must be UDP, TCP, or SCTP.\n Defaults to \"TCP\".\n type: string\n required:\n - containerPort\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - containerPort\n - protocol\n x-kubernetes-list-type: map\n readinessProbe:\n description: Probes are not allowed for ephemeral\n containers.\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n resizePolicy:\n description: Resources resize policy for the container.\n items:\n description: ContainerResizePolicy represents resource\n resize policy for the container.\n properties:\n resourceName:\n description: |-\n Name of the resource to which this resource resize policy applies.\n Supported values: cpu, memory.\n type: string\n restartPolicy:\n description: |-\n Restart policy to apply when specified resource is resized.\n If not specified, it defaults to NotRequired.\n type: string\n required:\n - resourceName\n - restartPolicy\n type: object\n type: array\n x-kubernetes-list-type: atomic\n resources:\n description: |-\n Resources are not allowed for ephemeral containers. Ephemeral containers use spare resources\n already allocated to the pod.\n properties:\n claims:\n description: |-\n Claims lists the names of resources, defined in spec.resourceClaims,\n that are used by this container.\n\n\n This is an alpha field and requires enabling the\n DynamicResourceAllocation feature gate.\n\n\n This field is immutable. It can only be set for containers.\n items:\n description: ResourceClaim references one entry\n in PodSpec.ResourceClaims.\n properties:\n name:\n description: |-\n Name must match the name of one entry in pod.spec.resourceClaims of\n the Pod where this field is used. It makes that resource available\n inside a container.\n type: string\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n limits:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Limits describes the maximum amount of compute resources allowed.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n type: object\n requests:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Requests describes the minimum amount of compute resources required.\n If Requests is omitted for a container, it defaults to Limits if that is explicitly specified,\n otherwise to an implementation-defined value. Requests cannot exceed Limits.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n type: object\n type: object\n restartPolicy:\n description: |-\n Restart policy for the container to manage the restart behavior of each\n container within a pod.\n This may only be set for init containers. You cannot set this field on\n ephemeral containers.\n type: string\n securityContext:\n description: |-\n Optional: SecurityContext defines the security options the ephemeral container should be run with.\n If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.\n properties:\n allowPrivilegeEscalation:\n description: |-\n AllowPrivilegeEscalation controls whether a process can gain more\n privileges than its parent process. This bool directly controls if\n the no_new_privs flag will be set on the container process.\n AllowPrivilegeEscalation is true always when the container is:\n 1) run as Privileged\n 2) has CAP_SYS_ADMIN\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n capabilities:\n description: |-\n The capabilities to add/drop when running containers.\n Defaults to the default set of capabilities granted by the container runtime.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n add:\n description: Added capabilities\n items:\n description: Capability represent POSIX\n capabilities type\n type: string\n type: array\n drop:\n description: Removed capabilities\n items:\n description: Capability represent POSIX\n capabilities type\n type: string\n type: array\n type: object\n privileged:\n description: |-\n Run container in privileged mode.\n Processes in privileged containers are essentially equivalent to root on the host.\n Defaults to false.\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n procMount:\n description: |-\n procMount denotes the type of proc mount to use for the containers.\n The default is DefaultProcMount which uses the container runtime defaults for\n readonly paths and masked paths.\n This requires the ProcMountType feature flag to be enabled.\n Note that this field cannot be set when spec.os.name is windows.\n type: string\n readOnlyRootFilesystem:\n description: |-\n Whether this container has a read-only root filesystem.\n Default is false.\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n runAsGroup:\n description: |-\n The GID to run the entrypoint of the container process.\n Uses runtime default if unset.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n runAsNonRoot:\n description: |-\n Indicates that the container must run as a non-root user.\n If true, the Kubelet will validate the image at runtime to ensure that it\n does not run as UID 0 (root) and fail to start the container if it does.\n If unset or false, no such validation will be performed.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: boolean\n runAsUser:\n description: |-\n The UID to run the entrypoint of the container process.\n Defaults to user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n seLinuxOptions:\n description: |-\n The SELinux context to be applied to the container.\n If unspecified, the container runtime will allocate a random SELinux context for each\n container. May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n level:\n description: Level is SELinux level label\n that applies to the container.\n type: string\n role:\n description: Role is a SELinux role label\n that applies to the container.\n type: string\n type:\n description: Type is a SELinux type label\n that applies to the container.\n type: string\n user:\n description: User is a SELinux user label\n that applies to the container.\n type: string\n type: object\n seccompProfile:\n description: |-\n The seccomp options to use by this container. If seccomp options are\n provided at both the pod & container level, the container options\n override the pod options.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n localhostProfile:\n description: |-\n localhostProfile indicates a profile defined in a file on the node should be used.\n The profile must be preconfigured on the node to work.\n Must be a descending path, relative to the kubelet's configured seccomp profile location.\n Must be set if type is \"Localhost\". Must NOT be set for any other type.\n type: string\n type:\n description: |-\n type indicates which kind of seccomp profile will be applied.\n Valid options are:\n\n\n Localhost - a profile defined in a file on the node should be used.\n RuntimeDefault - the container runtime default profile should be used.\n Unconfined - no profile should be applied.\n type: string\n required:\n - type\n type: object\n windowsOptions:\n description: |-\n The Windows specific settings applied to all containers.\n If unspecified, the options from the PodSecurityContext will be used.\n If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is linux.\n properties:\n gmsaCredentialSpec:\n description: |-\n GMSACredentialSpec is where the GMSA admission webhook\n (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the\n GMSA credential spec named by the GMSACredentialSpecName field.\n type: string\n gmsaCredentialSpecName:\n description: GMSACredentialSpecName is the\n name of the GMSA credential spec to use.\n type: string\n hostProcess:\n description: |-\n HostProcess determines if a container should be run as a 'Host Process' container.\n All of a Pod's containers must have the same effective HostProcess value\n (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers).\n In addition, if HostProcess is true then HostNetwork must also be set to true.\n type: boolean\n runAsUserName:\n description: |-\n The UserName in Windows to run the entrypoint of the container process.\n Defaults to the user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: string\n type: object\n type: object\n startupProbe:\n description: Probes are not allowed for ephemeral\n containers.\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n stdin:\n description: |-\n Whether this container should allocate a buffer for stdin in the container runtime. If this\n is not set, reads from stdin in the container will always result in EOF.\n Default is false.\n type: boolean\n stdinOnce:\n description: |-\n Whether the container runtime should close the stdin channel after it has been opened by\n a single attach. When stdin is true the stdin stream will remain open across multiple attach\n sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the\n first client attaches to stdin, and then remains open and accepts data until the client disconnects,\n at which time stdin is closed and remains closed until the container is restarted. If this\n flag is false, a container processes that reads from stdin will never receive an EOF.\n Default is false\n type: boolean\n targetContainerName:\n description: |-\n If set, the name of the container from PodSpec that this ephemeral container targets.\n The ephemeral container will be run in the namespaces (IPC, PID, etc) of this container.\n If not set then the ephemeral container uses the namespaces configured in the Pod spec.\n\n\n The container runtime must implement support for this feature. If the runtime does not\n support namespace targeting then the result of setting this field is undefined.\n type: string\n terminationMessagePath:\n description: |-\n Optional: Path at which the file to which the container's termination message\n will be written is mounted into the container's filesystem.\n Message written is intended to be brief final status, such as an assertion failure message.\n Will be truncated by the node if greater than 4096 bytes. The total message length across\n all containers will be limited to 12kb.\n Defaults to /dev/termination-log.\n Cannot be updated.\n type: string\n terminationMessagePolicy:\n description: |-\n Indicate how the termination message should be populated. File will use the contents of\n terminationMessagePath to populate the container status message on both success and failure.\n FallbackToLogsOnError will use the last chunk of container log output if the termination\n message file is empty and the container exited with an error.\n The log output is limited to 2048 bytes or 80 lines, whichever is smaller.\n Defaults to File.\n Cannot be updated.\n type: string\n tty:\n description: |-\n Whether this container should allocate a TTY for itself, also requires 'stdin' to be true.\n Default is false.\n type: boolean\n volumeDevices:\n description: volumeDevices is the list of block devices\n to be used by the container.\n items:\n description: volumeDevice describes a mapping of\n a raw block device within a container.\n properties:\n devicePath:\n description: devicePath is the path inside of\n the container that the device will be mapped\n to.\n type: string\n name:\n description: name must match the name of a persistentVolumeClaim\n in the pod\n type: string\n required:\n - devicePath\n - name\n type: object\n type: array\n volumeMounts:\n description: |-\n Pod volumes to mount into the container's filesystem. Subpath mounts are not allowed for ephemeral containers.\n Cannot be updated.\n items:\n description: VolumeMount describes a mounting of\n a Volume within a container.\n properties:\n mountPath:\n description: |-\n Path within the container at which the volume should be mounted. Must\n not contain ':'.\n type: string\n mountPropagation:\n description: |-\n mountPropagation determines how mounts are propagated from the host\n to container and the other way around.\n When not set, MountPropagationNone is used.\n This field is beta in 1.10.\n type: string\n name:\n description: This must match the Name of a Volume.\n type: string\n readOnly:\n description: |-\n Mounted read-only if true, read-write otherwise (false or unspecified).\n Defaults to false.\n type: boolean\n subPath:\n description: |-\n Path within the volume from which the container's volume should be mounted.\n Defaults to \"\" (volume's root).\n type: string\n subPathExpr:\n description: |-\n Expanded path within the volume from which the container's volume should be mounted.\n Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment.\n Defaults to \"\" (volume's root).\n SubPathExpr and SubPath are mutually exclusive.\n type: string\n required:\n - mountPath\n - name\n type: object\n type: array\n workingDir:\n description: |-\n Container's working directory.\n If not specified, the container runtime's default will be used, which\n might be configured in the container image.\n Cannot be updated.\n type: string\n required:\n - name\n type: object\n type: array\n hostAliases:\n description: |-\n HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts\n file if specified. This is only valid for non-hostNetwork pods.\n items:\n description: |-\n HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the\n pod's hosts file.\n properties:\n hostnames:\n description: Hostnames for the above IP address.\n items:\n type: string\n type: array\n ip:\n description: IP address of the host file entry.\n type: string\n type: object\n type: array\n hostIPC:\n description: |-\n Use the host's ipc namespace.\n Optional: Default to false.\n type: boolean\n hostNetwork:\n description: |-\n Host networking requested for this pod. Use the host's network namespace.\n If this option is set, the ports that will be used must be specified.\n Default to false.\n type: boolean\n hostPID:\n description: |-\n Use the host's pid namespace.\n Optional: Default to false.\n type: boolean\n hostUsers:\n description: |-\n Use the host's user namespace.\n Optional: Default to true.\n If set to true or not present, the pod will be run in the host user namespace, useful\n for when the pod needs a feature only available to the host user namespace, such as\n loading a kernel module with CAP_SYS_MODULE.\n When set to false, a new userns is created for the pod. Setting false is useful for\n mitigating container breakout vulnerabilities even allowing users to run their\n containers as root without actually having root privileges on the host.\n This field is alpha-level and is only honored by servers that enable the UserNamespacesSupport feature.\n type: boolean\n hostname:\n description: |-\n Specifies the hostname of the Pod\n If not specified, the pod's hostname will be set to a system-defined value.\n type: string\n imagePullSecrets:\n description: |-\n ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec.\n If specified, these secrets will be passed to individual puller implementations for them to use.\n More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod\n items:\n description: |-\n LocalObjectReference contains enough information to let you locate the\n referenced object inside the same namespace.\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n type: object\n x-kubernetes-map-type: atomic\n type: array\n initContainers:\n description: |-\n List of initialization containers belonging to the pod.\n Init containers are executed in order prior to containers being started. If any\n init container fails, the pod is considered to have failed and is handled according\n to its restartPolicy. The name for an init container or normal container must be\n unique among all containers.\n Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes.\n The resourceRequirements of an init container are taken into account during scheduling\n by finding the highest request/limit for each resource type, and then using the max of\n of that value or the sum of the normal containers. Limits are applied to init containers\n in a similar fashion.\n Init containers cannot currently be added or removed.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/\n items:\n description: A single application container that you want\n to run within a pod.\n properties:\n args:\n description: |-\n Arguments to the entrypoint.\n The container image's CMD is used if this is not provided.\n Variable references $(VAR_NAME) are expanded using the container's environment. If a variable\n cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will\n produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless\n of whether the variable exists or not. Cannot be updated.\n More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell\n items:\n type: string\n type: array\n command:\n description: |-\n Entrypoint array. Not executed within a shell.\n The container image's ENTRYPOINT is used if this is not provided.\n Variable references $(VAR_NAME) are expanded using the container's environment. If a variable\n cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will\n produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless\n of whether the variable exists or not. Cannot be updated.\n More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell\n items:\n type: string\n type: array\n env:\n description: |-\n List of environment variables to set in the container.\n Cannot be updated.\n items:\n description: EnvVar represents an environment variable\n present in a Container.\n properties:\n name:\n description: Name of the environment variable.\n Must be a C_IDENTIFIER.\n type: string\n value:\n description: |-\n Variable references $(VAR_NAME) are expanded\n using the previously defined environment variables in the container and\n any service environment variables. If a variable cannot be resolved,\n the reference in the input string will be unchanged. Double $$ are reduced\n to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e.\n \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\".\n Escaped references will never be expanded, regardless of whether the variable\n exists or not.\n Defaults to \"\".\n type: string\n valueFrom:\n description: Source for the environment variable's\n value. Cannot be used if value is not empty.\n properties:\n configMapKeyRef:\n description: Selects a key of a ConfigMap.\n properties:\n key:\n description: The key to select.\n type: string\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the ConfigMap\n or its key must be defined\n type: boolean\n required:\n - key\n type: object\n x-kubernetes-map-type: atomic\n fieldRef:\n description: |-\n Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`,\n spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.\n properties:\n apiVersion:\n description: Version of the schema the\n FieldPath is written in terms of,\n defaults to \"v1\".\n type: string\n fieldPath:\n description: Path of the field to select\n in the specified API version.\n type: string\n required:\n - fieldPath\n type: object\n x-kubernetes-map-type: atomic\n resourceFieldRef:\n description: |-\n Selects a resource of the container: only resources limits and requests\n (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.\n properties:\n containerName:\n description: 'Container name: required\n for volumes, optional for env vars'\n type: string\n divisor:\n anyOf:\n - type: integer\n - type: string\n description: Specifies the output format\n of the exposed resources, defaults\n to \"1\"\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n resource:\n description: 'Required: resource to\n select'\n type: string\n required:\n - resource\n type: object\n x-kubernetes-map-type: atomic\n secretKeyRef:\n description: Selects a key of a secret in\n the pod's namespace\n properties:\n key:\n description: The key of the secret to\n select from. Must be a valid secret\n key.\n type: string\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the Secret\n or its key must be defined\n type: boolean\n required:\n - key\n type: object\n x-kubernetes-map-type: atomic\n type: object\n required:\n - name\n type: object\n type: array\n envFrom:\n description: |-\n List of sources to populate environment variables in the container.\n The keys defined within a source must be a C_IDENTIFIER. All invalid keys\n will be reported as an event when the container is starting. When a key exists in multiple\n sources, the value associated with the last source will take precedence.\n Values defined by an Env with a duplicate key will take precedence.\n Cannot be updated.\n items:\n description: EnvFromSource represents the source\n of a set of ConfigMaps\n properties:\n configMapRef:\n description: The ConfigMap to select from\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the ConfigMap\n must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n prefix:\n description: An optional identifier to prepend\n to each key in the ConfigMap. Must be a C_IDENTIFIER.\n type: string\n secretRef:\n description: The Secret to select from\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: Specify whether the Secret\n must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n type: object\n type: array\n image:\n description: |-\n Container image name.\n More info: https://kubernetes.io/docs/concepts/containers/images\n This field is optional to allow higher level config management to default or override\n container images in workload controllers like Deployments and StatefulSets.\n type: string\n imagePullPolicy:\n description: |-\n Image pull policy.\n One of Always, Never, IfNotPresent.\n Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n type: string\n lifecycle:\n description: |-\n Actions that the management system should take in response to container lifecycle events.\n Cannot be updated.\n properties:\n postStart:\n description: |-\n PostStart is called immediately after a container is created. If the handler fails,\n the container is terminated and restarted according to its restart policy.\n Other management of the container blocks until the hook completes.\n More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks\n properties:\n exec:\n description: Exec specifies the action to\n take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in\n the request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a\n custom header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP\n server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n sleep:\n description: Sleep represents the duration\n that the container should sleep before being\n terminated.\n properties:\n seconds:\n description: Seconds is the number of\n seconds to sleep.\n format: int64\n type: integer\n required:\n - seconds\n type: object\n tcpSocket:\n description: |-\n Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept\n for the backward compatibility. There are no validation of this field and\n lifecycle hooks will fail in runtime when tcp handler is specified.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n type: object\n preStop:\n description: |-\n PreStop is called immediately before a container is terminated due to an\n API request or management event such as liveness/startup probe failure,\n preemption, resource contention, etc. The handler is not called if the\n container crashes or exits. The Pod's termination grace period countdown begins before the\n PreStop hook is executed. Regardless of the outcome of the handler, the\n container will eventually terminate within the Pod's termination grace\n period (unless delayed by finalizers). Other management of the container blocks until the hook completes\n or until the termination grace period is reached.\n More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks\n properties:\n exec:\n description: Exec specifies the action to\n take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in\n the request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a\n custom header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP\n server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n sleep:\n description: Sleep represents the duration\n that the container should sleep before being\n terminated.\n properties:\n seconds:\n description: Seconds is the number of\n seconds to sleep.\n format: int64\n type: integer\n required:\n - seconds\n type: object\n tcpSocket:\n description: |-\n Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept\n for the backward compatibility. There are no validation of this field and\n lifecycle hooks will fail in runtime when tcp handler is specified.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n type: object\n type: object\n livenessProbe:\n description: |-\n Periodic probe of container liveness.\n Container will be restarted if the probe fails.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n name:\n description: |-\n Name of the container specified as a DNS_LABEL.\n Each container in a pod must have a unique name (DNS_LABEL).\n Cannot be updated.\n type: string\n ports:\n description: |-\n List of ports to expose from the container. Not specifying a port here\n DOES NOT prevent that port from being exposed. Any port which is\n listening on the default \"0.0.0.0\" address inside a container will be\n accessible from the network.\n Modifying this array with strategic merge patch may corrupt the data.\n For more information See https://github.com/kubernetes/kubernetes/issues/108255.\n Cannot be updated.\n items:\n description: ContainerPort represents a network\n port in a single container.\n properties:\n containerPort:\n description: |-\n Number of port to expose on the pod's IP address.\n This must be a valid port number, 0 < x < 65536.\n format: int32\n type: integer\n hostIP:\n description: What host IP to bind the external\n port to.\n type: string\n hostPort:\n description: |-\n Number of port to expose on the host.\n If specified, this must be a valid port number, 0 < x < 65536.\n If HostNetwork is specified, this must match ContainerPort.\n Most containers do not need this.\n format: int32\n type: integer\n name:\n description: |-\n If specified, this must be an IANA_SVC_NAME and unique within the pod. Each\n named port in a pod must have a unique name. Name for the port that can be\n referred to by services.\n type: string\n protocol:\n default: TCP\n description: |-\n Protocol for port. Must be UDP, TCP, or SCTP.\n Defaults to \"TCP\".\n type: string\n required:\n - containerPort\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - containerPort\n - protocol\n x-kubernetes-list-type: map\n readinessProbe:\n description: |-\n Periodic probe of container service readiness.\n Container will be removed from service endpoints if the probe fails.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n resizePolicy:\n description: Resources resize policy for the container.\n items:\n description: ContainerResizePolicy represents resource\n resize policy for the container.\n properties:\n resourceName:\n description: |-\n Name of the resource to which this resource resize policy applies.\n Supported values: cpu, memory.\n type: string\n restartPolicy:\n description: |-\n Restart policy to apply when specified resource is resized.\n If not specified, it defaults to NotRequired.\n type: string\n required:\n - resourceName\n - restartPolicy\n type: object\n type: array\n x-kubernetes-list-type: atomic\n resources:\n description: |-\n Compute Resources required by this container.\n Cannot be updated.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n properties:\n claims:\n description: |-\n Claims lists the names of resources, defined in spec.resourceClaims,\n that are used by this container.\n\n\n This is an alpha field and requires enabling the\n DynamicResourceAllocation feature gate.\n\n\n This field is immutable. It can only be set for containers.\n items:\n description: ResourceClaim references one entry\n in PodSpec.ResourceClaims.\n properties:\n name:\n description: |-\n Name must match the name of one entry in pod.spec.resourceClaims of\n the Pod where this field is used. It makes that resource available\n inside a container.\n type: string\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n limits:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Limits describes the maximum amount of compute resources allowed.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n type: object\n requests:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Requests describes the minimum amount of compute resources required.\n If Requests is omitted for a container, it defaults to Limits if that is explicitly specified,\n otherwise to an implementation-defined value. Requests cannot exceed Limits.\n More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/\n type: object\n type: object\n restartPolicy:\n description: |-\n RestartPolicy defines the restart behavior of individual containers in a pod.\n This field may only be set for init containers, and the only allowed value is \"Always\".\n For non-init containers or when this field is not specified,\n the restart behavior is defined by the Pod's restart policy and the container type.\n Setting the RestartPolicy as \"Always\" for the init container will have the following effect:\n this init container will be continually restarted on\n exit until all regular containers have terminated. Once all regular\n containers have completed, all init containers with restartPolicy \"Always\"\n will be shut down. This lifecycle differs from normal init containers and\n is often referred to as a \"sidecar\" container. Although this init\n container still starts in the init container sequence, it does not wait\n for the container to complete before proceeding to the next init\n container. Instead, the next init container starts immediately after this\n init container is started, or after any startupProbe has successfully\n completed.\n type: string\n securityContext:\n description: |-\n SecurityContext defines the security options the container should be run with.\n If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.\n More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/\n properties:\n allowPrivilegeEscalation:\n description: |-\n AllowPrivilegeEscalation controls whether a process can gain more\n privileges than its parent process. This bool directly controls if\n the no_new_privs flag will be set on the container process.\n AllowPrivilegeEscalation is true always when the container is:\n 1) run as Privileged\n 2) has CAP_SYS_ADMIN\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n capabilities:\n description: |-\n The capabilities to add/drop when running containers.\n Defaults to the default set of capabilities granted by the container runtime.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n add:\n description: Added capabilities\n items:\n description: Capability represent POSIX\n capabilities type\n type: string\n type: array\n drop:\n description: Removed capabilities\n items:\n description: Capability represent POSIX\n capabilities type\n type: string\n type: array\n type: object\n privileged:\n description: |-\n Run container in privileged mode.\n Processes in privileged containers are essentially equivalent to root on the host.\n Defaults to false.\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n procMount:\n description: |-\n procMount denotes the type of proc mount to use for the containers.\n The default is DefaultProcMount which uses the container runtime defaults for\n readonly paths and masked paths.\n This requires the ProcMountType feature flag to be enabled.\n Note that this field cannot be set when spec.os.name is windows.\n type: string\n readOnlyRootFilesystem:\n description: |-\n Whether this container has a read-only root filesystem.\n Default is false.\n Note that this field cannot be set when spec.os.name is windows.\n type: boolean\n runAsGroup:\n description: |-\n The GID to run the entrypoint of the container process.\n Uses runtime default if unset.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n runAsNonRoot:\n description: |-\n Indicates that the container must run as a non-root user.\n If true, the Kubelet will validate the image at runtime to ensure that it\n does not run as UID 0 (root) and fail to start the container if it does.\n If unset or false, no such validation will be performed.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: boolean\n runAsUser:\n description: |-\n The UID to run the entrypoint of the container process.\n Defaults to user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n seLinuxOptions:\n description: |-\n The SELinux context to be applied to the container.\n If unspecified, the container runtime will allocate a random SELinux context for each\n container. May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n level:\n description: Level is SELinux level label\n that applies to the container.\n type: string\n role:\n description: Role is a SELinux role label\n that applies to the container.\n type: string\n type:\n description: Type is a SELinux type label\n that applies to the container.\n type: string\n user:\n description: User is a SELinux user label\n that applies to the container.\n type: string\n type: object\n seccompProfile:\n description: |-\n The seccomp options to use by this container. If seccomp options are\n provided at both the pod & container level, the container options\n override the pod options.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n localhostProfile:\n description: |-\n localhostProfile indicates a profile defined in a file on the node should be used.\n The profile must be preconfigured on the node to work.\n Must be a descending path, relative to the kubelet's configured seccomp profile location.\n Must be set if type is \"Localhost\". Must NOT be set for any other type.\n type: string\n type:\n description: |-\n type indicates which kind of seccomp profile will be applied.\n Valid options are:\n\n\n Localhost - a profile defined in a file on the node should be used.\n RuntimeDefault - the container runtime default profile should be used.\n Unconfined - no profile should be applied.\n type: string\n required:\n - type\n type: object\n windowsOptions:\n description: |-\n The Windows specific settings applied to all containers.\n If unspecified, the options from the PodSecurityContext will be used.\n If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is linux.\n properties:\n gmsaCredentialSpec:\n description: |-\n GMSACredentialSpec is where the GMSA admission webhook\n (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the\n GMSA credential spec named by the GMSACredentialSpecName field.\n type: string\n gmsaCredentialSpecName:\n description: GMSACredentialSpecName is the\n name of the GMSA credential spec to use.\n type: string\n hostProcess:\n description: |-\n HostProcess determines if a container should be run as a 'Host Process' container.\n All of a Pod's containers must have the same effective HostProcess value\n (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers).\n In addition, if HostProcess is true then HostNetwork must also be set to true.\n type: boolean\n runAsUserName:\n description: |-\n The UserName in Windows to run the entrypoint of the container process.\n Defaults to the user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: string\n type: object\n type: object\n startupProbe:\n description: |-\n StartupProbe indicates that the Pod has successfully initialized.\n If specified, no other probes are executed until this completes successfully.\n If this probe fails, the Pod will be restarted, just as if the livenessProbe failed.\n This can be used to provide different probe parameters at the beginning of a Pod's lifecycle,\n when it might take a long time to load data or warm a cache, than during steady-state operation.\n This cannot be updated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n properties:\n exec:\n description: Exec specifies the action to take.\n properties:\n command:\n description: |-\n Command is the command line to execute inside the container, the working directory for the\n command is root ('/') in the container's filesystem. The command is simply exec'd, it is\n not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use\n a shell, you need to explicitly call out to that shell.\n Exit status of 0 is treated as live/healthy and non-zero is unhealthy.\n items:\n type: string\n type: array\n type: object\n failureThreshold:\n description: |-\n Minimum consecutive failures for the probe to be considered failed after having succeeded.\n Defaults to 3. Minimum value is 1.\n format: int32\n type: integer\n grpc:\n description: GRPC specifies an action involving\n a GRPC port.\n properties:\n port:\n description: Port number of the gRPC service.\n Number must be in the range 1 to 65535.\n format: int32\n type: integer\n service:\n description: |-\n Service is the name of the service to place in the gRPC HealthCheckRequest\n (see https://github.com/grpc/grpc/blob/master/doc/health-checking.md).\n\n\n If this is not specified, the default behavior is defined by gRPC.\n type: string\n required:\n - port\n type: object\n httpGet:\n description: HTTPGet specifies the http request\n to perform.\n properties:\n host:\n description: |-\n Host name to connect to, defaults to the pod IP. You probably want to set\n \"Host\" in httpHeaders instead.\n type: string\n httpHeaders:\n description: Custom headers to set in the\n request. HTTP allows repeated headers.\n items:\n description: HTTPHeader describes a custom\n header to be used in HTTP probes\n properties:\n name:\n description: |-\n The header field name.\n This will be canonicalized upon output, so case-variant names will be understood as the same header.\n type: string\n value:\n description: The header field value\n type: string\n required:\n - name\n - value\n type: object\n type: array\n path:\n description: Path to access on the HTTP server.\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Name or number of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n scheme:\n description: |-\n Scheme to use for connecting to the host.\n Defaults to HTTP.\n type: string\n required:\n - port\n type: object\n initialDelaySeconds:\n description: |-\n Number of seconds after the container has started before liveness probes are initiated.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n periodSeconds:\n description: |-\n How often (in seconds) to perform the probe.\n Default to 10 seconds. Minimum value is 1.\n format: int32\n type: integer\n successThreshold:\n description: |-\n Minimum consecutive successes for the probe to be considered successful after having failed.\n Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.\n format: int32\n type: integer\n tcpSocket:\n description: TCPSocket specifies an action involving\n a TCP port.\n properties:\n host:\n description: 'Optional: Host name to connect\n to, defaults to the pod IP.'\n type: string\n port:\n anyOf:\n - type: integer\n - type: string\n description: |-\n Number or name of the port to access on the container.\n Number must be in the range 1 to 65535.\n Name must be an IANA_SVC_NAME.\n x-kubernetes-int-or-string: true\n required:\n - port\n type: object\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully upon probe failure.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this\n value overrides the value provided by the pod spec.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate.\n Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.\n format: int64\n type: integer\n timeoutSeconds:\n description: |-\n Number of seconds after which the probe times out.\n Defaults to 1 second. Minimum value is 1.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes\n format: int32\n type: integer\n type: object\n stdin:\n description: |-\n Whether this container should allocate a buffer for stdin in the container runtime. If this\n is not set, reads from stdin in the container will always result in EOF.\n Default is false.\n type: boolean\n stdinOnce:\n description: |-\n Whether the container runtime should close the stdin channel after it has been opened by\n a single attach. When stdin is true the stdin stream will remain open across multiple attach\n sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the\n first client attaches to stdin, and then remains open and accepts data until the client disconnects,\n at which time stdin is closed and remains closed until the container is restarted. If this\n flag is false, a container processes that reads from stdin will never receive an EOF.\n Default is false\n type: boolean\n terminationMessagePath:\n description: |-\n Optional: Path at which the file to which the container's termination message\n will be written is mounted into the container's filesystem.\n Message written is intended to be brief final status, such as an assertion failure message.\n Will be truncated by the node if greater than 4096 bytes. The total message length across\n all containers will be limited to 12kb.\n Defaults to /dev/termination-log.\n Cannot be updated.\n type: string\n terminationMessagePolicy:\n description: |-\n Indicate how the termination message should be populated. File will use the contents of\n terminationMessagePath to populate the container status message on both success and failure.\n FallbackToLogsOnError will use the last chunk of container log output if the termination\n message file is empty and the container exited with an error.\n The log output is limited to 2048 bytes or 80 lines, whichever is smaller.\n Defaults to File.\n Cannot be updated.\n type: string\n tty:\n description: |-\n Whether this container should allocate a TTY for itself, also requires 'stdin' to be true.\n Default is false.\n type: boolean\n volumeDevices:\n description: volumeDevices is the list of block devices\n to be used by the container.\n items:\n description: volumeDevice describes a mapping of\n a raw block device within a container.\n properties:\n devicePath:\n description: devicePath is the path inside of\n the container that the device will be mapped\n to.\n type: string\n name:\n description: name must match the name of a persistentVolumeClaim\n in the pod\n type: string\n required:\n - devicePath\n - name\n type: object\n type: array\n volumeMounts:\n description: |-\n Pod volumes to mount into the container's filesystem.\n Cannot be updated.\n items:\n description: VolumeMount describes a mounting of\n a Volume within a container.\n properties:\n mountPath:\n description: |-\n Path within the container at which the volume should be mounted. Must\n not contain ':'.\n type: string\n mountPropagation:\n description: |-\n mountPropagation determines how mounts are propagated from the host\n to container and the other way around.\n When not set, MountPropagationNone is used.\n This field is beta in 1.10.\n type: string\n name:\n description: This must match the Name of a Volume.\n type: string\n readOnly:\n description: |-\n Mounted read-only if true, read-write otherwise (false or unspecified).\n Defaults to false.\n type: boolean\n subPath:\n description: |-\n Path within the volume from which the container's volume should be mounted.\n Defaults to \"\" (volume's root).\n type: string\n subPathExpr:\n description: |-\n Expanded path within the volume from which the container's volume should be mounted.\n Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment.\n Defaults to \"\" (volume's root).\n SubPathExpr and SubPath are mutually exclusive.\n type: string\n required:\n - mountPath\n - name\n type: object\n type: array\n workingDir:\n description: |-\n Container's working directory.\n If not specified, the container runtime's default will be used, which\n might be configured in the container image.\n Cannot be updated.\n type: string\n required:\n - name\n type: object\n type: array\n nodeName:\n description: |-\n NodeName is a request to schedule this pod onto a specific node. If it is non-empty,\n the scheduler simply schedules this pod onto that node, assuming that it fits resource\n requirements.\n type: string\n nodeSelector:\n additionalProperties:\n type: string\n description: |-\n NodeSelector is a selector which must be true for the pod to fit on a node.\n Selector which must match a node's labels for the pod to be scheduled on that node.\n More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/\n type: object\n x-kubernetes-map-type: atomic\n os:\n description: |-\n Specifies the OS of the containers in the pod.\n Some pod and container fields are restricted if this is set.\n\n\n If the OS field is set to linux, the following fields must be unset:\n -securityContext.windowsOptions\n\n\n If the OS field is set to windows, following fields must be unset:\n - spec.hostPID\n - spec.hostIPC\n - spec.hostUsers\n - spec.securityContext.seLinuxOptions\n - spec.securityContext.seccompProfile\n - spec.securityContext.fsGroup\n - spec.securityContext.fsGroupChangePolicy\n - spec.securityContext.sysctls\n - spec.shareProcessNamespace\n - spec.securityContext.runAsUser\n - spec.securityContext.runAsGroup\n - spec.securityContext.supplementalGroups\n - spec.containers[*].securityContext.seLinuxOptions\n - spec.containers[*].securityContext.seccompProfile\n - spec.containers[*].securityContext.capabilities\n - spec.containers[*].securityContext.readOnlyRootFilesystem\n - spec.containers[*].securityContext.privileged\n - spec.containers[*].securityContext.allowPrivilegeEscalation\n - spec.containers[*].securityContext.procMount\n - spec.containers[*].securityContext.runAsUser\n - spec.containers[*].securityContext.runAsGroup\n properties:\n name:\n description: |-\n Name is the name of the operating system. The currently supported values are linux and windows.\n Additional value may be defined in future and can be one of:\n https://github.com/opencontainers/runtime-spec/blob/master/config.md#platform-specific-configuration\n Clients should expect to handle additional values and treat unrecognized values in this field as os: null\n type: string\n required:\n - name\n type: object\n overhead:\n additionalProperties:\n anyOf:\n - type: integer\n - type: string\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n description: |-\n Overhead represents the resource overhead associated with running a pod for a given RuntimeClass.\n This field will be autopopulated at admission time by the RuntimeClass admission controller. If\n the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests.\n The RuntimeClass admission controller will reject Pod create requests which have the overhead already\n set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value\n defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero.\n More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md\n type: object\n preemptionPolicy:\n description: |-\n PreemptionPolicy is the Policy for preempting pods with lower priority.\n One of Never, PreemptLowerPriority.\n Defaults to PreemptLowerPriority if unset.\n type: string\n priority:\n description: |-\n The priority value. Various system components use this field to find the\n priority of the pod. When Priority Admission Controller is enabled, it\n prevents users from setting this field. The admission controller populates\n this field from PriorityClassName.\n The higher the value, the higher the priority.\n format: int32\n type: integer\n priorityClassName:\n description: |-\n If specified, indicates the pod's priority. \"system-node-critical\" and\n \"system-cluster-critical\" are two special keywords which indicate the\n highest priorities with the former being the highest priority. Any other\n name must be defined by creating a PriorityClass object with that name.\n If not specified, the pod priority will be default or zero if there is no\n default.\n type: string\n readinessGates:\n description: |-\n If specified, all readiness gates will be evaluated for pod readiness.\n A pod is ready when all its containers are ready AND\n all conditions specified in the readiness gates have status equal to \"True\"\n More info: https://git.k8s.io/enhancements/keps/sig-network/580-pod-readiness-gates\n items:\n description: PodReadinessGate contains the reference to\n a pod condition\n properties:\n conditionType:\n description: ConditionType refers to a condition in\n the pod's condition list with matching type.\n type: string\n required:\n - conditionType\n type: object\n type: array\n resourceClaims:\n description: |-\n ResourceClaims defines which ResourceClaims must be allocated\n and reserved before the Pod is allowed to start. The resources\n will be made available to those containers which consume them\n by name.\n\n\n This is an alpha field and requires enabling the\n DynamicResourceAllocation feature gate.\n\n\n This field is immutable.\n items:\n description: |-\n PodResourceClaim references exactly one ResourceClaim through a ClaimSource.\n It adds a name to it that uniquely identifies the ResourceClaim inside the Pod.\n Containers that need access to the ResourceClaim reference it with this name.\n properties:\n name:\n description: |-\n Name uniquely identifies this resource claim inside the pod.\n This must be a DNS_LABEL.\n type: string\n source:\n description: Source describes where to find the ResourceClaim.\n properties:\n resourceClaimName:\n description: |-\n ResourceClaimName is the name of a ResourceClaim object in the same\n namespace as this pod.\n type: string\n resourceClaimTemplateName:\n description: |-\n ResourceClaimTemplateName is the name of a ResourceClaimTemplate\n object in the same namespace as this pod.\n\n\n The template will be used to create a new ResourceClaim, which will\n be bound to this pod. When this pod is deleted, the ResourceClaim\n will also be deleted. The pod name and resource name, along with a\n generated component, will be used to form a unique name for the\n ResourceClaim, which will be recorded in pod.status.resourceClaimStatuses.\n\n\n This field is immutable and no changes will be made to the\n corresponding ResourceClaim by the control plane after creating the\n ResourceClaim.\n type: string\n type: object\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n restartPolicy:\n description: |-\n Restart policy for all containers within the pod.\n One of Always, OnFailure, Never. In some contexts, only a subset of those values may be permitted.\n Default to Always.\n More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy\n type: string\n runtimeClassName:\n description: |-\n RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used\n to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run.\n If unset or empty, the \"legacy\" RuntimeClass will be used, which is an implicit class with an\n empty definition that uses the default runtime handler.\n More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class\n type: string\n schedulerName:\n description: |-\n If specified, the pod will be dispatched by specified scheduler.\n If not specified, the pod will be dispatched by default scheduler.\n type: string\n schedulingGates:\n description: |-\n SchedulingGates is an opaque list of values that if specified will block scheduling the pod.\n If schedulingGates is not empty, the pod will stay in the SchedulingGated state and the\n scheduler will not attempt to schedule the pod.\n\n\n SchedulingGates can only be set at pod creation time, and be removed only afterwards.\n\n\n This is a beta feature enabled by the PodSchedulingReadiness feature gate.\n items:\n description: PodSchedulingGate is associated to a Pod\n to guard its scheduling.\n properties:\n name:\n description: |-\n Name of the scheduling gate.\n Each scheduling gate must have a unique name field.\n type: string\n required:\n - name\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - name\n x-kubernetes-list-type: map\n securityContext:\n description: |-\n SecurityContext holds pod-level security attributes and common container settings.\n Optional: Defaults to empty. See type description for default values of each field.\n properties:\n fsGroup:\n description: |-\n A special supplemental group that applies to all containers in a pod.\n Some volume types allow the Kubelet to change the ownership of that volume\n to be owned by the pod:\n\n\n 1. The owning GID will be the FSGroup\n 2. The setgid bit is set (new files created in the volume will be owned by FSGroup)\n 3. The permission bits are OR'd with rw-rw----\n\n\n If unset, the Kubelet will not modify the ownership and permissions of any volume.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n fsGroupChangePolicy:\n description: |-\n fsGroupChangePolicy defines behavior of changing ownership and permission of the volume\n before being exposed inside Pod. This field will only apply to\n volume types which support fsGroup based ownership(and permissions).\n It will have no effect on ephemeral volume types such as: secret, configmaps\n and emptydir.\n Valid values are \"OnRootMismatch\" and \"Always\". If not specified, \"Always\" is used.\n Note that this field cannot be set when spec.os.name is windows.\n type: string\n runAsGroup:\n description: |-\n The GID to run the entrypoint of the container process.\n Uses runtime default if unset.\n May also be set in SecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence\n for that container.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n runAsNonRoot:\n description: |-\n Indicates that the container must run as a non-root user.\n If true, the Kubelet will validate the image at runtime to ensure that it\n does not run as UID 0 (root) and fail to start the container if it does.\n If unset or false, no such validation will be performed.\n May also be set in SecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: boolean\n runAsUser:\n description: |-\n The UID to run the entrypoint of the container process.\n Defaults to user specified in image metadata if unspecified.\n May also be set in SecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence\n for that container.\n Note that this field cannot be set when spec.os.name is windows.\n format: int64\n type: integer\n seLinuxOptions:\n description: |-\n The SELinux context to be applied to all containers.\n If unspecified, the container runtime will allocate a random SELinux context for each\n container. May also be set in SecurityContext. If set in\n both SecurityContext and PodSecurityContext, the value specified in SecurityContext\n takes precedence for that container.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n level:\n description: Level is SELinux level label that applies\n to the container.\n type: string\n role:\n description: Role is a SELinux role label that applies\n to the container.\n type: string\n type:\n description: Type is a SELinux type label that applies\n to the container.\n type: string\n user:\n description: User is a SELinux user label that applies\n to the container.\n type: string\n type: object\n seccompProfile:\n description: |-\n The seccomp options to use by the containers in this pod.\n Note that this field cannot be set when spec.os.name is windows.\n properties:\n localhostProfile:\n description: |-\n localhostProfile indicates a profile defined in a file on the node should be used.\n The profile must be preconfigured on the node to work.\n Must be a descending path, relative to the kubelet's configured seccomp profile location.\n Must be set if type is \"Localhost\". Must NOT be set for any other type.\n type: string\n type:\n description: |-\n type indicates which kind of seccomp profile will be applied.\n Valid options are:\n\n\n Localhost - a profile defined in a file on the node should be used.\n RuntimeDefault - the container runtime default profile should be used.\n Unconfined - no profile should be applied.\n type: string\n required:\n - type\n type: object\n supplementalGroups:\n description: |-\n A list of groups applied to the first process run in each container, in addition\n to the container's primary GID, the fsGroup (if specified), and group memberships\n defined in the container image for the uid of the container process. If unspecified,\n no additional groups are added to any container. Note that group memberships\n defined in the container image for the uid of the container process are still effective,\n even if they are not included in this list.\n Note that this field cannot be set when spec.os.name is windows.\n items:\n format: int64\n type: integer\n type: array\n sysctls:\n description: |-\n Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported\n sysctls (by the container runtime) might fail to launch.\n Note that this field cannot be set when spec.os.name is windows.\n items:\n description: Sysctl defines a kernel parameter to\n be set\n properties:\n name:\n description: Name of a property to set\n type: string\n value:\n description: Value of a property to set\n type: string\n required:\n - name\n - value\n type: object\n type: array\n windowsOptions:\n description: |-\n The Windows specific settings applied to all containers.\n If unspecified, the options within a container's SecurityContext will be used.\n If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.\n Note that this field cannot be set when spec.os.name is linux.\n properties:\n gmsaCredentialSpec:\n description: |-\n GMSACredentialSpec is where the GMSA admission webhook\n (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the\n GMSA credential spec named by the GMSACredentialSpecName field.\n type: string\n gmsaCredentialSpecName:\n description: GMSACredentialSpecName is the name\n of the GMSA credential spec to use.\n type: string\n hostProcess:\n description: |-\n HostProcess determines if a container should be run as a 'Host Process' container.\n All of a Pod's containers must have the same effective HostProcess value\n (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers).\n In addition, if HostProcess is true then HostNetwork must also be set to true.\n type: boolean\n runAsUserName:\n description: |-\n The UserName in Windows to run the entrypoint of the container process.\n Defaults to the user specified in image metadata if unspecified.\n May also be set in PodSecurityContext. If set in both SecurityContext and\n PodSecurityContext, the value specified in SecurityContext takes precedence.\n type: string\n type: object\n type: object\n serviceAccount:\n description: |-\n DeprecatedServiceAccount is a depreciated alias for ServiceAccountName.\n Deprecated: Use serviceAccountName instead.\n type: string\n serviceAccountName:\n description: |-\n ServiceAccountName is the name of the ServiceAccount to use to run this pod.\n More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\n type: string\n setHostnameAsFQDN:\n description: |-\n If true the pod's hostname will be configured as the pod's FQDN, rather than the leaf name (the default).\n In Linux containers, this means setting the FQDN in the hostname field of the kernel (the nodename field of struct utsname).\n In Windows containers, this means setting the registry value of hostname for the registry key HKEY_LOCAL_MACHINE\\\\SYSTEM\\\\CurrentControlSet\\\\Services\\\\Tcpip\\\\Parameters to FQDN.\n If a pod does not have FQDN, this has no effect.\n Default to false.\n type: boolean\n shareProcessNamespace:\n description: |-\n Share a single process namespace between all of the containers in a pod.\n When this is set containers will be able to view and signal processes from other containers\n in the same pod, and the first process in each container will not be assigned PID 1.\n HostPID and ShareProcessNamespace cannot both be set.\n Optional: Default to false.\n type: boolean\n subdomain:\n description: |-\n If specified, the fully qualified Pod hostname will be \"...svc.\".\n If not specified, the pod will not have a domainname at all.\n type: string\n terminationGracePeriodSeconds:\n description: |-\n Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request.\n Value must be non-negative integer. The value zero indicates stop immediately via\n the kill signal (no opportunity to shut down).\n If this value is nil, the default grace period will be used instead.\n The grace period is the duration in seconds after the processes running in the pod are sent\n a termination signal and the time when the processes are forcibly halted with a kill signal.\n Set this value longer than the expected cleanup time for your process.\n Defaults to 30 seconds.\n format: int64\n type: integer\n tolerations:\n description: If specified, the pod's tolerations.\n items:\n description: |-\n The pod this Toleration is attached to tolerates any taint that matches\n the triple using the matching operator .\n properties:\n effect:\n description: |-\n Effect indicates the taint effect to match. Empty means match all taint effects.\n When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.\n type: string\n key:\n description: |-\n Key is the taint key that the toleration applies to. Empty means match all taint keys.\n If the key is empty, operator must be Exists; this combination means to match all values and all keys.\n type: string\n operator:\n description: |-\n Operator represents a key's relationship to the value.\n Valid operators are Exists and Equal. Defaults to Equal.\n Exists is equivalent to wildcard for value, so that a pod can\n tolerate all taints of a particular category.\n type: string\n tolerationSeconds:\n description: |-\n TolerationSeconds represents the period of time the toleration (which must be\n of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default,\n it is not set, which means tolerate the taint forever (do not evict). Zero and\n negative values will be treated as 0 (evict immediately) by the system.\n format: int64\n type: integer\n value:\n description: |-\n Value is the taint value the toleration matches to.\n If the operator is Exists, the value should be empty, otherwise just a regular string.\n type: string\n type: object\n type: array\n topologySpreadConstraints:\n description: |-\n TopologySpreadConstraints describes how a group of pods ought to spread across topology\n domains. Scheduler will schedule pods in a way which abides by the constraints.\n All topologySpreadConstraints are ANDed.\n items:\n description: TopologySpreadConstraint specifies how to\n spread matching pods among the given topology.\n properties:\n labelSelector:\n description: |-\n LabelSelector is used to find matching pods.\n Pods that match this label selector are counted to determine the number of pods\n in their corresponding topology domain.\n properties:\n matchExpressions:\n description: matchExpressions is a list of label\n selector requirements. The requirements are\n ANDed.\n items:\n description: |-\n A label selector requirement is a selector that contains values, a key, and an operator that\n relates the key and values.\n properties:\n key:\n description: key is the label key that the\n selector applies to.\n type: string\n operator:\n description: |-\n operator represents a key's relationship to a set of values.\n Valid operators are In, NotIn, Exists and DoesNotExist.\n type: string\n values:\n description: |-\n values is an array of string values. If the operator is In or NotIn,\n the values array must be non-empty. If the operator is Exists or DoesNotExist,\n the values array must be empty. This array is replaced during a strategic\n merge patch.\n items:\n type: string\n type: array\n required:\n - key\n - operator\n type: object\n type: array\n matchLabels:\n additionalProperties:\n type: string\n description: |-\n matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels\n map is equivalent to an element of matchExpressions, whose key field is \"key\", the\n operator is \"In\", and the values array contains only \"value\". The requirements are ANDed.\n type: object\n type: object\n x-kubernetes-map-type: atomic\n matchLabelKeys:\n description: |-\n MatchLabelKeys is a set of pod label keys to select the pods over which\n spreading will be calculated. The keys are used to lookup values from the\n incoming pod labels, those key-value labels are ANDed with labelSelector\n to select the group of existing pods over which spreading will be calculated\n for the incoming pod. The same key is forbidden to exist in both MatchLabelKeys and LabelSelector.\n MatchLabelKeys cannot be set when LabelSelector isn't set.\n Keys that don't exist in the incoming pod labels will\n be ignored. A null or empty list means only match against labelSelector.\n\n\n This is a beta field and requires the MatchLabelKeysInPodTopologySpread feature gate to be enabled (enabled by default).\n items:\n type: string\n type: array\n x-kubernetes-list-type: atomic\n maxSkew:\n description: |-\n MaxSkew describes the degree to which pods may be unevenly distributed.\n When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference\n between the number of matching pods in the target topology and the global minimum.\n The global minimum is the minimum number of matching pods in an eligible domain\n or zero if the number of eligible domains is less than MinDomains.\n For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same\n labelSelector spread as 2/2/1:\n In this case, the global minimum is 1.\n | zone1 | zone2 | zone3 |\n | P P | P P | P |\n - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2;\n scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2)\n violate MaxSkew(1).\n - if MaxSkew is 2, incoming pod can be scheduled onto any zone.\n When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence\n to topologies that satisfy it.\n It's a required field. Default value is 1 and 0 is not allowed.\n format: int32\n type: integer\n minDomains:\n description: |-\n MinDomains indicates a minimum number of eligible domains.\n When the number of eligible domains with matching topology keys is less than minDomains,\n Pod Topology Spread treats \"global minimum\" as 0, and then the calculation of Skew is performed.\n And when the number of eligible domains with matching topology keys equals or greater than minDomains,\n this value has no effect on scheduling.\n As a result, when the number of eligible domains is less than minDomains,\n scheduler won't schedule more than maxSkew Pods to those domains.\n If value is nil, the constraint behaves as if MinDomains is equal to 1.\n Valid values are integers greater than 0.\n When value is not nil, WhenUnsatisfiable must be DoNotSchedule.\n\n\n For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same\n labelSelector spread as 2/2/2:\n | zone1 | zone2 | zone3 |\n | P P | P P | P P |\n The number of domains is less than 5(MinDomains), so \"global minimum\" is treated as 0.\n In this situation, new pod with the same labelSelector cannot be scheduled,\n because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones,\n it will violate MaxSkew.\n\n\n This is a beta field and requires the MinDomainsInPodTopologySpread feature gate to be enabled (enabled by default).\n format: int32\n type: integer\n nodeAffinityPolicy:\n description: |-\n NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector\n when calculating pod topology spread skew. Options are:\n - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations.\n - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations.\n\n\n If this value is nil, the behavior is equivalent to the Honor policy.\n This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag.\n type: string\n nodeTaintsPolicy:\n description: |-\n NodeTaintsPolicy indicates how we will treat node taints when calculating\n pod topology spread skew. Options are:\n - Honor: nodes without taints, along with tainted nodes for which the incoming pod\n has a toleration, are included.\n - Ignore: node taints are ignored. All nodes are included.\n\n\n If this value is nil, the behavior is equivalent to the Ignore policy.\n This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag.\n type: string\n topologyKey:\n description: |-\n TopologyKey is the key of node labels. Nodes that have a label with this key\n and identical values are considered to be in the same topology.\n We consider each as a \"bucket\", and try to put balanced number\n of pods into each bucket.\n We define a domain as a particular instance of a topology.\n Also, we define an eligible domain as a domain whose nodes meet the requirements of\n nodeAffinityPolicy and nodeTaintsPolicy.\n e.g. If TopologyKey is \"kubernetes.io/hostname\", each Node is a domain of that topology.\n And, if TopologyKey is \"topology.kubernetes.io/zone\", each zone is a domain of that topology.\n It's a required field.\n type: string\n whenUnsatisfiable:\n description: |-\n WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy\n the spread constraint.\n - DoNotSchedule (default) tells the scheduler not to schedule it.\n - ScheduleAnyway tells the scheduler to schedule the pod in any location,\n but giving higher precedence to topologies that would help reduce the\n skew.\n A constraint is considered \"Unsatisfiable\" for an incoming pod\n if and only if every possible node assignment for that pod would violate\n \"MaxSkew\" on some topology.\n For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same\n labelSelector spread as 3/1/1:\n | zone1 | zone2 | zone3 |\n | P P P | P | P |\n If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled\n to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies\n MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler\n won't make it *more* imbalanced.\n It's a required field.\n type: string\n required:\n - maxSkew\n - topologyKey\n - whenUnsatisfiable\n type: object\n type: array\n x-kubernetes-list-map-keys:\n - topologyKey\n - whenUnsatisfiable\n x-kubernetes-list-type: map\n volumes:\n description: |-\n List of volumes that can be mounted by containers belonging to the pod.\n More info: https://kubernetes.io/docs/concepts/storage/volumes\n items:\n description: Volume represents a named volume in a pod\n that may be accessed by any container in the pod.\n properties:\n awsElasticBlockStore:\n description: |-\n awsElasticBlockStore represents an AWS Disk resource that is attached to a\n kubelet's host machine and then exposed to the pod.\n More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore\n properties:\n fsType:\n description: |-\n fsType is the filesystem type of the volume that you want to mount.\n Tip: Ensure that the filesystem type is supported by the host operating system.\n Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.\n More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore\n TODO: how do we prevent errors in the filesystem from compromising the machine\n type: string\n partition:\n description: |-\n partition is the partition in the volume that you want to mount.\n If omitted, the default is to mount by volume name.\n Examples: For volume /dev/sda1, you specify the partition as \"1\".\n Similarly, the volume partition for /dev/sda is \"0\" (or you can leave the property empty).\n format: int32\n type: integer\n readOnly:\n description: |-\n readOnly value true will force the readOnly setting in VolumeMounts.\n More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore\n type: boolean\n volumeID:\n description: |-\n volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume).\n More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore\n type: string\n required:\n - volumeID\n type: object\n azureDisk:\n description: azureDisk represents an Azure Data Disk\n mount on the host and bind mount to the pod.\n properties:\n cachingMode:\n description: 'cachingMode is the Host Caching\n mode: None, Read Only, Read Write.'\n type: string\n diskName:\n description: diskName is the Name of the data\n disk in the blob storage\n type: string\n diskURI:\n description: diskURI is the URI of data disk in\n the blob storage\n type: string\n fsType:\n description: |-\n fsType is Filesystem type to mount.\n Must be a filesystem type supported by the host operating system.\n Ex. \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.\n type: string\n kind:\n description: 'kind expected values are Shared:\n multiple blob disks per storage account Dedicated:\n single blob disk per storage account Managed:\n azure managed data disk (only in managed availability\n set). defaults to shared'\n type: string\n readOnly:\n description: |-\n readOnly Defaults to false (read/write). ReadOnly here will force\n the ReadOnly setting in VolumeMounts.\n type: boolean\n required:\n - diskName\n - diskURI\n type: object\n azureFile:\n description: azureFile represents an Azure File Service\n mount on the host and bind mount to the pod.\n properties:\n readOnly:\n description: |-\n readOnly defaults to false (read/write). ReadOnly here will force\n the ReadOnly setting in VolumeMounts.\n type: boolean\n secretName:\n description: secretName is the name of secret\n that contains Azure Storage Account Name and\n Key\n type: string\n shareName:\n description: shareName is the azure share Name\n type: string\n required:\n - secretName\n - shareName\n type: object\n cephfs:\n description: cephFS represents a Ceph FS mount on\n the host that shares a pod's lifetime\n properties:\n monitors:\n description: |-\n monitors is Required: Monitors is a collection of Ceph monitors\n More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it\n items:\n type: string\n type: array\n path:\n description: 'path is Optional: Used as the mounted\n root, rather than the full Ceph tree, default\n is /'\n type: string\n readOnly:\n description: |-\n readOnly is Optional: Defaults to false (read/write). ReadOnly here will force\n the ReadOnly setting in VolumeMounts.\n More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it\n type: boolean\n secretFile:\n description: |-\n secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret\n More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it\n type: string\n secretRef:\n description: |-\n secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty.\n More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n type: object\n x-kubernetes-map-type: atomic\n user:\n description: |-\n user is optional: User is the rados user name, default is admin\n More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it\n type: string\n required:\n - monitors\n type: object\n cinder:\n description: |-\n cinder represents a cinder volume attached and mounted on kubelets host machine.\n More info: https://examples.k8s.io/mysql-cinder-pd/README.md\n properties:\n fsType:\n description: |-\n fsType is the filesystem type to mount.\n Must be a filesystem type supported by the host operating system.\n Examples: \"ext4\", \"xfs\", \"ntfs\". Implicitly inferred to be \"ext4\" if unspecified.\n More info: https://examples.k8s.io/mysql-cinder-pd/README.md\n type: string\n readOnly:\n description: |-\n readOnly defaults to false (read/write). ReadOnly here will force\n the ReadOnly setting in VolumeMounts.\n More info: https://examples.k8s.io/mysql-cinder-pd/README.md\n type: boolean\n secretRef:\n description: |-\n secretRef is optional: points to a secret object containing parameters used to connect\n to OpenStack.\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n type: object\n x-kubernetes-map-type: atomic\n volumeID:\n description: |-\n volumeID used to identify the volume in cinder.\n More info: https://examples.k8s.io/mysql-cinder-pd/README.md\n type: string\n required:\n - volumeID\n type: object\n configMap:\n description: configMap represents a configMap that\n should populate this volume\n properties:\n defaultMode:\n description: |-\n defaultMode is optional: mode bits used to set permissions on created files by default.\n Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.\n YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.\n Defaults to 0644.\n Directories within the path are not affected by this setting.\n This might be in conflict with other options that affect the file\n mode, like fsGroup, and the result can be other mode bits set.\n format: int32\n type: integer\n items:\n description: |-\n items if unspecified, each key-value pair in the Data field of the referenced\n ConfigMap will be projected into the volume as a file whose name is the\n key and content is the value. If specified, the listed keys will be\n projected into the specified paths, and unlisted keys will not be\n present. If a key is specified which is not present in the ConfigMap,\n the volume setup will error unless it is marked optional. Paths must be\n relative and may not contain the '..' path or start with '..'.\n items:\n description: Maps a string key to a path within\n a volume.\n properties:\n key:\n description: key is the key to project.\n type: string\n mode:\n description: |-\n mode is Optional: mode bits used to set permissions on this file.\n Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.\n YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.\n If not specified, the volume defaultMode will be used.\n This might be in conflict with other options that affect the file\n mode, like fsGroup, and the result can be other mode bits set.\n format: int32\n type: integer\n path:\n description: |-\n path is the relative path of the file to map the key to.\n May not be an absolute path.\n May not contain the path element '..'.\n May not start with the string '..'.\n type: string\n required:\n - key\n - path\n type: object\n type: array\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n optional:\n description: optional specify whether the ConfigMap\n or its keys must be defined\n type: boolean\n type: object\n x-kubernetes-map-type: atomic\n csi:\n description: csi (Container Storage Interface) represents\n ephemeral storage that is handled by certain external\n CSI drivers (Beta feature).\n properties:\n driver:\n description: |-\n driver is the name of the CSI driver that handles this volume.\n Consult with your admin for the correct name as registered in the cluster.\n type: string\n fsType:\n description: |-\n fsType to mount. Ex. \"ext4\", \"xfs\", \"ntfs\".\n If not provided, the empty value is passed to the associated CSI driver\n which will determine the default filesystem to apply.\n type: string\n nodePublishSecretRef:\n description: |-\n nodePublishSecretRef is a reference to the secret object containing\n sensitive information to pass to the CSI driver to complete the CSI\n NodePublishVolume and NodeUnpublishVolume calls.\n This field is optional, and may be empty if no secret is required. If the\n secret object contains more than one secret, all secret references are passed.\n properties:\n name:\n description: |-\n Name of the referent.\n More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names\n TODO: Add other useful fields. apiVersion, kind, uid?\n type: string\n type: object\n x-kubernetes-map-type: atomic\n readOnly:\n description: |-\n readOnly specifies a read-only configuration for the volume.\n Defaults to false (read/write).\n type: boolean\n volumeAttributes:\n additionalProperties:\n type: string\n description: |-\n volumeAttributes stores driver-specific properties that are passed to the CSI\n driver. Consult your driver's documentation for supported values.\n type: object\n required:\n - driver\n type: object\n downwardAPI:\n description: downwardAPI represents downward API about\n the pod that should populate this volume\n properties:\n defaultMode:\n description: |-\n Optional: mode bits to use on created files by default. Must be a\n Optional: mode bits used to set permissions on created files by default.\n Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.\n YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.\n Defaults to 0644.\n Directories within the path are not affected by this setting.\n This might be in conflict with other options that affect the file\n mode, like fsGroup, and the result can be other mode bits set.\n format: int32\n type: integer\n items:\n description: Items is a list of downward API volume\n file\n items:\n description: DownwardAPIVolumeFile represents\n information to create the file containing\n the pod field\n properties:\n fieldRef:\n description: 'Required: Selects a field\n of the pod: only annotations, labels,\n name and namespace are supported.'\n properties:\n apiVersion:\n description: Version of the schema the\n FieldPath is written in terms of,\n defaults to \"v1\".\n type: string\n fieldPath:\n description: Path of the field to select\n in the specified API version.\n type: string\n required:\n - fieldPath\n type: object\n x-kubernetes-map-type: atomic\n mode:\n description: |-\n Optional: mode bits used to set permissions on this file, must be an octal value\n between 0000 and 0777 or a decimal value between 0 and 511.\n YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.\n If not specified, the volume defaultMode will be used.\n This might be in conflict with other options that affect the file\n mode, like fsGroup, and the result can be other mode bits set.\n format: int32\n type: integer\n path:\n description: 'Required: Path is the relative\n path name of the file to be created. Must\n not be absolute or contain the ''..''\n path. Must be utf-8 encoded. The first\n item of the relative path must not start\n with ''..'''\n type: string\n resourceFieldRef:\n description: |-\n Selects a resource of the container: only resources limits and requests\n (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported.\n properties:\n containerName:\n description: 'Container name: required\n for volumes, optional for env vars'\n type: string\n divisor:\n anyOf:\n - type: integer\n - type: string\n description: Specifies the output format\n of the exposed resources, defaults\n to \"1\"\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n resource:\n description: 'Required: resource to\n select'\n type: string\n required:\n - resource\n type: object\n x-kubernetes-map-type: atomic\n required:\n - path\n type: object\n type: array\n type: object\n emptyDir:\n description: |-\n emptyDir represents a temporary directory that shares a pod's lifetime.\n More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir\n properties:\n medium:\n description: |-\n medium represents what type of storage medium should back this directory.\n The default is \"\" which means to use the node's default medium.\n Must be an empty string (default) or Memory.\n More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir\n type: string\n sizeLimit:\n anyOf:\n - type: integer\n - type: string\n description: |-\n sizeLimit is the total amount of local storage required for this EmptyDir volume.\n The size limit is also applicable for memory medium.\n The maximum usage on memory medium EmptyDir would be the minimum value between\n the SizeLimit specified here and the sum of memory limits of all containers in a pod.\n The default is nil which means that the limit is undefined.\n More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir\n pattern: ^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$\n x-kubernetes-int-or-string: true\n type: object\n ephemeral:\n description: |-\n ephemeral represents a volume that is handled by a cluster storage driver.\n The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts,\n and deleted when the pod is removed.\n\n\n Use this if:\n a) the volume is only needed while the pod runs,\n b) features of normal volumes like restoring from snapshot or capacity\n tracking are needed,\n c) the storage driver is specified through a storage class, and\n d) the storage driver supports dynamic volume provisioning through\n a PersistentVolumeClaim (see EphemeralVolumeSource for more\n information on the connection between this volume type\n and PersistentVolumeClaim).\n\n\n Use PersistentVolumeClaim or one of the vendor-specific\n APIs for volumes that persist for longer than the lifecycle\n of an individual pod.\n\n\n Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to\n be used that way - see the documentation of the driver for\n more information.\n\n\n A pod can use both types of ephemeral volumes and\n persistent volumes at the same time.\n properties:\n volumeClaimTemplate:\n description: |-\n Will be used to create a stand-alone PVC to provision the volume.\n The pod in which this EphemeralVolumeSource is embedded will be the\n owner of the PVC, i.e. the PVC will be deleted together with the\n pod. The name of the PVC will be `- ![\"conformance-icon\"](\"https://www.cncf.io/wp-content/uploads/2020/07/certified_kubernetes_color-1.png\")
![\"kubeasz-logo\"](\"docs/pics/kubeasz.svg\")
![\"Kubernetes教程_Kuboard_Github_Star\"](\"https://starchart.cc/eip-work/kuboard-press.svg\")
\n\n\n点击这里可以查看 [Kuboard 的安装文档](https://kuboard.cn/install/v3/install.html)\n\n## 在线演示\n\n![\"ha-3x\"](\"../pics/ha-3x.svg\")
\n\n- 注意1：确保各节点时区设置一致、时间同步。 如果你的环境没有提供NTP 时间同步，推荐集成安装[chrony](../guide/chrony.md)\n- 注意2：确保在干净的系统上开始安装，不要使用曾经装过kubeadm或其他k8s发行版的环境\n- 注意3：建议操作系统升级到新的稳定内核，请结合阅读[内核升级文档](../guide/kernel_upgrade.md)\n- 注意4：在公有云上创建多主集群，请结合阅读[在公有云上部署 kubeasz](kubeasz_on_public_cloud.md)\n\n## 高可用集群所需节点配置如下\n\n|角色|数量|描述|\n|:-|:-|:-|\n|部署节点|1|运行ansible/ezctl命令，一般复用第一个master节点|\n|etcd节点|3|注意etcd集群需要1,3,5,...奇数个节点，一般复用master节点|\n|master节点|2|高可用集群至少2个master节点|\n|node节点|n|运行应用负载的节点，可根据需要提升机器配置/增加节点数|\n\n机器配置：\n- master节点：4c/8g内存/50g硬盘\n- worker节点：建议8c/32g内存/200g硬盘以上\n\n注意：默认配置下容器运行时和kubelet会占用/var的磁盘空间，如果磁盘分区特殊，可以设置config.yml中的容器运行时和kubelet数据目录：`CONTAINERD_STORAGE_DIR` `DOCKER_STORAGE_DIR` `KUBELET_ROOT_DIR`\n\n在 kubeasz 2x 版本，多节点高可用集群安装可以使用2种方式\n\n- 1.按照本文步骤先规划准备，预先配置节点信息后，直接安装多节点高可用集群\n- 2.先部署单节点集群 [AllinOne部署](quickStart.md)，然后通过 [节点添加](../op/op-index.md) 扩容成高可用集群\n\n## 部署步骤\n\n以下示例创建一个4节点的多主高可用集群，文档中命令默认都需要root权限运行。\n\n### 1.基础系统配置\n\n+ 2c/4g内存/40g硬盘（该配置仅测试用）\n+ 最小化安装`Ubuntu 16.04 server`或者`CentOS 7 Minimal`\n+ 配置基础网络、更新源、SSH登录等\n\n### 2.在每个节点安装依赖工具\n\n推荐使用ansible in docker 容器化方式运行，无需安装额外依赖。\n\n### 3.准备ssh免密登陆\n\n配置从部署节点能够ssh免密登陆所有节点，并且设置python软连接\n\n``` bash\n#$IP为所有节点地址包括自身，按照提示输入yes 和root密码\nssh-copy-id $IP \n```\n\n### 4.在部署节点编排k8s安装\n\n- 4.1 下载项目源码、二进制及离线镜像\n\n下载工具脚本ezdown，举例使用kubeasz版本3.5.0\n\n``` bash\nexport release=3.5.0\nwget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown\nchmod +x ./ezdown\n```\n\n下载kubeasz代码、二进制、默认容器镜像（更多关于ezdown的参数，运行./ezdown 查看）\n\n``` bash\n# 国内环境\n./ezdown -D\n# 海外环境\n#./ezdown -D -m standard\n```\n\n【可选】下载额外容器镜像（cilium,flannel,prometheus等）\n\n``` bash\n# 按需下载\n./ezdown -X flannel\n./ezdown -X prometheus\n...\n```\n\n【可选】下载离线系统包 (适用于无法使用yum/apt仓库情形)\n\n``` bash\n./ezdown -P\n```\n\n上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录`/etc/kubeasz`\n\n- 4.2 创建集群配置实例\n\n``` bash\n# 容器化运行kubeasz\n./ezdown -S\n\n# 创建新集群 k8s-01\ndocker exec -it kubeasz ezctl new k8s-01\n2021-01-19 10:48:23 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-01\n2021-01-19 10:48:23 DEBUG set version of common plugins\n2021-01-19 10:48:23 DEBUG cluster k8s-01: files successfully created.\n2021-01-19 10:48:23 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-01/hosts'\n2021-01-19 10:48:23 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-01/config.yml'\n```\n然后根据提示配置'/etc/kubeasz/clusters/k8s-01/hosts' 和 '/etc/kubeasz/clusters/k8s-01/config.yml'：根据前面节点规划修改hosts 文件和其他集群层面的主要配置选项；其他集群组件等配置项可以在config.yml 文件中修改。\n\n- 4.3 开始安装\n如果你对集群安装流程不熟悉，请阅读项目首页 **安装步骤** 讲解后分步安装，并对 **每步都进行验证** \n\n``` bash\n#建议使用alias命令，查看~/.bashrc 文件应该包含：alias dk='docker exec -it kubeasz'\nsource ~/.bashrc\n\n# 一键安装，等价于执行docker exec -it kubeasz ezctl setup k8s-01 all\ndk ezctl setup k8s-01 all\n\n# 或者分步安装，具体使用 dk ezctl help setup 查看分步安装帮助信息\n# dk ezctl setup k8s-01 01\n# dk ezctl setup k8s-01 02\n# dk ezctl setup k8s-01 03\n# dk ezctl setup k8s-01 04\n...\n```\n\n更多ezctl使用帮助，请参考[这里](ezctl.md)\n\n[后一篇](01-CA_and_prerequisite.md)\n"
},
{
"path": "docs/setup/01-CA_and_prerequisite.md",
"content": "# 01-创建证书和环境准备\n\n本步骤主要完成: \n\n- (deprecated) role:os-harden，（未更新上游项目，未验证最新k8s集群安装，不建议启用）可选系统加固，符合linux安全基线，详见[upstream](https://github.com/dev-sec/ansible-collection-hardening/tree/master/roles/os_hardening)\n- (optional) role:chrony，[可选集群节点时间同步](../guide/chrony.md)\n- role:deploy，创建CA证书、集群组件访问apiserver所需的各种kubeconfig\n- role:prepare，系统基础环境配置、分发CA证书、kubectl客户端安装\n\n## deploy 角色\n\n主要任务讲解：roles/deploy/tasks/main.yml\n\n### 创建 CA 证书\n\nkubernetes 系统各组件需要使用 TLS 证书对通信进行加密，使用 CloudFlare 的 PKI 工具集生成自签名的 CA 证书，用来签名后续创建的其它 TLS 证书。[参考阅读](https://coreos.com/os/docs/latest/generate-self-signed-certificates.html)\n\n根据认证对象可以将证书分成三类：服务器证书`server cert`，客户端证书`client cert`，对等证书`peer cert`(既是`server cert`又是`client cert`)，在kubernetes 集群中需要的证书种类如下：\n\n+ `etcd` 节点需要标识自己服务的`server cert`，也需要`client cert`与`etcd`集群其他节点交互，当然可以分别指定2个证书，为方便这里使用一个对等证书\n+ `master` 节点需要标识 apiserver服务的`server cert`，也需要`client cert`连接`etcd`集群，这里也使用一个对等证书\n+ `kubectl` `calico` `kube-proxy` 只需要`client cert`，因此证书请求中 `hosts` 字段可以为空\n+ `kubelet` 需要标识自己服务的`server cert`，也需要`client cert`请求`apiserver`，也使用一个对等证书\n\n整个集群要使用统一的CA 证书，只需要在ansible控制端创建，然后分发给其他节点；为了保证安装的幂等性，如果已经存在CA 证书，就跳过创建CA 步骤\n\n#### 创建 CA 配置文件 [ca-config.json.j2](../../roles/deploy/templates/ca-config.json.j2)\n``` bash\n{\n \"signing\": {\n \"default\": {\n \"expiry\": \"{{ CERT_EXPIRY }}\"\n },\n \"profiles\": {\n \"kubernetes\": {\n \"usages\": [\n \"signing\",\n \"key encipherment\",\n \"server auth\",\n \"client auth\"\n ],\n \"expiry\": \"{{ CERT_EXPIRY }}\"\n },\n \"kcfg\": {\n \"usages\": [\n \"signing\",\n \"key encipherment\",\n \"client auth\"\n ],\n \"expiry\": \"{{ CUSTOM_EXPIRY }}\"\n }\n }\n }\n}\n```\n+ `signing`：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 `CA=TRUE`；\n+ `server auth`：表示可以用该 CA 对 server 提供的证书进行验证；\n+ `client auth`：表示可以用该 CA 对 client 提供的证书进行验证；\n+ `profile kubernetes` 包含了`server auth`和`client auth`，所以可以签发三种不同类型证书；expiry 证书有效期，默认50年\n+ `profile kcfg` 在后面客户端kubeconfig证书管理中用到\n\n#### 创建 CA 证书签名请求 [ca-csr.json.j2](../../roles/deploy/templates/ca-csr.json.j2)\n``` bash\n{\n \"CN\": \"kubernetes-ca\",\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"k8s\",\n \"OU\": \"System\"\n }\n ],\n \"ca\": {\n \"expiry\": \"876000h\"\n }\n}\n```\n- `ca expiry` 指定ca证书的有效期，默认100年\n\n#### 生成CA 证书和私钥\n``` bash\ncfssl gencert -initca ca-csr.json | cfssljson -bare ca\n```\n\n### 生成 kubeconfig 配置文件\n\nkubectl使用~/.kube/config 配置文件与kube-apiserver进行交互，且拥有管理 K8S集群的完全权限，\n\n准备kubectl使用的admin 证书签名请求 [admin-csr.json.j2](../../roles/deploy/templates/admin-csr.json.j2)\n\n``` bash\n{\n \"CN\": \"admin\",\n \"hosts\": [],\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"system:masters\",\n \"OU\": \"System\"\n }\n ]\n}\n\n```\n+ kubectl 使用客户端证书可以不指定hosts 字段\n+ 证书请求中 `O` 指定该证书的 Group 为 `system:masters`，而 `RBAC` 预定义的 `ClusterRoleBinding` 将 Group `system:masters` 与 ClusterRole `cluster-admin` 绑定，这就赋予了kubectl**所有集群权限**\n\n``` bash\n$ kubectl describe clusterrolebinding cluster-admin\nName: cluster-admin\nLabels: kubernetes.io/bootstrapping=rbac-defaults\nAnnotations: rbac.authorization.kubernetes.io/autoupdate=true\nRole:\n Kind: ClusterRole\n Name: cluster-admin\nSubjects:\n Kind Name Namespace\n ---- ---- ---------\n Group system:masters \n```\n\n#### 生成 admin 用户证书\n\n```\ncfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin\n```\n\n#### 生成 ~/.kube/config 配置文件\n\n使用`kubectl config` 生成kubeconfig 自动保存到 ~/.kube/config，生成后 `cat ~/.kube/config`可以验证配置文件包含 kube-apiserver 地址、证书、用户名等信息。\n\n```\nkubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=127.0.0.1:8443\nkubectl config set-credentials admin --client-certificate=admin.pem --embed-certs=true --client-key=admin-key.pem\nkubectl config set-context kubernetes --cluster=kubernetes --user=admin\nkubectl config use-context kubernetes\n```\n\n### 生成 kube-proxy.kubeconfig 配置文件\n\n创建 kube-proxy 证书请求\n\n``` bash\n{\n \"CN\": \"system:kube-proxy\",\n \"hosts\": [],\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"k8s\",\n \"OU\": \"System\"\n }\n ]\n}\n```\n+ kube-proxy 使用客户端证书可以不指定hosts 字段\n+ CN 指定该证书的 User 为 system:kube-proxy，预定义的 ClusterRoleBinding system:node-proxier 将User system:kube-proxy 与 Role system:node-proxier 绑定，授予了调用 kube-apiserver Proxy 相关 API 的权限；\n\n``` bash\n$ kubectl describe clusterrolebinding system:node-proxier\nName: system:node-proxier\nLabels: kubernetes.io/bootstrapping=rbac-defaults\nAnnotations: rbac.authorization.kubernetes.io/autoupdate=true\nRole:\n Kind: ClusterRole\n Name: system:node-proxier\nSubjects:\n Kind Name Namespace\n ---- ---- ---------\n User system:kube-proxy \n```\n\n#### 生成 system:kube-proxy 用户证书\n\n```\ncfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy\n```\n\n#### 生成 kube-proxy.kubeconfig\n\n使用`kubectl config` 生成kubeconfig 自动保存到 kube-proxy.kubeconfig\n\n```\nkubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=127.0.0.1:8443 --kubeconfig=kube-proxy.kubeconfig\nkubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --embed-certs=true --client-key=kube-proxy-key.pem --kubeconfig=kube-proxy.kubeconfig\nkubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig\nkubectl config use-context default --kubeconfig=kube-proxy.kubeconfig\n```\n\n### 创建kube-controller-manager 和 kube-scheduler 组件的kubeconfig 文件\n\n过程与创建kube-proxy.kubeconfig 类似，略。\n\n## prepare 角色\n\n请在另外窗口打开[roles/prepare/tasks/main.yml](../../roles/prepare/tasks/main.yml) 文件，比较简单直观\n\n1. 设置基础操作系统软件和系统参数，请阅读脚本中的注释内容\n1. 创建一些基础文件目录、环境变量以及添加本地镜像仓库`easzlab.io.local`的域名解析\n1. 分发kubeconfig等配置文件\n\n\n[后一篇](02-install_etcd.md)\n"
},
{
"path": "docs/setup/02-install_etcd.md",
"content": "## 02-安装etcd集群\n\nkuberntes 集群使用 etcd 存储所有数据，是最重要的组件之一，注意 etcd集群需要奇数个节点(1,3,5...)，本文档使用3个节点做集群。\n\n请在另外窗口打开[roles/etcd/tasks/main.yml](../../roles/etcd/tasks/main.yml) 文件，对照看以下讲解内容。\n\n### 创建etcd证书\n\n注意：证书是在部署节点创建好之后推送到目标etcd节点上去的，以增加ca证书的安全性\n\n创建ectd证书请求 [etcd-csr.json.j2](../../roles/etcd/templates/etcd-csr.json.j2)\n\n``` bash\n{\n \"CN\": \"etcd\",\n \"hosts\": [\n{% for host in groups['etcd'] %}\n \"{{ host }}\",\n{% endfor %}\n \"127.0.0.1\"\n ],\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"k8s\",\n \"OU\": \"System\"\n }\n ]\n}\n```\n+ etcd使用对等证书，hosts 字段必须指定授权使用该证书的 etcd 节点 IP，这里枚举了所有ectd节点的地址\n\n### 创建etcd 服务文件 [etcd.service.j2](../../roles/etcd/templates/etcd.service.j2)\n\n``` bash\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory={{ ETCD_DATA_DIR }}\nExecStart={{ bin_dir }}/etcd \\\n --name=etcd-{{ inventory_hostname }} \\\n --cert-file={{ ca_dir }}/etcd.pem \\\n --key-file={{ ca_dir }}/etcd-key.pem \\\n --peer-cert-file={{ ca_dir }}/etcd.pem \\\n --peer-key-file={{ ca_dir }}/etcd-key.pem \\\n --trusted-ca-file={{ ca_dir }}/ca.pem \\\n --peer-trusted-ca-file={{ ca_dir }}/ca.pem \\\n --initial-advertise-peer-urls=https://{{ inventory_hostname }}:2380 \\\n --listen-peer-urls=https://{{ inventory_hostname }}:2380 \\\n --listen-client-urls=https://{{ inventory_hostname }}:2379,http://127.0.0.1:2379 \\\n --advertise-client-urls=https://{{ inventory_hostname }}:2379 \\\n --initial-cluster-token=etcd-cluster-0 \\\n --initial-cluster={{ ETCD_NODES }} \\\n --initial-cluster-state={{ CLUSTER_STATE }} \\\n --data-dir={{ ETCD_DATA_DIR }} \\\n --wal-dir={{ ETCD_WAL_DIR }} \\\n --snapshot-count=50000 \\\n --auto-compaction-retention=1 \\\n --auto-compaction-mode=periodic \\\n --max-request-bytes=10485760 \\\n --quota-backend-bytes=8589934592\nRestart=always\nRestartSec=15\nLimitNOFILE=65536\nOOMScoreAdjust=-999\n\n[Install]\nWantedBy=multi-user.target\n```\n\n+ 完整参数列表请使用 `etcd --help` 查询\n+ 注意etcd 即需要服务器证书也需要客户端证书，为方便使用一个peer 证书代替两个证书\n+ `--initial-cluster-state` 值为 `new` 时，`--name` 的参数值必须位于 `--initial-cluster` 列表中\n+ `--snapshot-count` `--auto-compaction-retention` 一些性能优化参数，请查阅etcd项目文档\n+ 设置`--data-dir` 和`--wal-dir` 使用不同磁盘目录，可以避免磁盘io竞争，提高性能，具体请参考etcd项目文档\n\n### 验证etcd集群状态\n\n+ systemctl status etcd 查看服务状态\n+ journalctl -u etcd 查看运行日志\n+ 在任一 etcd 集群节点上执行如下命令\n\n``` bash\n# 根据hosts中配置设置shell变量 $NODE_IPS\nexport NODE_IPS=\"192.168.1.1 192.168.1.2 192.168.1.3\"\nfor ip in ${NODE_IPS}; do\n etcdctl \\\n --endpoints=https://${ip}:2379 \\\n --cacert=/etc/kubernetes/ssl/ca.pem \\\n --cert=/etc/kubernetes/ssl/etcd.pem \\\n --key=/etc/kubernetes/ssl/etcd-key.pem \\\n endpoint health; done\n\n# 预期结果\nhttps://192.168.1.1:2379 is healthy: successfully committed proposal: took = 2.210885ms\nhttps://192.168.1.2:2379 is healthy: successfully committed proposal: took = 2.784043ms\nhttps://192.168.1.3:2379 is healthy: successfully committed proposal: took = 3.275709ms\n\nfor ip in ${NODE_IPS}; do\n etcdctl \\\n --endpoints=https://${ip}:2379 \\\n --cacert=/etc/kubernetes/ssl/ca.pem \\\n --cert=/etc/kubernetes/ssl/etcd.pem \\\n --key=/etc/kubernetes/ssl/etcd-key.pem \\\n --write-out=table endpoint status; done\n\n# 预期结果\n+----------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |\n+----------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n| https://192.168.1.1:2379 | 5f64925bd78a482c | 3.6.4 | 3.6.0 | 38 MB | 28 MB | 28% | 8.6 GB | true | false | 269 | 6582307 | 6582307 | | | false |\n+----------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n+----------------------------+-----------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |\n+----------------------------+-----------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n| https://192.168.1.2:2379 | 18e1b1602639adb | 3.6.4 | 3.6.0 | 37 MB | 28 MB | 25% | 8.6 GB | false | false | 269 | 6582307 | 6582307 | | | false |\n+----------------------------+-----------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n+----------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |\n+----------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n| https://192.168.1.3:2379 | 3d375f7546465b4e | 3.6.4 | 3.6.0 | 37 MB | 28 MB | 26% | 8.6 GB | false | false | 269 | 6582308 | 6582308 | | | false |\n+----------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+\n```\n\n- 所有节点可达：etcdctl endpoint health 对所有三个节点都返回 healthy。\n- 有且仅有一个领导者：etcdctl endpoint status 显示一个节点 is leader: true，另外两个节点 is leader: false。\n- Raft 任期一致：所有三个节点的 raft term 值完全相同。\n- Raft 索引同步：所有节点的 raft index 值相差不大（跟随者与领导者的差距在可接受范围内）。\n- 无活跃告警：etcdctl alarm list 返回空。\n- 节点间网络稳定：没有频繁的领导者切换（通过监控 etcd_server_leader_changes_seen_total 指标）。\n- 磁盘空间充足：没有 NOSPACE 告警，且磁盘使用率在安全阈值内（例如低于80%）。\n\n### 磁盘性能\n\n快速的磁盘是 etcd 部署性能和稳定性的最关键因素。\n\n磁盘速度慢会增加 etcd 请求延迟，并可能损害集群稳定性。由于 etcd 的共识协议依赖于将元数据持久地存储到日志中，因此大多数 etcd 集群成员必须将每个请求写入磁盘。此外，etcd 还会逐步将其状态检查点写入磁盘，以便截断此日志。如果这些写入耗时过长，心跳可能会超时并触发选举，从而损害集群的稳定性。通常，要判断磁盘速度是否足以满足 etcd 的要求，可以使用fio等基准测试工具。\n\netcd 对磁盘写入延迟非常敏感。通常需要 50 的顺序 IOPS（例如，7200 RPM 磁盘）。对于负载较重的集群，建议使用 500 的顺序 IOPS（例如，典型的本地 SSD 或高性能虚拟化块设备）。请注意，大多数云提供商发布的是并发 IOPS，而不是顺序 IOPS；发布的并发 IOPS 可能比顺序 IOPS 高出 10 倍。要测量实际的顺序 IOPS，我们建议使用磁盘基准测试工具，例如diskbench或fio。\n\n``` bash\n# 测试示例\nmkdir test-data\nfio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=2200m --bs=2300 --name=mytest\n```\n\n\n\n[后一篇](03-container_runtime.md)\n"
},
{
"path": "docs/setup/03-container_runtime.md",
"content": "# 03-安装容器运行时\n\n项目根据k8s版本提供不同的默认容器运行时：\n\n- k8s 版本 < 1.24 时，支持docker containerd 可选\n- k8s 版本 >= 1.24 时，仅支持 containerd\n\n## 安装containerd\n\n作为 CNCF 毕业项目，containerd 致力于提供简洁、可靠、可扩展的容器运行时；它被设计用来集成到 kubernetes 等系统使用，而不是像 docker 那样独立使用。\n\n- 安装指南 https://github.com/containerd/cri/blob/master/docs/installation.md\n- 客户端 circtl 使用指南 https://github.com/containerd/cri/blob/master/docs/crictl.md\n- man 文档 https://github.com/containerd/containerd/tree/master/docs/man\n\n## kubeasz 集成安装 containerd\n\n- 注意：k8s 1.24以后，项目已经设置默认容器运行时为 containerd，无需手动修改\n- 执行安装：分步安装`ezctl setup xxxx 03`，一键安装`ezctl setup xxxx all`\n\n## 命令对比\n\n|命令 |docker |crictl（推荐） |ctr |\n|:- |:- |:- |:- |\n|查看容器列表 |docker ps |crictl ps |ctr -n k8s.io c ls |\n|查看容器详情 |docker inspect |crictl inspect |ctr -n k8s.io c info |\n|查看容器日志 |docker logs |crictl logs |无 |\n|容器内执行命令 |docker exec |crictl exec |无 |\n|挂载容器 |docker attach |crictl attach |无 |\n|容器资源使用 |docker stats |crictl stats |无 |\n|创建容器 |docker create |crictl create |ctr -n k8s.io c create |\n|启动容器 |docker start |crictl start |ctr -n k8s.io run |\n|停止容器 |docker stop |crictl stop |无 |\n|删除容器 |docker rm |crictl rm |ctr -n k8s.io c del |\n|查看镜像列表 |docker images |crictl images |ctr -n k8s.io i ls |\n|查看镜像详情 |docker inspect |crictl inspecti|无 |\n|拉取镜像 |docker pull |crictl pull |ctr -n k8s.io i pull |\n|推送镜像 |docker push |无 |ctr -n k8s.io i push |\n|删除镜像 |docker rmi |crictl rmi |ctr -n k8s.io i rm |\n|查看Pod列表 |无 |crictl pods |无 |\n|查看Pod详情 |无 |crictl inspectp|无 |\n|启动Pod |无 |crictl runp |无 |\n|停止Pod |无 |crictl stopp |无 |\n\n\n[后一篇](04-install_kube_master.md)\n"
},
{
"path": "docs/setup/04-install_kube_master.md",
"content": "# 04-安装kube_master节点\n\n部署master节点主要包含三个组件`apiserver` `scheduler` `controller-manager`，其中：\n\n- apiserver提供集群管理的REST API接口，包括认证授权、数据校验以及集群状态变更等\n - 只有API Server才直接操作etcd\n - 其他模块通过API Server查询或修改数据\n - 提供其他模块之间的数据交互和通信的枢纽\n- scheduler负责分配调度Pod到集群内的node节点\n - 监听kube-apiserver，查询还未分配Node的Pod\n - 根据调度策略为这些Pod分配节点\n- controller-manager由一系列的控制器组成，它通过apiserver监控整个集群的状态，并确保集群处于预期的工作状态\n\n## 高可用机制\n\n- apiserver 无状态服务，可以通过外部负载均衡实现高可用，如项目采用的两种高可用架构：HA-1x (#584)和 HA-2x (#585)\n- controller-manager 组件启动时会进行类似选举（leader）；当多副本存在时，如果原先leader挂掉，那么会选举出新的leader，从而保证高可用；\n- scheduler 类似选举机制\n\n## 安装流程\n\n``` bash\ncat playbooks/04.kube-master.yml\n- hosts: kube_master\n roles:\n - kube-lb # 四层负载均衡，监听在127.0.0.1:6443，转发到真实master节点apiserver服务\n - kube-master #\n - kube-node # 因为网络、监控等daemonset组件，master节点也推荐安装kubelet和kube-proxy服务\n ... \n```\n\n### 创建 kubernetes 证书签名请求\n\n``` bash\n{\n \"CN\": \"kubernetes\",\n \"hosts\": [\n \"127.0.0.1\",\n{% if groups['ex_lb']|length > 0 %}\n \"{{ hostvars[groups['ex_lb'][0]]['EX_APISERVER_VIP'] }}\",\n{% endif %}\n{% for host in groups['kube_master'] %}\n \"{{ host }}\",\n{% endfor %}\n \"{{ CLUSTER_KUBERNETES_SVC_IP }}\",\n{% for host in MASTER_CERT_HOSTS %}\n \"{{ host }}\",\n{% endfor %}\n \"kubernetes\",\n \"kubernetes.default\",\n \"kubernetes.default.svc\",\n \"kubernetes.default.svc.cluster\",\n \"kubernetes.default.svc.cluster.local\"\n ],\n \"key\": {\n \"algo\": \"rsa\",\n \"size\": 2048\n },\n \"names\": [\n {\n \"C\": \"CN\",\n \"ST\": \"HangZhou\",\n \"L\": \"XS\",\n \"O\": \"k8s\",\n \"OU\": \"System\"\n }\n ]\n}\n```\nkubernetes apiserver 使用对等证书，创建时hosts字段需要配置：\n- 如果配置 ex_lb，需要把 EX_APISERVER_VIP 也配置进去\n- 如果需要外部访问 apiserver，可选在config.yml配置 MASTER_CERT_HOSTS\n- `kubectl get svc` 将看到集群中由api-server 创建的默认服务 `kubernetes`，因此也要把 `kubernetes` 服务名和各个服务域名也添加进去\n\n### 创建apiserver的服务配置文件\n\n``` bash\n[Unit]\nDescription=Kubernetes API Server\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\nAfter=network.target\n\n[Service]\nExecStart={{ bin_dir }}/kube-apiserver \\\n --allow-privileged=true \\\n --anonymous-auth=false \\\n --api-audiences=api,istio-ca \\\n --authorization-mode=Node,RBAC \\\n --bind-address={{ inventory_hostname }} \\\n --client-ca-file={{ ca_dir }}/ca.pem \\\n --endpoint-reconciler-type=lease \\\n --etcd-cafile={{ ca_dir }}/ca.pem \\\n --etcd-certfile={{ ca_dir }}/kubernetes.pem \\\n --etcd-keyfile={{ ca_dir }}/kubernetes-key.pem \\\n --etcd-servers={{ ETCD_ENDPOINTS }} \\\n --kubelet-certificate-authority={{ ca_dir }}/ca.pem \\\n --kubelet-client-certificate={{ ca_dir }}/kubernetes.pem \\\n --kubelet-client-key={{ ca_dir }}/kubernetes-key.pem \\\n --secure-port={{ SECURE_PORT }} \\\n --service-account-issuer=https://kubernetes.default.svc \\\n --service-account-signing-key-file={{ ca_dir }}/ca-key.pem \\\n --service-account-key-file={{ ca_dir }}/ca.pem \\\n --service-cluster-ip-range={{ SERVICE_CIDR }} \\\n --service-node-port-range={{ NODE_PORT_RANGE }} \\\n --tls-cert-file={{ ca_dir }}/kubernetes.pem \\\n --tls-private-key-file={{ ca_dir }}/kubernetes-key.pem \\\n --requestheader-client-ca-file={{ ca_dir }}/ca.pem \\\n --requestheader-allowed-names= \\\n --requestheader-extra-headers-prefix=X-Remote-Extra- \\\n --requestheader-group-headers=X-Remote-Group \\\n --requestheader-username-headers=X-Remote-User \\\n --proxy-client-cert-file={{ ca_dir }}/aggregator-proxy.pem \\\n --proxy-client-key-file={{ ca_dir }}/aggregator-proxy-key.pem \\\n --enable-aggregator-routing=true \\\n --v=2\nRestart=always\nRestartSec=5\nType=notify\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\n```\n+ Kubernetes 对 API 访问需要依次经过认证、授权和准入控制(admission controll)，认证解决用户是谁的问题，授权解决用户能做什么的问题，Admission Control则是资源管理方面的作用。\n+ 关于authorization-mode=Node,RBAC v1.7+支持Node授权，配合NodeRestriction准入控制来限制kubelet仅可访问node、endpoint、pod、service以及secret、configmap、PV和PVC等相关的资源；需要注意的是v1.7中Node 授权是默认开启的，v1.8中需要显式配置开启，否则 Node无法正常工作\n+ 详细参数配置请参考`kube-apiserver --help`，关于认证、授权和准入控制请[阅读](https://github.com/feiskyer/kubernetes-handbook/blob/master/components/apiserver.md)\n+ 增加了访问kubelet使用的证书配置，防止匿名访问kubelet的安全漏洞，详见[漏洞说明](../mixes/01.fix_kubelet_annoymous_access.md)\n\n### 创建controller-manager 的服务文件\n\n``` bash\n[Unit]\nDescription=Kubernetes Controller Manager\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\n\n[Service]\nExecStart={{ bin_dir }}/kube-controller-manager \\\n --allocate-node-cidrs=true \\\n --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\n --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\n --bind-address=0.0.0.0 \\\n --cluster-cidr={{ CLUSTER_CIDR }} \\\n --cluster-name=kubernetes \\\n --cluster-signing-cert-file={{ ca_dir }}/ca.pem \\\n --cluster-signing-key-file={{ ca_dir }}/ca-key.pem \\\n --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\n --leader-elect=true \\\n --node-cidr-mask-size={{ NODE_CIDR_LEN }} \\\n --root-ca-file={{ ca_dir }}/ca.pem \\\n --service-account-private-key-file={{ ca_dir }}/ca-key.pem \\\n --service-cluster-ip-range={{ SERVICE_CIDR }} \\\n --use-service-account-credentials=true \\\n --v=2\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=multi-user.target\n```\n+ --cluster-cidr 指定 Cluster 中 Pod 的 CIDR 范围，该网段在各 Node 间必须路由可达(flannel/calico 等网络插件实现)\n+ --service-cluster-ip-range 参数指定 Cluster 中 Service 的CIDR范围，必须和 kube-apiserver 中的参数一致\n+ --cluster-signing-* 指定的证书和私钥文件用来签名为 TLS BootStrap 创建的证书和私钥\n+ --root-ca-file 用来对 kube-apiserver 证书进行校验，指定该参数后，才会在Pod 容器的 ServiceAccount 中放置该 CA 证书文件\n+ --leader-elect=true 使用多节点选主的方式选择主节点。只有主节点才会启动所有控制器，而其他从节点则仅执行选主算法\n\n### 创建scheduler 的服务文件\n\n``` bash\n[Unit]\nDescription=Kubernetes Scheduler\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\n\n[Service]\nExecStart={{ bin_dir }}/kube-scheduler \\\n --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\\n --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\\n --bind-address=0.0.0.0 \\\n --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\\n --leader-elect=true \\\n --v=2\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=multi-user.target\n```\n\n+ --leader-elect=true 部署多台机器组成的 master 集群时选举产生一个处于工作状态的 kube-controller-manager 进程\n\n### 在master 节点安装 node 服务: kubelet kube-proxy \n\n项目master 分支使用 DaemonSet 方式安装网络插件，如果master 节点不安装 kubelet 服务是无法安装网络插件的，如果 master 节点不安装网络插件，那么通过`apiserver` 方式无法访问 `dashboard` `kibana`等管理界面，[ISSUES #130](https://github.com/easzlab/kubeasz/issues/130)\n\n在master 节点也同时成为 node 节点后，默认业务 POD也会调度到 master节点；可以使用 `kubectl cordon`命令禁止业务 POD调度到 master节点。\n\n### master 集群的验证\n\n运行 `ansible-playbook 04.kube-master.yml` 成功后，验证 master节点的主要组件：\n\n``` bash\n# 查看进程状态\nsystemctl status kube-apiserver\nsystemctl status kube-controller-manager\nsystemctl status kube-scheduler\n# 查看进程运行日志\njournalctl -u kube-apiserver\njournalctl -u kube-controller-manager\njournalctl -u kube-scheduler\n```\n执行 `kubectl get componentstatus` 可以看到\n\n``` bash\nNAME STATUS MESSAGE ERROR\nscheduler Healthy ok \ncontroller-manager Healthy ok \netcd-0 Healthy {\"health\": \"true\"} \netcd-2 Healthy {\"health\": \"true\"} \netcd-1 Healthy {\"health\": \"true\"} \n```\n\n[后一篇](05-install_kube_node.md)\n"
},
{
"path": "docs/setup/05-install_kube_node.md",
"content": "## 05-安装kube_node节点\n\n`kube_node` 是集群中运行工作负载的节点，前置条件需要先部署好`kube_master`节点，它需要部署如下组件：\n\n``` bash\ncat playbooks/05.kube-node.yml\n- hosts: kube_node\n roles:\n - { role: kube-lb, when: \"inventory_hostname not in groups['kube_master']\" }\n - { role: kube-node, when: \"inventory_hostname not in groups['kube_master']\" }\n```\n\n+ kube-lb：由nginx裁剪编译的四层负载均衡，用于将请求转发到主节点的 apiserver服务\n+ kubelet：kube_node上最主要的组件\n+ kube-proxy： 发布应用服务与负载均衡\n\n### 创建cni 基础网络插件配置文件\n\n因为后续需要用 `DaemonSet Pod`方式运行k8s网络插件，所以kubelet.server服务必须开启cni相关参数，并且提供cni网络配置文件\n\n### 创建 kubelet 的服务文件\n\n+ 根据官方建议独立使用 kubelet 配置文件，详见roles/kube-node/templates/kubelet-config.yaml.j2\n+ 必须先创建工作目录 `/var/lib/kubelet`\n\n``` bash\n[Unit]\nDescription=Kubernetes Kubelet\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\n\n[Service]\nWorkingDirectory=/var/lib/kubelet\nExecStartPre=/bin/mount -o remount,rw '/sys/fs/cgroup'\n{% if KUBE_RESERVED_ENABLED == \"yes\" or SYS_RESERVED_ENABLED == \"yes\" %}\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpu/podruntime.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuacct/podruntime.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/podruntime.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/memory/podruntime.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/pids/podruntime.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/systemd/podruntime.slice\n\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpu/system.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuacct/system.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/memory/system.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/pids/system.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/systemd/system.slice\n\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/podruntime.slice\nExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice\n{% endif %}\nExecStart={{ bin_dir }}/kubelet \\\n --config=/var/lib/kubelet/config.yaml \\\n --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\\n --hostname-override={{ K8S_NODENAME }} \\\n --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\n --root-dir={{ KUBELET_ROOT_DIR }} \\\n --v=2\nRestart=always\nRestartSec=5\n\n[Install]\nWantedBy=multi-user.target\n```\n+ --ExecStartPre=/bin/mkdir -p xxx 对于某些系统（centos7）cpuset和hugetlb 是默认没有初始化system.slice 的，需要手动创建，否则在启用--kube-reserved-cgroup 时会报错Failed to start ContainerManager Failed to enforce System Reserved Cgroup Limits\n+ 关于kubelet资源预留相关配置请参考 https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/\n\n### 创建 kube-proxy kubeconfig 文件\n\n该步骤已经在 deploy节点完成，[roles/deploy/tasks/main.yml](../../roles/deploy/tasks/main.yml)\n\n+ 生成的kube-proxy.kubeconfig 配置文件需要移动到/etc/kubernetes/目录，后续kube-proxy服务启动参数里面需要指定\n\n### 创建 kube-proxy服务文件\n\n``` bash\n[Unit]\nDescription=Kubernetes Kube-Proxy Server\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\nAfter=network.target\n\n[Service]\nWorkingDirectory=/var/lib/kube-proxy\nExecStart={{ bin_dir }}/kube-proxy \\\n --config=/var/lib/kube-proxy/kube-proxy-config.yaml\nRestart=always\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\n```\n\n请注意 [kube-proxy-config](../../roles/kube-node/templates/kube-proxy-config.yaml.j2) 文件的注释说明\n\n### 验证 node 状态\n\n``` bash\nsystemctl status kubelet\t# 查看状态\nsystemctl status kube-proxy\njournalctl -u kubelet\t\t# 查看日志\njournalctl -u kube-proxy \n```\n运行 `kubectl get node` 可以看到类似\n\n``` bash\nNAME STATUS ROLES AGE VERSION\n192.168.1.42 Ready ![\"cilium_http_gsg\"](\"https://docs.cilium.io/en/stable/_images/cilium_http_gsg.png\")
\n\n根据文件[http-sw-app.yaml](../../../roles/cilium/files/star_war_example/http-sw-app.yaml) 创建 `$ kubectl create -f http-sw-app.yaml` 后，验证如下：\n\n``` bash\n$ kubectl get pods,svc\nNAME READY STATUS RESTARTS AGE\npod/deathstar-5fc7c7795d-djf2q 1/1 Running 0 4h\npod/deathstar-5fc7c7795d-hrgst 1/1 Running 0 4h\npod/tiefighter 1/1 Running 0 4h\npod/xwing 1/1 Running 0 4h\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nservice/deathstar ClusterIP 10.68.242.130 ![\"cilium_http_l3_l4_gsg\"](\"https://docs.cilium.io/en/stable/_images/cilium_http_l3_l4_gsg.png\")
\n\n根据文件[sw_l3_l4_policy.yaml](../../../roles/cilium/files/star_war_example/sw_l3_l4_policy.yaml) 创建 `$ kubectl apply -f sw_l3_l4_policy.yaml` 后，验证如下：\n\n``` bash\n$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing\nShip landed # 成功着陆\n\n$ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing\n# 失败超时\n```\n\n### 查看安全策略\n\n再次执行 `cilium endpoint list`，可以看到标签带`deathstar`的 POD 已经应用了 `Ingress`方向的策略：\n\n``` bash\n# kubectl exec -n kube-system cilium-6t5vx -- cilium endpoint list\nENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS \n ENFORCEMENT ENFORCEMENT \n643 Enabled Disabled 31371 k8s:class=deathstar f00d::ac14:0:0:283 172.20.0.246 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=empire \n1011 Enabled Disabled 31371 k8s:class=deathstar f00d::ac14:0:0:3f3 172.20.0.63 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=empire \n32030 Disabled Disabled 5350 k8s:class=tiefighter f00d::ac14:0:0:7d1e 172.20.0.201 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=empire \n45943 Disabled Disabled 14309 k8s:class=xwing f00d::ac14:0:0:b377 172.20.0.189 ready \n k8s:io.cilium.k8s.policy.serviceaccount=default \n k8s:io.kubernetes.pod.namespace=default \n k8s:org=alliance \n52035 Disabled Disabled 4 reserved:health f00d::ac14:0:0:cb43 172.20.0.92 ready \n```\n\n查看具体策略内容 `kubectl describe cnp rule1`\n\n### L7 安全策略\n\n上述的策略可以进行简单的安全防护了，但是“死星”的这个系统还有很多复杂的功能；比如它还提供了一个内部维护接口，如果被不合理调用将带来严重灾难性后果，也许“联盟”勇士劫持了一架“帝国”飞船正在进行这个任务（虽然我们内心希望他能够成功摧毁“死星”）。不幸的是“死星”系统设计者考虑到这个风险，它有办法严格限制每架飞船能够请求的权限。\n\n没有限制飞船请求权限时，如下运行：\n\n``` bash\n$ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port\nPanic: deathstar exploded\n\ngoroutine 1 [running]:\nmain.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)\n /code/src/github.com/empire/deathstar/\n temp/main.go:9 +0x64\nmain.main()\n /code/src/github.com/empire/deathstar/\n temp/main.go:5 +0x85\n```\n\n![\"cilium_http_l3_l4_l7_gsg\"](\"https://docs.cilium.io/en/stable/_images/cilium_http_l3_l4_l7_gsg.png\")
\n\n限制L7 的安全策略，根据文件[sw_l3_l4_l7_policy.yaml](../../../roles/cilium/files/star_war_example/sw_l3_l4_l7_policy.yaml) 创建 `$ kubectl apply -f sw_l3_l4_l7_policy.yaml` 后，验证如下：\n\n``` bash\n$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing\nShip landed\n$ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port\nAccess denied\n```\n\n我们同样可以使用 `kubectl desribe cnp`检查更新的策略，或者使用 `cilium` 命令行：\n\n``` bash\n$ kubectl exec -n kube-system cilium-6t5vx -- cilium policy get\n[\n {\n \"endpointSelector\": {\n \"matchLabels\": {\n \"any:class\": \"deathstar\",\n \"any:org\": \"empire\",\n \"k8s:io.kubernetes.pod.namespace\": \"default\"\n }\n },\n \"ingress\": [\n {\n \"fromEndpoints\": [\n {\n \"matchLabels\": {\n \"any:org\": \"empire\",\n \"k8s:io.kubernetes.pod.namespace\": \"default\"\n }\n }\n ],\n \"toPorts\": [\n {\n \"ports\": [\n {\n \"port\": \"80\",\n \"protocol\": \"TCP\"\n }\n ],\n \"rules\": {\n \"http\": [\n {\n \"path\": \"/v1/request-landing\",\n \"method\": \"POST\"\n }\n ]\n }\n }\n ]\n }\n ],\n \"labels\": [\n {\n \"key\": \"io.cilium.k8s.policy.name\",\n \"value\": \"rule1\",\n \"source\": \"k8s\"\n },\n {\n \"key\": \"io.cilium.k8s.policy.namespace\",\n \"value\": \"default\",\n \"source\": \"k8s\"\n }\n ]\n }\n]\nRevision: 267\n```\n我们看到 `cilium` 可以实现 `7层 HTTP `协议的请求方法（GET/PUT/POST等）、路径（/v1/request-landing）等等安全策略；另外，它还可以防护其他应用（如：Kafka, gRPC, Elasticsearch），可以去官网文档示例学习！\n\n## 参考资料\n\n- [cilium github](https://github.com/cilium/cilium)\n- [cilium doc](http://docs.cilium.io)\n"
},
{
"path": "docs/setup/network-plugin/cilium.md",
"content": "# 06-安装cilium网络组件\n\n`cilium` 是一个革新的网络与安全组件；基于 linux 内核新技术--`BPF`，它可以透明、零侵入地实现服务间安全策略与可视化，主要优势如下：\n\n- 支持L3/L4, L7(如：HTTP/gRPC/Kafka)的安全策略\n- 支持基于安全ID而不是地址+端口的传统防火墙策略\n- 支持基于Overlay或Native Routing的扁平多节点pod网络\n - Overlay VXLAN 方式类似于 flannel 的VXLAN后端\n- 高性能负载均衡，支持DSR\n- 支持事件、策略跟踪和监控集成\n\ncilium 项目文档比较完整，建议仔细阅读[官网文档]()\n\n## kubeasz 集成安装 cilium\n\nkubeasz 3.3.1 更新重写了cilium 安装流程，使用helm charts 方式，配置文件在 roles/cilium/templates/values.yaml.j2，请阅读原charts中values.yaml 文件后自定义修改。\n\n- https://docs.cilium.io/en/stable/installation/k8s-install-helm/#k8s-install-helm\n- 相关镜像已经离线打包并推送到本地镜像仓库，通过 `ezdown -X` 命令下载cilium等额外镜像\n\n### 0.检查系统内核版本\n\n- Linux kernel >= 4.9.17，如需升级请阅读文档[升级内核](guide/kernel_upgrade.md)\n- etcd >= 3.1.0 or consul >= 0.6.4\n\n### 1.选择cilium网络后安装\n\n- 参考[快速指南](../quickStart.md)，设置`/etc/kubeasz/clusters/xxx/hosts`文件中变量 `CLUSTER_NETWORK=\"cilium\"` \n- 下载额外镜像 `./ezdown -X cilium 和 ./ezdown -X network-check`\n- 执行集群安装 `dk ezctl setup xxx all`\n\n注意默认设置未集成cilium_hubble，可以在`/etc/kubeasz/clusters/xxx/config.yml`配置启用后再开始安装。\n\n- cilium_connectivity_check：检查集群cilium网络是否工作正常，非常实用\n- cilium_hubble：很酷很实用的监控、策略追踪排查工具\n\nCilium CLI 和 Hubble CLI 二进制已经默认包含在kubeasz-ext-bin 1.2.0及之后的版本中 https://github.com/kubeasz/dockerfiles/blob/master/kubeasz-ext-bin/Dockerfile\n\n### 2.验证\n\n一键安装完成后如下，注意cilium_connectivity_check 中带`multi-node`的检查任务需要多节点集群才能完成\n\n```\nkubectl get pod -A\nNAMESPACE NAME READY STATUS RESTARTS AGE\ncilium-test echo-a-5dd478f5d8-74xg5 1/1 Running 0 3m10s\ncilium-test echo-b-78c79f6cdd-t9vk6 1/1 Running 0 3m10s\ncilium-test echo-b-host-75c44b897-c8f5m 1/1 Running 0 3m10s\ncilium-test host-to-b-multi-node-clusterip-7895fd494c-92cb2 1/1 Running 0 2m59s\ncilium-test host-to-b-multi-node-headless-74bbc877b5-ffxxx 1/1 Running 0 2m59s\ncilium-test pod-to-a-allowed-cnp-598fc5c547-b885q 1/1 Running 0 2m59s\ncilium-test pod-to-a-b8b456c99-r6272 1/1 Running 0 2m59s\ncilium-test pod-to-a-denied-cnp-c78c44f5c-7xhkw 1/1 Running 0 2m59s\ncilium-test pod-to-b-intra-node-nodeport-6ccdb55779-j8gnd 1/1 Running 0 2m59s\ncilium-test pod-to-b-multi-node-clusterip-55d8448b5c-5b4nj 1/1 Running 0 2m59s\ncilium-test pod-to-b-multi-node-headless-5fbf655bb9-pszpr 1/1 Running 0 2m59s\ncilium-test pod-to-b-multi-node-nodeport-65f5b95569-qglb7 1/1 Running 0 2m59s\ncilium-test pod-to-external-1111-64496c754c-bvqlt 1/1 Running 0 2m59s\ncilium-test pod-to-external-fqdn-allow-baidu-cnp-6f96597855-c84zs 1/1 Running 0 2m59s\nkube-system cilium-7trcs 1/1 Running 0 3m42s\nkube-system cilium-hvclp 1/1 Running 0 3m42s\nkube-system cilium-operator-8566689975-vcxpp 1/1 Running 0 3m42s\nkube-system cilium-pw2sv 1/1 Running 0 3m42s\nkube-system cilium-qppnc 1/1 Running 0 3m42s\nkube-system coredns-84b58f6b4-m8x7s 1/1 Running 0 3m20s\nkube-system dashboard-metrics-scraper-864d79d497-92l2w 1/1 Running 0 3m14s\nkube-system hubble-relay-655dc744d7-8d9n7 1/1 Running 0 3m42s\nkube-system hubble-ui-54599d7967-lfkvk 2/2 Running 0 3m42s\nkube-system kubernetes-dashboard-5fc74cf5c6-pqdvc 1/1 Running 0 3m14s\nkube-system metrics-server-69797698d4-2jbg8 1/1 Running 0 3m17s\nkube-system node-local-dns-5n8gc 1/1 Running 0 3m19s\nkube-system node-local-dns-5pm2p 1/1 Running 0 3m19s\nkube-system node-local-dns-9x229 1/1 Running 0 3m19s\nkube-system node-local-dns-jz8lj 1/1 Running 0 3m19s\n```\n\n检查 cilium 节点状态\n\n```\ncilium status\n /¯¯\\\n /¯¯\\__/¯¯\\ Cilium: OK\n \\__/¯¯\\__/ Operator: OK\n /¯¯\\__/¯¯\\ Hubble: OK\n \\__/¯¯\\__/ ClusterMesh: disabled\n \\__/\n\nDaemonSet cilium Desired: 4, Ready: 4/4, Available: 4/4\nDeployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1\nDeployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1\nDeployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1\nContainers: cilium Running: 4\n cilium-operator Running: 1\n hubble-relay Running: 1\n hubble-ui Running: 1\nCluster Pods: 17/17 managed by Cilium\nImage versions hubble-relay easzlab.io.local:5000/cilium/hubble-relay:v1.11.6: 1\n hubble-ui easzlab.io.local:5000/cilium/hubble-ui:v0.9.0: 1\n hubble-ui easzlab.io.local:5000/cilium/hubble-ui-backend:v0.9.0: 1\n cilium easzlab.io.local:5000/cilium/cilium:v1.11.6: 4\n cilium-operator easzlab.io.local:5000/cilium/operator-generic:v1.11.6: 1\n```\n\n## cilium network policy\n\ncilium network policy 提供了比k8s network policy更丰富的网络安全策略功能，有兴趣的请阅读官网文档，以下是一个有趣的小例子：\n\n- [星战死星登陆系统](cilium-example.md)\n"
},
{
"path": "docs/setup/network-plugin/flannel.md",
"content": "## 06-安装flannel网络组件.md\n\n`Flannel`是最早应用到k8s集群的网络插件之一，简单高效，且提供多个后端`backend`模式供选择；本文介绍以`DaemonSet Pod`方式集成到k8s集群，需要在所有master节点和node节点安装。\n\n### kubeasz 集成安装flannel\n\n- 参考[快速指南](../quickStart.md)，设置`/etc/kubeasz/clusters/xxx/hosts`文件中变量 `CLUSTER_NETWORK=\"flannel\"`\n- 下载额外镜像 `./ezdown -X flannel`\n- 执行集群安装 `dk ezctl setup xxx all`\n\n### 配置介绍\n\nFlannel CNI 插件的配置文件可以包含多个`plugin` 或由其调用其他`plugin`；`Flannel DaemonSet Pod`运行以后会生成`/run/flannel/subnet.env `文件，例如：\n\n``` bash\nFLANNEL_NETWORK=10.1.0.0/16\nFLANNEL_SUBNET=10.1.17.1/24\nFLANNEL_MTU=1472\nFLANNEL_IPMASQ=true\n```\n然后它利用这个文件信息去配置和调用`bridge`插件来生成容器网络，调用`host-local`来管理`IP`地址，例如：\n\n``` bash\n{\n\t\"name\": \"mynet\",\n\t\"type\": \"bridge\",\n\t\"mtu\": 1472,\n\t\"ipMasq\": false,\n\t\"isGateway\": true,\n\t\"ipam\": {\n\t\t\"type\": \"host-local\",\n\t\t\"subnet\": \"10.1.17.0/24\"\n\t}\n}\n```\n- 更多相关介绍请阅读：\n - [flannel kubernetes 集成](https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md)\n - [flannel cni 插件](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel)\n - [更多 cni 插件](https://github.com/containernetworking/plugins)\n\n- `Flannel DaemonSet` yaml配置文件\n\n请阅读 `roles/flannel/templates/kube-flannel.yaml.j2` 内容，注意：\n\n+ 注意：本安装方式，flannel 通过 apiserver 接口读取 podCidr 信息，详见 https://github.com/coreos/flannel/issues/847；因此想要修改节点pod网段掩码，请在`clusters/xxxx/config.yml` 中修改`NODE_CIDR_LEN`配置项\n+ 配置相关RBAC 权限和 `service account`\n+ 配置`ConfigMap`包含 CNI配置和 flannel配置(指定backend等)，在文件中相关设置对应\n\n### 验证flannel网络\n\n执行flannel安装成功后可以验证如下：(需要等待镜像下载完成，有时候即便上一步已经配置了docker国内加速，还是可能比较慢，请确认以下容器运行起来以后，再执行后续验证步骤)\n\n``` bash\n# kubectl get pod --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\nkube-system kube-flannel-ds-m8mzm 1/1 Running 0 3m\nkube-system kube-flannel-ds-mnj6j 1/1 Running 0 3m\nkube-system kube-flannel-ds-mxn6k 1/1 Running 0 3m\n```\n在集群创建几个测试pod: `kubectl run test --image=busybox --replicas=3 sleep 30000`\n\n``` bash\n# kubectl get pod --all-namespaces -o wide|head -n 4\nNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE\ndefault busy-5956b54c8b-ld4gb 1/1 Running 0 9m 172.20.2.7 192.168.1.1\ndefault busy-5956b54c8b-lj9l9 1/1 Running 0 9m 172.20.1.5 192.168.1.2\ndefault busy-5956b54c8b-wwpkz 1/1 Running 0 9m 172.20.0.6 192.168.1.3\n\n# 查看路由\n# ip route\ndefault via 192.168.1.254 dev ens3 onlink \n192.168.1.0/24 dev ens3 proto kernel scope link src 192.168.1.1 \n172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown \n172.20.0.0/24 via 192.168.1.3 dev ens3 \n172.20.1.0/24 via 192.168.1.2 dev ens3 \n172.20.2.0/24 dev cni0 proto kernel scope link src 172.20.2.1 \n```\n在各节点上分别 ping 这三个POD IP地址，确保能通：\n\n``` bash\nping 172.20.2.7\nping 172.20.1.5\nping 172.20.0.6\n```\n\n"
},
{
"path": "docs/setup/network-plugin/kube-ovn.md",
"content": "## 06-安装kube-ovn网络组件.md\n\n(以下文档暂未更新，以插件官网文档为准)\n\n由灵雀云开源的网络组件 kube-ovn，将已被 openstack 社区采用的成熟网络虚拟化技术 ovs/ovn 引入 kubernetes 平台；为 kubernetes 网络打开了新的大门，令人耳目一新；强烈推荐大家试用该网络组件，反馈建议以帮助项目早日走向成熟。\n\n- 介绍 https://blog.csdn.net/alauda_andy/article/details/88886128\n- 项目地址 https://github.com/alauda/kube-ovn\n\n### 特性介绍\n\nkube-ovn 提供了针对企业应用场景下容器网络实用功能，并为实现更高级的网络管理控制提供了可能性；现有主要功能:\n\n- 1.Namespace 和子网的绑定，以及子网间的访问控制;\n- 2.静态IP分配;\n- 3.动态QoS;\n- 4.分布式和集中式网关;\n- 5.内嵌 LoadBalancer;\n- 6.Pod IP对外直接暴露\n- 7.流量镜像\n- 8.IPv6\n\n### kubeasz 集成安装 kube-ovn\n\nkube-ovn 的安装十分简单，详见项目的安装文档；基于 kubeasz，以下两步将安装一个集成了 kube-ovn 网络的 k8s 集群；\n\n- 在 ansible hosts 中设置变量 `CLUSTER_NETWORK=\"kube-ovn\"`\n- 执行安装 `ansible-playbook 90.setup.yml` 或者 `ezctl setup`\n\nkubeasz 项目为`kube-ovn`网络生成的 ansible role 如下：\n\n``` bash\nroles/kube-ovn\n├── defaults\n│ └── main.yml\t\t# kube-ovn 相关配置文件\n├── tasks\n│ └── main.yml\t\t# 安装执行文件\n└── templates\n ├── crd.yaml.j2\t # crd 模板\n ├── kube-ovn.yaml.j2\t# kube-ovn yaml 模板\n └── ovn.yaml.j2\t\t # ovn yaml 模板\n \n```\n\n安装成功后，可以验证所有 k8s 集群功能正常，查看集群的 pod 网络如下：\n\n```\n$ kubectl get pod --all-namespaces -o wide\nNAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\nkube-ovn kube-ovn-cni-5php2 1/1 Running 2 35h 192.168.1.43 192.168.1.43