Repository: morvencao/kube-sidecar-injector Branch: master Commit: 4e010f4cdee8 Files: 25 Total size: 48.9 MB Directory structure: gitextract__vpu_fhs/ ├── .dockerignore ├── .gitignore ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── bin/ │ └── kustomize ├── cmd/ │ ├── cert.go │ ├── main.go │ ├── webhook.go │ └── webhookconfig.go ├── deploy/ │ ├── clusterrole.yaml │ ├── clusterrolebinding.yaml │ ├── configmap.yaml │ ├── deployment.yaml │ ├── kustomization.yaml │ ├── namespace.yaml │ ├── nginx-configmap.yaml │ ├── service.yaml │ └── serviceaccount.yaml ├── go.mod ├── go.sum ├── medium-article.md ├── mutating-admission-webhook.xml └── prestop.sh ================================================ FILE CONTENTS ================================================ ================================================ FILE: .dockerignore ================================================ # Binaries for programs and plugins *.exe *.dll *.so *.dylib # Test binary, build with `go test -c` *.test # Output of the go coverage tool, specifically when used with LiteIDE *.out vendor/* ================================================ FILE: .gitignore ================================================ # Binaries for programs and plugins *.exe *.dll *.so *.dylib # Test binary, build with `go test -c` *.test # Output of the go coverage tool, specifically when used with LiteIDE *.out vendor/* build/_output # GOPATH .go ================================================ FILE: Dockerfile ================================================ # Build the sidecar-injector binary FROM golang:1.17 as builder WORKDIR /workspace # Copy the Go Modules manifests COPY go.mod go.mod COPY go.sum go.sum # cache deps before building and copying source so that we don't need to re-download as much # and so that source changes don't invalidate our downloaded layer RUN go mod download # Copy the go source COPY cmd/ cmd/ # Build RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o sidecar-injector ./cmd FROM alpine:latest # install curl for prestop script RUN apk --no-cache add curl WORKDIR / # install binary COPY --from=builder /workspace/sidecar-injector . # install the prestop script COPY ./prestop.sh . USER 65532:65532 ENTRYPOINT ["/sidecar-injector"] ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: Makefile ================================================ # Setting SHELL to bash allows bash commands to be executed by recipes. # This is a requirement for 'setup-envtest.sh' in the test target. # Options are set to exit when a recipe line exits non-zero or a piped command fails. SHELL = /usr/bin/env bash -o pipefail .SHELLFLAGS = -ec # Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set) ifeq (,$(shell go env GOBIN)) GOBIN=$(shell go env GOPATH)/bin else GOBIN=$(shell go env GOBIN) endif # Tools for deploy KUBECTL?=kubectl PWD=$(shell pwd) KUSTOMIZE?=$(PWD)/$(PERMANENT_TMP_GOPATH)/bin/kustomize KUSTOMIZE_VERSION?=v3.5.4 KUSTOMIZE_ARCHIVE_NAME?=kustomize_$(KUSTOMIZE_VERSION)_$(GOHOSTOS)_$(GOHOSTARCH).tar.gz kustomize_dir:=$(dir $(KUSTOMIZE)) IMAGE = quay.io/morvencao/sidecar-injector:latest all: build .PHONY: all ##@ General # The help target prints out all targets with their descriptions organized # beneath their categories. The categories are represented by '##@' and the # target descriptions by '##'. The awk commands is responsible for reading the # entire set of makefiles included in this invocation, looking for lines of the # file as xyz: ## something, and then pretty-format the target and help. Then, # if there's a line with ##@ something, that gets pretty-printed as a category. # More info on the usage of ANSI control characters for terminal formatting: # https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_parameters # More info on the awk command: # http://linuxcommand.org/lc3_adv_awk.php .PHONY: help help: ## Display this help. @awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST) ##@ Development .PHONY: fmt fmt: ## Run go fmt against code. go fmt ./... .PHONY: vet vet: ## Run go vet against code. go vet ./... .PHONY: test test: fmt vet ## Run tests. go test ./... -coverprofile cover.out ##@ Build .PHONY: build build: fmt vet ## Build binary. go build -o bin/sidecar-injector ./cmd/ .PHONY: docker-build docker-build: test ## Build docker image. docker build -t ${IMAGE} . .PHONY: docker-push docker-push: ## Push docker image. docker push ${IMAGE} ##@ Deployment deploy: kustomize cp deploy/kustomization.yaml deploy/kustomization.yaml.tmp cd deploy && $(KUSTOMIZE) edit set image sidecar-injector=$(IMAGE) $(KUSTOMIZE) build deploy | $(KUBECTL) apply -f - mv deploy/kustomization.yaml.tmp deploy/kustomization.yaml undeploy: kustomize $(KUSTOMIZE) build deploy | $(KUBECTL) delete --ignore-not-found -f - KUSTOMIZE = $(shell pwd)/bin/kustomize .PHONY: kustomize kustomize: ## Download kustomize locally if necessary. $(call go-get-tool,$(KUSTOMIZE),sigs.k8s.io/kustomize/kustomize/v3@v3.8.7) # go-get-tool will 'go get' any package $2 and install it to $1. PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST)))) define go-get-tool @[ -f $(1) ] || { \ set -e ;\ TMP_DIR=$$(mktemp -d) ;\ cd $$TMP_DIR ;\ go mod init tmp ;\ echo "Downloading $(2)" ;\ GOBIN=$(PROJECT_DIR)/bin go get $(2) ;\ rm -rf $$TMP_DIR ;\ } endef ================================================ FILE: README.md ================================================ # kube-sidecar-injector This repo is used for [a tutorial at Medium](https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74) to create a Kubernetes [MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19) that injects a nginx sidecar container into pod prior to persistence of the object. ## Prerequisites - [git](https://git-scm.com/downloads) - [go](https://golang.org/dl/) version v1.17+ - [docker](https://docs.docker.com/install/) version 19.03+ - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) version v1.19+ - Access to a Kubernetes v1.19+ cluster with the `admissionregistration.k8s.io/v1` API enabled. Verify that by the following command: ``` kubectl api-versions | grep admissionregistration.k8s.io ``` The result should be: ``` admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 ``` > Note: In addition, the `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver. ## Build and Deploy 1. Build and push docker image: ```bash make docker-build docker-push IMAGE=quay.io//sidecar-injector:latest ``` 2. Deploy the kube-sidecar-injector to kubernetes cluster: ```bash make deploy IMAGE=quay.io//sidecar-injector:latest ``` 3. Verify the kube-sidecar-injector is up and running: ```bash # kubectl -n sidecar-injector get pod # kubectl -n sidecar-injector get pod NAME READY STATUS RESTARTS AGE sidecar-injector-7c8bc5f4c9-28c84 1/1 Running 0 30s ``` ## How to use 1. Create a new namespace `test-ns` and label it with `sidecar-injector=enabled`: ``` # kubectl create ns test-ns # kubectl label namespace test-ns sidecar-injection=enabled # kubectl get namespace -L sidecar-injection NAME STATUS AGE SIDECAR-INJECTION default Active 26m test-ns Active 13s enabled kube-public Active 26m kube-system Active 26m sidecar-injector Active 17m ``` 2. Deploy an app in Kubernetes cluster, take `alpine` app as an example ```bash kubectl -n test-ns run alpine \ --image=alpine \ --restart=Never \ --command -- sleep infinity ``` 3. Verify sidecar container is injected: ``` # kubectl -n test-ns get pod NAME READY STATUS RESTARTS AGE alpine 2/2 Running 0 10s # kubectl -n test-ns get pod alpine -o jsonpath="{.spec.containers[*].name}" alpine sidecar-nginx ``` ## Troubleshooting Sometimes you may find that pod is injected with sidecar container as expected, check the following items: 1. The sidecar-injector pod is in running state and no error logs. 2. The namespace in which application pod is deployed has the correct labels(`sidecar-injector=enabled`) as configured in `mutatingwebhookconfiguration`. 3. Check if the application pod has annotation `sidecar-injector-webhook.morven.me/inject:"yes"`. ================================================ FILE: bin/kustomize ================================================ [File too large to display: 48.8 MB] ================================================ FILE: cmd/cert.go ================================================ package main import ( "bytes" "crypto/rand" "crypto/rsa" "crypto/x509" "crypto/x509/pkix" "encoding/pem" "math/big" "time" ) // generateCert generate a self-signed CA for given organization // and sign certificate with the CA for given common name and dns names // it resurns the CA, certificate and private key in PEM format func generateCert(orgs, dnsNames []string, commonName string) (*bytes.Buffer, *bytes.Buffer, *bytes.Buffer, error) { // init CA config ca := &x509.Certificate{ SerialNumber: big.NewInt(2022), Subject: pkix.Name{Organization: orgs}, NotBefore: time.Now(), NotAfter: time.Now().AddDate(1, 0, 0), // expired in 1 year IsCA: true, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, BasicConstraintsValid: true, } // generate private key for CA caPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096) if err != nil { return nil, nil, nil, err } // create the CA certificate caBytes, err := x509.CreateCertificate(rand.Reader, ca, ca, &caPrivateKey.PublicKey, caPrivateKey) if err != nil { return nil, nil, nil, err } // CA certificate with PEM encoded caPEM := new(bytes.Buffer) _ = pem.Encode(caPEM, &pem.Block{ Type: "CERTIFICATE", Bytes: caBytes, }) // new certificate config newCert := &x509.Certificate{ DNSNames: dnsNames, SerialNumber: big.NewInt(1024), Subject: pkix.Name{ CommonName: commonName, Organization: orgs, }, NotBefore: time.Now(), NotAfter: time.Now().AddDate(1, 0, 0), // expired in 1 year ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth}, KeyUsage: x509.KeyUsageDigitalSignature, } // generate new private key newPrivateKey, err := rsa.GenerateKey(rand.Reader, 4096) if err != nil { return nil, nil, nil, err } // sign the new certificate newCertBytes, err := x509.CreateCertificate(rand.Reader, newCert, ca, &newPrivateKey.PublicKey, caPrivateKey) if err != nil { return nil, nil, nil, err } // new certificate with PEM encoded newCertPEM := new(bytes.Buffer) _ = pem.Encode(newCertPEM, &pem.Block{ Type: "CERTIFICATE", Bytes: newCertBytes, }) // new private key with PEM encoded newPrivateKeyPEM := new(bytes.Buffer) _ = pem.Encode(newPrivateKeyPEM, &pem.Block{ Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(newPrivateKey), }) return caPEM, newCertPEM, newPrivateKeyPEM, nil } ================================================ FILE: cmd/main.go ================================================ package main import ( "context" "crypto/tls" "flag" "fmt" "log" "net/http" "os" "os/signal" "syscall" ) var ( infoLogger *log.Logger warningLogger *log.Logger errorLogger *log.Logger ) var ( port int sidecarConfigFile string webhookNamespace, webhookServiceName string ) func init() { // init loggers infoLogger = log.New(os.Stderr, "INFO: ", log.Ldate|log.Ltime|log.Lshortfile) warningLogger = log.New(os.Stderr, "WARNING: ", log.Ldate|log.Ltime|log.Lshortfile) errorLogger = log.New(os.Stderr, "ERROR: ", log.Ldate|log.Ltime|log.Lshortfile) // webhook server running namespace webhookNamespace = os.Getenv("POD_NAMESPACE") } func main() { // init command flags flag.IntVar(&port, "port", 8443, "Webhook server port.") flag.StringVar(&webhookServiceName, "service-name", "sidecar-injector", "Webhook service name.") flag.StringVar(&sidecarConfigFile, "sidecar-config-file", "/etc/webhook/config/sidecarconfig.yaml", "Sidecar injector configuration file.") // flag.StringVar(&certFile, "tlsCertFile", "/etc/webhook/certs/cert.pem", "x509 Certificate file.") // flag.StringVar(&keyFile, "tlsKeyFile", "/etc/webhook/certs/key.pem", "x509 private key file.") flag.Parse() dnsNames := []string{ webhookServiceName, webhookServiceName + "." + webhookNamespace, webhookServiceName + "." + webhookNamespace + ".svc", } commonName := webhookServiceName + "." + webhookNamespace + ".svc" org := "morven.me" caPEM, certPEM, certKeyPEM, err := generateCert([]string{org}, dnsNames, commonName) if err != nil { errorLogger.Fatalf("Failed to generate ca and certificate key pair: %v", err) } pair, err := tls.X509KeyPair(certPEM.Bytes(), certKeyPEM.Bytes()) if err != nil { errorLogger.Fatalf("Failed to load certificate key pair: %v", err) } sidecarConfig, err := loadConfig(sidecarConfigFile) if err != nil { errorLogger.Fatalf("Failed to load configuration: %v", err) } // create or update the mutatingwebhookconfiguration err = createOrUpdateMutatingWebhookConfiguration(caPEM, webhookServiceName, webhookNamespace) if err != nil { errorLogger.Fatalf("Failed to create or update the mutating webhook configuration: %v", err) } whsvr := &WebhookServer{ sidecarConfig: sidecarConfig, server: &http.Server{ Addr: fmt.Sprintf(":%v", port), TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}}, }, } // define http server and server handler mux := http.NewServeMux() mux.HandleFunc(webhookInjectPath, whsvr.serve) whsvr.server.Handler = mux // start webhook server in new rountine go func() { if err := whsvr.server.ListenAndServeTLS("", ""); err != nil { errorLogger.Fatalf("Failed to listen and serve webhook server: %v", err) } }() // listening OS shutdown singal signalChan := make(chan os.Signal, 1) signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM) <-signalChan infoLogger.Printf("Got OS shutdown signal, shutting down webhook server gracefully...") whsvr.server.Shutdown(context.Background()) } ================================================ FILE: cmd/webhook.go ================================================ package main import ( "crypto/sha256" "encoding/json" "fmt" "io/ioutil" "net/http" "strings" "gopkg.in/yaml.v2" admissionv1 "k8s.io/api/admission/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/serializer" ) var ( runtimeScheme = runtime.NewScheme() codecs = serializer.NewCodecFactory(runtimeScheme) deserializer = codecs.UniversalDeserializer() ) var ignoredNamespaces = []string{ metav1.NamespaceSystem, metav1.NamespacePublic, } const ( admissionWebhookAnnotationInjectKey = "sidecar-injector-webhook.morven.me/inject" admissionWebhookAnnotationStatusKey = "sidecar-injector-webhook.morven.me/status" ) type WebhookServer struct { sidecarConfig *Config server *http.Server } // Webhook Server parameters type WhSvrParameters struct { port int // webhook server port certFile string // path to the x509 certificate for https keyFile string // path to the x509 private key matching `CertFile` sidecarCfgFile string // path to sidecar injector configuration file } type Config struct { Containers []corev1.Container `yaml:"containers"` Volumes []corev1.Volume `yaml:"volumes"` } type patchOperation struct { Op string `json:"op"` Path string `json:"path"` Value interface{} `json:"value,omitempty"` } func loadConfig(configFile string) (*Config, error) { data, err := ioutil.ReadFile(configFile) if err != nil { return nil, err } infoLogger.Printf("New configuration: sha256sum %x", sha256.Sum256(data)) var cfg Config if err := yaml.Unmarshal(data, &cfg); err != nil { return nil, err } return &cfg, nil } // Check whether the target resoured need to be mutated func mutationRequired(ignoredList []string, metadata *metav1.ObjectMeta) bool { // skip special kubernete system namespaces for _, namespace := range ignoredList { if metadata.Namespace == namespace { infoLogger.Printf("Skip mutation for %v for it's in special namespace:%v", metadata.Name, metadata.Namespace) return false } } annotations := metadata.GetAnnotations() if annotations == nil { annotations = map[string]string{} } status := annotations[admissionWebhookAnnotationStatusKey] // determine whether to perform mutation based on annotation for the target resource var required bool if strings.ToLower(status) == "injected" { required = false } else { switch strings.ToLower(annotations[admissionWebhookAnnotationInjectKey]) { default: required = true case "n", "not", "false", "off": required = false } } infoLogger.Printf("Mutation policy for %v/%v: status: %q required:%v", metadata.Namespace, metadata.Name, status, required) return required } func addContainer(target, added []corev1.Container, basePath string) (patch []patchOperation) { first := len(target) == 0 var value interface{} for _, add := range added { value = add path := basePath if first { first = false value = []corev1.Container{add} } else { path = path + "/-" } patch = append(patch, patchOperation{ Op: "add", Path: path, Value: value, }) } return patch } func addVolume(target, added []corev1.Volume, basePath string) (patch []patchOperation) { first := len(target) == 0 var value interface{} for _, add := range added { value = add path := basePath if first { first = false value = []corev1.Volume{add} } else { path = path + "/-" } patch = append(patch, patchOperation{ Op: "add", Path: path, Value: value, }) } return patch } func updateAnnotation(target map[string]string, added map[string]string) (patch []patchOperation) { for key, value := range added { if target == nil || target[key] == "" { target = map[string]string{} patch = append(patch, patchOperation{ Op: "add", Path: "/metadata/annotations", Value: map[string]string{ key: value, }, }) } else { patch = append(patch, patchOperation{ Op: "replace", Path: "/metadata/annotations/" + key, Value: value, }) } } return patch } // create mutation patch for resoures func createPatch(pod *corev1.Pod, sidecarConfig *Config, annotations map[string]string) ([]byte, error) { var patch []patchOperation patch = append(patch, addContainer(pod.Spec.Containers, sidecarConfig.Containers, "/spec/containers")...) patch = append(patch, addVolume(pod.Spec.Volumes, sidecarConfig.Volumes, "/spec/volumes")...) patch = append(patch, updateAnnotation(pod.Annotations, annotations)...) return json.Marshal(patch) } // main mutation process func (whsvr *WebhookServer) mutate(ar *admissionv1.AdmissionReview) *admissionv1.AdmissionResponse { req := ar.Request var pod corev1.Pod if err := json.Unmarshal(req.Object.Raw, &pod); err != nil { warningLogger.Printf("Could not unmarshal raw object: %v", err) return &admissionv1.AdmissionResponse{ Result: &metav1.Status{ Message: err.Error(), }, } } infoLogger.Printf("AdmissionReview for Kind=%v, Namespace=%v Name=%v (%v) UID=%v patchOperation=%v UserInfo=%v", req.Kind, req.Namespace, req.Name, pod.Name, req.UID, req.Operation, req.UserInfo) // determine whether to perform mutation if !mutationRequired(ignoredNamespaces, &pod.ObjectMeta) { infoLogger.Printf("Skipping mutation for %s/%s due to policy check", pod.Namespace, pod.Name) return &admissionv1.AdmissionResponse{ Allowed: true, } } annotations := map[string]string{admissionWebhookAnnotationStatusKey: "injected"} patchBytes, err := createPatch(&pod, whsvr.sidecarConfig, annotations) if err != nil { return &admissionv1.AdmissionResponse{ Result: &metav1.Status{ Message: err.Error(), }, } } infoLogger.Printf("AdmissionResponse: patch=%v\n", string(patchBytes)) return &admissionv1.AdmissionResponse{ Allowed: true, Patch: patchBytes, PatchType: func() *admissionv1.PatchType { pt := admissionv1.PatchTypeJSONPatch return &pt }(), } } // Serve method for webhook server func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) { var body []byte if r.Body != nil { if data, err := ioutil.ReadAll(r.Body); err == nil { body = data } } if len(body) == 0 { warningLogger.Println("empty body") http.Error(w, "empty body", http.StatusBadRequest) return } // verify the content type is accurate contentType := r.Header.Get("Content-Type") if contentType != "application/json" { warningLogger.Printf("Content-Type=%s, expect application/json", contentType) http.Error(w, "invalid Content-Type, expect `application/json`", http.StatusUnsupportedMediaType) return } var admissionResponse *admissionv1.AdmissionResponse ar := admissionv1.AdmissionReview{} if _, _, err := deserializer.Decode(body, nil, &ar); err != nil { warningLogger.Printf("Can't decode body: %v", err) admissionResponse = &admissionv1.AdmissionResponse{ Result: &metav1.Status{ Message: err.Error(), }, } } else { admissionResponse = whsvr.mutate(&ar) } admissionReview := admissionv1.AdmissionReview{ TypeMeta: metav1.TypeMeta{ APIVersion: "admission.k8s.io/v1", Kind: "AdmissionReview", }, } if admissionResponse != nil { admissionReview.Response = admissionResponse if ar.Request != nil { admissionReview.Response.UID = ar.Request.UID } } resp, err := json.Marshal(admissionReview) if err != nil { warningLogger.Printf("Can't encode response: %v", err) http.Error(w, fmt.Sprintf("could not encode response: %v", err), http.StatusInternalServerError) } infoLogger.Printf("Ready to write reponse ...") if _, err := w.Write(resp); err != nil { warningLogger.Printf("Can't write response: %v", err) http.Error(w, fmt.Sprintf("could not write response: %v", err), http.StatusInternalServerError) } } ================================================ FILE: cmd/webhookconfig.go ================================================ package main import ( "bytes" "context" "os" "reflect" admissionregistrationv1 "k8s.io/api/admissionregistration/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/clientcmd" ) var ( webhookConfigName = "sidecar-injector-webhook" webhookInjectPath = "/inject" ) func createOrUpdateMutatingWebhookConfiguration(caPEM *bytes.Buffer, webhookService, webhookNamespace string) error { infoLogger.Println("Initializing the kube client...") kubeconfig := os.Getenv("KUBECONFIG") config, err := clientcmd.BuildConfigFromFlags("", kubeconfig) if err != nil { return err } clientset, err := kubernetes.NewForConfig(config) if err != nil { return err } mutatingWebhookConfigV1Client := clientset.AdmissionregistrationV1() infoLogger.Printf("Creating or updating the mutatingwebhookconfiguration: %s", webhookConfigName) fail := admissionregistrationv1.Fail sideEffect := admissionregistrationv1.SideEffectClassNone mutatingWebhookConfig := &admissionregistrationv1.MutatingWebhookConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: webhookConfigName, }, Webhooks: []admissionregistrationv1.MutatingWebhook{{ Name: "sidecar-injector.morven.me", AdmissionReviewVersions: []string{"v1", "v1beta1"}, SideEffects: &sideEffect, ClientConfig: admissionregistrationv1.WebhookClientConfig{ CABundle: caPEM.Bytes(), // self-generated CA for the webhook Service: &admissionregistrationv1.ServiceReference{ Name: webhookService, Namespace: webhookNamespace, Path: &webhookInjectPath, }, }, Rules: []admissionregistrationv1.RuleWithOperations{ { Operations: []admissionregistrationv1.OperationType{ admissionregistrationv1.Create, admissionregistrationv1.Update, }, Rule: admissionregistrationv1.Rule{ APIGroups: []string{""}, APIVersions: []string{"v1"}, Resources: []string{"pods"}, }, }, }, NamespaceSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{ "sidecar-injection": "enabled", }, }, FailurePolicy: &fail, }}, } foundWebhookConfig, err := mutatingWebhookConfigV1Client.MutatingWebhookConfigurations().Get(context.TODO(), webhookConfigName, metav1.GetOptions{}) if err != nil && apierrors.IsNotFound(err) { if _, err := mutatingWebhookConfigV1Client.MutatingWebhookConfigurations().Create(context.TODO(), mutatingWebhookConfig, metav1.CreateOptions{}); err != nil { warningLogger.Printf("Failed to create the mutatingwebhookconfiguration: %s", webhookConfigName) return err } infoLogger.Printf("Created mutatingwebhookconfiguration: %s", webhookConfigName) } else if err != nil { warningLogger.Printf("Failed to check the mutatingwebhookconfiguration: %s", webhookConfigName) return err } else { // there is an existing mutatingWebhookConfiguration if len(foundWebhookConfig.Webhooks) != len(mutatingWebhookConfig.Webhooks) || !(foundWebhookConfig.Webhooks[0].Name == mutatingWebhookConfig.Webhooks[0].Name && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].AdmissionReviewVersions, mutatingWebhookConfig.Webhooks[0].AdmissionReviewVersions) && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].SideEffects, mutatingWebhookConfig.Webhooks[0].SideEffects) && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].FailurePolicy, mutatingWebhookConfig.Webhooks[0].FailurePolicy) && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].Rules, mutatingWebhookConfig.Webhooks[0].Rules) && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].NamespaceSelector, mutatingWebhookConfig.Webhooks[0].NamespaceSelector) && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].ClientConfig.CABundle, mutatingWebhookConfig.Webhooks[0].ClientConfig.CABundle) && reflect.DeepEqual(foundWebhookConfig.Webhooks[0].ClientConfig.Service, mutatingWebhookConfig.Webhooks[0].ClientConfig.Service)) { mutatingWebhookConfig.ObjectMeta.ResourceVersion = foundWebhookConfig.ObjectMeta.ResourceVersion if _, err := mutatingWebhookConfigV1Client.MutatingWebhookConfigurations().Update(context.TODO(), mutatingWebhookConfig, metav1.UpdateOptions{}); err != nil { warningLogger.Printf("Failed to update the mutatingwebhookconfiguration: %s", webhookConfigName) return err } infoLogger.Printf("Updated the mutatingwebhookconfiguration: %s", webhookConfigName) } infoLogger.Printf("The mutatingwebhookconfiguration: %s already exists and has no change", webhookConfigName) } return nil } ================================================ FILE: deploy/clusterrole.yaml ================================================ kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sidecar-injector labels: app: sidecar-injector rules: - apiGroups: ["admissionregistration.k8s.io"] resources: ["mutatingwebhookconfigurations"] verbs: ["create", "get", "delete", "list", "patch", "update", "watch"] ================================================ FILE: deploy/clusterrolebinding.yaml ================================================ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: sidecar-injector labels: app: sidecar-injector roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: sidecar-injector subjects: - kind: ServiceAccount name: sidecar-injector namespace: sidecar-injector ================================================ FILE: deploy/configmap.yaml ================================================ apiVersion: v1 kind: ConfigMap metadata: name: sidecar-injector labels: app: sidecar-injector data: sidecarconfig.yaml: | containers: - name: sidecar-nginx image: nginx:1.12.2 imagePullPolicy: IfNotPresent volumeMounts: - name: nginx-conf mountPath: /etc/nginx volumes: - name: nginx-conf configMap: name: nginx-configmap ================================================ FILE: deploy/deployment.yaml ================================================ apiVersion: apps/v1 kind: Deployment metadata: name: sidecar-injector labels: app: sidecar-injector spec: replicas: 1 selector: matchLabels: app: sidecar-injector template: metadata: labels: app: sidecar-injector spec: serviceAccountName: sidecar-injector containers: - name: sidecar-injector image: sidecar-injector imagePullPolicy: Always args: - -service-name=sidecar-injector - -sidecar-config-file=/etc/webhook/config/sidecarconfig.yaml env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace lifecycle: preStop: exec: command: ["/bin/sh", "-c", "/prestop.sh"] volumeMounts: - name: webhook-config mountPath: /etc/webhook/config volumes: - name: webhook-config configMap: name: sidecar-injector ================================================ FILE: deploy/kustomization.yaml ================================================ namespace: sidecar-injector resources: - namespace.yaml - clusterrole.yaml - clusterrolebinding.yaml - deployment.yaml - service.yaml - serviceaccount.yaml - configmap.yaml - nginx-configmap.yaml images: - name: sidecar-injector newName: quay.io/morvencao/sidecar-injector newTag: latest apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization ================================================ FILE: deploy/namespace.yaml ================================================ apiVersion: v1 kind: Namespace metadata: name: sidecar-injector ================================================ FILE: deploy/nginx-configmap.yaml ================================================ apiVersion: v1 kind: ConfigMap metadata: name: nginx-configmap data: nginx.conf: | worker_processes 1; events { worker_connections 1024; } http { default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root html; index index.html index.htm; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } ================================================ FILE: deploy/service.yaml ================================================ apiVersion: v1 kind: Service metadata: name: sidecar-injector labels: app: sidecar-injector spec: ports: - port: 443 targetPort: 8443 selector: app: sidecar-injector ================================================ FILE: deploy/serviceaccount.yaml ================================================ apiVersion: v1 kind: ServiceAccount metadata: name: sidecar-injector labels: app: sidecar-injector ================================================ FILE: go.mod ================================================ module github.com/morvencao/kube-sidecar-injector go 1.17 require ( gopkg.in/yaml.v2 v2.4.0 k8s.io/api v0.19.15 k8s.io/apimachinery v0.19.15 k8s.io/client-go v0.19.15 ) require ( github.com/davecgh/go-spew v1.1.1 // indirect github.com/go-logr/logr v0.2.0 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/golang/protobuf v1.4.2 // indirect github.com/google/gofuzz v1.1.0 // indirect github.com/googleapis/gnostic v0.4.1 // indirect github.com/imdario/mergo v0.3.5 // indirect github.com/json-iterator/go v1.1.10 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.1 // indirect github.com/spf13/pflag v1.0.5 // indirect golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 // indirect golang.org/x/net v0.0.0-20201110031124-69a78807bb2b // indirect golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 // indirect golang.org/x/sys v0.0.0-20201112073958-5cba982894dd // indirect golang.org/x/text v0.3.3 // indirect golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect google.golang.org/appengine v1.6.5 // indirect google.golang.org/protobuf v1.24.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect k8s.io/klog/v2 v2.2.0 // indirect k8s.io/utils v0.0.0-20200729134348-d5654de09c73 // indirect sigs.k8s.io/structured-merge-diff/v4 v4.1.2 // indirect sigs.k8s.io/yaml v1.2.0 // indirect ) ================================================ FILE: go.sum ================================================ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= cloud.google.com/go v0.51.0/go.mod h1:hWtGJ6gnXH+KgDv+V0zFGDvpi07n3z8ZNj3T1RW0Gcw= cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= github.com/Azure/go-autorest/autorest v0.9.6/go.mod h1:/FALq9T/kS7b5J5qsQ+RSTUdAmGFqi0vUdVNNx8q630= github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= github.com/Azure/go-autorest/autorest/adal v0.8.2/go.mod h1:ZjhuQClTqx435SRJ2iMlOxPYt3d2C/T/7TiQCVZSn3Q= github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= github.com/Azure/go-autorest/autorest/date v0.2.0/go.mod h1:vcORJHLJEh643/Ioh9+vPmf1Ij9AEBM5FuBIXLmIy0g= github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= github.com/Azure/go-autorest/autorest/mocks v0.3.0/go.mod h1:a8FDP3DYzQ4RYfVAxAN3SVSiiO77gL2j2ronKKP0syM= github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= github.com/go-logr/logr v0.2.0 h1:QvGt2nLcHH0WK9orKa+ppBPAxREcH364nPUedEpK0TY= github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0= github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc= github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I= github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/imdario/mergo v0.3.5 h1:JboBksRwiiAJWvIYJVo46AfV+IAIKZpfrSzVKj42R4Q= github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68= github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs= github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b h1:uwuIcX0g4Yl1NC5XAz37xsr2lTtcqevgzYNVt49waME= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201112073958-5cba982894dd h1:5CtCZbICpIOFdgO940moixOPjc0178IU44m4EjOO5IY= golang.org/x/sys v0.0.0-20201112073958-5cba982894dd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0= google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM= google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.24.0 h1:UhZDfRO8JRQru4/+LlLE0BRKGF8L+PICnvYZmx/fEGA= google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= k8s.io/api v0.19.15 h1:i22aQYrQ9gaBHEAS9XvyR5ZfrTDAd+Q+JwWM+xIBv30= k8s.io/api v0.19.15/go.mod h1:rMRWjnIJQmurd/FdLobht6dCSbJQ+UDpyOwPaoFS7lI= k8s.io/apimachinery v0.19.15 h1:P37ni6/yFxRMrqgM75k/vt5xq9vnNiR3rJPTmWXrNho= k8s.io/apimachinery v0.19.15/go.mod h1:RMyblyny2ZcDQ/oVE+lC31u7XTHUaSXEK2IhgtwGxfc= k8s.io/client-go v0.19.15 h1:lDBvFBjDIExh0mFS6JbG+5B7ghuPhqXjBzlaxG81ToU= k8s.io/client-go v0.19.15/go.mod h1:OJMQWgHQJRDtO2BVtpkHUQOq/e5WHpXc02lSdPI0S/k= k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= k8s.io/klog/v2 v2.2.0 h1:XRvcwJozkgZ1UQJmfMGpvRthQHOvihEhYtDfAaxMz/A= k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6/go.mod h1:UuqjUnNftUyPE5H64/qeyjQoUZhGpeFDVdxjTeEVN2o= k8s.io/utils v0.0.0-20200729134348-d5654de09c73 h1:uJmqzgNWG7XyClnU/mLPBWwfKKF1K8Hf8whTseBgJcg= k8s.io/utils v0.0.0-20200729134348-d5654de09c73/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= sigs.k8s.io/structured-merge-diff/v4 v4.1.2 h1:Hr/htKFmJEbtMgS/UD0N+gtgctAqz81t3nu+sPzynno= sigs.k8s.io/structured-merge-diff/v4 v4.1.2/go.mod h1:j/nl6xW8vLS49O8YvXW1ocPhZawJtm+Yrr7PPRQ0Vg4= sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q= sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= ================================================ FILE: medium-article.md ================================================ # Diving into Kubernetes MutatingAdmissionWebhook [Admission controllers](https://kubernetes.io/docs/admin/admission-controllers/) are powerful tools for intercepting requests to the Kubernetes API server prior to persistence of the object. However, they are not very flexible due to the requirement that they are compiled into binary into `kube-apiserver` and configured by the cluster administrator. Starting in Kubernetes 1.7, [Initializers](https://v1-8.docs.kubernetes.io/docs/admin/extensible-admission-controllers/#initializers) and [External Admission Webhooks](https://v1-8.docs.kubernetes.io/docs/admin/extensible-admission-controllers/#external-admission-webhooks) are introduced to address this limitation. In Kubernetes 1.9, `Initializers` stays in alpha phase while `External Admission Webhooks` have been promoted to beta and split into [MutatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#mutatingadmissionwebhook-beta-in-19) and [ValidatingAdmissionWebhook](https://kubernetes.io/docs/admin/admission-controllers/#validatingadmissionwebhook-alpha-in-18-beta-in-19). `MutatingAdmissionWebhook` together with `ValidatingAdmissionWebhook` are a special kind of `admission controllers` which process mutating and validating on requests matching the rules defined in [MutatingWebhookConfiguration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#mutatingwebhookconfiguration-v1beta1-admissionregistration)(explained below). In this article, we'll dive into the details of `MutatingAdmissionWebhook` and write a working webhook admission server step by step. ## Benefit of `Webhooks` `Webhooks` allow Kubernetes cluster-admin to create additional mutating and validating admission plugins to the admission chain of `apiserver` without recompiling them. This provides end-developer with the freedom and flexibility to customize admission logic on multiple actions("CREATA", "UPDATE", "DELETE"...) on any resource. The possible applications are vast. Some common use cases includes: - Mutating resources before creating them. [Istio](https://github.com/istio), a representative example, injecting [Envoy](https://github.com/envoyproxy/envoy) sidecar container to target pods to implement traffic management and policy enforcement. - Automated provisioning of `StorageClass`. Observes creation of `PersistentVolumeClaim` objects and automatically adds storage class to them based on predefined policy. Users that do not need to care about `StorageClass` creating. - Validating complex custom resource. Make sure custom resource can only be created after its definition and all dependencies created and available. - Restricting namespace. On multi-tenant systems, avoid resources created in reserved namespaces. Besides the user-cases listed above, many more aplications can be created based on the power of `webhooks`. ## `Webhooks` vs `Initializers` Based on feedback from the community and use cases in alpha phase of both `External Admission Webhooks` and `Initializers`, the Kubernetes community decided to promote webhooks to beta and split it into `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook`. These updates make webhooks consistent with other admission controllers and enforce `mutate-before-validate`. `Initializers` can also implement dynamic admission control by modifying Kubernetes resources before they are actually created. If you're unfamiliar with `Initializers`, please refer to the acrtcle: https://medium.com/ibm-cloud/kubernetes-initializers-deep-dive-and-tutorial-3bc416e4e13e. So what's the difference between `Webhooks` and `Initializers`? - `Webhooks` can be applied on more actions, including 'mutate' or 'admit' on resoures 'CREATE' 'UPDATE' and 'DELETE', whereas `Initializers` can't 'admit' resources for 'DELETE' requests. - `Webhooks` are not allowed to query resources before created, while `Initializers` are capable of watching the uninitialized resources by the query parameter `?includeUninitialized=true`, which makes resources creating progress transparent. - Since the `Initializers` persist the 'pre-create' states to `etcd`, higher latency and increased `etcd` burden will be introduced accordingly, especially when `apiserver` upgrades or fails. `Webhooks`, however, consume less memory and computing resources. - More robustness on failures for `Webhooks` than `Initializers`. Failure policy can be configured in `Webhooks` configuraton to avoid hanging onto resources that are created. Buggy `Initializers`, on the other hand, may block all matched resources creating. Besides the difference listed above, `Initializer` is stuck in some open issues with long expected development time including quota replenishment bug. Promotion of `Webhooks` to beta may be a signal that more support for it in the future, but that depends. If stable behavior is preferred, suggest you choose `Webhooks`. ## How MutatingAdmissionWebhook works `MutatingAdmissionWebhook` intercepts requests matching the rules defined in `MutatingWebhookConfiguration` before presisting into [etcd](https://github.com/coreos/etcd). `MutatingAdmissionWebhook` executes the mutation by sending admission requests to webhook server. Webhook server is just plain http server that adhere to the [API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go). The following diagram describes how `MutatingAdmissionWebhook` works in details: ![](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/mutating-admission-webhook.jpg) The `MutatingAdmissionWebhook` needs three objects to function: 1. **MutatingWebhookConfiguration** `MutatingAdmissionWebhook` need to be registered in the `apiserver` by providing `MutatingWebhookConfiguration`. During the registration process, MutatingAdmissionWebhook states: - How to connect to the webhook admission server - How to verify the webhook admission server - The URL path of the webhook admission server - Rules defining which resource and what action it handles - How unrecognized errors from the webhook admission server are handled 2. **MutatingAdmissionWebhook itself** `MutatingAdmissionWebhook` is a plugin-style admission controller that can be configured into the `apiserver`. The `MutatingAdmissionWebhook` plugin get the list of interested admission webhooks from `MutatingWebhookConfiguration`. Then the `MutatingAdmissionWebhook` observes the requests to apiserver and intercepts requests matching the rules in admission webhooks and calls them in parallel. 3. **Webhook Admission Server** `Webhook Admission Server` is just plain http server that adhere to Kubernetes [API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go). For each request to the `apiserver`, the `MutatingAdmissionWebhook` sends an `admissionReview`([API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go) for reference) to the relevant webhook admission server. The webhook admission server gathers information like `object`, `oldobject`, and `userInfo` from `admissionReview`, and sends back a `admissionReview` response including `AdmissionResponse` whose `Allowed` and `Result` fields are filled with the admission decision and optional `Patch` to mutate the resoures. ## Tutorial for MutatingAdmissionWebhook Write a complete Webhook Admission Server may be intimidating. To make it easier, we'll write a simple Webhook Admission Server that implements injecting nginx sidecar container and volume. The complete code can be found in [kube-mutating-webhook-tutorial](https://github.com/morvencao/kube-mutating-webhook-tutorial). The project refers to [Kunernetes webhook example](https://github.com/kubernetes/kubernetes/tree/release-1.9/test/images/webhook) and [Istio sidecar injection implementation](https://github.com/istio/istio/tree/master/pilot/pkg/kube/inject). In the following sections, I'll show you how to write a working containerized webhook admission server and deploy it to a Kubernetes cluster. #### Prerequisites `MutatingAdmissionWebhook` requires a Kubernetes 1.9.0 or above with the `admissionregistration.k8s.io/v1beta1` API enabled. Verify that by the following command: ``` kubectl api-versions | grep admissionregistration.k8s.io/v1beta1 ``` The result should be: ``` admissionregistration.k8s.io/v1beta1 ``` In addition, the `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers should be added and listed in the correct order in the `admission-control` flag of `kube-apiserver`. ### Write the Webhook Server `Webhook Admission Server` is just plain http server that adhere to Kubernetes [API](https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/apis/admission/types.go). I'll paste some pseudo code to describe the main logic: ``` sidecarConfig, err := loadConfig(parameters.sidecarCfgFile) pair, err := tls.LoadX509KeyPair(parameters.certFile, parameters.keyFile) whsvr := &WebhookServer { sidecarConfig: sidecarConfig, server: &http.Server { Addr: fmt.Sprintf(":%v", 443), TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}}, }, } // define http server and server handler mux := http.NewServeMux() mux.HandleFunc("/mutate", whsvr.serve) whsvr.server.Handler = mux // start webhook server in new rountine go func() { if err := whsvr.server.ListenAndServeTLS("", ""); err != nil { glog.Errorf("Filed to listen and serve webhook server: %v", err) } }() ``` Explanation for the above code: - `sidecarCfgFile` contains sidecar injector template defined in `ConfigMap` below. - `certFile` and `keyFile` key pair that will be needed for TLS communication between `apiserver` and `webhook server`. - Line 19 starts https server listening on 443 on path '/mutate'. Next we'll focus on the main logic of handler function `serve`: ``` // Serve method for webhook server func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) { var body []byte if r.Body != nil { if data, err := ioutil.ReadAll(r.Body); err == nil { body = data } } var reviewResponse *v1beta1.AdmissionResponse ar := v1beta1.AdmissionReview{} deserializer := codecs.UniversalDeserializer() if _, _, err := deserializer.Decode(body, nil, &ar); err != nil { glog.Error(err) reviewResponse = toAdmissionResponse(err) } else { reviewResponse = mutate(ar) } response := v1beta1.AdmissionReview{} if reviewResponse != nil { response.Response = reviewResponse response.Response.UID = ar.Request.UID } // reset the Object and OldObject, they are not needed in a response. ar.Request.Object = runtime.RawExtension{} ar.Request.OldObject = runtime.RawExtension{} resp, err := json.Marshal(response) if err != nil { glog.Error(err) } if _, err := w.Write(resp); err != nil { glog.Error(err) } } ``` The `serve` function is plain http handler with `http request` and `response writer` parameters. - Firstly unmarshals the request to `AdmissionReview`, which contains information like `object`, `oldobject` and `userInfo`... - Then calls Webhook core function `mutate` to create `patch` that injects sidecar container and volume. - Finally, unmarshals the response with admission decision and optional patch, sends it back to `apiserver`. For the part of `mutate` function, you get the free rein to complete it in your preferred way. Let's take my implementation as an example: ``` // main mutation process func (whsvr *WebhookServer) mutate(ar *v1beta1.AdmissionReview) *v1beta1.AdmissionResponse { req := ar.Request var pod corev1.Pod if err := json.Unmarshal(req.Object.Raw, &pod); err != nil { glog.Errorf("Could not unmarshal raw object: %v", err) return &v1beta1.AdmissionResponse { Result: &metav1.Status { Message: err.Error(), }, } } // determine whether to perform mutation if !mutationRequired(ignoredNamespaces, &pod.ObjectMeta) { glog.Infof("Skipping mutation for %s/%s due to policy check", pod.Namespace, pod.Name) return &v1beta1.AdmissionResponse { Allowed: true, } } annotations := map[string]string{admissionWebhookAnnotationStatusKey: "injected"} patchBytes, err := createPatch(&pod, whsvr.sidecarConfig, annotations) return &v1beta1.AdmissionResponse { Allowed: true, Patch: patchBytes, PatchType: func() *v1beta1.PatchType { pt := v1beta1.PatchTypeJSONPatch return &pt }(), } } ``` From the code above, the `mutate` function calls [mutationRequired](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go#L98-L130) to detemine whether mutation is required or not. For those requiring mutation, the `mutate` function gets mutation 'patch' from another function [createPatch](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go#L196-L205). Pay attention to the little trick in function `mutationRequired`, we skip the `pods` without annotation `sidecar-injector-webhook.morven.me/inject: true`. That will be mentioned latter when we deployment applications. For complete code, please refer to https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go. #### Create Dockerfile and Build the Container Create the `build` script: ``` dep ensure CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o kube-mutating-webhook-tutorial . docker build --no-cache -t morvencao/sidecar-injector:v1 . rm -rf kube-mutating-webhook-tutorial docker push morvencao/sidecar-injector:v1 ``` And create `Dockerfile` as dependency of build script: ``` FROM alpine:latest ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial ENTRYPOINT ["./kube-mutating-webhook-tutorial"] ``` Before actually building the container, you need a [Docker](https://hub.docker.com/) ID account and change the image name&tag(in `Dockerfile` and `deployment.yaml`) to yours, then execute: ``` [root@mstnode kube-mutating-webhook-tutorial]# ./build Sending build context to Docker daemon 44.89MB Step 1/3 : FROM alpine:latest ---> 3fd9065eaf02 Step 2/3 : ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial ---> 432de60c2b3f Step 3/3 : ENTRYPOINT ["./kube-mutating-webhook-tutorial"] ---> Running in da6e956d1755 Removing intermediate container da6e956d1755 ---> 619faa936145 Successfully built 619faa936145 Successfully tagged morvencao/sidecar-injector:v1 The push refers to repository [docker.io/morvencao/sidecar-injector] efd05fe119bb: Pushed cd7100a72410: Layer already exists v1: digest: sha256:7a4889928ec5a8bcfb91b610dab812e5228d8dfbd2b540cd7a341c11f24729bf size: 739 ``` #### Create Sidecar Injection Configuration Now let's create a Kubernetes `ConfigMap`, which includes `container` and `volume` information that will be injected into the target pod. ``` apiVersion: v1 kind: ConfigMap metadata: name: sidecar-injector-webhook-configmap data: sidecarconfig.yaml: | containers: - name: sidecar-nginx image: nginx:1.12.2 imagePullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - name: nginx-conf mountPath: /etc/nginx volumes: - name: nginx-conf configMap: name: nginx-configmap ``` From the above manifest, another ConfigMap including `nginx conf` is required. Here we put it in [nginxconfigmap.yaml](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deploy/nginxconfigmap.yaml). Then deploy the two `ConfigMap`s to cluster: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/nginxconfigmap.yaml configmap "nginx-configmap" created [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/configmap.yaml configmap "sidecar-injector-webhook-configmap" created ``` #### Create Secret Including Signed key/cert Pair Supporting `TLS` for external webhook server is required, because admission is a high security operation. so we need to create TLS certificate signed by `Kubernetes CA` for to secure the communcation between webhook server and `apiserver`. For the complete creating and approving `CSR` process, please refer to https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/. For simplicity purposes, we refer to the [script](https://github.com/istio/istio/blob/master/install/kubernetes/webhook-create-signed-cert.sh) from `Istio` and write a similar script called `webhook-create-signed-cert.sh` to automatically create the cert/key pair and include it in a Kubernetes `secret`. ``` #!/bin/bash while [[ $# -gt 0 ]]; do case ${1} in --service) service="$2" shift ;; --secret) secret="$2" shift ;; --namespace) namespace="$2" shift ;; esac shift done [ -z ${service} ] && service=sidecar-injector-webhook-svc [ -z ${secret} ] && secret=sidecar-injector-webhook-certs [ -z ${namespace} ] && namespace=default csrName=${service}.${namespace} tmpdir=$(mktemp -d) echo "creating certs in tmpdir ${tmpdir} " cat <> ${tmpdir}/csr.conf [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1 = ${service} DNS.2 = ${service}.${namespace} DNS.3 = ${service}.${namespace}.svc EOF openssl genrsa -out ${tmpdir}/server-key.pem 2048 openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf # clean-up any previously created CSR for our service. Ignore errors if not present. kubectl delete csr ${csrName} 2>/dev/null || true # create server cert/key CSR and send to k8s API cat <&2 exit 1 fi echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem # create the secret with CA cert and server cert/key kubectl create secret generic ${secret} \ --from-file=key.pem=${tmpdir}/server-key.pem \ --from-file=cert.pem=${tmpdir}/server-cert.pem \ --dry-run -o yaml | kubectl -n ${namespace} apply -f - ``` Then execute it and a Kubernetes `secret` including cert/key pair is created: ``` [root@mstnode kube-mutating-webhook-tutorial]# ./deploy/webhook-create-signed-cert.sh creating certs in tmpdir /tmp/tmp.wXZywp0wAF Generating RSA private key, 2048 bit long modulus ...........................................+++ ..........+++ e is 65537 (0x10001) certificatesigningrequest "sidecar-injector-webhook-svc.default" created NAME AGE REQUESTOR CONDITION sidecar-injector-webhook-svc.default 0s https://mycluster.icp:9443/oidc/endpoint/OP#admin Pending certificatesigningrequest "sidecar-injector-webhook-svc.default" approved secret "sidecar-injector-webhook-certs" created ``` #### Create the Sidecar Injector Deployment and Service The `deployment` brings up 1 `pod` in which the `sidecar-injector` container is running. The container starts with special arguments: - `sidecarCfgFile` pointing to the sidecar injector configuration file mounted from `sidecar-injector-webhook-configmap` ConfigMap created above - `tlsCertFile` and `tlsKeyFile` are cert/key pair mounted from `sidecar-injector-webhook-certs` Secret create by script above - `alsologtostderr` `v=4` and `2>&1` are logging arguments ``` apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sidecar-injector-webhook-deployment labels: app: sidecar-injector spec: replicas: 1 template: metadata: labels: app: sidecar-injector spec: containers: - name: sidecar-injector image: morvencao/sidecar-injector:v1 imagePullPolicy: IfNotPresent args: - -sidecarCfgFile=/etc/webhook/config/sidecarconfig.yaml - -tlsCertFile=/etc/webhook/certs/cert.pem - -tlsKeyFile=/etc/webhook/certs/key.pem - -alsologtostderr - -v=4 - 2>&1 volumeMounts: - name: webhook-certs mountPath: /etc/webhook/certs readOnly: true - name: webhook-config mountPath: /etc/webhook/config volumes: - name: webhook-certs secret: secretName: sidecar-injector-webhook-certs - name: webhook-config configMap: name: sidecar-injector-webhook-configmap ``` The `service` exposes the `pod` defined above labeled by `app=sidecar-injector` to make it accessible in cluster. This `service` will be referred by the `MutatingWebhookConfiguration` in `clientConfig` section and by default `spec.ports.port` should be **443**(default https port). ``` apiVersion: v1 kind: Service metadata: name: sidecar-injector-webhook-svc labels: app: sidecar-injector spec: ports: - port: 443 targetPort: 443 selector: app: sidecar-injector ``` Next we deploy the above `Deployment` and `Service` to cluster and verify the `sidecar injector` webhook server is running: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/deployment.yaml deployment "sidecar-injector-webhook-deployment" created [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/service.yaml service "sidecar-injector-webhook-svc" created [root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 2m [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 3m ``` #### Configure webhook admission controller on the fly `MutatingWebhookConfiguration` specifies which webhook admission servers are enabled and which resources are subject to the admission server. It is recommended that you firstly deploy the webhook admission server and make sure it is working properly before creating the `MutatingWebhookConfiguration`. Otherwise, requests will be unconditionally accepted or rejected based on `failurePolicy`. For now, we create the `MutatingWebhookConfiguration` manifest with the following content: ``` apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: sidecar-injector-webhook-cfg labels: app: sidecar-injector webhooks: - name: sidecar-injector.morven.me clientConfig: service: name: sidecar-injector-webhook-svc namespace: default path: "/mutate" caBundle: ${CA_BUNDLE} rules: - operations: [ "CREATE" ] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] namespaceSelector: matchLabels: sidecar-injector: enabled ``` Line 8: `name` - name for the webhook, should be fully qualified. Mutiple mutating webhooks are sorted by providing order. Line 9: `clientConfig` - describes how to connect to the webhook admission server and the TLS certificate. In our case, we specify the `sidecar injector` service. Line 15: `rules` - specifies what resources and what actions the webhook server handles. In our case, only intercepts request for creating of pods. Line 20: `namespaceSelector` - `namespaceSelector` decides whether to send admission request the webhook server on an object based on whether the namespace for that object matches the selector. Before deploying the `MutatingWebhookConfiguration`, we need to replace the `${CA_BUNDLE}` with apiserver's default `caBundle`. Let's write the script `webhook-patch-ca-bundle.sh` to automate this process: ``` #!/bin/bash set -o errexit set -o nounset set -o pipefail ROOT=$(cd $(dirname $0)/../../; pwd) export CA_BUNDLE=$(kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n') if command -v envsubst >/dev/null 2>&1; then envsubst else sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g" fi ``` Then execute: ``` [root@mstnode kube-mutating-webhook-tutorial]# cat ./deploy/mutatingwebhook.yaml |\ > ./deploy/webhook-patch-ca-bundle.sh >\ > ./deploy/mutatingwebhook-ca-bundle.yaml ``` Finally we can deploy `MutatingWebhookConfiguration`: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deploy/mutatingwebhook-ca-bundle.yaml mutatingwebhookconfiguration "sidecar-injector-webhook-cfg" created [root@mstnode kube-mutating-webhook-tutorial]# kubectl get mutatingwebhookconfiguration NAME AGE sidecar-injector-webhook-cfg 11s ``` #### Verification and Troubleshooting Now it's time to verify sidecar injector works as expected and try to see how to troubleshoot if you encounter issues. Typically we create and deploy a sleep application in `default` namespace to see if the sidecar can be injected. ``` [root@mstnode kube-mutating-webhook-tutorial]# cat < apiVersion: extensions/v1beta1 > kind: Deployment > metadata: > name: sleep > spec: > replicas: 1 > template: > metadata: > annotations: > sidecar-injector-webhook.morven.me/inject: "true" > labels: > app: sleep > spec: > containers: > - name: sleep > image: tutum/curl > command: ["/bin/sleep","infinity"] > imagePullPolicy: IfNotPresent > EOF deployment "sleep" created ``` Pay close attention to the `spec.template.metadata.annotations` as there is a new annotation added: ``` sidecar-injector-webhook.morven.me/inject: "true" ``` The sidecar injector has some logic to check the existence of the above annotation before injecting sidecar container and volume. You're free to delete the logic or customize it before build the sidecar injector container. Check the `deployment` and `pod`: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 18m sleep 1 1 1 1 58s [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 18m sleep-6d79d8dc54-r66vz 1/1 Running 0 1m ``` It's not there. What's going on? Let's check the sidecar injector logs: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl logs -f sidecar-injector-webhook-deployment-bbb689d69-fdbgj I0314 08:48:15.140858 1 webhook.go:88] New configuration: sha256sum 21669464280f76170b88241fd79ecbca3dcebaec5c152a4a9a3e921ff742157f ``` We can't find any logs that indicate webhook server got admission request, seems that request hadn't been sent to `sidecar injector` webhook server. So there is a possibility that the issue is caused by configuration in `MutatingWebhookConfiguration`. Do a double check of `MutatingWebhookConfiguration` and we find following content: ``` namespaceSelector: matchLabels: sidecar-injector: enabled ``` #### Control sidecar injector with `namespaceSelector` We have configured 'namespaceSelector' in `MutatingWebhookConfiguration`, which means only resources in namespace matching the selector will be sent to webhook server. So we need label the `default` namespace with `sidecar-injector=enabled`: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl label namespace default sidecar-injector=enabled namespace "default" labeled [root@mstnode kube-mutating-webhook-tutorial]# kubectl get namespace -L sidecar-injector NAME STATUS AGE sidecar-injector default Active 1d enabled kube-public Active 1d kube-system Active 1d ``` We have configured the `MutatingWebhookConfiguration` resulting in the sidecar injection occuring at pod creation time. Kill the running pod and verify a new pod is created with the injected sidecar. ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-6d79d8dc54-r66vz pod "sleep-6d79d8dc54-r66vz" deleted [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 29m sleep-6d79d8dc54-b8ztx 0/2 ContainerCreating 0 3s sleep-6d79d8dc54-r66vz 1/1 Terminating 0 11m [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod sleep-6d79d8dc54-b8ztx -o yaml apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/psp: default sidecar-injector-webhook.morven.me/inject: "true" sidecar-injector-webhook.morven.me/status: injected labels: app: sleep pod-template-hash: "2835848710" name: sleep-6d79d8dc54-b8ztx namespace: default spec: containers: - command: - /bin/sleep - infinity image: tutum/curl imagePullPolicy: IfNotPresent name: sleep resources: {} volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-d7t2r readOnly: true - image: nginx:1.12.2 imagePullPolicy: IfNotPresent name: sidecar-nginx ports: - containerPort: 80 protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/nginx name: nginx-conf volumes: - name: default-token-d7t2r secret: defaultMode: 420 secretName: default-token-d7t2r - configMap: defaultMode: 420 name: nginx-configmap name: nginx-conf ... ``` We can see that sidecar container and volume have been injected into sleep application successfully. Until now, we have working sidecar injector with `MutatingAdmissionWebhook`. With `namespaceSelector` we can easily control whether the pods in specified namespace will be injected or not. But there is a problem for this, with the above configurations, all of the pods in `default` namespace will be injected with a sidecar, this may be not expected for some cases. #### Control sidecar injector with `annotation` Thanks to flexibility of `MutatingAdmissionWebhook`, we can easily customized the mutating logic to filter resources with specified annotations. Remember the annotation `sidecar-injector-webhook.morven.me/inject: "true"` mentioned above? It can be used as an extra control on sidecar injector. I have written [some code](https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/webhook.go#L98-L130) in webhook server to skip injecting for pod without the annotation. Let's give it a try. In this case, we create another sleep application without `sidecar-injector-webhook.morven.me/inject: "true"` annotation in `podTemplateSpec`: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl delete deployment sleep deployment "sleep" deleted [root@mstnode kube-mutating-webhook-tutorial]# cat < kind: Deployment > metadata: > name: sleep > spec: > replicas: 1 > template: > metadata: > labels: > app: sleep > spec: > containers: > - name: sleep > image: tutum/curl > command: ["/bin/sleep","infinity"] > imagePullPolicy: IfNotPresent > EOF deployment "sleep" created ``` And then verify the sidecar injector skipped the pod: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 45m sleep 1 1 1 1 17s [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 45m sleep-776b7bcdcd-4bz58 1/1 Running 0 21s ``` The output shows that the sleep application contains only one container, no extra container and volume injected. Then we patch the sleep deployment to add the additional annotation and verify it will be injected after recreated: ``` [root@mstnode kube-mutating-webhook-tutorial]# kubectl patch deployment sleep -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar-injector-webhook.morven.me/inject": "true"}}}}}' deployment "sleep" patched [root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-776b7bcdcd-4bz58 pod "sleep-776b7bcdcd-4bz58" deleted [root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 49m sleep-3e42ff9e6c-6f87b 0/2 ContainerCreating 0 18s sleep-776b7bcdcd-4bz58 1/1 Terminating 0 3m ``` As expected, the pod has been injected with extra sidecar container. Now, we got working sidecar injector with `mutatingAdmissionWebhook` and its coarse-grained control by `namespaceSelector` and fine-grained control by additional `annotation`. ## Conclusion `MutatingAdmissionWebhook` is one of easiest ways of extending Kubernetes with new policy controls, resources mutation... This feature will enable more workloads and support more ecosystem components, including [Istio](https://github.com/istio/istio) service mesh platform. Starting with Istio 0.5.0, Istio has refactored to support their auto injection code with `MutatingAdmissionWebhook` replacing `initializers`. ## Reference - http://blog.kubernetes.io/2018/01/extensible-admission-is-beta.html - https://docs.google.com/document/d/1c4kdkY3ha9rm0OIRbGleCeaHknZ-NR1nNtDp-i8eH8E/view - https://v1-8.docs.kubernetes.io/docs/admin/extensible-admission-controllers/ - https://github.com/kubernetes/kubernetes/tree/release-1.9/test/images/webhook ================================================ FILE: mutating-admission-webhook.xml ================================================ 7Vzbkps4EP0aV+0+2CUB4vI4M8lsHpKt2clWsvuIQbbZwUAAzyVfvwIkGwnJ2OZiT8qkamI3Qojuo9PdauGJfrd+/SN1k9WX2MfhRAP+60T/MNE0CKFN/iskb5XEhE4lWKaBX4nATvA1+InplUy6CXycUVklyuM4zIOEF3pxFGEv52RumsYvfLNFHPqcIHGXuCH46rlhU/o98PMVey4Adic+4WC5ore2ET0xd72nZRpvInq/iaYvyqM6vXZZX7R9tnL9+KUm0j9O9Ls0jvPq0/r1DoeFbnm13SvObsed4ig/5AJUXfDshhv66E+bOZ66SZDh9BmndJT5G9NMjl9Jx7erfB0SASQfszyNn/BdHMYpkURxRFreLoIwFERuGCwj8tUjQyMd67ek+zwgOr+hJ9aB7xe3uX1ZBTn+mrhecc8XgjAiK3WKi1GD4p5MbcWX0J3j8HareOG29AHJvfCrUklwq3oCaRyvcZ6+kSb0Aocai4JZZyh92UGDNVnVQKFTmUvBuNx2vDMI+UBtIreP1bBPwyK8ZmS6qxurrjjecgSqoDy2Zxj0NblBB9e7BWeoVfMatJqqN4wedO9IdG+GBfr94Jl8XOblY1Yij3CgKFvE5DnrtjJ/bGJ2YpqVrHdDGjjJ6+4c6+RPd42zwo6fgwX23rwQs27JuKue+bsRsWwQRMwNV4keOBB6oBw9xSPcu+sgLGz7CYfPuCCDMWAF2SUMVUYTVVB3JBMadQcVhEpUJWnDeL2j6nOwDvJHN1oW1H4inLhxXuHUgJMzJpy0Y+A0BsK+krgh8PCN5xF15/2A7Aq9w6BnjMpk+qVB7wGnWZDlRHnf4nCzxp8L/V8ROCoCHTQiAo2OCBwDlF82uZsH0fLGXwdZFsTRdzxfxfHTkLi8ovUwtKJRXXUz1z4rWm0ZWr+RLN2/4vVC8WqO6t/Ny8KrlF0fcRZvUg//RcTuFaIXANFRKdVWQrS7z8bTh9gvHoTxH2BLogqUjb4ud3FgMAC/UmoYehMMaKClUk29ssLAwOoMzN5Qafoou1q+i+URkNDAYJaXLYIINsGRf1MUhsg3L3RJYOPxZlCqTm2SQ5SK/UahqVWlNZXJNMZkKQ5JoPbMdy9TI73DQxyUs4CZ2BGWrSyBkCvHSq+ql5DaOnIEk+ZuusR5o6PSqtvHPszQshD6aug2QwukrJ1saJHdhzO0zLEPb2hisPTtn4KLZ8VCBhX8SwRTMAMkIKaSB5wG5GmKKubBFch2gGigUiALmi4KNH2xw3aBaADQqAt281OTDmjK4sNZebSnG3NliHCBZXTlfFmEQfKJjnIVp8FP0tANj4B+twTDEiILSQ6sSyZCHxVg1u/YNPQa5BULIQDp94KEIOEgRL+LFFSjLo64+qSnas5PuLVXjrLQmSmLsHbt4GBj9uX1zOG8ni7LYEaFWx1s5wAVkoDKPCeoNABmNrKgoVV/LZsHg45mu5Pk72kYK+6i28BBWvXX4G+ChkOcLHM6NWe+e/xwTZq7JM2mLimeDZU06+ryrWR7U5a40ckxFNhXF6Plhbs4WgTLTUpkcdQeWcnG07LTiWAmwaX9clV0VEMYv5Q7Ct7oVCvTDDGoy+NkDETqJo/IbfTObbiTIBKafUBSVs9V+795GHtPf6+CaF8gXTePUvNbD1jzf5XLK3bgEZBOqk2Q/23WCd2sXPlD/z4IWSS8c5acqyy7GSsC08+bNArosSwBFIf6QxGGjWBe4QEJMty3WrOkaJCpB2wI/AvsvcMSmmsWt7eafKjuf7IzVheIu9DvvgpHwlc4THdd0Ew0zxKeVw9i4SNTXDIzzfIQaLLYMo+Kf2dJdkdw+vye5u2rAHWnr0kmqdYHw8r2kw8LMb6SckXYGRCGmk58OISpS7JDIUzIO64QGx9i+pgkZshW5bqu82qqdd7rKu8ogDIEzrLHW+U11BsHrnWD94soxEfrCI2IKNmyWutCbj+p7J4sdJevwoZJWQoMZwAYk3oarNmsBjFU2bO1rmDLLT3SErCQjJrwxKzWbsmOe0pqdV3MasHeYYntDcRlwZ3TWkO90Hgom1oFmYISU7U3rKdeBbiiRbqc/0ameNFKIyMCwuffm1wsi0pdtuf5ET8H+GVPLFm/KsU/NjhTv+rUSuEpJo/qzssGxVNSC5PW6HaCPhxF3vvnpJI2GvysZGD6yj0d7mQb0dXndcV/SmoGM9NhOQkF3VTrZa5OdSC8MoxMvo94schw17KJcdxCZa/8znh6Ii5VCnzO/MCW0He+AJrKnSy8ixhkX4txTjJHhjOzawdfJDaQJTt7dDnPBDNLqx08Ih1rptcOYxAvgMy9rN7eXm/xGrBbe+SY/XoZ2dsK7yfk0vjpbJ054jrvjsVmxCXMkUMnoiGC1LSHCbmQPERUTgYxN6EbH3qbDOpl3mvI9YuGXIptMsOHXJDfEsPC/X4DrpP2A/cbcBGW5oMuw9ofdfFbpmYGMBsOQbmTbwziv8DoDO2Nzsx+ojNrX3RmglGiM3FTK/VwyuhMaG84LQ6mY3sTcOPp7pDUW7N/FYeUJXGUqX+j6d15JNjZIylW7Qb3SNvCE+sWWsIs7sUnIfV2jaMwLYKyAUDw+B7ARZzpfXnsQ9iQgJqCmQ41UzB9TzHOEPj5lTJWk3jj86asumLNcaSUlQQVwq5tqM0E2rm4tFXcf9wShTTao343wKFr2np+Hh83SEBnS1s1R6gUTIUXhPoh+QtIXCdHVAp03iHYsOkSRstP2cspF5SfmmBWf9tH+DUpTefeFeqQn9bvYYh+xZCNYeAEVaD69vZ2S8IpvlV3bHv6g+J9uR624tzB9dgDuJ53mgu0+ZBuLuNdpwHk6+7X3avmu5/Q1z/+Dw== ================================================ FILE: prestop.sh ================================================ set -exo pipefail # Point to the internal API server hostname APISERVER=https://kubernetes.default.svc # Path to ServiceAccount token SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount # Read this Pod's namespace NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) # Read the ServiceAccount bearer token TOKEN=$(cat ${SERVICEACCOUNT}/token) # Reference the internal certificate authority (CA) CACERT=${SERVICEACCOUNT}/ca.crt MutatingWebhookConfigurationName=sidecar-injector-webhook # Delete the validatingwebhookconfiguration with TOKEN curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X DELETE ${APISERVER}/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/${MutatingWebhookConfigurationName}