Showing preview only (321K chars total). Download the full file or copy to clipboard to get everything.
Repository: Azure/kubernetes-kms
Branch: master
Commit: 1a9b8f1fcd7f
Files: 79
Total size: 299.5 KB
Directory structure:
gitextract_y5iov8qc/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── dependabot.yml
│ ├── semantic.yml
│ └── workflows/
│ ├── codeql.yaml
│ ├── create-release.yml
│ ├── dependency-review.yml
│ └── scorecards.yml
├── .gitignore
├── .golangci.yml
├── .goreleaser.yml
├── .pipelines/
│ ├── nightly.yml
│ ├── pr.yml
│ └── templates/
│ ├── cleanup-template.yml
│ ├── cluster-health-template.yml
│ ├── e2e-kind-template.yml
│ ├── e2e-upgrade-template.yml
│ ├── kind-debug-template.yml
│ ├── manifest-template.yml
│ ├── prepare-deps.yaml
│ ├── scan-images-template.yml
│ └── unit-tests-template.yml
├── AUTHORS
├── CODEOWNERS
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── SECURITY.md
├── cmd/
│ └── server/
│ └── main.go
├── developers.md
├── docs/
│ ├── manual-install.md
│ ├── metrics.md
│ ├── rotation.md
│ └── testing.md
├── go.mod
├── go.sum
├── pkg/
│ ├── auth/
│ │ ├── auth.go
│ │ └── auth_test.go
│ ├── config/
│ │ └── azure_config.go
│ ├── consts/
│ │ └── consts.go
│ ├── metrics/
│ │ ├── exporter.go
│ │ ├── exporter_test.go
│ │ ├── prometheus_exporter.go
│ │ └── stats_reporter.go
│ ├── plugin/
│ │ ├── healthz.go
│ │ ├── healthz_test.go
│ │ ├── keyvault.go
│ │ ├── keyvault_test.go
│ │ ├── kms_v2_server.go
│ │ ├── kms_v2_server_test.go
│ │ ├── mock_keyvault/
│ │ │ └── keyvault_mock.go
│ │ ├── server.go
│ │ └── server_test.go
│ ├── utils/
│ │ ├── grpc.go
│ │ ├── grpc_test.go
│ │ ├── sanitize.go
│ │ └── sanitize_test.go
│ └── version/
│ ├── version.go
│ └── version_test.go
├── scripts/
│ ├── connect-registry.sh
│ ├── setup-kind-cluster.sh
│ ├── setup-kmsv2-kind-cluster.sh
│ └── setup-local-registry.sh
├── tests/
│ ├── client/
│ │ └── client_test.go
│ └── e2e/
│ ├── azure.json
│ ├── encryption-config.yaml
│ ├── helpers.bash
│ ├── kind-config.yaml
│ ├── kms.yaml
│ ├── kmsv2-encryption-config.yaml
│ ├── test.bats
│ └── testkmsv2.bats
└── tools/
├── go.mod
├── go.sum
└── tools.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug report
about: Create a report to help KMS Plugin for Key Vault improve
title: ''
labels: bug
assignees: ''
---
**Describe the bug**
**Steps To Reproduce**
**Expected behavior**
**KMS Plugin for Key Vault version**
**Kubernetes version**
**Additional context**
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md
================================================
---
name: Feature request
about: Suggest an idea for KMS Plugin for Key Vault
title: ''
labels: enhancement
assignees: ''
---
**Describe the request**
**Explain why KMS Plugin for Key Vault needs it**
**Describe the solution you'd like**
**Describe alternatives you've considered**
**Additional context**
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
<!-- Thank you for helping KMS Plugin for Key Vault with a pull request! -->
**Reason for Change**:
<!-- What does this PR improve or fix in KMS Plugin for Key Vault? Why is it needed? -->
**Issue Fixed**:
<!-- If this PR fixes GitHub issue 1234, add "Fixes #1234" to the next line. -->
**Notes for Reviewers**:
================================================
FILE: .github/dependabot.yml
================================================
version: 2
updates:
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "weekly"
commit-message:
prefix: "chore"
ignore:
- dependency-name: "*"
update-types:
- "version-update:semver-major"
- "version-update:semver-minor"
- package-ecosystem: github-actions
directory: /
schedule:
interval: daily
commit-message:
prefix: "chore"
- package-ecosystem: docker
directory: /
schedule:
interval: daily
commit-message:
prefix: "chore"
- package-ecosystem: gomod
directory: /tools
schedule:
interval: daily
commit-message:
prefix: "chore"
================================================
FILE: .github/semantic.yml
================================================
titleOnly: true
types:
- chore
- ci
- docs
- feat
- fix
- perf
- refactor
- release
- revert
- security
- test
================================================
FILE: .github/workflows/codeql.yaml
================================================
name: "CodeQL"
on:
push:
branches:
- master
pull_request:
branches:
- master
schedule:
- cron: "0 15 * * 1" # Mondays at 7:00 AM PST
permissions: read-all
jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
security-events: write
steps:
- name: Harden Runner
uses: step-security/harden-runner@6b3083af2869dc3314a0257a42f4af696cc79ba3 # v2.3.1
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab
- name: Initialize CodeQL
uses: github/codeql-action/init@b2c19fb9a2a485599ccf4ed5d65527d94bc57226
with:
languages: go
- name: Autobuild
uses: github/codeql-action/autobuild@b2c19fb9a2a485599ccf4ed5d65527d94bc57226
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@b2c19fb9a2a485599ccf4ed5d65527d94bc57226
================================================
FILE: .github/workflows/create-release.yml
================================================
name: create_release
on:
push:
tags:
- 'v*'
permissions:
contents: write
jobs:
create-release:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@eb238b55efaa70779f274895e782ed17c84f2895 # v2.6.1
with:
egress-policy: audit
- uses: actions/checkout@9bb56186c3b09b4f86b1c65136769dd318469633 # v4.1.2
with:
submodules: true
fetch-depth: 0
- name: Goreleaser
uses: goreleaser/goreleaser-action@336e29918d653399e599bfca99fadc1d7ffbc9f7 # v4.3.0
with:
version: "~> v2"
args: release --clean --fail-fast --timeout 60m --verbose
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
================================================
FILE: .github/workflows/dependency-review.yml
================================================
# Dependency Review Action
#
# This Action will scan dependency manifest files that change as part of a Pull Request,
# surfacing known-vulnerable versions of the packages declared or updated in the PR.
# Once installed, if the workflow run is marked as required,
# PRs introducing known-vulnerable packages will be blocked from merging.
#
# Source repository: https://github.com/actions/dependency-review-action
name: 'Dependency Review'
on: [pull_request]
permissions:
contents: read
jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@6b3083af2869dc3314a0257a42f4af696cc79ba3 # v2.3.1
with:
egress-policy: audit
- name: 'Checkout Repository'
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2
- name: 'Dependency Review'
uses: actions/dependency-review-action@0efb1d1d84fc9633afcdaad14c485cbbc90ef46c # v2.5.1
================================================
FILE: .github/workflows/scorecards.yml
================================================
# This workflow uses actions that are not certified by GitHub. They are provided
# by a third-party and are governed by separate terms of service, privacy
# policy, and support documentation.
name: Scorecard supply-chain security
on:
# For Branch-Protection check. Only the default branch is supported. See
# https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection
branch_protection_rule:
# To guarantee Maintained check is occasionally updated. See
# https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained
schedule:
- cron: '20 7 * * 2'
push:
branches: ["master"]
# Declare default permissions as read only.
permissions: read-all
jobs:
analysis:
name: Scorecard analysis
runs-on: ubuntu-latest
permissions:
# Needed to upload the results to code-scanning dashboard.
security-events: write
# Needed to publish results and get a badge (see publish_results below).
id-token: write
contents: read
actions: read
steps:
- name: Harden Runner
uses: step-security/harden-runner@6b3083af2869dc3314a0257a42f4af696cc79ba3 # v2.3.1
with:
egress-policy: audit
- name: "Checkout code"
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2
with:
persist-credentials: false
- name: "Run analysis"
uses: ossf/scorecard-action@0864cf19026789058feabb7e87baa5f140aac736 # v2.3.1
with:
results_file: results.sarif
results_format: sarif
# (Optional) "write" PAT token. Uncomment the `repo_token` line below if:
# - you want to enable the Branch-Protection check on a *public* repository, or
# - you are installing Scorecards on a *private* repository
# To create the PAT, follow the steps in https://github.com/ossf/scorecard-action#authentication-with-pat.
# repo_token: ${{ secrets.SCORECARD_TOKEN }}
# Public repositories:
# - Publish results to OpenSSF REST API for easy access by consumers
# - Allows the repository to include the Scorecard badge.
# - See https://github.com/ossf/scorecard-action#publishing-results.
# For private repositories:
# - `publish_results` will always be set to `false`, regardless
# of the value entered here.
publish_results: true
# Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
# format to the repository Actions tab.
- name: "Upload artifact"
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
with:
name: SARIF file
path: results.sarif
retention-days: 5
# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@8662eabe0e9f338a07350b7fd050732745f93848 # v2.3.1
with:
sarif_file: results.sarif
================================================
FILE: .gitignore
================================================
# Binaries for programs and plugins
*.exe
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Project-local glide cache, RE: https://github.com/Masterminds/glide/issues/736
.glide/
setenv.sh
kubernetes-kms
vendor
*.env
# Vscode files
.vscode
# OSX trash
.DS_Store
.idea/
_output/
# e2e output
tests/e2e/generated_manifests/*
# Go tools
.tools/
================================================
FILE: .golangci.yml
================================================
version: "2"
run:
go: "1.26"
linters:
default: none
enable:
- errorlint
- goconst
- gocyclo
- gosec
- govet
- ineffassign
- misspell
- nakedret
- prealloc
- revive
- staticcheck
- unconvert
- unused
- whitespace
exclusions:
generated: lax
presets:
- comments
- common-false-positives
- legacy
- std-error-handling
rules:
- linters:
- staticcheck
text: "SA1019: .*(v1beta1|KMSv1 is deprecated)"
paths:
- third_party$
- builtin$
- examples$
settings:
revive:
rules:
- name: var-naming
disabled: true
staticcheck:
checks:
- all
formatters:
enable:
- gofmt
- goimports
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
================================================
FILE: .goreleaser.yml
================================================
# refer to https://goreleaser.com for more options
version: 2
builds:
- skip: true
release:
prerelease: auto
header: |
## {{.Tag}} - {{ time "2006-01-02" }}
changelog:
disable: false
groups:
- title: Bug Fixes 🐞
regexp: ^.*fix[(\\w)]*:+.*$
- title: Build 🏭
regexp: ^.*build[(\\w)]*:+.*$
- title: Code Refactoring 💎
regexp: ^.*refactor[(\\w)]*:+.*$
- title: Code Style 🎶
regexp: ^.*style[(\\w)]*:+.*$
- title: Continuous Integration 💜
regexp: ^.*ci[(\\w)]*:+.*$
- title: Documentation 📘
regexp: ^.*docs[(\\w)]*:+.*$
- title: Features 🌈
regexp: ^.*feat[(\\w)]*:+.*$
- title: Maintenance 🔧
regexp: ^.*chore[(\\w)]*:+.*$
- title: Performance Improvements 🚀
regexp: ^.*perf[(\\w)]*:+.*$
- title: Revert Change ◀️
regexp: ^.*revert[(\\w)]*:+.*$
- title: Security Fix 🛡️
regexp: ^.*security[(\\w)]*:+.*$
- title: Testing 💚
regexp: ^.*test[(\\w)]*:+.*$
================================================
FILE: .pipelines/nightly.yml
================================================
trigger: none
schedules:
- cron: "0 0 * * *"
always: true
displayName: "Nightly Build & Test"
branches:
include:
- master
pool: staging-pool-amd64-mariner-2
jobs:
- template: templates/unit-tests-template.yml
- template: templates/e2e-upgrade-template.yml
================================================
FILE: .pipelines/pr.yml
================================================
trigger:
branches:
include:
- master
pr:
branches:
include:
- master
paths:
exclude:
- docs/*
- README.md
- .github/*
pool: staging-pool-amd64-mariner-2
jobs:
- template: templates/unit-tests-template.yml
- template: templates/e2e-kind-template.yml
================================================
FILE: .pipelines/templates/cleanup-template.yml
================================================
steps:
- script: |
kubectl logs -l component=azure-kms-provider -n kube-system --tail -1
kubectl get pods -o wide -A
displayName: "Get logs"
- script: make e2e-delete-kind
displayName: "Delete cluster"
================================================
FILE: .pipelines/templates/cluster-health-template.yml
================================================
steps:
- script: |
kubectl wait --for=condition=ready node --all
kubectl wait pod -n kube-system --for=condition=Ready --all
kubectl get nodes -owide
displayName: "Check cluster health"
================================================
FILE: .pipelines/templates/e2e-kind-template.yml
================================================
jobs:
- job:
timeoutInMinutes: 15
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
- name: REGISTRY_NAME
value: kind-registry
- name: REGISTRY_PORT
value: 5000
- name: KUBERNETES_VERSION
value: v1.32.3
- name: KIND_CLUSTER_NAME
value: kms
- name: KIND_NETWORK
value: kind
# contains the following environment variables:
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
# - USER_ASSIGNED_IDENTITY_ID
- group: kubernetes-kms
strategy:
matrix:
kmsv1_kind_v1_33_7:
KUBERNETES_VERSION: v1.33.7
kmsv1_kind_v1_34_3:
KUBERNETES_VERSION: v1.34.3
kmsv1_kind_v1_35_0:
KUBERNETES_VERSION: v1.35.0
steps:
- task: GoTool@0
inputs:
version: 1.26.2
- template: prepare-deps.yaml
- script: make e2e-install-prerequisites
displayName: "Install e2e test prerequisites"
- script: |
make e2e-setup-kind
displayName: "Setup kind cluster with azure kms plugin"
env:
REGISTRY_NAME: $(REGISTRY_NAME)
REGISTRY_PORT: $(REGISTRY_PORT)
KUBERNETES_VERSION: $(KUBERNETES_VERSION)
KIND_CLUSTER_NAME: $(KIND_CLUSTER_NAME)
KIND_NETWORK: $(KIND_NETWORK)
- template: cluster-health-template.yml
- template: kind-debug-template.yml
- script: make e2e-test
displayName: "Run e2e tests for KMS v1"
- template: cleanup-template.yml
- job:
timeoutInMinutes: 15
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
- name: REGISTRY_NAME
value: kind-registry
- name: REGISTRY_PORT
value: 5000
- name: KUBERNETES_VERSION
value: v1.32.3
- name: KIND_CLUSTER_NAME
value: kms
- name: KIND_NETWORK
value: kind
# contains the following environment variables:
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
# - USER_ASSIGNED_IDENTITY_ID
- group: kubernetes-kms
strategy:
matrix:
kmsv2_kind_v1_33_7:
KUBERNETES_VERSION: v1.33.7
kmsv2_kind_v1_34_3:
KUBERNETES_VERSION: v1.34.3
kmsv2_kind_v1_35_0:
KUBERNETES_VERSION: v1.35.0
steps:
- task: GoTool@0
inputs:
version: 1.26.2
- template: prepare-deps.yaml
- script: make e2e-install-prerequisites
displayName: "Install e2e test prerequisites"
- script: |
make e2e-kmsv2-setup-kind
displayName: "Setup kind cluster with azure kms plugin"
env:
REGISTRY_NAME: $(REGISTRY_NAME)
REGISTRY_PORT: $(REGISTRY_PORT)
KUBERNETES_VERSION: $(KUBERNETES_VERSION)
KIND_CLUSTER_NAME: $(KIND_CLUSTER_NAME)
KIND_NETWORK: $(KIND_NETWORK)
- template: cluster-health-template.yml
- template: kind-debug-template.yml
- script: make e2e-kmsv2-test
displayName: "Run e2e tests for KMS v2"
- template: cleanup-template.yml
================================================
FILE: .pipelines/templates/e2e-upgrade-template.yml
================================================
jobs:
- job: e2e_upgrade_tests
timeoutInMinutes: 10
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
- name: REGISTRY_NAME
value: kind-registry
- name: REGISTRY_PORT
value: 5000
- name: KUBERNETES_VERSION
value: v1.23.5
- name: KIND_CLUSTER_NAME
value: kms
- name: KIND_NETWORK
value: kind
# contains the following environment variables:
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
# - USER_ASSIGNED_IDENTITY_ID
- group: kubernetes-kms
steps:
- task: GoTool@0
inputs:
version: 1.26.2
- template: prepare-deps.yaml
- script: make e2e-install-prerequisites
displayName: "Install e2e test prerequisites"
- script: |
. scripts/setup-local-registry.sh
displayName: "Setup local registry"
env:
REGISTRY_NAME: $(REGISTRY_NAME)
REGISTRY_PORT: $(REGISTRY_PORT)
- script: |
version=$(git tag -l --sort=v:refname | tail -n 1)
echo "##vso[task.setvariable variable=LATEST_KMS_VERSION]$version"
echo "Latest released kms version - $version"
displayName: "Get latest released version"
- template: manifest-template.yml
parameters:
registry: mcr.microsoft.com/oss/v2/azure/kms
imageName: keyvault
imageVersion: $(LATEST_KMS_VERSION)
- script: |
. scripts/setup-kind-cluster.sh &
. scripts/connect-registry.sh &
wait
displayName: "Setup kind cluster with azure kms plugin"
env:
REGISTRY_NAME: $(REGISTRY_NAME)
REGISTRY_PORT: $(REGISTRY_PORT)
KUBERNETES_VERSION: $(KUBERNETES_VERSION)
KIND_CLUSTER_NAME: $(KIND_CLUSTER_NAME)
KIND_NETWORK: $(KIND_NETWORK)
- template: cluster-health-template.yml
- template: kind-debug-template.yml
- script: make e2e-test
displayName: "Run e2e tests"
- script: |
echo "##vso[task.setvariable variable=LOCAL_IMAGE_VERSION]$(git rev-parse --short HEAD)"
displayName: "Update Image version"
# This stage will upgrade kms plugin. The path (./tests/e2e/generated_manifests) is mounted in kind cluster.
# Any changes in the host will automatically be reflected in /etc/kubernetes/manifests mount path and that static pod is restarted with new changes.
# manifest-template updates these files with registry, imageName and version to desired upgrade values.
- template: manifest-template.yml
parameters:
registry: localhost:$(REGISTRY_PORT)
imageName: keyvault
imageVersion: e2e-$(LOCAL_IMAGE_VERSION)
- script: |
# wait for the kind network to exist
echo "waiting for upgraded kms pod to be Running"
for i in $(seq 1 25); do
image=$(kubectl get pods -n kube-system azure-kms-provider-kms-control-plane -o jsonpath="{.spec.containers[*].image}")
phase=$(kubectl get pods -n kube-system azure-kms-provider-kms-control-plane -o jsonpath="{.status.phase}")
echo "image - $image phase - $phase"
if [ "${image}" == "${REGISTRY}/${IMAGE_NAME}:e2e-${LOCAL_IMAGE_VERSION}" ] && [ "${phase}" == "Running" ]; then
break
else
sleep 5
fi
done
# Give additional 5s for plugin to start. Remove this once https://github.com/Azure/kubernetes-kms/issues/113 is fixed.
sleep 5
displayName: "Wait for kms upgrade"
- template: cluster-health-template.yml
- template: kind-debug-template.yml
- script: make e2e-test
displayName: "Run e2e tests"
- template: cleanup-template.yml
================================================
FILE: .pipelines/templates/kind-debug-template.yml
================================================
steps:
- script: |
docker exec kms-control-plane bash -c "cat /etc/kubernetes/manifests/kubernetes-kms.yaml"
docker exec kms-control-plane bash -c "cat /etc/kubernetes/manifests/kube-apiserver.yaml"
docker exec kms-control-plane bash -c "cat /etc/kubernetes/encryption-config.yaml"
docker exec kms-control-plane bash -c "journalctl -u kubelet > kubelet.log && cat kubelet.log"
docker exec kms-control-plane bash -c "cd /var/log/containers ; cat *"
docker network ls
displayName: "Debug logs"
condition: failed()
================================================
FILE: .pipelines/templates/manifest-template.yml
================================================
parameters:
- name: registry
type: string
- name: imageName
type: string
- name: imageVersion
type: string
steps:
- script: |
export REGISTRY=${{ parameters.registry }}
export IMAGE_NAME=${{ parameters.imageName }}
export IMAGE_VERSION=${{ parameters.imageVersion }}
make e2e-generate-manifests
echo "##vso[task.setvariable variable=REGISTRY]${{ parameters.registry }}"
echo "##vso[task.setvariable variable=IMAGE_NAME]${{ parameters.imageName }}"
displayName: "Generate Manifests"
================================================
FILE: .pipelines/templates/prepare-deps.yaml
================================================
steps:
- bash: |
for i in {1..10}; do
if sudo tdnf install -y kernel-headers make gcc glibc-devel binutils gettext; then
exit 0
fi
echo "waiting until rpm lock is free"
sleep 5
done
exit 1
================================================
FILE: .pipelines/templates/scan-images-template.yml
================================================
steps:
- script: |
export REGISTRY="e2e"
export IMAGE_VERSION="test"
export OUTPUT_TYPE="type=docker"
make docker-init-buildx docker-build
wget https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION:-0.70.0}/trivy_${TRIVY_VERSION:-0.70.0}_Linux-64bit.tar.gz
tar zxvf trivy_${TRIVY_VERSION:-0.70.0}_Linux-64bit.tar.gz
# show all vulnerabilities in the logs
./trivy image "${REGISTRY}/keyvault:${IMAGE_VERSION}"
./trivy image --exit-code 1 --ignore-unfixed --severity MEDIUM,HIGH,CRITICAL "${REGISTRY}/keyvault:${IMAGE_VERSION}" || exit 1
displayName: "Scan images for vulnerability"
================================================
FILE: .pipelines/templates/unit-tests-template.yml
================================================
jobs:
- job: unit_tests
timeoutInMinutes: 10
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
# contains the following environment variables:
# - AZURE_TENANT_ID
# - KEYVAULT_NAME
# - KEY_NAME
# - KEY_VERSION
# - USER_ASSIGNED_IDENTITY_ID
- group: kubernetes-kms
steps:
- task: GoTool@0
inputs:
version: 1.26.2
- template: prepare-deps.yaml
- script: make lint
displayName: Run lint
- script: make unit-test
displayName: Run unit tests
- script: make build
displayName: Build
- script: |
sudo ./_output/kubernetes-kms --version
displayName: Check binary version
- script: |
sudo mkdir /etc/kubernetes
echo -e '{\n "tenantId": "'$AZURE_TENANT_ID'",\n "useManagedIdentityExtension": true,\n "userAssignedIdentityID": "'$USER_ASSIGNED_IDENTITY_ID'",\n}' | sudo tee --append /etc/kubernetes/azure.json > /dev/null
sudo chown root:root /etc/kubernetes/azure.json && sudo chmod 600 /etc/kubernetes/azure.json
displayName: Setup azure.json on host
- script: |
sudo ./_output/kubernetes-kms --keyvault-name $KEYVAULT_NAME --key-name $KEY_NAME --key-version $KEY_VERSION --listen-addr "unix:///opt/azurekms.sock" > /dev/null &
echo Waiting 2 seconds for the server to start
sleep 2
sudo env "PATH=$PATH" make integration-test
displayName: Run integration tests
- template: scan-images-template.yml
================================================
FILE: AUTHORS
================================================
Rita Zhang <rita.z.zhang@gmail.com>
================================================
FILE: CODEOWNERS
================================================
# Ref: https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-code-owners
* @aramase @enj
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Microsoft Open Source Code of Conduct
This code of conduct outlines expectations for participation in Microsoft-managed open source communities, as well as steps for reporting unacceptable behavior. We are committed to providing a welcoming and inspiring community for all. People violating this code of conduct may be banned from the community.
Our open source communities strive to:
- **Be friendly and patient:** Remember you might not be communicating in someone else's primary spoken or programming language, and others may not have your level of understanding.
- **Be welcoming:** Our communities welcome and support people of all backgrounds and identities. This includes, but is not limited to members of any race, ethnicity, culture, national origin, color, immigration status, social and economic class, educational level, sex, sexual orientation, gender identity and expression, age, size, family status, political belief, religion, and mental and physical ability.
- **Be respectful:** We are a world-wide community of professionals, and we conduct ourselves professionally. Disagreement is no excuse for poor behavior and poor manners. Disrespectful and unacceptable behavior includes, but is not limited to:
Violent threats or language.
Discriminatory or derogatory jokes and language.
Posting sexually explicit or violent material.
Posting, or threatening to post, people's personally identifying information ("doxing").
Insults, especially those using discriminatory terms or slurs.
Behavior that could be perceived as sexual attention.
Advocating for or encouraging any of the above behaviors.
- **Understand disagreements:** Disagreements, both social and technical, are useful learning opportunities. Seek to understand the other viewpoints and resolve differences constructively.
- This code is not exhaustive or complete. It serves to capture our common understanding of a productive, collaborative environment. We expect the code to be followed in spirit as much as in the letter.
## Scope
This code of conduct applies to all repos and communities for Microsoft-managed open source projects regardless of whether or not the repo explicitly calls out its use of this code. The code also applies in public spaces when an individual is representing a project or its community. Examples include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
Note: Some Microsoft-managed communities have codes of conduct that pre-date this document and issue resolution process. While communities are not required to change their code, they are expected to use the resolution process outlined here. The review team will coordinate with the communities involved to address your concerns.
## Reporting Code of Conduct Issues
We encourage all communities to resolve issues on their own whenever possible. This builds a broader and deeper understanding and ultimately a healthier interaction. In the event that an issue cannot be resolved locally, please feel free to report your concerns by contacting opencode@microsoft.com. Your report will be handled in accordance with the issue resolution process described in the [Code of Conduct FAQ].
In your report please include:
- Your contact information.
- Names (real, usernames or pseudonyms) of any individuals involved. If there are additional witnesses, please include them as well.
- Your account of what occurred, and if you believe the incident is ongoing. If there is a publicly available record (e.g. a mailing list archive or a public chat log), please include a link or attachment.
- Any additional information that may be helpful.
All reports will be reviewed by a multi-person team and will result in a response that is deemed necessary and appropriate to the circumstances. Where additional perspectives are needed, the team may seek insight from others with relevant expertise or experience. The confidentiality of the person reporting the incident will be kept at all times. Involved parties are never part of the review team.
Anyone asked to stop unacceptable behavior is expected to comply immediately. If an individual engages in unacceptable behavior, the review team may take any action they deem appropriate, including a permanent ban from the community.
*This code of conduct is based on the [template] established by the [TODO Group] and used by numerous other large communities (e.g., [Facebook], [Yahoo], [Twitter], [GitHub]) and the Scope section from the [Contributor Covenant version 1.4].*
[Code of Conduct FAQ]: https://opensource.microsoft.com/codeofconduct/faq/
[template]: http://todogroup.org/opencodeofconduct
[TODO Group]: http://todogroup.org/
[Facebook]: https://code.facebook.com/pages/876921332402685/open-source-code-of-conduct
[Yahoo]: https://yahoo.github.io/codeofconduct
[Twitter]: https://engineering.twitter.com/opensource/code-of-conduct
[GitHub]: http://todogroup.org/opencodeofconduct/#opensource@github.com
[Contributor Covenant version 1.4]: http://contributor-covenant.org/version/1/4/
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
================================================
FILE: Dockerfile
================================================
FROM mcr.microsoft.com/oss/go/microsoft/golang:1.26.2-bookworm@sha256:61e607875d60ae21a7a4a49110fe7098355473fbc74ab13091e3c1160cc92f18 AS builder
WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download
# Copy the go source
COPY cmd/server/main.go main.go
COPY pkg/ pkg/
ARG TARGETARCH
ARG TARGETPLATFORM
ARG LDFLAGS
RUN MS_GO_NOSYSTEMCRYPTO=1 CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on go build -a -ldflags "${LDFLAGS:--X github.com/Azure/kubernetes-kms/pkg/version.BuildVersion=latest}" -o _output/kubernetes-kms main.go
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM --platform=${TARGETPLATFORM:-linux/amd64} mcr.microsoft.com/cbl-mariner/distroless/minimal:2.0-nonroot.20250402@sha256:c5e349966c9a8ffe5af65970300d2b6899592da1714490b46561f5d86a0ab1e0
WORKDIR /
COPY --from=builder /workspace/_output/kubernetes-kms .
ENTRYPOINT [ "/kubernetes-kms" ]
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) Microsoft Corporation. All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
================================================
FILE: Makefile
================================================
ORG_PATH=github.com/Azure
PROJECT_NAME := kubernetes-kms
REPO_PATH="$(ORG_PATH)/$(PROJECT_NAME)"
REGISTRY_NAME ?= upstreamk8sci
REPO_PREFIX ?= oss/azure/kms
REGISTRY ?= $(REGISTRY_NAME).azurecr.io/$(REPO_PREFIX)
LOCAL_REGISTRY_NAME ?= kind-registry
LOCAL_REGISTRY_PORT ?= 5000
IMAGE_NAME ?= keyvault
IMAGE_VERSION ?= v0.10.0
IMAGE_TAG := $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION)
CGO_ENABLED_FLAG := 0
# build variables
BUILD_VERSION_VAR := $(REPO_PATH)/pkg/version.BuildVersion
BUILD_DATE_VAR := $(REPO_PATH)/pkg/version.BuildDate
BUILD_DATE := $$(date +%Y-%m-%d-%H:%M)
GIT_VAR := $(REPO_PATH)/pkg/version.GitCommit
GIT_HASH := $$(git rev-parse --short HEAD)
LDFLAGS ?= "-X $(BUILD_DATE_VAR)=$(BUILD_DATE) -X $(BUILD_VERSION_VAR)=$(IMAGE_VERSION) -X $(GIT_VAR)=$(GIT_HASH)"
GO_FILES=$(shell go list ./... | grep -v /test/e2e)
TOOLS_MOD_DIR := ./tools
TOOLS_DIR := $(abspath ./.tools)
# docker env var
DOCKER_BUILDKIT = 1
export DOCKER_BUILDKIT
# Testing var
KIND_VERSION ?= 0.31.0
KUBERNETES_VERSION ?= v1.35.0
BATS_VERSION ?= 1.4.1
## --------------------------------------
## Linting
## --------------------------------------
$(TOOLS_DIR)/golangci-lint: $(TOOLS_MOD_DIR)/go.mod $(TOOLS_MOD_DIR)/go.sum $(TOOLS_MOD_DIR)/tools.go
cd $(TOOLS_MOD_DIR) && \
go build -o $(TOOLS_DIR)/golangci-lint github.com/golangci/golangci-lint/v2/cmd/golangci-lint
.PHONY: lint
lint: $(TOOLS_DIR)/golangci-lint
$(TOOLS_DIR)/golangci-lint run --timeout=5m -v
## --------------------------------------
## Images
## --------------------------------------
ALL_LINUX_ARCH ?= amd64 arm64
# Output type of docker buildx build
OUTPUT_TYPE ?= type=registry
BUILDX_BUILDER_NAME ?= img-builder
QEMU_VERSION ?= 5.2.0-2
# The architecture of the image
ARCH ?= amd64
.PHONY: build
build:
go build -a -ldflags $(LDFLAGS) -o _output/kubernetes-kms ./cmd/server/
.PHONY: docker-init-buildx
docker-init-buildx:
@if ! docker buildx ls | grep $(BUILDX_BUILDER_NAME); then \
docker run --rm --privileged mirror.gcr.io/multiarch/qemu-user-static:$(QEMU_VERSION) --reset -p yes; \
docker buildx create --name $(BUILDX_BUILDER_NAME) --use; \
docker buildx inspect $(BUILDX_BUILDER_NAME) --bootstrap; \
fi
.PHONY: docker-build
docker-build:
docker buildx build \
--build-arg LDFLAGS=$(LDFLAGS) \
--no-cache \
--platform="linux/$(ARCH)" \
--output=$(OUTPUT_TYPE) \
-t $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION)-linux-$(ARCH) . \
--progress=plain; \
@if [ "$(ARCH)" = "amd64" ] && [ "$(OUTPUT_TYPE)" = "type=docker" ]; then \
docker tag $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION)-linux-$(ARCH) $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION); \
fi
.PHONY: docker-build-all
docker-build-all:
@for arch in $(ALL_LINUX_ARCH); do \
$(MAKE) ARCH=$${arch} docker-build; \
done
.PHONY: docker-push-manifest
docker-push-manifest:
docker manifest create --amend $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION) $(foreach arch,$(ALL_LINUX_ARCH),$(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION)-linux-$(arch)); \
for arch in $(ALL_LINUX_ARCH); do \
docker manifest annotate --os linux --arch $${arch} $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION) $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION)-linux-$${arch}; \
done; \
docker manifest push --purge $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_VERSION); \
## --------------------------------------
## Testing
## --------------------------------------
.PHONY: integration-test
integration-test:
go test -v -count=1 -failfast github.com/Azure/kubernetes-kms/tests/client
.PHONY: unit-test
unit-test:
go test -race -v -count=1 -failfast `go list ./... | grep -v client`
## --------------------------------------
## E2E Testing
## --------------------------------------
e2e-install-prerequisites:
# Download and install kind
curl -L https://github.com/kubernetes-sigs/kind/releases/download/v${KIND_VERSION}/kind-linux-amd64 --output kind && chmod +x kind && sudo mv kind /usr/local/bin/
# Download and install kubectl
curl -LO https://dl.k8s.io/release/${KUBERNETES_VERSION}/bin/linux/amd64/kubectl && chmod +x ./kubectl && sudo mv kubectl /usr/local/bin/
# Download and install bats
curl -sSLO https://github.com/bats-core/bats-core/archive/v${BATS_VERSION}.tar.gz && tar -zxvf v${BATS_VERSION}.tar.gz && sudo bash bats-core-${BATS_VERSION}/install.sh /usr/local
e2e-setup-kind: setup-local-registry
./scripts/setup-kind-cluster.sh &
./scripts/connect-registry.sh &
sleep 90s
e2e-kmsv2-setup-kind: setup-local-registry
./scripts/setup-kmsv2-kind-cluster.sh &
./scripts/connect-registry.sh &
sleep 90s
.PHONY: setup-local-registry
setup-local-registry:
./scripts/setup-local-registry.sh
e2e-generate-manifests:
@mkdir -p tests/e2e/generated_manifests
envsubst < tests/e2e/azure.json > tests/e2e/generated_manifests/azure.json
envsubst < tests/e2e/kms.yaml > tests/e2e/generated_manifests/kms.yaml
e2e-delete-kind:
# delete kind e2e cluster created for tests
kind delete cluster --name kms
e2e-test:
# Run test suite with kind cluster
bats -t tests/e2e/test.bats
e2e-kmsv2-test:
# Run test suite with kind cluster
bats -t tests/e2e/testkmsv2.bats
================================================
FILE: README.md
================================================
# KMS Plugin for Key Vault
[](https://dev.azure.com/AzureContainerUpstream/Kubernetes%20KMS/_build/latest?definitionId=442&branchName=master)
[](https://goreportcard.com/report/Azure/kubernetes-kms)
[](https://api.securityscorecards.dev/projects/github.com/Azure/kubernetes-kms)
Enables encryption at rest of your Kubernetes data in etcd using Azure Key Vault.
From the Kubernetes documentation on [Encrypting Secret Data at Rest]:
> _[KMS Plugin for Key Vault is]_ the recommended choice for using a third party tool for key management. Simplifies key rotation, with a new data encryption key (DEK) generated for each encryption, and key encryption key (KEK) rotation controlled by the user.
⚠️ **NOTE**: Currently, KMS plugin for Key Vault does not support key rotation. If you create a new key version in KMS, decryption will fail since it won't match the key used for encryption when the cluster was created.
💡 **NOTE**: To integrate your application secrets from a key management system outside of Kubernetes, use [Azure Key Vault Provider for Secrets Store CSI Driver].
## Features
- Use a key in Key Vault for etcd encryption
- Use a key in Key Vault protected by a Hardware Security Module (HSM)
- Bring your own keys
- Store secrets, keys, and certs in etcd, but manage them as part of Kubernetes
## Getting Started
### Prerequisites
💡 Make sure you have a Kubernetes cluster version 1.10 or later, the minimum version that is supported by KMS Plugin for Key Vault.
### Azure Kubernetes Service (AKS)
Azure Kubernetes Service ([AKS]) creates managed, supported Kubernetes clusters on Azure.
To enable encryption at rest for Kubernetes resources in etcd, check out the KMS plugin for Key Vault on AKS feature in this [doc](https://docs.microsoft.com/en-us/azure/aks/use-kms-etcd-encryption).
### Setting up KMS Plugin manually
Refer to [doc](docs/manual-install.md) for steps to setup the KMS Key Vault plugin on an existing cluster.
## Verifying that Data is Encrypted
Now that Azure KMS provider is running in your cluster and the encryption configuration is setup, it will encrypt the data in etcd. Let's verify that is working:
1. Create a new secret:
```bash
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
```
2. Using `etcdctl`, read the secret from etcd:
```bash
sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/certs/ca.crt --cert=/etc/kubernetes/certs/etcdclient.crt --key=/etc/kubernetes/certs/etcdclient.key get /registry/secrets/default/secret1
```
3. Check that the stored secret is prefixed with `k8s:enc:kms:v1:azurekmsprovider` when KMSv1 is used for encryption, or with `k8s:enc:kms:v2:azurekmsprovider` when KMSv2 is used. This prefix indicates that the data has been encrypted by the Azure KMS provider.
4. Verify the secret is decrypted correctly when retrieved via the Kubernetes API:
```bash
kubectl get secrets secret1 -o yaml
```
The output should match `mykey: bXlkYXRh`, which is the encoded data of `mydata`.
## Rotation
Refer to [doc](docs/rotation.md) for steps to rotate the KMS Key on an existing cluster.
## Metrics
Refer to [doc](docs/metrics.md) for details on the metrics exposed by the KMS Key Vault plugin.
## Contributing
The KMS Plugin for Key Vault project welcomes contributions and suggestions. Please see [CONTRIBUTING](CONTRIBUTING.md) for details.
## Roadmap
You can view the public roadmap for the KMS plugin for Azure KeyVault on the GitHub Project [here](https://github.com/orgs/Azure/projects/440). Note that all target dates are aspirational and subject to change.
## Release
Currently, this project releases monthly to patch security vulnerabilities, and bi-monthly for new features. We target the **first week** of the month for release.
## Code of conduct
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information, see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
## Support
KMS Plugin for Key Vault is an open source project that is [**not** covered by the Microsoft Azure support policy](https://support.microsoft.com/en-us/help/2941892/support-for-linux-and-open-source-technology-in-azure). [Please search open issues here](https://github.com/Azure/kubernetes-kms/issues), and if your issue isn't already represented please [open a new one](https://github.com/Azure/kubernetes-kms/issues/new/choose). The project maintainers will respond to the best of their abilities.
[aks]: https://azure.microsoft.com/services/kubernetes-service/
[encrypting secret data at rest]: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#providers
[azure key vault provider for secrets store csi driver]: https://github.com/Azure/secrets-store-csi-driver-provider-azure
================================================
FILE: SECURITY.md
================================================
<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK -->
## Security
Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below.
## Reporting Security Issues
**Please do not report security vulnerabilities through public GitHub issues.**
Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report).
If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).
You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc).
Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
* Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
* Full paths of source file(s) related to the manifestation of the issue
* The location of the affected source code (tag/branch/commit or direct URL)
* Any special configuration required to reproduce the issue
* Step-by-step instructions to reproduce the issue
* Proof-of-concept or exploit code (if possible)
* Impact of the issue, including how an attacker might exploit the issue
This information will help us triage your report more quickly.
If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.
## Preferred Languages
We prefer all communications to be in English.
## Policy
Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).
<!-- END MICROSOFT SECURITY.MD BLOCK -->
================================================
FILE: cmd/server/main.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package main
import (
"context"
"flag"
"fmt"
"math"
"net"
"net/url"
"os"
"os/signal"
"strconv"
"syscall"
"time"
"github.com/Azure/kubernetes-kms/pkg/config"
"github.com/Azure/kubernetes-kms/pkg/metrics"
"github.com/Azure/kubernetes-kms/pkg/plugin"
"github.com/Azure/kubernetes-kms/pkg/utils"
"github.com/Azure/kubernetes-kms/pkg/version"
"google.golang.org/grpc"
"k8s.io/klog/v2"
kmsv1 "k8s.io/kms/apis/v1beta1"
kmsv2 "k8s.io/kms/apis/v2"
"monis.app/mlog"
)
var (
listenAddr = flag.String("listen-addr", "unix:///opt/azurekms.socket", "gRPC listen address")
keyvaultName = flag.String("keyvault-name", "", "Azure Key Vault name")
keyName = flag.String("key-name", "", "Azure Key Vault KMS key name")
keyVersion = flag.String("key-version", "", "Azure Key Vault KMS key version")
managedHSM = flag.Bool("managed-hsm", false, "Azure Key Vault Managed HSM. Refer to https://docs.microsoft.com/en-us/azure/key-vault/managed-hsm/overview for more details.")
logFormatJSON = flag.Bool("log-format-json", false, "set log formatter to json")
logLevel = flag.Uint("v", 0, "In order of increasing verbosity: 0=warning/error, 2=info, 4=debug, 6=trace, 10=all")
// TODO remove this flag in future release.
_ = flag.String("configFilePath", "/etc/kubernetes/azure.json", "[DEPRECATED] Path for Azure Cloud Provider config file")
configFilePath = flag.String("config-file-path", "/etc/kubernetes/azure.json", "Path for Azure Cloud Provider config file")
versionInfo = flag.Bool("version", false, "Prints the version information")
healthzPort = flag.Uint("healthz-port", 8787, "port for health check")
healthzPath = flag.String("healthz-path", "/healthz", "path for health check")
healthzTimeout = flag.Duration("healthz-timeout", 20*time.Second, "RPC timeout for health check")
metricsBackend = flag.String("metrics-backend", "prometheus", "Backend used for metrics")
metricsAddress = flag.String("metrics-addr", "8095", "The address the metric endpoint binds to")
proxyMode = flag.Bool("proxy-mode", false, "Proxy mode")
proxyAddress = flag.String("proxy-address", "", "proxy address")
proxyPort = flag.Int("proxy-port", 7788, "port for proxy")
)
func main() {
if err := setupKMSPlugin(); err != nil {
mlog.Fatal(err)
}
}
func setupKMSPlugin() error {
defer mlog.Setup()() // set up log flushing and attempt to flush on exit
flag.Parse()
ctx := withShutdownSignal(context.Background())
logFormat := mlog.FormatText
if *logFormatJSON {
logFormat = mlog.FormatJSON
}
if *logLevel > math.MaxUint8 {
return fmt.Errorf("invalid log level: %d", *logLevel)
}
if err := mlog.ValidateAndSetKlogLevelAndFormatGlobally(ctx, klog.Level(uint8(*logLevel)), logFormat); err != nil {
return fmt.Errorf("invalid --log-level set: %w", err)
}
if *versionInfo {
if err := version.PrintVersion(); err != nil {
return fmt.Errorf("failed to print version: %w", err)
}
return nil
}
// initialize metrics exporter
err := metrics.InitMetricsExporter(*metricsBackend, *metricsAddress)
if err != nil {
return fmt.Errorf("failed to initialize metrics exporter: %w", err)
}
mlog.Always("Starting KeyManagementServiceServer service", "version", version.BuildVersion, "buildDate", version.BuildDate)
pluginConfig := &plugin.Config{
KeyVaultName: *keyvaultName,
KeyName: *keyName,
KeyVersion: *keyVersion,
ManagedHSM: *managedHSM,
ProxyMode: *proxyMode,
ProxyAddress: *proxyAddress,
ProxyPort: *proxyPort,
ConfigFilePath: *configFilePath,
}
azureConfig, err := config.GetAzureConfig(pluginConfig.ConfigFilePath)
if err != nil {
return fmt.Errorf("failed to get azure config: %w", err)
}
kvClient, err := plugin.NewKeyVaultClient(
azureConfig,
pluginConfig.KeyVaultName,
pluginConfig.KeyName,
pluginConfig.KeyVersion,
pluginConfig.ProxyMode,
pluginConfig.ProxyAddress,
pluginConfig.ProxyPort,
pluginConfig.ManagedHSM,
)
if err != nil {
return fmt.Errorf("failed to create key vault client: %w", err)
}
// Initialize and run the GRPC server
proto, addr, err := utils.ParseEndpoint(*listenAddr)
if err != nil {
return fmt.Errorf("failed to parse endpoint: %w", err)
}
if err := os.Remove(addr); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to remove socket file %s: %w", addr, err)
}
listener, err := net.Listen(proto, addr)
if err != nil {
return fmt.Errorf("failed to listen addr: %s, proto: %s: %w", addr, proto, err)
}
opts := []grpc.ServerOption{
grpc.UnaryInterceptor(utils.UnaryServerInterceptor),
}
s := grpc.NewServer(opts...)
// register kms v1 server
kmsV1Server, err := plugin.NewKMSv1Server(kvClient)
if err != nil {
return fmt.Errorf("failed to create server: %w", err)
}
kmsv1.RegisterKeyManagementServiceServer(s, kmsV1Server)
// register kms v2 server
kmsV2Server, err := plugin.NewKMSv2Server(kvClient)
if err != nil {
return fmt.Errorf("failed to create kms V2 server: %w", err)
}
kmsv2.RegisterKeyManagementServiceServer(s, kmsV2Server)
mlog.Always("Listening for connections", "addr", listener.Addr().String())
go func() {
if err := s.Serve(listener); err != nil {
mlog.Fatal(fmt.Errorf("failed to serve kms server: %w", err))
}
}()
// Health check for kms v1 and v2
healthz := &plugin.HealthZ{
KMSv1Server: kmsV1Server,
KMSv2Server: kmsV2Server,
HealthCheckURL: &url.URL{
Host: net.JoinHostPort("", strconv.FormatUint(uint64(*healthzPort), 10)),
Path: *healthzPath,
},
UnixSocketPath: listener.Addr().String(),
RPCTimeout: *healthzTimeout,
}
go healthz.Serve()
<-ctx.Done()
// gracefully stop the grpc server
mlog.Always("terminating the server")
s.GracefulStop()
return nil
}
// withShutdownSignal returns a copy of the parent context that will close if
// the process receives termination signals.
func withShutdownSignal(ctx context.Context) context.Context {
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, syscall.SIGTERM, syscall.SIGINT, os.Interrupt)
nctx, cancel := context.WithCancel(ctx)
go func() {
<-signalChan
mlog.Always("received shutdown signal")
cancel()
}()
return nctx
}
================================================
FILE: developers.md
================================================
# Developers Guide
This guide explains how to set up your environment for developing the Azure kubernetes kms service.
## Prerequisites
- Go 1.9.0 or later
- dep
- kubectl 1.9 or later
- An Azure account (needed for creating Azure key vault)
- Git
- make
### Structure of the Code
The code for the kubernetes-kms project is organized as follows:
- The built binary is located in root `./kubernetes-kms`
- The `test/` directory contains `client.go`, which creates a connection against the grpc unix service at `/opt/azurekms.socket` then executes client-side API calls against the `KeyManagementService` service. This is used by the CI/CD pipeline.
Go dependencies are managed with [dep](https://github.com/golang/dep) and stored in the
`vendor/` directory.
### Git Conventions
We use Git for our version control system. The `master` branch is the
home of the current development candidate. Releases are tagged.
We accept changes to the code via GitHub Pull Requests (PRs). One
workflow for doing this is as follows:
1. Use `go get` to clone this repository: `go get github.com/Azure/kubernetes-kms`
2. Fork that repository into your GitHub account
3. Add your repository as a remote for `$GOPATH/github.com/Azure/kubernetes-kms`
4. Create a new working branch (`git checkout -b feat/my-feature`) and
do your work on that branch.
5. When you are ready for us to review, push your branch to GitHub, and
then open a new pull request with us.
### Build the Code
We use `make` and `Makefile` to build the binary and the Docker image. To start the build process:
1. Run `make build` to build the binary `/kubernetes-kms` for your OS
### Run the Code Locally
To test your code locally:
1. On a linux machine, you can run `sudo ./kubernetes-kms --configFilePath <PATH TO YOUR AZURE.JSON FILE>` to create the gRPC unix domain socket running at `/opt/azurekms.socket`. This will start the gRPC server.
2. Create an Azure resource group, a Key Vault, and update the key vault's access policy with:
```bash
az group create -n mykeyvaultrg -l eastus
az keyvault create -n k8skv -g mykeyvaultrg
az keyvault set-policy -n k8skv --key-permissions create decrypt encrypt get list --spn <YOUR SPN CLIENT ID>
```
If you do not have a service principal, please refer to this [doc](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?view=azure-cli-latest).
3. Populate a `azure.json` file locally. The gRPC server will look for this file in the path provided by `configFilePath`. By default, `configFilePath` is set to `etc/kubernetes/azure.json`.
```json
{
"tenantId": "<YOUR TENANT ID>",
"subscriptionId": "<YOUR SUBSCRIPTION ID>",
"aadClientId": "<YOUR CLIENT ID>",
"aadClientSecret": "<YOUR CLIENT SECRET>",
"resourceGroup": "mykeyvaultrg",
"location": "eastus",
"providerVaultName": "k8skv",
"providerKeyName": "mykey"
}
```
4. Test with the gRPC client, run `sudo GOPATH=[YOUR GOPATH] GOCACHE=off go test tests/client/client_test.go`.
5. Test racing condition with the gRPC client, run `sudo GOPATH=[YOUR GOPATH] go test test/client/client_test.go & sudo GOPATH=[YOUR GOPATH] go test test/client/client_test.go &`.
### Build image
1. Run `make build-image` to build the binary `/kubernetes-kms` for linux and Docker image `mcr.microsoft.com/k8s/kms/keyvault:latest`
================================================
FILE: docs/manual-install.md
================================================
# 🛠 Manual Configurations #
This guide demonstrates steps required to enable the KMS Plugin for Key Vault in an existing cluster.
### 1. Create a Keyvault
If you're bringing your own keys, skip this step.
```bash
KEYVAULT_NAME=k8skv
RG=mykubernetesrg
LOC=eastus
# create resource group that'll contain the keyvault instance
az group create -n $RG -l $LOC
# create keyvault
az keyvault create -n $KV_NAME -g $RG
# create key that will be used for encryption
az keyvault key create -n k8s --vault-name $KV_NAME --kty RSA --size 2048
```
### 2. Give the cluster identity permissions to access the keys in keyvault
The KMS Plugin uses the cluster service principal or managed identity to access the keyvault instance.
#### More on authentication methods
[`/etc/kubernetes/azure.json`](https://kubernetes-sigs.github.io/cloud-provider-azure/install/configs/) is a well-known JSON file in each node that provides the details about which method KMS Plugin uses for access to Keyvault:
| Authentication method | `/etc/kubernetes/azure.json` fields used |
| -------------------------------- | ------------------------------------------------------------------------------------------- |
| System-assigned managed identity | `useManagedIdentityExtension: true` and `userAssignedIdentityID:""` |
| User-assigned managed identity | `useManagedIdentityExtension: true` and `userAssignedIdentityID:"<UserAssignedIdentityID>"` |
| Service principal (default) | `aadClientID: "<AADClientID>"` and `aadClientSecret: "<AADClientSecret>"` |
#### Obtaining the ID of the cluster managed identity/service principal
After your cluster is provisioned, depending on your cluster identity configuration, run one of the following commands to retrieve the **ID** of your managed identity or service principal, which will be used for role assignment to access Keyvault:
| Cluster configuration | Command |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| AKS cluster with service principal | `az aks show -g <AKSResourceGroup> -n <AKSClusterName> --query servicePrincipalProfile.clientId -otsv` |
| AKS cluster with managed identity | `az aks show -g <AKSResourceGroup> -n <AKSClusterName> --query identityProfile.kubeletidentity.clientId -otsv` |
Assign the following permissions:
```bash
az keyvault set-policy -n $KEYVAULT_NAME --key-permissions decrypt encrypt --spn <YOUR SPN CLIENT ID>
```
### 3. Deploy the KMS Plugin
For all Kubernetes control plane nodes, add the static pod manifest to `/etc/kubernetes/manifests`
```yaml
apiVersion: v1
kind: Pod
metadata:
name: azure-kms-provider
namespace: kube-system
labels:
tier: control-plane
component: azure-kms-provider
spec:
priorityClassName: system-node-critical
hostNetwork: true
containers:
- name: azure-kms-provider
image: mcr.microsoft.com/oss/v2/azure/kms/keyvault:v0.10.0
imagePullPolicy: IfNotPresent
args:
- --listen-addr=unix:///opt/azurekms.socket # [OPTIONAL] gRPC listen address. Default is unix:///opt/azurekms.socket
- --keyvault-name=${KV_NAME} # [REQUIRED] Name of the keyvault. Must match criteria specified at https://docs.microsoft.com/en-us/azure/key-vault/general/about-keys-secrets-certificates#vault-name-and-object-name
- --key-name=${KEY_NAME} # [REQUIRED] Name of the keyvault key used for encrypt/decrypt
- --key-version=${KEY_VERSION} # [REQUIRED] Version of the key to use
- --log-format-json=false # [OPTIONAL] Set log formatter to json. Default is false.
- --healthz-port=8787 # [OPTIONAL] port for health check. Default is 8787
- --healthz-path=/healthz # [OPTIONAL] path for health check. Default is /healthz
- --healthz-timeout=20s # [OPTIONAL] RPC timeout for health check. Default is 20s
- --managed-hsm=false # [OPTIONAL] Use Azure Key Vault managed HSM. Default is false.
- -v=1
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsUser: 0
ports:
- containerPort: 8787 # Must match the value defined in --healthz-port
protocol: TCP
livenessProbe:
httpGet:
path: /healthz # Must match the value defined in --healthz-path
port: 8787 # Must match the value defined in --healthz-port
failureThreshold: 2
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 4
memory: 2Gi
volumeMounts:
- name: etc-kubernetes
mountPath: /etc/kubernetes
- name: etc-ssl
mountPath: /etc/ssl
readOnly: true
- name: sock
mountPath: /opt
volumes:
- name: etc-kubernetes
hostPath:
path: /etc/kubernetes
- name: etc-ssl
hostPath:
path: /etc/ssl
- name: sock
hostPath:
path: /opt
```
View logs from the kms pod:
```bash
kubectl logs -l component=azure-kms-provider -n kube-system
I0219 17:35:33.608840 1 main.go:60] "Starting KeyManagementServiceServer service" version="v0.0.11" buildDate="2021-02-19-17:33"
I0219 17:35:33.609090 1 azure_config.go:27] populating AzureConfig from /etc/kubernetes/azure.json
I0219 17:35:33.609420 1 auth.go:66] "azure: using client_id+client_secret to retrieve access token" clientID="9a7a##### REDACTED #####bb26" clientSecret="23T.##### REDACTED #####vw-r"
I0219 17:35:33.609568 1 keyvault.go:66] "using kms key for encrypt/decrypt" vaultName="k8skmskv" keyName="key1" keyVersion="5cdf48ea6bb9456ebf637e1130b7751a"
I0219 17:35:33.609897 1 main.go:86] Listening for connections on address: /opt/azurekms.socket
...
```
### 4. Create encryption configuration
Create a new encryption configuration file `/etc/kubernetes/manifests/encryptionconfig.yaml` using the appropriate properties for the `kms` provider:
```yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources: # List of kubernetes resources that will be encrypted in etcd using the KMS plugin
- secrets
providers:
- kms:
name: azurekmsprovider
endpoint: unix:///opt/azurekms.socket # This endpoint must match the value defined in --listen-addr for the KMS plugin
cachesize: 1000
- identity: {}
```
The encryption configuration file needs to be accessible by all the api servers.
### 5. Modify `/etc/kubernetes/kube-apiserver.yaml`
Add the following flag:
```yaml
--encryption-provider-config=/etc/kubernetes/encryptionconfig.yaml
```
Mount `/opt` to access the socket:
```yaml
...
volumeMounts:
- name: "sock"
mountPath: "/opt"
...
volumes:
- name: "sock"
hostPath:
path: "/opt"
```
### 6. Restart your API server
================================================
FILE: docs/metrics.md
================================================
# Metrics provided by KMS plugin for Key Vault
This project uses [opentelemetry](https://opentelemetry.io/) for reporting metrics. Please refer to it's status [here](https://github.com/open-telemetry/opentelemetry-go#project-status). Prometheus is the only exporter that's currently supported.
## List of metrics provided by the kms plugin
| Metric | Description | Tags |
| ------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| kms_request | Distribution of how long it took for an operation | `status=success OR error`<br><br>`operation=encrypt OR decrypt OR grpc_encrypt OR grpc_decrypt`<br><br>`error_message` |
### Sample Metrics output
```shell
# HELP kms_request Distribution of how long it took for an operation
# TYPE kms_request histogram
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.1"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.13492828476735633"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.18205642030260802"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.24564560522315804"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.3314454017339986"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.4472135954999578"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.6034176336545162"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.8141810630738084"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.0985605433061172"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.4822688982138947"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.9999999999999991"} 18
kms_request_bucket{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="+Inf"} 18
kms_request_sum{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 1.010053082
kms_request_count{operation="decrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 18
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.1"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.13492828476735633"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.18205642030260802"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.24564560522315804"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.3314454017339986"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.4472135954999578"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.6034176336545162"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.8141810630738084"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.0985605433061172"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.4822688982138947"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.9999999999999991"} 19
kms_request_bucket{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="+Inf"} 19
kms_request_sum{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 1.021080768
kms_request_count{operation="encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 19
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.1"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.13492828476735633"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.18205642030260802"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.24564560522315804"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.3314454017339986"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.4472135954999578"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.6034176336545162"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.8141810630738084"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.0985605433061172"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.4822688982138947"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.9999999999999991"} 1
kms_request_bucket{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="+Inf"} 1
kms_request_sum{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 0.053279316
kms_request_count{operation="grpc_encrypt",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 1
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.1"} 0
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.13492828476735633"} 11
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.18205642030260802"} 13
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.24564560522315804"} 13
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.3314454017339986"} 13
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.4472135954999578"} 13
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.6034176336545162"} 14
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.8141810630738084"} 14
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.0985605433061172"} 14
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.4822688982138947"} 14
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.9999999999999991"} 14
kms_request_bucket{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="+Inf"} 14
kms_request_sum{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 2.1240865880000004
kms_request_count{operation="grpc_status",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 14
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.1"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.13492828476735633"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.18205642030260802"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.24564560522315804"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.3314454017339986"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.4472135954999578"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.6034176336545162"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="0.8141810630738084"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.0985605433061172"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.4822688982138947"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="1.9999999999999991"} 9
kms_request_bucket{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success",le="+Inf"} 9
kms_request_sum{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 0.0007254060000000001
kms_request_count{operation="grpc_version",otel_scope_name="keyvaultkms",otel_scope_version="",status="success"} 9
```
================================================
FILE: docs/rotation.md
================================================
# Rotating KMS key
This guide demonstrates steps required to update your cluster to use a new KMS key for encryption.
> NOTE: Ensure to read the Kubernetes documentation on [Rotating a decryption key](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#rotating-a-decryption-key) before proceeding with the guide.
### 1. Generate a new key or rotate the existing key
* If this is a new key in a different keyvault, then give the cluster identity permissions to access the keys in keyvault. Refer to [doc](./manual-install.md#2-give-the-cluster-identity-permissions-to-access-the-keys-in-keyvault) for details.
* If this is a new version of the same key that's already being used, then proceed to the next step.
### 2. Deploy another instance of KMS plugin with new key
To rotate the encrypt/decrypt key in the cluster, you'll need to run 2 kms plugin pods simultaneously listening on different unix sockets before making the transition.
For all Kubernetes control plane nodes, add the static pod manifest to `/etc/kubernetes/manifests`
```yaml
apiVersion: v1
kind: Pod
metadata:
name: azure-kms-provider-2
namespace: kube-system
labels:
tier: control-plane
component: azure-kms-provider
spec:
priorityClassName: system-node-critical
hostNetwork: true
containers:
- name: azure-kms-provider
image: mcr.microsoft.com/oss/v2/azure/kms/keyvault:v0.10.0
imagePullPolicy: IfNotPresent
args:
- --listen-addr=unix:///opt/azurekms2.socket # unix:///opt/azurekms.socket is used by the primary kms plugin pod. So use a different listen address here for the new kms plugin pod.
- --keyvault-name=${KV_NAME} # [REQUIRED] Name of the keyvault
- --key-name=${KEY_NAME} # [REQUIRED] Name of the keyvault key used for encrypt/decrypt
- --key-version=${KEY_VERSION} # [REQUIRED] Version of the key to use
- --log-format-json=false # [OPTIONAL] Set log formatter to json. Default is false.
- --healthz-port=8788 # The port used here should be different than the one used by the primary kms plugin pod.
- --healthz-path=/healthz # [OPTIONAL] path for health check. Default is /healthz
- --healthz-timeout=20s # [OPTIONAL] RPC timeout for health check. Default is 20s
- --managed-hsm=false # [OPTIONAL] Use Azure Key Vault managed HSM. Default is false.
- -v=5
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsUser: 0
ports:
- containerPort: 8788 # Must match the value defined in --healthz-port
protocol: TCP
livenessProbe:
httpGet:
path: /healthz # Must match the value defined in --healthz-path
port: 8788 # Must match the value defined in --healthz-port
failureThreshold: 2
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: "4"
memory: 2Gi
volumeMounts:
- name: etc-kubernetes
mountPath: /etc/kubernetes
- name: etc-ssl
mountPath: /etc/ssl
readOnly: true
- name: sock
mountPath: /opt
volumes:
- name: etc-kubernetes
hostPath:
path: /etc/kubernetes
- name: etc-ssl
hostPath:
path: /etc/ssl
- name: sock
hostPath:
path: /opt
nodeSelector:
kubernetes.io/os: linux
```
View logs from the kms pod:
```bash
kubectl logs -l component=azure-kms-provider -n kube-system
I0219 17:35:33.608840 1 main.go:60] "Starting KeyManagementServiceServer service" version="v0.0.11" buildDate="2021-02-19-17:33"
I0219 17:35:33.609090 1 azure_config.go:27] populating AzureConfig from /etc/kubernetes/azure.json
I0219 17:35:33.609420 1 auth.go:66] "azure: using client_id+client_secret to retrieve access token" clientID="9a7a##### REDACTED #####bb26" clientSecret="23T.##### REDACTED #####vw-r"
I0219 17:35:33.609568 1 keyvault.go:66] "using kms key for encrypt/decrypt" vaultName="k8skmskv" keyName="key1" keyVersion="5cdf48ea6bb9456ebf637e1130b7751a"
I0219 17:35:33.609897 1 main.go:86] Listening for connections on address: /opt/azurekms2.socket
...
```
### 3. Add the new provider to encryption configuration in `/etc/kubernetes/manifests/encryptionconfig.yaml`
```yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources: # List of kubernetes resources that will be encrypted in etcd using the KMS plugin
- secrets
providers:
- kms:
name: azurekmsprovider
endpoint: unix:///opt/azurekms.socket # This endpoint must match the value defined in --listen-addr for the KMS plugin using old key
cachesize: 1000
- kms:
name: azurekmsprovider2
endpoint: unix:///opt/azurekms2.socket # This endpoint must match the value defined in --listen-addr for the KMS plugin using new key
cachesize: 1000
```
### 4. Restart all `kube-apiserver`
* Proceed to the next step if using a single `kube-apiserver`
* If using multiple control plane nodes, restart the `kube-apiserver` to ensure each server can still decrypt using the new key in the encryption config.
* To validate the decryption still works, run `kubectl get secret <secret name> -o yaml` with one of the existing secrets to confirm the data is returned and is valid.
### 5. Switch the order of provider in the encryption config
```yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources: # List of kubernetes resources that will be encrypted in etcd using the KMS plugin
- secrets
providers:
# kms provider with new key
- kms:
name: azurekmsprovider2
endpoint: unix:///opt/azurekms2.socket # This endpoint must match the value defined in --listen-addr for the KMS plugin using new key
cachesize: 1000
# kms provider with old key
- kms:
name: azurekmsprovider
endpoint: unix:///opt/azurekms.socket # This endpoint must match the value defined in --listen-addr for the KMS plugin using old key
cachesize: 1000
```
### 6. Restart all `kube-apiserver` again
Refer to [step 4](#4-restart-all-kube-apiserver) to again restart the `kube-apiserver` for the encryption config changes to take effect.
### 7. Decrypt and re-encrypt existing secrets with new key
Since secrets are encrypted on write, performing an update on a secret will encrypt that content.
Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all existing secrets with the new key.
> NOTE: For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
#### How does this work?
The first provider in the encryption configuration is used for new encrypt calls. For decrypt, all existing kms providers in encryption configuration will be tried until one of the decrypt call succeeds.
### 8. Remove the old provider from encryption configuration
Now that all the secrets have been re-encrypted with the new key, we can safely remove the old kms provider from the encryption configuration.
```yaml
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources: # List of kubernetes resources that will be encrypted in etcd using the KMS plugin
- secrets
providers:
# kms provider with new key
- kms:
name: azurekmsprovider2
endpoint: unix:///opt/azurekms2.socket # This endpoint must match the value defined in --listen-addr for the KMS plugin using new key
cachesize: 1000
```
================================================
FILE: docs/testing.md
================================================
# End-to-end testing for KMS Plugin for Keyvault
## Prerequisites
To run tests locally, following components are required:
1. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
1. [bats](https://bats-core.readthedocs.io/en/latest/installation.html)
1. [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)
To install the prerequisites, run the following command:
```bash
make e2e-install-prerequisites
```
The E2E test suite extracts runtime configurations through environment variables. Below is a list of environment variables to set before running the E2E test suite.
| Variable | Description |
| ------------------- | --------------------------------------------------------------------------------------------------- |
| AZURE_CLIENT_ID | The client ID of your service principal that has `encrypt, decrypt` access to the keyvault key. |
| AZURE_CLIENT_SECRET | The client secret of your service principal that has `encrypt, decrypt` access to the keyvault key. |
| AZURE_TENANT_ID | The Azure tenant ID. |
| KEYVAULT_NAME | The Azure Keyvault name. |
| KEY_NAME | The name of Keyvault key that will be used by the kms plugin. |
| KEY_VERSION | The version of Keyvault key that will be used by the kms plugin. |
## Running the tests
The e2e tests are run against a [kind](https://kind.sigs.k8s.io/) cluster that's created as part of the test script. The script also creates a local docker registry that's used for test images.
1. Setup cluster, registry and build image:
```bash
make e2e-setup-kind
```
- This creates the local registry
- Builds a kms plugin image with the latest changes and pushes to local registry
- Creates a kind cluster with connectivity to local registry and kms plugin enabled with custom image
1. Run the end-to-end tests:
```bash
make e2e-test
```
1. To delete the kind cluster after running tests:
```bash
make e2e-delete-kind
```
================================================
FILE: go.mod
================================================
module github.com/Azure/kubernetes-kms
go 1.26.2
require (
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible
github.com/Azure/go-autorest/autorest v0.11.28
github.com/Azure/go-autorest/autorest/adal v0.9.23
go.opentelemetry.io/otel v1.43.0
go.opentelemetry.io/otel/exporters/prometheus v0.60.0
go.opentelemetry.io/otel/metric v1.43.0
golang.org/x/crypto v0.48.0
google.golang.org/grpc v1.79.3
gopkg.in/yaml.v3 v3.0.1
k8s.io/apimachinery v0.27.1
k8s.io/klog/v2 v2.100.1
k8s.io/kms v0.35.1
monis.app/mlog v0.0.4
)
require (
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-logr/zapr v1.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc // indirect
github.com/inconshreveable/mousetrap v1.0.1 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.23.0
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.65.0 // indirect
github.com/prometheus/otlptranslator v0.0.2 // indirect
github.com/prometheus/procfs v0.17.0 // indirect
github.com/spf13/cobra v1.6.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/testify v1.11.1 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.43.0
go.opentelemetry.io/otel/trace v1.43.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
go.uber.org/zap v1.24.0 // indirect
golang.org/x/net v0.51.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/text v0.34.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/protobuf v1.36.10 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
k8s.io/component-base v0.27.1 // indirect
k8s.io/utils v0.0.0-20230209194617-a36077c30491 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
)
================================================
FILE: go.sum
================================================
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible h1:fcYLmCpyNYRnvJbPerq7U0hS+6+I79yEDJBqVNcqUzU=
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs=
github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
github.com/Azure/go-autorest/autorest v0.11.28 h1:ndAExarwr5Y+GaHE6VCaY1kyS/HwwGGyuimVhWsHOEM=
github.com/Azure/go-autorest/autorest v0.11.28/go.mod h1:MrkzG3Y3AH668QyF9KRk5neJnGgmhQ6krbhR8Q5eMvA=
github.com/Azure/go-autorest/autorest/adal v0.9.18/go.mod h1:XVVeme+LZwABT8K5Lc3hA4nAe8LDBVle26gTrguhhPQ=
github.com/Azure/go-autorest/autorest/adal v0.9.23 h1:Yepx8CvFxwNKpH6ja7RZ+sKX+DWYNldbLiALMC3BTz8=
github.com/Azure/go-autorest/autorest/adal v0.9.23/go.mod h1:5pcMqFkdPhviJdlEy3kC/v1ZLnQl0MH6XA5YCcMhy4c=
github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw=
github.com/Azure/go-autorest/autorest/date v0.3.0/go.mod h1:BI0uouVdmngYNUzGWeSYnokU+TrmwEsOqdt8Y6sso74=
github.com/Azure/go-autorest/autorest/mocks v0.4.1/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k=
github.com/Azure/go-autorest/autorest/mocks v0.4.2 h1:PGN4EDXnuQbojHbU0UWoNvmu9AGVwYHG9/fkDYhtAfw=
github.com/Azure/go-autorest/autorest/mocks v0.4.2/go.mod h1:Vy7OitM9Kei0i1Oj+LvyAWMXJHeKH1MVlzFugfVrmyU=
github.com/Azure/go-autorest/autorest/to v0.4.0 h1:oXVqrxakqqV1UZdSazDOPOLvOIz+XA683u8EctwboHk=
github.com/Azure/go-autorest/autorest/to v0.4.0/go.mod h1:fE8iZBn7LQR7zH/9XU2NcPR4o9jEImooCeWJcYV/zLE=
github.com/Azure/go-autorest/autorest/validation v0.3.1 h1:AgyqjAd94fwNAoTjl/WQXg4VvFeRFpO+UhNyRXqF1ac=
github.com/Azure/go-autorest/autorest/validation v0.3.1/go.mod h1:yhLgjC0Wda5DYXl6JAsWyUe4KVNffhoDhG0zVzUMo3E=
github.com/Azure/go-autorest/logger v0.2.1 h1:IG7i4p/mDa2Ce4TRyAO8IHnVhAVF3RFU+ZtXWSmf4Tg=
github.com/Azure/go-autorest/logger v0.2.1/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8=
github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo=
github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-logr/zapr v1.2.3 h1:a9vnzlIBPQBBkeaR9IuMUfmVOrQlkoC4YfPoFkX3T7A=
github.com/go-logr/zapr v1.2.3/go.mod h1:eIauM6P8qSvTw5o2ez6UEAfGjQKrxQTl5EoK+Qa2oG4=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.0.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc h1:GN2Lv3MGO7AS6PrRoT6yV5+wkrOpcszoIsO4+4ds248=
github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc/go.mod h1:+JKpmjMGhpgPL+rXZ5nsZieVzvarn86asRlBg4uNGnk=
github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.23.0 h1:ust4zpdl9r4trLY/gSjlm07PuiBq2ynaXXlptpfy8Uc=
github.com/prometheus/client_golang v1.23.0/go.mod h1:i/o0R9ByOnHX0McrTMTyhYvKE4haaf2mW08I+jGAjEE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE=
github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
github.com/prometheus/otlptranslator v0.0.2 h1:+1CdeLVrRQ6Psmhnobldo0kTp96Rj80DRXRd5OSnMEQ=
github.com/prometheus/otlptranslator v0.0.2/go.mod h1:P8AwMgdD7XEr6QRUJ2QWLpiAZTgTE2UYgjlu3svompI=
github.com/prometheus/procfs v0.17.0 h1:FuLQ+05u4ZI+SS/w9+BWEM2TXiHKsUQ9TADiRH7DuK0=
github.com/prometheus/procfs v0.17.0/go.mod h1:oPQLaDAMRbA+u8H5Pbfq+dl3VDAvHxMUOVhe0wYB2zw=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.6.1 h1:o94oiPyS4KD1mPy2fmcYYHHfCxLqYjJOhGsCHFZtEzA=
github.com/spf13/cobra v1.6.1/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I=
go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
go.opentelemetry.io/otel/exporters/prometheus v0.60.0 h1:cGtQxGvZbnrWdC2GyjZi0PDKVSLWP/Jocix3QWfXtbo=
go.opentelemetry.io/otel/exporters/prometheus v0.60.0/go.mod h1:hkd1EekxNo69PTV4OWFGZcKQiIqg0RfuWExcPKFvepk=
go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM=
go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg=
go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg=
go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfCGLEo89fDkw=
go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/atomic v1.10.0 h1:9qC72Qh0+3MqyJbAn8YU5xVq1frD8bn3JtD2oXtafVQ=
go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8=
go.uber.org/multierr v1.8.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak=
go.uber.org/zap v1.19.0/go.mod h1:xg/QME4nWcxGxrpdeYfq7UvYrLh66cuVKdrbD1XF/NI=
go.uber.org/zap v1.24.0 h1:FiJd5l1UOLj0wCgbSE0rwwXHzEdAZS6hiiSnxJN/D60=
go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 h1:gRkg/vSppuSQoDjxyiGfN4Upv/h/DQmIR10ZU8dh4Ww=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217/go.mod h1:7i2o+ce6H/6BluujYR+kqX3GKH+dChPTQU19wjRPiGk=
google.golang.org/grpc v1.79.3 h1:sybAEdRIEtvcD68Gx7dmnwjZKlyfuc61Dyo9pGXXkKE=
google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE=
google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
k8s.io/apimachinery v0.27.1 h1:EGuZiLI95UQQcClhanryclaQE6xjg1Bts6/L3cD7zyc=
k8s.io/apimachinery v0.27.1/go.mod h1:5ikh59fK3AJ287GUvpUsryoMFtH9zj/ARfWCo3AyXTM=
k8s.io/component-base v0.27.1 h1:kEB8p8lzi4gCs5f2SPU242vOumHJ6EOsOnDM3tTuDTM=
k8s.io/component-base v0.27.1/go.mod h1:UGEd8+gxE4YWoigz5/lb3af3Q24w98pDseXcXZjw+E0=
k8s.io/klog/v2 v2.100.1 h1:7WCHKK6K8fNhTqfBhISHQ97KrnJNFZMcQvKp7gP/tmg=
k8s.io/klog/v2 v2.100.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0=
k8s.io/kms v0.35.1 h1:kjv2r9g1mY7uL+l1RhyAZvWVZIA/4qIfBHXyjFGLRhU=
k8s.io/kms v0.35.1/go.mod h1:VT+4ekZAdrZDMgShK37vvlyHUVhwI9t/9tvh0AyCWmQ=
k8s.io/utils v0.0.0-20230209194617-a36077c30491 h1:r0BAOLElQnnFhE/ApUsg3iHdVYYPBjNSSOMowRZxxsY=
k8s.io/utils v0.0.0-20230209194617-a36077c30491/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
monis.app/mlog v0.0.4 h1:YEzh5sguG4ApywaRWnBU+mGP6SA4WxOqiJ36u+KtoeE=
monis.app/mlog v0.0.4/go.mod h1:LtOpnndFuRGqnLBwzBvpA1DaoKuud2/moLzYXIiNl1s=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 h1:PRbqxJClWWYMNV1dhaG4NsibJbArud9kFxnAMREiWFE=
sigs.k8s.io/structured-merge-diff/v4 v4.2.3/go.mod h1:qjx8mGObPmV2aSZepjQjbmb2ihdVs8cGKBraizNC69E=
sigs.k8s.io/yaml v1.3.0 h1:a2VclLzOGrwOHDiV8EfBGhvjHvP46CtW5j6POvhYGGo=
sigs.k8s.io/yaml v1.3.0/go.mod h1:GeOyir5tyXNByN85N/dRIT9es5UQNerPYEKK56eTBm8=
================================================
FILE: pkg/auth/auth.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package auth
import (
"crypto/rsa"
"crypto/x509"
"fmt"
"net/http"
"os"
"regexp"
"github.com/Azure/kubernetes-kms/pkg/config"
"github.com/Azure/kubernetes-kms/pkg/consts"
"github.com/Azure/go-autorest/autorest"
"github.com/Azure/go-autorest/autorest/adal"
"github.com/Azure/go-autorest/autorest/azure"
"golang.org/x/crypto/pkcs12"
"monis.app/mlog"
)
// GetKeyvaultToken() returns token for Keyvault endpoint.
func GetKeyvaultToken(config *config.AzureConfig, env *azure.Environment, resource string, proxyMode bool) (authorizer autorest.Authorizer, err error) {
servicePrincipalToken, err := GetServicePrincipalToken(config, env.ActiveDirectoryEndpoint, resource, proxyMode)
if err != nil {
return nil, err
}
authorizer = autorest.NewBearerAuthorizer(servicePrincipalToken)
return authorizer, nil
}
// GetServicePrincipalToken creates a new service principal token based on the configuration.
func GetServicePrincipalToken(config *config.AzureConfig, aadEndpoint, resource string, proxyMode bool) (adal.OAuthTokenProvider, error) {
oauthConfig, err := adal.NewOAuthConfig(aadEndpoint, config.TenantID)
if err != nil {
return nil, fmt.Errorf("failed to create OAuth config, error: %w", err)
}
if config.UseManagedIdentityExtension {
mlog.Info("using managed identity extension to retrieve access token")
msiEndpoint, err := adal.GetMSIVMEndpoint()
if err != nil {
return nil, fmt.Errorf("failed to get managed service identity endpoint, error: %w", err)
}
// using user-assigned managed identity to access keyvault
if len(config.UserAssignedIdentityID) > 0 {
mlog.Info("using User-assigned managed identity to retrieve access token", "clientID", redactClientCredentials(config.UserAssignedIdentityID))
return adal.NewServicePrincipalTokenFromMSIWithUserAssignedID(msiEndpoint,
resource,
config.UserAssignedIdentityID)
}
mlog.Info("using system-assigned managed identity to retrieve access token")
// using system-assigned managed identity to access keyvault
return adal.NewServicePrincipalTokenFromMSI(
msiEndpoint,
resource)
}
if len(config.ClientSecret) > 0 && len(config.ClientID) > 0 {
mlog.Info("azure: using client_id+client_secret to retrieve access token",
"clientID", redactClientCredentials(config.ClientID), "clientSecret", redactClientCredentials(config.ClientSecret))
spt, err := adal.NewServicePrincipalToken(
*oauthConfig,
config.ClientID,
config.ClientSecret,
resource)
if err != nil {
return nil, err
}
if proxyMode {
return addTargetTypeHeader(spt), nil
}
return spt, nil
}
if len(config.AADClientCertPath) > 0 && len(config.AADClientCertPassword) > 0 {
mlog.Info("using jwt client_assertion (client_cert+client_private_key) to retrieve access token")
certData, err := os.ReadFile(config.AADClientCertPath)
if err != nil {
return nil, fmt.Errorf("failed to read client certificate from file %s, error: %w", config.AADClientCertPath, err)
}
certificate, privateKey, err := decodePkcs12(certData, config.AADClientCertPassword)
if err != nil {
return nil, fmt.Errorf("failed to decode the client certificate, error: %w", err)
}
spt, err := adal.NewServicePrincipalTokenFromCertificate(
*oauthConfig,
config.ClientID,
certificate,
privateKey,
resource)
if err != nil {
return nil, err
}
if proxyMode {
return addTargetTypeHeader(spt), nil
}
return spt, nil
}
return nil, fmt.Errorf("no credentials provided for accessing keyvault")
}
// ParseAzureEnvironment returns azure environment by name.
func ParseAzureEnvironment(cloudName string) (*azure.Environment, error) {
var env azure.Environment
var err error
if cloudName == "" {
env = azure.PublicCloud
} else {
env, err = azure.EnvironmentFromName(cloudName)
}
return &env, err
}
// decodePkcs12 decodes a PKCS#12 client certificate by extracting the public certificate and
// the private RSA key.
func decodePkcs12(pkcs []byte, password string) (*x509.Certificate, *rsa.PrivateKey, error) {
privateKey, certificate, err := pkcs12.Decode(pkcs, password)
if err != nil {
return nil, nil, fmt.Errorf("decoding the PKCS#12 client certificate: %w", err)
}
rsaPrivateKey, isRsaKey := privateKey.(*rsa.PrivateKey)
if !isRsaKey {
return nil, nil, fmt.Errorf("PKCS#12 certificate must contain a RSA private key")
}
return certificate, rsaPrivateKey, nil
}
// redactClientCredentials applies regex to a sensitive string and return the redacted value.
func redactClientCredentials(sensitiveString string) string {
r := regexp.MustCompile(`^(\S{4})(\S|\s)*(\S{4})$`)
return r.ReplaceAllString(sensitiveString, "$1##### REDACTED #####$3")
}
// addTargetTypeHeader adds the target header if proxy mode is enabled.
func addTargetTypeHeader(spt *adal.ServicePrincipalToken) *adal.ServicePrincipalToken {
spt.SetSender(autorest.CreateSender(
(func() autorest.SendDecorator {
return func(s autorest.Sender) autorest.Sender {
return autorest.SenderFunc(func(r *http.Request) (*http.Response, error) {
r.Header.Set(consts.RequestHeaderTargetType, consts.TargetTypeAzureActiveDirectory)
return s.Do(r)
})
}
})()))
return spt
}
================================================
FILE: pkg/auth/auth_test.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package auth
import (
"reflect"
"strings"
"testing"
"github.com/Azure/kubernetes-kms/pkg/config"
"github.com/Azure/go-autorest/autorest/adal"
"github.com/Azure/go-autorest/autorest/azure"
)
func TestParseAzureEnvironment(t *testing.T) {
envNamesArray := []string{"AZURECHINACLOUD", "AZUREGERMANCLOUD", "AZUREPUBLICCLOUD", "AZUREUSGOVERNMENTCLOUD", ""}
for _, envName := range envNamesArray {
azureEnv, err := ParseAzureEnvironment(envName)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.EqualFold(envName, "") && !strings.EqualFold(azureEnv.Name, "AZUREPUBLICCLOUD") {
t.Fatalf("string doesn't match, expected AZUREPUBLICCLOUD, got %s", azureEnv.Name)
} else if !strings.EqualFold(envName, "") && !strings.EqualFold(envName, azureEnv.Name) {
t.Fatalf("string doesn't match, expected %s, got %s", envName, azureEnv.Name)
}
}
wrongEnvName := "AZUREWRONGCLOUD"
_, err := ParseAzureEnvironment(wrongEnvName)
if err == nil {
t.Fatalf("expected error for wrong azure environment name")
}
}
func TestRedactClientCredentials(t *testing.T) {
tests := []struct {
name string
clientID string
expected string
}{
{
name: "should redact client id",
clientID: "aabc0000-a83v-9h4m-000j-2c0a66b0c1f9",
expected: "aabc##### REDACTED #####c1f9",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
actual := redactClientCredentials(test.clientID)
if actual != test.expected {
t.Fatalf("expected: %s, got %s", test.expected, actual)
}
})
}
}
func TestGetServicePrincipalTokenFromMSIWithUserAssignedID(t *testing.T) {
tests := []struct {
name string
config *config.AzureConfig
proxyMode bool // The proxy mode doesn't matter if user-assigned managed identity is used to get service principal token
}{
{
name: "using user-assigned managed identity to access keyvault",
config: &config.AzureConfig{
UseManagedIdentityExtension: true,
UserAssignedIdentityID: "clientID",
TenantID: "TenantID",
ClientID: "AADClientID",
ClientSecret: "AADClientSecret",
},
proxyMode: false,
},
// The Azure service principal is ignored when
// UseManagedIdentityExtension is set to true
{
name: "using user-assigned managed identity over service principal if set to true",
config: &config.AzureConfig{
UseManagedIdentityExtension: true,
UserAssignedIdentityID: "clientID",
},
proxyMode: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
token, err := GetServicePrincipalToken(test.config, "https://login.microsoftonline.com/", "https://vault.azure.net", test.proxyMode)
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
msiEndpoint, err := adal.GetMSIVMEndpoint()
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
spt, err := adal.NewServicePrincipalTokenFromMSIWithUserAssignedID(msiEndpoint, "https://vault.azure.net", "clientID")
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
if !reflect.DeepEqual(token, spt) {
t.Fatalf("expected: %v, got: %v", spt, token)
}
})
}
}
func TestGetServicePrincipalTokenFromMSI(t *testing.T) {
tests := []struct {
name string
config *config.AzureConfig
proxyMode bool // The proxy mode doesn't matter if MSI is used to get service principal token
}{
{
name: "using system-assigned managed identity to access keyvault",
config: &config.AzureConfig{
UseManagedIdentityExtension: true,
},
proxyMode: false,
},
// The Azure service principal is ignored when
// UseManagedIdentityExtension is set to true
{
name: "using system-assigned managed identity over service principal if set to true",
config: &config.AzureConfig{
UseManagedIdentityExtension: true,
TenantID: "TenantID",
ClientID: "AADClientID",
ClientSecret: "AADClientSecret",
},
proxyMode: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
token, err := GetServicePrincipalToken(test.config, "https://login.microsoftonline.com/", "https://vault.azure.net", test.proxyMode)
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
msiEndpoint, err := adal.GetMSIVMEndpoint()
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
spt, err := adal.NewServicePrincipalTokenFromMSI(msiEndpoint, "https://vault.azure.net")
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
if !reflect.DeepEqual(token, spt) {
t.Fatalf("expected: %v, got: %v", spt, token)
}
})
}
}
func TestGetServicePrincipalToken(t *testing.T) {
tests := []struct {
name string
config *config.AzureConfig
}{
{
name: "using service-principal credentials to access keyvault",
config: &config.AzureConfig{
TenantID: "TenantID",
ClientID: "AADClientID",
ClientSecret: "AADClientSecret",
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
token, err := GetServicePrincipalToken(test.config, "https://login.microsoftonline.com/", "https://vault.azure.net", false)
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
env := &azure.PublicCloud
oauthConfig, err := adal.NewOAuthConfig(env.ActiveDirectoryEndpoint, test.config.TenantID)
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
spt, err := adal.NewServicePrincipalToken(*oauthConfig, test.config.ClientID, test.config.ClientSecret, "https://vault.azure.net")
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
if !reflect.DeepEqual(token, spt) {
t.Fatalf("expected: %+v, got: %+v", spt, token)
}
})
}
}
================================================
FILE: pkg/config/azure_config.go
================================================
package config
import (
"fmt"
"os"
"gopkg.in/yaml.v3"
"monis.app/mlog"
)
// AzureConfig is representing /etc/kubernetes/azure.json.
type AzureConfig struct {
Cloud string `json:"cloud" yaml:"cloud"`
TenantID string `json:"tenantId" yaml:"tenantId"`
ClientID string `json:"aadClientId" yaml:"aadClientId"`
ClientSecret string `json:"aadClientSecret" yaml:"aadClientSecret"`
UseManagedIdentityExtension bool `json:"useManagedIdentityExtension,omitempty" yaml:"useManagedIdentityExtension,omitempty"`
UserAssignedIdentityID string `json:"userAssignedIdentityID,omitempty" yaml:"userAssignedIdentityID,omitempty"`
AADClientCertPath string `json:"aadClientCertPath" yaml:"aadClientCertPath"`
AADClientCertPassword string `json:"aadClientCertPassword" yaml:"aadClientCertPassword"`
}
// GetAzureConfig returns configs in the azure.json cloud provider file.
func GetAzureConfig(configFile string) (config *AzureConfig, err error) {
cfg := AzureConfig{}
mlog.Trace("populating AzureConfig from config file", "configFile", configFile)
bytes, err := os.ReadFile(configFile)
if err != nil {
return nil, fmt.Errorf("failed to load config file %s, error: %w", configFile, err)
}
if err = yaml.Unmarshal(bytes, &cfg); err != nil {
return nil, fmt.Errorf("failed to unmarshal azure.json, error: %w", err)
}
return &cfg, nil
}
================================================
FILE: pkg/consts/consts.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package consts
const (
// In proxy mode, the header is added into the requests from kms-plugin.
// The proxy will check the header and forward the request to different destinations.
// e.g. When the value of the header "x-azure-proxy-target" is "KeyVault", the request
// is forwared to Azure Key Vault by the proxy.
RequestHeaderTargetType = "x-azure-proxy-target"
TargetTypeAzureActiveDirectory = "AzureActiveDirectory"
TargetTypeKeyVault = "KeyVault"
)
================================================
FILE: pkg/metrics/exporter.go
================================================
package metrics
import (
"fmt"
"strings"
"monis.app/mlog"
)
const (
prometheusExporter = "prometheus"
)
// InitMetricsExporter initializes new exporter.
func InitMetricsExporter(metricsBackend, metricsAddress string) error {
exporter := strings.ToLower(metricsBackend)
mlog.Always("metrics backend", "exporter", exporter)
switch exporter {
// Prometheus is the only exporter supported for now
case prometheusExporter:
return initPrometheusExporter(metricsAddress)
default:
return fmt.Errorf("unsupported metrics backend %v", metricsBackend)
}
}
================================================
FILE: pkg/metrics/exporter_test.go
================================================
package metrics
import (
"net/http"
"testing"
)
func TestInitMetricsExporter(t *testing.T) {
testCases := []struct {
name string
metricsBackend string
metricsAddress string
expectedError bool
}{
{
name: "With_Prometheus_Backend",
metricsBackend: "prometheus",
metricsAddress: "8095",
expectedError: false,
},
{
name: "With_Non_Prometheus_Backend",
metricsBackend: "nonprometheus",
expectedError: true,
},
{
name: "With_Uppercase_Backend_Name",
metricsBackend: "Prometheus",
metricsAddress: "8096",
expectedError: false,
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
err := InitMetricsExporter(testCase.metricsBackend, testCase.metricsAddress)
if testCase.expectedError && err == nil || !testCase.expectedError && err != nil {
t.Fatalf("expected error: %v, found: %v", testCase.expectedError, err)
}
// Reset handler to test /metrics repeatedly.
http.DefaultServeMux = new(http.ServeMux)
})
}
}
================================================
FILE: pkg/metrics/prometheus_exporter.go
================================================
package metrics
import (
"fmt"
"net/http"
"time"
promclient "github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/prometheus"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"monis.app/mlog"
)
const (
metricsEndpoint = "metrics"
)
func initPrometheusExporter(metricsAddress string) error {
registry := promclient.NewRegistry()
exporter, err := prometheus.New(
prometheus.WithRegisterer(registry))
if err != nil {
return err
}
mp := sdkmetric.NewMeterProvider(
sdkmetric.WithReader(exporter),
sdkmetric.WithView(sdkmetric.NewView(
sdkmetric.Instrument{Kind: sdkmetric.InstrumentKindHistogram},
sdkmetric.Stream{
Aggregation: sdkmetric.AggregationExplicitBucketHistogram{
Boundaries: promclient.ExponentialBucketsRange(0.1, 2, 11),
},
},
)),
)
otel.SetMeterProvider(mp)
http.Handle(fmt.Sprintf("/%s", metricsEndpoint), promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))
go func() {
server := &http.Server{
Addr: fmt.Sprintf(":%s", metricsAddress),
ReadHeaderTimeout: 5 * time.Second,
}
if err := server.ListenAndServe(); err != nil {
mlog.Fatal(err, "failed to register prometheus endpoint", "metricsAddress", metricsAddress)
}
}()
mlog.Always("Prometheus metrics server running", "address", metricsAddress)
return nil
}
================================================
FILE: pkg/metrics/stats_reporter.go
================================================
package metrics
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
)
const (
instrumentationName = "keyvaultkms"
errorMessageKey = "error_message"
statusTypeKey = "status"
operationTypeKey = "operation"
kmsRequestMetricName = "kms_request"
// ErrorStatusTypeValue sets status tag to "error".
ErrorStatusTypeValue = "error"
// SuccessStatusTypeValue sets status tag to "success".
SuccessStatusTypeValue = "success"
// EncryptOperationTypeValue sets operation tag to "encrypt".
EncryptOperationTypeValue = "encrypt"
// DecryptOperationTypeValue sets operation tag to "decrypt".
DecryptOperationTypeValue = "decrypt"
// GrpcOperationTypeValue sets operation tag to "grpc".
GrpcOperationTypeValue = "grpc"
)
type reporter struct {
histogram metric.Float64Histogram
}
// StatsReporter reports metrics.
type StatsReporter interface {
ReportRequest(ctx context.Context, operationType, status string, duration float64, errors ...string)
}
// NewStatsReporter instantiates otel reporter.
func NewStatsReporter() (StatsReporter, error) {
meter := otel.GetMeterProvider().Meter(instrumentationName)
metricCounter, err := meter.Float64Histogram(
kmsRequestMetricName,
metric.WithDescription("Distribution of how long it took for an operation"),
)
if err != nil {
return nil, err
}
return &reporter{
histogram: metricCounter,
}, nil
}
func (r *reporter) ReportRequest(ctx context.Context, operationType, status string, duration float64, errors ...string) {
labels := []attribute.KeyValue{
attribute.String(operationTypeKey, operationType),
attribute.String(statusTypeKey, status),
}
// Add errors
if (status == ErrorStatusTypeValue) && len(errors) > 0 {
for _, err := range errors {
labels = append(labels, attribute.String(errorMessageKey, err))
}
}
r.histogram.Record(ctx, duration, metric.WithAttributes(labels...))
}
================================================
FILE: pkg/plugin/healthz.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"context"
"fmt"
"net/http"
"net/url"
"time"
"github.com/Azure/kubernetes-kms/pkg/version"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"k8s.io/apimachinery/pkg/util/uuid"
kmsv1 "k8s.io/kms/apis/v1beta1"
kmsv2 "k8s.io/kms/apis/v2"
"monis.app/mlog"
)
const (
healthCheckPlainText = "healthcheck"
)
// HealthZ is the health check server for the KMS plugin.
type HealthZ struct {
KMSv1Server *KeyManagementServiceServer
KMSv2Server *KeyManagementServiceV2Server
HealthCheckURL *url.URL
UnixSocketPath string
RPCTimeout time.Duration
}
// Serve creates the http handler for serving health requests.
func (h *HealthZ) Serve() {
serveMux := http.NewServeMux()
serveMux.HandleFunc(h.HealthCheckURL.EscapedPath(), h.ServeHTTP)
server := &http.Server{
Addr: h.HealthCheckURL.Host,
ReadHeaderTimeout: 5 * time.Second,
Handler: serveMux,
}
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
mlog.Fatal(err, "failed to start health check server", "url", h.HealthCheckURL.String())
}
}
func (h *HealthZ) ServeHTTP(w http.ResponseWriter, _ *http.Request) {
mlog.Trace("Started health check")
ctx, cancel := context.WithTimeout(context.Background(), h.RPCTimeout)
defer cancel()
conn, err := h.dialUnixSocket()
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
return
}
defer conn.Close()
// create the kms client for v1
kmsClient := kmsv1.NewKeyManagementServiceClient(conn)
// create the kms client for v2
kmsV2Client := kmsv2.NewKeyManagementServiceClient(conn)
// check version response against KMS-Plugin's gRPC endpoint.
err = h.checkRPC(ctx, kmsClient, kmsV2Client)
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
return
}
// Both encryption and decryption calls are made for each version,
// resulting in a total of 4 calls to the keyvault.
// Additionally, a health check is performed every 10 seconds.
// v1 checks
// check the configured keyvault, key, key version and permissions are still
// valid to encrypt and decrypt with test data.
enc, err := h.KMSv1Server.Encrypt(ctx, &kmsv1.EncryptRequest{Plain: []byte(healthCheckPlainText)})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
dec, err := h.KMSv1Server.Decrypt(ctx, &kmsv1.DecryptRequest{Cipher: enc.Cipher})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if string(dec.Plain) != healthCheckPlainText {
http.Error(w, "plain text mismatch after decryption", http.StatusInternalServerError)
return
}
// v2 checks.
// appending a string to UUID allows us to differentiate the UUIDs generated by us from those generated by the API server.
uid := "local-healthz-check-" + string(uuid.NewUUID())
v2EncryptResponse, err := h.KMSv2Server.Encrypt(
ctx,
&kmsv2.EncryptRequest{
Plaintext: []byte(healthCheckPlainText),
Uid: uid,
},
)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
v2DecryptResponse, err := h.KMSv2Server.Decrypt(ctx, &kmsv2.DecryptRequest{
Ciphertext: v2EncryptResponse.Ciphertext,
KeyId: v2EncryptResponse.KeyId,
Uid: uid, // passing the same uid to track roundtrip encrypt/decrypt calls
Annotations: v2EncryptResponse.Annotations,
})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if string(v2DecryptResponse.Plaintext) != healthCheckPlainText {
http.Error(w, "plain text mismatch after decryption with KMSv2", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
if _, err = w.Write([]byte("ok")); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
mlog.Trace("Completed health check")
}
// checkRPC initiates a grpc request to validate the socket is responding
// sends a KMS VersionRequest and checks if the VersionResponse is valid.
func (h *HealthZ) checkRPC(
ctx context.Context,
kmsV1Client kmsv1.KeyManagementServiceClient,
kmsV2Client kmsv2.KeyManagementServiceClient,
) error {
v, err := kmsV1Client.Version(ctx, &kmsv1.VersionRequest{})
if err != nil {
return err
}
if v.Version != version.KMSv1APIVersion || v.RuntimeName != version.Runtime || v.RuntimeVersion != version.BuildVersion {
return fmt.Errorf("failed to get correct version response")
}
v2Status, err := kmsV2Client.Status(ctx, &kmsv2.StatusRequest{})
if err != nil {
return err
}
if v2Status.Version != version.KMSv2APIVersion {
return fmt.Errorf(
"failed to get correct version response for v2 expected: %s, got: %s",
version.KMSv2APIVersion,
v2Status.Version,
)
}
return nil
}
func (h *HealthZ) dialUnixSocket() (*grpc.ClientConn, error) {
return grpc.NewClient("unix://"+h.UnixSocketPath,
grpc.WithTransportCredentials(insecure.NewCredentials()))
}
================================================
FILE: pkg/plugin/healthz_test.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"testing"
"time"
"github.com/Azure/kubernetes-kms/pkg/metrics"
mockkeyvault "github.com/Azure/kubernetes-kms/pkg/plugin/mock_keyvault"
"github.com/Azure/azure-sdk-for-go/services/keyvault/2016-10-01/keyvault"
"google.golang.org/grpc"
kmsv1 "k8s.io/kms/apis/v1beta1"
kmsv2 "k8s.io/kms/apis/v2"
"monis.app/mlog"
)
func TestServe(t *testing.T) {
tests := []struct {
desc string
setEncryptResponse string
setDecryptResponse string
setEncryptError error
setDecryptError error
expectedHTTPStatusCode int
}{
{
desc: "failed to encrypt in health check",
setEncryptResponse: "",
setEncryptError: fmt.Errorf("failed to encrypt"),
expectedHTTPStatusCode: http.StatusServiceUnavailable,
},
{
desc: "failed to decrypt in health check",
setEncryptResponse: "",
setEncryptError: nil,
setDecryptResponse: "",
setDecryptError: fmt.Errorf("failed to decrypt"),
expectedHTTPStatusCode: http.StatusServiceUnavailable,
},
{
desc: "encrypt-decrypt mismatch",
setEncryptResponse: "bar",
setEncryptError: nil,
setDecryptResponse: "foo",
setDecryptError: nil,
expectedHTTPStatusCode: http.StatusServiceUnavailable,
},
{
desc: "successful health check",
setEncryptResponse: "bar",
setDecryptResponse: "healthcheck",
expectedHTTPStatusCode: http.StatusOK,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
socketPath := fmt.Sprintf("%s/kms.sock", getTempTestDir(t))
defer os.Remove(socketPath)
fakeKMSServer, fakeKMSV2Server, mockKVClient, err := setupFakeKMSServer(socketPath)
if err != nil {
t.Fatalf("failed to create fake kms server, err: %+v", err)
}
mockKVClient.SetEncryptResponse([]byte(test.setEncryptResponse), test.setEncryptError)
mockKVClient.SetDecryptResponse([]byte(test.setDecryptResponse), test.setDecryptError)
healthz := &HealthZ{
KMSv1Server: fakeKMSServer,
KMSv2Server: fakeKMSV2Server,
UnixSocketPath: socketPath,
RPCTimeout: 20 * time.Second,
HealthCheckURL: &url.URL{
Scheme: "http",
Host: net.JoinHostPort("localhost", "8080"),
Path: "/healthz",
},
}
server := httptest.NewServer(healthz)
defer server.Close()
respCode, body := doHealthCheck(t, server.URL)
if respCode != test.expectedHTTPStatusCode {
t.Fatalf("expected status code: %v, got: %v", test.expectedHTTPStatusCode, respCode)
}
if test.expectedHTTPStatusCode == http.StatusOK && string(body) != "ok" {
t.Fatalf("expected response body to be 'ok', got: %s", string(body))
}
})
}
}
func TestCheckRPC(t *testing.T) {
socketPath := fmt.Sprintf("%s/kms.sock", getTempTestDir(t))
defer os.Remove(socketPath)
fakeKMSV1Server, fakeKMSV2Server, mockKVClient, err := setupFakeKMSServer(socketPath)
if err != nil {
t.Fatalf("failed to create fake kms server, err: %+v", err)
}
healthz := &HealthZ{
KMSv1Server: fakeKMSV1Server,
KMSv2Server: fakeKMSV2Server,
UnixSocketPath: socketPath,
}
mockKVClient.SetEncryptResponse([]byte(healthCheckPlainText), nil)
mockKVClient.SetDecryptResponse([]byte(healthCheckPlainText), nil)
conn, err := healthz.dialUnixSocket()
if err != nil {
t.Fatalf("failed to create connection, err: %+v", err)
}
err = healthz.checkRPC(
context.TODO(),
kmsv1.NewKeyManagementServiceClient(conn),
kmsv2.NewKeyManagementServiceClient(conn),
)
if err != nil {
t.Fatalf("expected err to be nil, got: %+v", err)
}
}
func getTempTestDir(t *testing.T) string {
tmpDir, err := os.MkdirTemp("", "ut")
if err != nil {
t.Fatalf("expected err to be nil, got: %+v", err)
}
return tmpDir
}
func setupFakeKMSServer(socketPath string) (
*KeyManagementServiceServer,
*KeyManagementServiceV2Server,
*mockkeyvault.KeyVaultClient,
error,
) {
listener, err := net.Listen("unix", socketPath)
if err != nil {
return nil, nil, nil, err
}
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
return nil, nil, nil, err
}
kvClient := &mockkeyvault.KeyVaultClient{
KeyID: "mock-key-id",
Algorithm: keyvault.RSA15,
}
fakeKMSV1Server := &KeyManagementServiceServer{
kvClient: kvClient,
reporter: statsReporter,
}
fakeKMSV2Server := &KeyManagementServiceV2Server{
kvClient: kvClient,
reporter: statsReporter,
}
s := grpc.NewServer()
kmsv1.RegisterKeyManagementServiceServer(s, fakeKMSV1Server)
kmsv2.RegisterKeyManagementServiceServer(s, fakeKMSV2Server)
go func() {
if err := s.Serve(listener); err != nil {
mlog.Fatal(err, "failed to serve fake kms server")
}
}()
return fakeKMSV1Server, fakeKMSV2Server, kvClient, nil
}
func doHealthCheck(t *testing.T, url string) (int, []byte) {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
t.Fatalf("failed to create new http request, err: %+v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("failed to invoke http request, err: %+v", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
t.Fatalf("failed to read response body, err: %+v", err)
}
return resp.StatusCode, body
}
================================================
FILE: pkg/plugin/keyvault.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"context"
"crypto/sha256"
"encoding/base64"
"fmt"
"net/url"
"path"
"regexp"
"strings"
"github.com/Azure/kubernetes-kms/pkg/auth"
"github.com/Azure/kubernetes-kms/pkg/config"
"github.com/Azure/kubernetes-kms/pkg/consts"
"github.com/Azure/kubernetes-kms/pkg/utils"
"github.com/Azure/kubernetes-kms/pkg/version"
kv "github.com/Azure/azure-sdk-for-go/services/keyvault/2016-10-01/keyvault"
"github.com/Azure/go-autorest/autorest"
"github.com/Azure/go-autorest/autorest/azure"
"k8s.io/kms/pkg/service"
"monis.app/mlog"
)
// encryptionResponseVersion is validated prior to decryption.
// This is helpful in case we want to change anything about the data we send in the future.
var encryptionResponseVersion = "1"
const (
dateAnnotationKey = "date.azure.akv.io"
requestIDAnnotationKey = "x-ms-request-id.azure.akv.io"
keyvaultRegionAnnotationKey = "x-ms-keyvault-region.azure.akv.io"
versionAnnotationKey = "version.azure.akv.io"
algorithmAnnotationKey = "algorithm.azure.akv.io"
dateAnnotationValue = "Date"
requestIDAnnotationValue = "X-Ms-Request-Id"
keyvaultRegionAnnotationValue = "X-Ms-Keyvault-Region"
)
// Client interface for interacting with Keyvault.
type Client interface {
Encrypt(
ctx context.Context,
plain []byte,
encryptionAlgorithm kv.JSONWebKeyEncryptionAlgorithm,
) (*service.EncryptResponse, error)
Decrypt(
ctx context.Context,
cipher []byte,
encryptionAlgorithm kv.JSONWebKeyEncryptionAlgorithm,
apiVersion string,
annotations map[string][]byte,
decryptRequestKeyID string,
) ([]byte, error)
GetUserAgent() string
GetVaultURL() string
}
// KeyVaultClient is a client for interacting with Keyvault.
type KeyVaultClient struct {
baseClient kv.BaseClient
config *config.AzureConfig
vaultName string
keyName string
keyVersion string
vaultURL string
keyIDHash string
azureEnvironment *azure.Environment
}
// NewKeyVaultClient returns a new key vault client to use for kms operations.
func NewKeyVaultClient(
config *config.AzureConfig,
vaultName, keyName, keyVersion string,
proxyMode bool,
proxyAddress string,
proxyPort int,
managedHSM bool,
) (Client, error) {
// Sanitize vaultName, keyName, keyVersion. (https://github.com/Azure/kubernetes-kms/issues/85)
vaultName = utils.SanitizeString(vaultName)
keyName = utils.SanitizeString(keyName)
keyVersion = utils.SanitizeString(keyVersion)
// this should be the case for bring your own key, clusters bootstrapped with
// aks-engine or aks and standalone kms plugin deployments
if len(vaultName) == 0 || len(keyName) == 0 || len(keyVersion) == 0 {
return nil, fmt.Errorf("key vault name, key name and key version are required")
}
kvClient := kv.New()
err := kvClient.AddToUserAgent(version.GetUserAgent())
if err != nil {
return nil, fmt.Errorf("failed to add user agent to keyvault client, error: %w", err)
}
env, err := auth.ParseAzureEnvironment(config.Cloud)
if err != nil {
return nil, fmt.Errorf("failed to parse cloud environment: %s, error: %w", config.Cloud, err)
}
if proxyMode {
env.ActiveDirectoryEndpoint = fmt.Sprintf("http://%s:%d/", proxyAddress, proxyPort)
}
vaultResourceURL := getVaultResourceIdentifier(managedHSM, env)
if vaultResourceURL == azure.NotAvailable {
return nil, fmt.Errorf("keyvault resource identifier not available for cloud: %s", env.Name)
}
token, err := auth.GetKeyvaultToken(config, env, vaultResourceURL, proxyMode)
if err != nil {
return nil, fmt.Errorf("failed to get key vault token, error: %w", err)
}
kvClient.Authorizer = token
vaultURL, err := getVaultURL(vaultName, managedHSM, env)
if err != nil {
return nil, fmt.Errorf("failed to get vault url, error: %w", err)
}
keyIDHash, err := getKeyIDHash(*vaultURL, keyName, keyVersion)
if err != nil {
return nil, fmt.Errorf("failed to get key id hash, error: %w", err)
}
if proxyMode {
kvClient.RequestInspector = autorest.WithHeader(consts.RequestHeaderTargetType, consts.TargetTypeKeyVault)
vaultURL = getProxiedVaultURL(vaultURL, proxyAddress, proxyPort)
}
mlog.Always("using kms key for encrypt/decrypt", "vaultURL", *vaultURL, "keyName", keyName, "keyVersion", keyVersion)
client := &KeyVaultClient{
baseClient: kvClient,
config: config,
vaultName: vaultName,
keyName: keyName,
keyVersion: keyVersion,
vaultURL: *vaultURL,
azureEnvironment: env,
keyIDHash: keyIDHash,
}
return client, nil
}
// Encrypt encrypts the given plain text using the keyvault key.
func (kvc *KeyVaultClient) Encrypt(
ctx context.Context,
plain []byte,
encryptionAlgorithm kv.JSONWebKeyEncryptionAlgorithm,
) (*service.EncryptResponse, error) {
value := base64.RawURLEncoding.EncodeToString(plain)
params := kv.KeyOperationsParameters{
Algorithm: encryptionAlgorithm,
Value: &value,
}
result, err := kvc.baseClient.Encrypt(ctx, kvc.vaultURL, kvc.keyName, kvc.keyVersion, params)
if err != nil {
return nil, fmt.Errorf("failed to encrypt, error: %w", err)
}
if kvc.keyIDHash != fmt.Sprintf("%x", sha256.Sum256([]byte(*result.Kid))) {
return nil, fmt.Errorf(
"key id initialized does not match with the key id from encryption result, expected: %s, got: %s",
kvc.keyIDHash,
*result.Kid,
)
}
annotations := map[string][]byte{
dateAnnotationKey: []byte(result.Header.Get(dateAnnotationValue)),
requestIDAnnotationKey: []byte(result.Header.Get(requestIDAnnotationValue)),
keyvaultRegionAnnotationKey: []byte(result.Header.Get(keyvaultRegionAnnotationValue)),
versionAnnotationKey: []byte(encryptionResponseVersion),
algorithmAnnotationKey: []byte(encryptionAlgorithm),
}
return &service.EncryptResponse{
Ciphertext: []byte(*result.Result),
KeyID: kvc.keyIDHash,
Annotations: annotations,
}, nil
}
// Decrypt decrypts the given cipher text using the keyvault key.
func (kvc *KeyVaultClient) Decrypt(
ctx context.Context,
cipher []byte,
encryptionAlgorithm kv.JSONWebKeyEncryptionAlgorithm,
apiVersion string,
annotations map[string][]byte,
decryptRequestKeyID string,
) ([]byte, error) {
if apiVersion == version.KMSv2APIVersion {
err := kvc.validateAnnotations(annotations, decryptRequestKeyID, encryptionAlgorithm)
if err != nil {
return nil, err
}
}
value := string(cipher)
params := kv.KeyOperationsParameters{
Algorithm: encryptionAlgorithm,
Value: &value,
}
result, err := kvc.baseClient.Decrypt(ctx, kvc.vaultURL, kvc.keyName, kvc.keyVersion, params)
if err != nil {
return nil, fmt.Errorf("failed to decrypt, error: %w", err)
}
bytes, err := base64.RawURLEncoding.DecodeString(*result.Result)
if err != nil {
return nil, fmt.Errorf("failed to base64 decode result, error: %w", err)
}
return bytes, nil
}
func (kvc *KeyVaultClient) GetUserAgent() string {
return kvc.baseClient.UserAgent
}
func (kvc *KeyVaultClient) GetVaultURL() string {
return kvc.vaultURL
}
// ValidateAnnotations validates following annotations before decryption:
// - Algorithm.
// - Version.
// It also validates keyID that the API server checks.
func (kvc *KeyVaultClient) validateAnnotations(
annotations map[string][]byte,
keyID string,
encryptionAlgorithm kv.JSONWebKeyEncryptionAlgorithm,
) error {
if len(annotations) == 0 {
return fmt.Errorf("invalid annotations, annotations cannot be empty")
}
if keyID != kvc.keyIDHash {
return fmt.Errorf(
"key id %s does not match expected key id %s used for encryption",
keyID,
kvc.keyIDHash,
)
}
algorithm := string(annotations[algorithmAnnotationKey])
if algorithm != string(encryptionAlgorithm) {
return fmt.Errorf(
"algorithm %s does not match expected algorithm %s used for encryption",
algorithm,
encryptionAlgorithm,
)
}
version := string(annotations[versionAnnotationKey])
if version != encryptionResponseVersion {
return fmt.Errorf(
"version %s does not match expected version %s used for encryption",
version,
encryptionResponseVersion,
)
}
return nil
}
func getVaultURL(vaultName string, managedHSM bool, env *azure.Environment) (vaultURL *string, err error) {
// Key Vault name must be a 3-24 character string
if len(vaultName) < 3 || len(vaultName) > 24 {
return nil, fmt.Errorf("invalid vault name: %q, must be between 3 and 24 chars", vaultName)
}
// See docs for validation spec: https://docs.microsoft.com/en-us/azure/key-vault/about-keys-secrets-and-certificates#objects-identifiers-and-versioning
isValid := regexp.MustCompile(`^[-A-Za-z0-9]+$`).MatchString
if !isValid(vaultName) {
return nil, fmt.Errorf("invalid vault name: %q, must match [-a-zA-Z0-9]{3,24}", vaultName)
}
vaultDNSSuffixValue := getVaultDNSSuffix(managedHSM, env)
if vaultDNSSuffixValue == azure.NotAvailable {
return nil, fmt.Errorf("vault dns suffix not available for cloud: %s", env.Name)
}
vaultURI := fmt.Sprintf("https://%s.%s/", vaultName, vaultDNSSuffixValue)
return &vaultURI, nil
}
func getProxiedVaultURL(vaultURL *string, proxyAddress string, proxyPort int) *string {
proxiedVaultURL := fmt.Sprintf("http://%s:%d/%s", proxyAddress, proxyPort, strings.TrimPrefix(*vaultURL, "https://"))
return &proxiedVaultURL
}
func getVaultDNSSuffix(managedHSM bool, env *azure.Environment) string {
if managedHSM {
return env.ManagedHSMDNSSuffix
}
return env.KeyVaultDNSSuffix
}
func getVaultResourceIdentifier(managedHSM bool, env *azure.Environment) string {
if managedHSM {
return env.ResourceIdentifiers.ManagedHSM
}
return env.ResourceIdentifiers.KeyVault
}
func getKeyIDHash(vaultURL, keyName, keyVersion string) (string, error) {
if vaultURL == "" || keyName == "" || keyVersion == "" {
return "", fmt.Errorf("vault url, key name and key version cannot be empty")
}
baseURL, err := url.Parse(vaultURL)
if err != nil {
return "", fmt.Errorf("failed to parse vault url, error: %w", err)
}
urlPath := path.Join("keys", keyName, keyVersion)
keyID := baseURL.ResolveReference(
&url.URL{
Path: urlPath,
},
).String()
return fmt.Sprintf("%x", sha256.Sum256([]byte(keyID))), nil
}
================================================
FILE: pkg/plugin/keyvault_test.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"fmt"
"strings"
"testing"
"github.com/Azure/kubernetes-kms/pkg/auth"
"github.com/Azure/kubernetes-kms/pkg/config"
)
var (
testEnvs = []string{"", "AZUREPUBLICCLOUD", "AZURECHINACLOUD", "AZUREGERMANCLOUD", "AZUREUSGOVERNMENTCLOUD"}
vaultDNSSuffix = []string{"vault.azure.net", "vault.azure.net", "vault.azure.cn", "vault.microsoftazure.de", "vault.usgovcloudapi.net"}
)
func TestNewKeyVaultClientError(t *testing.T) {
tests := []struct {
desc string
config *config.AzureConfig
vaultName string
keyName string
keyVersion string
proxyMode bool
proxyAddress string
proxyPort int
managedHSM bool
}{
{
desc: "vault name not provided",
config: &config.AzureConfig{},
proxyMode: false,
},
{
desc: "key name not provided",
config: &config.AzureConfig{},
vaultName: "testkv",
proxyMode: false,
},
{
desc: "key version not provided",
config: &config.AzureConfig{},
vaultName: "testkv",
keyName: "k8s",
proxyMode: false,
},
{
desc: "no credentials in config",
config: &config.AzureConfig{},
vaultName: "testkv",
keyName: "key1",
keyVersion: "262067a9e8ba401aa8a746c5f1a7e147",
},
{
desc: "managed hsm not available in the azure environment",
config: &config.AzureConfig{ClientID: "clientid", ClientSecret: "clientsecret", Cloud: "AzureGermanCloud"},
vaultName: "testkv",
keyName: "key1",
keyVersion: "262067a9e8ba401aa8a746c5f1a7e147",
managedHSM: true,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
if _, err := NewKeyVaultClient(test.config, test.vaultName, test.keyName, test.keyVersion, test.proxyMode, test.proxyAddress, test.proxyPort, test.managedHSM); err == nil {
t.Fatalf("newKeyVaultClient() expected error, got nil")
}
})
}
}
func TestNewKeyVaultClient(t *testing.T) {
tests := []struct {
desc string
config *config.AzureConfig
vaultName string
keyName string
keyVersion string
proxyMode bool
proxyAddress string
proxyPort int
managedHSM bool
expectedVaultURL string
}{
{
desc: "no error",
config: &config.AzureConfig{ClientID: "clientid", ClientSecret: "clientsecret"},
vaultName: "testkv",
keyName: "key1",
keyVersion: "262067a9e8ba401aa8a746c5f1a7e147",
proxyMode: false,
expectedVaultURL: "https://testkv.vault.azure.net/",
},
{
desc: "no error with double quotes",
config: &config.AzureConfig{ClientID: "clientid", ClientSecret: "clientsecret"},
vaultName: "\"testkv\"",
keyName: "\"key1\"",
keyVersion: "\"262067a9e8ba401aa8a746c5f1a7e147\"",
proxyMode: false,
expectedVaultURL: "https://testkv.vault.azure.net/",
},
{
desc: "no error with proxy mode",
config: &config.AzureConfig{ClientID: "clientid", ClientSecret: "clientsecret"},
vaultName: "testkv",
keyName: "key1",
keyVersion: "262067a9e8ba401aa8a746c5f1a7e147",
proxyMode: true,
proxyAddress: "localhost",
proxyPort: 7788,
expectedVaultURL: "http://localhost:7788/testkv.vault.azure.net/",
},
{
desc: "no error with managed hsm",
config: &config.AzureConfig{ClientID: "clientid", ClientSecret: "clientsecret"},
vaultName: "testkv",
keyName: "key1",
keyVersion: "262067a9e8ba401aa8a746c5f1a7e147",
managedHSM: true,
proxyMode: false,
expectedVaultURL: "https://testkv.managedhsm.azure.net/",
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
kvClient, err := NewKeyVaultClient(test.config, test.vaultName, test.keyName, test.keyVersion, test.proxyMode, test.proxyAddress, test.proxyPort, test.managedHSM)
if err != nil {
t.Fatalf("newKeyVaultClient() failed with error: %v", err)
}
if kvClient == nil {
t.Fatalf("newKeyVaultClient() expected kv client to not be nil")
}
if !strings.Contains(kvClient.GetUserAgent(), "k8s-kms-keyvault") {
t.Fatalf("newKeyVaultClient() expected k8s-kms-keyvault user agent")
}
if kvClient.GetVaultURL() != test.expectedVaultURL {
t.Fatalf("expected vault URL: %v, got vault URL: %v", test.expectedVaultURL, kvClient.GetVaultURL())
}
})
}
}
func TestGetVaultURLError(t *testing.T) {
tests := []struct {
desc string
vaultName string
managedHSM bool
}{
{
desc: "vault name > 24",
vaultName: "longkeyvaultnamewhichisnotvalid",
},
{
desc: "vault name < 3",
vaultName: "kv",
},
{
desc: "vault name contains non alpha-numeric chars",
vaultName: "kv_test",
},
}
for _, test := range tests {
for idx := range testEnvs {
t.Run(fmt.Sprintf("%s/%s", test.desc, testEnvs[idx]), func(t *testing.T) {
azEnv, err := auth.ParseAzureEnvironment(testEnvs[idx])
if err != nil {
t.Fatalf("failed to parse azure environment from name, err: %+v", err)
}
if _, err = getVaultURL(test.vaultName, test.managedHSM, azEnv); err == nil {
t.Fatalf("getVaultURL() expected error, got nil")
}
})
}
}
}
func TestGetVaultURL(t *testing.T) {
vaultName := "testkv"
for idx := range testEnvs {
t.Run(testEnvs[idx], func(t *testing.T) {
azEnv, err := auth.ParseAzureEnvironment(testEnvs[idx])
if err != nil {
t.Fatalf("failed to parse azure environment from name, err: %+v", err)
}
vaultURL, err := getVaultURL(vaultName, false, azEnv)
if err != nil {
t.Fatalf("expected no error of getting vault URL, got error: %v", err)
}
expectedURL := "https://" + vaultName + "." + vaultDNSSuffix[idx] + "/"
if expectedURL != *vaultURL {
t.Fatalf("expected vault url: %s, got: %s", expectedURL, *vaultURL)
}
})
}
}
func TestGetKeyIDHash(t *testing.T) {
testCases := []struct {
name string
vaultURL string
keyName string
keyVersion string
expectedHash string
expectedError bool
expectedErrorString string
}{
{
name: "valid hash",
vaultURL: "https://example.vault.azure.net/",
keyName: "mykey",
keyVersion: "ABCD",
expectedHash: "567d783db3043fe298fe0d9eeedb0029a3815cdd4fe4b059d018c91e6acffe3b",
expectedError: false,
},
{
name: "invalid vault URL",
vaultURL: ":invalid-url:",
keyName: "mykey",
keyVersion: "ABCD",
expectedHash: "",
expectedError: true,
expectedErrorString: "failed to parse vault url, error: parse \":invalid-url:\": missing protocol scheme",
},
{
name: "empty vault name",
vaultURL: "",
keyName: "mykey",
keyVersion: "ABCD",
expectedHash: "",
expectedError: true,
expectedErrorString: "vault url, key name and key version cannot be empty",
},
{
name: "empty key name",
vaultURL: "https://example.vault.azure.net/",
keyName: "",
keyVersion: "ABCD",
expectedHash: "",
expectedError: true,
expectedErrorString: "vault url, key name and key version cannot be empty",
},
{
name: "empty key vesion",
vaultURL: "https://example.vault.azure.net/",
keyName: "mykey",
keyVersion: "",
expectedHash: "",
expectedError: true,
expectedErrorString: "vault url, key name and key version cannot be empty",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
hash, err := getKeyIDHash(tc.vaultURL, tc.keyName, tc.keyVersion)
if tc.expectedError {
if (err != nil) && (err.Error() != tc.expectedErrorString) {
t.Errorf("Expected error: %v, but got: %v", tc.expectedErrorString, err.Error())
} else if err == nil {
t.Errorf("Expected error: %v, but didn't get any", tc.expectedErrorString)
}
}
if hash != tc.expectedHash {
t.Errorf("Expected hash: %s, but got: %s", tc.expectedHash, hash)
}
})
}
}
================================================
FILE: pkg/plugin/kms_v2_server.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"context"
"fmt"
"time"
"github.com/Azure/kubernetes-kms/pkg/metrics"
"github.com/Azure/kubernetes-kms/pkg/version"
"github.com/Azure/azure-sdk-for-go/services/keyvault/2016-10-01/keyvault"
kmsv2 "k8s.io/kms/apis/v2"
"monis.app/mlog"
)
// KeyManagementServiceV2Server is a gRPC server.
type KeyManagementServiceV2Server struct {
kmsv2.UnimplementedKeyManagementServiceServer
kvClient Client
reporter metrics.StatsReporter
encryptionAlgorithm keyvault.JSONWebKeyEncryptionAlgorithm
}
// NewKMSv2Server creates an instance of the KMS Service Server with v2 apis.
func NewKMSv2Server(kvClient Client) (*KeyManagementServiceV2Server, error) {
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
return nil, fmt.Errorf("failed to create stats reporter: %w", err)
}
return &KeyManagementServiceV2Server{
kvClient: kvClient,
reporter: statsReporter,
encryptionAlgorithm: keyvault.RSAOAEP256,
}, nil
}
// Status returns the health status of the KMS plugin.
func (s *KeyManagementServiceV2Server) Status(ctx context.Context, _ *kmsv2.StatusRequest) (*kmsv2.StatusResponse, error) {
// We perform a simple encrypt/decrypt operation to verify the plugin's connectivity with Key Vault.
// The KMS invokes the Status API every minute, resulting in 120 calls per hour to the Key Vault.
// This volume of calls is well within the permissible limit of Key Vault.
encryptResponse, err := s.kvClient.Encrypt(ctx, []byte(healthCheckPlainText), s.encryptionAlgorithm)
if err != nil {
mlog.Error("failed to encrypt healthcheck call", err)
return nil, err
}
decryptedText, err := s.kvClient.Decrypt(
ctx,
encryptResponse.Ciphertext,
s.encryptionAlgorithm,
version.KMSv2APIVersion,
encryptResponse.Annotations,
encryptResponse.KeyID,
)
if err != nil {
mlog.Error("failed to decrypt healthcheck call", err)
return nil, err
}
if string(decryptedText) != healthCheckPlainText {
err = fmt.Errorf("decrypted text does not match")
mlog.Error("healthcheck failed", err)
return nil, err
}
return &kmsv2.StatusResponse{
Version: version.KMSv2APIVersion,
Healthz: "ok",
KeyId: encryptResponse.KeyID,
}, nil
}
// Encrypt message.
func (s *KeyManagementServiceV2Server) Encrypt(ctx context.Context, request *kmsv2.EncryptRequest) (*kmsv2.EncryptResponse, error) {
mlog.Debug("encrypt request received", "uid", request.Uid)
start := time.Now()
var err error
defer func() {
errors := ""
status := metrics.SuccessStatusTypeValue
if err != nil {
status = metrics.ErrorStatusTypeValue
errors = err.Error()
}
s.reporter.ReportRequest(ctx, metrics.EncryptOperationTypeValue, status, time.Since(start).Seconds(), errors)
}()
mlog.Info("encrypt request started", "uid", request.Uid)
encryptResponse, err := s.kvClient.Encrypt(ctx, request.Plaintext, s.encryptionAlgorithm)
if err != nil {
mlog.Error("failed to encrypt", err, "uid", request.Uid)
return &kmsv2.EncryptResponse{}, err
}
mlog.Info("encrypt request complete", "uid", request.Uid)
return &kmsv2.EncryptResponse{
Ciphertext: encryptResponse.Ciphertext,
KeyId: encryptResponse.KeyID,
Annotations: encryptResponse.Annotations,
}, nil
}
// Decrypt message.
func (s *KeyManagementServiceV2Server) Decrypt(ctx context.Context, request *kmsv2.DecryptRequest) (*kmsv2.DecryptResponse, error) {
mlog.Debug("decrypt request received", "uid", request.Uid)
start := time.Now()
var err error
defer func() {
errors := ""
status := metrics.SuccessStatusTypeValue
if err != nil {
status = metrics.ErrorStatusTypeValue
errors = err.Error()
}
s.reporter.ReportRequest(ctx, metrics.DecryptOperationTypeValue, status, time.Since(start).Seconds(), errors)
}()
mlog.Info("decrypt request started", "uid", request.Uid)
plainText, err := s.kvClient.Decrypt(
ctx,
request.Ciphertext,
s.encryptionAlgorithm,
version.KMSv2APIVersion,
request.Annotations,
request.KeyId,
)
if err != nil {
mlog.Error("failed to decrypt", err, "uid", request.Uid)
return &kmsv2.DecryptResponse{}, err
}
mlog.Info("decrypt request complete", "uid", request.Uid)
return &kmsv2.DecryptResponse{
Plaintext: plainText,
}, nil
}
================================================
FILE: pkg/plugin/kms_v2_server_test.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"bytes"
"context"
"errors"
"fmt"
"testing"
"github.com/Azure/azure-sdk-for-go/services/keyvault/2016-10-01/keyvault"
"github.com/Azure/kubernetes-kms/pkg/metrics"
mockkeyvault "github.com/Azure/kubernetes-kms/pkg/plugin/mock_keyvault"
"github.com/Azure/kubernetes-kms/pkg/version"
kmsv2 "k8s.io/kms/apis/v2"
)
func TestV2Encrypt(t *testing.T) {
tests := []struct {
desc string
input []byte
output []byte
err error
}{
{
desc: "failed to encrypt",
input: []byte("foo"),
output: []byte{},
err: fmt.Errorf("failed to encrypt"),
},
{
desc: "successfully encrypted",
input: []byte("foo"),
output: []byte("bar"),
err: nil,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
kvClient := &mockkeyvault.KeyVaultClient{
KeyID: "mock-key-id",
Algorithm: keyvault.RSA15,
}
kvClient.SetEncryptResponse(test.output, test.err)
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
t.Fatalf("failed to create stats reporter: %v", err)
}
kmsV2Server := KeyManagementServiceV2Server{
kvClient: kvClient,
reporter: statsReporter,
}
out, err := kmsV2Server.Encrypt(context.TODO(), &kmsv2.EncryptRequest{
Plaintext: test.input,
})
if !errors.Is(err, test.err) {
t.Fatalf("expected err: %v, got: %v", test.err, err)
}
if !bytes.Equal(out.GetCiphertext(), test.output) {
t.Fatalf("expected out: %v, got: %v", test.output, out)
}
if err == nil && (out.KeyId != kvClient.KeyID) {
t.Fatalf("expected key id: %v, got: %v", kvClient.KeyID, out.KeyId)
}
if err == nil && (len(out.Annotations) == 0) {
t.Fatalf("invalid annotations, annotations cannot be empty")
}
})
}
}
func TestV2Decrypt(t *testing.T) {
tests := []struct {
desc string
input []byte
output []byte
err error
annotations map[string][]byte
}{
{
desc: "empty annotations failed to decrypt",
input: []byte("bar"),
output: []byte{},
err: fmt.Errorf("invalid annotations, annotations cannot be empty"),
},
{
desc: "invalid keyid failed to decrypt",
input: []byte("bar"),
output: []byte{},
err: fmt.Errorf("key id \"invalid-key-id\" does not match expected key id \"mock-key-id\" used for encryption"),
annotations: map[string][]byte{
algorithmAnnotationKey: []byte(keyvault.RSA15),
versionAnnotationKey: []byte("1"),
},
},
{
desc: "invalid algorithm failed to decrypt",
input: []byte("bar"),
output: []byte{},
err: fmt.Errorf("algorithm \"insecure-algorithm\" does not match expected algorithm \"RSAOAEP256\" used for encryption"),
annotations: map[string][]byte{
algorithmAnnotationKey: []byte("insecure-algorithm"),
versionAnnotationKey: []byte("1"),
},
},
{
desc: "invalid version failed to decrypt",
input: []byte("bar"),
output: []byte{},
err: fmt.Errorf("version \"10\" does not match expected version \"1\" used for encryption"),
annotations: map[string][]byte{
algorithmAnnotationKey: []byte(keyvault.RSA15),
versionAnnotationKey: []byte("10"),
},
},
{
desc: "failed to decrypt",
input: []byte("foo"),
output: []byte{},
err: fmt.Errorf("failed to decrypt"),
annotations: map[string][]byte{
algorithmAnnotationKey: []byte(keyvault.RSA15),
versionAnnotationKey: []byte("1"),
},
},
{
desc: "successfully decrypted",
input: []byte("bar"),
output: []byte("foo"),
err: nil,
annotations: map[string][]byte{
algorithmAnnotationKey: []byte(keyvault.RSA15),
versionAnnotationKey: []byte("1"),
},
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
kvClient := &mockkeyvault.KeyVaultClient{
KeyID: "mock-key-id",
Algorithm: keyvault.RSAOAEP256,
}
kvClient.SetDecryptResponse(test.output, test.err)
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
t.Fatalf("failed to create stats reporter: %v", err)
}
kmsV2Server := KeyManagementServiceV2Server{
kvClient: kvClient,
reporter: statsReporter,
}
out, err := kmsV2Server.Decrypt(context.TODO(), &kmsv2.DecryptRequest{
Ciphertext: test.input,
Annotations: test.annotations,
KeyId: "mock-key-id",
})
if err != nil && (err.Error() != test.err.Error()) {
t.Fatalf("expected err: %v, got: %v", test.err, err)
}
if !bytes.Equal(out.GetPlaintext(), test.output) {
t.Fatalf("expected out: %v, got: %v", test.output, out)
}
})
}
}
func TestStatus(t *testing.T) {
kmsServer := KeyManagementServiceV2Server{}
mockKeyVaultClient := &mockkeyvault.KeyVaultClient{
KeyID: "mock-key-id",
}
mockKeyVaultClient.SetEncryptResponse([]byte(healthCheckPlainText), nil)
mockKeyVaultClient.SetDecryptResponse([]byte(healthCheckPlainText), nil)
kmsServer.kvClient = mockKeyVaultClient
v, err := kmsServer.Status(context.TODO(), &kmsv2.StatusRequest{})
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
if v.Version != version.KMSv2APIVersion {
t.Fatalf("expected version: %s, got: %s", version.KMSv2APIVersion, v.Version)
}
if v.Healthz != "ok" {
t.Fatalf("expected healthz response to be: %s, got: %s", "ok", v.Healthz)
}
if v.KeyId != "mock-key-id" {
t.Fatalf("expected key id: %s, got: %s", "mock-key-id", v.KeyId)
}
}
================================================
FILE: pkg/plugin/mock_keyvault/keyvault_mock.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package mockkeyvault
import (
"context"
"fmt"
"sync"
"github.com/Azure/azure-sdk-for-go/services/keyvault/2016-10-01/keyvault"
"k8s.io/kms/pkg/service"
)
type KeyVaultClient struct {
mutex sync.Mutex
encryptOut []byte
encryptErr error
decryptOut []byte
decryptErr error
KeyID string
Algorithm keyvault.JSONWebKeyEncryptionAlgorithm
}
func (kvc *KeyVaultClient) Encrypt(_ context.Context, _ []byte, _ keyvault.JSONWebKeyEncryptionAlgorithm) (*service.EncryptResponse, error) {
kvc.mutex.Lock()
defer kvc.mutex.Unlock()
return &service.EncryptResponse{
Ciphertext: kvc.encryptOut,
KeyID: kvc.KeyID,
Annotations: map[string][]byte{
"key-id.azure.akv.io": []byte(kvc.KeyID),
"algorithm.azure.akv.io": []byte(kvc.Algorithm),
"version.azure.akv.io": []byte("1"),
},
}, kvc.encryptErr
}
func (kvc *KeyVaultClient) Decrypt(_ context.Context, _ []byte, _ keyvault.JSONWebKeyEncryptionAlgorithm, _ string, _ map[string][]byte, _ string) ([]byte, error) {
kvc.mutex.Lock()
defer kvc.mutex.Unlock()
return kvc.decryptOut, kvc.decryptErr
}
func (kvc *KeyVaultClient) SetEncryptResponse(encryptOut []byte, err error) {
kvc.mutex.Lock()
defer kvc.mutex.Unlock()
kvc.encryptOut = encryptOut
kvc.encryptErr = err
}
func (kvc *KeyVaultClient) SetDecryptResponse(decryptOut []byte, err error) {
kvc.mutex.Lock()
defer kvc.mutex.Unlock()
kvc.decryptOut = decryptOut
kvc.decryptErr = err
}
func (kvc *KeyVaultClient) ValidateAnnotations(annotations map[string][]byte, keyID string) error {
if len(annotations) == 0 {
return fmt.Errorf("invalid annotations, annotations cannot be empty")
}
// validate key id
if keyID != kvc.KeyID {
return fmt.Errorf(
"key id %q does not match expected key id %q used for encryption",
string(annotations["key-id.azure.akv.io"]),
kvc.KeyID,
)
}
// validate algorithm
if string(annotations["algorithm.azure.akv.io"]) != string(kvc.Algorithm) {
return fmt.Errorf("algorithm %q does not match expected algorithm %q used for encryption", string(annotations["algorithm.azure.akv.io"]), kvc.Algorithm)
}
// validate version
if string(annotations["version.azure.akv.io"]) != "1" {
return fmt.Errorf(
"version %q does not match expected version %q used for encryption",
string(annotations["version.azure.akv.io"]),
"1",
)
}
return nil
}
func (kvc *KeyVaultClient) GetUserAgent() string {
return "k8s-kms-keyvault"
}
func (kvc *KeyVaultClient) GetVaultURL() string {
return "https://test.vault.azure.net"
}
================================================
FILE: pkg/plugin/server.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"context"
"fmt"
"time"
"github.com/Azure/kubernetes-kms/pkg/metrics"
"github.com/Azure/kubernetes-kms/pkg/version"
"github.com/Azure/azure-sdk-for-go/services/keyvault/2016-10-01/keyvault"
kmsv1 "k8s.io/kms/apis/v1beta1"
"monis.app/mlog"
)
// KeyManagementServiceServer is a gRPC server.
type KeyManagementServiceServer struct {
kmsv1.UnimplementedKeyManagementServiceServer
kvClient Client
reporter metrics.StatsReporter
encryptionAlgorithm keyvault.JSONWebKeyEncryptionAlgorithm
}
// Config is the configuration for the KMS plugin.
type Config struct {
ConfigFilePath string
KeyVaultName string
KeyName string
KeyVersion string
ManagedHSM bool
ProxyMode bool
ProxyAddress string
ProxyPort int
}
// NewKMSv1Server creates an instance of the KMS Service Server.
func NewKMSv1Server(kvClient Client) (*KeyManagementServiceServer, error) {
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
return nil, fmt.Errorf("failed to create stats reporter: %w", err)
}
return &KeyManagementServiceServer{
kvClient: kvClient,
reporter: statsReporter,
encryptionAlgorithm: keyvault.RSA15,
}, nil
}
// Version of kms.
func (s *KeyManagementServiceServer) Version(_ context.Context, _ *kmsv1.VersionRequest) (*kmsv1.VersionResponse, error) {
return &kmsv1.VersionResponse{
Version: version.KMSv1APIVersion,
RuntimeName: version.Runtime,
RuntimeVersion: version.BuildVersion,
}, nil
}
// Encrypt message.
func (s *KeyManagementServiceServer) Encrypt(ctx context.Context, request *kmsv1.EncryptRequest) (*kmsv1.EncryptResponse, error) {
start := time.Now()
var err error
defer func() {
errors := ""
status := metrics.SuccessStatusTypeValue
if err != nil {
status = metrics.ErrorStatusTypeValue
errors = err.Error()
}
s.reporter.ReportRequest(ctx, metrics.EncryptOperationTypeValue, status, time.Since(start).Seconds(), errors)
}()
mlog.Info("encrypt request started")
encryptResponse, err := s.kvClient.Encrypt(ctx, request.Plain, s.encryptionAlgorithm)
if err != nil {
mlog.Error("failed to encrypt", err)
return &kmsv1.EncryptResponse{}, err
}
mlog.Info("encrypt request complete")
return &kmsv1.EncryptResponse{
Cipher: encryptResponse.Ciphertext,
}, nil
}
// Decrypt message.
func (s *KeyManagementServiceServer) Decrypt(ctx context.Context, request *kmsv1.DecryptRequest) (*kmsv1.DecryptResponse, error) {
start := time.Now()
var err error
defer func() {
errors := ""
status := metrics.SuccessStatusTypeValue
if err != nil {
status = metrics.ErrorStatusTypeValue
errors = err.Error()
}
s.reporter.ReportRequest(ctx, metrics.DecryptOperationTypeValue, status, time.Since(start).Seconds(), errors)
}()
mlog.Info("decrypt request started")
plain, err := s.kvClient.Decrypt(
ctx,
request.Cipher,
s.encryptionAlgorithm,
request.Version,
nil,
"",
)
if err != nil {
mlog.Error("failed to decrypt", err)
return &kmsv1.DecryptResponse{}, err
}
mlog.Info("decrypt request complete")
return &kmsv1.DecryptResponse{Plain: plain}, nil
}
================================================
FILE: pkg/plugin/server_test.go
================================================
// Copyright (c) Microsoft and contributors. All rights reserved.
//
// This source code is licensed under the MIT license found in the
// LICENSE file in the root directory of this source tree.
package plugin
import (
"bytes"
"context"
"errors"
"fmt"
"testing"
"github.com/Azure/kubernetes-kms/pkg/metrics"
mockkeyvault "github.com/Azure/kubernetes-kms/pkg/plugin/mock_keyvault"
"github.com/Azure/kubernetes-kms/pkg/version"
kmsv1 "k8s.io/kms/apis/v1beta1"
)
func TestEncrypt(t *testing.T) {
tests := []struct {
desc string
input []byte
output []byte
err error
}{
{
desc: "failed to encrypt",
input: []byte("foo"),
output: []byte{},
err: fmt.Errorf("failed to encrypt"),
},
{
desc: "successfully encrypted",
input: []byte("foo"),
output: []byte("bar"),
err: nil,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
kvClient := &mockkeyvault.KeyVaultClient{}
kvClient.SetEncryptResponse(test.output, test.err)
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
t.Fatalf("failed to create stats reporter: %v", err)
}
kmsServer := KeyManagementServiceServer{
kvClient: kvClient,
reporter: statsReporter,
}
out, err := kmsServer.Encrypt(context.TODO(), &kmsv1.EncryptRequest{
Plain: test.input,
})
if !errors.Is(err, test.err) {
t.Fatalf("expected err: %v, got: %v", test.err, err)
}
if !bytes.Equal(out.GetCipher(), test.output) {
t.Fatalf("expected out: %v, got: %v", test.output, out)
}
})
}
}
func TestDecrypt(t *testing.T) {
tests := []struct {
desc string
input []byte
output []byte
err error
}{
{
desc: "failed to decrypt",
input: []byte("foo"),
output: []byte{},
err: fmt.Errorf("failed to decrypt"),
},
{
desc: "successfully decrypted",
input: []byte("bar"),
output: []byte("foo"),
err: nil,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
kvClient := &mockkeyvault.KeyVaultClient{}
kvClient.SetDecryptResponse(test.output, test.err)
statsReporter, err := metrics.NewStatsReporter()
if err != nil {
t.Fatalf("failed to create stats reporter: %v", err)
}
kmsServer := KeyManagementServiceServer{
kvClient: kvClient,
reporter: statsReporter,
}
out, err := kmsServer.Decrypt(context.TODO(), &kmsv1.DecryptRequest{
Cipher: test.input,
})
if !errors.Is(err, test.err) {
t.Fatalf("expected err: %v, got: %v", test.err, err)
}
if !bytes.Equal(out.GetPlain(), test.output) {
t.Fatalf("expected out: %v, got: %v", test.output, out)
}
})
}
}
func TestVersion(t *testing.T) {
kmsServer := KeyManagementServiceServer{}
version.BuildVersion = "latest"
v, err := kmsServer.Version(context.TODO(), &kmsv1.VersionRequest{})
if err != nil {
t.Fatalf("expected err to be nil, got: %v", err)
}
if v.Version != version.KMSv1APIVersion {
t.Fatalf("expected version: %s, got: %s", version.KMSv1APIVersion, v.Version)
}
if v.RuntimeName != version.Runtime {
t.Fatalf("expected runtime: %s, got: %s", version.Runtime, v.RuntimeName)
}
if v.RuntimeVersion != "latest" {
t.Fatalf("expected runtime version: %s, got: %s", version.BuildVersion, v.Version)
}
}
================================================
FILE: pkg/utils/grpc.go
================================================
package utils
import (
"context"
"fmt"
"strings"
"time"
"github.com/Azure/kubernetes-kms/pkg/metrics"
"google.golang.org/grpc"
"monis.app/mlog"
)
// ParseEndpoint returns unix socket's protocol and address.
func ParseEndpoint(ep string) (string, string, error) {
if strings.HasPrefix(strings.ToLower(ep), "unix://") {
s := strings.SplitN(ep, "://", 2)
if s[1] != "" {
return s[0], s[1], nil
}
}
return "", "", fmt.Errorf("invalid endpoint: %v", ep)
}
// UnaryServerInterceptor provides metrics around Unary RPCs.
func UnaryServerInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
var err error
start := time.Now()
reporter, err := metrics.NewStatsReporter()
if err != nil {
return nil, fmt.Errorf("failed to create stats reporter: %w", err)
}
defer func() {
errors := ""
status := metrics.SuccessStatusTypeValue
if err != nil {
status = metrics.ErrorStatusTypeValue
errors = err.Error()
}
reporter.ReportRequest(ctx, fmt.Sprintf("%s_%s", metrics.GrpcOperationTypeValue, getGRPCMethodName(info.FullMethod)), status, time.Since(start).Seconds(), errors)
}()
mlog.Trace("GRPC call", "method", info.FullMethod)
resp, err := handler(ctx, req)
if err != nil {
mlog.Error("GRPC request error", err)
}
return resp, err
}
func getGRPCMethodName(fullMethodName string) string {
fullMethodName = strings.TrimPrefix(fullMethodName, "/")
methodNames := strings.Split(fullMethodName, "/")
if len(methodNames) >= 2 {
return strings.ToLower(methodNames[1])
}
return "unknown"
}
================================================
FILE: pkg/utils/grpc_test.go
================================================
package utils
import "testing"
func TestParseEndpoint(t *testing.T) {
tests := []struct {
desc string
endpoint string
expectedProto string
expectedAddr string
expectedErr bool
}{
{
desc: "invalid endpoint",
endpoint: "udp:///provider/azure.sock",
expectedErr: true,
},
{
desc: "invalid unix endpoint",
endpoint: "unix://",
expectedProto: "",
expectedAddr: "",
expectedErr: true,
},
{
desc: "valid unix endpoint",
endpoint: "unix:///provider/azure.sock",
expectedProto: "unix",
expectedAddr: "/provider/azure.sock",
expectedErr: false,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
proto, addr, err := ParseEndpoint(test.endpoint)
if test.expectedErr && err == nil || !test.expectedErr && err != nil {
t.Fatalf("expected error: %v, got error: %v", test.expectedErr, err)
}
if proto != test.expectedProto {
t.Fatalf("expected proto: %v, got: %v", test.expectedProto, proto)
}
if addr != test.expectedAddr {
t.Fatalf("expected addr: %v, got: %v", test.expectedAddr, addr)
}
})
}
}
func TestGetGRPCMethodName(t *testing.T) {
testCases := []struct {
name string
input string
expectedOutput string
}{
{
name: "With_Correct_Method_Name",
input: "/v1beta1.KeyManagementService/Encrypt",
expectedOutput: "encrypt",
},
{
name: "With_Incorrect_Method_Name",
input: "/Encrypt",
expectedOutput: "unknown",
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
methodName := getGRPCMethodName(testCase.input)
if methodName != testCase.expectedOutput {
t.Fatalf("expected output: '%s', found: '%s'", testCase.expectedOutput, methodName)
}
})
}
}
================================================
FILE: pkg/utils/sanitize.go
================================================
package utils
import "strings"
// SanitizeString returns a string that does not have white spaces and double quotes.
func SanitizeString(s string) string {
return strings.TrimSpace(strings.Trim(strings.TrimSpace(s), "\""))
}
================================================
FILE: pkg/utils/sanitize_test.go
================================================
package utils
import "testing"
func TestSanitizeString(t *testing.T) {
testCases := []struct {
name string
input string
expectedOutput string
}{
{
name: "With_White_Spaces",
input: " hello ",
expectedOutput: "hello",
},
{
name: "With_Double_Quotes",
input: "\"hello\"",
expectedOutput: "hello",
},
{
name: "With_White_Spaces_And_Double_Quotes",
input: " \"hello\" ",
expectedOutput: "hello",
},
{
name: "With_Double_Quotes_And_White_Spaces",
input: "\" hello \"",
expectedOutput: "hello",
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
sanitizedString := SanitizeString(testCase.input)
if sanitizedString != testCase.expectedOutput {
t.Fatalf("expected output: '%s', found: '%s'", testCase.expectedOutput, sanitizedString)
}
})
}
}
================================================
FILE: pkg/version/version.go
================================================
package version
import (
"encoding/json"
"fmt"
"runtime"
)
var (
// BuildDate is the date when the binary was built.
BuildDate string
// GitCommit is the commit hash when the binary was built.
GitCommit string
// BuildVersion is the version of the KMS binary.
BuildVersion string
// KMSv1APIVersion is the version of the KMS V1 APIs.
KMSv1APIVersion = "v1beta1"
// KMSv2APIVersion is the version of the KMS V2 APIs.
KMSv2APIVersion = "v2beta1"
// Runtime of the plugin.
Runtime = "Microsoft AzureKMS"
)
// PrintVersion prints the current KMS plugin version.
func PrintVersion() (err error) {
pv := struct {
BuildVersion string
GitCommit string
BuildDate string
}{
BuildDate: BuildDate,
BuildVersion: BuildVersion,
GitCommit: GitCommit,
}
var res []byte
if res, err = json.Marshal(pv); err != nil {
return
}
fmt.Printf("%s\n", res)
return
}
// GetUserAgent returns UserAgent string to append to the agent identifier.
func GetUserAgent() string {
return fmt.Sprintf("k8s-kms-keyvault/%s (%s/%s) %s/%s", BuildVersion, runtime.GOOS, runtime.GOARCH, GitCommit, BuildDate)
}
================================================
FILE: pkg/version/version_test.go
================================================
package version
import (
"bytes"
"fmt"
"io"
"os"
"runtime"
"strings"
"testing"
)
func TestPrintVersion(t *testing.T) {
BuildDate = "Now"
BuildVersion = "version"
GitCommit = "hash"
old := os.Stdout // keep backup of the real stdout
r, w, _ := os.Pipe()
os.Stdout = w
err := PrintVersion()
outC := make(chan string)
// copy the output in a separate goroutine so printing can't block indefinitely
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, r)
outC <- strings.TrimSpace(buf.String())
}()
// back to normal state
w.Close()
os.Stdout = old // restoring the real stdout
out := <-outC
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
expected := `{"BuildVersion":"version","GitCommit":"hash","BuildDate":"Now"}`
if !strings.EqualFold(out, expected) {
t.Fatalf("string doesn't match, expected %s, got %s", expected, out)
}
}
func TestGetUserAgent(t *testing.T) {
BuildDate = "Now"
BuildVersion = "version"
GitCommit = "hash"
userAgent := GetUserAgent()
expectedUserAgent := fmt.Sprintf("k8s-kms-keyvault/version (%s/%s) hash/Now", runtime.GOOS, runtime.GOARCH)
if !strings.EqualFold(userAgent, expectedUserAgent) {
t.Fatalf("string doesn't match, expected %s, got %s", expectedUserAgent, userAgent)
}
}
================================================
FILE: scripts/connect-registry.sh
================================================
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
if [ "${KIND_NETWORK}" != "bridge" ]; then
# wait for the kind network to exist
for i in $(seq 1 25); do
if docker network ls | grep "${KIND_NETWORK}"; then
break
else
sleep 1
fi
done
containers=$(docker network inspect "${KIND_NETWORK}" -f "{{range .Containers}}{{.Name}} {{end}}")
needs_connect="true"
for c in $containers; do
if [ "$c" = "${REGISTRY_NAME}" ]; then
needs_connect="false"
fi
done
if [ "${needs_connect}" = "true" ]; then
echo "connecting ${KIND_NETWORK} network to ${REGISTRY_NAME}"
docker network connect "${KIND_NETWORK}" "${REGISTRY_NAME}" || true
fi
fi
================================================
FILE: scripts/setup-kind-cluster.sh
================================================
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
export ENCRYPTION_CONFIG_FILE=encryption-config.yaml
envsubst < ./tests/e2e/kind-config.yaml > ./tests/e2e/generated_manifests/kind-config.yaml
# create a cluster with the local registry enabled in containerd
# add encryption config and the kms static pod manifest with custom image
kind create cluster --retain --image kindest/node:"${KUBERNETES_VERSION}" --name "${KIND_CLUSTER_NAME}" --wait 2m --config=./tests/e2e/generated_manifests/kind-config.yaml
================================================
FILE: scripts/setup-kmsv2-kind-cluster.sh
================================================
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
export ENCRYPTION_CONFIG_FILE=kmsv2-encryption-config.yaml
envsubst < ./tests/e2e/kind-config.yaml > ./tests/e2e/generated_manifests/kind-config.yaml
# # create a cluster with the local registry enabled in containerd
# # add encryption config and the kms static pod manifest with custom image
kind create cluster --retain --image kindest/node:"${KUBERNETES_VERSION}" --name "${KIND_CLUSTER_NAME}" --wait 2m --config=./tests/e2e/generated_manifests/kind-config.yaml
================================================
FILE: scripts/setup-local-registry.sh
================================================
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
# create registry container unless it already exists
running="$(docker inspect -f '{{.State.Running}}' "${REGISTRY_NAME}" 2>/dev/null || true)"
if [ "${running}" != 'true' ]; then
echo "Creating local registry"
docker run \
-d --restart=always -p "${REGISTRY_PORT}:5000" --name "${REGISTRY_NAME}" \
mirror.gcr.io/registry:2
fi
# create hosts.toml for the local registry containerd config
# the certs.d directory is mounted into the kind node at /etc/containerd/certs.d
rm -rf tests/e2e/generated_manifests/certs.d
mkdir -p "tests/e2e/generated_manifests/certs.d/localhost:${REGISTRY_PORT}"
cat <<EOF > "tests/e2e/generated_manifests/certs.d/localhost:${REGISTRY_PORT}/hosts.toml"
[host."http://${REGISTRY_NAME}:5000"]
EOF
# Build and push kms image
export REGISTRY=localhost:${REGISTRY_PORT}
export IMAGE_NAME=keyvault
export IMAGE_VERSION=e2e-$(git rev-parse --short HEAD)
export OUTPUT_TYPE=type=docker
# push build image to local registry
echo "Build and push image to local registry"
make docker-init-buildx docker-build
docker push "${REGISTRY}/${IMAGE_NAME}:${IMAGE_VERSION}"
# generate manifest for local
make e2e-generate-manifests
================================================
FILE: tests/client/client_test.go
================================================
package test
import (
"bytes"
"context"
"fmt"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"k8s.io/apimachinery/pkg/util/uuid"
kmsv1 "k8s.io/kms/apis/v1beta1"
kmsv2 "k8s.io/kms/apis/v2"
)
const (
netProtocol = "unix"
pathToUnixSocket = "/opt/azurekms.sock"
version = "v1beta1"
)
var (
v1Client kmsv1.KeyManagementServiceClient
v2Client kmsv2.KeyManagementServiceClient
connection *grpc.ClientConn
t *testing.T
err error
)
func setupTestCase() {
if t != nil {
t.Log("setup test case")
connection, err = newUnixSocketConnection(pathToUnixSocket)
if err != nil {
fmt.Printf("%s", err)
}
v1Client = kmsv1.NewKeyManagementServiceClient(connection)
v2Client = kmsv2.NewKeyManagementServiceClient(connection)
}
}
func teardownTestCase() {
if t != nil {
t.Log("teardown test case")
connection.Close()
}
}
func TestEncryptDecrypt(t *testing.T) {
cases := []struct {
name string
want []byte
expected []byte
}{
{"text", []byte("secret"), []byte("secret")},
{"number", []byte("1234"), []byte("1234")},
{"special", []byte("!@#$%^&*()_"), []byte("!@#$%^&*()_")},
{"GUID", []byte("b32a58c6-48c1-4552-8ff0-47680f3416d0"), []byte("b32a58c6-48c1-4552-8ff0-47680f3416d0")},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
t.Cleanup(cancel)
v1EncryptRequest := kmsv1.EncryptRequest{Version: version, Plain: tc.want}
v1EncryptResponse, err := v1Client.Encrypt(ctx, &v1EncryptRequest)
if err != nil {
t.Fatalf("encrypt request for KMS v1 failed with error: %+v", err)
}
v1DecryptRequest := kmsv1.DecryptRequest{Version: version, Cipher: v1EncryptResponse.Cipher}
v1DecryptResponse, err := v1Client.Decrypt(ctx, &v1DecryptRequest)
if !bytes.Equal(v1DecryptResponse.Plain, tc.want) {
t.Fatalf("Expected secret, but got %s - %v", string(v1DecryptResponse.Plain), err)
}
uid := "integration-test-" + string(uuid.NewUUID())
v2EncryptRequest := kmsv2.EncryptRequest{
Plaintext: tc.want,
Uid: uid,
}
v2EncryptResponse, err := v2Client.Encrypt(ctx, &v2EncryptRequest)
if err != nil {
t.Fatalf("encrypt request for KMS v2 failed with error: %+v", err)
}
if v2EncryptResponse.KeyId == "" {
t.Fatalf("Returned KeyId is empty")
}
if v2EncryptResponse.Annotations == nil {
t.Fatalf("Returned Annotations is nil")
}
v2DecryptRequest := kmsv2.DecryptRequest{
Ciphertext: v2EncryptResponse.Ciphertext,
KeyId: v2EncryptResponse.KeyId,
Uid: uid,
Annotations: v2EncryptResponse.Annotations,
}
v2DecryptResponse, err := v2Client.Decrypt(ctx, &v2DecryptRequest)
if !bytes.Equal(v2DecryptResponse.Plaintext, tc.want) {
t.Fatalf("Expected secret, but got %s - %v", string(v2DecryptResponse.Plaintext), err)
}
})
}
}
// Check the KMS provider API version.
// Only matching version is supported now.
func TestV1Version(t *testing.T) {
cases := []struct {
name string
want string
expected string
}{
{"v1beta1", "v1beta1", "v1beta1"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
t.Cleanup(cancel)
request := &kmsv1.VersionRequest{Version: tc.want}
response, err := v1Client.Version(ctx, request)
if err != nil {
t.Fatalf("failed get version from remote KMS provider: %v", err)
}
if response.Version != tc.want {
t.Fatalf("KMS provider api version %s is not supported, only %s is supported now", tc.want, version)
}
})
}
}
func TestV2Version(t *testing.T) {
cases := []struct {
name string
want string
expected string
}{
{"v2beta1", "v2beta1", "v2beta1"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
t.Cleanup(cancel)
request := &kmsv2.StatusRequest{}
response, err := v2Client.Status(ctx, request)
if err != nil {
t.Fatalf("failed get status of remote KMS v2 provider: %v", err)
}
if response.Version != tc.want {
t.Fatalf("KMS v2 provider api version %s is not supported, only %s is supported now", tc.want, version)
}
})
}
}
func TestMain(m *testing.M) {
t = &testing.T{}
setupTestCase()
m.Run()
teardownTestCase()
}
func newUnixSocketConnection(path string) (*grpc.ClientConn, error) {
return grpc.NewClient("unix://"+path,
grpc.WithTransportCredentials(insecure.NewCredentials()))
}
================================================
FILE: tests/e2e/azure.json
================================================
{
"cloud": "AzurePublicCloud",
"tenantId": "$AZURE_TENANT_ID",
"useManagedIdentityExtension": true,
"userAssignedIdentityID": "$USER_ASSIGNED_IDENTITY_ID"
}
================================================
FILE: tests/e2e/encryption-config.yaml
================================================
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources:
- secrets
providers:
- kms:
name: azurekmsprovider
endpoint: unix:///opt/azurekms.socket
cachesize: 1000
- identity: {}
================================================
FILE: tests/e2e/helpers.bash
================================================
#!/bin/bash
assert_success() {
if [[ "$status" != 0 ]]; then
echo "expected: 0"
echo "actual: $status"
echo "output: $output"
return 1
fi
}
assert_equal() {
if [[ "$1" != "$2" ]]; then
echo "expected: $1"
echo "actual: $2"
return 1
fi
}
assert_match() {
if [[ ! "$2" =~ $1 ]]; then
echo "expected: $1"
gitextract_y5iov8qc/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── dependabot.yml
│ ├── semantic.yml
│ └── workflows/
│ ├── codeql.yaml
│ ├── create-release.yml
│ ├── dependency-review.yml
│ └── scorecards.yml
├── .gitignore
├── .golangci.yml
├── .goreleaser.yml
├── .pipelines/
│ ├── nightly.yml
│ ├── pr.yml
│ └── templates/
│ ├── cleanup-template.yml
│ ├── cluster-health-template.yml
│ ├── e2e-kind-template.yml
│ ├── e2e-upgrade-template.yml
│ ├── kind-debug-template.yml
│ ├── manifest-template.yml
│ ├── prepare-deps.yaml
│ ├── scan-images-template.yml
│ └── unit-tests-template.yml
├── AUTHORS
├── CODEOWNERS
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── SECURITY.md
├── cmd/
│ └── server/
│ └── main.go
├── developers.md
├── docs/
│ ├── manual-install.md
│ ├── metrics.md
│ ├── rotation.md
│ └── testing.md
├── go.mod
├── go.sum
├── pkg/
│ ├── auth/
│ │ ├── auth.go
│ │ └── auth_test.go
│ ├── config/
│ │ └── azure_config.go
│ ├── consts/
│ │ └── consts.go
│ ├── metrics/
│ │ ├── exporter.go
│ │ ├── exporter_test.go
│ │ ├── prometheus_exporter.go
│ │ └── stats_reporter.go
│ ├── plugin/
│ │ ├── healthz.go
│ │ ├── healthz_test.go
│ │ ├── keyvault.go
│ │ ├── keyvault_test.go
│ │ ├── kms_v2_server.go
│ │ ├── kms_v2_server_test.go
│ │ ├── mock_keyvault/
│ │ │ └── keyvault_mock.go
│ │ ├── server.go
│ │ └── server_test.go
│ ├── utils/
│ │ ├── grpc.go
│ │ ├── grpc_test.go
│ │ ├── sanitize.go
│ │ └── sanitize_test.go
│ └── version/
│ ├── version.go
│ └── version_test.go
├── scripts/
│ ├── connect-registry.sh
│ ├── setup-kind-cluster.sh
│ ├── setup-kmsv2-kind-cluster.sh
│ └── setup-local-registry.sh
├── tests/
│ ├── client/
│ │ └── client_test.go
│ └── e2e/
│ ├── azure.json
│ ├── encryption-config.yaml
│ ├── helpers.bash
│ ├── kind-config.yaml
│ ├── kms.yaml
│ ├── kmsv2-encryption-config.yaml
│ ├── test.bats
│ └── testkmsv2.bats
└── tools/
├── go.mod
├── go.sum
└── tools.go
SYMBOL INDEX (121 symbols across 25 files)
FILE: cmd/server/main.go
function main (line 58) | func main() {
function setupKMSPlugin (line 64) | func setupKMSPlugin() error {
function withShutdownSignal (line 191) | func withShutdownSignal(ctx context.Context) context.Context {
FILE: pkg/auth/auth.go
function GetKeyvaultToken (line 27) | func GetKeyvaultToken(config *config.AzureConfig, env *azure.Environment...
function GetServicePrincipalToken (line 37) | func GetServicePrincipalToken(config *config.AzureConfig, aadEndpoint, r...
function ParseAzureEnvironment (line 110) | func ParseAzureEnvironment(cloudName string) (*azure.Environment, error) {
function decodePkcs12 (line 123) | func decodePkcs12(pkcs []byte, password string) (*x509.Certificate, *rsa...
function redactClientCredentials (line 137) | func redactClientCredentials(sensitiveString string) string {
function addTargetTypeHeader (line 143) | func addTargetTypeHeader(spt *adal.ServicePrincipalToken) *adal.ServiceP...
FILE: pkg/auth/auth_test.go
function TestParseAzureEnvironment (line 19) | func TestParseAzureEnvironment(t *testing.T) {
function TestRedactClientCredentials (line 40) | func TestRedactClientCredentials(t *testing.T) {
function TestGetServicePrincipalTokenFromMSIWithUserAssignedID (line 63) | func TestGetServicePrincipalTokenFromMSIWithUserAssignedID(t *testing.T) {
function TestGetServicePrincipalTokenFromMSI (line 113) | func TestGetServicePrincipalTokenFromMSI(t *testing.T) {
function TestGetServicePrincipalToken (line 161) | func TestGetServicePrincipalToken(t *testing.T) {
FILE: pkg/config/azure_config.go
type AzureConfig (line 12) | type AzureConfig struct
function GetAzureConfig (line 24) | func GetAzureConfig(configFile string) (config *AzureConfig, err error) {
FILE: pkg/consts/consts.go
constant RequestHeaderTargetType (line 13) | RequestHeaderTargetType = "x-azure-proxy-target"
constant TargetTypeAzureActiveDirectory (line 14) | TargetTypeAzureActiveDirectory = "AzureActiveDirectory"
constant TargetTypeKeyVault (line 15) | TargetTypeKeyVault = "KeyVault"
FILE: pkg/metrics/exporter.go
constant prometheusExporter (line 11) | prometheusExporter = "prometheus"
function InitMetricsExporter (line 15) | func InitMetricsExporter(metricsBackend, metricsAddress string) error {
FILE: pkg/metrics/exporter_test.go
function TestInitMetricsExporter (line 8) | func TestInitMetricsExporter(t *testing.T) {
FILE: pkg/metrics/prometheus_exporter.go
constant metricsEndpoint (line 17) | metricsEndpoint = "metrics"
function initPrometheusExporter (line 20) | func initPrometheusExporter(metricsAddress string) error {
FILE: pkg/metrics/stats_reporter.go
constant instrumentationName (line 12) | instrumentationName = "keyvaultkms"
constant errorMessageKey (line 13) | errorMessageKey = "error_message"
constant statusTypeKey (line 14) | statusTypeKey = "status"
constant operationTypeKey (line 15) | operationTypeKey = "operation"
constant kmsRequestMetricName (line 16) | kmsRequestMetricName = "kms_request"
constant ErrorStatusTypeValue (line 18) | ErrorStatusTypeValue = "error"
constant SuccessStatusTypeValue (line 20) | SuccessStatusTypeValue = "success"
constant EncryptOperationTypeValue (line 22) | EncryptOperationTypeValue = "encrypt"
constant DecryptOperationTypeValue (line 24) | DecryptOperationTypeValue = "decrypt"
constant GrpcOperationTypeValue (line 26) | GrpcOperationTypeValue = "grpc"
type reporter (line 29) | type reporter struct
method ReportRequest (line 55) | func (r *reporter) ReportRequest(ctx context.Context, operationType, s...
type StatsReporter (line 34) | type StatsReporter interface
function NewStatsReporter (line 39) | func NewStatsReporter() (StatsReporter, error) {
FILE: pkg/plugin/healthz.go
constant healthCheckPlainText (line 26) | healthCheckPlainText = "healthcheck"
type HealthZ (line 30) | type HealthZ struct
method Serve (line 39) | func (h *HealthZ) Serve() {
method ServeHTTP (line 52) | func (h *HealthZ) ServeHTTP(w http.ResponseWriter, _ *http.Request) {
method checkRPC (line 141) | func (h *HealthZ) checkRPC(
method dialUnixSocket (line 169) | func (h *HealthZ) dialUnixSocket() (*grpc.ClientConn, error) {
FILE: pkg/plugin/healthz_test.go
function TestServe (line 30) | func TestServe(t *testing.T) {
function TestCheckRPC (line 108) | func TestCheckRPC(t *testing.T) {
function getTempTestDir (line 139) | func getTempTestDir(t *testing.T) string {
function setupFakeKMSServer (line 147) | func setupFakeKMSServer(socketPath string) (
function doHealthCheck (line 189) | func doHealthCheck(t *testing.T, url string) (int, []byte) {
FILE: pkg/plugin/keyvault.go
constant dateAnnotationKey (line 36) | dateAnnotationKey = "date.azure.akv.io"
constant requestIDAnnotationKey (line 37) | requestIDAnnotationKey = "x-ms-request-id.azure.akv.io"
constant keyvaultRegionAnnotationKey (line 38) | keyvaultRegionAnnotationKey = "x-ms-keyvault-region.azure.akv.io"
constant versionAnnotationKey (line 39) | versionAnnotationKey = "version.azure.akv.io"
constant algorithmAnnotationKey (line 40) | algorithmAnnotationKey = "algorithm.azure.akv.io"
constant dateAnnotationValue (line 41) | dateAnnotationValue = "Date"
constant requestIDAnnotationValue (line 42) | requestIDAnnotationValue = "X-Ms-Request-Id"
constant keyvaultRegionAnnotationValue (line 43) | keyvaultRegionAnnotationValue = "X-Ms-Keyvault-Region"
type Client (line 47) | type Client interface
type KeyVaultClient (line 66) | type KeyVaultClient struct
method Encrypt (line 150) | func (kvc *KeyVaultClient) Encrypt(
method Decrypt (line 190) | func (kvc *KeyVaultClient) Decrypt(
method GetUserAgent (line 223) | func (kvc *KeyVaultClient) GetUserAgent() string {
method GetVaultURL (line 227) | func (kvc *KeyVaultClient) GetVaultURL() string {
method validateAnnotations (line 235) | func (kvc *KeyVaultClient) validateAnnotations(
function NewKeyVaultClient (line 78) | func NewKeyVaultClient(
function getVaultURL (line 273) | func getVaultURL(vaultName string, managedHSM bool, env *azure.Environme...
function getProxiedVaultURL (line 294) | func getProxiedVaultURL(vaultURL *string, proxyAddress string, proxyPort...
function getVaultDNSSuffix (line 299) | func getVaultDNSSuffix(managedHSM bool, env *azure.Environment) string {
function getVaultResourceIdentifier (line 306) | func getVaultResourceIdentifier(managedHSM bool, env *azure.Environment)...
function getKeyIDHash (line 313) | func getKeyIDHash(vaultURL, keyName, keyVersion string) (string, error) {
FILE: pkg/plugin/keyvault_test.go
function TestNewKeyVaultClientError (line 22) | func TestNewKeyVaultClientError(t *testing.T) {
function TestNewKeyVaultClient (line 78) | func TestNewKeyVaultClient(t *testing.T) {
function TestGetVaultURLError (line 151) | func TestGetVaultURLError(t *testing.T) {
function TestGetVaultURL (line 186) | func TestGetVaultURL(t *testing.T) {
function TestGetKeyIDHash (line 207) | func TestGetKeyIDHash(t *testing.T) {
FILE: pkg/plugin/kms_v2_server.go
type KeyManagementServiceV2Server (line 22) | type KeyManagementServiceV2Server struct
method Status (line 44) | func (s *KeyManagementServiceV2Server) Status(ctx context.Context, _ *...
method Encrypt (line 81) | func (s *KeyManagementServiceV2Server) Encrypt(ctx context.Context, re...
method Decrypt (line 112) | func (s *KeyManagementServiceV2Server) Decrypt(ctx context.Context, re...
function NewKMSv2Server (line 30) | func NewKMSv2Server(kvClient Client) (*KeyManagementServiceV2Server, err...
FILE: pkg/plugin/kms_v2_server_test.go
function TestV2Encrypt (line 23) | func TestV2Encrypt(t *testing.T) {
function TestV2Decrypt (line 81) | func TestV2Decrypt(t *testing.T) {
function TestStatus (line 180) | func TestStatus(t *testing.T) {
FILE: pkg/plugin/mock_keyvault/keyvault_mock.go
type KeyVaultClient (line 17) | type KeyVaultClient struct
method Encrypt (line 28) | func (kvc *KeyVaultClient) Encrypt(_ context.Context, _ []byte, _ keyv...
method Decrypt (line 42) | func (kvc *KeyVaultClient) Decrypt(_ context.Context, _ []byte, _ keyv...
method SetEncryptResponse (line 48) | func (kvc *KeyVaultClient) SetEncryptResponse(encryptOut []byte, err e...
method SetDecryptResponse (line 55) | func (kvc *KeyVaultClient) SetDecryptResponse(decryptOut []byte, err e...
method ValidateAnnotations (line 62) | func (kvc *KeyVaultClient) ValidateAnnotations(annotations map[string]...
method GetUserAgent (line 93) | func (kvc *KeyVaultClient) GetUserAgent() string {
method GetVaultURL (line 97) | func (kvc *KeyVaultClient) GetVaultURL() string {
FILE: pkg/plugin/server.go
type KeyManagementServiceServer (line 22) | type KeyManagementServiceServer struct
method Version (line 56) | func (s *KeyManagementServiceServer) Version(_ context.Context, _ *kms...
method Encrypt (line 65) | func (s *KeyManagementServiceServer) Encrypt(ctx context.Context, requ...
method Decrypt (line 92) | func (s *KeyManagementServiceServer) Decrypt(ctx context.Context, requ...
type Config (line 30) | type Config struct
function NewKMSv1Server (line 42) | func NewKMSv1Server(kvClient Client) (*KeyManagementServiceServer, error) {
FILE: pkg/plugin/server_test.go
function TestEncrypt (line 22) | func TestEncrypt(t *testing.T) {
function TestDecrypt (line 71) | func TestDecrypt(t *testing.T) {
function TestVersion (line 120) | func TestVersion(t *testing.T) {
FILE: pkg/utils/grpc.go
function ParseEndpoint (line 16) | func ParseEndpoint(ep string) (string, string, error) {
function UnaryServerInterceptor (line 27) | func UnaryServerInterceptor(ctx context.Context, req interface{}, info *...
function getGRPCMethodName (line 53) | func getGRPCMethodName(fullMethodName string) string {
FILE: pkg/utils/grpc_test.go
function TestParseEndpoint (line 5) | func TestParseEndpoint(t *testing.T) {
function TestGetGRPCMethodName (line 50) | func TestGetGRPCMethodName(t *testing.T) {
FILE: pkg/utils/sanitize.go
function SanitizeString (line 6) | func SanitizeString(s string) string {
FILE: pkg/utils/sanitize_test.go
function TestSanitizeString (line 5) | func TestSanitizeString(t *testing.T) {
FILE: pkg/version/version.go
function PrintVersion (line 25) | func PrintVersion() (err error) {
function GetUserAgent (line 46) | func GetUserAgent() string {
FILE: pkg/version/version_test.go
function TestPrintVersion (line 13) | func TestPrintVersion(t *testing.T) {
function TestGetUserAgent (line 46) | func TestGetUserAgent(t *testing.T) {
FILE: tests/client/client_test.go
constant netProtocol (line 18) | netProtocol = "unix"
constant pathToUnixSocket (line 19) | pathToUnixSocket = "/opt/azurekms.sock"
constant version (line 20) | version = "v1beta1"
function setupTestCase (line 31) | func setupTestCase() {
function teardownTestCase (line 44) | func teardownTestCase() {
function TestEncryptDecrypt (line 51) | func TestEncryptDecrypt(t *testing.T) {
function TestV1Version (line 113) | func TestV1Version(t *testing.T) {
function TestV2Version (line 139) | func TestV2Version(t *testing.T) {
function TestMain (line 165) | func TestMain(m *testing.M) {
function newUnixSocketConnection (line 172) | func newUnixSocketConnection(path string) (*grpc.ClientConn, error) {
Condensed preview — 79 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (325K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 281,
"preview": "---\nname: Bug report\nabout: Create a report to help KMS Plugin for Key Vault improve\ntitle: ''\nlabels: bug\nassignees: ''"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"chars": 311,
"preview": "---\nname: Feature request\nabout: Suggest an idea for KMS Plugin for Key Vault\ntitle: ''\nlabels: enhancement\nassignees: '"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 316,
"preview": "<!-- Thank you for helping KMS Plugin for Key Vault with a pull request! -->\n\n**Reason for Change**:\n<!-- What does this"
},
{
"path": ".github/dependabot.yml",
"chars": 696,
"preview": "version: 2\nupdates:\n - package-ecosystem: \"gomod\"\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n commit-"
},
{
"path": ".github/semantic.yml",
"chars": 133,
"preview": "titleOnly: true\ntypes:\n - chore\n - ci\n - docs\n - feat\n - fix\n - perf\n - refactor\n - release\n - revert\n - secur"
},
{
"path": ".github/workflows/codeql.yaml",
"chars": 962,
"preview": "name: \"CodeQL\"\n\non:\n push:\n branches:\n - master\n pull_request:\n branches:\n - master\n schedule:\n - cron"
},
{
"path": ".github/workflows/create-release.yml",
"chars": 757,
"preview": "name: create_release\non:\n push:\n tags:\n - 'v*'\n\npermissions:\n contents: write\n\njobs:\n create-release:\n run"
},
{
"path": ".github/workflows/dependency-review.yml",
"chars": 970,
"preview": "# Dependency Review Action\n#\n# This Action will scan dependency manifest files that change as part of a Pull Request,\n# "
},
{
"path": ".github/workflows/scorecards.yml",
"chars": 3061,
"preview": "# This workflow uses actions that are not certified by GitHub. They are provided\n# by a third-party and are governed by "
},
{
"path": ".gitignore",
"chars": 443,
"preview": "# Binaries for programs and plugins\n*.exe\n*.dll\n*.so\n*.dylib\n\n# Test binary, build with `go test -c`\n*.test\n\n# Output of"
},
{
"path": ".golangci.yml",
"chars": 869,
"preview": "version: \"2\"\nrun:\n go: \"1.26\"\nlinters:\n default: none\n enable:\n - errorlint\n - goconst\n - gocyclo\n - gose"
},
{
"path": ".goreleaser.yml",
"chars": 977,
"preview": "# refer to https://goreleaser.com for more options\nversion: 2\nbuilds:\n- skip: true\nrelease:\n prerelease: auto\n header:"
},
{
"path": ".pipelines/nightly.yml",
"chars": 291,
"preview": "trigger: none\n\nschedules:\n - cron: \"0 0 * * *\"\n always: true\n displayName: \"Nightly Build & Test\"\n branches:\n "
},
{
"path": ".pipelines/pr.yml",
"chars": 302,
"preview": "trigger:\n branches:\n include:\n - master\n\npr:\n branches:\n include:\n - master\n paths:\n exclude:\n "
},
{
"path": ".pipelines/templates/cleanup-template.yml",
"chars": 231,
"preview": "steps:\n - script: |\n kubectl logs -l component=azure-kms-provider -n kube-system --tail -1\n kubectl get pods "
},
{
"path": ".pipelines/templates/cluster-health-template.yml",
"chars": 210,
"preview": "steps:\n - script: |\n kubectl wait --for=condition=ready node --all\n kubectl wait pod -n kube-system --for=con"
},
{
"path": ".pipelines/templates/e2e-kind-template.yml",
"chars": 3102,
"preview": "jobs:\n - job:\n timeoutInMinutes: 15\n cancelTimeoutInMinutes: 5\n workspace:\n clean: all\n variables:\n "
},
{
"path": ".pipelines/templates/e2e-upgrade-template.yml",
"chars": 3875,
"preview": "jobs:\n - job: e2e_upgrade_tests\n timeoutInMinutes: 10\n cancelTimeoutInMinutes: 5\n workspace:\n clean: all\n"
},
{
"path": ".pipelines/templates/kind-debug-template.yml",
"chars": 558,
"preview": "steps:\n - script: |\n docker exec kms-control-plane bash -c \"cat /etc/kubernetes/manifests/kubernetes-kms.yaml\"\n "
},
{
"path": ".pipelines/templates/manifest-template.yml",
"chars": 545,
"preview": "parameters:\n - name: registry\n type: string\n - name: imageName\n type: string\n - name: imageVersion\n type: st"
},
{
"path": ".pipelines/templates/prepare-deps.yaml",
"chars": 233,
"preview": "steps:\n- bash: |\n for i in {1..10}; do\n if sudo tdnf install -y kernel-headers make gcc glibc-devel binutils get"
},
{
"path": ".pipelines/templates/scan-images-template.yml",
"chars": 662,
"preview": "steps:\n - script: |\n export REGISTRY=\"e2e\"\n export IMAGE_VERSION=\"test\"\n export OUTPUT_TYPE=\"type=docker"
},
{
"path": ".pipelines/templates/unit-tests-template.yml",
"chars": 1578,
"preview": "jobs:\n - job: unit_tests\n timeoutInMinutes: 10\n cancelTimeoutInMinutes: 5\n workspace:\n clean: all\n var"
},
{
"path": "AUTHORS",
"chars": 36,
"preview": "Rita Zhang <rita.z.zhang@gmail.com>\n"
},
{
"path": "CODEOWNERS",
"chars": 158,
"preview": "# Ref: https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/abo"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 5260,
"preview": "# Microsoft Open Source Code of Conduct\n\nThis code of conduct outlines expectations for participation in Microsoft-manag"
},
{
"path": "CONTRIBUTING.md",
"chars": 581,
"preview": "# Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContr"
},
{
"path": "Dockerfile",
"chars": 1194,
"preview": "FROM mcr.microsoft.com/oss/go/microsoft/golang:1.26.2-bookworm@sha256:61e607875d60ae21a7a4a49110fe7098355473fbc74ab13091"
},
{
"path": "LICENSE",
"chars": 1162,
"preview": " MIT License\n\n Copyright (c) Microsoft Corporation. All rights reserved.\n\n Permission is hereby granted, free o"
},
{
"path": "Makefile",
"chars": 5099,
"preview": "ORG_PATH=github.com/Azure\nPROJECT_NAME := kubernetes-kms\nREPO_PATH=\"$(ORG_PATH)/$(PROJECT_NAME)\"\n\nREGISTRY_NAME ?= upstr"
},
{
"path": "README.md",
"chars": 5463,
"preview": "# KMS Plugin for Key Vault\n\n[![Build Status](https://dev.azure.com/AzureContainerUpstream/Kubernetes%20KMS/_apis/build/s"
},
{
"path": "SECURITY.md",
"chars": 2757,
"preview": "<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK -->\n\n## Security\n\nMicrosoft takes the security of our software products an"
},
{
"path": "cmd/server/main.go",
"chars": 6411,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "developers.md",
"chars": 3348,
"preview": "# Developers Guide\n\nThis guide explains how to set up your environment for developing the Azure kubernetes kms service.\n"
},
{
"path": "docs/manual-install.md",
"chars": 7873,
"preview": "# 🛠 Manual Configurations #\n\nThis guide demonstrates steps required to enable the KMS Plugin for Key Vault in an existin"
},
{
"path": "docs/metrics.md",
"chars": 10518,
"preview": "# Metrics provided by KMS plugin for Key Vault\n\nThis project uses [opentelemetry](https://opentelemetry.io/) for reporti"
},
{
"path": "docs/rotation.md",
"chars": 8275,
"preview": "# Rotating KMS key\n\nThis guide demonstrates steps required to update your cluster to use a new KMS key for encryption.\n\n"
},
{
"path": "docs/testing.md",
"chars": 2272,
"preview": "# End-to-end testing for KMS Plugin for Keyvault\n\n## Prerequisites\n\nTo run tests locally, following components are requi"
},
{
"path": "go.mod",
"chars": 3111,
"preview": "module github.com/Azure/kubernetes-kms\n\ngo 1.26.2\n\nrequire (\n\tgithub.com/Azure/azure-sdk-for-go v68.0.0+incompatible\n\tgi"
},
{
"path": "go.sum",
"chars": 22504,
"preview": "github.com/Azure/azure-sdk-for-go v68.0.0+incompatible h1:fcYLmCpyNYRnvJbPerq7U0hS+6+I79yEDJBqVNcqUzU=\ngithub.com/Azure/"
},
{
"path": "pkg/auth/auth.go",
"chars": 5399,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/auth/auth_test.go",
"chars": 6104,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/config/azure_config.go",
"chars": 1438,
"preview": "package config\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"gopkg.in/yaml.v3\"\n\t\"monis.app/mlog\"\n)\n\n// AzureConfig is representing /etc/kube"
},
{
"path": "pkg/consts/consts.go",
"chars": 680,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/metrics/exporter.go",
"chars": 565,
"preview": "package metrics\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"monis.app/mlog\"\n)\n\nconst (\n\tprometheusExporter = \"prometheus\"\n)\n\n// InitM"
},
{
"path": "pkg/metrics/exporter_test.go",
"chars": 1065,
"preview": "package metrics\n\nimport (\n\t\"net/http\"\n\t\"testing\"\n)\n\nfunc TestInitMetricsExporter(t *testing.T) {\n\ttestCases := []struct "
},
{
"path": "pkg/metrics/prometheus_exporter.go",
"chars": 1430,
"preview": "package metrics\n\nimport (\n\t\"fmt\"\n\t\"net/http\"\n\t\"time\"\n\n\tpromclient \"github.com/prometheus/client_golang/prometheus\"\n\t\"git"
},
{
"path": "pkg/metrics/stats_reporter.go",
"chars": 1955,
"preview": "package metrics\n\nimport (\n\t\"context\"\n\n\t\"go.opentelemetry.io/otel\"\n\t\"go.opentelemetry.io/otel/attribute\"\n\t\"go.opentelemet"
},
{
"path": "pkg/plugin/healthz.go",
"chars": 5154,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/healthz_test.go",
"chars": 5598,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/keyvault.go",
"chars": 10411,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/keyvault_test.go",
"chars": 8499,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/kms_v2_server.go",
"chars": 4456,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/kms_v2_server_test.go",
"chars": 5646,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/mock_keyvault/keyvault_mock.go",
"chars": 2725,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/server.go",
"chars": 3361,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/plugin/server_test.go",
"chars": 3302,
"preview": "// Copyright (c) Microsoft and contributors. All rights reserved.\n//\n// This source code is licensed under the MIT lice"
},
{
"path": "pkg/utils/grpc.go",
"chars": 1606,
"preview": "package utils\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/Azure/kubernetes-kms/pkg/metrics\"\n\n\t\"google.g"
},
{
"path": "pkg/utils/grpc_test.go",
"chars": 1868,
"preview": "package utils\n\nimport \"testing\"\n\nfunc TestParseEndpoint(t *testing.T) {\n\ttests := []struct {\n\t\tdesc string\n\t\ten"
},
{
"path": "pkg/utils/sanitize.go",
"chars": 228,
"preview": "package utils\n\nimport \"strings\"\n\n// SanitizeString returns a string that does not have white spaces and double quotes.\nf"
},
{
"path": "pkg/utils/sanitize_test.go",
"chars": 948,
"preview": "package utils\n\nimport \"testing\"\n\nfunc TestSanitizeString(t *testing.T) {\n\ttestCases := []struct {\n\t\tname strin"
},
{
"path": "pkg/version/version.go",
"chars": 1125,
"preview": "package version\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"runtime\"\n)\n\nvar (\n\t// BuildDate is the date when the binary was buil"
},
{
"path": "pkg/version/version_test.go",
"chars": 1273,
"preview": "package version\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"runtime\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestPrintVersion(t *testi"
},
{
"path": "scripts/connect-registry.sh",
"chars": 707,
"preview": "#!/usr/bin/env bash\n\nset -o errexit\nset -o nounset\nset -o pipefail\n\nif [ \"${KIND_NETWORK}\" != \"bridge\" ]; then\n # wait "
},
{
"path": "scripts/setup-kind-cluster.sh",
"chars": 524,
"preview": "#!/usr/bin/env bash\n\nset -o errexit\nset -o nounset\nset -o pipefail\n\nexport ENCRYPTION_CONFIG_FILE=encryption-config.yaml"
},
{
"path": "scripts/setup-kmsv2-kind-cluster.sh",
"chars": 534,
"preview": "#!/usr/bin/env bash\n\nset -o errexit\nset -o nounset\nset -o pipefail\n\nexport ENCRYPTION_CONFIG_FILE=kmsv2-encryption-confi"
},
{
"path": "scripts/setup-local-registry.sh",
"chars": 1224,
"preview": "#!/usr/bin/env bash\n\nset -o errexit\nset -o nounset\nset -o pipefail\n\n# create registry container unless it already exists"
},
{
"path": "tests/client/client_test.go",
"chars": 4652,
"preview": "package test\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"fmt\"\n\t\"testing\"\n\t\"time\"\n\n\t\"google.golang.org/grpc\"\n\t\"google.golang.org/grpc"
},
{
"path": "tests/e2e/azure.json",
"chars": 174,
"preview": "{\n \"cloud\": \"AzurePublicCloud\",\n \"tenantId\": \"$AZURE_TENANT_ID\",\n \"useManagedIdentityExtension\": true,\n \"use"
},
{
"path": "tests/e2e/encryption-config.yaml",
"chars": 267,
"preview": "kind: EncryptionConfiguration\napiVersion: apiserver.config.k8s.io/v1\nresources:\n - resources:\n - secrets\n provi"
},
{
"path": "tests/e2e/helpers.bash",
"chars": 638,
"preview": "#!/bin/bash\n\nassert_success() {\n if [[ \"$status\" != 0 ]]; then\n echo \"expected: 0\"\n echo \"actual: $status\"\n ec"
},
{
"path": "tests/e2e/kind-config.yaml",
"chars": 1320,
"preview": "kind: Cluster\napiVersion: kind.x-k8s.io/v1alpha4\ncontainerdConfigPatches:\n- |-\n [plugins.\"io.containerd.grpc.v1.cri\".re"
},
{
"path": "tests/e2e/kms.yaml",
"chars": 1640,
"preview": "apiVersion: v1\nkind: Pod\nmetadata:\n name: azure-kms-provider\n namespace: kube-system\n labels:\n tier: control-plane"
},
{
"path": "tests/e2e/kmsv2-encryption-config.yaml",
"chars": 245,
"preview": "kind: EncryptionConfiguration\napiVersion: apiserver.config.k8s.io/v1\nresources:\n - resources:\n - secrets\n provi"
},
{
"path": "tests/e2e/test.bats",
"chars": 2585,
"preview": "#!/usr/bin/env bats\n\nload helpers\n\nWAIT_TIME=120\nSLEEP_TIME=1\n\nexport ETCD_CA_CERT=/etc/kubernetes/pki/etcd/ca.crt\nexpor"
},
{
"path": "tests/e2e/testkmsv2.bats",
"chars": 3814,
"preview": "#!/usr/bin/env bats\n\nload helpers\n\nWAIT_TIME=120\nSLEEP_TIME=1\n\nexport ETCD_CA_CERT=/etc/kubernetes/pki/etcd/ca.crt\nexpor"
},
{
"path": "tools/go.mod",
"chars": 10826,
"preview": "module github.com/Azure/kubernetes-kms/tools\n\ngo 1.26.2\n\nrequire github.com/golangci/golangci-lint/v2 v2.7.2\n\nrequire (\n"
},
{
"path": "tools/go.sum",
"chars": 95215,
"preview": "4d63.com/gocheckcompilerdirectives v1.3.0 h1:Ew5y5CtcAAQeTVKUVFrE7EwHMrTO6BggtEj8BZSjZ3A=\n4d63.com/gocheckcompilerdirect"
},
{
"path": "tools/tools.go",
"chars": 120,
"preview": "//go:build tools\n// +build tools\n\npackage tools\n\nimport (\n\t_ \"github.com/golangci/golangci-lint/v2/cmd/golangci-lint\"\n)\n"
}
]
About this extraction
This page contains the full source code of the Azure/kubernetes-kms GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 79 files (299.5 KB), approximately 118.2k tokens, and a symbol index with 121 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.