Showing preview only (778K chars total). Download the full file or copy to clipboard to get everything.
Repository: mysticaltech/terraform-hcloud-kube-hetzner
Branch: master
Commit: 08e59a31197c
Files: 101
Total size: 742.6 KB
Directory structure:
gitextract_6abwv94a/
├── .claude/
│ └── skills/
│ ├── fix-issue/
│ │ └── SKILL.md
│ ├── kh-assistant/
│ │ └── SKILL.md
│ ├── prepare-release/
│ │ └── SKILL.md
│ ├── review-pr/
│ │ └── SKILL.md
│ ├── sync-docs/
│ │ └── SKILL.md
│ ├── test-changes/
│ │ └── SKILL.md
│ └── triage-issue/
│ └── SKILL.md
├── .extra/
│ └── k3s-selinux-next.rpm
├── .github/
│ ├── FUNDING.yml
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.yml
│ │ ├── config.yml
│ │ └── feature_request.yml
│ ├── dependabot.yaml
│ ├── release.yaml
│ ├── release.yml
│ └── workflows/
│ ├── generate-docs.yaml
│ ├── lint_pr.yaml
│ └── publish-release.yaml
├── .gitignore
├── .pre-commit-config.yaml
├── .terraform-docs.yml
├── CHANGELOG.md
├── LICENSE
├── README.md
├── SECURITY.md
├── agents.tf
├── autoscaler-agents.tf
├── control_planes.tf
├── data.tf
├── docs/
│ ├── add-robot-server.md
│ ├── customize-mount-path-longhorn.md
│ ├── llms.md
│ ├── private-network-egress.md
│ ├── ssh.md
│ └── terraform.md
├── examples/
│ ├── kustomization_user_deploy/
│ │ ├── README.md
│ │ ├── helm-chart/
│ │ │ ├── helm-chart.yaml.tpl
│ │ │ ├── kustomization.yaml.tpl
│ │ │ └── namespace.yaml.tpl
│ │ ├── letsencrypt/
│ │ │ ├── kustomization.yaml.tpl
│ │ │ └── letsencrypt.yaml.tpl
│ │ ├── multiple-namespaces/
│ │ │ ├── base/
│ │ │ │ ├── kustomization.yaml.tpl
│ │ │ │ └── pod.yaml.tpl
│ │ │ ├── kustomization.yaml.tpl
│ │ │ ├── namespace-a/
│ │ │ │ ├── kustomization.yaml.tpl
│ │ │ │ └── namespace-a.yaml.tpl
│ │ │ └── namespace-b/
│ │ │ ├── kustomization.yaml.tpl
│ │ │ └── namespace-b.yaml.tpl
│ │ └── simple-resources/
│ │ ├── demo-config-map.yaml.tpl
│ │ ├── demo-pod.yml.tpl
│ │ └── kustomization.yaml.tpl
│ ├── micro_os_rollback/
│ │ └── Readme.md
│ └── tls/
│ ├── ingress.yaml
│ ├── pod.yaml
│ └── service.yaml
├── init.tf
├── kube.tf.example
├── kubeconfig.tf
├── kustomization_backup.tf
├── kustomization_user.tf
├── kustomize/
│ ├── flannel-rbac.yaml
│ └── system-upgrade-controller.yaml
├── locals.tf
├── main.tf
├── modules/
│ ├── host/
│ │ ├── locals.tf
│ │ ├── main.tf
│ │ ├── out.tf
│ │ ├── templates/
│ │ │ └── cloudinit.yaml.tpl
│ │ ├── variables.tf
│ │ └── versions.tf
│ └── values_merger/
│ ├── main.tf
│ └── versions.tf
├── nat-router.tf
├── output.tf
├── packer-template/
│ └── hcloud-microos-snapshots.pkr.hcl
├── placement_groups.tf
├── scripts/
│ ├── cleanup.sh
│ └── create.sh
├── templates/
│ ├── autoscaler-cloudinit.yaml.tpl
│ ├── autoscaler.yaml.tpl
│ ├── calico.yaml.tpl
│ ├── ccm.yaml.tpl
│ ├── cert_manager.yaml.tpl
│ ├── cilium.yaml.tpl
│ ├── csi-driver-smb.yaml.tpl
│ ├── haproxy_ingress.yaml.tpl
│ ├── hcloud-ccm-helm.yaml.tpl
│ ├── hcloud-csi.yaml.tpl
│ ├── kube-hetzner-selinux.te
│ ├── kube_system_secrets.yaml.tpl
│ ├── kured.yaml.tpl
│ ├── longhorn.yaml.tpl
│ ├── nat-router-cloudinit.yaml.tpl
│ ├── nginx_ingress.yaml.tpl
│ ├── plans.yaml.tpl
│ ├── rancher.yaml.tpl
│ └── traefik_ingress.yaml.tpl
├── values-export.tf
├── values-merger.tf
├── variables.tf
└── versions.tf
================================================
FILE CONTENTS
================================================
================================================
FILE: .claude/skills/fix-issue/SKILL.md
================================================
---
name: fix-issue
description: Use when working on a GitHub issue - fetches issue details, analyzes codebase, implements fix following project methodology
args: issue_number
---
# Fix GitHub Issue
## Overview
Guided workflow for implementing fixes for GitHub issues following the project's CLAUDE.md methodology.
## Usage
```
/fix-issue <number>
```
## Workflow
```dot
digraph fix_flow {
rankdir=TB;
node [shape=box];
fetch [label="1. Fetch issue details"];
analyze [label="2. Analyze issue type"];
verify [label="3. Verify it's a real bug"];
investigate [label="4. Deep investigation"];
plan [label="5. Enter plan mode"];
implement [label="6. Implement fix"];
test [label="7. Test changes"];
commit [label="8. Commit & push"];
fetch -> analyze;
analyze -> verify;
verify -> investigate;
investigate -> plan;
plan -> implement;
implement -> test;
test -> commit;
}
```
## Step 1: Fetch Issue Details
```bash
# Get issue details
gh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner
# CRITICAL: Always read ALL comments - solutions may already be proposed
gh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --comments
```
## Step 2: Classify Issue Type
| Type | Description | Action |
|------|-------------|--------|
| 🔴 **BUG** | Reproducible defect | Fix it |
| 🟡 **EDGE CASE** | Fails in specific scenario | Evaluate effort vs impact |
| 🟠 **USER ERROR** | Misconfigured kube.tf | Help user, improve docs |
| ⚪ **OLD VERSION** | Fixed in newer release | Ask user to upgrade |
| 🔵 **FEATURE REQUEST** | New functionality | Move to Discussions |
| ❓ **NEEDS INFO** | Can't reproduce | Ask for more info |
### User Error Indicators
- kube.tf has obvious mistakes
- Error indicates syntax/config issue
- Using deprecated variable names
- Mixing incompatible options
- Missing required variables
### Actual Bug Indicators
- Reproducible with correct config
- Multiple users report same issue
- Error in module code, not user config
- Works in previous version, broke in update
## Step 3: Verify Before Fixing
**CRITICAL: Many issues are user configuration errors, NOT bugs.**
Before implementing any fix:
1. Check if the user's kube.tf is correct
2. Verify the issue exists in the latest version
3. Try to reproduce the issue locally
4. Check if there's already a PR addressing this
```bash
# Search for existing PRs
gh pr list --search "<error keyword>" --repo kube-hetzner/terraform-hcloud-kube-hetzner
# Check if issue is already mentioned in changelog
grep -i "<keyword>" CHANGELOG.md
```
## Step 4: Deep Investigation
Read these files to understand context:
```bash
# Always start with these
cat versions.tf # Provider/terraform versions
cat variables.tf # All configurable options
cat locals.tf # Core logic and computed values
# Then investigate specific areas based on the issue
```
### Key Files by Area
| Area | Files to Check |
|------|---------------|
| Network | `locals.tf` (subnet calculations), `network.tf` |
| Control Plane | `control_planes.tf`, `locals.tf` |
| Agents | `agents.tf`, `autoscaler.tf` |
| Load Balancer | `load_balancer.tf`, `init.tf` |
| CNI | `templates/cni/*.yaml.tpl` |
| Storage | `templates/longhorn.yaml.tpl` |
| Firewall | `firewall.tf` |
### For Complex Issues - Use AI Tools
```bash
# Codex CLI for deep reasoning
codex exec -m gpt-5.2-codex -s read-only -c model_reasoning_effort="xhigh" \
"Analyze this issue and identify root cause: <issue description>"
# Gemini for large context analysis
gemini --model gemini-3-pro-preview -p \
"@locals.tf @variables.tf Analyze how <feature> works and potential issues"
```
## Step 5: Enter Plan Mode
**MANDATORY: Always enter plan mode before implementing.**
Write a plan that includes:
- [ ] Issue number and title
- [ ] Root cause analysis
- [ ] Exact files to modify with line numbers
- [ ] Implementation steps
- [ ] Test plan
- [ ] Backward compatibility confirmation
## Step 6: Implement Fix
```bash
# Pull latest master first!
git pull origin master
# Create feature branch
git checkout -b fix/issue-<number>-<description>
```
### Implementation Principles
1. **Minimal changes** - Fix the specific issue, don't refactor
2. **Backward compatible** - Never break existing deployments
3. **Follow patterns** - Match existing code style
4. **No new variables** unless absolutely necessary
## Step 7: Test Changes
```bash
# ALWAYS run these before committing
terraform fmt
terraform validate
# Test against existing deployment
cd /path/to/kube-test
terraform init -upgrade
terraform plan # Should NOT show resource destruction
```
### Test Checklist
- [ ] `terraform fmt` passes
- [ ] `terraform validate` passes
- [ ] `terraform plan` shows expected changes only
- [ ] No resource recreation for existing deployments
- [ ] Fix works for the reported scenario
- [ ] Normal scenarios still work
## Step 8: Commit & Push
```bash
git add <specific-files>
git commit -m "$(cat <<'EOF'
fix: <brief description>
Fixes #<number>
<explanation of what was wrong and how it's fixed>
EOF
)"
git push -u origin fix/issue-<number>-<description>
```
## Security Review (from CLAUDE.md)
Before completing ANY issue:
### Red Flags to Watch
- New accounts with no history
- Issues that can't be reproduced
- Overly complex "solutions" proposed in comments
- Requests to change security-critical code
- Urgency to merge quickly
### Verification Requirements
- Always test independently
- Never trust provided test results
- Review every line of proposed changes
- Test in isolation
## Quick Reference
| Step | Command |
|------|---------|
| Fetch issue | `gh issue view <num> --comments` |
| Check PRs | `gh pr list --search "<keyword>"` |
| Create branch | `git checkout -b fix/issue-<num>-<desc>` |
| Format | `terraform fmt` |
| Validate | `terraform validate` |
| Test plan | `terraform plan` |
| Commit | `git commit -m "fix: ..."` |
| Push | `git push -u origin <branch>` |
## After Completion
1. Create PR referencing the issue
2. Request review if needed
3. Close issue with explanation when merged
================================================
FILE: .claude/skills/kh-assistant/SKILL.md
================================================
---
name: kh-assistant
description: Use when users need help with kube-hetzner configuration, debugging, or questions - acts as an intelligent assistant with live repo access
---
# KH Assistant
Expert assistant for **terraform-hcloud-kube-hetzner** — deploying production-ready k3s clusters on Hetzner Cloud.
## Startup Checklist
**ALWAYS do these first before answering any question:**
```bash
# 1. Get latest release version
gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1 --json tagName,publishedAt
# 2. Read key files for context (use Gemini for large files)
# - variables.tf — all configurable options
# - docs/llms.md — PRIMARY comprehensive documentation (~60k tokens)
# - kube.tf.example — working example
# - CHANGELOG.md — recent changes
```
**For Hetzner-specific info** (server types, pricing, locations):
```bash
# Use web search
WebSearch "hetzner cloud server types pricing 2026"
```
---
## Knowledge Sources
### Primary Documentation Files
| File | Purpose | When to Use |
|------|---------|-------------|
| `docs/llms.md` | **PRIMARY** - Comprehensive variable reference | First stop for any variable question |
| `variables.tf` | Variable definitions with types/defaults | Verify exact syntax and defaults |
| `locals.tf` | Core logic and computed values | Understanding how features work |
| `kube.tf.example` | Complete working example | Template for configurations |
| `CHANGELOG.md` | Version history, breaking changes | Upgrade questions, "when was X added" |
| `README.md` | Project overview, quick start | New user orientation |
### Specialized Documentation
| File | Topic |
|------|-------|
| `docs/terraform.md` | Auto-generated terraform docs |
| `docs/ssh.md` | SSH configuration, key formats |
| `docs/add-robot-server.md` | Hetzner dedicated server integration |
| `docs/private-network-egress.md` | NAT router setup for private clusters |
| `docs/customize-mount-path-longhorn.md` | Longhorn storage customization |
### GitHub (Live Data)
```bash
# Latest release
gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1
# Search issues for errors
gh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --search "<error>" --state all
# Search discussions for how-to
gh api repos/kube-hetzner/terraform-hcloud-kube-hetzner/discussions --jq '.[].title'
# Check if variable exists
grep 'variable "<name>"' variables.tf
```
---
## Critical Rules
### MUST Follow — Never Violate
| Rule | Explanation |
|------|-------------|
| **At least 1 control plane** | `control_plane_nodepools` must have at least one entry with `count >= 1` |
| **MicroOS ONLY** | Never suggest Ubuntu, Debian, or any other OS |
| **Network region coverage** | `network_region` must contain ALL node locations |
| **Odd control plane counts for HA** | Use 1, 3, or 5 — never 2 or 4 (quorum requirement) |
| **Autoscaler is separate** | `autoscaler_nodepools` is independent from `agent_nodepools` |
| **Latest version always** | Always fetch and use the latest release tag |
### Common Mistakes to Prevent
| Mistake | Correct |
|---------|---------|
| Empty control_plane_nodepools | At least one with count >= 1 |
| 2 control planes for "HA" | Use 3 (odd number for quorum) |
| Suggesting Ubuntu | MicroOS only |
| Location not in network_region | network_region must cover all locations |
| Confusing autoscaler with agents | Autoscaler pools are completely separate |
| Using old version | Always check latest release first |
---
## Common Issues Catalog
### Known Error Patterns
| Error | Cause | Solution |
|-------|-------|----------|
| `cannot sum empty list` | control_plane_nodepools is empty or all counts are 0 | Add at least one control plane with count >= 1 |
| `NAT router primary IPs will be replaced` | Pre-v2.19.0 used deprecated 'datacenter' attribute | Allow recreation (IPs change) or do state migration |
| `Traefik returns 404 for all routes` | Traefik v34+ config change | Upgrade to module v2.19.0+ |
| `SSH connection refused or timeout` | Key format, firewall, or node not ready | Check ssh_public_key format, verify firewall_ssh_source |
| `Node stuck in NotReady` | Network region mismatch or token issues | Ensure network_region contains all node locations |
| `Error creating network subnet` | Subnet CIDR conflicts | Check network_ipv4_cidr doesn't overlap with existing |
| `cloud-init failed` | MicroOS snapshot missing or wrong region | Recreate snapshot with packer in correct region |
### Debugging Workflow
```
1. Check Common Issues table above
2. Search GitHub issues: gh issue list --search "<error>" --state all
3. Search docs/llms.md for related variables
4. Check locals.tf for the logic
5. Provide: Root cause → Fix → Prevention
6. Link to relevant GitHub issues if found
```
---
## Hetzner Cloud Context
### Server Types (x86)
| Type | vCPU | RAM | Disk | Best For |
|------|------|-----|------|----------|
| `cpx11` | 2 | 2GB | 40GB | Minimal dev |
| `cpx21` | 3 | 4GB | 80GB | Dev/small workloads |
| `cpx31` | 4 | 8GB | 160GB | Production control plane |
| `cpx41` | 8 | 16GB | 240GB | Production workers |
| `cpx51` | 16 | 32GB | 360GB | Heavy workloads |
### Server Types (ARM — CAX, cost-optimized)
| Type | vCPU | RAM | Disk | Best For |
|------|------|-----|------|----------|
| `cax11` | 2 | 4GB | 40GB | ARM dev |
| `cax21` | 4 | 8GB | 80GB | ARM workloads |
| `cax31` | 8 | 16GB | 160GB | ARM production |
| `cax41` | 16 | 32GB | 320GB | ARM heavy |
### Locations
| Region | Locations | Network Zone |
|--------|-----------|--------------|
| Germany | `fsn1`, `nbg1` | `eu-central` |
| Finland | `hel1` | `eu-central` |
| USA East | `ash` | `us-east` |
| USA West | `hil` | `us-west` |
| Singapore | `sin` | `ap-southeast` |
**Rule**: All locations must be in the same `network_region`.
---
## Configuration Workflows
### Workflow: Creating kube.tf
```
1. FIRST: Get latest release
gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1
2. Ask clarifying questions:
- Use case: Production / Development / Testing?
- HA: Single node / 3 control planes / Super-HA (multi-location)?
- Budget: Which server types?
- Network: Public / Private with NAT router?
- CNI: Flannel (default) / Cilium / Calico?
- Storage: Longhorn needed?
- Ingress: Traefik (default) / Nginx / HAProxy?
3. Query variables.tf and docs/llms.md for relevant options
4. Generate complete config with:
- Module source and version (latest!)
- Required: hetzner_token, ssh keys
- Requested features
- Helpful comments
5. Validate syntax:
terraform fmt
terraform validate
```
### Workflow: Debugging
```
1. Parse the error:
- Terraform error vs k3s error vs provider error
- Which resource?
- What operation?
2. Check Common Issues Catalog (above)
3. Search GitHub:
gh issue list --search "<error keyword>" --state all
4. Read relevant code:
- locals.tf for logic
- variables.tf for options
- Specific .tf files based on error
5. Provide solution:
- Root cause explanation
- Fix (config change or upgrade)
- Prevention steps
- Link to related issues
```
### Workflow: Feature Questions
```
1. Check docs/llms.md FIRST (primary reference)
2. Verify in variables.tf (exact syntax)
3. Check kube.tf.example for usage
4. Search GitHub discussions for examples
5. Provide answer with file references
```
### Workflow: Upgrades
```
1. Get current and target versions
2. Read CHANGELOG.md for breaking changes between versions
3. Check for:
- Removed/renamed variables
- Changed defaults
- Required migrations
4. Generate upgrade steps:
- Update version in kube.tf
- terraform init -upgrade
- terraform plan (check for destructions!)
- terraform apply
5. Warn if terraform plan shows resource recreation
```
---
## Configuration Templates
### Minimal Development (Single Node)
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "<LATEST>" # Always fetch latest!
hetzner_token = var.hetzner_token
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
network_region = "eu-central"
control_plane_nodepools = [
{
name = "control-plane"
server_type = "cpx21"
location = "fsn1"
count = 1
}
]
agent_nodepools = []
# Single node: disable auto OS upgrades
automatically_upgrade_os = false
}
```
### Production HA (3 Control Planes + Workers)
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "<LATEST>"
hetzner_token = var.hetzner_token
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
network_region = "eu-central"
control_plane_nodepools = [
{
name = "control-plane"
server_type = "cpx31"
location = "fsn1"
count = 3 # Odd number for quorum!
}
]
agent_nodepools = [
{
name = "worker"
server_type = "cpx41"
location = "fsn1"
count = 3
}
]
enable_longhorn = true
# Security: restrict access to your IP
firewall_kube_api_source = ["YOUR_IP/32"]
firewall_ssh_source = ["YOUR_IP/32"]
}
```
### Private Cluster with NAT Router
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "<LATEST>"
hetzner_token = var.hetzner_token
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
network_region = "eu-central"
# Enable NAT router for private egress
create_nat_router = true
control_plane_nodepools = [
{
name = "control-plane"
server_type = "cpx31"
location = "fsn1"
count = 3
# Disable public IPs
disable_ipv4 = true
disable_ipv6 = true
}
]
agent_nodepools = [
{
name = "worker"
server_type = "cpx41"
location = "fsn1"
count = 3
disable_ipv4 = true
disable_ipv6 = true
}
]
# Optional: keep control plane LB private too
control_plane_lb_enable_public_interface = false
}
```
### Cilium with Hubble Observability
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "<LATEST>"
hetzner_token = var.hetzner_token
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
network_region = "eu-central"
# Use Cilium CNI
cni_plugin = "cilium"
# Full kube-proxy replacement
disable_kube_proxy = true
# Enable Hubble for observability
cilium_hubble_enabled = true
control_plane_nodepools = [
{
name = "control-plane"
server_type = "cpx31"
location = "fsn1"
count = 3
}
]
agent_nodepools = [
{
name = "worker"
server_type = "cpx41"
location = "fsn1"
count = 3
}
]
}
```
### Cost-Optimized ARM Cluster
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "<LATEST>"
hetzner_token = var.hetzner_token
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
network_region = "eu-central"
# ARM servers (CAX) are ~40% cheaper
control_plane_nodepools = [
{
name = "control-plane"
server_type = "cax21" # ARM
location = "fsn1"
count = 3
}
]
agent_nodepools = [
{
name = "worker-arm"
server_type = "cax31" # ARM
location = "fsn1"
count = 3
}
]
}
```
### Super-HA Multi-Location
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "<LATEST>"
hetzner_token = var.hetzner_token
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
# Must cover ALL locations used
network_region = "eu-central"
# Spread control planes across locations
control_plane_nodepools = [
{
name = "cp-fsn"
server_type = "cpx31"
location = "fsn1"
count = 1
},
{
name = "cp-nbg"
server_type = "cpx31"
location = "nbg1"
count = 1
},
{
name = "cp-hel"
server_type = "cpx31"
location = "hel1"
count = 1
}
]
# Spread workers too
agent_nodepools = [
{
name = "worker-fsn"
server_type = "cpx41"
location = "fsn1"
count = 2
},
{
name = "worker-nbg"
server_type = "cpx41"
location = "nbg1"
count = 2
},
{
name = "worker-hel"
server_type = "cpx41"
location = "hel1"
count = 2
}
]
enable_longhorn = true
}
```
---
## Quick Reference
### Variable Lookup
```bash
# Find specific variable
grep -A10 'variable "<name>"' variables.tf
# Search by keyword
grep -B2 -A10 'description.*<keyword>' variables.tf
# Use Gemini for comprehensive search
gemini --model gemini-3-pro-preview -p "@docs/llms.md Explain the <variable_name> variable"
```
### GitHub Commands
```bash
# Latest release
gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1
# Search issues
gh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --search "<query>" --state all
# View specific issue
gh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --comments
# Search discussions
gh api repos/kube-hetzner/terraform-hcloud-kube-hetzner/discussions --jq '.[].title'
```
### Validation
```bash
terraform fmt
terraform validate
terraform plan # Check for unexpected changes!
```
================================================
FILE: .claude/skills/prepare-release/SKILL.md
================================================
---
name: prepare-release
description: Use when preparing a release - generates changelog, updates version references, and creates release notes
---
# Prepare Release
## Overview
Prepare a new release by generating changelog entries, updating version references, and creating release notes.
## Usage
```
/prepare-release
```
## IMPORTANT: Releases are Manual
**NEVER create release tags automatically.** The maintainer handles all releases manually.
Your job:
- Prepare the changelog
- Update version references
- Generate release notes draft
- Commit preparation changes
User's job:
- Create the actual tag
- Push the tag
- Create GitHub release
## Workflow
```dot
digraph release_flow {
rankdir=TB;
node [shape=box];
analyze [label="1. Analyze changes since last release"];
classify [label="2. Classify release type"];
changelog [label="3. Update CHANGELOG.md"];
badges [label="4. Update version badges"];
gpt [label="5. Update GPT knowledge"];
notes [label="6. Generate release notes"];
commit [label="7. Commit preparation"];
analyze -> classify;
classify -> changelog;
changelog -> badges;
badges -> gpt;
gpt -> notes;
notes -> commit;
}
```
## Step 1: Analyze Changes
```bash
# Get latest release tag
LATEST=$(gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1 --json tagName --jq '.[0].tagName')
echo "Latest release: $LATEST"
# List commits since last release
git log $LATEST..HEAD --oneline
# Get detailed changes
git log $LATEST..HEAD --pretty=format:"- %s (%h)"
```
Use Gemini for comprehensive analysis:
```bash
gemini --model gemini-3-pro-preview -p \
"Analyze these git changes for a changelog. Categorize into: Features, Bug Fixes, Breaking Changes, Documentation. Ignore internal refactors.
$(git log $LATEST..HEAD --oneline)
$(git diff $LATEST..HEAD --stat)"
```
## Step 2: Classify Release Type
| Type | When | Example |
|------|------|---------|
| **PATCH** (x.x.X) | Bug fixes, docs, deps | 2.19.1 |
| **MINOR** (x.X.0) | New features, backward compatible | 2.20.0 |
| **MAJOR** (X.0.0) | Breaking changes | 3.0.0 |
### Breaking Change Indicators
- Variable removed or renamed
- Default value changes behavior
- Resource naming changes (causes recreation)
- Required migration steps
Use Codex for breaking change analysis:
```bash
codex exec -m gpt-5.2-codex -s read-only -c model_reasoning_effort="xhigh" \
"Analyze these changes for breaking changes affecting existing deployments: $(git diff $LATEST..HEAD -- variables.tf locals.tf)"
```
## Step 3: Update CHANGELOG.md
### Changelog Format
```markdown
## [Unreleased]
### ⚠️ Upgrade Notes
<!-- Migration guides, breaking change warnings, special upgrade steps -->
### 🚀 New Features
<!-- New functionality added -->
### 🐛 Bug Fixes
<!-- Bugs that were fixed -->
### 🔧 Changes
<!-- Non-breaking changes, refactors, improvements -->
### 📚 Documentation
<!-- Documentation updates -->
```
### Writing Good Entries
- Write from user's perspective
- Include issue/PR references: `(#1234)`
- Be specific about what changed
- Include migration steps for breaking changes
### Example Entries
```markdown
### 🚀 New Features
- **K3s v1.35 Support** - Added support for k3s v1.35 channel (#2029)
- **NAT Router IPv6** - NAT router now supports IPv6 egress (#2015)
### 🐛 Bug Fixes
- Fixed autoscaler not respecting max_nodes limit (#2018)
- Resolved firewall rules not applying to new nodes (#2012)
### ⚠️ Upgrade Notes
- **NAT Router users**: Run `terraform apply` twice after upgrade due to route changes
```
## Step 4: Update Version Badges
Update README.md badges if version references changed:
```markdown
[](https://k3s.io)
```
Check `versions.tf` for:
- Terraform version requirement
- Provider version requirements
- K3s default channel
## Step 5: Update GPT Knowledge (if applicable)
If significant changes, regenerate the Custom GPT knowledge base:
```bash
# Run the knowledge generation script from CLAUDE.md
python3 << 'PYEOF'
# ... (script from CLAUDE.md)
PYEOF
```
Update `meta.version` in the script to match new release.
## Step 6: Generate Release Notes
### Release Notes Template
```markdown
## 🚀 Release vX.Y.Z
### Highlights
- **Feature 1**: Brief description
- **Feature 2**: Brief description
### ⚠️ Upgrade Notes
[Any special upgrade instructions]
### What's Changed
#### New Features
- Feature description (#PR)
#### Bug Fixes
- Fix description (#PR)
#### Other Changes
- Change description (#PR)
### Full Changelog
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/compare/vPREV...vX.Y.Z
### Upgrade
\`\`\`tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "X.Y.Z"
# ...
}
\`\`\`
\`\`\`bash
terraform init -upgrade
terraform plan
terraform apply
\`\`\`
```
## Step 7: Commit Preparation
```bash
git add CHANGELOG.md README.md
git commit -m "$(cat <<'EOF'
chore: prepare release vX.Y.Z
- Update CHANGELOG.md with release notes
- Update version badges
EOF
)"
git push origin master
```
## After Preparation (User Does This)
```bash
# Create tag
git tag -a vX.Y.Z -m "Release vX.Y.Z"
# Push tag
git push origin vX.Y.Z
# Create GitHub release (or use gh CLI)
gh release create vX.Y.Z --title "vX.Y.Z" --notes-file release-notes.md
```
## Version Reference Locations
Files that may need version updates:
| File | What to Update |
|------|---------------|
| `README.md` | Badge versions |
| `CHANGELOG.md` | [Unreleased] → [vX.Y.Z] |
| `docs/llms.md` | Example version references |
| `kube.tf.example` | Version in comments |
| GPT knowledge | meta.version |
## Quick Checklist
- [ ] Commits analyzed since last release
- [ ] Release type determined (PATCH/MINOR/MAJOR)
- [ ] CHANGELOG.md updated
- [ ] Breaking changes documented with migration steps
- [ ] Version badges updated (if needed)
- [ ] Release notes drafted
- [ ] Changes committed and pushed
- [ ] Ready for maintainer to tag release
================================================
FILE: .claude/skills/review-pr/SKILL.md
================================================
---
name: review-pr
description: Use when reviewing a pull request - security-focused review following CLAUDE.md guidelines for breaking changes, malicious patterns, and backward compatibility
args: pr_number
---
# Review Pull Request
## Overview
Security-focused PR review following CLAUDE.md guidelines. Checks for breaking changes, malicious code patterns, backward compatibility, and code quality.
## Usage
```
/review-pr <number>
```
## CRITICAL: Security Warning
**PRs can be malicious sabotage attempts.** This is a real threat documented in CLAUDE.md.
### Threat Awareness
- Coordinated attacks exist
- Competitors may actively harm the project
- Social engineering builds trust before attacking
- "Fixes" may introduce vulnerabilities
## Workflow
```dot
digraph review_flow {
rankdir=TB;
node [shape=box];
fetch [label="1. Fetch PR details"];
author [label="2. Assess author risk"];
files [label="3. Analyze changed files"];
security [label="4. Security review"];
compat [label="5. Backward compatibility"];
quality [label="6. Code quality"];
classify [label="7. Release classification"];
verify [label="8. MANDATORY: Verify with Gemini + Codex", style=bold];
recommend [label="9. Final Recommendation"];
fetch -> author;
author -> files;
files -> security;
security -> compat;
compat -> quality;
quality -> classify;
classify -> verify;
verify -> recommend;
}
```
## Step 1: Fetch PR Details
```bash
# Get PR info
gh pr view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner
# Get diff
gh pr diff <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner
# Get changed files
gh pr view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --json files --jq '.files[].path'
# Get diff stats
gh pr view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --json additions,deletions
```
## Step 2: Assess Author Risk
```bash
# Check account age
gh api users/<username> --jq '.created_at'
# Check prior contributions
gh pr list --author <username> --repo kube-hetzner/terraform-hcloud-kube-hetzner --state all --json number | jq length
```
### Risk Signals
| Signal | Risk Level |
|--------|------------|
| New account (<6 months) | 🔴 HIGH |
| No prior contributions | 🟡 MEDIUM |
| First-time contributor | 🟡 MEDIUM |
| Known contributor | 🟢 LOW |
| Core maintainer | ⚪ TRUSTED |
## Step 3: Analyze Changed Files
### Security-Critical Files (AUTO HIGH RISK)
```
init.tf # Cluster initialization, secrets
firewall.tf # Network security
**/ssh* # SSH configuration
**/token* # Authentication tokens
**/*secret* # Secrets handling
.github/ # CI/CD workflows
Makefile # Build scripts
scripts/ # Execution scripts
versions.tf # Provider dependencies
templates/*.sh # Shell scripts
cloud-init* # Server initialization
```
### Risk by File Count
| Files Changed | Risk |
|---------------|------|
| 1-3 files | 🟢 LOW |
| 4-10 files | 🟡 MEDIUM |
| 11-20 files | 🟡 MEDIUM |
| >20 files | 🔴 HIGH |
### Risk by Diff Size
| Lines Changed | Risk |
|---------------|------|
| <50 lines | 🟢 LOW |
| 50-200 lines | 🟡 MEDIUM |
| 200-500 lines | 🟡 MEDIUM |
| >500 lines | 🔴 HIGH |
## Step 4: Security Review
### Checklist
- [ ] No hardcoded credentials or tokens
- [ ] No suspicious external URLs
- [ ] No obfuscated code
- [ ] Changes match stated purpose
- [ ] No unnecessary permission escalations
- [ ] CI/CD changes justified
- [ ] No bypassing existing security patterns
### Red Flags
| Pattern | Concern |
|---------|---------|
| Base64 encoded strings | Hidden payloads |
| External curl/wget calls | Code injection |
| Eval or exec statements | Command injection |
| Overly complex logic | Hiding malicious code |
| Unnecessary file access | Data exfiltration |
| Changes to .gitignore | Hiding tracks |
### Use AI for Deep Analysis
```bash
# Codex for security analysis
codex exec -m gpt-5.3-codex -s read-only -c model_reasoning_effort="xhigh" \
"Analyze this PR diff for security vulnerabilities and malicious patterns: $(gh pr diff <num>)"
# Gemini for broad context
gemini --model gemini-3-pro-preview -p \
"@locals.tf @init.tf Does this PR introduce any security concerns? $(gh pr diff <num>)"
```
## Step 5: Backward Compatibility
**CRITICAL: Any PR that causes resource recreation is a MAJOR release.**
### Breaking Change Indicators
- Removes or renames variables
- Changes variable defaults that affect behavior
- Modifies resource naming patterns
- Alters subnet/network calculations
- Changes resource keys (causes recreation)
- Removes outputs
- Modifies provider requirements
### Test for Breaking Changes
```bash
# Checkout PR locally
gh pr checkout <number>
# Test against existing cluster
cd /path/to/kube-test
terraform init -upgrade
terraform plan
```
**If `terraform plan` shows ANY resource destruction → MAJOR release required**
### Compatibility Checklist
- [ ] No variable removals
- [ ] No default changes that affect behavior
- [ ] No resource naming changes
- [ ] `terraform plan` shows no destruction
- [ ] Existing deployments unaffected
## Step 6: Code Quality
### Style
- [ ] Follows existing patterns
- [ ] Consistent naming
- [ ] Proper formatting (`terraform fmt`)
- [ ] No unnecessary complexity
### Logic
- [ ] Changes are correct
- [ ] Edge cases handled
- [ ] No regressions introduced
- [ ] Tests pass
## Step 7: Release Classification
### PATCH (x.x.PATCH)
- Bug fixes only
- No new features
- Fully backward compatible
- No terraform state impact
### MINOR (x.MINOR.0)
- New features (backward compatible)
- New optional variables with defaults
- Deprecation warnings (not removals)
### MAJOR (MAJOR.0.0)
- Breaking changes
- Removed/renamed variables
- Changed defaults affecting behavior
- State migrations required
- Resource recreations
## Step 8: MANDATORY - Verify with Gemini and Codex
**CRITICAL: Before making your final recommendation, you MUST run both Gemini and Codex to triple-verify the PR.**
This is not optional. External AI verification catches issues that may be missed in the initial review.
### Run Both in Parallel
```bash
# Gemini - Broad context analysis (run first or in parallel)
gemini --model gemini-3-pro-preview -p "@control_planes.tf @locals.tf @init.tf
Analyze this PR diff for the kube-hetzner terraform module:
$(gh pr diff <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner)
Questions:
1. Is this change consistent with existing patterns in the codebase?
2. Are there any security concerns?
3. Could this cause breaking changes or resource recreation?
4. Is this a legitimate bug fix or could it be malicious?"
# Codex - Deep reasoning security analysis (run in parallel)
codex exec -m gpt-5.3-codex -s read-only -c model_reasoning_effort="xhigh" \
"Analyze this Terraform PR for the kube-hetzner module.
DIFF:
$(gh pr diff <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner)
SECURITY ANALYSIS QUESTIONS:
1. Could this change introduce any security vulnerabilities?
2. Could this be a malicious change disguised as a bug fix?
3. Will this cause any Terraform state changes or resource recreation?
4. Is this pattern safe and consistent with Terraform best practices?
5. Any edge cases or potential issues?"
```
### Verification Checklist
- [ ] Gemini analysis completed
- [ ] Codex analysis completed
- [ ] Both agree the change is safe
- [ ] No concerns raised by either tool
- [ ] If concerns raised, they have been addressed or explained
### When Reviewers Disagree
If Gemini or Codex raises concerns that you didn't catch:
1. **Take the concern seriously** - investigate further
2. **Re-read the code** with the concern in mind
3. **Request changes** if the concern is valid
4. **Document** why the concern was dismissed if you determine it's a false positive
### Output in Final Review
Include a summary of external verification:
```markdown
### External AI Verification
| Reviewer | Verdict | Key Finding |
|----------|---------|-------------|
| Claude | ✅/❌ | <summary> |
| Gemini | ✅/❌ | <summary> |
| Codex | ✅/❌ | <summary> |
**Consensus:** All reviewers agree / Disagreement on X
```
---
## Step 9: Final Recommendation
### PR Review Output Template
```markdown
## PR Review: #<number>
**Title:** <title>
**Author:** @<username>
**Files:** <count> files changed (+<additions>/-<deletions>)
### Risk Assessment
| Factor | Value | Risk |
|--------|-------|------|
| Author tenure | X months | 🟢/🟡/🔴 |
| Prior contributions | N PRs | 🟢/🟡/🔴 |
| Files changed | N files | 🟢/🟡/🔴 |
| Lines changed | +X/-Y | 🟢/🟡/🔴 |
| Security-critical files | Yes/No | 🟢/🔴 |
| External dependencies | Yes/No | 🟢/🔴 |
**Overall Risk:** 🔴 HIGH / 🟡 MEDIUM / 🟢 LOW
### Security Review
- [ ] No hardcoded credentials
- [ ] No suspicious external URLs
- [ ] No obfuscated code
- [ ] Changes match stated purpose
### Backward Compatibility
- [ ] No breaking changes
- [ ] terraform plan shows no destruction
- [ ] Existing deployments unaffected
### Release Classification
**Type:** PATCH / MINOR / MAJOR
**Reason:** <explanation>
### External AI Verification
| Reviewer | Verdict | Key Finding |
|----------|---------|-------------|
| Claude | ✅/❌ | <summary> |
| Gemini | ✅/❌ | <summary> |
| Codex | ✅/❌ | <summary> |
**Consensus:** All agree / Disagreement on X
### Recommendation
**Action:** APPROVE / REQUEST CHANGES / CLOSE
**Notes:** <specific concerns or required changes>
```
## Quick Commands
```bash
# Approve PR
gh pr review <num> --approve --body "LGTM! ..."
# Request changes
gh pr review <num> --request-changes --body "Please address: ..."
# Comment
gh pr review <num> --comment --body "..."
# Merge (after approval)
gh pr merge <num> --squash --delete-branch
```
## Never Merge Directly to Master
All PRs go through staging branches first:
1. Create staging branch
2. Test thoroughly
3. Get AI review (Codex + Gemini)
4. Then merge to master
================================================
FILE: .claude/skills/sync-docs/SKILL.md
================================================
---
name: sync-docs
description: Use when documentation needs updating - ensures variables.tf, llms.md, kube.tf.example, and README are in sync
---
# Sync Documentation
## Overview
Ensure documentation is synchronized across all key files when variables or features change.
## Usage
```
/sync-docs
```
## Documentation Files
| File | Purpose | Priority |
|------|---------|----------|
| `variables.tf` | Source of truth for all variables | PRIMARY |
| `docs/llms.md` | Comprehensive variable reference | HIGH |
| `kube.tf.example` | Working example configuration | HIGH |
| `README.md` | Project overview and quick start | MEDIUM |
| `docs/terraform.md` | Auto-generated terraform docs | AUTO |
## Workflow
```dot
digraph sync_flow {
rankdir=TB;
node [shape=box];
extract [label="1. Extract from variables.tf"];
compare [label="2. Compare with llms.md"];
gaps [label="3. Identify gaps"];
update_llms [label="4. Update llms.md"];
update_example [label="5. Update kube.tf.example"];
update_readme [label="6. Update README if needed"];
verify [label="7. Verify consistency"];
extract -> compare;
compare -> gaps;
gaps -> update_llms;
update_llms -> update_example;
update_example -> update_readme;
update_readme -> verify;
}
```
## Step 1: Extract Variables from Source
Use Gemini for large file analysis:
```bash
# List all variables from variables.tf
gemini --model gemini-3-pro-preview -p "@variables.tf List ALL variable names defined in this file, one per line"
# Get variable details
gemini --model gemini-3-pro-preview -p "@variables.tf For variable '<name>', provide: type, default, description"
```
## Step 2: Find Undocumented Variables
```bash
# Compare variables.tf with llms.md
gemini --model gemini-3-pro-preview -p \
"@variables.tf @docs/llms.md List ALL variables from variables.tf that are NOT documented in llms.md. Output one per line."
```
## Step 3: Generate Documentation
### llms.md Format
```markdown
**Variable Name**
```tf
variable_name = "default_value"
```
* **`variable_name` (Type, Optional/Required):**
* **Default:** `default_value`
* **Purpose:** Clear explanation of what this does
* **Usage:** When and how to use it
* **Considerations:** Important notes, limitations, impacts
* **Example:** Practical usage example if helpful
```
### kube.tf.example Format
```tf
# Description of what this controls
# Additional context if needed
# variable_name = "default_value"
```
## Step 4: Update llms.md
For each undocumented variable:
1. Read variable definition from `variables.tf`
2. Understand its usage in `locals.tf` and other files
3. Write comprehensive documentation following the format above
4. Place in appropriate section of `llms.md`
### Section Organization in llms.md
| Section | Variables |
|---------|-----------|
| Cluster Basics | cluster_name, hetzner_token, ssh_* |
| Network | network_*, subnet_* |
| Control Plane | control_plane_* |
| Agents | agent_*, autoscaler_* |
| Load Balancer | lb_*, traefik_*, nginx_* |
| CNI | cni_*, cilium_*, calico_* |
| Storage | longhorn_* |
| Security | firewall_*, audit_* |
| Advanced | Additional/misc options |
## Step 5: Update kube.tf.example
Ensure new variables appear in the example with:
- Clear comment explaining purpose
- Commented out with default value
- Grouped with related variables
```bash
# Check what's in example vs variables.tf
gemini --model gemini-3-pro-preview -p \
"@variables.tf @kube.tf.example List variables from variables.tf missing from kube.tf.example"
```
## Step 6: Update README if Needed
Update README.md if:
- New major feature added
- New CNI or ingress option
- Significant capability change
Features section should match actual capabilities.
## Step 7: Verify Consistency
```bash
# Final verification
gemini --model gemini-3-pro-preview -p \
"@variables.tf @docs/llms.md @kube.tf.example Verify these files are consistent. List any discrepancies."
```
### Verification Checklist
- [ ] All variables.tf variables documented in llms.md
- [ ] All major variables appear in kube.tf.example
- [ ] README features match actual capabilities
- [ ] No typos in variable names across files
- [ ] Default values consistent across docs
## Common Sync Issues
### Variable renamed
1. Update in variables.tf
2. Search and replace in llms.md
3. Search and replace in kube.tf.example
4. Add to CHANGELOG.md (breaking change!)
### Variable removed
1. Remove from variables.tf
2. Remove from llms.md
3. Remove from kube.tf.example
4. Add to CHANGELOG.md (breaking change!)
### Default changed
1. Update in variables.tf
2. Update in llms.md
3. Update in kube.tf.example
4. Consider if this is a breaking change
## Quick Commands
```bash
# Regenerate terraform docs
terraform-docs markdown . > docs/terraform.md
# Search for variable across all docs
grep -r "variable_name" docs/ kube.tf.example README.md
# Find undocumented variables (quick check)
diff <(grep -oP 'variable "\K[^"]+' variables.tf | sort) \
<(grep -oP '`\K[a-z_]+(?=`)' docs/llms.md | sort -u) | grep "^<"
```
## After Sync
1. Run `terraform fmt`
2. Commit with message: `docs: sync documentation with variables.tf`
3. If breaking changes, update CHANGELOG.md
================================================
FILE: .claude/skills/test-changes/SKILL.md
================================================
---
name: test-changes
description: Use after making changes to run terraform fmt, validate, and plan against test environment
---
# Test Terraform Changes
## Overview
Run the standard validation suite for terraform changes against the test environment.
## Usage
```
/test-changes
```
## Test Environment
- **Module code:** `/Volumes/MysticalTech/Code/kube-hetzner`
- **Test cluster:** `/Users/karim/Code/kube-test`
## Workflow
```dot
digraph test_flow {
rankdir=TB;
node [shape=box];
fmt [label="1. terraform fmt"];
validate [label="2. terraform validate"];
init [label="3. terraform init -upgrade"];
plan [label="4. terraform plan"];
review [label="5. Review plan output"];
fmt -> validate;
validate -> init;
init -> plan;
plan -> review;
}
```
## Step 1: Format Check
```bash
cd /Volumes/MysticalTech/Code/kube-hetzner
terraform fmt -recursive
```
**Must pass before proceeding.**
## Step 2: Validate Module
```bash
cd /Volumes/MysticalTech/Code/kube-hetzner
terraform validate
```
**Must pass before proceeding.**
## Step 3: Initialize Test Environment
```bash
cd /Users/karim/Code/kube-test
terraform init -upgrade
```
This picks up changes from the local module.
## Step 4: Plan Against Test Cluster
```bash
cd /Users/karim/Code/kube-test
terraform plan
```
### What to Look For
#### Good Signs
- Only expected resources change
- No unexpected additions/deletions
- Changes match your intended modifications
#### Red Flags (STOP!)
| Output | Meaning | Action |
|--------|---------|--------|
| `will be destroyed` | Resource recreation | **STOP** - Breaking change |
| `must be replaced` | Resource recreation | **STOP** - Breaking change |
| `forces replacement` | Resource recreation | **STOP** - Breaking change |
| Unexpected changes | Side effects | Investigate before proceeding |
### Breaking Change = MAJOR Release
If `terraform plan` shows ANY resource destruction on existing infrastructure:
1. **STOP** - This is NOT backward compatible
2. The change requires a MAJOR version bump
3. Migration guide is required
4. Consider alternative approaches first
## Step 5: Review Plan Output
### Checklist
- [ ] `terraform fmt` passes
- [ ] `terraform validate` passes
- [ ] `terraform plan` shows expected changes only
- [ ] No resource destruction
- [ ] No unexpected side effects
- [ ] Changes are backward compatible
## Quick Reference
```bash
# Full test sequence
cd /Volumes/MysticalTech/Code/kube-hetzner && \
terraform fmt -recursive && \
terraform validate && \
cd /Users/karim/Code/kube-test && \
terraform init -upgrade && \
terraform plan
```
## Apply (Optional)
Only if plan looks correct and you want to test on actual infrastructure:
```bash
cd /Users/karim/Code/kube-test
terraform apply
```
**Caution:** This modifies real infrastructure. Only do this for thorough testing.
## Common Issues
### "Provider version constraints"
```bash
terraform init -upgrade
```
### "Module source has changed"
```bash
terraform init -upgrade
```
### "State lock"
Someone else may be running terraform. Wait or:
```bash
terraform force-unlock <lock-id>
```
### Validation errors
Check the error message - usually points to:
- Missing required variable
- Type mismatch
- Invalid reference
## AI-Assisted Review
For complex changes, get AI review:
```bash
# Codex for correctness
codex exec -m gpt-5.2-codex -s read-only -c model_reasoning_effort="xhigh" \
"Review these terraform changes for issues: $(git diff)"
# Gemini for broad impact
gemini --model gemini-3-pro-preview -p \
"@locals.tf @variables.tf Analyze impact of these changes: $(git diff)"
```
================================================
FILE: .claude/skills/triage-issue/SKILL.md
================================================
---
name: triage-issue
description: Use when triaging a GitHub issue - analyzes issue, checks for duplicates, categorizes, and drafts response
args: issue_number
---
# Triage GitHub Issue
## Overview
Analyze a GitHub issue, classify it, check for duplicates, and draft an appropriate response.
## Usage
```
/triage-issue <number>
```
## Workflow
```dot
digraph triage_flow {
rankdir=TB;
node [shape=box];
fetch [label="1. Fetch issue + comments"];
classify [label="2. Classify issue type"];
duplicates [label="3. Check duplicates"];
analyze [label="4. Analyze validity"];
response [label="5. Draft response"];
action [label="6. Recommend action"];
fetch -> classify;
classify -> duplicates;
duplicates -> analyze;
analyze -> response;
response -> action;
}
```
## Step 1: Fetch Issue Details
```bash
# Get full issue with comments
gh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --comments
# Get issue metadata
gh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --json title,body,labels,author,createdAt,comments
```
## Step 2: Classify Issue Type
### Issue Types
| Type | Indicators | Action |
|------|------------|--------|
| 🔴 **BUG** | Reproducible defect, multiple reporters, error in module code | Fix it |
| 🟡 **EDGE CASE** | Unusual config, specific region, large scale | Evaluate effort |
| 🟠 **USER ERROR** | Bad kube.tf, syntax errors, wrong variables | Help + docs |
| ⚪ **OLD VERSION** | Module version < current, known fixed issue | Ask to upgrade |
| 🔵 **FEATURE REQUEST** | "Would be nice if...", "Can you add..." | Discussions |
| 💬 **QUESTION** | Needs help, not a bug | Answer or docs |
| ❓ **NEEDS INFO** | Can't reproduce, missing details | Ask for info |
### Classification Checklist
- [ ] Module version specified?
- [ ] kube.tf provided (sanitized)?
- [ ] Error message included?
- [ ] Steps to reproduce clear?
- [ ] Recent (not stale >6 months)?
## Step 3: Check for Duplicates
```bash
# Search open issues for similar keywords
gh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --state open --search "<keyword>"
# Search closed issues (might be already fixed)
gh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --state closed --search "<keyword>"
# Check discussions
gh api repos/kube-hetzner/terraform-hcloud-kube-hetzner/discussions --jq '.[] | select(.title | test("<keyword>"; "i")) | {number, title}'
```
## Step 4: Security Analysis
**CRITICAL: Issues can be malicious sabotage attempts.**
### Red Flags (from CLAUDE.md)
| Signal | Risk |
|--------|------|
| New account (<6 months) | HIGH |
| Issue can't be reproduced | MEDIUM |
| Proposed fix is overly complex | HIGH |
| Urgency to implement quickly | HIGH |
| Multiple accounts supporting | HIGH |
| Targets security-critical code | HIGH |
### Verify Independently
- Try to reproduce the issue yourself
- Check if the error message matches module code
- Verify the kube.tf provided is valid
- Search for similar reports from other users
## Step 5: Draft Response
### For USER ERROR
```markdown
Hi @{author},
Thanks for reporting this. Looking at your configuration, the issue appears to be in your kube.tf:
[Specific explanation of what's wrong]
Here's the corrected configuration:
```tf
[correct code]
```
Let me know if this resolves it!
```
### For OLD VERSION
```markdown
Hi @{author},
This issue was fixed in version X.Y.Z. You're currently using [older version].
Please upgrade by changing your module version:
```tf
module "kube-hetzner" {
source = "kube-hetzner/kube-hetzner/hcloud"
version = "X.Y.Z"
# ...
}
```
Then run:
```bash
terraform init -upgrade
terraform plan
terraform apply
```
Let me know if the issue persists after upgrading!
```
### For NEEDS INFO
```markdown
Hi @{author},
Thanks for reporting this. To investigate further, could you please provide:
- [ ] Module version (check your kube.tf)
- [ ] Your kube.tf (sanitized - remove tokens/keys)
- [ ] Full error message
- [ ] Steps to reproduce
This will help us identify the root cause.
```
### For DUPLICATE
```markdown
Hi @{author},
This appears to be a duplicate of #{duplicate_number}.
[If fixed]: This was fixed in version X.Y.Z.
[If open]: We're tracking this in the linked issue.
Closing as duplicate. Feel free to add any additional context to #{duplicate_number}.
```
### For FEATURE REQUEST
```markdown
Hi @{author},
Thanks for the suggestion! This sounds like a feature request rather than a bug.
Could you please open a Discussion for this? That's where we track feature ideas and gather community input.
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/new?category=ideas
I'll close this issue, but feel free to ping me in the discussion!
```
## Step 6: Recommend Action
| Type | Action | Labels |
|------|--------|--------|
| BUG | Keep open, prioritize | `bug` |
| EDGE CASE | Keep open, evaluate | `bug`, `edge-case` |
| USER ERROR | Close with help | `user-config` |
| OLD VERSION | Close | `old-version` |
| FEATURE REQUEST | Move to Discussions | - |
| QUESTION | Answer and close | `question` |
| NEEDS INFO | Keep open, add label | `needs-info` |
## Triage Output Template
```markdown
## Triage Summary: Issue #<number>
**Title:** <title>
**Author:** @<username>
**Created:** <date>
### Classification
**Type:** <BUG/EDGE CASE/USER ERROR/OLD VERSION/FEATURE/QUESTION/NEEDS INFO>
**Confidence:** HIGH/MEDIUM/LOW
**Reason:** <why this classification>
### Checklist
- [ ] Module version: <version or "not specified">
- [ ] kube.tf provided: Yes/No/Partial
- [ ] Reproducible: Yes/No/Unknown
- [ ] Duplicate: No / Yes → #<number>
### Analysis
<What's actually happening and why>
### Recommended Action
**Action:** <FIX/HELP USER/CLOSE/MOVE TO DISCUSSIONS/NEEDS INFO>
**Priority:** HIGH/MEDIUM/LOW
**Response:** <drafted response above>
```
## Quick Commands
```bash
# Add label
gh issue edit <num> --add-label "bug"
# Close issue
gh issue close <num> --comment "Closing because..."
# Close as not planned
gh issue close <num> --reason "not planned" --comment "..."
# Transfer to discussions
gh issue transfer <num> --repo kube-hetzner/terraform-hcloud-kube-hetzner
```
================================================
FILE: .github/FUNDING.yml
================================================
# These are supported funding model platforms
github: mysticaltech
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.yml
================================================
name: Bug Report
description: File a bug report
title: "[Bug]: "
labels: ["bug", "triage"]
body:
- type: markdown
attributes:
value: |
**Kube-Hetzner Bug Report**
Before doing so, please research the project, the solution may be in the documentation, in other issues, or in the discussions. If none of the above gave you the answer, please explain the bug in detail and provide as much information as possible.
- type: textarea
id: description
attributes:
label: Description
description: A clear and concise description of what the bug is.
placeholder: What's happening?
validations:
required: true
- type: textarea
id: kube_tf
attributes:
label: Kube.tf file
description: Please share your kube.tf file, without sensitive values, and if possible, stripped of comments.
placeholder: Enter your kube.tf content goes here
render: terraform
validations:
required: true
- type: textarea
id: screenshots
attributes:
label: Screenshots
description: If applicable, add screenshots of the errors.
placeholder: Enter screenshots here
- type: input
id: platform
attributes:
label: Platform
description: Windows, Linux, Mac
placeholder: Enter platform here
validations:
required: true
================================================
FILE: .github/ISSUE_TEMPLATE/config.yml
================================================
blank_issues_enabled: true
contact_links:
- name: If you have questions, use the discussions
url: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions
about: Please ask and answer questions here.
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.yml
================================================
name: Feature Request
title: "[Feature Request]: "
description: "Submit a feature request for consideration"
labels: ["feature request"]
body:
- type: markdown
attributes:
value: |
**Kube-Hetzner Feature Request**
For feature exploration, please use our [discussions](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions) section. However, if you judge something to be extremely important or urgent, please let us know what you need and why, and even share tips on implementation.
Also please know that we are very open to PRs and will work with you to make it happen if fully relevent to the project. So if you can make it happen, and it falls within the project, please do not hesitate to ping us in the discussions about your idea.
- type: textarea
id: feature-description
attributes:
label: Description
description: Tell us more about your feature request.
placeholder: "E.g. Adding support for XYZ would greatly improve the user experience..."
validations:
required: true
================================================
FILE: .github/dependabot.yaml
================================================
version: 2
updates:
- package-ecosystem: "terraform"
directory: "/"
schedule:
interval: "daily"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
================================================
FILE: .github/release.yaml
================================================
changelog:
exclude:
labels:
- ignore-for-release
authors:
- octocat
categories:
- title: Breaking Changes 🛠
labels:
- Semver-Major
- breaking-change
- title: New Features 🎉
labels:
- Semver-Minor
- enhancement
- title: Bug Fixes 🐛
labels:
- Semver-Patch
- bug
- title: Other Changes
labels:
- "*"
================================================
FILE: .github/release.yml
================================================
# Configuration for auto-generated release notes
# https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes
changelog:
exclude:
labels:
- skip-changelog
authors:
- dependabot
- dependabot[bot]
categories:
- title: "🚀 New Features"
labels:
- enhancement
- feature
- title: "🐛 Bug Fixes"
labels:
- bug
- fix
- bugfix
- title: "📦 Packer / OS"
labels:
- packer
- microos
- title: "🔧 Configuration & Variables"
labels:
- configuration
- variables
- title: "📚 Documentation"
labels:
- documentation
- docs
- title: "🏗️ Infrastructure"
labels:
- infrastructure
- terraform
- title: "⬆️ Dependencies"
labels:
- dependencies
- title: "🔒 Security"
labels:
- security
- title: "Other Changes"
labels:
- "*"
================================================
FILE: .github/workflows/generate-docs.yaml
================================================
name: Generate terraform docs
on:
push:
branches:
- master
- staging
jobs:
docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.ref }}
fetch-depth: 0 # Necessary to fetch all history for create-pull-request to work correctly
- name: Render terraform docs and push changes back to PR
uses: terraform-docs/gh-actions@main
with:
working-dir: .
output-file: docs/terraform.md
output-method: inject
config-file: ".terraform-docs.yml"
- name: Create Pull Request
uses: peter-evans/create-pull-request@v8
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: Update Terraform documentation
title: "[AUTO] Update Terraform Documentation"
body: "Automated changes by GitHub Actions"
branch: "docs/update-${{ github.head_ref }}"
labels: documentation
================================================
FILE: .github/workflows/lint_pr.yaml
================================================
name: Lint
on:
pull_request:
jobs:
tfsec:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
name: Scan terraform files with tfsec
steps:
- name: Clone repo
uses: actions/checkout@v6
- name: tfsec
uses: aquasecurity/tfsec-pr-commenter-action@v1.3.1
with:
github_token: ${{ github.token }}
tfsec_args: --ignore-hcl-errors
- name: Run tfsec with reviewdog output on the PR
uses: reviewdog/action-tfsec@v1.30.0
with:
github_token: ${{ secrets.github_token }}
filter_mode: nofilter
fail_on_error: true
tfsec_flags: --ignore-hcl-errors
validate:
runs-on: ubuntu-latest
name: Validate terraform configuration
steps:
- name: Checkout
uses: actions/checkout@v6
- name: terraform validate
uses: dflook/terraform-validate@v2.2.3
fmt-check:
runs-on: ubuntu-latest
name: Check formatting of terraform files
steps:
- name: Checkout
uses: actions/checkout@v6
- name: terraform fmt
uses: dflook/terraform-fmt-check@v2.2.3
================================================
FILE: .github/workflows/publish-release.yaml
================================================
---
name: Publish a new Github Release
on:
push:
tags:
- '*'
workflow_dispatch:
jobs:
Release:
name: Release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0 # Full history for contributor extraction
- name: Get previous tag
id: prev_tag
run: |
PREV_TAG=$(git describe --tags --abbrev=0 HEAD^ 2>/dev/null || echo "")
echo "tag=${PREV_TAG}" >> "$GITHUB_OUTPUT"
- name: Extract changelog for this release
run: |
# Extract the [Unreleased] section from CHANGELOG.md
if [ -f CHANGELOG.md ]; then
# Get content between [Unreleased] and the next ## heading
awk '/^## \[Unreleased\]/{flag=1; next} /^## \[/{flag=0} flag' CHANGELOG.md > changelog_section.md
if [ -s changelog_section.md ]; then
echo "Found changelog content:"
cat changelog_section.md
else
echo "No unreleased changelog content found"
: > changelog_section.md
fi
else
echo "No CHANGELOG.md found"
: > changelog_section.md
fi
- name: Generate contributors list
env:
PREV_TAG: ${{ steps.prev_tag.outputs.tag }}
run: |
if [ -n "${PREV_TAG}" ]; then
RANGE="${PREV_TAG}..HEAD"
else
RANGE="HEAD"
fi
# Get unique contributors from commits and co-authors
CONTRIBUTORS=$(git log "${RANGE}" --format='%an' | sort -u | while read -r name; do
echo "* ${name}"
done)
# Also extract Co-authored-by names
COAUTHORS=$(git log "${RANGE}" --format='%b' | grep -i "co-authored-by" | sed 's/.*: //' | sed 's/ <.*//' | sort -u | while read -r name; do
[ -n "$name" ] && echo "* ${name}"
done)
# Combine and dedupe
ALL_CONTRIBUTORS=$(printf '%s\n%s' "${CONTRIBUTORS}" "${COAUTHORS}" | grep -v "^$" | sort -u)
# Write to file for multi-line output
{
echo "## 👥 Contributors"
echo ""
echo "Thanks to all contributors who made this release possible:"
echo ""
echo "${ALL_CONTRIBUTORS}"
} > contributors.md
echo "Generated contributors list:"
cat contributors.md
- name: Combine release notes
run: |
# Combine changelog + contributors into final release body
{
cat changelog_section.md
echo ""
cat contributors.md
} > release_body.md
echo "Final release body:"
cat release_body.md
- name: Create Release
uses: ncipollo/release-action@v1
with:
generateReleaseNotes: true
name: ${{ github.ref_name }}
token: ${{ secrets.GITHUB_TOKEN }}
bodyFile: release_body.md
appendBody: true
================================================
FILE: .gitignore
================================================
# Local .terraform directories
**/.terraform/*
# .tfstate files
*.tfstate
*.tfstate.*
# Crash log files
crash.log
crash.*.log
# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
*.tfvars
*.tfvars.json
# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json
# Include override files you do wish to add to version control using negated pattern
# !example_override.tf
# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*
# Ignore CLI configuration files
.terraformrc
terraform.rc
*_kubeconfig.yaml
*_kubeconfig.yaml-e
terraform.tfvars
plans-custom.yaml
kustomization.yaml
*kustomization_backup.yaml
kube.tf
.terraform.lock.hcl
issue_fix.patch
# AI related files
CLAUDE.md
kube-hetzner-knowledge.jsondata
# Misc
.DS_Store
requirements/*
# Ignore IntelliJ related files
.idea
# Local triage documentation
issues/
prs/
# Local analysis artifacts
.tldr/
.tldrignore
scripts/process_ideas_v3.py
tasks/
================================================
FILE: .pre-commit-config.yaml
================================================
default_install_hook_types:
- pre-commit
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.97.3
hooks:
- id: terraform_fmt
- id: terraform_validate
- id: terraform_tfsec
pass_filenames: false
- id: terraform_docs
args:
- '--args=--lockfile=false'
# - id: terraform_tflint
# args:
# - '--args=--only=terraform_deprecated_interpolation'
# - '--args=--only=terraform_deprecated_index'
# - '--args=--only=terraform_unused_declarations'
# - '--args=--only=terraform_comment_syntax'
# - '--args=--only=terraform_documented_outputs'
# - '--args=--only=terraform_documented_variables'
# - '--args=--only=terraform_typed_variables'
# - '--args=--only=terraform_module_pinned_source'
# - '--args=--only=terraform_naming_convention'
# - '--args=--only=terraform_required_version'
# - '--args=--only=terraform_required_providers'
# - '--args=--only=terraform_standard_module_structure'
# - '--args=--only=terraform_workspace_remote'
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: end-of-file-fixer
================================================
FILE: .terraform-docs.yml
================================================
formatter: "markdown table"
recursive:
enabled: false
path: modules
output:
file: docs/terraform.md
mode: inject
template: |-
<!-- BEGIN_TF_DOCS -->
{{ .Content }}
<!-- END_TF_DOCS -->
output-values:
enabled: false
from: ""
sort:
enabled: true
by: name
settings:
anchor: true
color: true
default: true
description: false
escape: true
hide-empty: false
html: true
indent: 3
lockfile: true
read-comments: true
required: true
sensitive: true
type: true
================================================
FILE: CHANGELOG.md
================================================
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### 📋 v2.19.1 Patch Release
This is a patch release for v2.19.0. **If upgrading from v2.18.x**, please review the full release notes below including upgrade notes, new features, and breaking changes.
**Patch fix:**
- **Audit Policy Bastion Connection** - Fixed missing bastion SSH settings in `audit_policy` provisioner, enabling audit policy deployment for NAT router / private network setups (#2042) - thanks @CounterClops
---
### ⚠️ Upgrade Notes (from v2.18.x)
#### NAT Router Users (created before v2.19.0)
If you created a NAT router **before v2.19.0** (when the hcloud provider used the now-deprecated `datacenter` attribute), you may see Terraform wanting to recreate your NAT router primary IPs. This would result in new IP addresses.
**To check if you're affected**, run `terraform plan` and look for changes to:
- `hcloud_primary_ip.nat_router_primary_ipv4`
- `hcloud_primary_ip.nat_router_primary_ipv6`
**If Terraform shows replacement**, you have two options:
1. **Allow the recreation** (simplest, but IPs will change):
```bash
terraform apply
```
2. **Migrate state manually** (preserves IPs):
```bash
# Remove old state entries
terraform state rm 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv4[0]'
terraform state rm 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv6[0]'
# Import with current IPs (get IDs from Hetzner Cloud Console)
terraform import 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv4[0]' <ipv4-id>
terraform import 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv6[0]' <ipv6-id>
terraform apply
```
#### Version Requirements
- Minimum Terraform version: `1.10.1`
- Minimum hcloud provider version: `1.59.0`
### 🚀 New Features
- **Hetzner Robot Integration** - Manage dedicated Robot servers via vSwitch and Cloud Controller Manager. New variables: `robot_ccm_enabled`, `robot_user`, `robot_password`, `vswitch_id`, `vswitch_subnet_index` (#1916)
- **Audit Logging** - Kubernetes audit logs with configurable policy via `k3s_audit_policy_config` and log rotation settings (#1825)
- **Control Plane Endpoint** - New `control_plane_endpoint` variable for stable external API server endpoint (e.g., external load balancers) (#1911)
- **NAT Router Control Plane Access** - Automatic port 6443 forwarding on NAT router when `control_plane_lb_enable_public_interface` is false (#2015)
- **Smaller Networks** - New `subnet_amount` variable enables networks smaller than /16 (#1971)
- **Custom Subnet Ranges** - Added `subnet_ip_range` to agent_nodepools for manual CIDR assignment (#1903)
- **Autoscaler Swap/ZRAM** - Added `swap_size` and `zram_size` support for autoscaler node pools (#2008)
- **Autoscaler Resources** - New `cluster_autoscaler_replicas`, `cluster_autoscaler_resource_limits`, `cluster_autoscaler_resource_values` (#2025)
- **Flannel Backend** - New `flannel_backend` variable to override flannel backend (wireguard-native, host-gw, etc.)
- **Cilium XDP Acceleration** - New `cilium_loadbalancer_acceleration_mode` variable (native, best-effort, disabled)
- **K3s v1.35 Support** - Added support for k3s v1.35 channel (#2029)
- **Packer Enhancements** - Configurable `kernel_type`, `sysctl_config_file`, and `timezone` for MicroOS snapshots (#2009, #2010)
### 🐛 Bug Fixes
- **Audit Policy Bastion Connection** _(v2.19.1)_ - Fixed missing bastion SSH settings in `audit_policy` provisioner, enabling audit policy deployment for NAT router / private network setups (#2042)
- **Longhorn Hotfix Tag Guidance** - Clarified `longhorn_version` as chart version and documented `longhorn_merge_values` for targeted Longhorn image hotfix tags (e.g. manager/instance-manager) (#2054)
- **Traefik v34 Compatibility** - Fixed HTTP to HTTPS redirection config for Traefik Helm Chart v34+ (#2028)
- **NAT Router IP Drift** - Fixed infinite replacement cycle by migrating from deprecated `datacenter` to `location` (#2021)
- **SELinux YAML Parsing** - Fixed cloud-init SCHEMA_ERROR caused by improper YAML formatting of SELinux policy
- **SELinux Missing Rules** - Added rules for JuiceFS (sock_file write) and SigNoz (blk_file getattr)
- **Kured Version Null** - Fixed potential null value issues with `kured_version` logic (#2032)
### 🔧 Changes
- **Default K3s Version** - Bumped from v1.31 to v1.33 (#2030)
- **Default System Upgrade Controller** - Bumped to v0.18.0
- **SELinux Policy Extraction** - Moved to dedicated template file for maintainability
- **terraform_data Migration** - Migrated from null_resource to terraform_data with automatic state migration (#1548)
- **remote-exec Refactor** - Improved provisioner compatibility with Terraform Stacks (#1893)
- **Custom GPT Updated** - [KH Assistant](https://chatgpt.com/g/g-67df95cd1e0c8191baedfa3179061581-kh-assistant) updated with v2.19.0 features, improved knowledge base, and cost calculator
---
## [2.19.0] - 2026-02-01
_Initial release of the v2.19 series. See above for full feature list._
---
## [2.18.5] - 2026-01-15
_See [GitHub releases](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/releases) for earlier versions._
================================================
FILE: LICENSE
================================================
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: README.md
================================================
<div align="center">
<!-- HERO SECTION -->
<img src="https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/kube-hetzner-logo.png" alt="Kube-Hetzner Logo" width="140" height="140">
# Kube-Hetzner
### Production-Ready Kubernetes on Hetzner Cloud
**HA by default • Auto-upgrading • Cost-optimized**
A highly optimized, easy-to-use, auto-upgradable Kubernetes cluster powered by k3s on MicroOS<br>deployed for peanuts on [Hetzner Cloud](https://hetzner.com)
[](https://terraform.io)
[](https://opentofu.org)
[](https://k3s.io)
[](LICENSE)
[](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/stargazers)
---
<table>
<tr>
<td width="50%" valign="top">
**💖 Love this project?**<br>
<a href="https://github.com/sponsors/mysticaltech">Become a sponsor</a> to help fund<br>maintenance and new features!
</td>
<td width="50%" valign="top">
**🤖 KH Assistant**<br>
<a href="https://chatgpt.com/g/g-67df95cd1e0c8191baedfa3179061581-kh-assistant">Custom GPT</a> or <code>/kh-assistant</code> <a href="https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/tree/master/.claude/skills/kh-assistant">skill</a><br>
AI-powered config generation & debugging!
</td>
</tr>
</table>
---
[Getting Started](#-getting-started) •
[Features](#-features) •
[Usage](#-usage) •
[Examples](#-examples) •
[Contributing](#-contributing)
</div>
---
## 📖 About The Project
[Hetzner Cloud](https://hetzner.com) offers exceptional value with data centers across Europe and the US. This project creates a **highly optimized Kubernetes installation** that's easy to maintain, secure, and automatically upgrades both nodes and Kubernetes—functionality similar to GKE's Auto-Pilot.
> *We are not Hetzner affiliates, but we strive to be the optimal solution for deploying Kubernetes on their platform.*
Built on the shoulders of giants:
- **[openSUSE MicroOS](https://en.opensuse.org/Portal:MicroOS)** — Immutable container OS with automatic updates
- **[k3s](https://k3s.io/)** — Certified, lightweight Kubernetes distribution
<div align="center">
<img src="https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/kubectl-pod-all-17022022.png" alt="Kube-Hetzner Screenshot" width="700">
</div>
### Why MicroOS over Ubuntu?
| Feature | Benefit |
|---------|---------|
| **Immutable filesystem** | Most of the OS is read-only—hardened by design |
| **Auto-ban abusive IPs** | SSH brute-force protection out of the box |
| **Rolling release** | Piggybacks on openSUSE Tumbleweed—always current |
| **BTRFS snapshots** | Automatic rollback if updates break something |
| **[Kured](https://github.com/kubereboot/kured) support** | Safe, HA-aware node reboots |
### Why k3s?
| Feature | Benefit |
|---------|---------|
| **Certified Kubernetes** | Automatically synced with upstream k8s |
| **Single binary** | Deploy with one command |
| **Batteries included** | Built-in [helm-controller](https://github.com/k3s-io/helm-controller) |
| **Easy upgrades** | Via [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller) |
---
## ✨ Features
<table>
<tr>
<td width="50%" valign="top">
### 🚀 Core Platform
- [x] **Maintenance-free** — Auto-upgrades MicroOS & k3s with rollback
- [x] **Multi-architecture** — Mix x86 and ARM (CAX) for cost savings
- [x] **Private networking** — Secure, low-latency node communication
- [x] **SELinux hardened** — Pre-configured security policies
### 🌐 Networking & CNI
- [x] **CNI flexibility** — Flannel, Calico, or Cilium
- [x] **Cilium XDP** — Hardware-level load balancing
- [x] **Wireguard encryption** — Optional encrypted overlay
- [x] **Dual-stack** — Full IPv4 & IPv6 support
- [x] **Custom subnets** — Define CIDR blocks per nodepool
### ⚖️ Load Balancing
- [x] **Ingress controllers** — Traefik, Nginx, or HAProxy
- [x] **Proxy Protocol** — Preserve client IPs
- [x] **Flexible LB** — Hetzner LB or Klipper
</td>
<td width="50%" valign="top">
### 🔄 High Availability
- [x] **HA by default** — 3 control-planes + 2 agents across AZs
- [x] **Super-HA** — Span multiple Hetzner locations
- [x] **Cluster autoscaler** — Automatic node scaling
- [x] **Single-node mode** — Perfect for development
### 💾 Storage
- [x] **Hetzner CSI** — Native block storage with encryption
- [x] **Longhorn** — Distributed storage with replication
- [x] **Custom mount paths** — Configurable storage locations
### 🔒 Security & Operations
- [x] **Audit logging** — Configurable retention policies
- [x] **Firewall rules** — Granular SSH/API access control
- [x] **NAT router** — Fully private clusters
- [x] **190+ variables** — Fine-tune everything
- [x] **Kustomization** — Extend with custom manifests
</td>
</tr>
</table>
---
## 🏁 Getting Started
### Prerequisites
<table>
<tr>
<th>Platform</th>
<th>Installation Command</th>
</tr>
<tr>
<td><strong>Homebrew</strong> (macOS/Linux)</td>
<td><code>brew install hashicorp/tap/terraform hashicorp/tap/packer kubectl hcloud</code></td>
</tr>
<tr>
<td><strong>Arch Linux</strong></td>
<td><code>yay -S terraform packer kubectl hcloud</code></td>
</tr>
<tr>
<td><strong>Debian/Ubuntu</strong></td>
<td><code>sudo apt install terraform packer kubectl</code></td>
</tr>
<tr>
<td><strong>Fedora/RHEL</strong></td>
<td><code>sudo dnf install terraform packer kubectl</code></td>
</tr>
<tr>
<td><strong>Windows</strong></td>
<td><code>choco install terraform packer kubernetes-cli hcloud</code></td>
</tr>
</table>
> **Required tools:** [terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) or [tofu](https://opentofu.org/docs/intro/install/), [packer](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli#installing-packer) (initial setup only), [kubectl](https://kubernetes.io/docs/tasks/tools/), [hcloud](https://github.com/hetznercloud/cli)
---
### ⚡ Quick Start
<table>
<tr>
<td>1️⃣</td>
<td><strong>Create a Hetzner project</strong> at <a href="https://console.hetzner.cloud/">console.hetzner.cloud</a> and grab an API token (Read & Write)</td>
</tr>
<tr>
<td>2️⃣</td>
<td><strong>Generate an SSH key pair</strong> (passphrase-less ed25519) — or see <a href="docs/ssh.md">SSH options</a></td>
</tr>
<tr>
<td>3️⃣</td>
<td><strong>Run the setup script</strong> — creates your project folder and MicroOS snapshot:</td>
</tr>
</table>
```sh
tmp_script=$(mktemp) && curl -sSL -o "${tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x "${tmp_script}" && "${tmp_script}" && rm "${tmp_script}"
```
<details>
<summary><strong>Fish shell version</strong></summary>
```fish
set tmp_script (mktemp); curl -sSL -o "{tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh; chmod +x "{tmp_script}"; bash "{tmp_script}"; rm "{tmp_script}"
```
</details>
<details>
<summary><strong>Save as alias for future use</strong></summary>
```sh
alias createkh='tmp_script=$(mktemp) && curl -sSL -o "${tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x "${tmp_script}" && "${tmp_script}" && rm "${tmp_script}"'
```
</details>
<details>
<summary><strong>What the script does</strong></summary>
```sh
mkdir /path/to/your/new/folder
cd /path/to/your/new/folder
curl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/kube.tf.example -o kube.tf
curl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/packer-template/hcloud-microos-snapshots.pkr.hcl -o hcloud-microos-snapshots.pkr.hcl
export HCLOUD_TOKEN="your_hcloud_token"
packer init hcloud-microos-snapshots.pkr.hcl
packer build hcloud-microos-snapshots.pkr.hcl
hcloud context create <project-name>
```
</details>
<table>
<tr>
<td>4️⃣</td>
<td><strong>Customize your <code>kube.tf</code></strong> — full reference in <a href="docs/terraform.md">terraform.md</a></td>
</tr>
</table>
---
### 🎯 Deploy
```sh
cd <your-project-folder>
terraform init --upgrade
terraform validate
terraform apply -auto-approve
```
**~5 minutes later:** Your cluster is ready! 🎉
> ⚠️ Once Terraform manages your cluster, avoid manual changes in the Hetzner UI. Use `hcloud` CLI to inspect resources.
---
## 🔧 Usage
View cluster details:
```sh
terraform output kubeconfig
terraform output -json kubeconfig | jq
```
### Connect via SSH
```sh
ssh root@<control-plane-ip> -i /path/to/private_key -o StrictHostKeyChecking=no
```
Restrict SSH access by configuring `firewall_ssh_source` in your kube.tf. See [SSH docs](docs/ssh.md#firewall-ssh-source-and-changing-ips) for dynamic IP handling.
### Connect via Kube API
```sh
kubectl --kubeconfig clustername_kubeconfig.yaml get nodes
```
Or set it as your default:
```sh
export KUBECONFIG=/<path-to>/clustername_kubeconfig.yaml
```
> **Tip:** If `create_kubeconfig = false`, generate it manually: `terraform output --raw kubeconfig > clustername_kubeconfig.yaml`
---
## 🌐 CNI Options
Default is **Flannel**. Switch by setting `cni_plugin` to `"calico"` or `"cilium"`.
### Cilium Configuration
Customize via `cilium_values` with [Cilium helm values](https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml).
| Feature | Variable |
|---------|----------|
| Full kube-proxy replacement | `disable_kube_proxy = true` |
| Hubble observability | `cilium_hubble_enabled = true` |
Access Hubble UI:
```sh
kubectl port-forward -n kube-system service/hubble-ui 12000:80
# or with Cilium CLI:
cilium hubble ui
```
---
## 📈 Scaling
### Manual Scaling
Adjust `count` in any nodepool and run `terraform apply`. Constraints:
- First control-plane nodepool minimum: **1**
- Drain nodes before removing: `kubectl drain <node-name>`
- Only remove nodepools from the **end** of the list
- Rename nodepools only when count is **0**
**Advanced:** Replace `count` with a `nodes` map for individual node control—see `kube.tf.example`.
### Autoscaling
Enable with `autoscaler_nodepools`. Powered by [Cluster Autoscaler](https://github.com/kubernetes/autoscaler).
> ⚠️ Autoscaled nodes use a snapshot from the initial control plane. Ensure disk sizes match.
---
## 🛡️ High Availability
Default: **3 control-planes + 3 agents** with automatic upgrades.
| Control Planes | Recommendation |
|----------------|----------------|
| 3+ (odd numbers) | Full HA with quorum maintenance |
| 2 | Disable auto OS upgrades, manual maintenance |
| 1 | Development only, disable auto upgrades |
See [Rancher's HA documentation](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/).
---
## 🔄 Automatic Upgrades
### OS Upgrades (MicroOS)
Handled by [Kured](https://github.com/kubereboot/kured)—safe, HA-aware reboots. Configure timeframes via [Kured options](https://kured.dev/docs/configuration/).
### K3s Upgrades
Managed by [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller). Customize the [upgrade plan template](templates/plans.yaml.tpl).
### Disable Automatic Upgrades
```tf
# Disable OS upgrades (required for <3 control planes)
automatically_upgrade_os = false
# Disable k3s upgrades
automatically_upgrade_k3s = false
```
<details>
<summary><strong>Manual upgrade commands</strong></summary>
**Selective k3s upgrade:**
```sh
kubectl label --overwrite node <node-name> k3s_upgrade=true
kubectl label node <node-name> k3s_upgrade- # disable
```
**Or delete upgrade plans:**
```sh
kubectl delete plan k3s-agent -n system-upgrade
kubectl delete plan k3s-server -n system-upgrade
```
**Manual OS upgrade:**
```sh
kubectl drain <node-name>
ssh root@<node-ip>
systemctl start transactional-update.service
reboot
```
</details>
### Component Upgrades
Use the `kustomization_backup.yaml` file created during installation:
1. Copy to `kustomization.yaml`
2. Update source URLs to latest versions
3. Apply: `kubectl apply -k ./`
---
## ⚙️ Customization
Most components use [Helm Chart](https://rancher.com/docs/k3s/latest/en/helm/) definitions via k3s Helm Controller.
See `kube.tf.example` for examples.
---
## 🖥️ Dedicated Servers
Integrate Hetzner Robot servers via [the dedicated server guide](docs/add-robot-server.md).
---
## ➕ Adding Extras
Use [Kustomize](https://kustomize.io) for additional deployments:
1. Create `extra-manifests/kustomization.yaml.tpl`
2. Supports Terraform templating via `extra_kustomize_parameters`
3. Applied after cluster setup with `kubectl apply -k`
Change folder name with `extra_kustomize_folder`. See [example](examples/kustomization_user_deploy).
---
## 📚 Examples
<details>
<summary><strong>Custom post-install actions (ArgoCD, etc.)</strong></summary>
For CRD-dependent applications:
```tf
extra_kustomize_deployment_commands = <<-EOT
kubectl -n argocd wait --for condition=established --timeout=120s crd/appprojects.argoproj.io
kubectl -n argocd wait --for condition=established --timeout=120s crd/applications.argoproj.io
kubectl apply -f /var/user_kustomize/argocd-projects.yaml
kubectl apply -f /var/user_kustomize/argocd-application-argocd.yaml
EOT
```
</details>
<details>
<summary><strong>Useful Cilium commands</strong></summary>
```sh
# Status
kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium status --verbose
# Monitor traffic
kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium monitor
# List services
kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium service list
```
[Full Cilium cheatsheet](https://docs.cilium.io/en/latest/cheatsheet)
</details>
<details>
<summary><strong>Cilium Egress Gateway with Floating IPs</strong></summary>
Control outgoing traffic with static IPs:
```tf
{
name = "egress",
server_type = "cx23",
location = "nbg1",
labels = ["node.kubernetes.io/role=egress"],
taints = ["node.kubernetes.io/role=egress:NoSchedule"],
floating_ip = true,
count = 1
}
```
Configure Cilium:
```tf
locals {
cluster_ipv4_cidr = "10.42.0.0/16"
}
cluster_ipv4_cidr = local.cluster_ipv4_cidr
cilium_values = <<-EOT
ipam:
mode: kubernetes
k8s:
requireIPv4PodCIDR: true
kubeProxyReplacement: true
routingMode: native
ipv4NativeRoutingCIDR: "10.0.0.0/8"
endpointRoutes:
enabled: true
loadBalancer:
acceleration: native
bpf:
masquerade: true
egressGateway:
enabled: true
MTU: 1450
EOT
```
Example policy:
```yaml
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: egress-sample
spec:
selectors:
- podSelector:
matchLabels:
org: empire
class: mediabot
io.kubernetes.pod.namespace: default
destinationCIDRs:
- "0.0.0.0/0"
excludedCIDRs:
- "10.0.0.0/8"
egressGateway:
nodeSelector:
matchLabels:
node.kubernetes.io/role: egress
egressIP: { FLOATING_IP }
```
[Full Egress Gateway docs](https://docs.cilium.io/en/stable/network/egress-gateway/)
</details>
<details>
<summary><strong>TLS with Cert-Manager (recommended)</strong></summary>
Cert-Manager handles HA certificate management (Traefik CE is stateless).
1. [Configure your issuer](https://cert-manager.io/docs/configuration/acme/)
2. Add annotations to Ingress:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
tls:
- hosts:
- "*.example.com"
secretName: example-com-letsencrypt-tls
rules:
- host: "*.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service
port:
number: 80
```
[Full Traefik + Cert-Manager guide](https://traefik.io/blog/secure-web-applications-with-traefik-proxy-cert-manager-and-lets-encrypt/)
> **Ingress-Nginx with HTTP challenge:** Add `lb_hostname = "cluster.example.org"` to work around [this known issue](https://github.com/cert-manager/cert-manager/issues/466).
</details>
<details>
<summary><strong>Managing snapshots</strong></summary>
**Create:**
```sh
export HCLOUD_TOKEN=<your-token>
packer build ./packer-template/hcloud-microos-snapshots.pkr.hcl
```
**Delete:**
```sh
hcloud image list
hcloud image delete <image-id>
```
</details>
<details>
<summary><strong>Single-node development cluster</strong></summary>
Set `automatically_upgrade_os = false` (attached volumes don't handle auto-reboots well).
Uses k3s [service load balancer](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer) instead of external LB. Ports 80 & 443 open automatically.
</details>
<details>
<summary><strong>Terraform Cloud deployment</strong></summary>
1. Create MicroOS snapshot in your project first
2. Configure SSH keys as Terraform Cloud variables (mark private key as sensitive):
```tf
ssh_public_key = var.ssh_public_key
ssh_private_key = var.ssh_private_key
```
> **Password-protected keys:** Requires `local` execution mode with your own agent.
</details>
<details>
<summary><strong>HelmChartConfig customization</strong></summary>
```yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rancher
namespace: kube-system
spec:
valuesContent: |-
# Your values.yaml customizations here
```
Works for all add-ons: Longhorn, Cert-manager, Traefik, etc.
</details>
<details>
<summary><strong>Encryption at rest (HCloud CSI)</strong></summary>
Create secret:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: encryption-secret
namespace: kube-system
stringData:
encryption-passphrase: foobar
```
Create storage class:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hcloud-volumes-encrypted
provisioner: csi.hetzner.cloud
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
csi.storage.k8s.io/node-publish-secret-name: encryption-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
```
</details>
<details>
<summary><strong>Encryption at rest (Longhorn)</strong></summary>
Create secret:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: longhorn-crypto
namespace: longhorn-system
stringData:
CRYPTO_KEY_VALUE: "your-encryption-key"
CRYPTO_KEY_PROVIDER: "secret"
CRYPTO_KEY_CIPHER: "aes-xts-plain64"
CRYPTO_KEY_HASH: "sha256"
CRYPTO_KEY_SIZE: "256"
CRYPTO_PBKDF: "argon2i"
```
Create storage class:
```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: longhorn-crypto-global
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
nodeSelector: "node-storage"
numberOfReplicas: "1"
staleReplicaTimeout: "2880"
fromBackup: ""
fsType: ext4
encrypted: "true"
csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"
```
[Longhorn encryption docs](https://longhorn.io/docs/1.4.0/advanced-resources/security/volume-encryption/)
</details>
<details>
<summary><strong>Namespace-based architecture assignment</strong></summary>
Enable admission controllers:
```tf
k3s_exec_server_args = "--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector"
```
Assign namespace to architecture:
```yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/arch=amd64
name: this-runs-on-amd64
```
With tolerations:
```yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/arch=arm64
scheduler.alpha.kubernetes.io/defaultTolerations: '[{ "operator" : "Equal", "effect" : "NoSchedule", "key" : "workload-type", "value" : "machine-learning" }]'
name: this-runs-on-arm64
```
</details>
<details>
<summary><strong>Backup and restore cluster (etcd S3)</strong></summary>
**Setup backup:**
1. Configure `etcd_s3_backup` in kube.tf
2. Add k3s_token output:
```tf
output "k3s_token" {
value = module.kube-hetzner.k3s_token
sensitive = true
}
```
**Restore:**
1. Add restoration config to kube.tf:
```tf
locals {
k3s_token = var.k3s_token
etcd_version = "v3.5.9"
etcd_snapshot_name = "name-of-the-snapshot"
etcd_s3_endpoint = "your-s3-endpoint"
etcd_s3_bucket = "your-s3-bucket"
etcd_s3_access_key = "your-s3-access-key"
etcd_s3_secret_key = var.etcd_s3_secret_key
}
variable "k3s_token" {
sensitive = true
type = string
}
variable "etcd_s3_secret_key" {
sensitive = true
type = string
}
module "kube-hetzner" {
k3s_token = local.k3s_token
postinstall_exec = compact([
(
local.etcd_snapshot_name == "" ? "" :
<<-EOF
export CLUSTERINIT=$(cat /etc/rancher/k3s/config.yaml | grep -i '"cluster-init": true')
if [ -n "$CLUSTERINIT" ]; then
k3s server \
--cluster-reset \
--etcd-s3 \
--cluster-reset-restore-path=${local.etcd_snapshot_name} \
--etcd-s3-endpoint=${local.etcd_s3_endpoint} \
--etcd-s3-bucket=${local.etcd_s3_bucket} \
--etcd-s3-access-key=${local.etcd_s3_access_key} \
--etcd-s3-secret-key=${local.etcd_s3_secret_key}
mv /etc/rancher/k3s/k3s.yaml /etc/rancher/k3s/k3s.backup.yaml
ETCD_VER=${local.etcd_version}
case "$(uname -m)" in
aarch64) ETCD_ARCH="arm64" ;;
x86_64) ETCD_ARCH="amd64" ;;
esac;
DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download
curl -L $DOWNLOAD_URL/$ETCD_VER/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz -o /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz
tar xzvf /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz -C /usr/local/bin --strip-components=1
nohup etcd --data-dir /var/lib/rancher/k3s/server/db/etcd &
echo $! > save_pid.txt
etcdctl del /registry/services/specs/traefik/traefik
etcdctl del /registry/services/endpoints/traefik/traefik
OLD_NODES=$(etcdctl get "" --prefix --keys-only | grep /registry/minions/ | cut -c 19-)
for NODE in $OLD_NODES; do
for KEY in $(etcdctl get "" --prefix --keys-only | grep $NODE); do
etcdctl del $KEY
done
done
kill -9 `cat save_pid.txt`
rm save_pid.txt
fi
EOF
)
])
}
```
2. Set environment variables:
```sh
export TF_VAR_k3s_token="..."
export TF_VAR_etcd_s3_secret_key="..."
```
3. Run `terraform apply`
</details>
<details>
<summary><strong>Pre-constructed private network (proxies)</strong></summary>
```tf
resource "hcloud_network" "k3s_proxied" {
name = "k3s-proxied"
ip_range = "10.0.0.0/8"
}
resource "hcloud_network_subnet" "k3s_proxy" {
network_id = hcloud_network.k3s_proxied.id
type = "cloud"
network_zone = "eu-central"
ip_range = "10.128.0.0/9"
}
resource "hcloud_server" "your_proxy_server" { ... }
resource "hcloud_server_network" "your_proxy_server" {
depends_on = [hcloud_server.your_proxy_server]
server_id = hcloud_server.your_proxy_server.id
network_id = hcloud_network.k3s_proxied.id
ip = "10.128.0.1"
}
module "kube-hetzner" {
existing_network_id = [hcloud_network.k3s_proxied.id] # Note: brackets required!
network_ipv4_cidr = "10.0.0.0/9"
additional_k3s_environment = {
"http_proxy" : "http://10.128.0.1:3128",
"HTTP_PROXY" : "http://10.128.0.1:3128",
"HTTPS_PROXY" : "http://10.128.0.1:3128",
"CONTAINERD_HTTP_PROXY" : "http://10.128.0.1:3128",
"CONTAINERD_HTTPS_PROXY" : "http://10.128.0.1:3128",
"NO_PROXY" : "127.0.0.0/8,10.0.0.0/8,",
}
}
```
</details>
<details>
<summary><strong>Placement groups</strong></summary>
Assign nodepools to placement groups:
```tf
agent_nodepools = [
{
...
placement_group = "special"
},
]
```
Legacy compatibility:
```tf
placement_group_compat_idx = 1
```
For >10 nodes, use map-based definition:
```tf
agent_nodepools = [
{
nodes = {
"0" : { placement_group = "pg-1" },
"30" : { placement_group = "pg-2" },
}
},
]
```
Disable globally: `placement_group_disable = true`
</details>
<details>
<summary><strong>Migrating from count to map-based nodes</strong></summary>
Set `append_index_to_node_name = false` to avoid node replacement:
```tf
agent_nodepools = [
{
name = "agent-large",
server_type = "cx33",
location = "nbg1",
labels = [],
taints = [],
nodes = {
"0" : {
append_index_to_node_name = false,
labels = ["my.extra.label=special"],
placement_group = "agent-large-pg-1",
},
"1" : {
append_index_to_node_name = false,
server_type = "cx43",
labels = ["my.extra.label=slightlybiggernode"],
placement_group = "agent-large-pg-2",
},
}
},
]
```
</details>
<details>
<summary><strong>Delete protection</strong></summary>
Protect resources from accidental deletion via Hetzner Console/API:
```tf
enable_delete_protection = {
floating_ip = true
load_balancer = true
volume = true
}
```
> Note: Terraform can still delete resources (provider lifts the lock).
</details>
<details>
<summary><strong>Private-only cluster (Wireguard)</strong></summary>
Requirements:
1. Pre-configured network
2. NAT gateway with public IP ([Hetzner guide](https://community.hetzner.com/tutorials/how-to-set-up-nat-for-cloud-networks))
3. Wireguard VPN access ([Hetzner guide](https://docs.hetzner.com/cloud/apps/list/wireguard/))
4. Route `0.0.0.0/0` through NAT gateway
Configuration:
```tf
existing_network_id = [YOURID]
network_ipv4_cidr = "10.0.0.0/9"
# In all nodepools:
disable_ipv4 = true
disable_ipv6 = true
# For autoscaler:
autoscaler_disable_ipv4 = true
autoscaler_disable_ipv6 = true
# Optional private LB:
control_plane_lb_enable_public_interface = false
```
</details>
<details>
<summary><strong>Private-only cluster (NAT Router)</strong></summary>
Fully private setup with:
- **Egress:** Single NAT router IP
- **SSH:** Through bastion (NAT router)
- **Control plane:** Through LB or NAT router port forwarding
- **Ingress:** Through agents LB only
> **August 11, 2025:** Hetzner removed legacy Router DHCP option. This module now automatically persists routes via the virtual gateway.
</details>
<details>
<summary><strong>Fix SELinux issues with udica</strong></summary>
Create targeted SELinux profiles instead of weakening cluster-wide security:
```sh
# Find container
crictl ps
# Generate inspection
crictl inspect <container-id> > container.json
# Create profile
udica -j container.json myapp --full-network-access
# Install module
semodule -i myapp.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}
```
Apply to deployment:
```yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: my-container
securityContext:
seLinuxOptions:
type: myapp.process
```
*Thanks @carolosf*
</details>
---
## 🔍 Debugging
### Quick Status Check
```sh
hcloud context create Kube-hetzner # First time only
hcloud server list # Check nodes
hcloud network describe k3s # Check network
hcloud loadbalancer describe k3s-traefik # Check LB
```
### SSH Troubleshooting
```sh
ssh root@<control-plane-ip> -i /path/to/private_key -o StrictHostKeyChecking=no
# View k3s logs
journalctl -u k3s # Control plane
journalctl -u k3s-agent # Agent nodes
# Check config
cat /etc/rancher/k3s/config.yaml
# Check uptime
last reboot
uptime
```
---
## 💣 Takedown
```sh
terraform destroy -auto-approve
```
**If destroy hangs** (LB or autoscaled nodes), use the cleanup script:
```sh
tmp_script=$(mktemp) && curl -sSL -o "${tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/cleanup.sh && chmod +x "${tmp_script}" && "${tmp_script}" && rm "${tmp_script}"
```
> ⚠️ This deletes everything including volumes. Dry-run option available.
---
## ⬆️ Upgrading the Module
Update `version` in your kube.tf and run `terraform apply`.
### Migrating from 1.x to 2.x
1. Run `createkh` to get new packer template
2. Update version to `>= 2.0`
3. Remove `extra_packages_to_install` and `opensuse_microos_mirror_link` (moved to packer)
4. Run `terraform init -upgrade && terraform apply`
---
## 🤝 Contributing
**Help wanted!** Consider asking Hetzner to add MicroOS as a default image (not just ISO) at [get.opensuse.org/microos](https://get.opensuse.org/microos). More requests = faster deployments for everyone!
### Development Workflow
1. Fork the project
2. Create your branch: `git checkout -b AmazingFeature`
3. Point your kube.tf `source` to local clone
4. Useful commands:
```sh
../kube-hetzner/scripts/cleanup.sh
packer build ../kube-hetzner/packer-template/hcloud-microos-snapshots.pkr.hcl
```
5. Update `kube.tf.example` if needed
6. Commit: `git commit -m 'Add AmazingFeature'`
7. Push: `git push origin AmazingFeature`
8. Open PR targeting `staging` branch
### Agent Skills
This project includes [agent skills](https://agentskills.io) in `.claude/skills/` — reusable workflows for any AI coding agent (Claude Code, Cursor, Windsurf, Codex, etc.):
| Skill | Purpose |
|-------|---------|
| `/kh-assistant` | Interactive help for configuration and debugging |
| `/fix-issue <num>` | Guided workflow for fixing GitHub issues |
| `/review-pr <num>` | Security-focused PR review |
| `/test-changes` | Run terraform fmt, validate, plan |
**PRs to improve these skills are welcome!** See `.claude/skills/` for the skill definitions.
---
## 💖 Support This Project
<div align="center">
If Kube-Hetzner saves you time and money, please consider supporting its development:
<a href="https://github.com/sponsors/mysticaltech">
<img src="https://img.shields.io/badge/Sponsor_on_GitHub-❤️-EA4AAA?style=for-the-badge&logo=github-sponsors" alt="Sponsor on GitHub">
</a>
<br><br>
Your sponsorship directly funds:
🐛 **Bug fixes** and issue response<br>
🚀 **New features** and improvements<br>
📚 **Documentation** maintenance<br>
🔒 **Security updates** and best practices
**Every contribution matters.** Thank you for keeping this project alive! 🙏
</div>
---
## 🙏 Acknowledgements
<div align="center">
<a href="https://www.hetzner.com"><img src="https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/hetzner-logo.svg" alt="Hetzner — Server · Cloud · Hosting" height="80"></a>
<br><br>
</div>
Thanks to **[Hetzner](https://www.hetzner.com)** for supporting this project with cloud credits.
---
<div align="center">
**[⬆ Back to Top](#kube-hetzner)**
Made with ❤️ by the Kube-Hetzner community
</div>
================================================
FILE: SECURITY.md
================================================
# Security Policy
## Reporting a Vulnerability
Please report any vulnerability findings privately via email to the top three contributors to the projects. You can find our emails by grepping the git logs.
In case you can't find the emails:
- [aleksasiriski](https://github.com/aleksasiriski): [sir@tmina.org](mailto:kube-hetzner@sir.tmina.org)
================================================
FILE: agents.tf
================================================
module "agents" {
source = "./modules/host"
providers = {
hcloud = hcloud,
}
for_each = local.agent_nodes
name = "${var.use_cluster_name_in_node_name ? "${var.cluster_name}-" : ""}${each.value.nodepool_name}${try(each.value.node_name_suffix, "")}"
microos_snapshot_id = substr(each.value.server_type, 0, 3) == "cax" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id
base_domain = var.base_domain
ssh_keys = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]
ssh_port = var.ssh_port
ssh_public_key = var.ssh_public_key
ssh_private_key = var.ssh_private_key
ssh_additional_public_keys = length(var.ssh_hcloud_key_label) > 0 ? concat(var.ssh_additional_public_keys, data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.public_key) : var.ssh_additional_public_keys
firewall_ids = each.value.disable_ipv4 && each.value.disable_ipv6 ? [] : [hcloud_firewall.k3s.id] # Cannot attach a firewall when public interfaces are disabled
placement_group_id = var.placement_group_disable ? null : (each.value.placement_group == null ? hcloud_placement_group.agent[each.value.placement_group_compat_idx].id : hcloud_placement_group.agent_named[each.value.placement_group].id)
location = each.value.location
server_type = each.value.server_type
backups = each.value.backups
ipv4_subnet_id = hcloud_network_subnet.agent[[for i, v in var.agent_nodepools : i if v.name == each.value.nodepool_name][0]].id
dns_servers = var.dns_servers
k3s_registries = var.k3s_registries
k3s_registries_update_script = local.k3s_registries_update_script
cloudinit_write_files_common = local.cloudinit_write_files_common
k3s_kubelet_config = var.k3s_kubelet_config
k3s_kubelet_config_update_script = local.k3s_kubelet_config_update_script
k3s_audit_policy_config = ""
k3s_audit_policy_update_script = ""
cloudinit_runcmd_common = local.cloudinit_runcmd_common
swap_size = each.value.swap_size
zram_size = each.value.zram_size
keep_disk_size = var.keep_disk_agents
disable_ipv4 = each.value.disable_ipv4
disable_ipv6 = each.value.disable_ipv6
ssh_bastion = local.ssh_bastion
network_id = data.hcloud_network.k3s.id
private_ipv4 = cidrhost(hcloud_network_subnet.agent[[for i, v in var.agent_nodepools : i if v.name == each.value.nodepool_name][0]].ip_range, each.value.index + (local.network_size >= 16 ? 101 : floor(pow(local.subnet_size, 2) * 0.4)))
labels = merge(local.labels, local.labels_agent_node)
automatically_upgrade_os = var.automatically_upgrade_os
network_gw_ipv4 = local.network_gw_ipv4
depends_on = [
hcloud_network_subnet.agent,
hcloud_placement_group.agent,
hcloud_server.nat_router,
terraform_data.nat_router_await_cloud_init,
]
}
locals {
k3s-agent-config = { for k, v in local.agent_nodes : k => merge(
{
node-name = module.agents[k].name
server = local.k3s_endpoint
token = local.k3s_token
# Kubelet arg precedence (last wins): local.kubelet_arg > v.kubelet_args > k3s_global_kubelet_args > k3s_agent_kubelet_args
kubelet-arg = concat(
local.kubelet_arg,
v.kubelet_args,
var.k3s_global_kubelet_args,
var.k3s_agent_kubelet_args
)
flannel-iface = local.flannel_iface
node-ip = module.agents[k].private_ipv4_address
node-label = v.labels
node-taint = v.taints
},
var.agent_nodes_custom_config,
local.prefer_bundled_bin_config,
# Force selinux=false if disable_selinux = true.
var.disable_selinux
? { selinux = false }
: (v.selinux == true ? { selinux = true } : {})
) }
agent_ips = {
for k, v in module.agents : k => coalesce(
v.ipv4_address,
v.ipv6_address,
v.private_ipv4_address
)
}
}
resource "terraform_data" "agent_config" {
for_each = local.agent_nodes
triggers_replace = {
agent_id = module.agents[each.key].id
config = sha1(yamlencode(local.k3s-agent-config[each.key]))
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.agent_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
# Generating k3s agent config file
provisioner "file" {
content = yamlencode(local.k3s-agent-config[each.key])
destination = "/tmp/config.yaml"
}
provisioner "remote-exec" {
inline = [local.k3s_config_update_script]
}
}
moved {
from = null_resource.agent_config
to = terraform_data.agent_config
}
resource "terraform_data" "agents" {
for_each = local.agent_nodes
triggers_replace = {
agent_id = module.agents[each.key].id
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.agent_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
# Install k3s agent
provisioner "remote-exec" {
inline = local.install_k3s_agent
}
# Start the k3s agent and wait for it to have started
provisioner "remote-exec" {
inline = concat(var.enable_longhorn || var.enable_iscsid ? ["systemctl enable --now iscsid"] : [], [
"timeout 120 systemctl start k3s-agent 2> /dev/null",
<<-EOT
timeout 120 bash <<EOF
until systemctl status k3s-agent > /dev/null; do
systemctl start k3s-agent 2> /dev/null
echo "Waiting for the k3s agent to start..."
sleep 2
done
EOF
EOT
])
}
depends_on = [
terraform_data.first_control_plane,
terraform_data.agent_config,
hcloud_network_subnet.agent
]
}
moved {
from = null_resource.agents
to = terraform_data.agents
}
resource "hcloud_volume" "longhorn_volume" {
for_each = { for k, v in local.agent_nodes : k => v if((v.longhorn_volume_size >= 10) && (v.longhorn_volume_size <= 10240) && var.enable_longhorn) }
labels = {
provisioner = "terraform"
cluster = var.cluster_name
scope = "longhorn"
}
name = "${var.cluster_name}-longhorn-${module.agents[each.key].name}"
size = local.agent_nodes[each.key].longhorn_volume_size
server_id = module.agents[each.key].id
automount = true
format = var.longhorn_fstype
delete_protection = var.enable_delete_protection.volume
}
resource "terraform_data" "configure_longhorn_volume" {
for_each = { for k, v in local.agent_nodes : k => v if((v.longhorn_volume_size >= 10) && (v.longhorn_volume_size <= 10240) && var.enable_longhorn) }
triggers_replace = {
agent_id = module.agents[each.key].id
}
# Start the k3s agent and wait for it to have started
provisioner "remote-exec" {
inline = [
"set -e",
"mkdir -p '${each.value.longhorn_mount_path}' >/dev/null",
"mountpoint -q '${each.value.longhorn_mount_path}' || mount -o discard,defaults ${hcloud_volume.longhorn_volume[each.key].linux_device} '${each.value.longhorn_mount_path}'",
"${var.longhorn_fstype == "ext4" ? "resize2fs" : "xfs_growfs"} ${hcloud_volume.longhorn_volume[each.key].linux_device}",
# Match any non-comment line (^[^#]) with any first field, followed by a space and your mount path in the second column.
# This prevents false positives like /data matching /data1.
"awk -v path='${each.value.longhorn_mount_path}' '$0 !~ /^#/ && $2 == path { found=1; exit } END { exit !found }' /etc/fstab || echo '${hcloud_volume.longhorn_volume[each.key].linux_device} ${each.value.longhorn_mount_path} ${var.longhorn_fstype} discard,nofail,defaults 0 0' | tee -a /etc/fstab >/dev/null"
]
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.agent_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
depends_on = [
hcloud_volume.longhorn_volume
]
}
moved {
from = null_resource.configure_longhorn_volume
to = terraform_data.configure_longhorn_volume
}
resource "hcloud_floating_ip" "agents" {
for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, "floating_ip"), false) }
type = "ipv4"
labels = local.labels
home_location = each.value.location
delete_protection = var.enable_delete_protection.floating_ip
}
resource "hcloud_floating_ip_assignment" "agents" {
for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, "floating_ip"), false) }
floating_ip_id = hcloud_floating_ip.agents[each.key].id
server_id = module.agents[each.key].id
depends_on = [
terraform_data.agents
]
}
resource "hcloud_rdns" "agents" {
for_each = { for k, v in local.agent_nodes : k => v if lookup(v, "floating_ip_rdns", null) != null }
floating_ip_id = hcloud_floating_ip.agents[each.key].id
ip_address = hcloud_floating_ip.agents[each.key].ip_address
dns_ptr = local.agent_nodes[each.key].floating_ip_rdns
depends_on = [
hcloud_floating_ip.agents
]
}
resource "terraform_data" "configure_floating_ip" {
for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, "floating_ip"), false) }
triggers_replace = {
agent_id = module.agents[each.key].id
floating_ip_id = hcloud_floating_ip.agents[each.key].id
}
provisioner "remote-exec" {
inline = [
# Reconfigure eth0:
# - add floating_ip as first and other IP as second address
# - add 172.31.1.1 as default gateway (In the Hetzner Cloud, the
# special private IP address 172.31.1.1 is the default
# gateway for the public network)
# The configuration is stored in file /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection
<<-EOT
ETH=eth1
if ip link show eth0 &>/dev/null; then
ETH=eth0
fi
NM_CONNECTION=$(nmcli -g GENERAL.CONNECTION device show "$ETH" 2>/dev/null)
if [ -z "$NM_CONNECTION" ]; then
echo "ERROR: No NetworkManager connection found for $ETH" >&2
exit 1
fi
nmcli connection modify "$NM_CONNECTION" \
ipv4.method manual \
ipv4.addresses ${hcloud_floating_ip.agents[each.key].ip_address}/32,${local.agent_ips[each.key]}/32 gw4 172.31.1.1 \
ipv4.route-metric 100 \
&& nmcli connection up "$NM_CONNECTION"
EOT
]
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.agent_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
depends_on = [
hcloud_floating_ip_assignment.agents
]
}
moved {
from = null_resource.configure_floating_ip
to = terraform_data.configure_floating_ip
}
================================================
FILE: autoscaler-agents.tf
================================================
locals {
cluster_prefix = var.use_cluster_name_in_node_name ? "${var.cluster_name}-" : ""
first_nodepool_snapshot_id = length(var.autoscaler_nodepools) == 0 ? "" : (
substr(var.autoscaler_nodepools[0].server_type, 0, 3) == "cax" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id
)
imageList = {
arm64 : tostring(data.hcloud_image.microos_arm_snapshot.id)
amd64 : tostring(data.hcloud_image.microos_x86_snapshot.id)
}
nodeConfigName = var.use_cluster_name_in_node_name ? "${var.cluster_name}-" : ""
cluster_config = {
imagesForArch : local.imageList
nodeConfigs : {
for index, nodePool in var.autoscaler_nodepools :
("${local.nodeConfigName}${nodePool.name}") => {
cloudInit = data.cloudinit_config.autoscaler_config[index].rendered
labels = nodePool.labels
taints = nodePool.taints
}
}
}
isUsingLegacyConfig = length(var.autoscaler_labels) > 0 || length(var.autoscaler_taints) > 0
autoscaler_yaml = length(var.autoscaler_nodepools) == 0 ? "" : templatefile(
"${path.module}/templates/autoscaler.yaml.tpl",
{
cloudinit_config = local.isUsingLegacyConfig ? base64encode(data.cloudinit_config.autoscaler_legacy_config[0].rendered) : ""
ca_image = var.cluster_autoscaler_image
ca_version = var.cluster_autoscaler_version
ca_replicas = var.cluster_autoscaler_replicas
ca_resource_limits = var.cluster_autoscaler_resource_limits
ca_resources = var.cluster_autoscaler_resource_values
cluster_autoscaler_extra_args = var.cluster_autoscaler_extra_args
cluster_autoscaler_log_level = var.cluster_autoscaler_log_level
cluster_autoscaler_log_to_stderr = var.cluster_autoscaler_log_to_stderr
cluster_autoscaler_stderr_threshold = var.cluster_autoscaler_stderr_threshold
cluster_autoscaler_server_creation_timeout = tostring(var.cluster_autoscaler_server_creation_timeout)
ssh_key = local.hcloud_ssh_key_id
ipv4_subnet_id = data.hcloud_network.k3s.id
snapshot_id = local.first_nodepool_snapshot_id
cluster_config = base64encode(jsonencode(local.cluster_config))
firewall_id = hcloud_firewall.k3s.id
cluster_name = local.cluster_prefix
node_pools = var.autoscaler_nodepools
enable_ipv4 = !(var.autoscaler_disable_ipv4 || local.use_nat_router)
enable_ipv6 = !(var.autoscaler_disable_ipv6 || local.use_nat_router)
})
# A concatenated list of all autoscaled nodes
autoscaled_nodes = length(var.autoscaler_nodepools) == 0 ? {} : {
for v in concat([
for k, v in data.
hcloud_servers.autoscaled_nodes : [for v in v.servers : v]
]...) : v.name => v
}
}
resource "terraform_data" "configure_autoscaler" {
count = length(var.autoscaler_nodepools) > 0 ? 1 : 0
triggers_replace = {
template = local.autoscaler_yaml
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.first_control_plane_ip
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
# Upload the autoscaler resource defintion
provisioner "file" {
content = local.autoscaler_yaml
destination = "/tmp/autoscaler.yaml"
}
# Create/Apply the definition
provisioner "remote-exec" {
inline = ["kubectl apply -f /tmp/autoscaler.yaml"]
}
depends_on = [
hcloud_load_balancer.cluster,
terraform_data.control_planes,
random_password.rancher_bootstrap,
hcloud_volume.longhorn_volume,
data.hcloud_image.microos_x86_snapshot
]
}
moved {
from = null_resource.configure_autoscaler
to = terraform_data.configure_autoscaler
}
data "cloudinit_config" "autoscaler_config" {
count = length(var.autoscaler_nodepools)
gzip = true
base64_encode = true
# Main cloud-config configuration file.
part {
filename = "init.cfg"
content_type = "text/cloud-config"
content = templatefile(
"${path.module}/templates/autoscaler-cloudinit.yaml.tpl",
{
hostname = "autoscaler"
dns_servers = var.dns_servers
has_dns_servers = local.has_dns_servers
sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)
swap_size = var.autoscaler_nodepools[count.index].swap_size
zram_size = var.autoscaler_nodepools[count.index].zram_size
k3s_config = yamlencode(merge(
{
server = local.k3s_endpoint
token = local.k3s_token
# Kubelet arg precedence (last wins): local.kubelet_arg > nodepool.kubelet_args > k3s_global_kubelet_args > k3s_autoscaler_kubelet_args
kubelet-arg = concat(local.kubelet_arg, var.autoscaler_nodepools[count.index].kubelet_args, var.k3s_global_kubelet_args, var.k3s_autoscaler_kubelet_args)
flannel-iface = local.flannel_iface
node-label = concat(local.default_agent_labels, [for k, v in var.autoscaler_nodepools[count.index].labels : "${k}=${v}"], var.autoscaler_nodepools[count.index].swap_size != "" || var.autoscaler_nodepools[count.index].zram_size != "" ? local.swap_node_label : [])
node-taint = compact(concat(local.default_agent_taints, [for taint in var.autoscaler_nodepools[count.index].taints : "${taint.key}=${tostring(taint.value)}:${taint.effect}"]))
selinux = !var.disable_selinux
},
var.agent_nodes_custom_config,
local.prefer_bundled_bin_config
))
install_k3s_agent_script = join("\n", concat(local.install_k3s_agent, ["systemctl start k3s-agent"]))
cloudinit_write_files_common = local.cloudinit_write_files_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common,
private_network_only = var.autoscaler_disable_ipv4 && var.autoscaler_disable_ipv6,
network_gw_ipv4 = local.network_gw_ipv4
}
)
}
}
data "cloudinit_config" "autoscaler_legacy_config" {
count = length(var.autoscaler_nodepools) > 0 && local.isUsingLegacyConfig ? 1 : 0
gzip = true
base64_encode = true
# Main cloud-config configuration file.
part {
filename = "init.cfg"
content_type = "text/cloud-config"
content = templatefile(
"${path.module}/templates/autoscaler-cloudinit.yaml.tpl",
{
hostname = "autoscaler"
dns_servers = var.dns_servers
has_dns_servers = local.has_dns_servers
sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)
swap_size = ""
zram_size = ""
k3s_config = yamlencode(merge(
{
server = local.k3s_endpoint
token = local.k3s_token
kubelet-arg = local.kubelet_arg
flannel-iface = local.flannel_iface
node-label = concat(local.default_agent_labels, var.autoscaler_labels)
node-taint = compact(concat(local.default_agent_taints, var.autoscaler_taints))
selinux = !var.disable_selinux
},
var.agent_nodes_custom_config,
local.prefer_bundled_bin_config
))
install_k3s_agent_script = join("\n", concat(local.install_k3s_agent, ["systemctl start k3s-agent"]))
cloudinit_write_files_common = local.cloudinit_write_files_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common,
private_network_only = var.autoscaler_disable_ipv4 && var.autoscaler_disable_ipv6,
network_gw_ipv4 = local.network_gw_ipv4,
}
)
}
}
data "hcloud_servers" "autoscaled_nodes" {
for_each = toset(var.autoscaler_nodepools[*].name)
with_selector = "hcloud/node-group=${local.cluster_prefix}${each.value}"
}
resource "terraform_data" "autoscaled_nodes_registries" {
for_each = local.autoscaled_nodes
triggers_replace = {
registries = var.k3s_registries
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = coalesce(each.value.ipv4_address, each.value.ipv6_address, try(one(each.value.network).ip, null))
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
provisioner "file" {
content = var.k3s_registries
destination = "/tmp/registries.yaml"
}
provisioner "remote-exec" {
inline = [local.k3s_registries_update_script]
}
}
moved {
from = null_resource.autoscaled_nodes_registries
to = terraform_data.autoscaled_nodes_registries
}
resource "terraform_data" "autoscaled_nodes_kubelet_config" {
for_each = var.k3s_kubelet_config != "" ? local.autoscaled_nodes : {}
triggers_replace = {
kubelet_config = var.k3s_kubelet_config
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = coalesce(each.value.ipv4_address, each.value.ipv6_address, try(one(each.value.network).ip, null))
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
provisioner "file" {
content = var.k3s_kubelet_config
destination = "/tmp/kubelet-config.yaml"
}
provisioner "remote-exec" {
inline = [local.k3s_kubelet_config_update_script]
}
}
moved {
from = null_resource.autoscaled_nodes_kubelet_config
to = terraform_data.autoscaled_nodes_kubelet_config
}
================================================
FILE: control_planes.tf
================================================
module "control_planes" {
source = "./modules/host"
providers = {
hcloud = hcloud,
}
for_each = local.control_plane_nodes
name = "${var.use_cluster_name_in_node_name ? "${var.cluster_name}-" : ""}${each.value.nodepool_name}"
microos_snapshot_id = substr(each.value.server_type, 0, 3) == "cax" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id
base_domain = var.base_domain
ssh_keys = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]
ssh_port = var.ssh_port
ssh_public_key = var.ssh_public_key
ssh_private_key = var.ssh_private_key
ssh_additional_public_keys = length(var.ssh_hcloud_key_label) > 0 ? concat(var.ssh_additional_public_keys, data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.public_key) : var.ssh_additional_public_keys
firewall_ids = each.value.disable_ipv4 && each.value.disable_ipv6 ? [] : [hcloud_firewall.k3s.id] # Cannot attach a firewall when public interfaces are disabled
placement_group_id = var.placement_group_disable ? null : (each.value.placement_group == null ? hcloud_placement_group.control_plane[each.value.placement_group_compat_idx].id : hcloud_placement_group.control_plane_named[each.value.placement_group].id)
location = each.value.location
server_type = each.value.server_type
backups = each.value.backups
ipv4_subnet_id = hcloud_network_subnet.control_plane[[for i, v in var.control_plane_nodepools : i if v.name == each.value.nodepool_name][0]].id
dns_servers = var.dns_servers
k3s_registries = var.k3s_registries
k3s_registries_update_script = local.k3s_registries_update_script
k3s_kubelet_config = var.k3s_kubelet_config
k3s_kubelet_config_update_script = local.k3s_kubelet_config_update_script
k3s_audit_policy_config = var.k3s_audit_policy_config
k3s_audit_policy_update_script = local.k3s_audit_policy_update_script
cloudinit_write_files_common = local.cloudinit_write_files_common
cloudinit_runcmd_common = local.cloudinit_runcmd_common
swap_size = each.value.swap_size
zram_size = each.value.zram_size
keep_disk_size = var.keep_disk_cp
disable_ipv4 = each.value.disable_ipv4
disable_ipv6 = each.value.disable_ipv6
ssh_bastion = local.ssh_bastion
network_id = data.hcloud_network.k3s.id
# We leave some room so 100 eventual Hetzner LBs that can be created perfectly safely
# It leaves the subnet with 254 x 254 - 100 = 64416 IPs to use, so probably enough.
private_ipv4 = cidrhost(hcloud_network_subnet.control_plane[[for i, v in var.control_plane_nodepools : i if v.name == each.value.nodepool_name][0]].ip_range, each.value.index + (local.network_size >= 16 ? 101 : floor(pow(local.subnet_size, 2) * 0.4)))
labels = merge(local.labels, local.labels_control_plane_node)
automatically_upgrade_os = var.automatically_upgrade_os
network_gw_ipv4 = local.network_gw_ipv4
depends_on = [
hcloud_network_subnet.control_plane,
hcloud_placement_group.control_plane,
hcloud_server.nat_router,
terraform_data.nat_router_await_cloud_init,
]
}
resource "hcloud_load_balancer" "control_plane" {
count = var.use_control_plane_lb ? 1 : 0
name = "${var.cluster_name}-control-plane"
load_balancer_type = var.control_plane_lb_type
location = var.load_balancer_location
labels = merge(local.labels, local.labels_control_plane_lb)
delete_protection = var.enable_delete_protection.load_balancer
lifecycle {
ignore_changes = [location]
}
}
resource "hcloud_load_balancer_network" "control_plane" {
count = var.use_control_plane_lb ? 1 : 0
load_balancer_id = hcloud_load_balancer.control_plane.*.id[0]
subnet_id = hcloud_network_subnet.control_plane.*.id[0]
enable_public_interface = var.control_plane_lb_enable_public_interface
ip = cidrhost(hcloud_network_subnet.control_plane.*.ip_range[0], -2)
# Keep existing LB IPs stable on upgrade.
lifecycle {
ignore_changes = [ip]
}
}
resource "hcloud_load_balancer_target" "control_plane" {
count = var.use_control_plane_lb ? 1 : 0
depends_on = [hcloud_load_balancer_network.control_plane]
type = "label_selector"
load_balancer_id = hcloud_load_balancer.control_plane.*.id[0]
label_selector = join(",", [for k, v in merge(local.labels, local.labels_control_plane_node) : "${k}=${v}"])
use_private_ip = true
}
resource "hcloud_load_balancer_service" "control_plane" {
count = var.use_control_plane_lb ? 1 : 0
load_balancer_id = hcloud_load_balancer.control_plane.*.id[0]
protocol = "tcp"
destination_port = "6443"
listen_port = "6443"
}
locals {
control_plane_endpoint_host = var.control_plane_endpoint != null ? one(compact(regexall("^(?:https?://)?(?:.*@)?(?:\\[([a-fA-F0-9:]+)\\]|([^:/?#]+))", var.control_plane_endpoint)[0])) : null
control_plane_ips = {
for k, v in module.control_planes : k => coalesce(
v.ipv4_address,
v.ipv6_address,
v.private_ipv4_address
)
}
k3s-config = { for k, v in local.control_plane_nodes : k => merge(
{
node-name = module.control_planes[k].name
server = length(module.control_planes) == 1 ? null : coalesce(
var.control_plane_endpoint,
"https://${
var.use_control_plane_lb ? hcloud_load_balancer_network.control_plane.*.ip[0] :
(
module.control_planes[k].private_ipv4_address == module.control_planes[keys(module.control_planes)[0]].private_ipv4_address ?
module.control_planes[keys(module.control_planes)[1]].private_ipv4_address :
module.control_planes[keys(module.control_planes)[0]].private_ipv4_address
)
}:6443"
)
token = local.k3s_token
disable-cloud-controller = true
disable-kube-proxy = var.disable_kube_proxy
disable = local.disable_extras
# Kubelet arg precedence (last wins): local.kubelet_arg > v.kubelet_args > k3s_global_kubelet_args > k3s_control_plane_kubelet_args
kubelet-arg = concat(local.kubelet_arg, v.kubelet_args, var.k3s_global_kubelet_args, var.k3s_control_plane_kubelet_args)
kube-apiserver-arg = local.kube_apiserver_arg
kube-controller-manager-arg = local.kube_controller_manager_arg
flannel-iface = local.flannel_iface
node-ip = module.control_planes[k].private_ipv4_address
advertise-address = module.control_planes[k].private_ipv4_address
node-label = v.labels
node-taint = v.taints
selinux = var.disable_selinux ? false : (v.selinux == true ? true : false)
cluster-cidr = var.cluster_ipv4_cidr
service-cidr = var.service_ipv4_cidr
cluster-dns = local.cluster_dns_ipv4
write-kubeconfig-mode = "0644" # needed for import into rancher
},
lookup(local.cni_k3s_settings, var.cni_plugin, {}),
var.use_control_plane_lb ? {
tls-san = concat(
compact([
hcloud_load_balancer.control_plane.*.ipv4[0],
hcloud_load_balancer_network.control_plane.*.ip[0],
var.kubeconfig_server_address != "" ? var.kubeconfig_server_address : null,
local.control_plane_endpoint_host,
!var.control_plane_lb_enable_public_interface && var.nat_router != null ? hcloud_server.nat_router[0].ipv4_address : null
]),
var.additional_tls_sans
)
} : {
tls-san = concat(
compact([
local.control_plane_endpoint_host,
module.control_planes[k].ipv4_address != "" ? module.control_planes[k].ipv4_address : null,
module.control_planes[k].ipv6_address != "" ? module.control_planes[k].ipv6_address : null,
try(one(module.control_planes[k].network).ip, null)
]),
var.additional_tls_sans)
},
local.etcd_s3_snapshots,
var.control_planes_custom_config,
local.prefer_bundled_bin_config
) }
}
resource "terraform_data" "control_plane_config" {
for_each = local.control_plane_nodes
triggers_replace = {
control_plane_id = module.control_planes[each.key].id
config = sha1(yamlencode(local.k3s-config[each.key]))
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.control_plane_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
# Generating k3s server config file
provisioner "file" {
content = yamlencode(local.k3s-config[each.key])
destination = "/tmp/config.yaml"
}
provisioner "remote-exec" {
inline = [local.k3s_config_update_script]
}
depends_on = [
terraform_data.first_control_plane,
hcloud_network_subnet.control_plane
]
}
moved {
from = null_resource.control_plane_config
to = terraform_data.control_plane_config
}
resource "terraform_data" "audit_policy" {
for_each = local.control_plane_nodes
triggers_replace = {
control_plane_id = module.control_planes[each.key].id
audit_policy = sha1(var.k3s_audit_policy_config)
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.control_plane_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
provisioner "file" {
content = var.k3s_audit_policy_config
destination = "/tmp/audit-policy.yaml"
}
provisioner "remote-exec" {
inline = [local.k3s_audit_policy_update_script]
}
depends_on = [
terraform_data.first_control_plane,
hcloud_network_subnet.control_plane
]
}
moved {
from = null_resource.audit_policy
to = terraform_data.audit_policy
}
resource "terraform_data" "authentication_config" {
for_each = local.control_plane_nodes
triggers_replace = {
control_plane_id = module.control_planes[each.key].id
authentication_config = sha1(var.authentication_config)
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.control_plane_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
provisioner "file" {
content = var.authentication_config
destination = "/tmp/authentication_config.yaml"
}
provisioner "remote-exec" {
inline = [local.k3s_authentication_config_update_script]
}
depends_on = [
terraform_data.first_control_plane,
hcloud_network_subnet.control_plane
]
}
moved {
from = null_resource.authentication_config
to = terraform_data.authentication_config
}
resource "terraform_data" "control_planes" {
for_each = local.control_plane_nodes
triggers_replace = {
control_plane_id = module.control_planes[each.key].id
}
connection {
user = "root"
private_key = var.ssh_private_key
agent_identity = local.ssh_agent_identity
host = local.control_plane_ips[each.key]
port = var.ssh_port
bastion_host = local.ssh_bastion.bastion_host
bastion_port = local.ssh_bastion.bastion_port
bastion_user = local.ssh_bastion.bastion_user
bastion_private_key = local.ssh_bastion.bastion_private_key
}
# Install k3s server
provisioner "remote-exec" {
inline = local.install_k3s_server
}
# Start the k3s server and wait for it to have started correctly
provisioner "remote-exec" {
inline = [
"systemctl start k3s 2> /dev/null",
# prepare the needed directories
"mkdir -p /var/post_install /var/user_kustomize",
# wait for the server to be ready
<<-EOT
timeout 360 bash <<EOF
until systemctl status k3s > /dev/null; do
systemctl start k3s 2> /dev/null
echo "Waiting for the k3s server to start..."
sleep 3
done
EOF
EOT
]
}
depends_on = [
terraform_data.first_control_plane,
terraform_data.control_plane_config,
terraform_data.authentication_config,
hcloud_network_subnet.control_plane
]
}
moved {
from = null_resource.control_planes
to = terraform_data.control_planes
}
================================================
FILE: data.tf
================================================
data "github_release" "hetzner_ccm" {
count = var.hetzner_ccm_version == null ? 1 : 0
repository = "hcloud-cloud-controller-manager"
owner = "hetznercloud"
retrieve_by = "latest"
}
data "github_release" "hetzner_csi" {
count = var.hetzner_csi_version == null && !var.disable_hetzner_csi ? 1 : 0
repository = "csi-driver"
owner = "hetznercloud"
retrieve_by = "latest"
}
// github_release for kured
data "github_release" "kured" {
count = var.kured_version == null ? 1 : 0
repository = "kured"
owner = "kubereboot"
retrieve_by = "latest"
}
// github_release for kured
data "github_release" "calico" {
count = var.calico_version == null && var.cni_plugin == "calico" ? 1 : 0
repository = "calico"
owner = "projectcalico"
retrieve_by = "latest"
}
data "hcloud_ssh_keys" "keys_by_selector" {
count = length(var.ssh_hcloud_key_label) > 0 ? 1 : 0
with_selector = var.ssh_hcloud_key_label
}
================================================
FILE: docs/add-robot-server.md
================================================
# Hetzner Robot Server Integration using HCCM v1.19+
This guide describes how to add Hetzner **robot servers** to a Kubernetes cluster with help of the [hcloud-cloud-controller-manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager), version 1.19 or newer.
It covers configuration for both k3s and Robot nodes, including networking, configuration, and caveats. Alternatives like WireGuard exist, but are not covered here.
---
## Prerequisites for connecting a Robot node to a new or already existing Cluster
- **Hetzner vSwitch**
- The recommended way is using a **vSwitch**, which connects the project-level Cloud subnets to the Robot node.
- This guide assumes the vSwitch has been created and is not currently connected to any subnet. The vSwitch can be created in the Hetzner Robot web-UI. See [Hetzner Docs](https://docs.hetzner.com/robot/dedicated-server/network/vswitch)
- Note down the vSwitch ID and the VLAN ID. Note: vSwitch IDs are in the number range of around 10000+, while VLAN ID range is account-specific and starts from 4000 by default.
- **Webservice User** created in Hetzner Robot account settings (for API access)
- This is required for `hccm` to list robot servers via the metadata endpoint:
- `https://169.254.169.254/hetzner/v1/metadata/instance-id`
- `hccm` version **1.19 or newer**
- **Operating System**: Ideally use the MicroOS image created by this project. Otherwise, any Linux distribution that supports k3s will work
- **Network CNI Configuration**:
- Flannel: Doesn't need additional configuration.
- Cilium: Doesn't need additional configuration, ensure `cilium_loadbalancer_acceleration_mode` is set to `"best-effort"` or `"disabled"`
- Calico: Untested
---
## 1. Connection from Kubernetes Cluster to vSwitch
In your kube.tf-configuration:
- Set `robot_ccm_enabled = true` and provide the Webservice User credentials in the `robot_user` and `robot_password` variables. All three are required to enable Robot server integration. If `robot_ccm_enabled` is true but credentials are not provided, the integration will not be activated.
- Set `vswitch_id = <vswitch_id from prerequisites>`
For manual configuration of the settings, see below:
<details>
<summary>Manual configuration of HCCM-settings and vSwitch connection</summary>
### 1. HCCM-settings
- **Update the `hcloud` Kubernetes secret** with your `robot-user` and `robot-password`.
- Set `robot.enabled: true` in `hetzner_ccm_values`.
- Set the correct `cluster-cidr` (the pod subnet for your cluster).
- Deploy `hccm` version **1.19 or newer**.
- Refer to [HCCM Github if required](https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/a0217eafe74c8704a5e8086cc774ceb3de8f04e3/chart/values.yaml#L54)
### 2. Connect the Existing Cluster Subnet manually to vSwitch
1. Choose a subnet CIDR to be used for the Robot nodes that doesn't conflict with the existing Cluster subnets, such as 10.201.0.0/16.
2. Connect the existing Cluster Cloud network to the previously created vSwitch in the web-UI and expose the routes to vSwitch.
- Follow the steps in [Hetzner docs](https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch) on how to connect the Cluster Subnets to the vSwitch. Use your selected subnet CIDR and VLAN ID.
</details>
---
## 2. Connect the Robot to the vSwitch
1. Follow the steps in "Step 2: Configure networking on your dedicated root servers" in [Hetzner docs](https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch/#step-2-configure-networking-on-your-dedicated-root-servers) to connect the Robot node to the vSwitch.
- Use your selected VLAN ID.
- If you created the Cloud->vSwitch connection via Terraform in the Step 1 of this guide, the default range for Robot is 10.201.0.0/16. The gateway is then at 10.201.0.1 and first Robot node should use private IP 10.201.0.2.
- Make sure to use MTU 1400 or less. Cilium is reported to be requiring MTU 1350 or less.
<details>
<summary>Robot Network configuration example for RHEL/AlmaLinux using nmcli</summary>
Assumptions (change these to your values!):
- vSwitch subnet: `10.201.0.0/16`
- VLAN ID: `4000` # "arbitrary" value, replace with your VLAN ID
- Main interface: `enp6s0`
> [!CAUTION]
> The routes and CIDR notations depend on your local setup and may vary depending on your network configuration.
```bash
nmcli connection add type vlan con-name vlan4000 ifname vlan4000 vlan.parent enp6s0 vlan.id 4000
nmcli connection modify vlan4000 802-3-ethernet.mtu 1400 # Important: vSwitch requires MTU 1400 max.
nmcli connection modify vlan4000 ipv4.addresses '10.201.0.2/16'
nmcli connection modify vlan4000 ipv4.gateway '10.201.0.1'
nmcli connection modify vlan4000 ipv4.method manual
# Route all 10.x IPs through the vSwitch gateway
nmcli connection modify vlan4000 +ipv4.routes "10.0.0.0/8 10.201.0.1"
# Apply the config
nmcli connection down vlan4000
nmcli connection up vlan4000
```
</details>
---
## 3. Verify Network connectivity
1. Log in to your Robot Node using SSH and ping one of the Cloud Control Plane nodes Private Network IP. (e.g., 10.255.0.101).
2. Log in to one of the Cloud Control Plane nodes using SSH and ping the Robot Node Private Network IP, such as 10.201.0.2.
<details>
<summary>Troubleshoot Robot Node networking</summary>
- Make sure the IP address and routing are correct on the Robot Node.
- Following examples assume Robot Node public IP 203.0.113.123, private IP 10.201.0.2, VLAN ID 4000 and device enp6s0.
- `ip route show` on the Robot Node should print similar to this:
```
default via 203.0.113.123 dev enp6s0 proto static onlink
10.0.0.0/8 via 10.201.0.1 dev enp6s0.4000 proto static onlink
10.201.0.0/16 dev enp6s0.4000 proto kernel scope link src 10.201.0.2
```
- `ip addr` on the Robot Node should include similar to this:
```
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether a8:a1:REDACTED brd ff:ff:ff:ff:ff:ff
inet 203.0.113.123/32 scope global enp6s0
valid_lft forever preferred_lft forever
inet6 2a01:REDACTED/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::REDACTED/64 scope link
valid_lft forever preferred_lft forever
3: enp6s0.4000@enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
link/ether a8:a1:REDACTED brd ff:ff:ff:ff:ff:ff
inet 10.201.0.2/16 brd 10.201.255.255 scope global enp6s0.4000
valid_lft forever preferred_lft forever
inet6 fe80::REDACTED/64 scope link
valid_lft forever preferred_lft forever
```
- You may want to try to "Refresh" the vSwitch connection in the Robot web-UIs vSwitches admin-panel. Select the vSwitch, then Robot Node and click Refresh.
- Try rebooting the Robot Node
</details>
---
## 4. Robot Node: k3s Agent Configuration
> [!IMPORTANT]
> If you set a Nodename for the k3s-agent, it must match the server name in the Hetzner Robot Web-UI.
1. **Create `/etc/rancher/k3s/config.yaml`** on the robot node:
```yaml
flannel-iface: enp6s0 # Set to your main interface (only needed for Flannel CNI)
prefer-bundled-bin: true
kubelet-arg:
- cloud-provider=external
- volume-plugin-dir=/var/lib/kubelet/volumeplugins
- kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi
- system-reserved=cpu=250m,memory=6000Mi # Optional: reserve some space for system
node-label:
- k3s_upgrade=true
- instance.hetzner.cloud/provided-by=robot # To prevent Hetzner CSI pods from being scheduled on robot nodes
node-taint: []
selinux: true
server: https://<API_SERVER_IP>:6443 # Replace with your API server IP
token: <CLUSTER_TOKEN> # Replace with your cluster token
```
---
## 5. Storage and Scheduling Notes
- **Hetzner Cloud Volumes** do **not** work on robot servers (CSI driver limitation).
- Use [Longhorn](https://longhorn.io/) or other external storage.
- Pods using cloud volumes cannot be scheduled on robot nodes.
- **Longhorn**: Install `open-iscsi` and start the service:
```bash
sudo dnf install -y iscsi-initiator-utils
sudo systemctl start iscsid
```
- **Node Scheduling**:
- Use taints and labels to control pod placement.
- To prevent Hetzner CSI pods from being scheduled on robot nodes, apply the label:
```
instance.hetzner.cloud/provided-by=robot
```
[Reference](https://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#integration-with-root-servers)
---
## 6. Caveats & Warnings
- This setup may not cover all edge cases (e.g., other CNIs, non-wireguard clusters, complex private networks).
- When destroying the cluster, it takes a few minutes for the vSwitch binding to be released on the Robot side.
- **Test your network thoroughly** before adding robot nodes to production clusters.
- **MTU Issues**: When using vSwitch, MTU configuration is critical:
- vSwitch has a maximum MTU of 1400
- Some users report needing even lower MTU values (e.g., 1350 or less) for stable operation
- This particularly affects Cilium CNI users
- Without proper MTU configuration, you may experience:
- Pods unable to connect to the Kubernetes API
- Network instability for pods not using host networking
- Intermittent connection issues
- Test different MTU values if you encounter network issues
---
## References
- [Hetzner Cloud Controller Manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager)
- [Hetzner vSwitch & Robot Networking](https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch)
- [Hetzner CSI Driver: Root Server Integration](https://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#integration-with-root-servers)
================================================
FILE: docs/customize-mount-path-longhorn.md
================================================
## How to use a custom mount path for Longhorn
<hr>
In order to use NVMe and external disks with Longhorn, you may need to mount an external disk to a location other than the default under the `/var/` folder. This can provide more storage capacity across the cluster, especially if you haven't disabled the default Longhorn disks.
> ⚠️ Note: You can set any mount path, but it must be within the `/var/` folder.
### How to set a custom mount path for your external disk?
1. You must enable Longhorn in your module.
```terraform
enable_longhorn = true
```
2. Set the Helm values for Longhorn. The `defaultDataPath` is important as this path is automatically created by Longhorn and will be the default storage class pointing to your primary disks (e.g., NVMe).
```yaml
longhorn_values = <<-EOT
defaultSettings:
nodeDrainPolicy: allow-if-replica-is-stopped
defaultDataPath: /var/longhorn
persistence:
defaultFsType: ext4
defaultClassReplicaCount: 3
defaultClass: true
EOT
```
3. In the `agent_nodepools` where you want to have a customized mount path, set the `longhorn_mount_path` variable. It's a good practice to define this path as a local variable to ensure consistency.
```terraform
locals {
custom_longhorn_path = "/var/lib/longhorn"
}
agent_nodepools = [
{
# ... other nodepool configuration
labels = ["role=monitoring", "storage=ssd"], # Label we use to filter nodes
longhorn_volume_size = 50,
longhorn_mount_path = local.custom_longhorn_path # This is the custom path
}
]
```
4. Apply the changes. As a result, your external disks will be mounted to the path defined in `local.custom_longhorn_path`.
### How to configure Longhorn to use the new path?
After setting the custom mount path, you need to configure Longhorn to recognize and use it. This typically involves:
1. Patching the Longhorn nodes to add the new disk.
2. Creating a new StorageClass that uses the new disk.
Here is an example of how you can achieve this with Terraform:
```terraform
# Find the nodes with the 'ssd' storage label
data "kubernetes_nodes" "ssd_nodes" {
metadata {
labels = {
"storage" = "ssd"
}
}
}
# Patch the selected Longhorn nodes to add the new disk
# IMPORTANT: The "path" value below must match the 'longhorn_mount_path' for the nodes
# selected by the 'storage=ssd' label.
resource "terraform_data" "longhorn_patch_external_disk" {
for_each = {
for node in data.kubernetes_nodes.ssd_nodes.nodes : node.metadata[0].name => node.metadata[0].name
}
provisioner "local-exec" {
command = <<-EOT
KUBECONFIG=${var.kubeconfig_path} kubectl -n longhorn-system patch nodes.longhorn.io ${each.key} --type merge -p '{
"spec": {
"disks": {
"external-ssd": {
"path": "${local.custom_longhorn_path}",
"allowScheduling": true,
"tags": ["ssd"]
}
}
}
}'
EOT
}
}
# Create a new StorageClass for the SSD-backed Longhorn storage
resource "kubernetes_manifest" "longhorn_ssd_storageclass" {
manifest = {
apiVersion = "storage.k8s.io/v1"
kind = "StorageClass"
metadata = {
name = "longhorn-ssd"
}
provisioner = "driver.longhorn.io"
parameters = {
numberOfReplicas = "3"
staleReplicaTimeout = "30"
diskSelector = "ssd"
fromBackup = ""
}
reclaimPolicy = "Delete"
allowVolumeExpansion = true
volumeBindingMode = "Immediate"
}
depends_on = [terraform_data.longhorn_patch_external_disk]
}
================================================
FILE: docs/llms.md
================================================
**An Intricate Guide to Configuring the `kube-hetzner` Terraform Module for k3s on Hetzner Cloud**
**Preamble: Understanding the Landscape**
Before diving into the specifics of the configuration file, it's crucial to understand the core components and philosophies at play:
* **Terraform:** An Infrastructure as Code (IaC) tool that allows you to define and provision data center infrastructure using a declarative configuration language. It manages the lifecycle of your resources.
* **Hetzner Cloud (hcloud):** The IaaS provider where your Kubernetes cluster will reside. Terraform will interact with the Hetzner Cloud API to create servers, networks, load balancers, etc.
* **k3s:** A lightweight, certified Kubernetes distribution. It's designed to be lean and easy to install, making it ideal for edge, IoT, CI, and, as in this case, relatively straightforward cloud deployments. The `kube-hetzner` module specifically targets k3s.
* **`kube-hetzner/kube-hetzner/hcloud` Module:** A community-maintained Terraform module that abstracts away the complexity of setting up a k3s cluster on Hetzner Cloud. It provides a set of configurable inputs to define your desired cluster topology and features.
* **Declarative Configuration:** You *declare* the desired state of your infrastructure, and Terraform, with the help of the module, figures out how to achieve that state.
* **Idempotency:** Applying the same Terraform configuration multiple times should result in the same state, without unintended side effects (though some module operations might have nuances).
This guide will walk through the provided Terraform configuration, explaining the purpose, implications, and interdependencies of each setting.
---
**Section 1: `locals` Block - Foundational Variables**
```terraform
locals {
# You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env
# within your shell, such as: export TF_VAR_hcloud_token=xxxxxxxxxxx
# If you choose to define it in the shell, this can be left as is.
# Your Hetzner token can be found in your Project > Security > API Token (Read & Write is required).
hcloud_token = "xxxxxxxxxxx"
}
```
* **Purpose:** The `locals` block defines local variables within your Terraform configuration. These are not exposed as input variables to the module but are used internally within this root configuration file.
* **`hcloud_token`:**
* **Significance:** This is arguably the most critical piece of sensitive information. It's the API token that grants Terraform programmatic access to your Hetzner Cloud account to create, modify, and delete resources.
* **Permissions:** As noted, the token *must* have "Read & Write" permissions. A read-only token would allow Terraform to plan but fail during the apply phase when attempting to create resources.
* **Security Considerations:**
* **Hardcoding (as shown):** `hcloud_token = "xxxxxxxxxxx"` is convenient for quick tests but is a **significant security risk** if this file is committed to version control (e.g., Git) or shared.
* **Environment Variable (Recommended):** The comment `export TF_VAR_hcloud_token=xxxxxxxxxxx` highlights the best practice. Terraform automatically picks up environment variables prefixed with `TF_VAR_`. So, `TF_VAR_hcloud_token` will populate a Terraform variable named `hcloud_token` (which we'll see defined later). This keeps the sensitive token out of your configuration files.
* **Other Secret Management:** For more advanced setups, tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault could be used, with Terraform fetching the token at runtime.
* **Interaction:** This `local.hcloud_token` is used as a fallback if the `var.hcloud_token` (populated by the environment variable or a `terraform.tfvars` file) is not set. The module instantiation later uses a conditional: `var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token`.
---
**Section 2: `module "kube-hetzner"` Block - The Core Orchestration**
This block is where the magic happens. It instantiates the `kube-hetzner` module, passing it all the necessary configurations.
```terraform
module "kube-hetzner" {
providers = {
hcloud = hcloud
}
hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
```
* **`module "kube-hetzner"`:** This declares an instance of a Terraform module. The name "kube-hetzner" here is arbitrary for this instance; you could call it "my_cluster" if you wished, though consistency with the module name is common.
* **`providers` Block:**
* **Purpose:** Terraform modules can define their own provider requirements. When a module uses a provider (like `hcloud`), the calling configuration (this root module) needs to explicitly pass that provider configuration to the child module.
* **`hcloud = hcloud`:** This line tells the `kube-hetzner` module to use the `hcloud` provider configuration defined in *this* root `main.tf` file (which we'll see at the end). This ensures that the module and the root configuration are using the same Hetzner account and settings.
* **`hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token`:**
* **Purpose:** This is an input variable for the `kube-hetzner` module itself. The module needs the Hetzner API token to function.
* **Logic:** This is a ternary conditional operator.
* `var.hcloud_token != ""`: It checks if the input variable `hcloud_token` (defined at the root level, typically populated by `TF_VAR_hcloud_token`) is not an empty string.
* `? var.hcloud_token`: If true (the environment variable is set), use its value.
* `: local.hcloud_token`: If false (the environment variable is not set or is empty), fall back to using the `hcloud_token` defined in the `locals` block at the top of this file.
* **Benefit:** This provides flexibility in how the token is supplied, prioritizing environment variables for better security practices.
```terraform
# Then fill or edit the below values. Only the first values starting with a * are obligatory; the rest can remain with their default values, or you
# could adapt them to your needs.
# * source can be specified in multiple ways:
# 1. For normal use, (the official version published on the Terraform Registry), use
source = "kube-hetzner/kube-hetzner/hcloud"
# When using the terraform registry as source, you can optionally specify a version number.
# See https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud for the available versions
# version = "2.15.3"
# 2. For local dev, path to the git repo
# source = "../../kube-hetzner/"
# 3. If you want to use the latest master branch (see https://developer.hashicorp.com/terraform/language/modules/sources#github), use
# source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"
```
* **`source` (Obligatory):**
* **Purpose:** This tells Terraform where to find the `kube-hetzner` module code.
* **Option 1 (Terraform Registry - Recommended for Users):** `kube-hetzner/kube-hetzner/hcloud`
* This is the standard way to use published modules. Terraform will download it from the public Terraform Registry.
* **`version`:** It's highly recommended to pin the module version (e.g., `version = "2.15.3"`). This ensures:
* **Reproducibility:** Your infrastructure builds are consistent over time.
* **Stability:** Prevents unexpected changes or breakages if a new, incompatible version of the module is released.
* **Controlled Upgrades:** You can consciously decide when to upgrade the module version after reviewing its changelog.
* **Option 2 (Local Path - For Module Developers/Contributors):** `source = "../../kube-hetzner/"`
* Used when you have a local copy of the module's source code, typically for development or testing modifications to the module itself. The path is relative to this `main.tf` file.
* **Option 3 (Direct Git Repository - For Bleeding Edge/Specific Commits):** `source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"`
* Pulls the module directly from the `master` branch of the GitHub repository. This is generally **not recommended for production** as `master` can be unstable.
* You can also specify a specific branch, tag, or commit hash using the `ref` query parameter (e.g., `source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner?ref=v2.15.3"`).
```terraform
# Note that some values, notably "location" and "public_key" have no effect after initializing the cluster.
# This is to keep Terraform from re-provisioning all nodes at once, which would lose data. If you want to update
# those, you should instead change the value here and manually re-provision each node. Grep for "lifecycle".
```
* **Important Note on Immutability:**
* This comment highlights a critical aspect of how this module (and often Terraform resources in general) handles certain changes.
* **`location` and `public_key`:** For existing server nodes, changing these attributes in the Terraform configuration *after* the initial `terraform apply` will not automatically trigger a change on the Hetzner Cloud server itself through a simple `terraform apply`.
* **Reasoning (Data Preservation):** If Terraform were to change the location of a server, it would mean destroying the old server and creating a new one, leading to data loss. Similarly, changing the primary SSH key might involve complex OS-level operations or re-provisioning.
* **Module's Approach (`lifecycle` block):** The module likely uses Terraform's `lifecycle` meta-argument, specifically `ignore_changes`, on these attributes for the server resources. This tells Terraform to create the resource with the initial value but then ignore any subsequent changes to that attribute in the configuration for plan/apply purposes.
* **Manual Intervention Required:** If you *need* to change these, you must:
1. Update the value in your Terraform configuration.
2. Manually re-provision the affected node(s). This could involve:
* Cordoning and draining the node in Kubernetes.
* Using `terraform taint <resource_address>` to mark the specific server resource for recreation on the next `apply`.
* Manually deleting the server in Hetzner Cloud and letting Terraform recreate it.
* This is a deliberate design choice to prevent accidental data loss or full cluster rebuilds for minor changes to sensitive, foundational attributes.
```terraform
# Customize the SSH port (by default 22)
# ssh_port = 2222
```
* **`ssh_port` (Optional):**
* **Default:** `22`
* **Purpose:** Allows you to specify a custom SSH port for the nodes created by the module. The module will configure the SSH daemon on the nodes to listen on this port and adjust firewall rules accordingly.
* **Use Case:** Security through obscurity (minor benefit) or if port 22 is blocked/used by something else in your environment.
* **Implication:** You'll need to specify this custom port when SSHing into the nodes (e.g., `ssh -p 2222 user@node_ip`).
```terraform
# * Your ssh public key
ssh_public_key = file("~/.ssh/id_ed25519.pub")
# * Your private key must be "ssh_private_key = null" when you want to use ssh-agent for a Yubikey-like device authentication or an SSH key-pair with a passphrase.
# For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md
ssh_private_key = file("~/.ssh/id_ed25519")
# You can add additional SSH public Keys to grant other team members root access to your cluster nodes.
# ssh_additional_public_keys = []
```
* **`ssh_public_key` (Obligatory):**
* **Purpose:** The content of your SSH public key. This key will be added to the `authorized_keys` file on all created nodes, allowing you to SSH into them as the root user (or the default user configured by the OS image).
* **`file("~/.ssh/id_ed25519.pub")`:** The `file()` function reads the content of the specified file. `~` is expanded to your home directory. Ensure this path is correct.
* **Security:** This is the primary means of accessing your nodes. Protect your corresponding private key.
* **`ssh_private_key` (Obligatory, but can be `null`):**
* **Purpose:** The content of your SSH private key. This is used by Terraform's provisioners (if the module uses them for direct SSH commands during setup) or by tools like Ansible if integrated. It's also used for generating the kubeconfig if it needs to SSH into a node to fetch it.
* **`file("~/.ssh/id_ed25519")`:** Reads the private key content.
* **`ssh_private_key = null` (Conditional Usage):**
* **When to use `null`:** If your private key is passphrase-protected, or if you're using an SSH agent (e.g., with a YubiKey or `ssh-add`), you *must* set this to `null`. Terraform cannot directly use a passphrase-protected key without the passphrase.
* **SSH Agent Reliance:** When `null`, Terraform (and underlying tools used by the module) will attempt to use an already configured SSH agent to authenticate. Ensure your key is added to the agent (`ssh-add ~/.ssh/your_private_key`).
* **Security:** Hardcoding the private key content via `file()` is less secure if the `.tf` file is shared. Using `null` with an SSH agent is generally preferred for keys with passphrases.
* **`ssh_additional_public_keys` (Optional):**
* **Default:** `[]` (empty list)
* **Purpose:** A list of strings, where each string is the content of an additional SSH public key. These keys will also be added to `authorized_keys` on the nodes.
* **Use Case:** Granting SSH access to other team members or automated systems without sharing your primary private key.
* **Format:** `ssh_additional_public_keys = [file("~/.ssh/teammate1.pub"), "ssh-rsa AAAAB3NzaC1yc2EAAA... user@host"]`
```terraform
# You can also add additional SSH public Keys which are saved in the hetzner cloud by a label.
# See https://docs.hetzner.com/cloud/#label-selector
# ssh_hcloud_key_label = "role=admin"
```
* **`ssh_hcloud_key_label` (Optional):**
* **Purpose:** Instead of providing raw public key content, you can specify a label. The module will then find SSH keys already uploaded to your Hetzner Cloud project that match this label and add them to the nodes.
* **Hetzner Cloud Feature:** This leverages Hetzner's ability to store and label SSH keys.
* **Use Case:** Managing a central repository of SSH keys in Hetzner Cloud and assigning them to servers based on roles or teams.
* **Format:** A string representing the label selector (e.g., `"team=devops"`, `"environment=production,role=admin"`).
```terraform
# If you use SSH agent and have issues with SSH connecting to your nodes, you can increase the number of auth tries (default is 2)
# ssh_max_auth_tries = 10
```
* **`ssh_max_auth_tries` (Optional):**
* **Default:** `2` (or a small number set by the underlying SSH client/library).
* **Purpose:** Controls the `MaxAuthTries` setting for SSH connections made by Terraform/module scripts.
* **Use Case:** If you have many keys loaded in your SSH agent, the server might close the connection before the correct key is tried. Increasing this value gives the SSH client more attempts to offer different keys.
* **Caution:** Setting this too high could theoretically make brute-force attacks slightly easier if other security measures are weak, but the primary defense is strong key management.
```terraform
# If you want to use an ssh key that is already registered within hetzner cloud, you can pass its id.
# If no id is passed, a new ssh key will be registered within hetzner cloud.
# It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` variables.
# hcloud_ssh_key_id = ""
```
* **`hcloud_ssh_key_id` (Optional):**
* **Purpose:** Allows you to use an SSH key that is *already registered* in your Hetzner Cloud project by specifying its unique ID.
* **Behavior:**
* **If ID provided:** The module will associate this existing Hetzner SSH key resource with the created servers. It will *not* create a new SSH key resource in Hetzner Cloud based on `ssh_public_key`.
* **If ID not provided (or empty string):** The module will create a *new* SSH key resource in Hetzner Cloud using the content from `ssh_public_key` and associate that new key with the servers.
* **Crucial Constraint:** The comment "It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` variables" is vital. Even if using an existing Hetzner key ID, the module might still need the raw public key content for other purposes (e.g., configuring `authorized_keys` directly if Hetzner's association method isn't solely relied upon, or for consistency). The private key is needed if SSH connections are made by provisioners. This ensures the keys Terraform *thinks* it's using match the key Hetzner *knows* about.
```terraform
# These can be customized, or left with the default values
# * For Hetzner locations see https://docs.hetzner.com/general/others/data-centers-and-connection/
network_region = "eu-central" # change to `us-east` if location is ash
```
* **`network_region` (Obligatory, though has a default in the module):**
* **Default (in module, not shown here):** Likely "eu-central".
* **Purpose:** Specifies the broad geographical region for your Hetzner Cloud private network. All servers and load balancers within the same private network must reside in locations that belong to this network region.
* **Hetzner Regions:**
* `eu-central`: Encompasses European locations like Falkenstein (`fsn1` - currently unavailable due to high demand), Nuremberg (`nbg1`), Helsinki (`hel1`).
* `us-east`: Encompasses Ashburn, VA (`ash`).
* `us-west`: Encompasses Hillsboro, OR (`hil`). (Check if supported by module if you intend to use it)
* **Constraint:** The `location` specified in your `control_plane_nodepools` and `agent_nodepools` *must* be compatible with this `network_region`. You cannot have a server in `fsn1` (Europe) in a network defined for `us-east`.
```terraform
# If you want to create the private network before calling this module,
# you can do so and pass its id here. For example if you want to use a proxy
# which only listens on your private network. Advanced use case.
#
# NOTE1: make sure to adapt network_ipv4_cidr, cluster_ipv4_cidr, and service_ipv4_cidr accordingly.
# If your network is created with 10.0.0.0/8, and you use subnet 10.128.0.0/9 for your
# non-k3s business, then adapting `network_ipv4_cidr = "10.0.0.0/9"` should be all you need.
#
# NOTE2: square brackets! This must be a list of length 1.
#
# existing_network_id = [hcloud_network.your_network.id]
```
* **`existing_network_id` (Optional, Advanced):**
* **Default:** Not set, meaning the module will create and manage its own Hetzner Cloud private network.
* **Purpose:** Allows you to use a pre-existing Hetzner Cloud private network for your Kubernetes cluster.
* **Format:** A list containing a single element: the ID of the existing Hetzner network (e.g., `[1234567]`). The comment `[hcloud_network.your_network.id]` shows how you'd reference a network created in the same Terraform configuration but outside this module.
* **Use Case:** Integrating the Kubernetes cluster into a larger, existing infrastructure on Hetzner Cloud where other services already reside on a specific private network.
* **Critical Considerations (NOTE1):** If you use an existing network, you are responsible for ensuring that the IP address ranges used by this module (`network_ipv4_cidr`, `cluster_ipv4_cidr`, `service_ipv4_cidr`) do not conflict with other subnets or IP ranges already in use on that existing network. You might need to adjust these CIDR parameters in the module configuration to fit within an available portion of your existing network's IP space. The example given (using `10.0.0.0/9` for k3s within a larger `10.0.0.0/8` network) illustrates this.
```terraform
# If you must change the network CIDR you can do so below, but it is highly advised against.
# network_ipv4_cidr = "10.0.0.0/8"
```
* **`network_ipv4_cidr` (Optional, Advanced):**
* **Default (in module):** Typically `10.0.0.0/8`.
* **Purpose:** Defines the overall IP address range for the Hetzner Cloud private network that the module will create (if `existing_network_id` is not used). All other internal Kubernetes CIDRs (for pods, services, and node subnets) will be carved out of this range.
* **Warning:** "highly advised against" changing this unless you have a very specific reason (e.g., conflict with on-premises networks if using VPN/interconnect, or needing a smaller/different range for a very specific setup). Changing it requires careful planning of all sub-CIDRs.
* **Impact:** If changed, `cluster_ipv4_cidr` and `service_ipv4_cidr` must be sub-ranges within this new `network_ipv4_cidr`.
```terraform
# The amount of subnets into which the network will be split. Must be a power of 2.
# subnet_amount = 256
```
* **`subnet_amount` (Number, Optional):**
* **Default:** `256`.
* **Purpose:** Determines into how many subnets the `network_ipv4_cidr` is divided.
* **Constraint:** Must be a power of 2. Each nodepool (control plane and agent) and potentially the NAT router takes one subnet. Ensure this is large enough for your planned number of nodepools.
```terraform
# Using the default configuration you can only create a maximum of 42 agent-nodepools.
# This is due to the creation of a subnet for each nodepool with CIDRs being in the shape of 10.[nodepool-index].0.0/16 which collides with k3s' cluster and service IP ranges (defaults below).
# Furthermore the maximum number of nodepools (controlplane and agent) is 50, due to a hard limit of 50 subnets per network, see https://docs.hetzner.com/cloud/networks/faq/.
# So to be able to create a maximum of 50 nodepools in total, the values below have to be changed to something outside that range, e.g. `10.200.0.0/16` and `10.201.0.0/16` for cluster and service respectively.
```
* **Explanation of Nodepool Subnet Allocation and Limits:**
* **Subnet per Nodepool:** The module creates a dedicated subnet within the Hetzner private network for each nodepool (both control plane and agent). This provides network isolation at the Hetzner level and allows for distinct IP ranges per nodepool.
* **Default Subnetting Scheme:** The module uses a scheme like `10.[nodepool-index].0.0/16` for these subnets. For example, the first nodepool might get `10.1.0.0/16`, the second `10.2.0.0/16`, and so on.
* **Collision Issue:** The default k3s cluster CIDR (`10.42.0.0/16`) and service CIDR (`10.43.0.0/16`) would collide if a nodepool index reached 42 or 43 using this scheme. This limits the number of *agent* nodepools to 42 if defaults are kept.
* **Hetzner Subnet Limit:** Hetzner Cloud has a hard limit of 50 subnets per private network. This is the ultimate cap on the total number of nodepools (control plane + agent).
* **Solution for >42 Nodepools:** To exceed 42 nodepools (up to the 50 limit), you *must* change `cluster_ipv4_cidr` and `service_ipv4_cidr` to ranges that won't collide with the `10.[0-49].0.0/16` nodepool subnet ranges. The example `10.200.0.0/16` and `10.201.0.0/16` achieves this.
```terraform
# If you must change the cluster CIDR you can do so below, but it is highly advised against.
# Never change this value after you already initialized a cluster. Complete cluster redeploy needed!
# The cluster CIDR must be a part of the network CIDR!
# cluster_ipv4_cidr = "10.42.0.0/16"
```
* **`cluster_ipv4_cidr` (Optional, Advanced):**
* **Default (in module):** `10.42.0.0/16` (a common default for k3s/Kubernetes).
* **Purpose:** This is the IP address range from which Kubernetes assigns IP addresses to Pods running within the cluster.
* **Critical Warning:** "Never change this value after you already initialized a cluster." Doing so would require a complete cluster redeployment because all existing Pods and network configurations would become invalid.
* **Constraint:** Must be a sub-range of `network_ipv4_cidr`.
* **Interdependency:** As explained above, may need to be changed if you require more than 42 nodepools.
```terraform
# If you must change the service CIDR you can do so below, but it is highly advised against.
# Never change this value after you already initialized a cluster. Complete cluster redeploy needed!
# The service CIDR must be a part of the network CIDR!
# service_ipv4_cidr = "10.43.0.0/16"
```
* **`service_ipv4_cidr` (Optional, Advanced):**
* **Default (in module):** `10.43.0.0/16` (a common default for k3s/Kubernetes).
* **Purpose:** This is the IP address range from which Kubernetes assigns virtual IP addresses to Services (e.g., ClusterIP services).
* **Critical Warning:** Same as `cluster_ipv4_cidr` – do not change post-initialization without a full redeploy.
* **Constraint:** Must be a sub-range of `network_ipv4_cidr`.
* **Interdependency:** May need to be changed if you require more than 42 nodepools.
```terraform
# If you must change the service IPv4 address of core-dns you can do so below, but it is highly advised against.
# Never change this value after you already initialized a cluster. Complete cluster redeploy needed!
# The service IPv4 address must be part of the service CIDR!
# cluster_dns_ipv4 = "10.43.0.10"
```
* **`cluster_dns_ipv4` (Optional, Advanced):**
* **Default (in module):** `10.43.0.10`.
* **Purpose:** Specifies the static IP address for the CoreDNS service (or KubeDNS) within the cluster. Pods use this IP to resolve internal and external domain names.
* **Critical Warning:** Same as above – do not change post-initialization.
* **Constraint:** This IP address *must* fall within the `service_ipv4_cidr` range. Typically, it's one of the first few usable IPs in that range (e.g., `.10`).
The subsequent sections on `control_plane_nodepools` and `agent_nodepools` are extensive. I will break them down carefully.
---
**Section 2.1: `control_plane_nodepools` - The Brains of the Operation**
```terraform
# For the control planes, at least three nodes are the minimum for HA. Otherwise, you need to turn off the automatic upgrades (see README).
# **It must always be an ODD number, never even!** Search the internet for "split-brain problem with etcd" or see https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/
# For instance, one is ok (non-HA), two is not ok, and three is ok (becomes HA). It does not matter if they are in the same nodepool or not! So they can be in different locations and of various types.
# Of course, you can choose any number of nodepools you want, with the location you want. The only constraint on the location is that you need to stay in the same network region, Europe, or the US.
# For the server type, the minimum instance supported is cx23. If you want to use arm64 use cax11; see https://www.hetzner.com/cloud.
# IMPORTANT: Before you create your cluster, you can do anything you want with the nodepools, but you need at least one of each, control plane and agent.
# Once the cluster is up and running, you can change nodepool count and even set it to 0 (in the case of the first control-plane nodepool, the minimum is 1).
# You can also rename it (if the count is 0), but do not remove a nodepool from the list.
# You can safely add or remove nodepools at the end of each list. That is due to how subnets and IPs get allocated (FILO).
# The maximum number of nodepools you can create combined for both lists is 50 (see above).
# Also, before decreasing the count of any nodepools to 0, it's essential to drain and cordon the nodes in question. Otherwise, it will leave your cluster in a bad state.
# Before initializing the cluster, you can change all parameters and add or remove any nodepools. You need at least one nodepool of each kind, control plane, and agent.
# ⚠️ The nodepool names are entirely arbitrary, but all lowercase, no special characters or underscore (dashes are allowed), and they must be unique.
# If you want to have a single node cluster, have one control plane nodepools with a count of 1, and one agent nodepool with a count of 0.
# Please note that changing labels and taints after the first run will have no effect. If needed, you can do that through Kubernetes directly.
# Multi-architecture clusters are OK for most use cases, as container underlying images tend to be multi-architecture too.
# * Example below:
control_plane_nodepools = [
{
name = "control-plane-nbg1",
server_type = "cx23",
location = "nbg1",
labels = [],
taints = [],
count = 1
# swap_size = "2G" # remember to add the suffix, examples: 512M, 1G
# zram_size = "2G" # remember to add the suffix, examples: 512M, 1G
# kubelet_args = ["kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi", "system-reserved=cpu=250m,memory=300Mi"]
# Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):
# placement_group = "default"
# Enable automatic backups via Hetzner (default: false)
# backups = true
# To disable public ips (default: false)
# WARNING: If both values are set to "true", your server will only be accessible via a private network. Make sure you have followed
# the instructions regarding this type of setup in README.md: "Use only private IPs in your cluster".
# disable_ipv4 = true
# disable_ipv6 = true
},
// ... more control plane nodepool examples ...
]
```
* **`control_plane_nodepools` (Obligatory, list of maps):**
* **Purpose:** Defines one or more groups of control plane nodes. Control plane nodes run the Kubernetes master components (API server, scheduler, controller manager) and, in k3s with an embedded database, `etcd` (or the k3s default, SQLite for single-node, or embedded etcd for HA).
* **Structure:** A list of maps, where each map defines a distinct nodepool.
* **High Availability (HA) Critical Logic:**
* **Minimum for HA:** 3 control plane nodes.
* **Odd Number:** Always use an odd number (1, 3, 5, etc.) for etcd quorum to prevent split-brain scenarios. 2 nodes are worse than 1 for HA.
* **Impact of Non-HA (1 control plane):** If you have only one control plane node, features like `automatically_upgrade_os` and `automatically_upgrade_k3s` (if not managed carefully) can lead to downtime. The module's README and comments often advise disabling automatic upgrades for single control plane setups.
* **Distribution:** HA control plane nodes can be in the same nodepool definition (e.g., `count = 3` in one map) or spread across multiple nodepool definitions (e.g., three maps, each with `count = 1`, potentially in different `location`s for better fault tolerance).
* **Minimum Requirements (Initial Cluster Create):**
* At least one control plane nodepool with `count >= 1`.
* (Typically) At least one agent nodepool with `count >= 1`, *unless* you are creating a single-node cluster where the control plane also acts as a worker (see below).
* **Lifecycle of Nodepools:**
* **Adding/Removing Nodepools:** You can safely add new nodepool definitions to the *end* of the list or remove nodepool definitions from the *end* of the list. This is due to how the module allocates subnets (FILO - First In, Last Out, or rather, sequentially). Modifying nodepools in the middle of the list can cause existing nodepools to be re-evaluated for their subnet, potentially leading to disruption.
* **Changing `count`:**
* **Increasing:** Generally safe. New nodes will be provisioned.
* **Decreasing (to > 0):** Terraform will select nodes to remove. Ensure workloads are drained from these nodes (`kubectl drain`) before applying, to prevent data loss or service interruption.
* **Decreasing to `0`:** The nodepool becomes effectively dormant. Its subnet remains. Before doing this, *all nodes in that pool must be drained and cordoned*.
* **Renaming:** A nodepool can be renamed *only if its `count` is 0*. Otherwise, Terraform will see it as destroying the old and creating a new one.
* **Removing from List:** Do not remove a nodepool definition from the list if it still has active nodes or if you intend to use it again. Set its `count` to 0 first.
* **Single-Node Cluster:**
* One control plane nodepool with `count = 1`.
* One (or more) agent nodepools with `count = 0`.
* The module typically automatically allows scheduling on the control plane in this scenario (or you'd set `allow_scheduling_on_control_plane = true`).
* **Multi-Architecture:** Mixing x86 (`cx` series) and ARM (`cax` series) nodes is generally fine. Kubernetes and container runtimes handle this, and many container images are multi-arch.
* **Nodepool Attributes (per map):**
* **`name` (String, Obligatory):**
* A unique, arbitrary name for this nodepool.
* Constraints: Lowercase, no special characters except dashes (`-`).
* Used for naming resources in Hetzner and for Kubernetes node labels/names.
* **`server_type` (String, Obligatory):**
* Hetzner server type:
* x86: e.g. `cx23` (2 vCPU, 4GB RAM, 40GB SSD), `cx33` (4 vCPU, 8GB RAM, 80GB SSD), `cx43` (8 vCPU, 16GB RAM, 160GB SSD).
* ARM: e.g. `cax11` (2 vCPU, 4GB RAM, 40GB SSD), `cax21` (4 vCPU, 8GB RAM, 80GB SSD).
* Minimum for control plane: `cx23` is often cited. More demanding setups (e.g., with Cilium CNI, Rancher) might require more RAM (e.g., `cx33`/`cx43` or `cax21`/`cax31`).
* **`location` (String, Obligatory):**
* Hetzner location (e.g., `fsn1`, `nbg1`, `hel1`, `ash`).
* Must be within the `network_region` defined earlier.
* For HA, distributing control plane nodes across different locations (within the same region) improves fault tolerance against a single location outage.
* **`labels` (List of Strings, Optional):**
* Default: `[]`.
* Kubernetes labels to apply to nodes in this pool. Format: `["key1=value1", "key2=value2"]`.
* **Lifecycle Note:** "changing labels and taints after the first run will have no effect." The module likely applies these only at node creation. Subsequent changes must be done via `kubectl label node ...`.
* **`taints` (List of Strings, Optional):**
* Default: `[]`.
* Kubernetes taints to apply to nodes in this pool. Format: `["key=value:Effect"]` (e.g., `"dedicated=control-plane:NoSchedule"`).
* Taints prevent pods from being scheduled on these nodes unless the pods have a corresponding toleration. Control planes often have taints to prevent regular workloads from running on them.
* **Lifecycle Note:** Same as labels – apply via `kubectl taint node ...` after initial creation if changes are needed.
* **`count` (Number, Obligatory):**
* Number of server instances to create in this specific nodepool.
* **`swap_size` (String, Optional):**
* Examples: `"512M"`, `"2G"`, `"4G"`.
* Configures a swap file of the specified size on the nodes.
* **K3s/Kubernetes Consideration:** Kubernetes traditionally doesn't work well with swap. However, recent versions of k3s/kubelet can support it if the `NodeSwap` feature gate is enabled and kubelet is configured correctly. The comment `Make sure you set "feature-gates=NodeSwap=true" if want to use swap_size` (seen later under `k3s_global_kubelet_args`) is relevant here. Use with caution and understanding of its implications on performance and scheduling.
* **`zram_size` (String, Optional):**
* Examples: `"512M"`, `"1G"`.
* Configures zRAM (compressed RAM block device, often used for swap) on the nodes.
* Can be an alternative or supplement to traditional disk-based swap, offering faster swap at the cost of CPU for compression/decompression.
* **`kubelet_args` (List of Strings, Optional):**
* Allows passing additional arguments directly to the `kubelet` process running on nodes in this specific pool.
* Example: `["kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi", "system-reserved=cpu=250m,memory=300Mi"]`
* This is for fine-grained resource reservation for Kubernetes system components (`kube-reserved`) and OS system components (`system-reserved`), ensuring they have enough resources and don't get starved by user pods.
* **Note:** There are also global `k3s_global_kubelet_args`, `k3s_control_plane_kubelet_args`, etc., defined later. These nodepool-specific `kubelet_args` likely supplement or override those for this pool.
* **`placement_group` (String, Optional):**
* Default: The module might create a default placement group or assign nodes to one.
* Purpose: Hetzner Placement Groups ensure that servers within the same group are located on different physical host systems (spread strategy). This improves fault tolerance against hardware failures on a single host.
* Value: Name of the placement group. If you specify the same name for multiple nodes/nodepools, they'll try to be in that group.
* Limit: Hetzner placement groups have limits (e.g., 10 servers per spread placement group). The module might manage creating multiple groups if a nodepool `count` exceeds this.
* **`backups` (Boolean, Optional):**
* Default: `false`.
* If `true`, enables Hetzner's automated server backup service for nodes in this pool. This incurs additional cost per server.
* **`disable_ipv4` (Boolean, Optional) / `disable_ipv6` (Boolean, Optional):**
* Default: `false` for both.
* If `true`, disables the public IPv4 or IPv6 interface on the server, respectively.
* **Warning:** If both are `true`, the server will *only* have a private IP address and will only be accessible via the Hetzner private network (e.g., from another server in the same network, or via a VPN/bastion host connected to that network). This is an advanced setup requiring careful network planning. The comment refers to a `README.md` section "Use only private IPs in your cluster" for guidance.
The example shows three control plane nodepools, each with one node, in different locations (`fsn1`, `nbg1`, `hel1`). This is a common pattern for a 3-node HA control plane, maximizing fault tolerance across Hetzner locations (within the same `network_region`).
---
**Section 2.2: `agent_nodepools` - The Workhorses**
```terraform
agent_nodepools = [
{
name = "agent-small",
server_type = "cx23",
location = "nbg1",
labels = [],
taints = [],
count = 1
# swap_size = "2G"
# zram_size = "2G"
# kubelet_args = ["kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi", "system-reserved=cpu=250m,memory=300Mi"]
# placement_group = "default"
# backups = true
},
{
name = "agent-large",
server_type = "cx33",
location = "nbg1",
labels = [],
taints = [],
count = 1
# placement_group = "default"
# backups = true
},
{
name = "storage",
server_type = "cx33",
location = "nbg1",
labels = [
"node.kubernetes.io/server-usage=storage" # Example label
],
taints = [], # Could add taints to only allow storage workloads
count = 1
# In the case of using Longhorn, you can use Hetzner volumes instead of using the node's own storage by specifying a value from 10 to 10240 (in GB)
# It will create one volume per node in the nodepool, and configure Longhorn to use them.
# Something worth noting is that Volume storage is slower than node storage, which is achieved by not mentioning longhorn_volume_size or setting it to 0.
# So for something like DBs, you definitely want node storage, for other things like backups, volume storage is fine, and cheaper.
# longhorn_volume_size = 20 # In GB
# backups = true
},
# Egress nodepool useful to route egress traffic using Hetzner Floating IPs
# used with Cilium's Egress Gateway feature
{
name = "egress",
server_type = "cx23",
location = "nbg1",
labels = [
"node.kubernetes.io/role=egress"
],
taints = [
"node.kubernetes.io/role=egress:NoSchedule" # Ensures only egress gateway pods run here
],
floating_ip = true # Special attribute for this module
# Optionally associate a reverse DNS entry with the floating IP(s).
# floating_ip_rns = "my.domain.com"
count = 1
},
# Arm based nodes
{
name = "agent-arm-small",
server_type = "cax11", # ARM server type
location = "nbg1",
labels = [],
taints = [],
count = 1
},
# For fine-grained control over the nodes in a node pool, replace the count variable with a nodes map.
# In this case, the node-pool variables are defaults which can be overridden on a per-node basis.
# Each key in the nodes map refers to a single node and must be an integer string ("1", "123", ...).
{
name = "agent-arm-medium",
server_type = "cax21", # Default server_type for this pool
location = "nbg1", # Default location
labels = [],
taints = [],
nodes = { # Overrides 'count' and allows per-node customization
"1" : { # Node identified as "1" within this pool
location = "fsn1" # Override location for this specific node
labels = [
"testing-labels=a1",
]
},
"20" : { # Node identified as "20"
labels = [
"testing-labels=b1",
]
# server_type could also be overridden here if needed
}
}
},
]
```
* **`agent_nodepools` (Obligatory, list of maps):**
* **Purpose:** Defines groups of agent (worker) nodes. These nodes run your actual application Pods.
* **Structure and Lifecycle:** Similar to `control_plane_nodepools` (list of maps, rules for adding/removing/renaming apply).
* **Minimum Requirement (Initial Cluster Create):** Typically, at least one agent nodepool with `count >= 1` is needed, unless it's a single-node cluster where the control plane also
gitextract_6abwv94a/ ├── .claude/ │ └── skills/ │ ├── fix-issue/ │ │ └── SKILL.md │ ├── kh-assistant/ │ │ └── SKILL.md │ ├── prepare-release/ │ │ └── SKILL.md │ ├── review-pr/ │ │ └── SKILL.md │ ├── sync-docs/ │ │ └── SKILL.md │ ├── test-changes/ │ │ └── SKILL.md │ └── triage-issue/ │ └── SKILL.md ├── .extra/ │ └── k3s-selinux-next.rpm ├── .github/ │ ├── FUNDING.yml │ ├── ISSUE_TEMPLATE/ │ │ ├── bug_report.yml │ │ ├── config.yml │ │ └── feature_request.yml │ ├── dependabot.yaml │ ├── release.yaml │ ├── release.yml │ └── workflows/ │ ├── generate-docs.yaml │ ├── lint_pr.yaml │ └── publish-release.yaml ├── .gitignore ├── .pre-commit-config.yaml ├── .terraform-docs.yml ├── CHANGELOG.md ├── LICENSE ├── README.md ├── SECURITY.md ├── agents.tf ├── autoscaler-agents.tf ├── control_planes.tf ├── data.tf ├── docs/ │ ├── add-robot-server.md │ ├── customize-mount-path-longhorn.md │ ├── llms.md │ ├── private-network-egress.md │ ├── ssh.md │ └── terraform.md ├── examples/ │ ├── kustomization_user_deploy/ │ │ ├── README.md │ │ ├── helm-chart/ │ │ │ ├── helm-chart.yaml.tpl │ │ │ ├── kustomization.yaml.tpl │ │ │ └── namespace.yaml.tpl │ │ ├── letsencrypt/ │ │ │ ├── kustomization.yaml.tpl │ │ │ └── letsencrypt.yaml.tpl │ │ ├── multiple-namespaces/ │ │ │ ├── base/ │ │ │ │ ├── kustomization.yaml.tpl │ │ │ │ └── pod.yaml.tpl │ │ │ ├── kustomization.yaml.tpl │ │ │ ├── namespace-a/ │ │ │ │ ├── kustomization.yaml.tpl │ │ │ │ └── namespace-a.yaml.tpl │ │ │ └── namespace-b/ │ │ │ ├── kustomization.yaml.tpl │ │ │ └── namespace-b.yaml.tpl │ │ └── simple-resources/ │ │ ├── demo-config-map.yaml.tpl │ │ ├── demo-pod.yml.tpl │ │ └── kustomization.yaml.tpl │ ├── micro_os_rollback/ │ │ └── Readme.md │ └── tls/ │ ├── ingress.yaml │ ├── pod.yaml │ └── service.yaml ├── init.tf ├── kube.tf.example ├── kubeconfig.tf ├── kustomization_backup.tf ├── kustomization_user.tf ├── kustomize/ │ ├── flannel-rbac.yaml │ └── system-upgrade-controller.yaml ├── locals.tf ├── main.tf ├── modules/ │ ├── host/ │ │ ├── locals.tf │ │ ├── main.tf │ │ ├── out.tf │ │ ├── templates/ │ │ │ └── cloudinit.yaml.tpl │ │ ├── variables.tf │ │ └── versions.tf │ └── values_merger/ │ ├── main.tf │ └── versions.tf ├── nat-router.tf ├── output.tf ├── packer-template/ │ └── hcloud-microos-snapshots.pkr.hcl ├── placement_groups.tf ├── scripts/ │ ├── cleanup.sh │ └── create.sh ├── templates/ │ ├── autoscaler-cloudinit.yaml.tpl │ ├── autoscaler.yaml.tpl │ ├── calico.yaml.tpl │ ├── ccm.yaml.tpl │ ├── cert_manager.yaml.tpl │ ├── cilium.yaml.tpl │ ├── csi-driver-smb.yaml.tpl │ ├── haproxy_ingress.yaml.tpl │ ├── hcloud-ccm-helm.yaml.tpl │ ├── hcloud-csi.yaml.tpl │ ├── kube-hetzner-selinux.te │ ├── kube_system_secrets.yaml.tpl │ ├── kured.yaml.tpl │ ├── longhorn.yaml.tpl │ ├── nat-router-cloudinit.yaml.tpl │ ├── nginx_ingress.yaml.tpl │ ├── plans.yaml.tpl │ ├── rancher.yaml.tpl │ └── traefik_ingress.yaml.tpl ├── values-export.tf ├── values-merger.tf ├── variables.tf └── versions.tf
Condensed preview — 101 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (794K chars).
[
{
"path": ".claude/skills/fix-issue/SKILL.md",
"chars": 6162,
"preview": "---\nname: fix-issue\ndescription: Use when working on a GitHub issue - fetches issue details, analyzes codebase, implemen"
},
{
"path": ".claude/skills/kh-assistant/SKILL.md",
"chars": 13776,
"preview": "---\nname: kh-assistant\ndescription: Use when users need help with kube-hetzner configuration, debugging, or questions - "
},
{
"path": ".claude/skills/prepare-release/SKILL.md",
"chars": 6071,
"preview": "---\nname: prepare-release\ndescription: Use when preparing a release - generates changelog, updates version references, a"
},
{
"path": ".claude/skills/review-pr/SKILL.md",
"chars": 10046,
"preview": "---\nname: review-pr\ndescription: Use when reviewing a pull request - security-focused review following CLAUDE.md guideli"
},
{
"path": ".claude/skills/sync-docs/SKILL.md",
"chars": 5266,
"preview": "---\nname: sync-docs\ndescription: Use when documentation needs updating - ensures variables.tf, llms.md, kube.tf.example,"
},
{
"path": ".claude/skills/test-changes/SKILL.md",
"chars": 3656,
"preview": "---\nname: test-changes\ndescription: Use after making changes to run terraform fmt, validate, and plan against test envir"
},
{
"path": ".claude/skills/triage-issue/SKILL.md",
"chars": 6282,
"preview": "---\nname: triage-issue\ndescription: Use when triaging a GitHub issue - analyzes issue, checks for duplicates, categorize"
},
{
"path": ".github/FUNDING.yml",
"chars": 68,
"preview": "# These are supported funding model platforms\n\ngithub: mysticaltech\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug_report.yml",
"chars": 1367,
"preview": "name: Bug Report\ndescription: File a bug report\ntitle: \"[Bug]: \"\nlabels: [\"bug\", \"triage\"]\nbody:\n - type: markdown\n "
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"chars": 227,
"preview": "blank_issues_enabled: true\ncontact_links:\n - name: If you have questions, use the discussions\n url: https://github.c"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.yml",
"chars": 1074,
"preview": "name: Feature Request\ntitle: \"[Feature Request]: \"\ndescription: \"Submit a feature request for consideration\"\nlabels: [\"f"
},
{
"path": ".github/dependabot.yaml",
"chars": 210,
"preview": "version: 2\nupdates:\n - package-ecosystem: \"terraform\"\n directory: \"/\"\n schedule:\n interval: \"daily\"\n\n - pac"
},
{
"path": ".github/release.yaml",
"chars": 417,
"preview": "changelog:\n exclude:\n labels:\n - ignore-for-release\n authors:\n - octocat\n categories:\n - title: Bre"
},
{
"path": ".github/release.yml",
"chars": 1004,
"preview": "# Configuration for auto-generated release notes\n# https://docs.github.com/en/repositories/releasing-projects-on-github/"
},
{
"path": ".github/workflows/generate-docs.yaml",
"chars": 970,
"preview": "name: Generate terraform docs\non:\n push:\n branches:\n - master\n - staging\n\njobs:\n docs:\n runs-on: ubunt"
},
{
"path": ".github/workflows/lint_pr.yaml",
"chars": 1170,
"preview": "name: Lint\n\non:\n pull_request:\n\njobs:\n tfsec:\n runs-on: ubuntu-latest\n permissions:\n contents: read\n p"
},
{
"path": ".github/workflows/publish-release.yaml",
"chars": 3015,
"preview": "---\nname: Publish a new Github Release\n\non:\n push:\n tags:\n - '*'\n workflow_dispatch:\n\njobs:\n Release:\n nam"
},
{
"path": ".gitignore",
"chars": 1308,
"preview": "# Local .terraform directories\n**/.terraform/*\n\n# .tfstate files\n*.tfstate\n*.tfstate.*\n\n# Crash log files\ncrash.log\ncras"
},
{
"path": ".pre-commit-config.yaml",
"chars": 1253,
"preview": "default_install_hook_types:\n - pre-commit\n\nrepos:\n - repo: https://github.com/antonbabenko/pre-commit-terraform\n re"
},
{
"path": ".terraform-docs.yml",
"chars": 511,
"preview": "formatter: \"markdown table\"\n\nrecursive:\n enabled: false\n path: modules\n\noutput:\n file: docs/terraform.md\n mode: inje"
},
{
"path": "CHANGELOG.md",
"chars": 5421,
"preview": "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Change"
},
{
"path": "LICENSE",
"chars": 1023,
"preview": "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentati"
},
{
"path": "README.md",
"chars": 31590,
"preview": "<div align=\"center\">\n\n<!-- HERO SECTION -->\n<img src=\"https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/"
},
{
"path": "SECURITY.md",
"chars": 348,
"preview": "# Security Policy\n\n## Reporting a Vulnerability\n\nPlease report any vulnerability findings privately via email to the top"
},
{
"path": "agents.tf",
"chars": 12334,
"preview": "module \"agents\" {\n source = \"./modules/host\"\n\n providers = {\n hcloud = hcloud,\n }\n\n for_each = local.agent_nodes\n"
},
{
"path": "autoscaler-agents.tf",
"chars": 10648,
"preview": "locals {\n cluster_prefix = var.use_cluster_name_in_node_name ? \"${var.cluster_name}-\" : \"\"\n first_nodepool_snapshot_id"
},
{
"path": "control_planes.tf",
"chars": 13572,
"preview": "module \"control_planes\" {\n source = \"./modules/host\"\n\n providers = {\n hcloud = hcloud,\n }\n\n for_each = local.cont"
},
{
"path": "data.tf",
"chars": 983,
"preview": "data \"github_release\" \"hetzner_ccm\" {\n count = var.hetzner_ccm_version == null ? 1 : 0\n repository = \"hcloud-cl"
},
{
"path": "docs/add-robot-server.md",
"chars": 9848,
"preview": "# Hetzner Robot Server Integration using HCCM v1.19+\n\nThis guide describes how to add Hetzner **robot servers** to a Kub"
},
{
"path": "docs/customize-mount-path-longhorn.md",
"chars": 3699,
"preview": "## How to use a custom mount path for Longhorn\n<hr>\n\nIn order to use NVMe and external disks with Longhorn, you may need"
},
{
"path": "docs/llms.md",
"chars": 225024,
"preview": "**An Intricate Guide to Configuring the `kube-hetzner` Terraform Module for k3s on Hetzner Cloud**\n\n**Preamble: Understa"
},
{
"path": "docs/private-network-egress.md",
"chars": 1265,
"preview": "# Private Network Egress & Hetzner DHCP (Aug 2025)\n\nOn **August 11, 2025**, Hetzner Cloud removed the legacy DHCP *Route"
},
{
"path": "docs/ssh.md",
"chars": 1752,
"preview": "Kube-Hetzner requires you to have a recent version of OpenSSH (>=6.5) installed on your client, and the use of a key-pai"
},
{
"path": "docs/terraform.md",
"chars": 66903,
"preview": "<!-- BEGIN_TF_DOCS -->\n### Requirements\n\n| Name | Version |\n|------|---------|\n| <a name=\"requirement_terraform\"></a> [t"
},
{
"path": "examples/kustomization_user_deploy/README.md",
"chars": 3603,
"preview": "# How to Install and Deploy Additional Resources with Terraform and Kube-Hetzner\n\nKube-Hetzner allows you to provide use"
},
{
"path": "examples/kustomization_user_deploy/helm-chart/helm-chart.yaml.tpl",
"chars": 250,
"preview": "apiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: argocd\n namespace: argocd\nspec:\n repo: https://argopro"
},
{
"path": "examples/kustomization_user_deploy/helm-chart/kustomization.yaml.tpl",
"chars": 115,
"preview": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace.yaml\n - helm-chart.yaml\n"
},
{
"path": "examples/kustomization_user_deploy/helm-chart/namespace.yaml.tpl",
"chars": 56,
"preview": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: argocd\n"
},
{
"path": "examples/kustomization_user_deploy/letsencrypt/kustomization.yaml.tpl",
"chars": 97,
"preview": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - letsencrypt.yaml\n"
},
{
"path": "examples/kustomization_user_deploy/letsencrypt/letsencrypt.yaml.tpl",
"chars": 462,
"preview": "apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt\n namespace: cert-manager\nspec:\n acme:"
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/base/kustomization.yaml.tpl",
"chars": 0,
"preview": ""
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/base/pod.yaml.tpl",
"chars": 141,
"preview": "apiVersion: v1\nkind: Pod\nmetadata:\n name: myapp-pod\n labels:\n app: myapp\nspec:\n containers:\n - name: nginx\n "
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/kustomization.yaml.tpl",
"chars": 108,
"preview": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace-a\n - namespace-b\n"
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-a/kustomization.yaml.tpl",
"chars": 130,
"preview": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace.yaml\n - ../base\nnamespace: na"
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-a/namespace-a.yaml.tpl",
"chars": 61,
"preview": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: namespace-a\n"
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-b/kustomization.yaml.tpl",
"chars": 130,
"preview": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace.yaml\n - ../base\nnamespace: na"
},
{
"path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-b/namespace-b.yaml.tpl",
"chars": 61,
"preview": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: namespace-b\n"
},
{
"path": "examples/kustomization_user_deploy/simple-resources/demo-config-map.yaml.tpl",
"chars": 106,
"preview": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: demo-config\ndata:\n someConfigKey: ${sealed_secrets_crt}\n"
},
{
"path": "examples/kustomization_user_deploy/simple-resources/demo-pod.yml.tpl",
"chars": 365,
"preview": "apiVersion: v1\nkind: Pod\nmetadata:\n name: demo\nspec:\n containers:\n - name: demo-container\n image: registry.k8s"
},
{
"path": "examples/kustomization_user_deploy/simple-resources/kustomization.yaml.tpl",
"chars": 101,
"preview": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - demo-config-map.yaml\n"
},
{
"path": "examples/micro_os_rollback/Readme.md",
"chars": 3078,
"preview": "# Rollback Node MicroOS Manually\n\nHow to manually rollback a MicroOS node to the last snapshot or be date.\n\n## Backgroun"
},
{
"path": "examples/tls/ingress.yaml",
"chars": 497,
"preview": "apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: nginx-ingress\n annotations:\n traefik.ingress.kubern"
},
{
"path": "examples/tls/pod.yaml",
"chars": 162,
"preview": "apiVersion: v1\nkind: Pod\nmetadata:\n labels:\n run: nginx\n name: nginx\nspec:\n containers:\n - image: nginx\n name:"
},
{
"path": "examples/tls/service.yaml",
"chars": 153,
"preview": "apiVersion: v1\nkind: Service\nmetadata:\n name: nginx-service\nspec:\n ports:\n - port: 80\n protocol: TCP\n targetPor"
},
{
"path": "init.tf",
"chars": 20598,
"preview": "resource \"hcloud_load_balancer\" \"cluster\" {\n count = local.has_external_load_balancer ? 0 : 1\n name = local.load_bala"
},
{
"path": "kube.tf.example",
"chars": 72817,
"preview": "locals {\n # You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env\n # within"
},
{
"path": "kubeconfig.tf",
"chars": 2553,
"preview": "resource \"ssh_sensitive_resource\" \"kubeconfig\" {\n # Note: moved from remote_file to ssh_sensitive_resource because\n # "
},
{
"path": "kustomization_backup.tf",
"chars": 248,
"preview": "resource \"local_file\" \"kustomization_backup\" {\n count = var.create_kustomization ? 1 : 0\n content = "
},
{
"path": "kustomization_user.tf",
"chars": 2652,
"preview": "locals {\n user_kustomization_templates = try(fileset(var.extra_kustomize_folder, \"**/*.yaml.tpl\"), toset([]))\n}\n\nresour"
},
{
"path": "kustomize/flannel-rbac.yaml",
"chars": 275,
"preview": "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: flannel-node-lister\nroleRef:\n apiGr"
},
{
"path": "kustomize/system-upgrade-controller.yaml",
"chars": 445,
"preview": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: system-upgrade-controller\n namespace: system-upgrade\nspec:\n tem"
},
{
"path": "locals.tf",
"chars": 57264,
"preview": "locals {\n # ssh_agent_identity is not set if the private key is passed directly, but if ssh agent is used, the public k"
},
{
"path": "main.tf",
"chars": 2906,
"preview": "resource \"random_password\" \"k3s_token\" {\n length = 48\n special = false\n}\n\ndata \"hcloud_image\" \"microos_x86_snapshot\" "
},
{
"path": "modules/host/locals.tf",
"chars": 533,
"preview": "locals {\n # ssh_agent_identity is not set if the private key is passed directly, but if ssh agent is used, the public k"
},
{
"path": "modules/host/main.tf",
"chars": 10491,
"preview": "resource \"random_string\" \"server\" {\n length = 3\n lower = true\n special = false\n numeric = false\n upper = false"
},
{
"path": "modules/host/out.tf",
"chars": 564,
"preview": "output \"ipv4_address\" {\n value = hcloud_server.server.ipv4_address\n}\n\noutput \"ipv6_address\" {\n value = hcloud_server.s"
},
{
"path": "modules/host/templates/cloudinit.yaml.tpl",
"chars": 2875,
"preview": "#cloud-config\n\nwrite_files:\n\n${cloudinit_write_files_common}\n\n# Apply DNS config\n%{ if has_dns_servers ~}\nmanage_resolv_"
},
{
"path": "modules/host/variables.tf",
"chars": 4260,
"preview": "variable \"name\" {\n description = \"Host name\"\n type = string\n}\nvariable \"microos_snapshot_id\" {\n description = "
},
{
"path": "modules/host/versions.tf",
"chars": 128,
"preview": "terraform {\n required_providers {\n hcloud = {\n source = \"hetznercloud/hcloud\"\n version = \">= 1.51.0\"\n "
},
{
"path": "modules/values_merger/main.tf",
"chars": 528,
"preview": "variable \"default_values\" {\n type = string\n default = \"\"\n}\n\nvariable \"override_values\" {\n type = string\n defau"
},
{
"path": "modules/values_merger/versions.tf",
"chars": 127,
"preview": "terraform {\n required_providers {\n deepmerge = {\n source = \"isometry/deepmerge\"\n version = \"~> 1.0\"\n }"
},
{
"path": "nat-router.tf",
"chars": 6922,
"preview": "locals {\n nat_gateway_ip = var.nat_router != null ? cidrhost(hcloud_network_subnet.nat_router[0].ip_range, 1) : \"\"\n\n n"
},
{
"path": "output.tf",
"chars": 5861,
"preview": "output \"cluster_name\" {\n value = var.cluster_name\n description = \"Shared suffix for all resources belonging to t"
},
{
"path": "packer-template/hcloud-microos-snapshots.pkr.hcl",
"chars": 7004,
"preview": "/*\n * Creates a MicroOS snapshot for Kube-Hetzner\n */\npacker {\n required_plugins {\n hcloud = {\n version = \">= 1"
},
{
"path": "placement_groups.tf",
"chars": 1928,
"preview": "locals {\n control_plane_placement_compat_groups = max(\n 0,\n [\n for cp_pool in var.control_plane_nodepools :\n"
},
{
"path": "scripts/cleanup.sh",
"chars": 6412,
"preview": "#!/usr/bin/env bash\n\nDRY_RUN=1\n\necho \"Welcome to the Kube-Hetzner cluster deletion script!\"\necho \" \"\necho \"We advise you"
},
{
"path": "scripts/create.sh",
"chars": 3540,
"preview": "#!/usr/bin/env bash\n\n# Check if terraform, packer and hcloud CLIs are present\ncommand -v ssh >/dev/null 2>&1 || {\n ec"
},
{
"path": "templates/autoscaler-cloudinit.yaml.tpl",
"chars": 4229,
"preview": "#cloud-config\n\nwrite_files:\n\n${cloudinit_write_files_common}\n\n- content: ${base64encode(k3s_config)}\n encoding: base64\n"
},
{
"path": "templates/autoscaler.yaml.tpl",
"chars": 6442,
"preview": "---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n labels:\n k8s-addon: cluster-autoscaler.addons.k8s.io\n k8s-app:"
},
{
"path": "templates/calico.yaml.tpl",
"chars": 10,
"preview": "${values}\n"
},
{
"path": "templates/ccm.yaml.tpl",
"chars": 990,
"preview": "---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: hcloud-cloud-controller-manager\n namespace: kube-system\nspec"
},
{
"path": "templates/cert_manager.yaml.tpl",
"chars": 351,
"preview": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: cert-manager\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmeta"
},
{
"path": "templates/cilium.yaml.tpl",
"chars": 261,
"preview": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: cilium\n namespace: kube-system\nspec:\n chart: ciliu"
},
{
"path": "templates/csi-driver-smb.yaml.tpl",
"chars": 339,
"preview": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: csi-driver-smb\n namespace: kube-system\nspec:\n char"
},
{
"path": "templates/haproxy_ingress.yaml.tpl",
"chars": 373,
"preview": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${target_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmCha"
},
{
"path": "templates/hcloud-ccm-helm.yaml.tpl",
"chars": 316,
"preview": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: hcloud-cloud-controller-manager\n namespace: kube-sy"
},
{
"path": "templates/hcloud-csi.yaml.tpl",
"chars": 274,
"preview": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: hcloud-csi\n namespace: kube-system\nspec:\n chart: h"
},
{
"path": "templates/kube-hetzner-selinux.te",
"chars": 4510,
"preview": "module kube_hetzner_selinux 1.0;\n\nrequire {\n type kernel_t, bin_t, kernel_generic_helper_t, iscsid_t, iscsid_exec_t, "
},
{
"path": "templates/kube_system_secrets.yaml.tpl",
"chars": 294,
"preview": "%{ for secret_name, secret_values in kube_system_secrets ~}\napiVersion: v1\nkind: Secret\nmetadata:\n name: ${secret_name}"
},
{
"path": "templates/kured.yaml.tpl",
"chars": 435,
"preview": "---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: kured\n namespace: kube-system\nspec:\n selector:\n matchLabe"
},
{
"path": "templates/longhorn.yaml.tpl",
"chars": 357,
"preview": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${longhorn_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmC"
},
{
"path": "templates/nat-router-cloudinit.yaml.tpl",
"chars": 6952,
"preview": "#cloud-config\npackage_reboot_if_required: false\npackage_update: true\npackage_upgrade: true\npackages: \n- fail2ban\n%{ if e"
},
{
"path": "templates/nginx_ingress.yaml.tpl",
"chars": 367,
"preview": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${target_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmCha"
},
{
"path": "templates/plans.yaml.tpl",
"chars": 3456,
"preview": "---\n# Doc: https://rancher.com/docs/k3s/latest/en/upgrades/automated/\n# agent plan\napiVersion: upgrade.cattle.io/v1\nkind"
},
{
"path": "templates/rancher.yaml.tpl",
"chars": 386,
"preview": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: cattle-system\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmet"
},
{
"path": "templates/traefik_ingress.yaml.tpl",
"chars": 353,
"preview": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${target_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmCha"
},
{
"path": "values-export.tf",
"chars": 1885,
"preview": "resource \"local_file\" \"cilium_values\" {\n count = var.export_values && var.cni_plugin == \"cilium\" ? 1 : 0\n co"
},
{
"path": "values-merger.tf",
"chars": 1747,
"preview": "module \"values_merger_cilium\" {\n source = \"./modules/values_merger\"\n default_values = local.cilium_values_de"
},
{
"path": "variables.tf",
"chars": 56548,
"preview": "variable \"hcloud_token\" {\n description = \"Hetzner Cloud API Token.\"\n type = string\n sensitive = true\n}\n\nvari"
},
{
"path": "versions.tf",
"chars": 682,
"preview": "terraform {\n required_version = \">= 1.10.1\"\n required_providers {\n github = {\n source = \"integrations/github\""
}
]
// ... and 1 more files (download for full content)
About this extraction
This page contains the full source code of the mysticaltech/terraform-hcloud-kube-hetzner GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 101 files (742.6 KB), approximately 201.7k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.