[ { "path": ".claude/skills/fix-issue/SKILL.md", "content": "---\nname: fix-issue\ndescription: Use when working on a GitHub issue - fetches issue details, analyzes codebase, implements fix following project methodology\nargs: issue_number\n---\n\n# Fix GitHub Issue\n\n## Overview\n\nGuided workflow for implementing fixes for GitHub issues following the project's CLAUDE.md methodology.\n\n## Usage\n\n```\n/fix-issue \n```\n\n## Workflow\n\n```dot\ndigraph fix_flow {\n rankdir=TB;\n node [shape=box];\n\n fetch [label=\"1. Fetch issue details\"];\n analyze [label=\"2. Analyze issue type\"];\n verify [label=\"3. Verify it's a real bug\"];\n investigate [label=\"4. Deep investigation\"];\n plan [label=\"5. Enter plan mode\"];\n implement [label=\"6. Implement fix\"];\n test [label=\"7. Test changes\"];\n commit [label=\"8. Commit & push\"];\n\n fetch -> analyze;\n analyze -> verify;\n verify -> investigate;\n investigate -> plan;\n plan -> implement;\n implement -> test;\n test -> commit;\n}\n```\n\n## Step 1: Fetch Issue Details\n\n```bash\n# Get issue details\ngh issue view --repo kube-hetzner/terraform-hcloud-kube-hetzner\n\n# CRITICAL: Always read ALL comments - solutions may already be proposed\ngh issue view --repo kube-hetzner/terraform-hcloud-kube-hetzner --comments\n```\n\n## Step 2: Classify Issue Type\n\n| Type | Description | Action |\n|------|-------------|--------|\n| 🔴 **BUG** | Reproducible defect | Fix it |\n| 🟡 **EDGE CASE** | Fails in specific scenario | Evaluate effort vs impact |\n| 🟠 **USER ERROR** | Misconfigured kube.tf | Help user, improve docs |\n| ⚪ **OLD VERSION** | Fixed in newer release | Ask user to upgrade |\n| 🔵 **FEATURE REQUEST** | New functionality | Move to Discussions |\n| ❓ **NEEDS INFO** | Can't reproduce | Ask for more info |\n\n### User Error Indicators\n- kube.tf has obvious mistakes\n- Error indicates syntax/config issue\n- Using deprecated variable names\n- Mixing incompatible options\n- Missing required variables\n\n### Actual Bug Indicators\n- Reproducible with correct config\n- Multiple users report same issue\n- Error in module code, not user config\n- Works in previous version, broke in update\n\n## Step 3: Verify Before Fixing\n\n**CRITICAL: Many issues are user configuration errors, NOT bugs.**\n\nBefore implementing any fix:\n1. Check if the user's kube.tf is correct\n2. Verify the issue exists in the latest version\n3. Try to reproduce the issue locally\n4. Check if there's already a PR addressing this\n\n```bash\n# Search for existing PRs\ngh pr list --search \"\" --repo kube-hetzner/terraform-hcloud-kube-hetzner\n\n# Check if issue is already mentioned in changelog\ngrep -i \"\" CHANGELOG.md\n```\n\n## Step 4: Deep Investigation\n\nRead these files to understand context:\n\n```bash\n# Always start with these\ncat versions.tf # Provider/terraform versions\ncat variables.tf # All configurable options\ncat locals.tf # Core logic and computed values\n\n# Then investigate specific areas based on the issue\n```\n\n### Key Files by Area\n\n| Area | Files to Check |\n|------|---------------|\n| Network | `locals.tf` (subnet calculations), `network.tf` |\n| Control Plane | `control_planes.tf`, `locals.tf` |\n| Agents | `agents.tf`, `autoscaler.tf` |\n| Load Balancer | `load_balancer.tf`, `init.tf` |\n| CNI | `templates/cni/*.yaml.tpl` |\n| Storage | `templates/longhorn.yaml.tpl` |\n| Firewall | `firewall.tf` |\n\n### For Complex Issues - Use AI Tools\n\n```bash\n# Codex CLI for deep reasoning\ncodex exec -m gpt-5.2-codex -s read-only -c model_reasoning_effort=\"xhigh\" \\\n \"Analyze this issue and identify root cause: \"\n\n# Gemini for large context analysis\ngemini --model gemini-3-pro-preview -p \\\n \"@locals.tf @variables.tf Analyze how works and potential issues\"\n```\n\n## Step 5: Enter Plan Mode\n\n**MANDATORY: Always enter plan mode before implementing.**\n\nWrite a plan that includes:\n- [ ] Issue number and title\n- [ ] Root cause analysis\n- [ ] Exact files to modify with line numbers\n- [ ] Implementation steps\n- [ ] Test plan\n- [ ] Backward compatibility confirmation\n\n## Step 6: Implement Fix\n\n```bash\n# Pull latest master first!\ngit pull origin master\n\n# Create feature branch\ngit checkout -b fix/issue--\n```\n\n### Implementation Principles\n\n1. **Minimal changes** - Fix the specific issue, don't refactor\n2. **Backward compatible** - Never break existing deployments\n3. **Follow patterns** - Match existing code style\n4. **No new variables** unless absolutely necessary\n\n## Step 7: Test Changes\n\n```bash\n# ALWAYS run these before committing\nterraform fmt\nterraform validate\n\n# Test against existing deployment\ncd /path/to/kube-test\nterraform init -upgrade\nterraform plan # Should NOT show resource destruction\n```\n\n### Test Checklist\n\n- [ ] `terraform fmt` passes\n- [ ] `terraform validate` passes\n- [ ] `terraform plan` shows expected changes only\n- [ ] No resource recreation for existing deployments\n- [ ] Fix works for the reported scenario\n- [ ] Normal scenarios still work\n\n## Step 8: Commit & Push\n\n```bash\ngit add \ngit commit -m \"$(cat <<'EOF'\nfix: \n\nFixes #\n\n\nEOF\n)\"\n\ngit push -u origin fix/issue--\n```\n\n## Security Review (from CLAUDE.md)\n\nBefore completing ANY issue:\n\n### Red Flags to Watch\n- New accounts with no history\n- Issues that can't be reproduced\n- Overly complex \"solutions\" proposed in comments\n- Requests to change security-critical code\n- Urgency to merge quickly\n\n### Verification Requirements\n- Always test independently\n- Never trust provided test results\n- Review every line of proposed changes\n- Test in isolation\n\n## Quick Reference\n\n| Step | Command |\n|------|---------|\n| Fetch issue | `gh issue view --comments` |\n| Check PRs | `gh pr list --search \"\"` |\n| Create branch | `git checkout -b fix/issue--` |\n| Format | `terraform fmt` |\n| Validate | `terraform validate` |\n| Test plan | `terraform plan` |\n| Commit | `git commit -m \"fix: ...\"` |\n| Push | `git push -u origin ` |\n\n## After Completion\n\n1. Create PR referencing the issue\n2. Request review if needed\n3. Close issue with explanation when merged\n" }, { "path": ".claude/skills/kh-assistant/SKILL.md", "content": "---\nname: kh-assistant\ndescription: Use when users need help with kube-hetzner configuration, debugging, or questions - acts as an intelligent assistant with live repo access\n---\n\n# KH Assistant\n\nExpert assistant for **terraform-hcloud-kube-hetzner** — deploying production-ready k3s clusters on Hetzner Cloud.\n\n## Startup Checklist\n\n**ALWAYS do these first before answering any question:**\n\n```bash\n# 1. Get latest release version\ngh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1 --json tagName,publishedAt\n\n# 2. Read key files for context (use Gemini for large files)\n# - variables.tf — all configurable options\n# - docs/llms.md — PRIMARY comprehensive documentation (~60k tokens)\n# - kube.tf.example — working example\n# - CHANGELOG.md — recent changes\n```\n\n**For Hetzner-specific info** (server types, pricing, locations):\n```bash\n# Use web search\nWebSearch \"hetzner cloud server types pricing 2026\"\n```\n\n---\n\n## Knowledge Sources\n\n### Primary Documentation Files\n\n| File | Purpose | When to Use |\n|------|---------|-------------|\n| `docs/llms.md` | **PRIMARY** - Comprehensive variable reference | First stop for any variable question |\n| `variables.tf` | Variable definitions with types/defaults | Verify exact syntax and defaults |\n| `locals.tf` | Core logic and computed values | Understanding how features work |\n| `kube.tf.example` | Complete working example | Template for configurations |\n| `CHANGELOG.md` | Version history, breaking changes | Upgrade questions, \"when was X added\" |\n| `README.md` | Project overview, quick start | New user orientation |\n\n### Specialized Documentation\n\n| File | Topic |\n|------|-------|\n| `docs/terraform.md` | Auto-generated terraform docs |\n| `docs/ssh.md` | SSH configuration, key formats |\n| `docs/add-robot-server.md` | Hetzner dedicated server integration |\n| `docs/private-network-egress.md` | NAT router setup for private clusters |\n| `docs/customize-mount-path-longhorn.md` | Longhorn storage customization |\n\n### GitHub (Live Data)\n\n```bash\n# Latest release\ngh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1\n\n# Search issues for errors\ngh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --search \"\" --state all\n\n# Search discussions for how-to\ngh api repos/kube-hetzner/terraform-hcloud-kube-hetzner/discussions --jq '.[].title'\n\n# Check if variable exists\ngrep 'variable \"\"' variables.tf\n```\n\n---\n\n## Critical Rules\n\n### MUST Follow — Never Violate\n\n| Rule | Explanation |\n|------|-------------|\n| **At least 1 control plane** | `control_plane_nodepools` must have at least one entry with `count >= 1` |\n| **MicroOS ONLY** | Never suggest Ubuntu, Debian, or any other OS |\n| **Network region coverage** | `network_region` must contain ALL node locations |\n| **Odd control plane counts for HA** | Use 1, 3, or 5 — never 2 or 4 (quorum requirement) |\n| **Autoscaler is separate** | `autoscaler_nodepools` is independent from `agent_nodepools` |\n| **Latest version always** | Always fetch and use the latest release tag |\n\n### Common Mistakes to Prevent\n\n| Mistake | Correct |\n|---------|---------|\n| Empty control_plane_nodepools | At least one with count >= 1 |\n| 2 control planes for \"HA\" | Use 3 (odd number for quorum) |\n| Suggesting Ubuntu | MicroOS only |\n| Location not in network_region | network_region must cover all locations |\n| Confusing autoscaler with agents | Autoscaler pools are completely separate |\n| Using old version | Always check latest release first |\n\n---\n\n## Common Issues Catalog\n\n### Known Error Patterns\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| `cannot sum empty list` | control_plane_nodepools is empty or all counts are 0 | Add at least one control plane with count >= 1 |\n| `NAT router primary IPs will be replaced` | Pre-v2.19.0 used deprecated 'datacenter' attribute | Allow recreation (IPs change) or do state migration |\n| `Traefik returns 404 for all routes` | Traefik v34+ config change | Upgrade to module v2.19.0+ |\n| `SSH connection refused or timeout` | Key format, firewall, or node not ready | Check ssh_public_key format, verify firewall_ssh_source |\n| `Node stuck in NotReady` | Network region mismatch or token issues | Ensure network_region contains all node locations |\n| `Error creating network subnet` | Subnet CIDR conflicts | Check network_ipv4_cidr doesn't overlap with existing |\n| `cloud-init failed` | MicroOS snapshot missing or wrong region | Recreate snapshot with packer in correct region |\n\n### Debugging Workflow\n\n```\n1. Check Common Issues table above\n2. Search GitHub issues: gh issue list --search \"\" --state all\n3. Search docs/llms.md for related variables\n4. Check locals.tf for the logic\n5. Provide: Root cause → Fix → Prevention\n6. Link to relevant GitHub issues if found\n```\n\n---\n\n## Hetzner Cloud Context\n\n### Server Types (x86)\n\n| Type | vCPU | RAM | Disk | Best For |\n|------|------|-----|------|----------|\n| `cpx11` | 2 | 2GB | 40GB | Minimal dev |\n| `cpx21` | 3 | 4GB | 80GB | Dev/small workloads |\n| `cpx31` | 4 | 8GB | 160GB | Production control plane |\n| `cpx41` | 8 | 16GB | 240GB | Production workers |\n| `cpx51` | 16 | 32GB | 360GB | Heavy workloads |\n\n### Server Types (ARM — CAX, cost-optimized)\n\n| Type | vCPU | RAM | Disk | Best For |\n|------|------|-----|------|----------|\n| `cax11` | 2 | 4GB | 40GB | ARM dev |\n| `cax21` | 4 | 8GB | 80GB | ARM workloads |\n| `cax31` | 8 | 16GB | 160GB | ARM production |\n| `cax41` | 16 | 32GB | 320GB | ARM heavy |\n\n### Locations\n\n| Region | Locations | Network Zone |\n|--------|-----------|--------------|\n| Germany | `fsn1`, `nbg1` | `eu-central` |\n| Finland | `hel1` | `eu-central` |\n| USA East | `ash` | `us-east` |\n| USA West | `hil` | `us-west` |\n| Singapore | `sin` | `ap-southeast` |\n\n**Rule**: All locations must be in the same `network_region`.\n\n---\n\n## Configuration Workflows\n\n### Workflow: Creating kube.tf\n\n```\n1. FIRST: Get latest release\n gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1\n\n2. Ask clarifying questions:\n - Use case: Production / Development / Testing?\n - HA: Single node / 3 control planes / Super-HA (multi-location)?\n - Budget: Which server types?\n - Network: Public / Private with NAT router?\n - CNI: Flannel (default) / Cilium / Calico?\n - Storage: Longhorn needed?\n - Ingress: Traefik (default) / Nginx / HAProxy?\n\n3. Query variables.tf and docs/llms.md for relevant options\n\n4. Generate complete config with:\n - Module source and version (latest!)\n - Required: hetzner_token, ssh keys\n - Requested features\n - Helpful comments\n\n5. Validate syntax:\n terraform fmt\n terraform validate\n```\n\n### Workflow: Debugging\n\n```\n1. Parse the error:\n - Terraform error vs k3s error vs provider error\n - Which resource?\n - What operation?\n\n2. Check Common Issues Catalog (above)\n\n3. Search GitHub:\n gh issue list --search \"\" --state all\n\n4. Read relevant code:\n - locals.tf for logic\n - variables.tf for options\n - Specific .tf files based on error\n\n5. Provide solution:\n - Root cause explanation\n - Fix (config change or upgrade)\n - Prevention steps\n - Link to related issues\n```\n\n### Workflow: Feature Questions\n\n```\n1. Check docs/llms.md FIRST (primary reference)\n2. Verify in variables.tf (exact syntax)\n3. Check kube.tf.example for usage\n4. Search GitHub discussions for examples\n5. Provide answer with file references\n```\n\n### Workflow: Upgrades\n\n```\n1. Get current and target versions\n2. Read CHANGELOG.md for breaking changes between versions\n3. Check for:\n - Removed/renamed variables\n - Changed defaults\n - Required migrations\n4. Generate upgrade steps:\n - Update version in kube.tf\n - terraform init -upgrade\n - terraform plan (check for destructions!)\n - terraform apply\n5. Warn if terraform plan shows resource recreation\n```\n\n---\n\n## Configuration Templates\n\n### Minimal Development (Single Node)\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"\" # Always fetch latest!\n\n hetzner_token = var.hetzner_token\n\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n\n network_region = \"eu-central\"\n\n control_plane_nodepools = [\n {\n name = \"control-plane\"\n server_type = \"cpx21\"\n location = \"fsn1\"\n count = 1\n }\n ]\n\n agent_nodepools = []\n\n # Single node: disable auto OS upgrades\n automatically_upgrade_os = false\n}\n```\n\n### Production HA (3 Control Planes + Workers)\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"\"\n\n hetzner_token = var.hetzner_token\n\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n\n network_region = \"eu-central\"\n\n control_plane_nodepools = [\n {\n name = \"control-plane\"\n server_type = \"cpx31\"\n location = \"fsn1\"\n count = 3 # Odd number for quorum!\n }\n ]\n\n agent_nodepools = [\n {\n name = \"worker\"\n server_type = \"cpx41\"\n location = \"fsn1\"\n count = 3\n }\n ]\n\n enable_longhorn = true\n\n # Security: restrict access to your IP\n firewall_kube_api_source = [\"YOUR_IP/32\"]\n firewall_ssh_source = [\"YOUR_IP/32\"]\n}\n```\n\n### Private Cluster with NAT Router\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"\"\n\n hetzner_token = var.hetzner_token\n\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n\n network_region = \"eu-central\"\n\n # Enable NAT router for private egress\n create_nat_router = true\n\n control_plane_nodepools = [\n {\n name = \"control-plane\"\n server_type = \"cpx31\"\n location = \"fsn1\"\n count = 3\n # Disable public IPs\n disable_ipv4 = true\n disable_ipv6 = true\n }\n ]\n\n agent_nodepools = [\n {\n name = \"worker\"\n server_type = \"cpx41\"\n location = \"fsn1\"\n count = 3\n disable_ipv4 = true\n disable_ipv6 = true\n }\n ]\n\n # Optional: keep control plane LB private too\n control_plane_lb_enable_public_interface = false\n}\n```\n\n### Cilium with Hubble Observability\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"\"\n\n hetzner_token = var.hetzner_token\n\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n\n network_region = \"eu-central\"\n\n # Use Cilium CNI\n cni_plugin = \"cilium\"\n\n # Full kube-proxy replacement\n disable_kube_proxy = true\n\n # Enable Hubble for observability\n cilium_hubble_enabled = true\n\n control_plane_nodepools = [\n {\n name = \"control-plane\"\n server_type = \"cpx31\"\n location = \"fsn1\"\n count = 3\n }\n ]\n\n agent_nodepools = [\n {\n name = \"worker\"\n server_type = \"cpx41\"\n location = \"fsn1\"\n count = 3\n }\n ]\n}\n```\n\n### Cost-Optimized ARM Cluster\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"\"\n\n hetzner_token = var.hetzner_token\n\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n\n network_region = \"eu-central\"\n\n # ARM servers (CAX) are ~40% cheaper\n control_plane_nodepools = [\n {\n name = \"control-plane\"\n server_type = \"cax21\" # ARM\n location = \"fsn1\"\n count = 3\n }\n ]\n\n agent_nodepools = [\n {\n name = \"worker-arm\"\n server_type = \"cax31\" # ARM\n location = \"fsn1\"\n count = 3\n }\n ]\n}\n```\n\n### Super-HA Multi-Location\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"\"\n\n hetzner_token = var.hetzner_token\n\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n\n # Must cover ALL locations used\n network_region = \"eu-central\"\n\n # Spread control planes across locations\n control_plane_nodepools = [\n {\n name = \"cp-fsn\"\n server_type = \"cpx31\"\n location = \"fsn1\"\n count = 1\n },\n {\n name = \"cp-nbg\"\n server_type = \"cpx31\"\n location = \"nbg1\"\n count = 1\n },\n {\n name = \"cp-hel\"\n server_type = \"cpx31\"\n location = \"hel1\"\n count = 1\n }\n ]\n\n # Spread workers too\n agent_nodepools = [\n {\n name = \"worker-fsn\"\n server_type = \"cpx41\"\n location = \"fsn1\"\n count = 2\n },\n {\n name = \"worker-nbg\"\n server_type = \"cpx41\"\n location = \"nbg1\"\n count = 2\n },\n {\n name = \"worker-hel\"\n server_type = \"cpx41\"\n location = \"hel1\"\n count = 2\n }\n ]\n\n enable_longhorn = true\n}\n```\n\n---\n\n## Quick Reference\n\n### Variable Lookup\n\n```bash\n# Find specific variable\ngrep -A10 'variable \"\"' variables.tf\n\n# Search by keyword\ngrep -B2 -A10 'description.*' variables.tf\n\n# Use Gemini for comprehensive search\ngemini --model gemini-3-pro-preview -p \"@docs/llms.md Explain the variable\"\n```\n\n### GitHub Commands\n\n```bash\n# Latest release\ngh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1\n\n# Search issues\ngh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --search \"\" --state all\n\n# View specific issue\ngh issue view --repo kube-hetzner/terraform-hcloud-kube-hetzner --comments\n\n# Search discussions\ngh api repos/kube-hetzner/terraform-hcloud-kube-hetzner/discussions --jq '.[].title'\n```\n\n### Validation\n\n```bash\nterraform fmt\nterraform validate\nterraform plan # Check for unexpected changes!\n```\n" }, { "path": ".claude/skills/prepare-release/SKILL.md", "content": "---\nname: prepare-release\ndescription: Use when preparing a release - generates changelog, updates version references, and creates release notes\n---\n\n# Prepare Release\n\n## Overview\n\nPrepare a new release by generating changelog entries, updating version references, and creating release notes.\n\n## Usage\n\n```\n/prepare-release\n```\n\n## IMPORTANT: Releases are Manual\n\n**NEVER create release tags automatically.** The maintainer handles all releases manually.\n\nYour job:\n- Prepare the changelog\n- Update version references\n- Generate release notes draft\n- Commit preparation changes\n\nUser's job:\n- Create the actual tag\n- Push the tag\n- Create GitHub release\n\n## Workflow\n\n```dot\ndigraph release_flow {\n rankdir=TB;\n node [shape=box];\n\n analyze [label=\"1. Analyze changes since last release\"];\n classify [label=\"2. Classify release type\"];\n changelog [label=\"3. Update CHANGELOG.md\"];\n badges [label=\"4. Update version badges\"];\n gpt [label=\"5. Update GPT knowledge\"];\n notes [label=\"6. Generate release notes\"];\n commit [label=\"7. Commit preparation\"];\n\n analyze -> classify;\n classify -> changelog;\n changelog -> badges;\n badges -> gpt;\n gpt -> notes;\n notes -> commit;\n}\n```\n\n## Step 1: Analyze Changes\n\n```bash\n# Get latest release tag\nLATEST=$(gh release list --repo kube-hetzner/terraform-hcloud-kube-hetzner --limit 1 --json tagName --jq '.[0].tagName')\necho \"Latest release: $LATEST\"\n\n# List commits since last release\ngit log $LATEST..HEAD --oneline\n\n# Get detailed changes\ngit log $LATEST..HEAD --pretty=format:\"- %s (%h)\"\n```\n\nUse Gemini for comprehensive analysis:\n\n```bash\ngemini --model gemini-3-pro-preview -p \\\n \"Analyze these git changes for a changelog. Categorize into: Features, Bug Fixes, Breaking Changes, Documentation. Ignore internal refactors.\n\n$(git log $LATEST..HEAD --oneline)\n$(git diff $LATEST..HEAD --stat)\"\n```\n\n## Step 2: Classify Release Type\n\n| Type | When | Example |\n|------|------|---------|\n| **PATCH** (x.x.X) | Bug fixes, docs, deps | 2.19.1 |\n| **MINOR** (x.X.0) | New features, backward compatible | 2.20.0 |\n| **MAJOR** (X.0.0) | Breaking changes | 3.0.0 |\n\n### Breaking Change Indicators\n- Variable removed or renamed\n- Default value changes behavior\n- Resource naming changes (causes recreation)\n- Required migration steps\n\nUse Codex for breaking change analysis:\n\n```bash\ncodex exec -m gpt-5.2-codex -s read-only -c model_reasoning_effort=\"xhigh\" \\\n \"Analyze these changes for breaking changes affecting existing deployments: $(git diff $LATEST..HEAD -- variables.tf locals.tf)\"\n```\n\n## Step 3: Update CHANGELOG.md\n\n### Changelog Format\n\n```markdown\n## [Unreleased]\n\n### ⚠️ Upgrade Notes\n\n\n### 🚀 New Features\n\n\n### 🐛 Bug Fixes\n\n\n### 🔧 Changes\n\n\n### 📚 Documentation\n\n```\n\n### Writing Good Entries\n\n- Write from user's perspective\n- Include issue/PR references: `(#1234)`\n- Be specific about what changed\n- Include migration steps for breaking changes\n\n### Example Entries\n\n```markdown\n### 🚀 New Features\n- **K3s v1.35 Support** - Added support for k3s v1.35 channel (#2029)\n- **NAT Router IPv6** - NAT router now supports IPv6 egress (#2015)\n\n### 🐛 Bug Fixes\n- Fixed autoscaler not respecting max_nodes limit (#2018)\n- Resolved firewall rules not applying to new nodes (#2012)\n\n### ⚠️ Upgrade Notes\n- **NAT Router users**: Run `terraform apply` twice after upgrade due to route changes\n```\n\n## Step 4: Update Version Badges\n\nUpdate README.md badges if version references changed:\n\n```markdown\n[![K3s](https://img.shields.io/badge/K3s-v1.35-FFC61C?style=flat-square&logo=k3s)](https://k3s.io)\n```\n\nCheck `versions.tf` for:\n- Terraform version requirement\n- Provider version requirements\n- K3s default channel\n\n## Step 5: Update GPT Knowledge (if applicable)\n\nIf significant changes, regenerate the Custom GPT knowledge base:\n\n```bash\n# Run the knowledge generation script from CLAUDE.md\npython3 << 'PYEOF'\n# ... (script from CLAUDE.md)\nPYEOF\n```\n\nUpdate `meta.version` in the script to match new release.\n\n## Step 6: Generate Release Notes\n\n### Release Notes Template\n\n```markdown\n## 🚀 Release vX.Y.Z\n\n### Highlights\n\n- **Feature 1**: Brief description\n- **Feature 2**: Brief description\n\n### ⚠️ Upgrade Notes\n\n[Any special upgrade instructions]\n\n### What's Changed\n\n#### New Features\n- Feature description (#PR)\n\n#### Bug Fixes\n- Fix description (#PR)\n\n#### Other Changes\n- Change description (#PR)\n\n### Full Changelog\n\nhttps://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/compare/vPREV...vX.Y.Z\n\n### Upgrade\n\n\\`\\`\\`tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"X.Y.Z\"\n # ...\n}\n\\`\\`\\`\n\n\\`\\`\\`bash\nterraform init -upgrade\nterraform plan\nterraform apply\n\\`\\`\\`\n```\n\n## Step 7: Commit Preparation\n\n```bash\ngit add CHANGELOG.md README.md\ngit commit -m \"$(cat <<'EOF'\nchore: prepare release vX.Y.Z\n\n- Update CHANGELOG.md with release notes\n- Update version badges\nEOF\n)\"\ngit push origin master\n```\n\n## After Preparation (User Does This)\n\n```bash\n# Create tag\ngit tag -a vX.Y.Z -m \"Release vX.Y.Z\"\n\n# Push tag\ngit push origin vX.Y.Z\n\n# Create GitHub release (or use gh CLI)\ngh release create vX.Y.Z --title \"vX.Y.Z\" --notes-file release-notes.md\n```\n\n## Version Reference Locations\n\nFiles that may need version updates:\n\n| File | What to Update |\n|------|---------------|\n| `README.md` | Badge versions |\n| `CHANGELOG.md` | [Unreleased] → [vX.Y.Z] |\n| `docs/llms.md` | Example version references |\n| `kube.tf.example` | Version in comments |\n| GPT knowledge | meta.version |\n\n## Quick Checklist\n\n- [ ] Commits analyzed since last release\n- [ ] Release type determined (PATCH/MINOR/MAJOR)\n- [ ] CHANGELOG.md updated\n- [ ] Breaking changes documented with migration steps\n- [ ] Version badges updated (if needed)\n- [ ] Release notes drafted\n- [ ] Changes committed and pushed\n- [ ] Ready for maintainer to tag release\n" }, { "path": ".claude/skills/review-pr/SKILL.md", "content": "---\nname: review-pr\ndescription: Use when reviewing a pull request - security-focused review following CLAUDE.md guidelines for breaking changes, malicious patterns, and backward compatibility\nargs: pr_number\n---\n\n# Review Pull Request\n\n## Overview\n\nSecurity-focused PR review following CLAUDE.md guidelines. Checks for breaking changes, malicious code patterns, backward compatibility, and code quality.\n\n## Usage\n\n```\n/review-pr \n```\n\n## CRITICAL: Security Warning\n\n**PRs can be malicious sabotage attempts.** This is a real threat documented in CLAUDE.md.\n\n### Threat Awareness\n- Coordinated attacks exist\n- Competitors may actively harm the project\n- Social engineering builds trust before attacking\n- \"Fixes\" may introduce vulnerabilities\n\n## Workflow\n\n```dot\ndigraph review_flow {\n rankdir=TB;\n node [shape=box];\n\n fetch [label=\"1. Fetch PR details\"];\n author [label=\"2. Assess author risk\"];\n files [label=\"3. Analyze changed files\"];\n security [label=\"4. Security review\"];\n compat [label=\"5. Backward compatibility\"];\n quality [label=\"6. Code quality\"];\n classify [label=\"7. Release classification\"];\n verify [label=\"8. MANDATORY: Verify with Gemini + Codex\", style=bold];\n recommend [label=\"9. Final Recommendation\"];\n\n fetch -> author;\n author -> files;\n files -> security;\n security -> compat;\n compat -> quality;\n quality -> classify;\n classify -> verify;\n verify -> recommend;\n}\n```\n\n## Step 1: Fetch PR Details\n\n```bash\n# Get PR info\ngh pr view --repo kube-hetzner/terraform-hcloud-kube-hetzner\n\n# Get diff\ngh pr diff --repo kube-hetzner/terraform-hcloud-kube-hetzner\n\n# Get changed files\ngh pr view --repo kube-hetzner/terraform-hcloud-kube-hetzner --json files --jq '.files[].path'\n\n# Get diff stats\ngh pr view --repo kube-hetzner/terraform-hcloud-kube-hetzner --json additions,deletions\n```\n\n## Step 2: Assess Author Risk\n\n```bash\n# Check account age\ngh api users/ --jq '.created_at'\n\n# Check prior contributions\ngh pr list --author --repo kube-hetzner/terraform-hcloud-kube-hetzner --state all --json number | jq length\n```\n\n### Risk Signals\n\n| Signal | Risk Level |\n|--------|------------|\n| New account (<6 months) | 🔴 HIGH |\n| No prior contributions | 🟡 MEDIUM |\n| First-time contributor | 🟡 MEDIUM |\n| Known contributor | 🟢 LOW |\n| Core maintainer | ⚪ TRUSTED |\n\n## Step 3: Analyze Changed Files\n\n### Security-Critical Files (AUTO HIGH RISK)\n\n```\ninit.tf # Cluster initialization, secrets\nfirewall.tf # Network security\n**/ssh* # SSH configuration\n**/token* # Authentication tokens\n**/*secret* # Secrets handling\n.github/ # CI/CD workflows\nMakefile # Build scripts\nscripts/ # Execution scripts\nversions.tf # Provider dependencies\ntemplates/*.sh # Shell scripts\ncloud-init* # Server initialization\n```\n\n### Risk by File Count\n\n| Files Changed | Risk |\n|---------------|------|\n| 1-3 files | 🟢 LOW |\n| 4-10 files | 🟡 MEDIUM |\n| 11-20 files | 🟡 MEDIUM |\n| >20 files | 🔴 HIGH |\n\n### Risk by Diff Size\n\n| Lines Changed | Risk |\n|---------------|------|\n| <50 lines | 🟢 LOW |\n| 50-200 lines | 🟡 MEDIUM |\n| 200-500 lines | 🟡 MEDIUM |\n| >500 lines | 🔴 HIGH |\n\n## Step 4: Security Review\n\n### Checklist\n\n- [ ] No hardcoded credentials or tokens\n- [ ] No suspicious external URLs\n- [ ] No obfuscated code\n- [ ] Changes match stated purpose\n- [ ] No unnecessary permission escalations\n- [ ] CI/CD changes justified\n- [ ] No bypassing existing security patterns\n\n### Red Flags\n\n| Pattern | Concern |\n|---------|---------|\n| Base64 encoded strings | Hidden payloads |\n| External curl/wget calls | Code injection |\n| Eval or exec statements | Command injection |\n| Overly complex logic | Hiding malicious code |\n| Unnecessary file access | Data exfiltration |\n| Changes to .gitignore | Hiding tracks |\n\n### Use AI for Deep Analysis\n\n```bash\n# Codex for security analysis\ncodex exec -m gpt-5.3-codex -s read-only -c model_reasoning_effort=\"xhigh\" \\\n \"Analyze this PR diff for security vulnerabilities and malicious patterns: $(gh pr diff )\"\n\n# Gemini for broad context\ngemini --model gemini-3-pro-preview -p \\\n \"@locals.tf @init.tf Does this PR introduce any security concerns? $(gh pr diff )\"\n```\n\n## Step 5: Backward Compatibility\n\n**CRITICAL: Any PR that causes resource recreation is a MAJOR release.**\n\n### Breaking Change Indicators\n\n- Removes or renames variables\n- Changes variable defaults that affect behavior\n- Modifies resource naming patterns\n- Alters subnet/network calculations\n- Changes resource keys (causes recreation)\n- Removes outputs\n- Modifies provider requirements\n\n### Test for Breaking Changes\n\n```bash\n# Checkout PR locally\ngh pr checkout \n\n# Test against existing cluster\ncd /path/to/kube-test\nterraform init -upgrade\nterraform plan\n```\n\n**If `terraform plan` shows ANY resource destruction → MAJOR release required**\n\n### Compatibility Checklist\n\n- [ ] No variable removals\n- [ ] No default changes that affect behavior\n- [ ] No resource naming changes\n- [ ] `terraform plan` shows no destruction\n- [ ] Existing deployments unaffected\n\n## Step 6: Code Quality\n\n### Style\n- [ ] Follows existing patterns\n- [ ] Consistent naming\n- [ ] Proper formatting (`terraform fmt`)\n- [ ] No unnecessary complexity\n\n### Logic\n- [ ] Changes are correct\n- [ ] Edge cases handled\n- [ ] No regressions introduced\n- [ ] Tests pass\n\n## Step 7: Release Classification\n\n### PATCH (x.x.PATCH)\n- Bug fixes only\n- No new features\n- Fully backward compatible\n- No terraform state impact\n\n### MINOR (x.MINOR.0)\n- New features (backward compatible)\n- New optional variables with defaults\n- Deprecation warnings (not removals)\n\n### MAJOR (MAJOR.0.0)\n- Breaking changes\n- Removed/renamed variables\n- Changed defaults affecting behavior\n- State migrations required\n- Resource recreations\n\n## Step 8: MANDATORY - Verify with Gemini and Codex\n\n**CRITICAL: Before making your final recommendation, you MUST run both Gemini and Codex to triple-verify the PR.**\n\nThis is not optional. External AI verification catches issues that may be missed in the initial review.\n\n### Run Both in Parallel\n\n```bash\n# Gemini - Broad context analysis (run first or in parallel)\ngemini --model gemini-3-pro-preview -p \"@control_planes.tf @locals.tf @init.tf\n\nAnalyze this PR diff for the kube-hetzner terraform module:\n\n$(gh pr diff --repo kube-hetzner/terraform-hcloud-kube-hetzner)\n\nQuestions:\n1. Is this change consistent with existing patterns in the codebase?\n2. Are there any security concerns?\n3. Could this cause breaking changes or resource recreation?\n4. Is this a legitimate bug fix or could it be malicious?\"\n\n# Codex - Deep reasoning security analysis (run in parallel)\ncodex exec -m gpt-5.3-codex -s read-only -c model_reasoning_effort=\"xhigh\" \\\n\"Analyze this Terraform PR for the kube-hetzner module.\n\nDIFF:\n$(gh pr diff --repo kube-hetzner/terraform-hcloud-kube-hetzner)\n\nSECURITY ANALYSIS QUESTIONS:\n1. Could this change introduce any security vulnerabilities?\n2. Could this be a malicious change disguised as a bug fix?\n3. Will this cause any Terraform state changes or resource recreation?\n4. Is this pattern safe and consistent with Terraform best practices?\n5. Any edge cases or potential issues?\"\n```\n\n### Verification Checklist\n\n- [ ] Gemini analysis completed\n- [ ] Codex analysis completed\n- [ ] Both agree the change is safe\n- [ ] No concerns raised by either tool\n- [ ] If concerns raised, they have been addressed or explained\n\n### When Reviewers Disagree\n\nIf Gemini or Codex raises concerns that you didn't catch:\n1. **Take the concern seriously** - investigate further\n2. **Re-read the code** with the concern in mind\n3. **Request changes** if the concern is valid\n4. **Document** why the concern was dismissed if you determine it's a false positive\n\n### Output in Final Review\n\nInclude a summary of external verification:\n\n```markdown\n### External AI Verification\n\n| Reviewer | Verdict | Key Finding |\n|----------|---------|-------------|\n| Claude | ✅/❌ | |\n| Gemini | ✅/❌ | |\n| Codex | ✅/❌ | |\n\n**Consensus:** All reviewers agree / Disagreement on X\n```\n\n---\n\n## Step 9: Final Recommendation\n\n### PR Review Output Template\n\n```markdown\n## PR Review: #\n\n**Title:** \n**Author:** @<username>\n**Files:** <count> files changed (+<additions>/-<deletions>)\n\n### Risk Assessment\n\n| Factor | Value | Risk |\n|--------|-------|------|\n| Author tenure | X months | 🟢/🟡/🔴 |\n| Prior contributions | N PRs | 🟢/🟡/🔴 |\n| Files changed | N files | 🟢/🟡/🔴 |\n| Lines changed | +X/-Y | 🟢/🟡/🔴 |\n| Security-critical files | Yes/No | 🟢/🔴 |\n| External dependencies | Yes/No | 🟢/🔴 |\n\n**Overall Risk:** 🔴 HIGH / 🟡 MEDIUM / 🟢 LOW\n\n### Security Review\n\n- [ ] No hardcoded credentials\n- [ ] No suspicious external URLs\n- [ ] No obfuscated code\n- [ ] Changes match stated purpose\n\n### Backward Compatibility\n\n- [ ] No breaking changes\n- [ ] terraform plan shows no destruction\n- [ ] Existing deployments unaffected\n\n### Release Classification\n\n**Type:** PATCH / MINOR / MAJOR\n**Reason:** <explanation>\n\n### External AI Verification\n\n| Reviewer | Verdict | Key Finding |\n|----------|---------|-------------|\n| Claude | ✅/❌ | <summary> |\n| Gemini | ✅/❌ | <summary> |\n| Codex | ✅/❌ | <summary> |\n\n**Consensus:** All agree / Disagreement on X\n\n### Recommendation\n\n**Action:** APPROVE / REQUEST CHANGES / CLOSE\n**Notes:** <specific concerns or required changes>\n```\n\n## Quick Commands\n\n```bash\n# Approve PR\ngh pr review <num> --approve --body \"LGTM! ...\"\n\n# Request changes\ngh pr review <num> --request-changes --body \"Please address: ...\"\n\n# Comment\ngh pr review <num> --comment --body \"...\"\n\n# Merge (after approval)\ngh pr merge <num> --squash --delete-branch\n```\n\n## Never Merge Directly to Master\n\nAll PRs go through staging branches first:\n\n1. Create staging branch\n2. Test thoroughly\n3. Get AI review (Codex + Gemini)\n4. Then merge to master\n" }, { "path": ".claude/skills/sync-docs/SKILL.md", "content": "---\nname: sync-docs\ndescription: Use when documentation needs updating - ensures variables.tf, llms.md, kube.tf.example, and README are in sync\n---\n\n# Sync Documentation\n\n## Overview\n\nEnsure documentation is synchronized across all key files when variables or features change.\n\n## Usage\n\n```\n/sync-docs\n```\n\n## Documentation Files\n\n| File | Purpose | Priority |\n|------|---------|----------|\n| `variables.tf` | Source of truth for all variables | PRIMARY |\n| `docs/llms.md` | Comprehensive variable reference | HIGH |\n| `kube.tf.example` | Working example configuration | HIGH |\n| `README.md` | Project overview and quick start | MEDIUM |\n| `docs/terraform.md` | Auto-generated terraform docs | AUTO |\n\n## Workflow\n\n```dot\ndigraph sync_flow {\n rankdir=TB;\n node [shape=box];\n\n extract [label=\"1. Extract from variables.tf\"];\n compare [label=\"2. Compare with llms.md\"];\n gaps [label=\"3. Identify gaps\"];\n update_llms [label=\"4. Update llms.md\"];\n update_example [label=\"5. Update kube.tf.example\"];\n update_readme [label=\"6. Update README if needed\"];\n verify [label=\"7. Verify consistency\"];\n\n extract -> compare;\n compare -> gaps;\n gaps -> update_llms;\n update_llms -> update_example;\n update_example -> update_readme;\n update_readme -> verify;\n}\n```\n\n## Step 1: Extract Variables from Source\n\nUse Gemini for large file analysis:\n\n```bash\n# List all variables from variables.tf\ngemini --model gemini-3-pro-preview -p \"@variables.tf List ALL variable names defined in this file, one per line\"\n\n# Get variable details\ngemini --model gemini-3-pro-preview -p \"@variables.tf For variable '<name>', provide: type, default, description\"\n```\n\n## Step 2: Find Undocumented Variables\n\n```bash\n# Compare variables.tf with llms.md\ngemini --model gemini-3-pro-preview -p \\\n \"@variables.tf @docs/llms.md List ALL variables from variables.tf that are NOT documented in llms.md. Output one per line.\"\n```\n\n## Step 3: Generate Documentation\n\n### llms.md Format\n\n```markdown\n**Variable Name**\n\n```tf\nvariable_name = \"default_value\"\n```\n\n* **`variable_name` (Type, Optional/Required):**\n * **Default:** `default_value`\n * **Purpose:** Clear explanation of what this does\n * **Usage:** When and how to use it\n * **Considerations:** Important notes, limitations, impacts\n * **Example:** Practical usage example if helpful\n```\n\n### kube.tf.example Format\n\n```tf\n # Description of what this controls\n # Additional context if needed\n # variable_name = \"default_value\"\n```\n\n## Step 4: Update llms.md\n\nFor each undocumented variable:\n\n1. Read variable definition from `variables.tf`\n2. Understand its usage in `locals.tf` and other files\n3. Write comprehensive documentation following the format above\n4. Place in appropriate section of `llms.md`\n\n### Section Organization in llms.md\n\n| Section | Variables |\n|---------|-----------|\n| Cluster Basics | cluster_name, hetzner_token, ssh_* |\n| Network | network_*, subnet_* |\n| Control Plane | control_plane_* |\n| Agents | agent_*, autoscaler_* |\n| Load Balancer | lb_*, traefik_*, nginx_* |\n| CNI | cni_*, cilium_*, calico_* |\n| Storage | longhorn_* |\n| Security | firewall_*, audit_* |\n| Advanced | Additional/misc options |\n\n## Step 5: Update kube.tf.example\n\nEnsure new variables appear in the example with:\n- Clear comment explaining purpose\n- Commented out with default value\n- Grouped with related variables\n\n```bash\n# Check what's in example vs variables.tf\ngemini --model gemini-3-pro-preview -p \\\n \"@variables.tf @kube.tf.example List variables from variables.tf missing from kube.tf.example\"\n```\n\n## Step 6: Update README if Needed\n\nUpdate README.md if:\n- New major feature added\n- New CNI or ingress option\n- Significant capability change\n\nFeatures section should match actual capabilities.\n\n## Step 7: Verify Consistency\n\n```bash\n# Final verification\ngemini --model gemini-3-pro-preview -p \\\n \"@variables.tf @docs/llms.md @kube.tf.example Verify these files are consistent. List any discrepancies.\"\n```\n\n### Verification Checklist\n\n- [ ] All variables.tf variables documented in llms.md\n- [ ] All major variables appear in kube.tf.example\n- [ ] README features match actual capabilities\n- [ ] No typos in variable names across files\n- [ ] Default values consistent across docs\n\n## Common Sync Issues\n\n### Variable renamed\n1. Update in variables.tf\n2. Search and replace in llms.md\n3. Search and replace in kube.tf.example\n4. Add to CHANGELOG.md (breaking change!)\n\n### Variable removed\n1. Remove from variables.tf\n2. Remove from llms.md\n3. Remove from kube.tf.example\n4. Add to CHANGELOG.md (breaking change!)\n\n### Default changed\n1. Update in variables.tf\n2. Update in llms.md\n3. Update in kube.tf.example\n4. Consider if this is a breaking change\n\n## Quick Commands\n\n```bash\n# Regenerate terraform docs\nterraform-docs markdown . > docs/terraform.md\n\n# Search for variable across all docs\ngrep -r \"variable_name\" docs/ kube.tf.example README.md\n\n# Find undocumented variables (quick check)\ndiff <(grep -oP 'variable \"\\K[^\"]+' variables.tf | sort) \\\n <(grep -oP '`\\K[a-z_]+(?=`)' docs/llms.md | sort -u) | grep \"^<\"\n```\n\n## After Sync\n\n1. Run `terraform fmt`\n2. Commit with message: `docs: sync documentation with variables.tf`\n3. If breaking changes, update CHANGELOG.md\n" }, { "path": ".claude/skills/test-changes/SKILL.md", "content": "---\nname: test-changes\ndescription: Use after making changes to run terraform fmt, validate, and plan against test environment\n---\n\n# Test Terraform Changes\n\n## Overview\n\nRun the standard validation suite for terraform changes against the test environment.\n\n## Usage\n\n```\n/test-changes\n```\n\n## Test Environment\n\n- **Module code:** `/Volumes/MysticalTech/Code/kube-hetzner`\n- **Test cluster:** `/Users/karim/Code/kube-test`\n\n## Workflow\n\n```dot\ndigraph test_flow {\n rankdir=TB;\n node [shape=box];\n\n fmt [label=\"1. terraform fmt\"];\n validate [label=\"2. terraform validate\"];\n init [label=\"3. terraform init -upgrade\"];\n plan [label=\"4. terraform plan\"];\n review [label=\"5. Review plan output\"];\n\n fmt -> validate;\n validate -> init;\n init -> plan;\n plan -> review;\n}\n```\n\n## Step 1: Format Check\n\n```bash\ncd /Volumes/MysticalTech/Code/kube-hetzner\nterraform fmt -recursive\n```\n\n**Must pass before proceeding.**\n\n## Step 2: Validate Module\n\n```bash\ncd /Volumes/MysticalTech/Code/kube-hetzner\nterraform validate\n```\n\n**Must pass before proceeding.**\n\n## Step 3: Initialize Test Environment\n\n```bash\ncd /Users/karim/Code/kube-test\nterraform init -upgrade\n```\n\nThis picks up changes from the local module.\n\n## Step 4: Plan Against Test Cluster\n\n```bash\ncd /Users/karim/Code/kube-test\nterraform plan\n```\n\n### What to Look For\n\n#### Good Signs\n- Only expected resources change\n- No unexpected additions/deletions\n- Changes match your intended modifications\n\n#### Red Flags (STOP!)\n\n| Output | Meaning | Action |\n|--------|---------|--------|\n| `will be destroyed` | Resource recreation | **STOP** - Breaking change |\n| `must be replaced` | Resource recreation | **STOP** - Breaking change |\n| `forces replacement` | Resource recreation | **STOP** - Breaking change |\n| Unexpected changes | Side effects | Investigate before proceeding |\n\n### Breaking Change = MAJOR Release\n\nIf `terraform plan` shows ANY resource destruction on existing infrastructure:\n1. **STOP** - This is NOT backward compatible\n2. The change requires a MAJOR version bump\n3. Migration guide is required\n4. Consider alternative approaches first\n\n## Step 5: Review Plan Output\n\n### Checklist\n\n- [ ] `terraform fmt` passes\n- [ ] `terraform validate` passes\n- [ ] `terraform plan` shows expected changes only\n- [ ] No resource destruction\n- [ ] No unexpected side effects\n- [ ] Changes are backward compatible\n\n## Quick Reference\n\n```bash\n# Full test sequence\ncd /Volumes/MysticalTech/Code/kube-hetzner && \\\nterraform fmt -recursive && \\\nterraform validate && \\\ncd /Users/karim/Code/kube-test && \\\nterraform init -upgrade && \\\nterraform plan\n```\n\n## Apply (Optional)\n\nOnly if plan looks correct and you want to test on actual infrastructure:\n\n```bash\ncd /Users/karim/Code/kube-test\nterraform apply\n```\n\n**Caution:** This modifies real infrastructure. Only do this for thorough testing.\n\n## Common Issues\n\n### \"Provider version constraints\"\n```bash\nterraform init -upgrade\n```\n\n### \"Module source has changed\"\n```bash\nterraform init -upgrade\n```\n\n### \"State lock\"\nSomeone else may be running terraform. Wait or:\n```bash\nterraform force-unlock <lock-id>\n```\n\n### Validation errors\nCheck the error message - usually points to:\n- Missing required variable\n- Type mismatch\n- Invalid reference\n\n## AI-Assisted Review\n\nFor complex changes, get AI review:\n\n```bash\n# Codex for correctness\ncodex exec -m gpt-5.2-codex -s read-only -c model_reasoning_effort=\"xhigh\" \\\n \"Review these terraform changes for issues: $(git diff)\"\n\n# Gemini for broad impact\ngemini --model gemini-3-pro-preview -p \\\n \"@locals.tf @variables.tf Analyze impact of these changes: $(git diff)\"\n```\n" }, { "path": ".claude/skills/triage-issue/SKILL.md", "content": "---\nname: triage-issue\ndescription: Use when triaging a GitHub issue - analyzes issue, checks for duplicates, categorizes, and drafts response\nargs: issue_number\n---\n\n# Triage GitHub Issue\n\n## Overview\n\nAnalyze a GitHub issue, classify it, check for duplicates, and draft an appropriate response.\n\n## Usage\n\n```\n/triage-issue <number>\n```\n\n## Workflow\n\n```dot\ndigraph triage_flow {\n rankdir=TB;\n node [shape=box];\n\n fetch [label=\"1. Fetch issue + comments\"];\n classify [label=\"2. Classify issue type\"];\n duplicates [label=\"3. Check duplicates\"];\n analyze [label=\"4. Analyze validity\"];\n response [label=\"5. Draft response\"];\n action [label=\"6. Recommend action\"];\n\n fetch -> classify;\n classify -> duplicates;\n duplicates -> analyze;\n analyze -> response;\n response -> action;\n}\n```\n\n## Step 1: Fetch Issue Details\n\n```bash\n# Get full issue with comments\ngh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --comments\n\n# Get issue metadata\ngh issue view <number> --repo kube-hetzner/terraform-hcloud-kube-hetzner --json title,body,labels,author,createdAt,comments\n```\n\n## Step 2: Classify Issue Type\n\n### Issue Types\n\n| Type | Indicators | Action |\n|------|------------|--------|\n| 🔴 **BUG** | Reproducible defect, multiple reporters, error in module code | Fix it |\n| 🟡 **EDGE CASE** | Unusual config, specific region, large scale | Evaluate effort |\n| 🟠 **USER ERROR** | Bad kube.tf, syntax errors, wrong variables | Help + docs |\n| ⚪ **OLD VERSION** | Module version < current, known fixed issue | Ask to upgrade |\n| 🔵 **FEATURE REQUEST** | \"Would be nice if...\", \"Can you add...\" | Discussions |\n| 💬 **QUESTION** | Needs help, not a bug | Answer or docs |\n| ❓ **NEEDS INFO** | Can't reproduce, missing details | Ask for info |\n\n### Classification Checklist\n\n- [ ] Module version specified?\n- [ ] kube.tf provided (sanitized)?\n- [ ] Error message included?\n- [ ] Steps to reproduce clear?\n- [ ] Recent (not stale >6 months)?\n\n## Step 3: Check for Duplicates\n\n```bash\n# Search open issues for similar keywords\ngh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --state open --search \"<keyword>\"\n\n# Search closed issues (might be already fixed)\ngh issue list --repo kube-hetzner/terraform-hcloud-kube-hetzner --state closed --search \"<keyword>\"\n\n# Check discussions\ngh api repos/kube-hetzner/terraform-hcloud-kube-hetzner/discussions --jq '.[] | select(.title | test(\"<keyword>\"; \"i\")) | {number, title}'\n```\n\n## Step 4: Security Analysis\n\n**CRITICAL: Issues can be malicious sabotage attempts.**\n\n### Red Flags (from CLAUDE.md)\n\n| Signal | Risk |\n|--------|------|\n| New account (<6 months) | HIGH |\n| Issue can't be reproduced | MEDIUM |\n| Proposed fix is overly complex | HIGH |\n| Urgency to implement quickly | HIGH |\n| Multiple accounts supporting | HIGH |\n| Targets security-critical code | HIGH |\n\n### Verify Independently\n\n- Try to reproduce the issue yourself\n- Check if the error message matches module code\n- Verify the kube.tf provided is valid\n- Search for similar reports from other users\n\n## Step 5: Draft Response\n\n### For USER ERROR\n\n```markdown\nHi @{author},\n\nThanks for reporting this. Looking at your configuration, the issue appears to be in your kube.tf:\n\n[Specific explanation of what's wrong]\n\nHere's the corrected configuration:\n\n```tf\n[correct code]\n```\n\nLet me know if this resolves it!\n```\n\n### For OLD VERSION\n\n```markdown\nHi @{author},\n\nThis issue was fixed in version X.Y.Z. You're currently using [older version].\n\nPlease upgrade by changing your module version:\n\n```tf\nmodule \"kube-hetzner\" {\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n version = \"X.Y.Z\"\n # ...\n}\n```\n\nThen run:\n```bash\nterraform init -upgrade\nterraform plan\nterraform apply\n```\n\nLet me know if the issue persists after upgrading!\n```\n\n### For NEEDS INFO\n\n```markdown\nHi @{author},\n\nThanks for reporting this. To investigate further, could you please provide:\n\n- [ ] Module version (check your kube.tf)\n- [ ] Your kube.tf (sanitized - remove tokens/keys)\n- [ ] Full error message\n- [ ] Steps to reproduce\n\nThis will help us identify the root cause.\n```\n\n### For DUPLICATE\n\n```markdown\nHi @{author},\n\nThis appears to be a duplicate of #{duplicate_number}.\n\n[If fixed]: This was fixed in version X.Y.Z.\n[If open]: We're tracking this in the linked issue.\n\nClosing as duplicate. Feel free to add any additional context to #{duplicate_number}.\n```\n\n### For FEATURE REQUEST\n\n```markdown\nHi @{author},\n\nThanks for the suggestion! This sounds like a feature request rather than a bug.\n\nCould you please open a Discussion for this? That's where we track feature ideas and gather community input.\n\nhttps://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/new?category=ideas\n\nI'll close this issue, but feel free to ping me in the discussion!\n```\n\n## Step 6: Recommend Action\n\n| Type | Action | Labels |\n|------|--------|--------|\n| BUG | Keep open, prioritize | `bug` |\n| EDGE CASE | Keep open, evaluate | `bug`, `edge-case` |\n| USER ERROR | Close with help | `user-config` |\n| OLD VERSION | Close | `old-version` |\n| FEATURE REQUEST | Move to Discussions | - |\n| QUESTION | Answer and close | `question` |\n| NEEDS INFO | Keep open, add label | `needs-info` |\n\n## Triage Output Template\n\n```markdown\n## Triage Summary: Issue #<number>\n\n**Title:** <title>\n**Author:** @<username>\n**Created:** <date>\n\n### Classification\n\n**Type:** <BUG/EDGE CASE/USER ERROR/OLD VERSION/FEATURE/QUESTION/NEEDS INFO>\n**Confidence:** HIGH/MEDIUM/LOW\n**Reason:** <why this classification>\n\n### Checklist\n\n- [ ] Module version: <version or \"not specified\">\n- [ ] kube.tf provided: Yes/No/Partial\n- [ ] Reproducible: Yes/No/Unknown\n- [ ] Duplicate: No / Yes → #<number>\n\n### Analysis\n\n<What's actually happening and why>\n\n### Recommended Action\n\n**Action:** <FIX/HELP USER/CLOSE/MOVE TO DISCUSSIONS/NEEDS INFO>\n**Priority:** HIGH/MEDIUM/LOW\n**Response:** <drafted response above>\n```\n\n## Quick Commands\n\n```bash\n# Add label\ngh issue edit <num> --add-label \"bug\"\n\n# Close issue\ngh issue close <num> --comment \"Closing because...\"\n\n# Close as not planned\ngh issue close <num> --reason \"not planned\" --comment \"...\"\n\n# Transfer to discussions\ngh issue transfer <num> --repo kube-hetzner/terraform-hcloud-kube-hetzner\n```\n" }, { "path": ".github/FUNDING.yml", "content": "# These are supported funding model platforms\n\ngithub: mysticaltech\n" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.yml", "content": "name: Bug Report\ndescription: File a bug report\ntitle: \"[Bug]: \"\nlabels: [\"bug\", \"triage\"]\nbody:\n - type: markdown\n attributes:\n value: |\n **Kube-Hetzner Bug Report**\n \n Before doing so, please research the project, the solution may be in the documentation, in other issues, or in the discussions. If none of the above gave you the answer, please explain the bug in detail and provide as much information as possible.\n - type: textarea\n id: description\n attributes:\n label: Description\n description: A clear and concise description of what the bug is.\n placeholder: What's happening?\n validations:\n required: true\n - type: textarea\n id: kube_tf\n attributes:\n label: Kube.tf file\n description: Please share your kube.tf file, without sensitive values, and if possible, stripped of comments.\n placeholder: Enter your kube.tf content goes here\n render: terraform\n validations:\n required: true\n - type: textarea\n id: screenshots\n attributes:\n label: Screenshots\n description: If applicable, add screenshots of the errors.\n placeholder: Enter screenshots here\n - type: input\n id: platform\n attributes:\n label: Platform\n description: Windows, Linux, Mac\n placeholder: Enter platform here\n validations:\n required: true\n" }, { "path": ".github/ISSUE_TEMPLATE/config.yml", "content": "blank_issues_enabled: true\ncontact_links:\n - name: If you have questions, use the discussions\n url: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions\n about: Please ask and answer questions here.\n" }, { "path": ".github/ISSUE_TEMPLATE/feature_request.yml", "content": "name: Feature Request\ntitle: \"[Feature Request]: \"\ndescription: \"Submit a feature request for consideration\"\nlabels: [\"feature request\"]\nbody:\n - type: markdown\n attributes:\n value: |\n **Kube-Hetzner Feature Request**\n\n For feature exploration, please use our [discussions](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions) section. However, if you judge something to be extremely important or urgent, please let us know what you need and why, and even share tips on implementation.\n\n Also please know that we are very open to PRs and will work with you to make it happen if fully relevent to the project. So if you can make it happen, and it falls within the project, please do not hesitate to ping us in the discussions about your idea.\n - type: textarea\n id: feature-description\n attributes:\n label: Description\n description: Tell us more about your feature request.\n placeholder: \"E.g. Adding support for XYZ would greatly improve the user experience...\"\n validations:\n required: true\n" }, { "path": ".github/dependabot.yaml", "content": "version: 2\nupdates:\n - package-ecosystem: \"terraform\"\n directory: \"/\"\n schedule:\n interval: \"daily\"\n\n - package-ecosystem: \"github-actions\"\n directory: \"/\"\n schedule:\n interval: \"daily\"\n" }, { "path": ".github/release.yaml", "content": "changelog:\n exclude:\n labels:\n - ignore-for-release\n authors:\n - octocat\n categories:\n - title: Breaking Changes 🛠\n labels:\n - Semver-Major\n - breaking-change\n - title: New Features 🎉\n labels:\n - Semver-Minor\n - enhancement\n - title: Bug Fixes 🐛\n labels:\n - Semver-Patch\n - bug\n - title: Other Changes\n labels:\n - \"*\"\n" }, { "path": ".github/release.yml", "content": "# Configuration for auto-generated release notes\n# https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes\n\nchangelog:\n exclude:\n labels:\n - skip-changelog\n authors:\n - dependabot\n - dependabot[bot]\n\n categories:\n - title: \"🚀 New Features\"\n labels:\n - enhancement\n - feature\n\n - title: \"🐛 Bug Fixes\"\n labels:\n - bug\n - fix\n - bugfix\n\n - title: \"📦 Packer / OS\"\n labels:\n - packer\n - microos\n\n - title: \"🔧 Configuration & Variables\"\n labels:\n - configuration\n - variables\n\n - title: \"📚 Documentation\"\n labels:\n - documentation\n - docs\n\n - title: \"🏗️ Infrastructure\"\n labels:\n - infrastructure\n - terraform\n\n - title: \"⬆️ Dependencies\"\n labels:\n - dependencies\n\n - title: \"🔒 Security\"\n labels:\n - security\n\n - title: \"Other Changes\"\n labels:\n - \"*\"\n" }, { "path": ".github/workflows/generate-docs.yaml", "content": "name: Generate terraform docs\non:\n push:\n branches:\n - master\n - staging\n\njobs:\n docs:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n with:\n ref: ${{ github.event.pull_request.head.ref }}\n fetch-depth: 0 # Necessary to fetch all history for create-pull-request to work correctly\n\n - name: Render terraform docs and push changes back to PR\n uses: terraform-docs/gh-actions@main\n with:\n working-dir: .\n output-file: docs/terraform.md\n output-method: inject\n config-file: \".terraform-docs.yml\"\n\n - name: Create Pull Request\n uses: peter-evans/create-pull-request@v8\n with:\n token: ${{ secrets.GITHUB_TOKEN }}\n commit-message: Update Terraform documentation\n title: \"[AUTO] Update Terraform Documentation\"\n body: \"Automated changes by GitHub Actions\"\n branch: \"docs/update-${{ github.head_ref }}\"\n labels: documentation\n" }, { "path": ".github/workflows/lint_pr.yaml", "content": "name: Lint\n\non:\n pull_request:\n\njobs:\n tfsec:\n runs-on: ubuntu-latest\n permissions:\n contents: read\n pull-requests: write\n name: Scan terraform files with tfsec\n steps:\n - name: Clone repo\n uses: actions/checkout@v6\n\n - name: tfsec\n uses: aquasecurity/tfsec-pr-commenter-action@v1.3.1\n with:\n github_token: ${{ github.token }}\n tfsec_args: --ignore-hcl-errors\n\n - name: Run tfsec with reviewdog output on the PR\n uses: reviewdog/action-tfsec@v1.30.0\n with:\n github_token: ${{ secrets.github_token }}\n filter_mode: nofilter\n fail_on_error: true\n tfsec_flags: --ignore-hcl-errors\n\n validate:\n runs-on: ubuntu-latest\n name: Validate terraform configuration\n steps:\n - name: Checkout\n uses: actions/checkout@v6\n\n - name: terraform validate\n uses: dflook/terraform-validate@v2.2.3\n\n fmt-check:\n runs-on: ubuntu-latest\n name: Check formatting of terraform files\n steps:\n - name: Checkout\n uses: actions/checkout@v6\n\n - name: terraform fmt\n uses: dflook/terraform-fmt-check@v2.2.3\n" }, { "path": ".github/workflows/publish-release.yaml", "content": "---\nname: Publish a new Github Release\n\non:\n push:\n tags:\n - '*'\n workflow_dispatch:\n\njobs:\n Release:\n name: Release\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n with:\n fetch-depth: 0 # Full history for contributor extraction\n\n - name: Get previous tag\n id: prev_tag\n run: |\n PREV_TAG=$(git describe --tags --abbrev=0 HEAD^ 2>/dev/null || echo \"\")\n echo \"tag=${PREV_TAG}\" >> \"$GITHUB_OUTPUT\"\n\n - name: Extract changelog for this release\n run: |\n # Extract the [Unreleased] section from CHANGELOG.md\n if [ -f CHANGELOG.md ]; then\n # Get content between [Unreleased] and the next ## heading\n awk '/^## \\[Unreleased\\]/{flag=1; next} /^## \\[/{flag=0} flag' CHANGELOG.md > changelog_section.md\n\n if [ -s changelog_section.md ]; then\n echo \"Found changelog content:\"\n cat changelog_section.md\n else\n echo \"No unreleased changelog content found\"\n : > changelog_section.md\n fi\n else\n echo \"No CHANGELOG.md found\"\n : > changelog_section.md\n fi\n\n - name: Generate contributors list\n env:\n PREV_TAG: ${{ steps.prev_tag.outputs.tag }}\n run: |\n if [ -n \"${PREV_TAG}\" ]; then\n RANGE=\"${PREV_TAG}..HEAD\"\n else\n RANGE=\"HEAD\"\n fi\n\n # Get unique contributors from commits and co-authors\n CONTRIBUTORS=$(git log \"${RANGE}\" --format='%an' | sort -u | while read -r name; do\n echo \"* ${name}\"\n done)\n\n # Also extract Co-authored-by names\n COAUTHORS=$(git log \"${RANGE}\" --format='%b' | grep -i \"co-authored-by\" | sed 's/.*: //' | sed 's/ <.*//' | sort -u | while read -r name; do\n [ -n \"$name\" ] && echo \"* ${name}\"\n done)\n\n # Combine and dedupe\n ALL_CONTRIBUTORS=$(printf '%s\\n%s' \"${CONTRIBUTORS}\" \"${COAUTHORS}\" | grep -v \"^$\" | sort -u)\n\n # Write to file for multi-line output\n {\n echo \"## 👥 Contributors\"\n echo \"\"\n echo \"Thanks to all contributors who made this release possible:\"\n echo \"\"\n echo \"${ALL_CONTRIBUTORS}\"\n } > contributors.md\n\n echo \"Generated contributors list:\"\n cat contributors.md\n\n - name: Combine release notes\n run: |\n # Combine changelog + contributors into final release body\n {\n cat changelog_section.md\n echo \"\"\n cat contributors.md\n } > release_body.md\n\n echo \"Final release body:\"\n cat release_body.md\n\n - name: Create Release\n uses: ncipollo/release-action@v1\n with:\n generateReleaseNotes: true\n name: ${{ github.ref_name }}\n token: ${{ secrets.GITHUB_TOKEN }}\n bodyFile: release_body.md\n appendBody: true\n" }, { "path": ".gitignore", "content": "# Local .terraform directories\n**/.terraform/*\n\n# .tfstate files\n*.tfstate\n*.tfstate.*\n\n# Crash log files\ncrash.log\ncrash.*.log\n\n# Exclude all .tfvars files, which are likely to contain sensitive data, such as\n# password, private keys, and other secrets. These should not be part of version \n# control as they are data points which are potentially sensitive and subject \n# to change depending on the environment.\n*.tfvars\n*.tfvars.json\n\n# Ignore override files as they are usually used to override resources locally and so\n# are not checked in\noverride.tf\noverride.tf.json\n*_override.tf\n*_override.tf.json\n\n# Include override files you do wish to add to version control using negated pattern\n# !example_override.tf\n\n# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan\n# example: *tfplan*\n\n# Ignore CLI configuration files\n.terraformrc\nterraform.rc\n\n*_kubeconfig.yaml\n*_kubeconfig.yaml-e\nterraform.tfvars\nplans-custom.yaml\nkustomization.yaml\n*kustomization_backup.yaml\nkube.tf\n.terraform.lock.hcl\nissue_fix.patch\n\n# AI related files\nCLAUDE.md\nkube-hetzner-knowledge.jsondata\n\n# Misc\n.DS_Store\n\nrequirements/*\n\n# Ignore IntelliJ related files\n.idea\n\n# Local triage documentation\nissues/\nprs/\n\n# Local analysis artifacts\n.tldr/\n.tldrignore\nscripts/process_ideas_v3.py\ntasks/\n" }, { "path": ".pre-commit-config.yaml", "content": "default_install_hook_types:\n - pre-commit\n\nrepos:\n - repo: https://github.com/antonbabenko/pre-commit-terraform\n rev: v1.97.3\n hooks:\n - id: terraform_fmt\n - id: terraform_validate\n - id: terraform_tfsec\n pass_filenames: false\n - id: terraform_docs\n args:\n - '--args=--lockfile=false'\n# - id: terraform_tflint\n# args:\n# - '--args=--only=terraform_deprecated_interpolation'\n# - '--args=--only=terraform_deprecated_index'\n# - '--args=--only=terraform_unused_declarations'\n# - '--args=--only=terraform_comment_syntax'\n# - '--args=--only=terraform_documented_outputs'\n# - '--args=--only=terraform_documented_variables'\n# - '--args=--only=terraform_typed_variables'\n# - '--args=--only=terraform_module_pinned_source'\n# - '--args=--only=terraform_naming_convention'\n# - '--args=--only=terraform_required_version'\n# - '--args=--only=terraform_required_providers'\n# - '--args=--only=terraform_standard_module_structure'\n# - '--args=--only=terraform_workspace_remote'\n - repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v5.0.0\n hooks:\n - id: end-of-file-fixer\n" }, { "path": ".terraform-docs.yml", "content": "formatter: \"markdown table\"\n\nrecursive:\n enabled: false\n path: modules\n\noutput:\n file: docs/terraform.md\n mode: inject\n template: |-\n <!-- BEGIN_TF_DOCS -->\n {{ .Content }}\n <!-- END_TF_DOCS -->\n\noutput-values:\n enabled: false\n from: \"\"\n\nsort:\n enabled: true\n by: name\n\nsettings:\n anchor: true\n color: true\n default: true\n description: false\n escape: true\n hide-empty: false\n html: true\n indent: 3\n lockfile: true\n read-comments: true\n required: true\n sensitive: true\n type: true\n" }, { "path": "CHANGELOG.md", "content": "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),\nand this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n\n## [Unreleased]\n\n### 📋 v2.19.1 Patch Release\n\nThis is a patch release for v2.19.0. **If upgrading from v2.18.x**, please review the full release notes below including upgrade notes, new features, and breaking changes.\n\n**Patch fix:**\n- **Audit Policy Bastion Connection** - Fixed missing bastion SSH settings in `audit_policy` provisioner, enabling audit policy deployment for NAT router / private network setups (#2042) - thanks @CounterClops\n\n---\n\n### ⚠️ Upgrade Notes (from v2.18.x)\n\n#### NAT Router Users (created before v2.19.0)\n\nIf you created a NAT router **before v2.19.0** (when the hcloud provider used the now-deprecated `datacenter` attribute), you may see Terraform wanting to recreate your NAT router primary IPs. This would result in new IP addresses.\n\n**To check if you're affected**, run `terraform plan` and look for changes to:\n- `hcloud_primary_ip.nat_router_primary_ipv4`\n- `hcloud_primary_ip.nat_router_primary_ipv6`\n\n**If Terraform shows replacement**, you have two options:\n\n1. **Allow the recreation** (simplest, but IPs will change):\n ```bash\n terraform apply\n ```\n\n2. **Migrate state manually** (preserves IPs):\n ```bash\n # Remove old state entries\n terraform state rm 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv4[0]'\n terraform state rm 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv6[0]'\n\n # Import with current IPs (get IDs from Hetzner Cloud Console)\n terraform import 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv4[0]' <ipv4-id>\n terraform import 'module.kube-hetzner.hcloud_primary_ip.nat_router_primary_ipv6[0]' <ipv6-id>\n\n terraform apply\n ```\n\n#### Version Requirements\n\n- Minimum Terraform version: `1.10.1`\n- Minimum hcloud provider version: `1.59.0`\n\n### 🚀 New Features\n\n- **Hetzner Robot Integration** - Manage dedicated Robot servers via vSwitch and Cloud Controller Manager. New variables: `robot_ccm_enabled`, `robot_user`, `robot_password`, `vswitch_id`, `vswitch_subnet_index` (#1916)\n- **Audit Logging** - Kubernetes audit logs with configurable policy via `k3s_audit_policy_config` and log rotation settings (#1825)\n- **Control Plane Endpoint** - New `control_plane_endpoint` variable for stable external API server endpoint (e.g., external load balancers) (#1911)\n- **NAT Router Control Plane Access** - Automatic port 6443 forwarding on NAT router when `control_plane_lb_enable_public_interface` is false (#2015)\n- **Smaller Networks** - New `subnet_amount` variable enables networks smaller than /16 (#1971)\n- **Custom Subnet Ranges** - Added `subnet_ip_range` to agent_nodepools for manual CIDR assignment (#1903)\n- **Autoscaler Swap/ZRAM** - Added `swap_size` and `zram_size` support for autoscaler node pools (#2008)\n- **Autoscaler Resources** - New `cluster_autoscaler_replicas`, `cluster_autoscaler_resource_limits`, `cluster_autoscaler_resource_values` (#2025)\n- **Flannel Backend** - New `flannel_backend` variable to override flannel backend (wireguard-native, host-gw, etc.)\n- **Cilium XDP Acceleration** - New `cilium_loadbalancer_acceleration_mode` variable (native, best-effort, disabled)\n- **K3s v1.35 Support** - Added support for k3s v1.35 channel (#2029)\n- **Packer Enhancements** - Configurable `kernel_type`, `sysctl_config_file`, and `timezone` for MicroOS snapshots (#2009, #2010)\n\n### 🐛 Bug Fixes\n\n- **Audit Policy Bastion Connection** _(v2.19.1)_ - Fixed missing bastion SSH settings in `audit_policy` provisioner, enabling audit policy deployment for NAT router / private network setups (#2042)\n- **Longhorn Hotfix Tag Guidance** - Clarified `longhorn_version` as chart version and documented `longhorn_merge_values` for targeted Longhorn image hotfix tags (e.g. manager/instance-manager) (#2054)\n- **Traefik v34 Compatibility** - Fixed HTTP to HTTPS redirection config for Traefik Helm Chart v34+ (#2028)\n- **NAT Router IP Drift** - Fixed infinite replacement cycle by migrating from deprecated `datacenter` to `location` (#2021)\n- **SELinux YAML Parsing** - Fixed cloud-init SCHEMA_ERROR caused by improper YAML formatting of SELinux policy\n- **SELinux Missing Rules** - Added rules for JuiceFS (sock_file write) and SigNoz (blk_file getattr)\n- **Kured Version Null** - Fixed potential null value issues with `kured_version` logic (#2032)\n\n### 🔧 Changes\n\n- **Default K3s Version** - Bumped from v1.31 to v1.33 (#2030)\n- **Default System Upgrade Controller** - Bumped to v0.18.0\n- **SELinux Policy Extraction** - Moved to dedicated template file for maintainability\n- **terraform_data Migration** - Migrated from null_resource to terraform_data with automatic state migration (#1548)\n- **remote-exec Refactor** - Improved provisioner compatibility with Terraform Stacks (#1893)\n- **Custom GPT Updated** - [KH Assistant](https://chatgpt.com/g/g-67df95cd1e0c8191baedfa3179061581-kh-assistant) updated with v2.19.0 features, improved knowledge base, and cost calculator\n\n---\n\n## [2.19.0] - 2026-02-01\n\n_Initial release of the v2.19 series. See above for full feature list._\n\n---\n\n## [2.18.5] - 2026-01-15\n\n_See [GitHub releases](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/releases) for earlier versions._\n" }, { "path": "LICENSE", "content": "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n" }, { "path": "README.md", "content": "<div align=\"center\">\n\n<!-- HERO SECTION -->\n<img src=\"https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/kube-hetzner-logo.png\" alt=\"Kube-Hetzner Logo\" width=\"140\" height=\"140\">\n\n# Kube-Hetzner\n\n### Production-Ready Kubernetes on Hetzner Cloud\n\n**HA by default • Auto-upgrading • Cost-optimized**\n\nA highly optimized, easy-to-use, auto-upgradable Kubernetes cluster powered by k3s on MicroOS<br>deployed for peanuts on [Hetzner Cloud](https://hetzner.com)\n\n[![Terraform](https://img.shields.io/badge/Terraform-%3E%3D1.10-844FBA?style=flat-square&logo=terraform)](https://terraform.io)  \n[![OpenTofu](https://img.shields.io/badge/OpenTofu-Compatible-FFDA18?style=flat-square&logo=opentofu)](https://opentofu.org)  \n[![K3s](https://img.shields.io/badge/K3s-v1.35-FFC61C?style=flat-square&logo=k3s)](https://k3s.io)  \n[![License](https://img.shields.io/github/license/kube-hetzner/terraform-hcloud-kube-hetzner?style=flat-square&color=blue)](LICENSE)  \n[![GitHub Stars](https://img.shields.io/github/stars/kube-hetzner/terraform-hcloud-kube-hetzner?style=flat-square&logo=github)](https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/stargazers)\n\n---\n\n<table>\n<tr>\n<td width=\"50%\" valign=\"top\">\n\n**💖 Love this project?**<br>\n<a href=\"https://github.com/sponsors/mysticaltech\">Become a sponsor</a> to help fund<br>maintenance and new features!\n\n</td>\n<td width=\"50%\" valign=\"top\">\n\n**🤖 KH Assistant**<br>\n<a href=\"https://chatgpt.com/g/g-67df95cd1e0c8191baedfa3179061581-kh-assistant\">Custom GPT</a> or <code>/kh-assistant</code> <a href=\"https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/tree/master/.claude/skills/kh-assistant\">skill</a><br>\nAI-powered config generation & debugging!\n\n</td>\n</tr>\n</table>\n\n---\n\n[Getting Started](#-getting-started) •\n[Features](#-features) •\n[Usage](#-usage) •\n[Examples](#-examples) •\n[Contributing](#-contributing)\n\n</div>\n\n---\n\n## 📖 About The Project\n\n[Hetzner Cloud](https://hetzner.com) offers exceptional value with data centers across Europe and the US. This project creates a **highly optimized Kubernetes installation** that's easy to maintain, secure, and automatically upgrades both nodes and Kubernetes—functionality similar to GKE's Auto-Pilot.\n\n> *We are not Hetzner affiliates, but we strive to be the optimal solution for deploying Kubernetes on their platform.*\n\nBuilt on the shoulders of giants:\n- **[openSUSE MicroOS](https://en.opensuse.org/Portal:MicroOS)** — Immutable container OS with automatic updates\n- **[k3s](https://k3s.io/)** — Certified, lightweight Kubernetes distribution\n\n<div align=\"center\">\n<img src=\"https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/kubectl-pod-all-17022022.png\" alt=\"Kube-Hetzner Screenshot\" width=\"700\">\n</div>\n\n### Why MicroOS over Ubuntu?\n\n| Feature | Benefit |\n|---------|---------|\n| **Immutable filesystem** | Most of the OS is read-only—hardened by design |\n| **Auto-ban abusive IPs** | SSH brute-force protection out of the box |\n| **Rolling release** | Piggybacks on openSUSE Tumbleweed—always current |\n| **BTRFS snapshots** | Automatic rollback if updates break something |\n| **[Kured](https://github.com/kubereboot/kured) support** | Safe, HA-aware node reboots |\n\n### Why k3s?\n\n| Feature | Benefit |\n|---------|---------|\n| **Certified Kubernetes** | Automatically synced with upstream k8s |\n| **Single binary** | Deploy with one command |\n| **Batteries included** | Built-in [helm-controller](https://github.com/k3s-io/helm-controller) |\n| **Easy upgrades** | Via [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller) |\n\n---\n\n## ✨ Features\n\n<table>\n<tr>\n<td width=\"50%\" valign=\"top\">\n\n### 🚀 Core Platform\n- [x] **Maintenance-free** — Auto-upgrades MicroOS & k3s with rollback\n- [x] **Multi-architecture** — Mix x86 and ARM (CAX) for cost savings\n- [x] **Private networking** — Secure, low-latency node communication\n- [x] **SELinux hardened** — Pre-configured security policies\n\n### 🌐 Networking & CNI\n- [x] **CNI flexibility** — Flannel, Calico, or Cilium\n- [x] **Cilium XDP** — Hardware-level load balancing\n- [x] **Wireguard encryption** — Optional encrypted overlay\n- [x] **Dual-stack** — Full IPv4 & IPv6 support\n- [x] **Custom subnets** — Define CIDR blocks per nodepool\n\n### ⚖️ Load Balancing\n- [x] **Ingress controllers** — Traefik, Nginx, or HAProxy\n- [x] **Proxy Protocol** — Preserve client IPs\n- [x] **Flexible LB** — Hetzner LB or Klipper\n\n</td>\n<td width=\"50%\" valign=\"top\">\n\n### 🔄 High Availability\n- [x] **HA by default** — 3 control-planes + 2 agents across AZs\n- [x] **Super-HA** — Span multiple Hetzner locations\n- [x] **Cluster autoscaler** — Automatic node scaling\n- [x] **Single-node mode** — Perfect for development\n\n### 💾 Storage\n- [x] **Hetzner CSI** — Native block storage with encryption\n- [x] **Longhorn** — Distributed storage with replication\n- [x] **Custom mount paths** — Configurable storage locations\n\n### 🔒 Security & Operations\n- [x] **Audit logging** — Configurable retention policies\n- [x] **Firewall rules** — Granular SSH/API access control\n- [x] **NAT router** — Fully private clusters\n- [x] **190+ variables** — Fine-tune everything\n- [x] **Kustomization** — Extend with custom manifests\n\n</td>\n</tr>\n</table>\n\n---\n\n## 🏁 Getting Started\n\n### Prerequisites\n\n<table>\n<tr>\n<th>Platform</th>\n<th>Installation Command</th>\n</tr>\n<tr>\n<td><strong>Homebrew</strong> (macOS/Linux)</td>\n<td><code>brew install hashicorp/tap/terraform hashicorp/tap/packer kubectl hcloud</code></td>\n</tr>\n<tr>\n<td><strong>Arch Linux</strong></td>\n<td><code>yay -S terraform packer kubectl hcloud</code></td>\n</tr>\n<tr>\n<td><strong>Debian/Ubuntu</strong></td>\n<td><code>sudo apt install terraform packer kubectl</code></td>\n</tr>\n<tr>\n<td><strong>Fedora/RHEL</strong></td>\n<td><code>sudo dnf install terraform packer kubectl</code></td>\n</tr>\n<tr>\n<td><strong>Windows</strong></td>\n<td><code>choco install terraform packer kubernetes-cli hcloud</code></td>\n</tr>\n</table>\n\n> **Required tools:** [terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) or [tofu](https://opentofu.org/docs/intro/install/), [packer](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli#installing-packer) (initial setup only), [kubectl](https://kubernetes.io/docs/tasks/tools/), [hcloud](https://github.com/hetznercloud/cli)\n\n---\n\n### ⚡ Quick Start\n\n<table>\n<tr>\n<td>1️⃣</td>\n<td><strong>Create a Hetzner project</strong> at <a href=\"https://console.hetzner.cloud/\">console.hetzner.cloud</a> and grab an API token (Read & Write)</td>\n</tr>\n<tr>\n<td>2️⃣</td>\n<td><strong>Generate an SSH key pair</strong> (passphrase-less ed25519) — or see <a href=\"docs/ssh.md\">SSH options</a></td>\n</tr>\n<tr>\n<td>3️⃣</td>\n<td><strong>Run the setup script</strong> — creates your project folder and MicroOS snapshot:</td>\n</tr>\n</table>\n\n```sh\ntmp_script=$(mktemp) && curl -sSL -o \"${tmp_script}\" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x \"${tmp_script}\" && \"${tmp_script}\" && rm \"${tmp_script}\"\n```\n\n<details>\n<summary><strong>Fish shell version</strong></summary>\n\n```fish\nset tmp_script (mktemp); curl -sSL -o \"{tmp_script}\" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh; chmod +x \"{tmp_script}\"; bash \"{tmp_script}\"; rm \"{tmp_script}\"\n```\n</details>\n\n<details>\n<summary><strong>Save as alias for future use</strong></summary>\n\n```sh\nalias createkh='tmp_script=$(mktemp) && curl -sSL -o \"${tmp_script}\" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x \"${tmp_script}\" && \"${tmp_script}\" && rm \"${tmp_script}\"'\n```\n</details>\n\n<details>\n<summary><strong>What the script does</strong></summary>\n\n```sh\nmkdir /path/to/your/new/folder\ncd /path/to/your/new/folder\ncurl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/kube.tf.example -o kube.tf\ncurl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/packer-template/hcloud-microos-snapshots.pkr.hcl -o hcloud-microos-snapshots.pkr.hcl\nexport HCLOUD_TOKEN=\"your_hcloud_token\"\npacker init hcloud-microos-snapshots.pkr.hcl\npacker build hcloud-microos-snapshots.pkr.hcl\nhcloud context create <project-name>\n```\n</details>\n\n<table>\n<tr>\n<td>4️⃣</td>\n<td><strong>Customize your <code>kube.tf</code></strong> — full reference in <a href=\"docs/terraform.md\">terraform.md</a></td>\n</tr>\n</table>\n\n---\n\n### 🎯 Deploy\n\n```sh\ncd <your-project-folder>\nterraform init --upgrade\nterraform validate\nterraform apply -auto-approve\n```\n\n**~5 minutes later:** Your cluster is ready! 🎉\n\n> ⚠️ Once Terraform manages your cluster, avoid manual changes in the Hetzner UI. Use `hcloud` CLI to inspect resources.\n\n---\n\n## 🔧 Usage\n\nView cluster details:\n```sh\nterraform output kubeconfig\nterraform output -json kubeconfig | jq\n```\n\n### Connect via SSH\n\n```sh\nssh root@<control-plane-ip> -i /path/to/private_key -o StrictHostKeyChecking=no\n```\n\nRestrict SSH access by configuring `firewall_ssh_source` in your kube.tf. See [SSH docs](docs/ssh.md#firewall-ssh-source-and-changing-ips) for dynamic IP handling.\n\n### Connect via Kube API\n\n```sh\nkubectl --kubeconfig clustername_kubeconfig.yaml get nodes\n```\n\nOr set it as your default:\n```sh\nexport KUBECONFIG=/<path-to>/clustername_kubeconfig.yaml\n```\n\n> **Tip:** If `create_kubeconfig = false`, generate it manually: `terraform output --raw kubeconfig > clustername_kubeconfig.yaml`\n\n---\n\n## 🌐 CNI Options\n\nDefault is **Flannel**. Switch by setting `cni_plugin` to `\"calico\"` or `\"cilium\"`.\n\n### Cilium Configuration\n\nCustomize via `cilium_values` with [Cilium helm values](https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml).\n\n| Feature | Variable |\n|---------|----------|\n| Full kube-proxy replacement | `disable_kube_proxy = true` |\n| Hubble observability | `cilium_hubble_enabled = true` |\n\nAccess Hubble UI:\n```sh\nkubectl port-forward -n kube-system service/hubble-ui 12000:80\n# or with Cilium CLI:\ncilium hubble ui\n```\n\n---\n\n## 📈 Scaling\n\n### Manual Scaling\n\nAdjust `count` in any nodepool and run `terraform apply`. Constraints:\n\n- First control-plane nodepool minimum: **1**\n- Drain nodes before removing: `kubectl drain <node-name>`\n- Only remove nodepools from the **end** of the list\n- Rename nodepools only when count is **0**\n\n**Advanced:** Replace `count` with a `nodes` map for individual node control—see `kube.tf.example`.\n\n### Autoscaling\n\nEnable with `autoscaler_nodepools`. Powered by [Cluster Autoscaler](https://github.com/kubernetes/autoscaler).\n\n> ⚠️ Autoscaled nodes use a snapshot from the initial control plane. Ensure disk sizes match.\n\n---\n\n## 🛡️ High Availability\n\nDefault: **3 control-planes + 3 agents** with automatic upgrades.\n\n| Control Planes | Recommendation |\n|----------------|----------------|\n| 3+ (odd numbers) | Full HA with quorum maintenance |\n| 2 | Disable auto OS upgrades, manual maintenance |\n| 1 | Development only, disable auto upgrades |\n\nSee [Rancher's HA documentation](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/).\n\n---\n\n## 🔄 Automatic Upgrades\n\n### OS Upgrades (MicroOS)\n\nHandled by [Kured](https://github.com/kubereboot/kured)—safe, HA-aware reboots. Configure timeframes via [Kured options](https://kured.dev/docs/configuration/).\n\n### K3s Upgrades\n\nManaged by [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller). Customize the [upgrade plan template](templates/plans.yaml.tpl).\n\n### Disable Automatic Upgrades\n\n```tf\n# Disable OS upgrades (required for <3 control planes)\nautomatically_upgrade_os = false\n\n# Disable k3s upgrades\nautomatically_upgrade_k3s = false\n```\n\n<details>\n<summary><strong>Manual upgrade commands</strong></summary>\n\n**Selective k3s upgrade:**\n```sh\nkubectl label --overwrite node <node-name> k3s_upgrade=true\nkubectl label node <node-name> k3s_upgrade- # disable\n```\n\n**Or delete upgrade plans:**\n```sh\nkubectl delete plan k3s-agent -n system-upgrade\nkubectl delete plan k3s-server -n system-upgrade\n```\n\n**Manual OS upgrade:**\n```sh\nkubectl drain <node-name>\nssh root@<node-ip>\nsystemctl start transactional-update.service\nreboot\n```\n</details>\n\n### Component Upgrades\n\nUse the `kustomization_backup.yaml` file created during installation:\n\n1. Copy to `kustomization.yaml`\n2. Update source URLs to latest versions\n3. Apply: `kubectl apply -k ./`\n\n---\n\n## ⚙️ Customization\n\nMost components use [Helm Chart](https://rancher.com/docs/k3s/latest/en/helm/) definitions via k3s Helm Controller.\n\nSee `kube.tf.example` for examples.\n\n---\n\n## 🖥️ Dedicated Servers\n\nIntegrate Hetzner Robot servers via [the dedicated server guide](docs/add-robot-server.md).\n\n---\n\n## ➕ Adding Extras\n\nUse [Kustomize](https://kustomize.io) for additional deployments:\n\n1. Create `extra-manifests/kustomization.yaml.tpl`\n2. Supports Terraform templating via `extra_kustomize_parameters`\n3. Applied after cluster setup with `kubectl apply -k`\n\nChange folder name with `extra_kustomize_folder`. See [example](examples/kustomization_user_deploy).\n\n---\n\n## 📚 Examples\n\n<details>\n<summary><strong>Custom post-install actions (ArgoCD, etc.)</strong></summary>\n\nFor CRD-dependent applications:\n\n```tf\nextra_kustomize_deployment_commands = <<-EOT\n kubectl -n argocd wait --for condition=established --timeout=120s crd/appprojects.argoproj.io\n kubectl -n argocd wait --for condition=established --timeout=120s crd/applications.argoproj.io\n kubectl apply -f /var/user_kustomize/argocd-projects.yaml\n kubectl apply -f /var/user_kustomize/argocd-application-argocd.yaml\nEOT\n```\n</details>\n\n<details>\n<summary><strong>Useful Cilium commands</strong></summary>\n\n```sh\n# Status\nkubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium status --verbose\n\n# Monitor traffic\nkubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium monitor\n\n# List services\nkubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium service list\n```\n\n[Full Cilium cheatsheet](https://docs.cilium.io/en/latest/cheatsheet)\n</details>\n\n<details>\n<summary><strong>Cilium Egress Gateway with Floating IPs</strong></summary>\n\nControl outgoing traffic with static IPs:\n\n```tf\n{\n name = \"egress\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [\"node.kubernetes.io/role=egress\"],\n taints = [\"node.kubernetes.io/role=egress:NoSchedule\"],\n floating_ip = true,\n count = 1\n}\n```\n\nConfigure Cilium:\n```tf\nlocals {\n cluster_ipv4_cidr = \"10.42.0.0/16\"\n}\n\ncluster_ipv4_cidr = local.cluster_ipv4_cidr\n\ncilium_values = <<-EOT\nipam:\n mode: kubernetes\nk8s:\n requireIPv4PodCIDR: true\nkubeProxyReplacement: true\nroutingMode: native\nipv4NativeRoutingCIDR: \"10.0.0.0/8\"\nendpointRoutes:\n enabled: true\nloadBalancer:\n acceleration: native\nbpf:\n masquerade: true\negressGateway:\n enabled: true\nMTU: 1450\nEOT\n```\n\nExample policy:\n```yaml\napiVersion: cilium.io/v2\nkind: CiliumEgressGatewayPolicy\nmetadata:\n name: egress-sample\nspec:\n selectors:\n - podSelector:\n matchLabels:\n org: empire\n class: mediabot\n io.kubernetes.pod.namespace: default\n destinationCIDRs:\n - \"0.0.0.0/0\"\n excludedCIDRs:\n - \"10.0.0.0/8\"\n egressGateway:\n nodeSelector:\n matchLabels:\n node.kubernetes.io/role: egress\n egressIP: { FLOATING_IP }\n```\n\n[Full Egress Gateway docs](https://docs.cilium.io/en/stable/network/egress-gateway/)\n</details>\n\n<details>\n<summary><strong>TLS with Cert-Manager (recommended)</strong></summary>\n\nCert-Manager handles HA certificate management (Traefik CE is stateless).\n\n1. [Configure your issuer](https://cert-manager.io/docs/configuration/acme/)\n2. Add annotations to Ingress:\n\n```yaml\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: my-ingress\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt\nspec:\n tls:\n - hosts:\n - \"*.example.com\"\n secretName: example-com-letsencrypt-tls\n rules:\n - host: \"*.example.com\"\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: my-service\n port:\n number: 80\n```\n\n[Full Traefik + Cert-Manager guide](https://traefik.io/blog/secure-web-applications-with-traefik-proxy-cert-manager-and-lets-encrypt/)\n\n> **Ingress-Nginx with HTTP challenge:** Add `lb_hostname = \"cluster.example.org\"` to work around [this known issue](https://github.com/cert-manager/cert-manager/issues/466).\n</details>\n\n<details>\n<summary><strong>Managing snapshots</strong></summary>\n\n**Create:**\n```sh\nexport HCLOUD_TOKEN=<your-token>\npacker build ./packer-template/hcloud-microos-snapshots.pkr.hcl\n```\n\n**Delete:**\n```sh\nhcloud image list\nhcloud image delete <image-id>\n```\n</details>\n\n<details>\n<summary><strong>Single-node development cluster</strong></summary>\n\nSet `automatically_upgrade_os = false` (attached volumes don't handle auto-reboots well).\n\nUses k3s [service load balancer](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer) instead of external LB. Ports 80 & 443 open automatically.\n</details>\n\n<details>\n<summary><strong>Terraform Cloud deployment</strong></summary>\n\n1. Create MicroOS snapshot in your project first\n2. Configure SSH keys as Terraform Cloud variables (mark private key as sensitive):\n\n```tf\nssh_public_key = var.ssh_public_key\nssh_private_key = var.ssh_private_key\n```\n\n> **Password-protected keys:** Requires `local` execution mode with your own agent.\n</details>\n\n<details>\n<summary><strong>HelmChartConfig customization</strong></summary>\n\n```yaml\napiVersion: helm.cattle.io/v1\nkind: HelmChartConfig\nmetadata:\n name: rancher\n namespace: kube-system\nspec:\n valuesContent: |-\n # Your values.yaml customizations here\n```\n\nWorks for all add-ons: Longhorn, Cert-manager, Traefik, etc.\n</details>\n\n<details>\n<summary><strong>Encryption at rest (HCloud CSI)</strong></summary>\n\nCreate secret:\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n name: encryption-secret\n namespace: kube-system\nstringData:\n encryption-passphrase: foobar\n```\n\nCreate storage class:\n```yaml\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n name: hcloud-volumes-encrypted\nprovisioner: csi.hetzner.cloud\nreclaimPolicy: Delete\nvolumeBindingMode: WaitForFirstConsumer\nallowVolumeExpansion: true\nparameters:\n csi.storage.k8s.io/node-publish-secret-name: encryption-secret\n csi.storage.k8s.io/node-publish-secret-namespace: kube-system\n```\n</details>\n\n<details>\n<summary><strong>Encryption at rest (Longhorn)</strong></summary>\n\nCreate secret:\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n name: longhorn-crypto\n namespace: longhorn-system\nstringData:\n CRYPTO_KEY_VALUE: \"your-encryption-key\"\n CRYPTO_KEY_PROVIDER: \"secret\"\n CRYPTO_KEY_CIPHER: \"aes-xts-plain64\"\n CRYPTO_KEY_HASH: \"sha256\"\n CRYPTO_KEY_SIZE: \"256\"\n CRYPTO_PBKDF: \"argon2i\"\n```\n\nCreate storage class:\n```yaml\nkind: StorageClass\napiVersion: storage.k8s.io/v1\nmetadata:\n name: longhorn-crypto-global\nprovisioner: driver.longhorn.io\nallowVolumeExpansion: true\nparameters:\n nodeSelector: \"node-storage\"\n numberOfReplicas: \"1\"\n staleReplicaTimeout: \"2880\"\n fromBackup: \"\"\n fsType: ext4\n encrypted: \"true\"\n csi.storage.k8s.io/provisioner-secret-name: \"longhorn-crypto\"\n csi.storage.k8s.io/provisioner-secret-namespace: \"longhorn-system\"\n csi.storage.k8s.io/node-publish-secret-name: \"longhorn-crypto\"\n csi.storage.k8s.io/node-publish-secret-namespace: \"longhorn-system\"\n csi.storage.k8s.io/node-stage-secret-name: \"longhorn-crypto\"\n csi.storage.k8s.io/node-stage-secret-namespace: \"longhorn-system\"\n```\n\n[Longhorn encryption docs](https://longhorn.io/docs/1.4.0/advanced-resources/security/volume-encryption/)\n</details>\n\n<details>\n<summary><strong>Namespace-based architecture assignment</strong></summary>\n\nEnable admission controllers:\n```tf\nk3s_exec_server_args = \"--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector\"\n```\n\nAssign namespace to architecture:\n```yaml\napiVersion: v1\nkind: Namespace\nmetadata:\n annotations:\n scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/arch=amd64\n name: this-runs-on-amd64\n```\n\nWith tolerations:\n```yaml\napiVersion: v1\nkind: Namespace\nmetadata:\n annotations:\n scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/arch=arm64\n scheduler.alpha.kubernetes.io/defaultTolerations: '[{ \"operator\" : \"Equal\", \"effect\" : \"NoSchedule\", \"key\" : \"workload-type\", \"value\" : \"machine-learning\" }]'\n name: this-runs-on-arm64\n```\n</details>\n\n<details>\n<summary><strong>Backup and restore cluster (etcd S3)</strong></summary>\n\n**Setup backup:**\n\n1. Configure `etcd_s3_backup` in kube.tf\n2. Add k3s_token output:\n\n```tf\noutput \"k3s_token\" {\n value = module.kube-hetzner.k3s_token\n sensitive = true\n}\n```\n\n**Restore:**\n\n1. Add restoration config to kube.tf:\n\n```tf\nlocals {\n k3s_token = var.k3s_token\n etcd_version = \"v3.5.9\"\n etcd_snapshot_name = \"name-of-the-snapshot\"\n etcd_s3_endpoint = \"your-s3-endpoint\"\n etcd_s3_bucket = \"your-s3-bucket\"\n etcd_s3_access_key = \"your-s3-access-key\"\n etcd_s3_secret_key = var.etcd_s3_secret_key\n}\n\nvariable \"k3s_token\" {\n sensitive = true\n type = string\n}\n\nvariable \"etcd_s3_secret_key\" {\n sensitive = true\n type = string\n}\n\nmodule \"kube-hetzner\" {\n k3s_token = local.k3s_token\n\n postinstall_exec = compact([\n (\n local.etcd_snapshot_name == \"\" ? \"\" :\n <<-EOF\n export CLUSTERINIT=$(cat /etc/rancher/k3s/config.yaml | grep -i '\"cluster-init\": true')\n if [ -n \"$CLUSTERINIT\" ]; then\n k3s server \\\n --cluster-reset \\\n --etcd-s3 \\\n --cluster-reset-restore-path=${local.etcd_snapshot_name} \\\n --etcd-s3-endpoint=${local.etcd_s3_endpoint} \\\n --etcd-s3-bucket=${local.etcd_s3_bucket} \\\n --etcd-s3-access-key=${local.etcd_s3_access_key} \\\n --etcd-s3-secret-key=${local.etcd_s3_secret_key}\n mv /etc/rancher/k3s/k3s.yaml /etc/rancher/k3s/k3s.backup.yaml\n\n ETCD_VER=${local.etcd_version}\n case \"$(uname -m)\" in\n aarch64) ETCD_ARCH=\"arm64\" ;;\n x86_64) ETCD_ARCH=\"amd64\" ;;\n esac;\n DOWNLOAD_URL=https://github.com/etcd-io/etcd/releases/download\n curl -L $DOWNLOAD_URL/$ETCD_VER/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz -o /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz\n tar xzvf /tmp/etcd-$ETCD_VER-linux-$ETCD_ARCH.tar.gz -C /usr/local/bin --strip-components=1\n\n nohup etcd --data-dir /var/lib/rancher/k3s/server/db/etcd &\n echo $! > save_pid.txt\n\n etcdctl del /registry/services/specs/traefik/traefik\n etcdctl del /registry/services/endpoints/traefik/traefik\n\n OLD_NODES=$(etcdctl get \"\" --prefix --keys-only | grep /registry/minions/ | cut -c 19-)\n for NODE in $OLD_NODES; do\n for KEY in $(etcdctl get \"\" --prefix --keys-only | grep $NODE); do\n etcdctl del $KEY\n done\n done\n\n kill -9 `cat save_pid.txt`\n rm save_pid.txt\n fi\n EOF\n )\n ])\n}\n```\n\n2. Set environment variables:\n```sh\nexport TF_VAR_k3s_token=\"...\"\nexport TF_VAR_etcd_s3_secret_key=\"...\"\n```\n\n3. Run `terraform apply`\n</details>\n\n<details>\n<summary><strong>Pre-constructed private network (proxies)</strong></summary>\n\n```tf\nresource \"hcloud_network\" \"k3s_proxied\" {\n name = \"k3s-proxied\"\n ip_range = \"10.0.0.0/8\"\n}\n\nresource \"hcloud_network_subnet\" \"k3s_proxy\" {\n network_id = hcloud_network.k3s_proxied.id\n type = \"cloud\"\n network_zone = \"eu-central\"\n ip_range = \"10.128.0.0/9\"\n}\n\nresource \"hcloud_server\" \"your_proxy_server\" { ... }\n\nresource \"hcloud_server_network\" \"your_proxy_server\" {\n depends_on = [hcloud_server.your_proxy_server]\n server_id = hcloud_server.your_proxy_server.id\n network_id = hcloud_network.k3s_proxied.id\n ip = \"10.128.0.1\"\n}\n\nmodule \"kube-hetzner\" {\n existing_network_id = [hcloud_network.k3s_proxied.id] # Note: brackets required!\n network_ipv4_cidr = \"10.0.0.0/9\"\n additional_k3s_environment = {\n \"http_proxy\" : \"http://10.128.0.1:3128\",\n \"HTTP_PROXY\" : \"http://10.128.0.1:3128\",\n \"HTTPS_PROXY\" : \"http://10.128.0.1:3128\",\n \"CONTAINERD_HTTP_PROXY\" : \"http://10.128.0.1:3128\",\n \"CONTAINERD_HTTPS_PROXY\" : \"http://10.128.0.1:3128\",\n \"NO_PROXY\" : \"127.0.0.0/8,10.0.0.0/8,\",\n }\n}\n```\n</details>\n\n<details>\n<summary><strong>Placement groups</strong></summary>\n\nAssign nodepools to placement groups:\n\n```tf\nagent_nodepools = [\n {\n ...\n placement_group = \"special\"\n },\n]\n```\n\nLegacy compatibility:\n```tf\nplacement_group_compat_idx = 1\n```\n\nFor >10 nodes, use map-based definition:\n```tf\nagent_nodepools = [\n {\n nodes = {\n \"0\" : { placement_group = \"pg-1\" },\n \"30\" : { placement_group = \"pg-2\" },\n }\n },\n]\n```\n\nDisable globally: `placement_group_disable = true`\n</details>\n\n<details>\n<summary><strong>Migrating from count to map-based nodes</strong></summary>\n\nSet `append_index_to_node_name = false` to avoid node replacement:\n\n```tf\nagent_nodepools = [\n {\n name = \"agent-large\",\n server_type = \"cx33\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n nodes = {\n \"0\" : {\n append_index_to_node_name = false,\n labels = [\"my.extra.label=special\"],\n placement_group = \"agent-large-pg-1\",\n },\n \"1\" : {\n append_index_to_node_name = false,\n server_type = \"cx43\",\n labels = [\"my.extra.label=slightlybiggernode\"],\n placement_group = \"agent-large-pg-2\",\n },\n }\n },\n]\n```\n</details>\n\n<details>\n<summary><strong>Delete protection</strong></summary>\n\nProtect resources from accidental deletion via Hetzner Console/API:\n\n```tf\nenable_delete_protection = {\n floating_ip = true\n load_balancer = true\n volume = true\n}\n```\n\n> Note: Terraform can still delete resources (provider lifts the lock).\n</details>\n\n<details>\n<summary><strong>Private-only cluster (Wireguard)</strong></summary>\n\nRequirements:\n1. Pre-configured network\n2. NAT gateway with public IP ([Hetzner guide](https://community.hetzner.com/tutorials/how-to-set-up-nat-for-cloud-networks))\n3. Wireguard VPN access ([Hetzner guide](https://docs.hetzner.com/cloud/apps/list/wireguard/))\n4. Route `0.0.0.0/0` through NAT gateway\n\nConfiguration:\n```tf\nexisting_network_id = [YOURID]\nnetwork_ipv4_cidr = \"10.0.0.0/9\"\n\n# In all nodepools:\ndisable_ipv4 = true\ndisable_ipv6 = true\n\n# For autoscaler:\nautoscaler_disable_ipv4 = true\nautoscaler_disable_ipv6 = true\n\n# Optional private LB:\ncontrol_plane_lb_enable_public_interface = false\n```\n</details>\n\n<details>\n<summary><strong>Private-only cluster (NAT Router)</strong></summary>\n\nFully private setup with:\n- **Egress:** Single NAT router IP\n- **SSH:** Through bastion (NAT router)\n- **Control plane:** Through LB or NAT router port forwarding\n- **Ingress:** Through agents LB only\n\n> **August 11, 2025:** Hetzner removed legacy Router DHCP option. This module now automatically persists routes via the virtual gateway.\n</details>\n\n<details>\n<summary><strong>Fix SELinux issues with udica</strong></summary>\n\nCreate targeted SELinux profiles instead of weakening cluster-wide security:\n\n```sh\n# Find container\ncrictl ps\n\n# Generate inspection\ncrictl inspect <container-id> > container.json\n\n# Create profile\nudica -j container.json myapp --full-network-access\n\n# Install module\nsemodule -i myapp.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}\n```\n\nApply to deployment:\n```yaml\napiVersion: apps/v1\nkind: Deployment\nspec:\n template:\n spec:\n containers:\n - name: my-container\n securityContext:\n seLinuxOptions:\n type: myapp.process\n```\n\n*Thanks @carolosf*\n</details>\n\n---\n\n## 🔍 Debugging\n\n### Quick Status Check\n\n```sh\nhcloud context create Kube-hetzner # First time only\nhcloud server list # Check nodes\nhcloud network describe k3s # Check network\nhcloud loadbalancer describe k3s-traefik # Check LB\n```\n\n### SSH Troubleshooting\n\n```sh\nssh root@<control-plane-ip> -i /path/to/private_key -o StrictHostKeyChecking=no\n\n# View k3s logs\njournalctl -u k3s # Control plane\njournalctl -u k3s-agent # Agent nodes\n\n# Check config\ncat /etc/rancher/k3s/config.yaml\n\n# Check uptime\nlast reboot\nuptime\n```\n\n---\n\n## 💣 Takedown\n\n```sh\nterraform destroy -auto-approve\n```\n\n**If destroy hangs** (LB or autoscaled nodes), use the cleanup script:\n\n```sh\ntmp_script=$(mktemp) && curl -sSL -o \"${tmp_script}\" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/cleanup.sh && chmod +x \"${tmp_script}\" && \"${tmp_script}\" && rm \"${tmp_script}\"\n```\n\n> ⚠️ This deletes everything including volumes. Dry-run option available.\n\n---\n\n## ⬆️ Upgrading the Module\n\nUpdate `version` in your kube.tf and run `terraform apply`.\n\n### Migrating from 1.x to 2.x\n\n1. Run `createkh` to get new packer template\n2. Update version to `>= 2.0`\n3. Remove `extra_packages_to_install` and `opensuse_microos_mirror_link` (moved to packer)\n4. Run `terraform init -upgrade && terraform apply`\n\n---\n\n## 🤝 Contributing\n\n**Help wanted!** Consider asking Hetzner to add MicroOS as a default image (not just ISO) at [get.opensuse.org/microos](https://get.opensuse.org/microos). More requests = faster deployments for everyone!\n\n### Development Workflow\n\n1. Fork the project\n2. Create your branch: `git checkout -b AmazingFeature`\n3. Point your kube.tf `source` to local clone\n4. Useful commands:\n ```sh\n ../kube-hetzner/scripts/cleanup.sh\n packer build ../kube-hetzner/packer-template/hcloud-microos-snapshots.pkr.hcl\n ```\n5. Update `kube.tf.example` if needed\n6. Commit: `git commit -m 'Add AmazingFeature'`\n7. Push: `git push origin AmazingFeature`\n8. Open PR targeting `staging` branch\n\n### Agent Skills\n\nThis project includes [agent skills](https://agentskills.io) in `.claude/skills/` — reusable workflows for any AI coding agent (Claude Code, Cursor, Windsurf, Codex, etc.):\n\n| Skill | Purpose |\n|-------|---------|\n| `/kh-assistant` | Interactive help for configuration and debugging |\n| `/fix-issue <num>` | Guided workflow for fixing GitHub issues |\n| `/review-pr <num>` | Security-focused PR review |\n| `/test-changes` | Run terraform fmt, validate, plan |\n\n**PRs to improve these skills are welcome!** See `.claude/skills/` for the skill definitions.\n\n---\n\n## 💖 Support This Project\n\n<div align=\"center\">\n\nIf Kube-Hetzner saves you time and money, please consider supporting its development:\n\n<a href=\"https://github.com/sponsors/mysticaltech\">\n<img src=\"https://img.shields.io/badge/Sponsor_on_GitHub-❤️-EA4AAA?style=for-the-badge&logo=github-sponsors\" alt=\"Sponsor on GitHub\">\n</a>\n\n<br><br>\n\nYour sponsorship directly funds:\n\n🐛 **Bug fixes** and issue response<br>\n🚀 **New features** and improvements<br>\n📚 **Documentation** maintenance<br>\n🔒 **Security updates** and best practices\n\n**Every contribution matters.** Thank you for keeping this project alive! 🙏\n\n</div>\n\n---\n\n## 🙏 Acknowledgements\n\n<div align=\"center\">\n<a href=\"https://www.hetzner.com\"><img src=\"https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/raw/master/.images/hetzner-logo.svg\" alt=\"Hetzner — Server · Cloud · Hosting\" height=\"80\"></a>\n<br><br>\n</div>\n\nThanks to **[Hetzner](https://www.hetzner.com)** for supporting this project with cloud credits.\n\n---\n\n<div align=\"center\">\n\n**[⬆ Back to Top](#kube-hetzner)**\n\nMade with ❤️ by the Kube-Hetzner community\n\n</div>\n" }, { "path": "SECURITY.md", "content": "# Security Policy\n\n## Reporting a Vulnerability\n\nPlease report any vulnerability findings privately via email to the top three contributors to the projects. You can find our emails by grepping the git logs.\n\nIn case you can't find the emails:\n\n- [aleksasiriski](https://github.com/aleksasiriski): [sir@tmina.org](mailto:kube-hetzner@sir.tmina.org)\n" }, { "path": "agents.tf", "content": "module \"agents\" {\n source = \"./modules/host\"\n\n providers = {\n hcloud = hcloud,\n }\n\n for_each = local.agent_nodes\n\n name = \"${var.use_cluster_name_in_node_name ? \"${var.cluster_name}-\" : \"\"}${each.value.nodepool_name}${try(each.value.node_name_suffix, \"\")}\"\n microos_snapshot_id = substr(each.value.server_type, 0, 3) == \"cax\" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id\n base_domain = var.base_domain\n ssh_keys = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]\n ssh_port = var.ssh_port\n ssh_public_key = var.ssh_public_key\n ssh_private_key = var.ssh_private_key\n ssh_additional_public_keys = length(var.ssh_hcloud_key_label) > 0 ? concat(var.ssh_additional_public_keys, data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.public_key) : var.ssh_additional_public_keys\n firewall_ids = each.value.disable_ipv4 && each.value.disable_ipv6 ? [] : [hcloud_firewall.k3s.id] # Cannot attach a firewall when public interfaces are disabled\n placement_group_id = var.placement_group_disable ? null : (each.value.placement_group == null ? hcloud_placement_group.agent[each.value.placement_group_compat_idx].id : hcloud_placement_group.agent_named[each.value.placement_group].id)\n location = each.value.location\n server_type = each.value.server_type\n backups = each.value.backups\n ipv4_subnet_id = hcloud_network_subnet.agent[[for i, v in var.agent_nodepools : i if v.name == each.value.nodepool_name][0]].id\n dns_servers = var.dns_servers\n k3s_registries = var.k3s_registries\n k3s_registries_update_script = local.k3s_registries_update_script\n cloudinit_write_files_common = local.cloudinit_write_files_common\n k3s_kubelet_config = var.k3s_kubelet_config\n k3s_kubelet_config_update_script = local.k3s_kubelet_config_update_script\n k3s_audit_policy_config = \"\"\n k3s_audit_policy_update_script = \"\"\n cloudinit_runcmd_common = local.cloudinit_runcmd_common\n swap_size = each.value.swap_size\n zram_size = each.value.zram_size\n keep_disk_size = var.keep_disk_agents\n disable_ipv4 = each.value.disable_ipv4\n disable_ipv6 = each.value.disable_ipv6\n ssh_bastion = local.ssh_bastion\n network_id = data.hcloud_network.k3s.id\n private_ipv4 = cidrhost(hcloud_network_subnet.agent[[for i, v in var.agent_nodepools : i if v.name == each.value.nodepool_name][0]].ip_range, each.value.index + (local.network_size >= 16 ? 101 : floor(pow(local.subnet_size, 2) * 0.4)))\n\n labels = merge(local.labels, local.labels_agent_node)\n\n automatically_upgrade_os = var.automatically_upgrade_os\n\n network_gw_ipv4 = local.network_gw_ipv4\n\n depends_on = [\n hcloud_network_subnet.agent,\n hcloud_placement_group.agent,\n hcloud_server.nat_router,\n terraform_data.nat_router_await_cloud_init,\n ]\n}\n\nlocals {\n k3s-agent-config = { for k, v in local.agent_nodes : k => merge(\n {\n node-name = module.agents[k].name\n server = local.k3s_endpoint\n token = local.k3s_token\n # Kubelet arg precedence (last wins): local.kubelet_arg > v.kubelet_args > k3s_global_kubelet_args > k3s_agent_kubelet_args\n kubelet-arg = concat(\n local.kubelet_arg,\n v.kubelet_args,\n var.k3s_global_kubelet_args,\n var.k3s_agent_kubelet_args\n )\n flannel-iface = local.flannel_iface\n node-ip = module.agents[k].private_ipv4_address\n node-label = v.labels\n node-taint = v.taints\n },\n var.agent_nodes_custom_config,\n local.prefer_bundled_bin_config,\n # Force selinux=false if disable_selinux = true.\n var.disable_selinux\n ? { selinux = false }\n : (v.selinux == true ? { selinux = true } : {})\n ) }\n\n agent_ips = {\n for k, v in module.agents : k => coalesce(\n v.ipv4_address,\n v.ipv6_address,\n v.private_ipv4_address\n )\n }\n}\n\nresource \"terraform_data\" \"agent_config\" {\n for_each = local.agent_nodes\n\n triggers_replace = {\n agent_id = module.agents[each.key].id\n config = sha1(yamlencode(local.k3s-agent-config[each.key]))\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.agent_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Generating k3s agent config file\n provisioner \"file\" {\n content = yamlencode(local.k3s-agent-config[each.key])\n destination = \"/tmp/config.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [local.k3s_config_update_script]\n }\n}\nmoved {\n from = null_resource.agent_config\n to = terraform_data.agent_config\n}\n\nresource \"terraform_data\" \"agents\" {\n for_each = local.agent_nodes\n\n triggers_replace = {\n agent_id = module.agents[each.key].id\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.agent_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Install k3s agent\n provisioner \"remote-exec\" {\n inline = local.install_k3s_agent\n }\n\n # Start the k3s agent and wait for it to have started\n provisioner \"remote-exec\" {\n inline = concat(var.enable_longhorn || var.enable_iscsid ? [\"systemctl enable --now iscsid\"] : [], [\n \"timeout 120 systemctl start k3s-agent 2> /dev/null\",\n <<-EOT\n timeout 120 bash <<EOF\n until systemctl status k3s-agent > /dev/null; do\n systemctl start k3s-agent 2> /dev/null\n echo \"Waiting for the k3s agent to start...\"\n sleep 2\n done\n EOF\n EOT\n ])\n }\n\n depends_on = [\n terraform_data.first_control_plane,\n terraform_data.agent_config,\n hcloud_network_subnet.agent\n ]\n}\nmoved {\n from = null_resource.agents\n to = terraform_data.agents\n}\n\nresource \"hcloud_volume\" \"longhorn_volume\" {\n for_each = { for k, v in local.agent_nodes : k => v if((v.longhorn_volume_size >= 10) && (v.longhorn_volume_size <= 10240) && var.enable_longhorn) }\n\n labels = {\n provisioner = \"terraform\"\n cluster = var.cluster_name\n scope = \"longhorn\"\n }\n name = \"${var.cluster_name}-longhorn-${module.agents[each.key].name}\"\n size = local.agent_nodes[each.key].longhorn_volume_size\n server_id = module.agents[each.key].id\n automount = true\n format = var.longhorn_fstype\n delete_protection = var.enable_delete_protection.volume\n}\n\nresource \"terraform_data\" \"configure_longhorn_volume\" {\n for_each = { for k, v in local.agent_nodes : k => v if((v.longhorn_volume_size >= 10) && (v.longhorn_volume_size <= 10240) && var.enable_longhorn) }\n\n triggers_replace = {\n agent_id = module.agents[each.key].id\n }\n\n # Start the k3s agent and wait for it to have started\n provisioner \"remote-exec\" {\n inline = [\n \"set -e\",\n \"mkdir -p '${each.value.longhorn_mount_path}' >/dev/null\",\n \"mountpoint -q '${each.value.longhorn_mount_path}' || mount -o discard,defaults ${hcloud_volume.longhorn_volume[each.key].linux_device} '${each.value.longhorn_mount_path}'\",\n \"${var.longhorn_fstype == \"ext4\" ? \"resize2fs\" : \"xfs_growfs\"} ${hcloud_volume.longhorn_volume[each.key].linux_device}\",\n # Match any non-comment line (^[^#]) with any first field, followed by a space and your mount path in the second column.\n # This prevents false positives like /data matching /data1.\n \"awk -v path='${each.value.longhorn_mount_path}' '$0 !~ /^#/ && $2 == path { found=1; exit } END { exit !found }' /etc/fstab || echo '${hcloud_volume.longhorn_volume[each.key].linux_device} ${each.value.longhorn_mount_path} ${var.longhorn_fstype} discard,nofail,defaults 0 0' | tee -a /etc/fstab >/dev/null\"\n ]\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.agent_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n depends_on = [\n hcloud_volume.longhorn_volume\n ]\n}\nmoved {\n from = null_resource.configure_longhorn_volume\n to = terraform_data.configure_longhorn_volume\n}\n\nresource \"hcloud_floating_ip\" \"agents\" {\n for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, \"floating_ip\"), false) }\n\n type = \"ipv4\"\n labels = local.labels\n home_location = each.value.location\n delete_protection = var.enable_delete_protection.floating_ip\n}\n\nresource \"hcloud_floating_ip_assignment\" \"agents\" {\n for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, \"floating_ip\"), false) }\n\n floating_ip_id = hcloud_floating_ip.agents[each.key].id\n server_id = module.agents[each.key].id\n\n depends_on = [\n terraform_data.agents\n ]\n}\n\nresource \"hcloud_rdns\" \"agents\" {\n for_each = { for k, v in local.agent_nodes : k => v if lookup(v, \"floating_ip_rdns\", null) != null }\n\n floating_ip_id = hcloud_floating_ip.agents[each.key].id\n ip_address = hcloud_floating_ip.agents[each.key].ip_address\n dns_ptr = local.agent_nodes[each.key].floating_ip_rdns\n\n depends_on = [\n hcloud_floating_ip.agents\n ]\n}\n\nresource \"terraform_data\" \"configure_floating_ip\" {\n for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, \"floating_ip\"), false) }\n\n triggers_replace = {\n agent_id = module.agents[each.key].id\n floating_ip_id = hcloud_floating_ip.agents[each.key].id\n }\n\n provisioner \"remote-exec\" {\n inline = [\n # Reconfigure eth0:\n # - add floating_ip as first and other IP as second address\n # - add 172.31.1.1 as default gateway (In the Hetzner Cloud, the\n # special private IP address 172.31.1.1 is the default\n # gateway for the public network)\n # The configuration is stored in file /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection\n <<-EOT\n ETH=eth1\n if ip link show eth0 &>/dev/null; then\n ETH=eth0\n fi\n\n NM_CONNECTION=$(nmcli -g GENERAL.CONNECTION device show \"$ETH\" 2>/dev/null)\n if [ -z \"$NM_CONNECTION\" ]; then\n echo \"ERROR: No NetworkManager connection found for $ETH\" >&2\n exit 1\n fi\n\n nmcli connection modify \"$NM_CONNECTION\" \\\n ipv4.method manual \\\n ipv4.addresses ${hcloud_floating_ip.agents[each.key].ip_address}/32,${local.agent_ips[each.key]}/32 gw4 172.31.1.1 \\\n ipv4.route-metric 100 \\\n && nmcli connection up \"$NM_CONNECTION\"\n EOT\n ]\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.agent_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n depends_on = [\n hcloud_floating_ip_assignment.agents\n ]\n}\nmoved {\n from = null_resource.configure_floating_ip\n to = terraform_data.configure_floating_ip\n}\n" }, { "path": "autoscaler-agents.tf", "content": "locals {\n cluster_prefix = var.use_cluster_name_in_node_name ? \"${var.cluster_name}-\" : \"\"\n first_nodepool_snapshot_id = length(var.autoscaler_nodepools) == 0 ? \"\" : (\n substr(var.autoscaler_nodepools[0].server_type, 0, 3) == \"cax\" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id\n )\n\n imageList = {\n arm64 : tostring(data.hcloud_image.microos_arm_snapshot.id)\n amd64 : tostring(data.hcloud_image.microos_x86_snapshot.id)\n }\n\n nodeConfigName = var.use_cluster_name_in_node_name ? \"${var.cluster_name}-\" : \"\"\n cluster_config = {\n imagesForArch : local.imageList\n nodeConfigs : {\n for index, nodePool in var.autoscaler_nodepools :\n (\"${local.nodeConfigName}${nodePool.name}\") => {\n cloudInit = data.cloudinit_config.autoscaler_config[index].rendered\n labels = nodePool.labels\n taints = nodePool.taints\n }\n }\n }\n\n isUsingLegacyConfig = length(var.autoscaler_labels) > 0 || length(var.autoscaler_taints) > 0\n\n autoscaler_yaml = length(var.autoscaler_nodepools) == 0 ? \"\" : templatefile(\n \"${path.module}/templates/autoscaler.yaml.tpl\",\n {\n cloudinit_config = local.isUsingLegacyConfig ? base64encode(data.cloudinit_config.autoscaler_legacy_config[0].rendered) : \"\"\n ca_image = var.cluster_autoscaler_image\n ca_version = var.cluster_autoscaler_version\n ca_replicas = var.cluster_autoscaler_replicas\n ca_resource_limits = var.cluster_autoscaler_resource_limits\n ca_resources = var.cluster_autoscaler_resource_values\n cluster_autoscaler_extra_args = var.cluster_autoscaler_extra_args\n cluster_autoscaler_log_level = var.cluster_autoscaler_log_level\n cluster_autoscaler_log_to_stderr = var.cluster_autoscaler_log_to_stderr\n cluster_autoscaler_stderr_threshold = var.cluster_autoscaler_stderr_threshold\n cluster_autoscaler_server_creation_timeout = tostring(var.cluster_autoscaler_server_creation_timeout)\n ssh_key = local.hcloud_ssh_key_id\n ipv4_subnet_id = data.hcloud_network.k3s.id\n snapshot_id = local.first_nodepool_snapshot_id\n cluster_config = base64encode(jsonencode(local.cluster_config))\n firewall_id = hcloud_firewall.k3s.id\n cluster_name = local.cluster_prefix\n node_pools = var.autoscaler_nodepools\n enable_ipv4 = !(var.autoscaler_disable_ipv4 || local.use_nat_router)\n enable_ipv6 = !(var.autoscaler_disable_ipv6 || local.use_nat_router)\n })\n # A concatenated list of all autoscaled nodes\n autoscaled_nodes = length(var.autoscaler_nodepools) == 0 ? {} : {\n for v in concat([\n for k, v in data.\n hcloud_servers.autoscaled_nodes : [for v in v.servers : v]\n ]...) : v.name => v\n }\n}\n\nresource \"terraform_data\" \"configure_autoscaler\" {\n count = length(var.autoscaler_nodepools) > 0 ? 1 : 0\n\n triggers_replace = {\n template = local.autoscaler_yaml\n }\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.first_control_plane_ip\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Upload the autoscaler resource defintion\n provisioner \"file\" {\n content = local.autoscaler_yaml\n destination = \"/tmp/autoscaler.yaml\"\n }\n\n # Create/Apply the definition\n provisioner \"remote-exec\" {\n inline = [\"kubectl apply -f /tmp/autoscaler.yaml\"]\n }\n\n depends_on = [\n hcloud_load_balancer.cluster,\n terraform_data.control_planes,\n random_password.rancher_bootstrap,\n hcloud_volume.longhorn_volume,\n data.hcloud_image.microos_x86_snapshot\n ]\n}\nmoved {\n from = null_resource.configure_autoscaler\n to = terraform_data.configure_autoscaler\n}\n\ndata \"cloudinit_config\" \"autoscaler_config\" {\n count = length(var.autoscaler_nodepools)\n\n gzip = true\n base64_encode = true\n\n # Main cloud-config configuration file.\n part {\n filename = \"init.cfg\"\n content_type = \"text/cloud-config\"\n content = templatefile(\n \"${path.module}/templates/autoscaler-cloudinit.yaml.tpl\",\n {\n hostname = \"autoscaler\"\n dns_servers = var.dns_servers\n has_dns_servers = local.has_dns_servers\n sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)\n swap_size = var.autoscaler_nodepools[count.index].swap_size\n zram_size = var.autoscaler_nodepools[count.index].zram_size\n k3s_config = yamlencode(merge(\n {\n server = local.k3s_endpoint\n token = local.k3s_token\n # Kubelet arg precedence (last wins): local.kubelet_arg > nodepool.kubelet_args > k3s_global_kubelet_args > k3s_autoscaler_kubelet_args\n kubelet-arg = concat(local.kubelet_arg, var.autoscaler_nodepools[count.index].kubelet_args, var.k3s_global_kubelet_args, var.k3s_autoscaler_kubelet_args)\n flannel-iface = local.flannel_iface\n node-label = concat(local.default_agent_labels, [for k, v in var.autoscaler_nodepools[count.index].labels : \"${k}=${v}\"], var.autoscaler_nodepools[count.index].swap_size != \"\" || var.autoscaler_nodepools[count.index].zram_size != \"\" ? local.swap_node_label : [])\n node-taint = compact(concat(local.default_agent_taints, [for taint in var.autoscaler_nodepools[count.index].taints : \"${taint.key}=${tostring(taint.value)}:${taint.effect}\"]))\n selinux = !var.disable_selinux\n },\n var.agent_nodes_custom_config,\n local.prefer_bundled_bin_config\n ))\n install_k3s_agent_script = join(\"\\n\", concat(local.install_k3s_agent, [\"systemctl start k3s-agent\"]))\n cloudinit_write_files_common = local.cloudinit_write_files_common\n cloudinit_runcmd_common = local.cloudinit_runcmd_common,\n private_network_only = var.autoscaler_disable_ipv4 && var.autoscaler_disable_ipv6,\n network_gw_ipv4 = local.network_gw_ipv4\n }\n )\n }\n}\n\ndata \"cloudinit_config\" \"autoscaler_legacy_config\" {\n count = length(var.autoscaler_nodepools) > 0 && local.isUsingLegacyConfig ? 1 : 0\n\n gzip = true\n base64_encode = true\n\n # Main cloud-config configuration file.\n part {\n filename = \"init.cfg\"\n content_type = \"text/cloud-config\"\n content = templatefile(\n \"${path.module}/templates/autoscaler-cloudinit.yaml.tpl\",\n {\n hostname = \"autoscaler\"\n dns_servers = var.dns_servers\n has_dns_servers = local.has_dns_servers\n sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)\n swap_size = \"\"\n zram_size = \"\"\n k3s_config = yamlencode(merge(\n {\n server = local.k3s_endpoint\n token = local.k3s_token\n kubelet-arg = local.kubelet_arg\n flannel-iface = local.flannel_iface\n node-label = concat(local.default_agent_labels, var.autoscaler_labels)\n node-taint = compact(concat(local.default_agent_taints, var.autoscaler_taints))\n selinux = !var.disable_selinux\n },\n var.agent_nodes_custom_config,\n local.prefer_bundled_bin_config\n ))\n install_k3s_agent_script = join(\"\\n\", concat(local.install_k3s_agent, [\"systemctl start k3s-agent\"]))\n cloudinit_write_files_common = local.cloudinit_write_files_common\n cloudinit_runcmd_common = local.cloudinit_runcmd_common,\n private_network_only = var.autoscaler_disable_ipv4 && var.autoscaler_disable_ipv6,\n network_gw_ipv4 = local.network_gw_ipv4,\n }\n )\n }\n}\n\ndata \"hcloud_servers\" \"autoscaled_nodes\" {\n for_each = toset(var.autoscaler_nodepools[*].name)\n with_selector = \"hcloud/node-group=${local.cluster_prefix}${each.value}\"\n}\n\nresource \"terraform_data\" \"autoscaled_nodes_registries\" {\n for_each = local.autoscaled_nodes\n triggers_replace = {\n registries = var.k3s_registries\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(each.value.ipv4_address, each.value.ipv6_address, try(one(each.value.network).ip, null))\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n provisioner \"file\" {\n content = var.k3s_registries\n destination = \"/tmp/registries.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [local.k3s_registries_update_script]\n }\n}\nmoved {\n from = null_resource.autoscaled_nodes_registries\n to = terraform_data.autoscaled_nodes_registries\n}\n\nresource \"terraform_data\" \"autoscaled_nodes_kubelet_config\" {\n for_each = var.k3s_kubelet_config != \"\" ? local.autoscaled_nodes : {}\n triggers_replace = {\n kubelet_config = var.k3s_kubelet_config\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(each.value.ipv4_address, each.value.ipv6_address, try(one(each.value.network).ip, null))\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n }\n\n provisioner \"file\" {\n content = var.k3s_kubelet_config\n destination = \"/tmp/kubelet-config.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [local.k3s_kubelet_config_update_script]\n }\n}\nmoved {\n from = null_resource.autoscaled_nodes_kubelet_config\n to = terraform_data.autoscaled_nodes_kubelet_config\n}\n" }, { "path": "control_planes.tf", "content": "module \"control_planes\" {\n source = \"./modules/host\"\n\n providers = {\n hcloud = hcloud,\n }\n\n for_each = local.control_plane_nodes\n\n name = \"${var.use_cluster_name_in_node_name ? \"${var.cluster_name}-\" : \"\"}${each.value.nodepool_name}\"\n microos_snapshot_id = substr(each.value.server_type, 0, 3) == \"cax\" ? data.hcloud_image.microos_arm_snapshot.id : data.hcloud_image.microos_x86_snapshot.id\n base_domain = var.base_domain\n ssh_keys = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]\n ssh_port = var.ssh_port\n ssh_public_key = var.ssh_public_key\n ssh_private_key = var.ssh_private_key\n ssh_additional_public_keys = length(var.ssh_hcloud_key_label) > 0 ? concat(var.ssh_additional_public_keys, data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.public_key) : var.ssh_additional_public_keys\n firewall_ids = each.value.disable_ipv4 && each.value.disable_ipv6 ? [] : [hcloud_firewall.k3s.id] # Cannot attach a firewall when public interfaces are disabled\n placement_group_id = var.placement_group_disable ? null : (each.value.placement_group == null ? hcloud_placement_group.control_plane[each.value.placement_group_compat_idx].id : hcloud_placement_group.control_plane_named[each.value.placement_group].id)\n location = each.value.location\n server_type = each.value.server_type\n backups = each.value.backups\n ipv4_subnet_id = hcloud_network_subnet.control_plane[[for i, v in var.control_plane_nodepools : i if v.name == each.value.nodepool_name][0]].id\n dns_servers = var.dns_servers\n k3s_registries = var.k3s_registries\n k3s_registries_update_script = local.k3s_registries_update_script\n k3s_kubelet_config = var.k3s_kubelet_config\n k3s_kubelet_config_update_script = local.k3s_kubelet_config_update_script\n k3s_audit_policy_config = var.k3s_audit_policy_config\n k3s_audit_policy_update_script = local.k3s_audit_policy_update_script\n cloudinit_write_files_common = local.cloudinit_write_files_common\n cloudinit_runcmd_common = local.cloudinit_runcmd_common\n swap_size = each.value.swap_size\n zram_size = each.value.zram_size\n keep_disk_size = var.keep_disk_cp\n disable_ipv4 = each.value.disable_ipv4\n disable_ipv6 = each.value.disable_ipv6\n ssh_bastion = local.ssh_bastion\n network_id = data.hcloud_network.k3s.id\n\n # We leave some room so 100 eventual Hetzner LBs that can be created perfectly safely\n # It leaves the subnet with 254 x 254 - 100 = 64416 IPs to use, so probably enough.\n private_ipv4 = cidrhost(hcloud_network_subnet.control_plane[[for i, v in var.control_plane_nodepools : i if v.name == each.value.nodepool_name][0]].ip_range, each.value.index + (local.network_size >= 16 ? 101 : floor(pow(local.subnet_size, 2) * 0.4)))\n\n labels = merge(local.labels, local.labels_control_plane_node)\n\n automatically_upgrade_os = var.automatically_upgrade_os\n\n network_gw_ipv4 = local.network_gw_ipv4\n\n depends_on = [\n hcloud_network_subnet.control_plane,\n hcloud_placement_group.control_plane,\n hcloud_server.nat_router,\n terraform_data.nat_router_await_cloud_init,\n ]\n}\n\nresource \"hcloud_load_balancer\" \"control_plane\" {\n count = var.use_control_plane_lb ? 1 : 0\n name = \"${var.cluster_name}-control-plane\"\n\n load_balancer_type = var.control_plane_lb_type\n location = var.load_balancer_location\n labels = merge(local.labels, local.labels_control_plane_lb)\n delete_protection = var.enable_delete_protection.load_balancer\n\n lifecycle {\n ignore_changes = [location]\n }\n}\n\nresource \"hcloud_load_balancer_network\" \"control_plane\" {\n count = var.use_control_plane_lb ? 1 : 0\n\n load_balancer_id = hcloud_load_balancer.control_plane.*.id[0]\n subnet_id = hcloud_network_subnet.control_plane.*.id[0]\n enable_public_interface = var.control_plane_lb_enable_public_interface\n ip = cidrhost(hcloud_network_subnet.control_plane.*.ip_range[0], -2)\n\n # Keep existing LB IPs stable on upgrade.\n lifecycle {\n ignore_changes = [ip]\n }\n}\n\nresource \"hcloud_load_balancer_target\" \"control_plane\" {\n count = var.use_control_plane_lb ? 1 : 0\n\n depends_on = [hcloud_load_balancer_network.control_plane]\n type = \"label_selector\"\n load_balancer_id = hcloud_load_balancer.control_plane.*.id[0]\n label_selector = join(\",\", [for k, v in merge(local.labels, local.labels_control_plane_node) : \"${k}=${v}\"])\n use_private_ip = true\n}\n\nresource \"hcloud_load_balancer_service\" \"control_plane\" {\n count = var.use_control_plane_lb ? 1 : 0\n\n load_balancer_id = hcloud_load_balancer.control_plane.*.id[0]\n protocol = \"tcp\"\n destination_port = \"6443\"\n listen_port = \"6443\"\n}\n\nlocals {\n control_plane_endpoint_host = var.control_plane_endpoint != null ? one(compact(regexall(\"^(?:https?://)?(?:.*@)?(?:\\\\[([a-fA-F0-9:]+)\\\\]|([^:/?#]+))\", var.control_plane_endpoint)[0])) : null\n\n control_plane_ips = {\n for k, v in module.control_planes : k => coalesce(\n v.ipv4_address,\n v.ipv6_address,\n v.private_ipv4_address\n )\n }\n\n k3s-config = { for k, v in local.control_plane_nodes : k => merge(\n {\n node-name = module.control_planes[k].name\n server = length(module.control_planes) == 1 ? null : coalesce(\n var.control_plane_endpoint,\n \"https://${\n var.use_control_plane_lb ? hcloud_load_balancer_network.control_plane.*.ip[0] :\n (\n module.control_planes[k].private_ipv4_address == module.control_planes[keys(module.control_planes)[0]].private_ipv4_address ?\n module.control_planes[keys(module.control_planes)[1]].private_ipv4_address :\n module.control_planes[keys(module.control_planes)[0]].private_ipv4_address\n )\n }:6443\"\n )\n token = local.k3s_token\n disable-cloud-controller = true\n disable-kube-proxy = var.disable_kube_proxy\n disable = local.disable_extras\n # Kubelet arg precedence (last wins): local.kubelet_arg > v.kubelet_args > k3s_global_kubelet_args > k3s_control_plane_kubelet_args\n kubelet-arg = concat(local.kubelet_arg, v.kubelet_args, var.k3s_global_kubelet_args, var.k3s_control_plane_kubelet_args)\n kube-apiserver-arg = local.kube_apiserver_arg\n kube-controller-manager-arg = local.kube_controller_manager_arg\n flannel-iface = local.flannel_iface\n node-ip = module.control_planes[k].private_ipv4_address\n advertise-address = module.control_planes[k].private_ipv4_address\n node-label = v.labels\n node-taint = v.taints\n selinux = var.disable_selinux ? false : (v.selinux == true ? true : false)\n cluster-cidr = var.cluster_ipv4_cidr\n service-cidr = var.service_ipv4_cidr\n cluster-dns = local.cluster_dns_ipv4\n write-kubeconfig-mode = \"0644\" # needed for import into rancher\n },\n lookup(local.cni_k3s_settings, var.cni_plugin, {}),\n var.use_control_plane_lb ? {\n tls-san = concat(\n compact([\n hcloud_load_balancer.control_plane.*.ipv4[0],\n hcloud_load_balancer_network.control_plane.*.ip[0],\n var.kubeconfig_server_address != \"\" ? var.kubeconfig_server_address : null,\n local.control_plane_endpoint_host,\n !var.control_plane_lb_enable_public_interface && var.nat_router != null ? hcloud_server.nat_router[0].ipv4_address : null\n ]),\n var.additional_tls_sans\n )\n } : {\n tls-san = concat(\n compact([\n local.control_plane_endpoint_host,\n module.control_planes[k].ipv4_address != \"\" ? module.control_planes[k].ipv4_address : null,\n module.control_planes[k].ipv6_address != \"\" ? module.control_planes[k].ipv6_address : null,\n try(one(module.control_planes[k].network).ip, null)\n ]),\n var.additional_tls_sans)\n },\n local.etcd_s3_snapshots,\n var.control_planes_custom_config,\n local.prefer_bundled_bin_config\n ) }\n}\n\nresource \"terraform_data\" \"control_plane_config\" {\n for_each = local.control_plane_nodes\n\n triggers_replace = {\n control_plane_id = module.control_planes[each.key].id\n config = sha1(yamlencode(local.k3s-config[each.key]))\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.control_plane_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Generating k3s server config file\n provisioner \"file\" {\n content = yamlencode(local.k3s-config[each.key])\n destination = \"/tmp/config.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [local.k3s_config_update_script]\n }\n\n depends_on = [\n terraform_data.first_control_plane,\n hcloud_network_subnet.control_plane\n ]\n}\nmoved {\n from = null_resource.control_plane_config\n to = terraform_data.control_plane_config\n}\n\nresource \"terraform_data\" \"audit_policy\" {\n for_each = local.control_plane_nodes\n\n triggers_replace = {\n control_plane_id = module.control_planes[each.key].id\n audit_policy = sha1(var.k3s_audit_policy_config)\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.control_plane_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n }\n\n provisioner \"file\" {\n content = var.k3s_audit_policy_config\n destination = \"/tmp/audit-policy.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [local.k3s_audit_policy_update_script]\n }\n\n depends_on = [\n terraform_data.first_control_plane,\n hcloud_network_subnet.control_plane\n ]\n}\nmoved {\n from = null_resource.audit_policy\n to = terraform_data.audit_policy\n}\n\nresource \"terraform_data\" \"authentication_config\" {\n for_each = local.control_plane_nodes\n\n triggers_replace = {\n control_plane_id = module.control_planes[each.key].id\n authentication_config = sha1(var.authentication_config)\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.control_plane_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n provisioner \"file\" {\n content = var.authentication_config\n destination = \"/tmp/authentication_config.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [local.k3s_authentication_config_update_script]\n }\n\n depends_on = [\n terraform_data.first_control_plane,\n hcloud_network_subnet.control_plane\n ]\n}\nmoved {\n from = null_resource.authentication_config\n to = terraform_data.authentication_config\n}\n\nresource \"terraform_data\" \"control_planes\" {\n for_each = local.control_plane_nodes\n\n triggers_replace = {\n control_plane_id = module.control_planes[each.key].id\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.control_plane_ips[each.key]\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Install k3s server\n provisioner \"remote-exec\" {\n inline = local.install_k3s_server\n }\n\n # Start the k3s server and wait for it to have started correctly\n provisioner \"remote-exec\" {\n inline = [\n \"systemctl start k3s 2> /dev/null\",\n # prepare the needed directories\n \"mkdir -p /var/post_install /var/user_kustomize\",\n # wait for the server to be ready\n <<-EOT\n timeout 360 bash <<EOF\n until systemctl status k3s > /dev/null; do\n systemctl start k3s 2> /dev/null\n echo \"Waiting for the k3s server to start...\"\n sleep 3\n done\n EOF\n EOT\n ]\n }\n\n depends_on = [\n terraform_data.first_control_plane,\n terraform_data.control_plane_config,\n terraform_data.authentication_config,\n hcloud_network_subnet.control_plane\n ]\n}\nmoved {\n from = null_resource.control_planes\n to = terraform_data.control_planes\n}\n" }, { "path": "data.tf", "content": "data \"github_release\" \"hetzner_ccm\" {\n count = var.hetzner_ccm_version == null ? 1 : 0\n repository = \"hcloud-cloud-controller-manager\"\n owner = \"hetznercloud\"\n retrieve_by = \"latest\"\n}\n\ndata \"github_release\" \"hetzner_csi\" {\n count = var.hetzner_csi_version == null && !var.disable_hetzner_csi ? 1 : 0\n repository = \"csi-driver\"\n owner = \"hetznercloud\"\n retrieve_by = \"latest\"\n}\n\n// github_release for kured\ndata \"github_release\" \"kured\" {\n count = var.kured_version == null ? 1 : 0\n repository = \"kured\"\n owner = \"kubereboot\"\n retrieve_by = \"latest\"\n}\n\n// github_release for kured\ndata \"github_release\" \"calico\" {\n count = var.calico_version == null && var.cni_plugin == \"calico\" ? 1 : 0\n repository = \"calico\"\n owner = \"projectcalico\"\n retrieve_by = \"latest\"\n}\n\ndata \"hcloud_ssh_keys\" \"keys_by_selector\" {\n count = length(var.ssh_hcloud_key_label) > 0 ? 1 : 0\n with_selector = var.ssh_hcloud_key_label\n}\n" }, { "path": "docs/add-robot-server.md", "content": "# Hetzner Robot Server Integration using HCCM v1.19+\n\nThis guide describes how to add Hetzner **robot servers** to a Kubernetes cluster with help of the [hcloud-cloud-controller-manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager), version 1.19 or newer.\nIt covers configuration for both k3s and Robot nodes, including networking, configuration, and caveats. Alternatives like WireGuard exist, but are not covered here.\n\n---\n\n## Prerequisites for connecting a Robot node to a new or already existing Cluster\n\n- **Hetzner vSwitch** \n - The recommended way is using a **vSwitch**, which connects the project-level Cloud subnets to the Robot node.\n - This guide assumes the vSwitch has been created and is not currently connected to any subnet. The vSwitch can be created in the Hetzner Robot web-UI. See [Hetzner Docs](https://docs.hetzner.com/robot/dedicated-server/network/vswitch)\n - Note down the vSwitch ID and the VLAN ID. Note: vSwitch IDs are in the number range of around 10000+, while VLAN ID range is account-specific and starts from 4000 by default.\n- **Webservice User** created in Hetzner Robot account settings (for API access)\n - This is required for `hccm` to list robot servers via the metadata endpoint:\n - `https://169.254.169.254/hetzner/v1/metadata/instance-id`\n- `hccm` version **1.19 or newer**\n- **Operating System**: Ideally use the MicroOS image created by this project. Otherwise, any Linux distribution that supports k3s will work\n- **Network CNI Configuration**: \n - Flannel: Doesn't need additional configuration.\n - Cilium: Doesn't need additional configuration, ensure `cilium_loadbalancer_acceleration_mode` is set to `\"best-effort\"` or `\"disabled\"`\n - Calico: Untested\n\n---\n\n## 1. Connection from Kubernetes Cluster to vSwitch\n\nIn your kube.tf-configuration:\n - Set `robot_ccm_enabled = true` and provide the Webservice User credentials in the `robot_user` and `robot_password` variables. All three are required to enable Robot server integration. If `robot_ccm_enabled` is true but credentials are not provided, the integration will not be activated.\n - Set `vswitch_id = <vswitch_id from prerequisites>`\n\nFor manual configuration of the settings, see below:\n\n<details>\n<summary>Manual configuration of HCCM-settings and vSwitch connection</summary>\n\n### 1. HCCM-settings\n\n- **Update the `hcloud` Kubernetes secret** with your `robot-user` and `robot-password`.\n- Set `robot.enabled: true` in `hetzner_ccm_values`.\n- Set the correct `cluster-cidr` (the pod subnet for your cluster).\n- Deploy `hccm` version **1.19 or newer**.\n- Refer to [HCCM Github if required](https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/a0217eafe74c8704a5e8086cc774ceb3de8f04e3/chart/values.yaml#L54)\n\n### 2. Connect the Existing Cluster Subnet manually to vSwitch \n\n1. Choose a subnet CIDR to be used for the Robot nodes that doesn't conflict with the existing Cluster subnets, such as 10.201.0.0/16.\n2. Connect the existing Cluster Cloud network to the previously created vSwitch in the web-UI and expose the routes to vSwitch. \n - Follow the steps in [Hetzner docs](https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch) on how to connect the Cluster Subnets to the vSwitch. Use your selected subnet CIDR and VLAN ID.\n\n</details>\n\n\n---\n\n## 2. Connect the Robot to the vSwitch \n\n1. Follow the steps in \"Step 2: Configure networking on your dedicated root servers\" in [Hetzner docs](https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch/#step-2-configure-networking-on-your-dedicated-root-servers) to connect the Robot node to the vSwitch.\n - Use your selected VLAN ID. \n - If you created the Cloud->vSwitch connection via Terraform in the Step 1 of this guide, the default range for Robot is 10.201.0.0/16. The gateway is then at 10.201.0.1 and first Robot node should use private IP 10.201.0.2. \n - Make sure to use MTU 1400 or less. Cilium is reported to be requiring MTU 1350 or less.\n\n<details>\n<summary>Robot Network configuration example for RHEL/AlmaLinux using nmcli</summary>\n\nAssumptions (change these to your values!):\n- vSwitch subnet: `10.201.0.0/16`\n- VLAN ID: `4000` # \"arbitrary\" value, replace with your VLAN ID\n- Main interface: `enp6s0`\n\n> [!CAUTION]\n> The routes and CIDR notations depend on your local setup and may vary depending on your network configuration.\n\n```bash\nnmcli connection add type vlan con-name vlan4000 ifname vlan4000 vlan.parent enp6s0 vlan.id 4000\n\nnmcli connection modify vlan4000 802-3-ethernet.mtu 1400 # Important: vSwitch requires MTU 1400 max.\nnmcli connection modify vlan4000 ipv4.addresses '10.201.0.2/16'\nnmcli connection modify vlan4000 ipv4.gateway '10.201.0.1'\nnmcli connection modify vlan4000 ipv4.method manual\n# Route all 10.x IPs through the vSwitch gateway\nnmcli connection modify vlan4000 +ipv4.routes \"10.0.0.0/8 10.201.0.1\"\n\n# Apply the config\nnmcli connection down vlan4000\nnmcli connection up vlan4000\n```\n\n</details>\n\n---\n## 3. Verify Network connectivity\n1. Log in to your Robot Node using SSH and ping one of the Cloud Control Plane nodes Private Network IP. (e.g., 10.255.0.101).\n2. Log in to one of the Cloud Control Plane nodes using SSH and ping the Robot Node Private Network IP, such as 10.201.0.2.\n\n\n<details>\n<summary>Troubleshoot Robot Node networking</summary>\n\n- Make sure the IP address and routing are correct on the Robot Node.\n- Following examples assume Robot Node public IP 203.0.113.123, private IP 10.201.0.2, VLAN ID 4000 and device enp6s0.\n- `ip route show` on the Robot Node should print similar to this:\n```\ndefault via 203.0.113.123 dev enp6s0 proto static onlink \n10.0.0.0/8 via 10.201.0.1 dev enp6s0.4000 proto static onlink \n10.201.0.0/16 dev enp6s0.4000 proto kernel scope link src 10.201.0.2 \n```\n- `ip addr` on the Robot Node should include similar to this:\n```\n2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000\n link/ether a8:a1:REDACTED brd ff:ff:ff:ff:ff:ff\n inet 203.0.113.123/32 scope global enp6s0\n valid_lft forever preferred_lft forever\n inet6 2a01:REDACTED/64 scope global \n valid_lft forever preferred_lft forever\n inet6 fe80::REDACTED/64 scope link \n valid_lft forever preferred_lft forever\n3: enp6s0.4000@enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000\n link/ether a8:a1:REDACTED brd ff:ff:ff:ff:ff:ff\n inet 10.201.0.2/16 brd 10.201.255.255 scope global enp6s0.4000\n valid_lft forever preferred_lft forever\n inet6 fe80::REDACTED/64 scope link \n valid_lft forever preferred_lft forever\n``` \n- You may want to try to \"Refresh\" the vSwitch connection in the Robot web-UIs vSwitches admin-panel. Select the vSwitch, then Robot Node and click Refresh.\n- Try rebooting the Robot Node\n\n</details>\n\n---\n\n## 4. Robot Node: k3s Agent Configuration\n\n> [!IMPORTANT]\n> If you set a Nodename for the k3s-agent, it must match the server name in the Hetzner Robot Web-UI.\n\n1. **Create `/etc/rancher/k3s/config.yaml`** on the robot node:\n\n ```yaml\n flannel-iface: enp6s0 # Set to your main interface (only needed for Flannel CNI)\n prefer-bundled-bin: true\n kubelet-arg:\n - cloud-provider=external\n - volume-plugin-dir=/var/lib/kubelet/volumeplugins\n - kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\n - system-reserved=cpu=250m,memory=6000Mi # Optional: reserve some space for system\n node-label:\n - k3s_upgrade=true\n - instance.hetzner.cloud/provided-by=robot # To prevent Hetzner CSI pods from being scheduled on robot nodes\n node-taint: []\n selinux: true\n server: https://<API_SERVER_IP>:6443 # Replace with your API server IP\n token: <CLUSTER_TOKEN> # Replace with your cluster token\n ```\n\n---\n\n## 5. Storage and Scheduling Notes\n\n- **Hetzner Cloud Volumes** do **not** work on robot servers (CSI driver limitation).\n - Use [Longhorn](https://longhorn.io/) or other external storage.\n - Pods using cloud volumes cannot be scheduled on robot nodes.\n- **Longhorn**: Install `open-iscsi` and start the service:\n ```bash\n sudo dnf install -y iscsi-initiator-utils\n sudo systemctl start iscsid\n ```\n- **Node Scheduling**:\n - Use taints and labels to control pod placement.\n - To prevent Hetzner CSI pods from being scheduled on robot nodes, apply the label:\n ```\n instance.hetzner.cloud/provided-by=robot\n ```\n [Reference](https://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#integration-with-root-servers)\n\n---\n\n## 6. Caveats & Warnings\n\n- This setup may not cover all edge cases (e.g., other CNIs, non-wireguard clusters, complex private networks).\n- When destroying the cluster, it takes a few minutes for the vSwitch binding to be released on the Robot side.\n- **Test your network thoroughly** before adding robot nodes to production clusters.\n- **MTU Issues**: When using vSwitch, MTU configuration is critical:\n - vSwitch has a maximum MTU of 1400\n - Some users report needing even lower MTU values (e.g., 1350 or less) for stable operation\n - This particularly affects Cilium CNI users\n - Without proper MTU configuration, you may experience:\n - Pods unable to connect to the Kubernetes API\n - Network instability for pods not using host networking\n - Intermittent connection issues\n - Test different MTU values if you encounter network issues\n\n---\n\n## References\n\n- [Hetzner Cloud Controller Manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager)\n- [Hetzner vSwitch & Robot Networking](https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch)\n- [Hetzner CSI Driver: Root Server Integration](https://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#integration-with-root-servers)\n" }, { "path": "docs/customize-mount-path-longhorn.md", "content": "## How to use a custom mount path for Longhorn\n<hr>\n\nIn order to use NVMe and external disks with Longhorn, you may need to mount an external disk to a location other than the default under the `/var/` folder. This can provide more storage capacity across the cluster, especially if you haven't disabled the default Longhorn disks.\n\n> ⚠️ Note: You can set any mount path, but it must be within the `/var/` folder.\n\n### How to set a custom mount path for your external disk?\n\n1. You must enable Longhorn in your module.\n ```terraform\n enable_longhorn = true\n ```\n\n2. Set the Helm values for Longhorn. The `defaultDataPath` is important as this path is automatically created by Longhorn and will be the default storage class pointing to your primary disks (e.g., NVMe).\n ```yaml\n longhorn_values = <<-EOT\n defaultSettings:\n nodeDrainPolicy: allow-if-replica-is-stopped\n defaultDataPath: /var/longhorn\n persistence:\n defaultFsType: ext4\n defaultClassReplicaCount: 3\n defaultClass: true\n EOT\n ```\n\n3. In the `agent_nodepools` where you want to have a customized mount path, set the `longhorn_mount_path` variable. It's a good practice to define this path as a local variable to ensure consistency.\n\n ```terraform\n locals {\n custom_longhorn_path = \"/var/lib/longhorn\"\n }\n\n agent_nodepools = [\n {\n # ... other nodepool configuration\n labels = [\"role=monitoring\", \"storage=ssd\"], # Label we use to filter nodes\n longhorn_volume_size = 50,\n longhorn_mount_path = local.custom_longhorn_path # This is the custom path\n }\n ]\n ```\n\n4. Apply the changes. As a result, your external disks will be mounted to the path defined in `local.custom_longhorn_path`.\n\n### How to configure Longhorn to use the new path?\n\nAfter setting the custom mount path, you need to configure Longhorn to recognize and use it. This typically involves:\n1. Patching the Longhorn nodes to add the new disk.\n2. Creating a new StorageClass that uses the new disk.\n\nHere is an example of how you can achieve this with Terraform:\n\n```terraform\n# Find the nodes with the 'ssd' storage label\ndata \"kubernetes_nodes\" \"ssd_nodes\" {\n metadata {\n labels = {\n \"storage\" = \"ssd\"\n }\n }\n}\n\n# Patch the selected Longhorn nodes to add the new disk\n# IMPORTANT: The \"path\" value below must match the 'longhorn_mount_path' for the nodes\n# selected by the 'storage=ssd' label.\nresource \"terraform_data\" \"longhorn_patch_external_disk\" {\n for_each = {\n for node in data.kubernetes_nodes.ssd_nodes.nodes : node.metadata[0].name => node.metadata[0].name\n }\n provisioner \"local-exec\" {\n command = <<-EOT\n KUBECONFIG=${var.kubeconfig_path} kubectl -n longhorn-system patch nodes.longhorn.io ${each.key} --type merge -p '{\n \"spec\": {\n \"disks\": {\n \"external-ssd\": {\n \"path\": \"${local.custom_longhorn_path}\",\n \"allowScheduling\": true,\n \"tags\": [\"ssd\"]\n }\n }\n }\n }'\n EOT\n }\n}\n\n# Create a new StorageClass for the SSD-backed Longhorn storage\nresource \"kubernetes_manifest\" \"longhorn_ssd_storageclass\" {\n manifest = {\n apiVersion = \"storage.k8s.io/v1\"\n kind = \"StorageClass\"\n metadata = {\n name = \"longhorn-ssd\"\n }\n provisioner = \"driver.longhorn.io\"\n parameters = {\n numberOfReplicas = \"3\"\n staleReplicaTimeout = \"30\"\n diskSelector = \"ssd\"\n fromBackup = \"\"\n }\n reclaimPolicy = \"Delete\"\n allowVolumeExpansion = true\n volumeBindingMode = \"Immediate\"\n }\n\n depends_on = [terraform_data.longhorn_patch_external_disk]\n}" }, { "path": "docs/llms.md", "content": "**An Intricate Guide to Configuring the `kube-hetzner` Terraform Module for k3s on Hetzner Cloud**\n\n**Preamble: Understanding the Landscape**\n\nBefore diving into the specifics of the configuration file, it's crucial to understand the core components and philosophies at play:\n\n* **Terraform:** An Infrastructure as Code (IaC) tool that allows you to define and provision data center infrastructure using a declarative configuration language. It manages the lifecycle of your resources.\n* **Hetzner Cloud (hcloud):** The IaaS provider where your Kubernetes cluster will reside. Terraform will interact with the Hetzner Cloud API to create servers, networks, load balancers, etc.\n* **k3s:** A lightweight, certified Kubernetes distribution. It's designed to be lean and easy to install, making it ideal for edge, IoT, CI, and, as in this case, relatively straightforward cloud deployments. The `kube-hetzner` module specifically targets k3s.\n* **`kube-hetzner/kube-hetzner/hcloud` Module:** A community-maintained Terraform module that abstracts away the complexity of setting up a k3s cluster on Hetzner Cloud. It provides a set of configurable inputs to define your desired cluster topology and features.\n* **Declarative Configuration:** You *declare* the desired state of your infrastructure, and Terraform, with the help of the module, figures out how to achieve that state.\n* **Idempotency:** Applying the same Terraform configuration multiple times should result in the same state, without unintended side effects (though some module operations might have nuances).\n\nThis guide will walk through the provided Terraform configuration, explaining the purpose, implications, and interdependencies of each setting.\n\n---\n\n**Section 1: `locals` Block - Foundational Variables**\n\n```terraform\nlocals {\n # You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env\n # within your shell, such as: export TF_VAR_hcloud_token=xxxxxxxxxxx\n # If you choose to define it in the shell, this can be left as is.\n\n # Your Hetzner token can be found in your Project > Security > API Token (Read & Write is required).\n hcloud_token = \"xxxxxxxxxxx\"\n}\n```\n\n* **Purpose:** The `locals` block defines local variables within your Terraform configuration. These are not exposed as input variables to the module but are used internally within this root configuration file.\n* **`hcloud_token`:**\n * **Significance:** This is arguably the most critical piece of sensitive information. It's the API token that grants Terraform programmatic access to your Hetzner Cloud account to create, modify, and delete resources.\n * **Permissions:** As noted, the token *must* have \"Read & Write\" permissions. A read-only token would allow Terraform to plan but fail during the apply phase when attempting to create resources.\n * **Security Considerations:**\n * **Hardcoding (as shown):** `hcloud_token = \"xxxxxxxxxxx\"` is convenient for quick tests but is a **significant security risk** if this file is committed to version control (e.g., Git) or shared.\n * **Environment Variable (Recommended):** The comment `export TF_VAR_hcloud_token=xxxxxxxxxxx` highlights the best practice. Terraform automatically picks up environment variables prefixed with `TF_VAR_`. So, `TF_VAR_hcloud_token` will populate a Terraform variable named `hcloud_token` (which we'll see defined later). This keeps the sensitive token out of your configuration files.\n * **Other Secret Management:** For more advanced setups, tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault could be used, with Terraform fetching the token at runtime.\n * **Interaction:** This `local.hcloud_token` is used as a fallback if the `var.hcloud_token` (populated by the environment variable or a `terraform.tfvars` file) is not set. The module instantiation later uses a conditional: `var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token`.\n\n---\n\n**Section 2: `module \"kube-hetzner\"` Block - The Core Orchestration**\n\nThis block is where the magic happens. It instantiates the `kube-hetzner` module, passing it all the necessary configurations.\n\n```terraform\nmodule \"kube-hetzner\" {\n providers = {\n hcloud = hcloud\n }\n hcloud_token = var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token\n```\n\n* **`module \"kube-hetzner\"`:** This declares an instance of a Terraform module. The name \"kube-hetzner\" here is arbitrary for this instance; you could call it \"my_cluster\" if you wished, though consistency with the module name is common.\n* **`providers` Block:**\n * **Purpose:** Terraform modules can define their own provider requirements. When a module uses a provider (like `hcloud`), the calling configuration (this root module) needs to explicitly pass that provider configuration to the child module.\n * **`hcloud = hcloud`:** This line tells the `kube-hetzner` module to use the `hcloud` provider configuration defined in *this* root `main.tf` file (which we'll see at the end). This ensures that the module and the root configuration are using the same Hetzner account and settings.\n* **`hcloud_token = var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token`:**\n * **Purpose:** This is an input variable for the `kube-hetzner` module itself. The module needs the Hetzner API token to function.\n * **Logic:** This is a ternary conditional operator.\n * `var.hcloud_token != \"\"`: It checks if the input variable `hcloud_token` (defined at the root level, typically populated by `TF_VAR_hcloud_token`) is not an empty string.\n * `? var.hcloud_token`: If true (the environment variable is set), use its value.\n * `: local.hcloud_token`: If false (the environment variable is not set or is empty), fall back to using the `hcloud_token` defined in the `locals` block at the top of this file.\n * **Benefit:** This provides flexibility in how the token is supplied, prioritizing environment variables for better security practices.\n\n```terraform\n # Then fill or edit the below values. Only the first values starting with a * are obligatory; the rest can remain with their default values, or you\n # could adapt them to your needs.\n\n # * source can be specified in multiple ways:\n # 1. For normal use, (the official version published on the Terraform Registry), use\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n # When using the terraform registry as source, you can optionally specify a version number.\n # See https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud for the available versions\n # version = \"2.15.3\"\n # 2. For local dev, path to the git repo\n # source = \"../../kube-hetzner/\"\n # 3. If you want to use the latest master branch (see https://developer.hashicorp.com/terraform/language/modules/sources#github), use\n # source = \"github.com/kube-hetzner/terraform-hcloud-kube-hetzner\"\n```\n\n* **`source` (Obligatory):**\n * **Purpose:** This tells Terraform where to find the `kube-hetzner` module code.\n * **Option 1 (Terraform Registry - Recommended for Users):** `kube-hetzner/kube-hetzner/hcloud`\n * This is the standard way to use published modules. Terraform will download it from the public Terraform Registry.\n * **`version`:** It's highly recommended to pin the module version (e.g., `version = \"2.15.3\"`). This ensures:\n * **Reproducibility:** Your infrastructure builds are consistent over time.\n * **Stability:** Prevents unexpected changes or breakages if a new, incompatible version of the module is released.\n * **Controlled Upgrades:** You can consciously decide when to upgrade the module version after reviewing its changelog.\n * **Option 2 (Local Path - For Module Developers/Contributors):** `source = \"../../kube-hetzner/\"`\n * Used when you have a local copy of the module's source code, typically for development or testing modifications to the module itself. The path is relative to this `main.tf` file.\n * **Option 3 (Direct Git Repository - For Bleeding Edge/Specific Commits):** `source = \"github.com/kube-hetzner/terraform-hcloud-kube-hetzner\"`\n * Pulls the module directly from the `master` branch of the GitHub repository. This is generally **not recommended for production** as `master` can be unstable.\n * You can also specify a specific branch, tag, or commit hash using the `ref` query parameter (e.g., `source = \"github.com/kube-hetzner/terraform-hcloud-kube-hetzner?ref=v2.15.3\"`).\n\n```terraform\n # Note that some values, notably \"location\" and \"public_key\" have no effect after initializing the cluster.\n # This is to keep Terraform from re-provisioning all nodes at once, which would lose data. If you want to update\n # those, you should instead change the value here and manually re-provision each node. Grep for \"lifecycle\".\n```\n\n* **Important Note on Immutability:**\n * This comment highlights a critical aspect of how this module (and often Terraform resources in general) handles certain changes.\n * **`location` and `public_key`:** For existing server nodes, changing these attributes in the Terraform configuration *after* the initial `terraform apply` will not automatically trigger a change on the Hetzner Cloud server itself through a simple `terraform apply`.\n * **Reasoning (Data Preservation):** If Terraform were to change the location of a server, it would mean destroying the old server and creating a new one, leading to data loss. Similarly, changing the primary SSH key might involve complex OS-level operations or re-provisioning.\n * **Module's Approach (`lifecycle` block):** The module likely uses Terraform's `lifecycle` meta-argument, specifically `ignore_changes`, on these attributes for the server resources. This tells Terraform to create the resource with the initial value but then ignore any subsequent changes to that attribute in the configuration for plan/apply purposes.\n * **Manual Intervention Required:** If you *need* to change these, you must:\n 1. Update the value in your Terraform configuration.\n 2. Manually re-provision the affected node(s). This could involve:\n * Cordoning and draining the node in Kubernetes.\n * Using `terraform taint <resource_address>` to mark the specific server resource for recreation on the next `apply`.\n * Manually deleting the server in Hetzner Cloud and letting Terraform recreate it.\n * This is a deliberate design choice to prevent accidental data loss or full cluster rebuilds for minor changes to sensitive, foundational attributes.\n\n```terraform\n # Customize the SSH port (by default 22)\n # ssh_port = 2222\n```\n\n* **`ssh_port` (Optional):**\n * **Default:** `22`\n * **Purpose:** Allows you to specify a custom SSH port for the nodes created by the module. The module will configure the SSH daemon on the nodes to listen on this port and adjust firewall rules accordingly.\n * **Use Case:** Security through obscurity (minor benefit) or if port 22 is blocked/used by something else in your environment.\n * **Implication:** You'll need to specify this custom port when SSHing into the nodes (e.g., `ssh -p 2222 user@node_ip`).\n\n```terraform\n # * Your ssh public key\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n # * Your private key must be \"ssh_private_key = null\" when you want to use ssh-agent for a Yubikey-like device authentication or an SSH key-pair with a passphrase.\n # For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n # You can add additional SSH public Keys to grant other team members root access to your cluster nodes.\n # ssh_additional_public_keys = []\n```\n\n* **`ssh_public_key` (Obligatory):**\n * **Purpose:** The content of your SSH public key. This key will be added to the `authorized_keys` file on all created nodes, allowing you to SSH into them as the root user (or the default user configured by the OS image).\n * **`file(\"~/.ssh/id_ed25519.pub\")`:** The `file()` function reads the content of the specified file. `~` is expanded to your home directory. Ensure this path is correct.\n * **Security:** This is the primary means of accessing your nodes. Protect your corresponding private key.\n* **`ssh_private_key` (Obligatory, but can be `null`):**\n * **Purpose:** The content of your SSH private key. This is used by Terraform's provisioners (if the module uses them for direct SSH commands during setup) or by tools like Ansible if integrated. It's also used for generating the kubeconfig if it needs to SSH into a node to fetch it.\n * **`file(\"~/.ssh/id_ed25519\")`:** Reads the private key content.\n * **`ssh_private_key = null` (Conditional Usage):**\n * **When to use `null`:** If your private key is passphrase-protected, or if you're using an SSH agent (e.g., with a YubiKey or `ssh-add`), you *must* set this to `null`. Terraform cannot directly use a passphrase-protected key without the passphrase.\n * **SSH Agent Reliance:** When `null`, Terraform (and underlying tools used by the module) will attempt to use an already configured SSH agent to authenticate. Ensure your key is added to the agent (`ssh-add ~/.ssh/your_private_key`).\n * **Security:** Hardcoding the private key content via `file()` is less secure if the `.tf` file is shared. Using `null` with an SSH agent is generally preferred for keys with passphrases.\n* **`ssh_additional_public_keys` (Optional):**\n * **Default:** `[]` (empty list)\n * **Purpose:** A list of strings, where each string is the content of an additional SSH public key. These keys will also be added to `authorized_keys` on the nodes.\n * **Use Case:** Granting SSH access to other team members or automated systems without sharing your primary private key.\n * **Format:** `ssh_additional_public_keys = [file(\"~/.ssh/teammate1.pub\"), \"ssh-rsa AAAAB3NzaC1yc2EAAA... user@host\"]`\n\n```terraform\n # You can also add additional SSH public Keys which are saved in the hetzner cloud by a label.\n # See https://docs.hetzner.com/cloud/#label-selector\n # ssh_hcloud_key_label = \"role=admin\"\n```\n\n* **`ssh_hcloud_key_label` (Optional):**\n * **Purpose:** Instead of providing raw public key content, you can specify a label. The module will then find SSH keys already uploaded to your Hetzner Cloud project that match this label and add them to the nodes.\n * **Hetzner Cloud Feature:** This leverages Hetzner's ability to store and label SSH keys.\n * **Use Case:** Managing a central repository of SSH keys in Hetzner Cloud and assigning them to servers based on roles or teams.\n * **Format:** A string representing the label selector (e.g., `\"team=devops\"`, `\"environment=production,role=admin\"`).\n\n```terraform\n # If you use SSH agent and have issues with SSH connecting to your nodes, you can increase the number of auth tries (default is 2)\n # ssh_max_auth_tries = 10\n```\n\n* **`ssh_max_auth_tries` (Optional):**\n * **Default:** `2` (or a small number set by the underlying SSH client/library).\n * **Purpose:** Controls the `MaxAuthTries` setting for SSH connections made by Terraform/module scripts.\n * **Use Case:** If you have many keys loaded in your SSH agent, the server might close the connection before the correct key is tried. Increasing this value gives the SSH client more attempts to offer different keys.\n * **Caution:** Setting this too high could theoretically make brute-force attacks slightly easier if other security measures are weak, but the primary defense is strong key management.\n\n```terraform\n # If you want to use an ssh key that is already registered within hetzner cloud, you can pass its id.\n # If no id is passed, a new ssh key will be registered within hetzner cloud.\n # It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` variables.\n # hcloud_ssh_key_id = \"\"\n```\n\n* **`hcloud_ssh_key_id` (Optional):**\n * **Purpose:** Allows you to use an SSH key that is *already registered* in your Hetzner Cloud project by specifying its unique ID.\n * **Behavior:**\n * **If ID provided:** The module will associate this existing Hetzner SSH key resource with the created servers. It will *not* create a new SSH key resource in Hetzner Cloud based on `ssh_public_key`.\n * **If ID not provided (or empty string):** The module will create a *new* SSH key resource in Hetzner Cloud using the content from `ssh_public_key` and associate that new key with the servers.\n * **Crucial Constraint:** The comment \"It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` variables\" is vital. Even if using an existing Hetzner key ID, the module might still need the raw public key content for other purposes (e.g., configuring `authorized_keys` directly if Hetzner's association method isn't solely relied upon, or for consistency). The private key is needed if SSH connections are made by provisioners. This ensures the keys Terraform *thinks* it's using match the key Hetzner *knows* about.\n\n```terraform\n # These can be customized, or left with the default values\n # * For Hetzner locations see https://docs.hetzner.com/general/others/data-centers-and-connection/\n network_region = \"eu-central\" # change to `us-east` if location is ash\n```\n\n* **`network_region` (Obligatory, though has a default in the module):**\n * **Default (in module, not shown here):** Likely \"eu-central\".\n * **Purpose:** Specifies the broad geographical region for your Hetzner Cloud private network. All servers and load balancers within the same private network must reside in locations that belong to this network region.\n * **Hetzner Regions:**\n * `eu-central`: Encompasses European locations like Falkenstein (`fsn1` - currently unavailable due to high demand), Nuremberg (`nbg1`), Helsinki (`hel1`).\n * `us-east`: Encompasses Ashburn, VA (`ash`).\n * `us-west`: Encompasses Hillsboro, OR (`hil`). (Check if supported by module if you intend to use it)\n * **Constraint:** The `location` specified in your `control_plane_nodepools` and `agent_nodepools` *must* be compatible with this `network_region`. You cannot have a server in `fsn1` (Europe) in a network defined for `us-east`.\n\n```terraform\n # If you want to create the private network before calling this module,\n # you can do so and pass its id here. For example if you want to use a proxy\n # which only listens on your private network. Advanced use case.\n #\n # NOTE1: make sure to adapt network_ipv4_cidr, cluster_ipv4_cidr, and service_ipv4_cidr accordingly.\n # If your network is created with 10.0.0.0/8, and you use subnet 10.128.0.0/9 for your\n # non-k3s business, then adapting `network_ipv4_cidr = \"10.0.0.0/9\"` should be all you need.\n #\n # NOTE2: square brackets! This must be a list of length 1.\n #\n # existing_network_id = [hcloud_network.your_network.id]\n```\n\n* **`existing_network_id` (Optional, Advanced):**\n * **Default:** Not set, meaning the module will create and manage its own Hetzner Cloud private network.\n * **Purpose:** Allows you to use a pre-existing Hetzner Cloud private network for your Kubernetes cluster.\n * **Format:** A list containing a single element: the ID of the existing Hetzner network (e.g., `[1234567]`). The comment `[hcloud_network.your_network.id]` shows how you'd reference a network created in the same Terraform configuration but outside this module.\n * **Use Case:** Integrating the Kubernetes cluster into a larger, existing infrastructure on Hetzner Cloud where other services already reside on a specific private network.\n * **Critical Considerations (NOTE1):** If you use an existing network, you are responsible for ensuring that the IP address ranges used by this module (`network_ipv4_cidr`, `cluster_ipv4_cidr`, `service_ipv4_cidr`) do not conflict with other subnets or IP ranges already in use on that existing network. You might need to adjust these CIDR parameters in the module configuration to fit within an available portion of your existing network's IP space. The example given (using `10.0.0.0/9` for k3s within a larger `10.0.0.0/8` network) illustrates this.\n\n```terraform\n # If you must change the network CIDR you can do so below, but it is highly advised against.\n # network_ipv4_cidr = \"10.0.0.0/8\"\n```\n\n* **`network_ipv4_cidr` (Optional, Advanced):**\n * **Default (in module):** Typically `10.0.0.0/8`.\n * **Purpose:** Defines the overall IP address range for the Hetzner Cloud private network that the module will create (if `existing_network_id` is not used). All other internal Kubernetes CIDRs (for pods, services, and node subnets) will be carved out of this range.\n * **Warning:** \"highly advised against\" changing this unless you have a very specific reason (e.g., conflict with on-premises networks if using VPN/interconnect, or needing a smaller/different range for a very specific setup). Changing it requires careful planning of all sub-CIDRs.\n * **Impact:** If changed, `cluster_ipv4_cidr` and `service_ipv4_cidr` must be sub-ranges within this new `network_ipv4_cidr`.\n\n```terraform\n # The amount of subnets into which the network will be split. Must be a power of 2.\n # subnet_amount = 256\n```\n\n* **`subnet_amount` (Number, Optional):**\n * **Default:** `256`.\n * **Purpose:** Determines into how many subnets the `network_ipv4_cidr` is divided.\n * **Constraint:** Must be a power of 2. Each nodepool (control plane and agent) and potentially the NAT router takes one subnet. Ensure this is large enough for your planned number of nodepools.\n\n```terraform\n # Using the default configuration you can only create a maximum of 42 agent-nodepools.\n # This is due to the creation of a subnet for each nodepool with CIDRs being in the shape of 10.[nodepool-index].0.0/16 which collides with k3s' cluster and service IP ranges (defaults below).\n # Furthermore the maximum number of nodepools (controlplane and agent) is 50, due to a hard limit of 50 subnets per network, see https://docs.hetzner.com/cloud/networks/faq/.\n # So to be able to create a maximum of 50 nodepools in total, the values below have to be changed to something outside that range, e.g. `10.200.0.0/16` and `10.201.0.0/16` for cluster and service respectively.\n```\n\n* **Explanation of Nodepool Subnet Allocation and Limits:**\n * **Subnet per Nodepool:** The module creates a dedicated subnet within the Hetzner private network for each nodepool (both control plane and agent). This provides network isolation at the Hetzner level and allows for distinct IP ranges per nodepool.\n * **Default Subnetting Scheme:** The module uses a scheme like `10.[nodepool-index].0.0/16` for these subnets. For example, the first nodepool might get `10.1.0.0/16`, the second `10.2.0.0/16`, and so on.\n * **Collision Issue:** The default k3s cluster CIDR (`10.42.0.0/16`) and service CIDR (`10.43.0.0/16`) would collide if a nodepool index reached 42 or 43 using this scheme. This limits the number of *agent* nodepools to 42 if defaults are kept.\n * **Hetzner Subnet Limit:** Hetzner Cloud has a hard limit of 50 subnets per private network. This is the ultimate cap on the total number of nodepools (control plane + agent).\n * **Solution for >42 Nodepools:** To exceed 42 nodepools (up to the 50 limit), you *must* change `cluster_ipv4_cidr` and `service_ipv4_cidr` to ranges that won't collide with the `10.[0-49].0.0/16` nodepool subnet ranges. The example `10.200.0.0/16` and `10.201.0.0/16` achieves this.\n\n```terraform\n # If you must change the cluster CIDR you can do so below, but it is highly advised against.\n # Never change this value after you already initialized a cluster. Complete cluster redeploy needed!\n # The cluster CIDR must be a part of the network CIDR!\n # cluster_ipv4_cidr = \"10.42.0.0/16\"\n```\n\n* **`cluster_ipv4_cidr` (Optional, Advanced):**\n * **Default (in module):** `10.42.0.0/16` (a common default for k3s/Kubernetes).\n * **Purpose:** This is the IP address range from which Kubernetes assigns IP addresses to Pods running within the cluster.\n * **Critical Warning:** \"Never change this value after you already initialized a cluster.\" Doing so would require a complete cluster redeployment because all existing Pods and network configurations would become invalid.\n * **Constraint:** Must be a sub-range of `network_ipv4_cidr`.\n * **Interdependency:** As explained above, may need to be changed if you require more than 42 nodepools.\n\n```terraform\n # If you must change the service CIDR you can do so below, but it is highly advised against.\n # Never change this value after you already initialized a cluster. Complete cluster redeploy needed!\n # The service CIDR must be a part of the network CIDR!\n # service_ipv4_cidr = \"10.43.0.0/16\"\n```\n\n* **`service_ipv4_cidr` (Optional, Advanced):**\n * **Default (in module):** `10.43.0.0/16` (a common default for k3s/Kubernetes).\n * **Purpose:** This is the IP address range from which Kubernetes assigns virtual IP addresses to Services (e.g., ClusterIP services).\n * **Critical Warning:** Same as `cluster_ipv4_cidr` – do not change post-initialization without a full redeploy.\n * **Constraint:** Must be a sub-range of `network_ipv4_cidr`.\n * **Interdependency:** May need to be changed if you require more than 42 nodepools.\n\n```terraform\n # If you must change the service IPv4 address of core-dns you can do so below, but it is highly advised against.\n # Never change this value after you already initialized a cluster. Complete cluster redeploy needed!\n # The service IPv4 address must be part of the service CIDR!\n # cluster_dns_ipv4 = \"10.43.0.10\"\n```\n\n* **`cluster_dns_ipv4` (Optional, Advanced):**\n * **Default (in module):** `10.43.0.10`.\n * **Purpose:** Specifies the static IP address for the CoreDNS service (or KubeDNS) within the cluster. Pods use this IP to resolve internal and external domain names.\n * **Critical Warning:** Same as above – do not change post-initialization.\n * **Constraint:** This IP address *must* fall within the `service_ipv4_cidr` range. Typically, it's one of the first few usable IPs in that range (e.g., `.10`).\n\nThe subsequent sections on `control_plane_nodepools` and `agent_nodepools` are extensive. I will break them down carefully.\n\n---\n\n**Section 2.1: `control_plane_nodepools` - The Brains of the Operation**\n\n```terraform\n # For the control planes, at least three nodes are the minimum for HA. Otherwise, you need to turn off the automatic upgrades (see README).\n # **It must always be an ODD number, never even!** Search the internet for \"split-brain problem with etcd\" or see https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/\n # For instance, one is ok (non-HA), two is not ok, and three is ok (becomes HA). It does not matter if they are in the same nodepool or not! So they can be in different locations and of various types.\n\n # Of course, you can choose any number of nodepools you want, with the location you want. The only constraint on the location is that you need to stay in the same network region, Europe, or the US.\n # For the server type, the minimum instance supported is cx23. If you want to use arm64 use cax11; see https://www.hetzner.com/cloud.\n\n # IMPORTANT: Before you create your cluster, you can do anything you want with the nodepools, but you need at least one of each, control plane and agent.\n # Once the cluster is up and running, you can change nodepool count and even set it to 0 (in the case of the first control-plane nodepool, the minimum is 1).\n # You can also rename it (if the count is 0), but do not remove a nodepool from the list.\n\n # You can safely add or remove nodepools at the end of each list. That is due to how subnets and IPs get allocated (FILO).\n # The maximum number of nodepools you can create combined for both lists is 50 (see above).\n # Also, before decreasing the count of any nodepools to 0, it's essential to drain and cordon the nodes in question. Otherwise, it will leave your cluster in a bad state.\n\n # Before initializing the cluster, you can change all parameters and add or remove any nodepools. You need at least one nodepool of each kind, control plane, and agent.\n # ⚠️ The nodepool names are entirely arbitrary, but all lowercase, no special characters or underscore (dashes are allowed), and they must be unique.\n\n # If you want to have a single node cluster, have one control plane nodepools with a count of 1, and one agent nodepool with a count of 0.\n\n # Please note that changing labels and taints after the first run will have no effect. If needed, you can do that through Kubernetes directly.\n\n # Multi-architecture clusters are OK for most use cases, as container underlying images tend to be multi-architecture too.\n\n # * Example below:\n\n control_plane_nodepools = [\n {\n name = \"control-plane-nbg1\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n # swap_size = \"2G\" # remember to add the suffix, examples: 512M, 1G\n # zram_size = \"2G\" # remember to add the suffix, examples: 512M, 1G\n # kubelet_args = [\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]\n\n # Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):\n # placement_group = \"default\"\n\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n\n # To disable public ips (default: false)\n # WARNING: If both values are set to \"true\", your server will only be accessible via a private network. Make sure you have followed\n # the instructions regarding this type of setup in README.md: \"Use only private IPs in your cluster\".\n # disable_ipv4 = true\n # disable_ipv6 = true\n },\n // ... more control plane nodepool examples ...\n ]\n```\n\n* **`control_plane_nodepools` (Obligatory, list of maps):**\n * **Purpose:** Defines one or more groups of control plane nodes. Control plane nodes run the Kubernetes master components (API server, scheduler, controller manager) and, in k3s with an embedded database, `etcd` (or the k3s default, SQLite for single-node, or embedded etcd for HA).\n * **Structure:** A list of maps, where each map defines a distinct nodepool.\n * **High Availability (HA) Critical Logic:**\n * **Minimum for HA:** 3 control plane nodes.\n * **Odd Number:** Always use an odd number (1, 3, 5, etc.) for etcd quorum to prevent split-brain scenarios. 2 nodes are worse than 1 for HA.\n * **Impact of Non-HA (1 control plane):** If you have only one control plane node, features like `automatically_upgrade_os` and `automatically_upgrade_k3s` (if not managed carefully) can lead to downtime. The module's README and comments often advise disabling automatic upgrades for single control plane setups.\n * **Distribution:** HA control plane nodes can be in the same nodepool definition (e.g., `count = 3` in one map) or spread across multiple nodepool definitions (e.g., three maps, each with `count = 1`, potentially in different `location`s for better fault tolerance).\n * **Minimum Requirements (Initial Cluster Create):**\n * At least one control plane nodepool with `count >= 1`.\n * (Typically) At least one agent nodepool with `count >= 1`, *unless* you are creating a single-node cluster where the control plane also acts as a worker (see below).\n * **Lifecycle of Nodepools:**\n * **Adding/Removing Nodepools:** You can safely add new nodepool definitions to the *end* of the list or remove nodepool definitions from the *end* of the list. This is due to how the module allocates subnets (FILO - First In, Last Out, or rather, sequentially). Modifying nodepools in the middle of the list can cause existing nodepools to be re-evaluated for their subnet, potentially leading to disruption.\n * **Changing `count`:**\n * **Increasing:** Generally safe. New nodes will be provisioned.\n * **Decreasing (to > 0):** Terraform will select nodes to remove. Ensure workloads are drained from these nodes (`kubectl drain`) before applying, to prevent data loss or service interruption.\n * **Decreasing to `0`:** The nodepool becomes effectively dormant. Its subnet remains. Before doing this, *all nodes in that pool must be drained and cordoned*.\n * **Renaming:** A nodepool can be renamed *only if its `count` is 0*. Otherwise, Terraform will see it as destroying the old and creating a new one.\n * **Removing from List:** Do not remove a nodepool definition from the list if it still has active nodes or if you intend to use it again. Set its `count` to 0 first.\n * **Single-Node Cluster:**\n * One control plane nodepool with `count = 1`.\n * One (or more) agent nodepools with `count = 0`.\n * The module typically automatically allows scheduling on the control plane in this scenario (or you'd set `allow_scheduling_on_control_plane = true`).\n * **Multi-Architecture:** Mixing x86 (`cx` series) and ARM (`cax` series) nodes is generally fine. Kubernetes and container runtimes handle this, and many container images are multi-arch.\n * **Nodepool Attributes (per map):**\n * **`name` (String, Obligatory):**\n * A unique, arbitrary name for this nodepool.\n * Constraints: Lowercase, no special characters except dashes (`-`).\n * Used for naming resources in Hetzner and for Kubernetes node labels/names.\n * **`server_type` (String, Obligatory):**\n * Hetzner server type:\n * x86: e.g. `cx23` (2 vCPU, 4GB RAM, 40GB SSD), `cx33` (4 vCPU, 8GB RAM, 80GB SSD), `cx43` (8 vCPU, 16GB RAM, 160GB SSD).\n * ARM: e.g. `cax11` (2 vCPU, 4GB RAM, 40GB SSD), `cax21` (4 vCPU, 8GB RAM, 80GB SSD).\n * Minimum for control plane: `cx23` is often cited. More demanding setups (e.g., with Cilium CNI, Rancher) might require more RAM (e.g., `cx33`/`cx43` or `cax21`/`cax31`).\n * **`location` (String, Obligatory):**\n * Hetzner location (e.g., `fsn1`, `nbg1`, `hel1`, `ash`).\n * Must be within the `network_region` defined earlier.\n * For HA, distributing control plane nodes across different locations (within the same region) improves fault tolerance against a single location outage.\n * **`labels` (List of Strings, Optional):**\n * Default: `[]`.\n * Kubernetes labels to apply to nodes in this pool. Format: `[\"key1=value1\", \"key2=value2\"]`.\n * **Lifecycle Note:** \"changing labels and taints after the first run will have no effect.\" The module likely applies these only at node creation. Subsequent changes must be done via `kubectl label node ...`.\n * **`taints` (List of Strings, Optional):**\n * Default: `[]`.\n * Kubernetes taints to apply to nodes in this pool. Format: `[\"key=value:Effect\"]` (e.g., `\"dedicated=control-plane:NoSchedule\"`).\n * Taints prevent pods from being scheduled on these nodes unless the pods have a corresponding toleration. Control planes often have taints to prevent regular workloads from running on them.\n * **Lifecycle Note:** Same as labels – apply via `kubectl taint node ...` after initial creation if changes are needed.\n * **`count` (Number, Obligatory):**\n * Number of server instances to create in this specific nodepool.\n * **`swap_size` (String, Optional):**\n * Examples: `\"512M\"`, `\"2G\"`, `\"4G\"`.\n * Configures a swap file of the specified size on the nodes.\n * **K3s/Kubernetes Consideration:** Kubernetes traditionally doesn't work well with swap. However, recent versions of k3s/kubelet can support it if the `NodeSwap` feature gate is enabled and kubelet is configured correctly. The comment `Make sure you set \"feature-gates=NodeSwap=true\" if want to use swap_size` (seen later under `k3s_global_kubelet_args`) is relevant here. Use with caution and understanding of its implications on performance and scheduling.\n * **`zram_size` (String, Optional):**\n * Examples: `\"512M\"`, `\"1G\"`.\n * Configures zRAM (compressed RAM block device, often used for swap) on the nodes.\n * Can be an alternative or supplement to traditional disk-based swap, offering faster swap at the cost of CPU for compression/decompression.\n * **`kubelet_args` (List of Strings, Optional):**\n * Allows passing additional arguments directly to the `kubelet` process running on nodes in this specific pool.\n * Example: `[\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]`\n * This is for fine-grained resource reservation for Kubernetes system components (`kube-reserved`) and OS system components (`system-reserved`), ensuring they have enough resources and don't get starved by user pods.\n * **Note:** There are also global `k3s_global_kubelet_args`, `k3s_control_plane_kubelet_args`, etc., defined later. These nodepool-specific `kubelet_args` likely supplement or override those for this pool.\n * **`placement_group` (String, Optional):**\n * Default: The module might create a default placement group or assign nodes to one.\n * Purpose: Hetzner Placement Groups ensure that servers within the same group are located on different physical host systems (spread strategy). This improves fault tolerance against hardware failures on a single host.\n * Value: Name of the placement group. If you specify the same name for multiple nodes/nodepools, they'll try to be in that group.\n * Limit: Hetzner placement groups have limits (e.g., 10 servers per spread placement group). The module might manage creating multiple groups if a nodepool `count` exceeds this.\n * **`backups` (Boolean, Optional):**\n * Default: `false`.\n * If `true`, enables Hetzner's automated server backup service for nodes in this pool. This incurs additional cost per server.\n * **`disable_ipv4` (Boolean, Optional) / `disable_ipv6` (Boolean, Optional):**\n * Default: `false` for both.\n * If `true`, disables the public IPv4 or IPv6 interface on the server, respectively.\n * **Warning:** If both are `true`, the server will *only* have a private IP address and will only be accessible via the Hetzner private network (e.g., from another server in the same network, or via a VPN/bastion host connected to that network). This is an advanced setup requiring careful network planning. The comment refers to a `README.md` section \"Use only private IPs in your cluster\" for guidance.\n\nThe example shows three control plane nodepools, each with one node, in different locations (`fsn1`, `nbg1`, `hel1`). This is a common pattern for a 3-node HA control plane, maximizing fault tolerance across Hetzner locations (within the same `network_region`).\n\n---\n\n**Section 2.2: `agent_nodepools` - The Workhorses**\n\n```terraform\n agent_nodepools = [\n {\n name = \"agent-small\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n # swap_size = \"2G\"\n # zram_size = \"2G\"\n # kubelet_args = [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]\n # placement_group = \"default\"\n # backups = true\n },\n {\n name = \"agent-large\",\n server_type = \"cx33\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n # placement_group = \"default\"\n # backups = true\n },\n {\n name = \"storage\",\n server_type = \"cx33\",\n location = \"nbg1\",\n labels = [\n \"node.kubernetes.io/server-usage=storage\" # Example label\n ],\n taints = [], # Could add taints to only allow storage workloads\n count = 1\n # In the case of using Longhorn, you can use Hetzner volumes instead of using the node's own storage by specifying a value from 10 to 10240 (in GB)\n # It will create one volume per node in the nodepool, and configure Longhorn to use them.\n # Something worth noting is that Volume storage is slower than node storage, which is achieved by not mentioning longhorn_volume_size or setting it to 0.\n # So for something like DBs, you definitely want node storage, for other things like backups, volume storage is fine, and cheaper.\n # longhorn_volume_size = 20 # In GB\n # backups = true\n },\n # Egress nodepool useful to route egress traffic using Hetzner Floating IPs\n # used with Cilium's Egress Gateway feature\n {\n name = \"egress\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [\n \"node.kubernetes.io/role=egress\"\n ],\n taints = [\n \"node.kubernetes.io/role=egress:NoSchedule\" # Ensures only egress gateway pods run here\n ],\n floating_ip = true # Special attribute for this module\n # Optionally associate a reverse DNS entry with the floating IP(s).\n # floating_ip_rns = \"my.domain.com\"\n count = 1\n },\n # Arm based nodes\n {\n name = \"agent-arm-small\",\n server_type = \"cax11\", # ARM server type\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n },\n # For fine-grained control over the nodes in a node pool, replace the count variable with a nodes map.\n # In this case, the node-pool variables are defaults which can be overridden on a per-node basis.\n # Each key in the nodes map refers to a single node and must be an integer string (\"1\", \"123\", ...).\n {\n name = \"agent-arm-medium\",\n server_type = \"cax21\", # Default server_type for this pool\n location = \"nbg1\", # Default location\n labels = [],\n taints = [],\n nodes = { # Overrides 'count' and allows per-node customization\n \"1\" : { # Node identified as \"1\" within this pool\n location = \"fsn1\" # Override location for this specific node\n labels = [\n \"testing-labels=a1\",\n ]\n },\n \"20\" : { # Node identified as \"20\"\n labels = [\n \"testing-labels=b1\",\n ]\n # server_type could also be overridden here if needed\n }\n }\n },\n ]\n```\n\n* **`agent_nodepools` (Obligatory, list of maps):**\n * **Purpose:** Defines groups of agent (worker) nodes. These nodes run your actual application Pods.\n * **Structure and Lifecycle:** Similar to `control_plane_nodepools` (list of maps, rules for adding/removing/renaming apply).\n * **Minimum Requirement (Initial Cluster Create):** Typically, at least one agent nodepool with `count >= 1` is needed, unless it's a single-node cluster where the control plane also acts as a worker (in which case, agent nodepool counts can be 0).\n * **Nodepool Attributes (per map):** Most attributes are the same as for `control_plane_nodepools` (`name`, `server_type`, `location`, `labels`, `taints`, `count`, `swap_size`, `zram_size`, `kubelet_args`, `placement_group`, `backups`, `disable_ipv4`/`ipv6`).\n * **Specific Agent Nodepool Attributes/Examples:**\n * **`longhorn_volume_size` (Number, Optional, specific to agent nodepools if Longhorn is enabled):**\n * If `enable_longhorn = true` (a global module setting), this attribute can be added to an agent nodepool definition.\n * **Purpose:** Instructs the module to create a Hetzner Cloud Volume of the specified size (in GB, e.g., `20` for 20GB) for *each node* in this pool. Longhorn will then be configured to use these dedicated Hetzner Volumes for its storage replicas instead of using the node's local disk.\n * **Trade-offs:**\n * **Hetzner Volumes:** Network-attached, potentially slower than local NVMe/SSD storage on the node, but can be larger, are independently manageable, and might be cheaper for bulk storage. Good for less I/O-intensive workloads or where data persistence independent of the node's lifecycle is paramount.\n * **Node Local Storage (if `longhorn_volume_size` is not set or 0):** Longhorn uses a directory on the node's filesystem. Faster I/O, but storage is tied to the node's disk.\n * **Recommendation:** The comment wisely suggests local storage for databases (high I/O) and Hetzner Volumes for backups or less critical storage.\n * **`floating_ip` (Boolean, Optional, specific to egress nodepool example):**\n * Default: `false`.\n * If `true`, the module will provision a Hetzner Floating IP and associate it with the node(s) in this pool. If `count > 1`, how the floating IP is managed across multiple nodes needs clarification from module docs (e.g., active/passive, or one FIP per node).\n * **Use Case (Egress Gateway):** As shown in the \"egress\" nodepool example, this is used with Cilium's Egress Gateway feature. This allows you to have a stable, predictable public IP address for outbound traffic originating from your cluster, which can be useful for whitelisting with external services.\n * The `labels` and `taints` on the \"egress\" pool ensure that only specific egress gateway pods (which would have tolerations for the taint) are scheduled there.\n * **`floating_ip_rns` (String, Optional):**\n * If `floating_ip = true`, this allows you to set a reverse DNS (PTR record) for the provisioned floating IP.\n * Use Case: Email servers or services where reverse DNS is important for reputation.\n * **`nodes` (Map of Maps, Optional, replaces `count`):**\n * **Purpose:** Provides fine-grained control over individual nodes within a single nodepool definition, overriding the nodepool-level defaults for `location`, `labels`, `taints`, `server_type`, etc., on a per-node basis.\n * **Structure:**\n * The top-level nodepool definition provides the defaults.\n * The `nodes` map's keys are arbitrary string identifiers for each node (e.g., `\"1\"`, `\"20\"`). These are not server IDs but logical identifiers within this Terraform definition.\n * Each value in the `nodes` map is another map specifying the attributes to override for that particular node.\n * **Example Breakdown (`agent-arm-medium`):**\n * Default `server_type`: `cax21`\n * Default `location`: `nbg1`\n * Node `\"1\"`: Overrides `location` to `fsn1` and adds specific `labels`. It will use the default `cax21` server type.\n * Node `\"20\"`: Uses default `location` (`nbg1`) and `server_type` (`cax21`) but has its own specific `labels`.\n * **Benefit:** Useful when you need slight variations for a few nodes within a larger, mostly homogeneous pool, without creating many separate small nodepool definitions.\n\n---\n\n**Section 2.3: Custom K3s Configuration Arguments**\n\n```terraform\n # Add additional configuration options for control planes here.\n # E.g to enable monitoring for etcd, proxy etc:\n # control_planes_custom_config = {\n # etcd-expose-metrics = true,\n # kube-controller-manager-arg = \"bind-address=0.0.0.0\",\n # kube-proxy-arg =\"metrics-bind-address=0.0.0.0\",\n # kube-scheduler-arg = \"bind-address=0.0.0.0\",\n # }\n\n # Add additional configuration options for agent nodes and autoscaler nodes here.\n # E.g to enable monitoring for proxy:\n # agent_nodes_custom_config = {\n # kube-proxy-arg =\"metrics-bind-address=0.0.0.0\",\n # }\n```\n\n* **`control_planes_custom_config` (Map, Optional):**\n * **Purpose:** Allows passing custom configuration parameters that will be translated into the k3s server configuration file (typically `/etc/rancher/k3s/config.yaml` or passed as CLI args to the k3s server process) specifically for control plane nodes.\n * **Format:** A map where keys are k3s configuration options (often matching CLI flags without the leading `--` or `config.yaml` keys) and values are their settings.\n * **Example Usage:**\n * `etcd-expose-metrics = true`: Enables Prometheus metrics endpoint for the embedded etcd.\n * `kube-controller-manager-arg = \"bind-address=0.0.0.0\"`: Makes the controller manager's metrics/health endpoint listen on all interfaces (use with caution, consider firewall implications). Similar for `kube-proxy-arg` and `kube-scheduler-arg`.\n * **Reference:** Consult the [k3s server configuration options documentation](https://rancher.com/docs/k3s/latest/en/installation/configuration/#configuration-file) for available keys.\n* **`agent_nodes_custom_config` (Map, Optional):**\n * **Purpose:** Similar to `control_planes_custom_config`, but applies to k3s agent configuration on agent nodes and nodes created by the cluster autoscaler.\n * **Example Usage:** `kube-proxy-arg =\"metrics-bind-address=0.0.0.0\"` enables kube-proxy metrics on agent nodes.\n * **Reference:** Consult the [k3s agent configuration options documentation](https://rancher.com/docs/k3s/latest/en/installation/configuration/#agent-configuration-file).\n\n---\n\n**Section 2.4: Network Security & CNI Options**\n\n```terraform\n # You can enable encrypted wireguard for the CNI by setting this to \"true\". Default is \"false\".\n # FYI, Hetzner says \"Traffic between cloud servers inside a Network is private and isolated, but not automatically encrypted.\"\n # Source: https://docs.hetzner.com/cloud/networks/faq/#is-traffic-inside-hetzner-cloud-networks-encrypted\n # It works with all CNIs that we support.\n # Just note, that if Cilium with cilium_values, the responsibility of enabling of disabling Wireguard falls on you.\n # enable_wireguard = true\n```\n\n* **`enable_wireguard` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** Enables WireGuard encryption for inter-node CNI (Container Network Interface) traffic. This encrypts pod-to-pod communication that traverses different nodes.\n * **Context:** Hetzner private networks provide isolation but not encryption-in-transit by default. Enabling WireGuard adds this layer of security.\n * **CNI Compatibility:** The comment states it works with supported CNIs (Flannel, Calico, Cilium).\n * **Cilium Specifics:** If you are using `cni_plugin = \"cilium\"` and also providing custom `cilium_values`, you become responsible for enabling/configuring WireGuard within those `cilium_values` yourself, as your custom values would likely override the module's default Cilium WireGuard setup.\n * **Performance:** WireGuard is generally efficient, but encryption always has some performance overhead.\n\n```terraform\n # Override the flannel backend used by k3s. When set, this takes precedence over enable_wireguard.\n # Valid values: vxlan, host-gw, wireguard-native.\n # flannel_backend = \"vxlan\"\n```\n\n* **`flannel_backend` (String, Optional):**\n * **Default:** `null` (defaults to `vxlan`, or `wireguard-native` if `enable_wireguard = true`).\n * **Purpose:** Explicitly configures the backend for the Flannel CNI.\n * **Robot Node Context:** For clusters involving Hetzner Robot nodes connected via vSwitch, `wireguard-native` is recommended to avoid MTU issues often seen with VXLAN in that topology.\n\n**Section 2.5: Load Balancer Configuration**\n\n```terraform\n # * LB location and type, the latter will depend on how much load you want it to handle, see https://www.hetzner.com/cloud/load-balancer\n load_balancer_type = \"lb11\"\n load_balancer_location = \"nbg1\"\n\n # Disable IPv6 for the load balancer, the default is false.\n # load_balancer_disable_ipv6 = true\n\n # Disables the public network of the load balancer. (default: false).\n # load_balancer_disable_public_network = true\n\n # Specifies the algorithm type of the load balancer. (default: round_robin).\n # load_balancer_algorithm_type = \"least_connections\"\n\n # Specifies the interval at which a health check is performed. Minimum is 3s (default: 15s).\n # load_balancer_health_check_interval = \"5s\"\n\n # Specifies the timeout of a single health check. Must not be greater than the health check interval. Minimum is 1s (default: 10s).\n # load_balancer_health_check_timeout = \"3s\"\n\n # Specifies the number of times a health check is retried before a target is marked as unhealthy. (default: 3)\n # load_balancer_health_check_retries = 3\n```\n\n* **Purpose:** This section configures the Hetzner Cloud Load Balancer that will typically sit in front of your agent nodes to distribute incoming traffic to services exposed via an Ingress controller (like Traefik, Nginx) or services of type `LoadBalancer`.\n* **`load_balancer_type` (String, Obligatory):**\n * **Default (in module, if any, but usually required here):** `lb11` is shown as an example.\n * **Values:** Hetzner offers various LB types (e.g., `lb11`, `lb21`, `lb31`) with different capacities for connections, requests per second, and included traffic. Choose based on expected load. `lb11` is the smallest/cheapest.\n * **Reference:** The comment points to the Hetzner Cloud Load Balancer documentation.\n* **`load_balancer_location` (String, Obligatory):**\n * **Purpose:** Specifies the Hetzner location where the Load Balancer instance will be provisioned.\n * **Best Practice:** Choose a location where you have agent nodes to minimize latency between the LB and its backend targets. It must be within the same `network_region` as your nodes.\n* **`load_balancer_disable_ipv6` (Boolean, Optional):**\n * **Default:** `false` (meaning IPv6 is enabled on the LB).\n * **Purpose:** If `true`, the Load Balancer will not be assigned a public IPv6 address and will not listen for traffic on IPv6.\n* **`load_balancer_disable_public_network` (Boolean, Optional):**\n * **Default:** `false` (meaning the LB has a public interface).\n * **Purpose:** If `true`, the Load Balancer will only have a private IP address and will only be accessible from within the Hetzner private network.\n * **Use Case:** For internal load balancing scenarios where you don't want to expose the LB to the public internet directly. External access would then require a VPN, bastion, or another proxy fronting this internal LB.\n* **`load_balancer_algorithm_type` (String, Optional):**\n * **Default:** `\"round_robin\"`.\n * **Purpose:** Defines the algorithm used by the Load Balancer to distribute traffic to its backend targets (your agent nodes).\n * **Values:**\n * `\"round_robin\"`: Distributes connections sequentially to each target.\n * `\"least_connections\"`: Sends new connections to the target that currently has the fewest active connections.\n* **`load_balancer_health_check_interval` (String, Optional):**\n * **Default:** `\"15s\"`. Minimum: `\"3s\"`.\n * **Purpose:** How often the Load Balancer performs health checks on its backend targets.\n * **Format:** String with a time unit suffix (e.g., `\"5s\"`, `\"1m\"`).\n* **`load_balancer_health_check_timeout` (String, Optional):**\n * **Default:** `\"10s\"`. Minimum: `\"1s\"`.\n * **Purpose:** The maximum time the Load Balancer will wait for a response from a target during a health check before considering it a failure.\n * **Constraint:** Must not be greater than `load_balancer_health_check_interval`.\n* **`load_balancer_health_check_retries` (Number, Optional):**\n * **Default:** `3`.\n * **Purpose:** The number of consecutive health check failures required before a target is marked as unhealthy and removed from the load balancing pool.\n\n---\n\n**Section 2.6: Optional Cluster Enhancements & Identifiers**\n\n```terraform\n ### The following values are entirely optional (and can be removed from this if unused)\n\n # You can refine a base domain name to be use in this form of nodename.base_domain for setting the reverse dns inside Hetzner\n # base_domain = \"mycluster.example.com\"\n```\n\n* **`base_domain` (String, Optional):**\n * **Purpose:** If set, the module may attempt to configure reverse DNS (PTR records) for your nodes' public IP addresses using a pattern like `nodename.your_base_domain`. For example, if a node is named `agent-pool1-node1` and `base_domain` is `k8s.example.com`, its reverse DNS might be set to `agent-pool1-node1.k8s.example.com`.\n * **Requirement:** You must own/control the `base_domain` and have appropriate DNS setup for this to be meaningful and verifiable. Hetzner's ability to set PTR records might also depend on whether the IP is from a range they allow custom PTR for.\n * **Impact:** Primarily affects how your server IPs are identified in reverse DNS lookups, which can be relevant for email sending or some logging/auditing systems.\n\n---\n\n**Section 2.7: Cluster Autoscaler Configuration**\n\n```terraform\n # Cluster Autoscaler\n # Providing at least one map for the array enables the cluster autoscaler feature, default is disabled.\n # ⚠️ Based on how the autoscaler works with this project, you can only choose either x86 instances or ARM server types for ALL autoscaler nodepools.\n # If you are curious, it's ok to have a multi-architecture cluster, as most underlying container images are multi-architecture too.\n #\n # ⚠️ Setting labels and taints will only work on cluster-autoscaler images versions released after > 20 October 2023. Or images built from master after that date.\n #\n # * Example below:\n # autoscaler_nodepools = [\n # {\n # name = \"autoscaled-small\"\n # server_type = \"cx33\"\n # location = \"nbg1\"\n # min_nodes = 0\n # max_nodes = 5\n # labels = { # Note: This is a map, not a list of strings like other labels\n # \"node.kubernetes.io/role\": \"peak-workloads\"\n # }\n # taints = [ # List of maps for taints\n # {\n # key= \"node.kubernetes.io/role\"\n # value= \"peak-workloads\"\n # effect= \"NoExecute\" # or NoSchedule, PreferNoSchedule\n # }\n # ]\n # # kubelet_args = [\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]\n # # swap_size = \"2G\"\n # # zram_size = \"2G\"\n # }\n # ]\n #\n # To disable public ips on your autoscaled nodes, uncomment the following lines:\n # autoscaler_disable_ipv4 = true\n # autoscaler_disable_ipv6 = true\n```\n\n* **`autoscaler_nodepools` (List of Maps, Optional):**\n * **Default:** Not set (or empty list), meaning Cluster Autoscaler is disabled.\n * **Purpose:** Enables and configures the Kubernetes Cluster Autoscaler for Hetzner Cloud. The Cluster Autoscaler automatically adjusts the number of nodes in specified nodepools based on pod scheduling demands (e.g., pending pods that cannot be scheduled due to resource shortages) or underutilization.\n * **Enabling:** Simply defining at least one map in this list will trigger the deployment of the Cluster Autoscaler components in your cluster.\n * **Architecture Constraint (⚠️):** \"you can only choose either x86 instances or ARM server types for ALL autoscaler nodepools.\" This implies a limitation in how the module or the Hetzner cloud provider for Cluster Autoscaler handles mixed-architecture autoscaling groups. You must commit to one architecture (e.g., all `cx` series or all `cax` series) for the pools managed by the autoscaler.\n * **Labels/Taints Versioning (⚠️):** The ability to set `labels` and `taints` directly in the `autoscaler_nodepools` definition depends on using a sufficiently new version of the Cluster Autoscaler image.\n * **Nodepool Attributes (per map within `autoscaler_nodepools`):**\n * **`name` (String, Obligatory):** A unique name for this autoscaled nodepool.\n * **`server_type` (String, Obligatory):** The Hetzner server type for nodes created in this pool (e.g., `cx33`, `cax21`). Must adhere to the single-architecture constraint mentioned above.\n * **`location` (String, Obligatory):** Hetzner location for nodes in this pool.\n * **`min_nodes` (Number, Obligatory):** The minimum number of nodes this pool can scale down to. Can be `0`.\n * **`max_nodes` (Number, Obligatory):** The maximum number of nodes this pool can scale up to.\n * **`labels` (Map of Strings, Optional):**\n * Kubernetes labels to apply to nodes provisioned by the autoscaler in this pool.\n * **Format Difference:** Note that this `labels` attribute is a *map* (`key: value`), unlike the `labels` in `control_plane_nodepools` and `agent_nodepools` which are lists of strings (`[\"key=value\"]`). This is likely due to how the Cluster Autoscaler itself expects these definitions.\n * **`taints` (List of Maps, Optional):**\n * Kubernetes taints to apply to nodes provisioned by the autoscaler in this pool.\n * **Format:** Each element in the list is a map with `key`, `value`, and `effect` (e.g., `NoSchedule`, `NoExecute`, `PreferNoSchedule`).\n * **`kubelet_args` (List of Strings, Optional):** Same purpose as in other nodepools, for passing custom arguments to kubelet on autoscaled nodes.\n * **`swap_size` (String, Optional):**\n * Examples: `\"512M\"`, `\"2G\"`, `\"4G\"`.\n * Configures a swap file of the specified size on autoscaled nodes.\n * **K3s/Kubernetes Consideration:** Kubernetes traditionally doesn't work well with swap. However, recent versions of k3s/kubelet can support it if the `NodeSwap` feature gate is enabled. Make sure you set `\"feature-gates=NodeSwap=true\"` in `k3s_global_kubelet_args` or `k3s_autoscaler_kubelet_args`.\n * When set, nodes will automatically receive the `node.kubernetes.io/server-swap=enabled` label.\n * **`zram_size` (String, Optional):**\n * Examples: `\"512M\"`, `\"1G\"`.\n * Configures zRAM (compressed RAM block device used for swap) on autoscaled nodes.\n * Uses zstd compression algorithm for optimal performance.\n * When set, nodes will automatically receive the `node.kubernetes.io/server-swap=enabled` label.\n* **`autoscaler_disable_ipv4` / `autoscaler_disable_ipv6` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** If `true`, disables public IPv4/IPv6 on nodes created by the Cluster Autoscaler. Similar implications as for regular nodepools (private network only access if both are true).\n\n```terraform\n # ⚠️ Deprecated, will be removed after a new Cluster Autoscaler version has been released which support the new way of setting labels and taints. See above.\n # Add extra labels on nodes started by the Cluster Autoscaler\n # This argument is not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set\n # autoscaler_labels = [\n # \"node.kubernetes.io/role=peak-workloads\"\n # ]\n\n # Add extra taints on nodes started by the Cluster Autoscaler\n # This argument is not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set\n # autoscaler_taints = [\n # \"node.kubernetes.io/role=specific-workloads:NoExecute\"\n # ]\n```\n\n* **`autoscaler_labels` / `autoscaler_taints` (List of Strings, Optional, Deprecated):**\n * **Status:** Marked as deprecated. These were older ways to apply labels/taints globally to all nodes created by the Cluster Autoscaler.\n * **Superseded by:** The per-nodepool `labels` (map) and `taints` (list of maps) within the `autoscaler_nodepools` definition offer more granular control and are the preferred method with newer Cluster Autoscaler versions.\n * **Logic:** If `autoscaler_nodepools` is not defined (i.e., autoscaler is disabled), these deprecated variables have no effect.\n\n```terraform\n # Configuration of the Cluster Autoscaler binary\n #\n # These arguments and variables are not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set.\n #\n # Image and version of Kubernetes Cluster Autoscaler for Hetzner Cloud:\n # - cluster_autoscaler_image: Image of Kubernetes Cluster Autoscaler for Hetzner Cloud to be used.\n # The default is the official image from the Kubernetes project: registry.k8s.io/autoscaling/cluster-autoscaler\n # - cluster_autoscaler_version: Version of Kubernetes Cluster Autoscaler for Hetzner Cloud. Should be aligned with Kubernetes version.\n # Available versions for the official image can be found at https://explore.ggcr.dev/?repo=registry.k8s.io%2Fautoscaling%2Fcluster-autoscaler\n #\n # Logging related arguments are managed using separate variables:\n # - cluster_autoscaler_log_level: Controls the verbosity of logs (--v), the value is from 0 to 5, default is 4, for max debug info set it to 5.\n # - cluster_autoscaler_log_to_stderr: Determines whether to log to stderr (--logtostderr).\n # - cluster_autoscaler_stderr_threshold: Sets the threshold for logs that go to stderr (--stderrthreshold).\n #\n # Server/node creation timeout variable:\n # - cluster_autoscaler_server_creation_timeout: Sets the timeout (in minutes) until which a newly created server/node has to become available before giving up and destroying it (defaults to 15, unit is minutes)\n #\n # Example:\n #\n # cluster_autoscaler_image = \"registry.k8s.io/autoscaling/cluster-autoscaler\"\n # cluster_autoscaler_version = \"v1.30.3\"\n # cluster_autoscaler_log_level = 4\n # cluster_autoscaler_log_to_stderr = true\n # cluster_autoscaler_stderr_threshold = \"INFO\"\n # cluster_autoscaler_server_creation_timeout = 15\n```\n\n* **Cluster Autoscaler Binary Configuration (Conditional on `autoscaler_nodepools` being set):**\n * **`cluster_autoscaler_image` (String, Optional):**\n * **Default:** `registry.k8s.io/autoscaling/cluster-autoscaler` (the official Kubernetes project image).\n * **Purpose:** Allows specifying a custom container image for the Cluster Autoscaler deployment. Useful for air-gapped environments, private registries, or custom builds.\n * **`cluster_autoscaler_version` (String, Optional):**\n * **Default:** The module likely picks a recent, compatible version.\n * **Purpose:** Specifies the version tag for the `cluster_autoscaler_image`.\n * **Recommendation:** Should generally be aligned with your Kubernetes cluster version (i.e., `install_k3s_version`). Mismatches can lead to incompatibility. The link provided helps find available official versions.\n * **`cluster_autoscaler_log_level` (Number, Optional):**\n * **Default:** `4`.\n * **Purpose:** Controls the verbosity of the Cluster Autoscaler logs (passed as the `--v` flag). Higher numbers mean more detailed logs. `5` is typically for maximum debug output.\n * **`cluster_autoscaler_log_to_stderr` (Boolean, Optional):**\n * **Default:** Likely `true`.\n * **Purpose:** Corresponds to the `--logtostderr` flag. If `true`, logs go to standard error.\n * **`cluster_autoscaler_stderr_threshold` (String, Optional):**\n * **Default:** Likely `\"INFO\"` or `\"ERROR\"`.\n * **Purpose:** Corresponds to the `--stderrthreshold` flag. Sets the minimum severity level for logs that are written to stderr (e.g., \"INFO\", \"WARNING\", \"ERROR\").\n * **`cluster_autoscaler_server_creation_timeout` (Number, Optional):**\n * **Default:** `15` (minutes).\n * **Purpose:** The maximum time (in minutes) the Cluster Autoscaler will wait for a newly provisioned node to become ready and join the cluster. If the timeout is exceeded, the autoscaler may assume the node provisioning failed and attempt to delete it and try again.\n * **`cluster_autoscaler_replicas` (Number, Optional):**\n * **Default:** `1`.\n * **Purpose:** Sets the replica count for the Cluster Autoscaler deployment. Increase to >1 for high availability (leader election is supported).\n * **`cluster_autoscaler_resource_limits` (Boolean, Optional):**\n * **Default:** `true`.\n * **Purpose:** Whether to apply resource requests/limits to the autoscaler pod.\n * **`cluster_autoscaler_resource_values` (Map, Optional):**\n * **Purpose:** Customizes the specific CPU and memory requests/limits for the autoscaler pod.\n\n```terraform\n # Additional Cluster Autoscaler binary configuration\n #\n # cluster_autoscaler_extra_args can be used for additional arguments. The default is an empty array.\n #\n # Please note that following arguments are managed by terraform-hcloud-kube-hetzner or the variables above and should not be set manually:\n # - --v=${var.cluster_autoscaler_log_level}\n # - --logtostderr=${var.cluster_autoscaler_log_to_stderr}\n # - --stderrthreshold=${var.cluster_autoscaler_stderr_threshold}\n # - --cloud-provider=hetzner\n # - --nodes ... (this defines the min/max/name for each autoscaled nodepool)\n #\n # See the Cluster Autoscaler FAQ for the full list of arguments: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca\n #\n # Example:\n #\n # cluster_autoscaler_extra_args = [\n # \"--ignore-daemonsets-utilization=true\",\n # \"--enforce-node-group-min-size=true\",\n # ]\n```\n\n* **`cluster_autoscaler_extra_args` (List of Strings, Optional):**\n * **Default:** `[]` (empty list).\n * **Purpose:** Allows passing arbitrary additional command-line arguments to the Cluster Autoscaler binary.\n * **Usage:** For advanced tuning or enabling features not directly exposed by other variables in this module.\n * **Managed Arguments (Do Not Set Manually):** The comment lists arguments that the module *already manages* based on other variables (like log levels, cloud provider, node group definitions). You should not try to set these via `cluster_autoscaler_extra_args` as it could conflict with the module's logic.\n * **Reference:** The Cluster Autoscaler FAQ link is the definitive source for all available CLI arguments.\n * **Example Args:**\n * `\"--ignore-daemonsets-utilization=true\"`: Tells the autoscaler to ignore resource requests from DaemonSet pods when calculating node utilization for scale-down decisions. Useful if DaemonSets reserve significant resources but aren't always actively using them.\n * `\"--enforce-node-group-min-size=true\"`: Ensures the autoscaler respects the `min_nodes` setting even if there are no pending pods, preventing it from scaling below the minimum due to other conditions.\n\n---\n\n**Section 2.8: Resource Protection and Backup Options**\n\n```terraform\n # Enable delete protection on compatible resources to prevent accidental deletion from the Hetzner Cloud Console.\n # This does not protect deletion from Terraform itself.\n # enable_delete_protection = {\n # floating_ip = true\n # load_balancer = true\n # volume = true # Applies to volumes created for Longhorn via longhorn_volume_size\n # }\n```\n\n* **`enable_delete_protection` (Map of Booleans, Optional):**\n * **Purpose:** Enables Hetzner Cloud's \"delete protection\" feature on specific resource types created by this module.\n * **Mechanism:** When delete protection is enabled on a resource in Hetzner Cloud, it cannot be deleted directly from the Hetzner Cloud Console (UI or hcloud CLI) until the protection is first disabled.\n * **Terraform Interaction:** This protection does *not* prevent `terraform destroy` from deleting the resources. Terraform will typically first disable the protection and then delete the resource.\n * **Scope:**\n * `floating_ip = true`: Protects Hetzner Floating IPs (e.g., for egress nodepools).\n * `load_balancer = true`: Protects the Hetzner Load Balancer.\n * `volume = true`: Protects Hetzner Volumes (e.g., those created if `longhorn_volume_size` is used in an agent nodepool).\n * **Benefit:** Adds an extra safety layer against accidental manual deletions in the Hetzner console.\n\n```terraform\n # Enable etcd snapshot backups to S3 storage.\n # Just provide a map with the needed settings (according to your S3 storage provider) and backups to S3 will\n # be enabled (with the default settings for etcd snapshots).\n # Cloudflare's R2 offers 10GB, 10 million reads and 1 million writes per month for free.\n # For proper context, have a look at https://docs.k3s.io/datastore/backup-restore.\n # You also can use additional parameters from https://docs.k3s.io/cli/etcd-snapshot, such as `etc-s3-folder`\n # etcd_s3_backup = {\n # etcd-s3-endpoint = \"xxxx.r2.cloudflarestorage.com\"\n # etcd-s3-access-key = \"<access-key>\"\n # etcd-s3-secret-key = \"<secret-key>\"\n # etcd-s3-bucket = \"k3s-etcd-snapshots\"\n # etcd-s3-region = \"<your-s3-bucket-region|usually required for aws>\"\n # # etcd-s3-folder = \"my-cluster-backups\" # Optional: subfolder within the bucket\n # # etcd-snapshot-schedule-cron = \"0 */12 * * *\" # Optional: cron for snapshot frequency, default is every 12 hours\n # # etcd-snapshot-retention = 5 # Optional: number of snapshots to retain, default is 5\n # }\n```\n\n* **`etcd_s3_backup` (Map of Strings, Optional):**\n * **Purpose:** Configures k3s's built-in capability to automatically take snapshots of its etcd datastore (or internal SQLite database if not HA) and upload them to an S3-compatible object storage service. This is crucial for disaster recovery.\n * **Enabling:** Simply providing this map with the necessary S3 details enables the feature.\n * **k3s Feature:** Leverages k3s's `--etcd-s3-*` server arguments.\n * **Parameters (within the map):**\n * `etcd-s3-endpoint` (String, Obligatory if enabling): The S3 API endpoint URL of your storage provider (e.g., AWS S3, MinIO, Cloudflare R2).\n * `etcd-s3-access-key` (String, Obligatory if enabling): Your S3 access key ID.\n * `etcd-s3-secret-key` (String, Obligatory if enabling, Sensitive): Your S3 secret access key. **Store this securely, e.g., using Terraform Cloud variables, Vault, or environment variables if possible, rather than hardcoding.**\n * `etcd-s3-bucket` (String, Obligatory if enabling): The name of the S3 bucket where snapshots will be stored.\n * `etcd-s3-region` (String, Optional but often required): The S3 region for your bucket (e.g., `us-east-1` for AWS). Some S3 providers might not require this if the endpoint is region-specific.\n * `etcd-s3-folder` (String, Optional): A subfolder path within the bucket to store snapshots.\n * `etcd-snapshot-schedule-cron` (String, Optional): A cron expression defining how often snapshots are taken. Default in k3s is typically `0 */12 * * *` (every 12 hours at minute 0).\n * `etcd-snapshot-retention` (Number, Optional): The number of snapshots to retain in S3. Older ones are deleted. Default in k3s is typically `5`.\n * **Reference:** The k3s documentation links are essential for understanding all available etcd snapshot options.\n\n---\n\n**Section 2.9: Storage Integrations (CSI - Container Storage Interface)**\n\n```terraform\n # To enable Hetzner Storage Box support, you can enable csi-driver-smb, default is \"false\".\n # enable_csi_driver_smb = true\n # If you want to specify the version for csi-driver-smb, set it below - otherwise it'll use the latest version available.\n # See https://github.com/kubernetes-csi/csi-driver-smb/releases for the available versions.\n # csi_driver_smb_version = \"v1.16.0\"\n```\n\n* **`enable_csi_driver_smb` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** If `true`, deploys the [Kubernetes CSI driver for SMB](https://github.com/kubernetes-csi/csi-driver-smb). This allows your Kubernetes cluster to provision and use PersistentVolumes (PVs) backed by SMB/CIFS shares.\n * **Hetzner Storage Box Context:** Hetzner Storage Boxes can be accessed via SMB/CIFS, making this driver relevant if you want to use Storage Box as persistent storage for your Kubernetes workloads.\n * **Mechanism:** The module will likely deploy the CSI driver components (controller, node plugins) as pods within your cluster.\n* **`csi_driver_smb_version` (String, Optional):**\n * **Default:** The module likely picks the latest stable version of the driver.\n * **Purpose:** Allows you to pin the `csi-driver-smb` to a specific version. Useful for stability or if you need a particular feature/fix from a specific version. The GitHub releases link provides available versions.\n\n```terraform\n # To enable iscid without setting enable_longhorn = true, set enable_iscsid = true. You will need this if\n # you install your own version of longhorn outside of this module.\n # Default is false. If enable_longhorn=true, this variable is ignored and iscsid is enabled anyway.\n # enable_iscsid = true\n```\n\n* **`enable_iscsid` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** Ensures that the iSCSI daemon (`iscsid` or `open-iscsi`) and related tools are installed and running on your cluster nodes.\n * **Relevance:** iSCSI is a protocol used by some storage solutions (like Longhorn, and potentially others you might install manually) to connect to block storage devices over a network.\n * **Logic:**\n * If `enable_longhorn = true` (a global module setting for Longhorn), `iscsid` is automatically enabled by the module because Longhorn requires it. This `enable_iscsid` variable is then ignored.\n * If you are *not* using the module's Longhorn integration (`enable_longhorn = false`) but plan to install Longhorn (or another iSCSI-dependent storage solution) *manually*, you would set `enable_iscsid = true` here to ensure the necessary OS-level iSCSI support is present.\n\n```terraform\n # To use local storage on the nodes, you can enable Longhorn, default is \"false\".\n # See a full recap on how to configure agent nodepools for longhorn here https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/373#discussioncomment-3983159\n # Also see Longhorn best practices here https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b\n # enable_longhorn = true\n```\n\n* **`enable_longhorn` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** If `true`, deploys [Longhorn](https://longhorn.io/), a cloud-native distributed block storage system for Kubernetes. Longhorn creates replicated PersistentVolumes using the local disk space on your agent nodes (or dedicated Hetzner Volumes if `longhorn_volume_size` is configured on agent nodepools).\n * **Benefits:** Provides resilient, replicated storage for stateful applications, snapshotting, backups, etc.\n * **Impact:** Deploys Longhorn components (manager, engine, UI) as pods in your cluster. It will also typically set up a default StorageClass for Longhorn.\n * **Dependencies:** As mentioned, enabling Longhorn implicitly enables `iscsid`.\n * **Configuration:** Can be further customized via `longhorn_replica_count`, `longhorn_fstype`, `longhorn_values`, and `longhorn_merge_values`.\n\n```terraform\n # By default, longhorn is pulled from https://charts.longhorn.io.\n # If you need a version of longhorn which assures compatibility with rancher you can set this variable to https://charts.rancher.io.\n # longhorn_repository = \"https://charts.rancher.io\"\n```\n\n* **`longhorn_repository` (String, Optional):**\n * **Default:** `\"https://charts.longhorn.io\"` (the official Longhorn Helm chart repository).\n * **Purpose:** Specifies the Helm chart repository URL from which to install Longhorn.\n * **Rancher Compatibility:** Rancher sometimes bundles or recommends specific versions/sources of Longhorn charts for optimal compatibility with its management platform. If using Rancher, you might need to set this to `\"https://charts.rancher.io\"` or another Rancher-provided URL.\n\n```terraform\n # The namespace for longhorn deployment, default is \"longhorn-system\".\n # longhorn_namespace = \"longhorn-system\"\n```\n\n* **`longhorn_namespace` (String, Optional):**\n * **Default:** `\"longhorn-system\"`.\n * **Purpose:** Specifies the Kubernetes namespace into which Longhorn components will be deployed.\n\n```terraform\n # The file system type for Longhorn, if enabled (ext4 is the default, otherwise you can choose xfs).\n # longhorn_fstype = \"xfs\"\n```\n\n* **`longhorn_fstype` (String, Optional):**\n * **Default:** `\"ext4\"`.\n * **Purpose:** When Longhorn formats the underlying storage (either local disk paths or Hetzner Volumes) for its replicas, this setting determines the filesystem type it will use.\n * **Options:** `\"ext4\"` or `\"xfs\"`. Both are robust Linux filesystems. `xfs` is sometimes preferred for large volumes or specific workloads, but `ext4` is a solid default.\n\n```terraform\n # how many replica volumes should longhorn create (default is 3).\n # longhorn_replica_count = 1\n```\n\n* **`longhorn_replica_count` (Number, Optional):**\n * **Default:** `3`.\n * **Purpose:** Sets the default number of replicas Longhorn will create for each PersistentVolume. For a volume to be highly available, it needs replicas on different nodes.\n * **Considerations:**\n * `3` replicas: Tolerates failure of 2 nodes (or the storage on them) hosting replicas for a given volume, provided the replicas are on distinct nodes. Requires at least 3 agent nodes with Longhorn-enabled storage.\n * `1` replica: No data redundancy. If the node hosting the replica fails, the data is lost (unless restored from a backup). Suitable for development, testing, or data that can be easily regenerated. Requires at least 1 agent node.\n * **Cost/Performance:** More replicas mean more disk space used and potentially more network traffic for replication, but higher availability.\n\n```terraform\n # When you enable Longhorn, you can go with the default settings and just modify the above two variables OR you can add a longhorn_values variable\n # with all needed helm values, see towards the end of the file in the advanced section.\n # If that file is present, the system will use it during the deploy, if not it will use the default values with the two variable above that can be customized.\n # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.\n```\n\n* **Longhorn Customization Path:**\n * **Simple:** Use `enable_longhorn`, `longhorn_replica_count`, `longhorn_fstype`.\n * **Advanced (full override):** Provide a `longhorn_values` block (discussed later) with custom Helm values. This replaces the module defaults.\n * **Advanced (targeted override):** Use `longhorn_merge_values` to merge selected keys on top of defaults (or on top of `longhorn_values` if set). Prefer this for small changes such as image tag overrides.\n * **Post-Deploy:** Kubernetes `HelmChartConfig` Custom Resource (if k3s supports/deploys it) can be used to modify Helm release values after the initial deployment by Terraform.\n\n```terraform\n # Also, you can choose to use a Hetzner volume with Longhorn. By default, it will use the nodes own storage space, but if you add an attribute of\n # longhorn_volume_size (⚠️ not a variable, just a possible agent nodepool attribute) with a value between 10 and 10240 GB to your agent nodepool definition, it will create and use the volume in question.\n # See the agent nodepool section for an example of how to do that.\n```\n\n* **Reiteration of `longhorn_volume_size`:** This just re-emphasizes the agent nodepool attribute `longhorn_volume_size` for using Hetzner Volumes with Longhorn, as discussed in the `agent_nodepools` section.\n\n```terraform\n # To disable Hetzner CSI storage, you can set the following to \"true\", default is \"false\".\n # disable_hetzner_csi = true\n```\n\n* **`disable_hetzner_csi` (Boolean, Optional):**\n * **Default:** `false` (meaning the Hetzner CSI driver *is* deployed by default).\n * **Purpose:** The [Hetzner Cloud CSI driver](https://github.com/hetznercloud/csi-driver) allows Kubernetes to dynamically provision PersistentVolumes backed by Hetzner Cloud Volumes. It's the standard way to use Hetzner's native block storage with Kubernetes.\n * **If `true`:** The module will *not* deploy the Hetzner Cloud CSI driver.\n * **Use Case for Disabling:**\n * You plan to use *only* Longhorn (or another storage solution) and don't want Hetzner Volumes managed via CSI.\n * You want to install and manage a specific version or configuration of the Hetzner CSI driver manually, outside of this module.\n\n```terraform\n # If you want to use a specific Hetzner CCM and CSI version, set them below; otherwise, leave them as-is for the latest versions.\n # See https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases for the available versions.\n # hetzner_ccm_version = \"\"\n```\n\n* **`hetzner_ccm_version` (String, Optional):**\n * **Default:** The module likely picks the latest stable version.\n * **Purpose:** Allows pinning the [Hetzner Cloud Controller Manager (CCM)](https://github.com/hetznercloud/hcloud-cloud-controller-manager) to a specific version.\n * **CCM Role:** The CCM is responsible for integrating Kubernetes with Hetzner Cloud specifics, such as:\n * Setting node addresses.\n * Managing Hetzner Load Balancers for services of type `LoadBalancer`.\n * Potentially other cloud-specific integrations.\n * **Reference:** The GitHub releases link provides available versions.\n\n```terraform\n # By default, new installations use Helm to install Hetzner CCM. You can use the legacy deployment method (using `kubectl apply`) by setting `hetzner_ccm_use_helm = false`.\n hetzner_ccm_use_helm = true\n```\n\n* **`hetzner_ccm_use_helm` (Boolean, Optional):**\n * **Default:** `true`.\n * **Purpose:** Controls the deployment method for the Hetzner CCM.\n * `true`: The module uses Helm to install and manage the CCM. This is generally the modern, preferred way.\n * `false`: The module uses a legacy method, likely applying raw Kubernetes YAML manifests (`kubectl apply -f ...`). This might be for compatibility with older module versions or specific needs.\n\n```terraform\n # To enable Hetzner CCM compatibility and connection with dedicated Robot servers, set the `robot_ccm_enabled` to \"true\", default is \"false\".\n robot_ccm_enabled = true\n```\n\n* **`robot_ccm_enabled` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** Enables the integration of Hetzner Robot dedicated servers via the Cloud Controller Manager (CCM). This is only activated if `robot_user` and `robot_password` are also provided.\n * `true`: The HCCM is configured to allow connections to Robot Nodes.\n * `false`: The HCCM won't handle connections to Robot Nodes\n\n```terraform\n # Hetzner Cloud vSwitch ID. If defined, a subnet will be created in the IP-range defined by vswitch_subnet_index.\n # vswitch_id = 12345\n\n # Subnet index (0-255) for vSwitch.\n # vswitch_subnet_index = 201\n```\n\n* **`vswitch_id` (Number, Optional):**\n * **Default:** `null`.\n * **Purpose:** Links a Hetzner vSwitch to the private network. Required for hybrid setups with Robot servers.\n* **`vswitch_subnet_index` (Number, Optional):**\n * **Default:** `201`.\n * **Purpose:** Defines which subnet index (within the `subnet_amount` range) is assigned to the vSwitch connection.\n\n```terraform\n # See https://github.com/hetznercloud/csi-driver/releases for the available versions.\n # hetzner_csi_version = \"\"\n```\n\n* **`hetzner_csi_version` (String, Optional):**\n * **Default:** The module likely picks the latest stable version.\n * **Purpose:** Allows pinning the Hetzner Cloud CSI driver to a specific version (if `disable_hetzner_csi` is `false`).\n * **Reference:** The GitHub releases link provides available versions.\n\n---\n\nExcellent! Let's continue our meticulous dissection.\n\n---\n\n**Section 2.10: Kured - Automated Node Reboot Management**\n\n```terraform\n # If you want to specify the Kured version, set it below - otherwise it'll use the latest version available.\n # See https://github.com/kubereboot/kured/releases for the available versions.\n # kured_version = \"\"\n```\n\n* **`kured_version` (String, Optional):**\n * **Default:** The module likely deploys the latest stable version of Kured.\n * **Purpose:** Allows you to specify a particular version of [Kured (KUbernetes REboot Daemon)](https://github.com/kubereboot/kured).\n * **Kured's Role:** Kured runs as a DaemonSet in your cluster. It watches for a \"sentinel\" file (e.g., `/var/run/reboot-required` on Debian/Ubuntu systems) that indicates a node needs to be rebooted (typically after OS package upgrades). When detected, Kured will:\n 1. Cordon the node (mark it unschedulable).\n 2. Drain the node (gracefully evict pods).\n 3. Execute the reboot command.\n 4. After reboot, it uncordons the node (or relies on other mechanisms to confirm health).\n * **Benefit:** Automates the reboot process for OS updates, which is crucial for maintaining security and stability, especially when `automatically_upgrade_os` is enabled.\n * **Reference:** The GitHub releases link helps find specific Kured versions.\n\n---\n\n**Section 2.11: Ingress Controller Configuration**\n\n```terraform\n # Default is \"traefik\".\n # If you want to enable the Nginx (https://kubernetes.github.io/ingress-nginx/) or HAProxy ingress controller instead of Traefik, you can set this to \"nginx\" or \"haproxy\".\n # By the default we load optimal Traefik, Nginx or HAProxy ingress controller config for Hetzner, however you may need to tweak it to your needs, so to do,\n # we allow you to add a traefik_values, nginx_values or haproxy_values, see towards the end of this file in the advanced section.\n # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.\n # If you want to disable both controllers set this to \"none\"\n # ingress_controller = \"nginx\"\n # Namespace in which to deploy the ingress controllers. Defaults to the ingress_controller variable, eg (haproxy, nginx, traefik)\n # ingress_target_namespace = \"\"\n```\n\n* **`ingress_controller` (String, Optional):**\n * **Default:** `\"traefik\"`.\n * **Purpose:** Specifies which Ingress controller to deploy in the cluster. An Ingress controller is responsible for fulfilling Ingress resources, which define rules for routing external HTTP/S traffic to services within the cluster.\n * **Options:**\n * `\"traefik\"`: Deploys [Traefik Proxy](https://traefik.io/traefik/). Known for its ease of use and dynamic configuration.\n * `\"nginx\"`: Deploys the [Ingress-NGINX controller](https://kubernetes.github.io/ingress-nginx/), a popular and robust choice based on NGINX.\n * `\"haproxy\"`: Deploys an Ingress controller based on [HAProxy](https://www.haproxy.org/), known for high performance and reliability.\n * `\"none\"`: Disables the automatic deployment of any Ingress controller by this module. You would then be responsible for installing one manually if needed.\n * **Module's Role:** The module typically deploys the chosen controller using its Helm chart and applies some Hetzner-specific optimal configurations (e.g., annotations for the Hetzner Load Balancer).\n * **Customization:** Further customization is possible via `traefik_values`, `nginx_values`, or `haproxy_values` blocks (discussed later).\n* **`ingress_target_namespace` (String, Optional):**\n * **Default:** The value of `ingress_controller` (e.g., if `ingress_controller = \"nginx\"`, the default namespace is `\"nginx\"`).\n * **Purpose:** Specifies the Kubernetes namespace into which the chosen Ingress controller components will be deployed.\n\n```terraform\n # You can change the number of replicas for selected ingress controller here. The default 0 means autoselecting based on number of agent nodes (1 node = 1 replica, 2 nodes = 2 replicas, 3+ nodes = 3 replicas)\n # ingress_replica_count = 1\n```\n\n* **`ingress_replica_count` (Number, Optional):**\n * **Default:** `0`.\n * **Purpose:** Controls the number of replicas (pods) for the deployed Ingress controller.\n * **Default Logic (`0`):**\n * 1 agent node -> 1 Ingress controller replica.\n * 2 agent nodes -> 2 Ingress controller replicas.\n * 3+ agent nodes -> 3 Ingress controller replicas.\n * This provides a sensible default for HA and load distribution across agent nodes.\n * **Manual Override:** Setting a specific number (e.g., `1`, `2`, `3`) overrides this auto-selection logic.\n * **Considerations:** More replicas provide higher availability and can handle more traffic, but also consume more resources. The Ingress controller pods are typically deployed as a DaemonSet (one per node) or a Deployment with a replica count, and their service is exposed via the Hetzner Load Balancer.\n\n```terraform\n # Use the klipperLB (similar to metalLB), instead of the default Hetzner one, that has an advantage of dropping the cost of the setup.\n # Automatically \"true\" in the case of single node cluster (as it does not make sense to use the Hetzner LB in that situation).\n # It can work with any ingress controller that you choose to deploy.\n # Please note that because the klipperLB points to all nodes, we automatically allow scheduling on the control plane when it is active.\n # enable_klipper_metal_lb = true\n```\n\n* **`enable_klipper_metal_lb` (Boolean, Optional):**\n * **Default:** `false` (unless it's a single-node cluster, then it's automatically `true`).\n * **Purpose:** If `true`, deploys [Klipper LoadBalancer](https://github.com/k3s-io/klipper-lb) (which is k3s's embedded service load balancer, similar in concept to MetalLB for bare-metal clusters).\n * **Mechanism:** Klipper LB allows services of type `LoadBalancer` to get an IP address from a pool of the nodes' own IP addresses. For external access, this typically means one of the node's public IPs is used by the Ingress controller's service.\n * **Advantage (Cost):** Avoids the need for a dedicated (and paid) Hetzner Cloud Load Balancer. Traffic goes directly to one of the nodes.\n * **Disadvantage:**\n * Less sophisticated load balancing than a dedicated cloud LB.\n * If the node whose IP is being used goes down, traffic to that IP stops until Kubernetes/Klipper reassigns it (if configured for HA with multiple nodes advertising).\n * HA with Klipper LB usually involves BGP or ARP announcements, which might have complexities in a cloud environment if not handled carefully by the implementation.\n * **Single-Node Cluster:** Automatically enabled because a dedicated Hetzner LB for a single node is redundant and costly.\n * **Scheduling Implication:** \"we automatically allow scheduling on the control plane when it is active.\" If Klipper LB is used, and you have control plane nodes, they might also participate in serving traffic directly. This means the taint that usually prevents workloads on control planes might be removed or adjusted by the module.\n\n```terraform\n # If you want to configure additional arguments for traefik, enter them here as a list and in the form of traefik CLI arguments; see https://doc.traefik.io/traefik/reference/static-configuration/cli/\n # They are the options that go into the additionalArguments section of the Traefik helm values file.\n # We already add \"providers.kubernetesingress.ingressendpoint.publishedservice\" by default so that Traefik works automatically with services such as External-DNS and ArgoCD.\n # Example:\n # traefik_additional_options = [\"--log.level=DEBUG\", \"--tracing=true\"]\n```\n\n* **`traefik_additional_options` (List of Strings, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Purpose:** Allows passing additional static configuration arguments directly to the Traefik Proxy binary. These are typically arguments you would find in Traefik's static configuration (e.g., `traefik.yml` or CLI flags).\n * **Mechanism:** These options are usually injected into the `additionalArguments` section of the Traefik Helm chart values.\n * **Default Added Option:** The module already adds `\"--providers.kubernetesingress.ingressendpoint.publishedservice=true\"` (or an equivalent Helm value). This is important for Traefik to correctly report its endpoint IP address in Ingress object statuses, which is then used by tools like ExternalDNS (to create DNS records) and ArgoCD (to determine application health/sync status).\n * **Example:** `[\"--log.level=DEBUG\", \"--tracing.jaeger=true\", \"--tracing.jaeger.samplingServerURL=http://jaeger-agent.observability:5778/sampling\"]`\n * **Reference:** The Traefik static configuration CLI reference is the definitive source.\n\n```terraform\n # By default traefik image tag is an empty string which uses latest image tag.\n # The default is \"\".\n # traefik_image_tag = \"v3.0.0-beta5\"\n```\n\n* **`traefik_image_tag` (String, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Default:** `\"\"` (empty string), which usually means the Traefik Helm chart will use its default version tag (often the latest stable release).\n * **Purpose:** Allows you to pin the Traefik Proxy container image to a specific version tag (e.g., `\"v2.10.5\"`, `\"v3.0.0\"`).\n * **Benefit:** Ensures version stability and allows controlled upgrades of Traefik.\n\n```terraform\n # By default traefik is configured to redirect http traffic to https, you can set this to \"false\" to disable the redirection.\n # The default is true.\n # traefik_redirect_to_https = false\n```\n\n* **`traefik_redirect_to_https` (Boolean, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Default:** `true`.\n * **Purpose:** Controls whether the default Traefik configuration includes a global middleware to redirect all HTTP traffic to HTTPS.\n * `true`: HTTP requests to the `web` entrypoint are redirected to the `websecure` entrypoint.\n * `false`: No automatic redirection. You would need to configure HTTPS redirection per Ingress route or handle it at the application level.\n\n```terraform\n # Enable or disable Horizontal Pod Autoscaler for traefik.\n # The default is true.\n # traefik_autoscaling = false\n```\n\n* **`traefik_autoscaling` (Boolean, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Default:** `true`.\n * **Purpose:** If `true`, the module configures a Horizontal Pod Autoscaler (HPA) for the Traefik deployment. The HPA will automatically scale the number of Traefik pods up or down based on CPU utilization (or other metrics if configured in custom `traefik_values`).\n * **Benefit:** Allows Traefik to handle varying loads more efficiently.\n * **Note:** This interacts with `ingress_replica_count`. If HPA is enabled, `ingress_replica_count` might set the initial/minimum replica count for the HPA.\n\n```terraform\n # Enable or disable pod disruption budget for traefik. Values are maxUnavailable: 33% and minAvailable: 1.\n # The default is true.\n # traefik_pod_disruption_budget = false\n```\n\n* **`traefik_pod_disruption_budget` (Boolean, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Default:** `true`.\n * **Purpose:** If `true`, creates a PodDisruptionBudget (PDB) for the Traefik deployment.\n * **PDB Role:** A PDB limits the number of pods of a replicated application that can be voluntarily disrupted at the same time (e.g., during node maintenance, upgrades, or when `kubectl drain` is used).\n * **Default Values:** The comment \"maxUnavailable: 33% and minAvailable: 1\" suggests the PDB is configured to allow at most 33% of Traefik pods to be unavailable, while ensuring at least 1 pod remains available.\n * **Benefit:** Improves the availability of Traefik during planned cluster operations.\n\n```terraform\n # Enable or disable default resource requests and limits for traefik. Values requested are 100m & 50Mi and limits 300m & 150Mi.\n # The default is true.\n # traefik_resource_limits = false\n```\n\n* **`traefik_resource_limits` (Boolean, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Default:** `true`.\n * **Purpose:** If `true`, the Traefik pods are configured with default CPU and memory requests and limits.\n * **Default Values Mentioned:**\n * Requests: CPU `100m` (0.1 core), Memory `50Mi`. This is what Kubernetes guarantees the pod will have.\n * Limits: CPU `300m` (0.3 core), Memory `150Mi`. The pod cannot exceed these limits.\n * **Benefit:** Helps with resource management and scheduling. Prevents Traefik from consuming excessive resources or being starved.\n * **If `false`:** Traefik pods might run without specific requests/limits, relying on defaults or potentially being less predictable in resource consumption.\n\n```terraform\n # If you want to configure additional ports for traefik, enter them here as a list of objects with name, port, and exposedPort properties.\n # Example:\n # traefik_additional_ports = [{name = \"example\", port = 1234, exposedPort = 1234}]\n```\n\n* **`traefik_additional_ports` (List of Maps, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Purpose:** Allows defining additional ports (entrypoints in Traefik terminology) for the Traefik service beyond the standard `web` (HTTP) and `websecure` (HTTPS) ports.\n * **Structure:** A list of maps, where each map defines a port:\n * `name` (String): A unique name for this entrypoint (e.g., \"tcp-echo\", \"metrics\").\n * `port` (Number): The port number Traefik will listen on internally for this entrypoint.\n * `exposedPort` (Number): The port number on the Traefik service (and thus on the Hetzner Load Balancer) that will map to the internal `port`. Often these are the same.\n * You might also need to specify `protocol` (e.g., `TCP`, `UDP`) if not HTTP/S, depending on how the Traefik Helm chart handles this.\n * **Use Case:** Exposing non-HTTP services (e.g., TCP or UDP applications, metrics endpoints on custom ports) through Traefik.\n\n```terraform\n # If you want to configure additional trusted IPs for traefik, enter them here as a list of IPs (strings).\n # Example for Cloudflare:\n # traefik_additional_trusted_ips = [\n # \"173.245.48.0/20\",\n # // ... more Cloudflare IP ranges ...\n # ]\n```\n\n* **`traefik_additional_trusted_ips` (List of Strings, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Purpose:** Configures Traefik's `forwardedHeaders.trustedIPs` (or equivalent proxy protocol settings). When Traefik is behind another proxy (like Cloudflare, or even the Hetzner Load Balancer if it's configured to use proxy protocol), the client's real IP address is often sent in headers like `X-Forwarded-For`. Traefik needs to be told which upstream proxy IPs are \"trusted\" to correctly parse these headers and use the real client IP.\n * **Format:** A list of IP addresses or CIDR ranges.\n * **Cloudflare Example:** The provided list contains Cloudflare's public IP ranges. If Cloudflare is proxying traffic to your Traefik instance, adding these IPs tells Traefik to trust the `X-Forwarded-For` (and similar) headers set by Cloudflare.\n * **Hetzner LB:** If the Hetzner LB is configured with `uses-proxyprotocol = \"true\"`, Traefik also needs to be configured to understand proxy protocol, and the LB's private network IP range might need to be trusted. The module might handle some of this automatically.\n\n---\n\n**Section 2.12: Kubernetes Core Components & Features**\n\n```terraform\n # If you want to disable the metric server set this to \"false\". Default is \"true\".\n # enable_metrics_server = false\n```\n\n* **`enable_metrics_server` (Boolean, Optional):**\n * **Default:** `true`.\n * **Purpose:** If `true`, deploys the [Kubernetes Metrics Server](https://github.com/kubernetes-sigs/metrics-server) into the cluster.\n * **Metrics Server Role:** Aggregates resource usage data (CPU, memory) from Kubelets on each node and exposes it through the Kubernetes Metrics API. This API is used by:\n * `kubectl top node` and `kubectl top pod` commands.\n * Horizontal Pod Autoscaler (HPA) to make scaling decisions based on CPU/memory utilization.\n * **If `false`:** `kubectl top` commands will not work, and HPAs relying on standard CPU/memory metrics will not function.\n\n```terraform\n # If you want to enable the k3s built-in local-storage controller set this to \"true\". Default is \"false\".\n # Warning: When enabled together with the Hetzner CSI, there will be two default storage classes: \"local-path\" and \"hcloud-volumes\"!\n # Even if patched to remove the \"default\" label, the local-path storage class will be reset as default on each reboot of\n # the node where the controller runs.\n # This is not a problem if you explicitly define which storageclass to use in your PVCs.\n # Workaround if you don't want two default storage classes: leave this to false and add the local-path-provisioner helm chart\n # as an extra (https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner#adding-extras).\n # enable_local_storage = false\n```\n\n* **`enable_local_storage` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** If `true`, enables k3s's built-in local path provisioner. This provisioner creates a StorageClass (typically named `local-path`) that can dynamically provision PersistentVolumes using local directory paths on the nodes.\n * **Use Case:** Simple single-node persistent storage without needing an external CSI driver or distributed storage system like Longhorn. Data is tied to the specific node.\n * **Warning (Multiple Default StorageClasses):**\n * If both this and the Hetzner CSI driver are enabled (which is default for Hetzner CSI), you'll have two StorageClasses marked as `default`: `local-path` and `hcloud-volumes`.\n * When a PersistentVolumeClaim (PVC) is created without specifying a `storageClassName`, Kubernetes uses the default StorageClass. Having multiple defaults can lead to ambiguity or unintended provisioning.\n * The comment notes that k3s might reset `local-path` as default on reboots, making it hard to permanently un-default it.\n * **Recommendation:**\n * If you need dynamic local storage and want to avoid the multiple-default issue, the comment suggests leaving `enable_local_storage = false` and instead deploying the `local-path-provisioner` via its Helm chart using the module's \"extra manifests\" feature. This gives more control over its configuration, including whether it's marked as default.\n * Alternatively, always explicitly specify `storageClassName` in your PVCs.\n\n```terraform\n # If you want to allow non-control-plane workloads to run on the control-plane nodes, set this to \"true\". The default is \"false\".\n # True by default for single node clusters, and when enable_klipper_metal_lb is true. In those cases, the value below will be ignored.\n # allow_scheduling_on_control_plane = true\n```\n\n* **`allow_scheduling_on_control_plane` (Boolean, Optional):**\n * **Default:** `false` (for multi-node clusters without Klipper LB).\n * **Purpose:** Controls whether regular application pods can be scheduled on control plane nodes.\n * `false`: Control plane nodes typically have taints (e.g., `node-role.kubernetes.io/master:NoSchedule` or `node-role.kubernetes.io/control-plane:NoSchedule`) that prevent most pods from running on them. This reserves control plane resources for critical Kubernetes components.\n * `true`: These taints are removed or modified, allowing user workloads to run on control plane nodes.\n * **Automatic `true` Scenarios:**\n * **Single-Node Clusters:** If you have only one control plane node and no (or zero-count) agent nodepools, this is effectively `true` because the control plane *must* also run workloads.\n * **`enable_klipper_metal_lb = true`:** As mentioned earlier, if Klipper LB is used, control plane nodes might participate in serving traffic, so scheduling is often allowed on them.\n * **Manual Setting:** You might set this to `true` in resource-constrained environments or for specific small clusters where you want to utilize all nodes for workloads.\n\n```terraform\n # If you want to disable the automatic upgrade of k3s, you can set below to \"false\".\n # Ideally, keep it on, to always have the latest Kubernetes version, but lock the initial_k3s_channel to a kube major version,\n # of your choice, like v1.25 or v1.26. That way you get the best of both worlds without the breaking changes risk.\n # For production use, always use an HA setup with at least 3 control-plane nodes and 2 agents, and keep this on for maximum security.\n\n # The default is \"true\" (in HA setup i.e. at least 3 control plane nodes & 2 agents, just keep it enabled since it works flawlessly).\n # automatically_upgrade_k3s = false\n```\n\n* **`automatically_upgrade_k3s` (Boolean, Optional):**\n * **Default:** `true` (especially in HA setups).\n * **Purpose:** Controls whether k3s versions are automatically upgraded on the nodes using Rancher's System Upgrade Controller.\n * `true`: The System Upgrade Controller will monitor for new k3s versions (based on `initial_k3s_channel` or `install_k3s_version` if it defines a channel) and apply them according to a plan (e.g., upgrading control planes one by one, then agents).\n * `false`: Disables automatic k3s upgrades. You would be responsible for manually upgrading k3s versions.\n * **Recommendation:**\n * **HA Setup:** Generally safe and recommended to keep `true` for security patches and new features. Pinning `initial_k3s_channel` to a specific minor version (e.g., `\"v1.29\"`) provides stability by only getting patch releases for that minor version.\n * **Non-HA Setup:** Can be risky if an upgrade fails on the single control plane. Often recommended to set to `false` or manage very carefully.\n * **Mechanism:** Uses the [System Upgrade Controller](https://github.com/rancher/system-upgrade-controller), which is deployed into the cluster.\n\n```terraform\n # By default nodes are drained before k3s upgrade, which will delete and transfer all pods to other nodes.\n # Set this to false to cordon nodes instead, which just prevents scheduling new pods on the node during upgrade\n # and keeps all pods running. This may be useful if you have pods which are known to be slow to start e.g.\n # because they have to mount volumes with many files which require to get the right security context applied.\n system_upgrade_use_drain = true\n```\n\n* **`system_upgrade_use_drain` (Boolean, Optional, relevant if `automatically_upgrade_k3s = true`):**\n * **Default:** `true`.\n * **Purpose:** Controls the behavior of the System Upgrade Controller when upgrading a node.\n * `true`: The node is cordoned and then drained (`kubectl drain`). Draining evicts all pods gracefully, allowing them to be rescheduled on other available nodes. This is the safest approach to ensure no workload interruption if pods can be moved.\n * `false`: The node is only cordoned. Existing pods continue to run on the node during the k3s upgrade process. New pods won't be scheduled there.\n * **Use Case for `false`:** If you have stateful applications or pods that are very slow to restart or have complex dependencies that make draining problematic or lengthy. However, this means those pods will experience a brief outage when the k3s service restarts on that node during the upgrade.\n\n```terraform\n # During k3s via system-upgrade-manager pods are evicted by default.\n # On small clusters this can lead to hanging upgrades and indefinitely unschedulable nodes,\n # in that case, set this to false to immediately delete pods before upgrading.\n # NOTE: Turning this flag off might lead to downtimes of services (which may be acceptable for your use case)\n # NOTE: This flag takes effect only when system_upgrade_use_drain is set to true.\n # system_upgrade_enable_eviction = false\n```\n\n* **`system_upgrade_enable_eviction` (Boolean, Optional, relevant if `automatically_upgrade_k3s = true` and `system_upgrade_use_drain = true`):**\n * **Default:** `true` (implied, as pods are evicted during drain by default).\n * **Purpose:** Fine-tunes the pod removal process during a `drain` operation initiated by the System Upgrade Controller.\n * `true` (Default behavior of `kubectl drain`): Uses the Kubernetes eviction API. This respects PodDisruptionBudgets (PDBs). If evicting a pod would violate a PDB (e.g., not enough replicas of an application would remain), the eviction might be delayed or fail.\n * `false`: Pods are deleted more forcefully/immediately (likely `kubectl delete pod --force --grace-period=0` or similar, bypassing PDB checks).\n * **Use Case for `false`:**\n * **Small Clusters:** In very small clusters (e.g., 2 nodes), if a PDB requires, say, 2 replicas of an app to be always available, draining one node might be impossible if the app only has 2 pods. This can stall the upgrade. Setting `system_upgrade_enable_eviction = false` would force pod deletion, allowing the upgrade to proceed but causing a brief downtime for that app.\n * **Warning:** Setting to `false` can lead to temporary service outages if PDBs are not respected.\n\n```terraform\n # The default is \"true\" (in HA setup it works wonderfully well, with automatic roll-back to the previous snapshot in case of an issue).\n # IMPORTANT! For non-HA clusters i.e. when the number of control-plane nodes is < 3, you have to turn it off.\n # automatically_upgrade_os = false\n```\n\n* **`automatically_upgrade_os` (Boolean, Optional):**\n * **Default:** `true` (for HA setups).\n * **Purpose:** Controls whether the underlying operating system packages on the nodes are automatically upgraded.\n * `true`: The module likely configures unattended upgrades (e.g., `unattended-upgrades` package on Debian/Ubuntu) or a similar mechanism to automatically install OS security patches and updates. Kured then handles the reboots if required.\n * `false`: Disables automatic OS upgrades. You would be responsible for manually updating the OS on each node.\n * **Critical Constraint for Non-HA:** \"For non-HA clusters ... you have to turn it off.\" If you have a single control plane, an automatic OS upgrade that requires a reboot (and is handled by Kured) will cause downtime for the entire Kubernetes API.\n * **Rollback Mention:** The comment \"automatic roll-back to the previous snapshot\" likely refers to features of the underlying OS or bootloader (e.g., transactional updates with `btrfs` snapshots as used by openSUSE MicroOS, which this module uses as the base OS image). If an OS upgrade fails, the system might be able to roll back to a pre-upgrade state.\n\n```terraform\n # If you need more control over kured and the reboot behaviour, you can pass additional options to kured.\n # For example limiting reboots to certain timeframes. For all options see: https://kured.dev/docs/configuration/\n # By default, the kured lock does not expire and is only released once a node successfully reboots. You can add the option\n # \"lock-ttl\" : \"30m\", if you have a single node which sometimes gets stuck. Note however, that in that case, kured continuous\n # draining the next node because the lock was released. You may end up with all nodes drained and your cluster completely down.\n # The default options are: `--reboot-command=/usr/bin/systemctl reboot --pre-reboot-node-labels=kured=rebooting --post-reboot-node-labels=kured=done --period=5m`\n # Defaults can be overridden by using the same key.\n # kured_options = {\n # \"reboot-days\": \"su\", # Example: only reboot on Sunday\n # \"start-time\": \"3am\",\n # \"end-time\": \"8am\",\n # \"time-zone\": \"Local\", # Or a specific IANA timezone like \"Europe/Berlin\"\n # \"lock-ttl\" : \"30m\",\n # }\n```\n\n* **`kured_options` (Map of Strings, Optional):**\n * **Purpose:** Allows passing additional command-line arguments to the Kured daemon to customize its behavior.\n * **Format:** A map where keys are Kured CLI option names (without leading `--`) and values are their settings.\n * **Default Kured Options (Managed by Module):** The comment lists some default arguments the module likely passes to Kured:\n * `--reboot-command=/usr/bin/systemctl reboot`: The command Kured uses to reboot the node.\n * `--pre-reboot-node-labels=kured=rebooting`: A label Kured adds to the node *before* rebooting.\n * `--post-reboot-node-labels=kured=done`: A label Kured adds *after* a successful reboot (or that it expects to be present).\n * `--period=5m`: How often Kured checks for the reboot-required sentinel.\n * **Overriding Defaults:** You can override these by providing the same key in your `kured_options` map.\n * **Example Customizations:**\n * `\"reboot-days\": \"su\"`: Restrict reboots to only occur on Sundays.\n * `\"start-time\": \"3am\"`, `\"end-time\": \"8am\"`: Define a maintenance window for reboots.\n * `\"time-zone\": \"Local\"` or `\"Europe/Berlin\"`: Specify the timezone for the maintenance window.\n * `\"lock-ttl\": \"30m\"`: Sets a Time-To-Live for Kured's distributed lock. Kured uses a lock (often a ConfigMap or Lease) to ensure only one node reboots at a time. If a node gets stuck during reboot and doesn't release the lock, this TTL would eventually expire the lock, allowing Kured to proceed with another node. **Warning:** As the comment notes, if the stuck node *doesn't* actually reboot, and the lock expires, Kured might start draining another node, potentially leading to multiple nodes being down if the issue is systemic. Use with caution.\n * **Reference:** The Kured documentation is the definitive source for all its configuration options.\n\n\n\n\n**Section 2.13: k3s Versioning and Naming**\n\n```terraform\n # Allows you to specify the k3s version. If defined, supersedes initial_k3s_channel.\n # See https://github.com/k3s-io/k3s/releases for the available versions.\n # install_k3s_version = \"v1.30.2+k3s2\"\n```\n\n* **`install_k3s_version` (String, Optional):**\n * **Purpose:** Allows you to specify an exact k3s version to install on all nodes.\n * **Format:** Should match a k3s release tag from their GitHub releases (e.g., `\"v1.30.2+k3s2\"`). The `+k3sX` suffix indicates a k3s-specific build/patch of that Kubernetes version.\n * **Precedence:** If both `install_k3s_version` and `initial_k3s_channel` are set, `install_k3s_version` takes precedence for the *initial* installation.\n * **Upgrades:** If `automatically_upgrade_k3s = true`, the System Upgrade Controller will still look for newer versions within the channel defined by `initial_k3s_channel` (or the channel this specific version belongs to) unless the `install_k3s_version` itself points to a specific channel behavior (less common for exact versions).\n * **Benefit:** Guarantees a specific k3s version is installed, useful for consistency, testing, or avoiding issues with very new/unstable releases from a channel.\n\n```terraform\n # Allows you to specify either stable, latest, testing or supported minor versions.\n # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels\n # ⚠️ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to one minor version below the latest stable,\n # e.g. v1.29 instead of the stable v1.30.\n # The default is \"v1.30\".\n # initial_k3s_channel = \"stable\"\n```\n\n* **`initial_k3s_channel` (String, Optional):**\n * **Default (in module):** `\"v1.30\"` (or another recent stable minor version channel).\n * **Purpose:** Specifies the k3s release channel from which to install k3s initially and, if `automatically_upgrade_k3s = true`, from which to pull subsequent upgrades.\n * **Channel Options:**\n * `\"stable\"`: Points to the latest stable k3s release.\n * `\"latest\"`: Points to the most recent k3s release, which might include release candidates or newer patches than \"stable\".\n * `\"testing\"`: For pre-release versions. Not for production.\n * Minor version channels (e.g., `\"v1.30\"`, `\"v1.29\"`): Installs the latest patch release within that specific Kubernetes minor version. This is **highly recommended for production** as it provides stability by avoiding automatic major/minor version jumps, while still allowing for security patches.\n * **Rancher Compatibility (⚠️):** Rancher often has specific Kubernetes version compatibility requirements. It's crucial to choose an `initial_k3s_channel` (or `install_k3s_version`) that is supported by the version of Rancher you intend to use (if `enable_rancher = true`). The advice to use one minor version below the absolute latest stable is good practice for broader addon compatibility.\n * **Reference:** The k3s documentation links explain channels in detail.\n\n```terraform\n # Allows to specify the version of the System Upgrade Controller for automated upgrades of k3s\n # See https://github.com/rancher/system-upgrade-controller/releases for the available versions.\n # sys_upgrade_controller_version = \"v0.14.2\"\n```\n\n* **`sys_upgrade_controller_version` (String, Optional, relevant if `automatically_upgrade_k3s = true`):**\n * **Default:** The module likely picks a recent, compatible version of the System Upgrade Controller.\n * **Purpose:** Allows you to pin the version of the Rancher System Upgrade Controller that is deployed to manage k3s upgrades.\n * **Benefit:** Version pinning for stability or if you need a specific feature/fix from a particular controller version.\n\n```terraform\n # The cluster name, by default \"k3s\"\n # cluster_name = \"\"\n```\n\n* **`cluster_name` (String, Optional):**\n * **Default (in module):** `\"k3s\"`.\n * **Purpose:** Sets a name for your Kubernetes cluster. This name might be used in:\n * The generated kubeconfig context name.\n * Naming of some cloud resources created by the module (e.g., prefixing firewall names, network names).\n * Internal k3s cluster identification.\n * **If `\"\"` (empty string) is provided:** The module will use its default, likely \"k3s\".\n\n```terraform\n # Whether to use the cluster name in the node name, in the form of {cluster_name}-{nodepool_name}, the default is \"true\".\n # use_cluster_name_in_node_name = false\n```\n\n* **`use_cluster_name_in_node_name` (Boolean, Optional):**\n * **Default:** `true`.\n * **Purpose:** Controls the naming convention for the Hetzner server instances (and thus Kubernetes node names).\n * `true`: Node names will be prefixed with the `cluster_name`, e.g., `k3s-cp-nbg1-1`, `mycluster-agent-small-1`.\n * `false`: Node names will likely just use the nodepool name and an index, e.g., `cp-nbg1-1`, `agent-small-1`.\n * **Benefit of `true`:** Helps differentiate nodes if you manage multiple clusters within the same Hetzner project.\n\n---\n\n**Section 2.14: Advanced k3s Configuration (Registries, Environment, Pre-install)**\n\n```terraform\n # Extra k3s registries. This is useful if you have private registries and you want to pull images without additional secrets.\n # Or if you want to proxy registries for various reasons like rate-limiting.\n # It will create the registries.yaml file, more info here https://docs.k3s.io/installation/private-registry.\n # Note that you do not need to get this right from the first time, you can update it when you want during the life of your cluster.\n # The default is blank.\n /* k3s_registries = <<-EOT\n mirrors:\n hub.my_registry.com:\n endpoint:\n - \"hub.my_registry.com\"\n configs:\n hub.my_registry.com:\n auth:\n username: username\n password: password\n EOT */\n```\n\n* **`k3s_registries` (String, Optional, Heredoc or File Content):**\n * **Default:** Blank/not set.\n * **Purpose:** Allows you to configure k3s's containerd (its container runtime) with custom image registry settings. This is typically done by creating a `registries.yaml` file on each node (e.g., in `/etc/rancher/k3s/registries.yaml`).\n * **Format:** The value should be a string containing the YAML content for `registries.yaml`. The example uses a Terraform heredoc (`<<-EOT ... EOT`) for multi-line string input.\n * **Use Cases:**\n * **Private Registries:** Configure authentication (username/password, certs) for pulling images from private container registries.\n * **Registry Mirrors/Proxies:** Define mirrors for public registries (like Docker Hub) to reduce reliance on them, overcome rate limits, or use a local caching proxy.\n * **Insecure Registries:** Configure containerd to allow pulling from registries that use self-signed certificates or HTTP (not recommended for production without other security measures).\n * **Lifecycle:** \"you can update it when you want during the life of your cluster.\" The module will likely re-apply this configuration to the nodes if changed. Containerd would then need to be restarted or reconfigured to pick up the changes.\n * **Reference:** The k3s private registry documentation is key.\n\n```terraform\n # Additional environment variables for the host OS on which k3s runs. See for example https://docs.k3s.io/advanced#configuring-an-http-proxy .\n # additional_k3s_environment = {\n # \"CONTAINERD_HTTP_PROXY\" : \"http://your.proxy:port\",\n # \"CONTAINERD_HTTPS_PROXY\" : \"http://your.proxy:port\",\n # \"NO_PROXY\" : \"127.0.0.0/8,10.0.0.0/8,\", # Note the trailing comma for NO_PROXY\n # }\n```\n\n* **`additional_k3s_environment` (Map of Strings, Optional):**\n * **Purpose:** Allows setting additional environment variables that will be available to the k3s server and agent processes when they start. This is often done by writing to `/etc/default/k3s` or `/etc/systemd/system/k3s.service.d/override.conf` (or similar for k3s-agent).\n * **Use Case (HTTP Proxy):** The primary example is configuring k3s and containerd to use an HTTP/S proxy for outbound connections (e.g., for pulling images or communicating with external services if the nodes are in a restricted network).\n * `CONTAINERD_HTTP_PROXY` / `CONTAINERD_HTTPS_PROXY`: For containerd image pulls.\n * `HTTP_PROXY` / `HTTPS_PROXY`: Might be needed for k3s itself or other components.\n * `NO_PROXY`: A comma-separated list of IP addresses, CIDRs, or domain names that should *not* go through the proxy (e.g., internal cluster IPs, local addresses, Hetzner metadata services). The trailing comma is often significant.\n * **Other Uses:** Setting any other environment variables required by k3s or its components.\n\n```terraform\n # Additional commands to execute on the host OS before the k3s install, for example fetching and installing certs.\n # preinstall_exec = [\n # \"curl https://somewhere.over.the.rainbow/ca.crt > /root/ca.crt\",\n # \"trust anchor --store /root/ca.crt\", # Command for openSUSE/SLE to add CA to trust store\n # ]\n```\n\n* **`preinstall_exec` (List of Strings, Optional):**\n * **Purpose:** A list of shell commands that will be executed on each node *before* the k3s installation script is run.\n * **Mechanism:** The module likely uses Terraform provisioners (`remote-exec`) or cloud-init user data to execute these commands.\n * **Use Cases:**\n * **Installing Custom CA Certificates:** As in the example, fetching a custom CA certificate and adding it to the system's trust store. This is necessary if k3s needs to communicate with internal services that use certificates signed by this custom CA (e.g., a private image registry, an internal authentication provider). The `trust anchor` command is specific to systems using `update-ca-certificates` with a certain backend; other distros might use `update-ca-certificates` directly or other commands.\n * Installing prerequisite packages not included in the base OS image.\n * Performing other OS-level customizations needed before k3s starts.\n * **Caution:** Keep these commands idempotent (safe to run multiple times without unintended side effects) if possible, as Terraform might re-run provisioners under certain conditions.\n\n```terraform\n # Structured authentication configuration. Multiple authentication providers support requires v1.30+ of\n # kubernetes.\n # https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration\n #\n # authentication_config = <<-EOT\n # apiVersion: apiserver.config.k8s.io/v1beta1\n # kind: AuthenticationConfiguration\n # jwt:\n # - issuer:\n # url: \"https://token.actions.githubusercontent.com\"\n # audiences:\n # - \"https://github.com/octo-org\"\n # claimMappings:\n # username:\n # claim: sub\n # prefix: \"gh:\"\n # groups:\n # claim: repository_owner\n # prefix: \"gh:\"\n # claimValidationRules:\n # - claim: repository\n # requiredValue: \"octo-org/octo-repo\"\n # # ... more rules ...\n # - issuer: # Example for a second OIDC provider\n # url: \"https://your.oidc.issuer\"\n # audiences:\n # - \"oidc_client_id\"\n # # ... claim mappings for second provider ...\n # EOT\n```\n\n* **`authentication_config` (String, Optional, Heredoc or File Content):**\n * **Purpose:** Allows configuring the Kubernetes API server with an `AuthenticationConfiguration` object. This is a more structured and flexible way to define authentication methods, especially for multiple OIDC providers or JWT issuers, compared to older flat CLI flags.\n * **Kubernetes Feature:** This is a standard Kubernetes API server feature, generally available from v1.19+ for OIDC and enhanced for multiple JWT issuers in v1.30+. k3s passes these configurations to its embedded API server.\n * **Format:** The value should be a string containing the YAML content for the `AuthenticationConfiguration` object.\n * **Use Case (Example - GitHub Actions OIDC):** The example shows how to configure the API server to trust OIDC tokens issued by GitHub Actions. This allows GitHub Actions workflows to authenticate to the Kubernetes cluster to deploy applications, manage resources, etc., without needing long-lived static credentials.\n * `issuer`: The OIDC provider's URL and expected audiences.\n * `claimMappings`: How to map claims from the OIDC token to Kubernetes usernames and groups.\n * `claimValidationRules`: Additional rules to validate specific claims in the token (e.g., ensuring the token is from a specific GitHub repository).\n * **Multiple Providers:** The structure allows defining multiple `jwt` issuers or other authentication mechanisms.\n * **Reference:** The Kubernetes authentication documentation link is crucial.\n\n---\n\n**Section 2.15: k3s Server and Agent Execution Arguments**\n\n```terraform\n # Additional flags to pass to the k3s server command (the control plane).\n # k3s_exec_server_args = \"--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector\"\n```\n\n* **`k3s_exec_server_args` (String or List of Strings, Optional):**\n * **Purpose:** Allows passing additional command-line arguments directly to the `k3s server` process that runs on control plane nodes.\n * **Format:** Can be a single string with space-separated arguments, or a list of strings where each element is an argument.\n * **Example:** `--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector`\n * `--kube-apiserver-arg`: This is a k3s-specific flag that allows you to pass arguments through to the underlying `kube-apiserver` binary that k3s embeds.\n * `enable-admission-plugins=...`: Enables specific Kubernetes admission controllers.\n * `PodTolerationRestriction`: Can restrict which tolerations pods can have based on namespace annotations.\n * `PodNodeSelector`: Can enforce or default node selectors for pods based on namespace annotations.\n * **Reference:** Consult the k3s server CLI options (`k3s server --help`) and the Kubernetes `kube-apiserver` documentation for available arguments.\n\n```terraform\n # Additional flags to pass to the k3s agent command (every agents nodes, including autoscaler nodepools).\n # k3s_exec_agent_args = \"--kubelet-arg kube-reserved=cpu=100m,memory=200Mi,ephemeral-storage=1Gi\"\n```\n\n* **`k3s_exec_agent_args` (String or List of Strings, Optional):**\n * **Purpose:** Allows passing additional command-line arguments directly to the `k3s agent` process that runs on agent nodes (and nodes created by the Cluster Autoscaler).\n * **Example:** `--kubelet-arg kube-reserved=cpu=100m,memory=200Mi,ephemeral-storage=1Gi`\n * `--kubelet-arg`: A k3s-specific flag to pass arguments through to the underlying `kubelet` binary.\n * `kube-reserved=...`: Reserves specified resources for Kubernetes system components on the agent node.\n * **Reference:** Consult k3s agent CLI options (`k3s agent --help`) and Kubernetes `kubelet` documentation.\n\n```terraform\n # The vars below here passes it to the k3s config.yaml. This way it persist across reboots\n # Make sure you set \"feature-gates=NodeSwap=true\" if want to use swap_size\n # Note: CloudDualStackNodeIPs was removed in K8s 1.32 (always enabled now)\n # see https://github.com/k3s-io/k3s/issues/8811#issuecomment-1856974516\n # k3s_global_kubelet_args = [\"kube-reserved=cpu=100m,ephemeral-storage=1Gi\", \"system-reserved=cpu=memory=200Mi\", \"image-gc-high-threshold=50\", \"image-gc-low-threshold=40\"]\n # k3s_control_plane_kubelet_args = []\n # k3s_agent_kubelet_args = []\n # k3s_autoscaler_kubelet_args = []\n```\n\n* **Kubelet Arguments via `config.yaml` (Persistent):**\n * **Purpose:** These variables allow configuring kubelet arguments by writing them into the k3s `config.yaml` file (e.g., `/etc/rancher/k3s/config.yaml`). Arguments set this way are persistent across k3s restarts and reboots, which is generally preferred over transient CLI args for kubelet settings.\n * **`k3s_global_kubelet_args` (List of Strings, Optional):**\n * Kubelet arguments to apply to *all* nodes (control plane, agent, autoscaled).\n * Example:\n * `\"kube-reserved=cpu=100m,ephemeral-storage=1Gi\"`\n * `\"system-reserved=cpu=memory=200Mi\"`\n * `\"image-gc-high-threshold=50\"`: Kubelet will start garbage collecting unused container images when disk usage for images exceeds 50%.\n * `\"image-gc-low-threshold=40\"`: Kubelet will stop garbage collecting images once disk usage drops below 40%.\n * `\"feature-gates=NodeSwap=true\"`: As per the comment, this is crucial if you intend to use the `swap_size` attribute on nodepools. `NodeSwap` enables kubelet's experimental swap support. Note: `CloudDualStackNodeIPs` was removed in K8s 1.32 (always enabled now).\n * **`k3s_control_plane_kubelet_args` (List of Strings, Optional):**\n * Kubelet arguments specific to control plane nodes. These would be merged with or override `k3s_global_kubelet_args`.\n * **`k3s_agent_kubelet_args` (List of Strings, Optional):**\n * Kubelet arguments specific to regular agent nodes (defined in `agent_nodepools`).\n * **`k3s_autoscaler_kubelet_args` (List of Strings, Optional):**\n * Kubelet arguments specific to nodes created by the Cluster Autoscaler (defined in `autoscaler_nodepools`).\n * **Nodepool-Specific `kubelet_args`:** Recall that individual nodepool definitions (e.g., within `control_plane_nodepools` or `agent_nodepools`) can also have a `kubelet_args` attribute. The order of precedence (global -> type-specific -> nodepool-specific) would need to be confirmed from the module's implementation, but typically more specific settings override general ones.\n \n \n \n\n**Section 2.16: Firewall and Security Settings**\n\n```terraform\n # If you want to allow all outbound traffic you can set this to \"false\". Default is \"true\".\n # restrict_outbound_traffic = false\n```\n\n* **`restrict_outbound_traffic` (Boolean, Optional):**\n * **Default:** `true`.\n * **Purpose:** Controls the default outbound traffic policy for the Hetzner Firewall associated with the cluster nodes.\n * `true`: The module configures the firewall to restrict outbound traffic. It will likely allow essential outbound traffic (e.g., for DNS, NTP, pulling images from common registries, k3s communication, Hetzner metadata) but might block other arbitrary outbound connections by default. You would then need to add `extra_firewall_rules` for any other specific outbound access your workloads require.\n * `false`: The firewall is configured to allow all outbound traffic from the nodes. This is simpler but less secure.\n * **Security Implication:** Restricting outbound traffic (`true`) is a good security practice (defense in depth) as it can limit the ability of a compromised pod/node to exfiltrate data or connect to malicious external command-and-control servers.\n\n```terraform\n # Allow access to the Kube API from the specified networks. The default is [\"0.0.0.0/0\", \"::/0\"].\n # Allowed values: null (disable Kube API rule entirely) or a list of allowed networks with CIDR notation.\n # For maximum security, it's best to disable it completely by setting it to null. However, in that case, to get access to the kube api,\n # you would have to connect to any control plane node via SSH, as you can run kubectl from within these.\n # Please be advised that this setting has no effect on the load balancer when the use_control_plane_lb variable is set to true. This is\n # because firewall rules cannot be applied to load balancers yet.\n # firewall_kube_api_source = null\n```\n\n* **`firewall_kube_api_source` (List of Strings or `null`, Optional):**\n * **Default (in module):** `[\"0.0.0.0/0\", \"::/0\"]` (Allow from anywhere on IPv4 and IPv6).\n * **Purpose:** Defines the source IP CIDR ranges allowed to access the Kubernetes API server (typically on port 6443) through the Hetzner Firewall.\n * **Values:**\n * List of CIDRs (e.g., `[\"YOUR_HOME_IP/32\", \"YOUR_OFFICE_IP_RANGE/24\"]`): Only allows access from these specified IPs. **This is highly recommended for security.**\n * `null`: Disables the firewall rule for the Kube API entirely. This means the Kube API server port (6443) would *not* be opened on the Hetzner Firewall for direct public access to the control plane nodes.\n * **Accessing API if `null`:** If set to `null`, you would need alternative methods to access the API:\n * SSH into a control plane node and run `kubectl` locally from there (as it can access the API via localhost or the private network).\n * Set up a VPN into the Hetzner private network.\n * Use an SSH tunnel (`ssh -L local_port:localhost:6443 user@control_plane_ip`) and point your local `kubectl` to `https://localhost:local_port`.\n * **`use_control_plane_lb = true` Implication:** If you are using a dedicated Hetzner Load Balancer in front of your control plane nodes (`use_control_plane_lb = true`), this `firewall_kube_api_source` setting (which applies to the *nodes'* firewall) has no direct effect on the accessibility of the API *through that load balancer*. Hetzner LBs currently do not support applying firewall rules directly to themselves. Access to the LB would be open, and security would rely on Kubernetes RBAC and authentication.\n * **Security Best Practice:** Restrict this to the minimum necessary IPs or use `null` and access via SSH/VPN.\n\n```terraform\n # Allow SSH access from the specified networks. Default: [\"0.0.0.0/0\", \"::/0\"]\n # Allowed values: null (disable SSH rule entirely) or a list of allowed networks with CIDR notation.\n # Ideally you would set your IP there. And if it changes after cluster deploy, you can always update this variable and apply again.\n # firewall_ssh_source = [\"1.2.3.4/32\"]\n```\n\n* **`firewall_ssh_source` (List of Strings or `null`, Optional):**\n * **Default (in module):** `[\"0.0.0.0/0\", \"::/0\"]` (Allow SSH from anywhere).\n * **Purpose:** Defines the source IP CIDR ranges allowed to access the SSH port (default 22, or custom `ssh_port`) on the cluster nodes through the Hetzner Firewall.\n * **Values:**\n * List of CIDRs (e.g., `[\"YOUR_HOME_IP/32\"]`): **Highly recommended.**\n * `null`: Disables the SSH firewall rule. This would make nodes inaccessible via public SSH unless you have another access path (e.g., Hetzner's web console, private network access from another server). Not generally recommended unless you have a very specific setup.\n * **Dynamic IP:** If your access IP changes, you can update this variable and re-run `terraform apply` to update the firewall rule.\n * **Security Best Practice:** Always restrict SSH access to known, trusted IP addresses.\n\n```terraform\n # By default, SELinux is enabled in enforcing mode on all nodes. For container-specific SELinux issues,\n # consider using the pre-installed 'udica' tool to create custom, targeted SELinux policies instead of\n # disabling SELinux globally. See the \"Fix SELinux issues with udica\" example in the README for details.\n # disable_selinux = false\n```\n\n* **`disable_selinux` (Boolean, Optional):**\n * **Default:** `false` (meaning SELinux is *enabled* in enforcing mode).\n * **Background:** The base OS image used by this module (openSUSE MicroOS) comes with SELinux enabled and enforcing by default. SELinux is a security module that provides mandatory access control (MAC).\n * **Purpose:**\n * `false`: Keeps SELinux enabled. This is generally better for security but can sometimes cause issues if containers are not SELinux-aware or if their default SELinux policies are too restrictive for their needs.\n * `true`: Disables SELinux (likely sets it to permissive or fully disabled) on the nodes. This can make it easier to get problematic containers running but reduces the overall security posture.\n * **Troubleshooting SELinux Issues:**\n * The comment recommends using `udica` (a tool pre-installed on MicroOS) to generate custom, targeted SELinux policies for containers that are having permission issues, rather than disabling SELinux globally. This allows you to grant only the necessary permissions.\n * Checking audit logs (`ausearch -m avc -ts recent`) is crucial for diagnosing SELinux denials.\n * **Recommendation:** Keep SELinux enabled (`disable_selinux = false`) and learn to work with it (e.g., using `udica`, understanding context labels) for better security. Disable it only as a last resort or for temporary debugging.\n\n```terraform\n # Adding extra firewall rules, like opening a port\n # More info on the format here https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/firewall\n # extra_firewall_rules = [\n # {\n # description = \"For Postgres\"\n # direction = \"in\"\n # protocol = \"tcp\"\n # port = \"5432\"\n # source_ips = [\"0.0.0.0/0\", \"::/0\"] # Be more restrictive in production!\n # destination_ips = [] # Won't be used for \"in\" rules on all nodes\n # },\n # {\n # description = \"To Allow ArgoCD access to resources via SSH\"\n # direction = \"out\"\n # protocol = \"tcp\"\n # port = \"22\"\n # source_ips = [] # Won't be used for \"out\" rules from all nodes\n # destination_ips = [\"0.0.0.0/0\", \"::/0\"] # Allow outbound SSH to anywhere\n # }\n # ]\n```\n\n* **`extra_firewall_rules` (List of Maps, Optional):**\n * **Purpose:** Allows you to define additional custom rules for the Hetzner Firewall that protects your cluster nodes. This is used for opening specific ports for applications running in your cluster or allowing specific outbound connections.\n * **Structure:** A list of maps, where each map defines a firewall rule. The attributes within each map correspond to the arguments of the `hcloud_firewall` resource's rule block.\n * **Rule Attributes:**\n * `description` (String, Optional): A human-readable description for the rule.\n * `direction` (String, Obligatory): `\"in\"` for inbound traffic to your nodes, or `\"out\"` for outbound traffic from your nodes.\n * `protocol` (String, Obligatory): Traffic protocol (e.g., `\"tcp\"`, `\"udp\"`, `\"icmp\"`).\n * `port` (String, Optional): Port number or range (e.g., `\"5432\"`, `\"8000-8080\"`). Required for TCP/UDP.\n * `source_ips` (List of Strings, Obligatory for `direction = \"in\"`): Source IP CIDRs allowed for inbound rules.\n * `destination_ips` (List of Strings, Obligatory for `direction = \"out\"`): Destination IP CIDRs allowed for outbound rules.\n * **Example 1 (Inbound Postgres):** Opens TCP port 5432 from any source. **Warning:** `[\"0.0.0.0/0\", \"::/0\"]` for `source_ips` is insecure for production databases; restrict it to known IPs.\n * **Example 2 (Outbound SSH for ArgoCD):** Allows nodes to make outbound SSH connections (TCP port 22) to any destination. This might be needed if ArgoCD (or another tool) needs to clone Git repositories via SSH from within the cluster.\n * **Reference:** The Hetzner provider documentation for the `hcloud_firewall` resource is the definitive guide for rule syntax.\n\n---\n\n**Section 2.17: CNI (Container Network Interface) Plugin Configuration**\n\n```terraform\n # If you want to configure a different CNI for k3s, use this flag\n # possible values: flannel (Default), calico, and cilium\n # As for Cilium, we allow infinite configurations via helm values, please check the CNI section of the readme over at https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/#cni.\n # Also, see the cilium_values at towards the end of this file, in the advanced section.\n # ⚠️ Depending on your setup, sometimes you need your control-planes to have more than\n # 2GB of RAM if you are going to use Cilium, otherwise the pods will not start.\n # cni_plugin = \"cilium\"\n```\n\n* **`cni_plugin` (String, Optional):**\n * **Default (in k3s, if not overridden by module):** Flannel (via VXLAN). The module might also default to Flannel.\n * **Purpose:** Specifies the CNI plugin to be installed and used by k3s for pod networking. The CNI plugin is responsible for allocating IP addresses to pods, connecting them to the network, and enforcing network policies (if supported).\n * **Options Provided by Module:**\n * `\"flannel\"`: A simple and widely used CNI plugin. Good for basic pod networking. k3s bundles it.\n * `\"calico\"`: A popular CNI known for its robust network policy enforcement and scalability. Can operate in IP-in-IP, VXLAN, or BGP modes (BGP mode is more for on-prem/bare-metal).\n * `\"cilium\"`: A powerful CNI based on eBPF. Offers advanced networking features (e.g., efficient load balancing, fine-grained network policies, Hubble observability, service mesh capabilities, transparent encryption, egress gateway).\n * **k3s Integration:** k3s can be started with CNI disabled (`--flannel-backend=none` or similar flags) allowing an external CNI like Calico or Cilium to be installed. This module handles that process.\n * **Cilium RAM Warning (⚠️):** Cilium, especially with all features enabled, can be more resource-intensive than Flannel. The comment warns that control plane nodes might need more than 2GB of RAM (e.g., `cx23` might be tight, consider `cx33`/`cx43` or higher) for Cilium pods to run reliably.\n * **Cilium Customization:** The module allows extensive Cilium configuration via the `cilium_values` block (discussed later).\n\n```terraform\n # You can choose the version of Cilium that you want. By default we keep the version up to date and configure Cilium with compatible settings according to the version.\n # See https://github.com/cilium/cilium/releases for the available versions.\n # cilium_version = \"v1.14.0\"\n```\n\n* **`cilium_version` (String, Optional, relevant if `cni_plugin = \"cilium\"`):**\n * **Default:** The module likely picks a recent, stable version of Cilium.\n * **Purpose:** Allows pinning the Cilium installation to a specific version.\n * **Benefit:** Version stability, access to specific features/fixes.\n * **Compatibility:** The module's default Cilium configurations (if `cilium_values` is not used) are likely tailored to be compatible with the Cilium version it defaults to or the one you specify. Major changes in Cilium versions can alter Helm chart values or feature availability.\n\n```terraform\n # Set native-routing mode (\"native\") or tunneling mode (\"tunnel\"). Default: tunnel\n # cilium_routing_mode = \"native\"\n```\n\n* **`cilium_routing_mode` (String, Optional, relevant if `cni_plugin = \"cilium\"`):**\n * **Default (in module, if not overridden by `cilium_values`):** `\"tunnel\"` (likely VXLAN or Geneve).\n * **Purpose:** Configures Cilium's routing mode for inter-node pod traffic.\n * `\"tunnel\"`: Pod traffic between nodes is encapsulated in an overlay tunnel (e.g., VXLAN). This is often simpler to set up as it doesn't require direct L2/L3 reachability for pod IPs between nodes beyond the tunnel endpoints.\n * `\"native\"` (Direct Routing): Pod IPs are directly routable on the underlying network. This requires the Hetzner private network (and its subnets for each node) to be configured to route traffic destined for pod CIDRs to the correct nodes. This can offer better performance by avoiding encapsulation overhead.\n * **Hetzner Context for Native Routing:** For native routing to work with Hetzner Cloud, the module (or Cilium itself, if configured appropriately) needs to manage routes within the Hetzner private network to ensure that traffic for a pod on Node B, originating from Node A, is correctly routed by Hetzner's network infrastructure to Node B. This often involves Cilium interacting with the cloud provider's routing tables or using features like Hetzner's \"Routes\" on their private networks.\n\n```terraform\n # Used when Cilium is configured in native routing mode. The CNI assumes that the underlying network stack will forward packets to this destination without the need to apply SNAT. Default: value of \"cluster_ipv4_cidr\"\n # cilium_ipv4_native_routing_cidr = \"10.0.0.0/8\"\n```\n\n* **`cilium_ipv4_native_routing_cidr` (String, Optional, relevant if `cni_plugin = \"cilium\"` and `cilium_routing_mode = \"native\"`):**\n * **Default (in module):** The value of `cluster_ipv4_cidr` (e.g., `\"10.42.0.0/16\"`).\n * **Purpose:** In Cilium's native routing mode, this tells Cilium which broader IP range encompasses all possible pod IPs across the cluster. Cilium uses this to understand which traffic should be routed directly versus potentially needing SNAT (Source Network Address Translation) for outbound traffic to destinations outside this CIDR.\n * **Typical Setting:** Often set to the overall network CIDR (e.g., `network_ipv4_cidr` like `\"10.0.0.0/8\"`) if all pod traffic within that larger network should be natively routed. If set to just `cluster_ipv4_cidr`, it implies only traffic within the pod CIDR itself is considered \"native\" by this specific Cilium setting. The exact behavior depends on Cilium's internal logic for this parameter.\n\n```terraform\n # Enables egress gateway to redirect and SNAT the traffic that leaves the cluster. Default: false\n # cilium_egress_gateway_enabled = true\n```\n\n* **`cilium_egress_gateway_enabled` (Boolean, Optional, relevant if `cni_plugin = \"cilium\"`):**\n * **Default:** `false`.\n * **Purpose:** If `true`, enables Cilium's Egress Gateway feature.\n * **Egress Gateway Role:** Allows you to route outbound traffic from specific pods (or all cluster outbound traffic) through a dedicated set of \"egress\" nodes. These egress nodes then SNAT the traffic, so all outbound connections appear to originate from the IP address(es) of these egress nodes.\n * **Use Case:**\n * Providing a stable, predictable source IP for outbound traffic for whitelisting with external services.\n * Applying common network policies or monitoring to all egress traffic.\n * **Integration:** Often used with the \"egress\" `agent_nodepool` example shown earlier, which had `floating_ip = true`. The floating IP(s) on the egress nodes become the source IP(s) for the SNAT'd traffic.\n\n```terraform\n # Enables Hubble Observability to collect and visualize network traffic. Default: false\n # cilium_hubble_enabled = true\n```\n\n* **`cilium_hubble_enabled` (Boolean, Optional, relevant if `cni_plugin = \"cilium\"`):**\n * **Default:** `false`.\n * **Purpose:** If `true`, deploys [Hubble](https://cilium.io/blog/2019/11/19/announcing-hubble/), Cilium's network observability platform.\n * **Hubble Features:**\n * Provides deep visibility into network traffic flows between pods, services, and external entities.\n * Offers a UI (Hubble UI) and CLI for exploring traffic, service dependencies, and network policy effects.\n * Can export flow logs and metrics.\n * **Impact:** Deploys Hubble components (e.g., Hubble Relay, Hubble UI pods) into the cluster.\n\n```terraform\n # Configures the list of Hubble metrics to collect.\n # cilium_hubble_metrics_enabled = [\n # \"policy:sourceContext=app|workload-name|pod|reserved-identity;destinationContext=app|workload-name|pod|dns|reserved-identity;labelsContext=source_namespace,destination_namespace\"\n # ]\n```\n\n* **`cilium_hubble_metrics_enabled` (List of Strings, Optional, relevant if `cilium_hubble_enabled = true`):**\n * **Purpose:** Specifies which types of metrics Hubble should collect and expose (typically for Prometheus consumption).\n * **Format:** A list of strings, where each string defines a metric configuration. The example shows a complex metric definition for policy-related traffic, broken down by various source/destination contexts (app, workload, pod, identity) and labeled by namespaces.\n * **Reference:** Consult the Hubble documentation for available metric types and configuration syntax.\n\n```terraform\n # Set the Cilium LoadBalancer & NodePort XDP Acceleration. Default: \"best-effort\".\n # The setting \"native\" enforces XDP Acceleration on ports and \"disabled\" disables the acceleration, \"best-effort\" enables the XDP Acceleration if the interface supports it.\n # See [Cilium XDP documentation](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#loadbalancer-nodeport-xdp-acceleration).\n # For Robot nodes connected over vSwitch, the XDP acceleration may not work on the Robot node and the setting therefore recommended to be set to \"best-effort\" or \"disabled\".\n # cilium_loadbalancer_acceleration_mode = \"best-effort\"\n```\n\n* **`cilium_loadbalancer_acceleration_mode` (String, Optional, relevant if `cni_plugin = \"cilium\"`):**\n * **Default:** `\"best-effort\"`.\n * **Purpose:** Specifies the Loadbalancer Acceleration mode for Cilium (loadBalancer.acceleration). \n\n```terraform\n # You can choose the version of Calico that you want. By default, the latest is used.\n # More info on available versions can be found at https://github.com/projectcalico/calico/releases\n # Please note that if you are getting 403s from Github, it's also useful to set the version manually. However there is rarely a need for that!\n # calico_version = \"v3.27.2\"\n```\n\n* **`calico_version` (String, Optional, relevant if `cni_plugin = \"calico\"`):**\n * **Default:** The module likely picks the latest stable version of Calico.\n * **Purpose:** Allows pinning the Calico installation to a specific version.\n * **GitHub 403s Note:** Sometimes, automated scripts fetching \"latest\" release information from GitHub can hit rate limits or encounter temporary issues. Pinning to a specific version can bypass such problems.\n\n```terraform\n # If you want to disable the k3s kube-proxy, use this flag. The default is \"false\".\n # Ensure that your CNI is capable of handling all the functionalities typically covered by kube-proxy.\n # disable_kube_proxy = true\n```\n\n* **`disable_kube_proxy` (Boolean, Optional):**\n * **Default:** `false` (k3s's embedded kube-proxy is enabled).\n * **Purpose:**\n * `false`: k3s runs its own kube-proxy component (usually a stripped-down version based on iptables or ipvs) on each node. Kube-proxy is responsible for implementing Kubernetes Services (ClusterIP, NodePort, LoadBalancer) by managing network rules (iptables, ipvs) on the nodes.\n * `true`: Disables k3s's internal kube-proxy.\n * **Requirement if `true`:** If you disable k3s's kube-proxy, your chosen CNI plugin *must* be capable of providing this service routing functionality itself.\n * **Cilium:** Can replace kube-proxy entirely using eBPF for service handling (often more efficient). This is a common reason to set `disable_kube_proxy = true` when using Cilium.\n * **Calico:** Can also work without kube-proxy in some configurations, but this needs careful setup.\n * **Flannel:** Typically relies on kube-proxy for service implementation.\n * **Benefit of Disabling (with capable CNI):** Can lead to better performance, simpler network path, and reduced overhead by having a single component (the CNI) manage all aspects of pod and service networking.\n\n```terraform\n # If you want to disable the k3s default network policy controller, use this flag!\n # Both Calico and Cilium cni_plugin values override this value to true automatically, the default is \"false\".\n # disable_network_policy = true\n```\n\n* **`disable_network_policy` (Boolean, Optional):**\n * **Default:** `false` (k3s's default network policy controller is enabled if no other CNI provides it).\n * **Purpose:**\n * `false`: If the chosen CNI (like Flannel) doesn't have its own network policy enforcement, k3s might enable a basic network policy controller.\n * `true`: Disables k3s's own network policy controller.\n * **Automatic Override:** \"Both Calico and Cilium cni_plugin values override this value to true automatically.\" This is because Calico and Cilium provide their own, more advanced network policy engines. When they are selected as the `cni_plugin`, the module ensures k3s's default (and potentially conflicting or redundant) network policy controller is disabled.\n\nLocked and loaded! Let's continue the detailed exploration.\n\n---\n\n**Section 2.18: Miscellaneous Operational Settings**\n\n```terraform\n # If you want to disable the automatic use of placement group \"spread\". See https://docs.hetzner.com/cloud/placement-groups/overview/\n # We advise to not touch that setting, unless you have a specific purpose.\n # The default is \"false\", meaning it's enabled by default.\n # placement_group_disable = true\n```\n\n* **`placement_group_disable` (Boolean, Optional):**\n * **Default:** `false` (meaning Hetzner Placement Groups with a \"spread\" strategy are *enabled* and used by the module by default).\n * **Purpose:** Controls whether the module attempts to use Hetzner Placement Groups for your cluster nodes.\n * `false`: The module will likely create one or more placement groups (with \"spread\" strategy, meaning servers in the group are on different physical hosts) and assign your nodes to them. This improves resilience against single physical host failures.\n * `true`: Disables the module's automatic use of placement groups. Servers will be provisioned without explicit placement group assignment, relying on Hetzner's default allocation.\n * **Recommendation:** \"We advise to not touch that setting, unless you have a specific purpose.\" Using placement groups is generally a good practice for HA. You might disable it if you are hitting Hetzner limits on placement groups per project, or for very specific testing scenarios.\n * **Interaction with Nodepool `placement_group`:** If a nodepool definition has its own `placement_group = \"group_name\"` attribute, that would likely take precedence for that specific nodepool, allowing for more granular control even if global placement groups are enabled.\n\n```terraform\n # By default, we allow ICMP ping in to the nodes, to check for liveness for instance. If you do not want to allow that, you can. Just set this flag to true (false by default).\n # block_icmp_ping_in = true\n```\n\n* **`block_icmp_ping_in` (Boolean, Optional):**\n * **Default:** `false` (meaning ICMP ping requests *are allowed* to the nodes by the Hetzner Firewall).\n * **Purpose:** Controls whether the Hetzner Firewall rule for ICMP (specifically echo-request, \"ping\") is configured to allow or block incoming pings to your cluster nodes.\n * `false`: Nodes will respond to pings. Useful for basic liveness checks and network troubleshooting.\n * `true`: Nodes will not respond to pings from external sources (blocked at the Hetzner Firewall level).\n * **Security Consideration:** Blocking ICMP can make it slightly harder for attackers to discover live hosts (though there are other methods). However, it also hinders legitimate network diagnostics. The security benefit is often considered minor compared to the operational inconvenience.\n\n```terraform\n # You can enable cert-manager (installed by Helm behind the scenes) with the following flag, the default is \"true\".\n # enable_cert_manager = false\n```\n\n* **`enable_cert_manager` (Boolean, Optional):**\n * **Default:** `true`.\n * **Purpose:** If `true`, deploys [cert-manager](https://cert-manager.io/) into your Kubernetes cluster.\n * **cert-manager Role:** A powerful tool for automating the management and issuance of TLS certificates within Kubernetes. It can:\n * Issue certificates from various sources like Let's Encrypt (for publicly trusted certs), Venafi, or self-signed CAs.\n * Automatically renew certificates before they expire.\n * Store certificates as Kubernetes Secrets, ready to be used by Ingress controllers, web applications, etc.\n * **Mechanism:** The module installs cert-manager using its Helm chart.\n * **If `false`:** cert-manager is not deployed. You would need to manage TLS certificates manually or use a different solution.\n * **Interaction with Rancher:** If `enable_rancher = true`, Rancher often deploys its own instance or version of cert-manager, or has specific requirements. The module might handle this interaction (e.g., not deploying a separate cert-manager if Rancher is enabled and provides it).\n\n```terraform\n # IP Addresses to use for the DNS Servers, the defaults are the ones provided by Hetzner https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/.\n # The number of different DNS servers is limited to 3 by Kubernetes itself.\n # It's always a good idea to have at least 1 IPv4 and 1 IPv6 DNS server for robustness.\n dns_servers = [\n \"1.1.1.1\", # Cloudflare Public DNS (IPv4)\n \"8.8.8.8\", # Google Public DNS (IPv4)\n \"2606:4700:4700::1111\", # Cloudflare Public DNS (IPv6)\n ]\n```\n\n* **`dns_servers` (List of Strings, Optional):**\n * **Default (in module):** Likely Hetzner's own recursive DNS servers (e.g., `185.12.64.1`, `185.12.64.2`, and their IPv6 equivalents).\n * **Purpose:** Specifies the upstream DNS servers that the nodes (and thus CoreDNS/kube-dns running in the cluster) will use for resolving external domain names. This configures the `/etc/resolv.conf` on the host nodes.\n * **Example Values:** The example shows public DNS servers from Cloudflare and Google.\n * **Kubernetes Limit:** \"The number of different DNS servers is limited to 3 by Kubernetes itself\" (actually, by the underlying Linux `resolv.conf` behavior, which typically only uses the first few).\n * **Recommendation:** Using a mix of reliable IPv4 and IPv6 DNS servers from different providers can improve DNS resolution robustness. Hetzner's own DNS servers are also a good choice as they are geographically close.\n\n---\n\n**Section 2.19: Control Plane Accessibility and Kubeconfig Options**\n\n```terraform\n # When this is enabled, rather than the first node, all external traffic will be routed via a control-plane loadbalancer, allowing for high availability.\n # The default is false.\n # use_control_plane_lb = true\n```\n\n* **`use_control_plane_lb` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** Controls how the Kubernetes API server is exposed for external access, especially in an HA control plane setup.\n * `false` (Default): The kubeconfig generated by the module might point to the IP address of the *first* control plane node, or if `kubeconfig_server_address` is set, to that address. If that first node goes down (in an HA setup), you'd need to manually update your kubeconfig to point to another live control plane node.\n * `true`: The module provisions an additional Hetzner Cloud Load Balancer specifically for the control plane nodes (on port 6443). The generated kubeconfig will then point to the IP address of this control plane LB.\n * **Benefit of `true`:** Provides a single, highly available endpoint for the Kubernetes API server. If one control plane node fails, the LB will route traffic to the remaining healthy ones.\n * **Cost Implication:** Enabling this incurs the cost of an additional Hetzner Load Balancer.\n * **Firewall Note:** As mentioned under `firewall_kube_api_source`, Hetzner LBs don't currently support firewall rules directly. Access to the control plane LB's public IP would be open, relying on Kubernetes authentication/RBAC.\n\n```terraform\n # When the above use_control_plane_lb is enabled, you can change the lb type for it, the default is \"lb11\".\n # control_plane_lb_type = \"lb21\"\n```\n\n* **`control_plane_lb_type` (String, Optional, relevant if `use_control_plane_lb = true`):**\n * **Default:** `\"lb11\"`.\n * **Purpose:** Allows you to specify the Hetzner Load Balancer type for the dedicated control plane load balancer.\n * **Consideration:** The API server traffic is usually not as high volume as application traffic, so `lb11` is often sufficient. Choose a larger type if you anticipate extremely high API load or have specific requirements.\n\n```terraform\n # When the above use_control_plane_lb is enabled, you can change to disable the public interface for control plane load balancer, the default is true.\n # control_plane_lb_enable_public_interface = false\n```\n\n* **`control_plane_lb_enable_public_interface` (Boolean, Optional, relevant if `use_control_plane_lb = true`):**\n * **Default:** `true` (meaning the control plane LB has a public IP).\n * **Purpose:**\n * `true`: The control plane LB gets a public IP, making the Kube API accessible from the internet (subject to Kubernetes authN/authZ).\n * `false`: The control plane LB only gets a private IP within the Hetzner network. The Kube API would only be accessible from within that private network (e.g., via VPN, bastion, or other servers in the same network).\n * **Use Case for `false`:** Enhanced security by not exposing the Kube API directly to the public internet, even via an LB.\n * **Integration with NAT Router:** When both `control_plane_lb_enable_public_interface = false` and `nat_router` are configured, the NAT router automatically forwards port 6443 to the control plane LB's private IP. This allows external kubectl access via the NAT router's public IP while keeping the control plane LB private. The generated kubeconfig will automatically use the NAT router's public IP as the server address.\n\n```terraform\n # Let's say you are not using the control plane LB solution above, and still want to have one hostname point to all your control-plane nodes.\n # You could create multiple A records of to let's say cp.cluster.my.org pointing to all of your control-plane nodes ips.\n # In which case, you need to define that hostname in the k3s TLS-SANs config to allow connection through it. It can be hostnames or IP addresses.\n # additional_tls_sans = [\"cp.cluster.my.org\"]\n```\n\n* **`additional_tls_sans` (List of Strings, Optional):**\n * **Purpose:** Allows you to add extra Subject Alternative Names (SANs) to the TLS certificate generated by k3s for its API server.\n * **Use Case (DNS Round Robin for API):** If you are *not* using `use_control_plane_lb = true` but want a single hostname for your API server (e.g., `cp.cluster.my.org`), you might create multiple A/AAAA DNS records for this hostname, each pointing to the public IP of one of your control plane nodes (DNS Round Robin).\n * For clients to connect to `https://cp.cluster.my.org:6443` without TLS certificate errors, this hostname *must* be listed as a SAN in the API server's certificate.\n * **Format:** A list of hostnames or IP addresses.\n * **Impact:** k3s will include these in its self-signed certificate for the API, or if integrating with an external CA, these SANs would be requested.\n\n```terraform\n # If you create a hostname with multiple A records pointing to all of your\n # control-plane nodes ips, you may want to use that hostname in the generated\n # kubeconfig.\n # kubeconfig_server_address = \"cp.cluster.my.org\"\n```\n\n* **`kubeconfig_server_address` (String, Optional):**\n * **Purpose:** Allows you to explicitly set the server address (hostname or IP) that will be written into the `server:` field of the generated kubeconfig file.\n * **Default Behavior:** Without this, the kubeconfig will automatically point to:\n * The public IP of the control plane LB (if `use_control_plane_lb = true` and `control_plane_lb_enable_public_interface = true`).\n * The public IP of the NAT router (if `use_control_plane_lb = true`, `control_plane_lb_enable_public_interface = false`, and `nat_router` is configured).\n * The private IP of the control plane LB (if `use_control_plane_lb = true`, `control_plane_lb_enable_public_interface = false`, and no `nat_router`).\n * The IP of the first control plane node (if no CP LB).\n * **Use Case:** If you've set up DNS Round Robin for your control plane nodes (as described for `additional_tls_sans`) and want your kubeconfig to use that hostname (e.g., `cp.cluster.my.org`) instead of a direct IP, or if you have a custom ingress setup.\n * **Requirement:** If you use a hostname here, ensure it resolves correctly and is included in the API server's TLS certificate SANs (via `additional_tls_sans` or default k3s behavior).\n\n```terraform\n # Optional external control plane endpoint URL (e.g. https://myapi.domain.com:6443).\n # Used as the k3s 'server' value for agents and secondary control planes.\n # control_plane_endpoint = \"https://myapi.domain.com:6443\"\n```\n\n* **`control_plane_endpoint` (String, Optional):**\n * **Default:** `null`.\n * **Purpose:** Specifies a custom external URL for the Kubernetes API server.\n * **Use Case:** When using an external load balancer (not managed by this module) or a specific DNS alias for your control plane, set this to ensure agents register correctly against that endpoint.\n\n```terraform\n # K3S audit-policy.yaml contents. Used to configure Kubernetes audit logging.\n # k3s_audit_policy_config = <<-EOT\n # apiVersion: audit.k8s.io/v1\n # kind: Policy\n # rules:\n # - level: Metadata\n # EOT\n # k3s_audit_log_path = \"/var/log/k3s-audit/audit.log\"\n # k3s_audit_log_maxage = 30\n # k3s_audit_log_maxbackup = 10\n # k3s_audit_log_maxsize = 100\n```\n\n* **`k3s_audit_policy_config` (String, Optional):**\n * **Purpose:** Defines the Kubernetes Audit Policy. If provided, k3s is configured to log audit events matching these rules.\n * **Format:** YAML string content for the policy file.\n* **`k3s_audit_log_*` variables:**\n * **Purpose:** Configure the rotation and retention of audit logs on control plane nodes.\n * `k3s_audit_log_path`: Path to audit log file (default: `/var/log/k3s-audit/audit.log`)\n * `k3s_audit_log_maxage`: Days to retain logs (default: `30`)\n * `k3s_audit_log_maxbackup`: Number of backup files to keep (default: `10`)\n * `k3s_audit_log_maxsize`: Max size in MB before rotation (default: `100`)\n\n```terraform\n # lb_hostname Configuration:\n #\n # Purpose:\n # The lb_hostname setting optimizes communication between services within the Kubernetes cluster\n # when they use domain names instead of direct service names. By associating a domain name directly\n # with the Hetzner Load Balancer, this setting can help reduce potential communication delays.\n #\n # Scenario:\n # If Service B communicates with Service A using a domain (e.g., `a.mycluster.domain.com`) that points\n # to an external Load Balancer, there can be a slowdown in communication.\n #\n # Guidance:\n # - If your internal services use domain names pointing to an external LB, set lb_hostname to a domain\n # like `mycluster.domain.com`.\n # - Create an A record pointing `mycluster.domain.com` to your LB's IP.\n # - Create a CNAME record for `a.mycluster.domain.com` (or xyz.com) pointing to `mycluster.domain.com`.\n #\n # Technical Note:\n # This setting sets the `load-balancer.hetzner.cloud/hostname` in the Hetzner LB definition, suitable for\n # HAProxy, Nginx and Traefik ingress controllers.\n #\n # Recommendation:\n # This setting is optional. If services communicate using direct service names, you can leave this unset.\n # For inter-namespace communication, use `.service_name` as per Kubernetes norms.\n #\n # Example:\n # lb_hostname = \"mycluster.domain.com\"\n```\n\n* **`lb_hostname` (String, Optional):**\n * **Purpose:** Sets the `load-balancer.hetzner.cloud/hostname` annotation on the Service object for your main Ingress controller (Traefik, Nginx, HAProxy).\n * **Hetzner CCM Behavior:** When the Hetzner Cloud Controller Manager (CCM) sees this annotation on a Service of type `LoadBalancer`, it attempts to associate the specified hostname with the provisioned Hetzner Load Balancer. This might involve creating/updating DNS records if you use Hetzner DNS, or it might just be informational for the LB itself.\n * **Scenario Explained:** The comment describes a scenario where internal cluster services communicate via external domain names that resolve to the main Hetzner LB. If Service B calls `a.mycluster.domain.com`, and this resolves to the LB IP, the traffic goes out to the LB and then back into the cluster to Service A. This can be inefficient (\"hairpinning\" or \"NAT loopback\" issues if not handled well).\n * **Optimization Goal:** By setting `lb_hostname`, the CCM might optimize this path, or it might simply ensure that if you CNAME your application hostnames (like `a.mycluster.domain.com`) to this `lb_hostname` (which itself points to the LB IP), the LB is aware of the primary domain it serves. The exact optimization depends on Hetzner CCM's capabilities.\n * **DNS Setup:** You are still responsible for:\n 1. Creating an A/AAAA record for `lb_hostname` (e.g., `mycluster.domain.com`) pointing to the Hetzner Load Balancer's public IP.\n 2. Creating CNAME records for your individual application services (e.g., `a.mycluster.domain.com` CNAME to `mycluster.domain.com`).\n * **Recommendation:** Optional. If your services primarily use internal Kubernetes service discovery (e.g., `service-a.namespace.svc.cluster.local`), this might not be necessary.\n\n---\n\n**Section 2.20: Rancher Integration**\n\n```terraform\n # You can enable Rancher (installed by Helm behind the scenes) with the following flag, the default is \"false\".\n # ⚠️ Rancher often doesn't support the latest Kubernetes version. You will need to set initial_k3s_channel to a supported version.\n # When Rancher is enabled, it automatically installs cert-manager too, and it uses rancher's own self-signed certificates.\n # See for options https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster#3-choose-your-ssl-configuration\n # The easiest thing is to leave everything as is (using the default rancher self-signed certificate) and put Cloudflare in front of it.\n # As for the number of replicas, by default it is set to the number of control plane nodes.\n # You can customized all of the above by adding a rancher_values variable see at the end of this file in the advanced section.\n # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.\n # IMPORTANT: Rancher's install is quite memory intensive, you will require at least 4GB if RAM, meaning cx23 server type (for your control plane).\n # ALSO, in order for Rancher to successfully deploy, you have to set the \"rancher_hostname\".\n # enable_rancher = true\n```\n\n* **`enable_rancher` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** If `true`, deploys [Rancher Manager](https://rancher.com/), a popular open-source platform for managing multiple Kubernetes clusters.\n * **Mechanism:** The module installs Rancher using its Helm chart.\n * **Key Considerations & Warnings:**\n * **Kubernetes Version Compatibility (⚠️):** Rancher has strict Kubernetes version compatibility. You *must* set `initial_k3s_channel` or `install_k3s_version` to a version supported by the Rancher version being installed. This often means using a slightly older, well-tested Kubernetes minor version.\n * **Cert-Manager:** Rancher typically bundles or requires its own instance of cert-manager. It often uses its own self-signed CA by default for its UI. The comment suggests that if `enable_rancher` is true, the module's `enable_cert_manager` might be implicitly handled or overridden.\n * **SSL Configuration:** Rancher offers options for SSL: Rancher-generated self-signed certs (default), Let's Encrypt, or bringing your own certs. The comment suggests the default self-signed cert is easiest if you put a proxy like Cloudflare (with its own valid cert) in front of Rancher.\n * **Replicas:** Rancher deployment replicas default to the number of control plane nodes for HA.\n * **Customization:** Advanced customization via `rancher_values` block (later).\n * **Resource Requirements (IMPORTANT):** Rancher is resource-intensive. Control plane nodes need significant RAM (at least 4GB, e.g., Hetzner `cx23`/`cx33` or higher). Insufficient resources will lead to installation failures or instability.\n * **`rancher_hostname` (REQUIRED):** You *must* set `rancher_hostname` if `enable_rancher = true`.\n\n```terraform\n # If using Rancher you can set the Rancher hostname, it must be unique hostname even if you do not use it.\n # If not pointing the DNS, you can just port-forward locally via kubectl to get access to the dashboard.\n # If you already set the lb_hostname above and are using a Hetzner LB, you do not need to set this one, as it will be used by default.\n # But if you set this one explicitly, it will have preference over the lb_hostname in rancher settings.\n # rancher_hostname = \"rancher.xyz.dev\"\n```\n\n* **`rancher_hostname` (String, Conditional Obligatory if `enable_rancher = true`):**\n * **Purpose:** Sets the hostname that Rancher will be configured to use for its UI and API. This hostname is embedded in Rancher's configuration and its TLS certificates.\n * **Requirement:** Must be set if `enable_rancher = true`.\n * **DNS:** You need to create a DNS A/AAAA record for this hostname pointing to the IP address where Rancher is accessible (e.g., the Hetzner Load Balancer IP, or a node IP if using Klipper LB).\n * **Local Access (No DNS):** If you don't set up public DNS, you can still access the Rancher UI by port-forwarding the Rancher service locally using `kubectl port-forward svc/rancher -n cattle-system <local_port>:443` and then accessing `https://localhost:<local_port>` (after adding the hostname to your local `/etc/hosts` file pointing to `127.0.0.1`).\n * **Interaction with `lb_hostname`:**\n * If `lb_hostname` is set and `rancher_hostname` is *not*, Rancher will default to using `lb_hostname`.\n * If `rancher_hostname` is explicitly set, it takes precedence for Rancher's configuration, even if `lb_hostname` is also set.\n\n```terraform\n # When Rancher is deployed, by default is uses the \"latest\" channel. But this can be customized.\n # The allowed values are \"stable\" or \"latest\".\n # rancher_install_channel = \"stable\"\n```\n\n* **`rancher_install_channel` (String, Optional, relevant if `enable_rancher = true`):**\n * **Default (in Rancher Helm chart):** Often `\"latest\"`.\n * **Purpose:** Specifies the Rancher release channel to use when installing Rancher via its Helm chart.\n * `\"latest\"`: Installs the most recent Rancher version, which might include newer features but could be less tested.\n * `\"stable\"`: Installs the version of Rancher marked as stable, generally recommended for production.\n * **Note:** This refers to the Rancher *application* version channel, distinct from the `initial_k3s_channel` for the Kubernetes version.\n\n```terraform\n # Finally, you can specify a bootstrap-password for your rancher instance. Minimum 48 characters long!\n # If you leave empty, one will be generated for you.\n # (Can be used by another rancher2 provider to continue setup of rancher outside this module.)\n # rancher_bootstrap_password = \"\"\n```\n\n* **`rancher_bootstrap_password` (String, Optional, Sensitive, relevant if `enable_rancher = true`):**\n * **Purpose:** Sets the initial bootstrap password for the default `admin` user in Rancher.\n * **Default:** If left empty or not provided, Rancher (or the module) will generate a random password. This password can usually be retrieved from a Kubernetes secret or logs after installation.\n * **Minimum Length:** The comment \"Minimum 48 characters long!\" indicates a strong recommendation or requirement from Rancher or the module for security.\n * **Automation Use:** \"Can be used by another rancher2 provider...\" If you're automating further Rancher configuration using the `rancher2` Terraform provider, knowing or setting this bootstrap password allows that provider to authenticate to the newly installed Rancher instance.\n * **Security:** If setting this, treat it as highly sensitive.\n\n```terraform\n # Separate from the above Rancher config (only use one or the other). You can import this cluster directly on an\n # an already active Rancher install. By clicking \"import cluster\" choosing \"generic\", giving it a name and pasting\n # the cluster registration url below. However, you can also ignore that and apply the url via kubectl as instructed\n # by Rancher in the wizard, and that would register your cluster too.\n # More information about the registration can be found here https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/\n # rancher_registration_manifest_url = \"https://rancher.xyz.dev/v3/import/xxxxxxxxxxxxxxxxxxYYYYYYYYYYYYYYYYYYYzzzzzzzzzzzzzzzzzzzzz.yaml\"\n```\n\n* **`rancher_registration_manifest_url` (String, Optional):**\n * **Purpose:** Used to register this newly created k3s cluster with an *existing, separate* Rancher Manager instance. This is an alternative to `enable_rancher = true` (which installs Rancher *within* this cluster).\n * **Mechanism:**\n 1. In your existing Rancher UI, you would go to \"Import Cluster,\" choose \"Generic,\" and Rancher will provide a registration command, often including a URL to a YAML manifest.\n 2. You paste that manifest URL here.\n 3. The module will then apply this manifest to your k3s cluster. The manifest typically deploys Rancher cluster agents, which connect back to your existing Rancher Manager and register the cluster.\n * **Alternative:** As the comment notes, you can also just run the `kubectl apply -f <url>` command manually after the cluster is up.\n\n\n\n---\n\n**Section 2.21: Kustomize and Post-Deployment Operations**\n\n```terraform\n # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).\n # extra_kustomize_deployment_commands=\"\"\n```\n\n* **`extra_kustomize_deployment_commands` (String or List of Strings, Optional):**\n * **Purpose:** Allows you to specify shell commands that will be executed *after* the module has run its main Kustomize deployment (which applies manifests for core components like CCM, CSI, Ingress, etc., based on your selections).\n * **Mechanism:** The module likely uses a `local-exec` or `remote-exec` provisioner (if commands need to run on a node) to execute these. If they are `kubectl` commands, they'd run from where Terraform is executed, using the generated kubeconfig.\n * **Use Cases:**\n * **Waiting for CRDs:** Some applications deployed via Helm or Kustomize install CustomResourceDefinitions (CRDs) first, and then CustomResources (CRs) that depend on those CRDs. There can be a race condition if the CRs are applied before the CRDs are fully registered. You could add a command here to wait for CRDs to become available (e.g., `kubectl wait --for condition=established crd/mycrd.example.com --timeout=120s`).\n * Applying additional Kubernetes manifests that depend on the core setup.\n * Running post-install scripts or triggering initial application setup jobs.\n * **Format:** Can be a single string with commands separated by `&&` or `\\n`, or a list of individual command strings.\n\n```terraform\n # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.\n # extra_kustomize_parameters={}\n```\n\n* **`extra_kustomize_parameters` (Map of Strings, Optional):**\n * **Purpose:** If you are using the module's \"extra manifests\" feature (where you can provide your own Kustomize setup in an `extra-manifests` directory), this map allows you to pass key-value parameters into a `kustomization.yaml.tpl` template file within that directory.\n * **Mechanism:** The module would process `extra-manifests/kustomization.yaml.tpl` as a template, substituting placeholders with values from this map, and then run `kustomize build` on the result.\n * **Use Case:** Parameterizing your custom Kustomize deployments based on Terraform inputs or computed values from the `kube-hetzner` module (e.g., passing in the cluster name, node IPs, etc., to your custom manifests).\n * **Reference:** The comment points to examples in the module's repository for how to use this feature.\n\n```terraform\n # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md\n```\n\n* **Documentation Pointer:** This directs users to example usage of the \"extra manifests\" feature, which is crucial for extending the module's capabilities with custom deployments.\n\n---\n\n**Section 2.22: Kubeconfig and Output Management**\n\n```terraform\n # It is best practice to turn this off, but for backwards compatibility it is set to \"true\" by default.\n # See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/349\n # When \"false\". The kubeconfig file can instead be created by executing: \"terraform output --raw kubeconfig > cluster_kubeconfig.yaml\"\n # Always be careful to not commit this file!\n # create_kubeconfig = false\n```\n\n* **`create_kubeconfig` (Boolean, Optional):**\n * **Default (in module, historically):** `true`. The comment suggests this might be changing or that `false` is now best practice.\n * **Purpose:**\n * `true`: The module, after setting up the cluster, will attempt to write the generated kubeconfig content to a local file (e.g., `cluster_kubeconfig.yaml` in the current directory).\n * `false`: The module does *not* write the kubeconfig to a file. You *must* retrieve it using `terraform output --raw kubeconfig > cluster_kubeconfig.yaml`.\n * **Best Practice (`false`):**\n * Avoids accidentally committing the sensitive kubeconfig file to version control if it's automatically created in the project directory.\n * Makes the user more deliberate about handling the kubeconfig.\n * **Issue #349:** Refers to a discussion likely about the security implications and best practices around kubeconfig generation.\n\n```terraform\n # Don't create the kustomize backup. This can be helpful for automation.\n # create_kustomization = false\n```\n\n* **`create_kustomization` (Boolean, Optional):**\n * **Default:** `true` (implied, as backups are usually made).\n * **Purpose:** The module internally uses Kustomize to generate and apply many of its Kubernetes manifests. It might create a backup of the generated Kustomize directory or files.\n * `true`: Creates this backup.\n * `false`: Skips creating the Kustomize backup.\n * **Use Case for `false`:** In CI/CD or fully automated environments, these backup files might be unnecessary clutter or could interfere with cleanup processes.\n\n```terraform\n # Export the values.yaml files used for the deployment of traefik, longhorn, cert-manager, etc.\n # This can be helpful to use them for later deployments like with ArgoCD.\n # The default is false.\n # export_values = true\n```\n\n* **`export_values` (Boolean, Optional):**\n * **Default:** `false`.\n * **Purpose:** If `true`, the module will output or save the effective `values.yaml` files that were used for deploying Helm charts like Traefik, Longhorn, cert-manager, etc.\n * **Benefit:**\n * **Debugging:** Allows you to see exactly what configuration was passed to each Helm chart.\n * **GitOps/ArgoCD:** You can take these exported values files and commit them to a Git repository to manage these applications via ArgoCD or a similar GitOps tool, ensuring consistency between the Terraform-managed deployment and subsequent GitOps management. This facilitates a transition or co-existence strategy.\n\n---\n\n**Section 2.23: Base OS Image Configuration**\n\n```terraform\n # MicroOS snapshot IDs to be used. Per default empty, the most recent image created using createkh will be used.\n # We recommend the default, but if you want to use specific IDs you can.\n # You can fetch the ids with the hcloud cli by running the \"hcloud image list --selector 'microos-snapshot=yes'\" command.\n # microos_x86_snapshot_id = \"1234567\"\n # microos_arm_snapshot_id = \"1234567\"\n```\n\n* **Background:** This module uses openSUSE MicroOS as the base operating system for the cluster nodes. MicroOS is a transactional, immutable-style OS designed for container workloads. The `createkh` tool (mentioned in the comment, part of the `kube-hetzner` project) is likely used to prepare and snapshot customized MicroOS images suitable for this module.\n* **`microos_x86_snapshot_id` (String, Optional):**\n * **Default:** Empty string (module uses the most recent `createkh`-generated x86 snapshot).\n * **Purpose:** Allows you to specify the exact Hetzner snapshot ID for the openSUSE MicroOS image to be used for x86-based nodes (e.g., `cx` series).\n* **`microos_arm_snapshot_id` (String, Optional):**\n * **Default:** Empty string (module uses the most recent `createkh`-generated ARM snapshot).\n * **Purpose:** Allows you to specify the exact Hetzner snapshot ID for the openSUSE MicroOS image to be used for ARM-based nodes (e.g., `cax` series).\n* **Recommendation:** \"We recommend the default\". Using the default ensures you get the latest tested and prepared image from the module maintainers.\n* **Use Case for Pinning:**\n * Ensuring absolute reproducibility if you need to rebuild a cluster exactly as it was.\n * If a new default snapshot introduces an issue, you can temporarily pin to a known good older snapshot ID.\n* **Fetching IDs:** The `hcloud image list --selector 'microos-snapshot=yes'` command helps you find available snapshot IDs created by `createkh` in your Hetzner project.\n\n---\n\n**Section 2.24: ADVANCED - Custom Helm Values Overrides**\n\nThis section introduces the mechanism for providing detailed, custom Helm chart values for various components deployed by the module. The general pattern is:\n\n* A variable named `component_values` (e.g., `cilium_values`, `traefik_values`).\n* The value can be a multi-line heredoc string containing YAML, or loaded from a file using `file(\"component-values.yaml\")`.\n* These values will be merged with or override the module's default Helm values for that component.\n* **Warning:** \"We advise you to use the default values, and only change them if you know what you are doing!\" Incorrect Helm values can easily break a component's deployment. Always refer to the official Helm chart documentation for the component in question.\n\n```terraform\n ### ADVANCED - Custom helm values for packages above (search _values if you want to located where those are mentioned upper in this file)\n # ⚠️ Inside the _values variable below are examples, up to you to find out the best helm values possible, we do not provide support for customized helm values.\n # Please understand that the indentation is very important, inside the EOTs, as those are proper yaml helm values.\n # We advise you to use the default values, and only change them if you know what you are doing!\n\n # You can inline the values here in heredoc-style (as the examples below with the <<-EOT to EOT). Please note that the indentation inside the EOT is important.\n # Or you can create a thepackage-values.yaml file with the content and use it here with the following syntax:\n # thepackage_values = file(\"thepackage-values.yaml\")\n\n # Cilium, all Cilium helm values can be found at https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml\n # Be careful when maintaining your own cilium_values, as the choice of available settings depends on the Cilium version used. See also the cilium_version setting to fix a specific version.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* cilium_values = <<-EOT\nipam:\n mode: kubernetes # Use Kubernetes host-scope IPAM\nk8s:\n requireIPv4PodCIDR: true # Ensure pod CIDRs are available\nkubeProxyReplacement: true # Cilium replaces kube-proxy\nroutingMode: native # Use native routing (direct routing)\nipv4NativeRoutingCIDR: \"10.0.0.0/8\" # Broader CIDR for native routing\nendpointRoutes:\n enabled: true # Manage routes for local endpoints\nloadBalancer:\n acceleration: native # Use eBPF for LB acceleration\nbpf:\n masquerade: true # Enable eBPF-based masquerading (SNAT)\nencryption:\n enabled: true # Enable transparent encryption\n type: wireguard # Use WireGuard for encryption\nMTU: 1450 # Set MTU, important for tunnels/encapsulation\n EOT */\n```\n\n* **`cilium_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for the Cilium deployment if `cni_plugin = \"cilium\"`.\n * The example shows various advanced Cilium settings:\n * `ipam.mode: kubernetes`: Cilium uses Kubernetes for IP address management.\n * `kubeProxyReplacement: true`: Cilium fully replaces kube-proxy functionality.\n * `routingMode: native`: Enables direct routing.\n * `encryption.enabled: true`, `encryption.type: wireguard`: Enables WireGuard encryption (overriding the simpler `enable_wireguard` flag if both are used, as `cilium_values` takes precedence for Cilium).\n * `MTU`: Setting the Maximum Transmission Unit is critical when using tunneling or encryption to avoid fragmentation.\n\n```terraform\n # Cert manager, all cert-manager helm values can be found at https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml\n # The following is an example, please note that the current indentation inside the EOT is important.\n # For cert-manager versions < v1.15.0, you need to set installCRDs: true instead of crds.enabled and crds.keep.\n /* cert_manager_values = <<-EOT\ncrds:\n enabled: true # Helm chart should manage CRDs (newer cert-manager versions)\n keep: true # Do not delete CRDs when Helm chart is uninstalled\nreplicaCount: 3 # Number of replicas for the main cert-manager controller\nwebhook:\n replicaCount: 3 # Replicas for the webhook component\ncainjector:\n replicaCount: 3 # Replicas for the CA injector component\n EOT */\n```\n\n* **`cert_manager_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for the cert-manager deployment if `enable_cert_manager = true` (and not overridden by Rancher).\n * Example shows:\n * `crds.enabled: true`, `crds.keep: true`: Modern way to manage CRD installation with Helm. The comment about `installCRDs: true` for older versions is important.\n * Setting `replicaCount` for various cert-manager components for HA.\n\n```terraform\n # csi-driver-smb, all csi-driver-smb helm values can be found at https://github.com/kubernetes-csi/csi-driver-smb/blob/master/charts/latest/csi-driver-smb/values.yaml\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* csi_driver_smb_values = <<-EOT\ncontroller:\n name: csi-smb-controller\n replicas: 1\n runOnMaster: false # Do not run controller on master nodes (old terminology)\n runOnControlPlane: false # Prefer not to run controller on control plane nodes\n resources: # Resource requests/limits for controller components\n csiProvisioner:\n limits:\n memory: 300Mi\n requests:\n cpu: 10m\n memory: 20Mi\n # ... similar for livenessProbe and smb sidecar ...\n EOT */\n```\n\n* **`csi_driver_smb_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for the CSI SMB driver if `enable_csi_driver_smb = true`.\n * Example shows configuring replica counts, node affinity (`runOnControlPlane`), and resource requests/limits for the driver's controller pod and its sidecars.\n\n```terraform\n # Longhorn, all Longhorn helm values can be found at https://github.com/longhorn/longhorn/blob/master/chart/values.yaml\n # longhorn_values replaces module defaults. Prefer longhorn_merge_values for targeted overrides.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* longhorn_values = <<-EOT\ndefaultSettings:\n defaultDataPath: /var/longhorn # Path on nodes where Longhorn stores data\npersistence:\n defaultFsType: ext4 # Filesystem for Longhorn volumes\n defaultClassReplicaCount: 3 # Default replica count for new volumes\n defaultClass: true # Make Longhorn's StorageClass the default\n EOT */\n```\n\n* **`longhorn_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for the Longhorn deployment if `enable_longhorn = true`.\n * Replaces the module's default Longhorn values.\n * Example shows setting default data path, filesystem type, replica count, and whether Longhorn's StorageClass should be the cluster-wide default.\n\n```terraform\n # Merge specific keys without replacing all defaults (recommended for hotfix image tags).\n /* longhorn_merge_values = <<-EOT\nimage:\n longhorn:\n manager:\n tag: v1.11.0-hotfix-1\n instanceManager:\n tag: v1.11.0-hotfix-1\n EOT */\n```\n\n* **`longhorn_merge_values` (String, Optional, Heredoc/File Content):**\n * Deep-merges your YAML on top of defaults (or on top of `longhorn_values` if set).\n * Recommended when you only need to override a subset of values, such as specific image tags.\n\n```terraform\n # If you want to use a specific Traefik helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # See https://github.com/traefik/traefik-helm-chart/releases for the available versions.\n # traefik_version = \"\"\n```\n\n* **`traefik_version` (String, Optional, specific to `ingress_controller = \"traefik\"`):**\n * **Purpose:** Allows pinning the Traefik *Helm chart version* itself, distinct from `traefik_image_tag` which pins the container image version. Helm chart versions can change structure, available values, etc., independently of the application image version.\n\n```terraform\n # Traefik, all Traefik helm values can be found at https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* traefik_values = <<-EOT\ndeployment:\n replicas: 1 # Override default replica count logic\nadditionalArguments: [] # Can add global static config args here too\nservice:\n enabled: true\n type: LoadBalancer # Ensure service is of type LoadBalancer\n annotations: # Annotations for the Hetzner Load Balancer\n \"load-balancer.hetzner.cloud/name\": \"k3s\" # Name for the LB in Hetzner console\n \"load-balancer.hetzner.cloud/use-private-ip\": \"true\" # LB uses private IP to connect to nodes\n \"load-balancer.hetzner.cloud/disable-private-ingress\": \"true\" # Disallow private network access to LB itself? (Check Hetzner docs for exact meaning)\n \"load-balancer.hetzner.cloud/location\": \"nbg1\" # Override LB location\n \"load-balancer.hetzner.cloud/type\": \"lb11\" # Override LB type\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"true\" # Enable PROXY protocol from LB to Traefik\n\nports: # Configure Traefik entrypoints\n web:\n http:\n redirections: # Redirect HTTP (web) to HTTPS (websecure)\n entryPoint:\n to: websecure\n scheme: https\n permanent: true\n\n proxyProtocol: # Configure PROXY protocol for web entrypoint\n trustedIPs:\n - 127.0.0.1/32 # Trust localhost\n - 10.0.0.0/8 # Trust private network IPs (e.g., from Hetzner LB)\n forwardedHeaders: # Configure trusted IPs for X-Forwarded-* headers\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n websecure: # Similar PROXY protocol and forwardedHeaders for HTTPS entrypoint\n proxyProtocol:\n trustedIPs:\n # ...\n forwardedHeaders:\n trustedIPs:\n # ...\n EOT */\n```\n\n* **`traefik_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for Traefik if `ingress_controller = \"traefik\"`.\n * The example is rich:\n * Setting replica count.\n * Crucially, setting Hetzner Load Balancer annotations directly on the Traefik service. This allows fine-grained control over the LB created by the Hetzner CCM for Traefik (name, location, type, private IP usage, PROXY protocol). This might override or supplement the global `load_balancer_*` settings if they apply to the Ingress LB.\n * Configuring HTTP to HTTPS redirection.\n * Configuring PROXY protocol and trusted IPs for `X-Forwarded-*` headers, essential when Traefik is behind the Hetzner LB (especially if PROXY protocol is enabled on the LB).\n\n```terraform\n # If you want to use a specific Nginx helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # See https://github.com/kubernetes/ingress-nginx?tab=readme-ov-file#supported-versions-table for the available versions.\n # nginx_version = \"\"\n```\n\n* **`nginx_version` (String, Optional, specific to `ingress_controller = \"nginx\"`):**\n * Pins the Ingress-NGINX *Helm chart version*.\n\n```terraform\n # Nginx, all Nginx helm values can be found at https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml\n # You can also have a look at https://kubernetes.github.io/ingress-nginx/, to understand how it works, and all the options at your disposal.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* nginx_values = <<-EOT\ncontroller:\n watchIngressWithoutClass: \"true\" # Watch Ingresses without an ingressClassName\n kind: \"DaemonSet\" # Deploy controller as a DaemonSet (one pod per node)\n config: # Pass custom Nginx config options\n \"use-forwarded-headers\": \"true\" # Trust X-Forwarded-* headers\n \"compute-full-forwarded-for\": \"true\" # Ensure X-Forwarded-For is accurate\n \"use-proxy-protocol\": \"true\" # Enable PROXY protocol listener\n service:\n annotations: # Annotations for the Hetzner LB for Nginx service\n \"load-balancer.hetzner.cloud/name\": \"k3s\"\n # ... other Hetzner LB annotations similar to Traefik example ...\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"true\"\n EOT */\n```\n\n* **`nginx_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for Ingress-NGINX if `ingress_controller = \"nginx\"`.\n * Example shows:\n * `watchIngressWithoutClass`: Useful if you have Ingress objects without `ingressClassName` specified.\n * `kind: \"DaemonSet\"`: Deploys Nginx controller pods on every (agent) node. Alternative is `Deployment`.\n * `config`: A map for passing Nginx-specific configurations (like `use-forwarded-headers`, `use-proxy-protocol`).\n * `service.annotations`: Similar to Traefik, for configuring the Hetzner LB for the Nginx service.\n\n```terraform\n # If you want to use a specific HAProxy helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # haproxy_version = \"\"\n```\n\n* **`haproxy_version` (String, Optional, specific to `ingress_controller = \"haproxy\"`):**\n * Pins the HAProxy Ingress *Helm chart version*.\n\n```terraform\n # If you want to configure additional proxy protocol trusted IPs for haproxy, enter them here as a list of IPs (strings).\n # Example for Cloudflare:\n # haproxy_additional_proxy_protocol_ips = [\n # \"173.245.48.0/20\",\n # // ... more Cloudflare IP ranges ...\n # ]\n```\n\n* **`haproxy_additional_proxy_protocol_ips` (List of Strings, Optional, specific to `ingress_controller = \"haproxy\"`):**\n * **Purpose:** Similar to `traefik_additional_trusted_ips`, this configures trusted source IPs for PROXY protocol when using HAProxy Ingress. If HAProxy receives PROXY protocol headers from these IPs, it will trust the client IP information within.\n * **Use Case:** When HAProxy is behind another proxy (like Cloudflare or the Hetzner LB using PROXY protocol).\n\n```terraform\n # Configure CPU and memory requests for each HAProxy pod\n # haproxy_requests_cpu = \"250m\"\n # haproxy_requests_memory = \"400Mi\"\n```\n\n* **`haproxy_requests_cpu` / `haproxy_requests_memory` (String, Optional, specific to `ingress_controller = \"haproxy\"`):**\n * **Purpose:** Sets default CPU and memory *requests* for HAProxy Ingress controller pods.\n * **Note:** These are just requests. Limits might be set separately via `haproxy_values` if needed.\n\n```terraform\n # Override values given to the HAProxy helm chart.\n # All HAProxy helm values can be found at https://github.com/haproxytech/helm-charts/blob/main/kubernetes-ingress/values.yaml\n # Default values can be found at https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/locals.tf\n /* haproxy_values = <<-EOT\n EOT */\n```\n\n* **`haproxy_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for HAProxy Ingress if `ingress_controller = \"haproxy\"`.\n * The example is empty, but you would populate it with HAProxy Ingress Helm chart values. The links point to the official chart values and potentially the module's own default values for HAProxy.\n\n```terraform\n # Rancher, all Rancher helm values can be found at https://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* rancher_values = <<-EOT\ningress:\n tls:\n source: \"rancher\" # Use Rancher's self-signed certs for Ingress\nhostname: \"rancher.example.com\" # Must match rancher_hostname\nreplicas: 1 # Override default replica count\nbootstrapPassword: \"supermario\" # Set bootstrap password (sensitive!)\n EOT */\n```\n\n* **`rancher_values` (String, Optional, Heredoc/File Content):**\n * Provides custom Helm values for the Rancher Manager deployment if `enable_rancher = true`.\n * Example shows:\n * `ingress.tls.source: \"rancher\"`: Tells Rancher's Ingress to use certificates managed by Rancher itself (often self-signed initially).\n * `hostname`: Must match the `rancher_hostname` variable.\n * `replicas`: Overrides the default replica count for Rancher pods.\n * `bootstrapPassword`: Sets the initial admin password (same as `rancher_bootstrap_password` variable, but this would be the Helm way to set it).\n * **Reference:** The Rancher chart options documentation is key.\n\n```terraform\n} # End of module \"kube-hetzner\"\n```\n\n---\n\n**Section 3: Provider and Terraform Block**\n\n```terraform\nprovider \"hcloud\" {\n token = var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token\n}\n\nterraform {\n required_version = \">= 1.10.1\"\n required_providers {\n hcloud = {\n source = \"hetznercloud/hcloud\"\n version = \">= 1.51.0\"\n }\n }\n}\n```\n\n* **`provider \"hcloud\"` Block:**\n * **Purpose:** Configures the Hetzner Cloud provider for Terraform. This is what allows Terraform to interact with the Hetzner API.\n * **`token`:** The Hetzner API token. The logic `var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token` is the same as used for the module input: it prioritizes the `TF_VAR_hcloud_token` environment variable, falling back to the `local.hcloud_token` if the environment variable is not set. This provider block is what the `kube-hetzner` module instance inherits when we pass `hcloud = hcloud` in its `providers` map.\n* **`terraform` Block:**\n * **`required_version`:** Specifies the minimum Terraform CLI version required to apply this configuration.\n * **`required_providers`:** Declares the providers needed by this root module and their source/version constraints.\n * `hcloud`: Specifies the official Hetzner Cloud provider from `hetznercloud/hcloud` on the Terraform Registry.\n * `version = \">= 1.51.0\"`: Constrains to use version 1.51.0 or newer of the Hetzner provider. It's good practice to use a lower bound and periodically update to newer provider versions for new features and bug fixes.\n\n---\n\n**Section 4: Outputs**\n\n```terraform\noutput \"kubeconfig\" {\n value = module.kube-hetzner.kubeconfig\n sensitive = true\n}\n\n# (The original file only had kubeconfig output, but I added more common/useful ones in the \"How-To\" version)\n# output \"cluster_name\" { value = module.kube-hetzner.cluster_name }\n# output \"control_plane_ips\" { value = module.kube-hetzner.control_plane_ips }\n# output \"agent_node_ips\" { value = module.kube-hetzner.agent_node_ips }\n# output \"load_balancer_ip\" { value = module.kube-hetzner.load_balancer_ipv4 }\n```\n\n* **`output \"kubeconfig\"` Block:**\n * **Purpose:** Defines an output variable named `kubeconfig`. Output variables expose data from your Terraform configuration after it's applied.\n * **`value = module.kube-hetzner.kubeconfig`:** The value of this output is taken from an output named `kubeconfig` that is exposed by the `kube-hetzner` module itself. This is the generated Kubernetes configuration file content.\n * **`sensitive = true`:** Marks this output as sensitive. Terraform will not display its value in plain text in the console during `apply` or `output` commands unless explicitly requested (e.g., `terraform output --raw kubeconfig`). This is crucial because the kubeconfig contains credentials.\n* **Other Potential Outputs (as shown in the \"How-To\" version):**\n * It's common to output other useful information like the cluster name, IP addresses of nodes, load balancer IPs, etc., by accessing corresponding outputs from the `kube-hetzner` module.\n\n---\n\n**Section 5: Input Variable Definition (for Root Module)**\n\n```terraform\nvariable \"hcloud_token\" {\n sensitive = true\n default = \"\"\n}\n```\n\n* **`variable \"hcloud_token\"` Block:**\n * **Purpose:** Declares an input variable named `hcloud_token` for this root Terraform configuration.\n * **`sensitive = true`:** Marks this input variable as sensitive. If you were to set it via a `terraform.tfvars` file or command line (`-var=\"hcloud_token=...\"`), Terraform would handle it with more care regarding logging.\n * **`default = \"\"`:** Provides a default value (empty string). This allows the logic `var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token` to work correctly:\n * If `TF_VAR_hcloud_token` is set in the environment, `var.hcloud_token` gets that value.\n * If not, `var.hcloud_token` defaults to `\"\"`, and the ternary operator then chooses `local.hcloud_token`.\n * This variable declaration is what allows `TF_VAR_hcloud_token` to populate `var.hcloud_token`.\n\n```terraform\nvariable \"robot_user\" {\n sensitive = true\n default = \"\"\n}\n```\n\n* **`variable \"robot_user\"` Block:**\n * **Purpose:** Declares an input variable named `robot_user` for this root Terraform configuration. The value should be retrieved from the Hetzner Robot Webservice UI. This variable is required for connecting Robot Nodes to the cluster and is only used when `robot_ccm_enabled` is set to `true`.\n * **`sensitive = true`:** Marks this input variable as sensitive. If you were to set it via a `terraform.tfvars` file or command line (`-var=\"hcloud_token=...\"`), Terraform would handle it with more care regarding logging.\n * **`default = \"\"`:** Provides a default value (empty string). This allows the logic `var.robot_user != \"\" ? var.robot_user : local.robot_user` to work correctly:\n * If `TF_VAR_robot_user` is set in the environment, `var.robot_user` gets that value.\n * If not, `var.robot_user` defaults to `\"\"`, and the ternary operator then chooses `local.robot_user`.\n * This variable declaration is what allows `TF_VAR_robot_user` to populate `var.robot_user`.\n\n```terraform\nvariable \"robot_password\" {\n sensitive = true\n default = \"\"\n}\n```\n\n* **`variable \"robot_password\"` Block:**\n * **Purpose:** Declares an input variable named `robot_password` for this root Terraform configuration. The value should be retrieved from the Hetzner Robot Webservice UI. This variable is required for connecting Robot Nodes to the cluster and is only used when `robot_ccm_enabled` is set to `true`.\n * **`sensitive = true`:** Marks this input variable as sensitive. If you were to set it via a `terraform.tfvars` file or command line (`-var=\"hcloud_token=...\"`), Terraform would handle it with more care regarding logging.\n * **`default = \"\"`:** Provides a default value (empty string). This allows the logic `var.robot_password != \"\" ? var.robot_password : local.robot_password` to work correctly:\n * If `TF_VAR_robot_password` is set in the environment, `var.robot_password` gets that value.\n * If not, `var.robot_password` defaults to `\"\"`, and the ternary operator then chooses `local.robot_password`.\n * This variable declaration is what allows `TF_VAR_robot_password` to populate `var.robot_password`.\n---\n\n**Conclusion of the Deep Dive**\n\nWe have now traversed the entirety of the provided Terraform configuration, dissecting each parameter, comment, and block of logic. This detailed explanation should provide a much deeper understanding of how the `kube-hetzner` module is configured and the implications of each choice.\n\nThe key takeaways are:\n\n* **Modularity:** The power of Terraform modules to abstract complexity.\n* **Declarative IaC:** Defining the \"what,\" not the \"how.\"\n* **Configuration Nuances:** Many settings have interdependencies, lifecycle considerations, and security implications.\n* **Provider Interaction:** The crucial role of the Hetzner Cloud provider and API token.\n* **k3s Specifics:** How k3s features (CNI, storage, upgrades, etc.) are exposed and managed through the module.\n* **Extensibility:** Options for custom Helm values, Kustomize overlays, and pre/post commands allow tailoring the deployment significantly.\n\nThis detailed walkthrough should serve as a comprehensive reference for anyone working with or seeking to understand this particular Terraform setup for deploying k3s on Hetzner Cloud.\n\n---\n\n**Additional Variables Added Since June 2024**\n\nThe following variables have been added to the `kube-hetzner` module since the initial documentation was created. These provide additional functionality and configuration options:\n\n**NAT Router Configuration**\n\n```terraform\n # Setup a NAT router, and automatically disable public ips on all control plane and agent nodes.\n # To use this, you must also set use_control_plane_lb = true, otherwise kubectl can never reach the cluster.\n # The NAT router will also function as a bastion. This makes securing the cluster easier, as all public traffic passes through a single strongly secured node.\n # It does however also introduce a single point of failure, so if you need high-availability on your egress, you should consider other configurations.\n # nat_router = {\n # server_type = \"cx33\"\n # location = \"nbg1\"\n # }\n```\n\n* **`nat_router` (Object, Optional):**\n * **Purpose:** Creates a dedicated NAT router server that acts as the single egress point for all cluster traffic. When enabled, all control plane and agent nodes are provisioned without public IPs.\n * **Requirements:** Must set `use_control_plane_lb = true` when using NAT router, as kubectl needs a public endpoint to reach the cluster.\n * **Benefits:**\n * Enhanced security by limiting public exposure to a single hardened node\n * Acts as a bastion host for SSH access to internal nodes\n * Simplifies firewall rules and security auditing\n * Automatically forwards Kubernetes API traffic (port 6443) when `control_plane_lb_enable_public_interface = false`\n * **Trade-offs:** Introduces a single point of failure for egress traffic\n * **Configuration:**\n * `server_type`: The Hetzner server type for the NAT router\n * `location`: The location where the NAT router should be deployed\n * `labels`: (Optional) Additional labels for the NAT router\n * `enable_sudo`: (Optional, default: false) Enable sudo access for the nat-router user\n * **Port Forwarding:** When the control plane LB has no public interface (`control_plane_lb_enable_public_interface = false`), the NAT router automatically configures iptables rules to forward incoming traffic on port 6443 to the control plane LB's private IP. This allows external kubectl access while keeping the control plane LB completely private.\n\n**k3s Binary Configuration**\n\n```terraform\n # Set to true if util-linux breaks on the OS (temporary regression fixed in util-linux v2.41.1).\n # k3s_prefer_bundled_bin = false\n```\n\n* **`k3s_prefer_bundled_bin` (Boolean, Optional):**\n * **Default:** `false`\n * **Purpose:** Forces k3s to use its bundled binaries instead of system binaries. This is a workaround for compatibility issues with certain OS utilities.\n * **Use Case:** Temporary fix for util-linux regression that affected k3s operation on certain MicroOS versions.\n\n**Load Balancer Hostname Configuration**\n\n```terraform\n # The lb_hostname setting optimizes communication between services within the Kubernetes cluster when they use domain names instead of direct service names.\n # By associating a domain name directly with the Hetzner Load Balancer, this setting can help reduce potential communication delays.\n # lb_hostname = \"mycluster.example.com\"\n```\n\n* **`lb_hostname` (String, Optional):**\n * **Purpose:** Associates a domain name directly with the Hetzner Load Balancer for optimized internal service communication.\n * **Technical Impact:** Sets the `load-balancer.hetzner.cloud/hostname` annotation in the LB definition.\n * **Use Case:** When services communicate using domain names (e.g., `a.mycluster.domain.com`) that point to the external LB, this can reduce communication delays.\n * **Setup:**\n 1. Set `lb_hostname` to a domain like `mycluster.domain.com`\n 2. Create an A record pointing to your LB's IP\n 3. Create CNAME records for service subdomains pointing to the main domain\n * **Note:** Optional - only needed if services communicate via domain names instead of direct service names.\n\n**Rancher Integration**\n\n```terraform\n # You can enable Rancher (installed by Helm behind the scenes) with the following flag, the default is \"false\".\n # enable_rancher = true\n \n # If using Rancher you can set the Rancher hostname\n # rancher_hostname = \"rancher.example.com\"\n \n # Rancher install channel\n # rancher_install_channel = \"stable\"\n \n # Bootstrap password for Rancher (min 48 characters)\n # rancher_bootstrap_password = \"\"\n \n # Rancher registration manifest URL\n # rancher_registration_manifest_url = \"\"\n```\n\n* **`enable_rancher` (Boolean, Optional):**\n * **Default:** `false`\n * **Purpose:** Installs Rancher Manager for Kubernetes cluster management UI.\n * **Requirements:** \n * Control plane nodes need at least 4GB RAM (cx21 or larger)\n * Must set `rancher_hostname`\n * May need to adjust `initial_k3s_channel` for compatibility\n * **Note:** Automatically installs cert-manager with self-signed certificates\n\n* **`rancher_hostname` (String, Required if `enable_rancher = true`):**\n * **Purpose:** The hostname for accessing Rancher UI. Must be unique even if not used with DNS.\n * **Default:** Falls back to `lb_hostname` if set and using Hetzner LB\n\n* **`rancher_install_channel` (String, Optional):**\n * **Default:** `\"stable\"`\n * **Options:** `\"stable\"` or `\"latest\"`\n * **Purpose:** Controls which Rancher release channel to use\n\n* **`rancher_bootstrap_password` (String, Optional):**\n * **Purpose:** Initial admin password for Rancher (minimum 48 characters)\n * **Default:** Auto-generated if not specified\n\n* **`rancher_registration_manifest_url` (String, Optional):**\n * **Purpose:** URL for importing this cluster into an existing Rancher installation\n * **Note:** Alternative to enabling Rancher directly - use one or the other\n\n**Kustomization Control**\n\n```terraform\n # Control kubeconfig and kustomization file generation\n # create_kubeconfig = false\n # create_kustomization = true\n # export_values = true\n```\n\n* **`create_kubeconfig` (Boolean, Optional):**\n * **Default:** `true` (for backward compatibility)\n * **Purpose:** Controls whether to create a kubeconfig file on disk\n * **Best Practice:** Set to `false` and use `terraform output --raw kubeconfig_data > cluster_kubeconfig.yaml` instead\n * **Security:** Prevents accidental commits of kubeconfig files\n\n* **`create_kustomization` (Boolean, Optional):**\n * **Default:** `true`\n * **Purpose:** Controls creation of kustomization backup files\n * **Use Case:** Set to `false` for automation scenarios\n\n* **`export_values` (Boolean, Optional):**\n * **Default:** `false`\n * **Purpose:** Exports values.yaml files for deployed components (traefik, longhorn, cert-manager, etc.)\n * **Use Case:** Useful for GitOps workflows with ArgoCD or similar tools\n\n**Extra Kustomization Parameters**\n\n```terraform\n # Extra commands and parameters for kustomization\n # extra_kustomize_deployment_commands = [\"kubectl wait --for=condition=ready pod -l app=myapp -n mynamespace --timeout=300s\"]\n # extra_kustomize_parameters = {\n # myvar = \"myvalue\"\n # }\n```\n\n* **`extra_kustomize_deployment_commands` (List of Strings, Optional):**\n * **Purpose:** Commands to execute after `kubectl apply -k`\n * **Use Cases:** \n * Wait for CRDs to be ready\n * Apply additional manifests\n * Post-install validation\n\n* **`extra_kustomize_parameters` (Map, Optional):**\n * **Purpose:** Variables passed to `extra-manifests/kustomization.yaml.tpl`\n * **Use Case:** Template variables for custom kustomization overlays\n\n**MicroOS Snapshot Control**\n\n```terraform\n # Use specific MicroOS snapshot IDs\n # microos_x86_snapshot_id = \"123456\"\n # microos_arm_snapshot_id = \"789012\"\n```\n\n* **`microos_x86_snapshot_id` / `microos_arm_snapshot_id` (String, Optional):**\n * **Default:** Uses the most recent snapshot created with `createkh`\n * **Purpose:** Pin to specific MicroOS snapshot versions\n * **Discovery:** `hcloud image list --selector 'microos-snapshot=yes'`\n * **Use Case:** Ensure consistency across deployments or rollback to known-good images\n\n**vSwitch Configuration**\n\n```terraform\n # To connect the Hetzner Cloud network to Robot servers via vSwitch subnet, create the vSwitch and set its ID to the `vswitch_id` (number).\n # Note that the VLAN ID is not the same as vSwitch ID. The vSwitch-subnet is assigned to 10.201.0.0/16 by default, can be changed via var.vswitch_subnet_index.\n # The vSwitch subnet is not created when the value is null. Default: null\n # vswitch_id = null\n```\n\n* **`vswitch_id` (number, Optional):**\n * **Purpose:** Connects the Cloud network to a pre-existing Hetzner vSwitch by creating a vSwitch-type subnet. It also exposes Cloud network routes to the vSwitch.\n * **Requirements:** vSwitch must exist\n * **Use Case:** The connection is required if Hetzner Robot instances are connected to the Hetzner Cloud instances via private networking \n\n* **`vswitch_subnet_index` (number, Optional):**\n * **Purpose:** Defines the subnet range index to be used in the vSwitch subnet creation. Default: 201, which then converts to 10.201.0.0/16 by default.\n\n**Additional Helm Values Customization**\n\nThe following variables allow deep customization of various components through Helm values:\n\n```terraform\n # Custom Cilium values\n # cilium_values = <<-EOT\n # ipam:\n # mode: kubernetes\n # EOT\n \n # Custom cert-manager values\n # cert_manager_values = <<-EOT\n # crds:\n # enabled: true\n # EOT\n \n # Custom Hetzner CCM values\n # hetzner_ccm_values = <<-EOT\n # networking:\n # enabled: true\n # EOT\n \n # Custom CSI driver SMB values\n # csi_driver_smb_values = <<-EOT\n # controller:\n # replicas: 2\n # EOT\n \n # Custom Longhorn values\n # longhorn_values = <<-EOT\n # defaultSettings:\n # defaultDataPath: /var/longhorn\n # EOT\n \n # Custom Rancher values\n # rancher_values = <<-EOT\n # hostname: rancher.example.com\n # replicas: 3\n # EOT\n```\n\nEach of these `*_values` variables:\n* **Purpose:** Override default Helm chart values for the respective component\n* **Format:** YAML string (heredoc syntax preserves formatting)\n* **Reference:** See each component's Helm chart documentation for available options\n* **Note:** Indentation within the heredoc is significant\n\n**Ingress Controller Versions and Values**\n\n```terraform\n # Specific ingress controller versions\n # traefik_version = \"30.1.0\"\n # nginx_version = \"4.11.3\"\n # haproxy_version = \"1.41.0\"\n \n # Custom Traefik values\n # traefik_values = <<-EOT\n # deployment:\n # replicas: 3\n # EOT\n \n # Custom Nginx values\n # nginx_values = <<-EOT\n # controller:\n # replicaCount: 3\n # EOT\n \n # Custom HAProxy configuration\n # haproxy_additional_proxy_protocol_ips = [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n # haproxy_requests_cpu = \"250m\"\n # haproxy_requests_memory = \"256Mi\"\n # haproxy_values = <<-EOT\n # controller:\n # replicaCount: 3\n # EOT\n```\n\n* **`traefik_version` / `nginx_version` / `haproxy_version` (String, Optional):**\n * **Purpose:** Pin to specific Helm chart versions for ingress controllers\n * **Default:** Latest available version\n * **Reference:** Check respective GitHub releases pages\n\n* **`traefik_values` / `nginx_values` / `haproxy_values` (String, Optional):**\n * **Purpose:** Override default Helm values for ingress controllers\n * **Format:** YAML heredoc string\n\n* **`haproxy_additional_proxy_protocol_ips` (List of Strings, Optional):**\n * **Purpose:** Additional trusted IPs for HAProxy proxy protocol\n * **Default:** Includes common private ranges\n\n* **`haproxy_requests_cpu` / `haproxy_requests_memory` (String, Optional):**\n * **Purpose:** Resource requests for HAProxy pods\n * **Format:** Kubernetes resource notation (e.g., \"250m\", \"256Mi\")\n\n---\n\nThis concludes the documentation of variables added to the `kube-hetzner` module since June 2024. These additions provide enhanced security options (NAT router), better integration capabilities (Rancher), more granular control over deployments (snapshot IDs, kubeconfig generation), and extensive customization options through Helm values overrides.\n" }, { "path": "docs/private-network-egress.md", "content": "# Private Network Egress & Hetzner DHCP (Aug 2025)\n\nOn **August 11, 2025**, Hetzner Cloud removed the legacy DHCP *Router option (code 3)* on private networks and now relies solely on *Classless Static Route (option 121)*. Any node that forwards outbound traffic through a NAT or VPN gateway on the private network must therefore install and persist a default route to the virtual gateway (typically the first IP of the prefix, e.g. `10.0.0.1`).\n\nStarting with this module version:\n\n- All nodes that attach to the Hetzner private network detect the relevant interface dynamically (no hardcoded `eth1`) and persist a `0.0.0.0/0` route via `${local.network_gw_ipv4}` in the active NetworkManager connection.\n- A runtime guard (`ip route add` with a high metric) ensures nodes regain egress immediately, even before NetworkManager reapplies the profile, without disturbing an existing public default route.\n- The route uses a higher metric so the public interface continues to be preferred on mixed deployments.\n\nNo manual `ip route add` commands are needed after reboots or DHCP renewals. If you roll your own images or bootstrap logic, make sure an equivalent persistent route exists or the nodes will lose outbound connectivity the next time the DHCP lease renews.\n" }, { "path": "docs/ssh.md", "content": "Kube-Hetzner requires you to have a recent version of OpenSSH (>=6.5) installed on your client, and the use of a key-pair generated with either of the following algorithms:\n\n- ssh-ed25519 (preferred, and most simple to use without passphrase)\n- rsa-sha2-512\n- rsa-sha2-256\n\nIf your key-pair is of the `ssh-ed25519` sort (useful command `ssh-keygen -t ed25519`), and without of passphrase, you do not need to do anything else. Just set `public_key` and `private_key` to their respective path values in your kube.tf file.\n\n---\n\nOtherwise, for a key-pair with passphrase or a device like a Yubikey, make sure you have an SSH agent running and your key is loaded with:\n\n```bash\neval ssh-agent $SHELL\nssh-add ~/.ssh/my_private-key_id\n```\n\nVerify it is loaded with:\n\n```bash\nssh-add -l\n```\n\nThen set `private_key = null` in your kube.tf file, as it will be read from the ssh-agent automatically.\n\n---\n\n## Firewall SSH source and changing IPs\n\nSSH access is controlled by the Hetzner Cloud firewall, and the module configures it via the `firewall_ssh_source` input. This is a list of CIDR blocks that are allowed to connect to the nodes over SSH (for a single IPv4 address, use a `/32`, for IPv6 a `/128`).\n\nIf your IP changes, you are not locked out permanently:\n\n- Update `firewall_ssh_source` in your `kube.tf` (or the corresponding variable in Terraform Cloud).\n- Run `terraform plan` and `terraform apply`.\n\nTerraform updates the firewall through the Hetzner API, so SSH access is not required to make this change. If you need access immediately, you can temporarily add a wider CIDR (for example `0.0.0.0/0` and/or `::/0`), apply, and then tighten it again once you are connected. A static IP or a VPN with a fixed egress IP also avoids future changes.\n" }, { "path": "docs/terraform.md", "content": "<!-- BEGIN_TF_DOCS -->\n### Requirements\n\n| Name | Version |\n|------|---------|\n| <a name=\"requirement_terraform\"></a> [terraform](#requirement\\_terraform) | >= 1.10.1 |\n| <a name=\"requirement_assert\"></a> [assert](#requirement\\_assert) | >= 0.16.0 |\n| <a name=\"requirement_github\"></a> [github](#requirement\\_github) | >= 6.4.0 |\n| <a name=\"requirement_hcloud\"></a> [hcloud](#requirement\\_hcloud) | >= 1.59.0 |\n| <a name=\"requirement_local\"></a> [local](#requirement\\_local) | >= 2.5.2 |\n| <a name=\"requirement_semvers\"></a> [semvers](#requirement\\_semvers) | >= 0.7.1 |\n| <a name=\"requirement_ssh\"></a> [ssh](#requirement\\_ssh) | 2.7.0 |\n\n### Providers\n\n| Name | Version |\n|------|---------|\n| <a name=\"provider_cloudinit\"></a> [cloudinit](#provider\\_cloudinit) | n/a |\n| <a name=\"provider_github\"></a> [github](#provider\\_github) | >= 6.4.0 |\n| <a name=\"provider_hcloud\"></a> [hcloud](#provider\\_hcloud) | >= 1.59.0 |\n| <a name=\"provider_local\"></a> [local](#provider\\_local) | >= 2.5.2 |\n| <a name=\"provider_random\"></a> [random](#provider\\_random) | n/a |\n| <a name=\"provider_ssh\"></a> [ssh](#provider\\_ssh) | 2.7.0 |\n| <a name=\"provider_terraform\"></a> [terraform](#provider\\_terraform) | n/a |\n\n### Modules\n\n| Name | Source | Version |\n|------|--------|---------|\n| <a name=\"module_agents\"></a> [agents](#module\\_agents) | ./modules/host | n/a |\n| <a name=\"module_control_planes\"></a> [control\\_planes](#module\\_control\\_planes) | ./modules/host | n/a |\n| <a name=\"module_values_merger_cert_manager\"></a> [values\\_merger\\_cert\\_manager](#module\\_values\\_merger\\_cert\\_manager) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_cilium\"></a> [values\\_merger\\_cilium](#module\\_values\\_merger\\_cilium) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_haproxy\"></a> [values\\_merger\\_haproxy](#module\\_values\\_merger\\_haproxy) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_hetzner_ccm\"></a> [values\\_merger\\_hetzner\\_ccm](#module\\_values\\_merger\\_hetzner\\_ccm) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_longhorn\"></a> [values\\_merger\\_longhorn](#module\\_values\\_merger\\_longhorn) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_nginx\"></a> [values\\_merger\\_nginx](#module\\_values\\_merger\\_nginx) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_rancher\"></a> [values\\_merger\\_rancher](#module\\_values\\_merger\\_rancher) | ./modules/values_merger | n/a |\n| <a name=\"module_values_merger_traefik\"></a> [values\\_merger\\_traefik](#module\\_values\\_merger\\_traefik) | ./modules/values_merger | n/a |\n\n### Resources\n\n| Name | Type |\n|------|------|\n| [hcloud_firewall.k3s](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/firewall) | resource |\n| [hcloud_floating_ip.agents](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/floating_ip) | resource |\n| [hcloud_floating_ip_assignment.agents](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/floating_ip_assignment) | resource |\n| [hcloud_load_balancer.cluster](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer) | resource |\n| [hcloud_load_balancer.control_plane](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer) | resource |\n| [hcloud_load_balancer_network.cluster](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer_network) | resource |\n| [hcloud_load_balancer_network.control_plane](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer_network) | resource |\n| [hcloud_load_balancer_service.control_plane](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer_service) | resource |\n| [hcloud_load_balancer_target.cluster](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer_target) | resource |\n| [hcloud_load_balancer_target.control_plane](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/load_balancer_target) | resource |\n| [hcloud_network.k3s](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/network) | resource |\n| [hcloud_network_route.nat_route_public_internet](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/network_route) | resource |\n| [hcloud_network_subnet.agent](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/network_subnet) | resource |\n| [hcloud_network_subnet.control_plane](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/network_subnet) | resource |\n| [hcloud_network_subnet.nat_router](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/network_subnet) | resource |\n| [hcloud_network_subnet.vswitch_subnet](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/network_subnet) | resource |\n| [hcloud_placement_group.agent](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/placement_group) | resource |\n| [hcloud_placement_group.agent_named](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/placement_group) | resource |\n| [hcloud_placement_group.control_plane](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/placement_group) | resource |\n| [hcloud_placement_group.control_plane_named](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/placement_group) | resource |\n| [hcloud_primary_ip.nat_router_primary_ipv4](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/primary_ip) | resource |\n| [hcloud_primary_ip.nat_router_primary_ipv6](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/primary_ip) | resource |\n| [hcloud_rdns.agents](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/rdns) | resource |\n| [hcloud_server.nat_router](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/server) | resource |\n| [hcloud_ssh_key.k3s](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/ssh_key) | resource |\n| [hcloud_volume.longhorn_volume](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/volume) | resource |\n| [local_file.cert_manager_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.cilium_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.csi_driver_smb_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.haproxy_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.hetzner_ccm_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.kustomization_backup](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.longhorn_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.nginx_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_file.traefik_values](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |\n| [local_sensitive_file.kubeconfig](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/sensitive_file) | resource |\n| [random_password.k3s_token](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/password) | resource |\n| [random_password.nat_router_vip_auth_pass](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/password) | resource |\n| [random_password.rancher_bootstrap](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/password) | resource |\n| [random_string.nat_router](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource |\n| [ssh_sensitive_resource.kubeconfig](https://registry.terraform.io/providers/loafoe/ssh/2.7.0/docs/resources/sensitive_resource) | resource |\n| [terraform_data.agent_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.agents](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.audit_policy](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.authentication_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.autoscaled_nodes_kubelet_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.autoscaled_nodes_registries](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.configure_autoscaler](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.configure_floating_ip](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.configure_longhorn_volume](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.control_plane_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.control_planes](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.first_control_plane](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.kube_system_secrets](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.kustomization](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.kustomization_user](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.kustomization_user_deploy](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [terraform_data.nat_router_await_cloud_init](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource |\n| [cloudinit_config.autoscaler_config](https://registry.terraform.io/providers/hashicorp/cloudinit/latest/docs/data-sources/config) | data source |\n| [cloudinit_config.autoscaler_legacy_config](https://registry.terraform.io/providers/hashicorp/cloudinit/latest/docs/data-sources/config) | data source |\n| [cloudinit_config.nat_router_config](https://registry.terraform.io/providers/hashicorp/cloudinit/latest/docs/data-sources/config) | data source |\n| [github_release.calico](https://registry.terraform.io/providers/integrations/github/latest/docs/data-sources/release) | data source |\n| [github_release.hetzner_ccm](https://registry.terraform.io/providers/integrations/github/latest/docs/data-sources/release) | data source |\n| [github_release.hetzner_csi](https://registry.terraform.io/providers/integrations/github/latest/docs/data-sources/release) | data source |\n| [github_release.kured](https://registry.terraform.io/providers/integrations/github/latest/docs/data-sources/release) | data source |\n| [hcloud_image.microos_arm_snapshot](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/data-sources/image) | data source |\n| [hcloud_image.microos_x86_snapshot](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/data-sources/image) | data source |\n| [hcloud_network.k3s](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/data-sources/network) | data source |\n| [hcloud_servers.autoscaled_nodes](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/data-sources/servers) | data source |\n| [hcloud_ssh_keys.keys_by_selector](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/data-sources/ssh_keys) | data source |\n\n### Inputs\n\n| Name | Description | Type | Default | Required |\n|------|-------------|------|---------|:--------:|\n| <a name=\"input_additional_k3s_environment\"></a> [additional\\_k3s\\_environment](#input\\_additional\\_k3s\\_environment) | Additional environment variables for the k3s binary. See for example https://docs.k3s.io/advanced#configuring-an-http-proxy . | `map(any)` | `{}` | no |\n| <a name=\"input_additional_tls_sans\"></a> [additional\\_tls\\_sans](#input\\_additional\\_tls\\_sans) | Additional TLS SANs to allow connection to control-plane through it. | `list(string)` | `[]` | no |\n| <a name=\"input_address_for_connectivity_test\"></a> [address\\_for\\_connectivity\\_test](#input\\_address\\_for\\_connectivity\\_test) | The address to test for external connectivity before proceeding with the installation. Defaults to Google's public DNS. | `string` | `\"8.8.8.8\"` | no |\n| <a name=\"input_agent_nodepools\"></a> [agent\\_nodepools](#input\\_agent\\_nodepools) | Number of agent nodes. | <pre>list(object({<br/> name = string<br/> server_type = string<br/> location = string<br/> backups = optional(bool)<br/> floating_ip = optional(bool)<br/> floating_ip_rdns = optional(string, null)<br/> labels = list(string)<br/> taints = list(string)<br/> longhorn_volume_size = optional(number)<br/> longhorn_mount_path = optional(string, \"/var/longhorn\")<br/> swap_size = optional(string, \"\")<br/> zram_size = optional(string, \"\")<br/> kubelet_args = optional(list(string), [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])<br/> selinux = optional(bool, true)<br/> placement_group_compat_idx = optional(number, 0)<br/> placement_group = optional(string, null)<br/> subnet_ip_range = optional(string, null)<br/> count = optional(number, null)<br/> disable_ipv4 = optional(bool, false)<br/> disable_ipv6 = optional(bool, false)<br/> network_id = optional(number, 0)<br/> nodes = optional(map(object({<br/> server_type = optional(string)<br/> location = optional(string)<br/> backups = optional(bool)<br/> floating_ip = optional(bool)<br/> floating_ip_rdns = optional(string, null)<br/> labels = optional(list(string))<br/> taints = optional(list(string))<br/> longhorn_volume_size = optional(number)<br/> longhorn_mount_path = optional(string, null)<br/> swap_size = optional(string, \"\")<br/> zram_size = optional(string, \"\")<br/> kubelet_args = optional(list(string), [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])<br/> selinux = optional(bool, true)<br/> placement_group_compat_idx = optional(number, 0)<br/> placement_group = optional(string, null)<br/> append_index_to_node_name = optional(bool, true)<br/> })))<br/> }))</pre> | `[]` | no |\n| <a name=\"input_agent_nodes_custom_config\"></a> [agent\\_nodes\\_custom\\_config](#input\\_agent\\_nodes\\_custom\\_config) | Additional configuration for agent nodes and autoscaler nodes that will be added to k3s's config.yaml. E.g to allow kube-proxy monitoring. | `any` | `{}` | no |\n| <a name=\"input_allow_scheduling_on_control_plane\"></a> [allow\\_scheduling\\_on\\_control\\_plane](#input\\_allow\\_scheduling\\_on\\_control\\_plane) | Whether to allow non-control-plane workloads to run on the control-plane nodes. | `bool` | `false` | no |\n| <a name=\"input_authentication_config\"></a> [authentication\\_config](#input\\_authentication\\_config) | Strucutred authentication configuration. This can be used to define external authentication providers. | `string` | `\"\"` | no |\n| <a name=\"input_automatically_upgrade_k3s\"></a> [automatically\\_upgrade\\_k3s](#input\\_automatically\\_upgrade\\_k3s) | Whether to automatically upgrade k3s based on the selected channel. | `bool` | `true` | no |\n| <a name=\"input_automatically_upgrade_os\"></a> [automatically\\_upgrade\\_os](#input\\_automatically\\_upgrade\\_os) | Whether to enable or disable automatic os updates. Defaults to true. Should be disabled for single-node clusters | `bool` | `true` | no |\n| <a name=\"input_autoscaler_disable_ipv4\"></a> [autoscaler\\_disable\\_ipv4](#input\\_autoscaler\\_disable\\_ipv4) | Disable IPv4 on nodes created by the Cluster Autoscaler. | `bool` | `false` | no |\n| <a name=\"input_autoscaler_disable_ipv6\"></a> [autoscaler\\_disable\\_ipv6](#input\\_autoscaler\\_disable\\_ipv6) | Disable IPv6 on nodes created by the Cluster Autoscaler. | `bool` | `false` | no |\n| <a name=\"input_autoscaler_labels\"></a> [autoscaler\\_labels](#input\\_autoscaler\\_labels) | Labels for nodes created by the Cluster Autoscaler. | `list(string)` | `[]` | no |\n| <a name=\"input_autoscaler_nodepools\"></a> [autoscaler\\_nodepools](#input\\_autoscaler\\_nodepools) | Cluster autoscaler nodepools. | <pre>list(object({<br/> name = string<br/> server_type = string<br/> location = string<br/> min_nodes = number<br/> max_nodes = number<br/> labels = optional(map(string), {})<br/> kubelet_args = optional(list(string), [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])<br/> taints = optional(list(object({<br/> key = string<br/> value = string<br/> effect = string<br/> })), [])<br/> swap_size = optional(string, \"\")<br/> zram_size = optional(string, \"\")<br/> }))</pre> | `[]` | no |\n| <a name=\"input_autoscaler_taints\"></a> [autoscaler\\_taints](#input\\_autoscaler\\_taints) | Taints for nodes created by the Cluster Autoscaler. | `list(string)` | `[]` | no |\n| <a name=\"input_base_domain\"></a> [base\\_domain](#input\\_base\\_domain) | Base domain of the cluster, used for reverse dns. | `string` | `\"\"` | no |\n| <a name=\"input_block_icmp_ping_in\"></a> [block\\_icmp\\_ping\\_in](#input\\_block\\_icmp\\_ping\\_in) | Block entering ICMP ping. | `bool` | `false` | no |\n| <a name=\"input_calico_values\"></a> [calico\\_values](#input\\_calico\\_values) | Just a stub for a future helm implementation. Now it can be used to replace the calico kustomize patch of the calico manifest. | `string` | `\"\"` | no |\n| <a name=\"input_calico_version\"></a> [calico\\_version](#input\\_calico\\_version) | Version of Calico. See https://github.com/projectcalico/calico/releases for the available versions. | `string` | `null` | no |\n| <a name=\"input_cert_manager_helmchart_bootstrap\"></a> [cert\\_manager\\_helmchart\\_bootstrap](#input\\_cert\\_manager\\_helmchart\\_bootstrap) | Whether the HelmChart cert\\_manager shall be run on control-plane nodes. | `bool` | `false` | no |\n| <a name=\"input_cert_manager_merge_values\"></a> [cert\\_manager\\_merge\\_values](#input\\_cert\\_manager\\_merge\\_values) | Additional Helm values to merge with defaults (or cert\\_manager\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_cert_manager_values\"></a> [cert\\_manager\\_values](#input\\_cert\\_manager\\_values) | Additional helm values file to pass to Cert-Manager as 'valuesContent' at the HelmChart. Defaults are set in locals.tf. For cert-manager versions prior to v1.15.0, you need to set 'installCRDs: true'. | `string` | `\"\"` | no |\n| <a name=\"input_cert_manager_version\"></a> [cert\\_manager\\_version](#input\\_cert\\_manager\\_version) | Version of cert\\_manager. | `string` | `\"*\"` | no |\n| <a name=\"input_cilium_egress_gateway_enabled\"></a> [cilium\\_egress\\_gateway\\_enabled](#input\\_cilium\\_egress\\_gateway\\_enabled) | Enables egress gateway to redirect and SNAT the traffic that leaves the cluster. | `bool` | `false` | no |\n| <a name=\"input_cilium_hubble_enabled\"></a> [cilium\\_hubble\\_enabled](#input\\_cilium\\_hubble\\_enabled) | Enables Hubble Observability to collect and visualize network traffic. | `bool` | `false` | no |\n| <a name=\"input_cilium_hubble_metrics_enabled\"></a> [cilium\\_hubble\\_metrics\\_enabled](#input\\_cilium\\_hubble\\_metrics\\_enabled) | Configures the list of Hubble metrics to collect | `list(string)` | `[]` | no |\n| <a name=\"input_cilium_ipv4_native_routing_cidr\"></a> [cilium\\_ipv4\\_native\\_routing\\_cidr](#input\\_cilium\\_ipv4\\_native\\_routing\\_cidr) | Used when Cilium is configured in native routing mode. The CNI assumes that the underlying network stack will forward packets to this destination without the need to apply SNAT. Default: value of \"cluster\\_ipv4\\_cidr\" | `string` | `null` | no |\n| <a name=\"input_cilium_loadbalancer_acceleration_mode\"></a> [cilium\\_loadbalancer\\_acceleration\\_mode](#input\\_cilium\\_loadbalancer\\_acceleration\\_mode) | Set Cilium loadbalancer.acceleration-mode. Supported values are \"disabled\", \"native\" and \"best-effort\". | `string` | `\"best-effort\"` | no |\n| <a name=\"input_cilium_merge_values\"></a> [cilium\\_merge\\_values](#input\\_cilium\\_merge\\_values) | Additional Helm values to merge with defaults (or cilium\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_cilium_routing_mode\"></a> [cilium\\_routing\\_mode](#input\\_cilium\\_routing\\_mode) | Set native-routing mode (\"native\") or tunneling mode (\"tunnel\"). | `string` | `\"tunnel\"` | no |\n| <a name=\"input_cilium_values\"></a> [cilium\\_values](#input\\_cilium\\_values) | Additional helm values file to pass to Cilium as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_cilium_version\"></a> [cilium\\_version](#input\\_cilium\\_version) | Version of Cilium. See https://github.com/cilium/cilium/releases for the available versions. | `string` | `\"1.17.0\"` | no |\n| <a name=\"input_cluster_autoscaler_extra_args\"></a> [cluster\\_autoscaler\\_extra\\_args](#input\\_cluster\\_autoscaler\\_extra\\_args) | Extra arguments for the Cluster Autoscaler deployment. | `list(string)` | `[]` | no |\n| <a name=\"input_cluster_autoscaler_image\"></a> [cluster\\_autoscaler\\_image](#input\\_cluster\\_autoscaler\\_image) | Image of Kubernetes Cluster Autoscaler for Hetzner Cloud to be used. | `string` | `\"registry.k8s.io/autoscaling/cluster-autoscaler\"` | no |\n| <a name=\"input_cluster_autoscaler_log_level\"></a> [cluster\\_autoscaler\\_log\\_level](#input\\_cluster\\_autoscaler\\_log\\_level) | Verbosity level of the logs for cluster-autoscaler | `number` | `4` | no |\n| <a name=\"input_cluster_autoscaler_log_to_stderr\"></a> [cluster\\_autoscaler\\_log\\_to\\_stderr](#input\\_cluster\\_autoscaler\\_log\\_to\\_stderr) | Determines whether to log to stderr or not | `bool` | `true` | no |\n| <a name=\"input_cluster_autoscaler_replicas\"></a> [cluster\\_autoscaler\\_replicas](#input\\_cluster\\_autoscaler\\_replicas) | Number of replicas for the cluster autoscaler deployment. Multiple replicas use leader election for HA. | `number` | `1` | no |\n| <a name=\"input_cluster_autoscaler_resource_limits\"></a> [cluster\\_autoscaler\\_resource\\_limits](#input\\_cluster\\_autoscaler\\_resource\\_limits) | Should cluster autoscaler enable default resource requests and limits. Default values are requests: 100m & 300Mi and limits: 100m & 300Mi. | `bool` | `true` | no |\n| <a name=\"input_cluster_autoscaler_resource_values\"></a> [cluster\\_autoscaler\\_resource\\_values](#input\\_cluster\\_autoscaler\\_resource\\_values) | Requests and limits for Cluster Autoscaler. | <pre>object({<br/> requests = object({<br/> cpu = string<br/> memory = string<br/> })<br/> limits = object({<br/> cpu = string<br/> memory = string<br/> })<br/> })</pre> | <pre>{<br/> \"limits\": {<br/> \"cpu\": \"100m\",<br/> \"memory\": \"300Mi\"<br/> },<br/> \"requests\": {<br/> \"cpu\": \"100m\",<br/> \"memory\": \"300Mi\"<br/> }<br/>}</pre> | no |\n| <a name=\"input_cluster_autoscaler_server_creation_timeout\"></a> [cluster\\_autoscaler\\_server\\_creation\\_timeout](#input\\_cluster\\_autoscaler\\_server\\_creation\\_timeout) | Timeout (in minutes) until which a newly created server/node has to become available before giving up and destroying it. | `number` | `15` | no |\n| <a name=\"input_cluster_autoscaler_stderr_threshold\"></a> [cluster\\_autoscaler\\_stderr\\_threshold](#input\\_cluster\\_autoscaler\\_stderr\\_threshold) | Severity level above which logs are sent to stderr instead of stdout | `string` | `\"INFO\"` | no |\n| <a name=\"input_cluster_autoscaler_version\"></a> [cluster\\_autoscaler\\_version](#input\\_cluster\\_autoscaler\\_version) | Version of Kubernetes Cluster Autoscaler for Hetzner Cloud. Should be aligned with Kubernetes version. Available versions for the official image can be found at https://explore.ggcr.dev/?repo=registry.k8s.io%2Fautoscaling%2Fcluster-autoscaler. | `string` | `\"v1.33.3\"` | no |\n| <a name=\"input_cluster_dns_ipv4\"></a> [cluster\\_dns\\_ipv4](#input\\_cluster\\_dns\\_ipv4) | Internal Service IPv4 address of core-dns. | `string` | `null` | no |\n| <a name=\"input_cluster_ipv4_cidr\"></a> [cluster\\_ipv4\\_cidr](#input\\_cluster\\_ipv4\\_cidr) | Internal Pod CIDR, used for the controller and currently for calico/cilium. | `string` | `\"10.42.0.0/16\"` | no |\n| <a name=\"input_cluster_name\"></a> [cluster\\_name](#input\\_cluster\\_name) | Name of the cluster. | `string` | `\"k3s\"` | no |\n| <a name=\"input_cni_plugin\"></a> [cni\\_plugin](#input\\_cni\\_plugin) | CNI plugin for k3s. | `string` | `\"flannel\"` | no |\n| <a name=\"input_control_plane_endpoint\"></a> [control\\_plane\\_endpoint](#input\\_control\\_plane\\_endpoint) | Optional external control plane endpoint URL (e.g. https://myapi.domain.com:6443). Used as the k3s 'server' value for agents and secondary control planes. | `string` | `null` | no |\n| <a name=\"input_control_plane_lb_enable_public_interface\"></a> [control\\_plane\\_lb\\_enable\\_public\\_interface](#input\\_control\\_plane\\_lb\\_enable\\_public\\_interface) | Enable or disable public interface for the control plane load balancer. Defaults to true. When disabled with nat\\_router enabled, the NAT router automatically forwards port 6443 to the private control plane LB. | `bool` | `true` | no |\n| <a name=\"input_control_plane_lb_type\"></a> [control\\_plane\\_lb\\_type](#input\\_control\\_plane\\_lb\\_type) | The type of load balancer to use for the control plane load balancer. Defaults to lb11, which is the cheapest one. | `string` | `\"lb11\"` | no |\n| <a name=\"input_control_plane_nodepools\"></a> [control\\_plane\\_nodepools](#input\\_control\\_plane\\_nodepools) | Number of control plane nodes. | <pre>list(object({<br/> name = string<br/> server_type = string<br/> location = string<br/> backups = optional(bool)<br/> labels = list(string)<br/> taints = list(string)<br/> count = number<br/> swap_size = optional(string, \"\")<br/> zram_size = optional(string, \"\")<br/> kubelet_args = optional(list(string), [\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])<br/> selinux = optional(bool, true)<br/> placement_group_compat_idx = optional(number, 0)<br/> placement_group = optional(string, null)<br/> disable_ipv4 = optional(bool, false)<br/> disable_ipv6 = optional(bool, false)<br/> network_id = optional(number, 0)<br/> }))</pre> | `[]` | no |\n| <a name=\"input_control_planes_custom_config\"></a> [control\\_planes\\_custom\\_config](#input\\_control\\_planes\\_custom\\_config) | Additional configuration for control planes that will be added to k3s's config.yaml. E.g to allow etcd monitoring. | `any` | `{}` | no |\n| <a name=\"input_create_kubeconfig\"></a> [create\\_kubeconfig](#input\\_create\\_kubeconfig) | Create the kubeconfig as a local file resource. Should be disabled for automatic runs. | `bool` | `true` | no |\n| <a name=\"input_create_kustomization\"></a> [create\\_kustomization](#input\\_create\\_kustomization) | Create the kustomization backup as a local file resource. Should be disabled for automatic runs. | `bool` | `true` | no |\n| <a name=\"input_csi_driver_smb_helmchart_bootstrap\"></a> [csi\\_driver\\_smb\\_helmchart\\_bootstrap](#input\\_csi\\_driver\\_smb\\_helmchart\\_bootstrap) | Whether the HelmChart csi\\_driver\\_smb shall be run on control-plane nodes. | `bool` | `false` | no |\n| <a name=\"input_csi_driver_smb_values\"></a> [csi\\_driver\\_smb\\_values](#input\\_csi\\_driver\\_smb\\_values) | Additional helm values file to pass to csi-driver-smb as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_csi_driver_smb_version\"></a> [csi\\_driver\\_smb\\_version](#input\\_csi\\_driver\\_smb\\_version) | Version of csi\\_driver\\_smb. See https://github.com/kubernetes-csi/csi-driver-smb/releases for the available versions. | `string` | `\"*\"` | no |\n| <a name=\"input_disable_hetzner_csi\"></a> [disable\\_hetzner\\_csi](#input\\_disable\\_hetzner\\_csi) | Disable hetzner csi driver. | `bool` | `false` | no |\n| <a name=\"input_disable_kube_proxy\"></a> [disable\\_kube\\_proxy](#input\\_disable\\_kube\\_proxy) | Disable kube-proxy in K3s (default false). | `bool` | `false` | no |\n| <a name=\"input_disable_network_policy\"></a> [disable\\_network\\_policy](#input\\_disable\\_network\\_policy) | Disable k3s default network policy controller (default false, automatically true for calico and cilium). | `bool` | `false` | no |\n| <a name=\"input_disable_selinux\"></a> [disable\\_selinux](#input\\_disable\\_selinux) | Disable SELinux on all nodes. | `bool` | `false` | no |\n| <a name=\"input_dns_servers\"></a> [dns\\_servers](#input\\_dns\\_servers) | IP Addresses to use for the DNS Servers, set to an empty list to use the ones provided by Hetzner. The length is limited to 3 entries, more entries is not supported by kubernetes | `list(string)` | <pre>[<br/> \"185.12.64.1\",<br/> \"185.12.64.2\",<br/> \"2a01:4ff:ff00::add:1\"<br/>]</pre> | no |\n| <a name=\"input_enable_cert_manager\"></a> [enable\\_cert\\_manager](#input\\_enable\\_cert\\_manager) | Enable cert manager. | `bool` | `true` | no |\n| <a name=\"input_enable_csi_driver_smb\"></a> [enable\\_csi\\_driver\\_smb](#input\\_enable\\_csi\\_driver\\_smb) | Whether or not to enable csi-driver-smb. | `bool` | `false` | no |\n| <a name=\"input_enable_delete_protection\"></a> [enable\\_delete\\_protection](#input\\_enable\\_delete\\_protection) | Enable or disable delete protection for resources in Hetzner Cloud. | <pre>object({<br/> floating_ip = optional(bool, false)<br/> load_balancer = optional(bool, false)<br/> volume = optional(bool, false)<br/> })</pre> | <pre>{<br/> \"floating_ip\": false,<br/> \"load_balancer\": false,<br/> \"volume\": false<br/>}</pre> | no |\n| <a name=\"input_enable_iscsid\"></a> [enable\\_iscsid](#input\\_enable\\_iscsid) | This is always true when enable\\_longhorn=true, however, you may also want this enabled if you perform your own installation of longhorn after this module runs. | `bool` | `false` | no |\n| <a name=\"input_enable_klipper_metal_lb\"></a> [enable\\_klipper\\_metal\\_lb](#input\\_enable\\_klipper\\_metal\\_lb) | Use klipper load balancer. | `bool` | `false` | no |\n| <a name=\"input_enable_local_storage\"></a> [enable\\_local\\_storage](#input\\_enable\\_local\\_storage) | Whether to enable or disable k3s local-storage. Warning: when enabled, there will be two default storage classes: \"local-path\" and \"hcloud-volumes\"! | `bool` | `false` | no |\n| <a name=\"input_enable_longhorn\"></a> [enable\\_longhorn](#input\\_enable\\_longhorn) | Whether or not to enable Longhorn. | `bool` | `false` | no |\n| <a name=\"input_enable_metrics_server\"></a> [enable\\_metrics\\_server](#input\\_enable\\_metrics\\_server) | Whether to enable or disable k3s metric server. | `bool` | `true` | no |\n| <a name=\"input_enable_rancher\"></a> [enable\\_rancher](#input\\_enable\\_rancher) | Enable rancher. | `bool` | `false` | no |\n| <a name=\"input_enable_wireguard\"></a> [enable\\_wireguard](#input\\_enable\\_wireguard) | Use wireguard-native as the backend for CNI. | `bool` | `false` | no |\n| <a name=\"input_etcd_s3_backup\"></a> [etcd\\_s3\\_backup](#input\\_etcd\\_s3\\_backup) | Etcd cluster state backup to S3 storage | `map(any)` | `{}` | no |\n| <a name=\"input_exclude_agents_from_external_load_balancers\"></a> [exclude\\_agents\\_from\\_external\\_load\\_balancers](#input\\_exclude\\_agents\\_from\\_external\\_load\\_balancers) | Add node.kubernetes.io/exclude-from-external-load-balancers=true label to agent nodes. Enable this if you use both the Terraform-managed ingress LB and CCM-managed LoadBalancer services, and want to prevent double-registration of agents to the CCM LBs. Note: This excludes agents from ALL CCM-managed LoadBalancer services, not just ingress. | `bool` | `false` | no |\n| <a name=\"input_existing_network_id\"></a> [existing\\_network\\_id](#input\\_existing\\_network\\_id) | If you want to create the private network before calling this module, you can do so and pass its id here. NOTE: make sure to adapt network\\_ipv4\\_cidr accordingly to a range which does not collide with your other nodes. | `list(string)` | `[]` | no |\n| <a name=\"input_export_values\"></a> [export\\_values](#input\\_export\\_values) | Export for deployment used values.yaml-files as local files. | `bool` | `false` | no |\n| <a name=\"input_extra_firewall_rules\"></a> [extra\\_firewall\\_rules](#input\\_extra\\_firewall\\_rules) | Additional firewall rules to apply to the cluster. | `list(any)` | `[]` | no |\n| <a name=\"input_extra_kustomize_deployment_commands\"></a> [extra\\_kustomize\\_deployment\\_commands](#input\\_extra\\_kustomize\\_deployment\\_commands) | Commands to be executed after the `kubectl apply -k <dir>` step. | `string` | `\"\"` | no |\n| <a name=\"input_extra_kustomize_folder\"></a> [extra\\_kustomize\\_folder](#input\\_extra\\_kustomize\\_folder) | Folder from where to upload extra manifests | `string` | `\"extra-manifests\"` | no |\n| <a name=\"input_extra_kustomize_parameters\"></a> [extra\\_kustomize\\_parameters](#input\\_extra\\_kustomize\\_parameters) | All values will be passed to the `kustomization.tmp.yml` template. | `any` | `{}` | no |\n| <a name=\"input_firewall_kube_api_source\"></a> [firewall\\_kube\\_api\\_source](#input\\_firewall\\_kube\\_api\\_source) | Source networks that have Kube API access to the servers. | `list(string)` | <pre>[<br/> \"0.0.0.0/0\",<br/> \"::/0\"<br/>]</pre> | no |\n| <a name=\"input_firewall_ssh_source\"></a> [firewall\\_ssh\\_source](#input\\_firewall\\_ssh\\_source) | Source networks that have SSH access to the servers. | `list(string)` | <pre>[<br/> \"0.0.0.0/0\",<br/> \"::/0\"<br/>]</pre> | no |\n| <a name=\"input_flannel_backend\"></a> [flannel\\_backend](#input\\_flannel\\_backend) | Override the flannel backend used by k3s. When set, this takes precedence over enable\\_wireguard. Valid values: vxlan, host-gw, wireguard-native. See https://docs.k3s.io/networking/basic-network-options for details. Use wireguard-native for Robot nodes with vSwitch to avoid MTU issues. | `string` | `null` | no |\n| <a name=\"input_haproxy_additional_proxy_protocol_ips\"></a> [haproxy\\_additional\\_proxy\\_protocol\\_ips](#input\\_haproxy\\_additional\\_proxy\\_protocol\\_ips) | Additional trusted proxy protocol IPs to pass to haproxy. | `list(string)` | `[]` | no |\n| <a name=\"input_haproxy_merge_values\"></a> [haproxy\\_merge\\_values](#input\\_haproxy\\_merge\\_values) | Additional Helm values to merge with defaults (or haproxy\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_haproxy_requests_cpu\"></a> [haproxy\\_requests\\_cpu](#input\\_haproxy\\_requests\\_cpu) | Setting for HAProxy controller.resources.requests.cpu | `string` | `\"250m\"` | no |\n| <a name=\"input_haproxy_requests_memory\"></a> [haproxy\\_requests\\_memory](#input\\_haproxy\\_requests\\_memory) | Setting for HAProxy controller.resources.requests.memory | `string` | `\"400Mi\"` | no |\n| <a name=\"input_haproxy_values\"></a> [haproxy\\_values](#input\\_haproxy\\_values) | Helm values file to pass to haproxy as 'valuesContent' at the HelmChart, overriding the default. | `string` | `\"\"` | no |\n| <a name=\"input_haproxy_version\"></a> [haproxy\\_version](#input\\_haproxy\\_version) | Version of HAProxy helm chart. | `string` | `\"\"` | no |\n| <a name=\"input_hcloud_ssh_key_id\"></a> [hcloud\\_ssh\\_key\\_id](#input\\_hcloud\\_ssh\\_key\\_id) | If passed, a key already registered within hetzner is used. Otherwise, a new one will be created by the module. | `string` | `null` | no |\n| <a name=\"input_hcloud_token\"></a> [hcloud\\_token](#input\\_hcloud\\_token) | Hetzner Cloud API Token. | `string` | n/a | yes |\n| <a name=\"input_hetzner_ccm_merge_values\"></a> [hetzner\\_ccm\\_merge\\_values](#input\\_hetzner\\_ccm\\_merge\\_values) | Additional Helm values to merge with defaults (or hetzner\\_ccm\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_hetzner_ccm_use_helm\"></a> [hetzner\\_ccm\\_use\\_helm](#input\\_hetzner\\_ccm\\_use\\_helm) | Whether to use the helm chart for the Hetzner CCM or the legacy manifest which is the default. | `bool` | `false` | no |\n| <a name=\"input_hetzner_ccm_values\"></a> [hetzner\\_ccm\\_values](#input\\_hetzner\\_ccm\\_values) | Additional helm values file to pass to Hetzner Controller Manager as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_hetzner_ccm_version\"></a> [hetzner\\_ccm\\_version](#input\\_hetzner\\_ccm\\_version) | Version of Kubernetes Cloud Controller Manager for Hetzner Cloud. See https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases for the available versions. | `string` | `null` | no |\n| <a name=\"input_hetzner_csi_values\"></a> [hetzner\\_csi\\_values](#input\\_hetzner\\_csi\\_values) | Additional helm values file to pass to hetzner csi as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_hetzner_csi_version\"></a> [hetzner\\_csi\\_version](#input\\_hetzner\\_csi\\_version) | Version of Container Storage Interface driver for Hetzner Cloud. See https://github.com/hetznercloud/csi-driver/releases for the available versions. | `string` | `null` | no |\n| <a name=\"input_ingress_controller\"></a> [ingress\\_controller](#input\\_ingress\\_controller) | The name of the ingress controller. | `string` | `\"traefik\"` | no |\n| <a name=\"input_ingress_max_replica_count\"></a> [ingress\\_max\\_replica\\_count](#input\\_ingress\\_max\\_replica\\_count) | Number of maximum replicas per ingress controller. Used for ingress HPA. Must be higher than number of replicas. | `number` | `10` | no |\n| <a name=\"input_ingress_replica_count\"></a> [ingress\\_replica\\_count](#input\\_ingress\\_replica\\_count) | Number of replicas per ingress controller. 0 means autodetect based on the number of agent nodes. | `number` | `0` | no |\n| <a name=\"input_ingress_target_namespace\"></a> [ingress\\_target\\_namespace](#input\\_ingress\\_target\\_namespace) | The namespace to deploy the ingress controller to. Defaults to ingress name. | `string` | `\"\"` | no |\n| <a name=\"input_initial_k3s_channel\"></a> [initial\\_k3s\\_channel](#input\\_initial\\_k3s\\_channel) | Allows you to specify an initial k3s channel. See https://update.k3s.io/v1-release/channels for available channels. | `string` | `\"v1.33\"` | no |\n| <a name=\"input_install_k3s_version\"></a> [install\\_k3s\\_version](#input\\_install\\_k3s\\_version) | Allows you to specify the k3s version (Example: v1.29.6+k3s2). Supersedes initial\\_k3s\\_channel. See https://github.com/k3s-io/k3s/releases for available versions. | `string` | `\"\"` | no |\n| <a name=\"input_k3s_agent_kubelet_args\"></a> [k3s\\_agent\\_kubelet\\_args](#input\\_k3s\\_agent\\_kubelet\\_args) | Kubelet args for agent nodes. | `list(string)` | `[]` | no |\n| <a name=\"input_k3s_audit_log_maxage\"></a> [k3s\\_audit\\_log\\_maxage](#input\\_k3s\\_audit\\_log\\_maxage) | Maximum number of days to retain audit log files | `number` | `30` | no |\n| <a name=\"input_k3s_audit_log_maxbackup\"></a> [k3s\\_audit\\_log\\_maxbackup](#input\\_k3s\\_audit\\_log\\_maxbackup) | Maximum number of audit log files to retain | `number` | `10` | no |\n| <a name=\"input_k3s_audit_log_maxsize\"></a> [k3s\\_audit\\_log\\_maxsize](#input\\_k3s\\_audit\\_log\\_maxsize) | Maximum size in megabytes of the audit log file before rotation | `number` | `100` | no |\n| <a name=\"input_k3s_audit_log_path\"></a> [k3s\\_audit\\_log\\_path](#input\\_k3s\\_audit\\_log\\_path) | Path where audit logs will be stored on control plane nodes | `string` | `\"/var/log/k3s-audit/audit.log\"` | no |\n| <a name=\"input_k3s_audit_policy_config\"></a> [k3s\\_audit\\_policy\\_config](#input\\_k3s\\_audit\\_policy\\_config) | K3S audit-policy.yaml contents. Used to configure Kubernetes audit logging. | `string` | `\"\"` | no |\n| <a name=\"input_k3s_autoscaler_kubelet_args\"></a> [k3s\\_autoscaler\\_kubelet\\_args](#input\\_k3s\\_autoscaler\\_kubelet\\_args) | Kubelet args for autoscaler nodes. | `list(string)` | `[]` | no |\n| <a name=\"input_k3s_control_plane_kubelet_args\"></a> [k3s\\_control\\_plane\\_kubelet\\_args](#input\\_k3s\\_control\\_plane\\_kubelet\\_args) | Kubelet args for control plane nodes. | `list(string)` | `[]` | no |\n| <a name=\"input_k3s_exec_agent_args\"></a> [k3s\\_exec\\_agent\\_args](#input\\_k3s\\_exec\\_agent\\_args) | Agents nodes are started with `k3s agent {k3s_exec_agent_args}`. Use this to add kubelet-arg for example. | `string` | `\"\"` | no |\n| <a name=\"input_k3s_exec_server_args\"></a> [k3s\\_exec\\_server\\_args](#input\\_k3s\\_exec\\_server\\_args) | The control plane is started with `k3s server {k3s_exec_server_args}`. Use this to add kube-apiserver-arg for example. | `string` | `\"\"` | no |\n| <a name=\"input_k3s_global_kubelet_args\"></a> [k3s\\_global\\_kubelet\\_args](#input\\_k3s\\_global\\_kubelet\\_args) | Global kubelet args for all nodes. | `list(string)` | `[]` | no |\n| <a name=\"input_k3s_kubelet_config\"></a> [k3s\\_kubelet\\_config](#input\\_k3s\\_kubelet\\_config) | K3S kubelet-config.yaml contents. Used to configure the kubelet. | `string` | `\"\"` | no |\n| <a name=\"input_k3s_prefer_bundled_bin\"></a> [k3s\\_prefer\\_bundled\\_bin](#input\\_k3s\\_prefer\\_bundled\\_bin) | Whether to use the bundled k3s mount binary instead of the one from the distro's util-linux package. | `bool` | `false` | no |\n| <a name=\"input_k3s_registries\"></a> [k3s\\_registries](#input\\_k3s\\_registries) | K3S registries.yml contents. It used to access private docker registries. | `string` | `\" \"` | no |\n| <a name=\"input_k3s_token\"></a> [k3s\\_token](#input\\_k3s\\_token) | k3s master token (must match when restoring a cluster). | `string` | `null` | no |\n| <a name=\"input_keep_disk_agents\"></a> [keep\\_disk\\_agents](#input\\_keep\\_disk\\_agents) | Whether to keep OS disks of nodes the same size when upgrading an agent node | `bool` | `false` | no |\n| <a name=\"input_keep_disk_cp\"></a> [keep\\_disk\\_cp](#input\\_keep\\_disk\\_cp) | Whether to keep OS disks of nodes the same size when upgrading a control-plane node | `bool` | `false` | no |\n| <a name=\"input_kubeconfig_server_address\"></a> [kubeconfig\\_server\\_address](#input\\_kubeconfig\\_server\\_address) | The hostname used for kubeconfig. | `string` | `\"\"` | no |\n| <a name=\"input_kured_options\"></a> [kured\\_options](#input\\_kured\\_options) | n/a | `map(string)` | `{}` | no |\n| <a name=\"input_kured_version\"></a> [kured\\_version](#input\\_kured\\_version) | Version of Kured. See https://github.com/kubereboot/kured/releases for the available versions. | `string` | `null` | no |\n| <a name=\"input_lb_hostname\"></a> [lb\\_hostname](#input\\_lb\\_hostname) | The Hetzner Load Balancer hostname, for either Traefik, HAProxy or Ingress-Nginx. | `string` | `\"\"` | no |\n| <a name=\"input_load_balancer_algorithm_type\"></a> [load\\_balancer\\_algorithm\\_type](#input\\_load\\_balancer\\_algorithm\\_type) | Specifies the algorithm type of the load balancer. | `string` | `\"round_robin\"` | no |\n| <a name=\"input_load_balancer_disable_ipv6\"></a> [load\\_balancer\\_disable\\_ipv6](#input\\_load\\_balancer\\_disable\\_ipv6) | Disable IPv6 for the load balancer. | `bool` | `false` | no |\n| <a name=\"input_load_balancer_disable_public_network\"></a> [load\\_balancer\\_disable\\_public\\_network](#input\\_load\\_balancer\\_disable\\_public\\_network) | Disables the public network of the load balancer. | `bool` | `false` | no |\n| <a name=\"input_load_balancer_health_check_interval\"></a> [load\\_balancer\\_health\\_check\\_interval](#input\\_load\\_balancer\\_health\\_check\\_interval) | Specifies the interval at which a health check is performed. Minimum is 3s. | `string` | `\"15s\"` | no |\n| <a name=\"input_load_balancer_health_check_retries\"></a> [load\\_balancer\\_health\\_check\\_retries](#input\\_load\\_balancer\\_health\\_check\\_retries) | Specifies the number of times a health check is retried before a target is marked as unhealthy. | `number` | `3` | no |\n| <a name=\"input_load_balancer_health_check_timeout\"></a> [load\\_balancer\\_health\\_check\\_timeout](#input\\_load\\_balancer\\_health\\_check\\_timeout) | Specifies the timeout of a single health check. Must not be greater than the health check interval. Minimum is 1s. | `string` | `\"10s\"` | no |\n| <a name=\"input_load_balancer_location\"></a> [load\\_balancer\\_location](#input\\_load\\_balancer\\_location) | Default load balancer location. | `string` | `\"nbg1\"` | no |\n| <a name=\"input_load_balancer_type\"></a> [load\\_balancer\\_type](#input\\_load\\_balancer\\_type) | Default load balancer server type. | `string` | `\"lb11\"` | no |\n| <a name=\"input_longhorn_fstype\"></a> [longhorn\\_fstype](#input\\_longhorn\\_fstype) | The longhorn fstype. | `string` | `\"ext4\"` | no |\n| <a name=\"input_longhorn_helmchart_bootstrap\"></a> [longhorn\\_helmchart\\_bootstrap](#input\\_longhorn\\_helmchart\\_bootstrap) | Whether the HelmChart longhorn shall be run on control-plane nodes. | `bool` | `false` | no |\n| <a name=\"input_longhorn_merge_values\"></a> [longhorn\\_merge\\_values](#input\\_longhorn\\_merge\\_values) | Helm values to merge with defaults (or longhorn\\_values if set). User values take precedence. Use for targeted overrides like image tags. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_longhorn_namespace\"></a> [longhorn\\_namespace](#input\\_longhorn\\_namespace) | Namespace for longhorn deployment, defaults to 'longhorn-system' | `string` | `\"longhorn-system\"` | no |\n| <a name=\"input_longhorn_replica_count\"></a> [longhorn\\_replica\\_count](#input\\_longhorn\\_replica\\_count) | Number of replicas per longhorn volume. | `number` | `3` | no |\n| <a name=\"input_longhorn_repository\"></a> [longhorn\\_repository](#input\\_longhorn\\_repository) | By default the official chart which may be incompatible with rancher is used. If you need to fully support rancher switch to https://charts.rancher.io. | `string` | `\"https://charts.longhorn.io\"` | no |\n| <a name=\"input_longhorn_values\"></a> [longhorn\\_values](#input\\_longhorn\\_values) | Helm values passed as valuesContent to the Longhorn HelmChart. When set, this replaces the module defaults. | `string` | `\"\"` | no |\n| <a name=\"input_longhorn_version\"></a> [longhorn\\_version](#input\\_longhorn\\_version) | Longhorn Helm chart version. | `string` | `\"*\"` | no |\n| <a name=\"input_microos_arm_snapshot_id\"></a> [microos\\_arm\\_snapshot\\_id](#input\\_microos\\_arm\\_snapshot\\_id) | MicroOS ARM snapshot ID to be used. Per default empty, the most recent image created using createkh will be used | `string` | `\"\"` | no |\n| <a name=\"input_microos_x86_snapshot_id\"></a> [microos\\_x86\\_snapshot\\_id](#input\\_microos\\_x86\\_snapshot\\_id) | MicroOS x86 snapshot ID to be used. Per default empty, the most recent image created using createkh will be used | `string` | `\"\"` | no |\n| <a name=\"input_nat_router\"></a> [nat\\_router](#input\\_nat\\_router) | Do you want to pipe all egress through a single nat router which is to be constructed? Note: Requires use\\_control\\_plane\\_lb=true when enabled. Automatically forwards port 6443 to the control plane LB when control\\_plane\\_lb\\_enable\\_public\\_interface=false. | <pre>object({<br/> server_type = string<br/> location = string<br/> labels = optional(map(string), {})<br/> enable_sudo = optional(bool, false)<br/> enable_redundancy = optional(bool, false)<br/> standby_location = optional(string, \"\")<br/> })</pre> | `null` | no |\n| <a name=\"input_nat_router_hcloud_token\"></a> [nat\\_router\\_hcloud\\_token](#input\\_nat\\_router\\_hcloud\\_token) | API Token used by the nat-router to change ip assignment when nat\\_router.enable\\_redundancy is true. | `string` | `\"\"` | no |\n| <a name=\"input_nat_router_subnet_index\"></a> [nat\\_router\\_subnet\\_index](#input\\_nat\\_router\\_subnet\\_index) | Subnet index for NAT router. Default 200 is safe for most deployments. Must not conflict with control plane (counting down from 255) or agent pools (counting up from 0). | `number` | `200` | no |\n| <a name=\"input_network_ipv4_cidr\"></a> [network\\_ipv4\\_cidr](#input\\_network\\_ipv4\\_cidr) | The main network cidr that all subnets will be created upon. | `string` | `\"10.0.0.0/8\"` | no |\n| <a name=\"input_network_region\"></a> [network\\_region](#input\\_network\\_region) | Default region for network. | `string` | `\"eu-central\"` | no |\n| <a name=\"input_nginx_merge_values\"></a> [nginx\\_merge\\_values](#input\\_nginx\\_merge\\_values) | Additional Helm values to merge with defaults (or nginx\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_nginx_values\"></a> [nginx\\_values](#input\\_nginx\\_values) | Additional helm values file to pass to nginx as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_nginx_version\"></a> [nginx\\_version](#input\\_nginx\\_version) | Version of Nginx helm chart. See https://github.com/kubernetes/ingress-nginx?tab=readme-ov-file#supported-versions-table for the available versions. | `string` | `\"\"` | no |\n| <a name=\"input_placement_group_disable\"></a> [placement\\_group\\_disable](#input\\_placement\\_group\\_disable) | Whether to disable placement groups. | `bool` | `false` | no |\n| <a name=\"input_postinstall_exec\"></a> [postinstall\\_exec](#input\\_postinstall\\_exec) | Additional to execute after the install calls, for example restoring a backup. | `list(string)` | `[]` | no |\n| <a name=\"input_preinstall_exec\"></a> [preinstall\\_exec](#input\\_preinstall\\_exec) | Additional to execute before the install calls, for example fetching and installing certs. | `list(string)` | `[]` | no |\n| <a name=\"input_rancher_bootstrap_password\"></a> [rancher\\_bootstrap\\_password](#input\\_rancher\\_bootstrap\\_password) | Rancher bootstrap password. | `string` | `\"\"` | no |\n| <a name=\"input_rancher_helmchart_bootstrap\"></a> [rancher\\_helmchart\\_bootstrap](#input\\_rancher\\_helmchart\\_bootstrap) | Whether the HelmChart rancher shall be run on control-plane nodes. | `bool` | `false` | no |\n| <a name=\"input_rancher_hostname\"></a> [rancher\\_hostname](#input\\_rancher\\_hostname) | The rancher hostname. | `string` | `\"\"` | no |\n| <a name=\"input_rancher_install_channel\"></a> [rancher\\_install\\_channel](#input\\_rancher\\_install\\_channel) | The rancher installation channel. | `string` | `\"stable\"` | no |\n| <a name=\"input_rancher_merge_values\"></a> [rancher\\_merge\\_values](#input\\_rancher\\_merge\\_values) | Additional Helm values to merge with defaults (or rancher\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_rancher_registration_manifest_url\"></a> [rancher\\_registration\\_manifest\\_url](#input\\_rancher\\_registration\\_manifest\\_url) | The url of a rancher registration manifest to apply. (see https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/). | `string` | `\"\"` | no |\n| <a name=\"input_rancher_values\"></a> [rancher\\_values](#input\\_rancher\\_values) | Additional helm values file to pass to Rancher as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_rancher_version\"></a> [rancher\\_version](#input\\_rancher\\_version) | Version of rancher. | `string` | `\"*\"` | no |\n| <a name=\"input_restrict_outbound_traffic\"></a> [restrict\\_outbound\\_traffic](#input\\_restrict\\_outbound\\_traffic) | Whether or not to restrict the outbound traffic. | `bool` | `true` | no |\n| <a name=\"input_robot_ccm_enabled\"></a> [robot\\_ccm\\_enabled](#input\\_robot\\_ccm\\_enabled) | Enables the integration of Hetzner Robot dedicated servers via the Cloud Controller Manager (CCM). If true, `robot_user` and `robot_password` must also be provided, otherwise the integration will not be activated. | `bool` | `false` | no |\n| <a name=\"input_robot_password\"></a> [robot\\_password](#input\\_robot\\_password) | Password for the Hetzner Robot webservice | `string` | `\"\"` | no |\n| <a name=\"input_robot_user\"></a> [robot\\_user](#input\\_robot\\_user) | User for the Hetzner Robot webservice | `string` | `\"\"` | no |\n| <a name=\"input_service_ipv4_cidr\"></a> [service\\_ipv4\\_cidr](#input\\_service\\_ipv4\\_cidr) | Internal Service CIDR, used for the controller and currently for calico/cilium. | `string` | `\"10.43.0.0/16\"` | no |\n| <a name=\"input_ssh_additional_public_keys\"></a> [ssh\\_additional\\_public\\_keys](#input\\_ssh\\_additional\\_public\\_keys) | Additional SSH public Keys. Use them to grant other team members root access to your cluster nodes. | `list(string)` | `[]` | no |\n| <a name=\"input_ssh_hcloud_key_label\"></a> [ssh\\_hcloud\\_key\\_label](#input\\_ssh\\_hcloud\\_key\\_label) | Additional SSH public Keys by hcloud label. e.g. role=admin | `string` | `\"\"` | no |\n| <a name=\"input_ssh_max_auth_tries\"></a> [ssh\\_max\\_auth\\_tries](#input\\_ssh\\_max\\_auth\\_tries) | The maximum number of authentication attempts permitted per connection. | `number` | `2` | no |\n| <a name=\"input_ssh_port\"></a> [ssh\\_port](#input\\_ssh\\_port) | The main SSH port to connect to the nodes. | `number` | `22` | no |\n| <a name=\"input_ssh_private_key\"></a> [ssh\\_private\\_key](#input\\_ssh\\_private\\_key) | SSH private Key. | `string` | n/a | yes |\n| <a name=\"input_ssh_public_key\"></a> [ssh\\_public\\_key](#input\\_ssh\\_public\\_key) | SSH public Key. | `string` | n/a | yes |\n| <a name=\"input_subnet_amount\"></a> [subnet\\_amount](#input\\_subnet\\_amount) | The amount of subnets into which the network will be split. Must be a power of 2. | `number` | `256` | no |\n| <a name=\"input_sys_upgrade_controller_version\"></a> [sys\\_upgrade\\_controller\\_version](#input\\_sys\\_upgrade\\_controller\\_version) | Version of the System Upgrade Controller for automated upgrades of k3s. v0.15.0+ supports the 'window' parameter for scheduling upgrades. See https://github.com/rancher/system-upgrade-controller/releases for available versions. | `string` | `\"v0.18.0\"` | no |\n| <a name=\"input_system_upgrade_enable_eviction\"></a> [system\\_upgrade\\_enable\\_eviction](#input\\_system\\_upgrade\\_enable\\_eviction) | Whether to directly delete pods during system upgrade (k3s) or evict them. Defaults to true. Disable this on small clusters to avoid system upgrades hanging since pods resisting eviction keep node unschedulable forever. NOTE: turning this off, introduces potential downtime of services of the upgraded nodes. | `bool` | `true` | no |\n| <a name=\"input_system_upgrade_schedule_window\"></a> [system\\_upgrade\\_schedule\\_window](#input\\_system\\_upgrade\\_schedule\\_window) | Schedule window for k3s automated upgrades (system-upgrade-controller v0.15.0+). When set, upgrade jobs will only be created within the specified time window. 'days' accepts lowercase day names (e.g. [\"monday\",\"tuesday\"]). 'startTime'/'endTime' use HH:MM format. 'timeZone' defaults to UTC. See https://docs.k3s.io/upgrades/automated#scheduling-upgrades | <pre>object({<br/> days = optional(list(string), [])<br/> startTime = optional(string, \"\")<br/> endTime = optional(string, \"\")<br/> timeZone = optional(string, \"UTC\")<br/> })</pre> | `null` | no |\n| <a name=\"input_system_upgrade_use_drain\"></a> [system\\_upgrade\\_use\\_drain](#input\\_system\\_upgrade\\_use\\_drain) | Wether using drain (true, the default), which will deletes and transfers all pods to other nodes before a node is being upgraded, or cordon (false), which just prevents schedulung new pods on the node during upgrade and keeps all pods running | `bool` | `true` | no |\n| <a name=\"input_traefik_additional_options\"></a> [traefik\\_additional\\_options](#input\\_traefik\\_additional\\_options) | Additional options to pass to Traefik as a list of strings. These are the ones that go into the additionalArguments section of the Traefik helm values file. | `list(string)` | `[]` | no |\n| <a name=\"input_traefik_additional_ports\"></a> [traefik\\_additional\\_ports](#input\\_traefik\\_additional\\_ports) | Additional ports to pass to Traefik. These are the ones that go into the ports section of the Traefik helm values file. | <pre>list(object({<br/> name = string<br/> port = number<br/> exposedPort = number<br/> }))</pre> | `[]` | no |\n| <a name=\"input_traefik_additional_trusted_ips\"></a> [traefik\\_additional\\_trusted\\_ips](#input\\_traefik\\_additional\\_trusted\\_ips) | Additional Trusted IPs to pass to Traefik. These are the ones that go into the trustedIPs section of the Traefik helm values file. | `list(string)` | `[]` | no |\n| <a name=\"input_traefik_autoscaling\"></a> [traefik\\_autoscaling](#input\\_traefik\\_autoscaling) | Should traefik enable Horizontal Pod Autoscaler. | `bool` | `true` | no |\n| <a name=\"input_traefik_image_tag\"></a> [traefik\\_image\\_tag](#input\\_traefik\\_image\\_tag) | Traefik image tag. Useful to use the beta version for new features. Example: v3.0.0-beta5 | `string` | `\"\"` | no |\n| <a name=\"input_traefik_merge_values\"></a> [traefik\\_merge\\_values](#input\\_traefik\\_merge\\_values) | Additional Helm values to merge with defaults (or traefik\\_values if set). User values take precedence. Requires valid YAML format. | `string` | `\"\"` | no |\n| <a name=\"input_traefik_pod_disruption_budget\"></a> [traefik\\_pod\\_disruption\\_budget](#input\\_traefik\\_pod\\_disruption\\_budget) | Should traefik enable pod disruption budget. Default values are maxUnavailable: 33% and minAvailable: 1. | `bool` | `true` | no |\n| <a name=\"input_traefik_provider_kubernetes_gateway_enabled\"></a> [traefik\\_provider\\_kubernetes\\_gateway\\_enabled](#input\\_traefik\\_provider\\_kubernetes\\_gateway\\_enabled) | Should traefik enable the kubernetes gateway provider. Default is false. | `bool` | `false` | no |\n| <a name=\"input_traefik_redirect_to_https\"></a> [traefik\\_redirect\\_to\\_https](#input\\_traefik\\_redirect\\_to\\_https) | Should traefik redirect http traffic to https. | `bool` | `true` | no |\n| <a name=\"input_traefik_resource_limits\"></a> [traefik\\_resource\\_limits](#input\\_traefik\\_resource\\_limits) | Should traefik enable default resource requests and limits. Default values are requests: 100m & 50Mi and limits: 300m & 150Mi. | `bool` | `true` | no |\n| <a name=\"input_traefik_resource_values\"></a> [traefik\\_resource\\_values](#input\\_traefik\\_resource\\_values) | Requests and limits for Traefik. | <pre>object({<br/> requests = object({<br/> cpu = string<br/> memory = string<br/> })<br/> limits = object({<br/> cpu = string<br/> memory = string<br/> })<br/> })</pre> | <pre>{<br/> \"limits\": {<br/> \"cpu\": \"300m\",<br/> \"memory\": \"150Mi\"<br/> },<br/> \"requests\": {<br/> \"cpu\": \"100m\",<br/> \"memory\": \"50Mi\"<br/> }<br/>}</pre> | no |\n| <a name=\"input_traefik_values\"></a> [traefik\\_values](#input\\_traefik\\_values) | Additional helm values file to pass to Traefik as 'valuesContent' at the HelmChart. | `string` | `\"\"` | no |\n| <a name=\"input_traefik_version\"></a> [traefik\\_version](#input\\_traefik\\_version) | Version of Traefik helm chart. See https://github.com/traefik/traefik-helm-chart/releases for the available versions. | `string` | `\"\"` | no |\n| <a name=\"input_use_cluster_name_in_node_name\"></a> [use\\_cluster\\_name\\_in\\_node\\_name](#input\\_use\\_cluster\\_name\\_in\\_node\\_name) | Whether to use the cluster name in the node name. | `bool` | `true` | no |\n| <a name=\"input_use_control_plane_lb\"></a> [use\\_control\\_plane\\_lb](#input\\_use\\_control\\_plane\\_lb) | Creates a dedicated load balancer for the Kubernetes API (port 6443). When enabled, kubectl and other API clients connect through this LB instead of directly to the first control plane node. Recommended for production clusters with multiple control plane nodes for high availability. Note: This is separate from the ingress load balancer for HTTP/HTTPS traffic. | `bool` | `false` | no |\n| <a name=\"input_vswitch_id\"></a> [vswitch\\_id](#input\\_vswitch\\_id) | Hetzner Cloud vSwitch ID. If defined, a subnet will be created in the IP-range defined by vswitch\\_subnet\\_index. The vSwitch must exist before this module is called. | `number` | `null` | no |\n| <a name=\"input_vswitch_subnet_index\"></a> [vswitch\\_subnet\\_index](#input\\_vswitch\\_subnet\\_index) | Subnet index (0-255) for vSwitch. Default 201 is safe for most deployments. Must not conflict with control plane (counting down from 255) or agent pools (counting up from 0). | `number` | `201` | no |\n\n### Outputs\n\n| Name | Description |\n|------|-------------|\n| <a name=\"output_agent_nodes\"></a> [agent\\_nodes](#output\\_agent\\_nodes) | The agent nodes |\n| <a name=\"output_agents_public_ipv4\"></a> [agents\\_public\\_ipv4](#output\\_agents\\_public\\_ipv4) | The public IPv4 addresses of the agent servers. |\n| <a name=\"output_agents_public_ipv6\"></a> [agents\\_public\\_ipv6](#output\\_agents\\_public\\_ipv6) | The public IPv6 addresses of the agent servers. |\n| <a name=\"output_cert_manager_values\"></a> [cert\\_manager\\_values](#output\\_cert\\_manager\\_values) | Helm values.yaml used for cert-manager |\n| <a name=\"output_cilium_values\"></a> [cilium\\_values](#output\\_cilium\\_values) | Helm values.yaml used for Cilium |\n| <a name=\"output_cluster_name\"></a> [cluster\\_name](#output\\_cluster\\_name) | Shared suffix for all resources belonging to this cluster. |\n| <a name=\"output_control_plane_nodes\"></a> [control\\_plane\\_nodes](#output\\_control\\_plane\\_nodes) | The control plane nodes |\n| <a name=\"output_control_planes_public_ipv4\"></a> [control\\_planes\\_public\\_ipv4](#output\\_control\\_planes\\_public\\_ipv4) | The public IPv4 addresses of the controlplane servers. |\n| <a name=\"output_control_planes_public_ipv6\"></a> [control\\_planes\\_public\\_ipv6](#output\\_control\\_planes\\_public\\_ipv6) | The public IPv6 addresses of the controlplane servers. |\n| <a name=\"output_csi_driver_smb_values\"></a> [csi\\_driver\\_smb\\_values](#output\\_csi\\_driver\\_smb\\_values) | Helm values.yaml used for SMB CSI driver |\n| <a name=\"output_domain_assignments\"></a> [domain\\_assignments](#output\\_domain\\_assignments) | Assignments of domains to IPs based on reverse DNS |\n| <a name=\"output_haproxy_values\"></a> [haproxy\\_values](#output\\_haproxy\\_values) | Helm values.yaml used for HAProxy |\n| <a name=\"output_ingress_public_ipv4\"></a> [ingress\\_public\\_ipv4](#output\\_ingress\\_public\\_ipv4) | The public IPv4 address of the Hetzner load balancer (with fallback to first control plane node) |\n| <a name=\"output_ingress_public_ipv6\"></a> [ingress\\_public\\_ipv6](#output\\_ingress\\_public\\_ipv6) | The public IPv6 address of the Hetzner load balancer (with fallback to first control plane node) |\n| <a name=\"output_k3s_endpoint\"></a> [k3s\\_endpoint](#output\\_k3s\\_endpoint) | A controller endpoint to register new nodes |\n| <a name=\"output_k3s_token\"></a> [k3s\\_token](#output\\_k3s\\_token) | The k3s token to register new nodes |\n| <a name=\"output_kubeconfig\"></a> [kubeconfig](#output\\_kubeconfig) | Kubeconfig file content with external IP address, or internal IP address if only private ips are available |\n| <a name=\"output_kubeconfig_data\"></a> [kubeconfig\\_data](#output\\_kubeconfig\\_data) | Structured kubeconfig data to supply to other providers |\n| <a name=\"output_kubeconfig_file\"></a> [kubeconfig\\_file](#output\\_kubeconfig\\_file) | Kubeconfig file content with external IP address, or internal IP address if only private ips are available |\n| <a name=\"output_lb_control_plane_ipv4\"></a> [lb\\_control\\_plane\\_ipv4](#output\\_lb\\_control\\_plane\\_ipv4) | The public IPv4 address of the Hetzner control plane load balancer |\n| <a name=\"output_lb_control_plane_ipv6\"></a> [lb\\_control\\_plane\\_ipv6](#output\\_lb\\_control\\_plane\\_ipv6) | The public IPv6 address of the Hetzner control plane load balancer |\n| <a name=\"output_longhorn_values\"></a> [longhorn\\_values](#output\\_longhorn\\_values) | Helm values.yaml used for Longhorn |\n| <a name=\"output_nat_router_public_ipv4\"></a> [nat\\_router\\_public\\_ipv4](#output\\_nat\\_router\\_public\\_ipv4) | The address of the nat router, if it exists. |\n| <a name=\"output_nat_router_public_ipv4_addresses\"></a> [nat\\_router\\_public\\_ipv4\\_addresses](#output\\_nat\\_router\\_public\\_ipv4\\_addresses) | The addresses of all nat routers, if they exist. |\n| <a name=\"output_nat_router_public_ipv6\"></a> [nat\\_router\\_public\\_ipv6](#output\\_nat\\_router\\_public\\_ipv6) | The address of the nat router, if it exists. |\n| <a name=\"output_nat_router_public_ipv6_addresses\"></a> [nat\\_router\\_public\\_ipv6\\_addresses](#output\\_nat\\_router\\_public\\_ipv6\\_addresses) | The addresses of all nat routers, if they exist. |\n| <a name=\"output_nat_router_ssh_port\"></a> [nat\\_router\\_ssh\\_port](#output\\_nat\\_router\\_ssh\\_port) | The non-root user as which you can ssh into the router. |\n| <a name=\"output_nat_router_username\"></a> [nat\\_router\\_username](#output\\_nat\\_router\\_username) | The non-root user as which you can ssh into the router. |\n| <a name=\"output_network_id\"></a> [network\\_id](#output\\_network\\_id) | The ID of the HCloud network. |\n| <a name=\"output_nginx_values\"></a> [nginx\\_values](#output\\_nginx\\_values) | Helm values.yaml used for nginx-ingress |\n| <a name=\"output_ssh_key_id\"></a> [ssh\\_key\\_id](#output\\_ssh\\_key\\_id) | The ID of the HCloud SSH key. |\n| <a name=\"output_traefik_values\"></a> [traefik\\_values](#output\\_traefik\\_values) | Helm values.yaml used for Traefik |\n| <a name=\"output_vswitch_subnet\"></a> [vswitch\\_subnet](#output\\_vswitch\\_subnet) | Attributes of the vSwitch subnet. |\n<!-- END_TF_DOCS -->\n" }, { "path": "examples/kustomization_user_deploy/README.md", "content": "# How to Install and Deploy Additional Resources with Terraform and Kube-Hetzner\n\nKube-Hetzner allows you to provide user-defined resources after the initial setup of the Kubernetes cluster. You can deploy additional resources using Kustomize scripts in the `extra-manifests` directory with the extension `.yaml.tpl`. These scripts are recursively copied onto the control plane and deployed with `kubectl apply -k`. The main entry point for these additional resources is the `kustomization.yaml.tpl` file. In this file, you need to list the names of other manifests without the `.tpl` extension in the resources section.\n\nWhen you execute terraform apply, the manifests in the extra-manifests directory, including the rendered versions of the `*.yaml.tpl` files, will be automatically deployed to the cluster.\n\n## Examples\n\nHere are some examples of common use cases for deploying additional resources:\n\n> **Note**: When trying out the demos, make sure that the files from the demo folders are located in the `extra-manifests` directory.\n\n### Deploying Simple Resources\n\nThe easiest use case is to deploy simple resources to the cluster. Since the Kustomize resources are [Terraform template](https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file) files, they can make use of parameters provided in the `extra_kustomize_parameters` map of the `kube.tf` file.\n\n#### `kube.tf`\n\n```\n...\nextra_kustomize_parameters = {\n my_config_key = \"somestring\"\n}\n...\n```\n\nThe variable defined in `kube.tf` can be used in any `.yaml.tpl` manifest.\n\n#### `configmap.tf`\n\n```\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: demo-config\n data:\n someConfigKey: ${my_config_key}\n```\n\nFor a full demo see the [simple-resources](simple-resources/) example.\n\n### Deploying a Helm Chart\n\nIf you want to deploy a Helm chart to your cluster, you can use the [Helm Chart controller](https://docs.k3s.io/helm) included in K3s. The Helm Chart controller provides the CRDs `HelmChart` and `HelmChartConfig`.\n\nFor a full demo see the [helm-chart](helm-chart/) example.\n\n### Multiple Namespaces\n\nIn more complex use cases, you may want to deploy to multiple namespaces with a common base. Kustomize supports this behavior, and it can be since Kube-Hetzner is considering all subdirectories of `extra-manifests`.\n\nFor a full demo see the [multiple-namespaces](multiple-namespaces/) example.\n\n### Using Letsencrypt with cert-manager\n\nYou can use letsencrypt issuer to issue tls certificate see [example](https://doc.traefik.io/traefik/user-guides/cert-manager/). You need to create a issuer type of `ClusterIssuer` to make is available in all namespaces, unlike in the traefik example. Also note that the `server` in the example is a stagging server, you would need a prod server to use in, well, production. The prod server link can be found at `https://letsencrypt.org/getting-started/`\n\nFor a full demo see the [letsencrypt](letsencrypt/)\n\n## Debugging\n\nTo check the existing kustomization, you can run the following command:\n\n```\n$ terraform state list | grep kustom\n ...\n module.kube-hetzner.terraform_data.kustomization\n module.kube-hetzner.terraform_data.kustomization_user[\"demo-config-map.yaml.tpl\"]\n module.kube-hetzner.terraform_data.kustomization_user[\"demo-pod.yaml.tpl\"]\n module.kube-hetzner.terraform_data.kustomization_user[\"kustomization.yaml.tpl\"]\n ...\n```\n\nIf you want to rerun just the kustomization part, you can use the following command:\n\n```\nterraform apply -replace='module.kube-hetzner.terraform_data.kustomization_user[\"kustomization.yaml.tpl\"]' --auto-approve\n```\n" }, { "path": "examples/kustomization_user_deploy/helm-chart/helm-chart.yaml.tpl", "content": "apiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: argocd\n namespace: argocd\nspec:\n repo: https://argoproj.github.io/argo-helm\n chart: argo-cd\n targetNamespace: argocd\n valuesContent: |-\n global:\n domain: argocd.example.com\n" }, { "path": "examples/kustomization_user_deploy/helm-chart/kustomization.yaml.tpl", "content": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace.yaml\n - helm-chart.yaml\n" }, { "path": "examples/kustomization_user_deploy/helm-chart/namespace.yaml.tpl", "content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: argocd\n" }, { "path": "examples/kustomization_user_deploy/letsencrypt/kustomization.yaml.tpl", "content": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - letsencrypt.yaml\n" }, { "path": "examples/kustomization_user_deploy/letsencrypt/letsencrypt.yaml.tpl", "content": "apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt\n namespace: cert-manager\nspec:\n acme:\n email: <youremail@domain.com> <--- change this to your email\n server: https://acme-v02.api.letsencrypt.org/directory | https://acme-staging-v02.api.letsencrypt.org/directory <-- pick one\n privateKeySecretRef:\n name: letsencrypt-account-key\n solvers:\n - http01:\n ingress:\n ingressClassName: traefik\n" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/base/kustomization.yaml.tpl", "content": "" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/base/pod.yaml.tpl", "content": "apiVersion: v1\nkind: Pod\nmetadata:\n name: myapp-pod\n labels:\n app: myapp\nspec:\n containers:\n - name: nginx\n image: nginx:1.7.9\n" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/kustomization.yaml.tpl", "content": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace-a\n - namespace-b\n" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-a/kustomization.yaml.tpl", "content": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace.yaml\n - ../base\nnamespace: namespace-a\n" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-a/namespace-a.yaml.tpl", "content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: namespace-a\n" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-b/kustomization.yaml.tpl", "content": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - namespace.yaml\n - ../base\nnamespace: namespace-b\n" }, { "path": "examples/kustomization_user_deploy/multiple-namespaces/namespace-b/namespace-b.yaml.tpl", "content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: namespace-b\n" }, { "path": "examples/kustomization_user_deploy/simple-resources/demo-config-map.yaml.tpl", "content": "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: demo-config\ndata:\n someConfigKey: ${sealed_secrets_crt}\n" }, { "path": "examples/kustomization_user_deploy/simple-resources/demo-pod.yml.tpl", "content": "apiVersion: v1\nkind: Pod\nmetadata:\n name: demo\nspec:\n containers:\n - name: demo-container\n image: registry.k8s.io/busybox\n command: [ \"/bin/sh\", \"-c\", \"env\" ]\n env:\n - name: DEMO_ENVIRONEMNT_VARIABLE\n valueFrom:\n configMapKeyRef:\n name: demo-config\n key: someConfigKey\n restartPolicy: Never\n" }, { "path": "examples/kustomization_user_deploy/simple-resources/kustomization.yaml.tpl", "content": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\n\nresources:\n - demo-config-map.yaml\n" }, { "path": "examples/micro_os_rollback/Readme.md", "content": "# Rollback Node MicroOS Manually\n\nHow to manually rollback a MicroOS node to the last snapshot or be date.\n\n## Background\n\nCertain versions of `linux-utils` (e.g., >2.40) may cause errors such as:\n\n```\n...cannot mount subpath... file exists... unmount...\n```\n\nFor more details, refer to the [Kubernetes issue #130999](https://github.com/kubernetes/kubernetes/issues/130999).\n\n## Step 1: Find Problematic Nodes\n\nRun the following command to identify nodes with issues:\n\n```bash\nkubectl get pods -o wide --all-namespaces | grep CreateContainerConfigError | awk '{printf \"%s%s\", $8, (NR == total ? \"\" : \",\")} {total=NR} END {print \"\"}'\n```\n\n**Note:** The output may include duplicates or irrelevant entries (e.g., pods with uptime of 12h).\n\n## Step 2: Manual Rollback Per Node\n\nSSH into each problematic node and execute the following command:\n\n```bash\nsnapper --iso list | tail -2 | head -1 | awk '{print $1}' | xargs -I{} snapper rollback {} && reboot\n```\n\n### Explanation of the Command\n\n1. `snapper --iso list`: Lists all snapshots with ISO timestamps.\n2. `tail -2`: Filters the last two snapshots.\n3. `head -1`: Selects the snapshot before the current one.\n4. `awk '{print $1}'`: Extracts the snapshot ID.\n5. `xargs -I{} snapper rollback {}`: Rolls back to the selected snapshot.\n6. `&& reboot`: Reboots the node if the rollback is successful.\n\n## Step 3: Automate Rollback with Ansible (Work in Progress)\n\nIf you have an inventory file, you can automate the rollback process using Ansible:\n\n```bash\nexport COMMA_SEPARATED_NODE_LIST=$(kubectl get pods -o wide --all-namespaces | grep CreateContainerConfigError | awk '{printf \"%s,\", $8} END {print \"\"}')\necho $COMMA_SEPARATED_NODE_LIST\n\nansible ${COMMA_SEPARATED_NODE_LIST} -i ansible/inventory.yml \\\n-m shell \\\n-a 'snapper --iso list | tail -2 | head -1 | awk \"{print $1}\" | xargs -I{} snapper rollback {} && reboot'\n```\n\n## Additional Notes\n\n### Snapshot List Example\n\nBelow is an example output of `snapper --iso list`:\n\n```bash\n # │ Type │ Pre # │ Date │ User │ Used Space │ Cleanup │ Description │ Userdata\n─────┼────────┼───────┼─────────────────────┼──────┼────────────┼─────────┼────────────────────────┼──────────────\n 0 │ single │ │ │ root │ │ │ current │\n 97 │ single │ │ 2025-06-03 00:33:31 │ root │ 274.03 MiB │ number │ Snapshot Update of #96 │ important=yes\n 98 │ single │ │ 2025-06-05 00:59:06 │ root │ 59.11 MiB │ number │ Snapshot Update of #97 │ important=yes\n 99 │ single │ │ 2025-06-06 01:17:29 │ root │ 22.16 MiB │ number │ Snapshot Update of #98 │ important=yes\n100* │ single │ │ 2025-06-08 01:49:48 │ root │ 38.02 MiB │ number │ Snapshot Update of #99 │\n```\n\n- `*`: Marks the current/running snapshot.\n- `+`: Marks the snapshot to be used on the next boot.\n\n### Alternative: Select Snapshot by Date\n\nTo rollback to a snapshot from a specific date (e.g., June 6, 2025):\n\n```bash\nsnapper --iso list | grep 06-06 | awk '{print $1}' | xargs -I{} snapper rollback {} && reboot\n```\n" }, { "path": "examples/tls/ingress.yaml", "content": "apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n name: nginx-ingress\n annotations:\n traefik.ingress.kubernetes.io/router.tls: \"true\"\n traefik.ingress.kubernetes.io/router.tls.certresolver: le\nspec:\n tls:\n - hosts:\n - example.com\n rules:\n - host: example.com\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: nginx-service\n port:\n number: 80\n" }, { "path": "examples/tls/pod.yaml", "content": "apiVersion: v1\nkind: Pod\nmetadata:\n labels:\n run: nginx\n name: nginx\nspec:\n containers:\n - image: nginx\n name: nginx\n ports:\n - containerPort: 80\n" }, { "path": "examples/tls/service.yaml", "content": "apiVersion: v1\nkind: Service\nmetadata:\n name: nginx-service\nspec:\n ports:\n - port: 80\n protocol: TCP\n targetPort: 80\n selector:\n run: nginx\n" }, { "path": "init.tf", "content": "resource \"hcloud_load_balancer\" \"cluster\" {\n count = local.has_external_load_balancer ? 0 : 1\n name = local.load_balancer_name\n\n load_balancer_type = var.load_balancer_type\n location = var.load_balancer_location\n labels = local.labels\n delete_protection = var.enable_delete_protection.load_balancer\n\n algorithm {\n type = var.load_balancer_algorithm_type\n }\n\n lifecycle {\n ignore_changes = [\n location,\n # Ignore changes to hcloud-ccm/service-uid label that is managed by the CCM.\n labels[\"hcloud-ccm/service-uid\"],\n ]\n }\n}\n\nresource \"hcloud_load_balancer_network\" \"cluster\" {\n count = local.has_external_load_balancer ? 0 : 1\n\n load_balancer_id = hcloud_load_balancer.cluster.*.id[0]\n # Use -2 to get the last usable IP in the subnet\n ip = cidrhost(\n (\n length(hcloud_network_subnet.agent) > 0\n ? hcloud_network_subnet.agent.*.ip_range[0]\n : hcloud_network_subnet.control_plane.*.ip_range[0]\n )\n , -2)\n subnet_id = (\n length(hcloud_network_subnet.agent) > 0\n ? hcloud_network_subnet.agent.*.id[0]\n : hcloud_network_subnet.control_plane.*.id[0]\n )\n enable_public_interface = true\n\n lifecycle {\n create_before_destroy = false\n ignore_changes = [\n ip,\n enable_public_interface\n ]\n }\n}\n\nresource \"hcloud_load_balancer_target\" \"cluster\" {\n count = local.has_external_load_balancer ? 0 : 1\n\n depends_on = [hcloud_load_balancer_network.cluster]\n type = \"label_selector\"\n load_balancer_id = hcloud_load_balancer.cluster.*.id[0]\n label_selector = join(\",\", concat(\n [for k, v in local.labels : \"${k}=${v}\"],\n [\n # Build label selector from lb_target_groups (respects allow_loadbalancer_target_on_control_plane)\n # Results in either: role in (control_plane_node,agent_node) or role in (agent_node)\n for key in keys(merge(local.lb_target_groups...)) :\n \"${key} in (${\n join(\",\", compact([\n for labels in local.lb_target_groups :\n try(labels[key], \"\")\n ]))\n })\"\n ]\n ))\n use_private_ip = true\n}\n\nlocals {\n first_control_plane_ip = coalesce(\n module.control_planes[keys(module.control_planes)[0]].ipv4_address,\n module.control_planes[keys(module.control_planes)[0]].ipv6_address,\n module.control_planes[keys(module.control_planes)[0]].private_ipv4_address\n )\n}\n\nresource \"terraform_data\" \"first_control_plane\" {\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.first_control_plane_ip\n port = var.ssh_port\n timeout = \"10m\" # Extended timeout to handle network migrations during upgrades\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Generating k3s master config file\n provisioner \"file\" {\n content = yamlencode(\n merge(\n {\n node-name = module.control_planes[keys(module.control_planes)[0]].name\n token = local.k3s_token\n cluster-init = true\n disable-cloud-controller = true\n disable-kube-proxy = var.disable_kube_proxy\n disable = local.disable_extras\n kubelet-arg = local.kubelet_arg\n kube-controller-manager-arg = local.kube_controller_manager_arg\n flannel-iface = local.flannel_iface\n node-ip = module.control_planes[keys(module.control_planes)[0]].private_ipv4_address\n advertise-address = module.control_planes[keys(module.control_planes)[0]].private_ipv4_address\n node-taint = local.control_plane_nodes[keys(module.control_planes)[0]].taints\n node-label = local.control_plane_nodes[keys(module.control_planes)[0]].labels\n cluster-cidr = var.cluster_ipv4_cidr\n service-cidr = var.service_ipv4_cidr\n cluster-dns = local.cluster_dns_ipv4\n },\n lookup(local.cni_k3s_settings, var.cni_plugin, {}),\n var.use_control_plane_lb ? {\n tls-san = concat(\n compact([\n hcloud_load_balancer.control_plane.*.ipv4[0],\n hcloud_load_balancer_network.control_plane.*.ip[0],\n var.kubeconfig_server_address != \"\" ? var.kubeconfig_server_address : null,\n !var.control_plane_lb_enable_public_interface && var.nat_router != null ? hcloud_server.nat_router[0].ipv4_address : null\n ]),\n var.additional_tls_sans\n )\n } : {\n tls-san = concat([local.first_control_plane_ip], var.additional_tls_sans)\n },\n local.etcd_s3_snapshots,\n var.control_planes_custom_config,\n (local.control_plane_nodes[keys(module.control_planes)[0]].selinux == true ? { selinux = true } : {}),\n local.prefer_bundled_bin_config\n )\n )\n\n destination = \"/tmp/config.yaml\"\n }\n\n # Install k3s server\n provisioner \"remote-exec\" {\n inline = local.install_k3s_server\n }\n\n # Upon reboot start k3s and wait for it to be ready to receive commands\n provisioner \"remote-exec\" {\n inline = [\n \"systemctl start k3s\",\n # prepare the needed directories\n \"mkdir -p /var/post_install /var/user_kustomize\",\n # wait for k3s to become ready\n <<-EOT\n timeout 120 bash <<EOF\n until systemctl status k3s > /dev/null; do\n systemctl start k3s\n echo \"Waiting for the k3s server to start...\"\n sleep 2\n done\n until [ -e /etc/rancher/k3s/k3s.yaml ]; do\n echo \"Waiting for kubectl config...\"\n sleep 2\n done\n until [[ \"\\$(kubectl get --raw='/readyz' 2> /dev/null)\" == \"ok\" ]]; do\n echo \"Waiting for the cluster to become ready...\"\n sleep 2\n done\n EOF\n EOT\n ]\n }\n\n depends_on = [\n hcloud_network_subnet.control_plane\n ]\n}\nmoved {\n from = null_resource.first_control_plane\n to = terraform_data.first_control_plane\n}\n\n# Needed for rancher setup\nresource \"random_password\" \"rancher_bootstrap\" {\n count = length(var.rancher_bootstrap_password) == 0 ? 1 : 0\n length = 48\n special = false\n}\n\nresource \"terraform_data\" \"kube_system_secrets\" {\n triggers_replace = {\n secrets_sha = sha256(yamlencode(local.kube_system_secrets))\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.first_control_plane_ip\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n }\n\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/kube_system_secrets.yaml.tpl\",\n {\n kube_system_secrets = local.kube_system_secrets,\n })\n destination = \"/var/post_install/kube_system_secrets.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [\n <<-EOT\n set -ex\n # Retry logic to handle temporary network connectivity issues during upgrades\n MAX_ATTEMPTS=30\n RETRY_INTERVAL=10\n for attempt in $(seq 1 $MAX_ATTEMPTS); do\n echo \"Attempt $attempt: Checking kubectl connectivity...\"\n if [ \"$(kubectl get --raw='/readyz' 2>/dev/null)\" = \"ok\" ]; then\n echo \"kubectl connectivity established, deploying secrets...\"\n\n kubectl apply -f /var/post_install/kube_system_secrets.yaml\n\n echo \"Secrets deployed successfully\"\n break\n else\n echo \"kubectl not ready yet, waiting $RETRY_INTERVAL seconds...\"\n sleep $RETRY_INTERVAL\n fi\n \n if [ $attempt -eq $MAX_ATTEMPTS ]; then\n echo \"Failed to establish kubectl connectivity after $MAX_ATTEMPTS attempts\"\n exit 1\n fi\n done\n\n rm /var/post_install/kube_system_secrets.yaml\n\n EOT\n ]\n }\n\n depends_on = [\n hcloud_load_balancer.cluster,\n terraform_data.control_planes,\n ]\n}\nmoved {\n from = null_resource.kube_system_secrets\n to = terraform_data.kube_system_secrets\n}\n\n# This is where all the setup of Kubernetes components happen\nresource \"terraform_data\" \"kustomization\" {\n triggers_replace = {\n # Redeploy helm charts when the underlying values change\n helm_values_yaml = join(\"---\\n\", [\n local.traefik_values,\n local.nginx_values,\n local.haproxy_values,\n local.calico_values,\n local.cilium_values,\n local.longhorn_values,\n local.csi_driver_smb_values,\n local.cert_manager_values,\n local.rancher_values,\n local.hetzner_csi_values,\n local.hetzner_ccm_values,\n\n ])\n # Redeploy when versions of addons need to be updated\n versions = join(\"\\n\", [\n coalesce(var.initial_k3s_channel, \"N/A\"),\n coalesce(var.install_k3s_version, \"N/A\"),\n coalesce(var.cluster_autoscaler_version, \"N/A\"),\n coalesce(var.hetzner_ccm_version, \"N/A\"),\n coalesce(var.hetzner_csi_version, \"N/A\"),\n coalesce(var.kured_version, \"N/A\"),\n coalesce(var.calico_version, \"N/A\"),\n coalesce(var.cilium_version, \"N/A\"),\n coalesce(var.traefik_version, \"N/A\"),\n coalesce(var.nginx_version, \"N/A\"),\n coalesce(var.haproxy_version, \"N/A\"),\n coalesce(var.cert_manager_version, \"N/A\"),\n coalesce(var.csi_driver_smb_version, \"N/A\"),\n coalesce(var.longhorn_version, \"N/A\"),\n coalesce(var.rancher_version, \"N/A\"),\n coalesce(var.sys_upgrade_controller_version, \"N/A\"),\n ])\n options = join(\"\\n\", [\n for option, value in local.kured_options : \"${option}=${value}\"\n ])\n ccm_use_helm = var.hetzner_ccm_use_helm\n system_upgrade_schedule_window = jsonencode(var.system_upgrade_schedule_window)\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.first_control_plane_ip\n port = var.ssh_port\n timeout = \"10m\" # Extended timeout to handle network migrations during upgrades\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Upload kustomization.yaml, containing Hetzner CSI & CSM, as well as kured.\n provisioner \"file\" {\n content = local.kustomization_backup_yaml\n destination = \"/var/post_install/kustomization.yaml\"\n }\n\n # Upload the flannel RBAC fix\n provisioner \"file\" {\n content = file(\"${path.module}/kustomize/flannel-rbac.yaml\")\n destination = \"/var/post_install/flannel-rbac.yaml\"\n }\n\n # Upload traefik ingress controller config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/traefik_ingress.yaml.tpl\",\n {\n version = var.traefik_version\n values = indent(4, local.traefik_values)\n target_namespace = local.ingress_controller_namespace\n })\n destination = \"/var/post_install/traefik_ingress.yaml\"\n }\n\n # Upload nginx ingress controller config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/nginx_ingress.yaml.tpl\",\n {\n version = var.nginx_version\n values = indent(4, local.nginx_values)\n target_namespace = local.ingress_controller_namespace\n })\n destination = \"/var/post_install/nginx_ingress.yaml\"\n }\n\n # Upload haproxy ingress controller config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/haproxy_ingress.yaml.tpl\",\n {\n version = var.haproxy_version\n values = indent(4, local.haproxy_values)\n target_namespace = local.ingress_controller_namespace\n })\n destination = \"/var/post_install/haproxy_ingress.yaml\"\n }\n\n # Upload the CCM patch config using the legacy deployment\n provisioner \"file\" {\n content = var.hetzner_ccm_use_helm ? \"\" : templatefile(\n \"${path.module}/templates/ccm.yaml.tpl\",\n {\n cluster_cidr_ipv4 = var.cluster_ipv4_cidr\n default_lb_location = var.load_balancer_location\n using_klipper_lb = local.using_klipper_lb\n })\n destination = \"/var/post_install/ccm.yaml\"\n }\n\n # Upload the CCM patch config using helm\n provisioner \"file\" {\n content = var.hetzner_ccm_use_helm ? templatefile(\n \"${path.module}/templates/hcloud-ccm-helm.yaml.tpl\",\n {\n values = indent(4, local.hetzner_ccm_values)\n version = coalesce(local.ccm_version, \"*\")\n using_klipper_lb = local.using_klipper_lb\n default_lb_location = var.load_balancer_location\n }\n ) : \"\"\n destination = \"/var/post_install/hcloud-ccm-helm.yaml\"\n }\n\n # Upload the calico patch config, for the kustomization of the calico manifest\n # This method is a stub which could be replaced by a more practical helm implementation\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/calico.yaml.tpl\",\n {\n values = trimspace(local.calico_values)\n })\n destination = \"/var/post_install/calico.yaml\"\n }\n\n # Upload the cilium install file\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/cilium.yaml.tpl\",\n {\n values = indent(4, local.cilium_values)\n version = var.cilium_version\n })\n destination = \"/var/post_install/cilium.yaml\"\n }\n\n # Upload the system upgrade controller plans config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/plans.yaml.tpl\",\n {\n channel = var.initial_k3s_channel\n version = var.install_k3s_version\n disable_eviction = !var.system_upgrade_enable_eviction\n drain = var.system_upgrade_use_drain\n upgrade_window = var.system_upgrade_schedule_window\n })\n destination = \"/var/post_install/plans.yaml\"\n }\n\n # Upload the Longhorn config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/longhorn.yaml.tpl\",\n {\n longhorn_namespace = var.longhorn_namespace\n longhorn_repository = var.longhorn_repository\n version = var.longhorn_version\n bootstrap = var.longhorn_helmchart_bootstrap\n values = indent(4, local.longhorn_values)\n })\n destination = \"/var/post_install/longhorn.yaml\"\n }\n\n # Upload the csi-driver config (ignored if csi is disabled)\n provisioner \"file\" {\n content = var.disable_hetzner_csi ? \"\" : templatefile(\n \"${path.module}/templates/hcloud-csi.yaml.tpl\",\n {\n version = coalesce(local.csi_version, \"*\")\n values = indent(4, local.hetzner_csi_values)\n }\n )\n destination = \"/var/post_install/hcloud-csi.yaml\"\n }\n\n # Upload the csi-driver-smb config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/csi-driver-smb.yaml.tpl\",\n {\n version = var.csi_driver_smb_version\n bootstrap = var.csi_driver_smb_helmchart_bootstrap\n values = indent(4, local.csi_driver_smb_values)\n })\n destination = \"/var/post_install/csi-driver-smb.yaml\"\n }\n\n # Upload the cert-manager config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/cert_manager.yaml.tpl\",\n {\n version = var.cert_manager_version\n bootstrap = var.cert_manager_helmchart_bootstrap\n values = indent(4, local.cert_manager_values)\n })\n destination = \"/var/post_install/cert_manager.yaml\"\n }\n\n # Upload the Rancher config\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/rancher.yaml.tpl\",\n {\n rancher_install_channel = var.rancher_install_channel\n version = var.rancher_version\n bootstrap = var.rancher_helmchart_bootstrap\n values = indent(4, local.rancher_values)\n })\n destination = \"/var/post_install/rancher.yaml\"\n }\n\n provisioner \"file\" {\n content = templatefile(\n \"${path.module}/templates/kured.yaml.tpl\",\n {\n options = local.kured_options\n }\n )\n destination = \"/var/post_install/kured.yaml\"\n }\n\n # Deploy our post-installation kustomization\n provisioner \"remote-exec\" {\n inline = concat([\n \"set -ex\",\n\n # This ugly hack is here, because terraform serializes the\n # embedded yaml files with \"- |2\", when there is more than\n # one yamldocument in the embedded file. Kustomize does not understand\n # that syntax and tries to parse the blocks content as a file, resulting\n # in weird errors. so gnu sed with funny escaping is used to\n # replace lines like \"- |3\" by \"- |\" (yaml block syntax).\n # due to indendation this should not changes the embedded\n # manifests themselves\n \"sed -i 's/^- |[0-9]\\\\+$/- |/g' /var/post_install/kustomization.yaml\",\n\n # Wait for k3s to become ready (we check one more time) because in some edge cases,\n # the cluster had become unvailable for a few seconds, at this very instant.\n <<-EOT\n timeout 360 bash <<EOF\n until [[ \"\\$(kubectl get --raw='/readyz' 2> /dev/null)\" == \"ok\" ]]; do\n echo \"Waiting for the cluster to become ready...\"\n sleep 2\n done\n EOF\n EOT\n ]\n ,\n var.hetzner_ccm_use_helm ? [\n \"echo 'Remove legacy ccm manifests if they exist'\",\n \"kubectl delete serviceaccount,deployment -n kube-system --field-selector 'metadata.name=hcloud-cloud-controller-manager' --selector='app.kubernetes.io/managed-by!=Helm'\",\n \"kubectl delete clusterrolebinding -n kube-system --field-selector 'metadata.name=system:hcloud-cloud-controller-manager' --selector='app.kubernetes.io/managed-by!=Helm'\",\n ] : [\n \"echo 'Uninstall helm ccm manifests if they exist'\",\n \"kubectl delete --ignore-not-found -n kube-system helmchart.helm.cattle.io/hcloud-cloud-controller-manager\",\n ],\n [\n # Ready, set, go for the kustomization\n \"kubectl apply -k /var/post_install\",\n \"echo 'Waiting for the system-upgrade-controller deployment to become available...'\",\n \"kubectl -n system-upgrade wait --for=condition=available --timeout=900s deployment/system-upgrade-controller\",\n \"sleep 7\", # important as the system upgrade controller CRDs sometimes don't get ready right away, especially with Cilium.\n \"kubectl -n system-upgrade apply -f /var/post_install/plans.yaml\",\n # Wait for system namespace deployments to become available\n \"for ns in kube-system ${var.enable_cert_manager ? \"cert-manager\" : \"\"} ${var.enable_longhorn ? var.longhorn_namespace : \"\"} ${local.ingress_controller_namespace} system-upgrade; do [ -n \\\"$ns\\\" ] && kubectl get ns $ns &>/dev/null && kubectl -n $ns wait deployment --all --for=condition=Available --timeout=300s || true; done\",\n # Wait for helm install jobs to complete (only in namespaces that have jobs)\n \"for ns in kube-system ${var.enable_longhorn ? var.longhorn_namespace : \"\"}; do [ -n \\\"$ns\\\" ] && kubectl get ns $ns &>/dev/null && kubectl -n $ns get job -o name 2>/dev/null | grep -q . && kubectl -n $ns wait job --all --for=condition=Complete --timeout=300s || true; done\"\n ],\n local.has_external_load_balancer ? [] : [\n <<-EOT\n timeout 360 bash <<EOF\n until [ -n \"\\$(kubectl get -n ${local.ingress_controller_namespace} service/${lookup(local.ingress_controller_service_names, var.ingress_controller)} --output=jsonpath='{.status.loadBalancer.ingress[0].${var.lb_hostname != \"\" ? \"hostname\" : \"ip\"}}' 2> /dev/null)\" ]; do\n echo \"Waiting for load-balancer to get an IP...\"\n sleep 2\n done\n EOF\n EOT\n ])\n }\n\n depends_on = [\n hcloud_load_balancer.cluster,\n terraform_data.control_planes,\n random_password.rancher_bootstrap,\n hcloud_volume.longhorn_volume,\n terraform_data.kube_system_secrets\n ]\n}\nmoved {\n from = null_resource.kustomization\n to = terraform_data.kustomization\n}\n" }, { "path": "kube.tf.example", "content": "locals {\n # You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env\n # within your shell, such as: export TF_VAR_hcloud_token=xxxxxxxxxxx. Or you can use .tfvars-files.\n # If you choose to define it in the shell, this can be left as is.\n\n # Your Hetzner token can be found in your Project > Security > API Token (Read & Write is required).\n hcloud_token = \"xxxxxxxxxxx\"\n\n # Credentials for the Hetzner Robot webservice\n robot_user = \"\"\n robot_password = \"\"\n}\n\nmodule \"kube-hetzner\" {\n providers = {\n hcloud = hcloud\n }\n hcloud_token = var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token\n robot_user = var.robot_user != \"\" ? var.robot_user : local.robot_user\n robot_password = var.robot_password != \"\" ? var.robot_password : local.robot_password\n\n # Then fill or edit the below values. Only the first values starting with a * are obligatory; the rest can remain with their default values, or you\n # could adapt them to your needs.\n\n # * source can be specified in multiple ways:\n # 1. For normal use, (the official version published on the Terraform Registry), use\n source = \"kube-hetzner/kube-hetzner/hcloud\"\n # When using the terraform registry as source, you can optionally specify a version number.\n # See https://registry.terraform.io/modules/kube-hetzner/kube-hetzner/hcloud for the available versions\n # version = \"2.15.3\"\n # 2. For local dev, path to the git repo\n # source = \"../../kube-hetzner/\"\n # 3. If you want to use the latest master branch (see https://developer.hashicorp.com/terraform/language/modules/sources#github), use\n # source = \"github.com/kube-hetzner/terraform-hcloud-kube-hetzner\"\n\n # Note that some values, notably \"location\" and \"public_key\" have no effect after initializing the cluster.\n # This is to keep Terraform from re-provisioning all nodes at once, which would lose data. If you want to update\n # those, you should instead change the value here and manually re-provision each node. Grep for \"lifecycle\".\n\n # Customize the SSH port (by default 22)\n # ssh_port = 2222\n\n # * Your ssh public key\n ssh_public_key = file(\"~/.ssh/id_ed25519.pub\")\n # * Your private key must be \"ssh_private_key = null\" when you want to use ssh-agent for a Yubikey-like device authentication or an SSH key-pair with a passphrase.\n # For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md\n ssh_private_key = file(\"~/.ssh/id_ed25519\")\n # You can add additional SSH public Keys to grant other team members root access to your cluster nodes.\n # ssh_additional_public_keys = []\n\n # You can also add additional SSH public Keys which are saved in the hetzner cloud by a label.\n # See https://docs.hetzner.cloud/#label-selector\n # ssh_hcloud_key_label = \"role=admin\"\n\n # If you use SSH agent and have issues with SSH connecting to your nodes, you can increase the number of auth tries (default is 2)\n # ssh_max_auth_tries = 10\n\n # If you want to use an ssh key that is already registered within hetzner cloud, you can pass its id.\n # If no id is passed, a new ssh key will be registered within hetzner cloud.\n # It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` variables.\n # hcloud_ssh_key_id = \"\"\n\n # These can be customized, or left with the default values\n # * For Hetzner locations see https://docs.hetzner.com/general/others/data-centers-and-connection/\n network_region = \"eu-central\" # change to `us-east` if location is ash\n\n # If you want to create the private network before calling this module,\n # you can do so and pass its id here. For example if you want to use a proxy\n # which only listens on your private network. Advanced use case.\n #\n # NOTE1: make sure to adapt network_ipv4_cidr, cluster_ipv4_cidr, and service_ipv4_cidr accordingly.\n # If your network is created with 10.0.0.0/8, and you use subnet 10.128.0.0/9 for your\n # non-k3s business, then adapting `network_ipv4_cidr = \"10.0.0.0/9\"` should be all you need.\n #\n # NOTE2: square brackets! This must be a list of length 1.\n #\n # existing_network_id = [hcloud_network.your_network.id]\n\n # If you must change the network CIDR you can do so below, but it is highly advised against.\n # network_ipv4_cidr = \"10.0.0.0/8\"\n\n # Using the default configuration you can only create a maximum of 42 agent-nodepools.\n # This is due to the creation of a subnet for each nodepool with CIDRs being in the shape of 10.[nodepool-index].0.0/16 which collides with k3s' cluster and service IP ranges (defaults below).\n # To create additional ones (or if you want to use different ranges for other reasons), set the `subnet_ip_range` explicitly for a node pool.\n # Furthermore the maximum number of nodepools (controlplane and agent) is 50, due to a hard limit of 50 subnets per network, see https://docs.hetzner.com/cloud/networks/faq/.\n # So to be able to create a maximum of 50 nodepools in total, the values below have to be changed to something outside that range, e.g. `10.200.0.0/16` and `10.201.0.0/16` for cluster and service respectively.\n\n # If you must change the cluster CIDR you can do so below, but it is highly advised against.\n # Never change this value after you already initialized a cluster. Complete cluster redeploy needed!\n # The cluster CIDR must be a part of the network CIDR!\n # cluster_ipv4_cidr = \"10.42.0.0/16\"\n\n # If you must change the service CIDR you can do so below, but it is highly advised against.\n # Never change this value after you already initialized a cluster. Complete cluster redeploy needed!\n # The service CIDR must be a part of the network CIDR!\n # service_ipv4_cidr = \"10.43.0.0/16\"\n\n # If you must change the service IPv4 address of core-dns you can do so below, but it is highly advised against.\n # Never change this value after you already initialized a cluster. Complete cluster redeploy needed!\n # The service IPv4 address must be part of the service CIDR!\n # cluster_dns_ipv4 = \"10.43.0.10\"\n\n # For the control planes, at least three nodes are the minimum for HA. Otherwise, you need to turn off the automatic upgrades (see README).\n # **It must always be an ODD number, never even!** Search the internet for \"split-brain problem with etcd\" or see https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/\n # For instance, one is ok (non-HA), two is not ok, and three is ok (becomes HA). It does not matter if they are in the same nodepool or not! So they can be in different locations and of various types.\n\n # Of course, you can choose any number of nodepools you want, with the location you want. The only constraint on the location is that you need to stay in the same network region, Europe, or the US.\n # For the server type, the minimum instance supported is cx23. If you want to use arm64 use cax11; see https://www.hetzner.com/cloud.\n\n # IMPORTANT: Before you create your cluster, you can do anything you want with the nodepools, but you need at least one of each, control plane and agent.\n # Once the cluster is up and running, you can change nodepool count and even set it to 0 (in the case of the first control-plane nodepool, the minimum is 1).\n # You can also rename it (if the count is 0), but do not remove a nodepool from the list.\n\n # You can safely add or remove nodepools at the end of each list. That is due to how subnets and IPs get allocated (FILO).\n # The maximum number of nodepools you can create combined for both lists is 50 (see above).\n # Also, before decreasing the count of any nodepools to 0, it's essential to drain and cordon the nodes in question. Otherwise, it will leave your cluster in a bad state.\n\n # Before initializing the cluster, you can change all parameters and add or remove any nodepools. You need at least one nodepool of each kind, control plane, and agent.\n # ⚠️ The nodepool names are entirely arbitrary, but all lowercase, no special characters or underscore (dashes are allowed), and they must be unique.\n\n # If you want to have a single node cluster, have one control plane nodepools with a count of 1, and one agent nodepool with a count of 0.\n\n # Please note that changing labels and taints after the first run will have no effect. If needed, you can do that through Kubernetes directly.\n\n # Multi-architecture clusters are OK for most use cases, as container underlying images tend to be multi-architecture too.\n\n # * Example below:\n\n control_plane_nodepools = [\n {\n name = \"control-plane-nbg1\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n # swap_size = \"2G\" # remember to add the suffix, examples: 512M, 1G\n # zram_size = \"2G\" # remember to add the suffix, examples: 512M, 1G\n # kubelet_args = [\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]\n\n # Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):\n # placement_group = \"default\"\n\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n\n # To disable public ips (default: false)\n # WARNING: If both values are set to \"true\", your server will only be accessible via a private network. Make sure you have followed\n # the instructions regarding this type of setup in README.md: \"Use only private IPs in your cluster\".\n # disable_ipv4 = true\n # disable_ipv6 = true\n },\n {\n name = \"control-plane-nbg1\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n\n # Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):\n # placement_group = \"default\"\n\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n\n # To disable public ips (default: false)\n # WARNING: If both values are set to \"true\", your server will only be accessible via a private network. Make sure you have followed\n # the instructions regarding this type of setup in README.md: \"Use only private IPs in your cluster\".\n # disable_ipv4 = true\n # disable_ipv6 = true\n },\n {\n name = \"control-plane-hel1\",\n server_type = \"cx23\",\n location = \"hel1\",\n labels = [],\n taints = [],\n count = 1\n\n # Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):\n # placement_group = \"default\"\n\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n\n # To disable public ips (default: false)\n # WARNING: If both values are set to \"true\", your server will only be accessible via a private network. Make sure you have followed\n # the instructions regarding this type of setup in README.md: \"Use only private IPs in your cluster\".\n # disable_ipv4 = true\n # disable_ipv6 = true\n }\n ]\n\n agent_nodepools = [\n {\n name = \"agent-small\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n # subnet_ip_range = \"10.0.0.0/16\" # Optional: override default subnet range\n # swap_size = \"2G\" # remember to add the suffix, examples: 512M, 1G\n # zram_size = \"2G\" # remember to add the suffix, examples: 512M, 1G\n # kubelet_args = [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]\n\n # Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):\n # placement_group = \"default\"\n\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n },\n {\n name = \"agent-large\",\n server_type = \"cx33\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n subnet_ip_range = \"10.100.0.0/16\"\n\n # Fine-grained control over placement groups (nodes in the same group are spread over different physical servers, 10 nodes per placement group max):\n # placement_group = \"default\"\n\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n },\n {\n name = \"storage\",\n server_type = \"cx33\",\n location = \"nbg1\",\n # Fully optional, just a demo.\n labels = [\n \"node.kubernetes.io/server-usage=storage\"\n ],\n taints = [],\n count = 1\n\n # In the case of using Longhorn, you can use Hetzner volumes instead of using the node's own storage by specifying a value from 10 to 10240 (in GB)\n # It will create one volume per node in the nodepool, and configure Longhorn to use them.\n # Something worth noting is that Volume storage is slower than node storage, which is achieved by not mentioning longhorn_volume_size or setting it to 0.\n # So for something like DBs, you definitely want node storage, for other things like backups, volume storage is fine, and cheaper.\n # longhorn_volume_size = 20\n # Set any path inside /var/ folder to have the ability to use an additional storage class along with the default one, which by default must be set in helm values to\n # /var/longhorn\n # longhorn_mount_path = \"/var/lib/longhorn\"\n # Enable automatic backups via Hetzner (default: false)\n # backups = true\n },\n # Egress nodepool useful to route egress traffic using Hetzner Floating IPs (https://docs.hetzner.com/cloud/floating-ips)\n # used with Cilium's Egress Gateway feature https://docs.cilium.io/en/stable/gettingstarted/egress-gateway/\n # See the https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner#examples for an example use case.\n {\n name = \"egress\",\n server_type = \"cx23\",\n location = \"nbg1\",\n labels = [\n \"node.kubernetes.io/role=egress\"\n ],\n taints = [\n \"node.kubernetes.io/role=egress:NoSchedule\"\n ],\n floating_ip = true\n # Optionally associate a reverse DNS entry with the floating IP(s).\n # This is useful in combination with the Egress Gateway feature for hosting certain services in the cluster, such as email servers.\n # floating_ip_rns = \"my.domain.com\"\n count = 1\n },\n # Arm based nodes\n {\n name = \"agent-arm-small\",\n server_type = \"cax11\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n count = 1\n },\n # For fine-grained control over the nodes in a node pool, replace the count variable with a nodes map.\n # In this case, the node-pool variables are defaults which can be overridden on a per-node basis.\n # Each key in the nodes map refers to a single node and must be an integer string (\"1\", \"123\", ...).\n {\n name = \"agent-arm-medium\",\n server_type = \"cax21\",\n location = \"nbg1\",\n labels = [],\n taints = [],\n nodes = {\n \"1\" : {\n location = \"fsn1\"\n labels = [\n \"testing-labels=a1\",\n ]\n },\n \"20\" : {\n labels = [\n \"testing-labels=b1\",\n ]\n }\n }\n },\n ]\n # Add additional configuration options for control planes here.\n # E.g to enable monitoring for etcd, proxy etc:\n # control_planes_custom_config = {\n # etcd-expose-metrics = true,\n # kube-controller-manager-arg = \"bind-address=0.0.0.0\",\n # kube-proxy-arg =\"metrics-bind-address=0.0.0.0\",\n # kube-scheduler-arg = \"bind-address=0.0.0.0\",\n # }\n\n # Add additional configuration options for agent nodes and autoscaler nodes here.\n # E.g to enable monitoring for proxy:\n # agent_nodes_custom_config = {\n # kube-proxy-arg =\"metrics-bind-address=0.0.0.0\",\n # }\n\n # You can enable encrypted wireguard for the CNI by setting this to \"true\". Default is \"false\".\n # FYI, Hetzner says \"Traffic between cloud servers inside a Network is private and isolated, but not automatically encrypted.\"\n # Source: https://docs.hetzner.com/cloud/networks/faq/#is-traffic-inside-hetzner-cloud-networks-encrypted\n # It works with all CNIs that we support.\n # Just note, that if Cilium with cilium_values, the responsibility of enabling of disabling Wireguard falls on you.\n # enable_wireguard = true\n\n # Override the flannel backend directly (takes precedence over enable_wireguard).\n # Valid values: vxlan (default), host-gw, wireguard-native\n # Use wireguard-native for Robot nodes with vSwitch to avoid MTU auto-calculation issues.\n # See https://docs.k3s.io/networking/basic-network-options for details.\n # flannel_backend = \"wireguard-native\"\n\n # * LB location and type, the latter will depend on how much load you want it to handle, see https://www.hetzner.com/cloud/load-balancer\n load_balancer_type = \"lb11\"\n load_balancer_location = \"nbg1\"\n\n # Disable IPv6 for the load balancer, the default is false.\n # load_balancer_disable_ipv6 = true\n\n # Disables the public network of the load balancer. (default: false).\n # load_balancer_disable_public_network = true\n\n # Specifies the algorithm type of the load balancer. (default: round_robin).\n # load_balancer_algorithm_type = \"least_connections\"\n\n # Specifies the interval at which a health check is performed. Minimum is 3s (default: 15s).\n # load_balancer_health_check_interval = \"5s\"\n\n # Specifies the timeout of a single health check. Must not be greater than the health check interval. Minimum is 1s (default: 10s).\n # load_balancer_health_check_timeout = \"3s\"\n\n # Specifies the number of times a health check is retried before a target is marked as unhealthy. (default: 3)\n # load_balancer_health_check_retries = 3\n\n\n # Setup a NAT router, and automatically disable public ips on all control plane and agent nodes.\n # To use this, you must also set use_control_plane_lb = true, otherwise kubectl can never\n # reach the cluster. The NAT router will also function as bastion. This makes securing the cluster\n # easier, as all public traffic passes through a single strongly secured node. It does\n # however also introduce a single point of failure, so if you need high-availability on your\n # egress, you should consider other configurations.\n # If a simple failover mechanism is acceptable, you can set enable_redundancy = true to deploy two NAT routers running keepalived to provide a failover mechanism.\n #\n # Hetzner removed the DHCP Router option from private networks on 2025-08-11, so the module\n # now ensures each node attached to the private network persists a default route via the\n # virtual gateway. No manual `ip route add` is required after reboots or DHCP renewals.\n #\n #\n # nat_router = {\n # server_type = \"cax21\"\n # location = \"nbg1\"\n # enable_sudo = false # optional, default to false. Set to true to add nat-router user to the sudo'ers. Note that ssh as root is disabled.\n # enable_redundancy = true # optional, default to false. Enable failover for the NAT router.\n # standby_location = \"fsn1\" # optional, default to \"\". Must be set if enable_redundancy is true.\n # labels = {} # optionally add labels.\n # }\n # nat_router_hcloud_token = \"\" # optional, default to \"\". Must be set if enable_redundancy is true. This token needs read/write to change the private alias ip in the nat_router subnetwork for failover\n\n\n ### The following values are entirely optional (and can be removed from this if unused)\n\n # You can refine a base domain name to be use in this form of nodename.base_domain for setting the reverse dns inside Hetzner\n # base_domain = \"mycluster.example.com\"\n\n # Cluster Autoscaler\n # Providing at least one map for the array enables the cluster autoscaler feature, default is disabled.\n # ⚠️ Based on how the autoscaler works with this project, you can only choose either x86 instances or ARM server types for ALL autoscaler nodepools.\n # If you are curious, it's ok to have a multi-architecture cluster, as most underlying container images are multi-architecture too.\n #\n # ⚠️ Setting labels and taints will only work on cluster-autoscaler images versions released after > 20 October 2023. Or images built from master after that date.\n #\n # * Example below:\n # autoscaler_nodepools = [\n # {\n # name = \"autoscaled-small\"\n # server_type = \"cx33\"\n # location = \"nbg1\"\n # # Add the arg --enforce-node-group-min-size=true in the cluster_autoscaler_extra_args option below if you want min_nodes to be effective\n # min_nodes = 0\n # max_nodes = 5\n # labels = {\n # \"node.kubernetes.io/role\": \"peak-workloads\"\n # }\n # taints = [\n # {\n # key= \"node.kubernetes.io/role\"\n # value= \"peak-workloads\"\n # effect= \"NoExecute\"\n # }\n # ]\n # # kubelet_args = [\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"]\n # }\n # ]\n #\n # To disable public ips on your autoscaled nodes, uncomment the following lines:\n # autoscaler_disable_ipv4 = true\n # autoscaler_disable_ipv6 = true\n\n # ⚠️ Deprecated, will be removed after a new Cluster Autoscaler version has been released which support the new way of setting labels and taints. See above.\n # Add extra labels on nodes started by the Cluster Autoscaler\n # This argument is not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set\n # autoscaler_labels = [\n # \"node.kubernetes.io/role=peak-workloads\"\n # ]\n\n # Add extra taints on nodes started by the Cluster Autoscaler\n # This argument is not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set\n # autoscaler_taints = [\n # \"node.kubernetes.io/role=specific-workloads:NoExecute\"\n # ]\n\n # Configuration of the Cluster Autoscaler binary\n #\n # These arguments and variables are not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set.\n #\n # Image and version of Kubernetes Cluster Autoscaler for Hetzner Cloud:\n # - cluster_autoscaler_image: Image of Kubernetes Cluster Autoscaler for Hetzner Cloud to be used.\n # The default is the official image from the Kubernetes project: registry.k8s.io/autoscaling/cluster-autoscaler\n # - cluster_autoscaler_version: Version of Kubernetes Cluster Autoscaler for Hetzner Cloud. Should be aligned with Kubernetes version.\n # Available versions for the official image can be found at https://explore.ggcr.dev/?repo=registry.k8s.io%2Fautoscaling%2Fcluster-autoscaler\n #\n # Logging related arguments are managed using separate variables:\n # - cluster_autoscaler_log_level: Controls the verbosity of logs (--v), the value is from 0 to 5, default is 4, for max debug info set it to 5.\n # - cluster_autoscaler_log_to_stderr: Determines whether to log to stderr (--logtostderr).\n # - cluster_autoscaler_stderr_threshold: Sets the threshold for logs that go to stderr (--stderrthreshold).\n #\n # Server/node creation timeout variable:\n # - cluster_autoscaler_server_creation_timeout: Sets the timeout (in minutes) until which a newly created server/node has to become available before giving up and destroying it (defaults to 15, unit is minutes)\n #\n # Example:\n #\n # cluster_autoscaler_image = \"registry.k8s.io/autoscaling/cluster-autoscaler\"\n # cluster_autoscaler_version = \"v1.33.3\"\n # cluster_autoscaler_log_level = 4\n # cluster_autoscaler_log_to_stderr = true\n # cluster_autoscaler_stderr_threshold = \"INFO\"\n # cluster_autoscaler_server_creation_timeout = 15\n\n # Additional Cluster Autoscaler binary configuration\n #\n # cluster_autoscaler_extra_args can be used for additional arguments. The default is an empty array.\n #\n # Please note that following arguments are managed by terraform-hcloud-kube-hetzner or the variables above and should not be set manually:\n # - --v=${var.cluster_autoscaler_log_level}\n # - --logtostderr=${var.cluster_autoscaler_log_to_stderr}\n # - --stderrthreshold=${var.cluster_autoscaler_stderr_threshold}\n # - --cloud-provider=hetzner\n # - --nodes ...\n #\n # See the Cluster Autoscaler FAQ for the full list of arguments: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca\n #\n # Example:\n #\n # cluster_autoscaler_extra_args = [\n # \"--ignore-daemonsets-utilization=true\",\n # \"--enforce-node-group-min-size=true\",\n # ]\n\n # Cluster Autoscaler Deployment Configuration\n #\n # Configure the number of replicas and resource limits/requests for the cluster autoscaler deployment.\n #\n # cluster_autoscaler_replicas: Number of replicas for the cluster autoscaler deployment (default: 1)\n # - Setting this to > 1 enables HA for the autoscaler itself (uses leader election)\n #\n # cluster_autoscaler_resource_limits: Enable or disable resource limits/requests (default: true)\n # - Set to false to remove resource constraints entirely\n #\n # cluster_autoscaler_resource_values: Customize CPU and memory resources (defaults: 100m CPU, 300Mi memory for both requests and limits)\n #\n # Example:\n #\n # cluster_autoscaler_replicas = 2\n # cluster_autoscaler_resource_limits = true\n # cluster_autoscaler_resource_values = {\n # requests = {\n # cpu = \"100m\"\n # memory = \"300Mi\"\n # }\n # limits = {\n # cpu = \"200m\"\n # memory = \"500Mi\"\n # }\n # }\n\n # Enable delete protection on compatible resources to prevent accidental deletion from the Hetzner Cloud Console.\n # This does not protect deletion from Terraform itself.\n # enable_delete_protection = {\n # floating_ip = true\n # load_balancer = true\n # volume = true\n # }\n\n # Enable etcd snapshot backups to S3 storage.\n # Just provide a map with the needed settings (according to your S3 storage provider) and backups to S3 will\n # be enabled (with the default settings for etcd snapshots).\n # Cloudflare's R2 offers 10GB, 10 million reads and 1 million writes per month for free.\n # For proper context, have a look at https://docs.k3s.io/datastore/backup-restore.\n # You also can use additional parameters from https://docs.k3s.io/cli/etcd-snapshot, such as `etc-s3-folder`\n # etcd_s3_backup = {\n # etcd-s3-endpoint = \"xxxx.r2.cloudflarestorage.com\"\n # etcd-s3-access-key = \"<access-key>\"\n # etcd-s3-secret-key = \"<secret-key>\"\n # etcd-s3-bucket = \"k3s-etcd-snapshots\"\n # etcd-s3-region = \"<your-s3-bucket-region|usually required for aws>\"\n # }\n\n # To enable Hetzner Storage Box support, you can enable csi-driver-smb, default is \"false\".\n # enable_csi_driver_smb = true\n # If you want to specify the version for csi-driver-smb, set it below - otherwise it'll use the latest version available.\n # See https://github.com/kubernetes-csi/csi-driver-smb/releases for the available versions.\n # csi_driver_smb_version = \"v1.16.0\"\n\n # To enable iscid without setting enable_longhorn = true, set enable_iscsid = true. You will need this if\n # you install your own version of longhorn outside of this module.\n # Default is false. If enable_longhorn=true, this variable is ignored and iscsid is enabled anyway.\n # enable_iscsid = true\n\n # To use local storage on the nodes, you can enable Longhorn, default is \"false\".\n # See a full recap on how to configure agent nodepools for longhorn here https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/discussions/373#discussioncomment-3983159\n # Also see Longhorn best practices here https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b\n # enable_longhorn = true\n\n # By default, longhorn is pulled from https://charts.longhorn.io.\n # If you need a version of longhorn which assures compatibility with rancher you can set this variable to https://charts.rancher.io.\n # longhorn_repository = \"https://charts.rancher.io\"\n\n # The namespace for longhorn deployment, default is \"longhorn-system\".\n # longhorn_namespace = \"longhorn-system\"\n\n # The file system type for Longhorn, if enabled (ext4 is the default, otherwise you can choose xfs).\n # longhorn_fstype = \"xfs\"\n\n # how many replica volumes should longhorn create (default is 3).\n # longhorn_replica_count = 1\n\n # When you enable Longhorn, you can go with the default settings and just modify the above two variables OR you can add a longhorn_values variable\n # with all needed helm values, see towards the end of the file in the advanced section. You can also use longhorn_merge_values.\n # If that file is present, the system will use it during the deploy, if not it will use the default values with the two variable above that can be customized.\n # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.\n\n # Also, you can choose to use a Hetzner volume with Longhorn. By default, it will use the nodes own storage space, but if you add an attribute of\n # longhorn_volume_size (⚠️ not a variable, just a possible agent nodepool attribute) with a value between 10 and 10240 GB to your agent nodepool definition, it will create and use the volume in question.\n # See the agent nodepool section for an example of how to do that.\n\n # To disable Hetzner CSI storage, you can set the following to \"true\", default is \"false\".\n # disable_hetzner_csi = true\n\n # If you want to use a specific Hetzner CCM and CSI version, set them below; otherwise, leave them as-is for the latest versions.\n # See https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases for the available versions.\n # hetzner_ccm_version = \"\"\n\n # By default, new installations use Helm to install Hetzner CCM. You can use the legacy deployment method (using `kubectl apply`) by setting `hetzner_ccm_use_helm = false`.\n hetzner_ccm_use_helm = true\n\n # To enable Hetzner CCM compatibility with dedicated Robot servers, set the `robot_ccm_enabled` to \"true\", default is \"false\".\n # Requirements for the CCM and Kubernetes Cluster to work with dedicated Robot servers:\n # - Create Robot Webservice credentials and set them to the `robot_user` and `robot_password` TF-variables. They are passed on via secrets to HCCM env.\n # - `hetzner_ccm_use_helm = true`\n # - `robot_ccm_enabled = true`\n # See more from https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md\n # robot_ccm_enabled = false\n\n # To connect the Hetzner Cloud network to Robot servers via vSwitch subnet, create the vSwitch and set its ID to the `vswitch_id` (number).\n # Note that the VLAN ID is not the same as vSwitch ID. The vSwitch-subnet is assigned to 10.201.0.0/16 by default, can be changed via var.vswitch_subnet_index.\n # The vSwitch subnet is not created when the value is null. Default: null\n # vswitch_id = null\n\n # See https://github.com/hetznercloud/csi-driver/releases for the available versions.\n # hetzner_csi_version = \"\"\n\n # If you want to specify the Kured version, set it below - otherwise it'll use the latest version available.\n # See https://github.com/kubereboot/kured/releases for the available versions.\n # kured_version = \"\"\n\n # Default is \"traefik\".\n # If you want to enable the Nginx (https://kubernetes.github.io/ingress-nginx/) or HAProxy ingress controller instead of Traefik, you can set this to \"nginx\" or \"haproxy\".\n # By the default we load optimal Traefik, Nginx or HAProxy ingress controller config for Hetzner, however you may need to tweak it to your needs, so to do,\n # we allow you to add a traefik_values, nginx_values or haproxy_values, see towards the end of this file in the advanced section.\n # You can also use *_merge_values to overlay defaults (or the *_values file if set).\n # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.\n # If you want to disable both controllers set this to \"none\"\n # ingress_controller = \"nginx\"\n # Namespace in which to deploy the ingress controllers. Defaults to the ingress_controller variable, eg (haproxy, nginx, traefik)\n # ingress_target_namespace = \"\"\n\n # You can change the number of replicas for selected ingress controller here. The default 0 means autoselecting based on number of agent nodes (1 node = 1 replica, 2 nodes = 2 replicas, 3+ nodes = 3 replicas)\n # ingress_replica_count = 1\n\n # Use the klipperLB (similar to metalLB), instead of the default Hetzner one, that has an advantage of dropping the cost of the setup.\n # Automatically \"true\" in the case of single node cluster (as it does not make sense to use the Hetzner LB in that situation).\n # It can work with any ingress controller that you choose to deploy.\n # Please note that because the klipperLB points to all nodes, we automatically allow scheduling on the control plane when it is active.\n # enable_klipper_metal_lb = true\n\n # When using an external load balancer, you can specify a stable control plane endpoint URL.\n # control_plane_endpoint = \"https://my-external-lb:6443\"\n\n # If you want to configure additional arguments for traefik, enter them here as a list and in the form of traefik CLI arguments; see https://doc.traefik.io/traefik/reference/static-configuration/cli/\n # They are the options that go into the additionalArguments section of the Traefik helm values file.\n # We already add \"providers.kubernetesingress.ingressendpoint.publishedservice\" by default so that Traefik works automatically with services such as External-DNS and ArgoCD.\n # Example:\n # traefik_additional_options = [\"--log.level=DEBUG\", \"--tracing=true\"]\n\n # By default traefik image tag is an empty string which uses latest image tag.\n # The default is \"\".\n # traefik_image_tag = \"v3.0.0-beta5\"\n\n # By default traefik is configured to redirect http traffic to https, you can set this to \"false\" to disable the redirection.\n # The default is true.\n # traefik_redirect_to_https = false\n\n # Enable or disable Horizontal Pod Autoscaler for traefik.\n # The default is true.\n # traefik_autoscaling = false\n\n # Enable or disable pod disruption budget for traefik. Values are maxUnavailable: 33% and minAvailable: 1.\n # The default is true.\n # traefik_pod_disruption_budget = false\n\n # Enable kubernetes gateway api (https://doc.traefik.io/traefik/providers/kubernetes-gateway/) provider support.\n # The default is false.\n # traefik_provider_kubernetes_gateway_enabled = true\n\n # Enable or disable default resource requests and limits for traefik. Values requested are 100m & 50Mi and limits 300m & 150Mi.\n # The default is true.\n # traefik_resource_limits = false\n\n # If you want to configure additional ports for traefik, enter them here as a list of objects with name, port, and exposedPort properties.\n # Example:\n # traefik_additional_ports = [{name = \"example\", port = 1234, exposedPort = 1234}]\n\n # If you want to configure additional trusted IPs for traefik, enter them here as a list of IPs (strings).\n # Example for Cloudflare:\n # traefik_additional_trusted_ips = [\n # \"173.245.48.0/20\",\n # \"103.21.244.0/22\",\n # \"103.22.200.0/22\",\n # \"103.31.4.0/22\",\n # \"141.101.64.0/18\",\n # \"108.162.192.0/18\",\n # \"190.93.240.0/20\",\n # \"188.114.96.0/20\",\n # \"197.234.240.0/22\",\n # \"198.41.128.0/17\",\n # \"162.158.0.0/15\",\n # \"104.16.0.0/13\",\n # \"104.24.0.0/14\",\n # \"172.64.0.0/13\",\n # \"131.0.72.0/22\",\n # \"2400:cb00::/32\",\n # \"2606:4700::/32\",\n # \"2803:f800::/32\",\n # \"2405:b500::/32\",\n # \"2405:8100::/32\",\n # \"2a06:98c0::/29\",\n # \"2c0f:f248::/32\"\n # ]\n\n # If you want to disable the metric server set this to \"false\". Default is \"true\".\n # enable_metrics_server = false\n\n # If you want to enable the k3s built-in local-storage controller set this to \"true\". Default is \"false\".\n # Warning: When enabled together with the Hetzner CSI, there will be two default storage classes: \"local-path\" and \"hcloud-volumes\"!\n # Even if patched to remove the \"default\" label, the local-path storage class will be reset as default on each reboot of\n # the node where the controller runs.\n # This is not a problem if you explicitly define which storageclass to use in your PVCs.\n # Workaround if you don't want two default storage classes: leave this to false and add the local-path-provisioner helm chart\n # as an extra (https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner#adding-extras).\n # enable_local_storage = false\n\n # If you want to allow non-control-plane workloads to run on the control-plane nodes, set this to \"true\". The default is \"false\".\n # True by default for single node clusters, and when enable_klipper_metal_lb is true. In those cases, the value below will be ignored.\n # allow_scheduling_on_control_plane = true\n\n # If you use both the Terraform-managed ingress LB AND CCM-managed LoadBalancer services, agents get registered to both.\n # Enable this to exclude agents from CCM LBs (adds node.kubernetes.io/exclude-from-external-load-balancers=true label).\n # WARNING: If allow_scheduling_on_control_plane=false, this leaves NO eligible targets for CCM LoadBalancer services.\n # exclude_agents_from_external_load_balancers = true\n\n # If you want to disable the automatic upgrade of k3s, you can set below to \"false\".\n # Ideally, keep it on, to always have the latest Kubernetes version, but lock the initial_k3s_channel to a kube major version,\n # of your choice, like v1.25 or v1.26. That way you get the best of both worlds without the breaking changes risk.\n # For production use, always use an HA setup with at least 3 control-plane nodes and 2 agents, and keep this on for maximum security.\n\n # The default is \"true\" (in HA setup i.e. at least 3 control plane nodes & 2 agents, just keep it enabled since it works flawlessly).\n # automatically_upgrade_k3s = false\n\n # By default nodes are drained before k3s upgrade, which will delete and transfer all pods to other nodes.\n # Set this to false to cordon nodes instead, which just prevents scheduling new pods on the node during upgrade\n # and keeps all pods running. This may be useful if you have pods which are known to be slow to start e.g.\n # because they have to mount volumes with many files which require to get the right security context applied.\n system_upgrade_use_drain = true\n\n # During k3s via system-upgrade-manager pods are evicted by default.\n # On small clusters this can lead to hanging upgrades and indefinitely unschedulable nodes,\n # in that case, set this to false to immediately delete pods before upgrading.\n # NOTE: Turning this flag off might lead to downtimes of services (which may be acceptable for your use case)\n # NOTE: This flag takes effect only when system_upgrade_use_drain is set to true.\n # system_upgrade_enable_eviction = false\n\n # The default is \"true\" (in HA setup it works wonderfully well, with automatic roll-back to the previous snapshot in case of an issue).\n # IMPORTANT! For non-HA clusters i.e. when the number of control-plane nodes is < 3, you have to turn it off.\n # automatically_upgrade_os = false\n\n # If you need more control over kured and the reboot behaviour, you can pass additional options to kured.\n # For example limiting reboots to certain timeframes. For all options see: https://kured.dev/docs/configuration/\n # By default, the kured lock does not expire and is only released once a node successfully reboots. You can add the option\n # \"lock-ttl\" : \"30m\", if you have a single node which sometimes gets stuck. Note however, that in that case, kured continuous\n # draining the next node because the lock was released. You may end up with all nodes drained and your cluster completely down.\n # The default options are: `--reboot-command=/usr/bin/systemctl reboot --pre-reboot-node-labels=kured=rebooting --post-reboot-node-labels=kured=done --period=5m`\n # Defaults can be overridden by using the same key.\n # kured_options = {\n # \"reboot-days\": \"su\",\n # \"start-time\": \"3am\",\n # \"end-time\": \"8am\",\n # \"time-zone\": \"Local\",\n # \"lock-ttl\" : \"30m\",\n # }\n\n # Allows you to specify the k3s version. If defined, supersedes initial_k3s_channel.\n # See https://github.com/k3s-io/k3s/releases for the available versions.\n # install_k3s_version = \"v1.33.7+k3s1\"\n\n # Allows you to specify either stable, latest, testing or supported minor versions.\n # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels\n # ⚠️ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to one minor version below the latest stable,\n # e.g. v1.32 instead of the stable v1.33.\n # The default is \"v1.33\".\n # initial_k3s_channel = \"stable\"\n\n # Allows to specify the version of the System Upgrade Controller for automated upgrades of k3s.\n # v0.15.0+ supports the 'window' parameter for scheduling upgrade times.\n # See https://github.com/rancher/system-upgrade-controller/releases for the available versions.\n # sys_upgrade_controller_version = \"v0.18.0\"\n\n # Schedule window for k3s automated upgrades (system-upgrade-controller v0.15.0+).\n # Restricts upgrade job creation to the specified time window.\n # See https://docs.k3s.io/upgrades/automated#scheduling-upgrades\n # system_upgrade_schedule_window = {\n # days = [\"monday\", \"tuesday\", \"wednesday\", \"thursday\", \"friday\"]\n # startTime = \"19:00\"\n # endTime = \"21:00\"\n # timeZone = \"UTC\"\n # }\n\n # The cluster name, by default \"k3s\"\n # cluster_name = \"\"\n\n # Whether to use the cluster name in the node name, in the form of {cluster_name}-{nodepool_name}, the default is \"true\".\n # use_cluster_name_in_node_name = false\n\n # Extra k3s registries. This is useful if you have private registries and you want to pull images without additional secrets.\n # Or if you want to proxy registries for various reasons like rate-limiting.\n # It will create the registries.yaml file, more info here https://docs.k3s.io/installation/private-registry.\n # Note that you do not need to get this right from the first time, you can update it when you want during the life of your cluster.\n # The default is blank.\n /* k3s_registries = <<-EOT\n mirrors:\n hub.my_registry.com:\n endpoint:\n - \"hub.my_registry.com\"\n configs:\n hub.my_registry.com:\n auth:\n username: username\n password: password\n EOT */\n\n # Additional environment variables for the host OS on which k3s runs. See for example https://docs.k3s.io/advanced#configuring-an-http-proxy .\n # additional_k3s_environment = {\n # \"CONTAINERD_HTTP_PROXY\" : \"http://your.proxy:port\",\n # \"CONTAINERD_HTTPS_PROXY\" : \"http://your.proxy:port\",\n # \"NO_PROXY\" : \"127.0.0.0/8,10.0.0.0/8,\",\n # }\n\n # Additional commands to execute on the host OS before the k3s install, for example fetching and installing certs.\n # preinstall_exec = [\n # \"curl https://somewhere.over.the.rainbow/ca.crt > /root/ca.crt\",\n # \"trust anchor --store /root/ca.crt\",\n # ]\n\n # Structured authentication configuration. Multiple authentication providers support requires v1.30+ of\n # kubernetes.\n # https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration\n #\n # authentication_config = <<-EOT\n # apiVersion: apiserver.config.k8s.io/v1beta1\n # kind: AuthenticationConfiguration\n # jwt:\n # - issuer:\n # url: \"https://token.actions.githubusercontent.com\"\n # audiences:\n # - \"https://github.com/octo-org\"\n # claimMappings:\n # username:\n # claim: sub\n # prefix: \"gh:\"\n # groups:\n # claim: repository_owner\n # prefix: \"gh:\"\n # claimValidationRules:\n # - claim: repository\n # requiredValue: \"octo-org/octo-repo\"\n # - claim: \"repository_visibility\"\n # requiredValue: \"public\"\n # - claim: \"ref\"\n # requiredValue: \"refs/heads/main\"\n # - claim: \"ref_type\"\n # requiredValue: \"branch\"\n # - issuer:\n # url: \"https://your.oidc.issuer\"\n # audiences:\n # - \"oidc_client_id\"\n # claimMappings:\n # username:\n # claim: oidc_username_claim\n # prefix: \"oidc:\"\n # groups:\n # claim: oidc_groups_claim\n # prefix: \"oidc:\"\n # EOT\n\n # Set to true if util-linux breaks on the OS (temporary regression fixed in util-linux v2.41.1).\n # k3s_prefer_bundled_bin = true\n\n # Additional flags to pass to the k3s server command (the control plane).\n # k3s_exec_server_args = \"--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector\"\n\n # Additional flags to pass to the k3s agent command (every agents nodes, including autoscaler nodepools).\n # k3s_exec_agent_args = \"--kubelet-arg kube-reserved=cpu=100m,memory=200Mi,ephemeral-storage=1Gi\"\n\n # The vars below here passes it to the k3s config.yaml. This way it persist across reboots\n # Make sure you set \"feature-gates=NodeSwap=true\" if want to use swap_size\n # Note: CloudDualStackNodeIPs was removed in K8s 1.32 (always enabled now)\n # see https://github.com/k3s-io/k3s/issues/8811#issuecomment-1856974516\n # k3s_global_kubelet_args = [\"kube-reserved=cpu=100m,ephemeral-storage=1Gi\", \"system-reserved=cpu=memory=200Mi\", \"image-gc-high-threshold=50\", \"image-gc-low-threshold=40\"]\n # k3s_control_plane_kubelet_args = []\n # k3s_agent_kubelet_args = []\n # k3s_autoscaler_kubelet_args = []\n\n # https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/\n # k3s_kubelet_config = <<-EOT\n # apiVersion: kubelet.config.k8s.io/v1beta1\n # kind: KubeletConfiguration\n # imageGCLowThresholdPercent: 40\n # imageGCHighThresholdPercent: 50\n # imageMaximumGCAge: 24h\n # EOT\n\n # Enable Kubernetes audit logging on control plane nodes\n # This will create an audit policy file and configure k3s to use it\n # Audit logs will be written to /var/log/k3s-audit/audit.log by default\n #\n # k3s_audit_policy_config = <<-EOT\n # apiVersion: audit.k8s.io/v1\n # kind: Policy\n # rules:\n # # Log pod changes at RequestResponse level\n # - level: RequestResponse\n # omitStages:\n # - RequestReceived\n # resources:\n # - group: \"\"\n # resources: [\"pods\", \"services\"]\n # namespaces: [\"default\", \"kube-system\"]\n # # Log all other resources at Metadata level\n # - level: Metadata\n # omitStages:\n # - RequestReceived\n # # Don't log requests to certain non-resource URL paths\n # - level: None\n # nonResourceURLs:\n # - /api*\n # - /version\n # - /healthz\n # - /readyz\n # EOT\n #\n # # Audit log configuration\n # k3s_audit_log_path = \"/var/log/k3s-audit/audit.log\" # Path to audit log file\n # k3s_audit_log_maxage = 30 # Days to retain audit logs\n # k3s_audit_log_maxbackup = 10 # Number of audit log files to keep\n # k3s_audit_log_maxsize = 100 # Max size in MB before rotation\n\n # If you want to allow all outbound traffic you can set this to \"false\". Default is \"true\".\n # restrict_outbound_traffic = false\n\n # Allow access to the Kube API from the specified networks. The default is [\"0.0.0.0/0\", \"::/0\"].\n # Allowed values: null (disable Kube API rule entirely) or a list of allowed networks with CIDR notation.\n # For maximum security, it's best to disable it completely by setting it to null. However, in that case, to get access to the kube api,\n # you would have to connect to any control plane node via SSH, as you can run kubectl from within these.\n # Please be advised that this setting has no effect on the load balancer when the use_control_plane_lb variable is set to true. This is\n # because firewall rules cannot be applied to load balancers yet.\n # firewall_kube_api_source = null\n\n # Allow SSH access from the specified networks. Default: [\"0.0.0.0/0\", \"::/0\"]\n # Allowed values: null (disable SSH rule entirely) or a list of allowed networks with CIDR notation.\n # Ideally you would set your IP there. And if it changes after cluster deploy, you can always update this variable and apply again.\n # firewall_ssh_source = [\"1.2.3.4/32\"]\n\n # By default, SELinux is enabled in enforcing mode on all nodes. For container-specific SELinux issues,\n # consider using the pre-installed 'udica' tool to create custom, targeted SELinux policies instead of\n # disabling SELinux globally. See the \"Fix SELinux issues with udica\" example in the README for details.\n # disable_selinux = false\n\n # Adding extra firewall rules, like opening a port\n # More info on the format here https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/firewall\n # extra_firewall_rules = [\n # {\n # description = \"For Postgres\"\n # direction = \"in\"\n # protocol = \"tcp\"\n # port = \"5432\"\n # source_ips = [\"0.0.0.0/0\", \"::/0\"]\n # destination_ips = [] # Won't be used for this rule\n # },\n # {\n # description = \"To Allow ArgoCD access to resources via SSH\"\n # direction = \"out\"\n # protocol = \"tcp\"\n # port = \"22\"\n # source_ips = [] # Won't be used for this rule\n # destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n # }\n # ]\n\n # If you want to configure a different CNI for k3s, use this flag\n # possible values: flannel (Default), calico, and cilium\n # As for Cilium, we allow infinite configurations via helm values, please check the CNI section of the readme over at https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/#cni.\n # Also, see the cilium_values at towards the end of this file, in the advanced section (or use cilium_merge_values).\n # ⚠️ Depending on your setup, sometimes you need your control-planes to have more than\n # 2GB of RAM if you are going to use Cilium, otherwise the pods will not start.\n # cni_plugin = \"cilium\"\n\n # You can choose the version of Cilium that you want. By default we keep the version up to date and configure Cilium with compatible settings according to the version.\n # See https://github.com/cilium/cilium/releases for the available versions.\n # cilium_version = \"v1.14.0\"\n\n # Set native-routing mode (\"native\") or tunneling mode (\"tunnel\"). Default: tunnel\n # cilium_routing_mode = \"native\"\n\n # Used when Cilium is configured in native routing mode. The CNI assumes that the underlying network stack will forward packets to this destination without the need to apply SNAT. Default: value of \"cluster_ipv4_cidr\"\n # cilium_ipv4_native_routing_cidr = \"10.0.0.0/8\"\n\n # Enables egress gateway to redirect and SNAT the traffic that leaves the cluster. Default: false\n # cilium_egress_gateway_enabled = true\n\n # Enables Hubble Observability to collect and visualize network traffic. Default: false\n # cilium_hubble_enabled = true\n\n # Configures the list of Hubble metrics to collect.\n # cilium_hubble_metrics_enabled = [\n # \"policy:sourceContext=app|workload-name|pod|reserved-identity;destinationContext=app|workload-name|pod|dns|reserved-identity;labelsContext=source_namespace,destination_namespace\"\n # ]\n\n # Set the Cilium LoadBalancer & NodePort XDP Acceleration. Default: \"best-effort\".\n # The setting \"native\" enforces XDP Acceleration on ports and \"disabled\" disabled the acceleration, \"best-effort\" enables the XDP Acceleration if the interface supports it.\n # See [Cilium XDP documentation](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#loadbalancer-nodeport-xdp-acceleration).\n # For Robot nodes connected over vSwitch, the XDP acceleration may not work on the Robot node and the setting therefore recommended to be set to \"best-effort\" or \"disabled\".\n # cilium_loadbalancer_acceleration_mode = \"best-effort\"\n\n # You can choose the version of Calico that you want. By default, the latest is used.\n # More info on available versions can be found at https://github.com/projectcalico/calico/releases\n # Please note that if you are getting 403s from Github, it's also useful to set the version manually. However there is rarely a need for that!\n # calico_version = \"v3.27.2\"\n\n # If you want to disable the k3s kube-proxy, use this flag. The default is \"false\".\n # Ensure that your CNI is capable of handling all the functionalities typically covered by kube-proxy.\n # disable_kube_proxy = true\n\n # If you want to disable the k3s default network policy controller, use this flag!\n # Both Calico and Cilium cni_plugin values override this value to true automatically, the default is \"false\".\n # disable_network_policy = true\n\n # If you want to disable the automatic use of placement group \"spread\". See https://docs.hetzner.com/cloud/placement-groups/overview/\n # We advise to not touch that setting, unless you have a specific purpose.\n # The default is \"false\", meaning it's enabled by default.\n # placement_group_disable = true\n\n # By default, we allow ICMP ping in to the nodes, to check for liveness for instance. If you do not want to allow that, you can. Just set this flag to true (false by default).\n # block_icmp_ping_in = true\n\n # You can enable cert-manager (installed by Helm behind the scenes) with the following flag, the default is \"true\".\n # enable_cert_manager = false\n\n # IP Addresses to use for the DNS Servers, the defaults are the ones provided by Hetzner https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/.\n # The number of different DNS servers is limited to 3 by Kubernetes itself.\n # It's always a good idea to have at least 1 IPv4 and 1 IPv6 DNS server for robustness.\n dns_servers = [\n \"1.1.1.1\",\n \"8.8.8.8\",\n \"2606:4700:4700::1111\",\n ]\n\n # When this is enabled, rather than the first node, all external traffic will be routed via a control-plane loadbalancer, allowing for high availability.\n # The default is false.\n # use_control_plane_lb = true\n\n # When the above use_control_plane_lb is enabled, you can change the lb type for it, the default is \"lb11\".\n # control_plane_lb_type = \"lb21\"\n\n # When the above use_control_plane_lb is enabled, you can change to disable the public interface for control plane load balancer, the default is true.\n # control_plane_lb_enable_public_interface = false\n\n # Let's say you are not using the control plane LB solution above, and still want to have one hostname point to all your control-plane nodes.\n # You could create multiple A records of to let's say cp.cluster.my.org pointing to all of your control-plane nodes ips.\n # In which case, you need to define that hostname in the k3s TLS-SANs config to allow connection through it. It can be hostnames or IP addresses.\n # additional_tls_sans = [\"cp.cluster.my.org\"]\n\n # If you create a hostname with multiple A records pointing to all of your\n # control-plane nodes ips, you may want to use that hostname in the generated\n # kubeconfig.\n # kubeconfig_server_address = \"cp.cluster.my.org\"\n\n # lb_hostname Configuration:\n #\n # Purpose:\n # The lb_hostname setting optimizes communication between services within the Kubernetes cluster\n # when they use domain names instead of direct service names. By associating a domain name directly\n # with the Hetzner Load Balancer, this setting can help reduce potential communication delays.\n #\n # Scenario:\n # If Service B communicates with Service A using a domain (e.g., `a.mycluster.domain.com`) that points\n # to an external Load Balancer, there can be a slowdown in communication.\n #\n # Guidance:\n # - If your internal services use domain names pointing to an external LB, set lb_hostname to a domain\n # like `mycluster.domain.com`.\n # - Create an A record pointing `mycluster.domain.com` to your LB's IP.\n # - Create a CNAME record for `a.mycluster.domain.com` (or xyz.com) pointing to `mycluster.domain.com`.\n #\n # Technical Note:\n # This setting sets the `load-balancer.hetzner.cloud/hostname` in the Hetzner LB definition, suitable for\n # HAProxy, Nginx and Traefik ingress controllers.\n #\n # Recommendation:\n # This setting is optional. If services communicate using direct service names, you can leave this unset.\n # For inter-namespace communication, use `.service_name` as per Kubernetes norms.\n #\n # Example:\n # lb_hostname = \"mycluster.domain.com\"\n\n # You can enable Rancher (installed by Helm behind the scenes) with the following flag, the default is \"false\".\n # ⚠️ Rancher often doesn't support the latest Kubernetes version. You will need to set initial_k3s_channel to a supported version.\n # When Rancher is enabled, it automatically installs cert-manager too, and it uses rancher's own self-signed certificates.\n # See for options https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster#3-choose-your-ssl-configuration\n # The easiest thing is to leave everything as is (using the default rancher self-signed certificate) and put Cloudflare in front of it.\n # As for the number of replicas, by default it is set to the number of control plane nodes.\n # You can customized all of the above by adding a rancher_values variable see at the end of this file in the advanced section (or use rancher_merge_values).\n # After the cluster is deployed, you can always use HelmChartConfig definition to tweak the configuration.\n # IMPORTANT: Rancher's install is quite memory intensive, you will require at least 4GB if RAM, meaning cx23 server type (for your control plane).\n # ALSO, in order for Rancher to successfully deploy, you have to set the \"rancher_hostname\".\n # enable_rancher = true\n\n # If using Rancher you can set the Rancher hostname, it must be unique hostname even if you do not use it.\n # If not pointing the DNS, you can just port-forward locally via kubectl to get access to the dashboard.\n # If you already set the lb_hostname above and are using a Hetzner LB, you do not need to set this one, as it will be used by default.\n # But if you set this one explicitly, it will have preference over the lb_hostname in rancher settings.\n # rancher_hostname = \"rancher.xyz.dev\"\n\n # When Rancher is deployed, by default is uses the \"latest\" channel. But this can be customized.\n # The allowed values are \"stable\" or \"latest\".\n # rancher_install_channel = \"stable\"\n\n # Finally, you can specify a bootstrap-password for your rancher instance. Minimum 48 characters long!\n # If you leave empty, one will be generated for you.\n # (Can be used by another rancher2 provider to continue setup of rancher outside this module.)\n # rancher_bootstrap_password = \"\"\n\n # Separate from the above Rancher config (only use one or the other). You can import this cluster directly on an\n # an already active Rancher install. By clicking \"import cluster\" choosing \"generic\", giving it a name and pasting\n # the cluster registration url below. However, you can also ignore that and apply the url via kubectl as instructed\n # by Rancher in the wizard, and that would register your cluster too.\n # More information about the registration can be found here https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/\n # rancher_registration_manifest_url = \"https://rancher.xyz.dev/v3/import/xxxxxxxxxxxxxxxxxxYYYYYYYYYYYYYYYYYYYzzzzzzzzzzzzzzzzzzzzz.yaml\"\n\n # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).\n # extra_kustomize_deployment_commands=\"\"\n\n # Extra values that will be passed to the `extra-manifests/kustomization.yaml.tpl` if its present.\n # extra_kustomize_parameters={}\n\n # See working examples for extra manifests or a HelmChart in examples/kustomization_user_deploy/README.md\n\n # It is best practice to turn this off, but for backwards compatibility it is set to \"true\" by default.\n # See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/349\n # When \"false\". The kubeconfig file can instead be created by executing: \"terraform output --raw kubeconfig > cluster_kubeconfig.yaml\"\n # Always be careful to not commit this file!\n # create_kubeconfig = false\n\n # Don't create the kustomize backup. This can be helpful for automation.\n # create_kustomization = false\n\n # Export the values.yaml files used for the deployment of traefik, longhorn, cert-manager, etc.\n # This can be helpful to use them for later deployments like with ArgoCD.\n # The default is false.\n # export_values = true\n\n # MicroOS snapshot IDs to be used. Per default empty, the most recent image created using createkh will be used.\n # We recommend the default, but if you want to use specific IDs you can.\n # You can fetch the ids with the hcloud cli by running the \"hcloud image list --selector 'microos-snapshot=yes'\" command.\n # microos_x86_snapshot_id = \"1234567\"\n # microos_arm_snapshot_id = \"1234567\"\n\n ### ADVANCED - Custom helm values for packages above (search _values if you want to located where those are mentioned upper in this file)\n # ⚠️ Inside the _values variable below are examples, up to you to find out the best helm values possible, we do not provide support for customized helm values.\n # Please understand that the indentation is very important, inside the EOTs, as those are proper yaml helm values.\n # We advise you to use the default values, and only change them if you know what you are doing!\n\n # You can inline the values here in heredoc-style (as the examples below with the <<-EOT to EOT). Please note that the indentation inside the EOT is important.\n # Or you can create a thepackage-values.yaml file with the content and use it here with the following syntax:\n # thepackage_values = file(\"thepackage-values.yaml\")\n # _values fully replaces the chart values used by the module. _merge_values keeps the defaults (or *_values if set) and overlays your YAML on top.\n\n # Cilium, all Cilium helm values can be found at https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/values.yaml\n # Be careful when maintaining your own cilium_values, as the choice of available settings depends on the Cilium version used. See also the cilium_version setting to fix a specific version.\n # If you want to merge extra values into defaults (or cilium_values), use cilium_merge_values.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* cilium_values = <<-EOT\n\nipam:\n mode: kubernetes\nk8s:\n requireIPv4PodCIDR: true\nkubeProxyReplacement: true\nroutingMode: native\nipv4NativeRoutingCIDR: \"10.0.0.0/8\"\nendpointRoutes:\n enabled: true\nloadBalancer:\n acceleration: native\nbpf:\n masquerade: true\nencryption:\n enabled: true\n type: wireguard\nMTU: 1450\n EOT */\n\n /* cilium_merge_values = <<-EOT\nencryption:\n enabled: true\n type: wireguard\n EOT */\n\n # If you want to use a specific cert-manager helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # cert_manager_version = \"\"\n\n # Cert manager, all cert-manager helm values can be found at https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml\n # If you want to merge extra values into defaults (or cert_manager_values), use cert_manager_merge_values.\n # The following is an example, please note that the current indentation inside the EOT is important.\n # For cert-manager versions < v1.15.0, you need to set installCRDs: true instead of crds.enabled and crds.keep.\n /* cert_manager_values = <<-EOT\ncrds:\n enabled: true\n keep: true\nreplicaCount: 3\nwebhook:\n replicaCount: 3\ncainjector:\n replicaCount: 3\n EOT */\n\n /* cert_manager_merge_values = <<-EOT\nwebhook:\n replicaCount: 2\n EOT */\n\n # Hetzner Cloud Controller Manager, all Hetzner Cloud Controller Manager helm values can be found at https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/chart/values.yaml\n # We advise you to not touch this and to let the defaults that are already set under the hood.\n # If you want to merge extra values into defaults (or hetzner_ccm_values), use hetzner_ccm_merge_values.\n # For advanced use cases like adding Hetzner Robot servers, see: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* hetzner_ccm_values = <<-EOT\nnetworking:\n enabled: true\nargs:\n cloud-provider: hcloud\n allow-untagged-cloud: \"\"\n route-reconciliation-period: 30s\n webhook-secure-port: \"0\"\nenv:\n HCLOUD_LOAD_BALANCERS_LOCATION:\n value: \"nbg1\"\n HCLOUD_LOAD_BALANCERS_USE_PRIVATE_IP:\n value: \"true\"\n HCLOUD_LOAD_BALANCERS_ENABLED:\n value: \"true\"\n HCLOUD_LOAD_BALANCERS_DISABLE_PRIVATE_INGRESS:\n value: \"true\"\n EOT */\n\n /* hetzner_ccm_merge_values = <<-EOT\nenv:\n HCLOUD_LOAD_BALANCERS_LOCATION:\n value: \"nbg1\"\n EOT */\n\n # csi-driver-smb, all csi-driver-smb helm values can be found at https://github.com/kubernetes-csi/csi-driver-smb/blob/master/charts/latest/csi-driver-smb/values.yaml\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* csi_driver_smb_values = <<-EOT\ncontroller:\n name: csi-smb-controller\n replicas: 1\n runOnMaster: false\n runOnControlPlane: false\n resources:\n csiProvisioner:\n limits:\n memory: 300Mi\n requests:\n cpu: 10m\n memory: 20Mi\n livenessProbe:\n limits:\n memory: 100Mi\n requests:\n cpu: 10m\n memory: 20Mi\n smb:\n limits:\n memory: 200Mi\n requests:\n cpu: 10m\n memory: 20Mi\n EOT */\n\n # Longhorn, all Longhorn helm values can be found at https://github.com/longhorn/longhorn/blob/master/chart/values.yaml\n # If you want to merge extra values into defaults (or longhorn_values), use longhorn_merge_values.\n # longhorn_values replaces the module defaults. For targeted overrides (for example hotfix image tags), prefer longhorn_merge_values.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* longhorn_values = <<-EOT\ndefaultSettings:\n defaultDataPath: /var/longhorn\npersistence:\n defaultFsType: ext4\n defaultClassReplicaCount: 3\n defaultClass: true\n EOT */\n\n /* longhorn_merge_values = <<-EOT\ndefaultSettings:\n defaultReplicaCount: 2\n EOT */\n\n # Example: apply upstream Longhorn hotfix tags without replacing all module defaults.\n /* longhorn_merge_values = <<-EOT\nimage:\n longhorn:\n manager:\n tag: v1.11.0-hotfix-1\n instanceManager:\n tag: v1.11.0-hotfix-1\n EOT */\n\n # If you want to use a specific Traefik helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # See https://github.com/traefik/traefik-helm-chart/releases for the available versions.\n # traefik_version = \"\"\n\n # Traefik, all Traefik helm values can be found at https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml\n # If you want to merge extra values into defaults (or traefik_values), use traefik_merge_values.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* traefik_values = <<-EOT\ndeployment:\n replicas: 1\nadditionalArguments: []\nservice:\n enabled: true\n type: LoadBalancer\n annotations:\n \"load-balancer.hetzner.cloud/name\": \"k3s\"\n \"load-balancer.hetzner.cloud/use-private-ip\": \"true\"\n \"load-balancer.hetzner.cloud/disable-private-ingress\": \"true\"\n \"load-balancer.hetzner.cloud/location\": \"nbg1\"\n \"load-balancer.hetzner.cloud/type\": \"lb11\"\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"true\"\n\nports:\n web:\n http:\n redirections:\n entryPoint:\n to: websecure\n scheme: https\n permanent: true\n\n proxyProtocol:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n forwardedHeaders:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n websecure:\n proxyProtocol:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n forwardedHeaders:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n EOT */\n\n /* traefik_merge_values = <<-EOT\nservice:\n annotations:\n \"load-balancer.hetzner.cloud/location\": \"fsn1\"\n EOT */\n\n # If you want to use a specific Nginx helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # See https://github.com/kubernetes/ingress-nginx?tab=readme-ov-file#supported-versions-table for the available versions.\n # nginx_version = \"\"\n\n # Nginx, all Nginx helm values can be found at https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml\n # You can also have a look at https://kubernetes.github.io/ingress-nginx/, to understand how it works, and all the options at your disposal.\n # If you want to merge extra values into defaults (or nginx_values), use nginx_merge_values.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* nginx_values = <<-EOT\ncontroller:\n watchIngressWithoutClass: \"true\"\n kind: \"DaemonSet\"\n config:\n \"use-forwarded-headers\": \"true\"\n \"compute-full-forwarded-for\": \"true\"\n \"use-proxy-protocol\": \"true\"\n service:\n annotations:\n \"load-balancer.hetzner.cloud/name\": \"k3s\"\n \"load-balancer.hetzner.cloud/use-private-ip\": \"true\"\n \"load-balancer.hetzner.cloud/disable-private-ingress\": \"true\"\n \"load-balancer.hetzner.cloud/location\": \"nbg1\"\n \"load-balancer.hetzner.cloud/type\": \"lb11\"\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"true\"\n EOT */\n\n /* nginx_merge_values = <<-EOT\ncontroller:\n kind: \"Deployment\"\n EOT */\n\n # If you want to use a specific HAProxy helm chart version, set it below; otherwise, leave them as-is for the latest versions.\n # haproxy_version = \"\"\n\n # If you want to configure additional proxy protocol trusted IPs for haproxy, enter them here as a list of IPs (strings).\n # Example for Cloudflare:\n # haproxy_additional_proxy_protocol_ips = [\n # \"173.245.48.0/20\",\n # \"103.21.244.0/22\",\n # \"103.22.200.0/22\",\n # \"103.31.4.0/22\",\n # \"141.101.64.0/18\",\n # \"108.162.192.0/18\",\n # \"190.93.240.0/20\",\n # \"188.114.96.0/20\",\n # \"197.234.240.0/22\",\n # \"198.41.128.0/17\",\n # \"162.158.0.0/15\",\n # \"104.16.0.0/13\",\n # \"104.24.0.0/14\",\n # \"172.64.0.0/13\",\n # \"131.0.72.0/22\",\n # \"2400:cb00::/32\",\n # \"2606:4700::/32\",\n # \"2803:f800::/32\",\n # \"2405:b500::/32\",\n # \"2405:8100::/32\",\n # \"2a06:98c0::/29\",\n # \"2c0f:f248::/32\"\n # ]\n\n # Configure CPU and memory requests for each HAProxy pod\n # haproxy_requests_cpu = \"250m\"\n # haproxy_requests_memory = \"400Mi\"\n\n # Override values given to the HAProxy helm chart.\n # All HAProxy helm values can be found at https://github.com/haproxytech/helm-charts/blob/main/kubernetes-ingress/values.yaml\n # Default values can be found at https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/locals.tf\n # If you want to merge extra values into defaults (or haproxy_values), use haproxy_merge_values.\n /* haproxy_values = <<-EOT\n EOT */\n\n /* haproxy_merge_values = <<-EOT\ncontroller:\n replicaCount: 2\n EOT */\n\n # Rancher, all Rancher helm values can be found at https://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/chart-options/\n # If you want to merge extra values into defaults (or rancher_values), use rancher_merge_values.\n # The following is an example, please note that the current indentation inside the EOT is important.\n /* rancher_values = <<-EOT\ningress:\n tls:\n source: \"rancher\"\nhostname: \"rancher.example.com\"\nreplicas: 1\nbootstrapPassword: \"supermario\"\n EOT */\n\n /* rancher_merge_values = <<-EOT\nreplicas: 3\n EOT */\n\n}\n\nprovider \"hcloud\" {\n token = var.hcloud_token != \"\" ? var.hcloud_token : local.hcloud_token\n}\n\nterraform {\n required_version = \">= 1.10.1\"\n required_providers {\n hcloud = {\n source = \"hetznercloud/hcloud\"\n version = \">= 1.51.0\"\n }\n }\n}\n\noutput \"kubeconfig\" {\n value = module.kube-hetzner.kubeconfig\n sensitive = true\n}\n\nvariable \"hcloud_token\" {\n sensitive = true\n default = \"\"\n}\n\nvariable \"robot_user\" {\n sensitive = true\n default = \"\"\n}\n\nvariable \"robot_password\" {\n sensitive = true\n default = \"\"\n}\n" }, { "path": "kubeconfig.tf", "content": "resource \"ssh_sensitive_resource\" \"kubeconfig\" {\n # Note: moved from remote_file to ssh_sensitive_resource because\n # remote_file does not support bastion hosts and ssh_sensitive_resource does.\n # The default behaviour is to run file blocks and commands at create time\n # You can also specify 'destroy' to run the commands at destroy time\n when = \"create\"\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n host = can(ipv6(local.first_control_plane_ip)) ? \"[${local.first_control_plane_ip}]\" : local.first_control_plane_ip\n port = var.ssh_port\n user = \"root\"\n private_key = var.ssh_private_key\n agent = var.ssh_private_key == null\n\n # An ssh-agent with your SSH private keys should be running\n # Use 'private_key' to set the SSH key otherwise\n\n timeout = \"15m\"\n\n commands = [\n \"cat /etc/rancher/k3s/k3s.yaml\"\n ]\n\n depends_on = [terraform_data.control_planes[0]]\n}\n\nlocals {\n kubeconfig_server_address = var.kubeconfig_server_address != \"\" ? var.kubeconfig_server_address : (var.use_control_plane_lb ?\n (\n var.control_plane_lb_enable_public_interface ?\n hcloud_load_balancer.control_plane.*.ipv4[0]\n : (\n var.nat_router != null ?\n hcloud_server.nat_router[0].ipv4_address\n : hcloud_load_balancer_network.control_plane.*.ip[0]\n )\n )\n :\n (can(local.first_control_plane_ip) ? local.first_control_plane_ip : \"unknown\")\n )\n kubeconfig_external = replace(replace(ssh_sensitive_resource.kubeconfig.result, \"127.0.0.1\", local.kubeconfig_server_address), \"default\", var.cluster_name)\n kubeconfig_parsed = yamldecode(local.kubeconfig_external)\n kubeconfig_data = {\n host = local.kubeconfig_parsed[\"clusters\"][0][\"cluster\"][\"server\"]\n client_certificate = base64decode(local.kubeconfig_parsed[\"users\"][0][\"user\"][\"client-certificate-data\"])\n client_key = base64decode(local.kubeconfig_parsed[\"users\"][0][\"user\"][\"client-key-data\"])\n cluster_ca_certificate = base64decode(local.kubeconfig_parsed[\"clusters\"][0][\"cluster\"][\"certificate-authority-data\"])\n cluster_name = var.cluster_name\n }\n}\n\nresource \"local_sensitive_file\" \"kubeconfig\" {\n count = var.create_kubeconfig ? 1 : 0\n content = local.kubeconfig_external\n filename = \"${var.cluster_name}_kubeconfig.yaml\"\n file_permission = \"600\"\n}\n" }, { "path": "kustomization_backup.tf", "content": "resource \"local_file\" \"kustomization_backup\" {\n count = var.create_kustomization ? 1 : 0\n content = local.kustomization_backup_yaml\n filename = \"${var.cluster_name}_kustomization_backup.yaml\"\n file_permission = \"600\"\n}\n" }, { "path": "kustomization_user.tf", "content": "locals {\n user_kustomization_templates = try(fileset(var.extra_kustomize_folder, \"**/*.yaml.tpl\"), toset([]))\n}\n\nresource \"terraform_data\" \"kustomization_user\" {\n for_each = local.user_kustomization_templates\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.first_control_plane_ip\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n provisioner \"remote-exec\" {\n inline = [\n \"mkdir -p $(dirname /var/user_kustomize/${each.key})\"\n ]\n }\n\n provisioner \"file\" {\n content = templatefile(\"${var.extra_kustomize_folder}/${each.key}\", var.extra_kustomize_parameters)\n destination = replace(\"/var/user_kustomize/${each.key}\", \".yaml.tpl\", \".yaml\")\n }\n\n triggers_replace = {\n manifest_sha1 = \"${sha1(templatefile(\"${var.extra_kustomize_folder}/${each.key}\", var.extra_kustomize_parameters))}\"\n }\n\n depends_on = [\n terraform_data.kustomization\n ]\n}\nmoved {\n from = null_resource.kustomization_user\n to = terraform_data.kustomization_user\n}\n\nresource \"terraform_data\" \"kustomization_user_deploy\" {\n count = length(local.user_kustomization_templates) > 0 ? 1 : 0\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = local.first_control_plane_ip\n port = var.ssh_port\n\n bastion_host = local.ssh_bastion.bastion_host\n bastion_port = local.ssh_bastion.bastion_port\n bastion_user = local.ssh_bastion.bastion_user\n bastion_private_key = local.ssh_bastion.bastion_private_key\n\n }\n\n # Remove templates after rendering, and apply changes.\n provisioner \"remote-exec\" {\n # Debugging: \"sh -c 'for file in $(find /var/user_kustomize -type f -name \\\"*.yaml\\\" | sort -n); do echo \\\"\\n### Template $${file}.tpl after rendering:\\\" && cat $${file}; done'\",\n inline = compact([\n \"rm -f /var/user_kustomize/**/*.yaml.tpl\",\n \"echo 'Applying user kustomization...'\",\n \"kubectl apply -k /var/user_kustomize/ --wait=true\",\n var.extra_kustomize_deployment_commands\n ])\n }\n\n lifecycle {\n replace_triggered_by = [\n terraform_data.kustomization_user\n ]\n }\n\n depends_on = [\n terraform_data.kustomization_user\n ]\n}\nmoved {\n from = null_resource.kustomization_user_deploy\n to = terraform_data.kustomization_user_deploy\n}\n" }, { "path": "kustomize/flannel-rbac.yaml", "content": "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: flannel-node-lister\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: system:node\nsubjects:\n- kind: Group\n name: system:nodes\n apiGroup: rbac.authorization.k8s.io \n" }, { "path": "kustomize/system-upgrade-controller.yaml", "content": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: system-upgrade-controller\n namespace: system-upgrade\nspec:\n template:\n spec:\n containers:\n - name: system-upgrade-controller\n volumeMounts:\n - name: ca-certificates\n mountPath: /var/lib/ca-certificates\n volumes:\n - name: ca-certificates\n hostPath:\n path: /var/lib/ca-certificates\n type: Directory\n" }, { "path": "locals.tf", "content": "locals {\n # ssh_agent_identity is not set if the private key is passed directly, but if ssh agent is used, the public key tells ssh agent which private key to use.\n # For terraforms provisioner.connection.agent_identity, we need the public key as a string.\n ssh_agent_identity = var.ssh_private_key == null ? var.ssh_public_key : null\n\n # If passed, a key already registered within hetzner is used.\n # Otherwise, a new one will be created by the module.\n hcloud_ssh_key_id = var.hcloud_ssh_key_id == null ? hcloud_ssh_key.k3s[0].id : var.hcloud_ssh_key_id\n\n # if given as a variable, we want to use the given token. This is needed to restore the cluster\n k3s_token = var.k3s_token == null ? random_password.k3s_token.result : var.k3s_token\n\n # k3s endpoint used for agent registration, respects control_plane_endpoint override\n k3s_endpoint = coalesce(var.control_plane_endpoint, \"https://${var.use_control_plane_lb ? hcloud_load_balancer_network.control_plane.*.ip[0] : module.control_planes[keys(module.control_planes)[0]].private_ipv4_address}:6443\")\n\n ccm_version = var.hetzner_ccm_version != null ? var.hetzner_ccm_version : data.github_release.hetzner_ccm[0].release_tag\n csi_version = length(data.github_release.hetzner_csi) == 0 ? var.hetzner_csi_version : data.github_release.hetzner_csi[0].release_tag\n kured_version = length(data.github_release.kured) == 0 ? var.kured_version : data.github_release.kured[0].release_tag\n calico_version = length(data.github_release.calico) == 0 ? var.calico_version : data.github_release.calico[0].release_tag\n\n # Determine kured YAML suffix based on version (>= 1.20.0 uses -combined.yaml, < 1.20.0 uses -dockerhub.yaml)\n kured_yaml_suffix = provider::semvers::compare(local.kured_version, \"1.20.0\") >= 0 ? \"combined\" : \"dockerhub\"\n\n cilium_ipv4_native_routing_cidr = coalesce(var.cilium_ipv4_native_routing_cidr, var.cluster_ipv4_cidr)\n\n # Check if the user has set custom DNS servers.\n has_dns_servers = length(var.dns_servers) > 0\n\n # Bit size of the \"network_ipv4_cidr\".\n network_size = 32 - split(\"/\", var.network_ipv4_cidr)[1]\n\n # Bit size of each subnet\n subnet_size = local.network_size - log(var.subnet_amount, 2)\n\n # Separate out IPv4 and IPv6 DNS hosts.\n dns_servers_ipv4 = [for ip in var.dns_servers : ip if provider::assert::ipv4(ip)]\n dns_servers_ipv6 = [for ip in var.dns_servers : ip if provider::assert::ipv6(ip)]\n\n use_robot_ccm = var.robot_ccm_enabled && var.robot_user != \"\" && var.robot_password != \"\"\n # Key of the kube_system_secret-items is the name of the Secret. Values of those items are the key-value pairs of Secret.\n kube_system_secrets = {\n \"hcloud\" = merge(\n {\n \"token\" = var.hcloud_token,\n \"network\" = data.hcloud_network.k3s.name\n },\n local.use_robot_ccm ? {\n \"robot-user\" = var.robot_user,\n \"robot-password\" = var.robot_password\n } : {}\n ),\n \"hcloud-csi\" = { \"token\" = var.hcloud_token }\n }\n\n additional_k3s_environment = join(\"\\n\",\n [\n for var_name, var_value in var.additional_k3s_environment :\n \"${var_name}=\\\"${var_value}\\\"\"\n ]\n )\n install_additional_k3s_environment = <<-EOT\n cat >> /etc/environment <<EOF\n ${local.additional_k3s_environment}\n EOF\n set -a; source /etc/environment; set +a;\n EOT\n\n install_system_alias = <<-EOT\n cat > /etc/profile.d/00-alias.sh <<EOF\n alias k=kubectl\n EOF\n EOT\n\n install_kubectl_bash_completion = <<-EOT\n cat > /etc/bash_completion.d/kubectl <<EOF\n if command -v kubectl >/dev/null; then\n source <(kubectl completion bash)\n complete -o default -F __start_kubectl k\n fi\n EOF\n EOT\n\n common_pre_install_k3s_commands = concat(\n [\n \"set -ex\",\n # rename the private network interface to eth1\n \"/etc/cloud/rename_interface.sh\",\n # prepare the k3s config directory\n \"mkdir -p /etc/rancher/k3s\",\n # move the config file into place and adjust permissions\n \"[ -f /tmp/config.yaml ] && mv /tmp/config.yaml /etc/rancher/k3s/config.yaml\",\n \"chmod 0600 /etc/rancher/k3s/config.yaml\",\n # if the server has already been initialized just stop here\n \"[ -e /etc/rancher/k3s/k3s.yaml ] && exit 0\",\n local.install_additional_k3s_environment,\n local.install_system_alias,\n local.install_kubectl_bash_completion,\n ],\n local.has_dns_servers ? [\n join(\"\\n\", compact([\n \"# Wait for NetworkManager to be ready\",\n \"if ! timeout 60 bash -c 'until systemctl is-active --quiet NetworkManager; do echo \\\"Waiting for NetworkManager to be ready...\\\"; sleep 2; done'; then\",\n \" echo \\\"ERROR: NetworkManager is not active after timeout\\\" >&2\",\n \" exit 0 # Don't fail cloud-init\",\n \"fi\",\n \"# Get the default interface\",\n \"IFACE=$(ip route show default 2>/dev/null | awk '/^default/ && /dev/ {for(i=1;i<=NF;i++) if($i==\\\"dev\\\") {print $(i+1); exit}}')\",\n \"if [ -z \\\"$IFACE\\\" ]; then\",\n \" # Fallback: try to get any interface that's up and has an IP\",\n \" IFACE=$(ip route show 2>/dev/null | awk '!/^default/ && /dev/ {for(i=1;i<=NF;i++) if($i==\\\"dev\\\") {print $(i+1); exit}}')\",\n \"fi\",\n \"if [ -n \\\"$IFACE\\\" ]; then\",\n \" CONNECTION=$(nmcli -g GENERAL.CONNECTION device show \\\"$IFACE\\\" 2>/dev/null | head -1)\",\n \" if [ -n \\\"$CONNECTION\\\" ]; then\",\n \" # Disable auto-DNS for both protocols when custom DNS servers are provided\",\n \" nmcli con mod \\\"$CONNECTION\\\" ipv4.ignore-auto-dns yes ipv6.ignore-auto-dns yes\",\n length(local.dns_servers_ipv4) > 0 ? \" nmcli con mod \\\"$CONNECTION\\\" ipv4.dns ${join(\",\", local.dns_servers_ipv4)}\" : \"\",\n length(local.dns_servers_ipv6) > 0 ? \" nmcli con mod \\\"$CONNECTION\\\" ipv6.dns ${join(\",\", local.dns_servers_ipv6)}\" : \"\",\n \" fi\",\n \"fi\"\n ]))\n ] : [],\n local.has_dns_servers ? [\"systemctl restart NetworkManager\"] : [],\n [\n join(\"\\n\", [\n \"# Ensure persistent private-network default route (Hetzner DHCP change Aug 11, 2025)\",\n \"set +e # Allow idempotent network adjustments\",\n \"METRIC=30000\",\n \"\",\n \"# Determine the private interface dynamically (no hardcoded eth1)\",\n \"PRIV_IF=$(ip -4 route show ${var.network_ipv4_cidr} 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i==\\\"dev\\\"){print $(i+1); exit}}' | head -n 1)\",\n \"if [ -z \\\"$PRIV_IF\\\" ]; then\",\n \" ROUTE_LINE=$(ip -4 route get ${local.network_gw_ipv4} 2>/dev/null)\",\n \" if [ -n \\\"$ROUTE_LINE\\\" ] && ! echo \\\"$ROUTE_LINE\\\" | grep -q ' via '; then\",\n \" PRIV_IF=$(echo \\\"$ROUTE_LINE\\\" | awk '{for(i=1;i<=NF;i++) if($i==\\\"dev\\\"){print $(i+1); exit}}' | head -n 1)\",\n \" fi\",\n \"fi\",\n \"if [ -n \\\"$PRIV_IF\\\" ]; then\",\n \" if systemctl is-active --quiet NetworkManager; then\",\n \" NM_CONN=$(nmcli -g GENERAL.CONNECTION device show \\\"$PRIV_IF\\\" 2>/dev/null | head -1)\",\n \" if [ -n \\\"$NM_CONN\\\" ]; then\",\n \" # Persist a default route via the private gateway with higher metric than public NICs\",\n \" ROUTE_READY=0\",\n \" ROUTE_LINE=$(nmcli -g ipv4.routes connection show \\\"$NM_CONN\\\" | tr ',' '\\\\n' | awk '$1==\\\"0.0.0.0/0\\\" && $2==\\\"${local.network_gw_ipv4}\\\"{print $0; exit}')\",\n \" if [ -n \\\"$ROUTE_LINE\\\" ]; then\",\n \" CUR_ROUTE_METRIC=$(echo \\\"$ROUTE_LINE\\\" | awk '{print $3}')\",\n \" if [ -z \\\"$CUR_ROUTE_METRIC\\\" ] || [ \\\"$CUR_ROUTE_METRIC\\\" != \\\"$METRIC\\\" ]; then\",\n \" nmcli connection modify \\\"$NM_CONN\\\" -ipv4.routes \\\"$ROUTE_LINE\\\" >/dev/null 2>&1 || true\",\n \" if nmcli connection modify \\\"$NM_CONN\\\" +ipv4.routes \\\"0.0.0.0/0 ${local.network_gw_ipv4} $METRIC\\\" >/dev/null 2>&1; then\",\n \" ROUTE_READY=1\",\n \" else\",\n \" echo \\\"Warning: Failed to update default route metric on $PRIV_IF. Node may be affected by Hetzner DHCP changes.\\\" >&2\",\n \" fi\",\n \" else\",\n \" ROUTE_READY=1\",\n \" fi\",\n \" else\",\n \" if nmcli connection modify \\\"$NM_CONN\\\" +ipv4.routes \\\"0.0.0.0/0 ${local.network_gw_ipv4} $METRIC\\\" >/dev/null 2>&1; then\",\n \" ROUTE_READY=1\",\n \" else\",\n \" echo \\\"Warning: Failed to persist default route on $PRIV_IF. Node may be affected by Hetzner DHCP changes.\\\" >&2\",\n \" fi\",\n \" fi\",\n \" if [ \\\"$ROUTE_READY\\\" -eq 1 ]; then\",\n \" nmcli connection modify \\\"$NM_CONN\\\" ipv4.never-default yes >/dev/null 2>&1 || true\",\n \" nmcli connection modify \\\"$NM_CONN\\\" ipv6.never-default yes >/dev/null 2>&1 || true\",\n \" nmcli connection modify \\\"$NM_CONN\\\" ipv4.route-metric $METRIC >/dev/null 2>&1 || true\",\n \" nmcli connection up \\\"$NM_CONN\\\" >/dev/null 2>&1 || true\",\n \" fi\",\n \" fi\",\n \" fi\",\n \" # Runtime guard to cover current leases before dispatcher hooks fire\",\n \" EXISTING_RT=$(ip -4 route show default dev \\\"$PRIV_IF\\\" | awk '$3==\\\"${local.network_gw_ipv4}\\\"{print $0; exit}')\",\n \" if [ -n \\\"$EXISTING_RT\\\" ]; then\",\n \" CUR_RT_METRIC=$(echo \\\"$EXISTING_RT\\\" | awk 'match($0,/metric ([0-9]+)/,m){print m[1]}')\",\n \" if [ -z \\\"$CUR_RT_METRIC\\\" ] || [ \\\"$CUR_RT_METRIC\\\" != \\\"$METRIC\\\" ]; then\",\n \" ip -4 route change default via ${local.network_gw_ipv4} dev \\\"$PRIV_IF\\\" metric $METRIC 2>/dev/null || true\",\n \" fi\",\n \" else\",\n \" ip -4 route add default via ${local.network_gw_ipv4} dev \\\"$PRIV_IF\\\" metric $METRIC 2>/dev/null || true\",\n \" fi\",\n \"else\",\n \" echo \\\"Info: Unable to identify interface that reaches ${local.network_gw_ipv4}; skipping private default route setup.\\\"\",\n \"fi\",\n \"\",\n \"set -e\"\n ])\n ],\n # User-defined commands to execute just before installing k3s.\n var.preinstall_exec,\n # Wait for a successful connection to the internet.\n [\"timeout 180s /bin/sh -c 'while ! ping -c 1 ${var.address_for_connectivity_test} >/dev/null 2>&1; do echo \\\"Ready for k3s installation, waiting for a successful connection to the internet...\\\"; sleep 5; done; echo Connected'\"]\n )\n\n common_post_install_k3s_commands = concat(var.postinstall_exec, [\"restorecon -v /usr/local/bin/k3s\"])\n\n kustomization_backup_yaml = yamlencode({\n apiVersion = \"kustomize.config.k8s.io/v1beta1\"\n kind = \"Kustomization\"\n resources = concat(\n [\n \"https://github.com/kubereboot/kured/releases/download/${local.kured_version}/kured-${local.kured_version}-${local.kured_yaml_suffix}.yaml\",\n \"https://github.com/rancher/system-upgrade-controller/releases/download/${var.sys_upgrade_controller_version}/system-upgrade-controller.yaml\",\n \"https://github.com/rancher/system-upgrade-controller/releases/download/${var.sys_upgrade_controller_version}/crd.yaml\"\n ],\n var.hetzner_ccm_use_helm ? [\"hcloud-ccm-helm.yaml\"] : [\"https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/${local.ccm_version}/ccm-networks.yaml\"],\n var.disable_hetzner_csi ? [] : [\"hcloud-csi.yaml\"],\n lookup(local.ingress_controller_install_resources, var.ingress_controller, []),\n lookup(local.cni_install_resources, var.cni_plugin, []),\n var.cni_plugin == \"flannel\" ? [\"flannel-rbac.yaml\"] : [],\n var.enable_longhorn ? [\"longhorn.yaml\"] : [],\n var.enable_csi_driver_smb ? [\"csi-driver-smb.yaml\"] : [],\n var.enable_cert_manager || var.enable_rancher ? [\"cert_manager.yaml\"] : [],\n var.enable_rancher ? [\"rancher.yaml\"] : [],\n var.rancher_registration_manifest_url != \"\" ? [var.rancher_registration_manifest_url] : []\n ),\n patches = concat([\n {\n target = {\n group = \"apps\"\n version = \"v1\"\n kind = \"Deployment\"\n name = \"system-upgrade-controller\"\n namespace = \"system-upgrade\"\n }\n patch = file(\"${path.module}/kustomize/system-upgrade-controller.yaml\")\n },\n {\n path = \"kured.yaml\"\n }\n ],\n var.hetzner_ccm_use_helm ? [] : [{ path = \"ccm.yaml\" }]\n )\n })\n\n apply_k3s_selinux = [\"/sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp\"]\n swap_node_label = [\"node.kubernetes.io/server-swap=enabled\"]\n\n k3s_install_command = \"curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true %{if var.install_k3s_version == \"\"}INSTALL_K3S_CHANNEL=${var.initial_k3s_channel}%{else}INSTALL_K3S_VERSION=${var.install_k3s_version}%{endif} INSTALL_K3S_EXEC='%s' sh -\"\n\n install_k3s_server = concat(\n local.common_pre_install_k3s_commands,\n [format(local.k3s_install_command, \"server ${var.k3s_exec_server_args}\")],\n var.disable_selinux ? [] : local.apply_k3s_selinux,\n local.common_post_install_k3s_commands\n )\n\n install_k3s_agent = concat(\n local.common_pre_install_k3s_commands,\n [format(local.k3s_install_command, \"agent ${var.k3s_exec_agent_args}\")],\n var.disable_selinux ? [] : local.apply_k3s_selinux,\n local.common_post_install_k3s_commands\n )\n\n control_plane_nodes = merge([\n for pool_index, nodepool_obj in var.control_plane_nodepools : {\n for node_index in range(nodepool_obj.count) :\n format(\"%s-%s-%s\", pool_index, node_index, nodepool_obj.name) => {\n nodepool_name : nodepool_obj.name,\n server_type : nodepool_obj.server_type,\n location : nodepool_obj.location,\n labels : concat(local.default_control_plane_labels, nodepool_obj.swap_size != \"\" || nodepool_obj.zram_size != \"\" ? local.swap_node_label : [], nodepool_obj.labels),\n taints : compact(concat(local.default_control_plane_taints, nodepool_obj.taints)),\n kubelet_args : nodepool_obj.kubelet_args,\n backups : nodepool_obj.backups,\n swap_size : nodepool_obj.swap_size,\n zram_size : nodepool_obj.zram_size,\n index : node_index\n selinux : nodepool_obj.selinux\n placement_group_compat_idx : nodepool_obj.placement_group_compat_idx,\n placement_group : nodepool_obj.placement_group,\n disable_ipv4 : nodepool_obj.disable_ipv4 || local.use_nat_router,\n disable_ipv6 : nodepool_obj.disable_ipv6 || local.use_nat_router,\n network_id : nodepool_obj.network_id,\n }\n }\n ]...)\n\n agent_nodes_from_integer_counts = merge([\n for pool_index, nodepool_obj in var.agent_nodepools : {\n # coalesce(nodepool_obj.count, 0) means we select those nodepools who's size is set by an integer count.\n for node_index in range(coalesce(nodepool_obj.count, 0)) :\n format(\"%s-%s-%s\", pool_index, node_index, nodepool_obj.name) => {\n nodepool_name : nodepool_obj.name,\n server_type : nodepool_obj.server_type,\n longhorn_volume_size : coalesce(nodepool_obj.longhorn_volume_size, 0),\n longhorn_mount_path : nodepool_obj.longhorn_mount_path,\n floating_ip : lookup(nodepool_obj, \"floating_ip\", false),\n floating_ip_rdns : lookup(nodepool_obj, \"floating_ip_rdns\", false),\n location : nodepool_obj.location,\n labels : concat(local.default_agent_labels, nodepool_obj.swap_size != \"\" || nodepool_obj.zram_size != \"\" ? local.swap_node_label : [], nodepool_obj.labels),\n taints : compact(concat(local.default_agent_taints, nodepool_obj.taints)),\n kubelet_args : nodepool_obj.kubelet_args,\n backups : lookup(nodepool_obj, \"backups\", false),\n swap_size : nodepool_obj.swap_size,\n zram_size : nodepool_obj.zram_size,\n index : node_index\n selinux : nodepool_obj.selinux\n placement_group_compat_idx : nodepool_obj.placement_group_compat_idx,\n placement_group : nodepool_obj.placement_group,\n disable_ipv4 : nodepool_obj.disable_ipv4 || local.use_nat_router,\n disable_ipv6 : nodepool_obj.disable_ipv6 || local.use_nat_router,\n network_id : nodepool_obj.network_id,\n }\n }\n ]...)\n\n agent_nodes_from_maps_for_counts = merge([\n for pool_index, nodepool_obj in var.agent_nodepools : {\n # coalesce(nodepool_obj.nodes, {}) means we select those nodepools who's size is set by an integer count.\n for node_key, node_obj in coalesce(nodepool_obj.nodes, {}) :\n format(\"%s-%s-%s\", pool_index, node_key, nodepool_obj.name) => merge(\n {\n nodepool_name : nodepool_obj.name,\n server_type : nodepool_obj.server_type,\n longhorn_volume_size : coalesce(nodepool_obj.longhorn_volume_size, 0),\n longhorn_mount_path : nodepool_obj.longhorn_mount_path,\n floating_ip : lookup(nodepool_obj, \"floating_ip\", false),\n floating_ip_rdns : lookup(nodepool_obj, \"floating_ip_rdns\", false),\n location : nodepool_obj.location,\n labels : concat(local.default_agent_labels, nodepool_obj.swap_size != \"\" || nodepool_obj.zram_size != \"\" ? local.swap_node_label : [], nodepool_obj.labels),\n taints : compact(concat(local.default_agent_taints, nodepool_obj.taints)),\n kubelet_args : nodepool_obj.kubelet_args,\n backups : lookup(nodepool_obj, \"backups\", false),\n swap_size : nodepool_obj.swap_size,\n zram_size : nodepool_obj.zram_size,\n selinux : nodepool_obj.selinux,\n placement_group_compat_idx : nodepool_obj.placement_group_compat_idx,\n placement_group : nodepool_obj.placement_group,\n index : floor(tonumber(node_key)),\n disable_ipv4 : nodepool_obj.disable_ipv4 || local.use_nat_router,\n disable_ipv6 : nodepool_obj.disable_ipv6 || local.use_nat_router,\n network_id : nodepool_obj.network_id,\n },\n { for key, value in node_obj : key => value if value != null },\n {\n labels : concat(local.default_agent_labels, nodepool_obj.swap_size != \"\" || nodepool_obj.zram_size != \"\" ? local.swap_node_label : [], nodepool_obj.labels, coalesce(node_obj.labels, [])),\n taints : compact(concat(local.default_agent_taints, nodepool_obj.taints, coalesce(node_obj.taints, []))),\n },\n (\n node_obj.append_index_to_node_name ? { node_name_suffix : \"-${floor(tonumber(node_key))}\" } : {}\n )\n )\n }\n ]...)\n\n\n agent_nodes = merge(\n local.agent_nodes_from_integer_counts,\n local.agent_nodes_from_maps_for_counts,\n )\n\n use_existing_network = length(var.existing_network_id) > 0\n\n use_nat_router = var.nat_router != null\n\n ssh_bastion = local.use_nat_router ? {\n bastion_host = hcloud_server.nat_router[0].ipv4_address\n bastion_port = var.ssh_port\n bastion_user = \"nat-router\"\n bastion_private_key = var.ssh_private_key\n } : {\n bastion_host = null\n bastion_port = null\n bastion_user = null\n bastion_private_key = null\n }\n\n # Create subnets from the base network CIDR.\n # Control planes allocate from the end of the range and agents from the start (0, 1, 2...)\n network_ipv4_subnets = [for index in range(var.subnet_amount) : cidrsubnet(var.network_ipv4_cidr, log(var.subnet_amount, 2), index)]\n\n # By convention the DNS service (usually core-dns) is assigned the 10th IP address in the service CIDR block\n cluster_dns_ipv4 = var.cluster_dns_ipv4 != null ? var.cluster_dns_ipv4 : cidrhost(var.service_ipv4_cidr, 10)\n\n # The gateway's IP address is always the first IP address of the subnet's IP range\n network_gw_ipv4 = cidrhost(var.network_ipv4_cidr, 1)\n\n # if we are in a single cluster config, we use the default klipper lb instead of Hetzner LB\n control_plane_count = sum([for v in var.control_plane_nodepools : v.count])\n agent_count = length(var.agent_nodepools) > 0 ? sum([for v in var.agent_nodepools : length(coalesce(v.nodes, {})) + coalesce(v.count, 0)]) : 0\n autoscaler_max_count = length(var.autoscaler_nodepools) > 0 ? sum([for v in var.autoscaler_nodepools : v.max_nodes]) : 0\n is_single_node_cluster = (local.control_plane_count + local.agent_count + local.autoscaler_max_count) == 1\n\n using_klipper_lb = var.enable_klipper_metal_lb || local.is_single_node_cluster\n\n has_external_load_balancer = local.using_klipper_lb || var.ingress_controller == \"none\"\n load_balancer_name = \"${var.cluster_name}-${var.ingress_controller}\"\n\n ingress_controller_service_names = {\n \"traefik\" = \"traefik\"\n \"nginx\" = \"nginx-ingress-nginx-controller\"\n \"haproxy\" = \"haproxy-kubernetes-ingress\"\n }\n\n ingress_controller_install_resources = {\n \"traefik\" = [\"traefik_ingress.yaml\"]\n \"nginx\" = [\"nginx_ingress.yaml\"]\n \"haproxy\" = [\"haproxy_ingress.yaml\"]\n }\n\n default_ingress_namespace_mapping = {\n \"traefik\" = \"traefik\"\n \"nginx\" = \"nginx\"\n \"haproxy\" = \"haproxy\"\n }\n\n ingress_controller_namespace = var.ingress_target_namespace != \"\" ? var.ingress_target_namespace : lookup(local.default_ingress_namespace_mapping, var.ingress_controller, \"\")\n ingress_replica_count = (var.ingress_replica_count > 0) ? var.ingress_replica_count : (local.agent_count > 2) ? 3 : (local.agent_count == 2) ? 2 : 1\n ingress_max_replica_count = (var.ingress_max_replica_count > local.ingress_replica_count) ? var.ingress_max_replica_count : local.ingress_replica_count\n\n # disable k3s extras\n disable_extras = concat(var.enable_local_storage ? [] : [\"local-storage\"], local.using_klipper_lb ? [] : [\"servicelb\"], [\"traefik\"], var.enable_metrics_server ? [] : [\"metrics-server\"])\n\n # Determine if scheduling should be allowed on control plane nodes, which will be always true for single node clusters and clusters or if scheduling is allowed on control plane nodes\n allow_scheduling_on_control_plane = local.is_single_node_cluster ? true : var.allow_scheduling_on_control_plane\n # Determine if loadbalancer target should be allowed on control plane nodes, which will be always true for single node clusters or if scheduling is allowed on control plane nodes\n allow_loadbalancer_target_on_control_plane = local.is_single_node_cluster ? true : var.allow_scheduling_on_control_plane\n\n # Build list of label maps to include in LB target selector based on allow_loadbalancer_target_on_control_plane\n lb_target_groups = (\n local.allow_loadbalancer_target_on_control_plane ?\n [local.labels_control_plane_node, local.labels_agent_node] :\n [local.labels_agent_node]\n )\n\n # Default k3s node labels\n default_agent_labels = concat(\n var.exclude_agents_from_external_load_balancers ? [\"node.kubernetes.io/exclude-from-external-load-balancers=true\"] : [],\n var.automatically_upgrade_k3s ? [\"k3s_upgrade=true\"] : []\n )\n default_control_plane_labels = concat(local.allow_loadbalancer_target_on_control_plane ? [] : [\"node.kubernetes.io/exclude-from-external-load-balancers=true\"], var.automatically_upgrade_k3s ? [\"k3s_upgrade=true\"] : [])\n\n # Default k3s node taints\n default_control_plane_taints = concat([], local.allow_scheduling_on_control_plane ? [] : [\"node-role.kubernetes.io/control-plane:NoSchedule\"])\n default_agent_taints = concat([], var.cni_plugin == \"cilium\" ? [\"node.cilium.io/agent-not-ready:NoExecute\"] : [])\n\n base_firewall_rules = concat(\n var.firewall_ssh_source == null ? [] : [\n # Allow all traffic to the ssh port\n {\n description = \"Allow Incoming SSH Traffic\"\n direction = \"in\"\n protocol = \"tcp\"\n port = var.ssh_port\n source_ips = var.firewall_ssh_source\n },\n ],\n var.firewall_kube_api_source == null ? [] : [\n {\n description = \"Allow Incoming Requests to Kube API Server\"\n direction = \"in\"\n protocol = \"tcp\"\n port = \"6443\"\n source_ips = var.firewall_kube_api_source\n }\n ],\n !var.restrict_outbound_traffic ? [] : [\n # Allow basic out traffic\n # ICMP to ping outside services\n {\n description = \"Allow Outbound ICMP Ping Requests\"\n direction = \"out\"\n protocol = \"icmp\"\n port = \"\"\n destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n },\n\n # DNS\n {\n description = \"Allow Outbound TCP DNS Requests\"\n direction = \"out\"\n protocol = \"tcp\"\n port = \"53\"\n destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n },\n {\n description = \"Allow Outbound UDP DNS Requests\"\n direction = \"out\"\n protocol = \"udp\"\n port = \"53\"\n destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n },\n\n # HTTP(s)\n {\n description = \"Allow Outbound HTTP Requests\"\n direction = \"out\"\n protocol = \"tcp\"\n port = \"80\"\n destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n },\n {\n description = \"Allow Outbound HTTPS Requests\"\n direction = \"out\"\n protocol = \"tcp\"\n port = \"443\"\n destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n },\n\n #NTP\n {\n description = \"Allow Outbound UDP NTP Requests\"\n direction = \"out\"\n protocol = \"udp\"\n port = \"123\"\n destination_ips = [\"0.0.0.0/0\", \"::/0\"]\n }\n ],\n !local.using_klipper_lb ? [] : [\n # Allow incoming web traffic for single node clusters, because we are using k3s servicelb there,\n # not an external load-balancer.\n {\n description = \"Allow Incoming HTTP Connections\"\n direction = \"in\"\n protocol = \"tcp\"\n port = \"80\"\n source_ips = [\"0.0.0.0/0\", \"::/0\"]\n },\n {\n description = \"Allow Incoming HTTPS Connections\"\n direction = \"in\"\n protocol = \"tcp\"\n port = \"443\"\n source_ips = [\"0.0.0.0/0\", \"::/0\"]\n }\n ],\n var.block_icmp_ping_in ? [] : [\n {\n description = \"Allow Incoming ICMP Ping Requests\"\n direction = \"in\"\n protocol = \"icmp\"\n port = \"\"\n source_ips = [\"0.0.0.0/0\", \"::/0\"]\n }\n ]\n )\n\n # create a new firewall list based on base_firewall_rules but with direction-protocol-port as key\n # this is needed to avoid duplicate rules\n firewall_rules = { for rule in local.base_firewall_rules : format(\"%s-%s-%s\", lookup(rule, \"direction\", \"null\"), lookup(rule, \"protocol\", \"null\"), lookup(rule, \"port\", \"null\")) => rule }\n\n # do the same for var.extra_firewall_rules\n extra_firewall_rules = { for rule in var.extra_firewall_rules : format(\"%s-%s-%s\", lookup(rule, \"direction\", \"null\"), lookup(rule, \"protocol\", \"null\"), lookup(rule, \"port\", \"null\")) => rule }\n\n # merge the two lists\n firewall_rules_merged = merge(local.firewall_rules, local.extra_firewall_rules)\n\n # convert the merged list back to a list\n firewall_rules_list = values(local.firewall_rules_merged)\n\n labels = {\n \"provisioner\" = \"terraform\",\n \"engine\" = \"k3s\"\n \"cluster\" = var.cluster_name\n }\n\n labels_control_plane_node = {\n role = \"control_plane_node\"\n }\n labels_control_plane_lb = {\n role = \"control_plane_lb\"\n }\n\n labels_agent_node = {\n role = \"agent_node\"\n }\n\n cni_install_resources = {\n \"calico\" = [\"https://raw.githubusercontent.com/projectcalico/calico/${coalesce(local.calico_version, \"v3.27.2\")}/manifests/calico.yaml\"]\n \"cilium\" = [\"cilium.yaml\"]\n }\n\n prefer_bundled_bin_config = var.k3s_prefer_bundled_bin ? { \"prefer-bundled-bin\" = true } : {}\n\n cni_install_resource_patches = {\n \"calico\" = [\"calico.yaml\"]\n }\n\n cni_k3s_settings = {\n \"flannel\" = {\n disable-network-policy = var.disable_network_policy\n flannel-backend = var.flannel_backend != null ? var.flannel_backend : (var.enable_wireguard ? \"wireguard-native\" : \"vxlan\")\n }\n \"calico\" = {\n disable-network-policy = true\n flannel-backend = \"none\"\n }\n \"cilium\" = {\n disable-network-policy = true\n flannel-backend = \"none\"\n }\n }\n\n etcd_s3_snapshots = length(keys(var.etcd_s3_backup)) > 0 ? merge(\n {\n \"etcd-s3\" = true\n },\n var.etcd_s3_backup) : {}\n\n kubelet_arg = concat([\"cloud-provider=external\", \"volume-plugin-dir=/var/lib/kubelet/volumeplugins\"], var.k3s_kubelet_config != \"\" ? [\"config=/etc/rancher/k3s/kubelet-config.yaml\"] : [])\n kube_controller_manager_arg = \"flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins\"\n flannel_iface = \"eth1\"\n\n kube_apiserver_arg = concat(\n var.authentication_config != \"\" ? [\"authentication-config=/etc/rancher/k3s/authentication_config.yaml\"] : [],\n var.k3s_audit_policy_config != \"\" ? [\n \"audit-policy-file=/etc/rancher/k3s/audit-policy.yaml\",\n \"audit-log-path=${var.k3s_audit_log_path}\",\n \"audit-log-maxage=${var.k3s_audit_log_maxage}\",\n \"audit-log-maxbackup=${var.k3s_audit_log_maxbackup}\",\n \"audit-log-maxsize=${var.k3s_audit_log_maxsize}\"\n ] : []\n )\n\n cilium_values_default = <<EOT\n# Enable Kubernetes host-scope IPAM mode (required for K3s + Hetzner CCM)\nipam:\n mode: kubernetes\nk8s:\n requireIPv4PodCIDR: true\n\n# Replace kube-proxy with Cilium\nkubeProxyReplacement: true\n\n%{if var.disable_kube_proxy}\n# Enable health check server (healthz) for the kube-proxy replacement\nkubeProxyReplacementHealthzBindAddr: \"0.0.0.0:10256\"\n%{endif~}\n\n# Access to Kube API Server (mandatory if kube-proxy is disabled)\nk8sServiceHost: \"127.0.0.1\"\nk8sServicePort: \"6444\"\n\n# Set Tunnel Mode or Native Routing Mode (supported by Hetzner CCM Route Controller)\nroutingMode: \"${var.cilium_routing_mode}\"\n%{if var.cilium_routing_mode == \"native\"~}\n# Set the native routable CIDR\nipv4NativeRoutingCIDR: \"${local.cilium_ipv4_native_routing_cidr}\"\n\n# Bypass iptables Connection Tracking for Pod traffic (only works in Native Routing Mode)\ninstallNoConntrackIptablesRules: true\n%{endif~}\n\n# Perform a gradual roll out on config update.\nrollOutCiliumPods: true\n\nendpointRoutes:\n # Enable use of per endpoint routes instead of routing via the cilium_host interface.\n enabled: true\n\nloadBalancer:\n # Enable LoadBalancer & NodePort XDP Acceleration (direct routing (routingMode=native) is recommended to achieve optimal performance)\n acceleration: \"${var.cilium_loadbalancer_acceleration_mode}\"\n\nbpf:\n # Enable eBPF-based Masquerading (\"The eBPF-based implementation is the most efficient implementation\")\n masquerade: true\n%{if var.enable_wireguard}\nencryption:\n enabled: true\n # Enable node encryption for node-to-node traffic\n nodeEncryption: true\n type: wireguard\n%{endif~}\n%{if var.cilium_egress_gateway_enabled}\negressGateway:\n enabled: true\n%{endif~}\n\n%{if var.cilium_hubble_enabled}\nhubble:\n relay:\n enabled: true\n ui:\n enabled: true\n metrics:\n enabled:\n%{for metric in var.cilium_hubble_metrics_enabled~}\n - \"${metric}\"\n%{endfor~}\n%{endif~}\n\n\nMTU: %{if local.use_robot_ccm} 1350 %{else} 1450 %{endif}\n EOT\n\n cilium_values = module.values_merger_cilium.values\n\n # Not to be confused with the other helm values, this is used for the calico.yaml kustomize patch\n # It also serves as a stub for a potential future use via helm values\n calico_values = var.calico_values != \"\" ? var.calico_values : <<EOT\nkind: DaemonSet\napiVersion: apps/v1\nmetadata:\n name: calico-node\n namespace: kube-system\n labels:\n k8s-app: calico-node\nspec:\n template:\n spec:\n volumes:\n - name: flexvol-driver-host\n hostPath:\n type: DirectoryOrCreate\n path: /var/lib/kubelet/volumeplugins/nodeagent~uds\n containers:\n - name: calico-node\n env:\n - name: CALICO_IPV4POOL_CIDR\n value: \"${var.cluster_ipv4_cidr}\"\n - name: FELIX_WIREGUARDENABLED\n value: \"${var.enable_wireguard}\"\n\n EOT\n\n longhorn_values_default = <<EOT\ndefaultSettings:\n%{if length(var.autoscaler_nodepools) != 0~}\n kubernetesClusterAutoscalerEnabled: true\n%{endif~}\n defaultDataPath: /var/longhorn\npersistence:\n defaultFsType: ${var.longhorn_fstype}\n defaultClassReplicaCount: ${var.longhorn_replica_count}\n %{if var.disable_hetzner_csi~}defaultClass: true%{else~}defaultClass: false%{endif~}\n EOT\n\n longhorn_values = module.values_merger_longhorn.values\n\n csi_driver_smb_values = var.csi_driver_smb_values != \"\" ? var.csi_driver_smb_values : <<EOT\n EOT\n\n hetzner_csi_values = var.hetzner_csi_values != \"\" ? var.hetzner_csi_values : <<-EOT\nnode:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n%{if !local.allow_scheduling_on_control_plane~}\n - key: \"node-role.kubernetes.io/control-plane\"\n operator: DoesNotExist\n%{endif~}\n - key: \"instance.hetzner.cloud/provided-by\"\n operator: NotIn\n values:\n - robot\nEOT\n\n nginx_values_default = <<EOT\ncontroller:\n watchIngressWithoutClass: \"true\"\n kind: \"Deployment\"\n replicaCount: ${local.ingress_replica_count}\n config:\n \"use-forwarded-headers\": \"true\"\n \"compute-full-forwarded-for\": \"true\"\n \"use-proxy-protocol\": \"${!local.using_klipper_lb}\"\n%{if !local.using_klipper_lb~}\n service:\n annotations:\n \"load-balancer.hetzner.cloud/name\": \"${local.load_balancer_name}\"\n \"load-balancer.hetzner.cloud/use-private-ip\": \"true\"\n \"load-balancer.hetzner.cloud/disable-private-ingress\": \"true\"\n \"load-balancer.hetzner.cloud/disable-public-network\": \"${var.load_balancer_disable_public_network}\"\n \"load-balancer.hetzner.cloud/ipv6-disabled\": \"${var.load_balancer_disable_ipv6}\"\n \"load-balancer.hetzner.cloud/location\": \"${var.load_balancer_location}\"\n \"load-balancer.hetzner.cloud/type\": \"${var.load_balancer_type}\"\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"${!local.using_klipper_lb}\"\n \"load-balancer.hetzner.cloud/algorithm-type\": \"${var.load_balancer_algorithm_type}\"\n \"load-balancer.hetzner.cloud/health-check-interval\": \"${var.load_balancer_health_check_interval}\"\n \"load-balancer.hetzner.cloud/health-check-timeout\": \"${var.load_balancer_health_check_timeout}\"\n \"load-balancer.hetzner.cloud/health-check-retries\": \"${var.load_balancer_health_check_retries}\"\n%{if var.lb_hostname != \"\"~}\n \"load-balancer.hetzner.cloud/hostname\": \"${var.lb_hostname}\"\n%{endif~}\n%{endif~}\n EOT\n\n nginx_values = module.values_merger_nginx.values\n\n hetzner_ccm_values_default = <<EOT\nnetworking:\n enabled: true\n clusterCIDR: \"${var.cluster_ipv4_cidr}\"\n%{if local.use_robot_ccm~}\nrobot:\n enabled: true\n%{endif~}\n\nargs:\n cloud-provider: hcloud\n allow-untagged-cloud: \"\"\n route-reconciliation-period: 30s\n webhook-secure-port: \"0\"\n%{if local.using_klipper_lb~}\n secure-port: \"10288\"\n%{endif~}\nenv:\n HCLOUD_LOAD_BALANCERS_LOCATION:\n value: \"${var.load_balancer_location}\"\n HCLOUD_LOAD_BALANCERS_USE_PRIVATE_IP:\n value: \"true\"\n HCLOUD_LOAD_BALANCERS_ENABLED:\n value: \"${!local.using_klipper_lb}\"\n HCLOUD_LOAD_BALANCERS_DISABLE_PRIVATE_INGRESS:\n value: \"true\"\n%{if local.use_robot_ccm~}\n HCLOUD_NETWORK_ROUTES_ENABLED:\n value: \"false\"\n%{endif~}\n# Use host network to avoid circular dependency with CNI\nhostNetwork: true\n EOT\n\n hetzner_ccm_values = module.values_merger_hetzner_ccm.values\n\n haproxy_values_default = <<EOT\ncontroller:\n kind: \"Deployment\"\n replicaCount: ${local.ingress_replica_count}\n ingressClass: null\n resources:\n requests:\n cpu: \"${var.haproxy_requests_cpu}\"\n memory: \"${var.haproxy_requests_memory}\"\n config:\n ssl-redirect: \"false\"\n forwarded-for: \"true\"\n%{if !local.using_klipper_lb~}\n proxy-protocol: \"${join(\n \", \",\n concat(\n [\"127.0.0.1/32\", \"10.0.0.0/8\"],\n var.haproxy_additional_proxy_protocol_ips\n )\n)}\"\n%{endif~}\n service:\n type: LoadBalancer\n enablePorts:\n quic: false\n stat: false\n prometheus: false\n%{if !local.using_klipper_lb~}\n annotations:\n \"load-balancer.hetzner.cloud/name\": \"${local.load_balancer_name}\"\n \"load-balancer.hetzner.cloud/use-private-ip\": \"true\"\n \"load-balancer.hetzner.cloud/disable-private-ingress\": \"true\"\n \"load-balancer.hetzner.cloud/disable-public-network\": \"${var.load_balancer_disable_public_network}\"\n \"load-balancer.hetzner.cloud/ipv6-disabled\": \"${var.load_balancer_disable_ipv6}\"\n \"load-balancer.hetzner.cloud/location\": \"${var.load_balancer_location}\"\n \"load-balancer.hetzner.cloud/type\": \"${var.load_balancer_type}\"\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"${!local.using_klipper_lb}\"\n \"load-balancer.hetzner.cloud/algorithm-type\": \"${var.load_balancer_algorithm_type}\"\n \"load-balancer.hetzner.cloud/health-check-interval\": \"${var.load_balancer_health_check_interval}\"\n \"load-balancer.hetzner.cloud/health-check-timeout\": \"${var.load_balancer_health_check_timeout}\"\n \"load-balancer.hetzner.cloud/health-check-retries\": \"${var.load_balancer_health_check_retries}\"\n%{if var.lb_hostname != \"\"~}\n \"load-balancer.hetzner.cloud/hostname\": \"${var.lb_hostname}\"\n%{endif~}\n%{endif~}\n EOT\n\nhaproxy_values = module.values_merger_haproxy.values\n\ntraefik_values_default = <<EOT\nimage:\n tag: ${var.traefik_image_tag}\ndeployment:\n replicas: ${local.ingress_replica_count}\nservice:\n enabled: true\n type: LoadBalancer\n%{if !local.using_klipper_lb~}\n annotations:\n \"load-balancer.hetzner.cloud/name\": \"${local.load_balancer_name}\"\n \"load-balancer.hetzner.cloud/use-private-ip\": \"true\"\n \"load-balancer.hetzner.cloud/disable-private-ingress\": \"true\"\n \"load-balancer.hetzner.cloud/disable-public-network\": \"${var.load_balancer_disable_public_network}\"\n \"load-balancer.hetzner.cloud/ipv6-disabled\": \"${var.load_balancer_disable_ipv6}\"\n \"load-balancer.hetzner.cloud/location\": \"${var.load_balancer_location}\"\n \"load-balancer.hetzner.cloud/type\": \"${var.load_balancer_type}\"\n \"load-balancer.hetzner.cloud/uses-proxyprotocol\": \"${!local.using_klipper_lb}\"\n \"load-balancer.hetzner.cloud/algorithm-type\": \"${var.load_balancer_algorithm_type}\"\n \"load-balancer.hetzner.cloud/health-check-interval\": \"${var.load_balancer_health_check_interval}\"\n \"load-balancer.hetzner.cloud/health-check-timeout\": \"${var.load_balancer_health_check_timeout}\"\n \"load-balancer.hetzner.cloud/health-check-retries\": \"${var.load_balancer_health_check_retries}\"\n%{if var.lb_hostname != \"\"~}\n \"load-balancer.hetzner.cloud/hostname\": \"${var.lb_hostname}\"\n%{endif~}\n%{endif~}\nports:\n%{if var.traefik_redirect_to_https || !local.using_klipper_lb~}\n web:\n%{if var.traefik_redirect_to_https~}\n http:\n redirections:\n entryPoint:\n to: websecure\n scheme: https\n permanent: true\n%{endif~}\n%{if !local.using_klipper_lb~}\n proxyProtocol:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n%{for ip in var.traefik_additional_trusted_ips~}\n - \"${ip}\"\n%{endfor~}\n forwardedHeaders:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n%{for ip in var.traefik_additional_trusted_ips~}\n - \"${ip}\"\n%{endfor~}\n%{endif~}\n%{endif~}\n%{if !local.using_klipper_lb~}\n websecure:\n proxyProtocol:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n%{for ip in var.traefik_additional_trusted_ips~}\n - \"${ip}\"\n%{endfor~}\n forwardedHeaders:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n%{for ip in var.traefik_additional_trusted_ips~}\n - \"${ip}\"\n%{endfor~}\n%{endif~}\n%{if var.traefik_additional_ports != \"\"~}\n%{for option in var.traefik_additional_ports~}\n ${option.name}:\n port: ${option.port}\n expose:\n default: true\n exposedPort: ${option.exposedPort}\n protocol: TCP\n observability:\n metrics: false\n accessLogs: false\n tracing: false\n%{if !local.using_klipper_lb~}\n proxyProtocol:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n%{for ip in var.traefik_additional_trusted_ips~}\n - \"${ip}\"\n%{endfor~}\n forwardedHeaders:\n trustedIPs:\n - 127.0.0.1/32\n - 10.0.0.0/8\n%{for ip in var.traefik_additional_trusted_ips~}\n - \"${ip}\"\n%{endfor~}\n%{endif~}\n%{endfor~}\n%{endif~}\n%{if var.traefik_pod_disruption_budget~}\npodDisruptionBudget:\n enabled: true\n maxUnavailable: 33%\n%{endif~}\n%{if var.traefik_provider_kubernetes_gateway_enabled~}\nproviders:\n kubernetesGateway:\n enabled: true\n%{endif~}\nadditionalArguments:\n - \"--providers.kubernetesingress.ingressendpoint.publishedservice=${local.ingress_controller_namespace}/traefik\"\n%{for option in var.traefik_additional_options~}\n - \"${option}\"\n%{endfor~}\n%{if var.traefik_resource_limits~}\nresources:\n requests:\n cpu: \"${var.traefik_resource_values.requests.cpu}\"\n memory: \"${var.traefik_resource_values.requests.memory}\"\n limits:\n cpu: \"${var.traefik_resource_values.limits.cpu}\"\n memory: \"${var.traefik_resource_values.limits.memory}\"\n%{endif~}\n%{if var.traefik_autoscaling~}\nautoscaling:\n enabled: true\n minReplicas: ${local.ingress_replica_count}\n maxReplicas: ${local.ingress_max_replica_count}\n%{endif~}\nEOT\n\ntraefik_values = module.values_merger_traefik.values\n\nrancher_values_default = <<EOT\nhostname: \"${var.rancher_hostname != \"\" ? var.rancher_hostname : var.lb_hostname}\"\nreplicas: ${length(local.control_plane_nodes)}\nbootstrapPassword: \"${length(var.rancher_bootstrap_password) == 0 ? resource.random_password.rancher_bootstrap[0].result : var.rancher_bootstrap_password}\"\nglobal:\n cattle:\n psp:\n enabled: false\n EOT\n\nrancher_values = module.values_merger_rancher.values\n\ncert_manager_values_default = <<EOT\ncrds:\n enabled: true\n keep: true\n%{if var.traefik_provider_kubernetes_gateway_enabled~}\nconfig:\n apiVersion: controller.config.cert-manager.io/v1alpha1\n kind: ControllerConfiguration\n enableGatewayAPI: true\n%{endif~}\n%{if var.ingress_controller == \"nginx\"~}\nextraArgs:\n - --feature-gates=ACMEHTTP01IngressPathTypeExact=false\n%{endif~}\n EOT\n\ncert_manager_values = module.values_merger_cert_manager.values\n\nkured_options = merge({\n \"reboot-command\" : \"/usr/bin/systemctl reboot\",\n \"pre-reboot-node-labels\" : \"kured=rebooting\",\n \"post-reboot-node-labels\" : \"kured=done\",\n \"period\" : \"5m\",\n \"reboot-sentinel\" : \"/sentinel/reboot-required\"\n}, var.kured_options)\n\nk3s_registries_update_script = <<EOF\nDATE=`date +%Y-%m-%d_%H-%M-%S`\nif cmp -s /tmp/registries.yaml /etc/rancher/k3s/registries.yaml; then\n echo \"No update required to the registries.yaml file\"\nelse\n echo \"Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_$DATE.yaml\"\n cp /etc/rancher/k3s/registries.yaml /tmp/registries_$DATE.yaml\n echo \"Updated registries.yaml detected, restart of k3s service required\"\n cp /tmp/registries.yaml /etc/rancher/k3s/registries.yaml\n if systemctl is-active --quiet k3s; then\n systemctl restart k3s || (echo \"Error: Failed to restart k3s. Restoring /etc/rancher/k3s/registries.yaml from backup\" && cp /tmp/registries_$DATE.yaml /etc/rancher/k3s/registries.yaml && systemctl restart k3s)\n elif systemctl is-active --quiet k3s-agent; then\n systemctl restart k3s-agent || (echo \"Error: Failed to restart k3s-agent. Restoring /etc/rancher/k3s/registries.yaml from backup\" && cp /tmp/registries_$DATE.yaml /etc/rancher/k3s/registries.yaml && systemctl restart k3s-agent)\n else\n echo \"No active k3s or k3s-agent service found\"\n fi\n echo \"k3s service or k3s-agent service restarted successfully\"\nfi\nEOF\n\nk3s_kubelet_config_update_script = <<EOF\nset -e\nDATE=`date +%Y-%m-%d_%H-%M-%S`\nBACKUP_FILE=\"/tmp/kubelet-config_$DATE.yaml\"\nHAS_BACKUP=false\n\nif cmp -s /tmp/kubelet-config.yaml /etc/rancher/k3s/kubelet-config.yaml; then\n echo \"No update required to the kubelet-config.yaml file\"\nelse\n if [ -f \"/etc/rancher/k3s/kubelet-config.yaml\" ]; then\n echo \"Backing up /etc/rancher/k3s/kubelet-config.yaml to $BACKUP_FILE\"\n cp /etc/rancher/k3s/kubelet-config.yaml \"$BACKUP_FILE\"\n HAS_BACKUP=true\n fi\n echo \"Updated kubelet-config.yaml detected, restart of k3s service required\"\n cp /tmp/kubelet-config.yaml /etc/rancher/k3s/kubelet-config.yaml\n\n restart_failed() {\n local SERVICE_NAME=\"$1\"\n echo \"Error: Failed to restart $SERVICE_NAME\"\n if [ \"$HAS_BACKUP\" = true ]; then\n echo \"Restoring from backup $BACKUP_FILE\"\n cp \"$BACKUP_FILE\" /etc/rancher/k3s/kubelet-config.yaml\n echo \"Attempting to restart $SERVICE_NAME with restored config...\"\n systemctl restart \"$SERVICE_NAME\" || echo \"Warning: Restart after restore also failed\"\n else\n echo \"No backup available to restore (first-time config)\"\n rm -f /etc/rancher/k3s/kubelet-config.yaml\n echo \"Attempting to restart $SERVICE_NAME without kubelet config...\"\n systemctl restart \"$SERVICE_NAME\" || echo \"Warning: Restart without config also failed\"\n fi\n exit 1\n }\n\n if systemctl is-active --quiet k3s; then\n systemctl restart k3s || restart_failed k3s\n elif systemctl is-active --quiet k3s-agent; then\n systemctl restart k3s-agent || restart_failed k3s-agent\n else\n echo \"Warning: No active k3s or k3s-agent service found, skipping restart\"\n fi\n echo \"k3s service or k3s-agent service (re)started successfully\"\nfi\nEOF\n\nk3s_config_update_script = <<EOF\nDATE=`date +%Y-%m-%d_%H-%M-%S`\nif cmp -s /tmp/config.yaml /etc/rancher/k3s/config.yaml; then\n echo \"No update required to the config.yaml file\"\nelse\n if [ -f \"/etc/rancher/k3s/config.yaml\" ]; then\n echo \"Backing up /etc/rancher/k3s/config.yaml to /tmp/config_$DATE.yaml\"\n cp /etc/rancher/k3s/config.yaml /tmp/config_$DATE.yaml\n fi\n echo \"Updated config.yaml detected, restart of k3s service required\"\n cp /tmp/config.yaml /etc/rancher/k3s/config.yaml\n if systemctl is-active --quiet k3s; then\n systemctl restart k3s || (echo \"Error: Failed to restart k3s. Restoring /etc/rancher/k3s/config.yaml from backup\" && cp /tmp/config_$DATE.yaml /etc/rancher/k3s/config.yaml && systemctl restart k3s)\n elif systemctl is-active --quiet k3s-agent; then\n systemctl restart k3s-agent || (echo \"Error: Failed to restart k3s-agent. Restoring /etc/rancher/k3s/config.yaml from backup\" && cp /tmp/config_$DATE.yaml /etc/rancher/k3s/config.yaml && systemctl restart k3s-agent)\n else\n echo \"No active k3s or k3s-agent service found\"\n fi\n echo \"k3s service or k3s-agent service (re)started successfully\"\nfi\nEOF\n\nk3s_audit_policy_update_script = <<EOF\nDATE=`date +%Y-%m-%d_%H-%M-%S`\nif [ -z \"${var.k3s_audit_policy_config}\" ] || [ \"${var.k3s_audit_policy_config}\" = \" \" ]; then\n echo \"No audit policy config provided via Terraform, skipping audit policy setup\"\n # Note: We intentionally DO NOT remove existing audit policies here.\n # This preserves any manually-configured audit policies for backward compatibility.\n exit 0\nfi\n\n# Config is provided, proceed with audit policy setup\nif cmp -s /tmp/audit-policy.yaml /etc/rancher/k3s/audit-policy.yaml; then\n echo \"No update required to the audit-policy.yaml file\"\nelse\n if [ -f \"/etc/rancher/k3s/audit-policy.yaml\" ]; then\n echo \"Backing up /etc/rancher/k3s/audit-policy.yaml to /tmp/audit-policy_$DATE.yaml\"\n cp /etc/rancher/k3s/audit-policy.yaml /tmp/audit-policy_$DATE.yaml\n fi\n echo \"Updated audit-policy.yaml detected, restart of k3s service required\"\n cp /tmp/audit-policy.yaml /etc/rancher/k3s/audit-policy.yaml\n if systemctl is-active --quiet k3s; then\n systemctl restart k3s || (echo \"Error: Failed to restart k3s. Restoring /etc/rancher/k3s/audit-policy.yaml from backup\" && cp /tmp/audit-policy_$DATE.yaml /etc/rancher/k3s/audit-policy.yaml && systemctl restart k3s)\n else\n echo \"k3s service is not active, skipping restart\"\n fi\n echo \"k3s service restarted successfully with new audit policy\"\nfi\n\n# Ensure audit log directory exists with proper permissions\nmkdir -p $(dirname ${var.k3s_audit_log_path})\nchmod 750 $(dirname ${var.k3s_audit_log_path})\nchown root:root $(dirname ${var.k3s_audit_log_path})\nEOF\n\nk3s_authentication_config_update_script = <<EOF\nDATE=`date +%Y-%m-%d_%H-%M-%S`\nif cmp -s /tmp/authentication_config.yaml /etc/rancher/k3s/authentication_config.yaml; then\n echo \"No update required to the authentication_config.yaml file\"\nelse\n if [ -f \"/etc/rancher/k3s/authentication_config.yaml\" ]; then\n echo \"Backing up /etc/rancher/k3s/authentication_config.yaml to /tmp/authentication_config_$DATE.yaml\"\n cp /etc/rancher/k3s/authentication_config.yaml /tmp/authentication_config_$DATE.yaml\n fi\n echo \"Updated authentication_config.yaml detected, restart of k3s service required\"\n cp /tmp/authentication_config.yaml /etc/rancher/k3s/authentication_config.yaml\n if systemctl is-active --quiet k3s; then\n systemctl restart k3s || (echo \"Error: Failed to restart k3s. Restoring /etc/rancher/k3s/authentication_config.yaml from backup\" && cp /tmp/authentication_config_$DATE.yaml /etc/rancher/k3s/authentication_config.yaml && systemctl restart k3s)\n elif systemctl is-active --quiet k3s-agent; then\n systemctl restart k3s-agent || (echo \"Error: Failed to restart k3s-agent. Restoring /etc/rancher/k3s/authentication_config.yaml from backup\" && cp /tmp/authentication_config_$DATE.yaml /etc/rancher/k3s/authentication_config.yaml && systemctl restart k3s-agent)\n else\n echo \"No active k3s or k3s-agent service found\"\n fi\n echo \"k3s service or k3s-agent service (re)started successfully\"\nfi\nEOF\n\ncloudinit_write_files_common = <<EOT\n# Script to rename the private interface to eth1 and unify NetworkManager connection naming\n- path: /etc/cloud/rename_interface.sh\n content: |\n #!/bin/bash\n set -euo pipefail\n sleep 8\n\n myinit() {\n # wait for a bit\n sleep 3\n\n # Somehow sometimes on private-ip only setups, the\n # interface may already be correctly named, and this\n # block should be skipped.\n if ! ip link show eth1 >/dev/null 2>&1; then\n # Find the private network interface by name, falling back to original logic.\n # The output of 'ip link show' is stored to avoid multiple calls.\n # Use '|| true' to prevent grep from causing script failure when no matches found\n IP_LINK_NO_FLANNEL=$(ip link show | grep -v 'flannel' || true)\n\n # Try to find an interface with a predictable name, e.g., enp1s0\n # Anchor pattern to second field to avoid false matches\n INTERFACE=$(awk '$2 ~ /^enp[0-9]+s[0-9]+:$/{sub(/:/,\"\",$2); print $2; exit}' <<< \"$IP_LINK_NO_FLANNEL\")\n\n # If no predictable name is found, use original logic as fallback\n if [ -z \"$INTERFACE\" ]; then\n INTERFACE=$(awk '/^3:/{p=$2} /^2:/{s=$2} END{iface=p?p:s; sub(/:/,\"\",iface); print iface}' <<< \"$IP_LINK_NO_FLANNEL\")\n fi\n\n # Ensure an interface was found\n if [ -z \"$INTERFACE\" ]; then\n echo \"ERROR: Failed to detect network interface for renaming to eth1\" >&2\n echo \"Available interfaces:\" >&2\n echo \"$IP_LINK_NO_FLANNEL\" >&2\n return 1\n fi\n\n MAC=$(cat \"/sys/class/net/$INTERFACE/address\") || return 1\n\n echo \"SUBSYSTEM==\\\"net\\\", ACTION==\\\"add\\\", DRIVERS==\\\"?*\\\", ATTR{address}==\\\"$MAC\\\", NAME=\\\"eth1\\\"\" > /etc/udev/rules.d/70-persistent-net.rules\n\n ip link set \"$INTERFACE\" down\n ip link set \"$INTERFACE\" name eth1\n ip link set eth1 up\n fi\n\n return 0\n }\n\n myrepeat () {\n # Current time + 300 seconds (5 minutes)\n local END_SECONDS=$((SECONDS + 300))\n while true; do\n >&2 echo \"loop\"\n if (( \"$SECONDS\" > \"$END_SECONDS\" )); then\n >&2 echo \"timeout reached\"\n exit 1\n fi\n # run command and check return code\n if $@ ; then\n >&2 echo \"break\"\n break\n else\n >&2 echo \"got failure exit code, repeating\"\n sleep 0.5\n fi\n done\n }\n\n myrename () {\n local eth=\"$1\"\n local eth_connection\n\n # In case of a private-only network, eth0 may not exist\n if ip link show \"$eth\" &>/dev/null; then\n eth_connection=$(nmcli -g GENERAL.CONNECTION device show \"$eth\" || echo '')\n nmcli connection modify \"$eth_connection\" \\\n con-name \"$eth\" \\\n connection.interface-name \"$eth\"\n fi\n }\n\n myrepeat myinit\n myrepeat myrename eth0\n myrepeat myrename eth1\n\n systemctl restart NetworkManager\n permissions: \"0744\"\n\n# Disable ssh password authentication\n- content: |\n Port ${var.ssh_port}\n PasswordAuthentication no\n X11Forwarding no\n MaxAuthTries ${var.ssh_max_auth_tries}\n AllowTcpForwarding no\n AllowAgentForwarding no\n AuthorizedKeysFile .ssh/authorized_keys\n path: /etc/ssh/sshd_config.d/kube-hetzner.conf\n\n# Set reboot method as \"kured\"\n- content: |\n REBOOT_METHOD=kured\n path: /etc/transactional-update.conf\n\n# Create Rancher repo config\n- content: |\n [rancher-k3s-common-stable]\n name=Rancher K3s Common (stable)\n baseurl=https://rpm.rancher.io/k3s/stable/common/microos/noarch\n enabled=1\n gpgcheck=1\n repo_gpgcheck=0\n gpgkey=https://rpm.rancher.io/public.key\n path: /etc/zypp/repos.d/rancher-k3s-common.repo\n\n# Create the kube_hetzner_selinux.te file, that allows in SELinux to not interfere with various needed services\n- path: /root/kube_hetzner_selinux.te\n encoding: base64\n content: ${base64encode(file(\"${path.module}/templates/kube-hetzner-selinux.te\"))}\n\n# Create the k3s registries file if needed\n%{if var.k3s_registries != \"\"}\n# Create k3s registries file\n- content: ${base64encode(var.k3s_registries)}\n encoding: base64\n path: /etc/rancher/k3s/registries.yaml\n%{endif}\n\n# Create the k3s kubelet config file if needed\n%{if var.k3s_kubelet_config != \"\"}\n# Create k3s kubelet config file\n- content: ${base64encode(var.k3s_kubelet_config)}\n encoding: base64\n path: /etc/rancher/k3s/kubelet-config.yaml\n%{endif}\nEOT\n\ncloudinit_runcmd_common = <<EOT\n# ensure that /var uses full available disk size, thanks to btrfs this is easy\n- [btrfs, 'filesystem', 'resize', 'max', '/var']\n\n# SELinux permission for the SSH alternative port\n%{if var.ssh_port != 22}\n# SELinux permission for the SSH alternative port.\n- [semanage, port, '-a', '-t', ssh_port_t, '-p', tcp, '${var.ssh_port}']\n%{endif}\n\n# Create and apply the necessary SELinux module for kube-hetzner\n- [checkmodule, '-M', '-m', '-o', '/root/kube_hetzner_selinux.mod', '/root/kube_hetzner_selinux.te']\n- ['semodule_package', '-o', '/root/kube_hetzner_selinux.pp', '-m', '/root/kube_hetzner_selinux.mod']\n- [semodule, '-i', '/root/kube_hetzner_selinux.pp']\n- [setsebool, '-P', 'virt_use_samba', '1']\n- [setsebool, '-P', 'domain_kernel_load_modules', '1']\n\n# Disable rebootmgr service as we use kured instead\n- [systemctl, disable, '--now', 'rebootmgr.service']\n\n# Bounds the amount of logs that can survive on the system\n- [sed, '-i', 's/#SystemMaxUse=/SystemMaxUse=3G/g', /etc/systemd/journald.conf]\n- [sed, '-i', 's/#MaxRetentionSec=/MaxRetentionSec=1week/g', /etc/systemd/journald.conf]\n\n# Reduces the default number of snapshots from 2-10 number limit, to 4 and from 4-10 number limit important, to 2\n- [sed, '-i', 's/NUMBER_LIMIT=\"2-10\"/NUMBER_LIMIT=\"4\"/g', /etc/snapper/configs/root]\n- [sed, '-i', 's/NUMBER_LIMIT_IMPORTANT=\"4-10\"/NUMBER_LIMIT_IMPORTANT=\"3\"/g', /etc/snapper/configs/root]\n\n# Allow network interface\n- [chmod, '+x', '/etc/cloud/rename_interface.sh']\n\n# Restart the sshd service to apply the new config\n- [systemctl, 'restart', 'sshd']\n\n# Make sure the network is up\n- [systemctl, restart, NetworkManager]\n- [systemctl, status, NetworkManager]\n\n# Cleanup some logs\n- [truncate, '-s', '0', '/var/log/audit/audit.log']\n\n# Create audit log directory for k3s\n- [mkdir, '-p', '${dirname(var.k3s_audit_log_path)}']\n- [chmod, '750', '${dirname(var.k3s_audit_log_path)}']\n- [chown, 'root:root', '${dirname(var.k3s_audit_log_path)}']\n\n# Add logic to truly disable SELinux if disable_selinux = true.\n# We'll do it by appending to cloudinit_runcmd_common.\n%{if var.disable_selinux}\n- [sed, '-i', '-E', 's/^SELINUX=[a-z]+/SELINUX=disabled/', '/etc/selinux/config']\n- [setenforce, '0']\n%{endif}\n\nEOT\n\n}\n\n# Cross-variable validations that can't be done in variable validation blocks\ncheck \"nat_router_requires_control_plane_lb\" {\n assert {\n condition = var.nat_router == null || var.use_control_plane_lb\n error_message = \"When nat_router is enabled, use_control_plane_lb must be set to true.\"\n }\n}\n\ncheck \"ccm_lb_has_eligible_targets\" {\n assert {\n condition = !(var.exclude_agents_from_external_load_balancers && !local.allow_loadbalancer_target_on_control_plane)\n error_message = \"Warning: exclude_agents_from_external_load_balancers=true with allow_scheduling_on_control_plane=false leaves NO eligible targets for CCM-managed LoadBalancer services. Either set allow_scheduling_on_control_plane=true or disable exclude_agents_from_external_load_balancers.\"\n }\n}\n\ncheck \"system_upgrade_window_requires_supported_controller_version\" {\n assert {\n condition = var.system_upgrade_schedule_window == null ? true : (\n try(provider::semvers::compare(trimprefix(var.sys_upgrade_controller_version, \"v\"), \"0.15.0\"), -1) >= 0\n )\n error_message = \"system_upgrade_schedule_window requires sys_upgrade_controller_version v0.15.0 or newer.\"\n }\n}\n" }, { "path": "main.tf", "content": "resource \"random_password\" \"k3s_token\" {\n length = 48\n special = false\n}\n\ndata \"hcloud_image\" \"microos_x86_snapshot\" {\n with_selector = \"microos-snapshot=yes\"\n with_architecture = \"x86\"\n most_recent = true\n}\n\ndata \"hcloud_image\" \"microos_arm_snapshot\" {\n with_selector = \"microos-snapshot=yes\"\n with_architecture = \"arm\"\n most_recent = true\n}\n\nresource \"hcloud_ssh_key\" \"k3s\" {\n count = var.hcloud_ssh_key_id == null ? 1 : 0\n name = var.cluster_name\n public_key = var.ssh_public_key\n labels = local.labels\n}\n\nresource \"hcloud_network\" \"k3s\" {\n count = local.use_existing_network ? 0 : 1\n name = var.cluster_name\n ip_range = var.network_ipv4_cidr\n labels = local.labels\n expose_routes_to_vswitch = var.vswitch_id != null\n}\n\ndata \"hcloud_network\" \"k3s\" {\n id = local.use_existing_network ? var.existing_network_id[0] : hcloud_network.k3s[0].id\n}\n\n\n# We start from the end of the subnets cidr array,\n# as we would have fewer control plane nodepools, than agent ones.\nresource \"hcloud_network_subnet\" \"control_plane\" {\n count = length(var.control_plane_nodepools)\n network_id = data.hcloud_network.k3s.id\n type = \"cloud\"\n network_zone = var.network_region\n ip_range = local.network_ipv4_subnets[var.subnet_amount - 1 - count.index]\n}\n\n# Here we start at the beginning of the subnets cidr array\nresource \"hcloud_network_subnet\" \"agent\" {\n count = length(var.agent_nodepools)\n network_id = data.hcloud_network.k3s.id\n type = \"cloud\"\n network_zone = var.network_region\n ip_range = coalesce(var.agent_nodepools[count.index].subnet_ip_range, local.network_ipv4_subnets[count.index])\n}\n\n# Subnet for NAT router and other peripherals\nresource \"hcloud_network_subnet\" \"nat_router\" {\n count = var.nat_router != null ? 1 : 0\n network_id = data.hcloud_network.k3s.id\n type = \"cloud\"\n network_zone = var.network_region\n ip_range = local.network_ipv4_subnets[var.nat_router_subnet_index]\n}\n\n# Subnet for vSwitch\nresource \"hcloud_network_subnet\" \"vswitch_subnet\" {\n count = var.vswitch_id != null ? 1 : 0\n network_id = data.hcloud_network.k3s.id\n type = \"vswitch\"\n network_zone = var.network_region\n ip_range = local.network_ipv4_subnets[var.vswitch_subnet_index]\n vswitch_id = var.vswitch_id\n}\n\nresource \"hcloud_firewall\" \"k3s\" {\n name = var.cluster_name\n labels = local.labels\n\n dynamic \"rule\" {\n for_each = local.firewall_rules_list\n content {\n description = rule.value.description\n direction = rule.value.direction\n protocol = rule.value.protocol\n port = lookup(rule.value, \"port\", null)\n destination_ips = lookup(rule.value, \"destination_ips\", [])\n source_ips = lookup(rule.value, \"source_ips\", [])\n }\n }\n}\n" }, { "path": "modules/host/locals.tf", "content": "locals {\n # ssh_agent_identity is not set if the private key is passed directly, but if ssh agent is used, the public key tells ssh agent which private key to use.\n # For terraforms provisioner.connection.agent_identity, we need the public key as a string.\n ssh_agent_identity = var.ssh_private_key == null ? var.ssh_public_key : null\n\n # the hosts name with its unique suffix attached\n name = \"${var.name}-${random_string.server.id}\"\n\n # check if the user has set dns servers\n has_dns_servers = length(var.dns_servers) > 0\n}\n" }, { "path": "modules/host/main.tf", "content": "resource \"random_string\" \"server\" {\n length = 3\n lower = true\n special = false\n numeric = false\n upper = false\n\n keepers = {\n # We re-create the apart of the name changes.\n name = var.name\n }\n}\n\nvariable \"network\" {\n type = object({\n network_id = number\n ip = string\n alias_ips = list(string)\n })\n default = null\n}\n\nresource \"hcloud_server\" \"server\" {\n name = local.name\n image = var.microos_snapshot_id\n server_type = var.server_type\n location = var.location\n ssh_keys = var.ssh_keys\n firewall_ids = var.firewall_ids\n placement_group_id = var.placement_group_id\n backups = var.backups\n user_data = data.cloudinit_config.config.rendered\n keep_disk = var.keep_disk_size\n public_net {\n ipv4_enabled = !var.disable_ipv4\n ipv6_enabled = !var.disable_ipv6\n }\n\n network {\n network_id = var.network_id\n ip = var.private_ipv4\n alias_ips = []\n }\n\n labels = var.labels\n\n # Prevent destroying the whole cluster if the user changes\n # any of the attributes that force to recreate the servers.\n lifecycle {\n ignore_changes = [\n location,\n ssh_keys,\n user_data,\n image,\n ]\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(self.ipv4_address, self.ipv6_address, try(one(self.network).ip, null))\n port = var.ssh_port\n\n bastion_host = var.ssh_bastion.bastion_host\n bastion_port = var.ssh_bastion.bastion_port\n bastion_user = var.ssh_bastion.bastion_user\n bastion_private_key = var.ssh_bastion.bastion_private_key\n\n timeout = \"10m\"\n }\n\n provisioner \"remote-exec\" {\n inline = [\n \"echo 'Waiting for system to become fully ready...'\",\n\n # Wait until the system is fully booted and in a running state.\n \"timeout 600 bash -c 'until systemctl is-system-running --quiet; do echo \\\"Waiting for system...\\\"; sleep 3; done'\",\n\n \"echo 'System is fully ready!'\"\n ]\n }\n\n provisioner \"remote-exec\" {\n inline = var.automatically_upgrade_os ? [\n <<-EOT\n echo \"Automatic OS updates are enabled\"\n EOT\n ] : [\n <<-EOT\n echo \"Automatic OS updates are disabled\"\n systemctl --now disable transactional-update.timer\n EOT\n ]\n }\n\n}\n\nresource \"terraform_data\" \"registries\" {\n triggers_replace = {\n registries = var.k3s_registries\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(hcloud_server.server.ipv4_address, hcloud_server.server.ipv6_address, try(one(hcloud_server.server.network).ip, null))\n port = var.ssh_port\n\n bastion_host = var.ssh_bastion.bastion_host\n bastion_port = var.ssh_bastion.bastion_port\n bastion_user = var.ssh_bastion.bastion_user\n bastion_private_key = var.ssh_bastion.bastion_private_key\n\n }\n\n provisioner \"file\" {\n content = var.k3s_registries\n destination = \"/tmp/registries.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [var.k3s_registries_update_script]\n }\n\n depends_on = [hcloud_server.server]\n}\nmoved {\n from = null_resource.registries\n to = terraform_data.registries\n}\n\nresource \"terraform_data\" \"kubelet_config\" {\n count = var.k3s_kubelet_config != \"\" ? 1 : 0\n\n triggers_replace = {\n kubelet_config = var.k3s_kubelet_config\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(hcloud_server.server.ipv4_address, hcloud_server.server.ipv6_address, try(one(hcloud_server.server.network).ip, null))\n port = var.ssh_port\n\n bastion_host = var.ssh_bastion.bastion_host\n bastion_port = var.ssh_bastion.bastion_port\n bastion_user = var.ssh_bastion.bastion_user\n bastion_private_key = var.ssh_bastion.bastion_private_key\n }\n\n provisioner \"file\" {\n content = var.k3s_kubelet_config\n destination = \"/tmp/kubelet-config.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [var.k3s_kubelet_config_update_script]\n }\n\n depends_on = [hcloud_server.server]\n}\nmoved {\n from = null_resource.kubelet_config\n to = terraform_data.kubelet_config\n}\n\nresource \"terraform_data\" \"audit_policy\" {\n count = var.k3s_audit_policy_config != \"\" ? 1 : 0\n\n triggers_replace = {\n audit_policy = var.k3s_audit_policy_config\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(hcloud_server.server.ipv4_address, hcloud_server.server.ipv6_address, try(one(hcloud_server.server.network).ip, null))\n port = var.ssh_port\n\n bastion_host = var.ssh_bastion.bastion_host\n bastion_port = var.ssh_bastion.bastion_port\n bastion_user = var.ssh_bastion.bastion_user\n bastion_private_key = var.ssh_bastion.bastion_private_key\n }\n\n provisioner \"file\" {\n content = var.k3s_audit_policy_config\n destination = \"/tmp/audit-policy.yaml\"\n }\n\n provisioner \"remote-exec\" {\n inline = [var.k3s_audit_policy_update_script]\n }\n\n depends_on = [hcloud_server.server]\n}\nmoved {\n from = null_resource.audit_policy\n to = terraform_data.audit_policy\n}\n\nresource \"hcloud_rdns\" \"server\" {\n count = (var.base_domain != \"\" && !var.disable_ipv4) ? 1 : 0\n\n server_id = hcloud_server.server.id\n ip_address = coalesce(hcloud_server.server.ipv4_address, try(one(hcloud_server.server.network).ip, null))\n dns_ptr = format(\"%s.%s\", local.name, var.base_domain)\n}\n\nresource \"hcloud_rdns\" \"server_ipv6\" {\n count = (var.base_domain != \"\" && !var.disable_ipv6) ? 1 : 0\n\n server_id = hcloud_server.server.id\n ip_address = hcloud_server.server.ipv6_address\n dns_ptr = format(\"%s.%s\", local.name, var.base_domain)\n}\n\n\ndata \"cloudinit_config\" \"config\" {\n gzip = true\n base64_encode = true\n\n # Main cloud-config configuration file.\n part {\n filename = \"init.cfg\"\n content_type = \"text/cloud-config\"\n content = templatefile(\n \"${path.module}/templates/cloudinit.yaml.tpl\",\n {\n hostname = local.name\n dns_servers = var.dns_servers\n has_dns_servers = local.has_dns_servers\n sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)\n cloudinit_write_files_common = var.cloudinit_write_files_common\n cloudinit_runcmd_common = var.cloudinit_runcmd_common\n swap_size = var.swap_size\n private_network_only = (var.disable_ipv4 && var.disable_ipv6)\n network_gw_ipv4 = var.network_gw_ipv4\n }\n )\n }\n}\n\nresource \"terraform_data\" \"zram\" {\n triggers_replace = {\n zram_size = var.zram_size\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(hcloud_server.server.ipv4_address, hcloud_server.server.ipv6_address, try(one(hcloud_server.server.network).ip, null))\n port = var.ssh_port\n\n bastion_host = var.ssh_bastion.bastion_host\n bastion_port = var.ssh_bastion.bastion_port\n bastion_user = var.ssh_bastion.bastion_user\n bastion_private_key = var.ssh_bastion.bastion_private_key\n\n }\n\n provisioner \"file\" {\n content = <<-EOT\n#!/bin/bash\n\n# Switching off swap\nswapoff /dev/zram0\n\nrmmod zram\n EOT\n destination = \"/usr/local/bin/k3s-swapoff\"\n }\n\n provisioner \"file\" {\n content = <<-EOT\n#!/bin/bash\n\n# get the amount of memory in the machine\n# load the dependency module\nmodprobe zram\n\n# initialize the device with zstd compression algorithm\necho zstd > /sys/block/zram0/comp_algorithm;\necho ${var.zram_size} > /sys/block/zram0/disksize\n\n# Creating the swap filesystem\nmkswap /dev/zram0\n\n# Switch the swaps on\nswapon -p 100 /dev/zram0\n EOT\n destination = \"/usr/local/bin/k3s-swapon\"\n }\n\n # Setup zram if it's enabled\n provisioner \"file\" {\n content = <<-EOT\n[Unit]\nDescription=Swap with zram\nAfter=multi-user.target\n\n[Service]\nType=oneshot\nRemainAfterExit=true\nExecStart=/usr/local/bin/k3s-swapon\nExecStop=/usr/local/bin/k3s-swapoff\n\n[Install]\nWantedBy=multi-user.target\n EOT\n destination = \"/etc/systemd/system/zram.service\"\n }\n\n provisioner \"remote-exec\" {\n inline = concat(var.zram_size != \"\" ? [\n \"chmod +x /usr/local/bin/k3s-swapon\",\n \"chmod +x /usr/local/bin/k3s-swapoff\",\n \"systemctl disable --now zram.service\",\n \"systemctl enable --now zram.service\",\n ] : [\n \"systemctl disable --now zram.service\",\n ])\n }\n\n depends_on = [hcloud_server.server]\n}\n\nmoved {\n from = null_resource.zram\n to = terraform_data.zram\n}\n\n# Resource to toggle transactional-update.timer based on automatically_upgrade_os setting\nresource \"terraform_data\" \"os_upgrade_toggle\" {\n triggers_replace = {\n os_upgrade_state = var.automatically_upgrade_os ? \"enabled\" : \"disabled\"\n server_id = hcloud_server.server.id\n }\n\n connection {\n user = \"root\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = coalesce(hcloud_server.server.ipv4_address, hcloud_server.server.ipv6_address, try(one(hcloud_server.server.network).ip, null))\n port = var.ssh_port\n\n bastion_host = var.ssh_bastion.bastion_host\n bastion_port = var.ssh_bastion.bastion_port\n bastion_user = var.ssh_bastion.bastion_user\n bastion_private_key = var.ssh_bastion.bastion_private_key\n\n }\n\n provisioner \"remote-exec\" {\n inline = [\n <<-EOT\n if [ \"${var.automatically_upgrade_os}\" = \"true\" ]; then\n echo \"automatically_upgrade_os changed to true, enabling transactional-update.timer\"\n systemctl enable --now transactional-update.timer || true\n else\n echo \"automatically_upgrade_os changed to false, disabling transactional-update.timer\"\n systemctl disable --now transactional-update.timer || true\n fi\n EOT\n ]\n }\n\n depends_on = [\n hcloud_server.server,\n terraform_data.registries\n ]\n}\n\nmoved {\n from = null_resource.os_upgrade_toggle\n to = terraform_data.os_upgrade_toggle\n}\n" }, { "path": "modules/host/out.tf", "content": "output \"ipv4_address\" {\n value = hcloud_server.server.ipv4_address\n}\n\noutput \"ipv6_address\" {\n value = hcloud_server.server.ipv6_address\n}\n\noutput \"private_ipv4_address\" {\n value = try(one(hcloud_server.server.network).ip, \"\")\n}\n\noutput \"name\" {\n value = hcloud_server.server.name\n}\n\noutput \"id\" {\n value = hcloud_server.server.id\n}\n\noutput \"domain_assignments\" {\n description = \"Assignment of domain to the primary IP of the server\"\n value = [\n for rdns in hcloud_rdns.server : {\n domain = rdns.dns_ptr\n ips = [rdns.ip_address]\n }\n ]\n}\n" }, { "path": "modules/host/templates/cloudinit.yaml.tpl", "content": "#cloud-config\n\nwrite_files:\n\n${cloudinit_write_files_common}\n\n# Apply DNS config\n%{ if has_dns_servers ~}\nmanage_resolv_conf: true\nresolv_conf:\n nameservers:\n%{ for dns_server in dns_servers ~}\n - ${dns_server}\n%{ endfor ~}\n%{ endif ~}\n\n# Add ssh authorized keys\nssh_authorized_keys:\n%{ for key in sshAuthorizedKeys ~}\n - ${key}\n%{ endfor ~}\n\n# Resize /var, not /, as that's the last partition in MicroOS image.\ngrowpart:\n devices: [\"/var\"]\n\n# Make sure the hostname is set correctly\nhostname: ${hostname}\npreserve_hostname: true\n\nruncmd:\n\n${cloudinit_runcmd_common}\n\n# Configure default routes based on public ip availability\n%{if private_network_only~}\n# Private-only setup: detect the private interface dynamically\n- |\n route_dev() {\n awk '{for(i=1;i<=NF;i++) if($i==\"dev\"){print $(i+1); exit}}'\n }\n PRIV_IF=$(ip -4 route get '${network_gw_ipv4}' 2>/dev/null | route_dev)\n if [ -z \"$PRIV_IF\" ]; then\n PRIV_IF=$(ip -4 route show scope link 2>/dev/null | route_dev)\n fi\n if [ -n \"$PRIV_IF\" ]; then\n ip route replace default via '${network_gw_ipv4}' dev \"$PRIV_IF\" metric 100\n else\n echo \"WARN: could not determine private interface for default route\" >&2\n fi\n%{else~}\n# Standard setup: detect public interface dynamically (ARM uses enp7s0, x86 uses eth0)\n- |\n route_dev() {\n awk '{for(i=1;i<=NF;i++) if($i==\"dev\"){print $(i+1); exit}}'\n }\n PUB_IF=$(ip -4 route get 172.31.1.1 2>/dev/null | route_dev)\n # Verify we didn't accidentally pick the private interface (can happen if network_ipv4_cidr overlaps 172.31.0.0/16)\n PRIV_IF=$(ip -4 route get '${network_gw_ipv4}' 2>/dev/null | route_dev)\n if [ -n \"$PRIV_IF\" ] && [ \"$PUB_IF\" = \"$PRIV_IF\" ]; then\n echo \"WARN: detected interface $PUB_IF matches private interface, clearing to trigger fallback\" >&2\n PUB_IF=\"\"\n fi\n if [ -z \"$PUB_IF\" ]; then\n echo \"WARN: could not detect public interface, falling back to eth0\" >&2\n PUB_IF=\"eth0\"\n fi\n ip route replace default via 172.31.1.1 dev \"$PUB_IF\" metric 100\n ip -6 route replace default via fe80::1 dev \"$PUB_IF\" metric 100\n%{endif~}\n\n%{if swap_size != \"\"~}\n- |\n btrfs subvolume create /var/lib/swap 2>/dev/null || true\n chmod 700 /var/lib/swap\n truncate -s 0 /var/lib/swap/swapfile\n chattr +C /var/lib/swap/swapfile\n fallocate -l ${swap_size} /var/lib/swap/swapfile\n chmod 600 /var/lib/swap/swapfile\n mkswap /var/lib/swap/swapfile\n swapon /var/lib/swap/swapfile\n if ! grep -q -F \"/var/lib/swap/swapfile\" /etc/fstab; then\n echo \"/var/lib/swap/swapfile none swap defaults 0 0\" | tee -a /etc/fstab\n fi\n cat <<' EOF' > /etc/systemd/system/swapon-late.service\n [Unit]\n Description=Activate all swap devices later\n After=default.target\n\n [Service]\n Type=oneshot\n ExecStart=/sbin/swapon -a\n\n [Install]\n WantedBy=default.target\n EOF\n systemctl daemon-reload\n systemctl enable swapon-late.service\n%{endif~}\n" }, { "path": "modules/host/variables.tf", "content": "variable \"name\" {\n description = \"Host name\"\n type = string\n}\nvariable \"microos_snapshot_id\" {\n description = \"MicroOS snapshot ID to be used. Per default empty, an initial snapshot will be created\"\n type = string\n default = \"\"\n}\nvariable \"base_domain\" {\n description = \"Base domain used for reverse dns\"\n type = string\n}\n\nvariable \"ssh_port\" {\n description = \"SSH port\"\n type = number\n}\n\nvariable \"ssh_public_key\" {\n description = \"SSH public Key\"\n type = string\n}\n\nvariable \"ssh_private_key\" {\n description = \"SSH private Key\"\n type = string\n}\n\nvariable \"ssh_additional_public_keys\" {\n description = \"Additional SSH public Keys. Use them to grant other team members root access to your cluster nodes\"\n type = list(string)\n default = []\n}\n\nvariable \"ssh_keys\" {\n description = \"List of SSH key IDs\"\n type = list(string)\n nullable = true\n}\n\nvariable \"firewall_ids\" {\n description = \"Set of firewall IDs\"\n type = set(number)\n nullable = true\n}\n\nvariable \"placement_group_id\" {\n description = \"Placement group ID\"\n type = number\n nullable = true\n}\n\nvariable \"labels\" {\n description = \"Labels\"\n type = map(any)\n nullable = true\n}\n\nvariable \"location\" {\n description = \"The server location\"\n type = string\n}\n\nvariable \"ipv4_subnet_id\" {\n description = \"The subnet id\"\n type = string\n}\n\nvariable \"private_ipv4\" {\n description = \"Private IP for the server\"\n type = string\n}\n\nvariable \"server_type\" {\n description = \"The server type\"\n type = string\n}\n\nvariable \"backups\" {\n description = \"Enable automatic backups via Hetzner\"\n type = bool\n default = false\n}\n\nvariable \"packages_to_install\" {\n description = \"Packages to install\"\n type = list(string)\n default = []\n}\n\nvariable \"dns_servers\" {\n type = list(string)\n description = \"IP Addresses to use for the DNS Servers, set to an empty list to use the ones provided by Hetzner\"\n}\n\nvariable \"automatically_upgrade_os\" {\n type = bool\n default = true\n}\n\nvariable \"k3s_registries\" {\n default = \"\"\n type = string\n}\n\nvariable \"k3s_registries_update_script\" {\n default = \"\"\n type = string\n}\n\nvariable \"k3s_kubelet_config\" {\n default = \"\"\n type = string\n}\n\nvariable \"k3s_kubelet_config_update_script\" {\n default = \"\"\n type = string\n}\n\nvariable \"k3s_audit_policy_config\" {\n description = \"K3S audit-policy.yaml contents\"\n type = string\n}\n\nvariable \"k3s_audit_policy_update_script\" {\n description = \"Script to update audit policy configuration\"\n type = string\n}\n\nvariable \"cloudinit_write_files_common\" {\n default = \"\"\n type = string\n}\n\nvariable \"cloudinit_runcmd_common\" {\n default = \"\"\n type = string\n}\n\nvariable \"swap_size\" {\n default = \"\"\n type = string\n\n validation {\n condition = can(regex(\"^$|[1-9][0-9]{0,3}(G|M)$\", var.swap_size))\n error_message = \"Invalid swap size. Examples: 512M, 1G\"\n }\n}\n\nvariable \"zram_size\" {\n default = \"\"\n type = string\n\n validation {\n condition = can(regex(\"^$|[1-9][0-9]{0,3}(G|M)$\", var.zram_size))\n error_message = \"Invalid zram size. Examples: 512M, 1G\"\n }\n}\n\nvariable \"keep_disk_size\" {\n type = bool\n default = false\n description = \"Whether to keep OS disks of nodes the same size when upgrading a node\"\n}\n\nvariable \"disable_ipv4\" {\n type = bool\n default = false\n description = \"Whether to disable ipv4 on the server. If you disable ipv4 and ipv6 make sure you have an access to your private network.\"\n}\n\nvariable \"disable_ipv6\" {\n type = bool\n default = false\n description = \"Whether to disable ipv4 on the server. If you disable ipv4 and ipv6 make sure you have an access to your private network.\"\n}\n\nvariable \"network_id\" {\n type = number\n default = null\n description = \"The network id to attach the server to.\"\n}\n\nvariable \"ssh_bastion\" {\n type = object({\n\n bastion_host = string\n bastion_port = number\n bastion_user = string\n bastion_private_key = string\n })\n}\n\nvariable \"network_gw_ipv4\" {\n type = string\n description = \"Default IPv4 gateway address for the node's primary network interface\"\n}\n" }, { "path": "modules/host/versions.tf", "content": "terraform {\n required_providers {\n hcloud = {\n source = \"hetznercloud/hcloud\"\n version = \">= 1.51.0\"\n }\n }\n}\n" }, { "path": "modules/values_merger/main.tf", "content": "variable \"default_values\" {\n type = string\n default = \"\"\n}\n\nvariable \"override_values\" {\n type = string\n default = \"\"\n}\n\nvariable \"merge_values\" {\n type = string\n default = \"\"\n}\n\nlocals {\n base_values = var.override_values != \"\" ? var.override_values : var.default_values\n final_values = var.merge_values != \"\" ? yamlencode(\n provider::deepmerge::mergo(\n yamldecode(local.base_values),\n yamldecode(var.merge_values)\n )\n ) : local.base_values\n}\n\noutput \"values\" {\n value = local.final_values\n}\n" }, { "path": "modules/values_merger/versions.tf", "content": "terraform {\n required_providers {\n deepmerge = {\n source = \"isometry/deepmerge\"\n version = \"~> 1.0\"\n }\n }\n}\n" }, { "path": "nat-router.tf", "content": "locals {\n nat_gateway_ip = var.nat_router != null ? cidrhost(hcloud_network_subnet.nat_router[0].ip_range, 1) : \"\"\n\n nat_router_ip = (\n var.nat_router != null && var.nat_router.enable_redundancy ?\n {\n 0 = cidrhost(hcloud_network_subnet.nat_router[0].ip_range, 2),\n 1 = cidrhost(hcloud_network_subnet.nat_router[0].ip_range, 3)\n } :\n {\n 0 = local.nat_gateway_ip\n }\n )\n\n nat_router_name_basename = \"nat-router\"\n nat_router_name = \"${var.use_cluster_name_in_node_name ? \"${var.cluster_name}-\" : \"\"}${local.nat_router_name_basename}\"\n}\n\nresource \"random_string\" \"nat_router\" {\n count = var.nat_router != null && var.nat_router.enable_redundancy ? 2 : 0\n\n length = 3\n lower = true\n special = false\n numeric = false\n upper = false\n\n keepers = {\n # Re-create when the stable name prefix changes.\n name = local.nat_router_name\n }\n}\n\nresource \"random_password\" \"nat_router_vip_auth_pass\" {\n count = var.nat_router != null && var.nat_router.enable_redundancy ? 1 : 0\n length = 8\n special = false\n}\n\ndata \"cloudinit_config\" \"nat_router_config\" {\n count = var.nat_router != null ? (var.nat_router.enable_redundancy ? 2 : 1) : 0\n\n gzip = true\n base64_encode = true\n\n # Main cloud-config configuration file.\n part {\n filename = \"init.cfg\"\n content_type = \"text/cloud-config\"\n content = templatefile(\n \"${path.module}/templates/nat-router-cloudinit.yaml.tpl\",\n {\n hostname = var.nat_router.enable_redundancy ? \"nat-router-${count.index}\" : \"nat-router\"\n dns_servers = var.dns_servers\n has_dns_servers = local.has_dns_servers\n sshAuthorizedKeys = concat([var.ssh_public_key], var.ssh_additional_public_keys)\n enable_sudo = var.nat_router.enable_sudo\n enable_redundancy = var.nat_router.enable_redundancy\n priority = count.index == 0 ? 150 : 100\n my_private_ip = local.nat_router_ip[count.index]\n peer_private_ip = var.nat_router.enable_redundancy ? local.nat_router_ip[(count.index == 0 ? 1 : 0)] : null\n hcloud_token = var.nat_router_hcloud_token\n network_id = data.hcloud_network.k3s.id\n vip = local.nat_gateway_ip\n vip_auth_pass = var.nat_router.enable_redundancy ? random_password.nat_router_vip_auth_pass[0].result : \"\"\n private_network_ipv4_range = data.hcloud_network.k3s.ip_range\n ssh_port = var.ssh_port\n ssh_max_auth_tries = var.ssh_max_auth_tries\n enable_cp_lb_port_forward = var.use_control_plane_lb && !var.control_plane_lb_enable_public_interface\n cp_lb_private_ip = try(hcloud_load_balancer_network.control_plane[0].ip, \"\")\n }\n )\n }\n}\n\nresource \"hcloud_network_route\" \"nat_route_public_internet\" {\n count = var.nat_router != null ? 1 : 0\n network_id = data.hcloud_network.k3s.id\n destination = \"0.0.0.0/0\"\n gateway = local.nat_gateway_ip\n}\n\nresource \"hcloud_primary_ip\" \"nat_router_primary_ipv4\" {\n # explicitly declare the ipv4 address, such that the address\n # is stable against possible replacements of the nat router\n count = var.nat_router != null ? (var.nat_router.enable_redundancy ? 2 : 1) : 0\n type = \"ipv4\"\n name = var.nat_router.enable_redundancy ? \"${local.nat_router_name}-${random_string.nat_router[count.index].id}-ipv4\" : \"${var.cluster_name}-nat-router-ipv4\"\n location = var.nat_router.enable_redundancy && count.index == 1 ? var.nat_router.standby_location : var.nat_router.location\n auto_delete = false\n assignee_type = \"server\"\n\n # Prevent recreation when user changes location after initial creation\n lifecycle {\n ignore_changes = [location]\n }\n}\n\nresource \"hcloud_primary_ip\" \"nat_router_primary_ipv6\" {\n # explicitly declare the ipv6 address, such that the address\n # is stable against possible replacements of the nat router\n count = var.nat_router != null ? (var.nat_router.enable_redundancy ? 2 : 1) : 0\n type = \"ipv6\"\n name = var.nat_router.enable_redundancy ? \"${local.nat_router_name}-${random_string.nat_router[count.index].id}-ipv6\" : \"${var.cluster_name}-nat-router-ipv6\"\n location = var.nat_router.enable_redundancy && count.index == 1 ? var.nat_router.standby_location : var.nat_router.location\n auto_delete = false\n assignee_type = \"server\"\n\n # Prevent recreation when user changes location after initial creation\n lifecycle {\n ignore_changes = [location]\n }\n}\n\nresource \"hcloud_server\" \"nat_router\" {\n count = var.nat_router != null ? (var.nat_router.enable_redundancy ? 2 : 1) : 0\n name = var.nat_router.enable_redundancy ? \"${local.nat_router_name}-${random_string.nat_router[count.index].id}\" : \"${var.cluster_name}-nat-router\"\n image = \"debian-12\"\n server_type = var.nat_router.server_type\n location = var.nat_router.enable_redundancy && count.index == 1 ? var.nat_router.standby_location : var.nat_router.location\n ssh_keys = length(var.ssh_hcloud_key_label) > 0 ? concat([local.hcloud_ssh_key_id], data.hcloud_ssh_keys.keys_by_selector[0].ssh_keys.*.id) : [local.hcloud_ssh_key_id]\n firewall_ids = [hcloud_firewall.k3s.id]\n user_data = data.cloudinit_config.nat_router_config[count.index].rendered\n keep_disk = false\n public_net {\n ipv4_enabled = true\n ipv4 = hcloud_primary_ip.nat_router_primary_ipv4[count.index].id\n ipv6_enabled = true\n ipv6 = hcloud_primary_ip.nat_router_primary_ipv6[count.index].id\n }\n\n network {\n network_id = data.hcloud_network.k3s.id\n ip = local.nat_router_ip[count.index]\n alias_ips = []\n }\n\n labels = merge(\n {\n role = \"nat_router\"\n },\n try(var.nat_router.labels, {}),\n )\n\n lifecycle {\n # Keepalived manages alias IPs during failover.\n ignore_changes = [network]\n }\n\n}\n\nresource \"terraform_data\" \"nat_router_await_cloud_init\" {\n count = var.nat_router != null ? (var.nat_router.enable_redundancy ? 2 : 1) : 0\n\n depends_on = [\n hcloud_network_route.nat_route_public_internet,\n hcloud_server.nat_router,\n ]\n\n triggers_replace = {\n config = data.cloudinit_config.nat_router_config[count.index].rendered\n }\n\n connection {\n user = \"nat-router\"\n private_key = var.ssh_private_key\n agent_identity = local.ssh_agent_identity\n host = hcloud_server.nat_router[count.index].ipv4_address\n port = var.ssh_port\n }\n\n provisioner \"remote-exec\" {\n inline = [\"cloud-init status --wait > /dev/null || echo 'Ready to move on'\"]\n # on_failure = continue # this will fail because the reboot \n }\n}\nmoved {\n from = null_resource.nat_router_await_cloud_init\n to = terraform_data.nat_router_await_cloud_init\n}\n" }, { "path": "output.tf", "content": "output \"cluster_name\" {\n value = var.cluster_name\n description = \"Shared suffix for all resources belonging to this cluster.\"\n}\n\noutput \"network_id\" {\n value = data.hcloud_network.k3s.id\n description = \"The ID of the HCloud network.\"\n}\n\noutput \"ssh_key_id\" {\n value = local.hcloud_ssh_key_id\n description = \"The ID of the HCloud SSH key.\"\n}\n\noutput \"control_planes_public_ipv4\" {\n value = [\n for obj in module.control_planes : obj.ipv4_address\n ]\n description = \"The public IPv4 addresses of the controlplane servers.\"\n}\n\noutput \"control_planes_public_ipv6\" {\n value = [\n for obj in module.control_planes : obj.ipv6_address\n ]\n description = \"The public IPv6 addresses of the controlplane servers.\"\n}\n\noutput \"agents_public_ipv4\" {\n value = [\n for obj in module.agents : obj.ipv4_address\n ]\n description = \"The public IPv4 addresses of the agent servers.\"\n}\n\noutput \"agents_public_ipv6\" {\n value = [\n for obj in module.agents : obj.ipv6_address\n ]\n description = \"The public IPv6 addresses of the agent servers.\"\n}\n\noutput \"ingress_public_ipv4\" {\n description = \"The public IPv4 address of the Hetzner load balancer (with fallback to first control plane node)\"\n value = local.has_external_load_balancer ? local.first_control_plane_ip : hcloud_load_balancer.cluster[0].ipv4\n}\n\noutput \"ingress_public_ipv6\" {\n description = \"The public IPv6 address of the Hetzner load balancer (with fallback to first control plane node)\"\n value = local.has_external_load_balancer ? module.control_planes[keys(module.control_planes)[0]].ipv6_address : (var.load_balancer_disable_ipv6 ? null : hcloud_load_balancer.cluster[0].ipv6)\n}\n\noutput \"lb_control_plane_ipv4\" {\n description = \"The public IPv4 address of the Hetzner control plane load balancer\"\n value = one(hcloud_load_balancer.control_plane[*].ipv4)\n}\n\noutput \"lb_control_plane_ipv6\" {\n description = \"The public IPv6 address of the Hetzner control plane load balancer\"\n value = one(hcloud_load_balancer.control_plane[*].ipv6)\n}\n\noutput \"k3s_endpoint\" {\n description = \"A controller endpoint to register new nodes\"\n value = local.k3s_endpoint\n}\n\noutput \"k3s_token\" {\n description = \"The k3s token to register new nodes\"\n value = local.k3s_token\n sensitive = true\n}\n\noutput \"control_plane_nodes\" {\n description = \"The control plane nodes\"\n value = [for node in module.control_planes : node]\n}\n\noutput \"agent_nodes\" {\n description = \"The agent nodes\"\n value = [for node in module.agents : node]\n}\n\noutput \"domain_assignments\" {\n description = \"Assignments of domains to IPs based on reverse DNS\"\n value = concat(\n # Propagate domain assignments from control plane and agent nodes.\n flatten([\n for node in concat(values(module.control_planes), values(module.agents)) :\n node.domain_assignments\n ]),\n # Get assignments from floating IPs.\n [for rdns in hcloud_rdns.agents : {\n domain = rdns.dns_ptr\n ips = [rdns.ip_address]\n }]\n )\n}\n\n# Keeping for backward compatibility\noutput \"kubeconfig_file\" {\n value = local.kubeconfig_external\n description = \"Kubeconfig file content with external IP address, or internal IP address if only private ips are available\"\n sensitive = true\n}\n\noutput \"kubeconfig\" {\n value = local.kubeconfig_external\n description = \"Kubeconfig file content with external IP address, or internal IP address if only private ips are available\"\n sensitive = true\n}\n\noutput \"kubeconfig_data\" {\n description = \"Structured kubeconfig data to supply to other providers\"\n value = local.kubeconfig_data\n sensitive = true\n}\n\noutput \"cilium_values\" {\n description = \"Helm values.yaml used for Cilium\"\n value = local.cilium_values\n sensitive = true\n}\n\noutput \"cert_manager_values\" {\n description = \"Helm values.yaml used for cert-manager\"\n value = local.cert_manager_values\n sensitive = true\n}\n\noutput \"csi_driver_smb_values\" {\n description = \"Helm values.yaml used for SMB CSI driver\"\n value = local.csi_driver_smb_values\n sensitive = true\n}\n\noutput \"longhorn_values\" {\n description = \"Helm values.yaml used for Longhorn\"\n value = local.longhorn_values\n sensitive = true\n}\n\noutput \"traefik_values\" {\n description = \"Helm values.yaml used for Traefik\"\n value = local.traefik_values\n sensitive = true\n}\n\noutput \"nginx_values\" {\n description = \"Helm values.yaml used for nginx-ingress\"\n value = local.nginx_values\n sensitive = true\n}\n\noutput \"haproxy_values\" {\n description = \"Helm values.yaml used for HAProxy\"\n value = local.haproxy_values\n sensitive = true\n}\n\noutput \"nat_router_public_ipv4\" {\n description = \"The address of the nat router, if it exists.\"\n value = try(hcloud_server.nat_router[0].ipv4_address, null)\n}\noutput \"nat_router_public_ipv6\" {\n description = \"The address of the nat router, if it exists.\"\n value = try(hcloud_server.nat_router[0].ipv6_address, null)\n}\noutput \"nat_router_public_ipv4_addresses\" {\n description = \"The addresses of all nat routers, if they exist.\"\n value = [for nat_router in hcloud_server.nat_router : nat_router.ipv4_address]\n}\noutput \"nat_router_public_ipv6_addresses\" {\n description = \"The addresses of all nat routers, if they exist.\"\n value = [for nat_router in hcloud_server.nat_router : nat_router.ipv6_address]\n}\noutput \"nat_router_username\" {\n description = \"The non-root user as which you can ssh into the router.\"\n value = \"nat-router\" # hard-coded in cloud-init template.\n}\noutput \"nat_router_ssh_port\" {\n description = \"The non-root user as which you can ssh into the router.\"\n value = var.ssh_port\n}\n\noutput \"vswitch_subnet\" {\n description = \"Attributes of the vSwitch subnet.\"\n value = try(hcloud_network_subnet.vswitch_subnet[0], null)\n}\n" }, { "path": "packer-template/hcloud-microos-snapshots.pkr.hcl", "content": "/*\n * Creates a MicroOS snapshot for Kube-Hetzner\n */\npacker {\n required_plugins {\n hcloud = {\n version = \">= 1.0.5\"\n source = \"github.com/hetznercloud/hcloud\"\n }\n }\n}\n\nvariable \"hcloud_token\" {\n type = string\n default = env(\"HCLOUD_TOKEN\")\n sensitive = true\n}\n\n# We download the OpenSUSE MicroOS x86 image from an automatically selected mirror.\nvariable \"opensuse_microos_x86_mirror_link\" {\n type = string\n default = \"https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-ContainerHost-OpenStack-Cloud.qcow2\"\n}\n\n# We download the OpenSUSE MicroOS ARM image from an automatically selected mirror.\nvariable \"opensuse_microos_arm_mirror_link\" {\n type = string\n default = \"https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2\"\n}\n\n# If you need to add other packages to the OS, do it here in the default value, like [\"vim\", \"curl\", \"wget\"]\n# When looking for packages, you need to search for OpenSUSE Tumbleweed packages, as MicroOS is based on Tumbleweed.\nvariable \"packages_to_install\" {\n type = list(string)\n default = []\n}\n\n# Timezone to set on the snapshot (e.g., \"Europe/Madrid\", \"UTC\", \"America/New_York\")\nvariable \"timezone\" {\n type = string\n default = \"UTC\"\n}\n\n# Path to a local file containing sysctl settings (one per line, e.g., \"vm.swappiness = 10\")\n# These will be installed to /etc/sysctl.d/99-custom.conf\nvariable \"sysctl_config_file\" {\n type = string\n default = \"\"\n}\n\n# Choose which kernel to use: \"default\" for the rolling release kernel or \"longterm\" for LTS kernel\nvariable \"kernel_type\" {\n type = string\n default = \"default\"\n validation {\n condition = contains([\"longterm\", \"default\"], var.kernel_type)\n error_message = \"The kernel_type must be either longterm or default.\"\n }\n}\n\nlocals {\n # Only install kernel-longterm if selected; kernel-default is already in the base image\n kernel_package_list = var.kernel_type == \"longterm\" ? [\"kernel-longterm\"] : []\n\n needed_packages = join(\" \", concat(local.kernel_package_list, [\"restorecond\", \"policycoreutils\", \"policycoreutils-python-utils\", \"setools-console\", \"audit\", \"bind-utils\", \"wireguard-tools\", \"fuse\", \"open-iscsi\", \"nfs-client\", \"xfsprogs\", \"cryptsetup\", \"lvm2\", \"git\", \"cifs-utils\", \"bash-completion\", \"mtr\", \"tcpdump\", \"udica\", \"qemu-guest-agent\"], var.packages_to_install))\n\n # Read sysctl config if file path is provided, otherwise empty (base64 encoded for safe transfer)\n sysctl_config_content = var.sysctl_config_file != \"\" ? base64encode(file(var.sysctl_config_file)) : \"\"\n\n # Commands to write sysctl config if provided (decode base64)\n sysctl_commands = local.sysctl_config_content != \"\" ? \"echo '${local.sysctl_config_content}' | base64 -d > /etc/sysctl.d/99-custom.conf\" : \"\"\n\n # Add local variables for inline shell commands\n download_image = \"wget --timeout=5 --waitretry=5 --tries=5 --retry-connrefused --inet4-only \"\n\n write_image = <<-EOT\n set -ex\n echo 'MicroOS image loaded, writing to disk... '\n qemu-img convert -p -f qcow2 -O host_device $(ls -a | grep -ie '^opensuse.*microos.*qcow2$') /dev/sda\n echo 'done. Rebooting...'\n sleep 1 && udevadm settle && reboot\n EOT\n\n # Kernel switching commands: remove kernel-default and lock it when using longterm\n # This ensures GRUB always boots the longterm kernel without complex configuration\n kernel_switch_commands = var.kernel_type == \"longterm\" ? join(\"\\n\", [\n \"zypper rm -y kernel-default\",\n \"zypper addlock kernel-default\",\n \"grub2-mkconfig -o /boot/grub2/grub.cfg\"\n ]) : \"true\"\n\n install_packages = <<-EOT\n set -ex\n echo \"First reboot successful, installing needed packages...\"\n transactional-update --continue pkg install -y ${local.needed_packages}\n transactional-update --continue shell <<- EOF\n setenforce 0\n rpm --import https://rpm.rancher.io/public.key\n zypper install -y https://github.com/k3s-io/k3s-selinux/releases/download/v1.6.stable.1/k3s-selinux-1.6-1.sle.noarch.rpm\n zypper addlock k3s-selinux\n restorecon -Rv /etc/selinux/targeted/policy\n restorecon -Rv /var/lib\n setenforce 1\n ${local.sysctl_commands}\n ${local.kernel_switch_commands}\n EOF\n sleep 1 && udevadm settle && reboot\n EOT\n\n clean_up = <<-EOT\n set -ex\n echo \"Second reboot successful, cleaning-up...\"\n rm -rf /etc/ssh/ssh_host_*\n echo \"Make sure to use NetworkManager\"\n touch /etc/NetworkManager/NetworkManager.conf\n echo \"Setting timezone to '${var.timezone}'...\"\n timedatectl set-timezone '${var.timezone}'\n sleep 1 && udevadm settle\n EOT\n}\n\n# Source for the MicroOS x86 snapshot\nsource \"hcloud\" \"microos-x86-snapshot\" {\n image = \"ubuntu-24.04\"\n rescue = \"linux64\"\n location = \"nbg1\"\n server_type = \"cx23\" # disk size of >= 40GiB is needed to install the MicroOS image\n snapshot_labels = {\n microos-snapshot = \"yes\"\n creator = \"kube-hetzner\"\n }\n snapshot_name = \"OpenSUSE MicroOS x86 by Kube-Hetzner\"\n ssh_username = \"root\"\n token = var.hcloud_token\n}\n\n# Source for the MicroOS ARM snapshot\nsource \"hcloud\" \"microos-arm-snapshot\" {\n image = \"ubuntu-24.04\"\n rescue = \"linux64\"\n location = \"nbg1\"\n server_type = \"cax11\" # disk size of >= 40GiB is needed to install the MicroOS image\n snapshot_labels = {\n microos-snapshot = \"yes\"\n creator = \"kube-hetzner\"\n }\n snapshot_name = \"OpenSUSE MicroOS ARM by Kube-Hetzner\"\n ssh_username = \"root\"\n token = var.hcloud_token\n}\n\n# Build the MicroOS x86 snapshot\nbuild {\n sources = [\"source.hcloud.microos-x86-snapshot\"]\n\n # Download the MicroOS x86 image\n provisioner \"shell\" {\n inline = [\"${local.download_image}${var.opensuse_microos_x86_mirror_link}\"]\n }\n\n # Write the MicroOS x86 image to disk\n provisioner \"shell\" {\n inline = [local.write_image]\n expect_disconnect = true\n }\n\n # Ensure connection to MicroOS x86 and do house-keeping\n provisioner \"shell\" {\n pause_before = \"5s\"\n inline = [local.install_packages]\n expect_disconnect = true\n }\n\n # Ensure connection to MicroOS x86 and do house-keeping\n provisioner \"shell\" {\n pause_before = \"5s\"\n inline = [local.clean_up]\n }\n}\n\n# Build the MicroOS ARM snapshot\nbuild {\n sources = [\"source.hcloud.microos-arm-snapshot\"]\n\n # Download the MicroOS ARM image\n provisioner \"shell\" {\n inline = [\"${local.download_image}${var.opensuse_microos_arm_mirror_link}\"]\n }\n\n # Write the MicroOS ARM image to disk\n provisioner \"shell\" {\n inline = [local.write_image]\n expect_disconnect = true\n }\n\n # Ensure connection to MicroOS ARM and do house-keeping\n provisioner \"shell\" {\n pause_before = \"5s\"\n inline = [local.install_packages]\n expect_disconnect = true\n }\n\n # Ensure connection to MicroOS ARM and do house-keeping\n provisioner \"shell\" {\n pause_before = \"5s\"\n inline = [local.clean_up]\n }\n}\n" }, { "path": "placement_groups.tf", "content": "locals {\n control_plane_placement_compat_groups = max(\n 0,\n [\n for cp_pool in var.control_plane_nodepools :\n cp_pool.placement_group_compat_idx + 1 if cp_pool.placement_group_compat_idx != null && cp_pool.placement_group == null\n ]...\n )\n control_plane_groups = toset(\n [\n for cp_pool in var.control_plane_nodepools :\n cp_pool.placement_group if cp_pool.placement_group != null\n ]\n )\n agent_placement_compat_groups = max(\n 0,\n [\n for ag_pool in var.agent_nodepools :\n ag_pool.placement_group_compat_idx + 1 if ag_pool.placement_group_compat_idx != null && ag_pool.placement_group == null\n ]...\n )\n agent_placement_groups = toset(\n concat(\n [\n for ag_pool in var.agent_nodepools :\n ag_pool.placement_group if ag_pool.placement_group != null\n ],\n concat(\n [\n for ag_pool in var.agent_nodepools :\n [\n for node, node_config in coalesce(ag_pool.nodes, {}) :\n node_config.placement_group if node_config.placement_group != null\n ]\n ]\n )...\n )\n )\n}\n\nresource \"hcloud_placement_group\" \"control_plane\" {\n count = local.control_plane_placement_compat_groups\n name = \"${var.cluster_name}-control-plane-${count.index + 1}\"\n labels = local.labels\n type = \"spread\"\n}\n\nresource \"hcloud_placement_group\" \"control_plane_named\" {\n for_each = local.control_plane_groups\n name = \"${var.cluster_name}-control-plane-${each.key}\"\n labels = local.labels\n type = \"spread\"\n}\n\nresource \"hcloud_placement_group\" \"agent\" {\n count = local.agent_placement_compat_groups\n name = \"${var.cluster_name}-agent-${count.index + 1}\"\n labels = local.labels\n type = \"spread\"\n}\n\nresource \"hcloud_placement_group\" \"agent_named\" {\n for_each = local.agent_placement_groups\n name = \"${var.cluster_name}-agent-${each.key}\"\n labels = local.labels\n type = \"spread\"\n}\n" }, { "path": "scripts/cleanup.sh", "content": "#!/usr/bin/env bash\n\nDRY_RUN=1\n\necho \"Welcome to the Kube-Hetzner cluster deletion script!\"\necho \" \"\necho \"We advise you to first run 'terraform destroy' and execute that script when it starts hanging because of resources still attached to the network.\"\necho \"In order to run this script need to have the hcloud CLI installed and configured with a context for the cluster you want to delete.\"\ncommand -v hcloud >/dev/null 2>&1 || { echo \"hcloud (Hetzner CLI) is not installed. Install it with 'brew install hcloud'.\"; exit 1; }\necho \"You can do so by running 'hcloud context create <cluster_name>' and inputting your HCLOUD_TOKEN.\"\necho \" \"\n\nif command -v tofu >/dev/null 2>&1 ; then\n terraform_command=tofu\nelif command -v terraform >/dev/null 2>&1 ; then\n terraform_command=terraform\nelse\n echo \"terraform or tofu is not installed. Install it with 'brew install terraform' or 'brew install opentofu'.\"\n exit 1\nfi\n\n\n# Try to guess the cluster name\nGUESSED_CLUSTER_NAME=$(sed -n 's/^[[:space:]]*cluster_name[[:space:]]*=[[:space:]]*\"\\([^\"]*\\)\".*/\\1/p' kube.tf 2>/dev/null)\n\nif [ -n \"$GUESSED_CLUSTER_NAME\" ]; then\n echo \"Cluster name '$GUESSED_CLUSTER_NAME' has been detected in the kube.tf file.\"\n read -p \"Enter the name of the cluster to delete (default: $GUESSED_CLUSTER_NAME): \" CLUSTER_NAME\n if [ -z \"$CLUSTER_NAME\" ]; then\n CLUSTER_NAME=\"$GUESSED_CLUSTER_NAME\"\n fi\nelse\n read -p \"Enter the name of the cluster to delete: \" CLUSTER_NAME\nfi\n\nwhile true; do\n read -p \"Do you want to perform a dry run? (yes/no): \" dry_run_input\n case $dry_run_input in\n [Yy]* ) DRY_RUN=1; break;;\n [Nn]* ) DRY_RUN=0; break;;\n * ) echo \"Please answer yes or no.\";;\n esac\ndone\n\nread -p \"Do you want to delete volumes? (yes/no, default: no): \" delete_volumes_input\nDELETE_VOLUMES=0\nif [[ \"$delete_volumes_input\" =~ ^([Yy]es|[Yy])$ ]]; then\n DELETE_VOLUMES=1\nfi\n\nread -p \"Do you want to delete MicroOS snapshots? (yes/no, default: no): \" delete_snapshots_input\nDELETE_SNAPSHOTS=0\nif [[ \"$delete_snapshots_input\" =~ ^([Yy]es|[Yy])$ ]]; then\n DELETE_SNAPSHOTS=1\nfi\n\nif (( DRY_RUN == 0 )); then\n echo \"WARNING: STUFF WILL BE DELETED!\"\nelse\n echo \"Performing a dry run, nothing will be deleted.\"\nfi\n\nHCLOUD_SELECTOR=(--selector='provisioner=terraform' --selector=\"cluster=$CLUSTER_NAME\")\nHCLOUD_OUTPUT_OPTIONS=(-o noheader -o 'columns=id')\n\nVOLUMES=()\nwhile IFS='' read -r line; do VOLUMES+=(\"$line\"); done < <(hcloud volume list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nSERVERS=()\nwhile IFS='' read -r line; do SERVERS+=(\"$line\"); done < <(hcloud server list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nPLACEMENT_GROUPS=()\nwhile IFS='' read -r line; do PLACEMENT_GROUPS+=(\"$line\"); done < <(hcloud placement-group list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nLOAD_BALANCERS=()\nwhile IFS='' read -r line; do LOAD_BALANCER+=(\"$line\"); done < <(hcloud load-balancer list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nINGRESS_LB=$(hcloud load-balancer list -o noheader -o columns=id,name | grep \"${CLUSTER_NAME}\" | cut -d ' ' -f1 )\n\nif [[ $INGRESS_LB != \"\" ]]; then\n LOAD_BALANCERS+=( \"$INGRESS_LB\" )\nfi\n\nFIREWALLS=()\nwhile IFS='' read -r line; do FIREWALLS+=(\"$line\"); done < <(hcloud firewall list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nNETWORKS=()\nwhile IFS='' read -r line; do NETWORKS+=(\"$line\"); done < <(hcloud network list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nSSH_KEYS=()\nwhile IFS='' read -r line; do SSH_KEYS+=(\"$line\"); done < <(hcloud ssh-key list \"${HCLOUD_SELECTOR[@]}\" \"${HCLOUD_OUTPUT_OPTIONS[@]}\")\n\nfunction detach_volumes() {\n for ID in \"${VOLUMES[@]}\"; do\n echo \"Detach volume: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud volume detach \"$ID\"\n fi\n done\n}\n\nfunction delete_volumes() {\n for ID in \"${VOLUMES[@]}\"; do\n echo \"Delete volume: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud volume delete \"$ID\"\n fi\n done\n}\n\nfunction delete_servers() {\n for ID in \"${SERVERS[@]}\"; do\n echo \"Delete server: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud server delete \"$ID\"\n fi\n done\n}\n\nfunction delete_placement_groups() {\n for ID in \"${PLACEMENT_GROUPS[@]}\"; do\n echo \"Delete placement-group: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud placement-group delete \"$ID\"\n fi\n done\n}\n\nfunction delete_load_balancer() {\n for ID in \"${LOAD_BALANCERS[@]}\"; do\n echo \"Delete load-balancer: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud load-balancer delete \"$ID\"\n fi\n done\n}\n\nfunction delete_firewalls() {\n for ID in \"${FIREWALLS[@]}\"; do\n echo \"Delete firewall: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud firewall delete \"$ID\"\n fi\n done\n}\n\nfunction delete_networks() {\n for ID in \"${NETWORKS[@]}\"; do\n echo \"Delete network: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud network delete \"$ID\"\n fi\n done\n}\n\nfunction delete_ssh_keys() {\n for ID in \"${SSH_KEYS[@]}\"; do\n echo \"Delete ssh-key: $ID\"\n if (( DRY_RUN == 0 )); then\n hcloud ssh-key delete \"$ID\"\n fi\n done\n}\n\nfunction delete_autoscaled_nodes() {\n local servers\n while IFS='' read -r line; do servers+=(\"$line\"); done < <(hcloud server list -o noheader -o 'columns=id,name' | grep \"${CLUSTER_NAME}\")\n\n for server_info in \"${servers[@]}\"; do\n local ID=$(echo \"$server_info\" | awk '{print $1}')\n local server_name=$(echo \"$server_info\" | awk '{print $2}')\n echo \"Delete autoscaled server: $ID (Name: $server_name)\"\n if (( DRY_RUN == 0 )); then\n hcloud server delete \"$ID\"\n fi\n done\n}\n\nfunction delete_snapshots() {\n local snapshots\n while IFS='' read -r line; do snapshots+=(\"$line\"); done < <(hcloud image list --selector 'microos-snapshot=yes' -o noheader -o 'columns=id,name')\n\n for snapshot_info in \"${snapshots[@]}\"; do\n local ID=$(echo \"$snapshot_info\" | awk '{print $1}')\n local snapshot_name=$(echo \"$snapshot_info\" | awk '{print $2}')\n echo \"Delete snapshot: $ID (Name: $snapshot_name)\"\n if (( DRY_RUN == 0 )); then\n hcloud image delete \"$ID\"\n fi\n done\n}\n\nif (( DRY_RUN > 0 )); then\n echo \"Dry run, nothing will be deleted!\"\nfi\n\ndetach_volumes\nif (( DELETE_VOLUMES == 1 )); then\n delete_volumes\nfi\ndelete_servers\ndelete_autoscaled_nodes\ndelete_placement_groups\ndelete_load_balancer\ndelete_networks\ndelete_firewalls\ndelete_ssh_keys\n\n\nif (( DELETE_SNAPSHOTS == 1 )); then\n delete_snapshots\nfi\n" }, { "path": "scripts/create.sh", "content": "#!/usr/bin/env bash\n\n# Check if terraform, packer and hcloud CLIs are present\ncommand -v ssh >/dev/null 2>&1 || {\n echo \"openssh is not installed. Install it with 'brew install openssh'.\"\n exit 1\n}\n\nif command -v tofu >/dev/null 2>&1 ; then\n terraform_command=tofu\nelif command -v terraform >/dev/null 2>&1 ; then\n terraform_command=terraform\nelse\n echo \"terraform or tofu is not installed. Install it with 'brew tap hashicorp/tap && brew install hashicorp/tap/terraform' or 'brew install opentofu'.\"\n exit 1\nfi\n\ncommand -v packer >/dev/null 2>&1 || {\n echo \"packer is not installed. Install it with 'brew install packer'.\"\n exit 1\n}\ncommand -v hcloud >/dev/null 2>&1 || {\n echo \"hcloud (Hetzner CLI) is not installed. Install it with 'brew install hcloud'.\"\n exit 1\n}\n\n# Ask for the folder name\nif [ -z \"${folder_name}\" ] ; then\n read -p \"Enter the name of the folder you want to create (leave empty to use the current directory instead, useful for upgrades): \" folder_name\nfi\n\n# Ask for the folder path only if folder_name is provided\nif [ -n \"$folder_name\" -a -z \"${folder_path}\" ]; then\n read -p \"Enter the path to create the folder in (default: current path): \" folder_path\nfi\n\n# Set default path if not provided\nif [ -z \"$folder_path\" ]; then\n folder_path=\".\"\nfi\n\n# Create the folder if folder_name is provided\nif [ -n \"$folder_name\" ]; then\n mkdir -p \"${folder_path}/${folder_name}\"\n folder_path=\"${folder_path}/${folder_name}\"\nfi\n\n# Download the required files only if they don't exist\nif [ ! -e \"${folder_path}/kube.tf\" ]; then\n curl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/kube.tf.example -o \"${folder_path}/kube.tf\"\nelse\n echo \"kube.tf already exists. Skipping download.\"\nfi\n\nif [ ! -e \"${folder_path}/hcloud-microos-snapshots.pkr.hcl\" ]; then\n curl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/packer-template/hcloud-microos-snapshots.pkr.hcl -o \"${folder_path}/hcloud-microos-snapshots.pkr.hcl\"\nelse\n echo \"hcloud-microos-snapshots.pkr.hcl already exists. Skipping download.\"\nfi\n\n# Ask if they want to create the MicroOS snapshots\nif [ -z \"${create_snapshots}\" ] ; then\n echo \" \"\n echo \"The snapshots are required and deployed using packer. If you need specific extra packages, you need to choose no and edit hcloud-microos-snapshots.pkr.hcl file manually. This is not needed in 99% of cases, as we already include the most common packages.\"\n echo \" \"\n read -p \"Do you want to create the MicroOS snapshots (we create one for x86 and one for ARM architectures) with packer now? (yes/no): \" create_snapshots\nfi\n\nif [[ \"$create_snapshots\" =~ ^([Yy]es|[Yy])$ ]]; then\n if [[ -z \"$HCLOUD_TOKEN\" ]]; then\n read -p \"Enter your HCLOUD_TOKEN: \" hcloud_token\n export HCLOUD_TOKEN=$hcloud_token\n fi\n echo \"Running packer build for hcloud-microos-snapshots.pkr.hcl\"\n cd \"${folder_path}\" && packer init hcloud-microos-snapshots.pkr.hcl && packer build hcloud-microos-snapshots.pkr.hcl\nelse\n echo \" \"\n echo \"You can create the snapshots later by running 'packer build hcloud-microos-snapshots.pkr.hcl' in the folder.\"\nfi\n\n# Output commands\necho \" \"\necho \"Remember, don't skip the hcloud cli, to activate it run 'hcloud context create <project-name>'. It is ideal to quickly debug and allows targeted cleanup when needed!\"\necho \" \"\necho \"Before running '${terraform_command} apply', go through the kube.tf file and fill it with your desired values.\"\n" }, { "path": "templates/autoscaler-cloudinit.yaml.tpl", "content": "#cloud-config\n\nwrite_files:\n\n${cloudinit_write_files_common}\n\n- content: ${base64encode(k3s_config)}\n encoding: base64\n path: /tmp/config.yaml\n\n- content: ${base64encode(install_k3s_agent_script)}\n encoding: base64\n path: /var/pre_install/install-k3s-agent.sh\n\n# Apply DNS config\n%{ if has_dns_servers ~}\nmanage_resolv_conf: true\nresolv_conf:\n nameservers:\n%{ for dns_server in dns_servers ~}\n - ${dns_server}\n%{ endfor ~}\n%{ endif ~}\n\n# Add ssh authorized keys\nssh_authorized_keys:\n%{ for key in sshAuthorizedKeys ~}\n - ${key}\n%{ endfor ~}\n\n# Resize /var, not /, as that's the last partition in MicroOS image.\ngrowpart:\n devices: [\"/var\"]\n\n# Make sure the hostname is set correctly\nhostname: ${hostname}\npreserve_hostname: true\n\nruncmd:\n\n${cloudinit_runcmd_common}\n\n# Configure default routes based on public ip availability\n%{if private_network_only~}\n# Private-only setup: detect the private interface dynamically\n- |\n route_dev() {\n awk '{for(i=1;i<=NF;i++) if($i==\"dev\"){print $(i+1); exit}}'\n }\n PRIV_IF=$(ip -4 route get '${network_gw_ipv4}' 2>/dev/null | route_dev)\n if [ -z \"$PRIV_IF\" ]; then\n PRIV_IF=$(ip -4 route show scope link 2>/dev/null | route_dev)\n fi\n if [ -n \"$PRIV_IF\" ]; then\n ip route replace default via '${network_gw_ipv4}' dev \"$PRIV_IF\" metric 100\n else\n echo \"WARN: could not determine private interface for default route\" >&2\n fi\n%{else~}\n# Standard setup: detect public interface dynamically (ARM uses enp7s0, x86 uses eth0)\n- |\n route_dev() {\n awk '{for(i=1;i<=NF;i++) if($i==\"dev\"){print $(i+1); exit}}'\n }\n PUB_IF=$(ip -4 route get 172.31.1.1 2>/dev/null | route_dev)\n # Verify we didn't accidentally pick the private interface (can happen if network_ipv4_cidr overlaps 172.31.0.0/16)\n PRIV_IF=$(ip -4 route get '${network_gw_ipv4}' 2>/dev/null | route_dev)\n if [ -n \"$PRIV_IF\" ] && [ \"$PUB_IF\" = \"$PRIV_IF\" ]; then\n echo \"WARN: detected interface $PUB_IF matches private interface, clearing to trigger fallback\" >&2\n PUB_IF=\"\"\n fi\n if [ -z \"$PUB_IF\" ]; then\n echo \"WARN: could not detect public interface, falling back to eth0\" >&2\n PUB_IF=\"eth0\"\n fi\n ip route replace default via 172.31.1.1 dev \"$PUB_IF\" metric 100\n ip -6 route replace default via fe80::1 dev \"$PUB_IF\" metric 100\n%{endif~}\n\n%{if swap_size != \"\"~}\n- |\n btrfs subvolume create /var/lib/swap 2>/dev/null || true\n chmod 700 /var/lib/swap\n truncate -s 0 /var/lib/swap/swapfile\n chattr +C /var/lib/swap/swapfile\n fallocate -l ${swap_size} /var/lib/swap/swapfile\n chmod 600 /var/lib/swap/swapfile\n mkswap /var/lib/swap/swapfile\n swapon /var/lib/swap/swapfile\n if ! grep -q -F \"/var/lib/swap/swapfile\" /etc/fstab; then\n echo \"/var/lib/swap/swapfile none swap defaults 0 0\" | tee -a /etc/fstab\n fi\n cat <<' EOF' > /etc/systemd/system/swapon-late.service\n [Unit]\n Description=Activate all swap devices later\n After=default.target\n\n [Service]\n Type=oneshot\n ExecStart=/sbin/swapon -a\n\n [Install]\n WantedBy=default.target\n EOF\n systemctl daemon-reload\n systemctl enable swapon-late.service\n%{endif~}\n\n%{if zram_size != \"\"~}\n- |\n cat <<' EOF' > /usr/local/bin/k3s-swapoff\n #!/bin/bash\n\n # Switching off swap\n swapoff /dev/zram0\n\n rmmod zram\n EOF\n chmod +x /usr/local/bin/k3s-swapoff\n\n cat <<' EOF' > /usr/local/bin/k3s-swapon\n #!/bin/bash\n\n # load the dependency module\n modprobe zram\n\n # initialize the device with zstd compression algorithm\n echo zstd > /sys/block/zram0/comp_algorithm;\n echo ZRAM_SIZE_PLACEHOLDER > /sys/block/zram0/disksize\n\n # Creating the swap filesystem\n mkswap /dev/zram0\n\n # Switch the swaps on\n swapon -p 100 /dev/zram0\n EOF\n sed -i 's/ZRAM_SIZE_PLACEHOLDER/${zram_size}/' /usr/local/bin/k3s-swapon\n chmod +x /usr/local/bin/k3s-swapon\n\n cat <<' EOF' > /etc/systemd/system/zram.service\n [Unit]\n Description=Swap with zram\n After=multi-user.target\n\n [Service]\n Type=oneshot\n RemainAfterExit=true\n ExecStart=/usr/local/bin/k3s-swapon\n ExecStop=/usr/local/bin/k3s-swapoff\n\n [Install]\n WantedBy=multi-user.target\n EOF\n systemctl daemon-reload\n systemctl enable --now zram.service\n%{endif~}\n\n# Start the install-k3s-agent service\n- ['/bin/bash', '/var/pre_install/install-k3s-agent.sh']\n" }, { "path": "templates/autoscaler.yaml.tpl", "content": "---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n labels:\n k8s-addon: cluster-autoscaler.addons.k8s.io\n k8s-app: cluster-autoscaler\n name: cluster-autoscaler\n namespace: kube-system\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n name: cluster-autoscaler\n labels:\n k8s-addon: cluster-autoscaler.addons.k8s.io\n k8s-app: cluster-autoscaler\nrules:\n - apiGroups: [\"\"]\n resources: [\"events\", \"endpoints\"]\n verbs: [\"create\", \"patch\"]\n - apiGroups: [\"\"]\n resources: [\"pods/eviction\"]\n verbs: [\"create\"]\n - apiGroups: [\"\"]\n resources: [\"pods/status\"]\n verbs: [\"update\"]\n - apiGroups: [\"\"]\n resources: [\"endpoints\"]\n resourceNames: [\"cluster-autoscaler\"]\n verbs: [\"get\", \"update\"]\n - apiGroups: [\"\"]\n resources: [\"nodes\"]\n verbs: [\"watch\", \"list\", \"get\", \"update\"]\n - apiGroups: [\"\"]\n resources:\n - \"namespaces\"\n - \"pods\"\n - \"services\"\n - \"replicationcontrollers\"\n - \"persistentvolumeclaims\"\n - \"persistentvolumes\"\n verbs: [\"watch\", \"list\", \"get\"]\n - apiGroups: [\"extensions\"]\n resources: [\"replicasets\", \"daemonsets\"]\n verbs: [\"watch\", \"list\", \"get\"]\n - apiGroups: [\"policy\"]\n resources: [\"poddisruptionbudgets\"]\n verbs: [\"watch\", \"list\"]\n - apiGroups: [\"apps\"]\n resources: [\"statefulsets\", \"replicasets\", \"daemonsets\"]\n verbs: [\"watch\", \"list\", \"get\"]\n - apiGroups: [\"storage.k8s.io\"]\n resources: [\"storageclasses\", \"csinodes\", \"csistoragecapacities\", \"csidrivers\", \"volumeattachments\"]\n verbs: [\"watch\", \"list\", \"get\"]\n - apiGroups: [\"batch\", \"extensions\"]\n resources: [\"jobs\"]\n verbs: [\"get\", \"list\", \"watch\", \"patch\"]\n - apiGroups: [\"coordination.k8s.io\"]\n resources: [\"leases\"]\n verbs: [\"create\"]\n - apiGroups: [\"coordination.k8s.io\"]\n resourceNames: [\"cluster-autoscaler\"]\n resources: [\"leases\"]\n verbs: [\"get\", \"update\"]\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n name: cluster-autoscaler\n namespace: kube-system\n labels:\n k8s-addon: cluster-autoscaler.addons.k8s.io\n k8s-app: cluster-autoscaler\nrules:\n - apiGroups: [\"\"]\n resources: [\"configmaps\"]\n verbs: [\"create\",\"list\",\"watch\"]\n - apiGroups: [\"\"]\n resources: [\"configmaps\"]\n resourceNames: [\"cluster-autoscaler-status\", \"cluster-autoscaler-priority-expander\"]\n verbs: [\"delete\", \"get\", \"update\", \"watch\"]\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n name: cluster-autoscaler\n labels:\n k8s-addon: cluster-autoscaler.addons.k8s.io\n k8s-app: cluster-autoscaler\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: cluster-autoscaler\nsubjects:\n - kind: ServiceAccount\n name: cluster-autoscaler\n namespace: kube-system\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n name: cluster-autoscaler\n namespace: kube-system\n labels:\n k8s-addon: cluster-autoscaler.addons.k8s.io\n k8s-app: cluster-autoscaler\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: Role\n name: cluster-autoscaler\nsubjects:\n - kind: ServiceAccount\n name: cluster-autoscaler\n namespace: kube-system\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: cluster-autoscaler\n namespace: kube-system\n labels:\n app: cluster-autoscaler\nspec:\n replicas: ${ca_replicas}\n selector:\n matchLabels:\n app: cluster-autoscaler\n template:\n metadata:\n labels:\n app: cluster-autoscaler\n annotations:\n prometheus.io/scrape: 'true'\n prometheus.io/port: '8085'\n spec:\n serviceAccountName: cluster-autoscaler\n tolerations:\n - effect: NoSchedule\n key: node-role.kubernetes.io/control-plane\n\n # Node affinity is used to force cluster-autoscaler to stick\n # to the control-plane node. This allows the cluster to reliably downscale\n # to zero worker nodes when needed.\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n - key: node-role.kubernetes.io/control-plane\n operator: Exists\n containers:\n - image: ${ca_image}:${ca_version}\n name: cluster-autoscaler\n %{~ if ca_resource_limits ~}\n resources:\n limits:\n cpu: ${ca_resources.limits.cpu}\n memory: ${ca_resources.limits.memory}\n requests:\n cpu: ${ca_resources.requests.cpu}\n memory: ${ca_resources.requests.memory}\n %{~ endif ~}\n ports:\n - containerPort: 8085\n command:\n - ./cluster-autoscaler\n - --v=${cluster_autoscaler_log_level}\n - --logtostderr=${cluster_autoscaler_log_to_stderr}\n - --stderrthreshold=${cluster_autoscaler_stderr_threshold}\n - --cloud-provider=hetzner\n %{~ for pool in node_pools ~}\n - --nodes=${pool.min_nodes}:${pool.max_nodes}:${pool.server_type}:${pool.location}:${cluster_name}${pool.name}\n %{~ endfor ~}\n %{~ for extra_arg in cluster_autoscaler_extra_args ~}\n - ${extra_arg}\n %{~ endfor ~}\n env:\n - name: HCLOUD_TOKEN\n valueFrom:\n secretKeyRef:\n name: hcloud\n key: token\n - name: HCLOUD_CLOUD_INIT\n value: ${cloudinit_config}\n - name: HCLOUD_CLUSTER_CONFIG\n value: ${cluster_config}\n - name: HCLOUD_SSH_KEY\n value: '${ssh_key}'\n - name: HCLOUD_IMAGE\n value: '${snapshot_id}'\n - name: HCLOUD_NETWORK\n value: '${ipv4_subnet_id}'\n - name: HCLOUD_FIREWALL\n value: '${firewall_id}'\n - name: HCLOUD_PUBLIC_IPV4\n value: '${enable_ipv4}'\n - name: HCLOUD_PUBLIC_IPV6\n value: '${enable_ipv6}'\n %{~ if cluster_autoscaler_server_creation_timeout != \"\" ~}\n - name: HCLOUD_SERVER_CREATION_TIMEOUT\n value: '${cluster_autoscaler_server_creation_timeout}'\n %{~ endif ~}\n volumeMounts:\n - name: ssl-certs\n mountPath: /etc/ssl/certs\n readOnly: true\n imagePullPolicy: \"Always\"\n volumes:\n - name: ssl-certs\n hostPath:\n path: \"/etc/ssl/certs\" # right place on MicroOS?\n" }, { "path": "templates/calico.yaml.tpl", "content": "${values}\n" }, { "path": "templates/ccm.yaml.tpl", "content": "---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: hcloud-cloud-controller-manager\n namespace: kube-system\nspec:\n template:\n spec:\n containers:\n - name: hcloud-cloud-controller-manager\n args:\n - \"--cloud-provider=hcloud\"\n - \"--leader-elect=false\"\n - \"--allow-untagged-cloud\"\n - \"--route-reconciliation-period=30s\"\n - \"--allocate-node-cidrs=true\"\n - \"--cluster-cidr=${cluster_cidr_ipv4}\"\n - \"--webhook-secure-port=0\"\n%{if using_klipper_lb~}\n - \"--secure-port=10288\"\n%{endif~}\n env:\n - name: \"HCLOUD_LOAD_BALANCERS_LOCATION\"\n value: \"${default_lb_location}\"\n - name: \"HCLOUD_LOAD_BALANCERS_USE_PRIVATE_IP\"\n value: \"true\"\n - name: \"HCLOUD_LOAD_BALANCERS_ENABLED\"\n value: \"${!using_klipper_lb}\"\n - name: \"HCLOUD_LOAD_BALANCERS_DISABLE_PRIVATE_INGRESS\"\n value: \"true\"\n" }, { "path": "templates/cert_manager.yaml.tpl", "content": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: cert-manager\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: cert-manager\n namespace: kube-system\nspec:\n chart: cert-manager\n repo: https://charts.jetstack.io\n version: \"${version}\"\n targetNamespace: cert-manager\n bootstrap: ${bootstrap}\n valuesContent: |-\n ${values}\n" }, { "path": "templates/cilium.yaml.tpl", "content": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: cilium\n namespace: kube-system\nspec:\n chart: cilium\n repo: https://helm.cilium.io/\n version: \"${version}\"\n targetNamespace: kube-system\n bootstrap: true\n valuesContent: |-\n ${values}\n" }, { "path": "templates/csi-driver-smb.yaml.tpl", "content": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: csi-driver-smb\n namespace: kube-system\nspec:\n chart: csi-driver-smb\n repo: https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts\n version: \"${version}\"\n targetNamespace: kube-system\n bootstrap: ${bootstrap}\n valuesContent: |-\n ${values}\n" }, { "path": "templates/haproxy_ingress.yaml.tpl", "content": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${target_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: haproxy\n namespace: kube-system\nspec:\n chart: kubernetes-ingress\n version: \"${version}\"\n repo: https://haproxytech.github.io/helm-charts\n targetNamespace: ${target_namespace}\n bootstrap: true\n valuesContent: |-\n ${values}\n" }, { "path": "templates/hcloud-ccm-helm.yaml.tpl", "content": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: hcloud-cloud-controller-manager\n namespace: kube-system\nspec:\n chart: hcloud-cloud-controller-manager\n repo: https://charts.hetzner.cloud\n version: \"${version}\"\n targetNamespace: kube-system\n bootstrap: true\n valuesContent: |-\n ${values}\n" }, { "path": "templates/hcloud-csi.yaml.tpl", "content": "---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: hcloud-csi\n namespace: kube-system\nspec:\n chart: hcloud-csi\n repo: https://charts.hetzner.cloud\n version: \"${version}\"\n targetNamespace: kube-system\n bootstrap: true\n valuesContent: |-\n ${values}\n" }, { "path": "templates/kube-hetzner-selinux.te", "content": "module kube_hetzner_selinux 1.0;\n\nrequire {\n type kernel_t, bin_t, kernel_generic_helper_t, iscsid_t, iscsid_exec_t, var_run_t, var_lib_t,\n init_t, unlabeled_t, systemd_logind_t, systemd_hostnamed_t, container_t,\n cert_t, container_var_lib_t, etc_t, usr_t, container_file_t, container_log_t,\n container_share_t, container_runtime_exec_t, container_runtime_t, var_log_t, proc_t, io_uring_t, fuse_device_t, http_port_t,\n container_var_run_t, fixed_disk_device_t, removable_device_t;\n class key { read view };\n class file { open read execute execute_no_trans create link lock rename write append setattr unlink getattr watch };\n class sock_file { watch write create unlink };\n class unix_dgram_socket create;\n class unix_stream_socket { connectto read write };\n class dir { add_name create getattr link lock read rename remove_name reparent rmdir setattr unlink search write watch };\n class lnk_file { read create };\n class system module_request;\n class filesystem associate;\n class bpf map_create;\n class io_uring sqpoll;\n class anon_inode { create map read write };\n class tcp_socket name_connect;\n class chr_file { open read write };\n class blk_file getattr;\n}\n\n#============= kernel_generic_helper_t ==============\nallow kernel_generic_helper_t bin_t:file execute_no_trans;\nallow kernel_generic_helper_t kernel_t:key { read view };\nallow kernel_generic_helper_t self:unix_dgram_socket create;\n\n#============= iscsid_t ==============\nallow iscsid_t iscsid_exec_t:file execute;\nallow iscsid_t var_run_t:sock_file write;\nallow iscsid_t var_run_t:unix_stream_socket connectto;\n\n#============= init_t ==============\nallow init_t unlabeled_t:dir { add_name remove_name rmdir search };\nallow init_t unlabeled_t:lnk_file create;\nallow init_t container_t:file { open read };\nallow init_t container_file_t:file { execute execute_no_trans };\nallow init_t fuse_device_t:chr_file { open read write };\nallow init_t http_port_t:tcp_socket name_connect;\n\n#============= systemd_logind_t ==============\nallow systemd_logind_t unlabeled_t:dir search;\n\n#============= systemd_hostnamed_t ==============\nallow systemd_hostnamed_t unlabeled_t:dir search;\n\n#============= container_t ==============\nallow container_t { cert_t container_log_t }:dir read;\nallow container_t { cert_t container_log_t }:lnk_file read;\nallow container_t cert_t:file { read open };\nallow container_t container_var_lib_t:dir { add_name remove_name write read create };\nallow container_t container_var_lib_t:file { append create open read write rename lock setattr getattr unlink };\nallow container_t container_var_lib_t:sock_file write;\nallow container_t etc_t:dir { add_name remove_name write create setattr watch };\nallow container_t etc_t:file { create setattr unlink write };\nallow container_t etc_t:sock_file { create unlink };\nallow container_t usr_t:dir { add_name create getattr link lock read rename remove_name reparent rmdir setattr unlink search write };\nallow container_t usr_t:file { append create execute getattr link lock read rename setattr unlink write };\nallow container_t container_file_t:file { open read write append getattr setattr lock };\nallow container_t container_file_t:sock_file watch;\nallow container_t container_log_t:file { open read write append getattr setattr watch };\nallow container_t container_share_t:dir { read write add_name remove_name };\nallow container_t container_share_t:file { read write create unlink };\nallow container_t container_runtime_exec_t:file { read execute execute_no_trans open };\nallow container_t container_runtime_t:unix_stream_socket { connectto read write };\nallow container_t kernel_t:system module_request;\nallow container_t var_log_t:dir { add_name write remove_name watch read };\nallow container_t var_log_t:file { create lock open read setattr write unlink getattr };\nallow container_t var_lib_t:dir { add_name remove_name write read create };\nallow container_t var_lib_t:file { append create open read write rename lock setattr getattr unlink };\nallow container_t proc_t:filesystem associate;\nallow container_t self:bpf map_create;\nallow container_t self:io_uring sqpoll;\nallow container_t io_uring_t:anon_inode { create map read write };\nallow container_t container_var_run_t:dir { add_name remove_name write };\nallow container_t container_var_run_t:file { create open read rename unlink write };\nallow container_t fixed_disk_device_t:blk_file getattr;\nallow container_t removable_device_t:blk_file getattr;\n" }, { "path": "templates/kube_system_secrets.yaml.tpl", "content": "%{ for secret_name, secret_values in kube_system_secrets ~}\napiVersion: v1\nkind: Secret\nmetadata:\n name: ${secret_name}\n namespace: kube-system\ntype: Opaque\ndata:\n%{ for secret_key, secret_value in secret_values ~}\n ${secret_key}: ${base64encode(secret_value)}\n%{ endfor ~}\n---\n%{ endfor ~}\n" }, { "path": "templates/kured.yaml.tpl", "content": "---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: kured\n namespace: kube-system\nspec:\n selector:\n matchLabels:\n name: kured\n template:\n metadata:\n labels:\n name: kured\n spec:\n serviceAccountName: kured\n containers:\n - name: kured\n command:\n - /usr/bin/kured\n %{~ for key, value in options ~}\n - --${key}=${value}\n %{~ endfor ~}\n" }, { "path": "templates/longhorn.yaml.tpl", "content": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${longhorn_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: longhorn\n namespace: kube-system\nspec:\n chart: longhorn\n repo: ${longhorn_repository}\n version: \"${version}\"\n targetNamespace: ${longhorn_namespace}\n bootstrap: ${bootstrap}\n valuesContent: |-\n ${values}\n" }, { "path": "templates/nat-router-cloudinit.yaml.tpl", "content": "#cloud-config\npackage_reboot_if_required: false\npackage_update: true\npackage_upgrade: true\npackages: \n- fail2ban\n%{ if enable_redundancy ~}\n- jq\n- keepalived\n%{ endif ~}\n\nwrite_files:\n - path: /etc/network/interfaces\n content: |\n auto eth0\n iface eth0 inet dhcp\n post-up echo 1 > /proc/sys/net/ipv4/ip_forward\n post-up iptables -t nat -A POSTROUTING -s '${ private_network_ipv4_range }' ! -d '${ private_network_ipv4_range }' -o eth0 -j MASQUERADE\n%{ if enable_cp_lb_port_forward ~}\n post-up iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 6443 -j DNAT --to-destination ${ cp_lb_private_ip }:6443\n post-up iptables -t nat -A POSTROUTING -d ${ cp_lb_private_ip } -p tcp --dport 6443 -j MASQUERADE\n%{ endif ~}\n append: true\n\n # Disable ssh password authentication\n - content: |\n Port ${ ssh_port }\n PasswordAuthentication no\n X11Forwarding no\n MaxAuthTries ${ ssh_max_auth_tries }\n AllowTcpForwarding yes\n AllowAgentForwarding yes\n AuthorizedKeysFile .ssh/authorized_keys\n%{ if enable_sudo ~}\n PermitRootLogin no\n%{ endif ~}\n path: /etc/ssh/sshd_config.d/kube-hetzner.conf\n - path: /etc/fail2ban/jail.d/sshd.local\n content: |\n [sshd]\n enabled = true\n port = ssh\n logpath = %(sshd_log)s\n maxretry = 5\n bantime = 86400\n\n%{ if enable_redundancy ~}\n # We might run into race condition of keepalived starting and the private subnet ip being attached to the interface\n # in that case, keepalived would go into fault state if we don't wait for it\n - path: /usr/local/bin/wait-for-ip.sh\n permissions: '0700'\n content: |\n #!/bin/bash\n TARGET_IP=\"${ my_private_ip }\"\n\n echo \"Waiting for $TARGET_IP to appear on private interface...\"\n while true; do\n INTERFACE=$(ip -o -4 addr show | awk -v target_ip=\"$TARGET_IP\" '$4 ~ (\"^\" target_ip \"/\") {print $2; exit}')\n if [ -n \"$INTERFACE\" ]; then\n break\n fi\n sleep 1\n done\n\n sed \"s/__NAT_PRIVATE_IFACE__/$INTERFACE/g\" /etc/keepalived/keepalived.conf.tmpl > /etc/keepalived/keepalived.conf\n echo \"Private interface is $INTERFACE, keepalived config rendered.\"\n - path: /etc/systemd/system/wait-for-private-ip.service\n content: |\n [Unit]\n Description=Wait for Private Network IP\n After=network.target\n Before=keepalived.service\n\n [Service]\n Type=oneshot\n ExecStart=/usr/local/bin/wait-for-ip.sh\n RemainAfterExit=yes\n\n [Install]\n WantedBy=multi-user.target\n - path: /etc/systemd/system/keepalived.service.d/override.conf\n content: |\n [Unit]\n # Require the specific IP wait service\n Requires=wait-for-private-ip.service\n After=wait-for-private-ip.service\n - path: /etc/keepalived/keepalived.conf.tmpl\n content: |\n global_defs {\n enable_script_security\n script_user keepalived_script\n max_auto_priority\n }\n\n vrrp_script check_internet {\n script \"/usr/local/bin/check_wan.sh\"\n interval 2\n fall 3\n rise 3\n }\n\n vrrp_instance VI_NAT {\n state BACKUP\n interface __NAT_PRIVATE_IFACE__\n virtual_router_id 51\n priority ${ priority }\n advert_int 1\n nopreempt\n unicast_src_ip ${ my_private_ip }\n unicast_peer {\n ${ peer_private_ip }\n }\n virtual_ipaddress {\n ${ vip } dev __NAT_PRIVATE_IFACE__\n }\n authentication {\n auth_type PASS\n auth_pass ${ vip_auth_pass }\n }\n\n track_script {\n check_internet\n }\n\n # Call Hetzner API for alias IP change\n notify_master \"/usr/local/bin/hcloud-alias-failover.sh\"\n }\n - path: /usr/local/bin/check_wan.sh\n owner: keepalived_script:keepalived_script\n defer: true\n permissions: '0744'\n content: |\n #!/bin/bash\n\n /usr/bin/ping -W 1 -c 1 8.8.8.8 >/dev/null 2>/dev/null\n GOOGLE_PING=$?\n /usr/bin/ping -W 1 -c 1 1.1.1.1 >/dev/null 2>/dev/null\n CF_PING=$?\n\n if [ $GOOGLE_PING -ne 0 ] && [ $CF_PING -ne 0 ]\n then\n exit 1\n else\n exit 0\n fi\n\n - path: /usr/local/bin/hcloud-alias-failover.sh\n owner: keepalived_script:keepalived_script\n defer: true\n permissions: '0700'\n content: |\n #!/bin/bash\n set -euo pipefail\n\n # Load environment variables, including HCLOUD_TOKEN\n ENV_FILE=\"/etc/keepalived/hcloud.env\"\n if [ -f \"$ENV_FILE\" ]\n then\n source \"$ENV_FILE\"\n else\n exit 1\n fi\n\n NET_ID=\"${ network_id }\"\n VIP=\"${ vip }\"\n PEER_IP=\"${ peer_private_ip }\"\n\n # Get own hcloud server id by calling metadata service\n MY_ID=$(curl -f -s http://169.254.169.254/hetzner/v1/metadata/instance-id)\n\n if [ -z \"$MY_ID\" ]\n then\n exit 1\n fi\n\n # Get peer id by server list filtered by role=nat_router and provided peer IP\n PEER_ID=$(curl -f -s -H \"Authorization: Bearer $HCLOUD_TOKEN\" \\\n \"https://api.hetzner.cloud/v1/servers?label_selector=role=nat_router\" | \\\n jq -r --arg peer_ip \"$PEER_IP\" --arg net_id \"$NET_ID\" '.servers[] | select(any(.private_net[]; .ip == $peer_ip and (.network | tostring) == $net_id)) | .id' | head -n 1)\n\n if [ -z \"$PEER_ID\" ] || [ \"$PEER_ID\" = \"null\" ]\n then\n exit 1\n fi\n\n # Remove from Peer\n curl -f -s -X POST \"https://api.hetzner.cloud/v1/servers/$PEER_ID/actions/change_alias_ips\" \\\n -H \"Authorization: Bearer $HCLOUD_TOKEN\" -H \"Content-Type: application/json\" \\\n -d \"{\\\"network\\\": $NET_ID, \\\"alias_ips\\\": []}\"\n\n # Assign to Me\n curl -f -s -X POST \"https://api.hetzner.cloud/v1/servers/$MY_ID/actions/change_alias_ips\" \\\n -H \"Authorization: Bearer $HCLOUD_TOKEN\" -H \"Content-Type: application/json\" \\\n -d \"{\\\"network\\\": $NET_ID, \\\"alias_ips\\\": [\\\"$VIP\\\"]}\"\n\n - path: /etc/keepalived/hcloud.env\n owner: keepalived_script:keepalived_script\n permissions: '0600'\n defer: true\n content: |\n export HCLOUD_TOKEN=\"${ hcloud_token }\"\n%{ endif ~}\n\nusers:\n - name: nat-router\n groups:\n%{ if enable_sudo ~}\n - sudo\n%{ endif ~}\n%{ if enable_sudo ~}\n sudo:\n - ALL=(ALL) NOPASSWD:ALL\n%{ endif ~}\n# Add ssh authorized keys\n ssh_authorized_keys:\n%{ for key in sshAuthorizedKeys ~}\n - ${key}\n%{ endfor ~}\n%{ if enable_redundancy ~}\n - name: keepalived_script\n%{ endif ~}\n\n# Apply DNS config\n%{ if has_dns_servers ~}\nmanage_resolv_conf: true\nresolv_conf:\n nameservers:\n%{ for dns_server in dns_servers ~}\n - ${dns_server}\n%{ endfor ~}\n%{ endif ~}\n\n\nruncmd:\n - [systemctl, 'enable', 'fail2ban']\n - [systemctl, 'restart', 'sshd']\n - [systemctl, 'restart', 'networking']\n%{ if enable_redundancy ~}\n - [systemctl, 'enable', 'keepalived']\n - [systemctl, 'restart', 'keepalived']\n%{ endif ~}\n" }, { "path": "templates/nginx_ingress.yaml.tpl", "content": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${target_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: nginx\n namespace: kube-system\nspec:\n chart: ingress-nginx\n version: \"${version}\"\n repo: https://kubernetes.github.io/ingress-nginx\n targetNamespace: ${target_namespace}\n bootstrap: true\n valuesContent: |-\n ${values}\n" }, { "path": "templates/plans.yaml.tpl", "content": "---\n# Doc: https://rancher.com/docs/k3s/latest/en/upgrades/automated/\n# agent plan\napiVersion: upgrade.cattle.io/v1\nkind: Plan\nmetadata:\n name: k3s-agent\n namespace: system-upgrade\n labels:\n k3s_upgrade: agent\nspec:\n concurrency: 1\n %{~ if version == \"\" ~}\n channel: https://update.k3s.io/v1-release/channels/${channel}\n %{~ else ~}\n version: ${version}\n %{~ endif ~}\n serviceAccountName: system-upgrade\n nodeSelector:\n matchExpressions:\n - {key: k3s_upgrade, operator: Exists}\n - {key: k3s_upgrade, operator: NotIn, values: [\"disabled\", \"false\"]}\n - {key: node-role.kubernetes.io/control-plane, operator: NotIn, values: [\"true\"]}\n - {key: kured, operator: NotIn, values: [\"rebooting\"]}\n tolerations:\n - {key: server-usage, effect: NoSchedule, operator: Equal, value: storage}\n - {operator: Exists}\n prepare:\n image: rancher/k3s-upgrade\n args: [\"prepare\", \"k3s-server\"]\n %{ if drain }drain:\n force: true\n disableEviction: ${disable_eviction}\n skipWaitForDeleteTimeout: 60%{ endif }\n %{ if !drain }cordon: true%{ endif }\n %{~ if upgrade_window != null ~}\n window:\n %{~ if length(try(upgrade_window.days, [])) > 0 ~}\n days:\n %{~ for day in try(upgrade_window.days, []) ~}\n - ${jsonencode(day)}\n %{~ endfor ~}\n %{~ endif ~}\n %{~ if coalesce(try(upgrade_window.startTime, \"\"), \"\") != \"\" ~}\n startTime: ${jsonencode(coalesce(try(upgrade_window.startTime, \"\"), \"\"))}\n %{~ endif ~}\n %{~ if coalesce(try(upgrade_window.endTime, \"\"), \"\") != \"\" ~}\n endTime: ${jsonencode(coalesce(try(upgrade_window.endTime, \"\"), \"\"))}\n %{~ endif ~}\n %{~ if coalesce(try(upgrade_window.timeZone, \"\"), \"\") != \"\" ~}\n timeZone: ${jsonencode(coalesce(try(upgrade_window.timeZone, \"\"), \"\"))}\n %{~ endif ~}\n %{~ endif ~}\n upgrade:\n image: rancher/k3s-upgrade\n---\n# server plan\napiVersion: upgrade.cattle.io/v1\nkind: Plan\nmetadata:\n name: k3s-server\n namespace: system-upgrade\n labels:\n k3s_upgrade: server\nspec:\n concurrency: 1\n %{~ if version == \"\" ~}\n channel: https://update.k3s.io/v1-release/channels/${channel}\n %{~ else ~}\n version: ${version}\n %{~ endif ~}\n serviceAccountName: system-upgrade\n nodeSelector:\n matchExpressions:\n - {key: k3s_upgrade, operator: Exists}\n - {key: k3s_upgrade, operator: NotIn, values: [\"disabled\", \"false\"]}\n - {key: node-role.kubernetes.io/control-plane, operator: In, values: [\"true\"]}\n - {key: kured, operator: NotIn, values: [\"rebooting\"]}\n tolerations:\n - {key: node-role.kubernetes.io/control-plane, effect: NoSchedule, operator: Exists}\n - {key: CriticalAddonsOnly, effect: NoExecute, operator: Exists}\n cordon: true\n %{~ if upgrade_window != null ~}\n window:\n %{~ if length(try(upgrade_window.days, [])) > 0 ~}\n days:\n %{~ for day in try(upgrade_window.days, []) ~}\n - ${jsonencode(day)}\n %{~ endfor ~}\n %{~ endif ~}\n %{~ if coalesce(try(upgrade_window.startTime, \"\"), \"\") != \"\" ~}\n startTime: ${jsonencode(coalesce(try(upgrade_window.startTime, \"\"), \"\"))}\n %{~ endif ~}\n %{~ if coalesce(try(upgrade_window.endTime, \"\"), \"\") != \"\" ~}\n endTime: ${jsonencode(coalesce(try(upgrade_window.endTime, \"\"), \"\"))}\n %{~ endif ~}\n %{~ if coalesce(try(upgrade_window.timeZone, \"\"), \"\") != \"\" ~}\n timeZone: ${jsonencode(coalesce(try(upgrade_window.timeZone, \"\"), \"\"))}\n %{~ endif ~}\n %{~ endif ~}\n upgrade:\n image: rancher/k3s-upgrade\n" }, { "path": "templates/rancher.yaml.tpl", "content": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: cattle-system\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: rancher\n namespace: kube-system\nspec:\n chart: rancher\n repo: https://releases.rancher.com/server-charts/${rancher_install_channel}\n version: \"${version}\"\n targetNamespace: cattle-system\n bootstrap: ${bootstrap}\n valuesContent: |-\n ${values}\n" }, { "path": "templates/traefik_ingress.yaml.tpl", "content": "---\napiVersion: v1\nkind: Namespace\nmetadata:\n name: ${target_namespace}\n---\napiVersion: helm.cattle.io/v1\nkind: HelmChart\nmetadata:\n name: traefik\n namespace: kube-system\nspec:\n chart: traefik\n version: \"${version}\"\n repo: https://traefik.github.io/charts\n targetNamespace: ${target_namespace}\n bootstrap: true\n valuesContent: |-\n ${values}\n" }, { "path": "values-export.tf", "content": "resource \"local_file\" \"cilium_values\" {\n count = var.export_values && var.cni_plugin == \"cilium\" ? 1 : 0\n content = local.cilium_values\n filename = \"cilium_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"cert_manager_values\" {\n count = var.export_values && var.enable_cert_manager ? 1 : 0\n content = local.cert_manager_values\n filename = \"cert_manager_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"hetzner_ccm_values\" {\n count = var.export_values && var.hetzner_ccm_use_helm ? 1 : 0\n content = local.hetzner_ccm_values\n filename = \"hetzner_ccm_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"csi_driver_smb_values\" {\n count = var.export_values && var.enable_csi_driver_smb ? 1 : 0\n content = local.csi_driver_smb_values\n filename = \"csi_driver_smb_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"longhorn_values\" {\n count = var.export_values && var.enable_longhorn ? 1 : 0\n content = local.longhorn_values\n filename = \"longhorn_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"traefik_values\" {\n count = var.export_values && var.ingress_controller == \"traefik\" ? 1 : 0\n content = local.traefik_values\n filename = \"traefik_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"nginx_values\" {\n count = var.export_values && var.ingress_controller == \"nginx\" ? 1 : 0\n content = local.nginx_values\n filename = \"nginx_values.yaml\"\n file_permission = \"600\"\n}\n\nresource \"local_file\" \"haproxy_values\" {\n count = var.export_values && var.ingress_controller == \"haproxy\" ? 1 : 0\n content = local.haproxy_values\n filename = \"haproxy_values.yaml\"\n file_permission = \"600\"\n}\n" }, { "path": "values-merger.tf", "content": "module \"values_merger_cilium\" {\n source = \"./modules/values_merger\"\n default_values = local.cilium_values_default\n override_values = var.cilium_values\n merge_values = var.cilium_merge_values\n}\n\nmodule \"values_merger_longhorn\" {\n source = \"./modules/values_merger\"\n default_values = local.longhorn_values_default\n override_values = var.longhorn_values\n merge_values = var.longhorn_merge_values\n}\n\nmodule \"values_merger_nginx\" {\n source = \"./modules/values_merger\"\n default_values = local.nginx_values_default\n override_values = var.nginx_values\n merge_values = var.nginx_merge_values\n}\n\nmodule \"values_merger_hetzner_ccm\" {\n source = \"./modules/values_merger\"\n default_values = local.hetzner_ccm_values_default\n override_values = var.hetzner_ccm_values\n merge_values = var.hetzner_ccm_merge_values\n}\n\nmodule \"values_merger_haproxy\" {\n source = \"./modules/values_merger\"\n default_values = local.haproxy_values_default\n override_values = var.haproxy_values\n merge_values = var.haproxy_merge_values\n}\n\nmodule \"values_merger_traefik\" {\n source = \"./modules/values_merger\"\n default_values = local.traefik_values_default\n override_values = var.traefik_values\n merge_values = var.traefik_merge_values\n}\n\nmodule \"values_merger_rancher\" {\n source = \"./modules/values_merger\"\n default_values = local.rancher_values_default\n override_values = var.rancher_values\n merge_values = var.rancher_merge_values\n}\n\nmodule \"values_merger_cert_manager\" {\n source = \"./modules/values_merger\"\n default_values = local.cert_manager_values_default\n override_values = var.cert_manager_values\n merge_values = var.cert_manager_merge_values\n}\n" }, { "path": "variables.tf", "content": "variable \"hcloud_token\" {\n description = \"Hetzner Cloud API Token.\"\n type = string\n sensitive = true\n}\n\nvariable \"k3s_token\" {\n description = \"k3s master token (must match when restoring a cluster).\"\n type = string\n sensitive = true\n default = null\n}\n\nvariable \"robot_user\" {\n type = string\n default = \"\"\n sensitive = true\n description = \"User for the Hetzner Robot webservice\"\n}\n\nvariable \"robot_password\" {\n type = string\n default = \"\"\n sensitive = true\n description = \"Password for the Hetzner Robot webservice\"\n}\n\nvariable \"robot_ccm_enabled\" {\n type = bool\n default = false\n description = \"Enables the integration of Hetzner Robot dedicated servers via the Cloud Controller Manager (CCM). If true, `robot_user` and `robot_password` must also be provided, otherwise the integration will not be activated.\"\n}\n\nvariable \"microos_x86_snapshot_id\" {\n description = \"MicroOS x86 snapshot ID to be used. Per default empty, the most recent image created using createkh will be used\"\n type = string\n default = \"\"\n}\n\nvariable \"microos_arm_snapshot_id\" {\n description = \"MicroOS ARM snapshot ID to be used. Per default empty, the most recent image created using createkh will be used\"\n type = string\n default = \"\"\n}\n\nvariable \"ssh_port\" {\n description = \"The main SSH port to connect to the nodes.\"\n type = number\n default = 22\n\n validation {\n condition = var.ssh_port >= 0 && var.ssh_port <= 65535\n error_message = \"The SSH port must use a valid range from 0 to 65535.\"\n }\n}\n\nvariable \"ssh_public_key\" {\n description = \"SSH public Key.\"\n type = string\n}\n\nvariable \"ssh_private_key\" {\n description = \"SSH private Key.\"\n type = string\n sensitive = true\n}\n\nvariable \"ssh_hcloud_key_label\" {\n description = \"Additional SSH public Keys by hcloud label. e.g. role=admin\"\n type = string\n default = \"\"\n}\n\nvariable \"ssh_additional_public_keys\" {\n description = \"Additional SSH public Keys. Use them to grant other team members root access to your cluster nodes.\"\n type = list(string)\n default = []\n}\n\nvariable \"authentication_config\" {\n description = \"Strucutred authentication configuration. This can be used to define external authentication providers.\"\n type = string\n default = \"\"\n}\n\nvariable \"hcloud_ssh_key_id\" {\n description = \"If passed, a key already registered within hetzner is used. Otherwise, a new one will be created by the module.\"\n type = string\n default = null\n}\n\nvariable \"ssh_max_auth_tries\" {\n description = \"The maximum number of authentication attempts permitted per connection.\"\n type = number\n default = 2\n}\n\nvariable \"network_region\" {\n description = \"Default region for network.\"\n type = string\n default = \"eu-central\"\n}\nvariable \"existing_network_id\" {\n # Unfortunately, we need this to be a list or null. If we only use a plain\n # string here, and check that existing_network_id is null, terraform will\n # complain that it cannot set `count` variables based on existing_network_id\n # != null, because that id is an output value from\n # hcloud_network.your_network.id, which terraform will only know after its\n # construction.\n description = \"If you want to create the private network before calling this module, you can do so and pass its id here. NOTE: make sure to adapt network_ipv4_cidr accordingly to a range which does not collide with your other nodes.\"\n type = list(string)\n default = []\n nullable = false\n validation {\n condition = length(var.existing_network_id) == 0 || (can(var.existing_network_id[0]) && length(var.existing_network_id) == 1)\n error_message = \"If you pass an existing_network_id, it must be enclosed in square brackets: [id]. This is necessary to be able to unambiguously distinguish between an empty network id (default) and a user-supplied network id.\"\n }\n}\nvariable \"network_ipv4_cidr\" {\n description = \"The main network cidr that all subnets will be created upon.\"\n type = string\n default = \"10.0.0.0/8\"\n}\n\nvariable \"subnet_amount\" {\n description = \"The amount of subnets into which the network will be split. Must be a power of 2.\"\n type = number\n default = 256\n validation {\n condition = floor(log(var.subnet_amount, 2)) == log(var.subnet_amount, 2)\n error_message = \"Subnet amount must be a power of 2.\"\n }\n validation {\n # Host bits = 32 - prefix, must have enough bits to create subnet_amount subnets\n condition = pow(2, 32 - tonumber(split(\"/\", var.network_ipv4_cidr)[1])) >= var.subnet_amount\n error_message = \"The network CIDR is too small for the requested subnet amount. Reduce subnet_amount or use a larger network.\"\n }\n validation {\n condition = var.subnet_amount >= length(var.control_plane_nodepools) + length(var.agent_nodepools) + (var.nat_router == null ? 0 : (var.nat_router.enable_redundancy == false ? 1 : 2))\n error_message = \"Subnet amount must be large enough so that a subnet for each agent pool, each control plane pool and (if enabled) the nat router can be created in the network.\"\n }\n}\n\nvariable \"cluster_ipv4_cidr\" {\n description = \"Internal Pod CIDR, used for the controller and currently for calico/cilium.\"\n type = string\n default = \"10.42.0.0/16\"\n}\n\nvariable \"service_ipv4_cidr\" {\n description = \"Internal Service CIDR, used for the controller and currently for calico/cilium.\"\n type = string\n default = \"10.43.0.0/16\"\n}\n\nvariable \"cluster_dns_ipv4\" {\n description = \"Internal Service IPv4 address of core-dns.\"\n type = string\n default = null\n}\n\n\nvariable \"nat_router\" {\n description = \"Do you want to pipe all egress through a single nat router which is to be constructed? Note: Requires use_control_plane_lb=true when enabled. Automatically forwards port 6443 to the control plane LB when control_plane_lb_enable_public_interface=false.\"\n nullable = true\n default = null\n type = object({\n server_type = string\n location = string\n labels = optional(map(string), {})\n enable_sudo = optional(bool, false)\n enable_redundancy = optional(bool, false)\n standby_location = optional(string, \"\")\n })\n\n validation {\n condition = var.nat_router == null || !var.nat_router.enable_redundancy || var.nat_router.standby_location != \"\"\n error_message = \"When nat_router.enable_redundancy is true, standby_location must be provided.\"\n }\n}\n\nvariable \"nat_router_hcloud_token\" {\n description = \"API Token used by the nat-router to change ip assignment when nat_router.enable_redundancy is true.\"\n type = string\n default = \"\"\n sensitive = true\n\n validation {\n condition = var.nat_router == null || !var.nat_router.enable_redundancy || var.nat_router_hcloud_token != \"\"\n error_message = \"When nat_router.enable_redundancy is true, nat_router_hcloud_token must be provided.\"\n }\n}\n\nvariable \"nat_router_subnet_index\" {\n type = number\n default = 200\n description = \"Subnet index for NAT router. Default 200 is safe for most deployments. Must not conflict with control plane (counting down from 255) or agent pools (counting up from 0).\"\n\n validation {\n condition = var.nat_router_subnet_index >= 0 && var.nat_router_subnet_index < var.subnet_amount\n error_message = \"NAT router subnet index must be between 0 and subnet_amount.\"\n }\n}\n\nvariable \"vswitch_subnet_index\" {\n type = number\n default = 201\n description = \"Subnet index (0-255) for vSwitch. Default 201 is safe for most deployments. Must not conflict with control plane (counting down from 255) or agent pools (counting up from 0).\"\n\n validation {\n condition = var.vswitch_subnet_index >= 0 && var.vswitch_subnet_index <= 255\n error_message = \"vSwitch subnet index must be between 0 and 255.\"\n }\n}\n\nvariable \"vswitch_id\" {\n description = \"Hetzner Cloud vSwitch ID. If defined, a subnet will be created in the IP-range defined by vswitch_subnet_index. The vSwitch must exist before this module is called.\"\n type = number\n default = null\n}\n\nvariable \"load_balancer_location\" {\n description = \"Default load balancer location.\"\n type = string\n default = \"nbg1\"\n}\n\nvariable \"load_balancer_type\" {\n description = \"Default load balancer server type.\"\n type = string\n default = \"lb11\"\n}\n\nvariable \"load_balancer_disable_ipv6\" {\n description = \"Disable IPv6 for the load balancer.\"\n type = bool\n default = false\n}\n\nvariable \"load_balancer_disable_public_network\" {\n description = \"Disables the public network of the load balancer.\"\n type = bool\n default = false\n}\n\nvariable \"load_balancer_algorithm_type\" {\n description = \"Specifies the algorithm type of the load balancer.\"\n type = string\n default = \"round_robin\"\n}\n\nvariable \"load_balancer_health_check_interval\" {\n description = \"Specifies the interval at which a health check is performed. Minimum is 3s.\"\n type = string\n default = \"15s\"\n}\n\nvariable \"load_balancer_health_check_timeout\" {\n description = \"Specifies the timeout of a single health check. Must not be greater than the health check interval. Minimum is 1s.\"\n type = string\n default = \"10s\"\n}\n\nvariable \"load_balancer_health_check_retries\" {\n description = \"Specifies the number of times a health check is retried before a target is marked as unhealthy.\"\n type = number\n default = 3\n}\n\nvariable \"exclude_agents_from_external_load_balancers\" {\n description = \"Add node.kubernetes.io/exclude-from-external-load-balancers=true label to agent nodes. Enable this if you use both the Terraform-managed ingress LB and CCM-managed LoadBalancer services, and want to prevent double-registration of agents to the CCM LBs. Note: This excludes agents from ALL CCM-managed LoadBalancer services, not just ingress.\"\n type = bool\n default = false\n}\n\nvariable \"control_plane_nodepools\" {\n description = \"Number of control plane nodes.\"\n type = list(object({\n name = string\n server_type = string\n location = string\n backups = optional(bool)\n labels = list(string)\n taints = list(string)\n count = number\n swap_size = optional(string, \"\")\n zram_size = optional(string, \"\")\n kubelet_args = optional(list(string), [\"kube-reserved=cpu=250m,memory=1500Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])\n selinux = optional(bool, true)\n placement_group_compat_idx = optional(number, 0)\n placement_group = optional(string, null)\n disable_ipv4 = optional(bool, false)\n disable_ipv6 = optional(bool, false)\n network_id = optional(number, 0)\n }))\n default = []\n validation {\n condition = length(\n [for control_plane_nodepool in var.control_plane_nodepools : control_plane_nodepool.name]\n ) == length(\n distinct(\n [for control_plane_nodepool in var.control_plane_nodepools : control_plane_nodepool.name]\n )\n )\n error_message = \"Names in control_plane_nodepools must be unique.\"\n }\n validation {\n condition = length(var.control_plane_nodepools) > 0\n error_message = \"At least one control plane nodepool is required. Kubernetes cannot run without control plane nodes.\"\n }\n validation {\n condition = length(var.control_plane_nodepools) == 0 || sum([for v in var.control_plane_nodepools : v.count]) >= 1\n error_message = \"At least one control plane node is required (total count across all control_plane_nodepools must be >= 1).\"\n }\n}\n\nvariable \"agent_nodepools\" {\n description = \"Number of agent nodes.\"\n type = list(object({\n name = string\n server_type = string\n location = string\n backups = optional(bool)\n floating_ip = optional(bool)\n floating_ip_rdns = optional(string, null)\n labels = list(string)\n taints = list(string)\n longhorn_volume_size = optional(number)\n longhorn_mount_path = optional(string, \"/var/longhorn\")\n swap_size = optional(string, \"\")\n zram_size = optional(string, \"\")\n kubelet_args = optional(list(string), [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])\n selinux = optional(bool, true)\n placement_group_compat_idx = optional(number, 0)\n placement_group = optional(string, null)\n subnet_ip_range = optional(string, null)\n count = optional(number, null)\n disable_ipv4 = optional(bool, false)\n disable_ipv6 = optional(bool, false)\n network_id = optional(number, 0)\n nodes = optional(map(object({\n server_type = optional(string)\n location = optional(string)\n backups = optional(bool)\n floating_ip = optional(bool)\n floating_ip_rdns = optional(string, null)\n labels = optional(list(string))\n taints = optional(list(string))\n longhorn_volume_size = optional(number)\n longhorn_mount_path = optional(string, null)\n swap_size = optional(string, \"\")\n zram_size = optional(string, \"\")\n kubelet_args = optional(list(string), [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])\n selinux = optional(bool, true)\n placement_group_compat_idx = optional(number, 0)\n placement_group = optional(string, null)\n append_index_to_node_name = optional(bool, true)\n })))\n }))\n default = []\n\n validation {\n condition = length(\n [for agent_nodepool in var.agent_nodepools : agent_nodepool.name]\n ) == length(\n distinct(\n [for agent_nodepool in var.agent_nodepools : agent_nodepool.name]\n )\n )\n error_message = \"Names in agent_nodepools must be unique.\"\n }\n\n validation {\n condition = alltrue([for agent_nodepool in var.agent_nodepools : (agent_nodepool.count == null) != (agent_nodepool.nodes == null)])\n error_message = \"Set either nodes or count per agent_nodepool, not both.\"\n }\n\n\n validation {\n condition = alltrue([for agent_nodepool in var.agent_nodepools :\n alltrue([for agent_key, agent_node in coalesce(agent_nodepool.nodes, {}) : can(tonumber(agent_key)) && tonumber(agent_key) == floor(tonumber(agent_key)) && 0 <= tonumber(agent_key) && tonumber(agent_key) < 154])\n ])\n # 154 because the private ip is derived from tonumber(key) + 101. See private_ipv4 in agents.tf\n error_message = \"The key for each individual node in a nodepool must be a stable integer in the range [0, 153] cast as a string.\"\n }\n\n validation {\n condition = length(var.agent_nodepools) == 0 ? true : sum([for agent_nodepool in var.agent_nodepools : length(coalesce(agent_nodepool.nodes, {})) + coalesce(agent_nodepool.count, 0)]) <= 100\n # 154 because the private ip is derived from tonumber(key) + 101. See private_ipv4 in agents.tf\n error_message = \"Hetzner does not support networks with more than 100 servers.\"\n }\n\n validation {\n condition = alltrue(flatten([\n for np in var.agent_nodepools : concat(\n [\n can(regex(\"^/var/[a-zA-Z0-9._-]+(/[a-zA-Z0-9._-]+)*$\", np.longhorn_mount_path)) &&\n !contains(split(\"/\", np.longhorn_mount_path), \"..\") &&\n !contains(split(\"/\", np.longhorn_mount_path), \".\")\n ],\n [\n for node in values(coalesce(np.nodes, {})) : (\n node.longhorn_mount_path == null || (\n can(regex(\"^/var/[a-zA-Z0-9._-]+(/[a-zA-Z0-9._-]+)*$\", node.longhorn_mount_path)) &&\n !contains(split(\"/\", node.longhorn_mount_path), \"..\") &&\n !contains(split(\"/\", node.longhorn_mount_path), \".\")\n )\n )\n ]\n )\n ]))\n error_message = \"Each longhorn_mount_path must be a valid, absolute path within a subdirectory of '/var/', not contain '.' or '..' components, and not end with a slash. This applies to both nodepool-level and node-level settings.\"\n }\n\n}\n\nvariable \"cluster_autoscaler_image\" {\n type = string\n default = \"registry.k8s.io/autoscaling/cluster-autoscaler\"\n description = \"Image of Kubernetes Cluster Autoscaler for Hetzner Cloud to be used.\"\n}\n\nvariable \"cluster_autoscaler_version\" {\n type = string\n default = \"v1.33.3\"\n description = \"Version of Kubernetes Cluster Autoscaler for Hetzner Cloud. Should be aligned with Kubernetes version. Available versions for the official image can be found at https://explore.ggcr.dev/?repo=registry.k8s.io%2Fautoscaling%2Fcluster-autoscaler.\"\n}\n\nvariable \"cluster_autoscaler_log_level\" {\n description = \"Verbosity level of the logs for cluster-autoscaler\"\n type = number\n default = 4\n\n validation {\n condition = var.cluster_autoscaler_log_level >= 0 && var.cluster_autoscaler_log_level <= 5\n error_message = \"The log level must be between 0 and 5.\"\n }\n}\n\nvariable \"cluster_autoscaler_log_to_stderr\" {\n description = \"Determines whether to log to stderr or not\"\n type = bool\n default = true\n}\n\nvariable \"cluster_autoscaler_stderr_threshold\" {\n description = \"Severity level above which logs are sent to stderr instead of stdout\"\n type = string\n default = \"INFO\"\n\n validation {\n condition = var.cluster_autoscaler_stderr_threshold == \"INFO\" || var.cluster_autoscaler_stderr_threshold == \"WARNING\" || var.cluster_autoscaler_stderr_threshold == \"ERROR\" || var.cluster_autoscaler_stderr_threshold == \"FATAL\"\n error_message = \"The stderr threshold must be one of the following: INFO, WARNING, ERROR, FATAL.\"\n }\n}\n\nvariable \"cluster_autoscaler_extra_args\" {\n type = list(string)\n default = []\n description = \"Extra arguments for the Cluster Autoscaler deployment.\"\n}\n\nvariable \"cluster_autoscaler_server_creation_timeout\" {\n type = number\n default = 15\n description = \"Timeout (in minutes) until which a newly created server/node has to become available before giving up and destroying it.\"\n}\n\nvariable \"cluster_autoscaler_replicas\" {\n type = number\n default = 1\n description = \"Number of replicas for the cluster autoscaler deployment. Multiple replicas use leader election for HA.\"\n\n validation {\n condition = var.cluster_autoscaler_replicas >= 1 && floor(var.cluster_autoscaler_replicas) == var.cluster_autoscaler_replicas\n error_message = \"Number of cluster autoscaler replicas must be a positive integer.\"\n }\n}\n\nvariable \"cluster_autoscaler_resource_limits\" {\n type = bool\n default = true\n description = \"Should cluster autoscaler enable default resource requests and limits. Default values are requests: 100m & 300Mi and limits: 100m & 300Mi.\"\n}\n\nvariable \"cluster_autoscaler_resource_values\" {\n type = object({\n requests = object({\n cpu = string\n memory = string\n })\n limits = object({\n cpu = string\n memory = string\n })\n })\n default = {\n requests = {\n cpu = \"100m\"\n memory = \"300Mi\"\n }\n limits = {\n cpu = \"100m\"\n memory = \"300Mi\"\n }\n }\n description = \"Requests and limits for Cluster Autoscaler.\"\n}\n\nvariable \"autoscaler_nodepools\" {\n description = \"Cluster autoscaler nodepools.\"\n type = list(object({\n name = string\n server_type = string\n location = string\n min_nodes = number\n max_nodes = number\n labels = optional(map(string), {})\n kubelet_args = optional(list(string), [\"kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi\", \"system-reserved=cpu=250m,memory=300Mi\"])\n taints = optional(list(object({\n key = string\n value = string\n effect = string\n })), [])\n swap_size = optional(string, \"\")\n zram_size = optional(string, \"\")\n }))\n default = []\n}\n\nvariable \"autoscaler_labels\" {\n description = \"Labels for nodes created by the Cluster Autoscaler.\"\n type = list(string)\n default = []\n}\n\nvariable \"autoscaler_taints\" {\n description = \"Taints for nodes created by the Cluster Autoscaler.\"\n type = list(string)\n default = []\n}\n\nvariable \"autoscaler_disable_ipv4\" {\n description = \"Disable IPv4 on nodes created by the Cluster Autoscaler.\"\n type = bool\n default = false\n}\n\nvariable \"autoscaler_disable_ipv6\" {\n description = \"Disable IPv6 on nodes created by the Cluster Autoscaler.\"\n type = bool\n default = false\n}\n\nvariable \"hetzner_ccm_version\" {\n type = string\n default = null\n description = \"Version of Kubernetes Cloud Controller Manager for Hetzner Cloud. See https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases for the available versions.\"\n}\n\nvariable \"hetzner_ccm_use_helm\" {\n type = bool\n default = false\n description = \"Whether to use the helm chart for the Hetzner CCM or the legacy manifest which is the default.\"\n}\n\nvariable \"hetzner_csi_version\" {\n type = string\n default = null\n description = \"Version of Container Storage Interface driver for Hetzner Cloud. See https://github.com/hetznercloud/csi-driver/releases for the available versions.\"\n}\n\nvariable \"hetzner_csi_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to hetzner csi as 'valuesContent' at the HelmChart.\"\n}\n\n\nvariable \"restrict_outbound_traffic\" {\n type = bool\n default = true\n description = \"Whether or not to restrict the outbound traffic.\"\n}\n\nvariable \"enable_klipper_metal_lb\" {\n type = bool\n default = false\n description = \"Use klipper load balancer.\"\n}\n\nvariable \"etcd_s3_backup\" {\n description = \"Etcd cluster state backup to S3 storage\"\n type = map(any)\n sensitive = true\n default = {}\n}\n\nvariable \"ingress_controller\" {\n type = string\n default = \"traefik\"\n description = \"The name of the ingress controller.\"\n\n validation {\n condition = contains([\"traefik\", \"nginx\", \"haproxy\", \"none\"], var.ingress_controller)\n error_message = \"Must be one of \\\"traefik\\\" or \\\"nginx\\\" or \\\"haproxy\\\" or \\\"none\\\"\"\n }\n}\n\nvariable \"ingress_replica_count\" {\n type = number\n default = 0\n description = \"Number of replicas per ingress controller. 0 means autodetect based on the number of agent nodes.\"\n\n validation {\n condition = var.ingress_replica_count >= 0\n error_message = \"Number of ingress replicas can't be below 0.\"\n }\n}\n\nvariable \"ingress_max_replica_count\" {\n type = number\n default = 10\n description = \"Number of maximum replicas per ingress controller. Used for ingress HPA. Must be higher than number of replicas.\"\n\n validation {\n condition = var.ingress_max_replica_count >= 0\n error_message = \"Number of ingress maximum replicas can't be below 0.\"\n }\n}\n\nvariable \"traefik_image_tag\" {\n type = string\n default = \"\"\n description = \"Traefik image tag. Useful to use the beta version for new features. Example: v3.0.0-beta5\"\n}\n\nvariable \"traefik_autoscaling\" {\n type = bool\n default = true\n description = \"Should traefik enable Horizontal Pod Autoscaler.\"\n}\n\nvariable \"traefik_redirect_to_https\" {\n type = bool\n default = true\n description = \"Should traefik redirect http traffic to https.\"\n}\n\nvariable \"traefik_pod_disruption_budget\" {\n type = bool\n default = true\n description = \"Should traefik enable pod disruption budget. Default values are maxUnavailable: 33% and minAvailable: 1.\"\n}\n\nvariable \"traefik_provider_kubernetes_gateway_enabled\" {\n type = bool\n default = false\n description = \"Should traefik enable the kubernetes gateway provider. Default is false.\"\n}\n\nvariable \"traefik_resource_limits\" {\n type = bool\n default = true\n description = \"Should traefik enable default resource requests and limits. Default values are requests: 100m & 50Mi and limits: 300m & 150Mi.\"\n}\n\nvariable \"traefik_resource_values\" {\n type = object({\n requests = object({\n cpu = string\n memory = string\n })\n limits = object({\n cpu = string\n memory = string\n })\n })\n default = {\n requests = {\n memory = \"50Mi\"\n cpu = \"100m\"\n }\n limits = {\n memory = \"150Mi\"\n cpu = \"300m\"\n }\n }\n description = \"Requests and limits for Traefik.\"\n}\n\nvariable \"traefik_additional_ports\" {\n type = list(object({\n name = string\n port = number\n exposedPort = number\n }))\n default = []\n description = \"Additional ports to pass to Traefik. These are the ones that go into the ports section of the Traefik helm values file.\"\n}\n\nvariable \"traefik_additional_options\" {\n type = list(string)\n default = []\n description = \"Additional options to pass to Traefik as a list of strings. These are the ones that go into the additionalArguments section of the Traefik helm values file.\"\n}\n\nvariable \"traefik_additional_trusted_ips\" {\n type = list(string)\n default = []\n description = \"Additional Trusted IPs to pass to Traefik. These are the ones that go into the trustedIPs section of the Traefik helm values file.\"\n}\n\nvariable \"traefik_version\" {\n type = string\n default = \"\"\n description = \"Version of Traefik helm chart. See https://github.com/traefik/traefik-helm-chart/releases for the available versions.\"\n}\n\nvariable \"traefik_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to Traefik as 'valuesContent' at the HelmChart.\"\n}\n\nvariable \"traefik_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or traefik_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.traefik_merge_values == \"\" || can(yamldecode(var.traefik_merge_values))\n error_message = \"traefik_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"nginx_version\" {\n type = string\n default = \"\"\n description = \"Version of Nginx helm chart. See https://github.com/kubernetes/ingress-nginx?tab=readme-ov-file#supported-versions-table for the available versions.\"\n}\n\nvariable \"nginx_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to nginx as 'valuesContent' at the HelmChart.\"\n}\n\nvariable \"nginx_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or nginx_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.nginx_merge_values == \"\" || can(yamldecode(var.nginx_merge_values))\n error_message = \"nginx_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"haproxy_requests_cpu\" {\n type = string\n default = \"250m\"\n description = \"Setting for HAProxy controller.resources.requests.cpu\"\n}\n\nvariable \"haproxy_requests_memory\" {\n type = string\n default = \"400Mi\"\n description = \"Setting for HAProxy controller.resources.requests.memory\"\n}\n\nvariable \"haproxy_additional_proxy_protocol_ips\" {\n type = list(string)\n default = []\n description = \"Additional trusted proxy protocol IPs to pass to haproxy.\"\n}\n\nvariable \"haproxy_version\" {\n type = string\n default = \"\"\n description = \"Version of HAProxy helm chart.\"\n}\n\nvariable \"haproxy_values\" {\n type = string\n default = \"\"\n description = \"Helm values file to pass to haproxy as 'valuesContent' at the HelmChart, overriding the default.\"\n}\n\nvariable \"haproxy_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or haproxy_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.haproxy_merge_values == \"\" || can(yamldecode(var.haproxy_merge_values))\n error_message = \"haproxy_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"allow_scheduling_on_control_plane\" {\n type = bool\n default = false\n description = \"Whether to allow non-control-plane workloads to run on the control-plane nodes.\"\n}\n\nvariable \"enable_metrics_server\" {\n type = bool\n default = true\n description = \"Whether to enable or disable k3s metric server.\"\n}\n\nvariable \"initial_k3s_channel\" {\n type = string\n default = \"v1.33\" # Please update kube.tf.example too when changing this variable\n description = \"Allows you to specify an initial k3s channel. See https://update.k3s.io/v1-release/channels for available channels.\"\n\n validation {\n condition = contains([\"stable\", \"latest\", \"testing\", \"v1.16\", \"v1.17\", \"v1.18\", \"v1.19\", \"v1.20\", \"v1.21\", \"v1.22\", \"v1.23\", \"v1.24\", \"v1.25\", \"v1.26\", \"v1.27\", \"v1.28\", \"v1.29\", \"v1.30\", \"v1.31\", \"v1.32\", \"v1.33\", \"v1.34\", \"v1.35\"], var.initial_k3s_channel)\n error_message = \"The initial k3s channel must be one of stable, latest or testing, or any of the minor kube versions like v1.26.\"\n }\n}\n\nvariable \"install_k3s_version\" {\n type = string\n default = \"\"\n description = \"Allows you to specify the k3s version (Example: v1.29.6+k3s2). Supersedes initial_k3s_channel. See https://github.com/k3s-io/k3s/releases for available versions.\"\n}\n\nvariable \"system_upgrade_enable_eviction\" {\n type = bool\n default = true\n description = \"Whether to directly delete pods during system upgrade (k3s) or evict them. Defaults to true. Disable this on small clusters to avoid system upgrades hanging since pods resisting eviction keep node unschedulable forever. NOTE: turning this off, introduces potential downtime of services of the upgraded nodes.\"\n}\n\nvariable \"system_upgrade_use_drain\" {\n type = bool\n default = true\n description = \"Wether using drain (true, the default), which will deletes and transfers all pods to other nodes before a node is being upgraded, or cordon (false), which just prevents schedulung new pods on the node during upgrade and keeps all pods running\"\n}\n\nvariable \"automatically_upgrade_k3s\" {\n type = bool\n default = true\n description = \"Whether to automatically upgrade k3s based on the selected channel.\"\n}\n\nvariable \"system_upgrade_schedule_window\" {\n type = object({\n days = optional(list(string), [])\n startTime = optional(string, \"\")\n endTime = optional(string, \"\")\n timeZone = optional(string, \"UTC\")\n })\n default = null\n description = \"Schedule window for k3s automated upgrades (system-upgrade-controller v0.15.0+). When set, upgrade jobs will only be created within the specified time window. 'days' accepts lowercase day names (e.g. [\\\"monday\\\",\\\"tuesday\\\"]). 'startTime'/'endTime' use HH:MM format. 'timeZone' defaults to UTC. See https://docs.k3s.io/upgrades/automated#scheduling-upgrades\"\n\n validation {\n condition = var.system_upgrade_schedule_window == null ? true : (\n length(try(var.system_upgrade_schedule_window.days, [])) > 0 ||\n coalesce(try(var.system_upgrade_schedule_window.startTime, \"\"), \"\") != \"\" ||\n coalesce(try(var.system_upgrade_schedule_window.endTime, \"\"), \"\") != \"\"\n )\n error_message = \"system_upgrade_schedule_window must have at least one of 'days', 'startTime', or 'endTime' set when not null.\"\n }\n\n validation {\n condition = var.system_upgrade_schedule_window == null ? true : alltrue([\n for day in try(var.system_upgrade_schedule_window.days, []) :\n can(regex(\"^(monday|tuesday|wednesday|thursday|friday|saturday|sunday)$\", day))\n ])\n error_message = \"system_upgrade_schedule_window.days must contain lowercase day names (monday-sunday).\"\n }\n\n validation {\n condition = var.system_upgrade_schedule_window == null ? true : alltrue([\n for time_value in [\n coalesce(try(var.system_upgrade_schedule_window.startTime, \"\"), \"\"),\n coalesce(try(var.system_upgrade_schedule_window.endTime, \"\"), \"\")\n ] :\n time_value == \"\" || can(regex(\"^([01][0-9]|2[0-3]):[0-5][0-9]$\", time_value))\n ])\n error_message = \"system_upgrade_schedule_window.startTime and endTime must use 24-hour HH:MM format when set.\"\n }\n\n validation {\n condition = var.system_upgrade_schedule_window == null ? true : (\n coalesce(try(var.system_upgrade_schedule_window.timeZone, \"\"), \"\") == \"\" ||\n can(regex(\"^[A-Za-z_]+(?:/[A-Za-z0-9_+\\\\-]+)*$\", coalesce(try(var.system_upgrade_schedule_window.timeZone, \"\"), \"\")))\n )\n error_message = \"system_upgrade_schedule_window.timeZone must be a valid IANA timezone name (for example, UTC or Europe/Budapest).\"\n }\n}\n\nvariable \"automatically_upgrade_os\" {\n type = bool\n default = true\n description = \"Whether to enable or disable automatic os updates. Defaults to true. Should be disabled for single-node clusters\"\n}\n\nvariable \"extra_firewall_rules\" {\n type = list(any)\n default = []\n description = \"Additional firewall rules to apply to the cluster.\"\n}\n\nvariable \"firewall_kube_api_source\" {\n type = list(string)\n default = [\"0.0.0.0/0\", \"::/0\"]\n description = \"Source networks that have Kube API access to the servers.\"\n}\n\nvariable \"firewall_ssh_source\" {\n type = list(string)\n default = [\"0.0.0.0/0\", \"::/0\"]\n description = \"Source networks that have SSH access to the servers.\"\n}\n\nvariable \"use_cluster_name_in_node_name\" {\n type = bool\n default = true\n description = \"Whether to use the cluster name in the node name.\"\n}\n\nvariable \"cluster_name\" {\n type = string\n default = \"k3s\"\n description = \"Name of the cluster.\"\n\n validation {\n condition = can(regex(\"^[a-z0-9\\\\-]+$\", var.cluster_name))\n error_message = \"The cluster name must be in the form of lowercase alphanumeric characters and/or dashes.\"\n }\n}\n\nvariable \"base_domain\" {\n type = string\n default = \"\"\n description = \"Base domain of the cluster, used for reverse dns.\"\n\n validation {\n condition = can(regex(\"^(?:(?:(?:[A-Za-z0-9])|(?:[A-Za-z0-9](?:[A-Za-z0-9\\\\-]+)?[A-Za-z0-9]))+(\\\\.))+([A-Za-z]{2,})([\\\\/?])?([\\\\/?][A-Za-z0-9\\\\-%._~:\\\\/?#\\\\[\\\\]@!\\\\$&\\\\'\\\\(\\\\)\\\\*\\\\+,;=]+)?$\", var.base_domain)) || var.base_domain == \"\"\n error_message = \"It must be a valid domain name (FQDN).\"\n }\n}\n\nvariable \"placement_group_disable\" {\n type = bool\n default = false\n description = \"Whether to disable placement groups.\"\n}\n\nvariable \"disable_kube_proxy\" {\n type = bool\n default = false\n description = \"Disable kube-proxy in K3s (default false).\"\n}\n\nvariable \"disable_network_policy\" {\n type = bool\n default = false\n description = \"Disable k3s default network policy controller (default false, automatically true for calico and cilium).\"\n}\n\nvariable \"cni_plugin\" {\n type = string\n default = \"flannel\"\n description = \"CNI plugin for k3s.\"\n\n validation {\n condition = contains([\"flannel\", \"calico\", \"cilium\"], var.cni_plugin)\n error_message = \"The cni_plugin must be one of \\\"flannel\\\", \\\"calico\\\", or \\\"cilium\\\".\"\n }\n}\n\nvariable \"cilium_egress_gateway_enabled\" {\n type = bool\n default = false\n description = \"Enables egress gateway to redirect and SNAT the traffic that leaves the cluster.\"\n}\n\nvariable \"cilium_hubble_enabled\" {\n type = bool\n default = false\n description = \"Enables Hubble Observability to collect and visualize network traffic.\"\n}\n\nvariable \"cilium_hubble_metrics_enabled\" {\n type = list(string)\n default = []\n description = \"Configures the list of Hubble metrics to collect\"\n}\n\nvariable \"cilium_ipv4_native_routing_cidr\" {\n type = string\n default = null\n description = \"Used when Cilium is configured in native routing mode. The CNI assumes that the underlying network stack will forward packets to this destination without the need to apply SNAT. Default: value of \\\"cluster_ipv4_cidr\\\"\"\n}\n\nvariable \"cilium_routing_mode\" {\n type = string\n default = \"tunnel\"\n description = \"Set native-routing mode (\\\"native\\\") or tunneling mode (\\\"tunnel\\\").\"\n\n validation {\n condition = contains([\"tunnel\", \"native\"], var.cilium_routing_mode)\n error_message = \"The cilium_routing_mode must be one of \\\"tunnel\\\" or \\\"native\\\".\"\n }\n}\n\nvariable \"cilium_loadbalancer_acceleration_mode\" {\n type = string\n default = \"best-effort\"\n description = \"Set Cilium loadbalancer.acceleration-mode. Supported values are \\\"disabled\\\", \\\"native\\\" and \\\"best-effort\\\".\"\n\n validation {\n condition = contains([\"disabled\", \"native\", \"best-effort\"], var.cilium_loadbalancer_acceleration_mode)\n error_message = \"The cilium_loadbalancer_acceleration_mode must be one of \\\"disabled\\\", \\\"native\\\" or \\\"best-effort\\\".\"\n }\n}\n\nvariable \"cilium_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to Cilium as 'valuesContent' at the HelmChart.\"\n}\n\nvariable \"cilium_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or cilium_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.cilium_merge_values == \"\" || can(yamldecode(var.cilium_merge_values))\n error_message = \"cilium_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"cilium_version\" {\n type = string\n default = \"1.17.0\"\n description = \"Version of Cilium. See https://github.com/cilium/cilium/releases for the available versions.\"\n}\n\nvariable \"calico_values\" {\n type = string\n default = \"\"\n description = \"Just a stub for a future helm implementation. Now it can be used to replace the calico kustomize patch of the calico manifest.\"\n}\n\nvariable \"enable_iscsid\" {\n type = bool\n default = false\n description = \"This is always true when enable_longhorn=true, however, you may also want this enabled if you perform your own installation of longhorn after this module runs.\"\n}\n\nvariable \"enable_longhorn\" {\n type = bool\n default = false\n description = \"Whether or not to enable Longhorn.\"\n}\n\nvariable \"longhorn_version\" {\n type = string\n default = \"*\"\n description = \"Longhorn Helm chart version.\"\n}\n\nvariable \"longhorn_helmchart_bootstrap\" {\n type = bool\n default = false\n description = \"Whether the HelmChart longhorn shall be run on control-plane nodes.\"\n}\n\nvariable \"longhorn_repository\" {\n type = string\n default = \"https://charts.longhorn.io\"\n description = \"By default the official chart which may be incompatible with rancher is used. If you need to fully support rancher switch to https://charts.rancher.io.\"\n}\n\nvariable \"longhorn_namespace\" {\n type = string\n default = \"longhorn-system\"\n description = \"Namespace for longhorn deployment, defaults to 'longhorn-system'\"\n}\n\nvariable \"longhorn_fstype\" {\n type = string\n default = \"ext4\"\n description = \"The longhorn fstype.\"\n\n validation {\n condition = contains([\"ext4\", \"xfs\"], var.longhorn_fstype)\n error_message = \"Must be one of \\\"ext4\\\" or \\\"xfs\\\"\"\n }\n}\n\nvariable \"longhorn_replica_count\" {\n type = number\n default = 3\n description = \"Number of replicas per longhorn volume.\"\n\n validation {\n condition = var.longhorn_replica_count > 0\n error_message = \"Number of longhorn replicas can't be below 1.\"\n }\n}\n\nvariable \"longhorn_values\" {\n type = string\n default = \"\"\n description = \"Helm values passed as valuesContent to the Longhorn HelmChart. When set, this replaces the module defaults.\"\n}\n\nvariable \"longhorn_merge_values\" {\n type = string\n default = \"\"\n description = \"Helm values to merge with defaults (or longhorn_values if set). User values take precedence. Use for targeted overrides like image tags. Requires valid YAML format.\"\n\n validation {\n condition = var.longhorn_merge_values == \"\" || can(yamldecode(var.longhorn_merge_values))\n error_message = \"longhorn_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"disable_hetzner_csi\" {\n type = bool\n default = false\n description = \"Disable hetzner csi driver.\"\n}\n\nvariable \"enable_csi_driver_smb\" {\n type = bool\n default = false\n description = \"Whether or not to enable csi-driver-smb.\"\n}\n\nvariable \"csi_driver_smb_version\" {\n type = string\n default = \"*\"\n description = \"Version of csi_driver_smb. See https://github.com/kubernetes-csi/csi-driver-smb/releases for the available versions.\"\n}\n\nvariable \"csi_driver_smb_helmchart_bootstrap\" {\n type = bool\n default = false\n description = \"Whether the HelmChart csi_driver_smb shall be run on control-plane nodes.\"\n}\n\nvariable \"csi_driver_smb_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to csi-driver-smb as 'valuesContent' at the HelmChart.\"\n}\n\nvariable \"enable_cert_manager\" {\n type = bool\n default = true\n description = \"Enable cert manager.\"\n}\n\nvariable \"cert_manager_version\" {\n type = string\n default = \"*\"\n description = \"Version of cert_manager.\"\n}\n\nvariable \"cert_manager_helmchart_bootstrap\" {\n type = bool\n default = false\n description = \"Whether the HelmChart cert_manager shall be run on control-plane nodes.\"\n}\n\nvariable \"cert_manager_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to Cert-Manager as 'valuesContent' at the HelmChart. Defaults are set in locals.tf. For cert-manager versions prior to v1.15.0, you need to set 'installCRDs: true'.\"\n}\n\nvariable \"cert_manager_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or cert_manager_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.cert_manager_merge_values == \"\" || can(yamldecode(var.cert_manager_merge_values))\n error_message = \"cert_manager_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"enable_rancher\" {\n type = bool\n default = false\n description = \"Enable rancher.\"\n}\n\nvariable \"rancher_version\" {\n type = string\n default = \"*\"\n description = \"Version of rancher.\"\n}\n\nvariable \"rancher_helmchart_bootstrap\" {\n type = bool\n default = false\n description = \"Whether the HelmChart rancher shall be run on control-plane nodes.\"\n}\n\nvariable \"rancher_install_channel\" {\n type = string\n default = \"stable\"\n description = \"The rancher installation channel.\"\n\n validation {\n condition = contains([\"stable\", \"latest\"], var.rancher_install_channel)\n error_message = \"The allowed values for the Rancher install channel are stable or latest.\"\n }\n}\n\nvariable \"rancher_hostname\" {\n type = string\n default = \"\"\n description = \"The rancher hostname.\"\n\n validation {\n condition = can(regex(\"^(?:(?:(?:[A-Za-z0-9])|(?:[A-Za-z0-9](?:[A-Za-z0-9\\\\-]+)?[A-Za-z0-9]))+(\\\\.))+([A-Za-z]{2,})([\\\\/?])?([\\\\/?][A-Za-z0-9\\\\-%._~:\\\\/?#\\\\[\\\\]@!\\\\$&\\\\'\\\\(\\\\)\\\\*\\\\+,;=]+)?$\", var.rancher_hostname)) || var.rancher_hostname == \"\"\n error_message = \"It must be a valid domain name (FQDN).\"\n }\n}\n\nvariable \"lb_hostname\" {\n type = string\n default = \"\"\n description = \"The Hetzner Load Balancer hostname, for either Traefik, HAProxy or Ingress-Nginx.\"\n\n validation {\n condition = can(regex(\"^(?:(?:(?:[A-Za-z0-9])|(?:[A-Za-z0-9](?:[A-Za-z0-9\\\\-]+)?[A-Za-z0-9]))+(\\\\.))+([A-Za-z]{2,})([\\\\/?])?([\\\\/?][A-Za-z0-9\\\\-%._~:\\\\/?#\\\\[\\\\]@!\\\\$&\\\\'\\\\(\\\\)\\\\*\\\\+,;=]+)?$\", var.lb_hostname)) || var.lb_hostname == \"\"\n error_message = \"It must be a valid domain name (FQDN).\"\n }\n}\n\nvariable \"kubeconfig_server_address\" {\n type = string\n default = \"\"\n description = \"The hostname used for kubeconfig.\"\n}\n\nvariable \"rancher_registration_manifest_url\" {\n type = string\n description = \"The url of a rancher registration manifest to apply. (see https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/).\"\n default = \"\"\n sensitive = true\n}\n\nvariable \"rancher_bootstrap_password\" {\n type = string\n default = \"\"\n description = \"Rancher bootstrap password.\"\n sensitive = true\n\n validation {\n condition = (length(var.rancher_bootstrap_password) >= 48) || (length(var.rancher_bootstrap_password) == 0)\n error_message = \"The Rancher bootstrap password must be at least 48 characters long.\"\n }\n}\n\nvariable \"rancher_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to Rancher as 'valuesContent' at the HelmChart.\"\n}\n\nvariable \"rancher_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or rancher_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.rancher_merge_values == \"\" || can(yamldecode(var.rancher_merge_values))\n error_message = \"rancher_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"kured_version\" {\n type = string\n default = null\n description = \"Version of Kured. See https://github.com/kubereboot/kured/releases for the available versions.\"\n}\n\nvariable \"kured_options\" {\n type = map(string)\n default = {}\n}\n\nvariable \"block_icmp_ping_in\" {\n type = bool\n default = false\n description = \"Block entering ICMP ping.\"\n}\n\nvariable \"use_control_plane_lb\" {\n type = bool\n default = false\n description = \"Creates a dedicated load balancer for the Kubernetes API (port 6443). When enabled, kubectl and other API clients connect through this LB instead of directly to the first control plane node. Recommended for production clusters with multiple control plane nodes for high availability. Note: This is separate from the ingress load balancer for HTTP/HTTPS traffic.\"\n}\n\nvariable \"control_plane_lb_type\" {\n type = string\n default = \"lb11\"\n description = \"The type of load balancer to use for the control plane load balancer. Defaults to lb11, which is the cheapest one.\"\n}\n\nvariable \"control_plane_lb_enable_public_interface\" {\n type = bool\n default = true\n description = \"Enable or disable public interface for the control plane load balancer. Defaults to true. When disabled with nat_router enabled, the NAT router automatically forwards port 6443 to the private control plane LB.\"\n}\n\nvariable \"dns_servers\" {\n type = list(string)\n\n default = [\n \"185.12.64.1\",\n \"185.12.64.2\",\n \"2a01:4ff:ff00::add:1\",\n ]\n description = \"IP Addresses to use for the DNS Servers, set to an empty list to use the ones provided by Hetzner. The length is limited to 3 entries, more entries is not supported by kubernetes\"\n\n validation {\n condition = length(var.dns_servers) <= 3\n error_message = \"The list must have no more than 3 items.\"\n }\n\n validation {\n condition = alltrue([for ip in var.dns_servers : provider::assert::ip(ip)])\n error_message = \"Some IP addresses are incorrect.\"\n }\n}\n\nvariable \"address_for_connectivity_test\" {\n description = \"The address to test for external connectivity before proceeding with the installation. Defaults to Google's public DNS.\"\n type = string\n default = \"8.8.8.8\"\n}\n\nvariable \"additional_k3s_environment\" {\n type = map(any)\n default = {}\n description = \"Additional environment variables for the k3s binary. See for example https://docs.k3s.io/advanced#configuring-an-http-proxy .\"\n}\n\nvariable \"preinstall_exec\" {\n type = list(string)\n default = []\n description = \"Additional to execute before the install calls, for example fetching and installing certs.\"\n}\n\nvariable \"postinstall_exec\" {\n type = list(string)\n default = []\n description = \"Additional to execute after the install calls, for example restoring a backup.\"\n}\n\n\nvariable \"extra_kustomize_deployment_commands\" {\n type = string\n default = \"\"\n description = \"Commands to be executed after the `kubectl apply -k <dir>` step.\"\n}\n\nvariable \"extra_kustomize_parameters\" {\n type = any\n default = {}\n description = \"All values will be passed to the `kustomization.tmp.yml` template.\"\n}\n\nvariable \"extra_kustomize_folder\" {\n type = string\n default = \"extra-manifests\"\n description = \"Folder from where to upload extra manifests\"\n}\n\nvariable \"create_kubeconfig\" {\n type = bool\n default = true\n description = \"Create the kubeconfig as a local file resource. Should be disabled for automatic runs.\"\n}\n\nvariable \"create_kustomization\" {\n type = bool\n default = true\n description = \"Create the kustomization backup as a local file resource. Should be disabled for automatic runs.\"\n}\n\nvariable \"export_values\" {\n type = bool\n default = false\n description = \"Export for deployment used values.yaml-files as local files.\"\n}\n\nvariable \"enable_wireguard\" {\n type = bool\n default = false\n description = \"Use wireguard-native as the backend for CNI.\"\n}\n\nvariable \"flannel_backend\" {\n type = string\n default = null\n description = \"Override the flannel backend used by k3s. When set, this takes precedence over enable_wireguard. Valid values: vxlan, host-gw, wireguard-native. See https://docs.k3s.io/networking/basic-network-options for details. Use wireguard-native for Robot nodes with vSwitch to avoid MTU issues.\"\n\n validation {\n condition = var.flannel_backend == null || contains([\"vxlan\", \"host-gw\", \"wireguard-native\"], var.flannel_backend)\n error_message = \"The flannel_backend must be one of: vxlan, host-gw, wireguard-native.\"\n }\n}\n\nvariable \"control_planes_custom_config\" {\n type = any\n default = {}\n description = \"Additional configuration for control planes that will be added to k3s's config.yaml. E.g to allow etcd monitoring.\"\n}\n\nvariable \"agent_nodes_custom_config\" {\n type = any\n default = {}\n description = \"Additional configuration for agent nodes and autoscaler nodes that will be added to k3s's config.yaml. E.g to allow kube-proxy monitoring.\"\n}\n\nvariable \"k3s_registries\" {\n description = \"K3S registries.yml contents. It used to access private docker registries.\"\n default = \" \"\n type = string\n}\n\nvariable \"k3s_kubelet_config\" {\n description = \"K3S kubelet-config.yaml contents. Used to configure the kubelet.\"\n default = \"\"\n type = string\n}\n\nvariable \"k3s_audit_policy_config\" {\n description = \"K3S audit-policy.yaml contents. Used to configure Kubernetes audit logging.\"\n default = \"\"\n type = string\n}\n\nvariable \"k3s_audit_log_path\" {\n description = \"Path where audit logs will be stored on control plane nodes\"\n default = \"/var/log/k3s-audit/audit.log\"\n type = string\n}\n\nvariable \"k3s_audit_log_maxage\" {\n description = \"Maximum number of days to retain audit log files\"\n default = 30\n type = number\n}\n\nvariable \"k3s_audit_log_maxbackup\" {\n description = \"Maximum number of audit log files to retain\"\n default = 10\n type = number\n}\n\nvariable \"k3s_audit_log_maxsize\" {\n description = \"Maximum size in megabytes of the audit log file before rotation\"\n default = 100\n type = number\n}\n\nvariable \"additional_tls_sans\" {\n description = \"Additional TLS SANs to allow connection to control-plane through it.\"\n default = []\n type = list(string)\n}\n\nvariable \"calico_version\" {\n type = string\n default = null\n description = \"Version of Calico. See https://github.com/projectcalico/calico/releases for the available versions.\"\n}\n\nvariable \"k3s_exec_server_args\" {\n type = string\n default = \"\"\n description = \"The control plane is started with `k3s server {k3s_exec_server_args}`. Use this to add kube-apiserver-arg for example.\"\n}\n\nvariable \"k3s_exec_agent_args\" {\n type = string\n default = \"\"\n description = \"Agents nodes are started with `k3s agent {k3s_exec_agent_args}`. Use this to add kubelet-arg for example.\"\n}\n\nvariable \"k3s_prefer_bundled_bin\" {\n type = bool\n default = false\n description = \"Whether to use the bundled k3s mount binary instead of the one from the distro's util-linux package.\"\n}\n\nvariable \"k3s_global_kubelet_args\" {\n type = list(string)\n default = []\n description = \"Global kubelet args for all nodes.\"\n}\n\nvariable \"k3s_control_plane_kubelet_args\" {\n type = list(string)\n default = []\n description = \"Kubelet args for control plane nodes.\"\n}\n\nvariable \"k3s_agent_kubelet_args\" {\n type = list(string)\n default = []\n description = \"Kubelet args for agent nodes.\"\n}\n\nvariable \"k3s_autoscaler_kubelet_args\" {\n type = list(string)\n default = []\n description = \"Kubelet args for autoscaler nodes.\"\n}\n\nvariable \"ingress_target_namespace\" {\n type = string\n default = \"\"\n description = \"The namespace to deploy the ingress controller to. Defaults to ingress name.\"\n}\n\nvariable \"enable_local_storage\" {\n type = bool\n default = false\n description = \"Whether to enable or disable k3s local-storage. Warning: when enabled, there will be two default storage classes: \\\"local-path\\\" and \\\"hcloud-volumes\\\"!\"\n}\n\nvariable \"disable_selinux\" {\n type = bool\n default = false\n description = \"Disable SELinux on all nodes.\"\n}\n\nvariable \"enable_delete_protection\" {\n type = object({\n floating_ip = optional(bool, false)\n load_balancer = optional(bool, false)\n volume = optional(bool, false)\n })\n default = {\n floating_ip = false\n load_balancer = false\n volume = false\n }\n description = \"Enable or disable delete protection for resources in Hetzner Cloud.\"\n}\n\nvariable \"keep_disk_agents\" {\n type = bool\n default = false\n description = \"Whether to keep OS disks of nodes the same size when upgrading an agent node\"\n}\n\nvariable \"keep_disk_cp\" {\n type = bool\n default = false\n description = \"Whether to keep OS disks of nodes the same size when upgrading a control-plane node\"\n}\n\n\nvariable \"sys_upgrade_controller_version\" {\n type = string\n default = \"v0.18.0\"\n description = \"Version of the System Upgrade Controller for automated upgrades of k3s. v0.15.0+ supports the 'window' parameter for scheduling upgrades. See https://github.com/rancher/system-upgrade-controller/releases for available versions.\"\n}\n\nvariable \"hetzner_ccm_values\" {\n type = string\n default = \"\"\n description = \"Additional helm values file to pass to Hetzner Controller Manager as 'valuesContent' at the HelmChart.\"\n}\n\nvariable \"hetzner_ccm_merge_values\" {\n type = string\n default = \"\"\n description = \"Additional Helm values to merge with defaults (or hetzner_ccm_values if set). User values take precedence. Requires valid YAML format.\"\n\n validation {\n condition = var.hetzner_ccm_merge_values == \"\" || can(yamldecode(var.hetzner_ccm_merge_values))\n error_message = \"hetzner_ccm_merge_values must be valid YAML format or empty string.\"\n }\n}\n\nvariable \"control_plane_endpoint\" {\n type = string\n description = \"Optional external control plane endpoint URL (e.g. https://myapi.domain.com:6443). Used as the k3s 'server' value for agents and secondary control planes.\"\n default = null\n validation {\n condition = var.control_plane_endpoint == null || can(regex(\"^https?://(?:(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\\\\.)*[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?|(?:[0-9]{1,3}\\\\.){3}[0-9]{1,3}|\\\\[[0-9a-fA-F:]+\\\\])(?::[0-9]{1,5})?(?:/.*)?$\", var.control_plane_endpoint))\n error_message = \"The control_plane_endpoint must be null or a valid URL (e.g., https://my-api.example.com:6443).\"\n }\n}\n" }, { "path": "versions.tf", "content": "terraform {\n required_version = \">= 1.10.1\"\n required_providers {\n github = {\n source = \"integrations/github\"\n version = \">= 6.4.0\"\n }\n hcloud = {\n source = \"hetznercloud/hcloud\"\n version = \">= 1.59.0\"\n }\n local = {\n source = \"hashicorp/local\"\n version = \">= 2.5.2\"\n }\n ssh = {\n source = \"loafoe/ssh\"\n version = \"2.7.0\"\n }\n assert = {\n source = \"hashicorp/assert\"\n version = \">= 0.16.0\"\n }\n semvers = {\n source = \"anapsix/semvers\"\n version = \">= 0.7.1\"\n }\n }\n}\n\n# Prevent provider picking up `GITHUB_TOKEN` env var and trying to authenticate\nprovider \"github\" {\n token = \"\"\n}\n" } ]