[
  {
    "path": ".gitignore",
    "content": "# Compiled files\n*.tfstate\n*.tfstate.backup\n*.terraform.tfstate.lock.info\n\n# Module directory\n.terraform/\n\n# IDE\n.idea\n.vscode\n.DS_Store\n\n# K8S\n*.conf\n"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2017 Stefan Prodan\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# k8s-scw-baremetal\n\nKubernetes Terraform installer for Scaleway bare-metal ARM and AMD64\n\n### Initial setup\n\nClone the repository and install the dependencies:\n\n```bash\n$ git clone https://github.com/stefanprodan/k8s-scw-baremetal.git\n$ cd k8s-scw-baremetal\n$ terraform init\n```\n\nNote that you'll need Terraform v0.10 or newer to run this project.\n\nBefore running the project you'll have to create an access token for Terraform to connect to the Scaleway API\n\nNow retrieve the `<ORGANIZATION_ID>` using your `<ACCESS-TOKEN>` from `/organizations` API endpoint:\n\n```bash\n$ curl https://account.scaleway.com/organizations -H \"X-Auth-Token: <ACCESS-TOKEN>\"\n```\n\nSample output (excerpt with organization ID):\n```bash\n\"organizations\": [{\"id\": \"xxxxxxxxxxxxx\", \"name\": \"Organization Name\"}],\n```\n\nUsing the token and your organization ID, create two environment variables:\n\n```bash\n$ export SCALEWAY_ORGANIZATION=\"<ORGANIZATION_ID>\"\n$ export SCALEWAY_TOKEN=\"<ACCESS-TOKEN>\"\n```\n\nTo configure your cluster, you'll need to have `jq` installed on your computer.\n\n### Usage\n\nCreate an AMD64 bare-metal Kubernetes cluster with one master and a node:\n\n```bash\n$ terraform workspace new amd64\n\n$ terraform apply \\\n -var region=par1 \\\n -var arch=x86_64 \\\n -var server_type=C2S \\\n -var nodes=1 \\\n -var server_type_node=C2S \\\n -var weave_passwd=ChangeMe \\\n -var docker_version=18.06 \\\n -var ubuntu_version=\"Ubuntu Bionic\"\n```\n\nThis will do the following:\n\n* reserves public IPs for each server\n* provisions three bare-metal servers with Ubuntu 16.04.1 LTS (the size of the `master` and the `node` may be different but must remain in the same type of architecture)\n* connects to the master server via SSH and installs Docker CE and kubeadm apt packages\n* runs kubeadm init on the master server and configures kubectl\n* downloads the kubectl admin config file on your local machine and replaces the private IP with the public one\n* creates a Kubernetes secret with the Weave Net password\n* installs Weave Net with encrypted overlay\n* installs cluster add-ons (Kubernetes dashboard, metrics server and Heapster)\n* starts the nodes in parallel and installs Docker CE and kubeadm\n* joins the nodes in the cluster using the kubeadm token obtained from the master\n\nScale up by increasing the number of nodes:\n\n```bash\n$ terraform apply \\\n -var nodes=3\n```\n\nTear down the whole infrastructure with:\n\n```bash\nterraform destroy -force\n```\n\nCreate an ARMv7 bare-metal Kubernetes cluster with one master and two nodes:\n\n```bash\n$ terraform workspace new arm\n\n$ terraform apply \\\n -var region=par1 \\\n -var arch=arm \\\n -var server_type=C1 \\\n -var nodes=2 \\\n -var server_type_node=C1 \\\n -var weave_passwd=ChangeMe \\\n -var docker_version=18.06 \\\n -var ubuntu_version=\"Ubuntu Xenial\"\n```\n\n### Remote control\n\nAfter applying the Terraform plan you'll see several output variables like the master public IP,\nthe kubeadmn join command and the current workspace admin config.\n\nIn order to run `kubectl` commands against the Scaleway cluster you can use the `kubectl_config` output variable:\n\nCheck if Heapster works:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n  top nodes\n\nNAME           CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%\narm-master-1   655m         16%       873Mi           45%\narm-node-1     147m         3%        618Mi           32%\narm-node-2     101m         2%        584Mi           30%\n```\n\nThe `kubectl` config file format is `<WORKSPACE>.conf` as in `arm.conf` or `amd64.conf`.\n\nIn order to access the dashboard you can use port forward:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n  -n kube-system port-forward deployment/kubernetes-dashboard 8888:9090\n```\n\nNow you can access the dashboard on your computer at `http://localhost:8888`.\n\n![Overview](https://github.com/stefanprodan/k8s-scw-baremetal/blob/master/screens/dash-overview.png)\n\n![Nodes](https://github.com/stefanprodan/k8s-scw-baremetal/blob/master/screens/dash-nodes.png)\n\n### Expose services outside the cluster\n\nSince we're running on bare-metal and Scaleway doesn't offer a load balancer, the easiest way to expose\napplications outside of Kubernetes is using a NodePort service.\n\nLet's deploy the [podinfo](https://github.com/stefanprodan/k8s-podinfo) app in the default namespace.\nPodinfo has a multi-arch Docker image and it will work on arm, arm64 or amd64.\n\nCreate the podinfo nodeport service:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n  apply -f https://raw.githubusercontent.com/stefanprodan/k8s-podinfo/7a8506e60fca086572f16de57f87bf5430e2df48/deploy/podinfo-svc-nodeport.yaml\n \nservice \"podinfo-nodeport\" created\n```\n\nCreate the podinfo deployment:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n  apply -f https://raw.githubusercontent.com/stefanprodan/k8s-podinfo/7a8506e60fca086572f16de57f87bf5430e2df48/deploy/podinfo-dep.yaml\n\ndeployment \"podinfo\" created\n```\n\nInspect the podinfo service to obtain the port number:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n  get svc --selector=app=podinfo\n\nNAME               TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE\npodinfo-nodeport   NodePort   10.104.132.14   <none>        9898:31190/TCP   3m\n```\n\nYou can access podinfo at `http://<MASTER_PUBLIC_IP>:31190` or using curl:\n\n```bash\n$ curl http://$(terraform output k8s_master_public_ip):31190\n\nruntime:\n  arch: arm\n  max_procs: \"4\"\n  num_cpu: \"4\"\n  num_goroutine: \"12\"\n  os: linux\n  version: go1.9.2\nlabels:\n  app: podinfo\n  pod-template-hash: \"1847780700\"\nannotations:\n  kubernetes.io/config.seen: 2018-01-08T00:39:45.580597397Z\n  kubernetes.io/config.source: api\nenvironment:\n  HOME: /root\n  HOSTNAME: podinfo-5d8ccd4c44-zrczc\n  KUBERNETES_PORT: tcp://10.96.0.1:443\n  KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443\n  KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1\n  KUBERNETES_PORT_443_TCP_PORT: \"443\"\n  KUBERNETES_PORT_443_TCP_PROTO: tcp\n  KUBERNETES_SERVICE_HOST: 10.96.0.1\n  KUBERNETES_SERVICE_PORT: \"443\"\n  KUBERNETES_SERVICE_PORT_HTTPS: \"443\"\n  PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\nexternalIP:\n  IPv4: 163.172.139.112\n```\n\n### Horizontal Pod Autoscaling\n\nStarting from Kubernetes 1.9 `kube-controller-manager` is configured by default with\n`horizontal-pod-autoscaler-use-rest-clients`.\nIn order to use HPA we need to install the metrics server to enable the new metrics API used by HPA v2.\nBoth Heapster and the metrics server have been deployed from Terraform\nwhen the master node was provisioned.\n\nThe metric server collects resource usage data from each node using Kubelet Summary API.\nCheck if the metrics server is running:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n get --raw \"/apis/metrics.k8s.io/v1beta1/nodes\" | jq\n```\n\n```json\n{\n  \"kind\": \"NodeMetricsList\",\n  \"apiVersion\": \"metrics.k8s.io/v1beta1\",\n  \"metadata\": {\n    \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes\"\n  },\n  \"items\": [\n    {\n      \"metadata\": {\n        \"name\": \"arm-master-1\",\n        \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes/arm-master-1\",\n        \"creationTimestamp\": \"2018-01-08T15:17:09Z\"\n      },\n      \"timestamp\": \"2018-01-08T15:17:00Z\",\n      \"window\": \"1m0s\",\n      \"usage\": {\n        \"cpu\": \"384m\",\n        \"memory\": \"935792Ki\"\n      }\n    },\n    {\n      \"metadata\": {\n        \"name\": \"arm-node-1\",\n        \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes/arm-node-1\",\n        \"creationTimestamp\": \"2018-01-08T15:17:09Z\"\n      },\n      \"timestamp\": \"2018-01-08T15:17:00Z\",\n      \"window\": \"1m0s\",\n      \"usage\": {\n        \"cpu\": \"130m\",\n        \"memory\": \"649020Ki\"\n      }\n    },\n    {\n      \"metadata\": {\n        \"name\": \"arm-node-2\",\n        \"selfLink\": \"/apis/metrics.k8s.io/v1beta1/nodes/arm-node-2\",\n        \"creationTimestamp\": \"2018-01-08T15:17:09Z\"\n      },\n      \"timestamp\": \"2018-01-08T15:17:00Z\",\n      \"window\": \"1m0s\",\n      \"usage\": {\n        \"cpu\": \"120m\",\n        \"memory\": \"614180Ki\"\n      }\n    }\n  ]\n}\n```\n\nLet's define a HPA that will maintain a minimum of two replicas and will scale up to ten\nif the CPU average is over 80% or if the memory goes over 200Mi.\n\n```yaml\napiVersion: autoscaling/v2beta1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: podinfo\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1beta1\n    kind: Deployment\n    name: podinfo\n  minReplicas: 2\n  maxReplicas: 10\n  metrics:\n  - type: Resource\n    resource:\n      name: cpu\n      targetAverageUtilization: 80\n  - type: Resource\n    resource:\n      name: memory\n      targetAverageValue: 200Mi\n```\n\nApply the podinfo HPA:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) \\\n  apply -f https://raw.githubusercontent.com/stefanprodan/k8s-podinfo/7a8506e60fca086572f16de57f87bf5430e2df48/deploy/podinfo-hpa.yaml\n\nhorizontalpodautoscaler \"podinfo\" created\n```\n\nAfter a couple of seconds the HPA controller will contact the metrics server and will fetch the CPU\nand memory usage:\n\n```bash\n$ kubectl --kubeconfig ./$(terraform output kubectl_config) get hpa\n\nNAME      REFERENCE            TARGETS                      MINPODS   MAXPODS   REPLICAS   AGE\npodinfo   Deployment/podinfo   2826240 / 200Mi, 15% / 80%   2         10        2          5m\n```\n\nIn order to increase the CPU usage we could run a load test with hey:\n\n```bash\n#install hey\ngo get -u github.com/rakyll/hey\n\n#do 10K requests rate limited at 20 QPS\nhey -n 10000 -q 10 -c 5 http://$(terraform output k8s_master_public_ip):31190\n```\n\nYou can monitor the autoscaler events with:\n\n```bash\n$ watch -n 5 kubectl --kubeconfig ./$(terraform output kubectl_config) describe hpa\n\nEvents:\n  Type    Reason             Age   From                       Message\n  ----    ------             ----  ----                       -------\n  Normal  SuccessfulRescale  7m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target\n  Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target\n```\n\nAfter the load tests finishes the autoscaler will remove replicas until the deployment reaches the initial replica count:\n\n```\nEvents:\n  Type    Reason             Age   From                       Message\n  ----    ------             ----  ----                       -------\n  Normal  SuccessfulRescale  20m   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target\n  Normal  SuccessfulRescale  16m   horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target\n  Normal  SuccessfulRescale  12m   horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target\n  Normal  SuccessfulRescale  6m    horizontal-pod-autoscaler  New size: 2; reason: All metrics below target\n```\n"
  },
  {
    "path": "addons/dashboard-rbac.yaml",
    "content": "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRoleBinding\nmetadata:\n  name: kubernetes-dashboard\n  labels:\n    k8s-app: kubernetes-dashboard\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: cluster-admin\nsubjects:\n- kind: ServiceAccount\n  name: kubernetes-dashboard\n  namespace: kube-system\n"
  },
  {
    "path": "addons/heapster-amd64.yaml",
    "content": "---\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  name: heapster\n  namespace: kube-system\nspec:\n  replicas: 1\n  template:\n    metadata:\n      labels:\n        task: monitoring\n        k8s-app: heapster\n    spec:\n      serviceAccountName: heapster\n      containers:\n      - name: heapster\n        image: k8s.gcr.io/heapster-amd64:v1.5.0\n        imagePullPolicy: IfNotPresent\n        command:\n        - /heapster\n        - --source=kubernetes.summary_api:''\n"
  },
  {
    "path": "addons/heapster-arm.yaml",
    "content": "---\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  name: heapster\n  namespace: kube-system\nspec:\n  replicas: 1\n  template:\n    metadata:\n      labels:\n        task: monitoring\n        k8s-app: heapster\n    spec:\n      serviceAccountName: heapster\n      containers:\n      - name: heapster\n        image: k8s.gcr.io/heapster-arm:v1.5.0\n        imagePullPolicy: IfNotPresent\n        command:\n        - /heapster\n        - --source=kubernetes.summary_api:''\n"
  },
  {
    "path": "addons/heapster-rbac.yaml",
    "content": "---\nkind: ClusterRoleBinding\napiVersion: rbac.authorization.k8s.io/v1beta1\nmetadata:\n  name: heapster\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: system:heapster\nsubjects:\n- kind: ServiceAccount\n  name: heapster\n  namespace: kube-system\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: heapster\n  namespace: kube-system\n---\napiVersion: v1\nkind: Service\nmetadata:\n  labels:\n    task: monitoring\n    kubernetes.io/cluster-service: 'true'\n    kubernetes.io/name: Heapster\n  name: heapster\n  namespace: kube-system\nspec:\n  ports:\n  - port: 80\n    targetPort: 8082\n  selector:\n    k8s-app: heapster\n"
  },
  {
    "path": "addons/metrics-server-amd64.yaml",
    "content": "---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: metrics-server\n  namespace: kube-system\n---\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  name: metrics-server\n  namespace: kube-system\n  labels:\n    k8s-app: metrics-server\nspec:\n  selector:\n    matchLabels:\n      k8s-app: metrics-server\n  template:\n    metadata:\n      name: metrics-server\n      labels:\n        k8s-app: metrics-server\n    spec:\n      serviceAccountName: metrics-server\n      containers:\n      - name: metrics-server\n        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1\n        imagePullPolicy: Always\n        command:\n        - /metrics-server\n        - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true\n\n"
  },
  {
    "path": "addons/metrics-server-arm.yaml",
    "content": "---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: metrics-server\n  namespace: kube-system\n---\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  name: metrics-server\n  namespace: kube-system\n  labels:\n    k8s-app: metrics-server\nspec:\n  selector:\n    matchLabels:\n      k8s-app: metrics-server\n  template:\n    metadata:\n      name: metrics-server\n      labels:\n        k8s-app: metrics-server\n    spec:\n      serviceAccountName: metrics-server\n      containers:\n      - name: metrics-server\n        image: gcr.io/google_containers/metrics-server-arm:v0.2.1\n        imagePullPolicy: Always\n        command:\n        - /metrics-server\n        - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true\n"
  },
  {
    "path": "addons/metrics-server-rbac.yaml",
    "content": "---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRoleBinding\nmetadata:\n  name: metrics-server:system:auth-delegator\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: system:auth-delegator\nsubjects:\n- kind: ServiceAccount\n  name: metrics-server\n  namespace: kube-system\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind: RoleBinding\nmetadata:\n  name: metrics-server-auth-reader\n  namespace: kube-system\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n  name: extension-apiserver-authentication-reader\nsubjects:\n- kind: ServiceAccount\n  name: metrics-server\n  namespace: kube-system\n---\napiVersion: apiregistration.k8s.io/v1beta1\nkind: APIService\nmetadata:\n  name: v1beta1.metrics.k8s.io\nspec:\n  service:\n    name: metrics-server\n    namespace: kube-system\n  group: metrics.k8s.io\n  version: v1beta1\n  insecureSkipTLSVerify: true\n  groupPriorityMinimum: 100\n  versionPriority: 100\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n  name: system:metrics-server\nrules:\n- apiGroups:\n  - \"\"\n  resources:\n  - pods\n  - nodes\n  - namespaces\n  verbs:\n  - get\n  - list\n  - watch\n- apiGroups:\n  - \"extensions\"\n  resources:\n  - deployments\n  verbs:\n  - get\n  - list\n  - watch\n- apiGroups:\n  - \"\"\n  resources:\n  - nodes/stats\n  verbs:\n  - get\n  - create\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: system:metrics-server\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: system:metrics-server\nsubjects:\n- kind: ServiceAccount\n  name: metrics-server\n  namespace: kube-system\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: metrics-server\n  namespace: kube-system\n  labels:\n    kubernetes.io/name: \"Metrics-server\"\nspec:\n  selector:\n    k8s-app: metrics-server\n  ports:\n  - port: 443\n    protocol: TCP\n    targetPort: 443\n"
  },
  {
    "path": "kubeadm/v1alpha3-config.yaml",
    "content": "---\napiVersion: kubeadm.k8s.io/v1alpha3\nkind: InitConfiguration\napiEndpoint:\n  advertiseAddress: CONFIG_CLUSTER_PRIVATE_IP\n  bindPort: 6443\nnodeRegistration:\n  kubeletExtraArgs:\n    \"feature-gates\": \"BlockVolume=true,CRIContainerLogRotation=true\"\n\n\n---\napiVersion: kubeadm.k8s.io/v1alpha3\nkind: ClusterConfiguration\nkubernetesVersion: CONFIG_KUBERNETES_VERSION\napiServerCertSANs:\n  - CONFIG_CLUSTER_PUBLIC_IP\napiServerExtraArgs:\n  authorization-mode: \"Node,RBAC\"\ncertificatesDir: /etc/kubernetes/pki\nclusterName: kubernetes\nimageRepository: k8s.gcr.io\n\n\n---\napiVersion: kubelet.config.k8s.io/v1beta1\nkind: KubeletConfiguration\ncontainerLogMaxFiles: 1\ncontainerLogMaxSize: CONFIG_CONTAINER_LOG_MAX_SIZE\nmaxPods: 110\nfeatureGates:\n  BlockVolume: true\n  CRIContainerLogRotation: true\nauthentication:\n  anonymous:\n    enabled: false\n  webhook:\n    enabled: true\nauthorization:\n  mode: Webhook\n\n\n---\napiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\n\n\n# ---\n# apiVersion: kubeadm.k8s.io/v1alpha3\n# kind: JoinConfiguration\n\n"
  },
  {
    "path": "kubeadm/v1beta1-config.yaml",
    "content": "---\napiVersion: kubeadm.k8s.io/v1beta1\nkind: InitConfiguration\nlocalAPIEndpoint:\n  advertiseAddress: CONFIG_CLUSTER_PRIVATE_IP\n  bindPort: 6443\nnodeRegistration:\n  kubeletExtraArgs:\n    \"feature-gates\": \"BlockVolume=true,CRIContainerLogRotation=true\"\n\n\n---\napiVersion: kubeadm.k8s.io/v1beta1\nkind: ClusterConfiguration\nkubernetesVersion: CONFIG_KUBERNETES_VERSION\napiServer:\n  extraArgs:\n    authorization-mode: \"Node,RBAC\"\n  certSANs:\n    - CONFIG_CLUSTER_PUBLIC_IP\n  timeoutForControlPlane: 4m0s\ncontrolPlaneEndpoint: CONFIG_CLUSTER_PRIVATE_IP:6443\ncontrollerManager:\n  extraArgs:\n    \"node-cidr-mask-size\": \"20\"\nscheduler:\n  extraArgs:\n    address: CONFIG_CLUSTER_PRIVATE_IP\ncertificatesDir: /etc/kubernetes/pki\nimageRepository: k8s.gcr.io\nuseHyperKubeImage: false\n\n\n---\napiVersion: kubelet.config.k8s.io/v1beta1\nkind: KubeletConfiguration\ncontainerLogMaxFiles: 1\ncontainerLogMaxSize: CONFIG_CONTAINER_LOG_MAX_SIZE\nmaxPods: 110\nfeatureGates:\n  BlockVolume: true\n  CRIContainerLogRotation: true\nauthentication:\n  anonymous:\n    enabled: false\n  webhook:\n    enabled: true\nauthorization:\n  mode: Webhook\n\n\n---\napiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\n\n\n# ---\n# apiVersion: kubeadm.k8s.io/v1beta1\n# kind: JoinConfiguration\n"
  },
  {
    "path": "main.tf",
    "content": "provider \"scaleway\" {\n  region  = \"${var.region}\"\n  version = \"1.8.0\"\n}\n\nprovider \"external\" {\n  version = \"1.0.0\"\n}\n\ndata \"scaleway_image\" \"ubuntu\" {\n  architecture = \"${var.arch}\"\n  name         = \"${var.ubuntu_version}\"\n}\n"
  },
  {
    "path": "master.tf",
    "content": "resource \"scaleway_ip\" \"k8s_master_ip\" {\n  count = 1\n}\n\nresource \"scaleway_server\" \"k8s_master\" {\n  count          = 1\n  name           = \"${terraform.workspace}-master-${count.index + 1}\"\n  image          = \"${data.scaleway_image.ubuntu.id}\"\n  type           = \"${var.server_type}\"\n  public_ip      = \"${element(scaleway_ip.k8s_master_ip.*.ip, count.index)}\"\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  connection {\n    type        = \"ssh\"\n    user        = \"root\"\n    private_key = \"${file(var.private_key)}\"\n  }\n  provisioner \"file\" {\n    source      = \"scripts/\"\n    destination = \"/tmp\"\n  }\n  provisioner \"file\" {\n    source      = \"addons/\"\n    destination = \"/tmp\"\n  }\n  provisioner \"file\" {\n    source      = \"kubeadm\"\n    destination = \"/tmp/\"\n  }\n  provisioner \"remote-exec\" {\n    inline = [\n      <<EOT\n#!/bin/bash\nset -e\nchmod +x /tmp/docker-install.sh\nchmod +x /tmp/kubeadm-install.sh\nchmod g+w -R /tmp/kubeadm/\n\nexport ubuntu_version=$(echo -n ${var.ubuntu_version} | cut -d \" \" -f 2 | awk '{print tolower($0)}')\n/tmp/docker-install.sh $${ubuntu_version} ${var.arch} ${var.docker_version} && \\\n/tmp/kubeadm-install.sh ${var.k8s_version} && \\\n\nmodify_kube_apiserver_config(){\n  while [[ ! -e /etc/kubernetes/manifests/kube-apiserver.yaml ]]; do\n    sleep 0.5s;\n  done && \\\n  sed -i 's/failureThreshold: [0-9]/failureThreshold: 18/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \\\n  sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \\\n  sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 240/g' /etc/kubernetes/manifests/kube-apiserver.yaml\n}\n\n# ref https://github.com/kubernetes/kubeadm/issues/413 (initialDelaySeconds is too eager)\nif [[ ${var.arch} == \"arm\" ]]; then modify_kube_apiserver_config & fi\n\nexport KUBEADM_VERSION=$(apt-cache madison kubeadm | grep $(echo ${var.k8s_version} | cut -c8-) | \\\n  awk 'NR==1 {print $3}' | rev | cut -c4- | rev)\n\ndpkg --compare-versions \"$${KUBEADM_VERSION}\" lt 1.13 && \\\n  export KUBEADM_CONFIG_FILE=/tmp/kubeadm/v1alpha3-config.yaml || \\\n  export KUBEADM_CONFIG_FILE=/tmp/kubeadm/v1beta1-config.yaml\n\ndpkg --compare-versions \"$${KUBEADM_VERSION}\" lt 1.12 && \\\n  export KUBEADM_CONFIG_FILE=\"\"\n\ndpkg --compare-versions \"$${KUBEADM_VERSION}\" lt 1.11 && \\\n  export VERBOSITY_EXTRA_ARGS='' || \\\n  export VERBOSITY_EXTRA_ARGS='--v ${var.kubeadm_verbosity}'\n\nif [[ -z \"$${KUBEADM_CONFIG_FILE}\" ]]; then\n  kubeadm init \\\n    --apiserver-advertise-address=${self.private_ip} \\\n    --apiserver-cert-extra-sans=${self.public_ip} \\\n    --kubernetes-version=${var.k8s_version} \\\n    --ignore-preflight-errors=KubeletVersion \\\n     $${VERBOSITY_EXTRA_ARGS};\nelse\n  sed -i 's/CONFIG_CLUSTER_PUBLIC_IP/${self.public_ip}/g' $${KUBEADM_CONFIG_FILE} && \\\n  sed -i 's/CONFIG_CLUSTER_PRIVATE_IP/${self.private_ip}/g' $${KUBEADM_CONFIG_FILE} && \\\n  sed -i \"s/CONFIG_KUBERNETES_VERSION/v$${KUBEADM_VERSION}/g\" $${KUBEADM_CONFIG_FILE} && \\\n  sed -i \"s/CONFIG_CONTAINER_LOG_MAX_SIZE/${var.container_log_max_size}/\" $${KUBEADM_CONFIG_FILE}\n\n  kubeadm init \\\n    --ignore-preflight-errors=KubeletVersion \\\n    --config=$${KUBEADM_CONFIG_FILE} \\\n     $${VERBOSITY_EXTRA_ARGS};\nfi && \\\n\nmkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && \\\nkubectl create secret -n kube-system generic weave-passwd --from-literal=weave-passwd=${var.weave_passwd} && \\\nkubectl apply -f \"https://cloud.weave.works/k8s/net?password-secret=weave-passwd&k8s-version=$(kubectl version | base64 | tr -d '\\n')\" && \\\nchmod +x /tmp/monitoring-install.sh && /tmp/monitoring-install.sh ${var.arch}\nEOT\n    ]\n  }\n  provisioner \"local-exec\" {\n    command    = \"./scripts/kubectl-conf.sh ${terraform.workspace} ${self.public_ip} ${self.private_ip} ${var.private_key}\"\n    on_failure = \"continue\"\n  }\n}\n\ndata \"external\" \"kubeadm_join\" {\n  program = [\"./scripts/kubeadm-token.sh\"]\n\n  query = {\n    host = \"${scaleway_ip.k8s_master_ip.0.ip}\"\n    key = \"${var.private_key}\"\n  }\n\n  depends_on = [\"scaleway_server.k8s_master\"]\n}\n"
  },
  {
    "path": "nodes.tf",
    "content": "resource \"scaleway_ip\" \"k8s_node_ip\" {\n  count = \"${var.nodes}\"\n}\n\nresource \"scaleway_server\" \"k8s_node\" {\n  count          = \"${var.nodes}\"\n  name           = \"${terraform.workspace}-node-${count.index + 1}\"\n  image          = \"${data.scaleway_image.ubuntu.id}\"\n  type           = \"${var.server_type_node}\"\n  public_ip      = \"${element(scaleway_ip.k8s_node_ip.*.ip, count.index)}\"\n  security_group = \"${scaleway_security_group.node_security_group.id}\"\n\n  connection {\n    type        = \"ssh\"\n    user        = \"root\"\n    private_key = \"${file(var.private_key)}\"\n  }\n  provisioner \"file\" {\n    source      = \"scripts/docker-install.sh\"\n    destination = \"/tmp/docker-install.sh\"\n  }\n  provisioner \"file\" {\n    source      = \"scripts/kubeadm-install.sh\"\n    destination = \"/tmp/kubeadm-install.sh\"\n  }\n  provisioner \"remote-exec\" {\n    inline = [\n      \"set -e\",\n      \"export ubuntu_version=$(echo -n ${var.ubuntu_version} | cut -d \\\" \\\" -f 2 | awk '{print tolower($0)}')\",\n      \"chmod +x /tmp/docker-install.sh && /tmp/docker-install.sh $${ubuntu_version} ${var.arch} ${var.docker_version}\",\n      \"chmod +x /tmp/kubeadm-install.sh && /tmp/kubeadm-install.sh ${var.k8s_version}\",\n      \"echo 'KUBELET_EXTRA_ARGS=${var.kubelet_extra_args}' > /etc/default/kubelet\",\n      \"${data.external.kubeadm_join.result.command}\",\n    ]\n  }\n  provisioner \"remote-exec\" {\n    inline = [\n      \"kubectl get pods --all-namespaces\",\n    ]\n\n    on_failure = \"continue\"\n\n    connection {\n      type = \"ssh\"\n      user = \"root\"\n      host = \"${scaleway_ip.k8s_master_ip.0.ip}\"\n    }\n  }\n}\n"
  },
  {
    "path": "outputs.tf",
    "content": "output \"k8s_master_public_ip\" {\n  value = \"${scaleway_ip.k8s_master_ip.0.ip}\"\n}\n\noutput \"kubeadm_join_command\" {\n  value = \"${data.external.kubeadm_join.result[\"command\"]}\"\n}\n\noutput \"nodes_public_ip\" {\n  value = \"${concat(scaleway_server.k8s_node.*.name, scaleway_server.k8s_node.*.public_ip)}\"\n}\n\noutput \"kubectl_config\" {\n  value = \"${terraform.workspace}.conf\"\n}\n"
  },
  {
    "path": "scripts/docker-install.sh",
    "content": "#!/usr/bin/env bash\n\nset -e\n\nUBUNTU_VERSION=$1\nARCH=$2\nDOCKER_VERSION=$3\n\nif [[ ${ARCH} == \"arm\" ]]; then export ARCH=armhf; fi\nif [[ ${ARCH} == \"x86_64\" ]]; then export ARCH=amd64; fi\n\napt-get update -qq\napt-get install -y -qq apt-transport-https ca-certificates curl git\ncurl -fsSL \"https://download.docker.com/linux/ubuntu/gpg\" | apt-key add -qq -\necho \"deb [arch=${ARCH}] https://download.docker.com/linux/ubuntu ${UBUNTU_VERSION} stable\" | \\\n  tee /etc/apt/sources.list.d/docker.list\napt-get update -qq\n\nif (( $(echo -n ${DOCKER_VERSION} | wc -c) > 5 )); then\n  export EXACT_DOCKER_VERSION=${DOCKER_VERSION}\nelse\n  export EXACT_DOCKER_VERSION=$(apt-cache madison docker-ce | \\\n    grep \"${DOCKER_VERSION}.*${UBUNTU_VERSION}\" | awk 'NR==1 {print $3}')\nfi\n\napt-get install -y -qq --no-install-recommends docker-ce=${EXACT_DOCKER_VERSION}\napt-mark hold docker-ce\ndocker version\n\n"
  },
  {
    "path": "scripts/kubeadm-install.sh",
    "content": "#!/usr/bin/env bash\n\nK8_VERSION=${1}\n\nset -e\n\ncurl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\necho \"deb http://apt.kubernetes.io/ kubernetes-xenial main\" | tee /etc/apt/sources.list.d/kubernetes.list\napt-get update -qq\n\ndeclare -a deps\nexport deps=(kubeadm kubelet kubectl cri-tools)\n\nfor dep in \"${deps[@]}\"; do\n\n  dep_version=$(echo \"${dep}_version\" | tr - _)\n  if [[ -z \"$(apt-cache madison ${dep} | grep ${K8_VERSION#\"stable-\"})\" ]]; then\n    export ${dep_version}=\"$(apt-cache madison \"${dep}\" | head -1 | awk '{print $3}')\"\n    echo -e \"\"\"\n\\033[33mWarning: ${dep} version ${K8_VERSION#\"stable-\"}.x is not available, \\\ninstalling ${dep} $(apt-cache madison \"${dep}\" | head -1 | awk '{print $3}') instead\\033[0m\n    \"\"\" && \\\n    sleep 2s\n  else\n    export ${dep_version}=\"$(apt-cache madison \"${dep}\" | grep \"${K8_VERSION#\"stable-\"}\" | head -1 | awk '{print $3}')\"\n  fi\n\n  apt-get install -qy --allow-downgrades \"${dep}\"=\"${!dep_version}\"\ndone\n\n"
  },
  {
    "path": "scripts/kubeadm-token.sh",
    "content": "#!/usr/bin/env bash\n\nset -e\n\n# Extract \"host\" and \"key_file\" argument from the input into HOST shell variable\neval \"$(jq -r '@sh \"HOST=\\(.host) KEY=\\(.key)\"')\"\n\n# Fetch the join command\nCMD=$(ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i $KEY \\\n    root@$HOST kubeadm token create --print-join-command)\n\n# Produce a JSON object containing the join command\njq -n --arg command \"$CMD\" '{\"command\":$command}'\n"
  },
  {
    "path": "scripts/kubectl-conf.sh",
    "content": "#!/usr/bin/env bash\n\nset -e\n\nWORKSPACE=$1\nPUBLIC_IP=$2\nPRIVATE_IP=$3\nKEY_FILE=$4\n\nscp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${KEY_FILE} root@${PUBLIC_IP}:/etc/kubernetes/admin.conf .\nsed -e \"s/${PRIVATE_IP}/${PUBLIC_IP}/g\" admin.conf > ${WORKSPACE}.conf\nrm admin.conf\n"
  },
  {
    "path": "scripts/monitoring-install.sh",
    "content": "#!/usr/bin/env bash\n\nset -e\n\nARCH=$1\n\nkubectl apply -f /tmp/dashboard-rbac.yaml\nkubectl apply -f /tmp/heapster-rbac.yaml\nkubectl apply -f /tmp/metrics-server-rbac.yaml\n\nif [ \"$ARCH\" == \"arm\" ]; then\n    curl -s https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard-arm.yaml | \\\n    sed -e 's/v2.0.0-alpha0/v1.8.3/g' | \\\n    kubectl apply -f -;\n    kubectl apply -f /tmp/heapster-arm.yaml;\n    kubectl apply -f /tmp/metrics-server-arm.yaml;\nelif [ \"$ARCH\" == \"x86_64\" ]; then\n    curl -s -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard.yaml | \\\n    sed -e 's/v2.0.0-alpha0/v1.8.3/g' | \\\n    kubectl apply -f -;\n    kubectl apply -f /tmp/heapster-amd64.yaml;\n    kubectl apply -f /tmp/metrics-server-amd64.yaml;\nfi\n"
  },
  {
    "path": "sg.tf",
    "content": "# Master\nresource \"scaleway_security_group\" \"master_security_group\" {\n  name        = \"sg.master.${terraform.workspace}\"\n  description = \"Master security group\"\n}\n\nresource \"scaleway_security_group_rule\" \"smtp_master_drop\" {\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  action    = \"drop\"\n  direction = \"inbound\"\n  ip_range  = \"0.0.0.0/0\"\n  protocol  = \"TCP\"\n  port      = 25\n}\n\nresource \"scaleway_security_group_rule\" \"ssh_master_accept\" {\n  count          = \"${length(var.ip_admin)}\"\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  action    = \"accept\"\n  direction = \"inbound\"\n  ip_range  = \"${element(var.ip_admin, count.index)}\"\n  protocol  = \"TCP\"\n  port      = 22\n}\n\nresource \"scaleway_security_group_rule\" \"https_master_dashboard_accept\" {\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  action    = \"accept\"\n  direction = \"inbound\"\n  ip_range  = \"${element(var.ip_admin, count.index)}\"\n  protocol  = \"TCP\"\n  port      = 8888\n}\n\nresource \"scaleway_security_group_rule\" \"https_master_management_accept\" {\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  action    = \"accept\"\n  direction = \"inbound\"\n  ip_range  = \"${element(var.ip_admin, count.index)}\"\n  protocol  = \"TCP\"\n  port      = 6443\n}\n\nresource \"scaleway_security_group_rule\" \"http_master_podinfo_accept\" {\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  action    = \"accept\"\n  direction = \"inbound\"\n  ip_range  = \"${element(var.ip_admin, count.index)}\"\n  protocol  = \"TCP\"\n  port      = 31190\n}\n\nresource \"scaleway_security_group_rule\" \"icmp_master_drop\" {\n  security_group = \"${scaleway_security_group.master_security_group.id}\"\n\n  action    = \"drop\"\n  direction = \"inbound\"\n  ip_range  = \"0.0.0.0/0\"\n  protocol  = \"ICMP\"\n}\n\n# Nodes\nresource \"scaleway_security_group\" \"node_security_group\" {\n  name        = \"sg.node.${terraform.workspace}\"\n  description = \"node security group\"\n}\n\nresource \"scaleway_security_group_rule\" \"ssh_node_accept\" {\n  security_group = \"${scaleway_security_group.node_security_group.id}\"\n\n  action    = \"accept\"\n  direction = \"inbound\"\n  ip_range  = \"${element(var.ip_admin, count.index)}\"\n  protocol  = \"TCP\"\n  port      = 22\n}\n\nresource \"scaleway_security_group_rule\" \"icmp_node_drop\" {\n  security_group = \"${scaleway_security_group.node_security_group.id}\"\n\n  action    = \"drop\"\n  direction = \"inbound\"\n  ip_range  = \"0.0.0.0/0\"\n  protocol  = \"ICMP\"\n}\n"
  },
  {
    "path": "terraform.tf",
    "content": "terraform {\n  required_version = \"<= 0.11.11\"\n}\n"
  },
  {
    "path": "variables.tf",
    "content": "variable \"ubuntu_version\" {\n  default = \"Ubuntu Xenial\"\n  description = <<EOT\n\nFor arm, choose from:\n  - Ubuntu Xenial\n\nFor x86_64, choose from:\n  - Ubuntu Xenial\n  - Ubuntu Bionic\n\nNotes:\n  - kubernetes only has xenial packages for debian\n  - currently arm is not working with ubuntu bionic (kubeadm init hangs)\n\nEOT\n}\n\nvariable \"docker_version\" {\n  default     = \"18.06\"\n  description = <<EOT\n\nSpecify the docker version either as\n\n  - Simplified 5 characters name such as:\n    - 17.03\n    - 18.06\n\n  - The exact release name such as:\n    - 17.03.0~ce-0~ubuntu-xenial\n    - 18.06.0~ce~3-0~ubuntu\n\nEOT\n}\n\nvariable \"k8s_version\" {\n  default = \"stable-1.13\"\n}\n\nvariable \"weave_passwd\" {\n  default = \"ChangeMe\"\n}\n\nvariable \"arch\" {\n  default     = \"arm\"\n  description = \"Values: arm arm64 x86_64\"\n}\n\nvariable \"region\" {\n  default     = \"par1\"\n  description = \"Values: par1 ams1\"\n}\n\nvariable \"server_type\" {\n  default     = \"C1\"\n  description = \"Use C1 for arm, ARM64-2GB for arm64 and C2S for x86_64\"\n}\n\nvariable \"server_type_node\" {\n  default     = \"C1\"\n  description = \"Use C1 for arm, ARM64-2GB for arm64 and C2S for x86_64\"\n}\n\nvariable \"nodes\" {\n  default = 2\n}\n\nvariable \"ip_admin\" {\n  type        = \"list\"\n  default     = [\"0.0.0.0/0\"]\n  description = \"IP access to services\"\n}\n\nvariable \"private_key\" {\n  type        = \"string\"\n  default     = \"~/.ssh/id_rsa\"\n  description = \"The path to your private key\"\n}\n\nvariable \"container_log_max_size\" {\n  default     = \"100Mi\"\n  description = \"The maximum file size for container logs, k8s 1.12+ only\"\n}\n\nvariable \"kubeadm_verbosity\" {\n  default     = \"0\"\n  description = \"The verbosity level of the kubeadm init logs\"\n}\n\nvariable \"kubelet_extra_args\" {\n  default = \"\"\n  description = \"Extra arguments used by kubelet systemd\"\n}\n"
  }
]