Repository: stefanprodan/k8s-scw-baremetal
Branch: master
Commit: 7829b914bc04
Files: 24
Total size: 32.9 KB

Directory structure:
gitextract_qqp53q9f/

├── .gitignore
├── LICENSE
├── README.md
├── addons/
│   ├── dashboard-rbac.yaml
│   ├── heapster-amd64.yaml
│   ├── heapster-arm.yaml
│   ├── heapster-rbac.yaml
│   ├── metrics-server-amd64.yaml
│   ├── metrics-server-arm.yaml
│   └── metrics-server-rbac.yaml
├── kubeadm/
│   ├── v1alpha3-config.yaml
│   └── v1beta1-config.yaml
├── main.tf
├── master.tf
├── nodes.tf
├── outputs.tf
├── scripts/
│   ├── docker-install.sh
│   ├── kubeadm-install.sh
│   ├── kubeadm-token.sh
│   ├── kubectl-conf.sh
│   └── monitoring-install.sh
├── sg.tf
├── terraform.tf
└── variables.tf

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
# Compiled files
*.tfstate
*.tfstate.backup
*.terraform.tfstate.lock.info

# Module directory
.terraform/

# IDE
.idea
.vscode
.DS_Store

# K8S
*.conf


================================================
FILE: LICENSE
================================================
MIT License

Copyright (c) 2017 Stefan Prodan

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


================================================
FILE: README.md
================================================
# k8s-scw-baremetal

Kubernetes Terraform installer for Scaleway bare-metal ARM and AMD64

### Initial setup

Clone the repository and install the dependencies:

```bash
$ git clone https://github.com/stefanprodan/k8s-scw-baremetal.git
$ cd k8s-scw-baremetal
$ terraform init
```

Note that you'll need Terraform v0.10 or newer to run this project.

Before running the project you'll have to create an access token for Terraform to connect to the Scaleway API

Now retrieve the `<ORGANIZATION_ID>` using your `<ACCESS-TOKEN>` from `/organizations` API endpoint:

```bash
$ curl https://account.scaleway.com/organizations -H "X-Auth-Token: <ACCESS-TOKEN>"
```

Sample output (excerpt with organization ID):
```bash
"organizations": [{"id": "xxxxxxxxxxxxx", "name": "Organization Name"}],
```

Using the token and your organization ID, create two environment variables:

```bash
$ export SCALEWAY_ORGANIZATION="<ORGANIZATION_ID>"
$ export SCALEWAY_TOKEN="<ACCESS-TOKEN>"
```

To configure your cluster, you'll need to have `jq` installed on your computer.

### Usage

Create an AMD64 bare-metal Kubernetes cluster with one master and a node:

```bash
$ terraform workspace new amd64

$ terraform apply \
 -var region=par1 \
 -var arch=x86_64 \
 -var server_type=C2S \
 -var nodes=1 \
 -var server_type_node=C2S \
 -var weave_passwd=ChangeMe \
 -var docker_version=18.06 \
 -var ubuntu_version="Ubuntu Bionic"
```

This will do the following:

* reserves public IPs for each server
* provisions three bare-metal servers with Ubuntu 16.04.1 LTS (the size of the `master` and the `node` may be different but must remain in the same type of architecture)
* connects to the master server via SSH and installs Docker CE and kubeadm apt packages
* runs kubeadm init on the master server and configures kubectl
* downloads the kubectl admin config file on your local machine and replaces the private IP with the public one
* creates a Kubernetes secret with the Weave Net password
* installs Weave Net with encrypted overlay
* installs cluster add-ons (Kubernetes dashboard, metrics server and Heapster)
* starts the nodes in parallel and installs Docker CE and kubeadm
* joins the nodes in the cluster using the kubeadm token obtained from the master

Scale up by increasing the number of nodes:

```bash
$ terraform apply \
 -var nodes=3
```

Tear down the whole infrastructure with:

```bash
terraform destroy -force
```

Create an ARMv7 bare-metal Kubernetes cluster with one master and two nodes:

```bash
$ terraform workspace new arm

$ terraform apply \
 -var region=par1 \
 -var arch=arm \
 -var server_type=C1 \
 -var nodes=2 \
 -var server_type_node=C1 \
 -var weave_passwd=ChangeMe \
 -var docker_version=18.06 \
 -var ubuntu_version="Ubuntu Xenial"
```

### Remote control

After applying the Terraform plan you'll see several output variables like the master public IP,
the kubeadmn join command and the current workspace admin config.

In order to run `kubectl` commands against the Scaleway cluster you can use the `kubectl_config` output variable:

Check if Heapster works:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
  top nodes

NAME           CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
arm-master-1   655m         16%       873Mi           45%
arm-node-1     147m         3%        618Mi           32%
arm-node-2     101m         2%        584Mi           30%
```

The `kubectl` config file format is `<WORKSPACE>.conf` as in `arm.conf` or `amd64.conf`.

In order to access the dashboard you can use port forward:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
  -n kube-system port-forward deployment/kubernetes-dashboard 8888:9090
```

Now you can access the dashboard on your computer at `http://localhost:8888`.

![Overview](https://github.com/stefanprodan/k8s-scw-baremetal/blob/master/screens/dash-overview.png)

![Nodes](https://github.com/stefanprodan/k8s-scw-baremetal/blob/master/screens/dash-nodes.png)

### Expose services outside the cluster

Since we're running on bare-metal and Scaleway doesn't offer a load balancer, the easiest way to expose
applications outside of Kubernetes is using a NodePort service.

Let's deploy the [podinfo](https://github.com/stefanprodan/k8s-podinfo) app in the default namespace.
Podinfo has a multi-arch Docker image and it will work on arm, arm64 or amd64.

Create the podinfo nodeport service:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
  apply -f https://raw.githubusercontent.com/stefanprodan/k8s-podinfo/7a8506e60fca086572f16de57f87bf5430e2df48/deploy/podinfo-svc-nodeport.yaml
 
service "podinfo-nodeport" created
```

Create the podinfo deployment:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
  apply -f https://raw.githubusercontent.com/stefanprodan/k8s-podinfo/7a8506e60fca086572f16de57f87bf5430e2df48/deploy/podinfo-dep.yaml

deployment "podinfo" created
```

Inspect the podinfo service to obtain the port number:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
  get svc --selector=app=podinfo

NAME               TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
podinfo-nodeport   NodePort   10.104.132.14   <none>        9898:31190/TCP   3m
```

You can access podinfo at `http://<MASTER_PUBLIC_IP>:31190` or using curl:

```bash
$ curl http://$(terraform output k8s_master_public_ip):31190

runtime:
  arch: arm
  max_procs: "4"
  num_cpu: "4"
  num_goroutine: "12"
  os: linux
  version: go1.9.2
labels:
  app: podinfo
  pod-template-hash: "1847780700"
annotations:
  kubernetes.io/config.seen: 2018-01-08T00:39:45.580597397Z
  kubernetes.io/config.source: api
environment:
  HOME: /root
  HOSTNAME: podinfo-5d8ccd4c44-zrczc
  KUBERNETES_PORT: tcp://10.96.0.1:443
  KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
  KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
  KUBERNETES_PORT_443_TCP_PORT: "443"
  KUBERNETES_PORT_443_TCP_PROTO: tcp
  KUBERNETES_SERVICE_HOST: 10.96.0.1
  KUBERNETES_SERVICE_PORT: "443"
  KUBERNETES_SERVICE_PORT_HTTPS: "443"
  PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
externalIP:
  IPv4: 163.172.139.112
```

### Horizontal Pod Autoscaling

Starting from Kubernetes 1.9 `kube-controller-manager` is configured by default with
`horizontal-pod-autoscaler-use-rest-clients`.
In order to use HPA we need to install the metrics server to enable the new metrics API used by HPA v2.
Both Heapster and the metrics server have been deployed from Terraform
when the master node was provisioned.

The metric server collects resource usage data from each node using Kubelet Summary API.
Check if the metrics server is running:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
 get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq
```

```json
{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": [
    {
      "metadata": {
        "name": "arm-master-1",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/arm-master-1",
        "creationTimestamp": "2018-01-08T15:17:09Z"
      },
      "timestamp": "2018-01-08T15:17:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "384m",
        "memory": "935792Ki"
      }
    },
    {
      "metadata": {
        "name": "arm-node-1",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/arm-node-1",
        "creationTimestamp": "2018-01-08T15:17:09Z"
      },
      "timestamp": "2018-01-08T15:17:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "130m",
        "memory": "649020Ki"
      }
    },
    {
      "metadata": {
        "name": "arm-node-2",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/arm-node-2",
        "creationTimestamp": "2018-01-08T15:17:09Z"
      },
      "timestamp": "2018-01-08T15:17:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "120m",
        "memory": "614180Ki"
      }
    }
  ]
}
```

Let's define a HPA that will maintain a minimum of two replicas and will scale up to ten
if the CPU average is over 80% or if the memory goes over 200Mi.

```yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: podinfo
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: podinfo
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 200Mi
```

Apply the podinfo HPA:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) \
  apply -f https://raw.githubusercontent.com/stefanprodan/k8s-podinfo/7a8506e60fca086572f16de57f87bf5430e2df48/deploy/podinfo-hpa.yaml

horizontalpodautoscaler "podinfo" created
```

After a couple of seconds the HPA controller will contact the metrics server and will fetch the CPU
and memory usage:

```bash
$ kubectl --kubeconfig ./$(terraform output kubectl_config) get hpa

NAME      REFERENCE            TARGETS                      MINPODS   MAXPODS   REPLICAS   AGE
podinfo   Deployment/podinfo   2826240 / 200Mi, 15% / 80%   2         10        2          5m
```

In order to increase the CPU usage we could run a load test with hey:

```bash
#install hey
go get -u github.com/rakyll/hey

#do 10K requests rate limited at 20 QPS
hey -n 10000 -q 10 -c 5 http://$(terraform output k8s_master_public_ip):31190
```

You can monitor the autoscaler events with:

```bash
$ watch -n 5 kubectl --kubeconfig ./$(terraform output kubectl_config) describe hpa

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  7m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
```

After the load tests finishes the autoscaler will remove replicas until the deployment reaches the initial replica count:

```
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  20m   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  16m   horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  12m   horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  6m    horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
```


================================================
FILE: addons/dashboard-rbac.yaml
================================================
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system


================================================
FILE: addons/heapster-amd64.yaml
================================================
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: k8s.gcr.io/heapster-amd64:v1.5.0
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes.summary_api:''


================================================
FILE: addons/heapster-arm.yaml
================================================
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: k8s.gcr.io/heapster-arm:v1.5.0
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes.summary_api:''


================================================
FILE: addons/heapster-rbac.yaml
================================================
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: heapster
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster


================================================
FILE: addons/metrics-server-amd64.yaml
================================================
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true



================================================
FILE: addons/metrics-server-arm.yaml
================================================
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-arm:v0.2.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true


================================================
FILE: addons/metrics-server-rbac.yaml
================================================
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443


================================================
FILE: kubeadm/v1alpha3-config.yaml
================================================
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
apiEndpoint:
  advertiseAddress: CONFIG_CLUSTER_PRIVATE_IP
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    "feature-gates": "BlockVolume=true,CRIContainerLogRotation=true"


---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: CONFIG_KUBERNETES_VERSION
apiServerCertSANs:
  - CONFIG_CLUSTER_PUBLIC_IP
apiServerExtraArgs:
  authorization-mode: "Node,RBAC"
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
imageRepository: k8s.gcr.io


---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxFiles: 1
containerLogMaxSize: CONFIG_CONTAINER_LOG_MAX_SIZE
maxPods: 110
featureGates:
  BlockVolume: true
  CRIContainerLogRotation: true
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook


---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration


# ---
# apiVersion: kubeadm.k8s.io/v1alpha3
# kind: JoinConfiguration



================================================
FILE: kubeadm/v1beta1-config.yaml
================================================
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: CONFIG_CLUSTER_PRIVATE_IP
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    "feature-gates": "BlockVolume=true,CRIContainerLogRotation=true"


---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: CONFIG_KUBERNETES_VERSION
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
  certSANs:
    - CONFIG_CLUSTER_PUBLIC_IP
  timeoutForControlPlane: 4m0s
controlPlaneEndpoint: CONFIG_CLUSTER_PRIVATE_IP:6443
controllerManager:
  extraArgs:
    "node-cidr-mask-size": "20"
scheduler:
  extraArgs:
    address: CONFIG_CLUSTER_PRIVATE_IP
certificatesDir: /etc/kubernetes/pki
imageRepository: k8s.gcr.io
useHyperKubeImage: false


---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxFiles: 1
containerLogMaxSize: CONFIG_CONTAINER_LOG_MAX_SIZE
maxPods: 110
featureGates:
  BlockVolume: true
  CRIContainerLogRotation: true
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook


---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration


# ---
# apiVersion: kubeadm.k8s.io/v1beta1
# kind: JoinConfiguration


================================================
FILE: main.tf
================================================
provider "scaleway" {
  region  = "${var.region}"
  version = "1.8.0"
}

provider "external" {
  version = "1.0.0"
}

data "scaleway_image" "ubuntu" {
  architecture = "${var.arch}"
  name         = "${var.ubuntu_version}"
}


================================================
FILE: master.tf
================================================
resource "scaleway_ip" "k8s_master_ip" {
  count = 1
}

resource "scaleway_server" "k8s_master" {
  count          = 1
  name           = "${terraform.workspace}-master-${count.index + 1}"
  image          = "${data.scaleway_image.ubuntu.id}"
  type           = "${var.server_type}"
  public_ip      = "${element(scaleway_ip.k8s_master_ip.*.ip, count.index)}"
  security_group = "${scaleway_security_group.master_security_group.id}"

  connection {
    type        = "ssh"
    user        = "root"
    private_key = "${file(var.private_key)}"
  }
  provisioner "file" {
    source      = "scripts/"
    destination = "/tmp"
  }
  provisioner "file" {
    source      = "addons/"
    destination = "/tmp"
  }
  provisioner "file" {
    source      = "kubeadm"
    destination = "/tmp/"
  }
  provisioner "remote-exec" {
    inline = [
      <<EOT
#!/bin/bash
set -e
chmod +x /tmp/docker-install.sh
chmod +x /tmp/kubeadm-install.sh
chmod g+w -R /tmp/kubeadm/

export ubuntu_version=$(echo -n ${var.ubuntu_version} | cut -d " " -f 2 | awk '{print tolower($0)}')
/tmp/docker-install.sh $${ubuntu_version} ${var.arch} ${var.docker_version} && \
/tmp/kubeadm-install.sh ${var.k8s_version} && \

modify_kube_apiserver_config(){
  while [[ ! -e /etc/kubernetes/manifests/kube-apiserver.yaml ]]; do
    sleep 0.5s;
  done && \
  sed -i 's/failureThreshold: [0-9]/failureThreshold: 18/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \
  sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml && \
  sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 240/g' /etc/kubernetes/manifests/kube-apiserver.yaml
}

# ref https://github.com/kubernetes/kubeadm/issues/413 (initialDelaySeconds is too eager)
if [[ ${var.arch} == "arm" ]]; then modify_kube_apiserver_config & fi

export KUBEADM_VERSION=$(apt-cache madison kubeadm | grep $(echo ${var.k8s_version} | cut -c8-) | \
  awk 'NR==1 {print $3}' | rev | cut -c4- | rev)

dpkg --compare-versions "$${KUBEADM_VERSION}" lt 1.13 && \
  export KUBEADM_CONFIG_FILE=/tmp/kubeadm/v1alpha3-config.yaml || \
  export KUBEADM_CONFIG_FILE=/tmp/kubeadm/v1beta1-config.yaml

dpkg --compare-versions "$${KUBEADM_VERSION}" lt 1.12 && \
  export KUBEADM_CONFIG_FILE=""

dpkg --compare-versions "$${KUBEADM_VERSION}" lt 1.11 && \
  export VERBOSITY_EXTRA_ARGS='' || \
  export VERBOSITY_EXTRA_ARGS='--v ${var.kubeadm_verbosity}'

if [[ -z "$${KUBEADM_CONFIG_FILE}" ]]; then
  kubeadm init \
    --apiserver-advertise-address=${self.private_ip} \
    --apiserver-cert-extra-sans=${self.public_ip} \
    --kubernetes-version=${var.k8s_version} \
    --ignore-preflight-errors=KubeletVersion \
     $${VERBOSITY_EXTRA_ARGS};
else
  sed -i 's/CONFIG_CLUSTER_PUBLIC_IP/${self.public_ip}/g' $${KUBEADM_CONFIG_FILE} && \
  sed -i 's/CONFIG_CLUSTER_PRIVATE_IP/${self.private_ip}/g' $${KUBEADM_CONFIG_FILE} && \
  sed -i "s/CONFIG_KUBERNETES_VERSION/v$${KUBEADM_VERSION}/g" $${KUBEADM_CONFIG_FILE} && \
  sed -i "s/CONFIG_CONTAINER_LOG_MAX_SIZE/${var.container_log_max_size}/" $${KUBEADM_CONFIG_FILE}

  kubeadm init \
    --ignore-preflight-errors=KubeletVersion \
    --config=$${KUBEADM_CONFIG_FILE} \
     $${VERBOSITY_EXTRA_ARGS};
fi && \

mkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && \
kubectl create secret -n kube-system generic weave-passwd --from-literal=weave-passwd=${var.weave_passwd} && \
kubectl apply -f "https://cloud.weave.works/k8s/net?password-secret=weave-passwd&k8s-version=$(kubectl version | base64 | tr -d '\n')" && \
chmod +x /tmp/monitoring-install.sh && /tmp/monitoring-install.sh ${var.arch}
EOT
    ]
  }
  provisioner "local-exec" {
    command    = "./scripts/kubectl-conf.sh ${terraform.workspace} ${self.public_ip} ${self.private_ip} ${var.private_key}"
    on_failure = "continue"
  }
}

data "external" "kubeadm_join" {
  program = ["./scripts/kubeadm-token.sh"]

  query = {
    host = "${scaleway_ip.k8s_master_ip.0.ip}"
    key = "${var.private_key}"
  }

  depends_on = ["scaleway_server.k8s_master"]
}


================================================
FILE: nodes.tf
================================================
resource "scaleway_ip" "k8s_node_ip" {
  count = "${var.nodes}"
}

resource "scaleway_server" "k8s_node" {
  count          = "${var.nodes}"
  name           = "${terraform.workspace}-node-${count.index + 1}"
  image          = "${data.scaleway_image.ubuntu.id}"
  type           = "${var.server_type_node}"
  public_ip      = "${element(scaleway_ip.k8s_node_ip.*.ip, count.index)}"
  security_group = "${scaleway_security_group.node_security_group.id}"

  connection {
    type        = "ssh"
    user        = "root"
    private_key = "${file(var.private_key)}"
  }
  provisioner "file" {
    source      = "scripts/docker-install.sh"
    destination = "/tmp/docker-install.sh"
  }
  provisioner "file" {
    source      = "scripts/kubeadm-install.sh"
    destination = "/tmp/kubeadm-install.sh"
  }
  provisioner "remote-exec" {
    inline = [
      "set -e",
      "export ubuntu_version=$(echo -n ${var.ubuntu_version} | cut -d \" \" -f 2 | awk '{print tolower($0)}')",
      "chmod +x /tmp/docker-install.sh && /tmp/docker-install.sh $${ubuntu_version} ${var.arch} ${var.docker_version}",
      "chmod +x /tmp/kubeadm-install.sh && /tmp/kubeadm-install.sh ${var.k8s_version}",
      "echo 'KUBELET_EXTRA_ARGS=${var.kubelet_extra_args}' > /etc/default/kubelet",
      "${data.external.kubeadm_join.result.command}",
    ]
  }
  provisioner "remote-exec" {
    inline = [
      "kubectl get pods --all-namespaces",
    ]

    on_failure = "continue"

    connection {
      type = "ssh"
      user = "root"
      host = "${scaleway_ip.k8s_master_ip.0.ip}"
    }
  }
}


================================================
FILE: outputs.tf
================================================
output "k8s_master_public_ip" {
  value = "${scaleway_ip.k8s_master_ip.0.ip}"
}

output "kubeadm_join_command" {
  value = "${data.external.kubeadm_join.result["command"]}"
}

output "nodes_public_ip" {
  value = "${concat(scaleway_server.k8s_node.*.name, scaleway_server.k8s_node.*.public_ip)}"
}

output "kubectl_config" {
  value = "${terraform.workspace}.conf"
}


================================================
FILE: scripts/docker-install.sh
================================================
#!/usr/bin/env bash

set -e

UBUNTU_VERSION=$1
ARCH=$2
DOCKER_VERSION=$3

if [[ ${ARCH} == "arm" ]]; then export ARCH=armhf; fi
if [[ ${ARCH} == "x86_64" ]]; then export ARCH=amd64; fi

apt-get update -qq
apt-get install -y -qq apt-transport-https ca-certificates curl git
curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq -
echo "deb [arch=${ARCH}] https://download.docker.com/linux/ubuntu ${UBUNTU_VERSION} stable" | \
  tee /etc/apt/sources.list.d/docker.list
apt-get update -qq

if (( $(echo -n ${DOCKER_VERSION} | wc -c) > 5 )); then
  export EXACT_DOCKER_VERSION=${DOCKER_VERSION}
else
  export EXACT_DOCKER_VERSION=$(apt-cache madison docker-ce | \
    grep "${DOCKER_VERSION}.*${UBUNTU_VERSION}" | awk 'NR==1 {print $3}')
fi

apt-get install -y -qq --no-install-recommends docker-ce=${EXACT_DOCKER_VERSION}
apt-mark hold docker-ce
docker version



================================================
FILE: scripts/kubeadm-install.sh
================================================
#!/usr/bin/env bash

K8_VERSION=${1}

set -e

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq

declare -a deps
export deps=(kubeadm kubelet kubectl cri-tools)

for dep in "${deps[@]}"; do

  dep_version=$(echo "${dep}_version" | tr - _)
  if [[ -z "$(apt-cache madison ${dep} | grep ${K8_VERSION#"stable-"})" ]]; then
    export ${dep_version}="$(apt-cache madison "${dep}" | head -1 | awk '{print $3}')"
    echo -e """
\033[33mWarning: ${dep} version ${K8_VERSION#"stable-"}.x is not available, \
installing ${dep} $(apt-cache madison "${dep}" | head -1 | awk '{print $3}') instead\033[0m
    """ && \
    sleep 2s
  else
    export ${dep_version}="$(apt-cache madison "${dep}" | grep "${K8_VERSION#"stable-"}" | head -1 | awk '{print $3}')"
  fi

  apt-get install -qy --allow-downgrades "${dep}"="${!dep_version}"
done



================================================
FILE: scripts/kubeadm-token.sh
================================================
#!/usr/bin/env bash

set -e

# Extract "host" and "key_file" argument from the input into HOST shell variable
eval "$(jq -r '@sh "HOST=\(.host) KEY=\(.key)"')"

# Fetch the join command
CMD=$(ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i $KEY \
    root@$HOST kubeadm token create --print-join-command)

# Produce a JSON object containing the join command
jq -n --arg command "$CMD" '{"command":$command}'


================================================
FILE: scripts/kubectl-conf.sh
================================================
#!/usr/bin/env bash

set -e

WORKSPACE=$1
PUBLIC_IP=$2
PRIVATE_IP=$3
KEY_FILE=$4

scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${KEY_FILE} root@${PUBLIC_IP}:/etc/kubernetes/admin.conf .
sed -e "s/${PRIVATE_IP}/${PUBLIC_IP}/g" admin.conf > ${WORKSPACE}.conf
rm admin.conf


================================================
FILE: scripts/monitoring-install.sh
================================================
#!/usr/bin/env bash

set -e

ARCH=$1

kubectl apply -f /tmp/dashboard-rbac.yaml
kubectl apply -f /tmp/heapster-rbac.yaml
kubectl apply -f /tmp/metrics-server-rbac.yaml

if [ "$ARCH" == "arm" ]; then
    curl -s https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard-arm.yaml | \
    sed -e 's/v2.0.0-alpha0/v1.8.3/g' | \
    kubectl apply -f -;
    kubectl apply -f /tmp/heapster-arm.yaml;
    kubectl apply -f /tmp/metrics-server-arm.yaml;
elif [ "$ARCH" == "x86_64" ]; then
    curl -s -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/alternative/kubernetes-dashboard.yaml | \
    sed -e 's/v2.0.0-alpha0/v1.8.3/g' | \
    kubectl apply -f -;
    kubectl apply -f /tmp/heapster-amd64.yaml;
    kubectl apply -f /tmp/metrics-server-amd64.yaml;
fi


================================================
FILE: sg.tf
================================================
# Master
resource "scaleway_security_group" "master_security_group" {
  name        = "sg.master.${terraform.workspace}"
  description = "Master security group"
}

resource "scaleway_security_group_rule" "smtp_master_drop" {
  security_group = "${scaleway_security_group.master_security_group.id}"

  action    = "drop"
  direction = "inbound"
  ip_range  = "0.0.0.0/0"
  protocol  = "TCP"
  port      = 25
}

resource "scaleway_security_group_rule" "ssh_master_accept" {
  count          = "${length(var.ip_admin)}"
  security_group = "${scaleway_security_group.master_security_group.id}"

  action    = "accept"
  direction = "inbound"
  ip_range  = "${element(var.ip_admin, count.index)}"
  protocol  = "TCP"
  port      = 22
}

resource "scaleway_security_group_rule" "https_master_dashboard_accept" {
  security_group = "${scaleway_security_group.master_security_group.id}"

  action    = "accept"
  direction = "inbound"
  ip_range  = "${element(var.ip_admin, count.index)}"
  protocol  = "TCP"
  port      = 8888
}

resource "scaleway_security_group_rule" "https_master_management_accept" {
  security_group = "${scaleway_security_group.master_security_group.id}"

  action    = "accept"
  direction = "inbound"
  ip_range  = "${element(var.ip_admin, count.index)}"
  protocol  = "TCP"
  port      = 6443
}

resource "scaleway_security_group_rule" "http_master_podinfo_accept" {
  security_group = "${scaleway_security_group.master_security_group.id}"

  action    = "accept"
  direction = "inbound"
  ip_range  = "${element(var.ip_admin, count.index)}"
  protocol  = "TCP"
  port      = 31190
}

resource "scaleway_security_group_rule" "icmp_master_drop" {
  security_group = "${scaleway_security_group.master_security_group.id}"

  action    = "drop"
  direction = "inbound"
  ip_range  = "0.0.0.0/0"
  protocol  = "ICMP"
}

# Nodes
resource "scaleway_security_group" "node_security_group" {
  name        = "sg.node.${terraform.workspace}"
  description = "node security group"
}

resource "scaleway_security_group_rule" "ssh_node_accept" {
  security_group = "${scaleway_security_group.node_security_group.id}"

  action    = "accept"
  direction = "inbound"
  ip_range  = "${element(var.ip_admin, count.index)}"
  protocol  = "TCP"
  port      = 22
}

resource "scaleway_security_group_rule" "icmp_node_drop" {
  security_group = "${scaleway_security_group.node_security_group.id}"

  action    = "drop"
  direction = "inbound"
  ip_range  = "0.0.0.0/0"
  protocol  = "ICMP"
}


================================================
FILE: terraform.tf
================================================
terraform {
  required_version = "<= 0.11.11"
}


================================================
FILE: variables.tf
================================================
variable "ubuntu_version" {
  default = "Ubuntu Xenial"
  description = <<EOT

For arm, choose from:
  - Ubuntu Xenial

For x86_64, choose from:
  - Ubuntu Xenial
  - Ubuntu Bionic

Notes:
  - kubernetes only has xenial packages for debian
  - currently arm is not working with ubuntu bionic (kubeadm init hangs)

EOT
}

variable "docker_version" {
  default     = "18.06"
  description = <<EOT

Specify the docker version either as

  - Simplified 5 characters name such as:
    - 17.03
    - 18.06

  - The exact release name such as:
    - 17.03.0~ce-0~ubuntu-xenial
    - 18.06.0~ce~3-0~ubuntu

EOT
}

variable "k8s_version" {
  default = "stable-1.13"
}

variable "weave_passwd" {
  default = "ChangeMe"
}

variable "arch" {
  default     = "arm"
  description = "Values: arm arm64 x86_64"
}

variable "region" {
  default     = "par1"
  description = "Values: par1 ams1"
}

variable "server_type" {
  default     = "C1"
  description = "Use C1 for arm, ARM64-2GB for arm64 and C2S for x86_64"
}

variable "server_type_node" {
  default     = "C1"
  description = "Use C1 for arm, ARM64-2GB for arm64 and C2S for x86_64"
}

variable "nodes" {
  default = 2
}

variable "ip_admin" {
  type        = "list"
  default     = ["0.0.0.0/0"]
  description = "IP access to services"
}

variable "private_key" {
  type        = "string"
  default     = "~/.ssh/id_rsa"
  description = "The path to your private key"
}

variable "container_log_max_size" {
  default     = "100Mi"
  description = "The maximum file size for container logs, k8s 1.12+ only"
}

variable "kubeadm_verbosity" {
  default     = "0"
  description = "The verbosity level of the kubeadm init logs"
}

variable "kubelet_extra_args" {
  default = ""
  description = "Extra arguments used by kubelet systemd"
}