Repository: yanc0/untrak
Branch: master
Commit: 163605dec1e1
Files: 19
Total size: 16.6 KB

Directory structure:
gitextract_7ejnjs7d/

├── .gitignore
├── .travis.yml
├── CHANGELOG.md
├── LICENSE
├── README.md
├── config/
│   ├── loader.go
│   └── structs.go
├── example/
│   └── manifests/
│       ├── resources_in.yaml
│       └── resources_out.yaml
├── go.mod
├── go.sum
├── kubernetes/
│   ├── non_namespaced.go
│   └── structs.go
├── main.go
├── outputs/
│   ├── text.go
│   └── yaml.go
├── untrak.yaml
└── utils/
    ├── commands.go
    └── strings.go

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
# untrak binary
untrak

# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, build with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out


================================================
FILE: .travis.yml
================================================
language: go
go:
- 1.14.x
script:
  - go get -v -d ./...
  - GOOS=linux GOARCH=amd64 go build -o untrak-linux
  - GOOS=darwin GOARCH=amd64 go build -o untrak-darwin
deploy:
  provider: releases
  api_key:
    secure: u76RGeXFqXOBTOiEFChuXS7C1aaLl7vX8J39hOyla22LAyIeSOmOK6atwMfz6zq3cui9wo/bQ6Mke3p5Fzm1bhvA1srLo+Ma59JYEzsMvyDE0A7OZG7W8S+iJYEn/rlLGBzjAozg0TlmOMh8s/H05TyUEVOOqFwlEyb/O4H4Zg/ooJdNv0A+MLYGqgCiyyDv2tuMzY6GNeiBLrR69iAJfEjkrOfzC+dnVKEHi9K6OV1UEWX/63VHd2SfvJmG9vOL++3QH05ToMP0Geuek0fvQ0c5EYjAZYW4d4zw3dxXsHLwN7GRjQuos0uGXGqeHaSkYfSbtDQhLNV3FRjXlp8r4fhjhHcWQm4GO5tZItMMWVFwA8S1v+aiZ4B2NwrewwOxRzyBqtYGV16yklkHngOwsbU3596TKQeWv9v7sCw7f1N+uoAYmbyf7Si95WZnvkdIW1DvADg/32+dOcsLOKWFo1wlkoA44O8qOUetfHFd9DDApvd1n6XfvsWkejOdk8X02R/1XB3fCFRxONegIsVXjhlodBS0AzoF398goDfiDtj9TCdOwHUMIGFGZ/SARJRFwkCNkzY46B1/P4yH/vzrxIsALAuPDGUjqfrERtpl289M/0SNz4j7hhDgVYq+BLzipErDMmT+3l+RRTR9rKcHWN2NNm2zkDNoNRVvyiZtpp0=
  file:
    - untrak-linux
    - untrak-darwin
  skip_cleanup: true
  on:
    tags: true
    all_branches: true


================================================
FILE: CHANGELOG.md
================================================
# Untrak Changelog

## v0.2.0 - 2020-10-22
Thanks to @almariah !

* Support building osx binaries on tags (#4)
* Compare only name and kind if the resourceIn is not namespaced (#4)
* Support flag to fail on untracked resources (#4)
* Add support for env variables to be substitute in args (#4)

## v0.1.1 - 2020-05-27
* Fix yaml parsing bug

## v0.1.0 - 2019-03-26
* Initial Release

================================================
FILE: LICENSE
================================================
MIT License

Copyright (c) 2019 Yann Coleu

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


================================================
FILE: README.md
================================================
# Untrak
Find untracked resources in Kubernetes cluster, garbage collect them.

[![Build Status](https://travis-ci.org/yanc0/untrak.svg?branch=master)](https://travis-ci.org/yanc0/untrak)

## Why?

When you use `kubectl apply`, `kustomize build` or `helm template` for injecting manifests through you CI/CD pipeline, kubernetes doesn't know when an object has been deleted from your repo. Your resources are now **untracked** from your delivery process and they are still managed by your clusters.

Untrak is a tool made for finding and deleting these untracked files on your cluster.


## How it works?

![untrak-schema.png](docs/untrak-schema.png)

Via a simple config file (`untrak.yaml`), this tool will internally execute commands that output YAMLs and find resources in your clusters that are not in your SCM anymore.

In a GitOps context, this is the tool you always dreamed of.

## Installation

Download latest version on [releases page](https://github.com/yanc0/untrak/releases)

- `chmod +x untrak`
- `sudo mv untrak /usr/local/bin`
- `untrak --help`

## Example

Put a `untrak.yaml` file in you SCM.

> Note: if you have multiple environments, you would need multiple untrak config files.
```yaml
# untrak.yaml
## git sources
in:
- cmd: "cat"
  args: ["example/manifests/resources.yaml"]

## cluster manifests
out:
- cmd: "kubectl"
  args: ["get", "cm,deploy,svc,ing", "-o", "yaml", "-n", "api"]

exclude:
- namespace
```
To show untracked resources in your cluster (out) simply launch `untrak` like so:

```
$ untrak -c untrak.yaml -o text
- api/ConfigMap/django-config-b4k42gm792
- api/ConfigMap/django-config-g55mctg456
- api/Ingress/my-ingress
```

If your manifests have the namespace set to non-namespaced resource, untrak will skip the namespace. A list of supported non-namespaced resource types that will be skipped are defined by default. If you have installed more non-namespaced resource types (eg., `CRDs`), you could add extra resource types to skip namespace in comparison:
```yaml
# untrak.yaml
## git sources
in:
...

## cluster manifests
out:
...

nonNamespaced:
- some_crd_type
```

You can use environment variables on command arguments (in/out):
```yaml
in:
- cmd: "cat"
  args: ["example/$SOME_FILE_NAME"]
...
```

If you need to garbage collect them, you can change the output format to yaml and pipe the result in kubectl:

```
$ untrak -c untrak.yaml -o yaml | kubectl delete -f -
configmap "django-config-b4k42gm792" deleted
configmap "django-config-g55mctg456" deleted
ingress.extensions "my-ingress" deleted
```

If you want to fail on untracked resources (exit status 1), you can use `-fail`:

```
$ untrak -c untrak.yaml -fail
```

> **Caution**: please test this tool extensively before deleting resources. The software is provided "as is", without warranty of any kind.


================================================
FILE: config/loader.go
================================================
package config

import (
	"io/ioutil"

	yaml "gopkg.in/yaml.v2"
)

// Load untrak config from path
func Load(path string) (*Config, error) {
	var cfg Config

	content, err := ioutil.ReadFile(path)
	if err != nil {
		return nil, err
	}

	err = yaml.Unmarshal(content, &cfg)
	if err != nil {
		return nil, err
	}

	return &cfg, nil
}


================================================
FILE: config/structs.go
================================================
package config

type CommandConfig struct {
	Cmd  string   `yaml:"cmd"`
	Args []string `yaml:"args"`
}

type Config struct {
	In            []*CommandConfig `yaml:"in"`
	Out           []*CommandConfig `yaml:"out"`
	Exclude       []string         `yaml:"exclude"`
	NonNamespaced []string         `yaml:"nonNamespaced"`
}


================================================
FILE: example/manifests/resources_in.yaml
================================================
---
apiVersion: v1
kind: Namespace
metadata:
  name: app
  namespace: app
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: django
    env: dev
  name: django
  namespace: api
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8000
  selector:
    app: django
    env: dev
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: django
    env: dev
  name: django
  namespace: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django
      env: dev
  template:
    metadata:
      labels:
        app: django
        env: dev
    spec:
      containers:
      - image: eu.gcr.io/example/django
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 20
          httpGet:
            path: /liveliness
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 3
          timeoutSeconds: 5
        name: django
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /readiness
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 5
---

================================================
FILE: example/manifests/resources_out.yaml
================================================
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: django
    env: dev
  name: django
  namespace: api
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8000
  selector:
    app: django
    env: dev
  type: ClusterIP
---
---
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: django
    env: dev
  name: django
  namespace: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: django
      env: dev
  template:
    metadata:
      labels:
        app: django
        env: dev
    spec:
      containers:
      - image: eu.gcr.io/example/django
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 20
          httpGet:
            path: /liveliness
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 3
          timeoutSeconds: 5
        name: django
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /readiness
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 5
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: django
    env: dev
  name: django
  namespace: api
spec:
  rules:
  - host: django.example.com
    http:
      paths:
      - backend:
          serviceName: django
          servicePort: 80
        path: /
---
apiVersion: v1
kind: Namespace
metadata:
  name: app

================================================
FILE: go.mod
================================================
module github.com/yanc0/untrak

go 1.12

require gopkg.in/yaml.v2 v2.3.0


================================================
FILE: go.sum
================================================
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=


================================================
FILE: kubernetes/non_namespaced.go
================================================
package kubernetes

var DefaultNonNamespacedResources = []string{
	"componentstatuse",
	"namespace",
	"node",
	"persistentvolume",
	"mutatingwebhookconfiguration",
	"validatingwebhookconfiguration",
	"customresourcedefinition",
	"apiservice",
	"certificatesigningrequest",
	"runtimeclass",
	"podsecuritypolicy",
	"clusterrolebinding",
	"clusterrole",
	"priorityclass",
	"csidriver",
	"csinode",
	"storageclass",
	"volumeattachment",
}

================================================
FILE: kubernetes/structs.go
================================================
package kubernetes

import (
	"fmt"
)

// Metadata of kubernetes resource
type Metadata struct {
	Name      string `yaml:"name"`
	Namespace string `yaml:"namespace,omitempty"`
}

// Resource is a minimal description of a kubernetes object
type Resource struct {
	APIVersion string      `yaml:"apiVersion"`
	Kind       string      `yaml:"kind"`
	Metadata   *Metadata   `yaml:"metadata"`
	Items      []*Resource `yaml:"items,omitempty"`
}

// ID of the resource
func (r *Resource) ID() string {
	return fmt.Sprintf("%s/%s/%s",
		r.Metadata.Namespace,
		r.Kind,
		r.Metadata.Name,
	)
}

//Empty return true if resource was not correctly loaded
func (r *Resource) Empty() bool {
	return r.APIVersion == "" ||
		r.Kind == "" ||
		r.Metadata == nil
}

================================================
FILE: main.go
================================================
package main

import (
	"bytes"
	"flag"
	"io"
	"log"
	"os"
	"os/exec"
	"sync"

	"github.com/yanc0/untrak/outputs"
	"github.com/yanc0/untrak/utils"
	yaml "gopkg.in/yaml.v2"

	"github.com/yanc0/untrak/kubernetes"

	"github.com/yanc0/untrak/config"
)

func main() {
	// Flags, command line parameters
	var cfgPathOpt = flag.String("config", "./untrak.yaml", "untrak Config Path")
	var outputOpt = flag.String("o", "text", "Output format")
	var failOpt = flag.Bool("fail", false, "Fail on untracked resources")
	flag.Parse()

	var wg sync.WaitGroup
	var resourcesIn []*kubernetes.Resource
	var resourcesOut []*kubernetes.Resource

	// Config Load
	cfg, err := config.Load(*cfgPathOpt)
	if err != nil {
		log.Printf("[ERR] Cannot load %s file: %v\n", *cfgPathOpt, err)
		os.Exit(1)
	}

	cfg.NonNamespaced = append(cfg.NonNamespaced, kubernetes.DefaultNonNamespacedResources...)

	wg.Add(1)
	go func() {
		defer wg.Done()
		resourcesIn, err = getKubernetesResources(cfg.In)
		if err != nil {
			log.Printf("[ERR] Failed to get Kubernetes resources (in): %v\n", err)
			os.Exit(1)
		}
	}()

	wg.Add(1)
	go func() {
		defer wg.Done()
		resourcesOut, err = getKubernetesResources(cfg.Out)
		if err != nil {
			log.Printf("[ERR] Failed to get Kubernetes resources (out): %v\n", err)
			os.Exit(1)
		}
	}()

	wg.Wait()

	untrackedResources := listUntrackedResources(resourcesIn, resourcesOut, cfg.Exclude, cfg.NonNamespaced)
	switch {
	case *outputOpt == "text":
		outputs.Text(untrackedResources)
	case *outputOpt == "yaml":
		outputs.YAML(untrackedResources)
	default:
		outputs.Text(untrackedResources)
	}

	if len(untrackedResources) > 0 && *failOpt {
		os.Exit(1)
	}
}

func getKubernetesResources(cfgs []*config.CommandConfig) ([]*kubernetes.Resource, error) {
	const yamlSeparator = "---\n"
	var resources []*kubernetes.Resource

	var wg sync.WaitGroup
	var mutex = &sync.Mutex{}

	for _, cfg := range cfgs {
		wg.Add(1)
		go func(cmd string, args ...string) {
			defer wg.Done()

			// substitute env variables if any has been set
			for i, _ := range args {
				args[i] = os.ExpandEnv(args[i])
			}

			c := exec.Command(cmd, args...)
			var outb, errb bytes.Buffer
			c.Stdout = &outb
			c.Stderr = &errb
			err := c.Run()
			if err != nil {
				log.Fatal(err, errb.String())
			}
			stdoutDec := yaml.NewDecoder(&outb)
			for {
				tempResource := &kubernetes.Resource{}
				err := stdoutDec.Decode(tempResource)
				if err != nil && err != io.EOF {
					log.Printf("[ERR] Failed to decode yaml stream: %s\n", err.Error())
					os.Exit(1)
				}
				if err == io.EOF {
					break
				}
				if tempResource.Kind == "List" {
					mutex.Lock()
					resources = append(resources, tempResource.Items...)
					mutex.Unlock()
					continue
				}
				// Resource can be empty if yaml file has return lines, separators or comments
				// for example:
				// # empty resource
				// ---
				// ---
				// YAML decoder consider these lines valid but resource will be uninitialized
				if !tempResource.Empty() {
					mutex.Lock()
					resources = append(resources, tempResource)
					mutex.Unlock()
				}
			}
		}(cfg.Cmd, cfg.Args...)
	}
	wg.Wait()
	return resources, nil
}

func listUntrackedResources(in []*kubernetes.Resource, out []*kubernetes.Resource, kindExclude []string, nonNamespaced []string) []*kubernetes.Resource {
	var untrackedResources []*kubernetes.Resource
	for _, resourceOut := range out {
		// Resource is in the exlude list, skip it
		if utils.StringInListCaseInsensitive(kindExclude, resourceOut.Kind) {
			continue
		}
		found := false
		for _, resourceIn := range in {

			// If input resource is not namespaced, compare only kind and Name
			if utils.StringInListCaseInsensitive(nonNamespaced, resourceIn.Kind) {
				if resourceOut.Kind == resourceIn.Kind && resourceOut.Metadata.Name == resourceIn.Metadata.Name {
					found = true
					break
				}
			}

			// If resource has been found in both IN an OUT, there is nothing to do
			if resourceOut.ID() == resourceIn.ID() {
				found = true
				break
			}
		}
		// If resource OUT is not found in IN, it is untracked
		if !found {
			untrackedResources = append(untrackedResources, resourceOut)
		}
	}

	return untrackedResources
}


================================================
FILE: outputs/text.go
================================================
package outputs

import (
	"fmt"

	"github.com/yanc0/untrak/kubernetes"
)

// Text output resources as text
func Text(resources []*kubernetes.Resource) {
	var output string
	for _, r := range resources {
		out := r.ID()
		output += fmt.Sprintf("- %s\n", out)
	}
	fmt.Println(output)
}


================================================
FILE: outputs/yaml.go
================================================
package outputs

import (
	"fmt"

	"github.com/yanc0/untrak/kubernetes"
	yaml "gopkg.in/yaml.v2"
)

// YAML outputs resources as YAML
func YAML(resources []*kubernetes.Resource) {
	var output string
	for _, r := range resources {
		out, err := yaml.Marshal(r)
		if err != nil {
			panic(err)
		}
		output += fmt.Sprintf("---\n%s\n", string(out))
	}
	fmt.Printf("%s", output)
}


================================================
FILE: untrak.yaml
================================================
---
# Untrak configuration
# All commands must produce YAML output on stdout either as:
#  - Kind: List - List Kubernetes resource type(from kubectl get stdout)
#  - Concatenated YAML with "---" separator
###

# Kubernetes resources from your versionned controlled configuration
in:
- cmd: "cat"
  args: ["example/manifests/resources_in.yaml"]
out:
- cmd: "cat"
  args: ["example/manifests/resources_out.yaml"]

# Kubernetes resources on your cluster
# check only configmaps, deployments, services and ingresses in api namespace
# out:
# - cmd: "kubectl"
#   args: ["get", "cm,deploy,svc,ing", "-o", "yaml", "-n", "api"]

# You can use environment variables on command args
# out:
# - cmd: "cat"
#   args: ["example/$SOME_FILE_NAME"]

# You can exclude some resource type from the comparison
exclude:
- namespace
- secret
- configmap

# Declare non-namespaced resource types to be considered in resource comparison.
# There are some defined resource types by default by default like namespace,
# node, clusterrole, etc.
# nonNamespaced:
# - some_crd_type


================================================
FILE: utils/commands.go
================================================
package utils

import (
	"bytes"
	"os/exec"
)

// Exec exec the command + args and returns stdout and stderr
func Exec(cmd string, args ...string) ([]byte, []byte, error) {
	c := exec.Command(cmd, args...)

	stdout := &bytes.Buffer{}
	stderr := &bytes.Buffer{}
	c.Stdout = stdout
	c.Stderr = stderr

	err := c.Run()
	if err != nil {
		return stdout.Bytes(), stderr.Bytes(), err
	}
	return stdout.Bytes(), stderr.Bytes(), nil
}


================================================
FILE: utils/strings.go
================================================
package utils

import (
	"strings"
)

// StringInListCaseInsensitive return true if str is in the list (case insensitive)
func StringInListCaseInsensitive(list []string, str string) bool {
	for _, s := range list {
		if strings.ToLower(s) == strings.ToLower(str) {
			return true
		}
	}
	return false
}