Showing preview only (307K chars total). Download the full file or copy to clipboard to get everything.
Repository: Clivern/Beetle
Branch: main
Commit: fddfc896762b
Files: 111
Total size: 281.8 KB
Directory structure:
gitextract_u0vtr08u/
├── .gitattributes
├── .github/
│ ├── CODEOWNERS
│ ├── FUNDING.yml
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ ├── auto-merge.yml
│ ├── boring-cyborg.yml
│ └── workflows/
│ ├── build.yml
│ ├── release.yml
│ └── release_pkg.yml
├── .gitignore
├── .go-version
├── .goreleaser.yml
├── .mergify.yml
├── .poodle.toml
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── assets/
│ └── img/
│ └── chart.drawio
├── beetle.go
├── beetle_test.go
├── bin/
│ └── release.sh
├── config.dist.yml
├── config.testing.yml
├── config.toml
├── core/
│ ├── cmd/
│ │ ├── apps.go
│ │ ├── deploy.go
│ │ ├── license.go
│ │ ├── root.go
│ │ ├── serve.go
│ │ └── version.go
│ ├── controller/
│ │ ├── application.go
│ │ ├── applications.go
│ │ ├── cluster.go
│ │ ├── clusters.go
│ │ ├── daemon.go
│ │ ├── deployment.go
│ │ ├── health_check.go
│ │ ├── health_check_test.go
│ │ ├── job.go
│ │ ├── jobs.go
│ │ ├── metrics.go
│ │ ├── namespace.go
│ │ ├── namespaces.go
│ │ ├── ready_check.go
│ │ ├── ready_check_test.go
│ │ └── worker.go
│ ├── kubernetes/
│ │ ├── application.go
│ │ ├── cluster.go
│ │ ├── cluster_test.go
│ │ ├── config.go
│ │ ├── configmap.go
│ │ ├── deployment.go
│ │ ├── deployment_strategy.go
│ │ ├── namespace.go
│ │ ├── namespace_test.go
│ │ └── pod.go
│ ├── middleware/
│ │ ├── auth.go
│ │ ├── correlation.go
│ │ ├── log.go
│ │ └── metric.go
│ ├── migration/
│ │ └── schema.go
│ ├── model/
│ │ ├── application.go
│ │ ├── cluster.go
│ │ ├── configmap.go
│ │ ├── configs.go
│ │ ├── dsn.go
│ │ ├── dsn_test.go
│ │ ├── job.go
│ │ ├── message.go
│ │ ├── metric.go
│ │ ├── migration.go
│ │ ├── namespace.go
│ │ ├── patch.go
│ │ └── request.go
│ ├── module/
│ │ ├── database.go
│ │ ├── database_test.go
│ │ ├── file_system.go
│ │ ├── http.go
│ │ ├── http_test.go
│ │ ├── prometheus.go
│ │ ├── remote.go
│ │ └── remote_test.go
│ └── util/
│ ├── helpers.go
│ └── helpers_test.go
├── deployment/
│ ├── docker/
│ │ ├── README.md
│ │ ├── docker-compose.yml
│ │ └── prometheus.yml
│ └── k8s/
│ └── incluster/
│ ├── README.md
│ ├── beetle.yaml
│ └── sample_app.yaml
├── go.mod
├── go.sum
├── pkg/
│ ├── expect.go
│ └── server_mock.go
├── renovate.json
├── sdk/
│ ├── application.go
│ ├── application_test.go
│ ├── client.go
│ ├── cluster.go
│ ├── cluster_test.go
│ ├── deployment.go
│ ├── deployment_test.go
│ ├── job.go
│ ├── job_test.go
│ ├── namespace.go
│ └── namespace_test.go
└── swagger.yaml
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
docs/* linguist-vendored
================================================
FILE: .github/CODEOWNERS
================================================
# Docs: https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners
* @clivern
================================================
FILE: .github/FUNDING.yml
================================================
github: # clivern
custom: clivern.com/sponsor/
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug report
about: Create a report to help us improve
---
**Describe the bug**
A clear and concise description of what the bug is.
**Development or production environment**
- OS: [e.g. Ubuntu 18.04]
- Go 1.13
**Additional context**
Add any other context about the problem here.
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md
================================================
---
name: Feature request
about: Suggest an idea for this project
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Additional context**
Add any other context or screenshots about the feature request here.
================================================
FILE: .github/auto-merge.yml
================================================
# https://github.com/bobvanderlinden/probot-auto-merge
blockingLabels:
- blocking
rules:
- minApprovals:
OWNER: 1
MEMBER: 2
- requiredLabels:
- merge
================================================
FILE: .github/boring-cyborg.yml
================================================
---
firstIssueWelcomeComment: "Thanks for opening your first issue here! Be sure to follow the issue template!"
firstPRMergeComment: "Awesome work, congrats on your first merged pull request!"
firstPRWelcomeComment: "Thanks for opening this pull request! Please check out our contributing guidelines."
labelPRBasedOnFilePath:
"🚧 CI":
- .github/workflows/*
"🚧 CSS":
- "**/*.css"
"🚧 Configuration":
- .github/*
"🚧 Dependencies":
- Dockerfile*
- composer.*
- package.json
- package-lock.json
- yarn.lock
- go.mod
- go.sum
- build.gradle
- Cargo.toml
- Cargo.lock
- Gemfile.lock
- Gemfile
"🚧 Docker":
- Dockerfile*
- .docker/**/*
"🚧 Documentation":
- README.md
- CONTRIBUTING.md
"🚧 Go":
- "**/*.go"
"🚧 Rust":
- "**/*.rs"
"🚧 Java":
- "**/*.java"
"🚧 Ruby":
- "**/*.rb"
"🚧 HTML":
- "**/*.htm"
- "**/*.html"
"🚧 Image":
- "**/*.gif"
- "**/*.jpg"
- "**/*.jpeg"
- "**/*.png"
- "**/*.webp"
"🚧 JSON":
- "**/*.json"
"🚧 JavaScript":
- "**/*.js"
- package.json
- package-lock.json
- yarn.lock
"🚧 MarkDown":
- "**/*.md"
"🚧 PHP":
- "**/*.php"
- composer.*
"🚧 Source":
- src/**/*
"🚧 TOML":
- "**/*.toml"
"🚧 Templates":
- "**/*.twig"
- "**/*.tpl"
"🚧 Tests":
- tests/**/*
"🚧 YAML":
- "**/*.yml"
- "**/*.yaml"
================================================
FILE: .github/workflows/build.yml
================================================
name: Build
on:
push:
pull_request:
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
go: ['1.19', '1.20.4']
name: Go ${{ matrix.go }} run
steps:
- uses: actions/checkout@v4
- name: Setup go
uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go }}
- name: Get dependencies
run: |
export PATH=${PATH}:`go env GOPATH`/bin
make install_revive
- name: Run make ci
run: |
export PATH=${PATH}:`go env GOPATH`/bin
go get -t .
make ci
git status
git diff > diff.log
cat diff.log
git clean -fd
git reset --hard
make verify
================================================
FILE: .github/workflows/release.yml
================================================
name: Release
on:
push:
tags:
- '*'
jobs:
goreleaser:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
-
name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.19
-
name: Run GoReleaser
uses: goreleaser/goreleaser-action@v3
with:
version: latest
args: release --rm-dist
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
================================================
FILE: .github/workflows/release_pkg.yml
================================================
name: ReleasePkg
on:
push:
tags:
- '*'
jobs:
release:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
-
name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.19
- name: Update checksum database
run: |
./bin/release.sh
================================================
FILE: .gitignore
================================================
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
config.prod.yml
# dist dir
dist
================================================
FILE: .go-version
================================================
1.20.4
================================================
FILE: .goreleaser.yml
================================================
# This is an example goreleaser.yaml file with some sane defaults.
# Make sure to check the documentation at http://goreleaser.com
---
archives:
-
replacements:
386: i386
amd64: x86_64
darwin: Darwin
linux: Linux
windows: Windows
before:
hooks:
- "go mod download"
- "go generate ./..."
builds:
-
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
- windows
changelog:
filters:
exclude:
- "^docs:"
- "^test:"
sort: asc
checksum:
name_template: checksums.txt
snapshot:
name_template: "{{ .Tag }}-next"
project_name: beetle
================================================
FILE: .mergify.yml
================================================
---
pull_request_rules:
-
actions:
merge:
method: squash
conditions:
- author!=Clivern
- approved-reviews-by=Clivern
- label=release
name: "Automatic Merge 🚀"
-
actions:
merge:
method: merge
conditions:
- author=Clivern
- label=release
name: "Automatic Merge 🚀"
-
actions:
merge:
method: squash
conditions:
- "author=renovate[bot]"
- label=release
name: "Automatic Merge for Renovate PRs 🚀"
-
actions:
comment:
message: "Nice! PR merged successfully."
conditions:
- merged
name: "Merge Done 🚀"
================================================
FILE: .poodle.toml
================================================
# API Definition For Beetle
# --------------------------
#
# In order to use this file:
# 1. Check & Install https://github.com/Clivern/Poodle
#
# 2. Now you can use poodle to interact with your local or hosted beetle.
# $ poodle call -f .poodle.toml
#
[Main]
id = "clivern_beetle"
name = "Clivern - Beetle"
description = "Beetle API Definitions"
timeout = "30s"
service_url = "{$serviceURL:http://127.0.0.1:8080}"
# These headers will be applied to all endpoints http calls
headers = []
[Security]
# Supported Types are basic, bearer and api_key and none
scheme = "none"
[Security.Basic]
username = "{$authUsername:default}"
password = "{$authPassword:default}"
header = ["Authorization", "Basic base64(username:password)"]
[Security.ApiKey]
header = ["X-API-KEY", "{$authApiKey:default}"]
# In case of bearer authentication, it is recommended to create another
# service or endpoint to generate the bearer tokens
[Security.Bearer]
header = ["Authorization", "Bearer {$authBearerToken:default}"]
[[Endpoint]]
id = "GetSystemHealth"
name = "Get System Health"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/_health"
body = ""
[[Endpoint]]
id = "GetSystemReadiness"
name = "Get System Readiness"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/_ready"
body = ""
[[Endpoint]]
id = "GetMetrics"
name = "Get Metrics"
description = ""
method = "get"
headers = []
parameters = []
public = true
uri = "/metrics"
body = ""
[[Endpoint]]
id = "GetClusters"
name = "Get Clusters"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/cluster"
body = ""
[[Endpoint]]
id = "GetCluster"
name = "Get Cluster"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/cluster/{$clusterName}"
body = ""
[[Endpoint]]
id = "GetClusterNamespaces"
name = "Get Cluster Namespaces"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/cluster/{$clusterName}/namespace"
body = ""
[[Endpoint]]
id = "GetClusterNamespace"
name = "Get Cluster Namespace"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/cluster/{$clusterName}/namespace/{$namespaceName}"
body = ""
[[Endpoint]]
id = "GetNamespaceApplications"
name = "Get Namespace Applications"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/cluster/{$clusterName}/namespace/{$namespaceName}/app"
body = ""
[[Endpoint]]
id = "GetNamespaceApplicationByID"
name = "Get Namespace Application by ID"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/cluster/{$clusterName}/namespace/{$namespaceName}/app/{$applicationId}"
body = ""
[[Endpoint]]
id = "GetJobs"
name = "Get Jobs"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
public = true
uri = "/api/v1/job"
body = ""
[[Endpoint]]
id = "GetJobByUUID"
name = "Get Job by UUID"
description = ""
method = "get"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
uri = "/api/v1/job/{$jobUUID}"
body = ""
[[Endpoint]]
id = "DeleteJobByUUID"
name = "Delete Job by UUID"
description = ""
method = "delete"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
uri = "/api/v1/job/{$jobUUID}"
body = ""
[[Endpoint]]
id = "DeployApplicationById"
name = "Deploy Application By ID"
description = ""
method = "post"
headers = [ ["Content-Type", "application/json"] ]
parameters = []
uri = "/api/v1/cluster/{$clusterName}/namespace/{$namespaceName}/app/{$applicationId}/deployment"
body = """
{
"version":"{$version}",
"strategy":"{$strategy:recreate}"
}
"""
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at hello@clivern.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq
================================================
FILE: CONTRIBUTING.md
================================================
## Contributing
- With issues:
- Use the search tool before opening a new issue.
- Please provide source code and commit sha if you found a bug.
- Review existing issues and provide feedback or react to them.
- With pull requests:
- Open your pull request against `main`
- Your pull request should have no more than two commits, if not you should squash them.
- It should pass all tests in the available continuous integrations systems such as TravisCI.
- You should add/modify tests to cover your proposed code changes.
- If your pull request contains a new feature, please document it on the README.
================================================
FILE: Dockerfile
================================================
FROM golang:1.20.2
ARG BEETLE_VERSION=1.0.2
ENV GO111MODULE=on
RUN mkdir -p /app/configs
RUN mkdir -p /app/var/logs
RUN mkdir -p /app/var/storage
RUN apt-get update
WORKDIR /app
RUN curl -sL https://github.com/Clivern/Beetle/releases/download/v${BEETLE_VERSION}/beetle_${BEETLE_VERSION}_Linux_x86_64.tar.gz | tar xz
RUN rm LICENSE
RUN rm README.md
COPY ./config.dist.yml /app/configs/
EXPOSE 8080
VOLUME /app/configs
VOLUME /app/var
RUN ./beetle version
CMD ["./beetle", "serve", "-c", "/app/configs/config.dist.yml"]
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2020 Clivern
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Makefile
================================================
GO ?= go
GOFMT ?= $(GO)fmt
pkgs = ./...
HUGO ?= hugo
help: Makefile
@echo
@echo " Choose a command run in Beetle:"
@echo
@sed -n 's/^##//p' $< | column -t -s ':' | sed -e 's/^/ /'
@echo
## install_revive: Install revive for linting.
install_revive:
@echo ">> ============= Install Revive ============= <<"
$(GO) install github.com/mgechev/revive@latest
## style: Check code style.
style:
@echo ">> ============= Checking Code Style ============= <<"
@fmtRes=$$($(GOFMT) -d $$(find . -path ./vendor -prune -o -name '*.go' -print)); \
if [ -n "$${fmtRes}" ]; then \
echo "gofmt checking failed!"; echo "$${fmtRes}"; echo; \
echo "Please ensure you are using $$($(GO) version) for formatting code."; \
exit 1; \
fi
## check_license: Check if license header on all files.
check_license:
@echo ">> ============= Checking License Header ============= <<"
@licRes=$$(for file in $$(find . -type f -iname '*.go' ! -path './vendor/*') ; do \
awk 'NR<=3' $$file | grep -Eq "(Copyright|generated|GENERATED)" || echo $$file; \
done); \
if [ -n "$${licRes}" ]; then \
echo "license header checking failed:"; echo "$${licRes}"; \
exit 1; \
fi
## test_short: Run test cases with short flag.
test_short:
@echo ">> ============= Running Short Tests ============= <<"
$(GO) test -mod=readonly -short $(pkgs)
## test: Run test cases.
test:
@echo ">> ============= Running All Tests ============= <<"
$(GO) test -mod=readonly -v -cover $(pkgs)
## lint: Lint the code.
lint:
@echo ">> ============= Lint All Files ============= <<"
revive -config config.toml -exclude vendor/... -formatter friendly ./...
## verify: Verify dependencies
verify:
@echo ">> ============= List Dependencies ============= <<"
$(GO) list -m all
@echo ">> ============= Verify Dependencies ============= <<"
$(GO) mod verify
## format: Format the code.
format:
@echo ">> ============= Formatting Code ============= <<"
$(GO) fmt $(pkgs)
## vet: Examines source code and reports suspicious constructs.
vet:
@echo ">> ============= Vetting Code ============= <<"
$(GO) vet $(pkgs)
## coverage: Create HTML coverage report
coverage:
@echo ">> ============= Coverage ============= <<"
rm -f coverage.html cover.out
$(GO) test -mod=readonly -coverprofile=cover.out $(pkgs)
go tool cover -html=cover.out -o coverage.html
## ci: Run all CI tests.
ci: style check_license test vet lint
@echo "\n==> All quality checks passed"
## run: Run the service
run:
-cp -n config.dist.yml config.prod.yml
$(GO) run beetle.go serve -c config.prod.yml
.PHONY: help
================================================
FILE: README.md
================================================
<p align="center">
<img src="https://raw.githubusercontent.com/clivern/Beetle/main/assets/img/gopher.png?v=1.0.4" width="180" />
<h3 align="center">Beetle</h3>
<p align="center">Kubernetes multi-cluster deployment automation service</p>
<p align="center">
<a href="https://github.com/Clivern/Beetle/actions"><img src="https://github.com/Clivern/Beetle/workflows/Build/badge.svg"></a>
<a href="https://github.com/Clivern/Beetle/actions"><img src="https://github.com/Clivern/Beetle/workflows/Release/badge.svg"></a>
<a href="https://github.com/Clivern/Beetle/releases"><img src="https://img.shields.io/badge/Version-v1.0.4-red.svg"></a>
<a href="https://goreportcard.com/report/github.com/Clivern/Beetle"><img src="https://goreportcard.com/badge/github.com/clivern/Beetle?v=1.0.4"></a>
<a href="https://godoc.org/github.com/clivern/beetle"><img src="https://godoc.org/github.com/clivern/beetle?status.svg"></a>
<a href="https://hub.docker.com/r/clivern/beetle"><img src="https://img.shields.io/badge/Docker-Latest-orange"></a>
<a href="https://github.com/Clivern/Beetle/blob/main/LICENSE"><img src="https://img.shields.io/badge/LICENSE-MIT-orange.svg"></a>
</p>
</p>
<br/>
<p align="center">
<img src="https://raw.githubusercontent.com/Clivern/Beetle/main/assets/img/chart.png?v=1.0.4" width="100%" />
</p>
<h4 align="center">
<a href="https://www.youtube.com/watch?v=54qQIYTZiAw" target="_blank">:unicorn: Check out the demo!</a>
</h4>
<br/>
Application deployment and management should be automated, auditable, and easy to understand and that's what beetle tries to achieve in a simple manner. Beetle automates the deployment and rollback of your applications in a multi-cluster, multi-namespaces kubernetes environments. Easy to integrate with through API endpoints & webhooks to fit a variety of workflows.
## Documentation
## Deployment
### On a Linux Server
Download [the latest beetle binary.](https://github.com/Clivern/Beetle/releases)
```zsh
$ curl -sL https://github.com/Clivern/Beetle/releases/download/vx.x.x/beetle_x.x.x_OS.tar.gz | tar xz
```
Create your config file as explained on [development part](#development) and run beetle with systemd or anything else you prefer.
```zsh
$ ./beetle serve -c /custom/path/config.prod.yml
```
## Development
Beetle uses Go Modules to manage dependencies. First Create a prod config file.
```zsh
$ git clone https://github.com/Clivern/Beetle.git
$ cp config.dist.yml config.prod.yml
```
Then add your default configs. You probably wondering how the following configs even work! let's pick one and explain.
The item mode: `${BEETLE_APP_MODE:-dev}` means that the mode is dev unless environment variable `BEETLE_APP_MODE` is defined. so you can always override the value by defining the environment variable `export BEETLE_APP_MODE=prod`. and same for others
```yaml
# App configs
app:
# Env mode (dev or prod)
mode: ${BEETLE_APP_MODE:-dev}
# HTTP port
port: ${BEETLE_API_PORT:-8080}
# App URL
domain: ${BEETLE_APP_DOMAIN:-http://127.0.0.1:8080}
# TLS configs
tls:
status: ${BEETLE_API_TLS_STATUS:-off}
pemPath: ${BEETLE_API_TLS_PEMPATH:-cert/server.pem}
keyPath: ${BEETLE_API_TLS_KEYPATH:-cert/server.key}
# Message Broker Configs
broker:
# Broker driver (native)
driver: ${BEETLE_BROKER_DRIVER:-native}
# Native driver configs
native:
# Queue max capacity
capacity: ${BEETLE_BROKER_NATIVE_CAPACITY:-5000}
# Number of concurrent workers
workers: ${BEETLE_BROKER_NATIVE_WORKERS:-4}
# API Configs
api:
key: ${BEETLE_API_KEY:- }
# Runtime, Requests/Response and Beetle Metrics
metrics:
prometheus:
# Route for the metrics endpoint
endpoint: ${BEETLE_METRICS_PROM_ENDPOINT:-/metrics}
# Application Database
database:
# Database driver (sqlite3, mysql)
driver: ${BEETLE_DATABASE_DRIVER:-sqlite3}
# Database Host
host: ${BEETLE_DATABASE_MYSQL_HOST:-localhost}
# Database Port
port: ${BEETLE_DATABASE_MYSQL_PORT:-3306}
# Database Name
name: ${BEETLE_DATABASE_MYSQL_DATABASE:-beetle.db}
# Database Username
username: ${BEETLE_DATABASE_MYSQL_USERNAME:-root}
# Database Password
password: ${BEETLE_DATABASE_MYSQL_PASSWORD:-root}
# Kubernetes Clusters
clusters:
-
name: ${BEETLE_KUBE_CLUSTER_01_NAME:-production}
inCluster: ${BEETLE_KUBE_CLUSTER_01_IN_CLUSTER:-false}
kubeconfig: ${BEETLE_KUBE_CLUSTER_01_CONFIG_FILE:-/app/configs/production-cluster-kubeconfig.yaml}
-
name: ${BEETLE_KUBE_CLUSTER_02_NAME:-staging}
inCluster: ${BEETLE_KUBE_CLUSTER_02_IN_CLUSTER:-false}
kubeconfig: ${BEETLE_KUBE_CLUSTER_02_CONFIG_FILE:-/app/configs/staging-cluster-kubeconfig.yaml}
# HTTP Webhook
webhook:
url: ${BEETLE_WEBHOOK_URL:- }
retry: ${BEETLE_WEBHOOK_RETRY:-3}
apiKey: ${BEETLE_WEBHOOK_API_KEY:- }
# Log configs
log:
# Log level, it can be debug, info, warn, error, panic, fatal
level: ${BEETLE_LOG_LEVEL:-info}
# output can be stdout or abs path to log file /var/logs/beetle.log
output: ${BEETLE_LOG_OUTPUT:-stdout}
# Format can be json
format: ${BEETLE_LOG_FORMAT:-json}
```
And then run the application.
```zsh
$ go build beetle.go
$ ./beetle serve -c /custom/path/config.prod.yml
// OR
$ make run
// OR
$ go run beetle.go serve -c /custom/path/config.prod.yml
```
## API Documentation
Go to https://editor.swagger.io/ and import this file https://raw.githubusercontent.com/Clivern/Beetle/main/swagger.yaml.
## Versioning
For transparency into our release cycle and in striving to maintain backward compatibility, Beetle is maintained under the [Semantic Versioning guidelines](https://semver.org/) and release process is predictable and business-friendly.
See the [Releases section of our GitHub project](https://github.com/clivern/beetle/releases) for changelogs for each release version of Beetle. It contains summaries of the most noteworthy changes made in each release.
## Bug tracker
If you have any suggestions, bug reports, or annoyances please report them to our issue tracker at https://github.com/clivern/beetle/issues
## Security Issues
If you discover a security vulnerability within Beetle, please send an email to [hello@clivern.com](mailto:hello@clivern.com)
## Contributing
We are an open source, community-driven project so please feel free to join us. see the [contributing guidelines](CONTRIBUTING.md) for more details.
## License
© 2020, clivern. Released under [MIT License](https://opensource.org/licenses/mit-license.php).
**Beetle** is authored and maintained by [@clivern](http://github.com/clivern).
================================================
FILE: assets/img/chart.drawio
================================================
<mxfile modified="2021-04-03T21:36:16.551Z" host="app.diagrams.net" agent="5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" etag="mCddqXOVAB87riqQZXYU" version="14.5.7" type="device"><diagram id="d1ab7348-05c3-a1e3-ca4d-12c340fd6b49" name="Page-1">7Vxdd5s6Fv01XmvmwSyQ+HyMk7jtNOnkXvfetvclC4Nsq8HIA3IS99ePBAKDhG2SgJMmcddqjBBCaG9tHZ1z8ACeLu8/JP5qcUlCFA2AHt4P4NkAAMPQbfaHl2zyEgc6ecE8waGotC2Y4F9IFOqidI1DlNYqUkIiilf1woDEMQporcxPEnJXrzYjUf2uK3+OlIJJ4Edq6Tcc0kXxXLa3PfER4flC3NoF4vmmfnAzT8g6FvcbADjLPvnppV+0JR40XfghuasUwfMBPE0Iofm35f0pivjYFsOWXzfecbbsd4Ji2uYCN55MJ+58/un6P2jpe7/Qt0syNGwBX0o3xYigkA2QOCQJXZA5if3ofFs6yp4a8XZ1drStc0HIihUarPAnonQj0PbXlLCiBV1G4iy6x/Q7v1yzxNGPypmze9FydrARB+rziiHg/a0UiKf/gMgS0WTDKiQo8im+rQPuC97My3rlpVcEs1sAXXDchVZ+iWA48PR6EylZJwESV1UReGhD1E/miCoNsS+V59kWZQA/CGznHew3A7YD8/vd+tEaFRIloc80acW/Lu/nXN21ebACWkjW0wj9mYntqFQtjldKE3JTyiQvmeEoOiURSdhxTGLOlxmJaVHENFHPPqz8FiUUM909ifA85vfEYZgRLG9VamUnAXgz6H4vBcRZx5IQsEUTd1udB7pAZVGReFvfzZoaTA/HxGzAxI6oGLZsmStGzv7fmi8NbCSg4fN/1SJ7zv9eJSRcBxSTeMDFP29omhSnP6+nKIkRZcsr0E+jdUpRUtRi/c/vmFdVmBH66aKc9GLt9afZWb0+vasUYH21DNccw12wNhLupuzndcS0hVVjuFPRfORPUXRFUpw9JjxLcpxKOl1I50ta+YJnEZrRvfRb+QGO5xdZtTNLZbCXfUS5UDkDNHL0wEw8TN0KNU1LZaYJNVsUP1Hn6lPD0WzH0rcfqTkym6WoJ5Wy3lWqBgUs1olnVCn7USolxlBRqTN0iyKyWqJMbvwlsxpG8TRd5VXeNetlapbVmshvT7OcN69ZZl2zAHx2zXI71awJ9edsgr3r1W+jV05rEr85vXLBYb2qkM1PV7mjbYbvOT13siyXL73Cg4ChxPivMoFyX4FMYzhyz8bGLsaGPpsMfoquQQeKZQNd86qfGjSmDRT9gp5KDas3+XJbbNbbQ8TG1gVTaNvd4BRayA3NXTjNkZ8wTfHDbtYWy3Q0WF9ddE9BxwWao+Lj2Jru9gZRi51KTdHJmkY4ZqAUjmtJ1BUwZBlWUJJ1eEooJcudMMpw3aEp1/5Um2O6WE/3WBQd4MhUR5N2NoUgHhJguzcIm4yEHYYbXmaBgypgzaN8EJwM1lEZLaiMt4gXwFF2s5NiUutNM1z052xBKY+MnPChAOMgjA0NM9thhuMQJVrA7gjGXDrZH17OzJLxNPHjMB3+JFHEBnGcBQrGU0yn6+AG0WFKAuxHQ2Yc3JHkZpjVHnKiDA3gaitmBfVMFL1Ok4YNsNVgS/YnxkUH9tGkcXLvVuFdiloTjIPmlkzAkmKKlFRYe7fAFE2YAcZ7fseEQLKy9OJYPJyxtdfYEWzQEf8uNbVV7nv8FPAujlZJ/qVeh7Fxych0zUfqGvOuzbJudEEb6NTX8CZ1sQyVNhD0Rpsm565Em8PhFtkArgZS4vCERzw58hEJbr4ucJwXj3FUVKqHa+r7SpDXFs1nKzplu4DascRgaJ64FpSZymBLNt+rBzykIwI8/HAb1cmODoZ18mDIXgNfGAUHA0AV/I1iD/pEQ96GkkSZrlZsgR8aF3JatNV3aMhrYcq8M/XRTLWekalAUkXZ49KaprKLR28XvmSw+5tKtRWvkO7psAcaO7yrX3J9U4zRdorkPeh4wjR5xF/nhNlJ/BchvI9WXZnOFtQsWPEHSHuOjtjtgFa33dnrB13eF/dbeLtfCfe3mSqm7Q62uSqZ72iwL1+FH1yhBLPR5vb3/pmUk3bfmLvPaeyY8n5Mcqm0nXNyQ6YlNdTRJLNcW1oSwMP6Va/f1zRq4Xt4JdPoRS0hJpQsoifY7laLtnq33b3DPPrtfFjmQR9W5rgccs9U7s1iRZbBh9cwzesPmLIOZoGt67zikTxXsouzKcXsqL6rMur6LjPHlRm2cZHd3ZZelDxUZ6AjbYosqT8drZxsYBvvs7Nf++v3s3KCIlL6TuljU1qWN9PwHktpU5dmh9uPP0G+j1l/K6IvioJ3ij4PRS15BfYerbqmLsWdQV8UhVKXj6Kiqs/3cjP544IVnYlMC4WyzA6idR42poI0pKodTDQoY1pN4ak6kXu04GzgaK6xKy3EArZizUFDgw0xa6M/g051PY4QomzsgH5y9Yn9P0HJLU86+63Aq0Yc9Q6QND2oeXXrBLhq3ogDNOOYQUGgq96zEr7TPDzKvl3gmBd8JST6zXDsAjrZyWur2aR2wyYKWr2Bpvpq/kZJmr9nc8pGIuE46ZNNStHy7QFmKCFMFTCzATB5Pe0OMEPd9f7F1zN991yjhK98aBWRbO2eZXm7+p8oQm9yJQSy19dW1z7DKK2tI8Gq2iwFYi8Sn1quZZ9miwxWQ/79kSdgiyDRQSfjQTelgpWSmfRrnSAt8ld5/cwSR8n5LeIGueBFDaN6WlYQICvzZap7IRt6sKMMV2jJmxarWHAOuA1hf/i9RyeeZQMLpeRHS06zaL15NY7kX5HuUxhv/W5eDTXocUGYPMjv55zHtzghcf6q4ZteIqBtyCJjms+8SIAWsYmX9x5K/iMz12lu7Pe8rhvl9rS0w1TUoK5ZsAE47+nA/TcdXXy8/Pvzp5+jzeU/11/+ALPvw7fj3NwmgEAXDmoJILA4bp8Ass093KYb/qiee2ju4XOsUJbjap5j6s0uMuhamucCu0xKeqTv1ZK2lKbTV66IZDm7+zOw5PqmyMhq/Rz1+k9eHhvnZ4vkxFeYETD1UxwM13gIhlG2pR9zL96Y9SoahoQOY0LxjD0C73h+sBkmOJ6XuQFP12tLNoYKttTEWlXqLuz4RiaoVvwPNgFZyZfKYBx+qZl7lHHAvSSf2DrK1kKKwuyp2EMB/V+TiME+AKyLusY+iAb/fqHWVp9eS8su3xMu4wXqNs6wGt9U7CJe0MgAQ12sP379esVKvqHpgpCbF4pUF7NRdkt6xzN4L04i95/F5dnKuwtC/8vwr89Ld/iet/A8FosrhYSLX6R8sFEiNQTll4s7M0qa79OV0dBIzhZG/Ss0GtAtjobZwXCdomS4fSkRjPlOa8wmAMUpe4q0SzvBBpr0c4DlO1hVcTJVcSrKOhcnW/XYXiW87QVap694kbA9BYoGq+2o64Sterfqv10jrLIg4fMxe3ub35hxtO0vzbwM+HgXx/4SR3zgP6LoFvGWOsLVM2Rcm5JxO8KVHW5/uDdX5O2vI8Pz/wM=</diagram></mxfile>
================================================
FILE: beetle.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package main
import (
"github.com/clivern/beetle/core/cmd"
)
var (
version = "dev"
commit = "none"
date = "unknown"
builtBy = "unknown"
)
func main() {
// Expose build info to cmd subpackage to avoid custom ldflags
cmd.Version = version
cmd.Commit = commit
cmd.Date = date
cmd.BuiltBy = builtBy
cmd.Execute()
}
================================================
FILE: beetle_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package main
import (
"bytes"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/pkg"
"github.com/drone/envsubst"
"github.com/spf13/viper"
)
var testingConfig = "config.testing.yml"
// TestMain test cases
func TestMain(t *testing.T) {
// LoadConfigFile
t.Run("LoadConfigFile", func(t *testing.T) {
fs := module.FileSystem{}
dir, _ := os.Getwd()
configFile := fmt.Sprintf("%s/%s", dir, testingConfig)
for {
if fs.FileExists(configFile) {
break
}
dir = filepath.Dir(dir)
configFile = fmt.Sprintf("%s/%s", dir, testingConfig)
}
t.Logf("Load Config File %s", configFile)
configUnparsed, _ := ioutil.ReadFile(configFile)
configParsed, _ := envsubst.EvalEnv(string(configUnparsed))
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer([]byte(configParsed)))
pkg.Expect(t, viper.GetString("app.mode"), "test")
})
}
================================================
FILE: bin/release.sh
================================================
#!/bin/bash
# Fetch latest version
export LATEST_VERSION=$(curl --silent "https://api.github.com/repos/clivern/beetle/releases/latest" | jq '.tag_name' | sed -E 's/.*"([^"]+)".*/\1/')
# Update go checksum database (sum.golang.org) immediately after release
curl --silent https://sum.golang.org/lookup/github.com/clivern/beetle@{$LATEST_VERSION}
================================================
FILE: config.dist.yml
================================================
# App configs
app:
# Env mode (dev or prod)
mode: ${BEETLE_APP_MODE:-dev}
# HTTP port
port: ${BEETLE_API_PORT:-8080}
# App URL
domain: ${BEETLE_APP_DOMAIN:-http://127.0.0.1:8080}
# TLS configs
tls:
status: ${BEETLE_API_TLS_STATUS:-off}
pemPath: ${BEETLE_API_TLS_PEMPATH:-cert/server.pem}
keyPath: ${BEETLE_API_TLS_KEYPATH:-cert/server.key}
# Message Broker Configs
broker:
# Broker driver (native)
driver: ${BEETLE_BROKER_DRIVER:-native}
# Native driver configs
native:
# Queue max capacity
capacity: ${BEETLE_BROKER_NATIVE_CAPACITY:-5000}
# Number of concurrent workers
workers: ${BEETLE_BROKER_NATIVE_WORKERS:-4}
# API Configs
api:
key: ${BEETLE_API_KEY:- }
# Runtime, Requests/Response and Beetle Metrics
metrics:
prometheus:
# Route for the metrics endpoint
endpoint: ${BEETLE_METRICS_PROM_ENDPOINT:-/metrics}
# Application Database
database:
# Database driver (sqlite3, mysql)
driver: ${BEETLE_DATABASE_DRIVER:-sqlite3}
# Hostname
host: ${BEETLE_DATABASE_MYSQL_HOST:-localhost}
# Port
port: ${BEETLE_DATABASE_MYSQL_PORT:-3306}
# Database
name: ${BEETLE_DATABASE_MYSQL_DATABASE:-beetle.db}
# Username
username: ${BEETLE_DATABASE_MYSQL_USERNAME:-root}
# Password
password: ${BEETLE_DATABASE_MYSQL_PASSWORD:-root}
# Kubernetes Clusters
clusters:
-
name: ${BEETLE_KUBE_CLUSTER_01_NAME:-production}
inCluster: ${BEETLE_KUBE_CLUSTER_01_IN_CLUSTER:-false}
kubeconfig: ${BEETLE_KUBE_CLUSTER_01_CONFIG_FILE:-/app/configs/production-cluster-kubeconfig.yaml}
-
name: ${BEETLE_KUBE_CLUSTER_02_NAME:-staging}
inCluster: ${BEETLE_KUBE_CLUSTER_02_IN_CLUSTER:-false}
kubeconfig: ${BEETLE_KUBE_CLUSTER_02_CONFIG_FILE:-/app/configs/staging-cluster-kubeconfig.yaml}
# HTTP Webhook
webhook:
url: ${BEETLE_WEBHOOK_URL:- }
retry: ${BEETLE_WEBHOOK_RETRY:-3}
apiKey: ${BEETLE_WEBHOOK_API_KEY:- }
# Log configs
log:
# Log level, it can be debug, info, warn, error, panic, fatal
level: ${BEETLE_LOG_LEVEL:-info}
# output can be stdout or abs path to log file /var/logs/beetle.log
output: ${BEETLE_LOG_OUTPUT:-stdout}
# Format can be json
format: ${BEETLE_LOG_FORMAT:-json}
================================================
FILE: config.testing.yml
================================================
# App configs
app:
# Env mode (dev or prod)
mode: ${BEETLE_APP_MODE:-test}
# HTTP port
port: ${BEETLE_API_PORT:-8080}
# App URL
domain: ${BEETLE_APP_DOMAIN:-http://127.0.0.1:8080}
# TLS configs
tls:
status: ${BEETLE_API_TLS_STATUS:-off}
pemPath: ${BEETLE_API_TLS_PEMPATH:-cert/server.pem}
keyPath: ${BEETLE_API_TLS_KEYPATH:-cert/server.key}
# Message Broker Configs
broker:
# Broker driver (native)
driver: ${BEETLE_BROKER_DRIVER:-native}
# Native driver configs
native:
# Queue max capacity
capacity: ${BEETLE_BROKER_NATIVE_CAPACITY:-5000}
# Number of concurrent workers
workers: ${BEETLE_BROKER_NATIVE_WORKERS:-4}
# API Configs
api:
key: ${BEETLE_API_KEY:- }
# Runtime, Requests/Response and Beetle Metrics
metrics:
prometheus:
# Route for the metrics endpoint
endpoint: ${BEETLE_METRICS_PROM_ENDPOINT:-/metrics}
# Application Database
database:
# Database driver (sqlite3, mysql)
driver: ${BEETLE_DATABASE_DRIVER:-sqlite3}
# Hostname
host: ${BEETLE_DATABASE_MYSQL_HOST:-localhost}
# Port
port: ${BEETLE_DATABASE_MYSQL_PORT:-3306}
# Database
name: ${BEETLE_DATABASE_MYSQL_DATABASE:-/tmp/beetle.db}
# Username
username: ${BEETLE_DATABASE_MYSQL_USERNAME:-root}
# Password
password: ${BEETLE_DATABASE_MYSQL_PASSWORD:- }
# Kubernetes Clusters
clusters:
-
name: ${BEETLE_KUBE_CLUSTER_01_NAME:-production}
inCluster: ${BEETLE_KUBE_CLUSTER_01_IN_CLUSTER:-false}
kubeconfig: ${BEETLE_KUBE_CLUSTER_01_CONFIG_FILE:-/app/configs/production-cluster-kubeconfig.yaml}
-
name: ${BEETLE_KUBE_CLUSTER_02_NAME:-staging}
inCluster: ${BEETLE_KUBE_CLUSTER_02_IN_CLUSTER:-false}
kubeconfig: ${BEETLE_KUBE_CLUSTER_02_CONFIG_FILE:-/app/configs/staging-cluster-kubeconfig.yaml}
# HTTP Webhook
webhook:
url: ${BEETLE_WEBHOOK_URL:- }
retry: ${BEETLE_WEBHOOK_RETRY:-3}
apiKey: ${BEETLE_WEBHOOK_API_KEY:- }
# Log configs
log:
# Log level, it can be debug, info, warn, error, panic, fatal
level: ${BEETLE_LOG_LEVEL:-info}
# output can be stdout or abs path to log file /var/logs/beetle.log
output: ${BEETLE_LOG_OUTPUT:-stdout}
# Format can be json
format: ${BEETLE_LOG_FORMAT:-json}
================================================
FILE: config.toml
================================================
ignoreGeneratedHeader = false
severity = "warning"
confidence = 0.8
errorCode = 0
warningCode = 0
[rule.blank-imports]
[rule.context-as-argument]
[rule.context-keys-type]
[rule.dot-imports]
[rule.error-return]
[rule.error-strings]
[rule.error-naming]
[rule.exported]
[rule.if-return]
[rule.increment-decrement]
[rule.var-naming]
[rule.var-declaration]
[rule.package-comments]
[rule.range]
[rule.receiver-naming]
[rule.time-naming]
[rule.unexported-return]
[rule.indent-error-flow]
[rule.errorf]
[rule.empty-block]
[rule.superfluous-else]
[rule.unused-parameter]
[rule.unreachable-code]
[rule.redefines-builtin-id]
================================================
FILE: core/cmd/apps.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package cmd
import (
"context"
"fmt"
"os"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/sdk"
"github.com/olekukonko/tablewriter"
"github.com/spf13/cobra"
)
var (
// Beetle API Server URL
apiURL string
// Beetle API Server API Key
apiKey string
// The Kubernetes Cluster
cluster string
// The Kubernetes Cluster Namespace
namespace string
)
var getCmd = &cobra.Command{
Use: "get",
Short: "Get resources",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println(`You must specify the type of resource to get. Current supported resources are (apps).`)
},
}
var appsCmd = &cobra.Command{
Use: "apps",
Short: "Get a list of applications with cluster id and namespace",
Run: func(cmd *cobra.Command, aras []string) {
// Usage
// $ ./beetle get apps -u "http://localhost:8080" -k "" -c "production" -n "default"
data := [][]string{}
client := sdk.Client{}
client.SetHTTPClient(module.NewHTTPClient(20))
client.SetAPIURL(apiURL)
client.SetAPIKey(apiKey)
apps, err := client.GetApplications(context.TODO(), cluster, namespace)
if err != nil {
data = append(data, []string{
fmt.Sprintf("Error: %s", err.Error()),
"",
"",
"",
})
} else {
for _, app := range apps.Applications {
version := "N/A"
if len(app.Containers) > 0 {
version = app.Containers[0].Version
}
data = append(data, []string{
app.ID,
app.Name,
fmt.Sprintf("%d", len(app.Containers)),
version,
})
}
}
table := tablewriter.NewWriter(os.Stdout)
table.SetHeader([]string{"ID", "Name", "Containers", "Version"})
table.SetAutoWrapText(false)
table.SetAutoFormatHeaders(true)
table.SetHeaderAlignment(tablewriter.ALIGN_LEFT)
table.SetAlignment(tablewriter.ALIGN_LEFT)
table.SetCenterSeparator("")
table.SetColumnSeparator("")
table.SetRowSeparator("")
table.SetHeaderLine(false)
table.SetBorder(false)
table.SetTablePadding("\t")
table.SetNoWhiteSpace(true)
table.AppendBulk(data)
table.Render()
},
}
func init() {
rootCmd.AddCommand(getCmd)
appsCmd.Flags().StringVarP(&namespace, "namespace", "n", "default", "The Kubernetes Cluster Namespace (eg. default)")
appsCmd.MarkFlagRequired("namespace")
appsCmd.Flags().StringVarP(&cluster, "cluster", "c", "", "The Kubernetes Cluster (eg. production)")
appsCmd.MarkFlagRequired("cluster")
appsCmd.Flags().StringVarP(&apiKey, "api_key", "k", "", "API Key of the Beetle API Server")
appsCmd.MarkFlagRequired("api_key")
appsCmd.Flags().StringVarP(&apiURL, "api_url", "u", "", "Beetle API Server URL (eg. https://example.com/)")
appsCmd.MarkFlagRequired("api_url")
getCmd.AddCommand(appsCmd)
}
================================================
FILE: core/cmd/deploy.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package cmd
import (
"context"
"fmt"
"time"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/sdk"
"github.com/briandowns/spinner"
"github.com/logrusorgru/aurora/v3"
"github.com/spf13/cobra"
)
var (
// Application ID
application string
// Application Version
version string
// Deployment Strategy
strategy string
// Ramped Strategy MaxSurge
maxSurge string
// Ramped Strategy MaxUnavailable
maxUnavailable string
// Whether to watch the deployment
watch bool
)
var deployCmd = &cobra.Command{
Use: "deploy",
Short: "Deploy a new application version",
Run: func(cmd *cobra.Command, aras []string) {
// Usage
// $ ./beetle deploy -u "http://localhost:8080" -k "" -c "production" -n "default" -a "toad" -s "recreate" -v "0.2.3" -w
client := sdk.Client{}
client.SetHTTPClient(module.NewHTTPClient(20))
client.SetAPIURL(apiURL)
client.SetAPIKey(apiKey)
spin := spinner.New(spinner.CharSets[26], 100*time.Millisecond)
spin.Color("green")
spin.Start()
job, err := client.CreateDeployment(context.TODO(), model.DeploymentRequest{
Cluster: cluster,
Namespace: namespace,
Application: application,
Version: version,
Strategy: strategy,
MaxSurge: maxSurge,
MaxUnavailable: maxUnavailable,
})
if err != nil {
fmt.Println(aurora.Red(fmt.Sprintf("Error: %s", err.Error())))
spin.Stop()
return
}
if watch {
for {
job, err := client.GetJob(context.TODO(), job.UUID)
if err != nil {
fmt.Println(aurora.Red(fmt.Sprintf("Error: %s", err.Error())))
spin.Stop()
return
}
if job.Status == model.JobFailed {
fmt.Println(aurora.Red(fmt.Sprintf(
"Deployment Request %s Failed!",
job.UUID,
)))
spin.Stop()
return
}
if job.Status == model.JobSuccess {
fmt.Println(aurora.Green(fmt.Sprintf(
"Deployment Request %s Succeeded!",
job.UUID,
)))
spin.Stop()
return
}
time.Sleep(2 * time.Second)
}
} else {
fmt.Println(aurora.Green(fmt.Sprintf(
"Deployment Request %s Submitted Successfully!",
job.UUID,
)))
}
spin.Stop()
},
}
func init() {
deployCmd.Flags().StringVarP(&application, "application", "a", "", "The Application ID")
deployCmd.MarkFlagRequired("application")
deployCmd.Flags().StringVarP(&version, "version", "v", "", "The Application Version")
deployCmd.MarkFlagRequired("version")
deployCmd.Flags().StringVarP(&strategy, "strategy", "s", "recreate", "The Deployment Strategy (recreate, ramped, canary or blue_green)")
deployCmd.MarkFlagRequired("strategy")
deployCmd.Flags().StringVarP(&maxSurge, "max_surge", "g", "50%", "Deployment Strategy MaxSurge")
deployCmd.Flags().StringVarP(&maxUnavailable, "max_unavailable", "b", "50%", "Deployment Strategy MaxUnavailable")
deployCmd.Flags().StringVarP(&namespace, "namespace", "n", "default", "The Kubernetes Cluster Namespace (eg. default)")
deployCmd.MarkFlagRequired("namespace")
deployCmd.Flags().StringVarP(&cluster, "cluster", "c", "", "The Kubernetes Cluster (eg. production)")
deployCmd.MarkFlagRequired("cluster")
deployCmd.Flags().StringVarP(&apiKey, "api_key", "k", "", "API Key of the Beetle API Server")
deployCmd.MarkFlagRequired("api_key")
deployCmd.Flags().StringVarP(&apiURL, "api_url", "u", "", "Beetle API Server URL (eg. https://example.com/)")
deployCmd.MarkFlagRequired("api_url")
deployCmd.Flags().BoolVarP(&watch, "watch", "w", false, "Watch the deployment")
rootCmd.AddCommand(deployCmd)
}
================================================
FILE: core/cmd/license.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package cmd
import (
"fmt"
"github.com/spf13/cobra"
)
var licenseCmd = &cobra.Command{
Use: "license",
Short: "Get License",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println(`MIT License
Copyright (c) 2020 Clivern
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.`)
},
}
func init() {
rootCmd.AddCommand(licenseCmd)
}
================================================
FILE: core/cmd/root.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package cmd
import (
"fmt"
"os"
"github.com/spf13/cobra"
)
var rootCmd = &cobra.Command{
Use: "beetle",
Short: `🔥 Kubernetes multi-cluster deployment automation service
Beetle is in early stages of development, and we'd love to hear your
feedback at <https://github.com/Clivern/Beetle>`,
}
// Execute runs cmd tool
func Execute() {
if err := rootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
================================================
FILE: core/cmd/serve.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package cmd
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"github.com/clivern/beetle/core/controller"
"github.com/clivern/beetle/core/middleware"
"github.com/clivern/beetle/core/module"
"github.com/drone/envsubst"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var config string
var serveCmd = &cobra.Command{
Use: "serve",
Short: "Start beetle server",
Run: func(cmd *cobra.Command, args []string) {
configUnparsed, err := ioutil.ReadFile(config)
if err != nil {
panic(fmt.Sprintf(
"Error while reading config file [%s]: %s",
config,
err.Error(),
))
}
configParsed, err := envsubst.EvalEnv(string(configUnparsed))
if err != nil {
panic(fmt.Sprintf(
"Error while parsing config file [%s]: %s",
config,
err.Error(),
))
}
viper.SetConfigType("yaml")
err = viper.ReadConfig(bytes.NewBufferString(configParsed))
if err != nil {
panic(fmt.Sprintf(
"Error while loading configs [%s]: %s",
config,
err.Error(),
))
}
if viper.GetString("log.output") != "stdout" {
fs := module.FileSystem{}
dir, _ := filepath.Split(viper.GetString("log.output"))
if !fs.DirExists(dir) {
if _, err := fs.EnsureDir(dir, 0775); err != nil {
panic(fmt.Sprintf(
"Directory [%s] creation failed with error: %s",
dir,
err.Error(),
))
}
}
if !fs.FileExists(viper.GetString("log.output")) {
f, err := os.Create(viper.GetString("log.output"))
if err != nil {
panic(fmt.Sprintf(
"Error while creating log file [%s]: %s",
viper.GetString("log.output"),
err.Error(),
))
}
defer f.Close()
}
}
if viper.GetString("log.output") == "stdout" {
gin.DefaultWriter = os.Stdout
log.SetOutput(os.Stdout)
} else {
f, _ := os.OpenFile(
viper.GetString("log.output"),
os.O_APPEND|os.O_CREATE|os.O_WRONLY,
0775,
)
gin.DefaultWriter = io.MultiWriter(f)
log.SetOutput(f)
}
lvl := strings.ToLower(viper.GetString("log.level"))
level, err := log.ParseLevel(lvl)
if err != nil {
level = log.InfoLevel
}
log.SetLevel(level)
if viper.GetString("app.mode") == "prod" {
gin.SetMode(gin.ReleaseMode)
gin.DefaultWriter = ioutil.Discard
gin.DisableConsoleColor()
}
if viper.GetString("log.format") == "json" {
log.SetFormatter(&log.JSONFormatter{})
} else {
log.SetFormatter(&log.TextFormatter{})
}
// Init DB Connection
db := module.Database{}
err = db.AutoConnect()
if err != nil {
panic(err.Error())
}
// Migrate Database
success := db.Migrate()
if !success {
panic("Error! Unable to migrate database tables.")
}
defer db.Close()
messages := make(chan string, viper.GetInt("app.broker.native.capacity"))
r := gin.Default()
r.Use(middleware.Correlation())
r.Use(middleware.Auth())
r.Use(middleware.Logger())
r.Use(middleware.Metric())
r.GET("/favicon.ico", func(c *gin.Context) {
c.String(http.StatusNoContent, "")
})
r.GET("/", controller.HealthCheck)
r.GET("/_health", controller.HealthCheck)
r.GET("/_ready", controller.ReadyCheck)
r.GET(viper.GetString("app.metrics.prometheus.endpoint"), gin.WrapH(controller.Metrics()))
r.GET("/api/v1/cluster", controller.Clusters)
r.GET("/api/v1/cluster/:cn", controller.Cluster)
r.GET("/api/v1/cluster/:cn/namespace", controller.Namespaces)
r.GET("/api/v1/cluster/:cn/namespace/:ns", controller.Namespace)
r.GET("/api/v1/cluster/:cn/namespace/:ns/app", controller.Applications)
r.GET("/api/v1/cluster/:cn/namespace/:ns/app/:id", controller.Application)
r.POST("/api/v1/cluster/:cn/namespace/:ns/app/:id/deployment", func(c *gin.Context) {
controller.CreateDeployment(c, messages)
})
r.GET("/api/v1/job", controller.Jobs)
r.GET("/api/v1/job/:uuid", controller.GetJob)
r.DELETE("/api/v1/job/:uuid", controller.DeleteJob)
for i := 0; i < viper.GetInt("app.broker.native.workers"); i++ {
go controller.Worker(i+1, messages)
}
go controller.Daemon()
var runerr error
if viper.GetBool("app.tls.status") {
runerr = r.RunTLS(
fmt.Sprintf(":%s", strconv.Itoa(viper.GetInt("app.port"))),
viper.GetString("app.tls.pemPath"),
viper.GetString("app.tls.keyPath"),
)
} else {
runerr = r.Run(
fmt.Sprintf(":%s", strconv.Itoa(viper.GetInt("app.port"))),
)
}
if runerr != nil {
panic(runerr.Error())
}
},
}
func init() {
serveCmd.Flags().StringVarP(&config, "config", "c", "config.prod.yml", "Absolute path to config file (required)")
serveCmd.MarkFlagRequired("config")
rootCmd.AddCommand(serveCmd)
}
================================================
FILE: core/cmd/version.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package cmd
import (
"fmt"
"github.com/clivern/beetle/core/module"
"github.com/spf13/cobra"
)
var (
// Version buildinfo item
Version = "dev"
// Commit buildinfo item
Commit = "none"
// Date buildinfo item
Date = "unknown"
// BuiltBy buildinfo item
BuiltBy = "unknown"
)
var versionCmd = &cobra.Command{
Use: "version",
Short: "Get current and latest version",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println(
fmt.Sprintf(
`Current Beetle Version %v Commit %v, Built @%v By %v.`,
Version,
Commit,
Date,
BuiltBy,
),
)
latest, err := module.GetLatestRelease()
if err != nil {
fmt.Printf("Error: %s \n", err.Error())
return
}
fmt.Printf(
"Latest release %s, Latest tag %s \n",
latest.Name,
latest.TagName,
)
},
}
func init() {
rootCmd.AddCommand(versionCmd)
}
================================================
FILE: core/controller/application.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"net/http"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Application controller
func Application(c *gin.Context) {
cn := c.Param("cn")
ns := c.Param("ns")
id := c.Param("id")
config := model.Configs{}
cluster, err := kubernetes.GetCluster(cn)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cn,
"error": err.Error(),
}).Info(`Cluster not found`)
c.Status(http.StatusNotFound)
return
}
config, err = cluster.GetConfig(context.TODO(), ns)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cn,
"namespace_name": ns,
"error": err.Error(),
}).Warn(`Error while fetching beetle configMap`)
}
for _, app := range config.Applications {
if app.ID == id {
application, err := cluster.GetApplication(
context.TODO(),
ns,
app.ID,
app.Name,
app.ImageFormat,
)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"application_id": id,
"cluster_name": cn,
"namespace_name": ns,
"error": err.Error(),
}).Warn(`Error while fetching application current version`)
}
c.JSON(http.StatusOK, gin.H{
"id": application.ID,
"name": application.Name,
"format": application.Format,
"containers": application.Containers,
})
return
}
}
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"application_id": id,
"cluster_name": cn,
"namespace_name": ns,
}).Info(`Application not found`)
c.Status(http.StatusNotFound)
}
================================================
FILE: core/controller/applications.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"net/http"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Applications controller
func Applications(c *gin.Context) {
cn := c.Param("cn")
ns := c.Param("ns")
config := model.Configs{}
cluster, err := kubernetes.GetCluster(cn)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cn,
"error": err.Error(),
}).Info(`Cluster not found`)
c.Status(http.StatusNotFound)
return
}
config, err = cluster.GetConfig(context.TODO(), ns)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cn,
"namespace_name": ns,
"error": err.Error(),
}).Warn(`Error while fetching beetle configMap`)
}
applications := []model.Application{}
for _, app := range config.Applications {
application, err := cluster.GetApplication(
context.TODO(),
ns,
app.ID,
app.Name,
app.ImageFormat,
)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"application_id": app.ID,
"cluster_name": cn,
"namespace_name": ns,
"error": err.Error(),
}).Warn(`Error while fetching application current version`)
continue
}
applications = append(applications, application)
}
c.JSON(http.StatusOK, gin.H{
"applications": applications,
})
}
================================================
FILE: core/controller/cluster.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"net/http"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Cluster controller
func Cluster(c *gin.Context) {
cn := c.Param("cn")
result := model.Cluster{}
cluster, err := kubernetes.GetCluster(cn)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cn,
"error": err.Error(),
}).Info(`Cluster not found`)
c.Status(http.StatusNotFound)
return
}
status, err := cluster.Ping(context.TODO())
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cn,
"error": err.Error(),
}).Error(`Error ping a cluster`)
}
result.Name = cluster.Name
result.Health = status
c.JSON(http.StatusOK, gin.H{
"name": result.Name,
"health": result.Health,
})
}
================================================
FILE: core/controller/clusters.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"net/http"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Clusters controller
func Clusters(c *gin.Context) {
result := []model.Cluster{}
clusters, err := kubernetes.GetClusters()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Error fetching clusters`)
c.Status(http.StatusInternalServerError)
return
}
var status bool
for _, cluster := range clusters {
status, err = cluster.Ping(context.TODO())
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"cluster_name": cluster.Name,
"error": err.Error(),
}).Error(`Error while ping a cluster`)
}
result = append(result, model.Cluster{
Name: cluster.Name,
Health: status,
})
}
c.JSON(http.StatusOK, gin.H{
"clusters": result,
})
}
================================================
FILE: core/controller/daemon.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"fmt"
"net/http"
"strconv"
"time"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/core/module"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"github.com/spf13/viper"
)
var (
pendingJobs = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "beetle",
Name: "workers_queue_pending_jobs",
Help: "The pending jobs in the queue",
})
failedJobs = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "beetle",
Name: "workers_queue_failed_jobs",
Help: "The failed jobs in the queue",
})
successJobs = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "beetle",
Name: "workers_queue_success_jobs",
Help: "The successful jobs in the queue",
})
onHoldJobs = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "beetle",
Name: "workers_queue_on_hold_jobs",
Help: "The on hold jobs in the queue",
})
)
func init() {
prometheus.MustRegister(pendingJobs)
prometheus.MustRegister(failedJobs)
prometheus.MustRegister(successJobs)
prometheus.MustRegister(onHoldJobs)
}
// Daemon function
func Daemon() {
var err error
var pendingJobsCount int
var failedJobsCount int
var successfulJobsCount int
var onHoldJobsCount int
var job model.Job
var parentJob model.Job
var deploymentRequest model.DeploymentRequest
var payload string
httpClient := module.NewHTTPClient(20)
db := module.Database{}
retry, err := strconv.Atoi(viper.GetString("app.webhook.retry"))
if err != nil {
panic(err.Error())
}
for {
err = db.AutoConnect()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": "",
"error": err.Error(),
}).Error(`Failure while connecting database`)
time.Sleep(2 * time.Second)
continue
}
// Update Metrics
pendingJobsCount = db.CountJobs(model.JobPending)
failedJobsCount = db.CountJobs(model.JobFailed)
successfulJobsCount = db.CountJobs(model.JobSuccess)
onHoldJobsCount = db.CountJobs(model.JobOnHold)
log.WithFields(log.Fields{
"correlation_id": "",
"pending_jobs_count": pendingJobsCount,
"failed_jobs_count": failedJobsCount,
"successful_jobs_count": successfulJobsCount,
"on_hold_jobs_count": onHoldJobsCount,
}).Debug(`Update metrics`)
pendingJobs.Set(float64(pendingJobsCount))
failedJobs.Set(float64(failedJobsCount))
successJobs.Set(float64(successfulJobsCount))
onHoldJobs.Set(float64(onHoldJobsCount))
// Run Pending Jobs (HTTP Notification)
job = db.GetPendingJobByType(model.JobDeploymentNotify)
if job.ID > 0 {
if job.Retry > retry {
now := time.Now()
job.Status = model.JobFailed
job.RunAt = &now
job.Result = fmt.Sprintf("Failed to deliver the notification")
db.UpdateJobByID(&job)
} else {
deploymentRequest.LoadFromJSON([]byte(job.Payload))
if job.Parent > 0 {
parentJob = db.GetJobByID(job.Parent)
if parentJob.ID > 0 {
deploymentRequest.Status = parentJob.Status
}
}
payload, _ = deploymentRequest.ConvertToJSON()
response, err := httpClient.Post(
context.TODO(),
viper.GetString("app.webhook.url"),
payload,
map[string]string{},
map[string]string{
"Content-Type": "application/json",
"X-API-KEY": viper.GetString("app.webhook.apiKey"),
"X-NOTIFICATION-ID": job.UUID,
"X-ACTION-NAME": job.Type,
"X-DEPLOYMENT-ID": parentJob.UUID,
},
)
if httpClient.GetStatusCode(response) != http.StatusOK || err != nil {
job.Status = model.JobFailed
job.Result = fmt.Sprintf("Failed to deliver the notification")
} else {
job.Status = model.JobSuccess
job.Result = fmt.Sprintf("Notification delivered successfully")
}
if job.Status == model.JobFailed && job.Retry <= retry {
job.Status = model.JobPending
}
now := time.Now()
job.Retry++
job.RunAt = &now
db.UpdateJobByID(&job)
}
}
time.Sleep(2 * time.Second)
}
}
================================================
FILE: core/controller/deployment.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"net/http"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/core/util"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// CreateDeployment controller
func CreateDeployment(c *gin.Context, messages chan<- string) {
rawBody, _ := c.GetRawData()
deploymentRequest := model.DeploymentRequest{}
_, err := deploymentRequest.LoadFromJSON(rawBody)
deploymentRequest.Cluster = c.Param("cn")
deploymentRequest.Namespace = c.Param("ns")
deploymentRequest.Application = c.Param("id")
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Info(`Invalid request`)
c.JSON(http.StatusBadRequest, gin.H{
"error": "Invalid request!",
})
return
}
err = deploymentRequest.Validate([]string{
model.RecreateStrategy,
model.RampedStrategy,
model.CanaryStrategy,
model.BlueGreenStrategy,
})
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Info(`Invalid request`)
c.JSON(http.StatusBadRequest, gin.H{
"error": err.Error(),
})
return
}
if deploymentRequest.MaxSurge == "" {
deploymentRequest.MaxSurge = "25%"
}
if deploymentRequest.MaxUnavailable == "" {
deploymentRequest.MaxUnavailable = "25%"
}
result, err := deploymentRequest.ConvertToJSON()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Info(`Invalid request`)
c.JSON(http.StatusBadRequest, gin.H{
"error": err.Error(),
})
return
}
// Then create async job
db := module.Database{}
err = db.AutoConnect()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Failure while connecting database`)
c.Status(http.StatusInternalServerError)
return
}
defer db.Close()
uuid := util.GenerateUUID4()
for db.JobExistByUUID(uuid) {
uuid = util.GenerateUUID4()
}
job := db.CreateJob(&model.Job{
UUID: uuid,
Payload: result,
Status: model.JobPending,
Parent: 0,
Type: model.JobDeploymentUpdate,
})
messageObj := model.Message{
UUID: c.Request.Header.Get("X-Correlation-ID"),
Job: job.ID,
}
message, _ := messageObj.ConvertToJSON()
// Send the job to workers
messages <- message
c.JSON(http.StatusAccepted, gin.H{
"id": job.ID,
"uuid": job.UUID,
"type": job.Type,
"status": job.Status,
"createdAt": job.CreatedAt,
})
}
================================================
FILE: core/controller/health_check.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"net/http"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// HealthCheck controller
func HealthCheck(c *gin.Context) {
status := "ok"
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"status": status,
}).Info(`Health check`)
c.JSON(http.StatusOK, gin.H{
"status": status,
})
}
================================================
FILE: core/controller/health_check_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"bytes"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/pkg"
"github.com/drone/envsubst"
"github.com/gin-gonic/gin"
"github.com/spf13/viper"
)
// TestHealthCheck test cases
func TestHealthCheck(t *testing.T) {
testingConfig := "config.testing.yml"
// LoadConfigFile
t.Run("LoadConfigFile", func(t *testing.T) {
fs := module.FileSystem{}
dir, _ := os.Getwd()
configFile := fmt.Sprintf("%s/%s", dir, testingConfig)
for {
if fs.FileExists(configFile) {
break
}
dir = filepath.Dir(dir)
configFile = fmt.Sprintf("%s/%s", dir, testingConfig)
}
t.Logf("Load Config File %s", configFile)
configUnparsed, _ := ioutil.ReadFile(configFile)
configParsed, _ := envsubst.EvalEnv(string(configUnparsed))
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer([]byte(configParsed)))
})
// TestHealthCheckController
t.Run("TestHealthCheckController", func(t *testing.T) {
gin.SetMode(gin.ReleaseMode)
gin.DefaultWriter = ioutil.Discard
gin.DisableConsoleColor()
router := gin.Default()
router.GET("/_health", HealthCheck)
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/_health", nil)
router.ServeHTTP(w, req)
pkg.Expect(t, viper.GetString("app.mode"), "test")
pkg.Expect(t, w.Code, 200)
pkg.Expect(t, strings.TrimSpace(w.Body.String()), `{"status":"ok"}`)
})
}
================================================
FILE: core/controller/job.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"fmt"
"net/http"
"github.com/clivern/beetle/core/module"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// GetJob controller
func GetJob(c *gin.Context) {
uuid := c.Param("uuid")
db := module.Database{}
err := db.AutoConnect()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Failure while connecting database`)
c.Status(http.StatusInternalServerError)
return
}
defer db.Close()
job := db.GetJobByUUID(uuid)
if job.ID < 1 {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"job_uuid": uuid,
}).Info(fmt.Sprintf(`Job not found`))
c.Status(http.StatusNotFound)
return
}
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"job_uuid": uuid,
}).Info(`Retrieve a job`)
c.JSON(http.StatusOK, gin.H{
"id": job.ID,
"uuid": job.UUID,
"status": job.Status,
"type": job.Type,
"runAt": job.RunAt,
"createdAt": job.CreatedAt,
"updatedAt": job.UpdatedAt,
})
}
// DeleteJob controller
func DeleteJob(c *gin.Context) {
uuid := c.Param("uuid")
db := module.Database{}
err := db.AutoConnect()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Failure while connecting database`)
c.Status(http.StatusInternalServerError)
return
}
defer db.Close()
job := db.GetJobByUUID(uuid)
if job.ID < 1 {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"job_uuid": uuid,
}).Info(`Job not found`)
c.Status(http.StatusNotFound)
return
}
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"job_uuid": uuid,
}).Info(`Deleting a job`)
db.DeleteJobByID(job.ID)
c.Status(http.StatusNoContent)
return
}
================================================
FILE: core/controller/jobs.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"net/http"
"github.com/clivern/beetle/core/module"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Jobs controller
func Jobs(c *gin.Context) {
db := module.Database{}
err := db.AutoConnect()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Failure while connecting database`)
c.Status(http.StatusInternalServerError)
return
}
defer db.Close()
c.JSON(http.StatusOK, gin.H{
"jobs": db.GetJobs(),
})
}
================================================
FILE: core/controller/metrics.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"net/http"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/spf13/viper"
)
var (
workersCount = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "beetle",
Name: "workers_count",
Help: "Number of Async Workers",
})
queueCapacity = prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: "beetle",
Name: "workers_queue_capacity",
Help: "The maximum number of messages queue can process",
})
)
func init() {
prometheus.MustRegister(workersCount)
prometheus.MustRegister(queueCapacity)
}
// Metrics controller
func Metrics() http.Handler {
workersCount.Set(float64(viper.GetInt("app.broker.native.workers")))
queueCapacity.Set(float64(viper.GetInt("app.broker.native.capacity")))
return promhttp.Handler()
}
================================================
FILE: core/controller/namespace.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"net/http"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Namespace controller
func Namespace(c *gin.Context) {
cn := c.Param("cn")
ns := c.Param("ns")
result := model.Namespace{}
clusters, err := kubernetes.GetClusters()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Failure to get clusters`)
c.Status(http.StatusInternalServerError)
return
}
for _, cluster := range clusters {
if cn != cluster.Name {
continue
}
result, err = cluster.GetNamespace(context.TODO(), ns)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"namespace_name": ns,
"cluster_name": cn,
"error": err.Error(),
}).Error(`Failure to get cluster namespace`)
}
}
if result.Name == "" {
c.Status(http.StatusNotFound)
return
}
c.JSON(http.StatusOK, gin.H{
"name": result.Name,
"uid": result.UID,
"status": result.Status,
})
}
================================================
FILE: core/controller/namespaces.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"net/http"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Namespaces controller
func Namespaces(c *gin.Context) {
cn := c.Param("cn")
result := []model.Namespace{}
clusters, err := kubernetes.GetClusters()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
}).Error(`Failure to get clusters`)
c.Status(http.StatusInternalServerError)
return
}
for _, cluster := range clusters {
if cn != cluster.Name {
continue
}
result, err = cluster.GetNamespaces(context.TODO())
if err != nil {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"error": err.Error(),
"cluster_name": cn,
}).Error(`Failure to get cluster namespaces`)
}
}
c.JSON(http.StatusOK, gin.H{
"namespaces": result,
})
}
================================================
FILE: core/controller/ready_check.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"net/http"
"github.com/clivern/beetle/core/module"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// ReadyCheck controller
func ReadyCheck(c *gin.Context) {
status := "ok"
db := module.Database{}
err := db.AutoConnect()
if err != nil {
status = "down"
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"status": status,
"error": err.Error(),
}).Error(`Failed ready check`)
c.Status(http.StatusInternalServerError)
return
}
err = db.Ping()
if err != nil {
status = "down"
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"status": status,
"error": err.Error(),
}).Error(`Failed ready check`)
c.Status(http.StatusInternalServerError)
return
}
defer db.Close()
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"status": status,
}).Info(`Passed ready check`)
c.JSON(http.StatusOK, gin.H{
"status": status,
})
}
================================================
FILE: core/controller/ready_check_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"bytes"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/pkg"
"github.com/drone/envsubst"
"github.com/gin-gonic/gin"
"github.com/spf13/viper"
)
// TestReadyCheck test cases
func TestReadyCheck(t *testing.T) {
testingConfig := "config.testing.yml"
// LoadConfigFile
t.Run("LoadConfigFile", func(t *testing.T) {
fs := module.FileSystem{}
dir, _ := os.Getwd()
configFile := fmt.Sprintf("%s/%s", dir, testingConfig)
for {
if fs.FileExists(configFile) {
break
}
dir = filepath.Dir(dir)
configFile = fmt.Sprintf("%s/%s", dir, testingConfig)
}
t.Logf("Load Config File %s", configFile)
configUnparsed, _ := ioutil.ReadFile(configFile)
configParsed, _ := envsubst.EvalEnv(string(configUnparsed))
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer([]byte(configParsed)))
})
// TestReadyCheckController
t.Run("TestReadyCheckController", func(t *testing.T) {
gin.SetMode(gin.ReleaseMode)
gin.DefaultWriter = ioutil.Discard
gin.DisableConsoleColor()
router := gin.Default()
router.GET("/_ready", ReadyCheck)
w := httptest.NewRecorder()
req, _ := http.NewRequest("GET", "/_ready", nil)
router.ServeHTTP(w, req)
pkg.Expect(t, viper.GetString("app.mode"), "test")
pkg.Expect(t, w.Code, 200)
pkg.Expect(t, strings.TrimSpace(w.Body.String()), `{"status":"ok"}`)
})
}
================================================
FILE: core/controller/worker.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package controller
import (
"context"
"fmt"
"strings"
"time"
"github.com/clivern/beetle/core/kubernetes"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/core/util"
log "github.com/sirupsen/logrus"
"github.com/spf13/viper"
)
// Worker controller
func Worker(workerID int, messages <-chan string) {
var ok bool
var err error
var job model.Job
var cluster *kubernetes.Cluster
var uuid string
messageObj := model.Message{}
deploymentRequest := model.DeploymentRequest{}
log.WithFields(log.Fields{
"correlation_id": util.GenerateUUID4(),
"worker_id": workerID,
}).Info(`Worker started`)
db := module.Database{}
for message := range messages {
ok, err = messageObj.LoadFromJSON([]byte(message))
if !ok || err != nil {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"message": message,
}).Warn(`Worker received invalid message`)
continue
}
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"job_id": messageObj.Job,
}).Info(`Worker received a new job`)
err = db.AutoConnect()
if err != nil {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"error": err.Error(),
}).Error(`Worker unable to connect to database`)
continue
}
job = db.GetJobByID(messageObj.Job)
ok, err = deploymentRequest.LoadFromJSON([]byte(job.Payload))
if !ok || err != nil {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"job_id": messageObj.Job,
"job_uuid": job.UUID,
"error": err.Error(),
}).Error(`Invalid job payload`)
// Job Failed
now := time.Now()
job.Status = model.JobFailed
job.RunAt = &now
job.Result = fmt.Sprintf("Invalid job payload, UUID %s", messageObj.UUID)
db.UpdateJobByID(&job)
db.ReleaseChildJobs(job.ID)
continue
}
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"job_id": messageObj.Job,
"job_uuid": job.UUID,
"deployment_request": deploymentRequest,
}).Info(`Worker accepted deployment request`)
// Notify if there is a webhook
if strings.TrimSpace(viper.GetString("app.webhook.url")) != "" {
uuid = util.GenerateUUID4()
for db.JobExistByUUID(uuid) {
uuid = util.GenerateUUID4()
}
db.CreateJob(&model.Job{
UUID: uuid,
Payload: job.Payload,
Status: model.JobOnHold,
Parent: messageObj.Job,
Type: model.JobDeploymentNotify,
})
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"job_id": messageObj.Job,
"job_uuid": job.UUID,
"deployment_request": deploymentRequest,
"webhook_url": viper.GetString("app.webhook.url"),
}).Info(`HTTP webhook enabled`)
} else {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"job_id": messageObj.Job,
"job_uuid": job.UUID,
"deployment_request": deploymentRequest,
}).Info(`HTTP webhook disabled`)
}
cluster, err = kubernetes.GetCluster(deploymentRequest.Cluster)
if err != nil {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"error": err.Error(),
"deployment_request": deploymentRequest,
}).Error(`Worker can not find the cluster`)
// Job Failed
now := time.Now()
job.Status = model.JobFailed
job.RunAt = &now
job.Result = fmt.Sprintf("Worker can not find the cluster, UUID %s", messageObj.UUID)
db.UpdateJobByID(&job)
db.ReleaseChildJobs(job.ID)
continue
}
ok, err = cluster.Ping(context.TODO())
if !ok || err != nil {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"error": err.Error(),
"deployment_request": deploymentRequest,
}).Error(`Worker unable to ping cluster`)
// Job Failed
now := time.Now()
job.Status = model.JobFailed
job.RunAt = &now
job.Result = fmt.Sprintf("Worker unable to ping cluster, UUID %s", messageObj.UUID)
db.UpdateJobByID(&job)
db.ReleaseChildJobs(job.ID)
continue
}
ok, err = cluster.Deploy(deploymentRequest)
if !ok || err != nil {
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"error": err.Error(),
"deployment_request": deploymentRequest,
}).Error(`Worker unable deploy`)
// Job Failed
now := time.Now()
job.Status = model.JobFailed
job.RunAt = &now
job.Result = fmt.Sprintf("Failure during deployment, UUID %s", messageObj.UUID)
db.UpdateJobByID(&job)
db.ReleaseChildJobs(job.ID)
continue
}
log.WithFields(log.Fields{
"correlation_id": messageObj.UUID,
"worker_id": workerID,
"deployment_request": deploymentRequest,
}).Info(`Deployment finished successfully`)
// Job Succeeded
now := time.Now()
job.Status = model.JobSuccess
job.RunAt = &now
job.Result = "Deployment finished successfully"
db.UpdateJobByID(&job)
db.ReleaseChildJobs(job.ID)
}
}
================================================
FILE: core/kubernetes/application.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"fmt"
"strings"
"github.com/clivern/beetle/core/model"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// GetApplication gets current application version
func (c *Cluster) GetApplication(ctx context.Context, namespace, id, name, format string) (model.Application, error) {
result := model.Application{}
err := c.Config()
if err != nil {
return result, err
}
data, err := c.ClientSet.AppsV1().Deployments(namespace).List(ctx, metav1.ListOptions{
LabelSelector: fmt.Sprintf(
"%s=%s,%s=%s",
"beetle.clivern.com/status",
"enabled",
"beetle.clivern.com/application-id",
id,
),
})
if err != nil {
return result, err
}
result.ID = id
result.Name = name
result.Format = format
result.Containers = []model.Container{}
for _, deployment := range data.Items {
for _, container := range deployment.Spec.Template.Spec.Containers {
result.Containers = append(result.Containers, model.Container{
Name: container.Name,
Image: container.Image,
Version: strings.Replace(
container.Image,
strings.Replace(format, "[.Release]", "", -1),
"",
-1,
),
Deployment: model.Deployment{
Name: deployment.ObjectMeta.Name,
UID: string(deployment.ObjectMeta.UID),
},
})
}
}
return result, nil
}
================================================
FILE: core/kubernetes/cluster.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"fmt"
"github.com/clivern/beetle/core/module"
"github.com/spf13/viper"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/kubernetes/fake"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
)
// Clusters struct
type Clusters struct {
Clusters []*Cluster `mapstructure:",clusters"`
}
// Cluster struct
type Cluster struct {
Name string `mapstructure:",name"`
Kubeconfig string `mapstructure:",kubeconfig"`
InCluster bool `mapstructure:",inCluster"`
ClientSet kubernetes.Interface
Fake bool
}
// GetClusters get a list of clusters
func GetClusters() ([]*Cluster, error) {
var clusters Clusters
err := viper.UnmarshalKey("app", &clusters)
if err != nil {
return nil, err
}
return clusters.Clusters, nil
}
// GetCluster get a list of clusters
func GetCluster(name string) (*Cluster, error) {
var clusters Clusters
err := viper.UnmarshalKey("app", &clusters)
if err != nil {
return nil, err
}
for _, cluster := range clusters.Clusters {
if name == cluster.Name {
return cluster, nil
}
}
return &Cluster{}, fmt.Errorf("Unable to find cluster %s", name)
}
// Override overrides the client set for testing
func (c *Cluster) Override(objects ...runtime.Object) {
c.Fake = true
c.ClientSet = fake.NewSimpleClientset(objects...)
}
// Config configs the client set for testing
func (c *Cluster) Config() error {
if c.Fake {
return nil
}
var config *rest.Config
var err error
if !c.InCluster {
fs := module.FileSystem{}
if !fs.FileExists(c.Kubeconfig) {
return fmt.Errorf(
"cluster [%s] config file [%s] not exist",
c.Name,
c.Kubeconfig,
)
}
config, err = clientcmd.BuildConfigFromFlags("", c.Kubeconfig)
if err != nil {
return err
}
} else {
config, err = rest.InClusterConfig()
if err != nil {
return err
}
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
return err
}
c.ClientSet = clientset
return nil
}
// Ping check the cluster
func (c *Cluster) Ping(ctx context.Context) (bool, error) {
err := c.Config()
if err != nil {
return false, err
}
data, err := c.ClientSet.CoreV1().RESTClient().Get().AbsPath("/api/v1").DoRaw(ctx)
if err != nil {
return false, err
}
return (string(data) != ""), nil
}
================================================
FILE: core/kubernetes/cluster_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"bytes"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/pkg"
"github.com/drone/envsubst"
"github.com/spf13/viper"
)
// TestCluster test cases
func TestCluster(t *testing.T) {
testingConfig := "config.testing.yml"
// LoadConfigFile
t.Run("LoadConfigFile", func(t *testing.T) {
fs := module.FileSystem{}
dir, _ := os.Getwd()
configFile := fmt.Sprintf("%s/%s", dir, testingConfig)
for {
if fs.FileExists(configFile) {
break
}
dir = filepath.Dir(dir)
configFile = fmt.Sprintf("%s/%s", dir, testingConfig)
}
t.Logf("Load Config File %s", configFile)
configUnparsed, _ := ioutil.ReadFile(configFile)
configParsed, _ := envsubst.EvalEnv(string(configUnparsed))
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer([]byte(configParsed)))
})
// TestGetClusters
t.Run("TestGetClusters", func(t *testing.T) {
clusters, err := GetClusters()
pkg.Expect(t, nil, err)
pkg.Expect(t, clusters[0].Name, "production")
pkg.Expect(t, clusters[0].Kubeconfig, "/app/configs/production-cluster-kubeconfig.yaml")
pkg.Expect(t, clusters[1].Name, "staging")
pkg.Expect(t, clusters[1].Kubeconfig, "/app/configs/staging-cluster-kubeconfig.yaml")
})
// TestGetCluster
t.Run("TestGetCluster", func(t *testing.T) {
cluster, err := GetCluster("production")
pkg.Expect(t, nil, err)
pkg.Expect(t, cluster.Name, "production")
pkg.Expect(t, cluster.Kubeconfig, "/app/configs/production-cluster-kubeconfig.yaml")
cluster, err = GetCluster("not-found")
pkg.Expect(t, fmt.Errorf("Unable to find cluster not-found"), err)
pkg.Expect(t, cluster.Name, "")
pkg.Expect(t, cluster.Kubeconfig, "")
})
}
================================================
FILE: core/kubernetes/config.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"fmt"
"github.com/clivern/beetle/core/model"
log "github.com/sirupsen/logrus"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// GetConfig gets a beetle configs for a specific namespace
func (c *Cluster) GetConfig(ctx context.Context, namespace string) (model.Configs, error) {
result := model.Configs{}
err := c.Config()
if err != nil {
return result, err
}
data, err := c.ClientSet.AppsV1().Deployments(namespace).List(ctx, metav1.ListOptions{
LabelSelector: fmt.Sprintf(
"%s=%s",
"beetle.clivern.com/status",
"enabled",
),
})
if err != nil {
return result, err
}
for _, deployment := range data.Items {
applicationName := ""
imageFormat := ""
applicationID := ""
status := "disabled"
for key, value := range deployment.ObjectMeta.Annotations {
if key == "beetle.clivern.com/application-name" {
applicationName = value
}
if key == "beetle.clivern.com/image-format" {
imageFormat = value
}
}
for key, value := range deployment.ObjectMeta.Labels {
if key == "beetle.clivern.com/status" {
status = value
}
if key == "beetle.clivern.com/application-id" {
applicationID = value
}
}
if status == "enabled" && applicationID != "" && imageFormat != "" {
result.Applications = append(result.Applications, model.App{
ID: applicationID,
Name: applicationName,
ImageFormat: imageFormat,
})
} else {
log.WithFields(log.Fields{
"application_id": applicationID,
}).Debug(`Application status disabled`)
}
}
result.Exists = true
return result, nil
}
================================================
FILE: core/kubernetes/configmap.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"github.com/clivern/beetle/core/model"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// GetConfigMap gets a configmap data
func (c *Cluster) GetConfigMap(ctx context.Context, namespace, name string) (model.ConfigMap, error) {
result := model.ConfigMap{}
err := c.Config()
if err != nil {
return result, err
}
configmap, err := c.ClientSet.CoreV1().ConfigMaps(namespace).Get(ctx, name, metav1.GetOptions{})
if err != nil {
return result, err
}
result.Name = configmap.ObjectMeta.Name
result.Namespace = configmap.ObjectMeta.Namespace
result.UID = string(configmap.ObjectMeta.UID)
result.CreationTimestamp = configmap.ObjectMeta.CreationTimestamp.String()
result.Data = configmap.Data
result.Labels = configmap.ObjectMeta.Labels
return result, nil
}
================================================
FILE: core/kubernetes/deployment.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"fmt"
"time"
"github.com/clivern/beetle/core/model"
log "github.com/sirupsen/logrus"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
types "k8s.io/apimachinery/pkg/types"
)
// GetDeployments gets a list of deployments
func (c *Cluster) GetDeployments(ctx context.Context, namespace, label string) ([]model.Deployment, error) {
result := []model.Deployment{}
err := c.Config()
if err != nil {
return result, err
}
data, err := c.ClientSet.AppsV1().Deployments(namespace).List(ctx, metav1.ListOptions{
LabelSelector: label,
})
if err != nil {
return result, err
}
for _, deployment := range data.Items {
result = append(result, model.Deployment{
Name: deployment.ObjectMeta.Name,
UID: string(deployment.ObjectMeta.UID),
})
}
return result, nil
}
// GetDeployment gets a deployment by name
func (c *Cluster) GetDeployment(ctx context.Context, namespace, name string) (model.Deployment, error) {
result := model.Deployment{}
err := c.Config()
if err != nil {
return result, err
}
deployment, err := c.ClientSet.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
if err != nil {
return result, err
}
result.Name = deployment.ObjectMeta.Name
result.UID = string(deployment.ObjectMeta.UID)
return result, nil
}
// PatchDeployment updates the deployment
func (c *Cluster) PatchDeployment(ctx context.Context, namespace, name, data string) (bool, error) {
err := c.Config()
if err != nil {
return false, err
}
_, err = c.ClientSet.AppsV1().Deployments(namespace).Patch(
ctx,
name,
types.JSONPatchType,
[]byte(data),
metav1.PatchOptions{},
)
if err != nil {
return false, err
}
return true, nil
}
// FetchDeploymentStatus get deployment status
func (c *Cluster) FetchDeploymentStatus(ctx context.Context, namespace, name string, limit int) (bool, error) {
err := c.Config()
if err != nil {
return false, err
}
// Wait till k8s pick the deployment
time.Sleep(10 * time.Second)
deployment, err := c.ClientSet.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
if err != nil {
return false, err
}
status := true
for i := 0; i < limit; i++ {
status = true
deployment, err = c.ClientSet.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
if err != nil {
return false, err
}
if int(deployment.Generation) != int(deployment.Status.ObservedGeneration) {
status = false
}
if int(deployment.Status.UnavailableReplicas) > 0 {
status = false
}
if int(int32(*deployment.Spec.Replicas)) != int(deployment.Status.AvailableReplicas) {
status = false
}
if !status {
log.WithFields(log.Fields{
"deployment.Generation": int(deployment.Generation),
"deployment.Status.ObservedGeneration": int(deployment.Status.ObservedGeneration),
"deployment.Spec.Replicas": int(int32(*deployment.Spec.Replicas)),
"deployment.Status.AvailableReplicas": int(deployment.Status.AvailableReplicas),
"deployment.Status.UnavailableReplicas": int(deployment.Status.UnavailableReplicas),
}).Debug(`Deployment Success`)
time.Sleep(2 * time.Second)
} else {
log.WithFields(log.Fields{
"deployment.Generation": int(deployment.Generation),
"deployment.Status.ObservedGeneration": int(deployment.Status.ObservedGeneration),
"deployment.Spec.Replicas": int(int32(*deployment.Spec.Replicas)),
"deployment.Status.AvailableReplicas": int(deployment.Status.AvailableReplicas),
"deployment.Status.UnavailableReplicas": int(deployment.Status.UnavailableReplicas),
}).Debug(`Deployment Success`)
return true, nil
}
}
log.WithFields(log.Fields{
"deployment.Generation": int(deployment.Generation),
"deployment.Status.ObservedGeneration": int(deployment.Status.ObservedGeneration),
"deployment.Spec.Replicas": int(int32(*deployment.Spec.Replicas)),
"deployment.Status.AvailableReplicas": int(deployment.Status.AvailableReplicas),
"deployment.Status.UnavailableReplicas": int(deployment.Status.UnavailableReplicas),
}).Debug(`Deployment failure`)
return false, fmt.Errorf(fmt.Sprintf(
"Deployment %s failed: namespace %s, Generation %d, ObservedGeneration %d,"+
" UnavailableReplicas %d, Replicas %d, AvailableReplicas %d",
name,
namespace,
int(deployment.Generation),
int(deployment.Status.ObservedGeneration),
int(deployment.Status.UnavailableReplicas),
int(int32(*deployment.Spec.Replicas)),
int(deployment.Status.AvailableReplicas),
))
}
================================================
FILE: core/kubernetes/deployment_strategy.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"fmt"
"strconv"
"strings"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/core/util"
)
// Deploy deploys an application
func (c *Cluster) Deploy(deploymentRequest model.DeploymentRequest) (bool, error) {
switch strategy := deploymentRequest.Strategy; strategy {
case model.RecreateStrategy:
return c.RecreateStrategy(deploymentRequest)
case model.RampedStrategy:
return c.RampedStrategy(deploymentRequest)
case model.CanaryStrategy:
return c.CanaryStrategy(deploymentRequest)
case model.BlueGreenStrategy:
return c.BlueGreenStrategy(deploymentRequest)
default:
return false, fmt.Errorf("Invalid deployment strategy %s", strategy)
}
}
// RecreateStrategy terminates the old version and release the new one.
//
// # This method is like running this command
//
// $ kubectl patch deployment toad-deployment --type=json -p '[
//
// {"op":"replace", "path":"/spec/strategy", "value":{"type":"Recreate"}},
// {"op":"replace","path":"/spec/template/spec/containers/0/image","value":"clivern/toad:release-0.2.4"}
//
// ]'
func (c *Cluster) RecreateStrategy(deploymentRequest model.DeploymentRequest) (bool, error) {
result := model.Application{}
patch := make(map[string][]model.PatchStringValue)
config, err := c.GetConfig(context.TODO(), deploymentRequest.Namespace)
if err != nil {
return false, err
}
for _, app := range config.Applications {
if app.ID == deploymentRequest.Application {
result, err = c.GetApplication(
context.TODO(),
deploymentRequest.Namespace,
app.ID,
app.Name,
app.ImageFormat,
)
if err != nil {
return false, err
}
break
}
}
i := 0
for _, container := range result.Containers {
if _, ok := patch[container.Deployment.Name]; !ok {
patch[container.Deployment.Name] = []model.PatchStringValue{}
}
patch[container.Deployment.Name] = append(patch[container.Deployment.Name], model.PatchStringValue{
Op: "replace",
Path: fmt.Sprintf("/spec/template/spec/containers/%d/image", i),
Value: strings.Replace(container.Image, container.Version, deploymentRequest.Version, -1),
})
i++
}
data := ""
status := true
for deploymentName, deploymentPatch := range patch {
data, err = util.ConvertToJSON(deploymentPatch)
if err != nil {
return false, err
}
// Enforce Recreate strategy
data = strings.Replace(
data,
`[`,
`[{"op":"replace","path":"/spec/strategy","value":{"type":"Recreate"}},`,
-1,
)
status, err = c.PatchDeployment(
context.TODO(),
deploymentRequest.Namespace,
deploymentName,
data,
)
if !status || err != nil {
return false, err
}
}
for deploymentName := range patch {
status, err = c.FetchDeploymentStatus(context.TODO(), deploymentRequest.Namespace, deploymentName, 600)
if !status || err != nil {
return false, err
}
}
return true, nil
}
// RampedStrategy releases a new version on a rolling update fashion, one after the other.
//
// it will set maxSurge as 25% and maxUnavailable as 25%
//
// # This method is like running this command
//
// $ kubectl patch deployment toad-deployment --type=json -p '[
//
// {"op":"replace", "path":"/spec/strategy", "value":{"type":"RollingUpdate"}},
// {"op":"replace", "path":"/spec/strategy/rollingUpdate", "value":{"maxSurge":""}},
// {"op":"replace", "path":"/spec/strategy/rollingUpdate", "value":{"maxUnavailable":""}},
// {"op":"replace","path":"/spec/template/spec/containers/0/image","value":"clivern/toad:release-0.2.4"}
//
// ]'
func (c *Cluster) RampedStrategy(deploymentRequest model.DeploymentRequest) (bool, error) {
result := model.Application{}
patch := make(map[string][]model.PatchStringValue)
config, err := c.GetConfig(context.TODO(), deploymentRequest.Namespace)
if err != nil {
return false, err
}
for _, app := range config.Applications {
if app.ID == deploymentRequest.Application {
result, err = c.GetApplication(
context.TODO(),
deploymentRequest.Namespace,
app.ID,
app.Name,
app.ImageFormat,
)
if err != nil {
return false, err
}
break
}
}
i := 0
for _, container := range result.Containers {
if _, ok := patch[container.Deployment.Name]; !ok {
patch[container.Deployment.Name] = []model.PatchStringValue{}
}
patch[container.Deployment.Name] = append(patch[container.Deployment.Name], model.PatchStringValue{
Op: "replace",
Path: fmt.Sprintf("/spec/template/spec/containers/%d/image", i),
Value: strings.Replace(container.Image, container.Version, deploymentRequest.Version, -1),
})
i++
}
data := ""
status := true
for deploymentName, deploymentPatch := range patch {
data, err = util.ConvertToJSON(deploymentPatch)
if err != nil {
return false, err
}
diff := ""
if strings.Contains(deploymentRequest.MaxSurge, "%") && strings.Contains(deploymentRequest.MaxUnavailable, "%") {
diff = fmt.Sprintf(
`[{"op":"replace","path":"/spec/strategy","value":{"type":"RollingUpdate"}},`+
`{"op":"replace", "path":"/spec/strategy/rollingUpdate", "value":{"maxSurge":"%s"}},`+
`{"op":"replace", "path":"/spec/strategy/rollingUpdate", "value":{"maxUnavailable":"%s"}},`,
deploymentRequest.MaxSurge,
deploymentRequest.MaxUnavailable,
)
} else {
maxSurge, err := strconv.Atoi(deploymentRequest.MaxSurge)
if err != nil {
return false, err
}
maxUnavailable, err := strconv.Atoi(deploymentRequest.MaxUnavailable)
if err != nil {
return false, err
}
diff = fmt.Sprintf(
`[{"op":"replace","path":"/spec/strategy","value":{"type":"RollingUpdate"}},`+
`{"op":"replace", "path":"/spec/strategy/rollingUpdate", "value":{"maxSurge":%d}},`+
`{"op":"replace", "path":"/spec/strategy/rollingUpdate", "value":{"maxUnavailable":%d}},`,
maxSurge,
maxUnavailable,
)
}
// Enforce RollingUpdate strategy
data = strings.Replace(
data,
`[`,
diff,
-1,
)
status, err = c.PatchDeployment(
context.TODO(),
deploymentRequest.Namespace,
deploymentName,
data,
)
if !status || err != nil {
return false, err
}
}
for deploymentName := range patch {
status, err = c.FetchDeploymentStatus(context.TODO(), deploymentRequest.Namespace, deploymentName, 1000)
if !status || err != nil {
return false, err
}
}
return true, nil
}
// BlueGreenStrategy releases a new version alongside the old version then switch traffic.
func (c *Cluster) BlueGreenStrategy(_ model.DeploymentRequest) (bool, error) {
return true, nil
}
// CanaryStrategy releases a new version to a subset of users, then proceed to a full rollout.
func (c *Cluster) CanaryStrategy(_ model.DeploymentRequest) (bool, error) {
return true, nil
}
================================================
FILE: core/kubernetes/namespace.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"context"
"strings"
"github.com/clivern/beetle/core/model"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// GetNamespaces gets a list of cluster namespaces
func (c *Cluster) GetNamespaces(ctx context.Context) ([]model.Namespace, error) {
result := []model.Namespace{}
err := c.Config()
if err != nil {
return result, err
}
data, err := c.ClientSet.CoreV1().Namespaces().List(ctx, metav1.ListOptions{})
if err != nil {
return result, err
}
for _, namespace := range data.Items {
result = append(result, model.Namespace{
Name: namespace.ObjectMeta.Name,
UID: string(namespace.ObjectMeta.UID),
Status: strings.ToLower(string(namespace.Status.Phase)),
})
}
return result, nil
}
// GetNamespace gets a namespace by name
func (c *Cluster) GetNamespace(ctx context.Context, name string) (model.Namespace, error) {
result := model.Namespace{}
err := c.Config()
if err != nil {
return result, err
}
namespace, err := c.ClientSet.CoreV1().Namespaces().Get(ctx, name, metav1.GetOptions{})
if err != nil {
return result, err
}
result.Name = namespace.ObjectMeta.Name
result.UID = string(namespace.ObjectMeta.UID)
result.Status = strings.ToLower(string(namespace.Status.Phase))
return result, nil
}
================================================
FILE: core/kubernetes/namespace_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
import (
"bytes"
"context"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
"github.com/clivern/beetle/core/module"
"github.com/clivern/beetle/pkg"
"github.com/drone/envsubst"
"github.com/spf13/viper"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// TestNamespace test cases
func TestNamespace(t *testing.T) {
testingConfig := "config.testing.yml"
// LoadConfigFile
t.Run("LoadConfigFile", func(t *testing.T) {
fs := module.FileSystem{}
dir, _ := os.Getwd()
configFile := fmt.Sprintf("%s/%s", dir, testingConfig)
for {
if fs.FileExists(configFile) {
break
}
dir = filepath.Dir(dir)
configFile = fmt.Sprintf("%s/%s", dir, testingConfig)
}
t.Logf("Load Config File %s", configFile)
configUnparsed, _ := ioutil.ReadFile(configFile)
configParsed, _ := envsubst.EvalEnv(string(configUnparsed))
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer([]byte(configParsed)))
})
// TestGetNamespaces
t.Run("TestGetNamespaces", func(t *testing.T) {
cluster, err := GetCluster("production")
pkg.Expect(t, nil, err)
pkg.Expect(t, cluster.Name, "production")
pkg.Expect(t, cluster.Kubeconfig, "/app/configs/production-cluster-kubeconfig.yaml")
cluster.Override(
&v1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "default",
UID: "9d0cdf8a-dedc-11e9-bf91-42010a800167",
},
},
&v1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "beetle",
UID: "9d0cdf8a-dedc-11e9-bf91-42010a800168",
},
},
)
namespaces, err := cluster.GetNamespaces(context.TODO())
pkg.Expect(t, nil, err)
pkg.Expect(t, namespaces[1].Name, "default")
pkg.Expect(t, namespaces[1].UID, "9d0cdf8a-dedc-11e9-bf91-42010a800167")
pkg.Expect(t, namespaces[1].Status, "")
pkg.Expect(t, namespaces[0].Name, "beetle")
pkg.Expect(t, namespaces[0].UID, "9d0cdf8a-dedc-11e9-bf91-42010a800168")
pkg.Expect(t, namespaces[0].Status, "")
})
// TestGetNamespaceBeetle
t.Run("TestGetNamespaceBeetle", func(t *testing.T) {
cluster, err := GetCluster("production")
pkg.Expect(t, nil, err)
pkg.Expect(t, cluster.Name, "production")
pkg.Expect(t, cluster.Kubeconfig, "/app/configs/production-cluster-kubeconfig.yaml")
cluster.Override(
&v1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "default",
UID: "9d0cdf8a-dedc-11e9-bf91-42010a800167",
},
},
&v1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "beetle",
UID: "9d0cdf8a-dedc-11e9-bf91-42010a800168",
},
},
)
namespace, err := cluster.GetNamespace(context.TODO(), "beetle")
pkg.Expect(t, nil, err)
pkg.Expect(t, namespace.Name, "beetle")
pkg.Expect(t, namespace.UID, "9d0cdf8a-dedc-11e9-bf91-42010a800168")
pkg.Expect(t, namespace.Status, "")
})
// TestGetNamespaceDefault
t.Run("TestGetNamespaceDefault", func(t *testing.T) {
cluster, err := GetCluster("production")
pkg.Expect(t, nil, err)
pkg.Expect(t, cluster.Name, "production")
pkg.Expect(t, cluster.Kubeconfig, "/app/configs/production-cluster-kubeconfig.yaml")
cluster.Override(
&v1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "default",
UID: "9d0cdf8a-dedc-11e9-bf91-42010a800167",
},
},
&v1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "beetle",
UID: "9d0cdf8a-dedc-11e9-bf91-42010a800168",
},
},
)
namespace, err := cluster.GetNamespace(context.TODO(), "default")
pkg.Expect(t, nil, err)
pkg.Expect(t, namespace.Name, "default")
pkg.Expect(t, namespace.UID, "9d0cdf8a-dedc-11e9-bf91-42010a800167")
pkg.Expect(t, namespace.Status, "")
})
}
================================================
FILE: core/kubernetes/pod.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package kubernetes
================================================
FILE: core/middleware/auth.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package middleware
import (
"net/http"
"strings"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
"github.com/spf13/viper"
)
// Auth middleware
func Auth() gin.HandlerFunc {
return func(c *gin.Context) {
path := c.Request.URL.Path
method := c.Request.Method
if strings.Contains(path, "/api/") {
apiKey := c.GetHeader("X-API-KEY")
if viper.GetString("app.api.key") != "" && apiKey != viper.GetString("app.api.key") {
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"http_method": method,
"http_path": path,
"api_key": apiKey,
}).Info(`Unauthorized access`)
c.AbortWithStatus(http.StatusUnauthorized)
}
}
}
}
================================================
FILE: core/middleware/correlation.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package middleware
import (
"strings"
"github.com/clivern/beetle/core/util"
"github.com/gin-gonic/gin"
)
// Correlation middleware
func Correlation() gin.HandlerFunc {
return func(c *gin.Context) {
corralationID := c.Request.Header.Get("X-Correlation-ID")
if strings.TrimSpace(corralationID) == "" {
c.Request.Header.Add("X-Correlation-ID", util.GenerateUUID4())
}
c.Next()
}
}
================================================
FILE: core/middleware/log.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package middleware
import (
"bytes"
"io/ioutil"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// Logger middleware
func Logger() gin.HandlerFunc {
return func(c *gin.Context) {
// before request
var bodyBytes []byte
// Workaround for issue https://github.com/gin-gonic/gin/issues/1651
if c.Request.Body != nil {
bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
}
c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"http_method": c.Request.Method,
"http_path": c.Request.URL.Path,
"request_body": string(bodyBytes),
}).Info("Request started")
c.Next()
// after request
status := c.Writer.Status()
size := c.Writer.Size()
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
"http_status": status,
"response_size": size,
}).Info(`Request finished`)
}
}
================================================
FILE: core/middleware/metric.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package middleware
import (
"strconv"
"time"
"github.com/gin-gonic/gin"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
)
var (
httpRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: "beetle",
Name: "total_http_requests",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
}, []string{"code", "method", "handler", "host", "url"})
requestDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Subsystem: "beetle",
Name: "request_duration_seconds",
Help: "The HTTP request latencies in seconds.",
},
[]string{"code", "method", "url"},
)
responseSize = prometheus.NewSummary(
prometheus.SummaryOpts{
Namespace: "beetle",
Name: "response_size_bytes",
Help: "The HTTP response sizes in bytes.",
},
)
)
func init() {
prometheus.MustRegister(httpRequests)
prometheus.MustRegister(requestDuration)
prometheus.MustRegister(responseSize)
}
// Metric middleware
func Metric() gin.HandlerFunc {
return func(c *gin.Context) {
// before request
start := time.Now()
c.Next()
// after request
elapsed := float64(time.Since(start)) / float64(time.Second)
log.WithFields(log.Fields{
"correlation_id": c.Request.Header.Get("X-Correlation-ID"),
}).Info(`Collecting metrics`)
// Collect Metrics
httpRequests.WithLabelValues(
strconv.Itoa(c.Writer.Status()),
c.Request.Method,
c.HandlerName(),
c.Request.Host,
c.Request.URL.Path,
).Inc()
requestDuration.WithLabelValues(
strconv.Itoa(c.Writer.Status()),
c.Request.Method,
c.Request.URL.Path,
).Observe(elapsed)
responseSize.Observe(float64(c.Writer.Size()))
}
}
================================================
FILE: core/migration/schema.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package migration
import (
"encoding/json"
"time"
"github.com/jinzhu/gorm"
)
// Job struct
type Job struct {
gorm.Model
UUID string `json:"uuid"`
Payload string `json:"payload"`
Status string `json:"status"`
Type string `json:"type"`
Result string `json:"result"`
Retry int `json:"retry"`
Parent int `json:"parent"`
RunAt time.Time `json:"run_at"`
}
// LoadFromJSON update object from json
func (j *Job) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &j)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (j *Job) ConvertToJSON() (string, error) {
data, err := json.Marshal(&j)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/application.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
)
// Container struct
type Container struct {
Name string `json:"name"`
Image string `json:"image"`
Version string `json:"version"`
Deployment Deployment `json:"deployment"`
}
// Application struct
type Application struct {
ID string `json:"id"`
Name string `json:"name"`
Format string `json:"format"`
Containers []Container `json:"containers"`
}
// Applications struct
type Applications struct {
Applications []Application `json:"applications"`
}
// Deployment struct
type Deployment struct {
Name string `json:"name"`
UID string `json:"uid"`
}
// LoadFromJSON update object from json
func (c *Application) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &c)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (c *Application) ConvertToJSON() (string, error) {
data, err := json.Marshal(&c)
if err != nil {
return "", err
}
return string(data), nil
}
// LoadFromJSON update object from json
func (c *Applications) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &c)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (c *Applications) ConvertToJSON() (string, error) {
data, err := json.Marshal(&c)
if err != nil {
return "", err
}
return string(data), nil
}
// LoadFromJSON update object from json
func (d *Deployment) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &d)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (d *Deployment) ConvertToJSON() (string, error) {
data, err := json.Marshal(&d)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/cluster.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
)
// Cluster struct
type Cluster struct {
Name string `json:"name"`
Health bool `json:"health"`
}
// Clusters struct
type Clusters struct {
Clusters []Cluster `json:"clusters"`
}
// LoadFromJSON update object from json
func (c *Cluster) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &c)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (c *Cluster) ConvertToJSON() (string, error) {
data, err := json.Marshal(&c)
if err != nil {
return "", err
}
return string(data), nil
}
// LoadFromJSON update object from json
func (c *Clusters) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &c)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (c *Clusters) ConvertToJSON() (string, error) {
data, err := json.Marshal(&c)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/configmap.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
)
// ConfigMap struct
type ConfigMap struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
UID string `json:"uid"`
CreationTimestamp string `json:"creation_timestamp"`
Data map[string]string `json:"data"`
Labels map[string]string `json:"labels"`
}
// LoadFromJSON update object from json
func (d *ConfigMap) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &d)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (d *ConfigMap) ConvertToJSON() (string, error) {
data, err := json.Marshal(&d)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/configs.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
// App struct
type App struct {
ID string
Name string
ImageFormat string
}
// Configs struct
type Configs struct {
Exists bool
Version string
Applications []App
}
================================================
FILE: core/model/dsn.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
"fmt"
)
// DSN struct
type DSN struct {
Driver string `json:"driver"`
Username string `json:"username"`
Password string `json:"password"`
Hostname string `json:"hostname"`
Port int `json:"port"`
Name string `json:"name"`
}
// ToString gets the dsn string
func (d *DSN) ToString() string {
if d.Driver == "mysql" {
return fmt.Sprintf(
"%s:%s@tcp(%s:%d)/%s?charset=utf8&parseTime=True",
d.Username,
d.Password,
d.Hostname,
d.Port,
d.Name,
)
}
// sqlite3 by default
return d.Name
}
// LoadFromJSON update object from json
func (d *DSN) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &d)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (d *DSN) ConvertToJSON() (string, error) {
data, err := json.Marshal(&d)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/dsn_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"testing"
"github.com/clivern/beetle/pkg"
)
// TestDsnToString test cases
func TestDsnToString(t *testing.T) {
t.Run("TestDsnToStringForMySQL", func(t *testing.T) {
dsn := DSN{
Driver: "mysql",
Username: "root",
Password: "root",
Hostname: "127.0.0.1",
Port: 3306,
Name: "beetle",
}
pkg.Expect(t, "root:root@tcp(127.0.0.1:3306)/beetle?charset=utf8&parseTime=True", dsn.ToString())
})
t.Run("TestDsnToStringForSQLLite", func(t *testing.T) {
dsn := DSN{
Driver: "sqlite3",
Name: "/path/to/beetle.db",
}
pkg.Expect(t, "/path/to/beetle.db", dsn.ToString())
})
}
================================================
FILE: core/model/job.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
"time"
)
var (
// JobPending pending job type
JobPending = "pending"
// JobFailed failed job type
JobFailed = "failed"
// JobSuccess success job type
JobSuccess = "success"
// JobOnHold on hold job type
JobOnHold = "on_hold"
// JobDeploymentUpdate deployment update
JobDeploymentUpdate = "deployment.update"
// JobDeploymentNotify deployment notify
JobDeploymentNotify = "deployment.notify"
)
// Job struct
type Job struct {
ID int `json:"id"`
UUID string `json:"uuid"`
Payload string `json:"payload"`
Status string `json:"status"`
Type string `json:"type"`
Result string `json:"result"`
Retry int `json:"retry"`
Parent int `json:"parent"`
RunAt *time.Time `json:"run_at"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// Jobs struct
type Jobs struct {
Jobs []Job `json:"jobs"`
}
// LoadFromJSON update object from json
func (j *Job) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &j)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (j *Job) ConvertToJSON() (string, error) {
data, err := json.Marshal(&j)
if err != nil {
return "", err
}
return string(data), nil
}
// LoadFromJSON update object from json
func (j *Jobs) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &j)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (j *Jobs) ConvertToJSON() (string, error) {
data, err := json.Marshal(&j)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/message.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
)
// Message struct
type Message struct {
UUID string `json:"uuid"`
Job int `json:"job"`
}
// LoadFromJSON update object from json
func (m *Message) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &m)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (m *Message) ConvertToJSON() (string, error) {
data, err := json.Marshal(&m)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/metric.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
"strconv"
"github.com/prometheus/client_golang/prometheus"
)
const (
// COUNTER is a Prometheus COUNTER metric
COUNTER string = "counter"
// GAUGE is a Prometheus GAUGE metric
GAUGE string = "gauge"
// HISTOGRAM is a Prometheus HISTOGRAM metric
HISTOGRAM string = "histogram"
// SUMMARY is a Prometheus SUMMARY metric
SUMMARY string = "summary"
)
// Metric struct
type Metric struct {
Type string `json:"type"`
Name string `json:"name"`
Help string `json:"help"`
Method string `json:"method"`
Value string `json:"value"`
Labels prometheus.Labels `json:"labels"`
Buckets []float64 `json:"buckets"`
}
// LoadFromJSON update object from json
func (m *Metric) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &m)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (m *Metric) ConvertToJSON() (string, error) {
data, err := json.Marshal(&m)
if err != nil {
return "", err
}
return string(data), nil
}
// LabelKeys gets a list of label keys
func (m *Metric) LabelKeys() []string {
keys := []string{}
for k := range m.Labels {
keys = append(keys, k)
}
return keys
}
// LabelValues gets a list of label values
func (m *Metric) LabelValues() []string {
values := []string{}
for _, v := range m.Labels {
values = append(values, v)
}
return values
}
// GetValueAsFloat gets a list of label values
func (m *Metric) GetValueAsFloat() (float64, error) {
value, err := strconv.ParseFloat(m.Value, 64)
if err != nil {
return 0, nil
}
return value, nil
}
================================================
FILE: core/model/migration.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
"time"
)
// Migration struct
type Migration struct {
ID int `json:"id"`
Flag string `json:"file"`
RunAt time.Time `json:"run_at"`
}
// LoadFromJSON update object from json
func (m *Migration) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &m)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (m *Migration) ConvertToJSON() (string, error) {
data, err := json.Marshal(&m)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/namespace.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
)
// Namespace struct
type Namespace struct {
Name string `json:"name"`
UID string `json:"uid"`
Status string `json:"status"`
}
// Namespaces struct
type Namespaces struct {
Namespaces []Namespace `json:"namespaces"`
}
// LoadFromJSON update object from json
func (d *Namespace) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &d)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (d *Namespace) ConvertToJSON() (string, error) {
data, err := json.Marshal(&d)
if err != nil {
return "", err
}
return string(data), nil
}
// LoadFromJSON update object from json
func (d *Namespaces) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &d)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (d *Namespaces) ConvertToJSON() (string, error) {
data, err := json.Marshal(&d)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/model/patch.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
// PatchStringValue specifies a patch operation for a string.
type PatchStringValue struct {
Op string `json:"op"`
Path string `json:"path"`
Value string `json:"value"`
}
// PatchUInt32Value specifies a patch operation for a uint32.
type PatchUInt32Value struct {
Op string `json:"op"`
Path string `json:"path"`
Value uint32 `json:"value"`
}
================================================
FILE: core/model/request.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package model
import (
"encoding/json"
"fmt"
"reflect"
)
var (
// RecreateStrategy var
RecreateStrategy = "recreate"
// RampedStrategy var
RampedStrategy = "ramped"
// CanaryStrategy var
CanaryStrategy = "canary"
// BlueGreenStrategy var
BlueGreenStrategy = "blue_green"
)
// DeploymentRequest struct
type DeploymentRequest struct {
Cluster string `json:"cluster"`
Namespace string `json:"namespace"`
Application string `json:"application"`
Version string `json:"version"`
Strategy string `json:"strategy"`
Status string `json:"status"`
// Ramped Strategy
MaxSurge string `json:"maxSurge"`
MaxUnavailable string `json:"maxUnavailable"`
}
// LoadFromJSON update object from json
func (d *DeploymentRequest) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &d)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (d *DeploymentRequest) ConvertToJSON() (string, error) {
data, err := json.Marshal(&d)
if err != nil {
return "", err
}
return string(data), nil
}
// Validate validates the request
func (d *DeploymentRequest) Validate(strategies []string) error {
if d.Version == "" {
return fmt.Errorf(
"Error! version is required",
)
}
if !In(d.Strategy, strategies) {
return fmt.Errorf(
"Error! strategy %s is invalid",
d.Strategy,
)
}
return nil
}
// In check if value is on array
func In(val interface{}, array interface{}) bool {
switch reflect.TypeOf(array).Kind() {
case reflect.Slice:
s := reflect.ValueOf(array)
for i := 0; i < s.Len(); i++ {
if reflect.DeepEqual(val, s.Index(i).Interface()) {
return true
}
}
}
return false
}
================================================
FILE: core/module/database.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"fmt"
"time"
"github.com/clivern/beetle/core/migration"
"github.com/clivern/beetle/core/model"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql"
_ "github.com/jinzhu/gorm/dialects/sqlite"
log "github.com/sirupsen/logrus"
"github.com/spf13/viper"
)
// Database struct
type Database struct {
Connection *gorm.DB
}
// Connect connects to a MySQL database
func (db *Database) Connect(dsn model.DSN) error {
var err error
// Reuse db connections http://go-database-sql.org/surprises.html
if db.Ping() == nil {
return nil
}
db.Connection, err = gorm.Open(dsn.Driver, dsn.ToString())
if err != nil {
return err
}
return nil
}
// Ping check the db connection
func (db *Database) Ping() error {
if db.Connection == nil {
return fmt.Errorf("No DB Connections Found")
}
err := db.Connection.DB().Ping()
if err != nil {
return err
}
// Cleanup stale connections http://go-database-sql.org/surprises.html
db.Connection.DB().SetMaxOpenConns(5)
db.Connection.DB().SetConnMaxLifetime(time.Duration(10) * time.Second)
dbStats := db.Connection.DB().Stats()
log.WithFields(log.Fields{
"dbStats.maxOpenConnections": int(dbStats.MaxOpenConnections),
"dbStats.openConnections": int(dbStats.OpenConnections),
"dbStats.inUse": int(dbStats.InUse),
"dbStats.idle": int(dbStats.Idle),
}).Debug(`Open DB Connection`)
return nil
}
// AutoConnect connects to a MySQL database using loaded configs
func (db *Database) AutoConnect() error {
var err error
// Reuse db connections http://go-database-sql.org/surprises.html
if db.Ping() == nil {
return nil
}
dsn := model.DSN{
Driver: viper.GetString("app.database.driver"),
Username: viper.GetString("app.database.username"),
Password: viper.GetString("app.database.password"),
Hostname: viper.GetString("app.database.host"),
Port: viper.GetInt("app.database.port"),
Name: viper.GetString("app.database.name"),
}
db.Connection, err = gorm.Open(dsn.Driver, dsn.ToString())
if err != nil {
return err
}
return nil
}
// Migrate migrates the database
func (db *Database) Migrate() bool {
status := true
db.Connection.AutoMigrate(&migration.Job{})
status = status && db.Connection.HasTable(&migration.Job{})
return status
}
// Rollback drop tables
func (db *Database) Rollback() bool {
status := true
db.Connection.DropTableIfExists(&migration.Job{})
status = status && !db.Connection.HasTable(&migration.Job{})
return status
}
// HasTable checks if table exists
func (db *Database) HasTable(table string) bool {
return db.Connection.HasTable(table)
}
// CreateJob creates a new job
func (db *Database) CreateJob(job *model.Job) *model.Job {
db.Connection.Create(job)
return job
}
// JobExistByID check if job exists
func (db *Database) JobExistByID(id int) bool {
job := model.Job{}
db.Connection.Where("id = ?", id).First(&job)
return job.ID > 0
}
// GetJobByID gets a job by id
func (db *Database) GetJobByID(id int) model.Job {
job := model.Job{}
db.Connection.Where("id = ?", id).First(&job)
return job
}
// GetJobs gets jobs
func (db *Database) GetJobs() []model.Job {
jobs := []model.Job{}
db.Connection.Select("*").Find(&jobs)
return jobs
}
// JobExistByUUID check if job exists
func (db *Database) JobExistByUUID(uuid string) bool {
job := model.Job{}
db.Connection.Where("uuid = ?", uuid).First(&job)
return job.ID > 0
}
// GetJobByUUID gets a job by uuid
func (db *Database) GetJobByUUID(uuid string) model.Job {
job := model.Job{}
db.Connection.Where("uuid = ?", uuid).First(&job)
return job
}
// GetPendingJobByType gets a job by uuid
func (db *Database) GetPendingJobByType(jobType string) model.Job {
job := model.Job{}
db.Connection.Where("status = ? AND type = ?", model.JobPending, jobType).First(&job)
return job
}
// CountJobs count jobs by status
func (db *Database) CountJobs(status string) int {
count := 0
db.Connection.Model(&model.Job{}).Where("status = ?", status).Count(&count)
return count
}
// DeleteJobByID deletes a job by id
func (db *Database) DeleteJobByID(id int) {
db.Connection.Unscoped().Where("id=?", id).Delete(&migration.Job{})
}
// DeleteJobByUUID deletes a job by uuid
func (db *Database) DeleteJobByUUID(uuid string) {
db.Connection.Unscoped().Where("uuid=?", uuid).Delete(&migration.Job{})
}
// UpdateJobByID updates a job by ID
func (db *Database) UpdateJobByID(job *model.Job) {
db.Connection.Save(&job)
}
// Close closes MySQL database connection
func (db *Database) Close() error {
return db.Connection.Close()
}
// ReleaseChildJobs count jobs by status
func (db *Database) ReleaseChildJobs(parentID int) {
db.Connection.Model(&model.Job{}).Where(
"parent = ? AND status = ?",
parentID,
model.JobOnHold,
).Update("status", model.JobPending)
}
================================================
FILE: core/module/database_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"bytes"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/pkg"
"github.com/drone/envsubst"
"github.com/spf13/viper"
)
var testingConfig = "config.testing.yml"
// TestDatabase test cases
func TestDatabase(t *testing.T) {
// LoadConfigFile
t.Run("LoadConfigFile", func(t *testing.T) {
fs := FileSystem{}
dir, _ := os.Getwd()
configFile := fmt.Sprintf("%s/%s", dir, testingConfig)
for {
if fs.FileExists(configFile) {
break
}
dir = filepath.Dir(dir)
configFile = fmt.Sprintf("%s/%s", dir, testingConfig)
}
t.Logf("Load Config File %s", configFile)
configUnparsed, _ := ioutil.ReadFile(configFile)
configParsed, _ := envsubst.EvalEnv(string(configUnparsed))
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer([]byte(configParsed)))
})
// TestDatabaseConnection
t.Run("TestDatabaseConnection", func(t *testing.T) {
db := Database{}
err := db.Connect(model.DSN{
Driver: viper.GetString("app.database.driver"),
Username: viper.GetString("app.database.username"),
Password: viper.GetString("app.database.password"),
Hostname: viper.GetString("app.database.host"),
Port: viper.GetInt("app.database.port"),
Name: viper.GetString("app.database.name"),
})
pkg.Expect(t, nil, err)
defer db.Close()
pkg.Expect(t, true, db.Rollback())
pkg.Expect(t, true, db.Migrate())
pkg.Expect(t, true, db.HasTable("jobs"))
})
// TestJobCRUD
t.Run("TestJobCRUD", func(t *testing.T) {
db := Database{}
err := db.Connect(model.DSN{
Driver: viper.GetString("app.database.driver"),
Username: viper.GetString("app.database.username"),
Password: viper.GetString("app.database.password"),
Hostname: viper.GetString("app.database.host"),
Port: viper.GetInt("app.database.port"),
Name: viper.GetString("app.database.name"),
})
pkg.Expect(t, nil, err)
defer db.Close()
pkg.Expect(t, true, db.Rollback())
pkg.Expect(t, true, db.Migrate())
pkg.Expect(t, true, db.HasTable("jobs"))
// Delete the job if it exists
db.DeleteJobByID(1)
db.DeleteJobByUUID("dddde755-5f99-4e51-a517-77878986a07e")
// Create the job
job := db.CreateJob(&model.Job{
UUID: "dddde755-5f99-4e51-a517-77878986a07e",
Parent: 0,
})
pkg.Expect(t, 1, job.ID)
pkg.Expect(t, "dddde755-5f99-4e51-a517-77878986a07e", job.UUID)
job1 := db.GetJobByID(1)
job2 := db.GetJobByUUID("dddde755-5f99-4e51-a517-77878986a07e")
pkg.Expect(t, job1.ID, job2.ID)
pkg.Expect(t, job1.UUID, job2.UUID)
job1.UUID = "dddde755-5f99-4e51-a517-77878986a07n"
db.UpdateJobByID(&job1)
job3 := db.GetJobByID(1)
pkg.Expect(t, "dddde755-5f99-4e51-a517-77878986a07n", job3.UUID)
pkg.Expect(t, job1.UUID, job3.UUID)
pkg.Expect(t, 0, db.GetJobByUUID("dddde755-5f99-4e51-a517-77878986a07ek").ID)
pkg.Expect(t, false, db.JobExistByUUID("dddde755-5f99-4e51-a517-77878986a07eo"))
pkg.Expect(t, false, db.JobExistByID(20))
})
}
================================================
FILE: core/module/file_system.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"os"
)
// FileSystem struct
type FileSystem struct{}
// PathExists reports whether the path exists
func (fs *FileSystem) PathExists(path string) bool {
if _, err := os.Stat(path); os.IsNotExist(err) {
return false
}
return true
}
// FileExists reports whether the named file exists
func (fs *FileSystem) FileExists(path string) bool {
if fi, err := os.Stat(path); err == nil {
if fi.Mode().IsRegular() {
return true
}
}
return false
}
// DirExists reports whether the dir exists
func (fs *FileSystem) DirExists(path string) bool {
if fi, err := os.Stat(path); err == nil {
if fi.Mode().IsDir() {
return true
}
}
return false
}
// EnsureDir ensures that directory exists
func (fs *FileSystem) EnsureDir(dirName string, mode int) (bool, error) {
err := os.MkdirAll(dirName, os.FileMode(mode))
if err == nil || os.IsExist(err) {
return true, nil
}
return false, err
}
================================================
FILE: core/module/http.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"bytes"
"context"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"strings"
"time"
)
// HTTPClient struct
type HTTPClient struct {
Timeout time.Duration
}
// NewHTTPClient creates an instance of http client
func NewHTTPClient(timeout int) *HTTPClient {
return &HTTPClient{
Timeout: time.Duration(timeout),
}
}
// Get http call
func (h *HTTPClient) Get(ctx context.Context, endpoint string, parameters, headers map[string]string) (*http.Response, error) {
endpoint, err := h.buildParameters(endpoint, parameters)
if err != nil {
return nil, err
}
req, _ := http.NewRequest("GET", endpoint, nil)
req = req.WithContext(ctx)
for k, v := range headers {
req.Header.Add(k, v)
}
client := http.Client{
Timeout: time.Second * h.Timeout,
}
resp, err := client.Do(req)
if err != nil {
return resp, err
}
return resp, err
}
// Post http call
func (h *HTTPClient) Post(ctx context.Context, endpoint string, data string, parameters, headers map[string]string) (*http.Response, error) {
endpoint, err := h.buildParameters(endpoint, parameters)
if err != nil {
return nil, err
}
req, _ := http.NewRequest("POST", endpoint, bytes.NewBufferString(data))
req = req.WithContext(ctx)
for k, v := range headers {
req.Header.Add(k, v)
}
client := http.Client{
Timeout: time.Second * h.Timeout,
}
resp, err := client.Do(req)
if err != nil {
return resp, err
}
return resp, err
}
// Put http call
func (h *HTTPClient) Put(ctx context.Context, endpoint string, data string, parameters, headers map[string]string) (*http.Response, error) {
endpoint, err := h.buildParameters(endpoint, parameters)
if err != nil {
return nil, err
}
req, _ := http.NewRequest("PUT", endpoint, bytes.NewBufferString(data))
req = req.WithContext(ctx)
for k, v := range headers {
req.Header.Add(k, v)
}
client := http.Client{
Timeout: time.Second * h.Timeout,
}
resp, err := client.Do(req)
if err != nil {
return resp, err
}
return resp, err
}
// Patch http call
func (h *HTTPClient) Patch(ctx context.Context, endpoint string, data string, parameters, headers map[string]string) (*http.Response, error) {
endpoint, err := h.buildParameters(endpoint, parameters)
if err != nil {
return nil, err
}
req, _ := http.NewRequest("PATCH", endpoint, bytes.NewBufferString(data))
req = req.WithContext(ctx)
for k, v := range headers {
req.Header.Add(k, v)
}
client := http.Client{
Timeout: time.Second * h.Timeout,
}
resp, err := client.Do(req)
if err != nil {
return resp, err
}
return resp, err
}
// Delete http call
func (h *HTTPClient) Delete(ctx context.Context, endpoint string, parameters, headers map[string]string) (*http.Response, error) {
endpoint, err := h.buildParameters(endpoint, parameters)
if err != nil {
return nil, err
}
req, _ := http.NewRequest("DELETE", endpoint, nil)
req = req.WithContext(ctx)
for k, v := range headers {
req.Header.Add(k, v)
}
client := http.Client{
Timeout: time.Second * h.Timeout,
}
resp, err := client.Do(req)
if err != nil {
return resp, err
}
return resp, err
}
// buildParameters add parameters to URL
func (h *HTTPClient) buildParameters(endpoint string, parameters map[string]string) (string, error) {
u, err := url.Parse(endpoint)
if err != nil {
return "", err
}
q := u.Query()
for k, v := range parameters {
q.Set(k, v)
}
u.RawQuery = q.Encode()
return u.String(), nil
}
// BuildData build body data
func (h *HTTPClient) BuildData(parameters map[string]string) string {
var items []string
for k, v := range parameters {
items = append(items, fmt.Sprintf("%s=%s", k, v))
}
return strings.Join(items, "&")
}
// ToString response body to string
func (h *HTTPClient) ToString(response *http.Response) (string, error) {
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
return "", err
}
return string(body), nil
}
// GetStatusCode response status code
func (h *HTTPClient) GetStatusCode(response *http.Response) int {
return response.StatusCode
}
// GetHeaderValue get response header value
func (h *HTTPClient) GetHeaderValue(response *http.Response, key string) string {
return response.Header.Get(key)
}
================================================
FILE: core/module/http_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"context"
"net/http"
"strings"
"testing"
"github.com/clivern/beetle/pkg"
)
// TestHttpGet test cases
func TestHttpGet(t *testing.T) {
t.Run("TestHttpGet", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Get(
context.TODO(),
"https://httpbin.org/get",
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusOK, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, true, strings.Contains(body, "value1"))
pkg.Expect(t, true, strings.Contains(body, "arg1"))
pkg.Expect(t, true, strings.Contains(body, "arg1=value1"))
pkg.Expect(t, true, strings.Contains(body, "X-Api-Key"))
pkg.Expect(t, true, strings.Contains(body, "hipp-123"))
pkg.Expect(t, nil, error)
})
}
// TestHttpDelete test cases
func TestHttpDelete(t *testing.T) {
t.Run("TestHttpDelete", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Delete(
context.TODO(),
"https://httpbin.org/delete",
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusOK, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, true, strings.Contains(body, "value1"))
pkg.Expect(t, true, strings.Contains(body, "arg1"))
pkg.Expect(t, true, strings.Contains(body, "arg1=value1"))
pkg.Expect(t, true, strings.Contains(body, "X-Api-Key"))
pkg.Expect(t, true, strings.Contains(body, "hipp-123"))
pkg.Expect(t, nil, error)
})
}
// TestHttpPost test cases
func TestHttpPost(t *testing.T) {
t.Run("TestHttpPost", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Post(
context.TODO(),
"https://httpbin.org/post",
`{"Username":"admin", "Password":"12345"}`,
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusOK, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, true, strings.Contains(body, `"12345"`))
pkg.Expect(t, true, strings.Contains(body, `"Username"`))
pkg.Expect(t, true, strings.Contains(body, `"admin"`))
pkg.Expect(t, true, strings.Contains(body, `"Password"`))
pkg.Expect(t, true, strings.Contains(body, "value1"))
pkg.Expect(t, true, strings.Contains(body, "arg1"))
pkg.Expect(t, true, strings.Contains(body, "arg1=value1"))
pkg.Expect(t, true, strings.Contains(body, "X-Api-Key"))
pkg.Expect(t, true, strings.Contains(body, "hipp-123"))
pkg.Expect(t, nil, error)
})
}
// TestHttpPut test cases
func TestHttpPut(t *testing.T) {
t.Run("TestHttpPut", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Put(
context.TODO(),
"https://httpbin.org/put",
`{"Username":"admin", "Password":"12345"}`,
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusOK, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, true, strings.Contains(body, `"12345"`))
pkg.Expect(t, true, strings.Contains(body, `"Username"`))
pkg.Expect(t, true, strings.Contains(body, `"admin"`))
pkg.Expect(t, true, strings.Contains(body, `"Password"`))
pkg.Expect(t, true, strings.Contains(body, "value1"))
pkg.Expect(t, true, strings.Contains(body, "arg1"))
pkg.Expect(t, true, strings.Contains(body, "arg1=value1"))
pkg.Expect(t, true, strings.Contains(body, "X-Api-Key"))
pkg.Expect(t, true, strings.Contains(body, "hipp-123"))
pkg.Expect(t, nil, error)
})
}
// TestHttpGetStatusCode1 test cases
func TestHttpGetStatusCode1(t *testing.T) {
t.Run("TestHttpGetStatusCode1", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Get(
context.TODO(),
"https://httpbin.org/status/200",
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusOK, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, "", body)
pkg.Expect(t, nil, error)
})
}
// TestHttpGetStatusCode2 test cases
func TestHttpGetStatusCode2(t *testing.T) {
t.Run("TestHttpGetStatusCode2", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Get(
context.TODO(),
"https://httpbin.org/status/500",
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusInternalServerError, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, "", body)
pkg.Expect(t, nil, error)
})
}
// TestHttpGetStatusCode3 test cases
func TestHttpGetStatusCode3(t *testing.T) {
t.Run("TestHttpGetStatusCode3", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Get(
context.TODO(),
"https://httpbin.org/status/404",
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusNotFound, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, "", body)
pkg.Expect(t, nil, error)
})
}
// TestHttpGetStatusCode4 test cases
func TestHttpGetStatusCode4(t *testing.T) {
t.Run("TestHttpGetStatusCode4", func(t *testing.T) {
httpClient := NewHTTPClient(20)
response, error := httpClient.Get(
context.TODO(),
"https://httpbin.org/status/201",
map[string]string{"arg1": "value1"},
map[string]string{"X-Api-Key": "hipp-123"},
)
pkg.Expect(t, http.StatusCreated, httpClient.GetStatusCode(response))
pkg.Expect(t, nil, error)
body, error := httpClient.ToString(response)
pkg.Expect(t, "", body)
pkg.Expect(t, nil, error)
})
}
// TestBuildParameters test cases
func TestBuildParameters(t *testing.T) {
t.Run("TestBuildParameters", func(t *testing.T) {
httpClient := NewHTTPClient(20)
url, error := httpClient.buildParameters("http://127.0.0.1", map[string]string{"arg1": "value1"})
pkg.Expect(t, "http://127.0.0.1?arg1=value1", url)
pkg.Expect(t, nil, error)
})
}
// TestBuildData test cases
func TestBuildData(t *testing.T) {
t.Run("TestBuildData", func(t *testing.T) {
httpClient := NewHTTPClient(20)
pkg.Expect(t, httpClient.BuildData(map[string]string{}), "")
pkg.Expect(t, httpClient.BuildData(map[string]string{"arg1": "value1"}), "arg1=value1")
})
}
================================================
FILE: core/module/prometheus.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"fmt"
"github.com/clivern/beetle/core/model"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
)
// Prometheus struct
type Prometheus struct{}
// NewPrometheus create a new instance of prometheus backend
func NewPrometheus() *Prometheus {
return &Prometheus{}
}
// Send sends metrics to prometheus
func (p *Prometheus) Send(metrics []model.Metric) error {
log.Info(fmt.Sprintf(
"Send %d metrics to prometheus backend",
len(metrics),
))
for _, metric := range metrics {
switch metric.Type {
case model.COUNTER:
p.Counter(metric)
case model.GAUGE:
p.Gauge(metric)
case model.HISTOGRAM:
p.Histogram(metric)
case model.SUMMARY:
p.Summary(metric)
default:
return fmt.Errorf("metric with type %s not implemented yet", metric.Type)
}
}
return nil
}
// Summary updates or creates a summary
func (p *Prometheus) Summary(item model.Metric) error {
var metric prometheus.Summary
value, _ := item.GetValueAsFloat()
opts := prometheus.SummaryOpts{
Name: item.Name,
Help: item.Help,
}
if len(item.Labels) > 0 {
vec := prometheus.NewSummaryVec(opts, item.LabelKeys())
err := prometheus.Register(vec)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
vec = are.ExistingCollector.(*prometheus.SummaryVec)
} else {
return err
}
}
metric = vec.With(item.Labels).(prometheus.Summary)
} else {
metric = prometheus.NewSummary(opts)
err := prometheus.Register(metric)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
metric = are.ExistingCollector.(prometheus.Summary)
} else {
return err
}
}
}
if item.Method == "observe" {
metric.Observe(value)
} else {
return fmt.Errorf("method %s is not implemented yet", item.Method)
}
return nil
}
// Counter updates or creates a counter
func (p *Prometheus) Counter(item model.Metric) error {
var metric prometheus.Counter
value, _ := item.GetValueAsFloat()
opts := prometheus.CounterOpts{
Name: item.Name,
Help: item.Help,
}
if len(item.Labels) > 0 {
vec := prometheus.NewCounterVec(opts, item.LabelKeys())
err := prometheus.Register(vec)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
vec = are.ExistingCollector.(*prometheus.CounterVec)
} else {
return err
}
}
metric = vec.With(item.Labels)
} else {
metric = prometheus.NewCounter(opts)
err := prometheus.Register(metric)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
metric = are.ExistingCollector.(prometheus.Counter)
} else {
return err
}
}
}
switch item.Method {
case "inc":
metric.Inc()
case "add":
metric.Add(value)
default:
return fmt.Errorf("method %s is not implemented yet", item.Method)
}
return nil
}
// Histogram updates or creates a histogram
func (p *Prometheus) Histogram(item model.Metric) error {
var metric prometheus.Histogram
value, _ := item.GetValueAsFloat()
opts := prometheus.HistogramOpts{
Name: item.Name,
Help: item.Help,
Buckets: item.Buckets,
}
if len(item.Labels) > 0 {
vec := prometheus.NewHistogramVec(opts, item.LabelKeys())
err := prometheus.Register(vec)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
vec = are.ExistingCollector.(*prometheus.HistogramVec)
} else {
return err
}
}
metric = vec.With(item.Labels).(prometheus.Histogram)
} else {
metric = prometheus.NewHistogram(opts)
err := prometheus.Register(metric)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
metric = are.ExistingCollector.(prometheus.Histogram)
} else {
return err
}
}
}
if item.Method == "observe" {
metric.Observe(value)
} else {
return fmt.Errorf("method %s is not implemented yet", item.Method)
}
return nil
}
// Gauge updates or creates a gauge
func (p *Prometheus) Gauge(item model.Metric) error {
var metric prometheus.Gauge
value, _ := item.GetValueAsFloat()
opts := prometheus.GaugeOpts{
Name: item.Name,
Help: item.Help,
}
if len(item.Labels) > 0 {
vec := prometheus.NewGaugeVec(opts, item.LabelKeys())
err := prometheus.Register(vec)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
vec = are.ExistingCollector.(*prometheus.GaugeVec)
} else {
return err
}
}
metric = vec.With(item.Labels)
} else {
metric = prometheus.NewGauge(opts)
err := prometheus.Register(metric)
if err != nil {
if are, ok := err.(prometheus.AlreadyRegisteredError); ok {
metric = are.ExistingCollector.(prometheus.Gauge)
} else {
return err
}
}
}
switch item.Method {
case "set":
metric.Set(value)
case "inc":
metric.Inc()
case "dec":
metric.Dec()
case "add":
metric.Add(value)
case "sub":
metric.Sub(value)
default:
return fmt.Errorf("method %s is not implemented yet", item.Method)
}
return nil
}
================================================
FILE: core/module/remote.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"context"
"encoding/json"
"fmt"
"net/http"
)
// ReleaseURL remote release URL
const ReleaseURL = "https://api.github.com/repos/Clivern/Beetle/releases/latest"
// LatestRelease struct
type LatestRelease struct {
Name string `json:"name"`
TagName string `json:"tag_name"`
}
// LoadFromJSON update object from json
func (lr *LatestRelease) LoadFromJSON(data []byte) (bool, error) {
err := json.Unmarshal(data, &lr)
if err != nil {
return false, err
}
return true, nil
}
// ConvertToJSON convert object to json
func (lr *LatestRelease) ConvertToJSON() (string, error) {
data, err := json.Marshal(&lr)
if err != nil {
return "", err
}
return string(data), nil
}
// GetLatestRelease gets the latest beetle release
func GetLatestRelease() (LatestRelease, error) {
result := LatestRelease{}
httpClient := NewHTTPClient(20)
response, err := httpClient.Get(
context.TODO(),
ReleaseURL,
map[string]string{},
map[string]string{},
)
if http.StatusOK != httpClient.GetStatusCode(response) || err != nil {
return result, fmt.Errorf("Error: Unable to fetch latest release")
}
body, err := httpClient.ToString(response)
if err != nil {
return result, fmt.Errorf("Error: Unable to fetch latest release")
}
ok, err := result.LoadFromJSON([]byte(body))
if !ok || err != nil {
return result, fmt.Errorf("Error: Invalid remote response")
}
return result, nil
}
================================================
FILE: core/module/remote_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package module
import (
"strings"
"testing"
"github.com/clivern/beetle/pkg"
)
// TestRemote test cases
func TestRemote(t *testing.T) {
t.Run("TestRemote", func(t *testing.T) {
result, err := GetLatestRelease()
pkg.Expect(t, true, strings.Contains(result.Name, "."))
pkg.Expect(t, true, strings.Contains(result.TagName, "."))
pkg.Expect(t, nil, err)
})
}
================================================
FILE: core/util/helpers.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package util
import (
"encoding/json"
"io/ioutil"
"os"
"path/filepath"
"reflect"
"strings"
"github.com/satori/go.uuid"
)
// InArray check if value is on array
func InArray(val interface{}, array interface{}) bool {
switch reflect.TypeOf(array).Kind() {
case reflect.Slice:
s := reflect.ValueOf(array)
for i := 0; i < s.Len(); i++ {
if reflect.DeepEqual(val, s.Index(i).Interface()) {
return true
}
}
}
return false
}
// GenerateUUID4 create a UUID
func GenerateUUID4() string {
u := uuid.Must(uuid.NewV4(), nil)
return u.String()
}
// ListFiles lists all files inside a dir
func ListFiles(basePath string) []string {
var files []string
err := filepath.Walk(basePath, func(path string, info os.FileInfo, err error) error {
if basePath != path && !info.IsDir() {
files = append(files, path)
}
return nil
})
if err != nil {
return files
}
return files
}
// ReadFile get the file content
func ReadFile(path string) string {
data, err := ioutil.ReadFile(path)
if err != nil {
return err.Error()
}
return string(data)
}
// FilterFiles filters files list based on specific sub-strings
func FilterFiles(files, filters []string) []string {
var filteredFiles []string
for _, file := range files {
ok := true
for _, filter := range filters {
ok = ok && strings.Contains(file, filter)
}
if ok {
filteredFiles = append(filteredFiles, file)
}
}
return filteredFiles
}
// Unset remove element at position i
func Unset(a []string, i int) []string {
a[i] = a[len(a)-1]
a[len(a)-1] = ""
return a[:len(a)-1]
}
// ConvertToJSON convert object to json
func ConvertToJSON(val interface{}) (string, error) {
data, err := json.Marshal(val)
if err != nil {
return "", err
}
return string(data), nil
}
================================================
FILE: core/util/helpers_test.go
================================================
// Copyright 2020 Clivern. All rights reserved.
// Use of this source code is governed by the MIT
// license that can be found in the LICENSE file.
package util
import (
"testing"
"github.com/clivern/beetle/core/model"
"github.com/clivern/beetle/pkg"
)
// TestInArray test cases
func TestInArray(t *testing.T) {
// TestInArray
t.Run("TestInArray", func(t *testing.T) {
pkg.Expect(t, InArray("A", []string{"A", "B", "C", "D"}), true)
pkg.Expect(t, InArray("B", []string{"A", "B", "C", "D"}), true)
pkg.Expect(t, InArray("H", []string{"A", "B", "C", "D"}), false)
pkg.Expect(t, InArray(1, []int{2, 3, 1}), true)
pkg.Expect(t, InArray(9, []int{2, 3, 1}), false)
payload := []model.PatchStringValue{
model.PatchStringValue{
Op: "replace",
Path: "/spec/template/spec/containers/0/image",
Value: "clivern/toad:release-0.2.4",
},
}
data, err := ConvertToJSON(payload)
pkg.Expect(t, data, `[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"clivern/toad:release-0.2.4"}]`)
pkg.Expect(t, err, nil)
})
}
================================================
FILE: deployment/docker/README.md
================================================
⚠️ This only to test beetle with prometheus and grafana.
================================================
FILE: deployment/docker/docker-compose.yml
================================================
version: '3'
services:
# Redis Service
redis:
image: 'redis:7.2-alpine'
volumes:
- 'redis_data:/data'
ports:
- '6379:6379'
restart: always
# Prometheus Service
prometheus:
image: 'prom/prometheus:v2.53.0'
volumes:
- './:/etc/prometheus'
command: '--config.file=/etc/prometheus/prometheus.yml'
ports:
- '9090:9090'
restart: always
# Grafana Service
grafana:
image: 'grafana/grafana:9.5.20'
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
ports:
- '3000:3000'
depends_on:
- prometheus
restart: always
volumes:
redis_data: null
================================================
FILE: deployment/docker/prometheus.yml
================================================
# my global config
global:
evaluation_interval: 15s
scrape_interval: 15s
rule_files: ~
scrape_configs:
-
job_name: prometheus
scrape_interval: 5s
static_configs:
-
targets:
- "localhost:9090"
-
job_name: beetle
metrics_path: /metrics
scrape_interval: 5s
static_configs:
-
targets:
- "xx.ngrok.io"
================================================
FILE: deployment/k8s/incluster/README.md
================================================
## Running Beetle inside Kubernetes Cluster
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml
$ kubectl get pods -n ingress-nginx \
-l app.kubernetes.io/name=ingress-nginx --watch
$ kubectl get svc --namespace=ingress-nginx
```
Update `beetle.yaml` with the database credentials. The following part inside the file
```bash
....
# Application Database
database:
# Database driver (sqlite3, mysql)
driver: ${BEETLE_DATABASE_DRIVER:-mysql}
# Hostname
host: ${BEETLE_DATABASE_MYSQL_HOST:-REPLACE_WITH_MYSQL_HOSTNAME}
# Port
port: ${BEETLE_DATABASE_MYSQL_PORT:-3306}
# Database
name: ${BEETLE_DATABASE_MYSQL_DATABASE:-REPLACE_WITH_MYSQL_DATABASE}
# Username
username: ${BEETLE_DATABASE_MYSQL_USERNAME:-REPLACE_WITH_MYSQL_USERNAME}
# Password
password: ${BEETLE_DATABASE_MYSQL_PASSWORD:-REPLACE_WITH_MYSQL_PASSWORD}
....
```
Deploy a sample application and Beetle API server
```bash
$ kubectl apply -f sample_app.yaml --record
$ kubectl apply -f beetle.yaml --record
$ kubectl get ingress
# Update /etc/hosts with the ingress IP
# 167.x.x.x example.com
$ kubectl describe ingress toad-ing
$ kubectl describe ingress beetle-ing
$ curl http://example.com/toad/_ready
$ curl http://example.com/beetle/_ready
```
Interact with Beetle API server
```bash
# Get clusters
$ curl http://example.com/beetle/api/v1/cluster -H "X-API-KEY: 1234" -s | jq .
# Get cluster
$ curl http://example.com/beetle/api/v1/cluster/production -H "X-API-KEY: 1234" -s | jq .
# Get cluster namespaces
$ curl http://example.com/beetle/api/v1/cluster/production/namespace -H "X-API-KEY: 1234" -s | jq .
# Get namespace
$ curl http://example.com/beetle/api/v1/cluster/production/namespace/default -H "X-API-KEY: 1234" -s | jq .
# Get namespace applications
$ curl http://example.com/beetle/api/v1/cluster/production/namespace/default/app -H "X-API-KEY: 1234" -s | jq .
# Get application `toad`
$ curl http://example.com/beetle/api/v1/cluster/production/namespace/default/app/toad -H "X-API-KEY: 1234" -s | jq .
# Get Async Jobs
$ curl -X GET http://example.com/beetle/api/v1/job -H "X-API-KEY: 1234" -s | jq .
# Deploy a new version with recreate strategy
$ curl -X POST \
-H "X-API-KEY: 1234" \
-d '{"version":"0.2.4","strategy":"recreate"}' \
http://example.com/beetle/api/v1/cluster/production/namespace/default/app/toad/deployment
# Get application `toad` version
$ curl http://example.com/beetle/api/v1/cluster/production/namespace/default/app/toad -H "X-API-KEY: 1234" -s | jq .
# Another deployment with ramped strategy
$ curl -X POST \
-H "X-API-KEY: 1234" \
-d '{"version":"0.2.3","strategy":"ramped", "maxSurge": "1", "maxUnavailable": "0"}' \
http://example.com/beetle/api/v1/cluster/production/namespace/default/app/toad/deployment
```
================================================
FILE: deployment/k8s/incluster/beetle.yaml
================================================
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: beetle-service-account
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: beetle-service-account
namespace: default
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"]
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: beetle-service-account
roleRef:
kind: ClusterRole
name: beetle-service-account
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: beetle-service-account
namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
name: incluster-beetle-configs
namespace: default
data:
config.dist.yml: |-
---
# App configs
app:
# Env mode (dev or prod)
mode: ${BEETLE_APP_MODE:-prod}
# HTTP port
port: ${BEETLE_API_PORT:-8080}
# App URL
domain: ${BEETLE_APP_DOMAIN:-http://127.0.0.1:8080}
# TLS configs
tls:
status: ${BEETLE_API_TLS_STATUS:-off}
pemPath: ${BEETLE_API_TLS_PEMPATH:-cert/server.pem}
keyPath: ${BEETLE_API_TLS_KEYPATH:-cert/server.key}
# Message Broker Configs
broker:
# Broker driver (native)
driver: ${BEETLE_BROKER_DRIVER:-native}
# Native driver configs
native:
# Queue max capacity
capacity: ${BEETLE_BROKER_NATIVE_CAPACITY:-5000}
# Number of concurrent workers
workers: ${BEETLE_BROKER_NATIVE_WORKERS:-4}
# API Configs
api:
key: ${BEETLE_API_KEY:-1234}
# Runtime, Requests/Response and Beetle Metrics
metrics:
prometheus:
# Route for the metrics endpoint
endpoint: ${BEETLE_METRICS_PROM_ENDPOINT:-/metrics}
# Application Database
database:
# Database driver (sqlite3, mysql)
driver: ${BEETLE_DATABASE_DRIVER:-mysql}
# Hostname
host: ${BEETLE_DATABASE_MYSQL_HOST:-REPLACE_WITH_MYSQL_HOSTNAME}
# Port
port: ${BEETLE_DATABASE_MYSQL_PORT:-3306}
# Database
name: ${BEETLE_DATABASE_MYSQL_DATABASE:-REPLACE_WITH_MYSQL_DATABASE}
# Username
username: ${BEETLE_DATABASE_MYSQL_USERNAME:-REPLACE_WITH_MYSQL_USERNAME}
# Password
password: ${BEETLE_DATABASE_MYSQL_PASSWORD:-REPLACE_WITH_MYSQL_PASSWORD}
# Kubernetes Clusters
clusters:
-
name: ${BEETLE_KUBE_CLUSTER_01_NAME:-production}
inCluster: ${BEETLE_KUBE_CLUSTER_01_IN_CLUSTER:-true}
kubeconfig: ${BEETLE_KUBE_CLUSTER_01_CONFIG_FILE:- }
# HTTP Webhook
webhook:
url: ${BEETLE_WEBHOOK_URL:-https://httpbin.org/anything}
retry: ${BEETLE_WEBHOOK_RETRY:-3}
apiKey: ${BEETLE_WEBHOOK_API_KEY:-12345}
# Log configs
log:
# Log level, it can be debug, info, warn, error, panic, fatal
level: ${BEETLE_LOG_LEVEL:-info}
# output can be stdout or abs path to log file /var/logs/beetle.log
output: ${BEETLE_LOG_OUTPUT:-stdout}
# Format can be json
format: ${BEETLE_LOG_FORMAT:-json}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: beetle-deployment
spec:
replicas: 2
selector:
matchLabels:
app: beetle
template:
metadata:
labels:
app: beetle
name: beetle
spec:
serviceAccount: beetle-service-account
serviceAccountName: beetle-service-account
containers:
-
image: "clivern/beetle:1.0.2"
name: beetle-app
volumeMounts:
-
mountPath: /app/configs
name: incluster-beetle-configs-volume
volumes:
-
configMap:
name: incluster-beetle-configs
name: incluster-beetle-configs-volume
---
apiVersion: v1
kind: Service
metadata:
name: beetle-svc
labels:
app: beetle
spec:
ports:
-
port: 80
targetPort: 8080
selector:
app: beetle
type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# example.com/beetle rewrites to example.com/
# example.com/beetle/ rewrites to example.com/
# example.com/beetle/_ready rewrites to example.com/_ready
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: beetle-ing
spec:
rules:
- host: example.com
http:
paths:
- path: /beetle(/|$)(.*)
pathType: Prefix
backend:
service:
name: beetle-svc
port:
number: 80
================================================
FILE: deployment/k8s/incluster/sample_app.yaml
================================================
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
beetle.clivern.com/status: enabled
beetle.clivern.com/application-id: toad
annotations:
beetle.clivern.com/application-name: Toad
beetle.clivern.com/image-format: "clivern/toad:release-[.Release]"
name: toad-deployment
spec:
replicas: 2
selector:
matchLabels:
app: toad
template:
metadata:
labels:
app: toad
name: toad
spec:
containers:
-
image: "clivern/toad:release-0.2.3"
name: toad-app
---
apiVersion: v1
kind: Service
metadata:
name: toad-svc
labels:
app: toad
spec:
ports:
-
port: 80
targetPort: 8080
selector:
app: toad
type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# example.com/toad rewrites to example.com/
# example.com/toad/ rewrites to example.com/
# example.com/toad/_ready rewrites to example.com/_ready
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: toad-ing
spec:
rules:
- host: example.com
http:
paths:
- path: /toad(/|$)(.*)
pathType: Prefix
backend:
service:
name: toad-svc
port:
number: 80
================================================
FILE: go.mod
================================================
module github.com/clivern/beetle
go 1.20
require (
github.com/briandowns/spinner v1.23.0
github.com/drone/envsubst v1.0.3
github.com/gin-gonic/gin v1.10.0
github.com/jinzhu/gorm v1.9.16
github.com/logrusorgru/aurora/v3 v3.0.0
github.com/olekukonko/tablewriter v0.0.5
github.com/prometheus/client_golang v1.18.0
github.com/satori/go.uuid v1.2.0
github.com/sirupsen/logrus v1.9.3
github.com/spf13/cobra v1.8.1
github.com/spf13/viper v1.18.2
k8s.io/api v0.27.4
k8s.io/apimachinery v0.27.4
k8s.io/client-go v0.27.4
)
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/bytedance/sonic v1.11.6 // indirect
github.com/bytedance/sonic/loader v0.1.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/cloudwego/base64x v0.1.4 // indirect
github.com/cloudwego/iasm v0.2.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/emicklei/go-restful/v3 v3.9.0 // indirect
github.com/evanphx/json-patch v4.12.0+incompatible // indirect
github.com/fatih/color v1.14.1 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.1 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.20.0 // indirect
github.com/go-sql-driver/mysql v1.5.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/gnostic v0.5.7-v3refs // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/gofuzz v1.1.0 // indirect
github.com/google/uuid v1.4.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/imdario/mergo v0.3.6 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/mattn/go-sqlite3 v1.14.0 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_model v0.5.0 // indirect
github.com/prometheus/common v0.45.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/sagikazarmark/locafero v0.4.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.12 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.9.0 // indirect
golang.org/x/arch v0.8.0 // indirect
golang.org/x/crypto v0.23.0 // indirect
golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
golang.org/x/net v0.25.0 // indirect
golang.org/x/oauth2 v0.15.0 // indirect
golang.org/x/sys v0.20.0 // indirect
golang.org/x/term v0.20.0 // indirect
golang.org/x/text v0.15.0 // indirect
golang.org/x/time v0.5.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.34.1 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/klog/v2 v2.90.1 // indirect
k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f // indirect
k8s.io/utils v0.0.0-20230209194617-a36077c30491 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)
================================================
FILE: go.sum
================================================
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.44.3/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI=
cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk=
cloud.google.com/go v0.75.0/go.mod h1:VGuuCn7PG0dwsd5XPVm2Mm3wlh3EL55/79EKB6hlPTY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/PuerkitoBio/goquery v1.5.1/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc=
github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/briandowns/spinner v1.23.0 h1:alDF2guRWqa/FOZZYWjlMIx2L6H0wyewPxo/CH4Pt2A=
github.com/briandowns/spinner v1.23.0/go.mod h1:rPG4gmXeN3wQV/TsAY4w8lPdIM6RX3yqeBQJSrbXjuE=
github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s=
github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
github.com/bytedance/sonic v1.11.6 h1:oUp34TzMlL+OY1OUWxHqsdkgC/Zfc85zGqw9siXjrc0=
github.com/bytedance/sonic v1.11.6/go.mod h1:LysEHSvpvDySVdC2f87zGWf6CIKJcAvqab1ZaiQtds4=
github.com/bytedance/sonic/loader v0.1.1 h1:c+e5Pt1k/cy5wMveRDyk2X4B9hF4g7an8N3zCYjJFNM=
github.com/bytedance/sonic/loader v0.1.1/go.mod h1:ncP89zfokxS5LZrJxl5z0UJcsk4M4yY2JpfqGeCtNLU=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cloudwego/base64x v0.1.4 h1:jwCgWpFanWmN8xoIUHa2rtzmkd5J2plF/dnLS6Xd/0Y=
github.com/cloudwego/base64x v0.1.4/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w=
github.com/cloudwego/iasm v0.2.0 h1:1KNIy1I1H9hNNFEEH3DVnI4UujN+1zjpuk6gwHLTssg=
github.com/cloudwego/iasm v0.2.0/go.mod h1:8rXZaNYT2n95jn+zTI1sDr+IgcD2GVs0nlbbQPiEFhY=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/denisenkom/go-mssqldb v0.0.0-20191124224453-732737034ffd h1:83Wprp6ROGeiHFAP8WJdI2RoxALQYgdllERc3N5N2DM=
github.com/denisenkom/go-mssqldb v0.0.0-20191124224453-732737034ffd/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
github.com/drone/envsubst v1.0.3 h1:PCIBwNDYjs50AsLZPYdfhSATKaRg/FJmDc2D6+C2x8g=
github.com/drone/envsubst v1.0.3/go.mod h1:N2jZmlMufstn1KEqvbHjw40h1KyTmnVzHcSc9bFiJ2g=
github.com/emicklei/go-restful/v3 v3.9.0 h1:XwGDlfxEnQZzuopoqxwSEllNcCOM9DhhFyhFIIGKwxE=
github.com/emicklei/go-restful/v3 v3.9.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 h1:Yzb9+7DPaBjB8zlTR87/ElzFsnQfuHnVUVqpZZIcV5Y=
github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5/go.mod h1:a2zkGnVExMxdzMo3M0Hi/3sEU+cWnZpSni0O6/Yb/P0=
github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84=
github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
github.com/fatih/color v1.14.1 h1:qfhVLaG5s+nCROl1zJsZRxFeYrHLqWroPOQ8BWiNb4w=
github.com/fatih/color v1.14.1/go.mod h1:2oHN61fhTpgcxD3TSWCgKDiH1+x4OiDVVGH8WlgGZGg=
github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
github.com/gabriel-vasile/mimetype v1.4.2 h1:w5qFW6JKBz9Y393Y4q372O9A7cUSequkh1Q7OhCmWKU=
github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA=
github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.9.1 h1:4idEAncQnU5cB7BeOkPtxjfCSye0AAm1R0RVIqJ+Jmg=
github.com/gin-gonic/gin v1.9.1/go.mod h1:hPrL7YrpYKXt5YId3A/Tnip5kqbEAP+KLuI3SUcPTeU=
github.com/gin-gonic/gin v1.10.0 h1:nTuyha1TYqgedzytsKYqna+DfLos46nTv2ygFy86HFU=
github.com/gin-gonic/gin v1.10.0/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0=
github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE=
github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
github.com/go-openapi/jsonreference v0.20.1 h1:FBLnyygC4/IZZr893oiomc9XaghoveYTrLC1F86HID8=
github.com/go-openapi/jsonreference v0.20.1/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k=
github.com/go-openapi/swag v0.22.3 h1:yMBqmnQ0gyZvEb/+KzuWZOXgllrXT4SADYbvDaXHv/g=
github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.14.0 h1:vgvQWe3XCz3gIeFDm/HnTIbj6UGmg/+t63MyGU2n5js=
github.com/go-playground/validator/v10 v10.14.0/go.mod h1:9iXMNT7sEkjXb0I+enO7QXmzG6QCsPWY4zveKFVRSyU=
github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBExVwjEviJTixqxL8=
github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 h1:p104kn46Q8WdvHunIJ9dAyjPVtrBPhSr3KT2yUst43I=
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-sql/civil v0.0.0-20
gitextract_u0vtr08u/ ├── .gitattributes ├── .github/ │ ├── CODEOWNERS │ ├── FUNDING.yml │ ├── ISSUE_TEMPLATE/ │ │ ├── bug_report.md │ │ └── feature_request.md │ ├── auto-merge.yml │ ├── boring-cyborg.yml │ └── workflows/ │ ├── build.yml │ ├── release.yml │ └── release_pkg.yml ├── .gitignore ├── .go-version ├── .goreleaser.yml ├── .mergify.yml ├── .poodle.toml ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── assets/ │ └── img/ │ └── chart.drawio ├── beetle.go ├── beetle_test.go ├── bin/ │ └── release.sh ├── config.dist.yml ├── config.testing.yml ├── config.toml ├── core/ │ ├── cmd/ │ │ ├── apps.go │ │ ├── deploy.go │ │ ├── license.go │ │ ├── root.go │ │ ├── serve.go │ │ └── version.go │ ├── controller/ │ │ ├── application.go │ │ ├── applications.go │ │ ├── cluster.go │ │ ├── clusters.go │ │ ├── daemon.go │ │ ├── deployment.go │ │ ├── health_check.go │ │ ├── health_check_test.go │ │ ├── job.go │ │ ├── jobs.go │ │ ├── metrics.go │ │ ├── namespace.go │ │ ├── namespaces.go │ │ ├── ready_check.go │ │ ├── ready_check_test.go │ │ └── worker.go │ ├── kubernetes/ │ │ ├── application.go │ │ ├── cluster.go │ │ ├── cluster_test.go │ │ ├── config.go │ │ ├── configmap.go │ │ ├── deployment.go │ │ ├── deployment_strategy.go │ │ ├── namespace.go │ │ ├── namespace_test.go │ │ └── pod.go │ ├── middleware/ │ │ ├── auth.go │ │ ├── correlation.go │ │ ├── log.go │ │ └── metric.go │ ├── migration/ │ │ └── schema.go │ ├── model/ │ │ ├── application.go │ │ ├── cluster.go │ │ ├── configmap.go │ │ ├── configs.go │ │ ├── dsn.go │ │ ├── dsn_test.go │ │ ├── job.go │ │ ├── message.go │ │ ├── metric.go │ │ ├── migration.go │ │ ├── namespace.go │ │ ├── patch.go │ │ └── request.go │ ├── module/ │ │ ├── database.go │ │ ├── database_test.go │ │ ├── file_system.go │ │ ├── http.go │ │ ├── http_test.go │ │ ├── prometheus.go │ │ ├── remote.go │ │ └── remote_test.go │ └── util/ │ ├── helpers.go │ └── helpers_test.go ├── deployment/ │ ├── docker/ │ │ ├── README.md │ │ ├── docker-compose.yml │ │ └── prometheus.yml │ └── k8s/ │ └── incluster/ │ ├── README.md │ ├── beetle.yaml │ └── sample_app.yaml ├── go.mod ├── go.sum ├── pkg/ │ ├── expect.go │ └── server_mock.go ├── renovate.json ├── sdk/ │ ├── application.go │ ├── application_test.go │ ├── client.go │ ├── cluster.go │ ├── cluster_test.go │ ├── deployment.go │ ├── deployment_test.go │ ├── job.go │ ├── job_test.go │ ├── namespace.go │ └── namespace_test.go └── swagger.yaml
SYMBOL INDEX (209 symbols across 74 files)
FILE: beetle.go
function main (line 18) | func main() {
FILE: beetle_test.go
function TestMain (line 25) | func TestMain(t *testing.T) {
FILE: core/cmd/apps.go
function init (line 99) | func init() {
FILE: core/cmd/deploy.go
function init (line 116) | func init() {
FILE: core/cmd/license.go
function init (line 41) | func init() {
FILE: core/cmd/root.go
function Execute (line 23) | func Execute() {
FILE: core/cmd/serve.go
function init (line 202) | func init() {
FILE: core/cmd/version.go
function init (line 55) | func init() {
FILE: core/controller/application.go
function Application (line 19) | func Application(c *gin.Context) {
FILE: core/controller/applications.go
function Applications (line 19) | func Applications(c *gin.Context) {
FILE: core/controller/cluster.go
function Cluster (line 19) | func Cluster(c *gin.Context) {
FILE: core/controller/clusters.go
function Clusters (line 19) | func Clusters(c *gin.Context) {
FILE: core/controller/daemon.go
function init (line 52) | func init() {
function Daemon (line 60) | func Daemon() {
FILE: core/controller/deployment.go
function CreateDeployment (line 19) | func CreateDeployment(c *gin.Context, messages chan<- string) {
FILE: core/controller/health_check.go
function HealthCheck (line 15) | func HealthCheck(c *gin.Context) {
FILE: core/controller/health_check_test.go
function TestHealthCheck (line 27) | func TestHealthCheck(t *testing.T) {
FILE: core/controller/job.go
function GetJob (line 18) | func GetJob(c *gin.Context) {
function DeleteJob (line 66) | func DeleteJob(c *gin.Context) {
FILE: core/controller/jobs.go
function Jobs (line 17) | func Jobs(c *gin.Context) {
FILE: core/controller/metrics.go
function init (line 31) | func init() {
function Metrics (line 37) | func Metrics() http.Handler {
FILE: core/controller/namespace.go
function Namespace (line 19) | func Namespace(c *gin.Context) {
FILE: core/controller/namespaces.go
function Namespaces (line 19) | func Namespaces(c *gin.Context) {
FILE: core/controller/ready_check.go
function ReadyCheck (line 17) | func ReadyCheck(c *gin.Context) {
FILE: core/controller/ready_check_test.go
function TestReadyCheck (line 27) | func TestReadyCheck(t *testing.T) {
FILE: core/controller/worker.go
function Worker (line 23) | func Worker(workerID int, messages <-chan string) {
FILE: core/kubernetes/application.go
method GetApplication (line 18) | func (c *Cluster) GetApplication(ctx context.Context, namespace, id, nam...
FILE: core/kubernetes/cluster.go
type Clusters (line 22) | type Clusters struct
type Cluster (line 27) | type Cluster struct
method Override (line 68) | func (c *Cluster) Override(objects ...runtime.Object) {
method Config (line 74) | func (c *Cluster) Config() error {
method Ping (line 118) | func (c *Cluster) Ping(ctx context.Context) (bool, error) {
function GetClusters (line 36) | func GetClusters() ([]*Cluster, error) {
function GetCluster (line 49) | func GetCluster(name string) (*Cluster, error) {
FILE: core/kubernetes/cluster_test.go
function TestCluster (line 23) | func TestCluster(t *testing.T) {
FILE: core/kubernetes/config.go
method GetConfig (line 18) | func (c *Cluster) GetConfig(ctx context.Context, namespace string) (mode...
FILE: core/kubernetes/configmap.go
method GetConfigMap (line 16) | func (c *Cluster) GetConfigMap(ctx context.Context, namespace, name stri...
FILE: core/kubernetes/deployment.go
method GetDeployments (line 20) | func (c *Cluster) GetDeployments(ctx context.Context, namespace, label s...
method GetDeployment (line 48) | func (c *Cluster) GetDeployment(ctx context.Context, namespace, name str...
method PatchDeployment (line 70) | func (c *Cluster) PatchDeployment(ctx context.Context, namespace, name, ...
method FetchDeploymentStatus (line 93) | func (c *Cluster) FetchDeploymentStatus(ctx context.Context, namespace, ...
FILE: core/kubernetes/deployment_strategy.go
method Deploy (line 18) | func (c *Cluster) Deploy(deploymentRequest model.DeploymentRequest) (boo...
method RecreateStrategy (line 49) | func (c *Cluster) RecreateStrategy(deploymentRequest model.DeploymentReq...
method RampedStrategy (line 143) | func (c *Cluster) RampedStrategy(deploymentRequest model.DeploymentReque...
method BlueGreenStrategy (line 257) | func (c *Cluster) BlueGreenStrategy(_ model.DeploymentRequest) (bool, er...
method CanaryStrategy (line 262) | func (c *Cluster) CanaryStrategy(_ model.DeploymentRequest) (bool, error) {
FILE: core/kubernetes/namespace.go
method GetNamespaces (line 17) | func (c *Cluster) GetNamespaces(ctx context.Context) ([]model.Namespace,...
method GetNamespace (line 44) | func (c *Cluster) GetNamespace(ctx context.Context, name string) (model....
FILE: core/kubernetes/namespace_test.go
function TestNamespace (line 26) | func TestNamespace(t *testing.T) {
FILE: core/middleware/auth.go
function Auth (line 17) | func Auth() gin.HandlerFunc {
FILE: core/middleware/correlation.go
function Correlation (line 16) | func Correlation() gin.HandlerFunc {
FILE: core/middleware/log.go
function Logger (line 16) | func Logger() gin.HandlerFunc {
FILE: core/middleware/metric.go
function init (line 42) | func init() {
function Metric (line 49) | func Metric() gin.HandlerFunc {
FILE: core/migration/schema.go
type Job (line 15) | type Job struct
method LoadFromJSON (line 29) | func (j *Job) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 38) | func (j *Job) ConvertToJSON() (string, error) {
FILE: core/model/application.go
type Container (line 12) | type Container struct
type Application (line 20) | type Application struct
method LoadFromJSON (line 39) | func (c *Application) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 48) | func (c *Application) ConvertToJSON() (string, error) {
type Applications (line 28) | type Applications struct
method LoadFromJSON (line 57) | func (c *Applications) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 66) | func (c *Applications) ConvertToJSON() (string, error) {
type Deployment (line 33) | type Deployment struct
method LoadFromJSON (line 75) | func (d *Deployment) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 84) | func (d *Deployment) ConvertToJSON() (string, error) {
FILE: core/model/cluster.go
type Cluster (line 12) | type Cluster struct
method LoadFromJSON (line 23) | func (c *Cluster) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 32) | func (c *Cluster) ConvertToJSON() (string, error) {
type Clusters (line 18) | type Clusters struct
method LoadFromJSON (line 41) | func (c *Clusters) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 50) | func (c *Clusters) ConvertToJSON() (string, error) {
FILE: core/model/configmap.go
type ConfigMap (line 12) | type ConfigMap struct
method LoadFromJSON (line 22) | func (d *ConfigMap) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 31) | func (d *ConfigMap) ConvertToJSON() (string, error) {
FILE: core/model/configs.go
type App (line 8) | type App struct
type Configs (line 15) | type Configs struct
FILE: core/model/dsn.go
type DSN (line 13) | type DSN struct
method ToString (line 23) | func (d *DSN) ToString() string {
method LoadFromJSON (line 40) | func (d *DSN) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 49) | func (d *DSN) ConvertToJSON() (string, error) {
FILE: core/model/dsn_test.go
function TestDsnToString (line 14) | func TestDsnToString(t *testing.T) {
FILE: core/model/job.go
type Job (line 33) | type Job struct
method LoadFromJSON (line 53) | func (j *Job) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 62) | func (j *Job) ConvertToJSON() (string, error) {
type Jobs (line 48) | type Jobs struct
method LoadFromJSON (line 71) | func (j *Jobs) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 80) | func (j *Jobs) ConvertToJSON() (string, error) {
FILE: core/model/message.go
type Message (line 12) | type Message struct
method LoadFromJSON (line 18) | func (m *Message) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 27) | func (m *Message) ConvertToJSON() (string, error) {
FILE: core/model/metric.go
constant COUNTER (line 16) | COUNTER string = "counter"
constant GAUGE (line 18) | GAUGE string = "gauge"
constant HISTOGRAM (line 20) | HISTOGRAM string = "histogram"
constant SUMMARY (line 22) | SUMMARY string = "summary"
type Metric (line 26) | type Metric struct
method LoadFromJSON (line 37) | func (m *Metric) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 46) | func (m *Metric) ConvertToJSON() (string, error) {
method LabelKeys (line 55) | func (m *Metric) LabelKeys() []string {
method LabelValues (line 66) | func (m *Metric) LabelValues() []string {
method GetValueAsFloat (line 77) | func (m *Metric) GetValueAsFloat() (float64, error) {
FILE: core/model/migration.go
type Migration (line 13) | type Migration struct
method LoadFromJSON (line 20) | func (m *Migration) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 29) | func (m *Migration) ConvertToJSON() (string, error) {
FILE: core/model/namespace.go
type Namespace (line 12) | type Namespace struct
method LoadFromJSON (line 24) | func (d *Namespace) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 33) | func (d *Namespace) ConvertToJSON() (string, error) {
type Namespaces (line 19) | type Namespaces struct
method LoadFromJSON (line 42) | func (d *Namespaces) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 51) | func (d *Namespaces) ConvertToJSON() (string, error) {
FILE: core/model/patch.go
type PatchStringValue (line 8) | type PatchStringValue struct
type PatchUInt32Value (line 15) | type PatchUInt32Value struct
FILE: core/model/request.go
type DeploymentRequest (line 25) | type DeploymentRequest struct
method LoadFromJSON (line 39) | func (d *DeploymentRequest) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 48) | func (d *DeploymentRequest) ConvertToJSON() (string, error) {
method Validate (line 57) | func (d *DeploymentRequest) Validate(strategies []string) error {
function In (line 75) | func In(val interface{}, array interface{}) bool {
FILE: core/module/database.go
type Database (line 22) | type Database struct
method Connect (line 27) | func (db *Database) Connect(dsn model.DSN) error {
method Ping (line 45) | func (db *Database) Ping() error {
method AutoConnect (line 73) | func (db *Database) AutoConnect() error {
method Migrate (line 100) | func (db *Database) Migrate() bool {
method Rollback (line 108) | func (db *Database) Rollback() bool {
method HasTable (line 116) | func (db *Database) HasTable(table string) bool {
method CreateJob (line 121) | func (db *Database) CreateJob(job *model.Job) *model.Job {
method JobExistByID (line 127) | func (db *Database) JobExistByID(id int) bool {
method GetJobByID (line 136) | func (db *Database) GetJobByID(id int) model.Job {
method GetJobs (line 145) | func (db *Database) GetJobs() []model.Job {
method JobExistByUUID (line 154) | func (db *Database) JobExistByUUID(uuid string) bool {
method GetJobByUUID (line 163) | func (db *Database) GetJobByUUID(uuid string) model.Job {
method GetPendingJobByType (line 172) | func (db *Database) GetPendingJobByType(jobType string) model.Job {
method CountJobs (line 181) | func (db *Database) CountJobs(status string) int {
method DeleteJobByID (line 190) | func (db *Database) DeleteJobByID(id int) {
method DeleteJobByUUID (line 195) | func (db *Database) DeleteJobByUUID(uuid string) {
method UpdateJobByID (line 200) | func (db *Database) UpdateJobByID(job *model.Job) {
method Close (line 205) | func (db *Database) Close() error {
method ReleaseChildJobs (line 210) | func (db *Database) ReleaseChildJobs(parentID int) {
FILE: core/module/database_test.go
function TestDatabase (line 25) | func TestDatabase(t *testing.T) {
FILE: core/module/file_system.go
type FileSystem (line 12) | type FileSystem struct
method PathExists (line 15) | func (fs *FileSystem) PathExists(path string) bool {
method FileExists (line 23) | func (fs *FileSystem) FileExists(path string) bool {
method DirExists (line 33) | func (fs *FileSystem) DirExists(path string) bool {
method EnsureDir (line 43) | func (fs *FileSystem) EnsureDir(dirName string, mode int) (bool, error) {
FILE: core/module/http.go
type HTTPClient (line 19) | type HTTPClient struct
method Get (line 31) | func (h *HTTPClient) Get(ctx context.Context, endpoint string, paramet...
method Post (line 61) | func (h *HTTPClient) Post(ctx context.Context, endpoint string, data s...
method Put (line 91) | func (h *HTTPClient) Put(ctx context.Context, endpoint string, data st...
method Patch (line 121) | func (h *HTTPClient) Patch(ctx context.Context, endpoint string, data ...
method Delete (line 151) | func (h *HTTPClient) Delete(ctx context.Context, endpoint string, para...
method buildParameters (line 181) | func (h *HTTPClient) buildParameters(endpoint string, parameters map[s...
method BuildData (line 200) | func (h *HTTPClient) BuildData(parameters map[string]string) string {
method ToString (line 211) | func (h *HTTPClient) ToString(response *http.Response) (string, error) {
method GetStatusCode (line 224) | func (h *HTTPClient) GetStatusCode(response *http.Response) int {
method GetHeaderValue (line 229) | func (h *HTTPClient) GetHeaderValue(response *http.Response, key strin...
function NewHTTPClient (line 24) | func NewHTTPClient(timeout int) *HTTPClient {
FILE: core/module/http_test.go
function TestHttpGet (line 17) | func TestHttpGet(t *testing.T) {
function TestHttpDelete (line 42) | func TestHttpDelete(t *testing.T) {
function TestHttpPost (line 67) | func TestHttpPost(t *testing.T) {
function TestHttpPut (line 97) | func TestHttpPut(t *testing.T) {
function TestHttpGetStatusCode1 (line 127) | func TestHttpGetStatusCode1(t *testing.T) {
function TestHttpGetStatusCode2 (line 148) | func TestHttpGetStatusCode2(t *testing.T) {
function TestHttpGetStatusCode3 (line 169) | func TestHttpGetStatusCode3(t *testing.T) {
function TestHttpGetStatusCode4 (line 190) | func TestHttpGetStatusCode4(t *testing.T) {
function TestBuildParameters (line 211) | func TestBuildParameters(t *testing.T) {
function TestBuildData (line 222) | func TestBuildData(t *testing.T) {
FILE: core/module/prometheus.go
type Prometheus (line 17) | type Prometheus struct
method Send (line 25) | func (p *Prometheus) Send(metrics []model.Metric) error {
method Summary (line 54) | func (p *Prometheus) Summary(item model.Metric) error {
method Counter (line 97) | func (p *Prometheus) Counter(item model.Metric) error {
method Histogram (line 146) | func (p *Prometheus) Histogram(item model.Metric) error {
method Gauge (line 191) | func (p *Prometheus) Gauge(item model.Metric) error {
function NewPrometheus (line 20) | func NewPrometheus() *Prometheus {
FILE: core/module/remote.go
constant ReleaseURL (line 15) | ReleaseURL = "https://api.github.com/repos/Clivern/Beetle/releases/latest"
type LatestRelease (line 18) | type LatestRelease struct
method LoadFromJSON (line 24) | func (lr *LatestRelease) LoadFromJSON(data []byte) (bool, error) {
method ConvertToJSON (line 33) | func (lr *LatestRelease) ConvertToJSON() (string, error) {
function GetLatestRelease (line 42) | func GetLatestRelease() (LatestRelease, error) {
FILE: core/module/remote_test.go
function TestRemote (line 15) | func TestRemote(t *testing.T) {
FILE: core/util/helpers.go
function InArray (line 19) | func InArray(val interface{}, array interface{}) bool {
function GenerateUUID4 (line 35) | func GenerateUUID4() string {
function ListFiles (line 41) | func ListFiles(basePath string) []string {
function ReadFile (line 58) | func ReadFile(path string) string {
function FilterFiles (line 67) | func FilterFiles(files, filters []string) []string {
function Unset (line 85) | func Unset(a []string, i int) []string {
function ConvertToJSON (line 92) | func ConvertToJSON(val interface{}) (string, error) {
FILE: core/util/helpers_test.go
function TestInArray (line 15) | func TestInArray(t *testing.T) {
FILE: pkg/expect.go
function Expect (line 13) | func Expect(t *testing.T, got, want interface{}) {
FILE: pkg/server_mock.go
function ServerMock (line 13) | func ServerMock(uri, response string, statusCode int) *httptest.Server {
FILE: sdk/application.go
method GetApplications (line 16) | func (c *Client) GetApplications(ctx context.Context, cluster, namespace...
method GetApplication (line 56) | func (c *Client) GetApplication(ctx context.Context, cluster, namespace,...
FILE: sdk/application_test.go
function TestApplicationCRUD (line 26) | func TestApplicationCRUD(t *testing.T) {
FILE: sdk/client.go
type Client (line 12) | type Client struct
method SetHTTPClient (line 19) | func (c *Client) SetHTTPClient(httpClient *module.HTTPClient) {
method SetAPIURL (line 24) | func (c *Client) SetAPIURL(APIURL string) {
method SetAPIKey (line 29) | func (c *Client) SetAPIKey(APIKey string) {
FILE: sdk/cluster.go
method GetClusters (line 16) | func (c *Client) GetClusters(ctx context.Context) (model.Clusters, error) {
method GetCluster (line 56) | func (c *Client) GetCluster(ctx context.Context, cluster string) (model....
FILE: sdk/cluster_test.go
function TestClusterCRUD (line 26) | func TestClusterCRUD(t *testing.T) {
FILE: sdk/deployment.go
method CreateDeployment (line 16) | func (c *Client) CreateDeployment(ctx context.Context, request model.Dep...
FILE: sdk/deployment_test.go
function TestDeploymentCRUD (line 26) | func TestDeploymentCRUD(t *testing.T) {
FILE: sdk/job.go
method GetJobs (line 16) | func (c *Client) GetJobs(ctx context.Context) (model.Jobs, error) {
method GetJob (line 56) | func (c *Client) GetJob(ctx context.Context, uuid string) (model.Job, er...
method DeleteJob (line 96) | func (c *Client) DeleteJob(ctx context.Context, uuid string) (bool, erro...
FILE: sdk/job_test.go
function TestJobCRUD (line 25) | func TestJobCRUD(t *testing.T) {
FILE: sdk/namespace.go
method GetNamespaces (line 16) | func (c *Client) GetNamespaces(ctx context.Context, cluster string) (mod...
method GetNamespace (line 56) | func (c *Client) GetNamespace(ctx context.Context, cluster, namespace st...
FILE: sdk/namespace_test.go
function TestNamespaceCRUD (line 26) | func TestNamespaceCRUD(t *testing.T) {
Condensed preview — 111 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (315K chars).
[
{
"path": ".gitattributes",
"chars": 25,
"preview": "docs/* linguist-vendored\n"
},
{
"path": ".github/CODEOWNERS",
"chars": 121,
"preview": "# Docs: https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners\n* @clivern"
},
{
"path": ".github/FUNDING.yml",
"chars": 47,
"preview": "github: # clivern\ncustom: clivern.com/sponsor/\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 293,
"preview": "---\nname: Bug report\nabout: Create a report to help us improve\n\n---\n\n**Describe the bug**\nA clear and concise descriptio"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"chars": 423,
"preview": "---\nname: Feature request\nabout: Suggest an idea for this project\n\n---\n\n**Is your feature request related to a problem? "
},
{
"path": ".github/auto-merge.yml",
"chars": 156,
"preview": "# https://github.com/bobvanderlinden/probot-auto-merge\nblockingLabels:\n- blocking\nrules:\n- minApprovals:\n OWNER: 1\n ME"
},
{
"path": ".github/boring-cyborg.yml",
"chars": 1653,
"preview": "---\nfirstIssueWelcomeComment: \"Thanks for opening your first issue here! Be sure to follow the issue template!\"\nfirstPRM"
},
{
"path": ".github/workflows/build.yml",
"chars": 761,
"preview": "name: Build\n\non:\n push:\n pull_request:\n\njobs:\n build:\n runs-on: ubuntu-latest\n strategy:\n fail-fast: false"
},
{
"path": ".github/workflows/release.yml",
"chars": 541,
"preview": "name: Release\n\non:\n push:\n tags:\n - '*'\n\njobs:\n goreleaser:\n runs-on: ubuntu-latest\n steps:\n -\n "
},
{
"path": ".github/workflows/release_pkg.yml",
"chars": 401,
"preview": "name: ReleasePkg\n\non:\n push:\n tags:\n - '*'\n\njobs:\n release:\n runs-on: ubuntu-latest\n steps:\n -\n "
},
{
"path": ".gitignore",
"chars": 226,
"preview": "# Binaries for programs and plugins\n*.exe\n*.exe~\n*.dll\n*.so\n*.dylib\n\n# Test binary, build with `go test -c`\n*.test\n\n# Ou"
},
{
"path": ".go-version",
"chars": 7,
"preview": "1.20.4\n"
},
{
"path": ".goreleaser.yml",
"chars": 620,
"preview": "# This is an example goreleaser.yaml file with some sane defaults.\n# Make sure to check the documentation at http://gore"
},
{
"path": ".mergify.yml",
"chars": 654,
"preview": "---\npull_request_rules:\n -\n actions:\n merge:\n method: squash\n conditions:\n - author!=Clivern\n "
},
{
"path": ".poodle.toml",
"chars": 4599,
"preview": "# API Definition For Beetle\n# --------------------------\n#\n# In order to use this file:\n# 1. Check & Install https://git"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 3349,
"preview": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nIn the interest of fostering an open and welcoming environment, w"
},
{
"path": "CONTRIBUTING.md",
"chars": 620,
"preview": "## Contributing\n\n- With issues:\n - Use the search tool before opening a new issue.\n - Please provide source code and c"
},
{
"path": "Dockerfile",
"chars": 528,
"preview": "FROM golang:1.20.2\n\nARG BEETLE_VERSION=1.0.2\n\nENV GO111MODULE=on\n\nRUN mkdir -p /app/configs\nRUN mkdir -p /app/var/logs\nR"
},
{
"path": "LICENSE",
"chars": 1064,
"preview": "MIT License\n\nCopyright (c) 2020 Clivern\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof"
},
{
"path": "Makefile",
"chars": 2660,
"preview": "GO ?= go\nGOFMT ?= $(GO)fmt\npkgs = ./...\nHUGO ?= hugo\n\n\nhelp: Makefile\n\t@echo\n\t@echo \" Choose a"
},
{
"path": "README.md",
"chars": 6954,
"preview": "<p align=\"center\">\n <img src=\"https://raw.githubusercontent.com/clivern/Beetle/main/assets/img/gopher.png?v=1.0.4\" wi"
},
{
"path": "assets/img/chart.drawio",
"chars": 3757,
"preview": "<mxfile modified=\"2021-04-03T21:36:16.551Z\" host=\"app.diagrams.net\" agent=\"5.0 (Macintosh; Intel Mac OS X 10_15_4) Apple"
},
{
"path": "beetle.go",
"chars": 479,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "beetle_test.go",
"chars": 1091,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "bin/release.sh",
"chars": 347,
"preview": "#!/bin/bash\n\n# Fetch latest version\nexport LATEST_VERSION=$(curl --silent \"https://api.github.com/repos/clivern/beetle/r"
},
{
"path": "config.dist.yml",
"chars": 2515,
"preview": "# App configs\napp:\n # Env mode (dev or prod)\n mode: ${BEETLE_APP_MODE:-dev}\n # HTTP port\n port: ${BEETLE_API"
},
{
"path": "config.testing.yml",
"chars": 2518,
"preview": "# App configs\napp:\n # Env mode (dev or prod)\n mode: ${BEETLE_APP_MODE:-test}\n # HTTP port\n port: ${BEETLE_AP"
},
{
"path": "config.toml",
"chars": 614,
"preview": "ignoreGeneratedHeader = false\nseverity = \"warning\"\nconfidence = 0.8\nerrorCode = 0\nwarningCode = 0\n\n[rule.blank-imports]\n"
},
{
"path": "core/cmd/apps.go",
"chars": 2843,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/cmd/deploy.go",
"chars": 3754,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/cmd/license.go",
"chars": 1467,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/cmd/root.go",
"chars": 571,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/cmd/serve.go",
"chars": 4867,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/cmd/version.go",
"chars": 999,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/application.go",
"chars": 2026,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/applications.go",
"chars": 1713,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/cluster.go",
"chars": 1138,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/clusters.go",
"chars": 1202,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/daemon.go",
"chars": 4224,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/deployment.go",
"chars": 2844,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/health_check.go",
"chars": 534,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/health_check_test.go",
"chars": 1645,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/job.go",
"chars": 2176,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/jobs.go",
"chars": 724,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/metrics.go",
"chars": 1018,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/namespace.go",
"chars": 1338,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/namespaces.go",
"chars": 1171,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/ready_check.go",
"chars": 1231,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/ready_check_test.go",
"chars": 1638,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/controller/worker.go",
"chars": 5546,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/application.go",
"chars": 1470,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/cluster.go",
"chars": 2490,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/cluster_test.go",
"chars": 1909,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/config.go",
"chars": 1775,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/configmap.go",
"chars": 968,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/deployment.go",
"chars": 4758,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/deployment_strategy.go",
"chars": 6917,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/namespace.go",
"chars": 1433,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/namespace_test.go",
"chars": 3802,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/kubernetes/pod.go",
"chars": 168,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/middleware/auth.go",
"chars": 882,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/middleware/correlation.go",
"chars": 548,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/middleware/log.go",
"chars": 1126,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/middleware/metric.go",
"chars": 1892,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/migration/schema.go",
"chars": 930,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/application.go",
"chars": 1984,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/cluster.go",
"chars": 1146,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/configmap.go",
"chars": 952,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/configs.go",
"chars": 361,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/dsn.go",
"chars": 1093,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/dsn_test.go",
"chars": 788,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/job.go",
"chars": 1870,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/message.go",
"chars": 671,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/metric.go",
"chars": 1835,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/migration.go",
"chars": 727,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/namespace.go",
"chars": 1196,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/patch.go",
"chars": 521,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/model/request.go",
"chars": 1860,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/database.go",
"chars": 5002,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/database_test.go",
"chars": 3175,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/file_system.go",
"chars": 1077,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/http.go",
"chars": 4414,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/http_test.go",
"chars": 6876,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/prometheus.go",
"chars": 5159,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/remote.go",
"chars": 1574,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/module/remote_test.go",
"chars": 519,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/util/helpers.go",
"chars": 1922,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "core/util/helpers_test.go",
"chars": 1072,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "deployment/docker/README.md",
"chars": 56,
"preview": "⚠️ This only to test beetle with prometheus and grafana."
},
{
"path": "deployment/docker/docker-compose.yml",
"chars": 873,
"preview": "version: '3'\n\nservices:\n\n # Redis Service\n redis:\n image: 'redis:7.2-alpine'\n volumes:\n -"
},
{
"path": "deployment/docker/prometheus.yml",
"chars": 380,
"preview": "# my global config\nglobal:\n evaluation_interval: 15s\n scrape_interval: 15s\nrule_files: ~\nscrape_configs:\n -\n job_n"
},
{
"path": "deployment/k8s/incluster/README.md",
"chars": 2910,
"preview": "## Running Beetle inside Kubernetes Cluster\n\n```bash\n$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ing"
},
{
"path": "deployment/k8s/incluster/beetle.yaml",
"chars": 5128,
"preview": "---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: beetle-service-account\n namespace: default\n\n\n---\napiVersion: "
},
{
"path": "deployment/k8s/incluster/sample_app.yaml",
"chars": 1265,
"preview": "---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n labels:\n beetle.clivern.com/status: enabled\n beetle.clivern.c"
},
{
"path": "go.mod",
"chars": 4918,
"preview": "module github.com/clivern/beetle\n\ngo 1.20\n\nrequire (\n\tgithub.com/briandowns/spinner v1.23.0\n\tgithub.com/drone/envsubst v"
},
{
"path": "go.sum",
"chars": 77703,
"preview": "cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ncloud.google.com/go v0.34.0/go.mod h1"
},
{
"path": "pkg/expect.go",
"chars": 417,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "pkg/server_mock.go",
"chars": 527,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "renovate.json",
"chars": 49,
"preview": "{\n \"extends\": [\n \"config:base\"\n ]\n}\n"
},
{
"path": "sdk/application.go",
"chars": 2209,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/application_test.go",
"chars": 3392,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/client.go",
"chars": 636,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/cluster.go",
"chars": 2034,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/cluster_test.go",
"chars": 2183,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/deployment.go",
"chars": 1362,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/deployment_test.go",
"chars": 2108,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/job.go",
"chars": 2487,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/job_test.go",
"chars": 3061,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/namespace.go",
"chars": 2127,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "sdk/namespace_test.go",
"chars": 2580,
"preview": "// Copyright 2020 Clivern. All rights reserved.\n// Use of this source code is governed by the MIT\n// license that can be"
},
{
"path": "swagger.yaml",
"chars": 11630,
"preview": "swagger: '2.0'\ninfo:\n description: |\n Application deployment and management should be automated, auditable, and easy"
}
]
About this extraction
This page contains the full source code of the Clivern/Beetle GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 111 files (281.8 KB), approximately 103.7k tokens, and a symbol index with 209 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.