Showing preview only (216K chars total). Download the full file or copy to clipboard to get everything.
Repository: gojek/weaver
Branch: master
Commit: dad97e4bb0b1
Files: 84
Total size: 197.2 KB
Directory structure:
gitextract_25hsj6d8/
├── .gitignore
├── .travis.yml
├── CONTRIBUTING.md
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── acl.go
├── acl_test.go
├── backend.go
├── backend_test.go
├── cmd/
│ └── weaver-server/
│ └── main.go
├── config/
│ ├── config.go
│ ├── config_test.go
│ ├── newrelic.go
│ ├── proxy.go
│ └── statsd.go
├── deployment/
│ └── weaver/
│ ├── .helmignore
│ ├── Chart.yaml
│ ├── README.md
│ ├── templates/
│ │ ├── _helpers.tpl
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ └── statefulset.yaml
│ ├── values-env.yaml
│ └── values.yaml
├── docker-compose.yml
├── docs/
│ └── weaver_acls.md
├── endpoint.go
├── endpoint_test.go
├── etcd/
│ ├── aclkey.go
│ ├── routeloader.go
│ └── routeloader_test.go
├── examples/
│ └── body_lookup/
│ ├── Dockerfile
│ ├── README.md
│ ├── estimate_acl.json
│ ├── estimator/
│ │ ├── Chart.yaml
│ │ ├── templates/
│ │ │ ├── _helpers.tpl
│ │ │ ├── deployment.yaml
│ │ │ └── service.yaml
│ │ ├── values-id.yaml
│ │ ├── values-sg.yaml
│ │ └── values.yaml
│ └── main.go
├── go.mod
├── go.sum
├── goreleaser.yml
├── pkg/
│ ├── instrumentation/
│ │ ├── newrelic.go
│ │ └── statsd.go
│ ├── logger/
│ │ └── logger.go
│ ├── matcher/
│ │ ├── matcher.go
│ │ └── matcher_test.go
│ ├── shard/
│ │ ├── domain.go
│ │ ├── hashring.go
│ │ ├── hashring_test.go
│ │ ├── lookup.go
│ │ ├── lookup_test.go
│ │ ├── modulo.go
│ │ ├── modulo_test.go
│ │ ├── no.go
│ │ ├── no_test.go
│ │ ├── prefix_lookup.go
│ │ ├── prefix_lookup_test.go
│ │ ├── s2.go
│ │ ├── s2_test.go
│ │ └── shard.go
│ └── util/
│ ├── s2.go
│ ├── s2_test.go
│ ├── util.go
│ └── util_test.go
├── server/
│ ├── error.go
│ ├── error_test.go
│ ├── handler.go
│ ├── handler_test.go
│ ├── loader.go
│ ├── mock.go
│ ├── recovery.go
│ ├── recovery_test.go
│ ├── router.go
│ ├── router_test.go
│ ├── server.go
│ └── wrapped_response_writer.go
├── sharder.go
└── weaver.conf.yaml.sample
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Windows Ignores
Thumbs.db
ehthumbs.db
ehthumbs_vista.db
*.stackdump
[Dd]esktop.ini
$RECYCLE.BIN/
*.cab
*.msi
*.msix
*.msm
*.msp
*.lnk
*.rdb
# macOS Ignores
.DS_Store
.AppleDouble
.LSOverride
Icon
._*
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
# Text Editor Ignores
*~
\#*\#
.\#*
*.swp
[._]*.un~
[._]*.s[a-v][a-z]
[._]*.sw[a-p]
[._]s[a-rt-v][a-z]
[._]ss[a-gi-z]
[._]sw[a-p]
tags
TAGS
*.sublime-workspace
*.sublime-project
# IDE Ignores
/.idea
*.iml
.vscode/
# Go Ignores
.exe
*.exe~
*.dll
*.so
*.dylib
*.test
*.out
# Project Ignores
dev/
tmp/
temp/
out/
build/
vendor/
.vendor-new/
/**/application.yml
expt/
cmd/debug
api/*.*
weaver.conf.yaml
================================================
FILE: .travis.yml
================================================
language: go
go: 1.11.5
env:
global:
- ETCD_VER=v3.3.0
- GO111MODULE=on
- DOCKER_LATEST=latest
matrix:
exclude:
go: 1.11.5
setup_etcd: &setup_etcd
before_script:
- curl -L https://github.com/coreos/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
- mkdir -p /tmp/etcd
- tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd --strip-components=1
- /tmp/etcd/etcd --advertise-client-urls http://127.0.0.1:12379 --listen-client-urls http://127.0.0.1:12379 > /dev/null &
build_docker_image: &build_docker_image
before_script:
- |
docker-compose build dev_weaver
[ ! -z $TRAVIS_TAG ] && docker tag weaver_dev_weaver:$DOCKER_LATEST gojektech/weaver:$TRAVIS_TAG
[ ! -z $TRAVIS_TAG ] && docker tag gojektech/weaver:$TRAVIS_TAG gojektech/weaver:$DOCKER_LATEST
[ ! -z $DOCKER_PASSWORD ] && echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin
docker images
services:
- docker
stages:
- test
- name: deploy
if: (repo == gojektech/weaver) AND (branch == master) AND (tag IS present)
# travis is stupid to do its own custom install and script things
# This is the only way to skip them from running
install: echo "Skip global installing..."
script: echo "Skip global script..."
jobs:
include:
- stage: test
- name: "Make Spec"
<<: *setup_etcd
script: make test
- name: "Docker Spec"
<<: *build_docker_image
script: make docker-spec
after_script: make docker-clean
- stage: deploy
- name: "Release Builds"
script: curl -sL https://git.io/goreleaser | bash -s -- --rm-dist --skip-validate
- name: "Docker Builds"
<<: *build_docker_image
script: docker push gojektech/weaver:$TRAVIS_TAG && docker push gojektech/weaver:$DOCKER_LATEST
notifications:
email: false
================================================
FILE: CONTRIBUTING.md
================================================
# Weaver - Contributing
Weaver `github.com/gojektech/weaver` is an open-source project.
It is licensed using the [Apache License 2.0][1].
We appreciate pull requests; here are our guidelines:
1. [File an issue][2]
(if there isn't one already). If your patch
is going to be large it might be a good idea to get the
discussion started early. We are happy to discuss it in a
new issue beforehand, and you can always email
<tech+heimdall@go-jek.com> about future work.
2. Please use [Effective Go Community Guidelines][3].
3. We ask that you squash all the commits together before
pushing and that your commit message references the bug.
## Issue Reporting
- Check that the issue has not already been reported.
- Be clear, concise and precise in your description of the problem.
- Open an issue with a descriptive title and a summary in grammatically correct,
complete sentences.
- Include any relevant code to the issue summary.
## Pull Requests
- Please read this [how to GitHub][4] blog post.
- Use a topic branch to easily amend a pull request later, if necessary.
- Write [good commit messages][5].
- Use the same coding conventions as the rest of the project.
- Open a [pull request][6] that relates to *only* one subject with a clear title
and description in grammatically correct, complete sentences.
Much Thanks! ❤❤❤
GO-JEK Tech
[1]: http://www.apache.org/licenses/LICENSE-2.0
[2]: https://github.com/gojektech/heimdall/issues
[3]: https://golang.org/doc/effective_go.html
[4]: http://gun.io/blog/how-to-github-fork-branch-and-pull-request
[5]: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
[6]: https://help.github.com/articles/using-pull-requests
================================================
FILE: Dockerfile
================================================
FROM golang:1.11.5-alpine as base
ENV GO111MODULE on
RUN apk --no-cache add gcc g++ make ca-certificates git
RUN mkdir /weaver
WORKDIR /weaver
ADD go.mod .
ADD go.sum .
RUN go mod download
FROM base AS weaver_base
ADD . /weaver
RUN make setup
RUN make build
FROM alpine:latest
COPY --from=weaver_base /weaver/out/weaver-server /usr/local/bin/weaver
ENTRYPOINT ["weaver", "start"]
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: Makefile
================================================
.PHONY: all
all: build fmt vet lint test coverage
default: build fmt vet lint test
ALL_PACKAGES=$(shell go list ./... | grep -v "vendor")
APP_EXECUTABLE="out/weaver-server"
COMMIT_HASH=$(shell git rev-parse --verify head | cut -c-1-8)
BUILD_DATE=$(shell date +%Y-%m-%dT%H:%M:%S%z)
setup:
GO111MODULE=on go get -u golang.org/x/lint/golint
GO111MODULE=on go get github.com/mattn/goveralls
compile:
mkdir -p out/
GO111MODULE=on go build -o $(APP_EXECUTABLE) -ldflags "-X main.BuildDate=$(BUILD_DATE) -X main.Commit=$(COMMIT_HASH) -s -w" ./cmd/weaver-server
build: deps compile fmt vet lint
deps:
GO111MODULE=on go mod tidy -v
install:
GO111MODULE=on go install ./...
fmt:
GO111MODULE=on go fmt ./...
vet:
GO111MODULE=on go vet ./...
lint:
@if [[ `golint $(All_PACKAGES) | { grep -vwE "exported (var|function|method|type|const) \S+ should have comment" || true; } | wc -l | tr -d ' '` -ne 0 ]]; then \
golint $(ALL_PACKAGES) | { grep -vwE "exported (var|function|method|type|const) \S+ should have comment" || true; }; \
exit 2; \
fi;
test: copy-config
GO111MODULE=on go test ./...
test-cover-html:
@echo "mode: count" > coverage-all.out
$(foreach pkg, $(ALL_PACKAGES),\
go test -coverprofile=coverage.out -covermode=count $(pkg);\
tail -n +2 coverage.out >> coverage-all.out;)
GO111MODULE=on go tool cover -html=coverage-all.out -o out/coverage.html
copy-config:
cp weaver.conf.yaml.sample weaver.conf.yaml
clean:
go clean && rm -rf ./vendor ./build ./weaver.conf.yaml
docker-clean:
docker-compose down
docker-spec: docker-clean
docker-compose build
docker-compose run --entrypoint "make test" dev_weaver
docker-server:
docker-compose run --entrypoint "make local-server" dev_weaver
docker-up:
docker-compose up -d
local-server: compile
$(APP_EXECUTABLE) start
coverage:
goveralls -service=travis-ci
================================================
FILE: README.md
================================================
# Weaver - A modern HTTP Proxy with Advanced features
<p align="center"><img src="docs/weaver-logo.png" width="360"></p>
<a href="https://travis-ci.org/gojektech/weaver"><img src="https://travis-ci.org/gojektech/weaver.svg?branch=master" alt="Build Status"></img></a> [](https://goreportcard.com/report/github.com/gojekfarm/weaver)
<a href="https://golangci.com"><img src="https://golangci.com/badges/github.com/gojektech/weaver.svg"></img></a>
[](https://coveralls.io/github/gojektech/weaver?branch=master)
[](https://github.com/gojektech/weaver/releases)
* [Description](#description)
* [Features](#features)
* [Installation](#installation)
* [Architecture](#architecture)
* [Configuration](#configuration)
* [Contributing](#contributing)
* [License](#license)
## Description
Weaver is a Layer-7 Load Balancer with Dynamic Sharding Strategies.
It is a modern HTTP reverse proxy with advanced features.
## Features:
- Sharding request based on headers/path/body fields
- Emits Metrics on requests per route per backend
- Dynamic configuring of different routes (No restarts!)
- Is Fast
- Supports multiple algorithms for sharding requests (consistent hashing, modulo, s2 etc)
- Packaged as a single self contained binary
- Logs on failures (Observability)
## Installation
### Build from source
- Clone the repo:
```
git clone git@github.com:gojektech/weaver.git
```
- Build to create weaver binary
```
make build
```
### Binaries for various architectures
Download the binary for a release from: [here](https://github.com/gojekfarm/weaver/releases)
## Architecture
<p align="center"><img src="docs/weaver_architecture.png" width="860"></p>
Weaver uses `etcd` as a control plane to match the incoming requests against a particular route config and shard the traffic to different backends based on some sharding strategy.
Weaver can be configured for different routes matching different paths with various sharding strategies through a simple route config named ACL.
The various sharding strategies supported by weaver are:
- Consistent hashing (hashring)
- Simple lookup based
- Modulo
- Prefix lookup
- S2 based
## Deploying to Kubernetes
Currently we support deploying to kubernetes officially. You can check the doc [here](deployment/weaver)
## Examples
We have examples defined to deploy it to kubernetes and using acls. Please checkout out [examples](examples/body_lookup)
## Configuration
### Defining ACL's
Details on configuring weaver can be found [here](docs/weaver_acls.md)
### Please note
As the famous saying goes, `All Load balancers are proxies, but not every proxy is a load balancer`, weaver currently does not support load balancing.
## Contributing
If you'd like to contribute to the project, refer to the [contributing documentation](https://github.com/gojektech/weaver/blob/master/CONTRIBUTING.md)
## License
```
Copyright 2018, GO-JEK Tech (http://gojek.tech)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```
================================================
FILE: acl.go
================================================
package weaver
import (
"encoding/json"
"fmt"
)
// ACL - Connects to an external endpoint
type ACL struct {
ID string `json:"id"`
Criterion string `json:"criterion"`
EndpointConfig *EndpointConfig `json:"endpoint"`
Endpoint *Endpoint
}
// GenACL - Generates an ACL from JSON
func (acl *ACL) GenACL(val string) error {
return json.Unmarshal([]byte(val), &acl)
}
func (acl ACL) String() string {
return fmt.Sprintf("ACL(%s, %s)", acl.ID, acl.Criterion)
}
================================================
FILE: acl_test.go
================================================
package weaver
import (
"testing"
"github.com/stretchr/testify/assert"
)
const (
validACLDoc = `{
"id": "gojek_hello",
"criterion" : "Method('POST') && Path('/gojek/hello-service')",
"endpoint" : {
"shard_expr": ".serviceType",
"matcher": "body",
"shard_func": "lookup",
"shard_config": {
"999": {
"backend_name": "hello_backend",
"backend":"http://hello.golabs.io"
}
}
}
}`
)
var genACLTests = []struct {
aclDoc string
expectedErr error
}{
{
validACLDoc,
nil,
},
}
func TestGenACL(t *testing.T) {
acl := &ACL{}
for _, tt := range genACLTests {
actualErr := acl.GenACL(tt.aclDoc)
if actualErr != tt.expectedErr {
assert.Failf(t, "Unexpected error message", "want: %v got: %v", tt.expectedErr, actualErr)
}
}
}
================================================
FILE: backend.go
================================================
package weaver
import (
"net"
"net/http"
"net/http/httputil"
"net/url"
"time"
"github.com/gojektech/weaver/config"
"github.com/pkg/errors"
)
type Backend struct {
Handler http.Handler
Server *url.URL
Name string
}
type BackendOptions struct {
Timeout time.Duration
}
func NewBackend(name string, serverURL string, options BackendOptions) (*Backend, error) {
server, err := url.Parse(serverURL)
if err != nil {
return nil, errors.Wrapf(err, "URL Parsing failed for: %s", serverURL)
}
return &Backend{
Name: name,
Handler: newWeaverReverseProxy(server, options),
Server: server,
}, nil
}
func newWeaverReverseProxy(target *url.URL, options BackendOptions) *httputil.ReverseProxy {
proxyConfig := config.Proxy()
proxy := httputil.NewSingleHostReverseProxy(target)
proxy.Transport = &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: options.Timeout,
KeepAlive: proxyConfig.ProxyDialerKeepAliveInMS(),
DualStack: true,
}).DialContext,
MaxIdleConns: proxyConfig.ProxyMaxIdleConns(),
IdleConnTimeout: proxyConfig.ProxyIdleConnTimeoutInMS(),
DisableKeepAlives: !proxyConfig.KeepAliveEnabled(),
}
return proxy
}
================================================
FILE: backend_test.go
================================================
package weaver
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewBackend(t *testing.T) {
serverURL := "http://localhost"
backendOptions := BackendOptions{}
backend, err := NewBackend("foobar", serverURL, backendOptions)
require.NoError(t, err, "should not have failed to create new backend")
assert.NotNil(t, backend.Handler)
assert.Equal(t, serverURL, backend.Server.String())
}
func TestNewBackendFailsWhenURLIsInvalid(t *testing.T) {
serverURL := ":"
backendOptions := BackendOptions{}
backend, err := NewBackend("foobar", serverURL, backendOptions)
require.Error(t, err, "should have failed to create new backend")
assert.Nil(t, backend)
}
================================================
FILE: cmd/weaver-server/main.go
================================================
package main
import (
"context"
"fmt"
"log"
"os"
"os/signal"
"syscall"
raven "github.com/getsentry/raven-go"
"github.com/gojektech/weaver/config"
"github.com/gojektech/weaver/etcd"
"github.com/gojektech/weaver/pkg/instrumentation"
"github.com/gojektech/weaver/pkg/logger"
"github.com/gojektech/weaver/server"
cli "gopkg.in/urfave/cli.v1"
)
func main() {
app := cli.NewApp()
app.Name = "weaver"
app.Usage = "run weaver-server"
app.Version = fmt.Sprintf("%s built on %s (commit: %s)", Version, BuildDate, Commit)
app.Description = "An Advanced HTTP Reverse Proxy with Dynamic Sharding Strategies"
app.Commands = []cli.Command{
{
Name: "start",
Description: "Start weaver server",
Action: startWeaver,
},
}
app.Run(os.Args)
}
func startWeaver(_ *cli.Context) error {
sigC := make(chan os.Signal, 1)
signal.Notify(sigC, syscall.SIGINT, syscall.SIGKILL, syscall.SIGTERM)
config.Load()
raven.SetDSN(config.SentryDSN())
logger.SetupLogger()
err := instrumentation.InitiateStatsDMetrics()
if err != nil {
log.Printf("StatsD: Error initiating client %s", err)
}
defer instrumentation.CloseStatsDClient()
instrumentation.InitNewRelic()
defer instrumentation.ShutdownNewRelic()
routeLoader, err := etcd.NewRouteLoader()
if err != nil {
log.Printf("StartServer: failed to initialise etcd route loader: %s", err)
}
ctx, cancel := context.WithCancel(context.Background())
go server.StartServer(ctx, routeLoader)
sig := <-sigC
log.Printf("Received %d, shutting down", sig)
defer cancel()
server.ShutdownServer(ctx)
return nil
}
// Build information (will be injected during build)
var (
Version = "1.0.0"
Commit = "n/a"
BuildDate = "n/a"
)
================================================
FILE: config/config.go
================================================
package config
import (
"fmt"
"net"
"net/http"
"strconv"
"strings"
"time"
etcd "github.com/coreos/etcd/client"
newrelic "github.com/newrelic/go-agent"
"github.com/spf13/viper"
)
var appConfig Config
type Config struct {
proxyHost string
proxyPort int
etcdKeyPrefix string
loggerLevel string
etcdEndpoints []string
etcdDialTimeout time.Duration
statsDConfig StatsDConfig
newRelicConfig newrelic.Config
sentryDSN string
serverReadTimeout time.Duration
serverWriteTimeout time.Duration
proxyConfig ProxyConfig
}
func Load() {
viper.SetDefault("LOGGER_LEVEL", "error")
viper.SetDefault("SERVER_PORT", "8080")
viper.SetDefault("PROXY_PORT", "8081")
viper.SetConfigName("weaver.conf")
viper.AddConfigPath("./")
viper.AddConfigPath("../")
viper.AddConfigPath("../../")
viper.SetConfigType("yaml")
viper.ReadInConfig()
viper.AutomaticEnv()
appConfig = Config{
proxyHost: extractStringValue("PROXY_HOST"),
proxyPort: extractIntValue("PROXY_PORT"),
etcdKeyPrefix: extractStringValue("ETCD_KEY_PREFIX"),
loggerLevel: extractStringValue("LOGGER_LEVEL"),
etcdEndpoints: strings.Split(extractStringValue("ETCD_ENDPOINTS"), ","),
etcdDialTimeout: time.Duration(extractIntValue("ETCD_DIAL_TIMEOUT")),
statsDConfig: loadStatsDConfig(),
newRelicConfig: loadNewRelicConfig(),
proxyConfig: loadProxyConfig(),
sentryDSN: extractStringValue("SENTRY_DSN"),
serverReadTimeout: time.Duration(extractIntValue("SERVER_READ_TIMEOUT")),
serverWriteTimeout: time.Duration(extractIntValue("SERVER_WRITE_TIMEOUT")),
}
}
func ServerReadTimeoutInMillis() time.Duration {
return appConfig.serverReadTimeout * time.Millisecond
}
func ServerWriteTimeoutInMillis() time.Duration {
return appConfig.serverWriteTimeout * time.Millisecond
}
func ProxyServerAddress() string {
return fmt.Sprintf("%s:%d", appConfig.proxyHost, appConfig.proxyPort)
}
func ETCDKeyPrefix() string {
return appConfig.etcdKeyPrefix
}
func NewRelicConfig() newrelic.Config {
return appConfig.newRelicConfig
}
func SentryDSN() string {
return appConfig.sentryDSN
}
func StatsD() StatsDConfig {
return appConfig.statsDConfig
}
func Proxy() ProxyConfig {
return appConfig.proxyConfig
}
func NewETCDClient() (etcd.Client, error) {
return etcd.New(etcd.Config{
Endpoints: appConfig.etcdEndpoints,
HeaderTimeoutPerRequest: appConfig.etcdDialTimeout * time.Second,
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
},
})
}
func LogLevel() string {
return appConfig.loggerLevel
}
func extractStringValue(key string) string {
checkPresenceOf(key)
return viper.GetString(key)
}
func extractBoolValue(key string) bool {
checkPresenceOf(key)
return viper.GetBool(key)
}
func extractBoolValueDefaultToFalse(key string) bool {
if !viper.IsSet(key) {
return false
}
return viper.GetBool(key)
}
func extractIntValue(key string) int {
checkPresenceOf(key)
v, err := strconv.Atoi(viper.GetString(key))
if err != nil {
panic(fmt.Sprintf("key %s is not a valid Integer value", key))
}
return v
}
func checkPresenceOf(key string) {
if !viper.IsSet(key) {
panic(fmt.Sprintf("key %s is not set", key))
}
}
================================================
FILE: config/config_test.go
================================================
package config
import (
"fmt"
"os"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestShouldLoadConfigFromFile(t *testing.T) {
Load()
assert.NotEmpty(t, LogLevel())
assert.NotNil(t, loadStatsDConfig().Prefix())
assert.NotNil(t, loadStatsDConfig().FlushPeriodInSeconds())
assert.NotNil(t, loadStatsDConfig().Port())
assert.NotNil(t, loadStatsDConfig().Enabled())
}
func TestShouldLoadFromEnvVars(t *testing.T) {
configVars := map[string]string{
"LOGGER_LEVEL": "info",
"NEW_RELIC_APP_NAME": "newrelic",
"NEW_RELIC_LICENSE_KEY": "licence",
"NEW_RELIC_ENABLED": "true",
"STATSD_PREFIX": "weaver",
"STATSD_FLUSH_PERIOD_IN_SECONDS": "20",
"STATSD_HOST": "statsd",
"STATSD_PORT": "8125",
"STATSD_ENABLED": "true",
"ETCD_KEY_PREFIX": "weaver",
"PROXY_DIALER_TIMEOUT_IN_MS": "10",
"PROXY_DIALER_KEEP_ALIVE_IN_MS": "10",
"PROXY_MAX_IDLE_CONNS": "200",
"PROXY_IDLE_CONN_TIMEOUT_IN_MS": "20",
"SENTRY_DSN": "dsn",
"SERVER_READ_TIMEOUT": "100",
"SERVER_WRITE_TIMEOUT": "100",
}
for k, v := range configVars {
err := os.Setenv(k, v)
require.NoError(t, err, fmt.Sprintf("failed to set env for %s key", k))
}
Load()
expectedStatsDConfig := StatsDConfig{
prefix: "weaver",
flushPeriodInSeconds: 20,
host: "statsd",
port: 8125,
enabled: true,
}
assert.Equal(t, "info", LogLevel())
assert.Equal(t, "newrelic", loadNewRelicConfig().AppName)
assert.Equal(t, "licence", loadNewRelicConfig().License)
assert.True(t, loadNewRelicConfig().Enabled)
assert.Equal(t, expectedStatsDConfig, loadStatsDConfig())
assert.Equal(t, "weaver", ETCDKeyPrefix())
assert.Equal(t, "dsn", SentryDSN())
assert.Equal(t, time.Duration(10)*time.Millisecond, Proxy().ProxyDialerTimeoutInMS())
assert.Equal(t, time.Duration(10)*time.Millisecond, Proxy().ProxyDialerKeepAliveInMS())
assert.Equal(t, 200, Proxy().ProxyMaxIdleConns())
assert.Equal(t, time.Duration(20)*time.Millisecond, Proxy().ProxyIdleConnTimeoutInMS())
assert.Equal(t, time.Duration(100)*time.Millisecond, ServerReadTimeoutInMillis())
assert.Equal(t, time.Duration(100)*time.Millisecond, ServerWriteTimeoutInMillis())
}
================================================
FILE: config/newrelic.go
================================================
package config
import (
newrelic "github.com/newrelic/go-agent"
)
func loadNewRelicConfig() newrelic.Config {
config := newrelic.NewConfig(extractStringValue("NEW_RELIC_APP_NAME"),
extractStringValue("NEW_RELIC_LICENSE_KEY"))
config.Enabled = extractBoolValue("NEW_RELIC_ENABLED")
return config
}
================================================
FILE: config/proxy.go
================================================
package config
import "time"
type ProxyConfig struct {
proxyDialerTimeoutInMS int
proxyDialerKeepAliveInMS int
proxyMaxIdleConns int
proxyIdleConnTimeoutInMS int
keepAliveEnabled bool
}
func loadProxyConfig() ProxyConfig {
return ProxyConfig{
proxyDialerTimeoutInMS: extractIntValue("PROXY_DIALER_TIMEOUT_IN_MS"),
proxyDialerKeepAliveInMS: extractIntValue("PROXY_DIALER_KEEP_ALIVE_IN_MS"),
proxyMaxIdleConns: extractIntValue("PROXY_MAX_IDLE_CONNS"),
proxyIdleConnTimeoutInMS: extractIntValue("PROXY_IDLE_CONN_TIMEOUT_IN_MS"),
keepAliveEnabled: extractBoolValueDefaultToFalse("PROXY_KEEP_ALIVE_ENABLED"),
}
}
func (pc ProxyConfig) ProxyDialerTimeoutInMS() time.Duration {
return time.Duration(pc.proxyDialerTimeoutInMS) * time.Millisecond
}
func (pc ProxyConfig) ProxyDialerKeepAliveInMS() time.Duration {
return time.Duration(pc.proxyDialerKeepAliveInMS) * time.Millisecond
}
func (pc ProxyConfig) ProxyMaxIdleConns() int {
return pc.proxyMaxIdleConns
}
func (pc ProxyConfig) ProxyIdleConnTimeoutInMS() time.Duration {
return time.Duration(pc.proxyIdleConnTimeoutInMS) * time.Millisecond
}
func (pc ProxyConfig) KeepAliveEnabled() bool {
return pc.keepAliveEnabled
}
================================================
FILE: config/statsd.go
================================================
package config
type StatsDConfig struct {
prefix string
flushPeriodInSeconds int
host string
port int
enabled bool
}
func loadStatsDConfig() StatsDConfig {
return StatsDConfig{
prefix: extractStringValue("STATSD_PREFIX"),
flushPeriodInSeconds: extractIntValue("STATSD_FLUSH_PERIOD_IN_SECONDS"),
host: extractStringValue("STATSD_HOST"),
port: extractIntValue("STATSD_PORT"),
enabled: extractBoolValue("STATSD_ENABLED"),
}
}
func (sdc StatsDConfig) Prefix() string {
return sdc.prefix
}
func (sdc StatsDConfig) FlushPeriodInSeconds() int {
return sdc.flushPeriodInSeconds
}
func (sdc StatsDConfig) Host() string {
return sdc.host
}
func (sdc StatsDConfig) Port() int {
return sdc.port
}
func (sdc StatsDConfig) Enabled() bool {
return sdc.enabled
}
================================================
FILE: deployment/weaver/.helmignore
================================================
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
================================================
FILE: deployment/weaver/Chart.yaml
================================================
apiVersion: v1
appVersion: "1.0"
description: A Helm chart to deploy weaver along with etcd and statsd (configurable)
name: weaver
version: 0.1.0
maintainers:
- name: Gowtham Sai
email: dev@gowtham.me
================================================
FILE: deployment/weaver/README.md
================================================
# Deploying to Kubernetes
You can deploy to Kubernetes with the helm charts available in this repo.
### Deploying with ETCD
By default, helm install will deploy weaver with etcd. But you can disable deploying etcd if you want to reuse existing ETCD.
```sh
helm upgrade --debug --install proxy-cluster ./deployment/weaver -f ./deployment/weaver/values-env.yaml
```
This will deploy weaver with env values specified in the values-env.yaml file. In case if you want to expose weaver to outside kubernetes you can use NodePort to do that.
```sh
helm upgrade --debug --install proxy-cluster ./deployment/weaver --set service.type=NodePort -f ./deployment/weaver/values-env.yaml
```
This will deploy along with service of type NodePort. So you can access weaver from outside your kube cluster using kube cluster address and NodePort. In production, you might want to consider ingress/load balancer.
### Deploying without ETCD
You can disable deploying ETCD in case if you want to use existing ETCD in your cluster. To do this, all you have to do is to pass `etcd.enabled` value from command line.
```sh
helm upgrade --debug --install proxy-cluster ./deployment/weaver --set etcd.enabled=false -f ./deployment/weaver/values-env.yaml
```
This will disable deploying etcd to cluster. But you have to pass etcd host env variable `ETCD_ENDPOINTS` to make weaver work.
### Bucket List
1. Helm charts here won't support deploying statsd and sentry yet.
2. NEWRELIC key can be set to anything if you don't want to monitor your app using newrelic.
3. Similarly statsd and sentry host can be set to any value.
================================================
FILE: deployment/weaver/templates/_helpers.tpl
================================================
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "weaver.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "weaver.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "weaver.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
================================================
FILE: deployment/weaver/templates/deployment.yaml
================================================
apiVersion: extensions/v1beta1
kind: Deployment
name: {{ template "weaver.fullname" . }}
metadata:
name: {{ template "weaver.fullname" . }}
labels:
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/name: {{ template "weaver.name" . }}
spec:
replicas: {{ .Values.replicaCount }}
template:
metadata:
{{- if .Values.podAnnotations }}
# Allows custom annotations to be specified
annotations:
{{- toYaml .Values.podAnnotations | nindent 8 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "weaver.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: {{ template "weaver.name" . }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.targetPort }}
protocol: TCP
{{- if .Values.weaver.env }}
env:
{{- toYaml .Values.weaver.env | nindent 12 }}
{{- end }}
{{- if .Values.resources }}
resources:
# Minikube when high resource requests are specified by default.
{{- toYaml .Values.resources | nindent 12 }}
{{- end }}
{{- if .Values.etcd.enabled }}
initContainers:
- name: init-etcd-dns
image: busybox
command: ['sh', '-c', 'until nslookup etcd; do echo waiting for etcd dns to be resolving; sleep 1; done;']
- name: init-etcd-health
image: busybox
command: ['sh', '-c', 'until wget --spider etcd:2379/health; do echo waiting for etcd to be up; sleep 1; done;']
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
# Node selectors can be important on mixed Windows/Linux clusters.
{{- toYaml .Values.nodeSelector | nindent 8 }}
{{- end }}
================================================
FILE: deployment/weaver/templates/service.yaml
================================================
apiVersion: v1
kind: Service
metadata:
{{- if .Values.service.annotations }}
annotations:
{{- toYaml .Values.service.annotations | nindent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "weaver.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: server
app.kubernetes.io/part-of: weaver
name: {{ template "weaver.fullname" . }}
spec:
# Provides options for the service so chart users have the full choice
type: "{{ .Values.service.type }}"
clusterIP: "{{ .Values.service.clusterIP }}"
{{- if .Values.service.externalIPs }}
externalIPs:
{{- toYaml .Values.service.externalIPs | nindent 4 }}
{{- end }}
{{- if .Values.service.loadBalancerIP }}
loadBalancerIP: "{{ .Values.service.loadBalancerIP }}"
{{- end }}
{{- if .Values.service.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{- toYaml .Values.service.loadBalancerSourceRanges | nindent 4 }}
{{- end }}
ports:
- name: http
port: {{ .Values.service.port }}
protocol: TCP
targetPort: {{ .Values.service.targetPort }}
{{- if (and (eq .Values.service.type "NodePort") (not (empty .Values.service.nodePort))) }}
nodePort: {{ .Values.service.nodePort }}
{{- end }}
selector:
app.kubernetes.io/name: {{ template "weaver.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
---
{{- if .Values.etcd.enabled }}
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
{{- if .Values.service.annotations }}
{{- toYaml .Values.service.annotations | nindent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: "etcd"
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: control-pane
app.kubernetes.io/part-of: weaver
name: "etcd"
spec:
ports:
- port: {{ .Values.etcd.peerPort }}
name: etcd-server
- port: {{ .Values.etcd.clientPort }}
name: etcd-client
clusterIP: None
selector:
app.kubernetes.io/name: "etcd"
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
================================================
FILE: deployment/weaver/templates/statefulset.yaml
================================================
{{- if .Values.etcd.enabled }}
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
{{- if .Values.service.annotations }}
annotations:
{{- toYaml .Values.service.annotations | indent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: "etcd"
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: control-pane
app.kubernetes.io/part-of: weaver
name: "etcd"
spec:
serviceName: "etcd"
replicas: {{ .Values.etcd.replicas }}
template:
metadata:
{{- if .Values.service.annotations }}
annotations:
{{- toYaml .Values.service.annotations | indent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: "etcd"
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: control-pane
app.kubernetes.io/part-of: weaver
name: "etcd"
spec:
{{- if .Values.affinity }}
affinity:
{{ toYaml .Values.affinity | indent 8 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
containers:
- name: "etcd"
image: "{{ .Values.etcd.image.repository }}:{{ .Values.etcd.image.tag }}"
imagePullPolicy: "{{ .Values.etcd.image.pullPolicy }}"
ports:
- containerPort: {{ .Values.etcd.peerPort }}
name: peer
- containerPort: {{ .Values.etcd.clientPort }}
name: client
resources:
{{ toYaml .Values.resources | indent 10 }}
env:
- name: INITIAL_CLUSTER_SIZE
value: {{ .Values.etcd.replicas | quote }}
- name: SET_NAME
value: "etcd"
{{- if .Values.extraEnv }}
{{ toYaml .Values.extraEnv | indent 8 }}
{{- end }}
volumeMounts:
- name: datadir
mountPath: /var/run/etcd
lifecycle:
preStop:
exec:
command:
- "/bin/sh"
- "-ec"
- |
EPS=""
for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}:2379"
done
HOSTNAME=$(hostname)
member_hash() {
etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}:2380 | cut -d':' -f1 | cut -d'[' -f1
}
SET_ID=${HOSTNAME##*[^0-9]}
if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
echo "Removing ${HOSTNAME} from etcd cluster"
ETCDCTL_ENDPOINT=${EPS} etcdctl member remove $(member_hash)
if [ $? -eq 0 ]; then
# Remove everything otherwise the cluster will no longer scale-up
rm -rf /var/run/etcd/*
fi
fi
command:
- "/bin/sh"
- "-ec"
- |
HOSTNAME=$(hostname)
echo $HOSTNAME
# store member id into PVC for later member replacement
collect_member() {
while ! etcdctl member list &>/dev/null; do sleep 1; done
etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}:2380 | cut -d':' -f1 | cut -d'[' -f1 > /var/run/etcd/member_id
exit 0
}
eps() {
EPS=""
for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}:2379"
done
echo ${EPS}
}
member_hash() {
etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}:2380 | cut -d':' -f1 | cut -d'[' -f1
}
# we should wait for other pods to be up before trying to join
# otherwise we got "no such host" errors when trying to resolve other members
for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
while true; do
echo "Waiting for ${SET_NAME}-${i}.${SET_NAME} to come up"
ping -W 1 -c 1 ${SET_NAME}-${i}.${SET_NAME} > /dev/null && break
sleep 1s
done
done
# re-joining after failure?
echo $(eps)
if [ -e /var/run/etcd/member_id ]; then
echo "Re-joining etcd member"
member_id=$(cat /var/run/etcd/member_id)
# re-join member
ETCDCTL_ENDPOINT=$(eps) etcdctl member update ${member_id} http://${HOSTNAME}.${SET_NAME}:2380 | true
exec etcd --name ${HOSTNAME} \
--listen-peer-urls http://0.0.0.0:2380 \
--listen-client-urls http://0.0.0.0:2379\
--advertise-client-urls http://${HOSTNAME}.${SET_NAME}:2379 \
--data-dir /var/run/etcd/default.etcd
fi
# etcd-SET_ID
SET_ID=${HOSTNAME##*[^0-9]}
# adding a new member to existing cluster (assuming all initial pods are available)
if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
export ETCDCTL_ENDPOINT=$(eps)
# member already added?
MEMBER_HASH=$(member_hash)
if [ -n "${MEMBER_HASH}" ]; then
# the member hash exists but for some reason etcd failed
# as the datadir has not be created, we can remove the member
# and retrieve new hash
etcdctl member remove ${MEMBER_HASH}
fi
echo "Adding new member"
etcdctl member add ${HOSTNAME} http://${HOSTNAME}.${SET_NAME}:2380 | grep "^ETCD_" > /var/run/etcd/new_member_envs
if [ $? -ne 0 ]; then
echo "Exiting"
rm -f /var/run/etcd/new_member_envs
exit 1
fi
cat /var/run/etcd/new_member_envs
source /var/run/etcd/new_member_envs
collect_member &
exec etcd --name ${HOSTNAME} \
--listen-peer-urls http://0.0.0.0:2380 \
--listen-client-urls http://0.0.0.0:2379 \
--advertise-client-urls http://${HOSTNAME}.${SET_NAME}:2379 \
--data-dir /var/run/etcd/default.etcd \
--initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}:2380 \
--initial-cluster ${ETCD_INITIAL_CLUSTER} \
--initial-cluster-state ${ETCD_INITIAL_CLUSTER_STATE}
fi
PEERS=""
for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}:2380"
done
collect_member &
# join member
exec etcd --name ${HOSTNAME} \
--initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}:2380 \
--listen-peer-urls http://0.0.0.0:2380 \
--listen-client-urls http://0.0.0.0:2379 \
--advertise-client-urls http://${HOSTNAME}.${SET_NAME}:2379 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster ${PEERS} \
--initial-cluster-state new \
--data-dir /var/run/etcd/default.etcd
{{- if .Values.etcd.persistentVolume.enabled }}
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
# upstream recommended max is 700M
storage: "{{ .Values.etcd.persistentVolume.storage }}"
{{- if .Values.etcd.persistentVolume.storageClass }}
{{- if (eq "-" .Values.etcd.persistentVolume.storageClass) }}
storageClassName: ""
{{- else }}
storageClassName: "{{ .Values.etcd.persistentVolume.storageClass }}"
{{- end }}
{{- end }}
{{- else }}
volumes:
- name: datadir
{{- if .Values.etcd.memoryMode }}
emptyDir:
medium: Memory
{{- else }}
emptyDir: {}
{{- end }}
{{- end }}
{{- end }}
================================================
FILE: deployment/weaver/values-env.yaml
================================================
weaver:
env:
- name: "PROXY_HOST"
value: "0.0.0.0"
- name: "PROXY_PORT"
value: "8080"
- name: "PROXY_DIALER_TIMEOUT_IN_MS"
value: "1000"
- name: "PROXY_DIALER_KEEP_ALIVE_IN_MS"
value: "100"
- name: "PROXY_IDLE_CONN_TIMEOUT_IN_MS"
value : "100"
- name: "PROXY_MAX_IDLE_CONNS"
value: "100"
- name: "SENTRY_DSN"
value: "sentry"
- name: "SERVER_READ_TIMEOUT"
value: "100"
- name: "SERVER_WRITE_TIMEOUT"
value: "100"
- name: "ETCD_KEY_PREFIX"
value: "weaver"
- name: "ETCD_ENDPOINTS"
value: "http://etcd:2379"
- name: "ETCD_DIAL_TIMEOUT"
value: "5"
- name: "NEW_RELIC_APP_NAME"
value: "weaver"
- name: "NEW_RELIC_LICENSE_KEY"
value: "weaver-temp-license"
- name: "STATSD_ENABLED"
value: "false"
- name: "STATSD_PREFIX"
value: "weaver"
- name: "STATSD_PORT"
value: "8125"
- name: "STATSD_FLUSH_PERIOD_IN_SECONDS"
value: "10"
- name: "STATSD_HOST"
value: "localhost"
- name: "NEW_RELIC_ENABLED"
value: "false"
================================================
FILE: deployment/weaver/values.yaml
================================================
# Default values for weaver.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: gojektech/weaver
tag: stable
pullPolicy: IfNotPresent
nameOverride: ""
fullnameOverride: ""
service:
type: ClusterIP
port: 80
targetPort: 8080
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}
weaver:
env: []
# ETCD Specific configurations
etcd:
enabled: true
peerPort: 2380
clientPort: 2379
component: "etcd"
replicas: 3
image:
repository: "k8s.gcr.io/etcd-amd64"
tag: "2.2.5"
pullPolicy: "IfNotPresent"
resources: {}
persistentVolume:
enabled: false
# storage: "1Gi"
## etcd data Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: "-"
## This is only available when persistentVolume is false:
## If persistentVolume is not enabled, one can choose to use memory mode for ETCD by setting memoryMode to "true".
## The system will create a volume with "medium: Memory"
memoryMode: false
================================================
FILE: docker-compose.yml
================================================
version: '3.4'
services:
dev_statsd:
image: gojektech/statsd:0.7.2
hostname: dev-statsd
ports:
- "18124:8124"
- "18125:8125"
- "18126:8126"
networks:
- weaver_net
dev_etcd:
image: quay.io/coreos/etcd:v2.3.8
hostname: dev-etcd
entrypoint: ["/etcd"]
command: [
"-name",
"etcd0",
"-advertise-client-urls",
"http://localhost:2379,http://localhost:4001",
"-listen-client-urls",
"http://0.0.0.0:2379,http://0.0.0.0:4001",
"-initial-advertise-peer-urls",
"http://localhost:2380",
"-listen-peer-urls",
"http://0.0.0.0:2380",
"-initial-cluster-token",
"etcd-cluster-1",
"-initial-cluster",
"etcd0=http://localhost:2380",
"-initial-cluster-state",
"new"
]
ports:
- "12379:2379"
networks:
- weaver_net
dev_weaver:
build:
context: .
target: weaver_base
hostname: dev-weaver
depends_on:
- dev_etcd
- dev_statsd
environment:
STATD_PORT: 8125
STATSD_HOST: dev_statsd
ETCD_ENDPOINTS: "http://dev_etcd:2379"
stdin_open: true
tty: true
networks:
- weaver_net
networks:
weaver_net:
================================================
FILE: docs/weaver_acls.md
================================================
## Weaver ACLs
Weaver ACL is a document formatted in JSON used to decide the destination of downstream traffic. An example of an ACL is
like the following
``` json
{
"id": "gojek_hello",
"criterion" : "Method(`POST`) && Path(`/gojek/hello-service`)",
"endpoint" : {
"shard_expr": ".serviceType",
"matcher": "body",
"shard_func": "lookup",
"shard_config": {
"999": {
"backend_name": "hello_backend",
"backend":"http://hello.golabs.io"
}
}
}
}
```
The description of each field is as follows
| Field Name | Description |
|---|---|
| `id` | The name of the service |
| `criterion` | The criterion expressed based on [Vulcand Routing](https://godoc.org/github.com/vulcand/route) |
| `endpoint` | The endpoint description (see below) |
For endpoints the keys descriptions are as following:
| Field Name | Description |
|---|---|
| `matcher` | The value to match can be `body`, `path` , `header` or `param` |
| `shard_expr` | Shard expression, the expression to evaluate request based on the matcher |
| `shard_func` | The function of the sharding (See Below) |
| `shard_config` | The backends for each evaluated value |
For each `shard_config` value there are the value evaluated as the result of expression of `shard_expr`. We need to
describe backends for each value.
| Field Name | Description |
|---|---|
| `backend_name` | unique name for the evaluated value |
| `backend` | The URI in which the packet will be forwarded |
---
## ACL examples:
Possible **`shard_func`** values accepted by weaver are : `none`, `lookup`, `prefix-lookup`, `modulo` , `hashring`, `s2`.
Sample ACLs for each accepted **`shard_func`** are provided below.
**`none`**:
``` json
{
"id": "gojek_hello",
"criterion" : "Method(`GET`) && Path(`/gojek/hello-service`)",
"endpoint" : {
"shard_func": "none",
"shard_config": {
"backend_name": "hello_backend",
"backend":"http://hello.golabs.io"
}
}
}
```
Details: Just forwards the traffic. HTTP `GET` to `weaver.io/gojek/hello-service` will be forwarded to backend at `http://hello.golabs.io`.
---
**`lookup`**:
``` json
{
"id": "gojek_hello",
"criterion" : "Method(`POST`) && Path(`/gojek/hello-service`)",
"endpoint" : {
"shard_expr": ".serviceType",
"matcher": "body",
"shard_func": "lookup",
"shard_config": {
"999": {
"backend_name": "hello_backend",
"backend":"http://hello.golabs.io"
},
"6969": {
"backend_name": "bye_backend",
"backend":"http://bye.golabs.io"
}
}
}
}
```
Details: HTTP `POST` to `weaver.io/gojek/hello-service` will be forwarded based on the `shard_expr` field's value within request body. The request body shall be similar to:
``` json
{
"serviceType": "999",
...
}
```
In this scenario the value evaluated by `shard_expr` will be `999`. This will forward the request to `http://hello.golabs.io`. However if the value evaluated by the `shard_expr` is not found within `shard_config`, weaver returns `503` error.
---
**`prefix-lookup`**:
``` json
{
"id": "gojek_prefix_hello",
"criterion": "Method(`PUT`) && Path(`/gojek/hello/world`)",
"endpoint": {
"shard_expr": ".orderNo",
"matcher": "body",
"shard_func": "prefix-lookup",
"shard_config": {
"backends": {
"default": {
"backend_name": "backend_1",
"backend": "http://backend1"
},
"AB-": {
"backend_name": "backend2",
"backend": "http://backend2"
},
"AD-": {
"backend_name": "backend3",
"backend": "http://backend3"
}
},
"prefix_splitter": "-"
}
}
}
```
Details: HTTP `PUT` to `weaver.io/gojek/hello/world` will be forwarded based on the `shard_expr` field's value within request body. The request body shall be similar to:
``` json
{
"orderNo": "AD-2132315",
...
}
```
In this scenario the value evaluated by `shard_expr` will be `AD-2132315`. This value will be split according to `prefix_splitter` in `shard_config.backends` evaluating to `AD-`. This will forward the request to `http://backend3`. However if the value evaluated by the `shard_expr` is not found within `shard_config`, weaver will fallback to the value in the `default` key. If `default` key is not found within `shard_config.backends`, weaver returns `503` error.
---
**`modulo`**:
``` json
{
"id": "drivers-by-driver-id",
"criterion": "Method(`GET`) && PathRegexp(`/v2/drivers/\\d+`)",
"endpoint": {
"shard_config": {
"0": {
"backend_name": "backend1",
"backend": "http://backend1"
},
"1": {
"backend_name": "backend2",
"backend": "http://backend2"
},
"2": {
"backend_name": "backend3",
"backend": "http://backend3"
},
"3": {
"backend_name": "backend4",
"backend": "http://backend4"
}
},
"shard_expr": "/v2/drivers/(\\d+)",
"matcher": "path",
"shard_func": "modulo"
}
}
```
Details: HTTP `GET` to `weaver.io/v2/drivers/2156545453242` will be forwarded based on the value captured by regex in `shard_expr` from `/v2/drivers/2156545453242` path.
The value must be an integer. The backend is selected based on the modulo operation between extracted value (`2156545453242`) with number of backends in the `shard_config`. In this scenario the result is `2156545453242 % 4 = 2`. This will forward the request to `http://backend3`.
---
**`hashring`**:
``` json
{
"id": "driver-location",
"criterion": "Method(`GET`) && PathRegexp(`/gojek/driver/location`)",
"endpoint": {
"shard_config": {
"totalVirtualBackends": 1000,
"backends": {
"0-249": {
"backend_name": "backend1",
"backend": "http://backend1"
},
"250-499": {
"backend_name": "backend2",
"backend": "http://backend2"
},
"500-749": {
"backend_name": "backend3",
"backend": "http://backend3"
},
"750-999": {
"backend_name": "backend4",
"backend": "http://backend4"
}
}
},
"shard_expr": "DriverID",
"matcher": "header",
"shard_func": "hashring"
}
}
```
Details: HTTP `PUT` to `weaver.io/gojek/driver/location` will be forwarded based on the result of hashing function from the here.
In this scenario the key by which we select the backend is obtained by using value of DriverID header since matcher is header. For example if request had DriverID: 34345 header, and hashring calculated hash of that values as hash(34345): 555, it will select backend with 500-749 key. This will forward the request to `http://backend3`
---
**`s2`**:
``` json
{
"id": "nearby-driver-service-get-nearby",
"criterion": "Method(`GET`) && PathRegexp(`/gojek/nearby`)",
"endpoint": {
"shard_config": {
"shard_key_separator": ",",
"shard_key_position": -1,
"backends": {
"3477275891585777700": {
"backend_name": "backend1",
"backend": "http://backend1",
"timeout": 300
},
"3477284687678800000": {
"backend_name": "backend2",
"backend": "http://backend2",
"timeout": 300
},
"3477302279864844300": {
"backend_name": "backend3",
"backend": "http://backend3",
"timeout": 300
},
"3477290185236939000": {
"backend_name": "backend4",
"backend": "http://backend4",
"timeout": 300
}
}
},
"shard_expr": "X-Location",
"matcher": "header",
"shard_func": "s2"
}
}
```
Details: HTTP `GET` to `weaver.io/gojek/nearby` will be forwarded based on the result of s2id calculation from X-Location header in the form of lat and long separated by , in accordance to shard_key_separator value. e.g -6.2428103,106.7940571. Weaver calculated s2id from lat and long because shard_key_position value is -1.
================================================
FILE: endpoint.go
================================================
package weaver
import (
"encoding/json"
"fmt"
"net/http"
"github.com/gojektech/weaver/pkg/matcher"
"github.com/pkg/errors"
)
// EndpointConfig - Defines a config for external service
type EndpointConfig struct {
Matcher string `json:"matcher"`
ShardExpr string `json:"shard_expr"`
ShardFunc string `json:"shard_func"`
ShardConfig json.RawMessage `json:"shard_config"`
}
func (endpointConfig *EndpointConfig) genShardKeyFunc() (shardKeyFunc, error) {
matcherFunc, found := matcher.New(endpointConfig.Matcher)
if !found {
return nil, errors.WithStack(fmt.Errorf("failed to find a matcherMux for: %s", endpointConfig.Matcher))
}
return func(req *http.Request) (string, error) {
return matcherFunc(req, endpointConfig.ShardExpr)
}, nil
}
type Endpoint struct {
sharder Sharder
shardKeyFunc shardKeyFunc
}
func NewEndpoint(endpointConfig *EndpointConfig, sharder Sharder) (*Endpoint, error) {
if sharder == nil {
return nil, errors.New("nil sharder passed in")
}
shardKeyFunc, err := endpointConfig.genShardKeyFunc()
if err != nil {
return nil, errors.Wrapf(err, "failed to generate shardKeyFunc for %s", endpointConfig.ShardExpr)
}
return &Endpoint{
sharder: sharder,
shardKeyFunc: shardKeyFunc,
}, nil
}
func (endpoint *Endpoint) Shard(request *http.Request) (*Backend, error) {
shardKey, err := endpoint.shardKeyFunc(request)
if err != nil {
return nil, errors.Wrapf(err, "failed to find shardKey")
}
return endpoint.sharder.Shard(shardKey)
}
type shardKeyFunc func(*http.Request) (string, error)
================================================
FILE: endpoint_test.go
================================================
package weaver
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewEndpoint(t *testing.T) {
endpointConfig := &EndpointConfig{
Matcher: "path",
ShardExpr: "/.*",
ShardFunc: "lookup",
ShardConfig: json.RawMessage(`{}`),
}
sharder := &stubSharder{}
endpoint, err := NewEndpoint(endpointConfig, sharder)
require.NoError(t, err, "should not fail to create an endpoint from endpointConfig")
assert.NotNil(t, endpoint, "should create an endpoint")
assert.Equal(t, sharder, endpoint.sharder)
}
func TestNewEndpoint_SharderIsNil(t *testing.T) {
endpointConfig := &EndpointConfig{
Matcher: "path",
ShardExpr: "/.*",
ShardFunc: "lookup",
ShardConfig: json.RawMessage(`{}`),
}
endpoint, err := NewEndpoint(endpointConfig, nil)
assert.Error(t, err, "should fail to create an endpoint when sharder is nil")
assert.Nil(t, endpoint)
}
type stubSharder struct {
}
func (stub *stubSharder) Shard(key string) (*Backend, error) {
return nil, nil
}
================================================
FILE: etcd/aclkey.go
================================================
package etcd
import (
"fmt"
"github.com/gojektech/weaver"
)
const (
// ACLKeyFormat - Format for a ACL's key in a KV Store
ACLKeyFormat = "/%s/acls/%s/acl"
)
// ACLKey - Points to a stored ACL
type ACLKey string
// GenACLKey - Generate an ACL Key given etcd's node key
func GenACLKey(key string) ACLKey {
return ACLKey(fmt.Sprintf("%s/acl", key))
}
func GenKey(acl *weaver.ACL, pfx string) ACLKey {
return ACLKey(fmt.Sprintf(ACLKeyFormat, pfx, acl.ID))
}
================================================
FILE: etcd/routeloader.go
================================================
package etcd
import (
"context"
"encoding/json"
"fmt"
"sort"
"github.com/gojektech/weaver"
"github.com/gojektech/weaver/pkg/shard"
etcd "github.com/coreos/etcd/client"
"github.com/gojektech/weaver/config"
"github.com/gojektech/weaver/pkg/logger"
"github.com/gojektech/weaver/server"
"github.com/pkg/errors"
)
func NewRouteLoader() (*RouteLoader, error) {
etcdClient, err := config.NewETCDClient()
if err != nil {
return nil, err
}
return &RouteLoader{
etcdClient: etcdClient,
namespace: config.ETCDKeyPrefix(),
}, nil
}
// RouteLoader - To store and modify proxy configuration
type RouteLoader struct {
etcdClient etcd.Client
namespace string
}
// PutACL - Upserts a given ACL
func (routeLoader *RouteLoader) PutACL(acl *weaver.ACL) (ACLKey, error) {
key := GenKey(acl, routeLoader.namespace)
val, err := json.Marshal(acl)
if err != nil {
return "", err
}
_, err = etcd.NewKeysAPI(routeLoader.etcdClient).Set(context.Background(), string(key), string(val), nil)
if err != nil {
return "", fmt.Errorf("fail to PUT %s:%s with %s", key, acl, err.Error())
}
return key, nil
}
// GetACL - Fetches an ACL given an ACLKey
func (routeLoader *RouteLoader) GetACL(key ACLKey) (*weaver.ACL, error) {
res, err := etcd.NewKeysAPI(routeLoader.etcdClient).Get(context.Background(), string(key), nil)
if err != nil {
return nil, fmt.Errorf("fail to GET %s with %s", key, err.Error())
}
acl := &weaver.ACL{}
if err := json.Unmarshal([]byte(res.Node.Value), acl); err != nil {
return nil, err
}
sharder, err := shard.New(acl.EndpointConfig.ShardFunc, acl.EndpointConfig.ShardConfig)
if err != nil {
return nil, errors.Wrapf(err, "failed to initialize sharder '%s'", acl.EndpointConfig.ShardFunc)
}
acl.Endpoint, err = weaver.NewEndpoint(acl.EndpointConfig, sharder)
if err != nil {
return nil, errors.Wrapf(err, "failed to create a new Endpoint for key: %s", key)
}
return acl, nil
}
// DelACL - Deletes an ACL given an ACLKey
func (routeLoader *RouteLoader) DelACL(key ACLKey) error {
_, err := etcd.NewKeysAPI(routeLoader.etcdClient).Delete(context.Background(), string(key), nil)
if err != nil {
return fmt.Errorf("fail to DELETE %s with %s", key, err.Error())
}
return nil
}
func (routeLoader *RouteLoader) WatchRoutes(ctx context.Context, upsertRouteFunc server.UpsertRouteFunc, deleteRouteFunc server.DeleteRouteFunc) {
etc, key := initEtcd(routeLoader)
watcher := etc.Watcher(key, &etcd.WatcherOptions{Recursive: true})
logger.Infof("starting etcd watcher on %s", key)
for {
res, err := watcher.Next(ctx)
if err != nil {
logger.Errorf("stopping etcd watcher on %s: %v", key, err)
return
}
logger.Debugf("registered etcd watcher event on %v with action %s", res, res.Action)
switch res.Action {
case "set":
fallthrough
case "update":
logger.Debugf("fetching node key %s", res.Node.Key)
acl, err := routeLoader.GetACL(ACLKey(res.Node.Key))
if err != nil {
logger.Errorf("error in fetching %s: %v", res.Node.Key, err)
continue
}
logger.Infof("upserting %v to router", acl)
err = upsertRouteFunc(acl)
if err != nil {
logger.Errorf("error in upserting %v: %v ", acl, err)
continue
}
case "delete":
acl := &weaver.ACL{}
err := acl.GenACL(res.PrevNode.Value)
if err != nil {
logger.Errorf("error in unmarshalling %s: %v", res.PrevNode.Value, err)
continue
}
logger.Infof("deleteing %v to router", acl)
err = deleteRouteFunc(acl)
if err != nil {
logger.Errorf("error in deleting %v: %v ", acl, err)
continue
}
}
}
}
func (routeLoader *RouteLoader) BootstrapRoutes(ctx context.Context, upsertRouteFunc server.UpsertRouteFunc) error {
// TODO: Consider error scenarios and return an error when it makes sense
etc, key := initEtcd(routeLoader)
logger.Infof("bootstrapping router using etcd on %s", key)
res, err := etc.Get(ctx, key, nil)
if err != nil {
logger.Infof("creating router namespace on etcd using %s", key)
_, _ = etc.Set(ctx, key, "", &etcd.SetOptions{
Dir: true,
})
}
if res != nil {
sort.Sort(res.Node.Nodes)
for _, nd := range res.Node.Nodes {
logger.Debugf("fetching node key %s", nd.Key)
acl, err := routeLoader.GetACL(GenACLKey(nd.Key))
if err != nil {
logger.Errorf("error in fetching %s: %v", nd.Key, err)
continue
}
logger.Infof("upserting %v to router", acl)
err = upsertRouteFunc(acl)
if err != nil {
logger.Errorf("error in upserting %v: %v ", acl, err)
continue
}
}
}
return nil
}
func initEtcd(routeLoader *RouteLoader) (etcd.KeysAPI, string) {
key := fmt.Sprintf("/%s/acls/", routeLoader.namespace)
etc := etcd.NewKeysAPI(routeLoader.etcdClient)
return etc, key
}
================================================
FILE: etcd/routeloader_test.go
================================================
package etcd
import (
"context"
"encoding/json"
"errors"
"reflect"
"testing"
"time"
"github.com/gojektech/weaver"
etcd "github.com/coreos/etcd/client"
"github.com/gojektech/weaver/config"
"github.com/gojektech/weaver/pkg/logger"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
)
// Notice: This test uses time.Sleep, TODO: fix it
type RouteLoaderSuite struct {
suite.Suite
ng *RouteLoader
}
func (es *RouteLoaderSuite) SetupTest() {
config.Load()
logger.SetupLogger()
var err error
es.ng, err = NewRouteLoader()
assert.NoError(es.T(), err)
}
func (es *RouteLoaderSuite) TestNewRouteLoader() {
assert.NotNil(es.T(), es.ng)
}
func TestRouteLoaderSuite(tst *testing.T) {
suite.Run(tst, new(RouteLoaderSuite))
}
func (es *RouteLoaderSuite) TestPutACL() {
aclPut := &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && Path(`/ping`)",
EndpointConfig: &weaver.EndpointConfig{
ShardFunc: "lookup",
Matcher: "path",
ShardExpr: "*",
ShardConfig: json.RawMessage(`{
"GF-": {
"backend_name": "foobar",
"backend": "http://customer-locations-primary"
},
"R-": {
"timeout": 100.0,
"backend_name": "foobar",
"backend": "http://customer-locations-secondary"
}
}`),
},
}
key, err := es.ng.PutACL(aclPut)
assert.Nil(es.T(), err, "fail to PUT %s", aclPut)
aclGet, err := es.ng.GetACL(key)
assert.Nil(es.T(), err, "fail to GET with %s", key)
assert.Equal(es.T(), aclPut.ID, aclGet.ID, "PUT %s =/= GET %s", aclPut, aclGet)
assert.Nil(es.T(), es.ng.DelACL(key), "fail to DELETE %+v", aclPut)
}
func (es *RouteLoaderSuite) TestBootstrapRoutes() {
aclPut := &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && Path(`/ping`)",
EndpointConfig: &weaver.EndpointConfig{
ShardFunc: "lookup",
Matcher: "path",
ShardExpr: "*",
ShardConfig: json.RawMessage(`{}`),
},
}
key, err := es.ng.PutACL(aclPut)
assert.NoError(es.T(), err, "failed to PUT %s", aclPut)
aclsChan := make(chan *weaver.ACL, 1)
es.ng.BootstrapRoutes(context.Background(), genRouteProcessorMock(aclsChan))
deepEqual(es.T(), aclPut, <-aclsChan)
assert.Nil(es.T(), es.ng.DelACL(key), "fail to DELETE %+v", aclPut)
}
func (es *RouteLoaderSuite) TestBootstrapRoutesSucceedWhenARouteUpsertFails() {
aclPut := &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && Path(`/ping`)",
EndpointConfig: &weaver.EndpointConfig{
ShardFunc: "lookup",
Matcher: "path",
ShardExpr: "*",
ShardConfig: json.RawMessage(`{
"GF-": {
"backend_name": "foobar",
"backend": "http://customer-locations-primary"
},
"R-": {
"timeout": 100.0,
"backend_name": "foobar",
"backend": "http://customer-locations-secondary"
}
}`),
},
}
key, err := es.ng.PutACL(aclPut)
require.NoError(es.T(), err, "failed to PUT %s", aclPut)
err = es.ng.BootstrapRoutes(context.Background(), failingUpsertRouteFunc)
require.NoError(es.T(), err, "should not have failed to bootstrap routes")
assert.Nil(es.T(), es.ng.DelACL(key), "fail to DELETE %+v", aclPut)
}
func (es *RouteLoaderSuite) TestBootstrapRoutesSucceedWhenARouteDoesntExist() {
err := es.ng.BootstrapRoutes(context.Background(), successUpsertRouteFunc)
require.NoError(es.T(), err, "should not have failed to bootstrap routes")
}
func (es *RouteLoaderSuite) TestBootstrapRoutesSucceedWhenARouteHasInvalidData() {
aclPut := newTestACL("path")
value := `{ "blah": "a }`
key := "abc"
_, err := etcd.NewKeysAPI(es.ng.etcdClient).Set(context.Background(), key, value, nil)
require.NoError(es.T(), err, "failed to PUT %s", aclPut)
err = es.ng.BootstrapRoutes(context.Background(), successUpsertRouteFunc)
require.NoError(es.T(), err, "should not have failed to bootstrap routes")
assert.Nil(es.T(), es.ng.DelACL(ACLKey(key)), "fail to DELETE %+v", aclPut)
}
func (es *RouteLoaderSuite) TestWatchRoutesUpsertRoutesWhenRoutesSet() {
newACL := newTestACL("path")
aclsUpserted := make(chan *weaver.ACL, 1)
watchCtx, cancelWatch := context.WithCancel(context.Background())
defer cancelWatch()
go es.ng.WatchRoutes(watchCtx, genRouteProcessorMock(aclsUpserted), successUpsertRouteFunc)
time.Sleep(100 * time.Millisecond)
key, err := es.ng.PutACL(newACL)
require.NoError(es.T(), err, "fail to PUT %+v", newACL)
deepEqual(es.T(), newACL, <-aclsUpserted)
assert.Nil(es.T(), es.ng.DelACL(key), "fail to DELETE %+v", newACL)
}
func (es *RouteLoaderSuite) TestWatchRoutesUpsertRoutesWhenRoutesUpdated() {
newACL := newTestACL("path")
updatedACL := newTestACL("header")
_, err := es.ng.PutACL(newACL)
aclsUpserted := make(chan *weaver.ACL, 1)
watchCtx, cancelWatch := context.WithCancel(context.Background())
defer cancelWatch()
go es.ng.WatchRoutes(watchCtx, genRouteProcessorMock(aclsUpserted), successUpsertRouteFunc)
time.Sleep(100 * time.Millisecond)
key, err := es.ng.PutACL(updatedACL)
require.NoError(es.T(), err, "fail to PUT %+v", updatedACL)
deepEqual(es.T(), updatedACL, <-aclsUpserted)
assert.Nil(es.T(), es.ng.DelACL(key), "fail to DELETE %+v", updatedACL)
}
func (es *RouteLoaderSuite) TestWatchRoutesDeleteRouteWhenARouteIsDeleted() {
newACL := newTestACL("path")
key, err := es.ng.PutACL(newACL)
require.NoError(es.T(), err, "fail to PUT ACL %+v", newACL)
aclsDeleted := make(chan *weaver.ACL, 1)
watchCtx, cancelWatch := context.WithCancel(context.Background())
defer cancelWatch()
go es.ng.WatchRoutes(watchCtx, successUpsertRouteFunc, genRouteProcessorMock(aclsDeleted))
time.Sleep(100 * time.Millisecond)
err = es.ng.DelACL(key)
require.NoError(es.T(), err, "fail to Delete %+v", newACL)
deepEqual(es.T(), newACL, <-aclsDeleted)
}
func newTestACL(matcher string) *weaver.ACL {
return &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && Path(`/ping`)",
EndpointConfig: &weaver.EndpointConfig{
ShardFunc: "lookup",
Matcher: matcher,
ShardExpr: "*",
ShardConfig: json.RawMessage(`{
"GF-": {
"backend_name": "foobar",
"backend": "http://customer-locations-primary"
},
"R-": {
"timeout": 100.0,
"backend_name": "foobar",
"backend": "http://customer-locations-secondary"
}
}`),
},
}
}
func genRouteProcessorMock(c chan *weaver.ACL) func(*weaver.ACL) error {
return func(acl *weaver.ACL) error {
c <- acl
return nil
}
}
func deepEqual(t *testing.T, expected *weaver.ACL, actual *weaver.ACL) {
assert.Equal(t, expected.ID, actual.ID)
assert.Equal(t, expected.Criterion, actual.Criterion)
assertEqualJSON(t, expected.EndpointConfig.ShardConfig, actual.EndpointConfig.ShardConfig)
assert.Equal(t, expected.EndpointConfig.ShardFunc, actual.EndpointConfig.ShardFunc)
assert.Equal(t, expected.EndpointConfig.Matcher, actual.EndpointConfig.Matcher)
assert.Equal(t, expected.EndpointConfig.ShardExpr, actual.EndpointConfig.ShardExpr)
}
func assertEqualJSON(t *testing.T, json1, json2 json.RawMessage) {
var jsonVal1 interface{}
var jsonVal2 interface{}
err1 := json.Unmarshal(json1, &jsonVal1)
err2 := json.Unmarshal(json2, &jsonVal2)
assert.NoError(t, err1, "failed to parse json string")
assert.NoError(t, err2, "failed to parse json string")
assert.True(t, reflect.DeepEqual(jsonVal1, jsonVal2))
}
func failingUpsertRouteFunc(acl *weaver.ACL) error {
return errors.New("error")
}
func successUpsertRouteFunc(acl *weaver.ACL) error {
return nil
}
================================================
FILE: examples/body_lookup/Dockerfile
================================================
FROM golang:1.11.5-alpine as base
ENV GO111MODULE off
RUN mkdir /estimate
ADD . /estimate
WORKDIR /estimate
RUN go build main.go
FROM alpine:latest
COPY --from=base /estimate/main /usr/local/bin/estimator
ENTRYPOINT ["estimator"]
================================================
FILE: examples/body_lookup/README.md
================================================
# Backend Lookup
In this example, will deploy etcd and weaver to kubernetes and apply a simple etcd to shard between Singapore estimator and Indonesian Estimator based on key body lookup.
### Setup
```
# To kubernetes cluster in local and set current context
minikube start
minikube status # verify it is up and running
# You can check dashboard by running following command
minikube dashboard
# Deploying helm components
helm init
```
### Deploying weaver
Now we have running kubernetes cluster in local. Let's deploy weaver and etcd in kubernetes to play with routes.
1. Clone the repo
2. On root folder of the project, run the following commands
```sh
# Connect to kubernetes docker image
eval $(minikube docker-env)
# Build docker weaver image
docker build . -t weaver:stable
# Deploy weaver to kubernetes
helm upgrade --debug --install proxy-cluster ./deployment/weaver --set service.type=NodePort -f ./deployment/weaver/values-env.yaml
```
We are setting service type as NodePort so that we can access it from local machine.
We have deployed weaver successfully to kubernetes under release name , you can check the same in dashboard.
### Deploying simple service
Now we have to deploy simple service to kubernetes and shard request using weaver.
Navigate to examples/body_lookup/ and run the following commands.
1. Build docker image for estimate service
2. Deploy docker image to 2 sharded clusters
```
# Building docker image for estimate
docker build . -t estimate:stable
# Deploying it Singapore Cluster
helm upgrade --debug --install singapore-cluster ./examples/body_lookup/estimator -f ./examples/body_lookup/estimator/values-sg.yaml
# Deploying it to Indonesian Cluster
helm upgrade --debug --install indonesia-cluster ./examples/body_lookup/estimator -f ./examples/body_lookup/estimator/values-id.yaml
```
We have a service called estimator which is sharded (Indonesian cluster, and Singapore cluster) which returns an Amount and Currency.
### Deploying Weaver ACLS
Let's deploy acl to etcd and check weaver in action.
1. Copy acls to weaver pod
2. Load acs to etcd
We have to apply acls to etcd so that we can lookup for that acl and load it. In order to apply a acl, first will copy to one of the pod
and deploy using curl request by issuing following commands.
```sh
# You can get pod name by running this command - kubectx get pods | grep weaver | awk '{print $1}'
kubectl cp examples/body_lookup/estimate_acl.json proxy-cluster-weaver-79fb49db6f-tng8r:/go/
# Set path in etcd using curl command
curl -v etcd:2379/v2/keys/weaver/acls/estimate/acl -XPUT --data-urlencode "value@estimate_acl.json"
```
Once we set the acl in etcd, as weaver is watching for path changes continuously it just loads the acl and starts sharding requests.
### Weaver in action
Now you have wevaer which is exposed using NodePort service type. This mean you can just shard your request based on currency lookup in body as we defined in the estimate_acl.json file.
1. Get Cluster Info
2. Send request to weaver to see response from estimator
```sh
# Get cluster ip from cluster-info
kubectl cluster-info
# Using cluster ip make a curl request to weaver
curl -X POST ${CLUSTER_IP}:${NODE_PORT}/estimate -d '{"currency": "SGD"}' # This is served by singapore shard
# {"Amount": 23.23, "Currency": "SGD"}
# Getting estimate from Indonesia shard
curl -X POST ${CLUSTER_IP}:${NODE_PORT}/estimate -d '{"currency": "IDR"}' # This is served by singapore shard
# {"Amount": 81223.23, "Currency": "IDR"}
```
================================================
FILE: examples/body_lookup/estimate_acl.json
================================================
{
"id": "estimator",
"criterion" : "Method(`POST`) && Path(`/estimate`)",
"endpoint" : {
"shard_expr": ".currency",
"matcher": "body",
"shard_func": "lookup",
"shard_config": {
"IDR": {
"backend_name": "indonesia-cluster",
"backend":"http://indonesia-cluster-estimator"
},
"SGD": {
"backend_name": "singapore-cluster",
"backend":"http://singapore-cluster-estimator"
}
}
}
}
================================================
FILE: examples/body_lookup/estimator/Chart.yaml
================================================
apiVersion: v1
appVersion: "1.0"
description: A Helm chart to deploy estimator
name: estimator
version: 0.1.0
maintainers:
- name: Gowtham Sai
email: dev@gowtham.me
================================================
FILE: examples/body_lookup/estimator/templates/_helpers.tpl
================================================
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "estimator.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "estimator.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "estimator.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
================================================
FILE: examples/body_lookup/estimator/templates/deployment.yaml
================================================
apiVersion: extensions/v1beta1
kind: Deployment
name: {{ template "estimator.fullname" . }}
metadata:
name: {{ template "estimator.fullname" . }}
labels:
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/name: {{ template "estimator.name" . }}
spec:
replicas: {{ .Values.replicaCount }}
template:
metadata:
{{- if .Values.podAnnotations }}
# Allows custom annotations to be specified
annotations:
{{- toYaml .Values.podAnnotations | indent 8 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "estimator.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
containers:
- name: {{ template "estimator.name" . }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.containerPort }}
protocol: TCP
env:
- name: "MAX_AMOUNT"
value: {{ .Values.env.maxAmount | quote }}
- name: "MIN_AMOUNT"
value: {{ .Values.env.minAmount | quote }}
- name: "CURRENCY"
value: {{ .Values.env.currency | quote }}
{{- if .Values.resources }}
resources:
# Minikube when high resource requests are specified by default.
{{- toYaml .Values.resources | indent 12 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
# Node selectors can be important on mixed Windows/Linux clusters.
{{- toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
================================================
FILE: examples/body_lookup/estimator/templates/service.yaml
================================================
apiVersion: v1
kind: Service
metadata:
{{- if .Values.service.annotations }}
annotations:
{{- toYaml .Values.service.annotations | indent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "estimator.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: server
app.kubernetes.io/part-of: estimator
name: {{ template "estimator.fullname" . }}
spec:
# Provides options for the service so chart users have the full choice
type: "{{ .Values.service.type }}"
clusterIP: "{{ .Values.service.clusterIP }}"
{{- if .Values.service.externalIPs }}
externalIPs:
{{- toYaml .Values.service.externalIPs | indent 4 }}
{{- end }}
{{- if .Values.service.loadBalancerIP }}
loadBalancerIP: "{{ .Values.service.loadBalancerIP }}"
{{- end }}
{{- if .Values.service.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{- toYaml .Values.service.loadBalancerSourceRanges | indent 4 }}
{{- end }}
ports:
- name: http
port: {{ .Values.service.port }}
protocol: TCP
targetPort: {{ .Values.service.containerPort }}
{{- if (and (eq .Values.service.type "NodePort") (not (empty .Values.service.nodePort))) }}
nodePort: {{ .Values.service.nodePort }}
{{- end }}
selector:
app.kubernetes.io/name: {{ template "estimator.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
================================================
FILE: examples/body_lookup/estimator/values-id.yaml
================================================
# Default env values for Indonesia cluster
env:
maxAmount: 100000
minAmount: 1000
currency: IDR
================================================
FILE: examples/body_lookup/estimator/values-sg.yaml
================================================
# Default env values for Singapore cluster
env:
maxAmount: 100
minAmount: 10
currency: SGD
================================================
FILE: examples/body_lookup/estimator/values.yaml
================================================
# Default values for weaver.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 2
image:
repository: estimator
tag: stable
pullPolicy: IfNotPresent
nameOverride: ""
fullnameOverride: ""
service:
type: ClusterIP
port: 80
containerPort: 8080
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}
env:
maxAmount: "100"
minAmount: "10"
currency: "SGD"
================================================
FILE: examples/body_lookup/main.go
================================================
package main
import (
"encoding/json"
"fmt"
"log"
"math"
"math/rand"
"net/http"
"os"
"strconv"
)
type estimationRequest struct {
Amount float64
Currency string
}
func getEnv(key string, fallback string) string {
if value, ok := os.LookupEnv(key); ok {
return value
}
return fallback
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi there, welcome to weaver!")
}
func handlePing(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "pong")
}
func getAmount() float64 {
maxCap, _ := strconv.ParseFloat(getEnv("MAX_AMOUNT", "100"), 64)
minCap, _ := strconv.ParseFloat(getEnv("MIN_AMOUNT", "100"), 64)
randomValue := minCap + rand.Float64()*(maxCap-minCap)
return math.Round(randomValue*100) / 100
}
func handleEstimate(w http.ResponseWriter, r *http.Request) {
estimatedValue := estimationRequest{Amount: getAmount(), Currency: getEnv("CURRENCY", "IDR")}
respEncoder := json.NewEncoder(w)
respEncoder.Encode(estimatedValue)
}
func main() {
http.HandleFunc("/", handler)
http.HandleFunc("/ping", handlePing)
http.HandleFunc("/estimate", handleEstimate)
log.Fatal(http.ListenAndServe(":8080", nil))
}
================================================
FILE: go.mod
================================================
module github.com/gojektech/weaver
require (
github.com/certifi/gocertifi v0.0.0-20170123212243-03be5e6bb987 // indirect
github.com/coreos/bbolt v1.3.2 // indirect
github.com/coreos/etcd v3.3.0+incompatible
github.com/coreos/go-semver v0.2.0 // indirect
github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142 // indirect
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect
github.com/getsentry/raven-go v0.0.0-20161115135411-3f7439d3e74d
github.com/ghodss/yaml v1.0.0 // indirect
github.com/gogo/protobuf v1.2.0 // indirect
github.com/gojekfarm/hashring v0.0.0-20180330151038-7bba2fd52501
github.com/golang/geo v0.0.0-20170430223333-5747e9816367
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef // indirect
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c // indirect
github.com/gorilla/websocket v1.4.0 // indirect
github.com/gravitational/trace v0.0.0-20171118015604-0bd13642feb8 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.7.0 // indirect
github.com/hashicorp/hcl v0.0.0-20170217164738-630949a3c5fa // indirect
github.com/jonboulle/clockwork v0.1.0 // indirect
github.com/kr/pretty v0.1.0 // indirect
github.com/magiconair/properties v0.0.0-20170113111004-b3b15ef068fd // indirect
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 // indirect
github.com/mitchellh/mapstructure v0.0.0-20170125051937-db1efb556f84 // indirect
github.com/newrelic/go-agent v1.11.0
github.com/onsi/ginkgo v1.7.0 // indirect
github.com/onsi/gomega v1.4.3 // indirect
github.com/pelletier/go-buffruneio v0.2.0 // indirect
github.com/pelletier/go-toml v0.0.0-20170227222904-361678322880 // indirect
github.com/philhofer/fwd v1.0.0 // indirect
github.com/pkg/errors v0.8.0
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/pquerna/ffjson v0.0.0-20181028064349-e517b90714f7 // indirect
github.com/prometheus/client_golang v0.9.2 // indirect
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 // indirect
github.com/savaki/jq v0.0.0-20161209013833-0e6baecebbf8
github.com/sirupsen/logrus v1.0.3
github.com/soheilhy/cmux v0.1.4 // indirect
github.com/spaolacci/murmur3 v0.0.0-20170819071325-9f5d223c6079 // indirect
github.com/spf13/afero v0.0.0-20170217164146-9be650865eab // indirect
github.com/spf13/cast v0.0.0-20170221152302-f820543c3592 // indirect
github.com/spf13/jwalterweatherman v0.0.0-20170109133355-fa7ca7e836cf // indirect
github.com/spf13/pflag v1.0.0 // indirect
github.com/spf13/viper v1.0.0
github.com/stretchr/objx v0.1.1 // indirect
github.com/stretchr/testify v1.2.2
github.com/tinylib/msgp v1.1.0 // indirect
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 // indirect
github.com/ugorji/go v0.0.0-20171019201919-bdcc60b419d1 // indirect
github.com/vulcand/predicate v1.0.0 // indirect
github.com/vulcand/route v0.0.0-20160805191529-61904570391b
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
go.etcd.io/bbolt v1.3.2 // indirect
golang.org/x/crypto v0.0.0-20190131182504-b8fe1690c613 // indirect
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c // indirect
google.golang.org/grpc v1.18.0 // indirect
gopkg.in/airbrake/gobrake.v2 v2.0.9 // indirect
gopkg.in/alexcesaro/statsd.v2 v2.0.0
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect
gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2 // indirect
gopkg.in/urfave/cli.v1 v1.20.0
)
================================================
FILE: go.sum
================================================
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/certifi/gocertifi v0.0.0-20170123212243-03be5e6bb987 h1:GMSZ85uysw01MMLfnHGjTj/QfUdJcGHuDabY6kWKnVk=
github.com/certifi/gocertifi v0.0.0-20170123212243-03be5e6bb987/go.mod h1:GJKEexRPVJrBSOjoqN5VNOIKJ5Q3RViH6eu3puDRwx4=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/coreos/bbolt v1.3.2 h1:wZwiHHUieZCquLkDL0B8UhzreNWsPHooDAG3q34zk0s=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v3.3.0+incompatible h1:v3H7yHgF+94suF7Xg6V7Haq6Anac3X6WosuKGTTJCGM=
github.com/coreos/etcd v3.3.0+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-semver v0.2.0 h1:3Jm3tLmsgAYcjC+4Up7hJrFBPr+n7rAqYeSw/SZazuY=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142 h1:3jFq2xL4ZajGK4aZY8jz+DAF0FHjI51BXjjSwCzS1Dk=
github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbpBpLoyyu8B6e44T7hJy6potg=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/getsentry/raven-go v0.0.0-20161115135411-3f7439d3e74d h1:l+MZegqjcffeVt3U7OldySISIA+wDlizPTz9Ki2u3k4=
github.com/getsentry/raven-go v0.0.0-20161115135411-3f7439d3e74d/go.mod h1:KungGk8q33+aIAZUIVWZDr2OfAEBsO49PX4NzFV5kcQ=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gogo/protobuf v1.2.0 h1:xU6/SpYbvkNYiptHJYEDRseDLvYE7wSqhYYNy0QSUzI=
github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gojekfarm/hashring v0.0.0-20180330151038-7bba2fd52501 h1:M711duYkMPGeglzA822WpcmfuLES3eafw7zH7ch+JxM=
github.com/gojekfarm/hashring v0.0.0-20180330151038-7bba2fd52501/go.mod h1:OiCsMsLqQGrKJrRQdHIK8PyJXCv7yo73ZeBweRM4u0w=
github.com/golang/geo v0.0.0-20170430223333-5747e9816367 h1:vfvm90sLVQQU3gbQ+EpAF/Y9SFNvjqCxkyy92aXMnK0=
github.com/golang/geo v0.0.0-20170430223333-5747e9816367/go.mod h1:vgWZ7cu0fq0KY3PpEHsocXOWJpRtkcbKemU4IUw0M60=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef h1:veQD95Isof8w9/WXiA+pa3tz3fJXkt5B7QaRBrM62gk=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c h1:964Od4U6p2jUkFxvCydnIczKteheJEzHRToSGK3Bnlw=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/gravitational/trace v0.0.0-20171118015604-0bd13642feb8 h1:beahEEOlfVHRfa7JFDl3oetCjvCga+p7iU+5RN81evY=
github.com/gravitational/trace v0.0.0-20171118015604-0bd13642feb8/go.mod h1:RvdOUHE4SHqR3oXlFFKnGzms8a5dugHygGw1bqDstYI=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.7.0 h1:tPFY/SM+d656aSgLWO2Eckc3ExwpwwybwdN5Ph20h1A=
github.com/grpc-ecosystem/grpc-gateway v1.7.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/hashicorp/hcl v0.0.0-20170217164738-630949a3c5fa h1:10wM7X2JKPrmcvtI9Qy2xsoQI1CBA8dd6LqjyGKlD0c=
github.com/hashicorp/hcl v0.0.0-20170217164738-630949a3c5fa/go.mod h1:oZtUIOe8dh44I2q6ScRibXws4Ajl+d+nod3AaR9vL5w=
github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/jonboulle/clockwork v0.1.0 h1:VKV+ZcuP6l3yW9doeqz6ziZGgcynBVQO+obU0+0hcPo=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/magiconair/properties v0.0.0-20170113111004-b3b15ef068fd h1:iWbe8Xk8p4yQR6ZVjhS21WRnSM79D/l7YN1mOCdM/wQ=
github.com/magiconair/properties v0.0.0-20170113111004-b3b15ef068fd/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329 h1:2gxZ0XQIU/5z3Z3bUBu+FXuk2pFbkN6tcwi/pjyaDic=
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/mitchellh/mapstructure v0.0.0-20170125051937-db1efb556f84 h1:rrg06yhhsqEELubsnYWqadxdi0CYJ97s899oUXDIrkY=
github.com/mitchellh/mapstructure v0.0.0-20170125051937-db1efb556f84/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/newrelic/go-agent v1.11.0 h1:jnd8+H6dB+93UTJHFT1wJoij5spKNN/xZ0nkw0kvt7o=
github.com/newrelic/go-agent v1.11.0/go.mod h1:a8Fv1b/fYhFSReoTU6HDkTYIMZeSVNffmoS726Y0LzQ=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0 h1:WSHQ+IS43OoUrWtD1/bbclrwK8TTH5hzp+umCiuxHgs=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v1.4.3 h1:RE1xgDvH7imwFD45h+u2SgIfERHlS2yNG4DObb5BSKU=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/pelletier/go-buffruneio v0.2.0 h1:U4t4R6YkofJ5xHm3dJzuRpPZ0mr5MMCoAWooScCR7aA=
github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
github.com/pelletier/go-toml v0.0.0-20170227222904-361678322880 h1:3UCAtS/p4J0cFiubq1hFsel1jlSxSp6SZEZoAFUREh8=
github.com/pelletier/go-toml v0.0.0-20170227222904-361678322880/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/philhofer/fwd v1.0.0 h1:UbZqGr5Y38ApvM/V/jEljVxwocdweyH+vmYvRPBnbqQ=
github.com/philhofer/fwd v1.0.0/go.mod h1:gk3iGcWd9+svBvR0sR+KPcfE+RNWozjowpeBVG3ZVNU=
github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pquerna/ffjson v0.0.0-20181028064349-e517b90714f7 h1:gGBSHPOU7g8YjTbhwn+lvFm2VDEhhA+PwDIlstkgSxE=
github.com/pquerna/ffjson v0.0.0-20181028064349-e517b90714f7/go.mod h1:YARuvh7BUWHNhzDq2OM5tzR2RiCcN2D7sapiKyCel/M=
github.com/prometheus/client_golang v0.9.2 h1:awm861/B8OKDd2I/6o1dy3ra4BamzKhYOiGItCeZ740=
github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 h1:S/YWwWx/RA8rT8tKFRuGUZhuA90OyIBpPCXkcbwU8DE=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20181126121408-4724e9255275 h1:PnBWHBf+6L0jOqq0gIVUe6Yk0/QMZ640k6NvkxcBf+8=
github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a h1:9a8MnZMP0X2nLJdBg+pBmGgkJlSaKC2KaQmTCk1XDtE=
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/savaki/jq v0.0.0-20161209013833-0e6baecebbf8 h1:ajJQhvqPSQFJJ4aV5mDAMx8F7iFi6Dxfo6y62wymLNs=
github.com/savaki/jq v0.0.0-20161209013833-0e6baecebbf8/go.mod h1:Nw/CCOXNyF5JDd6UpYxBwG5WWZ2FOJ/d5QnXL4KQ6vY=
github.com/sirupsen/logrus v1.0.3 h1:B5C/igNWoiULof20pKfY4VntcIPqKuwEmoLZrabbUrc=
github.com/sirupsen/logrus v1.0.3/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/spaolacci/murmur3 v0.0.0-20170819071325-9f5d223c6079 h1:lDiM+yMjW7Ork8mhl0YN0qO1Z02qGoe1vwzGc1gP/8U=
github.com/spaolacci/murmur3 v0.0.0-20170819071325-9f5d223c6079/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v0.0.0-20170217164146-9be650865eab h1:IVAbBHQR8rXL2Fc8Zba/lMF7KOnTi70lqdx91UTuAwQ=
github.com/spf13/afero v0.0.0-20170217164146-9be650865eab/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/cast v0.0.0-20170221152302-f820543c3592 h1:xwZ8A+Sp00knVqYp3nzyQJ931wd7IVQGan4Iur2QpX8=
github.com/spf13/cast v0.0.0-20170221152302-f820543c3592/go.mod h1:r2rcYCSwa1IExKTDiTfzaxqT2FNHs8hODu4LnUfgKEg=
github.com/spf13/jwalterweatherman v0.0.0-20170109133355-fa7ca7e836cf h1:F3R4gmObzwZfjwH3hCs9WIyyTPjL5yC/cfdsW5hORhI=
github.com/spf13/jwalterweatherman v0.0.0-20170109133355-fa7ca7e836cf/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
github.com/spf13/pflag v1.0.0 h1:oaPbdDe/x0UncahuwiPxW1GYJyilRAdsPnq3e1yaPcI=
github.com/spf13/pflag v1.0.0/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/viper v1.0.0 h1:RUA/ghS2i64rlnn4ydTfblY8Og8QzcPtCcHvgMn+w/I=
github.com/spf13/viper v1.0.0/go.mod h1:A8kyI5cUJhb8N+3pkfONlcEcZbueH6nhAm0Fq7SrnBM=
github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/tinylib/msgp v1.1.0 h1:9fQd+ICuRIu/ue4vxJZu6/LzxN0HwMds2nq/0cFvxHU=
github.com/tinylib/msgp v1.1.0/go.mod h1:+d+yLhGm8mzTaHzB+wgMYrodPfmZrzkirds8fDWklFE=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 h1:LnC5Kc/wtumK+WB441p7ynQJzVuNRJiqddSIE3IlSEQ=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v0.0.0-20171019201919-bdcc60b419d1 h1:UvhxfNjNqlZ/x3cDyqxMhoiUpemd3zXkVQApN6bM/lg=
github.com/ugorji/go v0.0.0-20171019201919-bdcc60b419d1/go.mod h1:hnLbHMwcvSihnDhEfx2/BzKp2xb0Y+ErdfYcrs9tkJQ=
github.com/vulcand/predicate v1.0.0 h1:c5lVsC9SKrQjdWNwTTG3RkADPKhSw1SrUZWq6LJL21k=
github.com/vulcand/predicate v1.0.0/go.mod h1:mlccC5IRBoc2cIFmCB8ZM62I3VDb6p2GXESMHa3CnZg=
github.com/vulcand/route v0.0.0-20160805191529-61904570391b h1:eCB3pa/SYYqLuajbklwy+mJAYNU1U8JQLv3M9gwKSeg=
github.com/vulcand/route v0.0.0-20160805191529-61904570391b/go.mod h1:Pn2LM+/AaNyDRnlxKzatwCJiGBR/ZnRILFto79oYeUg=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
go.etcd.io/bbolt v1.3.2 h1:Z/90sZLPOeCy2PwprqkFa25PdkusRzaj9P8zm/KNyvk=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
golang.org/x/crypto v0.0.0-20190131182504-b8fe1690c613 h1:MQ/ZZiDsUapFFiMS+vzwXkCTeEKaum+Do5rINYJDmxc=
golang.org/x/crypto v0.0.0-20190131182504-b8fe1690c613/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc h1:a3CU5tJYVj92DY2LaA1kUkrsqD5/3mLDhx2NcNqyW+0=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f h1:Bl/8QSvNqXvPGPGXa2z5xUTmV7VDcZyvRZ+QQXkXTZQ=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e h1:o3PsSEY8E4eXWkXrIP9YJALUkVZqzHJT5DOasTyn8Vs=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c h1:fqgJT0MGcGpPgpWU7VRdRjuArfcOvC4AoJmILihzhDg=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0NQvRW8DG4Yk3Q6T9cu9RcFQDu1tc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/grpc v1.18.0 h1:IZl7mfBGfbhYx2p2rKRtYgDFw6SBz+kclmxYrCksPPA=
google.golang.org/grpc v1.18.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
gopkg.in/airbrake/gobrake.v2 v2.0.9 h1:7z2uVWwn7oVeeugY1DtlPAy5H+KYgB1KeKTnqjNatLo=
gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
gopkg.in/alexcesaro/statsd.v2 v2.0.0 h1:FXkZSCZIH17vLCO5sO2UucTHsH9pc+17F6pl3JVCwMc=
gopkg.in/alexcesaro/statsd.v2 v2.0.0/go.mod h1:i0ubccKGzBVNBpdGV5MocxyA/XlLUJzA7SLonnE4drU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2 h1:OAj3g0cR6Dx/R07QgQe8wkA9RNjB2u4i700xBkIT4e0=
gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2/go.mod h1:Xk6kEKp8OKb+X14hQBKWaSkCsqBpgog8nAV2xsGOxlo=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/urfave/cli.v1 v1.20.0 h1:NdAVW6RYxDif9DhDHaAortIu956m2c0v+09AZBPTbE0=
gopkg.in/urfave/cli.v1 v1.20.0/go.mod h1:vuBzUtMdQeixQj8LVd+/98pzhxNGQoyuPBlsXHOQNO0=
gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
================================================
FILE: goreleaser.yml
================================================
builds:
- main: ./cmd/weaver-server/
goos:
- linux
- darwin
- windows
goarch:
- amd64
- 386
archive:
replacements:
amd64: 64-bit
386: 32-bit
darwin: macOS
format: zip
================================================
FILE: pkg/instrumentation/newrelic.go
================================================
package instrumentation
import (
"context"
"log"
"net/http"
"time"
"github.com/gojektech/weaver/config"
newrelic "github.com/newrelic/go-agent"
)
type ctxKey int
const txKey ctxKey = 0
var newRelicApp newrelic.Application
func InitNewRelic() newrelic.Application {
cfg := config.NewRelicConfig()
if cfg.Enabled {
app, err := newrelic.NewApplication(cfg)
if err != nil {
log.Fatalf(err.Error())
}
newRelicApp = app
}
return newRelicApp
}
func ShutdownNewRelic() {
if config.NewRelicConfig().Enabled {
newRelicApp.Shutdown(time.Second)
}
}
func NewRelicApp() newrelic.Application {
return newRelicApp
}
func StartRedisSegmentNow(op string, coll string, txn newrelic.Transaction) newrelic.DatastoreSegment {
s := newrelic.DatastoreSegment{
Product: newrelic.DatastoreRedis,
Collection: coll,
Operation: op,
}
s.StartTime = newrelic.StartSegmentNow(txn)
return s
}
func NewContext(ctx context.Context, w http.ResponseWriter) context.Context {
if config.NewRelicConfig().Enabled {
tx, ok := w.(newrelic.Transaction)
if !ok {
return ctx
}
return context.WithValue(ctx, txKey, tx)
}
return ctx
}
func NewContextWithTransaction(ctx context.Context, tx newrelic.Transaction) context.Context {
return context.WithValue(ctx, txKey, tx)
}
func GetTx(ctx context.Context) (newrelic.Transaction, bool) {
tx, ok := ctx.Value(txKey).(newrelic.Transaction)
return tx, ok
}
================================================
FILE: pkg/instrumentation/statsd.go
================================================
package instrumentation
import (
"fmt"
"net/http"
"time"
"github.com/gojektech/weaver/config"
"github.com/gojektech/weaver/pkg/logger"
statsd "gopkg.in/alexcesaro/statsd.v2"
)
var statsD *statsd.Client
func InitiateStatsDMetrics() error {
statsDConfig := config.StatsD()
if statsDConfig.Enabled() {
flushPeriod := time.Duration(statsDConfig.FlushPeriodInSeconds()) * time.Second
address := fmt.Sprintf("%s:%d", statsDConfig.Host(), statsDConfig.Port())
var err error
statsD, err = statsd.New(statsd.Address(address),
statsd.Prefix(statsDConfig.Prefix()), statsd.FlushPeriod(flushPeriod))
if err != nil {
logger.Errorf("StatsD: Error initiating client %s", err)
return err
}
logger.Infof("StatsD: Sending metrics")
}
return nil
}
func StatsDClient() *statsd.Client {
return statsD
}
func CloseStatsDClient() {
if statsD != nil {
logger.Infof("StatsD: Shutting down")
statsD.Close()
}
}
func NewTiming() statsd.Timing {
if statsD != nil {
return statsD.NewTiming()
}
return statsd.Timing{}
}
func IncrementTotalRequestCount() {
incrementProbe("request.total.count")
}
func IncrementAPIRequestCount(apiName string) {
incrementProbe(fmt.Sprintf("request.api.%s.count", apiName))
}
func IncrementAPIStatusCount(apiName string, httpStatusCode int) {
incrementProbe(fmt.Sprintf("request.api.%s.status.%d.count", apiName, httpStatusCode))
}
func IncrementAPIBackendRequestCount(apiName, backendName string) {
incrementProbe(fmt.Sprintf("request.api.%s.backend.%s.count", apiName, backendName))
}
func IncrementAPIBackendStatusCount(apiName, backendName string, httpStatusCode int) {
incrementProbe(fmt.Sprintf("request.api.%s.backend.%s.status.%d.count", apiName, backendName, httpStatusCode))
}
func IncrementCrashCount() {
incrementProbe("request.internal.crash.count")
}
func IncrementNotFound() {
incrementProbe(fmt.Sprintf("request.internal.%d.count", http.StatusNotFound))
}
func IncrementInternalAPIStatusCount(aclName string, statusCode int) {
incrementProbe(fmt.Sprintf("request.api.%s.internal.status.%d.count", aclName, statusCode))
}
func TimeTotalLatency(timing statsd.Timing) {
if statsD != nil {
timing.Send("request.time.total")
}
return
}
func TimeAPILatency(apiName string, timing statsd.Timing) {
if statsD != nil {
timing.Send(fmt.Sprintf("request.api.%s.time.total", apiName))
}
return
}
func TimeAPIBackendLatency(apiName, backendName string, timing statsd.Timing) {
if statsD != nil {
timing.Send(fmt.Sprintf("request.api.%s.backend.%s.time.total", apiName, backendName))
}
return
}
func incrementProbe(key string) {
if statsD == nil {
return
}
go statsD.Increment(key)
}
================================================
FILE: pkg/logger/logger.go
================================================
package logger
import (
"net/http"
"os"
"github.com/gojektech/weaver/config"
"github.com/gojektech/weaver/pkg/util"
"github.com/sirupsen/logrus"
)
var logger *logrus.Logger
func SetupLogger() {
level, err := logrus.ParseLevel(config.LogLevel())
if err != nil {
level = logrus.WarnLevel
}
logger = &logrus.Logger{
Out: os.Stdout,
Hooks: make(logrus.LevelHooks),
Level: level,
Formatter: &logrus.JSONFormatter{},
}
}
func AddHook(hook logrus.Hook) {
logger.Hooks.Add(hook)
}
func Debug(args ...interface{}) {
logger.Debug(args...)
}
func Debugf(format string, args ...interface{}) {
logger.Debugf(format, args...)
}
func Debugln(args ...interface{}) {
logger.Debugln(args...)
}
func Debugrf(r *http.Request, format string, args ...interface{}) {
httpRequestLogEntry(r).Debugf(format, args...)
}
func Error(args ...interface{}) {
logger.Error(args...)
}
func Errorf(format string, args ...interface{}) {
logger.Errorf(format, args...)
}
func Errorln(args ...interface{}) {
logger.Errorln(args...)
}
func Errorrf(r *http.Request, format string, args ...interface{}) {
httpRequestLogEntry(r).Errorf(format, args...)
}
func ErrorWithFieldsf(fields logrus.Fields, format string, args ...interface{}) {
logger.WithFields(fields).Errorf(format, args...)
}
func Fatal(args ...interface{}) {
logger.Fatal(args...)
}
func Fatalf(format string, args ...interface{}) {
logger.Fatalf(format, args...)
}
func Fatalln(args ...interface{}) {
logger.Fatalln(args...)
}
func Info(args ...interface{}) {
logger.Info(args...)
}
func Infof(format string, args ...interface{}) {
logger.Infof(format, args...)
}
func Infoln(args ...interface{}) {
logger.Infoln(args...)
}
func Inforf(r *http.Request, format string, args ...interface{}) {
httpRequestLogEntry(r).Infof(format, args...)
}
func InfoWithFieldsf(fields logrus.Fields, format string, args ...interface{}) {
logger.WithFields(fields).Infof(format, args...)
}
func ProxyInfo(aclName string, downstreamHost string, r *http.Request, responseStatus int, rw http.ResponseWriter) {
logger.WithFields(logrus.Fields{
"type": "proxy",
"downstream_host": downstreamHost,
"api_name": aclName,
"request": httpRequestFields(r),
"response": httpResponseFields(responseStatus, rw),
}).Info("proxy")
}
func httpRequestFields(r *http.Request) logrus.Fields {
requestHeaders := map[string]string{}
for k := range r.Header {
normalizedKey := util.ToSnake(k)
if normalizedKey == "authorization" {
continue
}
requestHeaders[normalizedKey] = r.Header.Get(k)
}
return logrus.Fields{
"uri": r.URL.String(),
"query": r.URL.Query(),
"method": r.Method,
"headers": requestHeaders,
}
}
func httpResponseFields(responseStatus int, rw http.ResponseWriter) logrus.Fields {
responseHeaders := map[string]string{}
for k := range rw.Header() {
responseHeaders[util.ToSnake(k)] = rw.Header().Get(k)
}
return logrus.Fields{
"status": responseStatus,
"headers": responseHeaders,
}
}
func Warn(args ...interface{}) {
logger.Warn(args...)
}
func Warnf(format string, args ...interface{}) {
logger.Warnf(format, args...)
}
func Warnln(args ...interface{}) {
logger.Warnln(args...)
}
func WithField(key string, value interface{}) *logrus.Entry {
return logger.WithField(key, value)
}
func WithFields(fields logrus.Fields) *logrus.Entry {
return logger.WithFields(fields)
}
func httpRequestLogEntry(r *http.Request) *logrus.Entry {
return logger.WithFields(logrus.Fields{
"request_method": r.Method,
"request_host": r.Host,
"request_url": r.URL.String(),
})
}
================================================
FILE: pkg/matcher/matcher.go
================================================
package matcher
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"regexp"
"strconv"
"strings"
"github.com/pkg/errors"
"github.com/savaki/jq"
)
func New(matcherName string) (MatcherFunc, bool) {
mf, found := matcherMux[matcherName]
return mf, found
}
type MatcherFunc func(request *http.Request, shardExpr string) (shardKey string, err error)
var matcherMux = map[string]MatcherFunc{
"header": func(req *http.Request, expr string) (string, error) {
return req.Header.Get(expr), nil
},
"multi-headers": func(req *http.Request, expr string) (string, error) {
headers := strings.Split(expr, ",")
var headerValues strings.Builder
headersCount := len(headers)
if headersCount == 0 {
return "", nil
}
for idx, header := range headers {
headerValue := req.Header.Get(header)
headerValues.Grow(len(headerValue))
headerValues.WriteString(headerValue)
if (idx + 1) != headersCount {
headerValues.Grow(1)
headerValues.WriteString(",")
}
}
return headerValues.String(), nil
},
"param": func(req *http.Request, expr string) (string, error) {
return req.URL.Query().Get(expr), nil
},
"path": func(req *http.Request, expr string) (string, error) {
rex := regexp.MustCompile(expr)
match := rex.FindStringSubmatch(req.URL.Path)
if len(match) == 0 {
return "", fmt.Errorf("no match found for expr: %s", expr)
}
return match[1], nil
},
"body": func(req *http.Request, expr string) (string, error) {
requestBody, err := ioutil.ReadAll(req.Body)
if err != nil {
return "", errors.Wrapf(err, "failed to read request body for expr: %s", expr)
}
req.Body = ioutil.NopCloser(bytes.NewBuffer(requestBody))
var bodyKey interface{}
op, err := jq.Parse(expr)
if err != nil {
return "", errors.Wrapf(err, "failed to parse shard expr: %s", expr)
}
key, err := op.Apply(requestBody)
if err != nil {
return "", errors.Wrapf(err, "failed to apply parsed shard expr: %s", expr)
}
if err := json.Unmarshal(key, &bodyKey); err != nil {
return "", errors.Wrapf(err, "failed to unmarshal data for shard expr: %s", expr)
}
switch v := bodyKey.(type) {
case string:
return v, nil
case float64:
return strconv.FormatFloat(v, 'f', -1, 64), nil
default:
return "", errors.New("failed to type assert bodyKey")
}
},
}
================================================
FILE: pkg/matcher/matcher_test.go
================================================
package matcher
import (
"bytes"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestBodyMatcher(t *testing.T) {
body := bytes.NewReader([]byte(`{ "drivers": { "id": "123", "name": "hello"} }`))
req := httptest.NewRequest("GET", "/drivers", body)
expr := ".drivers.id"
key, err := matcherMux["body"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "123", key)
}
func TestBodyMatcherParseInt(t *testing.T) {
body := bytes.NewReader([]byte(`{ "routeRequests": [{ "id": "123", "serviceType": 1}] }`))
req := httptest.NewRequest("GET", "/drivers", body)
expr := ".routeRequests.[0].serviceType"
key, err := matcherMux["body"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "1", key)
}
func TestBodyMatcherParseTypeAssertFail(t *testing.T) {
body := bytes.NewReader([]byte(`{ "routeRequests": [{ "id": "123", "serviceType": []}] }`))
req := httptest.NewRequest("GET", "/drivers", body)
expr := ".routeRequests.[0].serviceType"
key, err := matcherMux["body"](req, expr)
require.Error(t, err, "should have failed to match a key")
require.Equal(t, "", key)
assert.Equal(t, "failed to type assert bodyKey", err.Error())
}
func TestBodyMatcherFail(t *testing.T) {
body := bytes.NewReader([]byte(`{ "drivers": { "id": "123", "name": "hello"} }`))
req := httptest.NewRequest("GET", "/drivers", body)
expr := ".drivers.blah"
key, err := matcherMux["body"](req, expr)
require.Error(t, err, "should have failed to match a key")
assert.Equal(t, "", key)
}
func TestHeaderMatcher(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
req.Header.Add("Hello", "World")
expr := "Hello"
key, err := matcherMux["header"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "World", key)
}
func TestHeadersCsvMatcherWithSingleHeader(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
req.Header.Add("H1", "One")
req.Header.Add("H2", "Two")
req.Header.Add("H3", "Three")
expr := "H2"
key, err := matcherMux["multi-headers"](req, expr)
require.NoError(t, err, "should not have failed to extract headers")
assert.Equal(t, "Two", key)
}
func TestHeadersCsvMatcherWithSingleHeaderWhenNoneArePresent(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
expr := "H1"
key, err := matcherMux["multi-headers"](req, expr)
require.NoError(t, err, "should not have failed to extract headers")
assert.Equal(t, "", key)
}
func TestHeadersCsvMatcherWithZeroHeaders(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
req.Header.Add("H1", "One")
req.Header.Add("H2", "Two")
req.Header.Add("H3", "Three")
expr := ""
key, err := matcherMux["multi-headers"](req, expr)
require.NoError(t, err, "should not have failed to extract headers")
assert.Equal(t, "", key)
}
func TestHeadersCsvMatcherWithMultipleHeaders(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
req.Header.Add("H1", "One")
req.Header.Add("H2", "Two")
req.Header.Add("H3", "Three")
expr := "H1,H3"
key, err := matcherMux["multi-headers"](req, expr)
require.NoError(t, err, "should not have failed to extract headers")
assert.Equal(t, "One,Three", key)
}
func TestHeadersCsvMatcherWithMultipleHeadersWhenSomeArePresent(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
req.Header.Add("H3", "Three")
expr := "H1,H3"
key, err := matcherMux["multi-headers"](req, expr)
require.NoError(t, err, "should not have failed to extract headers")
assert.Equal(t, ",Three", key)
}
func TestHeadersCsvMatcherWithMultipleHeadersWhenNoneArePresent(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
expr := "H1,H3"
key, err := matcherMux["multi-headers"](req, expr)
require.NoError(t, err, "should not have failed to extract headers")
assert.Equal(t, ",", key)
}
func TestHeaderMatcherFail(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers", nil)
expr := "Hello"
key, err := matcherMux["header"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "", key)
}
func TestParamMatcher(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers?url=blah", nil)
expr := "url"
key, err := matcherMux["param"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "blah", key)
}
func TestParamMatcherFail(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers?url=blah", nil)
expr := "hello"
key, err := matcherMux["param"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "", key)
}
func TestPathMatcher(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers/123", nil)
expr := `/drivers/(\d+)`
key, err := matcherMux["path"](req, expr)
require.NoError(t, err, "should not have failed to match a key")
assert.Equal(t, "123", key)
}
func TestPathMatcherFail(t *testing.T) {
req := httptest.NewRequest("GET", "/drivers/123", nil)
expr := `/drivers/blah`
key, err := matcherMux["path"](req, expr)
require.Error(t, err, "should have failed to match a key")
assert.Equal(t, "", key)
}
================================================
FILE: pkg/shard/domain.go
================================================
package shard
import (
"fmt"
"time"
"github.com/gojektech/weaver"
"github.com/gojektech/weaver/config"
"github.com/pkg/errors"
)
type CustomError struct {
ExitMessage string
}
func (e *CustomError) Error() string {
return fmt.Sprintf("[error] %s", e.ExitMessage)
}
func Error(msg string) error {
return &CustomError{msg}
}
type BackendDefinition struct {
BackendName string `json:"backend_name"`
BackendURL string `json:"backend"`
Timeout *float64 `json:"timeout,omitempty"`
}
func (bd BackendDefinition) Validate() error {
if bd.BackendName == "" {
return errors.WithStack(fmt.Errorf("missing backend name in shard config: %+v", bd))
}
if bd.BackendURL == "" {
return errors.WithStack(fmt.Errorf("missing backend url in shard config: %+v", bd))
}
return nil
}
func toBackends(shardConfig map[string]BackendDefinition) (map[string]*weaver.Backend, error) {
backends := map[string]*weaver.Backend{}
for key, backendDefinition := range shardConfig {
if err := backendDefinition.Validate(); err != nil {
return nil, errors.Wrapf(err, "failed to validate backend definition")
}
backend, err := parseBackend(backendDefinition)
if err != nil {
return nil, errors.Wrapf(err, "failed to parseBackends from backendDefinition")
}
backends[key] = backend
}
return backends, nil
}
func parseBackend(shardConfig BackendDefinition) (*weaver.Backend, error) {
timeoutInDuration := config.Proxy().ProxyDialerTimeoutInMS()
if shardConfig.Timeout != nil {
timeoutInDuration = time.Duration(*shardConfig.Timeout)
}
backendOptions := weaver.BackendOptions{
Timeout: timeoutInDuration * time.Millisecond,
}
return weaver.NewBackend(shardConfig.BackendName, shardConfig.BackendURL, backendOptions)
}
================================================
FILE: pkg/shard/hashring.go
================================================
package shard
import (
"encoding/json"
"fmt"
"regexp"
"strconv"
"github.com/gojekfarm/hashring"
"github.com/gojektech/weaver"
"github.com/pkg/errors"
)
func NewHashRingStrategy(data json.RawMessage) (weaver.Sharder, error) {
cfg := HashRingStrategyConfig{}
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, err
}
if err := cfg.Validate(); err != nil {
return nil, err
}
hashRing, backends, err := hashringBackends(cfg)
if err != nil {
return nil, err
}
return &HashRingStrategy{
hashRing: hashRing,
backends: backends,
}, nil
}
type HashRingStrategy struct {
hashRing *hashring.HashRingCluster
backends map[string]*weaver.Backend
}
func (rs HashRingStrategy) Shard(key string) (*weaver.Backend, error) {
serverName := rs.hashRing.GetServer(key)
return rs.backends[serverName], nil
}
type HashRingStrategyConfig struct {
TotalVirtualBackends *int `json:"totalVirtualBackends"`
Backends map[string]BackendDefinition `json:"backends"`
}
func (hrCfg HashRingStrategyConfig) Validate() error {
if hrCfg.Backends == nil || len(hrCfg.Backends) == 0 {
return fmt.Errorf("No Shard Backends Specified Or Specified Incorrectly")
}
for _, backend := range hrCfg.Backends {
if err := backend.Validate(); err != nil {
return errors.Wrapf(err, "failed to validate backendDefinition for backend: %s", backend.BackendName)
}
}
return nil
}
func hashringBackends(cfg HashRingStrategyConfig) (*hashring.HashRingCluster, map[string]*weaver.Backend, error) {
if cfg.TotalVirtualBackends == nil || *cfg.TotalVirtualBackends < 0 {
defaultBackends := 1000
cfg.TotalVirtualBackends = &defaultBackends
}
backendDetails := map[string]*weaver.Backend{}
hashRingCluster := hashring.NewHashRingCluster(*cfg.TotalVirtualBackends)
virtualNodesFound := map[int]bool{}
maxValue := 0
rangeRegexp := regexp.MustCompile("^([\\d]+)-([\\d]+)$")
for k, v := range cfg.Backends {
matches := rangeRegexp.FindStringSubmatch(k)
if len(matches) != 3 {
return nil, nil, fmt.Errorf("Invalid range key format: %s", k)
}
end, _ := strconv.Atoi(matches[2])
start, _ := strconv.Atoi(matches[1])
if end <= start {
return nil, nil, fmt.Errorf("Invalid range key %d-%d for backends", start, end)
}
for i := start; i <= end; i++ {
if _, ok := virtualNodesFound[i]; ok {
return nil, nil, fmt.Errorf("Overlap seen in range key %d", i)
}
virtualNodesFound[i] = true
if maxValue < i {
maxValue = i
}
}
backend, err := parseBackend(v)
if err != nil {
return nil, nil, err
}
backendDetails[backend.Name] = backend
hashRingCluster.AddServer(backend.Name, k)
}
if maxValue != *cfg.TotalVirtualBackends-1 {
return nil, nil, fmt.Errorf("Shard is out of bounds Max %d found %d", *cfg.TotalVirtualBackends-1, maxValue)
}
for i := 0; i < *cfg.TotalVirtualBackends; i++ {
if _, ok := virtualNodesFound[i]; !ok {
return nil, nil, fmt.Errorf("Shard is missing coverage for %d", i)
}
}
return hashRingCluster, backendDetails, nil
}
================================================
FILE: pkg/shard/hashring_test.go
================================================
package shard_test
import (
"encoding/json"
"fmt"
"math"
"runtime"
"strconv"
"testing"
"time"
"github.com/gojektech/weaver/pkg/shard"
"github.com/stretchr/testify/assert"
)
func TestNewHashringStrategy(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 1000,
"backends": {
"0-250": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"251-500": { "backend_name": "foobar2", "backend": "http://shard01.local"},
"501-725": { "backend_name": "foobar3", "backend": "http://shard02.local"},
"726-999": { "backend_name": "foobar4", "backend": "http://shard03.local"}
}
}`)
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, err)
assert.NotNil(t, hashRingStrategy)
}
func TestShouldFailToCreateWhenWrongBackends(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 500,
"backends": "foo"
}`)
expectedErr := fmt.Errorf("json: cannot unmarshal string into Go struct field HashRingStrategyConfig.backends of type map[string]shard.BackendDefinition")
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr.Error(), err.Error())
assert.Nil(t, hashRingStrategy)
}
func TestShouldFailToCreateWhenNoBackends(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 500
}`)
expectedErr := fmt.Errorf("No Shard Backends Specified Or Specified Incorrectly")
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr, err)
assert.Nil(t, hashRingStrategy)
}
func TestShouldFailToCreateWhenNoBackendURL(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 1000,
"backends": {
"0-999": { "timeout": 100, "backend_name": "foobar1", "backend": "ht$tp://shard00.local"}
}
}`)
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Contains(t, err.Error(), "first path segment in URL cannot contain colon")
assert.Nil(t, hashRingStrategy)
}
func TestShouldFailToCreateWhenTotalVirtualBackendsIsIncorrect(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": "foo",
"backends": {
"0-10" : { "backend_name": "foo"}
}
}`)
expectedErr := fmt.Errorf("json: cannot unmarshal string into Go struct field HashRingStrategyConfig.totalVirtualBackends of type int")
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr.Error(), err.Error())
assert.Nil(t, hashRingStrategy)
}
func TestShouldFailToCreateWhenBackendURLIsMissing(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"foo" : { "backend_name": "foo"}
}
}`)
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Contains(t, err.Error(), "missing backend url in shard config:")
assert.Nil(t, hashRingStrategy)
}
func TestShouldDefaultTotalVirtualBackendsWhenValueMissing(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"0-999" : { "backend_name": "foo", "backend": "http://backend01"}
}
}`)
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, err)
assert.NotNil(t, hashRingStrategy)
}
func TestShouldFailToCreateWithIncorrectRange(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"999-0" : { "backend_name": "foo", "backend": "http://blah"}
}
}`)
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, hashRingStrategy)
assert.Contains(t, err.Error(), "Invalid range key 999-0 for backends")
}
func TestShouldFailToCreateWithIncorrectRangeSpec(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"999-999-0" : { "backend_name": "foo", "backend": "http://blah"}
}
}`)
hashRingStrategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, hashRingStrategy)
assert.Contains(t, err.Error(), "Invalid range key format:")
}
func TestShouldFailToCreateHashRingOutOfBounds(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 500,
"backends": {
"0-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-500": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
expectedErr := fmt.Errorf("Shard is out of bounds Max %d found %d", 499, 500)
_, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr, err)
}
func TestShouldFailToCreateHashRingOnOverlap(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 500,
"backends": {
"0-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"249-499": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
expectedErr := fmt.Errorf("Overlap seen in range key %d", 249)
_, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr, err)
}
func TestShouldFailToCreateHashRingForMissingValuesInTheRangeInMiddle(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 500,
"backends": {
"0-248": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-499": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
expectedErr := fmt.Errorf("Shard is missing coverage for %d", 249)
_, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr, err)
}
func TestShouldFailToCreateHashRingForMissingValuesInTheRangeAtStart(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 500,
"backends": {
"1-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-499": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
expectedErr := fmt.Errorf("Shard is missing coverage for %d", 0)
_, err := shard.NewHashRingStrategy(shardConfig)
assert.Equal(t, expectedErr, err)
}
func TestShouldCheckBackendConfiguration(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 10,
"backends": {
"0-4": "foo",
"5-9": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
expectedErr := fmt.Errorf("json: cannot unmarshal string into Go struct field HashRingStrategyConfig.backends of type shard.BackendDefinition")
strategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, strategy)
assert.Equal(t, expectedErr.Error(), err.Error())
}
func TestShouldCheckBackendConfigurationForBackendName(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 10,
"backends": {
"0-4": {"foo": "bar"},
"5-9": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
strategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, strategy)
assert.Contains(t, err.Error(), "missing backend name in shard config:")
}
func TestShouldCheckBackendConfigurationForBackendUrl(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 10,
"backends": {
"0-4": {"foo": "bar", "backend_name": "foo"},
"5-9": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
strategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, strategy)
assert.Contains(t, err.Error(), "missing backend url in shard config:")
}
func TestShouldCheckBackendConfigurationForTimeout(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 10,
"backends": {
"0-4": {"backend": "http://foo", "backend_name": "foo", "timeout": "abc"},
"5-9": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
expectedErr := fmt.Errorf("json: cannot unmarshal string into Go struct field BackendDefinition.timeout of type float64")
strategy, err := shard.NewHashRingStrategy(shardConfig)
assert.Nil(t, strategy)
assert.Equal(t, expectedErr.Error(), err.Error())
}
func TestShouldShardConsistently(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 10,
"backends": {
"0-4": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"5-9": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
strategy, _ := shard.NewHashRingStrategy(shardConfig)
expectedBackend := "foobar2"
backend, err := strategy.Shard("1")
assert.Nil(t, err)
assert.NotNil(t, backend)
assert.Equal(t, expectedBackend, backend.Name, "Should return foobar2 for key 1")
backend, err = strategy.Shard("1")
assert.Equal(t, expectedBackend, backend.Name, "Should return foobar2 for key 1")
}
func TestShouldShardConsistentlyOverALargeRange(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 10,
"backends": {
"0-4": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"5-9": { "backend_name": "foobar2", "backend": "http://shard01.local"}
}
}`)
strategy, _ := shard.NewHashRingStrategy(shardConfig)
shardList := []string{}
for i := 0; i < 10000; i++ {
backend, err := strategy.Shard(strconv.Itoa(i))
assert.Nil(t, err, "Failed to Shard for key %d", i)
if err != nil {
t.Log(err)
return
}
shardList = append(shardList, backend.Name)
}
for i := 0; i < 10000; i++ {
backend, err := strategy.Shard(strconv.Itoa(i))
assert.Nil(t, err, "Failed to Re - Shard for key %d", i)
if err != nil {
t.Log(err)
return
}
assert.Equal(t, shardList[i], backend.Name, "Sharded inconsistently for key %d %s -> %s", i, shardList[i], backend.Name)
}
}
func TestShouldShardConsistentlyAcrossRuns(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 1000,
"backends": {
"0-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-499": { "backend_name": "foobar2", "backend": "http://shard01.local"},
"500-749": { "backend_name": "foobar3", "backend": "http://shard02.local"},
"750-999": { "backend_name": "foobar4", "backend": "http://shard03.local"}
}
}`)
strategy, _ := shard.NewHashRingStrategy(shardConfig)
shardList := []string{}
for i := 0; i < 10000; i++ {
backend, err := strategy.Shard(strconv.Itoa(i))
assert.Nil(t, err, "Failed to Shard for key %d", i)
if err != nil {
t.Log(err)
return
}
shardList = append(shardList, backend.Name)
}
strategy2, _ := shard.NewHashRingStrategy(shardConfig)
for i := 0; i < 10000; i++ {
backend, err := strategy2.Shard(strconv.Itoa(i))
assert.Nil(t, err, "Failed to Re - Shard for key %d", i)
if err != nil {
t.Log(err)
return
}
assert.Equal(t, shardList[i], backend.Name, "Sharded inconsistently for key %d %s -> %s", i, shardList[i], backend.Name)
}
}
func TestShouldShardUniformally(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 1000,
"backends": {
"0-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-499": { "backend_name": "foobar2", "backend": "http://shard01.local"},
"500-749": { "backend_name": "foobar3", "backend": "http://shard02.local"},
"750-999": { "backend_name": "foobar4", "backend": "http://shard03.local"}
}
}`)
strategy, _ := shard.NewHashRingStrategy(shardConfig)
shardDistribution := map[string]int{}
numKeys := 1000000
for i := 0; i < numKeys; i++ {
backend, err := strategy.Shard(strconv.Itoa(i))
assert.Nil(t, err, "Failed to Shard for key %d", i)
if err != nil {
t.Log(err)
return
}
shardDistribution[backend.Name] = shardDistribution[backend.Name] + 1
}
mean := float64(0)
for _, v := range shardDistribution {
mean += float64(v)
}
mean = mean / 4
sd := float64(0)
for _, v := range shardDistribution {
sd += math.Pow(float64(v)-mean, 2)
}
sd = (math.Sqrt(sd/4) / float64(numKeys)) * float64(100)
assert.True(t, (sd < float64(2.5)), "Standard Deviation should be less than 2.5% -> %f", sd)
t.Log("Standard Deviation:", sd)
}
func bToMb(b uint64) uint64 {
return b / 1024 / 1024
}
func PrintMemUsage() runtime.MemStats {
var m runtime.MemStats
runtime.ReadMemStats(&m)
// For info on each, see: https://golang.org/pkg/runtime/#MemStats
fmt.Printf("Alloc = %v MiB", bToMb(m.Alloc))
fmt.Printf("\tTotalAlloc = %v MiB", bToMb(m.TotalAlloc))
fmt.Printf("\tHeapAlloc = %v MiB", bToMb(m.HeapAlloc))
fmt.Printf("\tSys = %v MiB", bToMb(m.Sys))
fmt.Printf("\tNumGC = %v\n", m.NumGC)
return m
}
func TestShouldShardWithoutLeakingMemory(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 1000,
"backends": {
"0-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-499": { "backend_name": "foobar2", "backend": "http://shard01.local"},
"500-749": { "backend_name": "foobar3", "backend": "http://shard02.local"},
"750-999": { "backend_name": "foobar4", "backend": "http://shard03.local"}
}
}`)
strategy, _ := shard.NewHashRingStrategy(shardConfig)
numKeys := 1000
PrintMemUsage()
strategy.Shard("1")
for j := 0; j < 1000; j++ {
for i := 0; i < numKeys; i++ {
_, _ = strategy.Shard(strconv.Itoa(i))
}
}
PrintMemUsage()
}
func TestToMeasureTimeForSharding(t *testing.T) {
shardConfig := json.RawMessage(`{
"totalVirtualBackends": 1000,
"backends": {
"0-249": { "timeout": 100, "backend_name": "foobar1", "backend": "http://shard00.local"},
"250-499": { "backend_name": "foobar2", "backend": "http://shard01.local"},
"500-749": { "backend_name": "foobar3", "backend": "http://shard02.local"},
"750-999": { "backend_name": "foobar4", "backend": "http://shard03.local"}
}
}`)
strategy, _ := shard.NewHashRingStrategy(shardConfig)
numKeys := 1000
start := time.Now()
for j := 0; j < 1000; j++ {
for i := 0; i < numKeys; i++ {
_, _ = strategy.Shard(strconv.Itoa(i))
}
}
elapsed := time.Since(start)
fmt.Printf("Elapsed Time %v", elapsed)
}
================================================
FILE: pkg/shard/lookup.go
================================================
package shard
import (
"encoding/json"
"github.com/gojektech/weaver"
)
func NewLookupStrategy(data json.RawMessage) (weaver.Sharder, error) {
shardConfig := map[string]BackendDefinition{}
if err := json.Unmarshal(data, &shardConfig); err != nil {
return nil, err
}
backends, err := toBackends(shardConfig)
if err != nil {
return nil, err
}
return &LookupStrategy{
backends: backends,
}, nil
}
type LookupStrategy struct {
backends map[string]*weaver.Backend
}
func (ls *LookupStrategy) Shard(key string) (*weaver.Backend, error) {
return ls.backends[key], nil
}
================================================
FILE: pkg/shard/lookup_test.go
================================================
package shard
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewLookupStrategy(t *testing.T) {
shardConfig := json.RawMessage(`{
"R-": { "timeout": 100, "backend_name": "foobar", "backend": "http://ride-service"},
"GK-": { "backend_name": "foobar", "backend": "http://go-kilat"}
}`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := lookupStrategy.Shard("GK-")
require.NoError(t, err, "should not have failed when finding shard")
assert.Equal(t, "http://go-kilat", backend.Server.String())
assert.NotNil(t, backend.Handler)
}
func TestNewLookupStrategyFailWhenTimeoutIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"R-": { "timeout": "abc", "backend": "http://ride-service"},
"GK-": { "backend": "http://go-kilat"}
}`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, lookupStrategy)
}
func TestNewLookupStrategyFailWhenNoBackendGiven(t *testing.T) {
shardConfig := json.RawMessage(`{
"R-": { "timeout": "abc", "backend_name": "hello"},
"GK-": { "backend_name": "mello", "backend": "http://go-kilat"}
}`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, lookupStrategy)
}
func TestNewLookupStrategyFailWhenBackendIsNotString(t *testing.T) {
shardConfig := json.RawMessage(`{
"R-": { "backend": 123 },
"GK-": { "backend": "http://go-kilat"}
}`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, lookupStrategy)
}
func TestNewLookupStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
shardConfig := json.RawMessage(`{
"R-": { "backend": ":"},
"GK-": { "backend": "http://go-kilat"}
}`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, lookupStrategy)
}
func TestNewLookupStrategyFailsWhenConfigIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`[]`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Equal(t, err.Error(), "json: cannot unmarshal array into Go value of type map[string]shard.BackendDefinition")
assert.Nil(t, lookupStrategy, "should have failed to parse the shard config")
}
func TestNewLookupStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"foo": "hello",
"hello": []
}`)
lookupStrategy, err := NewLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Contains(t, err.Error(), "cannot unmarshal string into Go value of type shard.BackendDefinition")
assert.Nil(t, lookupStrategy, "should have failed to parse the shard config")
}
================================================
FILE: pkg/shard/modulo.go
================================================
package shard
import (
"encoding/json"
"strconv"
"github.com/gojektech/weaver"
"github.com/pkg/errors"
)
func NewModuloStrategy(data json.RawMessage) (weaver.Sharder, error) {
shardConfig := map[string]BackendDefinition{}
if err := json.Unmarshal(data, &shardConfig); err != nil {
return nil, err
}
backends, err := toBackends(shardConfig)
if err != nil {
return nil, err
}
return &ModuloStrategy{
backends: backends,
}, nil
}
type ModuloStrategy struct {
backends map[string]*weaver.Backend
}
func (ms ModuloStrategy) Shard(key string) (*weaver.Backend, error) {
id, err := strconv.Atoi(key)
if err != nil {
return nil, errors.Wrapf(err, "not an integer key: %s", key)
}
modulo := id % (len(ms.backends))
return ms.backends[strconv.Itoa(modulo)], nil
}
================================================
FILE: pkg/shard/modulo_test.go
================================================
package shard
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewModuloShardStrategy(t *testing.T) {
shardConfig := json.RawMessage(`{
"0": { "timeout": 100, "backend_name": "foobar", "backend": "http://shard00.local"},
"1": { "backend_name": "foobar1", "backend": "http://shard01.local"},
"2": { "backend_name": "foobar2", "backend": "http://shard02.local"}
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := moduloStrategy.Shard("5678987")
require.NoError(t, err, "should not have failed to find backend")
assert.Equal(t, "http://shard02.local", backend.Server.String())
assert.NotNil(t, backend.Handler)
}
func TestNewModuloShardStrategyFailure(t *testing.T) {
shardConfig := json.RawMessage(`{
"0": { "timeout": 100, "backend_name": "foobar", "backend": "http://shard00.local"},
"1": { "backend_name": "foobar1", "backend": "http://shard01.local"},
"2": { "backend_name": "foobar2", "backend": "http://shard02.local"}
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := moduloStrategy.Shard("abcd")
require.Error(t, err, "should have failed to find backend")
assert.Nil(t, backend)
}
func TestNewModuloStrategyFailWhenTimeoutIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"0": { "backend_name": "A", "timeout": "abc", "backend": "http://shard00.local"},
"1": { "backend_name": "B", "backend": "http://shard01.local"}
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, moduloStrategy)
assert.Contains(t, err.Error(), "cannot unmarshal string into Go struct field BackendDefinition.timeout of type float64")
}
func TestNewModuloStrategyFailWhenNoBackendGiven(t *testing.T) {
shardConfig := json.RawMessage(`{
"0": { "backend_name": "hello"},
"1": { "backend_name": "mello", "backend": "http://shard01.local"}
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, moduloStrategy)
assert.Contains(t, err.Error(), "missing backend url in shard config:")
}
func TestNewModuloStrategyFailWhenBackendIsNotString(t *testing.T) {
shardConfig := json.RawMessage(`{
"0": { "backend_name": "hello", "backend": 123 },
"1": { "backend_name": "mello", "backend": "http://shard01.local"}
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, moduloStrategy)
assert.Contains(t, err.Error(), "cannot unmarshal number")
}
func TestNewModuloStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
shardConfig := json.RawMessage(`{
"0": { "backend_name": "hello", "backend": ":"},
"1": { "backend_name": "mello", "backend": "http://shard01.local"}
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, moduloStrategy)
assert.Contains(t, err.Error(), "URL Parsing failed for")
}
func TestNewModuloStrategyFailsWhenConfigIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`[]`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Contains(t, err.Error(), "json: cannot unmarshal array into Go value of type map[string]shard.BackendDefinition")
assert.Nil(t, moduloStrategy, "should have failed to parse the shard config")
}
func TestNewModuloStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"foo": "hello",
"hello": []
}`)
moduloStrategy, err := NewModuloStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Contains(t, err.Error(), "cannot unmarshal string into Go value of type shard.BackendDefinition")
assert.Nil(t, moduloStrategy, "should have failed to parse the shard config")
}
================================================
FILE: pkg/shard/no.go
================================================
package shard
import (
"encoding/json"
"fmt"
"github.com/gojektech/weaver"
"github.com/pkg/errors"
)
func NewNoStrategy(data json.RawMessage) (weaver.Sharder, error) {
cfg := NoStrategyConfig{}
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, err
}
if err := cfg.Validate(); err != nil {
return nil, err
}
backendOptions := weaver.BackendOptions{}
backend, err := weaver.NewBackend(cfg.BackendName, cfg.BackendURL, backendOptions)
if err != nil {
return nil, errors.WithStack(fmt.Errorf("failed to create backend: %s: %+v", err, cfg))
}
return &NoStrategy{
backend: backend,
}, nil
}
type NoStrategy struct {
backend *weaver.Backend
}
func (ns *NoStrategy) Shard(key string) (*weaver.Backend, error) {
return ns.backend, nil
}
type NoStrategyConfig struct {
BackendDefinition `json:",inline"`
}
================================================
FILE: pkg/shard/no_test.go
================================================
package shard
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewNoStrategy(t *testing.T) {
shardConfig := json.RawMessage(`{ "backend_name": "foobar", "backend": "http://localhost" }`)
noStrategy, err := NewNoStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := noStrategy.Shard("whatever")
require.NoError(t, err, "should not have failed when finding shard")
assert.Equal(t, "http://localhost", backend.Server.String())
assert.NotNil(t, backend.Handler)
}
func TestNewNoStrategyFailsWhenConfigIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`[]`)
noStrategy, err := NewNoStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Equal(t, "json: cannot unmarshal array into Go value of type shard.NoStrategyConfig", err.Error())
assert.Nil(t, noStrategy, "should have failed to parse the shard config")
}
func TestNewNoStrategyFailsWhenBackendURLInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"backend_name": "foobar",
"backend": "http$://google.com"
}`)
noStrategy, err := NewNoStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Contains(t, err.Error(), "failed to create backend:")
assert.Nil(t, noStrategy, "should have failed to parse the shard config")
}
func TestNewNoStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{ "backend_name": "foobar", "backend": [] }`)
noStrategy, err := NewNoStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Contains(t, err.Error(), "cannot unmarshal array into Go struct field NoStrategyConfig.backend of type string")
assert.Nil(t, noStrategy, "should have failed to parse the shard config")
}
func TestNoStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
shardConfig := json.RawMessage(`{ "server": ":" }`)
noStrategy, err := NewNoStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, noStrategy)
}
================================================
FILE: pkg/shard/prefix_lookup.go
================================================
package shard
import (
"encoding/json"
"errors"
"strings"
"github.com/gojektech/weaver"
)
const (
defaultBackendKey = "default"
defaultPrefixSplitter = "-"
)
type prefixLookupConfig struct {
PrefixSplitter string `json:"prefix_splitter"`
Backends map[string]BackendDefinition `json:"backends"`
}
func (plg prefixLookupConfig) Validate() error {
if len(plg.Backends) == 0 {
return errors.New("no backends specified")
}
return nil
}
func NewPrefixLookupStrategy(data json.RawMessage) (weaver.Sharder, error) {
prefixLookupConfig := &prefixLookupConfig{}
if err := json.Unmarshal(data, &prefixLookupConfig); err != nil {
return nil, err
}
if err := prefixLookupConfig.Validate(); err != nil {
return nil, err
}
backends, err := toBackends(prefixLookupConfig.Backends)
if err != nil {
return nil, err
}
prefixSplitter := prefixLookupConfig.PrefixSplitter
if prefixSplitter == "" {
prefixSplitter = defaultPrefixSplitter
}
return &PrefixLookupStrategy{
backends: backends,
prefixSplitter: prefixSplitter,
}, nil
}
type PrefixLookupStrategy struct {
backends map[string]*weaver.Backend
prefixSplitter string
}
func (pls *PrefixLookupStrategy) Shard(key string) (*weaver.Backend, error) {
prefix := strings.SplitAfter(key, pls.prefixSplitter)[0]
if pls.backends[prefix] == nil {
return pls.backends[defaultBackendKey], nil
}
return pls.backends[prefix], nil
}
================================================
FILE: pkg/shard/prefix_lookup_test.go
================================================
package shard
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNewPrefixLookupStrategy(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R_": { "timeout": 100, "backend_name": "foobar", "backend": "http://ride-service"},
"GK_": { "backend_name": "foobar", "backend": "http://go-kilat"}
},
"prefix_splitter": "_"
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := prefixLookupStrategy.Shard("GK_123444")
require.NoError(t, err, "should not have failed when finding shard")
assert.Equal(t, "http://go-kilat", backend.Server.String())
assert.NotNil(t, backend.Handler)
}
func TestNewPrefixLookupStrategyWithDefaultPrefixSplitter(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R-": { "timeout": 100, "backend_name": "foobar", "backend": "http://ride-service"},
"GK-": { "backend_name": "foobar", "backend": "http://go-kilat"}
}
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := prefixLookupStrategy.Shard("GK-123444")
require.NoError(t, err, "should not have failed when finding shard")
assert.Equal(t, "http://go-kilat", backend.Server.String())
assert.NotNil(t, backend.Handler)
}
func TestNewPrefixLookupStrategyForNoPrefix(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R-": { "timeout": 100, "backend_name": "foobar", "backend": "http://ride-service"},
"GK-": { "backend_name": "foobar", "backend": "http://go-kilat"},
"default": { "backend_name": "hello", "backend": "http://sm"}
},
"prefix_splitter": "-"
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.NoError(t, err, "should not have failed to parse the shard config")
backend, err := prefixLookupStrategy.Shard("123444")
require.NoError(t, err, "should not have failed when finding shard")
assert.Equal(t, "http://sm", backend.Server.String())
assert.NotNil(t, backend.Handler)
}
func TestNewPrefixLookupStrategyFailWhenTimeoutIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R-": { "timeout": "abc", "backend": "http://ride-service"},
"GK-": { "backend": "http://go-kilat"},
"default": { "backend_name": "hello", "backend": "http://sm"}
},
"prefix_splitter": "-"
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, prefixLookupStrategy)
}
func TestNewPrefixLookupStrategyFailWhenNoBackendGiven(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R-": { "timeout": "abc", "backend_name": "hello"},
"GK-": { "backend_name": "mello", "backend": "http://go-kilat"},
"default": { "backend_name": "hello", "backend": "http://sm"}
},
"prefix_splitter": "-"
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, prefixLookupStrategy)
}
func TestNewPrefixLookupStrategyFailWhenBackendIsNotString(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R-": { "backend": 123 },
"GK-": { "backend": "http://go-kilat"},
"default": { "backend_name": "hello", "backend": "http://sm"}
},
"prefix_splitter": "-"
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, prefixLookupStrategy)
}
func TestNewPrefixLookupStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
shardConfig := json.RawMessage(`{
"backends": {
"R-": { "backend": ":"},
"GK-": { "backend": "http://go-kilat"},
"default": { "backend_name": "hello", "backend": "http://sm"}
},
"prefix_splitter": "-"
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Nil(t, prefixLookupStrategy)
}
func TestNewPrefixLookupStrategyFailsWhenConfigIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`[]`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Equal(t, err.Error(), "json: cannot unmarshal array into Go value of type shard.prefixLookupConfig")
assert.Nil(t, prefixLookupStrategy, "should have failed to parse the shard config")
}
func TestNewPrefixLookupStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
shardConfig := json.RawMessage(`{
"foo": "hello",
"hello": []
}`)
prefixLookupStrategy, err := NewPrefixLookupStrategy(shardConfig)
require.Error(t, err, "should have failed to parse the shard config")
assert.Contains(t, err.Error(), "no backends specified")
assert.Nil(t, prefixLookupStrategy, "should have failed to parse the shard config")
}
================================================
FILE: pkg/shard/s2.go
================================================
package shard
import (
"encoding/json"
"fmt"
"strconv"
"strings"
"github.com/gojektech/weaver"
"github.com/gojektech/weaver/pkg/util"
geos2 "github.com/golang/geo/s2"
)
var (
defaultBackendS2id = "default"
)
func NewS2Strategy(data json.RawMessage) (weaver.Sharder, error) {
cfg := S2StrategyConfig{}
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, err
}
if err := cfg.Validate(); err != nil {
return nil, err
}
s2Backends := make(map[string]*weaver.Backend, len(cfg.Backends))
for s2id, backend := range cfg.Backends {
var err error
if s2Backends[s2id], err = parseBackend(backend); err != nil {
return nil, err
}
}
if cfg.ShardKeyPosition == nil {
defaultPos := -1
cfg.ShardKeyPosition = &defaultPos
}
return &S2Strategy{
backends: s2Backends,
shardKeySeparator: cfg.ShardKeySeparator,
shardKeyPosition: *cfg.ShardKeyPosition,
}, nil
}
type S2Strategy struct {
backends map[string]*weaver.Backend
shardKeySeparator string
shardKeyPosition int
}
type S2StrategyConfig struct {
ShardKeySeparator string `json:"shard_key_separator"`
ShardKeyPosition *int `json:"shard_key_position,omitempty"`
Backends map[string]BackendDefinition `json:"backends"`
}
func (s2cfg S2StrategyConfig) Validate() error {
if s2cfg.ShardKeySeparator == "" {
return Error("missing required config: shard_key_separator")
}
if err := s2cfg.validateS2IDs(); err != nil {
return err
}
return nil
}
func (s2cfg S2StrategyConfig) validateS2IDs() error {
backendCount := len(s2cfg.Backends)
s2IDs := make([]uint64, backendCount)
for k := range s2cfg.Backends {
if k != defaultBackendS2id {
id, err := strconv.ParseUint(k, 10, 64)
if err != nil {
return fmt.Errorf("[error] Bad S2 ID found in backends: %s", k)
}
s2IDs = append(s2IDs, id)
}
}
if util.ContainsOverlappingS2IDs(s2IDs) {
return fmt.Errorf("[error] Overlapping S2 IDs found in backends: %v", s2cfg.Backends)
}
return nil
}
func s2idFromLatLng(latLng []string) (s2id geos2.CellID, err error) {
if len(latLng) != 2 {
err = Error("lat lng key is not valid")
return
}
lat, err := strconv.ParseFloat(latLng[0], 64)
if err != nil {
err = Error("fail to parse latitude")
return
}
lng, err := strconv.ParseFloat(latLng[1], 64)
if err != nil {
err = Error("fail to parse longitude")
return
}
s2LatLng := geos2.LatLngFromDegrees(lat, lng)
if !s2LatLng.IsValid() {
err = Error("fail to convert lat-long to geos2 objects")
return
}
s2id = geos2.CellIDFromLatLng(s2LatLng)
return
}
func s2idFromSmartID(smartIDComponents []string, pos int) (s2id geos2.CellID, err error) {
if len(smartIDComponents) <= pos {
err = Error("failed to get location from smart-id")
return
}
s2idStr := smartIDComponents[pos]
s2idUint, err := strconv.ParseUint(s2idStr, 10, 64)
if err != nil {
err = Error("failed to parse s2id")
return
}
s2id = geos2.CellID(s2idUint)
return
}
func (s2s *S2Strategy) s2ID(key string) (uint64, error) {
shardKeyComponents := strings.Split(key, s2s.shardKeySeparator)
var s2CellID geos2.CellID
var err error
switch s2s.shardKeyPosition {
case -1:
s2CellID, err = s2idFromLatLng(shardKeyComponents)
default:
s2CellID, err = s2idFromSmartID(shardKeyComponents, s2s.shardKeyPosition)
}
return uint64(s2CellID), err
}
func (s2s *S2Strategy) Shard(key string) (*weaver.Backend, error) {
s2id, err := s2s.s2ID(key)
if err != nil {
return nil, err
}
s2CellID := geos2.CellID(s2id)
for s2Str, backendConfig := range s2s.backends {
cellInt, err := strconv.ParseUint(s2Str, 10, 64)
if err != nil {
continue
}
shardCellID := geos2.CellID(cellInt)
if shardCellID.Contains(s2CellID) {
return backendConfig, nil
}
}
if _, ok := s2s.backends[defaultBackendS2id]; ok {
return s2s.backends[defaultBackendS2id], nil
}
return nil, Error("fail to find backend")
}
================================================
FILE: pkg/shard/s2_test.go
================================================
package shard
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
)
var (
s2ShardConfigWithoutDefault = json.RawMessage(`{
"backends": {
"3344472479136481280": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"},
"3346530764903677952": { "backend_name": "jkt-b", "backend": "http://jkt.b.local"}
},
"shard_key_separator": ","
}`)
s2ShardConfig = json.RawMessage(`{
"backends": {
"3344472479136481280": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"},
"3346530764903677952": { "backend_name": "jkt-b", "backend": "http://jkt.b.local"},
"default": {"backend_name": "jkt-c", "backend": "http://jkt.c.local"}
},
"shard_key_separator": ","
}`)
s2ShardConfigBad = json.RawMessage(`{
"backends": {
"123454321": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"},
"123459876": "bad-data"
},
"shard_key_separator": ","
}`)
s2ShardConfigInvalidS2ID = json.RawMessage(`{
"backends": {
"45435hb344hj3b": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"}
},
"shard_key_separator": ","
}`)
s2ShardConfigOverlapping = json.RawMessage(`{
"backends": {
"3344474311656055761": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"},
"3344474311656055760": { "backend_name": "jkt-b", "backend": "http://jkt.b.local"},
"3344474311656055744": { "backend_name": "jkt-c", "backend": "http://jkt.c.local"},
"3344474311656055552": { "backend_name": "jkt-d", "backend": "http://jkt.d.local"},
"3344474311656055808": { "backend_name": "jkt-e", "backend": "http://jkt.e.local"},
"3573054715985026048": { "backend_name": "surbaya-a", "backend": "http://surbaya.a.local"}
},
"shard_key_separator": ","
}`)
s2SmartIDShardConfig = json.RawMessage(`{
"backends": {
"3344472479136481280": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"},
"3346530764903677952": { "backend_name": "jkt-b", "backend": "http://jkt.b.local"},
"default": {"backend_name": "jkt-c", "backend": "http://jkt.c.local"}
},
"shard_key_separator": "-",
"shard_key_position":2
}`)
s2SmartIDShardConfigBad = json.RawMessage(`{
"backends": {
"3344472479136481280": { "backend_name": "jkt-a", "backend": "http://jkt.a.local"},
"3346530764903677952": { "backend_name": "jkt-b", "backend": "http://jkt.b.local"},
"default": {"backend_name": "jkt-c", "backend": "http://jkt.c.local"}
},
"shard_key_position":2
}`)
)
func TestNewS2StrategySuccess(t *testing.T) {
strategy, err := NewS2Strategy(s2ShardConfig)
assert.NotNil(t, strategy)
assert.Nil(t, err)
}
func TestNewS2StrategyFailure(t *testing.T) {
sharder, err := NewS2Strategy(s2ShardConfigBad)
assert.Nil(t, sharder)
assert.NotNil(t, err)
assert.Contains(t, err.Error(), "json: cannot unmarshal string")
}
func TestNewS2StrategyFailureWithMissingKeySeparator(t *testing.T) {
sharder, err := NewS2Strategy(s2SmartIDShardConfigBad)
assert.Nil(t, sharder)
assert.NotNil(t, err)
assert.Equal(t, "[error] missing required config: shard_key_separator", err.Error())
}
func TestNewS2StrategyFailureWithOverlappingShards(t *testing.T) {
sharder, err := NewS2Strategy(s2ShardConfigOverlapping)
assert.Nil(t, sharder)
assert.NotNil(t, err)
assert.Contains(t, err.Error(), "[error] Overlapping S2 IDs found in backends:")
}
func TestNewS2StrategyFailureWithInvalidS2ID(t *testing.T) {
sharder, err := NewS2Strategy(s2ShardConfigInvalidS2ID)
assert.Nil(t, sharder)
assert.NotNil(t, err)
assert.Contains(t, err.Error(), "[error] Bad S2 ID found in backends:")
}
func TestS2StrategyS2IDSuccess(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: ",", shardKeyPosition: -1}
actualS2ID, err := strategy.s2ID("-6.1751,106.865")
expectedS2ID := uint64(3344474311656055761)
assert.Nil(t, err)
assert.Equal(t, expectedS2ID, actualS2ID)
actualS2ID, err = strategy.s2ID("-6.1751,110.865")
expectedS2ID = uint64(3346531974082111711)
assert.Nil(t, err)
assert.Equal(t, expectedS2ID, actualS2ID)
}
func TestS2StrategyS2IDFailureForInvalidLat(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: ",", shardKeyPosition: -1}
_, err := strategy.s2ID("-qwerty6.1751,32.865")
assert.NotNil(t, err)
}
func TestS2StrategyS2IDFailureForInvalidLng(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: ",", shardKeyPosition: -1}
_, err := strategy.s2ID("-6.1751,qwerty1232.865")
assert.NotNil(t, err)
}
func TestS2StrategyS2IDFailureForInvalidLatLngObject(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: ",", shardKeyPosition: -1}
_, err := strategy.s2ID("1116.1751,1232.865")
assert.NotNil(t, err)
}
func TestS2StrategyS2IDFailureForInvalidAlphanumeric(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: ",", shardKeyPosition: -1}
_, err := strategy.s2ID("-6.17511232.865")
assert.NotNil(t, err)
_, err = strategy.s2ID("abc,1232.865")
assert.NotNil(t, err)
}
func TestS2StrategyS2IDWithSmartIDSuccess(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: "-", shardKeyPosition: 2}
actualS2ID, err := strategy.s2ID("v1-foo-3344474403281829888")
expectedS2ID := uint64(3344474403281829888)
assert.Nil(t, err)
assert.Equal(t, expectedS2ID, actualS2ID)
actualS2ID, err = strategy.s2ID("v1-foo-3346532139293212672")
expectedS2ID = uint64(3346532139293212672)
assert.Nil(t, err)
assert.Equal(t, expectedS2ID, actualS2ID)
}
func TestS2StrategyS2IDFailureForInvalidS2ID(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: "-", shardKeyPosition: 2}
_, err := strategy.s2ID("v1-foo-bar")
assert.NotNil(t, err)
assert.Equal(t, "[error] failed to parse s2id", err.Error())
}
func TestS2StrategyS2IDFailureForInvalidSmartID(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: "-", shardKeyPosition: 2}
_, err := strategy.s2ID("booyeah")
assert.NotNil(t, err)
assert.Equal(t, "[error] failed to get location from smart-id", err.Error())
}
func TestS2StrategyS2IDFailureForInvalidSeparatorConfig(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: "&", shardKeyPosition: 2}
_, err := strategy.s2ID("v1-foo-3344474403281829888")
assert.NotNil(t, err)
assert.Equal(t, "[error] failed to get location from smart-id", err.Error())
}
func TestS2StrategyS2IDFailureForInvalidPositionConfig(t *testing.T) {
strategy := S2Strategy{shardKeySeparator: "-", shardKeyPosition: 3}
_, err := strategy.s2ID("v1-foo-3344474403281829888")
assert.NotNil(t, err)
assert.Equal(t, "[error] failed to get location from smart-id", err.Error())
strategy = S2Strategy{shardKeySeparator: "-", shardKeyPosition: 4}
_, err = strategy.s2ID("v1-foo-3344474403281829888")
assert.NotNil(t, err)
assert.Equal(t, "[error] failed to get location from smart-id", err.Error())
}
func TestS2StrategyShardSuccess(t *testing.T) {
strategy, _ := NewS2Strategy(s2ShardConfig)
backend, err := strategy.Shard("-6.1751,106.865")
expectedBackend := "jkt-a"
assert.Nil(t, err)
assert.Equal(t, expectedBackend, backend.Name)
backend, err = strategy.Shard("-6.1751,110.865")
expectedBackend = "jkt-b"
assert.Nil(t, err)
assert.Equal(t, expectedBackend, backend.Name)
}
func TestS2StrategySmartIDShardSuccess(t *testing.T) {
strategy, _ := NewS2Strategy(s2SmartIDShardConfig)
backend, err := strategy.Shard("v1-foo-3344474403281829888")
expectedBackend := "jkt-a"
assert.Nil(t, err)
assert.Equal(t, expectedBackend, backend.Name)
backend, err = strategy.Shard("v1-foo-3346532139293212672")
expectedBackend = "jkt-b"
assert.Nil(t, err)
assert.Equal(t, expectedBackend, backend.Name)
backend, err = strategy.Shard("v1-foo-2534")
expectedBackend = "jkt-c"
assert.Nil(t, err)
assert.Equal(t, expectedBackend, backend.Name)
}
func TestS2StrategyShardSuccessForDefaultBackend(t *testing.T) {
strategy, _ := NewS2Strategy(s2ShardConfig)
backend, err := strategy.Shard("-34.1751,106.865")
expectedBackend := "jkt-c"
assert.Nil(t, err)
assert.Equal(t, expectedBackend, backend.Name)
}
func TestS2StrategyShardFailure(t *testing.T) {
strategy, _ := NewS2Strategy(s2ShardConfig)
backendForWrongLatLng, err := strategy.Shard("-126.1751,906.865")
assert.NotNil(t, err)
assert.Nil(t, backendForWrongLatLng)
backendForWrongInputNum, err := strategy.Shard("-126.1751865")
assert.NotNil(t, err)
assert.Nil(t, backendForWrongInputNum)
backendForWrongInputAlpha, err := strategy.Shard("abc,xyz")
assert.NotNil(t, err)
assert.Nil(t, backendForWrongInputAlpha)
strategy, _ = NewS2Strategy(s2ShardConfigWithoutDefault)
noBackendForCorrectInput, err := strategy.Shard("-6.1751,126.865")
assert.NotNil(t, err)
assert.Equal(t, "[error] fail to find backend", err.Error())
assert.Nil(t, noBackendForCorrectInput)
}
func TestGeneratesCustomError(t *testing.T) {
ce := CustomError{ExitMessage: "Error for custom error"}
err := ce.Error()
assert.Equal(t, err, "[error] Error for custom error")
}
================================================
FILE: pkg/shard/shard.go
================================================
package shard
import (
"encoding/json"
"fmt"
"github.com/gojektech/weaver"
)
func New(name string, cfg json.RawMessage) (weaver.Sharder, error) {
newSharder, found := shardFuncTable[name]
if !found {
return nil, fmt.Errorf("failed to find sharder with name '%s'", name)
}
return newSharder(cfg)
}
type sharderGenerator func(json.RawMessage) (weaver.Sharder, error)
var shardFuncTable = map[string]sharderGenerator{
"lookup": NewLookupStrategy,
"prefix-lookup": NewPrefixLookupStrategy,
"none": NewNoStrategy,
"modulo": NewModuloStrategy,
"hashring": NewHashRingStrategy,
"s2": NewS2Strategy,
}
================================================
FILE: pkg/util/s2.go
================================================
package util
import (
"sort"
"github.com/golang/geo/s2"
)
type s2List []s2.CellID
func (s2l s2List) Len() int {
return len(s2l)
}
func (s2l s2List) Less(i int, j int) bool {
return s2l[i].Level() < s2l[j].Level()
}
func (s2l s2List) Swap(i int, j int) {
s2l[i], s2l[j] = s2l[j], s2l[i]
}
func toS2List(ints []uint64) s2List {
lst := s2List{}
for _, id := range ints {
lst = append(lst, s2.CellID(id))
}
return lst
}
func ContainsOverlappingS2IDs(ids []uint64) bool {
lst := toS2List(ids)
sort.Sort(lst)
length := lst.Len()
for i := 0; i < length; i++ {
higher := lst[i]
for j := i + 1; j < length; j++ {
lower := lst[j]
if higher.Level() < lower.Level() && higher.Contains(lower) {
return true
}
}
}
return false
}
================================================
FILE: pkg/util/s2_test.go
================================================
package util
import (
"testing"
"github.com/golang/geo/s2"
"github.com/stretchr/testify/assert"
)
func TestS2Len(t *testing.T) {
list := s2List{
s2.CellID(23458045904904),
s2.CellID(23458222224904),
s2.CellID(23888885904904),
s2.CellID(23458999999904),
}
assert.Equal(t, list.Len(), 4)
}
func TestS2Less(t *testing.T) {
list := s2List{
s2.CellID(4542091330435678208),
s2.CellID(4542051748017078272),
}
assert.True(t, list.Less(0, 1))
}
func TestS2Swap(t *testing.T) {
first := s2.CellID(4542091330435678208)
second := s2.CellID(4542051748017078272)
list := s2List{
first,
second,
}
list.Swap(0, 1)
assert.Equal(t, list[0], second)
assert.Equal(t, list[1], first)
}
func TestContainsOverlappingS2IDsTrueCase(t *testing.T) {
list := []uint64{
4542051748017078272,
4542091330435678208,
}
assert.True(t, ContainsOverlappingS2IDs(list))
}
func TestContainsOverlappingS2IDsFalseCase(t *testing.T) {
list := []uint64{
4542051748017078272,
1504976331727699968,
}
assert.False(t, ContainsOverlappingS2IDs(list))
}
func TestToS2List(t *testing.T) {
idList := []uint64{
4542051748017078272,
4542091330435678208,
}
expectedS2List := s2List{
s2.CellID(idList[0]),
s2.CellID(idList[1]),
}
assert.Equal(t, expectedS2List, toS2List(idList))
}
================================================
FILE: pkg/util/util.go
================================================
package util
import (
"strings"
"unicode"
)
func ToSnake(in string) string {
runes := []rune(in)
length := len(runes)
var out []rune
for i := 0; i < length; i++ {
if i > 0 && unicode.IsUpper(runes[i]) && ((i+1 < length && unicode.IsLower(runes[i+1])) || unicode.IsLower(runes[i-1])) {
out = append(out, '_')
}
out = append(out, unicode.ToLower(runes[i]))
}
return strings.Replace(string(out), "-", "", -1)
}
func BoolToOnOff(on bool) string {
if on {
return "on"
}
return "off"
}
================================================
FILE: pkg/util/util_test.go
================================================
package util
import (
"testing"
)
type SnakeTest struct {
input string
output string
}
var tests = []SnakeTest{
{"a", "a"},
{"snake", "snake"},
{"A", "a"},
{"ID", "id"},
{"MOTD", "motd"},
{"Snake", "snake"},
{"SnakeTest", "snake_test"},
{"Snake-Test", "snake_test"},
{"SnakeID", "snake_id"},
{"Snake_ID", "snake_id"},
{"SnakeIDGoogle", "snake_id_google"},
{"LinuxMOTD", "linux_motd"},
{"OMGWTFBBQ", "omgwtfbbq"},
{"omg_wtf_bbq", "omg_wtf_bbq"},
}
func TestToSnake(t *testing.T) {
for _, test := range tests {
if ToSnake(test.input) != test.output {
t.Errorf(`ToSnake("%s"), wanted "%s", got \%s"`, test.input, test.output, ToSnake(test.input))
}
}
}
================================================
FILE: server/error.go
================================================
package server
import (
"encoding/json"
"net/http"
"github.com/gojektech/weaver/pkg/instrumentation"
)
type weaverResponse struct {
Errors []errorDetails `json:"errors"`
}
type errorDetails struct {
Code string `json:"code"`
Message string `json:"message"`
MessageTitle string `json:"message_title"`
MessageSeverity string `json:"message_severity"`
}
func notFoundError(w http.ResponseWriter, r *http.Request) {
instrumentation.IncrementNotFound()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusNotFound)
errorResponse := weaverResponse{
Errors: []errorDetails{
{
Code: "weaver:route:not_found",
Message: "Something went wrong",
MessageTitle: "Failure",
MessageSeverity: "failure",
},
},
}
response, _ := json.Marshal(errorResponse)
w.Write(response)
}
func internalServerError(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
errorResponse := weaverResponse{
Errors: []errorDetails{
{
Code: "weaver:service:unavailable",
Message: "Something went wrong",
MessageTitle: "Internal error",
MessageSeverity: "failure",
},
},
}
response, _ := json.Marshal(errorResponse)
w.Write(response)
}
// TODO: decouple instrumentation from this errors function
type err503Handler struct {
ACLName string
}
func (eh err503Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
failureHTTPStatus := http.StatusServiceUnavailable
instrumentation.IncrementInternalAPIStatusCount(eh.ACLName, failureHTTPStatus)
errorResponse := weaverResponse{
Errors: []errorDetails{
{
Code: "weaver:service:unavailable",
Message: "Something went wrong",
MessageTitle: "Failure",
MessageSeverity: "failure",
},
},
}
response, _ := json.Marshal(errorResponse)
w.WriteHeader(failureHTTPStatus)
w.Write(response)
return
}
================================================
FILE: server/error_test.go
================================================
package server
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
)
func Test404Handler(t *testing.T) {
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/hello", nil)
notFoundError(w, r)
assert.Equal(t, http.StatusNotFound, w.Code)
assert.Equal(t, "{\"errors\":[{\"code\":\"weaver:route:not_found\",\"message\":\"Something went wrong\",\"message_title\":\"Failure\",\"message_severity\":\"failure\"}]}", w.Body.String())
}
func Test500Handler(t *testing.T) {
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/hello", nil)
internalServerError(w, r)
assert.Equal(t, http.StatusInternalServerError, w.Code)
assert.Equal(t, "{\"errors\":[{\"code\":\"weaver:service:unavailable\",\"message\":\"Something went wrong\",\"message_title\":\"Internal error\",\"message_severity\":\"failure\"}]}", w.Body.String())
}
func Test503Handler(t *testing.T) {
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/hello", nil)
err503Handler{}.ServeHTTP(w, r)
assert.Equal(t, http.StatusServiceUnavailable, w.Code)
assert.Equal(t, "{\"errors\":[{\"code\":\"weaver:service:unavailable\",\"message\":\"Something went wrong\",\"message_title\":\"Failure\",\"message_severity\":\"failure\"}]}", w.Body.String())
}
================================================
FILE: server/handler.go
================================================
package server
import (
"net/http"
"github.com/gojektech/weaver/config"
"github.com/gojektech/weaver/pkg/instrumentation"
"github.com/gojektech/weaver/pkg/logger"
newrelic "github.com/newrelic/go-agent"
)
type proxy struct {
router *Router
}
func (proxy *proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rw := &wrapperResponseWriter{ResponseWriter: w}
if r.URL.Path == "/ping" || r.URL.Path == "/" {
proxy.pingHandler(rw, r)
return
}
timing := instrumentation.NewTiming()
defer instrumentation.TimeTotalLatency(timing)
instrumentation.IncrementTotalRequestCount()
acl, err := proxy.router.Route(r)
if err != nil || acl == nil {
logger.Errorrf(r, "failed to find route: %+v for request: %s", err, r.URL.String())
notFoundError(rw, r)
return
}
backend, err := acl.Endpoint.Shard(r)
if backend == nil || err != nil {
logger.Errorrf(r, "failed to find backend for acl %s for: %s, error: %s", acl.ID, r.URL.String(), err)
err503Handler{ACLName: acl.ID}.ServeHTTP(rw, r)
return
}
instrumentation.IncrementAPIBackendRequestCount(acl.ID, backend.Name)
instrumentation.IncrementAPIRequestCount(acl.ID)
apiTiming := instrumentation.NewTiming()
defer instrumentation.TimeAPILatency(acl.ID, apiTiming)
apiBackendTiming := instrumentation.NewTiming()
defer instrumentation.TimeAPIBackendLatency(acl.ID, backend.Name, apiBackendTiming)
var s newrelic.ExternalSegment
if txn, ok := w.(newrelic.Transaction); ok {
s = newrelic.StartExternalSegment(txn, r)
}
backend.Handler.ServeHTTP(rw, r)
s.End()
logger.ProxyInfo(acl.ID, backend.Server.String(), r, rw.statusCode, rw)
instrumentation.IncrementAPIStatusCount(acl.ID, rw.statusCode)
instrumentation.IncrementAPIBackendStatusCount(acl.ID, backend.Name, rw.statusCode)
}
func (proxy *proxy) pingHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write([]byte("{}"))
}
func wrapNewRelicHandler(proxy *proxy) http.Handler {
if !config.NewRelicConfig().Enabled {
return proxy
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path
if path == "/ping" {
proxy.ServeHTTP(w, r)
return
}
_, next := newrelic.WrapHandleFunc(instrumentation.NewRelicApp(), path,
func(w http.ResponseWriter, r *http.Request) {
proxy.ServeHTTP(w, r)
})
next(w, r)
})
}
================================================
FILE: server/handler_test.go
================================================
package server
import (
"bytes"
"encoding/json"
"fmt"
"github.com/gojektech/weaver/pkg/shard"
"net/http"
"net/http/httptest"
"testing"
"github.com/gojektech/weaver"
"github.com/gojektech/weaver/pkg/logger"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
)
type ProxySuite struct {
suite.Suite
rtr *Router
}
func (ps *ProxySuite) SetupTest() {
logger.SetupLogger()
routeLoader := &mockRouteLoader{}
ps.rtr = NewRouter(routeLoader)
require.NotNil(ps.T(), ps.rtr)
}
func TestProxySuite(t *testing.T) {
suite.Run(t, new(ProxySuite))
}
func (ps *ProxySuite) TestProxyHandlerOnSuccessfulRouting() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusForbidden)
_, _ = w.Write([]byte("foobar"))
}))
acl := &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && PathRegexp(`/(GF-|R-).*`)",
EndpointConfig: &weaver.EndpointConfig{
Matcher: "path",
ShardExpr: "/(GF-|R-|).*",
ShardFunc: "lookup",
ShardConfig: json.RawMessage(fmt.Sprintf(`{
"GF-": {
"backend_name": "foo",
"backend": "%s"
},
"R-": {
"backend_name": "bar",
"timeout": 100.0,
"backend": "http://iamgone"
}
}`, server.URL)),
},
}
sharder, err := shard.New(acl.EndpointConfig.ShardFunc, acl.EndpointConfig.ShardConfig)
require.NoError(ps.T(), err, "should not have failed to init a sharder")
acl.Endpoint, err = weaver.NewEndpoint(acl.EndpointConfig, sharder)
require.NoError(ps.T(), err, "should not have failed to set endpoint")
_ = ps.rtr.UpsertRoute(acl.Criterion, acl)
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/GF-1234", nil)
proxy := proxy{router: ps.rtr}
proxy.ServeHTTP(w, r)
assert.Equal(ps.T(), http.StatusForbidden, w.Code)
assert.Equal(ps.T(), "foobar", w.Body.String())
}
func (ps *ProxySuite) TestProxyHandlerOnBodyBasedMatcherWithModuloSharding() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("foobar"))
}))
acl := &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && PathRegexp(`/drivers`)",
EndpointConfig: &weaver.EndpointConfig{
Matcher: "body",
ShardExpr: ".drivers.id",
ShardFunc: "modulo",
ShardConfig: json.RawMessage(fmt.Sprintf(`{
"0": {
"backend_name": "foo",
"backend": "%s"
},
"1": {
"backend_name": "bar",
"timeout": 100.0,
"backend": "http://shard01"
}
}`, server.URL)),
},
}
sharder, err := shard.New(acl.EndpointConfig.ShardFunc, acl.EndpointConfig.ShardConfig)
require.NoError(ps.T(), err, "should not have failed to init a sharder")
acl.Endpoint, err = weaver.NewEndpoint(acl.EndpointConfig, sharder)
require.NoError(ps.T(), err, "should not have failed to set endpoint")
_ = ps.rtr.UpsertRoute(acl.Criterion, acl)
w := httptest.NewRecorder()
body := bytes.NewReader([]byte(`{ "drivers": { "id": "122" } }`))
r := httptest.NewRequest("GET", "/drivers", body)
proxy := proxy{router: ps.rtr}
proxy.ServeHTTP(w, r)
assert.Equal(ps.T(), http.StatusOK, w.Code)
assert.Equal(ps.T(), "foobar", w.Body.String())
}
func (ps *ProxySuite) TestProxyHandlerOnPathBasedMatcherWithModuloSharding() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("foobar"))
}))
acl := &weaver.ACL{
ID: "svc-01",
Criterion: "Method(`GET`) && PathRegexp(`/drivers`)",
EndpointConfig: &weaver.EndpointConfig{
Matcher: "path",
ShardExpr: `/drivers/(\d+)`,
ShardFunc: "modulo",
ShardConfig: json.RawMessage(fmt.Sprintf(`{
"0": {
"backend_name": "foo",
"backend": "http://shard01"
},
"1": {
"backend_name": "bar",
"timeout":100.0,
"backend":"%s"
}
}`, server.URL)),
},
}
sharder, err := shard.New(acl.EndpointConfig.ShardFunc, acl.EndpointConfig.ShardConfig)
require.NoError(ps.T(), err, "should not have failed to init a sharder")
acl.Endpoint, err = weaver.NewEndpoint(acl.EndpointConfig, sharder)
require.NoError(ps.T(), err, "should not have failed to set endpoint")
_ = ps.rtr.UpsertRoute(acl.Criterion, acl)
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/drivers/123", nil)
proxy := proxy{router: ps.rtr}
proxy.ServeHTTP(w, r)
assert.Equal(ps.T(), http.StatusOK, w.Code)
assert.Equal(ps.T(), "foobar", w.Body.String())
}
func (ps *ProxySuite) TestProxyHandlerOnFailureRouting() {
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/GF-1234", nil)
proxy := proxy{router: ps.rtr}
proxy.ServeHTTP
gitextract_25hsj6d8/ ├── .gitignore ├── .travis.yml ├── CONTRIBUTING.md ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── acl.go ├── acl_test.go ├── backend.go ├── backend_test.go ├── cmd/ │ └── weaver-server/ │ └── main.go ├── config/ │ ├── config.go │ ├── config_test.go │ ├── newrelic.go │ ├── proxy.go │ └── statsd.go ├── deployment/ │ └── weaver/ │ ├── .helmignore │ ├── Chart.yaml │ ├── README.md │ ├── templates/ │ │ ├── _helpers.tpl │ │ ├── deployment.yaml │ │ ├── service.yaml │ │ └── statefulset.yaml │ ├── values-env.yaml │ └── values.yaml ├── docker-compose.yml ├── docs/ │ └── weaver_acls.md ├── endpoint.go ├── endpoint_test.go ├── etcd/ │ ├── aclkey.go │ ├── routeloader.go │ └── routeloader_test.go ├── examples/ │ └── body_lookup/ │ ├── Dockerfile │ ├── README.md │ ├── estimate_acl.json │ ├── estimator/ │ │ ├── Chart.yaml │ │ ├── templates/ │ │ │ ├── _helpers.tpl │ │ │ ├── deployment.yaml │ │ │ └── service.yaml │ │ ├── values-id.yaml │ │ ├── values-sg.yaml │ │ └── values.yaml │ └── main.go ├── go.mod ├── go.sum ├── goreleaser.yml ├── pkg/ │ ├── instrumentation/ │ │ ├── newrelic.go │ │ └── statsd.go │ ├── logger/ │ │ └── logger.go │ ├── matcher/ │ │ ├── matcher.go │ │ └── matcher_test.go │ ├── shard/ │ │ ├── domain.go │ │ ├── hashring.go │ │ ├── hashring_test.go │ │ ├── lookup.go │ │ ├── lookup_test.go │ │ ├── modulo.go │ │ ├── modulo_test.go │ │ ├── no.go │ │ ├── no_test.go │ │ ├── prefix_lookup.go │ │ ├── prefix_lookup_test.go │ │ ├── s2.go │ │ ├── s2_test.go │ │ └── shard.go │ └── util/ │ ├── s2.go │ ├── s2_test.go │ ├── util.go │ └── util_test.go ├── server/ │ ├── error.go │ ├── error_test.go │ ├── handler.go │ ├── handler_test.go │ ├── loader.go │ ├── mock.go │ ├── recovery.go │ ├── recovery_test.go │ ├── router.go │ ├── router_test.go │ ├── server.go │ └── wrapped_response_writer.go ├── sharder.go └── weaver.conf.yaml.sample
SYMBOL INDEX (354 symbols across 52 files)
FILE: acl.go
type ACL (line 9) | type ACL struct
method GenACL (line 18) | func (acl *ACL) GenACL(val string) error {
method String (line 22) | func (acl ACL) String() string {
FILE: acl_test.go
constant validACLDoc (line 10) | validACLDoc = `{
function TestGenACL (line 37) | func TestGenACL(t *testing.T) {
FILE: backend.go
type Backend (line 14) | type Backend struct
type BackendOptions (line 20) | type BackendOptions struct
function NewBackend (line 24) | func NewBackend(name string, serverURL string, options BackendOptions) (...
function newWeaverReverseProxy (line 37) | func newWeaverReverseProxy(target *url.URL, options BackendOptions) *htt...
FILE: backend_test.go
function TestNewBackend (line 10) | func TestNewBackend(t *testing.T) {
function TestNewBackendFailsWhenURLIsInvalid (line 21) | func TestNewBackendFailsWhenURLIsInvalid(t *testing.T) {
FILE: cmd/weaver-server/main.go
function main (line 20) | func main() {
function startWeaver (line 37) | func startWeaver(_ *cli.Context) error {
FILE: config/config.go
type Config (line 18) | type Config struct
function Load (line 35) | func Load() {
function ServerReadTimeoutInMillis (line 66) | func ServerReadTimeoutInMillis() time.Duration {
function ServerWriteTimeoutInMillis (line 70) | func ServerWriteTimeoutInMillis() time.Duration {
function ProxyServerAddress (line 74) | func ProxyServerAddress() string {
function ETCDKeyPrefix (line 78) | func ETCDKeyPrefix() string {
function NewRelicConfig (line 82) | func NewRelicConfig() newrelic.Config {
function SentryDSN (line 86) | func SentryDSN() string {
function StatsD (line 90) | func StatsD() StatsDConfig {
function Proxy (line 94) | func Proxy() ProxyConfig {
function NewETCDClient (line 98) | func NewETCDClient() (etcd.Client, error) {
function LogLevel (line 113) | func LogLevel() string {
function extractStringValue (line 117) | func extractStringValue(key string) string {
function extractBoolValue (line 122) | func extractBoolValue(key string) bool {
function extractBoolValueDefaultToFalse (line 127) | func extractBoolValueDefaultToFalse(key string) bool {
function extractIntValue (line 135) | func extractIntValue(key string) int {
function checkPresenceOf (line 145) | func checkPresenceOf(key string) {
FILE: config/config_test.go
function TestShouldLoadConfigFromFile (line 13) | func TestShouldLoadConfigFromFile(t *testing.T) {
function TestShouldLoadFromEnvVars (line 23) | func TestShouldLoadFromEnvVars(t *testing.T) {
FILE: config/newrelic.go
function loadNewRelicConfig (line 7) | func loadNewRelicConfig() newrelic.Config {
FILE: config/proxy.go
type ProxyConfig (line 5) | type ProxyConfig struct
method ProxyDialerTimeoutInMS (line 23) | func (pc ProxyConfig) ProxyDialerTimeoutInMS() time.Duration {
method ProxyDialerKeepAliveInMS (line 27) | func (pc ProxyConfig) ProxyDialerKeepAliveInMS() time.Duration {
method ProxyMaxIdleConns (line 31) | func (pc ProxyConfig) ProxyMaxIdleConns() int {
method ProxyIdleConnTimeoutInMS (line 35) | func (pc ProxyConfig) ProxyIdleConnTimeoutInMS() time.Duration {
method KeepAliveEnabled (line 39) | func (pc ProxyConfig) KeepAliveEnabled() bool {
function loadProxyConfig (line 13) | func loadProxyConfig() ProxyConfig {
FILE: config/statsd.go
type StatsDConfig (line 3) | type StatsDConfig struct
method Prefix (line 21) | func (sdc StatsDConfig) Prefix() string {
method FlushPeriodInSeconds (line 25) | func (sdc StatsDConfig) FlushPeriodInSeconds() int {
method Host (line 29) | func (sdc StatsDConfig) Host() string {
method Port (line 33) | func (sdc StatsDConfig) Port() int {
method Enabled (line 37) | func (sdc StatsDConfig) Enabled() bool {
function loadStatsDConfig (line 11) | func loadStatsDConfig() StatsDConfig {
FILE: endpoint.go
type EndpointConfig (line 13) | type EndpointConfig struct
method genShardKeyFunc (line 20) | func (endpointConfig *EndpointConfig) genShardKeyFunc() (shardKeyFunc,...
type Endpoint (line 31) | type Endpoint struct
method Shard (line 52) | func (endpoint *Endpoint) Shard(request *http.Request) (*Backend, erro...
function NewEndpoint (line 36) | func NewEndpoint(endpointConfig *EndpointConfig, sharder Sharder) (*Endp...
type shardKeyFunc (line 61) | type shardKeyFunc
FILE: endpoint_test.go
function TestNewEndpoint (line 11) | func TestNewEndpoint(t *testing.T) {
function TestNewEndpoint_SharderIsNil (line 26) | func TestNewEndpoint_SharderIsNil(t *testing.T) {
type stubSharder (line 39) | type stubSharder struct
method Shard (line 42) | func (stub *stubSharder) Shard(key string) (*Backend, error) {
FILE: etcd/aclkey.go
constant ACLKeyFormat (line 11) | ACLKeyFormat = "/%s/acls/%s/acl"
type ACLKey (line 15) | type ACLKey
function GenACLKey (line 18) | func GenACLKey(key string) ACLKey {
function GenKey (line 22) | func GenKey(acl *weaver.ACL, pfx string) ACLKey {
FILE: etcd/routeloader.go
function NewRouteLoader (line 19) | func NewRouteLoader() (*RouteLoader, error) {
type RouteLoader (line 31) | type RouteLoader struct
method PutACL (line 37) | func (routeLoader *RouteLoader) PutACL(acl *weaver.ACL) (ACLKey, error) {
method GetACL (line 51) | func (routeLoader *RouteLoader) GetACL(key ACLKey) (*weaver.ACL, error) {
method DelACL (line 75) | func (routeLoader *RouteLoader) DelACL(key ACLKey) error {
method WatchRoutes (line 83) | func (routeLoader *RouteLoader) WatchRoutes(ctx context.Context, upser...
method BootstrapRoutes (line 131) | func (routeLoader *RouteLoader) BootstrapRoutes(ctx context.Context, u...
function initEtcd (line 165) | func initEtcd(routeLoader *RouteLoader) (etcd.KeysAPI, string) {
FILE: etcd/routeloader_test.go
type RouteLoaderSuite (line 22) | type RouteLoaderSuite struct
method SetupTest (line 28) | func (es *RouteLoaderSuite) SetupTest() {
method TestNewRouteLoader (line 38) | func (es *RouteLoaderSuite) TestNewRouteLoader() {
method TestPutACL (line 46) | func (es *RouteLoaderSuite) TestPutACL() {
method TestBootstrapRoutes (line 76) | func (es *RouteLoaderSuite) TestBootstrapRoutes() {
method TestBootstrapRoutesSucceedWhenARouteUpsertFails (line 97) | func (es *RouteLoaderSuite) TestBootstrapRoutesSucceedWhenARouteUpsert...
method TestBootstrapRoutesSucceedWhenARouteDoesntExist (line 126) | func (es *RouteLoaderSuite) TestBootstrapRoutesSucceedWhenARouteDoesnt...
method TestBootstrapRoutesSucceedWhenARouteHasInvalidData (line 131) | func (es *RouteLoaderSuite) TestBootstrapRoutesSucceedWhenARouteHasInv...
method TestWatchRoutesUpsertRoutesWhenRoutesSet (line 144) | func (es *RouteLoaderSuite) TestWatchRoutesUpsertRoutesWhenRoutesSet() {
method TestWatchRoutesUpsertRoutesWhenRoutesUpdated (line 162) | func (es *RouteLoaderSuite) TestWatchRoutesUpsertRoutesWhenRoutesUpdat...
method TestWatchRoutesDeleteRouteWhenARouteIsDeleted (line 181) | func (es *RouteLoaderSuite) TestWatchRoutesDeleteRouteWhenARouteIsDele...
function TestRouteLoaderSuite (line 42) | func TestRouteLoaderSuite(tst *testing.T) {
function newTestACL (line 201) | func newTestACL(matcher string) *weaver.ACL {
function genRouteProcessorMock (line 224) | func genRouteProcessorMock(c chan *weaver.ACL) func(*weaver.ACL) error {
function deepEqual (line 231) | func deepEqual(t *testing.T, expected *weaver.ACL, actual *weaver.ACL) {
function assertEqualJSON (line 240) | func assertEqualJSON(t *testing.T, json1, json2 json.RawMessage) {
function failingUpsertRouteFunc (line 252) | func failingUpsertRouteFunc(acl *weaver.ACL) error {
function successUpsertRouteFunc (line 256) | func successUpsertRouteFunc(acl *weaver.ACL) error {
FILE: examples/body_lookup/main.go
type estimationRequest (line 14) | type estimationRequest struct
function getEnv (line 19) | func getEnv(key string, fallback string) string {
function handler (line 26) | func handler(w http.ResponseWriter, r *http.Request) {
function handlePing (line 30) | func handlePing(w http.ResponseWriter, r *http.Request) {
function getAmount (line 34) | func getAmount() float64 {
function handleEstimate (line 41) | func handleEstimate(w http.ResponseWriter, r *http.Request) {
function main (line 47) | func main() {
FILE: pkg/instrumentation/newrelic.go
type ctxKey (line 13) | type ctxKey
constant txKey (line 15) | txKey ctxKey = 0
function InitNewRelic (line 19) | func InitNewRelic() newrelic.Application {
function ShutdownNewRelic (line 32) | func ShutdownNewRelic() {
function NewRelicApp (line 38) | func NewRelicApp() newrelic.Application {
function StartRedisSegmentNow (line 42) | func StartRedisSegmentNow(op string, coll string, txn newrelic.Transacti...
function NewContext (line 53) | func NewContext(ctx context.Context, w http.ResponseWriter) context.Cont...
function NewContextWithTransaction (line 64) | func NewContextWithTransaction(ctx context.Context, tx newrelic.Transact...
function GetTx (line 68) | func GetTx(ctx context.Context) (newrelic.Transaction, bool) {
FILE: pkg/instrumentation/statsd.go
function InitiateStatsDMetrics (line 15) | func InitiateStatsDMetrics() error {
function StatsDClient (line 37) | func StatsDClient() *statsd.Client {
function CloseStatsDClient (line 41) | func CloseStatsDClient() {
function NewTiming (line 48) | func NewTiming() statsd.Timing {
function IncrementTotalRequestCount (line 56) | func IncrementTotalRequestCount() {
function IncrementAPIRequestCount (line 60) | func IncrementAPIRequestCount(apiName string) {
function IncrementAPIStatusCount (line 64) | func IncrementAPIStatusCount(apiName string, httpStatusCode int) {
function IncrementAPIBackendRequestCount (line 68) | func IncrementAPIBackendRequestCount(apiName, backendName string) {
function IncrementAPIBackendStatusCount (line 72) | func IncrementAPIBackendStatusCount(apiName, backendName string, httpSta...
function IncrementCrashCount (line 76) | func IncrementCrashCount() {
function IncrementNotFound (line 80) | func IncrementNotFound() {
function IncrementInternalAPIStatusCount (line 84) | func IncrementInternalAPIStatusCount(aclName string, statusCode int) {
function TimeTotalLatency (line 88) | func TimeTotalLatency(timing statsd.Timing) {
function TimeAPILatency (line 96) | func TimeAPILatency(apiName string, timing statsd.Timing) {
function TimeAPIBackendLatency (line 104) | func TimeAPIBackendLatency(apiName, backendName string, timing statsd.Ti...
function incrementProbe (line 112) | func incrementProbe(key string) {
FILE: pkg/logger/logger.go
function SetupLogger (line 14) | func SetupLogger() {
function AddHook (line 28) | func AddHook(hook logrus.Hook) {
function Debug (line 32) | func Debug(args ...interface{}) {
function Debugf (line 36) | func Debugf(format string, args ...interface{}) {
function Debugln (line 40) | func Debugln(args ...interface{}) {
function Debugrf (line 44) | func Debugrf(r *http.Request, format string, args ...interface{}) {
function Error (line 48) | func Error(args ...interface{}) {
function Errorf (line 52) | func Errorf(format string, args ...interface{}) {
function Errorln (line 56) | func Errorln(args ...interface{}) {
function Errorrf (line 60) | func Errorrf(r *http.Request, format string, args ...interface{}) {
function ErrorWithFieldsf (line 64) | func ErrorWithFieldsf(fields logrus.Fields, format string, args ...inter...
function Fatal (line 68) | func Fatal(args ...interface{}) {
function Fatalf (line 72) | func Fatalf(format string, args ...interface{}) {
function Fatalln (line 76) | func Fatalln(args ...interface{}) {
function Info (line 80) | func Info(args ...interface{}) {
function Infof (line 84) | func Infof(format string, args ...interface{}) {
function Infoln (line 88) | func Infoln(args ...interface{}) {
function Inforf (line 92) | func Inforf(r *http.Request, format string, args ...interface{}) {
function InfoWithFieldsf (line 96) | func InfoWithFieldsf(fields logrus.Fields, format string, args ...interf...
function ProxyInfo (line 100) | func ProxyInfo(aclName string, downstreamHost string, r *http.Request, r...
function httpRequestFields (line 110) | func httpRequestFields(r *http.Request) logrus.Fields {
function httpResponseFields (line 129) | func httpResponseFields(responseStatus int, rw http.ResponseWriter) logr...
function Warn (line 141) | func Warn(args ...interface{}) {
function Warnf (line 145) | func Warnf(format string, args ...interface{}) {
function Warnln (line 149) | func Warnln(args ...interface{}) {
function WithField (line 153) | func WithField(key string, value interface{}) *logrus.Entry {
function WithFields (line 157) | func WithFields(fields logrus.Fields) *logrus.Entry {
function httpRequestLogEntry (line 161) | func httpRequestLogEntry(r *http.Request) *logrus.Entry {
FILE: pkg/matcher/matcher.go
function New (line 17) | func New(matcherName string) (MatcherFunc, bool) {
type MatcherFunc (line 22) | type MatcherFunc
FILE: pkg/matcher/matcher_test.go
function TestBodyMatcher (line 12) | func TestBodyMatcher(t *testing.T) {
function TestBodyMatcherParseInt (line 24) | func TestBodyMatcherParseInt(t *testing.T) {
function TestBodyMatcherParseTypeAssertFail (line 36) | func TestBodyMatcherParseTypeAssertFail(t *testing.T) {
function TestBodyMatcherFail (line 49) | func TestBodyMatcherFail(t *testing.T) {
function TestHeaderMatcher (line 61) | func TestHeaderMatcher(t *testing.T) {
function TestHeadersCsvMatcherWithSingleHeader (line 73) | func TestHeadersCsvMatcherWithSingleHeader(t *testing.T) {
function TestHeadersCsvMatcherWithSingleHeaderWhenNoneArePresent (line 87) | func TestHeadersCsvMatcherWithSingleHeaderWhenNoneArePresent(t *testing....
function TestHeadersCsvMatcherWithZeroHeaders (line 97) | func TestHeadersCsvMatcherWithZeroHeaders(t *testing.T) {
function TestHeadersCsvMatcherWithMultipleHeaders (line 111) | func TestHeadersCsvMatcherWithMultipleHeaders(t *testing.T) {
function TestHeadersCsvMatcherWithMultipleHeadersWhenSomeArePresent (line 125) | func TestHeadersCsvMatcherWithMultipleHeadersWhenSomeArePresent(t *testi...
function TestHeadersCsvMatcherWithMultipleHeadersWhenNoneArePresent (line 137) | func TestHeadersCsvMatcherWithMultipleHeadersWhenNoneArePresent(t *testi...
function TestHeaderMatcherFail (line 147) | func TestHeaderMatcherFail(t *testing.T) {
function TestParamMatcher (line 158) | func TestParamMatcher(t *testing.T) {
function TestParamMatcherFail (line 169) | func TestParamMatcherFail(t *testing.T) {
function TestPathMatcher (line 180) | func TestPathMatcher(t *testing.T) {
function TestPathMatcherFail (line 191) | func TestPathMatcherFail(t *testing.T) {
FILE: pkg/shard/domain.go
type CustomError (line 12) | type CustomError struct
method Error (line 16) | func (e *CustomError) Error() string {
function Error (line 20) | func Error(msg string) error {
type BackendDefinition (line 24) | type BackendDefinition struct
method Validate (line 30) | func (bd BackendDefinition) Validate() error {
function toBackends (line 42) | func toBackends(shardConfig map[string]BackendDefinition) (map[string]*w...
function parseBackend (line 61) | func parseBackend(shardConfig BackendDefinition) (*weaver.Backend, error) {
FILE: pkg/shard/hashring.go
function NewHashRingStrategy (line 14) | func NewHashRingStrategy(data json.RawMessage) (weaver.Sharder, error) {
type HashRingStrategy (line 35) | type HashRingStrategy struct
method Shard (line 40) | func (rs HashRingStrategy) Shard(key string) (*weaver.Backend, error) {
type HashRingStrategyConfig (line 45) | type HashRingStrategyConfig struct
method Validate (line 50) | func (hrCfg HashRingStrategyConfig) Validate() error {
function hashringBackends (line 64) | func hashringBackends(cfg HashRingStrategyConfig) (*hashring.HashRingClu...
FILE: pkg/shard/hashring_test.go
function TestNewHashringStrategy (line 16) | func TestNewHashringStrategy(t *testing.T) {
function TestShouldFailToCreateWhenWrongBackends (line 31) | func TestShouldFailToCreateWhenWrongBackends(t *testing.T) {
function TestShouldFailToCreateWhenNoBackends (line 42) | func TestShouldFailToCreateWhenNoBackends(t *testing.T) {
function TestShouldFailToCreateWhenNoBackendURL (line 52) | func TestShouldFailToCreateWhenNoBackendURL(t *testing.T) {
function TestShouldFailToCreateWhenTotalVirtualBackendsIsIncorrect (line 64) | func TestShouldFailToCreateWhenTotalVirtualBackendsIsIncorrect(t *testin...
function TestShouldFailToCreateWhenBackendURLIsMissing (line 77) | func TestShouldFailToCreateWhenBackendURLIsMissing(t *testing.T) {
function TestShouldDefaultTotalVirtualBackendsWhenValueMissing (line 88) | func TestShouldDefaultTotalVirtualBackendsWhenValueMissing(t *testing.T) {
function TestShouldFailToCreateWithIncorrectRange (line 99) | func TestShouldFailToCreateWithIncorrectRange(t *testing.T) {
function TestShouldFailToCreateWithIncorrectRangeSpec (line 110) | func TestShouldFailToCreateWithIncorrectRangeSpec(t *testing.T) {
function TestShouldFailToCreateHashRingOutOfBounds (line 121) | func TestShouldFailToCreateHashRingOutOfBounds(t *testing.T) {
function TestShouldFailToCreateHashRingOnOverlap (line 134) | func TestShouldFailToCreateHashRingOnOverlap(t *testing.T) {
function TestShouldFailToCreateHashRingForMissingValuesInTheRangeInMiddle (line 147) | func TestShouldFailToCreateHashRingForMissingValuesInTheRangeInMiddle(t ...
function TestShouldFailToCreateHashRingForMissingValuesInTheRangeAtStart (line 160) | func TestShouldFailToCreateHashRingForMissingValuesInTheRangeAtStart(t *...
function TestShouldCheckBackendConfiguration (line 173) | func TestShouldCheckBackendConfiguration(t *testing.T) {
function TestShouldCheckBackendConfigurationForBackendName (line 188) | func TestShouldCheckBackendConfigurationForBackendName(t *testing.T) {
function TestShouldCheckBackendConfigurationForBackendUrl (line 201) | func TestShouldCheckBackendConfigurationForBackendUrl(t *testing.T) {
function TestShouldCheckBackendConfigurationForTimeout (line 214) | func TestShouldCheckBackendConfigurationForTimeout(t *testing.T) {
function TestShouldShardConsistently (line 228) | func TestShouldShardConsistently(t *testing.T) {
function TestShouldShardConsistentlyOverALargeRange (line 248) | func TestShouldShardConsistentlyOverALargeRange(t *testing.T) {
function TestShouldShardConsistentlyAcrossRuns (line 280) | func TestShouldShardConsistentlyAcrossRuns(t *testing.T) {
function TestShouldShardUniformally (line 315) | func TestShouldShardUniformally(t *testing.T) {
function bToMb (line 356) | func bToMb(b uint64) uint64 {
function PrintMemUsage (line 360) | func PrintMemUsage() runtime.MemStats {
function TestShouldShardWithoutLeakingMemory (line 372) | func TestShouldShardWithoutLeakingMemory(t *testing.T) {
function TestToMeasureTimeForSharding (line 397) | func TestToMeasureTimeForSharding(t *testing.T) {
FILE: pkg/shard/lookup.go
function NewLookupStrategy (line 9) | func NewLookupStrategy(data json.RawMessage) (weaver.Sharder, error) {
type LookupStrategy (line 25) | type LookupStrategy struct
method Shard (line 29) | func (ls *LookupStrategy) Shard(key string) (*weaver.Backend, error) {
FILE: pkg/shard/lookup_test.go
function TestNewLookupStrategy (line 11) | func TestNewLookupStrategy(t *testing.T) {
function TestNewLookupStrategyFailWhenTimeoutIsInvalid (line 27) | func TestNewLookupStrategyFailWhenTimeoutIsInvalid(t *testing.T) {
function TestNewLookupStrategyFailWhenNoBackendGiven (line 39) | func TestNewLookupStrategyFailWhenNoBackendGiven(t *testing.T) {
function TestNewLookupStrategyFailWhenBackendIsNotString (line 51) | func TestNewLookupStrategyFailWhenBackendIsNotString(t *testing.T) {
function TestNewLookupStrategyFailWhenBackendIsNotAValidURL (line 63) | func TestNewLookupStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
function TestNewLookupStrategyFailsWhenConfigIsInvalid (line 75) | func TestNewLookupStrategyFailsWhenConfigIsInvalid(t *testing.T) {
function TestNewLookupStrategyFailsWhenConfigValueIsInvalid (line 85) | func TestNewLookupStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
FILE: pkg/shard/modulo.go
function NewModuloStrategy (line 11) | func NewModuloStrategy(data json.RawMessage) (weaver.Sharder, error) {
type ModuloStrategy (line 27) | type ModuloStrategy struct
method Shard (line 31) | func (ms ModuloStrategy) Shard(key string) (*weaver.Backend, error) {
FILE: pkg/shard/modulo_test.go
function TestNewModuloShardStrategy (line 11) | func TestNewModuloShardStrategy(t *testing.T) {
function TestNewModuloShardStrategyFailure (line 28) | func TestNewModuloShardStrategyFailure(t *testing.T) {
function TestNewModuloStrategyFailWhenTimeoutIsInvalid (line 44) | func TestNewModuloStrategyFailWhenTimeoutIsInvalid(t *testing.T) {
function TestNewModuloStrategyFailWhenNoBackendGiven (line 57) | func TestNewModuloStrategyFailWhenNoBackendGiven(t *testing.T) {
function TestNewModuloStrategyFailWhenBackendIsNotString (line 70) | func TestNewModuloStrategyFailWhenBackendIsNotString(t *testing.T) {
function TestNewModuloStrategyFailWhenBackendIsNotAValidURL (line 83) | func TestNewModuloStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
function TestNewModuloStrategyFailsWhenConfigIsInvalid (line 96) | func TestNewModuloStrategyFailsWhenConfigIsInvalid(t *testing.T) {
function TestNewModuloStrategyFailsWhenConfigValueIsInvalid (line 106) | func TestNewModuloStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
FILE: pkg/shard/no.go
function NewNoStrategy (line 11) | func NewNoStrategy(data json.RawMessage) (weaver.Sharder, error) {
type NoStrategy (line 32) | type NoStrategy struct
method Shard (line 36) | func (ns *NoStrategy) Shard(key string) (*weaver.Backend, error) {
type NoStrategyConfig (line 40) | type NoStrategyConfig struct
FILE: pkg/shard/no_test.go
function TestNewNoStrategy (line 11) | func TestNewNoStrategy(t *testing.T) {
function TestNewNoStrategyFailsWhenConfigIsInvalid (line 24) | func TestNewNoStrategyFailsWhenConfigIsInvalid(t *testing.T) {
function TestNewNoStrategyFailsWhenBackendURLInvalid (line 34) | func TestNewNoStrategyFailsWhenBackendURLInvalid(t *testing.T) {
function TestNewNoStrategyFailsWhenConfigValueIsInvalid (line 47) | func TestNewNoStrategyFailsWhenConfigValueIsInvalid(t *testing.T) {
function TestNoStrategyFailWhenBackendIsNotAValidURL (line 57) | func TestNoStrategyFailWhenBackendIsNotAValidURL(t *testing.T) {
FILE: pkg/shard/prefix_lookup.go
constant defaultBackendKey (line 12) | defaultBackendKey = "default"
constant defaultPrefixSplitter (line 13) | defaultPrefixSplitter = "-"
type prefixLookupConfig (line 16) | type prefixLookupConfig struct
method Validate (line 21) | func (plg prefixLookupConfig) Validate() error {
function NewPrefixLookupStrategy (line 29) | func NewPrefixLookupStrategy(data json.RawMessage) (weaver.Sharder, erro...
type PrefixLookupStrategy (line 56) | type PrefixLookupStrategy struct
method Shard (line 61) | func (pls *PrefixLookupStrategy) Shard(key string) (*weaver.Backend, e...
FILE: pkg/shard/prefix_lookup_test.go
function TestNewPrefixLookupStrategy (line 11) | func TestNewPrefixLookupStrategy(t *testing.T) {
function TestNewPrefixLookupStrategyWithDefaultPrefixSplitter (line 30) | func TestNewPrefixLookupStrategyWithDefaultPrefixSplitter(t *testing.T) {
function TestNewPrefixLookupStrategyForNoPrefix (line 48) | func TestNewPrefixLookupStrategyForNoPrefix(t *testing.T) {
function TestNewPrefixLookupStrategyFailWhenTimeoutIsInvalid (line 68) | func TestNewPrefixLookupStrategyFailWhenTimeoutIsInvalid(t *testing.T) {
function TestNewPrefixLookupStrategyFailWhenNoBackendGiven (line 84) | func TestNewPrefixLookupStrategyFailWhenNoBackendGiven(t *testing.T) {
function TestNewPrefixLookupStrategyFailWhenBackendIsNotString (line 100) | func TestNewPrefixLookupStrategyFailWhenBackendIsNotString(t *testing.T) {
function TestNewPrefixLookupStrategyFailWhenBackendIsNotAValidURL (line 116) | func TestNewPrefixLookupStrategyFailWhenBackendIsNotAValidURL(t *testing...
function TestNewPrefixLookupStrategyFailsWhenConfigIsInvalid (line 132) | func TestNewPrefixLookupStrategyFailsWhenConfigIsInvalid(t *testing.T) {
function TestNewPrefixLookupStrategyFailsWhenConfigValueIsInvalid (line 142) | func TestNewPrefixLookupStrategyFailsWhenConfigValueIsInvalid(t *testing...
FILE: pkg/shard/s2.go
function NewS2Strategy (line 18) | func NewS2Strategy(data json.RawMessage) (weaver.Sharder, error) {
type S2Strategy (line 48) | type S2Strategy struct
method s2ID (line 129) | func (s2s *S2Strategy) s2ID(key string) (uint64, error) {
method Shard (line 144) | func (s2s *S2Strategy) Shard(key string) (*weaver.Backend, error) {
type S2StrategyConfig (line 54) | type S2StrategyConfig struct
method Validate (line 60) | func (s2cfg S2StrategyConfig) Validate() error {
method validateS2IDs (line 70) | func (s2cfg S2StrategyConfig) validateS2IDs() error {
function s2idFromLatLng (line 89) | func s2idFromLatLng(latLng []string) (s2id geos2.CellID, err error) {
function s2idFromSmartID (line 113) | func s2idFromSmartID(smartIDComponents []string, pos int) (s2id geos2.Ce...
FILE: pkg/shard/s2_test.go
function TestNewS2StrategySuccess (line 72) | func TestNewS2StrategySuccess(t *testing.T) {
function TestNewS2StrategyFailure (line 78) | func TestNewS2StrategyFailure(t *testing.T) {
function TestNewS2StrategyFailureWithMissingKeySeparator (line 85) | func TestNewS2StrategyFailureWithMissingKeySeparator(t *testing.T) {
function TestNewS2StrategyFailureWithOverlappingShards (line 92) | func TestNewS2StrategyFailureWithOverlappingShards(t *testing.T) {
function TestNewS2StrategyFailureWithInvalidS2ID (line 99) | func TestNewS2StrategyFailureWithInvalidS2ID(t *testing.T) {
function TestS2StrategyS2IDSuccess (line 106) | func TestS2StrategyS2IDSuccess(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidLat (line 119) | func TestS2StrategyS2IDFailureForInvalidLat(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidLng (line 125) | func TestS2StrategyS2IDFailureForInvalidLng(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidLatLngObject (line 131) | func TestS2StrategyS2IDFailureForInvalidLatLngObject(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidAlphanumeric (line 137) | func TestS2StrategyS2IDFailureForInvalidAlphanumeric(t *testing.T) {
function TestS2StrategyS2IDWithSmartIDSuccess (line 146) | func TestS2StrategyS2IDWithSmartIDSuccess(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidS2ID (line 159) | func TestS2StrategyS2IDFailureForInvalidS2ID(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidSmartID (line 166) | func TestS2StrategyS2IDFailureForInvalidSmartID(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidSeparatorConfig (line 173) | func TestS2StrategyS2IDFailureForInvalidSeparatorConfig(t *testing.T) {
function TestS2StrategyS2IDFailureForInvalidPositionConfig (line 180) | func TestS2StrategyS2IDFailureForInvalidPositionConfig(t *testing.T) {
function TestS2StrategyShardSuccess (line 192) | func TestS2StrategyShardSuccess(t *testing.T) {
function TestS2StrategySmartIDShardSuccess (line 205) | func TestS2StrategySmartIDShardSuccess(t *testing.T) {
function TestS2StrategyShardSuccessForDefaultBackend (line 223) | func TestS2StrategyShardSuccessForDefaultBackend(t *testing.T) {
function TestS2StrategyShardFailure (line 231) | func TestS2StrategyShardFailure(t *testing.T) {
function TestGeneratesCustomError (line 252) | func TestGeneratesCustomError(t *testing.T) {
FILE: pkg/shard/shard.go
function New (line 10) | func New(name string, cfg json.RawMessage) (weaver.Sharder, error) {
type sharderGenerator (line 19) | type sharderGenerator
FILE: pkg/util/s2.go
type s2List (line 9) | type s2List
method Len (line 11) | func (s2l s2List) Len() int {
method Less (line 15) | func (s2l s2List) Less(i int, j int) bool {
method Swap (line 19) | func (s2l s2List) Swap(i int, j int) {
function toS2List (line 23) | func toS2List(ints []uint64) s2List {
function ContainsOverlappingS2IDs (line 31) | func ContainsOverlappingS2IDs(ids []uint64) bool {
FILE: pkg/util/s2_test.go
function TestS2Len (line 10) | func TestS2Len(t *testing.T) {
function TestS2Less (line 20) | func TestS2Less(t *testing.T) {
function TestS2Swap (line 28) | func TestS2Swap(t *testing.T) {
function TestContainsOverlappingS2IDsTrueCase (line 40) | func TestContainsOverlappingS2IDsTrueCase(t *testing.T) {
function TestContainsOverlappingS2IDsFalseCase (line 48) | func TestContainsOverlappingS2IDsFalseCase(t *testing.T) {
function TestToS2List (line 56) | func TestToS2List(t *testing.T) {
FILE: pkg/util/util.go
function ToSnake (line 8) | func ToSnake(in string) string {
function BoolToOnOff (line 24) | func BoolToOnOff(on bool) string {
FILE: pkg/util/util_test.go
type SnakeTest (line 7) | type SnakeTest struct
function TestToSnake (line 29) | func TestToSnake(t *testing.T) {
FILE: server/error.go
type weaverResponse (line 10) | type weaverResponse struct
type errorDetails (line 14) | type errorDetails struct
function notFoundError (line 21) | func notFoundError(w http.ResponseWriter, r *http.Request) {
function internalServerError (line 42) | func internalServerError(w http.ResponseWriter, r *http.Request) {
type err503Handler (line 63) | type err503Handler struct
method ServeHTTP (line 67) | func (eh err503Handler) ServeHTTP(w http.ResponseWriter, r *http.Reque...
FILE: server/error_test.go
function Test404Handler (line 11) | func Test404Handler(t *testing.T) {
function Test500Handler (line 21) | func Test500Handler(t *testing.T) {
function Test503Handler (line 31) | func Test503Handler(t *testing.T) {
FILE: server/handler.go
type proxy (line 12) | type proxy struct
method ServeHTTP (line 16) | func (proxy *proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
method pingHandler (line 67) | func (proxy *proxy) pingHandler(w http.ResponseWriter, r *http.Request) {
function wrapNewRelicHandler (line 73) | func wrapNewRelicHandler(proxy *proxy) http.Handler {
FILE: server/handler_test.go
type ProxySuite (line 19) | type ProxySuite struct
method SetupTest (line 25) | func (ps *ProxySuite) SetupTest() {
method TestProxyHandlerOnSuccessfulRouting (line 38) | func (ps *ProxySuite) TestProxyHandlerOnSuccessfulRouting() {
method TestProxyHandlerOnBodyBasedMatcherWithModuloSharding (line 84) | func (ps *ProxySuite) TestProxyHandlerOnBodyBasedMatcherWithModuloShar...
method TestProxyHandlerOnPathBasedMatcherWithModuloSharding (line 131) | func (ps *ProxySuite) TestProxyHandlerOnPathBasedMatcherWithModuloShar...
method TestProxyHandlerOnFailureRouting (line 177) | func (ps *ProxySuite) TestProxyHandlerOnFailureRouting() {
method TestProxyHandlerOnMissingBackend (line 188) | func (ps *ProxySuite) TestProxyHandlerOnMissingBackend() {
method TestHealthCheckWithPingRoute (line 224) | func (ps *ProxySuite) TestHealthCheckWithPingRoute() {
method TestHealthCheckWithDefaultRoute (line 234) | func (ps *ProxySuite) TestHealthCheckWithDefaultRoute() {
function TestProxySuite (line 34) | func TestProxySuite(t *testing.T) {
FILE: server/loader.go
type UpsertRouteFunc (line 9) | type UpsertRouteFunc
type DeleteRouteFunc (line 10) | type DeleteRouteFunc
type RouteLoader (line 12) | type RouteLoader interface
FILE: server/mock.go
type mockRouteLoader (line 9) | type mockRouteLoader struct
method BootstrapRoutes (line 13) | func (mrl *mockRouteLoader) BootstrapRoutes(ctx context.Context, upser...
method WatchRoutes (line 18) | func (mrl *mockRouteLoader) WatchRoutes(ctx context.Context, upsertRou...
FILE: server/recovery.go
function Recover (line 12) | func Recover(next http.Handler) http.Handler {
FILE: server/recovery_test.go
type testHandler (line 13) | type testHandler struct
method ServeHTTP (line 15) | func (th testHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
function TestRecoverMiddleware (line 19) | func TestRecoverMiddleware(t *testing.T) {
FILE: server/router.go
type Router (line 13) | type Router struct
method Route (line 20) | func (router *Router) Route(req *http.Request) (*weaver.ACL, error) {
method WatchRouteUpdates (line 45) | func (router *Router) WatchRouteUpdates(routeSyncCtx context.Context) {
method BootstrapRoutes (line 49) | func (router *Router) BootstrapRoutes(ctx context.Context) error {
method upsertACL (line 53) | func (router *Router) upsertACL(acl *weaver.ACL) error {
method deleteACL (line 57) | func (router *Router) deleteACL(acl *weaver.ACL) error {
type apiName (line 18) | type apiName
function NewRouter (line 38) | func NewRouter(loader RouteLoader) *Router {
FILE: server/router_test.go
type RouterSuite (line 20) | type RouterSuite struct
method SetupTest (line 26) | func (rs *RouterSuite) SetupTest() {
method TestRouteNotFound (line 38) | func (rs *RouterSuite) TestRouteNotFound() {
method TestRouteInvalidACL (line 46) | func (rs *RouterSuite) TestRouteInvalidACL() {
method TestRouteReturnsACL (line 58) | func (rs *RouterSuite) TestRouteReturnsACL() {
method TestBootstrapRoutesUseBootstrapRoutesOfRouteLoader (line 99) | func (rs *RouterSuite) TestBootstrapRoutesUseBootstrapRoutesOfRouteLoa...
method TestBootstrapRoutesUseBootstrapRoutesOfRouteLoaderFail (line 113) | func (rs *RouterSuite) TestBootstrapRoutesUseBootstrapRoutesOfRouteLoa...
method TestWatchRouteUpdatesCallsWatchRoutesOfLoader (line 127) | func (rs *RouterSuite) TestWatchRouteUpdatesCallsWatchRoutesOfLoader() {
function TestRouterSuite (line 34) | func TestRouterSuite(t *testing.T) {
FILE: server/server.go
type Weaver (line 14) | type Weaver struct
function ShutdownServer (line 18) | func ShutdownServer(ctx context.Context) {
function StartServer (line 22) | func StartServer(ctx context.Context, routeLoader RouteLoader) {
FILE: server/wrapped_response_writer.go
type wrapperResponseWriter (line 5) | type wrapperResponseWriter struct
method Header (line 10) | func (w *wrapperResponseWriter) Header() http.Header {
method Write (line 14) | func (w *wrapperResponseWriter) Write(data []byte) (int, error) {
method WriteHeader (line 18) | func (w *wrapperResponseWriter) WriteHeader(statusCode int) {
FILE: sharder.go
type Sharder (line 3) | type Sharder interface
Condensed preview — 84 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (222K chars).
[
{
"path": ".gitignore",
"chars": 804,
"preview": "# Windows Ignores\nThumbs.db\nehthumbs.db\nehthumbs_vista.db\n*.stackdump\n[Dd]esktop.ini\n$RECYCLE.BIN/\n*.cab\n*.msi\n*.msix\n*."
},
{
"path": ".travis.yml",
"chars": 1918,
"preview": "language: go\n\ngo: 1.11.5\n\nenv:\n global:\n - ETCD_VER=v3.3.0\n - GO111MODULE=on\n - DOCKER_LATEST=latest\n\nmatrix:\n"
},
{
"path": "CONTRIBUTING.md",
"chars": 1721,
"preview": "# Weaver - Contributing\n\nWeaver `github.com/gojektech/weaver` is an open-source project. \nIt is licensed using the [Apac"
},
{
"path": "Dockerfile",
"chars": 388,
"preview": "FROM golang:1.11.5-alpine as base\n\nENV GO111MODULE on\n\nRUN apk --no-cache add gcc g++ make ca-certificates git\nRUN mkdir"
},
{
"path": "LICENSE",
"chars": 11357,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "Makefile",
"chars": 1847,
"preview": ".PHONY: all\n\nall: build fmt vet lint test coverage\ndefault: build fmt vet lint test\n\nALL_PACKAGES=$(shell go list ./... "
},
{
"path": "README.md",
"chars": 3703,
"preview": "# Weaver - A modern HTTP Proxy with Advanced features\n\n<p align=\"center\"><img src=\"docs/weaver-logo.png\" width=\"360\"></p"
},
{
"path": "acl.go",
"chars": 501,
"preview": "package weaver\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n)\n\n// ACL - Connects to an external endpoint\ntype ACL struct {\n\tID "
},
{
"path": "acl_test.go",
"chars": 786,
"preview": "package weaver\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nconst (\n\tvalidACLDoc = `{\n\t\t\"id\": \"gojek_h"
},
{
"path": "backend.go",
"chars": 1217,
"preview": "package weaver\n\nimport (\n\t\"net\"\n\t\"net/http\"\n\t\"net/http/httputil\"\n\t\"net/url\"\n\t\"time\"\n\n\t\"github.com/gojektech/weaver/confi"
},
{
"path": "backend_test.go",
"chars": 724,
"preview": "package weaver\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/require\"\n)\n\nfun"
},
{
"path": "cmd/weaver-server/main.go",
"chars": 1719,
"preview": "package main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"syscall\"\n\n\traven \"github.com/getsentry/raven-go\"\n\t\""
},
{
"path": "config/config.go",
"chars": 3410,
"preview": "package config\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"net/http\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\tetcd \"github.com/coreos/etcd/client\"\n"
},
{
"path": "config/config_test.go",
"chars": 2416,
"preview": "package config\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/te"
},
{
"path": "config/newrelic.go",
"chars": 305,
"preview": "package config\n\nimport (\n\tnewrelic \"github.com/newrelic/go-agent\"\n)\n\nfunc loadNewRelicConfig() newrelic.Config {\n\tconfig"
},
{
"path": "config/proxy.go",
"chars": 1231,
"preview": "package config\n\nimport \"time\"\n\ntype ProxyConfig struct {\n\tproxyDialerTimeoutInMS int\n\tproxyDialerKeepAliveInMS int\n\tpr"
},
{
"path": "config/statsd.go",
"chars": 894,
"preview": "package config\n\ntype StatsDConfig struct {\n\tprefix string\n\tflushPeriodInSeconds int\n\thost "
},
{
"path": "deployment/weaver/.helmignore",
"chars": 342,
"preview": "# Patterns to ignore when building packages.\n# This supports shell glob matching, relative path matching, and\n# negation"
},
{
"path": "deployment/weaver/Chart.yaml",
"chars": 207,
"preview": "apiVersion: v1\nappVersion: \"1.0\"\ndescription: A Helm chart to deploy weaver along with etcd and statsd (configurable)\nna"
},
{
"path": "deployment/weaver/README.md",
"chars": 1612,
"preview": "# Deploying to Kubernetes\n\nYou can deploy to Kubernetes with the helm charts available in this repo.\n\n### Deploying with"
},
{
"path": "deployment/weaver/templates/_helpers.tpl",
"chars": 1042,
"preview": "{{/* vim: set filetype=mustache: */}}\n{{/*\nExpand the name of the chart.\n*/}}\n{{- define \"weaver.name\" -}}\n{{- default ."
},
{
"path": "deployment/weaver/templates/deployment.yaml",
"chars": 2083,
"preview": "apiVersion: extensions/v1beta1\nkind: Deployment\nname: {{ template \"weaver.fullname\" . }}\nmetadata:\n name: {{ template \""
},
{
"path": "deployment/weaver/templates/service.yaml",
"chars": 2361,
"preview": "apiVersion: v1\nkind: Service\nmetadata:\n {{- if .Values.service.annotations }}\n annotations:\n {{- toYaml .Values.ser"
},
{
"path": "deployment/weaver/templates/statefulset.yaml",
"chars": 8588,
"preview": "{{- if .Values.etcd.enabled }}\napiVersion: apps/v1beta1\nkind: StatefulSet\nmetadata:\n {{- if .Values.service.annotations"
},
{
"path": "deployment/weaver/values-env.yaml",
"chars": 1105,
"preview": "weaver:\n env:\n - name: \"PROXY_HOST\"\n value: \"0.0.0.0\"\n - name: \"PROXY_PORT\"\n value: \"8080\"\n - name: "
},
{
"path": "deployment/weaver/values.yaml",
"chars": 1815,
"preview": "# Default values for weaver.\n# This is a YAML-formatted file.\n# Declare variables to be passed into your templates.\n\nrep"
},
{
"path": "docker-compose.yml",
"chars": 1261,
"preview": "version: '3.4'\n\nservices:\n dev_statsd:\n image: gojektech/statsd:0.7.2\n hostname: dev-statsd\n ports:\n - \"1"
},
{
"path": "docs/weaver_acls.md",
"chars": 8072,
"preview": "## Weaver ACLs\n\nWeaver ACL is a document formatted in JSON used to decide the destination of downstream traffic. An exam"
},
{
"path": "endpoint.go",
"chars": 1593,
"preview": "package weaver\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\n\t\"github.com/gojektech/weaver/pkg/matcher\"\n\t\"github.com/pk"
},
{
"path": "endpoint_test.go",
"chars": 1058,
"preview": "package weaver\n\nimport (\n\t\"encoding/json\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testif"
},
{
"path": "etcd/aclkey.go",
"chars": 467,
"preview": "package etcd\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/gojektech/weaver\"\n)\n\nconst (\n\t// ACLKeyFormat - Format for a ACL's key in a "
},
{
"path": "etcd/routeloader.go",
"chars": 4735,
"preview": "package etcd\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"sort\"\n\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/gojektech"
},
{
"path": "etcd/routeloader_test.go",
"chars": 7513,
"preview": "package etcd\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"reflect\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/gojektech/weave"
},
{
"path": "examples/body_lookup/Dockerfile",
"chars": 235,
"preview": "FROM golang:1.11.5-alpine as base\n\nENV GO111MODULE off\n\nRUN mkdir /estimate\nADD . /estimate\nWORKDIR /estimate\n\nRUN go bu"
},
{
"path": "examples/body_lookup/README.md",
"chars": 3526,
"preview": "# Backend Lookup\n\nIn this example, will deploy etcd and weaver to kubernetes and apply a simple etcd to shard between Si"
},
{
"path": "examples/body_lookup/estimate_acl.json",
"chars": 458,
"preview": "{\n \"id\": \"estimator\",\n \"criterion\" : \"Method(`POST`) && Path(`/estimate`)\",\n \"endpoint\" : {\n \"shard_expr\": \".curre"
},
{
"path": "examples/body_lookup/estimator/Chart.yaml",
"chars": 172,
"preview": "apiVersion: v1\nappVersion: \"1.0\"\ndescription: A Helm chart to deploy estimator\nname: estimator \nversion: 0.1.0\nmaintaine"
},
{
"path": "examples/body_lookup/estimator/templates/_helpers.tpl",
"chars": 1051,
"preview": "{{/* vim: set filetype=mustache: */}}\n{{/*\nExpand the name of the chart.\n*/}}\n{{- define \"estimator.name\" -}}\n{{- defaul"
},
{
"path": "examples/body_lookup/estimator/templates/deployment.yaml",
"chars": 1818,
"preview": "apiVersion: extensions/v1beta1\nkind: Deployment\nname: {{ template \"estimator.fullname\" . }}\nmetadata:\n name: {{ templat"
},
{
"path": "examples/body_lookup/estimator/templates/service.yaml",
"chars": 1522,
"preview": "apiVersion: v1\nkind: Service\nmetadata:\n {{- if .Values.service.annotations }}\n annotations:\n {{- toYaml .Values.ser"
},
{
"path": "examples/body_lookup/estimator/values-id.yaml",
"chars": 103,
"preview": "# Default env values for Indonesia cluster\nenv:\n maxAmount: 100000\n minAmount: 1000\n currency: IDR\n\n"
},
{
"path": "examples/body_lookup/estimator/values-sg.yaml",
"chars": 97,
"preview": "# Default env values for Singapore cluster\nenv:\n maxAmount: 100\n minAmount: 10\n currency: SGD\n"
},
{
"path": "examples/body_lookup/estimator/values.yaml",
"chars": 905,
"preview": "# Default values for weaver.\n# This is a YAML-formatted file.\n# Declare variables to be passed into your templates.\n\nrep"
},
{
"path": "examples/body_lookup/main.go",
"chars": 1165,
"preview": "package main\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"log\"\n\t\"math\"\n\t\"math/rand\"\n\t\"net/http\"\n\t\"os\"\n\t\"strconv\"\n)\n\ntype estimati"
},
{
"path": "go.mod",
"chars": 3630,
"preview": "module github.com/gojektech/weaver\n\nrequire (\n\tgithub.com/certifi/gocertifi v0.0.0-20170123212243-03be5e6bb987 // indire"
},
{
"path": "go.sum",
"chars": 17252,
"preview": "cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ngithub.com/beorn7/perks v0.0.0-201803"
},
{
"path": "goreleaser.yml",
"chars": 224,
"preview": "builds:\n - main: ./cmd/weaver-server/\n goos:\n - linux\n - darwin\n - windows\n goarch:\n - amd64\n"
},
{
"path": "pkg/instrumentation/newrelic.go",
"chars": 1427,
"preview": "package instrumentation\n\nimport (\n\t\"context\"\n\t\"log\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/gojektech/weaver/config\"\n\tnewrelic"
},
{
"path": "pkg/instrumentation/statsd.go",
"chars": 2689,
"preview": "package instrumentation\n\nimport (\n\t\"fmt\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/gojektech/weaver/config\"\n\t\"github.com/gojekte"
},
{
"path": "pkg/logger/logger.go",
"chars": 3646,
"preview": "package logger\n\nimport (\n\t\"net/http\"\n\t\"os\"\n\n\t\"github.com/gojektech/weaver/config\"\n\t\"github.com/gojektech/weaver/pkg/util"
},
{
"path": "pkg/matcher/matcher.go",
"chars": 2342,
"preview": "package matcher\n\nimport (\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"g"
},
{
"path": "pkg/matcher/matcher_test.go",
"chars": 5286,
"preview": "package matcher\n\nimport (\n\t\"bytes\"\n\t\"net/http/httptest\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/s"
},
{
"path": "pkg/shard/domain.go",
"chars": 1758,
"preview": "package shard\n\nimport (\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/gojektech/weaver/config\"\n\t\"github.co"
},
{
"path": "pkg/shard/hashring.go",
"chars": 3063,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"regexp\"\n\t\"strconv\"\n\n\t\"github.com/gojekfarm/hashring\"\n\t\"github.com/goje"
},
{
"path": "pkg/shard/hashring_test.go",
"chars": 13802,
"preview": "package shard_test\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"math\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/goje"
},
{
"path": "pkg/shard/lookup.go",
"chars": 587,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\n\t\"github.com/gojektech/weaver\"\n)\n\nfunc NewLookupStrategy(data json.RawMessage)"
},
{
"path": "pkg/shard/lookup_test.go",
"chars": 3054,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify"
},
{
"path": "pkg/shard/modulo.go",
"chars": 788,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"strconv\"\n\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/pkg/errors\"\n)\n\nfunc New"
},
{
"path": "pkg/shard/modulo_test.go",
"chars": 4186,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify"
},
{
"path": "pkg/shard/no.go",
"chars": 844,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/pkg/errors\"\n)\n\nfunc NewNoSt"
},
{
"path": "pkg/shard/no_test.go",
"chars": 2175,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify"
},
{
"path": "pkg/shard/prefix_lookup.go",
"chars": 1458,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"errors\"\n\t\"strings\"\n\n\t\"github.com/gojektech/weaver\"\n)\n\nconst (\n\tdefaultBackend"
},
{
"path": "pkg/shard/prefix_lookup_test.go",
"chars": 5070,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify"
},
{
"path": "pkg/shard/s2.go",
"chars": 3961,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/gojek"
},
{
"path": "pkg/shard/s2_test.go",
"chars": 8865,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar (\n\ts2ShardConfigWithou"
},
{
"path": "pkg/shard/shard.go",
"chars": 654,
"preview": "package shard\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\n\t\"github.com/gojektech/weaver\"\n)\n\nfunc New(name string, cfg json.RawMes"
},
{
"path": "pkg/util/s2.go",
"chars": 762,
"preview": "package util\n\nimport (\n\t\"sort\"\n\n\t\"github.com/golang/geo/s2\"\n)\n\ntype s2List []s2.CellID\n\nfunc (s2l s2List) Len() int {\n\tr"
},
{
"path": "pkg/util/s2_test.go",
"chars": 1294,
"preview": "package util\n\nimport (\n\t\"testing\"\n\n\t\"github.com/golang/geo/s2\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestS2Len(t"
},
{
"path": "pkg/util/util.go",
"chars": 509,
"preview": "package util\n\nimport (\n\t\"strings\"\n\t\"unicode\"\n)\n\nfunc ToSnake(in string) string {\n\trunes := []rune(in)\n\tlength := len(run"
},
{
"path": "pkg/util/util_test.go",
"chars": 681,
"preview": "package util\n\nimport (\n\t\"testing\"\n)\n\ntype SnakeTest struct {\n\tinput string\n\toutput string\n}\n\nvar tests = []SnakeTest{\n\t"
},
{
"path": "server/error.go",
"chars": 2020,
"preview": "package server\n\nimport (\n\t\"encoding/json\"\n\t\"net/http\"\n\n\t\"github.com/gojektech/weaver/pkg/instrumentation\"\n)\n\ntype weaver"
},
{
"path": "server/error_test.go",
"chars": 1289,
"preview": "package server\n\nimport (\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc Test"
},
{
"path": "server/handler.go",
"chars": 2409,
"preview": "package server\n\nimport (\n\t\"net/http\"\n\n\t\"github.com/gojektech/weaver/config\"\n\t\"github.com/gojektech/weaver/pkg/instrument"
},
{
"path": "server/handler_test.go",
"chars": 6520,
"preview": "package server\n\nimport (\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"github.com/gojektech/weaver/pkg/shard\"\n\t\"net/http\"\n\t\"net/htt"
},
{
"path": "server/loader.go",
"chars": 314,
"preview": "package server\n\nimport (\n\t\"context\"\n\n\t\"github.com/gojektech/weaver\"\n)\n\ntype UpsertRouteFunc func(*weaver.ACL) error\ntype"
},
{
"path": "server/mock.go",
"chars": 483,
"preview": "package server\n\nimport (\n\t\"context\"\n\n\t\"github.com/stretchr/testify/mock\"\n)\n\ntype mockRouteLoader struct {\n\tmock.Mock\n}\n\n"
},
{
"path": "server/recovery.go",
"chars": 868,
"preview": "package server\n\nimport (\n\t\"fmt\"\n\t\"net/http\"\n\n\traven \"github.com/getsentry/raven-go\"\n\t\"github.com/gojektech/weaver/pkg/in"
},
{
"path": "server/recovery_test.go",
"chars": 772,
"preview": "package server\n\nimport (\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"testing\"\n\n\t\"github.com/gojektech/weaver/config\"\n\t\"github.com"
},
{
"path": "server/router.go",
"chars": 1263,
"preview": "package server\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/pkg/errors\"\n\t\"github"
},
{
"path": "server/router_test.go",
"chars": 3584,
"preview": "package server\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"github.com/gojektech/weaver\"\n\t\"github.com/gojektech/wea"
},
{
"path": "server/server.go",
"chars": 1346,
"preview": "package server\n\nimport (\n\t\"context\"\n\t\"log\"\n\t\"net/http\"\n\n\t\"github.com/gojektech/weaver/config\"\n\t\"github.com/gojektech/wea"
},
{
"path": "server/wrapped_response_writer.go",
"chars": 442,
"preview": "package server\n\nimport \"net/http\"\n\ntype wrapperResponseWriter struct {\n\tstatusCode int\n\thttp.ResponseWriter\n}\n\nfunc (w *"
},
{
"path": "sharder.go",
"chars": 80,
"preview": "package weaver\n\ntype Sharder interface {\n\tShard(key string) (*Backend, error)\n}\n"
},
{
"path": "weaver.conf.yaml.sample",
"chars": 656,
"preview": "SERVER_HOST: \"127.0.0.1\"\nSERVER_PORT: \"8080\"\nPROXY_HOST: \"127.0.0.1\"\nPROXY_PORT: \"8081\"\nPROXY_MAX_IDLE_CONNS: \"50\"\nPROXY"
}
]
About this extraction
This page contains the full source code of the gojek/weaver GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 84 files (197.2 KB), approximately 62.1k tokens, and a symbol index with 354 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.