Showing preview only (309K chars total). Download the full file or copy to clipboard to get everything.
Repository: vmware/vmware-go-kcl
Branch: master
Commit: 898bb33c52af
Files: 53
Total size: 292.5 KB
Directory structure:
gitextract_khtdggzz/
├── .gitignore
├── .gitreview
├── CONTRIBUTING.md
├── HyperMake
├── LICENSE
├── README.md
├── clientlibrary/
│ ├── checkpoint/
│ │ ├── checkpointer.go
│ │ ├── dynamodb-checkpointer.go
│ │ └── dynamodb-checkpointer_test.go
│ ├── config/
│ │ ├── config.go
│ │ ├── config_test.go
│ │ ├── initial-stream-pos.go
│ │ └── kcl-config.go
│ ├── interfaces/
│ │ ├── inputs.go
│ │ ├── record-processor-checkpointer.go
│ │ ├── record-processor.go
│ │ └── sequence-number.go
│ ├── metrics/
│ │ ├── cloudwatch/
│ │ │ └── cloudwatch.go
│ │ ├── interfaces.go
│ │ └── prometheus/
│ │ └── prometheus.go
│ ├── partition/
│ │ └── partition.go
│ ├── utils/
│ │ ├── awserr.go
│ │ ├── random.go
│ │ ├── random_test.go
│ │ └── uuid.go
│ └── worker/
│ ├── common-shard-consumer.go
│ ├── fan-out-shard-consumer.go
│ ├── polling-shard-consumer.go
│ ├── record-processor-checkpointer.go
│ ├── worker-fan-out.go
│ └── worker.go
├── go.mod
├── go.sum
├── logger/
│ ├── logger.go
│ ├── logger_test.go
│ ├── logrus.go
│ ├── zap/
│ │ ├── zap.go
│ │ └── zap_test.go
│ └── zerolog/
│ ├── zerolog.go
│ └── zerolog_test.go
├── support/
│ ├── scripts/
│ │ ├── check.sh
│ │ ├── ci.sh
│ │ ├── functions.sh
│ │ └── test.sh
│ └── toolchain/
│ ├── HyperMake
│ └── docker/
│ └── Dockerfile
└── test/
├── lease_stealing_util_test.go
├── logger_test.go
├── record_processor_test.go
├── record_publisher_test.go
├── worker_custom_test.go
├── worker_lease_stealing_test.go
└── worker_test.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
/gen
/vendor
!/vendor/manifest
/bin
/pkg
/tmp
/log
/vms
/run
/go
.hmake
.hmakerc
.project
.idea
.vscode
*_mock_test.go
filenames
.DS_Store
================================================
FILE: .gitreview
================================================
[gerrit]
host=review.ec.eng.vmware.com
port=29418
project=cascade-kinesis-client
defaultbranch=develop
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to vmware-go-kcl
The vmware-go-kcl project team welcomes contributions from the community. Before you start working with vmware-go-kcl, please read our [Developer Certificate of Origin](https://cla.vmware.com/dco). All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch.
## Contribution Flow
This is a rough outline of what a contributor's workflow looks like:
- Create a topic branch from where you want to base your work
- Make commits of logical units
- Make sure your commit messages are in the proper format (see below)
- Push your changes to a topic branch in your fork of the repository
- Submit a pull request
Example:
``` shell
git remote add upstream https://github.com/vmware/vmware-go-kcl.git
git checkout -b my-new-feature master
git commit -a
git push origin my-new-feature
```
### Staying In Sync With Upstream
When your branch gets out of sync with the vmware/master branch, use the following to update:
``` shell
git checkout my-new-feature
git fetch -a
git pull --rebase upstream master
git push --force-with-lease origin my-new-feature
```
### Updating pull requests
If your PR fails to pass CI or needs changes based on code review, you'll most likely want to squash these changes into
existing commits.
If your pull request contains a single commit or your changes are related to the most recent commit, you can simply
amend the commit.
``` shell
git add .
git commit --amend
git push --force-with-lease origin my-new-feature
```
If you need to squash changes into an earlier commit, you can use:
``` shell
git add .
git commit --fixup <commit>
git rebase -i --autosquash master
git push --force-with-lease origin my-new-feature
```
Be sure to add a comment to the PR indicating your new changes are ready to review, as GitHub does not generate a
notification when you git push.
### Formatting Commit Messages
We follow the conventions on [How to Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
Be sure to include any related GitHub issue references in the commit message. See
[GFM syntax](https://guides.github.com/features/mastering-markdown/#GitHub-flavored-markdown) for referencing issues
and commits.
## Reporting Bugs and Creating Issues
When opening a new issue, try to roughly follow the commit message format conventions above.
================================================
FILE: HyperMake
================================================
---
format: hypermake.v0
name: cascade-kinesis-client
description: Kinesis Client in Go
targets:
rebuild-toolchain:
description: build toolchain image
watches:
- support/toolchain/docker
build: support/toolchain/docker
toolchain:
description: placeholder for additional toolchain dependencies
deps:
description: download dependencies to local cache
after:
- toolchain
watches:
- go.mod
cmds:
- go mod download
- go mod vendor
- go mod tidy
build:
description: build source code
after:
- 'build-*'
test:
description: run unit tests
after:
- deps
- check
always: true
cmds:
- ./support/scripts/test.sh
ci:
description: run CI tests
after:
- deps
cmds:
- ./support/scripts/ci.sh
checkfmt:
description: check code format
after:
- toolchain
watches:
- support/scripts/check.sh
always: true
cmds:
- ./support/scripts/check.sh fmt
lint:
description: run lint to check code
after:
- toolchain
watches:
- support/scripts/check.sh
always: true
cmds:
- ./support/scripts/check.sh lint
scanast:
description: run Go AST security scan
after:
- toolchain
watches:
- '**/**/*.go'
- './support/scripts/check.sh'
cmds:
- ./support/scripts/check.sh scanast
check:
description: run all code checks
after:
- checkfmt
- lint
- scanast
settings:
default-targets:
- test
docker:
image: 'vmware/go-kcl-toolchain:0.1.4'
src-volume: /go/src/github.com/vmware/vmware-go-kcl
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2018 VMware, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# VMware-Go-KCL
[](https://goreportcard.com/report/github.com/vmware/vmware-go-kcl)
[](https://opensource.org/licenses/MIT)
## Overview
[Amazon Kinesis](https://aws.amazon.com/kinesis/data-streams/) enables real-time processing of streaming data at massive scale. Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing.
The **VMware Kinesis Client Library for GO** (VMware-Go-KCL) enables Go developers to easily consume and process data from [Amazon Kinesis][kinesis].
**VMware-Go-KCL** brings Go/Kubernetes community with Go language native implementation of KCL matching **exactly the same** API and functional spec of original [Java KCL v2.0](https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html) without the resource overhead of installing Java based MultiLangDaemon.
Besides, [vmware-go-kcl-v2](https://github.com/vmware/vmware-go-kcl-v2) is the v2 version of VMWare KCL for the Go programming language by utilizing [AWS Go SDK V2](https://github.com/aws/aws-sdk-go-v2).
## Try it out
### Prerequisites
- Install [Go](https://golang.org/)
- Install [docker](https://www.docker.com)
- Install [HyperMake](https://evo-cloud.github.io/hmake)
- Config [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
Make sure hmake is version 1.3.1 or above and go is version 1.11 or above
```sh
hmake --version
1.3.1
```
Make sure to launch Docker daemon with specified DNS server `--dns DNS-SERVER-IP`
On Ubuntu, update the file `/etc/default/docker` to put `--dns DNS-SERVER-IP` in `DOCKER_OPTS`.
On Mac, set DNS in _Docker Preferences_ – _Daemon_ – _Insecure registries_
### Build & Run
```sh
hmake
# security scan
hmake scanast
# run test
hmake check
# run integration test
# update the worker_test.go to let it point to your Kinesis stream
hmake test
```
## Documentation
VMware-Go-KCL matches exactly the same interface and programming model from original Amazon KCL, the best place for getting reference, tutorial is from Amazon itself:
- [Developing Consumers Using the Kinesis Client Library](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html)
- [Troubleshooting](https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html)
- [Advanced Topics](https://docs.aws.amazon.com/streams/latest/dev/advanced-consumers.html)
## Contributing
The vmware-go-kcl project team welcomes contributions from the community. Before you start working with vmware-go-kcl, please read our [Developer Certificate of Origin](https://cla.vmware.com/dco). All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to [CONTRIBUTING.md](CONTRIBUTING.md).
## License
MIT License
================================================
FILE: clientlibrary/checkpoint/checkpointer.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package checkpoint
import (
"errors"
"fmt"
par "github.com/vmware/vmware-go-kcl/clientlibrary/partition"
)
const (
LeaseKeyKey = "ShardID"
LeaseOwnerKey = "AssignedTo"
LeaseTimeoutKey = "LeaseTimeout"
SequenceNumberKey = "Checkpoint"
ParentShardIdKey = "ParentShardId"
ClaimRequestKey = "ClaimRequest"
// We've completely processed all records in this shard.
ShardEnd = "SHARD_END"
// ErrShardClaimed is returned when shard is claimed
ErrShardClaimed = "Shard is already claimed by another node"
)
type ErrLeaseNotAcquired struct {
cause string
}
func (e ErrLeaseNotAcquired) Error() string {
return fmt.Sprintf("lease not acquired: %s", e.cause)
}
// Checkpointer handles checkpointing when a record has been processed
type Checkpointer interface {
// Init initialises the Checkpoint
Init() error
// GetLease attempts to gain a lock on the given shard
GetLease(*par.ShardStatus, string) error
// CheckpointSequence writes a checkpoint at the designated sequence ID
CheckpointSequence(*par.ShardStatus) error
// FetchCheckpoint retrieves the checkpoint for the given shard
FetchCheckpoint(*par.ShardStatus) error
// RemoveLeaseInfo to remove lease info for shard entry because the shard no longer exists
RemoveLeaseInfo(string) error
// RemoveLeaseOwner to remove lease owner for the shard entry to make the shard available for reassignment
RemoveLeaseOwner(string) error
// New Lease Stealing Methods
// ListActiveWorkers returns active workers and their shards
ListActiveWorkers(map[string]*par.ShardStatus) (map[string][]*par.ShardStatus, error)
// ClaimShard claims a shard for stealing
ClaimShard(*par.ShardStatus, string) error
}
// ErrSequenceIDNotFound is returned by FetchCheckpoint when no SequenceID is found
var ErrSequenceIDNotFound = errors.New("SequenceIDNotFoundForShard")
// ErrShardNotAssigned is returned by ListActiveWorkers when no AssignedTo is found
var ErrShardNotAssigned = errors.New("AssignedToNotFoundForShard")
================================================
FILE: clientlibrary/checkpoint/dynamodb-checkpointer.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package checkpoint
import (
"errors"
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/client"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbiface"
"github.com/vmware/vmware-go-kcl/clientlibrary/config"
par "github.com/vmware/vmware-go-kcl/clientlibrary/partition"
"github.com/vmware/vmware-go-kcl/clientlibrary/utils"
"github.com/vmware/vmware-go-kcl/logger"
)
const (
// ErrInvalidDynamoDBSchema is returned when there are one or more fields missing from the table
ErrInvalidDynamoDBSchema = "The DynamoDB schema is invalid and may need to be re-created"
// NumMaxRetries is the max times of doing retry
NumMaxRetries = 10
)
// DynamoCheckpoint implements the Checkpoint interface using DynamoDB as a backend
type DynamoCheckpoint struct {
log logger.Logger
TableName string
leaseTableReadCapacity int64
leaseTableWriteCapacity int64
LeaseDuration int
svc dynamodbiface.DynamoDBAPI
kclConfig *config.KinesisClientLibConfiguration
Retries int
lastLeaseSync time.Time
}
func NewDynamoCheckpoint(kclConfig *config.KinesisClientLibConfiguration) *DynamoCheckpoint {
checkpointer := &DynamoCheckpoint{
log: kclConfig.Logger,
TableName: kclConfig.TableName,
leaseTableReadCapacity: int64(kclConfig.InitialLeaseTableReadCapacity),
leaseTableWriteCapacity: int64(kclConfig.InitialLeaseTableWriteCapacity),
LeaseDuration: kclConfig.FailoverTimeMillis,
kclConfig: kclConfig,
Retries: NumMaxRetries,
}
return checkpointer
}
// WithDynamoDB is used to provide DynamoDB service
func (checkpointer *DynamoCheckpoint) WithDynamoDB(svc dynamodbiface.DynamoDBAPI) *DynamoCheckpoint {
checkpointer.svc = svc
return checkpointer
}
// Init initialises the DynamoDB Checkpoint
func (checkpointer *DynamoCheckpoint) Init() error {
checkpointer.log.Infof("Creating DynamoDB session")
s, err := session.NewSession(&aws.Config{
Region: aws.String(checkpointer.kclConfig.RegionName),
Endpoint: aws.String(checkpointer.kclConfig.DynamoDBEndpoint),
Credentials: checkpointer.kclConfig.DynamoDBCredentials,
Retryer: client.DefaultRetryer{
NumMaxRetries: checkpointer.Retries,
MinRetryDelay: client.DefaultRetryerMinRetryDelay,
MinThrottleDelay: client.DefaultRetryerMinThrottleDelay,
MaxRetryDelay: client.DefaultRetryerMaxRetryDelay,
MaxThrottleDelay: client.DefaultRetryerMaxRetryDelay,
},
})
if err != nil {
// no need to move forward
checkpointer.log.Fatalf("Failed in getting DynamoDB session for creating Worker: %+v", err)
}
if checkpointer.svc == nil {
checkpointer.svc = dynamodb.New(s)
}
if !checkpointer.doesTableExist() {
return checkpointer.createTable()
}
return nil
}
// GetLease attempts to gain a lock on the given shard
func (checkpointer *DynamoCheckpoint) GetLease(shard *par.ShardStatus, newAssignTo string) error {
newLeaseTimeout := time.Now().Add(time.Duration(checkpointer.LeaseDuration) * time.Millisecond).UTC()
newLeaseTimeoutString := newLeaseTimeout.Format(time.RFC3339)
currentCheckpoint, err := checkpointer.getItem(shard.ID)
if err != nil {
return err
}
isClaimRequestExpired := shard.IsClaimRequestExpired(checkpointer.kclConfig)
var claimRequest string
if checkpointer.kclConfig.EnableLeaseStealing {
if currentCheckpointClaimRequest, ok := currentCheckpoint[ClaimRequestKey]; ok && currentCheckpointClaimRequest.S != nil {
claimRequest = *currentCheckpointClaimRequest.S
if newAssignTo != claimRequest && !isClaimRequestExpired {
checkpointer.log.Debugf("another worker: %s has a claim on this shard. Not going to renew the lease", claimRequest)
return errors.New(ErrShardClaimed)
}
}
}
assignedVar, assignedToOk := currentCheckpoint[LeaseOwnerKey]
leaseVar, leaseTimeoutOk := currentCheckpoint[LeaseTimeoutKey]
var conditionalExpression string
var expressionAttributeValues map[string]*dynamodb.AttributeValue
if !leaseTimeoutOk || !assignedToOk {
conditionalExpression = "attribute_not_exists(AssignedTo)"
} else {
assignedTo := *assignedVar.S
leaseTimeout := *leaseVar.S
currentLeaseTimeout, err := time.Parse(time.RFC3339, leaseTimeout)
if err != nil {
return err
}
if checkpointer.kclConfig.EnableLeaseStealing {
if time.Now().UTC().Before(currentLeaseTimeout) && assignedTo != newAssignTo && !isClaimRequestExpired {
return ErrLeaseNotAcquired{"current lease timeout not yet expired"}
}
} else {
if time.Now().UTC().Before(currentLeaseTimeout) && assignedTo != newAssignTo {
return ErrLeaseNotAcquired{"current lease timeout not yet expired"}
}
}
checkpointer.log.Debugf("Attempting to get a lock for shard: %s, leaseTimeout: %s, assignedTo: %s, newAssignedTo: %s", shard.ID, currentLeaseTimeout, assignedTo, newAssignTo)
conditionalExpression = "ShardID = :id AND AssignedTo = :assigned_to AND LeaseTimeout = :lease_timeout"
expressionAttributeValues = map[string]*dynamodb.AttributeValue{
":id": {
S: aws.String(shard.ID),
},
":assigned_to": {
S: aws.String(assignedTo),
},
":lease_timeout": {
S: aws.String(leaseTimeout),
},
}
}
marshalledCheckpoint := map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String(shard.ID),
},
LeaseOwnerKey: {
S: aws.String(newAssignTo),
},
LeaseTimeoutKey: {
S: aws.String(newLeaseTimeoutString),
},
}
if len(shard.ParentShardId) > 0 {
marshalledCheckpoint[ParentShardIdKey] = &dynamodb.AttributeValue{S: aws.String(shard.ParentShardId)}
}
if checkpoint := shard.GetCheckpoint(); checkpoint != "" {
marshalledCheckpoint[SequenceNumberKey] = &dynamodb.AttributeValue{
S: aws.String(checkpoint),
}
}
if checkpointer.kclConfig.EnableLeaseStealing {
if claimRequest != "" && claimRequest == newAssignTo && !isClaimRequestExpired {
if expressionAttributeValues == nil {
expressionAttributeValues = make(map[string]*dynamodb.AttributeValue)
}
conditionalExpression = conditionalExpression + " AND ClaimRequest = :claim_request"
expressionAttributeValues[":claim_request"] = &dynamodb.AttributeValue{
S: &claimRequest,
}
}
}
err = checkpointer.conditionalUpdate(conditionalExpression, expressionAttributeValues, marshalledCheckpoint)
if err != nil {
if utils.AWSErrCode(err) == dynamodb.ErrCodeConditionalCheckFailedException {
return ErrLeaseNotAcquired{dynamodb.ErrCodeConditionalCheckFailedException}
}
return err
}
shard.Mux.Lock()
shard.AssignedTo = newAssignTo
shard.LeaseTimeout = newLeaseTimeout
shard.Mux.Unlock()
return nil
}
// CheckpointSequence writes a checkpoint at the designated sequence ID
func (checkpointer *DynamoCheckpoint) CheckpointSequence(shard *par.ShardStatus) error {
leaseTimeout := shard.GetLeaseTimeout().UTC().Format(time.RFC3339)
marshalledCheckpoint := map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String(shard.ID),
},
SequenceNumberKey: {
S: aws.String(shard.GetCheckpoint()),
},
LeaseOwnerKey: {
S: aws.String(shard.GetLeaseOwner()),
},
LeaseTimeoutKey: {
S: aws.String(leaseTimeout),
},
}
if len(shard.ParentShardId) > 0 {
marshalledCheckpoint[ParentShardIdKey] = &dynamodb.AttributeValue{S: &shard.ParentShardId}
}
return checkpointer.saveItem(marshalledCheckpoint)
}
// FetchCheckpoint retrieves the checkpoint for the given shard
func (checkpointer *DynamoCheckpoint) FetchCheckpoint(shard *par.ShardStatus) error {
checkpoint, err := checkpointer.getItem(shard.ID)
if err != nil {
return err
}
sequenceID, ok := checkpoint[SequenceNumberKey]
if !ok {
return ErrSequenceIDNotFound
}
checkpointer.log.Debugf("Retrieved Shard Iterator %s", *sequenceID.S)
shard.SetCheckpoint(aws.StringValue(sequenceID.S))
if assignedTo, ok := checkpoint[LeaseOwnerKey]; ok {
shard.SetLeaseOwner(aws.StringValue(assignedTo.S))
}
// Use up-to-date leaseTimeout to avoid ConditionalCheckFailedException when claiming
if leaseTimeout, ok := checkpoint[LeaseTimeoutKey]; ok && leaseTimeout.S != nil {
currentLeaseTimeout, err := time.Parse(time.RFC3339, aws.StringValue(leaseTimeout.S))
if err != nil {
return err
}
shard.LeaseTimeout = currentLeaseTimeout
}
return nil
}
// RemoveLeaseInfo to remove lease info for shard entry in dynamoDB because the shard no longer exists in Kinesis
func (checkpointer *DynamoCheckpoint) RemoveLeaseInfo(shardID string) error {
err := checkpointer.removeItem(shardID)
if err != nil {
checkpointer.log.Errorf("Error in removing lease info for shard: %s, Error: %+v", shardID, err)
} else {
checkpointer.log.Infof("Lease info for shard: %s has been removed.", shardID)
}
return err
}
// RemoveLeaseOwner to remove lease owner for the shard entry
func (checkpointer *DynamoCheckpoint) RemoveLeaseOwner(shardID string) error {
input := &dynamodb.UpdateItemInput{
TableName: aws.String(checkpointer.TableName),
Key: map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String(shardID),
},
},
UpdateExpression: aws.String("remove " + LeaseOwnerKey),
ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
":assigned_to": {
S: aws.String(checkpointer.kclConfig.WorkerID),
},
},
ConditionExpression: aws.String("AssignedTo = :assigned_to"),
}
_, err := checkpointer.svc.UpdateItem(input)
return err
}
// ListActiveWorkers returns a map of workers and their shards
func (checkpointer *DynamoCheckpoint) ListActiveWorkers(shardStatus map[string]*par.ShardStatus) (map[string][]*par.ShardStatus, error) {
err := checkpointer.syncLeases(shardStatus)
if err != nil {
return nil, err
}
workers := map[string][]*par.ShardStatus{}
for _, shard := range shardStatus {
if shard.GetCheckpoint() == ShardEnd {
continue
}
leaseOwner := shard.GetLeaseOwner()
if leaseOwner == "" {
checkpointer.log.Debugf("Shard Not Assigned Error. ShardID: %s, WorkerID: %s", shard.ID, checkpointer.kclConfig.WorkerID)
return nil, ErrShardNotAssigned
}
if w, ok := workers[leaseOwner]; ok {
workers[leaseOwner] = append(w, shard)
} else {
workers[leaseOwner] = []*par.ShardStatus{shard}
}
}
return workers, nil
}
// ClaimShard places a claim request on a shard to signal a steal attempt
func (checkpointer *DynamoCheckpoint) ClaimShard(shard *par.ShardStatus, claimID string) error {
err := checkpointer.FetchCheckpoint(shard)
if err != nil && err != ErrSequenceIDNotFound {
return err
}
leaseTimeoutString := shard.GetLeaseTimeout().Format(time.RFC3339)
conditionalExpression := `ShardID = :id AND LeaseTimeout = :lease_timeout AND attribute_not_exists(ClaimRequest)`
expressionAttributeValues := map[string]*dynamodb.AttributeValue{
":id": {
S: aws.String(shard.ID),
},
":lease_timeout": {
S: aws.String(leaseTimeoutString),
},
}
marshalledCheckpoint := map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: &shard.ID,
},
LeaseTimeoutKey: {
S: &leaseTimeoutString,
},
SequenceNumberKey: {
S: &shard.Checkpoint,
},
ClaimRequestKey: {
S: &claimID,
},
}
if leaseOwner := shard.GetLeaseOwner(); leaseOwner == "" {
conditionalExpression += " AND attribute_not_exists(AssignedTo)"
} else {
marshalledCheckpoint[LeaseOwnerKey] = &dynamodb.AttributeValue{S: &leaseOwner}
conditionalExpression += "AND AssignedTo = :assigned_to"
expressionAttributeValues[":assigned_to"] = &dynamodb.AttributeValue{S: &leaseOwner}
}
if checkpoint := shard.GetCheckpoint(); checkpoint == "" {
conditionalExpression += " AND attribute_not_exists(Checkpoint)"
} else if checkpoint == ShardEnd {
conditionalExpression += " AND Checkpoint <> :checkpoint"
expressionAttributeValues[":checkpoint"] = &dynamodb.AttributeValue{S: aws.String(ShardEnd)}
} else {
conditionalExpression += " AND Checkpoint = :checkpoint"
expressionAttributeValues[":checkpoint"] = &dynamodb.AttributeValue{S: &checkpoint}
}
if shard.ParentShardId == "" {
conditionalExpression += " AND attribute_not_exists(ParentShardId)"
} else {
marshalledCheckpoint[ParentShardIdKey] = &dynamodb.AttributeValue{S: aws.String(shard.ParentShardId)}
conditionalExpression += " AND ParentShardId = :parent_shard"
expressionAttributeValues[":parent_shard"] = &dynamodb.AttributeValue{S: &shard.ParentShardId}
}
return checkpointer.conditionalUpdate(conditionalExpression, expressionAttributeValues, marshalledCheckpoint)
}
func (checkpointer *DynamoCheckpoint) syncLeases(shardStatus map[string]*par.ShardStatus) error {
log := checkpointer.kclConfig.Logger
if (checkpointer.lastLeaseSync.Add(time.Duration(checkpointer.kclConfig.LeaseSyncingTimeIntervalMillis) * time.Millisecond)).After(time.Now()) {
return nil
}
checkpointer.lastLeaseSync = time.Now()
input := &dynamodb.ScanInput{
ProjectionExpression: aws.String(fmt.Sprintf("%s,%s,%s", LeaseKeyKey, LeaseOwnerKey, SequenceNumberKey)),
Select: aws.String("SPECIFIC_ATTRIBUTES"),
TableName: aws.String(checkpointer.kclConfig.TableName),
}
err := checkpointer.svc.ScanPages(input,
func(pages *dynamodb.ScanOutput, lastPage bool) bool {
results := pages.Items
for _, result := range results {
shardId, foundShardId := result[LeaseKeyKey]
assignedTo, foundAssignedTo := result[LeaseOwnerKey]
checkpoint, foundCheckpoint := result[SequenceNumberKey]
if !foundShardId || !foundAssignedTo || !foundCheckpoint {
continue
}
if shard, ok := shardStatus[aws.StringValue(shardId.S)]; ok {
shard.SetLeaseOwner(aws.StringValue(assignedTo.S))
shard.SetCheckpoint(aws.StringValue(checkpoint.S))
}
}
return !lastPage
})
if err != nil {
log.Debugf("Error performing SyncLeases. Error: %+v ", err)
return err
}
log.Debugf("Lease sync completed. Next lease sync will occur in %s", time.Duration(checkpointer.kclConfig.LeaseSyncingTimeIntervalMillis)*time.Millisecond)
return nil
}
func (checkpointer *DynamoCheckpoint) createTable() error {
input := &dynamodb.CreateTableInput{
AttributeDefinitions: []*dynamodb.AttributeDefinition{
{
AttributeName: aws.String(LeaseKeyKey),
AttributeType: aws.String("S"),
},
},
KeySchema: []*dynamodb.KeySchemaElement{
{
AttributeName: aws.String(LeaseKeyKey),
KeyType: aws.String("HASH"),
},
},
ProvisionedThroughput: &dynamodb.ProvisionedThroughput{
ReadCapacityUnits: aws.Int64(checkpointer.leaseTableReadCapacity),
WriteCapacityUnits: aws.Int64(checkpointer.leaseTableWriteCapacity),
},
TableName: aws.String(checkpointer.TableName),
}
_, err := checkpointer.svc.CreateTable(input)
return err
}
func (checkpointer *DynamoCheckpoint) doesTableExist() bool {
input := &dynamodb.DescribeTableInput{
TableName: aws.String(checkpointer.TableName),
}
_, err := checkpointer.svc.DescribeTable(input)
return err == nil
}
func (checkpointer *DynamoCheckpoint) saveItem(item map[string]*dynamodb.AttributeValue) error {
return checkpointer.putItem(&dynamodb.PutItemInput{
TableName: aws.String(checkpointer.TableName),
Item: item,
})
}
func (checkpointer *DynamoCheckpoint) conditionalUpdate(conditionExpression string, expressionAttributeValues map[string]*dynamodb.AttributeValue, item map[string]*dynamodb.AttributeValue) error {
return checkpointer.putItem(&dynamodb.PutItemInput{
ConditionExpression: aws.String(conditionExpression),
TableName: aws.String(checkpointer.TableName),
Item: item,
ExpressionAttributeValues: expressionAttributeValues,
})
}
func (checkpointer *DynamoCheckpoint) putItem(input *dynamodb.PutItemInput) error {
_, err := checkpointer.svc.PutItem(input)
return err
}
func (checkpointer *DynamoCheckpoint) getItem(shardID string) (map[string]*dynamodb.AttributeValue, error) {
item, err := checkpointer.svc.GetItem(&dynamodb.GetItemInput{
TableName: aws.String(checkpointer.TableName),
Key: map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String(shardID),
},
},
})
return item.Item, err
}
func (checkpointer *DynamoCheckpoint) removeItem(shardID string) error {
_, err := checkpointer.svc.DeleteItem(&dynamodb.DeleteItemInput{
TableName: aws.String(checkpointer.TableName),
Key: map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String(shardID),
},
},
})
return err
}
================================================
FILE: clientlibrary/checkpoint/dynamodb-checkpointer_test.go
================================================
/*
* Copyright (c) 2019 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package checkpoint
import (
"errors"
"sync"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbiface"
"github.com/stretchr/testify/assert"
cfg "github.com/vmware/vmware-go-kcl/clientlibrary/config"
par "github.com/vmware/vmware-go-kcl/clientlibrary/partition"
)
func TestDoesTableExist(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
checkpoint := &DynamoCheckpoint{
TableName: "TableName",
svc: svc,
}
if !checkpoint.doesTableExist() {
t.Error("Table exists but returned false")
}
svc = &mockDynamoDB{tableExist: false}
checkpoint.svc = svc
if checkpoint.doesTableExist() {
t.Error("Table does not exist but returned true")
}
}
func TestGetLeaseNotAquired(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
err := checkpoint.GetLease(&par.ShardStatus{
ID: "0001",
Checkpoint: "",
Mux: &sync.RWMutex{},
}, "abcd-efgh")
if err != nil {
t.Errorf("Error getting lease %s", err)
}
err = checkpoint.GetLease(&par.ShardStatus{
ID: "0001",
Checkpoint: "",
Mux: &sync.RWMutex{},
}, "ijkl-mnop")
if err == nil || !errors.As(err, &ErrLeaseNotAcquired{}) {
t.Errorf("Got a lease when it was already held by abcd-efgh: %s", err)
}
}
func TestGetLeaseAquired(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
marshalledCheckpoint := map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String("0001"),
},
LeaseOwnerKey: {
S: aws.String("abcd-efgh"),
},
LeaseTimeoutKey: {
S: aws.String(time.Now().AddDate(0, -1, 0).UTC().Format(time.RFC3339)),
},
SequenceNumberKey: {
S: aws.String("deadbeef"),
},
}
input := &dynamodb.PutItemInput{
TableName: aws.String("TableName"),
Item: marshalledCheckpoint,
}
checkpoint.svc.PutItem(input)
shard := &par.ShardStatus{
ID: "0001",
Checkpoint: "deadbeef",
Mux: &sync.RWMutex{},
}
err := checkpoint.GetLease(shard, "ijkl-mnop")
if err != nil {
t.Errorf("Lease not aquired after timeout %s", err)
}
id, ok := svc.item[SequenceNumberKey]
if !ok {
t.Error("Expected checkpoint to be set by GetLease")
} else if *id.S != "deadbeef" {
t.Errorf("Expected checkpoint to be deadbeef. Got '%s'", *id.S)
}
// release owner info
err = checkpoint.RemoveLeaseOwner(shard.ID)
assert.Nil(t, err)
status := &par.ShardStatus{
ID: shard.ID,
Mux: &sync.RWMutex{},
}
checkpoint.FetchCheckpoint(status)
// checkpointer and parent shard id should be the same
assert.Equal(t, shard.Checkpoint, status.Checkpoint)
assert.Equal(t, shard.ParentShardId, status.ParentShardId)
// Only the lease owner has been wiped out
assert.Equal(t, "", status.GetLeaseOwner())
}
func TestGetLeaseShardClaimed(t *testing.T) {
leaseTimeout := time.Now().Add(-100 * time.Second).UTC()
svc := &mockDynamoDB{
tableExist: true,
item: map[string]*dynamodb.AttributeValue{
ClaimRequestKey: {S: aws.String("ijkl-mnop")},
LeaseTimeoutKey: {S: aws.String(leaseTimeout.Format(time.RFC3339))},
},
}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000).
WithLeaseStealing(true)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
err := checkpoint.GetLease(&par.ShardStatus{
ID: "0001",
Checkpoint: "",
LeaseTimeout: leaseTimeout,
Mux: &sync.RWMutex{},
}, "abcd-efgh")
if err == nil || err.Error() != ErrShardClaimed {
t.Errorf("Got a lease when it was already claimed by by ijkl-mnop: %s", err)
}
err = checkpoint.GetLease(&par.ShardStatus{
ID: "0001",
Checkpoint: "",
LeaseTimeout: leaseTimeout,
Mux: &sync.RWMutex{},
}, "ijkl-mnop")
if err != nil {
t.Errorf("Error getting lease %s", err)
}
}
func TestGetLeaseClaimRequestExpiredOwner(t *testing.T) {
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000).
WithLeaseStealing(true)
// Not expired
leaseTimeout := time.Now().
Add(-time.Duration(kclConfig.LeaseStealingClaimTimeoutMillis) * time.Millisecond).
Add(1 * time.Second).
UTC()
svc := &mockDynamoDB{
tableExist: true,
item: map[string]*dynamodb.AttributeValue{
LeaseOwnerKey: {S: aws.String("abcd-efgh")},
ClaimRequestKey: {S: aws.String("ijkl-mnop")},
LeaseTimeoutKey: {S: aws.String(leaseTimeout.Format(time.RFC3339))},
},
}
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
err := checkpoint.GetLease(&par.ShardStatus{
ID: "0001",
Checkpoint: "",
LeaseTimeout: leaseTimeout,
Mux: &sync.RWMutex{},
}, "abcd-efgh")
if err == nil || err.Error() != ErrShardClaimed {
t.Errorf("Got a lease when it was already claimed by ijkl-mnop: %s", err)
}
}
func TestGetLeaseClaimRequestExpiredClaimer(t *testing.T) {
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000).
WithLeaseStealing(true)
// Not expired
leaseTimeout := time.Now().
Add(-time.Duration(kclConfig.LeaseStealingClaimTimeoutMillis) * time.Millisecond).
Add(121 * time.Second).
UTC()
svc := &mockDynamoDB{
tableExist: true,
item: map[string]*dynamodb.AttributeValue{
LeaseOwnerKey: {S: aws.String("abcd-efgh")},
ClaimRequestKey: {S: aws.String("ijkl-mnop")},
LeaseTimeoutKey: {S: aws.String(leaseTimeout.Format(time.RFC3339))},
},
}
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
err := checkpoint.GetLease(&par.ShardStatus{
ID: "0001",
Checkpoint: "",
LeaseTimeout: leaseTimeout,
Mux: &sync.RWMutex{},
}, "ijkl-mnop")
if err == nil || !errors.As(err, &ErrLeaseNotAcquired{}) {
t.Errorf("Got a lease when it was already claimed by ijkl-mnop: %s", err)
}
}
func TestFetchCheckpointWithStealing(t *testing.T) {
future := time.Now().AddDate(0, 1, 0)
svc := &mockDynamoDB{
tableExist: true,
item: map[string]*dynamodb.AttributeValue{
SequenceNumberKey: {S: aws.String("deadbeef")},
LeaseOwnerKey: {S: aws.String("abcd-efgh")},
LeaseTimeoutKey: {
S: aws.String(future.Format(time.RFC3339)),
},
},
}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000).
WithLeaseStealing(true)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
status := &par.ShardStatus{
ID: "0001",
Checkpoint: "",
LeaseTimeout: time.Now(),
Mux: &sync.RWMutex{},
}
checkpoint.FetchCheckpoint(status)
leaseTimeout, _ := time.Parse(time.RFC3339, *svc.item[LeaseTimeoutKey].S)
assert.Equal(t, leaseTimeout, status.LeaseTimeout)
}
func TestGetLeaseConditional(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000).
WithLeaseStealing(true)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
marshalledCheckpoint := map[string]*dynamodb.AttributeValue{
LeaseKeyKey: {
S: aws.String("0001"),
},
LeaseOwnerKey: {
S: aws.String("abcd-efgh"),
},
LeaseTimeoutKey: {
S: aws.String(time.Now().Add(-1 * time.Second).UTC().Format(time.RFC3339)),
},
SequenceNumberKey: {
S: aws.String("deadbeef"),
},
ClaimRequestKey: {
S: aws.String("ijkl-mnop"),
},
}
input := &dynamodb.PutItemInput{
TableName: aws.String("TableName"),
Item: marshalledCheckpoint,
}
checkpoint.svc.PutItem(input)
shard := &par.ShardStatus{
ID: "0001",
Checkpoint: "deadbeef",
ClaimRequest: "ijkl-mnop",
Mux: &sync.RWMutex{},
}
err := checkpoint.FetchCheckpoint(shard)
if err != nil {
t.Errorf("Could not fetch checkpoint %s", err)
}
err = checkpoint.GetLease(shard, "ijkl-mnop")
if err != nil {
t.Errorf("Lease not aquired after timeout %s", err)
}
assert.Equal(t, *svc.expressionAttributeValues[":claim_request"].S, "ijkl-mnop")
assert.Contains(t, svc.conditionalExpression, " AND ClaimRequest = :claim_request")
}
type mockDynamoDB struct {
dynamodbiface.DynamoDBAPI
tableExist bool
item map[string]*dynamodb.AttributeValue
conditionalExpression string
expressionAttributeValues map[string]*dynamodb.AttributeValue
}
func (m *mockDynamoDB) ScanPages(*dynamodb.ScanInput, func(*dynamodb.ScanOutput, bool) bool) error {
return nil
}
func (m *mockDynamoDB) DescribeTable(*dynamodb.DescribeTableInput) (*dynamodb.DescribeTableOutput, error) {
if !m.tableExist {
return &dynamodb.DescribeTableOutput{}, awserr.New(dynamodb.ErrCodeResourceNotFoundException, "doesNotExist", errors.New(""))
}
return &dynamodb.DescribeTableOutput{}, nil
}
func (m *mockDynamoDB) PutItem(input *dynamodb.PutItemInput) (*dynamodb.PutItemOutput, error) {
item := input.Item
if shardID, ok := item[LeaseKeyKey]; ok {
m.item[LeaseKeyKey] = shardID
}
if owner, ok := item[LeaseOwnerKey]; ok {
m.item[LeaseOwnerKey] = owner
}
if timeout, ok := item[LeaseTimeoutKey]; ok {
m.item[LeaseTimeoutKey] = timeout
}
if checkpoint, ok := item[SequenceNumberKey]; ok {
m.item[SequenceNumberKey] = checkpoint
}
if parent, ok := item[ParentShardIdKey]; ok {
m.item[ParentShardIdKey] = parent
}
if claimRequest, ok := item[ClaimRequestKey]; ok {
m.item[ClaimRequestKey] = claimRequest
}
if input.ConditionExpression != nil {
m.conditionalExpression = *input.ConditionExpression
}
m.expressionAttributeValues = input.ExpressionAttributeValues
return nil, nil
}
func (m *mockDynamoDB) GetItem(input *dynamodb.GetItemInput) (*dynamodb.GetItemOutput, error) {
return &dynamodb.GetItemOutput{
Item: m.item,
}, nil
}
func (m *mockDynamoDB) UpdateItem(input *dynamodb.UpdateItemInput) (*dynamodb.UpdateItemOutput, error) {
exp := input.UpdateExpression
if aws.StringValue(exp) == "remove "+LeaseOwnerKey {
delete(m.item, LeaseOwnerKey)
}
return nil, nil
}
func (m *mockDynamoDB) CreateTable(input *dynamodb.CreateTableInput) (*dynamodb.CreateTableOutput, error) {
return &dynamodb.CreateTableOutput{}, nil
}
func TestListActiveWorkers(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithLeaseStealing(true)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
err := checkpoint.Init()
if err != nil {
t.Errorf("Checkpoint initialization failed: %+v", err)
}
shardStatus := map[string]*par.ShardStatus{
"0000": {ID: "0000", AssignedTo: "worker_1", Checkpoint: "", Mux: &sync.RWMutex{}},
"0001": {ID: "0001", AssignedTo: "worker_2", Checkpoint: "", Mux: &sync.RWMutex{}},
"0002": {ID: "0002", AssignedTo: "worker_4", Checkpoint: "", Mux: &sync.RWMutex{}},
"0003": {ID: "0003", AssignedTo: "worker_0", Checkpoint: "", Mux: &sync.RWMutex{}},
"0004": {ID: "0004", AssignedTo: "worker_1", Checkpoint: "", Mux: &sync.RWMutex{}},
"0005": {ID: "0005", AssignedTo: "worker_3", Checkpoint: "", Mux: &sync.RWMutex{}},
"0006": {ID: "0006", AssignedTo: "worker_3", Checkpoint: "", Mux: &sync.RWMutex{}},
"0007": {ID: "0007", AssignedTo: "worker_0", Checkpoint: "", Mux: &sync.RWMutex{}},
"0008": {ID: "0008", AssignedTo: "worker_4", Checkpoint: "", Mux: &sync.RWMutex{}},
"0009": {ID: "0009", AssignedTo: "worker_2", Checkpoint: "", Mux: &sync.RWMutex{}},
"0010": {ID: "0010", AssignedTo: "worker_0", Checkpoint: ShardEnd, Mux: &sync.RWMutex{}},
}
workers, err := checkpoint.ListActiveWorkers(shardStatus)
if err != nil {
t.Error(err)
}
for workerID, shards := range workers {
assert.Equal(t, 2, len(shards))
for _, shard := range shards {
assert.Equal(t, workerID, shard.AssignedTo)
}
}
}
func TestListActiveWorkersErrShardNotAssigned(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithLeaseStealing(true)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
err := checkpoint.Init()
if err != nil {
t.Errorf("Checkpoint initialization failed: %+v", err)
}
shardStatus := map[string]*par.ShardStatus{
"0000": {ID: "0000", Mux: &sync.RWMutex{}},
}
_, err = checkpoint.ListActiveWorkers(shardStatus)
if err != ErrShardNotAssigned {
t.Error("Expected ErrShardNotAssigned when shard is missing AssignedTo value")
}
}
func TestClaimShard(t *testing.T) {
svc := &mockDynamoDB{tableExist: true, item: map[string]*dynamodb.AttributeValue{}}
kclConfig := cfg.NewKinesisClientLibConfig("appName", "test", "us-west-2", "abc").
WithInitialPositionInStream(cfg.LATEST).
WithMaxRecords(10).
WithMaxLeasesForWorker(1).
WithShardSyncIntervalMillis(5000).
WithFailoverTimeMillis(300000).
WithLeaseStealing(true)
checkpoint := NewDynamoCheckpoint(kclConfig).WithDynamoDB(svc)
checkpoint.Init()
marshalledCheckpoint := map[string]*dynamodb.AttributeValue{
"ShardID": {
S: aws.String("0001"),
},
"AssignedTo": {
S: aws.String("abcd-efgh"),
},
"LeaseTimeout": {
S: aws.String(time.Now().AddDate(0, -1, 0).UTC().Format(time.RFC3339)),
},
"Checkpoint": {
S: aws.String("deadbeef"),
},
}
input := &dynamodb.PutItemInput{
TableName: aws.String("TableName"),
Item: marshalledCheckpoint,
}
checkpoint.svc.PutItem(input)
shard := &par.ShardStatus{
ID: "0001",
Checkpoint: "deadbeef",
Mux: &sync.RWMutex{},
}
err := checkpoint.ClaimShard(shard, "ijkl-mnop")
if err != nil {
t.Errorf("Shard not claimed %s", err)
}
claimRequest, ok := svc.item[ClaimRequestKey]
if !ok {
t.Error("Expected claimRequest to be set by ClaimShard")
} else if *claimRequest.S != "ijkl-mnop" {
t.Errorf("Expected checkpoint to be ijkl-mnop. Got '%s'", *claimRequest.S)
}
status := &par.ShardStatus{
ID: shard.ID,
Mux: &sync.RWMutex{},
}
checkpoint.FetchCheckpoint(status)
// asiggnedTo, checkpointer, and parent shard id should be the same
assert.Equal(t, shard.AssignedTo, status.AssignedTo)
assert.Equal(t, shard.Checkpoint, status.Checkpoint)
assert.Equal(t, shard.ParentShardId, status.ParentShardId)
}
================================================
FILE: clientlibrary/config/config.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package config
import (
"log"
"math"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
creds "github.com/aws/aws-sdk-go/aws/credentials"
"github.com/vmware/vmware-go-kcl/clientlibrary/metrics"
"github.com/vmware/vmware-go-kcl/logger"
)
const (
// LATEST start after the most recent data record (fetch new data).
LATEST InitialPositionInStream = iota + 1
// TRIM_HORIZON start from the oldest available data record
TRIM_HORIZON
// AT_TIMESTAMP start from the record at or after the specified server-side Timestamp.
AT_TIMESTAMP
// The location in the shard from which the KinesisClientLibrary will start fetching records from
// when the application starts for the first time and there is no checkpoint for the shard.
DefaultInitialPositionInStream = LATEST
// Fail over time in milliseconds. A worker which does not renew it's lease within this time interval
// will be regarded as having problems and it's shards will be assigned to other workers.
// For applications that have a large number of shards, this may be set to a higher number to reduce
// the number of DynamoDB IOPS required for tracking leases.
DefaultFailoverTimeMillis = 10000
// Period before the end of lease during which a lease is refreshed by the owner.
DefaultLeaseRefreshPeriodMillis = 5000
// Max records to fetch from Kinesis in a single GetRecords call.
DefaultMaxRecords = 10000
// The default value for how long the {@link ShardConsumer} should sleep if no records are returned
// from the call to
DefaultIdletimeBetweenReadsMillis = 1000
// Don't call processRecords() on the record processor for empty record lists.
DefaultDontCallProcessRecordsForEmptyRecordList = false
// Interval in milliseconds between polling to check for parent shard completion.
// Polling frequently will take up more DynamoDB IOPS (when there are leases for shards waiting on
// completion of parent shards).
DefaultParentShardPollIntervalMillis = 10000
// Shard sync interval in milliseconds - e.g. wait for this long between shard sync tasks.
DefaultShardSyncIntervalMillis = 60000
// Cleanup leases upon shards completion (don't wait until they expire in Kinesis).
// Keeping leases takes some tracking/resources (e.g. they need to be renewed, assigned), so by
// default we try to delete the ones we don't need any longer.
DefaultCleanupLeasesUponShardsCompletion = true
// Backoff time in milliseconds for Amazon Kinesis Client Library tasks (in the event of failures).
DefaultTaskBackoffTimeMillis = 500
// KCL will validate client provided sequence numbers with a call to Amazon Kinesis before
// checkpointing for calls to {@link RecordProcessorCheckpointer#checkpoint(String)} by default.
DefaultValidateSequenceNumberBeforeCheckpointing = true
// The max number of leases (shards) this worker should process.
// This can be useful to avoid overloading (and thrashing) a worker when a host has resource constraints
// or during deployment.
// NOTE: Setting this to a low value can cause data loss if workers are not able to pick up all shards in the
// stream due to the max limit.
DefaultMaxLeasesForWorker = math.MaxInt16
// Max leases to steal from another worker at one time (for load balancing).
// Setting this to a higher number can allow for faster load convergence (e.g. during deployments, cold starts),
// but can cause higher churn in the system.
DefaultMaxLeasesToStealAtOneTime = 1
// The Amazon DynamoDB table used for tracking leases will be provisioned with this read capacity.
DefaultInitialLeaseTableReadCapacity = 10
// The Amazon DynamoDB table used for tracking leases will be provisioned with this write capacity.
DefaultInitialLeaseTableWriteCapacity = 10
// The Worker will skip shard sync during initialization if there are one or more leases in the lease table. This
// assumes that the shards and leases are in-sync. This enables customers to choose faster startup times (e.g.
// during incremental deployments of an application).
DefaultSkipShardSyncAtStartupIfLeasesExist = false
// The amount of milliseconds to wait before graceful shutdown forcefully terminates.
DefaultShutdownGraceMillis = 5000
// Lease stealing defaults to false for backwards compatibility.
DefaultEnableLeaseStealing = false
// Interval between rebalance tasks defaults to 5 seconds.
DefaultLeaseStealingIntervalMillis = 5000
// Number of milliseconds to wait before another worker can aquire a claimed shard
DefaultLeaseStealingClaimTimeoutMillis = 120000
// Number of milliseconds to wait before syncing with lease table (dynamodDB)
DefaultLeaseSyncingIntervalMillis = 60000
)
type (
// InitialPositionInStream Used to specify the Position in the stream where a new application should start from
// This is used during initial application bootstrap (when a checkpoint doesn't exist for a shard or its parents)
InitialPositionInStream int
// Class that houses the entities needed to specify the Position in the stream from where a new application should
// start.
InitialPositionInStreamExtended struct {
Position InitialPositionInStream
// The time stamp of the data record from which to start reading. Used with
// shard iterator type AT_TIMESTAMP. A time stamp is the Unix epoch date with
// precision in milliseconds. For example, 2016-04-04T19:58:46.480-00:00 or
// 1459799926.480. If a record with this exact time stamp does not exist, the
// iterator returned is for the next (later) record. If the time stamp is older
// than the current trim horizon, the iterator returned is for the oldest untrimmed
// data record (TRIM_HORIZON).
Timestamp *time.Time `type:"Timestamp" timestampFormat:"unix"`
}
// Configuration for the Kinesis Client Library.
// Note: There is no need to configure credential provider. Credential can be get from InstanceProfile.
KinesisClientLibConfiguration struct {
// ApplicationName is name of application. Kinesis allows multiple applications to consume the same stream.
ApplicationName string
// DynamoDBEndpoint is an optional endpoint URL that overrides the default generated endpoint for a DynamoDB client.
// If this is empty, the default generated endpoint will be used.
DynamoDBEndpoint string
// KinesisEndpoint is an optional endpoint URL that overrides the default generated endpoint for a Kinesis client.
// If this is empty, the default generated endpoint will be used.
KinesisEndpoint string
// KinesisCredentials is used to access Kinesis
KinesisCredentials *creds.Credentials
// DynamoDBCredentials is used to access DynamoDB
DynamoDBCredentials *creds.Credentials
// TableName is name of the dynamo db table for managing kinesis stream default to ApplicationName
TableName string
// StreamName is the name of Kinesis stream
StreamName string
// EnableEnhancedFanOutConsumer enables enhanced fan-out consumer
// See: https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html
// Either consumer name or consumer ARN must be specified when Enhanced Fan-Out is enabled.
EnableEnhancedFanOutConsumer bool
// EnhancedFanOutConsumerName is the name of the enhanced fan-out consumer to create. If this isn't set the ApplicationName will be used.
EnhancedFanOutConsumerName string
// EnhancedFanOutConsumerARN is the ARN of an already created enhanced fan-out consumer, if this is set no automatic consumer creation will be attempted
EnhancedFanOutConsumerARN string
// WorkerID used to distinguish different workers/processes of a Kinesis application
WorkerID string
// InitialPositionInStream specifies the Position in the stream where a new application should start from
InitialPositionInStream InitialPositionInStream
// InitialPositionInStreamExtended provides actual AT_TIMESTAMP value
InitialPositionInStreamExtended InitialPositionInStreamExtended
// credentials to access Kinesis/Dynamo: https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/
// Note: No need to configure here. Use NewEnvCredentials for testing and EC2RoleProvider for production
// FailoverTimeMillis Lease duration (leases not renewed within this period will be claimed by others)
FailoverTimeMillis int
// LeaseRefreshPeriodMillis is the period before the end of lease during which a lease is refreshed by the owner.
LeaseRefreshPeriodMillis int
// MaxRecords Max records to read per Kinesis getRecords() call
MaxRecords int
// IdleTimeBetweenReadsInMillis Idle time between calls to fetch data from Kinesis
IdleTimeBetweenReadsInMillis int
// CallProcessRecordsEvenForEmptyRecordList Call the IRecordProcessor::processRecords() API even if
// GetRecords returned an empty record list.
CallProcessRecordsEvenForEmptyRecordList bool
// ParentShardPollIntervalMillis Wait for this long between polls to check if parent shards are done
ParentShardPollIntervalMillis int
// ShardSyncIntervalMillis Time between tasks to sync leases and Kinesis shards
ShardSyncIntervalMillis int
// CleanupTerminatedShardsBeforeExpiry Clean up shards we've finished processing (don't wait for expiration)
CleanupTerminatedShardsBeforeExpiry bool
// kinesisClientConfig Client Configuration used by Kinesis client
// dynamoDBClientConfig Client Configuration used by DynamoDB client
// Note: we will use default client provided by AWS SDK
// TaskBackoffTimeMillis Backoff period when tasks encounter an exception
TaskBackoffTimeMillis int
// ValidateSequenceNumberBeforeCheckpointing whether KCL should validate client provided sequence numbers
ValidateSequenceNumberBeforeCheckpointing bool
// RegionName The region name for the service
RegionName string
// ShutdownGraceMillis The number of milliseconds before graceful shutdown terminates forcefully
ShutdownGraceMillis int
// Operation parameters
// Max leases this Worker can handle at a time
MaxLeasesForWorker int
// Max leases to steal at one time (for load balancing)
MaxLeasesToStealAtOneTime int
// Read capacity to provision when creating the lease table (dynamoDB).
InitialLeaseTableReadCapacity int
// Write capacity to provision when creating the lease table.
InitialLeaseTableWriteCapacity int
// Worker should skip syncing shards and leases at startup if leases are present
// This is useful for optimizing deployments to large fleets working on a stable stream.
SkipShardSyncAtWorkerInitializationIfLeasesExist bool
// Logger used to log message.
Logger logger.Logger
// MonitoringService publishes per worker-scoped metrics.
MonitoringService metrics.MonitoringService
// EnableLeaseStealing turns on lease stealing
EnableLeaseStealing bool
// LeaseStealingIntervalMillis The number of milliseconds between rebalance tasks
LeaseStealingIntervalMillis int
// LeaseStealingClaimTimeoutMillis The number of milliseconds to wait before another worker can aquire a claimed shard
LeaseStealingClaimTimeoutMillis int
// LeaseSyncingTimeInterval The number of milliseconds to wait before syncing with lease table (dynamoDB)
LeaseSyncingTimeIntervalMillis int
}
)
var positionMap = map[InitialPositionInStream]*string{
LATEST: aws.String("LATEST"),
TRIM_HORIZON: aws.String("TRIM_HORIZON"),
AT_TIMESTAMP: aws.String("AT_TIMESTAMP"),
}
func InitalPositionInStreamToShardIteratorType(pos InitialPositionInStream) *string {
return positionMap[pos]
}
func empty(s string) bool {
return len(strings.TrimSpace(s)) == 0
}
// checkIsValueNotEmpty makes sure the value is not empty.
func checkIsValueNotEmpty(key string, value string) {
if empty(value) {
// There is no point to continue for incorrect configuration. Fail fast!
log.Panicf("Non-empty value expected for %v, actual: %v", key, value)
}
}
// checkIsValuePositive makes sure the value is possitive.
func checkIsValuePositive(key string, value int) {
if value <= 0 {
// There is no point to continue for incorrect configuration. Fail fast!
log.Panicf("Positive value expected for %v, actual: %v", key, value)
}
}
================================================
FILE: clientlibrary/config/config_test.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package config
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/vmware/vmware-go-kcl/logger"
)
func TestConfig(t *testing.T) {
kclConfig := NewKinesisClientLibConfig("appName", "StreamName", "us-west-2", "workerId").
WithFailoverTimeMillis(500).
WithMaxRecords(100).
WithInitialPositionInStream(TRIM_HORIZON).
WithIdleTimeBetweenReadsInMillis(20).
WithCallProcessRecordsEvenForEmptyRecordList(true).
WithTaskBackoffTimeMillis(10).
WithEnhancedFanOutConsumerName("fan-out-consumer")
assert.Equal(t, "appName", kclConfig.ApplicationName)
assert.Equal(t, 500, kclConfig.FailoverTimeMillis)
assert.Equal(t, 10, kclConfig.TaskBackoffTimeMillis)
assert.True(t, kclConfig.EnableEnhancedFanOutConsumer)
assert.Equal(t, "fan-out-consumer", kclConfig.EnhancedFanOutConsumerName)
assert.Equal(t, false, kclConfig.EnableLeaseStealing)
assert.Equal(t, 5000, kclConfig.LeaseStealingIntervalMillis)
contextLogger := kclConfig.Logger.WithFields(logger.Fields{"key1": "value1"})
contextLogger.Debugf("Starting with default logger")
contextLogger.Infof("Default logger is awesome")
}
func TestConfigLeaseStealing(t *testing.T) {
kclConfig := NewKinesisClientLibConfig("appName", "StreamName", "us-west-2", "workerId").
WithFailoverTimeMillis(500).
WithMaxRecords(100).
WithInitialPositionInStream(TRIM_HORIZON).
WithIdleTimeBetweenReadsInMillis(20).
WithCallProcessRecordsEvenForEmptyRecordList(true).
WithTaskBackoffTimeMillis(10).
WithLeaseStealing(true).
WithLeaseStealingIntervalMillis(10000)
assert.Equal(t, "appName", kclConfig.ApplicationName)
assert.Equal(t, 500, kclConfig.FailoverTimeMillis)
assert.Equal(t, 10, kclConfig.TaskBackoffTimeMillis)
assert.Equal(t, true, kclConfig.EnableLeaseStealing)
assert.Equal(t, 10000, kclConfig.LeaseStealingIntervalMillis)
contextLogger := kclConfig.Logger.WithFields(logger.Fields{"key1": "value1"})
contextLogger.Debugf("Starting with default logger")
contextLogger.Infof("Default logger is awesome")
}
func TestConfigDefaultEnhancedFanOutConsumerName(t *testing.T) {
kclConfig := NewKinesisClientLibConfig("appName", "StreamName", "us-west-2", "workerId")
assert.Equal(t, "appName", kclConfig.ApplicationName)
assert.False(t, kclConfig.EnableEnhancedFanOutConsumer)
assert.Equal(t, "appName", kclConfig.EnhancedFanOutConsumerName)
}
func TestEmptyEnhancedFanOutConsumerName(t *testing.T) {
assert.PanicsWithValue(t, "Non-empty value expected for EnhancedFanOutConsumerName, actual: ", func() {
NewKinesisClientLibConfig("app", "stream", "us-west-2", "worker").WithEnhancedFanOutConsumerName("")
})
}
func TestConfigWithEnhancedFanOutConsumerARN(t *testing.T) {
kclConfig := NewKinesisClientLibConfig("app", "stream", "us-west-2", "worker").
WithEnhancedFanOutConsumerARN("consumer:arn")
assert.True(t, kclConfig.EnableEnhancedFanOutConsumer)
assert.Equal(t, "consumer:arn", kclConfig.EnhancedFanOutConsumerARN)
}
func TestEmptyEnhancedFanOutConsumerARN(t *testing.T) {
assert.PanicsWithValue(t, "Non-empty value expected for EnhancedFanOutConsumerARN, actual: ", func() {
NewKinesisClientLibConfig("app", "stream", "us-west-2", "worker").WithEnhancedFanOutConsumerARN("")
})
}
================================================
FILE: clientlibrary/config/initial-stream-pos.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package config
import (
"time"
)
func newInitialPositionAtTimestamp(timestamp *time.Time) *InitialPositionInStreamExtended {
return &InitialPositionInStreamExtended{Position: AT_TIMESTAMP, Timestamp: timestamp}
}
func newInitialPosition(position InitialPositionInStream) *InitialPositionInStreamExtended {
return &InitialPositionInStreamExtended{Position: position, Timestamp: nil}
}
================================================
FILE: clientlibrary/config/kcl-config.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package config
import (
"log"
"time"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/vmware/vmware-go-kcl/clientlibrary/metrics"
"github.com/vmware/vmware-go-kcl/clientlibrary/utils"
"github.com/vmware/vmware-go-kcl/logger"
)
// NewKinesisClientLibConfig creates a default KinesisClientLibConfiguration based on the required fields.
func NewKinesisClientLibConfig(applicationName, streamName, regionName, workerID string) *KinesisClientLibConfiguration {
return NewKinesisClientLibConfigWithCredentials(applicationName, streamName, regionName, workerID,
nil, nil)
}
// NewKinesisClientLibConfigWithCredential creates a default KinesisClientLibConfiguration based on the required fields and unique credentials.
func NewKinesisClientLibConfigWithCredential(applicationName, streamName, regionName, workerID string,
creds *credentials.Credentials) *KinesisClientLibConfiguration {
return NewKinesisClientLibConfigWithCredentials(applicationName, streamName, regionName, workerID, creds, creds)
}
// NewKinesisClientLibConfigWithCredentials creates a default KinesisClientLibConfiguration based on the required fields and specific credentials for each service.
func NewKinesisClientLibConfigWithCredentials(applicationName, streamName, regionName, workerID string,
kiniesisCreds, dynamodbCreds *credentials.Credentials) *KinesisClientLibConfiguration {
checkIsValueNotEmpty("ApplicationName", applicationName)
checkIsValueNotEmpty("StreamName", streamName)
checkIsValueNotEmpty("RegionName", regionName)
if empty(workerID) {
workerID = utils.MustNewUUID()
}
// populate the KCL configuration with default values
return &KinesisClientLibConfiguration{
ApplicationName: applicationName,
KinesisCredentials: kiniesisCreds,
DynamoDBCredentials: dynamodbCreds,
TableName: applicationName,
EnhancedFanOutConsumerName: applicationName,
StreamName: streamName,
RegionName: regionName,
WorkerID: workerID,
InitialPositionInStream: DefaultInitialPositionInStream,
InitialPositionInStreamExtended: *newInitialPosition(DefaultInitialPositionInStream),
FailoverTimeMillis: DefaultFailoverTimeMillis,
LeaseRefreshPeriodMillis: DefaultLeaseRefreshPeriodMillis,
MaxRecords: DefaultMaxRecords,
IdleTimeBetweenReadsInMillis: DefaultIdletimeBetweenReadsMillis,
CallProcessRecordsEvenForEmptyRecordList: DefaultDontCallProcessRecordsForEmptyRecordList,
ParentShardPollIntervalMillis: DefaultParentShardPollIntervalMillis,
ShardSyncIntervalMillis: DefaultShardSyncIntervalMillis,
CleanupTerminatedShardsBeforeExpiry: DefaultCleanupLeasesUponShardsCompletion,
TaskBackoffTimeMillis: DefaultTaskBackoffTimeMillis,
ValidateSequenceNumberBeforeCheckpointing: DefaultValidateSequenceNumberBeforeCheckpointing,
ShutdownGraceMillis: DefaultShutdownGraceMillis,
MaxLeasesForWorker: DefaultMaxLeasesForWorker,
MaxLeasesToStealAtOneTime: DefaultMaxLeasesToStealAtOneTime,
InitialLeaseTableReadCapacity: DefaultInitialLeaseTableReadCapacity,
InitialLeaseTableWriteCapacity: DefaultInitialLeaseTableWriteCapacity,
SkipShardSyncAtWorkerInitializationIfLeasesExist: DefaultSkipShardSyncAtStartupIfLeasesExist,
EnableLeaseStealing: DefaultEnableLeaseStealing,
LeaseStealingIntervalMillis: DefaultLeaseStealingIntervalMillis,
LeaseStealingClaimTimeoutMillis: DefaultLeaseStealingClaimTimeoutMillis,
LeaseSyncingTimeIntervalMillis: DefaultLeaseSyncingIntervalMillis,
Logger: logger.GetDefaultLogger(),
}
}
// WithKinesisEndpoint is used to provide an alternative Kinesis endpoint
func (c *KinesisClientLibConfiguration) WithKinesisEndpoint(kinesisEndpoint string) *KinesisClientLibConfiguration {
c.KinesisEndpoint = kinesisEndpoint
return c
}
// WithDynamoDBEndpoint is used to provide an alternative DynamoDB endpoint
func (c *KinesisClientLibConfiguration) WithDynamoDBEndpoint(dynamoDBEndpoint string) *KinesisClientLibConfiguration {
c.DynamoDBEndpoint = dynamoDBEndpoint
return c
}
// WithTableName to provide alternative lease table in DynamoDB
func (c *KinesisClientLibConfiguration) WithTableName(tableName string) *KinesisClientLibConfiguration {
c.TableName = tableName
return c
}
func (c *KinesisClientLibConfiguration) WithInitialPositionInStream(initialPositionInStream InitialPositionInStream) *KinesisClientLibConfiguration {
c.InitialPositionInStream = initialPositionInStream
c.InitialPositionInStreamExtended = *newInitialPosition(initialPositionInStream)
return c
}
func (c *KinesisClientLibConfiguration) WithTimestampAtInitialPositionInStream(timestamp *time.Time) *KinesisClientLibConfiguration {
c.InitialPositionInStream = AT_TIMESTAMP
c.InitialPositionInStreamExtended = *newInitialPositionAtTimestamp(timestamp)
return c
}
func (c *KinesisClientLibConfiguration) WithFailoverTimeMillis(failoverTimeMillis int) *KinesisClientLibConfiguration {
checkIsValuePositive("FailoverTimeMillis", failoverTimeMillis)
c.FailoverTimeMillis = failoverTimeMillis
return c
}
func (c *KinesisClientLibConfiguration) WithLeaseRefreshPeriodMillis(leaseRefreshPeriodMillis int) *KinesisClientLibConfiguration {
checkIsValuePositive("LeaseRefreshPeriodMillis", leaseRefreshPeriodMillis)
c.LeaseRefreshPeriodMillis = leaseRefreshPeriodMillis
return c
}
func (c *KinesisClientLibConfiguration) WithShardSyncIntervalMillis(shardSyncIntervalMillis int) *KinesisClientLibConfiguration {
checkIsValuePositive("ShardSyncIntervalMillis", shardSyncIntervalMillis)
c.ShardSyncIntervalMillis = shardSyncIntervalMillis
return c
}
func (c *KinesisClientLibConfiguration) WithMaxRecords(maxRecords int) *KinesisClientLibConfiguration {
checkIsValuePositive("MaxRecords", maxRecords)
c.MaxRecords = maxRecords
return c
}
// WithMaxLeasesForWorker configures maximum lease this worker can handles. It determines how maximun number of shards
// this worker can handle.
func (c *KinesisClientLibConfiguration) WithMaxLeasesForWorker(n int) *KinesisClientLibConfiguration {
checkIsValuePositive("MaxLeasesForWorker", n)
c.MaxLeasesForWorker = n
return c
}
/**
* Controls how long the KCL will sleep if no records are returned from Kinesis
*
* <p>
* This value is only used when no records are returned; if records are returned, the {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask} will
* immediately retrieve the next set of records after the call to
* {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor#processRecords(ProcessRecordsInput)}
* has returned. Setting this value to high may result in the KCL being unable to catch up. If you are changing this
* value it's recommended that you enable {@link #withCallProcessRecordsEvenForEmptyRecordList(boolean)}, and
* monitor how far behind the records retrieved are by inspecting
* {@link com.amazonaws.services.kinesis.clientlibrary.types.ProcessRecordsInput#getMillisBehindLatest()}, and the
* <a href=
* "http://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html#kinesis-metrics-stream">CloudWatch
* Metric: GetRecords.MillisBehindLatest</a>
* </p>
*
* @param IdleTimeBetweenReadsInMillis
* how long to sleep between GetRecords calls when no records are returned.
* @return KinesisClientLibConfiguration
*/
func (c *KinesisClientLibConfiguration) WithIdleTimeBetweenReadsInMillis(idleTimeBetweenReadsInMillis int) *KinesisClientLibConfiguration {
checkIsValuePositive("IdleTimeBetweenReadsInMillis", idleTimeBetweenReadsInMillis)
c.IdleTimeBetweenReadsInMillis = idleTimeBetweenReadsInMillis
return c
}
func (c *KinesisClientLibConfiguration) WithCallProcessRecordsEvenForEmptyRecordList(callProcessRecordsEvenForEmptyRecordList bool) *KinesisClientLibConfiguration {
c.CallProcessRecordsEvenForEmptyRecordList = callProcessRecordsEvenForEmptyRecordList
return c
}
func (c *KinesisClientLibConfiguration) WithTaskBackoffTimeMillis(taskBackoffTimeMillis int) *KinesisClientLibConfiguration {
checkIsValuePositive("TaskBackoffTimeMillis", taskBackoffTimeMillis)
c.TaskBackoffTimeMillis = taskBackoffTimeMillis
return c
}
func (c *KinesisClientLibConfiguration) WithLogger(logger logger.Logger) *KinesisClientLibConfiguration {
if logger == nil {
log.Panic("Logger cannot be null")
}
c.Logger = logger
return c
}
// WithMonitoringService sets the monitoring service to use to publish metrics.
func (c *KinesisClientLibConfiguration) WithMonitoringService(mService metrics.MonitoringService) *KinesisClientLibConfiguration {
// Nil case is handled downward (at worker creation) so no need to do it here.
// Plus the user might want to be explicit about passing a nil monitoring service here.
c.MonitoringService = mService
return c
}
// WithEnhancedFanOutConsumer sets EnableEnhancedFanOutConsumer. If enhanced fan-out is enabled and ConsumerName is not specified ApplicationName is used as ConsumerName.
// For more info see: https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html
// Note: You can register up to twenty consumers per stream to use enhanced fan-out.
func (c *KinesisClientLibConfiguration) WithEnhancedFanOutConsumer(enable bool) *KinesisClientLibConfiguration {
c.EnableEnhancedFanOutConsumer = enable
return c
}
// WithEnhancedFanOutConsumerName enables enhanced fan-out consumer with the specified name
// For more info see: https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html
// Note: You can register up to twenty consumers per stream to use enhanced fan-out.
func (c *KinesisClientLibConfiguration) WithEnhancedFanOutConsumerName(consumerName string) *KinesisClientLibConfiguration {
checkIsValueNotEmpty("EnhancedFanOutConsumerName", consumerName)
c.EnhancedFanOutConsumerName = consumerName
c.EnableEnhancedFanOutConsumer = true
return c
}
// WithEnhancedFanOutConsumerARN enables enhanced fan-out consumer with the specified consumer ARN
// For more info see: https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html
// Note: You can register up to twenty consumers per stream to use enhanced fan-out.
func (c *KinesisClientLibConfiguration) WithEnhancedFanOutConsumerARN(consumerARN string) *KinesisClientLibConfiguration {
checkIsValueNotEmpty("EnhancedFanOutConsumerARN", consumerARN)
c.EnhancedFanOutConsumerARN = consumerARN
c.EnableEnhancedFanOutConsumer = true
return c
}
func (c *KinesisClientLibConfiguration) WithLeaseStealing(enableLeaseStealing bool) *KinesisClientLibConfiguration {
c.EnableLeaseStealing = enableLeaseStealing
return c
}
func (c *KinesisClientLibConfiguration) WithLeaseStealingIntervalMillis(leaseStealingIntervalMillis int) *KinesisClientLibConfiguration {
c.LeaseStealingIntervalMillis = leaseStealingIntervalMillis
return c
}
func (c *KinesisClientLibConfiguration) WithLeaseSyncingIntervalMillis(leaseSyncingIntervalMillis int) *KinesisClientLibConfiguration {
c.LeaseSyncingTimeIntervalMillis = leaseSyncingIntervalMillis
return c
}
================================================
FILE: clientlibrary/interfaces/inputs.go
================================================
/*
* Copyright (c) 2020 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package interfaces
import (
"time"
"github.com/aws/aws-sdk-go/aws"
ks "github.com/aws/aws-sdk-go/service/kinesis"
)
const (
/**
* Indicates that the entire application is being shutdown, and if desired the record processor will be given a
* final chance to checkpoint. This state will not trigger a direct call to
* {@link com.amazonaws.services.kinesis.clientlibrary.interfaces.v2.IRecordProcessor#shutdown(ShutdownInput)}, but
* instead depend on a different interface for backward compatibility.
*/
REQUESTED ShutdownReason = iota + 1
/**
* Terminate processing for this RecordProcessor (resharding use case).
* Indicates that the shard is closed and all records from the shard have been delivered to the application.
* Applications SHOULD checkpoint their progress to indicate that they have successfully processed all records
* from this shard and processing of child shards can be started.
*/
TERMINATE
/**
* Processing will be moved to a different record processor (fail over, load balancing use cases).
* Applications SHOULD NOT checkpoint their progress (as another record processor may have already started
* processing data).
*/
ZOMBIE
)
// Containers for the parameters to the IRecordProcessor
type (
/**
* Reason the RecordProcessor is being shutdown.
* Used to distinguish between a fail-over vs. a termination (shard is closed and all records have been delivered).
* In case of a fail over, applications should NOT checkpoint as part of shutdown,
* since another record processor may have already started processing records for that shard.
* In case of termination (resharding use case), applications SHOULD checkpoint their progress to indicate
* that they have successfully processed all the records (processing of child shards can then begin).
*/
ShutdownReason int
InitializationInput struct {
// The shardId that the record processor is being initialized for.
ShardId string
// The last extended sequence number that was successfully checkpointed by the previous record processor.
ExtendedSequenceNumber *ExtendedSequenceNumber
}
ProcessRecordsInput struct {
// The time that this batch of records was received by the KCL.
CacheEntryTime *time.Time
// The time that this batch of records was prepared to be provided to the RecordProcessor.
CacheExitTime *time.Time
// The records received from Kinesis. These records may have been de-aggregated if they were published by the KPL.
Records []*ks.Record
// A checkpointer that the RecordProcessor can use to checkpoint its progress.
Checkpointer IRecordProcessorCheckpointer
// How far behind this batch of records was when received from Kinesis.
MillisBehindLatest int64
}
ShutdownInput struct {
// ShutdownReason shows why RecordProcessor is going to be shutdown.
ShutdownReason ShutdownReason
// Checkpointer is used to record the current progress.
Checkpointer IRecordProcessorCheckpointer
}
)
var shutdownReasonMap = map[ShutdownReason]*string{
REQUESTED: aws.String("REQUESTED"),
TERMINATE: aws.String("TERMINATE"),
ZOMBIE: aws.String("ZOMBIE"),
}
func ShutdownReasonMessage(reason ShutdownReason) *string {
return shutdownReasonMap[reason]
}
================================================
FILE: clientlibrary/interfaces/record-processor-checkpointer.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package interfaces
type (
IPreparedCheckpointer interface {
GetPendingCheckpoint() *ExtendedSequenceNumber
/**
* This method will record a pending checkpoint.
*
* @error ThrottlingError Can't store checkpoint. Can be caused by checkpointing too frequently.
* Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.
* @error ShutdownError The record processor instance has been shutdown. Another instance may have
* started processing some of these records already.
* The application should abort processing via this RecordProcessor instance.
* @error InvalidStateError Can't store checkpoint.
* Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).
* @error KinesisClientLibDependencyError Encountered an issue when storing the checkpoint. The application can
* backoff and retry.
* @error IllegalArgumentError The sequence number being checkpointed is invalid because it is out of range,
* i.e. it is smaller than the last check point value (prepared or committed), or larger than the greatest
* sequence number seen by the associated record processor.
*/
Checkpoint() error
}
/**
* Used by RecordProcessors when they want to checkpoint their progress.
* The Kinesis Client Library will pass an object implementing this interface to RecordProcessors, so they can
* checkpoint their progress.
*/
IRecordProcessorCheckpointer interface {
/**
* This method will checkpoint the progress at the provided sequenceNumber. This method is analogous to
* {@link #checkpoint()} but provides the ability to specify the sequence number at which to
* checkpoint.
*
* @param sequenceNumber A sequence number at which to checkpoint in this shard. Upon failover,
* the Kinesis Client Library will start fetching records after this sequence number.
* @error ThrottlingError Can't store checkpoint. Can be caused by checkpointing too frequently.
* Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.
* @error ShutdownError The record processor instance has been shutdown. Another instance may have
* started processing some of these records already.
* The application should abort processing via this RecordProcessor instance.
* @error InvalidStateError Can't store checkpoint.
* Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).
* @error KinesisClientLibDependencyError Encountered an issue when storing the checkpoint. The application can
* backoff and retry.
* @error IllegalArgumentError The sequence number is invalid for one of the following reasons:
* 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the
* greatest sequence number seen by the associated record processor.
* 2.) It is not a valid sequence number for a record in this shard.
*/
Checkpoint(sequenceNumber *string) error
/**
* This method will record a pending checkpoint at the provided sequenceNumber.
*
* @param sequenceNumber A sequence number at which to prepare checkpoint in this shard.
* @return an IPreparedCheckpointer object that can be called later to persist the checkpoint.
*
* @error ThrottlingError Can't store pending checkpoint. Can be caused by checkpointing too frequently.
* Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.
* @error ShutdownError The record processor instance has been shutdown. Another instance may have
* started processing some of these records already.
* The application should abort processing via this RecordProcessor instance.
* @error InvalidStateError Can't store pending checkpoint.
* Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).
* @error KinesisClientLibDependencyError Encountered an issue when storing the pending checkpoint. The
* application can backoff and retry.
* @error IllegalArgumentError The sequence number is invalid for one of the following reasons:
* 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the
* greatest sequence number seen by the associated record processor.
* 2.) It is not a valid sequence number for a record in this shard.
*/
PrepareCheckpoint(sequenceNumber *string) (IPreparedCheckpointer, error)
}
)
================================================
FILE: clientlibrary/interfaces/record-processor.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package interfaces
type (
// IRecordProcessor is the interface for some callback functions invoked by KCL will
// The main task of using KCL is to provide implementation on IRecordProcessor interface.
// Note: This is exactly the same interface as Amazon KCL IRecordProcessor v2
IRecordProcessor interface {
/**
* Invoked by the Amazon Kinesis Client Library before data records are delivered to the RecordProcessor instance
* (via processRecords).
*
* @param initializationInput Provides information related to initialization
*/
Initialize(initializationInput *InitializationInput)
/**
* Process data records. The Amazon Kinesis Client Library will invoke this method to deliver data records to the
* application.
* Upon fail over, the new instance will get records with sequence number > checkpoint position
* for each partition key.
*
* @param processRecordsInput Provides the records to be processed as well as information and capabilities related
* to them (eg checkpointing).
*/
ProcessRecords(processRecordsInput *ProcessRecordsInput)
/**
* Invoked by the Amazon Kinesis Client Library to indicate it will no longer send data records to this
* RecordProcessor instance.
*
* <h2><b>Warning</b></h2>
*
* When the value of {@link ShutdownInput#getShutdownReason()} is
* {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason#TERMINATE} it is required that you
* checkpoint. Failure to do so will result in an IllegalArgumentException, and the KCL no longer making progress.
*
* @param shutdownInput
* Provides information and capabilities (eg checkpointing) related to shutdown of this record processor.
*/
Shutdown(shutdownInput *ShutdownInput)
}
// IRecordProcessorFactory is interface for creating IRecordProcessor. Each Worker can have multiple threads
// for processing shard. Client can choose either creating one processor per shard or sharing them.
IRecordProcessorFactory interface {
/**
* Returns a record processor to be used for processing data records for a (assigned) shard.
*
* @return Returns a processor object.
*/
CreateProcessor() IRecordProcessor
}
)
================================================
FILE: clientlibrary/interfaces/sequence-number.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/awslabs/amazon-kinesis-client
/*
* Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package interfaces
// ExtendedSequenceNumber represents a two-part sequence number for records aggregated by the Kinesis Producer Library.
//
// The KPL combines multiple user records into a single Kinesis record. Each user record therefore has an integer
// sub-sequence number, in addition to the regular sequence number of the Kinesis record. The sub-sequence number
// is used to checkpoint within an aggregated record.
type ExtendedSequenceNumber struct {
SequenceNumber *string
SubSequenceNumber int64
}
================================================
FILE: clientlibrary/metrics/cloudwatch/cloudwatch.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package cloudwatch
import (
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
cwatch "github.com/aws/aws-sdk-go/service/cloudwatch"
"github.com/aws/aws-sdk-go/service/cloudwatch/cloudwatchiface"
"github.com/vmware/vmware-go-kcl/logger"
)
// Buffer metrics for at most this long before publishing to CloudWatch.
const DEFAULT_CLOUDWATCH_METRICS_BUFFER_DURATION = 10 * time.Second
type MonitoringService struct {
appName string
streamName string
workerID string
region string
credentials *credentials.Credentials
logger logger.Logger
// control how often to publish to CloudWatch
bufferDuration time.Duration
stop *chan struct{}
waitGroup *sync.WaitGroup
svc cloudwatchiface.CloudWatchAPI
shardMetrics *sync.Map
}
type cloudWatchMetrics struct {
sync.Mutex
processedRecords int64
processedBytes int64
behindLatestMillis []float64
leasesHeld int64
leaseRenewals int64
getRecordsTime []float64
processRecordsTime []float64
}
// NewMonitoringService returns a Monitoring service publishing metrics to CloudWatch.
func NewMonitoringService(region string, creds *credentials.Credentials) *MonitoringService {
return NewMonitoringServiceWithOptions(region, creds, logger.GetDefaultLogger(), DEFAULT_CLOUDWATCH_METRICS_BUFFER_DURATION)
}
// NewMonitoringServiceWithOptions returns a Monitoring service publishing metrics to
// CloudWatch with the provided credentials, buffering duration and logger.
func NewMonitoringServiceWithOptions(region string, creds *credentials.Credentials, logger logger.Logger, bufferDur time.Duration) *MonitoringService {
return &MonitoringService{
region: region,
credentials: creds,
logger: logger,
bufferDuration: bufferDur,
}
}
func (cw *MonitoringService) Init(appName, streamName, workerID string) error {
cw.appName = appName
cw.streamName = streamName
cw.workerID = workerID
cfg := &aws.Config{Region: aws.String(cw.region)}
cfg.Credentials = cw.credentials
s, err := session.NewSession(cfg)
if err != nil {
cw.logger.Errorf("Error in creating session for cloudwatch. %+v", err)
return err
}
cw.svc = cwatch.New(s)
cw.shardMetrics = new(sync.Map)
stopChan := make(chan struct{})
cw.stop = &stopChan
wg := sync.WaitGroup{}
cw.waitGroup = &wg
return nil
}
func (cw *MonitoringService) Start() error {
cw.waitGroup.Add(1)
// entering eventloop for sending metrics to CloudWatch
go cw.eventloop()
return nil
}
func (cw *MonitoringService) Shutdown() {
cw.logger.Infof("Shutting down cloudwatch metrics system...")
close(*cw.stop)
cw.waitGroup.Wait()
cw.logger.Infof("Cloudwatch metrics system has been shutdown.")
}
// Start daemon to flush metrics periodically
func (cw *MonitoringService) eventloop() {
defer cw.waitGroup.Done()
for {
if err := cw.flush(); err != nil {
cw.logger.Errorf("Error sending metrics to CloudWatch. %+v", err)
}
select {
case <-*cw.stop:
cw.logger.Infof("Shutting down monitoring system")
if err := cw.flush(); err != nil {
cw.logger.Errorf("Error sending metrics to CloudWatch. %+v", err)
}
return
case <-time.After(cw.bufferDuration):
}
}
}
func (cw *MonitoringService) flushShard(shard string, metric *cloudWatchMetrics) bool {
metric.Lock()
defaultDimensions := []*cwatch.Dimension{
{
Name: aws.String("Shard"),
Value: &shard,
},
{
Name: aws.String("KinesisStreamName"),
Value: &cw.streamName,
},
}
leaseDimensions := []*cwatch.Dimension{
{
Name: aws.String("Shard"),
Value: &shard,
},
{
Name: aws.String("KinesisStreamName"),
Value: &cw.streamName,
},
{
Name: aws.String("WorkerID"),
Value: &cw.workerID,
},
}
metricTimestamp := time.Now()
data := []*cwatch.MetricDatum{
{
Dimensions: defaultDimensions,
MetricName: aws.String("RecordsProcessed"),
Unit: aws.String("Count"),
Timestamp: &metricTimestamp,
Value: aws.Float64(float64(metric.processedRecords)),
},
{
Dimensions: defaultDimensions,
MetricName: aws.String("DataBytesProcessed"),
Unit: aws.String("Bytes"),
Timestamp: &metricTimestamp,
Value: aws.Float64(float64(metric.processedBytes)),
},
{
Dimensions: leaseDimensions,
MetricName: aws.String("RenewLease.Success"),
Unit: aws.String("Count"),
Timestamp: &metricTimestamp,
Value: aws.Float64(float64(metric.leaseRenewals)),
},
{
Dimensions: leaseDimensions,
MetricName: aws.String("CurrentLeases"),
Unit: aws.String("Count"),
Timestamp: &metricTimestamp,
Value: aws.Float64(float64(metric.leasesHeld)),
},
}
if len(metric.behindLatestMillis) > 0 {
data = append(data, &cwatch.MetricDatum{
Dimensions: defaultDimensions,
MetricName: aws.String("MillisBehindLatest"),
Unit: aws.String("Milliseconds"),
Timestamp: &metricTimestamp,
StatisticValues: &cwatch.StatisticSet{
SampleCount: aws.Float64(float64(len(metric.behindLatestMillis))),
Sum: sumFloat64(metric.behindLatestMillis),
Maximum: maxFloat64(metric.behindLatestMillis),
Minimum: minFloat64(metric.behindLatestMillis),
}})
}
if len(metric.getRecordsTime) > 0 {
data = append(data, &cwatch.MetricDatum{
Dimensions: defaultDimensions,
MetricName: aws.String("KinesisDataFetcher.getRecords.Time"),
Unit: aws.String("Milliseconds"),
Timestamp: &metricTimestamp,
StatisticValues: &cwatch.StatisticSet{
SampleCount: aws.Float64(float64(len(metric.getRecordsTime))),
Sum: sumFloat64(metric.getRecordsTime),
Maximum: maxFloat64(metric.getRecordsTime),
Minimum: minFloat64(metric.getRecordsTime),
}})
}
if len(metric.processRecordsTime) > 0 {
data = append(data, &cwatch.MetricDatum{
Dimensions: defaultDimensions,
MetricName: aws.String("RecordProcessor.processRecords.Time"),
Unit: aws.String("Milliseconds"),
Timestamp: &metricTimestamp,
StatisticValues: &cwatch.StatisticSet{
SampleCount: aws.Float64(float64(len(metric.processRecordsTime))),
Sum: sumFloat64(metric.processRecordsTime),
Maximum: maxFloat64(metric.processRecordsTime),
Minimum: minFloat64(metric.processRecordsTime),
}})
}
// Publish metrics data to cloud watch
_, err := cw.svc.PutMetricData(&cwatch.PutMetricDataInput{
Namespace: aws.String(cw.appName),
MetricData: data,
})
if err == nil {
metric.processedRecords = 0
metric.processedBytes = 0
metric.behindLatestMillis = []float64{}
metric.leaseRenewals = 0
metric.getRecordsTime = []float64{}
metric.processRecordsTime = []float64{}
} else {
cw.logger.Errorf("Error in publishing cloudwatch metrics. Error: %+v", err)
}
metric.Unlock()
return true
}
func (cw *MonitoringService) flush() error {
cw.logger.Debugf("Flushing metrics data. Stream: %s, Worker: %s", cw.streamName, cw.workerID)
// publish per shard metrics
cw.shardMetrics.Range(func(k, v interface{}) bool {
shard, metric := k.(string), v.(*cloudWatchMetrics)
return cw.flushShard(shard, metric)
})
return nil
}
func (cw *MonitoringService) IncrRecordsProcessed(shard string, count int) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.processedRecords += int64(count)
}
func (cw *MonitoringService) IncrBytesProcessed(shard string, count int64) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.processedBytes += count
}
func (cw *MonitoringService) MillisBehindLatest(shard string, millSeconds float64) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.behindLatestMillis = append(m.behindLatestMillis, millSeconds)
}
func (cw *MonitoringService) LeaseGained(shard string) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.leasesHeld++
}
func (cw *MonitoringService) LeaseLost(shard string) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.leasesHeld--
}
func (cw *MonitoringService) LeaseRenewed(shard string) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.leaseRenewals++
}
func (cw *MonitoringService) RecordGetRecordsTime(shard string, time float64) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.getRecordsTime = append(m.getRecordsTime, time)
}
func (cw *MonitoringService) RecordProcessRecordsTime(shard string, time float64) {
m := cw.getOrCreatePerShardMetrics(shard)
m.Lock()
defer m.Unlock()
m.processRecordsTime = append(m.processRecordsTime, time)
}
func (cw *MonitoringService) getOrCreatePerShardMetrics(shard string) *cloudWatchMetrics {
var i interface{}
var ok bool
if i, ok = cw.shardMetrics.Load(shard); !ok {
m := &cloudWatchMetrics{}
cw.shardMetrics.Store(shard, m)
return m
}
return i.(*cloudWatchMetrics)
}
func sumFloat64(slice []float64) *float64 {
sum := float64(0)
for _, num := range slice {
sum += num
}
return &sum
}
func maxFloat64(slice []float64) *float64 {
if len(slice) < 1 {
return aws.Float64(0)
}
max := slice[0]
for _, num := range slice {
if num > max {
max = num
}
}
return &max
}
func minFloat64(slice []float64) *float64 {
if len(slice) < 1 {
return aws.Float64(0)
}
min := slice[0]
for _, num := range slice {
if num < min {
min = num
}
}
return &min
}
================================================
FILE: clientlibrary/metrics/interfaces.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package metrics
type MonitoringService interface {
Init(appName, streamName, workerID string) error
Start() error
IncrRecordsProcessed(string, int)
IncrBytesProcessed(string, int64)
MillisBehindLatest(string, float64)
LeaseGained(string)
LeaseLost(string)
LeaseRenewed(string)
RecordGetRecordsTime(string, float64)
RecordProcessRecordsTime(string, float64)
Shutdown()
}
// NoopMonitoringService implements MonitoringService by does nothing.
type NoopMonitoringService struct{}
func (NoopMonitoringService) Init(appName, streamName, workerID string) error { return nil }
func (NoopMonitoringService) Start() error { return nil }
func (NoopMonitoringService) Shutdown() {}
func (NoopMonitoringService) IncrRecordsProcessed(shard string, count int) {}
func (NoopMonitoringService) IncrBytesProcessed(shard string, count int64) {}
func (NoopMonitoringService) MillisBehindLatest(shard string, millSeconds float64) {}
func (NoopMonitoringService) LeaseGained(shard string) {}
func (NoopMonitoringService) LeaseLost(shard string) {}
func (NoopMonitoringService) LeaseRenewed(shard string) {}
func (NoopMonitoringService) RecordGetRecordsTime(shard string, time float64) {}
func (NoopMonitoringService) RecordProcessRecordsTime(shard string, time float64) {}
================================================
FILE: clientlibrary/metrics/prometheus/prometheus.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package prometheus
import (
"net/http"
prom "github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/vmware/vmware-go-kcl/logger"
)
// MonitoringService publishes kcl metrics to Prometheus.
// It might be trick if the service onboarding with KCL already uses Prometheus.
type MonitoringService struct {
listenAddress string
namespace string
streamName string
workerID string
region string
logger logger.Logger
processedRecords *prom.CounterVec
processedBytes *prom.CounterVec
behindLatestMillis *prom.GaugeVec
leasesHeld *prom.GaugeVec
leaseRenewals *prom.CounterVec
getRecordsTime *prom.HistogramVec
processRecordsTime *prom.HistogramVec
}
// NewMonitoringService returns a Monitoring service publishing metrics to Prometheus.
func NewMonitoringService(listenAddress, region string, logger logger.Logger) *MonitoringService {
return &MonitoringService{
listenAddress: listenAddress,
region: region,
logger: logger,
}
}
func (p *MonitoringService) Init(appName, streamName, workerID string) error {
p.namespace = appName
p.streamName = streamName
p.workerID = workerID
p.processedBytes = prom.NewCounterVec(prom.CounterOpts{
Name: p.namespace + `_processed_bytes`,
Help: "Number of bytes processed",
}, []string{"kinesisStream", "shard"})
p.processedRecords = prom.NewCounterVec(prom.CounterOpts{
Name: p.namespace + `_processed_records`,
Help: "Number of records processed",
}, []string{"kinesisStream", "shard"})
p.behindLatestMillis = prom.NewGaugeVec(prom.GaugeOpts{
Name: p.namespace + `_behind_latest_millis`,
Help: "The amount of milliseconds processing is behind",
}, []string{"kinesisStream", "shard"})
p.leasesHeld = prom.NewGaugeVec(prom.GaugeOpts{
Name: p.namespace + `_leases_held`,
Help: "The number of leases held by the worker",
}, []string{"kinesisStream", "shard", "workerID"})
p.leaseRenewals = prom.NewCounterVec(prom.CounterOpts{
Name: p.namespace + `_lease_renewals`,
Help: "The number of successful lease renewals",
}, []string{"kinesisStream", "shard", "workerID"})
p.getRecordsTime = prom.NewHistogramVec(prom.HistogramOpts{
Name: p.namespace + `_get_records_duration_milliseconds`,
Help: "The time taken to fetch records and process them",
}, []string{"kinesisStream", "shard"})
p.processRecordsTime = prom.NewHistogramVec(prom.HistogramOpts{
Name: p.namespace + `_process_records_duration_milliseconds`,
Help: "The time taken to process records",
}, []string{"kinesisStream", "shard"})
metrics := []prom.Collector{
p.processedBytes,
p.processedRecords,
p.behindLatestMillis,
p.leasesHeld,
p.leaseRenewals,
p.getRecordsTime,
p.processRecordsTime,
}
for _, metric := range metrics {
err := prom.Register(metric)
if err != nil {
return err
}
}
return nil
}
func (p *MonitoringService) Start() error {
http.Handle("/metrics", promhttp.Handler())
go func() {
p.logger.Infof("Starting Prometheus listener on %s", p.listenAddress)
err := http.ListenAndServe(p.listenAddress, nil)
if err != nil {
p.logger.Errorf("Error starting Prometheus metrics endpoint. %+v", err)
}
p.logger.Infof("Stopped metrics server")
}()
return nil
}
func (p *MonitoringService) Shutdown() {}
func (p *MonitoringService) IncrRecordsProcessed(shard string, count int) {
p.processedRecords.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName}).Add(float64(count))
}
func (p *MonitoringService) IncrBytesProcessed(shard string, count int64) {
p.processedBytes.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName}).Add(float64(count))
}
func (p *MonitoringService) MillisBehindLatest(shard string, millSeconds float64) {
p.behindLatestMillis.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName}).Set(millSeconds)
}
func (p *MonitoringService) LeaseGained(shard string) {
p.leasesHeld.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName, "workerID": p.workerID}).Inc()
}
func (p *MonitoringService) LeaseLost(shard string) {
p.leasesHeld.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName, "workerID": p.workerID}).Dec()
}
func (p *MonitoringService) LeaseRenewed(shard string) {
p.leaseRenewals.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName, "workerID": p.workerID}).Inc()
}
func (p *MonitoringService) RecordGetRecordsTime(shard string, time float64) {
p.getRecordsTime.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName}).Observe(time)
}
func (p *MonitoringService) RecordProcessRecordsTime(shard string, time float64) {
p.processRecordsTime.With(prom.Labels{"shard": shard, "kinesisStream": p.streamName}).Observe(time)
}
================================================
FILE: clientlibrary/partition/partition.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package worker
import (
"sync"
"time"
"github.com/vmware/vmware-go-kcl/clientlibrary/config"
)
type ShardStatus struct {
ID string
ParentShardId string
Checkpoint string
AssignedTo string
Mux *sync.RWMutex
LeaseTimeout time.Time
// Shard Range
StartingSequenceNumber string
// child shard doesn't have end sequence number
EndingSequenceNumber string
ClaimRequest string
}
func (ss *ShardStatus) GetLeaseOwner() string {
ss.Mux.RLock()
defer ss.Mux.RUnlock()
return ss.AssignedTo
}
func (ss *ShardStatus) SetLeaseOwner(owner string) {
ss.Mux.Lock()
defer ss.Mux.Unlock()
ss.AssignedTo = owner
}
func (ss *ShardStatus) GetCheckpoint() string {
ss.Mux.RLock()
defer ss.Mux.RUnlock()
return ss.Checkpoint
}
func (ss *ShardStatus) SetCheckpoint(c string) {
ss.Mux.Lock()
defer ss.Mux.Unlock()
ss.Checkpoint = c
}
func (ss *ShardStatus) GetLeaseTimeout() time.Time {
ss.Mux.Lock()
defer ss.Mux.Unlock()
return ss.LeaseTimeout
}
func (ss *ShardStatus) SetLeaseTimeout(timeout time.Time) {
ss.Mux.Lock()
defer ss.Mux.Unlock()
ss.LeaseTimeout = timeout
}
func (ss *ShardStatus) IsClaimRequestExpired(kclConfig *config.KinesisClientLibConfiguration) bool {
if leaseTimeout := ss.GetLeaseTimeout(); leaseTimeout.IsZero() {
return false
} else {
return leaseTimeout.
Before(time.Now().UTC().Add(time.Duration(-kclConfig.LeaseStealingClaimTimeoutMillis) * time.Millisecond))
}
}
================================================
FILE: clientlibrary/utils/awserr.go
================================================
/*
* Copyright (c) 2021 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package utils
import (
"github.com/aws/aws-sdk-go/aws/awserr"
)
func AWSErrCode(err error) string {
awsErr, _ := err.(awserr.Error)
if awsErr != nil {
return awsErr.Code()
}
return ""
}
================================================
FILE: clientlibrary/utils/random.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// Package utils
package utils
import (
"crypto/rand"
"math/big"
"time"
)
const letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
const (
letterIdxBits = 6 // 6 bits to represent a letter index
letterIdxMask = 1<<letterIdxBits - 1 // All 1-bits, as many as letterIdxBits
letterIdxMax = 63 / letterIdxBits // # of letter indices fitting in 63 bits
)
func RandStringBytesMaskImpr(n int) string {
b := make([]byte, n)
seed := time.Now().UTC().UnixNano()
rnd, _ := rand.Int(rand.Reader, big.NewInt(seed))
// A rand.Int64() generates 64 random bits, enough for letterIdxMax letters!
for i, cache, remain := n-1, rnd.Int64(), letterIdxMax; i >= 0; {
if remain == 0 {
rnd, _ = rand.Int(rand.Reader, big.NewInt(seed))
cache, remain = rnd.Int64(), letterIdxMax
}
if idx := int(cache & letterIdxMask); idx < len(letterBytes) {
b[i] = letterBytes[idx]
i--
}
cache >>= letterIdxBits
remain--
}
return string(b)
}
================================================
FILE: clientlibrary/utils/random_test.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package utils
import (
"fmt"
"testing"
"time"
)
func TestRandom(t *testing.T) {
for i := 0; i < 10; i++ {
s1 := RandStringBytesMaskImpr(10)
s2 := RandStringBytesMaskImpr(10)
if s1 == s2 {
t.Fatalf("failed in generating random string. s1: %s, s2: %s", s1, s2)
}
fmt.Println(s1)
fmt.Println(s2)
}
}
func TestRandomNum(t *testing.T) {
for i := 0; i < 10; i++ {
seed := time.Now().UTC().Second()
s1 := RandStringBytesMaskImpr(seed)
s2 := RandStringBytesMaskImpr(seed)
if s1 == s2 {
t.Fatalf("failed in generating random string. s1: %s, s2: %s", s1, s2)
}
fmt.Println(s1)
fmt.Println(s2)
}
}
================================================
FILE: clientlibrary/utils/uuid.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package utils
import (
guuid "github.com/google/uuid"
)
// MustNewUUID generates a new UUID and panics if failed
func MustNewUUID() string {
id, err := guuid.NewUUID()
if err != nil {
panic(err)
}
return id.String()
}
================================================
FILE: clientlibrary/worker/common-shard-consumer.go
================================================
/*
* Copyright (c) 2021 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package worker
import (
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/kinesis"
"github.com/aws/aws-sdk-go/service/kinesis/kinesisiface"
deagg "github.com/awslabs/kinesis-aggregation/go/deaggregator"
chk "github.com/vmware/vmware-go-kcl/clientlibrary/checkpoint"
"github.com/vmware/vmware-go-kcl/clientlibrary/config"
kcl "github.com/vmware/vmware-go-kcl/clientlibrary/interfaces"
"github.com/vmware/vmware-go-kcl/clientlibrary/metrics"
par "github.com/vmware/vmware-go-kcl/clientlibrary/partition"
)
type shardConsumer interface {
getRecords() error
}
// commonShardConsumer implements common functionality for regular and enhanced fan-out consumers
type commonShardConsumer struct {
shard *par.ShardStatus
kc kinesisiface.KinesisAPI
checkpointer chk.Checkpointer
recordProcessor kcl.IRecordProcessor
kclConfig *config.KinesisClientLibConfiguration
mService metrics.MonitoringService
}
// Cleanup the internal lease cache
func (sc *commonShardConsumer) releaseLease() {
log := sc.kclConfig.Logger
log.Infof("Release lease for shard %s", sc.shard.ID)
sc.shard.SetLeaseOwner("")
// Release the lease by wiping out the lease owner for the shard
// Note: we don't need to do anything in case of error here and shard lease will eventually be expired.
if err := sc.checkpointer.RemoveLeaseOwner(sc.shard.ID); err != nil {
log.Errorf("Failed to release shard lease or shard: %s Error: %+v", sc.shard.ID, err)
}
// reporting lease lose metrics
sc.mService.LeaseLost(sc.shard.ID)
}
// getStartingPosition gets kinesis stating position.
// First try to fetch checkpoint. If checkpoint is not found use InitialPositionInStream
func (sc *commonShardConsumer) getStartingPosition() (*kinesis.StartingPosition, error) {
err := sc.checkpointer.FetchCheckpoint(sc.shard)
if err != nil && err != chk.ErrSequenceIDNotFound {
return nil, err
}
checkpoint := sc.shard.GetCheckpoint()
if checkpoint != "" {
sc.kclConfig.Logger.Debugf("Start shard: %v at checkpoint: %v", sc.shard.ID, checkpoint)
return &kinesis.StartingPosition{
Type: aws.String("AFTER_SEQUENCE_NUMBER"),
SequenceNumber: &checkpoint,
}, nil
}
shardIteratorType := config.InitalPositionInStreamToShardIteratorType(sc.kclConfig.InitialPositionInStream)
sc.kclConfig.Logger.Debugf("No checkpoint recorded for shard: %v, starting with: %v", sc.shard.ID, aws.StringValue(shardIteratorType))
if sc.kclConfig.InitialPositionInStream == config.AT_TIMESTAMP {
return &kinesis.StartingPosition{
Type: shardIteratorType,
Timestamp: sc.kclConfig.InitialPositionInStreamExtended.Timestamp,
}, nil
}
return &kinesis.StartingPosition{
Type: shardIteratorType,
}, nil
}
// Need to wait until the parent shard finished
func (sc *commonShardConsumer) waitOnParentShard() error {
if len(sc.shard.ParentShardId) == 0 {
return nil
}
pshard := &par.ShardStatus{
ID: sc.shard.ParentShardId,
Mux: &sync.RWMutex{},
}
for {
if err := sc.checkpointer.FetchCheckpoint(pshard); err != nil {
return err
}
// Parent shard is finished.
if pshard.GetCheckpoint() == chk.ShardEnd {
return nil
}
time.Sleep(time.Duration(sc.kclConfig.ParentShardPollIntervalMillis) * time.Millisecond)
}
}
func (sc *commonShardConsumer) processRecords(getRecordsStartTime time.Time, records []*kinesis.Record, millisBehindLatest *int64, recordCheckpointer kcl.IRecordProcessorCheckpointer) {
log := sc.kclConfig.Logger
getRecordsTime := time.Since(getRecordsStartTime).Milliseconds()
sc.mService.RecordGetRecordsTime(sc.shard.ID, float64(getRecordsTime))
log.Debugf("Received %d original records.", len(records))
// De-aggregate the records if they were published by the KPL.
dars, err := deagg.DeaggregateRecords(records)
if err != nil {
// The error is caused by bad KPL publisher and just skip the bad records
// instead of being stuck here.
log.Errorf("Error in de-aggregating KPL records: %+v", err)
}
input := &kcl.ProcessRecordsInput{
Records: dars,
MillisBehindLatest: aws.Int64Value(millisBehindLatest),
Checkpointer: recordCheckpointer,
}
recordLength := len(input.Records)
recordBytes := int64(0)
log.Debugf("Received %d de-aggregated records, MillisBehindLatest: %v", recordLength, input.MillisBehindLatest)
for _, r := range input.Records {
recordBytes += int64(len(r.Data))
}
if recordLength > 0 || sc.kclConfig.CallProcessRecordsEvenForEmptyRecordList {
processRecordsStartTime := time.Now()
// Delivery the events to the record processor
input.CacheEntryTime = &getRecordsStartTime
input.CacheExitTime = &processRecordsStartTime
sc.recordProcessor.ProcessRecords(input)
processedRecordsTiming := time.Since(processRecordsStartTime).Milliseconds()
sc.mService.RecordProcessRecordsTime(sc.shard.ID, float64(processedRecordsTiming))
}
sc.mService.IncrRecordsProcessed(sc.shard.ID, recordLength)
sc.mService.IncrBytesProcessed(sc.shard.ID, recordBytes)
sc.mService.MillisBehindLatest(sc.shard.ID, float64(*millisBehindLatest))
}
================================================
FILE: clientlibrary/worker/fan-out-shard-consumer.go
================================================
/*
* Copyright (c) 2021 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package worker
import (
"errors"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/kinesis"
chk "github.com/vmware/vmware-go-kcl/clientlibrary/checkpoint"
kcl "github.com/vmware/vmware-go-kcl/clientlibrary/interfaces"
)
// FanOutShardConsumer is responsible for consuming data records of a (specified) shard.
// Note: FanOutShardConsumer only deal with one shard.
// For more info see: https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html
type FanOutShardConsumer struct {
commonShardConsumer
consumerARN string
consumerID string
stop *chan struct{}
}
// getRecords subscribes to a shard and reads events from it.
// Precondition: it currently has the lease on the shard.
func (sc *FanOutShardConsumer) getRecords() error {
defer sc.releaseLease()
log := sc.kclConfig.Logger
// If the shard is child shard, need to wait until the parent finished.
if err := sc.waitOnParentShard(); err != nil {
// If parent shard has been deleted by Kinesis system already, just ignore the error.
if err != chk.ErrSequenceIDNotFound {
log.Errorf("Error in waiting for parent shard: %v to finish. Error: %+v", sc.shard.ParentShardId, err)
return err
}
}
shardSub, err := sc.subscribeToShard()
if err != nil {
log.Errorf("Unable to subscribe to shard %s: %v", sc.shard.ID, err)
return err
}
defer func() {
if shardSub == nil || shardSub.EventStream == nil {
log.Debugf("Nothing to close, EventStream is nil")
return
}
err = shardSub.EventStream.Close()
if err != nil {
log.Errorf("Unable to close event stream for %s: %v", sc.shard.ID, err)
}
}()
input := &kcl.InitializationInput{
ShardId: sc.shard.ID,
ExtendedSequenceNumber: &kcl.ExtendedSequenceNumber{SequenceNumber: aws.String(sc.shard.GetCheckpoint())},
}
sc.recordProcessor.Initialize(input)
recordCheckpointer := NewRecordProcessorCheckpoint(sc.shard, sc.checkpointer)
var continuationSequenceNumber *string
refreshLeaseTimer := time.After(time.Until(sc.shard.LeaseTimeout.Add(-time.Duration(sc.kclConfig.LeaseRefreshPeriodMillis) * time.Millisecond)))
for {
getRecordsStartTime := time.Now()
select {
case <-*sc.stop:
shutdownInput := &kcl.ShutdownInput{ShutdownReason: kcl.REQUESTED, Checkpointer: recordCheckpointer}
sc.recordProcessor.Shutdown(shutdownInput)
return nil
case <-refreshLeaseTimer:
log.Debugf("Refreshing lease on shard: %s for worker: %s", sc.shard.ID, sc.consumerID)
err = sc.checkpointer.GetLease(sc.shard, sc.consumerID)
if err != nil {
if errors.As(err, &chk.ErrLeaseNotAcquired{}) {
log.Warnf("Failed in acquiring lease on shard: %s for worker: %s", sc.shard.ID, sc.consumerID)
return nil
}
log.Errorf("Error in refreshing lease on shard: %s for worker: %s. Error: %+v", sc.shard.ID, sc.consumerID, err)
return err
}
refreshLeaseTimer = time.After(time.Until(sc.shard.LeaseTimeout.Add(-time.Duration(sc.kclConfig.LeaseRefreshPeriodMillis) * time.Millisecond)))
case event, ok := <-shardSub.EventStream.Events():
if !ok {
// need to resubscribe to shard
log.Debugf("Event stream ended, refreshing subscription on shard: %s for worker: %s", sc.shard.ID, sc.consumerID)
if continuationSequenceNumber == nil || *continuationSequenceNumber == "" {
log.Debugf("No continuation sequence number")
return nil
}
shardSub, err = sc.resubscribe(shardSub, continuationSequenceNumber)
if err != nil {
return err
}
continue
}
subEvent, ok := event.(*kinesis.SubscribeToShardEvent)
if !ok {
log.Errorf("Received unexpected event type: %T", event)
continue
}
continuationSequenceNumber = subEvent.ContinuationSequenceNumber
sc.processRecords(getRecordsStartTime, subEvent.Records, subEvent.MillisBehindLatest, recordCheckpointer)
// The shard has been closed, so no new records can be read from it
if continuationSequenceNumber == nil {
log.Infof("Shard %s closed", sc.shard.ID)
shutdownInput := &kcl.ShutdownInput{ShutdownReason: kcl.TERMINATE, Checkpointer: recordCheckpointer}
sc.recordProcessor.Shutdown(shutdownInput)
return nil
}
}
}
}
func (sc *FanOutShardConsumer) subscribeToShard() (*kinesis.SubscribeToShardOutput, error) {
startPosition, err := sc.getStartingPosition()
if err != nil {
return nil, err
}
return sc.kc.SubscribeToShard(&kinesis.SubscribeToShardInput{
ConsumerARN: &sc.consumerARN,
ShardId: &sc.shard.ID,
StartingPosition: startPosition,
})
}
func (sc *FanOutShardConsumer) resubscribe(shardSub *kinesis.SubscribeToShardOutput, continuationSequence *string) (*kinesis.SubscribeToShardOutput, error) {
err := shardSub.EventStream.Close()
if err != nil {
sc.kclConfig.Logger.Errorf("Unable to close event stream for %s: %v", sc.shard.ID, err)
return nil, err
}
startPosition := &kinesis.StartingPosition{
Type: aws.String("AFTER_SEQUENCE_NUMBER"),
SequenceNumber: continuationSequence,
}
shardSub, err = sc.kc.SubscribeToShard(&kinesis.SubscribeToShardInput{
ConsumerARN: &sc.consumerARN,
ShardId: &sc.shard.ID,
StartingPosition: startPosition,
})
if err != nil {
sc.kclConfig.Logger.Errorf("Unable to resubscribe to shard %s: %v", sc.shard.ID, err)
return nil, err
}
return shardSub, nil
}
================================================
FILE: clientlibrary/worker/polling-shard-consumer.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package worker
import (
"errors"
"math"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/kinesis"
chk "github.com/vmware/vmware-go-kcl/clientlibrary/checkpoint"
kcl "github.com/vmware/vmware-go-kcl/clientlibrary/interfaces"
"github.com/vmware/vmware-go-kcl/clientlibrary/metrics"
"github.com/vmware/vmware-go-kcl/clientlibrary/utils"
)
// PollingShardConsumer is responsible for polling data records from a (specified) shard.
// Note: PollingShardConsumer only deal with one shard.
type PollingShardConsumer struct {
commonShardConsumer
streamName string
stop *chan struct{}
consumerID string
mService metrics.MonitoringService
}
func (sc *PollingShardConsumer) getShardIterator() (*string, error) {
startPosition, err := sc.getStartingPosition()
if err != nil {
return nil, err
}
shardIterArgs := &kinesis.GetShardIteratorInput{
ShardId: &sc.shard.ID,
ShardIteratorType: startPosition.Type,
StartingSequenceNumber: startPosition.SequenceNumber,
Timestamp: startPosition.Timestamp,
StreamName: &sc.streamName,
}
iterResp, err := sc.kc.GetShardIterator(shardIterArgs)
if err != nil {
return nil, err
}
return iterResp.ShardIterator, nil
}
// getRecords continously poll one shard for data record
// Precondition: it currently has the lease on the shard.
func (sc *PollingShardConsumer) getRecords() error {
defer sc.releaseLease()
log := sc.kclConfig.Logger
// If the shard is child shard, need to wait until the parent finished.
if err := sc.waitOnParentShard(); err != nil {
// If parent shard has been deleted by Kinesis system already, just ignore the error.
if err != chk.ErrSequenceIDNotFound {
log.Errorf("Error in waiting for parent shard: %v to finish. Error: %+v", sc.shard.ParentShardId, err)
return err
}
}
shardIterator, err := sc.getShardIterator()
if err != nil {
log.Errorf("Unable to get shard iterator for %s: %v", sc.shard.ID, err)
return err
}
// Start processing events and notify record processor on shard and starting checkpoint
input := &kcl.InitializationInput{
ShardId: sc.shard.ID,
ExtendedSequenceNumber: &kcl.ExtendedSequenceNumber{SequenceNumber: aws.String(sc.shard.GetCheckpoint())},
}
sc.recordProcessor.Initialize(input)
recordCheckpointer := NewRecordProcessorCheckpoint(sc.shard, sc.checkpointer)
retriedErrors := 0
for {
if time.Now().UTC().After(sc.shard.GetLeaseTimeout().Add(-time.Duration(sc.kclConfig.LeaseRefreshPeriodMillis) * time.Millisecond)) {
log.Debugf("Refreshing lease on shard: %s for worker: %s", sc.shard.ID, sc.consumerID)
err = sc.checkpointer.GetLease(sc.shard, sc.consumerID)
if err != nil {
if errors.As(err, &chk.ErrLeaseNotAcquired{}) {
log.Warnf("Failed in acquiring lease on shard: %s for worker: %s", sc.shard.ID, sc.consumerID)
return nil
}
// log and return error
log.Errorf("Error in refreshing lease on shard: %s for worker: %s. Error: %+v",
sc.shard.ID, sc.consumerID, err)
return err
}
}
getRecordsStartTime := time.Now()
log.Debugf("Trying to read %d record from iterator: %v", sc.kclConfig.MaxRecords, aws.StringValue(shardIterator))
getRecordsArgs := &kinesis.GetRecordsInput{
Limit: aws.Int64(int64(sc.kclConfig.MaxRecords)),
ShardIterator: shardIterator,
}
// Get records from stream and retry as needed
getResp, err := sc.kc.GetRecords(getRecordsArgs)
if err != nil {
if utils.AWSErrCode(err) == kinesis.ErrCodeProvisionedThroughputExceededException || utils.AWSErrCode(err) == kinesis.ErrCodeKMSThrottlingException {
log.Errorf("Error getting records from shard %v: %+v", sc.shard.ID, err)
retriedErrors++
// exponential backoff
// https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.RetryAndBackoff
time.Sleep(time.Duration(math.Exp2(float64(retriedErrors))*100) * time.Millisecond)
continue
}
log.Errorf("Error getting records from Kinesis that cannot be retried: %+v Request: %s", err, getRecordsArgs)
return err
}
// reset the retry count after success
retriedErrors = 0
sc.processRecords(getRecordsStartTime, getResp.Records, getResp.MillisBehindLatest, recordCheckpointer)
// The shard has been closed, so no new records can be read from it
if getResp.NextShardIterator == nil {
log.Infof("Shard %s closed", sc.shard.ID)
shutdownInput := &kcl.ShutdownInput{ShutdownReason: kcl.TERMINATE, Checkpointer: recordCheckpointer}
sc.recordProcessor.Shutdown(shutdownInput)
return nil
}
shardIterator = getResp.NextShardIterator
// Idle between each read, the user is responsible for checkpoint the progress
// This value is only used when no records are returned; if records are returned, it should immediately
// retrieve the next set of records.
if len(getResp.Records) == 0 && aws.Int64Value(getResp.MillisBehindLatest) < int64(sc.kclConfig.IdleTimeBetweenReadsInMillis) {
time.Sleep(time.Duration(sc.kclConfig.IdleTimeBetweenReadsInMillis) * time.Millisecond)
}
select {
case <-*sc.stop:
shutdownInput := &kcl.ShutdownInput{ShutdownReason: kcl.REQUESTED, Checkpointer: recordCheckpointer}
sc.recordProcessor.Shutdown(shutdownInput)
return nil
default:
}
}
}
================================================
FILE: clientlibrary/worker/record-processor-checkpointer.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package worker
import (
"github.com/aws/aws-sdk-go/aws"
chk "github.com/vmware/vmware-go-kcl/clientlibrary/checkpoint"
kcl "github.com/vmware/vmware-go-kcl/clientlibrary/interfaces"
par "github.com/vmware/vmware-go-kcl/clientlibrary/partition"
)
type (
/* Objects of this class are prepared to checkpoint at a specific sequence number. They use an
* IRecordProcessorCheckpointer to do the actual checkpointing, so their checkpoint is subject to the same 'didn't go
* backwards' validation as a normal checkpoint.
*/
PreparedCheckpointer struct {
pendingCheckpointSequenceNumber *kcl.ExtendedSequenceNumber
checkpointer kcl.IRecordProcessorCheckpointer
}
/**
* This class is used to enable RecordProcessors to checkpoint their progress.
* The Amazon Kinesis Client Library will instantiate an object and provide a reference to the application
* RecordProcessor instance. Amazon Kinesis Client Library will create one instance per shard assignment.
*/
RecordProcessorCheckpointer struct {
shard *par.ShardStatus
checkpoint chk.Checkpointer
}
)
func NewRecordProcessorCheckpoint(shard *par.ShardStatus, checkpoint chk.Checkpointer) kcl.IRecordProcessorCheckpointer {
return &RecordProcessorCheckpointer{
shard: shard,
checkpoint: checkpoint,
}
}
func (pc *PreparedCheckpointer) GetPendingCheckpoint() *kcl.ExtendedSequenceNumber {
return pc.pendingCheckpointSequenceNumber
}
func (pc *PreparedCheckpointer) Checkpoint() error {
return pc.checkpointer.Checkpoint(pc.pendingCheckpointSequenceNumber.SequenceNumber)
}
func (rc *RecordProcessorCheckpointer) Checkpoint(sequenceNumber *string) error {
// checkpoint the last sequence of a closed shard
if sequenceNumber == nil {
rc.shard.SetCheckpoint(chk.ShardEnd)
} else {
rc.shard.SetCheckpoint(aws.StringValue(sequenceNumber))
}
return rc.checkpoint.CheckpointSequence(rc.shard)
}
func (rc *RecordProcessorCheckpointer) PrepareCheckpoint(sequenceNumber *string) (kcl.IPreparedCheckpointer, error) {
return &PreparedCheckpointer{}, nil
}
================================================
FILE: clientlibrary/worker/worker-fan-out.go
================================================
/*
* Copyright (c) 2021 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package worker
import (
"fmt"
"math"
"time"
"github.com/aws/aws-sdk-go/service/kinesis"
"github.com/vmware/vmware-go-kcl/clientlibrary/utils"
)
// fetchConsumerARNWithRetry tries to fetch consumer ARN. Retries 10 times with exponential backoff in case of an error
func (w *Worker) fetchConsumerARNWithRetry() (string, error) {
for retry := 0; ; retry++ {
consumerARN, err := w.fetchConsumerARN()
if err == nil {
return consumerARN, nil
}
if retry < 10 {
sleepDuration := time.Duration(math.Exp2(float64(retry))*100) * time.Millisecond
w.kclConfig.Logger.Errorf("Could not get consumer ARN: %v, retrying after: %s", err, sleepDuration)
time.Sleep(sleepDuration)
continue
}
return consumerARN, err
}
}
// fetchConsumerARN gets enhanced fan-out consumerARN.
// Registers enhanced fan-out consumer if the consumer is not found
func (w *Worker) fetchConsumerARN() (string, error) {
log := w.kclConfig.Logger
log.Debugf("Fetching stream consumer ARN")
streamDescription, err := w.kc.DescribeStream(&kinesis.DescribeStreamInput{
StreamName: &w.kclConfig.StreamName,
})
if err != nil {
log.Errorf("Could not describe stream: %v", err)
return "", err
}
streamConsumerDescription, err := w.kc.DescribeStreamConsumer(&kinesis.DescribeStreamConsumerInput{
ConsumerName: &w.kclConfig.EnhancedFanOutConsumerName,
StreamARN: streamDescription.StreamDescription.StreamARN,
})
if err == nil {
log.Infof("Enhanced fan-out consumer found, consumer status: %s", *streamConsumerDescription.ConsumerDescription.ConsumerStatus)
if *streamConsumerDescription.ConsumerDescription.ConsumerStatus != kinesis.ConsumerStatusActive {
return "", fmt.Errorf("consumer is not in active status yet, current status: %s", *streamConsumerDescription.ConsumerDescription.ConsumerStatus)
}
return *streamConsumerDescription.ConsumerDescription.ConsumerARN, nil
}
if utils.AWSErrCode(err) == kinesis.ErrCodeResourceNotFoundException {
log.Infof("Enhanced fan-out consumer not found, registering new consumer with name: %s", w.kclConfig.EnhancedFanOutConsumerName)
out, err := w.kc.RegisterStreamConsumer(&kinesis.RegisterStreamConsumerInput{
ConsumerName: &w.kclConfig.EnhancedFanOutConsumerName,
StreamARN: streamDescription.StreamDescription.StreamARN,
})
if err != nil {
log.Errorf("Could not register enhanced fan-out consumer: %v", err)
return "", err
}
if *out.Consumer.ConsumerStatus != kinesis.ConsumerStatusActive {
return "", fmt.Errorf("consumer is not in active status yet, current status: %s", *out.Consumer.ConsumerStatus)
}
return *out.Consumer.ConsumerARN, nil
}
log.Errorf("Could not describe stream consumer: %v", err)
return "", err
}
================================================
FILE: clientlibrary/worker/worker.go
================================================
/*
* Copyright (c) 2018 VMware, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
* associated documentation files (the "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is furnished to do
* so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or substantial
* portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
* NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
// Package worker
// The implementation is derived from https://github.com/patrobinson/gokini
//
// Copyright 2018 Patrick robinson
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
package worker
import (
"crypto/rand"
"errors"
"math/big"
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/kinesis"
"github.com/aws/aws-sdk-go/service/kinesis/kinesisiface"
chk "github.com/vmware/vmware-go-kcl/clientlibrary/checkpoint"
"github.com/vmware/vmware-go-kcl/clientlibrary/config"
kcl "github.com/vmware/vmware-go-kcl/clientlibrary/interfaces"
"github.com/vmware/vmware-go-kcl/clientlibrary/metrics"
par "github.com/vmware/vmware-go-kcl/clientlibrary/partition"
)
//Worker is the high level class that Kinesis applications use to start processing data. It initializes and oversees
//different components (e.g. syncing shard and lease information, tracking shard assignments, and processing data from
//the shards).
type Worker struct {
streamName string
regionName string
workerID string
consumerARN string
processorFactory kcl.IRecordProcessorFactory
kclConfig *config.KinesisClientLibConfiguration
kc kinesisiface.KinesisAPI
checkpointer chk.Checkpointer
mService metrics.MonitoringService
stop *chan struct{}
waitGroup *sync.WaitGroup
done bool
randomSeed int64
shardStatus map[string]*par.ShardStatus
shardStealInProgress bool
}
// NewWorker constructs a Worker instance for processing Kinesis stream data.
func NewWorker(factory kcl.IRecordProcessorFactory, kclConfig *config.KinesisClientLibConfiguration) *Worker {
mService := kclConfig.MonitoringService
if mService == nil {
// Replaces nil with noop monitor service (not emitting any metrics).
mService = metrics.NoopMonitoringService{}
}
return &Worker{
streamName: kclConfig.StreamName,
regionName: kclConfig.RegionName,
workerID: kclConfig.WorkerID,
processorFactory: factory,
kclConfig: kclConfig,
mService: mService,
done: false,
randomSeed: time.Now().UTC().UnixNano(),
}
}
// WithKinesis is used to provide Kinesis service for either custom implementation or unit testing.
func (w *Worker) WithKinesis(svc kinesisiface.KinesisAPI) *Worker {
w.kc = svc
return w
}
// WithCheckpointer is used to provide a custom checkpointer service for non-dynamodb implementation
// or unit testing.
func (w *Worker) WithCheckpointer(checker chk.Checkpointer) *Worker {
w.checkpointer = checker
return w
}
// Start Run starts consuming data from the stream, and pass it to the application record processors.
func (w *Worker) Start() error {
log := w.kclConfig.Logger
if err := w.initialize(); err != nil {
log.Errorf("Failed to initialize Worker: %+v", err)
return err
}
// Start monitoring service
log.Infof("Starting monitoring service.")
if err := w.mService.Start(); err != nil {
log.Errorf("Failed to start monitoring service: %+v", err)
return err
}
log.Infof("Starting worker event loop.")
w.waitGroup.Add(1)
go func() {
defer w.waitGroup.Done()
// entering event loop
w.eventLoop()
}()
return nil
}
// Shutdown signals worker to shut down. Worker will try initiating shutdown of all record processors.
func (w *Worker) Shutdown() {
log := w.kclConfig.Logger
log.Infof("Worker shutdown in requested.")
if w.done || w.stop == nil {
return
}
close(*w.stop)
w.done = true
w.waitGroup.Wait()
w.mService.Shutdown()
log.Infof("Worker loop is complete. Exiting from worker.")
}
// initialize
func (w *Worker) initialize() error {
log := w.kclConfig.Logger
log.Infof("Worker initialization in progress...")
// Create default Kinesis session
if w.kc == nil {
// create session for Kinesis
log.Infof("Creating Kinesis session")
s, err := session.NewSession(&aws.Config{
Region: aws.String(w.regionName),
Endpoint: &w.kclConfig.KinesisEndpoint,
Credentials: w.kclConfig.KinesisCredentials,
})
if err != nil {
// no need to move forward
log.Fatalf("Failed in getting Kinesis session for creating Worker: %+v", err)
}
w.kc = kinesis.New(s)
} else {
log.Infof("Use custom Kinesis service.")
}
// Create default dynamodb based checkpointer implementation
if w.checkpointer == nil {
log.Infof("Creating DynamoDB based checkpointer")
w.checkpointer = chk.NewDynamoCheckpoint(w.kclConfig)
} else {
log.Infof("Use custom checkpointer implementation.")
}
if w.kclConfig.EnableEnhancedFanOutConsumer {
log.Debugf("Enhanced fan-out is enabled")
w.consumerARN = w.kclConfig.EnhancedFanOutConsumerARN
if w.consumerARN == "" {
var err error
w.consumerARN, err = w.fetchConsumerARNWithRetry()
if err != nil {
log.Errorf("Failed to fetch consumer ARN for: %s, %v", w.kclConfig.EnhancedFanOutConsumerName, err)
return err
}
}
}
err := w.mService.Init(w.kclConfig.ApplicationName, w.streamName, w.workerID)
if err != nil {
log.Errorf("Failed to start monitoring service: %+v", err)
}
log.Infof("Initializing Checkpointer")
if err := w.checkpointer.Init(); err != nil {
log.Errorf("Failed to start Checkpointer: %+v", err)
return err
}
w.shardStatus = make(map[string]*par.ShardStatus)
stopChan := make(chan struct{})
w.stop = &stopChan
w.waitGroup = &sync.WaitGroup{}
log.Infof("Initialization complete.")
return nil
}
// newShardConsumer creates shard consumer for the specified shard
func (w *Worker) newShardConsumer(shard *par.ShardStatus) shardConsumer {
common := commonShardConsumer{
shard: shard,
kc: w.kc,
checkpointer: w.checkpointer,
recordProcessor: w.processorFactory.CreateProcessor(),
kclConfig: w.kclConfig,
mService: w.mService,
}
if w.kclConfig.EnableEnhancedFanOutConsumer {
w.kclConfig.Logger.Infof("Start enhanced fan-out shard consumer for shard: %v", shard.ID)
return &FanOutShardConsumer{
commonShardConsumer: common,
consumerARN: w.consumerARN,
consumerID: w.workerID,
stop: w.stop,
}
}
w.kclConfig.Logger.Infof("Start polling shard consumer for shard: %v", shard.ID)
return &PollingShardConsumer{
commonShardConsumer: common,
streamName: w.streamName,
consumerID: w.workerID,
stop: w.stop,
mService: w.mService,
}
}
// eventLoop
func (w *Worker) eventLoop() {
log := w.kclConfig.Logger
var foundShards int
for {
// Add [-50%, +50%] random jitter to ShardSyncIntervalMillis. When multiple workers
// starts at the same time, this decreases the probability of them calling
// kinesis.DescribeStream at the same time, and hit the hard-limit on aws API calls.
// On average the period remains the same so that doesn't affect behavior.
rnd, _ := rand.Int(rand.Reader, big.NewInt(int64(w.kclConfig.ShardSyncIntervalMillis)))
shardSyncSleep := w.kclConfig.ShardSyncIntervalMillis/2 + int(rnd.Int64())
err := w.syncShard()
if err != nil {
log.Errorf("Error syncing shards: %+v, Retrying in %d ms...", err, shardSyncSleep)
time.Sleep(time.Duration(shardSyncSleep) * time.Millisecond)
continue
}
if foundShards == 0 || foundShards != len(w.shardStatus) {
foundShards = len(w.shardStatus)
log.Infof("Found %d shards", foundShards)
}
// Count the number of leases held by this worker excluding the processed shard
counter := 0
for _, shard := range w.shardStatus {
if shard.GetLeaseOwner() == w.workerID && shard.GetCheckpoint() != chk.ShardEnd {
counter++
}
}
// max number of lease has not been reached yet
if counter < w.kclConfig.MaxLeasesForWorker {
for _, shard := range w.shardStatus {
// already owner of the shard
if shard.GetLeaseOwner() == w.workerID {
continue
}
err := w.checkpointer.FetchCheckpoint(shard)
if err != nil {
// checkpoint may not exist yet is not an error condition.
if err != chk.ErrSequenceIDNotFound {
log.Warnf("Couldn't fetch checkpoint: %+v", err)
// move on to next shard
continue
}
}
// The shard is closed and we have processed all records
if shard.GetCheckpoint() == chk.ShardEnd {
continue
}
var stealShard bool
if w.kclConfig.EnableLeaseStealing && shard.ClaimRequest != "" {
upcomingStealingInterval := time.Now().UTC().Add(time.Duration(w.kclConfig.LeaseStealingIntervalMillis) * time.Millisecond)
if shard.GetLeaseTimeout().Before(upcomingStealingInterval) && !shard.IsClaimRequestExpired(w.kclConfig) {
if shard.ClaimRequest == w.workerID {
stealShard = true
log.Debugf("Stealing shard: %s", shard.ID)
} else {
log.Debugf("Shard being stolen: %s", shard.ID)
continue
}
}
}
err = w.checkpointer.GetLease(shard, w.workerID)
if err != nil {
// cannot get lease on the shard
if !errors.As(err, &chk.ErrLeaseNotAcquired{}) {
log.Errorf("Cannot get lease: %+v", err)
}
continue
}
if stealShard {
log.Debugf("Successfully stole shard: %+v", shard.ID)
w.shardStealInProgress = false
}
// log metrics on got lease
w.mService.LeaseGained(shard.ID)
w.waitGroup.Add(1)
go func(shard *par.ShardStatus) {
defer w.waitGroup.Done()
if err := w.newShardConsumer(shard).getRecords(); err != nil {
log.Errorf("Error in getRecords: %+v", err)
}
}(shard)
// exit from for loop and not to grab more shard for now.
break
}
}
if w.kclConfig.EnableLeaseStealing {
err = w.rebalance()
if err != nil {
log.Warnf("Error in rebalance: %+v", err)
}
}
select {
case <-*w.stop:
log.Infof("Shutting down...")
return
case <-time.After(time.Duration(shardSyncSleep) * time.Millisecond):
log.Debugf("Waited %d ms to sync shards...", shardSyncSleep)
}
}
}
func (w *Worker) rebalance() error {
log := w.kclConfig.Logger
workers, err := w.checkpointer.ListActiveWorkers(w.shardStatus)
if err != nil {
log.Debugf("Error listing workers. workerID: %s. Error: %+v ", w.workerID, err)
return err
}
// Only attempt to steal one shard at time, to allow for linear convergence
if w.shardStealInProgress {
shardInfo := make(map[string]bool)
err := w.getShardIDs("", shardInfo)
if err != nil {
return err
}
for _, shard := range w.shardStatus {
if shard.ClaimRequest != "" && shard.ClaimRequest == w.workerID {
log.Debugf("Steal in progress. workerID: %s", w.workerID)
return nil
}
// Our shard steal was stomped on by a Checkpoint.
// We could deal with that, but instead just try again
w.shardStealInProgress = false
}
}
var numShards int
for _, shards := range workers {
numShards += len(shards)
}
numWorkers := len(workers)
// 1:1 shards to workers is optimal, so we cannot possibly rebalance
if numWorkers >= numShards {
log.Debugf("Optimal shard allocation, not stealing any shards. workerID: %s, %v > %v. ", w.workerID, numWorkers, numShards)
return nil
}
currentShards, ok := workers[w.workerID]
var numCurrentShards int
if !ok {
numCurrentShards = 0
numWorkers++
} else {
numCurrentShards = len(currentShards)
}
optimalShards := numShards / numWorkers
// We have more than or equal optimal shards, so no rebalancing can take place
if numCurrentShards >= optimalShards || numCurrentShards == w.kclConfig.MaxLeasesForWorker {
log.Debugf("We have enough shards, not attempting to steal any. workerID: %s", w.workerID)
return nil
}
var workerSteal string
for worker, shards := range workers {
if worker != w.workerID && len(shards) > optimalShards {
workerSteal = worker
optimalShards = len(shards)
}
}
// Not all shards are allocated so fallback to default shard allocation mechanisms
if workerSteal == "" {
log.Infof("Not all shards are allocated, not stealing any. workerID: %s", w.workerID)
return nil
}
// Steal a random shard from the worker with the most shards
w.shardStealInProgress = true
rnd, _ := rand.Int(rand.Reader, big.NewInt(int64(len(workers[workerSteal]))))
randIndex := int(rnd.Int64())
shardToSteal := workers[workerSteal][randIndex]
log.Debugf("Stealing shard %s from %s", shardToSteal, workerSteal)
err = w.checkpointer.ClaimShard(w.shardStatus[shardToSteal.ID], w.workerID)
if err != nil {
w.shardStealInProgress = false
return err
}
return nil
}
// List all shards and store them into shardStatus table
// If shard has been removed, need to exclude it from cached shard status.
func (w *Worker) getShardIDs(nextToken string, shardInfo map[string]bool) error {
log := w.kclConfig.Logger
args := &kinesis.ListShardsInput{}
// When you have a nextToken, you can't set the streamName
if nextToken != "" {
args.NextToken = aws.String(nextToken)
} else {
args.StreamName = aws.String(w.streamName)
}
listShards, err := w.kc.ListShards(args)
if err != nil {
log.Errorf("Error in ListShards: %s Error: %+v Request: %s", w.streamName, err, args)
return err
}
for _, s := range listShards.Shards {
// record avail shardId from fresh reading from Kinesis
shardInfo[*s.ShardId] = true
// found new shard
if _, ok := w.shardStatus[*s.ShardId]; !ok {
log.Infof("Found new shard with id %s", *s.ShardId)
w.shardStatus[*s.ShardId] = &par.ShardStatus{
ID: *s.ShardId,
ParentShardId: aws.StringValue(s.ParentShardId),
Mux: &sync.RWMutex{},
StartingSequenceNumber: aws.StringValue(s.SequenceNumberRange.StartingSequenceNumber),
EndingSequenceNumber: aws.StringValue(s.SequenceNumberRange.EndingSequenceNumber),
}
}
}
if listShards.NextToken != nil {
err := w.getShardIDs(aws.StringValue(listShards.NextToken), shardInfo)
if err != nil {
log.Errorf("Error in ListShards: %s Error: %+v Request: %s", w.streamName, err, args)
return err
}
}
return nil
}
// syncShard to sync the cached shard info with actual shard info from Kinesis
func (w *Worker) syncShard() error {
log := w.kclConfig.Logger
shardInfo := make(map[string]bool)
err := w.getShardIDs("", shardInfo)
if err != nil {
return err
}
for _, shard := range w.shardStatus {
// The cached shard no longer existed, remove it.
if _, ok := shardInfo[shard.ID]; !ok {
// remove the shard from local status cache
delete(w.shardStatus, shard.ID)
// remove the shard entry in dynamoDB as well
// Note: syncShard runs periodically. we don't need to do anything in case of error here.
if err := w.checkpointer.RemoveLeaseInfo(shard.ID); err != nil {
log.Errorf("Failed to remove shard lease info: %s Error: %+v", shard.ID, err)
}
}
}
return nil
}
================================================
FILE: go.mod
================================================
module github.com/vmware/vmware-go-kcl
go 1.17
require (
github.com/aws/aws-sdk-go v1.41.7
github.com/awslabs/kinesis-aggregation/go v0.0.0-20210630091500-54e17340d32f
github.com/golang/protobuf v1.5.2
github.com/google/uuid v1.3.0
github.com/prometheus/client_golang v1.11.1
github.com/prometheus/common v0.32.1
github.com/rs/zerolog v1.25.0
github.com/sirupsen/logrus v1.8.1
github.com/stretchr/testify v1.7.0
go.uber.org/zap v1.19.1
gopkg.in/natefinch/lumberjack.v2 v2.0.0
)
require (
github.com/BurntSushi/toml v0.4.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.7.0 // indirect
golang.org/x/sys v0.1.0 // indirect
google.golang.org/protobuf v1.27.1 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
)
================================================
FILE: go.sum
================================================
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v0.4.1 h1:GaI7EiDXDRfa8VshkTj7Fym7ha+y8/XxIgD2okUIjLw=
github.com/BurntSushi/toml v0.4.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
github.com/aws/aws-sdk-go v1.19.48/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/aws/aws-sdk-go v1.41.7 h1:vlpR8Cky3ZxUVNINgeRZS6N0p6zmFvu/ZqRRwrTI25U=
github.com/aws/aws-sdk-go v1.41.7/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q=
github.com/awslabs/kinesis-aggregation/go v0.0.0-20210630091500-54e17340d32f h1:Pf0BjJDga7C98f0vhw+Ip5EaiE07S3lTKpIYPNS0nMo=
github.com/awslabs/kinesis-aggregation/go v0.0.0-20210630091500-54e17340d32f/go.mod h1:SghidfnxvX7ribW6nHI7T+IBbc9puZ9kk5Tx/88h8P4=
github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0m
gitextract_khtdggzz/
├── .gitignore
├── .gitreview
├── CONTRIBUTING.md
├── HyperMake
├── LICENSE
├── README.md
├── clientlibrary/
│ ├── checkpoint/
│ │ ├── checkpointer.go
│ │ ├── dynamodb-checkpointer.go
│ │ └── dynamodb-checkpointer_test.go
│ ├── config/
│ │ ├── config.go
│ │ ├── config_test.go
│ │ ├── initial-stream-pos.go
│ │ └── kcl-config.go
│ ├── interfaces/
│ │ ├── inputs.go
│ │ ├── record-processor-checkpointer.go
│ │ ├── record-processor.go
│ │ └── sequence-number.go
│ ├── metrics/
│ │ ├── cloudwatch/
│ │ │ └── cloudwatch.go
│ │ ├── interfaces.go
│ │ └── prometheus/
│ │ └── prometheus.go
│ ├── partition/
│ │ └── partition.go
│ ├── utils/
│ │ ├── awserr.go
│ │ ├── random.go
│ │ ├── random_test.go
│ │ └── uuid.go
│ └── worker/
│ ├── common-shard-consumer.go
│ ├── fan-out-shard-consumer.go
│ ├── polling-shard-consumer.go
│ ├── record-processor-checkpointer.go
│ ├── worker-fan-out.go
│ └── worker.go
├── go.mod
├── go.sum
├── logger/
│ ├── logger.go
│ ├── logger_test.go
│ ├── logrus.go
│ ├── zap/
│ │ ├── zap.go
│ │ └── zap_test.go
│ └── zerolog/
│ ├── zerolog.go
│ └── zerolog_test.go
├── support/
│ ├── scripts/
│ │ ├── check.sh
│ │ ├── ci.sh
│ │ ├── functions.sh
│ │ └── test.sh
│ └── toolchain/
│ ├── HyperMake
│ └── docker/
│ └── Dockerfile
└── test/
├── lease_stealing_util_test.go
├── logger_test.go
├── record_processor_test.go
├── record_publisher_test.go
├── worker_custom_test.go
├── worker_lease_stealing_test.go
└── worker_test.go
SYMBOL INDEX (355 symbols across 39 files)
FILE: clientlibrary/checkpoint/checkpointer.go
constant LeaseKeyKey (line 38) | LeaseKeyKey = "ShardID"
constant LeaseOwnerKey (line 39) | LeaseOwnerKey = "AssignedTo"
constant LeaseTimeoutKey (line 40) | LeaseTimeoutKey = "LeaseTimeout"
constant SequenceNumberKey (line 41) | SequenceNumberKey = "Checkpoint"
constant ParentShardIdKey (line 42) | ParentShardIdKey = "ParentShardId"
constant ClaimRequestKey (line 43) | ClaimRequestKey = "ClaimRequest"
constant ShardEnd (line 46) | ShardEnd = "SHARD_END"
constant ErrShardClaimed (line 49) | ErrShardClaimed = "Shard is already claimed by another node"
type ErrLeaseNotAcquired (line 52) | type ErrLeaseNotAcquired struct
method Error (line 56) | func (e ErrLeaseNotAcquired) Error() string {
type Checkpointer (line 61) | type Checkpointer interface
FILE: clientlibrary/checkpoint/dynamodb-checkpointer.go
constant ErrInvalidDynamoDBSchema (line 49) | ErrInvalidDynamoDBSchema = "The DynamoDB schema is invalid and may need ...
constant NumMaxRetries (line 52) | NumMaxRetries = 10
type DynamoCheckpoint (line 56) | type DynamoCheckpoint struct
method WithDynamoDB (line 84) | func (checkpointer *DynamoCheckpoint) WithDynamoDB(svc dynamodbiface.D...
method Init (line 90) | func (checkpointer *DynamoCheckpoint) Init() error {
method GetLease (line 122) | func (checkpointer *DynamoCheckpoint) GetLease(shard *par.ShardStatus,...
method CheckpointSequence (line 236) | func (checkpointer *DynamoCheckpoint) CheckpointSequence(shard *par.Sh...
method FetchCheckpoint (line 261) | func (checkpointer *DynamoCheckpoint) FetchCheckpoint(shard *par.Shard...
method RemoveLeaseInfo (line 291) | func (checkpointer *DynamoCheckpoint) RemoveLeaseInfo(shardID string) ...
method RemoveLeaseOwner (line 304) | func (checkpointer *DynamoCheckpoint) RemoveLeaseOwner(shardID string)...
method ListActiveWorkers (line 327) | func (checkpointer *DynamoCheckpoint) ListActiveWorkers(shardStatus ma...
method ClaimShard (line 354) | func (checkpointer *DynamoCheckpoint) ClaimShard(shard *par.ShardStatu...
method syncLeases (line 415) | func (checkpointer *DynamoCheckpoint) syncLeases(shardStatus map[strin...
method createTable (line 455) | func (checkpointer *DynamoCheckpoint) createTable() error {
method doesTableExist (line 479) | func (checkpointer *DynamoCheckpoint) doesTableExist() bool {
method saveItem (line 487) | func (checkpointer *DynamoCheckpoint) saveItem(item map[string]*dynamo...
method conditionalUpdate (line 494) | func (checkpointer *DynamoCheckpoint) conditionalUpdate(conditionExpre...
method putItem (line 503) | func (checkpointer *DynamoCheckpoint) putItem(input *dynamodb.PutItemI...
method getItem (line 508) | func (checkpointer *DynamoCheckpoint) getItem(shardID string) (map[str...
method removeItem (line 520) | func (checkpointer *DynamoCheckpoint) removeItem(shardID string) error {
function NewDynamoCheckpoint (line 69) | func NewDynamoCheckpoint(kclConfig *config.KinesisClientLibConfiguration...
FILE: clientlibrary/checkpoint/dynamodb-checkpointer_test.go
function TestDoesTableExist (line 46) | func TestDoesTableExist(t *testing.T) {
function TestGetLeaseNotAquired (line 63) | func TestGetLeaseNotAquired(t *testing.T) {
function TestGetLeaseAquired (line 94) | func TestGetLeaseAquired(t *testing.T) {
function TestGetLeaseShardClaimed (line 160) | func TestGetLeaseShardClaimed(t *testing.T) {
function TestGetLeaseClaimRequestExpiredOwner (line 200) | func TestGetLeaseClaimRequestExpiredOwner(t *testing.T) {
function TestGetLeaseClaimRequestExpiredClaimer (line 237) | func TestGetLeaseClaimRequestExpiredClaimer(t *testing.T) {
function TestFetchCheckpointWithStealing (line 274) | func TestFetchCheckpointWithStealing(t *testing.T) {
function TestGetLeaseConditional (line 312) | func TestGetLeaseConditional(t *testing.T) {
type mockDynamoDB (line 365) | type mockDynamoDB struct
method ScanPages (line 373) | func (m *mockDynamoDB) ScanPages(*dynamodb.ScanInput, func(*dynamodb.S...
method DescribeTable (line 377) | func (m *mockDynamoDB) DescribeTable(*dynamodb.DescribeTableInput) (*d...
method PutItem (line 384) | func (m *mockDynamoDB) PutItem(input *dynamodb.PutItemInput) (*dynamod...
method GetItem (line 420) | func (m *mockDynamoDB) GetItem(input *dynamodb.GetItemInput) (*dynamod...
method UpdateItem (line 426) | func (m *mockDynamoDB) UpdateItem(input *dynamodb.UpdateItemInput) (*d...
method CreateTable (line 436) | func (m *mockDynamoDB) CreateTable(input *dynamodb.CreateTableInput) (...
function TestListActiveWorkers (line 440) | func TestListActiveWorkers(t *testing.T) {
function TestListActiveWorkersErrShardNotAssigned (line 478) | func TestListActiveWorkersErrShardNotAssigned(t *testing.T) {
function TestClaimShard (line 499) | func TestClaimShard(t *testing.T) {
FILE: clientlibrary/config/config.go
constant LATEST (line 51) | LATEST InitialPositionInStream = iota + 1
constant TRIM_HORIZON (line 53) | TRIM_HORIZON
constant AT_TIMESTAMP (line 55) | AT_TIMESTAMP
constant DefaultInitialPositionInStream (line 59) | DefaultInitialPositionInStream = LATEST
constant DefaultFailoverTimeMillis (line 65) | DefaultFailoverTimeMillis = 10000
constant DefaultLeaseRefreshPeriodMillis (line 68) | DefaultLeaseRefreshPeriodMillis = 5000
constant DefaultMaxRecords (line 71) | DefaultMaxRecords = 10000
constant DefaultIdletimeBetweenReadsMillis (line 75) | DefaultIdletimeBetweenReadsMillis = 1000
constant DefaultDontCallProcessRecordsForEmptyRecordList (line 78) | DefaultDontCallProcessRecordsForEmptyRecordList = false
constant DefaultParentShardPollIntervalMillis (line 83) | DefaultParentShardPollIntervalMillis = 10000
constant DefaultShardSyncIntervalMillis (line 86) | DefaultShardSyncIntervalMillis = 60000
constant DefaultCleanupLeasesUponShardsCompletion (line 91) | DefaultCleanupLeasesUponShardsCompletion = true
constant DefaultTaskBackoffTimeMillis (line 94) | DefaultTaskBackoffTimeMillis = 500
constant DefaultValidateSequenceNumberBeforeCheckpointing (line 98) | DefaultValidateSequenceNumberBeforeCheckpointing = true
constant DefaultMaxLeasesForWorker (line 105) | DefaultMaxLeasesForWorker = math.MaxInt16
constant DefaultMaxLeasesToStealAtOneTime (line 110) | DefaultMaxLeasesToStealAtOneTime = 1
constant DefaultInitialLeaseTableReadCapacity (line 113) | DefaultInitialLeaseTableReadCapacity = 10
constant DefaultInitialLeaseTableWriteCapacity (line 116) | DefaultInitialLeaseTableWriteCapacity = 10
constant DefaultSkipShardSyncAtStartupIfLeasesExist (line 121) | DefaultSkipShardSyncAtStartupIfLeasesExist = false
constant DefaultShutdownGraceMillis (line 124) | DefaultShutdownGraceMillis = 5000
constant DefaultEnableLeaseStealing (line 127) | DefaultEnableLeaseStealing = false
constant DefaultLeaseStealingIntervalMillis (line 130) | DefaultLeaseStealingIntervalMillis = 5000
constant DefaultLeaseStealingClaimTimeoutMillis (line 133) | DefaultLeaseStealingClaimTimeoutMillis = 120000
constant DefaultLeaseSyncingIntervalMillis (line 136) | DefaultLeaseSyncingIntervalMillis = 60000
type InitialPositionInStream (line 142) | type InitialPositionInStream
type InitialPositionInStreamExtended (line 146) | type InitialPositionInStreamExtended struct
type KinesisClientLibConfiguration (line 161) | type KinesisClientLibConfiguration struct
function InitalPositionInStreamToShardIteratorType (line 293) | func InitalPositionInStreamToShardIteratorType(pos InitialPositionInStre...
function empty (line 297) | func empty(s string) bool {
function checkIsValueNotEmpty (line 302) | func checkIsValueNotEmpty(key string, value string) {
function checkIsValuePositive (line 310) | func checkIsValuePositive(key string, value int) {
FILE: clientlibrary/config/config_test.go
function TestConfig (line 29) | func TestConfig(t *testing.T) {
function TestConfigLeaseStealing (line 54) | func TestConfigLeaseStealing(t *testing.T) {
function TestConfigDefaultEnhancedFanOutConsumerName (line 76) | func TestConfigDefaultEnhancedFanOutConsumerName(t *testing.T) {
function TestEmptyEnhancedFanOutConsumerName (line 84) | func TestEmptyEnhancedFanOutConsumerName(t *testing.T) {
function TestConfigWithEnhancedFanOutConsumerARN (line 90) | func TestConfigWithEnhancedFanOutConsumerARN(t *testing.T) {
function TestEmptyEnhancedFanOutConsumerARN (line 98) | func TestEmptyEnhancedFanOutConsumerARN(t *testing.T) {
FILE: clientlibrary/config/initial-stream-pos.go
function newInitialPositionAtTimestamp (line 40) | func newInitialPositionAtTimestamp(timestamp *time.Time) *InitialPositio...
function newInitialPosition (line 44) | func newInitialPosition(position InitialPositionInStream) *InitialPositi...
FILE: clientlibrary/config/kcl-config.go
function NewKinesisClientLibConfig (line 48) | func NewKinesisClientLibConfig(applicationName, streamName, regionName, ...
function NewKinesisClientLibConfigWithCredential (line 54) | func NewKinesisClientLibConfigWithCredential(applicationName, streamName...
function NewKinesisClientLibConfigWithCredentials (line 60) | func NewKinesisClientLibConfigWithCredentials(applicationName, streamNam...
method WithKinesisEndpoint (line 107) | func (c *KinesisClientLibConfiguration) WithKinesisEndpoint(kinesisEndpo...
method WithDynamoDBEndpoint (line 113) | func (c *KinesisClientLibConfiguration) WithDynamoDBEndpoint(dynamoDBEnd...
method WithTableName (line 119) | func (c *KinesisClientLibConfiguration) WithTableName(tableName string) ...
method WithInitialPositionInStream (line 124) | func (c *KinesisClientLibConfiguration) WithInitialPositionInStream(init...
method WithTimestampAtInitialPositionInStream (line 130) | func (c *KinesisClientLibConfiguration) WithTimestampAtInitialPositionIn...
method WithFailoverTimeMillis (line 136) | func (c *KinesisClientLibConfiguration) WithFailoverTimeMillis(failoverT...
method WithLeaseRefreshPeriodMillis (line 142) | func (c *KinesisClientLibConfiguration) WithLeaseRefreshPeriodMillis(lea...
method WithShardSyncIntervalMillis (line 148) | func (c *KinesisClientLibConfiguration) WithShardSyncIntervalMillis(shar...
method WithMaxRecords (line 154) | func (c *KinesisClientLibConfiguration) WithMaxRecords(maxRecords int) *...
method WithMaxLeasesForWorker (line 162) | func (c *KinesisClientLibConfiguration) WithMaxLeasesForWorker(n int) *K...
method WithIdleTimeBetweenReadsInMillis (line 188) | func (c *KinesisClientLibConfiguration) WithIdleTimeBetweenReadsInMillis...
method WithCallProcessRecordsEvenForEmptyRecordList (line 194) | func (c *KinesisClientLibConfiguration) WithCallProcessRecordsEvenForEmp...
method WithTaskBackoffTimeMillis (line 199) | func (c *KinesisClientLibConfiguration) WithTaskBackoffTimeMillis(taskBa...
method WithLogger (line 205) | func (c *KinesisClientLibConfiguration) WithLogger(logger logger.Logger)...
method WithMonitoringService (line 214) | func (c *KinesisClientLibConfiguration) WithMonitoringService(mService m...
method WithEnhancedFanOutConsumer (line 224) | func (c *KinesisClientLibConfiguration) WithEnhancedFanOutConsumer(enabl...
method WithEnhancedFanOutConsumerName (line 232) | func (c *KinesisClientLibConfiguration) WithEnhancedFanOutConsumerName(c...
method WithEnhancedFanOutConsumerARN (line 242) | func (c *KinesisClientLibConfiguration) WithEnhancedFanOutConsumerARN(co...
method WithLeaseStealing (line 249) | func (c *KinesisClientLibConfiguration) WithLeaseStealing(enableLeaseSte...
method WithLeaseStealingIntervalMillis (line 254) | func (c *KinesisClientLibConfiguration) WithLeaseStealingIntervalMillis(...
method WithLeaseSyncingIntervalMillis (line 259) | func (c *KinesisClientLibConfiguration) WithLeaseSyncingIntervalMillis(l...
FILE: clientlibrary/interfaces/inputs.go
constant REQUESTED (line 50) | REQUESTED ShutdownReason = iota + 1
constant TERMINATE (line 58) | TERMINATE
constant ZOMBIE (line 65) | ZOMBIE
type ShutdownReason (line 78) | type ShutdownReason
type InitializationInput (line 80) | type InitializationInput struct
type ProcessRecordsInput (line 88) | type ProcessRecordsInput struct
type ShutdownInput (line 105) | type ShutdownInput struct
function ShutdownReasonMessage (line 120) | func ShutdownReasonMessage(reason ShutdownReason) *string {
FILE: clientlibrary/interfaces/record-processor-checkpointer.go
type IPreparedCheckpointer (line 37) | type IPreparedCheckpointer interface
type IRecordProcessorCheckpointer (line 64) | type IRecordProcessorCheckpointer interface
FILE: clientlibrary/interfaces/record-processor.go
type IRecordProcessor (line 40) | type IRecordProcessor interface
type IRecordProcessorFactory (line 78) | type IRecordProcessorFactory interface
FILE: clientlibrary/interfaces/sequence-number.go
type ExtendedSequenceNumber (line 41) | type ExtendedSequenceNumber struct
FILE: clientlibrary/metrics/cloudwatch/cloudwatch.go
constant DEFAULT_CLOUDWATCH_METRICS_BUFFER_DURATION (line 44) | DEFAULT_CLOUDWATCH_METRICS_BUFFER_DURATION = 10 * time.Second
type MonitoringService (line 46) | type MonitoringService struct
method Init (line 91) | func (cw *MonitoringService) Init(appName, streamName, workerID string...
method Start (line 114) | func (cw *MonitoringService) Start() error {
method Shutdown (line 121) | func (cw *MonitoringService) Shutdown() {
method eventloop (line 129) | func (cw *MonitoringService) eventloop() {
method flushShard (line 149) | func (cw *MonitoringService) flushShard(shard string, metric *cloudWat...
method flush (line 272) | func (cw *MonitoringService) flush() error {
method IncrRecordsProcessed (line 283) | func (cw *MonitoringService) IncrRecordsProcessed(shard string, count ...
method IncrBytesProcessed (line 290) | func (cw *MonitoringService) IncrBytesProcessed(shard string, count in...
method MillisBehindLatest (line 297) | func (cw *MonitoringService) MillisBehindLatest(shard string, millSeco...
method LeaseGained (line 304) | func (cw *MonitoringService) LeaseGained(shard string) {
method LeaseLost (line 311) | func (cw *MonitoringService) LeaseLost(shard string) {
method LeaseRenewed (line 318) | func (cw *MonitoringService) LeaseRenewed(shard string) {
method RecordGetRecordsTime (line 325) | func (cw *MonitoringService) RecordGetRecordsTime(shard string, time f...
method RecordProcessRecordsTime (line 331) | func (cw *MonitoringService) RecordProcessRecordsTime(shard string, ti...
method getOrCreatePerShardMetrics (line 338) | func (cw *MonitoringService) getOrCreatePerShardMetrics(shard string) ...
type cloudWatchMetrics (line 63) | type cloudWatchMetrics struct
function NewMonitoringService (line 76) | func NewMonitoringService(region string, creds *credentials.Credentials)...
function NewMonitoringServiceWithOptions (line 82) | func NewMonitoringServiceWithOptions(region string, creds *credentials.C...
function sumFloat64 (line 350) | func sumFloat64(slice []float64) *float64 {
function maxFloat64 (line 358) | func maxFloat64(slice []float64) *float64 {
function minFloat64 (line 371) | func minFloat64(slice []float64) *float64 {
FILE: clientlibrary/metrics/interfaces.go
type MonitoringService (line 30) | type MonitoringService interface
type NoopMonitoringService (line 45) | type NoopMonitoringService struct
method Init (line 47) | func (NoopMonitoringService) Init(appName, streamName, workerID string...
method Start (line 48) | func (NoopMonitoringService) Start() error ...
method Shutdown (line 49) | func (NoopMonitoringService) Shutdown() ...
method IncrRecordsProcessed (line 51) | func (NoopMonitoringService) IncrRecordsProcessed(shard string, count ...
method IncrBytesProcessed (line 52) | func (NoopMonitoringService) IncrBytesProcessed(shard string, count in...
method MillisBehindLatest (line 53) | func (NoopMonitoringService) MillisBehindLatest(shard string, millSeco...
method LeaseGained (line 54) | func (NoopMonitoringService) LeaseGained(shard string) ...
method LeaseLost (line 55) | func (NoopMonitoringService) LeaseLost(shard string) ...
method LeaseRenewed (line 56) | func (NoopMonitoringService) LeaseRenewed(shard string) ...
method RecordGetRecordsTime (line 57) | func (NoopMonitoringService) RecordGetRecordsTime(shard string, time f...
method RecordProcessRecordsTime (line 58) | func (NoopMonitoringService) RecordProcessRecordsTime(shard string, ti...
FILE: clientlibrary/metrics/prometheus/prometheus.go
type MonitoringService (line 41) | type MonitoringService struct
method Init (line 67) | func (p *MonitoringService) Init(appName, streamName, workerID string)...
method Start (line 120) | func (p *MonitoringService) Start() error {
method Shutdown (line 134) | func (p *MonitoringService) Shutdown() {}
method IncrRecordsProcessed (line 136) | func (p *MonitoringService) IncrRecordsProcessed(shard string, count i...
method IncrBytesProcessed (line 140) | func (p *MonitoringService) IncrBytesProcessed(shard string, count int...
method MillisBehindLatest (line 144) | func (p *MonitoringService) MillisBehindLatest(shard string, millSecon...
method LeaseGained (line 148) | func (p *MonitoringService) LeaseGained(shard string) {
method LeaseLost (line 152) | func (p *MonitoringService) LeaseLost(shard string) {
method LeaseRenewed (line 156) | func (p *MonitoringService) LeaseRenewed(shard string) {
method RecordGetRecordsTime (line 160) | func (p *MonitoringService) RecordGetRecordsTime(shard string, time fl...
method RecordProcessRecordsTime (line 164) | func (p *MonitoringService) RecordProcessRecordsTime(shard string, tim...
function NewMonitoringService (line 59) | func NewMonitoringService(listenAddress, region string, logger logger.Lo...
FILE: clientlibrary/partition/partition.go
type ShardStatus (line 37) | type ShardStatus struct
method GetLeaseOwner (line 51) | func (ss *ShardStatus) GetLeaseOwner() string {
method SetLeaseOwner (line 57) | func (ss *ShardStatus) SetLeaseOwner(owner string) {
method GetCheckpoint (line 63) | func (ss *ShardStatus) GetCheckpoint() string {
method SetCheckpoint (line 69) | func (ss *ShardStatus) SetCheckpoint(c string) {
method GetLeaseTimeout (line 75) | func (ss *ShardStatus) GetLeaseTimeout() time.Time {
method SetLeaseTimeout (line 81) | func (ss *ShardStatus) SetLeaseTimeout(timeout time.Time) {
method IsClaimRequestExpired (line 87) | func (ss *ShardStatus) IsClaimRequestExpired(kclConfig *config.Kinesis...
FILE: clientlibrary/utils/awserr.go
function AWSErrCode (line 25) | func AWSErrCode(err error) string {
FILE: clientlibrary/utils/random.go
constant letterBytes (line 29) | letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
constant letterIdxBits (line 31) | letterIdxBits = 6
constant letterIdxMask (line 32) | letterIdxMask = 1<<letterIdxBits - 1
constant letterIdxMax (line 33) | letterIdxMax = 63 / letterIdxBits
function RandStringBytesMaskImpr (line 36) | func RandStringBytesMaskImpr(n int) string {
FILE: clientlibrary/utils/random_test.go
function TestRandom (line 27) | func TestRandom(t *testing.T) {
function TestRandomNum (line 39) | func TestRandomNum(t *testing.T) {
FILE: clientlibrary/utils/uuid.go
function MustNewUUID (line 26) | func MustNewUUID() string {
FILE: clientlibrary/worker/common-shard-consumer.go
type shardConsumer (line 37) | type shardConsumer interface
type commonShardConsumer (line 42) | type commonShardConsumer struct
method releaseLease (line 52) | func (sc *commonShardConsumer) releaseLease() {
method getStartingPosition (line 69) | func (sc *commonShardConsumer) getStartingPosition() (*kinesis.Startin...
method waitOnParentShard (line 100) | func (sc *commonShardConsumer) waitOnParentShard() error {
method processRecords (line 124) | func (sc *commonShardConsumer) processRecords(getRecordsStartTime time...
FILE: clientlibrary/worker/fan-out-shard-consumer.go
type FanOutShardConsumer (line 35) | type FanOutShardConsumer struct
method getRecords (line 44) | func (sc *FanOutShardConsumer) getRecords() error {
method subscribeToShard (line 135) | func (sc *FanOutShardConsumer) subscribeToShard() (*kinesis.SubscribeT...
method resubscribe (line 148) | func (sc *FanOutShardConsumer) resubscribe(shardSub *kinesis.Subscribe...
FILE: clientlibrary/worker/polling-shard-consumer.go
type PollingShardConsumer (line 46) | type PollingShardConsumer struct
method getShardIterator (line 54) | func (sc *PollingShardConsumer) getShardIterator() (*string, error) {
method getRecords (line 75) | func (sc *PollingShardConsumer) getRecords() error {
FILE: clientlibrary/worker/record-processor-checkpointer.go
type PreparedCheckpointer (line 35) | type PreparedCheckpointer struct
method GetPendingCheckpoint (line 58) | func (pc *PreparedCheckpointer) GetPendingCheckpoint() *kcl.ExtendedSe...
method Checkpoint (line 62) | func (pc *PreparedCheckpointer) Checkpoint() error {
type RecordProcessorCheckpointer (line 45) | type RecordProcessorCheckpointer struct
method Checkpoint (line 66) | func (rc *RecordProcessorCheckpointer) Checkpoint(sequenceNumber *stri...
method PrepareCheckpoint (line 77) | func (rc *RecordProcessorCheckpointer) PrepareCheckpoint(sequenceNumbe...
function NewRecordProcessorCheckpoint (line 51) | func NewRecordProcessorCheckpoint(shard *par.ShardStatus, checkpoint chk...
FILE: clientlibrary/worker/worker-fan-out.go
method fetchConsumerARNWithRetry (line 32) | func (w *Worker) fetchConsumerARNWithRetry() (string, error) {
method fetchConsumerARN (line 50) | func (w *Worker) fetchConsumerARN() (string, error) {
FILE: clientlibrary/worker/worker.go
type Worker (line 54) | type Worker struct
method WithKinesis (line 97) | func (w *Worker) WithKinesis(svc kinesisiface.KinesisAPI) *Worker {
method WithCheckpointer (line 104) | func (w *Worker) WithCheckpointer(checker chk.Checkpointer) *Worker {
method Start (line 110) | func (w *Worker) Start() error {
method Shutdown (line 135) | func (w *Worker) Shutdown() {
method initialize (line 152) | func (w *Worker) initialize() error {
method newShardConsumer (line 221) | func (w *Worker) newShardConsumer(shard *par.ShardStatus) shardConsumer {
method eventLoop (line 250) | func (w *Worker) eventLoop() {
method rebalance (line 364) | func (w *Worker) rebalance() error {
method getShardIDs (line 451) | func (w *Worker) getShardIDs(nextToken string, shardInfo map[string]bo...
method syncShard (line 498) | func (w *Worker) syncShard() error {
function NewWorker (line 77) | func NewWorker(factory kcl.IRecordProcessorFactory, kclConfig *config.Ki...
FILE: logger/logger.go
type Fields (line 29) | type Fields
constant Debug (line 33) | Debug = "debug"
constant Info (line 35) | Info = "info"
constant Warn (line 37) | Warn = "warn"
constant Error (line 39) | Error = "error"
constant Fatal (line 41) | Fatal = "fatal"
type Logger (line 45) | type Logger interface
type Configuration (line 63) | type Configuration struct
function GetDefaultLogger (line 98) | func GetDefaultLogger() Logger {
function normalizeConfig (line 103) | func normalizeConfig(config *Configuration) {
FILE: logger/logger_test.go
function TestLogrusLoggerWithConfig (line 29) | func TestLogrusLoggerWithConfig(t *testing.T) {
function TestLogrusLogger (line 46) | func TestLogrusLogger(t *testing.T) {
FILE: logger/logrus.go
type LogrusLogEntry (line 32) | type LogrusLogEntry struct
method Debugf (line 124) | func (l *LogrusLogEntry) Debugf(format string, args ...interface{}) {
method Infof (line 128) | func (l *LogrusLogEntry) Infof(format string, args ...interface{}) {
method Warnf (line 132) | func (l *LogrusLogEntry) Warnf(format string, args ...interface{}) {
method Errorf (line 136) | func (l *LogrusLogEntry) Errorf(format string, args ...interface{}) {
method Fatalf (line 140) | func (l *LogrusLogEntry) Fatalf(format string, args ...interface{}) {
method Panicf (line 144) | func (l *LogrusLogEntry) Panicf(format string, args ...interface{}) {
method WithFields (line 148) | func (l *LogrusLogEntry) WithFields(fields Fields) Logger {
type LogrusLogger (line 36) | type LogrusLogger struct
method Debugf (line 94) | func (l *LogrusLogger) Debugf(format string, args ...interface{}) {
method Infof (line 98) | func (l *LogrusLogger) Infof(format string, args ...interface{}) {
method Warnf (line 102) | func (l *LogrusLogger) Warnf(format string, args ...interface{}) {
method Errorf (line 106) | func (l *LogrusLogger) Errorf(format string, args ...interface{}) {
method Fatalf (line 110) | func (l *LogrusLogger) Fatalf(format string, args ...interface{}) {
method Panicf (line 114) | func (l *LogrusLogger) Panicf(format string, args ...interface{}) {
method WithFields (line 118) | func (l *LogrusLogger) WithFields(fields Fields) Logger {
function NewLogrusLogger (line 42) | func NewLogrusLogger(lLogger logrus.FieldLogger) Logger {
function NewLogrusLoggerWithConfig (line 50) | func NewLogrusLoggerWithConfig(config Configuration) Logger {
function getFormatter (line 154) | func getFormatter(isJSON bool) logrus.Formatter {
function convertToLogrusFields (line 164) | func convertToLogrusFields(fields Fields) logrus.Fields {
FILE: logger/zap/zap.go
type ZapLogger (line 33) | type ZapLogger struct
method Debugf (line 94) | func (l *ZapLogger) Debugf(format string, args ...interface{}) {
method Infof (line 98) | func (l *ZapLogger) Infof(format string, args ...interface{}) {
method Warnf (line 102) | func (l *ZapLogger) Warnf(format string, args ...interface{}) {
method Errorf (line 106) | func (l *ZapLogger) Errorf(format string, args ...interface{}) {
method Fatalf (line 110) | func (l *ZapLogger) Fatalf(format string, args ...interface{}) {
method Panicf (line 114) | func (l *ZapLogger) Panicf(format string, args ...interface{}) {
method WithFields (line 118) | func (l *ZapLogger) WithFields(fields logger.Fields) logger.Logger {
function NewZapLogger (line 48) | func NewZapLogger(logger *uzap.SugaredLogger) logger.Logger {
function NewZapLoggerWithConfig (line 56) | func NewZapLoggerWithConfig(config logger.Configuration) logger.Logger {
function getEncoder (line 128) | func getEncoder(isJSON bool) zapcore.Encoder {
function getZapLevel (line 137) | func getZapLevel(level string) zapcore.Level {
FILE: logger/zap/zap_test.go
function TestZapLoggerWithConfig (line 12) | func TestZapLoggerWithConfig(t *testing.T) {
function TestZapLogger (line 30) | func TestZapLogger(t *testing.T) {
FILE: logger/zerolog/zerolog.go
type zeroLogger (line 32) | type zeroLogger struct
method Debugf (line 88) | func (z *zeroLogger) Debugf(format string, args ...interface{}) {
method Infof (line 92) | func (z *zeroLogger) Infof(format string, args ...interface{}) {
method Warnf (line 96) | func (z *zeroLogger) Warnf(format string, args ...interface{}) {
method Errorf (line 100) | func (z *zeroLogger) Errorf(format string, args ...interface{}) {
method Fatalf (line 104) | func (z *zeroLogger) Fatalf(format string, args ...interface{}) {
method Panicf (line 108) | func (z *zeroLogger) Panicf(format string, args ...interface{}) {
method WithFields (line 112) | func (z *zeroLogger) WithFields(keyValues logger.Fields) logger.Logger {
function NewZerologLogger (line 37) | func NewZerologLogger() logger.Logger {
function NewZerologLoggerWithConfig (line 54) | func NewZerologLoggerWithConfig(config logger.Configuration) logger.Logg...
function getZeroLogLevel (line 123) | func getZeroLogLevel(level string) zerolog.Level {
function normalizeConfig (line 140) | func normalizeConfig(config *logger.Configuration) {
FILE: logger/zerolog/zerolog_test.go
function TestZeroLogLoggerWithConfig (line 8) | func TestZeroLogLoggerWithConfig(t *testing.T) {
function TestZeroLogLogger (line 26) | func TestZeroLogLogger(t *testing.T) {
FILE: test/lease_stealing_util_test.go
type LeaseStealingTest (line 19) | type LeaseStealingTest struct
method WithBackoffSeconds (line 44) | func (lst *LeaseStealingTest) WithBackoffSeconds(backoff int) *LeaseSt...
method WithMaxRetries (line 49) | func (lst *LeaseStealingTest) WithMaxRetries(retries int) *LeaseSteali...
method publishSomeData (line 54) | func (lst *LeaseStealingTest) publishSomeData() (stop func()) {
method getShardCountByWorker (line 80) | func (lst *LeaseStealingTest) getShardCountByWorker() map[string]int {
method Run (line 115) | func (lst *LeaseStealingTest) Run(assertions LeaseStealingAssertions) {
function NewLeaseStealingTest (line 30) | func NewLeaseStealingTest(t *testing.T, config *TestClusterConfig, worke...
type LeaseStealingAssertions (line 110) | type LeaseStealingAssertions struct
type TestWorkerFactory (line 173) | type TestWorkerFactory interface
type TestClusterConfig (line 178) | type TestClusterConfig struct
type TestCluster (line 188) | type TestCluster struct
method addWorker (line 206) | func (tc *TestCluster) addWorker(workerID string, config *cfg.KinesisC...
method SpawnWorker (line 213) | func (tc *TestCluster) SpawnWorker() (string, *wk.Worker) {
method Shutdown (line 225) | func (tc *TestCluster) Shutdown() {
function NewTestCluster (line 196) | func NewTestCluster(t *testing.T, config *TestClusterConfig, workerFacto...
FILE: test/logger_test.go
function TestZapLoggerWithConfig (line 35) | func TestZapLoggerWithConfig(t *testing.T) {
function TestZapLogger (line 53) | func TestZapLogger(t *testing.T) {
function TestLogrusLoggerWithConfig (line 64) | func TestLogrusLoggerWithConfig(t *testing.T) {
function TestLogrusLogger (line 82) | func TestLogrusLogger(t *testing.T) {
function TestLogrusLoggerWithFieldsAtInit (line 91) | func TestLogrusLoggerWithFieldsAtInit(t *testing.T) {
FILE: test/record_processor_test.go
function recordProcessorFactory (line 30) | func recordProcessorFactory(t *testing.T) kc.IRecordProcessorFactory {
type dumpRecordProcessorFactory (line 35) | type dumpRecordProcessorFactory struct
method CreateProcessor (line 39) | func (d *dumpRecordProcessorFactory) CreateProcessor() kc.IRecordProce...
type dumpRecordProcessor (line 46) | type dumpRecordProcessor struct
method Initialize (line 51) | func (dd *dumpRecordProcessor) Initialize(input *kc.InitializationInpu...
method ProcessRecords (line 57) | func (dd *dumpRecordProcessor) ProcessRecords(input *kc.ProcessRecords...
method Shutdown (line 81) | func (dd *dumpRecordProcessor) Shutdown(input *kc.ShutdownInput) {
FILE: test/record_publisher_test.go
constant specstr (line 40) | specstr = `{"name":"kube-qQyhk","networking":{"containerNetworkCidr":"10...
function NewKinesisClient (line 43) | func NewKinesisClient(t *testing.T, regionName, endpoint string, credent...
function NewDynamoDBClient (line 58) | func NewDynamoDBClient(t *testing.T, regionName, endpoint string, creden...
function continuouslyPublishSomeData (line 72) | func continuouslyPublishSomeData(t *testing.T, kc kinesisiface.KinesisAP...
function publishToAllShards (line 115) | func publishToAllShards(t *testing.T, kc kinesisiface.KinesisAPI, shards...
function publishSomeData (line 125) | func publishSomeData(t *testing.T, kc kinesisiface.KinesisAPI) {
function publishRecord (line 149) | func publishRecord(t *testing.T, kc kinesisiface.KinesisAPI, hashKey *st...
function publishRecords (line 167) | func publishRecords(t *testing.T, kc kinesisiface.KinesisAPI) {
function publishAggregateRecord (line 190) | func publishAggregateRecord(t *testing.T, kc kinesisiface.KinesisAPI) {
function generateAggregateRecord (line 207) | func generateAggregateRecord(numRecords int, content string) []byte {
FILE: test/worker_custom_test.go
function TestWorkerInjectCheckpointer (line 39) | func TestWorkerInjectCheckpointer(t *testing.T) {
function TestWorkerInjectKinesis (line 90) | func TestWorkerInjectKinesis(t *testing.T) {
function TestWorkerInjectKinesisAndCheckpointer (line 131) | func TestWorkerInjectKinesisAndCheckpointer(t *testing.T) {
FILE: test/worker_lease_stealing_test.go
function TestLeaseStealing (line 12) | func TestLeaseStealing(t *testing.T) {
type leaseStealingWorkerFactory (line 28) | type leaseStealingWorkerFactory struct
method CreateKCLConfig (line 36) | func (wf *leaseStealingWorkerFactory) CreateKCLConfig(workerID string,...
method CreateWorker (line 58) | func (wf *leaseStealingWorkerFactory) CreateWorker(workerID string, kc...
function newLeaseStealingWorkerFactory (line 32) | func newLeaseStealingWorkerFactory(t *testing.T) *leaseStealingWorkerFac...
function TestLeaseStealingInjectCheckpointer (line 63) | func TestLeaseStealingInjectCheckpointer(t *testing.T) {
type leaseStealingWorkerFactoryCustom (line 79) | type leaseStealingWorkerFactoryCustom struct
method CreateWorker (line 89) | func (wfc *leaseStealingWorkerFactoryCustom) CreateWorker(workerID str...
function newleaseStealingWorkerFactoryCustomChk (line 83) | func newleaseStealingWorkerFactoryCustomChk(t *testing.T) *leaseStealing...
function TestLeaseStealingWithMaxLeasesForWorker (line 95) | func TestLeaseStealingWithMaxLeasesForWorker(t *testing.T) {
type leaseStealingWorkerFactoryMaxLeases (line 111) | type leaseStealingWorkerFactoryMaxLeases struct
method CreateKCLConfig (line 123) | func (wfm *leaseStealingWorkerFactoryMaxLeases) CreateKCLConfig(worker...
function newleaseStealingWorkerFactoryMaxLeases (line 116) | func newleaseStealingWorkerFactoryMaxLeases(t *testing.T, maxLeases int)...
FILE: test/worker_test.go
constant appName (line 45) | appName = "appName"
constant streamName (line 46) | streamName = "kcl-test"
constant regionName (line 47) | regionName = "us-west-2"
constant workerID (line 48) | workerID = "test-worker"
constant consumerName (line 49) | consumerName = "enhanced-fan-out-consumer"
constant metricsSystem (line 52) | metricsSystem = "cloudwatch"
function TestWorker (line 56) | func TestWorker(t *testing.T) {
function TestWorkerWithTimestamp (line 84) | func TestWorkerWithTimestamp(t *testing.T) {
function TestWorkerWithSigInt (line 106) | func TestWorkerWithSigInt(t *testing.T) {
function TestWorkerStatic (line 136) | func TestWorkerStatic(t *testing.T) {
function TestWorkerAssumeRole (line 153) | func TestWorkerAssumeRole(t *testing.T) {
function TestEnhancedFanOutConsumer (line 175) | func TestEnhancedFanOutConsumer(t *testing.T) {
function TestEnhancedFanOutConsumerDefaultConsumerName (line 204) | func TestEnhancedFanOutConsumerDefaultConsumerName(t *testing.T) {
function TestEnhancedFanOutConsumerARN (line 233) | func TestEnhancedFanOutConsumerARN(t *testing.T) {
function runTest (line 265) | func runTest(kclConfig *cfg.KinesisClientLibConfiguration, triggersig bo...
function getMetricsConfig (line 324) | func getMetricsConfig(kclConfig *cfg.KinesisClientLibConfiguration, serv...
Condensed preview — 53 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (318K chars).
[
{
"path": ".gitignore",
"chars": 139,
"preview": "/gen\n/vendor\n!/vendor/manifest\n/bin\n/pkg\n/tmp\n/log\n/vms\n/run\n/go\n.hmake\n.hmakerc\n.project\n.idea\n.vscode\n*_mock_test.go\nf"
},
{
"path": ".gitreview",
"chars": 103,
"preview": "[gerrit]\nhost=review.ec.eng.vmware.com\nport=29418\nproject=cascade-kinesis-client\ndefaultbranch=develop\n"
},
{
"path": "CONTRIBUTING.md",
"chars": 2448,
"preview": "# Contributing to vmware-go-kcl\n\nThe vmware-go-kcl project team welcomes contributions from the community. Before you st"
},
{
"path": "HyperMake",
"chars": 1676,
"preview": "---\nformat: hypermake.v0\n\nname: cascade-kinesis-client\ndescription: Kinesis Client in Go\n\ntargets:\n rebuild-toolchain:\n"
},
{
"path": "LICENSE",
"chars": 1088,
"preview": "MIT License\r\n\r\nCopyright (c) 2018 VMware, Inc.\r\n\r\nPermission is hereby granted, free of charge, to any person obtaining "
},
{
"path": "README.md",
"chars": 3279,
"preview": "# VMware-Go-KCL\n\n\n[![Go Report Card](https://gorepo"
},
{
"path": "clientlibrary/checkpoint/checkpointer.go",
"chars": 4265,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/checkpoint/dynamodb-checkpointer.go",
"chars": 18807,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/checkpoint/dynamodb-checkpointer_test.go",
"chars": 18179,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/config/config.go",
"chars": 13852,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/config/config_test.go",
"chars": 4323,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/config/initial-stream-pos.go",
"chars": 2160,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/config/kcl-config.go",
"chars": 13468,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/interfaces/inputs.go",
"chars": 5005,
"preview": "/*\n * Copyright (c) 2020 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/interfaces/record-processor-checkpointer.go",
"chars": 6441,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/interfaces/record-processor.go",
"chars": 4003,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/interfaces/sequence-number.go",
"chars": 2286,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/metrics/cloudwatch/cloudwatch.go",
"chars": 11714,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/metrics/interfaces.go",
"chars": 3708,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/metrics/prometheus/prometheus.go",
"chars": 7033,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/partition/partition.go",
"chars": 3715,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/utils/awserr.go",
"chars": 1303,
"preview": "/*\n * Copyright (c) 2021 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/utils/random.go",
"chars": 2092,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/utils/random_test.go",
"chars": 1737,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/utils/uuid.go",
"chars": 1335,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/worker/common-shard-consumer.go",
"chars": 6225,
"preview": "/*\n * Copyright (c) 2021 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/worker/fan-out-shard-consumer.go",
"chars": 6459,
"preview": "/*\n * Copyright (c) 2021 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/worker/polling-shard-consumer.go",
"chars": 7611,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/worker/record-processor-checkpointer.go",
"chars": 3182,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/worker/worker-fan-out.go",
"chars": 3833,
"preview": "/*\n * Copyright (c) 2021 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "clientlibrary/worker/worker.go",
"chars": 16884,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "go.mod",
"chars": 1196,
"preview": "module github.com/vmware/vmware-go-kcl\n\ngo 1.17\n\nrequire (\n\tgithub.com/aws/aws-sdk-go v1.41.7\n\tgithub.com/awslabs/kinesi"
},
{
"path": "go.sum",
"chars": 51681,
"preview": "cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ncloud.google.com/go v0.34.0/go.mod h1"
},
{
"path": "logger/logger.go",
"chars": 3859,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "logger/logger_test.go",
"chars": 1979,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "logger/logrus.go",
"chars": 4763,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "logger/zap/zap.go",
"chars": 4823,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "logger/zap/zap_test.go",
"chars": 970,
"preview": "package zap_test\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/vmware/vmware-go-kcl/logger\"\n\t"
},
{
"path": "logger/zerolog/zerolog.go",
"chars": 4613,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "logger/zerolog/zerolog_test.go",
"chars": 829,
"preview": "package zerolog\n\nimport (\n\t\"github.com/vmware/vmware-go-kcl/logger\"\n\t\"testing\"\n)\n\nfunc TestZeroLogLoggerWithConfig(t *te"
},
{
"path": "support/scripts/check.sh",
"chars": 2142,
"preview": "#!/usr/bin/env bash\n\n. support/scripts/functions.sh\n\ncheckfmt() {\n local files=\"$(gofmt -l $(local_go_pkgs))\"\n if "
},
{
"path": "support/scripts/ci.sh",
"chars": 242,
"preview": "#!/bin/bash\n\n# Run only the integration tests\n# go test -race ./test\necho \"Warning: Cannot find a good way to inject AWS"
},
{
"path": "support/scripts/functions.sh",
"chars": 1530,
"preview": "set -ex\n\n# PROJ_ROOT specifies the project root\nexport PROJ_ROOT=\"$HMAKE_PROJECT_DIR\"\n\n# Add /go in GOPATH because that'"
},
{
"path": "support/scripts/test.sh",
"chars": 134,
"preview": "#!/bin/bash\n. support/scripts/functions.sh\n\n# Run only the unit tests and not integration tests\ngo test -cover -race $(l"
},
{
"path": "support/toolchain/HyperMake",
"chars": 534,
"preview": "---\nformat: hypermake.v0\n\nname: go-kcl\ndescription: VMWare Go-KCL Amazon Kinesis Client Library in Go\n\ntargets:\n rebuil"
},
{
"path": "support/toolchain/docker/Dockerfile",
"chars": 418,
"preview": "FROM golang:1.17\nENV PATH /go/bin:/src/bin:/root/go/bin:/usr/local/go/bin:$PATH\nENV GOPATH /go:/src\nRUN go install githu"
},
{
"path": "test/lease_stealing_util_test.go",
"chars": 6157,
"preview": "package test\n\nimport (\n\t\"fmt\"\n\t\"sync\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/s"
},
{
"path": "test/logger_test.go",
"chars": 3409,
"preview": "/*\n * Copyright (c) 2019 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "test/record_processor_test.go",
"chars": 3692,
"preview": "/*\n * Copyright (c) 2020 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "test/record_publisher_test.go",
"chars": 7701,
"preview": "/*\n * Copyright (c) 2020 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "test/worker_custom_test.go",
"chars": 5536,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "test/worker_lease_stealing_test.go",
"chars": 4089,
"preview": "package test\n\nimport (\n\t\"testing\"\n\n\tchk \"github.com/vmware/vmware-go-kcl/clientlibrary/checkpoint\"\n\tcfg \"github.com/vmwa"
},
{
"path": "test/worker_test.go",
"chars": 10838,
"preview": "/*\n * Copyright (c) 2018 VMware, Inc.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy"
}
]
About this extraction
This page contains the full source code of the vmware/vmware-go-kcl GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 53 files (292.5 KB), approximately 90.3k tokens, and a symbol index with 355 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.