Showing preview only (1,156K chars total). Download the full file or copy to clipboard to get everything.
Repository: monzo/phosphor
Branch: master
Commit: ee1fad17320a
Files: 163
Total size: 1.1 MB
Directory structure:
gitextract_lw0vpfqr/
├── .gitignore
├── .travis.yml
├── Dockerfile
├── Godeps/
│ ├── Godeps.json
│ └── Readme
├── LICENCE
├── README.md
├── apps/
│ ├── phosphor/
│ │ └── main.go
│ └── phosphord/
│ └── main.go
├── dist.sh
├── internal/
│ ├── util/
│ │ └── stringarray.go
│ └── version/
│ └── version.go
├── phosphor/
│ ├── README.md
│ ├── context.go
│ ├── domain.go
│ ├── handler.go
│ ├── ingester.go
│ ├── marshaling.go
│ ├── memorystore.go
│ ├── options.go
│ ├── phosphor.go
│ └── store.go
├── phosphord/
│ ├── README.md
│ ├── forwarder.go
│ ├── options.go
│ ├── phosphord.go
│ ├── test/
│ │ └── test.go
│ └── transport/
│ ├── nsq.go
│ └── transport.go
├── proto/
│ ├── trace.pb.go
│ └── trace.proto
├── script/
│ └── buildprotobufs.sh
└── vendor/
├── code.google.com/
│ └── p/
│ └── snappy-go/
│ └── snappy/
│ ├── decode.go
│ ├── encode.go
│ ├── snappy.go
│ └── snappy_test.go
├── github.com/
│ ├── bitly/
│ │ └── go-nsq/
│ │ ├── .travis.yml
│ │ ├── ChangeLog.md
│ │ ├── LICENSE
│ │ ├── README.md
│ │ ├── UPGRADING.md
│ │ ├── api_request.go
│ │ ├── command.go
│ │ ├── command_test.go
│ │ ├── config.go
│ │ ├── config_test.go
│ │ ├── conn.go
│ │ ├── consumer.go
│ │ ├── consumer_test.go
│ │ ├── delegates.go
│ │ ├── errors.go
│ │ ├── message.go
│ │ ├── mock_test.go
│ │ ├── producer.go
│ │ ├── producer_test.go
│ │ ├── protocol.go
│ │ ├── states.go
│ │ ├── test/
│ │ │ ├── ca.pem
│ │ │ ├── server.key
│ │ │ └── server.pem
│ │ ├── test.sh
│ │ └── version.go
│ ├── cihub/
│ │ └── seelog/
│ │ ├── LICENSE.txt
│ │ ├── README.markdown
│ │ ├── behavior_adaptive_test.go
│ │ ├── behavior_adaptivelogger.go
│ │ ├── behavior_asynclogger.go
│ │ ├── behavior_asyncloop_test.go
│ │ ├── behavior_asynclooplogger.go
│ │ ├── behavior_asynctimer_test.go
│ │ ├── behavior_asynctimerlogger.go
│ │ ├── behavior_synclogger.go
│ │ ├── behavior_synclogger_test.go
│ │ ├── cfg_config.go
│ │ ├── cfg_errors.go
│ │ ├── cfg_logconfig.go
│ │ ├── cfg_logconfig_test.go
│ │ ├── cfg_parser.go
│ │ ├── cfg_parser_test.go
│ │ ├── common_closer.go
│ │ ├── common_constraints.go
│ │ ├── common_constraints_test.go
│ │ ├── common_context.go
│ │ ├── common_context_test.go
│ │ ├── common_exception.go
│ │ ├── common_exception_test.go
│ │ ├── common_flusher.go
│ │ ├── common_loglevel.go
│ │ ├── dispatch_custom.go
│ │ ├── dispatch_customdispatcher_test.go
│ │ ├── dispatch_dispatcher.go
│ │ ├── dispatch_filterdispatcher.go
│ │ ├── dispatch_filterdispatcher_test.go
│ │ ├── dispatch_splitdispatcher.go
│ │ ├── dispatch_splitdispatcher_test.go
│ │ ├── doc.go
│ │ ├── format.go
│ │ ├── format_test.go
│ │ ├── internals_baseerror.go
│ │ ├── internals_byteverifiers_test.go
│ │ ├── internals_fsutils.go
│ │ ├── internals_xmlnode.go
│ │ ├── internals_xmlnode_test.go
│ │ ├── log.go
│ │ ├── logger.go
│ │ ├── writers_bufferedwriter.go
│ │ ├── writers_bufferedwriter_test.go
│ │ ├── writers_connwriter.go
│ │ ├── writers_consolewriter.go
│ │ ├── writers_filewriter.go
│ │ ├── writers_filewriter_test.go
│ │ ├── writers_formattedwriter.go
│ │ ├── writers_formattedwriter_test.go
│ │ ├── writers_rollingfilewriter.go
│ │ ├── writers_rollingfilewriter_test.go
│ │ └── writers_smtpwriter.go
│ ├── golang/
│ │ └── protobuf/
│ │ └── proto/
│ │ ├── Makefile
│ │ ├── all_test.go
│ │ ├── clone.go
│ │ ├── clone_test.go
│ │ ├── decode.go
│ │ ├── encode.go
│ │ ├── equal.go
│ │ ├── equal_test.go
│ │ ├── extensions.go
│ │ ├── extensions_test.go
│ │ ├── lib.go
│ │ ├── message_set.go
│ │ ├── message_set_test.go
│ │ ├── pointer_reflect.go
│ │ ├── pointer_unsafe.go
│ │ ├── properties.go
│ │ ├── proto3_proto/
│ │ │ ├── proto3.pb.go
│ │ │ └── proto3.proto
│ │ ├── proto3_test.go
│ │ ├── size2_test.go
│ │ ├── size_test.go
│ │ ├── text.go
│ │ ├── text_parser.go
│ │ ├── text_parser_test.go
│ │ └── text_test.go
│ └── mreiferson/
│ ├── go-options/
│ │ ├── .travis.yml
│ │ ├── LICENSE
│ │ ├── README.md
│ │ ├── example_test.go
│ │ └── options.go
│ └── go-snappystream/
│ ├── .travis.yml
│ ├── LICENSE
│ ├── README.md
│ ├── fixturedata_test.go
│ ├── reader.go
│ ├── reader_test.go
│ ├── readwrite_test.go
│ ├── snappystream.go
│ ├── writer.go
│ └── writer_test.go
└── golang.org/
└── x/
└── net/
└── context/
├── context.go
├── context_test.go
├── ctxhttp/
│ ├── cancelreq.go
│ ├── cancelreq_go14.go
│ ├── ctxhttp.go
│ └── ctxhttp_test.go
└── withtimeout_test.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.DS_Store
dist/
================================================
FILE: .travis.yml
================================================
language: go
go:
- 1.5
- tip
env:
- GO15VENDOREXPERIMENT=1
install: true
script:
- "go install ./apps/phosphor"
- "go install ./apps/phosphord"
- "go test $(go list ./... | grep -v /vendor/)"
================================================
FILE: Dockerfile
================================================
FROM busybox
ADD dist/docker/bin/ /phosphor_bin/
RUN cd / && ln -s /phosphor_bin/* . \
&& cd /bin && ln -s /phosphor_bin/* .
EXPOSE 7750 7760 7760/udp
================================================
FILE: Godeps/Godeps.json
================================================
{
"ImportPath": "github.com/mondough/phosphor",
"GoVersion": "go1.5",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "code.google.com/p/snappy-go/snappy",
"Comment": "null-15",
"Rev": "12e4b4183793ac4b061921e7980845e750679fd0"
},
{
"ImportPath": "github.com/bitly/go-nsq",
"Comment": "v1.0.4-13-ga3aee1d",
"Rev": "a3aee1d8e104a99d8fedffe2c45832df6a96d735"
},
{
"ImportPath": "github.com/cihub/seelog",
"Comment": "go1.1-55-gc40fd0a",
"Rev": "c40fd0af694fa48ec870c030f495c26a5bffcf55"
},
{
"ImportPath": "github.com/golang/protobuf/proto",
"Rev": "16256d3ce6929458613798ee44b7914a3f59f5c6"
},
{
"ImportPath": "github.com/mreiferson/go-options",
"Rev": "7c174072188d0cfbe6f01bb457626abb22bdff52"
},
{
"ImportPath": "github.com/mreiferson/go-snappystream",
"Comment": "v0.2.2-8-ga5260a3",
"Rev": "a5260a307b3e7dd583283c1e2717445244d506c7"
},
{
"ImportPath": "golang.org/x/net/context",
"Rev": "47990a1ba55743e6ef1affd3a14e5bac8553615d"
}
]
}
================================================
FILE: Godeps/Readme
================================================
This directory tree is generated automatically by godep.
Please do not edit.
See https://github.com/tools/godep for more information.
================================================
FILE: LICENCE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2015 Matt Heath
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
Phosphor is a Distributed Tracing system, similar to [Google's Dapper](https://research.google.com/pubs/pub36356.html), [Twitter's Zipkin](https://twitter.github.io/zipkin), and [Hailo's Trace Service](https://speakerdeck.com/mattheath/scaling-microservices-in-go-high-load-strategy-2015?slide=45).
It is comprised of a few simple components:
- [Phosphor Client](https://github.com/mondough/phosphor-go), used to send traces from applications
- [Phosphor Daemon](https://github.com/mondough/phosphor/tree/master/phosphord), collects traces and forwards onto the main server
- [Phosphor Server](https://github.com/mondough/phosphor/tree/master/phosphor), stores traces and aggregated trace information
- Phosphor UI, view trace and debug information about your infrastructure
## Dependencies
- [NSQ](https://nsq.io) is used as the delivery transport between PhosphorD and the Phosphor Server
## Caveats
This system is currently in development, and some components are not yet open source. In particular, the persistence layer in this repository is an in-memory mock, and is therefore not appropriate for production usage. Additional storage adaptors will be added in the near future.
================================================
FILE: apps/phosphor/main.go
================================================
package main
import (
"flag"
"fmt"
"math/rand"
"os"
"os/signal"
"syscall"
"time"
"github.com/mondough/phosphor/internal/util"
"github.com/mondough/phosphor/internal/version"
"github.com/mondough/phosphor/phosphor"
"github.com/mreiferson/go-options"
)
func phosphorFlagSet() *flag.FlagSet {
flagSet := flag.NewFlagSet("phosphor", flag.ExitOnError)
// basic options
flagSet.Bool("version", false, "print version string")
flagSet.Bool("verbose", false, "enable verbose logging")
flagSet.Int64("worker-id", 0, "unique seed for message ID generation (int) in range [0,4096) (will default to a hash of hostname)")
flagSet.String("https-address", "", "<addr>:<port> to listen on for HTTPS clients")
flagSet.String("http-address", "0.0.0.0:7750", "<addr>:<port> to listen on for HTTP clients")
// NSQ Transport options
nsqLookupdHTTPAddrs := util.StringArray{}
flagSet.Var(&nsqLookupdHTTPAddrs, "nsqlookupd-http-address", "nsqlookupd HTTP address (may be given multiple times)")
nsqdHTTPAddrs := util.StringArray{}
flagSet.Var(&nsqdHTTPAddrs, "nsqd-http-address", "nsqd HTTP address (may be given multiple times)")
flagSet.String("nsq-topic", "phosphor", "NSQ topic name to recieve traces from")
flagSet.String("nsq-channel", "phosphor-server", "NSQ channel name to recieve traces from. This should be the same for all instances of the phosphor servers to spread ingestion work.")
flagSet.Int("nsq-max-inflight", 200, "Number of traces to allow NSQ to keep inflight")
flagSet.Int("nsq-num-handlers", 10, "Number of concurrent NSQ handlers to run")
return flagSet
}
func main() {
flagSet := phosphorFlagSet()
flagSet.Parse(os.Args[1:])
// Globally seed rand
rand.Seed(time.Now().UTC().UnixNano())
if flagSet.Lookup("version").Value.(flag.Getter).Get().(bool) {
fmt.Println(version.String("phosphor"))
return
}
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
opts := phosphor.NewOptions()
cfg := map[string]interface{}{}
options.Resolve(opts, flagSet, cfg)
p := phosphor.New(opts)
p.Run()
<-signalChan
p.Exit()
}
================================================
FILE: apps/phosphord/main.go
================================================
package main
import (
"flag"
"fmt"
"math/rand"
"os"
"os/signal"
"runtime"
"syscall"
"time"
log "github.com/cihub/seelog"
"github.com/mreiferson/go-options"
"github.com/mondough/phosphor/internal/util"
"github.com/mondough/phosphor/internal/version"
"github.com/mondough/phosphor/phosphord"
)
func phosphordFlagSet() *flag.FlagSet {
flagSet := flag.NewFlagSet("phosphord", flag.ExitOnError)
// basic options
flagSet.Bool("version", false, "print version string")
flagSet.Bool("verbose", false, "enable verbose logging")
// forwarder options
flagSet.String("udp-address", "0.0.0.0:7760", "<addr>:<port> to listen for UDP traces")
flagSet.Int("num-forwarders", 20, "set the number of workers which buffer and forward traces")
flagSet.Int("buffer-size", 200, "set the maximum number of traces buffered per worker before batch sending")
flagSet.Int("flush-interval", 2000, "set the maximum flush interval in ms")
// NSQ Transport options
flagSet.String("nsq-topic", "phosphor", "NSQ topic name to recieve traces from")
nsqdTCPAddrs := util.StringArray{}
flagSet.Var(&nsqdTCPAddrs, "nsqd-tcp-address", "nsqd TCP address (may be given multiple times)")
return flagSet
}
func main() {
flagSet := phosphordFlagSet()
flagSet.Parse(os.Args[1:])
defer log.Flush()
// Globally seed rand
rand.Seed(time.Now().UTC().UnixNano())
// Use ALL the CPUs
runtime.GOMAXPROCS(runtime.NumCPU())
// Immediately print and exit the version number
if flagSet.Lookup("version").Value.(flag.Getter).Get().(bool) {
fmt.Println(version.String("phosphord"))
return
}
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
opts := phosphord.NewOptions()
cfg := map[string]interface{}{}
options.Resolve(opts, flagSet, cfg)
p := phosphord.New(opts)
p.Run()
<-signalChan
p.Exit()
}
================================================
FILE: dist.sh
================================================
#!/bin/bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
rm -rf $DIR/dist/docker
mkdir -p $DIR/dist/docker
rm -rf $DIR/.godeps
mkdir -p $DIR/.godeps
export GOPATH=$DIR/vendor:$GOPATH
arch=$(go env GOARCH)
version=$(awk '/const Version/ {print $NF}' < $DIR/internal/version/version.go | sed 's/"//g')
goversion=$(go version | awk '{print $3}')
for os in linux darwin freebsd; do
echo "... building v$version for $os/$arch"
BUILD=$(mktemp -d -t phosphor)
TARGET="phosphor-$version.$os-$arch.$goversion"
for app in phosphor phosphord; do
GOOS=$os GOARCH=$arch CGO_ENABLED=0 go build -o $BUILD/$TARGET/bin/$app ./apps/$app
done
pushd $BUILD
if [ "$os" == "linux" ]; then
cp -r $TARGET/bin $DIR/dist/docker/
fi
tar czvf $TARGET.tar.gz $TARGET
mv $TARGET.tar.gz $DIR/dist
popd
rm -r $BUILD
done
docker build -t mondough/phosphor:v$version .
if [[ ! $version == *"-"* ]]; then
echo "Tagging mondough/phosphor:v$version as the latest release."
docker tag -f mondough/phosphor:v$version mondough/phosphor:latest
fi
================================================
FILE: internal/util/stringarray.go
================================================
// The StringArray type is borrowed from NSQ
// https://github.com/bitly/nsq/blob/master/util/string_array.go
package util
import (
"strings"
)
type StringArray []string
func (a *StringArray) Set(s string) error {
*a = append(*a, s)
return nil
}
func (a *StringArray) String() string {
return strings.Join(*a, ",")
}
================================================
FILE: internal/version/version.go
================================================
package version
import (
"fmt"
"runtime"
)
// Version of the binaries
const Version = "0.0.1"
// String returns our formatted version string
func String(app string) string {
return fmt.Sprintf("%s v%s (built w/%s)", app, Version, runtime.Version())
}
================================================
FILE: phosphor/README.md
================================================
# Phosphor
The Phosphor server receives traces from PhosphorD via NSQ and stores these for later retrieval via its API.
## Usage
```
-http-address string
<addr>:<port> to listen on for HTTP clients (default "0.0.0.0:7750")
-https-address string
<addr>:<port> to listen on for HTTPS clients
-nsq-channel string
NSQ channel name to recieve traces from. This should be the same for all instances of the phosphor servers to spread ingestion work. (default "phosphor-server")
-nsq-max-inflight int
Number of traces to allow NSQ to keep inflight (default 200)
-nsq-num-handlers int
Number of concurrent NSQ handlers to run (default 10)
-nsq-topic string
NSQ topic name to recieve traces from (default "phosphor")
-nsqd-http-address value
nsqd HTTP address (may be given multiple times)
-nsqlookupd-http-address value
nsqlookupd HTTP address (may be given multiple times)
-verbose
enable verbose logging
-version
print version string
-worker-id int
unique seed for message ID generation (int) in range [0,4096) (will default to a hash of hostname)
```
================================================
FILE: phosphor/context.go
================================================
package phosphor
import (
"fmt"
"golang.org/x/net/context"
)
func phosphorFromContext(ctx context.Context) (*Phosphor, error) {
if p, ok := ctx.Value("phosphor").(*Phosphor); ok {
return p, nil
}
return nil, fmt.Errorf("Couldn't retrieve Phosphor from Context")
}
================================================
FILE: phosphor/domain.go
================================================
package phosphor
import (
"errors"
"sync"
"time"
)
// NewTrace initialises and returns a new Trace
func NewTrace() *Trace {
return &Trace{
Annotation: make([]*Annotation, 0),
}
}
// Trace represents a full trace of a request
// comprised of a number of Annotations
type Trace struct {
sync.Mutex
Annotation []*Annotation `json:"annotations"`
}
// AppendAnnotation to a Trace
func (t *Trace) AppendAnnotation(a *Annotation) error {
if t == nil {
return errors.New("Trace is Nil")
}
t.Annotation = append(t.Annotation, a)
return nil
}
// AnnotationType represents an Enum of types of Anotations which Phosphor supports
type AnnotationType int32
const (
UnknownAnnotationType = AnnotationType(0) // No idea...
// Calls
Req = AnnotationType(1) // Client Request dispatch
Rsp = AnnotationType(2) // Client Response received
In = AnnotationType(3) // Server Request received
Out = AnnotationType(4) // Server Response dispatched
Timeout = AnnotationType(5) // Client timed out waiting
// Developer initiated annotations
// @todo
// Annotation = AnnotationType(6)
)
// An Annotation represents the smallest individually recorded component of a trace
// These can be assembled into spans, and entire traces of a request to our systems
type Annotation struct {
TraceId string // Global Trace Identifier
SpanId string // Identifier for this span, non unique - eg. RPC calls would have 4 annotation with this id
ParentSpanId string // Parent span - eg. nested RPC calls
Timestamp time.Time // Timestamp the event occured, can only be compared on the same machine
Duration time.Duration // Optional: duration of the event, eg. RPC call
Hostname string // Hostname this event originated from
Origin string // Fully qualified name of the message origin
Destination string // Optional: Fully qualified name of the message destination
AnnotationType AnnotationType // The type of Annotation
Async bool // If the request was fired asynchronously
Payload string // The payload, eg. RPC body, or Annotation
PayloadSize int32 // Bytes of payload
KeyValue map[string]string // Key value debug information
}
================================================
FILE: phosphor/handler.go
================================================
package phosphor
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"golang.org/x/net/context"
log "github.com/cihub/seelog"
"github.com/mondough/phosphor/internal/version"
)
// Index
// @todo return version information etc
func Index(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
fmt.Fprintf(w, fmt.Sprintf(`{
"name": "phosphor",
"version": "%s"
}`, version.Version))
}
// TraceLookup retrieves a trace from the persistence layer
func TraceLookup(ctx context.Context) func(http.ResponseWriter, *http.Request) {
p, err := phosphorFromContext(ctx)
if err != nil {
panic(err)
}
return func(w http.ResponseWriter, r *http.Request) {
traceId := r.URL.Query().Get("traceId")
if traceId == "" {
errorResponse(r, w, http.StatusBadRequest, errors.New("traceId param not provided"))
return
}
log.Debugf("Trace lookup - TraceId: %s", traceId)
t, err := p.Store.ReadTrace(traceId)
if err != nil {
log.Errorf("Trace lookup failed: %s", err)
errorResponse(r, w, http.StatusInternalServerError, fmt.Errorf("could not load trace: %s", err))
return
}
// If we don't find the trace return 404
if t == nil {
log.Debugf("Trace not found: %s", traceId)
errorResponse(r, w, http.StatusNotFound, errors.New("traceId not found"))
return
}
// Return trace
response(
r,
w,
map[string]interface{}{
"trace": prettyFormatTrace(t),
},
)
}
}
// response sends the response back to the client, marshaling to JSON
func response(r *http.Request, w http.ResponseWriter, resp interface{}) {
writeResponse(r, w, http.StatusOK, resp)
}
// errorResponse marshals an error to JSON and returns this to the client
func errorResponse(r *http.Request, w http.ResponseWriter, code int, err error) {
resp := map[string]interface{}{
"error": err.Error(),
}
writeResponse(r, w, code, resp)
}
// response marshals a response to json and returns to the client
func writeResponse(r *http.Request, w http.ResponseWriter, code int, resp interface{}) {
// Deal with CORS
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "DELETE, GET, HEAD, OPTIONS, POST, PUT")
w.Header().Set("Access-Control-Allow-Credentials", "true")
// Allow any headers
if wantedHeaders := r.Header.Get("Access-Control-Request-Headers"); wantedHeaders != "" {
w.Header().Set("Access-Control-Allow-Headers", wantedHeaders)
}
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
b, err := json.Marshal(resp)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
fmt.Fprintln(w, `{"error":"failed to marshal json"}`)
return
}
w.WriteHeader(code)
fmt.Fprintln(w, string(b))
}
================================================
FILE: phosphor/ingester.go
================================================
package phosphor
import (
"fmt"
"os"
nsq "github.com/bitly/go-nsq"
log "github.com/cihub/seelog"
"github.com/golang/protobuf/proto"
traceproto "github.com/mondough/phosphor/proto"
)
var (
topic = "trace"
channel = "phosphor-server"
maxInFlight = 200
concurrency = 10
)
// Run the trace ingester, ingesting traces into the provided store
func (p *Phosphor) RunIngester() {
cfg := nsq.NewConfig()
cfg.UserAgent = fmt.Sprintf("phosphor go-nsq/%s", nsq.VERSION)
cfg.MaxInFlight = p.opts.NSQMaxInflight
consumer, err := nsq.NewConsumer(p.opts.NSQTopicName, p.opts.NSQChannelName, cfg)
if err != nil {
log.Critical(err)
os.Exit(1)
}
consumer.AddConcurrentHandlers(&IngestionHandler{
store: p.Store,
}, p.opts.NSQNumHandlers)
if len(p.opts.NSQDHTTPAddresses) != 0 {
err = consumer.ConnectToNSQDs(p.opts.NSQDHTTPAddresses)
if err != nil {
log.Critical(err)
os.Exit(1)
}
} else {
err = consumer.ConnectToNSQLookupds(p.opts.NSQLookupdHTTPAddresses)
if err != nil {
log.Critical(err)
os.Exit(1)
}
}
// Block until exit
select {
case <-consumer.StopChan:
case <-p.exitChan:
}
}
// IngestionHandler exists to match the NSQ handler interface
type IngestionHandler struct {
store Store
}
// HandleMessage delivered by NSQ
func (ih *IngestionHandler) HandleMessage(message *nsq.Message) error {
p := &traceproto.Annotation{}
err := proto.Unmarshal(message.Body, p)
if err != nil {
// returning an error to NSQ will requeue this
// failure to unmarshal is permanent
return nil
}
a := ProtoToAnnotation(p)
log.Debugf("Received annotation: %+v", a)
// Write to our store
ih.store.StoreAnnotation(a)
return nil
}
================================================
FILE: phosphor/marshaling.go
================================================
package phosphor
import (
"sort"
"time"
"github.com/mondough/phosphor/proto"
)
func prettyFormatTrace(t *Trace) interface{} {
return map[string]interface{}{
"annotations": formatAnnotations(t.Annotation),
}
}
func formatAnnotations(ans []*Annotation) interface{} {
sort.Sort(ByTime(ans))
// Convert to proto
pa := AnnotationsToProto(ans)
// Format nicely as JSON
m := make([]interface{}, 0, len(pa))
for _, a := range pa {
m = append(m, formatAnnotation(a))
}
return m
}
func formatAnnotation(a *traceproto.Annotation) interface{} {
return map[string]interface{}{
"trace_id": a.TraceId,
"span_id": a.SpanId,
"parent_id": a.ParentId,
"type": a.Type.String(),
"async": a.Async,
"timestamp": a.Timestamp,
"duration": a.Duration,
"hostname": a.Hostname,
"origin": a.Origin,
"destination": a.Destination,
"payload": a.Payload,
"key_value": a.KeyValue,
}
}
type ByTime []*Annotation
func (s ByTime) Len() int {
return len(s)
}
func (s ByTime) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s ByTime) Less(i, j int) bool {
return s[i].Timestamp.Before(s[j].Timestamp)
}
// ---
// ProtoToAnnotation converts a proto annotation to our domain
func ProtoToAnnotation(p *traceproto.Annotation) *Annotation {
if p == nil {
return &Annotation{}
}
return &Annotation{
TraceId: p.TraceId,
SpanId: p.SpanId,
ParentSpanId: p.ParentId,
Timestamp: microsecondInt64ToTime(p.Timestamp),
Duration: microsecondInt64ToDuration(p.Duration),
Hostname: p.Hostname,
Origin: p.Origin,
Destination: p.Destination,
AnnotationType: protoToAnnotationType(p.Type),
Async: p.Async,
Payload: p.Payload,
PayloadSize: int32(len(p.Payload)),
KeyValue: protoToKeyValue(p.KeyValue),
}
}
// protoToAnnotationType converts a annotation type in our proto to our domain
func protoToAnnotationType(p traceproto.AnnotationType) AnnotationType {
// Ensure we are within bounds
at := int32(p)
if at > 6 || at < 1 {
at = 0
}
return AnnotationType(at)
}
// annotationTypeToProto converts a annotation type in our domain to proto format
func annotationTypeToProto(at AnnotationType) traceproto.AnnotationType {
// Ensure we are within bounds
p := int32(at)
if p > 6 || p < 1 {
p = 0
}
return traceproto.AnnotationType(p)
}
// microsecondInt64ToTime converts an integer number of microseconds
// since the epoch to a time
func microsecondInt64ToTime(i int64) time.Time {
µsec := i % 1e6
sec := (i - µsec) / 1e6
return time.Unix(sec, µsec*1e3)
}
// timeToMicrosecondInt64 converts a time to µseconds since epoch as int64
func timeToMicrosecondInt64(t time.Time) int64 {
sec := t.Unix() * 1e6
µsec := int64(t.Nanosecond() / 1e3)
return sec + µsec
}
// microsecondInt64ToDuration converts an integer number
// of microseconds to a duration
func microsecondInt64ToDuration(i int64) time.Duration {
return time.Duration(i) * time.Microsecond
}
// durationToMicrosecondInt64 returns a duration to the nearest µs
func durationToMicrosecondInt64(d time.Duration) int64 {
return d.Nanoseconds() / 1e3
}
// protoToKeyValue converts a repeated set of proto key values
// to a map of keys => values
func protoToKeyValue(p []*traceproto.KeyValue) map[string]string {
ret := make(map[string]string)
for _, kv := range p {
if p == nil {
continue
}
ret[kv.Key] = kv.Value
}
return ret
}
// keyValueToProto converts a map of keys => values to a repeated set
// of proto key values
func keyValueToProto(m map[string]string) []*traceproto.KeyValue {
ret := make([]*traceproto.KeyValue, 0, len(m))
for k, v := range m {
kv := &traceproto.KeyValue{
Key: k,
Value: v,
}
ret = append(ret, kv)
}
return ret
}
// AnnotationsToProto converts a slice of domain annotations to our proto format
func AnnotationsToProto(a []*Annotation) []*traceproto.Annotation {
ret := make([]*traceproto.Annotation, 0, len(a))
for _, v := range a {
ret = append(ret, AnnotationToProto(v))
}
return ret
}
// AnnotationToProto converts a domain annotation to our proto format
func AnnotationToProto(a *Annotation) *traceproto.Annotation {
if a == nil {
return &traceproto.Annotation{}
}
return &traceproto.Annotation{
TraceId: a.TraceId,
SpanId: a.SpanId,
ParentId: a.ParentSpanId,
Type: annotationTypeToProto(a.AnnotationType),
Async: a.Async,
Timestamp: timeToMicrosecondInt64(a.Timestamp),
Duration: durationToMicrosecondInt64(a.Duration),
Hostname: a.Hostname,
Origin: a.Origin,
Destination: a.Destination,
Payload: a.Payload,
KeyValue: keyValueToProto(a.KeyValue),
}
}
================================================
FILE: phosphor/memorystore.go
================================================
package phosphor
import (
"sync"
"time"
log "github.com/cihub/seelog"
)
type MemoryStore struct {
sync.RWMutex
traces map[string]*Trace
}
// NewMemoryStore initialises and returns a new MemoryStore
func NewMemoryStore() *MemoryStore {
s := &MemoryStore{
traces: make(map[string]*Trace),
}
// run stats worker
go s.statsLoop()
return s
}
// ReadTrace retrieves a full Trace, composed of Annotations from the store by ID
func (s *MemoryStore) ReadTrace(id string) (*Trace, error) {
if s == nil {
return nil, ErrStoreNotInitialised
}
s.RLock()
defer s.RUnlock()
return s.traces[id], nil
}
// StoreAnnotation into the store, if the trace doesn't not already exist
// this will be created for the global trace ID
func (s *MemoryStore) StoreAnnotation(a *Annotation) error {
s.Lock()
defer s.Unlock()
if s == nil {
return ErrStoreNotInitialised
}
if a == nil {
return ErrInvalidAnnotation
}
if a.TraceId == "" {
return ErrInvalidTraceId
}
// Load our current trace
t := s.traces[a.TraceId]
// Initialise a new trace if we don't have it already
if t == nil {
t = NewTrace()
}
// Add the new annotation to this
t.AppendAnnotation(a)
// Store it back
s.traces[a.TraceId] = t
return nil
}
// statsLoop loops and outputs stats every 5 seconds
func (s *MemoryStore) statsLoop() {
tick := time.NewTicker(5 * time.Second)
// @todo listen for shutdown, stop ticker and exit cleanly
for {
<-tick.C // block until tick
s.printStats()
}
}
// printStats about the status of the memorystore to stdout
func (s *MemoryStore) printStats() {
// Get some data while under the mutex
s.RLock()
count := len(s.traces)
s.RUnlock()
// Separate processing and logging outside of mutex
log.Infof("[MemoryStore] Traces stored: %v", count)
}
================================================
FILE: phosphor/options.go
================================================
package phosphor
import (
"crypto/md5"
"hash/crc32"
"io"
"log"
"os"
)
type Options struct {
// basic options
ID int64 `flag:"worker-id" cfg:"id"`
Verbose bool `flag:"verbose"`
HTTPAddress string `flag:"http-address"`
HTTPSAddress string `flag:"https-address"`
// NSQ Transport options
NSQLookupdHTTPAddresses []string `flag:"nsqlookupd-http-address"`
NSQDHTTPAddresses []string `flag:"nsqd-http-address"`
NSQTopicName string `flag:"nsq-topic"`
NSQChannelName string `flag:"nsq-channel"`
NSQMaxInflight int `flag:"nsq-max-inflight"`
NSQNumHandlers int `flag:"nsq-num-handlers"`
}
func NewOptions() *Options {
hostname, err := os.Hostname()
if err != nil {
log.Fatal(err)
}
h := md5.New()
io.WriteString(h, hostname)
defaultID := int64(crc32.ChecksumIEEE(h.Sum(nil)) % 1024)
return &Options{
ID: defaultID,
HTTPAddress: "0.0.0.0:7750",
NSQTopicName: "phosphor",
NSQChannelName: "phosphor-server",
NSQMaxInflight: 200,
NSQNumHandlers: 10,
}
}
================================================
FILE: phosphor/phosphor.go
================================================
package phosphor
import (
"net/http"
"golang.org/x/net/context"
log "github.com/cihub/seelog"
)
type Phosphor struct {
opts *Options
Store Store
exitChan chan struct{}
}
func New(opts *Options) *Phosphor {
return &Phosphor{
opts: opts,
// Store: opts.Store,
exitChan: make(chan struct{}),
}
}
func (p *Phosphor) Run() {
log.Infof("Phosphor starting up")
defer log.Flush()
// Store a reference to phosphor in our context which we can pass
// to other areas of the application, eg the HTTP api
ctx := context.Background()
ctx = context.WithValue(ctx, "phosphor", p)
// Initialise a persistent store
// if p.Store == nil {
p.Store = NewMemoryStore()
// }
// Initialise trace ingestion
go p.RunIngester()
// Set up API and serve requests
http.HandleFunc("/", Index)
http.HandleFunc("/trace", TraceLookup(ctx))
go http.ListenAndServe(p.opts.HTTPAddress, nil)
}
func (p *Phosphor) Exit() {
log.Infof("Phosphor exiting")
select {
case <-p.exitChan: // check if already closed
default:
close(p.exitChan)
}
}
================================================
FILE: phosphor/store.go
================================================
package phosphor
import "errors"
type Store interface {
ReadTrace(id string) (*Trace, error)
StoreAnnotation(a *Annotation) error
}
var (
ErrStoreNotInitialised = errors.New("Store is not initialised")
ErrInvalidAnnotation = errors.New("Annotation is invalid")
ErrInvalidTrace = errors.New("Trace is invalid")
ErrInvalidTraceId = errors.New("TraceId is invalid")
)
================================================
FILE: phosphord/README.md
================================================
# PhosphorD
PhosphorD is a local forwarder, like StatsD, which receives traces from the Phosphor client, and forwards to the [Phosphor server](https://github.com/mondough/phosphor/tree/master/phosphor).
Currently this receives Traces over UDP, which prevents clients blocking, but is reasonably reliable on a local machine. In the event this blocks, traces will be dropped and lost.
A future improvement would make this configurable to read from local files, mirroring the behaviour of Dapper Daemons as described in the [Google Dapper](https://research.google.com/pubs/pub36356.html) paper.
## Usage
```
-buffer-size int
set the maximum number of traces buffered per worker before batch sending (default 200)
-flush-interval int
set the maximum flush interval in ms (default 2000)
-nsq-topic string
NSQ topic name to recieve traces from (default "phosphor")
-nsqd-tcp-address value
nsqd TCP address (may be given multiple times)
-num-forwarders int
set the number of workers which buffer and forward traces (default 20)
-udp-address string
<addr>:<port> to listen for UDP traces (default "0.0.0.0:7760")
-verbose
enable verbose logging
-version
print version string
```
================================================
FILE: phosphord/forwarder.go
================================================
package phosphord
import (
"encoding/json"
"time"
log "github.com/cihub/seelog"
"github.com/golang/protobuf/proto"
pb "github.com/mondough/phosphor/proto"
)
func (p *PhosphorD) forward(id int) {
log.Debugf("[Forwarder %v] started", id)
var (
b []byte
i int
decoded *pb.Annotation
js []byte
buf = make([][]byte, 0, p.opts.BufferSize)
metricsTick = time.NewTicker(5 * time.Second)
timeoutTick = time.NewTicker(time.Duration(p.opts.FlushInterval) * time.Millisecond)
)
for {
select {
case <-p.exitChan:
return
case b = <-p.traceChan:
i++
// Log the frame if we're in verbose mode
if p.opts.Verbose {
decoded = &pb.Annotation{}
if err := proto.Unmarshal(b, decoded); err != nil {
log.Warnf("[Forwarder %v] Couldn't decode trace frame", id)
continue
}
js, _ = json.Marshal(decoded)
log.Tracef("[Forwarder %v] Received message: %s", id, string(js))
}
// Add message to our buffer
buf = append(buf, b)
// Forward on if we're at our buffer size
if len(buf) >= p.opts.BufferSize {
p.sendTraces(id, &buf)
}
case <-timeoutTick.C:
p.sendTraces(id, &buf)
case <-metricsTick.C:
log.Debugf("[Forwarder %v] Processed %v messages", id, i)
}
}
}
func (p *PhosphorD) sendTraces(id int, buf *[][]byte) error {
// Don't publish empty buffers
if buf == nil || len(*buf) == 0 {
return nil
}
// Attempt to publish
log.Debugf("[Forwarder %v] Sending %v traces", id, len(*buf))
if err := p.tr.MultiPublish(*buf); err != nil {
// we return an error here, but currently ignore it
// therefore the behaviour will be reattempting to republish the
// buffer when the next trace arrives to this forwarder
return err
}
// Empty the buffer on success
*buf = nil
return nil
}
================================================
FILE: phosphord/options.go
================================================
package phosphord
type Options struct {
// basic options
Verbose bool `flag:"verbose"`
UDPAddress string `flag:"udp-address"`
NumForwarders int `flag:"num-forwarders"`
BufferSize int `flag:"buffer-size"`
FlushInterval int `flag:"flush-interval"`
// NSQ Transport options
NSQDTCPAddresses []string `flag:"nsqd-tcp-address"`
NSQTopicName string `flag:"nsq-topic"`
NSQMaxInflight int
NSQNumHandlers int
}
func NewOptions() *Options {
return &Options{
UDPAddress: "0.0.0.0:7760",
NumForwarders: 20,
BufferSize: 200,
FlushInterval: 2000,
NSQTopicName: "phosphor",
NSQMaxInflight: 200,
NSQNumHandlers: 10,
}
}
================================================
FILE: phosphord/phosphord.go
================================================
package phosphord
import (
"bytes"
"net"
"os"
"runtime"
"time"
log "github.com/cihub/seelog"
"github.com/mondough/phosphor/phosphord/transport"
)
const (
UDP = "udp"
)
var (
packetSize = 65536 - 8 - 20 // 8-byte UDP header, 20-byte IP header
)
type PhosphorD struct {
opts *Options
tr transport.Transport
traceChan chan []byte
exitChan chan struct{}
}
func New(opts *Options) *PhosphorD {
// Initialise our transport
// TODO ensure this doesn't connect until we Run()
tr, err := transport.NewNSQTransport(opts.NSQTopicName, opts.NSQDTCPAddresses)
if err != nil {
log.Criticalf(err.Error())
os.Exit(1)
}
return &PhosphorD{
opts: opts,
tr: tr,
traceChan: make(chan []byte),
exitChan: make(chan struct{}),
}
}
func (p *PhosphorD) Run() {
log.Infof("PhosphorD started at %v using %v CPUs", time.Now(), runtime.NumCPU())
// Fire up a number of forwarders to process inbound messages
log.Infof("Starting %v forwarders with buffer size of %v", p.opts.NumForwarders, p.opts.BufferSize)
for i := 0; i < p.opts.NumForwarders; i++ {
go p.forward(i)
}
// Bind and listen to UDP traffic
go p.listen()
}
// Exit and shut down
func (p *PhosphorD) Exit() {
log.Infof("PhosphorD exiting")
select {
case <-p.exitChan: // check if already closed
default:
close(p.exitChan)
}
}
// listen on a UDP socket for trace frames
func (p *PhosphorD) listen() {
// Resolve bind address
address, err := net.ResolveUDPAddr(UDP, p.opts.UDPAddress)
if err != nil {
log.Errorf("Failed to resolve address: %s", err.Error())
return
}
// Take the resolved address and attempt to listen on the UDP socket
listener, err := net.ListenUDP(UDP, address)
if err != nil {
log.Errorf("ListenUDP error: %s", err.Error())
return
}
defer listener.Close()
// Listen loop
log.Infof("Listening on %s for UDP trace frames", address.String())
for {
message := make([]byte, packetSize)
n, _, err := listener.ReadFrom(message)
if err != nil {
continue
}
buf := bytes.NewBuffer(message[0:n])
// log.Infof("Packet received from %s: %s", remaddr, string(message[0:n]))
// Attempt to push into our channel to be processed by a worker
select {
// Successfully write inbound message to queue
case p.traceChan <- buf.Bytes():
// Stop listening and shut down
case <-p.exitChan:
return
// Drop message to prevent blocking
default:
}
}
}
================================================
FILE: phosphord/test/test.go
================================================
package main
import (
"fmt"
"net"
"time"
"github.com/golang/protobuf/proto"
pb "github.com/mondough/phosphor/proto"
)
const (
MAX_PACKET_SIZE = 1500 - 8 - 20 // 8-byte UDP header, 20-byte IP header
)
func main() {
// Make example trace frame
t := &pb.Annotation{
TraceId: "aasldjaskjdlsakjdkasjdklasjdlasjdkljdas",
SpanId: "8yf8sdg76sg897b98fbuys8b9s6rvs6ducghkfhi27tuw",
ParentId: "97as8d7s9a7a7dv32hrkqehfkuh23hq8d7h4g7iygs7ih",
Type: pb.AnnotationType_CLIENT_SEND,
Timestamp: time.Now().UnixNano() / 1e3,
Duration: 1231312,
Hostname: "somehostname",
Origin: "some.api",
Destination: "some.service",
Payload: `{"boop":123}`,
}
// Marshal to bytes
b, err := proto.Marshal(t)
if err != nil {
panic(err)
}
fmt.Printf("Encoded: %s\n", string(b))
fmt.Printf("Encoded bytes: %v\n", b)
// Send via UDP!
// Get a conn
c, err := net.DialTimeout("udp", "localhost:7760", time.Second)
if err != nil {
panic(err)
}
// Write into the connection
var i int
for j := 0; j < 20; j++ {
for i = 0; i < 500; i++ {
_, err := c.Write([]byte(b))
if err != nil {
panic(err)
}
}
time.Sleep(500 * time.Millisecond)
}
fmt.Println("Sent", i, "messages")
// fmt.Println("Sent %v bytes", n)
}
================================================
FILE: phosphord/transport/nsq.go
================================================
package transport
import (
"errors"
"math/rand"
nsq "github.com/bitly/go-nsq"
log "github.com/cihub/seelog"
"github.com/mondough/phosphor/internal/util"
)
var (
ErrPublishFailure = errors.New("Failed to publish to NSQD")
ErrNoConfiguredNodes = errors.New("No NSQD nodes are configured")
)
// NewNSQTransport initialises a Transport over NSQ
func NewNSQTransport(topic string, nsqdTCPAddrs util.StringArray) (Transport, error) {
// Currently using default config
cfg := nsq.NewConfig()
// Create a producer for each nsqd node provided
producers := make(map[string]*nsq.Producer)
producersIndex := make([]*nsq.Producer, 0, len(nsqdTCPAddrs))
for _, addr := range nsqdTCPAddrs {
producer, err := nsq.NewProducer(addr, cfg)
if err != nil {
log.Warnf("failed to create nsq.Producer - %s", err)
}
producers[addr] = producer
producersIndex = append(producersIndex, producers[addr])
}
return &NSQPublisher{
topic: topic,
producers: producers,
producersIndex: producersIndex,
}, nil
}
type NSQPublisher struct {
topic string
producers map[string]*nsq.Producer
producersIndex []*nsq.Producer
}
func (p *NSQPublisher) MultiPublish(body [][]byte) error {
if len(p.producers) == 0 {
return ErrNoConfiguredNodes
}
// Round robin, from a random starting position
i := rand.Intn(len(p.producers)) - 1
// Attempt up to our number of configured nodes
for attempt := 0; attempt < len(p.producers); attempt++ {
// Move to next host, or cycle back around
i++
if i >= len(p.producers) {
i = 0
}
// Attempt to publish
pd := p.producersIndex[i]
if err := pd.MultiPublish(p.topic, body); err == nil {
// success!
return nil
}
}
// We've run out of nodes, and not managed to publish
return ErrPublishFailure
}
================================================
FILE: phosphord/transport/transport.go
================================================
package transport
type Transport interface {
MultiPublish(body [][]byte) error
}
================================================
FILE: proto/trace.pb.go
================================================
// Code generated by protoc-gen-go.
// source: github.com/mondough/phosphor/proto/trace.proto
// DO NOT EDIT!
/*
Package traceproto is a generated protocol buffer package.
It is generated from these files:
github.com/mondough/phosphor/proto/trace.proto
It has these top-level messages:
Annotation
KeyValue
*/
package traceproto
import proto "github.com/golang/protobuf/proto"
import fmt "fmt"
import math "math"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
type AnnotationType int32
const (
AnnotationType_UNKNOWN AnnotationType = 0
AnnotationType_CLIENT_SEND AnnotationType = 1
AnnotationType_CLIENT_RECV AnnotationType = 2
AnnotationType_SERVER_RECV AnnotationType = 3
AnnotationType_SERVER_SEND AnnotationType = 4
AnnotationType_TIMEOUT AnnotationType = 5
AnnotationType_ANNOTATION AnnotationType = 6
)
var AnnotationType_name = map[int32]string{
0: "UNKNOWN",
1: "CLIENT_SEND",
2: "CLIENT_RECV",
3: "SERVER_RECV",
4: "SERVER_SEND",
5: "TIMEOUT",
6: "ANNOTATION",
}
var AnnotationType_value = map[string]int32{
"UNKNOWN": 0,
"CLIENT_SEND": 1,
"CLIENT_RECV": 2,
"SERVER_RECV": 3,
"SERVER_SEND": 4,
"TIMEOUT": 5,
"ANNOTATION": 6,
}
func (x AnnotationType) String() string {
return proto.EnumName(AnnotationType_name, int32(x))
}
type Annotation struct {
// The ID of the trace this annotation is a component of
TraceId string `protobuf:"bytes,1,opt,name=trace_id" json:"trace_id,omitempty"`
// The span this trace corresponds to, in the case this
// is representing a service (REQ/REP) call
SpanId string `protobuf:"bytes,2,opt,name=span_id" json:"span_id,omitempty"`
// The parent span this trace corresponds to, allowing us
// to correlate trace frames and reconstruct the request
ParentId string `protobuf:"bytes,3,opt,name=parent_id" json:"parent_id,omitempty"`
// The type of annotation we're capturing
Type AnnotationType `protobuf:"varint,4,opt,name=type,enum=traceproto.AnnotationType" json:"type,omitempty"`
// Flag to indicate this is an asynchronous span, which will not have a
// response - eg. just client send and server recv annotations
Async bool `protobuf:"varint,5,opt,name=async" json:"async,omitempty"`
// Time since the epoch in microseconds
Timestamp int64 `protobuf:"varint,6,opt,name=timestamp" json:"timestamp,omitempty"`
// Duration in microseconds
// This should only be used to measure time on the same node
// eg. the duration of service / rpc calls
Duration int64 `protobuf:"varint,7,opt,name=duration" json:"duration,omitempty"`
// Machine hostname, container name etc
Hostname string `protobuf:"bytes,8,opt,name=hostname" json:"hostname,omitempty"`
// Origin of this annotation, likely a service or application for a RPC
Origin string `protobuf:"bytes,9,opt,name=origin" json:"origin,omitempty"`
// Destination of this annotations action
// eg. the service which a request was destined for
// likely not set for annotations
Destination string `protobuf:"bytes,10,opt,name=destination" json:"destination,omitempty"`
// Payload as a string - eg. JSON encoded
Payload string `protobuf:"bytes,11,opt,name=payload" json:"payload,omitempty"`
// Repeated series of key value fields for arbitrary data
KeyValue []*KeyValue `protobuf:"bytes,12,rep,name=key_value" json:"key_value,omitempty"`
}
func (m *Annotation) Reset() { *m = Annotation{} }
func (m *Annotation) String() string { return proto.CompactTextString(m) }
func (*Annotation) ProtoMessage() {}
func (m *Annotation) GetKeyValue() []*KeyValue {
if m != nil {
return m.KeyValue
}
return nil
}
type KeyValue struct {
Key string `protobuf:"bytes,1,opt,name=key" json:"key,omitempty"`
Value string `protobuf:"bytes,2,opt,name=value" json:"value,omitempty"`
}
func (m *KeyValue) Reset() { *m = KeyValue{} }
func (m *KeyValue) String() string { return proto.CompactTextString(m) }
func (*KeyValue) ProtoMessage() {}
func init() {
proto.RegisterEnum("traceproto.AnnotationType", AnnotationType_name, AnnotationType_value)
}
================================================
FILE: proto/trace.proto
================================================
syntax="proto3";
package traceproto;
message Annotation {
// The ID of the trace this annotation is a component of
string trace_id = 1;
// The span this trace corresponds to, in the case this
// is representing a service (REQ/REP) call
string span_id = 2;
// The parent span this trace corresponds to, allowing us
// to correlate trace frames and reconstruct the request
string parent_id = 3;
// The type of annotation we're capturing
AnnotationType type = 4;
// Flag to indicate this is an asynchronous span, which will not have a
// response - eg. just client send and server recv annotations
bool async = 5;
// Time since the epoch in microseconds
int64 timestamp = 6;
// Duration in microseconds
// This should only be used to measure time on the same node
// eg. the duration of service / rpc calls
int64 duration = 7;
// Machine hostname, container name etc
string hostname = 8;
// Origin of this annotation, likely a service or application for a RPC
string origin = 9;
// Destination of this annotations action
// eg. the service which a request was destined for
// likely not set for annotations
string destination = 10;
// Payload as a string - eg. JSON encoded
string payload = 11;
// Repeated series of key value fields for arbitrary data
repeated KeyValue key_value = 12;
}
enum AnnotationType {
UNKNOWN = 0;
CLIENT_SEND = 1;
CLIENT_RECV = 2;
SERVER_RECV = 3;
SERVER_SEND = 4;
TIMEOUT = 5;
ANNOTATION = 6;
}
message KeyValue {
string key = 1;
string value = 2;
}
================================================
FILE: script/buildprotobufs.sh
================================================
#!/bin/bash
# Basic path locations
ROOT=$(cd $(dirname -- "$0" ) && cd .. && pwd)
MESSAGEPATH=${ROOT}/proto
# SRCPATH is the path to our src directory - everything from here is fully qualified
# This depends on your storing your code in your GOPATH
# eg. xxx/github.com/mondough/phosphor
SRCPATH=$(cd ${ROOT}/../../.. && pwd)
# Cakes are important. and delicious. and should be given out for success.
function dispatchCake() {
printf "\n \033[1;33m*\033[0m \033[1;33m*\033[0m \033[1;33m*\033[0m \n"
printf " \033[1;33m*\033[0m\033[0;31m|\033[0m_\033[1;33m*\033[0m\033[0;31m|\033[0m_\033[1;33m*\033[0m\033[0;31m|\033[0m_\033[1;33m*\033[0m \n"
printf " .-'\`\033[0;31m|\033[0m \033[0;31m|\033[0m \033[0;31m|\033[0m \033[0;31m|\033[0m\`'-. \n"
printf " |\`-............-'| \n"
printf " | | \n"
printf " \ _ .-. _ / \n"
printf " ,-|'-' '-' '-' '-'|-, \n"
printf " /\` \._ _./ \`\ \n"
printf " '._ \`\"\"\"\"\"\"\"\"\"\` _.'\n"
printf " \`''--..........--''\` \n\n"
printf " \033[1;5;7;32m GREAT SUCCESS! \033[0m\n\n"
printf "\n\n"
}
# Show which protobufs were found
printf "\nLocating protobufs...\n"
find ${MESSAGEPATH} -name '*.proto' -exec echo {} \;
echo ""
# Clean out current protos
find ${MESSAGEPATH} -name '*.pb.go' | xargs rm -f
# Try to rebuild all the things
echo "Generating Go protobuf classes..."
find $MESSAGEPATH -name '*.proto' -exec protoc -I${SRCPATH} --go_out=${SRCPATH} {} \;
printf "Complete\n\n"
# GREAT SUCCESS
dispatchCake
================================================
FILE: vendor/code.google.com/p/snappy-go/snappy/decode.go
================================================
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package snappy
import (
"encoding/binary"
"errors"
)
// ErrCorrupt reports that the input is invalid.
var ErrCorrupt = errors.New("snappy: corrupt input")
// DecodedLen returns the length of the decoded block.
func DecodedLen(src []byte) (int, error) {
v, _, err := decodedLen(src)
return v, err
}
// decodedLen returns the length of the decoded block and the number of bytes
// that the length header occupied.
func decodedLen(src []byte) (blockLen, headerLen int, err error) {
v, n := binary.Uvarint(src)
if n == 0 {
return 0, 0, ErrCorrupt
}
if uint64(int(v)) != v {
return 0, 0, errors.New("snappy: decoded block is too large")
}
return int(v), n, nil
}
// Decode returns the decoded form of src. The returned slice may be a sub-
// slice of dst if dst was large enough to hold the entire decoded block.
// Otherwise, a newly allocated slice will be returned.
// It is valid to pass a nil dst.
func Decode(dst, src []byte) ([]byte, error) {
dLen, s, err := decodedLen(src)
if err != nil {
return nil, err
}
if len(dst) < dLen {
dst = make([]byte, dLen)
}
var d, offset, length int
for s < len(src) {
switch src[s] & 0x03 {
case tagLiteral:
x := uint(src[s] >> 2)
switch {
case x < 60:
s += 1
case x == 60:
s += 2
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-1])
case x == 61:
s += 3
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-2]) | uint(src[s-1])<<8
case x == 62:
s += 4
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-3]) | uint(src[s-2])<<8 | uint(src[s-1])<<16
case x == 63:
s += 5
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-4]) | uint(src[s-3])<<8 | uint(src[s-2])<<16 | uint(src[s-1])<<24
}
length = int(x + 1)
if length <= 0 {
return nil, errors.New("snappy: unsupported literal length")
}
if length > len(dst)-d || length > len(src)-s {
return nil, ErrCorrupt
}
copy(dst[d:], src[s:s+length])
d += length
s += length
continue
case tagCopy1:
s += 2
if s > len(src) {
return nil, ErrCorrupt
}
length = 4 + int(src[s-2])>>2&0x7
offset = int(src[s-2])&0xe0<<3 | int(src[s-1])
case tagCopy2:
s += 3
if s > len(src) {
return nil, ErrCorrupt
}
length = 1 + int(src[s-3])>>2
offset = int(src[s-2]) | int(src[s-1])<<8
case tagCopy4:
return nil, errors.New("snappy: unsupported COPY_4 tag")
}
end := d + length
if offset > d || end > len(dst) {
return nil, ErrCorrupt
}
for ; d < end; d++ {
dst[d] = dst[d-offset]
}
}
if d != dLen {
return nil, ErrCorrupt
}
return dst[:d], nil
}
================================================
FILE: vendor/code.google.com/p/snappy-go/snappy/encode.go
================================================
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package snappy
import (
"encoding/binary"
)
// We limit how far copy back-references can go, the same as the C++ code.
const maxOffset = 1 << 15
// emitLiteral writes a literal chunk and returns the number of bytes written.
func emitLiteral(dst, lit []byte) int {
i, n := 0, uint(len(lit)-1)
switch {
case n < 60:
dst[0] = uint8(n)<<2 | tagLiteral
i = 1
case n < 1<<8:
dst[0] = 60<<2 | tagLiteral
dst[1] = uint8(n)
i = 2
case n < 1<<16:
dst[0] = 61<<2 | tagLiteral
dst[1] = uint8(n)
dst[2] = uint8(n >> 8)
i = 3
case n < 1<<24:
dst[0] = 62<<2 | tagLiteral
dst[1] = uint8(n)
dst[2] = uint8(n >> 8)
dst[3] = uint8(n >> 16)
i = 4
case int64(n) < 1<<32:
dst[0] = 63<<2 | tagLiteral
dst[1] = uint8(n)
dst[2] = uint8(n >> 8)
dst[3] = uint8(n >> 16)
dst[4] = uint8(n >> 24)
i = 5
default:
panic("snappy: source buffer is too long")
}
if copy(dst[i:], lit) != len(lit) {
panic("snappy: destination buffer is too short")
}
return i + len(lit)
}
// emitCopy writes a copy chunk and returns the number of bytes written.
func emitCopy(dst []byte, offset, length int) int {
i := 0
for length > 0 {
x := length - 4
if 0 <= x && x < 1<<3 && offset < 1<<11 {
dst[i+0] = uint8(offset>>8)&0x07<<5 | uint8(x)<<2 | tagCopy1
dst[i+1] = uint8(offset)
i += 2
break
}
x = length
if x > 1<<6 {
x = 1 << 6
}
dst[i+0] = uint8(x-1)<<2 | tagCopy2
dst[i+1] = uint8(offset)
dst[i+2] = uint8(offset >> 8)
i += 3
length -= x
}
return i
}
// Encode returns the encoded form of src. The returned slice may be a sub-
// slice of dst if dst was large enough to hold the entire encoded block.
// Otherwise, a newly allocated slice will be returned.
// It is valid to pass a nil dst.
func Encode(dst, src []byte) ([]byte, error) {
if n := MaxEncodedLen(len(src)); len(dst) < n {
dst = make([]byte, n)
}
// The block starts with the varint-encoded length of the decompressed bytes.
d := binary.PutUvarint(dst, uint64(len(src)))
// Return early if src is short.
if len(src) <= 4 {
if len(src) != 0 {
d += emitLiteral(dst[d:], src)
}
return dst[:d], nil
}
// Initialize the hash table. Its size ranges from 1<<8 to 1<<14 inclusive.
const maxTableSize = 1 << 14
shift, tableSize := uint(32-8), 1<<8
for tableSize < maxTableSize && tableSize < len(src) {
shift--
tableSize *= 2
}
var table [maxTableSize]int
// Iterate over the source bytes.
var (
s int // The iterator position.
t int // The last position with the same hash as s.
lit int // The start position of any pending literal bytes.
)
for s+3 < len(src) {
// Update the hash table.
b0, b1, b2, b3 := src[s], src[s+1], src[s+2], src[s+3]
h := uint32(b0) | uint32(b1)<<8 | uint32(b2)<<16 | uint32(b3)<<24
p := &table[(h*0x1e35a7bd)>>shift]
// We need to to store values in [-1, inf) in table. To save
// some initialization time, (re)use the table's zero value
// and shift the values against this zero: add 1 on writes,
// subtract 1 on reads.
t, *p = *p-1, s+1
// If t is invalid or src[s:s+4] differs from src[t:t+4], accumulate a literal byte.
if t < 0 || s-t >= maxOffset || b0 != src[t] || b1 != src[t+1] || b2 != src[t+2] || b3 != src[t+3] {
s++
continue
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
d += emitLiteral(dst[d:], src[lit:s])
}
// Extend the match to be as long as possible.
s0 := s
s, t = s+4, t+4
for s < len(src) && src[s] == src[t] {
s++
t++
}
// Emit the copied bytes.
d += emitCopy(dst[d:], s-t, s-s0)
lit = s
}
// Emit any final pending literal bytes and return.
if lit != len(src) {
d += emitLiteral(dst[d:], src[lit:])
}
return dst[:d], nil
}
// MaxEncodedLen returns the maximum length of a snappy block, given its
// uncompressed length.
func MaxEncodedLen(srcLen int) int {
// Compressed data can be defined as:
// compressed := item* literal*
// item := literal* copy
//
// The trailing literal sequence has a space blowup of at most 62/60
// since a literal of length 60 needs one tag byte + one extra byte
// for length information.
//
// Item blowup is trickier to measure. Suppose the "copy" op copies
// 4 bytes of data. Because of a special check in the encoding code,
// we produce a 4-byte copy only if the offset is < 65536. Therefore
// the copy op takes 3 bytes to encode, and this type of item leads
// to at most the 62/60 blowup for representing literals.
//
// Suppose the "copy" op copies 5 bytes of data. If the offset is big
// enough, it will take 5 bytes to encode the copy op. Therefore the
// worst case here is a one-byte literal followed by a five-byte copy.
// That is, 6 bytes of input turn into 7 bytes of "compressed" data.
//
// This last factor dominates the blowup, so the final estimate is:
return 32 + srcLen + srcLen/6
}
================================================
FILE: vendor/code.google.com/p/snappy-go/snappy/snappy.go
================================================
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package snappy implements the snappy block-based compression format.
// It aims for very high speeds and reasonable compression.
//
// The C++ snappy implementation is at http://code.google.com/p/snappy/
package snappy
/*
Each encoded block begins with the varint-encoded length of the decoded data,
followed by a sequence of chunks. Chunks begin and end on byte boundaries. The
first byte of each chunk is broken into its 2 least and 6 most significant bits
called l and m: l ranges in [0, 4) and m ranges in [0, 64). l is the chunk tag.
Zero means a literal tag. All other values mean a copy tag.
For literal tags:
- If m < 60, the next 1 + m bytes are literal bytes.
- Otherwise, let n be the little-endian unsigned integer denoted by the next
m - 59 bytes. The next 1 + n bytes after that are literal bytes.
For copy tags, length bytes are copied from offset bytes ago, in the style of
Lempel-Ziv compression algorithms. In particular:
- For l == 1, the offset ranges in [0, 1<<11) and the length in [4, 12).
The length is 4 + the low 3 bits of m. The high 3 bits of m form bits 8-10
of the offset. The next byte is bits 0-7 of the offset.
- For l == 2, the offset ranges in [0, 1<<16) and the length in [1, 65).
The length is 1 + m. The offset is the little-endian unsigned integer
denoted by the next 2 bytes.
- For l == 3, this tag is a legacy format that is no longer supported.
*/
const (
tagLiteral = 0x00
tagCopy1 = 0x01
tagCopy2 = 0x02
tagCopy4 = 0x03
)
================================================
FILE: vendor/code.google.com/p/snappy-go/snappy/snappy_test.go
================================================
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package snappy
import (
"bytes"
"flag"
"fmt"
"io"
"io/ioutil"
"math/rand"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
)
var download = flag.Bool("download", false, "If true, download any missing files before running benchmarks")
func roundtrip(b, ebuf, dbuf []byte) error {
e, err := Encode(ebuf, b)
if err != nil {
return fmt.Errorf("encoding error: %v", err)
}
d, err := Decode(dbuf, e)
if err != nil {
return fmt.Errorf("decoding error: %v", err)
}
if !bytes.Equal(b, d) {
return fmt.Errorf("roundtrip mismatch:\n\twant %v\n\tgot %v", b, d)
}
return nil
}
func TestEmpty(t *testing.T) {
if err := roundtrip(nil, nil, nil); err != nil {
t.Fatal(err)
}
}
func TestSmallCopy(t *testing.T) {
for _, ebuf := range [][]byte{nil, make([]byte, 20), make([]byte, 64)} {
for _, dbuf := range [][]byte{nil, make([]byte, 20), make([]byte, 64)} {
for i := 0; i < 32; i++ {
s := "aaaa" + strings.Repeat("b", i) + "aaaabbbb"
if err := roundtrip([]byte(s), ebuf, dbuf); err != nil {
t.Errorf("len(ebuf)=%d, len(dbuf)=%d, i=%d: %v", len(ebuf), len(dbuf), i, err)
}
}
}
}
}
func TestSmallRand(t *testing.T) {
rand.Seed(27354294)
for n := 1; n < 20000; n += 23 {
b := make([]byte, n)
for i, _ := range b {
b[i] = uint8(rand.Uint32())
}
if err := roundtrip(b, nil, nil); err != nil {
t.Fatal(err)
}
}
}
func TestSmallRegular(t *testing.T) {
for n := 1; n < 20000; n += 23 {
b := make([]byte, n)
for i, _ := range b {
b[i] = uint8(i%10 + 'a')
}
if err := roundtrip(b, nil, nil); err != nil {
t.Fatal(err)
}
}
}
func benchDecode(b *testing.B, src []byte) {
encoded, err := Encode(nil, src)
if err != nil {
b.Fatal(err)
}
// Bandwidth is in amount of uncompressed data.
b.SetBytes(int64(len(src)))
b.ResetTimer()
for i := 0; i < b.N; i++ {
Decode(src, encoded)
}
}
func benchEncode(b *testing.B, src []byte) {
// Bandwidth is in amount of uncompressed data.
b.SetBytes(int64(len(src)))
dst := make([]byte, MaxEncodedLen(len(src)))
b.ResetTimer()
for i := 0; i < b.N; i++ {
Encode(dst, src)
}
}
func readFile(b *testing.B, filename string) []byte {
src, err := ioutil.ReadFile(filename)
if err != nil {
b.Fatalf("failed reading %s: %s", filename, err)
}
if len(src) == 0 {
b.Fatalf("%s has zero length", filename)
}
return src
}
// expand returns a slice of length n containing repeated copies of src.
func expand(src []byte, n int) []byte {
dst := make([]byte, n)
for x := dst; len(x) > 0; {
i := copy(x, src)
x = x[i:]
}
return dst
}
func benchWords(b *testing.B, n int, decode bool) {
// Note: the file is OS-language dependent so the resulting values are not
// directly comparable for non-US-English OS installations.
data := expand(readFile(b, "/usr/share/dict/words"), n)
if decode {
benchDecode(b, data)
} else {
benchEncode(b, data)
}
}
func BenchmarkWordsDecode1e3(b *testing.B) { benchWords(b, 1e3, true) }
func BenchmarkWordsDecode1e4(b *testing.B) { benchWords(b, 1e4, true) }
func BenchmarkWordsDecode1e5(b *testing.B) { benchWords(b, 1e5, true) }
func BenchmarkWordsDecode1e6(b *testing.B) { benchWords(b, 1e6, true) }
func BenchmarkWordsEncode1e3(b *testing.B) { benchWords(b, 1e3, false) }
func BenchmarkWordsEncode1e4(b *testing.B) { benchWords(b, 1e4, false) }
func BenchmarkWordsEncode1e5(b *testing.B) { benchWords(b, 1e5, false) }
func BenchmarkWordsEncode1e6(b *testing.B) { benchWords(b, 1e6, false) }
// testFiles' values are copied directly from
// https://code.google.com/p/snappy/source/browse/trunk/snappy_unittest.cc.
// The label field is unused in snappy-go.
var testFiles = []struct {
label string
filename string
}{
{"html", "html"},
{"urls", "urls.10K"},
{"jpg", "house.jpg"},
{"pdf", "mapreduce-osdi-1.pdf"},
{"html4", "html_x_4"},
{"cp", "cp.html"},
{"c", "fields.c"},
{"lsp", "grammar.lsp"},
{"xls", "kennedy.xls"},
{"txt1", "alice29.txt"},
{"txt2", "asyoulik.txt"},
{"txt3", "lcet10.txt"},
{"txt4", "plrabn12.txt"},
{"bin", "ptt5"},
{"sum", "sum"},
{"man", "xargs.1"},
{"pb", "geo.protodata"},
{"gaviota", "kppkn.gtb"},
}
// The test data files are present at this canonical URL.
const baseURL = "https://snappy.googlecode.com/svn/trunk/testdata/"
func downloadTestdata(basename string) (errRet error) {
filename := filepath.Join("testdata", basename)
f, err := os.Create(filename)
if err != nil {
return fmt.Errorf("failed to create %s: %s", filename, err)
}
defer f.Close()
defer func() {
if errRet != nil {
os.Remove(filename)
}
}()
resp, err := http.Get(baseURL + basename)
if err != nil {
return fmt.Errorf("failed to download %s: %s", baseURL+basename, err)
}
defer resp.Body.Close()
_, err = io.Copy(f, resp.Body)
if err != nil {
return fmt.Errorf("failed to write %s: %s", filename, err)
}
return nil
}
func benchFile(b *testing.B, n int, decode bool) {
filename := filepath.Join("testdata", testFiles[n].filename)
if stat, err := os.Stat(filename); err != nil || stat.Size() == 0 {
if !*download {
b.Fatal("test data not found; skipping benchmark without the -download flag")
}
// Download the official snappy C++ implementation reference test data
// files for benchmarking.
if err := os.Mkdir("testdata", 0777); err != nil && !os.IsExist(err) {
b.Fatalf("failed to create testdata: %s", err)
}
for _, tf := range testFiles {
if err := downloadTestdata(tf.filename); err != nil {
b.Fatalf("failed to download testdata: %s", err)
}
}
}
data := readFile(b, filename)
if decode {
benchDecode(b, data)
} else {
benchEncode(b, data)
}
}
// Naming convention is kept similar to what snappy's C++ implementation uses.
func Benchmark_UFlat0(b *testing.B) { benchFile(b, 0, true) }
func Benchmark_UFlat1(b *testing.B) { benchFile(b, 1, true) }
func Benchmark_UFlat2(b *testing.B) { benchFile(b, 2, true) }
func Benchmark_UFlat3(b *testing.B) { benchFile(b, 3, true) }
func Benchmark_UFlat4(b *testing.B) { benchFile(b, 4, true) }
func Benchmark_UFlat5(b *testing.B) { benchFile(b, 5, true) }
func Benchmark_UFlat6(b *testing.B) { benchFile(b, 6, true) }
func Benchmark_UFlat7(b *testing.B) { benchFile(b, 7, true) }
func Benchmark_UFlat8(b *testing.B) { benchFile(b, 8, true) }
func Benchmark_UFlat9(b *testing.B) { benchFile(b, 9, true) }
func Benchmark_UFlat10(b *testing.B) { benchFile(b, 10, true) }
func Benchmark_UFlat11(b *testing.B) { benchFile(b, 11, true) }
func Benchmark_UFlat12(b *testing.B) { benchFile(b, 12, true) }
func Benchmark_UFlat13(b *testing.B) { benchFile(b, 13, true) }
func Benchmark_UFlat14(b *testing.B) { benchFile(b, 14, true) }
func Benchmark_UFlat15(b *testing.B) { benchFile(b, 15, true) }
func Benchmark_UFlat16(b *testing.B) { benchFile(b, 16, true) }
func Benchmark_UFlat17(b *testing.B) { benchFile(b, 17, true) }
func Benchmark_ZFlat0(b *testing.B) { benchFile(b, 0, false) }
func Benchmark_ZFlat1(b *testing.B) { benchFile(b, 1, false) }
func Benchmark_ZFlat2(b *testing.B) { benchFile(b, 2, false) }
func Benchmark_ZFlat3(b *testing.B) { benchFile(b, 3, false) }
func Benchmark_ZFlat4(b *testing.B) { benchFile(b, 4, false) }
func Benchmark_ZFlat5(b *testing.B) { benchFile(b, 5, false) }
func Benchmark_ZFlat6(b *testing.B) { benchFile(b, 6, false) }
func Benchmark_ZFlat7(b *testing.B) { benchFile(b, 7, false) }
func Benchmark_ZFlat8(b *testing.B) { benchFile(b, 8, false) }
func Benchmark_ZFlat9(b *testing.B) { benchFile(b, 9, false) }
func Benchmark_ZFlat10(b *testing.B) { benchFile(b, 10, false) }
func Benchmark_ZFlat11(b *testing.B) { benchFile(b, 11, false) }
func Benchmark_ZFlat12(b *testing.B) { benchFile(b, 12, false) }
func Benchmark_ZFlat13(b *testing.B) { benchFile(b, 13, false) }
func Benchmark_ZFlat14(b *testing.B) { benchFile(b, 14, false) }
func Benchmark_ZFlat15(b *testing.B) { benchFile(b, 15, false) }
func Benchmark_ZFlat16(b *testing.B) { benchFile(b, 16, false) }
func Benchmark_ZFlat17(b *testing.B) { benchFile(b, 17, false) }
================================================
FILE: vendor/github.com/bitly/go-nsq/.travis.yml
================================================
language: go
go:
- 1.4.2
env:
- NSQ_DOWNLOAD=nsq-0.2.30.linux-amd64.go1.3 GOARCH=amd64
- NSQ_DOWNLOAD=nsq-0.2.30.linux-amd64.go1.3 GOARCH=386
- NSQ_DOWNLOAD=nsq-0.2.31.linux-amd64.go1.3.1 GOARCH=amd64
- NSQ_DOWNLOAD=nsq-0.2.31.linux-amd64.go1.3.1 GOARCH=386
- NSQ_DOWNLOAD=nsq-0.3.0.linux-amd64.go1.3.3 GOARCH=amd64
- NSQ_DOWNLOAD=nsq-0.3.0.linux-amd64.go1.3.3 GOARCH=386
- NSQ_DOWNLOAD=nsq-0.3.1.linux-amd64.go1.4.1 GOARCH=amd64
- NSQ_DOWNLOAD=nsq-0.3.1.linux-amd64.go1.4.1 GOARCH=386
- NSQ_DOWNLOAD=nsq-0.3.2.linux-amd64.go1.4.1 GOARCH=amd64
- NSQ_DOWNLOAD=nsq-0.3.2.linux-amd64.go1.4.1 GOARCH=386
install:
- go get github.com/bitly/go-simplejson
- go get github.com/mreiferson/go-snappystream
script:
- wget http://bitly-downloads.s3.amazonaws.com/nsq/$NSQ_DOWNLOAD.tar.gz
- tar zxvf $NSQ_DOWNLOAD.tar.gz
- export PATH=$NSQ_DOWNLOAD/bin:$PATH
- pushd $TRAVIS_BUILD_DIR
- ./test.sh
- popd
notifications:
email: false
sudo: false
================================================
FILE: vendor/github.com/bitly/go-nsq/ChangeLog.md
================================================
## go-nsq Change Log
### 1.0.4 - 2015-04-07
**Upgrading from 1.0.3**: There are no backward incompatible changes.
* #133 - fix `ErrNotConnected` race during `Producer` connection (thanks @jeddenlea)
* #132 - fix `RDY` redistribution after backoff with no connections
* #128 - fix backoff stall when using `RequeueWithoutBackoff`
* #127 - fix handling of connection closing when resuming after backoff (thanks @jnewmano)
* #126 - allow `BackoffStrategy` to be set via flag (thanks @twmb)
* #125 - add pluggable consumer `BackoffStrategy`; add full-jitter strategy (thanks @hden)
* #124 - add `DialTimeout` and `LocalAddr` config (thanks @yashkin)
* #119 - add `Producer.Ping()` method (thanks @zulily)
* #122 - refactor log level string handling
* #120 - fix `Message` data races on `responded`
* #114 - fix lookupd jitter having no effect (thanks @judwhite)
### 1.0.3 - 2015-02-07
**Upgrading from 1.0.2**: There are no backward incompatible changes.
* #104 - fix reconnect address bug (thanks @ryanslade)
* #106 - fix backoff reconnect deadlock (thanks @ryanslade)
* #107 - fix out-of-bounds error when removing nsqlookupd addresses (thanks @andreas)
* #108 - fix potential logger race conditions (thanks @judwhite)
* #111 - fix resolved address error in reconnect loop (thanks @twmb)
### 1.0.2 - 2015-01-21
**Upgrading from 1.0.1**: There are no backward incompatible changes.
* #102 - TLS min/max config defaults (thanks @twmb)
* #99 - fix `Consumer.Stop()` race and `Producer.Stop()` deadlock (thanks @tylertreat)
* #92 - expose `Message.NSQDAddress`
* #95 - cleanup panic during `Consumer.Stop()` if handlers are deadlocked
* #98 - add `tls-min-version` option (thanks @twmb)
* #93 - expose a way to get `Consumer` runtime stats (thanks @dcarney)
* #94 - allow `#ephemeral` topic names (thanks @jamesgroat)
### 1.0.1 - 2014-11-09
**Upgrading from 1.0.0**: There are no backward incompatible changes functionally, however this
release no longer compiles with Go `1.0.x`.
* #89 - don't spam connection teardown cleanup messages
* #91 - add consumer `DisconnectFrom*`
* #87 - allow `heartbeat_interval` and `output_buffer_timeout` to be disabled
* #86 - pluggable `nsqlookupd` behaviors
* #83 - send `RDY` before `FIN`/`REQ` (forwards compatibility with bitly/nsq#404)
* #82 - fix panic when conn isn't assigned
* #75/#76 - minor config related bug fixes
* #75/#77/#78 - add `tls-cert` and `tls-key` config options
### 1.0.0 - 2014-08-11
**Upgrading from 0.3.7**: The public API was significantly refactored and is not backwards
compatible, please read [UPGRADING](UPGRADING.md).
* #58 - support `IDENTIFY` `msg_timeout`
* #54 - per-connection TLS config and set `ServerName`
* #49 - add common connect helpers
* #43/#63 - more flexible `nsqlookupd` URL specification
* #35 - `AUTH` support
* #41/#62 - use package private RNG
* #36 - support 64 character topic/channel names
* #30/#38/#39/#42/#45/#46/#48/#51/#52/#65/#70 - refactor public API (see [UPGRADING](UPGRADING.md))
### 0.3.7 - 2014-05-25
**Upgrading from 0.3.6**: There are no backward incompatible changes. **THIS IS THE LAST STABLE
RELEASE PROVIDING THIS API**. Future releases will be based on the api in #30 and **will not be
backwards compatible!**
This is a bug fix release relating to the refactoring done in `0.3.6`.
* #32 - fix potential panic for race condition when # conns == 0
* #33/#34 - more granular connection locking
### 0.3.6 - 2014-04-29
**Upgrading from 0.3.5**: There are no backward incompatible changes.
This release includes a significant internal refactoring, designed
to better encapsulate responsibility, see #19.
Specifically:
* make `Conn` public
* move transport responsibilities into `Conn` from `Reader`/`Writer`
* supply callbacks for hooking into `Conn` events
As part of the refactoring, a few additional clean exit related
issues were resolved:
* wait group now includes all exit related goroutines
* ensure that readLoop exits before exiting cleanup
* always check messagesInFlight at readLoop exit
* close underlying connection last
### 0.3.5 - 2014-04-05
**Upgrading from 0.3.4**: There are no backward incompatible changes.
This release includes a few new features such as support for channel
sampling and sending along a user agent string (which is now displayed
in `nsqadmin`).
Also, a critical bug fix for potential deadlocks (thanks @kjk
for reporting and help testing).
New Features/Improvements:
* #27 - reader logs disambiguate topic/channel
* #22 - channel sampling
* #23 - user agent
Bug Fixes:
* #24 - fix racey reader IDENTIFY buffering
* #29 - fix recursive RLock deadlocks
### 0.3.4 - 2013-11-19
**Upgrading from 0.3.3**: There are no backward incompatible changes.
This is a bug fix release, notably potential deadlocks in `Message.Requeue()` and `Message.Touch()`
as well as a potential busy loop cleaning up closed connections with in-flight messages.
New Features/Improvements:
* #14 - add `Reader.Configure()`
* #18 - return an exported error when an `nsqlookupd` address is already configured
Bug Fixes:
* #15 - dont let `handleError()` loop if already connected
* #17 - resolve potential deadlocks on `Message` responders
* #16 - eliminate busy loop when draining `finishedMessages`
### 0.3.3 - 2013-10-21
**Upgrading from 0.3.2**: This release requires NSQ binary version `0.2.23+` for compression
support.
This release contains significant `Reader` refactoring of the RDY handling code paths. The
motivation is documented in #1 however the commits in #8 identify individual changes. Additionally,
we eliminated deadlocks during connection cleanup in `Writer`.
As a result, both user-facing APIs should now be considerably more robust and stable. Additionally,
`Reader` should behave better when backing off.
New Features/Improvements:
* #9 - ability to ignore publish responses in `Writer`
* #12 - `Requeue()` method on `Message`
* #6 - `Touch()` method on `Message`
* #4 - snappy/deflate feature negotiation
Bug Fixes:
* #8 - `Reader` RDY handling refactoring (race conditions, deadlocks, consolidation)
* #13 - fix `Writer` deadlocks
* #10 - stop accessing simplejson internals
* #5 - fix `max-in-flight` race condition
### 0.3.2 - 2013-08-26
**Upgrading from 0.3.1**: This release requires NSQ binary version `0.2.22+` for TLS support.
New Features/Improvements:
* #227 - TLS feature negotiation
* #164/#202/#255 - add `Writer`
* #186 - `MaxBackoffDuration` of `0` disables backoff
* #175 - support for `nsqd` config option `--max-rdy-count`
* #169 - auto-reconnect to hard-coded `nsqd`
Bug Fixes:
* #254/#256/#257 - new connection RDY starvation
* #250 - `nsqlookupd` polling improvements
* #243 - limit `IsStarved()` to connections w/ in-flight messages
* #169 - use last RDY count for `IsStarved()`; redistribute RDY state
* #204 - fix early termination blocking
* #177 - support `broadcast_address`
* #161 - connection pool goroutine safety
### 0.3.1 - 2013-02-07
**Upgrading from 0.3.0**: This release requires NSQ binary version `0.2.17+` for `TOUCH` support.
* #119 - add TOUCH command
* #133 - improved handling of errors/magic
* #127 - send IDENTIFY (missed in #90)
* #16 - add backoff to Reader
### 0.3.0 - 2013-01-07
**Upgrading from 0.2.4**: There are no backward incompatible changes to applications
written against the public `nsq.Reader` API.
However, there *are* a few backward incompatible changes to the API for applications that
directly use other public methods, or properties of a few NSQ data types:
`nsq.Message` IDs are now a type `nsq.MessageID` (a `[16]byte` array). The signatures of
`nsq.Finish()` and `nsq.Requeue()` reflect this change.
`nsq.SendCommand()` and `nsq.Frame()` were removed in favor of `nsq.SendFramedResponse()`.
`nsq.Subscribe()` no longer accepts `shortId` and `longId`. If upgrading your consumers
before upgrading your `nsqd` binaries to `0.2.16-rc.1` they will not be able to send the
optional custom identifiers.
* #90 performance optimizations
* #81 reader performance improvements / MPUB support
### 0.2.4 - 2012-10-15
* #69 added IsStarved() to reader API
### 0.2.3 - 2012-10-11
* #64 timeouts on reader queries to lookupd
* #54 fix crash issue with reader cleaning up from unexpectedly closed nsqd connections
### 0.2.2 - 2012-10-09
* Initial public release
================================================
FILE: vendor/github.com/bitly/go-nsq/LICENSE
================================================
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
================================================
FILE: vendor/github.com/bitly/go-nsq/README.md
================================================
## go-nsq
The official Go package for [NSQ][nsq].
[][travis] [](https://godoc.org/github.com/bitly/go-nsq)
The latest stable release is **[1.0.4][latest_tag]**.
NOTE: The public API has been refactored as of `v1.0.0` and is not backwards compatible with
previous releases. **[0.3.7][legacy]** is the last stable release compatible with the legacy API.
Please read the [UPGRADING](UPGRADING.md) guide.
### Docs
See [godoc][nsq_gopkgdoc].
See the [main repo apps][apps] directory for examples of clients built using this package.
[nsq]: https://github.com/bitly/nsq
[nsq_gopkgdoc]: http://godoc.org/github.com/bitly/go-nsq
[protocol]: http://bitly.github.io/nsq/clients/tcp_protocol_spec.html
[apps]: https://github.com/bitly/nsq/tree/master/apps
[consumer]: http://godoc.org/github.com/bitly/go-nsq#Consumer
[producer]: http://godoc.org/github.com/bitly/go-nsq#Producer
[pr30]: https://github.com/bitly/go-nsq/pull/30
[legacy]: https://github.com/bitly/go-nsq/releases/tag/v0.3.7
[travis]: http://travis-ci.org/bitly/go-nsq
[latest_tag]: https://github.com/bitly/go-nsq/releases/tag/v1.0.4
================================================
FILE: vendor/github.com/bitly/go-nsq/UPGRADING.md
================================================
This outlines the backwards incompatible changes that were made to the public API after the
`v0.3.7` stable release, and and how to migrate existing legacy codebases.
#### Background
The original `go-nsq` codebase is some of our earliest Go code, and one of our first attempts at a
public Go library.
We've learned a lot over the last 2 years and we wanted `go-nsq` to reflect the experiences we've
had working with the library as well as the general Go conventions and best practices we picked up
along the way.
The diff can be seen via: https://github.com/bitly/go-nsq/compare/v0.3.7...HEAD
The bulk of the refactoring came via: https://github.com/bitly/go-nsq/pull/30
#### Naming
Previously, the high-level types we exposed were named `nsq.Reader` and `nsq.Writer`. These
reflected internal naming conventions we had used at bitly for some time but conflated semantics
with what a typical Go developer would expect (they obviously did not implement `io.Reader` and
`io.Writer`).
We renamed these types to `nsq.Consumer` and `nsq.Producer`, which more effectively communicate
their purpose and is consistent with the NSQ documentation.
#### Configuration
In the previous API there were inconsistent and confusing ways to configure your clients.
Now, configuration is performed *before* creating an `nsq.Consumer` or `nsq.Producer` by creating
an `nsq.Config` struct. The only valid way to do this is via `nsq.NewConfig` (i.e. using a struct
literal will panic due to invalid internal state).
The `nsq.Config` struct has exported variables that can be set directly in a type-safe manner. You
can also call `cfg.Validate()` to check that the values are correct and within range.
`nsq.Config` also exposes a convenient helper method `Set(k string, v interface{})` that can set
options by *coercing* the supplied `interface{}` value.
This is incredibly convenient if you're reading options from a config file or in a serialized
format that does not exactly match the native types.
It is both flexible and forgiving.
#### Improving the nsq.Handler interface
`go-nsq` attempts to make writing the common use case consumer incredibly easy.
You specify a type that implements the `nsq.Handler` interface, the interface method is called per
message, and the return value of said method indicates to the library what the response to `nsqd`
should be (`FIN` or `REQ`), all the while managing flow control and backoff.
However, more advanced use cases require the ability to respond to a message *later*
("asynchronously", if you will). Our original API provided a *second* message handler interface
called `nsq.AsyncHandler`.
Unfortunately, it was never obvious from the name alone (or even the documentation) how to properly
use this form. The API was needlessly complex, involving the garbage creation of wrapping structs
to track state and respond to messages.
We originally had the same problem in `pynsq`, our Python client library, and we were able to
resolve the tension and expose an API that was robust and supported all use cases.
The new `go-nsq` message handler interface exposes only `nsq.Handler`, and its `HandleMessage`
method remains identical (specifically, `nsq.AsyncHandler` has been removed).
Additionally, the API to configure handlers has been improved to provide better first-class support
for common operations. We've added `AddConcurrentHandlers` (for quickly spawning multiple handler
goroutines).
For the most common use case, where you want `go-nsq` to respond to messages on your behalf, there
are no changes required! In fact, we've made it even easier to implement the `nsq.Handler`
interface for simple functions by providing the `nsq.HandlerFunc` type (in the spirit of the Go
standard library's `http.HandlerFunc`):
```go
r, err := nsq.NewConsumer("test_topic", "test_channel", nsq.NewConfig())
if err != nil {
log.Fatalf(err.Error())
}
r.AddHandler(nsq.HandlerFunc(func(m *nsq.Message) error {
return doSomeWork(m)
})
err := r.ConnectToNSQD(nsqdAddr)
if err != nil {
log.Fatalf(err.Error())
}
<-r.StopChan
```
In the new API, we've made the `nsq.Message` struct more robust, giving it the ability to proxy
responses. If you want to usurp control of the message from `go-nsq`, you simply call
`msg.DisableAutoResponse()`.
This is effectively the same as if you had used `nsq.AsyncHandler`, only you don't need to manage
`nsq.FinishedMessage` structs or implement a separate interface. Instead you just keep/pass
references to the `nsq.Message` itself, and when you're ready to respond you call `msg.Finish()`,
`msg.Requeue(<duration>)` or `msg.Touch(<duration>)`. Additionally, this means you can make this
decision on a *per-message* basis rather than for the lifetime of the handler.
Here is an example:
```go
type myHandler struct {}
func (h *myHandler) HandleMessage(m *nsq.Message) error {
m.DisableAutoResponse()
workerChan <- m
return nil
}
go func() {
for m := range workerChan {
err := doSomeWork(m)
if err != nil {
m.Requeue(-1)
continue
}
m.Finish()
}
}()
cfg := nsq.NewConfig()
cfg.MaxInFlight = 1000
r, err := nsq.NewConsumer("test_topic", "test_channel", cfg)
if err != nil {
log.Fatalf(err.Error())
}
r.AddConcurrentHandlers(&myHandler{}, 20)
err := r.ConnectToNSQD(nsqdAddr)
if err != nil {
log.Fatalf(err.Error())
}
<-r.StopChan
```
#### Requeue without backoff
As a side effect of the message handler restructuring above, it is now trivial to respond to a
message without triggering a backoff state in `nsq.Consumer` (which was not possible in the
previous API).
The `nsq.Message` type now has a `msg.RequeueWithoutBackoff()` method for this purpose.
#### Producer Error Handling
Previously, `Writer` (now `Producer`) returned a triplicate of `frameType`, `responseBody`, and
`error` from calls to `*Publish`.
This required the caller to check both `error` and `frameType` to confirm success. `Producer`
publish methods now return only `error`.
#### Logging
One of the challenges library implementors face is how to provide feedback via logging, while
exposing an interface that follows the standard library and still provides a means to control and
configure the output.
In the new API, we've provided a method on `Consumer` and `Producer` called `SetLogger` that takes
an interface compatible with the Go standard library `log.Logger` (which can be instantiated via
`log.NewLogger`) and a traditional log level integer `nsq.LogLevel{Debug,Info,Warning,Error}`:
Output(maxdepth int, s string) error
This gives the user the flexibility to control the format, destination, and verbosity while still
conforming to standard library logging conventions.
#### Misc.
Un-exported `NewDeadlineTransport` and `ApiRequest`, which never should have been exported in the
first place.
`nsq.Message` serialization switched away from `binary.{Read,Write}` for performance and
`nsq.Message` now implements the `io.WriterTo` interface.
================================================
FILE: vendor/github.com/bitly/go-nsq/api_request.go
================================================
package nsq
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"net"
"net/http"
"time"
)
type deadlinedConn struct {
Timeout time.Duration
net.Conn
}
func (c *deadlinedConn) Read(b []byte) (n int, err error) {
c.Conn.SetReadDeadline(time.Now().Add(c.Timeout))
return c.Conn.Read(b)
}
func (c *deadlinedConn) Write(b []byte) (n int, err error) {
c.Conn.SetWriteDeadline(time.Now().Add(c.Timeout))
return c.Conn.Write(b)
}
func newDeadlineTransport(timeout time.Duration) *http.Transport {
transport := &http.Transport{
Dial: func(netw, addr string) (net.Conn, error) {
c, err := net.DialTimeout(netw, addr, timeout)
if err != nil {
return nil, err
}
return &deadlinedConn{timeout, c}, nil
},
}
return transport
}
type wrappedResp struct {
Status string `json:"status_txt"`
StatusCode int `json:"status_code"`
Data interface{} `json:"data"`
}
// stores the result in the value pointed to by ret(must be a pointer)
func apiRequestNegotiateV1(method string, endpoint string, body io.Reader, ret interface{}) error {
httpclient := &http.Client{Transport: newDeadlineTransport(2 * time.Second)}
req, err := http.NewRequest(method, endpoint, body)
if err != nil {
return err
}
req.Header.Add("Accept", "application/vnd.nsq; version=1.0")
resp, err := httpclient.Do(req)
if err != nil {
return err
}
respBody, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return err
}
if resp.StatusCode != 200 {
return fmt.Errorf("got response %s %q", resp.Status, respBody)
}
if len(respBody) == 0 {
respBody = []byte("{}")
}
if resp.Header.Get("X-NSQ-Content-Type") == "nsq; version=1.0" {
return json.Unmarshal(respBody, ret)
}
wResp := &wrappedResp{
Data: ret,
}
if err = json.Unmarshal(respBody, wResp); err != nil {
return err
}
// wResp.StatusCode here is equal to resp.StatusCode, so ignore it
return nil
}
================================================
FILE: vendor/github.com/bitly/go-nsq/command.go
================================================
package nsq
import (
"bytes"
"encoding/binary"
"encoding/json"
"fmt"
"io"
"strconv"
"time"
)
var byteSpace = []byte(" ")
var byteNewLine = []byte("\n")
// Command represents a command from a client to an NSQ daemon
type Command struct {
Name []byte
Params [][]byte
Body []byte
}
// String returns the name and parameters of the Command
func (c *Command) String() string {
if len(c.Params) > 0 {
return fmt.Sprintf("%s %s", c.Name, string(bytes.Join(c.Params, byteSpace)))
}
return string(c.Name)
}
// WriteTo implements the WriterTo interface and
// serializes the Command to the supplied Writer.
//
// It is suggested that the target Writer is buffered
// to avoid performing many system calls.
func (c *Command) WriteTo(w io.Writer) (int64, error) {
var total int64
var buf [4]byte
n, err := w.Write(c.Name)
total += int64(n)
if err != nil {
return total, err
}
for _, param := range c.Params {
n, err := w.Write(byteSpace)
total += int64(n)
if err != nil {
return total, err
}
n, err = w.Write(param)
total += int64(n)
if err != nil {
return total, err
}
}
n, err = w.Write(byteNewLine)
total += int64(n)
if err != nil {
return total, err
}
if c.Body != nil {
bufs := buf[:]
binary.BigEndian.PutUint32(bufs, uint32(len(c.Body)))
n, err := w.Write(bufs)
total += int64(n)
if err != nil {
return total, err
}
n, err = w.Write(c.Body)
total += int64(n)
if err != nil {
return total, err
}
}
return total, nil
}
// Identify creates a new Command to provide information about the client. After connecting,
// it is generally the first message sent.
//
// The supplied map is marshaled into JSON to provide some flexibility
// for this command to evolve over time.
//
// See http://nsq.io/clients/tcp_protocol_spec.html#identify for information
// on the supported options
func Identify(js map[string]interface{}) (*Command, error) {
body, err := json.Marshal(js)
if err != nil {
return nil, err
}
return &Command{[]byte("IDENTIFY"), nil, body}, nil
}
// Auth sends credentials for authentication
//
// After `Identify`, this is usually the first message sent, if auth is used.
func Auth(secret string) (*Command, error) {
return &Command{[]byte("AUTH"), nil, []byte(secret)}, nil
}
// Register creates a new Command to add a topic/channel for the connected nsqd
func Register(topic string, channel string) *Command {
params := [][]byte{[]byte(topic)}
if len(channel) > 0 {
params = append(params, []byte(channel))
}
return &Command{[]byte("REGISTER"), params, nil}
}
// UnRegister creates a new Command to remove a topic/channel for the connected nsqd
func UnRegister(topic string, channel string) *Command {
params := [][]byte{[]byte(topic)}
if len(channel) > 0 {
params = append(params, []byte(channel))
}
return &Command{[]byte("UNREGISTER"), params, nil}
}
// Ping creates a new Command to keep-alive the state of all the
// announced topic/channels for a given client
func Ping() *Command {
return &Command{[]byte("PING"), nil, nil}
}
// Publish creates a new Command to write a message to a given topic
func Publish(topic string, body []byte) *Command {
var params = [][]byte{[]byte(topic)}
return &Command{[]byte("PUB"), params, body}
}
// DeferredPublish creates a new Command to write a message to a given topic
// where the message will queue at the channel level until the timeout expires
func DeferredPublish(topic string, delay time.Duration, body []byte) *Command {
var params = [][]byte{[]byte(topic), []byte(strconv.Itoa(int(delay / time.Millisecond)))}
return &Command{[]byte("DPUB"), params, body}
}
// MultiPublish creates a new Command to write more than one message to a given topic
// (useful for high-throughput situations to avoid roundtrips and saturate the pipe)
func MultiPublish(topic string, bodies [][]byte) (*Command, error) {
var params = [][]byte{[]byte(topic)}
num := uint32(len(bodies))
bodySize := 4
for _, b := range bodies {
bodySize += len(b) + 4
}
body := make([]byte, 0, bodySize)
buf := bytes.NewBuffer(body)
err := binary.Write(buf, binary.BigEndian, &num)
if err != nil {
return nil, err
}
for _, b := range bodies {
err = binary.Write(buf, binary.BigEndian, int32(len(b)))
if err != nil {
return nil, err
}
_, err = buf.Write(b)
if err != nil {
return nil, err
}
}
return &Command{[]byte("MPUB"), params, buf.Bytes()}, nil
}
// Subscribe creates a new Command to subscribe to the given topic/channel
func Subscribe(topic string, channel string) *Command {
var params = [][]byte{[]byte(topic), []byte(channel)}
return &Command{[]byte("SUB"), params, nil}
}
// Ready creates a new Command to specify
// the number of messages a client is willing to receive
func Ready(count int) *Command {
var params = [][]byte{[]byte(strconv.Itoa(count))}
return &Command{[]byte("RDY"), params, nil}
}
// Finish creates a new Command to indiciate that
// a given message (by id) has been processed successfully
func Finish(id MessageID) *Command {
var params = [][]byte{id[:]}
return &Command{[]byte("FIN"), params, nil}
}
// Requeue creates a new Command to indicate that
// a given message (by id) should be requeued after the given delay
// NOTE: a delay of 0 indicates immediate requeue
func Requeue(id MessageID, delay time.Duration) *Command {
var params = [][]byte{id[:], []byte(strconv.Itoa(int(delay / time.Millisecond)))}
return &Command{[]byte("REQ"), params, nil}
}
// Touch creates a new Command to reset the timeout for
// a given message (by id)
func Touch(id MessageID) *Command {
var params = [][]byte{id[:]}
return &Command{[]byte("TOUCH"), params, nil}
}
// StartClose creates a new Command to indicate that the
// client would like to start a close cycle. nsqd will no longer
// send messages to a client in this state and the client is expected
// finish pending messages and close the connection
func StartClose() *Command {
return &Command{[]byte("CLS"), nil, nil}
}
// Nop creates a new Command that has no effect server side.
// Commonly used to respond to heartbeats
func Nop() *Command {
return &Command{[]byte("NOP"), nil, nil}
}
================================================
FILE: vendor/github.com/bitly/go-nsq/command_test.go
================================================
package nsq
import (
"bytes"
"testing"
)
func BenchmarkCommand(b *testing.B) {
b.StopTimer()
data := make([]byte, 2048)
cmd := Publish("test", data)
var buf bytes.Buffer
b.StartTimer()
for i := 0; i < b.N; i++ {
cmd.WriteTo(&buf)
}
}
================================================
FILE: vendor/github.com/bitly/go-nsq/config.go
================================================
package nsq
import (
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/ioutil"
"log"
"math"
"math/rand"
"net"
"os"
"reflect"
"strconv"
"strings"
"sync"
"time"
"unsafe"
)
// Define handlers for setting config defaults, and setting config values from command line arguments or config files
type configHandler interface {
HandlesOption(c *Config, option string) bool
Set(c *Config, option string, value interface{}) error
Validate(c *Config) error
}
type defaultsHandler interface {
SetDefaults(c *Config) error
}
// BackoffStrategy defines a strategy for calculating the duration of time
// a consumer should backoff for a given attempt
type BackoffStrategy interface {
Calculate(attempt int) time.Duration
}
// ExponentialStrategy implements an exponential backoff strategy (default)
type ExponentialStrategy struct {
cfg *Config
}
// Calculate returns a duration of time: 2 ^ attempt
func (s *ExponentialStrategy) Calculate(attempt int) time.Duration {
backoffDuration := s.cfg.BackoffMultiplier *
time.Duration(math.Pow(2, float64(attempt)))
return backoffDuration
}
func (s *ExponentialStrategy) setConfig(cfg *Config) {
s.cfg = cfg
}
// FullJitterStrategy implements http://www.awsarchitectureblog.com/2015/03/backoff.html
type FullJitterStrategy struct {
cfg *Config
rngOnce sync.Once
rng *rand.Rand
}
// Calculate returns a random duration of time [0, 2 ^ attempt]
func (s *FullJitterStrategy) Calculate(attempt int) time.Duration {
// lazily initialize the RNG
s.rngOnce.Do(func() {
if s.rng != nil {
return
}
s.rng = rand.New(rand.NewSource(time.Now().UnixNano()))
})
backoffDuration := s.cfg.BackoffMultiplier *
time.Duration(math.Pow(2, float64(attempt)))
return time.Duration(s.rng.Intn(int(backoffDuration)))
}
func (s *FullJitterStrategy) setConfig(cfg *Config) {
s.cfg = cfg
}
// Config is a struct of NSQ options
//
// The only valid way to create a Config is via NewConfig, using a struct literal will panic.
// After Config is passed into a high-level type (like Consumer, Producer, etc.) the values are no
// longer mutable (they are copied).
//
// Use Set(option string, value interface{}) as an alternate way to set parameters
type Config struct {
initialized bool
// used to Initialize, Validate
configHandlers []configHandler
DialTimeout time.Duration `opt:"dial_timeout" default:"1s"`
// Deadlines for network reads and writes
ReadTimeout time.Duration `opt:"read_timeout" min:"100ms" max:"5m" default:"60s"`
WriteTimeout time.Duration `opt:"write_timeout" min:"100ms" max:"5m" default:"1s"`
// LocalAddr is the local address to use when dialing an nsqd.
// If empty, a local address is automatically chosen.
LocalAddr net.Addr `opt:"local_addr"`
// Duration between polling lookupd for new producers, and fractional jitter to add to
// the lookupd pool loop. this helps evenly distribute requests even if multiple consumers
// restart at the same time
//
// NOTE: when not using nsqlookupd, LookupdPollInterval represents the duration of time between
// reconnection attempts
LookupdPollInterval time.Duration `opt:"lookupd_poll_interval" min:"10ms" max:"5m" default:"60s"`
LookupdPollJitter float64 `opt:"lookupd_poll_jitter" min:"0" max:"1" default:"0.3"`
// Maximum duration when REQueueing (for doubling of deferred requeue)
MaxRequeueDelay time.Duration `opt:"max_requeue_delay" min:"0" max:"60m" default:"15m"`
DefaultRequeueDelay time.Duration `opt:"default_requeue_delay" min:"0" max:"60m" default:"90s"`
// Backoff strategy, defaults to exponential backoff. Overwrite this to define alternative backoff algrithms.
BackoffStrategy BackoffStrategy `opt:"backoff_strategy" default:"exponential"`
// Maximum amount of time to backoff when processing fails 0 == no backoff
MaxBackoffDuration time.Duration `opt:"max_backoff_duration" min:"0" max:"60m" default:"2m"`
// Unit of time for calculating consumer backoff
BackoffMultiplier time.Duration `opt:"backoff_multiplier" min:"0" max:"60m" default:"1s"`
// Maximum number of times this consumer will attempt to process a message before giving up
MaxAttempts uint16 `opt:"max_attempts" min:"0" max:"65535" default:"5"`
// Duration to wait for a message from a producer when in a state where RDY
// counts are re-distributed (ie. max_in_flight < num_producers)
LowRdyIdleTimeout time.Duration `opt:"low_rdy_idle_timeout" min:"1s" max:"5m" default:"10s"`
// Duration between redistributing max-in-flight to connections
RDYRedistributeInterval time.Duration `opt:"rdy_redistribute_interval" min:"1ms" max:"5s" default:"5s"`
// Identifiers sent to nsqd representing this client
// UserAgent is in the spirit of HTTP (default: "<client_library_name>/<version>")
ClientID string `opt:"client_id"` // (defaults: short hostname)
Hostname string `opt:"hostname"`
UserAgent string `opt:"user_agent"`
// Duration of time between heartbeats. This must be less than ReadTimeout
HeartbeatInterval time.Duration `opt:"heartbeat_interval" default:"30s"`
// Integer percentage to sample the channel (requires nsqd 0.2.25+)
SampleRate int32 `opt:"sample_rate" min:"0" max:"99"`
// To set TLS config, use the following options:
//
// tls_v1 - Bool enable TLS negotiation
// tls_root_ca_file - String path to file containing root CA
// tls_insecure_skip_verify - Bool indicates whether this client should verify server certificates
// tls_cert - String path to file containing public key for certificate
// tls_key - String path to file containing private key for certificate
// tls_min_version - String indicating the minimum version of tls acceptable ('ssl3.0', 'tls1.0', 'tls1.1', 'tls1.2')
//
TlsV1 bool `opt:"tls_v1"`
TlsConfig *tls.Config `opt:"tls_config"`
// Compression Settings
Deflate bool `opt:"deflate"`
DeflateLevel int `opt:"deflate_level" min:"1" max:"9" default:"6"`
Snappy bool `opt:"snappy"`
// Size of the buffer (in bytes) used by nsqd for buffering writes to this connection
OutputBufferSize int64 `opt:"output_buffer_size" default:"16384"`
// Timeout used by nsqd before flushing buffered writes (set to 0 to disable).
//
// WARNING: configuring clients with an extremely low
// (< 25ms) output_buffer_timeout has a significant effect
// on nsqd CPU usage (particularly with > 50 clients connected).
OutputBufferTimeout time.Duration `opt:"output_buffer_timeout" default:"250ms"`
// Maximum number of messages to allow in flight (concurrency knob)
MaxInFlight int `opt:"max_in_flight" min:"0" default:"1"`
// The server-side message timeout for messages delivered to this client
MsgTimeout time.Duration `opt:"msg_timeout" min:"0"`
// secret for nsqd authentication (requires nsqd 0.2.29+)
AuthSecret string `opt:"auth_secret"`
}
// NewConfig returns a new default nsq configuration.
//
// This must be used to initialize Config structs. Values can be set directly, or through Config.Set()
func NewConfig() *Config {
c := &Config{
configHandlers: []configHandler{&structTagsConfig{}, &tlsConfig{}},
initialized: true,
}
if err := c.setDefaults(); err != nil {
panic(err.Error())
}
return c
}
// Set takes an option as a string and a value as an interface and
// attempts to set the appropriate configuration option.
//
// It attempts to coerce the value into the right format depending on the named
// option and the underlying type of the value passed in.
//
// Calls to Set() that take a time.Duration as an argument can be input as:
//
// "1000ms" (a string parsed by time.ParseDuration())
// 1000 (an integer interpreted as milliseconds)
// 1000*time.Millisecond (a literal time.Duration value)
//
// Calls to Set() that take bool can be input as:
//
// "true" (a string parsed by strconv.ParseBool())
// true (a boolean)
// 1 (an int where 1 == true and 0 == false)
//
// It returns an error for an invalid option or value.
func (c *Config) Set(option string, value interface{}) error {
c.assertInitialized()
option = strings.Replace(option, "-", "_", -1)
for _, h := range c.configHandlers {
if h.HandlesOption(c, option) {
return h.Set(c, option, value)
}
}
return fmt.Errorf("invalid option %s", option)
}
func (c *Config) assertInitialized() {
if !c.initialized {
panic("Config{} must be created with NewConfig()")
}
}
// Validate checks that all values are within specified min/max ranges
func (c *Config) Validate() error {
c.assertInitialized()
for _, h := range c.configHandlers {
if err := h.Validate(c); err != nil {
return err
}
}
return nil
}
func (c *Config) setDefaults() error {
for _, h := range c.configHandlers {
hh, ok := h.(defaultsHandler)
if ok {
if err := hh.SetDefaults(c); err != nil {
return err
}
}
}
return nil
}
type structTagsConfig struct{}
// Handle options that are listed in StructTags
func (h *structTagsConfig) HandlesOption(c *Config, option string) bool {
val := reflect.ValueOf(c).Elem()
typ := val.Type()
for i := 0; i < typ.NumField(); i++ {
field := typ.Field(i)
opt := field.Tag.Get("opt")
if opt == option {
return true
}
}
return false
}
// Set values based on parameters in StructTags
func (h *structTagsConfig) Set(c *Config, option string, value interface{}) error {
val := reflect.ValueOf(c).Elem()
typ := val.Type()
for i := 0; i < typ.NumField(); i++ {
field := typ.Field(i)
opt := field.Tag.Get("opt")
if option != opt {
continue
}
min := field.Tag.Get("min")
max := field.Tag.Get("max")
fieldVal := val.FieldByName(field.Name)
dest := unsafeValueOf(fieldVal)
coercedVal, err := coerce(value, field.Type)
if err != nil {
return fmt.Errorf("failed to coerce option %s (%v) - %s",
option, value, err)
}
if min != "" {
coercedMinVal, _ := coerce(min, field.Type)
if valueCompare(coercedVal, coercedMinVal) == -1 {
return fmt.Errorf("invalid %s ! %v < %v",
option, coercedVal.Interface(), coercedMinVal.Interface())
}
}
if max != "" {
coercedMaxVal, _ := coerce(max, field.Type)
if valueCompare(coercedVal, coercedMaxVal) == 1 {
return fmt.Errorf("invalid %s ! %v > %v",
option, coercedVal.Interface(), coercedMaxVal.Interface())
}
}
if coercedVal.Type().String() == "nsq.BackoffStrategy" {
v := coercedVal.Interface().(BackoffStrategy)
if v, ok := v.(interface {
setConfig(*Config)
}); ok {
v.setConfig(c)
}
}
dest.Set(coercedVal)
return nil
}
return fmt.Errorf("unknown option %s", option)
}
func (h *structTagsConfig) SetDefaults(c *Config) error {
val := reflect.ValueOf(c).Elem()
typ := val.Type()
for i := 0; i < typ.NumField(); i++ {
field := typ.Field(i)
opt := field.Tag.Get("opt")
defaultVal := field.Tag.Get("default")
if defaultVal == "" || opt == "" {
continue
}
if err := c.Set(opt, defaultVal); err != nil {
return err
}
}
hostname, err := os.Hostname()
if err != nil {
log.Fatalf("ERROR: unable to get hostname %s", err.Error())
}
c.ClientID = strings.Split(hostname, ".")[0]
c.Hostname = hostname
c.UserAgent = fmt.Sprintf("go-nsq/%s", VERSION)
return nil
}
func (h *structTagsConfig) Validate(c *Config) error {
val := reflect.ValueOf(c).Elem()
typ := val.Type()
for i := 0; i < typ.NumField(); i++ {
field := typ.Field(i)
min := field.Tag.Get("min")
max := field.Tag.Get("max")
if min == "" && max == "" {
continue
}
value := val.FieldByName(field.Name)
if min != "" {
coercedMinVal, _ := coerce(min, field.Type)
if valueCompare(value, coercedMinVal) == -1 {
return fmt.Errorf("invalid %s ! %v < %v",
field.Name, value.Interface(), coercedMinVal.Interface())
}
}
if max != "" {
coercedMaxVal, _ := coerce(max, field.Type)
if valueCompare(value, coercedMaxVal) == 1 {
return fmt.Errorf("invalid %s ! %v > %v",
field.Name, value.Interface(), coercedMaxVal.Interface())
}
}
}
if c.HeartbeatInterval > c.ReadTimeout {
return fmt.Errorf("HeartbeatInterval %v must be less than ReadTimeout %v", c.HeartbeatInterval, c.ReadTimeout)
}
return nil
}
// Parsing for higher order TLS settings
type tlsConfig struct {
certFile string
keyFile string
}
func (t *tlsConfig) HandlesOption(c *Config, option string) bool {
switch option {
case "tls_root_ca_file", "tls_insecure_skip_verify", "tls_cert", "tls_key", "tls_min_version":
return true
}
return false
}
func (t *tlsConfig) Set(c *Config, option string, value interface{}) error {
if c.TlsConfig == nil {
c.TlsConfig = &tls.Config{
MinVersion: tls.VersionTLS10,
MaxVersion: tls.VersionTLS12, // enable TLS_FALLBACK_SCSV prior to Go 1.5: https://go-review.googlesource.com/#/c/1776/
}
}
val := reflect.ValueOf(c.TlsConfig).Elem()
switch option {
case "tls_cert", "tls_key":
if option == "tls_cert" {
t.certFile = value.(string)
} else {
t.keyFile = value.(string)
}
if t.certFile != "" && t.keyFile != "" && len(c.TlsConfig.Certificates) == 0 {
cert, err := tls.LoadX509KeyPair(t.certFile, t.keyFile)
if err != nil {
return err
}
c.TlsConfig.Certificates = []tls.Certificate{cert}
}
return nil
case "tls_root_ca_file":
filename, ok := value.(string)
if !ok {
return fmt.Errorf("ERROR: %v is not a string", value)
}
tlsCertPool := x509.NewCertPool()
caCertFile, err := ioutil.ReadFile(filename)
if err != nil {
return fmt.Errorf("ERROR: failed to read custom Certificate Authority file %s", err)
}
if !tlsCertPool.AppendCertsFromPEM(caCertFile) {
return fmt.Errorf("ERROR: failed to append certificates from Certificate Authority file")
}
c.TlsConfig.RootCAs = tlsCertPool
return nil
case "tls_insecure_skip_verify":
fieldVal := val.FieldByName("InsecureSkipVerify")
dest := unsafeValueOf(fieldVal)
coercedVal, err := coerce(value, fieldVal.Type())
if err != nil {
return fmt.Errorf("failed to coerce option %s (%v) - %s",
option, value, err)
}
dest.Set(coercedVal)
return nil
case "tls_min_version":
version, ok := value.(string)
if !ok {
return fmt.Errorf("ERROR: %v is not a string", value)
}
switch version {
case "ssl3.0":
c.TlsConfig.MinVersion = tls.VersionSSL30
case "tls1.0":
c.TlsConfig.MinVersion = tls.VersionTLS10
case "tls1.1":
c.TlsConfig.MinVersion = tls.VersionTLS11
case "tls1.2":
c.TlsConfig.MinVersion = tls.VersionTLS12
default:
return fmt.Errorf("ERROR: %v is not a tls version", value)
}
return nil
}
return fmt.Errorf("unknown option %s", option)
}
func (t *tlsConfig) Validate(c *Config) error {
return nil
}
// because Config contains private structs we can't use reflect.Value
// directly, instead we need to "unsafely" address the variable
func unsafeValueOf(val reflect.Value) reflect.Value {
uptr := unsafe.Pointer(val.UnsafeAddr())
return reflect.NewAt(val.Type(), uptr).Elem()
}
func valueCompare(v1 reflect.Value, v2 reflect.Value) int {
switch v1.Type().String() {
case "int", "int16", "int32", "int64":
if v1.Int() > v2.Int() {
return 1
} else if v1.Int() < v2.Int() {
return -1
}
return 0
case "uint", "uint16", "uint32", "uint64":
if v1.Uint() > v2.Uint() {
return 1
} else if v1.Uint() < v2.Uint() {
return -1
}
return 0
case "float32", "float64":
if v1.Float() > v2.Float() {
return 1
} else if v1.Float() < v2.Float() {
return -1
}
return 0
case "time.Duration":
if v1.Interface().(time.Duration) > v2.Interface().(time.Duration) {
return 1
} else if v1.Interface().(time.Duration) < v2.Interface().(time.Duration) {
return -1
}
return 0
}
panic("impossible")
}
func coerce(v interface{}, typ reflect.Type) (reflect.Value, error) {
var err error
if typ.Kind() == reflect.Ptr {
return reflect.ValueOf(v), nil
}
switch typ.String() {
case "string":
v, err = coerceString(v)
case "int", "int16", "int32", "int64":
v, err = coerceInt64(v)
case "uint", "uint16", "uint32", "uint64":
v, err = coerceUint64(v)
case "float32", "float64":
v, err = coerceFloat64(v)
case "bool":
v, err = coerceBool(v)
case "time.Duration":
v, err = coerceDuration(v)
case "net.Addr":
v, err = coerceAddr(v)
case "nsq.BackoffStrategy":
v, err = coerceBackoffStrategy(v)
default:
v = nil
err = fmt.Errorf("invalid type %s", typ.String())
}
return valueTypeCoerce(v, typ), err
}
func valueTypeCoerce(v interface{}, typ reflect.Type) reflect.Value {
val := reflect.ValueOf(v)
if reflect.TypeOf(v) == typ {
return val
}
tval := reflect.New(typ).Elem()
switch typ.String() {
case "int", "int16", "int32", "int64":
tval.SetInt(val.Int())
case "uint", "uint16", "uint32", "uint64":
tval.SetUint(val.Uint())
case "float32", "float64":
tval.SetFloat(val.Float())
default:
tval.Set(val)
}
return tval
}
func coerceString(v interface{}) (string, error) {
switch v := v.(type) {
case string:
return v, nil
case int, int16, int32, int64, uint, uint16, uint32, uint64:
return fmt.Sprintf("%d", v), nil
case float32, float64:
return fmt.Sprintf("%f", v), nil
}
return fmt.Sprintf("%s", v), nil
}
func coerceDuration(v interface{}) (time.Duration, error) {
switch v := v.(type) {
case string:
return time.ParseDuration(v)
case int, int16, int32, int64:
// treat like ms
return time.Duration(reflect.ValueOf(v).Int()) * time.Millisecond, nil
case uint, uint16, uint32, uint64:
// treat like ms
return time.Duration(reflect.ValueOf(v).Uint()) * time.Millisecond, nil
case time.Duration:
return v, nil
}
return 0, errors.New("invalid value type")
}
func coerceAddr(v interface{}) (net.Addr, error) {
switch v := v.(type) {
case string:
return net.ResolveTCPAddr("tcp", v)
case net.Addr:
return v, nil
}
return nil, errors.New("invalid value type")
}
func coerceBackoffStrategy(v interface{}) (BackoffStrategy, error) {
switch v := v.(type) {
case string:
switch v {
case "", "exponential":
return &ExponentialStrategy{}, nil
case "full_jitter":
return &FullJitterStrategy{}, nil
}
case BackoffStrategy:
return v, nil
}
return nil, errors.New("invalid value type")
}
func coerceBool(v interface{}) (bool, error) {
switch v := v.(type) {
case bool:
return v, nil
case string:
return strconv.ParseBool(v)
case int, int16, int32, int64:
return reflect.ValueOf(v).Int() != 0, nil
case uint, uint16, uint32, uint64:
return reflect.ValueOf(v).Uint() != 0, nil
}
return false, errors.New("invalid value type")
}
func coerceFloat64(v interface{}) (float64, error) {
switch v := v.(type) {
case string:
return strconv.ParseFloat(v, 64)
case int, int16, int32, int64:
return float64(reflect.ValueOf(v).Int()), nil
case uint, uint16, uint32, uint64:
return float64(reflect.ValueOf(v).Uint()), nil
case float32:
return float64(v), nil
case float64:
return v, nil
}
return 0, errors.New("invalid value type")
}
func coerceInt64(v interface{}) (int64, error) {
switch v := v.(type) {
case string:
return strconv.ParseInt(v, 10, 64)
case int, int16, int32, int64:
return reflect.ValueOf(v).Int(), nil
case uint, uint16, uint32, uint64:
return int64(reflect.ValueOf(v).Uint()), nil
}
return 0, errors.New("invalid value type")
}
func coerceUint64(v interface{}) (uint64, error) {
switch v := v.(type) {
case string:
return strconv.ParseUint(v, 10, 64)
case int, int16, int32, int64:
return uint64(reflect.ValueOf(v).Int()), nil
case uint, uint16, uint32, uint64:
return reflect.ValueOf(v).Uint(), nil
}
return 0, errors.New("invalid value type")
}
================================================
FILE: vendor/github.com/bitly/go-nsq/config_test.go
================================================
package nsq
import (
"math/rand"
"net"
"reflect"
"testing"
"time"
)
func TestConfigSet(t *testing.T) {
c := NewConfig()
if err := c.Set("not a real config value", struct{}{}); err == nil {
t.Error("No error when setting an invalid value")
}
if err := c.Set("tls_v1", "lol"); err == nil {
t.Error("No error when setting `tls_v1` to an invalid value")
}
if err := c.Set("tls_v1", true); err != nil {
t.Errorf("Error setting `tls_v1` config. %s", err)
}
if err := c.Set("tls-insecure-skip-verify", true); err != nil {
t.Errorf("Error setting `tls-insecure-skip-verify` config. %v", err)
}
if c.TlsConfig.InsecureSkipVerify != true {
t.Errorf("Error setting `tls-insecure-skip-verify` config: %v", c.TlsConfig)
}
if err := c.Set("tls-min-version", "tls1.2"); err != nil {
t.Errorf("Error setting `tls-min-version` config: %s", err)
}
if err := c.Set("tls-min-version", "tls1.3"); err == nil {
t.Error("No error when setting `tls-min-version` to an invalid value")
}
if err := c.Set("local_addr", &net.TCPAddr{}); err != nil {
t.Errorf("Error setting `local_addr` config: %s", err)
}
if err := c.Set("local_addr", "1.2.3.4:27015"); err != nil {
t.Errorf("Error setting `local_addr` config: %s", err)
}
if err := c.Set("dial_timeout", "5s"); err != nil {
t.Errorf("Error setting `dial_timeout` config: %s", err)
}
if c.LocalAddr.String() != "1.2.3.4:27015" {
t.Error("Failed to assign `local_addr` config")
}
if reflect.ValueOf(c.BackoffStrategy).Type().String() != "*nsq.ExponentialStrategy" {
t.Error("Failed to set default `exponential` backoff strategy")
}
if err := c.Set("backoff_strategy", "full_jitter"); err != nil {
t.Errorf("Failed to assign `backoff_strategy` config: %v", err)
}
if reflect.ValueOf(c.BackoffStrategy).Type().String() != "*nsq.FullJitterStrategy" {
t.Error("Failed to set `full_jitter` backoff strategy")
}
}
func TestConfigValidate(t *testing.T) {
c := NewConfig()
if err := c.Validate(); err != nil {
t.Error("initialized config is invalid")
}
c.DeflateLevel = 100
if err := c.Validate(); err == nil {
t.Error("no error set for invalid value")
}
}
func TestExponentialBackoff(t *testing.T) {
expected := []time.Duration{
1 * time.Second,
2 * time.Second,
8 * time.Second,
32 * time.Second,
}
backoffTest(t, expected, func(c *Config) BackoffStrategy {
return &ExponentialStrategy{cfg: c}
})
}
func TestFullJitterBackoff(t *testing.T) {
expected := []time.Duration{
566028617 * time.Nanosecond,
1365407263 * time.Nanosecond,
5232470547 * time.Nanosecond,
21467499218 * time.Nanosecond,
}
backoffTest(t, expected, func(c *Config) BackoffStrategy {
return &FullJitterStrategy{cfg: c, rng: rand.New(rand.NewSource(99))}
})
}
func backoffTest(t *testing.T, expected []time.Duration, cb func(c *Config) BackoffStrategy) {
config := NewConfig()
attempts := []int{0, 1, 3, 5}
s := cb(config)
for i := range attempts {
result := s.Calculate(attempts[i])
if result != expected[i] {
t.Fatalf("wrong backoff duration %v for attempt %d (should be %v)",
result, attempts[i], expected[i])
}
}
}
================================================
FILE: vendor/github.com/bitly/go-nsq/conn.go
================================================
package nsq
import (
"bufio"
"bytes"
"compress/flate"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/mreiferson/go-snappystream"
)
// IdentifyResponse represents the metadata
// returned from an IDENTIFY command to nsqd
type IdentifyResponse struct {
MaxRdyCount int64 `json:"max_rdy_count"`
TLSv1 bool `json:"tls_v1"`
Deflate bool `json:"deflate"`
Snappy bool `json:"snappy"`
AuthRequired bool `json:"auth_required"`
}
// AuthResponse represents the metadata
// returned from an AUTH command to nsqd
type AuthResponse struct {
Identity string `json:"identity"`
IdentityUrl string `json:"identity_url"`
PermissionCount int64 `json:"permission_count"`
}
type msgResponse struct {
msg *Message
cmd *Command
success bool
backoff bool
}
// Conn represents a connection to nsqd
//
// Conn exposes a set of callbacks for the
// various events that occur on a connection
type Conn struct {
// 64bit atomic vars need to be first for proper alignment on 32bit platforms
messagesInFlight int64
maxRdyCount int64
rdyCount int64
lastRdyCount int64
lastMsgTimestamp int64
mtx sync.Mutex
config *Config
conn *net.TCPConn
tlsConn *tls.Conn
addr string
delegate ConnDelegate
logger logger
logLvl LogLevel
logFmt string
logGuard sync.RWMutex
r io.Reader
w io.Writer
cmdChan chan *Command
msgResponseChan chan *msgResponse
exitChan chan int
drainReady chan int
closeFlag int32
stopper sync.Once
wg sync.WaitGroup
readLoopRunning int32
}
// NewConn returns a new Conn instance
func NewConn(addr string, config *Config, delegate ConnDelegate) *Conn {
if !config.initialized {
panic("Config must be created with NewConfig()")
}
return &Conn{
addr: addr,
config: config,
delegate: delegate,
maxRdyCount: 2500,
lastMsgTimestamp: time.Now().UnixNano(),
cmdChan: make(chan *Command),
msgResponseChan: make(chan *msgResponse),
exitChan: make(chan int),
drainReady: make(chan int),
}
}
// SetLogger assigns the logger to use as well as a level.
//
// The format parameter is expected to be a printf compatible string with
// a single %s argument. This is useful if you want to provide additional
// context to the log messages that the connection will print, the default
// is '(%s)'.
//
// The logger parameter is an interface that requires the following
// method to be implemented (such as the the stdlib log.Logger):
//
// Output(calldepth int, s string)
//
func (c *Conn) SetLogger(l logger, lvl LogLevel, format string) {
c.logGuard.Lock()
defer c.logGuard.Unlock()
c.logger = l
c.logLvl = lvl
c.logFmt = format
if c.logFmt == "" {
c.logFmt = "(%s)"
}
}
func (c *Conn) getLogger() (logger, LogLevel, string) {
c.logGuard.RLock()
defer c.logGuard.RUnlock()
return c.logger, c.logLvl, c.logFmt
}
// Connect dials and bootstraps the nsqd connection
// (including IDENTIFY) and returns the IdentifyResponse
func (c *Conn) Connect() (*IdentifyResponse, error) {
dialer := &net.Dialer{
LocalAddr: c.config.LocalAddr,
Timeout: c.config.DialTimeout,
}
conn, err := dialer.Dial("tcp", c.addr)
if err != nil {
return nil, err
}
c.conn = conn.(*net.TCPConn)
c.r = conn
c.w = conn
_, err = c.Write(MagicV2)
if err != nil {
c.Close()
return nil, fmt.Errorf("[%s] failed to write magic - %s", c.addr, err)
}
resp, err := c.identify()
if err != nil {
return nil, err
}
if resp != nil && resp.AuthRequired {
if c.config.AuthSecret == "" {
c.log(LogLevelError, "Auth Required")
return nil, errors.New("Auth Required")
}
err := c.auth(c.config.AuthSecret)
if err != nil {
c.log(LogLevelError, "Auth Failed %s", err)
return nil, err
}
}
c.wg.Add(2)
atomic.StoreInt32(&c.readLoopRunning, 1)
go c.readLoop()
go c.writeLoop()
return resp, nil
}
// Close idempotently initiates connection close
func (c *Conn) Close() error {
atomic.StoreInt32(&c.closeFlag, 1)
if c.conn != nil && atomic.LoadInt64(&c.messagesInFlight) == 0 {
return c.conn.CloseRead()
}
return nil
}
// IsClosing indicates whether or not the
// connection is currently in the processing of
// gracefully closing
func (c *Conn) IsClosing() bool {
return atomic.LoadInt32(&c.closeFlag) == 1
}
// RDY returns the current RDY count
func (c *Conn) RDY() int64 {
return atomic.LoadInt64(&c.rdyCount)
}
// LastRDY returns the previously set RDY count
func (c *Conn) LastRDY() int64 {
return atomic.LoadInt64(&c.lastRdyCount)
}
// SetRDY stores the specified RDY count
func (c *Conn) SetRDY(rdy int64) {
atomic.StoreInt64(&c.rdyCount, rdy)
atomic.StoreInt64(&c.lastRdyCount, rdy)
}
// MaxRDY returns the nsqd negotiated maximum
// RDY count that it will accept for this connection
func (c *Conn) MaxRDY() int64 {
return c.maxRdyCount
}
// LastMessageTime returns a time.Time representing
// the time at which the last message was received
func (c *Conn) LastMessageTime() time.Time {
return time.Unix(0, atomic.LoadInt64(&c.lastMsgTimestamp))
}
// RemoteAddr returns the configured destination nsqd address
func (c *Conn) RemoteAddr() net.Addr {
return c.conn.RemoteAddr()
}
// String returns the fully-qualified address
func (c *Conn) String() string {
return c.addr
}
// Read performs a deadlined read on the underlying TCP connection
func (c *Conn) Read(p []byte) (int, error) {
c.conn.SetReadDeadline(time.Now().Add(c.config.ReadTimeout))
return c.r.Read(p)
}
// Write performs a deadlined write on the underlying TCP connection
func (c *Conn) Write(p []byte) (int, error) {
c.conn.SetWriteDeadline(time.Now().Add(c.config.WriteTimeout))
return c.w.Write(p)
}
// WriteCommand is a goroutine safe method to write a Command
// to this connection, and flush.
func (c *Conn) WriteCommand(cmd *Command) error {
c.mtx.Lock()
_, err := cmd.WriteTo(c)
if err != nil {
goto exit
}
err = c.Flush()
exit:
c.mtx.Unlock()
if err != nil {
c.log(LogLevelError, "IO error - %s", err)
c.delegate.OnIOError(c, err)
}
return err
}
type flusher interface {
Flush() error
}
// Flush writes all buffered data to the underlying TCP connection
func (c *Conn) Flush() error {
if f, ok := c.w.(flusher); ok {
return f.Flush()
}
return nil
}
func (c *Conn) identify() (*IdentifyResponse, error) {
ci := make(map[string]interface{})
ci["client_id"] = c.config.ClientID
ci["hostname"] = c.config.Hostname
ci["user_agent"] = c.config.UserAgent
ci["short_id"] = c.config.ClientID // deprecated
ci["long_id"] = c.config.Hostname // deprecated
ci["tls_v1"] = c.config.TlsV1
ci["deflate"] = c.config.Deflate
ci["deflate_level"] = c.config.DeflateLevel
ci["snappy"] = c.config.Snappy
ci["feature_negotiation"] = true
if c.config.HeartbeatInterval == -1 {
ci["heartbeat_interval"] = -1
} else {
ci["heartbeat_interval"] = int64(c.config.HeartbeatInterval / time.Millisecond)
}
ci["sample_rate"] = c.config.SampleRate
ci["output_buffer_size"] = c.config.OutputBufferSize
if c.config.OutputBufferTimeout == -1 {
ci["output_buffer_timeout"] = -1
} else {
ci["output_buffer_timeout"] = int64(c.config.OutputBufferTimeout / time.Millisecond)
}
ci["msg_timeout"] = int64(c.config.MsgTimeout / time.Millisecond)
cmd, err := Identify(ci)
if err != nil {
return nil, ErrIdentify{err.Error()}
}
err = c.WriteCommand(cmd)
if err != nil {
return nil, ErrIdentify{err.Error()}
}
frameType, data, err := ReadUnpackedResponse(c)
if err != nil {
return nil, ErrIdentify{err.Error()}
}
if frameType == FrameTypeError {
return nil, ErrIdentify{string(data)}
}
// check to see if the server was able to respond w/ capabilities
// i.e. it was a JSON response
if data[0] != '{' {
return nil, nil
}
resp := &IdentifyResponse{}
err = json.Unmarshal(data, resp)
if err != nil {
return nil, ErrIdentify{err.Error()}
}
c.log(LogLevelDebug, "IDENTIFY response: %+v", resp)
c.maxRdyCount = resp.MaxRdyCount
if resp.TLSv1 {
c.log(LogLevelInfo, "upgrading to TLS")
err := c.upgradeTLS(c.config.TlsConfig)
if err != nil {
return nil, ErrIdentify{err.Error()}
}
}
if resp.Deflate {
c.log(LogLevelInfo, "upgrading to Deflate")
err := c.upgradeDeflate(c.config.DeflateLevel)
if err != nil {
return nil, ErrIdentify{err.Error()}
}
}
if resp.Snappy {
c.log(LogLevelInfo, "upgrading to Snappy")
err := c.upgradeSnappy()
if err != nil {
return nil, ErrIdentify{err.Error()}
}
}
// now that connection is bootstrapped, enable read buffering
// (and write buffering if it's not already capable of Flush())
c.r = bufio.NewReader(c.r)
if _, ok := c.w.(flusher); !ok {
c.w = bufio.NewWriter(c.w)
}
return resp, nil
}
func (c *Conn) upgradeTLS(tlsConf *tls.Config) error {
// create a local copy of the config to set ServerName for this connection
var conf tls.Config
if tlsConf != nil {
conf = *tlsConf
}
host, _, err := net.SplitHostPort(c.addr)
if err != nil {
return err
}
conf.ServerName = host
c.tlsConn = tls.Client(c.conn, &conf)
err = c.tlsConn.Handshake()
if err != nil {
return err
}
c.r = c.tlsConn
c.w = c.tlsConn
frameType, data, err := ReadUnpackedResponse(c)
if err != nil {
return err
}
if frameType != FrameTypeResponse || !bytes.Equal(data, []byte("OK")) {
return errors.New("invalid response from TLS upgrade")
}
return nil
}
func (c *Conn) upgradeDeflate(level int) error {
conn := net.Conn(c.conn)
if c.tlsConn != nil {
conn = c.tlsConn
}
fw, _ := flate.NewWriter(conn, level)
c.r = flate.NewReader(conn)
c.w = fw
frameType, data, err := ReadUnpackedResponse(c)
if err != nil {
return err
}
if frameType != FrameTypeResponse || !bytes.Equal(data, []byte("OK")) {
return errors.New("invalid response from Deflate upgrade")
}
return nil
}
func (c *Conn) upgradeSnappy() error {
conn := net.Conn(c.conn)
if c.tlsConn != nil {
conn = c.tlsConn
}
c.r = snappystream.NewReader(conn, snappystream.SkipVerifyChecksum)
c.w = snappystream.NewWriter(conn)
frameType, data, err := ReadUnpackedResponse(c)
if err != nil {
return err
}
if frameType != FrameTypeResponse || !bytes.Equal(data, []byte("OK")) {
return errors.New("invalid response from Snappy upgrade")
}
return nil
}
func (c *Conn) auth(secret string) error {
cmd, err := Auth(secret)
if err != nil {
return err
}
err = c.WriteCommand(cmd)
if err != nil {
return err
}
frameType, data, err := ReadUnpackedResponse(c)
if err != nil {
return err
}
if frameType == FrameTypeError {
return errors.New("Error authenticating " + string(data))
}
resp := &AuthResponse{}
err = json.Unmarshal(data, resp)
if err != nil {
return err
}
c.log(LogLevelInfo, "Auth accepted. Identity: %q %s Permissions: %d",
resp.Identity, resp.IdentityUrl, resp.PermissionCount)
return nil
}
func (c *Conn) readLoop() {
delegate := &connMessageDelegate{c}
for {
if atomic.LoadInt32(&c.closeFlag) == 1 {
goto exit
}
frameType, data, err := ReadUnpackedResponse(c)
if err != nil {
if !strings.Contains(err.Error(), "use of closed network connection") {
c.log(LogLevelError, "IO error - %s", err)
c.delegate.OnIOError(c, err)
}
goto exit
}
if frameType == FrameTypeResponse && bytes.Equal(data, []byte("_heartbeat_")) {
c.log(LogLevelDebug, "heartbeat received")
c.delegate.OnHeartbeat(c)
err := c.WriteCommand(Nop())
if err != nil {
c.log(LogLevelError, "IO error - %s", err)
c.delegate.OnIOError(c, err)
goto exit
}
continue
}
switch frameType {
case FrameTypeResponse:
c.delegate.OnResponse(c, data)
case FrameTypeMessage:
msg, err := DecodeMessage(data)
if err != nil {
c.log(LogLevelError, "IO error - %s", err)
c.delegate.OnIOError(c, err)
goto exit
}
msg.Delegate = delegate
msg.NSQDAddress = c.String()
atomic.AddInt64(&c.rdyCount, -1)
atomic.AddInt64(&c.messagesInFlight, 1)
atomic.StoreInt64(&c.lastMsgTimestamp, time.Now().UnixNano())
c.delegate.OnMessage(c, msg)
case FrameTypeError:
c.log(LogLevelError, "protocol error - %s", data)
c.delegate.OnError(c, data)
default:
c.log(LogLevelError, "IO error - %s", err)
c.delegate.OnIOError(c, fmt.Errorf("unknown frame type %d", frameType))
}
}
exit:
atomic.StoreInt32(&c.readLoopRunning, 0)
// start the connection close
messagesInFlight := atomic.LoadInt64(&c.messagesInFlight)
if messagesInFlight == 0 {
// if we exited readLoop with no messages in flight
// we need to explicitly trigger the close because
// writeLoop won't
c.close()
} else {
c.log(LogLevelWarning, "delaying close, %d outstanding messages", messagesInFlight)
}
c.wg.Done()
c.log(LogLevelInfo, "readLoop exiting")
}
func (c *Conn) writeLoop() {
for {
select {
case <-c.exitChan:
c.log(LogLevelInfo, "breaking out of writeLoop")
// Indicate drainReady because we will not pull any more off msgResponseChan
close(c.drainReady)
goto exit
case cmd := <-c.cmdChan:
err := c.WriteCommand(cmd)
if err != nil {
c.log(LogLevelError, "error sending command %s - %s", cmd, err)
c.close()
continue
}
case resp := <-c.msgResponseChan:
// Decrement this here so it is correct even if we can't respond to nsqd
msgsInFlight := atomic.AddInt64(&c.messagesInFlight, -1)
if resp.success {
c.log(LogLevelDebug, "FIN %s", resp.msg.ID)
c.delegate.OnMessageFinished(c, resp.msg)
c.delegate.OnResume(c)
} else {
c.log(LogLevelDebug, "REQ %s", resp.msg.ID)
c.delegate.OnMessageRequeued(c, resp.msg)
if resp.backoff {
c.delegate.OnBackoff(c)
} else {
c.delegate.OnContinue(c)
}
}
err := c.WriteCommand(resp.cmd)
if err != nil {
c.log(LogLevelError, "error sending command %s - %s", resp.cmd, err)
c.close()
continue
}
if msgsInFlight == 0 &&
atomic.LoadInt32(&c.closeFlag) == 1 {
c.close()
continue
}
}
}
exit:
c.wg.Done()
c.log(LogLevelInfo, "writeLoop exiting")
}
func (c *Conn) close() {
// a "clean" connection close is orchestrated as follows:
//
// 1. CLOSE cmd sent to nsqd
// 2. CLOSE_WAIT response received from nsqd
// 3. set c.closeFlag
// 4. readLoop() exits
// a. if messages-in-flight > 0 delay close()
// i. writeLoop() continues receiving on c.msgResponseChan chan
// x. when messages-in-flight == 0 call close()
// b. else call close() immediately
// 5. c.exitChan close
// a. writeLoop() exits
// i. c.drainReady close
// 6a. launch cleanup() goroutine (we're racing with intraprocess
// routed messages, see comments below)
// a. wait on c.drainReady
// b. loop and receive on c.msgResponseChan chan
// until messages-in-flight == 0
// i. ensure that readLoop has exited
// 6b. launch waitForCleanup() goroutine
// b. wait on waitgroup (covers readLoop() and writeLoop()
// and cleanup goroutine)
// c. underlying TCP connection close
// d. trigger Delegate OnClose()
//
c.stopper.Do(func() {
c.log(LogLevelInfo, "beginning close")
close(c.exitChan)
c.conn.CloseRead()
c.wg.Add(1)
go c.cleanup()
go c.waitForCleanup()
})
}
func (c *Conn) cleanup() {
<-c.drainReady
ticker := time.NewTicker(100 * time.Millisecond)
lastWarning := time.Now()
// writeLoop has exited, drain any remaining in flight messages
for {
// we're racing with readLoop which potentially has a message
// for handling so infinitely loop until messagesInFlight == 0
// and readLoop has exited
var msgsInFlight int64
select {
case <-c.msgResponseChan:
msgsInFlight = atomic.AddInt64(&c.messagesInFlight, -1)
case <-ticker.C:
msgsInFlight = atomic.LoadInt64(&c.messagesInFlight)
}
if msgsInFlight > 0 {
if time.Now().Sub(lastWarning) > time.Second {
c.log(LogLevelWarning, "draining... waiting for %d messages in flight", msgsInFlight)
lastWarning = time.Now()
}
continue
}
// until the readLoop has exited we cannot be sure that there
// still won't be a race
if atomic.LoadInt32(&c.readLoopRunning) == 1 {
if time.Now().Sub(lastWarning) > time.Second {
c.log(LogLevelWarning, "draining... readLoop still running")
lastWarning = time.Now()
}
continue
}
goto exit
}
exit:
ticker.Stop()
c.wg.Done()
c.log(LogLevelInfo, "finished draining, cleanup exiting")
}
func (c *Conn) waitForCleanup() {
// this blocks until readLoop and writeLoop
// (and cleanup goroutine above) have exited
c.wg.Wait()
c.conn.CloseWrite()
c.log(LogLevelInfo, "clean close complete")
c.delegate.OnClose(c)
}
func (c *Conn) onMessageFinish(m *Message) {
c.msgResponseChan <- &msgResponse{msg: m, cmd: Finish(m.ID), success: true}
}
func (c *Conn) onMessageRequeue(m *Message, delay time.Duration, backoff bool) {
if delay == -1 {
// linear delay
delay = c.config.DefaultRequeueDelay * time.Duration(m.Attempts)
// bound the requeueDelay to configured max
if delay > c.config.MaxRequeueDelay {
delay = c.config.MaxRequeueDelay
}
}
c.msgResponseChan <- &msgResponse{msg: m, cmd: Requeue(m.ID, delay), success: false, backoff: backoff}
}
func (c *Conn) onMessageTouch(m *Message) {
select {
case c.cmdChan <- Touch(m.ID):
case <-c.exitChan:
}
}
func (c *Conn) log(lvl LogLevel, line string, args ...interface{}) {
logger, logLvl, logFmt := c.getLogger()
if logger == nil {
return
}
if logLvl > lvl {
return
}
logger.Output(2, fmt.Sprintf("%-4s %s %s", lvl,
fmt.Sprintf(logFmt, c.String()),
fmt.Sprintf(line, args...)))
}
================================================
FILE: vendor/github.com/bitly/go-nsq/consumer.go
================================================
package nsq
import (
"bytes"
"errors"
"fmt"
"log"
"math"
"math/rand"
"net"
"net/url"
"os"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
)
// Handler is the message processing interface for Consumer
//
// Implement this interface for handlers that return whether or not message
// processing completed successfully.
//
// When the return value is nil Consumer will automatically handle FINishing.
//
// When the returned value is non-nil Consumer will automatically handle REQueing.
type Handler interface {
HandleMessage(message *Message) error
}
// HandlerFunc is a convenience type to avoid having to declare a struct
// to implement the Handler interface, it can be used like this:
//
// consumer.AddHandler(nsq.HandlerFunc(func(m *Message) error {
// // handle the message
// }))
type HandlerFunc func(message *Message) error
// HandleMessage implements the Handler interface
func (h HandlerFunc) HandleMessage(m *Message) error {
return h(m)
}
// DiscoveryFilter is an interface accepted by `SetBehaviorDelegate()`
// for filtering the nsqds returned from discovery via nsqlookupd
type DiscoveryFilter interface {
Filter([]string) []string
}
// FailedMessageLogger is an interface that can be implemented by handlers that wish
// to receive a callback when a message is deemed "failed" (i.e. the number of attempts
// exceeded the Consumer specified MaxAttemptCount)
type FailedMessageLogger interface {
LogFailedMessage(message *Message)
}
// ConsumerStats represents a snapshot of the state of a Consumer's connections and the messages
// it has seen
type ConsumerStats struct {
MessagesReceived uint64
MessagesFinished uint64
MessagesRequeued uint64
Connections int
}
var instCount int64
type backoffSignal int
const (
backoffFlag backoffSignal = iota
continueFlag
resumeFlag
)
// Consumer is a high-level type to consume from NSQ.
//
// A Consumer instance is supplied a Handler that will be executed
// concurrently via goroutines to handle processing the stream of messages
// consumed from the specified topic/channel. See: Handler/HandlerFunc
// for details on implementing the interface to create handlers.
//
// If configured, it will poll nsqlookupd instances and handle connection (and
// reconnection) to any discovered nsqds.
type Consumer struct {
// 64bit atomic vars need to be first for proper alignment on 32bit platforms
messagesReceived uint64
messagesFinished uint64
messagesRequeued uint64
totalRdyCount int64
backoffDuration int64
backoffCounter int32
maxInFlight int32
mtx sync.RWMutex
logger logger
logLvl LogLevel
logGuard sync.RWMutex
behaviorDelegate interface{}
id int64
topic string
channel string
config Config
rng *rand.Rand
needRDYRedistributed int32
backoffMtx sync.RWMutex
incomingMessages chan *Message
rdyRetryMtx sync.RWMutex
rdyRetryTimers map[string]*time.Timer
pendingConnections map[string]*Conn
connections map[string]*Conn
nsqdTCPAddrs []string
// used at connection close to force a possible reconnect
lookupdRecheckChan chan int
lookupdHTTPAddrs []string
lookupdQueryIndex int
wg sync.WaitGroup
runningHandlers int32
stopFlag int32
connectedFlag int32
stopHandler sync.Once
exitHandler sync.Once
// read from this channel to block until consumer is cleanly stopped
StopChan chan int
exitChan chan int
}
// NewConsumer creates a new instance of Consumer for the specified topic/channel
//
// The only valid way to create a Config is via NewConfig, using a struct literal will panic.
// After Config is passed into NewConsumer the values are no longer mutable (they are copied).
func NewConsumer(topic string, channel string, config *Config) (*Consumer, error) {
config.assertInitialized()
if err := config.Validate(); err != nil {
return nil, err
}
if !IsValidTopicName(topic) {
return nil, errors.New("invalid topic name")
}
if !IsValidChannelName(channel) {
return nil, errors.New("invalid channel name")
}
r := &Consumer{
id: atomic.AddInt64(&instCount, 1),
topic: topic,
channel: channel,
config: *config,
logger: log.New(os.Stderr, "", log.Flags()),
logLvl: LogLevelInfo,
maxInFlight: int32(config.MaxInFlight),
incomingMessages: make(chan *Message),
rdyRetryTimers: make(map[string]*time.Timer),
pendingConnections: make(map[string]*Conn),
connections: make(map[string]*Conn),
lookupdRecheckChan: make(chan int, 1),
rng: rand.New(rand.NewSource(time.Now().UnixNano())),
StopChan: make(chan int),
exitChan: make(chan int),
}
r.wg.Add(1)
go r.rdyLoop()
return r, nil
}
// Stats retrieves the current connection and message statistics for a Consumer
func (r *Consumer) Stats() *ConsumerStats {
return &ConsumerStats{
MessagesReceived: atomic.LoadUint64(&r.messagesReceived),
MessagesFinished: atomic.LoadUint64(&r.messagesFinished),
MessagesRequeued: atomic.LoadUint64(&r.messagesRequeued),
Connections: len(r.conns()),
}
}
func (r *Consumer) conns() []*Conn {
r.mtx.RLock()
conns := make([]*Conn, 0, len(r.connections))
for _, c := range r.connections {
conns = append(conns, c)
}
r.mtx.RUnlock()
return conns
}
// SetLogger assigns the logger to use as well as a level
//
// The logger parameter is an interface that requires the following
// method to be implemented (such as the the stdlib log.Logger):
//
// Output(calldepth int, s string)
//
func (r *Consumer) SetLogger(l logger, lvl LogLevel) {
r.logGuard.Lock()
defer r.logGuard.Unlock()
r.logger = l
r.logLvl = lvl
}
func (r *Consumer) getLogger() (logger, LogLevel) {
r.logGuard.RLock()
defer r.logGuard.RUnlock()
return r.logger, r.logLvl
}
// SetBehaviorDelegate takes a type implementing one or more
// of the following interfaces that modify the behavior
// of the `Consumer`:
//
// DiscoveryFilter
//
func (r *Consumer) SetBehaviorDelegate(cb interface{}) {
matched := false
if _, ok := cb.(DiscoveryFilter); ok {
matched = true
}
if !matched {
panic("behavior delegate does not have any recognized methods")
}
r.behaviorDelegate = cb
}
// perConnMaxInFlight calculates the per-connection max-in-flight count.
//
// This may change dynamically based on the number of connections to nsqd the Consumer
// is responsible for.
func (r *Consumer) perConnMaxInFlight() int64 {
b := float64(r.getMaxInFlight())
s := b / float64(len(r.conns()))
return int64(math.Min(math.Max(1, s), b))
}
// IsStarved indicates whether any connections for this consumer are blocked on processing
// before being able to receive more messages (ie. RDY count of 0 and not exiting)
func (r *Consumer) IsStarved() bool {
for _, conn := range r.conns() {
threshold := int64(float64(atomic.LoadInt64(&conn.lastRdyCount)) * 0.85)
inFlight := atomic.LoadInt64(&conn.messagesInFlight)
if inFlight >= threshold && inFlight > 0 && !conn.IsClosing() {
return true
}
}
return false
}
func (r *Consumer) getMaxInFlight() int32 {
return atomic.LoadInt32(&r.maxInFlight)
}
// ChangeMaxInFlight sets a new maximum number of messages this comsumer instance
// will allow in-flight, and updates all existing connections as appropriate.
//
// For example, ChangeMaxInFlight(0) would pause message flow
//
// If already connected, it updates the reader RDY state for each connection.
func (r *Consumer) ChangeMaxInFlight(maxInFlight int) {
if r.getMaxInFlight() == int32(maxInFlight) {
return
}
atomic.StoreInt32(&r.maxInFlight, int32(maxInFlight))
for _, c := range r.conns() {
r.maybeUpdateRDY(c)
}
}
// ConnectToNSQLookupd adds an nsqlookupd address to the list for this Consumer instance.
//
// If it is the first to be added, it initiates an HTTP request to discover nsqd
// producers for the configured topic.
//
// A goroutine is spawned to handle continual polling.
func (r *Consumer) ConnectToNSQLookupd(addr string) error {
if atomic.LoadInt32(&r.stopFlag) == 1 {
return errors.New("consumer stopped")
}
if atomic.LoadInt32(&r.runningHandlers) == 0 {
return errors.New("no handlers")
}
if err := validatedLookupAddr(addr); err != nil {
return err
}
atomic.StoreInt32(&r.connectedFlag, 1)
r.mtx.Lock()
for _, x := range r.lookupdHTTPAddrs {
if x == addr {
r.mtx.Unlock()
return nil
}
}
r.lookupdHTTPAddrs = append(r.lookupdHTTPAddrs, addr)
numLookupd := len(r.lookupdHTTPAddrs)
r.mtx.Unlock()
// if this is the first one, kick off the go loop
if numLookupd == 1 {
r.queryLookupd()
r.wg.Add(1)
go r.lookupdLoop()
}
return nil
}
// ConnectToNSQLookupds adds multiple nsqlookupd address to the list for this Consumer instance.
//
// If adding the first address it initiates an HTTP request to discover nsqd
// producers for the configured topic.
//
// A goroutine is spawned to handle continual polling.
func (r *Consumer) ConnectToNSQLookupds(addresses []string) error {
for _, addr := range addresses {
err := r.ConnectToNSQLookupd(addr)
if err != nil {
return err
}
}
return nil
}
func validatedLookupAddr(addr string) error {
if strings.Contains(addr, "/") {
_, err := url.Parse(addr)
if err != nil {
return err
}
return nil
}
if !strings.Contains(addr, ":") {
return errors.New("missing port")
}
return nil
}
// poll all known lookup servers every LookupdPollInterval
func (r *Consumer) lookupdLoop() {
// add some jitter so that multiple consumers discovering the same topic,
// when restarted at the same time, dont all connect at once.
jitter := time.Duration(int64(r.rng.Float64() *
r.config.LookupdPollJitter * float64(r.config.LookupdPollInterval)))
var ticker *time.Ticker
select {
case <-time.After(jitter):
case <-r.exitChan:
goto exit
}
ticker = time.NewTicker(r.config.LookupdPollInterval)
for {
select {
case <-ticker.C:
r.queryLookupd()
case <-r.lookupdRecheckChan:
r.queryLookupd()
case <-r.exitChan:
goto exit
}
}
exit:
if ticker != nil {
ticker.Stop()
}
r.log(LogLevelInfo, "exiting lookupdLoop")
r.wg.Done()
}
// return the next lookupd endpoint to query
// keeping track of which one was last used
func (r *Consumer) nextLookupdEndpoint() string {
r.mtx.RLock()
if r.lookupdQueryIndex >= len(r.lookupdHTTPAddrs) {
r.lookupdQueryIndex = 0
}
addr := r.lookupdHTTPAddrs[r.lookupdQueryIndex]
num := len(r.lookupdHTTPAddrs)
r.mtx.RUnlock()
r.lookupdQueryIndex = (r.lookupdQueryIndex + 1) % num
urlString := addr
if !strings.Contains(urlString, "://") {
urlString = "http://" + addr
}
u, err := url.Parse(urlString)
if err != nil {
panic(err)
}
if u.Path == "/" || u.Path == "" {
u.Path = "/lookup"
}
v, err := url.ParseQuery(u.RawQuery)
v.Add("topic", r.topic)
u.RawQuery = v.Encode()
return u.String()
}
type lookupResp struct {
Channels []string `json:"channels"`
Producers []*peerInfo `json:"producers"`
Timestamp int64 `json:"timestamp"`
}
type peerInfo struct {
RemoteAddress string `json:"remote_address"`
Hostname string `json:"hostname"`
BroadcastAddress string `json:"broadcast_address"`
TCPPort int `json:"tcp_port"`
HTTPPort int `json:"http_port"`
Version string `json:"version"`
}
// make an HTTP req to one of the configured nsqlookupd instances to discover
// which nsqd's provide the topic we are consuming.
//
// initiate a connection to any new producers that are identified.
func (r *Consumer) queryLookupd() {
endpoint := r.nextLookupdEndpoint()
r.log(LogLevelInfo, "querying nsqlookupd %s", endpoint)
var data lookupResp
err := apiRequestNegotiateV1("GET", endpoint, nil, &data)
if err != nil {
r.log(LogLevelError, "error querying nsqlookupd (%s) - %s", endpoint, err)
return
}
var nsqdAddrs []string
for _, producer := range data.Producers {
broadcastAddress := producer.BroadcastAddress
port := producer.TCPPort
joined := net.JoinHostPort(broadcastAddress, strconv.Itoa(port))
nsqdAddrs = append(nsqdAddrs, joined)
}
// apply filter
if discoveryFilter, ok := r.behaviorDelegate.(DiscoveryFilter); ok {
nsqdAddrs = discoveryFilter.Filter(nsqdAddrs)
}
for _, addr := range nsqdAddrs {
err = r.ConnectToNSQD(addr)
if err != nil && err != ErrAlreadyConnected {
r.log(LogLevelError, "(%s) error connecting to nsqd - %s", addr, err)
continue
}
}
}
// ConnectToNSQDs takes multiple nsqd addresses to connect directly to.
//
// It is recommended to use ConnectToNSQLookupd so that topics are discovered
// automatically. This method is useful when you want to connect to local instance.
func (r *Consumer) ConnectToNSQDs(addresses []string) error {
for _, addr := range addresses {
err := r.ConnectToNSQD(addr)
if err != nil {
return err
}
}
return nil
}
// ConnectToNSQD takes a nsqd address to connect directly to.
//
// It is recommended to use ConnectToNSQLookupd so that topics are discovered
// automatically. This method is useful when you want to connect to a single, local,
// instance.
func (r *Consumer) ConnectToNSQD(addr string) error {
if atomic.LoadInt32(&r.stopFlag) == 1 {
return errors.New("consumer stopped")
}
if atomic.LoadInt32(&r.runningHandlers) == 0 {
return errors.New("no handlers")
}
atomic.StoreInt32(&r.connectedFlag, 1)
logger, logLvl := r.getLogger()
conn := NewConn(addr, &r.config, &consumerConnDelegate{r})
conn.SetLogger(logger, logLvl,
fmt.Sprintf("%3d [%s/%s] (%%s)", r.id, r.topic, r.channel))
r.mtx.Lock()
_, pendingOk := r.pendingConnections[addr]
_, ok := r.connections[addr]
if ok || pendingOk {
r.mtx.Unlock()
return ErrAlreadyConnected
}
r.pendingConnections[addr] = conn
if idx := indexOf(addr, r.nsqdTCPAddrs); idx == -1 {
r.nsqdTCPAddrs = append(r.nsqdTCPAddrs, addr)
}
r.mtx.Unlock()
r.log(LogLevelInfo, "(%s) connecting to nsqd", addr)
cleanupConnection := func() {
r.mtx.Lock()
delete(r.pendingConnections, addr)
r.mtx.Unlock()
conn.Close()
}
resp, err := conn.Connect()
if err != nil {
cleanupConnection()
return err
}
if resp != nil {
if resp.MaxRdyCount < int64(r.getMaxInFlight()) {
r.log(LogLevelWarning,
"(%s) max RDY count %d < consumer max in flight %d, truncation possible",
conn.String(), resp.MaxRdyCount, r.getMaxInFlight())
}
}
cmd := Subscribe(r.topic, r.channel)
err = conn.WriteCommand(cmd)
if err != nil {
cleanupConnection()
return fmt.Errorf("[%s] failed to subscribe to %s:%s - %s",
conn, r.topic, r.channel, err.Error())
}
r.mtx.Lock()
delete(r.pendingConnections, addr)
r.connections[addr] = conn
r.mtx.Unlock()
// pre-emptive signal to existing connections to lower their RDY count
for _, c := range r.conns() {
r.maybeUpdateRDY(c)
}
return nil
}
func indexOf(n string, h []string) int {
for i, a := range h {
if n == a {
return i
}
}
return -1
}
// DisconnectFromNSQD closes the connection to and removes the specified
// `nsqd` address from the list
func (r *Consumer) DisconnectFromNSQD(addr string) error {
r.mtx.Lock()
defer r.mtx.Unlock()
idx := indexOf(addr, r.nsqdTCPAddrs)
if idx == -1 {
return ErrNotConnected
}
// slice delete
r.nsqdTCPAddrs = append(r.nsqdTCPAddrs[:idx], r.nsqdTCPAddrs[idx+1:]...)
pendingConn, pendingOk := r.pendingConnections[addr]
conn, ok := r.connections[addr]
if ok {
conn.Close()
} else if pendingOk {
pendingConn.Close()
}
return nil
}
// DisconnectFromNSQLookupd removes the specified `nsqlookupd` address
// from the list used for periodic discovery.
func (r *Consumer) DisconnectFromNSQLookupd(addr string) error {
r.mtx.Lock()
defer r.mtx.Unlock()
idx := indexOf(addr, r.lookupdHTTPAddrs)
if idx == -1 {
return ErrNotConnected
}
if len(r.lookupdHTTPAddrs) == 1 {
return fmt.Errorf("cannot disconnect from only remaining nsqlookupd HTTP address %s", addr)
}
r.lookupdHTTPAddrs = append(r.lookupdHTTPAddrs[:idx], r.lookupdHTTPAddrs[idx+1:]...)
return nil
}
func (r *Consumer) onConnMessage(c *Conn, msg *Message) {
atomic.AddInt64(&r.totalRdyCount, -1)
atomic.AddUint64(&r.messagesReceived, 1)
r.incomingMessages <- msg
r.maybeUpdateRDY(c)
}
func (r *Consumer) onConnMessageFinished(c *Conn, msg *Message) {
atomic.AddUint64(&r.messagesFinished, 1)
}
func (r *Consumer) onConnMessageRequeued(c *Conn, msg *Message) {
atomic.AddUint64(&r.messagesRequeued, 1)
}
func (r *Consumer) onConnBackoff(c *Conn) {
r.startStopContinueBackoff(c, backoffFlag)
}
func (r *Consumer) onConnContinue(c *Conn) {
r.startStopContinueBackoff(c, continueFlag)
}
func (r *Consumer) onConnResume(c *Conn) {
r.startStopContinueBackoff(c, resumeFlag)
}
func (r *Consumer) onConnResponse(c *Conn, data []byte) {
switch {
case bytes.Equal(data, []byte("CLOSE_WAIT")):
// server is ready for us to close (it ack'd our StartClose)
// we can assume we will not receive any more messages over this channel
// (but we can still write back responses)
r.log(LogLevelInfo, "(%s) received CLOSE_WAIT from nsqd", c.String())
c.Close()
}
}
func (r *Consumer) onConnError(c *Conn, data []byte) {}
func (r *Consumer) onConnHeartbeat(c *Conn) {}
func (r *Consumer) onConnIOError(c *Conn, err error) {
c.Close()
}
func (r *Consumer) onConnClose(c *Conn) {
var hasRDYRetryTimer bool
// remove this connections RDY count from the consumer's total
rdyCount := c.RDY()
atomic.AddInt64(&r.totalRdyCount, -rdyCount)
r.rdyRetryMtx.Lock()
if timer, ok := r.rdyRetryTimers[c.String()]; ok {
// stop any pending retry of an old RDY update
timer.Stop()
delete(r.rdyRetryTimers, c.String())
hasRDYRetryTimer = true
}
r.rdyRetryMtx.Unlock()
r.mtx.Lock()
delete(r.connections, c.String())
left := len(r.connections)
r.mtx.Unlock()
r.log(LogLevelWarning, "there are %d connections left alive", left)
if (hasRDYRetryTimer || rdyCount > 0) &&
(int32(left) == r.getMaxInFlight() || r.inBackoff()) {
// we're toggling out of (normal) redistribution cases and this conn
// had a RDY count...
//
// trigger RDY redistribution to make sure this RDY is moved
// to a new connection
atomic.StoreInt32(&r.needRDYRedistributed, 1)
}
// we were the last one (and stopping)
if atomic.LoadInt32(&r.stopFlag) == 1 {
if left == 0 {
r.stopHandlers()
}
return
}
r.mtx.RLock()
numLookupd := len(r.lookupdHTTPAddrs)
reconnect := indexOf(c.String(), r.nsqdTCPAddrs) >= 0
r.mtx.RUnlock()
if numLookupd > 0 {
// trigger a poll of the lookupd
select {
case r.lookupdRecheckChan <- 1:
default:
}
} else if reconnect {
// there are no lookupd and we still have this nsqd TCP address in our list...
// try to reconnect after a bit
go func(addr string) {
for {
r.log(LogLevelInfo, "(%s) re-connecting in %s", addr, r.config.LookupdPollInterval)
time.Sleep(r.config.LookupdPollInterval)
if atomic.LoadInt32(&r.stopFlag) == 1 {
break
}
r.mtx.RLock()
reconnect := indexOf(addr, r.nsqdTCPAddrs) >= 0
r.mtx.RUnlock()
if !reconnect {
r.log(LogLevelWarning, "(%s) skipped reconnect after removal...", addr)
return
}
err := r.ConnectToNSQD(addr)
if err != nil && err != ErrAlreadyConnected {
r.log(LogLevelError, "(%s) error connecting to nsqd - %s", addr, err)
continue
}
break
}
}(c.String())
}
}
func (r *Consumer) startStopContinueBackoff(conn *Conn, signal backoffSignal) {
// prevent many async failures/successes from immediately resulting in
// max backoff/normal rate (by ensuring that we dont continually incr/decr
// the counter during a backoff period)
r.backoffMtx.Lock()
if r.inBackoffTimeout() {
r.backoffMtx.Unlock()
return
}
defer r.backoffMtx.Unlock()
// update backoff state
backoffUpdated := false
backoffCounter := atomic.LoadInt32(&r.backoffCounter)
switch signal {
case resumeFlag:
if backoffCounter > 0 {
backoffCounter--
backoffUpdated = true
}
case backoffFlag:
nextBackoff := r.config.BackoffStrategy.Calculate(int(backoffCounter) + 1)
if nextBackoff <= r.config.MaxBackoffDuration {
backoffCounter++
backoffUpdated = true
}
}
atomic.StoreInt32(&r.backoffCounter, backoffCounter)
if r.backoffCounter == 0 && backoffUpdated {
// exit backoff
count := r.perConnMaxInFlight()
r.log(LogLevelWarning, "exiting backoff, returning all to RDY %d", count)
for _, c := range r.conns() {
r.updateRDY(c, count)
}
} else if r.backoffCounter > 0 {
// start or continue backoff
backoffDuration := r.config.BackoffStrategy.Calculate(int(backoffCounter))
if backoffDuration > r.config.MaxBackoffDuration {
backoffDuration = r.config.MaxBackoffDuration
}
r.log(LogLevelWarning, "backing off for %.04f seconds (backoff level %d), setting all to RDY 0",
backoffDuration.Seconds(), backoffCounter)
// send RDY 0 immediately (to *all* connections)
for _, c := range r.conns() {
r.updateRDY(c, 0)
}
r.backoff(backoffDuration)
}
}
func (r *Consumer) backoff(d time.Duration) {
atomic.StoreInt64(&r.backoffDuration, d.Nanoseconds())
time.AfterFunc(d, r.resume)
}
func (r *Consumer) resume() {
if atomic.LoadInt32(&r.stopFlag) == 1 {
atomic.StoreInt64(&r.backoffDuration, 0)
return
}
// pick a random connection to test the waters
conns := r.conns()
if len(conns) == 0 {
r.log(LogLevelWarning, "no connection available to resume")
r.log(LogLevelWarning, "backing off for %.04f seconds", 1)
r.backoff(time.Second)
return
}
idx := r.rng.Intn(len(conns))
choice := conns[idx]
r.log(LogLevelWarning,
"(%s) backoff timeout expired, sending RDY 1",
choice.String())
// while in backoff only ever let 1 message at a time through
err := r.updateRDY(choice, 1)
if err != nil {
r.log(LogLevelWarning, "(%s) error resuming RDY 1 - %s", choice.String(), err)
r.log(LogLevelWarning, "backing off for %.04f seconds", 1)
r.backoff(time.Second)
return
}
atomic.StoreInt64(&r.backoffDuration, 0)
}
func (r *Consumer) inBackoff() bool {
return atomic.LoadInt32(&r.backoffCounter) > 0
}
func (r *Consumer) inBackoffTimeout() bool {
return atomic.LoadInt64(&r.backoffDuration) > 0
}
func (r *Consumer) maybeUpdateRDY(conn *Conn) {
inBackoff := r.inBackoff()
inBackoffTimeout := r.inBackoffTimeout()
if inBackoff || inBackoffTimeout {
r.log(LogLevelDebug, "(%s) skip sending RDY inBackoff:%v || inBackoffTimeout:%v",
conn, inBackoff, inBackoffTimeout)
return
}
remain := conn.RDY()
lastRdyCount := conn.LastRDY()
count := r.perConnMaxInFlight()
// refill when at 1, or at 25%, or if connections have changed and we're imbalanced
if remain <= 1 || remain < (lastRdyCount/4) || (count > 0 && count < remain) {
r.log(LogLevelDebug, "(%s) sending RDY %d (%d remain from last RDY %d)",
conn, count, remain, lastRdyCount)
r.updateRDY(conn, count)
} else {
r.log(LogLevelDebug, "(%s) skip sending RDY %d (%d remain out of last RDY %d)",
conn, count, remain, lastRdyCount)
}
}
func (r *Consumer) rdyLoop() {
redistributeTicker := time.NewTicker(r.config.RDYRedistributeInterval)
for {
select {
case <-redistributeTicker.C:
r.redistributeRDY()
case <-r.exitChan:
goto exit
}
}
exit:
redistributeTicker.Stop()
r.log(LogLevelInfo, "rdyLoop exiting")
r.wg.Done()
}
func (r *Consumer) updateRDY(c *Conn, count int64) error {
if c.IsClosing() {
return ErrClosing
}
// never exceed the nsqd's configured max RDY count
if count > c.MaxRDY() {
count = c.MaxRDY()
}
// stop any pending retry of an old RDY update
r.rdyRetryMtx.Lock()
if timer, ok := r.rdyRetryTimers[c.String()]; ok {
timer.Stop()
delete(r.rdyRetryTimers, c.String())
}
r.rdyRetryMtx.Unlock()
// never exceed our global max in flight. truncate if possible.
// this could help a new connection get partial max-in-flight
rdyCount := c.RDY()
maxPossibleRdy := int64(r.getMaxInFlight()) - atomic.LoadInt64(&r.totalRdyCount) + rdyCount
if maxPossibleRdy > 0 && maxPossibleRdy < count {
count = maxPossibleRdy
}
if maxPossibleRdy <= 0 && count > 0 {
if rdyCount == 0 {
// we wanted to exit a zero RDY count but we couldn't send it...
// in order to prevent eternal starvation we reschedule this attempt
// (if any other RDY update succeeds this timer will be stopped)
r.rdyRetryMtx.Lock()
r.rdyRetryTimers[c.String()] = time.AfterFunc(5*time.Second,
func() {
r.updateRDY(c, count)
})
r.rdyRetryMtx.Unlock()
}
return ErrOverMaxInFlight
}
return r.sendRDY(c, count)
}
func (r *Consumer) sendRDY(c *Conn, count int64) error {
if count == 0 && c.LastRDY() == 0 {
// no need to send. It's already that RDY count
return nil
}
atomic.AddInt64(&r.totalRdyCount, -c.RDY()+count)
c.SetRDY(count)
err := c.WriteCommand(Ready(int(count)))
if err != nil {
r.log(LogLevelError, "(%s) error sending RDY %d - %s", c.String(), count, err)
return err
}
return nil
}
func (r *Consumer) redistributeRDY() {
if r.inBackoffTimeout() {
return
}
// if an external heuristic set needRDYRedistributed we want to wait
// until we can actually redistribute to proceed
conns := r.conns()
if len(conns) == 0 {
return
}
maxInFlight := r.getMaxInFlight()
if len(conns) > int(maxInFlight) {
r.log(LogLevelDebug, "redistributing RDY state (%d conns > %d max_in_flight)",
len(conns), maxInFlight)
atomic.StoreInt32(&r.needRDYRedistributed, 1)
}
if r.inBackoff() && len(conns) > 1 {
r.log(LogLevelDebug, "redistributing RDY state (in backoff and %d conns > 1)", len(conns))
atomic.StoreInt32(&r.needRDYRedistributed, 1)
}
if !atomic.CompareAndSwapInt32(&r.needRDYRedistributed, 1, 0) {
return
}
possibleConns := make([]*Conn, 0, len(conns))
for _, c := range conns {
lastMsgDuration := time.Now().Sub(c.LastMessageTime())
rdyCount := c.RDY()
r.log(LogLevelDebug, "(%s) rdy: %d (last message received %s)",
c.String(), rdyCount, lastMsgDuration)
if rdyCount > 0 && lastMsgDuration > r.config.LowRdyIdleTimeout {
r.log(LogLevelDebug, "(%s) idle connection, giving up RDY", c.String())
r.updateRDY(c, 0)
}
possibleConns = append(possibleConns, c)
}
availableMaxInFlight := int64(maxInFlight) - atomic.LoadInt64(&r.totalRdyCount)
if r.inBackoff() {
availableMaxInFlight = 1 - atomic.LoadInt64(&r.totalRdyCount)
}
for len(possibleConns) > 0 && availableMaxInFlight > 0 {
availableMaxInFlight--
i := r.rng.Int() % len(possibleConns)
c := possibleConns[i]
// delete
possibleConns = append(possibleConns[:i], possibleConns[i+1:]...)
r.log(LogLevelDebug, "(%s) redistributing RDY", c.String())
r.updateRDY(c, 1)
}
}
// Stop will initiate a graceful stop of the Consumer (permanent)
//
// NOTE: receive on StopChan to block until this process completes
func (r *Consumer) Stop() {
if !atomic.CompareAndSwapInt32(&r.stopFlag, 0, 1) {
return
}
r.log(LogLevelInfo, "stopping...")
if len(r.conns()) == 0 {
r.stopHandlers()
} else {
for _, c := range r.conns() {
err := c.WriteCommand(StartClose())
if err != nil {
r.log(LogLevelError, "(%s) error sending CLS - %s", c.String(), err)
}
}
time.AfterFunc(time.Second*30, func() {
// if we've waited this long handlers are blocked on processing messages
// so we can't just stopHandlers (if any adtl. messages were pending processing
// we would cause a panic on channel close)
//
// instead, we just bypass handler closing and skip to the final exit
r.exit()
})
}
}
func (r *Consumer) stopHandlers() {
r.stopHandler.Do(func() {
r.log(LogLevelInfo, "stopping handlers")
close(r.incomingMessages)
})
}
// AddHandler sets the Handler for messages received by this Consumer. This can be called
// multiple times to add additional handlers. Handler will have a 1:1 ratio to message handling goroutines.
//
// This panics if called after connecting to NSQD or NSQ Lookupd
//
// (see Handler or HandlerFunc for details on implementing this interface)
func (r *Consumer) AddHandler(handler Handler) {
r.AddConcurrentHandlers(handler, 1)
}
// AddConcurrentHandlers sets the Handler for messages received by this Consumer. It
// takes a second argument which indicates the number of goroutines to spawn for
// message handling.
//
// This panics if called after connecting to NSQD or NSQ Lookupd
//
// (see Handler or HandlerFunc for details on implementing this interface)
func (r *Consumer) AddConcurrentHandlers(handler Handler, concurrency int) {
if atomic.LoadInt32(&r.connectedFlag) == 1 {
panic("already connected")
}
atomic.AddInt32(&r.runningHandlers, int32(concurrency))
for i := 0; i < concurrency; i++ {
go r.handlerLoop(handler)
}
}
func (r *Consumer) handlerLoop(handler Handler) {
r.log(LogLevelDebug, "starting Handler")
for {
message, ok := <-r.incomingMessages
if !ok {
goto exit
}
if r.shouldFailMessage(message, handler) {
message.Finish()
continue
}
err := handler.HandleMessage(message)
if err != nil {
r.log(LogLevelError, "Handler returned error (%s) for msg %s", err, message.ID)
if !message.IsAutoResponseDisabled() {
message.Requeue(-1)
}
continue
}
if !message.IsAutoResponseDisabled() {
message.Finish()
}
}
exit:
r.log(LogLevelDebug, "stopping Handler")
if atomic.AddInt32(&r.runningHandlers, -1) == 0 {
r.exit()
}
}
func (r *Consumer) shouldFailMessage(message *Message, handler interface{}) bool {
// message passed the max number of attempts
if r.config.MaxAttempts > 0 && message.Attempts > r.config.MaxAttempts {
r.log(LogLevelWarning, "msg %s attempted %d times, giving up",
message.ID, message.Attempts)
logger, ok := handler.(FailedMessageLogger)
if ok {
logger.LogFailedMessage(message)
}
return true
}
return false
}
func (r *Consumer) exit() {
r.exitHandler.Do(func() {
close(r.exitChan)
r.wg.Wait()
close(r.StopChan)
})
}
func (r *Consumer) log(lvl LogLevel, line string, args ...interface{}) {
logger, logLvl := r.getLogger()
if logger == nil {
return
}
if logLvl > lvl {
return
}
logger.Output(2, fmt.Sprintf("%-4s %3d [%s/%s] %s",
lvl, r.id, r.topic, r.channel,
fmt.Sprintf(line, args...)))
}
================================================
FILE: vendor/github.com/bitly/go-nsq/consumer_test.go
================================================
package nsq
import (
"bytes"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"net"
"net/http"
"os"
"strconv"
"strings"
"testing"
"time"
)
type MyTestHandler struct {
t *testing.T
q *Consumer
messagesSent int
messagesReceived int
messagesFailed int
}
var nullLogger = log.New(ioutil.Discard, "", log.LstdFlags)
func (h *MyTestHandler) LogFailedMessage(message *Message) {
h.messagesFailed++
h.q.Stop()
}
func (h *MyTestHandler) HandleMessage(message *Message) error {
if string(message.Body) == "TOBEFAILED" {
h.messagesReceived++
return errors.New("fail this message")
}
data := struct {
Msg string
}{}
err := json.Unmarshal(message.Body, &data)
if err != nil {
return err
}
msg := data.Msg
if msg != "single" && msg != "double" {
h.t.Error("message 'action' was not correct: ", msg, data)
}
h.messagesReceived++
return nil
}
func SendMessage(t *testing.T, port int, topic string, method string, body []byte) {
httpclient := &http.Client{}
endpoint := fmt.Sprintf("http://127.0.0.1:%d/%s?topic=%s", port, method, topic)
req, err := http.NewRequest("POST", endpoint, bytes.NewBuffer(body))
resp, err := httpclient.Do(req)
if err != nil {
t.Fatalf(err.Error())
return
}
resp.Body.Close()
}
func TestConsumer(t *testing.T) {
consumerTest(t, nil)
}
func TestConsumerTLS(t *testing.T) {
consumerTest(t, func(c *Config) {
c.TlsV1 = true
c.TlsConfig = &tls.Config{
InsecureSkipVerify: true,
}
})
}
func TestConsumerDeflate(t *testing.T) {
consumerTest(t, func(c *Config) {
c.Deflate = true
})
}
func TestConsumerSnappy(t *testing.T) {
consumerTest(t, func(c *Config) {
c.Snappy = true
})
}
func TestConsumerTLSDeflate(t *testing.T) {
consumerTest(t, func(c *Config) {
c.TlsV1 = true
c.TlsConfig = &tls.Config{
InsecureSkipVerify: true,
}
c.Deflate = true
})
}
func TestConsumerTLSSnappy(t *testing.T) {
consumerTest(t, func(c *Config) {
c.TlsV1 = true
c.TlsConfig = &tls.Config{
InsecureSkipVerify: true,
}
c.Snappy = true
})
}
func TestConsumerTLSClientCert(t *testing.T) {
envDl := os.Getenv("NSQ_DOWNLOAD")
if strings.HasPrefix(envDl, "nsq-0.2.24") || strings.HasPrefix(envDl, "nsq-0.2.27") {
t.Log("skipping due to older nsqd")
return
}
cert, _ := tls.LoadX509KeyPair("./test/client.pem", "./test/client.key")
consumerTest(t, func(c *Config) {
c.TlsV1 = true
c.TlsConfig = &tls.Config{
Certificates: []tls.Certificate{cert},
InsecureSkipVerify: true,
}
})
}
func TestConsumerTLSClientCertViaSet(t *testing.T) {
envDl := os.Getenv("NSQ_DOWNLOAD")
if strings.HasPrefix(envDl, "nsq-0.2.24") || strings.HasPrefix(envDl, "nsq-0.2.27") {
t.Log("skipping due to older nsqd")
return
}
consumerTest(t, func(c *Config) {
c.Set("tls_v1", true)
c.Set("tls_cert", "./test/client.pem")
c.Set("tls_key", "./test/client.key")
c.Set("tls_insecure_skip_verify", true)
})
}
func consumerTest(t *testing.T, cb func(c *Config)) {
config := NewConfig()
laddr := "127.0.0.1"
// so that the test can simulate binding consumer to specified address
config.LocalAddr, _ = net.ResolveTCPAddr("tcp", laddr+":0")
// so that the test can simulate reaching max requeues and a call to LogFailedMessage
config.DefaultRequeueDelay = 0
// so that the test wont timeout from backing off
config.MaxBackoffDuration = time.Millisecond * 50
if cb != nil {
cb(config)
}
topicName := "rdr_test"
if config.Deflate {
topicName = topicName + "_deflate"
} else if config.Snappy {
topicName = topicName + "_snappy"
}
if config.TlsV1 {
topicName = topicName + "_tls"
}
topicName = topicName + strconv.Itoa(int(time.Now().Unix()))
q, _ := NewConsumer(topicName, "ch", config)
// q.SetLogger(nullLogger, LogLevelInfo)
h := &MyTestHandler{
t: t,
q: q,
}
q.AddHandler(h)
SendMessage(t, 4151, topicName, "put", []byte(`{"msg":"single"}`))
SendMessage(t, 4151, topicName, "mput", []byte("{\"msg\":\"double\"}\n{\"msg\":\"double\"}"))
SendMessage(t, 4151, topicName, "put", []byte("TOBEFAILED"))
h.messagesSent = 4
addr := "127.0.0.1:4150"
err := q.ConnectToNSQD(addr)
if err != nil {
t.Fatal(err)
}
stats := q.Stats()
if stats.Connections == 0 {
t.Fatal("stats report 0 connections (should be > 0)")
}
err = q.ConnectToNSQD(addr)
if err == nil {
t.Fatal("should not be able to connect to the same NSQ twice")
}
conn := q.conns()[0]
if !strings.HasPrefix(conn.conn.LocalAddr().String(), laddr) {
t.Fatal("connection should be bound to the specified address:", conn.conn.LocalAddr())
}
err = q.DisconnectFromNSQD("1.2.3.4:4150")
if err == nil {
t.Fatal("should not be able to disconnect from an unknown nsqd")
}
err = q.ConnectToNSQD("1.2.3.4:4150")
if err == nil {
t.Fatal("should not be able to connect to non-existent nsqd")
}
err = q.DisconnectFromNSQD("1.2.3.4:4150")
if err != nil {
t.Fatal("should be able to disconnect from an nsqd - " + err.Error())
}
<-q.StopChan
stats = q.Stats()
if stats.Connections != 0 {
t.Fatalf("stats report %d active connections (should be 0)", stats.Connections)
}
stats = q.Stats()
if stats.MessagesReceived != uint64(h.messagesReceived+h.messagesFailed) {
t.Fatalf("stats report %d messages received (should be %d)",
stats.MessagesReceived,
h.messagesReceived+h.messagesFailed)
}
if h.messagesReceived != 8 || h.messagesSent != 4 {
t.Fatalf("end of test. should have handled a diff number of messages (got %d, sent %d)", h.messagesReceived, h.messagesSent)
}
if h.messagesFailed != 1 {
t.Fatal("failed message not done")
}
}
================================================
FILE: vendor/github.com/bitly/go-nsq/delegates.go
================================================
package nsq
import "time"
type logger interface {
Output(calldepth int, s string) error
}
// LogLevel specifies the severity of a given log message
type LogLevel int
// Log levels
const (
LogLevelDebug LogLevel = iota
LogLevelInfo
LogLevelWarning
LogLevelError
)
// String returns the string form for a given LogLevel
func (lvl LogLevel) String() string {
switch lvl {
case LogLevelInfo:
return "INF"
case LogLevelWarning:
return "WRN"
case LogLevelError:
return "ERR"
}
return "DBG"
}
// MessageDelegate is an interface of methods that are used as
// callbacks in Message
type MessageDelegate interface {
// OnFinish is called when the Finish() method
// is triggered on the Message
OnFinish(*Message)
// OnRequeue is called when the Requeue() method
// is triggered on the Message
OnRequeue(m *Message, delay time.Duration, backoff bool)
// OnTouch is called when the Touch() method
// is triggered on the Message
OnTouch(*Message)
}
type connMessageDelegate struct {
c *Conn
}
func (d *connMessageDelegate) OnFinish(m *Message) { d.c.onMessageFinish(m) }
func (d *connMessageDelegate) OnRequeue(m *Message, t time.Duration, b bool) {
d.c.onMessageRequeue(m, t, b)
}
func (d *connMessageDelegate) OnTouch(m *Message) { d.c.onMessageTouch(m) }
// ConnDelegate is an interface of methods that are used as
// callbacks in Conn
type ConnDelegate interface {
// OnResponse is called when the connection
// receives a FrameTypeResponse from nsqd
OnResponse(*Conn, []byte)
// OnError is called when the connection
// receives a FrameTypeError from nsqd
OnError(*Conn, []byte)
// OnMessage is called when the connection
// receives a FrameTypeMessage from nsqd
OnMessage(*Conn, *Message)
// OnMessageFinished is called when the connection
// handles a FIN command from a message handler
OnMessageFinished(*Conn, *Message)
// OnMessageRequeued is called when the connection
// handles a REQ command from a message handler
OnMessageRequeued(*Conn, *Message)
// OnBackoff is called when the connection triggers a backoff state
OnBackoff(*Conn)
// OnContinue is called when the connection finishes a message without adjusting backoff state
OnContinue(*Conn)
// OnResume is called when the connection triggers a resume state
OnResume(*Conn)
// OnIOError is called when the connection experiences
// a low-level TCP transport error
OnIOError(*Conn, error)
// OnHeartbeat is called when the connection
// receives a heartbeat from nsqd
OnHeartbeat(*Conn)
// OnClose is called when the connection
// closes, after all cleanup
OnClose(*Conn)
}
// keeps the exported Consumer struct clean of the exported methods
// required to implement the ConnDelegate interface
type consumerConnDelegate struct {
r *Consumer
}
func (d *consumerConnDelegate) OnResponse(c *Conn, data []byte) { d.r.onConnResponse(c, data) }
func (d *consumerConnDelegate) OnError(c *Conn, data []byte) { d.r.onConnError(c, data) }
func (d *consumerConnDelegate) OnMessage(c *Conn, m *Message) { d.r.onConnMessage(c, m) }
func (d *consumerConnDelegate) OnMessageFinished(c *Conn, m *Message) { d.r.onConnMessageFinished(c, m) }
func (d *consumerConnDelegate) OnMessageRequeued(c *Conn, m *Message) { d.r.onConnMessageRequeued(c, m) }
func (d *consumerConnDelegate) OnBackoff(c *Conn) { d.r.onConnBackoff(c) }
func (d *consumerConnDelegate) OnContinue(c *Conn) { d.r.onConnContinue(c) }
func (d *consumerConnDelegate) OnResume(c *Conn) { d.r.onConnResume(c) }
func (d *consumerConnDelegate) OnIOError(c *Conn, err error) { d.r.onConnIOError(c, err) }
func (d *consumerConnDelegate) OnHeartbeat(c *Conn) { d.r.onConnHeartbeat(c) }
func (d *consumerConnDelegate) OnClose(c *Conn) { d.r.onConnClose(c) }
// keeps the exported Producer struct clean of the exported methods
// required to implement the ConnDelegate interface
type producerConnDelegate struct {
w *Producer
}
func (d *producerConnDelegate) OnResponse(c *Conn, data []byte) { d.w.onConnResponse(c, data) }
func (d *producerConnDelegate) OnError(c *Conn, data []byte) { d.w.onConnError(c, data) }
func (d *producerConnDelegate) OnMessage(c *Conn, m *Message) {}
func (d *producerConnDelegate) OnMessageFinished(c *Conn, m *Message) {}
func (d *producerConnDelegate) OnMessageRequeued(c *Conn, m *Message) {}
func (d *producerConnDelegate) OnBackoff(c *Conn) {}
func (d *producerConnDelegate) OnContinue(c *Conn) {}
func (d *producerConnDelegate) OnResume(c *Conn) {}
func (d *producerConnDelegate) OnIOError(c *Conn, err error) { d.w.onConnIOError(c, err) }
func (d *producerConnDelegate) OnHeartbeat(c *Conn) { d.w.onConnHeartbeat(c) }
func (d *producerConnDelegate) OnClose(c *Conn) { d.w.onConnClose(c) }
================================================
FILE: vendor/github.com/bitly/go-nsq/errors.go
================================================
package nsq
import (
"errors"
"fmt"
)
// ErrNotConnected is returned when a publish command is made
// against a Producer that is not connected
var ErrNotConnected = errors.New("not connected")
// ErrStopped is returned when a publish command is
// made against a Producer that has been stopped
var ErrStopped = errors.New("stopped")
// ErrClosing is returned when a connection is closing
var ErrClosing = errors.New("closing")
// ErrAlreadyConnected is returned from ConnectToNSQD when already connected
var ErrAlreadyConnected = errors.New("already connected")
// ErrOverMaxInFlight is returned from Consumer if over max-in-flight
var ErrOverMaxInFlight = errors.New("over configure max-inflight")
// ErrIdentify is returned from Conn as part of the IDENTIFY handshake
type ErrIdentify struct {
Reason string
}
// Error returns a stringified error
func (e ErrIdentify) Error() string {
return fmt.Sprintf("failed to IDENTIFY - %s", e.Reason)
}
// ErrProtocol is returned from Producer when encountering
// an NSQ protocol level error
type ErrProtocol struct {
Reason string
}
// Error returns a stringified error
func (e ErrProtocol) Error() string {
return e.Reason
}
================================================
FILE: vendor/github.com/bitly/go-nsq/message.go
================================================
package nsq
import (
"bytes"
"encoding/binary"
"io"
"io/ioutil"
"sync/atomic"
"time"
)
// The number of bytes for a Message.ID
const MsgIDLength = 16
// MessageID is the ASCII encoded hexadecimal message ID
type MessageID [MsgIDLength]byte
// Message is the fundamental data type containing
// the id, body, and metadata
type Message struct {
ID MessageID
Body []byte
Timestamp int64
Attempts uint16
NSQDAddress string
Delegate MessageDelegate
autoResponseDisabled int32
responded int32
}
// NewMessage creates a Message, initializes some metadata,
// and returns a pointer
func NewMessage(id MessageID, body []byte) *Message {
return &Message{
ID: id,
Body: body,
Timestamp: time.Now().UnixNano(),
}
}
// DisableAutoResponse disables the automatic response that
// would normally be sent when a handler.HandleMessage
// returns (FIN/REQ based on the error value returned).
//
// This is useful if you want to batch, buffer, or asynchronously
// respond to messages.
func (m *Message) DisableAutoResponse() {
atomic.StoreInt32(&m.autoResponseDisabled, 1)
}
// IsAutoResponseDisabled indicates whether or not this message
// will be responded to automatically
func (m *Message) IsAutoResponseDisabled() bool {
return atomic.LoadInt32(&m.autoResponseDisabled) == 1
}
// HasResponded indicates whether or not this message has been responded to
func (m *Message) HasResponded() bool {
return atomic.LoadInt32(&m.responded) == 1
}
// Finish sends a FIN command to the nsqd which
// sent this message
func (m *Message) Finish() {
if !atomic.CompareAndSwapInt32(&m.responded, 0, 1) {
return
}
m.Delegate.OnFinish(m)
}
// Touch sends a TOUCH command to the nsqd which
// sent this message
func (m *Message) Touch() {
if m.HasResponded() {
return
}
m.Delegate.OnTouch(m)
}
// Requeue sends a REQ command to the nsqd which
// sent this message, using the supplied delay.
//
// A delay of -1 will automatically calculate
// based on the number of attempts and the
// configured default_requeue_delay
func (m *Message) Requeue(delay time.Duration) {
m.doRequeue(delay, true)
}
// RequeueWithoutBackoff sends a REQ command to the nsqd which
// sent this message, using the supplied delay.
//
// Notably, using this method to respond does not trigger a backoff
// event on the configured Delegate.
func (m *Message) RequeueWithoutBackoff(delay time.Duration) {
m.doRequeue(delay, false)
}
func (m *Message) doRequeue(delay time.Duration, backoff bool) {
if !atomic.CompareAndSwapInt32(&m.responded, 0, 1) {
return
}
m.Delegate.OnRequeue(m, delay, backoff)
}
// WriteTo implements the WriterTo interface and serializes
// the message into the supplied producer.
//
// It is suggested that the target Writer is buffered to
// avoid performing many system calls.
func (m *Message) WriteTo(w io.Writer) (int64, error) {
var buf [10]byte
var total int64
binary.BigEndian.PutUint64(buf[:8], uint64(m.Timestamp))
binary.BigEndian.PutUint16(buf[8:10], uint16(m.Attempts))
n, err := w.Write(buf[:])
total += int64(n)
if err != nil {
return total, err
}
n, err = w.Write(m.ID[:])
total += int64(n)
if err != nil {
return total, err
}
n, err = w.Write(m.Body)
total += int64(n)
if err != nil {
return total, err
}
return total, nil
}
// DecodeMessage deseralizes data (as []byte) and creates a new Message
func DecodeMessage(b []byte) (*Message, error) {
var msg Message
msg.Timestamp = int64(binary.BigEndian.Uint64(b[:8]))
msg.Attempts = binary.BigEndian.Uint16(b[8:10])
buf := bytes.NewBuffer(b[10:])
_, err := io.ReadFull(buf, msg.ID[:])
if err != nil {
return nil, err
}
msg.Body, err = ioutil.ReadAll(buf)
if err != nil {
return nil, err
}
return &msg, nil
}
================================================
FILE: vendor/github.com/bitly/go-nsq/mock_test.go
================================================
package nsq
import (
"bufio"
"bytes"
"encoding/binary"
"errors"
"fmt"
"io"
"log"
"net"
"strconv"
"testing"
"time"
)
type tbLog interface {
Log(...interface{})
}
type testLogger struct {
tbLog
}
func (tl *testLogger) Output(maxdepth int, s string) error {
tl.Log(s)
return nil
}
func newTestLogger(tbl tbLog) logger {
return &testLogger{tbl}
}
type instruction struct {
delay time.Duration
frameType int32
body []byte
}
type mockNSQD struct {
script []instruction
got [][]byte
tcpAddr *net.TCPAddr
tcpListener net.Listener
exitChan chan int
}
func newMockNSQD(script []instruction, addr string) *mockNSQD {
n := &mockNSQD{
script: script,
exitChan: make(chan int),
}
tcpListener, err := net.Listen("tcp", addr)
if err != nil {
log.Fatalf("FATAL: listen (%s) failed - %s", n.tcpAddr.String(), err)
}
n.tcpListener = tcpListener
n.tcpAddr = tcpListener.Addr().(*net.TCPAddr)
go n.listen()
return n
}
func (n *mockNSQD) listen() {
log.Printf("TCP: listening on %s", n.tcpListener.Addr().String())
for {
conn, err := n.tcpListener.Accept()
if err != nil {
break
}
go n.handle(conn)
}
log.Printf("TCP: closing %s", n.tcpListener.Addr().String())
close(n.exitChan)
}
func (n *mockNSQD) handle(conn net.Conn) {
var idx int
log.Printf("TCP: new client(%s)", conn.RemoteAddr())
buf := make([]byte, 4)
_, err := io.ReadFull(conn, buf)
if err != nil {
log.Fatalf("ERROR: failed to read protocol version - %s", err)
}
readChan := make(chan []byte)
readDoneChan := make(chan int)
scriptTime := time.After(n.script[0].delay)
rdr := bufio.NewReader(conn)
go func() {
for {
line, err := rdr.ReadBytes('\n')
if err != nil {
return
}
// trim the '\n'
line = line[:len(line)-1]
readChan <- line
<-readDoneChan
}
}()
var rdyCount int
for idx < len(n.script) {
select {
case line := <-readChan:
log.Printf("mock: %s", line)
n.got = append(n.got, line)
params := bytes.Split(line, []byte(" "))
switch {
case bytes.Equal(params[0], []byte("IDENTIFY")):
l := make([]byte, 4)
_, err := io.ReadFull(rdr, l)
if err != nil {
log.Printf(err.Error())
goto exit
}
size := int32(binary.BigEndian.Uint32(l))
b := make([]byte, size)
_, err = io.ReadFull(rdr, b)
if err != nil {
log.Printf(err.Error())
goto exit
}
log.Printf("%s", b)
case bytes.Equal(params[0], []byte("RDY")):
rdy, _ := strconv.Atoi(string(params[1]))
rdyCount = rdy
case bytes.Equal(params[0], []byte("FIN")):
case bytes.Equal(params[0], []byte("REQ")):
}
readDoneChan <- 1
case <-scriptTime:
inst := n.script[idx]
if bytes.Equal(inst.body, []byte("exit")) {
goto exit
}
if inst.frameType == FrameTypeMessage {
if rdyCount == 0 {
log.Printf("!!! RDY == 0")
scriptTime = time.After(n.script[idx+1].delay)
continue
}
rdyCount--
}
_, err := conn.Write(framedResponse(inst.frameType, inst.body))
if err != nil {
log.Printf(err.Error())
goto exit
}
scriptTime = time.After(n.script[idx+1].delay)
idx++
}
}
exit:
n.tcpListener.Close()
conn.Close()
}
func framedResponse(frameType int32, data []byte) []byte {
var w bytes.Buffer
beBuf := make([]byte, 4)
size := uint32(len(data)) + 4
binary.BigEndian.PutUint32(beBuf, size)
_, err := w.Write(beBuf)
if err != nil {
return nil
}
binary.BigEndian.PutUint32(beBuf, uint32(frameType))
_, err = w.Write(beBuf)
if err != nil {
return nil
}
_, err = w.Write(data)
return w.Bytes()
}
type testHandler struct{}
func (h *testHandler) HandleMessage(message *Message) error {
switch string(message.Body) {
case "requeue":
message.Requeue(-1)
return nil
case "requeue_no_backoff_1":
if message.Attempts > 1 {
return nil
}
message.RequeueWithoutBackoff(-1)
return nil
case "bad":
return errors.New("bad")
}
return nil
}
func frameMessage(m *Message) []byte {
var b bytes.Buffer
m.WriteTo(&b)
return b.Bytes()
}
func TestConsumerBackoff(t *testing.T) {
msgIDGood := MessageID{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0', 'a', 's', 'd', 'f', 'g', 'h'}
msgGood := NewMessage(msgIDGood, []byte("good"))
msgIDBad := MessageID{'z', 'x', 'c', 'v', 'b', '6', '7', '8', '9', '0', 'a', 's', 'd', 'f', 'g', 'h'}
msgBad := NewMessage(msgIDBad, []byte("bad"))
script := []instruction{
// SUB
instruction{0, FrameTypeResponse, []byte("OK")},
// IDENTIFY
instruction{0, FrameTypeResponse, []byte("OK")},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgGood)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgGood)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgGood)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgBad)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgBad)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgGood)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgGood)},
// needed to exit test
instruction{200 * time.Millisecond, -1, []byte("exit")},
}
addr, _ := net.ResolveTCPAddr("tcp", "127.0.0.1:0")
n := newMockNSQD(script, addr.String())
topicName := "test_consumer_commands" + strconv.Itoa(int(time.Now().Unix()))
config := NewConfig()
config.MaxInFlight = 5
config.BackoffMultiplier = 10 * time.Millisecond
q, _ := NewConsumer(topicName, "ch", config)
q.SetLogger(newTestLogger(t), LogLevelDebug)
q.AddHandler(&testHandler{})
err := q.ConnectToNSQD(n.tcpAddr.String())
if err != nil {
t.Fatalf(err.Error())
}
<-n.exitChan
for i, r := range n.got {
log.Printf("%d: %s", i, r)
}
expected := []string{
"IDENTIFY",
"SUB " + topicName + " ch",
"RDY 5",
fmt.Sprintf("FIN %s", msgIDGood),
fmt.Sprintf("FIN %s", msgIDGood),
fmt.Sprintf("FIN %s", msgIDGood),
"RDY 5",
"RDY 0",
fmt.Sprintf("REQ %s 0", msgIDBad),
"RDY 1",
"RDY 0",
fmt.Sprintf("REQ %s 0", msgIDBad),
"RDY 1",
"RDY 0",
fmt.Sprintf("FIN %s", msgIDGood),
"RDY 1",
"RDY 5",
fmt.Sprintf("FIN %s", msgIDGood),
}
if len(n.got) != len(expected) {
t.Fatalf("we got %d commands != %d expected", len(n.got), len(expected))
}
for i, r := range n.got {
if string(r) != expected[i] {
t.Fatalf("cmd %d bad %s != %s", i, r, expected[i])
}
}
}
func TestConsumerRequeueNoBackoff(t *testing.T) {
msgIDGood := MessageID{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0', 'a', 's', 'd', 'f', 'g', 'h'}
msgIDRequeue := MessageID{'r', 'e', 'q', 'v', 'b', '6', '7', '8', '9', '0', 'a', 's', 'd', 'f', 'g', 'h'}
msgIDRequeueNoBackoff := MessageID{'r', 'e', 'q', 'n', 'b', 'a', 'c', 'k', '9', '0', 'a', 's', 'd', 'f', 'g', 'h'}
msgGood := NewMessage(msgIDGood, []byte("good"))
msgRequeue := NewMessage(msgIDRequeue, []byte("requeue"))
msgRequeueNoBackoff := NewMessage(msgIDRequeueNoBackoff, []byte("requeue_no_backoff_1"))
script := []instruction{
// SUB
instruction{0, FrameTypeResponse, []byte("OK")},
// IDENTIFY
instruction{0, FrameTypeResponse, []byte("OK")},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgRequeue)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgRequeueNoBackoff)},
instruction{20 * time.Millisecond, FrameTypeMessage, frameMessage(msgGood)},
// needed to exit test
instruction{100 * time.Millisecond, -1, []byte("exit")},
}
addr, _ := net.ResolveTCPAddr("tcp", "127.0.0.1:0")
n := newMockNSQD(script, addr.String())
topicName := "test_requeue" + strconv.Itoa(int(time.Now().Unix()))
config := NewConfig()
config.MaxInFlight = 1
config.BackoffMultiplier = 10 * time.Millisecond
q, _ := NewConsumer(topicName, "ch", config)
q.SetLogger(newTestLogger(t), LogLevelDebug)
q.AddHandler(&testHandler{})
err := q.ConnectToNSQD(n.tcpAddr.String())
if err != nil {
t.Fatalf(err.Error())
}
select {
case <-n.exitChan:
log.Printf("clean exit")
case <-time.After(500 * time.Millisecond):
log.Printf("timeout")
}
for i, r := range n.got {
log.Printf("%d: %s", i, r)
}
expected := []string{
"IDENTIFY",
"SUB " + topicName + " ch",
"RDY 1",
"RDY 1",
"RDY 0",
fmt.Sprintf("REQ %s 0", msgIDRequeue),
"RDY 1",
"RDY 0",
fmt.Sprintf("REQ %s 0", msgIDRequeueNoBackoff),
"RDY 1",
"RDY 1",
fmt.Sprintf("FIN %s", msgIDGood),
}
if len(n.
gitextract_lw0vpfqr/
├── .gitignore
├── .travis.yml
├── Dockerfile
├── Godeps/
│ ├── Godeps.json
│ └── Readme
├── LICENCE
├── README.md
├── apps/
│ ├── phosphor/
│ │ └── main.go
│ └── phosphord/
│ └── main.go
├── dist.sh
├── internal/
│ ├── util/
│ │ └── stringarray.go
│ └── version/
│ └── version.go
├── phosphor/
│ ├── README.md
│ ├── context.go
│ ├── domain.go
│ ├── handler.go
│ ├── ingester.go
│ ├── marshaling.go
│ ├── memorystore.go
│ ├── options.go
│ ├── phosphor.go
│ └── store.go
├── phosphord/
│ ├── README.md
│ ├── forwarder.go
│ ├── options.go
│ ├── phosphord.go
│ ├── test/
│ │ └── test.go
│ └── transport/
│ ├── nsq.go
│ └── transport.go
├── proto/
│ ├── trace.pb.go
│ └── trace.proto
├── script/
│ └── buildprotobufs.sh
└── vendor/
├── code.google.com/
│ └── p/
│ └── snappy-go/
│ └── snappy/
│ ├── decode.go
│ ├── encode.go
│ ├── snappy.go
│ └── snappy_test.go
├── github.com/
│ ├── bitly/
│ │ └── go-nsq/
│ │ ├── .travis.yml
│ │ ├── ChangeLog.md
│ │ ├── LICENSE
│ │ ├── README.md
│ │ ├── UPGRADING.md
│ │ ├── api_request.go
│ │ ├── command.go
│ │ ├── command_test.go
│ │ ├── config.go
│ │ ├── config_test.go
│ │ ├── conn.go
│ │ ├── consumer.go
│ │ ├── consumer_test.go
│ │ ├── delegates.go
│ │ ├── errors.go
│ │ ├── message.go
│ │ ├── mock_test.go
│ │ ├── producer.go
│ │ ├── producer_test.go
│ │ ├── protocol.go
│ │ ├── states.go
│ │ ├── test/
│ │ │ ├── ca.pem
│ │ │ ├── server.key
│ │ │ └── server.pem
│ │ ├── test.sh
│ │ └── version.go
│ ├── cihub/
│ │ └── seelog/
│ │ ├── LICENSE.txt
│ │ ├── README.markdown
│ │ ├── behavior_adaptive_test.go
│ │ ├── behavior_adaptivelogger.go
│ │ ├── behavior_asynclogger.go
│ │ ├── behavior_asyncloop_test.go
│ │ ├── behavior_asynclooplogger.go
│ │ ├── behavior_asynctimer_test.go
│ │ ├── behavior_asynctimerlogger.go
│ │ ├── behavior_synclogger.go
│ │ ├── behavior_synclogger_test.go
│ │ ├── cfg_config.go
│ │ ├── cfg_errors.go
│ │ ├── cfg_logconfig.go
│ │ ├── cfg_logconfig_test.go
│ │ ├── cfg_parser.go
│ │ ├── cfg_parser_test.go
│ │ ├── common_closer.go
│ │ ├── common_constraints.go
│ │ ├── common_constraints_test.go
│ │ ├── common_context.go
│ │ ├── common_context_test.go
│ │ ├── common_exception.go
│ │ ├── common_exception_test.go
│ │ ├── common_flusher.go
│ │ ├── common_loglevel.go
│ │ ├── dispatch_custom.go
│ │ ├── dispatch_customdispatcher_test.go
│ │ ├── dispatch_dispatcher.go
│ │ ├── dispatch_filterdispatcher.go
│ │ ├── dispatch_filterdispatcher_test.go
│ │ ├── dispatch_splitdispatcher.go
│ │ ├── dispatch_splitdispatcher_test.go
│ │ ├── doc.go
│ │ ├── format.go
│ │ ├── format_test.go
│ │ ├── internals_baseerror.go
│ │ ├── internals_byteverifiers_test.go
│ │ ├── internals_fsutils.go
│ │ ├── internals_xmlnode.go
│ │ ├── internals_xmlnode_test.go
│ │ ├── log.go
│ │ ├── logger.go
│ │ ├── writers_bufferedwriter.go
│ │ ├── writers_bufferedwriter_test.go
│ │ ├── writers_connwriter.go
│ │ ├── writers_consolewriter.go
│ │ ├── writers_filewriter.go
│ │ ├── writers_filewriter_test.go
│ │ ├── writers_formattedwriter.go
│ │ ├── writers_formattedwriter_test.go
│ │ ├── writers_rollingfilewriter.go
│ │ ├── writers_rollingfilewriter_test.go
│ │ └── writers_smtpwriter.go
│ ├── golang/
│ │ └── protobuf/
│ │ └── proto/
│ │ ├── Makefile
│ │ ├── all_test.go
│ │ ├── clone.go
│ │ ├── clone_test.go
│ │ ├── decode.go
│ │ ├── encode.go
│ │ ├── equal.go
│ │ ├── equal_test.go
│ │ ├── extensions.go
│ │ ├── extensions_test.go
│ │ ├── lib.go
│ │ ├── message_set.go
│ │ ├── message_set_test.go
│ │ ├── pointer_reflect.go
│ │ ├── pointer_unsafe.go
│ │ ├── properties.go
│ │ ├── proto3_proto/
│ │ │ ├── proto3.pb.go
│ │ │ └── proto3.proto
│ │ ├── proto3_test.go
│ │ ├── size2_test.go
│ │ ├── size_test.go
│ │ ├── text.go
│ │ ├── text_parser.go
│ │ ├── text_parser_test.go
│ │ └── text_test.go
│ └── mreiferson/
│ ├── go-options/
│ │ ├── .travis.yml
│ │ ├── LICENSE
│ │ ├── README.md
│ │ ├── example_test.go
│ │ └── options.go
│ └── go-snappystream/
│ ├── .travis.yml
│ ├── LICENSE
│ ├── README.md
│ ├── fixturedata_test.go
│ ├── reader.go
│ ├── reader_test.go
│ ├── readwrite_test.go
│ ├── snappystream.go
│ ├── writer.go
│ └── writer_test.go
└── golang.org/
└── x/
└── net/
└── context/
├── context.go
├── context_test.go
├── ctxhttp/
│ ├── cancelreq.go
│ ├── cancelreq_go14.go
│ ├── ctxhttp.go
│ └── ctxhttp_test.go
└── withtimeout_test.go
SYMBOL INDEX (1810 symbols across 129 files)
FILE: apps/phosphor/main.go
function phosphorFlagSet (line 18) | func phosphorFlagSet() *flag.FlagSet {
function main (line 41) | func main() {
FILE: apps/phosphord/main.go
function phosphordFlagSet (line 21) | func phosphordFlagSet() *flag.FlagSet {
function main (line 42) | func main() {
FILE: internal/util/stringarray.go
type StringArray (line 10) | type StringArray
method Set (line 12) | func (a *StringArray) Set(s string) error {
method String (line 17) | func (a *StringArray) String() string {
FILE: internal/version/version.go
constant Version (line 9) | Version = "0.0.1"
function String (line 12) | func String(app string) string {
FILE: phosphor/context.go
function phosphorFromContext (line 9) | func phosphorFromContext(ctx context.Context) (*Phosphor, error) {
FILE: phosphor/domain.go
function NewTrace (line 10) | func NewTrace() *Trace {
type Trace (line 18) | type Trace struct
method AppendAnnotation (line 25) | func (t *Trace) AppendAnnotation(a *Annotation) error {
type AnnotationType (line 36) | type AnnotationType
constant UnknownAnnotationType (line 39) | UnknownAnnotationType = AnnotationType(0)
constant Req (line 42) | Req = AnnotationType(1)
constant Rsp (line 43) | Rsp = AnnotationType(2)
constant In (line 44) | In = AnnotationType(3)
constant Out (line 45) | Out = AnnotationType(4)
constant Timeout (line 46) | Timeout = AnnotationType(5)
type Annotation (line 55) | type Annotation struct
FILE: phosphor/handler.go
function Index (line 17) | func Index(w http.ResponseWriter, r *http.Request) {
function TraceLookup (line 26) | func TraceLookup(ctx context.Context) func(http.ResponseWriter, *http.Re...
function response (line 66) | func response(r *http.Request, w http.ResponseWriter, resp interface{}) {
function errorResponse (line 71) | func errorResponse(r *http.Request, w http.ResponseWriter, code int, err...
function writeResponse (line 80) | func writeResponse(r *http.Request, w http.ResponseWriter, code int, res...
FILE: phosphor/ingester.go
method RunIngester (line 23) | func (p *Phosphor) RunIngester() {
type IngestionHandler (line 60) | type IngestionHandler struct
method HandleMessage (line 65) | func (ih *IngestionHandler) HandleMessage(message *nsq.Message) error {
FILE: phosphor/marshaling.go
function prettyFormatTrace (line 10) | func prettyFormatTrace(t *Trace) interface{} {
function formatAnnotations (line 16) | func formatAnnotations(ans []*Annotation) interface{} {
function formatAnnotation (line 31) | func formatAnnotation(a *traceproto.Annotation) interface{} {
type ByTime (line 48) | type ByTime
method Len (line 50) | func (s ByTime) Len() int {
method Swap (line 53) | func (s ByTime) Swap(i, j int) {
method Less (line 56) | func (s ByTime) Less(i, j int) bool {
function ProtoToAnnotation (line 63) | func ProtoToAnnotation(p *traceproto.Annotation) *Annotation {
function protoToAnnotationType (line 86) | func protoToAnnotationType(p traceproto.AnnotationType) AnnotationType {
function annotationTypeToProto (line 97) | func annotationTypeToProto(at AnnotationType) traceproto.AnnotationType {
function microsecondInt64ToTime (line 109) | func microsecondInt64ToTime(i int64) time.Time {
function timeToMicrosecondInt64 (line 117) | func timeToMicrosecondInt64(t time.Time) int64 {
function microsecondInt64ToDuration (line 126) | func microsecondInt64ToDuration(i int64) time.Duration {
function durationToMicrosecondInt64 (line 131) | func durationToMicrosecondInt64(d time.Duration) int64 {
function protoToKeyValue (line 137) | func protoToKeyValue(p []*traceproto.KeyValue) map[string]string {
function keyValueToProto (line 150) | func keyValueToProto(m map[string]string) []*traceproto.KeyValue {
function AnnotationsToProto (line 163) | func AnnotationsToProto(a []*Annotation) []*traceproto.Annotation {
function AnnotationToProto (line 172) | func AnnotationToProto(a *Annotation) *traceproto.Annotation {
FILE: phosphor/memorystore.go
type MemoryStore (line 10) | type MemoryStore struct
method ReadTrace (line 28) | func (s *MemoryStore) ReadTrace(id string) (*Trace, error) {
method StoreAnnotation (line 41) | func (s *MemoryStore) StoreAnnotation(a *Annotation) error {
method statsLoop (line 73) | func (s *MemoryStore) statsLoop() {
method printStats (line 86) | func (s *MemoryStore) printStats() {
function NewMemoryStore (line 16) | func NewMemoryStore() *MemoryStore {
FILE: phosphor/options.go
type Options (line 11) | type Options struct
function NewOptions (line 27) | func NewOptions() *Options {
FILE: phosphor/phosphor.go
type Phosphor (line 11) | type Phosphor struct
method Run (line 27) | func (p *Phosphor) Run() {
method Exit (line 50) | func (p *Phosphor) Exit() {
function New (line 18) | func New(opts *Options) *Phosphor {
FILE: phosphor/store.go
type Store (line 5) | type Store interface
FILE: phosphord/forwarder.go
method forward (line 13) | func (p *PhosphorD) forward(id int) {
method sendTraces (line 60) | func (p *PhosphorD) sendTraces(id int, buf *[][]byte) error {
FILE: phosphord/options.go
type Options (line 3) | type Options struct
function NewOptions (line 18) | func NewOptions() *Options {
FILE: phosphord/phosphord.go
constant UDP (line 16) | UDP = "udp"
type PhosphorD (line 23) | type PhosphorD struct
method Run (line 50) | func (p *PhosphorD) Run() {
method Exit (line 64) | func (p *PhosphorD) Exit() {
method listen (line 74) | func (p *PhosphorD) listen() {
function New (line 32) | func New(opts *Options) *PhosphorD {
FILE: phosphord/test/test.go
constant MAX_PACKET_SIZE (line 14) | MAX_PACKET_SIZE = 1500 - 8 - 20
function main (line 17) | func main() {
FILE: phosphord/transport/nsq.go
function NewNSQTransport (line 19) | func NewNSQTransport(topic string, nsqdTCPAddrs util.StringArray) (Trans...
type NSQPublisher (line 43) | type NSQPublisher struct
method MultiPublish (line 49) | func (p *NSQPublisher) MultiPublish(body [][]byte) error {
FILE: phosphord/transport/transport.go
type Transport (line 3) | type Transport interface
FILE: proto/trace.pb.go
type AnnotationType (line 26) | type AnnotationType
method String (line 57) | func (x AnnotationType) String() string {
constant AnnotationType_UNKNOWN (line 29) | AnnotationType_UNKNOWN AnnotationType = 0
constant AnnotationType_CLIENT_SEND (line 30) | AnnotationType_CLIENT_SEND AnnotationType = 1
constant AnnotationType_CLIENT_RECV (line 31) | AnnotationType_CLIENT_RECV AnnotationType = 2
constant AnnotationType_SERVER_RECV (line 32) | AnnotationType_SERVER_RECV AnnotationType = 3
constant AnnotationType_SERVER_SEND (line 33) | AnnotationType_SERVER_SEND AnnotationType = 4
constant AnnotationType_TIMEOUT (line 34) | AnnotationType_TIMEOUT AnnotationType = 5
constant AnnotationType_ANNOTATION (line 35) | AnnotationType_ANNOTATION AnnotationType = 6
type Annotation (line 61) | type Annotation struct
method Reset (line 95) | func (m *Annotation) Reset() { *m = Annotation{} }
method String (line 96) | func (m *Annotation) String() string { return proto.CompactTextString(...
method ProtoMessage (line 97) | func (*Annotation) ProtoMessage() {}
method GetKeyValue (line 99) | func (m *Annotation) GetKeyValue() []*KeyValue {
type KeyValue (line 106) | type KeyValue struct
method Reset (line 111) | func (m *KeyValue) Reset() { *m = KeyValue{} }
method String (line 112) | func (m *KeyValue) String() string { return proto.CompactTextString(m) }
method ProtoMessage (line 113) | func (*KeyValue) ProtoMessage() {}
function init (line 115) | func init() {
FILE: vendor/code.google.com/p/snappy-go/snappy/decode.go
function DecodedLen (line 16) | func DecodedLen(src []byte) (int, error) {
function decodedLen (line 23) | func decodedLen(src []byte) (blockLen, headerLen int, err error) {
function Decode (line 38) | func Decode(dst, src []byte) ([]byte, error) {
FILE: vendor/code.google.com/p/snappy-go/snappy/encode.go
constant maxOffset (line 12) | maxOffset = 1 << 15
function emitLiteral (line 15) | func emitLiteral(dst, lit []byte) int {
function emitCopy (line 53) | func emitCopy(dst []byte, offset, length int) int {
function Encode (line 81) | func Encode(dst, src []byte) ([]byte, error) {
function MaxEncodedLen (line 152) | func MaxEncodedLen(srcLen int) int {
FILE: vendor/code.google.com/p/snappy-go/snappy/snappy.go
constant tagLiteral (line 34) | tagLiteral = 0x00
constant tagCopy1 (line 35) | tagCopy1 = 0x01
constant tagCopy2 (line 36) | tagCopy2 = 0x02
constant tagCopy4 (line 37) | tagCopy4 = 0x03
FILE: vendor/code.google.com/p/snappy-go/snappy/snappy_test.go
function roundtrip (line 23) | func roundtrip(b, ebuf, dbuf []byte) error {
function TestEmpty (line 38) | func TestEmpty(t *testing.T) {
function TestSmallCopy (line 44) | func TestSmallCopy(t *testing.T) {
function TestSmallRand (line 57) | func TestSmallRand(t *testing.T) {
function TestSmallRegular (line 70) | func TestSmallRegular(t *testing.T) {
function benchDecode (line 82) | func benchDecode(b *testing.B, src []byte) {
function benchEncode (line 95) | func benchEncode(b *testing.B, src []byte) {
function readFile (line 105) | func readFile(b *testing.B, filename string) []byte {
function expand (line 117) | func expand(src []byte, n int) []byte {
function benchWords (line 126) | func benchWords(b *testing.B, n int, decode bool) {
function BenchmarkWordsDecode1e3 (line 137) | func BenchmarkWordsDecode1e3(b *testing.B) { benchWords(b, 1e3, true) }
function BenchmarkWordsDecode1e4 (line 138) | func BenchmarkWordsDecode1e4(b *testing.B) { benchWords(b, 1e4, true) }
function BenchmarkWordsDecode1e5 (line 139) | func BenchmarkWordsDecode1e5(b *testing.B) { benchWords(b, 1e5, true) }
function BenchmarkWordsDecode1e6 (line 140) | func BenchmarkWordsDecode1e6(b *testing.B) { benchWords(b, 1e6, true) }
function BenchmarkWordsEncode1e3 (line 141) | func BenchmarkWordsEncode1e3(b *testing.B) { benchWords(b, 1e3, false) }
function BenchmarkWordsEncode1e4 (line 142) | func BenchmarkWordsEncode1e4(b *testing.B) { benchWords(b, 1e4, false) }
function BenchmarkWordsEncode1e5 (line 143) | func BenchmarkWordsEncode1e5(b *testing.B) { benchWords(b, 1e5, false) }
function BenchmarkWordsEncode1e6 (line 144) | func BenchmarkWordsEncode1e6(b *testing.B) { benchWords(b, 1e6, false) }
constant baseURL (line 174) | baseURL = "https://snappy.googlecode.com/svn/trunk/testdata/"
function downloadTestdata (line 176) | func downloadTestdata(basename string) (errRet error) {
function benchFile (line 200) | func benchFile(b *testing.B, n int, decode bool) {
function Benchmark_UFlat0 (line 226) | func Benchmark_UFlat0(b *testing.B) { benchFile(b, 0, true) }
function Benchmark_UFlat1 (line 227) | func Benchmark_UFlat1(b *testing.B) { benchFile(b, 1, true) }
function Benchmark_UFlat2 (line 228) | func Benchmark_UFlat2(b *testing.B) { benchFile(b, 2, true) }
function Benchmark_UFlat3 (line 229) | func Benchmark_UFlat3(b *testing.B) { benchFile(b, 3, true) }
function Benchmark_UFlat4 (line 230) | func Benchmark_UFlat4(b *testing.B) { benchFile(b, 4, true) }
function Benchmark_UFlat5 (line 231) | func Benchmark_UFlat5(b *testing.B) { benchFile(b, 5, true) }
function Benchmark_UFlat6 (line 232) | func Benchmark_UFlat6(b *testing.B) { benchFile(b, 6, true) }
function Benchmark_UFlat7 (line 233) | func Benchmark_UFlat7(b *testing.B) { benchFile(b, 7, true) }
function Benchmark_UFlat8 (line 234) | func Benchmark_UFlat8(b *testing.B) { benchFile(b, 8, true) }
function Benchmark_UFlat9 (line 235) | func Benchmark_UFlat9(b *testing.B) { benchFile(b, 9, true) }
function Benchmark_UFlat10 (line 236) | func Benchmark_UFlat10(b *testing.B) { benchFile(b, 10, true) }
function Benchmark_UFlat11 (line 237) | func Benchmark_UFlat11(b *testing.B) { benchFile(b, 11, true) }
function Benchmark_UFlat12 (line 238) | func Benchmark_UFlat12(b *testing.B) { benchFile(b, 12, true) }
function Benchmark_UFlat13 (line 239) | func Benchmark_UFlat13(b *testing.B) { benchFile(b, 13, true) }
function Benchmark_UFlat14 (line 240) | func Benchmark_UFlat14(b *testing.B) { benchFile(b, 14, true) }
function Benchmark_UFlat15 (line 241) | func Benchmark_UFlat15(b *testing.B) { benchFile(b, 15, true) }
function Benchmark_UFlat16 (line 242) | func Benchmark_UFlat16(b *testing.B) { benchFile(b, 16, true) }
function Benchmark_UFlat17 (line 243) | func Benchmark_UFlat17(b *testing.B) { benchFile(b, 17, true) }
function Benchmark_ZFlat0 (line 244) | func Benchmark_ZFlat0(b *testing.B) { benchFile(b, 0, false) }
function Benchmark_ZFlat1 (line 245) | func Benchmark_ZFlat1(b *testing.B) { benchFile(b, 1, false) }
function Benchmark_ZFlat2 (line 246) | func Benchmark_ZFlat2(b *testing.B) { benchFile(b, 2, false) }
function Benchmark_ZFlat3 (line 247) | func Benchmark_ZFlat3(b *testing.B) { benchFile(b, 3, false) }
function Benchmark_ZFlat4 (line 248) | func Benchmark_ZFlat4(b *testing.B) { benchFile(b, 4, false) }
function Benchmark_ZFlat5 (line 249) | func Benchmark_ZFlat5(b *testing.B) { benchFile(b, 5, false) }
function Benchmark_ZFlat6 (line 250) | func Benchmark_ZFlat6(b *testing.B) { benchFile(b, 6, false) }
function Benchmark_ZFlat7 (line 251) | func Benchmark_ZFlat7(b *testing.B) { benchFile(b, 7, false) }
function Benchmark_ZFlat8 (line 252) | func Benchmark_ZFlat8(b *testing.B) { benchFile(b, 8, false) }
function Benchmark_ZFlat9 (line 253) | func Benchmark_ZFlat9(b *testing.B) { benchFile(b, 9, false) }
function Benchmark_ZFlat10 (line 254) | func Benchmark_ZFlat10(b *testing.B) { benchFile(b, 10, false) }
function Benchmark_ZFlat11 (line 255) | func Benchmark_ZFlat11(b *testing.B) { benchFile(b, 11, false) }
function Benchmark_ZFlat12 (line 256) | func Benchmark_ZFlat12(b *testing.B) { benchFile(b, 12, false) }
function Benchmark_ZFlat13 (line 257) | func Benchmark_ZFlat13(b *testing.B) { benchFile(b, 13, false) }
function Benchmark_ZFlat14 (line 258) | func Benchmark_ZFlat14(b *testing.B) { benchFile(b, 14, false) }
function Benchmark_ZFlat15 (line 259) | func Benchmark_ZFlat15(b *testing.B) { benchFile(b, 15, false) }
function Benchmark_ZFlat16 (line 260) | func Benchmark_ZFlat16(b *testing.B) { benchFile(b, 16, false) }
function Benchmark_ZFlat17 (line 261) | func Benchmark_ZFlat17(b *testing.B) { benchFile(b, 17, false) }
FILE: vendor/github.com/bitly/go-nsq/api_request.go
type deadlinedConn (line 13) | type deadlinedConn struct
method Read (line 18) | func (c *deadlinedConn) Read(b []byte) (n int, err error) {
method Write (line 23) | func (c *deadlinedConn) Write(b []byte) (n int, err error) {
function newDeadlineTransport (line 28) | func newDeadlineTransport(timeout time.Duration) *http.Transport {
type wrappedResp (line 41) | type wrappedResp struct
function apiRequestNegotiateV1 (line 48) | func apiRequestNegotiateV1(method string, endpoint string, body io.Reade...
FILE: vendor/github.com/bitly/go-nsq/command.go
type Command (line 17) | type Command struct
method String (line 24) | func (c *Command) String() string {
method WriteTo (line 36) | func (c *Command) WriteTo(w io.Writer) (int64, error) {
function Identify (line 91) | func Identify(js map[string]interface{}) (*Command, error) {
function Auth (line 102) | func Auth(secret string) (*Command, error) {
function Register (line 107) | func Register(topic string, channel string) *Command {
function UnRegister (line 116) | func UnRegister(topic string, channel string) *Command {
function Ping (line 126) | func Ping() *Command {
function Publish (line 131) | func Publish(topic string, body []byte) *Command {
function DeferredPublish (line 138) | func DeferredPublish(topic string, delay time.Duration, body []byte) *Co...
function MultiPublish (line 145) | func MultiPublish(topic string, bodies [][]byte) (*Command, error) {
function Subscribe (line 175) | func Subscribe(topic string, channel string) *Command {
function Ready (line 182) | func Ready(count int) *Command {
function Finish (line 189) | func Finish(id MessageID) *Command {
function Requeue (line 197) | func Requeue(id MessageID, delay time.Duration) *Command {
function Touch (line 204) | func Touch(id MessageID) *Command {
function StartClose (line 213) | func StartClose() *Command {
function Nop (line 219) | func Nop() *Command {
FILE: vendor/github.com/bitly/go-nsq/command_test.go
function BenchmarkCommand (line 8) | func BenchmarkCommand(b *testing.B) {
FILE: vendor/github.com/bitly/go-nsq/config.go
type configHandler (line 23) | type configHandler interface
type defaultsHandler (line 29) | type defaultsHandler interface
type BackoffStrategy (line 35) | type BackoffStrategy interface
type ExponentialStrategy (line 40) | type ExponentialStrategy struct
method Calculate (line 45) | func (s *ExponentialStrategy) Calculate(attempt int) time.Duration {
method setConfig (line 51) | func (s *ExponentialStrategy) setConfig(cfg *Config) {
type FullJitterStrategy (line 56) | type FullJitterStrategy struct
method Calculate (line 64) | func (s *FullJitterStrategy) Calculate(attempt int) time.Duration {
method setConfig (line 78) | func (s *FullJitterStrategy) setConfig(cfg *Config) {
type Config (line 89) | type Config struct
method Set (line 215) | func (c *Config) Set(option string, value interface{}) error {
method assertInitialized (line 226) | func (c *Config) assertInitialized() {
method Validate (line 233) | func (c *Config) Validate() error {
method setDefaults (line 243) | func (c *Config) setDefaults() error {
function NewConfig (line 185) | func NewConfig() *Config {
type structTagsConfig (line 255) | type structTagsConfig struct
method HandlesOption (line 258) | func (h *structTagsConfig) HandlesOption(c *Config, option string) bool {
method Set (line 272) | func (h *structTagsConfig) Set(c *Config, option string, value interfa...
method SetDefaults (line 321) | func (h *structTagsConfig) SetDefaults(c *Config) error {
method Validate (line 348) | func (h *structTagsConfig) Validate(c *Config) error {
type tlsConfig (line 387) | type tlsConfig struct
method HandlesOption (line 392) | func (t *tlsConfig) HandlesOption(c *Config, option string) bool {
method Set (line 400) | func (t *tlsConfig) Set(c *Config, option string, value interface{}) e...
method Validate (line 472) | func (t *tlsConfig) Validate(c *Config) error {
function unsafeValueOf (line 478) | func unsafeValueOf(val reflect.Value) reflect.Value {
function valueCompare (line 483) | func valueCompare(v1 reflect.Value, v2 reflect.Value) int {
function coerce (line 517) | func coerce(v interface{}, typ reflect.Type) (reflect.Value, error) {
function valueTypeCoerce (line 546) | func valueTypeCoerce(v interface{}, typ reflect.Type) reflect.Value {
function coerceString (line 565) | func coerceString(v interface{}) (string, error) {
function coerceDuration (line 577) | func coerceDuration(v interface{}) (time.Duration, error) {
function coerceAddr (line 593) | func coerceAddr(v interface{}) (net.Addr, error) {
function coerceBackoffStrategy (line 603) | func coerceBackoffStrategy(v interface{}) (BackoffStrategy, error) {
function coerceBool (line 618) | func coerceBool(v interface{}) (bool, error) {
function coerceFloat64 (line 632) | func coerceFloat64(v interface{}) (float64, error) {
function coerceInt64 (line 648) | func coerceInt64(v interface{}) (int64, error) {
function coerceUint64 (line 660) | func coerceUint64(v interface{}) (uint64, error) {
FILE: vendor/github.com/bitly/go-nsq/config_test.go
function TestConfigSet (line 11) | func TestConfigSet(t *testing.T) {
function TestConfigValidate (line 58) | func TestConfigValidate(t *testing.T) {
function TestExponentialBackoff (line 69) | func TestExponentialBackoff(t *testing.T) {
function TestFullJitterBackoff (line 81) | func TestFullJitterBackoff(t *testing.T) {
function backoffTest (line 93) | func backoffTest(t *testing.T, expected []time.Duration, cb func(c *Conf...
FILE: vendor/github.com/bitly/go-nsq/conn.go
type IdentifyResponse (line 23) | type IdentifyResponse struct
type AuthResponse (line 33) | type AuthResponse struct
type msgResponse (line 39) | type msgResponse struct
type Conn (line 50) | type Conn struct
method SetLogger (line 121) | func (c *Conn) SetLogger(l logger, lvl LogLevel, format string) {
method getLogger (line 133) | func (c *Conn) getLogger() (logger, LogLevel, string) {
method Connect (line 142) | func (c *Conn) Connect() (*IdentifyResponse, error) {
method Close (line 187) | func (c *Conn) Close() error {
method IsClosing (line 198) | func (c *Conn) IsClosing() bool {
method RDY (line 203) | func (c *Conn) RDY() int64 {
method LastRDY (line 208) | func (c *Conn) LastRDY() int64 {
method SetRDY (line 213) | func (c *Conn) SetRDY(rdy int64) {
method MaxRDY (line 220) | func (c *Conn) MaxRDY() int64 {
method LastMessageTime (line 226) | func (c *Conn) LastMessageTime() time.Time {
method RemoteAddr (line 231) | func (c *Conn) RemoteAddr() net.Addr {
method String (line 236) | func (c *Conn) String() string {
method Read (line 241) | func (c *Conn) Read(p []byte) (int, error) {
method Write (line 247) | func (c *Conn) Write(p []byte) (int, error) {
method WriteCommand (line 254) | func (c *Conn) WriteCommand(cmd *Command) error {
method Flush (line 277) | func (c *Conn) Flush() error {
method identify (line 284) | func (c *Conn) identify() (*IdentifyResponse, error) {
method upgradeTLS (line 378) | func (c *Conn) upgradeTLS(tlsConf *tls.Config) error {
method upgradeDeflate (line 407) | func (c *Conn) upgradeDeflate(level int) error {
method upgradeSnappy (line 425) | func (c *Conn) upgradeSnappy() error {
method auth (line 442) | func (c *Conn) auth(secret string) error {
method readLoop (line 474) | func (c *Conn) readLoop() {
method writeLoop (line 545) | func (c *Conn) writeLoop() {
method close (line 598) | func (c *Conn) close() {
method cleanup (line 636) | func (c *Conn) cleanup() {
method waitForCleanup (line 677) | func (c *Conn) waitForCleanup() {
method onMessageFinish (line 686) | func (c *Conn) onMessageFinish(m *Message) {
method onMessageRequeue (line 690) | func (c *Conn) onMessageRequeue(m *Message, delay time.Duration, backo...
method onMessageTouch (line 702) | func (c *Conn) onMessageTouch(m *Message) {
method log (line 709) | func (c *Conn) log(lvl LogLevel, line string, args ...interface{}) {
function NewConn (line 89) | func NewConn(addr string, config *Config, delegate ConnDelegate) *Conn {
type flusher (line 272) | type flusher interface
FILE: vendor/github.com/bitly/go-nsq/consumer.go
type Handler (line 28) | type Handler interface
type HandlerFunc (line 38) | type HandlerFunc
method HandleMessage (line 41) | func (h HandlerFunc) HandleMessage(m *Message) error {
type DiscoveryFilter (line 47) | type DiscoveryFilter interface
type FailedMessageLogger (line 54) | type FailedMessageLogger interface
type ConsumerStats (line 60) | type ConsumerStats struct
type backoffSignal (line 69) | type backoffSignal
constant backoffFlag (line 72) | backoffFlag backoffSignal = iota
constant continueFlag (line 73) | continueFlag
constant resumeFlag (line 74) | resumeFlag
type Consumer (line 86) | type Consumer struct
method Stats (line 191) | func (r *Consumer) Stats() *ConsumerStats {
method conns (line 200) | func (r *Consumer) conns() []*Conn {
method SetLogger (line 217) | func (r *Consumer) SetLogger(l logger, lvl LogLevel) {
method getLogger (line 225) | func (r *Consumer) getLogger() (logger, LogLevel) {
method SetBehaviorDelegate (line 238) | func (r *Consumer) SetBehaviorDelegate(cb interface{}) {
method perConnMaxInFlight (line 256) | func (r *Consumer) perConnMaxInFlight() int64 {
method IsStarved (line 264) | func (r *Consumer) IsStarved() bool {
method getMaxInFlight (line 275) | func (r *Consumer) getMaxInFlight() int32 {
method ChangeMaxInFlight (line 285) | func (r *Consumer) ChangeMaxInFlight(maxInFlight int) {
method ConnectToNSQLookupd (line 303) | func (r *Consumer) ConnectToNSQLookupd(addr string) error {
method ConnectToNSQLookupds (line 344) | func (r *Consumer) ConnectToNSQLookupds(addresses []string) error {
method lookupdLoop (line 369) | func (r *Consumer) lookupdLoop() {
method nextLookupdEndpoint (line 405) | func (r *Consumer) nextLookupdEndpoint() string {
method queryLookupd (line 453) | func (r *Consumer) queryLookupd() {
method ConnectToNSQDs (line 489) | func (r *Consumer) ConnectToNSQDs(addresses []string) error {
method ConnectToNSQD (line 504) | func (r *Consumer) ConnectToNSQD(addr string) error {
method DisconnectFromNSQD (line 589) | func (r *Consumer) DisconnectFromNSQD(addr string) error {
method DisconnectFromNSQLookupd (line 615) | func (r *Consumer) DisconnectFromNSQLookupd(addr string) error {
method onConnMessage (line 633) | func (r *Consumer) onConnMessage(c *Conn, msg *Message) {
method onConnMessageFinished (line 640) | func (r *Consumer) onConnMessageFinished(c *Conn, msg *Message) {
method onConnMessageRequeued (line 644) | func (r *Consumer) onConnMessageRequeued(c *Conn, msg *Message) {
method onConnBackoff (line 648) | func (r *Consumer) onConnBackoff(c *Conn) {
method onConnContinue (line 652) | func (r *Consumer) onConnContinue(c *Conn) {
method onConnResume (line 656) | func (r *Consumer) onConnResume(c *Conn) {
method onConnResponse (line 660) | func (r *Consumer) onConnResponse(c *Conn, data []byte) {
method onConnError (line 671) | func (r *Consumer) onConnError(c *Conn, data []byte) {}
method onConnHeartbeat (line 673) | func (r *Consumer) onConnHeartbeat(c *Conn) {}
method onConnIOError (line 675) | func (r *Consumer) onConnIOError(c *Conn, err error) {
method onConnClose (line 679) | func (r *Consumer) onConnClose(c *Conn) {
method startStopContinueBackoff (line 758) | func (r *Consumer) startStopContinueBackoff(conn *Conn, signal backoff...
method backoff (line 814) | func (r *Consumer) backoff(d time.Duration) {
method resume (line 819) | func (r *Consumer) resume() {
method inBackoff (line 852) | func (r *Consumer) inBackoff() bool {
method inBackoffTimeout (line 856) | func (r *Consumer) inBackoffTimeout() bool {
method maybeUpdateRDY (line 860) | func (r *Consumer) maybeUpdateRDY(conn *Conn) {
method rdyLoop (line 884) | func (r *Consumer) rdyLoop() {
method updateRDY (line 902) | func (r *Consumer) updateRDY(c *Conn, count int64) error {
method sendRDY (line 945) | func (r *Consumer) sendRDY(c *Conn, count int64) error {
method redistributeRDY (line 961) | func (r *Consumer) redistributeRDY() {
method Stop (line 1021) | func (r *Consumer) Stop() {
method stopHandlers (line 1049) | func (r *Consumer) stopHandlers() {
method AddHandler (line 1062) | func (r *Consumer) AddHandler(handler Handler) {
method AddConcurrentHandlers (line 1073) | func (r *Consumer) AddConcurrentHandlers(handler Handler, concurrency ...
method handlerLoop (line 1084) | func (r *Consumer) handlerLoop(handler Handler) {
method shouldFailMessage (line 1119) | func (r *Consumer) shouldFailMessage(message *Message, handler interfa...
method exit (line 1135) | func (r *Consumer) exit() {
method log (line 1143) | func (r *Consumer) log(lvl LogLevel, line string, args ...interface{}) {
function NewConsumer (line 146) | func NewConsumer(topic string, channel string, config *Config) (*Consume...
function validatedLookupAddr (line 354) | func validatedLookupAddr(addr string) error {
type lookupResp (line 434) | type lookupResp struct
type peerInfo (line 440) | type peerInfo struct
function indexOf (line 578) | func indexOf(n string, h []string) int {
FILE: vendor/github.com/bitly/go-nsq/consumer_test.go
type MyTestHandler (line 20) | type MyTestHandler struct
method LogFailedMessage (line 30) | func (h *MyTestHandler) LogFailedMessage(message *Message) {
method HandleMessage (line 35) | func (h *MyTestHandler) HandleMessage(message *Message) error {
function SendMessage (line 58) | func SendMessage(t *testing.T, port int, topic string, method string, bo...
function TestConsumer (line 70) | func TestConsumer(t *testing.T) {
function TestConsumerTLS (line 74) | func TestConsumerTLS(t *testing.T) {
function TestConsumerDeflate (line 83) | func TestConsumerDeflate(t *testing.T) {
function TestConsumerSnappy (line 89) | func TestConsumerSnappy(t *testing.T) {
function TestConsumerTLSDeflate (line 95) | func TestConsumerTLSDeflate(t *testing.T) {
function TestConsumerTLSSnappy (line 105) | func TestConsumerTLSSnappy(t *testing.T) {
function TestConsumerTLSClientCert (line 115) | func TestConsumerTLSClientCert(t *testing.T) {
function TestConsumerTLSClientCertViaSet (line 131) | func TestConsumerTLSClientCertViaSet(t *testing.T) {
function consumerTest (line 145) | func consumerTest(t *testing.T, cb func(c *Config)) {
FILE: vendor/github.com/bitly/go-nsq/delegates.go
type logger (line 5) | type logger interface
type LogLevel (line 10) | type LogLevel
method String (line 21) | func (lvl LogLevel) String() string {
constant LogLevelDebug (line 14) | LogLevelDebug LogLevel = iota
constant LogLevelInfo (line 15) | LogLevelInfo
constant LogLevelWarning (line 16) | LogLevelWarning
constant LogLevelError (line 17) | LogLevelError
type MessageDelegate (line 35) | type MessageDelegate interface
type connMessageDelegate (line 49) | type connMessageDelegate struct
method OnFinish (line 53) | func (d *connMessageDelegate) OnFinish(m *Message) { d.c.onMessageFini...
method OnRequeue (line 54) | func (d *connMessageDelegate) OnRequeue(m *Message, t time.Duration, b...
method OnTouch (line 57) | func (d *connMessageDelegate) OnTouch(m *Message) { d.c.onMessageTouch...
type ConnDelegate (line 61) | type ConnDelegate interface
type consumerConnDelegate (line 106) | type consumerConnDelegate struct
method OnResponse (line 110) | func (d *consumerConnDelegate) OnResponse(c *Conn, data []byte) ...
method OnError (line 111) | func (d *consumerConnDelegate) OnError(c *Conn, data []byte) ...
method OnMessage (line 112) | func (d *consumerConnDelegate) OnMessage(c *Conn, m *Message) ...
method OnMessageFinished (line 113) | func (d *consumerConnDelegate) OnMessageFinished(c *Conn, m *Message) ...
method OnMessageRequeued (line 114) | func (d *consumerConnDelegate) OnMessageRequeued(c *Conn, m *Message) ...
method OnBackoff (line 115) | func (d *consumerConnDelegate) OnBackoff(c *Conn) ...
method OnContinue (line 116) | func (d *consumerConnDelegate) OnContinue(c *Conn) ...
method OnResume (line 117) | func (d *consumerConnDelegate) OnResume(c *Conn) ...
method OnIOError (line 118) | func (d *consumerConnDelegate) OnIOError(c *Conn, err error) ...
method OnHeartbeat (line 119) | func (d *consumerConnDelegate) OnHeartbeat(c *Conn) ...
method OnClose (line 120) | func (d *consumerConnDelegate) OnClose(c *Conn) ...
type producerConnDelegate (line 124) | type producerConnDelegate struct
method OnResponse (line 128) | func (d *producerConnDelegate) OnResponse(c *Conn, data []byte) ...
method OnError (line 129) | func (d *producerConnDelegate) OnError(c *Conn, data []byte) ...
method OnMessage (line 130) | func (d *producerConnDelegate) OnMessage(c *Conn, m *Message) {}
method OnMessageFinished (line 131) | func (d *producerConnDelegate) OnMessageFinished(c *Conn, m *Message) {}
method OnMessageRequeued (line 132) | func (d *producerConnDelegate) OnMessageRequeued(c *Conn, m *Message) {}
method OnBackoff (line 133) | func (d *producerConnDelegate) OnBackoff(c *Conn) {}
method OnContinue (line 134) | func (d *producerConnDelegate) OnContinue(c *Conn) {}
method OnResume (line 135) | func (d *producerConnDelegate) OnResume(c *Conn) {}
method OnIOError (line 136) | func (d *producerConnDelegate) OnIOError(c *Conn, err error) ...
method OnHeartbeat (line 137) | func (d *producerConnDelegate) OnHeartbeat(c *Conn) ...
method OnClose (line 138) | func (d *producerConnDelegate) OnClose(c *Conn) ...
FILE: vendor/github.com/bitly/go-nsq/errors.go
type ErrIdentify (line 26) | type ErrIdentify struct
method Error (line 31) | func (e ErrIdentify) Error() string {
type ErrProtocol (line 37) | type ErrProtocol struct
method Error (line 42) | func (e ErrProtocol) Error() string {
FILE: vendor/github.com/bitly/go-nsq/message.go
constant MsgIDLength (line 13) | MsgIDLength = 16
type MessageID (line 16) | type MessageID
type Message (line 20) | type Message struct
method DisableAutoResponse (line 50) | func (m *Message) DisableAutoResponse() {
method IsAutoResponseDisabled (line 56) | func (m *Message) IsAutoResponseDisabled() bool {
method HasResponded (line 61) | func (m *Message) HasResponded() bool {
method Finish (line 67) | func (m *Message) Finish() {
method Touch (line 76) | func (m *Message) Touch() {
method Requeue (line 89) | func (m *Message) Requeue(delay time.Duration) {
method RequeueWithoutBackoff (line 98) | func (m *Message) RequeueWithoutBackoff(delay time.Duration) {
method doRequeue (line 102) | func (m *Message) doRequeue(delay time.Duration, backoff bool) {
method WriteTo (line 114) | func (m *Message) WriteTo(w io.Writer) (int64, error) {
function NewMessage (line 36) | func NewMessage(id MessageID, body []byte) *Message {
function DecodeMessage (line 143) | func DecodeMessage(b []byte) (*Message, error) {
FILE: vendor/github.com/bitly/go-nsq/mock_test.go
type tbLog (line 17) | type tbLog interface
type testLogger (line 21) | type testLogger struct
method Output (line 25) | func (tl *testLogger) Output(maxdepth int, s string) error {
function newTestLogger (line 30) | func newTestLogger(tbl tbLog) logger {
type instruction (line 34) | type instruction struct
type mockNSQD (line 40) | type mockNSQD struct
method listen (line 66) | func (n *mockNSQD) listen() {
method handle (line 81) | func (n *mockNSQD) handle(conn net.Conn) {
function newMockNSQD (line 48) | func newMockNSQD(script []instruction, addr string) *mockNSQD {
function framedResponse (line 168) | func framedResponse(frameType int32, data []byte) []byte {
type testHandler (line 190) | type testHandler struct
method HandleMessage (line 192) | func (h *testHandler) HandleMessage(message *Message) error {
function frameMessage (line 209) | func frameMessage(m *Message) []byte {
function TestConsumerBackoff (line 215) | func TestConsumerBackoff(t *testing.T) {
function TestConsumerRequeueNoBackoff (line 289) | func TestConsumerRequeueNoBackoff(t *testing.T) {
function TestConsumerBackoffDisconnect (line 360) | func TestConsumerBackoffDisconnect(t *testing.T) {
FILE: vendor/github.com/bitly/go-nsq/producer.go
type producerConn (line 12) | type producerConn interface
type Producer (line 25) | type Producer struct
method Ping (line 100) | func (w *Producer) Ping() error {
method SetLogger (line 118) | func (w *Producer) SetLogger(l logger, lvl LogLevel) {
method getLogger (line 126) | func (w *Producer) getLogger() (logger, LogLevel) {
method String (line 134) | func (w *Producer) String() string {
method Stop (line 141) | func (w *Producer) Stop() {
method PublishAsync (line 161) | func (w *Producer) PublishAsync(topic string, body []byte, doneChan ch...
method MultiPublishAsync (line 173) | func (w *Producer) MultiPublishAsync(topic string, body [][]byte, done...
method Publish (line 184) | func (w *Producer) Publish(topic string, body []byte) error {
method MultiPublish (line 190) | func (w *Producer) MultiPublish(topic string, body [][]byte) error {
method sendCommand (line 198) | func (w *Producer) sendCommand(cmd *Command) error {
method sendCommandAsync (line 209) | func (w *Producer) sendCommandAsync(cmd *Command, doneChan chan *Produ...
method connect (line 238) | func (w *Producer) connect() error {
method close (line 275) | func (w *Producer) close() {
method router (line 288) | func (w *Producer) router() {
method popTransaction (line 315) | func (w *Producer) popTransaction(frameType int32, data []byte) {
method transactionCleanup (line 324) | func (w *Producer) transactionCleanup() {
method log (line 351) | func (w *Producer) log(lvl LogLevel, line string, args ...interface{}) {
method onConnResponse (line 365) | func (w *Producer) onConnResponse(c *Conn, data []byte) { w.responseCh...
method onConnError (line 366) | func (w *Producer) onConnError(c *Conn, data []byte) { w.errorChan ...
method onConnHeartbeat (line 367) | func (w *Producer) onConnHeartbeat(c *Conn) {}
method onConnIOError (line 368) | func (w *Producer) onConnIOError(c *Conn, err error) { w.close() }
method onConnClose (line 369) | func (w *Producer) onConnClose(c *Conn) {
type ProducerTransaction (line 53) | type ProducerTransaction struct
method finish (line 60) | func (t *ProducerTransaction) finish() {
function NewProducer (line 70) | func NewProducer(addr string, config *Config) (*Producer, error) {
FILE: vendor/github.com/bitly/go-nsq/producer_test.go
type ConsumerHandler (line 19) | type ConsumerHandler struct
method LogFailedMessage (line 26) | func (h *ConsumerHandler) LogFailedMessage(message *Message) {
method HandleMessage (line 31) | func (h *ConsumerHandler) HandleMessage(message *Message) error {
function TestProducerConnection (line 43) | func TestProducerConnection(t *testing.T) {
function TestProducerPing (line 70) | func TestProducerPing(t *testing.T) {
function TestProducerPublish (line 92) | func TestProducerPublish(t *testing.T) {
function TestProducerMultiPublish (line 116) | func TestProducerMultiPublish(t *testing.T) {
function TestProducerPublishAsync (line 143) | func TestProducerPublishAsync(t *testing.T) {
function TestProducerMultiPublishAsync (line 178) | func TestProducerMultiPublishAsync(t *testing.T) {
function TestProducerHeartbeat (line 217) | func TestProducerHeartbeat(t *testing.T) {
function readMessages (line 264) | func readMessages(topicName string, t *testing.T, msgCount int) {
type mockProducerConn (line 292) | type mockProducerConn struct
method String (line 308) | func (m *mockProducerConn) String() string {
method SetLogger (line 312) | func (m *mockProducerConn) SetLogger(logger logger, level LogLevel, pr...
method Connect (line 314) | func (m *mockProducerConn) Connect() (*IdentifyResponse, error) {
method Close (line 318) | func (m *mockProducerConn) Close() error {
method WriteCommand (line 323) | func (m *mockProducerConn) WriteCommand(cmd *Command) error {
method router (line 330) | func (m *mockProducerConn) router() {
function newMockProducerConn (line 298) | func newMockProducerConn(delegate ConnDelegate) producerConn {
function BenchmarkProducer (line 342) | func BenchmarkProducer(b *testing.B) {
FILE: vendor/github.com/bitly/go-nsq/protocol.go
constant FrameTypeResponse (line 18) | FrameTypeResponse int32 = 0
constant FrameTypeError (line 19) | FrameTypeError int32 = 1
constant FrameTypeMessage (line 20) | FrameTypeMessage int32 = 2
function IsValidTopicName (line 26) | func IsValidTopicName(name string) bool {
function IsValidChannelName (line 31) | func IsValidChannelName(name string) bool {
function isValidName (line 35) | func isValidName(name string) bool {
function ReadResponse (line 50) | func ReadResponse(r io.Reader) ([]byte, error) {
function UnpackResponse (line 79) | func UnpackResponse(response []byte) (int32, []byte, error) {
function ReadUnpackedResponse (line 90) | func ReadUnpackedResponse(r io.Reader) (int32, []byte, error) {
FILE: vendor/github.com/bitly/go-nsq/states.go
constant StateInit (line 5) | StateInit = iota
constant StateDisconnected (line 6) | StateDisconnected
constant StateConnected (line 7) | StateConnected
constant StateSubscribed (line 8) | StateSubscribed
constant StateClosing (line 11) | StateClosing
FILE: vendor/github.com/bitly/go-nsq/version.go
constant VERSION (line 8) | VERSION = "1.0.5-alpha"
FILE: vendor/github.com/cihub/seelog/behavior_adaptive_test.go
function countSequencedRowsInFile (line 37) | func countSequencedRowsInFile(filePath string) (int64, error) {
function Test_Adaptive (line 72) | func Test_Adaptive(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/behavior_adaptivelogger.go
type asyncAdaptiveLogger (line 45) | type asyncAdaptiveLogger struct
method processItem (line 89) | func (asnAdaptiveLogger *asyncAdaptiveLogger) processItem() (closed bo...
method calcAdaptiveInterval (line 109) | func (asnAdaptiveLogger *asyncAdaptiveLogger) calcAdaptiveInterval(msg...
method processQueue (line 117) | func (asnAdaptiveLogger *asyncAdaptiveLogger) processQueue() {
function newAsyncAdaptiveLogger (line 53) | func newAsyncAdaptiveLogger(
FILE: vendor/github.com/cihub/seelog/behavior_asynclogger.go
constant MaxQueueSize (line 35) | MaxQueueSize = 10000
type msgQueueItem (line 38) | type msgQueueItem struct
type asyncLogger (line 45) | type asyncLogger struct
method innerLog (line 63) | func (asnLogger *asyncLogger) innerLog(
method Close (line 71) | func (asnLogger *asyncLogger) Close() {
method Flush (line 87) | func (asnLogger *asyncLogger) Flush() {
method flushQueue (line 97) | func (asnLogger *asyncLogger) flushQueue(lockNeeded bool) {
method processQueueElement (line 108) | func (asnLogger *asyncLogger) processQueueElement() {
method addMsgToQueue (line 117) | func (asnLogger *asyncLogger) addMsgToQueue(
function newAsyncLogger (line 52) | func newAsyncLogger(config *logConfig) *asyncLogger {
FILE: vendor/github.com/cihub/seelog/behavior_asyncloop_test.go
function Test_Asyncloop (line 32) | func Test_Asyncloop(t *testing.T) {
function Test_AsyncloopOff (line 85) | func Test_AsyncloopOff(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/behavior_asynclooplogger.go
type asyncLoopLogger (line 29) | type asyncLoopLogger struct
method processItem (line 45) | func (asnLoopLogger *asyncLoopLogger) processItem() (closed bool) {
method processQueue (line 61) | func (asnLoopLogger *asyncLoopLogger) processQueue() {
function newAsyncLoopLogger (line 34) | func newAsyncLoopLogger(config *logConfig) *asyncLoopLogger {
FILE: vendor/github.com/cihub/seelog/behavior_asynctimer_test.go
function Test_Asynctimer (line 32) | func Test_Asynctimer(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/behavior_asynctimerlogger.go
type asyncTimerLogger (line 34) | type asyncTimerLogger struct
method processItem (line 56) | func (asnTimerLogger *asyncTimerLogger) processItem() (closed bool) {
method processQueue (line 72) | func (asnTimerLogger *asyncTimerLogger) processQueue() {
function newAsyncTimerLogger (line 40) | func newAsyncTimerLogger(config *logConfig, interval time.Duration) (*as...
FILE: vendor/github.com/cihub/seelog/behavior_synclogger.go
type syncLogger (line 33) | type syncLogger struct
method innerLog (line 46) | func (syncLogger *syncLogger) innerLog(
method Close (line 54) | func (syncLogger *syncLogger) Close() {
method Flush (line 65) | func (syncLogger *syncLogger) Flush() {
function newSyncLogger (line 38) | func newSyncLogger(config *logConfig) *syncLogger {
FILE: vendor/github.com/cihub/seelog/behavior_synclogger_test.go
function Test_Sync (line 32) | func Test_Sync(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/cfg_config.go
function LoggerFromConfigAsFile (line 35) | func LoggerFromConfigAsFile(fileName string) (LoggerInterface, error) {
function LoggerFromConfigAsBytes (line 51) | func LoggerFromConfigAsBytes(data []byte) (LoggerInterface, error) {
function LoggerFromConfigAsString (line 61) | func LoggerFromConfigAsString(data string) (LoggerInterface, error) {
function LoggerFromParamConfigAsFile (line 67) | func LoggerFromParamConfigAsFile(fileName string, parserParams *CfgParse...
function LoggerFromParamConfigAsBytes (line 84) | func LoggerFromParamConfigAsBytes(data []byte, parserParams *CfgParsePar...
function LoggerFromParamConfigAsString (line 95) | func LoggerFromParamConfigAsString(data string, parserParams *CfgParsePa...
function LoggerFromWriterWithMinLevel (line 100) | func LoggerFromWriterWithMinLevel(output io.Writer, minLevel LogLevel) (...
function LoggerFromWriterWithMinLevelAndFormat (line 111) | func LoggerFromWriterWithMinLevelAndFormat(output io.Writer, minLevel Lo...
function LoggerFromXMLDecoder (line 135) | func LoggerFromXMLDecoder(xmlParser *xml.Decoder, rootNode xml.Token) (L...
function LoggerFromCustomReceiver (line 167) | func LoggerFromCustomReceiver(receiver CustomReceiver) (LoggerInterface,...
FILE: vendor/github.com/cihub/seelog/cfg_errors.go
type unexpectedChildElementError (line 36) | type unexpectedChildElementError struct
function newUnexpectedChildElementError (line 40) | func newUnexpectedChildElementError(msg string) *unexpectedChildElementE...
type missingArgumentError (line 45) | type missingArgumentError struct
function newMissingArgumentError (line 49) | func newMissingArgumentError(nodeName, attrName string) *missingArgument...
type unexpectedAttributeError (line 54) | type unexpectedAttributeError struct
function newUnexpectedAttributeError (line 58) | func newUnexpectedAttributeError(nodeName, attr string) *unexpectedAttri...
FILE: vendor/github.com/cihub/seelog/cfg_logconfig.go
type loggerTypeFromString (line 31) | type loggerTypeFromString
constant syncloggerTypeFromString (line 34) | syncloggerTypeFromString = iota
constant asyncLooploggerTypeFromString (line 35) | asyncLooploggerTypeFromString
constant asyncTimerloggerTypeFromString (line 36) | asyncTimerloggerTypeFromString
constant adaptiveLoggerTypeFromString (line 37) | adaptiveLoggerTypeFromString
constant defaultloggerTypeFromString (line 38) | defaultloggerTypeFromString = asyncLooploggerTypeFromString
constant syncloggerTypeFromStringStr (line 42) | syncloggerTypeFromStringStr = "sync"
constant asyncloggerTypeFromStringStr (line 43) | asyncloggerTypeFromStringStr = "asyncloop"
constant asyncTimerloggerTypeFromStringStr (line 44) | asyncTimerloggerTypeFromStringStr = "asynctimer"
constant adaptiveLoggerTypeFromStringStr (line 45) | adaptiveLoggerTypeFromStringStr = "adaptive"
type asyncTimerLoggerData (line 49) | type asyncTimerLoggerData struct
type adaptiveLoggerData (line 54) | type adaptiveLoggerData struct
function getLoggerTypeFromString (line 68) | func getLoggerTypeFromString(logTypeString string) (level loggerTypeFrom...
type logConfig (line 80) | type logConfig struct
method IsAllowed (line 117) | func (config *logConfig) IsAllowed(level LogLevel, context LogContextI...
function newConfig (line 89) | func newConfig(
FILE: vendor/github.com/cihub/seelog/cfg_logconfig_test.go
function TestConfig (line 32) | func TestConfig(t *testing.T) {
function getFirstContext (line 93) | func getFirstContext() (LogContextInterface, error) {
function getSecondContext (line 97) | func getSecondContext() (LogContextInterface, error) {
FILE: vendor/github.com/cihub/seelog/cfg_parser.go
constant seelogConfigID (line 40) | seelogConfigID = "seelog"
constant outputsID (line 41) | outputsID = "outputs"
constant formatsID (line 42) | formatsID = "formats"
constant minLevelID (line 43) | minLevelID = "minlevel"
constant maxLevelID (line 44) | maxLevelID = "maxlevel"
constant levelsID (line 45) | levelsID = "levels"
constant exceptionsID (line 46) | exceptionsID = "exceptions"
constant exceptionID (line 47) | exceptionID = "exception"
constant funcPatternID (line 48) | funcPatternID = "funcpattern"
constant filePatternID (line 49) | filePatternID = "filepattern"
constant formatID (line 50) | formatID = "format"
constant formatAttrID (line 51) | formatAttrID = "format"
constant formatKeyAttrID (line 52) | formatKeyAttrID = "id"
constant outputFormatID (line 53) | outputFormatID = "formatid"
constant pathID (line 54) | pathID = "path"
constant fileWriterID (line 55) | fileWriterID = "file"
constant smtpWriterID (line 56) | smtpWriterID = "smtp"
constant senderaddressID (line 57) | senderaddressID = "senderaddress"
constant senderNameID (line 58) | senderNameID = "sendername"
constant recipientID (line 59) | recipientID = "recipient"
constant mailHeaderID (line 60) | mailHeaderID = "header"
constant mailHeaderNameID (line 61) | mailHeaderNameID = "name"
constant mailHeaderValueID (line 62) | mailHeaderValueID = "value"
constant addressID (line 63) | addressID = "address"
constant hostNameID (line 64) | hostNameID = "hostname"
constant hostPortID (line 65) | hostPortID = "hostport"
constant userNameID (line 66) | userNameID = "username"
constant userPassID (line 67) | userPassID = "password"
constant cACertDirpathID (line 68) | cACertDirpathID = "cacertdirpath"
constant subjectID (line 69) | subjectID = "subject"
constant splitterDispatcherID (line 70) | splitterDispatcherID = "splitter"
constant consoleWriterID (line 71) | consoleWriterID = "console"
constant customReceiverID (line 72) | customReceiverID = "custom"
constant customNameAttrID (line 73) | customNameAttrID = "name"
constant customNameDataAttrPrefix (line 74) | customNameDataAttrPrefix = "data-"
constant filterDispatcherID (line 75) | filterDispatcherID = "filter"
constant filterLevelsAttrID (line 76) | filterLevelsAttrID = "levels"
constant rollingfileWriterID (line 77) | rollingfileWriterID = "rollingfile"
constant rollingFileTypeAttr (line 78) | rollingFileTypeAttr = "type"
constant rollingFilePathAttr (line 79) | rollingFilePathAttr = "filename"
constant rollingFileMaxSizeAttr (line 80) | rollingFileMaxSizeAttr = "maxsize"
constant rollingFileMaxRollsAttr (line 81) | rollingFileMaxRollsAttr = "maxrolls"
constant rollingFileNameModeAttr (line 82) | rollingFileNameModeAttr = "namemode"
constant rollingFileDataPatternAttr (line 83) | rollingFileDataPatternAttr = "datepattern"
constant rollingFileArchiveAttr (line 84) | rollingFileArchiveAttr = "archivetype"
constant rollingFileArchivePathAttr (line 85) | rollingFileArchivePathAttr = "archivepath"
constant bufferedWriterID (line 86) | bufferedWriterID = "buffered"
constant bufferedSizeAttr (line 87) | bufferedSizeAttr = "size"
constant bufferedFlushPeriodAttr (line 88) | bufferedFlushPeriodAttr = "flushperiod"
constant loggerTypeFromStringAttr (line 89) | loggerTypeFromStringAttr = "type"
constant asyncLoggerIntervalAttr (line 90) | asyncLoggerIntervalAttr = "asyncinterval"
constant adaptLoggerMinIntervalAttr (line 91) | adaptLoggerMinIntervalAttr = "mininterval"
constant adaptLoggerMaxIntervalAttr (line 92) | adaptLoggerMaxIntervalAttr = "maxinterval"
constant adaptLoggerCriticalMsgCountAttr (line 93) | adaptLoggerCriticalMsgCountAttr = "critmsgcount"
constant predefinedPrefix (line 94) | predefinedPrefix = "std:"
constant connWriterID (line 95) | connWriterID = "conn"
constant connWriterAddrAttr (line 96) | connWriterAddrAttr = "addr"
constant connWriterNetAttr (line 97) | connWriterNetAttr = "net"
constant connWriterReconnectOnMsgAttr (line 98) | connWriterReconnectOnMsgAttr = "reconnectonmsg"
constant connWriterUseTLSAttr (line 99) | connWriterUseTLSAttr = "tls"
constant connWriterInsecureSkipVerifyAttr (line 100) | connWriterInsecureSkipVerifyAttr = "insecureskipverify"
type CustomReceiverProducer (line 105) | type CustomReceiverProducer
type CfgParseParams (line 109) | type CfgParseParams struct
method String (line 133) | func (cfg *CfgParseParams) String() string {
type elementMapEntry (line 137) | type elementMapEntry struct
function init (line 144) | func init() {
function fillPredefinedFormats (line 163) | func fillPredefinedFormats() error {
function configFromXMLDecoder (line 197) | func configFromXMLDecoder(xmlParser *xml.Decoder, rootNode xml.Token) (*...
function configFromXMLDecoderWithConfig (line 204) | func configFromXMLDecoderWithConfig(xmlParser *xml.Decoder, rootNode xml...
function configFromReader (line 224) | func configFromReader(reader io.Reader) (*logConfig, error) {
function configFromReaderWithConfig (line 231) | func configFromReaderWithConfig(reader io.Reader, cfg *CfgParseParams) (...
function configFromXMLNodeWithConfig (line 244) | func configFromXMLNodeWithConfig(config *xmlNode, cfg *CfgParseParams) (...
function getConstraints (line 303) | func getConstraints(node *xmlNode) (logLevelConstraints, error) {
function parseLevels (line 350) | func parseLevels(str string) ([]LogLevel, error) {
function getExceptions (line 365) | func getExceptions(config *xmlNode) ([]*logLevelException, error) {
function checkDistinctExceptions (line 425) | func checkDistinctExceptions(exceptions []*logLevelException) error {
function getFormats (line 444) | func getFormats(config *xmlNode) (map[string]*formatter, error) {
function getloggerTypeFromStringData (line 499) | func getloggerTypeFromStringData(config *xmlNode) (logType loggerTypeFro...
function getOutputsTree (line 562) | func getOutputsTree(config *xmlNode, formats map[string]*formatter, cfg ...
function getCurrentFormat (line 600) | func getCurrentFormat(node *xmlNode, formatFromParent *formatter, format...
function createInnerReceivers (line 621) | func createInnerReceivers(node *xmlNode, format *formatter, formats map[...
function createSplitter (line 640) | func createSplitter(node *xmlNode, formatFromParent *formatter, formats ...
function createCustomReceiver (line 663) | func createCustomReceiver(node *xmlNode, formatFromParent *formatter, fo...
function createFilter (line 717) | func createFilter(node *xmlNode, formatFromParent *formatter, formats ma...
function createfileWriter (line 750) | func createfileWriter(node *xmlNode, formatFromParent *formatter, format...
function createSMTPWriter (line 779) | func createSMTPWriter(node *xmlNode, formatFromParent *formatter, format...
function createConsoleWriter (line 889) | func createConsoleWriter(node *xmlNode, formatFromParent *formatter, for...
function createconnWriter (line 912) | func createconnWriter(node *xmlNode, formatFromParent *formatter, format...
function createRollingFileWriter (line 982) | func createRollingFileWriter(node *xmlNode, formatFromParent *formatter,...
function createbufferedWriter (line 1112) | func createbufferedWriter(node *xmlNode, formatFromParent *formatter, fo...
function checkUnexpectedAttribute (line 1171) | func checkUnexpectedAttribute(node *xmlNode, expectedAttrs ...string) er...
type expectedElementInfo (line 1188) | type expectedElementInfo struct
function optionalElement (line 1194) | func optionalElement(name string) expectedElementInfo {
function mandatoryElement (line 1197) | func mandatoryElement(name string) expectedElementInfo {
function multipleElements (line 1200) | func multipleElements(name string) expectedElementInfo {
function multipleMandatoryElements (line 1203) | func multipleMandatoryElements(name string) expectedElementInfo {
function checkExpectedElements (line 1207) | func checkExpectedElements(node *xmlNode, elements ...expectedElementInf...
FILE: vendor/github.com/cihub/seelog/cfg_parser_test.go
type customTestReceiverOutput (line 35) | type customTestReceiverOutput struct
type customTestReceiver (line 43) | type customTestReceiver struct
method ReceiveMessage (line 45) | func (cr *customTestReceiver) ReceiveMessage(message string, level Log...
method String (line 51) | func (cr *customTestReceiver) String() string {
method AfterParse (line 55) | func (cr *customTestReceiver) AfterParse(initArgs CustomReceiverInitAr...
method Flush (line 62) | func (cr *customTestReceiver) Flush() {
method Close (line 66) | func (cr *customTestReceiver) Close() error {
function getTestFileName (line 73) | func getTestFileName(testName, postfix string) string {
type parserTest (line 82) | type parserTest struct
function getParserTests (line 90) | func getParserTests() []parserTest {
function configsAreEqual (line 1024) | func configsAreEqual(conf1 *logConfig, conf2 interface{}) bool {
function testLogFileFilter (line 1040) | func testLogFileFilter(fn string) bool {
function cleanupAfterCfgTest (line 1044) | func cleanupAfterCfgTest(t *testing.T) {
function parseTest (line 1058) | func parseTest(test parserTest, t *testing.T) {
function TestParser (line 1083) | func TestParser(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/common_constraints.go
type logLevelConstraints (line 34) | type logLevelConstraints interface
type minMaxConstraints (line 39) | type minMaxConstraints struct
method IsAllowed (line 60) | func (minMaxConstr *minMaxConstraints) IsAllowed(level LogLevel) bool {
method String (line 64) | func (minMaxConstr *minMaxConstraints) String() string {
function newMinMaxConstraints (line 45) | func newMinMaxConstraints(min LogLevel, max LogLevel) (*minMaxConstraint...
type listConstraints (line 71) | type listConstraints struct
method String (line 93) | func (listConstr *listConstraints) String() string {
method IsAllowed (line 132) | func (listConstr *listConstraints) IsAllowed(level LogLevel) bool {
method AllowedLevels (line 143) | func (listConstr *listConstraints) AllowedLevels() map[LogLevel]bool {
function newListConstraints (line 76) | func newListConstraints(allowList []LogLevel) (*listConstraints, error) {
function createMapFromList (line 112) | func createMapFromList(allowedList []LogLevel) (map[LogLevel]bool, error) {
function validateOffLevel (line 122) | func validateOffLevel(allowedLevels map[LogLevel]bool) error {
type offConstraints (line 149) | type offConstraints struct
method IsAllowed (line 156) | func (offConstr *offConstraints) IsAllowed(level LogLevel) bool {
method String (line 160) | func (offConstr *offConstraints) String() string {
function newOffConstraints (line 152) | func newOffConstraints() (*offConstraints, error) {
FILE: vendor/github.com/cihub/seelog/common_constraints_test.go
function TestInvalidminMaxConstraints (line 31) | func TestInvalidminMaxConstraints(t *testing.T) {
function TestInvalidLogLevels (line 41) | func TestInvalidLogLevels(t *testing.T) {
function TestlistConstraintsWithDuplicates (line 63) | func TestlistConstraintsWithDuplicates(t *testing.T) {
function TestlistConstraintsWithOffInList (line 88) | func TestlistConstraintsWithOffInList(t *testing.T) {
type logLevelTestCase (line 100) | type logLevelTestCase struct
function TestValidminMaxConstraints (line 116) | func TestValidminMaxConstraints(t *testing.T) {
function TestValidlistConstraints (line 147) | func TestValidlistConstraints(t *testing.T) {
function TestValidListoffConstraints (line 178) | func TestValidListoffConstraints(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/common_context.go
function init (line 38) | func init() {
function setWorkDir (line 42) | func setWorkDir() {
type LogContextInterface (line 53) | type LogContextInterface interface
function currentContext (line 73) | func currentContext() (LogContextInterface, error) {
function extractCallerInfo (line 77) | func extractCallerInfo(skip int) (fullPath string, shortPath string, fun...
function specificContext (line 112) | func specificContext(skip int) (LogContextInterface, error) {
type logContext (line 129) | type logContext struct
method IsValid (line 138) | func (context *logContext) IsValid() bool {
method Func (line 142) | func (context *logContext) Func() string {
method Line (line 146) | func (context *logContext) Line() int {
method ShortPath (line 150) | func (context *logContext) ShortPath() string {
method FullPath (line 154) | func (context *logContext) FullPath() string {
method FileName (line 158) | func (context *logContext) FileName() string {
method CallTime (line 162) | func (context *logContext) CallTime() time.Time {
constant errorContextFunc (line 167) | errorContextFunc = "Func() error:"
constant errorContextShortPath (line 168) | errorContextShortPath = "ShortPath() error:"
constant errorContextFullPath (line 169) | errorContextFullPath = "FullPath() error:"
constant errorContextFileName (line 170) | errorContextFileName = "FileName() error:"
type errorContext (line 174) | type errorContext struct
method IsValid (line 179) | func (errContext *errorContext) IsValid() bool {
method Line (line 183) | func (errContext *errorContext) Line() int {
method Func (line 187) | func (errContext *errorContext) Func() string {
method ShortPath (line 191) | func (errContext *errorContext) ShortPath() string {
method FullPath (line 195) | func (errContext *errorContext) FullPath() string {
method FileName (line 199) | func (errContext *errorContext) FileName() string {
method CallTime (line 203) | func (errContext *errorContext) CallTime() time.Time {
FILE: vendor/github.com/cihub/seelog/common_context_test.go
constant shortPath (line 35) | shortPath = "common_context_test.go"
function init (line 38) | func init() {
function fullPath (line 48) | func fullPath(t *testing.T) string {
function TestContext (line 62) | func TestContext(t *testing.T) {
function innerContext (line 94) | func innerContext() (context LogContextInterface, err error) {
function TestInnerContext (line 98) | func TestInnerContext(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/common_exception.go
type logLevelException (line 42) | type logLevelException struct
method MatchesContext (line 78) | func (logLevelEx *logLevelException) MatchesContext(context LogContext...
method IsAllowed (line 83) | func (logLevelEx *logLevelException) IsAllowed(level LogLevel) bool {
method FuncPattern (line 88) | func (logLevelEx *logLevelException) FuncPattern() string {
method FilePattern (line 93) | func (logLevelEx *logLevelException) FilePattern() string {
method initFuncPatternParts (line 98) | func (logLevelEx *logLevelException) initFuncPatternParts(funcPattern ...
method initFilePatternParts (line 109) | func (logLevelEx *logLevelException) initFilePatternParts(filePattern ...
method match (line 119) | func (logLevelEx *logLevelException) match(funcPath string, filePath s...
method String (line 126) | func (logLevelEx *logLevelException) String() string {
function newLogLevelException (line 53) | func newLogLevelException(funcPattern string, filePattern string, constr...
function splitPattern (line 139) | func splitPattern(pattern string) []string {
function stringMatchesPattern (line 162) | func stringMatchesPattern(patternparts []string, testString string) bool {
FILE: vendor/github.com/cihub/seelog/common_exception_test.go
type exceptionTestCase (line 31) | type exceptionTestCase struct
function TestMatchingCorrectness (line 55) | func TestMatchingCorrectness(t *testing.T) {
function TestAsterisksReducing (line 77) | func TestAsterisksReducing(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/common_flusher.go
type flusherInterface (line 29) | type flusherInterface interface
FILE: vendor/github.com/cihub/seelog/common_loglevel.go
type LogLevel (line 28) | type LogLevel
method String (line 74) | func (level LogLevel) String() string {
constant TraceLvl (line 32) | TraceLvl = iota
constant DebugLvl (line 33) | DebugLvl
constant InfoLvl (line 34) | InfoLvl
constant WarnLvl (line 35) | WarnLvl
constant ErrorLvl (line 36) | ErrorLvl
constant CriticalLvl (line 37) | CriticalLvl
constant Off (line 38) | Off
constant TraceStr (line 43) | TraceStr = "trace"
constant DebugStr (line 44) | DebugStr = "debug"
constant InfoStr (line 45) | InfoStr = "info"
constant WarnStr (line 46) | WarnStr = "warn"
constant ErrorStr (line 47) | ErrorStr = "error"
constant CriticalStr (line 48) | CriticalStr = "critical"
constant OffStr (line 49) | OffStr = "off"
function LogLevelFromString (line 63) | func LogLevelFromString(levelStr string) (level LogLevel, found bool) {
FILE: vendor/github.com/cihub/seelog/dispatch_custom.go
function RegisterReceiver (line 59) | func RegisterReceiver(name string, receiver CustomReceiver) {
function customReceiverByName (line 67) | func customReceiverByName(name string) (creceiver CustomReceiver, err er...
type CustomReceiverInitArgs (line 81) | type CustomReceiverInitArgs struct
type CustomReceiver (line 101) | type CustomReceiver interface
type customReceiverDispatcher (line 147) | type customReceiverDispatcher struct
method Dispatch (line 192) | func (disp *customReceiverDispatcher) Dispatch(
method Flush (line 211) | func (disp *customReceiverDispatcher) Flush() {
method Close (line 216) | func (disp *customReceiverDispatcher) Close() error {
method String (line 227) | func (disp *customReceiverDispatcher) String() string {
function newCustomReceiverDispatcher (line 156) | func newCustomReceiverDispatcher(formatter *formatter, customReceiverNam...
function newCustomReceiverDispatcherByValue (line 179) | func newCustomReceiverDispatcherByValue(formatter *formatter, customRece...
FILE: vendor/github.com/cihub/seelog/dispatch_customdispatcher_test.go
type testCustomDispatcherMessageReceiver (line 31) | type testCustomDispatcherMessageReceiver struct
function TestCustomDispatcher_Message (line 35) | func TestCustomDispatcher_Message(t *testing.T) {
type testCustomDispatcherFlushReceiver (line 85) | type testCustomDispatcherFlushReceiver struct
function TestCustomDispatcher_Flush (line 89) | func TestCustomDispatcher_Flush(t *testing.T) {
type testCustomDispatcherCloseReceiver (line 132) | type testCustomDispatcherCloseReceiver struct
function TestCustomDispatcher_Close (line 136) | func TestCustomDispatcher_Close(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/dispatch_dispatcher.go
type dispatcherInterface (line 37) | type dispatcherInterface interface
type dispatcher (line 43) | type dispatcher struct
method Dispatch (line 89) | func (disp *dispatcher) Dispatch(
method Flush (line 109) | func (disp *dispatcher) Flush() {
method Close (line 126) | func (disp *dispatcher) Close() error {
method Writers (line 153) | func (disp *dispatcher) Writers() []*formattedWriter {
method Dispatchers (line 157) | func (disp *dispatcher) Dispatchers() []dispatcherInterface {
method String (line 161) | func (disp *dispatcher) String() string {
function createDispatcher (line 51) | func createDispatcher(formatter *formatter, receivers []interface{}) (*d...
FILE: vendor/github.com/cihub/seelog/dispatch_filterdispatcher.go
type filterDispatcher (line 33) | type filterDispatcher struct
method Dispatch (line 53) | func (filter *filterDispatcher) Dispatch(
method String (line 64) | func (filter *filterDispatcher) String() string {
function newFilterDispatcher (line 39) | func newFilterDispatcher(formatter *formatter, receivers []interface{}, ...
FILE: vendor/github.com/cihub/seelog/dispatch_filterdispatcher_test.go
function TestfilterDispatcher_Pass (line 31) | func TestfilterDispatcher_Pass(t *testing.T) {
function TestfilterDispatcher_Deny (line 51) | func TestfilterDispatcher_Deny(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/dispatch_splitdispatcher.go
type splitDispatcher (line 32) | type splitDispatcher struct
method String (line 45) | func (splitter *splitDispatcher) String() string {
function newSplitDispatcher (line 36) | func newSplitDispatcher(formatter *formatter, receivers []interface{}) (...
FILE: vendor/github.com/cihub/seelog/dispatch_splitdispatcher_test.go
function init (line 34) | func init() {
function TestsplitDispatcher (line 42) | func TestsplitDispatcher(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/format.go
constant FormatterSymbol (line 38) | FormatterSymbol = '%'
constant formatterSymbolString (line 41) | formatterSymbolString = "%"
constant formatterParameterStart (line 42) | formatterParameterStart = '('
constant formatterParameterEnd (line 43) | formatterParameterEnd = ')'
constant DateDefaultFormat (line 48) | DateDefaultFormat = "2006-01-02"
constant TimeFormat (line 49) | TimeFormat = "15:04:05"
function init (line 57) | func init() {
type FormatterFunc (line 78) | type FormatterFunc
type FormatterFuncCreator (line 82) | type FormatterFuncCreator
function errorAliasReserved (line 111) | func errorAliasReserved(name string) error {
function RegisterCustomFormatter (line 130) | func RegisterCustomFormatter(name string, creator FormatterFuncCreator) ...
type formatter (line 143) | type formatter struct
method buildFormatterFuncs (line 162) | func (formatter *formatter) buildFormatterFuncs() error {
method extractFormatterFunc (line 198) | func (formatter *formatter) extractFormatterFunc(index int) (Formatter...
method extractLetterSequence (line 220) | func (formatter *formatter) extractLetterSequence(index int) string {
method findFormatterFunc (line 238) | func (formatter *formatter) findFormatterFunc(letters string) (Formatt...
method findFormatterFuncParametrized (line 251) | func (formatter *formatter) findFormatterFuncParametrized(letters stri...
method findparameter (line 279) | func (formatter *formatter) findparameter(startIndex int) (string, int...
method Format (line 298) | func (formatter *formatter) Format(message string, level LogLevel, con...
method String (line 311) | func (formatter *formatter) String() string {
function newFormatter (line 150) | func newFormatter(formatString string) (*formatter, error) {
constant wrongLogLevel (line 318) | wrongLogLevel = "WRONG_LOGLEVEL"
constant wrongEscapeCode (line 319) | wrongEscapeCode = "WRONG_ESCAPE"
function formatterLevel (line 352) | func formatterLevel(message string, level LogLevel, context LogContextIn...
function formatterLev (line 360) | func formatterLev(message string, level LogLevel, context LogContextInte...
function formatterLEVEL (line 368) | func formatterLEVEL(message string, level LogLevel, context LogContextIn...
function formatterLEV (line 372) | func formatterLEV(message string, level LogLevel, context LogContextInte...
function formatterl (line 376) | func formatterl(message string, level LogLevel, context LogContextInterf...
function formatterMsg (line 384) | func formatterMsg(message string, level LogLevel, context LogContextInte...
function formatterFullPath (line 388) | func formatterFullPath(message string, level LogLevel, context LogContex...
function formatterFile (line 392) | func formatterFile(message string, level LogLevel, context LogContextInt...
function formatterRelFile (line 396) | func formatterRelFile(message string, level LogLevel, context LogContext...
function FormatterFunction (line 400) | func FormatterFunction(message string, level LogLevel, context LogContex...
function FormatterFunctionShort (line 404) | func FormatterFunctionShort(message string, level LogLevel, context LogC...
function formatterLine (line 410) | func formatterLine(message string, level LogLevel, context LogContextInt...
function formatterTime (line 414) | func formatterTime(message string, level LogLevel, context LogContextInt...
function formatterUTCTime (line 418) | func formatterUTCTime(message string, level LogLevel, context LogContext...
function formatterNs (line 422) | func formatterNs(message string, level LogLevel, context LogContextInter...
function formatterUTCNs (line 426) | func formatterUTCNs(message string, level LogLevel, context LogContextIn...
function formattern (line 430) | func formattern(message string, level LogLevel, context LogContextInterf...
function formattert (line 434) | func formattert(message string, level LogLevel, context LogContextInterf...
function createDateTimeFormatterFunc (line 438) | func createDateTimeFormatterFunc(dateTimeFormat string) FormatterFunc {
function createUTCDateTimeFormatterFunc (line 448) | func createUTCDateTimeFormatterFunc(dateTimeFormat string) FormatterFunc {
function createANSIEscapeFunc (line 458) | func createANSIEscapeFunc(escapeCodeString string) FormatterFunc {
FILE: vendor/github.com/cihub/seelog/format_test.go
constant TestFuncName (line 35) | TestFuncName = "TestFormats"
type formatTest (line 38) | type formatTest struct
function TestFormats (line 128) | func TestFormats(t *testing.T) {
function TestDateFormat (line 160) | func TestDateFormat(t *testing.T) {
function TestDateParameterizedFormat (line 167) | func TestDateParameterizedFormat(t *testing.T) {
function createTestFormatter (line 196) | func createTestFormatter(format string) FormatterFunc {
function TestCustomFormatterRegistration (line 202) | func TestCustomFormatterRegistration(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/internals_baseerror.go
type baseError (line 4) | type baseError struct
method Error (line 8) | func (be baseError) Error() string {
FILE: vendor/github.com/cihub/seelog/internals_byteverifiers_test.go
type bytesVerifier (line 35) | type bytesVerifier struct
method Write (line 55) | func (verifier *bytesVerifier) Write(bytes []byte) (n int, err error) {
method ExpectBytes (line 86) | func (verifier *bytesVerifier) ExpectBytes(bytes []byte) {
method MustNotExpect (line 91) | func (verifier *bytesVerifier) MustNotExpect() {
method Close (line 104) | func (verifier *bytesVerifier) Close() error {
function newBytesVerifier (line 42) | func newBytesVerifier(t *testing.T) (*bytesVerifier, error) {
type nullWriter (line 109) | type nullWriter struct
method Write (line 112) | func (writer *nullWriter) Write(bytes []byte) (n int, err error) {
method Close (line 116) | func (writer *nullWriter) Close() error {
FILE: vendor/github.com/cihub/seelog/internals_fsutils.go
constant defaultFilePermissions (line 16) | defaultFilePermissions = 0666
constant defaultDirectoryPermissions (line 17) | defaultDirectoryPermissions = 0767
constant maxDirNumberReadAsync (line 22) | maxDirNumberReadAsync = 1000
type cannotOpenFileError (line 25) | type cannotOpenFileError struct
function newCannotOpenFileError (line 29) | func newCannotOpenFileError(fname string) *cannotOpenFileError {
type notDirectoryError (line 33) | type notDirectoryError struct
function newNotDirectoryError (line 37) | func newNotDirectoryError(dname string) *notDirectoryError {
type fileFilter (line 43) | type fileFilter
type filePathFilter (line 47) | type filePathFilter
function getSubdirNames (line 51) | func getSubdirNames(dirPath string) ([]string, error) {
function getAllSubdirAbsPaths (line 84) | func getAllSubdirAbsPaths(dirPath string) (res []string, err error) {
function getSubdirAbsPaths (line 105) | func getSubdirAbsPaths(dirPath string) ([]string, error) {
function getOpenFilesInDir (line 119) | func getOpenFilesInDir(dirPath string, fFilter fileFilter) ([]*os.File, ...
function isRegular (line 168) | func isRegular(m os.FileMode) bool {
function getDirFilePaths (line 174) | func getDirFilePaths(dirPath string, fpFilter filePathFilter, pathIsName...
function getOpenFilesByDirectoryAsync (line 235) | func getOpenFilesByDirectoryAsync(
function copyFile (line 278) | func copyFile(sf *os.File, dst string) (int64, error) {
function fileExists (line 289) | func fileExists(path string) (bool, error) {
function createDirectory (line 302) | func createDirectory(dirPath string) error {
function tryRemoveFile (line 325) | func tryRemoveFile(filePath string) (err error) {
function unzip (line 335) | func unzip(archiveName string) (map[string][]byte, error) {
function createZip (line 372) | func createZip(archiveName string, files map[string][]byte) error {
FILE: vendor/github.com/cihub/seelog/internals_xmlnode.go
type xmlNode (line 35) | type xmlNode struct
method String (line 49) | func (node *xmlNode) String() string {
method unmarshal (line 70) | func (node *xmlNode) unmarshal(startEl xml.StartElement) error {
method add (line 84) | func (node *xmlNode) add(child *xmlNode) {
method hasChildren (line 92) | func (node *xmlNode) hasChildren() bool {
function newNode (line 42) | func newNode() *xmlNode {
function unmarshalConfig (line 98) | func unmarshalConfig(reader io.Reader) (*xmlNode, error) {
function unmarshalNode (line 117) | func unmarshalNode(xmlParser *xml.Decoder, curToken xml.Token) (node *xm...
function getNextToken (line 165) | func getNextToken(xmlParser *xml.Decoder) (tok xml.Token, err error) {
FILE: vendor/github.com/cihub/seelog/internals_xmlnode_test.go
type xmlNodeTest (line 65) | type xmlNodeTest struct
function getXMLTests (line 72) | func getXMLTests() []xmlNodeTest {
function TestXmlNode (line 175) | func TestXmlNode(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/log.go
constant staticFuncCallDepth (line 35) | staticFuncCallDepth = 3
constant loggerFuncCallDepth (line 36) | loggerFuncCallDepth = 3
function init (line 50) | func init() {
function createLoggerFromConfig (line 69) | func createLoggerFromConfig(config *logConfig) (LoggerInterface, error) {
function UseLogger (line 134) | func UseLogger(logger LoggerInterface) error {
function ReplaceLogger (line 171) | func ReplaceLogger(logger LoggerInterface) error {
function Tracef (line 199) | func Tracef(format string, params ...interface{}) {
function Debugf (line 207) | func Debugf(format string, params ...interface{}) {
function Infof (line 215) | func Infof(format string, params ...interface{}) {
function Warnf (line 222) | func Warnf(format string, params ...interface{}) error {
function Errorf (line 231) | func Errorf(format string, params ...interface{}) error {
function Criticalf (line 240) | func Criticalf(format string, params ...interface{}) error {
function Trace (line 249) | func Trace(v ...interface{}) {
function Debug (line 256) | func Debug(v ...interface{}) {
function Info (line 263) | func Info(v ...interface{}) {
function Warn (line 270) | func Warn(v ...interface{}) error {
function Error (line 279) | func Error(v ...interface{}) error {
function Critical (line 288) | func Critical(v ...interface{}) error {
function Flush (line 303) | func Flush() {
FILE: vendor/github.com/cihub/seelog/logger.go
function reportInternalError (line 34) | func reportInternalError(err error) {
type LoggerInterface (line 39) | type LoggerInterface interface
type innerLoggerInterface (line 119) | type innerLoggerInterface interface
type allowedContextCache (line 125) | type allowedContextCache
type commonLogger (line 128) | type commonLogger struct
method SetAdditionalStackDepth (line 150) | func (cLogger *commonLogger) SetAdditionalStackDepth(depth int) error {
method Tracef (line 160) | func (cLogger *commonLogger) Tracef(format string, params ...interface...
method Debugf (line 164) | func (cLogger *commonLogger) Debugf(format string, params ...interface...
method Infof (line 168) | func (cLogger *commonLogger) Infof(format string, params ...interface{...
method Warnf (line 172) | func (cLogger *commonLogger) Warnf(format string, params ...interface{...
method Errorf (line 178) | func (cLogger *commonLogger) Errorf(format string, params ...interface...
method Criticalf (line 184) | func (cLogger *commonLogger) Criticalf(format string, params ...interf...
method Trace (line 190) | func (cLogger *commonLogger) Trace(v ...interface{}) {
method Debug (line 194) | func (cLogger *commonLogger) Debug(v ...interface{}) {
method Info (line 198) | func (cLogger *commonLogger) Info(v ...interface{}) {
method Warn (line 202) | func (cLogger *commonLogger) Warn(v ...interface{}) error {
method Error (line 208) | func (cLogger *commonLogger) Error(v ...interface{}) error {
method Critical (line 214) | func (cLogger *commonLogger) Critical(v ...interface{}) error {
method traceWithCallDepth (line 220) | func (cLogger *commonLogger) traceWithCallDepth(callDepth int, message...
method debugWithCallDepth (line 224) | func (cLogger *commonLogger) debugWithCallDepth(callDepth int, message...
method infoWithCallDepth (line 228) | func (cLogger *commonLogger) infoWithCallDepth(callDepth int, message ...
method warnWithCallDepth (line 232) | func (cLogger *commonLogger) warnWithCallDepth(callDepth int, message ...
method errorWithCallDepth (line 236) | func (cLogger *commonLogger) errorWithCallDepth(callDepth int, message...
method criticalWithCallDepth (line 240) | func (cLogger *commonLogger) criticalWithCallDepth(callDepth int, mess...
method Closed (line 245) | func (cLogger *commonLogger) Closed() bool {
method fillUnusedLevels (line 249) | func (cLogger *commonLogger) fillUnusedLevels() {
method fillUnusedLevelsByContraint (line 261) | func (cLogger *commonLogger) fillUnusedLevelsByContraint(constraint lo...
method log (line 272) | func (cLogger *commonLogger) log(
method processLogMsg (line 301) | func (cLogger *commonLogger) processLogMsg(
method isAllowed (line 317) | func (cLogger *commonLogger) isAllowed(level LogLevel, context LogCont...
function newCommonLogger (line 138) | func newCommonLogger(config *logConfig, internalLogger innerLoggerInterf...
type logMessage (line 339) | type logMessage struct
method String (line 365) | func (message *logMessage) String() string {
type logFormattedMessage (line 343) | type logFormattedMessage struct
method String (line 369) | func (message *logFormattedMessage) String() string {
function newLogMessage (line 348) | func newLogMessage(params []interface{}) fmt.Stringer {
function newLogFormattedMessage (line 356) | func newLogFormattedMessage(format string, params []interface{}) *logFor...
FILE: vendor/github.com/cihub/seelog/writers_bufferedwriter.go
type bufferedWriter (line 37) | type bufferedWriter struct
method writeBigChunk (line 82) | func (bufWriter *bufferedWriter) writeBigChunk(bytes []byte) (n int, e...
method Write (line 95) | func (bufWriter *bufferedWriter) Write(bytes []byte) (n int, err error) {
method Close (line 118) | func (bufWriter *bufferedWriter) Close() error {
method Flush (line 127) | func (bufWriter *bufferedWriter) Flush() {
method flushInner (line 135) | func (bufWriter *bufferedWriter) flushInner() (n int, err error) {
method flushBuffer (line 142) | func (bufWriter *bufferedWriter) flushBuffer() {
method flushPeriodically (line 149) | func (bufWriter *bufferedWriter) flushPeriodically() {
method String (line 159) | func (bufWriter *bufferedWriter) String() string {
function newBufferedWriter (line 48) | func newBufferedWriter(innerWriter io.Writer, bufferSize int, flushPerio...
FILE: vendor/github.com/cihub/seelog/writers_bufferedwriter_test.go
function TestChunkWriteOnFilling (line 31) | func TestChunkWriteOnFilling(t *testing.T) {
function TestFlushByTimePeriod (line 46) | func TestFlushByTimePeriod(t *testing.T) {
function TestBigMessageMustPassMemoryBuffer (line 62) | func TestBigMessageMustPassMemoryBuffer(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/writers_connwriter.go
type connWriter (line 35) | type connWriter struct
method Close (line 70) | func (connWriter *connWriter) Close() error {
method Write (line 78) | func (connWriter *connWriter) Write(bytes []byte) (n int, err error) {
method String (line 98) | func (connWriter *connWriter) String() string {
method connect (line 102) | func (connWriter *connWriter) connect() error {
method neededConnectOnMsg (line 133) | func (connWriter *connWriter) neededConnectOnMsg() bool {
function newConnWriter (line 47) | func newConnWriter(netName string, addr string, reconnectOnMsg bool) *co...
function newTLSWriter (line 58) | func newTLSWriter(netName string, addr string, reconnectOnMsg bool, conf...
FILE: vendor/github.com/cihub/seelog/writers_consolewriter.go
type consoleWriter (line 30) | type consoleWriter struct
method Write (line 41) | func (console *consoleWriter) Write(bytes []byte) (int, error) {
method String (line 45) | func (console *consoleWriter) String() string {
function newConsoleWriter (line 34) | func newConsoleWriter() (writer *consoleWriter, err error) {
FILE: vendor/github.com/cihub/seelog/writers_filewriter.go
type fileWriter (line 35) | type fileWriter struct
method Close (line 48) | func (fw *fileWriter) Close() error {
method Write (line 60) | func (fw *fileWriter) Write(bytes []byte) (n int, err error) {
method createFile (line 69) | func (fw *fileWriter) createFile() error {
method String (line 90) | func (fw *fileWriter) String() string {
function newFileWriter (line 41) | func newFileWriter(fileName string) (writer *fileWriter, err error) {
FILE: vendor/github.com/cihub/seelog/writers_filewriter_test.go
constant messageLen (line 37) | messageLen = 10
function TestSimpleFileWriter (line 42) | func TestSimpleFileWriter(t *testing.T) {
function simplefileWriterGetter (line 49) | func simplefileWriterGetter(testCase *fileWriterTestCase) (io.WriteClose...
type fileWriterTestCase (line 54) | type fileWriterTestCase struct
function createSimplefileWriterTestCase (line 66) | func createSimplefileWriterTestCase(fileName string, writeCount int) *fi...
type fileWriterTester (line 78) | type fileWriterTester struct
method testCase (line 137) | func (tester *fileWriterTester) testCase(testCase *fileWriterTestCase,...
method test (line 188) | func (tester *fileWriterTester) test() {
method performWrite (line 195) | func (tester *fileWriterTester) performWrite(fileWriter io.Writer, cou...
method checkRequiredFilesExist (line 206) | func (tester *fileWriterTester) checkRequiredFilesExist(testCase *file...
method checkJustRequiredFilesExist (line 234) | func (tester *fileWriterTester) checkJustRequiredFilesExist(testCase *...
function newFileWriterTester (line 84) | func newFileWriterTester(
function isWriterTestFile (line 92) | func isWriterTestFile(fn string) bool {
function cleanupWriterTest (line 96) | func cleanupWriterTest(t *testing.T) {
function getWriterTestResultFiles (line 113) | func getWriterTestResultFiles() ([]string, error) {
FILE: vendor/github.com/cihub/seelog/writers_formattedwriter.go
type formattedWriter (line 33) | type formattedWriter struct
method Write (line 46) | func (formattedWriter *formattedWriter) Write(message string, level Lo...
method String (line 52) | func (formattedWriter *formattedWriter) String() string {
method Writer (line 56) | func (formattedWriter *formattedWriter) Writer() io.Writer {
method Format (line 60) | func (formattedWriter *formattedWriter) Format() *formatter {
function newFormattedWriter (line 38) | func newFormattedWriter(writer io.Writer, formatter *formatter) (*format...
FILE: vendor/github.com/cihub/seelog/writers_formattedwriter_test.go
function TestformattedWriter (line 31) | func TestformattedWriter(t *testing.T) {
FILE: vendor/github.com/cihub/seelog/writers_rollingfilewriter.go
constant rollingLogHistoryDelimiter (line 40) | rollingLogHistoryDelimiter = "."
type rollingType (line 44) | type rollingType
constant rollingTypeSize (line 47) | rollingTypeSize = iota
constant rollingTypeTime (line 48) | rollingTypeTime
type rollingNameMode (line 52) | type rollingNameMode
constant rollingNameModePostfix (line 55) | rollingNameModePostfix = iota
constant rollingNameModePrefix (line 56) | rollingNameModePrefix
function rollingNameModeFromString (line 64) | func rollingNameModeFromString(rollingNameStr string) (rollingNameMode, ...
type rollingIntervalType (line 74) | type rollingIntervalType
constant rollingIntervalAny (line 77) | rollingIntervalAny = iota
constant rollingIntervalDaily (line 78) | rollingIntervalDaily
function rollingIntervalTypeFromString (line 85) | func rollingIntervalTypeFromString(rollingTypeStr string) (rollingInterv...
function rollingTypeFromString (line 100) | func rollingTypeFromString(rollingTypeStr string) (rollingType, bool) {
type rollingArchiveType (line 111) | type rollingArchiveType
constant rollingArchiveNone (line 114) | rollingArchiveNone = iota
constant rollingArchiveZip (line 115) | rollingArchiveZip
function rollingArchiveTypeFromString (line 123) | func rollingArchiveTypeFromString(rollingArchiveTypeStr string) (rolling...
type rollerVirtual (line 140) | type rollerVirtual interface
type rollingFileWriter (line 156) | type rollingFileWriter struct
method hasRollName (line 186) | func (rw *rollingFileWriter) hasRollName(file string) bool {
method createFullFileName (line 198) | func (rw *rollingFileWriter) createFullFileName(originalName, rollname...
method getSortedLogHistory (line 208) | func (rw *rollingFileWriter) getSortedLogHistory() ([]string, error) {
method createFileAndFolderIfNeeded (line 233) | func (rw *rollingFileWriter) createFileAndFolderIfNeeded() error {
method deleteOldRolls (line 269) | func (rw *rollingFileWriter) deleteOldRolls(history []string) error {
method getFileRollName (line 331) | func (rw *rollingFileWriter) getFileRollName(fileName string) string {
method Write (line 341) | func (rw *rollingFileWriter) Write(bytes []byte) (n int, err error) {
method Close (line 432) | func (rw *rollingFileWriter) Close() error {
function newRollingFileWriter (line 170) | func newRollingFileWriter(fpath string, rtype rollingType, atype rolling...
type rollingFileWriterSize (line 452) | type rollingFileWriterSize struct
method needsToRoll (line 467) | func (rws *rollingFileWriterSize) needsToRoll() (bool, error) {
method isFileRollNameValid (line 471) | func (rws *rollingFileWriterSize) isFileRollNameValid(rname string) bo...
method sortFileRollNamesAsc (line 489) | func (rws *rollingFileWriterSize) sortFileRollNamesAsc(fs []string) ([...
method getNewHistoryRollFileName (line 495) | func (rws *rollingFileWriterSize) getNewHistoryRollFileName(lastRollNa...
method getCurrentModifiedFileName (line 503) | func (rws *rollingFileWriterSize) getCurrentModifiedFileName(originalF...
method String (line 507) | func (rws *rollingFileWriterSize) String() string {
function newRollingFileWriterSize (line 457) | func newRollingFileWriterSize(fpath string, atype rollingArchiveType, ap...
type rollSizeFileTailsSlice (line 479) | type rollSizeFileTailsSlice
method Len (line 481) | func (p rollSizeFileTailsSlice) Len() int { return len(p) }
method Less (line 482) | func (p rollSizeFileTailsSlice) Less(i, j int) bool {
method Swap (line 487) | func (p rollSizeFileTailsSlice) Swap(i, j int) { p[i], p[j] = p[j], p[...
type rollingFileWriterTime (line 521) | type rollingFileWriterTime struct
method needsToRoll (line 540) | func (rwt *rollingFileWriterTime) needsToRoll() (bool, error) {
method isFileRollNameValid (line 568) | func (rwt *rollingFileWriterTime) isFileRollNameValid(rname string) bo...
method sortFileRollNamesAsc (line 589) | func (rwt *rollingFileWriterTime) sortFileRollNamesAsc(fs []string) ([...
method getNewHistoryRollFileName (line 595) | func (rwt *rollingFileWriterTime) getNewHistoryRollFileName(lastRollNa...
method getCurrentModifiedFileName (line 599) | func (rwt *rollingFileWriterTime) getCurrentModifiedFileName(originalF...
method String (line 609) | func (rwt *rollingFileWriterTime) String() string {
function newRollingFileWriterTime (line 528) | func newRollingFileWriterTime(fpath string, atype rollingArchiveType, ap...
type rollTimeFileTailsSlice (line 576) | type rollTimeFileTailsSlice struct
method Len (line 581) | func (p rollTimeFileTailsSlice) Len() int { return len(p.data) }
method Less (line 582) | func (p rollTimeFileTailsSlice) Less(i, j int) bool {
method Swap (line 587) | func (p rollTimeFileTailsSlice) Swap(i, j int) { p.data[i], p.data[j] ...
FILE: vendor/github.com/cihub/seelog/writers_rollingfilewriter_test.go
function createRollingSizeFileWriterTestCase (line 35) | func createRollingSizeFileWriterTestCase(
function createRollingDatefileWriterTestCase (line 47) | func createRollingDatefileWriterTestCase(
function TestRollingFileWriter (line 58) | func TestRollingFileWriter(t *testing.T) {
function rollingFileWriterGetter (line 65) | func rollingFileWriterGetter(testCase *fileWriterTestCase) (io.WriteClos...
FILE: vendor/github.com/cihub/seelog/writers_smtpwriter.go
constant DefaultSubjectPhrase (line 40) | DefaultSubjectPhrase = "Diagnostic message from server: "
constant rfc5321SubjectPattern (line 43) | rfc5321SubjectPattern = "From: %s <%s>\nSubject: %s\n"
type smtpWriter (line 47) | type smtpWriter struct
method Write (line 183) | func (smtpw *smtpWriter) Write(data []byte) (int, error) {
method Close (line 215) | func (smtpw *smtpWriter) Close() error {
function newSMTPWriter (line 61) | func newSMTPWriter(sa, sn string, ras []string, hn, hp, un, pwd string, ...
function prepareMessage (line 76) | func prepareMessage(senderAddr, senderName, subject string, body []byte,...
function getTLSConfig (line 90) | func getTLSConfig(pemFileDirPaths []string, hostName string) (config *tl...
function sendMailWithTLSConfig (line 137) | func sendMailWithTLSConfig(config *tls.Config, addr string, a smtp.Auth,...
FILE: vendor/github.com/golang/protobuf/proto/all_test.go
function old (line 53) | func old() *Buffer {
function equalbytes (line 61) | func equalbytes(b1, b2 []byte, t *testing.T) {
function initGoTestField (line 73) | func initGoTestField() *GoTestField {
function initGoTest_RequiredGroup (line 83) | func initGoTest_RequiredGroup() *GoTest_RequiredGroup {
function initGoTest_OptionalGroup (line 89) | func initGoTest_OptionalGroup() *GoTest_OptionalGroup {
function initGoTest_RepeatedGroup (line 95) | func initGoTest_RepeatedGroup() *GoTest_RepeatedGroup {
function initGoTest (line 101) | func initGoTest(setdefaults bool) *GoTest {
function fail (line 139) | func fail(msg string, b *bytes.Buffer, s string, t *testing.T) {
function hex (line 263) | func hex(c uint8) uint8 {
function equal (line 276) | func equal(b []byte, s string, t *testing.T) bool {
function overify (line 293) | func overify(t *testing.T, pb *GoTest, expected string) {
function TestNumericPrimitives (line 328) | func TestNumericPrimitives(t *testing.T) {
type fakeMarshaler (line 399) | type fakeMarshaler struct
method Marshal (line 404) | func (f fakeMarshaler) Marshal() ([]byte, error) {
method String (line 408) | func (f fakeMarshaler) String() string {
method ProtoMessage (line 412) | func (f fakeMarshaler) ProtoMessage() {}
method Reset (line 414) | func (f fakeMarshaler) Reset() {}
function TestMarshalerEncoding (line 417) | func TestMarshalerEncoding(t *testing.T) {
function TestBytesPrimitives (line 456) | func TestBytesPrimitives(t *testing.T) {
function TestStringPrimitives (line 470) | func TestStringPrimitives(t *testing.T) {
function TestRequiredBit (line 486) | func TestRequiredBit(t *testing.T) {
function checkInitialized (line 501) | func checkInitialized(pb *GoTest, t *testing.T) {
function TestReset (line 544) | func TestReset(t *testing.T) {
function TestEncodeDecode1 (line 565) | func TestEncodeDecode1(t *testing.T) {
function TestEncodeDecode2 (line 589) | func TestEncodeDecode2(t *testing.T) {
function TestEncodeDecode3 (line 627) | func TestEncodeDecode3(t *testing.T) {
function TestEncodeDecode4 (line 679) | func TestEncodeDecode4(t *testing.T) {
function TestEncodeDecode5 (line 754) | func TestEncodeDecode5(t *testing.T) {
function TestEncodeDecode6 (line 842) | func TestEncodeDecode6(t *testing.T) {
function TestEncodeDecodeBytes1 (line 897) | func TestEncodeDecodeBytes1(t *testing.T) {
function TestEncodeDecodeBytes2 (line 928) | func TestEncodeDecodeBytes2(t *testing.T) {
function TestSkippingUnrecognizedFields (line 950) | func TestSkippingUnrecognizedFields(t *testing.T) {
function TestSubmessageUnrecognizedFields (line 1002) | func TestSubmessageUnrecognizedFields(t *testing.T) {
function TestNegativeInt32 (line 1051) | func TestNegativeInt32(t *testing.T) {
function TestBigRepeated (line 1085) | func TestBigRepeated(t *testing.T) {
function TestTypeMismatch (line 1196) | func TestTypeMismatch(t *testing.T) {
function encodeDecode (line 1213) | func encodeDecode(t *testing.T, in, out Message, msg string) {
function TestPackedNonPackedDecoderSwitching (line 1223) | func TestPackedNonPackedDecoderSwitching(t *testing.T) {
function TestProto1RepeatedGroup (line 1242) | func TestProto1RepeatedGroup(t *testing.T) {
function TestEnum (line 1264) | func TestEnum(t *testing.T) {
function TestPrintingNilEnumFields (line 1282) | func TestPrintingNilEnumFields(t *testing.T) {
function TestRequiredFieldEnforcement (line 1288) | func TestRequiredFieldEnforcement(t *testing.T) {
function TestTypedNilMarshal (line 1309) | func TestTypedNilMarshal(t *testing.T) {
type nonNillableInt (line 1318) | type nonNillableInt
method Marshal (line 1320) | func (nni nonNillableInt) Marshal() ([]byte, error) {
type NNIMessage (line 1324) | type NNIMessage struct
method Reset (line 1328) | func (*NNIMessage) Reset() {}
method String (line 1329) | func (*NNIMessage) String() string { return "" }
method ProtoMessage (line 1330) | func (*NNIMessage) ProtoMessage() {}
type nillableMessage (line 1333) | type nillableMessage struct
method Marshal (line 1337) | func (nm *nillableMessage) Marshal() ([]byte, error) {
type NMMessage (line 1341) | type NMMessage struct
method Reset (line 1345) | func (*NMMessage) Reset() {}
method String (line 1346) | func (*NMMessage) String() string { return "" }
method ProtoMessage (line 1347) | func (*NMMessage) ProtoMessage() {}
function TestNilMarshaler (line 1350) | func TestNilMarshaler(t *testing.T) {
function TestAllSetDefaults (line 1367) | func TestAllSetDefaults(t *testing.T) {
function TestSetDefaultsWithSetField (line 1399) | func TestSetDefaultsWithSetField(t *testing.T) {
function TestSetDefaultsWithSubMessage (line 1410) | func TestSetDefaultsWithSubMessage(t *testing.T) {
function TestSetDefaultsWithRepeatedSubMessage (line 1430) | func TestSetDefaultsWithRepeatedSubMessage(t *testing.T) {
function TestSetDefaultWithRepeatedNonMessage (line 1445) | func TestSetDefaultWithRepeatedNonMessage(t *testing.T) {
function TestMaximumTagNumber (line 1456) | func TestMaximumTagNumber(t *testing.T) {
function TestJSON (line 1473) | func TestJSON(t *testing.T) {
function TestBadWireType (line 1512) | func TestBadWireType(t *testing.T) {
function TestBytesWithInvalidLength (line 1522) | func TestBytesWithInvalidLength(t *testing.T) {
function TestLengthOverflow (line 1528) | func TestLengthOverflow(t *testing.T) {
function TestVarintOverflow (line 1534) | func TestVarintOverflow(t *testing.T) {
function TestUnmarshalFuzz (line 1542) | func TestUnmarshalFuzz(t *testing.T) {
function TestMergeMessages (line 1556) | func TestMergeMessages(t *testing.T) {
function TestExtensionMarshalOrder (line 1586) | func TestExtensionMarshalOrder(t *testing.T) {
function TestMessageSetMarshalOrder (line 1669) | func TestMessageSetMarshalOrder(t *testing.T) {
function TestUnmarshalMergesMessages (line 1706) | func TestUnmarshalMergesMessages(t *testing.T) {
function TestEncodingSizes (line 1749) | func TestEncodingSizes(t *testing.T) {
function TestRequiredNotSetError (line 1771) | func TestRequiredNotSetError(t *testing.T) {
function fuzzUnmarshal (line 1835) | func fuzzUnmarshal(t *testing.T, data []byte) {
function TestMapFieldMarshal (line 1848) | func TestMapFieldMarshal(t *testing.T) {
function TestMapFieldRoundTrips (line 1893) | func TestMapFieldRoundTrips(t *testing.T) {
function testMsg (line 1930) | func testMsg() *GoTest {
function bytesMsg (line 1942) | func bytesMsg() *GoTest {
function benchmarkMarshal (line 1952) | func benchmarkMarshal(b *testing.B, pb Message, marshal func(Message) ([...
function benchmarkBufferMarshal (line 1961) | func benchmarkBufferMarshal(b *testing.B, pb Message) {
function benchmarkSize (line 1970) | func benchmarkSize(b *testing.B, pb Message) {
function newOf (line 1977) | func newOf(pb Message) Message {
function benchmarkUnmarshal (line 1985) | func benchmarkUnmarshal(b *testing.B, pb Message, unmarshal func([]byte,...
function benchmarkBufferUnmarshal (line 1996) | func benchmarkBufferUnmarshal(b *testing.B, pb Message) {
function BenchmarkMarshal (line 2006) | func BenchmarkMarshal(b *testing.B) {
function BenchmarkBufferMarshal (line 2010) | func BenchmarkBufferMarshal(b *testing.B) {
function BenchmarkSize (line 2014) | func BenchmarkSize(b *testing.B) {
function BenchmarkUnmarshal (line 2018) | func BenchmarkUnmarshal(b *testing.B) {
function BenchmarkBufferUnmarshal (line 2022) | func BenchmarkBufferUnmarshal(b *testing.B) {
function BenchmarkMarshalBytes (line 2026) | func BenchmarkMarshalBytes(b *testing.B) {
function BenchmarkBufferMarshalBytes (line 2030) | func BenchmarkBufferMarshalBytes(b *testing.B) {
function BenchmarkSizeBytes (line 2034) | func BenchmarkSizeBytes(b *testing.B) {
function BenchmarkUnmarshalBytes (line 2038) | func BenchmarkUnmarshalBytes(b *testing.B) {
function BenchmarkBufferUnmarshalBytes (line 2042) | func BenchmarkBufferUnmarshalBytes(b *testing.B) {
function BenchmarkUnmarshalUnrecognizedFields (line 2046) | func BenchmarkUnmarshalUnrecognizedFields(b *testing.B) {
FILE: vendor/github.com/golang/protobuf/proto/clone.go
function Clone (line 44) | func Clone(pb Message) Message {
function Merge (line 60) | func Merge(dst, src Message) {
function mergeStruct (line 77) | func mergeStruct(out, in reflect.Value) {
function mergeAny (line 101) | func mergeAny(out, in reflect.Value) {
function mergeExtension (line 182) | func mergeExtension(out, in map[int32]Extension) {
FILE: vendor/github.com/golang/protobuf/proto/clone_test.go
function init (line 62) | func init() {
function TestClone (line 71) | func TestClone(t *testing.T) {
function TestCloneNil (line 100) | func TestCloneNil(t *testing.T) {
function TestMerge (line 219) | func TestMerge(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/decode.go
function DecodeVarint (line 59) | func DecodeVarint(buf []byte) (x uint64, n int) {
method DecodeVarint (line 81) | func (p *Buffer) DecodeVarint() (x uint64, err error) {
method DecodeFixed64 (line 109) | func (p *Buffer) DecodeFixed64() (x uint64, err error) {
method DecodeFixed32 (line 132) | func (p *Buffer) DecodeFixed32() (x uint64, err error) {
method DecodeZigzag64 (line 151) | func (p *Buffer) DecodeZigzag64() (x uint64, err error) {
method DecodeZigzag32 (line 163) | func (p *Buffer) DecodeZigzag32() (x uint64, err error) {
method DecodeRawBytes (line 178) | func (p *Buffer) DecodeRawBytes(alloc bool) (buf []byte, err error) {
method DecodeStringBytes (line 208) | func (p *Buffer) DecodeStringBytes() (s string, err error) {
method skipAndSave (line 219) | func (o *Buffer) skipAndSave(t reflect.Type, tag, wire int, base structP...
method skip (line 246) | func (o *Buffer) skip(t reflect.Type, tag, wire int) error {
type Unmarshaler (line 287) | type Unmarshaler interface
function Unmarshal (line 298) | func Unmarshal(buf []byte, pb Message) error {
function UnmarshalMerge (line 309) | func UnmarshalMerge(buf []byte, pb Message) error {
method Unmarshal (line 321) | func (p *Buffer) Unmarshal(pb Message) error {
method unmarshalType (line 344) | func (o *Buffer) unmarshalType(st reflect.Type, prop *StructProperties, ...
constant boolPoolSize (line 448) | boolPoolSize = 16
constant uint32PoolSize (line 449) | uint32PoolSize = 8
constant uint64PoolSize (line 450) | uint64PoolSize = 4
method dec_bool (line 454) | func (o *Buffer) dec_bool(p *Properties, base structPointer) error {
method dec_proto3_bool (line 468) | func (o *Buffer) dec_proto3_bool(p *Properties, base structPointer) error {
method dec_int32 (line 478) | func (o *Buffer) dec_int32(p *Properties, base structPointer) error {
method dec_proto3_int32 (line 487) | func (o *Buffer) dec_proto3_int32(p *Properties, base structPointer) err...
method dec_int64 (line 497) | func (o *Buffer) dec_int64(p *Properties, base structPointer) error {
method dec_proto3_int64 (line 506) | func (o *Buffer) dec_proto3_int64(p *Properties, base structPointer) err...
method dec_string (line 516) | func (o *Buffer) dec_string(p *Properties, base structPointer) error {
method dec_proto3_string (line 525) | func (o *Buffer) dec_proto3_string(p *Properties, base structPointer) er...
method dec_slice_byte (line 535) | func (o *Buffer) dec_slice_byte(p *Properties, base structPointer) error {
method dec_slice_bool (line 545) | func (o *Buffer) dec_slice_bool(p *Properties, base structPointer) error {
method dec_slice_packed_bool (line 556) | func (o *Buffer) dec_slice_packed_bool(p *Properties, base structPointer...
method dec_slice_int32 (line 579) | func (o *Buffer) dec_slice_int32(p *Properties, base structPointer) error {
method dec_slice_packed_int32 (line 589) | func (o *Buffer) dec_slice_packed_int32(p *Properties, base structPointe...
method dec_slice_int64 (line 613) | func (o *Buffer) dec_slice_int64(p *Properties, base structPointer) error {
method dec_slice_packed_int64 (line 624) | func (o *Buffer) dec_slice_packed_int64(p *Properties, base structPointe...
method dec_slice_string (line 648) | func (o *Buffer) dec_slice_string(p *Properties, base structPointer) err...
method dec_slice_slice_byte (line 659) | func (o *Buffer) dec_slice_slice_byte(p *Properties, base structPointer)...
method dec_new_map (line 670) | func (o *Buffer) dec_new_map(p *Properties, base structPointer) error {
method dec_struct_group (line 742) | func (o *Buffer) dec_struct_group(p *Properties, base structPointer) err...
method dec_struct_message (line 753) | func (o *Buffer) dec_struct_message(p *Properties, base structPointer) (...
method dec_slice_struct_message (line 785) | func (o *Buffer) dec_slice_struct_message(p *Properties, base structPoin...
method dec_slice_struct_group (line 790) | func (o *Buffer) dec_slice_struct_group(p *Properties, base structPointe...
method dec_slice_struct (line 795) | func (o *Buffer) dec_slice_struct(p *Properties, is_group bool, base str...
FILE: vendor/github.com/golang/protobuf/proto/encode.go
type RequiredNotSetError (line 54) | type RequiredNotSetError struct
method Error (line 58) | func (e *RequiredNotSetError) Error() string {
constant maxVarintBytes (line 75) | maxVarintBytes = 10
function EncodeVarint (line 83) | func EncodeVarint(x uint64) []byte {
method EncodeVarint (line 99) | func (p *Buffer) EncodeVarint(x uint64) error {
function sizeVarint (line 108) | func sizeVarint(x uint64) (n int) {
method EncodeFixed64 (line 122) | func (p *Buffer) EncodeFixed64(x uint64) error {
function sizeFixed64 (line 135) | func sizeFixed64(x uint64) int {
method EncodeFixed32 (line 142) | func (p *Buffer) EncodeFixed32(x uint64) error {
function sizeFixed32 (line 151) | func sizeFixed32(x uint64) int {
method EncodeZigzag64 (line 158) | func (p *Buffer) EncodeZigzag64(x uint64) error {
function sizeZigzag64 (line 163) | func sizeZigzag64(x uint64) int {
method EncodeZigzag32 (line 170) | func (p *Buffer) EncodeZigzag32(x uint64) error {
function sizeZigzag32 (line 175) | func sizeZigzag32(x uint64) int {
method EncodeRawBytes (line 182) | func (p *Buffer) EncodeRawBytes(b []byte) error {
function sizeRawBytes (line 188) | func sizeRawBytes(b []byte) int {
method EncodeStringBytes (line 195) | func (p *Buffer) EncodeStringBytes(s string) error {
function sizeStringBytes (line 201) | func sizeStringBytes(s string) int {
type Marshaler (line 207) | type Marshaler interface
function Marshal (line 213) | func Marshal(pb Message) ([]byte, error) {
method Marshal (line 234) | func (p *Buffer) Marshal(pb Message) error {
function Size (line 261) | func Size(pb Message) (n int) {
method enc_bool (line 287) | func (o *Buffer) enc_bool(p *Properties, base structPointer) error {
method enc_proto3_bool (line 301) | func (o *Buffer) enc_proto3_bool(p *Properties, base structPointer) error {
function size_bool (line 311) | func size_bool(p *Properties, base structPointer) int {
function size_proto3_bool (line 319) | func size_proto3_bool(p *Properties, base structPointer) int {
method enc_int32 (line 328) | func (o *Buffer) enc_int32(p *Properties, base structPointer) error {
method enc_proto3_int32 (line 339) | func (o *Buffer) enc_proto3_int32(p *Properties, base structPointer) err...
function size_int32 (line 350) | func size_int32(p *Properties, base structPointer) (n int) {
function size_proto3_int32 (line 361) | func size_proto3_int32(p *Properties, base structPointer) (n int) {
method enc_uint32 (line 374) | func (o *Buffer) enc_uint32(p *Properties, base structPointer) error {
method enc_proto3_uint32 (line 385) | func (o *Buffer) enc_proto3_uint32(p *Properties, base structPointer) er...
function size_uint32 (line 396) | func size_uint32(p *Properties, base structPointer) (n int) {
function size_proto3_uint32 (line 407) | func size_proto3_uint32(p *Properties, base structPointer) (n int) {
method enc_int64 (line 419) | func (o *Buffer) enc_int64(p *Properties, base structPointer) error {
method enc_proto3_int64 (line 430) | func (o *Buffer) enc_proto3_int64(p *Properties, base structPointer) err...
function size_int64 (line 441) | func size_int64(p *Properties, base structPointer) (n int) {
function size_proto3_int64 (line 452) | func size_proto3_int64(p *Properties, base structPointer) (n int) {
method enc_string (line 464) | func (o *Buffer) enc_string(p *Properties, base structPointer) error {
method enc_proto3_string (line 475) | func (o *Buffer) enc_proto3_string(p *Properties, base structPointer) er...
function size_string (line 485) | func size_string(p *Properties, base structPointer) (n int) {
function size_proto3_string (line 496) | func size_proto3_string(p *Properties, base structPointer) (n int) {
function isNil (line 507) | func isNil(v reflect.Value) bool {
method enc_struct_message (line 516) | func (o *Buffer) enc_struct_message(p *Properties, base structPointer) e...
function size_struct_message (line 539) | func size_struct_message(p *Properties, base structPointer) int {
method enc_struct_group (line 561) | func (o *Buffer) enc_struct_group(p *Properties, base structPointer) err...
function size_struct_group (line 577) | func size_struct_group(p *Properties, base structPointer) (n int) {
method enc_slice_bool (line 590) | func (o *Buffer) enc_slice_bool(p *Properties, base structPointer) error {
function size_slice_bool (line 607) | func size_slice_bool(p *Properties, base structPointer) int {
method enc_slice_packed_bool (line 617) | func (o *Buffer) enc_slice_packed_bool(p *Properties, base structPointer...
function size_slice_packed_bool (line 635) | func size_slice_packed_bool(p *Properties, base structPointer) (n int) {
method enc_slice_byte (line 648) | func (o *Buffer) enc_slice_byte(p *Properties, base structPointer) error {
method enc_proto3_slice_byte (line 658) | func (o *Buffer) enc_proto3_slice_byte(p *Properties, base structPointer...
function size_slice_byte (line 668) | func size_slice_byte(p *Properties, base structPointer) (n int) {
function size_proto3_slice_byte (line 678) | func size_proto3_slice_byte(p *Properties, base structPointer) (n int) {
method enc_slice_int32 (line 689) | func (o *Buffer) enc_slice_int32(p *Properties, base structPointer) error {
function size_slice_int32 (line 703) | func size_slice_int32(p *Properties, base structPointer) (n int) {
method enc_slice_packed_int32 (line 718) | func (o *Buffer) enc_slice_packed_int32(p *Properties, base structPointe...
function size_slice_packed_int32 (line 737) | func size_slice_packed_int32(p *Properties, base structPointer) (n int) {
method enc_slice_uint32 (line 757) | func (o *Buffer) enc_slice_uint32(p *Properties, base structPointer) err...
function size_slice_uint32 (line 771) | func size_slice_uint32(p *Properties, base structPointer) (n int) {
method enc_slice_packed_uint32 (line 787) | func (o *Buffer) enc_slice_packed_uint32(p *Properties, base structPoint...
function size_slice_packed_uint32 (line 805) | func size_slice_packed_uint32(p *Properties, base structPointer) (n int) {
method enc_slice_int64 (line 823) | func (o *Buffer) enc_slice_int64(p *Properties, base structPointer) error {
function size_slice_int64 (line 836) | func size_slice_int64(p *Properties, base structPointer) (n int) {
method enc_slice_packed_int64 (line 850) | func (o *Buffer) enc_slice_packed_int64(p *Properties, base structPointe...
function size_slice_packed_int64 (line 868) | func size_slice_packed_int64(p *Properties, base structPointer) (n int) {
method enc_slice_slice_byte (line 886) | func (o *Buffer) enc_slice_slice_byte(p *Properties, base structPointer)...
function size_slice_slice_byte (line 899) | func size_slice_slice_byte(p *Properties, base structPointer) (n int) {
method enc_slice_string (line 913) | func (o *Buffer) enc_slice_string(p *Properties, base structPointer) err...
function size_slice_string (line 923) | func size_slice_string(p *Properties, base structPointer) (n int) {
method enc_slice_struct_message (line 934) | func (o *Buffer) enc_slice_struct_message(p *Properties, base structPoin...
function size_slice_struct_message (line 969) | func size_slice_struct_message(p *Properties, base structPointer) (n int) {
method enc_slice_struct_group (line 996) | func (o *Buffer) enc_slice_struct_group(p *Properties, base structPointe...
function size_slice_struct_group (line 1023) | func size_slice_struct_group(p *Properties, base structPointer) (n int) {
method enc_map (line 1041) | func (o *Buffer) enc_map(p *Properties, base structPointer) error {
function size_map (line 1067) | func size_map(p *Properties, base structPointer) int {
method enc_new_map (line 1073) | func (o *Buffer) enc_new_map(p *Properties, base structPointer) error {
function size_new_map (line 1120) | func size_new_map(p *Properties, base structPointer) int {
function mapEncodeScratch (line 1143) | func mapEncodeScratch(mapType reflect.Type) (keycopy, valcopy reflect.Va...
method enc_struct (line 1175) | func (o *Buffer) enc_struct(prop *StructProperties, base structPointer) ...
function size_struct (line 1210) | func size_struct(prop *StructProperties, base structPointer) (n int) {
method enc_len_struct (line 1230) | func (o *Buffer) enc_len_struct(prop *StructProperties, base structPoint...
method enc_len_thing (line 1235) | func (o *Buffer) enc_len_thing(enc func() error, state *errorState) error {
type errorState (line 1264) | type errorState struct
method shouldContinue (line 1275) | func (s *errorState) shouldContinue(err error, prop *Properties) bool {
FILE: vendor/github.com/golang/protobuf/proto/equal.go
function Equal (line 67) | func Equal(a, b Message) bool {
function equalStruct (line 91) | func equalStruct(v1, v2 reflect.Value) bool {
function equalAny (line 144) | func equalAny(v1, v2 reflect.Value) bool {
function equalExtensions (line 207) | func equalExtensions(base reflect.Type, em1, em2 map[int32]Extension) bo...
FILE: vendor/github.com/golang/protobuf/proto/equal_test.go
function init (line 52) | func init() {
function TestEqual (line 185) | func TestEqual(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/extensions.go
type ExtensionRange (line 51) | type ExtensionRange struct
type extendableProto (line 56) | type extendableProto interface
type ExtensionDesc (line 66) | type ExtensionDesc struct
method repeated (line 74) | func (ed *ExtensionDesc) repeated() bool {
type Extension (line 80) | type Extension struct
function SetRawExtension (line 95) | func SetRawExtension(base extendableProto, id int32, b []byte) {
function isExtensionField (line 100) | func isExtensionField(pb extendableProto, field int32) bool {
function checkExtensionTypes (line 110) | func checkExtensionTypes(pb extendableProto, extension *ExtensionDesc) e...
type extPropKey (line 123) | type extPropKey struct
function extensionProperties (line 135) | func extensionProperties(ed *ExtensionDesc) *Properties {
function encodeExtensionMap (line 159) | func encodeExtensionMap(m map[int32]Extension) error {
function sizeExtensionMap (line 187) | func sizeExtensionMap(m map[int32]Extension) (n int) {
function HasExtension (line 212) | func HasExtension(pb extendableProto, extension *ExtensionDesc) bool {
function ClearExtension (line 219) | func ClearExtension(pb extendableProto, extension *ExtensionDesc) {
function GetExtension (line 226) | func GetExtension(pb extendableProto, extension *ExtensionDesc) (interfa...
function decodeExtension (line 262) | func decodeExtension(b []byte, extension *ExtensionDesc) (interface{}, e...
function GetExtensions (line 297) | func GetExtensions(pb Message, es []*ExtensionDesc) (extensions []interf...
function SetExtension (line 317) | func SetExtension(pb extendableProto, extension *ExtensionDesc, value in...
function RegisterExtension (line 344) | func RegisterExtension(desc *ExtensionDesc) {
function RegisteredExtensions (line 360) | func RegisteredExtensions(pb Message) map[int32]*ExtensionDesc {
FILE: vendor/github.com/golang/protobuf/proto/extensions_test.go
function TestGetExtensionsWithMissingExtensions (line 41) | func TestGetExtensionsWithMissingExtensions(t *testing.T) {
function TestGetExtensionStability (line 62) | func TestGetExtensionStability(t *testing.T) {
function TestExtensionsRoundTrip (line 96) | func TestExtensionsRoundTrip(t *testing.T) {
function TestNilExtension (line 139) | func TestNilExtension(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/lib.go
type Message (line 219) | type Message interface
type Stats (line 227) | type Stats struct
constant collectStats (line 238) | collectStats = false
function GetStats (line 243) | func GetStats() Stats { return stats }
type Buffer (line 250) | type Buffer struct
method Reset (line 273) | func (p *Buffer) Reset() {
method SetBuf (line 280) | func (p *Buffer) SetBuf(s []byte) {
method Bytes (line 286) | func (p *Buffer) Bytes() []byte { return p.buf }
method DebugPrint (line 388) | func (p *Buffer) DebugPrint(s string, b []byte) {
function NewBuffer (line 268) | func NewBuffer(e []byte) *Buffer {
function Bool (line 294) | func Bool(v bool) *bool {
function Int32 (line 300) | func Int32(v int32) *int32 {
function Int (line 307) | func Int(v int) *int32 {
function Int64 (line 315) | func Int64(v int64) *int64 {
function Float32 (line 321) | func Float32(v float32) *float32 {
function Float64 (line 327) | func Float64(v float64) *float64 {
function Uint32 (line 333) | func Uint32(v uint32) *uint32 {
function Uint64 (line 339) | func Uint64(v uint64) *uint64 {
function String (line 345) | func String(v string) *string {
function EnumName (line 351) | func EnumName(m map[int32]string, v int32) string {
function UnmarshalJSONEnum (line 365) | func UnmarshalJSONEnum(m map[string]int32, data []byte, enumName string)...
function SetDefaults (line 502) | func SetDefaults(pb Message) {
function setDefaults (line 507) | func setDefaults(v reflect.Value, recur, zeros bool) {
type defaultMessage (line 649) | type defaultMessage struct
type scalarField (line 654) | type scalarField struct
function buildDefaultMessage (line 661) | func buildDefaultMessage(t reflect.Type) (dm defaultMessage) {
type mapKeys (line 784) | type mapKeys
method Len (line 786) | func (s mapKeys) Len() int { return len(s) }
method Swap (line 787) | func (s mapKeys) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
method Less (line 788) | func (s mapKeys) Less(i, j int) bool {
FILE: vendor/github.com/golang/protobuf/proto/message_set.go
type _MessageSet_Item (line 69) | type _MessageSet_Item struct
type MessageSet (line 74) | type MessageSet struct
method find (line 89) | func (ms *MessageSet) find(pb Message) *_MessageSet_Item {
method Has (line 103) | func (ms *MessageSet) Has(pb Message) bool {
method Unmarshal (line 110) | func (ms *MessageSet) Unmarshal(pb Message) error {
method Marshal (line 120) | func (ms *MessageSet) Marshal(pb Message) error {
method Reset (line 144) | func (ms *MessageSet) Reset() { *ms = MessageSet{} }
method String (line 145) | func (ms *MessageSet) String() string { return CompactTextString(ms) }
method ProtoMessage (line 146) | func (*MessageSet) ProtoMessage() {}
type messageTypeIder (line 85) | type messageTypeIder interface
function skipVarint (line 150) | func skipVarint(buf []byte) []byte {
function MarshalMessageSet (line 159) | func MarshalMessageSet(m map[int32]Extension) ([]byte, error) {
function UnmarshalMessageSet (line 188) | func UnmarshalMessageSet(buf []byte, m map[int32]Extension) error {
function MarshalMessageSetJSON (line 219) | func MarshalMessageSetJSON(m map[int32]Extension) ([]byte, error) {
function UnmarshalMessageSetJSON (line 262) | func UnmarshalMessageSetJSON(buf []byte, m map[int32]Extension) error {
type messageSetDesc (line 276) | type messageSetDesc struct
function RegisterMessageSetType (line 282) | func RegisterMessageSetType(m Message, fieldNum int32, name string) {
FILE: vendor/github.com/golang/protobuf/proto/message_set_test.go
function TestUnmarshalMessageSetWithDuplicate (line 39) | func TestUnmarshalMessageSetWithDuplicate(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/pointer_reflect.go
type structPointer (line 46) | type structPointer struct
function toStructPointer (line 52) | func toStructPointer(v reflect.Value) structPointer {
function structPointer_IsNil (line 57) | func structPointer_IsNil(p structPointer) bool {
function structPointer_Interface (line 62) | func structPointer_Interface(p structPointer, _ reflect.Type) interface{} {
type field (line 69) | type field
method IsValid (line 80) | func (f field) IsValid() bool { return f != nil }
function toField (line 72) | func toField(f *reflect.StructField) field {
function structPointer_field (line 83) | func structPointer_field(p structPointer, f field) reflect.Value {
function structPointer_ifield (line 98) | func structPointer_ifield(p structPointer, f field) interface{} {
function structPointer_Bytes (line 103) | func structPointer_Bytes(p structPointer, f field) *[]byte {
function structPointer_BytesSlice (line 108) | func structPointer_BytesSlice(p structPointer, f field) *[][]byte {
function structPointer_Bool (line 113) | func structPointer_Bool(p structPointer, f field) **bool {
function structPointer_BoolVal (line 118) | func structPointer_BoolVal(p structPointer, f field) *bool {
function structPointer_BoolSlice (line 123) | func structPointer_BoolSlice(p structPointer, f field) *[]bool {
function structPointer_String (line 128) | func structPointer_String(p structPointer, f field) **string {
function structPointer_StringVal (line 133) | func structPointer_StringVal(p structPointer, f field) *string {
function structPointer_StringSlice (line 138) | func structPointer_StringSlice(p structPointer, f field) *[]string {
function structPointer_ExtMap (line 143) | func structPointer_ExtMap(p structPointer, f field) *map[int32]Extension {
function structPointer_Map (line 148) | func structPointer_Map(p structPointer, f field, typ reflect.Type) refle...
function structPointer_SetStructPointer (line 153) | func structPointer_SetStructPointer(p structPointer, f field, q structPo...
function structPointer_GetStructPointer (line 158) | func structPointer_GetStructPointer(p structPointer, f field) structPoin...
function structPointer_StructPointerSlice (line 163) | func structPointer_StructPointerSlice(p structPointer, f field) structPo...
type structPointerSlice (line 169) | type structPointerSlice struct
method Len (line 173) | func (p structPointerSlice) Len() int { return p.v.Le...
method Index (line 174) | func (p structPointerSlice) Index(i int) structPointer { return struct...
method Append (line 175) | func (p structPointerSlice) Append(q structPointer) {
type word32 (line 190) | type word32 struct
function word32_IsNil (line 195) | func word32_IsNil(p word32) bool {
function word32_Set (line 200) | func word32_Set(p word32, o *Buffer, x uint32) {
function word32_Get (line 235) | func word32_Get(p word32) uint32 {
function structPointer_Word32 (line 249) | func structPointer_Word32(p structPointer, f field) word32 {
type word32Val (line 255) | type word32Val struct
function word32Val_Set (line 260) | func word32Val_Set(p word32Val, x uint32) {
function word32Val_Get (line 278) | func word32Val_Get(p word32Val) uint32 {
function structPointer_Word32Val (line 292) | func structPointer_Word32Val(p structPointer, f field) word32Val {
type word32Slice (line 298) | type word32Slice struct
method Append (line 302) | func (p word32Slice) Append(x uint32) {
method Len (line 321) | func (p word32Slice) Len() int {
method Index (line 325) | func (p word32Slice) Index(i int) uint32 {
function structPointer_Word32Slice (line 339) | func structPointer_Word32Slice(p structPointer, f field) word32Slice {
type word64 (line 344) | type word64 struct
function word64_Set (line 348) | func word64_Set(p word64, o *Buffer, x uint64) {
function word64_IsNil (line 379) | func word64_IsNil(p word64) bool {
function word64_Get (line 383) | func word64_Get(p word64) uint64 {
function structPointer_Word64 (line 396) | func structPointer_Word64(p structPointer, f field) word64 {
type word64Val (line 401) | type word64Val struct
function word64Val_Set (line 405) | func word64Val_Set(p word64Val, o *Buffer, x uint64) {
function word64Val_Get (line 420) | func word64Val_Get(p word64Val) uint64 {
function structPointer_Word64Val (line 433) | func structPointer_Word64Val(p structPointer, f field) word64Val {
type word64Slice (line 437) | type word64Slice struct
method Append (line 441) | func (p word64Slice) Append(x uint64) {
method Len (line 460) | func (p word64Slice) Len() int {
method Index (line 464) | func (p word64Slice) Index(i int) uint64 {
function structPointer_Word64Slice (line 477) | func structPointer_Word64Slice(p structPointer, f field) word64Slice {
FILE: vendor/github.com/golang/protobuf/proto/pointer_unsafe.go
type structPointer (line 53) | type structPointer
function toStructPointer (line 56) | func toStructPointer(v reflect.Value) structPointer {
function structPointer_IsNil (line 61) | func structPointer_IsNil(p structPointer) bool {
function structPointer_Interface (line 67) | func structPointer_Interface(p structPointer, t reflect.Type) interface{} {
type field (line 73) | type field
method IsValid (line 84) | func (f field) IsValid() bool {
function toField (line 76) | func toField(f *reflect.StructField) field {
constant invalidField (line 81) | invalidField = ^field(0)
function structPointer_Bytes (line 89) | func structPointer_Bytes(p structPointer, f field) *[]byte {
function structPointer_BytesSlice (line 94) | func structPointer_BytesSlice(p structPointer, f field) *[][]byte {
function structPointer_Bool (line 99) | func structPointer_Bool(p structPointer, f field) **bool {
function structPointer_BoolVal (line 104) | func structPointer_BoolVal(p structPointer, f field) *bool {
function structPointer_BoolSlice (line 109) | func structPointer_BoolSlice(p structPointer, f field) *[]bool {
function structPointer_String (line 114) | func structPointer_String(p structPointer, f field) **string {
function structPointer_StringVal (line 119) | func structPointer_StringVal(p structPointer, f field) *string {
function structPointer_StringSlice (line 124) | func structPointer_StringSlice(p structPointer, f field) *[]string {
function structPointer_ExtMap (line 129) | func structPointer_ExtMap(p structPointer, f field) *map[int32]Extension {
function structPointer_Map (line 134) | func structPointer_Map(p structPointer, f field, typ reflect.Type) refle...
function structPointer_SetStructPointer (line 139) | func structPointer_SetStructPointer(p structPointer, f field, q structPo...
function structPointer_GetStructPointer (line 144) | func structPointer_GetStructPointer(p structPointer, f field) structPoin...
function structPointer_StructPointerSlice (line 149) | func structPointer_StructPointerSlice(p structPointer, f field) *structP...
type structPointerSlice (line 154) | type structPointerSlice
method Len (line 156) | func (v *structPointerSlice) Len() int { return len(*...
method Index (line 157) | func (v *structPointerSlice) Index(i int) structPointer { return (*v)[...
method Append (line 158) | func (v *structPointerSlice) Append(p structPointer) { *v = append(...
type word32 (line 161) | type word32
function word32_IsNil (line 164) | func word32_IsNil(p word32) bool {
function word32_Set (line 169) | func word32_Set(p word32, o *Buffer, x uint32) {
function word32_Get (line 179) | func word32_Get(p word32) uint32 {
function structPointer_Word32 (line 184) | func structPointer_Word32(p structPointer, f field) word32 {
type word32Val (line 189) | type word32Val
function word32Val_Set (line 192) | func word32Val_Set(p word32Val, x uint32) {
function word32Val_Get (line 197) | func word32Val_Get(p word32Val) uint32 {
function structPointer_Word32Val (line 202) | func structPointer_Word32Val(p structPointer, f field) word32Val {
type word32Slice (line 207) | type word32Slice
method Append (line 209) | func (v *word32Slice) Append(x uint32) { *v = append(*v, x) }
method Len (line 210) | func (v *word32Slice) Len() int { return len(*v) }
method Index (line 211) | func (v *word32Slice) Index(i int) uint32 { return (*v)[i] }
function structPointer_Word32Slice (line 214) | func structPointer_Word32Slice(p structPointer, f field) *word32Slice {
type word64 (line 219) | type word64
function word64_Set (line 221) | func word64_Set(p word64, o *Buffer, x uint64) {
function word64_IsNil (line 230) | func word64_IsNil(p word64) bool {
function word64_Get (line 234) | func word64_Get(p word64) uint64 {
function structPointer_Word64 (line 238) | func structPointer_Word64(p structPointer, f field) word64 {
type word64Val (line 243) | type word64Val
function word64Val_Set (line 245) | func word64Val_Set(p word64Val, o *Buffer, x uint64) {
function word64Val_Get (line 249) | func word64Val_Get(p word64Val) uint64 {
function structPointer_Word64Val (line 253) | func structPointer_Word64Val(p structPointer, f field) word64Val {
type word64Slice (line 258) | type word64Slice
method Append (line 260) | func (v *word64Slice) Append(x uint64) { *v = append(*v, x) }
method Len (line 261) | func (v *word64Slice) Len() int { return len(*v) }
method Index (line 262) | func (v *word64Slice) Index(i int) uint64 { return (*v)[i] }
function structPointer_Word64Slice (line 264) | func structPointer_Word64Slice(p structPointer, f field) *word64Slice {
FILE: vendor/github.com/golang/protobuf/proto/properties.go
constant debug (line 48) | debug bool = false
constant WireVarint (line 52) | WireVarint = 0
constant WireFixed64 (line 53) | WireFixed64 = 1
constant WireBytes (line 54) | WireBytes = 2
constant WireStartGroup (line 55) | WireStartGroup = 3
constant WireEndGroup (line 56) | WireEndGroup = 4
constant WireFixed32 (line 57) | WireFixed32 = 5
constant startSize (line 60) | startSize = 10
type encoder (line 65) | type encoder
type valueEncoder (line 68) | type valueEncoder
type sizer (line 73) | type sizer
type valueSizer (line 77) | type valueSizer
type decoder (line 82) | type decoder
type valueDecoder (line 85) | type valueDecoder
type tagMap (line 90) | type tagMap struct
method get (line 99) | func (p *tagMap) get(t int) (int, bool) {
method put (line 111) | func (p *tagMap) put(t int, fi int) {
constant tagMapFastLimit (line 97) | tagMapFastLimit = 1024
type StructProperties (line 127) | type StructProperties struct
method Len (line 140) | func (sp *StructProperties) Len() int { return len(sp.order) }
method Less (line 141) | func (sp *StructProperties) Less(i, j int) bool {
method Swap (line 144) | func (sp *StructProperties) Swap(i, j int) { sp.order[i], sp.order[j] ...
type Properties (line 147) | type Properties struct
method String (line 189) | func (p *Properties) String() string {
method Parse (line 221) | func (p *Properties) Parse(s string) {
method setEncAndDec (line 306) | func (p *Properties) setEncAndDec(typ reflect.Type, f *reflect.StructF...
method Init (line 584) | func (p *Properties) Init(typ reflect.Type, name, tag string, f *refle...
method init (line 588) | func (p *Properties) init(typ reflect.Type, name, tag string, f *refle...
function logNoSliceEnc (line 299) | func logNoSliceEnc(t1, t2 reflect.Type) {
function isMarshaler (line 562) | func isMarshaler(t reflect.Type) bool {
function isUnmarshaler (line 573) | func isUnmarshaler(t reflect.Type) bool {
function GetProperties (line 609) | func GetProperties(t reflect.Type) *StructProperties {
function getPropertiesLocked (line 633) | func getPropertiesLocked(t reflect.Type) *StructProperties {
function propByIndex (line 707) | func propByIndex(t reflect.Type, x []int) *Properties {
function getbase (line 717) | func getbase(pb Message) (t reflect.Type, b structPointer, err error) {
function RegisterEnum (line 737) | func RegisterEnum(typeName string, unusedNameMap map[int32]string, value...
FILE: vendor/github.com/golang/protobuf/proto/proto3_proto/proto3.pb.go
type Message_Humour (line 24) | type Message_Humour
method String (line 46) | func (x Message_Humour) String() string {
constant Message_UNKNOWN (line 27) | Message_UNKNOWN Message_Humour = 0
constant Message_PUNS (line 28) | Message_PUNS Message_Humour = 1
constant Message_SLAPSTICK (line 29) | Message_SLAPSTICK Message_Humour = 2
constant Message_BILL_BAILEY (line 30) | Message_BILL_BAILEY Message_Humour = 3
type Message (line 50) | type Message struct
method Reset (line 65) | func (m *Message) Reset() { *m = Message{} }
method String (line 66) | func (m *Message) String() string { return proto.CompactTextString(m) }
method ProtoMessage (line 67) | func (*Message) ProtoMessage() {}
method GetNested (line 69) | func (m *Message) GetNested() *Nested {
method GetTerrain (line 76) | func (m *Message) GetTerrain() map[string]*Nested {
method GetProto2Field (line 83) | func (m *Message) GetProto2Field() *testdata.SubDefaults {
method GetProto2Value (line 90) | func (m *Message) GetProto2Value() map[string]*testdata.SubDefaults {
type Nested (line 97) | type Nested struct
method Reset (line 101) | func (m *Nested) Reset() { *m = Nested{} }
method String (line 102) | func (m *Nested) String() string { return proto.CompactTextString(m) }
method ProtoMessage (line 103) | func (*Nested) ProtoMessage() {}
type MessageWithMap (line 105) | type MessageWithMap struct
method Reset (line 109) | func (m *MessageWithMap) Reset() { *m = MessageWithMap{} }
method String (line 110) | func (m *MessageWithMap) String() string { return proto.CompactTextStr...
method ProtoMessage (line 111) | func (*MessageWithMap) ProtoMessage() {}
method GetByteMapping (line 113) | func (m *MessageWithMap) GetByteMapping() map[bool][]byte {
function init (line 120) | func init() {
FILE: vendor/github.com/golang/protobuf/proto/proto3_test.go
function TestProto3ZeroValues (line 42) | func TestProto3ZeroValues(t *testing.T) {
function TestRoundTripProto3 (line 62) | func TestRoundTripProto3(t *testing.T) {
function TestProto3SetDefaults (line 96) | func TestProto3SetDefaults(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/size2_test.go
function TestVarintSize (line 42) | func TestVarintSize(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/size_test.go
function init (line 49) | func init() {
function TestSize (line 129) | func TestSize(t *testing.T) {
FILE: vendor/github.com/golang/protobuf/proto/text.go
type writer (line 65) | type writer interface
type textWriter (line 71) | type textWriter struct
method WriteString (line 78) | func (w *textWriter) WriteString(s string) (n int, err error) {
method Write (line 92) | func (w *textWriter) Write(p []byte) (n int, err error) {
method WriteByte (line 141) | func (w *textWriter) WriteByte(c byte) error {
method indent (line 153) | func (w *textWriter) indent() { w.ind++ }
method unindent (line 155) | func (w *textWriter) unindent() {
method writeIndent (line 710) | func (w *textWriter) writeIndent() {
function writeName (line 163) | func writeName(w *textWriter, props *Properties) error {
type raw (line 178) | type raw interface
function writeStruct (line 182) | func writeStruct(w *textWriter, sv reflect.Value) error {
function writeRaw (line 379) | func writeRaw(w *textWriter, b []byte) error {
function writeAny (line 400) | func writeAny(w *textWriter, v reflect.Value, props *Properties) error {
function isprint (line 472) | func isprint(c byte) bool {
function writeString (line 481) | func writeString(w *textWriter, s string) error {
function writeMessageSet (line 517) | func writeMessageSet(w *textWriter, ms *MessageSet) error {
function writeUnknownStruct (line 555) | func writeUnknownStruct(w *textWriter, data []byte) (err error) {
function writeUnknownInt (line 622) | func writeUnknownInt(w *textWriter, x uint64, err error) error {
type int32Slice (line 631) | type int32Slice
method Len (line 633) | func (s int32Slice) Len() int { return len(s) }
method Less (line 634) | func (s int32Slice) Less(i, j int) bool { return s[i] < s[j] }
method Swap (line 635) | func (s int32Slice) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
function writeExtensions (line 639) | func writeExtensions(w *textWriter, pv reflect.Value) error {
function writeExtension (line 692) | func writeExtension(w *textWriter, name string, pb interface{}) error {
function marshalText (line 726) | func marshalText(w io.Writer, pb Message, compact bool) error {
function MarshalText (line 770) | func MarshalText(w io.Writer, pb Message) error {
function MarshalTextString (line 775) | func MarshalTextString(pb Message) string {
function CompactText (line 782) | func CompactText(w io.Writer, pb Message) error { return marshalText(w, ...
function CompactTextString (line 785) | func CompactTextString(pb Message) string {
FILE: vendor/github.com/golang/protobuf/proto/text_parser.go
type ParseError (line 47) | type ParseError struct
method Error (line 53) | func (p *ParseError) Error() string {
type token (line 61) | type token struct
method String (line 69) | func (t *token) String() string {
type textParser (line 76) | type textParser struct
method errorf (line 92) | func (p *textParser) errorf(format string, a ...interface{}) *ParseErr...
method skipWhitespace (line 122) | func (p *textParser) skipWhitespace() {
method advance (line 146) | func (p *textParser) advance() {
method back (line 325) | func (p *textParser) back() { p.backed = true }
method next (line 328) | func (p *textParser) next() *token {
method consumeToken (line 358) | func (p *textParser) consumeToken(s string) error {
method missingRequiredFieldError (line 371) | func (p *textParser) missingRequiredFieldError(sv reflect.Value) *Requ...
method checkForColon (line 399) | func (p *textParser) checkForColon(props *Properties, typ reflect.Type...
method readStruct (line 439) | func (p *textParser) readStruct(sv reflect.Value, terminator string) e...
method consumeOptionalSeparator (line 628) | func (p *textParser) consumeOptionalSeparator() error {
method readAny (line 639) | func (p *textParser) readAny(v reflect.Value, props *Properties) error {
function newTextParser (line 84) | func newTextParser(s string) *textParser {
function isIdentOrNumberChar (line 100) | func isIdentOrNumberChar(c byte) bool {
function isWhitespace (line 114) | func isWhitespace(c byte) bool {
function unquoteC (line 201) | func unquoteC(s string, quote rune) (string, error) {
function unescape (line 244) | func unescape(s string) (ch string, tail string, err error) {
function unhex (line 311) | func unhex(b byte) (v byte, ok bool) {
function structFieldByName (line 388) | func structFieldByName(st reflect.Type, name string) (int, *Properties, ...
function UnmarshalText (line 761) | func UnmarshalText(s string, pb Message) error {
FILE: vendor/github.com/golang/protobuf/proto/text_parser_test.go
type UnmarshalTextTest (line 44) | type UnmarshalTextTest struct
function buildExtStructTest (line 50) | func buildExtStructTest(text string) UnmarshalTextTest {
function buildExtDataTest (line 60) | func buildExtDataTest(text string) UnmarshalTextTest {
function buildExtRepStringTest (line 69) | func buildExtRepStringTest(text string) UnmarshalTextTest {
function TestUnmarshalText (line 396) | func TestUnmarshalText(t *testing.T) {
function TestUnmarshalTextCustomMessage (line 423) | func TestUnmarshalTextCustomMessage(t *testing.T) {
function TestRepeatedEnum (line 434) | func TestRepeatedEnum(t *testing.T) {
function TestProto3TextParsing (line 447) | func TestProto3TextParsing(t *testing.T) {
function TestMapParsing (line 462) | func TestMapParsing(t *testing.T) {
function init (line 491) | func init() {
function BenchmarkUnmarshalText (line 505) | func BenchmarkUnmarshalText(b *testing.B) {
FILE: vendor/github.com/golang/protobuf/proto/text_test.go
type textMessage (line 50) | type textMessage struct
method MarshalText (line 53) | func (*textMessage) MarshalText() ([]byte, error) {
method UnmarshalText (line 57) | func (*textMessage) UnmarshalText(bytes []byte) error {
method Reset (line 64) | func (*textMessage) Reset() {}
method String (line 65) | func (*textMessage) String() string { return "" }
method ProtoMessage (line 66) | func (*textMessage) ProtoMessage() {}
function newTestMessage (line 68) | func newTestMessage() *pb.MyMessage {
constant text (line 126) | text = `count: 42
function TestMarshalText (line 166) | func TestMarshalText(t *testing.T) {
function TestMarshalTextCustomMessage (line 177) | func TestMarshalTextCustomMessage(t *testing.T) {
function TestMarshalTextNil (line 187) | func TestMarshalTextNil(t *testing.T) {
function TestMarshalTextUnknownEnum (line 201) | func TestMarshalTextUnknownEnum(t *testing.T) {
function BenchmarkMarshalTextBuffered (line 211) | func BenchmarkMarshalTextBuffered(b *testing.B) {
function BenchmarkMarshalTextUnbuffered (line 220) | func BenchmarkMarshalTextUnbuffered(b *testing.B) {
function compact (line 228) | func compact(src string) string {
function TestCompactText (line 275) | func TestCompactText(t *testing.T) {
function TestStringEscaping (line 282) | func TestStringEscaping(t *testing.T) {
type limitedWriter (line 332) | type limitedWriter struct
method Write (line 339) | func (w *limitedWriter) Write(p []byte) (n int, err error) {
function TestMarshalTextFailing (line 351) | func TestMarshalTextFailing(t *testing.T) {
function TestFloats (line 369) | func TestFloats(t *testing.T) {
function TestRepeatedNilText (line 390) | func TestRepeatedNilText(t *testing.T) {
function TestProto3Text (line 411) | func TestProto3Text(t *testing.T) {
FILE: vendor/github.com/mreiferson/go-options/example_test.go
type Options (line 11) | type Options struct
function ExampleResolve (line 17) | func ExampleResolve() {
FILE: vendor/github.com/mreiferson/go-options/options.go
function Resolve (line 36) | func Resolve(options interface{}, flagSet *flag.FlagSet, cfg map[string]...
function coerceBool (line 100) | func coerceBool(v interface{}) (bool, error) {
function coerceInt64 (line 112) | func coerceInt64(v interface{}) (int64, error) {
function coerceDuration (line 122) | func coerceDuration(v interface{}, arg string) (time.Duration, error) {
function coerceStringSlice (line 148) | func coerceStringSlice(v interface{}) ([]string, error) {
function coerceFloat64Slice (line 165) | func coerceFloat64Slice(v interface{}) ([]float64, error) {
function coerceString (line 195) | func coerceString(v interface{}) (string, error) {
function coerce (line 203) | func coerce(v interface{}, opt interface{}, arg string) (interface{}, er...
function hasArg (line 257) | func hasArg(s string) bool {
FILE: vendor/github.com/mreiferson/go-snappystream/reader.go
type reader (line 18) | type reader struct
method WriteTo (line 63) | func (r *reader) WriteTo(w io.Writer) (int64, error) {
method read (line 139) | func (r *reader) read(b []byte) (int, error) {
method Read (line 145) | func (r *reader) Read(b []byte) (int, error) {
method nextFrame (line 164) | func (r *reader) nextFrame(w io.Writer) (int, error) {
method decodeBlock (line 213) | func (r *reader) decodeBlock(w io.Writer) (int, error) {
method readStreamID (line 251) | func (r *reader) readStreamID() error {
method discardBlock (line 269) | func (r *reader) discardBlock() error {
method readBlock (line 275) | func (r *reader) readBlock() ([]byte, error) {
function NewReader (line 48) | func NewReader(r io.Reader, verifyChecksum bool) io.Reader {
type bufferFallbackWriter (line 112) | type bufferFallbackWriter struct
method Write (line 123) | func (w *bufferFallbackWriter) Write(b []byte) (int, error) {
function decodeLength (line 296) | func decodeLength(b []byte) uint32 {
function unmaskChecksum (line 300) | func unmaskChecksum(c uint32) uint32 {
function noeof (line 307) | func noeof(n int, err error) (int, error) {
function noeof64 (line 316) | func noeof64(n int64, err error) (int64, error) {
FILE: vendor/github.com/mreiferson/go-snappystream/reader_test.go
function TestReader_skippable (line 17) | func TestReader_skippable(t *testing.T) {
function TestReader_unskippable (line 62) | func TestReader_unskippable(t *testing.T) {
function TestReaderStreamID (line 91) | func TestReaderStreamID(t *testing.T) {
function TestReader_maxPad (line 133) | func TestReader_maxPad(t *testing.T) {
function TestReader_maxSkippable (line 152) | func TestReader_maxSkippable(t *testing.T) {
function TestReader_maxBlock (line 171) | func TestReader_maxBlock(t *testing.T) {
function TestReader_maxUnskippable (line 206) | func TestReader_maxUnskippable(t *testing.T) {
function TestReader_blockTooLarge (line 241) | func TestReader_blockTooLarge(t *testing.T) {
function TestReader_corruption (line 276) | func TestReader_corruption(t *testing.T) {
function TestReader_unexpectedEOF (line 360) | func TestReader_unexpectedEOF(t *testing.T) {
function compressedChunkGreaterN (line 427) | func compressedChunkGreaterN(t *testing.T, n int) []byte {
function compressedChunk (line 459) | func compressedChunk(t *testing.T, src []byte) []byte {
function uncompressedChunk (line 480) | func uncompressedChunk(t *testing.T, src []byte) []byte {
function opaqueChunk (line 496) | func opaqueChunk(b byte, n int) []byte {
function TestReaderWriteTo (line 522) | func TestReaderWriteTo(t *testing.T) {
function TestReaderWriteToPreviousError (line 551) | func TestReaderWriteToPreviousError(t *testing.T) {
type readerErrorFirst (line 583) | type readerErrorFirst struct
method Read (line 596) | func (r *readerErrorFirst) Read(b []byte) (int, error) {
function readErrorFirst (line 589) | func readErrorFirst(r io.Reader, err error) io.Reader {
function TestReaderWriteToWriteError (line 604) | func TestReaderWriteToWriteError(t *testing.T) {
type writerUnwritable (line 633) | type writerUnwritable struct
method Write (line 637) | func (w *writerUnwritable) Write([]byte) (int, error) {
function unwritable (line 641) | func unwritable(err error) io.Writer {
function encodedString (line 645) | func encodedString(s string) io.Reader {
FILE: vendor/github.com/mreiferson/go-snappystream/readwrite_test.go
constant TestFileSize (line 11) | TestFileSize = 10 << 20
function dummyBytesReader (line 19) | func dummyBytesReader(p []byte) io.Reader {
function testWriteThenRead (line 23) | func testWriteThenRead(t *testing.T, name string, bs []byte) {
function testBufferedWriteThenRead (line 59) | func testBufferedWriteThenRead(t *testing.T, name string, bs []byte) {
function TestWriterReader (line 100) | func TestWriterReader(t *testing.T) {
function TestBufferedWriterReader (line 116) | func TestBufferedWriterReader(t *testing.T) {
function TestWriterChunk (line 132) | func TestWriterChunk(t *testing.T) {
function BenchmarkWriterManpage (line 162) | func BenchmarkWriterManpage(b *testing.B) {
function BenchmarkBufferedWriterManpage (line 165) | func BenchmarkBufferedWriterManpage(b *testing.B) {
function BenchmarkBufferedWriterManpageNoCopy (line 168) | func BenchmarkBufferedWriterManpageNoCopy(b *testing.B) {
function BenchmarkWriterJSON (line 172) | func BenchmarkWriterJSON(b *testing.B) {
function BenchmarkBufferedWriterJSON (line 175) | func BenchmarkBufferedWriterJSON(b *testing.B) {
function BenchmarkBufferedWriterJSONNoCopy (line 178) | func BenchmarkBufferedWriterJSONNoCopy(b *testing.B) {
function BenchmarkWriterRandom (line 184) | func BenchmarkWriterRandom(b *testing.B) {
function BenchmarkBufferedWriterRandom (line 187) | func BenchmarkBufferedWriterRandom(b *testing.B) {
function BenchmarkBufferedWriterRandomNoCopy (line 190) | func BenchmarkBufferedWriterRandomNoCopy(b *testing.B) {
function BenchmarkWriterConstant (line 196) | func BenchmarkWriterConstant(b *testing.B) {
function BenchmarkBufferedWriterConstant (line 199) | func BenchmarkBufferedWriterConstant(b *testing.B) {
function BenchmarkBufferedWriterConstantNoCopy (line 202) | func BenchmarkBufferedWriterConstantNoCopy(b *testing.B) {
function benchmarkWriterBytes (line 206) | func benchmarkWriterBytes(b *testing.B, p []byte) {
function benchmarkBufferedWriterBytes (line 214) | func benchmarkBufferedWriterBytes(b *testing.B, p []byte) {
function benchmarkBufferedWriterBytesNoCopy (line 221) | func benchmarkBufferedWriterBytesNoCopy(b *testing.B, p []byte) {
function benchmarkEncode (line 231) | func benchmarkEncode(b *testing.B, enc func() io.WriteCloser, bs []byte) {
function BenchmarkReaderManpage (line 252) | func BenchmarkReaderManpage(b *testing.B) {
function BenchmarkReaderManpage_buffered (line 255) | func BenchmarkReaderManpage_buffered(b *testing.B) {
function BenchmarkReaderManpageNoCopy (line 258) | func BenchmarkReaderManpageNoCopy(b *testing.B) {
function BenchmarkReaderJSON (line 262) | func BenchmarkReaderJSON(b *testing.B) {
function BenchmarkReaderJSON_buffered (line 265) | func BenchmarkReaderJSON_buffered(b *testing.B) {
function BenchmarkReaderJSONNoCopy (line 268) | func BenchmarkReaderJSONNoCopy(b *testing.B) {
function BenchmarkReaderRandom (line 273) | func BenchmarkReaderRandom(b *testing.B) {
function BenchmarkReaderRandom_buffered (line 276) | func BenchmarkReaderRandom_buffered(b *testing.B) {
function BenchmarkReaderRandomNoCopy (line 279) | func BenchmarkReaderRandomNoCopy(b *testing.B) {
function BenchmarkReaderConstant (line 284) | func BenchmarkReaderConstant(b *testing.B) {
function BenchmarkReaderConstant_buffered (line 287) | func BenchmarkReaderConstant_buffered(b *testing.B) {
function BenchmarkReaderConstantNoCopy (line 290) | func BenchmarkReaderConstantNoCopy(b *testing.B) {
function encodeAndBenchmarkReader (line 299) | func encodeAndBenchmarkReader(b *testing.B, p []byte) {
function encodeAndBenchmarkReader_buffered (line 315) | func encodeAndBenchmarkReader_buffered(b *testing.B, p []byte) {
function encodeAndBenchmarkReaderNoCopy (line 333) | func encodeAndBenchmarkReaderNoCopy(b *testing.B, p []byte) {
function benchmarkDecode (line 347) | func benchmarkDecode(b *testing.B, dec func(io.Reader) io.Reader, size i...
function encodeStreamBytes (line 366) | func encodeStreamBytes(b []byte, buffer bool) ([]byte, error) {
function encodeStream (line 373) | func encodeStream(r io.Reader, buffer bool) ([]byte, error) {
function randBytes (line 399) | func randBytes(b *testing.B, size int) []byte {
type writeCloserNoCopy (line 411) | type writeCloserNoCopy struct
type nopWriteCloser (line 419) | type nopWriteCloser struct
method Close (line 423) | func (w *nopWriteCloser) Close() error {
FILE: vendor/github.com/mreiferson/go-snappystream/snappystream.go
constant Ext (line 13) | Ext = ".sz"
constant MediaType (line 16) | MediaType = "application/x-snappy-framed"
constant ContentEncoding (line 20) | ContentEncoding = "x-snappy-framed"
constant MaxBlockSize (line 24) | MaxBlockSize = 65536
constant VerifyChecksum (line 30) | VerifyChecksum = true
constant SkipVerifyChecksum (line 31) | SkipVerifyChecksum = false
constant blockCompressed (line 35) | blockCompressed = 0x00
constant blockUncompressed (line 36) | blockUncompressed = 0x01
constant blockPadding (line 37) | blockPadding = 0xfe
constant blockStreamIdentifier (line 38) | blockStreamIdentifier = 0xff
function maskChecksum (line 46) | func maskChecksum(c uint32) uint32 {
function init (line 52) | func init() {
FILE: vendor/github.com/mreiferson/go-snappystream/writer.go
type BufferedWriter (line 27) | type BufferedWriter struct
method ReadFrom (line 50) | func (w *BufferedWriter) ReadFrom(r io.Reader) (int64, error) {
method Write (line 63) | func (w *BufferedWriter) Write(p []byte) (int, error) {
method Flush (line 79) | func (w *BufferedWriter) Flush() error {
method Close (line 90) | func (w *BufferedWriter) Close() error {
function NewBufferedWriter (line 38) | func NewBufferedWriter(w io.Writer) *BufferedWriter {
type writer (line 107) | type writer struct
method Write (line 137) | func (w *writer) Write(p []byte) (int, error) {
method write (line 162) | func (w *writer) write(p []byte) (int, error) {
function NewWriter (line 128) | func NewWriter(w io.Writer) io.Writer {
function writeHeader (line 214) | func writeHeader(hdr []byte, btype byte, enc, dec []byte) {
FILE: vendor/github.com/mreiferson/go-snappystream/writer_test.go
function TestBufferedWriterClose (line 12) | func TestBufferedWriterClose(t *testing.T) {
function TestBufferedWriter_compression (line 34) | func TestBufferedWriter_compression(t *testing.T) {
function TestBufferedWriterFlush (line 83) | func TestBufferedWriterFlush(t *testing.T) {
FILE: vendor/golang.org/x/net/context/context.go
type Context (line 50) | type Context interface
type emptyCtx (line 150) | type emptyCtx
method Deadline (line 152) | func (*emptyCtx) Deadline() (deadline time.Time, ok bool) {
method Done (line 156) | func (*emptyCtx) Done() <-chan struct{} {
method Err (line 160) | func (*emptyCtx) Err() error {
method Value (line 164) | func (*emptyCtx) Value(key interface{}) interface{} {
method String (line 168) | func (e *emptyCtx) String() string {
function Background (line 187) | func Background() Context {
function TODO (line 196) | func TODO() Context {
type CancelFunc (line 203) | type CancelFunc
function WithCancel (line 211) | func WithCancel(parent Context) (ctx Context, cancel CancelFunc) {
function newCancelCtx (line 218) | func newCancelCtx(parent Context) cancelCtx {
function propagateCancel (line 226) | func propagateCancel(parent Context, child canceler) {
function parentCancelCtx (line 256) | func parentCancelCtx(parent Context) (*cancelCtx, bool) {
function removeChild (line 272) | func removeChild(parent Context, child canceler) {
type canceler (line 286) | type canceler interface
type cancelCtx (line 293) | type cancelCtx struct
method Done (line 303) | func (c *cancelCtx) Done() <-chan struct{} {
method Err (line 307) | func (c *cancelCtx) Err() error {
method String (line 313) | func (c *cancelCtx) String() string {
method cancel (line 319) | func (c *cancelCtx) cancel(removeFromParent bool, err error) {
function WithDeadline (line 351) | func WithDeadline(parent Context, deadline time.Time) (Context, CancelFu...
type timerCtx (line 379) | type timerCtx struct
method Deadline (line 386) | func (c *timerCtx) Deadline() (deadline time.Time, ok bool) {
method String (line 390) | func (c *timerCtx) String() string {
method cancel (line 394) | func (c *timerCtx) cancel(removeFromParent bool, err error) {
function WithTimeout (line 418) | func WithTimeout(parent Context, timeout time.Duration) (Context, Cancel...
function WithValue (line 427) | func WithValue(parent Context, key interface{}, val interface{}) Context {
type valueCtx (line 433) | type valueCtx struct
method String (line 438) | func (c *valueCtx) String() string {
method Value (line 442) | func (c *valueCtx) Value(key interface{}) interface{} {
FILE: vendor/golang.org/x/net/context/context_test.go
type otherContext (line 20) | type otherContext struct
function TestBackground (line 24) | func TestBackground(t *testing.T) {
function TestTODO (line 39) | func TestTODO(t *testing.T) {
function TestWithCancel (line 54) | func TestWithCancel(t *testing.T) {
function TestParentFinishesChild (line 95) | func TestParentFinishesChild(t *testing.T) {
function TestChildFinishesFirst (line 172) | func TestChildFinishesFirst(t *testing.T) {
function testDeadline (line 232) | func testDeadline(c Context, wait time.Duration, t *testing.T) {
function TestDeadline (line 243) | func TestDeadline(t *testing.T) {
function TestTimeout (line 260) | func TestTimeout(t *testing.T) {
function TestCanceledTimeout (line 277) | func TestCanceledTimeout(t *testing.T) {
type key1 (line 293) | type key1
type key2 (line 294) | type key2
function TestValues (line 300) | func TestValues(t *testing.T) {
function TestAllocs (line 348) | func TestAllocs(t *testing.T) {
function TestSimultaneousCancels (line 413) | func TestSimultaneousCancels(t *testing.T) {
function TestInterlockedCancels (line 461) | func TestInterlockedCancels(t *testing.T) {
function TestLayersCancel (line 478) | func TestLayersCancel(t *testing.T) {
function TestLayersTimeout (line 482) | func TestLayersTimeout(t *testing.T) {
function testLayers (line 486) | func testLayers(t *testing.T, seed int64, testTimeout bool) {
function TestCancelRemoves (line 555) | func TestCancelRemoves(t *testing.T) {
FILE: vendor/golang.org/x/net/context/ctxhttp/cancelreq.go
function canceler (line 11) | func canceler(client *http.Client, req *http.Request) func() {
FILE: vendor/golang.org/x/net/context/ctxhttp/cancelreq_go14.go
type requestCanceler (line 11) | type requestCanceler interface
function canceler (line 15) | func canceler(client *http.Client, req *http.Request) func() {
FILE: vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go
function Do (line 20) | func Do(ctx context.Context, client *http.Client, req *http.Request) (*h...
function Get (line 49) | func Get(ctx context.Context, client *http.Client, url string) (*http.Re...
function Head (line 58) | func Head(ctx context.Context, client *http.Client, url string) (*http.R...
function Post (line 67) | func Post(ctx context.Context, client *http.Client, url string, bodyType...
function PostForm (line 77) | func PostForm(ctx context.Context, client *http.Client, url string, data...
FILE: vendor/golang.org/x/net/context/ctxhttp/ctxhttp_test.go
constant requestDuration (line 18) | requestDuration = 100 * time.Millisecond
constant requestBody (line 19) | requestBody = "ok"
function TestNoTimeout (line 22) | func TestNoTimeout(t *testing.T) {
function TestCancel (line 30) | func TestCancel(t *testing.T) {
function TestCancelAfterRequest (line 47) | func TestCancelAfterRequest(t *testing.T) {
function doRequest (line 62) | func doRequest(ctx context.Context) (*http.Response, error) {
FILE: vendor/golang.org/x/net/context/withtimeout_test.go
function ExampleWithTimeout (line 14) | func ExampleWithTimeout() {
Condensed preview — 163 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,224K chars).
[
{
"path": ".gitignore",
"chars": 17,
"preview": ".DS_Store\ndist/\n\n"
},
{
"path": ".travis.yml",
"chars": 208,
"preview": "language: go\n\ngo:\n - 1.5\n - tip\n\nenv:\n - GO15VENDOREXPERIMENT=1\n\ninstall: true\n\nscript:\n - \"go install ./apps/phosph"
},
{
"path": "Dockerfile",
"chars": 157,
"preview": "FROM busybox\n\nADD dist/docker/bin/ /phosphor_bin/\nRUN cd / && ln -s /phosphor_bin/* . \\\n && cd /bin && ln -s /phospho"
},
{
"path": "Godeps/Godeps.json",
"chars": 1033,
"preview": "{\n\t\"ImportPath\": \"github.com/mondough/phosphor\",\n\t\"GoVersion\": \"go1.5\",\n\t\"Packages\": [\n\t\t\"./...\"\n\t],\n\t\"Deps\": [\n\t\t{\n\t\t\t\""
},
{
"path": "Godeps/Readme",
"chars": 136,
"preview": "This directory tree is generated automatically by godep.\n\nPlease do not edit.\n\nSee https://github.com/tools/godep for mo"
},
{
"path": "LICENCE",
"chars": 11307,
"preview": "Apache License\n Version 2.0, January 2004\n http://www.apache.org/licens"
},
{
"path": "README.md",
"chars": 1332,
"preview": "\n\nPhosphor is a Distributed Tracing system, similar to [Google's Dapper](https://research.google.com/p"
},
{
"path": "apps/phosphor/main.go",
"chars": 2112,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"os\"\n\t\"os/signal\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/mondough/phosphor"
},
{
"path": "apps/phosphord/main.go",
"chars": 1853,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"os\"\n\t\"os/signal\"\n\t\"runtime\"\n\t\"syscall\"\n\t\"time\"\n\n\tlog \"github.com/ci"
},
{
"path": "dist.sh",
"chars": 1101,
"preview": "#!/bin/bash\n\nset -e\n\nDIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nrm -rf $DIR/dist/docker\nmkdir -p $DIR/dist/do"
},
{
"path": "internal/util/stringarray.go",
"chars": 326,
"preview": "// The StringArray type is borrowed from NSQ\n// https://github.com/bitly/nsq/blob/master/util/string_array.go\n\npackage u"
},
{
"path": "internal/version/version.go",
"chars": 257,
"preview": "package version\n\nimport (\n\t\"fmt\"\n\t\"runtime\"\n)\n\n// Version of the binaries\nconst Version = \"0.0.1\"\n\n// String returns our"
},
{
"path": "phosphor/README.md",
"chars": 1092,
"preview": "# Phosphor\n\nThe Phosphor server receives traces from PhosphorD via NSQ and stores these for later retrieval via its API."
},
{
"path": "phosphor/context.go",
"chars": 277,
"preview": "package phosphor\n\nimport (\n\t\"fmt\"\n\n\t\"golang.org/x/net/context\"\n)\n\nfunc phosphorFromContext(ctx context.Context) (*Phosph"
},
{
"path": "phosphor/domain.go",
"chars": 2230,
"preview": "package phosphor\n\nimport (\n\t\"errors\"\n\t\"sync\"\n\t\"time\"\n)\n\n// NewTrace initialises and returns a new Trace\nfunc NewTrace() "
},
{
"path": "phosphor/handler.go",
"chars": 2776,
"preview": "package phosphor\n\nimport (\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"net/http\"\n\n\t\"golang.org/x/net/context\"\n\n\tlog \"github.com/"
},
{
"path": "phosphor/ingester.go",
"chars": 1683,
"preview": "package phosphor\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\tnsq \"github.com/bitly/go-nsq\"\n\tlog \"github.com/cihub/seelog\"\n\t\"github.com/gola"
},
{
"path": "phosphor/marshaling.go",
"chars": 4711,
"preview": "package phosphor\n\nimport (\n\t\"sort\"\n\t\"time\"\n\n\t\"github.com/mondough/phosphor/proto\"\n)\n\nfunc prettyFormatTrace(t *Trace) in"
},
{
"path": "phosphor/memorystore.go",
"chars": 1790,
"preview": "package phosphor\n\nimport (\n\t\"sync\"\n\t\"time\"\n\n\tlog \"github.com/cihub/seelog\"\n)\n\ntype MemoryStore struct {\n\tsync.RWMutex\n\tt"
},
{
"path": "phosphor/options.go",
"chars": 1066,
"preview": "package phosphor\n\nimport (\n\t\"crypto/md5\"\n\t\"hash/crc32\"\n\t\"io\"\n\t\"log\"\n\t\"os\"\n)\n\ntype Options struct {\n\t// basic options\n\tID"
},
{
"path": "phosphor/phosphor.go",
"chars": 1051,
"preview": "package phosphor\n\nimport (\n\t\"net/http\"\n\n\t\"golang.org/x/net/context\"\n\n\tlog \"github.com/cihub/seelog\"\n)\n\ntype Phosphor str"
},
{
"path": "phosphor/store.go",
"chars": 388,
"preview": "package phosphor\n\nimport \"errors\"\n\ntype Store interface {\n\tReadTrace(id string) (*Trace, error)\n\tStoreAnnotation(a *Anno"
},
{
"path": "phosphord/README.md",
"chars": 1232,
"preview": "# PhosphorD\n\nPhosphorD is a local forwarder, like StatsD, which receives traces from the Phosphor client, and forwards t"
},
{
"path": "phosphord/forwarder.go",
"chars": 1822,
"preview": "package phosphord\n\nimport (\n\t\"encoding/json\"\n\t\"time\"\n\n\tlog \"github.com/cihub/seelog\"\n\t\"github.com/golang/protobuf/proto\""
},
{
"path": "phosphord/options.go",
"chars": 676,
"preview": "package phosphord\n\ntype Options struct {\n\t// basic options\n\tVerbose bool `flag:\"verbose\"`\n\tUDPAddress string "
},
{
"path": "phosphord/phosphord.go",
"chars": 2414,
"preview": "package phosphord\n\nimport (\n\t\"bytes\"\n\t\"net\"\n\t\"os\"\n\t\"runtime\"\n\t\"time\"\n\n\tlog \"github.com/cihub/seelog\"\n\n\t\"github.com/mondo"
},
{
"path": "phosphord/test/test.go",
"chars": 1285,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"time\"\n\n\t\"github.com/golang/protobuf/proto\"\n\n\tpb \"github.com/mondough/phosphor/pro"
},
{
"path": "phosphord/transport/nsq.go",
"chars": 1805,
"preview": "package transport\n\nimport (\n\t\"errors\"\n\t\"math/rand\"\n\n\tnsq \"github.com/bitly/go-nsq\"\n\tlog \"github.com/cihub/seelog\"\n\n\t\"git"
},
{
"path": "phosphord/transport/transport.go",
"chars": 83,
"preview": "package transport\n\ntype Transport interface {\n\tMultiPublish(body [][]byte) error\n}\n"
},
{
"path": "proto/trace.pb.go",
"chars": 4108,
"preview": "// Code generated by protoc-gen-go.\n// source: github.com/mondough/phosphor/proto/trace.proto\n// DO NOT EDIT!\n\n/*\nPackag"
},
{
"path": "proto/trace.proto",
"chars": 1564,
"preview": "syntax=\"proto3\";\n\npackage traceproto;\n\nmessage Annotation {\n // The ID of the trace this annotation is a component of\n "
},
{
"path": "script/buildprotobufs.sh",
"chars": 1634,
"preview": "#!/bin/bash\n\n# Basic path locations\nROOT=$(cd $(dirname -- \"$0\" ) && cd .. && pwd)\nMESSAGEPATH=${ROOT}/proto\n\n# SRCPATH "
},
{
"path": "vendor/code.google.com/p/snappy-go/snappy/decode.go",
"chars": 2858,
"preview": "// Copyright 2011 The Snappy-Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// l"
},
{
"path": "vendor/code.google.com/p/snappy-go/snappy/encode.go",
"chars": 5061,
"preview": "// Copyright 2011 The Snappy-Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// l"
},
{
"path": "vendor/code.google.com/p/snappy-go/snappy/snappy.go",
"chars": 1679,
"preview": "// Copyright 2011 The Snappy-Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// l"
},
{
"path": "vendor/code.google.com/p/snappy-go/snappy/snappy_test.go",
"chars": 8173,
"preview": "// Copyright 2011 The Snappy-Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// l"
},
{
"path": "vendor/github.com/bitly/go-nsq/.travis.yml",
"chars": 972,
"preview": "language: go\ngo:\n - 1.4.2\nenv:\n - NSQ_DOWNLOAD=nsq-0.2.30.linux-amd64.go1.3 GOARCH=amd64\n - NSQ_DOWNLOAD=nsq-0.2.30.l"
},
{
"path": "vendor/github.com/bitly/go-nsq/ChangeLog.md",
"chars": 8410,
"preview": "## go-nsq Change Log\n\n### 1.0.4 - 2015-04-07\n\n**Upgrading from 1.0.3**: There are no backward incompatible changes.\n\n * "
},
{
"path": "vendor/github.com/bitly/go-nsq/LICENSE",
"chars": 1023,
"preview": "Permission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentati"
},
{
"path": "vendor/github.com/bitly/go-nsq/README.md",
"chars": 1235,
"preview": "## go-nsq\n\nThe official Go package for [NSQ][nsq].\n\n[![Build Status](https://secure.travis-ci.org/bitly/go-nsq.png?branc"
},
{
"path": "vendor/github.com/bitly/go-nsq/UPGRADING.md",
"chars": 7015,
"preview": "This outlines the backwards incompatible changes that were made to the public API after the\n`v0.3.7` stable release, and"
},
{
"path": "vendor/github.com/bitly/go-nsq/api_request.go",
"chars": 1928,
"preview": "package nsq\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"net\"\n\t\"net/http\"\n\t\"time\"\n)\n\ntype deadlinedConn struct"
},
{
"path": "vendor/github.com/bitly/go-nsq/command.go",
"chars": 6171,
"preview": "package nsq\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"strconv\"\n\t\"time\"\n)\n\nvar byteSpace = []"
},
{
"path": "vendor/github.com/bitly/go-nsq/command_test.go",
"chars": 248,
"preview": "package nsq\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n)\n\nfunc BenchmarkCommand(b *testing.B) {\n\tb.StopTimer()\n\tdata := make([]byte, "
},
{
"path": "vendor/github.com/bitly/go-nsq/config.go",
"chars": 19533,
"preview": "package nsq\n\nimport (\n\t\"crypto/tls\"\n\t\"crypto/x509\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"math\"\n\t\"math/rand\"\n\t\"net\"\n\t\"os"
},
{
"path": "vendor/github.com/bitly/go-nsq/config_test.go",
"chars": 3128,
"preview": "package nsq\n\nimport (\n\t\"math/rand\"\n\t\"net\"\n\t\"reflect\"\n\t\"testing\"\n\t\"time\"\n)\n\nfunc TestConfigSet(t *testing.T) {\n\tc := NewC"
},
{
"path": "vendor/github.com/bitly/go-nsq/conn.go",
"chars": 17755,
"preview": "package nsq\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"compress/flate\"\n\t\"crypto/tls\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t"
},
{
"path": "vendor/github.com/bitly/go-nsq/consumer.go",
"chars": 30104,
"preview": "package nsq\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"log\"\n\t\"math\"\n\t\"math/rand\"\n\t\"net\"\n\t\"net/url\"\n\t\"os\"\n\t\"strconv\"\n\t\"strings"
},
{
"path": "vendor/github.com/bitly/go-nsq/consumer_test.go",
"chars": 5629,
"preview": "package nsq\n\nimport (\n\t\"bytes\"\n\t\"crypto/tls\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"net\"\n\t\"net/http\"\n\t\""
},
{
"path": "vendor/github.com/bitly/go-nsq/delegates.go",
"chars": 4967,
"preview": "package nsq\n\nimport \"time\"\n\ntype logger interface {\n\tOutput(calldepth int, s string) error\n}\n\n// LogLevel specifies the "
},
{
"path": "vendor/github.com/bitly/go-nsq/errors.go",
"chars": 1187,
"preview": "package nsq\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n)\n\n// ErrNotConnected is returned when a publish command is made\n// against a Pro"
},
{
"path": "vendor/github.com/bitly/go-nsq/message.go",
"chars": 3784,
"preview": "package nsq\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"sync/atomic\"\n\t\"time\"\n)\n\n// The number of bytes fo"
},
{
"path": "vendor/github.com/bitly/go-nsq/mock_test.go",
"chars": 11909,
"preview": "package nsq\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"net\"\n\t\"strconv\"\n\t\"testing\"\n\t\""
},
{
"path": "vendor/github.com/bitly/go-nsq/producer.go",
"chars": 9468,
"preview": "package nsq\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n)\n\ntype producerConn interface {\n\tString() stri"
},
{
"path": "vendor/github.com/bitly/go-nsq/producer_test.go",
"chars": 8233,
"preview": "package nsq\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"syn"
},
{
"path": "vendor/github.com/bitly/go-nsq/protocol.go",
"chars": 2372,
"preview": "package nsq\n\nimport (\n\t\"encoding/binary\"\n\t\"errors\"\n\t\"io\"\n\t\"regexp\"\n)\n\n// MagicV1 is the initial identifier sent when con"
},
{
"path": "vendor/github.com/bitly/go-nsq/states.go",
"chars": 218,
"preview": "package nsq\n\n// states\nconst (\n\tStateInit = iota\n\tStateDisconnected\n\tStateConnected\n\tStateSubscribed\n\t// StateClosing me"
},
{
"path": "vendor/github.com/bitly/go-nsq/test/ca.pem",
"chars": 1436,
"preview": "-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIJAPYpAVNDj2lgMA0GCSqGSIb3DQEBBQUAMIGRMQswCQYD\nVQQGEwJERTEMMAoGA1UECAwDTlJ"
},
{
"path": "vendor/github.com/bitly/go-nsq/test/server.key",
"chars": 1708,
"preview": "-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDFHWaPfRA5nU/F\nE8AVoFj2TAgMRISLduWlbAgDnMt"
},
{
"path": "vendor/github.com/bitly/go-nsq/test/server.pem",
"chars": 1424,
"preview": "-----BEGIN CERTIFICATE-----\nMIID7zCCAtegAwIBAgIJAMsErP97ZQmgMA0GCSqGSIb3DQEBBQUAMIGNMQswCQYD\nVQQGEwJERTEMMAoGA1UECAwDTlJ"
},
{
"path": "vendor/github.com/bitly/go-nsq/test.sh",
"chars": 1255,
"preview": "#!/bin/bash\nset -e\n\n# a helper script to run tests\n\nif ! which nsqd >/dev/null; then\n echo \"missing nsqd binary\" && e"
},
{
"path": "vendor/github.com/bitly/go-nsq/version.go",
"chars": 249,
"preview": "// Package nsq is the official Go package for NSQ (http://nsq.io/)\n//\n// It provides high-level Consumer and Producer ty"
},
{
"path": "vendor/github.com/cihub/seelog/LICENSE.txt",
"chars": 1565,
"preview": "Copyright (c) 2012, Cloud Instruments Co., Ltd. <info@cin.io>\r\nAll rights reserved.\r\n\r\nRedistribution and use in source "
},
{
"path": "vendor/github.com/cihub/seelog/README.markdown",
"chars": 4095,
"preview": "Seelog\n=======\n\nSeelog is a powerful and easy-to-learn logging framework that provides functionality for flexible dispat"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_adaptive_test.go",
"chars": 3082,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_adaptivelogger.go",
"chars": 4380,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_asynclogger.go",
"chars": 4048,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_asyncloop_test.go",
"chars": 3167,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_asynclooplogger.go",
"chars": 2344,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_asynctimer_test.go",
"chars": 2343,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_asynctimerlogger.go",
"chars": 2635,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_synclogger.go",
"chars": 2327,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/behavior_synclogger_test.go",
"chars": 2295,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/cfg_config.go",
"chars": 7041,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/cfg_errors.go",
"chars": 2344,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/cfg_logconfig.go",
"chars": 4691,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/cfg_logconfig_test.go",
"chars": 3174,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/cfg_parser.go",
"chars": 37380,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/cfg_parser_test.go",
"chars": 41306,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_closer.go",
"chars": 1396,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_constraints.go",
"chars": 5250,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_constraints_test.go",
"chars": 5728,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_context.go",
"chars": 5697,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_context_test.go",
"chars": 3679,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_exception.go",
"chars": 6295,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_exception_test.go",
"chars": 3462,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_flusher.go",
"chars": 1585,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/common_loglevel.go",
"chars": 2544,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_custom.go",
"chars": 9830,
"preview": "// Copyright (c) 2013 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_customdispatcher_test.go",
"chars": 4879,
"preview": "// Copyright (c) 2013 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_dispatcher.go",
"chars": 5454,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_filterdispatcher.go",
"chars": 2462,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_filterdispatcher_test.go",
"chars": 2252,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_splitdispatcher.go",
"chars": 1926,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/dispatch_splitdispatcher_test.go",
"chars": 2179,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/doc.go",
"chars": 6436,
"preview": "// Copyright (c) 2014 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/format.go",
"chars": 14798,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/format_test.go",
"chars": 8551,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/internals_baseerror.go",
"chars": 151,
"preview": "package seelog\n\n// Base struct for custom errors.\ntype baseError struct {\n\tmessage string\n}\n\nfunc (be baseError) Error()"
},
{
"path": "vendor/github.com/cihub/seelog/internals_byteverifiers_test.go",
"chars": 4105,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/internals_fsutils.go",
"chars": 9718,
"preview": "package seelog\n\nimport (\n\t\"archive/zip\"\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sync\"\n)\n\n// File and"
},
{
"path": "vendor/github.com/cihub/seelog/internals_xmlnode.go",
"chars": 4333,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/internals_xmlnode_test.go",
"chars": 5817,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/log.go",
"chars": 10095,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/logger.go",
"chars": 12296,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_bufferedwriter.go",
"chars": 4746,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_bufferedwriter_test.go",
"chars": 2519,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_connwriter.go",
"chars": 3852,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_consolewriter.go",
"chars": 1930,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_filewriter.go",
"chars": 2733,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_filewriter_test.go",
"chars": 6431,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_formattedwriter.go",
"chars": 2332,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_formattedwriter_test.go",
"chars": 2098,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_rollingfilewriter.go",
"chars": 17896,
"preview": "// Copyright (c) 2013 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_rollingfilewriter_test.go",
"chars": 6702,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/cihub/seelog/writers_smtpwriter.go",
"chars": 6612,
"preview": "// Copyright (c) 2012 - Cloud Instruments Co., Ltd.\n//\n// All rights reserved.\n//\n// Redistribution and use in source an"
},
{
"path": "vendor/github.com/golang/protobuf/proto/Makefile",
"chars": 1869,
"preview": "# Go support for Protocol Buffers - Google's data interchange format\n#\n# Copyright 2010 The Go Authors. All rights rese"
},
{
"path": "vendor/github.com/golang/protobuf/proto/all_test.go",
"chars": 60762,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/clone.go",
"chars": 5873,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2011 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/clone_test.go",
"chars": 6164,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2011 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/decode.go",
"chars": 21785,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/encode.go",
"chars": 32590,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/equal.go",
"chars": 7100,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2011 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/equal_test.go",
"chars": 7600,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2011 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/extensions.go",
"chars": 11731,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/extensions_test.go",
"chars": 5243,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2014 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/lib.go",
"chars": 20957,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/message_set.go",
"chars": 8722,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/message_set_test.go",
"chars": 2543,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2014 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/pointer_reflect.go",
"chars": 13674,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2012 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/pointer_unsafe.go",
"chars": 9331,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2012 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/properties.go",
"chars": 21617,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/proto3_proto/proto3.pb.go",
"chars": 4101,
"preview": "// Code generated by protoc-gen-go.\n// source: proto3_proto/proto3.proto\n// DO NOT EDIT!\n\n/*\nPackage proto3_proto is a g"
},
{
"path": "vendor/github.com/golang/protobuf/proto/proto3_proto/proto3.proto",
"chars": 2301,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2014 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/proto3_test.go",
"chars": 4057,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2014 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/size2_test.go",
"chars": 2334,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2012 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/size_test.go",
"chars": 6579,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2012 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/text.go",
"chars": 18838,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/text_parser.go",
"chars": 19072,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/text_parser_test.go",
"chars": 12165,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/golang/protobuf/proto/text_test.go",
"chars": 11041,
"preview": "// Go support for Protocol Buffers - Google's data interchange format\n//\n// Copyright 2010 The Go Authors. All rights r"
},
{
"path": "vendor/github.com/mreiferson/go-options/.travis.yml",
"chars": 66,
"preview": "language: go\ngo:\n - 1.3\n - tip\nnotifications:\n email: false\n\n"
},
{
"path": "vendor/github.com/mreiferson/go-options/LICENSE",
"chars": 1023,
"preview": "Permission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentati"
},
{
"path": "vendor/github.com/mreiferson/go-options/README.md",
"chars": 365,
"preview": "# go-options\n\nResolve configuration values set via command line flags, config files, and default struct values.\n\n[![Buil"
},
{
"path": "vendor/github.com/mreiferson/go-options/example_test.go",
"chars": 811,
"preview": "package options_test\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/mreiferson/go-options\"\n)\n\ntype Options struct {\n\tMax"
},
{
"path": "vendor/github.com/mreiferson/go-options/options.go",
"chars": 6774,
"preview": "// options resolves configuration values set via command line flags, config files, and default\n// struct values\npackage "
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/.travis.yml",
"chars": 182,
"preview": "language: go\ngo:\n - 1.2.2\n - 1.3.1\nenv:\n - GOARCH=amd64\n - GOARCH=386\ninstall:\n - go get code.google.com/p/snappy-g"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/LICENSE",
"chars": 1023,
"preview": "Permission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentati"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/README.md",
"chars": 561,
"preview": "## go-snappystream\n\na Go package for framed snappy streams.\n\n[![Build Status](https://secure.travis-ci.org/mreiferson/go"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/fixturedata_test.go",
"chars": 161725,
"preview": "package snappystream\n\nvar testDataMan = []byte(`\n.TH XARGS 1L \\\" -*- nroff -*-\n.SH NAME\nxargs \\- build and execute comma"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/reader.go",
"chars": 8994,
"preview": "package snappystream\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"hash/crc32\"\n\t\"io\"\n\t\"io/ioutil\"\n\n\t\"code.google.com/p/snappy-go/snappy\"\n)"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/reader_test.go",
"chars": 16626,
"preview": "package snappystream\n\nimport (\n\t\"bytes\"\n\t\"crypto/rand\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"code.google.co"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/readwrite_test.go",
"chars": 11980,
"preview": "package snappystream\n\nimport (\n\t\"bytes\"\n\t\"crypto/rand\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"testing\"\n)\n\nconst TestFileSize = 10 << 20 //"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/snappystream.go",
"chars": 1649,
"preview": "// snappystream wraps snappy-go and supplies a Reader and Writer\n// for the snappy framed stream format:\n// https://"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/writer.go",
"chars": 6003,
"preview": "package snappystream\n\nimport (\n\t\"bufio\"\n\t\"errors\"\n\t\"fmt\"\n\t\"hash/crc32\"\n\t\"io\"\n\n\t\"code.google.com/p/snappy-go/snappy\"\n)\n\nv"
},
{
"path": "vendor/github.com/mreiferson/go-snappystream/writer_test.go",
"chars": 2768,
"preview": "package snappystream\n\nimport (\n\t\"bytes\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"testing\"\n)\n\n// This test ensures that all BufferedWriter m"
},
{
"path": "vendor/golang.org/x/net/context/context.go",
"chars": 14094,
"preview": "// Copyright 2014 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/golang.org/x/net/context/context_test.go",
"chars": 14932,
"preview": "// Copyright 2014 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/golang.org/x/net/context/ctxhttp/cancelreq.go",
"chars": 355,
"preview": "// Copyright 2015 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/golang.org/x/net/context/ctxhttp/cancelreq_go14.go",
"chars": 467,
"preview": "// Copyright 2015 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go",
"chars": 2259,
"preview": "// Copyright 2015 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/golang.org/x/net/context/ctxhttp/ctxhttp_test.go",
"chars": 1609,
"preview": "// Copyright 2015 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/golang.org/x/net/context/withtimeout_test.go",
"chars": 677,
"preview": "// Copyright 2014 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
}
]
About this extraction
This page contains the full source code of the monzo/phosphor GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 163 files (1.1 MB), approximately 316.3k tokens, and a symbol index with 1810 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.