Repository: quipo/statsd Branch: master Commit: 3d6a5565f314 Files: 33 Total size: 107.1 KB Directory structure: gitextract_x5826k8v/ ├── .travis.yml ├── LICENSE ├── README.md ├── bufferedclient.go ├── bufferedclient_test.go ├── client.go ├── client_test.go ├── event/ │ ├── absolute.go │ ├── absolute_test.go │ ├── fabsolute.go │ ├── fabsolute_test.go │ ├── fgauge.go │ ├── fgauge_test.go │ ├── fgaugedelta.go │ ├── fgaugedelta_test.go │ ├── gauge.go │ ├── gauge_test.go │ ├── gaugedelta.go │ ├── gaugedelta_test.go │ ├── increment.go │ ├── increment_test.go │ ├── interface.go │ ├── precisiontiming.go │ ├── precisiontiming_test.go │ ├── timing.go │ ├── timing_test.go │ ├── total.go │ └── total_test.go ├── interface.go ├── mock/ │ ├── mockableclient.go │ └── mockableclient_test.go ├── noopclient.go └── stdoutclient.go ================================================ FILE CONTENTS ================================================ ================================================ FILE: .travis.yml ================================================ language: go go: - 1.9.x - master ================================================ FILE: LICENSE ================================================ The MIT License (MIT) Copyright (c) 2014 Lorenzo Alberton Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ # StatsD client (Golang) [![Build Status](https://travis-ci.org/quipo/statsd.png?branch=master)](https://travis-ci.org/quipo/statsd) [![GoDoc](https://godoc.org/github.com/quipo/statsd?status.png)](http://godoc.org/github.com/quipo/statsd) ## Introduction Go Client library for [StatsD](https://github.com/etsy/statsd/). Contains a direct and a buffered client. The buffered version will hold and aggregate values for the same key in memory before flushing them at the defined frequency. This client library was inspired by the one embedded in the [Bit.ly NSQ](https://github.com/bitly/nsq/blob/master/util/statsd_client.go) project, and extended to support some extra custom events used at DataSift. ## Installation go get github.com/quipo/statsd ## Supported event types * `Increment` - Count occurrences per second/minute of a specific event * `Decrement` - Count occurrences per second/minute of a specific event * `Timing` - To track a duration event * `PrecisionTiming` - To track a duration event * `Gauge` (int) / `FGauge` (float) - Gauges are a constant data type. They are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again * `GaugeDelta` (int) / `FGaugeDelta` (float) - Same as above, but as a delta change to the previous value rather than a new absolute value * `Absolute` (int) / `FAbsolute` (float) - Absolute-valued metric (not averaged/aggregated) * `Total` - Continously increasing value, e.g. read operations since boot ## Sample usage ```go package main import ( "log" "os" "time" "github.com/quipo/statsd" ) func main() { // init prefix := "myproject." statsdclient := statsd.NewStatsdClient("localhost:8125", prefix) err := statsdclient.CreateSocket() if nil != err { log.Println(err) os.Exit(1) } interval := time.Second * 2 // aggregate stats and flush every 2 seconds stats := statsd.NewStatsdBuffer(interval, statsdclient) defer stats.Close() // not buffered: send immediately statsdclient.Incr("mymetric", 4) // buffered: aggregate in memory before flushing stats.Incr("mymetric", 1) stats.Incr("mymetric", 3) stats.Incr("mymetric", 1) stats.Incr("mymetric", 1) } ``` The string `%HOST%` in the metric name will automatically be replaced with the hostname of the server the event is sent from. ## [Changelog](https://github.com/quipo/statsd/releases) * `HEAD`: * * [`v.1.4.0`](https://github.com/quipo/statsd/releases/tag/1.4.0) * Fixed behaviour of Gauge with positive numbers: the previous behaviour was the same as GaugeDelta (FGauge already had the correct behaviour) * Added more tests * Small optimisation: replace string formatting with concatenation (thanks to @agnivade) * [`v.1.3.0`](https://github.com/quipo/statsd/releases/tag/v.1.3.0): * Added stdout client ("echo" service for debugging) * Fixed [issue #23](https://github.com/quipo/statsd/issues/23): GaugeDelta event Stats() should not send an absolute value of 0 * Fixed FGauge's collation in the buffered client to only preserve the last value in the batch (it mistakenly had the same implementation of FGaugeDelta's collation) * Fixed FGaugeDelta with negative value not to send a 0 value first (it mistakenly had the same implementation of FGauge) * Added many tests * Added compile-time checks that the default events implement the Event interface * [`v.1.2.0`](https://github.com/quipo/statsd/releases/tag/1.2.0): Sample rate support (thanks to [Hongjian Zhu](https://github.com/hongjianzhu)) * [`v.1.1.0`](https://github.com/quipo/statsd/releases/tag/1.1.0): * Added `SendEvents` function to `Statsd` interface; * Using interface in buffered client constructor; * Added/Fixed tests * [`v.1.0.0`](https://github.com/quipo/statsd/releases/tag/1.0.0): First stable release * `v.0.0.9`: Added memoization to reduce memory allocations * `v.0.0.8`: Pre-release ## Author Lorenzo Alberton * Web: [http://alberton.info](http://alberton.info) * Twitter: [@lorenzoalberton](https://twitter.com/lorenzoalberton) * Linkedin: [/in/lorenzoalberton](https://www.linkedin.com/in/lorenzoalberton) ## Copyright See [LICENSE](LICENSE) document ================================================ FILE: bufferedclient.go ================================================ package statsd import ( "log" "os" "time" "github.com/quipo/statsd/event" ) // request to close the buffered statsd collector type closeRequest struct { reply chan error } // StatsdBuffer is a client library to aggregate events in memory before // flushing aggregates to StatsD, useful if the frequency of events is extremely high // and sampling is not desirable type StatsdBuffer struct { statsd Statsd flushInterval time.Duration eventChannel chan event.Event events map[string]event.Event closeChannel chan closeRequest Logger Logger Verbose bool } // NewStatsdBuffer Factory func NewStatsdBuffer(interval time.Duration, client Statsd) *StatsdBuffer { sb := &StatsdBuffer{ flushInterval: interval, statsd: client, eventChannel: make(chan event.Event, 100), events: make(map[string]event.Event), closeChannel: make(chan closeRequest), Logger: log.New(os.Stdout, "[BufferedStatsdClient] ", log.Ldate|log.Ltime), Verbose: true, } go sb.collector() return sb } // CreateSocket creates a UDP connection to a StatsD server func (sb *StatsdBuffer) CreateSocket() error { return sb.statsd.CreateSocket() } // CreateTCPSocket creates a TCP connection to a StatsD server func (sb *StatsdBuffer) CreateTCPSocket() error { return sb.statsd.CreateTCPSocket() } // Incr - Increment a counter metric. Often used to note a particular event func (sb *StatsdBuffer) Incr(stat string, count int64) error { if 0 != count { sb.eventChannel <- &event.Increment{Name: stat, Value: count} } return nil } // Decr - Decrement a counter metric. Often used to note a particular event func (sb *StatsdBuffer) Decr(stat string, count int64) error { if 0 != count { sb.eventChannel <- &event.Increment{Name: stat, Value: -count} } return nil } // Timing - Track a duration event func (sb *StatsdBuffer) Timing(stat string, delta int64) error { sb.eventChannel <- event.NewTiming(stat, delta) return nil } // PrecisionTiming - Track a duration event // the time delta has to be a duration func (sb *StatsdBuffer) PrecisionTiming(stat string, delta time.Duration) error { sb.eventChannel <- event.NewPrecisionTiming(stat, delta) return nil } // Gauge - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again func (sb *StatsdBuffer) Gauge(stat string, value int64) error { sb.eventChannel <- &event.Gauge{Name: stat, Value: value} return nil } // GaugeDelta records a delta from the previous value (as int64) func (sb *StatsdBuffer) GaugeDelta(stat string, value int64) error { sb.eventChannel <- &event.GaugeDelta{Name: stat, Value: value} return nil } // FGauge is a Gauge working with float64 values func (sb *StatsdBuffer) FGauge(stat string, value float64) error { sb.eventChannel <- &event.FGauge{Name: stat, Value: value} return nil } // FGaugeDelta records a delta from the previous value (as float64) func (sb *StatsdBuffer) FGaugeDelta(stat string, value float64) error { sb.eventChannel <- &event.FGaugeDelta{Name: stat, Value: value} return nil } // Absolute - Send absolute-valued metric (not averaged/aggregated) func (sb *StatsdBuffer) Absolute(stat string, value int64) error { sb.eventChannel <- &event.Absolute{Name: stat, Values: []int64{value}} return nil } // FAbsolute - Send absolute-valued metric (not averaged/aggregated) func (sb *StatsdBuffer) FAbsolute(stat string, value float64) error { sb.eventChannel <- &event.FAbsolute{Name: stat, Values: []float64{value}} return nil } // Total - Send a metric that is continously increasing, e.g. read operations since boot func (sb *StatsdBuffer) Total(stat string, value int64) error { sb.eventChannel <- &event.Total{Name: stat, Value: value} return nil } // SendEvents - Sends stats from all the event objects. func (sb *StatsdBuffer) SendEvents(events map[string]event.Event) error { for _, e := range events { sb.eventChannel <- e } return nil } // avoid too many allocations by memoizing the "type|key" pair for an event // @see https://gobyexample.com/closures func initMemoisedKeyMap() func(typ string, key string) string { m := make(map[string]map[string]string) return func(typ string, key string) string { if _, ok := m[typ]; !ok { m[typ] = make(map[string]string) } k, ok := m[typ][key] if !ok { m[typ][key] = typ + "|" + key return m[typ][key] } return k // memoized value } } // handle flushes and updates in one single thread (instead of locking the events map) func (sb *StatsdBuffer) collector() { // on a panic event, flush all the pending stats before panicking defer func(sb *StatsdBuffer) { if r := recover(); r != nil { sb.Logger.Println("Caught panic, flushing stats before throwing the panic again") err := sb.flush() if nil != err { sb.Logger.Println("Error flushing stats", err.Error()) } panic(r) } }(sb) keyFor := initMemoisedKeyMap() // avoid allocations (https://gobyexample.com/closures) ticker := time.NewTicker(sb.flushInterval) for { select { case <-ticker.C: //sb.Logger.Println("Flushing stats") err := sb.flush() if nil != err { sb.Logger.Println("Error flushing stats", err.Error()) } case e := <-sb.eventChannel: //sb.Logger.Println("Received ", e.String()) // issue #28: unable to use Incr and PrecisionTiming with the same key (also fixed #27) k := keyFor(e.TypeString(), e.Key()) // avoid allocations if e2, ok := sb.events[k]; ok { //sb.Logger.Println("Updating existing event") err := e2.Update(e) if nil != err { sb.Logger.Println("Error updating stats", err.Error()) } sb.events[k] = e2 } else { //sb.Logger.Println("Adding new event") sb.events[k] = e } case c := <-sb.closeChannel: if sb.Verbose { sb.Logger.Println("Asked to terminate. Flushing stats before returning.") } c.reply <- sb.flush() return } } } // Close sends a close event to the collector asking to stop & flush pending stats // and closes the statsd client func (sb *StatsdBuffer) Close() (err error) { // 1. send a close event to the collector req := closeRequest{reply: make(chan error)} sb.closeChannel <- req // 2. wait for the collector to drain the queue and respond err = <-req.reply // 3. close the statsd client err2 := sb.statsd.Close() if err != nil { return err } return err2 } // send the events to StatsD and reset them. // This function is NOT thread-safe, so it must only be invoked synchronously // from within the collector() goroutine func (sb *StatsdBuffer) flush() (err error) { n := len(sb.events) if n == 0 { return nil } if err := sb.statsd.SendEvents(sb.events); err != nil { sb.Logger.Println(err) return err } sb.events = make(map[string]event.Event) return nil } ================================================ FILE: bufferedclient_test.go ================================================ package statsd import ( "math" "os" "reflect" "regexp" "sort" "strconv" "strings" "testing" "time" ) // ------------------------------------------------------------------- type KVint64 struct { Key string Value int64 } type KVfloat64 struct { Key string Value float64 } type KVint64Sorter []KVint64 type KVfloat64Sorter []KVfloat64 func (a KVint64Sorter) Len() int { return len(a) } func (a KVint64Sorter) Swap(i, j int) { a[i], a[j] = a[j], a[i] } func (a KVint64Sorter) Less(i, j int) bool { if a[i].Key == a[j].Key { return a[i].Value < a[j].Value } return a[i].Key < a[j].Key } func (a KVfloat64Sorter) Len() int { return len(a) } func (a KVfloat64Sorter) Swap(i, j int) { a[i], a[j] = a[j], a[i] } func (a KVfloat64Sorter) Less(i, j int) bool { if a[i].Key == a[j].Key { return a[i].Value < a[j].Value } return a[i].Key < a[j].Key } // Normalise the number of decimal places for easier comparisons in the tests func (a KVfloat64Sorter) Normalise(precision int) { for k, v := range a { v.Value = toFixed(v.Value, precision) a[k] = v } } func round(num float64) int { return int(num + math.Copysign(0.5, num)) } func toFixed(num float64, precision int) float64 { output := math.Pow(10, float64(precision)) return float64(round(num*output)) / output } // ------------------------------------------------------------------- func TestBufferedInt64(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexTotal := regexp.MustCompile(`^(.*)\:([+\-]?\d+)\|(\w).*$`) prefix := "myproject." tt := []struct { name string function string suffix string input []KVint64 expected []KVint64 }{ { name: "total", function: "total", suffix: "t", input: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"x:b:c", 5}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"g.h.i", 1}, {"x:b:c", 5}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, { name: "gauge", function: "gauge", suffix: "g", input: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", 2}, // this should override the previous one {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 2}, {"d:e:f", 2}, {"g.h.i", 1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, { name: "gaugedelta", function: "gaugedelta", suffix: "g", input: []KVint64{ {"a:b:c", +5}, {"d:e:f", -2}, {"a:b:c", -2}, {"g.h.i", +1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 3}, {"d:e:f", -2}, {"g.h.i", +1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, { name: "increment", function: "increment", suffix: "c", input: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", -2}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 3}, {"d:e:f", 2}, {"g.h.i", 1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) buffered := NewStatsdBuffer(time.Millisecond*20, client) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = buffered.CreateSocket() if nil != err { t.Fatal(err) } defer buffered.Close() for _, entry := range tc.input { switch tc.function { // send metric case "total": err = buffered.Total(entry.Key, entry.Value) case "gauge": err = buffered.Gauge(entry.Key, entry.Value) case "gaugedelta": err = buffered.GaugeDelta(entry.Key, entry.Value) case "increment": if entry.Value < 0 { err = buffered.Decr(entry.Key, int64(math.Abs(float64(entry.Value)))) } else { err = buffered.Incr(entry.Key, entry.Value) } } if nil != err { t.Error(err) } } received := 0 var actual []KVint64 for received < len(tc.expected) { batch := <-ch for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexTotal.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseInt(vv[2], 10, 64) if err != nil { t.Error(err) } actual = append(actual, KVint64{Key: vv[1][len(prefix):], Value: v}) } } sort.Sort(KVint64Sorter(actual)) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func TestBufferedFloat64(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexTotal := regexp.MustCompile(`^(.*)\:([+\-]?\d+(?:\.\d+)?)\|(\w).*$`) prefix := "myproject." tt := []struct { name string function string suffix string input KVfloat64Sorter expected KVfloat64Sorter }{ { name: "fgauge", function: "fgauge", suffix: "g", input: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.3}, {"a:b:c", 2.2}, // this should override the previous one {"g.h.i", 1.2}, {"zz.%HOST%", 1.1}, // also test %HOST% replacement }, expected: KVfloat64Sorter{ {"a:b:c", 2.2}, {"d:e:f", 2.3}, {"g.h.i", 1.2}, {"zz." + hostname, 1.1}, // also test %HOST% replacement }, }, { name: "fgaugedelta", function: "fgaugedelta", suffix: "g", input: KVfloat64Sorter{ {"a:b:c", +5.1}, {"d:e:f", -2.2}, {"a:b:c", -2.1}, {"g.h.i", +1.3}, {"zz.%HOST%", 1.4}, // also test %HOST% replacement }, expected: KVfloat64Sorter{ {"a:b:c", 3.0}, {"d:e:f", -2.2}, {"g.h.i", +1.3}, {"zz." + hostname, 1.4}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) buffered := NewStatsdBuffer(time.Millisecond*20, client) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = buffered.CreateSocket() if nil != err { t.Fatal(err) } defer buffered.Close() for _, entry := range tc.input { switch tc.function { // send metric case "fgauge": err = buffered.FGauge(entry.Key, entry.Value) case "fgaugedelta": err = buffered.FGaugeDelta(entry.Key, entry.Value) } if nil != err { t.Error(err) } } received := 0 var actual KVfloat64Sorter for received < len(tc.expected) { batch := <-ch for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexTotal.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseFloat(vv[2], 64) if err != nil { t.Error(err) } actual = append(actual, KVfloat64{Key: vv[1][len(prefix):], Value: v}) } } actual.Normalise(2) // keep 2 decimal digits sort.Sort(actual) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: Expected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func TestBufferedAbsolute(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexAbsolute := regexp.MustCompile(`^(.*)\:([+\-]?\d+)\|(\w).*$`) prefix := "myproject." tt := []struct { name string suffix string input KVint64Sorter expected KVint64Sorter }{ { name: "absolute", suffix: "a", input: KVint64Sorter{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", 8}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: KVint64Sorter{ {"a:b:c", 5}, {"a:b:c", 8}, {"d:e:f", 2}, {"g.h.i", 1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) buffered := NewStatsdBuffer(time.Millisecond*20, client) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = buffered.CreateSocket() if nil != err { t.Fatal(err) } defer buffered.Close() for _, entry := range tc.input { err = buffered.Absolute(entry.Key, entry.Value) if nil != err { t.Error(err) } } received := 0 var actual KVint64Sorter for received < len(tc.expected) { batch := <-ch for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexAbsolute.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseInt(vv[2], 10, 64) if err != nil { t.Error(err) } actual = append(actual, KVint64{Key: vv[1][len(prefix):], Value: v}) } } sort.Sort(actual) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: \nExpected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func TestBufferedFAbsolute(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexAbsolute := regexp.MustCompile(`^(.*)\:([+\-]?\d+(?:\.\d+)?)\|(\w).*$`) prefix := "myproject." tt := []struct { name string suffix string input KVfloat64Sorter expected KVfloat64Sorter }{ { name: "fabsolute", suffix: "a", input: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.1}, {"x:b:c", 5.1}, {"g.h.i", 1.1}, {"zz.%HOST%", 1.5}, // also test %HOST% replacement }, expected: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.1}, {"g.h.i", 1.1}, {"x:b:c", 5.1}, {"zz." + hostname, 1.5}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) buffered := NewStatsdBuffer(time.Millisecond*20, client) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = buffered.CreateSocket() if nil != err { t.Fatal(err) } defer buffered.Close() for _, entry := range tc.input { err = buffered.FAbsolute(entry.Key, entry.Value) if nil != err { t.Error(err) } } received := 0 var actual KVfloat64Sorter for received < len(tc.expected) { batch := <-ch for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexAbsolute.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseFloat(vv[2], 64) if err != nil { t.Error(err) } actual = append(actual, KVfloat64{Key: vv[1][len(prefix):], Value: toFixed(v, 2)}) } } sort.Sort(actual) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: \nExpected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } ================================================ FILE: client.go ================================================ package statsd import ( "errors" "fmt" "log" "math/rand" "net" "os" "strings" "time" "github.com/quipo/statsd/event" ) // Logger interface compatible with log.Logger type Logger interface { Println(v ...interface{}) } // UDPPayloadSize is the number of bytes to send at one go through the udp socket. // SendEvents will try to pack as many events into one udp packet. // Change this value as per network capabilities // For example to change to 16KB // import "github.com/quipo/statsd" // func init() { // statsd.UDPPayloadSize = 16 * 1024 // } var UDPPayloadSize = 512 // Hostname is exported so clients can set it to something different than the default var Hostname string var errNotConnected = fmt.Errorf("cannot send stats, not connected to StatsD server") // errors var ( ErrInvalidCount = errors.New("count is less than 0") ErrInvalidSampleRate = errors.New("sample rate is larger than 1 or less then 0") ) func init() { if host, err := os.Hostname(); nil == err { Hostname = host } } type socketType string const ( udpSocket socketType = "udp" tcpSocket socketType = "tcp" ) // StatsdClient is a client library to send events to StatsD type StatsdClient struct { conn net.Conn addr string prefix string sockType socketType Logger Logger } // NewStatsdClient - Factory func NewStatsdClient(addr string, prefix string) *StatsdClient { // allow %HOST% in the prefix string prefix = strings.Replace(prefix, "%HOST%", Hostname, 1) return &StatsdClient{ addr: addr, prefix: prefix, Logger: log.New(os.Stdout, "[StatsdClient] ", log.Ldate|log.Ltime), } } // String returns the StatsD server address func (c *StatsdClient) String() string { return c.addr } // CreateSocket creates a UDP connection to a StatsD server func (c *StatsdClient) CreateSocket() error { conn, err := net.DialTimeout(string(udpSocket), c.addr, 5*time.Second) if err != nil { return err } c.conn = conn c.sockType = udpSocket return nil } // CreateTCPSocket creates a TCP connection to a StatsD server func (c *StatsdClient) CreateTCPSocket() error { conn, err := net.DialTimeout(string(tcpSocket), c.addr, 5*time.Second) if err != nil { return err } c.conn = conn c.sockType = tcpSocket return nil } // Close the UDP connection func (c *StatsdClient) Close() error { if nil == c.conn { return nil } return c.conn.Close() } // See statsd data types here: http://statsd.readthedocs.org/en/latest/types.html // or also https://github.com/b/statsd_spec // Incr - Increment a counter metric. Often used to note a particular event func (c *StatsdClient) Incr(stat string, count int64) error { return c.IncrWithSampling(stat, count, 1) } // IncrWithSampling - Increment a counter metric with sampling between 0 and 1 func (c *StatsdClient) IncrWithSampling(stat string, count int64, sampleRate float32) error { if err := checkSampleRate(sampleRate); err != nil { return err } if !shouldFire(sampleRate) { return nil // ignore this call } if err := checkCount(count); err != nil { return err } return c.send(stat, "%d|c", count, sampleRate) } // Decr - Decrement a counter metric. Often used to note a particular event func (c *StatsdClient) Decr(stat string, count int64) error { return c.DecrWithSampling(stat, count, 1) } // DecrWithSampling - Decrement a counter metric with sampling between 0 and 1 func (c *StatsdClient) DecrWithSampling(stat string, count int64, sampleRate float32) error { if err := checkSampleRate(sampleRate); err != nil { return err } if !shouldFire(sampleRate) { return nil // ignore this call } if err := checkCount(count); err != nil { return err } return c.send(stat, "%d|c", -count, sampleRate) } // Timing - Track a duration event // the time delta must be given in milliseconds func (c *StatsdClient) Timing(stat string, delta int64) error { return c.TimingWithSampling(stat, delta, 1) } // TimingWithSampling - Track a duration event func (c *StatsdClient) TimingWithSampling(stat string, delta int64, sampleRate float32) error { if err := checkSampleRate(sampleRate); err != nil { return err } if !shouldFire(sampleRate) { return nil // ignore this call } return c.send(stat, "%d|ms", delta, sampleRate) } // PrecisionTiming - Track a duration event // the time delta has to be a duration func (c *StatsdClient) PrecisionTiming(stat string, delta time.Duration) error { return c.send(stat, "%.6f|ms", float64(delta)/float64(time.Millisecond), 1) } // Gauge - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again. If you specify // delta to be true, that specifies that the gauge should be updated, not set. Due to the // underlying protocol, you can't explicitly set a gauge to a negative number without // first setting it to zero. func (c *StatsdClient) Gauge(stat string, value int64) error { return c.GaugeWithSampling(stat, value, 1) } // GaugeWithSampling - Gauges are a constant data type. func (c *StatsdClient) GaugeWithSampling(stat string, value int64, sampleRate float32) error { if err := checkSampleRate(sampleRate); err != nil { return err } if !shouldFire(sampleRate) { return nil // ignore this call } if value < 0 { err := c.send(stat, "%d|g", 0, 1) if nil != err { return err } } return c.send(stat, "%d|g", value, sampleRate) } // GaugeDelta -- Send a change for a gauge func (c *StatsdClient) GaugeDelta(stat string, value int64) error { // Gauge Deltas are always sent with a leading '+' or '-'. The '-' takes care of itself but the '+' must added by hand if value < 0 { return c.send(stat, "%d|g", value, 1) } return c.send(stat, "+%d|g", value, 1) } // FGauge -- Send a floating point value for a gauge func (c *StatsdClient) FGauge(stat string, value float64) error { return c.FGaugeWithSampling(stat, value, 1) } // FGaugeWithSampling - Gauges are a constant data type. func (c *StatsdClient) FGaugeWithSampling(stat string, value float64, sampleRate float32) error { if err := checkSampleRate(sampleRate); err != nil { return err } if !shouldFire(sampleRate) { return nil } if value < 0 { err := c.send(stat, "%d|g", 0, 1) if nil != err { return err } } return c.send(stat, "%g|g", value, sampleRate) } // FGaugeDelta -- Send a floating point change for a gauge func (c *StatsdClient) FGaugeDelta(stat string, value float64) error { if value < 0 { return c.send(stat, "%g|g", value, 1) } return c.send(stat, "+%g|g", value, 1) } // Absolute - Send absolute-valued metric (not averaged/aggregated) func (c *StatsdClient) Absolute(stat string, value int64) error { return c.send(stat, "%d|a", value, 1) } // FAbsolute - Send absolute-valued floating point metric (not averaged/aggregated) func (c *StatsdClient) FAbsolute(stat string, value float64) error { return c.send(stat, "%g|a", value, 1) } // Total - Send a metric that is continously increasing, e.g. read operations since boot func (c *StatsdClient) Total(stat string, value int64) error { return c.send(stat, "%d|t", value, 1) } // write a UDP packet with the statsd event func (c *StatsdClient) send(stat string, format string, value interface{}, sampleRate float32) error { if c.conn == nil { return errNotConnected } stat = strings.Replace(stat, "%HOST%", Hostname, 1) metricString := c.prefix + stat + ":" + fmt.Sprintf(format, value) if sampleRate != 1 { metricString = fmt.Sprintf("%s|@%f", metricString, sampleRate) } // if sending tcp append a newline if c.sockType == tcpSocket { metricString += "\n" } _, err := fmt.Fprint(c.conn, metricString) return err } // SendEvent - Sends stats from an event object func (c *StatsdClient) SendEvent(e event.Event) error { if c.conn == nil { return errNotConnected } for _, stat := range e.Stats() { //fmt.Printf("SENDING EVENT %s%s\n", c.prefix, strings.Replace(stat, "%HOST%", Hostname, 1)) _, err := fmt.Fprintf(c.conn, "%s%s", c.prefix, strings.Replace(stat, "%HOST%", Hostname, 1)) if nil != err { return err } } return nil } // SendEvents - Sends stats from all the event objects. // Tries to bundle many together into one fmt.Fprintf based on UDPPayloadSize. func (c *StatsdClient) SendEvents(events map[string]event.Event) error { if c.conn == nil { return errNotConnected } var n int var stats = make([]string, 0) for _, e := range events { for _, stat := range e.Stats() { stat = c.prefix + strings.Replace(stat, "%HOST%", Hostname, 1) _n := n + len(stat) + 1 if _n > UDPPayloadSize { // with this last event, the UDP payload would be too big if _, err := fmt.Fprintf(c.conn, strings.Join(stats, "\n")+"\n"); err != nil { return err } // reset payload after flushing, and add the last event stats = []string{stat} n = len(stat) continue } // can fit more into the current payload n = _n stats = append(stats, stat) } } if len(stats) != 0 { if _, err := fmt.Fprintf(c.conn, strings.Join(stats, "\n")+"\n"); err != nil { return err } } return nil } func checkCount(c int64) error { if c <= 0 { return ErrInvalidCount } return nil } func checkSampleRate(r float32) error { if r < 0 || r > 1 { return ErrInvalidSampleRate } return nil } func shouldFire(sampleRate float32) bool { if sampleRate == 1 { return true } r := rand.New(rand.NewSource(time.Now().Unix())) return r.Float32() <= sampleRate } ================================================ FILE: client_test.go ================================================ package statsd import ( "bytes" "fmt" "math" "net" "os" "reflect" "regexp" "sort" "strconv" "strings" "sync" "testing" "time" "github.com/quipo/statsd/event" ) // MockNetConn is a mock for net.Conn type MockNetConn struct { buf bytes.Buffer } func (mock *MockNetConn) Read(b []byte) (n int, err error) { return mock.buf.Read(b) } func (mock *MockNetConn) Write(b []byte) (n int, err error) { return mock.buf.Write(append(b, '\n')) } func (mock MockNetConn) Close() error { mock.buf.Truncate(0) return nil } func (mock MockNetConn) LocalAddr() net.Addr { return nil } func (mock MockNetConn) RemoteAddr() net.Addr { return nil } func (mock MockNetConn) SetDeadline(t time.Time) error { return nil } func (mock MockNetConn) SetReadDeadline(t time.Time) error { return nil } func (mock MockNetConn) SetWriteDeadline(t time.Time) error { return nil } /* // TODO: use this function instead mocking net.Conn // usage: client, server := GetTestConnection("tcp", t) // usage: client, server := GetTestConnection("udp", t) func GetTestConnection(connType string, t *testing.T) (client, server net.Conn) { ln, err := net.Listen(connType, "127.0.0.1") if nil != err { t.Error("TCP errpr:", err) } go func() { defer ln.Close() server, err = ln.Accept() if nil != err { t.Error("TCP Accept errpr:", err) } }() client, err = net.Dial(connType, ln.Addr().String()) if nil != err { t.Error("TCP Dial error:", err) } return client, server } */ func TestClientInt64(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexTotal := regexp.MustCompile(`^(.*)\:([+\-]?\d+)\|(\w).*$`) prefix := "myproject." tt := []struct { name string function string suffix string input []KVint64 expected []KVint64 }{ { name: "total", function: "total", suffix: "t", input: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"x:b:c", 5}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"g.h.i", 1}, {"x:b:c", 5}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, { name: "gauge", function: "gauge", suffix: "g", input: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", 2}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 5}, {"a:b:c", 2}, {"d:e:f", 2}, {"g.h.i", 1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, { name: "gaugedelta", function: "gaugedelta", suffix: "g", input: []KVint64{ {"a:b:c", +5}, {"d:e:f", -2}, {"a:b:c", -2}, {"g.h.i", +1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", +5}, {"d:e:f", -2}, {"a:b:c", -2}, {"g.h.i", +1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, { name: "increment", function: "increment", suffix: "c", input: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", -2}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: []KVint64{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", -2}, {"g.h.i", 1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = client.CreateSocket() if nil != err { t.Fatal(err) } defer client.Close() for _, entry := range tc.input { switch tc.function { // send metric case "total": err = client.Total(entry.Key, entry.Value) case "gauge": err = client.Gauge(entry.Key, entry.Value) case "gaugedelta": err = client.GaugeDelta(entry.Key, entry.Value) case "increment": if entry.Value < 0 { err = client.Decr(entry.Key, int64(math.Abs(float64(entry.Value)))) } else { err = client.Incr(entry.Key, entry.Value) } } if nil != err { t.Error(err) } } received := 0 var actual []KVint64 for batch := range ch { for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexTotal.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseInt(vv[2], 10, 64) if err != nil { t.Error(err) } actual = append(actual, KVint64{Key: vv[1][len(prefix):], Value: v}) } } sort.Sort(KVint64Sorter(actual)) sort.Sort(KVint64Sorter(tc.expected)) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: \nExpected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func TestClientFloat64(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexTotal := regexp.MustCompile(`^(.*)\:([+\-]?\d+(?:\.\d+)?)\|(\w).*$`) prefix := "myproject." tt := []struct { name string function string suffix string input KVfloat64Sorter expected KVfloat64Sorter }{ { name: "fgauge", function: "fgauge", suffix: "g", input: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.3}, {"a:b:c", -2.2}, {"g.h.i", 1.2}, {"zz.%HOST%", 1.1}, // also test %HOST% replacement }, expected: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.3}, {"a:b:c", 0}, {"a:b:c", -2.2}, {"g.h.i", 1.2}, {"zz." + hostname, 1.1}, // also test %HOST% replacement }, }, { name: "fgaugedelta", function: "fgaugedelta", suffix: "g", input: KVfloat64Sorter{ {"a:b:c", +5.1}, {"d:e:f", -2.2}, {"a:b:c", -2.1}, {"g.h.i", +1.3}, {"zz.%HOST%", 1.4}, // also test %HOST% replacement }, expected: KVfloat64Sorter{ {"a:b:c", +5.1}, {"d:e:f", -2.2}, {"a:b:c", -2.1}, {"g.h.i", +1.3}, {"zz." + hostname, 1.4}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = client.CreateSocket() if nil != err { t.Fatal(err) } //defer client.Close() for _, entry := range tc.input { switch tc.function { // send metric case "fgauge": err = client.FGauge(entry.Key, entry.Value) case "fgaugedelta": err = client.FGaugeDelta(entry.Key, entry.Value) } if nil != err { t.Error(err) } } client.Close() received := 0 var actual KVfloat64Sorter for batch := range ch { for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexTotal.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseFloat(vv[2], 64) if err != nil { t.Error(err) } actual = append(actual, KVfloat64{Key: vv[1][len(prefix):], Value: v}) } } actual.Normalise(2) // keep 2 decimal digits sort.Sort(actual) sort.Sort(tc.expected) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: Expected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func TestClientAbsolute(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexAbsolute := regexp.MustCompile(`^(.*)\:([+\-]?\d+)\|(\w).*$`) prefix := "myproject." tt := []struct { name string suffix string input KVint64Sorter expected KVint64Sorter }{ { name: "absolute", suffix: "a", input: KVint64Sorter{ {"a:b:c", 5}, {"d:e:f", 2}, {"a:b:c", 8}, {"g.h.i", 1}, {"zz.%HOST%", 1}, // also test %HOST% replacement }, expected: KVint64Sorter{ {"a:b:c", 5}, {"a:b:c", 8}, {"d:e:f", 2}, {"g.h.i", 1}, {"zz." + hostname, 1}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = client.CreateSocket() if nil != err { t.Fatal(err) } defer client.Close() for _, entry := range tc.input { err = client.Absolute(entry.Key, entry.Value) if nil != err { t.Error(err) } } received := 0 var actual KVint64Sorter for received < len(tc.expected) { batch := <-ch for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexAbsolute.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseInt(vv[2], 10, 64) if err != nil { t.Error(err) } actual = append(actual, KVint64{Key: vv[1][len(prefix):], Value: v}) } } sort.Sort(actual) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: \nExpected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func TestClientFAbsolute(t *testing.T) { hostname, err := os.Hostname() if nil != err { t.Fatal("Cannot read host name:", err) } regexAbsolute := regexp.MustCompile(`^(.*)\:([+\-]?\d+(?:\.\d+)?)\|(\w).*$`) prefix := "myproject." tt := []struct { name string suffix string input KVfloat64Sorter expected KVfloat64Sorter }{ { name: "fabsolute", suffix: "a", input: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.1}, {"x:b:c", 5.1}, {"g.h.i", 1.1}, {"zz.%HOST%", 1.5}, // also test %HOST% replacement }, expected: KVfloat64Sorter{ {"a:b:c", 5.2}, {"d:e:f", 2.1}, {"g.h.i", 1.1}, {"x:b:c", 5.1}, {"zz." + hostname, 1.5}, // also test %HOST% replacement }, }, } for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { ln, udpAddr := newLocalListenerUDP(t) defer ln.Close() t.Log("Starting new UDP listener at", udpAddr.String()) time.Sleep(50 * time.Millisecond) client := NewStatsdClient(udpAddr.String(), prefix) ch := make(chan string) go doListenUDP(t, ln, ch, len(tc.expected)) time.Sleep(50 * time.Millisecond) err = client.CreateSocket() if nil != err { t.Fatal(err) } defer client.Close() for _, entry := range tc.input { err = client.FAbsolute(entry.Key, entry.Value) if nil != err { t.Error(err) } } received := 0 var actual KVfloat64Sorter for received < len(tc.expected) { batch := <-ch for _, x := range strings.Split(batch, "\n") { x = strings.TrimSpace(x) if "" == x { continue } if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) return } received++ vv := regexAbsolute.FindStringSubmatch(x) //t.Log(vv, x) if len(vv) < 4 { t.Error("Expecting more tokens", len(vv)) continue } if vv[3] != tc.suffix { t.Errorf("Metric without expected suffix: expected '%s', actual '%s'", tc.suffix, vv[3]) } v, err := strconv.ParseFloat(vv[2], 64) if err != nil { t.Error(err) } actual = append(actual, KVfloat64{Key: vv[1][len(prefix):], Value: toFixed(v, 2)}) } } sort.Sort(actual) if !reflect.DeepEqual(tc.expected, actual) { t.Errorf("did not receive all metrics: \nExpected: \n%T %v, \nActual: \n%T %v ", tc.expected, tc.expected, actual, actual) } time.Sleep(500 * time.Millisecond) }) } } func newLocalListenerUDP(t *testing.T) (*net.UDPConn, *net.UDPAddr) { addr := fmt.Sprintf(":%d", getFreePort()) udpAddr, err := net.ResolveUDPAddr("udp", addr) if err != nil { t.Error("UDP error:", err) return nil, nil } ln, err := net.ListenUDP("udp", udpAddr) if err != nil { t.Error("UDP Listen error:", err) return ln, udpAddr } t.Logf("Started new local UDP listener @ %s\n", udpAddr) return ln, udpAddr } func doListenUDP(t *testing.T, conn *net.UDPConn, ch chan string, n int) { var wg sync.WaitGroup wg.Add(n) for n > 0 { // Handle the connection in a new goroutine. // The loop then returns to accepting, so that // multiple connections may be served concurrently. go func(c *net.UDPConn, ch chan string, wg *sync.WaitGroup) { t.Logf("Reading from UDP socket @ %s\n", conn.LocalAddr().String()) buffer := make([]byte, 1024) size, err := c.Read(buffer) // size, address, err := sock.ReadFrom(buffer) <- This starts printing empty and nil values below immediatly if err != nil { t.Logf("Error reading from UDP socket. Buffer: %s, Size: %d, Error: %s\n", string(buffer), size, err) //t.Fatal(err) } t.Logf("Read buffer: \n------------------\n%s\n------------------\n* Size: %d\n", string(buffer), size) ch <- string(buffer[:size]) wg.Done() }(conn, ch, &wg) n-- } wg.Wait() close(ch) t.Logf("Finished listening on UDP socket @ %s\n", conn.LocalAddr().String()) } func doListenTCP(t *testing.T, conn net.Listener, ch chan string, n int) { for n > 0 { // read n non-empty lines from TCP socket t.Logf("doListenTCP iteration") client, err := conn.Accept() if err != nil { t.Error(err) return } buf := make([]byte, 1024) size, err := client.Read(buf) if err != nil { if err.Error() == "EOF" { return } t.Error(err) return } t.Logf("Read from TCP socket:\n----------\n%s\n----------\n", string(buf)) for _, s := range bytes.Split(buf[:size], []byte{'\n'}) { if len(s) > 0 { n-- ch <- string(s) } } } close(ch) } func newLocalListenerTCP(t *testing.T) (string, net.Listener) { addr := fmt.Sprintf("127.0.0.1:%d", getFreePort()) ln, err := net.Listen("tcp", addr) if err != nil { t.Fatal(err) } return addr, ln } func TestTCP(t *testing.T) { addr, ln := newLocalListenerTCP(t) defer ln.Close() t.Log("Starting new TCP listener at", addr) time.Sleep(50 * time.Millisecond) prefix := "myproject." client := NewStatsdClient(addr, prefix) ch := make(chan string) s := map[string]int64{ "a:b:c": 5, "d:e:f": 2, "x:b:c": 5, "g.h.i": 1, } expected := make(map[string]int64) for k, v := range s { expected[k] = v } // also test %HOST% replacement s["zz.%HOST%"] = 1 hostname, err := os.Hostname() expected["zz."+hostname] = 1 if nil != err { t.Error("Cannot read host name:", err.Error()) } t.Logf("Sending stats to TCP Socket") err = client.CreateTCPSocket() if nil != err { t.Error(err) } defer client.Close() for k, v := range s { err = client.Total(k, v) if nil != err { t.Error(err) } } time.Sleep(60 * time.Millisecond) go doListenTCP(t, ln, ch, len(s)) time.Sleep(50 * time.Millisecond) actual := make(map[string]int64) re := regexp.MustCompile(`^(.*)\:(\d+)\|(\w).*$`) for i := len(s); i > 0; i-- { //t.Logf("ITERATION %d\n", i) x, open := <-ch if !open { //t.Logf("CLOSED _____") break } x = strings.TrimSpace(x) if "" == x { //t.Logf("EMPTY STRING *****") break } //fmt.Println(x) if !strings.HasPrefix(x, prefix) { t.Errorf("Metric without expected prefix: expected '%s', actual '%s'", prefix, x) break } vv := re.FindStringSubmatch(x) if vv[3] != "t" { t.Errorf("Metric without expected suffix: expected 't', actual '%s'", vv[3]) } v, err := strconv.ParseInt(vv[2], 10, 64) if err != nil { t.Error(err) } actual[vv[1][len(prefix):]] = v } if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v \n", expected, expected, actual, actual) } } func TestSendEvents(t *testing.T) { c := NewStatsdClient("127.0.0.1:1201", "test") c.conn = &MockNetConn{} // mock connection // override with a small size UDPPayloadSize = 40 e1 := &event.Increment{Name: "test1", Value: 123} e2 := &event.Increment{Name: "test2", Value: 432} e3 := &event.Increment{Name: "test3", Value: 111} e4 := &event.Gauge{Name: "test4", Value: 12435} events := map[string]event.Event{ "test1": e1, "test2": e2, "test3": e3, "test4": e4, } err := c.SendEvents(events) if nil != err { t.Error(err) } b1 := make([]byte, UDPPayloadSize*3) n, err2 := c.conn.Read(b1) if nil != err2 { t.Error(err2) } cleanPayload := strings.Replace(strings.TrimSpace(string(b1[:n])), "\n\n", "\n", -1) nStats := len(strings.Split(cleanPayload, "\n")) if nStats != len(events) { t.Errorf("Was expecting %d events, got %d: %s", len(events), nStats, string(b1)) } } // getFreePort Ask the kernel for a free open port that is ready to use func getFreePort() int { addr, err := net.ResolveTCPAddr("tcp", "localhost:0") if err != nil { panic(err) } l, err := net.ListenTCP("tcp", addr) if err != nil { panic(err) } defer l.Close() return l.Addr().(*net.TCPAddr).Port } ================================================ FILE: event/absolute.go ================================================ package event import "fmt" // Absolute is a metric that is not averaged/aggregated. // We keep each value distinct and then we flush them all individually. type Absolute struct { Name string Values []int64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *Absolute) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Values = append(e.Values, e2.Payload().([]int64)...) return nil } // Payload returns the aggregated value for this event func (e Absolute) Payload() interface{} { return e.Values } // Stats returns an array of StatsD events as they travel over UDP func (e Absolute) Stats() []string { ret := make([]string, 0, len(e.Values)) for _, v := range e.Values { ret = append(ret, fmt.Sprintf("%s:%d|a", e.Name, v)) } return ret } // Key returns the name of this metric func (e Absolute) Key() string { return e.Name } // SetKey sets the name of this metric func (e *Absolute) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e Absolute) Type() int { return EventAbsolute } // TypeString returns a name for this type of metric func (e Absolute) TypeString() string { return "Absolute" } // String returns a debug-friendly representation of this metric func (e Absolute) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Values: %v}", e.TypeString(), e.Name, e.Values) } ================================================ FILE: event/absolute_test.go ================================================ package event import ( "reflect" "testing" ) func TestAbsoluteUpdate(t *testing.T) { e1 := &Absolute{Name: "test", Values: []int64{15}} e2 := &Absolute{Name: "test", Values: []int64{-10}} e3 := &Absolute{Name: "test", Values: []int64{8}} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:15|a", "test:-10|a", "test:8|a"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/fabsolute.go ================================================ package event import "fmt" // FAbsolute is a metric that is not averaged/aggregated. // We keep each value distinct and then we flush them all individually. type FAbsolute struct { Name string Values []float64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *FAbsolute) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Values = append(e.Values, e2.Payload().([]float64)...) return nil } // Payload returns the aggregated value for this event func (e FAbsolute) Payload() interface{} { return e.Values } // Stats returns an array of StatsD events as they travel over UDP func (e FAbsolute) Stats() []string { ret := make([]string, 0, len(e.Values)) for _, v := range e.Values { ret = append(ret, fmt.Sprintf("%s:%g|a", e.Name, v)) } return ret } // Key returns the name of this metric func (e FAbsolute) Key() string { return e.Name } // SetKey sets the name of this metric func (e *FAbsolute) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e FAbsolute) Type() int { return EventFAbsolute } // TypeString returns a name for this type of metric func (e FAbsolute) TypeString() string { return "FAbsolute" } // String returns a debug-friendly representation of this metric func (e FAbsolute) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Values: %v}", e.TypeString(), e.Name, e.Values) } ================================================ FILE: event/fabsolute_test.go ================================================ package event import ( "reflect" "testing" ) func TestFAbsoluteUpdate(t *testing.T) { e1 := &FAbsolute{Name: "test", Values: []float64{15.3}} e2 := &FAbsolute{Name: "test", Values: []float64{-10.1}} e3 := &FAbsolute{Name: "test", Values: []float64{8.3}} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:15.3|a", "test:-10.1|a", "test:8.3|a"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/fgauge.go ================================================ package event import "fmt" // FGauge - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again type FGauge struct { Name string Value float64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *FGauge) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Value = e2.Payload().(float64) return nil } // Payload returns the aggregated value for this event func (e FGauge) Payload() interface{} { return e.Value } // Stats returns an array of StatsD events as they travel over UDP func (e FGauge) Stats() []string { if e.Value < 0 { // because a leading '+' or '-' in the value of a gauge denotes a delta, to send // a negative gauge value we first set the gauge absolutely to 0, then send the // negative value as a delta from 0 (that's just how the spec works :-) return []string{ fmt.Sprintf("%s:%d|g", e.Name, 0), fmt.Sprintf("%s:%g|g", e.Name, e.Value), } } return []string{fmt.Sprintf("%s:%g|g", e.Name, e.Value)} } // Key returns the name of this metric func (e FGauge) Key() string { return e.Name } // SetKey sets the name of this metric func (e *FGauge) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e FGauge) Type() int { return EventFGauge } // TypeString returns a name for this type of metric func (e FGauge) TypeString() string { return "FGauge" } // String returns a debug-friendly representation of this metric func (e FGauge) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %g}", e.TypeString(), e.Name, e.Value) } ================================================ FILE: event/fgauge_test.go ================================================ package event import ( "reflect" "testing" ) func TestFGaugeUpdate(t *testing.T) { e1 := &FGauge{Name: "test", Value: float64(15.1)} e2 := &FGauge{Name: "test", Value: float64(-10.1)} e3 := &FGauge{Name: "test", Value: float64(8.4)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:8.4|g"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } func TestFGaugeUpdateNegative(t *testing.T) { e1 := &FGauge{Name: "test", Value: float64(-10.1)} e2 := &FGauge{Name: "test", Value: float64(-3.4)} err := e1.Update(e2) if nil != err { t.Error(err) } expected := []string{"test:0|g", "test:-3.4|g"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/fgaugedelta.go ================================================ package event import "fmt" // FGaugeDelta - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again type FGaugeDelta struct { Name string Value float64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *FGaugeDelta) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Value += e2.Payload().(float64) return nil } // Payload returns the aggregated value for this event func (e FGaugeDelta) Payload() interface{} { return e.Value } // Stats returns an array of StatsD events as they travel over UDP func (e FGaugeDelta) Stats() []string { return []string{fmt.Sprintf("%s:%+g|g", e.Name, e.Value)} } // Key returns the name of this metric func (e FGaugeDelta) Key() string { return e.Name } // SetKey sets the name of this metric func (e *FGaugeDelta) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e FGaugeDelta) Type() int { return EventFGaugeDelta } // TypeString returns a name for this type of metric func (e FGaugeDelta) TypeString() string { return "FGaugeDelta" } // String returns a debug-friendly representation of this metric func (e FGaugeDelta) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %g}", e.TypeString(), e.Name, e.Value) } ================================================ FILE: event/fgaugedelta_test.go ================================================ package event import ( "reflect" "testing" ) func TestFGaugeDeltaUpdate(t *testing.T) { e1 := &FGaugeDelta{Name: "test", Value: float64(15.1)} e2 := &FGaugeDelta{Name: "test", Value: float64(-10.0)} e3 := &FGaugeDelta{Name: "test", Value: float64(15.1)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:+20.2|g"} actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } func TestFGaugeDeltaUpdateNegative(t *testing.T) { e1 := &FGaugeDelta{Name: "test", Value: float64(-15.1)} e2 := &FGaugeDelta{Name: "test", Value: float64(10.0)} e3 := &FGaugeDelta{Name: "test", Value: float64(-15.1)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:-20.2|g"} actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/gauge.go ================================================ package event import "fmt" // Gauge - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again type Gauge struct { Name string Value int64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *Gauge) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Value = e2.Payload().(int64) return nil } // Payload returns the aggregated value for this event func (e Gauge) Payload() interface{} { return e.Value } // Stats returns an array of StatsD events as they travel over UDP func (e Gauge) Stats() []string { if e.Value < 0 { // because a leading '+' or '-' in the value of a gauge denotes a delta, to send // a negative gauge value we first set the gauge absolutely to 0, then send the // negative value as a delta from 0 (that's just how the spec works :-) return []string{ fmt.Sprintf("%s:%d|g", e.Name, 0), fmt.Sprintf("%s:%d|g", e.Name, e.Value), } } return []string{fmt.Sprintf("%s:%d|g", e.Name, e.Value)} } // Key returns the name of this metric func (e Gauge) Key() string { return e.Name } // SetKey sets the name of this metric func (e *Gauge) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e Gauge) Type() int { return EventGauge } // TypeString returns a name for this type of metric func (e Gauge) TypeString() string { return "Gauge" } // String returns a debug-friendly representation of this metric func (e Gauge) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %d}", e.TypeString(), e.Name, e.Value) } ================================================ FILE: event/gauge_test.go ================================================ package event import ( "reflect" "testing" ) func TestGaugeUpdate(t *testing.T) { e1 := &Gauge{Name: "test", Value: int64(15)} e2 := &Gauge{Name: "test", Value: int64(-10)} e3 := &Gauge{Name: "test", Value: int64(8)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:8|g"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } func TestGaugeUpdateNegative(t *testing.T) { e1 := &Gauge{Name: "test", Value: int64(-10)} e2 := &Gauge{Name: "test", Value: int64(-3)} err := e1.Update(e2) if nil != err { t.Error(err) } expected := []string{"test:0|g", "test:-3|g"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/gaugedelta.go ================================================ package event import "fmt" // GaugeDelta - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again type GaugeDelta struct { Name string Value int64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *GaugeDelta) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Value += e2.Payload().(int64) return nil } // Payload returns the aggregated value for this event func (e GaugeDelta) Payload() interface{} { return e.Value } // Stats returns an array of StatsD events as they travel over UDP func (e GaugeDelta) Stats() []string { return []string{fmt.Sprintf("%s:%+d|g", e.Name, e.Value)} } // Key returns the name of this metric func (e GaugeDelta) Key() string { return e.Name } // SetKey sets the name of this metric func (e *GaugeDelta) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e GaugeDelta) Type() int { return EventGaugeDelta } // TypeString returns a name for this type of metric func (e GaugeDelta) TypeString() string { return "GaugeDelta" } // String returns a debug-friendly representation of this metric func (e GaugeDelta) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %d}", e.TypeString(), e.Name, e.Value) } ================================================ FILE: event/gaugedelta_test.go ================================================ package event import ( "reflect" "testing" ) func TestGaugeDeltaUpdate(t *testing.T) { e1 := &GaugeDelta{Name: "test", Value: int64(15)} e2 := &GaugeDelta{Name: "test", Value: int64(-10)} e3 := &GaugeDelta{Name: "test", Value: int64(15)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:+20|g"} actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } func TestGaugeDeltaUpdateNegative(t *testing.T) { e1 := &GaugeDelta{Name: "test", Value: int64(-15)} e2 := &GaugeDelta{Name: "test", Value: int64(10)} e3 := &GaugeDelta{Name: "test", Value: int64(-15)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:-20|g"} actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/increment.go ================================================ package event import "fmt" // Increment represents a metric whose value is averaged over a minute type Increment struct { Name string Value int64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *Increment) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Value += e2.Payload().(int64) return nil } // Payload returns the aggregated value for this event func (e Increment) Payload() interface{} { return e.Value } // Stats returns an array of StatsD events as they travel over UDP func (e Increment) Stats() []string { return []string{fmt.Sprintf("%s:%d|c", e.Name, e.Value)} } // Key returns the name of this metric func (e Increment) Key() string { return e.Name } // SetKey sets the name of this metric func (e *Increment) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e Increment) Type() int { return EventIncr } // TypeString returns a name for this type of metric func (e Increment) TypeString() string { return "Increment" } // String returns a debug-friendly representation of this metric func (e Increment) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %d}", e.TypeString(), e.Name, e.Value) } ================================================ FILE: event/increment_test.go ================================================ package event import ( "reflect" "testing" ) func TestIncrementUpdate(t *testing.T) { e1 := &Increment{Name: "test", Value: int64(15)} e2 := &Increment{Name: "test", Value: int64(-10)} e3 := &Increment{Name: "test", Value: int64(8)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:13|c"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/interface.go ================================================ package event // constant event type identifiers const ( EventIncr = iota EventTiming EventAbsolute EventTotal EventGauge EventGaugeDelta EventFGauge EventFGaugeDelta EventFAbsolute EventPrecisionTiming ) // Event is an interface to a generic StatsD event, used by the buffered client collator type Event interface { Stats() []string Type() int TypeString() string Payload() interface{} Update(e2 Event) error String() string Key() string SetKey(string) } // compile-time assertion to verify default events implement the Event interface func _() { var _ Event = (*Absolute)(nil) // assert *Absolute implements Event var _ Event = (*FAbsolute)(nil) // assert *FAbsolute implements Event var _ Event = (*Gauge)(nil) // assert *Gauge implements Event var _ Event = (*FGauge)(nil) // assert *FGauge implements Event var _ Event = (*GaugeDelta)(nil) // assert *GaugeDelta implements Event var _ Event = (*FGaugeDelta)(nil) // assert *FGaugeDelta implements Event var _ Event = (*Increment)(nil) // assert *Increment implements Event var _ Event = (*PrecisionTiming)(nil) // assert *PrecisionTiming implements Event var _ Event = (*Timing)(nil) // assert *Timing implements Event var _ Event = (*Total)(nil) // assert *Total implements Event } ================================================ FILE: event/precisiontiming.go ================================================ package event import ( "fmt" "time" ) // PrecisionTiming keeps min/max/avg information about a timer over a certain interval type PrecisionTiming struct { Name string Min time.Duration Max time.Duration Value time.Duration Count int64 } // NewPrecisionTiming is a factory for a Timing event, setting the Count to 1 to prevent div_by_0 errors func NewPrecisionTiming(k string, delta time.Duration) *PrecisionTiming { return &PrecisionTiming{Name: k, Min: delta, Max: delta, Value: delta, Count: 1} } // Update the event with metrics coming from a new one of the same type and with the same key func (e *PrecisionTiming) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } p := e2.Payload().(PrecisionTiming) e.Count += p.Count e.Value += p.Value e.Min = time.Duration(minInt64(int64(e.Min), int64(p.Min))) e.Max = time.Duration(maxInt64(int64(e.Max), int64(p.Min))) return nil } // Payload returns the aggregated value for this event func (e PrecisionTiming) Payload() interface{} { return e } // Stats returns an array of StatsD events as they travel over UDP func (e PrecisionTiming) Stats() []string { return []string{ fmt.Sprintf("%s.count:%d|c", e.Name, e.Count), fmt.Sprintf("%s.avg:%.6f|ms", e.Name, float64(int64(e.Value)/e.Count)/1000000), // make sure e.Count != 0 fmt.Sprintf("%s.min:%.6f|ms", e.Name, e.durationToMs(e.Min)), fmt.Sprintf("%s.max:%.6f|ms", e.Name, e.durationToMs(e.Max)), } } // durationToMs converts time.Duration into the corresponding value in milliseconds func (e PrecisionTiming) durationToMs(x time.Duration) float64 { return float64(x) / float64(time.Millisecond) } // Key returns the name of this metric func (e PrecisionTiming) Key() string { return e.Name } // SetKey sets the name of this metric func (e *PrecisionTiming) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e PrecisionTiming) Type() int { return EventPrecisionTiming } // TypeString returns a name for this type of metric func (e PrecisionTiming) TypeString() string { return "PrecisionTiming" } // String returns a debug-friendly representation of this metric func (e PrecisionTiming) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %+v}", e.TypeString(), e.Name, e.Payload()) } ================================================ FILE: event/precisiontiming_test.go ================================================ package event import ( "reflect" "testing" "time" ) func TestPrecisionTimingUpdate(t *testing.T) { e1 := NewPrecisionTiming("test", 5*time.Microsecond) e2 := NewPrecisionTiming("test", 3*time.Microsecond) e3 := NewPrecisionTiming("test", 7*time.Microsecond) err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test.count:3|c", "test.avg:0.005000|ms", "test.min:0.003000|ms", "test.max:0.007000|ms"} actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/timing.go ================================================ package event import "fmt" // Timing keeps min/max/avg information about a timer over a certain interval type Timing struct { Name string Min int64 Max int64 Value int64 Count int64 } // NewTiming is a factory for a Timing event, setting the Count to 1 to prevent div_by_0 errors func NewTiming(k string, delta int64) *Timing { return &Timing{Name: k, Min: delta, Max: delta, Value: delta, Count: 1} } // Update the event with metrics coming from a new one of the same type and with the same key func (e *Timing) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } p := e2.Payload().(map[string]int64) e.Count += p["cnt"] e.Value += p["val"] e.Min = minInt64(e.Min, p["min"]) e.Max = maxInt64(e.Max, p["max"]) return nil } // Payload returns the aggregated value for this event func (e Timing) Payload() interface{} { return map[string]int64{ "min": e.Min, "max": e.Max, "val": e.Value, "cnt": e.Count, } } // Stats returns an array of StatsD events as they travel over UDP func (e Timing) Stats() []string { return []string{ fmt.Sprintf("%s.count:%d|c", e.Name, e.Count), fmt.Sprintf("%s.avg:%d|ms", e.Name, int64(e.Value/e.Count)), // make sure e.Count != 0 fmt.Sprintf("%s.min:%d|ms", e.Name, e.Min), fmt.Sprintf("%s.max:%d|ms", e.Name, e.Max), } } // Key returns the name of this metric func (e Timing) Key() string { return e.Name } // SetKey sets the name of this metric func (e *Timing) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e Timing) Type() int { return EventTiming } // TypeString returns a name for this type of metric func (e Timing) TypeString() string { return "Timing" } // String returns a debug-friendly representation of this metric func (e Timing) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %+v}", e.TypeString(), e.Name, e.Payload()) } func minInt64(v1, v2 int64) int64 { if v1 <= v2 { return v1 } return v2 } func maxInt64(v1, v2 int64) int64 { if v1 >= v2 { return v1 } return v2 } ================================================ FILE: event/timing_test.go ================================================ package event import ( "reflect" "testing" ) func TestTimingUpdate(t *testing.T) { e1 := NewTiming("test", 5) e2 := NewTiming("test", 3) e3 := NewTiming("test", 7) err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test.count:3|c", "test.avg:5|ms", "test.min:3|ms", "test.max:7|ms"} actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: event/total.go ================================================ package event import "fmt" // Total represents a metric that is continously increasing, e.g. read operations since boot type Total struct { Name string Value int64 } // Update the event with metrics coming from a new one of the same type and with the same key func (e *Total) Update(e2 Event) error { if e.Type() != e2.Type() { return fmt.Errorf("statsd event type conflict: %s vs %s ", e.String(), e2.String()) } e.Value += e2.Payload().(int64) return nil } // Payload returns the aggregated value for this event func (e Total) Payload() interface{} { return e.Value } // Stats returns an array of StatsD events as they travel over UDP func (e Total) Stats() []string { return []string{fmt.Sprintf("%s:%d|t", e.Name, e.Value)} } // Key returns the name of this metric func (e Total) Key() string { return e.Name } // SetKey sets the name of this metric func (e *Total) SetKey(key string) { e.Name = key } // Type returns an integer identifier for this type of metric func (e Total) Type() int { return EventTotal } // TypeString returns a name for this type of metric func (e Total) TypeString() string { return "Total" } // String returns a debug-friendly representation of this metric func (e Total) String() string { return fmt.Sprintf("{Type: %s, Key: %s, Value: %d}", e.TypeString(), e.Name, e.Value) } ================================================ FILE: event/total_test.go ================================================ package event import ( "reflect" "testing" ) func TestTotalUpdate(t *testing.T) { e1 := &Total{Name: "test", Value: int64(15)} e2 := &Total{Name: "test", Value: int64(-10)} e3 := &Total{Name: "test", Value: int64(8)} err := e1.Update(e2) if nil != err { t.Error(err) } err = e1.Update(e3) if nil != err { t.Error(err) } expected := []string{"test:13|t"} // only the last value is flushed actual := e1.Stats() if !reflect.DeepEqual(expected, actual) { t.Errorf("did not receive all metrics: Expected: %T %v, Actual: %T %v ", expected, expected, actual, actual) } } ================================================ FILE: interface.go ================================================ package statsd import ( "time" "github.com/quipo/statsd/event" ) // Statsd is an interface to a StatsD client (buffered/unbuffered) type Statsd interface { CreateSocket() error CreateTCPSocket() error Close() error Incr(stat string, count int64) error Decr(stat string, count int64) error Timing(stat string, delta int64) error PrecisionTiming(stat string, delta time.Duration) error Gauge(stat string, value int64) error GaugeDelta(stat string, value int64) error Absolute(stat string, value int64) error Total(stat string, value int64) error FGauge(stat string, value float64) error FGaugeDelta(stat string, value float64) error FAbsolute(stat string, value float64) error SendEvents(events map[string]event.Event) error } ================================================ FILE: mock/mockableclient.go ================================================ package mock // // A mockable client allowing arbitrary functions to be called statsd.Statsd methods. // // This is particularly helpful in unit test scenarios where it is desired to simulate calls // without actually writing to a network or filesystem. // // A default implementation is provided that records all calls and can be used for verification // in unit tests. // // The default implementations of these methods are no-ops, so without any further configuration, // &MockStatsdClient{} is equivalent to statsd.NoopClient. But utility methods are also provided that // allow recording calls for the purposes of verification during unit testing. // import ( "sync" "time" "github.com/quipo/statsd/event" ) type statelessStatsdFunction func() error type intMetricStatsdFunction func(string, int64) error type floatMetricStatsdFunction func(string, float64) error type durationMetricStatsdFunction func(string, time.Duration) error type eventsStatsdFunction func(events map[string]event.Event) error // MockStatsdClient at its simplest provides a layer of indirection so that // arbitrary functions can be used as the targets of calls to the Statsd interface // // In its basic state, it will act like a noop client. // // There are also helper functions that will set functions for specific // calls to record those events to caller-provided slices. // This is particularly helpful for unit testing that needs to verify // that certain metrics have been recorded by the system under test type MockStatsdClient struct { CreateSocketFn statelessStatsdFunction CreateTCPSocketFn statelessStatsdFunction CloseFn statelessStatsdFunction IncrFn intMetricStatsdFunction DecrFn intMetricStatsdFunction TimingFn intMetricStatsdFunction PrecisionTimingFn durationMetricStatsdFunction GaugeFn intMetricStatsdFunction GaugeDeltaFn intMetricStatsdFunction AbsoluteFn intMetricStatsdFunction TotalFn intMetricStatsdFunction FGaugeFn floatMetricStatsdFunction FGaugeDeltaFn floatMetricStatsdFunction FAbsoluteFn floatMetricStatsdFunction SendEventsFn eventsStatsdFunction } // Implement statsd interface func (msc *MockStatsdClient) CreateSocket() error { if msc.CreateSocketFn == nil { return nil } return msc.CreateSocketFn() } func (msc *MockStatsdClient) CreateTCPSocket() error { if msc.CreateTCPSocketFn == nil { return nil } return msc.CreateTCPSocketFn() } func (msc *MockStatsdClient) Close() error { if msc.CloseFn == nil { return nil } return msc.CloseFn() } func (msc *MockStatsdClient) Incr(stat string, count int64) error { if msc.IncrFn == nil { return nil } return msc.IncrFn(stat, count) } func (msc *MockStatsdClient) Decr(stat string, count int64) error { if msc.DecrFn == nil { return nil } return msc.DecrFn(stat, count) } func (msc *MockStatsdClient) Timing(stat string, delta int64) error { if msc.TimingFn == nil { return nil } return msc.TimingFn(stat, delta) } func (msc *MockStatsdClient) PrecisionTiming(stat string, delta time.Duration) error { if msc.PrecisionTimingFn == nil { return nil } return msc.PrecisionTimingFn(stat, delta) } func (msc *MockStatsdClient) Gauge(stat string, value int64) error { if msc.GaugeFn == nil { return nil } return msc.GaugeFn(stat, value) } func (msc *MockStatsdClient) GaugeDelta(stat string, value int64) error { if msc.GaugeDeltaFn == nil { return nil } return msc.GaugeDeltaFn(stat, value) } func (msc *MockStatsdClient) Absolute(stat string, value int64) error { if msc.AbsoluteFn == nil { return nil } return msc.AbsoluteFn(stat, value) } func (msc *MockStatsdClient) Total(stat string, value int64) error { if msc.TotalFn == nil { return nil } return msc.TotalFn(stat, value) } func (msc *MockStatsdClient) FGauge(stat string, value float64) error { if msc.FGaugeFn == nil { return nil } return msc.FGaugeFn(stat, value) } func (msc *MockStatsdClient) FGaugeDelta(stat string, value float64) error { if msc.FGaugeDeltaFn == nil { return nil } return msc.FGaugeDeltaFn(stat, value) } func (msc *MockStatsdClient) FAbsolute(stat string, value float64) error { if msc.FAbsoluteFn == nil { return nil } return msc.FAbsoluteFn(stat, value) } func (msc *MockStatsdClient) SendEvents(events map[string]event.Event) error { if msc.SendEventsFn == nil { return nil } return msc.SendEventsFn(events) } // Mocking helpers that record seen events for verification during unit testing type Int64Event struct { MetricName string EventValue int64 } type Float64Event struct { MetricName string EventValue float64 } type DurationEvent struct { MetricName string EventValue time.Duration } // UnvaluedEvents are useful for recording things like calls to Close() or CreateSocket() type UnvaluedEvent struct { } // Fluent-style constructors for recording events to caller-provided slices // This means that during unit tests, one can do something like // // incrEvents := []statsd.Int64Event // decrEvents := []statsd.Int64Event // statsdClient = &MockStatsdClient{}.RecordIncrEventsTo(&incrEvents).RecordDecrEventsTo(&decrEvents) // ... Execute code under test that records metrics // ... Verify that incrEvents and decrEvents have seen the expected events func (msc *MockStatsdClient) RecordCreateSocketEventsTo(createSocketEvents *[]UnvaluedEvent) *MockStatsdClient { eventLock := &sync.Mutex{} msc.CreateSocketFn = func() error { recordUnvaluedEvent(eventLock, createSocketEvents) return nil } return msc } func (msc *MockStatsdClient) RecordCreateTCPSocketEventsTo(createTcpSocketEvents *[]UnvaluedEvent) *MockStatsdClient { eventLock := &sync.Mutex{} msc.CreateTCPSocketFn = func() error { recordUnvaluedEvent(eventLock, createTcpSocketEvents) return nil } return msc } func (msc *MockStatsdClient) RecordCloseEventsTo(closeEvents *[]UnvaluedEvent) *MockStatsdClient { eventLock := &sync.Mutex{} msc.CloseFn = func() error { recordUnvaluedEvent(eventLock, closeEvents) return nil } return msc } func (msc *MockStatsdClient) RecordIncrEventsTo(incrEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.IncrFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, incrEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordDecrEventsTo(decrEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.DecrFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, decrEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordTimingEventsTo(timingEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.TimingFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, timingEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordPrecisionTimingEventsTo(timingEvents *[]DurationEvent) *MockStatsdClient { eventLock := &sync.Mutex{} msc.PrecisionTimingFn = func(metricName string, eventValue time.Duration) error { recordDurationEvent(eventLock, timingEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordGaugeEventsTo(gaugeEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.GaugeFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, gaugeEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordGaugeDeltaEventsTo(gaugeDeltaEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.GaugeDeltaFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, gaugeDeltaEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordAbsoluteEventsTo(absoluteEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.AbsoluteFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, absoluteEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordTotalEventsTo(totalEvents *[]Int64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.TotalFn = func(metricName string, eventValue int64) error { recordInt64Event(eventLock, totalEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordFGaugeEventsTo(fgaugeEvents *[]Float64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.FGaugeFn = func(metricName string, eventValue float64) error { recordFloat64Event(eventLock, fgaugeEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordFGaugeDeltaEventsTo(fgaugeDeltaEvents *[]Float64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.FGaugeDeltaFn = func(metricName string, eventValue float64) error { recordFloat64Event(eventLock, fgaugeDeltaEvents, metricName, eventValue) return nil } return msc } func (msc *MockStatsdClient) RecordFAbsoluteEventsTo(fabsoluteEvents *[]Float64Event) *MockStatsdClient { eventLock := &sync.Mutex{} msc.FAbsoluteFn = func(metricName string, eventValue float64) error { recordFloat64Event(eventLock, fabsoluteEvents, metricName, eventValue) return nil } return msc } func recordDurationEvent(eventLock sync.Locker, events *[]DurationEvent, metricName string, eventValue time.Duration) { newEvent := DurationEvent{ MetricName: metricName, EventValue: eventValue, } eventLock.Lock() defer eventLock.Unlock() *events = append(*events, newEvent) } func recordFloat64Event(eventLock sync.Locker, events *[]Float64Event, metricName string, eventValue float64) { newEvent := Float64Event{ MetricName: metricName, EventValue: eventValue, } eventLock.Lock() defer eventLock.Unlock() *events = append(*events, newEvent) } func recordInt64Event(eventLock sync.Locker, events *[]Int64Event, metricName string, eventValue int64) { newEvent := Int64Event{ MetricName: metricName, EventValue: eventValue, } eventLock.Lock() defer eventLock.Unlock() *events = append(*events, newEvent) } func recordUnvaluedEvent(eventLock sync.Locker, events *[]UnvaluedEvent) { eventLock.Lock() defer eventLock.Unlock() *events = append(*events, UnvaluedEvent{}) } ================================================ FILE: mock/mockableclient_test.go ================================================ package mock import ( "reflect" "testing" "github.com/quipo/statsd/event" ) func TestNoopBehavior(t *testing.T) { mockClient := MockStatsdClient{} err := mockClient.CreateSocket() if err != nil { t.Fail() } err = mockClient.CreateTCPSocket() if err != nil { t.Fail() } err = mockClient.Close() if err != nil { t.Fail() } err = mockClient.Incr("incr", 1) if err != nil { t.Fail() } err = mockClient.Decr("decr", 2) if err != nil { t.Fail() } err = mockClient.Timing("timing", 3) if err != nil { t.Fail() } err = mockClient.PrecisionTiming("precisionTiming", 4) if err != nil { t.Fail() } err = mockClient.Gauge("gauge", 5) if err != nil { t.Fail() } err = mockClient.GaugeDelta("gaugeDelta", 6) if err != nil { t.Fail() } err = mockClient.Absolute("absolute", 7) if err != nil { t.Fail() } err = mockClient.Total("total", 8) if err != nil { t.Fail() } err = mockClient.FGauge("fgauge", 10.0) if err != nil { t.Fail() } err = mockClient.FGaugeDelta("fgaugeDelta", 11.0) if err != nil { t.Fail() } err = mockClient.FAbsolute("fabsolute", 12.0) if err != nil { t.Fail() } err = mockClient.SendEvents(make(map[string]event.Event)) if err != nil { t.Fail() } } func TestMockStatsdClient_RecordCreateSocketEventsTo(t *testing.T) { var createSocketEvents []UnvaluedEvent mockClient := (&MockStatsdClient{}).RecordCreateSocketEventsTo(&createSocketEvents) err := mockClient.CreateSocket() if err != nil { t.Logf("Got non-nil err from mock CreateSocket") t.Fail() } expectedEvents := []UnvaluedEvent{UnvaluedEvent{}} if !reflect.DeepEqual(createSocketEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, createSocketEvents) t.Fail() } } func TestMockStatsdClient_RecordCreateTCPSocketEventsTo(t *testing.T) { var createTCPSocketEvents []UnvaluedEvent mockClient := (&MockStatsdClient{}).RecordCreateTCPSocketEventsTo(&createTCPSocketEvents) err := mockClient.CreateTCPSocket() if err != nil { t.Logf("Got non-nil err from mock CreateTCPSocket") t.Fail() } expectedEvents := []UnvaluedEvent{UnvaluedEvent{}} if !reflect.DeepEqual(createTCPSocketEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, createTCPSocketEvents) t.Fail() } } func TestMockStatsdClient_RecordCloseEventsTo(t *testing.T) { var closeEvents []UnvaluedEvent mockClient := (&MockStatsdClient{}).RecordCloseEventsTo(&closeEvents) err := mockClient.Close() if err != nil { t.Logf("Got non-nil err from mock Close") t.Fail() } expectedEvents := []UnvaluedEvent{UnvaluedEvent{}} if !reflect.DeepEqual(closeEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, closeEvents) t.Fail() } } func TestMockStatsdClient_RecordIncrEventsTo(t *testing.T) { var incrEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordIncrEventsTo(&incrEvents) err := mockClient.Incr("incr", 1) if err != nil { t.Logf("Got non-nil err from mock Incr") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "incr", EventValue: 1}} if !reflect.DeepEqual(incrEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, incrEvents) t.Fail() } } func TestMockStatsdClient_Decr(t *testing.T) { var decrEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordDecrEventsTo(&decrEvents) err := mockClient.Decr("decr", 1) if err != nil { t.Logf("Got non-nil err from mock Decr") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "decr", EventValue: 1}} if !reflect.DeepEqual(decrEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, decrEvents) t.Fail() } } func TestMockStatsdClient_Timing(t *testing.T) { var timingEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordIncrEventsTo(&timingEvents) err := mockClient.Incr("timing", 1) if err != nil { t.Logf("Got non-nil err from mock Timing") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "timing", EventValue: 1}} if !reflect.DeepEqual(timingEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, timingEvents) t.Fail() } } func TestMockStatsdClient_RecordPrecisionTimingEventsTo(t *testing.T) { var durationEvents []DurationEvent mockClient := (&MockStatsdClient{}).RecordPrecisionTimingEventsTo(&durationEvents) err := mockClient.PrecisionTiming("precisionTiming", 1) if err != nil { t.Logf("Got non-nil err from mock PrecisionTiming") t.Fail() } expectedEvents := []DurationEvent{DurationEvent{MetricName: "precisionTiming", EventValue: 1}} if !reflect.DeepEqual(durationEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, durationEvents) t.Fail() } } func TestMockStatsdClient_RecordGaugeEventsTo(t *testing.T) { var gaugeEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordGaugeEventsTo(&gaugeEvents) err := mockClient.Gauge("gauge", 1) if err != nil { t.Logf("Got non-nil err from mock Gauge") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "gauge", EventValue: 1}} if !reflect.DeepEqual(gaugeEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, gaugeEvents) t.Fail() } } func TestMockStatsdClient_RecordFGaugeDeltaEventsTo(t *testing.T) { var gaugeDeltaEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordGaugeDeltaEventsTo(&gaugeDeltaEvents) err := mockClient.GaugeDelta("gaugeDelta", 1) if err != nil { t.Logf("Got non-nil err from mock GaugeDelta") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "gaugeDelta", EventValue: 1}} if !reflect.DeepEqual(gaugeDeltaEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, gaugeDeltaEvents) t.Fail() } } func TestMockStatsdClient_RecordAbsoluteEventsTo(t *testing.T) { var absoluteEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordAbsoluteEventsTo(&absoluteEvents) err := mockClient.Absolute("absolute", 1) if err != nil { t.Logf("Got non-nil err from mock Absolute") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "absolute", EventValue: 1}} if !reflect.DeepEqual(absoluteEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, absoluteEvents) t.Fail() } } func TestMockStatsdClient_RecordTotalEventsTo(t *testing.T) { var totalEvents []Int64Event mockClient := (&MockStatsdClient{}).RecordTotalEventsTo(&totalEvents) err := mockClient.Total("total", 1) if err != nil { t.Logf("Got non-nil err from mock Total") t.Fail() } expectedEvents := []Int64Event{Int64Event{MetricName: "total", EventValue: 1}} if !reflect.DeepEqual(totalEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, totalEvents) t.Fail() } } func TestMockStatsdClient_RecordFGaugeEventsTo(t *testing.T) { var fgaugeEvents []Float64Event mockClient := (&MockStatsdClient{}).RecordFGaugeEventsTo(&fgaugeEvents) err := mockClient.FGauge("fgauge", 1) if err != nil { t.Logf("Got non-nil err from mock FGauge") t.Fail() } expectedEvents := []Float64Event{Float64Event{MetricName: "fgauge", EventValue: 1}} if !reflect.DeepEqual(fgaugeEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, fgaugeEvents) t.Fail() } } func TestMockStatsdClient_RecordFGaugeDeltaEventsTo2(t *testing.T) { var fgaugeDeltaEvents []Float64Event mockClient := (&MockStatsdClient{}).RecordFGaugeDeltaEventsTo(&fgaugeDeltaEvents) err := mockClient.FGaugeDelta("fgaugeDelta", 1) if err != nil { t.Logf("Got non-nil err from mock FGaugeDelta") t.Fail() } expectedEvents := []Float64Event{Float64Event{MetricName: "fgaugeDelta", EventValue: 1}} if !reflect.DeepEqual(fgaugeDeltaEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, fgaugeDeltaEvents) t.Fail() } } func TestMockStatsdClient_RecordFAbsoluteEventsTo(t *testing.T) { var fabsoluteEvents []Float64Event mockClient := (&MockStatsdClient{}).RecordFAbsoluteEventsTo(&fabsoluteEvents) err := mockClient.FAbsolute("fabsolute", 1) if err != nil { t.Logf("Got non-nil err from mock FAbsolute") t.Fail() } expectedEvents := []Float64Event{Float64Event{MetricName: "fabsolute", EventValue: 1}} if !reflect.DeepEqual(fabsoluteEvents, expectedEvents) { t.Logf("Expected %s, saw %s", expectedEvents, fabsoluteEvents) t.Fail() } } //func TestRecordingBuilders(t *testing.T) { // var createTcpSocketEvents []UnvaluedEvent // var closeEvents []UnvaluedEvent // var incrEvents []Int64Event // var decrEvents []Int64Event // var timingEvents []Int64Event // var precidionTimingEvents []DurationEvent // var gaugeEvents []Int64Event // var gaugeDeltaEvents []Int64Event // var absoluteEvents []Int64Event // var totalEvents []Int64Event // var fgaugeEvents []Float64Event // var fgaugeDeltaEvents []Float64Event // var fabsoluteEvents []Float64Event // // mockClient := &MockStatsdClient{}. // RecordCreateSocketEventsTo(&createSocketEvents). // RecordCreateTCPSocketEventsTo(&createTcpSocketEvents). // RecordCloseEventsTo(&closeEvents). // RecordIncrEventsTo(&incrEvents). // RecordDecrEventsTo(&decrEvents). // RecordTimingEventsTo(&timingEvents). // RecordPrecisionTimingEventsTo(&precidionTimingEvents). // RecordGaugeEventsTo(&gaugeEvents). // RecordGaugeDeltaEventsTo(&gaugeDeltaEvents). // RecordAbsoluteEventsTo(&absoluteEvents). // RecordTotalEventsTo(&totalEvents). // RecordFGaugeEventsTo(&fgaugeEvents). // RecordFGaugeDeltaEventsTo(&fgaugeDeltaEvents). // RecordFAbsoluteEventsTo(&fabsoluteEvents) // // err := mockClient.CreateSocket() // if err != nil { // t.Fail() // } // err = mockClient.CreateTCPSocket() // if err != nil { // t.Fail() // } // err = mockClient.Close() // if err != nil { // t.Fail() // } // err = mockClient.Incr("incr", 1) // if err != nil { // t.Fail() // } // err = mockClient.Decr("decr", 1) // if err != nil { // t.Fail() // } // err = mockClient.Timing("timing", 1) // if err != nil { // t.Fail() // } //} ================================================ FILE: noopclient.go ================================================ package statsd //@author https://github.com/wyndhblb/statsd import ( "time" "github.com/quipo/statsd/event" ) // NoopClient implements a "no-op" statsd in case there is no statsd server type NoopClient struct{} // CreateSocket does nothing func (s NoopClient) CreateSocket() error { return nil } // CreateTCPSocket does nothing func (s NoopClient) CreateTCPSocket() error { return nil } // Close does nothing func (s NoopClient) Close() error { return nil } // Incr does nothing func (s NoopClient) Incr(stat string, count int64) error { return nil } // Decr does nothing func (s NoopClient) Decr(stat string, count int64) error { return nil } // Timing does nothing func (s NoopClient) Timing(stat string, count int64) error { return nil } // PrecisionTiming does nothing func (s NoopClient) PrecisionTiming(stat string, delta time.Duration) error { return nil } // Gauge does nothing func (s NoopClient) Gauge(stat string, value int64) error { return nil } // GaugeDelta does nothing func (s NoopClient) GaugeDelta(stat string, value int64) error { return nil } // Absolute does nothing func (s NoopClient) Absolute(stat string, value int64) error { return nil } // Total does nothing func (s NoopClient) Total(stat string, value int64) error { return nil } // FGauge does nothing func (s NoopClient) FGauge(stat string, value float64) error { return nil } // FGaugeDelta does nothing func (s NoopClient) FGaugeDelta(stat string, value float64) error { return nil } // FAbsolute does nothing func (s NoopClient) FAbsolute(stat string, value float64) error { return nil } // SendEvents does nothing func (s NoopClient) SendEvents(events map[string]event.Event) error { return nil } ================================================ FILE: stdoutclient.go ================================================ package statsd import ( "fmt" "log" "os" "strings" "time" "github.com/quipo/statsd/event" ) // StdoutClient implements a "no-op" statsd in case there is no statsd server type StdoutClient struct { FD *os.File prefix string Logger Logger } // NewStdoutClient - Factory func NewStdoutClient(filename string, prefix string) *StdoutClient { var err error // allow %HOST% in the prefix string prefix = strings.Replace(prefix, "%HOST%", Hostname, 1) var fh *os.File if filename == "" { fh = os.Stdout } else { fh, err = os.OpenFile(filename, os.O_WRONLY, 0644) if nil != err { fmt.Printf("Cannot open file '%s' for stats output: %s\n", filename, err.Error()) } } return &StdoutClient{ FD: fh, prefix: prefix, Logger: log.New(os.Stdout, "[StdoutClient] ", log.Ldate|log.Ltime), } } // CreateSocket does nothing func (s *StdoutClient) CreateSocket() error { if s.FD == nil { s.FD = os.Stdout } return nil } // CreateTCPSocket does nothing func (s *StdoutClient) CreateTCPSocket() error { if s.FD == nil { s.FD = os.Stdout } return nil } // Close does nothing func (s *StdoutClient) Close() error { return nil } // Incr - Increment a counter metric. Often used to note a particular event func (s *StdoutClient) Incr(stat string, count int64) error { if 0 != count { return s.send(stat, "%d|c", count) } return nil } // Decr - Decrement a counter metric. Often used to note a particular event func (s *StdoutClient) Decr(stat string, count int64) error { if 0 != count { return s.send(stat, "%d|c", -count) } return nil } // Timing - Track a duration event // the time delta must be given in milliseconds func (s *StdoutClient) Timing(stat string, delta int64) error { return s.send(stat, "%d|ms", delta) } // PrecisionTiming - Track a duration event // the time delta has to be a duration func (s *StdoutClient) PrecisionTiming(stat string, delta time.Duration) error { return s.send(stat, "%.6f|ms", float64(delta)/float64(time.Millisecond)) } // Gauge - Gauges are a constant data type. They are not subject to averaging, // and they don’t change unless you change them. That is, once you set a gauge value, // it will be a flat line on the graph until you change it again. If you specify // delta to be true, that specifies that the gauge should be updated, not set. Due to the // underlying protocol, you can't explicitly set a gauge to a negative number without // first setting it to zero. func (s *StdoutClient) Gauge(stat string, value int64) error { if value < 0 { err := s.send(stat, "%d|g", 0) if nil != err { return err } return s.send(stat, "%d|g", value) } return s.send(stat, "%d|g", value) } // GaugeDelta -- Send a change for a gauge func (s *StdoutClient) GaugeDelta(stat string, value int64) error { // Gauge Deltas are always sent with a leading '+' or '-'. The '-' takes care of itself but the '+' must added by hand if value < 0 { return s.send(stat, "%d|g", value) } return s.send(stat, "+%d|g", value) } // FGauge -- Send a floating point value for a gauge func (s *StdoutClient) FGauge(stat string, value float64) error { if value < 0 { err := s.send(stat, "%d|g", 0) if nil != err { return err } return s.send(stat, "%g|g", value) } return s.send(stat, "%g|g", value) } // FGaugeDelta -- Send a floating point change for a gauge func (s *StdoutClient) FGaugeDelta(stat string, value float64) error { if value < 0 { return s.send(stat, "%g|g", value) } return s.send(stat, "+%g|g", value) } // Absolute - Send absolute-valued metric (not averaged/aggregated) func (s *StdoutClient) Absolute(stat string, value int64) error { return s.send(stat, "%d|a", value) } // FAbsolute - Send absolute-valued floating point metric (not averaged/aggregated) func (s *StdoutClient) FAbsolute(stat string, value float64) error { return s.send(stat, "%g|a", value) } // Total - Send a metric that is continously increasing, e.g. read operations since boot func (s *StdoutClient) Total(stat string, value int64) error { return s.send(stat, "%d|t", value) } // write a UDP packet with the statsd event func (s *StdoutClient) send(stat string, format string, value interface{}) error { stat = strings.Replace(stat, "%HOST%", Hostname, 1) // if sending tcp append a newline format = fmt.Sprintf("%s%s:%s\n", s.prefix, stat, format) _, err := fmt.Fprintf(s.FD, format, value) return err } // SendEvent - Sends stats from an event object func (s *StdoutClient) SendEvent(e event.Event) error { for _, stat := range e.Stats() { //fmt.Printf("SENDING EVENT %s%s\n", s.prefix, strings.Replace(stat, "%HOST%", Hostname, 1)) _, err := fmt.Fprintf(s.FD, "%s%s", s.prefix, strings.Replace(stat, "%HOST%", Hostname, 1)) if nil != err { return err } } return nil } // SendEvents - Sends stats from all the event objects. // Tries to bundle many together into one fmt.Fprintf based on UDPPayloadSize. func (s *StdoutClient) SendEvents(events map[string]event.Event) error { var n int var stats = make([]string, 0) for _, e := range events { for _, stat := range e.Stats() { stat = fmt.Sprintf("%s%s", s.prefix, strings.Replace(stat, "%HOST%", Hostname, 1)) _n := n + len(stat) + 1 if _n > UDPPayloadSize { // with this last event, the UDP payload would be too big if _, err := fmt.Fprintf(s.FD, strings.Join(stats, "\n")); err != nil { return err } // reset payload after flushing, and add the last event stats = []string{stat} n = len(stat) continue } // can fit more into the current payload n = _n stats = append(stats, stat) } } if len(stats) != 0 { if _, err := fmt.Fprintf(s.FD, strings.Join(stats, "\n")); err != nil { return err } } return nil }