Showing preview only (294K chars total). Download the full file or copy to clipboard to get everything.
Repository: JustinAzoff/bro-pdns
Branch: main
Commit: 1255b1e4a055
Files: 34
Total size: 282.2 KB
Directory structure:
gitextract_ljrd_5b5/
├── .gitignore
├── .travis.yml
├── LICENSE
├── Makefile
├── README.md
├── aggregate.go
├── aggregate_test.go
├── dns-ans-query.bro
├── go.mod
├── go.sum
├── index.go
├── local_test.txt
├── main.go
├── read.go
├── read_ascii.go
├── read_json.go
├── read_test.go
├── store.go
├── store_clickhouse.go
├── store_pg.go
├── store_sql_common.go
├── store_sqlite.go
├── store_test.go
├── template/
│ └── index.html
├── test_data/
│ ├── bad_ttl.log
│ ├── dns_json.log
│ ├── dns_json_iso8601.json
│ ├── garbage.log
│ ├── nbtstat.log
│ ├── reddit_1.txt
│ ├── reddit_2.txt
│ └── reddit_dns_2016-04-01.log
├── version.go
└── web.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.py[co]
# Packages
*.egg
*.egg-info
dist
build
eggs
parts
bin
develop-eggs
.installed.cfg
# Installer logs
pip-log.txt
# Unit test / coverage reports
.coverage
.tox
bro-pdns
bro-pdns_linux_amd64
================================================
FILE: .travis.yml
================================================
language: go
go:
- "1.16"
services:
- postgresql
- docker
env:
- PG_TEST_URL=postgres://postgres:password@localhost/pdns_test?sslmode=disable
- CH_TEST_URL=tcp://localhost:9000/default
before_script:
- psql -c 'create database pdns_test;' -U postgres
- docker run -d -p 127.0.0.1:9000:9000 -p 127.0.0.1:8123:8123 --name test-clickhouse-server --ulimit nofile=262144:262144 yandex/clickhouse-server
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2022 Justin Azoff
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Makefile
================================================
all: build test
build:
go get -t -v ./...
go build
test:
go test -v ./...
static:
go get -t -v ./...
go build --ldflags '-extldflags "-static"'
.PHONY: rpm
rpm: build
rpm: VERSION=$(shell ./zeek-pdns version)
rpm:
fpm -f -s dir -t rpm -n zeek-pdns -v $(VERSION) \
--iteration=1 \
--architecture native \
--description "Zeek Passive DNS" \
./zeek-pdns=/usr/bin/
================================================
FILE: README.md
================================================
Passive DNS for Zeek
===================
This is an extremely simple implementation of a passive DNS collection system
that utilizes Zeek for DNS log collection.
Passive DNS collection can be used for various security or troubleshooting
purposes. Many queries to raw DNS logs can be done faster by using
the aggregated data in the passive DNS database, which is more compact.
This tool uses the Zeek DNS logs to build a database of unique query+type+answer
tuples.
It produces a table like this:
pdns=# select * from dns where answer='74.125.225.18' order by last desc limit 4;
query | type | answer | count | ttl | first | last
------------------+------+---------------+-------+-----+---------------+------------
www.google.com | A | 74.125.225.18 | 7517 | 198 | 2014-09-03 .. | 2014-10-30 ..
t0.gstatic.com | A | 74.125.225.18 | 266 | 300 | 2014-09-03 .. | 2014-10-30 ..
googlegroups.com | A | 74.125.225.18 | 266 | 300 | 2014-09-03 .. | 2014-10-30 ..
t3.gstatic.com | A | 74.125.225.18 | 291 | 300 | 2014-09-03 .. | 2014-10-30 ..
This is helpful because the PTR record itself for 74.125.225.18 is ord08s12-in-f18.1e100.net.
Examples of questions this database can answer faster than the raw logs:
* Did anything ever resolve example.com, and if so, when was the first time?
* What IPs has example.com resolved to?
* What other names resolve to this IP?
Requirements
------------
* go compiler ( to build )
* postgresql ( optional )
* clickhouse ( optional )
Build
-----
$ go build
Index logs
----------
# for postgresql
export PDNS_STORE_TYPE="postgresql"
export PDNS_STORE_URI="postgres://pdns:foo@localhost/pdns?sslmode=disable"
# for clickhouse
export PDNS_STORE_TYPE="clickhouse"
export PDNS_STORE_URI="tcp://localhost:9000/?database=pdns"
# for built in sqlite
export PDNS_STORE_TYPE="sqlite"
export PDNS_STORE_URI="/path/to/passivedns.sqlite"
# then finally index logs
find /usr/local/zeek/logs -name 'dns*' | sort -n | xargs -n 50 zeek-pdns index
Query Database
--------------
# suffix search:
$ zeek-pdns like tuples google.com
$ zeek-pdns like individual google.com
# exact match
$ zeek-pdns find tuples google.com
$ zeek-pdns find individual google.com
Start HTTP server
-----------------
$ zeek-pdns web
Query HTTP API
--------------
$ curl localhost:8080/dns/like/tuples/google.com
$ curl localhost:8080/dns/like/individual/google.com
$ curl localhost:8080/dns/find/tuples/google.com
$ curl localhost:8080/dns/find/individual/google.com
================================================
FILE: aggregate.go
================================================
package main
import (
"encoding/json"
"errors"
"io"
"log"
"strconv"
"strings"
"time"
opendecompress "github.com/JustinAzoff/go-opendecompress"
)
var MAX_SANE_VALUE_LEN = 1000
func stripDecimal(value string) string {
if value == "-" {
return "0"
}
idx := strings.Index(value, ".")
if idx == -1 {
return value
}
return value[:idx]
}
type DNSRecord struct {
ts string
query string
qtype string
answers []string
ttls []string
}
type uniqueTuple struct {
query string
answer string
qtype string
}
type uniqueIndividual struct {
value string
which string // "Q" or "A"
}
type queryStat struct {
count uint
first string
last string
ttl string
}
type aggregationResult struct {
Duration time.Duration
TotalRecords uint
SkippedRecords uint
Tuples []aggregatedTuple
TuplesLen int
Individual []aggregatedIndividual
IndividualLen int
}
type aggregatedTuple struct {
uniqueTuple
queryStat
}
type aggregatedIndividual struct {
uniqueIndividual
queryStat
}
type DNSAggregator struct {
queries map[uniqueTuple]*queryStat
values map[uniqueIndividual]*queryStat
totalRecords uint
skippedRecords uint
start time.Time
}
func NewDNSAggregator() *DNSAggregator {
queries := make(map[uniqueTuple]*queryStat)
values := make(map[uniqueIndividual]*queryStat)
return &DNSAggregator{
queries: queries,
values: values,
start: time.Now(),
}
}
func (d *DNSAggregator) SkipRecord() {
d.skippedRecords++
}
func (d *DNSAggregator) AddRecord(r DNSRecord) {
if len(r.query) > MAX_SANE_VALUE_LEN {
log.Printf("Skipping record with insane query length: %#v\n", r)
d.skippedRecords++
return
}
r.query = strings.TrimRight(r.query, "\u0000")
if strings.ContainsRune(r.query, '\u0000') {
log.Printf("Skipping record with null byte in query: %#v\n", r)
d.skippedRecords++
return
}
d.totalRecords++
query_value := uniqueIndividual{value: r.query, which: "Q"}
arec := d.values[query_value]
if arec == nil {
arec = &queryStat{
first: r.ts,
last: r.ts,
count: 1,
}
d.values[query_value] = arec
} else {
arec.count++
arec.last = r.ts
}
for idx, answer := range r.answers {
if len(answer) > MAX_SANE_VALUE_LEN {
log.Printf("Skipping record with insane answer length: %#v\n", r)
d.skippedRecords++
return
}
if answer == "-" {
continue
}
ttl := stripDecimal(r.ttls[idx])
//Validate that a ttl fits in a 32bit int
_, err := strconv.ParseInt(ttl, 10, 32)
if err != nil {
log.Printf("Skipping record with insane ttl: %#v\n", r)
d.skippedRecords++
return
}
if len(ttl) > 0 && ttl[0] == '-' {
ttl = "0"
}
uquery := uniqueTuple{
query: r.query,
answer: answer,
qtype: r.qtype,
}
rec := d.queries[uquery]
if rec == nil {
rec = &queryStat{
first: r.ts,
last: r.ts,
ttl: ttl,
count: 1,
}
d.queries[uquery] = rec
} else {
rec.count++
rec.last = r.ts
rec.ttl = ttl
}
answer_value := uniqueIndividual{value: answer, which: "A"}
arec := d.values[answer_value]
if arec == nil {
arec = &queryStat{
first: r.ts,
last: r.ts,
ttl: ttl,
count: 1,
}
d.values[answer_value] = arec
} else {
arec.count++
arec.last = r.ts
arec.ttl = ttl
}
}
}
func (d *DNSAggregator) GetResult() aggregationResult {
var result aggregationResult
for q, stat := range d.queries {
agg := aggregatedTuple{
uniqueTuple: q,
queryStat: *stat,
}
result.Tuples = append(result.Tuples, agg)
}
for value, stat := range d.values {
agg := aggregatedIndividual{
uniqueIndividual: value,
queryStat: *stat,
}
result.Individual = append(result.Individual, agg)
}
result.TotalRecords = d.totalRecords
result.SkippedRecords = d.skippedRecords
result.Duration = time.Since(d.start)
result.TuplesLen = len(result.Tuples)
result.IndividualLen = len(result.Individual)
return result
}
//timeCompare compares timestamps, doesn't care about subsecond
func timeCompare(a, b string) int {
a = stripDecimal(a)
b = stripDecimal(b)
if strings.Contains(a, "-") {
//Formatted timestamps are the same length, and can just be
//Compared as is
if a < b {
return -1
} else if a > b {
return 1
} else {
return 0
}
} else {
ai, err := strconv.ParseInt(a, 10, 64)
if err != nil {
log.Printf("Invalid timestamp: %v", a)
return 0
}
bi, err := strconv.ParseInt(b, 10, 64)
if err != nil {
log.Printf("Invalid timestamp: %v", b)
return 0
}
if ai < bi {
return -1
} else if ai > bi {
return 1
} else {
return 0
}
}
}
func (d *DNSAggregator) Merge(other *DNSAggregator) {
for q, stat := range other.queries {
rec := d.queries[q]
if rec == nil {
d.queries[q] = stat
} else {
rec.count += stat.count
if timeCompare(stat.first, rec.first) < 0 {
rec.first = stat.first
}
if timeCompare(stat.last, rec.last) > 0 {
rec.last = stat.last
}
rec.ttl = stat.ttl
}
}
for q, stat := range other.values {
rec := d.values[q]
if rec == nil {
d.values[q] = stat
} else {
rec.count += stat.count
if timeCompare(stat.first, rec.first) < 0 {
rec.first = stat.first
}
if timeCompare(stat.last, rec.last) > 0 {
rec.last = stat.last
}
rec.ttl = stat.ttl
}
}
return
}
func aggregate(aggregator *DNSAggregator, fn string) error {
f, err := opendecompress.Open(fn)
if err != nil {
return err
}
defer f.Close()
br, err := NewBroReader(f)
if err != nil {
return err
}
for {
rec, err := br.Next()
if errors.Is(err, io.ErrUnexpectedEOF) {
log.Printf("Possible truncated file %s: %v", fn, err)
break
}
if err != nil {
return err
}
if rec == nil {
break
}
ts := rec.GetTimestamp("ts")
query := rec.GetString("query")
qtype_name := rec.GetString("qtype_name")
answers := rec.GetStringList("answers")
ttls := rec.GetStringList("TTLs")
if rec.Error() != nil {
if rec.IsMissingFieldError() {
log.Printf("Skipping record with missing fields: %s", rec)
aggregator.SkipRecord()
continue
} else {
return rec.Error()
}
}
dns_record := DNSRecord{
ts: ts,
query: query,
qtype: qtype_name,
answers: answers,
ttls: ttls,
}
aggregator.AddRecord(dns_record)
}
return nil
}
func (ar *aggregationResult) ShallowCopy() aggregationResult {
return aggregationResult{
Duration: ar.Duration,
TotalRecords: ar.TotalRecords,
SkippedRecords: ar.SkippedRecords,
TuplesLen: ar.TuplesLen,
IndividualLen: ar.IndividualLen,
}
}
type JSONTuple struct {
Query string `json:"query"`
Type string `json:"type"`
Answer string `json:"answer"`
TTL string `json:"ttl"`
Count uint `json:"count"`
First string `json:"first"`
Last string `json:"last"`
}
func (ar *aggregationResult) TupleJSONReader(reverseQuery bool) io.ReadCloser {
pr, pw := io.Pipe()
encoder := json.NewEncoder(pw)
go func() {
defer pw.Close()
var q string
for _, t := range ar.Tuples {
if reverseQuery {
q = Reverse(t.query)
} else {
q = t.query
}
v := JSONTuple{
Query: q,
Type: t.qtype,
Answer: t.answer,
TTL: t.ttl,
Count: t.count,
First: t.first,
Last: t.last,
}
err := encoder.Encode(v)
if err != nil {
pr.CloseWithError(err)
return
}
}
}()
return pr
}
type JSONIndividual struct {
Value string `json:"value"`
Which string `json:"which"`
Count uint `json:"count"`
First string `json:"first"`
Last string `json:"last"`
}
func (ar *aggregationResult) IndividualJSONReader(reverseQuery bool) io.ReadCloser {
pr, pw := io.Pipe()
encoder := json.NewEncoder(pw)
go func() {
defer pw.Close()
var q string
for _, t := range ar.Individual {
if t.which == "Q" && reverseQuery {
q = Reverse(t.value)
} else {
q = t.value
}
v := JSONIndividual{
Value: q,
Which: t.which,
Count: t.count,
First: t.first,
Last: t.last,
}
err := encoder.Encode(v)
if err != nil {
pr.CloseWithError(err)
return
}
}
}()
return pr
}
================================================
FILE: aggregate_test.go
================================================
package main
import (
"fmt"
"io/ioutil"
"os"
"sort"
"testing"
)
type ByValue []aggregatedIndividual
func (a ByValue) Len() int { return len(a) }
func (a ByValue) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a ByValue) Less(i, j int) bool { return a[i].value < a[j].value }
type ByTuple []aggregatedTuple
func (a ByTuple) Len() int { return len(a) }
func (a ByTuple) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a ByTuple) Less(i, j int) bool { return a[i].query+a[i].answer < a[j].query+a[j].answer }
func ExampleAggregate() {
ag := NewDNSAggregator()
ag.AddRecord(DNSRecord{
ts: "10",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
ag.AddRecord(DNSRecord{
ts: "20",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
res := ag.GetResult()
sort.Sort(ByTuple(res.Tuples))
sort.Sort(ByValue(res.Individual))
fmt.Printf("Tuples:\n")
for _, r := range res.Tuples {
fmt.Printf("%#v\n", r)
}
fmt.Printf("\nIndividual:\n")
for _, r := range res.Individual {
fmt.Printf("%#v\n", r)
}
// Output:
//Tuples:
//main.aggregatedTuple{uniqueTuple:main.uniqueTuple{query:"www.example.com", answer:"1.2.3.4", qtype:"A"}, queryStat:main.queryStat{count:0x2, first:"10", last:"20", ttl:"300"}}
//
//Individual:
//main.aggregatedIndividual{uniqueIndividual:main.uniqueIndividual{value:"1.2.3.4", which:"A"}, queryStat:main.queryStat{count:0x2, first:"10", last:"20", ttl:"300"}}
//main.aggregatedIndividual{uniqueIndividual:main.uniqueIndividual{value:"www.example.com", which:"Q"}, queryStat:main.queryStat{count:0x2, first:"10", last:"20", ttl:""}}
}
func ExampleAggregateMerge() {
ag := NewDNSAggregator()
ag.AddRecord(DNSRecord{
ts: "10",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
ag.AddRecord(DNSRecord{
ts: "200",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
ag2 := NewDNSAggregator()
ag2.AddRecord(DNSRecord{
ts: "30",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
ag2.AddRecord(DNSRecord{
ts: "30",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.5"},
ttls: []string{"300"},
})
ag2.AddRecord(DNSRecord{
ts: "40",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.5"},
ttls: []string{"300"},
})
ag.Merge(ag2)
res := ag.GetResult()
sort.Sort(ByTuple(res.Tuples))
sort.Sort(ByValue(res.Individual))
fmt.Printf("Tuples:\n")
for _, r := range res.Tuples {
fmt.Printf("%#v\n", r)
}
fmt.Printf("\nIndividual:\n")
for _, r := range res.Individual {
fmt.Printf("%#v\n", r)
}
// Output:
//Tuples:
//main.aggregatedTuple{uniqueTuple:main.uniqueTuple{query:"www.example.com", answer:"1.2.3.4", qtype:"A"}, queryStat:main.queryStat{count:0x3, first:"10", last:"200", ttl:"300"}}
//main.aggregatedTuple{uniqueTuple:main.uniqueTuple{query:"www.example.com", answer:"1.2.3.5", qtype:"A"}, queryStat:main.queryStat{count:0x2, first:"30", last:"40", ttl:"300"}}
//
//Individual:
//main.aggregatedIndividual{uniqueIndividual:main.uniqueIndividual{value:"1.2.3.4", which:"A"}, queryStat:main.queryStat{count:0x3, first:"10", last:"200", ttl:"300"}}
//main.aggregatedIndividual{uniqueIndividual:main.uniqueIndividual{value:"1.2.3.5", which:"A"}, queryStat:main.queryStat{count:0x2, first:"30", last:"40", ttl:"300"}}
//main.aggregatedIndividual{uniqueIndividual:main.uniqueIndividual{value:"www.example.com", which:"Q"}, queryStat:main.queryStat{count:0x5, first:"10", last:"200", ttl:""}}
}
func BenchmarkAggregate(b *testing.B) {
aggregator := NewDNSAggregator()
var total uint
for i := 0; i < b.N; i++ {
err := aggregate(aggregator, "test_data/dns_json.log")
if err != nil {
b.Fatal(err)
}
aggregated := aggregator.GetResult()
total += aggregated.TotalRecords
}
}
func ExampleResultTupleJSONReader() {
ag := NewDNSAggregator()
ag.AddRecord(DNSRecord{
ts: "10",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
ag.AddRecord(DNSRecord{
ts: "20",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.5"},
ttls: []string{"300"},
})
res := ag.GetResult()
sort.Sort(ByTuple(res.Tuples))
reader := res.TupleJSONReader(false)
body, err := ioutil.ReadAll(reader)
if err != nil {
fmt.Fprintf(os.Stderr, "%v", err)
}
fmt.Printf("%s", body)
// Output:
//{"query":"www.example.com","type":"A","answer":"1.2.3.4","ttl":"300","count":1,"first":"10","last":"10"}
//{"query":"www.example.com","type":"A","answer":"1.2.3.5","ttl":"300","count":1,"first":"20","last":"20"}
}
func ExampleResultIndividualJSONReader() {
ag := NewDNSAggregator()
ag.AddRecord(DNSRecord{
ts: "10",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.4"},
ttls: []string{"300"},
})
ag.AddRecord(DNSRecord{
ts: "20",
query: "www.example.com",
qtype: "A",
answers: []string{"1.2.3.5"},
ttls: []string{"300"},
})
res := ag.GetResult()
sort.Sort(ByValue(res.Individual))
reader := res.IndividualJSONReader(false)
body, err := ioutil.ReadAll(reader)
if err != nil {
fmt.Fprintf(os.Stderr, "%v", err)
}
fmt.Printf("%s", body)
// Output:
//{"value":"1.2.3.4","which":"A","count":1,"first":"10","last":"10"}
//{"value":"1.2.3.5","which":"A","count":1,"first":"20","last":"20"}
//{"value":"www.example.com","which":"Q","count":2,"first":"10","last":"20"}
}
================================================
FILE: dns-ans-query.bro
================================================
##! Add the peer to the connection logs.
module DNS;
export {
redef record DNS::Info += {
ans_query: vector of string &optional &log;
};
}
event dns_query_reply(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count)
{
if(!c?$dns)
return;
if(!c$dns?$ans_query)
c$dns$ans_query = vector();
c$dns$ans_query[|c$dns$ans_query|] = query;
}
================================================
FILE: go.mod
================================================
module github.com/JustinAzoff/zeek-pdns
go 1.16
require (
github.com/ClickHouse/clickhouse-go v1.4.3
github.com/JustinAzoff/go-opendecompress v0.0.0-20210404020920-ed787af23844
github.com/buger/jsonparser v1.1.1
github.com/gorilla/mux v1.7.3
github.com/jmoiron/sqlx v1.2.0
github.com/lib/pq v1.3.0
github.com/mattn/go-sqlite3 v2.0.2+incompatible
github.com/pkg/errors v0.9.1
github.com/spf13/cobra v0.0.5
github.com/spf13/viper v1.6.2
github.com/stretchr/testify v1.7.0
)
================================================
FILE: go.sum
================================================
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/ClickHouse/clickhouse-go v1.4.3 h1:iAFMa2UrQdR5bHJ2/yaSLffZkxpcOYQMCUuKeNXGdqc=
github.com/ClickHouse/clickhouse-go v1.4.3/go.mod h1:EaI/sW7Azgz9UATzd5ZdZHRUhHgv5+JMS9NSr2smCJI=
github.com/JustinAzoff/go-opendecompress v0.0.0-20210404020920-ed787af23844 h1:dozqygFyq+gPF1qHj5BXtUwap2ZWNYkXv5SOBn2re7o=
github.com/JustinAzoff/go-opendecompress v0.0.0-20210404020920-ed787af23844/go.mod h1:HhAjXnUSLm0cIlX5FnaucNpSOkh/u1CFjRpZsy5TLTA=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/bkaradzic/go-lz4 v1.0.0 h1:RXc4wYsyz985CkXXeX04y4VnZFGG8Rd43pRaHsOXAKk=
github.com/bkaradzic/go-lz4 v1.0.0/go.mod h1:0YdlkowM3VswSROI7qDxhRvJ3sLhlFrRRwjwegp5jy4=
github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58 h1:F1EaeKL/ta07PY/k9Os/UFtwERei2/XzGemhpGnBKNg=
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58/go.mod h1:EOBUe0h4xcZ5GoxqC5SDxFQ8gwyZPKQoEzownBlhI80=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-sql-driver/mysql v1.4.0 h1:7LxgVwFb2hIQtMm87NdgAVfXjnt4OePseqT1tKx+opk=
github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/mux v1.7.3 h1:gnP5JzjVOuiZD07fKKToCAOjS0yOpj/qPETTXCCS6hw=
github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jmoiron/sqlx v1.2.0 h1:41Ip0zITnmWNR/vHV+S4m+VoUivnWY5E4OJfLZjCJMA=
github.com/jmoiron/sqlx v1.2.0/go.mod h1:1FEQNm3xlJgrMD+FBdI9+xvCksHtbpVBBw5dYhBSsks=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.11.13 h1:eSvu8Tmq6j2psUJqJrLcWH6K3w5Dwc+qipbaA6eVEN4=
github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE=
github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mattn/go-sqlite3 v1.9.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/mattn/go-sqlite3 v2.0.2+incompatible h1:qzw9c2GNT8UFrgWNDhCTqRqYUSmu/Dav/9Z58LGpk7U=
github.com/mattn/go-sqlite3 v2.0.2+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pierrec/lz4 v2.0.5+incompatible h1:2xWsjqPFWcplujydGg4WmhC/6fZqK42wMM8aXeqhl0I=
github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s=
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
github.com/spf13/viper v1.6.2 h1:7aKfF+e8/k68gda3LOjo5RxiUqddoFxVq4BKBPrxk5E=
github.com/spf13/viper v1.6.2/go.mod h1:t3iDnF5Jlj76alVNuyFBk5oUMCvsrkbvZK0WQdfDi5k=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
google.golang.org/appengine v1.1.0 h1:igQkv0AAhEIvTEpD5LIpAfav2eeVO9HBTjvKHVJPRSs=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
================================================
FILE: index.go
================================================
package main
import (
"fmt"
"log"
)
func index(store Store, filenames []string) error {
var didWork bool
store.Begin()
aggregator := NewDNSAggregator()
var emptyStoreResult UpdateResult
aggMap := make(map[string]aggregationResult)
for _, fn := range filenames {
indexed, err := store.IsLogIndexed(fn)
if err != nil {
return fmt.Errorf("store.IsLogIndexed: %w", err)
}
if indexed {
log.Printf("%s: Already indexed", fn)
continue
}
fileAgg := NewDNSAggregator()
err = aggregate(fileAgg, fn)
if err != nil {
return fmt.Errorf("Error Aggregating %s: %w", fn, err)
}
aggregator.Merge(fileAgg)
aggregated := fileAgg.GetResult()
log.Printf("%s: Aggregation: Duration=%0.1f TotalRecords=%d SkippedRecords=%d Tuples=%d Individual=%d",
fn,
aggregated.Duration.Seconds(),
aggregated.TotalRecords,
aggregated.SkippedRecords,
aggregated.TuplesLen,
aggregated.IndividualLen,
)
aggMap[fn] = aggregated.ShallowCopy()
didWork = true
}
if !didWork {
return nil
//TODO: rollback transaction
}
aggregated := aggregator.GetResult()
result, err := store.Update(aggregated)
if err != nil {
return fmt.Errorf("store.Update: %w", err)
}
log.Printf("batch: Store: Duration=%0.1f Inserted=%d Updated=%d", result.Duration.Seconds(), result.Inserted, result.Updated)
for fn, aggregated := range aggMap {
err = store.SetLogIndexed(fn, aggregated, emptyStoreResult)
if err != nil {
return fmt.Errorf("store.SetLogIndexed: %w", err)
}
}
err = store.Commit()
if err != nil {
return fmt.Errorf("store.Commit: %w", err)
}
return nil
}
================================================
FILE: local_test.txt
================================================
docker run --rm -t -i -p 5432:5432 -e POSTGRES_DB=pdns_test -e POSTGRES_PASSWORD=pdns postgres
PG_TEST_URL="postgres://postgres:password@$(docker-machine ip)/pdns_test?sslmode=disable" go test -v
================================================
FILE: main.go
================================================
package main
import (
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func getStore() Store {
storeType := viper.GetString("store.type")
storeUri := viper.GetString("store.uri")
mystore, err := NewStore(storeType, storeUri)
if err != nil {
log.Fatal(err)
}
return mystore
}
var RootCmd = &cobra.Command{
Use: "zeek-pdns",
Short: "Passive DNS Collection for BRO",
Run: nil,
}
var IndexCmd = &cobra.Command{
Use: "index",
Short: "Index one or more dns log files",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
err := index(mystore, args)
if err != nil {
log.Fatal(err)
}
},
}
var FindCmd = &cobra.Command{
Use: "find",
Short: "find records",
Run: nil,
}
var FindTupleCmd = &cobra.Command{
Use: "tuples",
Short: "find dns tuples",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
for _, value := range args {
recs, err := mystore.FindTuples(value)
if err != nil {
log.Fatal(err)
}
recs.Display()
}
},
}
var FindIndividualCmd = &cobra.Command{
Use: "individual",
Short: "find an individual dns value",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
for _, value := range args {
recs, err := mystore.FindIndividual(value)
if err != nil {
log.Fatal(err)
}
recs.Display()
}
},
}
var LikeCmd = &cobra.Command{
Use: "like",
Short: "find records like something",
Run: nil,
}
var LikeTupleCmd = &cobra.Command{
Use: "tuples",
Short: "find like dns tuples",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
for _, value := range args {
recs, err := mystore.LikeTuples(value)
if err != nil {
log.Fatal(err)
}
recs.Display()
}
},
}
var LikeIndividualCmd = &cobra.Command{
Use: "individual",
Short: "find like individual dns values",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
for _, value := range args {
recs, err := mystore.LikeIndividual(value)
if err != nil {
log.Fatal(err)
}
recs.Display()
}
},
}
var DeleteOldCmd = &cobra.Command{
Use: "delete-old",
Short: "delete old records",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
days := viper.GetInt("deleteold.days")
rows, err := mystore.DeleteOld(int64(days))
if err != nil {
log.Fatal(err)
}
log.Printf("Deleted %d records", rows)
},
}
var WebCmd = &cobra.Command{
Use: "web",
Short: "start http API",
Run: func(cmd *cobra.Command, args []string) {
mystore := getStore()
bind := viper.GetString("http.listen")
startWeb(mystore, bind)
},
}
var VersionCmd = &cobra.Command{
Use: "version",
Short: "Output version number",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println(VERSION)
},
}
func init() {
RootCmd.AddCommand(IndexCmd)
RootCmd.AddCommand(FindCmd)
FindCmd.AddCommand(FindIndividualCmd)
FindCmd.AddCommand(FindTupleCmd)
RootCmd.AddCommand(LikeCmd)
LikeCmd.AddCommand(LikeIndividualCmd)
LikeCmd.AddCommand(LikeTupleCmd)
DeleteOldCmd.Flags().Int64("days", 365, "Age in days of records to be deleted")
viper.BindPFlag("deleteold.days", DeleteOldCmd.Flags().Lookup("days"))
viper.BindEnv("deleteold.days", "PDNS_DELETE_OLD_DAYS")
RootCmd.AddCommand(DeleteOldCmd)
WebCmd.Flags().String("listen", ":8080", "Address to listen on")
viper.BindPFlag("http.listen", WebCmd.Flags().Lookup("listen"))
viper.BindEnv("http.listen", "PDNS_HTTP_LISTEN")
RootCmd.AddCommand(WebCmd)
RootCmd.AddCommand(VersionCmd)
RootCmd.PersistentFlags().String("store", "sqlite", "Backend data store")
viper.BindPFlag("store.type", RootCmd.PersistentFlags().Lookup("store"))
viper.BindEnv("store.type", "PDNS_STORE_TYPE")
RootCmd.PersistentFlags().String("uri", "db.sqlite", "Backend data store URI")
viper.BindPFlag("store.uri", RootCmd.PersistentFlags().Lookup("uri"))
viper.BindEnv("store.uri", "PDNS_STORE_URI")
viper.AutomaticEnv()
}
func main() {
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(-1)
}
}
================================================
FILE: read.go
================================================
package main
import (
"bufio"
"fmt"
"io"
)
type Reader interface {
Next() (Record, error)
}
type Record interface {
String() string
GetString(string) string
GetTimestamp(string) string
GetStringList(string) []string
GetFloat(string) float64
Error() error
IsMissingFieldError() bool
}
func NewBroReader(r io.Reader) (Reader, error) {
wrapped := bufio.NewReader(r)
first_byte, err := wrapped.Peek(1)
if err != nil {
return nil, err
}
switch first_byte[0] {
case '#':
return NewBroAsciiReader(wrapped), nil
case '{':
return NewBroJSONReader(wrapped), nil
default:
return nil, fmt.Errorf("Unable to determine file type, first byte was %q", first_byte)
}
}
================================================
FILE: read_ascii.go
================================================
package main
import (
"bufio"
"encoding/hex"
"fmt"
"io"
"log"
"strconv"
"strings"
)
func grab_value(line string) string {
val := strings.Split(line, " ")[1]
return val
}
func extract_sep(line string) string {
sep := grab_value(line)
sepchar, err := hex.DecodeString(sep[2:])
if err != nil {
log.Panic(err)
}
return string(sepchar)
}
type BroAsciiReader struct {
r io.Reader
br *bufio.Reader
sep string
fields []string
fieldsMap map[string]int
types []string
timeFields map[int]bool
newHeaders bool
}
type ASCIIRecord struct {
line *string
cols *[]string
fields *map[string]int
err error
}
func (r *ASCIIRecord) String() string {
return *r.line
}
func (r *ASCIIRecord) GetString(field string) string {
idx, ok := (*r.fields)[field]
if !ok {
r.err = fmt.Errorf("Invalid field %s", field)
return ""
}
return (*r.cols)[idx]
}
func (r *ASCIIRecord) GetTimestamp(field string) string {
return r.GetString(field)
}
func (r *ASCIIRecord) GetStringList(field string) []string {
raw := r.GetString(field)
spl := strings.Split(raw, ",")
return spl
}
func (r *ASCIIRecord) GetStringByIndex(index int) string {
return (*r.cols)[index]
}
func (r *ASCIIRecord) GetFloat(field string) float64 {
idx, ok := (*r.fields)[field]
if !ok {
r.err = fmt.Errorf("Invalid field %s", field)
return 0.0
}
val := (*r.cols)[idx]
fl, err := strconv.ParseFloat(val, 64)
if err != nil {
panic(err)
}
return fl
}
func (r *ASCIIRecord) GetFloatByIndex(index int) float64 {
val := (*r.cols)[index]
fl, err := strconv.ParseFloat(val, 64)
if err != nil {
panic(err)
}
return fl
}
func (r *ASCIIRecord) IsMissingFieldError() bool {
//TODO: handle here or jsut skip in Next?
return false
}
func (r *ASCIIRecord) Error() error {
if r.err != nil {
return fmt.Errorf("Error parsing %s: %w", r, r.err)
}
return nil
}
func (r *ASCIIRecord) GetFieldIndex(field string) int {
idx, ok := (*r.fields)[field]
if ok {
return idx
}
r.err = fmt.Errorf("Invalid field %s", field)
return -1
}
func NewBroAsciiReader(r io.Reader) *BroAsciiReader {
br := bufio.NewReader(r)
tf := make(map[int]bool)
return &BroAsciiReader{r: r, br: br, timeFields: tf}
}
func (b *BroAsciiReader) Next() (Record, error) {
line, err := b.br.ReadString('\n')
if err == io.EOF {
return nil, nil
}
if err != nil {
return nil, err
}
line = strings.Trim(line, "\n")
if strings.HasPrefix(line, "#") {
b.handleHeader(line)
return b.Next()
}
parts := strings.Split(line, "\t")
rec := ASCIIRecord{
line: &line,
cols: &parts,
fields: &b.fieldsMap,
}
return &rec, nil
}
func (b *BroAsciiReader) handleHeader(line string) error {
b.newHeaders = true
if strings.HasPrefix(line, "#separator") {
b.sep = extract_sep(line)
} else if strings.HasPrefix(line, "#fields") {
b.fields = strings.Split(line, "\t")[1:]
b.fieldsMap = make(map[string]int)
for idx, f := range b.fields {
b.fieldsMap[f] = idx
}
} else if strings.HasPrefix(line, "#types") {
b.types = strings.Split(line, "\t")[1:]
for idx, typ := range b.types {
if typ == "time" {
b.timeFields[idx] = true
}
}
}
return nil
}
func (b *BroAsciiReader) HeadersChanged() bool {
return b.newHeaders
}
func (b *BroAsciiReader) HandledHeaders() {
b.newHeaders = false
}
================================================
FILE: read_json.go
================================================
package main
import (
"bufio"
"fmt"
"io"
"strings"
"github.com/buger/jsonparser"
)
type BroJSONReader struct {
r *bufio.Reader
}
type JSONRecord struct {
line []byte
err error
}
func (r *JSONRecord) String() string {
return strings.Trim(string(r.line), "\n")
}
func (r *JSONRecord) GetString(field string) string {
val, err := jsonparser.GetString(r.line, field)
r.err = err
return val
}
func (r *JSONRecord) GetStringList(field string) []string {
var strings []string
jsonparser.ArrayEach(r.line, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
strings = append(strings, string(value))
r.err = err
}, field)
return strings
}
func (r *JSONRecord) GetFloat(field string) float64 {
val, err := jsonparser.GetFloat(r.line, field)
r.err = err
return val
}
func (r *JSONRecord) IsMissingFieldError() bool {
return r.err == jsonparser.KeyPathNotFoundError
}
func (r *JSONRecord) Error() error {
if r.err != nil {
return fmt.Errorf("Error parsing %s: %w", r, r.err)
}
return nil
}
func NewBroJSONReader(r *bufio.Reader) *BroJSONReader {
return &BroJSONReader{r: r}
}
func (b *BroJSONReader) Next() (Record, error) {
line, err := b.r.ReadBytes('\n')
if err == io.EOF {
return nil, nil
}
if err != nil {
return nil, err
}
rec := JSONRecord{
line: line,
}
return &rec, nil
}
//GetTimestamp tries to get a field that may be a String or a float64
//Yes, this is terrible, but supporting both timestamps and iso8601 at the same
//time is tricky
//TODO: use a struct type that is a string or float and can be lazy evaluated
//for sorting and for final representation when being upserted
func (r *JSONRecord) GetTimestamp(field string) string {
val, err := jsonparser.GetString(r.line, field)
if err == nil {
return val
}
fval, err := jsonparser.GetFloat(r.line, field)
r.err = err
return fmt.Sprintf("%f", fval)
}
================================================
FILE: read_test.go
================================================
package main
import (
"testing"
)
func BenchmarkReadASCII(b *testing.B) {
for i := 0; i < b.N; i++ {
fn := "test_data/reddit_dns_2016-04-01.log"
aggregator := NewDNSAggregator()
aggregate(aggregator, fn)
}
}
func BenchmarkReadJSON(b *testing.B) {
for i := 0; i < b.N; i++ {
fn := "test_data/dns_json.log"
aggregator := NewDNSAggregator()
aggregate(aggregator, fn)
}
}
================================================
FILE: store.go
================================================
package main
import (
"errors"
"fmt"
"strings"
"time"
"log"
)
type Store interface {
Init() error
Clear() error
Begin() error
Commit() error
IsLogIndexed(filename string) (bool, error)
SetLogIndexed(filename string, ar aggregationResult, ur UpdateResult) error
Update(aggregationResult) (UpdateResult, error)
FindQueryTuples(query string) (tupleResults, error)
FindTuples(query string) (tupleResults, error)
FindIndividual(value string) (individualResults, error)
LikeTuples(query string) (tupleResults, error)
LikeIndividual(value string) (individualResults, error)
DeleteOld(days int64) (int64, error)
Close() error
}
type tupleResult struct {
Query string
Type string
Answer string
Count uint
TTL uint
First string
Last string
}
type tupleResults []tupleResult
func (tr tupleResults) Display() {
if len(tr) == 0 {
return
}
header := []string{"Query", "Type", "Answer", "Count", "TTL", "First", "Last"}
fmt.Println(strings.Join(header, "\t"))
for _, rec := range tr {
fmt.Println(rec)
}
}
func (tr tupleResult) String() string {
count := fmt.Sprintf("%d", tr.Count)
ttl := fmt.Sprintf("%d", tr.TTL)
s := []string{tr.Query, tr.Type, tr.Answer, count, ttl, tr.First, tr.Last}
return strings.Join(s, "\t")
}
type individualResult struct {
Value string
Which string
Count uint
First string
Last string
}
type individualResults []individualResult
func (ir individualResults) Display() {
if len(ir) == 0 {
return
}
header := []string{"Value", "Which", "Count", "First", "Last"}
fmt.Println(strings.Join(header, "\t"))
for _, rec := range ir {
fmt.Println(rec)
}
}
func (ir individualResult) String() string {
count := fmt.Sprintf("%d", ir.Count)
s := []string{ir.Value, ir.Which, count, ir.First, ir.Last}
return strings.Join(s, "\t")
}
type UpdateResult struct {
Inserted uint
Updated uint
Duration time.Duration
}
var storeFactories = map[string]func(string) (Store, error){
"clickhouse": NewCHStore,
"sqlite": NewSQLiteStore,
"postgresql": NewPGStore,
}
func NewStore(storeType string, filename string) (Store, error) {
storeFactory, ok := storeFactories[storeType]
if !ok {
return nil, errors.New("Invalid store type")
}
s, err := storeFactory(filename)
if err != nil {
return nil, err
}
err = s.Init()
if err != nil {
return nil, err
}
return s, err
}
//ToTS ensures a string is a unix timestamp
//Yes, this is terrible, but supporting both timestamps and iso8601 at the same
//time is tricky
func ToTS(t string) string {
//If it doesn't have a dash, it should be a unix timestamp already
if !strings.Contains(t, "-") {
return t
}
parsed, err := time.Parse(time.RFC3339, t)
if err != nil {
log.Fatalf("Unparsable timestamp, don't know what to do here: %v", t)
}
return fmt.Sprintf("%d", parsed.Unix())
}
================================================
FILE: store_clickhouse.go
================================================
package main
import (
"database/sql"
"fmt"
"net/url"
"time"
_ "github.com/ClickHouse/clickhouse-go"
"github.com/jmoiron/sqlx"
)
var chschema = []string{
`
CREATE TABLE IF NOT EXISTS tuples (
whatever Date DEFAULT '2000-01-01',
query String,
type String,
answer String,
ttl AggregateFunction(anyLast, UInt16),
first AggregateFunction(min, DateTime),
last AggregateFunction(max, DateTime),
count AggregateFunction(sum, UInt64)
) ENGINE = AggregatingMergeTree(whatever, (query, type, answer), 8192);
`,
`
CREATE TABLE IF NOT EXISTS individual (
whatever Date DEFAULT '2000-01-01',
which Enum8('Q'=0, 'A'=1),
value String,
first AggregateFunction(min, DateTime),
last AggregateFunction(max, DateTime),
count AggregateFunction(sum, UInt64)
) ENGINE = AggregatingMergeTree(whatever, (which, value), 8192);
`,
`
CREATE TABLE IF NOT EXISTS filenames (
day Date DEFAULT toDate(ts),
ts DateTime DEFAULT now(),
filename String,
aggregation_time Float64,
total_records UInt64,
skipped_records UInt64,
tuples UInt64,
individual UInt64,
store_time Float64,
inserted UInt64,
updated UInt64
) ENGINE = MergeTree(day, (filename), 8192);
`}
const tuples_temp_stmt = `
CREATE TEMPORARY TABLE tuples_temp (
query String,
type String,
answer String,
ttl String,
first String,
last String,
count UInt64
) ENGINE = Memory`
const individual_temp_stmt = `
CREATE TEMPORARY TABLE individual_temp (
which Enum8('Q'=0, 'A'=1),
value String,
first String,
last String,
count UInt64
) ENGINE = Memory`
type CHStore struct {
conn *sqlx.DB
}
func NewCHStore(uri string) (Store, error) {
_, err := url.Parse(uri)
if err != nil {
return nil, err
}
conn, err := sqlx.Open("clickhouse", uri)
if err != nil {
return nil, err
}
err = conn.Ping()
if err != nil {
return nil, err
}
return &CHStore{
conn: conn,
}, nil
}
func (s *CHStore) Close() error {
return s.Close()
}
func (s *CHStore) Exec(stmt string) error {
_, err := s.conn.Exec(stmt)
return err
}
func (s *CHStore) Init() error {
for _, stmt := range chschema {
err := s.Exec(stmt)
if err != nil {
return err
}
}
return nil
}
func (s *CHStore) Clear() error {
stmts := []string{
"drop table filenames",
"drop table individual",
"drop table tuples",
}
for _, stmt := range stmts {
err := s.Exec(stmt)
if err != nil {
return err
}
}
return nil
}
func (s *CHStore) Begin() error {
return fmt.Errorf("clickhouse doesn't support transactions")
}
func (s *CHStore) Commit() error {
//log.Printf("clickhouse doesn't support transactions")
return nil
}
//DeleteOld Deletes records that haven't been seen in DAYS, returns the total records deleted
func (s *CHStore) DeleteOld(days int64) (int64, error) {
return 0, fmt.Errorf("clickhouse doesn't support delete")
}
func (s *CHStore) Update(ar aggregationResult) (UpdateResult, error) {
var result UpdateResult
var err error
start := time.Now()
s.Exec("DROP TABLE tuples_temp")
s.Exec("DROP TABLE individual_temp")
err = s.Exec(tuples_temp_stmt)
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
err = s.Exec(individual_temp_stmt)
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
tx, err := s.conn.Begin()
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
stmt, err := tx.Prepare(`INSERT INTO tuples_temp
(query, type, answer, ttl, first, last, count)
values (?,?,?,?,?,?,?)`,
)
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
// Ok, now let's update stuff
// tuples
for _, q := range ar.Tuples {
//Update the tuples table
query := Reverse(q.query)
_, err := stmt.Exec(query, q.qtype, q.answer, q.ttl, ToTS(q.first), ToTS(q.last), uint64(q.count))
if err != nil {
return result, fmt.Errorf("CHStore.Update failed to run query: %w", err)
}
}
err = tx.Commit()
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
err = s.Exec(`INSERT INTO tuples (query, type, answer, ttl, first, last, count) SELECT
query, type, answer,
anyLastState(toUInt16(ttl)),
minState(toDateTime(toFloat64(first))),
maxState(toDateTime(toFloat64(last))),
sumState(count) from tuples_temp group by query, type, answer`,
)
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
tx, err = s.conn.Begin()
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
// Individuals
stmt, err = tx.Prepare(`INSERT INTO individual_temp
(value, which, first, last, count)
values (?,?,?,?,?)`,
)
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
for _, q := range ar.Individual {
//Update the tuples table
value := q.value
if q.which == "Q" {
value = Reverse(value)
}
_, err := stmt.Exec(value, q.which, ToTS(q.first), ToTS(q.last), uint64(q.count))
if err != nil {
return result, fmt.Errorf("CHStore.Update failed to run query: %w", err)
}
}
err = tx.Commit()
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
err = s.Exec(`INSERT INTO individual (which, value, first, last, count) SELECT which, value,
minState(toDateTime(toFloat64(first))),
maxState(toDateTime(toFloat64(last))),
sumState(count) from individual_temp group by which, value`)
if err != nil {
return result, fmt.Errorf("CHStore.Update failed: %w", err)
}
result.Updated = uint(ar.TuplesLen + ar.IndividualLen)
result.Duration = time.Since(start)
return result, nil
}
func (s *CHStore) IsLogIndexed(filename string) (bool, error) {
var fn string
err := s.conn.QueryRow("SELECT filename FROM filenames WHERE filename=?", filename).Scan(&fn)
switch {
case err == sql.ErrNoRows:
return false, nil
case err != nil:
return false, err
default:
return true, nil
}
}
func (s *CHStore) SetLogIndexed(filename string, ar aggregationResult, ur UpdateResult) error {
tx, _ := s.conn.Begin()
q := `INSERT INTO filenames (filename,
aggregation_time, total_records, skipped_records, tuples, individual,
store_time, inserted, updated)
VALUES (?,?,?,?,?,?,?,?,?)`
_, err := tx.Exec(q, filename,
ar.Duration.Seconds(), uint64(ar.TotalRecords), uint64(ar.SkippedRecords), len(ar.Tuples), len(ar.Individual),
ur.Duration.Seconds(), uint64(ur.Inserted), uint64(ur.Updated))
if err != nil {
return err
}
return tx.Commit()
}
func (s *CHStore) FindQueryTuples(query string) (tupleResults, error) {
tr := []tupleResult{}
query = Reverse(query)
err := s.conn.Select(&tr, "SELECT * FROM tuples WHERE query = ?", query)
reverseQuery(tr)
return tr, err
}
func (s *CHStore) FindTuples(query string) (tupleResults, error) {
tr := []tupleResult{}
rquery := Reverse(query)
err := s.conn.Select(&tr, "SELECT query, type, answer, anyLastMerge(ttl) as ttl, minMerge(first) as first, maxMerge(last) as last, sumMerge(count) as count from tuples WHERE query = ? OR answer = ? group by query, type, answer ORDER BY query, answer", rquery, query)
reverseQuery(tr)
return tr, err
}
func (s *CHStore) LikeTuples(query string) (tupleResults, error) {
tr := []tupleResult{}
rquery := Reverse(query)
err := s.conn.Select(&tr, "SELECT query, type, answer, anyLastMerge(ttl) as ttl, minMerge(first) as first, maxMerge(last) as last, sumMerge(count) as count from tuples WHERE query like ? OR answer like ? group by query, type, answer ORDER BY query, answer", rquery+"%", query+"%")
reverseQuery(tr)
return tr, err
}
func (s *CHStore) FindIndividual(value string) (individualResults, error) {
rvalue := Reverse(value)
tr := []individualResult{}
err := s.conn.Select(&tr, `SELECT which, value, minMerge(first) as first, maxMerge(last) as last, sumMerge(count) as count from individual WHERE (which='A' AND value = ?) OR (which='Q' AND value = ?) group by which, value ORDER BY value`, value, rvalue)
reverseValue(tr)
return tr, err
}
func (s *CHStore) LikeIndividual(value string) (individualResults, error) {
rvalue := Reverse(value)
tr := []individualResult{}
err := s.conn.Select(&tr, `SELECT which, value, minMerge(first) as first, maxMerge(last) as last, sumMerge(count) as count from individual WHERE (which='A' AND value like ?) OR (which='Q' AND value like ?) group by which, value ORDER BY value`, value+"%", rvalue+"%")
reverseValue(tr)
return tr, err
}
================================================
FILE: store_pg.go
================================================
package main
import (
"database/sql"
"fmt"
"log"
"strings"
"time"
"github.com/jmoiron/sqlx"
"github.com/lib/pq"
)
const pgschema = `
set synchronous_commit to off;
CREATE TABLE IF NOT EXISTS tuples (
query text,
type text,
answer text,
count bigint,
ttl integer,
first timestamp,
last timestamp,
PRIMARY KEY (query, type, answer)
) ;
CREATE INDEX tuples_query ON tuples(query varchar_pattern_ops);
CREATE INDEX tuples_answer ON tuples(answer varchar_pattern_ops);
-- CREATE INDEX tuples_first ON tuples(first);
-- CREATE INDEX tuples_last ON tuples(last);
CREATE TABLE IF NOT EXISTS individual (
which char(1),
value text,
count bigint,
first timestamp,
last timestamp,
PRIMARY KEY (which, value)
);
CREATE INDEX individual_value ON individual(value varchar_pattern_ops);
-- CREATE INDEX individual_first ON individual(first);
-- CREATE INDEX individual_last ON individual(last);
CREATE TABLE IF NOT EXISTS filenames (
filename text PRIMARY KEY UNIQUE NOT NULL,
time timestamp DEFAULT now(),
aggregation_time real,
total_records int,
skipped_records int,
tuples int,
individual int,
store_time real,
inserted int,
updated int
);
CREATE OR REPLACE FUNCTION update_individual(w char(1), v text, c integer,f timestamp,l timestamp) RETURNS CHAR(1) AS
$$
BEGIN
LOOP
-- first try to update the key
UPDATE individual SET count=count+c,
first=least(f, first),
last =greatest(l, last)
WHERE value=v AND which=w;
IF found THEN
RETURN 'U';
END IF;
-- not there, so try to insert the key
-- if someone else inserts the same key concurrently,
-- we could get a unique-key failure
BEGIN
INSERT INTO individual (value, which, count, first, last) VALUES (v,w,c,f,l);
RETURN 'I';
EXCEPTION WHEN unique_violation THEN
-- do nothing, and loop to try the UPDATE again
END;
END LOOP;
END;
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION update_tuples(q text, ty text, a text, tt integer, c integer ,f timestamp,l timestamp) RETURNS CHAR(1) AS
$$
BEGIN
LOOP
-- first try to update the key
UPDATE tuples SET count=count+c,
ttl=tt,
first=least(f, first),
last =greatest(l, last)
WHERE query=q AND type=ty AND answer=a;
IF found THEN
RETURN 'U';
END IF;
-- not there, so try to insert the key
-- if someone else inserts the same key concurrently,
-- we could get a unique-key failure
BEGIN
INSERT INTO tuples (query, type, answer, ttl, count, first, last) VALUES (q, ty, a, tt, c, f, l);
RETURN 'I';
EXCEPTION WHEN unique_violation THEN
-- do nothing, and loop to try the UPDATE again
END;
END LOOP;
END;
$$
LANGUAGE plpgsql;
`
type PGStore struct {
conn *sqlx.DB
*SQLCommonStore
}
func NewPGStore(uri string) (Store, error) {
conn, err := sqlx.Open("postgres", uri)
if err != nil {
return nil, err
}
common := &SQLCommonStore{conn: conn}
return &PGStore{conn: conn, SQLCommonStore: common}, nil
}
func (s *PGStore) Close() error {
return s.Close()
}
func (s *PGStore) Init() error {
_, err := s.conn.Exec(pgschema)
// Ignore a duplicte table error message
if pqerr, ok := err.(*pq.Error); ok {
if pqerr.Code == "42P07" {
return nil
}
}
return err
}
func genFullBatchSelect(tmpl string, batchSize int) string {
var queries []string
numParams := strings.Count(tmpl, "$")
arg := 1
for i := 0; i < batchSize; i++ {
var args []interface{}
for p := 0; p < numParams; p++ {
args = append(args, arg)
arg++
}
queries = append(queries, fmt.Sprintf(tmpl, args...))
}
fullq := fmt.Sprintf("SELECT %s", strings.Join(queries, " || "))
return fullq
}
var BATCHSIZE = 200
func (s *PGStore) Update(ar aggregationResult) (UpdateResult, error) {
var result UpdateResult
start := time.Now()
tx, err := s.BeginTx()
if err != nil {
return result, err
}
//Setup the 2 different prepared statements
updateTupleTmpl := "update_tuples($%d, $%d, $%d, $%d, $%d, to_timestamp($%d)::timestamp, to_timestamp($%d)::timestamp)"
updateTupleBatch, err := tx.Prepare(genFullBatchSelect(updateTupleTmpl, BATCHSIZE))
if err != nil {
return result, err
}
defer updateTupleBatch.Close()
updateIndividualTmpl := "update_individual($%d, $%d, $%d, to_timestamp($%d)::timestamp, to_timestamp($%d)::timestamp)"
updateIndividualeBatch, err := tx.Prepare(genFullBatchSelect(updateIndividualTmpl, BATCHSIZE))
if err != nil {
return result, err
}
defer updateIndividualeBatch.Close()
var arguments []interface{}
batchCounter := 0
runBatch := func(tmpl string, preparedBatch *sql.Stmt, arguments []interface{}, batchSize int) {
if batchSize == 0 {
return
}
var stmt *sql.Stmt
if batchSize == BATCHSIZE {
stmt = preparedBatch
} else {
stmt, err = tx.Prepare(genFullBatchSelect(tmpl, batchSize))
defer stmt.Close()
}
res, err := stmt.Query(arguments...)
//log.Printf("Fullq is: %s", fullq)
//log.Printf("Arguments is: %#v", arguments)
if err != nil {
log.Fatal(err)
}
res.Next()
var update_result string
res.Scan(&update_result)
res.Close()
for _, ch := range update_result {
if ch == 'I' {
result.Inserted++
} else {
result.Updated++
}
}
}
// Ok, now let's update stuff
for _, q := range ar.Tuples {
//Update the tuples table
query := Reverse(q.query)
arguments = append(arguments, query, q.qtype, q.answer, q.ttl, q.count, ToTS(q.first), ToTS(q.last))
batchCounter++
if batchCounter == BATCHSIZE {
runBatch(updateTupleTmpl, updateTupleBatch, arguments, batchCounter)
arguments = arguments[:0]
batchCounter = 0
}
}
runBatch(updateTupleTmpl, updateTupleBatch, arguments, batchCounter)
arguments = arguments[:0]
batchCounter = 0
for _, q := range ar.Individual {
value := q.value
if q.which == "Q" {
value = Reverse(value)
}
arguments = append(arguments, q.which, value, q.count, ToTS(q.first), ToTS(q.last))
batchCounter++
if batchCounter == BATCHSIZE {
runBatch(updateIndividualTmpl, updateIndividualeBatch, arguments, batchCounter)
arguments = arguments[:0]
batchCounter = 0
}
}
runBatch(updateIndividualTmpl, updateIndividualeBatch, arguments, batchCounter)
result.Duration = time.Since(start)
return result, s.Commit()
}
================================================
FILE: store_sql_common.go
================================================
package main
import (
"database/sql"
"errors"
"time"
"github.com/jmoiron/sqlx"
)
func Reverse(s string) string {
runes := []rune(s)
for i, j := 0, len(runes)-1; i < j; i, j = i+1, j-1 {
runes[i], runes[j] = runes[j], runes[i]
}
return string(runes)
}
type SQLCommonStore struct {
conn *sqlx.DB
tx *sql.Tx
txDepth int
}
func (s *SQLCommonStore) Clear() error {
_, err := s.conn.Exec("DELETE FROM filenames;DELETE FROM individual;DELETE FROM tuples;")
return err
}
func (s *SQLCommonStore) Begin() error {
_, err := s.BeginTx()
return err
}
func (s *SQLCommonStore) BeginTx() (*sql.Tx, error) {
if s.tx != nil {
s.txDepth += 1
//log.Printf("Returning existing transaction: depth=%d\n", s.txDepth)
return s.tx, nil
}
//log.Printf("new transaction\n")
tx, err := s.conn.Begin()
if err != nil {
return tx, err
}
s.tx = tx
s.txDepth += 1
return s.tx, nil
}
func (s *SQLCommonStore) Commit() error {
if s.tx == nil {
return errors.New("Commit outside of transaction")
}
s.txDepth -= 1
if s.txDepth > 0 {
//log.Printf("Not commiting stacked transaction: depth=%d\n", s.txDepth)
return nil // No OP
}
//log.Printf("Commiting transaction: depth=%d\n", s.txDepth)
err := s.tx.Commit()
s.tx = nil
return err
}
func (s *SQLCommonStore) IsLogIndexed(filename string) (bool, error) {
tx, err := s.BeginTx()
if err != nil {
return false, err
}
defer s.Commit()
var fn string
err = tx.QueryRow("SELECT filename FROM filenames WHERE filename=$1", filename).Scan(&fn)
switch {
case err == sql.ErrNoRows:
return false, nil
case err != nil:
return false, err
default:
return true, nil
}
}
func (s *SQLCommonStore) SetLogIndexed(filename string, ar aggregationResult, ur UpdateResult) error {
tx, err := s.BeginTx()
defer s.Commit()
if err != nil {
return err
}
q := `INSERT INTO filenames (filename,
aggregation_time, total_records, skipped_records, tuples, individual,
store_time, inserted, updated)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)`
_, err = tx.Exec(q, filename,
ar.Duration.Seconds(), ar.TotalRecords, ar.SkippedRecords, ar.TuplesLen, ar.IndividualLen,
ur.Duration.Seconds(), ur.Inserted, ur.Updated)
return err
}
func reverseQuery(tr tupleResults) {
for idx, rec := range tr {
rec.Query = Reverse(rec.Query)
tr[idx] = rec
}
}
func reverseValue(tr individualResults) {
for idx, rec := range tr {
if rec.Which == "Q" {
rec.Value = Reverse(rec.Value)
tr[idx] = rec
}
}
}
func (s *SQLCommonStore) FindQueryTuples(query string) (tupleResults, error) {
tr := []tupleResult{}
query = Reverse(query)
err := s.conn.Select(&tr, "SELECT * FROM tuples WHERE query = $1", query)
reverseQuery(tr)
return tr, err
}
func (s *SQLCommonStore) FindTuples(query string) (tupleResults, error) {
tr := []tupleResult{}
rquery := Reverse(query)
err := s.conn.Select(&tr, "SELECT * FROM tuples WHERE query = $1 OR answer = $2 ORDER BY query, answer", rquery, query)
reverseQuery(tr)
return tr, err
}
func (s *SQLCommonStore) LikeTuples(query string) (tupleResults, error) {
tr := []tupleResult{}
rquery := Reverse(query)
err := s.conn.Select(&tr, "SELECT * FROM tuples WHERE query like $1 OR answer like $2 ORDER BY query, answer", rquery+"%", query+"%")
reverseQuery(tr)
return tr, err
}
func (s *SQLCommonStore) FindIndividual(value string) (individualResults, error) {
rvalue := Reverse(value)
tr := []individualResult{}
err := s.conn.Select(&tr, "SELECT * FROM individual WHERE (which='A' AND value = $1) OR (which='Q' AND value = $2) ORDER BY value", value, rvalue)
reverseValue(tr)
return tr, err
}
func (s *SQLCommonStore) LikeIndividual(value string) (individualResults, error) {
rvalue := Reverse(value)
tr := []individualResult{}
err := s.conn.Select(&tr, "SELECT * FROM individual WHERE (which='A' AND value like $1) OR (which='Q' AND value like $2) ORDER BY value", value+"%", rvalue+"%")
reverseValue(tr)
return tr, err
}
//DeleteOld Deletes records that haven't been seen in DAYS, returns the total records deleted
func (s *SQLCommonStore) DeleteOld(days int64) (int64, error) {
var deletedRows int64
cutoff := time.Now().Add(time.Duration(-1*days) * time.Hour * 24)
res, err := s.conn.Exec("DELETE FROM individual WHERE last < $1", cutoff)
if err != nil {
return deletedRows, err
}
rows, err := res.RowsAffected()
deletedRows += rows
if err != nil {
return deletedRows, err
}
res, err = s.conn.Exec("DELETE FROM tuples WHERE last < $1", cutoff)
if err != nil {
return deletedRows, err
}
rows, err = res.RowsAffected()
deletedRows += rows
return deletedRows, err
}
================================================
FILE: store_sqlite.go
================================================
package main
import (
"time"
"github.com/jmoiron/sqlx"
_ "github.com/mattn/go-sqlite3"
)
const schema = `
CREATE TABLE IF NOT EXISTS tuples (
query character varying,
type character varying,
answer character varying,
count integer,
ttl integer,
first REAL,
last REAL,
PRIMARY KEY (query, type, answer)
) ;
CREATE INDEX IF NOT EXISTS tuples_query ON tuples(query);
CREATE INDEX IF NOT EXISTS tuples_answer ON tuples(answer);
CREATE INDEX IF NOT EXISTS tuples_first ON tuples(first);
CREATE INDEX IF NOT EXISTS tuples_last ON tuples(last);
CREATE TABLE IF NOT EXISTS individual (
which char(1),
value character varying,
count integer,
first REAL,
last REAL,
PRIMARY KEY (which, value)
);
CREATE INDEX IF NOT EXISTS individual_first ON individual(first);
CREATE INDEX IF NOT EXISTS individual_last ON individual(last);
CREATE TABLE IF NOT EXISTS filenames (
filename character varying PRIMARY KEY UNIQUE NOT NULL,
time REAL DEFAULT (datetime('now', 'localtime')),
aggregation_time real,
total_records int,
skipped_records int,
tuples int,
individual int,
store_time real,
inserted int,
updated int
);
PRAGMA case_sensitive_like=ON;
-- PRAGMA journal_mode=WAL;
-- PRAGMA synchronous=off;
PRAGMA temp_store = MEMORY;
PRAGMA cache_size = 5000;
`
type SQLiteStore struct {
conn *sqlx.DB
*SQLCommonStore
}
func NewSQLiteStore(uri string) (Store, error) {
conn, err := sqlx.Open("sqlite3", uri)
if err != nil {
return nil, err
}
common := &SQLCommonStore{conn: conn}
return &SQLiteStore{conn: conn, SQLCommonStore: common}, nil
}
func (s *SQLiteStore) Close() error {
return s.Close()
}
func (s *SQLiteStore) Init() error {
_, err := s.conn.Exec(schema)
return err
}
func (s *SQLiteStore) Update(ar aggregationResult) (UpdateResult, error) {
var result UpdateResult
start := time.Now()
tx, err := s.BeginTx()
if err != nil {
return result, err
}
//Setup the 4 different prepared statements
update_tuples, err := tx.Prepare(`UPDATE tuples SET
count=count+$1,
ttl=$2,
first=min(datetime($3, 'unixepoch'), first),
last =max(datetime($4, 'unixepoch'), last)
WHERE query=$5 AND type=$6 AND answer=$7`)
if err != nil {
return result, err
}
defer update_tuples.Close()
insert_tuples, err := tx.Prepare(`INSERT INTO tuples (query, type, answer, ttl, count, first, last)
VALUES ($1, $2, $3, $4, $5, datetime($6, 'unixepoch'), datetime($7,'unixepoch'))`)
if err != nil {
return result, err
}
defer insert_tuples.Close()
update_individual, err := tx.Prepare(`UPDATE individual SET
count=count+$1,
first=min(datetime($2, 'unixepoch'), first),
last =max(datetime($3, 'unixepoch'), last)
WHERE value=$4 AND which=$5`)
if err != nil {
return result, err
}
defer update_individual.Close()
insert_individual, err := tx.Prepare(`INSERT INTO individual (value, which, count, first, last)
VALUES ($1, $2, $3, datetime($4, 'unixepoch'), datetime($5,'unixepoch'))`)
if err != nil {
return result, err
}
defer insert_individual.Close()
// Ok, now let's update stuff
for _, q := range ar.Tuples {
//Update the tuples table
query := Reverse(q.query)
res, err := update_tuples.Exec(q.count, q.ttl, ToTS(q.first), ToTS(q.last), query, q.qtype, q.answer)
if err != nil {
return result, err
}
rows, err := res.RowsAffected()
if err != nil {
return result, err
}
if rows == 0 {
_, err := insert_tuples.Exec(query, q.qtype, q.answer, q.ttl, q.count, ToTS(q.first), ToTS(q.last))
if err != nil {
return result, err
}
result.Inserted++
} else {
result.Updated++
}
}
for _, q := range ar.Individual {
value := q.value
if q.which == "Q" {
value = Reverse(value)
}
res, err := update_individual.Exec(q.count, ToTS(q.first), ToTS(q.last), value, q.which)
if err != nil {
return result, err
}
rows, err := res.RowsAffected()
if err != nil {
return result, err
}
if rows == 0 {
_, err := insert_individual.Exec(value, q.which, q.count, ToTS(q.first), ToTS(q.last))
if err != nil {
return result, err
}
result.Inserted++
} else {
result.Updated++
}
}
result.Duration = time.Since(start)
return result, s.Commit()
}
================================================
FILE: store_test.go
================================================
package main
import (
"fmt"
"os"
"testing"
"github.com/stretchr/testify/assert"
)
type storeTest struct {
storetype string
uri string
}
var testStores = []storeTest{
{"sqlite", ":memory:"},
}
func init() {
pgTestUrl := os.Getenv("PG_TEST_URL")
if pgTestUrl != "" {
testStores = append(testStores, storeTest{"postgresql", pgTestUrl})
}
chTestUrl := os.Getenv("CH_TEST_URL")
if chTestUrl != "" {
testStores = append(testStores, storeTest{"clickhouse", chTestUrl})
}
}
func doTestLogIndexed(t *testing.T, s Store) {
s.Clear()
s.Init()
testFilename := "test.log"
indexed, err := s.IsLogIndexed(testFilename)
if err != nil {
t.Fatal(err)
}
if indexed != false {
t.Errorf("IsLogIndexed(%q) == %t, want false", testFilename, indexed)
}
var ar aggregationResult
var ur UpdateResult
err = s.SetLogIndexed(testFilename, ar, ur)
if err != nil {
t.Fatal(err)
}
indexed, err = s.IsLogIndexed(testFilename)
if err != nil {
t.Fatal(err)
}
if indexed != true {
t.Errorf("IsLogIndexed(%q) == %t, want true", testFilename, indexed)
}
}
func LoadFile(t *testing.T, s Store, fn string) UpdateResult {
aggregator := NewDNSAggregator()
err := aggregate(aggregator, fn)
if err != nil {
t.Fatal(err)
}
aggregated := aggregator.GetResult()
result, err := s.Update(aggregated)
if err != nil {
t.Fatal(err)
}
return result
}
func doTestUpdating(t *testing.T, s Store, forward bool) {
s.Clear()
s.Init()
var files []string
if forward {
files = []string{"test_data/reddit_1.txt", "test_data/reddit_2.txt"}
} else {
files = []string{"test_data/reddit_2.txt", "test_data/reddit_1.txt"}
}
result_a := LoadFile(t, s, files[0])
result_b := LoadFile(t, s, files[1])
// Hack for now. Clickhouse store doesn't report inserted vs updated
//TODO: add a method to Store interface to return a bool for this
expected_inserted := 31
expected_updated := 0
if _, ok := s.(*CHStore); ok {
expected_inserted = 0
expected_updated = 31
}
assert.EqualValues(t, result_a.Inserted, expected_inserted)
assert.EqualValues(t, result_a.Updated, expected_updated)
assert.EqualValues(t, result_b.Inserted, 0)
assert.EqualValues(t, result_b.Updated, 31)
recs, err := s.FindIndividual("www.reddit.com")
if err != nil {
t.Fatalf("Failed to find: %v", err)
return
}
//www.reddit.com Q 2 2016-04-01 00:03:03 2016-04-01 21:55:04
if assert.Equal(t, len(recs), 1) {
rec := recs[0]
assert.Equal(t, rec.Value, "www.reddit.com")
assert.Equal(t, rec.Which, "Q")
assert.EqualValues(t, rec.Count, 2)
//This is stupid, but I need to fix things so that they return actual dates
//and get a handle on the timezone BS.
//So for now, ignore the ' ' vs 'T' difference, and the hour
assert.Regexp(t, "2016-04-01...:03:03", rec.First)
assert.Regexp(t, "2016-04-01...:55:04", rec.Last)
}
//www.reddit.com A 198.41.208.138 2 300 2016-04-01 00:03:03 2016-04-01 21:55:04
trecs, err := s.FindTuples("198.41.208.138")
if err != nil {
t.Fatalf("Failed to find: %v", err)
return
}
if assert.Equal(t, len(trecs), 1) {
rec := trecs[0]
assert.Equal(t, rec.Query, "www.reddit.com")
assert.Equal(t, rec.Type, "A")
assert.Equal(t, rec.Answer, "198.41.208.138")
assert.EqualValues(t, rec.Count, 2)
//This is stupid, but I need to fix things so that they return actual dates
//and get a handle on the timezone BS.
//So for now, ignore the ' ' vs 'T' difference, and the hour
assert.Regexp(t, "2016-04-01...:03:03", rec.First)
assert.Regexp(t, "2016-04-01...:55:04", rec.Last)
}
}
// Output:
//A: Inserted=31 Updated=0
//B: Inserted=0 Updated=31
//Individual records: 1
//www.reddit.com Q 2 2016-04-01 00:03:03 2016-04-01 21:55:04
//Tuple records: 1
//www.reddit.com A 198.41.208.138 2 300 2016-04-01 00:03:03 2016-04-01 21:55:04
func BenchmarkUpdateStores(b *testing.B) {
for _, ts := range testStores {
b.Run(fmt.Sprintf("Indexing/%s", ts.storetype), func(b *testing.B) {
store, err := NewStore(ts.storetype, ts.uri)
if err != nil {
b.Fatalf("NewStore failed: %s", err)
}
aggregator := NewDNSAggregator()
err = aggregate(aggregator, "big.log.gz")
if err != nil {
b.Fatal(err)
}
aggregated := aggregator.GetResult()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := store.Update(aggregated)
if err != nil {
b.Fatalf("store.Update failed: %s", err)
}
}
})
}
}
func testFile(t *testing.T, s Store, fn string) error {
aggregator := NewDNSAggregator()
err := aggregate(aggregator, fn)
if err != nil {
return err
}
aggregated := aggregator.GetResult()
_, err = s.Update(aggregated)
if err != nil {
return err
}
return nil
}
func TestIndexingFiles(t *testing.T) {
allFiles := []string{
"./test_data/nbtstat.log",
"./test_data/garbage.log",
"./test_data/bad_ttl.log",
"./test_data/dns_json_iso8601.json",
}
for _, ts := range testStores {
t.Run(fmt.Sprintf("Indexing/%s", ts.storetype), func(t *testing.T) {
store, err := NewStore(ts.storetype, ts.uri)
if err != nil {
t.Fatalf("can't create store at %s: %v", ts.uri, err)
}
for _, fn := range allFiles {
t.Run(fn, func(t *testing.T) {
testFile(t, store, fn)
})
}
})
}
}
func doTestStore(t *testing.T, store Store) {
t.Run("testLogIndexed", func(t *testing.T) {
doTestLogIndexed(t, store)
})
t.Run("forward", func(t *testing.T) {
doTestUpdating(t, store, true)
})
t.Run("reverse", func(t *testing.T) {
doTestUpdating(t, store, false)
})
}
func TestStoreIndexing(t *testing.T) {
for _, ts := range testStores {
t.Run(ts.storetype, func(t *testing.T) {
store, err := NewStore(ts.storetype, ts.uri)
if err != nil {
t.Fatalf("can't create store at %s: %v", ts.uri, err)
}
doTestStore(t, store)
})
}
}
================================================
FILE: template/index.html
================================================
<!DOCTYPE html>
<html>
<head>
<title>pdns{{if .Query}} - {{.Query }}{{end}}</title>
<style type="text/css">
table, td, th {
border: 1px solid black;
border-collapse: collapse;
white-space: nowrap;
padding-left: 0.5em;
padding-right: 0.5em;
}
</style>
</head>
<body>
<form method="GET">
<fieldset>
<legend>pdns search</legend>
<label for="query"> Query </label>
<input type="search" id="query" name="query" value="{{.Query}}" autofocus>
<label for="exact"> Exact match </label>
<input type="checkbox" id="exact" name="exact" {{if .Exact}}checked{{end}} >
<input type="submit" value="Search">
</fieldset>
</form>
{{ if .Error }}
Error searching: {{.Error }}
{{ end }}
{{ if .Individual }}
<h1>Individual values</h1>
<table width="100%" border="1">
<thead>
<tr>
<th>Value</th>
<th>Which</th>
<th>Count</th>
<th>First</th>
<th>Last</th>
</tr>
</thead>
<tbody>
{{range $val := .Individual}}
<tr>
<td> {{$val.Value}} </td>
<td> {{$val.Which}} </td>
<td> {{$val.Count}} </td>
<td> {{$val.First}} </td>
<td> {{$val.Last}} </td>
</tr>
{{end}}
</tbody>
</table>
{{ end }}
{{ if .Tuples }}
<h1>Tuples</h1>
<table width="100%" border="1">
<thead>
<tr>
<th>Query</th>
<th>Type</th>
<th>Answer</th>
<th>TTL</th>
<th>Count</th>
<th>First</th>
<th>Last</th>
</tr>
</thead>
<tbody>
{{range $val := .Tuples}}
<tr>
<td> {{$val.Query}} </td>
<td> {{$val.Type}} </td>
<td> {{$val.Answer}} </td>
<td> {{$val.TTL}} </td>
<td> {{$val.Count}} </td>
<td> {{$val.First}} </td>
<td> {{$val.Last}} </td>
</tr>
{{end}}
</tbody>
</table>
{{ end }}
</body>
</html>
================================================
FILE: test_data/bad_ttl.log
================================================
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[3286499344.0,14.0],"rejected":false}
================================================
FILE: test_data/dns_json.log
================================================
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.37572,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518031,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226060.051722,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201758,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226064.375771,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226073.518083,"uid":"CWsRPl2Qx6gNhHARM","id.orig_h":"fe80::5ef9:38ff:fe95:536c","id.orig_p":5353,"id.resp_h":"ff02::fb","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226093.757659,"uid":"CyCXT53AEW082wEJuk","id.orig_h":"192.168.2.161","id.orig_p":33275,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15917,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":true}
{"ts":1504226093.772416,"uid":"CMbrzzpSPxeegCR4","id.orig_h":"192.168.2.161","id.orig_p":5841,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":15037,"rtt":0.001594,"query":"ssl.google-analytics.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["0.0.0.0"],"TTLs":[0.0],"rejected":false}
{"ts":1504226100.889636,"uid":"CeRmRa4wChxaXX5bPl","id.orig_h":"192.168.2.144","id.orig_p":52291,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":38092,"rtt":0.013108,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.190","74.125.136.91","74.125.136.136","74.125.136.93"],"TTLs":[75.0,75.0,75.0,75.0],"rejected":false}
{"ts":1504226100.892947,"uid":"CeRJhn3RZX4EGTnUqe","id.orig_h":"192.168.2.144","id.orig_p":51541,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":62630,"rtt":0.014942,"query":"sb.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:80f::200e"],"TTLs":[171.0],"rejected":false}
{"ts":1504226100.986226,"uid":"CUGNDW1T74UvdK9Mf9","id.orig_h":"192.168.2.144","id.orig_p":63234,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":4160,"rtt":0.019922,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.177.101","64.233.177.102","64.233.177.138","64.233.177.100","64.233.177.113","64.233.177.139"],"TTLs":[196.0,196.0,196.0,196.0,196.0,196.0],"rejected":false}
{"ts":1504226100.990028,"uid":"CWDOPh2uTQkhU7WH04","id.orig_h":"192.168.2.144","id.orig_p":61705,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9459,"rtt":0.017509,"query":"safebrowsing.cache.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:811::200e"],"TTLs":[95.0],"rejected":false}
{"ts":1504226037.090323,"uid":"CC3BJqLx0zgN89c35","id.orig_h":"192.168.2.157","id.orig_p":53477,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":36570,"rtt":0.012916,"query":"www.reddit.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["reddit.map.fastly.net","151.101.57.140"],"TTLs":[100.0,14.0],"rejected":false}
{"ts":1504226037.107365,"uid":"ConzS63HBcUhAmsxj1","id.orig_h":"192.168.2.157","id.orig_p":37538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":51137,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226037.121395,"uid":"CjjfFwN6sIpL3CNX6","id.orig_h":"192.168.2.157","id.orig_p":57316,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":46466,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.300376,"uid":"CC5fjfToo9iIu1S6d","id.orig_h":"192.168.2.157","id.orig_p":60315,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":26918,"rtt":0.012398,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["173.194.175.106","173.194.175.99","173.194.175.147","173.194.175.103","173.194.175.105","173.194.175.104"],"TTLs":[153.0,153.0,153.0,153.0,153.0,153.0],"rejected":false}
{"ts":1504226061.31575,"uid":"Cwb8jt3ylC0gNoPhP2","id.orig_h":"192.168.2.157","id.orig_p":43538,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":57700,"rtt":0.009332,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:400d:c08::6a"],"TTLs":[59.0],"rejected":false}
{"ts":1504226061.326262,"uid":"CbHWTgnel1m6H88hg","id.orig_h":"192.168.2.157","id.orig_p":57906,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":23801,"query":"www.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.114845,"uid":"Cseinn4ur1mzgm7Kb1","id.orig_h":"192.168.2.157","id.orig_p":34365,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41784,"rtt":0.012139,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["151.101.57.140"],"TTLs":[13.0],"rejected":false}
{"ts":1504226077.129055,"uid":"CjMZsD3QI6tfxwInEf","id.orig_h":"192.168.2.157","id.orig_p":35859,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":14171,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226077.151007,"uid":"CYkgGr4Pvo594QvbEl","id.orig_h":"192.168.2.157","id.orig_p":59605,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":32601,"query":"reddit.map.fastly.net","qclass":1,"qclass_name":"C_INTERNET","qtype":15,"qtype_name":"MX","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1504226083.730927,"uid":"Cc4BXO1HuOstqlUuZe","id.orig_h":"192.168.2.161","id.orig_p":28208,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":9969,"rtt":0.01341,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::5e"],"TTLs":[161.0],"rejected":false}
{"ts":1504226083.749612,"uid":"CPx0AD3djrVpt5vcdf","id.orig_h":"192.168.2.161","id.orig_p":12786,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":13082,"rtt":0.012164,"query":"update.googleapis.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.136.94"],"TTLs":[116.0],"rejected":false}
{"ts":1504226060.051483,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":32769,"qclass_name":"qclass-32769","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
{"ts":1504226061.201694,"uid":"CYmiFV2KquvFq4lac4","id.orig_h":"192.168.2.144","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
================================================
FILE: test_data/dns_json_iso8601.json
================================================
{"ts":"2019-08-27T21:00:00.058798Z","uid":"CERjsr4nnMSFf6Q617","id.orig_h":"192.168.2.116","id.orig_p":53678,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":30824,"rtt":0.007465,"query":"googlemail.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c06::11"],"TTLs":[299.0],"rejected":false}
{"ts":"2019-08-27T21:00:00.050787Z","uid":"Cxx0no4PbWXq8e8aha","id.orig_h":"192.168.2.116","id.orig_p":56470,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":41183,"rtt":0.007586,"query":"googlemail.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["74.125.196.17","74.125.196.18","74.125.196.83","74.125.196.19"],"TTLs":[299.0,299.0,299.0,299.0],"rejected":false}
{"ts":"2019-08-27T21:05:00.455676Z","uid":"CacroF3sLUaPW1Tpw7","id.orig_h":"192.168.2.116","id.orig_p":40955,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":1304,"rtt":0.009959,"query":"googlemail.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.185.19","64.233.185.18","64.233.185.17","64.233.185.83"],"TTLs":[299.0,299.0,299.0,299.0],"rejected":false}
{"ts":"2019-08-27T21:05:00.466232Z","uid":"Cs1FUC32JFma4IAahi","id.orig_h":"192.168.2.116","id.orig_p":53514,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":21792,"rtt":0.008193,"query":"googlemail.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["2607:f8b0:4002:c0c::13"],"TTLs":[299.0],"rejected":false}
{"ts":"2019-08-27T21:05:25.005297Z","uid":"CfPP9AblRknTTsXY8","id.orig_h":"192.168.2.133","id.orig_p":64423,"id.resp_h":"192.168.2.1","id.resp_p":53,"proto":"udp","trans_id":52998,"rtt":0.000628,"query":"googlemail.l.google.com","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["64.233.185.83","64.233.185.17","64.233.185.18","64.233.185.19"],"TTLs":[274.0,274.0,274.0,274.0],"rejected":false}
================================================
FILE: test_data/garbage.log
================================================
{"ts":1546319860.813048,"uid":"CrkLPv18FirmddtvQ","id.orig_h":"1.2.3.4","id.orig_p":58517,"id.resp_h":"4.3.2.1","id.resp_p":53,"proto":"udp","trans_id":3083,"query":"\u001e\u00d1$ppyt\u00a38+\u0093\u0016\u0036\u00dc\u00f5\u00a5\u00ac[5\u00e9*y\u00a2\u00b3\u00876\u00be#\u00bd;}\u000a\u001a\u00f1@\u0026\u0000\u000fz\u0018\u0009.no\u0026\u00e4\u00bele\u00fc\u008c","qclass":29467,"qclass_name":"qclass-29467","qtype":35599,"qtype_name":"query-35599","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
================================================
FILE: test_data/nbtstat.log
================================================
{"ts":1538959195.632702,"uid":"CDC4zv1yqQAZJ4dDXe","id.orig_h":"10.1.2.3","id.orig_p":137,"id.resp_h":"10.4.5.6","id.resp_p":137,"proto":"udp","trans_id":38607,"query":"*\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","qclass":1,"qclass_name":"C_INTERNET","qtype":33,"qtype_name":"NBSTAT","AA":false,"TC":false,"RD":false,"RA":false,"Z":1,"rejected":false}
================================================
FILE: test_data/reddit_1.txt
================================================
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path dns
#open 2016-04-01-00-00-40
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected cc
#types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool string
1459468983.750000 COKlryLloBE6EB7oe 192.168.1.1 62834 198.41.222.24 53 udp 22543 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.142,198.41.208.142,198.41.208.143,198.41.208.140,198.41.209.139,198.41.209.140,198.41.209.141,198.41.208.138,198.41.209.136,198.41.208.137,198.41.209.137,198.41.208.139,198.41.209.143,198.41.208.141,198.41.209.138 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
================================================
FILE: test_data/reddit_2.txt
================================================
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path dns
#open 2016-04-01-00-00-40
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected cc
#types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool string
1459547704.500000 CSJeQh1Uu4zoeDAkTa 192.168.1.1 54496 173.245.58.24 53 udp 45516 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.138,198.41.208.142,198.41.208.143,198.41.209.136,198.41.209.143,198.41.209.140,198.41.208.137,198.41.208.141,198.41.209.137,198.41.209.142,198.41.208.139,198.41.208.140,198.41.209.141,198.41.209.139,198.41.209.138 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
================================================
FILE: test_data/reddit_dns_2016-04-01.log
================================================
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path dns
#open 2016-04-01-00-00-40
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected cc
#types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool string
1459468983.743478 COKlryLloBE6EB7oe 192.168.1.1 62834 198.41.222.24 53 udp 22543 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.142,198.41.208.142,198.41.208.143,198.41.208.140,198.41.209.139,198.41.209.140,198.41.209.141,198.41.208.138,198.41.209.136,198.41.208.137,198.41.209.137,198.41.208.139,198.41.209.143,198.41.208.141,198.41.209.138 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459469040.755063 C8MFK740YjXEMPgXY6 192.168.1.1 53975 173.245.58.24 53 udp 3270 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.140,198.41.209.139,198.41.209.138,198.41.208.138,198.41.209.140,198.41.208.142,198.41.209.142,198.41.208.141,198.41.208.139,198.41.209.137,198.41.209.141,198.41.209.136,198.41.208.143,198.41.208.137,198.41.209.143 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459470277.597403 C0LfI41RHTGQRhAmPa 192.168.1.1 49499 198.41.223.24 53 udp 12439 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.142,198.41.208.141,198.41.209.141,198.41.209.139,198.41.209.138,198.41.209.140,198.41.208.143,198.41.208.142,198.41.209.136,198.41.209.143,198.41.208.140,198.41.208.137,198.41.208.139,198.41.209.137,198.41.208.138 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459483828.090695 CmX7tILbwd9U3or7d 192.168.1.1 53595 198.41.223.24 53 udp 55870 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.142,198.41.208.138,198.41.208.137,198.41.209.136,198.41.208.140,198.41.209.141,198.41.209.142,198.41.209.137,198.41.209.143,198.41.208.139,198.41.208.141,198.41.209.138,198.41.209.139,198.41.208.143,198.41.209.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459483828.097083 CrvRfW1GzsGjJ5anci 192.168.1.1 58683 198.41.222.24 53 udp 5753 www.reddit.com 1 C_INTERNET 28 AAAA 0 NOERROR F F F F 0 - - F -
1459509366.896876 CbHpRD47aSOLmD8O03 192.168.1.1 63648 198.41.222.24 53 udp 29743 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.139,198.41.209.136,198.41.208.137,198.41.208.143,198.41.208.142,198.41.209.138,198.41.208.141,198.41.209.142,198.41.209.141,198.41.209.139,198.41.209.137,198.41.208.138,198.41.208.140,198.41.209.140,198.41.209.143 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459513647.596456 Cbt9jc4T1qDQNNZG8d 192.168.1.1 52854 173.245.58.24 53 udp 10074 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.141,198.41.208.142,198.41.209.136,198.41.208.143,198.41.209.140,198.41.209.139,198.41.209.142,198.41.208.138,198.41.209.143,198.41.209.138,198.41.208.140,198.41.209.141,198.41.209.137,198.41.208.139,198.41.208.137 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459513648.271360 CBgKHC4jPHcFPbKY26 192.168.1.1 55307 198.41.223.24 53 udp 43077 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.142,198.41.209.142,198.41.209.141,198.41.209.139,198.41.208.140,198.41.208.138,198.41.208.137,198.41.209.143,198.41.208.143,198.41.209.138,198.41.209.137,198.41.209.140,198.41.208.139,198.41.208.141,198.41.209.136 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459514126.069800 COtFV7311oufk5QDkk 192.168.1.1 49392 198.41.222.24 53 udp 7588 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.139,198.41.209.138,198.41.209.136,198.41.208.142,198.41.209.141,198.41.208.138,198.41.209.137,198.41.208.140,198.41.209.143,198.41.208.137,198.41.209.139,198.41.209.140,198.41.208.143,198.41.209.142,198.41.208.141 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459514286.737869 CmC0er1hh7XQkd4Rk4 192.168.1.1 59509 198.41.223.24 53 udp 7091 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.143,198.41.208.143,198.41.209.137,198.41.209.140,198.41.208.138,198.41.209.142,198.41.208.142,198.41.208.137,198.41.209.139,198.41.208.139,198.41.208.140,198.41.209.138,198.41.209.141,198.41.208.141,198.41.209.136 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459514655.777144 ClgK2b4tj2FT3rWdic 192.168.1.1 53318 198.41.222.24 53 udp 11474 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.142,198.41.208.141,198.41.208.143,198.41.208.137,198.41.208.139,198.41.209.141,198.41.209.136,198.41.209.139,198.41.209.142,198.41.208.138,198.41.209.143,198.41.209.137,198.41.209.140,198.41.209.138,198.41.208.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459514965.558995 Cb3ZHoHMoFuFvxu8f 192.168.1.1 51384 173.245.58.24 53 udp 62422 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.141,198.41.209.137,198.41.208.140,198.41.209.138,198.41.209.142,198.41.208.141,198.41.208.142,198.41.208.137,198.41.209.139,198.41.209.140,198.41.208.143,198.41.209.143,198.41.208.138,198.41.208.139,198.41.209.136 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459515235.058652 C2UlHW2B3q5VbiSgqf 192.168.1.1 57516 198.41.222.24 53 udp 17273 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.138,198.41.208.139,198.41.209.140,198.41.209.137,198.41.208.140,198.41.209.142,198.41.209.141,198.41.208.137,198.41.208.143,198.41.208.142,198.41.209.136,198.41.209.143,198.41.209.138,198.41.208.141,198.41.209.139 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459515390.544248 Cb3K8O22OeM6vRMXml 192.168.1.1 58386 198.41.223.24 53 udp 57698 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.137,198.41.209.140,198.41.209.138,198.41.208.143,198.41.208.142,198.41.209.139,198.41.209.136,198.41.208.141,198.41.208.139,198.41.209.143,198.41.208.138,198.41.209.141,198.41.209.137,198.41.209.142,198.41.208.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459515993.135468 CeOksW1glap2bMkV2 192.168.1.1 52012 198.41.223.24 53 udp 38164 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.141,198.41.209.137,198.41.209.142,198.41.209.138,198.41.209.136,198.41.208.143,198.41.208.140,198.41.208.137,198.41.209.143,198.41.208.138,198.41.209.141,198.41.208.139,198.41.209.140,198.41.209.139,198.41.208.142 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459516120.670290 CcCxQi4LnGC36SMxJh 192.168.1.1 55753 198.41.222.24 53 udp 15884 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.142,198.41.208.140,198.41.208.139,198.41.208.142,198.41.209.138,198.41.209.136,198.41.209.137,198.41.208.143,198.41.208.137,198.41.208.141,198.41.209.140,198.41.209.141,198.41.208.138,198.41.209.143,198.41.209.139 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459516435.132207 C5Ms4b4xjaY48q9ovf 192.168.1.1 57171 173.245.58.24 53 udp 48836 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.137,198.41.208.139,198.41.209.139,198.41.209.141,198.41.208.141,198.41.209.138,198.41.208.143,198.41.209.136,198.41.208.142,198.41.209.142,198.41.208.137,198.41.209.143,198.41.208.138,198.41.209.140,198.41.208.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459516541.938562 CbDzCN1i5RGaABK5ce 192.168.1.1 59680 198.41.223.24 53 udp 30304 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.137,198.41.208.138,198.41.209.139,198.41.208.141,198.41.208.142,198.41.209.141,198.41.209.140,198.41.208.140,198.41.209.138,198.41.208.143,198.41.209.143,198.41.208.137,198.41.209.142,198.41.208.139,198.41.209.136 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459516796.374825 Cp4Jic255bEVtg6H3d 192.168.1.1 57628 173.245.58.24 53 udp 48076 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.139,198.41.209.142,198.41.209.138,198.41.209.141,198.41.208.140,198.41.209.137,198.41.209.143,198.41.208.141,198.41.209.136,198.41.209.140,198.41.208.138,198.41.209.139,198.41.208.143,198.41.208.142,198.41.208.137 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459516796.378706 CDbu8S1JyrNHwfhPd9 192.168.1.1 60808 198.41.222.24 53 udp 43846 www.reddit.com 1 C_INTERNET 28 AAAA 0 NOERROR F F F F 0 - - F -
1459517120.939082 Ch7wDi4S4qTUS5NRI6 192.168.1.1 63215 173.245.58.24 53 udp 55635 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.142,198.41.208.140,198.41.209.138,198.41.209.137,198.41.208.141,198.41.208.138,198.41.208.143,198.41.209.139,198.41.208.139,198.41.209.141,198.41.209.140,198.41.209.136,198.41.208.137,198.41.209.143,198.41.208.142 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459517494.235804 CsfYwa1SZtLt7cmU3j 192.168.1.1 54030 198.41.222.24 53 udp 40499 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.136,198.41.208.139,198.41.209.139,198.41.208.142,198.41.209.141,198.41.208.143,198.41.209.138,198.41.208.137,198.41.209.137,198.41.209.143,198.41.209.142,198.41.208.140,198.41.208.138,198.41.208.141,198.41.209.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459517912.258693 C3xEqm1bM5l2hWQjG8 192.168.1.1 63989 198.41.223.24 53 udp 59875 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.138,198.41.209.138,198.41.209.142,198.41.209.137,198.41.209.136,198.41.208.141,198.41.208.139,198.41.209.141,198.41.208.140,198.41.209.143,198.41.209.140,198.41.208.142,198.41.208.143,198.41.208.137,198.41.209.139 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459518341.550836 CX06yG1EEpNhAKZpG4 192.168.1.1 55433 198.41.222.24 53 udp 17934 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.143,198.41.209.136,198.41.208.142,198.41.208.137,198.41.208.138,198.41.209.137,198.41.209.142,198.41.208.143,198.41.209.141,198.41.208.141,198.41.208.139,198.41.208.140,198.41.209.139,198.41.209.138,198.41.209.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459518950.808185 CrQlZk1aoVEk18SBn1 192.168.1.1 57431 198.41.223.24 53 udp 12622 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.140,198.41.209.142,198.41.208.137,198.41.208.143,198.41.209.136,198.41.209.141,198.41.209.140,198.41.209.138,198.41.209.143,198.41.208.142,198.41.208.141,198.41.208.138,198.41.208.139,198.41.209.137,198.41.209.139 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459518950.794330 CSDGGT2i3suYULyudf 192.168.1.1 62552 173.245.58.24 53 udp 39934 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.138,198.41.209.143,198.41.209.136,198.41.209.140,198.41.208.137,198.41.208.139,198.41.209.139,198.41.208.141,198.41.208.140,198.41.208.143,198.41.208.138,198.41.209.142,198.41.208.142,198.41.209.141,198.41.209.137 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459519280.468503 C5p8qO1g8RjY32lXoe 192.168.1.1 53544 173.245.58.24 53 udp 53008 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.141,198.41.209.140,198.41.209.143,198.41.208.138,198.41.209.138,198.41.209.137,198.41.208.142,198.41.208.139,198.41.208.140,198.41.209.139,198.41.208.137,198.41.209.142,198.41.209.136,198.41.208.143,198.41.208.141 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459519707.516203 CbfBZ74hw3GFrUNDTh 192.168.1.1 57819 173.245.58.24 53 udp 42329 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.139,198.41.208.140,198.41.209.141,198.41.208.138,198.41.209.138,198.41.209.143,198.41.209.136,198.41.209.142,198.41.209.137,198.41.209.139,198.41.208.142,198.41.208.143,198.41.208.137,198.41.208.141,198.41.209.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520012.295282 CEjVdD1e0LqWV0tYPf 192.168.1.1 64019 198.41.222.24 53 udp 61858 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.143,198.41.209.139,198.41.209.138,198.41.209.137,198.41.208.142,198.41.208.140,198.41.209.140,198.41.208.141,198.41.208.138,198.41.209.143,198.41.208.137,198.41.209.136,198.41.208.139,198.41.209.142,198.41.209.141 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520122.898406 CsCmxNS1FaLTTtsEe 192.168.1.1 53778 198.41.223.24 53 udp 17311 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.138,198.41.209.143,198.41.208.143,198.41.208.142,198.41.209.142,198.41.209.136,198.41.208.137,198.41.209.137,198.41.209.140,198.41.209.139,198.41.209.141,198.41.208.140,198.41.208.138,198.41.208.139,198.41.208.141 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520194.714102 ClHYVA2rSyGTxfLex8 192.168.1.2 62020 173.245.58.24 53 udp 28486 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.139,198.41.208.139,198.41.208.141,198.41.209.143,198.41.209.142,198.41.208.142,198.41.208.138,198.41.209.138,198.41.209.140,198.41.209.136,198.41.208.140,198.41.209.141,198.41.208.143,198.41.209.137,198.41.208.137 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520194.714842 CqvTqU2iHvn3ZIjly2 192.168.1.2 56508 173.245.58.24 53 udp 34689 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.142,198.41.208.139,198.41.208.143,198.41.208.138,198.41.209.142,198.41.209.136,198.41.209.141,198.41.208.137,198.41.208.140,198.41.209.143,198.41.209.140,198.41.209.138,198.41.209.137,198.41.208.141,198.41.209.139 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520363.149589 CldhW6uEeo9jYhfC4 192.168.1.1 54622 173.245.58.24 53 udp 13951 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.142,198.41.209.142,198.41.209.139,198.41.209.138,198.41.209.143,198.41.208.137,198.41.208.138,198.41.208.143,198.41.208.140,198.41.209.137,198.41.209.136,198.41.208.141,198.41.209.140,198.41.209.141,198.41.208.139 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520437.322170 CnepGwdHbupHN7iCd 192.168.1.1 49550 198.41.222.24 53 udp 26886 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.142,198.41.209.139,198.41.209.141,198.41.208.137,198.41.209.137,198.41.208.140,198.41.208.142,198.41.209.140,198.41.209.143,198.41.208.138,198.41.208.139,198.41.209.136,198.41.208.143,198.41.209.138,198.41.208.141 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520523.892531 CdLpEC2QaCnEa6jMRk 192.168.1.2 53906 198.41.223.24 53 udp 17542 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.141,198.41.208.139,198.41.208.137,198.41.209.141,198.41.209.138,198.41.208.143,198.41.208.140,198.41.209.143,198.41.209.140,198.41.209.137,198.41.208.142,198.41.209.139,198.41.209.136,198.41.208.138,198.41.209.142 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520673.010387 CqWg3e6QVNbqxZx21 192.168.1.1 49691 198.41.223.24 53 udp 22197 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.141,198.41.208.138,198.41.209.137,198.41.209.142,198.41.209.139,198.41.208.143,198.41.208.141,198.41.208.140,198.41.208.142,198.41.209.138,198.41.209.136,198.41.208.139,198.41.209.143,198.41.209.140,198.41.208.137 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459520826.923034 CSvNPg2G1CKH3U7V6j 192.168.1.1 51938 198.41.222.24 53 udp 58493 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.141,198.41.209.139,198.41.209.137,198.41.208.143,198.41.208.137,198.41.208.138,198.41.208.142,198.41.208.139,198.41.209.138,198.41.209.136,198.41.208.140,198.41.209.142,198.41.208.141,198.41.209.143,198.41.209.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459521134.745701 CXNf9dwbAGYkj4zX5 192.168.1.1 59777 173.245.58.24 53 udp 54917 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.143,198.41.208.142,198.41.209.141,198.41.208.139,198.41.209.136,198.41.208.138,198.41.208.141,198.41.209.142,198.41.209.140,198.41.209.137,198.41.208.143,198.41.209.139,198.41.209.138,198.41.208.137,198.41.208.140 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459521408.473273 CawQs73CWM1TSaUqni 192.168.1.1 65084 198.41.223.24 53 udp 11212 reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.209.143,198.41.208.140,198.41.209.136,198.41.208.141,198.41.209.139,198.41.208.139,198.41.209.137,198.41.208.138,198.41.208.137,198.41.209.142,198.41.209.140,198.41.208.142,198.41.208.143,198.41.209.141,198.41.209.138 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000 F -
1459521436.669136 CsDjBv4W99bmo8MfU2 192.168.1.1 58039 198.41.222.24 53 udp 52231 www.reddit.com 1 C_INTERNET 1 A 0 NOERROR T F F F 0 198.41.208.140,198.41.209.138,198.41.209.142,198.41.208.139,198.41.209.143,198.41.208.137,198.41.208.142,198.41.208.141,198.41.209.139,198.41.208.143,198.41.209.140,198.41.208.138,198.41.209.136,198.41.209.137,198.41.209.141 300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,300.000000,3
gitextract_ljrd_5b5/ ├── .gitignore ├── .travis.yml ├── LICENSE ├── Makefile ├── README.md ├── aggregate.go ├── aggregate_test.go ├── dns-ans-query.bro ├── go.mod ├── go.sum ├── index.go ├── local_test.txt ├── main.go ├── read.go ├── read_ascii.go ├── read_json.go ├── read_test.go ├── store.go ├── store_clickhouse.go ├── store_pg.go ├── store_sql_common.go ├── store_sqlite.go ├── store_test.go ├── template/ │ └── index.html ├── test_data/ │ ├── bad_ttl.log │ ├── dns_json.log │ ├── dns_json_iso8601.json │ ├── garbage.log │ ├── nbtstat.log │ ├── reddit_1.txt │ ├── reddit_2.txt │ └── reddit_dns_2016-04-01.log ├── version.go └── web.go
SYMBOL INDEX (150 symbols across 16 files)
FILE: aggregate.go
function stripDecimal (line 17) | func stripDecimal(value string) string {
type DNSRecord (line 28) | type DNSRecord struct
type uniqueTuple (line 36) | type uniqueTuple struct
type uniqueIndividual (line 41) | type uniqueIndividual struct
type queryStat (line 46) | type queryStat struct
type aggregationResult (line 53) | type aggregationResult struct
method ShallowCopy (line 324) | func (ar *aggregationResult) ShallowCopy() aggregationResult {
method TupleJSONReader (line 344) | func (ar *aggregationResult) TupleJSONReader(reverseQuery bool) io.Rea...
method IndividualJSONReader (line 384) | func (ar *aggregationResult) IndividualJSONReader(reverseQuery bool) i...
type aggregatedTuple (line 63) | type aggregatedTuple struct
type aggregatedIndividual (line 67) | type aggregatedIndividual struct
type DNSAggregator (line 72) | type DNSAggregator struct
method SkipRecord (line 89) | func (d *DNSAggregator) SkipRecord() {
method AddRecord (line 93) | func (d *DNSAggregator) AddRecord(r DNSRecord) {
method GetResult (line 179) | func (d *DNSAggregator) GetResult() aggregationResult {
method Merge (line 240) | func (d *DNSAggregator) Merge(other *DNSAggregator) {
function NewDNSAggregator (line 80) | func NewDNSAggregator() *DNSAggregator {
function timeCompare (line 205) | func timeCompare(a, b string) int {
function aggregate (line 274) | func aggregate(aggregator *DNSAggregator, fn string) error {
type JSONTuple (line 334) | type JSONTuple struct
type JSONIndividual (line 376) | type JSONIndividual struct
FILE: aggregate_test.go
type ByValue (line 11) | type ByValue
method Len (line 13) | func (a ByValue) Len() int { return len(a) }
method Swap (line 14) | func (a ByValue) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
method Less (line 15) | func (a ByValue) Less(i, j int) bool { return a[i].value < a[j].value }
type ByTuple (line 17) | type ByTuple
method Len (line 19) | func (a ByTuple) Len() int { return len(a) }
method Swap (line 20) | func (a ByTuple) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
method Less (line 21) | func (a ByTuple) Less(i, j int) bool { return a[i].query+a[i].answer <...
function ExampleAggregate (line 23) | func ExampleAggregate() {
function ExampleAggregateMerge (line 62) | func ExampleAggregateMerge() {
function BenchmarkAggregate (line 128) | func BenchmarkAggregate(b *testing.B) {
function ExampleResultTupleJSONReader (line 141) | func ExampleResultTupleJSONReader() {
function ExampleResultIndividualJSONReader (line 173) | func ExampleResultIndividualJSONReader() {
FILE: index.go
function index (line 8) | func index(store Store, filenames []string) error {
FILE: main.go
function getStore (line 12) | func getStore() Store {
function init (line 145) | func init() {
function main (line 179) | func main() {
FILE: read.go
type Reader (line 9) | type Reader interface
type Record (line 13) | type Record interface
function NewBroReader (line 23) | func NewBroReader(r io.Reader) (Reader, error) {
FILE: read_ascii.go
function grab_value (line 13) | func grab_value(line string) string {
function extract_sep (line 18) | func extract_sep(line string) string {
type BroAsciiReader (line 27) | type BroAsciiReader struct
method Next (line 116) | func (b *BroAsciiReader) Next() (Record, error) {
method handleHeader (line 138) | func (b *BroAsciiReader) handleHeader(line string) error {
method HeadersChanged (line 158) | func (b *BroAsciiReader) HeadersChanged() bool {
method HandledHeaders (line 161) | func (b *BroAsciiReader) HandledHeaders() {
type ASCIIRecord (line 39) | type ASCIIRecord struct
method String (line 46) | func (r *ASCIIRecord) String() string {
method GetString (line 50) | func (r *ASCIIRecord) GetString(field string) string {
method GetTimestamp (line 58) | func (r *ASCIIRecord) GetTimestamp(field string) string {
method GetStringList (line 61) | func (r *ASCIIRecord) GetStringList(field string) []string {
method GetStringByIndex (line 66) | func (r *ASCIIRecord) GetStringByIndex(index int) string {
method GetFloat (line 69) | func (r *ASCIIRecord) GetFloat(field string) float64 {
method GetFloatByIndex (line 82) | func (r *ASCIIRecord) GetFloatByIndex(index int) float64 {
method IsMissingFieldError (line 90) | func (r *ASCIIRecord) IsMissingFieldError() bool {
method Error (line 94) | func (r *ASCIIRecord) Error() error {
method GetFieldIndex (line 101) | func (r *ASCIIRecord) GetFieldIndex(field string) int {
function NewBroAsciiReader (line 110) | func NewBroAsciiReader(r io.Reader) *BroAsciiReader {
FILE: read_json.go
type BroJSONReader (line 12) | type BroJSONReader struct
method Next (line 59) | func (b *BroJSONReader) Next() (Record, error) {
type JSONRecord (line 16) | type JSONRecord struct
method String (line 21) | func (r *JSONRecord) String() string {
method GetString (line 25) | func (r *JSONRecord) GetString(field string) string {
method GetStringList (line 30) | func (r *JSONRecord) GetStringList(field string) []string {
method GetFloat (line 38) | func (r *JSONRecord) GetFloat(field string) float64 {
method IsMissingFieldError (line 44) | func (r *JSONRecord) IsMissingFieldError() bool {
method Error (line 48) | func (r *JSONRecord) Error() error {
method GetTimestamp (line 78) | func (r *JSONRecord) GetTimestamp(field string) string {
function NewBroJSONReader (line 55) | func NewBroJSONReader(r *bufio.Reader) *BroJSONReader {
FILE: read_test.go
function BenchmarkReadASCII (line 7) | func BenchmarkReadASCII(b *testing.B) {
function BenchmarkReadJSON (line 14) | func BenchmarkReadJSON(b *testing.B) {
FILE: store.go
type Store (line 11) | type Store interface
type tupleResult (line 28) | type tupleResult struct
method String (line 50) | func (tr tupleResult) String() string {
type tupleResults (line 38) | type tupleResults
method Display (line 40) | func (tr tupleResults) Display() {
type individualResult (line 57) | type individualResult struct
method String (line 76) | func (ir individualResult) String() string {
type individualResults (line 64) | type individualResults
method Display (line 66) | func (ir individualResults) Display() {
type UpdateResult (line 82) | type UpdateResult struct
function NewStore (line 94) | func NewStore(storeType string, filename string) (Store, error) {
function ToTS (line 113) | func ToTS(t string) string {
FILE: store_clickhouse.go
constant tuples_temp_stmt (line 53) | tuples_temp_stmt = `
constant individual_temp_stmt (line 64) | individual_temp_stmt = `
type CHStore (line 73) | type CHStore struct
method Close (line 97) | func (s *CHStore) Close() error {
method Exec (line 100) | func (s *CHStore) Exec(stmt string) error {
method Init (line 105) | func (s *CHStore) Init() error {
method Clear (line 114) | func (s *CHStore) Clear() error {
method Begin (line 129) | func (s *CHStore) Begin() error {
method Commit (line 132) | func (s *CHStore) Commit() error {
method DeleteOld (line 138) | func (s *CHStore) DeleteOld(days int64) (int64, error) {
method Update (line 142) | func (s *CHStore) Update(ar aggregationResult) (UpdateResult, error) {
method IsLogIndexed (line 235) | func (s *CHStore) IsLogIndexed(filename string) (bool, error) {
method SetLogIndexed (line 247) | func (s *CHStore) SetLogIndexed(filename string, ar aggregationResult,...
method FindQueryTuples (line 262) | func (s *CHStore) FindQueryTuples(query string) (tupleResults, error) {
method FindTuples (line 269) | func (s *CHStore) FindTuples(query string) (tupleResults, error) {
method LikeTuples (line 277) | func (s *CHStore) LikeTuples(query string) (tupleResults, error) {
method FindIndividual (line 284) | func (s *CHStore) FindIndividual(value string) (individualResults, err...
method LikeIndividual (line 292) | func (s *CHStore) LikeIndividual(value string) (individualResults, err...
function NewCHStore (line 77) | func NewCHStore(uri string) (Store, error) {
FILE: store_pg.go
constant pgschema (line 15) | pgschema = `
type PGStore (line 111) | type PGStore struct
method Close (line 125) | func (s *PGStore) Close() error {
method Init (line 129) | func (s *PGStore) Init() error {
method Update (line 159) | func (s *PGStore) Update(ar aggregationResult) (UpdateResult, error) {
function NewPGStore (line 116) | func NewPGStore(uri string) (Store, error) {
function genFullBatchSelect (line 141) | func genFullBatchSelect(tmpl string, batchSize int) string {
FILE: store_sql_common.go
function Reverse (line 11) | func Reverse(s string) string {
type SQLCommonStore (line 19) | type SQLCommonStore struct
method Clear (line 25) | func (s *SQLCommonStore) Clear() error {
method Begin (line 29) | func (s *SQLCommonStore) Begin() error {
method BeginTx (line 34) | func (s *SQLCommonStore) BeginTx() (*sql.Tx, error) {
method Commit (line 49) | func (s *SQLCommonStore) Commit() error {
method IsLogIndexed (line 64) | func (s *SQLCommonStore) IsLogIndexed(filename string) (bool, error) {
method SetLogIndexed (line 82) | func (s *SQLCommonStore) SetLogIndexed(filename string, ar aggregation...
method FindQueryTuples (line 113) | func (s *SQLCommonStore) FindQueryTuples(query string) (tupleResults, ...
method FindTuples (line 120) | func (s *SQLCommonStore) FindTuples(query string) (tupleResults, error) {
method LikeTuples (line 128) | func (s *SQLCommonStore) LikeTuples(query string) (tupleResults, error) {
method FindIndividual (line 135) | func (s *SQLCommonStore) FindIndividual(value string) (individualResul...
method LikeIndividual (line 143) | func (s *SQLCommonStore) LikeIndividual(value string) (individualResul...
method DeleteOld (line 152) | func (s *SQLCommonStore) DeleteOld(days int64) (int64, error) {
function reverseQuery (line 98) | func reverseQuery(tr tupleResults) {
function reverseValue (line 104) | func reverseValue(tr individualResults) {
FILE: store_sqlite.go
constant schema (line 11) | schema = `
type SQLiteStore (line 57) | type SQLiteStore struct
method Close (line 71) | func (s *SQLiteStore) Close() error {
method Init (line 75) | func (s *SQLiteStore) Init() error {
method Update (line 80) | func (s *SQLiteStore) Update(ar aggregationResult) (UpdateResult, erro...
function NewSQLiteStore (line 62) | func NewSQLiteStore(uri string) (Store, error) {
FILE: store_test.go
type storeTest (line 11) | type storeTest struct
function init (line 20) | func init() {
function doTestLogIndexed (line 31) | func doTestLogIndexed(t *testing.T, s Store) {
function LoadFile (line 59) | func LoadFile(t *testing.T, s Store, fn string) UpdateResult {
function doTestUpdating (line 73) | func doTestUpdating(t *testing.T, s Store, forward bool) {
function BenchmarkUpdateStores (line 149) | func BenchmarkUpdateStores(b *testing.B) {
function testFile (line 174) | func testFile(t *testing.T, s Store, fn string) error {
function TestIndexingFiles (line 188) | func TestIndexingFiles(t *testing.T) {
function doTestStore (line 210) | func doTestStore(t *testing.T, store Store) {
function TestStoreIndexing (line 222) | func TestStoreIndexing(t *testing.T) {
FILE: version.go
constant VERSION (line 4) | VERSION = "0.7.0"
FILE: web.go
type pdnsHandler (line 16) | type pdnsHandler struct
method handleSearchTuples (line 20) | func (h *pdnsHandler) handleSearchTuples(w http.ResponseWriter, req *h...
method handleSearchIndividual (line 44) | func (h *pdnsHandler) handleSearchIndividual(w http.ResponseWriter, re...
method handleUI (line 77) | func (h *pdnsHandler) handleUI(w http.ResponseWriter, req *http.Reques...
type Results (line 69) | type Results struct
function startWeb (line 111) | func startWeb(s Store, bind string) {
Condensed preview — 34 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (317K chars).
[
{
"path": ".gitignore",
"chars": 199,
"preview": "*.py[co]\n\n# Packages\n*.egg\n*.egg-info\ndist\nbuild\neggs\nparts\nbin\ndevelop-eggs\n.installed.cfg\n\n# Installer logs\npip-log.tx"
},
{
"path": ".travis.yml",
"chars": 416,
"preview": "language: go\n\ngo:\n - \"1.16\"\n\nservices:\n - postgresql\n - docker\n\nenv:\n - PG_TEST_URL=postgres://postgres:password@loc"
},
{
"path": "LICENSE",
"chars": 1069,
"preview": "MIT License\n\nCopyright (c) 2022 Justin Azoff\n\nPermission is hereby granted, free of charge, to any person obtaining a co"
},
{
"path": "Makefile",
"chars": 372,
"preview": "all: build test\nbuild:\n\tgo get -t -v ./...\n\tgo build\ntest:\n\tgo test -v ./...\nstatic:\n\tgo get -t -v ./...\n\tgo build --ldf"
},
{
"path": "README.md",
"chars": 2660,
"preview": "Passive DNS for Zeek\n===================\n\nThis is an extremely simple implementation of a passive DNS collection system\n"
},
{
"path": "aggregate.go",
"chars": 8142,
"preview": "package main\n\nimport (\n\t\"encoding/json\"\n\t\"errors\"\n\t\"io\"\n\t\"log\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\topendecompress \"github.co"
},
{
"path": "aggregate_test.go",
"chars": 5731,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"sort\"\n\t\"testing\"\n)\n\ntype ByValue []aggregatedIndividual\n\nfunc (a ByVa"
},
{
"path": "dns-ans-query.bro",
"chars": 401,
"preview": "##! Add the peer to the connection logs.\n\nmodule DNS;\n\nexport {\n redef record DNS::Info += {\n ans_query: vecto"
},
{
"path": "go.mod",
"chars": 485,
"preview": "module github.com/JustinAzoff/zeek-pdns\n\ngo 1.16\n\nrequire (\n\tgithub.com/ClickHouse/clickhouse-go v1.4.3\n\tgithub.com/Just"
},
{
"path": "go.sum",
"chars": 19328,
"preview": "cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ngithub.com/BurntSushi/toml v0.3.1 h1:"
},
{
"path": "index.go",
"chars": 1602,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n)\n\nfunc index(store Store, filenames []string) error {\n\tvar didWork bool\n\tstore.Beg"
},
{
"path": "local_test.txt",
"chars": 197,
"preview": "docker run --rm -t -i -p 5432:5432 -e POSTGRES_DB=pdns_test -e POSTGRES_PASSWORD=pdns postgres\nPG_TEST_URL=\"postgres://p"
},
{
"path": "main.go",
"chars": 4067,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\n\t\"github.com/spf13/cobra\"\n\t\"github.com/spf13/viper\"\n)\n\nfunc getStore() Store"
},
{
"path": "read.go",
"chars": 684,
"preview": "package main\n\nimport (\n\t\"bufio\"\n\t\"fmt\"\n\t\"io\"\n)\n\ntype Reader interface {\n\tNext() (Record, error)\n}\n\ntype Record interface"
},
{
"path": "read_ascii.go",
"chars": 3327,
"preview": "package main\n\nimport (\n\t\"bufio\"\n\t\"encoding/hex\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"strconv\"\n\t\"strings\"\n)\n\nfunc grab_value(line string"
},
{
"path": "read_json.go",
"chars": 1891,
"preview": "package main\n\nimport (\n\t\"bufio\"\n\t\"fmt\"\n\t\"io\"\n\t\"strings\"\n\n\t\"github.com/buger/jsonparser\"\n)\n\ntype BroJSONReader struct {\n\t"
},
{
"path": "read_test.go",
"chars": 387,
"preview": "package main\n\nimport (\n\t\"testing\"\n)\n\nfunc BenchmarkReadASCII(b *testing.B) {\n\tfor i := 0; i < b.N; i++ {\n\t\tfn := \"test_d"
},
{
"path": "store.go",
"chars": 2823,
"preview": "package main\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\t\"log\"\n)\n\ntype Store interface {\n\tInit() error\n\tClear() error"
},
{
"path": "store_clickhouse.go",
"chars": 8450,
"preview": "package main\n\nimport (\n\t\"database/sql\"\n\t\"fmt\"\n\t\"net/url\"\n\t\"time\"\n\n\t_ \"github.com/ClickHouse/clickhouse-go\"\n\t\"github.com/"
},
{
"path": "store_pg.go",
"chars": 6423,
"preview": "package main\n\nimport (\n\t\"database/sql\"\n\t\"fmt\"\n\t\"log\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/jmoiron/sqlx\"\n\n\t\"github.com/lib/pq"
},
{
"path": "store_sql_common.go",
"chars": 4628,
"preview": "package main\n\nimport (\n\t\"database/sql\"\n\t\"errors\"\n\t\"time\"\n\n\t\"github.com/jmoiron/sqlx\"\n)\n\nfunc Reverse(s string) string {\n"
},
{
"path": "store_sqlite.go",
"chars": 4185,
"preview": "package main\n\nimport (\n\t\"time\"\n\n\t\"github.com/jmoiron/sqlx\"\n\n\t_ \"github.com/mattn/go-sqlite3\"\n)\n\nconst schema = `\nCREATE "
},
{
"path": "store_test.go",
"chars": 5817,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype storeTest struct {\n\tstoret"
},
{
"path": "template/index.html",
"chars": 1718,
"preview": "<!DOCTYPE html>\n<html>\n<head>\n\t<title>pdns{{if .Query}} - {{.Query }}{{end}}</title>\n\n<style type=\"text/css\">\ntable, td,"
},
{
"path": "test_data/bad_ttl.log",
"chars": 452,
"preview": "{\"ts\":1504226037.090323,\"uid\":\"CC3BJqLx0zgN89c35\",\"id.orig_h\":\"192.168.2.157\",\"id.orig_p\":53477,\"id.resp_h\":\"192.168.2.1"
},
{
"path": "test_data/dns_json.log",
"chars": 84711,
"preview": "{\"ts\":1504226037.090323,\"uid\":\"CC3BJqLx0zgN89c35\",\"id.orig_h\":\"192.168.2.157\",\"id.orig_p\":53477,\"id.resp_h\":\"192.168.2.1"
},
{
"path": "test_data/dns_json_iso8601.json",
"chars": 2407,
"preview": "{\"ts\":\"2019-08-27T21:00:00.058798Z\",\"uid\":\"CERjsr4nnMSFf6Q617\",\"id.orig_h\":\"192.168.2.116\",\"id.orig_p\":53678,\"id.resp_h\""
},
{
"path": "test_data/garbage.log",
"chars": 516,
"preview": "{\"ts\":1546319860.813048,\"uid\":\"CrkLPv18FirmddtvQ\",\"id.orig_h\":\"1.2.3.4\",\"id.orig_p\":58517,\"id.resp_h\":\"4.3.2.1\",\"id.resp"
},
{
"path": "test_data/nbtstat.log",
"chars": 395,
"preview": "{\"ts\":1538959195.632702,\"uid\":\"CDC4zv1yqQAZJ4dDXe\",\"id.orig_h\":\"10.1.2.3\",\"id.orig_p\":137,\"id.resp_h\":\"10.4.5.6\",\"id.res"
},
{
"path": "test_data/reddit_1.txt",
"chars": 966,
"preview": "#separator \\x09\n#set_separator\t,\n#empty_field\t(empty)\n#unset_field\t-\n#path\tdns\n#open\t2016-04-01-00-00-40\n#fields\tts\tuid\t"
},
{
"path": "test_data/reddit_2.txt",
"chars": 967,
"preview": "#separator \\x09\n#set_separator\t,\n#empty_field\t(empty)\n#unset_field\t-\n#path\tdns\n#open\t2016-04-01-00-00-40\n#fields\tts\tuid\t"
},
{
"path": "test_data/reddit_dns_2016-04-01.log",
"chars": 110814,
"preview": "#separator \\x09\n#set_separator\t,\n#empty_field\t(empty)\n#unset_field\t-\n#path\tdns\n#open\t2016-04-01-00-00-40\n#fields\tts\tuid\t"
},
{
"path": "version.go",
"chars": 43,
"preview": "package main\n\nconst (\n\tVERSION = \"0.7.0\"\n)\n"
},
{
"path": "web.go",
"chars": 2736,
"preview": "package main\n\nimport (\n\t\"embed\"\n\t\"encoding/json\"\n\t\"log\"\n\t\"net/http\"\n\t\"text/template\"\n\n\t\"github.com/gorilla/mux\"\n)\n\n//go:"
}
]
About this extraction
This page contains the full source code of the JustinAzoff/bro-pdns GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 34 files (282.2 KB), approximately 124.2k tokens, and a symbol index with 150 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.