[
  {
    "path": ".gitignore",
    "content": ".idea\n.vscode\n\ndb/*\n\n# code coverage\n*.out\ncover.html\ncoverage.txt\n\n# profiling\n*.test\nprofile_cpu.pdf"
  },
  {
    "path": ".travis.yml",
    "content": "language: go\n\nsudo: false\ndist: trusty\n\nbranches:\n  only:\n  - master\n\ngo:\n- 1.15.x\n\nbefore_install:\n- make deps-ci\n\nscript:\n- make test-ci\n\nafter_success:\n- bash <(curl -s \"https://codecov.io/bash\")\t\n- $HOME/gopath/bin/goveralls -coverprofile=coverage.txt -service=travis-ci -repotoken $COVERALLS_TOKEN"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2019 Adil H\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "Makefile",
    "content": "test:\n\tgo test -race ./pkg/...\ntest-cover:\n\tgo test -race -coverprofile cover.out -covermode=atomic  ./pkg/...\n\tgo tool cover -html=cover.out -o cover.html\n\topen cover.html\ntest-ci:\n\tgo test -race -coverprofile=coverage.txt -covermode=atomic ./pkg/...\nbench:\n\tgo test -run=XXX -bench=. -benchtime=5s ./pkg/blero/\ndeps:\n\tgo mod download\ndeps-ci: deps\n\tgo get golang.org/x/tools/cmd/cover\n\tgo get github.com/mattn/goveralls"
  },
  {
    "path": "README.md",
    "content": "# Goblero \n\nPure Go, Simple, Embedded, Persistent Job Queue, backed by [BadgerDB](https://github.com/dgraph-io/badger)\n\n[![Build Status](https://travis-ci.org/didil/goblero.svg?branch=master)](https://travis-ci.org/didil/goblero)\n[![Coverage Status](https://coveralls.io/repos/github/didil/goblero/badge.svg?branch=master)](https://coveralls.io/github/didil/goblero?branch=master)\n[![goreportcard](https://goreportcard.com/badge/github.com/didil/goblero)](https://goreportcard.com/report/github.com/didil/goblero)\n[![codebeat badge](https://codebeat.co/badges/1d261e4f-36ff-42b5-b015-e31eb7aa7e7d)](https://codebeat.co/projects/github-com-didil-goblero-master)\n[![Documentation](https://godoc.org/github.com/didil/goblero/pkg/blero?status.svg)](https://godoc.org/github.com/didil/goblero/pkg/blero)\n\n\n**DO NOT USE IN PRODUCTION** This library is still in Alpha / Work In Progress \n\n## About Goblero\nIntro article: [Introducing Goblero: a Go Embedded Job Queue](https://medium.com/@didil/introducing-goblero-a-go-embedded-job-queue-6dfec8e24d4c)\n- Pure Go library, no cgo\n- Simple, embedded, persistent job queue\n- Provides in-process job processing to any Go app\n- The jobs/status changes are persisted to disk after each operation and pending jobs can continue processing after an app restart or a crash\n- Allows multiple \"processors\", each processor/worker processes one job at a time then is assigned a new job, etc\n- The storage engine used is [BadgerDB](https://github.com/dgraph-io/badger)\n\n*P.S: Why is the library named Goblero ? Go for the Go programming language obviously, and Badger in french is \"Blaireau\", but blero is easier to pronounce :)* \n\n## Usage \nThe full API is documented on [godoc.org](https://godoc.org/github.com/didil/goblero/pkg/blero). There is also a demo repo [goblero-demo](https://github.com/didil/goblero-demo/tree/master)\n\nGet package\n````\ngo get -u github.com/didil/goblero/pkg/blero\n````\nAPI\n\n````\n// Create a new Blero backend\nbl := blero.New(\"db/\")\n\n// Start Blero\nbl.Start()\n\n// defer Stopping Blero\ndefer bl.Stop()\n\n// register a processor\nbl.RegisterProcessorFunc(func(j *blero.Job) error {\n  // Do some processing, access job name with j.Name, job data with j.Data\n})\n\n// enqueue a job\nbl.EnqueueJob(\"MyJob\", []byte(\"My Job Data\"))\n\n````\n\n## Benchmarks\n````\n# Core i5 laptop / 8GB Ram / SSD \nmake bench\nBenchmarkEnqueue/EnqueueJob-4          50000            159942 ns/op (~ 6250 ops/s)\nBenchmarkEnqueue/dequeueJob-4           5000           2767260 ns/op (~  361 ops/s)\n\n````\n\n## Todo:\n- Restart interrupted jobs after app restart/crashes\n- Sweep completed jobs from the \"complete\" queue\n- Failed Jobs retry options\n- Allow batch enqueuing\n- Add support for Go contexts\n- Test in real conditions under high load\n- Expose Prometheus Metrics in an Http handler\n- Optimize performance / Locking\n\n\n## Contributing\nAll contributions (PR, feedback, bug reports, ideas, etc.) are welcome !\n\n[![](https://sourcerer.io/fame/didil/didil/goblero/images/0)](https://sourcerer.io/fame/didil/didil/goblero/links/0)[![](https://sourcerer.io/fame/didil/didil/goblero/images/1)](https://sourcerer.io/fame/didil/didil/goblero/links/1)[![](https://sourcerer.io/fame/didil/didil/goblero/images/2)](https://sourcerer.io/fame/didil/didil/goblero/links/2)[![](https://sourcerer.io/fame/didil/didil/goblero/images/3)](https://sourcerer.io/fame/didil/didil/goblero/links/3)[![](https://sourcerer.io/fame/didil/didil/goblero/images/4)](https://sourcerer.io/fame/didil/didil/goblero/links/4)[![](https://sourcerer.io/fame/didil/didil/goblero/images/5)](https://sourcerer.io/fame/didil/didil/goblero/links/5)[![](https://sourcerer.io/fame/didil/didil/goblero/images/6)](https://sourcerer.io/fame/didil/didil/goblero/links/6)[![](https://sourcerer.io/fame/didil/didil/goblero/images/7)](https://sourcerer.io/fame/didil/didil/goblero/links/7)"
  },
  {
    "path": "go.mod",
    "content": "module github.com/didil/goblero\n\ngo 1.26\n\nrequire (\n\tgithub.com/dgraph-io/badger v1.6.2\n\tgithub.com/stretchr/testify v1.11.1\n)\n\nrequire (\n\tgithub.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect\n\tgithub.com/cespare/xxhash/v2 v2.3.0 // indirect\n\tgithub.com/davecgh/go-spew v1.1.1 // indirect\n\tgithub.com/dgraph-io/ristretto v0.2.0 // indirect\n\tgithub.com/dustin/go-humanize v1.0.1 // indirect\n\tgithub.com/golang/protobuf v1.5.4 // indirect\n\tgithub.com/pkg/errors v0.9.1 // indirect\n\tgithub.com/pmezard/go-difflib v1.0.0 // indirect\n\tgithub.com/stretchr/objx v0.5.3 // indirect\n\tgolang.org/x/net v0.52.0 // indirect\n\tgolang.org/x/sys v0.42.0 // indirect\n\tgoogle.golang.org/protobuf v1.36.11 // indirect\n\tgopkg.in/yaml.v3 v3.0.1 // indirect\n)\n"
  },
  {
    "path": "go.sum",
    "content": "github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 h1:cTp8I5+VIoKjsnZuH8vjyaysT/ses3EvZeaV/1UkF2M=\ngithub.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8=\ngithub.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=\ngithub.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=\ngithub.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=\ngithub.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=\ngithub.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=\ngithub.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=\ngithub.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=\ngithub.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=\ngithub.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=\ngithub.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=\ngithub.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=\ngithub.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/dgraph-io/badger v1.6.2 h1:mNw0qs90GVgGGWylh0umH5iag1j6n/PeJtNvL6KY/x8=\ngithub.com/dgraph-io/badger v1.6.2/go.mod h1:JW2yswe3V058sS0kZ2h/AXeDSqFjxnZcRrVH//y2UQE=\ngithub.com/dgraph-io/ristretto v0.0.2/go.mod h1:KPxhHT9ZxKefz+PCeOGsrHpl1qZ7i70dGTu2u+Ahh6E=\ngithub.com/dgraph-io/ristretto v0.2.0 h1:XAfl+7cmoUDWW/2Lx8TGZQjjxIQ2Ley9DSf52dru4WE=\ngithub.com/dgraph-io/ristretto v0.2.0/go.mod h1:8uBHCU/PBV4Ag0CJrP47b9Ofby5dqWNh4FicAdoqFNU=\ngithub.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=\ngithub.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 h1:fAjc9m62+UWV/WAFKLNi6ZS0675eEUC9y3AlwSbQu1Y=\ngithub.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=\ngithub.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=\ngithub.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=\ngithub.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=\ngithub.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=\ngithub.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=\ngithub.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=\ngithub.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=\ngithub.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=\ngithub.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=\ngithub.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=\ngithub.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs=\ngithub.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=\ngithub.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=\ngithub.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=\ngithub.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=\ngithub.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=\ngithub.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=\ngithub.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=\ngithub.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=\ngithub.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=\ngithub.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=\ngithub.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=\ngithub.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=\ngithub.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=\ngithub.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=\ngithub.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=\ngithub.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=\ngithub.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=\ngithub.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=\ngithub.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=\ngithub.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=\ngithub.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=\ngithub.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4=\ngithub.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0=\ngithub.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=\ngithub.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=\ngithub.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=\ngithub.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=\ngithub.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=\ngithub.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=\ngolang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=\ngolang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=\ngolang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=\ngolang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=\ngolang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=\ngolang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=\ngolang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngoogle.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=\ngoogle.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=\ngopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=\ngopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=\ngopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=\ngopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\n"
  },
  {
    "path": "pkg/blero/backend.go",
    "content": "package blero\n\nimport \"fmt\"\n\n// Blero struct\ntype Blero struct {\n\tdispatcher *dispatcher\n\tqueue      *queue\n}\n\n// New creates new Blero Backend\nfunc New(dbPath string) *Blero {\n\tbl := &Blero{}\n\tpStore := newProcessorsStore()\n\tbl.dispatcher = newDispatcher(pStore)\n\tbl.queue = newQueue(queueOpts{DBPath: dbPath})\n\treturn bl\n}\n\n// Start Blero\nfunc (bl *Blero) Start() error {\n\tfmt.Println(\"Starting Blero ...\")\n\terr := bl.queue.start()\n\tif err != nil {\n\t\treturn err\n\t}\n\tbl.dispatcher.startLoop(bl.queue)\n\treturn nil\n}\n\n// Stop Blero and Release resources\nfunc (bl *Blero) Stop() error {\n\tfmt.Println(\"Stopping Blero ...\")\n\tbl.dispatcher.stopLoop()\n\treturn bl.queue.stop()\n}\n\n// EnqueueJob enqueues a new Job and returns the job id\nfunc (bl *Blero) EnqueueJob(name string, data []byte) (uint64, error) {\n\tjID, err := bl.queue.enqueueJob(name, data)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\n\t// signal that a new job was enqueued\n\tbl.dispatcher.signalLoop()\n\n\treturn jID, nil\n}\n\n// EnqueueJobs enqueues new Jobs\n/*func (bl *Blero) EnqueueJobs(names string) (uint64, error) {\n\n}*/\n\n// RegisterProcessor registers a new processor and returns the processor id\nfunc (bl *Blero) RegisterProcessor(p Processor) int {\n\treturn bl.dispatcher.registerProcessor(p)\n}\n\n// RegisterProcessorFunc registers a new ProcessorFunc and returns the processor id\nfunc (bl *Blero) RegisterProcessorFunc(f func(j *Job) error) int {\n\treturn bl.dispatcher.registerProcessor(ProcessorFunc(f))\n}\n\n// UnregisterProcessor unregisters a processor\n// No more jobs will be assigned but if will not cancel a job that already started processing\nfunc (bl *Blero) UnregisterProcessor(pID int) {\n\tbl.dispatcher.unregisterProcessor(pID)\n}\n"
  },
  {
    "path": "pkg/blero/backend_test.go",
    "content": "package blero\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestBlero_StartNoDBPath(t *testing.T) {\n\tbl := New(\"\")\n\terr := bl.Start()\n\tassert.EqualError(t, err, \"DBPath is required\")\n}\n\nfunc TestBlero_StartDBPathDoesntExist(t *testing.T) {\n\tbl := New(\"/tmp/12354/56547/45459/blero/dbx/\")\n\terr := bl.Start()\n\tassert.EqualError(t, err, \"Error Creating Dir: \\\"/tmp/12354/56547/45459/blero/dbx/\\\": mkdir /tmp/12354/56547/45459/blero/dbx/: no such file or directory\")\n}\n\nfunc BenchmarkEnqueue(b *testing.B) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tif err != nil {\n\t\tb.Error(err)\n\t}\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tjobName := \"MyJob\"\n\tjobData := []byte(\"MyJobData\")\n\tb.ResetTimer()\n\n\tb.Run(\"EnqueueJob\", func(b *testing.B) {\n\t\tfor n := 0; n < b.N; n++ {\n\t\t\t_, err := bl.EnqueueJob(jobName, jobData)\n\t\t\tif err != nil {\n\t\t\t\tb.Error(err)\n\t\t\t}\n\t\t}\n\t})\n\n\tb.ResetTimer()\n\n\tb.Run(\"dequeueJob\", func(b *testing.B) {\n\t\tfor n := 0; n < b.N; n++ {\n\t\t\t_, err := bl.queue.dequeueJob()\n\t\t\tif err != nil {\n\t\t\t\tb.Error(err)\n\t\t\t}\n\t\t}\n\t})\n\n}\n"
  },
  {
    "path": "pkg/blero/dispatch.go",
    "content": "package blero\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"sync\"\n)\n\n// dispatcher struct\ntype dispatcher struct {\n\tdispatchL sync.Mutex\n\tch        chan int\n\tquitCh    chan struct{}\n\tpStore    *processorsStore\n}\n\n// newDispatcher creates new Dispatcher\nfunc newDispatcher(pStore *processorsStore) *dispatcher {\n\td := &dispatcher{}\n\td.ch = make(chan int, 100)\n\td.quitCh = make(chan struct{})\n\td.pStore = pStore\n\treturn d\n}\n\n// testable stderr\nvar stdErr io.ReadWriter = os.Stderr\n\n// startLoop starts the dispatcher assignment loop\nfunc (d *dispatcher) startLoop(q *queue) {\n\tgo func() {\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-d.ch:\n\t\t\t\terr := d.assignJobs(q)\n\t\t\t\tif err != nil {\n\t\t\t\t\tfmt.Fprintf(stdErr, \"Cannot assign jobs: %v\", err)\n\t\t\t\t}\n\t\t\tcase <-d.quitCh: // loop was stopped\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}()\n}\n\n// signalLoop signals to the dispatcher loop that an assignment check might need to run\nfunc (d *dispatcher) signalLoop() {\n\tgo func() {\n\t\tselect {\n\t\tcase <-d.quitCh: // loop was stopped\n\t\t\treturn\n\t\tcase d.ch <- 1:\n\t\t\treturn\n\t\t}\n\t}()\n}\n\n// stopLoop stops the dispatcher assignment loop\nfunc (d *dispatcher) stopLoop() {\n\tclose(d.quitCh)\n}\n\n// registerProcessor registers a new processor\nfunc (d *dispatcher) registerProcessor(p Processor) int {\n\td.dispatchL.Lock()\n\tdefer d.dispatchL.Unlock()\n\n\tpID := d.pStore.registerProcessor(p)\n\n\t// signal that the processor is now available\n\td.signalLoop()\n\n\treturn pID\n}\n\n// unregisterProcessor unregisters a processor\n// No more jobs will be assigned but if will not cancel a job that already started processing\nfunc (d *dispatcher) unregisterProcessor(pID int) {\n\td.dispatchL.Lock()\n\tdefer d.dispatchL.Unlock()\n\n\td.pStore.unregisterProcessor(pID)\n}\n\n// assignJobs assigns pending jobs from the queue to free processors\nfunc (d *dispatcher) assignJobs(q *queue) error {\n\td.dispatchL.Lock()\n\tdefer d.dispatchL.Unlock()\n\n\tpIDs := d.pStore.getAvailableProcessorsIDs()\n\n\tfor _, pID := range pIDs {\n\t\terr := d.assignJob(q, pID)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// assignJob assigns a pending job processor #pID and starts the run\n// NOT THREAD SAFE !! only call from assignJobs\nfunc (d *dispatcher) assignJob(q *queue, pID int) error {\n\tp := d.pStore.getProcessor(pID)\n\tif p == nil {\n\t\treturn fmt.Errorf(\"Processor %v not found\", pID)\n\t}\n\n\tj, err := q.dequeueJob()\n\tif err != nil {\n\t\treturn err\n\t}\n\t// no jobs to assign\n\tif j == nil {\n\t\treturn nil\n\t}\n\n\tif d.pStore.isProcessorBusy(pID) {\n\t\treturn fmt.Errorf(\"Cannot assign job %v to Processor %v. Processor busy\", j.ID, pID)\n\t}\n\n\td.pStore.setProcessing(pID, j.ID)\n\n\tgo d.runJob(q, pID, p, j)\n\n\treturn nil\n}\n\n// unassignJob unmarks a job as assigned to #pID\nfunc (d *dispatcher) unassignJob(pID int) {\n\td.dispatchL.Lock()\n\tdefer d.dispatchL.Unlock()\n\n\td.pStore.unsetProcessing(pID)\n}\n\n// runJob runs a job on the corresponding processor and moves it to the right queue depending on results\nfunc (d *dispatcher) runJob(q *queue, pID int, p Processor, j *Job) {\n\tdefer d.processorDone(pID)\n\terr := p.Run(j)\n\tif err != nil {\n\t\tfmt.Printf(\"Processor: %v. Job %v failed with err: %v\\n\", pID, j.ID, err)\n\t\terr := q.markJobDone(j.ID, jobFailed)\n\t\tif err != nil {\n\t\t\tfmt.Printf(\"markJobDone -> %v jobFailed failed: %v\\n\", j.ID, err)\n\t\t}\n\t\treturn\n\t}\n\n\terr = q.markJobDone(j.ID, jobComplete)\n\tif err != nil {\n\t\tfmt.Printf(\"markJobDone -> %v jobComplete failed: %v\\n\", j.ID, err)\n\t}\n}\n\nfunc (d *dispatcher) processorDone(pID int) {\n\td.unassignJob(pID)\n\n\t// signal that the processor might now be available\n\td.signalLoop()\n}\n"
  },
  {
    "path": "pkg/blero/dispatch_test.go",
    "content": "package blero\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"runtime\"\n\t\"sync\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/dgraph-io/badger\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/mock\"\n)\n\ntype noOpProcessor struct{}\n\nfunc (p *noOpProcessor) Run(j *Job) error {\n\treturn nil\n}\n\nfunc TestBlero_RegisterUnregisterProcessor(t *testing.T) {\n\tbl := New(testDBPath)\n\tp1 := &noOpProcessor{}\n\tp2 := &noOpProcessor{}\n\t// test usage of ProcessorFunc\n\tp3 := func(j *Job) error {\n\t\treturn nil\n\t}\n\n\td := bl.dispatcher\n\tpStore := d.pStore\n\n\tpID1 := bl.RegisterProcessor(p1)\n\tpID2 := bl.RegisterProcessor(p2)\n\tpID3 := bl.RegisterProcessorFunc(p3)\n\tassert.Len(t, pStore.processors, 3)\n\tassert.Equal(t, 3, pStore.maxProcessorID)\n\n\tbl.UnregisterProcessor(pID2)\n\tassert.Len(t, pStore.processors, 2)\n\tassert.Equal(t, p1, pStore.processors[pID1])\n\tassert.Equal(t, nil, pStore.processors[pID2])\n\tassert.NotNil(t, pStore.processors[pID3])\n\n\tbl.UnregisterProcessor(pID3)\n\tassert.Len(t, pStore.processors, 1)\n\tassert.Equal(t, p1, pStore.processors[pID1])\n\tassert.Equal(t, nil, pStore.processors[pID2])\n\tassert.Equal(t, nil, pStore.processors[pID3])\n}\n\ntype testProcessor struct {\n\tmock.Mock\n}\n\nfunc (m *testProcessor) Run(j *Job) error {\n\t_ = m.Called(j)\n\n\tvar err error\n\tif j.Name == \"MyOtherOtherJob\" {\n\t\terr = fmt.Errorf(\"%v errors out\", j.Name)\n\t}\n\n\treturn err\n}\n\nfunc TestBlero_assignJobs(t *testing.T) {\n\tbl := New(testDBPath)\n\t// only start the queue and not the dispatch loop to allow manual jobs assignment\n\terr := bl.queue.start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\td := bl.dispatcher\n\tq := bl.queue\n\n\tj1Name := \"MyJob\"\n\tj2Name := \"MyOtherJob\"\n\tj3Name := \"MyOtherOtherJob\"\n\n\tp1 := new(testProcessor)\n\tp2 := new(testProcessor)\n\tp3 := new(testProcessor)\n\n\tfor _, p := range []*testProcessor{p1, p2, p3} {\n\t\tp.On(\"Run\", mock.AnythingOfType(\"*blero.Job\"))\n\t\tbl.RegisterProcessor(p)\n\t}\n\n\t// test assign without jobs\n\terr = d.assignJobs(q)\n\tassert.NoError(t, err)\n\n\t// enqueue jobs\n\tj1ID, err := bl.EnqueueJob(j1Name, nil)\n\tassert.NoError(t, err)\n\tp1.AssertNumberOfCalls(t, \"Run\", 0)\n\tp2.AssertNumberOfCalls(t, \"Run\", 0)\n\tp3.AssertNumberOfCalls(t, \"Run\", 0)\n\n\tj2ID, err := bl.EnqueueJob(j2Name, nil)\n\tassert.NoError(t, err)\n\tp1.AssertNumberOfCalls(t, \"Run\", 0)\n\tp2.AssertNumberOfCalls(t, \"Run\", 0)\n\tp3.AssertNumberOfCalls(t, \"Run\", 0)\n\n\tj3ID, err := bl.EnqueueJob(j3Name, nil)\n\tassert.NoError(t, err)\n\tp1.AssertNumberOfCalls(t, \"Run\", 0)\n\tp2.AssertNumberOfCalls(t, \"Run\", 0)\n\tp3.AssertNumberOfCalls(t, \"Run\", 0)\n\n\terr = d.assignJobs(q)\n\tassert.NoError(t, err)\n\n\t// wait for jobs to be processed\n\ttime.Sleep(50 * time.Millisecond)\n\n\tp1.AssertNumberOfCalls(t, \"Run\", 1)\n\tp2.AssertNumberOfCalls(t, \"Run\", 1)\n\tp3.AssertNumberOfCalls(t, \"Run\", 1)\n\n\terr = q.db.View(func(txn *badger.Txn) error {\n\t\t// check that job 1 is in the complete queue\n\t\t_, err := txn.Get([]byte(\"q:complete:\" + jIDString(j1ID)))\n\t\tassert.NoError(t, err)\n\n\t\t// check that job 2 is in the complete queue\n\t\t_, err = txn.Get([]byte(\"q:complete:\" + jIDString(j2ID)))\n\t\tassert.NoError(t, err)\n\n\t\t// check that job 3 is in the failed queue\n\t\t_, err = txn.Get([]byte(\"q:failed:\" + jIDString(j3ID)))\n\t\tassert.NoError(t, err)\n\n\t\treturn nil\n\t})\n\tassert.NoError(t, err)\n\n\tp1.AssertExpectations(t)\n\tp2.AssertExpectations(t)\n\tp3.AssertExpectations(t)\n}\n\nfunc TestBlero_AutoProcessing_ProcessorFirst(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tvar m sync.Mutex\n\tvar calls []string\n\n\tbl.RegisterProcessor(ProcessorFunc(func(j *Job) error {\n\t\tm.Lock()\n\t\tcalls = append(calls, j.Name)\n\t\tm.Unlock()\n\t\treturn nil\n\t}))\n\n\t// simulate wait period\n\ttime.Sleep(50 * time.Millisecond)\n\n\tj1Name := \"MyJob\"\n\tj2Name := \"MyOtherJob\"\n\n\tbl.EnqueueJob(j1Name, nil)\n\tbl.EnqueueJob(j2Name, nil)\n\n\t// wait for jobs to be processed\n\ttime.Sleep(50 * time.Millisecond)\n\n\tm.Lock()\n\tassert.Len(t, calls, 2)\n\tassert.ElementsMatch(t, []string{j1Name, j2Name}, calls)\n\tm.Unlock()\n}\n\nfunc TestBlero_AutoProcessing_JobsFirst(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tvar m sync.Mutex\n\tvar calls []string\n\n\tj1Name := \"MyJob\"\n\tj2Name := \"MyOtherJob\"\n\n\tbl.EnqueueJob(j1Name, nil)\n\tbl.EnqueueJob(j2Name, nil)\n\n\t// simulate wait period\n\ttime.Sleep(50 * time.Millisecond)\n\n\tbl.RegisterProcessor(ProcessorFunc(func(j *Job) error {\n\t\tm.Lock()\n\t\tcalls = append(calls, j.Name)\n\t\tm.Unlock()\n\t\treturn nil\n\t}))\n\n\t// wait for jobs to be processed\n\ttime.Sleep(50 * time.Millisecond)\n\n\tm.Lock()\n\tassert.Len(t, calls, 2)\n\tassert.ElementsMatch(t, []string{j1Name, j2Name}, calls)\n\tm.Unlock()\n}\n\nfunc TestBlero_AutoProcessing_GoRoutinesHanging(t *testing.T) {\n\tbl := New(testDBPath)\n\t// only start the queue and not the dispatch loop to not run assign and show goroutines hanging problem\n\terr := bl.queue.start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\n\tfor index := 0; index < 300; index++ {\n\t\tbl.EnqueueJob(\"FakeJob\", nil)\n\t}\n\n\tbl.Stop()\n\n\t// simulate wait period\n\ttime.Sleep(50 * time.Millisecond)\n\n\tassert.True(t, runtime.NumGoroutine() < 20)\n}\n\ntype safeBuffer struct {\n\tbytes.Buffer\n\tm sync.Mutex\n}\n\nfunc (b *safeBuffer) Read(p []byte) (n int, err error) {\n\tb.m.Lock()\n\tdefer b.m.Unlock()\n\treturn b.Buffer.Read(p)\n}\n\nfunc (b *safeBuffer) Write(p []byte) (n int, err error) {\n\tb.m.Lock()\n\tdefer b.m.Unlock()\n\treturn b.Buffer.Write(p)\n}\n\nfunc Test_dispatcherAssignFails(t *testing.T) {\n\tstdErr = new(safeBuffer)\n\tpStore := newProcessorsStore()\n\t// introduce error by registering nil processor\n\tpStore.registerProcessor(nil)\n\td := newDispatcher(pStore)\n\n\td.startLoop(nil)\n\n\t// send assign signal\n\td.signalLoop()\n\n\ttime.Sleep(50 * time.Millisecond)\n\terrText, err := ioutil.ReadAll(stdErr)\n\tassert.NoError(t, err)\n\n\tassert.Equal(t, \"Cannot assign jobs: Processor 1 not found\", string(errText))\n}\n"
  },
  {
    "path": "pkg/blero/jobstatus_string.go",
    "content": "// Code generated by \"stringer -type jobStatus .\"; DO NOT EDIT.\n\npackage blero\n\nimport \"strconv\"\n\nconst _jobStatus_name = \"pendinginprogresscompletefailed\"\n\nvar _jobStatus_index = [...]uint8{0, 7, 17, 25, 31}\n\nfunc (i jobStatus) String() string {\n\tif i >= jobStatus(len(_jobStatus_index)-1) {\n\t\treturn \"jobStatus(\" + strconv.FormatInt(int64(i), 10) + \")\"\n\t}\n\treturn _jobStatus_name[_jobStatus_index[i]:_jobStatus_index[i+1]]\n}\n"
  },
  {
    "path": "pkg/blero/jobstatus_string_test.go",
    "content": "package blero\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestJobStatus_String(t *testing.T) {\n\tassert.Equal(t, \"pending\", jobPending.String())\n\tassert.Equal(t, \"inprogress\", jobInProgress.String())\n\tassert.Equal(t, \"complete\", jobComplete.String())\n\tassert.Equal(t, \"failed\", jobFailed.String())\n\n\t// unknown\n\tassert.Equal(t, \"jobStatus(50)\", jobStatus(50).String())\n}\n"
  },
  {
    "path": "pkg/blero/processing.go",
    "content": "package blero\n\n// Job represents a Goblero job definition\ntype Job struct {\n\tID   uint64\n\tName string\n\tData []byte\n}\n\n// Processor interface\ntype Processor interface {\n\tRun(j *Job) error\n}\n\n// ProcessorFunc is a processor function\ntype ProcessorFunc func(j *Job) error\n\n// Run allows using ProcessorFunc as a Processor\nfunc (pf ProcessorFunc) Run(j *Job) error {\n\treturn pf(j)\n}\n\n// processorsStore struct\ntype processorsStore struct {\n\tmaxProcessorID int\n\tprocessors     map[int]Processor\n\tprocessing     map[int]uint64\n}\n\n// newProcessorsStore creates a new ProcessorsStore\nfunc newProcessorsStore() *processorsStore {\n\tpStore := &processorsStore{}\n\tpStore.processors = make(map[int]Processor)\n\tpStore.processing = make(map[int]uint64)\n\treturn pStore\n}\n\n// registerProcessor registers a new processor\nfunc (pStore *processorsStore) registerProcessor(p Processor) int {\n\tpStore.maxProcessorID++\n\tpStore.processors[pStore.maxProcessorID] = p\n\n\treturn pStore.maxProcessorID\n}\n\n// unregisterProcessor unregisters a processor\nfunc (pStore *processorsStore) unregisterProcessor(pID int) {\n\tdelete(pStore.processors, pID)\n}\n\n// getAvailableProcessorsIDs returns the currently free processors\nfunc (pStore *processorsStore) getAvailableProcessorsIDs() []int {\n\tvar pIDs []int\n\tfor pID := range pStore.processors {\n\t\tif _, ok := pStore.processing[pID]; !ok {\n\t\t\tpIDs = append(pIDs, pID)\n\t\t}\n\t}\n\treturn pIDs\n}\n\n// getProcessor fetches processors by ID\nfunc (pStore *processorsStore) getProcessor(pID int) Processor {\n\treturn pStore.processors[pID]\n}\n\n// isProcessorBusy checks if a processor is already working on a job\nfunc (pStore *processorsStore) isProcessorBusy(pID int) bool {\n\t_, ok := pStore.processing[pID]\n\treturn ok\n}\n\n// setProcessing sets a processor as working on a job\nfunc (pStore *processorsStore) setProcessing(pID int, jID uint64) {\n\tpStore.processing[pID] = jID\n}\n\n// unsetProcessing unsets a processor as working on a job\nfunc (pStore *processorsStore) unsetProcessing(pID int) {\n\tdelete(pStore.processing, pID)\n}\n"
  },
  {
    "path": "pkg/blero/queue.go",
    "content": "package blero\n\nimport (\n\t\"bytes\"\n\t\"encoding/gob\"\n\t\"errors\"\n\t\"fmt\"\n\t\"os\"\n\t\"sync\"\n\n\t\"github.com/dgraph-io/badger\"\n)\n\n// queueOpts struct\ntype queueOpts struct {\n\tDBPath string\n\tLogger badger.Logger\n}\n\n// queue struct\ntype queue struct {\n\topts queueOpts\n\tdb   *badger.DB\n\tseq  *badger.Sequence\n\tdbL  sync.Mutex\n}\n\n// newQueue creates new ueue\nfunc newQueue(opts queueOpts) *queue {\n\tq := &queue{opts: opts}\n\treturn q\n}\n\ntype badgerLogger struct{}\n\nfunc (l *badgerLogger) Infof(format string, a ...interface{}) {\n\t// ignore badger info logs\n}\nfunc (l *badgerLogger) Errorf(format string, a ...interface{}) {\n\tfmt.Fprintf(os.Stderr, format, a...)\n}\nfunc (l *badgerLogger) Warningf(format string, a ...interface{}) {\n\tfmt.Fprintf(os.Stderr, format, a...)\n}\n\nfunc (l *badgerLogger) Debugf(format string, a ...interface{}) {\n\t// Leave blank for the time being,\n\t// track it in the future.\n}\n\n// start Queue\nfunc (q *queue) start() error {\n\t// validate opts\n\tif q.opts.DBPath == \"\" {\n\t\treturn errors.New(\"DBPath is required\")\n\t}\n\n\tx := badger.Options{}\n\n\tx.Dir = \"ds\"\n\n\t// open db\n\tbadgerOpts := badger.DefaultOptions(q.opts.DBPath)\n\tbadgerOpts.Logger = &badgerLogger{}\n\tbadgerOpts.SyncWrites = true\n\n\tdb, err := badger.Open(badgerOpts)\n\tif err != nil {\n\t\treturn err\n\t}\n\tq.db = db\n\n\t// init sequence\n\tq.seq, err = db.GetSequence([]byte(\"standard\"), 1000)\n\treturn err\n}\n\n// stop Queue and Release resources\nfunc (q *queue) stop() error {\n\t// release sequence\n\terr := q.seq.Release()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// close db\n\terr = q.db.Close()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc getNextSeq(seq *badger.Sequence) (num uint64, err error) {\n\tdefer func() {\n\t\tr := recover()\n\t\tif r != nil {\n\t\t\t// recover from panic and send err instead\n\t\t\terr = r.(error)\n\t\t}\n\t}()\n\n\tnum, err = seq.Next()\n\treturn num, err\n}\n\n// enqueueJob enqueues a new Job to the Pending queue\nfunc (q *queue) enqueueJob(name string, data []byte) (uint64, error) {\n\tnum, err := getNextSeq(q.seq)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\tj := &Job{ID: num + 1, Name: name, Data: data}\n\tjKey := getJobKey(jobPending, j.ID)\n\n\terr = q.db.Update(func(txn *badger.Txn) error {\n\t\tb, err := encodeJob(j)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\terr = txn.Set([]byte(jKey), b)\n\n\t\treturn err\n\t})\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\n\treturn j.ID, nil\n}\n\nfunc encodeJob(j *Job) ([]byte, error) {\n\tvar b bytes.Buffer\n\terr := gob.NewEncoder(&b).Encode(j)\n\treturn b.Bytes(), err\n}\n\n// jobStatus Enum Type\ntype jobStatus uint8\n\nconst (\n\t// jobPending : waiting to be processed\n\tjobPending jobStatus = iota\n\t// jobInProgress : processing in progress\n\tjobInProgress\n\t// jobComplete : processing complete\n\tjobComplete\n\t// jobFailed : processing errored out\n\tjobFailed\n)\n\nfunc getQueueKeyPrefix(status jobStatus) string {\n\treturn fmt.Sprintf(\"q:%v:\", status)\n}\n\nfunc getJobKey(status jobStatus, jID uint64) string {\n\treturn getQueueKeyPrefix(status) + jIDString(jID)\n}\n\nfunc jIDString(jID uint64) string {\n\treturn fmt.Sprintf(\"%020d\", jID)\n}\n\n// dequeueJob moves the next pending job from the pending status to inprogress\nfunc (q *queue) dequeueJob() (*Job, error) {\n\tvar j *Job\n\n\tq.dbL.Lock()\n\tdefer q.dbL.Unlock()\n\terr := q.db.Update(func(txn *badger.Txn) error {\n\t\tprefix := []byte(getQueueKeyPrefix(jobPending))\n\t\tk, v, err := getFirstKVForPrefix(txn, prefix)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\t// iteration is done, no job was found\n\t\tif k == nil {\n\t\t\treturn nil\n\t\t}\n\n\t\tj, err = decodeJob(v)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\t// Move from from Pending queue to InProgress queue\n\t\terr = moveItem(txn, k, []byte(getJobKey(jobInProgress, j.ID)), v)\n\n\t\treturn err\n\t})\n\n\treturn j, err\n}\n\nfunc getFirstKVForPrefix(txn *badger.Txn, prefix []byte) ([]byte, []byte, error) {\n\titOpts := badger.DefaultIteratorOptions\n\titOpts.PrefetchValues = true\n\titOpts.PrefetchSize = 1\n\tit := txn.NewIterator(itOpts)\n\n\t// go to smallest key after prefix\n\tit.Seek(prefix)\n\tdefer it.Close()\n\t// iteration done, no item found\n\tif !it.ValidForPrefix(prefix) {\n\t\treturn nil, nil, nil\n\t}\n\n\titem := it.Item()\n\n\tk := item.KeyCopy(nil)\n\n\tv, err := item.ValueCopy(nil)\n\treturn k, v, err\n}\n\n// markJobDone moves a job from the inprogress status to complete/failed\nfunc (q *queue) markJobDone(id uint64, status jobStatus) error {\n\tif status != jobComplete && status != jobFailed {\n\t\treturn errors.New(\"Can only move to Complete or Failed Status\")\n\t}\n\n\tkey := []byte(getJobKey(jobInProgress, id))\n\n\tq.dbL.Lock()\n\tdefer q.dbL.Unlock()\n\terr := q.db.Update(func(txn *badger.Txn) error {\n\t\tb, err := getBytesForKey(txn, key)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\t// Move from from InProgress queue to dest queue\n\t\terr = moveItem(txn, key, []byte(getJobKey(status, id)), b)\n\n\t\treturn err\n\t})\n\n\treturn err\n}\n\nfunc decodeJob(b []byte) (*Job, error) {\n\tvar j *Job\n\terr := gob.NewDecoder(bytes.NewBuffer(b)).Decode(&j)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn j, nil\n}\n\nfunc getJobForKey(txn *badger.Txn, key []byte) (*Job, error) {\n\tb, err := getBytesForKey(txn, key)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tj, err := decodeJob(b)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn j, nil\n}\n\nfunc getBytesForKey(txn *badger.Txn, key []byte) ([]byte, error) {\n\titem, err := txn.Get(key)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tb, err := item.ValueCopy(nil)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn b, nil\n}\n\nfunc moveItem(txn *badger.Txn, oldKey []byte, newKey []byte, b []byte) error {\n\t// remove from Source queue\n\terr := txn.Delete(oldKey)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// create in Dest queue\n\terr = txn.Set(newKey, b)\n\treturn err\n}\n"
  },
  {
    "path": "pkg/blero/queue_test.go",
    "content": "package blero\n\nimport (\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/dgraph-io/badger\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nconst testDBPath = \"../../db/test\"\n\nfunc deleteDBFolder(dbPath string) {\n\t// prevent accidental deletion of non badgerdb folder\n\tif _, err := os.Stat(filepath.Join(dbPath, \"MANIFEST\")); os.IsNotExist(err) {\n\t\tpanic(\"Attempted to delete non badgerdb folder \" + dbPath)\n\t}\n\n\terr := os.RemoveAll(dbPath)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n}\n\n/*\nfunc TestBlero_StopQueueAlreadyStopped(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// delete folder\n\tdefer deleteDBFolder(testDBPath)\n\tbl.Stop()\n\n\terr = bl.queue.stop()\n\tassert.EqualError(t, err, \"Writes are blocked, possibly due to DropAll or Close\")\n}*/\n\nfunc TestBlero_BadgerLogger(t *testing.T) {\n\tlogger := &badgerLogger{}\n\t// test logger\n\tlogger.Infof(\"[badgerLogger]TEST Infof\")\n\tlogger.Warningf(\"[badgerLogger]TEST Warningf\")\n\tlogger.Errorf(\"[badgerLogger]TEST Errorf\")\n}\n\nfunc TestBlero_EnqueueJob(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\tq := bl.queue\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tjName := \"TestJob\"\n\tjData := []byte(\"TestJob Args\")\n\n\tjID, err := bl.EnqueueJob(jName, jData)\n\tassert.NoError(t, err)\n\n\tassert.Equal(t, uint64(1), jID)\n\n\tvar j *Job\n\terr = q.db.View(func(txn *badger.Txn) error {\n\t\tj, err = getJobForKey(txn, []byte(\"q:pending:\"+jIDString(jID)))\n\t\tassert.NoError(t, err)\n\n\t\treturn nil\n\t})\n\tassert.NoError(t, err)\n\n\tassert.Equal(t, jID, j.ID)\n\tassert.Equal(t, jName, j.Name)\n\tassert.Equal(t, jData, j.Data)\n}\n\nfunc TestBlero_EnqueueJob_Concurrent(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tch := make(chan uint64)\n\tgo func() {\n\t\tid, err := bl.EnqueueJob(\"TestJob\", nil)\n\t\tassert.NoError(t, err)\n\n\t\tch <- id\n\t}()\n\n\tgo func() {\n\t\tid, err := bl.EnqueueJob(\"TestJob\", nil)\n\t\tassert.NoError(t, err)\n\n\t\tch <- id\n\t}()\n\n\tid1 := <-ch\n\tid2 := <-ch\n\n\tassert.ElementsMatch(t, []uint64{1, 2}, []uint64{id1, id2})\n}\n\nfunc TestBlero_EnqueueJobQueueStopped(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// delete folder\n\tdefer deleteDBFolder(testDBPath)\n\tbl.Stop()\n\n\t_, err = bl.EnqueueJob(\"TestJob\", nil)\n\tassert.EqualError(t, err, \"runtime error: invalid memory address or nil pointer dereference\")\n}\n\nfunc TestBlero_DequeueJob(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tq := bl.queue\n\n\tj1Name := \"TestJob\"\n\tj1ID, err := bl.EnqueueJob(j1Name, nil)\n\tassert.NoError(t, err)\n\n\tj2Name := \"TestJob\"\n\tj2ID, err := bl.EnqueueJob(j2Name, nil)\n\tassert.NoError(t, err)\n\n\tj, err := q.dequeueJob()\n\tassert.NoError(t, err)\n\n\tassert.Equal(t, j1ID, j.ID)\n\tassert.Equal(t, j1Name, j.Name)\n\n\terr = q.db.View(func(txn *badger.Txn) error {\n\t\t// check that job 1 is not in the pending queue anymore\n\t\t_, err := txn.Get([]byte(\"q:pending:\" + jIDString(j1ID)))\n\t\tassert.EqualError(t, err, badger.ErrKeyNotFound.Error())\n\n\t\t// check that job 2 is still in the pending queue\n\t\t_, err = txn.Get([]byte(\"q:pending:\" + jIDString(j2ID)))\n\t\tassert.NoError(t, err)\n\n\t\t// get job 1 from inprogress queue\n\t\tcompleteJob, err := getJobForKey(txn, []byte(\"q:inprogress:\"+jIDString(j1ID)))\n\t\tassert.NoError(t, err)\n\n\t\tassert.Equal(t, j1ID, completeJob.ID)\n\t\tassert.Equal(t, j1Name, completeJob.Name)\n\t\treturn nil\n\t})\n\tassert.NoError(t, err)\n}\n\nfunc TestBlero_DequeueJob_Concurrent(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tq := bl.queue\n\n\tj1Name := \"TestJob\"\n\tj1ID, err := bl.EnqueueJob(j1Name, nil)\n\tassert.NoError(t, err)\n\n\tj2Name := \"TestJob\"\n\tj2ID, err := bl.EnqueueJob(j2Name, nil)\n\tassert.NoError(t, err)\n\n\tch := make(chan *Job)\n\n\tgo func() {\n\t\tj, err := q.dequeueJob()\n\t\tassert.NoError(t, err)\n\t\tch <- j\n\t}()\n\n\tgo func() {\n\t\tj, err := q.dequeueJob()\n\t\tassert.NoError(t, err)\n\t\tch <- j\n\t}()\n\n\tj1 := <-ch\n\tj2 := <-ch\n\n\tjobs := []*Job{j1, j2}\n\tsort.Slice(jobs, func(i, j int) bool {\n\t\treturn jobs[i].ID < jobs[j].ID\n\t})\n\n\tassert.Equal(t, jobs[0].ID, j1ID)\n\tassert.Equal(t, jobs[0].Name, j1Name)\n\n\tassert.Equal(t, jobs[1].ID, j2ID)\n\tassert.Equal(t, jobs[1].Name, j2Name)\n}\n\nfunc TestBlero_MarkJobDone(t *testing.T) {\n\tbl := New(testDBPath)\n\terr := bl.Start()\n\tassert.NoError(t, err)\n\n\t// stop gracefully\n\tdefer deleteDBFolder(testDBPath)\n\tdefer bl.Stop()\n\n\tq := bl.queue\n\n\tj1Name := \"TestJob\"\n\tj1ID, err := bl.EnqueueJob(j1Name, nil)\n\tassert.NoError(t, err)\n\n\tj2Name := \"TestJob\"\n\tj2ID, err := bl.EnqueueJob(j2Name, nil)\n\tassert.NoError(t, err)\n\n\t// move job 1 to inprogress\n\t_, err = q.dequeueJob()\n\tassert.NoError(t, err)\n\t// move job 2 to inprogress\n\t_, err = q.dequeueJob()\n\tassert.NoError(t, err)\n\n\terr = q.markJobDone(j1ID, jobComplete)\n\tassert.NoError(t, err)\n\n\terr = q.markJobDone(j2ID, jobFailed)\n\tassert.NoError(t, err)\n\n\terr = q.db.View(func(txn *badger.Txn) error {\n\t\t// check that job 1 is not in the inprogress queue anymore\n\t\t_, err := txn.Get([]byte(\"q:inprogress:\" + jIDString(j1ID)))\n\t\tassert.EqualError(t, err, badger.ErrKeyNotFound.Error())\n\n\t\t// check that job 2 is not in the inprogress queue anymore\n\t\t_, err = txn.Get([]byte(\"q:inprogress:\" + jIDString(j2ID)))\n\t\tassert.EqualError(t, err, badger.ErrKeyNotFound.Error())\n\n\t\t// check that job 1 is now in the complete queue\n\t\tcompleteJob, err := getJobForKey(txn, []byte(\"q:complete:\"+jIDString(j1ID)))\n\t\tassert.NoError(t, err)\n\n\t\tassert.Equal(t, j1ID, completeJob.ID)\n\t\tassert.Equal(t, j1Name, completeJob.Name)\n\n\t\tfailedJob, err := getJobForKey(txn, []byte(\"q:failed:\"+jIDString(j2ID)))\n\t\tassert.NoError(t, err)\n\n\t\tassert.Equal(t, j2ID, failedJob.ID)\n\t\tassert.Equal(t, j2Name, failedJob.Name)\n\n\t\treturn nil\n\t})\n\tassert.NoError(t, err)\n\n\t// check random job id is not in queue error\n\terr = q.markJobDone(uint64(4151231), jobComplete)\n\tassert.EqualError(t, err, \"Key not found\")\n\n\t// check moving job to pending error\n\terr = q.markJobDone(j2ID, jobPending)\n\tassert.EqualError(t, err, \"Can only move to Complete or Failed Status\")\n}\n\nfunc TestBlero_moveItemErr(t *testing.T) {\n\ttxn := &badger.Txn{}\n\terr := moveItem(txn, nil, nil, nil)\n\tassert.EqualError(t, err, \"No sets or deletes are allowed in a read-only transaction\")\n}\n\nfunc TestBlero_decodeJobErr(t *testing.T) {\n\t_, err := decodeJob(nil)\n\tassert.EqualError(t, err, \"EOF\")\n}\n"
  }
]