Showing preview only (250K chars total). Download the full file or copy to clipboard to get everything.
Repository: Automattic/kue
Branch: master
Commit: c5647b1a8890
Files: 61
Total size: 234.6 KB
Directory structure:
gitextract_wlr7h9ez/
├── .gitignore
├── .npmignore
├── .travis.yml
├── History.md
├── LICENSE
├── Makefile
├── Readme.md
├── bin/
│ └── kue-dashboard
├── examples/
│ ├── delayed.js
│ ├── events.js
│ ├── many.js
│ ├── shutdown.js
│ ├── stale.js
│ └── video.js
├── index.js
├── lib/
│ ├── http/
│ │ ├── index.js
│ │ ├── middleware/
│ │ │ └── provides.js
│ │ ├── public/
│ │ │ ├── javascripts/
│ │ │ │ ├── caustic.js
│ │ │ │ ├── job.js
│ │ │ │ ├── jquery.ext.js
│ │ │ │ ├── loading.js
│ │ │ │ ├── main.js
│ │ │ │ ├── progress.js
│ │ │ │ ├── search.js
│ │ │ │ └── utils.js
│ │ │ └── stylesheets/
│ │ │ ├── actions.styl
│ │ │ ├── config.styl
│ │ │ ├── context-menu.styl
│ │ │ ├── error.styl
│ │ │ ├── job.styl
│ │ │ ├── main.css
│ │ │ ├── main.styl
│ │ │ ├── menu.styl
│ │ │ ├── mixins.styl
│ │ │ └── scrollbar.styl
│ │ ├── routes/
│ │ │ ├── index.js
│ │ │ └── json.js
│ │ └── views/
│ │ ├── _filter.pug
│ │ ├── _job.pug
│ │ ├── _menu.pug
│ │ ├── _row.pug
│ │ ├── _search.pug
│ │ ├── _sort.pug
│ │ ├── job/
│ │ │ └── list.pug
│ │ └── layout.pug
│ ├── kue.js
│ ├── queue/
│ │ ├── events.js
│ │ ├── job.js
│ │ ├── test_mode.js
│ │ └── worker.js
│ └── redis.js
├── package.json
└── test/
├── jsonapi.js
├── mocha.opts
├── prefix.coffee
├── shutdown.coffee
├── tdd/
│ ├── kue.spec.js
│ └── redis.spec.js
├── test.coffee
├── test.js
└── test_mode.js
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.idea
.DS_Store
node_modules
*.sock
*.rdb
test/incomplete
*.swp
================================================
FILE: .npmignore
================================================
support
test
examples
*.sock
================================================
FILE: .travis.yml
================================================
language: node_js
node_js:
- "4"
- "6"
- "8"
- "node"
services:
- redis-server
================================================
FILE: History.md
================================================
0.11.5 / 2016-11-05
===================
* Fix even more redis command callbacks
* Fix redis commands SLC integration #978
0.11.4 / 2016-10-21
===================
* adding reds module to optional dependencies
0.11.3 / 2016-10-21
===================
* Fix making reds module optional, #969
0.11.2 / 2016-10-14
===================
* Update packages to remove CVEs, #932
* Make reds an optional dependency, #922
* Remove unnecessary dependency to lodash-deep, #921
* Expose shutdown in process worker ctx, #912
* Add ioredis support to watchStuckJobs, #884
0.11.1 / 2016-06-15
===================
* Upgrade redis to 2.6
* Add switch for each job event
0.11.0 / 2016-05-13
===================
* force node_redis version to 2.4.x, Closes #857
* Converting Job ids back into integers, #855
* Fix LPUSH crash during shutdown, #854
* Install kue-dashboard script, #853
* Add start event to documentation, #841
* Add parameter for testMode.enter to continue processing jobs, #821
* Modern Node.js versions support, #812
* Don't start the next job until the current one is totally finished, Closes #806
* Store multiple instances of jobs in jobs id map to emit events for all, #750
0.10.6 / 2016-04-27
===================
* Redis Cluster fix, Closes #861
0.10.5 / 2016-01-14
===================
* Attempts surpassing max attempts on delay jobs upon failure, resulting in infinite retries, Fixes #797
* Add yargs dependency for kue-dashboard, #796
0.10.4 / 2016-01-14
===================
* fix zpop callback on shutdown
* fix connection_options in test.js
* Unit tests for redis.js #779
* Tests for kue.js #778
0.10.3 / 2015-11-20
===================
* Fixing Job processing order without munging the job id, Closes #708, Closes #678
0.10.2 / 2015-11-20
===================
* Add support for ioredis, Closes #652
* Add support for Redis Cluster, Closes #642
* Fix `this.state` on refreshTTL
0.10.0 / 2015-11-20
===================
* Update TTL on job progress, Closes #694
* Upgrade to node_redis 2.3, #717
* Fix LPUSH vs connection quit race when shutting down
* Restart task btn, #754
* Fix uncaught exception in job.js, #751
* Added kue-dashboard script for conveniently running the dashboard #611
* Fixed invalid CSS on production, #755
* Connection string not supporting DB number #725
* Fix attempts remaining logic, #742
* Update jade, #741
* Properly set job IDs in test mode, #727
* Enhanced Job.log formatting, #630
* Use node's util#format() in Job.log, #724
0.9.6 / 2015-10-06
===================
* Fix redirection issue
0.9.5 / 2015-09-16
===================
* When no ttl is set for jobs, don't let high priorities to conflict, fixes #697
* Fix redirection issue, closes #685
* Get progress_data along with other redis fields, PR #642
* Grab only password from Redis URL, fixes #681
* Add remove job event, PR #665
0.9.4 / 2015-07-17
===================
* Job that doesn't call done() retries twice, fixes #669
0.9.3 / 2015-05-07
===================
* Fix unlocking promotion lock, Closes #608
0.9.2 / 2015-05-07
===================
* Fix duplicate job promotion/ttl race, Closes #601
0.9.1 / 2015-05-05
===================
* Filter only jobs that have ttl set, Fixes #590
0.9.0 / 2015-05-02
===================
* Upgrade to express 4.x, Closes #537
* Move `job.reprocess` done callback to the last, Closes #387, Closes #385
* Standardize signature of `.shutdown()` callback, Closes #454
* Turn off search indexes by default, Closes #412
* Improve delayed job promotion feature, Closes #533, fixes #312, closes #352
* Use a distributed redis lock to hide job promotion from user, Closes #556
* Deprecate `.promote` and update documentation
* Document Javascript API to query queue state, Closes #455
* Add jobEvents flag to switch off job events for memory optimization, Closes #401
* Add idle event to capture unsuccessful zpop's in between of worker get Job, should fix #538
* Add TTL for active jobs, Closes #544
* Document `jobEvents` queue config, Closes #557
* Bulk job create API now processes all jobs in case of intermediate errors, Closes #552
* Merge `red job remove buttons and tooltips` PR, Closes #566
* Add a in-memory test Kue mode, Closes #561
* Update reds package to `0.2.5`
* Merge PR #594, bad redirect URL in old express versions, fixes #592
* update dependency to forked warlock repo to fix redis connection cleanup on shutdown, fixes #578
* Update job hash with the worker ID, Closes #580
0.8.12 / 2015-03-22
===================
* Bulk job create JSON API, Closes #334, Closes #500, Closes #527
* Add feature to specify redis connection string/url, Closes #540
* Mention kue-ui in readme, Closes #502
* Add an extra parameter to the progress method to notify extra contextual data, Closes #466, Closes #427, Closes #313
* Document job event callback arguments, Closes #542
* Fix typo in documentation, Closes #506
* Document importance of using Kue `error` listeners, Closes #409
* Document Queue maintenance and job.removeOnComplete( true ), Closes #439
* Document how to query all the active jobs programmatically, Closes #418
* Document to explain how "stuck queued jobs" happens, Closes #451
* Document on proper error handling to prevent stuck jobs, Closes #391
0.8.11 / 2014-12-15
===================
* Fix shutdown on re-attemptable jobs, Closes #469
* Fix race condition in delaying jobs when re-attempts, Closes #483
* Make `watchStuckJobs` aware of queue prefix, Closes #452
* Send along error message when emitting a failed event, Closes #461
0.8.10 / 2014-12-13
===================
* Add more tests, Closes #280
* More atomic job state changes, Closes #411
* Documentation: error passed to done should be string or standard JS error object, Closes #394
* Documentation: backoff documentation, Closes #435
* Documentation: correct `promote` usage, Closes #413
* Add job enqueue event, Closes #458
* Watch for errors with non-string err.stack, Closes #426
* Fix web app redirect path for express 4.0, Closes #393
* `removeBadJob` should do pessimistic job removal from all state ZSETs, Closes #438
* Add stats json api by type and state, Closes #477
* Don't let concurrent graceful shutdowns on subsequent`Queue#shutdown`calls, Closes #479
* Fix `cleanup` global leak, Closes #475
0.8.9 / 2014-10-01
==================
* Properly update status flags on resume, Closes #423
0.8.8 / 2014-09-12
==================
* Fix tests to limited shutdown timeouts
* Add a redis lua watchdog to fix stuck inactive jobs, fixes #130
* Stuck inactive jobs watchdog, Closes #130
0.8.7 / 2014-09-12
==================
* Shutdown timeout problems and races, fixes #406
0.8.6 / 2014-08-30
==================
* Quit redis connections on shutdown & let the process exit, closes #398
0.8.5 / 2014-08-10
==================
* Fix typo in removeOnComplete
0.8.4 / 2014-08-08
==================
* Emit event 'job failed attempt' after job successfully updated, closes #377
* Fix delaying jobs when failed, closes #384
* Implement `job.removeOnComplete`, closes #383
* Make searchKeys chainable, closes #379
* Add extra job options to JSON API, closes #378
0.8.3 / 2014-07-13
==================
* Inject other Redis clients compatible with node_redis #344
* Add support to connect to Redis using Linux sockets #362
* Add .save callback sample code in documentation #367
0.8.2 / 2014-07-08
==================
* Fix broken failure backoff #360
* Merge web console redirection fix #357
* Add db selection option to redis configuration #354
* Get number of jobs with given state and type #349
* Add Queue.prototype.delayed function #351
0.8.1 / 2014-06-13
==================
* Fix wrong parameter orders in complete event #343s
* Graceful shutdown bug fix #328
0.8.0 / 2014-06-11
==================
* Implement backoff on failure retries #300
* Allow passing back worker results via done to event handlers #170
* Allow job producer to specify which keys of `job.data` to be indexed for search #284
* Waffle.io Badge #332
* Dropping monkey-patch style redis client connections
* Update docs: Worker Pause/Resume-ability
* Update docs: Reliability of Queue event handlers over Job event handlers
0.7.9 / 2014-06-01
==================
* Graceful shutdown bug fix #336
* More robust graceful shutdown under heavy load #328
0.7.6 / 2014-05-02
==================
* Fixed broken monkey-patch style redis connections #323
0.7.0 / 2014-01-24
==================
* Suppress "undefined" messages on String errors. Closes #230
* Fix cannot read property id of undefined errors. Closes #252
* Parameterize limit of jobs checked in promotion cycles. Closes #244
* Graceful shutdown
* Worker pause/resume ability, Closes #163
* Ensure event subscription before job save. Closes #179
* Fix Queue singleton
* Fix failed event being called in first attempt. Closes #142
* Disable search (Search index memory leaks). See #58 & #218
* Emit error events on both kue and job
* JS/Coffeescript tests added (Mocha+Should)
* Travis support added
0.6.2 / 2013-04-03
==================
* Fix redirection to active for mounted apps
0.6.1 / 2013-03-25
==================
* Fixed issue preventing polling for new jobs. Closes #192
0.6.0 / 2013-03-20
==================
* Make pollForJobs actually use ms argument. Closes #158
* Support delay over HTTP POST. Closes #165
* Fix natural sorting. Closes #174
* Update `updated_at` timestamp during `log`, `progress`, `attempt`, or `state` changes. Closes #188
* Fix redirection to /active. Closes #190
0.5.0 / 2012-11-16
==================
* add POST /job to create a job
* fix /job/search hang
0.4.2 / 2012-11-08
==================
* Revert "Fix delay() not really delaying"
* Revert "If a job with a delay has more attempts, honor the original delay"
0.4.1 / 2012-09-25
==================
* fix: if a job with a delay has more attempts, honor the original delay [mathrawka]
0.4.0 / 2012-06-28
==================
* Added 0.8.0 support
0.3.4 / 2012-02-23
==================
* Changed: reduce polling by using BLPOP to notify workers of activity [Davide Bertola]
0.3.3 / 2011-11-28
==================
* Fixed: use relative stats route to support mounting [alexkwolfe]
* Fixed 0.6.x support
* Removed empty Makefile
0.3.2 / 2011-10-04
==================
* Removed unnecessary "pooling"
* Fixed multiple event emitting. Closes #73
* Fixed menu styling
0.3.1 / 2011-08-25
==================
* Fixed auto event subscription. Closes #68
* Changed: one redis connection for all workers
* Removed user-select: none from everything. Closes #50
0.3.0 / 2011-08-11
==================
* Added search capabilities
* Added `workTime` stat
* Added removal of stale jobs example
* Added Queue-level job events, useful for removing stale jobs etc. Closes * Changed: lazy load reds search [David Wood]
* Fixed `Job#error` for modules that throw strings or emit `error` events with strings [guillermo] #51
* Fixed `Job#remove(fn)`
* Fixed proxy issue with paths, use relative paths [booo]
0.2.0 / 2011-07-25
==================
* Added infinite scroll
* Added delayed job support
* Added configurable redis support [davidwood]
* Added job windowing. Closes #28
* Added `Job#delay(ms)`
* Removed job scrollIntoView
* Removed fancy scrollbar (for infinite scroll / windowing :( )
* Removed "More" button
* Fixed z-index for actions
* Fixed job mapping. Closes #43
0.1.0 / 2011-07-19
==================
* Added exposing of progress via redis pubsub
* Added pubsub job events "complete" and "failed"
* Fixed: capping of progress > 100 == 100
* UI: scroll details into view
0.0.3 / 2011-07-07
==================
* Added caustic to aid in template management
* Added job attempt support. Closes #31
* Added `Job.attempts(n)`
* Added minified jQuery
* Added cluster integration docs. Closes #13
* Added GET _/jobs/:from..:to_ to JSON API
* Fixed: hide "More" on sort
* Fixed: hide "More" on filter
* Fixed: removed "error" emission, blows up when no one is listening
0.0.2 / 2011-07-05
==================
* Added support to update state from UI. Closes #26
* Added support to alter priority in UI. Closes #25
* Added filtering by type. Closes #20
0.0.1 / 2011-07-04
==================
* Initial release
================================================
FILE: LICENSE
================================================
The MIT License
Copyright (c) 2011 LearnBoost <tj@learnboost.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: Makefile
================================================
REPORTER = spec
all: build
build:
@./node_modules/coffee-script/bin/coffee \
-c \
-o lib src
test-tdd:
@./node_modules/.bin/mocha \
--reporter $(REPORTER) \
--require should \
--require sinon \
--ui tdd \
test/tdd/*.js
test-bdd:
@./node_modules/.bin/mocha \
--reporter $(REPORTER) \
--require should \
--ui bdd \
test/*.js
test-bdd-coffee:
@./node_modules/.bin/mocha \
--compilers coffee:coffee-script \
--reporter $(REPORTER) \
--require should \
--require coffee-script/register \
--ui bdd \
test/*.coffee
test-all: test-tdd test-bdd test-bdd-coffee
.PHONY: test-all
================================================
FILE: Readme.md
================================================
# Kue
## Kue is no longer maintained
Please see e.g. [Bull](https://github.com/OptimalBits/bull) as an alternative. Thank you!
[](https://travis-ci.org/Automattic/kue)
[](http://badge.fury.io/js/kue)
[](https://david-dm.org/Automattic/kue)
[](https://gitter.im/Automattic/kue?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
Kue is a priority job queue backed by [redis](http://redis.io), built for [node.js](http://nodejs.org).
**PROTIP** This is the latest Kue documentation, make sure to also read the [changelist](History.md).
## Upgrade Notes (Please Read)
- [0.9 -> 0.10](https://github.com/Automattic/kue/wiki/Upgrading-to-0.10.x)
- [0.8 -> 0.9](https://github.com/Automattic/kue/wiki/Upgrading-to-0.9.x)
## Installation
- Latest release:
$ npm install kue
- Master branch:
$ npm install http://github.com/Automattic/kue/tarball/master
[](https://nodei.co/npm/kue/)
## Features
- Delayed jobs
- Distribution of parallel work load
- Job event and progress pubsub
- Job TTL
- Optional retries with backoff
- Graceful workers shutdown
- Full-text search capabilities
- RESTful JSON API
- Rich integrated UI
- Infinite scrolling
- UI progress indication
- Job specific logging
- Powered by Redis
## Overview
- [Creating Jobs](#creating-jobs)
- [Jobs Priority](#job-priority)
- [Failure Attempts](#failure-attempts)
- [Failure Backoff](#failure-backoff)
- [Job TTL](#job-ttl)
- [Job Logs](#job-logs)
- [Job Progress](#job-progress)
- [Job Events](#job-events)
- [Queue Events](#queue-events)
- [Delayed Jobs](#delayed-jobs)
- [Processing Jobs](#processing-jobs)
- [Processing Concurrency](#processing-concurrency)
- [Pause Processing](#pause-processing)
- [Updating Progress](#updating-progress)
- [Graceful Shutdown](#graceful-shutdown)
- [Error Handling](#error-handling)
- [Queue Maintenance](#queue-maintenance)
- [Redis Connection Settings](#redis-connection-settings)
- [User-Interface](#user-interface)
- [JSON API](#json-api)
- [Parallel Processing With Cluster](#parallel-processing-with-cluster)
- [Securing Kue](#securing-kue)
- [Testing](#testing)
- [Screencasts](#screencasts)
- [License](#license)
## Creating Jobs
First create a job `Queue` with `kue.createQueue()`:
```js
var kue = require('kue')
, queue = kue.createQueue();
```
Calling `queue.create()` with the type of job ("email"), and arbitrary job data will return a `Job`, which can then be `save()`ed, adding it to redis, with a default priority level of "normal". The `save()` method optionally accepts a callback, responding with an `error` if something goes wrong. The `title` key is special-cased, and will display in the job listings within the UI, making it easier to find a specific job.
```js
var job = queue.create('email', {
title: 'welcome email for tj'
, to: 'tj@learnboost.com'
, template: 'welcome-email'
}).save( function(err){
if( !err ) console.log( job.id );
});
```
### Job Priority
To specify the priority of a job, simply invoke the `priority()` method with a number, or priority name, which is mapped to a number.
```js
queue.create('email', {
title: 'welcome email for tj'
, to: 'tj@learnboost.com'
, template: 'welcome-email'
}).priority('high').save();
```
The default priority map is as follows:
```js
{
low: 10
, normal: 0
, medium: -5
, high: -10
, critical: -15
};
```
### Failure Attempts
By default jobs only have _one_ attempt, that is when they fail, they are marked as a failure, and remain that way until you intervene. However, Kue allows you to specify this, which is important for jobs such as transferring an email, which upon failure, may usually retry without issue. To do this invoke the `.attempts()` method with a number.
```js
queue.create('email', {
title: 'welcome email for tj'
, to: 'tj@learnboost.com'
, template: 'welcome-email'
}).priority('high').attempts(5).save();
```
### Failure Backoff
Job retry attempts are done as soon as they fail, with no delay, even if your job had a delay set via `Job#delay`. If you want to delay job re-attempts upon failures (known as backoff) you can use `Job#backoff` method in different ways:
```js
// Honor job's original delay (if set) at each attempt, defaults to fixed backoff
job.attempts(3).backoff( true )
// Override delay value, fixed backoff
job.attempts(3).backoff( {delay: 60*1000, type:'fixed'} )
// Enable exponential backoff using original delay (if set)
job.attempts(3).backoff( {type:'exponential'} )
// Use a function to get a customized next attempt delay value
job.attempts(3).backoff( function( attempts, delay ){
//attempts will correspond to the nth attempt failure so it will start with 0
//delay will be the amount of the last delay, not the initial delay unless attempts === 0
return my_customized_calculated_delay;
})
```
In the last scenario, provided function will be executed (via eval) on each re-attempt to get next attempt delay value, meaning that you can't reference external/context variables within it.
### Job TTL
Job producers can set an expiry value for the time their job can live in active state, so that if workers didn't reply in timely fashion, Kue will fail it with `TTL exceeded` error message preventing that job from being stuck in active state and spoiling concurrency.
```js
queue.create('email', {title: 'email job with TTL'}).ttl(milliseconds).save();
```
### Job Logs
Job-specific logs enable you to expose information to the UI at any point in the job's life-time. To do so simply invoke `job.log()`, which accepts a message string as well as variable-arguments for sprintf-like support:
```js
job.log('$%d sent to %s', amount, user.name);
```
or anything else (uses [util.inspect()](https://nodejs.org/api/util.html#util_util_inspect_object_options) internally):
```js
job.log({key: 'some key', value: 10});
job.log([1,2,3,5,8]);
job.log(10.1);
```
### Job Progress
Job progress is extremely useful for long-running jobs such as video conversion. To update the job's progress simply invoke `job.progress(completed, total [, data])`:
```js
job.progress(frames, totalFrames);
```
data can be used to pass extra information about the job. For example a message or an object with some extra contextual data to the current status.
### Job Events
Job-specific events are fired on the `Job` instances via Redis pubsub. The following events are currently supported:
- `enqueue` the job is now queued
- `start` the job is now running
- `promotion` the job is promoted from delayed state to queued
- `progress` the job's progress ranging from 0-100
- `failed attempt` the job has failed, but has remaining attempts yet
- `failed` the job has failed and has no remaining attempts
- `complete` the job has completed
- `remove` the job has been removed
For example this may look something like the following:
```js
var job = queue.create('video conversion', {
title: 'converting loki\'s to avi'
, user: 1
, frames: 200
});
job.on('complete', function(result){
console.log('Job completed with data ', result);
}).on('failed attempt', function(errorMessage, doneAttempts){
console.log('Job failed');
}).on('failed', function(errorMessage){
console.log('Job failed');
}).on('progress', function(progress, data){
console.log('\r job #' + job.id + ' ' + progress + '% complete with data ', data );
});
```
**Note** that Job level events are not guaranteed to be received upon process restarts, since restarted node.js process will lose the reference to the specific Job object. If you want a more reliable event handler look for [Queue Events](#queue-events).
**Note** Kue stores job objects in memory until they are complete/failed to be able to emit events on them. If you have a huge concurrency in uncompleted jobs, turn this feature off and use queue level events for better memory scaling.
```js
kue.createQueue({jobEvents: false})
```
Alternatively, you can use the job level function `events` to control whether events are fired for a job at the job level.
```js
var job = queue.create('test').events(false).save();
```
### Queue Events
Queue-level events provide access to the job-level events previously mentioned, however scoped to the `Queue` instance to apply logic at a "global" level. An example of this is removing completed jobs:
```js
queue.on('job enqueue', function(id, type){
console.log( 'Job %s got queued of type %s', id, type );
}).on('job complete', function(id, result){
kue.Job.get(id, function(err, job){
if (err) return;
job.remove(function(err){
if (err) throw err;
console.log('removed completed job #%d', job.id);
});
});
});
```
The events available are the same as mentioned in "Job Events", however prefixed with "job ".
### Delayed Jobs
Delayed jobs may be scheduled to be queued for an arbitrary distance in time by invoking the `.delay(ms)` method, passing the number of milliseconds relative to _now_. Alternatively, you can pass a JavaScript `Date` object with a specific time in the future.
This automatically flags the `Job` as "delayed".
```js
var email = queue.create('email', {
title: 'Account renewal required'
, to: 'tj@learnboost.com'
, template: 'renewal-email'
}).delay(milliseconds)
.priority('high')
.save();
```
Kue will check the delayed jobs with a timer, promoting them if the scheduled delay has been exceeded, defaulting to a check of top 1000 jobs every second.
## Processing Jobs
Processing jobs is simple with Kue. First create a `Queue` instance much like we do for creating jobs, providing us access to redis etc, then invoke `queue.process()` with the associated type.
Note that unlike what the name `createQueue` suggests, it currently returns a singleton `Queue` instance. So you can configure and use only a single `Queue` object within your node.js process.
In the following example we pass the callback `done` to `email`, When an error occurs we invoke `done(err)` to tell Kue something happened, otherwise we invoke `done()` only when the job is complete. If this function responds with an error it will be displayed in the UI and the job will be marked as a failure. The error object passed to done, should be of standard type `Error`.
```js
var kue = require('kue')
, queue = kue.createQueue();
queue.process('email', function(job, done){
email(job.data.to, done);
});
function email(address, done) {
if(!isValidEmail(address)) {
//done('invalid to address') is possible but discouraged
return done(new Error('invalid to address'));
}
// email send stuff...
done();
}
```
Workers can also pass job result as the second parameter to done `done(null,result)` to store that in `Job.result` key. `result` is also passed through `complete` event handlers so that job producers can receive it if they like to.
### Processing Concurrency
By default a call to `queue.process()` will only accept one job at a time for processing. For small tasks like sending emails this is not ideal, so we may specify the maximum active jobs for this type by passing a number:
```js
queue.process('email', 20, function(job, done){
// ...
});
```
### Pause Processing
Workers can temporarily pause and resume their activity. That is, after calling `pause` they will receive no jobs in their process callback until `resume` is called. The `pause` function gracefully shutdowns this worker, and uses the same internal functionality as the `shutdown` method in [Graceful Shutdown](#graceful-shutdown).
```js
queue.process('email', function(job, ctx, done){
ctx.pause( 5000, function(err){
console.log("Worker is paused... ");
setTimeout( function(){ ctx.resume(); }, 10000 );
});
});
```
**Note** *The `ctx` parameter from Kue `>=0.9.0` is the second argument of the process callback function and `done` is idiomatically always the last*
**Note** *The `pause` method signature is changed from Kue `>=0.9.0` to move the callback function to the last.*
### Updating Progress
For a "real" example, let's say we need to compile a PDF from numerous slides with [node-canvas](https://github.com/Automattic/node-canvas). Our job may consist of the following data, note that in general you should _not_ store large data in the job it-self, it's better to store references like ids, pulling them in while processing.
```js
queue.create('slideshow pdf', {
title: user.name + "'s slideshow"
, slides: [...] // keys to data stored in redis, mongodb, or some other store
});
```
We can access this same arbitrary data within a separate process while processing, via the `job.data` property. In the example we render each slide one-by-one, updating the job's log and progress.
```js
queue.process('slideshow pdf', 5, function(job, done){
var slides = job.data.slides
, len = slides.length;
function next(i) {
var slide = slides[i]; // pretend we did a query on this slide id ;)
job.log('rendering %dx%d slide', slide.width, slide.height);
renderSlide(slide, function(err){
if (err) return done(err);
job.progress(i, len, {nextSlide : i == len ? 'itsdone' : i + 1});
if (i == len) done()
else next(i + 1);
});
}
next(0);
});
```
### Graceful Shutdown
`Queue#shutdown([timeout,] fn)` signals all workers to stop processing after their current active job is done. Workers will wait `timeout` milliseconds for their active job's done to be called or mark the active job `failed` with shutdown error reason. When all workers tell Kue they are stopped `fn` is called.
```javascript
var queue = require('kue').createQueue();
process.once( 'SIGTERM', function ( sig ) {
queue.shutdown( 5000, function(err) {
console.log( 'Kue shutdown: ', err||'' );
process.exit( 0 );
});
});
```
**Note** *that `shutdown` method signature is changed from Kue `>=0.9.0` to move the callback function to the last.*
## Error Handling
All errors either in Redis client library or Queue are emitted to the `Queue` object. You should bind to `error` events to prevent uncaught exceptions or debug kue errors.
```javascript
var queue = require('kue').createQueue();
queue.on( 'error', function( err ) {
console.log( 'Oops... ', err );
});
```
### Prevent from Stuck Active Jobs
Kue marks a job complete/failed when `done` is called by your worker, so you should use proper error handling to prevent uncaught exceptions in your worker's code and node.js process exiting before in handle jobs get done.
This can be achieved in two ways:
1. Wrapping your worker's process function in [Domains](https://nodejs.org/api/domain.html)
```js
queue.process('my-error-prone-task', function(job, done){
var domain = require('domain').create();
domain.on('error', function(err){
done(err);
});
domain.run(function(){ // your process function
throw new Error( 'bad things happen' );
done();
});
});
```
**Notice -** Domains are [deprecated](https://nodejs.org/api/documentation.html#documentation_stability_index) from Nodejs with **stability 0** and it's not recommended to use.
This is the softest and best solution, however is not built-in with Kue. Please refer to [this discussion](https://github.com/kriskowal/q/issues/120). You can comment on this feature in the related open Kue [issue](https://github.com/Automattic/kue/pull/403).
You can also use promises to do something like
```js
queue.process('my-error-prone-task', function(job, done){
Promise.method( function(){ // your process function
throw new Error( 'bad things happen' );
})().nodeify(done)
});
```
but this won't catch exceptions in your async call stack as domains do.
2. Binding to `uncaughtException` and gracefully shutting down the Kue, however this is not a recommended error handling idiom in javascript since you are losing the error context.
```js
process.once( 'uncaughtException', function(err){
console.error( 'Something bad happened: ', err );
queue.shutdown( 1000, function(err2){
console.error( 'Kue shutdown result: ', err2 || 'OK' );
process.exit( 0 );
});
});
```
### Unstable Redis connections
Kue currently uses client side job state management and when redis crashes in the middle of that operations, some stuck jobs or index inconsistencies will happen. The consequence is that certain number of jobs will be stuck, and be pulled out by worker only when new jobs are created, if no more new jobs are created, they stuck forever. So we **strongly** suggest that you run watchdog to fix this issue by calling:
```js
queue.watchStuckJobs(interval)
```
`interval` is in milliseconds and defaults to 1000ms
Kue will be refactored to fully atomic job state management from version 1.0 and this will happen by lua scripts and/or BRPOPLPUSH combination. You can read more [here](https://github.com/Automattic/kue/issues/130) and [here](https://github.com/Automattic/kue/issues/38).
## Queue Maintenance
Queue object has two type of methods to tell you about the number of jobs in each state
```js
queue.inactiveCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
if( total > 100000 ) {
console.log( 'We need some back pressure here' );
}
});
```
you can also query on an specific job type:
```js
queue.failedCount( 'my-critical-job', function( err, total ) {
if( total > 10000 ) {
console.log( 'This is tOoOo bad' );
}
});
```
and iterating over job ids
```js
queue.inactive( function( err, ids ) { // others are active, complete, failed, delayed
// you may want to fetch each id to get the Job object out of it...
});
```
however the second one doesn't scale to large deployments, there you can use more specific `Job` static methods:
```js
kue.Job.rangeByState( 'failed', 0, n, 'asc', function( err, jobs ) {
// you have an array of maximum n Job objects here
});
```
or
```js
kue.Job.rangeByType( 'my-job-type', 'failed', 0, n, 'asc', function( err, jobs ) {
// you have an array of maximum n Job objects here
});
```
**Note** *that the last two methods are subject to change in later Kue versions.*
### Programmatic Job Management
If you did none of above in [Error Handling](#error-handling) section or your process lost active jobs in any way, you can recover from them when your process is restarted. A blind logic would be to re-queue all stuck jobs:
```js
queue.active( function( err, ids ) {
ids.forEach( function( id ) {
kue.Job.get( id, function( err, job ) {
// Your application should check if job is a stuck one
job.inactive();
});
});
});
```
**Note** *in a clustered deployment your application should be aware not to involve a job that is valid, currently inprocess by other workers.*
### Job Cleanup
Jobs data and search indexes eat up redis memory space, so you will need some job-keeping process in real world deployments. Your first chance is using automatic job removal on completion.
```javascript
queue.create( ... ).removeOnComplete( true ).save()
```
But if you eventually/temporally need completed job data, you can setup an on-demand job removal script like below to remove top `n` completed jobs:
```js
kue.Job.rangeByState( 'complete', 0, n, 'asc', function( err, jobs ) {
jobs.forEach( function( job ) {
job.remove( function(){
console.log( 'removed ', job.id );
});
});
});
```
**Note** *that you should provide enough time for `.remove` calls on each job object to complete before your process exits, or job indexes will leak*
## Redis Connection Settings
By default, Kue will connect to Redis using the client default settings (port defaults to `6379`, host defaults to `127.0.0.1`, prefix defaults to `q`). `Queue#createQueue(options)` accepts redis connection options in `options.redis` key.
```javascript
var kue = require('kue');
var q = kue.createQueue({
prefix: 'q',
redis: {
port: 1234,
host: '10.0.50.20',
auth: 'password',
db: 3, // if provided select a non-default redis db
options: {
// see https://github.com/mranney/node_redis#rediscreateclient
}
}
});
```
`prefix` controls the key names used in Redis. By default, this is simply `q`. Prefix generally shouldn't be changed unless you need to use one Redis instance for multiple apps. It can also be useful for providing an isolated testbed across your main application.
You can also specify the connection information as a URL string.
```js
var q = kue.createQueue({
redis: 'redis://example.com:1234?redis_option=value&redis_option=value'
});
```
#### Connecting using Unix Domain Sockets
Since [node_redis](https://github.com/mranney/node_redis) supports Unix Domain Sockets, you can also tell Kue to do so. See [unix-domain-socket](https://github.com/mranney/node_redis#unix-domain-socket) for your redis server configuration.
```javascript
var kue = require('kue');
var q = kue.createQueue({
prefix: 'q',
redis: {
socket: '/data/sockets/redis.sock',
auth: 'password',
options: {
// see https://github.com/mranney/node_redis#rediscreateclient
}
}
});
```
#### Replacing Redis Client Module
Any node.js redis client library that conforms (or when adapted) to [node_redis](https://github.com/mranney/node_redis) API can be injected into Kue. You should only provide a `createClientFactory` function as a redis connection factory instead of providing node_redis connection options.
Below is a sample code to enable [redis-sentinel](https://github.com/ortoo/node-redis-sentinel) to connect to [Redis Sentinel](http://redis.io/topics/sentinel) for automatic master/slave failover.
```javascript
var kue = require('kue');
var Sentinel = require('redis-sentinel');
var endpoints = [
{host: '192.168.1.10', port: 6379},
{host: '192.168.1.11', port: 6379}
];
var opts = options || {}; // Standard node_redis client options
var masterName = 'mymaster';
var sentinel = Sentinel.Sentinel(endpoints);
var q = kue.createQueue({
redis: {
createClientFactory: function(){
return sentinel.createClient(masterName, opts);
}
}
});
```
**Note** *that all `<0.8.x` client codes should be refactored to pass redis options to `Queue#createQueue` instead of monkey patched style overriding of `redis#createClient` or they will be broken from Kue `0.8.x`.*
#### Using ioredis client with cluster support
```javascript
var Redis = require('ioredis');
var kue = require('kue');
// using https://github.com/72squared/vagrant-redis-cluster
var queue = kue.createQueue({
redis: {
createClientFactory: function () {
return new Redis.Cluster([{
port: 7000
}, {
port: 7001
}]);
}
}
});
```
## User-Interface
The UI is a small [Express](https://github.com/strongloop/express) application.
A script is provided in `bin/` for running the interface as a standalone application
with default settings. You may pass in options for the port, redis-url, and prefix. For example:
```
node_modules/kue/bin/kue-dashboard -p 3050 -r redis://127.0.0.1:3000 -q prefix
```
You can fire it up from within another application too:
```js
var kue = require('kue');
kue.createQueue(...);
kue.app.listen(3000);
```
The title defaults to "Kue", to alter this invoke:
```js
kue.app.set('title', 'My Application');
```
**Note** *that if you are using non-default Kue options, `kue.createQueue(...)` must be called before accessing `kue.app`.*
### Third-party interfaces
You can also use [Kue-UI](https://github.com/StreetHub/kue-ui) web interface contributed by [Arnaud Bénard](https://github.com/arnaudbenard)
## JSON API
Along with the UI Kue also exposes a JSON API, which is utilized by the UI.
### GET /job/search?q=
Query jobs, for example "GET /job/search?q=avi video":
```js
["5", "7", "10"]
```
By default kue indexes the whole Job data object for searching, but this can be customized via calling `Job#searchKeys` to tell kue which keys on Job data to create index for:
```javascript
var kue = require('kue');
queue = kue.createQueue();
queue.create('email', {
title: 'welcome email for tj'
, to: 'tj@learnboost.com'
, template: 'welcome-email'
}).searchKeys( ['to', 'title'] ).save();
```
Search feature is turned off by default from Kue `>=0.9.0`. Read more about this [here](https://github.com/Automattic/kue/issues/412). You should enable search indexes and add [reds](https://www.npmjs.com/package/reds) in your dependencies if you need to:
```javascript
var kue = require('kue');
q = kue.createQueue({
disableSearch: false
});
```
```
npm install reds --save
```
### GET /stats
Currently responds with state counts, and worker activity time in milliseconds:
```js
{"inactiveCount":4,"completeCount":69,"activeCount":2,"failedCount":0,"workTime":20892}
```
### GET /job/:id
Get a job by `:id`:
```js
{"id":"3","type":"email","data":{"title":"welcome email for tj","to":"tj@learnboost.com","template":"welcome-email"},"priority":-10,"progress":"100","state":"complete","attempts":null,"created_at":"1309973155248","updated_at":"1309973155248","duration":"15002"}
```
### GET /job/:id/log
Get job `:id`'s log:
```js
['foo', 'bar', 'baz']
```
### GET /jobs/:from..:to/:order?
Get jobs with the specified range `:from` to `:to`, for example "/jobs/0..2", where `:order` may be "asc" or "desc":
```js
[{"id":"12","type":"email","data":{"title":"welcome email for tj","to":"tj@learnboost.com","template":"welcome-email"},"priority":-10,"progress":0,"state":"active","attempts":null,"created_at":"1309973299293","updated_at":"1309973299293"},{"id":"130","type":"email","data":{"title":"welcome email for tj","to":"tj@learnboost.com","template":"welcome-email"},"priority":-10,"progress":0,"state":"active","attempts":null,"created_at":"1309975157291","updated_at":"1309975157291"}]
```
### GET /jobs/:state/:from..:to/:order?
Same as above, restricting by `:state` which is one of:
- active
- inactive
- failed
- complete
### GET /jobs/:type/:state/:from..:to/:order?
Same as above, however restricted to `:type` and `:state`.
### DELETE /job/:id
Delete job `:id`:
$ curl -X DELETE http://local:3000/job/2
{"message":"job 2 removed"}
### POST /job
Create a job:
$ curl -H "Content-Type: application/json" -X POST -d \
'{
"type": "email",
"data": {
"title": "welcome email for tj",
"to": "tj@learnboost.com",
"template": "welcome-email"
},
"options" : {
"attempts": 5,
"priority": "high"
}
}' http://localhost:3000/job
{"message": "job created", "id": 3}
You can create multiple jobs at once by passing an array. In this case, the response will be an array too, preserving the order:
$ curl -H "Content-Type: application/json" -X POST -d \
'[{
"type": "email",
"data": {
"title": "welcome email for tj",
"to": "tj@learnboost.com",
"template": "welcome-email"
},
"options" : {
"attempts": 5,
"priority": "high"
}
},
{
"type": "email",
"data": {
"title": "followup email for tj",
"to": "tj@learnboost.com",
"template": "followup-email"
},
"options" : {
"delay": 86400,
"attempts": 5,
"priority": "high"
}
}]' http://localhost:3000/job
[
{"message": "job created", "id": 4},
{"message": "job created", "id": 5}
]
Note: when inserting multiple jobs in bulk, if one insertion fails Kue will keep processing the remaining jobs in order. The response array will contain the ids of the jobs added successfully, and any failed element will be an object describing the error: `{"error": "error reason"}`.
## Parallel Processing With Cluster
The example below shows how you may use [Cluster](http://nodejs.org/api/cluster.html) to spread the job processing load across CPUs. Please see [Cluster module's documentation](http://nodejs.org/api/cluster.html) for more detailed examples on using it.
When cluster `.isMaster` the file is being executed in context of the master process, in which case you may perform tasks that you only want once, such as starting the web app bundled with Kue. The logic in the `else` block is executed _per worker_.
```js
var kue = require('kue')
, cluster = require('cluster')
, queue = kue.createQueue();
var clusterWorkerSize = require('os').cpus().length;
if (cluster.isMaster) {
kue.app.listen(3000);
for (var i = 0; i < clusterWorkerSize; i++) {
cluster.fork();
}
} else {
queue.process('email', 10, function(job, done){
var pending = 5
, total = pending;
var interval = setInterval(function(){
job.log('sending!');
job.progress(total - pending, total);
--pending || done();
pending || clearInterval(interval);
}, 1000);
});
}
```
This will create an `email` job processor (worker) per each of your machine CPU cores, with each you can handle 10 concurrent email jobs, leading to total `10 * N` concurrent email jobs processed in your `N` core machine.
Now when you visit Kue's UI in the browser you'll see that jobs are being processed roughly `N` times faster! (if you have `N` cores).
## Securing Kue
Through the use of app mounting you may customize the web application, enabling TLS, or adding additional middleware like `basic-auth-connect`.
```bash
$ npm install --save basic-auth-connect
```
```js
var basicAuth = require('basic-auth-connect');
var app = express.createServer({ ... tls options ... });
app.use(basicAuth('foo', 'bar'));
app.use(kue.app);
app.listen(3000);
```
## Testing
Enable test mode to push all jobs into a `jobs` array. Make assertions against
the jobs in that array to ensure code under test is correctly enqueuing jobs.
```js
queue = require('kue').createQueue();
before(function() {
queue.testMode.enter();
});
afterEach(function() {
queue.testMode.clear();
});
after(function() {
queue.testMode.exit()
});
it('does something cool', function() {
queue.createJob('myJob', { foo: 'bar' }).save();
queue.createJob('anotherJob', { baz: 'bip' }).save();
expect(queue.testMode.jobs.length).to.equal(2);
expect(queue.testMode.jobs[0].type).to.equal('myJob');
expect(queue.testMode.jobs[0].data).to.eql({ foo: 'bar' });
});
```
**IMPORTANT:** By default jobs aren't processed when created during test mode. You can enable job processing by passing true to testMode.enter
```js
before(function() {
queue.testMode.enter(true);
});
```
## Screencasts
- [Introduction](http://www.screenr.com/oyNs) to Kue
- API [walkthrough](https://vimeo.com/26963384) to Kue
## Contributing
**We love contributions!**
When contributing, follow the simple rules:
* Don't violate [DRY](http://programmer.97things.oreilly.com/wiki/index.php/Don%27t_Repeat_Yourself) principles.
* [Boy Scout Rule](http://programmer.97things.oreilly.com/wiki/index.php/The_Boy_Scout_Rule) needs to have been applied.
* Your code should look like all the other code – this project should look like it was written by one person, always.
* If you want to propose something – just create an issue and describe your question with as much description as you can.
* If you think you have some general improvement, consider creating a pull request with it.
* If you add new code, it should be covered by tests. No tests – no code.
* If you add a new feature, don't forget to update the documentation for it.
* If you find a bug (or at least you think it is a bug), create an issue with the library version and test case that we can run and see what are you talking about, or at least full steps by which we can reproduce it.
## License
(The MIT License)
Copyright (c) 2011 LearnBoost <tj@learnboost.com>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: bin/kue-dashboard
================================================
#!/usr/bin/env node
var kue = require('kue');
var argv = require('yargs')
.usage('Usage: $0 [options]')
.example('$0 -p 3050 -r redis://10.0.0.4:6379 -q q')
.describe('r', 'Redis url')
.describe('p', 'Dashboard port')
.describe('q', 'Prefix to use')
.default('p', 3000)
.default('r', 'redis://127.0.0.1:6379')
.default('q', 'q')
.help('h')
.alias('h', 'help')
.argv
;
kue.createQueue({
redis: argv.r,
prefix: argv.q
});
kue.app.listen(argv.p);
console.log("Running on http://127.0.0.1:" + argv.p);
================================================
FILE: examples/delayed.js
================================================
var kue = require( '../' );
// create our job queue
var jobs = kue.createQueue();
// one minute
var minute = 60000;
var email = jobs.create( 'email', {
title: 'Account renewal required', to: 'tj@learnboost.com', template: 'renewal-email'
} ).delay( minute )
.priority( 'high' )
.save();
email.on( 'promotion', function () {
console.log( 'renewal job promoted' );
} );
email.on( 'complete', function () {
console.log( 'renewal job completed' );
} );
jobs.create( 'email', {
title: 'Account expired', to: 'tj@learnboost.com', template: 'expired-email'
} ).delay( minute * 10 )
.priority( 'high' )
.save();
jobs.promote();
jobs.process( 'email', 10, function ( job, done ) {
setTimeout( function () {
done();
}, Math.random() * 5000 );
} );
// start the UI
kue.app.listen( 3000 );
console.log( 'UI started on port 3000' );
================================================
FILE: examples/events.js
================================================
var kue = require( '../' );
// create our job queue
var jobs = kue.createQueue();
// start redis with $ redis-server
// create some jobs at random,
// usually you would create these
// in your http processes upon
// user input etc.
function create() {
var name = [ 'tobi', 'loki', 'jane', 'manny' ][ Math.random() * 4 | 0 ];
var job = jobs.create( 'video conversion', {
title: 'converting ' + name + '\'s to avi', user: 1, frames: 200
} );
job.on( 'complete', function () {
console.log( " Job complete" );
} ).on( 'failed', function () {
console.log( " Job failed" );
} ).on( 'progress', function ( progress ) {
process.stdout.write( '\r job #' + job.id + ' ' + progress + '% complete' );
} );
job.save();
setTimeout( create, Math.random() * 2000 | 0 );
}
create();
// process video conversion jobs, 1 at a time.
jobs.process( 'video conversion', 1, function ( job, done ) {
var frames = job.data.frames;
function next( i ) {
// pretend we are doing some work
convertFrame( i, function ( err ) {
if ( err ) return done( err );
// report progress, i/frames complete
job.progress( i, frames );
if ( i >= frames ) done()
else next( i + Math.random() * 10 );
} );
}
next( 0 );
} );
function convertFrame( i, fn ) {
setTimeout( fn, Math.random() * 50 );
}
// start the UI
kue.app.listen( 3000 );
console.log( 'UI started on port 3000' );
================================================
FILE: examples/many.js
================================================
var kue = require( '../' )
, express = require( 'express' );
// create our job queue
var jobs = kue.createQueue();
function create() {
var name = [ 'tobi', 'loki', 'jane', 'manny' ][ Math.random() * 4 | 0 ];
jobs.create( 'video conversion', {
title: 'converting ' + name + '\'s to avi', user: 1, frames: 200
} ).save();
setTimeout( create, Math.random() * 3000 | 0 );
}
create();
function create2() {
var name = [ 'tobi', 'loki', 'jane', 'manny' ][ Math.random() * 4 | 0 ];
jobs.create( 'email', {
title: 'emailing ' + name + '', body: 'hello'
} ).save();
setTimeout( create2, Math.random() * 1000 | 0 );
}
create2();
// process video conversion jobs, 2 at a time.
jobs.process( 'video conversion', 2, function ( job, done ) {
console.log( 'video' );
setTimeout( done, Math.random() * 5000 );
} );
// process 10 emails at a time
jobs.process( 'email', 10, function ( job, done ) {
console.log( 'email' );
setTimeout( done, Math.random() * 2000 );
} );
// start the UI
kue.app.listen( 3000 );
console.log( 'UI started on port 3000' );
================================================
FILE: examples/shutdown.js
================================================
var kue = require( '../' )
var jobs = kue.createQueue()
function generateJobs() {
for ( var i = 0; i < 12; i++ ) {
console.log( 'Creating Job #' + i );
jobs.create( 'long render', {
title: 'rendering frame #' + i
} ).save();
}
}
jobs.process( 'long render', 4, function ( job, done ) {
console.log( 'Starting ' + job.data.title );
setTimeout( function () {
console.log( 'Finished ' + job.data.title );
done();
}, 3000 );
} )
generateJobs();
setTimeout( function () {
console.log( '[ Shutting down when all jobs finish... ]' );
jobs.shutdown( function ( err ) {
console.log( '[ All jobs finished. Kue is shut down. ]' );
process.exit( 0 );
} )
}, 4200 )
================================================
FILE: examples/stale.js
================================================
var kue = require( '../' )
, express = require( 'express' );
// create our job queue
var jobs = kue.createQueue()
, Job = kue.Job;
// start redis with $ redis-server
// create some jobs at random,
// usually you would create these
// in your http processes upon
// user input etc.
function create() {
var name = [ 'tobi', 'loki', 'jane', 'manny' ][ Math.random() * 4 | 0 ];
console.log( '- creating job for %s', name );
jobs.create( 'video conversion', {
title: 'converting ' + name + '\'s to avi', user: 1, frames: 200
} ).save();
setTimeout( create, Math.random() * 3000 | 0 );
}
create();
// process video conversion jobs, 3 at a time.
jobs.process( 'video conversion', 3, function ( job, done ) {
var frames = job.data.frames;
console.log( "job process %d", job.id );
function next( i ) {
// pretend we are doing some work
convertFrame( i, function ( err ) {
if ( err ) return done( err );
// report progress, i/frames complete
job.progress( i, frames );
if ( i == frames ) done()
else next( i + 5 );
} );
}
next( 0 );
} );
function convertFrame( i, fn ) {
setTimeout( fn, Math.random() * 100 );
}
// remove stale jobs
jobs.on( 'job complete', function ( id ) {
Job.get( id, function ( err, job ) {
if ( err ) return;
job.remove( function ( err ) {
if ( err ) throw err;
console.log( 'removed completed job #%d', job.id );
} );
} );
} );
// start the UI
var app = express.createServer();
app.use( kue.app );
app.listen( 3000 );
console.log( 'UI started on port 3000' );
================================================
FILE: examples/video.js
================================================
var kue = require( '../' )
, express = require( 'express' );
// create our job queue
var jobs = kue.createQueue();
// start redis with $ redis-server
// create some jobs at random,
// usually you would create these
// in your http processes upon
// user input etc.
function create() {
var name = [ 'tobi', 'loki', 'jane', 'manny' ][ Math.random() * 4 | 0 ];
console.log( '- creating job for %s', name );
jobs.create( 'video conversion', {
title: 'converting ' + name + '\'s to avi', user: 1, frames: 200
} ).save();
setTimeout( create, Math.random() * 3000 | 0 );
}
create();
// process video conversion jobs, 3 at a time.
jobs.process( 'video conversion', 3, function ( job, done ) {
var frames = job.data.frames;
console.log( "job process %d", job.id );
function next( i ) {
// pretend we are doing some work
convertFrame( i, function ( err ) {
if ( err ) return done( err );
// report progress, i/frames complete
job.progress( i, frames );
if ( i == frames ) done()
else next( i + 1 );
} );
}
next( 0 );
} );
function convertFrame( i, fn ) {
setTimeout( fn, Math.random() * 100 );
}
// start the UI
var app = express.createServer();
app.use( express.basicAuth( 'foo', 'bar' ) );
app.use( kue.app );
app.listen( 3000 );
console.log( 'UI started on port 3000' );
================================================
FILE: index.js
================================================
module.exports = require('./lib/kue');
================================================
FILE: lib/http/index.js
================================================
/*!
* q - http
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Module dependencies.
*/
var express = require('express');
// setup
var app = express()
, bodyParser = require('body-parser')
, provides = require('./middleware/provides')
, stylus = require('stylus')
, routes = require('./routes')
, pug = require('pug')
, json = require('./routes/json')
, util = require('util')
, nib = require('nib');
// expose the app
module.exports = app;
// stylus config
function compile( str, path ) {
return stylus(str)
.set('filename', path)
.use(nib());
}
// config
app.set('view options', { doctype: 'html' });
app.set('view engine', 'pug');
app.engine('pug', pug.renderFile);
app.set('views', __dirname + '/views');
app.set('title', 'Kue');
app.locals = { inspect: util.inspect };
// middlewares
app.use(stylus.middleware({ src: __dirname + '/public', compile: compile }));
app.use(express.static(__dirname + '/public'));
// JSON api
app.get('/stats', provides('json'), json.stats);
app.get('/job/search', provides('json'), json.search);
app.get('/jobs/:from..:to/:order?', provides('json'), json.jobRange);
app.get('/jobs/:type/:state/:from..:to/:order?', provides('json'), json.jobTypeRange);
app.get('/jobs/:type/:state/stats', provides('json'), json.jobTypeStateStats);
app.get('/jobs/:state/:from..:to/:order?', provides('json'), json.jobStateRange);
app.get('/job/types', provides('json'), json.types);
app.get('/job/:id', provides('json'), json.job);
app.get('/job/:id/log', provides('json'), json.log);
app.put('/job/:id/state/:state', provides('json'), json.updateState);
app.put('/job/:id/priority/:priority', provides('json'), json.updatePriority);
app.delete('/job/:id', provides('json'), json.remove);
app.post('/job', provides('json'), bodyParser.json(), json.createJob);
app.get('/inactive/:id', provides('json'), json.inactive);
// routes
app.get('/', routes.jobs('active'));
app.get('/active', routes.jobs('active'));
app.get('/inactive', routes.jobs('inactive'));
app.get('/failed', routes.jobs('failed'));
app.get('/complete', routes.jobs('complete'));
app.get('/delayed', routes.jobs('delayed'));
================================================
FILE: lib/http/middleware/provides.js
================================================
/**
* Specify that the route provides `type`.
*
* @param {String} type
* @return {Function}
* @api private
*/
module.exports = function( type ) {
return function( req, res, next ) {
if( req.accepts(type) ) return next();
next('route');
}
};
================================================
FILE: lib/http/public/javascripts/caustic.js
================================================
/*!
* EventEmitter
* Copyright (c) 2011 TJ Holowaychuk <tj@vision-media.ca>
* MIT Licensed
*/
/**
* EventEmitter.
*/
function EventEmitter() {
this.callbacks = {};
}
/**
* Listen on the given `event` with `fn`.
*
* @param {String} event
* @param {Function} fn
*/
EventEmitter.prototype.on = function (event, fn) {
(this.callbacks[event] = this.callbacks[event] || [])
.push(fn);
return this;
};
/**
* Emit `event` with the given args.
*
* @param {String} event
* @param {Mixed} ...
*/
EventEmitter.prototype.emit = function (event) {
var args = Array.prototype.slice.call(arguments, 1)
, callbacks = this.callbacks[event];
if (callbacks) {
for (var i = 0, len = callbacks.length; i < len; ++i) {
callbacks[i].apply(this, args)
}
}
return this;
};
/*!
* caustic
* Copyright(c) 2011 TJ Holowaychuk <tj@vision-media.ca>
* MIT Licensed
*/
// TODO: `make caustic.js` should wrap in an anonymous function
// TODO: `make caustic.min.js`
// TODO: compile sub-views such as User etc based on the given
// html, as there's no need to keep traversing each time.
/**
* Convert callback `fn` to a function when a string is given.
*
* @param {Type} name
* @return {Type}
* @api private
*/
function callback(fn) {
return 'string' == typeof fn
? function (obj) {
return obj[fn]();
}
: fn;
}
/**
* Initialize a new view with the given `name`
* or string of html. When a `name` is given an element
* with the id `name + "-template"` will be used.
*
* Examples:
*
* var user = new View('user');
* var list = new View('<ul class="list"><li></li></ul>');
*
* @param {String} name
* @api public
*/
function View(name) {
if (!(this instanceof View)) return new View(name);
EventEmitter.call(this);
var html;
if (~name.indexOf('<')) html = name;
else html = $('#' + name + '-template').html();
this.el = $(html);
this.visit(this.el);
}
/**
* Inherit from `EventEmitter.prototype`.
*/
View.prototype.__proto__ = EventEmitter.prototype;
/**
* Visit `el`.
*
* @param {jQuery} el
* @param {Boolean} ignore
* @api private
*/
View.prototype.visit = function (el, ignore) {
var self = this
, type = el.get(0).nodeName
, classes = el.attr('class').split(/ +/)
, method = 'visit' + type;
if (this[method] && !ignore) this[method](el, classes[0]);
el.children().each(function (i, el) {
self.visit($(el));
});
};
/**
* Visit INPUT tag.
*
* @param {jQuery} el
* @api public
*/
View.prototype.visitINPUT = function (el) {
var self = this
, name = el.attr('name')
, type = el.attr('type');
switch (type) {
case 'text':
this[name] = function (val) {
if (0 == arguments.length) return el.val();
el.val(val);
return this;
}
this[name].isEmpty = function () {
return '' == el.val();
};
this[name].clear = function () {
el.val('');
return self;
};
break;
case 'checkbox':
this[name] = function (val) {
if (0 == arguments.length) return el.attr('checked');
switch (typeof val) {
case 'function':
el.change(function (e) {
val.call(self, el.attr('checked'), e);
});
break;
default:
el.attr('checked', val
? 'checked'
: val);
}
return this;
}
break;
}
};
/**
* Visit FORM.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitFORM = function (el, name) {
var self = this;
this.submit = function (val) {
switch (typeof val) {
case 'function':
el.submit(function (e) {
val.call(self, e, el);
return false;
});
break;
}
}
};
/**
* Visit A tag.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitA = function (el, name) {
var self = this;
el.click(function (e) {
self.emit(name, e, el);
});
this[name] = function (fn) {
el.click(function (e) {
fn.call(self, e, el);
return false;
});
return this;
}
};
/**
* Visit P, TD, SPAN, or DIV tag.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitP =
View.prototype.visitTD =
View.prototype.visitSPAN =
View.prototype.visitDIV = function (el, name) {
var self = this;
this[name] = function (val) {
if (0 == arguments.length) return el;
el.empty().append(val.el || val);
return this;
};
};
/**
* Visit UL tag.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitUL = function (el, name) {
var self = this;
this.children = [];
this[name] = el;
// TODO: move these out
/**
* Add `val` to this list.
*
* @param {String|jQuery|View} val
* @return {View} for chaining
* @api public
*/
el.add = function (val) {
var li = $('<li>');
self.children.push(val);
el.append(li.append(val.el || val));
return this;
};
/**
* Return the list item `View`s as an array.
*
* @return {Array}
* @api public
*/
el.items = function () {
return self.children;
};
/**
* Iterate the list `View`s, calling `fn(item, i)`.
*
* @param {Function} fn
* @return {View} for chaining
* @api public
*/
el.each = function (fn) {
for (var i = 0, len = self.children.length; i < len; ++i) {
fn(self.children[i], i);
}
return this;
};
/**
* Map the list `View`s, calling `fn(item, i)`.
*
* @param {String|function} fn
* @return {Array}
* @api public
*/
el.map = function (fn) {
var ret = []
, fn = callback(fn);
for (var i = 0, len = self.children.length; i < len; ++i) {
ret.push(fn(self.children[i], i));
}
return ret;
};
};
/**
* Visit TABLE.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitTABLE = function (el, name) {
this[name] = el;
this[name].add = function (val) {
this.append(val.el || val);
};
};
/**
* Visit CANVAS.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitCANVAS = function (el, name) {
this[name] = el.get(0);
};
/**
* Visit H1-H5 tags.
*
* @param {jQuery} el
* @api private
*/
View.prototype.visitH1 =
View.prototype.visitH2 =
View.prototype.visitH3 =
View.prototype.visitH4 =
View.prototype.visitH5 = function (el, name) {
var self = this;
this[name] = function (val) {
if (0 == arguments.length) return el.text();
el.text(val.el || val);
return this;
};
};
/**
* Remove the view from the DOM.
*
* @return {View}
* @api public
*/
View.prototype.remove = function () {
var parent = this.el.parent()
, type = parent.get(0).nodeName;
if ('LI' == type) parent.remove();
else this.el.remove();
return this;
};
/**
* Append this view's element to `val`.
*
* @param {String|jQuery} val
* @return {View}
* @api public
*/
View.prototype.appendTo = function (val) {
this.el.appendTo(val.el || val);
return this;
};
================================================
FILE: lib/http/public/javascripts/job.js
================================================
/*!
* kue - Job
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Initialize a new `Job` with the given `data`.
*
* @param {Object} obj
*/
function Job(data) {
this.update(data);
}
/**
* Show progress indicator.
*
* @param {Boolean} val
* @return {Job} for chaining
*/
Job.prototype.showProgress = function (val) {
this._showProgress = val;
return this;
};
/**
* Show error message when `val` is true.
*
* @param {Boolean} val
* @return {Job} for chaining
*/
Job.prototype.showErrorMessage = function (val) {
this._showError = val;
return this;
};
/**
* Remove the job and callback `fn()`.
*
* @param {Function} fn
*/
Job.prototype.remove = function (fn) {
request('DELETE', './job/' + this.id, fn);
return this;
};
/**
* Restart the job and callback `fn()`.
*
* @param {Function} fn
*/
Job.prototype.restart = function (fn) {
request('GET', './inactive/' + this.id, fn);
return this;
};
/**
* Update the job with the given `data`.
*
* @param {Object} data
* @return {Job} for chaining
*/
Job.prototype.update = function (data) {
for (var key in data) this[key] = data[key];
if (!this.data) this.data = {};
return this;
};
/**
* Render the job, returning an oQuery object.
*
* @param {Boolean} isNew
* @return {oQuery}
*/
Job.prototype.render = function (isNew) {
var self = this
, id = this.id
, view = this.view
, keys = Object.keys(this.data).sort()
, data;
if (isNew) {
view = this.view = View('job');
view.remove(function () {
this.remove();
self.remove();
});
view.restart(function () {
this.restart();
self.restart();
});
var canvas = view.progress
, ctx = this.ctx = canvas.getContext('2d')
, progress = new Progress;
progress.size(canvas.width);
this._progress = progress;
// initially hide the logs
view.log.hide();
// populate title and id
view.el.attr('id', 'job-' + id);
view.id(id);
// show job data
for (var i = 0, len = keys.length; i < len; ++i) {
data = this.data[keys[i]];
if ('object' == typeof data) data = JSON.stringify(data);
var row = View('row');
// row.title(keys[i] + ':').value(data);
row.title(keys[i] + ':').value($('<p></p>').text(data).html());
view.data.add(row);
}
// alter state
view.state(this.state);
view.state().click(function () {
var select = o('<select>%s</select>', options(states, self.state));
o(this).replaceWith(select);
select.change(function () {
self.updateState(select.val());
});
return false;
});
// alter priority
view.priority(priority(this));
view.priority().click(function () {
var select = o('<select>%s</select>', options(priorities, self.priority));
o(this).replaceWith(select);
select.change(function () {
self.updatePriority(select.val());
})
return false;
});
// show details
view.el.find('.contents').toggle(function () {
view.details().addClass('show');
self.showDetails = true;
}, function () {
view.details().removeClass('show');
self.showDetails = false;
});
}
this.renderUpdate();
return view.el;
};
/**
* Update this jobs state to `state`.
*
* @param {String} state
*/
Job.prototype.updateState = function (state) {
request('PUT', './job/' + this.id + '/state/' + state);
};
/**
* Update this jobs priority to `n`.
*
* @param {Number} n
*/
Job.prototype.updatePriority = function (n) {
request('PUT', './job/' + this.id + '/priority/' + n);
};
/**
* Update the job view.
*/
Job.prototype.renderUpdate = function () {
// TODO: templates
var view = this.view
, showError = this._showError
, showProgress = this._showProgress;
// type
view.type(this.type);
// errors
if (showError && this.error) {
view.errorMessage(this.error.split('\n')[0]);
} else {
view.errorMessage().remove();
}
// attempts
if (this.attempts.made) {
view.attempts(this.attempts.made + '/' + this.attempts.max);
} else {
view.attempts().parent().remove();
}
// title
view.title(this.data.title
? this.data.title
: 'untitled');
// details
this.renderTimestamp('created_at');
this.renderTimestamp('updated_at');
this.renderTimestamp('failed_at');
// delayed
if ('delayed' == this.state) {
var delay = parseInt(this.delay, 10)
, creation = parseInt(this.created_at, 10)
, remaining = relative(creation + delay - Date.now());
view.title((this.data.title || '') + ' <em>( ' + remaining + ' )</em>');
}
// inactive
if ('inactive' == this.state) view.log.remove();
// completion
if ('complete' == this.state) {
view.duration(relative(this.duration));
view.updated_at().prev().text('Completed: ');
view.priority().parent().hide();
} else {
view.duration().parent().remove();
}
// error
if ('failed' == this.state) {
view.error().show().find('pre').text(this.error);
} else {
view.error().hide();
}
// progress indicator
if (showProgress) this._progress.update(this.progress).draw(this.ctx);
// logs
if (this.showDetails) {
request('GET', './job/' + this.id + '/log', function (log) {
var ul = view.log.show();
// return early if log hasnt changed
if (ul.text() === log) return;
ul.find('li').remove();
log.forEach(function (line) {
ul.append(o('<li>%s</li>', line));
});
});
}
};
/**
* Render timestamp for the given `prop`.
*
* @param {String} prop
*/
Job.prototype.renderTimestamp = function (prop) {
var val = this[prop]
, view = this.view;
if (val) {
view[prop]().text(relative(Date.now() - val) + ' ago');
} else {
view[prop]().parent().remove();
}
};
================================================
FILE: lib/http/public/javascripts/jquery.ext.js
================================================
// proxy to allow formatting
// and because $ is ugly
var o = function (val) {
var args = arguments
, options = args[1]
, i = 0;
if ('string' != typeof val) return $(val);
if (!~val.indexOf('<')) return $(val);
val = val.replace(/%([sd])/g, function (_, specifier) {
var arg = args[++i];
switch (specifier) {
case 's':
return String(arg)
case 'd':
return arg | 0;
}
});
val = val.replace(/\{(\w+)\}/g, function (_, name) {
return options[name];
});
return $(val);
};
for (var key in $) o[key] = $[key];
================================================
FILE: lib/http/public/javascripts/loading.js
================================================
/*!
* kue - LoadingIndicator
* Copyright (c) 2011 LearnBoost <dev@learnboost.com>
* MIT Licensed
*/
/**
* Initialize a new `LoadingIndicator`.
*/
function LoadingIndicator() {
this.size(0);
this.fontSize(9);
this.font('helvetica, arial, sans-serif');
}
/**
* Set size to `n`.
*
* @param {Number} n
* @return {LoadingIndicator} for chaining
* @api public
*/
LoadingIndicator.prototype.size = function (n) {
this._size = n;
return this;
};
/**
* Set font size to `n`.
*
* @param {Number} n
* @return {LoadingIndicator} for chaining
* @api public
*/
LoadingIndicator.prototype.fontSize = function (n) {
this._fontSize = n;
return this;
};
/**
* Set font `family`.
*
* @param {String} family
* @return {LoadingIndicator} for chaining
*/
LoadingIndicator.prototype.font = function (family) {
this._font = family;
return this;
};
/**
* Update pos to `n`.
*
* @param {Number} n
* @return {LoadingIndicator} for chaining
*/
LoadingIndicator.prototype.update = function (n) {
this.pos = n;
return this;
};
/**
* Draw on `ctx`.
*
* @param {CanvasRenderingContext2d} ctx
* @return {LoadingIndicator} for chaining
*/
LoadingIndicator.prototype.draw = function (ctx) {
var pos = this.pos % 360
, size = this._size
, half = size / 2
, x = half
, y = half
, rad = half - 1
, fontSize = this._fontSize;
ctx.font = fontSize + 'px ' + this._font;
ctx.clearRect(0, 0, size, size);
// outer circle
ctx.strokeStyle = '#9f9f9f';
ctx.beginPath();
ctx.arc(x, y, rad, pos, Math.PI / 2 + pos, false);
ctx.stroke();
// inner circle
ctx.strokeStyle = '#eee';
ctx.beginPath();
ctx.arc(x, y, rad - 3, -pos, Math.PI / 2 - pos, false);
ctx.stroke();
// text
var text = 'Loading'
, w = ctx.measureText(text).width;
ctx.fillText(
text
, x - w / 2 + 1
, y + fontSize / 2 - 1);
return this;
};
================================================
FILE: lib/http/public/javascripts/main.js
================================================
/*!
* kue - http - main
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
// TODO: clean up
// TODO: server-side config for this stuff
// TODO: optimize! many of these jQuery objects can be cached
/**
* Active state.
*/
var active;
/**
* Active type filter.
*/
var filter;
/**
* Number of jobs fetched when "more" is clicked.
*/
var more = 10;
/**
* Number of jobs shown.
*/
var to = more;
/**
* Sort order.
*/
var sort = 'asc';
/**
* Loading indicator.
*/
var loading;
/**
* Initialize UI.
*/
function init(state) {
var canvas = o('#loading canvas').get(0)
, ctx = canvas.getContext('2d');
loading = new LoadingIndicator;
loading.ctx = ctx;
loading.size(canvas.width);
pollStats(1000);
show(state)();
o('li.inactive a').click(show('inactive'));
o('li.complete a').click(show('complete'));
o('li.active a').click(show('active'));
o('li.failed a').click(show('failed'));
o('li.delayed a').click(show('delayed'));
o('#filter').change(function () {
filter = $(this).val();
});
o('#sort').change(function () {
sort = $(this).val();
o('#jobs .job').remove();
});
onpopstate = function (e) {
if (e.state) show(e.state.state)();
};
}
/**
* Show loading indicator.
*/
function showLoading() {
var n = 0;
o('#loading').show();
showLoading.timer = setInterval(function () {
loading.update(++n).draw(loading.ctx);
}, 50);
}
/**
* Hide loading indicator.
*/
function hideLoading() {
o('#loading').hide();
clearInterval(showLoading.timer);
}
/**
* Infinite scroll.
*/
function infiniteScroll() {
if (infiniteScroll.bound) return;
var body = o('body');
hideLoading();
infiniteScroll.bound = true;
o(window).scroll(function (e) {
var top = body.scrollTop()
, height = body.innerHeight()
, windowHeight = window.innerHeight
, pad = 30;
if (top + windowHeight + pad >= height) {
to += more;
infiniteScroll.bound = false;
showLoading();
o(window).unbind('scroll');
}
});
}
/**
* Show jobs with `state`.
*
* @param {String} state
* @param {Boolean} init
* @return {Function}
*/
function show(state) {
return function () {
active = state;
if (pollForJobs.timer) {
clearTimeout(pollForJobs.timer);
delete pollForJobs.timer;
}
history.pushState({ state: state }, state, state);
o('#jobs .job').remove();
o('#menu li a').removeClass('active');
o('#menu li.' + state + ' a').addClass('active');
pollForJobs(state, 1000);
return false;
}
}
/**
* Poll for jobs with `state` every `ms`.
*
* @param {String} state
* @param {Number} ms
*/
function pollForJobs(state, ms) {
o('h1').text(state);
refreshJobs(state, function () {
infiniteScroll();
if (!pollForJobs.timer) pollForJobs.timer = setTimeout(function () {
delete pollForJobs.timer;
pollForJobs(state, ms);
}, ms);
});
};
/**
* Re-request and refresh job elements.
*
* @param {String} state
* @param {Function} fn
*/
function refreshJobs(state, fn) {
// TODO: clean this crap up
var jobHeight = o('#jobs .job .block').outerHeight(true)
, top = o(window).scrollTop()
, height = window.innerHeight
, visibleFrom = Math.max(0, Math.floor(top / jobHeight))
, visibleTo = Math.floor((top + height) / jobHeight)
, url = './jobs/'
+ (filter ? filter + '/' : '')
+ state + '/0..' + to
+ '/' + sort;
// var color = ['blue', 'red', 'yellow', 'green', 'purple'][Math.random() * 5 | 0];
request(url, function (jobs) {
var len = jobs.length
, job
, el;
// remove jobs which have changed their state
o('#jobs .job').each(function (i, el) {
var el = $(el)
, id = (el.attr('id') || '').replace('job-', '')
, found = jobs.some(function (job) {
return job && id == job.id;
});
if (!found) el.remove();
});
for (var i = 0; i < len; ++i) {
if (!jobs[i]) continue;
// exists
if (o('#job-' + jobs[i].id).length) {
if (i < visibleFrom || i > visibleTo) continue;
el = o('#job-' + jobs[i].id);
// el.css('background-color', color);
job = el.get(0).job;
job.update(jobs[i])
.showProgress('active' == active)
.showErrorMessage('failed' == active)
.render();
// new
} else {
job = new Job(jobs[i]);
el = job.showProgress('active' == active)
.showErrorMessage('failed' == active)
.render(true);
el.get(0).job = job;
el.appendTo('#jobs');
}
}
fn();
});
}
/**
* Poll for stats every `ms`.
*
* @param {Number} ms
*/
function pollStats(ms) {
request('./stats', function (data) {
o('li.inactive .count').text(data.inactiveCount);
o('li.active .count').text(data.activeCount);
o('li.complete .count').text(data.completeCount);
o('li.failed .count').text(data.failedCount);
o('li.delayed .count').text(data.delayedCount);
setTimeout(function () {
pollStats(ms);
}, ms);
});
}
/**
* Request `url` and invoke `fn(res)`.
*
* @param {String} url
* @param {Function} fn
*/
function request(url, fn) {
var method = 'GET';
if ('string' == typeof fn) {
method = url;
url = fn;
fn = arguments[2];
}
fn = fn || function () {
};
o.ajax({ type: method, url: url })
.success(function (res) {
res.error
? error(res.error)
: fn(res);
});
}
/**
* Display error `msg`.
*
* @param {String} msg
*/
function error(msg) {
o('#error').text(msg).addClass('show');
setTimeout(function () {
o('#error').removeClass('show');
}, 4000);
}
================================================
FILE: lib/http/public/javascripts/progress.js
================================================
/*!
* kue - Progress
* Copyright (c) 2011 LearnBoost <dev@learnboost.com>
* MIT Licensed
*/
/**
* Initialize a new `Progress` indicator.
*/
function Progress() {
this.percent = 0;
this.size(0);
this.fontSize(12);
this.font('helvetica, arial, sans-serif');
}
/**
* Set progress size to `n`.
*
* @param {Number} n
* @return {Progress} for chaining
* @api public
*/
Progress.prototype.size = function (n) {
this._size = n;
return this;
};
/**
* Set text to `str`.
*
* @param {String} str
* @return {Progress} for chaining
* @api public
*/
Progress.prototype.text = function (str) {
this._text = str;
return this;
};
/**
* Set font size to `n`.
*
* @param {Number} n
* @return {Progress} for chaining
* @api public
*/
Progress.prototype.fontSize = function (n) {
this._fontSize = n;
return this;
};
/**
* Set font `family`.
*
* @param {String} family
* @return {Progress} for chaining
*/
Progress.prototype.font = function (family) {
this._font = family;
return this;
};
/**
* Update percentage to `n`.
*
* @param {Number} n
* @return {Progress} for chaining
*/
Progress.prototype.update = function (n) {
this.percent = n;
return this;
};
/**
* Draw on `ctx`.
*
* @param {CanvasRenderingContext2d} ctx
* @return {Progress} for chaining
*/
Progress.prototype.draw = function (ctx) {
var percent = Math.min(this.percent, 100)
, size = this._size
, half = size / 2
, x = half
, y = half
, rad = half - 1
, fontSize = this._fontSize;
ctx.font = fontSize + 'px ' + this._font;
var angle = Math.PI * 2 * (percent / 100);
ctx.clearRect(0, 0, size, size);
// outer circle
ctx.strokeStyle = '#9f9f9f';
ctx.beginPath();
ctx.arc(x, y, rad, 0, angle, false);
ctx.stroke();
// inner circle
ctx.strokeStyle = '#eee';
ctx.beginPath();
ctx.arc(x, y, rad - 1, 0, angle, true);
ctx.stroke();
// text
var text = this._text || (percent | 0) + '%'
, w = ctx.measureText(text).width;
ctx.fillText(
text
, x - w / 2 + 1
, y + fontSize / 2 - 1);
return this;
};
================================================
FILE: lib/http/public/javascripts/search.js
================================================
o(function () {
var search = o('#search');
search.keyup(function () {
var val = search.val().trim()
, jobs = o('#jobs .job');
// show all
if (val.length < 2) return jobs.show();
// query
o.get('./job/search?q=' + encodeURIComponent(val), function (ids) {
jobs.each(function (i, el) {
var id = el.id.replace('job-', '');
if (~ids.indexOf(id)) {
o(el).show();
} else {
o(el).hide();
}
});
});
});
});
================================================
FILE: lib/http/public/javascripts/utils.js
================================================
/*!
* kue - utils
* Copyright (c) 2010 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Format `ms` in words.
*
* @param {Number} ms
* @return {String}
*/
function relative(ms) {
var sec = 1000
, min = 60 * sec
, hour = 60 * min;
function n(n, name) {
n = Math.round(n);
return n + ' ' + name + (n > 1 ? 's' : '');
}
if (isNaN(ms)) return '';
if (ms < sec) return 'less than one second';
if (ms < min) return n(ms / sec, 'second');
if (ms < hour) return n(ms / min, 'minute');
return n(ms / hour, 'hour');
// TODO: larger than an hour or so, we should
// have some nice date formatting
}
/**
* Default job states.
*/
var states = {
active: 'active', inactive: 'inactive', failed: 'failed', complete: 'complete', delayed: 'delayed'
};
/**
* Default job priority map.
*/
var priorities = {
'10': 'low', '0': 'normal', '-5': 'medium', '-10': 'high', '-15': 'critical'
};
/**
* Return priority string for `job`.
*
* @param {Job} job
* @return {String}
*/
function priority(job) {
return priorities[job.priority] || job.priority;
}
/**
* Generate options from `obj`.
*
* @param {Object} obj
* @param {String} selected
* @return {String}
*/
function options(obj, selected) {
var html = '';
for (var key in obj) {
html += '<option value="' + key + '" '
+ (key == selected ? 'selected="selected"' : '')
+ '>' + obj[key] + '</option>\n';
}
return html;
}
================================================
FILE: lib/http/public/stylesheets/actions.styl
================================================
#actions
fixed: top -2px right -2px
z-index: 20
#sort
#filter
#search
float: left
margin: 0
padding: 5px 10px
border: 1px solid #eee
border-radius: 0 0 0 5px
-webkit-appearance: none
color: dark
outline: none
&:hover
border-color: #eee - 10%
#sort
#filter
cursor: pointer
#sort
#filter
border-radius: 0
border-left: none
================================================
FILE: lib/http/public/stylesheets/config.styl
================================================
// general colors
dark = #3b3b3b
light = #666
lighter = #777
bg = #fff
// status colors
inactive-color = #00CCCC
complete-color = #00CC7A
active-color = #CCC500
failed-color = #c00
// menu config
menu-bg = dark
menu-fg = lighter
menu-intensity = 13%
menu-colored = false
// job config
job-bg = white
// scrollbar
scroll-bg = transparent
scroll-thumb = menu-bg
scroll-width = 6px
================================================
FILE: lib/http/public/stylesheets/context-menu.styl
================================================
@import 'mixins'
highlight-color = #00B3E9
.context-menu
display: none
reset-list()
decorated-box()
li
&:last-child a
border-bottom: none
a
display: block
background: white
padding: 5px 10px
border: 1px solid transparent
border-bottom: 1px solid #eee
font-size: 12px
&:hover
background: linear-gradient(bottom, highlight-color, highlight-color + 50%)
color: white
border: 1px solid white
&:active
background: linear-gradient(bottom, highlight-color + 10%, highlight-color + 10% + 50%)
================================================
FILE: lib/http/public/stylesheets/error.styl
================================================
#error
fixed: top -50px right 15px
padding: 20px
transition: top 500ms, opacity 500ms
opacity: 0
background: rgba(dark, .2)
border: 1px solid rgba(dark, .3)
border-radius: 5px
color: dark
&.show
top: 15px
opacity: 1
================================================
FILE: lib/http/public/stylesheets/job.styl
================================================
@import 'mixins'
#job-template
display: none
bar(color)
background: linear-gradient(top, color + 20%, color)
border: 1px solid rgba(white, .2)
color: white
// generic blocks
.block
decorated-box()
width: 90%
margin: 10px 25px
padding: 20px 25px
h2
margin: 0
absolute: top 5px left -15px
padding: 5px
font-size: 10px
border-radius: left 5px right 2px
background: linear-gradient(left, menu-fg - 10%, 50% menu-fg + 5%)
box-shadow: -1px 0 1px 1px rgba(black, .1)
color: white
text-shadow: 1px 1px 1px #444
.type
color: lighter + 20%
// job delay
.job td.title em
color: lighter + 20%
// job blocks
.job .block
position: relative
background: job-bg
cursor: pointer
table td:first-child
display: none
.progress
absolute: top 15px right 20px
.attempts
display: none
absolute: top right
padding: 5px 8px
border-radius: 2px
font-size: 10px
.remove
absolute: top 30px right -6px
/*background: white*/
background: #F05151
color: white
display: block
width: size = 20px
height: size
line-height: size
text-align: center
font-size: 12px
font-weight: bold
outline: none
border: 1px solid #eee
border-radius: size
transition: opacity 200ms, top 300ms
opacity: 0
&:hover
border: 1px solid #eee - 10%
&:active
border: 1px solid #eee - 20%
.restart
absolute: top 30px right -6px
/*background: white*/
background: #00e600
color: white
display: block
width: size = 20px
height: size
line-height: size
text-align: center
font-size: 12px
font-weight: bold
outline: none
border: 1px solid #eee
border-radius: size
transition: opacity 200ms, top 300ms
opacity: 0
&:hover
border: 1px solid #eee - 10%
&:active
border: 1px solid #eee - 20%
&:hover
.remove
opacity: 1
top: -6px
.restart
opacity: 1
top: 16px
// details
.job .details
background: dark
width: 89%
margin-top: -10px
margin-left: 35px
border-radius: bottom 5px
box-shadow: inset 0 1px 10px 0 rgba(black, .8)
transition: padding 200ms, height 200ms
height: 0
overflow: hidden
table
width: 100%
td:first-child
width: 60px
color: light + 30%
&.show
padding: 15px 20px
height: auto
// job log
.job ul.log
reset-list()
margin: 5px
padding: 10px
max-height: 100px
overflow-y: auto
border-radius: 5px
width: 95%
li
padding: 5px 0
border-bottom: 1px dotted light - 35%
color: light
&:last-child
border-bottom: none
// scrollbar
.job .details
::-webkit-scrollbar
width: 2px
::-webkit-scrollbar-thumb:vertical
background: light + 20%
::-webkit-scrollbar-track
border: 1px solid rgba(white, .1)
// sections
.job .details > div
padding: 10px 0
border-bottom: 1px solid light - 35%
&:last-child
border-bottom: none
================================================
FILE: lib/http/public/stylesheets/main.css
================================================
body {
padding: 50px 120px;
}
#menu {
margin: 0;
padding: 0;
position: fixed;
top: 0;
left: 0;
height: 100%;
width: 80px;
background: #3b3b3b;
border-right: 1px solid #232323;
-webkit-box-shadow: 0 0 0 1px rgba(255,255,255,0.5);
box-shadow: 0 0 0 1px rgba(255,255,255,0.5);
}
#menu li {
margin: 0;
list-style: none;
}
#menu li {
position: relative;
text-align: center;
}
#menu li .count {
position: absolute;
top: 15px;
left: 0;
text-shadow: 1px 1px 1px #333;
width: 100%;
color: #808080;
}
#menu li a {
display: block;
padding: 40px 0 10px 0;
color: #777;
border-top: 1px solid #545454;
border-bottom: 1px solid #333;
font-size: 12px;
background: #393939;
}
#menu li a:hover {
background: #434343;
}
#menu li a:active,
#menu li a.active {
background: #343434;
-webkit-box-shadow: inset 0 0 3px 2px #282828, inset 0 -5px 10px 2px #303030;
box-shadow: inset 0 0 3px 2px #282828, inset 0 -5px 10px 2px #303030;
border-bottom: 1px solid #222;
}
.context-menu {
display: none;
margin: 0;
padding: 0;
border: 1px solid #eee;
border-bottom-color: rgba(0,0,0,0.25);
border-left-color: rgba(0,0,0,0.2);
border-right-color: rgba(0,0,0,0.2);
-webkit-box-shadow: 0 2px 2px 0 rgba(0,0,0,0.1);
box-shadow: 0 2px 2px 0 rgba(0,0,0,0.1);
-webkit-border-radius: 4px;
border-radius: 4px;
}
.context-menu li {
margin: 0;
list-style: none;
}
.context-menu li:last-child a {
border-bottom: none;
}
.context-menu li a {
display: block;
background: #fff;
padding: 5px 10px;
border: 1px solid transparent;
border-bottom: 1px solid #eee;
font-size: 12px;
}
.context-menu li a:hover {
background: -webkit-linear-gradient(bottom, #00b3e9, #74dfff);
background: -moz-linear-gradient(bottom, #00b3e9, #74dfff);
background: -o-linear-gradient(bottom, #00b3e9, #74dfff);
background: -ms-linear-gradient(bottom, #00b3e9, #74dfff);
background: linear-gradient(to top, #00b3e9, #74dfff);
color: #fff;
border: 1px solid #fff;
}
.context-menu li a:active {
background: -webkit-linear-gradient(bottom, #06c5ff, #83e2ff);
background: -moz-linear-gradient(bottom, #06c5ff, #83e2ff);
background: -o-linear-gradient(bottom, #06c5ff, #83e2ff);
background: -ms-linear-gradient(bottom, #06c5ff, #83e2ff);
background: linear-gradient(to top, #06c5ff, #83e2ff);
}
#job-template {
display: none;
}
.block {
border: 1px solid #eee;
border-bottom-color: rgba(0,0,0,0.25);
border-left-color: rgba(0,0,0,0.2);
border-right-color: rgba(0,0,0,0.2);
-webkit-box-shadow: 0 2px 2px 0 rgba(0,0,0,0.1);
box-shadow: 0 2px 2px 0 rgba(0,0,0,0.1);
-webkit-border-radius: 4px;
border-radius: 4px;
width: 90%;
margin: 10px 25px;
padding: 20px 25px;
}
.block h2 {
margin: 0;
position: absolute;
top: 5px;
left: -15px;
padding: 5px;
font-size: 10px;
-webkit-border-top-left-radius: 5px;
border-top-left-radius: 5px;
-webkit-border-bottom-left-radius: 5px;
border-bottom-left-radius: 5px;
-webkit-border-top-right-radius: 2px;
border-top-right-radius: 2px;
-webkit-border-bottom-right-radius: 2px;
border-bottom-right-radius: 2px;
background: -webkit-linear-gradient(left, #6b6b6b, #7e7e7e 50%);
background: -moz-linear-gradient(left, #6b6b6b, #7e7e7e 50%);
background: -o-linear-gradient(left, #6b6b6b, #7e7e7e 50%);
background: -ms-linear-gradient(left, #6b6b6b, #7e7e7e 50%);
background: linear-gradient(to right, #6b6b6b, #7e7e7e 50%);
-webkit-box-shadow: -1px 0 1px 1px rgba(0,0,0,0.1);
box-shadow: -1px 0 1px 1px rgba(0,0,0,0.1);
color: #fff;
text-shadow: 1px 1px 1px #444;
}
.block .type {
color: #929292;
}
.job td.title em {
color: #929292;
}
.job .block {
position: relative;
background: #fff;
cursor: pointer;
}
.job .block table td:first-child {
display: none;
}
.job .block .progress {
position: absolute;
top: 15px;
right: 20px;
}
.job .block .attempts {
display: none;
position: absolute;
top: 0;
right: 0;
padding: 5px 8px;
-webkit-border-radius: 2px;
border-radius: 2px;
font-size: 10px;
}
.job .block .remove {
position: absolute;
top: 30px;
right: -6px;
/*background: white*/
background: #f05151;
color: #fff;
display: block;
width: 20px;
height: 20px;
line-height: 20px;
text-align: center;
font-size: 12px;
font-weight: bold;
outline: none;
border: 1px solid #eee;
-webkit-border-radius: 20px;
border-radius: 20px;
-webkit-transition: opacity 200ms, top 300ms;
-moz-transition: opacity 200ms, top 300ms;
-o-transition: opacity 200ms, top 300ms;
-ms-transition: opacity 200ms, top 300ms;
transition: opacity 200ms, top 300ms;
opacity: 0;
-ms-filter: "progid:DXImageTransform.Microsoft.Alpha(Opacity=0)";
filter: alpha(opacity=0);
}
.job .block .remove:hover {
border: 1px solid #d6d6d6;
}
.job .block .remove:active {
border: 1px solid #bebebe;
}
.job .block .restart {
position: absolute;
top: 30px;
right: -6px;
/*background: white*/
background: #00e600;
color: #fff;
display: block;
width: 20px;
height: 20px;
line-height: 20px;
text-align: center;
font-size: 12px;
font-weight: bold;
outline: none;
border: 1px solid #eee;
-webkit-border-radius: 20px;
border-radius: 20px;
-webkit-transition: opacity 200ms, top 300ms;
-moz-transition: opacity 200ms, top 300ms;
-o-transition: opacity 200ms, top 300ms;
-ms-transition: opacity 200ms, top 300ms;
transition: opacity 200ms, top 300ms;
opacity: 0;
-ms-filter: "progid:DXImageTransform.Microsoft.Alpha(Opacity=0)";
filter: alpha(opacity=0);
}
.job .block .restart:hover {
border: 1px solid #d6d6d6;
}
.job .block .restart:active {
border: 1px solid #bebebe;
}
.job .block:hover .remove {
opacity: 1;
-ms-filter: none;
filter: none;
top: -6px;
}
.job .block:hover .restart {
opacity: 1;
-ms-filter: none;
filter: none;
top: 16px;
}
.job .details {
background: #3b3b3b;
width: 89%;
margin-top: -10px;
margin-left: 35px;
-webkit-border-bottom-left-radius: 5px;
border-bottom-left-radius: 5px;
-webkit-border-bottom-right-radius: 5px;
border-bottom-right-radius: 5px;
-webkit-box-shadow: inset 0 1px 10px 0 rgba(0,0,0,0.8);
box-shadow: inset 0 1px 10px 0 rgba(0,0,0,0.8);
-webkit-transition: padding 200ms, height 200ms;
-moz-transition: padding 200ms, height 200ms;
-o-transition: padding 200ms, height 200ms;
-ms-transition: padding 200ms, height 200ms;
transition: padding 200ms, height 200ms;
height: 0;
overflow: hidden;
}
.job .details table {
width: 100%;
}
.job .details table td:first-child {
width: 60px;
color: #949494;
}
.job .details.show {
padding: 15px 20px;
height: auto;
}
.job ul.log {
margin: 0;
padding: 0;
margin: 5px;
padding: 10px;
max-height: 100px;
overflow-y: auto;
-webkit-border-radius: 5px;
border-radius: 5px;
width: 95%;
}
.job ul.log li {
margin: 0;
list-style: none;
}
.job ul.log li {
padding: 5px 0;
border-bottom: 1px dotted #424242;
color: #666;
}
.job ul.log li:last-child {
border-bottom: none;
}
.job .details ::-webkit-scrollbar {
width: 2px;
}
.job .details ::-webkit-scrollbar-thumb:vertical {
background: #858585;
}
.job .details ::-webkit-scrollbar-track {
border: 1px solid rgba(255,255,255,0.1);
}
.job .details > div {
padding: 10px 0;
border-bottom: 1px solid #424242;
}
.job .details > div:last-child {
border-bottom: none;
}
#actions {
position: fixed;
top: -2px;
right: -2px;
z-index: 20;
}
#sort,
#filter,
#search {
float: left;
margin: 0;
padding: 5px 10px;
border: 1px solid #eee;
-webkit-border-radius: 0 0 0 5px;
border-radius: 0 0 0 5px;
-webkit-appearance: none;
color: #3b3b3b;
outline: none;
}
#sort:hover,
#filter:hover,
#search:hover {
border-color: #d6d6d6;
}
#sort,
#filter {
cursor: pointer;
}
#sort,
#filter {
-webkit-border-radius: 0;
border-radius: 0;
border-left: none;
}
#error {
position: fixed;
top: -50px;
right: 15px;
padding: 20px;
-webkit-transition: top 500ms, opacity 500ms;
-moz-transition: top 500ms, opacity 500ms;
-o-transition: top 500ms, opacity 500ms;
-ms-transition: top 500ms, opacity 500ms;
transition: top 500ms, opacity 500ms;
opacity: 0;
-ms-filter: "progid:DXImageTransform.Microsoft.Alpha(Opacity=0)";
filter: alpha(opacity=0);
background: rgba(59,59,59,0.2);
border: 1px solid rgba(59,59,59,0.3);
-webkit-border-radius: 5px;
border-radius: 5px;
color: #3b3b3b;
}
#error.show {
top: 15px;
opacity: 1;
-ms-filter: none;
filter: none;
}
body {
font: 13px "helvetica neue", helvetica, arial, sans-serif;
-webkit-font-smoothing: antialiased;
background: #fff;
color: #666;
}
h1,
h2,
h3 {
margin: 0 0 25px 0;
padding: 0;
font-weight: normal;
text-transform: capitalize;
color: #666;
}
h2 {
font-size: 16px;
margin-top: 20px;
}
pre {
margin-top: 20px;
}
a {
text-decoration: none;
cursor: pointer;
}
table {
border-collapse: separate;
border-spacing: 0;
vertical-align: middle;
}
table tr td {
padding: 2px 5px;
}
#loading {
width: 100%;
text-align: center;
margin-top: 40px;
margin-left: 20px;
}
#loading canvas {
margin: 0 auto;
}
================================================
FILE: lib/http/public/stylesheets/main.styl
================================================
font-smoothing()
-webkit-font-smoothing: arguments
@import 'nib'
@import 'config'
@import 'scrollbar'
@import 'menu'
@import 'context-menu'
@import 'job'
@import 'actions'
@import 'error'
body
font: 13px "helvetica neue", helvetica, arial, sans-serif
font-smoothing: antialiased
background: bg
color: light
h1, h2, h3
margin: 0 0 25px 0
padding: 0
font-weight: normal
text-transform: capitalize
color: light
h2
font-size: 16px
margin-top: 20px
button
a.button
input[type='submit']
bold-button(glow:#00ABFA)
pre
margin-top: 20px
a
text-decoration: none
cursor: pointer
table
reset-table()
tr td
padding: 2px 5px
#loading
width: 100%
text-align: center
margin-top: 40px
margin-left: 20px
canvas
margin: 0 auto
================================================
FILE: lib/http/public/stylesheets/menu.styl
================================================
@import 'mixins'
#menu
reset-list()
fixed: top left
height: 100%
width: 80px
background: menu-bg
border-right: 1px solid menu-bg - 40%
box-shadow: 0 0 0 1px rgba(white, .5)
li
position: relative
text-align: center
if menu-colored
&.inactive
border-right: 1px solid inactive-color
&.active
border-right: 1px solid active-color
&.complete
border-right: 1px solid complete-color
&.failed
border-right: 1px solid failed-color
.count
absolute: top 15px left
text-shadow: 1px 1px 1px menu-bg - menu-intensity
width: 100%
color: menu-fg + (menu-intensity / 2)
a
display: block
padding: 40px 0 10px 0
color: menu-fg
border-top: 1px solid menu-bg + menu-intensity
border-bottom: 1px solid menu-bg - menu-intensity
font-size: 12px
background: menu-bg -= 2%
&:hover
background: menu-bg + 5%
&:active
&.active
background: #343434
box-shadow: inset 0 0 3px 2px menu-bg - 30%, inset 0 -5px 10px 2px menu-bg - 15%
border-bottom: 1px solid menu-bg - 40%
================================================
FILE: lib/http/public/stylesheets/mixins.styl
================================================
reset-list()
margin: 0
padding: 0
li
margin: 0
list-style: none
decorated-box()
border: 1px solid #eee
border-bottom-color: rgba(black, .25)
border-left-color: rgba(black, .2)
border-right-color: rgba(black, .2)
box-shadow: 0 2px 2px 0 rgba(black, .1)
border-radius: 4px
================================================
FILE: lib/http/public/stylesheets/scrollbar.styl
================================================
width = scroll-width
pad-x = 60px
pad-y = 40px
body
padding: 50px 120px
/*
html
overflow: auto
body
position: absolute
top: pad-y
left: pad-x
bottom: pad-y
right: pad-x
padding: 0 pad-x
overflow-y: scroll
overflow-x: hidden
::-webkit-scrollbar
background: scroll-bg
width: width
::-webkit-scrollbar-button:start:decrement
::-webkit-scrollbar-button:start:increment
display: none
::-webkit-scrollbar-track
border-radius: (width / 2)
box-shadow: inset 0 0 1px rgba(black, .2), inset 0 4px 10px rgba(black, .2)
border: 1px solid rgba(white, .5)
::-webkit-scrollbar-track-piece
background: transparent
::-webkit-scrollbar-thumb:vertical
height: 30px
transition: background-color 300ms ease-out
background: rgba(scroll-thumb, .5)
border-radius: (width / 2)
&:window-inactive
background: rgba(scroll-thumb, .2)
*/
================================================
FILE: lib/http/routes/index.js
================================================
/*!
* kue - http - routes
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Module dependencies.
*/
var Queue = require('../../kue')
, Job = require('../../queue/job')
, queue = Queue.createQueue();
/**
* Serve the index page.
*/
exports.jobs = function( state ) {
return function( req, res ) {
queue.types(function( err, types ) {
res.render('job/list', {
state: state, types: types, title: req.app.get('title')
});
});
};
};
================================================
FILE: lib/http/routes/json.js
================================================
/*!
* kue - http - routes - json
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Module dependencies.
*/
var Queue = require('../../kue')
, Job = require('../../queue/job')
, lodash = require('lodash')
, queue = Queue.createQueue();
/**
* Search instance.
*/
var search;
function getSearch() {
if( search ) return search;
var reds = require('reds');
reds.createClient = require('../../redis').createClient;
return search = reds.createSearch(queue.client.getKey('search'));
}
/**
* Get statistics including:
*
* - inactive count
* - active count
* - complete count
* - failed count
* - delayed count
*
*/
exports.stats = function( req, res ) {
get(queue)
('inactiveCount')
('completeCount')
('activeCount')
('failedCount')
('delayedCount')
('workTime')
(function( err, obj ) {
if( err ) return res.json({ error: err.message });
res.json(obj);
});
};
/**
* Get job types.
*/
exports.types = function( req, res ) {
queue.types(function( err, types ) {
if( err ) return res.json({ error: err.message });
res.json(types);
});
};
/**
* Get jobs by range :from..:to.
*/
exports.jobRange = function( req, res ) {
var from = parseInt(req.params.from, 10)
, to = parseInt(req.params.to, 10)
, order = req.params.order;
Job.range(from, to, order, function( err, jobs ) {
if( err ) return res.json({ error: err.message });
res.json(jobs);
});
};
/**
* Get jobs by :state, and range :from..:to.
*/
exports.jobStateRange = function( req, res ) {
var state = req.params.state
, from = parseInt(req.params.from, 10)
, to = parseInt(req.params.to, 10)
, order = req.params.order;
Job.rangeByState(state, from, to, order, function( err, jobs ) {
if( err ) return res.json({ error: err.message });
res.json(jobs);
});
};
/**
* Get jobs by :type, :state, and range :from..:to.
*/
exports.jobTypeRange = function( req, res ) {
var type = req.params.type
, state = req.params.state
, from = parseInt(req.params.from, 10)
, to = parseInt(req.params.to, 10)
, order = req.params.order;
Job.rangeByType(type, state, from, to, order, function( err, jobs ) {
if( err ) return res.json({ error: err.message });
res.json(jobs);
});
};
/**
* Get jobs stats by :type and :state
*/
exports.jobTypeStateStats = function( req, res ) {
var type = req.params.type
, state = req.params.state;
queue.cardByType(type, state, function( err, count ) {
if( err ) return res.json({ error: err.message });
res.json({ count: count });
});
};
/**
* Get job by :id.
*/
exports.job = function( req, res ) {
var id = req.params.id;
Job.get(id, function( err, job ) {
if( err ) return res.json({ error: err.message });
res.json(job);
});
};
/**
* Restart job by :id.
*/
exports.inactive = function( req, res ) {
var id = req.params.id;
Job.get(id, function( err, job ) {
if( err ) return res.json({ error: err.message });
job.inactive();
res.json({ message: 'job ' + id + ' inactive' });
});
};
/**
* Create a job.
*/
exports.createJob = function( req, res ) {
var body = req.body;
function _create( args, next ) {
if( !args.type ) return next({ error: 'Must provide job type' }, null, 400);
var job = new Job(args.type, args.data || {});
var options = args.options || {};
if( options.attempts ) job.attempts(parseInt(options.attempts));
if( options.priority ) job.priority(options.priority);
if( options.delay ) job.delay(options.delay);
if( options.searchKeys ) job.searchKeys(options.searchKeys);
if( options.backoff ) job.backoff(options.backoff);
if( options.removeOnComplete ) job.removeOnComplete(options.removeOnComplete);
if( options.ttl ) job.ttl(options.ttl);
job.save(function( err ) {
if( err ) {
return next({ error: err.message }, null, 500);
}
else {
return next(null, { message: 'job created', id: job.id });
}
});
}
if( !lodash.isEmpty(body) ) {
if( lodash.isArray(body) ) {
var returnErrorCode = 0; // Default: we don't have any error
var i = 0, len = body.length;
var result = [];
-function _iterate() {
_create(body[ i ], function( err, status, errCode ) {
result.push(err || status);
if( err ) {
// Set an error code for the response
if( !returnErrorCode ) {
returnErrorCode = errCode || 500;
}
}
// Keep processing even after an error
i++;
if( i < len ) {
_iterate();
}
else {
// If we had an error code, return it
if( returnErrorCode ) {
res.status(returnErrorCode);
}
res.json(result);
}
})
}()
}
else {
_create(body, function( err, status, errCode ) {
if( err ) {
res.status(errCode || 500).json(err);
}
else {
res.json(status);
}
})
}
}
else {
res.status(204); // "No content" status code
res.end();
}
};
/**
* Remove job :id.
*/
exports.remove = function( req, res ) {
var id = req.params.id;
Job.remove(id, function( err ) {
if( err ) return res.json({ error: err.message });
res.json({ message: 'job ' + id + ' removed' });
});
};
/**
* Update job :id :priority.
*/
exports.updatePriority = function( req, res ) {
var id = req.params.id
, priority = parseInt(req.params.priority, 10);
if( isNaN(priority) ) return res.json({ error: 'invalid priority' });
Job.get(id, function( err, job ) {
if( err ) return res.json({ error: err.message });
job.priority(priority);
job.save(function( err ) {
if( err ) return res.json({ error: err.message });
res.json({ message: 'updated priority' });
});
});
};
/**
* Update job :id :state.
*/
exports.updateState = function( req, res ) {
var id = req.params.id
, state = req.params.state;
Job.get(id, function( err, job ) {
if( err ) return res.json({ error: err.message });
job.state(state);
job.save(function( err ) {
if( err ) return res.json({ error: err.message });
res.json({ message: 'updated state' });
});
});
};
/**
* Search and respond with ids.
*/
exports.search = function( req, res ) {
getSearch().query(req.query.q).end(function( err, ids ) {
if( err ) return res.json({ error: err.message });
res.json(ids);
});
};
/**
* Get log for job :id.
*/
exports.log = function( req, res ) {
var id = req.params.id;
Job.log(id, function( err, log ) {
if( err ) return res.json({ error: err.message });
res.json(log);
});
};
/**
* Data fetching helper.
*/
function get( obj ) {
var pending = 0
, res = {}
, callback
, done;
return function _( arg ) {
switch(typeof arg) {
case 'function':
callback = arg;
break;
case 'string':
++pending;
obj[ arg ](function( err, val ) {
if( done ) return;
if( err ) return done = true, callback(err);
res[ arg ] = val;
--pending || callback(null, res);
});
break;
}
return _;
};
}
================================================
FILE: lib/http/views/_filter.pug
================================================
select#filter
option(value='') filter by
each type in types
option(value=type)= type
================================================
FILE: lib/http/views/_job.pug
================================================
.job
.block.contents
h2.id
a.remove(title='Delete Job') x
a.restart(title='Restart Job') ↻
canvas.progress(width=50, height=50)
table.meta
tbody
tr
td Type:
td.type
tr
td Title:
td.title
tr
td Error:
td.errorMessage
.details
.data
table.data
tbody
tr
td State:
td.state
tr
td Priority:
td.priority
tr
td Attempts:
td.attempts
tr.time
td Duration:
td.duration
tr.time
td Created:
td.created_at
tr.time
td Updated:
td.updated_at
tr.time
td Failed:
td.failed_at
.error
pre
ul.log
================================================
FILE: lib/http/views/_menu.pug
================================================
ul#menu
li.inactive
a(href='./inactive')
.count 0
| Queued
li.active
a.active(href='./active')
.count 0
| Active
li.failed
a(href='./failed')
.count 0
| Failed
li.complete
a(href='./complete')
.count 0
| Complete
li.delayed
a(href='./delayed')
.count 0
| Delayed
================================================
FILE: lib/http/views/_row.pug
================================================
tr
td.title
td.value
================================================
FILE: lib/http/views/_search.pug
================================================
input#search(type='text', placeholder='Search')
================================================
FILE: lib/http/views/_sort.pug
================================================
select#sort
option(value='asc') sort
option(value='asc') asc
option(value='desc') desc
================================================
FILE: lib/http/views/job/list.pug
================================================
extends ../layout
block body
h1 #{state}
script.
o(function(){
init('#{state}');
});
#jobs
#loading: canvas(width=50, height=50)
================================================
FILE: lib/http/views/layout.pug
================================================
html
head
title= title
link(rel='stylesheet', href='./stylesheets/main.css')
script(src='./javascripts/utils.js')
script(src='./javascripts/jquery.min.js')
script(src='./javascripts/jquery.ext.js')
script(src='./javascripts/caustic.js')
script(src='./javascripts/progress.js')
script(src='./javascripts/loading.js')
script(src='./javascripts/job.js')
script(src='./javascripts/search.js')
script(src='./javascripts/main.js')
body
include _menu
#actions
include _search
include _filter
include _sort
#content
block body
script(type='text/template')#job-template
include _job
script(type='text/template')#row-template
include _row
#error
================================================
FILE: lib/kue.js
================================================
/*!
* kue
* Copyright (c) 2013 Automattic <behradz@gmail.com>
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Module dependencies.
*/
var EventEmitter = require('events').EventEmitter
, Worker = require('./queue/worker')
, events = require('./queue/events')
, Job = require('./queue/job')
, Warlock = require('node-redis-warlock')
, _ = require('lodash')
, redis = require('./redis')
, noop = function(){};
/**
* Expose `Queue`.
*/
exports = module.exports = Queue;
/**
* Library version.
*/
exports.version = require('../package.json').version;
/**
* Expose `Job`.
*/
exports.Job = Job;
/**
* Server instance (that is lazily required)
*/
var app;
/**
* Expose the server.
*/
Object.defineProperty(exports, 'app', {
get: function() {
return app || (app = require('./http'));
}
});
/**
* Expose the RedisClient factory.
*/
exports.redis = redis;
/**
* Create a new `Queue`.
*
* @return {Queue}
* @api public
*/
exports.createQueue = function( options ) {
if( !Queue.singleton ) {
Queue.singleton = new Queue(options);
}
events.subscribe();
return Queue.singleton;
};
/**
* Store workers
*/
exports.workers = [];
/**
* Initialize a new job `Queue`.
*
* @api public
*/
function Queue( options ) {
options = options || {};
this.name = options.name || 'kue';
this.id = [ 'kue', require('os').hostname(), process.pid ].join(':');
this._options = options;
this.promoter = null;
this.workers = exports.workers;
this.shuttingDown = false;
Job.disableSearch = options.disableSearch !== false;
options.jobEvents !== undefined ? Job.jobEvents = options.jobEvents : '';
redis.configureFactory(options, this);
this.client = Worker.client = Job.client = redis.createClient();
}
/**
* Inherit from `EventEmitter.prototype`.
*/
Queue.prototype.__proto__ = EventEmitter.prototype;
/**
* Create a `Job` with the given `type` and `data`.
*
* @param {String} type
* @param {Object} data
* @return {Job}
* @api public
*/
Queue.prototype.create =
Queue.prototype.createJob = function( type, data ) {
return new Job(type, data);
};
/**
* Proxy to auto-subscribe to events.
*
* @api public
*/
var on = EventEmitter.prototype.on;
Queue.prototype.on = function( event ) {
if( 0 == event.indexOf('job') ) events.subscribe();
return on.apply(this, arguments);
};
/**
* Promote delayed jobs, checking every `ms`,
* defaulting to 1 second.
*
* @params {Number} ms
* @deprecated
*/
Queue.prototype.promote = function( ms, l ) {
console.warn('promote method is deprecated, you don\'t need to call this anymore. You can safely remove it from your code now.');
};
/**
* sets up promotion & ttl timers
*/
Queue.prototype.setupTimers = function() {
if( this.warlock === undefined ) {
this.lockClient = redis.createClient();
this.warlock = new Warlock(this.lockClient);
}
this.checkJobPromotion(this._options.promotion);
this.checkActiveJobTtl(this._options.promotion);
};
/**
* This new method is called by Kue when created
*
* Promote delayed jobs, checking every `ms`,
* defaulting to 1 second.
*
* @params {Number} ms
*/
Queue.prototype.checkJobPromotion = function( promotionOptions ) {
promotionOptions = promotionOptions || {};
var client = this.client
, self = this
, timeout = promotionOptions.interval || 1000
, lockTtl = promotionOptions.lockTtl || 2000
//, lockTtl = timeout
, limit = promotionOptions.limit || 1000;
clearInterval(this.promoter);
this.promoter = setInterval(function() {
self.warlock.lock('promotion', lockTtl, function( err, unlock ) {
if( err ) {
// Something went wrong and we weren't able to set a lock
self.emit('error', err);
return;
}
if( typeof unlock === 'function' ) {
// If the lock is set successfully by this process, an unlock function is passed to our callback.
client.zrangebyscore(client.getKey('jobs:delayed'), 0, Date.now(), 'LIMIT', 0, limit, function( err, ids ) {
if( err || !ids.length ) return unlock();
//TODO do a ZREMRANGEBYRANK jobs:delayed 0 ids.length-1
var doUnlock = _.after(ids.length, unlock);
ids.forEach(function( id ) {
id = client.stripFIFO(id);
Job.get(id, function( err, job ) {
if( err ) return doUnlock();
events.emit(id, 'promotion');
job.inactive(doUnlock);
});
});
});
} else {
// The lock was not established by us, be silent
}
});
}, timeout);
};
Queue.prototype.checkActiveJobTtl = function( ttlOptions ) {
ttlOptions = ttlOptions || {};
var client = this.client
, self = this
, timeout = ttlOptions.interval || 1000
, lockTtl = 2000
, limit = ttlOptions.limit || 1000;
clearInterval(this.activeJobsTtlTimer);
this.activeJobsTtlTimer = setInterval(function() {
self.warlock.lock('activeJobsTTL', lockTtl, function( err, unlock ) {
if( err ) {
// Something went wrong and we weren't able to set a lock
self.emit('error', err);
return;
}
if( typeof unlock === 'function' ) {
// If the lock is set successfully by this process, an unlock function is passed to our callback.
// filter only jobs set with a ttl (timestamped) between a large number and current time
client.zrangebyscore(client.getKey('jobs:active'), 100000, Date.now(), 'LIMIT', 0, limit, function( err, ids ) {
if( err || !ids.length ) return unlock();
var idsRemaining = ids.slice();
var doUnlock = _.after(ids.length, function(){
self.removeAllListeners( 'job ttl exceeded ack' );
waitForAcks && clearTimeout( waitForAcks );
unlock && unlock();
});
self.on( 'job ttl exceeded ack', function( id ) {
idsRemaining.splice( idsRemaining.indexOf( id ), 1 );
doUnlock();
});
var waitForAcks = setTimeout( function(){
idsRemaining.forEach( function( id ){
id = client.stripFIFO(id);
Job.get(id, function( err, job ) {
if( err ) return doUnlock();
job.failedAttempt( { error: true, message: 'TTL exceeded' }, doUnlock );
});
});
}, 1000 );
ids.forEach(function( id ) {
id = client.stripFIFO(id);
events.emit(id, 'ttl exceeded');
});
});
} else {
// The lock was not established by us, be silent
}
});
}, timeout);
};
/**
* Runs a LUA script to diff inactive jobs ZSET cardinality
* and helper pop LIST length each `ms` milliseconds and syncs helper LIST.
*
* @param {Number} ms interval for periodical script runs
* @api public
*/
Queue.prototype.watchStuckJobs = function( ms ) {
var client = this.client
, self = this
, ms = ms || 1000;
var prefix = this.client.prefix;
if( this.client.constructor.name == 'Redis' || this.client.constructor.name == 'Cluster') {
// {prefix}:jobs format is needed in using ioredis cluster to keep they keys in same node
prefix = '{' + prefix + '}';
}
var script =
'local msg = redis.call( "keys", "' + prefix + ':jobs:*:inactive" )\n\
local need_fix = 0\n\
for i,v in ipairs(msg) do\n\
local queue = redis.call( "zcard", v )\n\
local jt = string.match(v, "' + prefix + ':jobs:(.*):inactive")\n\
local pending = redis.call( "LLEN", "' + prefix + ':" .. jt .. ":jobs" )\n\
if queue > pending then\n\
need_fix = need_fix + 1\n\
for j=1,(queue-pending) do\n\
redis.call( "lpush", "' + prefix + ':"..jt..":jobs", 1 )\n\
end\n\
end\n\
end\n\
return need_fix';
clearInterval(this.stuck_job_watch);
client.script('LOAD', script, function( err, sha ) {
if( err ) {
return self.emit('error', err);
}
this.stuck_job_watch = setInterval(function() {
client.evalsha(sha, 0, function( err, fixes ) {
if( err ) return clearInterval(this.stuck_job_watch);
}.bind(this));
}.bind(this), ms);
}.bind(this));
};
/**
* Get setting `name` and invoke `fn(err, res)`.
*
* @param {String} name
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.setting = function( name, fn ) {
fn = fn || noop;
this.client.hget(this.client.getKey('settings'), name, fn);
return this;
};
/**
* Process jobs with the given `type`, invoking `fn(job)`.
*
* @param {String} type
* @param {Number|Function} n
* @param {Function} fn
* @api public
*/
Queue.prototype.process = function( type, n, fn ) {
var self = this;
if( 'function' == typeof n ) fn = n, n = 1;
while( n-- ) {
var worker = new Worker(this, type).start(fn);
worker.id = [ self.id, type, self.workers.length + 1 ].join(':');
worker.on('error', function( err ) {
self.emit('error', err);
});
worker.on('job complete', function( job ) {
// guard against emit after shutdown
if( self.client ) {
self.client.incrby(self.client.getKey('stats:work-time'), job.duration, noop);
}
});
// Save worker so we can access it later
self.workers.push(worker);
}
this.setupTimers();
};
/**
* Graceful shutdown
*
* @param {Number} timeout in milliseconds to wait for workers to finish
* @param {String} type specific worker type to shutdown
* @param {Function} fn callback
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.shutdown = function( timeout, type, fn ) {
var self = this
, n = self.workers.length;
if( arguments.length === 1 ) {
fn = timeout;
type = '';
timeout = null;
} else if( arguments.length === 2 ) {
fn = type;
type = '';
}
var origFn = fn || function() {
};
if( this.shuttingDown && type === '' ) { // a global shutdown already has been called
return fn(new Error('Shutdown already in progress'));
}
if( type === '' ) { // this is a global shutdown call
this.shuttingDown = true;
}
var cleanup = function() {
if( self.shuttingDown ) {
self.workers = [];
exports.workers = [];
self.removeAllListeners();
Queue.singleton = null;
events.unsubscribe();
// destroy redis client and pubsub
redis.reset();
self.client && self.client.quit();
self.client = null;
self.lockClient && self.lockClient.quit();
self.lockClient = null;
}
};
// Wrap `fn` to only call after all workers finished
fn = function( err ) {
if( err ) {
return origFn(err);
}
if( !--n ) {
cleanup();
origFn.apply(null, arguments);
}
};
// shut down promoter interval
if( self.shuttingDown ) {
if( self.promoter ) {
clearInterval(self.promoter);
self.promoter = null;
}
if( self.activeJobsTtlTimer ) {
clearInterval(self.activeJobsTtlTimer);
self.activeJobsTtlTimer = null;
}
}
if( !self.workers.length ) {
cleanup();
origFn();
} else {
// Shut down workers 1 by 1
self.workers.forEach(function( worker ) {
if( self.shuttingDown || worker.type == type ) {
worker.shutdown(timeout, fn);
} else {
fn && fn();
}
});
}
return this;
};
/**
* Get the job types present and callback `fn(err, types)`.
*
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.types = function( fn ) {
fn = fn || noop;
this.client.smembers(this.client.getKey('job:types'), fn);
return this;
};
/**
* Return job ids with the given `state`, and callback `fn(err, ids)`.
*
* @param {String} state
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.state = function( state, fn ) {
var self = this;
this.client.zrange(this.client.getKey('jobs:' + state), 0, -1, function(err,ids){
var fixedIds = [];
ids.forEach(function(id){
fixedIds.push(self.client.stripFIFO(id));
});
fn(err,fixedIds);
});
return this;
};
/**
* Get queue work time in milliseconds and invoke `fn(err, ms)`.
*
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.workTime = function( fn ) {
this.client.get(this.client.getKey('stats:work-time'), function( err, n ) {
if( err ) return fn(err);
fn(null, parseInt(n, 10));
});
return this;
};
/**
* Get cardinality of jobs with given `state` and `type` and callback `fn(err, n)`.
*
* @param {String} type
* @param {String} state
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.cardByType = function( type, state, fn ) {
fn = fn || noop;
this.client.zcard(this.client.getKey('jobs:' + type + ':' + state), fn);
return this;
};
/**
* Get cardinality of `state` and callback `fn(err, n)`.
*
* @param {String} state
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.card = function( state, fn ) {
fn = fn || noop;
this.client.zcard(this.client.getKey('jobs:' + state), fn);
return this;
};
/**
* Completed jobs.
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.complete = function( fn ) {
return this.state('complete', fn);
};
/**
* Failed jobs.
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.failed = function( fn ) {
return this.state('failed', fn);
};
/**
* Inactive jobs (queued).
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.inactive = function( fn ) {
return this.state('inactive', fn);
};
/**
* Active jobs (mid-process).
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.active = function( fn ) {
return this.state('active', fn);
};
/**
* Delayed jobs.
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.delayed = function( fn ) {
return this.state('delayed', fn);
};
/**
* Completed jobs of type `type` count.
* @param {String} type is optional
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.completeCount = function( type, fn ) {
if( 1 == arguments.length ) {
fn = type;
return this.card('complete', fn);
}
return this.cardByType(type, 'complete', fn);
};
/**
* Failed jobs of type `type` count.
* @param {String} type is optional
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.failedCount = function( type, fn ) {
if( 1 == arguments.length ) {
fn = type;
return this.card('failed', fn);
}
return this.cardByType(type, 'failed', fn);
};
/**
* Inactive jobs (queued) of type `type` count.
* @param {String} type is optional
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.inactiveCount = function( type, fn ) {
if( 1 == arguments.length ) {
fn = type;
return this.card('inactive', fn);
}
return this.cardByType(type, 'inactive', fn);
};
/**
* Active jobs (mid-process) of type `type` count.
* @param {String} type is optional
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.activeCount = function( type, fn ) {
if( 1 == arguments.length ) {
fn = type;
return this.card('active', fn);
}
return this.cardByType(type, 'active', fn);
};
/**
* Delayed jobs of type `type` count.
* @param {String} type is optional
* @param {Function} fn
* @return {Queue} for chaining
* @api public
*/
Queue.prototype.delayedCount = function( type, fn ) {
if( 1 == arguments.length ) {
fn = type;
return this.card('delayed', fn);
}
return this.cardByType(type, 'delayed', fn);
};
/**
* Test mode for convenience in test suites
* @api public
*/
Queue.prototype.testMode = require('./queue/test_mode');
================================================
FILE: lib/queue/events.js
================================================
/*!
* kue - events
* Copyright (c) 2013 Automattic <behradz@gmail.com>
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Module dependencies.
*/
var redis = require('../redis');
/**
* Job map.
*/
exports.jobs = {};
/**
* Pub/sub key.
*/
exports.key = 'events';
/**
* Add `job` to the jobs map, used
* to grab the in-process object
* so we can emit relative events.
*
* @param {Job} job
* @api private
*/
exports.callbackQueue = [];
exports.add = function( job, callback ) {
if( job.id ) {
if(!exports.jobs[ job.id ])
exports.jobs[ job.id ] = [];
exports.jobs[ job.id ].push(job);
}
// if (!exports.subscribed) exports.subscribe();
if( !exports.subscribeStarted ) exports.subscribe();
if( !exports.subscribed ) {
exports.callbackQueue.push(callback);
} else {
callback();
}
};
/**
* Remove `job` from the jobs map.
*
* @param {Job} job
* @api private
*/
exports.remove = function( job ) {
delete exports.jobs[ job.id ];
};
/**
* Subscribe to "q:events".
*
* @api private
*/
exports.subscribe = function() {
// if (exports.subscribed) return;
if( exports.subscribeStarted ) return;
var client = redis.pubsubClient();
client.on('message', exports.onMessage);
client.subscribe(client.getKey(exports.key), function() {
exports.subscribed = true;
while( exports.callbackQueue.length ) {
process.nextTick(exports.callbackQueue.shift());
}
});
exports.queue = require('../kue').singleton;
// exports.subscribed = true;
exports.subscribeStarted = true;
};
exports.unsubscribe = function() {
var client = redis.pubsubClient();
client.unsubscribe();
client.removeAllListeners();
exports.subscribeStarted = false;
};
/**
* Message handler.
*
* @api private
*/
exports.onMessage = function( channel, msg ) {
// TODO: only subscribe on {Queue,Job}#on()
msg = JSON.parse(msg);
// map to Job when in-process
var jobs = exports.jobs[ msg.id ];
if( jobs && jobs.length > 0 ) {
for (var i = 0; i < jobs.length; i++) {
var job = jobs[i];
job.emit.apply(job, msg.args);
if( [ 'complete', 'failed' ].indexOf(msg.event) !== -1 ) exports.remove(job);
}
}
// emit args on Queues
msg.args[ 0 ] = 'job ' + msg.args[ 0 ];
msg.args.splice(1, 0, msg.id);
if( exports.queue ) {
exports.queue.emit.apply(exports.queue, msg.args);
}
};
/**
* Emit `event` for for job `id` with variable args.
*
* @param {Number} id
* @param {String} event
* @param {Mixed} ...
* @api private
*/
exports.emit = function( id, event ) {
var client = redis.client()
, msg = JSON.stringify({
id: id, event: event, args: [].slice.call(arguments, 1)
});
client.publish(client.getKey(exports.key), msg, function () {});
};
================================================
FILE: lib/queue/job.js
================================================
/*!
* kue - Job
* Copyright (c) 2013 Automattic <behradz@gmail.com>
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
*/
/**
* Module dependencies.
*/
var EventEmitter = require('events').EventEmitter
, events = require('./events')
, redis = require('../redis')
, _ = require('lodash')
, util = require('util')
, noop = function() {};
/**
* Expose `Job`.
*/
exports = module.exports = Job;
exports.disableSearch = true;
exports.jobEvents = true;
/**
* Search instance.
*/
var search;
function getSearch() {
if( search ) return search;
var reds = require('reds');
reds.createClient = require('../redis').createClient;
return search = reds.createSearch(redis.client().getKey('search'));
}
/**
* Default job priority map.
*/
var priorities = exports.priorities = {
low: 10, normal: 0, medium: -5, high: -10, critical: -15
};
/**
* Map `jobs` by the given array of `ids`.
*
* @param {Object} jobs
* @param {Array} ids
* @return {Array}
* @api private
*/
function map( jobs, ids ) {
var ret = [];
ids.forEach(function( id ) {
if( jobs[ id ] ) ret.push(jobs[ id ]);
});
ret = ret.sort(function( a, b ) {
return parseInt(a.id) - parseInt(b.id);
});
return ret;
}
/**
* Return a function that handles fetching
* of jobs by the ids fetched.
*
* @param {Function} fn
* @param {String} order
* @param {String} jobType
* @return {Function}
* @api private
*/
function get( fn, order, jobType) {
return function( err, ids ) {
if( err ) return fn(err);
var pending = ids.length
, jobs = {};
if( !pending ) return fn(null, ids);
ids.forEach(function( id ) {
id = redis.client().stripFIFO(id); // turn zid back to regular job id
exports.get(id, jobType, function( err, job ) {
if( err ) {
console.error(err);
} else {
jobs[ redis.client().createFIFO(job.id) ] = job;
}
--pending || fn(null, 'desc' == order
? map(jobs, ids).reverse()
: map(jobs, ids));
});
});
}
}
/**
* Get with the range `from`..`to`
* and invoke callback `fn(err, ids)`.
*
* @param {Number} from
* @param {Number} to
* @param {String} order
* @param {Function} fn
* @api public
*/
exports.range = function( from, to, order, fn ) {
redis.client().zrange(redis.client().getKey('jobs'), from, to, get(fn, order));
};
/**
* Get jobs of `state`, with the range `from`..`to`
* and invoke callback `fn(err, ids)`.
*
* @param {String} state
* @param {Number} from
* @param {Number} to
* @param {String} order
* @param {Function} fn
* @api public
*/
exports.rangeByState = function( state, from, to, order, fn ) {
redis.client().zrange(redis.client().getKey('jobs:' + state), from, to, get(fn, order));
};
/**
* Get jobs of `type` and `state`, with the range `from`..`to`
* and invoke callback `fn(err, ids)`.
*
* @param {String} type
* @param {String} state
* @param {Number} from
* @param {Number} to
* @param {String} order
* @param {Function} fn
* @api public
*/
exports.rangeByType = function( type, state, from, to, order, fn ) {
redis.client().zrange(redis.client().getKey('jobs:' + type + ':' + state), from, to, get(fn, order, type));
};
/**
* Get job with `id` and callback `fn(err, job)`.
*
* @param {Number} id
* @param {String} jobType is optional
* @param {Function} fn
* @api public
*/
exports.get = function( id, jobType, fn ) {
if (id === null || id === undefined) {
return fn(new Error('invalid id param'));
}
if (typeof jobType === 'function' && !fn) {
fn = jobType;
jobType = '';
}
var client = redis.client()
, job = new Job;
job.id = id;
job.zid = client.createFIFO(id);
client.hgetall(client.getKey('job:' + job.id), function( err, hash ) {
if( err ) return fn(err);
if( !hash ) {
exports.removeBadJob(job.id, jobType);
return fn(new Error('job "' + job.id + '" doesnt exist'));
}
if( !hash.type ) {
exports.removeBadJob(job.id, jobType);
return fn(new Error('job "' + job.id + '" is invalid'))
}
// TODO: really lame, change some methods so
// we can just merge these
job.type = hash.type;
job._ttl = hash.ttl;
job._delay = hash.delay;
job.priority(Number(hash.priority));
job._progress = hash.progress;
job._attempts = Number(hash.attempts);
job._max_attempts = Number(hash.max_attempts);
job._state = hash.state;
job._error = hash.error;
job.created_at = hash.created_at;
job.promote_at = hash.promote_at;
job.updated_at = hash.updated_at;
job.failed_at = hash.failed_at;
job.started_at = hash.started_at;
job.duration = hash.duration;
job.workerId = hash.workerId;
job._removeOnComplete = hash.removeOnComplete;
try {
if( hash.data ) job.data = JSON.parse(hash.data);
if( hash.result ) job.result = JSON.parse(hash.result);
if( hash.progress_data ) job.progress_data = JSON.parse(hash.progress_data);
if( hash.backoff ) {
var source = 'job._backoff = ' + hash.backoff + ';';
// require('vm').runInContext( source );
eval(source);
}
} catch(e) {
err = e;
}
fn(err, job);
});
};
/**
* Remove all references to an invalid job. Will remove leaky keys in redis keys:TYPE:STATE when
* exports.rangeByType is used.
*
* @param {Number} id
* @param {String} jobType
*/
exports.removeBadJob = function( id, jobType) {
var client = redis.client();
var zid = client.createFIFO(id);
client.multi()
.del(client.getKey('job:' + id + ':log'))
.del(client.getKey('job:' + id))
.zrem(client.getKey('jobs:inactive'), zid)
.zrem(client.getKey('jobs:active'), zid)
.zrem(client.getKey('jobs:complete'), zid)
.zrem(client.getKey('jobs:failed'), zid)
.zrem(client.getKey('jobs:delayed'), zid)
.zrem(client.getKey('jobs'), zid)
.zrem(client.getKey('jobs:' + jobType + ':inactive'), zid)
.zrem(client.getKey('jobs:' + jobType+ ':active'), zid)
.zrem(client.getKey('jobs:' + jobType + ':complete'), zid)
.zrem(client.getKey('jobs:' + jobType + ':failed'), zid)
.zrem(client.getKey('jobs:' + jobType + ':delayed'), zid)
.exec();
if( !exports.disableSearch ) {
getSearch().remove(id);
}
};
/**
* Remove job `id` if it exists and invoke callback `fn(err)`.
*
* @param {Number} id
* @param {Function} fn
* @api public
*/
exports.remove = function( id, fn ) {
fn = fn || noop;
exports.get(id, function( err, job ) {
if( err ) return fn(err);
if( !job ) return fn(new Error('failed to find job ' + id));
job.remove(fn);
});
};
/**
* Get log for job `id` and callback `fn(err, log)`.
*
* @param {Number} id
* @param {Function} fn
* @return {Type}
* @api public
*/
exports.log = function( id, fn ) {
/*redis*/
Job.client/*()*/.lrange(Job.client.getKey('job:' + id + ':log'), 0, -1, fn);
};
/**
* Initialize a new `Job` with the given `type` and `data`.
*
* @param {String} type
* @param {Object} data
* @api public
*/
function Job( type, data ) {
this.type = type;
this.data = data || {};
this._max_attempts = 1;
this._jobEvents = exports.jobEvents;
// this.client = redis.client();
this.client = Job.client/* || (Job.client = redis.client())*/;
this.priority('normal');
this.on('error', function( err ) {
});// prevent uncaught exceptions on failed job errors
}
/**
* Inherit from `EventEmitter.prototype`.
*/
Job.prototype.__proto__ = EventEmitter.prototype;
/**
* Return JSON-friendly object.
*
* @return {Object}
* @api public
*/
Job.prototype.toJSON = function() {
return {
id: this.id
, type: this.type
, data: this.data
, result: this.result
, priority: this._priority
, progress: this._progress || 0
, progress_data: this.progress_data
, state: this._state
, error: this._error
, created_at: this.created_at
, promote_at: this.promote_at
, updated_at: this.updated_at
, failed_at: this.failed_at
, started_at: this.started_at
, duration: this.duration
, delay: this._delay
, workerId: this.workerId
, ttl: this._ttl
, attempts: {
made: Number(this._attempts) || 0
, remaining: this._attempts > 0 ? this._max_attempts - this._attempts : Number(this._max_attempts) || 1
, max: Number(this._max_attempts) || 1
}
};
};
Job.prototype.refreshTtl = function() {
('active' === this.state() && this._ttl > 0)
?
this.client.zadd(this.client.getKey('jobs:' + this.state()), Date.now() + parseInt(this._ttl), this.zid, noop)
:
noop();
};
/**
* Log `str` with sprintf-style variable args or anything (objects,arrays,numbers,etc).
*
* Examples:
*
* job.log('preparing attachments');
* job.log('sending email to %s at %s', user.name, user.email);
* job.log({key: 'some key', value: 10});
* job.log([1,2,3]);
*
* Specifiers:
*
* - %s : string
* - %d : integer
*
* @param {String} str
* @param {Mixed} ...
* @return {Job} for chaining
* @api public
*/
Job.prototype.log = function( str ) {
if(typeof str === 'string') {
var formatted = util.format.apply(util, arguments);
}else{
var formatted = util.inspect(str);
}
this.client.rpush(this.client.getKey('job:' + this.id + ':log'), formatted, noop);
this.set('updated_at', Date.now());
return this;
};
/**
* Set job `key` to `val`.
*
* @param {String} key
* @param {String} val
* @param {String} fn
* @return {Job} for chaining
* @api public
*/
Job.prototype.set = function( key, val, fn ) {
this.client.hset(this.client.getKey('job:' + this.id), key, val, fn || noop);
return this;
};
/**
* Get job `key`
*
* @param {String} key
* @param {Function} fn
* @return {Job} for chaining
* @api public
*/
Job.prototype.get = function( key, fn ) {
this.client.hget(this.client.getKey('job:' + this.id), key, fn || noop);
return this;
};
/**
* Set the job progress by telling the job
* how `complete` it is relative to `total`.
* data can be used to pass extra data to job subscribers
*
* @param {Number} complete
* @param {Number} total
* @param {Object} data
* @return {Job} for chaining
* @api public
*/
Job.prototype.progress = function( complete, total, data ) {
if( 0 == arguments.length ) return this._progress;
var n = Math.min(100, complete * 100 / total | 0);
this.set('progress', n);
// If this stringify fails because of a circular structure, even the one in events.emit would.
// So it does not make sense to try/catch this.
if( data ) this.set('progress_data', JSON.stringify(data));
this.set('updated_at', Date.now());
this.refreshTtl();
events.emit(this.id, 'progress', n, data);
return this;
};
/**
* Set the job delay in `ms`.
*
* @param {Number|Date} delay in ms or execution date
* @return {Job|Number}
* @api public
*/
Job.prototype.delay = function( ms ) {
if( 0 == arguments.length ) return this._delay;
if( _.isDate(ms) ) {
ms = parseInt(ms.getTime() - Date.now())
}
if( ms > 0 ) {
this._delay = ms;
}
return this;
};
/**
* Sets the jobEvents flag for the job.
* Can be used to override the global exports.jobEvents setting
*
* @param {Boolean} events True if job events should be emitted, false if job events should not be emitted.
* @return {Job} Returns `this` for chaining
*/
Job.prototype.events = function (events) {
this._jobEvents = !!events;
return this;
};
Job.prototype.removeOnComplete = function( param ) {
if( 0 == arguments.length ) return this._removeOnComplete;
this._removeOnComplete = param;
return this;
};
Job.prototype.backoff = function( param ) {
if( 0 == arguments.length ) return this._backoff;
this._backoff = param;
return this;
};
/**
*
* @param param
* @returns {*}
*/
Job.prototype.ttl = function( param ) {
if( 0 == arguments.length ) return this._ttl;
if( param > 0 ) {
this._ttl = param;
}
return this;
};
Job.prototype._getBackoffImpl = function() {
var self = this
var supported_backoffs = {
fixed: function( delay ) {
return function( attempts ) {
return delay;
};
}
, exponential: function( delay ) {
return function( attempts ) {
return Math.round(Math.min(
(self._backoff.maxDelay || Infinity) - (delay * 0.5),
delay * 0.5 * ( Math.pow(2, attempts) - 1)
));
};
}
};
if( _.isPlainObject(this._backoff) ) {
return supported_backoffs[ this._backoff.type ](this._backoff.delay || this._delay);
} else {
return this._backoff;
}
};
/**
* Set or get the priority `level`, which is one
* of "low", "normal", "medium", and "high", or
* a number in the range of -10..10.
*
* @param {String|Number} level
* @return {Job|Number} for chaining
* @api public
*/
Job.prototype.priority = function( level ) {
if( 0 == arguments.length ) return this._priority;
this._priority = null == priorities[ level ]
? level
: priorities[ level ];
return this;
};
/**
* Increment attempts, invoking callback `fn(remaining, attempts, max)`.
*
* @param {Function} fn
* @return {Job} for chaining
* @api public
*/
Job.prototype.attempt = function( fn ) {
var client = this.client
, id = this.id
, key = client.getKey('job:' + id);
this._attempts = this._attempts || 0;
if( this._attempts < this._max_attempts ) {
client.hincrby(key, 'attempts', 1, function( err, attempts ) {
this._attempts = attempts;
fn(err, Math.max(0, this._max_attempts - attempts), attempts, this._max_attempts);
}.bind(this));
} else {
fn(null, 0, this._attempts, this._max_attempts);
}
return this;
};
/**
* Try to reattempt the job seand called onFailedAttempt, or call onFailed
* @param remaining total left attempts
* @param attempts
* @param onFailedAttempt
* @param onFailed
* @param clbk
*/
Job.prototype.reattempt = function( attempts, clbk ) {
clbk = clbk || noop;
if( this.backoff() ) {
var delay = this.delay();
if( _.isFunction(this._getBackoffImpl()) ) {
try {
delay = this._getBackoffImpl().apply(this, [ attempts, delay ]);
} catch(e) {
clbk(e);
}
}
var self = this;
this.delay(delay).update(function( err ) {
if( err ) return clbk(err);
self.delayed(clbk);
});
} else {
this.inactive(clbk);
}
};
/**
* Set max attempts to `n`.
*
* @param {Number} n
* @return {Job} for chaining
* @api public
*/
Job.prototype.attempts = function( n ) {
this._max_attempts = n;
return this;
};
Job.prototype.failedAttempt = function( theErr, fn ) {
this.error(theErr).failed(function() {
this.attempt(function( error, remaining, attempts/*, max*/ ) {
if( error ) {
this.emit( 'error', error );
return fn && fn( error );
}
if( remaining > 0 ) {
this.reattempt(attempts, function( err ) {
if( err ) {
this.emit( 'error', err );
return fn && fn( err );
}
fn && fn( err, true, attempts );
}.bind(this));
} else if( remaining === 0 ) {
fn && fn( null, false, attempts );
} else {
fn && fn( new Error('Attempts Exceeded') );
}
}.bind(this));
}.bind(this));
return this;
};
Job.prototype.searchKeys = function( keys ) {
if( 0 == arguments.length ) return this._searchKeys;
this._searchKeys = keys || [];
if( !_.isArray(this._searchKeys) ) {
this._searchKeys = [ this._searchKeys ];
}
return this;
};
/**
* Remove the job and callback `fn(err)`.
*
* @param {Function} fn
* @return {Job} for chaining
* @api public
*/
Job.prototype.remove = function( fn ) {
var client = this.client;
client.multi()
.zrem(client.getKey('jobs:' + this.state()), this.zid)
.zrem(client.getKey('jobs:' + this.type + ':' + this.state()), this.zid)
.zrem(client.getKey('jobs'), this.zid)
.del(client.getKey('job:' + this.id + ':log'))
.del(client.getKey('job:' + this.id))
.exec(function( err ) {
// events.remove(this);
events.emit(this.id, 'remove', this.type);
if( !exports.disableSearch ) {
getSearch().remove(this.id, fn);
} else {
fn && fn(err);
}
}.bind(this));
return this;
};
/**
* Set state to `state`.
*
* @param {String} state
* @param fn
* @return {Job} for chaining
* @api public
*/
Job.prototype.state = function( state, fn ) {
if( 0 == arguments.length ) return this._state;
var client = this.client
, fn = fn || noop;
var oldState = this._state;
var multi = client.multi();
if( oldState && oldState != '' && oldState != state ) {
multi
.zrem(client.getKey('jobs:' + oldState), this.zid)
.zrem(client.getKey('jobs:' + this.type + ':' + oldState), this.zid);
}
multi
.hset(client.getKey('job:' + this.id), 'state', state)
.zadd(client.getKey('jobs:' + state), this._priority, this.zid)
.zadd(client.getKey('jobs:' + this.type + ':' + state), this._priority, this.zid);
// use promote_at as score when job moves to delayed
('delayed' === state) ? multi.zadd(client.getKey('jobs:' + state), parseInt(this.promote_at), this.zid) : noop();
('active' === state && this._ttl > 0) ? multi.zadd(client.getKey('jobs:' + state), Date.now() + parseInt(this._ttl), this.zid) : noop();
('active' === state && !this._ttl) ? multi.zadd(client.getKey('jobs:' + state), this._priority<0?this._priority:-this._priority, this.zid) : noop();
('inactive' === state) ? multi.lpush(client.getKey(this.type + ':jobs'), 1) : noop();
this.set('updated_at', Date.now());
this._state = state;
multi.exec(function( err, replies ) {
if( !err ) {
(this._state === 'inactive') ? events.emit(this.id, 'enqueue', this.type) : noop();
}
return fn(err);
}.bind(this));
return this;
};
/**
* Set the job's failure `err`.
*
* @param {Error} err
* @return {Job} for chaining
* @api public
*/
Job.prototype.error = function( err ) {
var str, summary;
if( 0 == arguments.length ) return this._error;
if( 'string' == typeof err ) {
str = err;
summary = '';
} else {
if( err.stack && 'string' === typeof err.stack ) {
str = err.stack
} else { //TODO what happens to CallSite[] err.stack?
str = err.message
}
summary = ('string' === typeof str) ? str.split('\n')[ 0 ] : '';
}
this.set('error', str);
this.log('%s', summary);
events.emit(this.id, 'error', str);
return this;
};
/**
* Set state to "complete", and progress to 100%.
*/
Job.prototype.complete = function( clbk ) {
return this.set('progress', 100).state('complete', clbk);
};
/**
* Set state to "failed".
*/
Job.prototype.failed = function( clbk ) {
this.failed_at = Date.now();
return this.set('failed_at', this.failed_at).state('failed', clbk);
};
/**
* Set state to "inactive".
*/
Job.prototype.inactive = function( clbk ) {
return this.state('inactive', clbk);
};
/**
* Set state to "active".
*/
Job.prototype.active = function( clbk ) {
return this.state('active', clbk);
};
/**
* Set state to "delayed".
*/
Job.prototype.delayed = function( clbk ) {
return this.state('delayed', clbk);
};
/**
* Save the job, optionally invoking the callback `fn(err)`.
*
* @param {Function} fn
* @return {Job} for chaining
* @api public
*/
Job.prototype.save = function( fn ) {
var client = this.client
, fn = fn || noop
, max = this._max_attempts
, self = this;
// update
if( this.id ) return this.update(fn);
// incr id
client.incr(client.getKey('ids'), function( err, id ) {
if( err ) return fn(err);
// add the job for event mapping
self.id = id;
self.zid = client.createFIFO(id);
self.subscribe(function() {
self._state = self._state || (this._delay ? 'delayed' : 'inactive');
if( max ) { self.set('max_attempts', max); }
client.sadd(client.getKey('job:types'), self.type, noop);
self.set('type', self.type);
var now = Date.now();
self.created_at = now;
self.set('created_at', self.created_at);
self.promote_at = now + (self._delay || 0);
self.set('promote_at', self.promote_at);
self.update(fn);
}.bind(this));
}.bind(this));
return this;
};
/**
* Update the job and callback `fn(err)`.
*
* @param {Function} fn
* @api public
*/
Job.prototype.update = function( fn ) {
var json;
// serialize json data
try {
json = JSON.stringify(this.data);
} catch(err) {
fn(err);
return this;
}
// delay
if( this._delay ) {
this.set('delay', this._delay);
if( this.created_at ) {
var timestamp = parseInt(this.failed_at || this.created_at, 10)
, delay = parseInt(this._delay);
this.promote_at = timestamp + delay;
this.set('promote_at', this.promote_at);
}
}
if( this._ttl ) {
this.set('ttl', this._ttl);
}
if( this._removeOnComplete ) this.set('removeOnComplete', this._removeOnComplete);
if( this._backoff ) {
if( _.isPlainObject(this._backoff) ) this.set('backoff', JSON.stringify(this._backoff));
else this.set('backoff', this._backoff.toString());
}
// updated timestamp
this.set('updated_at', Date.now());
this.refreshTtl();
// priority
this.set('priority', this._priority);
this.client.zadd(this.client.getKey('jobs'), this._priority, this.zid, noop);
// data
this.set('data', json, function() {
// state
this.state(this._state, fn);
}.bind(this));
if( !exports.disableSearch ) {
if( this.searchKeys() ) {
this.searchKeys().forEach(function( key ) {
var value = _.get(this.data, key);
if( !_.isString(value) ) {
value = JSON.stringify(value);
}
getSearch().index(value, this.id);
}.bind(this));
} else {
getSearch().index(json, this.id);
}
}
return this;
};
/**
* Subscribe this job for event mapping.
*
* @return {Job} for chaining
* @api public
*/
Job.prototype.subscribe = function( callback ) {
if( this._jobEvents ) {
events.add(this, callback);
} else {
callback && callback();
}
return this;
};
================================================
FILE: lib/queue/test_mode.js
================================================
var Job = require('./job'),
_ = require('lodash');
var originalJobSave = Job.prototype.save,
originalJobUpdate = Job.prototype.update,
processQueue,
jobs;
function testJobSave( fn ) {
if(processQueue) {
jobs.push(this);
originalJobSave.call(this, fn);
} else {
this.id = _.uniqueId();
jobs.push(this);
if( _.isFunction(fn) ) fn();
}
};
function testJobUpdate( fn ) {
if(processQueue) {
originalJobUpdate.call(this, fn);
} else {
if( _.isFunction(fn) ) fn();
}
};
/**
* Array of jobs added to the queue
* @api public
*/
module.exports.jobs = jobs = [];
module.exports.processQueue = processQueue = false;
/**
* Enable test mode.
* @api public
*/
module.exports.enter = function(process) {
processQueue = process || false;
Job.prototype.save = testJobSave;
Job.prototype.update = testJobUpdate;
};
/**
* Disable test mode.
* @api public
*/
module.exports.exit = function() {
Job.prototype.save = originalJobSave;
Job.prototype.update = originalJobUpdate;
};
/**
* Clear the array of queued jobs
* @api public
*/
module.exports.clear = function() {
jobs.length = 0;
};
================================================
FILE: lib/queue/worker.js
================================================
/*!
* kue - Worker
* Copyright (c) 2013 Automattic <behradz@gmail.com>
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
* Author: behradz@gmail.com
*/
/**
* Module dependencies.
*/
var EventEmitter = require('events').EventEmitter
, redis = require('../redis')
, events = require('./events')
, Job = require('./job')
, noop = function() {};
/**
* Expose `Worker`.
*/
module.exports = Worker;
/**
* Redis connections used by `getJob()` when blocking.
*/
var clients = {};
/**
* Initialize a new `Worker` with the given Queue
* targetting jobs of `type`.
*
* @param {Queue} queue
* @param {String} type
* @api private
*/
function Worker( queue, type ) {
this.queue = queue;
this.type = type;
this.client = Worker.client || (Worker.client = redis.createClient());
this.running = true;
this.job = null;
}
/**
* Inherit from `EventEmitter.prototype`.
*/
Worker.prototype.__proto__ = EventEmitter.prototype;
/**
* Start processing jobs with the given `fn`,
*
* @param {Function} fn
* @return {Worker} for chaining
* @api private
*/
Worker.prototype.start = function( fn ) {
var self = this;
self.idle();
if( !self.running ) return;
if (self.ttlExceededCb)
self.queue.removeListener('job ttl exceeded', self.ttlExceededCb);
self.ttlExceededCb = function(id) {
if( self.job && self.job.id && self.job.id === id ) {
self.failed( self.job, { error: true, message: 'TTL exceeded' }, fn );
events.emit(id, 'ttl exceeded ack');
}
}
/*
listen if current job ttl received,
so that this worker can fail current stuck job and continue,
in case user's process callback is stuck and done is not called in time
*/
this.queue.on( 'job ttl exceeded', self.ttlExceededCb);
self.getJob(function( err, job ) {
if( err ) self.error(err, job);
if( !job || err ) return process.nextTick(function() {
self.start(fn);
});
self.process(job, fn);
});
return this;
};
/**
* Error handler, currently does nothing.
*
* @param {Error} err
* @param {Job} job
* @return {Worker} for chaining
* @api private
*/
Worker.prototype.error = function( err, job ) {
this.emit('error', err, job);
return this;
};
/**
* Process a failed `job`. Set's the job's state
* to "failed" unless more attempts remain, in which
* case the job is marked as "inactive" or "delayed"
* and remains in the queue.
*
* @param {Job} job
* @param {Object} theErr
* @param {Function} fn
* @return {Worker} for chaining
* @api private
*/
Worker.prototype.failed = function( job, theErr, fn ) {
var self = this;
job.failedAttempt( theErr, function( err, hasAttempts, attempt ) {
if( err ) return self.error(err, job);
if( hasAttempts ) {
self.emitJobEvent( 'failed attempt', job, theErr.message || theErr.toString(), attempt );
} else {
self.emitJobEvent( 'failed', job, theErr.message || theErr.toString() );
}
fn && self.start(fn);
});
return this;
};
/**
* Process `job`, marking it as active,
* invoking the given callback `fn(job)`,
* if the job fails `Worker#failed()` is invoked,
* otherwise the job is marked as "complete".
*
* @param {Job} job
* @param {Function} fn
* @return {Worker} for chaining
* @api public
*/
Worker.prototype.process = function( job, fn ) {
var self = this
, start = new Date();
this.job = job;
job.set( 'started_at', job.started_at = start.getTime() );
job.set( 'workerId', job.workerId = this.id );
/*
store job.id around given done to the caller,
so that we can later match against it when done is called
*/
var createDoneCallback = function( jobId ) {
return function( err, result ) {
if( self.drop_user_callbacks ) {
//console.warn( 'Worker started to shutdown, ignoring execution of done callback' );
//job.log( 'Worker started to shutdown, ignoring execution of done callback' );
return;
}
/*
if no job in hand, or the current job in hand
doesn't match called done callback's jobId
then ignore running callers done.
*/
if( self.job === null || self.job && self.job.id && self.job.id !== jobId ) {
//console.warn( 'This job has already been finished, ignoring execution of done callback' );
//job.log( 'This job has already been finished, ignoring execution of done callback' );
return;
}
if( err ) {
return self.failed(job, err, fn);
}
job.set('duration', job.duration = new Date - start);
if( result ) {
try {
job.result = result;
job.set('result', JSON.stringify(result), noop);
} catch(e) {
job.set('result', JSON.stringify({ error: true, message: 'Invalid JSON Result: "' + result + '"' }), noop);
}
}
job.complete(function() {
job.attempt(function() {
if( job.removeOnComplete() ) {
job.remove();
}
self.emitJobEvent('complete', job, result);
self.start(fn);
});
}.bind(this));
};
};
var doneCallback = createDoneCallback( job.id );
var workerCtx = {
/**
* @author behrad
* @pause: let the processor to tell worker not to continue processing new jobs
*/
pause: function( timeout, fn ) {
if( arguments.length === 1 ) {
fn = timeout;
timeout = 5000;
}
self.queue.shutdown(Number(timeout), self.type, fn);
},
/**
* @author behrad
* @pause: let the processor to trigger restart for they job processing
*/
resume: function() {
if( self.resume() ) {
self.start(fn);
}
},
shutdown: function() {
self.shutdown();
}
};
job.active(function() {
self.emitJobEvent('start', job, job.type);
if( fn.length === 2 ) { // user provided a two argument function, doesn't need workerCtx
fn(job, doneCallback);
} else { // user wants workerCtx parameter, make done callback the last
fn(job, workerCtx, doneCallback);
}
}.bind(this));
return this;
};
/**
* Atomic ZPOP implementation.
*
* @param {String} key
* @param {Function} fn
* @api private
*/
Worker.prototype.zpop = function( key, fn ) {
this.client
.multi()
.zrange(key, 0, 0)
.zremrangebyrank(key, 0, 0)
.exec(function( err, res ) {
if( err || !res || !res[ 0 ] || !res[ 0 ].length ) return fn(err);
var id = res[ 0 ][ 0 ] || res[ 0 ][ 1 ][ 0 ];
fn(null, this.client.stripFIFO(id));
}.bind(this));
};
/**
* Attempt to fetch the next job.
*
* @param {Function} fn
* @api private
*/
Worker.prototype.getJob = function( fn ) {
var self = this;
if( !self.running ) {
return fn('Already Shutdown');
}
// alloc a client for this job type
var client = clients[ self.type ] || (clients[ self.type ] = redis.createClient());
// BLPOP indicates we have a new inactive job to process
client.blpop(client.getKey(self.type + ':jobs'), 0, function( err ) {
if( err || !self.running ) {
if( self.client && self.client.connected && !self.client.closing ) {
self.client.lpush(self.client.getKey(self.type + ':jobs'), 1, noop);
}
return fn(err); // SAE: Added to avoid crashing redis on zpop
}
// Set job to a temp value so shutdown() knows to wait
self.job = true;
self.zpop(self.client.getKey('jobs:' + self.type + ':inactive'), function( err, id ) {
if( err || !id ) {
self.idle();
return fn(err /*|| "No job to pop!"*/);
}
Job.get(id, fn);
});
});
};
/**
* emits worker idle event and nullifies current job in hand
*/
Worker.prototype.idle = function() {
this.job = null;
this.emit('idle');
return this;
};
/**
* Gracefully shut down the worker
*
* @param {Function} fn
* @param {int} timeout
* @api private
*/
Worker.prototype.shutdown = function( timeout, fn ) {
var self = this, shutdownTimer = null;
if( arguments.length === 1 ) {
fn = timeout;
timeout = null;
}
// Wrap `fn` so we don't pass `job` to it
var _fn = function( job ) {
if( job && self.job && job.id != self.job.id ) {
return; // simply ignore older job events currently being received until the right one comes...
}
shutdownTimer && clearTimeout(shutdownTimer);
self.removeAllListeners();
self.job = null;
//Safeyly kill any blpop's that are waiting.
(self.type in clients) && clients[ self.type ].quit();
delete clients[ self.type ];
self.cleaned_up = true;
//fix half-blob job fetches if any
self.client.lpush(self.client.getKey(self.type + ':jobs'), 1, fn || noop);
};
if( !this.running ) return _fn();
this.running = false;
// As soon as we're free, signal that we're done
if( !this.job ) {
return _fn();
}
this.on('idle', _fn);
this.on('job complete', _fn);
this.on('job failed', _fn);
this.on('job failed attempt', _fn);
if( timeout ) {
shutdownTimer = setTimeout(function() {
// shutdown timeout reached...
if( self.job ) {
self.drop_user_callbacks = true;
self.removeAllListeners();
if( self.job === true ) {
self.once('idle', _fn);
} else {
// a job is running, fail it and call _fn when failed
self.once('job failed', _fn);
self.once('job failed attempt', _fn);
self.failed(self.job, { error: true, message: 'Shutdown' });
}
} else {
// no job running, just finish immediately
_fn();
}
}.bind(this), timeout);
}
};
Worker.prototype.emitJobEvent = function( event, job, arg1, arg2 ) {
if( this.cleaned_up ) return;
events.emit(job.id, event, arg1, arg2);
this.emit('job ' + event, job);
};
Worker.prototype.resume = function() {
if( this.running ) return false;
this.cleaned_up = false;
this.drop_user_callbacks = false;
this.running = true;
return true;
};
================================================
FILE: lib/redis.js
================================================
/*!
* kue - RedisClient factory
* Copyright (c) 2013 Automattic <behradz@gmail.com>
* Copyright (c) 2011 LearnBoost <tj@learnboost.com>
* MIT Licensed
* Author: behradz@gmail.com
*/
/**
* Module dependencies.
*/
var redis = require('redis');
var url = require('url');
/**
*
* @param options
* @param queue
*/
exports.configureFactory = function( options, queue ) {
options.prefix = options.prefix || 'q';
if( typeof options.redis === 'string' ) {
// parse the url
var conn_info = url.parse(options.redis, true /* parse query string */);
if( conn_info.protocol !== 'redis:' ) {
throw new Error('kue connection string must use the redis: protocol');
}
options.redis = {
port: conn_info.port || 6379,
host: conn_info.hostname,
db: (conn_info.pathname ? conn_info.pathname.substr(1) : null) || conn_info.query.db || 0,
// see https://github.com/mranney/node_redis#rediscreateclient
options: conn_info.query
};
if( conn_info.auth ) {
options.redis.auth = conn_info.auth.replace(/.*?:/, '');
}
}
options.redis = options.redis || {};
// guarantee that redis._client has not been populated.
// may warrant some more testing - i was running into cases where shutdown
// would call redis.reset but an event would be emitted after the reset
// which would re-create the client and cache it in the redis module.
exports.reset();
/**
* Create a RedisClient.
*
* @return {RedisClient}
* @api private
*/
exports.createClient = function() {
var clientFactoryMethod = options.redis.createClientFactory || exports.createClientFactory;
var client = clientFactoryMethod(options);
client.on('error', function( err ) {
queue.emit('error', err);
});
client.prefix = options.prefix;
// redefine getKey to use the configured prefix
client.getKey = function( key ) {
if( client.constructor.name == 'Redis' || client.constructor.name == 'Cluster') {
// {prefix}:jobs format is needed in using ioredis cluster to keep they keys in same node
// otherwise multi commands fail, since they use ioredis's pipeline.
return '{' + this.prefix + '}:' + key;
}
return this.prefix + ':' + key;
};
client.createFIFO = function( id ) {
//Create an id for the zset to preserve FIFO order
var idLen = '' + id.toString().length;
var len = 2 - idLen.length;
while (len--) idLen = '0' + idLen;
return idLen + '|' + id;
};
// Parse out original ID from zid
client.stripFIFO = function( zid ) {
if ( typeof zid === 'string' ) {
return +zid.substr(zid.indexOf('|')+1);
} else {
// Sometimes this gets called with an undefined
// it seems to be OK to have that not resolve to an id
return zid;
}
};
return client;
};
};
/**
* Create a RedisClient from options
* @param options
* @return {RedisClient}
* @api private
*/
exports.createClientFactory = function( options ) {
var socket = options.redis.socket;
var port = !socket ? (options.redis.port || 6379) : null;
var host = !socket ? (options.redis.host || '127.0.0.1') : null;
var db = !socket ? (options.redis.db || 0) : null;
var client = redis.createClient(socket || port, host, options.redis.options);
if( options.redis.auth ) {
client.auth(options.redis.auth);
}
if( db >= 0 ){
client.select(db);
}
return client;
};
/**
* Create or return the existing RedisClient.
*
* @return {RedisClient}
* @api private
*/
exports.client = function() {
return exports._client || (exports._client = exports.createClient());
};
/**
* Return the pubsub-specific redis client.
*
* @return {RedisClient}
* @api private
*/
exports.pubsubClient = function() {
return exports._pubsub || (exports._pubsub = exports.createClient());
};
/**
* Resets internal variables to initial state
*
* @api private
*/
exports.reset = function() {
exports._client && exports._client.quit();
exports._pubsub && exports._pubsub.quit();
exports._client = null;
exports._pubsub = null;
};
================================================
FILE: package.json
================================================
{
"name": "kue",
"version": "0.11.5",
"description": "Feature rich priority job queue backed by redis",
"homepage": "http://automattic.github.io/kue/",
"keywords": [
"job",
"queue",
"worker",
"redis"
],
"license": "MIT",
"author": "TJ Holowaychuk <tj@learnboost.com>",
"contributors": [
{
"name": "Behrad Zari",
"email": "behradz@gmail.com"
}
],
"repository": {
"type": "git",
"url": "https://github.com/Automattic/kue.git"
},
"bugs": {
"url": "https://github.com/Automattic/kue/issues"
},
"dependencies": {
"body-parser": "^1.12.2",
"express": "^4.12.2",
"lodash": "^4.0.0",
"nib": "~1.1.2",
"node-redis-warlock": "~0.2.0",
"pug": "^2.0.0-beta3",
"redis": "~2.6.0-2",
"stylus": "~0.54.5",
"yargs": "^4.0.0"
},
"devDependencies": {
"async": "^1.4.2",
"chai": "^3.3.0",
"coffee-script": "~1.10.0",
"mocha": "^2.3.3",
"should": "^3.1.0",
"sinon": "^1.17.2",
"supertest": "^1.1.0"
},
"main": "index",
"bin": {
"kue-dashboard": "bin/kue-dashboard"
},
"scripts": {
"test": "make test-all"
},
"optionalDependencies": {
"reds": "^0.2.5"
}
}
================================================
FILE: test/jsonapi.js
================================================
var request = require( 'supertest' ),
kue = require( '../index' ),
async = require( 'async' ),
chai = require( 'chai' ),
queue = kue.createQueue( { disableSearch: false } ), //customize queue before accessing kue.app
app = kue.app,
type = 'test:inserts';
expect = chai.expect;
function jobsPopulate( count ) {
var priority = [ 10, 0, -5, -10, -15 ],
jobs = [];
for ( var i = 0; i < count; i++ ) {
jobs.push( {
type: type,
data: {
title: i,
data: type + ':data'
},
options: {
// random priority
priority: priority[ Math.floor( Math.random() * 5 ) ]
}
} );
}
// return array only if length > 1
return jobs.length === 1 ? jobs[ 0 ] : jobs;
}
describe( 'JSON API', function () {
var scope = {};
before( function ( done ) {
scope.queue = queue;
// delete all jobs to get a clean state
kue.Job.rangeByType( type, 'inactive', 0, 100, 'asc', function ( err, jobs ) {
if ( err ) return done( err );
if ( !jobs.length ) return done();
async.each( jobs, function ( job, asyncDone ) {
job.remove( asyncDone );
}, done );
} );
} );
after( function ( done ) {
scope.queue.shutdown( 200, function ( err ) {
scope.queue = null;
done( err );
} );
} );
describe( 'create, get, update and delete', function () {
it( 'should insert a job and respond with an id', function ( done ) {
request( app )
.post( '/job' )
.send( jobsPopulate( 1 ) )
.expect( 200 )
.expect( function ( res ) {
res.body.message.should.equal( 'job created' );
res.body.id.should.be.a.Number;
Object.keys( res.body ).should.have.lengthOf( 2 );
scope.jobId = res.body.id;
} )
.end( done );
} );
it( 'should insert multiple jobs and respond with ids', function ( done ) {
var jobCount = 5;
request( app )
.post( '/job' )
.send( jobsPopulate( jobCount ) )
.expect( 200 )
.expect( function ( res ) {
var created = res.body;
created.should.be.ok;
created.length.should.equal( jobCount );
for ( var i = 0; i < jobCount; i++ ) {
var job = created[ i ];
job.message.should.be.equal( 'job created' );
job.id.should.be.a.Number;
Object.keys( job ).should.have.lengthOf( 2 );
}
} )
.end( done );
} );
it( 'get job by id: job is inactive', function ( done ) {
request( app )
.get( '/job/' + scope.jobId )
.expect( function ( res ) {
res.body.id.should.eql( scope.jobId );
res.body.type.should.eql( type );
res.body.state.should.eql( 'inactive' );
} )
.end( done );
} );
it( 'change state', function ( done ) {
request( app )
.put( '/job/' + scope.jobId + '/state/active' )
.expect( function ( res ) {
expect( res.body.message ).to.exist;
} )
.end( done );
} );
it( 'get job by id: job is now active', function ( done ) {
request( app )
.get( '/job/' + scope.jobId )
.expect( function ( res ) {
res.body.id.should.eql( scope.jobId );
res.body.type.should.eql( type );
res.body.state.should.eql( 'active' );
} )
.end( done );
} );
it( 'delete job by id', function ( done ) {
request( app )
.del( '/job/' + scope.jobId )
.expect( function ( res ) {
expect( res.body.message ).to.contain( scope.jobId );
} )
.end( done );
} );
} );
describe( 'search', function () {
it( 'search by query: not found', function ( done ) {
request( app )
.get( '/job/search' )
.query( {} )
.expect( function ( res ) {
res.body.length.should.eql( 0 );
} )
.end( done );
} );
it( 'search by query: found', function ( done ) {
request( app )
.get( '/job/search' )
.query( {
q: type + ':data'
} )
.expect( function ( res ) {
// we created 6 jobs, one was deleted, 5 left
res.body.length.should.eql( 5 );
} )
.end( done );
} );
} );
describe( 'range', function () {
it( 'range from...to', function ( done ) {
request( app )
.get( '/jobs/0..3' )
.expect( function ( res ) {
res.body.length.should.eql( 4 );
} )
.end( done );
} );
it( 'range from...to with type and state', function ( done ) {
request( app )
.get( '/jobs/' + type + '/inactive/0..20/asc' )
.expect( function ( res ) {
res.body.length.should.eql( 5 );
} )
.end( done );
} );
} );
describe( 'stats', function () {
it( 'get stats', function ( done ) {
request( app )
.get( '/stats' )
.expect( function ( res ) {
expect( res.body.inactiveCount ).to.exist;
expect( res.body.completeCount ).to.exist;
expect( res.body.activeCount ).to.exist;
expect( res.body.delayedCount ).to.exist;
} )
.end( done );
} );
} );
describe( 'error cases', function () {
it( 'should return 204 status code when POST /job body is empty', function ( done ) {
request( app )
.post( '/job' )
.send( [] )
.expect( 204 )
.expect( function ( res ) {
res.text.should.have.lengthOf( 0 );
} )
.end( done );
} );
it( 'should insert jobs including an invalid job, respond with ids and error', function ( done ) {
var jobs = jobsPopulate( 3 );
delete jobs[ 1 ].type;
request( app )
.post( '/job' )
.send( jobs )
.expect( 400 ) // Expect a bad request
.expect( function ( res ) {
var created = res.body;
created.should.be.ok;
created.length.should.equal( 3 ); // should still have 3 objects in the response
// The first one succeeded
created[ 0 ].message.should.be.equal( 'job created' );
created[ 0 ].id.should.be.a.Number;
Object.keys( created[ 0 ] ).should.have.lengthOf( 2 );
// The second one failed
created[ 1 ].error.should.equal( 'Must provide job type' );
Object.keys( created[ 1 ] ).should.have.lengthOf( 1 );
// The third one succeeded
created[ 2 ].message.should.be.equal( 'job created' );
created[ 2 ].id.should.be.a.Number;
Object.keys( created[ 2 ] ).should.have.lengthOf( 2 );
} )
.end( done );
} );
} );
} );
================================================
FILE: test/mocha.opts
================================================
--compilers coffee:coffee-script
--require should
--reporter spec
--ui bdd
--timeout 10000
================================================
FILE: test/prefix.coffee
================================================
kue = require '../'
describe 'Kue - Prefix', ->
makeJobs = (queueName) ->
opts =
prefix: queueName
promotion:
interval: 10
jobs = kue.createQueue opts
return jobs
stopJobs = (jobs, callback) ->
jobs.shutdown callback
# expected redis activity
#
# 1397744169.196792 "subscribe" "q:events"
# 1397744169.196852 "unsubscribe"
it 'should use prefix q by default', (done) ->
jobs = kue.createQueue()
jobs.client.prefix.should.equal 'q'
stopJobs jobs, done
# expected redis activity
#
# 1397744498.330456 "subscribe" "testPrefix1:events"
# 1397744498.330638 "unsubscribe"
# 1397744498.330907 "subscribe" "testPrefix2:events"
# 1397744498.331148 "unsubscribe"
it 'should accept and store prefix', (done) ->
jobs = makeJobs('testPrefix1')
jobs.client.prefix.should.equal 'testPrefix1'
stopJobs jobs, (err) ->
jobs2 = makeJobs('testPrefix2')
jobs2.client.prefix.should.equal 'testPrefix2'
stopJobs jobs2, done
it 'should process and complete a job using a prefix', (testDone) ->
jobs = makeJobs('simplePrefixTest')
job = jobs.create('simplePrefixJob')
job.on 'complete', () ->
stopJobs jobs, testDone
job.save()
jobs.process 'simplePrefixJob', (job, done) ->
done()
# expected redis activity
#
# 1397744498.333423 "subscribe" "jobCompleteTest:events"
# 1397744498.334002 "info"
# 1397744498.334358 "zcard" "jobCompleteTest:jobs:inactive"
# 1397744498.335262 "info"
# 1397744498.335578 "incr" "jobCompleteTest:ids"
# etc...
it 'store queued jobs in different prefixes', (testDone) ->
jobs = makeJobs('jobCompleteTest')
jobs.inactiveCount (err, count) ->
prevCount = count
jobs.create( 'fakeJob', {} ).save()
f = ->
jobs.inactiveCount (err, count) ->
count.should.equal prevCount + 1
stopJobs jobs, testDone
setTimeout f, 10
it 'should not pick up an inactive job from another prefix', (testDone) ->
jobs = makeJobs('inactiveJobs')
# create a job but do not process
job = jobs.create('inactiveJob', {} ).save (err) ->
# stop the 'inactiveJobs' prefix
stopJobs jobs, (err) ->
jobs = makeJobs('inactiveJobs2')
# verify count of inactive jobs is 0 for this prefix
jobs.inactiveCount (err, count) ->
count.should.equal 0
stopJobs jobs, testDone
it 'should properly switch back to default queue', (testDone) ->
jobs = makeJobs('notDefault')
stopJobs jobs, (err) ->
jobs = kue.createQueue()
job = jobs.create('defaultPrefixJob')
job.on 'complete', () ->
stopJobs jobs, testDone
job.save()
jobs.process 'defaultPrefixJob', (job, done) ->
done()
================================================
FILE: test/shutdown.coffee
================================================
should = require 'should'
kue = require '../'
describe 'Kue', ->
before (done) ->
jobs = kue.createQueue()
jobs.client.flushdb done
after (done) ->
jobs = kue.createQueue()
jobs.client.flushdb done
describe 'Shutdown', ->
it 'should return singleton from createQueue', (done) ->
jobs = kue.createQueue()
jobsToo = kue.createQueue()
jobs.should.equal jobsToo
jobs.shutdown done
it 'should destroy singleton on shutdown', (done) ->
jobs = kue.createQueue()
jobs.shutdown (err) ->
# test that new jobs object is a different reference
newJobs = kue.createQueue()
newJobs.should.not.equal jobs
newJobs.shutdown done
it 'should clear properties on shutdown', (done) ->
jobs = kue.createQueue({promotion:{interval:200}})
jobs.shutdown (err) ->
should(jobs.workers).be.empty
should(jobs.client).be.empty
should(jobs.promoter).be.empty
done()
it 'should be able to pause/resume the worker', (done) ->
jobs = kue.createQueue()
job_data =
title: 'resumable jobs'
to: 'tj@learnboost.com'
total_jobs = 3
for i in [0...total_jobs]
jobs.create('resumable-jobs', job_data).save()
jobs.process 'resumable-jobs', 1, (job, ctx, job_done) ->
job_done()
if( !--total_jobs )
jobs.shutdown 1000, done
else
ctx.pause()
setTimeout ctx.resume, 100
it 'should not clear properties on single type shutdown', (testDone) ->
jobs = kue.createQueue()
fn = (err) ->
jobs.client.should.not.be.empty
jobs.shutdown 10, testDone
jobs.shutdown 10, 'fooJob', fn
it 'should shutdown one worker type on single type shutdown', (testDone) ->
jobs = kue.createQueue()
# set up two worker types
jobs.process 'runningTask', (job, done) ->
done()
jobs.workers.should.have.length 1
jobs.process 'shutdownTask', (job, done) ->
done()
jobs.workers.should.have.length 2
fn = (err) ->
# verify shutdownTask is not running but runningTask is
for worker in jobs.workers
switch worker.type
when 'shutdownTask'
worker.should.have.property 'running', false
when 'runningTask'
worker.should.have.property 'running', true
# kue should still be running
jobs.promoter.should.not.be.empty
jobs.client.should.not.be.empty
jobs.shutdown 10, testDone
jobs.shutdown 10, 'shutdownTask', fn
it 'should fail active job when shutdown timer expires', (testDone) ->
jobs = kue.createQueue()
jobId = null
jobs.process 'long-task', (job, done) ->
jobId = job.id
fn = ->
done()
setTimeout fn, 10000
jobs.create('long-task', {}).save()
# need to make sure long-task has had enough time to get into active state
waitForJobToRun = ->
fn = (err) ->
kue.Job.get jobId, (err, job) ->
job.should.have.property '_state', "failed"
job.should.have.property '_error', "Shutdown"
testDone()
# shutdown timer is shorter than job length
jobs.shutdown 10, fn
setTimeout waitForJobToRun, 50
it 'should not call graceful shutdown twice on subsequent calls', (testDone) ->
jobs = kue.createQueue()
jobs.process 'test-subsequent-shutdowns', (job, done) ->
done()
setTimeout ()->
jobs.shutdown 100, (err)->
should.not.exist(err)
, 50
setTimeout ()->
jobs.shutdown 100, (err)->
should.exist err, 'expected `err` to exist'
err.should.be.an.instanceOf(Error)
.with.property('message', 'Shutdown already in progress')
testDone()
, 60
jobs.create('test-subsequent-shutdowns', {}).save()
it 'should fail active re-attemptable job when shutdown timer expires', (testDone) ->
jobs = kue.createQueue()
jobId = null
jobs.process 'shutdown-reattemptable-jobs', (job, done) ->
jobId = job.id
setTimeout done, 500
jobs.create('shutdown-reattemptable-jobs', { title: 'shutdown-reattemptable-jobs' }).attempts(2).save()
# need to make sure long-task has had enough time to get into active state
waitForJobToRun = ->
fn = (err) ->
kue.Job.get jobId, (err, job) ->
job.should.have.property '_state', "inactive"
job.should.have.property '_attempts', "1"
job.should.have.property '_error', "Shutdown"
testDone()
# shutdown timer is shorter than job length
jobs.shutdown 100, fn
setTimeout waitForJobToRun, 50
================================================
FILE: test/tdd/kue.spec.js
================================================
var sinon = require('sinon');
var kue = require('../../lib/kue');
var redis = require('../../lib/redis');
var events = require('../../lib/queue/events');
var Job = require('../../lib/queue/job');
var Worker = require('../../lib/queue/worker');
var _ = require('lodash');
var EventEmitter = require('events').EventEmitter;
var redisClient = {};
describe('Kue', function () {
beforeEach(function(){
sinon.stub(events, 'subscribe');
sinon.stub(redis, 'configureFactory', function () {
redis.createClient = sinon.stub();
});
});
afterEach(function(){
events.subscribe.restore();
redis.configureFactory.restore();
});
describe('Function: createQueue', function () {
it('should subscribe to queue events', function () {
var queue = kue.createQueue();
events.subscribe.called.should.be.true;
});
it('should set the correct default values', function () {
var queue = kue.createQueue();
queue.name.should.equal('kue');
queue.id.should.equal([ 'kue', require("os").hostname(), process.pid ].join(':'));
(queue.promoter === null).should.be.true;
queue.workers.should.eql(kue.workers);
queue.shuttingDown.should.be.false;
});
it('should allow a custom name option', function () {
it('should set the correct default values', function () {
var queue = kue.createQueue({
name: 'name'
});
queue.name.should.equal('name');
});
});
});
describe('Function: create', function() {
var queue;
beforeEach(function(){
queue = kue.createQueue();
});
it('should return a new Job instance', function () {
var data = {
key: 'value'
};
var job = queue.create('type', data);
job.type.should.equal('type');
job.data.should.eql(data);
});
});
describe('Function: on', function() {
var queue, noop;
beforeEach(function(){
queue = kue.createQueue();
events.subscribe.reset();
noop = function () {};
});
it('should subscribe to events when subscribing to the job event', function () {
queue.on('job', noop);
events.subscribe.called.should.be.true;
});
it('should proxy the event listener', function (done) {
queue.on('event', function (data) {
data.should.equal('data');
done();
});
queue.emit('event', 'data');
});
});
describe('Function: setupTimers', function() {
var queue;
beforeEach(function(){
queue = kue.createQueue();
sinon.stub(queue, 'checkJobPromotion');
sinon.stub(queue, 'checkActiveJobTtl');
});
afterEach(function(){
queue.checkJobPromotion.restore();
queue.checkActiveJobTtl.restore();
});
it('should setup a warlock client if it is not setup yet', function () {
queue.warlock = undefined;
queue.setupTimers();
queue.warlock.should.exist;
});
it('should call checkJobPromotion', function () {
queue.setupTimers();
queue.checkJobPromotion.called.should.be.true;
});
it('should call checkActiveJobTtl', function () {
queue.setupTimers();
queue.checkActiveJobTtl.called.should.be.true;
});
});
describe('Function: checkJobPromotion', function() {
var queue, unlock, clock, timeout, client, ids, job;
beforeEach(function(){
unlock = sinon.spy();
timeout = 1000;
ids = [1, 2, 3];
client = {
zrangebyscore: sinon.stub().callsArgWith(6, null, ids),
getKey: sinon.stub().returnsArg(0),
stripFIFO: sinon.stub().returnsArg(0)
};
job = {
inactive: sinon.stub().callsArg(0)
};
queue = kue.createQueue();
queue.client = client;
sinon.stub(Job, 'get').callsArgWith(1, null, job);
sinon.stub(queue.warlock, 'lock').callsArgWith(2, null, unlock);
sinon.stub(events, 'emit');
clock = sinon.useFakeTimers();
});
afterEach(function(){
Job.get.restore();
queue.warlock.lock.restore();
events.emit.restore();
clock.restore();
});
it('should set the promotion lock', function () {
queue.checkJobPromotion();
clock.tick(timeout);
queue.warlock.lock.calledWith('promotion', 2000).should.be.true;
});
it('should allow an override for the lockTtl', function () {
queue.checkJobPromotion({ lockTtl: 5000 });
clock.tick(timeout);
queue.warlock.lock.calledWith('promotion', 5000).should.be.true;
});
it('should load all delayed jobs that should be run job', function () {
queue.checkJobPromotion();
clock.tick(timeout);
client.zrangebyscore.calledWith(client.getKey('jobs:delayed'), 0, sinon.match.any, "LIMIT", 0, 1000).should.be.true;
});
it('should get each job', function () {
queue.checkJobPromotion();
clock.tick(timeout);
Job.get.callCount.should.equal(3);
Job.get.calledWith(ids[0]).should.be.true;
Job.get.calledWith(ids[1]).should.be.true;
Job.get.calledWith(ids[2]).should.be.true;
});
it('should emit promotion for each job', function () {
queue.checkJobPromotion();
clock.tick(timeout);
events.emit.callCount.should.equal(3);
events.emit.calledWith(ids[0], 'promotion').should.be.true;
events.emit.calledWith(ids[1], 'promotion').should.be.true;
events.emit.calledWith(ids[2], 'promotion').should.be.true;
});
it('should set each job to inactive', function () {
queue.checkJobPromotion();
clock.tick(timeout);
job.inactive.callCount.should.equal(3);
});
it('should unlock promotion', function () {
queue.checkJobPromotion();
clock.tick(timeout);
unlock.calledOnce.should.be.true;
});
gitextract_wlr7h9ez/
├── .gitignore
├── .npmignore
├── .travis.yml
├── History.md
├── LICENSE
├── Makefile
├── Readme.md
├── bin/
│ └── kue-dashboard
├── examples/
│ ├── delayed.js
│ ├── events.js
│ ├── many.js
│ ├── shutdown.js
│ ├── stale.js
│ └── video.js
├── index.js
├── lib/
│ ├── http/
│ │ ├── index.js
│ │ ├── middleware/
│ │ │ └── provides.js
│ │ ├── public/
│ │ │ ├── javascripts/
│ │ │ │ ├── caustic.js
│ │ │ │ ├── job.js
│ │ │ │ ├── jquery.ext.js
│ │ │ │ ├── loading.js
│ │ │ │ ├── main.js
│ │ │ │ ├── progress.js
│ │ │ │ ├── search.js
│ │ │ │ └── utils.js
│ │ │ └── stylesheets/
│ │ │ ├── actions.styl
│ │ │ ├── config.styl
│ │ │ ├── context-menu.styl
│ │ │ ├── error.styl
│ │ │ ├── job.styl
│ │ │ ├── main.css
│ │ │ ├── main.styl
│ │ │ ├── menu.styl
│ │ │ ├── mixins.styl
│ │ │ └── scrollbar.styl
│ │ ├── routes/
│ │ │ ├── index.js
│ │ │ └── json.js
│ │ └── views/
│ │ ├── _filter.pug
│ │ ├── _job.pug
│ │ ├── _menu.pug
│ │ ├── _row.pug
│ │ ├── _search.pug
│ │ ├── _sort.pug
│ │ ├── job/
│ │ │ └── list.pug
│ │ └── layout.pug
│ ├── kue.js
│ ├── queue/
│ │ ├── events.js
│ │ ├── job.js
│ │ ├── test_mode.js
│ │ └── worker.js
│ └── redis.js
├── package.json
└── test/
├── jsonapi.js
├── mocha.opts
├── prefix.coffee
├── shutdown.coffee
├── tdd/
│ ├── kue.spec.js
│ └── redis.spec.js
├── test.coffee
├── test.js
└── test_mode.js
SYMBOL INDEX (44 symbols across 18 files)
FILE: examples/events.js
function create (line 14) | function create() {
function next (line 40) | function next( i ) {
function convertFrame (line 54) | function convertFrame( i, fn ) {
FILE: examples/many.js
function create (line 8) | function create() {
function create2 (line 18) | function create2() {
FILE: examples/shutdown.js
function generateJobs (line 6) | function generateJobs() {
FILE: examples/stale.js
function create (line 16) | function create() {
function next (line 32) | function next( i ) {
function convertFrame (line 46) | function convertFrame( i, fn ) {
FILE: examples/video.js
function create (line 15) | function create() {
function next (line 31) | function next( i ) {
function convertFrame (line 45) | function convertFrame( i, fn ) {
FILE: lib/http/index.js
function compile (line 31) | function compile( str, path ) {
FILE: lib/http/public/javascripts/caustic.js
function EventEmitter (line 11) | function EventEmitter() {
function callback (line 68) | function callback(fn) {
function View (line 90) | function View(name) {
FILE: lib/http/public/javascripts/job.js
function Job (line 13) | function Job(data) {
FILE: lib/http/public/javascripts/loading.js
function LoadingIndicator (line 11) | function LoadingIndicator() {
FILE: lib/http/public/javascripts/main.js
function init (line 51) | function init(state) {
function showLoading (line 85) | function showLoading() {
function hideLoading (line 97) | function hideLoading() {
function infiniteScroll (line 106) | function infiniteScroll() {
function show (line 135) | function show(state) {
function pollForJobs (line 158) | function pollForJobs(state, ms) {
function refreshJobs (line 176) | function refreshJobs(state, fn) {
function pollStats (line 240) | function pollStats(ms) {
function request (line 260) | function request(url, fn) {
function error (line 286) | function error(msg) {
FILE: lib/http/public/javascripts/progress.js
function Progress (line 11) | function Progress() {
FILE: lib/http/public/javascripts/utils.js
function relative (line 14) | function relative(ms) {
function priority (line 56) | function priority(job) {
function options (line 68) | function options(obj, selected) {
FILE: lib/http/routes/json.js
function getSearch (line 21) | function getSearch() {
function _create (line 158) | function _create( args, next ) {
function get (line 305) | function get( obj ) {
FILE: lib/kue.js
function Queue (line 88) | function Queue( options ) {
FILE: lib/queue/job.js
function getSearch (line 35) | function getSearch() {
function map (line 59) | function map( jobs, ids ) {
function get (line 81) | function get( fn, order, jobType) {
function Job (line 289) | function Job( type, data ) {
FILE: lib/queue/test_mode.js
function testJobSave (line 9) | function testJobSave( fn ) {
function testJobUpdate (line 20) | function testJobUpdate( fn ) {
FILE: lib/queue/worker.js
function Worker (line 40) | function Worker( queue, type ) {
FILE: test/jsonapi.js
function jobsPopulate (line 13) | function jobsPopulate( count ) {
Condensed preview — 61 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (254K chars).
[
{
"path": ".gitignore",
"chars": 64,
"preview": ".idea\n.DS_Store\nnode_modules\n*.sock\n*.rdb\ntest/incomplete\n*.swp\n"
},
{
"path": ".npmignore",
"chars": 29,
"preview": "support\ntest\nexamples\n*.sock\n"
},
{
"path": ".travis.yml",
"chars": 89,
"preview": "language: node_js\nnode_js:\n - \"4\"\n - \"6\"\n - \"8\"\n - \"node\"\nservices:\n - redis-server\n"
},
{
"path": "History.md",
"chars": 12417,
"preview": "0.11.5 / 2016-11-05\n===================\n\n* Fix even more redis command callbacks\n* Fix redis commands SLC integration #9"
},
{
"path": "LICENSE",
"chars": 1091,
"preview": "The MIT License\n\nCopyright (c) 2011 LearnBoost <tj@learnboost.com>\n\nPermission is hereby granted, free of charge, to any"
},
{
"path": "Makefile",
"chars": 650,
"preview": "REPORTER = spec\n\nall: build\n\nbuild:\n\t@./node_modules/coffee-script/bin/coffee \\\n -c \\\n -o lib src\n\ntest-tdd:\n\t@"
},
{
"path": "Readme.md",
"chars": 33546,
"preview": "# Kue\n\n## Kue is no longer maintained\n\nPlease see e.g. [Bull](https://github.com/OptimalBits/bull) as an alternative. Th"
},
{
"path": "bin/kue-dashboard",
"chars": 522,
"preview": "#!/usr/bin/env node\nvar kue = require('kue');\nvar argv = require('yargs')\n\t.usage('Usage: $0 [options]')\n\t.example('$0 -"
},
{
"path": "examples/delayed.js",
"chars": 855,
"preview": "var kue = require( '../' );\n\n// create our job queue\n\nvar jobs = kue.createQueue();\n\n// one minute\n\nvar minute = 60000;\n"
},
{
"path": "examples/events.js",
"chars": 1436,
"preview": "var kue = require( '../' );\n\n// create our job queue\n\nvar jobs = kue.createQueue();\n\n// start redis with $ redis-server\n"
},
{
"path": "examples/many.js",
"chars": 1083,
"preview": "var kue = require( '../' )\n , express = require( 'express' );\n\n// create our job queue\n\nvar jobs = kue.createQueue("
},
{
"path": "examples/shutdown.js",
"chars": 712,
"preview": "var kue = require( '../' )\n\nvar jobs = kue.createQueue()\n\n\nfunction generateJobs() {\n for ( var i = 0; i < 12; i++ ) {\n"
},
{
"path": "examples/stale.js",
"chars": 1590,
"preview": "var kue = require( '../' )\n , express = require( 'express' );\n\n// create our job queue\n\nvar jobs = kue.createQueue("
},
{
"path": "examples/video.js",
"chars": 1348,
"preview": "var kue = require( '../' )\n , express = require( 'express' );\n\n// create our job queue\n\nvar jobs = kue.createQueue("
},
{
"path": "index.js",
"chars": 38,
"preview": "module.exports = require('./lib/kue');"
},
{
"path": "lib/http/index.js",
"chars": 2232,
"preview": "/*!\n * q - http\n * Copyright (c) 2011 LearnBoost <tj@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Module dependencies.\n *"
},
{
"path": "lib/http/middleware/provides.js",
"chars": 259,
"preview": "/**\n * Specify that the route provides `type`.\n *\n * @param {String} type\n * @return {Function}\n * @api private\n */\n\nmod"
},
{
"path": "lib/http/public/javascripts/caustic.js",
"chars": 7932,
"preview": "/*!\n * EventEmitter\n * Copyright (c) 2011 TJ Holowaychuk <tj@vision-media.ca>\n * MIT Licensed\n */\n\n/**\n * EventEmitter.\n"
},
{
"path": "lib/http/public/javascripts/job.js",
"chars": 6453,
"preview": "/*!\n * kue - Job\n * Copyright (c) 2011 LearnBoost <tj@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Initialize a new `Job`"
},
{
"path": "lib/http/public/javascripts/jquery.ext.js",
"chars": 644,
"preview": "// proxy to allow formatting\n// and because $ is ugly\n\nvar o = function (val) {\n var args = arguments\n , optio"
},
{
"path": "lib/http/public/javascripts/loading.js",
"chars": 2002,
"preview": "/*!\n * kue - LoadingIndicator\n * Copyright (c) 2011 LearnBoost <dev@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Initiali"
},
{
"path": "lib/http/public/javascripts/main.js",
"chars": 6355,
"preview": "/*!\n * kue - http - main\n * Copyright (c) 2011 LearnBoost <tj@learnboost.com>\n * MIT Licensed\n */\n\n// TODO: clean up\n// "
},
{
"path": "lib/http/public/javascripts/progress.js",
"chars": 2202,
"preview": "/*!\n * kue - Progress\n * Copyright (c) 2011 LearnBoost <dev@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Initialize a new"
},
{
"path": "lib/http/public/javascripts/search.js",
"chars": 601,
"preview": "o(function () {\n var search = o('#search');\n search.keyup(function () {\n var val = search.val().trim()\n "
},
{
"path": "lib/http/public/javascripts/utils.js",
"chars": 1520,
"preview": "/*!\n * kue - utils\n * Copyright (c) 2010 LearnBoost <tj@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Format `ms` in words"
},
{
"path": "lib/http/public/stylesheets/actions.styl",
"chars": 356,
"preview": "\n#actions\n fixed: top -2px right -2px\n z-index: 20\n\n#sort\n#filter\n#search\n float: left\n margin: 0\n padding: 5px 10p"
},
{
"path": "lib/http/public/stylesheets/config.styl",
"chars": 390,
"preview": "// general colors\n\ndark = #3b3b3b\nlight = #666\nlighter = #777\n\nbg = #fff\n\n// status colors\n\ninactive-color = #00CCCC\ncom"
},
{
"path": "lib/http/public/stylesheets/context-menu.styl",
"chars": 589,
"preview": "@import 'mixins'\n\nhighlight-color = #00B3E9\n\n.context-menu\n display: none\n reset-list()\n decorated-box()\n li\n &:l"
},
{
"path": "lib/http/public/stylesheets/error.styl",
"chars": 243,
"preview": "\n#error\n fixed: top -50px right 15px\n padding: 20px\n transition: top 500ms, opacity 500ms\n opacity: 0\n background: "
},
{
"path": "lib/http/public/stylesheets/job.styl",
"chars": 2977,
"preview": "@import 'mixins'\n\n#job-template\n display: none\n\nbar(color)\n background: linear-gradient(top, color + 20%, color)\n bor"
},
{
"path": "lib/http/public/stylesheets/main.css",
"chars": 9186,
"preview": "body {\n padding: 50px 120px;\n}\n#menu {\n margin: 0;\n padding: 0;\n position: fixed;\n top: 0;\n left: 0;\n height: 100"
},
{
"path": "lib/http/public/stylesheets/main.styl",
"chars": 771,
"preview": "font-smoothing()\n -webkit-font-smoothing: arguments\n\n@import 'nib'\n@import 'config'\n@import 'scrollbar'\n@import 'menu'\n"
},
{
"path": "lib/http/public/stylesheets/menu.styl",
"chars": 1147,
"preview": "@import 'mixins'\n\n#menu\n reset-list()\n fixed: top left\n height: 100%\n width: 80px\n background: menu-bg\n border-rig"
},
{
"path": "lib/http/public/stylesheets/mixins.styl",
"chars": 298,
"preview": "reset-list()\n margin: 0\n padding: 0\n li\n margin: 0\n list-style: none\n\ndecorated-box()\n border: 1px solid #eee\n"
},
{
"path": "lib/http/public/stylesheets/scrollbar.styl",
"chars": 863,
"preview": "width = scroll-width\npad-x = 60px\npad-y = 40px\n\nbody\n padding: 50px 120px\n\n/*\nhtml\n overflow: auto\n\nbody\n position: a"
},
{
"path": "lib/http/routes/index.js",
"chars": 503,
"preview": "/*!\n * kue - http - routes\n * Copyright (c) 2011 LearnBoost <tj@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Module depen"
},
{
"path": "lib/http/routes/json.js",
"chars": 7390,
"preview": "/*!\n * kue - http - routes - json\n * Copyright (c) 2011 LearnBoost <tj@learnboost.com>\n * MIT Licensed\n */\n\n/**\n * Modul"
},
{
"path": "lib/http/views/_filter.pug",
"chars": 103,
"preview": "select#filter\n option(value='') filter by\n each type in types\n option(value=type)= type\n"
},
{
"path": "lib/http/views/_job.pug",
"chars": 1255,
"preview": ".job\n .block.contents\n h2.id\n a.remove(title='Delete Job') x\n a.restart(title='Restart Job') &#x"
},
{
"path": "lib/http/views/_menu.pug",
"chars": 443,
"preview": "ul#menu\n li.inactive\n a(href='./inactive')\n .count 0\n | Queued\n li.active\n a.a"
},
{
"path": "lib/http/views/_row.pug",
"chars": 28,
"preview": "tr\n td.title\n td.value"
},
{
"path": "lib/http/views/_search.pug",
"chars": 47,
"preview": "input#search(type='text', placeholder='Search')"
},
{
"path": "lib/http/views/_sort.pug",
"chars": 98,
"preview": "select#sort\n option(value='asc') sort\n option(value='asc') asc\n option(value='desc') desc"
},
{
"path": "lib/http/views/job/list.pug",
"chars": 175,
"preview": "extends ../layout\n\nblock body\n h1 #{state}\n\n script.\n o(function(){\n init('#{state}');\n });"
},
{
"path": "lib/http/views/layout.pug",
"chars": 852,
"preview": "html\n head\n title= title\n link(rel='stylesheet', href='./stylesheets/main.css')\n script(src='./j"
},
{
"path": "lib/kue.js",
"chars": 16346,
"preview": "/*!\n * kue\n * Copyright (c) 2013 Automattic <behradz@gmail.com>\n * Copyright (c) 2011 LearnBoost <tj@learnboost.com>\n * "
},
{
"path": "lib/queue/events.js",
"chars": 2822,
"preview": "/*!\n * kue - events\n * Copyright (c) 2013 Automattic <behradz@gmail.com>\n * Copyright (c) 2011 LearnBoost <tj@learnboost"
},
{
"path": "lib/queue/job.js",
"chars": 22488,
"preview": "/*!\n * kue - Job\n * Copyright (c) 2013 Automattic <behradz@gmail.com>\n * Copyright (c) 2011 LearnBoost <tj@learnboost.co"
},
{
"path": "lib/queue/test_mode.js",
"chars": 1177,
"preview": "var Job = require('./job'),\n _ = require('lodash');\n\nvar originalJobSave = Job.prototype.save,\n originalJobUpd"
},
{
"path": "lib/queue/worker.js",
"chars": 10061,
"preview": "/*!\n * kue - Worker\n * Copyright (c) 2013 Automattic <behradz@gmail.com>\n * Copyright (c) 2011 LearnBoost <tj@learnboost"
},
{
"path": "lib/redis.js",
"chars": 4179,
"preview": "/*!\n * kue - RedisClient factory\n * Copyright (c) 2013 Automattic <behradz@gmail.com>\n * Copyright (c) 2011 LearnBoost <"
},
{
"path": "package.json",
"chars": 1208,
"preview": "{\n \"name\": \"kue\",\n \"version\": \"0.11.5\",\n \"description\": \"Feature rich priority job queue backed by redis\",\n \"homepag"
},
{
"path": "test/jsonapi.js",
"chars": 7063,
"preview": "var request = require( 'supertest' ),\r\n kue = require( '../index' ),\r\n async = require( 'async' ),\r\n chai"
},
{
"path": "test/mocha.opts",
"chars": 90,
"preview": "--compilers coffee:coffee-script\n--require should\n--reporter spec\n--ui bdd\n--timeout 10000"
},
{
"path": "test/prefix.coffee",
"chars": 2980,
"preview": "kue = require '../'\n\ndescribe 'Kue - Prefix', ->\n\n makeJobs = (queueName) ->\n opts =\n prefix: queueName\n "
},
{
"path": "test/shutdown.coffee",
"chars": 4933,
"preview": "should = require 'should'\n\nkue = require '../'\n\n\ndescribe 'Kue', ->\n\n before (done) ->\n jobs = kue.createQueue()\n "
},
{
"path": "test/tdd/kue.spec.js",
"chars": 22157,
"preview": "var sinon = require('sinon');\nvar kue = require('../../lib/kue');\nvar redis = require('../../lib/redis');\nvar events = r"
},
{
"path": "test/tdd/redis.spec.js",
"chars": 6012,
"preview": "var sinon = require('sinon');\nvar r = require('redis');\nvar redis = require('../../lib/redis');\n\ndescribe('redis', funct"
},
{
"path": "test/test.coffee",
"chars": 16020,
"preview": "_ = require 'lodash'\nasync = require 'async'\nshould = require 'should'\nkue = require '../'\nutil = require 'util'\n\ndescri"
},
{
"path": "test/test.js",
"chars": 6792,
"preview": "var kue = require( '../' );\n\ndescribe('CONNECTION', function(){\n\tvar jobs = null;\n\n\tafterEach( function ( done ) {\n\t\tjob"
},
{
"path": "test/test_mode.js",
"chars": 2545,
"preview": "var kue = require('../'),\n _ = require('lodash'),\n queue = kue.createQueue();\n\ndescribe('Test Mode', function() {\n"
}
]
About this extraction
This page contains the full source code of the Automattic/kue GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 61 files (234.6 KB), approximately 65.2k tokens, and a symbol index with 44 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.