Queue Options

Description

When creating the Queue object you can supply options to customize the way the queue works. Each one of the Queue object options is discussed below.

Option	Type	Default
host	String	localhost
port	Integer	28015
db	String	rjqJobQueue
name	String	rjqJobList
masterInterval	Integer	310 seconds
changeFeed	Boolean	true
concurrency	Integer	1
removeFinishedJobs	Bool / Int	180 days

Options Example

This example is to show all the options being customized:

const Queue = require('rethinkdb-job-queue')
const options = {
  host: 'proddb.domain.com',
  port: 10000,
  db: 'JobQueue',
  name: 'VideoProcess',
  masterInterval: 1000,
  changeFeed: true,
  concurrency: 50,
  removeFinishedJobs: false
}
const q = new Queue(options)

Queue host Option

Default: localhost

Valid: IP Address or DNS name of the RethinkDB server.

The host option can be ignored if you are running RethinkDB on the same machine as the nodejs service. If not, point the queue to the server you have RethinkDB running on.

host Example

This example points the Queue object at an IP address:

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ host: '192.168.1.10' })

This example points the Queue object at a DNS name:

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ host: 'proddb.domain.com' })

Queue port Option

Default: 28015

Valid: Integer up to 65535.

The port option allows you to specify the TCP port used by the RethinkDB database.

port Example

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ port: 8000 })

Queue db Option

Default: rjqJobQueue

Valid: Any string value. Don't use spaces.

Set the db option to be the name of your job database in RethinkDB. The db option will either create a new database if it does not exist, or use an existing database. If you would like your existing database to hold your job Tables, simply set the db option to the existing name you are using in RethinkDB.

Important: There is no API call in rethinkdb-job-queue that can drop the database. The Queue.drop method only drops the queue Table in the database, not the database itself.

db Example

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ db: 'JobQueue' })

Queue name Option

Default: rjqJobList

Valid: Any string value. Don't use spaces.

The name option defines the name of the Queue, the Queue object, and the Table created in RethinkDB. You can create multiple Queue objects with different names causing multiple Tables to be created in the RethinkDB database.

Important: It is highly recommended not to set the name option to the same name of an existing table in your database. The Queue.drop API call will drop the Table from the database. Always use a name related to the jobs you are processing.

name Example

This example creates two queues using the default options except for the name. One of the queues is called 'registration' and the other 'video'. This example will create two Tables in the RethinkDB database called 'registration' and 'video'.

const Queue = require('rethinkdb-job-queue')
const regoQ = new Queue({ name: 'registration' })
const videoProcessQ = new Queue({ name: 'video' })

Queue masterInterval Option

Default: 310 seconds

Valid: Enable with a positive Integer. Disable with false or 0.

The masterInterval option determines how often the Queue Master review process is carried out against the queue. Please refer to the Queue Master document for more detail.

The default value of 310 seconds was chosen because it is 10 seconds past the job timeout default value of 300 seconds. This is to improve detection of failed nodejs processes on queue startup.

Warning: Setting the masterInterval value too low will cause excessive database load. Every time this period lapses queries are run against the database. Keep this value as high as possible.

masterInterval Example

This example shows setting the masterInterval to one minute to ensure the queue is kept clean and failed jobs get processed.

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ masterInterval: 60 })

Queue changeFeed Option

Default: true

Valid: true or false.

If enabled the changeFeed option will cause the Queue object to raise events for the entire queue rather than just the local Queue object.

Behind the scenes a RethinkDB change feed is created for the whole of the queue Table in the database. Once the changes are received by the Queue object, they are checked to ensure they are not local changes. If the changes are detected as local changes (a change made by self) they are ignored because the events will be raised by the Queue object. If the changes are not local then the relevant event is raised.

See the Queue Events document for more detail.

changeFeed Example

This example simply disables the changeFeed option.

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ changeFeed: false })

Queue concurrency Option

Default: 1

Valid: Any positive Integer.

The Queue object can process many jobs at once. Keep in mind the blocking nature of long running tasks. If your jobs are external and will not block the node process, feel free to push the concurrency value through the roof.

concurrency Example

This example sets the concurrency option to 100. If more than 100 jobs are available on the queue then 100 will be processed at once.

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ concurrency: 100 })

Queue removeFinishedJobs Option

Default: 180 days

Valid: true, false or a positive Integer.

The removeFinishedJobs option is used to control how the queue removes jobs which are considered to be finished from the database.

A job is considered finished when it has a status of either completed, cancelled, or terminated. See the Job Status document for more detail.

Finished jobs will be permanently removed from the database based on the values you assign to removeFinishedJobs.

This is one of the more flexible options available on Queue objects. You have three values you can assign to the removeFinishedJobs option.

Value 1: true

Setting the removeFinishedJobs option to true will cause jobs to be removed from the queue immediately when they are finished. They will no longer be in the database.

Value 2: false or 0

Setting the removeFinishedJobs option to false or 0 will prevent the queue from ever removing jobs from the database.

Value 3: Integer as number of days

By setting the removeFinishedJobs option to a number such as 180, the Queue Master review process will permanently remove jobs that were finished 180 days ago.

removeFinishedJobs Example

This example shows setting removeFinishedJobs option to false causing the queue to never remove jobs from the database:

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ removeFinishedJobs: false })

This example shows setting the removeFinishedJobs option to 30 days. The Queue Master review process will remove finished jobs when they are a month old.

const Queue = require('rethinkdb-job-queue')
const q = new Queue({ removeFinishedJobs: 30 })