Bee-Queue 1.0 makes stalled-job recovery explicit

A worker can finish an external operation and disappear before confirming that its job completed. A queue cannot infer the outcome from silence alone. Bee-Queue 1.0 chooses to retry jobs whose workers stop reporting, providing an at-least-once processing model. Understanding that choice is more useful than assuming a Redis-backed queue removes uncertainty about execution.

Eli Skeggs's 18 October announcement describes Bee-Queue as a Redis-backed Node.js library intended for short, real-time jobs and distributed worker pools. Mixmax reports using it for tens of millions of jobs each day. Those are the publisher's production figures, not a benchmark or a claim about my own use.

I like Redis as a foundation for explicit data operations. A job library is interesting when it explains the protocol it builds from those operations. Bee-Queue's versioned documentation makes both the storage layout and the recovery rule visible, including work the application must arrange itself.

Waiting jobs move into an active list

The 1.0.0 README describes a hash mapping job identifiers to JSON records containing data and options. Separate lists contain waiting and active job identifiers. Sets retain successful and failed identifiers, and a sorted set represents delayed jobs.

Idle workers use BRPOPLPUSH to move a job identifier from the waiting list to the active list. The documentation calls this a non-polling arrangement following Redis's reliable-queue pattern. A job leaves the waiting list while remaining represented in the active list, which gives recovery logic somewhere to find work that has not completed.

The worker connection dedicated to that blocking command is separate from other connections. Pub/Sub event delivery can require another dedicated connection when event reception or delayed-job activation is enabled. Queue instances therefore have connection costs related to their enabled roles, rather than simply one connection per application process.

Producers and workers share a named queue and Redis instance. The API creates a job with arbitrary data and saves it before processing. A handler can use a completion callback or return a promise. The README's callback form can be reduced to this complete example:

const Queue = require('bee-queue');
const queue = new Queue('addition');

queue.process(function (job, done) {
  done(null, job.data.x + job.data.y);
});

queue.createJob({x: 2, y: 3}).save();

The handler can run in another process using the same queue name. The queue abstraction supplies delivery and result handling, while the application still supplies the operation performed for the job. Increasing the worker pool adds processing capacity within the constraints of that operation and the shared Redis service.

Silence puts a job back into waiting

Workers periodically report that they are still processing their active jobs. The stalling set is a snapshot of the active list at the beginning of a stall interval. Reporting workers remove their job identifiers from that set. Identifiers remaining when the next check runs describe work whose worker did not report during the interval.

checkStalledJobs re-enqueues those identifiers and takes a new snapshot of the active list. This is a concrete failure detector based on missing reports. It cannot prove that the earlier worker performed no external side effect before becoming silent.

The README consequently says Bee-Queue attempts at-least-once delivery. A worker that crashes or disconnects before confirming completion can cause another worker to process the job. Repeated processing is compatible with that guarantee, even if the normal path usually executes a job once.

For an operation with external effects, this means that the job identifier and the target system's behavior matter. A retry can encounter an effect already performed by the previous attempt. The queue's active-list and heartbeat protocol describes delivery state; it does not supply an atomic transaction across every external service a handler might call.

I regard the explicit delivery model as a useful part of the library's design. It lets an application reason about repeated execution without confusing atomic Redis operations with exactly-once effects outside Redis. Those are different claims, and the documented guarantee is the former delivery model.

Applications must arrange recurring stall checks

The library deliberately leaves repeated stall checks to the user. Its README explains that automatically checking from every queue instance would add redundant work without giving the same timing guarantee as a smaller, deliberately scheduled set of checkers. Starting workers is therefore not the whole recovery arrangement.

The documented default stallInterval is 5,000 milliseconds. Increasing it reduces reporting overhead and the probability of false positives, but delays retrying a genuinely stalled job. Choosing the interval changes both detection sensitivity and recovery latency because silence is the evidence the mechanism uses.

Delayed retries have another explicit requirement. At least one queue instance must enable activateDelayedJobs for fixed or exponential delayed-retry strategies. The library exposes that role separately, so configuring a retry policy and providing a process that activates delayed work are related deployment decisions.

The initial feature set also limits scope. Its README identifies job priority and all-worker pause/resume among significant non-features. The announcement says Mixmax chose to build on the existing Bee-Queue project with original author Lewis Ellis instead of creating a new queue from scratch. Its stated aim is a small library focused on performance and stability.

The README lists Redis 2.8 or later as the minimum but recommends 3.2 or later because of observed delays on older versions. It also documents callbacks and promises for saving jobs. The job identifier becomes available after the save completes, which separates constructing a local job object from confirming that it has been enqueued.

Events are separate from stored progress

Bee-Queue sends progress and result events through Redis Pub/Sub. Its reportProgress documentation explicitly says progress is not persisted to Redis, although receiving queue instances can update corresponding in-memory job objects. A progress event therefore has a different retention model from a stored job record.

That distinction matters for a producer waiting for results within an HTTP request, one of the use cases described by the README. The event mechanism supplies live feedback, while the job hash and active list supply persistent queue state. Bee-Queue 1.0 makes those roles visible and assigns stalled-job checking to the application, with retries preserving the possibility of repeated processing after an ambiguous worker failure.