Horizontal Scaling

Deploy multiple workers sharing the same database.

Deploy multiple worker processes sharing the same database for horizontal scaling. Workers coordinate via database-level locking (FOR UPDATE SKIP LOCKED) — no external coordination required.

your pub/subPostgres LISTEN · Redis NATS · in-process
1 · wake
worker A
2 · claimwins
1 · wake
worker B
2 · claimskipped
1 · wake
worker C
2 · claimskipped
your databaseFOR UPDATE SKIP LOCKED

Identical Workers

The simplest approach: deploy the same worker configuration on multiple machines or processes.

TypeScript

const workerRunner = createWorkerRunner({
  client,
  concurrency: 10,
  processors: createProcessors({ client, jobTypes, processors: { ... } }),
});

Each worker runner registers under a new id from the state adapter's generateWorkerId, a random UUID by default, so processes running the same code never share an id. A restarted process registers under a new id, and once the lease of its previous registration expires, an idle worker runner reclaims the jobs that registration left behind. To keep a stable id per process (a pod name, for example), pass workerId; a process restarting under it waits until the previous registration's lease expires, then reclaims it and starts. Worker runners compete for available jobs — when one acquires a job, others skip it. The notify adapter (Redis, PostgreSQL LISTEN/NOTIFY, etc.) ensures worker runners wake up immediately when new jobs are queued.

Specialized Workers

A worker only processes the job types in its processor registry. This lets you run different worker topologies optimized for different workloads — a worker that doesn't define a processor for a job type simply ignores it. The same mechanism powers prioritization: reserve capacity for an urgent workload by giving it a worker of its own.

For CPU-heavy work, spawn each worker in its own thread so they get true parallelism. Each thread creates its own client, state adapter, and worker — they share nothing except the database:

image-worker-thread.ts

// Each thread runs independently with its own database connection
const stateAdapter = createPgStateAdapter({ stateProvider });
const client = createClient({
  stateAdapter,
  notifyAdapter,
  jobTypes: imageJobTypes,
});

const workerRunner = createWorkerRunner({
  client,
  concurrency: 1,
  processors: createProcessors({
    client,
    jobTypes,
    processors: {
      "images.resize": { attemptHandler: resizeHandler },
      "images.transcode": { attemptHandler: transcodeHandler },
    },
  }),
});
await workerRunner.start();

main.ts

import { Worker } from "node:worker_threads";

// 10 threads for CPU-heavy image processing
for (let i = 0; i < 10; i++) {
  new Worker("./image-worker-thread.ts");
}

// Lightweight async I/O — single worker in main thread, high concurrency
const notificationWorkerRunner = createWorkerRunner({
  client,
  concurrency: 100,
  processors: createProcessors({
    client,
    jobTypes,
    processors: {
      "notifications.send-email": { attemptHandler: emailHandler },
      "notifications.send-sms": { attemptHandler: smsHandler },
    },
  }),
});

This works because chains are stored in the database, not in worker memory. A chain that starts with images.resize (picked up by an image worker thread) can continueWith to notifications.send-email (picked up by the notification worker in the main thread) — the handoff happens through the database.

You can also combine slices when a single worker should handle multiple domains:

TypeScript

const workerRunner = createWorkerRunner({
  client,
  processors: [orderProcessors, notificationProcessors],
});

See Feature Slices for organizing job types and processors into independent modules.

Splitting a Registry

When one createProcessors registry (appProcessors below) holds the processors for every job type, split it across workers with pickProcessors and omitProcessors. pickProcessors keeps only the listed job types; omitProcessors keeps everything else, so the two together cover every job type. Each processor keeps its handler, its slice's middleware chain, and its backoffConfig:

TypeScript

import { omitProcessors, pickProcessors } from "queuert";

const imageJobTypeNames = ["images.resize", "images.transcode"] as const;

const imageWorkerRunner = createWorkerRunner({
  client,
  concurrency: 1,
  processors: pickProcessors(appProcessors, imageJobTypeNames),
});

const defaultWorkerRunner = createWorkerRunner({
  client,
  concurrency: 100,
  processors: omitProcessors(appProcessors, imageJobTypeNames),
});

Job type names are type-checked against the registry, and both functions throw on a name the registry has no processor for. Neither accepts a selection that leaves no processors. To pick from several slices, pick from each one and pass the results as an array: processors: [pickProcessors(orderProcessors, [...]), pickProcessors(billingProcessors, [...])].

See Also

esc

Search guides, integrations and every exported symbol.

↑ ↓ to navigate↵ to open