Horizontal Scaling
Deploy multiple workers sharing the same database.
Deploy multiple worker processes sharing the same database for horizontal scaling. Workers coordinate via database-level locking (FOR UPDATE SKIP LOCKED) — no external coordination required.
Identical Workers
The simplest approach: deploy the same worker configuration on multiple machines or processes.
TypeScript
const workerRunner = createWorkerRunner({
client,
concurrency: 10,
processors: createProcessors({ client, jobTypes, processors: { ... } }),
});Each worker runner registers under a new id from the state adapter's generateWorkerId, a random UUID by default, so processes running the same code never share an id. A restarted process registers under a new id, and once the lease of its previous registration expires, an idle worker runner reclaims the jobs that registration left behind. To keep a stable id per process (a pod name, for example), pass workerId; a process restarting under it waits until the previous registration's lease expires, then reclaims it and starts. Worker runners compete for available jobs — when one acquires a job, others skip it. The notify adapter (Redis, PostgreSQL LISTEN/NOTIFY, etc.) ensures worker runners wake up immediately when new jobs are queued.
Specialized Workers
A worker only processes the job types in its processor registry. This lets you run different worker topologies optimized for different workloads — a worker that doesn't define a processor for a job type simply ignores it. The same mechanism powers prioritization: reserve capacity for an urgent workload by giving it a worker of its own.
For CPU-heavy work, spawn each worker in its own thread so they get true parallelism. Each thread creates its own client, state adapter, and worker — they share nothing except the database:
image-worker-thread.ts
// Each thread runs independently with its own database connection
const stateAdapter = createPgStateAdapter({ stateProvider });
const client = createClient({
stateAdapter,
notifyAdapter,
jobTypes: imageJobTypes,
});
const workerRunner = createWorkerRunner({
client,
concurrency: 1,
processors: createProcessors({
client,
jobTypes,
processors: {
"images.resize": { attemptHandler: resizeHandler },
"images.transcode": { attemptHandler: transcodeHandler },
},
}),
});
await workerRunner.start();main.ts
import { Worker } from "node:worker_threads";
// 10 threads for CPU-heavy image processing
for (let i = 0; i < 10; i++) {
new Worker("./image-worker-thread.ts");
}
// Lightweight async I/O — single worker in main thread, high concurrency
const notificationWorkerRunner = createWorkerRunner({
client,
concurrency: 100,
processors: createProcessors({
client,
jobTypes,
processors: {
"notifications.send-email": { attemptHandler: emailHandler },
"notifications.send-sms": { attemptHandler: smsHandler },
},
}),
});This works because chains are stored in the database, not in worker memory. A chain that starts with images.resize (picked up by an image worker thread) can continueWith to notifications.send-email (picked up by the notification worker in the main thread) — the handoff happens through the database.
You can also combine slices when a single worker should handle multiple domains:
TypeScript
const workerRunner = createWorkerRunner({
client,
processors: [orderProcessors, notificationProcessors],
});See Feature Slices for organizing job types and processors into independent modules.
Splitting a Registry
When one createProcessors registry (appProcessors below) holds the processors for every job type, split it across workers with pickProcessors and omitProcessors. pickProcessors keeps only the listed job types; omitProcessors keeps everything else, so the two together cover every job type. Each processor keeps its handler, its slice's middleware chain, and its backoffConfig:
TypeScript
import { omitProcessors, pickProcessors } from "queuert";
const imageJobTypeNames = ["images.resize", "images.transcode"] as const;
const imageWorkerRunner = createWorkerRunner({
client,
concurrency: 1,
processors: pickProcessors(appProcessors, imageJobTypeNames),
});
const defaultWorkerRunner = createWorkerRunner({
client,
concurrency: 100,
processors: omitProcessors(appProcessors, imageJobTypeNames),
});Job type names are type-checked against the registry, and both functions throw on a name the registry has no processor for. Neither accepts a selection that leaves no processors. To pick from several slices, pick from each one and pass the results as an array: processors: [pickProcessors(orderProcessors, [...]), pickProcessors(billingProcessors, [...])].
See Also
- examples/postgresql/multi-worker — multiple workers sharing a PostgreSQL database
- examples/sqlite/multi-worker — worker processes on one host sharing a SQLite database file, notified over IPC
- examples/mysql/multi-worker — worker processes sharing a MySQL database, polling for new jobs
- examples/redis/multi-worker — worker processes sharing Redis for state, notified over Redis pub/sub
- Prioritization — reserving worker capacity for urgent workloads
- Worker Runner — worker lifecycle and configuration
- State Adapters — database adapter setup