Redis State Adapter
Keep chains, jobs and workers in Redis or Redis Cluster with ioredis or node-redis, using optimistic transactions.
The Redis state adapter in @queuert/redis keeps Queuert's chains, jobs and workers in Redis or Redis Cluster. Redis holds no other data of your application, and it has no interactive transactions, so the adapter's transactions are optimistic: they never wait for each other, and a transaction that conflicts runs again.
Experimental
@queuert/redis is experimental: its API may change in a minor release.Clients
Install
Terminal
npm install @queuert/redisRequires Redis 6 or later, standalone or Redis Cluster. migrateToLatest() checks the version.
Setup
Each tab shows the example's RedisStateProvider and the lines that create the adapter and bring its keys to the latest schema with migrateToLatest():
src/provider.ts
import { createHash } from "node:crypto";
import type { RedisStateProvider } from "@queuert/redis";
import type { RedisClientType } from "redis";
export const createNodeRedisStateProvider = ({
client,
}: {
client: RedisClientType;
}): RedisStateProvider => {
const scriptShas = new Map<string, string>();
return {
eval: async (script, keys, args) => {
let sha = scriptShas.get(script);
if (sha === undefined) {
sha = createHash("sha1").update(script).digest("hex");
scriptShas.set(script, sha);
}
try {
return await client.evalSha(sha, { keys, arguments: args });
} catch (error) {
if (!(error instanceof Error) || !error.message.startsWith("NOSCRIPT")) throw error;
return client.eval(script, { keys, arguments: args });
}
},
};
};src/index.ts
await using drivers = new AsyncDisposableStack();
const redis = drivers.adopt(
await createRedisClient({ url: redisContainer.connectionUrl }).connect(),
async (redis) => {
await redis.close();
},
);
await using stateAdapter = createRedisStateAdapter({
stateProvider: createNodeRedisStateProvider({ client: redis }),
keyPrefix: "{queuert}:",
});
await stateAdapter.migrateToLatest();src/provider.ts
import { createHash } from "node:crypto";
import type { RedisStateProvider } from "@queuert/redis";
import type { Redis } from "ioredis";
export const createIoredisStateProvider = ({ client }: { client: Redis }): RedisStateProvider => {
const scriptShas = new Map<string, string>();
return {
eval: async (script, keys, args) => {
let sha = scriptShas.get(script);
if (sha === undefined) {
sha = createHash("sha1").update(script).digest("hex");
scriptShas.set(script, sha);
}
try {
return await client.evalsha(sha, keys.length, ...keys, ...args);
} catch (error) {
if (!(error instanceof Error) || !error.message.startsWith("NOSCRIPT")) throw error;
return client.eval(script, keys.length, ...keys, ...args);
}
},
};
};src/index.ts
await using drivers = new AsyncDisposableStack();
const redis = drivers.adopt(new Redis(redisContainer.connectionUrl), async (redis) => {
await redis.quit();
});
await using stateAdapter = createRedisStateAdapter({
stateProvider: createIoredisStateProvider({ client: redis }),
keyPrefix: "{queuert}:",
});
await stateAdapter.migrateToLatest();Configuration
createRedisStateAdapter
- stateProvider requiredThe
RedisStateProviderfor your client; see State provider. - keyPrefix "{queuert}:"Prefix of every key the adapter writes. It must contain a non-empty hash tag such as
{queuert}, or the factory throws; see Redis Cluster. - The most tries of one transaction. After that many conflicting tries,
withTransactionthrowsRedisTransactionConflictError. - The most blocker chains a single job may declare. Declaring more throws
BlockerLimitExceededError; see Job Blockers. - countCap 10_000The per-status count at which
countChainsByTypeNameandcountJobsByTypeName(and the dashboard's type counts) stop counting and reporthasMore. A higher cap means a longer index scan per count. - generateJobId crypto.randomUUIDCalled for every job created without a caller-supplied
id. - validateJobId accept allRuns on every caller-supplied and generated ID before writing, throwing
InvalidJobIdErroron failure. - generateWorkerId crypto.randomUUIDCalled once for each worker runner created without a
workerId; must return a new id on every call. - validateWorkerId accept allRuns on the configured or generated worker id;
createWorkerRunnerthrowsInvalidWorkerIdErroron failure.
maxBlockersPerJob and countCap must be positive integers; anything else throws when the adapter is created. The resolved values are readable on stateAdapter.config.
State provider
A RedisStateProvider is one method over any Redis client: the adapter reads and writes only through Lua scripts. Each Setup tab is a complete provider:
RedisStateProvider
- eval requiredRuns a Lua script with the given keys and arguments and returns its reply. Every key shares the adapter's hash tag, so on a cluster the script runs on one node. The examples cache each script's SHA-1 and send
EVALSHA, falling back toEVALwhen Redis answersNOSCRIPT. - Releases resources the provider owns. Leave it out when the provider wraps a client you close yourself; when present, it must be idempotent.
Transactions
withTransaction reads from Redis and records the version of every record it reads, buffers every write, and commits with one Lua script that applies the writes only if none of those versions changed. When one did, the adapter runs the callback again with fresh reads. A callback that throws after reading records another transaction has since changed runs again too. After maxTransactionTries conflicting tries it throws RedisTransactionConflictError, with the last callback's error as its cause when that try threw.
- Callbacks may run more than once. Keep side effects out of the callback and register them with
onTransactionCommit, which runs once, after the commit. - Pass the adapter's context. Create chains and complete jobs with the context
withTransactionpasses to its callback. The adapter's transactions carry their own commit and rollback callbacks, so they need nowithTxHooks. - A resent commit applies once. A commit leaves a marker in
{keyPrefix}commit:*and a claim of job attempts leaves the jobs it claimed in{keyPrefix}claim:*, both for 60 seconds. A client that resends a script after a reconnect, as ioredis does by default, gets the first reply back instead of applying it twice. - No other database joins the transaction. A write to your application database cannot commit or roll back together with Queuert's state in Redis.
src/index.ts
const chain = await stateAdapter.withTransaction(async (txCtx) => {
const created = await client.createChain({
...txCtx,
typeName: "send_welcome_email",
input: { email: "alice@example.com" },
});
stateAdapter.onTransactionCommit(txCtx, () => {
console.log(`Queued chain ${created.id}`);
});
return created;
});src/index.ts
const chain = await stateAdapter.withTransaction(async (txCtx) => {
const created = await client.createChain({
...txCtx,
typeName: "send_welcome_email",
input: { email: "alice@example.com" },
});
stateAdapter.onTransactionCommit(txCtx, () => {
console.log(`Queued chain ${created.id}`);
});
return created;
});With an application database
Because no other database joins its transactions, the adapter fits work that starts from a committed fact in your application database. Run a PostgreSQL or SQLite Queuert as a transactional outbox: create its chain in the same transaction as your application write, and let its handler start the chain in the Redis Queuert. Give that chain a deduplication key derived from the outbox job, so a handler that runs again after a failure returns the chain it already started instead of creating a second one.
Locks
A read with lock: "exclusive" of a pending job, such as the one completeChain takes, marks the job for up to 60 seconds; workers skip a marked job until the transaction commits or rolls back. A completeChain handler that runs longer than that can see a worker claim the job. Workers look past marked jobs through at most the first 1,000 due jobs of each type, so a worker claims nothing of a type whose first 1,000 due jobs are all marked.
No transaction ever waits for a lock. Where PostgreSQL would make a second transaction wait for the first, the Redis adapter lets both run and the one that commits second runs again. In the conformance cases that check a wait, each outcome matches some order of the two transactions; those cases are left out of the suite and listed in redisStateAdapterLockWaitCaseNames.
Migrations
migrateToLatest() applies the adapter's schema migrations and records each one in the {keyPrefix}migrations set. Run it on every boot: processes that run it at once take turns through a lease in {keyPrefix}migration-lock, so one migrates and the others wait and then find nothing left to do.
Write fence
The set {keyPrefix}schema-fences holds the schema versions whose code may write. The commit script of every transaction that creates, continues, completes, reschedules, blocks, unblocks or deletes jobs, or starts, stops or extends the lease of a worker, checks that the set holds the adapter's version before it writes anything, and so does the script that starts job attempts. Without it the script writes nothing and the adapter throws SchemaFencedError. Until migrateToLatest() has run, the set is empty and every such write is fenced. A breaking migration removes the versions of older code, so older processes fail those writes instead of corrupting state. Reads still work, and older workers report the error through workerError and keep retrying. Stop them once you have deployed the newer version.
The check runs when the transaction commits, so a transaction that began before a breaking migration and commits after it is fenced too, whatever it wrote before the migration.
Schema
Every key starts with keyPrefix, left out below. Records hold the state, and every other key is an index the adapter derives from them, so the layout reads like the PostgreSQL schema: a job record per job, a worker record per worker, and an index per PostgreSQL index, named after it where one matches. The adapter's job lock, commit and claim markers expire on their own.
records
- job:<id> hashOne job, the Redis form of a
jobrow with itsjob_blockerrows.vis the version of its last write,dthe JSON fields the Lua scripts read and change (ids, type name, status, chain status, attempt fields and timestamps), anduthe JSON fields no script reads: input, output, last attempt error, trace contexts, deduplication key,completedByand blockers. A field that isnullis left out of both. - worker:<id> hashOne worker, the Redis form of a
workerrow: its job types, concurrency, start time, lease and token ind. - deduplication:<[typeName,key]> hashThe head ids of the chains created with one deduplication key: the chains still running and the newest one. It serves what
chain_deduplication_idxserves in PostgreSQL. - sequence stringThe counter every commit and claim increments; its value is the version the write stamps on records and blocker sets.
- set-versions hashThe version of the last write that changed each non-empty set the adapter keeps, such as blocked-by:<chainId>, so a transaction that read a set conflicts when it changed.
- schema-fences setThe schema versions whose code may write; see Write fence.
- migrations setThe names of the applied schema migrations; see Migrations.
- migration-lock stringThe lease of the process running migrations, expiring on its own.
- job-lock:<id> stringMarks a pending job an exclusive read holds, for up to 60 seconds.
- commit:<token>, claim:<token> stringMarkers of an applied commit and of the jobs a claim took, kept 60 seconds so a resent script applies once.
Indexes
Every write of a record updates its index entries in the same script, and so does the claim of job attempts. A job sits in an index while it matches the index's condition, such as status = 'pending' for due:, with the job id as the member and the named timestamp as the score. The hashes count records per type name or blocker chain, and a field goes away when its count reaches zero.
indexes
- chain-jobs:<chainId> zset chain_index_idxThe jobs of a chain scored by chain index: chain listings and the chain's tail.
- jobs:createdAt:all:<typeName> zset job_idxEvery job of a type by creation time: job listings without a status filter.
- jobs:createdAt:<status>:<typeName> zsetThe jobs of a type in one status by creation time: per-status counts and listings.
- jobs:scheduledAt:pending:<typeName> zset job_pending_idxPending jobs by scheduled time: listings of pending jobs.
- due:<typeName> zset job_pending_idxPending jobs by scheduled time, with the creation order in the member to break ties: the claim scans it for due jobs.
- jobs:scheduledAt:blocked:<typeName> zset job_blocked_idxBlocked jobs by scheduled time: listings of blocked jobs.
- jobs:attemptAt:running:<typeName> zset job_running_idxRunning jobs by attempt start: listings of running jobs.
- jobs:completedAt:completed:<typeName> zset job_completed_idxCompleted jobs by completion time: listings of completed jobs.
- running:<workerId> zset job_attempt_by_idxThe jobs a worker is running, by attempt start: a worker's jobs and their count.
- job-type-names hashThe number of jobs of each type: the job type names in use.
- chains:createdAt:all:<typeName> zset chain_idxChain heads by creation time: chain listings without a status filter.
- chains:createdAt:<status>:<typeName> zset chain_running_idxChain heads in one status by creation time: per-status counts and listings.
- chains:completedAt:completed:<typeName> zset chain_completed_idxCompleted chain heads by completion time.
- chain-type-names hashThe number of chains of each type: the chain type names in use.
- blocked-by:<chainId> set job_blocker_chain_idxThe jobs a chain blocks: unblocking them when it completes, and refusing to delete it while they exist.
- blocked-jobs:<chainId> zset job_blocker_chain_idxThe jobs a chain blocks by creation time: listings of a chain's blocked jobs.
- blocker-references hashThe number of jobs each chain blocks: chain listings filtered by whether a chain blocks others.
- workers, workers-of-type:<typeName> zset worker_idxWorkers by start time, all of them and per job type: worker listings.
- worker-leases, worker-leases-of-type:<typeName> zsetWorkers by lease end, all of them and per job type: live and expired counts.
Schema changes
A schema change ships as a numbered migration that migrateToLatest() applies. A change to an index definition ships with a migration that rebuilds the indexes: it drops every index key and computes them again from the job and worker records, in batches that each block Redis briefly. A breaking change also replaces the version in schema-fences, so older processes stop writing while it runs.
Redis configuration
Redis is the only copy of Queuert's state, so configure it as a database, not a cache:
- Persistence. Turn on AOF (
appendonly yes); without it a restart loses every chain, job and worker written since the last snapshot. - No eviction. Set
maxmemory-policy noeviction, so a full Redis rejects writes instead of deleting job records or the expiring lock keys. - Memory. Every job stays in memory until its chain is deleted, which a cleanup job does on a schedule. A job takes about 1.5 KB with its index entries, plus its input, output and blockers, so set
maxmemoryfor the number of jobs you keep, not for the queue length.
Redis Cluster
Every key starts with keyPrefix, whose hash tag sends all of them to one slot, so each script runs on one node. A cluster gives that Queuert availability, not more throughput: one node holds all of its state. Adapters with different hash tags can land on different nodes. Replication to replicas is asynchronous, so a failover can lose the last writes the primary acknowledged.
Testing your provider
@queuert/redis/conformance checks a provider you write. redisStateProviderConformanceCases are checks of the eval contract followed by every state adapter case the Redis adapter meets, including checks of its write fence. createRedisStateProviderConformanceContext takes a createStateProvider factory; each case builds its own adapter over a new provider under a key prefix of its own, migrates it, and closes it when it ends. The suite drives no test runner: your test file owns the client in its own hooks and registers one test per case:
src/provider.spec.ts
describe("redis/state-redis provider conformance", () => {
let redis: AcquiredRedis;
let client: RedisClientType;
let context: RedisStateProviderConformanceContext;
beforeAll(async () => {
redis = await acquireRedis("redis:8");
client = createClient({ url: redis.connectionUrl });
await client.connect();
context = createRedisStateProviderConformanceContext({
createStateProvider: async () => createNodeRedisStateProvider({ client }),
});
}, 60_000);
afterAll(async () => {
await client.close();
await redis[Symbol.asyncDispose]();
});
for (const conformanceCase of redisStateProviderConformanceCases) {
test(conformanceCase.title, async () => {
await conformanceCase.run(context);
});
}
});src/provider.spec.ts
describe("redis/state-ioredis provider conformance", () => {
let redis: AcquiredRedis;
let client: Redis;
let context: RedisStateProviderConformanceContext;
beforeAll(async () => {
redis = await acquireRedis("redis:8");
client = new Redis(redis.connectionUrl);
context = createRedisStateProviderConformanceContext({
createStateProvider: async () => createIoredisStateProvider({ client }),
});
}, 60_000);
afterAll(async () => {
await client.quit();
await redis[Symbol.asyncDispose]();
});
for (const conformanceCase of redisStateProviderConformanceCases) {
test(conformanceCase.title, async () => {
await conformanceCase.run(context);
});
}
});The full API is in the conformance reference.
Examples
@queuert/redis
- GitHub Experimental Redis state storage via @queuert/redis with the node-redis client — a chain created in an optimistic transaction, with its side effect in onTransactionCommit because the callback may run more than once.
$ pnpm --filter example-redis-state-redis startsrc/provider.tssrc/index.ts - GitHub Experimental Redis state storage via @queuert/redis with the ioredis client — a chain created in an optimistic transaction, with its side effect in onTransactionCommit because the callback may run more than once.
$ pnpm --filter example-redis-state-ioredis startsrc/provider.tssrc/index.ts - GitHub Three worker processes (forked from the main process) share one Redis instance for state and compete for jobs, waking each other through the Redis notify adapter — the main process creates a chain per order in one optimistic transaction. Horizontal Scaling guide →
$ pnpm --filter example-redis-multi-worker startsrc/index.tssrc/worker.ts
The API reference for every export is in @queuert/redis.