Redis State Adapter

Keep chains, jobs and workers in Redis or Redis Cluster with ioredis or node-redis, using optimistic transactions.

The Redis state adapter in @queuert/redis keeps Queuert's chains, jobs and workers in Redis or Redis Cluster. Redis holds no other data of your application, and it has no interactive transactions, so the adapter's transactions are optimistic: they never wait for each other, and a transaction that conflicts runs again.

Experimental

The Redis state adapter in @queuert/redis is experimental: its API may change in a minor release.

Clients

Install

Terminal

npm install @queuert/redis

Requires Redis 6 or later, standalone or Redis Cluster. migrateToLatest() checks the version.

Setup

Each tab shows the example's RedisStateProvider and the lines that create the adapter and bring its keys to the latest schema with migrateToLatest():

src/provider.ts

import { createHash } from "node:crypto";

import type { RedisStateProvider } from "@queuert/redis";
import type { RedisClientType } from "redis";

export const createNodeRedisStateProvider = ({
  client,
}: {
  client: RedisClientType;
}): RedisStateProvider => {
  const scriptShas = new Map<string, string>();
  return {
    eval: async (script, keys, args) => {
      let sha = scriptShas.get(script);
      if (sha === undefined) {
        sha = createHash("sha1").update(script).digest("hex");
        scriptShas.set(script, sha);
      }
      try {
        return await client.evalSha(sha, { keys, arguments: args });
      } catch (error) {
        if (!(error instanceof Error) || !error.message.startsWith("NOSCRIPT")) throw error;
        return client.eval(script, { keys, arguments: args });
      }
    },
  };
};

src/index.ts

await using drivers = new AsyncDisposableStack();
const redis = drivers.adopt(
  await createRedisClient({ url: redisContainer.connectionUrl }).connect(),
  async (redis) => {
    await redis.close();
  },
);
await using stateAdapter = createRedisStateAdapter({
  stateProvider: createNodeRedisStateProvider({ client: redis }),
  keyPrefix: "{queuert}:",
});
await stateAdapter.migrateToLatest();

Configuration

createRedisStateAdapter

  • stateProvider required
    The RedisStateProvider for your client; see State provider.
  • keyPrefix "{queuert}:"
    Prefix of every key the adapter writes. It must contain a non-empty hash tag such as {queuert}, or the factory throws; see Redis Cluster.
  • The most tries of one transaction. After that many conflicting tries, withTransaction throws RedisTransactionConflictError.
  • The most blocker chains a single job may declare. Declaring more throws BlockerLimitExceededError; see Job Blockers.
  • countCap 10_000
    The per-status count at which countChainsByTypeName and countJobsByTypeName (and the dashboard's type counts) stop counting and report hasMore. A higher cap means a longer index scan per count.
  • generateJobId crypto.randomUUID
    Called for every job created without a caller-supplied id.
  • validateJobId accept all
    Runs on every caller-supplied and generated ID before writing, throwing InvalidJobIdError on failure.
  • generateWorkerId crypto.randomUUID
    Called once for each worker runner created without a workerId; must return a new id on every call.
  • validateWorkerId accept all
    Runs on the configured or generated worker id; createWorkerRunner throws InvalidWorkerIdError on failure.

maxBlockersPerJob and countCap must be positive integers; anything else throws when the adapter is created. The resolved values are readable on stateAdapter.config.

State provider

A RedisStateProvider is one method over any Redis client: the adapter reads and writes only through Lua scripts. Each Setup tab is a complete provider:

RedisStateProvider

  • eval required
    Runs a Lua script with the given keys and arguments and returns its reply. Every key shares the adapter's hash tag, so on a cluster the script runs on one node. The examples cache each script's SHA-1 and send EVALSHA, falling back to EVAL when Redis answers NOSCRIPT.
  • Releases resources the provider owns. Leave it out when the provider wraps a client you close yourself; when present, it must be idempotent.

Transactions

withTransaction reads from Redis and records the version of every record it reads, buffers every write, and commits with one Lua script that applies the writes only if none of those versions changed. When one did, the adapter runs the callback again with fresh reads. A callback that throws after reading records another transaction has since changed runs again too. After maxTransactionTries conflicting tries it throws RedisTransactionConflictError, with the last callback's error as its cause when that try threw.

  • Callbacks may run more than once. Keep side effects out of the callback and register them with onTransactionCommit, which runs once, after the commit.
  • Pass the adapter's context. Create chains and complete jobs with the context withTransaction passes to its callback. The adapter's transactions carry their own commit and rollback callbacks, so they need no withTxHooks.
  • A resent commit applies once. A commit leaves a marker in {keyPrefix}commit:* and a claim of job attempts leaves the jobs it claimed in {keyPrefix}claim:*, both for 60 seconds. A client that resends a script after a reconnect, as ioredis does by default, gets the first reply back instead of applying it twice.
  • No other database joins the transaction. A write to your application database cannot commit or roll back together with Queuert's state in Redis.

src/index.ts

const chain = await stateAdapter.withTransaction(async (txCtx) => {
  const created = await client.createChain({
    ...txCtx,
    typeName: "send_welcome_email",
    input: { email: "alice@example.com" },
  });
  stateAdapter.onTransactionCommit(txCtx, () => {
    console.log(`Queued chain ${created.id}`);
  });
  return created;
});

With an application database

Because no other database joins its transactions, the adapter fits work that starts from a committed fact in your application database. Run a PostgreSQL or SQLite Queuert as a transactional outbox: create its chain in the same transaction as your application write, and let its handler start the chain in the Redis Queuert. Give that chain a deduplication key derived from the outbox job, so a handler that runs again after a failure returns the chain it already started instead of creating a second one.

Locks

A read with lock: "exclusive" of a pending job, such as the one completeChain takes, marks the job for up to 60 seconds; workers skip a marked job until the transaction commits or rolls back. A completeChain handler that runs longer than that can see a worker claim the job. Workers look past marked jobs through at most the first 1,000 due jobs of each type, so a worker claims nothing of a type whose first 1,000 due jobs are all marked.

No transaction ever waits for a lock. Where PostgreSQL would make a second transaction wait for the first, the Redis adapter lets both run and the one that commits second runs again. In the conformance cases that check a wait, each outcome matches some order of the two transactions; those cases are left out of the suite and listed in redisStateAdapterLockWaitCaseNames.

Migrations

migrateToLatest() applies the adapter's schema migrations and records each one in the {keyPrefix}migrations set. Run it on every boot: processes that run it at once take turns through a lease in {keyPrefix}migration-lock, so one migrates and the others wait and then find nothing left to do.

Write fence

The set {keyPrefix}schema-fences holds the schema versions whose code may write. The commit script of every transaction that creates, continues, completes, reschedules, blocks, unblocks or deletes jobs, or starts, stops or extends the lease of a worker, checks that the set holds the adapter's version before it writes anything, and so does the script that starts job attempts. Without it the script writes nothing and the adapter throws SchemaFencedError. Until migrateToLatest() has run, the set is empty and every such write is fenced. A breaking migration removes the versions of older code, so older processes fail those writes instead of corrupting state. Reads still work, and older workers report the error through workerError and keep retrying. Stop them once you have deployed the newer version.

The check runs when the transaction commits, so a transaction that began before a breaking migration and commits after it is fenced too, whatever it wrote before the migration.

Schema

Every key starts with keyPrefix, left out below. Records hold the state, and every other key is an index the adapter derives from them, so the layout reads like the PostgreSQL schema: a job record per job, a worker record per worker, and an index per PostgreSQL index, named after it where one matches. The adapter's job lock, commit and claim markers expire on their own.

records

  • job:<id> hash
    One job, the Redis form of a job row with its job_blocker rows. v is the version of its last write, d the JSON fields the Lua scripts read and change (ids, type name, status, chain status, attempt fields and timestamps), and u the JSON fields no script reads: input, output, last attempt error, trace contexts, deduplication key, completedBy and blockers. A field that is null is left out of both.
  • worker:<id> hash
    One worker, the Redis form of a worker row: its job types, concurrency, start time, lease and token in d.
  • deduplication:<[typeName,key]> hash
    The head ids of the chains created with one deduplication key: the chains still running and the newest one. It serves what chain_deduplication_idx serves in PostgreSQL.
  • sequence string
    The counter every commit and claim increments; its value is the version the write stamps on records and blocker sets.
  • set-versions hash
    The version of the last write that changed each non-empty set the adapter keeps, such as blocked-by:<chainId>, so a transaction that read a set conflicts when it changed.
  • schema-fences set
    The schema versions whose code may write; see Write fence.
  • migrations set
    The names of the applied schema migrations; see Migrations.
  • migration-lock string
    The lease of the process running migrations, expiring on its own.
  • job-lock:<id> string
    Marks a pending job an exclusive read holds, for up to 60 seconds.
  • commit:<token>, claim:<token> string
    Markers of an applied commit and of the jobs a claim took, kept 60 seconds so a resent script applies once.

Indexes

Every write of a record updates its index entries in the same script, and so does the claim of job attempts. A job sits in an index while it matches the index's condition, such as status = 'pending' for due:, with the job id as the member and the named timestamp as the score. The hashes count records per type name or blocker chain, and a field goes away when its count reaches zero.

indexes

  • chain-jobs:<chainId> zset chain_index_idx
    The jobs of a chain scored by chain index: chain listings and the chain's tail.
  • jobs:createdAt:all:<typeName> zset job_idx
    Every job of a type by creation time: job listings without a status filter.
  • jobs:createdAt:<status>:<typeName> zset
    The jobs of a type in one status by creation time: per-status counts and listings.
  • jobs:scheduledAt:pending:<typeName> zset job_pending_idx
    Pending jobs by scheduled time: listings of pending jobs.
  • due:<typeName> zset job_pending_idx
    Pending jobs by scheduled time, with the creation order in the member to break ties: the claim scans it for due jobs.
  • jobs:scheduledAt:blocked:<typeName> zset job_blocked_idx
    Blocked jobs by scheduled time: listings of blocked jobs.
  • jobs:attemptAt:running:<typeName> zset job_running_idx
    Running jobs by attempt start: listings of running jobs.
  • jobs:completedAt:completed:<typeName> zset job_completed_idx
    Completed jobs by completion time: listings of completed jobs.
  • running:<workerId> zset job_attempt_by_idx
    The jobs a worker is running, by attempt start: a worker's jobs and their count.
  • job-type-names hash
    The number of jobs of each type: the job type names in use.
  • chains:createdAt:all:<typeName> zset chain_idx
    Chain heads by creation time: chain listings without a status filter.
  • chains:createdAt:<status>:<typeName> zset chain_running_idx
    Chain heads in one status by creation time: per-status counts and listings.
  • chains:completedAt:completed:<typeName> zset chain_completed_idx
    Completed chain heads by completion time.
  • chain-type-names hash
    The number of chains of each type: the chain type names in use.
  • blocked-by:<chainId> set job_blocker_chain_idx
    The jobs a chain blocks: unblocking them when it completes, and refusing to delete it while they exist.
  • blocked-jobs:<chainId> zset job_blocker_chain_idx
    The jobs a chain blocks by creation time: listings of a chain's blocked jobs.
  • blocker-references hash
    The number of jobs each chain blocks: chain listings filtered by whether a chain blocks others.
  • workers, workers-of-type:<typeName> zset worker_idx
    Workers by start time, all of them and per job type: worker listings.
  • worker-leases, worker-leases-of-type:<typeName> zset
    Workers by lease end, all of them and per job type: live and expired counts.

Schema changes

A schema change ships as a numbered migration that migrateToLatest() applies. A change to an index definition ships with a migration that rebuilds the indexes: it drops every index key and computes them again from the job and worker records, in batches that each block Redis briefly. A breaking change also replaces the version in schema-fences, so older processes stop writing while it runs.

Redis configuration

Redis is the only copy of Queuert's state, so configure it as a database, not a cache:

  • Persistence. Turn on AOF (appendonly yes); without it a restart loses every chain, job and worker written since the last snapshot.
  • No eviction. Set maxmemory-policy noeviction, so a full Redis rejects writes instead of deleting job records or the expiring lock keys.
  • Memory. Every job stays in memory until its chain is deleted, which a cleanup job does on a schedule. A job takes about 1.5 KB with its index entries, plus its input, output and blockers, so set maxmemory for the number of jobs you keep, not for the queue length.

Redis Cluster

Every key starts with keyPrefix, whose hash tag sends all of them to one slot, so each script runs on one node. A cluster gives that Queuert availability, not more throughput: one node holds all of its state. Adapters with different hash tags can land on different nodes. Replication to replicas is asynchronous, so a failover can lose the last writes the primary acknowledged.

Testing your provider

@queuert/redis/conformance checks a provider you write. redisStateProviderConformanceCases are checks of the eval contract followed by every state adapter case the Redis adapter meets, including checks of its write fence. createRedisStateProviderConformanceContext takes a createStateProvider factory; each case builds its own adapter over a new provider under a key prefix of its own, migrates it, and closes it when it ends. The suite drives no test runner: your test file owns the client in its own hooks and registers one test per case:

src/provider.spec.ts

describe("redis/state-redis provider conformance", () => {
  let redis: AcquiredRedis;
  let client: RedisClientType;
  let context: RedisStateProviderConformanceContext;

  beforeAll(async () => {
    redis = await acquireRedis("redis:8");
    client = createClient({ url: redis.connectionUrl });
    await client.connect();
    context = createRedisStateProviderConformanceContext({
      createStateProvider: async () => createNodeRedisStateProvider({ client }),
    });
  }, 60_000);

  afterAll(async () => {
    await client.close();
    await redis[Symbol.asyncDispose]();
  });

  for (const conformanceCase of redisStateProviderConformanceCases) {
    test(conformanceCase.title, async () => {
      await conformanceCase.run(context);
    });
  }
});

The full API is in the conformance reference.

Examples

@queuert/redis

  • redis/state-redis node-redis Docker · Redis
    GitHub
    Experimental Redis state storage via @queuert/redis with the node-redis client — a chain created in an optimistic transaction, with its side effect in onTransactionCommit because the callback may run more than once.
    $ pnpm --filter example-redis-state-redis startsrc/provider.tssrc/index.ts
  • redis/state-ioredis ioredis Docker · Redis
    GitHub
    Experimental Redis state storage via @queuert/redis with the ioredis client — a chain created in an optimistic transaction, with its side effect in onTransactionCommit because the callback may run more than once.
    $ pnpm --filter example-redis-state-ioredis startsrc/provider.tssrc/index.ts
  • redis/multi-worker multiple workers on Redis Docker · Redis
    GitHub
    Three worker processes (forked from the main process) share one Redis instance for state and compete for jobs, waking each other through the Redis notify adapter — the main process creates a chain per order in one optimistic transaction. Horizontal Scaling guide →
    $ pnpm --filter example-redis-multi-worker startsrc/index.tssrc/worker.ts

The API reference for every export is in @queuert/redis.

esc

Search guides, integrations and every exported symbol.

↑ ↓ to navigate↵ to open