Job Processing

How an attempt runs, finishing in your own transaction, and attempt ownership.

Overview

This document describes how Queuert processes jobs: the transactional design, how an attempt runs from acquisition to complete, and how ownership is enforced without holding a lock for the duration of the attempt.

Transactional Design

Queuert's core design principle is that jobs are created inside the same database transaction as your application state changes. This follows the transactional outbox pattern:

TypeScript

await withTxHooks(async (txHooks) =>
  db.transaction(async (tx) => {
    // Application state change
    const image = await tx.images.create({ ... });

    // Job creation in the same transaction
    // The transaction context property name matches your StateProvider
    await client.createChain({
      tx,
      txHooks,
      typeName: "process-image",
      input: { imageId: image.id },
    });
  }),
);

Why This Matters

  1. Atomicity: If the transaction rolls back, the job is never created. No orphaned jobs.
  2. Consistency: The job always references valid application state.
  3. No dual-write problem: You don't need to coordinate between your database and a separate job queue.

Extending to Job Processing

The same principle extends to job processing. Queuert opens no transaction around an attempt: the handler does its work — reads, external API calls, long computations — and then opens its own transaction, writes its domain changes, and calls complete inside it. The outcome (an output that completes the chain, a continueWith that adds the next job, or a reschedule) commits or rolls back together with those domain writes.

TypeScript

attemptHandler: async ({ job, complete }) => {
  const { paymentId } = await paymentAPI.charge(job.input.amount);

  return withTxHooks(async (txHooks) =>
    db.transaction(async (tx) => {
      await tx.orders.update({ where: { id: job.input.orderId }, data: { paymentId } });
      return complete({ tx, txHooks, output: { paymentId } });
    }),
  );
},

complete registers its observability events and notifications through the state provider's onTransactionCommit, so they are released only after your transaction commits. If it rolls back, nothing leaks out; see Tx Hooks.

The Attempt Lifecycle

acquirepending→running
attempt handler
your workno transaction
your transaction
commitdomain writes complete({ ...txCtx, outcome })
commit detection
committed
attempt completed
not committed
reschedulerunning→pending
  1. Acquire. The worker starts an attempt under its id, which it holds while the worker's lease is live; see Worker Runner.
  2. Handler. Your handler does its work, then opens a transaction, calls complete in it, and returns the complete result.
  3. Commit detection. The worker records the attempt as completed if the commit callbacks of a complete call ran, and otherwise fails it; see Errors and Commit Detection.

complete

complete ends the attempt. Whether the job ends up completed depends on the outcome you pass. It takes your transaction context spread alongside exactly one outcome — the same convention as client.createChain:

  • complete({ ...txCtx, output }) — completes the job and its chain, and unblocks jobs waiting on the chain.
  • complete({ ...txCtx, continueWith: { typeName, input, blockers?, schedule? } }) — completes the job and adds the next job to the chain.
  • complete({ ...txCtx, reschedule: { afterMs } | { at } }) — returns the job to pending without counting the attempt as failed.

The transaction context is required, including every field the provider's context type requires, such as the txHooks of a provider that carries them in its context. complete performs several writes (for example completing the chain and unblocking its dependents), so it always needs your transaction. It throws if the state adapter's isTransaction reports the context as outside an open transaction, for example when you pass a pool or the root ORM handle. On PostgreSQL, Queuert supports only READ COMMITTED, the default; see PostgreSQL State Adapter. On SQLite with an async provider it must hold the provider's write lock; see SQLite State Adapter. On Redis it must be the context the adapter's withTransaction passed; see Redis State Adapter.

Return the complete result from the handler. The handler's return type is the finished job, so forgetting complete is a compile error. Return it out of your transaction and hooks callbacks: return withTxHooks(async (txHooks) => db.transaction(async (tx) => complete({ tx, txHooks, output }))). The type only proves complete was called; the worker does not compare the returned value at runtime, and whether the attempt completed is decided by commit detection.

With a synchronous state adapter, complete is synchronous. Processors built with createProcessors over a client of a synchronous state adapter give the handler a complete that returns the job directly and throws synchronously, so it runs inside the driver's own transaction and an error rolls that transaction back: return withTxHooks((txHooks) => db.transaction(() => complete({ db, txHooks, output }))()). The handler itself stays async; fence is synchronous the same way.

One complete per attempt. A second or concurrent call throws an Error before writing anything and leaves the attempt running, whether or not the first one's transaction has committed yet. A call that threw before writing (for example a validation error), or whose transaction or savepoint rolled back, does not count, so you can catch the error and call complete again. complete cannot be called after the handler has ended.

Attempt Ownership

Every write tied to an attempt is conditional on the job row: the job must be running and still hold the attempt token it was given when this attempt acquired it. Each acquisition gets a new token, so an earlier attempt, or one of a chain deleted and recreated under the same id, never matches. A conditional write that does not match writes nothing at all — no completed job, no chain update, no continuation. This is what makes complete safe without Queuert holding a lock for the duration of the attempt.

When complete's write misses, complete writes nothing, throws and aborts the attempt's signal; let the error propagate so your transaction rolls back with it.

fence

complete checks ownership for its own transaction only. For writes in the other transactions your handler opens, call fence({ ...txCtx }) at the start of that transaction. It takes the same transaction context as complete, with the same isolation and SQLite write-lock requirements, though like a read it needs only part of it: fence({ tx }) without the txHooks some providers carry is enough. It locks the job row and its chain head and checks that the job is running under this attempt's token. Reclaiming the job, completing it with client.completeChain and deleting its chain all lock or update the same rows, so they wait until your transaction commits or rolls back: every write that commits after a successful fence committed while the attempt owned the job. On Redis nothing waits: the transaction that commits second runs again, and its fence then throws; see Locks.

TypeScript

attemptHandler: async ({ signal, fence, complete }) => {
  for (const batch of batches) {
    if (signal.aborted) break;
    await db.transaction(async (tx) => {
      await fence({ tx });
      await writeBatch(tx, batch);
    });
  }
  return withTxHooks(async (txHooks) =>
    db.transaction(async (tx) => complete({ tx, txHooks, output: null })),
  );
},

When the attempt no longer owns the job, fence throws the same error complete would and aborts signal; let it propagate so the transaction rolls back. fence writes nothing, so it can be called any number of times. In the transaction that calls complete, call it before complete: complete ends the attempt, so a later fence treats it as lost. While a fenced transaction is open, reclaim, client writes to the chain and new chains that use it as a blocker wait for it, so keep fenced transactions short. On PostgreSQL, a fenced transaction that then locks other chains, for example by deleting them, can deadlock with a client call that locks the same chains in a different order; PostgreSQL aborts one of the two transactions, and a handler that lets the error propagate is rescheduled. Transactions without fence are not checked against ownership: their writes can commit after the attempt was lost, so keep them idempotent.

Errors and Commit Detection

The attempt counts as completed once a successful complete call's commit callbacks run, which the state provider's onTransactionCommit does after your transaction commits, so a successful attempt needs no extra database round-trip. If no complete call's commit callbacks ran — the handler threw before committing, or returned without calling complete — the attempt fails: the worker runs one reschedule under the attempt token, which reschedules the job with backoff while the attempt still owns it and matches nothing if a complete committed or the attempt was lost. An error thrown after a committed complete does not roll the completion back; it is reported as a workerError.

Commit callbacks stand for a committed write only when the state provider meets these requirements:

  • Commit callbacks run only after COMMIT. A provider that carries TxHooks in its context creates them with withTxHooks around the transaction, so they flush after COMMIT; created inside the transaction, they would flush before COMMIT, and a COMMIT that then fails leaves nothing written. When you open the transaction yourself for such a provider, wrap it the same way.
  • Callbacks follow savepoints. A complete inside a database savepoint that rolls back must have its commit callbacks dropped with it. A provider with transaction events does this itself; with TxHooks in the context, pair the database savepoint with txHooks.withSavepoint (or a createSavepoint handle rolled back alongside it) and pass the savepoint's hooks to complete. See Tx Hooks.
  • A try the driver reruns drops its callbacks. When the driver reruns the transaction callback, the rolled-back try's complete must not count. With TxHooks, wrap the rerun callback in txHooks.retryBoundary. See Tx Hooks.
  • A rolled-back transaction is reported as rolled back. A knex transaction resolves after a bare trx.rollback(), which withTxHooks takes for a commit, unless it is given { doNotRejectOnRollback: false }. See Tx Hooks. MikroORM fires no rollback event when PostgreSQL answers COMMIT with ROLLBACK after a caught statement error, so let statement errors propagate out of em.transactional.

If any of them is broken, the worker can count the attempt as completed although nothing was written. The job then stays running until the worker is next idle and reclaims it as a failed attempt; see Worker Runner.

See Job Processing Reliability for each scenario with code examples, and Error Handling for handling permanently failing jobs.

Abort Signal

Every attempt handler receives a signal that aborts when the worker is stopping, when its worker lost its lease, or when the attempt is lost (taken by another worker, deleted, or completed externally). Your own committed complete never aborts it. On abort, stop working and commit a complete — with partial output or { reschedule } — or throw to reschedule with backoff. If the attempt was lost, complete and fence write nothing and throw. A handler that returns without a committed complete fails the attempt.

The batch handler below stops at the next item once its signal aborts and completes with the number of items it processed, so a worker that stops mid-batch still commits partial output:

src/signals.ts

"process-batch": {
  attemptHandler: async ({ signal, job, complete }) => {
    let processed = 0;
    for (const item of job.input.items) {
      if (signal.aborted) break;
      await processItem(item);
      processed++;
    }
    return stateAdapter.withTransaction(async (txCtx) =>
      complete({ ...txCtx, output: { processed } }),
    );
  },
},

Completing a chain from outside the worker also aborts the signal of the attempt still running it. The await-payment-confirmation handler below polls the payment provider until its signal aborts, then throws signal.reason:

src/signals.ts

"await-payment-confirmation": {
  attemptHandler: async ({ signal, job, complete }) => {
    while (!signal.aborted) {
      if (await pollPaymentProvider(job.input.paymentId)) {
        return stateAdapter.withTransaction(async (txCtx) =>
          complete({ ...txCtx, output: { confirmedBy: "poll" } }),
        );
      }
    }
    throw signal.reason;
  },
},

A payment webhook completes the job with completeChain while the handler polls. The handler stops polling, and since its attempt no longer owns the job, nothing it does afterwards is written:

src/signals.ts

await stateAdapter.withTransaction(async (txCtx) =>
  client.completeChain({
    ...txCtx,
    typeName: "await-payment-confirmation",
    id: payment.id,
    handler: async ({ job, completeJob }) => {
      await completeJob(job, async ({ complete }) =>
        complete({ output: { confirmedBy: "webhook" } }),
      );
    },
  }),
);

See Also

esc

Search guides, integrations and every exported symbol.

↑ ↓ to navigate↵ to open