Deduplication

Prevent duplicate chains with deduplication keys.

Deduplication prevents duplicate chains from being created. When you start a chain with a deduplication key, Queuert checks if a chain of the same type with that key already exists and returns the existing chain instead of creating a new one.

createChain #1key: sync:user:123
creates
chain abc-123deduplicated: false
createChain #2key: sync:user:123
matches
chain abc-123deduplicated: true
createChain #3key: sync:user:123
matches
chain abc-123deduplicated: true

Both calls below pass the same key while the first chain is still running, so the second returns the first chain: secondSync.deduplicated is true and secondSync.id equals firstSync.id.

src/deduplication.ts

const firstSync = await stateAdapter.withTransaction(async (txCtx) =>
  client.createChain({
    ...txCtx,
    typeName: "sync-user",
    input: { userId: "123" },
    deduplication: { key: "sync:user:123", scope: "running" },
  }),
);

const secondSync = await stateAdapter.withTransaction(async (txCtx) =>
  client.createChain({
    ...txCtx,
    typeName: "sync-user",
    input: { userId: "123" },
    deduplication: { key: "sync:user:123", scope: "running" },
  }),
);

Deduplication Modes

The required scope option controls what chains to check for duplicates, and a missing or unknown value throws a TypeError:

  • running — Dedup against running chains only, so the same key creates a new chain once the previous one completes
  • any — Dedup against any existing chain with this key

The sync-user chains above use running: once the first sync completes, the same key creates a new chain. A welcome email should go out once per user, so it uses any, and calling this again after the email was sent still returns the original chain:

src/deduplication.ts

const sendWelcomeEmail = async (userId: string) =>
  stateAdapter.withTransaction(async (txCtx) =>
    client.createChain({
      ...txCtx,
      typeName: "send-welcome-email",
      input: { userId },
      deduplication: { key: `welcome:${userId}`, scope: "any" },
    }),
  );

Self-Scheduling Recurring Chains

A recurring chain schedules its next occurrence from its own terminal job. Under scope: "running" the ordering matters: complete first, then create the next chain in the same transaction. complete({ output: ... }) applies the completion inside your transaction, so by the time createChain runs, the chain being completed already reads as completed and cannot match itself.

src/deduplication.ts

"health-check": {
  attemptHandler: async ({ job, complete }) =>
    stateAdapter.withTransaction(async (txCtx) => {
      const completedJob = await complete({
        ...txCtx,
        output: { checkedAt: new Date().toISOString() },
      });

      await client.createChain({
        ...txCtx,
        typeName: "health-check",
        input: { serviceId: job.input.serviceId },
        schedule: { afterMs: 5 * 60 * 1000 },
        deduplication: { key: `health:${job.input.serviceId}`, scope: "running" },
      });

      return completedJob;
    }),
},

Scheduling before completing matches the still-running chain and silently suppresses the next occurrence, ending the recurrence. The same applies to scheduling from a mid-chain job: that chain is genuinely still running, so schedule from the terminal job. Because the next occurrence is always the running chain under the key, starting the same health check from outside while its next occurrence is scheduled returns that running chain rather than starting a second recurrence.

See examples/guides/src/deduplication.ts for the runnable code. See also Scheduling and Tx Hooks.

esc

Search guides, integrations and every exported symbol.

↑ ↓ to navigate↵ to open