OpenTelemetry Tracing

OpenTelemetry span hierarchy, span kinds and attributes.

Tracing through the @queuert/otel adapter provides end-to-end visibility into chain execution, including job dependencies, retry attempts, and blocker relationships.

Span hierarchy

Queuert uses a four-level span hierarchy. Span kinds use OpenTelemetry's PRODUCER/CONSUMER semantics. The chain has both a PRODUCER (creation) and CONSUMER (completion) span for symmetry.

trace

chain render-invoice-pdf

  • PRODUCER create chain.render-invoice-pdf
  • PRODUCER create job.render-invoice-pdf
  • CONSUMER start job-attempt.render-invoice-pdf ERROR
  • CONSUMER start job-attempt.render-invoice-pdf OK
  • PRODUCER create job.sync-to-crm
    link → create job.render-invoice-pdf
  • CONSUMER start job-attempt.sync-to-crm OK
  • CONSUMER complete chain.render-invoice-pdf
    link → create chain.render-invoice-pdf

The row under each span lists its attributes, defined under Span attributes.

Spans

  • Created: createChain(). Ended: on commit. Duration: until the transaction commits.
    queuert.chain.id queuert.chain.type queuert.chain.deduplicated
  • Created: createChain(), complete({ continueWith: ... }). Ended: on commit. Duration: until the transaction commits.
    queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.chain.deduplicated
  • Created: createChain() with blockers. Ended: on commit. Duration: until the transaction commits.
    queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.blocker.chain.id queuert.blocker.chain.type queuert.blocker.index
  • Created: blocker chain completes. Ended: immediately. Duration: ~0ms. No attributes.
  • Created: worker claims job. Ended: attempt completes/fails. Duration: processing time.
    queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.job.attempt queuert.worker.id queuert.continued_with.job_id queuert.continued_with.job_type
  • Created: workerless completion. Ended: immediately. Duration: ~0ms.
    queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.continued_with.job_id queuert.continued_with.job_type
  • Created: final job completes. Ended: immediately. Duration: ~0ms.
    queuert.chain.id queuert.chain.type

Blocker relationships

When a job has blockers (dependencies on other chains), each blocker gets a PRODUCER/CONSUMER span pair as a child of the blocked job's PRODUCER span. The PRODUCER (await chain.{type}) is created at createChain time with a link to the blocker chain. The CONSUMER (complete chain.{type}) is created when the blocker chain completes, so the time between them represents the blocking duration.

trace

POST /orders

  • SERVER POST /orders
  • PRODUCER create chain.capture-payment
  • PRODUCER create job.capture-payment
  • CONSUMER start job-attempt.capture-payment OK
  • CONSUMER complete chain.capture-payment
    link → create chain.capture-payment
  • PRODUCER create chain.ship-order
  • PRODUCER create job.ship-order
  • PRODUCER await chain.capture-payment
    link → create chain.capture-payment
  • CONSUMER complete chain.capture-payment
  • CONSUMER start job-attempt.ship-order OK
  • CONSUMER complete chain.ship-order
    link → create chain.ship-order
  1. PRODUCER created in createChain and ended when the transaction commits, when the job has blockers — one PRODUCER span per blocker, as a child of the job's PRODUCER span, with a link to the blocker chain's trace context.
  2. Persisted — the PRODUCER span context is stored in the job_blocker table (trace_context column) so the CONSUMER can be created by another process.
  3. CONSUMER created when the blocker chain has completed — the PRODUCER span context is read from job_blocker and a CONSUMER span is created as its child. If the blocker chain is already completed at creation time, createChain creates the CONSUMER span right away instead.

Workerless completion

When a job is completed via completeChain (without a worker), there is no job-attempt. Instead, a CONSUMER job span marks the completion, and if the chain is fully completed, a CONSUMER chain span closes the trace:

trace

chain approve-refund

  • PRODUCER create chain.approve-refund
  • PRODUCER create job.approve-refund
  • CONSUMER complete job.approve-refund
  • CONSUMER complete chain.approve-refund
    link → create chain.approve-refund

The CONSUMER job span is a child of the PRODUCER job span and carries the same chain/job attributes. When continueWith is called during workerless completion, the CONSUMER chain span is omitted (the chain continues).

The final complete chain.{type} CONSUMER span includes a link to the chain's PRODUCER span context (chainTraceContext), connecting the chain's creation and completion across processes and time.

Deduplication

When createChain is called with deduplication options and a matching chain already exists, no new chain is created. Deduplication is not an error — per OpenTelemetry status conventions, the span status remains UNSET. When deduplication occurs, the spans:

  • get the attribute queuert.chain.deduplicated: true;
  • reference the existing chain's IDs;
  • link the create job span to the existing chain's trace context, if available.

trace

two calls with the same deduplication key

  • SERVER POST /digests
  • PRODUCER create chain.daily-digest
    queuert.chain.id: 0193…a1
  • PRODUCER create job.daily-digest
  • SERVER POST /digests
  • PRODUCER create chain.daily-digest
    queuert.chain.id: 0193…a1 queuert.chain.deduplicated: true
  • PRODUCER create job.daily-digest
    link → create chain.daily-digest (first) queuert.chain.deduplicated: true

Span attributes

Span attributes

esc

Search guides, integrations and every exported symbol.

↑ ↓ to navigate↵ to open