OpenTelemetry Tracing
OpenTelemetry span hierarchy, span kinds and attributes.
Tracing through the @queuert/otel adapter provides end-to-end visibility into chain execution, including job dependencies, retry attempts, and blocker relationships.
Span hierarchy
Queuert uses a four-level span hierarchy. Span kinds use OpenTelemetry's PRODUCER/CONSUMER semantics. The chain has both a PRODUCER (creation) and CONSUMER (completion) span for symmetry.
trace
chain render-invoice-pdf
- PRODUCER create chain.render-invoice-pdf
- PRODUCER create job.render-invoice-pdf
- CONSUMER start job-attempt.render-invoice-pdf ERROR
- CONSUMER start job-attempt.render-invoice-pdf OK
- PRODUCER create job.sync-to-crmlink → create job.render-invoice-pdf
- CONSUMER start job-attempt.sync-to-crm OK
- CONSUMER complete chain.render-invoice-pdflink → create chain.render-invoice-pdf
The row under each span lists its attributes, defined under Span attributes.
Spans
- create chain.{type} PRODUCERCreated:
createChain(). Ended: on commit. Duration: until the transaction commits.queuert.chain.id queuert.chain.type queuert.chain.deduplicated - create job.{type} PRODUCERCreated:
createChain(),complete({ continueWith: ... }). Ended: on commit. Duration: until the transaction commits.queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.chain.deduplicated - await chain.{type} PRODUCERCreated:
createChain()with blockers. Ended: on commit. Duration: until the transaction commits.queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.blocker.chain.id queuert.blocker.chain.type queuert.blocker.index - complete chain.{type} CONSUMERCreated: blocker chain completes. Ended: immediately. Duration: ~0ms. No attributes.
- start job-attempt.{type} CONSUMERCreated: worker claims job. Ended: attempt completes/fails. Duration: processing time.queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.job.attempt queuert.worker.id queuert.continued_with.job_id queuert.continued_with.job_type
- complete job.{type} CONSUMERCreated: workerless completion. Ended: immediately. Duration: ~0ms.queuert.chain.id queuert.chain.type queuert.job.id queuert.job.type queuert.continued_with.job_id queuert.continued_with.job_type
- complete chain.{type} CONSUMERCreated: final job completes. Ended: immediately. Duration: ~0ms.queuert.chain.id queuert.chain.type
Blocker relationships
When a job has blockers (dependencies on other chains), each blocker gets a PRODUCER/CONSUMER span pair as a child of the blocked job's PRODUCER span. The PRODUCER (await chain.{type}) is created at createChain time with a link to the blocker chain. The CONSUMER (complete chain.{type}) is created when the blocker chain completes, so the time between them represents the blocking duration.
trace
POST /orders
- SERVER POST /orders
- PRODUCER create chain.capture-payment
- PRODUCER create job.capture-payment
- CONSUMER start job-attempt.capture-payment OK
- CONSUMER complete chain.capture-paymentlink → create chain.capture-payment
- PRODUCER create chain.ship-order
- PRODUCER create job.ship-order
- PRODUCER await chain.capture-paymentlink → create chain.capture-payment
- CONSUMER complete chain.capture-payment
- CONSUMER start job-attempt.ship-order OK
- CONSUMER complete chain.ship-orderlink → create chain.ship-order
- PRODUCER created in
createChainand ended when the transaction commits, when the job has blockers — one PRODUCER span per blocker, as a child of the job's PRODUCER span, with a link to the blocker chain's trace context. - Persisted — the PRODUCER span context is stored in the
job_blockertable (trace_contextcolumn) so the CONSUMER can be created by another process. - CONSUMER created when the blocker chain has completed — the PRODUCER span context is read from
job_blockerand a CONSUMER span is created as its child. If the blocker chain is already completed at creation time,createChaincreates the CONSUMER span right away instead.
Workerless completion
When a job is completed via completeChain (without a worker), there is no job-attempt. Instead, a CONSUMER job span marks the completion, and if the chain is fully completed, a CONSUMER chain span closes the trace:
trace
chain approve-refund
- PRODUCER create chain.approve-refund
- PRODUCER create job.approve-refund
- CONSUMER complete job.approve-refund
- CONSUMER complete chain.approve-refundlink → create chain.approve-refund
The CONSUMER job span is a child of the PRODUCER job span and carries the same chain/job attributes. When continueWith is called during workerless completion, the CONSUMER chain span is omitted (the chain continues).
Chain completion link
The final complete chain.{type} CONSUMER span includes a link to the chain's PRODUCER span context (chainTraceContext), connecting the chain's creation and completion across processes and time.
Deduplication
When createChain is called with deduplication options and a matching chain already exists, no new chain is created. Deduplication is not an error — per OpenTelemetry status conventions, the span status remains UNSET. When deduplication occurs, the spans:
- get the attribute
queuert.chain.deduplicated: true; - reference the existing chain's IDs;
- link the
create jobspan to the existing chain's trace context, if available.
trace
two calls with the same deduplication key
- SERVER POST /digests
- PRODUCER create chain.daily-digestqueuert.chain.id: 0193…a1
- PRODUCER create job.daily-digest
- SERVER POST /digests
- PRODUCER create chain.daily-digestqueuert.chain.id: 0193…a1 queuert.chain.deduplicated: true
- PRODUCER create job.daily-digestlink → create chain.daily-digest (first) queuert.chain.deduplicated: true
Span attributes
Span attributes
- queuert.chain.id stringChain ID.
- queuert.chain.type stringChain type name.
- queuert.chain.deduplicated boolean
truewhen chain was deduplicated; absent otherwise. - queuert.job.id stringJob ID.
- queuert.job.type stringJob type name.
- queuert.job.attempt numberAttempt number.
- queuert.worker.id stringWorker ID processing the attempt.
- ID of the continuation job; present when the job continued.
- Type name of the continuation job; present when the job continued.
- queuert.blocker.chain.id stringBlocker chain ID.
- queuert.blocker.chain.type stringBlocker chain type name.
- queuert.blocker.index numberIndex of the blocker in the blockers array.