By This Hour Development Desk

TypeScript background jobs move slow, fallible, or independently scheduled work out of an HTTP request. That separation improves responsiveness, but it also removes the simple request-and-response boundary that often hides failure decisions. A worker may run a message more than once, lose a process during an external call, or receive malformed data long after the original request completed.

The practical goal is not to guarantee that a handler runs only once. A durable design assumes delivery can repeat and makes the intended business effect safe when it does. This workflow gives each job a defined contract, a bounded retry policy, a durable idempotency decision, usable telemetry, and a recovery path for work that cannot complete.

What you will learn

By the end, you will have a repeatable design review and implementation sequence for asynchronous work outside the request lifecycle:

  1. Define a small job contract that crosses the queue boundary with an operation key, validated data, and correlation context.
  2. Classify failures, retry only safe transient work with a cap, and make duplicate execution converge on one durable result.
  3. Operate workers with traces, metrics, structured logs, graceful shutdown, redelivery, and reconciliation.

This approach deliberately separates delivery from effect. The queue may redeliver a job after a crash or timeout. Your application should still produce either one recorded business outcome or one clear, durable failure state. That distinction is central to making retries safe: AWS’s discussion of idempotent APIs and retries emphasizes a caller-supplied token, durable state, and a consistent result for repeated requests.

The trade-off is more up-front state and discipline. You must store operation records, retain enough information to diagnose failures, and decide which side effects can be repeated safely. In return, a timeout stops being an invitation for guesswork.

Before you start

Start with one concrete business operation, not a generic “run task” abstraction. Write down its input, intended outcome, owner, maximum useful age, dependencies, and recovery owner. For example, “create a requested export” is an operation; “call a service” is an implementation detail. Decide what the producer may assume after enqueueing: normally, only that work was accepted for later processing, not that it succeeded.

Choose durable storage appropriate to your system for two kinds of records: queued work and operation state. The worker needs an atomic or otherwise reliable way to determine whether an operation key is new, completed, already being processed, or invalid because its payload differs from the original. Keep the queue adapter behind a narrow interface because acknowledgement, visibility, leasing, and delayed-delivery rules vary by provider.

Define these failure classes before writing retry code:

  • Transient: a temporary dependency, network, or capacity failure that may succeed later.
  • Permanent: invalid input, missing required data, or a business rule violation that another attempt will not repair.
  • Unknown: an interruption or ambiguous external outcome. Treat it as potentially repeated work and resolve it through the idempotency record or reconciliation rather than declaring success.

Also establish operating limits: maximum attempts, a retry window, concurrency, a dead-letter or quarantine destination, and an alert owner. Retry limits prevent failing work from creating sustained load; AWS’s reliability guidance recommends limiting retries rather than allowing them to continue indefinitely at its guidance on limiting retries for interaction failures.

Step 1: Design the job contract and queue boundary

Make the message a versioned statement of intent. Include an immutable job identifier for delivery diagnostics, a stable operation key for business deduplication, a job type, payload data, creation time, and correlation ID. Validate the type and fields before business processing. Do not let a consumer reconstruct essential intent from current HTTP-session state, logs, or a mutable object that was never stored with the job.

The operation key must come from the caller or be deterministically established when the business operation begins. It is not a fresh random value generated on every attempt. Store a normalized representation or fingerprint of the meaningful parameters alongside that key. If the same key arrives with changed parameters, reject it rather than silently treating two distinct requests as one. This prevents a token from hiding a caller defect.

The queue boundary should return a simple outcome: accepted for processing, rejected before enqueueing, or already associated with a prior operation. Keep acknowledgement separate from the handler’s final business result. Acknowledging too early can lose work; acknowledging only after a durable completed or quarantined state makes redelivery useful.

Worked example: an export request crosses from an API into a worker

Scenario: A client asks for an account export and repeats its request after its connection times out. The worker must recognize both deliveries as the same requested export.

Example: The boundary replaces an unstructured task description with explicit identifiers and request context that can survive independent worker execution.

Before
{ "task": "make export", "accountId": "a-42" }

After
{
  "jobId": "job-701",
  "operationKey": "client-request-884",
  "type": "account-export.requested",
  "payload": { "accountId": "a-42", "format": "csv" },
  "correlationId": "req-193",
  "createdAt": "2025-01-01T00:00:00Z"
}

What this shows: jobId identifies a delivery, while operationKey identifies the business intent. A retry or redelivery may have a different delivery identifier in some designs, but it must retain the operation key.

Validate that every consumer-required field is present and that the payload matches the expected type. Then persist or enqueue the complete message according to your system’s durability rules. Test the boundary by submitting the same operation key twice and by submitting that key with a modified format. The first case should converge on the prior operation; the second should be an explicit conflict. The cost is a little more schema maintenance, but the contract becomes inspectable during incidents.

Step 2: Bound retries and make repeated execution safe

Build the handler around a durable operation record. First, look up or create the record keyed by operationKey and compare its stored parameters with the incoming payload. If it is already complete, return the recorded outcome without repeating the side effect. If it is active, use your lease or ownership rules to avoid concurrent processing. If the work is new, record that processing has begun before performing the effect.

After a transient error, calculate an exponential delay, add jitter, and stop at a fixed attempt or elapsed-time limit. Jitter spreads retries that would otherwise return to a recovering dependency together. Do not retry permanent validation and data errors. For an uncertain result from an external system, query or reconcile using the same stable operation key where possible; blindly issuing a second effect is the unsafe option.

Worked example: a temporary dependency failure receives a bounded retry

Scenario: An export worker has a retry-safe operation record and a dependency reports a temporary failure on attempt two. The team allows no more than five attempts.

Example: The policy grows the delay, varies it slightly, and routes the final failed attempt away from the normal queue.

attempt: 2
baseDelayMs: 1000
maximumDelayMs: 30000
jitterMs: random value from 0 to 500

rawDelayMs: 1000 * 2^(2 - 1)
delayMs: minimum(2000 + jitterMs, 30000)

attempt < 5: requeue after delayMs
attempt = 5: write quarantine record

What this shows: The attempt count is part of the worker state, not an accidental loop counter. A cap gives operators a definite point at which intervention replaces automated pressure on a dependency.

Persist the outcome and any external reference in the same durable decision flow your storage model supports. If the process stops between an external call and marking completion, a later attempt must inspect durable evidence before calling again. Record the error classification and the next action. Send permanent failures and exhausted transient failures to a durable dead-letter or quarantine path with the message, attempt history, error context, and correlation ID. Quarantine is not disposal: assign review, correction, replay, or cancellation procedures.

Step 3: Instrument execution and recover safely

Instrument every job from dequeue through acknowledgement or quarantine. OpenTelemetry’s JavaScript support can be used in TypeScript and Node.js workers to create active spans and attach job identifiers, attempt numbers, outcomes, and error events. Use the correlation ID to connect producer logs, worker logs, and relevant downstream work. Avoid logging sensitive payload content; prefer identifiers, job type, state transition, duration, and an error category.

At minimum, publish queue age, attempt count, failure rate, processing latency, active jobs, retry count, and quarantine count. Check these together. Low worker activity with rising queue age indicates intake or capacity trouble; rising attempts and latency can point to a struggling dependency. Traces explain an individual path, metrics reveal a trend, and structured logs preserve exact state transitions.

Worked example: an interrupted worker is recorded as unfinished, not successful

Scenario: A worker has started an export but receives shutdown while it still owns in-flight work. The system must preserve recovery evidence.

Example: The shutdown path stops new intake, gives active work a bounded chance to finish, and leaves unfinished work eligible for redelivery or reconciliation.

state: accepting

on shutdown:
  state: draining
  stop receiving new jobs
  wait until deadline for active jobs
  persist unfinished operation state
  release or allow lease expiry
  exit without success acknowledgement

on next delivery:
  inspect operation record
  reconcile or resume safely

What this shows: A shutdown is a state transition, not proof that active business work finished. The next worker has enough durable context to make a safe decision.

Node.js worker threads provide independent JavaScript execution threads and message passing, but termination can stop execution abruptly. Whether your jobs run in worker threads or separate processes, forced termination must never be treated as a successful acknowledgement. Stop intake first, drain for a defined period, then preserve or release in-flight ownership so redelivery can occur. Reconciliation should periodically find records stuck in processing beyond their lease or expected duration and resolve them through evidence, retry, or quarantine.

Common mistakes to avoid

  • Retrying every exception: this amplifies invalid requests and repeated side effects. Require an explicit transient classification.
  • Using the message ID as deduplication: delivery IDs may change on redelivery. Deduplicate with the stable business operation key.
  • Marking completion only in memory: a restart erases the decision. Persist the operation status and relevant result.
  • Acknowledging before durable completion: this creates lost work if the process stops after acknowledgement.
  • Logging only error text: without operation key, attempt, job type, and correlation ID, an incident cannot be reconstructed reliably.

Pre-publish checklist

  • Confirm every job has validated type, payload, operation key, and correlation ID.
  • Test duplicate delivery, changed parameters under the same key, transient failure, permanent failure, and an ambiguous interruption.
  • Verify retries have backoff, jitter, a hard limit, and a quarantine destination.
  • Verify completed outcomes and parameter fingerprints are durable and repeated requests receive consistent handling.
  • Verify dashboards expose age, attempts, failures, latency, and quarantine volume, and that traces and logs carry correlation context.
  • Simulate shutdown during active work and confirm no forced stop is recorded as success.

Sources