Async job queue patterns in production: retry, dead letter, and concurrency

An async job queue in production needs at least three things to hold up under real traffic: controlled retry with backoff, a dead letter queue for jobs that never recover, and concurrency handling across workers. Without those three pieces, failed jobs vanish silently or pile up until the system stalls. It’s the minimum setup we apply at Snowinch before treating a queue as production-ready.

The problem that shows up after the first deploy

In development the queue always works. Jobs run, tasks complete, nothing looks wrong. In production the first transient error hits, an external service down, a network timeout, a load spike, and you discover the system wasn’t built to absorb it.

Failed jobs sit in error state with no automatic retry. Or they retry forever at the same rate, burning resources and hiding the real issue. Or they disappear silently because nobody configured where exhausted jobs go.

These aren’t edge cases, they’re normal behavior under pressure. The gap between a queue that holds and one that breaks is configuration, not the base architecture.

Retry with exponential backoff

Immediate retry is almost always wrong. If a job fails because an external service is overloaded, retrying instantly adds load to a system already in trouble. Exponential backoff, increasing delay between attempts, gives dependents time to recover.

// Example with BullMQ, check your installed version for available options
// https://docs.bullmq.io/guide/retrying-failing-jobs
const queue = new Queue('notifications', { connection: redisConnection });

await queue.add('send-email', payload, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 2000, // first retry after 2s, then 4s, 8s, 16s, 32s
  },
});

Attempt count depends on job type. For critical work (payments, must-not-lose notifications) you might go to 8–10 attempts with a max delay of a few minutes. For non-critical work (cache refresh, stats) 2–3 attempts is often enough.

Important detail: backoff is measured from the first failure, not the original schedule time. A job scheduled for 10:00 that fails might retry at 10:00:02, 10:00:06, 10:00:14, not 10:02, 10:04, 10:08. That matters when jobs have timing constraints.

Dead letter queue

When a job exhausts all attempts, it has to land somewhere. The dead letter queue (DLQ) is that destination, not a trash bin, but an inspectable archive for manual or automated replay after you fix the underlying issue.

// Worker that moves exhausted jobs to DLQ with context
const worker = new Worker('notifications', async (job) => {
  await processJob(job.data);
}, {
  connection: redisConnection,
});

worker.on('failed', async (job, error) => {
  if (job && job.attemptsMade >= job.opts.attempts) {
    const dlq = new Queue('notifications-dlq', { connection: redisConnection });
    await dlq.add('failed-job', {
      originalJob: job.data,
      error: error.message,
      failedAt: new Date().toISOString(),
      attemptsMade: job.attemptsMade,
    });
  }
});

The DLQ needs monitoring. A job in the DLQ isn’t solved, it’s waiting to be examined. Without alerts on DLQ volume, failures accumulate invisibly.

-- If you use Postgres as queue storage (e.g. pg-boss)
-- alert when DLQ exceeds a threshold (tune to your volume)
SELECT count(*) FROM jobs
WHERE queue_name = 'notifications-dlq'
  AND created_at > now() - interval '1 hour';

Worker concurrency and idempotency

With multiple worker instances in parallel, two workers can pick up the same job almost at once, during failover, restart, or a queue locking bug. The job runs twice.

The fix isn’t lowering concurrency, it’s making jobs idempotent: processing the same job twice must have the same effect as processing it once.

async function processJob(job: Job) {
  const jobId = job.id;

  const alreadyProcessed = await db.transaction(async (trx) => {
    const existing = await trx('processed_jobs')
      .where({ job_id: jobId })
      .first();

    if (existing) return true;

    await trx('processed_jobs').insert({
      job_id: jobId,
      processed_at: new Date(),
    });

    return false;
  });

  if (alreadyProcessed) {
    return; // expected, not an error
  }

  await doActualWork(job.data);
}

Same pattern as webhook idempotency in the Stripe webhooks article, applies identically to job queues.

Graceful shutdown

When a worker stops, deploy, restart, scale-down, in-flight jobs can be cut mid-run. Graceful shutdown waits for current jobs to finish before exiting.

const worker = new Worker('notifications', processor, {
  connection: redisConnection,
});

process.on('SIGTERM', async () => {
  await worker.close();
  process.exit(0);
});

Shutdown timeout must be shorter than the orchestrator’s kill window (Kubernetes defaults to ~30s before SIGKILL). If a job outlasts that, it still gets killed, long jobs need resumable design or smaller chunks.

Minimum monitoring

A queue without monitoring is a black box. Metrics that matter regardless of stack:

• Queue depth: pending jobs. Steady growth means workers can’t keep up.
• Failure rate by job type: not just “how many failed” but “which jobs fail”. One hot type usually means implementation bug, not infra.
• DLQ volume per hour: alert on threshold, values depend on domain and normal volume.
• Processing latency: enqueue-to-complete time. Sudden spikes mean stressed workers or slower jobs.

What this article doesn’t cover

Multi-datacenter queues, exactly-once guarantees (significant coordination overhead), and large-scale fan-out are real but show up at volumes that deserve their own discussion. For most systems growing from MVP to production, the patterns here are enough without extra infrastructure beyond Redis or Postgres.

Operational summary

• Exponential backoff retry, not immediate retry, growing delay protects dependents under pressure.
• DLQ isn’t a trash bin: inspectable archive that needs active monitoring and a replay process.
• Jobs must be idempotent, double processing is normal, not exceptional, in distributed systems.
• Graceful shutdown avoids mid-run cuts: timeout must match max job runtime.
• Monitor queue depth, failure rate by type, DLQ volume, and latency, without those four, the queue is a black box.

Tell us your context, constraints, and goals: we’ll say whether working together makes sense and how to set up a first step.

Email hello@snowinch.com