Azure Service Bus (3) — dead-letter queue and retry policies

The third part of the series about Azure Service Bus. We have functional producers and consumers. Now we answer the inevitable question: what happens when processing fails?

The lifecycle of a failed message

Recap from part 2: with AutoCompleteMessages = false, a message is removed from the queue only when you call CompleteMessageAsync. If the handler throws an exception, the message is abandoned and returns to the queue for redelivery.

Each redelivery increments DeliveryCount. When it exceeds MaxDeliveryCount (default: 10), Service Bus automatically moves the message to the dead-letter queue (DLQ) — a sub-queue attached to each queue and each subscription.

Message --> processing fails --> abandon --> returns to queue (DeliveryCount+1)
      --> ... up to MaxDeliveryCount times ...
      --> automatically moved to DLQ (orders/$DeadLetterQueue)

The DLQ is the safety net: no message is lost, but no poison message blocks the processing of others indefinitely.

# Configure MaxDeliveryCount at creation
az servicebus queue create \
  --name order-processing \
  --namespace-name my-servicebus \
  --resource-group my-rg \
  --max-delivery-count 5

Transient vs. permanent errors

The key to smart retry is error classification. Not all failures deserve a retry:

• Transient — database timeout, temporarily unavailable downstream service, throttling (429). Retry makes sense: after 30 seconds it will probably work.
• Permanent — malformed message, failed business validation, non-existent entity. Retry is useless: the same message will fail identically every time. Send it directly to the DLQ.

private async Task OnMessageAsync(ProcessMessageEventArgs args)
{
    using var scope = _scopeFactory.CreateScope();
    var handler = scope.ServiceProvider.GetRequiredService<IOrderHandler>();

    try
    {
        var evt = JsonConvert.DeserializeObject<OrderPlacedEvent>(
            args.Message.Body.ToString());

        if (evt is null)
        {
            // PERMANENT error: malformed message -- direct to DLQ, no retry
            await args.DeadLetterMessageAsync(args.Message,
                deadLetterReason: "DeserializationFailed",
                deadLetterErrorDescription: "Body is not a valid OrderPlacedEvent");
            return;
        }

        await handler.HandleAsync(evt, args.CancellationToken);
        await args.CompleteMessageAsync(args.Message);
    }
    catch (BusinessValidationException ex)
    {
        // PERMANENT error: business rule violated -- DLQ with context
        await args.DeadLetterMessageAsync(args.Message,
            deadLetterReason: "BusinessValidationFailed",
            deadLetterErrorDescription: ex.Message);
    }
    catch (Exception)
    {
        // TRANSIENT (or unknown) error: abandon -- returns for retry
        // After MaxDeliveryCount attempts, automatically goes to DLQ
        await args.AbandonMessageAsync(args.Message);
        throw;
    }
}

DeadLetterMessageAsync with explicit deadLetterReason is essential: when you analyze the DLQ a week later, the reason saves you from digging through logs.

Retry with exponential backoff

Simple abandon redelivers the message almost immediately — useful for momentary errors, but counterproductive when the downstream needs time to recover (rapid consecutive retries can worsen the problem).

Service Bus does not have native per-message backoff, but you can elegantly build it with scheduled messages: instead of abandoning, you copy the message with progressively delayed ScheduledEnqueueTime:

private async Task RetryWithBackoffAsync(ProcessMessageEventArgs args)
{
    var retryCount = args.Message.ApplicationProperties.TryGetValue("RetryCount", out var rc)
        ? (int)rc : 0;

    const int maxRetries = 5;

    if (retryCount >= maxRetries)
    {
        await args.DeadLetterMessageAsync(args.Message,
            deadLetterReason: "MaxRetriesExceeded",
            deadLetterErrorDescription: $"Failed after {maxRetries} attempts with backoff");
        return;
    }

    // Exponential backoff: 10s, 20s, 40s, 80s, 160s
    var delay = TimeSpan.FromSeconds(10 * Math.Pow(2, retryCount));

    var retryMessage = new ServiceBusMessage(args.Message.Body)
    {
        ContentType = args.Message.ContentType,
        MessageId = $"{args.Message.MessageId}-retry-{retryCount + 1}",
        Subject = args.Message.Subject,
        CorrelationId = args.Message.CorrelationId,
        ScheduledEnqueueTime = DateTimeOffset.UtcNow.Add(delay)
    };

    foreach (var prop in args.Message.ApplicationProperties)
        retryMessage.ApplicationProperties[prop.Key] = prop.Value;
    retryMessage.ApplicationProperties["RetryCount"] = retryCount + 1;

    await _sender.SendMessageAsync(retryMessage);
    await args.CompleteMessageAsync(args.Message); // original is completed
}

Important note: MessageId must differ per retry if you have duplicate detection enabled — otherwise Service Bus silently drops the scheduled copy.

For retry within a processing (HTTP call to downstream), use Microsoft.Extensions.Http.Resilience — the two retry layers complement each other: resilience handler for micro network failures, scheduled messages for macro processing failures.

Monitoring and reprocessing the DLQ

Reading dead messages

// The DLQ is a sub-queue -- accessed via SubQueue.DeadLetter
var dlqReceiver = client.CreateReceiver("order-processing",
    new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });

// Peek: read without consuming -- for inspection
var messages = await dlqReceiver.PeekMessagesAsync(maxMessages: 50);

foreach (var msg in messages)
{
    Console.WriteLine($"Reason: {msg.DeadLetterReason}");
    Console.WriteLine($"Description: {msg.DeadLetterErrorDescription}");
    Console.WriteLine($"DeliveryCount: {msg.DeliveryCount}");
}

Reprocessing service

After you fix the cause (bug fix, downstream recovered), messages in the DLQ are resent to the main queue:

public async Task<int> ReprocessDeadLettersAsync(
    string queueName, int maxMessages, CancellationToken ct)
{
    var dlqReceiver = _client.CreateReceiver(queueName,
        new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
    var sender = _client.CreateSender(queueName);

    var reprocessed = 0;
    var messages = await dlqReceiver.ReceiveMessagesAsync(
        maxMessages, TimeSpan.FromSeconds(5), ct);

    foreach (var msg in messages)
    {
        var retryMessage = new ServiceBusMessage(msg.Body)
        {
            ContentType = msg.ContentType,
            MessageId = $"{msg.MessageId}-reprocessed",
            Subject = msg.Subject,
            CorrelationId = msg.CorrelationId
        };
        // Reset RetryCount -- start from zero after fix
        foreach (var prop in msg.ApplicationProperties
                     .Where(p => p.Key != "RetryCount"))
            retryMessage.ApplicationProperties[prop.Key] = prop.Value;

        await sender.SendMessageAsync(retryMessage, ct);
        await dlqReceiver.CompleteMessageAsync(msg, ct);
        reprocessed++;
    }

    return reprocessed;
}

Alerts on DLQ depth

# Alert when DLQ exceeds 10 messages
az monitor metrics alert create \
  --name "dlq-depth-alert" \
  --resource-group my-rg \
  --scopes $SERVICEBUS_ID \
  --condition "avg DeadletteredMessages > 10" \
  --window-size 5m \
  --evaluation-frequency 5m \
  --action $ACTION_GROUP_ID

A silently growing DLQ is an invisible failure. Alerting on DeadletteredMessages is mandatory in production — a message in the DLQ is a client whose email was not sent or whose order was not processed.

What’s next

The most subtle problem remains: what happens if you save the order in the database but publishing the message fails? Or vice versa? In part four: Outbox pattern — consistency between database state and published messages.

Questions? Write to me at contact@ludoprogramming.com.