Do you know that Azure Service Bus supports duplicate detection — a feature that automatically prevents the same message from being processed more than once?
📝 One-Minute Brief
Duplicate Detection in Azure Service Bus prevents the same message from being accepted more than once within a defined time window.
It works by comparing each incoming message’s ID against previously received ones.
This ensures reliable, idempotent message processing even when senders retry due to network or system errors.
Duplicate Detection prevents a Service Bus queue or topic from storing more than one message with the same MessageId during a configurable time window. If another message with the same MessageId arrives within that window, the send operation succeeds, but the duplicate is discarded (it never lands in the entity). Available in Standard and Premium tiers.
Why it matters
This is important because it helps you maintain data integrity and idempotent message processing in distributed systems, especially when network retries or transient errors cause the same message to be sent multiple times.
- Idempotence made simple: retries and network timeouts often produce duplicates. With Duplicate Detection, the broker guarantees you won’t store them during the window—no extra consumer logic required.
- Operationally light: you turn it on when creating the entity and then only tune the window to match throughput and risk.
- Business-aligned: set MessageId to a stable business ID so resends are predictably deduped.
How it works
When enabled, Service Bus keeps a recent history of MessageId values for the duration of the configured window. If a new message arrives with a MessageId already seen in that window, the send is acknowledged but the message is dropped.
Partitioning/Sessions:
- On partitioned entities, uniqueness is MessageId + PartitionKey.
- With sessions, set PartitionKey = SessionId so dedupe behaves as expected.
Scheduled messages are deduped the same way.
Step-by-Step
Create the queue with Duplicate Detection
- In the namespace, go to Queues and then add a Queue.
- Check Enable duplicate detection and (for a quick PoC) set the window to 20 seconds.
- Leave Partitioning and Sessions disabled for a no-surprises test.
- Create the entity.
Send messages with the same MessageId (in-window vs out-of-window)
- Open the Queue → Service Bus Explorer→ Send.
- Set Message ID = ORDER-123 and use these two cases:
- Inside the window (deduped): type hello in the Message Body, click Send, then click Send again immediately.
- Outside the window: wait around 25 to 30 seconds, change Message Body to poc=true, for example, and send twice again with the same MessageId, but this time separated by more than 20 seconds.
Peek to verify results
- Go to Peek to inspect active messages.
- What you should see and why it proves the concept:
- “hello” appears only once → duplicates sent inside the 20-second window were discarded.
- “poc=true” appears twice → each send happened after the window expired, so both were stored.
Tips & gotchas
- Pick a good MessageId: use a stable, meaningful ID per business operation. Random IDs defeat dedupe.
- Window sizing: start with 10 minutes in production and adjust. Larger windows provide more protection but reduce peak throughput.
- Partitioning & sessions: if you enable them, remember dedupe keys include PartitionKey (and with sessions you should set PartitionKey = SessionId). For a simple PoC, leave both off.
- Not a toggle: you can only enable Duplicate Detection when creating the entity. Later, you can change the window, but you can’t turn the feature on/off.
- Observability: there’s no “deduplicated count” metric. Validate behavior via Peek and Active messages.
FAQ
- Can I enable it on an existing queue?
- No. Create a new queue/topic with Duplicate Detection enabled and migrate producers/consumers.
- What window values are allowed?
- From 20 seconds to 7 days (default is 10 minutes). You can change the window later.
- Does it work with topics and subscriptions?
- Yes. Deduping happens at the topic level (before messages fan out to subscriptions).
- What does the sender see for duplicates?
- The send call succeeds, and the duplicate is discarded at the broker.
- Which tiers support it?
- Standard and Premium. Not available in Basic.
Take Away
Azure Service Bus’s duplicate detection is a small switch that brings big, practical benefits: it helps keep your data clean, your processes predictable, and your teams confident that what gets processed is what was intended once. By quietly filtering out repeated sends, it reduces the risk of inconsistent records, accidental double actions, and security exposures that can come from handling the same message multiple times. The result is a safer, more reliable pipeline with fewer manual checks, less operational noise, and more time spent on what matters. It’s a simple way to strengthen trust, protect data integrity, and improve overall system resilience.
To lazy to read? We’ve got you covered! Check out our video version of this content!
Hope you find this helpful! If you enjoyed the content or found it useful and wish to support our efforts to create more, you can contribute towards purchasing a Star Wars Lego for Sandro’s son!
