Webhooks are how our customers' systems learn that something happened: a label was created, a parcel moved between hubs, a delivery completed, an exception occurred. When they work, they're invisible — a customer's ERP simply stays in sync with reality. When they fail, automations break silently, orders get stuck, and nobody notices until a human does. That asymmetry is why webhook reliability isn't a nice-to-have; it's a core promise.
Why "just send an HTTP request" isn't enough
The naive version of webhooks is a single line of code: when something happens, POST it to the customer's URL. It works perfectly in a demo and fails constantly in production, because the real world violates every assumption that one line makes.
- The receiver is temporarily down, and the event is lost
- The network times out after the receiver already processed the event, so a retry duplicates it
- Two events arrive out of order, so "delivered" lands before "in transit"
- The receiver is slow, and sending blocks everything behind it
Each of these is common at scale, and each corrupts a customer's data in a way that's hard to detect. A trustworthy webhook system is really a set of defences against these four failures.
The durable outbox
The foundation is to never lose an event, even if the receiver is offline. We do that with a durable outbox: when something happens, we first write the event to our own storage, inside the same transaction that recorded the underlying change. Only then does a separate delivery worker attempt to send it.
This decoupling is the key insight. The event's existence never depends on the receiver being available. If the receiver is down, the event waits safely in the outbox until it can be delivered.
Retries that don't give up too soon or hammer too hard
When a delivery fails, we retry with exponential backoff — waiting a little after the first failure, longer after the second, and so on — for up to 24 hours. That window is generous enough to ride out most outages on the receiver's side without either giving up prematurely or hammering a struggling endpoint into the ground.
A receiver that comes back online after a two-hour outage finds its missed events waiting and arriving in an orderly stream, not a thundering herd.
Signing, idempotency, and ordering
Reliability on the wire isn't enough; the receiver needs to trust and correctly interpret what arrives. Three properties make that possible:
- Signing — every payload is cryptographically signed, so the receiver can verify it genuinely came from us and wasn't tampered with
- Idempotency — every event carries a unique id, so a receiver can safely ignore a duplicate caused by a retry
- Ordering — every event carries a monotonic sequence number, so a receiver can detect and reorder events that arrive out of sequence
Together these let a receiver treat our stream as authoritative even when the network does its worst.
Make it easy on their side, too
All of this reliability on our side only helps if it's easy to consume on theirs. So we publish the exact signing scheme, ship idempotency keys the receiver can rely on, and expose a replay endpoint. That replay endpoint matters: a customer recovering from their own downtime can ask us to re-send a window of events and catch up completely, without opening a support ticket.
Good webhook design assumes the receiver will occasionally have problems too, and gives them the tools to recover on their own.
Observability
Finally, we treat webhook delivery as a first-class thing to observe. Customers can see delivery attempts, failures, and retries for their endpoints. When something's wrong, the answer to "did you send it?" is a clear, auditable "yes, here are the six attempts and the responses" — not a shrug.
Evolving without breaking anyone
A webhook system isn't finished the day it ships; it has to evolve as new event types appear and payloads gain fields — and it has to do so without breaking the integrations customers built months ago. Backward compatibility is a promise you make the first time someone consumes your events, and honouring it is as much a part of reliability as delivery itself.
The guiding principle is additive change. New fields can be added to a payload freely, because a well-written consumer ignores fields it doesn't recognise. What you must never do is rename or remove an existing field, or change the meaning of one, without warning — those are the changes that silently break receivers who were parsing exactly what you documented. When a genuinely breaking change is unavoidable, it belongs behind a version, so existing consumers keep receiving the shape they were built against while new ones can opt into the change.
- Add fields, don't rename or remove them — additive change is safe
- Version the payload when a breaking change is truly unavoidable
- Introduce new event types as opt-in, so existing subscriptions are undisturbed
- Document every event and field, and treat that documentation as a contract
There's a human side to this discipline, too. Customers integrate once and then, ideally, forget about it — their systems just stay in sync. Every time you break that quietly, you erode the trust that made webhooks worth offering, and you generate exactly the kind of confusing, hard-to-diagnose failure that damages a relationship. So changes are communicated ahead of time, deprecations come with generous timelines, and nothing that a customer depends on disappears without notice.
Treating your event schema as a public contract, evolved additively and versioned carefully, is what lets the system grow for years without a trail of broken integrations behind it. It's slower than changing things freely, and it's the difference between webhooks customers rely on and webhooks they've learned to distrust.
The takeaway
A webhook system couriers can trust is built from durable storage, patient retries, signed and sequenced payloads, self-service replay, and real observability. None of it shows up in a demo, because a demo never has an outage. All of it shows up the first time production does — which is exactly when a customer's business is depending on the events arriving anyway.
