Reliability & Operations

Webhook Replay, DLQ, and Reliable Delivery

At-least-once delivery means some webhooks may arrive twice. Learn why idempotency on the receiver is non-negotiable, how exponential backoff works, when a delivery ends up in the DLQ, and how to replay safely without duplicating side effects.

At-least-once delivery: the promise and the reality

When a webhook delivery fails — a timeout, a 5xx error, a network blip — the platform retries. The guarantee is that your endpoint will receive the event at least one time. Not at most once, and not exactly once. At least once.

What it protects against

You will not lose an event because your endpoint was temporarily unavailable or slow. If a delivery attempt hangs, times out, or fails with a 5xx error, Centrali retries with exponential backoff until the event succeeds or the retry window expires.

What it doesn't protect against

Your endpoint may receive the same event twice. Your code must be idempotent — if the same webhook arrives twice (because of a retry after your endpoint processed it, or a manual replay), you must not create two records or charge twice.

Why idempotency matters on your receiver

The same webhook will arrive at least once. Your handler must recognize when it has already processed an event and return success without repeating the side effect. Here's the pattern:

javascript
// Node.js / Express handler
app.post('/webhooks/centrali', async (req, res) => {
  const event = req.body;


  // 1. Use the event ID as an idempotency key
  const existing = await db.query(
    'SELECT id FROM events_processed WHERE event_id = ?',
    [event.recordId] // Centrali includes recordId in the payload
  );


  if (existing.length > 0) {
    // Already processed — return success
    return res.status(200).json({ ok: true, duplicate: true });
  }


  // 2. Process the event
  if (event.event === 'record_created') {
    await db.query(
      'INSERT INTO user_records (id, data) VALUES (?, ?)',
      [event.recordId, JSON.stringify(event.data)]
    );
  }


  // 3. Mark it processed atomically
  await db.query(
    'INSERT INTO events_processed (event_id, processed_at) VALUES (?, NOW())',
    [event.recordId]
  );


  return res.status(200).json({ ok: true });
});

Why check-then-process isn't enough: Between the time you check for an existing record and the time you insert, another copy of the same webhook may arrive. Use a database unique constraint on the event ID, or wrap both operations in a transaction. Many teams get this wrong and silently create duplicate records on retries.

Exponential backoff: spacing out the retries

Retrying immediately on failure is wasteful. If an endpoint is overloaded, hammering it faster makes it worse. Backoff spreads the retries over time, giving the endpoint a chance to recover.

Centrali's default backoff schedule (5 retries)

130 seconds
T+30s
22 minutes
T+2m 30s
310 minutes
T+12m 30s
430 minutes
T+42m 30s
5terminal failure
DLQ

You can customize this schedule per subscription — set attempt count (1–20), choose exponential or fixed backoff, and set a retry window (up to 7 days). For most use cases, the default 5 retries over ~40 minutes works well.

Design for it anyway: no delivery platform is immune to an outage, ours included. Treat missing webhooks as a normal condition rather than an exception — reconcile against the stored event record when a delivery window looks quiet, and keep your handler idempotent so a catch-up burst is safe.

The dead-letter queue: when retries are exhausted

A delivery lands in the DLQ when all retries are exhausted or the retry window expires. You can inspect, replay, or discard DLQ deliveries — no need to delete the subscription and set it up again.

When does a delivery land in the DLQ?

  • •Max attempts reached: All retries completed and the endpoint keeps returning 4xx or 5xx errors.
  • •Retry window expired: The delivery failed for longer than your retry-window setting (default 7 days). Centrali stops retrying even if attempts remain.

How to inspect and replay DLQ deliveries

Access the DLQ via the REST API or the console. Each delivery shows the reason it dead-lettered, the last error, and the stored payload.

bash
// List dead-lettered deliveries
curl -X GET https://api.centrali.io/data/workspace/{workspace}/api/v1/webhook-subscriptions/deliveries/dlq \
  -H "Authorization: Bearer $CENTRALI_TOKEN"


// Response includes:
// {
//   "data": [
//     {
//       "id": "dlv_...",
//       "dlqReason": "max_attempts",
//       "lastError": "Connection timeout",
//       "requestPayload": { ... },
//       "responseBody": "Error message...",
//       "attemptCount": 5,
//       "retryWindowMs": 604800000
//     }
//   ]
// }


// Replay a DLQ delivery
curl -X POST https://api.centrali.io/data/workspace/{workspace}/api/v1/webhook-subscriptions/deliveries/{id}/retry \
  -H "Authorization: Bearer $CENTRALI_TOKEN"


// This creates a new delivery attempt with the same payload.
// The replay increments the attempt counter fresh, so the retry schedule starts over.

No automatic alerting (yet): Centrali does not send you a notification when a delivery lands in the DLQ. Build a saved query that counts DLQ deliveries per subscription and set up a query alert, or poll the DLQ endpoint on a schedule.

Safe replay patterns

Replay is powerful but requires care. Your endpoint will receive the same payload and signature twice (or more). Here's how to handle it safely.

Approach 1: Idempotent processing (recommended)

Make your handler idempotent using the event ID as a key. This is the right default because it works for both retries and manual replays.

// Upsert using event ID as the idempotency key
async function handleWebhook(event) {
  // The event.recordId is globally unique and stable across retries/replays
  const result = await db.query(
    `INSERT INTO webhooks_received (event_id, event_data, received_at)
     VALUES (?, ?, NOW())
     ON CONFLICT(event_id) DO UPDATE SET
       last_seen_at = NOW()`,
    [event.recordId, JSON.stringify(event)]
  );


  // Trigger downstream work only on first receipt
  if (result.affectedRows === 1) {
    await processNewEvent(event);
  }


  return { ok: true };
}

Approach 2: Verify you fixed it first

If your endpoint has been crashing or timing out, wait until you've deployed a fix before replaying. Replaying into a broken endpoint lands it back in the DLQ. Fix the bug, deploy, verify with a test, then replay.

Approach 3: Discard if you don't need it

For time-sensitive events (notifications, alerts), discarding an old DLQ delivery may be fine. Discard does not delete it — it marks it as intentionally skipped and removed from the DLQ.

// Discard a DLQ delivery (mark as handled without replaying)
curl -X POST https://api.centrali.io/data/workspace/{workspace}/api/v1/webhook-subscriptions/deliveries/{id}/discard \
  -H "Authorization: Bearer $CENTRALI_TOKEN"

Related guides

Signed Webhook Delivery

Understand HMAC signing, secret rotation, and how to verify incoming webhooks.

Verify Webhook Signatures

Cross-provider signature verification patterns and provider-specific schemes.

Webhook Event Storage

Why storing webhook events as queryable records outlasts provider retention windows.

Best Webhook Infrastructure Tools

Compare Centrali, Svix, Hookdeck, Inngest, and roll-your-own approaches.

Compare: Centrali vs Svix

Side-by-side comparison of Centrali and Svix webhook capabilities.

Compare: Centrali vs Hookdeck

See how Centrali stacks up against Hookdeck for webhook delivery.

FAQ

See your webhooks in action

Inspect every delivery, see retry attempts, handle DLQ events, and replay with confidence. Build webhook endpoints that handle retries and duplicates gracefully.

Free tier includes webhook delivery logs and replay