GHL
Opportunity stage change
event_84f2GHL → n8n RELIABILITY SYSTEM / APPLICATION PROOF
A deterministic recovery pattern for stage-change webhooks: receive fast, persist first, process idempotently, retry safely, and reconcile against the source of truth.
100 synthetic stage changes · 3 intentionally dropped · zero client data
01 / DURABLE PATH
The webhook request is acknowledged only after the event is durably captured. Business logic happens downstream, where retries cannot lose the original payload.
Opportunity stage change
event_84f2Validate + normalize
respond < 500msRaw payload + idempotency key
persisted firstRoute + enrich + sync
safe to retryState + notification
observable result02 / EVENT LEDGER
03 / FAILURE INSPECTOR
The source-of-truth audit shows the stage change, but the ingress ledger has no matching correlation key.
04 / RECONCILIATION
Neither transport is enough alone. The recovery cycle compares recent GHL transitions with the local ledger and replays only the missing state changes.
100 authoritative transitions
3 correlation keys absent
3 recovery events queued
100 states reconciled
Validate, persist, respond. Keep enrichment and API calls out of the request path.
A stable event key turns retries and duplicates into safe no-ops.
Exponential retry for transient faults; dead-letter queue and alert for terminal failures.
Scheduled comparisons close the gap that webhooks alone cannot prove is zero.
05 / PRODUCTION HANDOFF
Correlation IDs, payload hashes, timestamps, stage version, processing receipt.
Timeouts, duplicates, out-of-order updates, rate limits, malformed payloads.
Failure budget, queue depth, replay count, reconciliation drift, alert ownership.