DROP/OUT
RECOVERY LAB
SYNTHETIC PROOF · NO EXTERNAL CALLS
DOWNLOAD BLUEPRINT

GHL → n8n RELIABILITY SYSTEM / APPLICATION PROOF

Make the missing
3% impossible
to ignore.

A deterministic recovery pattern for stage-change webhooks: receive fast, persist first, process idempotently, retry safely, and reconcile against the source of truth.

100 synthetic stage changes · 3 intentionally dropped · zero client data

SIMULATION 01
00:00:00.000
97%RECONCILED
EXPECTED100
RECEIVED97
MISSING3
DUPLICATES2
DRIFT DETECTEDReconciliation required

01 / DURABLE PATH

Persist before processing.

The webhook request is acknowledged only after the event is durably captured. Business logic happens downstream, where retries cannot lose the original payload.

01SOURCE

GHL

Opportunity stage change

event_84f2
HTTPS
02INGRESS

Webhook

Validate + normalize

respond < 500ms
WRITE
03DURABILITY

Event ledger

Raw payload + idempotency key

persisted first
QUEUE
04PROCESS

n8n worker

Route + enrich + sync

safe to retry
UPSERT
05DESTINATIONS

Airtable + Slack

State + notification

observable result

02 / EVENT LEDGER

Every event gets a receipt.

3 MISSING
EVENTGHL STAGEINGRESSPROCESSORRESULT
Showing diagnostic sampleCorrelation key: opportunity_id + stage + occurred_at

03 / FAILURE INSPECTOR

Find the layer, not a scapegoat.

FAULT / 01

No delivery receipt

The source-of-truth audit shows the stage change, but the ingress ledger has no matching correlation key.

DETECT
Scheduled GHL reconciliation
RECOVER
Fetch current opportunity state and enqueue a synthetic replay event
PROTECT
Idempotent upsert prevents double-processing

04 / RECONCILIATION

Webhooks for speed. Polling for truth.

Neither transport is enough alone. The recovery cycle compares recent GHL transitions with the local ledger and replays only the missing state changes.

00:00GHL AUDIT

100 authoritative transitions

00:01LEDGER DIFF

3 correlation keys absent

00:02SAFE REPLAY

3 recovery events queued

00:03VERIFIED

100 states reconciled

01

Fast acknowledgement

Validate, persist, respond. Keep enrichment and API calls out of the request path.

02

Idempotency everywhere

A stable event key turns retries and duplicates into safe no-ops.

03

Bounded retries + DLQ

Exponential retry for transient faults; dead-letter queue and alert for terminal failures.

04

Source reconciliation

Scheduled comparisons close the gap that webhooks alone cannot prove is zero.

05 / PRODUCTION HANDOFF

Build the recovery contract
before the happy path.

  1. 01
    Instrument

    Correlation IDs, payload hashes, timestamps, stage version, processing receipt.

  2. 02
    Pressure test

    Timeouts, duplicates, out-of-order updates, rate limits, malformed payloads.

  3. 03
    Operate

    Failure budget, queue depth, replay count, reconciliation drift, alert ownership.