Designing integrations for failure

An integration crosses a boundary you do not control. A payment gateway, messaging service, identity provider, or delivery API can be slow, unavailable, or return a response your system cannot confidently interpret. The goal is not to pretend those failures will not happen. It is to keep your own records honest while you work out what happened outside them.

Persist intent before delivery

Before asking another service to act, save what your application intends to do and the internal record it belongs to. A durable queue or outbox can then deliver that work and preserve it across a process restart. If the process fails after saving the intent but before sending the request, a worker can resume it. If it fails after sending, the record still shows that the outcome needs resolving.

const operation = await store.create({
  id: newId(),
  state: "pending",
  idempotencyKey: stableRequestKey,
});

await outbox.enqueue({ operationId: operation.id });

This is a conceptual example, not code from a particular integration. The important part is that the intent exists before delivery, and that the delivery worker can find it again.

Distinguish failure from uncertainty

A clear validation rejection is different from a timeout after the request left your server. In the first case, the provider has said it did not accept the operation. In the second, the provider may have acted and the response may simply have been lost. Treating both cases as a safe-to-repeat failure is how systems create duplicate work or charges.

• Known rejection: store the response and show what must be corrected before another attempt.
• Temporary service failure: queue a bounded retry if the operation is safe to repeat.
• Unknown outcome: keep the operation pending and query or reconcile it before initiating another one.
• Permanent configuration error: surface it for an operator instead of retrying forever.

Make retries safe by design

Use a stable idempotency key when the provider supports one, and enforce uniqueness in your own records as well. A retry should refer to the original business operation, not create a new one because the browser or worker tried again. Keep retries bounded, add backoff to avoid hammering a struggling provider, and distinguish operations that are safe to repeat from those that need a status check first.

For webhooks, verify the provider's signature before applying a result, look up the operation by a stable reference, and make processing idempotent. Persist the result before acknowledging it. A repeated delivery should return a consistent acknowledgement without applying the business change twice.

Keep the rest of the system moving

Do not hold a user-facing request open while an unreliable provider retries in the background. Record a meaningful state such as pending, failed, or needs review, then let the rest of the application continue according to its own rules. A reservation, booking, or order may need a manual path while a payment is uncertain; that decision belongs in the domain workflow, not in a generic retry loop.

Give unresolved work somewhere to go

Some operations cannot be settled automatically. Keep the provider reference, relevant response, amount or payload needed for comparison, and the reason the system stopped. Give an operator a view of the unresolved state and a documented way to verify it. Reconciliation is part of the integration, not cleanup left for a support inbox.

1. Persist the business operation before delivery.
2. Classify known rejection, temporary outage, and unknown outcome separately.
3. Use stable idempotency keys and bounded retries where repetition is safe.
4. Verify and deduplicate callbacks before applying their effects.
5. Provide an operator path for anything that remains unresolved.

A resilient integration does not promise that every provider call succeeds. It makes the state visible, prevents avoidable duplicates, and gives the business a safe next step when the answer is not yet known.