Building a Salesforce Integration Architecture Designed for Production Scale

#The estate, as it actually presented

The brief that arrives is almost never "design our integration architecture". It is "the billing feed broke on Saturday and nobody noticed until Tuesday".

Trace that back and the same shape appears. A subscription business runs a billing platform that owns the paid relationship, an entitlement service that decides what a customer is allowed to use, and a support desk that answers when any of it goes wrong. Three systems, one customer, and three different opinions about who that customer is.

Salesforce was brought in to hold the commercial and service view, so each of the three was pointed at it, in turn, by whoever was available in that quarter. Each integration worked. Every one of them was demonstrated successfully on the day it was built, because a demonstration has a fresh token, an unlocked record, a test file with a dozen rows in it and a network that is behaving.

The days after are a different environment. The billing extract runs nightly and writes record by record, which is unremarkable in a quiet month and still running at mid-morning in the renewal month. The entitlement service polls on a timer, because nobody knew the platform could announce its own changes, so it spends an org-wide allowance asking whether anything happened. The support desk queries for a matching contact and inserts one if it finds nothing, so the day somebody replays a queue to test a change, finance finds two of everything. And the certificate behind the billing feed expires on a date that lives in nobody's calendar.

So the work was not to rebuild three interfaces that already ran. It was to decide, per interface and in writing, what each one does under volume, under failure and under replay.

#What we designed, and why

The API family is chosen per interface, not per estate

The platform offers several API families and at small scale most of them can be made to produce the same result, which is exactly the trap. A row-at-a-time synchronous approach passes every test anybody writes, so it ships, and the wrong choice only becomes visible under volume on a day nobody scheduled.

The nightly billing extract moves a large set with nobody waiting for it, so it belongs on Bulk API 2.0: asynchronous, records processed independently, results returned as separate successful and failed sets. That last property is the one teams forget. A bulk job that finishes is not a bulk job that worked, and somebody has to read the failed set.

The entitlement change is a single fact that Salesforce needs to know about promptly, and previously it was discovered by polling. Change Data Capture and Platform Events invert that: the platform announces the change instead of being asked. Replacing polling was the single largest reduction in consumption in this estate, and none of it was work anybody wanted done.

The support desk lookup has an agent waiting on a screen, so it stays synchronous on REST API, with related writes collapsed into a Composite request rather than chained round trips. Three questions decide every one of these: what is the peak volume, is anybody waiting, and what happens to the rest of the set when one record fails.

The specific sizing, concurrency behaviour and allowances differ by edition and by release, so the only correct source is the current Salesforce developer documentation checked against the org itself, rather than a figure somebody carried in from a previous project.

Authentication designed for how it fails

Most documentation treats OAuth as a setup step: register a connected app, pick a flow, exchange a token, done. That framing is why authentication is the most common cause of an integration failing overnight in an otherwise healthy estate. The flow determines the failure mode, and the failure mode is what you actually operate.

JWT bearer suits the unattended billing feed, and it fails in two specific ways. The signing certificate expires, and nothing in the delivery process reminds anybody, because the person who uploaded it has moved on. Or the integration user is deactivated, loses a permission set, or is caught by a login restriction added for an unrelated reason, at which point every call fails identically and the error does not explain itself.

Client credentials moves the failure surface onto a secret and the user the connected app runs as, which turns rotation into a scheduled task rather than an emergency, provided somebody schedules it. Web server flow is for acting on behalf of a person, and its refresh token is revocable by a password reset or a policy change. An unattended process depending on a token a human obtained interactively is a design defect, not an operational one.

Three rules followed for every interface regardless of flow. Cache the access token and reuse it until it fails, because authenticating per call is pure waste against a shared allowance. Guard the refresh so a fleet of workers finding an expired token at the same moment does not stampede the token endpoint. And treat an invalid session as routine: reauthenticate once, replay once, then escalate. A token expiring at two in the morning must be a non-event, because an integration that pages somebody for it teaches the team to ignore the page.

Idempotency, which everything else depends on

This is the decision that separates a retryable integration from one that duplicates records, and every other technique here assumes it.

Each integrated object carries an external ID field holding the identifier from the source system, marked unique, and every write is an upsert against that field. Send the same message five times and the result is one record updated five times. Send it once and the result is the same record. That property is what makes a retry safe and a replay a recovery tool rather than a risk.

What the estate had instead was query-then-insert, which is a race condition with a friendly interface. Two workers processing the same message both find nothing and both insert, and it costs twice the calls to do it.

Two extensions matter in practice. Child records need their own external ID rather than a lookup resolved by name, or the parent stays idempotent while the lines duplicate underneath it. And anything with a financial or notification side effect needs an idempotency key the receiving side recognises. Writing a payment record twice is survivable. Sending the payment twice is not.

Retries, backoff, and the errors that must never be retried

Retry logic written without classification is a denial of service you built and aimed at yourself. The call fails, is retried immediately, fails again for the same deterministic reason, and consumes the shared allowance faster than the original traffic ever did.

A transient failure is one where the same call, sent later, could plausibly succeed: a row lock, a request limit condition, a gateway or server-side error, a connection reset, an expired session. A deterministic failure returns the same answer however many times you send it: a validation rule, a missing required field, an insufficient access error, a malformed query, a reference to a deleted record. Only the first class is retried, with exponential backoff and jitter rather than a fixed interval that synchronises workers into waves, and with both the attempts and the elapsed time capped.

Row locks earned their own treatment because they were the most common transient failure here and the most commonly mishandled. Backoff helps, but the real fix was upstream: group records by parent so concurrent workers stop contending on the same rollup.

The one that is routinely misclassified is the connection reset, where the outcome is unknown and the write may or may not have landed. That case is only recoverable because of the previous decision. With an upsert on an external ID you simply send it again. Without one, somebody has to go and look.

Partial failure is a business decision

When a set is sent together and some records fail, something has to decide whether the successes stand. That decision is usually taken by whoever wrote the code, on the basis of whatever the default happened to be, and it is not a technical decision at all.

All-or-none says the batch is a unit, and it is correct wherever the records have referential meaning to each other. A subscription written without its entitlement lines is worse than nothing at all, because the first is a data incident somebody will act on and the second is an obvious gap. Independent processing says each record stands on its own, and it is correct for high-volume loosely related records. Rejecting an entire preference sync because one row has a malformed postcode is an outage you chose.

We asked the business, per interface, and wrote the answer into the interface contract. Then we tested it, because the difference only appears in a failure test, and failure tests are the ones that get skipped.

#Error handling that produces a record, not a log line

An error written to a log is not error handling. It is a record of an event nobody will read, in a place nobody will look, until an incident forces a search and the retention window has already closed.

What replaced it is a failure record with enough on it to act: the business identifier from the source system rather than only a platform record id, because the person resolving it works in the source; the operation attempted and the interface it belongs to; the payload or a durable reference to it, so the work can be replayed without being reconstructed by hand; the error exactly as returned; the attempt count; a retryable flag set by the classification above; and a status somebody can move.

That last field is what makes the difference. A failure queue that can only be read is a graveyard. One where a person corrects the data, marks the item for replay and watches it succeed is an operational tool, and it is what stops every integration exception becoming an engineering ticket. Failures are grouped by error class before anybody is alerted, because forty rows failing one validation rule is one problem with one fix, and forty separate alerts is how a team learns to ignore alerts.

Alongside it runs a scheduled reconciliation comparing counts and key fields on both sides. Error handling catches what fails loudly. Reconciliation catches what fails silently, which is the category that produces the Tuesday morning conversation.

#Version pinning, and the discipline of moving

Every client calls a specific API version and pinning one takes a minute. Salesforce keeps older versions available for a long published window and then retires them, which is generous, and which is precisely why the consequence lands years later when everybody involved has moved on.

The failure is never the pin. It is that nothing causes the pin to move. So the mechanism was built rather than intended: the version held in one configuration value per client instead of a literal repeated through a codebase, a register recording which interface uses which version so the retirement question is answered in minutes rather than weeks, and a version review attached to the platform release cycle, which arrives on a schedule whether anybody plans for it or not.

Then the move is tested against the failure cases, not the happy path. A version change removes fields and endpoints, but it also changes what a response contains and how an error is shaped, and a success-path test will not surface either.

#Implementation

The sequence mattered more than the components.

External IDs and the data contract first. Which system is the source of truth for each field, and what identifier each carries. Nothing else is safe to build before this is settled, and it is close to irreversible once the org holds real data.

Then the authentication boundary, so credentials, renewal, throttling and the token cache exist in one place before three clients each solve them differently.

Then one interface end to end, chosen as the billing extract because it had the clearest volume story and the most obvious failure. Proving the pattern once, including the failure path and the replay, is what made the remaining interfaces cheap.

Then the failure record and reconciliation, before the remaining interfaces were migrated rather than after, so every new interface arrived with somewhere for its failures to go.

#The decisions that were contested

Replacing polling with events. The entitlement team preferred polling because it was working and they understood it. The argument that landed was not elegance. It was that the polling was spending an org-wide allowance shared with every other integration, report refresh and mobile client, and that the cost was invisible precisely because it was spread across everybody else.

Not putting every interface behind middleware. There was appetite for a single uniform layer on principle. We argued that the layer earns its place through shared concerns, credentials, rate limiting, one failure queue, and that a low-volume interface with one consumer does not become better by acquiring a hop. Uniformity is not an outcome.

Keeping all-or-none on the subscription load. The engineering preference was independent processing everywhere, because it is simpler to operate. The business chose all-or-none for the interface where a header without its lines becomes a customer-facing error, and that was the correct call even though it makes reprocessing more work.

Building the failure record before migrating the rest. It looked like a delay against interfaces that were already running. It was the decision that made the following ones fast, because no interface needed to invent its own error path.

#What changes

The outcomes worth claiming from this design are operational, and they follow from the decisions rather than from effort.

The second-order effect is the one leaders notice. The conversation changes from "is the integration up" to "what is in the failure queue this morning", which is a question with an answer and an owner.

#What we would tell you before starting

Settle the external IDs and the data contract before anybody writes a callout. Almost every integration of this shape that goes badly goes badly there, and the cost of changing it later is measured in reconciliation work rather than in code.

Then be honest about scale. If you have one interface, one consumer and volume that fits comfortably in a synchronous call, most of this is over-engineering and a well-written point-to-point integration is the right answer. The design above earns its cost when several systems need the same record, when volume peaks somewhere other than where you tested, and when the people who built the original interfaces are no longer the people running them.