# Building a Salesforce Integration Architecture Designed for Production Scale > Three systems, one customer record, and three integrations that each worked perfectly on the day they were built. The design work was about the days after. - Source: https://synconai.com/case-studies/integration-architecture-for-production-scale - Publisher: SynconAI (https://synconai.com) - Type: Case study - Evidence: Illustrative scenario. This describes how SynconAI approaches the problem. It is not an account of one named customer, and no outcome is claimed as measured. - Organisation: A subscription media business with a billing platform, an entitlement service and a support desk that all needed the same customer record - Industry: Communications, Media & Technology - Products: Sales Cloud, Service Cloud, Platform Events, Change Data Capture, REST API - Service: Salesforce Integration Services (https://synconai.com/salesforce-integration-services) - Reading time: 12 minutes ## In short A subscription business with billing, entitlement and support systems writing to one Salesforce customer record chose an API family per interface, moved polling onto Change Data Capture and Platform Events, made every write idempotent through an external ID and upsert, classified which errors may be retried, and set all-or-none per interface as a business decision rather than a default. ## Key points - Choose the API family from peak volume and who is waiting for the answer. The wrong choice works in testing and only shows itself on the busiest day of the year. - Authentication is an operational concern. A token that expires overnight fails differently from one that was configured wrongly, and only one of those is found by a test. - An external ID with an upsert is what makes a retry safe. Without it, every retry and every replay is another chance to create a second record for one customer. - All-or-none inside a batch is a business decision. Ask what should happen to the good rows when one row is bad, then write the answer into the interface contract. ## Three systems into one customer record, and what catches the failures Three source systems write to the same Salesforce customer record. Every call passes through one authentication and gateway boundary rather than each system holding its own credentials, so token renewal, secret rotation and rate limiting are solved once. Each interface then uses the API family its volume and latency actually require: bulk loading for the nightly billing extract, event subscription for entitlement changes, and a synchronous call for the support desk lookup a person is waiting on. Writes land on Account, Contact, Asset and Case through external ID fields, so sending the same message twice produces one record rather than two. Anything that fails leaves the path into a failure record carrying the source identifier and the payload, and a scheduled reconciliation compares both sides so a message lost in silence is found by the system rather than by a customer. ### 1. Source systems - Billing platform - Entitlement service - Support desk - Self-service web Four producers, each with its own identifier for the same customer and its own idea of when a change matters. ### 2. Authentication and gateway - JWT bearer - Client credentials - Token cache - Client-side rate limiter One boundary owns credentials, renewal and throttling. No source system holds a Salesforce secret of its own. ### 3. Pattern per interface - Bulk API 2.0 load - Platform Events - Change Data Capture - REST API call - Composite request Chosen from volume, latency and who is waiting, not from what the previous project used. ### 4. Salesforce writes - Account - Contact - Asset - Case - External ID upsert Every write is an upsert on a unique external ID, so a replay is a repeat rather than a duplicate. ### 5. Failure and reconciliation - Integration Failure record - Retryable flag - Replay action - Scheduled reconciliation Failures become records a person can correct and replay. Reconciliation finds what failed without raising an error at all. ## The estate, as it actually presented The brief that arrives is almost never "design our integration architecture". It is "the billing feed broke on Saturday and nobody noticed until Tuesday". Trace that back and the same shape appears. A subscription business runs a billing platform that owns the paid relationship, an entitlement service that decides what a customer is allowed to use, and a support desk that answers when any of it goes wrong. Three systems, one customer, and three different opinions about who that customer is. Salesforce was brought in to hold the commercial and service view, so each of the three was pointed at it, in turn, by whoever was available in that quarter. Each integration worked. Every one of them was demonstrated successfully on the day it was built, because a demonstration has a fresh token, an unlocked record, a test file with a dozen rows in it and a network that is behaving. The days after are a different environment. The billing extract runs nightly and writes record by record, which is unremarkable in a quiet month and still running at mid-morning in the renewal month. The entitlement service polls on a timer, because nobody knew the platform could announce its own changes, so it spends an org-wide allowance asking whether anything happened. The support desk queries for a matching contact and inserts one if it finds nothing, so the day somebody replays a queue to test a change, finance finds two of everything. And the certificate behind the billing feed expires on a date that lives in nobody's calendar. ::: warn The tell for this estate is not an error rate. It is that one named engineer can say which integration failed by looking at which report is wrong. That knowledge is load-bearing, it is nowhere in writing, and it leaves when they do. ::: So the work was not to rebuild three interfaces that already ran. It was to decide, per interface and in writing, what each one does under volume, under failure and under replay. ## What we designed, and why ### The API family is chosen per interface, not per estate The platform offers several API families and at small scale most of them can be made to produce the same result, which is exactly the trap. A row-at-a-time synchronous approach passes every test anybody writes, so it ships, and the wrong choice only becomes visible under volume on a day nobody scheduled. The nightly billing extract moves a large set with nobody waiting for it, so it belongs on Bulk API 2.0: asynchronous, records processed independently, results returned as separate successful and failed sets. That last property is the one teams forget. A bulk job that finishes is not a bulk job that worked, and somebody has to read the failed set. The entitlement change is a single fact that Salesforce needs to know about promptly, and previously it was discovered by polling. Change Data Capture and Platform Events invert that: the platform announces the change instead of being asked. Replacing polling was the single largest reduction in consumption in this estate, and none of it was work anybody wanted done. The support desk lookup has an agent waiting on a screen, so it stays synchronous on REST API, with related writes collapsed into a Composite request rather than chained round trips. Three questions decide every one of these: what is the peak volume, is anybody waiting, and what happens to the rest of the set when one record fails. The specific sizing, concurrency behaviour and allowances differ by edition and by release, so the only correct source is the current Salesforce developer documentation checked against the org itself, rather than a figure somebody carried in from a previous project. ### Authentication designed for how it fails Most documentation treats OAuth as a setup step: register a connected app, pick a flow, exchange a token, done. That framing is why authentication is the most common cause of an integration failing overnight in an otherwise healthy estate. The flow determines the failure mode, and the failure mode is what you actually operate. **JWT bearer** suits the unattended billing feed, and it fails in two specific ways. The signing certificate expires, and nothing in the delivery process reminds anybody, because the person who uploaded it has moved on. Or the integration user is deactivated, loses a permission set, or is caught by a login restriction added for an unrelated reason, at which point every call fails identically and the error does not explain itself. **Client credentials** moves the failure surface onto a secret and the user the connected app runs as, which turns rotation into a scheduled task rather than an emergency, provided somebody schedules it. **Web server flow** is for acting on behalf of a person, and its refresh token is revocable by a password reset or a policy change. An unattended process depending on a token a human obtained interactively is a design defect, not an operational one. Three rules followed for every interface regardless of flow. Cache the access token and reuse it until it fails, because authenticating per call is pure waste against a shared allowance. Guard the refresh so a fleet of workers finding an expired token at the same moment does not stampede the token endpoint. And treat an invalid session as routine: reauthenticate once, replay once, then escalate. A token expiring at two in the morning must be a non-event, because an integration that pages somebody for it teaches the team to ignore the page. ### Idempotency, which everything else depends on This is the decision that separates a retryable integration from one that duplicates records, and every other technique here assumes it. Each integrated object carries an external ID field holding the identifier from the source system, marked unique, and every write is an upsert against that field. Send the same message five times and the result is one record updated five times. Send it once and the result is the same record. That property is what makes a retry safe and a replay a recovery tool rather than a risk. What the estate had instead was query-then-insert, which is a race condition with a friendly interface. Two workers processing the same message both find nothing and both insert, and it costs twice the calls to do it. Two extensions matter in practice. Child records need their own external ID rather than a lookup resolved by name, or the parent stays idempotent while the lines duplicate underneath it. And anything with a financial or notification side effect needs an idempotency key the receiving side recognises. Writing a payment record twice is survivable. Sending the payment twice is not. ### Retries, backoff, and the errors that must never be retried Retry logic written without classification is a denial of service you built and aimed at yourself. The call fails, is retried immediately, fails again for the same deterministic reason, and consumes the shared allowance faster than the original traffic ever did. A transient failure is one where the same call, sent later, could plausibly succeed: a row lock, a request limit condition, a gateway or server-side error, a connection reset, an expired session. A deterministic failure returns the same answer however many times you send it: a validation rule, a missing required field, an insufficient access error, a malformed query, a reference to a deleted record. Only the first class is retried, with exponential backoff and jitter rather than a fixed interval that synchronises workers into waves, and with both the attempts and the elapsed time capped. Row locks earned their own treatment because they were the most common transient failure here and the most commonly mishandled. Backoff helps, but the real fix was upstream: group records by parent so concurrent workers stop contending on the same rollup. The one that is routinely misclassified is the connection reset, where the outcome is unknown and the write may or may not have landed. That case is only recoverable because of the previous decision. With an upsert on an external ID you simply send it again. Without one, somebody has to go and look. ### Partial failure is a business decision When a set is sent together and some records fail, something has to decide whether the successes stand. That decision is usually taken by whoever wrote the code, on the basis of whatever the default happened to be, and it is not a technical decision at all. All-or-none says the batch is a unit, and it is correct wherever the records have referential meaning to each other. A subscription written without its entitlement lines is worse than nothing at all, because the first is a data incident somebody will act on and the second is an obvious gap. Independent processing says each record stands on its own, and it is correct for high-volume loosely related records. Rejecting an entire preference sync because one row has a malformed postcode is an outage you chose. We asked the business, per interface, and wrote the answer into the interface contract. Then we tested it, because the difference only appears in a failure test, and failure tests are the ones that get skipped. ## Error handling that produces a record, not a log line An error written to a log is not error handling. It is a record of an event nobody will read, in a place nobody will look, until an incident forces a search and the retention window has already closed. What replaced it is a failure record with enough on it to act: the business identifier from the source system rather than only a platform record id, because the person resolving it works in the source; the operation attempted and the interface it belongs to; the payload or a durable reference to it, so the work can be replayed without being reconstructed by hand; the error exactly as returned; the attempt count; a retryable flag set by the classification above; and a status somebody can move. That last field is what makes the difference. A failure queue that can only be read is a graveyard. One where a person corrects the data, marks the item for replay and watches it succeed is an operational tool, and it is what stops every integration exception becoming an engineering ticket. Failures are grouped by error class before anybody is alerted, because forty rows failing one validation rule is one problem with one fix, and forty separate alerts is how a team learns to ignore alerts. Alongside it runs a scheduled reconciliation comparing counts and key fields on both sides. Error handling catches what fails loudly. Reconciliation catches what fails silently, which is the category that produces the Tuesday morning conversation. ## Version pinning, and the discipline of moving Every client calls a specific API version and pinning one takes a minute. Salesforce keeps older versions available for a long published window and then retires them, which is generous, and which is precisely why the consequence lands years later when everybody involved has moved on. The failure is never the pin. It is that nothing causes the pin to move. So the mechanism was built rather than intended: the version held in one configuration value per client instead of a literal repeated through a codebase, a register recording which interface uses which version so the retirement question is answered in minutes rather than weeks, and a version review attached to the platform release cycle, which arrives on a schedule whether anybody plans for it or not. Then the move is tested against the failure cases, not the happy path. A version change removes fields and endpoints, but it also changes what a response contains and how an error is shaped, and a success-path test will not surface either. ## Implementation The sequence mattered more than the components. **External IDs and the data contract first.** Which system is the source of truth for each field, and what identifier each carries. Nothing else is safe to build before this is settled, and it is close to irreversible once the org holds real data. **Then the authentication boundary**, so credentials, renewal, throttling and the token cache exist in one place before three clients each solve them differently. **Then one interface end to end**, chosen as the billing extract because it had the clearest volume story and the most obvious failure. Proving the pattern once, including the failure path and the replay, is what made the remaining interfaces cheap. **Then the failure record and reconciliation**, before the remaining interfaces were migrated rather than after, so every new interface arrived with somewhere for its failures to go. ::: note We deliberately did not automate replay for the interfaces with financial side effects. An automatic replay of a payment-adjacent message is a class of error that is far more expensive than a person pressing a button, and some things should stay manual on purpose. ::: ## The decisions that were contested **Replacing polling with events.** The entitlement team preferred polling because it was working and they understood it. The argument that landed was not elegance. It was that the polling was spending an org-wide allowance shared with every other integration, report refresh and mobile client, and that the cost was invisible precisely because it was spread across everybody else. **Not putting every interface behind middleware.** There was appetite for a single uniform layer on principle. We argued that the layer earns its place through shared concerns, credentials, rate limiting, one failure queue, and that a low-volume interface with one consumer does not become better by acquiring a hop. Uniformity is not an outcome. **Keeping all-or-none on the subscription load.** The engineering preference was independent processing everywhere, because it is simpler to operate. The business chose all-or-none for the interface where a header without its lines becomes a customer-facing error, and that was the correct call even though it makes reprocessing more work. **Building the failure record before migrating the rest.** It looked like a delay against interfaces that were already running. It was the decision that made the following ones fast, because no interface needed to invent its own error path. ## What changes The outcomes worth claiming from this design are operational, and they follow from the decisions rather than from effort. | Before | After | What made the difference | | --- | --- | --- | | One API approach used everywhere | An API family chosen per interface | Volume, latency and partial-failure behaviour asked per interface | | Entitlement changes discovered by polling | The platform announces its own changes | Change Data Capture and Platform Events replacing a timer | | Query then insert, duplicates after a replay | Upsert on a unique external ID | Idempotency made a property of the write, not of the caller | | Every error retried the same way | Transient and deterministic errors classified | A written classification the client reads, not a per-engineer opinion | | Batch behaviour inherited from a default | All-or-none decided and recorded per interface | The business asked what should happen to the good rows | | Failures written to a log nobody reads | A failure record with an owner and a replay action | The source identifier and payload captured, and a status somebody can move | | Silent losses found by a customer | Silent losses found by scheduled reconciliation | Comparing both sides rather than waiting for an error | The second-order effect is the one leaders notice. The conversation changes from "is the integration up" to "what is in the failure queue this morning", which is a question with an answer and an owner. ## What we would tell you before starting Settle the external IDs and the data contract before anybody writes a callout. Almost every integration of this shape that goes badly goes badly there, and the cost of changing it later is measured in reconciliation work rather than in code. Then be honest about scale. If you have one interface, one consumer and volume that fits comfortably in a synchronous call, most of this is over-engineering and a well-written point-to-point integration is the right answer. The design above earns its cost when several systems need the same record, when volume peaks somewhere other than where you tested, and when the people who built the original interfaces are no longer the people running them. ## Questions ### Should every interface go through middleware, or can some call Salesforce directly? Both are defensible, and the test is whether anything needs to be shared. A single point-to-point interface with one consumer rarely earns a middleware hop. Once three systems need the same customer view, the shared concerns are what justify the layer: one place holding credentials, one rate limiter protecting an org-wide allowance, one failure queue instead of three. We route through a boundary for the shared concerns, not for the sake of a diagram. ### How do you know which Salesforce API family an interface should use? Three questions settle it. What is the peak volume, not the average. Is anybody waiting for the answer on a screen. And what should happen to the rest of the set when one record fails. High volume with nobody waiting points at Bulk API 2.0. A person waiting points at a synchronous REST or Composite call. A change the platform can announce points at Change Data Capture or Platform Events rather than polling. Confirm current sizing and concurrency behaviour in the Salesforce developer documentation for your edition before the design is fixed. ### Is version pinning enough to protect us from Salesforce releases? Pinning protects you from the next release and exposes you to the retirement. Salesforce publishes long support windows for older API versions, which is generous and is exactly why the problem arrives years later when the original team has moved on. The pin is a minute of work. The mechanism that moves it, a single configuration value per client, a register of which interface uses which version, and a review attached to the release cycle, is the part that has to be built deliberately. --- SynconAI. https://synconai.com/case-studies/integration-architecture-for-production-scale