Data & Integration

Architecture

Pick the integration pattern before you pick the middleware

Tool selection is the loudest part of an integration project and the least consequential. The pattern decides whether the thing survives its second year.

Network patch panel with cables routed between portsData & Integration

Integration projects begin with a tool comparison because a tool comparison feels like progress. It has a spreadsheet, it has vendors, it has a decision at the end of it.

The trouble is that every mainstream integration platform can implement every pattern. The tool rarely determines whether the integration works. The pattern does, and the pattern is usually chosen by accident, in the first sprint, by whoever wrote the first callout.

Four questions that pick the pattern

Before any tooling conversation, answer these for each interface. Not for "the integration" as a whole, per interface, because a single programme usually needs three different patterns.

  1. How stale can this data be? Milliseconds, seconds, minutes, overnight. Be honest; most "real time" requirements are "within a minute" requirements with a confident sponsor.
  2. What is the volume, at peak? Not average. The Monday morning batch, the end-of-month load, the Black Friday spike.
  3. Which direction, and who owns the field? For every field, exactly one system is the source of truth. Write it down.
  4. What must happen when the other side is down? Fail the user's save, queue and retry, drop it, or alert someone.

That last question is the one that eliminates most bad designs on the spot.

The six patterns

Request and reply

Salesforce calls out and waits for the answer. The user is waiting too.

Use it when the answer is genuinely needed to continue, a credit check that gates the next step, an address validation, a real-time inventory read. Keep it out of triggers and out of save paths wherever you can.

The cost: your availability becomes the product of every system in the chain. If the ERP is down, your users cannot save. Set aggressive timeouts, and decide in advance what the user sees when it fails.

Fire and forget

Salesforce sends and does not wait. The user's save completes regardless.

This is the right default for most outbound notifications: tell the ERP a record changed, tell the warehouse a case closed. It requires somewhere for failures to land, because nobody is watching in real time.

The cost: you need a retry story and a dead-letter destination. Fire-and-forget without those is fire-and-lose.

Batch

Scheduled bulk movement, usually overnight, usually large.

Wildly unfashionable and frequently correct. If finance reconciles once a day, a nightly batch is simpler, cheaper and easier to reason about than an event stream nobody can replay.

The cost: latency, obviously, plus a batch window that shrinks as data grows. Watch that window; batch jobs fail quietly until they fail loudly.

Event driven

The source publishes an event; interested consumers subscribe. On the platform this is Platform Events for events you define, and Change Data Capture for record-level changes you did not have to model.

Excellent when several consumers care about the same change, or when the consumer list will grow. It decouples producer from consumer, which is the whole point.

The cost: eventual consistency, replay and ordering semantics you must actually understand, and a debugging story that is harder than "read the log". Subscribers can miss events; you need to know your replay window and design for it.

Remote call-in

An external system calls Salesforce. REST, SOAP, or a Bulk API job for volume.

Straightforward, and the pattern most likely to be built without governance. Every inbound integration needs a named integration user, a scoped permission set, and API limit awareness. Orgs discover the org-wide API limit the day someone points a badly written loop at it.

Data virtualisation

Do not copy the data at all; read it in place. Salesforce Connect and external objects are the platform expression of this.

The right answer more often than teams expect, particularly for large reference datasets that are read occasionally and never edited in Salesforce. Nothing to sync means nothing to reconcile.

The cost: query performance depends on the remote system, and the feature set on external objects is narrower than on standard objects. Validate reporting and search needs before committing.

Quick reference

RequirementPattern
User cannot proceed without the answerRequest and reply
Downstream should know, user should not waitFire and forget
Reconcile large volumes on a scheduleBatch
Several consumers care about the same changeEvent driven
External system initiatesRemote call-in
Large reference data, read rarely, never edited hereData virtualisation

The three rules that outlive the pattern

One owner per field. Bidirectional sync where both sides can write the same field without a documented precedence rule is not an integration, it is a race condition with a schedule. Write the ownership matrix. Publish it. It is the most useful artefact the project will produce.

Idempotency everywhere. Every retry-capable pattern will retry. Every consumer must handle the same message twice without creating a duplicate or double-counting a value. External IDs and upsert are your friends here, use them rather than query-then-insert, which is both slower and racier.

Observability before go-live. You need to be able to answer, without a developer: did this record sync, when, and if not, why. If the only way to answer that is a debug log, support will escalate every question to engineering forever.

Then choose the tool

With patterns settled, tool selection gets much easier and much less contentious, because you are now comparing platforms against a concrete list: does it support these six interfaces, at this volume, with this retry and observability story, and can our team operate it.

That is a two-day evaluation instead of a two-month debate, and it is the right order to do it in.

More on how we approach this: MuleSoft and integration services.

Common questions

Answered, directly.

The questions this piece settles about Data & Integration, answered in full on this page.

Request and reply, fire and forget, batch, event driven, remote call-in, and data virtualisation. Each trades latency against coupling and failure behaviour, and a single programme usually needs three of them across different interfaces.

When several consumers care about the same change, or the consumer list will grow. Events decouple producer from consumer, at the cost of eventual consistency and replay semantics you must design for. Consumers have to be idempotent, because replays and retries are normal operation.

Without one, every sync becomes a matching problem solved by name or email, which produces duplicates at a slow steady rate nobody traces back to the original decision. With one, integrations use upsert, which is idempotent, faster and far simpler to reason about.

Free architect conversation

Talk to an architect, not a sales rep.

Free integration audit. 60 seconds to brief us, and a certified architect replies within one business day.

What are you trying to connect?

Pick the closest fit. The audit is free, and "you do not need middleware" is an answer we give often.

What is being connected?

Optional. Choose any that apply, or skip ahead.

Where does your org stand today?

Optional. A few sentences is plenty: what is working, what is stuck, and what you want to be true. Or skip ahead and tell us on the call.

Who should the architect reach?

A certified architect will reply to these details.

Takes about 30–60 seconds · No obligation · Architect replies within one business day

Protected by reCAPTCHA. Google's Privacy Policy and Terms apply.