Connecting two systems via an API can feel deceptively simple. You read the docs, write a few HTTP calls, map some fields, and data starts flowing. Job done — until it isn't.
The gap between a working demo and a production-grade integration is where most projects run into trouble. Here's what we've learned from building integrations that need to work reliably, day after day, without constant attention.
The demo trap
Most API documentation shows the happy path: well-formed requests, clean responses, predictable data. Real-world APIs are messier. Fields that should be populated are null. Rate limits kick in at inconvenient times. Authentication tokens expire. Endpoints change behaviour between versions.
A good integration isn't just about making the call — it's about handling every way that call can fail, and recovering gracefully when it does.
Design for failure
Every integration should answer these questions before writing a line of code:
- What happens when the remote system is down? Queue and retry? Alert and pause? Fall back to cached data? The right answer depends on the business context.
- What happens when the data is unexpected? A field that's usually a string arrives as null, or an array is empty when it shouldn't be. Your integration should handle this without crashing or corrupting data.
- What happens when rate limits are hit? Respect backoff headers. Implement queuing. Don't hammer an API that's asking you to slow down.
- What happens when credentials expire? Token refresh should be automatic and tested — not something that breaks at 3am on a Saturday.
Logging is not optional
When an integration stops working — and at some point, it will — the first question is always "what happened?" If the answer is "we don't know", you're in trouble.
Good integration logging captures:
- Every request and response (or at least the metadata — status codes, timestamps, record counts)
- Errors with enough context to diagnose without reproducing
- Data transformations — what came in, what was mapped, what went out
- Timing — how long calls take, and whether that's changing over time
This doesn't mean logging everything verbatim — that creates its own problems (storage, security, noise). It means logging the right things at the right level of detail.
Idempotency matters
If your integration processes a record twice, what happens? If the answer is "it creates a duplicate" or "it overwrites changes", you have a problem that will eventually surface in production.
Design your integrations to be idempotent: processing the same input twice should produce the same result as processing it once. This usually means:
- Using unique identifiers to detect duplicates
- Tracking what's been processed (and what hasn't)
- Using upsert patterns instead of blind inserts
Version and monitor
APIs change. Sometimes with notice, sometimes without. A field gets renamed. A response format shifts. A deprecated endpoint finally gets removed.
Protect yourself by:
- Pinning to specific API versions where possible
- Monitoring response shapes — if the structure changes, you want to know before it breaks your data
- Testing integrations regularly, not just when they're first built
- Keeping integration code modular so changes to one connector don't cascade through the system
The payoff
A well-built integration is invisible. Data flows between systems. Teams trust the numbers. Nobody is copying data between spreadsheets or chasing discrepancies. That's the goal — and it's worth investing in up front to get there reliably.
The best integration is one nobody has to think about. It just works — and when it doesn't, it tells you exactly why.