Before the enterprise data gateway went live, I ran a dedicated audit on it and fixed 15 issues. The ones that scared me most were not the ones that threw errors. They were the five below — they throw nothing, and quietly hand you a wrong conclusion.
1. Storing a guess as the answer
On a fuzzy search by company name, the system took the first candidate and wrote "search term → company" into the alias table. From then on, the same term hit the cache directly and returned that company — without ever checking with the upstream again.
A guess had been dressed up as a fact. The fix was simple: only exact-match APIs may write aliases; fuzzy search never does.
2. Counting our own blocks as upstream failures
The upstream failure rate was defined as "not a cache hit, and the call failed". But requests stopped by our own risk checks happen to satisfy both conditions.
So one subsidiary sending too many requests could push the "upstream failure rate" to 100%, trigger a false outage alert, and even trip the circuit breaker on a perfectly healthy provider. Authentication failures and rate-limit rejections have to be excluded from that statistic.
3. No channel configured, yet the UI says "sent"
When no alert channel was configured, the send function returned a line of explanatory text — and the success check only looked for the word "failed" in that text. The interface showed "sent" while not a single alert had actually gone out.
After the fix there are three structured states (illustrative code):
// before: no "failed" in the string means success
const allOk = !result.includes('failed');
// after: three explicit states
// sent delivered
// failed delivery failed
// undelivered not delivered (no channel configured)4. Two retries, one quota charge
Quota was deducted once, outside the retry loop, while every retry inside the loop really did call the official API again. Over time the gateway's usage count fell below reality: when the true limit arrived, the upstream errored first while the dashboard still showed quota to spare.
The right way is to deduct once per real external call, and refund in full when all attempts fail.
5. Monitoring that becomes a cost
My first idea was to probe the upstream for real. Then it sank in: every probe is a billed call, so the more diligently you monitor, the more you spend.
In the end the upstream is only checked at the configuration level, and its health is read from the success rate of real business calls.
Takeaway
None of these would survive one plain test: when you add the breakdowns together, do they equal the total? Cache, upstream and blocked must sum to total calls; the usage the gateway records must match the provider's bill. I later added automated tests that do exactly this kind of reconciliation.