Sometimes, we’re too clever for our own good. A couple of recent trips highlighted several more significant issues to discuss around integrations and patterns. It might be that Tech leaders really are clever—but clever isn’t always the same as smart.
Take for example, times we overcomplicate choices. Continuing our conversation with the “Business”, it might make sense to set some expectations about integrations.
Have you considered what the additional functions will do to your infrastructure sizing and expected performance? Make sure your integration requirements include the often-neglected “Non-Functional Requirements”—we need to be able to back up and recover the system, keep users happy about performance, and meet our other reliability and scalability goals. The last thing you want is a system that doesn’t perform as well as its individual components. Remember- you want the Business to be your ally, and they leave it to you to understand what an NFR is.
You need to perform a bare metal recovery test, especially of any integrations. We inherited a pet duck from our neighbors, named “Ducky”. She swims around in her kiddie pool in the back yard, under the shade of a very large Tangelo tree. She is the happiest being I think I have ever met.
Do you know why? She doesn’t have to worry about the steps needed to recover a system with multiple integrations, that may or may not be recoverable in a logical order.
That’s why.
Think about an e-commerce system connected to an ERP for a moment. If the ERP has a brief system failure, does the E-commerce system understand that it needs to resend just the orders that were missed for the outage? (Hint—good systems do this pretty well, but you should still check it. Because when it comes to data integrity, pretty well isn’t good enough.)
Is the juice worth the squeeze? Good integrations are expensive. Your awesome technical teammates are going to work through edge cases, error handling through multiple platforms, message timing, and multiple development languages. Can your business teammates make the business case how this effort is going to deliver Some-X ROI or greater? Will your business teammates defend the investment needed for quality integrations, especially as compared to the consequences of failing to make it?
Point to point vs integration platform (such as an Enterprise Service Bus). At the moment, here’s what I recommend to keep the decision relatively straightforward: If you must connect only two or three systems, Point to Point integrations make sense. If you are being asked to connect more than that, however, the number of integrations grows polynomially, and maintenance and technical debt will become an issue.
And so, following Bob’s rule of making the architecture cleaner than when you found it, an integration platform (such as an Enterprise Services Bus) makes a lot of sense. It is a big investment, however. Make sure that your colleagues understand what they are getting into.
One of your colleagues might have been reading about how “Microservices” are a thing, and that it makes all integration easy. Just like EAI platforms made integration easy, ESBs make it easy, and, if you like cloud stuff, IaaS makes it easy. That is, it doesn’t. Nothing makes integration easy.
Microservices architecture is just another way for you to own all of the architecture. Be prepared to manage the costs and staff levels to maintain it. Microservices does give you more flexibility—and you need the wisdom to use it. This is a controversial position I am taking, and I expect to hear disagreements. But, when you ride the microservices tiger, you may not get to choose when you dismount. You now own all of the choices in how the target systems work together. Be sure your team is ready for the adventure.
Real Time “Extract- Transform-Load” between systems is still an integration. Taking data from one system to another, even if we pretend we are not doing an “Integration”, means that we are doing an integration. The same points come up on cleanliness, architecture, sizing, currency, and so on. As my grandmother used to say “You are not fooling anybody!” The vendors that push this are using semantics to get around nervous stakeholders, not really inventing anything new. Let’s chalk this up to being part of the same discussion about investing in an integration platform.
Internet research on Integration isn’t necessarily real research. And some business executives think it is. I am continually bewildered that some people are taking medical, career or relationship advice from Tik Tok-ers. While I am sure that it is meant with the best of intentions, it is hard to believe that someone who doesn’t know us, or our situation, and has never had to do the hard work of making it work could give more than a handwaving opinion that would have relevance.
And, yet, some of our friends in the Business, doing a bit of research on the internet, will come up with “out-of-the-box” solutions that may or may not introduce a whole host of other problems. We do owe it to them to investigate their solutions. AND, we should ask that everybody keep an open mind on the solutions, taking into account security, scalability, and overall architecture.
Pro tip: Figure out a way to reframe the solution that will work (that is, your preferred solution) so it resembles what your business colleagues have discovered. This will make achieving consensus much easier.
When it comes to integration, following the long held KJR rule of “Keep it Simple”, turns out to be really complicated. My question for you—How do you make it look simple in your conversations without committing to something that won’t work in practice?