All Posts in Architecture

Enterprise Technical Architecture Management

|In: Architecture|Last Updated:

Sometimes, we’re too clever for our own good. A couple of recent trips highlighted several more significant issues to discuss around integrations and patterns. It might be that Tech leaders really are clever—but clever isn’t always the same as smart.

Take for example, times we overcomplicate choices. Continuing our conversation with the “Business”, it might make sense to set some expectations about integrations.

Have you considered what the additional functions will do to your infrastructure sizing and expected performance? Make sure your integration requirements include the often-neglected “Non-Functional Requirements”—we need to be able to back up and recover the system, keep users happy about performance, and meet our other reliability and scalability goals. The last thing you want is a system that doesn’t perform as well as its individual components. Remember- you want the Business to be your ally, and they leave it to you to understand what an NFR is.

You need to perform a bare metal recovery test, especially of any integrations. We inherited a pet duck from our neighbors, named “Ducky”. She swims around in her kiddie pool in the back yard, under the shade of a very large Tangelo tree. She is the happiest being I think I have ever met.

Do you know why? She doesn’t have to worry about the steps needed to recover a system with multiple integrations, that may or may not be recoverable in a logical order.

That’s why.

Think about an e-commerce system connected to an ERP for a moment. If the ERP has a brief system failure, does the E-commerce system understand that it needs to resend just the orders that were missed for the outage?  (Hint—good systems do this pretty well, but you should still check it. Because when it comes to data integrity, pretty well isn’t good enough.)

Is the juice worth the squeeze?  Good integrations are expensive. Your awesome technical teammates are going to work through edge cases, error handling through multiple platforms, message timing, and multiple development languages. Can your business teammates make the business case how this effort is going to deliver Some-X ROI or greater?  Will your business teammates defend the investment needed for quality integrations, especially as compared to the consequences of failing to make it?

Point to point vs integration platform (such as an Enterprise Service Bus). At the moment, here’s what I recommend to keep the decision relatively straightforward: If you must connect only two or three systems, Point to Point integrations make sense. If you are being asked to connect more than that, however, the number of integrations grows polynomially, and maintenance and technical debt will become an issue.

And so, following Bob’s rule of making the architecture cleaner than when you found it, an integration platform (such as an Enterprise Services Bus) makes a lot of sense. It is a big investment, however. Make sure that your colleagues understand what they are getting into.

One of your colleagues might have been reading about how “Microservices” are a thing, and that it makes all integration easy. Just like EAI platforms made integration easy, ESBs make it easy, and, if you like cloud stuff, IaaS makes it easy. That is, it doesn’t. Nothing makes integration easy.

Microservices architecture is just another way for you to own all of the architecture. Be prepared to manage the costs and staff levels to maintain it. Microservices does give you more flexibility—and you need the wisdom to use it. This is a controversial position I am taking, and I expect to hear disagreements. But, when you ride the microservices tiger, you may not get to choose when you dismount. You now own all of the choices in how the target systems work together. Be sure your team is ready for the adventure.

Real Time “Extract- Transform-Load” between systems is still an integration. Taking data from one system to another, even if we pretend we are not doing an “Integration”, means that we are doing an integration. The same points come up on cleanliness, architecture, sizing, currency, and so on. As my grandmother used to say “You are not fooling anybody!” The vendors that push this are using semantics to get around nervous stakeholders, not really inventing anything new. Let’s chalk this up to being part of the same discussion about investing in an integration platform.

Internet research on Integration isn’t necessarily real research. And some business executives think it is. I am continually bewildered that some people are taking medical, career or relationship advice from Tik Tok-ers. While I am sure that it is meant with the best of intentions, it is hard to believe that someone who doesn’t know us, or our situation, and has never had to do the hard work of making it work could give more than a handwaving opinion that would have relevance.

And, yet, some of our friends in the Business, doing a bit of research on the internet, will come up with “out-of-the-box” solutions that may or may not introduce a whole host of other problems. We do owe it to them to investigate their solutions. AND, we should ask that everybody keep an open mind on the solutions, taking into account security, scalability, and overall architecture.

Pro tip: Figure out a way to reframe the solution that will work (that is, your preferred solution) so it resembles what your business colleagues have discovered. This will make achieving consensus much easier.

When it comes to integration, following the long held KJR rule of “Keep it Simple”, turns out to be really complicated. My question for you—How do you make it look simple in your conversations without committing to something that won’t work in practice?

Comments

|In: Architecture|Last Updated:

Bob here. This week we share a conversation about one of IT’s critical success factors – integration. The past couple of weeks Greg has focused on this as one of IT’s central challenges, so much so that he proposed changing the CIO acronym to stand for “Chief Integration Officer.”

Now I’ve been involved in IT integration since the ancient days of Electronic Data Interchange service bureaus. Call it 1980 or thereabouts, and even back then we ran into the problem of conflicting semantics, where the sender’s definition of a data field didn’t match the receiver’s definition. The more integration projects I’ve been involved in, the more I came to the conclusion that the technological solutions to simplifying and streamlining integration did a terrific job of making the easy aspects of integration even easier while doing little or nothing to solve the hard problems, like resolving semantic mis-matches.

What do you think, Greg? Am I overstating the challenge? Or am I just out of date and it was solved a long time ago?

Greg says:  I wouldn’t say you’re out of date, although, in my opinion at least, semantic matching has gotten easier. But Integrations are as hard as they have ever been.  The problem has just shifted, a bit like squeezing a tube of refrigerated cookie dough.

Bob says: Now we know what Greg snacks on.

Greg says: Ahem. As I was sayin’, integration is as hard as it ever was. The problems are still in familiar places – especially, volume, performance and speed.  In some ways, a lot of integration projects now feel like near-real-time Extract-Transform-Load problems. The expectations are really high, such as thinking we can make a 1 MB payload of multiple features move between systems instantaneously.

Then there’s security and trust. Can we ensure that the payload we are moving is trustworthy?  Passing bounds testing is simple, but are we sure that it is data that we want to be committed to another system?

Bob: So it’s just as hard, just differently hard. Okay. Then a follow-up: I’ve often wondered if pre-built integration might be the dominant reason IT shops find ERP solutions attractive.

Somewhere or other I read a piece of advice for deciding when to use an ERP module vs a best-of-breed application. The advice: If the ERP module provides even 50% of the functionality the business needs, choose it over integrating the best of breed. My question: do you agree? And to the extent you agree, do you think pre-built integration is a driving force behind ERP? Maybe on a 1 to 5 scale, where 5 says it’s the main driver and everything else is pretext; 1 means it’s nice, but not all that important.

Greg: Prebuilt integrations do help CIOs! Especially if you call them Chief Integration Officers (of course).

ERPs do still have a long way to go, but have made improvements.  Using your scale, I think ERP prebuilt integrations probably are at a solid 2, making it so that as a CIO, you can focus on some of the more value-oriented challenges of an implementation.

CRM systems lead the way.  I am constantly amazed at the creativity of certain CRM brands and the prebuilt integrations offered to make them more enticing.  I also notice that these same brands don’t market themselves to tech leadership – they spend their marketing money talking to Sales leaders about the ease of implementation. At the risk of being controversial, my guess is that they are giving Sales leaders talking points to deflect any concerns IT might raise about the true cost and complexity of the implementation effort.

Bob: Interesting bit (to me, at least). In the early cloud days, AWS and Azure started as Infrastructure as a Service providers (IaaS) and then moved up the stack to become Platform as a Service (PaaS) providers. But while nobody was looking, Salesforce, which started life as a Software as Service (SaaS), grew down the stack with its Force development environment (now Salesforce Platform), turning itself into a PaaS provider from the opposite direction.

Which gets me to the world of integration tools like enterprise service buses (ESB), enterprise application integration (EAI) tools, APIs, and Integration as a Service vendors. My question: Do these work better than custom interfaces and integrations? Do they, that is, succeed at simplifying inter-application integration?

Greg:

That is a great question, and like a friend of mine often says, “It depends!”

ESBs, Integration as a Service, and EAI tools make a lot of sense when we are facing-

Lots of systems that all need to talk to each other (less so when all the systems need to talk to just one system).

Older platforms need to talk to newer platform. I tip my hat to the remaining RPG and COBOL programmers who figure out integrations everyday.

A need for a “Clean” architecture. The initial efforts need to be worth the outcome of orderliness and stability (And they usually are).

The biggest drawback that we face are the initial upfront costs, as well as the ongoing costs.  Business Leaders may face a bit of sticker shock with this approach.  This conversation is likely to require a lot of explanation into the gnarly bits of Tech.

Bob: I think we’ll have to hold off on the gnarly bits. Greg, it’s been a pleasure figuring out if and where we agree and disagree. Everyone who’s read this far – add your agreements and disagreements in the Comments.

Comments