All Posts in Technology

|In: Architecture|Last Updated:

Bob here. This week we share a conversation about one of IT’s critical success factors – integration. The past couple of weeks Greg has focused on this as one of IT’s central challenges, so much so that he proposed changing the CIO acronym to stand for “Chief Integration Officer.”

Now I’ve been involved in IT integration since the ancient days of Electronic Data Interchange service bureaus. Call it 1980 or thereabouts, and even back then we ran into the problem of conflicting semantics, where the sender’s definition of a data field didn’t match the receiver’s definition. The more integration projects I’ve been involved in, the more I came to the conclusion that the technological solutions to simplifying and streamlining integration did a terrific job of making the easy aspects of integration even easier while doing little or nothing to solve the hard problems, like resolving semantic mis-matches.

What do you think, Greg? Am I overstating the challenge? Or am I just out of date and it was solved a long time ago?

Greg says:  I wouldn’t say you’re out of date, although, in my opinion at least, semantic matching has gotten easier. But Integrations are as hard as they have ever been.  The problem has just shifted, a bit like squeezing a tube of refrigerated cookie dough.

Bob says: Now we know what Greg snacks on.

Greg says: Ahem. As I was sayin’, integration is as hard as it ever was. The problems are still in familiar places – especially, volume, performance and speed.  In some ways, a lot of integration projects now feel like near-real-time Extract-Transform-Load problems. The expectations are really high, such as thinking we can make a 1 MB payload of multiple features move between systems instantaneously.

Then there’s security and trust. Can we ensure that the payload we are moving is trustworthy?  Passing bounds testing is simple, but are we sure that it is data that we want to be committed to another system?

Bob: So it’s just as hard, just differently hard. Okay. Then a follow-up: I’ve often wondered if pre-built integration might be the dominant reason IT shops find ERP solutions attractive.

Somewhere or other I read a piece of advice for deciding when to use an ERP module vs a best-of-breed application. The advice: If the ERP module provides even 50% of the functionality the business needs, choose it over integrating the best of breed. My question: do you agree? And to the extent you agree, do you think pre-built integration is a driving force behind ERP? Maybe on a 1 to 5 scale, where 5 says it’s the main driver and everything else is pretext; 1 means it’s nice, but not all that important.

Greg: Prebuilt integrations do help CIOs! Especially if you call them Chief Integration Officers (of course).

ERPs do still have a long way to go, but have made improvements.  Using your scale, I think ERP prebuilt integrations probably are at a solid 2, making it so that as a CIO, you can focus on some of the more value-oriented challenges of an implementation.

CRM systems lead the way.  I am constantly amazed at the creativity of certain CRM brands and the prebuilt integrations offered to make them more enticing.  I also notice that these same brands don’t market themselves to tech leadership – they spend their marketing money talking to Sales leaders about the ease of implementation. At the risk of being controversial, my guess is that they are giving Sales leaders talking points to deflect any concerns IT might raise about the true cost and complexity of the implementation effort.

Bob: Interesting bit (to me, at least). In the early cloud days, AWS and Azure started as Infrastructure as a Service providers (IaaS) and then moved up the stack to become Platform as a Service (PaaS) providers. But while nobody was looking, Salesforce, which started life as a Software as Service (SaaS), grew down the stack with its Force development environment (now Salesforce Platform), turning itself into a PaaS provider from the opposite direction.

Which gets me to the world of integration tools like enterprise service buses (ESB), enterprise application integration (EAI) tools, APIs, and Integration as a Service vendors. My question: Do these work better than custom interfaces and integrations? Do they, that is, succeed at simplifying inter-application integration?

Greg:

That is a great question, and like a friend of mine often says, “It depends!”

ESBs, Integration as a Service, and EAI tools make a lot of sense when we are facing-

Lots of systems that all need to talk to each other (less so when all the systems need to talk to just one system).

Older platforms need to talk to newer platform. I tip my hat to the remaining RPG and COBOL programmers who figure out integrations everyday.

A need for a “Clean” architecture. The initial efforts need to be worth the outcome of orderliness and stability (And they usually are).

The biggest drawback that we face are the initial upfront costs, as well as the ongoing costs.  Business Leaders may face a bit of sticker shock with this approach.  This conversation is likely to require a lot of explanation into the gnarly bits of Tech.

Bob: I think we’ll have to hold off on the gnarly bits. Greg, it’s been a pleasure figuring out if and where we agree and disagree. Everyone who’s read this far – add your agreements and disagreements in the Comments.

Comments

|In: Technology|Last Updated:

If your career included a technical phase it’s likely that the first project you were involved with included integration as a deliverable. The last IT project you found yourself involved in probably included integration as a deliverable as well.

It would be unsurprising if they were the same project.

Regardless of the amazing coverage of your ERP or CRM, or the depth of a new point solution … wait. I need to start that sentence over: Not “regardless,” but “because of” the amazing coverage of the systems you have or are in the process of implementing, they will have data and functional overlaps. To take an easy-to-understand example, your ERP and CRM systems both manage data about your customers somewhere in the depths of their databases.

Failure to integrate them means that any time you want information about a customer, or knowledge about customers in the aggregate, the two systems will disagree.

These disagreements are gaps. Your business sponsor will either find them, or hear about them from someone else who did. And, they’ll have sufficient sophistication to know the gaps could “easily” be closed by integrating the systems. And they’ll want to know why you didn’t take this obvious step.

Let’s role-play the conversation you would have to have with the business sponsor to get yourself off the hook. You’ll need to ask the business sponsor a few questions, to help them understand the tradeoffs the team will need to make in order for the project to move forward.

Let’s rehearse a few of the points that go into this conversation, starting with:

  • Where does one system start, and where does the other one pick up the mission?

Where systems overlap, that is, which is the source of truth?  Breaking this down further, what we are really seeing are three overlaps—Overlapping Data (both systems might need a street address), Overlapping Functional Logic (both systems need to make sure that a delivery is going to a valid location) and Overlapping Business Logic (both systems are involved in order fulfillment).

To say this gets messy is an understatement.  Ideally, if you ask your CRM and ERP systems the same question about an order, you should get the same answer, in terms of payments, fulfillment stage, delivery location, billing location and customer. But depending on how your solution will synchronize them, the answer might be to let them disagree. Is this okay? Which gets to the next question:

  • If you must choose, which system needs to be “right”?

In our conversations with our colleagues, we do need to ask which system should be considered the System of Record, which systems depend on the information from this system, and when they all must agree.

  • What’s the flip side of the coin?

How often, that is, do the two systems need to re-synchronize? Near-real time? Overnight through a batch process? At month’s end as part of closing the books? This is when you give your business sponsor the bad news about synchronization: The closer we get to real time, the more complex the engineering and the higher the cost. Not to mention the higher cost. If the business sponsor wants real time or nearly so, are they willing to pay for it?

  • Every system has data that is in some way, shape, or form, “dirty.”

CRM systems, for example, are really, really good at helping you stay connected to customers. They’ll track every interaction imaginable. They are also notorious for creating an almost schizophrenic portfolio of contacts that are, in fact, the same person, but with one letter in their name different, slightly different addresses, birthdays, and so on. It is not uncommon to have 10+ entries associated with the same human being. Which of the ten should your ERP system synchronize to?

It’s a good question with no right answer. The dirty-data problem mucks up expensive marketing campaigns, recalls or RMAs … even interpersonal interactions. CRMs are likely the worst offender, but not the only one for introducing bad data to other systems.

  • Is data cleansing in your company’s future?

Without it you’ll never finish implementing the new system. With it comes expense, implementation delays, and the certainty that three years from now you’ll have to cleanse all that data all over again.

Will this conversation with your business sponsor be easy? Sure it will. Conversations about trade-offs are always fun and games, aren’t they? But especially with your business sponsor, and then recapping the results to your team, you are going to gain trust and build alignment—which might at least make later conversations easier for everyone.

Comments