Ann Landers used to advise that if something seems too good to be true, it probably is.
This week’s example is an IT organization extolled in a recent trade press article. Among its virtues, and those of its prominently featured CIO, are 70-hour work weeks, strong coffee, and amazing focus — its staff regularly deploys major applications in less than half the time estimated by professional services companies vying for its business. A Gartner analyst cited in the article credits it with highly mature processes which take incredible care of “internal customers.”
Through entirely accidental circumstances I fell into possession of a letter of apology its CIO sent to customers last week — real paying customers, not internal ones. Some snippets:
” … the vast majority of system problems we have are problems related to updates. These update problems have been manifesting themselves as inventory update failures, missing orders, missing images, incorrect status synchs, etc. At the end of the day, all of these problems boil down to [the company’s] failure (read, my failure) to architect a system that can handle real-time updates properly.”
This is simply irresistible, as Robert Palmer might have sang. Call me sarcastic, call me snide, call me an armchair quarterback — other than updates, what do transaction processing systems do? This sounds like ManagementSpeak for “problems with everything.”
“In the current system, inventory updates, orders, image data, status changes, etc., are all written to small files which are then sent back and forth between systems. The sending system writes and sends the file and automatically assumes that the receiving system processed the file. This “fire and forget” approach is killing us. In reality, a file might not send properly, become corrupted in transfer, or produce errors when the receiving system attempts to process it. In most cases, we don’t know when we have problems. The architecture is horribly architected.”
Well, yes, it is. One might think the CIO and his staff had never heard of transaction processing. It’s understandable. The technology has only been around for thirty years or so.
Oh — and repeat after me: Architect is a noun!
“In the new system, there are database tables between each system that we call “queue tables.” For example, when [the company’s] shopping database receives an order, that order is written to a queue table. Oracle picks up that order, and processes the order in the Oracle database. Once processed, the order is immediately written to another queue table.”
When one kludge doesn’t work, replace it with another. Memo to CIO: This is a solved problem. If the transaction processing built into Oracle isn’t to your liking, IBM, BEA, and quite a few other companies sell outstanding middleware that reliably posts transactions, on-line as they happen, to databases. That’s why it’s called On Line Transaction Processing (OLTP).
“We’ve added extra staff to watch the queue tables and handle any errors in communication.”
Good idea. Combine kludges with brute force. It sure beats using the automated alerts built into commercial transaction processing monitors.
“With that said, it’s critically important that I prepare you for the worst. When we began the Oracle implementation in February of this year, Oracle told us it would take 12-18 months. We cut the project scope down as much as we could for Phase I, forced our own developers to work incredible hours, and greatly increased the number of Oracle consultants on the project, all in an attempt to launch in early August.”
Hey, that’s the way to fix a problem with system quality — create a death march. Nice phrasing, by the way: “… forced our own developers to work incredible hours.” Alotta guys would have provided a financial incentive.
Okay, fair is fair. I’m unencumbered by facts, so my suspicion — that the high-visibility article in the trade press is part of a marketing campaign by the CIO to help him escape to a new job before the current one comes crashing down around his ears — is uninformed by first-hand knowledge of the situation.
Nor is my assessment of their future system architecture based on anything more than my own interpretation of the CIO’s apology letter. Maybe the plan really is Just Fine.
It doesn’t, however, take much inferring to figure that a CIO who focuses on pleasing internal customers, works 70-hour weeks, expects the same of everyone working for him, and takes pride on ultra-fast delivery is likely to take a few architectural shortcuts.
The problem is as it so often is: When you save on architecture in the present, you’re usually mortgaging your future.