Before you can be strategic you have to be competent.

That’s according to Keep the Joint Running: A Manifesto for 21st Century Information Technology, (me, 2012), the source of all IT management wisdom worth wisdoming.

An unglamorous but essential ingredient of IT organizational competence is software quality assurance (SQA), the nuts-and-bolts discipline that makes sure a given application does what it’s supposed to do and doesn’t do anything else.

SQA isn’t just one practice. It’s several. It checks:

Software engineering – whether code adheres to the overall system architecture, is properly structured, and conforms to coding style standards.

Unit testing – whether a module correctly turns each possible input into the expected output.

Integration testing – whether a module interacts properly with all the other modules the team is creating.

Regression testing – whether the new modules break anything that’s already in production.

Stress testing – whether the whole system will perform well enough once everyone starts to bang on it.

User acceptance – whether the new modules are aesthetically pleasing enough; also, whether they do what the business needs them to do – do they, that is, effectively support, drive, and manage the business processes they’re supposed to support, drive, and manage.

Ideally, IT’s SQA function will establish and maintain automated test suites for all production applications and keep them current, to ensure efficient and correct unit, integration, regression, and stress testing.

In practice, creating and managing automated test suites is really, really hard.

This looks like a fabulous opportunity for generative AI, doesn’t it? Instead of asking it to generate a mathematical proof in the style of William Shakespeare, point your generative AI tool of choice to your library of production application code and tell it to … generate? … an automated test suite.

Generative AI, that is, could take one of the most fundamental but time-consuming and expensive aspects of IT competence and turn it into a button-push.

Brilliant!

Except for this annoying tidbit that’s been an issue since the earliest days of “big data,” generative AI’s forgotten precursor: How to perform SQA on big data analytics, let alone on generative AI’s responses to the problems assigned of it.

Way, way, way back we had data warehouses. Data warehouses start with data cleansing, so your business statisticians could rely on both the content and architecture of the data they analyzed.

But data warehouse efforts were bulky. They took too long, were anything but flexible, and frequently collapsed under their own weight, which is why big data, in the form of Hadoop and its hyperscale brethren, became popular. You just dumped your data into some data lakes, deferring data cleansing and structuring … turning that data into something analyzable … until the time came to analyze it. It was schema on demand, shifting responsibility from the IT-based data warehouse team to the company’s newly re-named statisticians, now “data scientists.”

The missing piece: SQA.

In scientific disciplines, researchers rely on the peer review process to spot bad statistics, along with all the other flaws they might have missed.

In a business environment, responsibility for detecting even such popular and easily anticipated management practices as solving for the number has no obvious organizational home.

Which gets us to this week’s conundrum. We might call it SQA*2. Imagine you ask your friendly generative AI to automagically generate an automated test suite. It happily complies. The SQA*2 challenge? How do you test the generative AI’s automated test suite to make sure the flaws it uncovers are truly flaws, and that it doesn’t miss some flaws that are present – feed it into another generative AI?

Bob’s last word: It’s easy, and gratifying, to point out all the potential gaps, defects, fallacies, and potential pitfalls embedded in generative-AI implementations. In the generative-AI vs human beings competition, we can rely on confirmation bias to assure ourselves that generative-AI’s numerous flaws will be thoroughly explored.

But even in the technology’s current level of development, we Homo sapiens need to consider the don’t-have-to-outrun-the-bear aspect of the situation:

Generative-AI doesn’t have to be perfect. It just has to be better at solving a problem than the best human beings are.

This week’s SQA*2 example … the automated generation of automated test suites … exemplifies the challenge we carbon-based technologies are going to increasingly face as we try to justify our existence given our silicon-based competition.

Bob’s sales pitch: You are required to read Isaac Asimov’s short story in which he predicts the rise of generative AI. Titled “The Jokester,” it’s classic Asimov, and well worth your time and attention (and yes, I did say the same thing twice).

Now on CIO.com’s CIO Survival Guide: “5 IT management practices certain to kill IT productivity.” What’s it about? The headline is accurate.

What do you do when someone else’s evidence and your logic collide? You might:

• Use ad hominem “logic” to cast aspersions on the someone else.
• Not waste time bothering with ad hominem logic – just deny the other side’s evidence.
• Create a strawman version of what you’re trying to refute – something that resembles it, only with a few fatal flaws added to let you “prove” it can’t work.
• Embrace your confirmation bias. Find someone on the internet … anyone on the internet … who asserts evidence supporting whatever position you like best. Cite them as the only authority worth paying attention to.
• Redefine the criteria for being right.
• Find a historical case of a scientist whose colleagues ridiculed their theories, only to be vindicated in the end. Claim they’re your ideological brethren. Whether the historical scientist was actually subjected to ridicule or not doesn’t matter.
• Or, reverse the last stratagem: Find a theory that was popular once upon a time, only to ultimately be proven wrong. Those who disagree with you would have agreed with it.

Which brings us to the sad, sad case of why Waterfall methodologies persist.

In case you’re unfamiliar with the term, a Waterfall methodology is one that designs a project plan in terms of a linear progression. For application development these would typically start with a feasibility study, followed by requirements definition, specifications, coding, testing, training, and deployment.

With Waterfall methodologies, design has to be complete before development can begin, and if anything has been left out, or a design assumption has changed (as in, “Oh, you wanted wings on that airplane?”) whoever is championing the change goes through a change control process, which includes the project manager rippling the change through the entire project plan.

Agile, in contrast, is a family of methodologies, not a methodology. What its variants have in common is that they build software iteratively and incrementally. The list of Things the Application is Supposed to Do is called the backlog.

Projects still start with some form of feasibility study, one of whose work products is the initial backlog; another is defining the “minimum viable product” – the irreducible core of the planned application that everything else hooks onto.

From that point forward there is a process for deciding which items in the backlog have the highest priority. As for anything left out of the backlog or a change in design assumptions, these are pushed into the backlog where they are subjected to the same priority-setting process as everything else.

This will do as a barebones sketch. If you want more, please refer to chapter 3 of There’s no such thing as an IT project, Fixing Agile.

The best available evidence, from the widely respected Standish Group, reports that Agile projects are fully successful at nearly twice the rate as Waterfall projects, which fail completely about two and a half times as often as their Agile counterparts.

Case closed? If only.

Some Waterfall proponents counter with one or more of the denial strategies that started this article. For example a popular strawman is that Agile can’t work because in order to optimize the whole you have to suboptimize the parts, which supposedly can’t happen because in Agile, each developer does whatever he or she wants to build a capability that accretes onto the accumulating application.

This is a strawman. Agile projects build to a well-defined architecture, to user-interface standards, and to an overall coherent plan: Anything added to the backlog has to be consistent with the rest of the backlog.

Meanwhile, the implication is that in Waterfall projects, designers can optimize the whole. This assertion is, in a way, accurate. Designers can optimize the whole, so long as the target “the whole” is shooting at doesn’t change over the life of the project.

Good luck with that. The specifics of what a business needs to succeed within a given application’s domain change all the time.

So by the time a Waterfall-developed application is done, the criteria for “done” have changed.

Bob’s last word: The specifics of what a business needs to succeed within a given application’s domain changes all the time in successful businesses. Leaders of successful businesses know that their success depends on continuous adaptation to changing circumstances.

Success, that is, depends on agility.

Bob’s sales pitch: “To optimize the whole you must sub-optimize the parts” is a well-recognized principle here in KJR-land. If you’d like some guidance on how to apply it to your own real-world situations, you’ll find it in chapter 2 of Keep the Joint Running: A Manifesto for 21^st Century Information Technology, which explores this concept in depth.