It’s a scandal! It’s a waste of taxpayer dollars (roughly $1 per taxpayer)! It’s proof, if I’d just take my partisan blinders off, as one commenter put it in response to last week’s column, that the federal government is totally incompetent at project management!
Bad project execution is a partisan issue? Seriously?
One failed project no more proves the federal government is “bad at project management” than NASA’s three blindingly successful Mars rover missions prove it’s good at it.
For that matter, the myriad, usually buried-in-the-back-yard private sector project failures tell us Healthcare.gov’s problems probably have little to do with the .gov part. Big projects are risky, for well-known reasons.
Any number of commentators have emitted strongly-held opinions about this mess, whether or not they know anything at all about software development, systems integration, or project management.
And yet, few of the mistakes that mattered were either intrinsic to government or were project management failures. What they were was entirely unoriginal. For example:
Integration engineering: It is by now well known that Healthcare.gov integrates information from more than a dozen back-end databases, synchronously, in real time. It is the big flaw.
But project management isn’t software engineering. Project managers make sure the tasks in a project adhere to the budget, schedule … and specifications. They don’t pass judgment on the specifications, unless they also happen to be software engineers.
The design was questionable at best. When a system has to consult a bunch of other systems … Healthcare.gov retrieves applicant information from quite a few other government databases … its response time can’t be any faster than the slowest of them. And as those systems weren’t designed to handle the additional processing load from Healthcare.gov, this was risk piled on risk.
My guess, based on no knowledge at all of this project, but plenty of experience in how these things happen: Either an engineering purist insisted on design elegance, or a non-technical manager overruled the experts for political reasons. Regardless, this problem had nothing to do with project management.
Agile? Quite a few commentators have suggested this should have been an Agile project. They’re wrong.
First, see above. Agile wouldn’t have avoided bad integration engineering. Second, Agile shines when planners can break a big effort into small chunks; when the whole organization has to “learn its way into” how the software should work; and when user-interface design dominates the effort. In the more common software development situations, that is.
But not Healthcare.gov. It had a detailed spec. It’s called the Affordable Care Act. Its business logic is complex. This is where you do want waterfall, not iteration and high levels of user involvement.
Slavish Adherence to Deadlines (SAD?): This was a sponsorship issue, making it a project management problem but not a problem of having bad project managers.
It’s banal and ubiquitous: In big projects, when early phases go long, everyone tells each other they can make up the time in later phases. Eventually, the only phase left to make it up in is testing.
And as everyone in the software business knows, you always test software, and test it thoroughly. The only question is whether you test it before it goes into production or after.
Deadline-adherence is a choice. Sometimes, as in Y2K remediation or product rollouts, the deadlines are real. Usually, SAD comes either from politics trumping engineering or the next topic, which is …
Optimism bias: I’m speculating but it’s a pretty safe bet. Throughout any big effort, sponsors … who are supposed to be professional optimists because otherwise they’d never do anything as crazy as sponsoring the project … often pressure project managers to commit to an aggressive schedule, or accept risks they shouldn’t.
Among the reasons is the time-honored tradition of Solving for the ROI. Another is the sponsor’s reputation for delivering the goods.
With Healthcare.gov, on the front end contractors knew that price and schedule would be key to winning the business. At the back end, everyone in the Obama administration’s decision hierarchy had committed to a date.
I doubt any of them lied, except to themselves. They succumbed to optimism bias.
Learning?
There’s nothing to learn from this little contretemps. Nothing at all.
The problems with the Healthcare.gov project are well-known, entirely preventable mistakes. If you want lessons to learn, maybe these: Make sure your sponsors know how to sponsor, and make sure key engineering decisions get a thorough vetting.
But really, I sure hope neither of these are lessons. Reminders, perhaps … of lessons learned long ago, over and over and over.
Excellent analysis. The only 2 things I would add are:
1. They should have told Obama 3 months ago that there were problems and what the problems were. While I only consider him to be an expert in Constitutional law, I just have a strong hunch that he has an instinctive knack for dealing with these kinds of projects and would have recognized the extreme danger of adding id verification a few months before the launch date.
2. He should have found an experienced, trustworthy IT manager to advise him, since he probably felt he didn’t have the technical expertise himself. Once he found that person, he should have trusted him or her blindly, even if he fully didn’t understand. Everyone has blind spots, and if you’re going to succeed, you’ve got to find people different than you, but with a good track record because sometimes you just have to trust the other guy blindly – if they have the track record.
Well Robert – as for #1 “they should have told him…” Given this is his signature ‘accomplishment’, one would think he would be interested enough to ask. As for #2 – you are exactly right – that is what experienced and competent executives do.
Of all the comments regarding healthcare.gov, that have gotten my attention, this piece is the most concise, well written, and accurate, synopsis of them all.
Having many years of .gov experience under my belt, I can say the motivators described are spot on.
‘Nuf said.
The most nonsensical statements I heard were the commentators saying “In the private sector this would never happen”. Are they kidding or just willfully ignorant? This happens *all the time*. It’s just that 6 or 7 network news outlets aren’t pounding away on these websites to demonstrate to the world how bad they are.
As far as the optimists go, I like to ask them: What in our past experience suggests we will successful this time? I never get a satisfactory answer.
Just a point of information from a space enthusiast (who actually worked on the Apollo project): NASA had four “blindingly successful Mars rover missions,” not three. In reverse chronological order: Curiosity, Opportunity, Spirit, and the one I’ll bet you forgot, the tiny Sojourner. Sojourner wasn’t nearly as ambitious as its successors, but it fulfilled all its mission goals beautifully, especially the proof-of-concept bouncing ball landing.
I stand corrected. Thanks!
That is the absolute best summary of how big projects ‘work’ in IT. I was pretty sure the signup process was going to be a failure before it started and then every piece of info that came out about it, such as multiple contractors or Obama’s campaign IT manager refusing the gig, simply confirmed ‘the banality of the mistakes.’
Having a crack team from Google, Microsoft and Cisco is going to help fer sure (sarcasm).
Indeed. Testing — especially load testing — is almost always sacrificed for the almighty delivery date. Sponsors never learn. I think it was a tech-savvy Santayana who said, “Those who cannot remember the past are condemned to point fingers and retest…or not.”
Another example of “high quality, low price, fast schedule: pick two.”
The schedule was clearly very intense. I assume that the original plan, to let each of the states build its own portal, would have been easier to implement, with fewer moving parts for each state, but too many of the states declined.
The cost, as Bob pointed out (~$1 per American or even ~$10 per American who was not previously insured but will be covered under ACA) was reasonable.
So quality suffered.
Yes, spot on.
I could add that there is a “partisan” issue in all of this though—some players in this government are much more prone to optimism than others, especially about what the government can accomplish by throwing more money at things.
But take that or leave it. I’m not going to back it up with deep analysis.
Clever deflection Bob, but what I said was – your ~defense~ of ‘Bad project execution’ was a partisan. (Re: your quote that the last column was “… -not- ripped from today’s headlines”). The true scandal is that America elected an arrogant and incompetent man, who didnt have any prior executive exerience to recognize the typical basic issues you have identified.
George … Arrogant? Not exactly a fact-based statement. Incompetent? Compared to the administration that preceded it, that’s a tough one to swallow. I’m pretty sure I’m not the one wearing partisan blinders.
Speaking of which, read a bit about the Medicare Part D rollout. Software problems are a bipartisan problem, just as they aren’t restricted to an economic sector.
Very much on point, Bob. Too often we make everything about politics and the failure of government, when this is just a basic failure.
Sadly, as much as you hope that these are just lesson reminders, I can assure you that there are many for whom these are neither lessons nor reminders of lessons — they have yet to get it.
I also like the additions that Robert Harris provided.
-ASB: http://XeeMe.com/AndrewBaker
Hi Bob,
As usual, your analysis is right on. From all outward appearances, this project was driven and decisions mandated by people who had no clue as to what they were doing or signing up for. Enjoyed your analysis very much. Great job.
I don’t know enough of the specifics to offer any analysis-based input on the technical issues involved. Your integration engineering theory could certainly be spot on. However, having been in the middle of a few of these sorts of issues (admittedly only at a large corporatation level), I’d also be curious to see if anyone has gone back verified that there are no duplexing mismatches between the servers and switches. No, I’m not a networking guru. However, over the course of 12 years in an infrastructure organization (and 16 years as a developer before that), I’ve seen a huge number of bizarre performance issues caused by such mismatches, particularly when new systems are involved. And that number is comparable to the quantity of poorly tuned database queries I’ve had to deal with.
I wish I knew who to ask.
Have a great day,
Derek
Excellent observations! I would have added something about scope-creep or requirement changes? As Robert mentions, it sounds like there were some last minute changes – not that the PM was responsible, but it doesn’t sound like there was a very good process for addressing changes. Are you going to have a separate column re: ‘accountability’ and the silliness of Congress holding these hearings? 🙂
As you might remember, I don’t do “holding people accountable.” It’s a popular practice … just the ticket for managers who want the most important facts concealed, enjoy a culture of mutual finger pointing, and who have perfected the circular firing squad.
Whether you are for or against the Act, the reality was that those “in system” who deal with smaller intertwined databases KNEW it would implode–just a matter of time. Enough blame to go around, but the bottom line is that there could not be a realistic test until it went live…and POOF! Another great blurb, Bob!
Hmmm…looks like a Bob Day…Mr. Lewis and Mr. Harris!
Clearly the healthcare.gov problems are due to standard US government policy: the award goes to the lowest bidder. This leads to use of the lowest paid/least qualified staff. Then you can make up for your low bid by “scope changes.” Only, most projects are not this visible.
excellent points. I completely agree about testing… the “we’ll crunch the timeline later in the project” mentality happens all the time. I agree that this was more of a “waterfall” type of project with specific requirements and a deadline, some Agile methodology might have helped. Iterative development of features might have highlighted the design problems earlier. A prototype phase that tested the design might have helped greatly. I’m also curious when this project started. on a lot of projects, the vendor selection and contract signing take such a big chunk of time that the timeline for the work is shortened. Vendors sign off without really understanding the requirements. none of the issues with healthcare.gov surprise me. it would have actually surprised me more if everything went smoothly!
Great article, I appreciated your distinction between Agile and waterfall methods.
But, you don’t give credit to the role that a good Business Analyst plays in discovering, documenting, validating and managing business requirements. You said that Project Managers ‘don’t pass judgment on the specifications, unless they also happen to be software engineers.’, but between the PM and the software engineers is the BA who does review the requirements, analyzes them for consistency, ambiguity, business need, and on and on. Just what value do BA’s bring to the software development lifecycle party? Go and see at http://www.iiba.org/Careers/What-is-Business-Analysis.aspx