All Posts in Leadership

|In: Industry Commentary|Last Updated:

“We have to plan for the future,” a systems executive told me recently. Well yes, that’s probably far more useful than planning for the past or present. It is, however, much harder.

Two weeks ago we explored the subject of worthless statistics and the odd preference so many people have for bad information over unbiased ignorance. (See “Pie charts and bar charts may bring comfort, but wisdom is another matter,” May 5, page 74.)

This week we talk about forecasting, an activity that fulfills the need many people have to know and plan for the future. Throughout history that need has created professions and industries. In the earliest days, the tools of the forecaster’s trade were tea leaves, entrails, the Zodiac, and crystal balls. More recently, tools like Doppler radar and computer simulations have been added to the arsenal.

Forecasting hasn’t improved much, but forecasting tools sure have. (A recent article in the local newspaper compared the performance of the National Weather Service, television meteorologists, and a guy who predicts the weather using a combination of Radio Shack gear and homemade tools. The backyard hobbyist won by a wide margin.)

And that brings us to the most pernicious forecasting tool yet devised: the market survey. How many articles have you read in this and other reputable journals presenting some industry forecast or other in which the future is presented as a fact: “Within five years, the market for bubble memory will grow to over $5 billion”?

Oops. Bubble memory flopped. It must have been personal digital assistants I was thinking of.

The accuracy of both market and weather forecasting begins to plummet beyond about one day into the future for the same reason: The systems we’re trying to predict are chaotic. And although the mathematics of chaos theory are daunting, one of its basic conclusions is pretty straightforward: When systems (actually, nonlinear systems) reach a certain level of complexity you can’t predict their future state very well.

Now what’s involved in predicting a future market? Oh, just the behavior of large numbers of individual humans. If someone would just invent Isaac Asimov’s psychohistory we could get somewhere.

Market surveys seem so scientific. But what are they really asking? “Do you expect to implement Windows NT Advanced Server in your enterprise within the next three years?”

Now who’s going to answer with an unequivocal “no?” We’re talking about a Microsoft product and a three-year planning horizon. And what does “implement” mean? Replace Novell’s NetWare? Or install a test server? Forecasts of NT Advanced Server sales have no value because there are unpredictable factors involved: Will Microsoft ship the next release on time? Will it have any crippling bugs? Will Bill Gates burn out, causing all of Microsoft’s customers to lose confidence? Will the sun turn nova and wipe out my capital-procurement budget?

Not that I want to single out NT Advanced Server. Whether it’s network computers or wireless data communication, any innovative technology just entering the market faces too many imponderables for forecasts to mean much. Why? Because market success depends as much on unanticipated details as on the basic ideas. Is the advertising campaign exceptionally stupid? Has innovation extended the life of a moribund technology? Did the designer misread the market?

So don’t let forecasts paraded as facts serve as a substitute for your own insight. Except, of course, for the enlightened predictions of your friendly IS Survival Guide soothsayer.

Satire alert

A few weeks ago, I expressed some concern about the Software Publishers Association’s and the Business Software Alliance’s admission that they inflate their software piracy estimates. (See “The Romans had some words for it,” April 7.) Some alert InfoWorld readers told me that the admission was a joke, originally published in the San Jose Mercury News and later picked up as fact elsewhere. By the time I read it in Edupage, the humor had been removed and it looked legitimate.

So here’s the situation: The numbers are still suspect for all the reasons expressed in the satire, but the SPA and BSA haven’t admitted it after all.

Comments

|In: Leadership|Last Updated:

Evolutionary theory has to account for all the bizarre complexity of the natural world: the tail feathers of peacocks; the mating rituals of praying mantises; the popularity of Beavis and Butthead. One interesting question: Why do prey animals herd?

Herds are easy targets for predators. So why do animals join them?

One ingenious theory has it that even though the herd as a whole makes an easy target, each individual member is less likely to get eaten – they can hide behind the herd. One critter – usually old or infirm – gets eaten and the rest escape. When you’re solitary, your risk goes up.

Predators hunt in packs for entirely different reasons. Human beings, as omnivores, appear to have the instincts of both predators and prey: We hunt in packs, herd when in danger.

Which explains the popularity of “research reports” showing how many of our peers are adopting some technology or other. These reports show us how big our herd is and where it seems to be going. Infused with this knowledge we can stay in the middle of our herd, safely out of trouble.

And so it was that I found myself reading an “executive report” last week with several dozen bar charts. A typical chart segmented respondents into five categories, and showed how many of the twenty or so “yes” responses fell into each one.

Academic journals impose a discipline – peer review – which usually catches egregious statistical nonsense. But while academic publication requires peer review, business publication requires only a printing press.

Which lead to this report’s distribution to a large number of CIOs. I wonder how many of them looked at the bar charts, murmured, “No error bars,” to themselves, and tossed this information-free report into the trash.

We read over and over again about information glut. I sometimes wonder if what we really have is nonsense glut, with no more actual new information each year than a century ago.

Bar charts without error bars – those pesky black lines that show how uncertain we are about each bar’s true value – are only one symptom of the larger epidemic. We’re inundated with nonsense because we not only tolerate it, we embrace it.

Don’t believe me? Here’s a question: faced with a report like this and a critique by one of your analysts pointing out its deficiencies, would you say, “Thanks for the analysis,” as you shred the offending pages, or would you say, “Well, any information is better than none at all.”

Thomas Jefferson once said, “Ignorance is preferable to error,” and as usual, Tom is worth listening to. Next time you’re faced with some analysis or other take the time to read it critically. Look for sample sizes so small that comparisons are meaningless, like the bar charts I’ve been complaining about.

Also look for leading questions, like, “Would you prefer a delicious, flame-broiled hamburger, or a greasy, nasty looking fried chunk of cow?” (If your source has an axe to grind and doesn’t tell you the exact question asked, you can be pretty sure of the phrasing.)

Look for graphs presenting “data” with no hint as to how items were scored. How many graphs have you seen that divide the known universe into quadrants? You know the ones: every company is given a dot, the dots are all over the landscape, the upper right quadrant is “good”, and you have no clue why each dot landed where it did because the two axes both represent matters of opinion (“vendor stability” or “industry presence”).

Readers David Cassell and Tony Olsen, both statisticians, recently acquainted me with two measures, Data Density, and the Data-Ink Ratio, from Edward Tufte’s wonderful book, The Visual Display of Quantitative Information:.

To calculate the Data Density divide the number of data points by the total graph area. You express the result in dpsi – data per square inch.

You calculate the Data-Ink Ratio by dividing the amount of ink used to display non-redundant data by the total ink used to print the graph. Use care when scraping the ink off the page – one sneeze and you’re out of luck.

Comments