LinuxPlanet - Print - Getting the Real Facts: How Industry Analyst Reports Can Trick Readers

Getting the Real Facts: How Industry Analyst Reports Can Trick Readers

By: Maria Winslow
Monday, November 14, 2005 10:12:15 AM EST
URL: http://www.linuxplanet.com/linuxplanet/reports/6078/1/

Start from the Beginning

Microsoft's "Get the Facts" advertising campaign makes the claim that Windows offers a lower total cost of ownership (TCO) than Linux, and backs it up with reports from well-known industry analyst firms. But Linux advocates claim that the TCO of Linux is lower, and some other studies back them up. How can you separate the fact from the fiction? In Part 2 of this series, I'll offer tips on how to read a study and make up your own mind about its relevance. Next month, in Part 3, I'll analyze specific studies referenced by Microsoft in their "Get the Facts" campaign.

Various studies on the total cost of ownership (TCO) of Windows versus Linux have arrived at vastly different conclusions. How is this possible? The short answer is that if you are abstract enough with your goals and methodology, are selective with the costs that you include, and ask the right questions, then you can arrive at any conclusion you want.

Here are some of the things you need to consider as you read a report.

First, read the whole report.

A lot of people make decisions based on press releases or summaries of analyst studies, but these brief sources don't provide enough information. Tedious as it may sometimes be, you must read the whole study with a critical eye in order to determine the relevance of the advice to your organization. In fact, you may need to read the report at least twice in order to judge it properly. Skim it first for a "common sense" test, then delve in with the following issues in mind.

Are the Assumptions Realistic?

Some studies take a premise, form it into a question, then report on how the surveyed organizations responded. That's a good idea, but only if the assumptions make sense for the real world. Your first task while reading the study is to assess the usefulness of the premise. If it's not realistic for your situation, then discount the study.

One of the surveys referenced by Microsoft, for example, asked participants about the wholesale replacement of Windows systems with Linux. But no one indiscriminately replaces one technology with another one wholesale. It was an unrealistic question that produced a meaningless and misleading result.

Lesson: abstract assumptions do not translate well to the real world.

Was the Study Based on the Actual Experience of Respondees?

It also makes a big difference whether the data was collected from the real-world experiences of interviewees or from their opinion. When analysts ask survey particpants about a hypothetical situation, the answers they get can only describe perceptions on the topic, not reality on the ground. For example, while it is a common perception that Linux administration is more expensive than Windows administration, the actual reliable data suggests otherwise. Therefore a survey asking participants without significant Linux deployments about this topic will yield a suspect result.

As you read the questions and responses of a survey, think about whether the answers are based on speculation or experience. Speculation may be interesting--it can be a good barometer of industry-wide thought on a topic--but it doesn't give you an accurate picture of what's happening in real-world settings.

Lesson: real-world data is more valuable than opinion.

Did the Study Include All Relevant Costs?

Any study of TCO must include all relevant costs in order to be reliable. Go over the numbers for the studies that examine cost issues to make sure they look realistic. For either return on investment (ROI) or TCO, make sure the study authors covered all the bases. For a brief refresher on TCO and what should be included, see the first article in this series, Getting the Real Facts: How to Read an Analyst's Report.

One of the Microsoft-sponsored studies actually omitted a good number of client access licenses (CALs), which are typically a large portion of the licensing cost. This is a serious oversight, resulting in a misleadingly low cost for the Windows deployment.

Whether intentional or due to oversight, omissions of related costs in ROI and TCO studies are inexcusable and will yield warped findings.

Lesson: omissions can make the data unrealistic in practice.

Is the Methodology Clear?

The methodology of the analysts should be apparent as you read the study. In general, a study should seem reproducible given the descriptions by the authors. Many studies will even include a section on methodology. A vague or confusing description of the techniques they used to arrive at their results is not a good sign.

Survey authors should indicate who they interviewed, including position and company size. The primary questions should also be provided. Studies involving cost should break down those costs into itemized lists. After reading the study, you should have a good idea of how the authors collected and analyzed the data. If not, then nothing was proved.

Lesson: If you don't know how the authors collected and analyzed the data, you can't be sure it's valid.

Who Sponsored the Study?

And finally, it really does matter who sponsored the study. Any reputable analyst firm will discose the funding source, if there is one. Naturally, any study sponsored by a company that shows its products in a favorable light must be considered less reliable than any independent study. This point is just plain common sense.

Lesson: vendor-sponsored reports can be biased.

What to Look For in a Study

So how do you know who to believe? If you're going to follow the advice in an analyst report, then read the document and ask yourself these questions:

Who sponsored the study?
Were the interviews based on experience or guessing?
Was there actual data from real-world settings?
Are cost estimates realistic and inclusive?
Do the authors sufficiently describe their methodology so that I can understand how they arrived at the data?

If you're going to follow advice from analysts then read the reports and look for signs of solid methodology and independence in the research. Think about the advice you are being offered, and decide for yourself if it's helpful or not. There is a wide discrepancy in the available analysis.

Next month, I'll do this work for you as I investigate the details of the studies referenced in Microsoft's "Get the Facts" advertising campaign.

About the author: As an open source practice leader with Virtuas, Maria Winslow assists clients in understanding the technical and budgetary impact open source software will have on their computing environments. Her recent book, "The Practical Manager's Guide to Open Source", guides IT directors and system administrators through the process of finding practical uses for open source that will integrate seamlessly into existing infrastructures, as well as understanding the costs and savings. Ms. Winslow is a frequent speaker and author on the topic of open source, and is a contributing editor of open source applications at LinuxPlanet and Linux Today. You can reach her at maria.winslow at windows-linux dot com.