The Trouble with Market Research

What I found when I dug deeper into this report gives a good illustration as to why you should be wary before you spend thousands of dollars on these reports. I won’t burden you with too many details except to say that the analyst firm made a big deal of the 900 respondents who were surveyed in the report. What the firm didn’t say was that the respondents weren’t randomly selected – they were invited to opt in to the survey, which is another way of saying they were self-selected.

Now, you see a lot of surveys on the Internet, and the vast majority of what I’ve seen are just that kind. You are asked if you’d like to participate in a survey when you’re visiting a website of some sort. While these surveys can produce useful information, especially if they’re asking visitors to the site whether they found what they wanted, or whether using the site was easy enough, that’s not the same thing as being a statistically valid survey. To be valid, participants must be randomly chosen. While there can be some limits on the randomness – for example, you can limit your questions only to people of a specific gender or people who smoke or drive cars – you can’t base a scientific survey on people who volunteer to take part.

Unfortunately, actually finding out this information can be difficult. You might see some kind of attention-grabbing headline that highlights a fact in the alleged survey, and then offers to sell you the complete report for several thousand dollars. It’s only when you buy the report that you find out that you’ve been misled. The study doesn’t necessarily present accurate survey data, but rather something else that fits the needs of the company presenting the study.

Fortunately, these companies, if they’re even slightly reliable, will let you reach out to the researchers and ask them. Usually, they’ll tell you. In the study I reference above, the principal researcher admitted that the age of 55 just sort of came out in a meeting. He also admitted that the survey population was self-selected. In other words, the survey wasn’t performed scientifically, and the study population’s demographics were chosen on a whim.

The problem with this sort of study is that analyst firms target companies that need good marketing data, and need it quickly so they can react to a rapidly changing marketplace. If a study is offered that seems to reflect on something you’re trying to find out, there’s a strong temptation to spend the money and use the information in the study to guide your business decisions. But do you want to base your decisions on flawed data? Probably not.

In reality, it never pays to move too quickly on these offers. When any research firm presents an unsolicited offer, it pays to spend a couple of days verifying the information, and the methodology, before you buy. While it’s possible that weak methodology may still produce useful results, at least you should know before you spend the money. Then, when you base your decisions on the data, you’ll know where the holes are.