Handle With Care

Some polls zero in on what customers want, but others focus on what a company wants to build. You need to know what kind of survey it is and the orientation of the company that commissioned the survey before evaluating the strength of its findings. The former could be useful; the latter could be disastrous if applied to your own product efforts.

Poll results often conflict. One may say that boomers are not saving enough for retirement while another says boomers are saving more than you think for retirement. Which is it? If the poll samples differed sharply, it could be both are right. The devil is in the details, so you better know those details if you're planning to use the results to support a product initiative.

Poll result analyses are of uneven quality. Some state the most positive information first, and downplay or even omit the negative information, while others give the pro and con fair play. Sometimes you don't know which is which unless you see the original report, which is hard to do if you don't work for the company that commissioned it and/or if you don't want to pay for it. Fortunately, some research firms do provide the original to those who ask for it (such as news outlets). Even so, if you only see a summary from someone who read the original, you still don't know the whole story. If you're making product decisions based on the findings, get the original.

Some poll reports do not show the research parameters. Who was sampled? How? When? Where? What is the margin of error? etc. Reputable research firms routinely spell this out, but poll result summaries don't always include it. Also, some reports bury this in the footnotes or conclusion, making you hunt for it–and making you wonder what else to hunt for.

Poll analyses often include commentary from the sponsoring organization but not from independent outsiders. The commentary can be interesting, but may lack balance. Look for outside opinion too.

Not everyone who sees the results understands them the same way. When the results are swept into product discussions, this can make for healthy debate or destructive disputes. To avoid the later, use reports that define not only parameters, but also mission, terminology and other clarifying details.

Poll results get misquoted, distorted and incorrectly sourced. This often happens after the data gets some buzz. When someone cites "a survey I read" to support a product point, it's best to ask for the source of the survey so you can check it yourself.

Despite all this, polls and surveys are mission critical to informing decision makers in product development, rollout and sales. When well executed and reported, they provide telling insights to consumer needs, competitor strength, target markets and more. They spark thoughts and ideas. However, not all that glitters in poll/survey reports is gold, so … handle with care.