Judith Tanur, a statistics professor at Stony Brook University, says surveys can be strange tools, as evidenced by research she conducted years ago on employee satisfaction surveys.
When the survey results reflected employee dissatisfaction, Tanur says, they were probably filled out by people who were taking them seriously. But when the surveys reflected satisfaction, they were probably filled out by people who wanted to avoid the meetings they would have to attend if there was widespread employee dissatisfaction.
In other words, she adds, "high levels of measured satisfaction actually reflect dissatisfaction with the system.
Thanks for reading this article on the outrageous proliferation of surveys in contemporary America.
You know what we're talking about: When you buy a computer, the retailer sends you a survey; so does the computer company. When you order a film or a sundress or a box of fruit online, you are asked to complete a survey. And when you kvetch to your phone/cable/Internet provider, you are asked to stay on the line to take a short survey.
What do all these surveys mean? To help us sort through the surge of surveys, please take just a few minutes to complete the following survey. It will help us do a better job of surveying, um, serving you.
The Survey Of Surveys
1. When did you first notice this trend-gone-wild in which every company you deal with wants to follow up any transaction — even no transaction — with a survey?
a) When I was asked to take a survey by Inhabitots baby/parenting Web site with the chance to win a free floral diaper bag!
b) When the Planetary Society took the consumer-first charge a little too far by surveying people about how many planets they believe are in the solar system (answer "d" was "12 with more to come — the 9 traditional planets plus 2003 UB313, Ceres, Charon and all other round objects orbiting the Sun").
c) When the Restroom Association of Singapore asked consumers a series of questions, including, "What will you do if the toilet seat is dirty?"
d) When I started reading this story and realized it's a survey.
e) When I started reading this survey and realized it's a story.
2. What is your idea of survey nirvana?
a) One of the many Web sites that offer people rewards for taking surveys. "While you won't get rich, doing a few paid surveys can help you make money with little effort," according to Sara Noel, creator of the Frugal Village Web site. She recommends MySurvey.com and Global Test Market, among others.
b) A college-level course such as "Constructing Questionnaires and Surveys" at the University of Pittsburgh, taught by Clement Stone.
c) Questionnaires like the TreeHugger survey from mid-April, which asked: "Do We Run Too Many Surveys?" Sixty percent of 481 respondents checked "No."
d) Do-it-yourself survey sites — like InstantSurvey, Zoomerang and SurveyMonkey — which are used by a slew of organizations, including NPR. For better or worse, now anybody can create countless online surveys. In fact, you are liable to be surveyed about this survey on surveys. Quite meta.
e) Online forums — like those at DoItYourself and Flowerpod — that do not allow users to conduct surveys. At. All.
3) Do you think the potency of surveys is diminished by their overabundance?
a) Absolutely. I agree with Fred Reicheld, author of The Ultimate Question: Driving Good Profits And True Growth, who wrote that "too many surveys are marketing campaigns in disguise" and "surveys confuse transactions with relationships."
b) Yes. Judith Tanur, a statistics professor at Stony Brook University, has it right when she says, "The concept is called 'respondent burden' and often refers to taking too much of a respondent's time in a single survey. But as surveys proliferate, potential respondents are reluctant to contribute increasing amounts of their time. Serious surveys — for example, for the government to collect data that are needed for policy purposes — especially those done by telephone, are suffering drastically declining response rates, probably because of this overabundance. And if response rates are low, surveys become much less dependable tools for reporting on issues like unemployment, health, etc., and thus for making policy. This is a very big and very serious problem."
c) Abundance? What abundance? I echo the poet Phyllis McGinley, who wrote: "Before unpolled generations trample me/ Won't somebody sample me?"
d) No way. I love surveys and concur with the San Diego State University "Blackboard Support" page that reads: "Surveys are great for any type of information gathering where you either don't need or don't want specific names attached to the results."
e) Surveys rock! If CareerBuilder.com didn't survey people to discover that birth order affects job choices, and that 4 in 10 American employees have gained weight at their current jobs, how would I ever know these things?
Customer Surveys Department
In the 1940s — the early days of public surveys — interviews were conducted face to face, says Nancy Mathiowetz, past president of the American Association for Public Opinion Research and editor of Public Opinion Quarterly. The process "was very expensive and took weeks, months to plan and collect the data."
In the 1970s, surveyors moved to telephone data collection. "Quicker but still costly," Mathiowetz says. Then came the Internet, interactive voice recognition and other methods of collecting data that involve less cost and quicker turnaround times for corporations thirsty for consumer information.
There are two salient reasons for the burgeoning survey industry, she says. First, the corporate desire to make empirically based decisions "requires the collection of data to examine satisfaction, how people view products, etc. Second, the marginal cost of data collection — for some modes of data collection — has dropped."
But in the end, does an organization actually use the information it gleans from a survey, or is the survey an end in itself, a signal to the customer that a company cares?
"It's sometimes one and sometimes the other and sometimes both," says Tanur. For example, "research I did years ago suggested that employee satisfaction surveys are strange tools."
When the survey results reflected employee dissatisfaction, Tanur says, they were probably filled out by people who were taking them seriously. But when the surveys reflected satisfaction, they were probably filled out by people who wanted to avoid the meetings they would have to attend if there was widespread employee dissatisfaction. In other words, she adds, "high levels of measured satisfaction actually reflect dissatisfaction with the system."
The challenge, says Mathiowetz, is that people "need to be able to sort through the good, the bad and the ugly" surveys. "The survey and opinion research profession, as well as the media, needs to do a better job of helping the public understand the differences between junk data and high-quality surveys."
The Mother Of All Surveys
Some surveys are essential, Mathiowetz says. "It's important for a democracy to have a vehicle for the general public to express their concerns, as well as for our policymakers to hear from the full population."
She is referring to surveys conducted for the public good, which includes the mother of all surveys: the decennial national census. The next one is scheduled for 2010. Some 144,000 census workers will be surveying Americans across the country.
Without the U.S. census — coming soon to a front door near you — and the vast array of social and government surveys, Mathiowetz says, "we know little about ourselves as a population."
So while she is concerned that people are bombarded with requests for all kinds of information, she says some surveys can be meaningful. "We do need that information to know the percent of the population that is unemployed, without health insurance, living below poverty."