The Question Stock & 2 Smoking Barrels (Devasheesh Nautiyal Group_A2)
Of all the things that was discussed in class regarding creation of a questionnaire, I believe it cannot be seen in isolation with respect to its purpose (provide insight). There are a lot of factors that affect the reliability and validity of the inference, out of many references that I came across, the following two caught my eye. Though seemingly obvious, the outcome can be startling.
Barrel#1: Time Lapse:
Some surveys are conducted over a long period of time, mostly in multiple iterations with considerable time in between the first and the last survey. As per the ongoing craze around incremental improvements in all the facets of industry, the surveys are also brought up to speed with time. The learning that came out of it is incredible.
Above is the visual representation for religious tolerance of countries all over the world with red marking the most intolerant countries and vice versa. So what could be wrong about this survey if you did not see it yet! Though I am still trying to find out what went wrong for India, a Bangladeshi National apparently was able to decipher the cause for the ‘Red Bangladesh’.
What had happened was, to gauge religious tolerance, the survey was using 0 for tolerance and 1 as intolerance in 1990s. During the process of ‘improving’ the survey, the value references were swapped for the Bangladesh survey (1 for tolerance and 0 as intolerance) leading to the erroneous interpretation.
Barrel #2: Mr. Who!!
Who conducts the survey holds direct connect to how valid the response is or/and how fierce the backlash to the erroneous survey will be, for the conductor and the organizer.
Case and Point, a survey the New York Times wanted to conduct in the prestigious Yale University campus. The responsibility was given to the Yale College Council (YCC), who in their own right decided to conduct the survey among themselves (28 in all). The conduct was widely criticized on part of both the parties, YCC not considering a random sample and NYT publishing the discovery made from the data provided by YCC. Here the question is not about sampling, but of the reluctance on the part of the conductors and the organisers.
The next and the most frequently experienced surveyor based issue is of when she tries to influence our ratings most probably for improving her overall ratings. For an average performance, demands are made for ‘Rockstar Ratings’ which speaks volumes about itself. As it happened to a witness
“Some years ago when I would take my car to the dealer for service. The same conversation ensued each time I returned to pick up the car. The clerk whom I paid for the service would shove a paper survey at me and say “Please complete our service satisfaction survey” suggesting with her body language that my keys would be held hostage until I finished the thing. “But,” I would protest, “How do I know whether I am satisfied or not until I have driven the car for a while?” Impervious to my logic, she would shrug and tell me that she needed it done now because that was their process, and if I didn’t know how satisfied I was, I should guess.”
As we saw above a good questionnaire, apart from being good should also be consistent and conducted by responsible and accountable. Something very important but seemingly obvious enough not to be noticed before the trigger has been pulled!!!