Councilwoman Kitty Brown is concerned that the mail survey will not incorporate enough of the public and would like a companion survey to be published in the New Paltz Times.
While I admire Kitty Brown’s desire to broaden inclusion, in this instance, as a rigid methodologist when it comes to survey research*, I have to advise the town to not take this approach. In order for the science of the mail survey to be sound, the first and only exposure to the questions needs to happen when people receive them in the mail. It needs to be an independent, stand alone document, one that is especially not embedded within a newspaper that is currently reporting on the issues it addresses. The intent of including a survey in the project was to provide a scientifically valid sample of the town residents and offering the survey via the newspaper diminishes the validity of the mail survey’s results.
That said, I also have some serious concerns about the mail survey methodology. Done well, random sampling methods include contact with a small number of people, the results of which can represent the entire population under study. The answers obtained from a scientific probability survey are not just answers from those individuals who responded but more importantly, because of the design and methods by which the data is collected, can be used to generalize to the population as a whole. We want a methodology that ensures results are an estimate of what would have been obtained if all adults in the New Paltz were interviewed.
Firms typically chose to use mail surveys over telephone surveys because of the significant difference in cost. (A sound telephone survey would cut into at least half the budget of this entire project.) But, the trade off for lower cost is that mail surveys have notoriously low response rates, making the case for representativeness a tough sell. The consultants estimate a return rate of 8% after mailing out the survey to 1000 randomly selected households. How do we know the responses of these 80 residents represent the views of all New Paltzians? We can’t know, the response rate is too low and the sample size too small to justify the science of random sampling. (Which btw, assumes 100% response rate, but lower rates have proven sufficient, just not that low). My advice is to mail out more surveys, possibly staggered in waves of 500 over time, in order to generate 300 interviews with a margin of error of +/- 6%.
Then what? Population parameters provided by the U.S. Census can be compared to the demographics of the survey sample to ensure representativeness. If it is close but not quite close enough, a statistical process called weighting can be employed, but this should only be done if guided by strict rules… in lay people’s terms, it should only “tweak” the data, not stretch the truth, so to speak. In the end, when we review the data, we must ask: do the demographics reflect our population, as we know New Paltz to be, based on census data? 82% white, 48% college educated, 54% homeowners, 28% households with children? If the composition of the resulting sample is similar to the make up of our community based on census data, we can be confident the survey yielded information that can be generalized to the entire population of New Paltz.
* My M.S. degree is in Social Research and just last year, after eleven years, I left my job as Assistant Director at the Marist Institute for Public Opinion