.png)
Councilwoman Kitty Brown is concerned that the mail survey will not incorporate enough of the public and would like a companion survey to be published in the New Paltz Times.
While I admire Kitty Brown’s desire to broaden inclusion, in this instance, as a rigid methodologist when it comes to survey research*, I have to advise the town to not take this approach. In order for the science of the mail survey to be sound, the first and only exposure to the questions needs to happen when people receive them in the mail. It needs to be an independent, stand alone document, one that is especially not embedded within a newspaper that is currently reporting on the issues it addresses. The intent of including a survey in the project was to provide a scientifically valid sample of the town residents and offering the survey via the newspaper diminishes the validity of the mail survey’s results.
That said, I also have some serious concerns about the mail survey methodology. Done well, random sampling methods include contact with a small number of people, the results of which can represent the entire population under study. The answers obtained from a scientific probability survey are not just answers from those individuals who responded but more importantly, because of the design and methods by which the data is collected, can be used to generalize to the population as a whole. We want a methodology that ensures results are an estimate of what would have been obtained if all adults in the New Paltz were interviewed.
Firms typically chose to use mail surveys over telephone surveys because of the significant difference in cost. (A sound telephone survey would cut into at least half the budget of this entire project.) But, the trade off for lower cost is that mail surveys have notoriously low response rates, making the case for representativeness a tough sell. The consultants estimate a return rate of 8% after mailing out the survey to 1000 randomly selected households. How do we know the responses of these 80 residents represent the views of all New Paltzians? We can’t know, the response rate is too low and the sample size too small to justify the science of random sampling. (Which btw, assumes 100% response rate, but lower rates have proven sufficient, just not that low). My advice is to mail out more surveys, possibly staggered in waves of 500 over time, in order to generate 300 interviews with a margin of error of +/- 6%.
Then what? Population parameters provided by the U.S. Census can be compared to the demographics of the survey sample to ensure representativeness. If it is close but not quite close enough, a statistical process called weighting can be employed, but this should only be done if guided by strict rules… in lay people’s terms, it should only “tweak” the data, not stretch the truth, so to speak. In the end, when we review the data, we must ask: do the demographics reflect our population, as we know New Paltz to be, based on census data? 82% white, 48% college educated, 54% homeowners, 28% households with children? If the composition of the resulting sample is similar to the make up of our community based on census data, we can be confident the survey yielded information that can be generalized to the entire population of New Paltz.
kt
* My M.S. degree is in Social Research and just last year, after eleven years, I left my job as Assistant Director at the Marist Institute for Public Opinion
4 comments:
Clearly this is your forte, KT. I always get nervous about formal surveys, because they seem to spend so much time on making sure the sample is representative that to the rest of us it seems like the sample is being manipulated. Frankly, when you talk about statistical modeling it sounds like making up answers to me.
Well, that's not really true. It's more like going to the auto mechanic. Cars are sufficiently complex that most of us only understand how they work in part - hence the need for specialists in the field. However, a certain level of trust in your mechanic is needed, because he can tell you things that you're just going to have to take his word on. If a survey is developed that's representative, we can always ask, metaphorically at least, to see the parts that were replaced, but when it comes down to it we have to trust in people with more expertise in the area.
For auto repairs I trust Bill's Garage on North Chestnut. For survey analysis, I'll put my money on KT to spot any hanky-panky. I haven't tried anyone else in either case, but I'm okay with that.
There are other issues with the survey. One is why money is being spent designing, mailing, and analyzing it (unscientific sample notwithstanding) when almost all the questions it would ask have already been asked in at least two recent town-wide surveys -- the open space survey and the Transportation and Land Use survey. Now we're paying more consultants to rephrase questions we've already received and answered (at great expense) at least twice, and worse yet, as in both of those cases, the data will ultimately be weighted very low in influencing the final policy advice, largely because of its unscientific nature and low response rate (not to mention that some elected officials will constantly insist that there's some "silent majority" out there that is much larger than the response pool that wants shopping malls, and this survey method won't allow us to definitively declare that incorrect).
Another issue is the questions themselves. I went to two of the forums. All the questions asked at both of them were development questions, not truly planning questions. Curiously, in a town that's littered with vast tracts of abandoned farm land, and where the number of CSA's has increased from 2 to 5 just in the last few years, and in economic and environmental circumstances in which long-haul trucking of produce is decidedly NOT the way of the future, the question of restoration of an agricultural economy (which would preserve open space, as well) never came up -- only residential, retail, and industrial development.
I continue to have concerns about the purpose and expense of paid planning consultants -- especially when they come from Buffalo. We have such incredible human resources to draw upon right here at home -- enginees, lawyers, educators, farmers, business associations, and families that have lived here since the late 1600's. Why did we hire consultants from Buffalo without first forming a local blue-ribbon panel? Who can show that we're going to come out ahead on the expense, or get a plan that suits this community?
Isn't the money really being spent just so our elected officials can claim they got expert advice?
Correction: Toni Hokanson is not in favor of a newspaper survey. The statistical integrity of the survey must be preserved. Any other survey results could not be mixed with the "official" mailed survey. I do not believe the public should be misled into thinking that the newspaper survey counts. Toni
Kitty Brown always seems to miss the point. The survey should be to collect public ideas and suggestions for the future of the community. Planning for the future should not be about a popularity contest for certain views, but a process to consider the best ideas. What would Kitty have to say if a newspaper poll called for extensive new commercial development? She would would likely cry foul and say the results are not scientific. Don't call the thing a poll at all Kitty, call it a solicitation for ideas and you may get it right this time!
Post a Comment