Pro-gun survey provides textbook example of 'poll’ designed to elicit specific response
In an attempt to counter the impact of two public opinion surveys that indicate New Hampshire voters support maintaining the current license requirement for carrying a concealed handgun in public by a 3 to 1 margin, pro-gun advocates have released their own poll that purports to show Granite Staters oppose the requirement.
The survey, commissioned by the Women’s Defense League of New Hampshire, provides a textbook example of how to design a poll guaranteed to elicit a specific response.
The two earlier surveys were administered by experienced, national pollsters. Public Policy Polling surveyed 747 New Hampshire voters in late April and found Granite Staters support the license requirement by a 71% to 23% margin. A subsequent poll by SurveyUSA measured support at an almost identical 73% to 24% margin.
The Defense League poll was apparently conducted by the Women’s Independent Majority Polling Project, a New Hampshire-based organization that registered with the Secretary of State’s office just two weeks ago. The registration was filed by conservative activist Susan Olsen, who is also on the Women’s Defense League board of directors. Emily Sandblade, the Free Stater and former State Representative who presented the survey results, identified herself as the group’s statistician.
We had some questions about poll’s methodology. For example, there’s no breakdown of responses by age groups. But frankly, after seeing the wording of the survey questions, details of the survey methodology are fairly irrelevant.
Biased questions poison everything that comes below
The Pew Research Center describes “order effects” which create survey bias. “Researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions – in particular those directly preceding other questions – can provide context for the questions that follow.”
The Defense League survey began with a question on self-defense. “Do you support or oppose the right of a person to defend herself with a firearm in a dangerous situation?” This question, intentionally or not, frames the issue of concealed carry in terms of using of a firearm for protection rather than to commit a crime. That bias provides a context for the questions that follow.
As polling guru Nate Silver notes, “when you ask biased questions first, they are infectious, potentially poisoning everything that comes below.”
But it gets worse. Before asking the question to measure support for eliminating the concealed carry requirement, the Defense League survey also included statements declaring, “Vermont has never required a concealed carry license, and yet their crime rate is lower than New Hampshire's” and “The state law governing concealed carry permits was set up to bar union member Irish and Italian immigrants and their families from possessing a concealed firearm.”
Does the question seem fair and unbiased?
The concealed carry questions from PPP and SurveyUSA were simple and straightforward. PPP asked, “Do you support or oppose allowing people to carry a concealed gun without a license?”
SurveyUSA added some context. “Under current New Hampshire law, people are required to have a permit to carry a concealed handgun in public places. In order to get a permit, people need to offer three character references and pay a $10 processing fee. Do you … strongly support? Support? Oppose? Or strongly oppose … requiring permits to carry a concealed handgun in public in New Hampshire?”
Contrast the wording of those questions with the wording of the Defense League question. “Do you support or oppose the ability of local police to make decisions about issuing permits based on their judgment of suitability – even when such decisions can be based on political, union membership, gender, or racial bias?” they asked.
“Perhaps the best test of any poll question is your reaction to it,” the National Council on Public Polls notes. “On the face of it, does the question seem fair and unbiased? Does it present a balanced set of choices? … You should examine carefully both the results and the exact wording of the questions.” Indeed.