Notes on the JMI/zinc tank polling experiment
2017-05-05
Vancouver B.C. (5 May 2017) – Yesterday, we reported that the BC Liberals appear to have secured a lead in the race to form the next provincial government. In that poll, the BC Liberals were the choice of 39% of decided voters; the BC NDP held 34% and the BC Greens, 23%.
How confident are we that this will be the popular outcome?
We are confident that we captured public opinion as of our May 1 and May 2 online and telephone polls. But we caveated that this is not a prediction of the popular outcome of the May 9 election. We expect the race will narrow as election day approaches.
Some Green supporters will reconsider their Green support and select the party that has the potential to keep the bad guys out.
The undecideds who will in fact vote will firm up their decision and, if the prediction of one aggregator is correct, lean toward “the safe incumbent”. While our recipes differ, we (and at least one aggregator) now favour the BC Liberals. This is as much about the number of reasonably safe BC Liberal seats as it is about what B.C.’s decided voters tell us.
You’ll recall in 2013, pollsters favoured the BC NDP, who enjoyed a huge advantage in the polls leading to election day. Despite this advantage, the BC NDP did not form the government. Did the pollsters get it wrong or did voters change their minds in the final moments of the campaign? Might it have been a bit of both? Both online and telephone polling in 2013 pointed to a popular victory for the NDP. (See the polling and actual results here.)
In post-election debates about methodology, the polling industry was trounced while pollsters who didn’t publicly share their findings publicly congratulated themselves for predicting the BC Liberal win. Pundits ridiculed online polling and the pollsters ignoring the many telephone polls that also failed to predict an NDP win.
In 2017, we are favour our telephone work over our online. Why?
By comparing results of telephone and online modes separately, we observe some glaring differences in the character and preferences of online and telephone respondents. Hence, this BC election polling experiment.
THE EXPERIMENT CONTINUES
In our 2017 experiment, we added automatic telephone surveys to the mix in our third measure and combined the findings in our report.
What if we had reported on just the online component?
The statistical tie noted in late April converts back to an advantage for the BC NDP, although the NDP support level was up only one point (from 37% to 38%). BC Liberal support fell from 38% to 34%. Even so, neither the difference in support levels between the two parties, nor the BC Liberal decline would fall within the (5 pt) error rate of a pure probability population samples (which this research is not).
A look at online results
The online sample still finds the BC Liberals in the lead among men (42%), while women lean toward supporting the BC NDP (42%). And some of the expected age-related trends are evident. Voters under 35 years are most likely to support the BC NDP (41%). Voters aged 35 to 54 are divided with about equal numbers casting their support for the BC NDP and the BC Liberals (37% and 35%).
Regionally our online polling finds the BC NDP heavily favoured in City of Vancouver, the Victoria area, the rest of Vancouver Island and Gulf Islands (41%, 41% and 51%). The BC Greens run a very close second to the BC NDP in the Victoria area.
The online polling also introduces potential anomalies.
In the North, we find the BC Greens sharing the lead with the more typical lead, the BC Liberals (41% and 37%), while the BC NDP garner just 10%. Similarly, in the Southern Interior, our polling this wave may have overrepresented the BC NDP, which shares the lead with the BC Liberals (at 35%).
A look at phone results
The phone sample finds the BC Liberals lead among men and women (41% and 40%). Among those reached by phone who are under 35 years, the BC NDP and BC Greens are tied (at 36%). The BC Liberals hold commanding leads among older cohorts (35-54 year olds: 44%; 55+: 49%).
City of Vancouver favours the BC NDP (at 46%). The balance of Metro Vancouver favours the BC Liberals (45%). The Victoria area heavily favours the BC Greens (41%), followed by the BC NDP (at 34%). The balance of Vancouver Island (and Gulf Islands) favours the BC NDP (at 35%), followed by the BC Greens and BC Liberals in a dead heat (at 30% and 29%). The North heavily favours the BC Liberals (at 52%). The Southern Interior and Coast region favours the BC Liberals followed by the BC NDP and the BC Greens (at 27% and 23%).
OUR CONCLUSION
Here, at a glance, is a comparison of the final results in total, online and by telephone.
We are advocates of online polling. We get our predictions right most of the time. For example, our City of Vancouver municipal predictions nailed the outcomes for mayor and council in 2014 and mayor 2011. We’ve had similar success for private clients. But in 2013 something went wrong for us and for every pollster who had the courage to publish.
We were early adopters of mixed-mode designs. Conventional wisdom points to online’s benefit in gathering data from people who would never be reached using telephone research. The same argument in reverse points to the benefit of telephone research in gathering the opinions of people who would never join a research panel or complete an online survey.
The phone results from our May 1 and 2 polling are more in line with expected demographics and the regional distributions of the actual 2013 election outcome. The population reached in telephone polling is more typical of the public that are also votes. For these reasons, along with some anomalous results in this wave’s regional results, we favour our telephone research. That is not to say the online findings are not to be considered. The online findings have picked up on meaningful trends, like steadily growing support for the BC Greens and tumultuous times in BC Liberal support, which appear to have been affected by some third-party negative campaigning and media revelations.
In marketing and public affairs research applications, the mixed mode approach works well. But in polling research, the biases inherent in telephone research are also the biases we see on election day. Are we over-representing some demographic groups’ party preference in our online research? We don’t think so. But we are over-representing non-voters. Some groups of “decided voters”, who complete a survey online, may not actually show up to vote. Much works remain to reform and update a now dated voting experience. Much work also remains among pollsters who must carry on working to find a more effective way to identify decided vote.
Share. Discuss.
For more information contact: Barb Justason, Justason Market Intelligence Inc.+1 604 783 4165 / Barb@JustasonMI.com / JMI.co Brian Singh, zinc tank +1 403 861 9462 / brian@zinc-tank.com / zinc-tank.com – 30 –
