What Would We Do Without Pollsters?

80 Replies to “What Would We Do Without Pollsters?”

1. Nice….and very predictable.
   Hmmm, I wonder how many other polls are sneakily done like this?

2. Move over Decima, you have competition.

3. “That said, it’s important to note that the gap between the Conservatives and Liberals has narrowed quite a bit.”
   There’s the money shot. Predetermined meme; the poll was just “fill in the numbers”.
   Pure fantasy.

4. *sigh*
   Do you know why oversampling is done? To eliminate possible bias. Do you know how the overall figures are calculated? By pro-rating the oversampling back (weighting it down) when the percentages from each province/region are obtained and combined.

5. I think I’ll undersample CTV from now on.

6. Hmm… I wonder if polling firms regularly over-sample certain geographic regions so as to get results they want?
   If one will do it…
   Fancy math… like the IPCC computer program, perhaps???
   Arrogant pollsters… they’re a dime a dozen.

7. MsMew, do you actually know anything about statistics? If I pulled a stunt like that in a medical study the results would be useless. Because they don’t represent reality, they represent the choices made by the study designer. Aka bias.
   If half the sample is from one discrete and -small- regional population, and the other half is taken from five or six other discrete regional populations, you can’t generalize the study to the whole country. Its not science, its propaganda.
   Yet here’s CTV treating this foolishness as if it were serious business. Hence CTV’s ever shrinking viewer ship.
   *sigh*

8. Oversampling – in this case, stratified sampling, from Quebec assumes that the population there is homogeneous and also will come up with different responses than the entire rest of Canada.
   But, you could get different responses from the West, the North, as well.
   And Quebec has only 20% of the population, so, how accurate is such a poll (1,000/500) as a measure of the whole population?

9. “Do you know why oversampling is done? To eliminate possible bias”
   Ya, that’s why it looks like they decided not to poll anywhere west of Mississauga. That’s why 50% of those polled were from a province that supports a seperatist party, and where the governing party is polling the lowest.
   Good God, do you even think before you write stuff like that?

10. Good Lord. This is too easy.
    Oversampling is done to eliminate bias, not to create it. It allows for greater stratification in the sample, which is the very reverse of what ET claims it is–one stratifies a sample precisely because one recognizes that the population is NOT homogeneous. For a number of different reasons, too small a sample might be skewed. I’m speculating, but Quebec has an anglophone population, a significant Native one, and a francophone one. Suppose a sondage of 20% produced a skew, for whatever reason, so that the sub-sample from the anglophone population was disproportionately small. That’s why one oversamples–to make that random skew less likely.
    The point is that, after all the results are in from across the country, the Quebec sample is weighted back down to the proper 20% of the national sample. Some here, including the poster, seems to imagine that the oversampling from Quebec is simply lumped into the national results without weighting. That would create serious bias, of course, but that’s not what oversampling is.

11. What more do we need to know?
    Peter Donolo is a political strategist and communications advisor associated with the Liberal Party of Canada. From 1993 to 1999, he was the Director of Communications in the office of Prime Minister Jean Chrétien.
    No bias here, folks. Just move along …

12. It allows for greater stratification in the sample
    Too hastily written, sorry. It does that, but more pertinent here, it allows for more accurate results from stratification in this case.

13. MsMew, I’m afraid that sounds like, well, gobbleddygook.
    Much like the IPCC stuff.
    Sorry, not buying.

14. Yet Saskatchewan and Manitoba, which also have significant Native populations, weren’t declared “heterogenous”.
    I can also assume that the Cree, Innu and Inuit populations of Labrador weren’t declared “heterogenous” from Newfoundland either.
    Ethnic/racial politics, gotta love it.

15. * fart * oops , so why bother with the rest of the country at all . . Just phone 500 more in Quebec and voila !

16. Nope, not “gobbleddygook.” Survey methodology 101. Time for some of you folks to do a little homework before you make greater fools of yourselves.

17. “it allows for more accurate results from stratification in this case.”
    Not if you over-sample the smallest, least homogenous part of the total population Ma’am.
    Quebec opinion is in the tail of the curve on most issues. Furthermore I’ll bet you a donut that Montreal is the source of most of the Quebec sample, skewing it even farther. Oversampling what you know is the two sigma tail does not give information about the center of the curve. You can’t get anything useful out of a sampling like that just by changing the weighting. You’d have to know what the outcome was supposed to be first.
    Oh, wait…
    BTW, while I may be an over trained, glorified nurse, ET is the gold plated Real Thing(TM). I’d suggest you go crack a textbook if you want to disagree with her on stats and study design.

18. MsMew, why Quebec vs. any other province? Apparently the GTA alone has a population of about 5.5 million, vs. a Quebec provincial population of under 8 million.
    And how do you decide which categories are significant enough to start fiddling with? There’s a francophone category and an anglophone category, okay. Is there an age category? Income? Is churchgoing significant? Birth order?
    I’m not trying to be sarcastic; I just think that there’s inherent bias in these attempts to eliminate bias.

19. This looks to be much ado about nothing. Oversampling Quebec on this issue seems to be an appropriate and reasonable response.
    If 20% of the survey population is from Quebec, that sample is probably too small (200 people) to make any valid claims, however adding the extra 500 to give a sample size of 700, probably does.
    The results of the 500 people from the oversample are not used in any of the original survey results.
    The bias in this, or any other survey would most likely not come from data manipulation, but from the way the questions are phrased and the answer interpreted.

20. The full Strategic Counsel report is here, there is more than was quoted by CTV.
    http://www.thestrategiccounsel.com/our_news/polls/2009-05-28%20-%20GMCTV%20FINAL.pdf

21. SDH: Then the same principle applies nationally. The population of Prince Edward Island is 140,000 vs. a national population of about 34,000,000.

22. The population of PEI is also a darn sight more homogeneous than the population of Quebec.

23. I get a kick out of the way polls are presented, ie +/- 3% 19 times out of 20.
    19 times out of 20 =5% so why don’t they say +/-8%

24. ms Mew – the population of Ontario is the most non-homogeneous of the entire country. Check out Canada Stats to see the ratio of immigrants in Ontario. And any other province in the West has more immigrants than Quebec. What about New Brunswick – with its francophone, anglophone and indigeneous?
    So, I don’t buy your explanation that oversampling was required to rule out Quebec bias. The statistical facts of Canada is the heterogeneity is higher outside of Quebec – and – furthermore, you can’t assume that a Quebecois anglophone will vote on these questions differently than a Quebecois francophone.
    SDH – I didn’t say that 20% of the survey sample population was from Quebec but that 20% of the Canadian population is Quebecois. The sample population of the survey need not match the provincial ratio. After all, if you selected your respondents by provincial ratio, then, your sample from such provinces as PEI, Nunavut, NB, etc..would be equally small. And who says that the sample must reflect provincial opinions?
    My question is why the researchers assumed that Quebecers would have biased responses and therefore, they took a larger sample from that area (which is why this is also a stratified sample), to rule out bias. Why wouldn’t they assume that, eg, ‘red-neck Alberta’ would provide biased sample responses?

25. Well, yes. I was responding to SDH, taking his/her point to what I took to be it’s logical conclusion, i.e. that 1,000 people out of a population of 34,000,000 isn’t really a big enough sample to be reliable.
    My original point was that there is an inherent bias in adjusting statistics based on assumptions about which population subdivisions are significant.
    People who like this sort of thing should google:
    Iowahawk “sampling error”
    Nobody’s ever regretted googling Iowahawk.

26. Homogeneous? Who cares if one province has more people that lie about their sexuality than another? Batb’s comment about Donolo’s background sums up this poll very well,liberal BS,trotted out as facts. Just like that Drummond fellow always giving his opinions on the economy,yet it is never mentioned that before he worked for TD, he worked for the libs for about 25 years.

27. Oh for crying out loud. This is getting outrageous.
    See, Tony, there are two different measures in that formulation, not one. One is called a confidence interval: that’s 95% (19 times out of twenty). That tells you that the results of the sampling will statistically speaking be found to fall within the range(+/- 3%) 95% of the time.
    I ought to charge for this.

28. In the sciences we just called it ‘cooking it in’.

29. I actually think I understand this oversample stuff. If you poll everyone in Quebec,you will get a true response. However if you only poll ten people,you may get a skewed sample. Therefore you poll more and then bring it back to reflect the size of the original polling.Eureka,quebec is a skewed province,and the ROC is just screwed.

30. There is something fishy about this oversamnple stuff. Why is Quebec opinion so critical with this little pissant poll yet regular polls by this outfit do not oversample Quebec? What is so critical about this poll? Why do we need to know who is the prefered dinner guest out of PMSH and MI and why is it so critical that we get a ‘true’ feeling of how Quebec feels on this? Its a push poll designed to counteract the MI ads pure and simple…..what a piece of shit these outfits are?

31. MsMew is correct in her analysis of the process
    and I go with ET in questioning the need to over sample queerbeck, because as she, ET, states , Terana is a large and with greater diversity population, and it was not over sampled
    as someone mentioned, the questions asked in the poll, the time of day it was taken, and who did the polling (what persons/s) are all critical in polling

32. I’d trust a poll that oversampled homosexuals from Qwebec, with an average income over $100k. At least then the underemployed druggies would not be a significant factor!!!

33. for god’s sake why does anyone ever concern themselves with what Quebecers allegedly think?
    you know they’re going to ‘think'(vote) whatever complements and compliments their position a la exchequer vis a vis the ‘current federal position’…always have always will…
    the Quebecers are NOT serious members of MY Canada…i disabused myself of that notion not long after i started wearing long pants and i first met that bald guy …the one married the nutbar out west here…gee…he was SO intellectual and grooving with Fidel and all….it was cool..cool times period.

34. I suspect the Leftist Big Media and the Leftist pollsters are colluding to IPCC-ize data to get results they want to propagandize.
    They want people to think the Iggy Ads are “backfiring” and so on and so forth.
    Of course pollsters can be biased and corrupt, though I’m not going to point any fingers of specificity. Just food for thought. Hey, if scientists can be corrupt…

35. And, then, this brilliant salvo in the TorStar yesterday under the headline “Tory ads attacking Ignatieff backfiring on Harper, poll finds”:
    “The Conservatives’ biting partisan attack ads aimed at Michael Ignatieff are taking a toll — on Prime Minister Stephen Harper as well as Liberal leader, according to a new poll.”
    How’s that for talking out of both sides of one’s mouth? And how’s that for a “partisan” headline? (The poll was conducted by TorStar and Angus Reid whose VP is Mario Canseco. ‘Anyone know his background?)
    I long ago stopped paying attention to polls except to note that the MSM tends to promote polls whose results show the CPC and PMSH in a bad light — and which are often proven wrong come election time.
    Yawn.

36. Comments at CTV quickly closed…right after an honest non-partisan opinion with first hand experience is posted…
    and the title using ‘most Canadians’ was very misleading.
    ’nuff said…media bias is rampant in this article.
    I’d love to have Prime Minister Harper and family over for coffee or a meal anytime.

37. Talk about SKEWED and screwed up polling!
    According to Donolo Quebec is one province but warrants 50% of respondents to his polls. We get the game now, it gets the desired results in favour of the Liberals. It’s a “national” poll? Yeah, right. How about a “half national” poll?
    What a freaking conniving bunch of desperate people.
    As to the Conservative truth ads, they’re working. We know by the reaction from the Liberals and their toadies.
    Statements calling them “biting, partisan attack ads” is really rich when they are the facts about Iggy, some right from his own mouth. It also tells us the Liberals know those facts are not helpful and want to hide the truth from the voters.
    If the ads are hurting or biting Iggy who’s to blame? We didn’t make up the facts, they can’t be denied or glossed over.

38. Wow. This thread makes me want to bang my head against a wall.
    MsMew’s overview of Statistics 101 is sound. While I haven’t read the actual survey methodology for this poll, over-sampling is common in national (or other large population) research projects and, as others have mentioned, the final results are then weighted back down to the correct representation.
    The reason for doing so isn’t to bias the overall results, but rather to provide more reliable and representative results for the sub-set.
    Let’s say that I have a population with 990 boys and 10 girls and I am conducting a survey of 100. Such a sample would normally only include one girl, which would be pretty much useless from an analytical stand-point. I’d be able to talk about the results as a whole, but I would not be able to gain any insight into the girls, as the margin of error would be far too high. To compensate for this, I would over-sample the girls.
    To keep things simple, lets say that I’ll speak to 25 girls and still speak to 99 boys. When I calculate the final numbers for the total population, I will first adjust (weight) the girls results back down to their proportion of the population, i.e. divide their results 1/25.
    The end result is that I can get better insight into the girls’ opinions, but they still only represent 1% of the overall results.
    The “Why oversample Quebec?” in a second.

39. Do all the weighting people note they rarely if ever give the questions asked and how they are asked? I used to answer polls to see which way the bias was going. When the pollsters said who they were it was so fast I never caught who it was. Nanos is noted as being the best pollster. Why? It’s the questions he asks and he is the only pollster that calls cellphones. The rest talk to your grandma or your kids.

40. Who gives a rats-ass what Quebec thinks anyway.
    They have had a love-in with all things socialist for decades…just like the MSM.

41. Several of you are bothered/confused/suspicious as to why Quebec is being oversampled.
    There are a bunch of reasons for this, some that are actually research-related, others that are not.
    The first is pure marketing. Quebec has a unique media market that is interested in Quebec. The Quebec media wants to report on what Quebecers think, whereas the (typically Torontocentric) English media are generally happy to report on broader Canadian results. Because of this, there is a greater need / value in having more statistically reliable results for Quebec than Manitoba.
    Similarly, the media still enjoy / cling to the old “two solitudes” political narrative. It may be lazy and (perhaps) outdated, but the national media are interested in comparing what Quebecers think compared to the ROC. Again, in order to do so, you need to oversample Quebec.
    On the research side, there are several solid reasons to oversample the Quebec population. As you all know, Quebec has a disproportionate (ad significant) influence on national politics both in the number of HOC seats and the focus of national policies. You can hate that all you want but, as long as the situation exists, smart research says that you dig deeper into such a group.
    The Quebec populace is also far more politically volatile. Toronto may have a more ethnically diverse population, but when was the last time that the city didn’t send an army of Liberals to parliament? 1984? The swings in both popular vote and seat count can be fairly wild in Quebec, hence the need to dig deeper into (and have greater confidence in) the numbers.
    Can pollsters be biased? Of course. Can their biases influence the results of their studies? Yes, but this would more commonly occur in the nature of the questions asked or by sampling non-representative populations (and then trying to suggest that their opinions have broader application). Most commonly, the bias occurs with the interpretation and/or presentation of the results.
    Hope this helps.

42. a sample would include quebec at about 20%. the oversample is quebec at 100% giving a quebec bias to the poll of 46% , only appropriate when you are trying with all your lieberal heart to promote iggy.

43. Just to wrap-up, Speedy hits on a very important point. Very few pollsters have properly adapted to call screening and the growing percentage of people who do not have landlines.
    The concern is, of course, that the people who actually answer the phones and take the poll are not representative of the average person (regardless of province).
    I’m not sure that Nanos has it exactly right, but it at least represents an attempt to address this growing problem.
    Bias in actual questions (which I mentioned in my second post) is a clear concern, but has nothing to do with weighting or over-sampling.

44. following MsMew’s line of thinking – my question is how come they didn’t do 500 respondents from Alberta. After all – Alberta is 100% Conservative (well except for the “strategic voting” blip in Edmonton Strathcona).
    Wouldn’t doing an over sample in Alberta acheive the same thing?
    Just askin.
    As I learned in Statistics – you can make “statistics” say whatever you want them to say.
    I would suggest that this poll is a prime example of that.

45. Jim – I acknowledge your analysis of ‘why to oversample’ but I don’t agree with your analysis of why to oversample Quebec.
    Such an act gives an overweight to the Quebec response that I feel it doesn’t merit. You ask when Toronto didn’t return a Liberal Set to parliament – and, with the introduction of multiculturalism, Toronto The City of Immigrants moved into the grip of the political party, the Liberals, who based their voting power on creating isolate sets of dependent immigrant groups.
    Equally, when did Quebec ever return a non-socialist Set? And, with the disastrous enabling of its own political party – when did it move away from the Bloc – its very own socialist party? Quebec isn’t volatile; it’s totally predictable. Indeed, in all of Canada, Quebec is the most predictable; it will vote for whichever party will give it the most federal money with the least responsibility.
    Its 75 seats in the House, heavily held by the Bloc, mean that the government will almost always function as a minority. But, with the new seats in the West, due to their increased population, Quebec’s generation of dominance in Canada is fading. At the moment, Quebec has 11 more seats than the West, despite their populations being equal. The change in seat counts will deal with that.
    Therefore, my question remains – why oversample Quebec?

46. Alberta Girl – You’ve completely misunderstood MsMew’s explanation and apparently only attended the first day of your stats class.
    AGAIN, over-sampling does NOT mean that the sample population becomes over-represented in the national figures. The results are weighted to restore the correct representation in the overall results. The oversampling is simply to increase the reliability of the subset.

47. Half from Quebec !?
    Aren’t they the same freaks that think Justin Trudeau represents their interests?

48. What would the poll results have been if 500 of the respondents were from the prairies and the other 500 from the rest of Canada?
    It is incredible that main street media has the gall and lack of integrity to put the results of such a misleading poll on the national news.
    Doesn’t the President of CTV take some responsibility for what his network is delivering to Canadians under their license – bring on more cable,satellite and the World Wide Web because the media have proven they can’t be trusted and need a watchdog – and I haven’t even commented on the CBC.

49. ET – I’m pinched for time, and I pretty much agree with your feelings if not all of your facts.
    Quebec seat totals have varied significantly over the past 25 years. Huge majorities for the Tories in 84 & 88, a Bloc sweep in 93, Liberal rebounds in the subsequent two elections, a Bloc comeback post-adscam, surprise Tory inroads in the first Harper victory, and another Bloc rebound this past year.
    I’m not saying that it is right or fair – just a valid / interesting group for researchers to study.

50. Pollsters are part of the opinion creation industry. I believe this game was described as manufactured consent by Bernays. They give us the opinions we should have to compliment the political agenda of the day.
    In any real sense opinion polling can be related to fishing for the right answers.