Wednesday, November 01, 2006

COMPAS: Hacktacular!


A couple commenters have suggested that the company responsible for the polling for the recent SSM survey -- COMPAS Public Opinion & Customer Research -- might have its own ethical issues.

Back here, commenter Mike wrote:

As for [COMPAS president] Conrad Winn, well no surprise there. He was my Political Science prof back in 1986 at Carleton and considered himself a 'ultilitarian'. Meaning he would do what it took to get a paycheck, as this 'study' shows.

The very next (anonymous) commenter makes an interesting claim:

A reliable source told me that Conrad Winn was at McVety's anti ss-marriage shindig in Ottawa last week telling the wankers how polls could be really useful in making their case - COMPAS is obviously a sham of a polling company.

Admittedly, these are just two peoples' opinions, but it's the substance of the survey itself and the results that raise one's eyebrows just a tad. Recall the actual questions in the poll itself:

  • Should an individual minister, rabbi, iman or other clergy have the freedom not to marry a same sex couple if this were against the clergy’s religious beliefs?

  • Should a school teacher have the freedom to disagree with the same sex law in a letter to a newspaper?

  • Should a religious person who prints brochures for a living have the freedom to recommend another printer to a homosexual group wanting some brochures printed?

  • So long as there are enough marriage commissioners available for gay marriages, should individual commissioners be allowed not to officiate at gay marriages if this is against their religious beliefs?

As we've already pointed out here, the first question is utterly irrelevant. There has never been any debate over whether clergy have an obligation to marry any couple they would prefer not to in the context of a religious ceremony. This religious freedom is already guaranteed under the Charter, so the very asking of the question is simply pointless.

Questions two and three clearly represent the recent incidents involving B.C. teacher Chris Kempling and printer Scott Brockie, both incidents which are addressed here, where you can see that the questions don't even remotely describe what actually happened.

Kempling was not just a teacher who wrote a letter, he was a guidance counselor who was continuously and openly hostile to gays, while Brockie did not just "recommend" another printer to gay customers, he refused to serve them -- a very different situation. In short, the questions clearly try to present those events in the most harmless and innocuous light possible, deliberately leaving out the actual relevant details.

But it's the last question that's the most entertaining, asking whether (publicly-funded) marriage commissioners should be allowed to refuse to officiate at same-sex marriages as long as there are other commissioners available to do the job? But what does it mean to say that there are other commissioners "available?" Available at that location? In the same town? Within a day's drive? And, more importantly, where is the obvious followup question -- what if there is no other commissioner available? That's the question that's just screaming to be asked and yet ... nothing.

So, we can see that the survey itself is worthless shit with no value whatsoever. But is that the fault of COMPAS? Admittedly, I have no expertise when it comes to public polling logistics, so it's quite possible that the function of a public opinion company like COMPAS is to simply ask whatever questions it's given (by, in this case, the National Post and the knuckle-dragging Neandertals at the Institute for Canadian Values).

But (and, again, I'm speaking from a position of pure ignorance) wouldn't a responsible polling organization take a look at the questions, then say something like, "Um ... we can ask these questions as you gave them to us if that's what you want but, in our opinion, they have some fundamental flaws that you might want to look at first." Is that part of the job of a polling company? Or is their job to just ask what they're given, no questions asked. (And even if the company asks what it's given, wouldn't it have to know that the results are going to be questionable, if not downright valueless?)

In any event, I think it's safe to say that that survey is total crap, which brings us to the obvious final question -- what's the deal with COMPAS? Does it have a history of this kind of rancid, dishonest rubbish? By all means, let's hear about COMPAS. The lines are open, and operators are standing by.

Go wild.

8 comments:

Anonymous said...

Just to flesh things out, the newly anointed president of Canwest Global is Derek Burney - head of Harper's transition team, and just for giggles a former Mulroney era backroom advisor.

Anonymous said...

This really shouldn't be news.

Almost from the day the National Post launched it has been buying polling, or publishing polling commissioned by ideological fellow travellers, that endeavors to show that Canadians are supportive of their far right agenda. The one about how a solid majority of Canadians support private health care was a classic.

The purpose of this polling is to recast that agenda as actually being "mainstream", while putting the National Post's ideological opponents on the defensive and generating favorable publicity for the cause.

Usually such polling is published without disclosing the full methodology, including the specific questions asked of respondents and the options they were given as answers. This prevents the kind of negative publicity the COMPAS poll has received, where it is clear the questions and answers were selected to get results pleasing to COMPAS' paymasters.

It goes without saying that any competent pollster can manipulate the methodology to get whatever results they want, which is why a poll that is released without a full methodology (which is now the norm, even if it wasn't followed in this particular instance) should be treated as extremely suspect.

Anonymous said...

This seems similar to "push polling" used by the right wing in the US, generally for political candidates.

Sort of reminds me of that joke about an accountant, mathematician and statistician when asked what 2+2 is? The statistician's response is "what do you want it to be?"

Anonymous said...

LOL!

That was exactly my point Mike, but you made it much more succinctly and humorously!

Anonymous said...

As a statistician, I'll just say that I object to shot against my job. Both the humourous one, and the perversion of "polling" that these things pass themselves off as.

No attempt made to determine interesting information, obvious questions not asked, they are a waste of time and money. It's like asking "Are you glad you stopped beating your kids?" Loaded questions that don't provide any useful information at all.

For instance, from the article: "64% of those surveyed supporting a full or partial review of the existing same-sex marriage legislation "to make sure that freedom of speech and freedom of religion are fully protected." The question is asking if a law should be checked to make sure it doesn't hurt freedoms. Well, no shit, of course it should. Let's hope those checks are done...ask this question in a non-loaded way, and you get 90% support or more.

All this means is 36% of people know what you are getting at, and refuse to give you the answer you are trying to get. Which should be a worryingly high % for the pollsters. That's a lot of people who can smell the crap, even over the phone talking to a stranger. Good for them.

Anonymous said...

"The purpose of this polling is to recast [the National Post's] agenda as actually being "mainstream", while putting the National Post's ideological opponents on the defensive and generating favorable publicity for the cause."

One poll that the National Post can't fake (but wishes it could) is the market choices of people who refuse to buy their stupid newspaper.

Re: the joke - the way i heard it, it was 3 accountants being interviewed for a job, ... this in the aftermath of Enron, and Arthur Anderson Accounting.

Anonymous said...

asking whether (publicly-funded) marriage commissioners should be allowed to refuse to officiate at same-sex marriages as long as there are other commissioners available to do the job?

Marriage Commissioners aren't "publicly funded" (at least here in Alberta) - they are paid by the couple. However, because they are licensed by the government, they are still public officials.

Anonymous said...

excellent post!