CBC : it’s fine to consult, as long as you do it properly by Francis Fortier

May 14, 2014

Source: IRIS

On May 5, the President of CBC/Radio-Canada, Hubert T. Lacroix, speaking to the Canadian Club of Montreal, laid out the CBC’s financial situation. The speech, ostensibly intended to explain the impact of the federal government's budget cuts on the public broadcaster, was instead mostly devoted to a lecture on the new ways of consuming radio and television content. And since, in its President’s words, CBC/Radio-Canada faces “decision time”, the speech also served as the launchpad for an online survey: “Transforming CBC/Radio-Canada for the Future: Let us know what you think.”

This desire to consult the population and CBC employees is very laudable. However, if this consultation is to be valid and representative of participants’ opinions, it needs to follow proven methodological rules. Unfortunately, after having submitted it to a few tests, we have concluded that this is not the case for three major reasons which we explain below: technical problems, faulty sampling methodology and the political direction which permeates the entire survey, the design of the questions and the available answers, which predetermine the results.

1. Sample problems

There is a lot of debate about the scientific validity of online surveys. In fact, the questions and criticisms are mostly aimed at the way in which web-based panels are put together. Unfortunately, the CBC/Radio-Canada “survey” does not respect even the most minimal criteria that would qualify this “consultation” as a true survey.

The questionnaire is accessible to anyone from anywhere. You can participate as casually as you would on a news site with a framed question like “Should we trade such and such a player?” or “Do you prefer your coffee with milk or cream?”. Even if you were in a part of the world that had no access to most of the public broadcaster’s content, you could still fill in this survey. This type of sampling raises obvious problems, because it is in no way representative of the population, as would be the case with proper probability sampling.

If this survey obtained answers from 25 million Canadians, we would have a very good assessment of the issue. However, in this case we have another problem: there is no way to know whether each respondent is in fact a different person. In its current form, the survey can be filled in as often as one likes. There is no security in place to determine whether the respondent is human or a machine; it would be easily possible to automate participation. Even if is not necessary to go as far as administering the Turing test to everyone who fills out the survey, there are forms of simple, accessible security that would do the trick very well. To paraphrase the end of Mr. Lacroix’s speech, “Let’s go, take your smartphone, your tablet, your laptop or your desktop and do the survey!” Who is to say that certain people will not have responded to this appeal a little too enthusiastically, showing their love for the public broadcaster by responding to the survey several times? We simply have no proof.

The creators of this survey are unlikely to believe that they have designed a probabilistic survey; let’s leave them the benefit of the doubt until they present their results. However, as we have seen, the methodological problems go beyond this one issue. In fact, in examining the questionnaire more closely, we can ask ourselves if we need to await the compilation of the responses to have a pretty good idea of the results.

2. Political direction

When a survey is ordered, let’s not be naïve, there is always a purpose behind it. However, at the design stage, professional practice is to avoid and to eliminate this direction as much as possible to obtain responses as representative as possible of what participants actually think. This survey is totally devoid of this honest effort to seek a certain objectivity, which is already obvious in the first few lines. Here is the introduction to the questionnaire:

CBC/Radio-Canada is making a transition to the future. The broadcasting industry is dramatically changing and we must make tough choices to ensure that we are able to seize opportunities and position ourselves to meet the evolving needs of Canadians.

In the questions below, you’ll find real issues facing us as we develop a strategic plan that will take us to 2020 and beyond. This survey was developed to solicit input from Canadians as to the kinds of services they expect from their public broadcaster now and in the future.”

 So before we have even begun the survey, two thoughts have been planted:

-        The changes are inevitable

-        Difficult but necessary choices must be made to save CBC/Radio-Canada.

These two a priori political statements will obviously orient the answers eventually.

3. Biased questions

The third important element in the art of survey design is to formulate questions in such as way as to avoid pushing people into specific answers. We have demonstrated the effort to present the inevitability of the changes in the introduction to the survey. Each question also has an introduction that constantly reminds the respondent of this idea. Here is an example drawn from the consultation:

Children are increasingly consuming television content online.

Q6:  Looking towards 2020, do you think that our children’s programming should remain on conventional television or be available online only?

  •  Keep it on the conventional television service.
  •  Move all children’s programming online.

Since children consume more and more television content online, why would I want to prevent a child from doing what pleases him or her, or is most convenient? Why continue to offer an obsolete service? In this way, the respondent is pushed to choose the option that proposes to transfer programming to the internet.

This is not an isolated case. All the survey questions are biased in the direction of pushing the respondent towards new rather than traditional modes of distribution for audio-visual products. Of course, it’s not unusual to see an introduction to a survey question. However, this type of presentation generally serves to offer information that is indispensable to the ability to answer the question, but without prejudicing the answer. On reading these questions, we can see that it will be difficult to rely on the results of this consultation to know what the respondents really think.

Furthermore, if we look carefully, we see that the majority of the questions only offer two possible answers. This is what we call the false choice dilemma. It provides the illusion of freedom of choice while eliminating the possibility of a middle course or a non-response. It’s one or the other. This dichotomy of choice almost automatically creates conditions in which the respondent will follow the orientation preferred by the survey designers.

In the end, this laudable consultation will lead to results of which the validity cannot be ensured. We will never know how many people responded, whether they represent the Canadian population or whether they even live in Canada. Neither will we know if they are answering sincerely or after having been influenced by the political discourse that precedes the questionnaire.  Finally, we cannot know whether their answers are simply the reflection of the way in which the questions were formulated.  So whatever the good intentions that may have motivated this survey, we will not learn very much from its results about what the population actually wants for CBC/Radio-Canada’s future.