Questions, questions

by Feb 13, 2012 No Comments

When David Cameron announced in early January that he wanted to “settle this issue in a fair and decisive way”, he effectively signalled the start of a long and fractious battle to determine Scotland’s constitutional future.

The first few weeks threw more heat than light on the issue as politicians from all sides of the debate argued over process. When should the independence referendum be held? Who should oversee the ballot? How many questions should be posed? Then, with the publication of the Scottish Government’s consultation paper on 25 January, a concrete question for the referendum was proposed and the nature of the debate changed.

For those of us who design opinion questions on a daily basis this was an important moment.

The wording, style and tone of a question are crucial in determining the value and quality of the answers received; as the saying goes, “Ask a stupid question, get a stupid answer.” There are two key tests in determining the quality of a question. Firstly, it needs to be clear so a respondent knows exactly what they are being asked. Secondly, it should be fair, neither leading nor biased.

The Scottish Government’s proposed question – “Do you agree that Scotland should be an independent country?” – has generally been considered to be clear, though it does rely on a definition of “an independent country” being articulated clearly between now and the ballot taking place. Previous referenda in the UK have highlighted the importance of simplicity and clarity, ensuring all sections of the electorate are clear on the question being asked. Indeed, the Electoral Commission insisted on a change to the question originally proposed by the UK Government ahead of the 2011 AV referendum, because it considered that “particularly those with lower levels of education or literacy, found the question hard work and did not understand it”.

More criticism has been made of the proposed question in terms of its perceived fairness. The statement goes in one direction only and, it is argued, tends to invite a respondent to agree with it. One key point here is to consider what responses would be if the question was asked the opposite direction, such as: “Do you agree that Scotland should stay in the United Kingdom?” Previous referenda have resolved this issue by offering the two alternatives on offer and giving them equal weight. The 1997 devolution referendum provides a useful example as voters were asked to choose between the statements: “I agree that there should be a Scottish Parliament” and “I do not agree that there should be a Scottish Parliament.” In reality, we’ll have a better idea of whether a question is clear and fair once the options have been tested rigorously and analysed against differently worded questions, with robust sample sizes, to measure attitudes to independence. In addition a phase of “cognitive testing” of various question options would be useful.

Cognitive testing is an approach we use to assess how questions are understood and answered by respondents. After a respondent has answered the question put, a researcher interviews the respondent to explore how they went about answering it. This process can be used to refine questions before further testing. We have successfully used cognitive testing techniques in a range of settings; this includes testing the question on “national identity” before the 2011 census, when our testing informed the wording of the final census question, ensuring that respondents were clearer about what was required and better able to express their identity/ identities fully.

There are likely to be a range of alternative questions proposed before the referendum is held. Each needs to be tested thoroughly, including the use of cognitive techniques, before the final decision is made.

Leave a Reply