Cambridge Professor Sheila Bird, vice-president of the Royal Statistical Society, says the company hired by the Electoral Commission to research the public views on ballot paper design – Cragg Ross Dawson – did not interview enough people for the results to have scientific standing, and used flawed designs that provided confusing information.

“The standard of reporting was appalling. There were too few subjects to give proper qualification of what the rejection rate of sample papers was,” she says.

Cragg Ross Dawson says it carried out the research according to the terms of the contract it was given.

“The research was always meant to be qualititative. It was not intended to have a statistical basis,” said director Tim Porter.

The election was marred by 100,000 spoiled ballots, with many voters complaining the design of the ballot paper was confusing. The Scottish Office ordered that the design of the papers be changed for the 2007 election.

In researching different ballot designs, Cragg Ross Dawson sought a quota sample of four sets of 25 urban voters without party political affiliations. They were then shown five different paper designs and asked for their opinions and asked to make a ‘mock vote’.

Bird says the methodology employed by Cragg Ross Dawson was flawed:

“The order in which interviewees were asked to use and assess the five design options for ballot forms was ‘rotated’ to ensure that each design was presented first to the same number of voters, but may not have been formally randomised. Failure to randomise the order of presentation of ballot forms would be a serious design fault because of the potential for learning or other effects from the previously-seen forms. Interviewees who make a mistake on the first form may well get the second form right.

“Notably, Cragg Ross Dawson did not report its results for the first-presented design separately. The quota sample of 100 voters was, of course, wholly insufficient to quantify properly the likely “rejected ballot rate” associated with the first presented design.”

In order to ascertain how the research group of 100 behaved, Bird used Freedom of Information law in an attempt to access the trial ballot forms, only to be told they had been destroyed. She was also told that the Gould Report had not asked for access to the forms.

Without having had access to this data, Bird says there is no way Gould could have arrived at the 4 per cent rejection rate mentioned in his report.

“When quantitative answers are needed, qualitative studies are a masquerade. And sadly, even the man who was conducting the postmortem fell into that trap. The 4 per cent figure Gould plucked from the air was a masquerade as well,” she said.

Bird also says that in future elections, the order of the parties must be randomised separately in each of the regions to ensure listing fairness.