Nate Silver: "Excluding Cellphones Introduces Statistically Significant Bias in Polls"

Excluding Cellphones Introduces Statistically Significant Bias in Polls

This is a graph, per the CDC, of the percentage of "cellphone-only" adults in the United States. The fraction, as of the second half of 2009, was 23 percent of all U.S. adults, or 25 percent of all U.S. households. These adults -- about a quarter of the population -- are simply ignored by many pollsters.



This is not a new problem -- in fact, it's one we've written about on several occasions. But it's continuing to get worse. The percentage of people who have replaced their landlines with cellphones has climbed at a remarkably steady rate. (There may have been an especially large leap from the second half of 2008 to the first half of 2009, when the recession was at its worst and many people were looking for ways to trim household costs.)

Bear in mind that these figures are already somewhat out of date. The fraction could be in the high 20s by the time we get to November. And if the current trends hold, it could be in the mid or high 30s by the time we get to 2012. Nor does the figure include so-called "cellphone-mostly" households, which is when the house has a landline, but rarely or never uses it to receive incoming calls; another 15 percent of the population falls into this category.

Cellphone-only households are different from their landline-using counterparts. They tend to be younger, poorer, more urban, less white, and more Internet-savvy. All of these characteristics are correlated with political viewpoints and voting behavior.

The pollsters' usual defense mechanism against this is to weight their polls by demogrpahics -- something which they need to do anyway, since polls are subject to many forms of non-response bias (for instance, it's harder to get men on the phone then women). But this is potentilly an inadequate response for several reasons. First, some characteristics that correlate with both cellphone usage and political preferences may not correspond to those that are most commonly used to weight polls. It is somewhat rare, for instance, for pollsters to weight their polls by characteristics like urban/rural location or marital status, which are predictive of both cellphone usage and political beliefs. Being cellphone-dependent also appears to be significantly correlated with media consumption habits (in particular, getting more of one's news from the Internet and less from television), which also seems to be increasingly important in determining one's political views. And there are some characteristics that may be even more subtle. For instance, there are some hints in the CDC data (such as the higher prevalance of binge drinking) that cellphone-only adults are less "domestic" and more "bohemian". I suspect that, in young adults, this is correlated with more liberal political views.

Secondly, even where weighting occurs, one may encounter problems when upweighting from very small subsamples. It is now very difficult, for instance, to get young people on the phone when using a landline-only sample. About half of all adults from age 25-29, for instance, are cellphone-only, and two-thirds are either cellphone-only or cellphone-mostly. (The numbers are actually slightly better for adults aged 18-24, who are more likely to be living in a college dormatory, or still to be living at home, where a landline will usually be available.) Couple this with the fact that young people have grown up in a call-screening culture, and their response rates are often completely inadequate. Say that you're supposed to have 100 people aged 18-29 in a poll of 500 adults, but in fact you only get 30 because of problems with call-screening and cellphone usage. The margin of error on a sample of that size is 18 percent. And yet, you may essentially let each of these young people speak on behalf of two or three of their peers, to compensate for the ones you haven't gotten in contact with.

A new study from Pew, in fact, has found that these weighting schemes may have become inadequate. In their experiment, a weighted landline-only sample produced a generic ballot result of Republicans 47, Democrats 41, whereas a weighted landline-plus-cellphone sample had the generic ballot tied 44-44. That six-point net difference is statistically significant, and needless to say, could have huge implications for where the parties finish in November.

<snip>

The most obvious solution, of course, is to call cellphones, which more and more of the major-media pollsters (like Gallup, Pew, ABC/Post, CBS/NYT, NBC/WSJ and AP-Gfk) are doing, as are some academic polling centers like Quinnipiac.

http://www.fivethirtyeight.com/2010/05/study-excluding-cellphones-introduces.html?utm_source=twitterfeed&utm_medium=twitter