.desc-wrapper { height:300px!important; }


Yes, The Election Polls Were Wrong. Here’s How We Fix It.

The cracks in the polling industry have been readily apparent for years and it was only a matter of time before we had a major polling miss in this country. On Monday I offered the following warning about the accuracy of the polls leading up to election day. I knew a surprise was coming thanks to my own online surveys throughout the year.

For years, I’ve been advising clients to move their polls from phone to online. Political and issue-oriented clients have been more resistant to the change, while private firms have embraced the lower cost, quick turnarounds, and improved accuracy offered by online surveys.

Much of the resistance from clients in the public affairs space (i.e. campaigns, associations, public affairs groups, etc.) is due to guidance from groups like FiveThirtyEight, AAPOR, AP, etc. who for years maligned many of the publicly available online polls.

Granted, in some cases their skepticism and criticism was warranted. But it was totally unfair and wrong to advise media and organizations to simply avoid “unreliable” online polls. There’s a lot of good online polling being done right now such as the USC Dornsife/L.A. Times tracking poll that correctly predicted Trump’s victory.

Another factor is resistance to change. Too many pollsters are wedded to phone polling due to the revenue streams associated with a methodology they have used for years that gave them significant professional and financial success.

Those days are now over.

People lack the time or patience to answer a 15-minute phone survey, and the respondents who do stick with it are probably not an accurate reflection of any group other than partisans or people who are natural joiners or volunteers.

Consumers have tools to avoid being interrupted by telephone polls via caller-id and blocking technologies just as they successfully avoid TV commercials with subscription services and time-shifting.

The ability to avoid or the lack of desire to participate in phone polls have led to record low response and cooperation rates. This has been common knowledge for years and it biases survey results. Adding cell phone interviews to this stew helps, but is not a cure, because it’s very expensive and difficult to reach the 40% to 50% of the population without a landline phone.

Respondents are simply more honest answering an online survey compared to surveys that are administered by a live interviewer over the phone. This is especially true when testing voting intentions involving two candidates with off-the-chart negative ratings competing in a highly charged media environment.

Short online surveys are the way to go.

Let me add one caveat.

Most online surveys use a panel of pre-recruited individuals or households who have agreed to take part in online market research. Typically, there are not enough of these panelists to conduct a statistically relevant poll in a smaller geography such as a congressional or state legislative district. However, online is a viable and a preferred option for statewide and larger congressional districts.

Okay. So, we should go ahead and convert our phone polls to online?


You can’t just convert your phone poll to an online survey. It’s not that simple. Online respondents require different questions and different methods to interpret the results.

This was the mistake newspapers made in the early days of online news. They simply took their print product, which was declining in readership, and replicated it online. Not taking advantage of the new technology was a huge mistake. It’s the same for polling. Simply taking a phone poll and asking the same questions online is not advisable.

What do I suggest? I don’t pretend to have all of the answers, but these are some ideas that have worked for me.

Shift To Online Polling

There are just too many issues in phone polling, ranging from non-response (i.e. getting a representative sample of people to talk with you) to coverage (i.e. reaching certain segments of the population such as prepaid cell phone households)

Purchase quality online sample or grow your own online panels. Online panel quality matters. A lot. For example, a panel built from coupon clippers and entrants to online sweepstakes, contests, and giveaways will skew your results — unless that’s your target audience.

Respondents are more likely to be honest when answering an online poll. I’ve tested it and so have others. Use this to your advantage. For example, respondents are more likely to state their actual household income when answering an online survey. Ask for their household income in the survey and then use it to weight the data. Too many disregard income questions in phone polls fearing that respondents are less than honest in their answers. If you believe this, then don’t ask the question, or find a methodology such as online where people will accurately answer the question. Stop wasting peoples’ time. It reflects badly on all of us in the industry.

Proper Weighting

We know some population groups are over- or under-represented in a survey sample regardless of methodology. We have to do a better job of weighting (i.e. assigning “corrective” values to each one of the sample responses of a survey) regardless of mode (i.e. online, phone, mail, or in-person surveys) to ensure results reflect the profile of our desired audience for our survey.

This election provides a great example. YouGovUS, an online polling firm, polled throughout this election cycle. While they weighted their results across a host of demographic variables, it doesn’t appear they used household income in their weighting scheme. That was a mistake.

As you can see below, their samples skewed lower income, and Clinton outperformed with lower-income voters while Trump over performed with upper income households. Who knows? If they had factored income in their weightings, perhaps they would have had a better read on this election.

Surveys should be heavily weighted. Just using simple demographics such as gender, age, and ethnicity to weight a survey is insufficient. You have to include attitudinal and behavioral measures in addition to demos such as income if you have any chance of getting useable results. Harris Interactive did a lot of great work in this area in the early days of online polling.

Beware Digital and Social Media Signaling

Use digital and social media analytics to augment your polling, not to replace it. Digital plays a role and in some ways replaces qualitative research. However, in the current media environment, language has been weaponized, especially online. The data you collect from social media will reflect the socially desirable aspects of peoples’ personality or beliefs. People typically put on their best face online so it’s really difficult to determine what is real or simply social signaling.

Shorter Surveys and Polls

People lack the time or patience to answer a 15-minute survey and the respondents that do are probably not an accurate reflection of the overall population. Use third-party data or purchase demographic data from panel providers for variables you need to weight the data. If you have a lot to test, launch multiple short surveys instead of long surveys that respondents hate.

Vary Your Sampling

For political polling, don’t rely solely on verified registered voter files for your samples. Conversely, don’t rely on samples of the general population using screening questions to determine voting status or intent. Do both.

Pollsters are like generals, they’re always fighting the last war and every election is different. Chances are, you will miss. Tight screens might work for one cycle and then be absolutely the wrong option in another cycle. Vary your sampling and you will have a wider range of possible outcomes to review and analyze.

There are no easy solutions to cure what ails the polling industry. Technology has given us better analysis tools while making it more difficult to collect data to analyze.

Typically, when faced with these situations, people focus on what they know and keep doing the same things over and over until they’re forced to change.

Our jobs are too important to simply ignore the serious issues we now face or to leave it to the next generation to address. We need to change and evolve instead of deriding or attacking other ideas on how we should do our jobs.

The suggestions above are just that, suggestions. If you have a better mousetrap or idea, I would love to hear it, and I hope others in the industry are ready to be open-minded, too.

Could The Polls Be Wrong? It’s Possible.

Over this final weekend before the election, Nate Silver FiveThirtyEight has taken fire for being too cautious in forecasting the outcome of tomorrow’s election. Some have charged that his probabilities for a Clinton win are too low. Frankly, I don’t blame him. I would be cautious too.

Silver and crew use a results-validated model to make election predictions. Like any model, the results are only as accurate as its inputs. Garbage in, garbage out.

Silver’s models contain a large number of public phone-based polls that typically include some combination of landline and cell phone interviews. This would make me nervous. Very nervous.

The accuracy of phone polling is declining and has been declining for years for a host of reasons. It’s one reason I’ve shifted everything to online polling. Also, media sponsored public polls are generally less reliable than private campaign polls.

Polling by phone is an increasingly expensive and difficult endeavor. Today, phone polling should include some degree of cell phone interviews to augment standard landline phone interviewing. Adding cell phone interviews to this stew helps, but is not a panacea for reaching the 40% to 50% of the population without a landline phone.

Missing from most cell phone surveys are interviews with voters who are not on a cell phone contract. Pre-paid and pay-as-you-go cell phones are two growth areas in the cell phone industry and at least a quarter of the cell phone population is probably missing from most polls. In the UK and Europe, pre-paid is the bulk of the cell phone market.

Also, including cell phone interviews is very expensive. Legally, cell phone interviews must be hand dialed which increases the labor expense for phone polling and reduces the quantity of available public polls. This is especially true at the state level where local media lack the funds to conduct a series of strong phone polls.

However, simply including cell phone interviews doesn’t cure all the issues faced in polling by phone. Getting a representative sample of respondents willing to complete a survey is difficult. On a good day, we may get 11% of a sample to fully respond to a poll by phone. This is tragically and historically low.

Response rates to surveys also vary by season. Rates are low during holidays (i.e. December) especially among consumer audiences. It’s also difficult to get a representative sample during the summer when people are on vacation, traveling, watching youth soccer and baseball, etc.

October is an equally problematic time for polling. Historically, response rates for October phone-based polls can decline up to 25%. October is the first month of the last quarter of the year and is typically a huge month for business travel. You also have Halloween, a bank holiday, benchmark tests for school students under “No Child Left Behind”, fall breaks, youth sports, etc. October isn’t as bad as December for polling, but it’s not far off. It’s entirely possible that respondents to phone polls in October are not representative of who will show up on election day.

This non-response issue is the likely reason we’ve seen such crazy numbers from the October polls. Take a look at the NBC News/Wall Street Journal polls from October and November. The swing of Clinton from +6 to +9 and +11 and back to +4 are more likely the result of changes in survey response rates than a genuine change in the voting intentions of the electorate.

NBC News/Wall Street Journal Poll

It’s wild swings like we see in the NBC Poll and a lot of prior research on phone polling that makes me question most of the current phone based polls. As I mentioned earlier, I’ve shifted to online polling where I typically see more stable numbers without the wild fluctuations we have seen in the phone polls.

Sadly, there are not that many online polls this cycle. Outside of YouGov and newer entries from Google Surveys and SurveyMonkey, we just don’t have a lot to work with. We also don’t have a lot of information on the source of the online sample used by these groups.

Most online surveys use an online panel of pre-recruited individuals or households who have agreed to take part in online market research surveys. Most are compensated in some way and the quality of these samples vary. The quality of the respondents matters a great deal, so it’s hard to gauge the accuracy of these polls without knowing who and how their survey respondents were recruited.

Two polls widely viewed as outliers this cycle are the IBD/TIPP Tracking poll (Phone) and the LA Times/USC Tracking poll (Online). Both polls have been remarkably stable over the final months of this election and both indicate a much closer race than the majority of other public polls. Also, both polls have given Trump leads over the past few months.

One possible reason for the outlier status of these two polls is the use of weighting.

Weighting is a technique in survey research where survey results are re-balanced to more accurately reflect the population you’re polling. A demographic profile (based on known data such as a census age distribution) is often used to rebalance survey results to better reflect real-world results.

From what I can tell, both the IBD/TIPP Tracking poll and the LA Times/USC Tracking poll heavily weight their results beyond measures such as gender, age, party affiliation, and ethnicity. If their use of weighting cures some of the deficiencies in terms of non-response (i.e. households who don’t respond or answer a phone poll) and coverage (i.e. prepaid cell phone households not in the cell phone samples), they might not be outliers after all.

This is especially true for the LA Times/USC Tracking poll which has consistently given Trump a better chance of winning than pretty much any other polling organization. This poll is conducted online, uses a different set of questions than most “traditional” polls, and weights the results across a broad range of measures. This poll is conducted among a sample of respondents that were recruited specifically for this type of research. Their methodology could be an antidote for the problems with traditional polling that I mentioned earlier, or it could be a major factor in being very wrong about this election. They are basically in a go big or go home situation that I totally support. Stick to your guns and if you are wrong, tell us why.

There is some recent precedence for properly weighted online polls outperforming phone polls, namely, Brexit.

Online polls outperformed phone with the UK Brexit vote earlier this year. Phone polls tended to show a win for the “Remain” camp. But online polls by TNS UK and Opinium Research accurately predicted a “Leave” win, and were both viewed as outliers by most pundits. Both also used extensive weighting to ensure their results properly reflected the views of potential voters.

It’s entirely possible that the phone polls of this election could be wrong, and if they are, a large component of Silver’s data-driven election model, will also be wrong. Nate Silver and crew are right to be cautious. It’s not out of the realm of possibility that we are shocked by the election results tomorrow night.

Based on the public and private polling I’ve tracked and conducted, I can easily envision three very different scenarios occurring tomorrow night.

Scenario 1: Clinton cruises to an easy three or four-point win in the popular voter and handily wins the electoral vote

Scenario 2: The election is exceptionally close late into the evening and we are up waiting for results from Colorado, Arizona, and Nevada to find out the winner

Scenario 3: Trumps win the popular vote but loses the electoral vote

Scenario three is my nightmare situation, leaving everyone unhappy and providing zero closure to an overly-stressed electorate.

Conventional wisdom has been wrong all year, and I’m totally expecting some type of surprise tomorrow night. It could be a decisive win by Clinton, a close win by Trump, or a deadlocked election.

So forgive me if I don’t jump on the Nate Silver bashing bandwagon. This has been a crazy year and I expect tomorrow to be the same. I just hope we can learn something from this whole episode that helps us learn more about our fellow citizens and how to properly capture their actual thoughts, concerns, and beliefs for future elections.

Journalists Are Out of Touch With Reality

Headlines coming out of the third and final 2016 presidential debate all followed a similar theme: Trump will not commit to accepting the election results if he loses.

The four largest newspapers in the country ran front page headlines that adhered to the same script, echoing post-debate comments from broadcast commentators:

  • Washington Post: “Trump Refuses To Say Whether He’ll Accept Election Results”
  • The Wall Street Journal: “Trump Won’t Commit to Accepting Vote if He Loses”
  • New York Times: “Trump Won’t Say if He Will Accept Election Results”
  • USA Today: “Keep You In Suspense: Trump Won’t Commit To Accepting Vote Results”

Out of a ninety-minute debate, viewed by more than 71 million Americans and covering a variety of topics, this was the major takeaway?

Despite the blaring and repetitive headlines, Americans’ view of this election and whether it’s rigged is unlikely to be shaped by an increasingly out of touch and out of time press.

The belief that Trump’s failure to commit to the outcome of the election was a major gaffe simply illustrates just how out of touch major news organizations are with the country they cover.

Americans increasingly believe the system(s) is rigged against them and view corruption as widespread in government and business. Few will be troubled by Trump’s “rigged” rhetoric since most already believe corruption is a huge problem in the country.

According to Gallup, three in four Americans (75%) perceive corruption as widespread in the country’s government. While this number is from 2014, it’s been steadily increasing since 2007.

So what do Americans fear most? Corrupt government officials. According to the 2016 Chapman University Survey of American Fears, six in ten Americans (61%) identified corrupt government officials as their top fear, eclipsing both terrorist attacks (41%) and not having enough money for the future (40%). Government corruption was also their top fear in 2015.

Americans’ view of government and government officials is dismal and their assessment of the economy and businesses are not any better.

Seventy-one percent of Americans think the U.S. economic system is “rigged in favor of certain groups” according to a June 2016 poll conducted by Marketplace and Edison . This belief was shared regardless of political affiliation or ethnicity.

In a follow-up poll conducted in October 2016, nine out of ten Americans who believed the economic system was rigged in favor of certain groups agreed the U.S. economic system is rigged to benefit politicians (89%) and corporations (86%).

Amidst this backdrop of distrust, why do journalists think voters care if Trump accepts the outcome of November elections? They probably don’t. If anything, they probably agree that it is rigged.

And as much as the legacy press highlights “the gaffe”, their ability to influence voters’ opinion on this topic is increasingly weak. Americans view the press just as negatively as the aforementioned government officials.

In a poll conducted in 2016 by the Media Insight Project for the American Press Institute, less than one in ten Americans (6%) have a great deal of confidence in the press. Four in ten (41%) Americans said they have hardly any confidence at all in the press. In the same poll, more than one out of three (38%) Americans said they have had an experience with a news and information source that made them trust it less.

One of the great challenges ahead is finding a way to restore faith in institutions and most importantly, the press. Sadly, the damage will take decades to repair. These types of initiatives are often generational and will require the passing of significant time before peoples’ memories of this era fade.

The first step to restoring faith in the press is for the media to admit there’s a problem. Second, media organizations need to accept reality and understand how the public actually thinks.

Too many journalists and thought-leaders view the country as they wish it to be and have segregated themselves into hive minds that shelter them from opposing opinions.

Journalists and legacy media are some of the worst offenders of this phenomenon, often opting to challenge the grammar of a dissenting voice than understanding the beliefs and judgments of the messenger. Granted, sheltering yourself, regardless of your profession, against the trolls and vitriol of social media is tempting. However, it skews your view of the country, which is a big problem if you’re a journalist.

Without trust in the press, it’s unlikely faith in any government or political institution will be repaired anytime soon. The divisions and distrust we see today will only continue to grow.

It’s high time journalists and the media return to reality, perhaps survey their audiences to learn how they think, focus coverage on issues the public actually cares about, and then report it accurately.

Staying Ahead of the Curve (Keeping An Eye On DC)

With all of the talk about an amnesty bill in DC, I don’t’ think it is a coincidence that AARP is expanding its Latino-oriented, Spanish-language media properties. The expansion will will debut with the Spring issue of AARP's quarterly bilingual magazine, AARP VIVA Su Segunda Juventud. The brand will also extend into a TV show, radio series and Web site.

AARP is usually ahead of the curve in DC and is a great example of an organization that keeps an eye on the political tealeaves when looking for opportunities to grow. This is an important lesson for every business. One change in law can have a dramatic impact on your business. Keep an eye on Washington and plan accordingly. It could be the difference between a very good year, or very very bad one.