Statement on Survey Methodology

Why we poll who we poll

District surveys are generally harder to get right than statewide surveys, because with declining response ratesand increasing selection bias, getting a representative sample for a survey from the smaller universe of voters in any given district is increasingly challenging.

For this reason, we’ve moved away from phone-only surveys to multi-modal surveys that utilize a combination of contact methods to conduct the poll, including landline phones, cell phones and the internet. You can look through the crosstabs of any of our surveys to get an idea of the different types of voters that each contact method reaches.

While academics and media outlets often prefer Random Digit Dialing (also know by the acronym RDD) for conducting surveys, our experience is that working off of an enhanced voter list with quotas or targets for certain types of voters generates the most accurate results in political campaign polls.

Using voter quotas places a premium on correctly projecting the makeup of the electorate. Failing to accurately project the electorate can doom your poll before you draft it. Fortunately, compared to most states, North Carolina’s approach to voter registration makes the job a bit easier. When voters in our state register to vote, they designate a party affiliation, note their gender, their age, and, until recent changes at the DMV, their race. This wealth of data, when cross-referenced with an individual voter’s history of participation in previous elections and further enhanced through various databases, provides a strong foundation to build a turnout model for our surveys.

What we think about when we build a turnout model

Rather than walk through all the State House and State Senate districts and how we arrive at the turnout model for each district, we will use a statewide example to illustrate some of the considerations that go into building a polling model. 

We take several demographics into consideration. In our experience, there are three tiers of data that need to be considered when building a model and setting contact quotas for a survey:

Tier 1: Party affiliation, Gender, Age and Race. Setting accurate contact quotas are fundamental to getting an accurate poll in a North Carolina district election. If you don’t properly account for these demographic characteristics, it is hard to get a quality survey.

Tier 2:  Vote history. This is the toughest target to set.  Since 2008 and especially since 2016, voter turnout has hit historic levels around the country and in North Carolina.  We’ve seen both a surge in voter registration and a surge in the percentage of registered voters who turn out to vote. The 2018 election set a record for turnout in a North Carolina mid-term election and that was without a high-profile statewide race on the ballot to drive voter interest. Likewise, the 2020 presidential election exceeded every forecast for turnout and blew away all previous voter participation records. Correctly identifying which voters to include in your sample can have a big impact on the accuracy of your results.

Tier 3:  Education, Income, Ideology. These demographics need to be considered but we usually choose to keep an eye on them as opposed to setting quotas for them.  There are two reasons for this: First, the Census Bureau conducts something called the American Community Survey that includes data on things like income and educational attainment (in North Carolina, they estimate $56,642 as the median income family income and that 32% of North Carolina residents have at least a college degree). In statewide surveys, this information is pretty accurate. At a district or county level, the margin of error in the data increases and it is harder to utilize as an accurate benchmark. Second, our experience shows voters are more likely to fudge on the income and educational attainment questions than others. This works both ways on the income question, with upper income voters understating their earnings and lower income voters overstating theirs. Voters also tend to slightly overstate their educational attainment. Ideological trends tend to change slowly over time. We’ve found that with experience you can identify an outlier and that it is particularly important to keep an eye on the ideology of Unaffiliated voters in any given poll, but it isn’t wise or effective to set quotas based on ideology.

 So how does all this work in practice?  Let’s look at statewide turnout in North Carolina in a few key demographics in the 2014 through 2020 elections:

We want to draw your attention to a few key numbers:

• Turnout has been about 50% higher in presidential elections than in mid-term elections.
• Republican voters turn out at higher rate than Democratic voters, who turn out at a higher rate the Unaffiliated voters.
• Women turn out at a higher rate than men.
• White voters turn out at a higher rate than black voters.
• And while we didn’t include it in this chart for simplicity’s sake, older voters turn out at higher rates than younger voters.

With those turnout numbers in mind, here’s a chart showing the vote share over the same period within the demographics:

Looking through this chart, you can see there some data points are consistent and some vary greatly. Of the voters whose gender we know, turnout has been consistently about 55% female and 45% male. Because the registration system no longer consistently collects information on voters’ race, the percentage of both white and black voters appears to be declining, but you can see the ratio of white to black voter turnout has consistently remained around 3.5:1.

Setting targets for party registration is a bit trickier. The percentage of North Carolina voters who choose to register “Unaffiliated” has surpassed the registration of both major political parties.
If you simply looked at statewide trends, you could be forgiven for assuming that every district poll ought to have more Unaffiliated voters in it every year. After all, anyone can look at this chart of statewide registration trends:

And while many counties are following the statewide trend, some are not. 

As an example, take Currituck County, where Democratic registration share is falling rapidly and Republicans could overtake Unaffiliated voters as the largest group of voters later this year:

Or Mecklenburg County, where Democratic registration is stable but Unaffiliated voters share is growing at the expense of Republican registration:

We review the trends in districts before we set our registration targets. (If you have a Differentiators Data subscription, you can use it to review these registration trends at the county level across the state.) 

Once we’ve reviewed and developed the data we need, we develop a turnout model.

For our 2022 legislative surveys, we will take a relatively cautious approach and focus on the 2018 election, a good year for Democrats, and the 2020 election, a relatively neutral environment for the two parties, and set targets based on a modified average of those two elections.

If we were thinking about it using the statewide example and the demographics we’ve reviewed, we would want our poll to come within a point or two of a statewide model that looks like this:

• Men 45% Women 55%
• Democrat 35% Republican 33% Unaffiliated 31%
• White 72% Black 20% All other races 8%

As the survey is conducted, we make sure that our pool of respondents reflects the model.