Home > Applying Political Science, California Politics, Elections, Research > CrowdPAC, Candidate Ideology, and Measurement Validity

CrowdPAC, Candidate Ideology, and Measurement Validity

So there’s this new organization called CrowdPAC that aims to offer “The best objective data on US political candidates.” Using a somewhat undefined methodology (although there is more technical information available here), built on the work of Stanford Political Scientist and co-founder Dr. Adam Bonica, it rates each candidate in all of the various congressional contests on a 10L to 10C scale, with 10L being the most liberal and 10C being the most conservative. (There are a handful of people they rate 10L+ or 10C+, which means they are off the scale liberal or off the scale conservative.) Dr. Bonica’s work recently received some coverage over at the Monkey Cage, most notably because it challenges the dominant narrative that modern partisan polarization is being driven by Republicans rather than Democrats.

I am particularly happy to see this kind of work, especially because it is about candidates and not elected officials. (Much of my current research interest revolves around Proposition 14 and its effects, which are supposed to be about candidate behavior.) Boris Shor has another set of data on candidate positions, but he hasn’t released an update for 2014 yet.

I’m a little dubious of this data, however, and it is the score that they give Jerry McNerney (CA-9), the local representative here in the Central Valley, that spurred my concern. In 2012, according to Shor’s data, McNerney was more conservative than the average Democrat. I made the following graph last election cycle to demonstrate the ideological positions of McNerney and his 2012 Republican challenger, Ricky Gill. McNerney scored relatively close to 0, which is politically moderate.


Shor’s measure tracks well with other measures of ideology as revealed through voting in Congress. The most commonly used measure of ideology in congressional studies is Poole and Rosenthal’s NOMINATE scores. (There are others, but this is the one that everyone talks about.) NOMINATE scores range from -1 (the most liberal member of Congress) to +1 (the most conservative member of Congress). Here too, McNerney comes out fairly moderate (-0.225). All of this should not be surprising for a candidate who has made veterans’ issues and clean energy his primary policy talking points.

So how does McNerney score in the CrowdPAC data? He gets a 9.4L, which is more liberal than Nancy Pelosi (6.8L), Barbara Lee (8.5L), and Maxine Waters (7.1L)–all of whom have strong reputations as liberal stalwarts in both Democratic and Republican circles. (Their respective NOMINATE scores are -0.398, -0.694, and -0.594.) Indeed, as the following graph shows, McNerney’s score puts him near the liberal edge of all Democratic candidates running in California. (Note: I’ve reversed the sign for liberal rankings so that a 10L is a -10 in the following graphs.)


So something seems a bit off here.

One of the concepts we talk about in research methods is measurement validity–the degree to which a measure accurately captures the concept that you are trying to measure. One way to assess measurement validity is to compare its values with other, more well known measures of the same concept. (We call this convergent validity and use it to assess a measure’s content validity. This isn’t the whole of how we should test for content validity, but it is instructive as a start.) So let’s do that.

The following graph compares NOMINATE scores from the 112th Congress with the CrowdPAC measure for California’s candidates. While the two measures appropriately lump Republicans (upper right) and Democrats (lower left) together, there actually isn’t that strong of a correlation between the two measures within each party. Perhaps this disparity isn’t that surprising given that Dr. Bonica’s work disagrees with Poole and Roenthal’s on some important points, but it does raise concerns for me.


An alternative measure that I have been playing with recently is the National Journal’s conservative ranking. Similar to Poole and Rosenthal, National Journal examines member’s voting records and rates them on a liberal-conservative index. In this index, 0 is the most liberal a member can be and 100 is the most conservative. The rankings shown in the graph below are for current members of Congress. Again, while the groupings are roughly correct, within each party there isn’t much connection between the two scores. In one case, Eric Swalwell (CA-15), CrowdPAC offers a dramatically different rating than National Journal. (Removing Swalwell actually lowers the correlation coefficient for the Democrats to 0.015.) McNerney by National Journal’s measure is on the moderate side of the California Democratic delegation with a score of 38.5.


For the record, the correlation coefficient for the National Journal rating and NOMINATE for Democrats is 0.645 and for Republicans is 0.698. Unlike all of the above coefficients, these last two are statistically significant at conventional levels (i.e., α = 0.05).

All of this is to say that I am somewhat torn about this new measure. On the one hand, it is fantastic to get any measure of candidate ideology. It allows us to assess questions of whether competitive districts lead to more moderate candidates. The answer according to this data appears to be: No, competitive districts do not lead to more moderate candidates, even controlling for the presence of an incumbent. The following graph shows the CrowdPAC score plotted against the normalized presidential vote (NPV; i.e., how much more or less a district supported Obama in 2012 compared to the average district nationwide). If the competitiveness of a district was related to the ideology of the candidates running in it, we ought to see the dots getting closer to each other as the NPV gets closer to zero. (The lines are LOWESS regression lines for each party’s members.) They don’t.


On the other hand, I have serious doubts about the validity of the measure. So my hope is that in future iterations it will get better.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: