Congress, Earmarks, and Doing Analysis Right

This morning the Washington Post has an “explosive” story (yes, I mean that with all due sarcasm; feel free to add more if you think it necessary–and you probably should) about how Members of Congress (MC’s) sometimes push for earmarks (federal funding for projects in their districts) that affect areas near or next to where they live. The Post identified 33 Members who pushed for such projects. The not so subtle implication from the story is that MC’s are using their positions within Congress to improve their property values, thereby making themselves richer. The story comes on the heels of recent news about MC’s possibly using insider information to game the stock market–although see here and here [edit: links added] for evidence that MC’s generally do worse than the market–and the Senate’s recent vote to explicitly ban insider trading (the STOCK Act).

The Post’s story, however, is a classic example of the media doing data analysis poorly in the name of gotcha story. Here are a few criticisms:

(1) We ought to expect a correlation between where Members of Congress live and the location of the projects they push for. First, MC’s are supposed to fight for projects in their districts. Second, MC’s live in the districts they represent. Third, districts are a function of population; urban districts will be geographically small because they have more people living closely together. Taken together, these three conditions mean that it should be more likely that urban members will more frequently “benefit” from the location of the projects. Given the fact that the Post allows for projects to be “within two miles” of a MC’s home while still insinuating that the Member is “benefiting” from the project, this urban bias should be exacerbated. Eyeballing the Post’s data, the number of urban representatives clearly outnumbers the number of rural representatives. Even within rural districts, the population is not distributed evenly–people tend to live close to each other rather than apart.

(2) Quoting the story, “Mere proximity to a lawmaker’s property does not establish that an earmark was unwarranted. In some cases, the public benefit of the spending was large, improving life for thousands. In others, the benefit appeared narrower. In some cases, the work was within a mile or two of the properties; in others, it was directly in front of the lawmaker’s land.” Despite this statement, the implication from the title and the story’s lead is that, in fact, the projects were unwarranted.

(3) The story provides no context for the number of projects that happen to be located close to an MC’s home. How many projects in total are there? Moreover, when were they authorized? The closest we come to any kind of context is a statement that since the moratorium on earmarks was agreed to by Congress last year (2011), a single Member, Sen. Claire McCaskill (D-MO), has identified over “100 special spending provisions.” The data analyzed by the Post, however, stretches back to 2008, several year before the earmark moratorium. By comparison, the CBO identified over 11,500 separate earmarks in the FY2008 appropriations bills.

Given (1) and (2), and given the time frame for the analysis and therefore the number of such projects that exist, random chance alone would mean that some projects end up close to a Member’s home. What we need to know, and what the Post’s story doesn’t tell us, is whether the colocation of projects and Members’ homes occur more frequently than we might expect from random chance. My expectation, and I am happy to be proven wrong, is that they do not.

At the end of the day, what we have here is a classic example of a major data analysis error: selecting on the dependent variable. Rather than examining all of the possible locations where earmark projects might be located, the Post began with the locations of Members’ homes and looked to see if there were any such projects nearby. It is not at all surprising, therefore, that they found a connection.

About these ads