UCLA's press release begins:
While the editorial page of The Wall Street Journal is conservative, the newspaper's news pages are liberal, even more liberal than The New York Times. The Drudge Report may have a right-wing reputation, but it leans left. Coverage by public television and radio is conservative compared to the rest of the mainstream media. Meanwhile, almost all major media outlets tilt to the left.
These are just a few of the surprising findings from a UCLA-led study, which is believed to be the first successful attempt at objectively quantifying bias in a range of media outlets and ranking them accordingly.
The "fact" that the Drudge Report "leans left" is not just a "surprising finding," though. It's a flashing red light that says, "Warning! Bad Data Ahead!" For any study that kicks out a result like this must surely have something fundamentally wrong with it.
The other "surprising findings," in contrast, are not so surprising to serious media critics--notably those on the left. It's long been known that The Wall Street Journal does more hard-hitting reporting than The New York Times--the non-ideological translation of the UCLA press release--and FAIR (Fairness and Accuracy In Reporting) has long pointed out that public broadcasting is quite corporate-friendly and citizen/activist averse--particularly in its flagship programming. (See for example, back in 1993, "New Study Reveals Public TV Bias And Debunks Conservative Myths".) So what we have here--at least as advertised--is a mix of bad data signalling a flawed methodology, and an ideologically slanted report of stuff that left/liberal media critics have been saying for over a decade.
Big news, no?
In fact, there are a number of anomalies that stick out, indicating that something is seriously wrong with the methodology. For example, the ACLU and the NRA have very similar scores--49.8 and 45.9--both centrist, though slightly to the right. The authors take note of the ACLU's position, and explain why--its opposition to campaign finance restrictions was cited repeatedly by Congressional opponents:
"In fact, slightly more than one-eight of all ACLU citations in Congress were due to one person alone, Mitch McConnell (R.-KY), perhaps the chief critic of McCain-Feingold."
It does not seem to occur to the authors that this may be indicative of a much more pervasive problem with their basic, untested assumption that politicians cite sources for ideological reasons, rather than pragmatic ones. And they did not seem to be troubled by NRA's position at all, focusing instead on the RAND Corporation as a second anomaly that needed special attention.
There's also the little matter of calling organizations like the NRA and the ACLU "think tanks." They're not. Putting advocacy organizations and think tanks into the same dataset creates a classic example of comparing apples and oranges.
A prominent critic of an earlier incarnation of this paper, Geoff Nunberg, pointed out (among other things) that the list of groups used is deeply problematic:
Start with the list of groups from which G & M drew their initial sample. They describe this simply as a list of "the most prominent think tanks," but that isn't quite accurate. In fact their list was drawn from the 200 links included on the site wheretodoresearch.com (which actually describes it merely as a list of "major think tanks and policy groups"). The list was compiled by one Saguee Saraf, a free-lance researcher with a masters degree in history who lists among his achievements that he was named Man of the Year by the Cheshire (Connecticut) Republican Town Committee.
Saraf gives no indication of how his list was compiled, or what criteria were used -- nor, more to the point, do Groseclose and Milyo say why they consider the list authoritative. In fact its contents are a jumble of think tanks, lobbying groups, trade associations, and advocacy groups, assembled in a catch-as-catch-can manner....
Then, too, Groseclose and Milyo's survey of the citations of groups in the Congressional Record shows some results that would most kindly be described as puzzling. In their list of the "twenty think tanks most cited by members of Congress," for example, they list in 13th place the Alexis de Tocqueville Institution (which they refer to as the "Alexis de Tocqueville Institute"), which comes in ahead of Common Cause (14th), the Family Research Council (16th), and the Economic Policy Institute (19th), not to mention a number of much better-known groups that appear on Saraf's list but not in G & M's top 20, like the NRA and the Hoover Institution.
FAIR itself currently has a front-paged a comment (permalink here) on this report, with an excerpt from an article in FAIR's newsletter, Extra!, which comments on an earlier incarnation of the paper. The bolded text is where they really poke the hole in the center of this whole charade:
A study being published in the Quarterly Journal of Economics has been getting some attention for its supposed quantification of bias in the media. An earlier incarnation of the study was discussed in the May/June 2005 issue of Extra!:Academics Tim Groseclose and Jeff Milyo got considerable attention for a paper they wrote called "A Measure of Media Bias" (12/04), which deduced a "strong liberal bias" from an analysis of news outlets' use of "think tanks." (The groups the study looks at are actually a combination of think tanks and advocacy groups.)
The report used a peculiar Rube Goldberg–like method to calculate media bias from think tank citations: Taking the Americans for Democratic Action ratings of congressional voting records as its yardstick, it assumed that media outlets have ideologies similar to those of members of Congress who cited the same think tanks that the media outlets did.
This approach is based on the problematic notion that politicians cite the think tanks that they most agree with rather than the ones whose citation will be the most politically effective; a problem the researchers acknowledge when they attempt to explain away some curious anomalies that their method produces. (The National Rifle Association comes out as a centrist group; the Rand Corporation turns out to be left-leaning.)
If the authors truly wanted to rank media outlets on the ADA scale, the simpler method would be to look at the ADA ratings of congressmembers quoted by those news outlets. One suspects that the authors avoided this obvious approach because the results would have been less to their liking: Studies in Extra! have repeatedly found various media outlets quote Republicans more often than Democrats, by ratios ranging from 3 to 2 on NPR (5–6/04) to 3 to 1 on nightly network news (5–6/02) to a startling 5 to 1 on Fox News Special Report (7–8/04). Fox News, according to Groseclose and Milyo's method, is a "centrist" news outlet.
The above is an excerpt from an article by Michael Dolny, "Right, Center Think Tanks Still Most Quoted: Study of cites debunks 'liberal media' claims". Dolny, a sociology professor, has been doing these studies annually since 1996, when he was an intern at FAIR. Unlike Groseclose and Milyo, he actually restricts himself to just looking at think tanks. The article explains:
The study counts citations of the 25 most prominent think tanks of right, center and left, using the Nexis news media database. Citations are counted in what Nexis designates to be major newspapers, as well as in Nexis’ transcripts file, which includes the major broadcast and cable news outlets. Because stories included in the Nexis database change over time, figures for previous years are recalculated for comparison purposes rather than taken from previous editions of the study.
Conservative or right-leaning think tanks garnered 50 percent of citations among the 25 most-cited think tanks, the same percentage as last year, and near their 10-year average of 51 percent of citations. Centrist think tanks declined slightly this year, garnering 33 percent of the citations, compared to 37 percent last year and 36 percent as their 10-year average. Progressive or left-leaning think tanks had the greatest percentage increase this year, receiving 16 percent of citations, up from last year’s 13 percent and their 10-year average of 14 percent.
This is what's really happening in the media, when it comes to citing think-tank sources--an overwhelming predominance of conservatives. Dolny's work is not nearly as complicated as Groseclose & Milyo's. But, then it doesn't have to be. It's not trying to stand reality on its head.
Groseclose & Milyo published in an economic journal precisely because it uses economic models that come out of an ideological project to reinterpret the world in terms of conservative economic ideology. While there's no direct link between the economic ideology and the outcome of their research, it is not surprising that the two go together. They reflect a common attitude of beginning with an ideological outlook, and then looking for data to support it. There is nothing wrong with this per se. But if it's not accompanied by a willingness to take criticism seriously, and respond to it fairly, then the end result is not good science, and arguably, even, not science at all.
Bumper Stickers & MORE
