I’ve been checking the results I scraped (warning: 9Mb zip file, 100Mb uncompressed) against the national and provincial totals. And things aren’t adding up. That’s not a problem with the scraper, it’s a problem with the results as posted online.

Say, for example, we’re looking at the results for the 1979 election in Ragusa, a province with relatively few comunes.

Take the results for the DC in each of these comuni: Acate, Chiaramonte Gulfi, Comiso, Giarratana, Ispica, Modica, Monterosso Almo, Pozzallo, Ragusa, Santa Croce, Scicli, and Vittoria, and add them up. You should get (1438+2814+5190+922+2531+14781+1100+2564+17509+1534+4832+8812) = 64027. But the aggregate results reported for Ragusa as a province suggest 64054.

You can repeat this exercise for the Provincia di Trieste, which mercifully only has/had six comuni. Again, the totals don’t seem to add up: the DC vote is reported as 50,936, but the comune-level votes sum to 50854.

So either (a) I’ve made a mistake adding up correct (as-reported) results, (b) the Ministry of the Interior has made a mistake adding up correct (as-reported) results, (c) comune vote totals need not sum to provincia vote totals, or (d) the as-reported results at comune level do not sum to as-reported provincia vote-totals, or (e) some other scenario.

This might be due to rounding errors or other trivial mistakes — except that in some province the discrepancy is between 1,000 and 6,000 votes.

Any ideas?

There were (by my count) fifteen divisions, none of which were as neat as the divisions on the Select Committee report on hacking.

As before, I’ve used the results of the ideal point analysis to place legislators and amendments in a common space. Click to embiggen:

The strongest divide on the Joint Committee seemed to be between chambers rather than between parties. Towards the pro-reform end, Dan Poulter and John Stevenson (both Cons.) distinguished themselves, whilst at the opposite end Baronesses Andrews and Symons (Lab.) often made common cause with Lord Norton and Baroness Shephard.

Having said that, the votes which distinguish Poulter and Stevenson from the rest do seem to be more partisan. They occupy their extreme position because of a vote on para. 45, to insert “This lack of transparency has hampered Parliamentary scrutiny of the draft Bill”. Voting against that amendment comes out as pro-reform, but Poulter and Stevenson (presumably) voted against not because of beliefs about reform but because they did not want to embarass the government.

The full list of amendments I have is as follows:

1. Para 22 (now 23): to leave out “The Committee on a majority either agrees that the reformed second chamber of legislature should be elected [or] on a majority does not agree that the reformed second chamber of legislature has to be elected” and insert “agrees that the reformed second chamber of legislature should have an electoral mandate”.
2. Para 34: to leave out “would enhance” and insert “would not enhance”
3. Para 63 (now 67): to leave out “A majority, while acknowledging that the balance of power would shift, consider that the remaining pillars on which Commons primacy rests would suffice to ensure its continuation”.
4. Para 63 (now 67): to leave out paragraph 63 (now paragraph 67) and insert a new paragraph—”We received no clear evidence and reached no conclusion as to the limits of primacy. The Commons will continue to be the chamber through which the Government is elected and enjoy privilege in respect of finance. (We consider later the matter of the Parliament Acts.) Beyond that, it is difficult to reach conclusions as to the extent to which the Commons will be able to assert itself over an elected or largely elected second chamber. Even if the Parliament Acts remain in force, they are blunt weapons for determining outcomes and the regular use of their provisions would have the potential to create tension within our constitutional arrangements. The Draft Bill provides for no new arrangements for resolving disputes between the two chambers. We have not addressed how such disputes should be resolved. We consider that this will need to be addressed by Government before a Bill is introduced.”
5. Para 43 (now 45): to insert “This lack of transparency has hampered Parliamentary scrutiny of the draft Bill”.
6. Para 102 (now 107): that the paragraph be agreed to
7. Para 162 (now 167): to leave out “disagree with the Government’s proposal and recommend instead the possibility of re-election for future term be available”.
8. Para 168: to replace with “A majority of the Committee considers on balance that a 15-year term is to be preferred”.
9. Para 217 (now 222): to leave out “The Committee believes” and insert “Some members of the Committee believe”.
10. Para 218 (now 223): to leave out “Accordingly, we recommend that IPSA should make no provision for members of the reformed House to deal with personal casework, as opposed to policy work.”.
11. Para 262 (now 267): to add “We believe that these members should have the right to sit, but not to vote, in a reformed House”.
12. Para 284 (now 289): to leave out “bishops should no longer have ex officio seats in the reformed House of Lords.”
13. Para 285 (now 290): to leave out “that the number of reserved seats ought to be reduced to seven”.
14. Para 362 (now 367): to leave out “We leave the evidence of Lord Pannick and Lord Goldsmith to speak for itself”.
15. Para 376 (now 384): to add a paragraph as follows: “The Committee recommends that, in view of the significance of the constitutional change brought forward by an elected House of Lords, the Government should submit the decision to a referendum.”

Voting for amendments 1, 6, 7, 8 and 11 is considered “pro-reform”, on this analysis, although six and eight are far less discriminating. Voting for amendments 2-5, 9-10, 12-15 is considered “anti-reform”, although 2, 10 and 12 are far less discriminating.

If you want to replicate this analysis, data are here and here.

Code is as follows:

library(ltm)
library(ggplot2)
library(plyr)

ggsave("lords_pos.svg",width=4,height=7)
ggsave("lords_pos.pdf",width=5,height=7)


They’ve come up with a anti/pro-Murdoch score, based on number of anti-Murdoch amendments voted for, less the number of pro-Murdoch amendments voted for. That’s based on the Guardian’s own coding of pro- or anti.

That kind of information can also be recovered by ideal point analysis. The idea behind ideal point analysis is to assume that each legislator has an ideal point along the left-right spectrum, that they vote for (vote=1) proposals close to them, and reject (vote=0) proposals far from them. (Technically, they vote for proposals which move the status quo closer to them, but we can ignore that). Votes and legislators’ positions along the left-right space are recalibrated in order to make the most likely set of positions given the pattern of votes we see.

Here’s a plot of voters (Select Committee members) and motions in the same space. Click to embiggen, apologies for over-plotting.

Higher numbers on this graph indicate a more anti-Murdoch position. Lower numbers indicate a pro-Murdoch position.

The difference between ideal points analysis, and scoring people, is the treatment of missing information. The position for Adrian Sanders essentially ignores his abstention on amendments 4 and 9. If you view abstention as equivalent to absence, this is fair. If you view it as submission to the majority outcome (without the courage to say so), then these should be recoded as 1s, but might not have affected his position (4 would have made the report less critical, 9 more).

This information also gives information on the direction in which each amendment discriminated between positions close on the line. The sign of the discrimination parameter gives exactly the same information as the Guardian’s hand-coding (with the exception of one “neutral” amendment, which on this analysis comes out as anti-Murdoch), so we could have arrived at the same outcome without necessarily reading the text of the amendments.

To replicate this in R, use the code below.

library(ggplot2)
library(plyr)
library(ltm)
dat<-read.csv("Hacking select committee amendments - ALL AMENDMENTS.csv",header=F,skip=2,nrow=16,na.strings=c("","NA"))
title<-read.csv("Hacking select committee amendments - ALL AMENDMENTS.csv",header=F,nrow=1,as.is=TRUE)
party<-read.csv("Hacking select committee amendments - ALL AMENDMENTS.csv",header=F,nrow=1,skip=1,as.is=TRUE)
names(dat)<-as.character(as.vector(title[1,]))

## Get horizontal justification right
plot.df2<-plot.df2[order(plot.df2$Type,plot.df2$Position),]
plot.df2$hjust<-rep(c(1,0),length.out=nrow(plot.df2))
ggplot(plot.df2,aes(x=Type,y=Position)) + geom_point() + geom_text(aes(label=plot.text,hjust=hjust,vjust=0),size=2,angle=0,vjust=1) + theme_bw()
ggsave("hacking_pos.svg",width=4,height=7)
ggsave("hacking_pos.pdf",width=5,height=7)


About this graph:
This graph is based on poll results publicly available at http://www.sondaggipoliticoelettorali.it/. It shows the top six parties over the relevant period. A smoothed line (“loess fit”) is added to aid interpretation. The current values of that smoothed line are plotted at the end. The smoothed line does not (yet!) take account of variations in polling sample sizes or house effects.

Through the course of the Leveson inquiry, both James and Rupert Murdoch have argued that News International exercises very little influence of any kind, either over politicians or over voters. Rupert Murdoch argued that the Sun’s claim that it won the 1992 election for the Conservatives was both `tasteless and wrong’.

A number of commentators have rushed to agree with statements of this kind. Philip Collins wrote that the idea that `the public are dupes who digest their views whole from The Sun’ was `amazingly patronising’. Prospect political editor James Macintyre thought Murdoch was right to deny the power of the Sun to decide elections. And Andrew Roberts, in a piece that combined historical knowledge with astonishing non sequiturs, said that it was foolish to think of Rupert Murdoch as a kind of Darth Vader figure (though here Roberts was most likely thinking of Emperor Palpatine, whose mind control powers were much greater).

These claims, if they were true, would be comforting. If newspaper proprietors, through the editorial slant of the newspapers they own, do not influence voters, then politicians do not need to court newspaper proprietors in order to curry their favour. Policies — particularly those antithetical to the interests of those proprietors, or disliked by them or ideological or principled grounds — could be set freely by parties and their leaders. In other words, parties could present their platforms to the electorate and expect them to be voted on as such, rather than being handicapped from the outset.

A skeptical attitude towards proprietor influence is by no means outlandish. Indeed, many of those who have studied the media have tended to be very skeptical about the influence of the media.

The earliest empirical work on media influence, carried out in the United States in the forties and fifties, famously argued for a ‘minimal effects’ interpretation of media influence. Media influence — they argued — is much less important than the cues we take from others around is: family, friends, work colleagues, and opinion leaders. These cues, which are persistent and are not chosen anew every day, are far more important than the cues from the television stations we watch or the newspapers we buy on a given day. Whilst there might be an association between the media one consumes and the party one votes for, that association is present because left- and right-wing voters buy left- and right-wing newspapers, and not the other way around.

Whilst some skepticism might be warranted, some of the claims we’ve seen in defence of Murdoch seem either to miss the point, or to be too far-fetched to be credible. Those who rush to deny that Rupert Murdoch makes election runners into election winners — in other words, those who think that “It was the Sun wot won it” is a great calumny perpetrated on Labour politicians — are certainly right to say that Murdoch doesn’t determine the `winners’ of elections. But it’s possible to exercise a great deal of influence without determining the winner of an election.

If, instead of asking whether Murdoch causes certain parties to win or lose elections, we ask, `how many votes can Murdoch swing’, then the claim of negligible proprietorial influence becomes untenable. Are we really prepared to claim that the number of votes swung by News International newspapers is zero? Or if not zero, as likely to swing votes one way as the other? That really does seem far-fetched.

What we need then, is some kind of evidence which bears on the matter, evidence capable of disentangling the link between prior political inclination and choice of media. Fortunately, we have just the right kind of evidence — born from an improbable concatenation of events.

The evidence I am talking about comes from a paper by two American political scientists, Jonathan Ladd and Gabriel Lenz. Ladd and Lenz, like many other political scientists, use the British Election Study (BES), one of the longest running and best election surveys in the world. They were particularly interested in the 1997 study, because they had noted two things: first, that the Sun had `switched’ to Labour in a big way on the second day of the campaign; and second, that the BES had started surveying a large number of potential voters quite some time before this switch.

Bivariate relationship only; effect strength increases after matching on covariates

Because the election study was a panel study — that is, the same respondents had been polled on multiple occasions — Ladd and Lenz had a large bank of voters, some of whom read the Sun, and some of whom didn’t.

And because the BES sampled so many voters, and asked so many questions about age, education, occupation, and left-right positioning, Ladd and Lenz were able to match up Sun-readers to statistical doppelgangers, who were nearly identical in every respect save reading the Sun, and compare their behaviour and reported vote intention both before and after the Sun’s switch. It was, in other words, a way of creating an identical twin study after the fact, and thus accounting for many of the background characteristics that make us both more likely to vote a given way and read a particular newspaper.

What did Ladd and Lenz find when they compared these two groups — those who had read the Sun as it switched, and their statistical doppelgangers who had not? Based on their matching, they estimated that somewhere between 8% and 20% of Sun readers — between 312,000 and 780,000 voters — voted Labour as a result of that paper’s switch. That number is large — certainly not large enough to account for the 3.9 million votes that Labour won in excess of the Tories, but large enough to substantially alter the number of seats that Labour won in that landslide year.

That number is also large enough to matter in elections that don’t produce a clear winner. Many people have interpreted the fact that Murdoch newspapers always back `winners’ to mean that Murdoch picks likely winners and sides with them. That is, rather than making winners he discovers them. But in some elections, there is no real winner, and here margins matter. The Conservatives won a plurality of votes in the last election, but failed to win a plurality of seats. Coalitions excluding the Conservatives were mathematically possible after the 2010 election, and would have been much more likely if as few as six to ten seats had changed hands. If Ladd and Lenz are to be believed, the vote swing necessary to make up that six to ten seat swing is well within News International’s grasp.

I am not suggesting that we re-run the 2010 election and ban the Sun from expressing its opinion. I am arguing that we should be very aware of the potential of newspaper proprietors like Rupert Murdoch to influence public opinion in significant ways, and ways which can be deleterious to the public good. Being aware of this influence does not automatically lead to support for restrictions on ownership. One might consistently believe that newspaper proprietors have tremendous influence, but that that influence is perfectly licit. But I am not sure that that view would hold much water if estimates of the persuasive ability of newspapers of the kind put forward by Ladd and Lenz were readily believed.

Of course, good reasons for restrictions on newspaper ownership are no guarantee of their introduction. It would take a very courageous Labour party, confident of winning in a near-landslide, to forgo the support of News International newspapers by promising to undo the mistakes it made in the 2003 Communications Act. But it is perhaps more likely that Labour will win back some courage than it is likely that Rupert Murdoch will give up on the influence he claims not to wield.

For the past thirty years, the state’s role in funding broadcasting
has been under attack. The rise of free market ideas and considerable
scepticism regarding the role of the state and state-owned enterprises
signalled a period of decreasing political support for public service
broadcasting. The introduction of cable and satellite television and,
subsequently, the advent of digital terrestrial television, have ended
or considerably ameliorated previous problems of spectrum scarcity.
Public service broadcasters have, in many cases, had to accept that
competition in media markets is the default option.

Because of these developments, a declinist narrative has taken over the
study of public service broadcasting. Academics such as Michael Tracey
have concluded that public broadcasting has struggled to survive in the
face of this political, ideological and technological change and is now
experiencing a terminal decline (Tracey, 1998).

The purpose of this report is to give an overview of how public service
broadcasters around the world are structured and funded, and what is
required of them through law. The report examines the extent to which
public service broadcasters’ output is distinctive, of high quality and
capable of making a difference. The report concentrates exclusively on
television broadcasters and predominantly on broadcasters operating at
a national level. This comparative perspective should enable policymak-
ers to decide whether the objectives that they set for public broadcast-
ers are commonly shared, and whether they are feasible. It suggests
that public service broadcasting enjoys far ruder health than the declinist
narrative suggests.

That’s fine.

Because this kind of stuff is most fun(!) when replicated, I’m posting all of the original data and R code I used for my analysis.

Apologies if it’s not as tidy as you might like.

Here’s the zip file.

As such, it’s important to know whether the results of the REF could be approximated using other proxy measures. Patrick Dunleavy has argued vocally for using bibliometrics, for example. In this series of posts, I’m going to investigate how predictable the REF is — and whether we can generate predictions now based on leading indicators of “research excellence”.

Warning: if you’re using versions of Internet Explorer before 9, you may need to click on the PDFs

In the previous post, I gave a model of RAE/REF performance based on ESRC grant income, the number of successful ESRC applications, staff numbers, and University grouping. In this post, I’m going to use a slightly different model to generate predictions for 2014 based on research income until 2012.

The model is similar, but lacks the standardized FTE staff numbers. Here’s the readout for those playing along at home.


Calls:
my.mod.lmer: lmer(formula = StdRating ~ log1p(RunSum5) + log1p(RunCount5) +
Grouping + I(PanelMembers > 0) + (1 | Institution), data = subset(esrc.5,
Year %in% c(1992, 1996, 2001, 2008)))

===============================
(Intercept)            0.320***
                      (0.035)
log1p(RunSum5)         0.008**
                      (0.003)
log1p(RunCount5)       0.040
                      (0.034)
Grouping1994           0.159***
                      (0.044)
GroupingRussell        0.158***
                      (0.045)
I(PanelMembers > 0)    0.074*
                      (0.036)
-------------------------------
Aldrich-Nelson R-sq.
McFadden R-sq.
Cox-Snell R-sq.
Nagelkerke R-sq.
Likelihood-ratio
p
Log-likelihood          60.281
Deviance              -156.191
AIC                   -104.562
BIC                    -76.295
N                      253
===============================

The interesting part comes when we feed in data on ESRC income up to 2012, and generate predictions on that basis. I’ve used the mcmcsamp command in the lme4 package for R to sample from the posterior distribution of the model parameters. That means that not only can I generate credible/confidence intervals for the estimated quantity (standardized RAE/REF score), but I can also generate credible intervals for auxiliary quantities such as rank information — which in this context may be moee important.

Here, then, is the first graph showing the expected REF scores for each institution. Because the total levels of funding for 2012 are partial, the average REF score across all institutions was down on the 2008 level. I’ve consequently removed this by making the mean 2014 prediction equal to the mean 2008 result. Differences from 2008, after this correction, are plotted to the left of the graph. Credible intervals are 80% credible intervals.

Click to see in PDF

Perhaps more interesting from an inside-baseball perspective are the predicted ranks and associated 80% credible intervals. Here, the (rounded) 1st decile and 9th decile ranks are given at the right hand side of the plot.

Click to see in PDF
Note that some of the estimated gains in median rank w.r.t 2008 rank are a bit ropey at the top and bottom as a result of rounding.

What can we say on the basis of these predictions?

• Essex and Oxford seem to be in a two-horse race for the top position
• The lowest eleven-placed departments are largely fitting amongst themselves
• The biggest gainers w.r.t. 2008 are Liverpool (up 32), Westminster (25) and Leeds (up 18)
• The biggest losers w.r.t. 2008 are Queen’s (down 27 spots) and Bradford (down 21).
• Cambridge seems to do particularly badly on this prediction, especially relative to
  some rankings which place them 3rd in the world for Politics and IR

Of course, these are predictions about a ranking which will be carried out in 2013/2014, made on the basis of data available in the first quarter of 2012. Predicting rankings is doubly difficult, because of the loss of information inherent in ranking instead of producing point estimates.

Nevertheless, the 80% credible intervals surrounding the rank information may provide a useful ballpark estimate. Indeed, I am happy to wager an all-costs-personally-incurred lecture at your institution if you manage to outperform (or underperform) the range of ranks indicated in the right hand side of the graph. Of course, you would have to wager the same, and given that these are 80% credible intervals, and the stakes are even, then I have considerable “edge”.

In the next post, I’ll upload all of the materials I used to produce these estimates. Comments welcome in the meantime.

As such, it’s important to know whether the results of the REF could be approximated using other proxy measures. Patrick Dunleavy has argued vocally for using bibliometrics, for example. In this series of posts, I’m going to investigate how predictable the REF is — and whether we can generate predictions now based on leading indicators of “research excellence”.

In previous posts I’ve explored the variability of RAE scores (and how to put them on a common metric), and how to link these scores to ESRC grant income. In this post I’m going to develop a fairly full multilevel model of RAE scores.

The first problem we face is that our observations — RAE scores for years 1992, 1996, 2001, and 2008, and associated grant income for the five years preceding each assessment exercise — are not independent. We have multiple observations for each institution over time. So a straightforward OLS regression isn’t enough. Here, I’m using a multilevel model with a random intercept for each institution. These help mop up a bit of the variation in scores, and indirectly address the problem of AHRC-intensive universities.

The second problem is that our dependent variable — RAE scores standardized to lie in the range [0,1] — are bounded, and so are likely to give rise to more severe errors near the bounds. This turns out not to be a problem, since there’s no evidence of heteroskedasticity, and taking the log of the outcome doesn’t really change the findings.

I’m going to include a number of variables in the model. They are as follows:

• log1p(RunSum5): this variable is based on the running sum of ESRC grant money calculated over the past five years. For the RAE in 2008, this variable is based on grant money over the period 2003:2008. This variable is then transformed by adding one to it, and then taking the log.
• log1p(RunCount5): this variable is based on the running sum of successful ESRC applications. This is calculated in the same way as the above variable
• log1p(StdStaffFTE): this variable is based on the number of full-time equivalent staff submitted to the RAE. Again, it has been transformed in a similar fashion to the above variables
• log1p(StdStaffFTE) * log1p(RunSum5): this variable represents the interaction effect of the two constituent terms
• Grouping: this variable is a factor with three levels — “Russell”, “1994″, or “Other”. This reflects group membership in 2011, i.e., before the recent defections from the 1994 group
• PanelMembers: this is a dichotomous variable which takes the value 1 if the institution has members on the RAE/REF subject panel

Each of these variables seems reasonable to me. The bivariate correlations shown in the previous post suggest including ESRC grant money; but equally a department with a large number of successful grant applicants, each with moderate income, might result in a better RAE/REF score than a department with a single superstar researcher, motivating including ESRC grant successes.

The number of full time REF submitted staff reflects a size effect (moderated slightly by strategic submission). The Group variable is mostly a way of mopping up extra variance. Panel membership has been shown by previous research to have an effect.

The full model, in its ugliest form, is pasted below

Calls:
my.mod.lmer: lmer(formula = StdRating ~ log1p(RunSum5) + log1p(StdStaffFTE) +
log1p(RunCount5) + Grouping + (1 | Institution) + I(PanelMembers >
0), data = subset(esrc.5, Year %in% c(1992, 1996, 2001, 2008)))

===============================
(Intercept)            0.156**
                      (0.053)
log1p(RunSum5)         0.007*
                      (0.003)
log1p(StdStaffFTE)     0.083***
                      (0.023)
log1p(RunCount5)       0.036
                      (0.034)
Grouping1994           0.111**
                      (0.040)
GroupingRussell        0.088*
                      (0.043)
I(PanelMembers > 0)    0.074*
                      (0.036)
-------------------------------
Aldrich-Nelson R-sq.
McFadden R-sq.
Cox-Snell R-sq.
Nagelkerke R-sq.
Likelihood-ratio
p
Log-likelihood          62.799
Deviance              -167.869
AIC                   -107.597
BIC                    -75.797
N                      253
===============================

In terms of the explanatory power, the pseudo-R-squared based on the square of correlation between fitted and actual values is high (0.64), higher than the Nagelkerke-style pseudo-R-square based on log-likelihood ratio (0.47).

What does this mean in human-accessible terms? Well, it’s simplest to start with the interpretation of the dummy variables. Universities in the 1994 group, other things being equal, are associated with standardized RAE scores which are 0.111 units greater on a zero-to-one scale — or almost half of the way between 3* and 4*, on the 2014 REF scale.

The impact of ESRC funding isn’t huge. An extra million over the five year period will gain you just less than 0.1 units on this standardized scale, slightly less than the amount you would gain from being a Russell group university.

Having a member on the relevant subject panel does have a significant effect. The substantive magnitude of this effect is less than an extra million in ESRC funding over the five years, but less than being a member of the 1994 group or Russell group. Still, at 0.07 on this standardized scale, or a quarter of a unit on the REF 2014 scale, it’s nothing to sniff at.

The impact of FTE staff submitted seems to be the most substantively important variable, which perhaps reflects a sheer size effect, and perhaps reflects strategic submission and accumulation of talent in larger departments. Submitting an additional staff member (assuming constant-quality staff) is associated with an increase of 0.69 units, far greater than any of the changes discussed so far.

Because information on FTE staff submissions won’t be known until after the REF, I won’t be using this extremely important variable to predict REF scores. But it is extremely useful for retrodiction, and suggests that a poaching strategy could have quite a considerable pay-off.