Universities across the country are preparing their submissions to the Research Excellence Framework (REF). This is the fourth such research assessment exercise, and the first three were indeed called by that name. It also seems the most onerous of the series, with an added impact element that has been the subject of much interesting commentary.

As such, it’s important to know whether the results of the REF could be approximated using other proxy measures. Patrick Dunleavy has argued vocally for using bibliometrics, for example. In this series of posts, I’m going to investigate how predictable the REF is — and whether we can generate predictions now based on leading indicators of “research excellence”.

Now that I’ve explained where my information on ESRC grant money comes from, I can start to use it to explain performance in research assessment exercises. I’m going to show the simple bivariate association in this post, and move on to different models in subsequent posts.

In order to predict performance in the research assessment exercise at time t, I use a running sum of ESRC grants with a five-year window. So, my predictor of performance in the 2008 RAE is based on ESRC research income in the period 2003:2008.

The figure below shows the Spearman correlation between ESRC grant income and RAE success for four different years and two different disciplines. The axis ticks on the x-axis are a bit messed up — they’ve been put on a log(1+x) scale to make it possible to plot zeros and multimillion pound departments on a reasonable scale.

Economics tends to have a much stronger association between ESRC income and RAE performance, no doubt due in part to its greater reliance on that body as the predominant source of funding. But the correlations across all years and both subjects are reasonable: and the correlation between ESRC money and RAE performance for the last year available, 0.708, is greater in magnitude (and much more defensible in its method) than the correlation between that performance and performance in the preceding exercise.

These simple bivariate relationships show that there’s something interesting going on here, worth exploring. At this point it’s worth exploring a couple of possible objections.

Research grants don’t measure the excellence of research, just ability to craft a grant proposal.

Entirely possible. But the same argument — about a disjuncture between intrinsic quality and recognition thereof — can be made for many types of external recognition, including publishing and citations. At least research grants are based on evaluation by peers and are (unlike citation counts) unambiguously positive indicators.

I do 4* research, but I’ve never received an ESRC grant.

I imagine that’s a reasonably common scenario. But I’m not claiming that the relationship between ESRC grant income and research ‘excellence’ holds at an individual level, only at the departmental level.

My department attracts most of its research income from the AHRC, not the ESRC.

If this is genuinely a characterisation of the department, then yes, these estimates may be problematic. If research income is related to quality, if there are multiple funders of research, and if revenue from one source does not correlate with revenue from the ESRC, then scoring on the basis of ESRC income would systematically underestimate quality. Unfortunately, I couldn’t get AHRC grant data in a readily available format. I can only say that in the full model I present later, random institution-intercepts account for this somewhat.

ESRC grant income arrives today — but the research outputs arrive tomorrow. You shouldn’t use grant income today to predict REF performance today; you should only use grant income today to predict REF performance tomorrow.

That’s possible. ESRC income may be a leading indicator of quality. And so I experimented with different lag structures — one year, two years, and so on. The strength of the correlation doesn’t vary much, giving us few empirical reasons to lag the data. And you might think that ESRC grant income is even a lagged indicator of research quality. Most people know researchers who follow the PhDComics grant cycle.

Universities across the country are preparing their submissions to the Research Excellence Framework (REF). This is the fourth such research assessment exercise, and the first three were indeed called by that name. It also seems the most onerous of the series, with an added impact element that has been the subject of much interesting commentary.

As such, it’s important to know whether the results of the REF could be approximated using other proxy measures. Patrick Dunleavy has argued vocally for using bibliometrics, for example. In this series of posts, I’m going to investigate how predictable the REF is — and whether we can generate predictions now based on leading indicators of “research excellence”.

The main proxy measure that I’m going to use to retrodict/predict performance in the REF/RAE is the amount of money received in grants from the Economic and Social Research Council. I’m going to defend a number of in-principle objections to using this as a proxy in the next post. For the moment, I just want to discuss where the data come from, and what’s been done to them.

All the data here come from a ScraperWiki scraper written by Julian Todd.

The scraper outputs data with information on “Discipline”, “Grant holder”, “Grant amount”, start- and end-dates, “Institution”, a free-text description, and a number of other fields.

Here, I’m restricting myself to grants with Discipline fields which contain either “[p|P]olitic*” or (for comparison) “[E|e]con*”.

I have then recoded all of the 293 institutions listed into either one of the institutions which submitted RAE entries in previous years, or a large generic “Other” category.

Given that I will eventually be using this data to predict outcomes across a large number of universities, it’s helpful to see the number of institutions and dispersion of grants across each year. As the figure below shows, there are on average twenty institutions per year winning grants in the discipline of Politics and IR; the figures for Economics are slightly higher.

Yearly number of institutions winning ESRC grants in selected disciplines

What surprised me was how dispersed grants were across these twenty-odd institutions. Recently there’s been much talk about the concentration of resources amongst a top-tier of universities. Perhaps that talk is a motivated reaction to figures like these, which show that the share of the top-three, top-five, and top-ten institutions on a yearly basis.

Certainly, I expected to see the top-five institutions taking up something like half of the annual per-discipline spend. The figures are slightly more dispersed for economics (not shown here).

The good news is that because ESRC money is (relatively) dispersed, we can use it as a meaningful predictor of RAE/REF performance for a wide range of institutions. That’s what I’ll discuss in the next post.

Universities across the country are preparing their submissions to the Research Excellence Framework (REF). This is the fourth such research assessment exercise, and the first three were indeed called by that name. It also seems the most onerous of the series, with an added impact element that has been the subject of much interesting commentary.

As such, it’s important to know whether the results of the REF could be approximated using other proxy measures. Patrick Dunleavy has argued vocally for using bibliometrics, for example. In this series of posts, I’m going to investigate how predictable the REF is — and whether we can generate predictions now based on leading indicators of “research excellence”.

One of the most obvious ways in which to assess the predictability of the REF, or similar exercises, is to check whether performance at time (t) predicts performance at time (t+1). The more past performance predicts present performance, the more we can rely on historical allocations, putting our fingers on the scale for departments that have made big improvements.

In order to do this, we must put RAE/REF rankings on a common metric. The current REF uses five grades (Unclassified, 1*, 2*, 3*, 4* — one wonders what extra force the stars add), the 1996 and 2001 RAEs seven grades (1,2,3a,3b,4,5,5*), and the 1992 RAE six (as above, but a single ’3′ grade in place of 3a and 3b).

If we assume that these grades are all equally spaced — and you might reasonably think that a 2001 ’3a’ and ’3b’ are closer together than a 2001 ’1′ and ’2′ — then we can convert these grades to a common metric by setting the lowest grade to zero, the highest grade to one, and dividing accordingly.

The figure above shows the pairwise correlations between average REF/RAE grades for departments across the four research exercises so far, for departments submitting returns in “Politics and International Relations”. All of the correlations reported are rank correlations, because in 1992 and 1996 the grade the department received was a single grade, rather than an average of grades.

The correlations are all reasonably strong. They wouldn’t pass the (somewhat arbitrary) 0.7 rule of thumb for test-retest reliability — but since we are allowing for department performance to vary over time, that’s not a problem.

These correlations are weaker than similar correlations for the REF subject heading “Economics and econometrics”, all of which are greater than 0.7, and one of which (1992 to 1996) is very high indeed, at 0.9.

We can show some of the variability that is present in the data by using a bumps chart. Plotting all of the year-to-year variation in a bumps chart gets messy very quickly, so here I’ve focused on nine of the more variable departments.

No doubt individuals familiar with the histories of these departments will be able to say what happened such that RAE performance was so volatile. Volatile departments such as these represent exceptions to the general rule of stability across RAE cycles. Whether the new REF, with its additional impact elements, will shake up the rank-ordering of universities substantially, remains to be seen.

It seems like calls for evidence on pluralism are ten-a-penny these days. Here are some submissions that I’ve done with wonderful colleagues from UEA, Shaun Hargreaves-Heap, Michael Harker, John Street, and Daithí Mac Sithígh.

• Submission to Ofcom on media plurality
• Submission to the Commons Select Committee on Culture, Media and Sport

The publication of an interim report from the Giovannini Commission on the costs of politics has raised quite a stink.

The report confirms that Italian deputies are grossly overpaid in relation to comparable parliamentarians in France, Belgium, Germany, the Netherlands, and Austria.

Yet the Chamber of Deputies press office has put out a press release claiming, in essence, that Italian deputies have to pay lots of taxes, poor things, and that the total “cost per deputy” is lower than in other countries. The first claim is disingenuous unless taxation in other countries is also taken into account, and the second claim seems to me to be wrong.

Can anyone help me understand this?

Here’s a paper I’ve been working on. Abstract is below; data and Sweave source [rename to .Rnw] also available.

One important characteristic of justice, and a fortiori of our judicial system, is impartiality. One type of impartiality in judicial practice is impartiality between litigants who command status and material resources — the `haves’ — and litigants who lack resources, the `have-nots’. Investigation of relative status advantage in litigation outcomes, which springs from the work of Marc Galanter, has talked past a particularly British tradition emphasizing the conservative bias of the judiciary, and in particular its defence of property owners. In order to investigate these charges, I investigate the success of appeals to the House of Lords between 1969 and 2003 using logistic regression. I build both on general theories of relative status advantage and the advantages of `repeat players’, and on more particularly British interpretations of the judiciary. I find partial support for theories of relative status advantage, insofar as governmental actors have significant advantages over all other actors, but businesses and associations have no advantages over individual litigants. Contrary to expectations of a uniformly conservative judiciary, appellants challenging liberal outcomes were less likely to succeed rather than more.

Comments welcome.

Mark Pack asked about the trade union vote. Here’s a weighted table pulled from the BES pre-electoral polling, the only wave that included a question about trade union membership.

Weighting was done in R with the survey package using the standard YouGov weights.

I was very struck recently by the FT’s ranking of EU finance ministers.

It’s a rare venture into rating ministers other than Presidents or Prime Ministers. And, it’s been repeated over time.

I thought it might be useful to gather data on rankings over time, and construct a measure of finance minister “greatness” comparing across time. To do this, I turned these ranks into a series of pairwise contests, and then estimated a Bradley-Terry model using these 1632 different contests over 5 years, assuming that ministers’ ability remains constant over time, and ignoring ties. The results are shown below.

Note that this figure is based on the FT’s rankings for ministers’ political skills, not their economic ranking (surprisingly, the two sets of rankings are not correlated).

The top two finance ministers are both one-shot wonders: the FT must really regret rating George Papaconstantinou so highly last year. Eduard Janota we can know less about since he died unexpectedly of a heart attack, and was in any case a minister in a caretaker government.

The full data is available, and the R source code used to create the graphs is here.

Obviously the next step is to try and model the ability of “players” in the model using a range of covariates, perhaps the ones included in this interesting paper.

I’m quite surprised that Monti wants to stay until 2013.

That’s a long time for a technocratic government. Assume that elections in 2013 take place exactly five years after those of 2008, on the 13th and 14th. By that time, Monti would have been in office for 514 days.

That’s a month longer than Dini who, IMHO, had a firmer parliamentary base than Monti. And it’s longer than any technocratic government I can find except the Berov government that led Bulgaria between late 1992 and 1994.

Is Italy as a polity really as badly-off as Bulgaria in the early nineties?
Continue Reading »

It is a good thing that Berlusconi has resigned as Prime Minister. Hallelujah.

Some people are upset about the way in which Berlusconi was forced to leave office. Those who are upset fall into two camps. One left-leaning camp is upset that Berlusconi was forced out by bond traders. One right-leaning camp (at least the version I read today in the Telegraph) is upset that Berlusconi was forced out by `Europe’, broadly defined.

I think it’s broadly correct to say that Berlusconi had to leave because the cost of servicing Italian debt had grown to unsustainable levels. But I think that saying baldly that the markets made Berlusconi resign is too simplistic.

First, had it not been the markets, it might well have been something else. At the beginning of Berlusconi’s tenure, I tried to predict how long his government would last, based on certain universal characteristics well-known to predict government duration. I predicted that his government was 95% certain to fall before January 2012. I later thought that he was close to resigning when he was charged for abuse of office and for paying for an under-age prostitute. He wasn’t, and didn’t. But the point still stands. Had the credit crisis not erupted, another parochial crisis might have done Berlusconi in.

Second, Berlusconi reaped a whirlwind of his own sowing. A lot of things had to happen for Berlusconi to be forced out. Not the least of these is the fact that Italian debt had to grow to considerable levels. Berlusconi has some responsibility for this. The second-most important relationship in Italian politics has been between Berlusconi and Giulio Tremonti (the most important is the relationship between Bossi and Berlusconi). Tremonti has always been on the side of fiscal probity. Berlusconi has consistently opposed him (not, I believe, because of any ideological differences, but simply because he resents being beholden to someone who is not disposable, who understands his brief and who commands some respect in the markets).

Third, Berlusconi was already politically weakened. Berlusconi started this legislature with 344 deputies in his majority, 28 more than he needed for a majority. He has frittered away that majority by consistently marginalizing Gianfranco Fini, causing supporters of the latter to leave the PdL and form their own party. Some of those people Berlusconi won back. That’s consistent with the idea of Berlusconi as a great salesman, and a great tactician but poor strategic. His utter neglect of strategy means that he has put himself at the mercy of a number of centrists who have little idea of loyalty. Berlusconi should know this well — he tried to bribe many of these people to bring about the fall of the second Prodi government in 2008.

Bond markets have a lot of influence in politics. It’s right for people to follow them, and their effects. But we shouldn’t ignore the effects of domestic factors. Markets might have pushed Berlusconi over the cliff. But politically, no one was there to catch him when he fell.