In late March, Patrick Wu and colleagues released a working paper suggesting that ChatGPT could produce a ranking of US Senators from left to right.
They asked ChatGPT to compare senators two at a time, and say which one was more liberal.
They then fed this into a model for pairwise comparisons, and generated scores for each senator, scores which correlate very well with other measures of how liberal or conservative senators are.
I decided I would try and repeat this exercise for the UK. Although the very latest version of ChatGPT (v. 4) is still being rolled out, it’s easy to play around with ChatGPT 3.5, and there are packages that make interacting with ChatGPT programmatically easier as long as you sign up for a developer account.
For my first attempt, I tried comparing all MPs using the davinci GPT3.5 model. This is not a ChatGPT model – it’s not had humans in the loop in quite the same way.
I asked davinci: “Of the following two MPs, which is more left-wing on economic issues, [NAME] and [NAME]?”.
It felt like a good idea at the time to restrict it to economic issues, both on grounds of greater specificity and because I didn’t want GPT3.5 to compare one set of MPs on economic left-right only to compare a second set on sociocultural left-right.
Experimenting with davinci was wild – it would answer my prompts confidently, and pretty soon I had a large number of comparisons. It’s not possible to make all comparisons in the way Wu et al did: there are around pairwise 210,000 comparisons between MPs (650, times 650 minus one, divided by two), and I was still asking GPT3.5 to return two or three answers to check how random/determined its answers were.
I then decided to back off and try ChatGPT on a smaller task: comparing Conservative MPs who were elected before the 2019 election.
This worked much better (although I still had problems with the connection timing out). What seemed to make a big difference was actually the option for ChatGPT to refuse to answer. Sometimes this is a perfectly reasonable response – I, as someone who is professionally somewhat interested in UK politics, would find it hard to compare some of these MPs.
Here’s a plot of the top-ten most “right-wing MPs” according to a Bradley-Terry model trained on around 18,000 pairwise comparisons. The absolute values of the numbers don’t really matter – what matters is relative position and uncertainty. The bars are 83% confidence intervals, so if two bars overlap it’s possible we could have got results like this if the MPs in fact had the same position.
This doesn’t look crazy – Liam Fox has in the past been described as leader of the [Thatcherite] Conservative right, and Liz Truss’ economic policy was notoriously too right-wing for the markets.
Similarly, if we plot the ten most left-wing MPs, we get something that again seems to make sense.
Robert Halfon is a trade union member, and has campaigned volubly for a Conservatism that benefits the working class, understood not just to be one that involves tax cuts but which might conceivably involve greater public provision. Jesse Norman has in the past been in favour of cooperative ideals.
If we step back from the extremes, and use data on the group/faction affiliations of different MPs, we get a picture that again seems to make sense. Here’s a box-plot showing averages for different groups (ERG, Common Sense, One Nation).
The average position of One Nation is lower (i.e., to the left of) Common Sense, which in turn is to the left of the ERG.
I’ve said that these estimates “make sense” three times now – but at this time I don’t plan on using these estimates in future research or publishing the full list. This is because I can’t trust the reasons ChatGPT gives.
Here’s an example of a ChatGPT response which is half right:
“Based on publicly available information, Chloe Smith is more left-wing on economic issues than John Redwood. John Redwood is a Conservative MP who has been a vocal advocate for free-market policies, including deregulation and lower taxes. He has also been critical of government spending and has advocated for austerity measures. Chloe Smith, on the other hand, is a Conservative MP who has expressed support for policies such as the living wage, increased investment in public services, and a more progressive tax system”
I think everything about John Redwood rings true – but I must have missed the bit where Chloe Smith expressed support for a “more progressive tax system”. This is not an isolated claim – progressive tax crops up in just under 1,000 responses.
You can’t even quiz ChatGPT about its rationales. Often if you try to probe you run in to guardrails, or ChatGPT says it doesn’t have access to public statements made by [X] in relation to taxation.
It’s a paradoxical feature of these ChatGPT responses that they more they purported to give reasons for these pairwise judgements, the less confident I felt in the judgements themselves. If whatever God looks over Conservative MPs had come down from the heavens and given me a rank-ordered list of MPs, I’d probably have run with it. As it is, I’m sticking a pin in this for the moment.