In a recent article in the British Journal of Political Science, Hübscher, Sattler, and Wagner (2023) investigate the effects of austerity on party system polarization. They regress election-on-election changes in polarization on a measure of fiscal consolidation and find that an average fiscal consolidation package increases polarization by 0.036 units, or around 0.1 standard deviations. The size of this effect does not change when the authors include a range of additional controls and country and period fixed effects. As is standard, the authors measure party system polarization as the share-weighted standard deviation of party positions,
\[
\varsigma_{V} = \sqrt{\sum_i^{N} v_i \cdot \left(\theta_i - \bar{\theta}\right)^2}
\]
where \(v\) and \(\theta\) represent vote shares and party positions respectively, and where \(\bar{\theta}\) gives the vote-share weighted system mean. The authors take their measures of vote share and party position from Döring, Huber, and Manow (2022) for parliamentary systems.
Although Döring, Huber, and Manow (2022) provide excellent coverage of party positions, this coverage is not perfect, and the measure is somewhat idiosyncratic. The measure is an average of measures taken at different points in time. For some parties, the measure might be based on left-right scales from Castles and Mair (1982). For other parties, the measures might be based on more recent expert surveys. Additionally, even with the unstinting efforts of the ParlGov team to provide left-right measures for all parties, some party positions are missing. The number of missing positions is small ( 31), and the average vote share of parties with missing positions is small ( 3.5%). Perhaps for these reasons, the authors ignore missing observations when they calculate polarization.
Table 1: Largest ten parties missing a left-right position in Hübscher, Sattler, and Wagner (2023)
| Italy |
2013 |
2 |
25.555 |
Five Star Movement |
| France |
2012 |
6 |
6.909 |
Left Front |
| Ireland |
2016 |
2 |
4.212 |
Independent Alliance |
| Italy |
2013 |
2 |
3.204 |
Left Ecology Freedom |
| Italy |
2013 |
2 |
2.250 |
Civil Revolution |
| Germany |
2013 |
9 |
2.200 |
Pirates |
| Italy |
2001 |
5 |
2.170 |
The Girasole (‘Sunflower’) |
| Portugal |
1991 |
10 |
1.706 |
National Solidarity Party |
| Spain |
2015 |
12 |
0.869 |
Basque Country Unite |
| Spain |
2016 |
6 |
0.768 |
Basque Country Unite |
Table 1 gives some impression of this pattern by listing the ten largest parties for which Hübscher, Sattler, and Wagner (2023) lack a left-right position. The list includes a number of parties on the far-left (Left Ecology Freedom, Left Front, Civil Revolution).
The standard way of dealing with missing data is to use multiple imputation. It might seem a bit strange to use multiple imputation here. Normally, we take a data-set, multiply impute data-sets with missing values filled in, and run analyses on the multiply-imputed data-sets, and combine those results. Here, we multiply impute, perform a summarizing operation on the data-set, and then run analyses on summaries of those data-sets. That is, we impute party positions only in order that we can calculate their weighted standard deviation (=polarization).
One way to improve multiple imputation is to bring in auxiliary variables – variables which might not feature in the analysis, but which are correlated with the missing values we’re trying to predict. Here, I brought in a bunch of other measures from Chapel-Hill, V-Party, the Manifesto Project, and WikiTags. I interpolated data from Chapel-Hill by carrying the last value forward, but the data from V-Party and the Manifesto Project is specific to each election year. These three measures therefore vary over time. The WikiTags measure doesn’t vary over time, but has much broader coverage.
Here’s what the model looks like with and without multiple imputation. I’ve kept everything about the original model the same, even those parts I disagree with, like regressing on the difference between polarization in two successive years, or a change score.
Table 2: Main models
| |
Original |
MI |
| Constant |
0.003 |
0.003 |
| |
[−0.023, 0.028] |
[−0.023, 0.029] |
| Fiscal consolidation |
0.019 |
0.019 |
| |
[0.009, 0.030] |
[0.008, 0.029] |
| Num.Obs. |
166 |
166 |
| R2 |
0.074 |
0.068 |
| Num.Imp. |
|
20 |
It looks from Table 2 like multiple imputation hasn’t affected the analysis at all – the coefficient value is unchanged, and if anything the confidence interval is slightly narrower than it was before. The coefficient value gives the change in polarization associated with a fiscal consolidation capable of reducing the deficit by one percentage point of GDP. Given that the average (median) fiscal consolidation package is worth 1.9 percentage points, the effect of an average fiscal consolidation on polarization is 0.036 units. Given the standard deviation of polarization in the original data is 0.39 units, that’s a standardized effect size of around 0.1 comparing “austerity” to “no austerity” countries. That kind of effect size would ordinarily be regarded as “small”.
The great thing about multiple imputation, however, is that it allows us to use other measures which are more affected by missingness. I decided to re-estimate the model using those other variables. Let’s see what happens.
Table 3: Alternative models
| |
Original |
V-Party |
Chapel-Hill |
MARPOR |
Wiki |
| Constant |
0.003 |
−0.017 |
0.008 |
−0.006 |
0.003 |
| |
[−0.023, 0.028] |
[−0.059, 0.024] |
[−0.067, 0.082] |
[−0.064, 0.052] |
[−0.011, 0.017] |
| Fiscal consolidation |
0.019 |
0.014 |
0.013 |
0.009 |
0.005 |
| |
[0.009, 0.030] |
[−0.004, 0.032] |
[−0.021, 0.046] |
[−0.015, 0.033] |
[−0.001, 0.011] |
| Num.Obs. |
166 |
166 |
166 |
166 |
166 |
| R2 |
0.074 |
0.018 |
0.005 |
0.003 |
0.020 |
| Num.Imp. |
|
20 |
20 |
20 |
20 |
As Table 3 shows, when we use any of these other measures, the effect of fiscal consolidation is smaller than what it was before. In the V-Party data – the left-right measure I trust the most – the effect is three-quarters of what it was before. In the MARPOR data – the left-right measure I trust the least – the effect is half what it was before. What’s more, in all of the other models, the 95% confidence interval on the effect runs below zero, and so the finding doesn’t reach conventional levels of statistical significance.
It’s for this reason that I think that the effect of austerity on polarization depends on the measure of party position that you use. I think it’s hard to argue that the ParlGov measure – a time-invariant aggregate of many different sources over different periods – is preferable to any of the measures here. I’d therefore go further, and say that whilst the effect of austerity on polarization is very likely positive, there is room for doubt about this, and the magnitude of the effect may be smaller than we first thought.
Response from the authors
(Points where the post has been amended in response to comments in bold in square brackets)
Given these challenges for this type of macro analysis, we think that the results from your replication are in fact encouraging. We agree with your conclusion that “the effect of austerity on polarization is very likely positive” (p.4), and that we do not (and we would add: cannot) show that it exists with certainty. We mostly tried to refer to our results as the “macro pattern” in order to make clear that we see the macro results as part of a larger analysis. We are aware of the limits that this type of country-level macro analysis has. Your reference to the 5% p-value cutoff raises the question what the appropriate threshold for this type of country-level analysis is (or, at which p-value an effect is “very likely”, in your own expression). We do not know and do not have a strong position on this and hence did not point to the 5% p-value cutoff in previous versions [but were asked to do so in the final version.] Our figure 2, which presents the predicted effect of austerity on the change in polarization uses 90% confidence intervals, and the interpretation of this figure presumably would be the same for most of the other indicators that you use (certainly for the Wiki tags). That the results are not exactly identical with different indicators is something we would expect. Given the trade-offs involved in this type of analysis that we discuss in our point 4 below, it is nice to see that in your analysis, this macro pattern still holds. We substantiate this pattern with additional, micro-level evidence (see also our last point below)
As we describe in the article, we initially started with CMP data and hence the set of parties provided by CMP. We outline the reasons for this decision in point 4 below. We continued to work with this dataset when we added indicators from other sources, also to be able to compare results across different outcome variables. We did not select parties as we wished and in a way that they give us the best results.
Relatedly, and a comment on your Table 1: our analysis does not include Italy 2018 nor Spain 2019. We drop elections after 2016 because our austerity data ends in 2014 (01_HSW_BJPS_Macro1.do, line 281) This means that 5 Stelle 2018, Free and Equal 2018 and Voice 2019 in the list are not really missing parties. The only large party that is missing is 5 Stelle in 2013 [fixed in post above]
One important aspect that you could mention is that your replication is based on indicators that vary over time except the indicator you got from WikiTags (p. 2). (At least this is how we read your analysis.) In our paper, we actually deliberately chose an indicator that is stable over time. We discuss the reasons in the article. The most important one is to not mix supply-side effects (changes in party positions) with demand-side effects (changes in voter behavior). We discuss this fairly extensively on p. 3-4. We also point to this again on p. 7. In fact, we initially used the CMP scores and did not find an effect of austerity on polarization. We found this puzzling given the clear effect for non-mainstream parties. When we then plotted the CMP-based polarization variable over time, we found that the over-time polarization (annual averages across all countries) was declining (with ups and downs) rather than increasing. The time-invariant Parlgov data, however, leads to a gradual increase in polarization that is consistent with the rise of non-mainstream parties (Figure 1). We think it is fair to say that much of the discussion of polarization today is based on the idea that polarization is rising. We attributed these diverging patters between CMP- and Parlgov-based polarization to the convergence of mainstream parties, i.e. a supply-side effect, in the time-varying CMP data. The question is now how to empirically address the mixture of supply vs. demand effects on time-varying data and how to isolate the effect of vote shares from the effect of party positions. Our question to you would be to what extent your weaker results are due to different measurement, additional data points or supply-side effects? We agree that a time-invariant indicator as we use it is imperfect, so we are eager to hear about better solutions to this issue using macro data. We are not convinced that data that does not distinguish between these two types of mechanisms is the best solution given our theoretical focus on voter behavior.
Regarding effect size (p.3), you compare our effect to the cross-country variation in polarization. The purpose of our paper, as we say on page 8, is to examine how austerity contributed to the increase of polarization over time. The average increase in polarization in our dataset from one election to the next is 0.028 (with a standard deviation of 0.14). We interpreted our results in light of this average change in the text on page 9. The question now is what is the right comparison? As we asked ourselves, is the effect that we find small because Ireland does not become Italy? We do not think so. The reason is that the increase in polarization over time is small compared to cross-country variation in polarization. If cross-country variation is the benchmark, then the increase in polarization that political systems have seen in the past decades should not really be a major issue. We do not think that this is the interpretation in the political science literature. We think that austerity is relevant because it can increase polarization in a country compared to the degree of polarization in this country at an earlier stage. This is why we do not compare our estimated impact to the cross-country variation. Here, a look at non-mainstream party vote share helps because its interpretation is more intuitive. As we write, an average (large) consolidation package increases non-mainstream party vote share by ca. 1.3 (2.9) percentage points. [The cross-country standard deviation in non-mainstream party vote share is 12.4; the average increase over time is 1.28 with a standard deviation of 6.7.] For comparison, this estimate is within the range of other estimates of the impact of the economy on radical party vote share (e.g. Colantone & Stanig 2008 find that a one standard deviation increase in import shock leads, ceteris paribus, to higher support for radical-right parties by around 1.7 percentage points, see their page 945).
A final point that we think is important: the cross-national analysis with polarization as the outcome is just one part of the paper. We also examine different outcome variables, look into specific cases and provide experimental evidence from four countries. The results from the micro analysis, for instance, are quite clear. This is a multi-method paper exactly because each of the parts has limits (external validity for the survey experiment - which is why we have the macro part, identification for the macro analysis - which is why we have the micro part). There has been a move away from macro analyses towards micro analyses for these reasons. But many of these micro studies are motivated with macro patterns (such as a rise of radical parties). We, therefore, think that it is worthwhile to tie the micro findings back to these macro patterns - and we go a long way to do this, e.g. at the end of the micro section - even if the macro analysis comes with trade-offs. In our view, an evaluation of the effect of austerity, as you do it at the end of your paper, has to be judged against all the evidence that we provide. So, we very much agree with your approach of using multiple ways of answering the same question – while you use different data sources, we use different methods.
References
Döring, Holger, Constantin Huber, and Philip Manow. 2022.
“Parliaments and Governments Database (ParlGov): Information on Parties, Elections and Cabinets in Established Democracies.” http://parlgov.org.
Hübscher, Evelyne, Thomas Sattler, and Markus Wagner. 2023. “Does Austerity Cause Polarization?” British Journal of Political Science.