Measuring Populism in Political Parties: Appraisal of a New Approach

Populism has become a pervasive concept in political science research. However, a central and basic question remains unanswered: which European parties are more populist than others? Despite the increasing wealth of studies on populism in parties, we lack data that measures populism in political parties in a valid and precise manner, that recognizes that populism is constituted by multiple dimensions, and that ensures full coverage of all parties in Europe. In this article, we first appraise the weaknesses of existing approaches. Arguing that parties’ populism should be measured as a latent construct, we then advocate a new approach to operationalizing and measuring populism in political parties using expert surveys. Relying on the Populism and Political Parties Expert Survey spanning 250 political parties in 28 European countries, we show that populism is best measured in a multi-dimensional and continuous manner. We subsequently illustrate the advantages of our approach for empirical analysis in political science.


Introduction
Populism has become a ubiquitous concept in political science. As political parties like the League in Italy and the National Rally in France have become key actors in European politics, populism research has matured. There is increasing consensus among scholars on the definition of populism and its core characteristics (Rovira Kaltwasser et al., 2017). We know more about the systemic causes of populism (Hawkins et al., 2017) and about the individual-level causes of populist radical right and populist radical left support (Burgoon et al., 2019;Rooduijn & Burgoon, 2018;Van Hauwaert & Van Kessel, 2018). In short, the steady increase of populist parties within European party systems is paralleled by significant advances in populism research (Rooduijn, 2019).
However, a central and most basic question remains unanswered: which European parties are more populist than others? Despite the increasing wealth of studies on party-based populism, we so far lack data that measures populism in a valid and precise manner in political parties, that recognizes that populism is constituted by multiple related but distinct dimensions, and that ensures full coverage of all parties in Europe.
Current approaches to measuring populism in political parties often rely on less precise measurements or categorizations of populism. Most large-N applications of populism rely on classification "by fiat" based on literature reviews or country specialists, as Hawkins and Castanho Silva (2018) note. Such approaches suffer from reliability and validity problems as they are based on a heterogeneous set of studies applying varying conceptions of populism. Many studies moreover choose to focus on populism's "usual suspects" or examine a limited set of country cases only-making the empirical material less suitable for systematic comparative analysis. As different scholars apply contrasting definitions, or operationalize definitions of populism in different ways, it is ultimately difficult to disentangle definitional and conceptual preferences from empirical manifestations. Moreover, the reliance on a very limited number of coders or experts, raises doubts about the reliability of populism measurement in some studies. These issues highlight a problem in party research on populism; without a manner of empirically establishing parties' level of populism in a precise and reliable fashion for a broad range of parties and party systems, we cannot make meaningful comparisons between parties and countries.
While the study of populism in Europe originated in supply-side partylevel research (Canovan, 1999;Mudde, 2004), the most important empirical advances to the study of populism in recent years stem from demand-side individual-level research (Akkerman et al., 2014;Castanho Silva et al., 2018;Hawkins et al., 2012;Schulz et al., 2018;Van Hauwaert & Van Kessel, 2018;Wuttke et al., 2020). Demand-side measurements of populism have demonstrated the value of a precise, continuous populism measure, while showing the importance of accurately identifying and measuring populism's constitutive dimensions. We therefore argue that empirical studies of populism in parties stand much to gain from the methodologies applied to measure the levels of populism among citizens.
In this article, we take stock of these challenges to the empirical study of populism in political parties and, subsequently, propose an alternative method of operationalizing and measuring populism in political parties. We put forward a way of measuring parties' populism using expert surveys that allows for precise, interval-level measurement of populism, encompasses the full range of political parties in 28 European countries that recognizes the multidimensionality of the concept of populism. Subsequently, we explore the advantages of our new approach and compare our approach to other attempts of measuring populism among political parties.

Measuring Populism on the Supply Side: The Need for a New Approach
After prolonged scholarly debate, three dominant definitions of populism have crystallized in the literature: the ideational approach to populism (Hawkins, 2009;Mudde, 2004Mudde, , 2017, populism as a strategy (Weyland, 2001(Weyland, , 2017, and populism as a style (Jagers & Walgrave, 2007;Moffitt & Tormey, 2014). The ideational approach is the most readily used approach by studies that empirically measure populism. The ideational approach to populism is most applicable for empirical analysis as its concise definition is amenable to clear operationalization and as it is applicable to various different actors and domains (Mudde, 2017, p. 35). Our focus here lies on the ideational approach to populism.
When populism research gained traction in the 1990s and 2000s, the adjudication of populism in political parties was based on qualitative, and often inductive, approaches (e.g., Betz, 1994;Mudde, 2007;Taggart, 2000). Since the 2010s, scholars increasingly resorted to quantitative and deductive approaches to measuring populism. Populism has been measured using textual, content analysis methods on the basis of party manifestos, speeches, or press releases. Textual approaches employ hand-coding or machine coding, while researchers focus on sentence level, paragraphs, statements, or on the document as a whole (e.g., Bernhard & Kriesi, 2019;Hawkins et al., 2019;Hawkins & Castanho Silva, 2018;Rooduijn & Pauwels, 2011). A few have directed their efforts to measuring populism using expert surveys Wiesehomeier, 2018). These studies have done important, groundbreaking work in the study of populism.
Yet, we contend that existing approaches to measuring populism nevertheless suffer from a number of weaknesses. Specifically, we argue that existing approaches to measuring populism among political parties often lack construct validity and/or fail to appreciate the multi-dimensionality of populism. In addition, the empirical strategies employed often lack precision. In terms of coverage, current approaches usually study only a limited set of political parties. Clearly, all approaches discussed have their individual strengths, be they in terms of construct validity, concept multi-dimensionality, measurement precision or case coverage. Yet, arguably none of the existing approaches fit all four criteria outlined below.

Construct Validity
Ideational approaches consider populism to be a set of ideas about politics that understands politics as a Manichean struggle between the will of the homogenous people and the corrupt elite (Hawkins & Rovira Kaltwasser, 2018;Mudde, 2004Mudde, , 2017. This definition of populism is multidimensional and consists of a number of components. In line with the existing conceptual literature, we distill five components. Political sovereignty should reside with the ordinary people (1). The ordinary people are an indivisible or homogenous community (2) whose interests are united by a general will (3). The elite is portrayed as corrupt (4) and the juxtaposition between the ordinary people and the elite is of Manichean proportions, that is, a moral struggle between good and bad (5). For a measurement of the ideational approach to populism to have construct validity, it should capture all relevant components (Hawkins & Rovira Kaltwasser, 2018, p. 3). Most existing measures of populism, however, fall short of capturing all components.
In the 2014 Chapel Hill Expert Survey (CHES),  aim to measure populism in European parties. Populism is measured using an item measuring the "salience of anti-establishment and anti-elite rhetoric" and with an item measuring the "salience of reducing political corruption". This conceptualization of populism does not capture the full scope of the ideational approach to populism. First, the empirical emphasis on anti-elite rhetoric overlooks other core components of the ideational definition of populism. Not measured are parties' stances regarding the sovereignty of the people, the existence of a homogenous general will, and the antagonistic relation between the people and the elite. As such, the ideational conception of populism is only partially operationalized. Measuring the saliency of antielite rhetoric, moreover, seems to conflate positional and saliency-based measures of party ideology. Scholars have long recognized that parties' ideological positions and parties' issue emphasis or salience, are interrelated but distinct logics of party politics (Carmines & Stimson, 1986;Downs, 1957). Whereas positions reflect a party's stance on a certain issue or problem, saliency reflects the importance the party attaches to that issue. Existing CHES measures of party dispositions toward the European Union (EU) therefore also measure parties' EU positions and EU salience separately (Bakker et al., 2015a;. Second, while the ideational approach speaks of the opposition between the "ordinary people" and the "corrupt elite", (Mudde 2004, p. 543) this does not mean that populists necessarily aim to fight "actual" corruption in a literal sense, that is, "the misuse of entrusted power for private gain" (Heywood, 2009). Rather, populist parties regard the very act of participation in politics to be corrupting (Taggart, 2018). Thus, an item on the importance of reducing actual corruption does not adequately address the complexity of the populist notion of the corrupt elite.
In a 2017 "flash" survey focusing on a subset of countries, CHES addresses the lack of a people-centered question by including an additional item from the demand-side literature, focusing on the juxtaposition between the people and the elite (Akkerman et al., 2014). However, the CHES wording of the question embeds the people-centeredness of populism within the context of direct (i.e., referendums) versus representative democracy. Framing the question as such gives the impression that populists are per definition opposed to representative democracy and it requires experts to associate populism with plebiscitary democracy. To be sure, there is an elective affinity between populism and referendums (Jacobs et al., 2018). However, support for referendums does not constitute a defining feature of the ideational approach.
All in all, while the CHES items do capture certain elements of the ideational approach to populism, the items do not fully cover the constitutive dimensions of the concept. Moreover, not all items clearly separate populism from adjacent concepts often associated with populism, such as corruption and referendums. As Sartori notes, a concept must be "mutually exclusive" (Sartori, 1970).
Also using expert surveys, Wiesehomeier (2018) provided an empirical application of the ideational approach for the first time in a wide range of Latin American countries with expert surveys fielded in 2015 and 2018/2019. Populism was operationalized with items measuring people-centrism, a moral conception of anti-elitism and the aforementioned CHES anti-elite saliency item. Wiesehomeier operationalized "people-centrism" and the "moral anti-elite" in terms of their opposing poles. Indeed, when delineating a concept it can be helpful to conceptualize concepts in terms of their opposites, that is, in terms of what they are not, as Sartori (1970) famously noted. In the case of populism, however, this is not straightforward. Some regard the opposite pole of populism to be both elitism and pluralism (Hawkins & Rovira Kaltwasser, 2018;Mudde, 2004). Individual-level studies of citizens' populist attitudes find that populism, elitism and pluralism do not load on a single dimension, but on three separate dimensions (Akkerman et al., 2014). Moreover, in terms of survey design, juxtaposing multiple constructs results in "double-barreled questions" which can lead to validity problems as it is unclear which construct was most salient to the respondent (Olson, 2008). To measure people-centrism Wiesehomeier juxtaposes a candidate's or party's "identification with the authentic common people" with "referring to citizens and their unique interests." Yet, these two constructs are not mutually exclusive. Left-wing populist movements in Europe, such as Podemos in Spain, hold more pluralistic worldviews but combine this with references to the "ordinary people." And while the ideational definition of populism certainly contains a moral component, this is not adequately captured by a measurement contrasting "demonizing and vilifying opponents" with "treating opponents with respect." A different challenge to the construct validity of populism measures in other research endeavors is that parties' degree of populism may depend on the specific political domain or arena. There is evidence that the degree of populism found highly depends on the specific area of party behavior. For instance, the degree of populism in manifestos is generally low (Rooduijn et al., 2014). In contrast, speeches are more prone to exhibit populism than party manifestos (Hawkins & Castanho Silva, 2018) and certain types of speeches are more disposed to populist phrases than others (Hawkins et al., 2019). Hence, it is not clear if all domains of party behavior or party communication are equally valid for measuring populism. Measures that are highly context-dependent may therefore be less useful for generalizable assessments of parties' populism.

Multi-Dimensionality
Populism's multi-dimensionality is an essential characteristic of the ideational approach. Empirical renderings of populism should not only encompass all of these dimensions, they should also measure populism's constitutive dimensions separately. Only a multi-dimensional measure of populism allows operationalization of the ideational approach in a fine-grained and transparent manner. Moreover, a multi-dimensional measurement of populism allows us to explore the diversity among parties' "populisms." Using a "holistic" coding method, Hawkins and colleagues (Hawkins, 2009;Hawkins et al., 2019;Hawkins & Castanho Silva, 2018) convincingly measure populism in line with the ideational approach. Holistic grading provides an overall assessment of the degree of populism as apparent in the speeches of presidents and prime-ministers (Hawkins et al., 2019). While the coders are instructed to empirically gauge whether core dimensions of populism are present in a given text, the approach cannot empirically differentiate between the multiple dimensions of populism-as it is measured in a holistic way.
Other textual approaches have been more geared towards assessing populism in multi-dimensional fashion. These studies have explicitly coded the constitutive components of populism (Bernhard & Kriesi, 2019;Manucci & Weber, 2017). Yet, as Rooduijn et al. argue, measures of populism should examine the extent to which its core components co-occur in a party's statement (Rooduijn et al., 2014;Rooduijn & Pauwels, 2011). It is unclear, therefore, whether studies that measure the occurrence of elements of populism separately are, in fact, gauging populism in a party, or merely the presence of sub-dimensions of populism (see Bernhard & Kriesi, 2019;Manucci & Weber, 2017).

Measurement Precision
Following a Sartorian approach, most empirical work on populist parties relies on a dichotomous understanding of parties' populism. Parties are either populist, or they are not populist. These classifications are either based on in-depth qualitative research (e.g., Mudde, 2007;van Kessel, 2015), or on judgments by a small number of scholars, as is the case in the PopuList . Many large-N studies of populism rely on such dichotomous measurements of populism (Leininger & Meijers, 2020). These dichotomous measures of populism cannot account for the multi-dimensionality of populism and cannot differentiate between degrees of populism.
In democracy research, some advocate a dichotomous measurement of democracy (Boix et al., 2013;Przeworski et al., 2000). Yet, others argue that dichotomous approaches are not able to account for the multi-dimensionality of democracy. Moreover, dichotomous measures of democracy unite regimes that strongly differ from one another in significant aspects of democracy. Unsurprisingly, therefore, multidimensional and continuous measures of democracy have gained traction within democracy research (Coppedge et al., 2011).
The need for a more precise measurement of populism is also apparent when looking at "contentious cases" of populism. The Dutch Socialist Party (SP) is such a case. Mudde and March (2005), March (2011), and the PopuList  consider the SP to be a populist party. According to van Kessel's (2015, p. 60) classification, however, the SP is not a populist party, as it toned down its populist rhetoric since 2012. Similarly, while Zaslove (2008) considered Berlusconi's Forza Italia (FI) a populist party, Mudde (2011, p. 13) and van Kessel (2015, p. 54) consider it to be a "borderline case." We contend that such ambiguity indicates that different scholars rely on different implicit benchmarks of populism. In most cases, however, it remains opaque what informs that benchmark and how one can determine where the benchmark should lie.
Conceptualizing populism as a dichotomous construct is also exceedingly difficult when parties strategically employ populist discourse. As Mudde (2007, p. 40) notes, "[p]arties may appear schizophrenic if their rhetoric diverges from their ideology and the researcher is left with the dilemma of which image to trust". Mudde notes that adjudicating a parties' populism becomes all the more difficult if a party continues to voice populism "strategically." Yet, deciding that one party's use of populism is "sincere" and another's "strategic" is inevitably arbitrary. If we are to regard populism not as an "all or nothing" affair but as a matter of degree, we can appreciate parties' limited application of populism precisely as a low degree of populism. A continuous measure of populism would therefore allow us to avoid conceptual confusion on whether populism is "sincere" or "strategic." We argue that it is the multi-dimensionality of the ideational approach to populism that makes adjudicating populism in a dichotomous manner difficult. As with any multi-dimensional concept, the adherence of the objects of inquiry to the constitutive dimensions are likely to vary between the dimensions. If a party exhibits some dimensions of populism strongly, but other dimensions less clearly, deciding whether a party is populist or not is fraught with difficulty. Left populist parties, for example, tend to have a more pluralistic perspective of the people (Kioupkiolis & Katsambekis, 2018;Stavrakakis & Katsambekis, 2014).
From individual-level research, we have seen important advances in measuring citizens' populist attitudes in a continuous manner (Akkerman et al., 2014;Hawkins et al., 2012). A continuous measure permits a fine-grained analysis of the correlates of populism. For instance, demand-side research shows that an individual's degree of populism is distinct from political trust or external political efficacy (Geurkink et al., 2020). In party-level research, textual approaches have made important first steps in measuring populism in a continuous manner. Rooduijn et al. (2014) measure the salience of populist discourse in party manifestos on a 0-100 scale. Bernhard and Kriesi (2019) measure the salience of populist rhetoric in press releases in three dimensions aggregated to a 0-3 index. Pauwels (2011) measured the relative share of words pertaining to "populism" in party manifestos and internal party communication. Unlike these salience measures of populism, the holistic approach advocated by Hawkins produces a non-dichotomous positional measure of populism measured as an 0-2 index (Hawkins, 2009;Hawkins et al., 2019).

Coverage
Existing approaches to measuring populism are often confined to a selection of political parties and countries. We lack a dataset that measures populism comprehensive in Europe's party systems.
Many approaches to measuring populism rely on manually coded textual approaches. Human coding is a time consuming and costly affair. Some, therefore, choose to measure the degree of populism only in the "usual suspects," that is, parties generally considered to be populist (Franzmann & Lewandowsky, 2020). Other studies provide a more complete picture of countries' party systems. Rooduijn et al. (2014) code the most relevant parties in France, Germany, Italy, The Netherlands and the United Kingdom over four election periods in the 1990s and the 2000s. Manucci and Weber (2017) measured populism for parties with more than 3 per cent of the vote in Austrian, German, Dutch, Swiss, and UK elections since the 1970s. Bernhard and Kriesi (2019) study the salience of populism in press releases during national election campaigns in 11 European countries.
Indicative of the promise of expert surveys, Wiesehomeier (2018) measures the degree of populism in 165 political parties and 18 presidents in 18 Latin American countries in her 2015 expert survey. The most comprehensive dataset of populism to the date is the Global Populism Database which covers 215 chief executives, that is, presidents or prime-ministers, in 66 countries across the world from 2000 to 2018 (Hawkins et al., 2019). While a focus on the executive is suitable for comparative work on populism on the country-level, its applicability for party-level research is more limited.
All in all, existing approaches to measuring populism have made important advances. However, they also suffer from a number of drawbacks. While expert survey-based approaches show great promise of measuring populism in a multi-dimensional manner, their operationalization is not completely in keeping with the ideational definition of populism Wiesehomeier, 2018). A few text-based approaches make an important contribution given that they go beyond the ubiquitous dichotomous approach to measuring populism (Bernhard & Kriesi, 2019;Manucci & Weber, 2017;Rooduijn et al., 2014;Rooduijn & Pauwels, 2011). Yet, their coverage of parties and countries is limited, and the degree of populism found in such textual measures are highly dependent on the type of textual output. The holistic approach applied to speeches captures the ideational approach to populism, but fails to empirically address its multidimensionality and mostly limited to the speeches of chief executives (Hawkins et al., 2019).

A New Approach to Measuring Populism in Political Parties
To assuage the challenge to measuring populism in a precise, valid, multidimensional way for a large array of parties in Europe, we propose to measure European parties' populism with an expert survey.
Expert surveys are a well-established method for collecting data on parties' ideological positions and party characteristics. Initially used to estimate parties' left-right positions (Castles & Mair, 1984), expert surveys have subsequently been used to assess party stances on European integration (Marks et al., 2007;Ray, 1999) as well as on other policy issues such as immigration, the environment, or ethical questions such as abortion and euthanasia (Bakker et al., 2015b;Benoit & Laver, 2006). The use of expert surveys in party politics research is however not limited to programmatic characteristics of parties. Expert surveys can also be used to assess parties' organizational characteristics and party-citizen linkages (Carlin et al., 2015;Kitschelt & Kselman, 2013), as well as parties' attitudes toward representation and issues such as media freedom (Benoit & Laver, 2006;Ruth-Lovell et al., 2019).
There are several advantages to using expert surveys for measuring party positions, and as an extension for measuring parties' populism. First, expert surveys do not prioritize a single domain of party behavior (Benoit & Laver, 2006;Marks et al., 2007). When an expert judges a party, he or she takes various domains into consideration, including "political speeches, debates", and "opinions of party leaders" (Benoit & Laver, 2006, p. 73). Second, in contrast to qualitative case studies which rely on a very small number of experts, expert surveys convey a scholarly consensus about a party's placement as they reflect the central tendency of multiple experts' qualitative judgments. Moreover, the collection of more information increases certainty in the accuracy of the estimates produced. In short, expert surveys "collect the best knowledge and wisdom of a population of experts, based on their evaluation of all the evidence at their disposal, and summarize their consensus in a set of tractable estimates" (Benoit & Laver, 2006, p. 73). Third, expert surveys proclivity to the measurement of multiple dimensions is especially important given the multi-dimensional nature of populism.
We have conducted an expert survey, entitled Populism and Political Parties Expert Survey (POPPA), in 28 European countries covering 250 political parties to estimate parties' populism as well as to measure parties' ideological positions and characteristics pertaining to their organization and political style. 1 The countries included are Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Lithuania, Malta, The Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland, United Kingdom. 2 For Belgium, we distinguish between Flanders and Wallonia given that the two regions have different party systems. The survey aimed to include all relevant parties in each party system. All parties that were represented in parliament in 2017 and/or 2018 were included in the survey. If elections were held in 2017 or 2018, we include both parties with seats in the previous parliament and parties that newly entered parliament. 3 In some cases, electoral coalitions of parties hold seats in parliament (e.g., the Coalition for Bulgaria, KzB). In such cases we select the party or coalition for which ParlGov has allotted seats (Döring and Manow, 2019). 4 All party lists were validated by country experts.
The expert survey was fielded between April and June 2018. In total, 861 invitations were sent to country-experts in the 28 countries with expertise in party politics. Experts have been selected on the basis of publications records in Web of Science as well as biographical information. In the end, the survey has 294 recorded responses-amounting to a response rate of approximately 34 per cent. 5 Experts were explicitly instructed to evaluate party positions and party characteristics as apparent in the party leadership during a clearly demarcated timeframe, the spring of 2018, as it is important to prescribe experts which aspects of the party should be evaluated and at what time period should be taken into consideration in order to attain valid expert judgments (Budge, 2000;Marks et al., 2007). 6

Operationalizing Populism
Experts were asked to evaluate all parties in their respective party system on 16 different items pertaining to populism, party ideology, party organization and political style on a 11-point scale (0-10). As different country experts likely adhere to different definitions of populism, we avoided asking a direct question on populism. Instead, we operationalize populism in a multi-dimensional way that is in line with the ideational approach. Individual-level research on populist attitudes has demonstrated that populism is a latent construct (Akkerman et al., 2014;Hawkins et al., 2012;Van Hauwaert et al., 2019). That is, populism is measured indirectly through a number of survey items that share common variance-referring to the underlying construct of populism. Also on the party-level we believe populism to be a latent construct-constituted by the various components identified by the ideational approach to populism. Following the ideational approach, we distill five components of populism (Hawkins & Rovira Kaltwasser, 2018;Mudde, 2004Mudde, , 2017.
In order to avoid double-barreled questions, we designed a different survey item for each dimension. Table 1 lists all 16 items included in the expert survey with a short description.
The juxtaposition between the elite and "the people" is central to the ideational definition. The "anti-elitism" item captures parties' overall anti-elite attitude and the "people-centrism" item assesses the extent to which sovereignty should lie exclusively with the ordinary people. Populism in the ideational sense also has a moral component, which we measure with the item "Manichean worldview." The item measures the extent to which parties regard politics as a moral struggle between good and bad, that is, a Manichean worldview (Mudde, 2004(Mudde, , 2017. Furthermore, we measure the supposition that the people and its interest are homogenous using two distinct, but closely related items. We make a distinction between the unity of political interests and the homogeneity of the political community (i.e., the people). The item "general will" assesses parties' belief that the ordinary people's interests are singular, or in other words, that the ordinary people share one general will (Abts & Rummens, 2007). The item "indivisible people" measures parties' agreement with the supposition that the ordinary people are a homogenous or indivisible entity (Abts & Rummens, 2007;Canovan, 1999;Taggart, 2000). 7 Table 2 shows the results of an iterated principal exploratory factor analysis (EFA) on the mean expert judgment on the five items operationalizing populism. 8 The EFA confirms our expectation that the five items constitute a latent construct of populism. Moreover, each of the five items cover unique variance in that latent construct. On the basis of the predicted regression scores yielded by the factor analysis, we subsequently operationalize populism as a weighted arithmetic mean of the five items. 9 While EFA is commonly used for scale building, it does not allow a precise assessment of the added value of each item for measuring the latent construct (Van Hauwaert et al., 2019). By contrast, ordinal item response theory (IRT) allows us to evaluate the relationship between experts' assessments of parties on the five items and the underlying latent construct of populism. Originated in quantitative educational research, IRT was designed to measure the latent construct "ability" in students' test results (Hamlbleton et al., 1991). In our case, IRT models can help assessing whether expert assessments of a party on a certain survey item is a function of the latent construct of a parties' populism. 10 Manichean worldview "Politics is a moral struggle between good and bad" Indivisible people "The ordinary people to be indivisible (i.e., the people are seen as homogenous)" General will "The ordinary people's interests to be singular (i.e., a 'general will')" People-centrism "Sovereignty should lie exclusively with the ordinary people (i.e., the ordinary people, not the elites, should have the final say in politics)" Anti-elitism "Anti-elite dispositions"

Party ideology
Left-Right (General) "Overall ideology (i.e., the general left-right scale)" Left-Right (Economy) "Active government role in the economy versus reduced gov. role in the economy" Immigration "Strongly opposed to immigration versus strongly in favour of immigration" European integration "Strongly in favour of EU integration versus strongly opposed to EU integration" Nativism "Exclusive idea of who can and should belong to the nation-state" Civil liberties versus Law & order "Civil liberties (i.e., personal freedoms) versus law and order (i.e., stricter policing, harsher sentencing)" Lifestyle: Traditional versus Liberal "traditional view on moral values (such as conventional notions of the family: i.e., marriage and child-rearing) versus liberal views on moral values (i.e., acceptance of homosexuality, same-sex marriage etc.)"

Political style
Complex versus Common-sense politics "Political decision-making is a complex process versus common-sense solutions to political problems" Emotional appeal "Appeal to emotions in their political communication with the voter (i.e., fear, hope, anger, happiness)"

Party organization
Personalized leadership "Characterized by more or less personal leadership" Intra-party democracy "Characterized by more or less intra-party democracy (i.e., party members play a role in decision making, room for internal debate, decision-making is inclusive of various factions and organizational layers within the party)" In their study of the individual-level measurements of populist attitudes, Van Hauwaert et al. (2019) have demonstrated the utility of IRT analyses for comparing various scales capturing latent populist attitudes in individuals. Ordinal IRT is particularly suited to assess latent constructs in individuals' ordinal survey responses, but also "when experts assign ordinal ratings to countries, to party manifestos, to candidates, etc." (Jackman, 2009, p. 137) In addition, IRT modeling allows us to account for differential item functioning as well as random errors related to variation in expert reliability (Clinton & Lewis, 2008;Marquardt & Pemstein, 2018).
We compare the extent to which our five items measure the latent construct of populism with the three items from the 2017 CHES "flash survey" . In line with Marquandt and Pemstein (2018), the IRT analyses are conducted on the expert-level clustered per expert. While we argue that the CHES approach does not operationalize the ideational definition to populism in its entirety, it is an open question to what extent their items measure a latent construct of populism, and how they compare to our items. Figure 1 depicts the Test Information Function (TIF) curve for the 5-item populism scale as well as for the CHES scale. 11 The TIF displays the information and measurement curves of the populism items from our dataset. On the left y-axis, the level of information yielded by the item or construct is shown. "Information" denotes the reliability or measurement precision of the entire latent construct. The x-axis shows the theta (θ) values which represent the continuum of the latent construct, that is, parties' populism. The flatter the  Table A1 in the online appendix shows the results of the iterated principal exploratory factor analysis without a set minimum Eigenvalue.
tails of theta, the better the fringes of the latent construct are measured by the items. The right y-axis shows the associated measurement error for each level of theta.
The TIF in Figure 1a shows that our latent populism variable is able to measure the latent construct of populism with considerable precision (i.e., information) in the theta interval [−2.5 2.5] of the latent scale. On the fringes of the scale precision decreases and associated levels of measurement error increase. Figure 1b shows that the precision of measurement of the three CHES items is considerably lower in comparison to our five-item measure. Moreover, the theta interval in which the CHES items reliably measure the latent construct is smaller at [−2 2].
The Item Information Function in Figure 2 shows how much the individual expert survey items contribute to the latent construct. On the left panel, we see that the item measuring "general will" contributes most to the latent construct at a precision level of 4.5, whereas "Manichean worldview" contributes least to the latent construct at a precision level of 1.6. The diversity of the items strongly contributes to the coverage of the fringes of the latent construct, especially for the less populist end.
By and large, this strongly suggests that populism is a latent construct that can only be fully captured with a multi-dimensional empirical approach. 12 The right-hand panel of Figure 2 shows how much the three CHES items contribute to the latent construct. It becomes clear that the "anti-elite salience" item contributes substantively to the latent construct. Measurement precision of the "people-versus-elite" item is higher for the tails of the theta interval, but overall the item contributes less information to the latent construct. By contrast, the item measuring "corruption salience" hardly adds information to the construct. All in all, we see that our five-item populism measure performs better than the 2017 CHES measure.

Comparing Populism Measures
Having ascertained that the populism items load strongly on one factor and that we are able to measure the latent construct with sufficient precision, the question still remains whether the predicted populism variable weighted by factor regression scores identifies a different set of parties than other measures of populism.
First we compare our measure with the 2014 CHES "salience of anti-elite rhetoric" item . 13 The CHES anti-elite salience measure is often used as a measure of parties' degree of populism (Norris & Inglehart, 2019;. To examine the degree to which the two measures coincide, we map the ordering of parties according to both measures using the Spearman's Rho correlation (see also Benoit & Laver, 2006, pp. 99-100). The overall Spearman's Rho correlation coefficient for all cases is 0.72 pointing to considerable overlap, which is to be expected. However, there are also some important differences. Table 3 shows the results for three cases, Austria, Belgium-Flanders and Hungary (see Table A2 in the online appendix for the full table). We see theoretically relevant differences between the two measures. 14 In Belgium-Flanders there is a high correlation in the ordering between our populism measure and the CHES anti-elite measure. However, in Austria and Hungary the correlation is extremely low. In Austria, NEOS (The New Austrian and Liberal Forum) is clearly an anti-elitist or anti-establishment party, but it is also the least populist party of Austria. In Hungary, the governing party Fidesz scores low on the CHES anti-elitism measure. This is unsurprising as Fidesz constitutes the national political elite. By contrast, our populism variable lists Fidesz and Jobbik as Hungary's most populist parties, respectively.
Differentiating between parties' populism and anti-elitism can be essential for drawing valid inferences. For instance, Norris and Inglehart (2019) uses the CHES anti-elite salience measure as a proxy for a populism-pluralism continuum. Parties such as Fidesz, the Norwegian Progress Party, and the Danish People's Party score low on the anti-elite salience dimension but high on the authoritarian dimension. The authors therefore conclude that these parties demonstrate "socially conservative values, but . . . endorse[s] less populist philosophies toward governance" (2019, p. 240).
We also compare our expert survey dataset with the Global Populism Database (GPD, Hawkins et al., 2019) and with the PopuList dataset (Rooduijn  We also see two deviant cases. While SYRIZA has a populism score of 7.63, prime-minister Alexis Tsipras' speeches received a GPD populism score of 0.25 on the 0-2 scale. And while UK prime-minister Theresa May's speeches were moderately populist in the GPD data (0.49), the UK Conservatives scored low on our populism measure (2.26). Besides methodological differences, these divergences may reflect differences between populism voiced in speeches and populism as apparent in party overall ideology as well as differences between the degree of populism of the party leader and the party as a whole. Next, we compare our measure of populism with the PopuList, a dichotomous coding of populist parties on the basis of country experts. The pointbiserial r of 0.673 shows that the two measures generally correlate well. Figure  3b displays the number of cases included or not included in the PopuList according to their populism score. 16 The majority of the cases with a score higher than 7.5 are included in the PopuList, but nine cases were not. These include the Belgian radical left PVDA-PTB, the Greek parties KKE and Golden Dawn, and the Portuguese BE and CDU. Among cases with moderate populism scores (between 5 and 7.5), we see that some cases are included in the PopuList, while others are not. For instance, among radical left parties the Dutch SP and the German Die Linke are included in the PopuList, but the French, PCF, the Spanish IU, and the Polish party Razem were not. On the right side of the spectrum, we see that the Flemish N-VA was not included in the PopuList, but the Brothers of Italy (FdI) was. Save four cases, the vast majority with a score lower than five are not included in the PopuList. One of the exceptions is the Bulgarian "Citizens for European Development" (GERB). Yet, also in the GPD this party scores low on populism (0.13). We strongly believe that these results point to key strengths of our approach. The range of moderately populist parties (between 5 and 7.5) shows how difficult it is to adjudicate parties' populism in a dichotomous manner. A continuous approach avoids crude, and perhaps arbitrary, classification. Moreover, the discrepancy between different methods highlights the importance of consulting multiple sources to assess parties' populism, as our expert survey explicitly does.

Party Ideology, Political Style and Party Organization
Populist parties are found across the left-right spectrum (Mudde & Rovira Kaltwasser, 2013;Rooduijn & Akkerman, 2017). While most populist parties, such as the French National Rally or the Italian League, can be found on the far right side of the spectrum, others, such as the Spanish party Podemos or the German Left Party, are found on the left. The ideational approach, in particular, contends that populism cannot be the sole defining ideological characteristic of a party. Rather, populism is attached to a host ideology such as nativism or a left-wing position on the economy (Mudde, 2007;Stanley, 2008).
To assess parties' ideological profiles we also asked the country experts to evaluate parties' positions on the "overall left-right" dimension, on the "economic left-right" issue, on "nativism," on the "immigration" issue and the issue of "European integration." 17 In addition, we asked experts to assess parties on moral issues pertaining to the tension between civil liberties and law and order as well as on traditional versus liberal social lifestyle.
While the expert survey clearly prioritizes the ideational approach to populism, other approaches regard populism to denote a particular political style (Moffitt & Tormey, 2014) or organizational strategy (Heinisch & Mazzoleni, 2016;Weyland, 2001Weyland, , 2017. The expert survey includes two items to assess political style and two items for party organization. To empirically assess a parties' political style, experts evaluated the extent to which a party appealed to emotions in their political communications ("emotional appeal") and to extent to which they regard political decision-making to be complex versus advocating common-sense solutions ("complex versus common-sense politics"). Party organization was measured with an item gauging the extent to which the party is characterized by personalized leadership ("personalized leadership"). Experts were also instructed to assess the extent to which members are involved in decision making, the degree to which there is space within the party for internal debate, and the degree to which decision making is inclusive ("intra-party democracy"). Table 4 shows the EFA including all the expert survey's items. The EFA yields two factors. The first factor shows that besides the five ideational populism items also the two items pertaining to political style as well as the EU integration item load on the same factor. The second factor includes the items pertaining to party ideology as well as the items on party organization. This dimension can perhaps be best described as "left-right cum organization" dimension. The fact that political style but not political organization loads on  Table A4 in the online appendix shows the results of the iterated principal exploratory factor analysis without a set minimum Eigenvalue.
the same factor as the five populism items suggests that a populism definition based on political style approximates ideational definitions. A definition based on party organization, on the other hand, produces a different set of parties-at least in the European context. As virtually all populist parties, within the European context, exhibit high levels of Euroscepticism (de Vries & Edwards, 2009), it is not surprising that the European integration item loads on the first factor.

Advantages of a New Approach
Measuring populism with an expert survey in a precise and multi-dimensional way for wide range of political parties offers researchers a number of advantages. For instance, it allows us to assess the distribution of the parties' degree of populism vis-à-vis other ideological indicators. Figure 4 plots the populism score and overall left-right score for all 250 political parties included in the dataset complemented with a quadratic prediction plot. As expected, we see a curvilinear pattern in which far left and far right parties display higher levels of populism. That said, we also see a number of rather centrist parties with considerable levels of populism, such as the Romanian Social Democratic Party (PSD), the Lithuanian Peasant and Greens Union (LVLS) and the Cypriot Citizen's Alliance (SYPOL). If we consider contentious cases of populist parties, the advantages of a precise, continuous measure of populism come to the fore. As noted, there is debate in the literature over whether parties such as the Dutch Socialist Party (SP) or the Berlusconi's Go Italy! (FI) are cases of populist parties. Figure 5 shows the positions of Dutch and Italian political parties on the populism scale and on the left-right scale. And while parties such as the Five Star Movement (M5S), the League (LN), the Freedom Party (PVV) and Forum for Democracy (FvD) are clear cases of populism in both countries, the SP and the FI can be considered as moderate cases of populism. In contrast to a dichotomous operationalization of populism, empirical analyses based on our populism variable are thus able to exploit the variation in populism among parties.
One of the strengths of measuring the constitutive dimensions of the ideational conception of populism in a disaggregated way is that it allows us to explore the extent to which the nature of populism among various parties varies. It is likely that different types of populist parties differ in the kind of populism they express. Figure 6 shows the promise of assessing the variation in adherence to the different dimensions of populism by comparing the radical right and the radical left. The radical left displays lower values on the item  "indivisible people," which assesses the extent to which the ordinary people are a homogenous community, and lower values on the item "general will," which estimates the extent to which the ordinary people's interests are singular. This is in line with the argument that the populist left holds a more pluralistic notion of the people (Kioupkiolis & Katsambekis, 2018;Stavrakakis & Katsambekis, 2014).

Populism and Ideological Dimensions
Prominent applications of the ideational approach combine populism with other ideological dimensions (Mudde, 2007;Stanley, 2008). A continuous measure of populism for a wide array of political parties allows us to assess the association between populism and these attaching ideological dimensions. In Table 5, we regress four important determinants of party ideology on the populism variable: economic left-right dimension, nativism, EU integration, and traditional-liberal lifestyle. Overall, we see that populist parties tend to be slightly more left-wing in economic terms, more nativist, less supportive of EU integration and slightly more liberal in terms of social lifestyle. This view is broadly in line with qualitative assessments of populist parties' ideologies (Mudde, 2007;van Kessel, 2015). The small positive effect of liberal lifestyle positions is, in fact, not surprising given the moral politics of left populist parties as well as of "libertarian" populist radical right parties such as the Danish The New Right (D) or the Dutch Freedom Party (PVV) (see also Lancaster, 2019;Spierings et al., 2017).
At the same time, these average effects hide substantial variation between countries and between individual parties on these ideological dimensions. To illustrate this point, we plot parties' degree of populism and the four ideological dimensions for Greece, Germany, Spain, the Netherlands and Italy in Figures 7 and 8. These five countries combine North and South Europe with populist parties across the left-right ideological spectrum. 18 The OLS analysis in Table 5 shows a statistically significant effect for a left-wing economic position. However, if we split the cases up into countries and we examine the data more descriptively we see that the picture is more nuanced. There are also populist parties with centrist economic positions, such as the Italian M5S or the Greek radical right party Independent Greeks (ANEL). And, as noted above, other radical right parties adopt clear economic right-wing positions, such as the Dutch FvD. And finally, Euroscepticism is an important component of populist parties, both on the left and the right (Halikiopoulou et al., 2012;Meijers, 2017).
At the same time, research recognizes that populist parties also differ in their opposition to the European integration process. Some populist parties reject the EU outright, while other populist parties are more accommodative, taking a more reformist position (de Wilde & Trenz, 2012;Pirro & van Kessel, 2017;Vasilopoulou, 2011). Our data reflects this nuanced picture. Populist parties such as the Dutch Freedom Party (PVV) and Forum for Democracy (FvD), the Alternative for Germany (AfD), Golden Dawn (XA) in Greece, and the Italian League (LN) are highly Eurosceptic. Yet, the Five   Star Movement (M5S) is moderately Eurosceptic, while Podemos (P) and SYRIZA are populist parties with moderate pro-EU positions. Nativism is recognized as a core ideological trait of populist radical right parties (Mudde, 2007), while the literature atones that nativism is less salient for populist left-parties (McGowan & Keith, 2016;Mudde & Rovira Kaltwasser, 2013). We also see that there is considerable variation in terms of nativism among the individual cases. While populist radical right parties are strongly nativist, the three left-wing parties Podemos, SYRIZA, and Popular Unity (LAE) are clearly non-nativist whilst still scoring high on the populism dimension. This confirms the picture that populist left parties see non-native elements in society as less of threat than populist radical right parties. At the same time, the Dutch left-wing party SP exhibits moderately nativist positions.
Authoritarianism is a core characteristic of populist radical right parties (Mudde, 2007). Authoritarianism refers to support for stronger positions regarding law and order, as well as support for conservative positions regarding social lifestyle (Mudde, 2007). Here we focus on social lifestyle. 19 Turning to the individual cases, we see interesting patterns. Political parties that score higher on the populism variable, also diverge considerably on the lifestyle variable. The five countries are highly illustrative. There are, for example, differences between the Alternative for Germany (AfD), Golden Dawn (XA), the Forum for Democracy (FvD), the Freedom Party (PVV) (all populist radical right parties), and Podemos regarding lifestyle. The AfD, Golden Dawn and the Independent Greeks espouse the most traditional views of society of all populist parties in the five countries. However, the FvD and the PVV, also populist radical right parties, are located more in the middle of the 11-point scale-appearing to be less morally conservative than the other populist radical right parties. Podemos, and the other left-wing parties, on the other hand, are clearly the most liberal populist parties.

Conclusion
Despite the flourishing of populism research, a measure of populism in political parties that is in line with the theoretical underpinnings of populism, that is precise, and that offers comprehensive coverage of Europe's political parties has been lacking. While previous attempts to measure populism on the supply-side of politics have clearly been highly instrumental in making both theoretical and empirical advances, existing approaches suffer from a number of weaknesses. First, the operationalization of populism is often not consistent with the conceptual work on populism, as spelled out by the dominant ideational approach (Hawkins, 2009;Mudde, 2004Mudde, , 2017. Some studies deviate from the ideational approach's conceptual thrust, while others fail to capture the multidimensionality of the concept of populism. Second, existing studies put forward imprecise renderings of populism as a dichotomous measure. The inability to accurately assess the degree of populism impedes the potential explanatory quality of populism and hampers the adjudication of border line cases of populism. Third, many studies have focused on a limited subset of parties or countries or revolve around populism as apparent in the executive branch of government. To address these challenges, we propose an alternative method of measuring populism in political parties with the 2018 Populism and Political Parties Expert Survey (POPPA). We find that expert surveys measuring the constitutive dimensions of populism in a fine-grained manner for a wide range of political parties shows great promise. We argue that populism is a latent construct and that the ideational approach to populism can be operationalized with five separate items: "Manichean worldview," "indivisible people," "general will," "people-centrism," and "anti-elitism." Loading strongly on a single factor, these five dimensions constitute the latent construct of populism. IRT analyses indicate that the items capture the latent construct of populism with satisfactory levels of measurement precision.
A fine-grained, continuous measure of populism allows us to identify and differentiate between parties with low, moderate and high levels of populism. This alleviates the problem of determining whether a party with slight populist tendencies is populist or not. The added empirical value of our populism measure comes to the fore when we assess the qualitative differences between our populism measure and a measure assessing the salience of anti-elite rhetoric . As populism and anti-elite rhetoric are conceptually and empirically not the same, our data shows that we should be careful in using anti-elite salience as a proxy for populism, as do Norris and Inglehart (2019).
Comparing the radical right and the radical left, we show that measuring the multidimensional concept of populism in a disaggregated manner harbors great promise for comparing the kind of populism parties espouse-as the radical left exhibits a more pluralistic perspective of the "ordinary people." Further research should be done to explore and explain this diversity in populist ideology. Examining populism in conjunction with its "attaching" ideologies, we see that populist parties tend to hold economically left-of-centre positions, they are Eurosceptic, they are nativist and, on average, they are slightly more liberal in terms of social lifestyle. At the same time, we show that these average estimates mask a high degree of variation between parties.
While we encourage users of the data to employ our populism variable based on the ideational approach to populism, scholars are also free to use the expert survey's items in any way they see fit. Those who regard populism to be a political style, for instance, may choose to combine the items "Manichean worldview" and "anti-elitism" with items measuring stylistic elements of party behavior, such as "complex versus common-sense politics" and "emotional appeal." And while we estimated the mean expert judgment per party, we also provide researchers with the median estimates of the items, as advised by Lindstädt, Proksch, and Slapin (2020). Moreover, one can operationalize populism using different aggregation rules, as Wuttke et al. (2020) propose.
We believe that our approach to measuring populism among political parties is an important step forward for the comparative politics literature on populism. The proposed populism measure enables us to better understand and explain the ubiquity and diversity of populism in European democracies. Descriptively, our data conveys a consensus of expert judgments regarding which parties can be considered populist. While it is perhaps most geared to quantitative empirical research, qualitative studies of populism can also benefit from our measure for case selection purposes. Moreover, a continuous measure of populism for all relevant parties in the party system also permits a broadening of the empirical focus of populism research to the study of populism among mainstream parties. The full coverage of parties also alleviates concerns regarding "selection on the dependent variable" in populism research.
And while we encourage scholars to think of and examine populism in a continuous manner, our data also allows scholars to create dichotomous classifications (i.e., for logistic or multinomial regression models) in a systematic way-either based on theoretical insight or on empirical benchmarks (i.e., z-scores). Given our multi-dimensional approach to measuring populism and given the data's extensive coverage of cases, future research can examine variety and diversity of populism(s) as apparent in different party systems.
In addition, the data can be used in conjunction with other party-or country-level data sets. As such, scholars are able to examine the explanatory power of parties' populism for a wide range of political science research questions. For instance, the data allows scholars to explore how populism affects the dimensionality of political competition in European party systems. Future research could also address to which extent populism shapes parties' voting behavior in national parliaments or in the European Parliament. In addition, the data allows systematic study of the effects of populism on the quality of democracy in European countries.
While our empirical focus lies on Europe, our approach may be successfully implemented to study other regions in the world. While textual approaches have the benefit of measuring populism in the past (Hawkins & Castanho Silva, 2018;Rooduijn et al., 2014), future rounds of the expert survey will allow us to collect data longitudinally-as the CHES projects have convincingly demonstrated (Bakker et al., 2015a). In addition, the success of our expert survey approach to measuring populism has demonstrated that its potential for including measures of parties' positions to key attributes of liberal, representative democracies. Such an addition to our approach would allow us to address unanswered questions regarding the quality of democracy and the rule of law in modern-day democracies.
Future applications of our approach should consider including bridging questions or anchoring vignettes which multiple experts from multiple countries code to assess with more precision commonalities and differences in experts' scale interpretation (Bakker et al., 2014;Marquardt & Pemstein, 2018). Although we would like to emphasize the distinct advantages of expert surveys, we would also like to caution against "expert fatigue." We should be careful that the success of expert surveys will not herald its downfall as experts are confronted with an abundance of expert surveys on a variety of topics. The community should therefore think about creative ways to coordinate the fielding of expert surveys.

Authors' Note
This work has been previously presented at the 9th Annual Conference of the European Political Science Association (EPSA) in Belfast, 20-22 June 2019 and at the General Conference of the European Consortium for Political Research (ECPR), Wroclaw, 4-7 September 2019. It was also presented at the 5th Prague Populism Conference Conference programme, 27-28 May 2019 and at the workshop "Rechtspopulismus in Westeuropa: Schwesterparteien oder entfernte Cousins?" at the Bonner Akademie für Forschung und Lehre praktischer Politik (BAPP), January 10, 2019.

Supplemental Material
Supplemental material for this article is available online at the CPS website http:// journals.sagepub.com/doi/suppl/10.1177/0010414020938081 Notes 1. The data is publicly available at the Harvard Dataverse: https://doi.org/10.7910/ DVN/8NEL7B. An interactive Web app made with R Shiny that allows users to explore the data is available at: https://poppa.shinyapps.io/poppa/ 2. Iceland, Latvia and Luxembourg were initially included as well, but have been dropped due to insufficient response. 3. In some countries that had upcoming elections we included parties that were doing well in national polls. 4. For instance, in the KzB case we selected the KzB, not its constituting parties such as the BSP. 5. We have recorded responses from eight or more experts per country, except for Bulgaria, Cyprus, Lithuania, Malta and Norway. Hence, the data from these countries should be used with caution. Figure A1 in the online appendix shows the number of experts measured per party. Over 90 per cent of the party observations have five or more expert judgments and 70 per cent have more than eight judgments. The number of expert judgments per party is moreover not related to party size (Pearson's R between "vote share" and "no. of experts" is −0.0033, n = 212). 6. The codebook of the expert survey with a full list of parties can be found in the appendix. 7. With respect to reliability, inspection of the average standard deviations per party in Figure A3 in the online appendix show that the five items perform similarly in comparison with other expert surveys, such as the CHES 2014 data. 8. Replication files for the analyses in this manuscript can be found at: https://doi. org/10.7910/DVN/22K7WU. 9. Alternatively, we can predict parties' populism score on the basis of ordinal IRT prediction scores. These yield substantively similar results to the factor-predicted scores. Figure A4 in the online appendix shows a plot of parties' left-right positions and an IRT-predicted populism score. 10. Alternatively, Wuttke et al. (2020) do not consider populism on the individuallevel to be a latent construct. Rather, populism is seen as a "non-compensatory, multi-dimensional" construct in which the minimum score of all components constitutes the overall populism score. This "Goertzian" necessary conditions approach implies all components have equal weight in the overall construct. Both approaches are feasible with the data presented here. Figure A2 (a replication of Figure 4 presented below) in the online appendix shows a plot of parties' left-right positions and a populism measure with "Goertzian" aggregation. 11. Graphs in this article are made with Stata schemes by Bischof (2017). 12. All items contribute substantially to the latent construct. Yet, while excluding the item "indivisible people" decreases the level of information, the 4-item IRT also yields a high information level of around 12 with a similar theta interval (see Figure A5 in the online appendix). However, we believe the ideational definition of populism is best captures by all five items, both theoretically and empirically. 13. The high congruence of the left-right measures between our dataset and CHES as well the Manifesto Project points to high overall congruent validity of our data (see Figure A6). 14. The different timing of the surveys may also account for some of the divergence.
Yet, arguably, the most important difference between the two measures is theoretical in nature. 15. For Poland, we selected the speeches by prime-minister Beata Szydło. 16. Table A3 in the appendix shows an overview of relevant case comparisons. 17. The wording of the items "left-right (economy)," "immigration" and "European integration" were taken from the 2014 CHES survey . 18. Figure A7 in the appendix shows scatterplots with overlaid linear prediction plot for all parties and all countries for each of the four items: "Left-Right (Economic)," "Nativism," "EU Support" and "Trad.-Lib. Lifestyle." 19. Due to multicollinearity concerns we could not examine the relationship between populism and law and order and social lifestyle simultaneously in the OLS regression. Table A5 in the appendix shows Pearson's R correlation coefficients for the items "Left-Right (Economic)," "Nativism," "Immigration," "Law & Order" and "Trad.-Lib. Lifestyle."