Digital Trace Data in the Study of Public Opinion: An Indicator of Attention Toward Politics Rather Than Political Support
Abstract
In this article, we examine the relationship between metrics documenting politics-related Twitter activity with election results and trends in opinion polls. Various studies have proposed the possibility of inferring public opinion based on digital trace data collected on Twitter and even the possibility to predict election results based on aggregates of mentions of political actors. Yet, a systematic attempt at a validation of Twitter as an indicator for political support is lacking. In this article, building on social science methodology, we test the validity of the relationship between various Twitter-based metrics of public attention toward politics with election results and opinion polls. All indicators tested in this article suggest caution in the attempt to infer public opinion or predict election results based on Twitter messages. In all tested metrics, indicators based on Twitter mentions of political parties differed strongly from parties’ results in elections or opinion polls. This leads us to question the power of Twitter to infer levels of political support of political actors. Instead, Twitter appears to promise insights into temporal dynamics of public attention toward politics.
Get full access to this article
View all access and purchase options for this article.
References
Adcock R., Collier D. (2001). Measurement validity: A shared standard for qualitatitve and quantitative research. The American Political Science Review, 95, 529–546.
Asur S., Huberman B. A. (2010). Predicting the future with social media. In Huang X. J., King I., Raghavan V., Rueger S. (Eds.), WI-IAT 2010: Proceedings of the 2010 IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (pp. 492–499). Washington, DC: IEEE.
Barberá P., Rivero G. (2014). Understanding the political representativeness of twitter users. Social Science Computer Review. Retrieved from http://ssc.sagepub.com/content/early/2014/11/27/0894439314558836.abstract
Bermingham A., Smeaton A. F. (2011). On using Twitter to monitor political sentiment and predict election results. In Bandyopadhyay S., Okumura M., SAAIP: Proceedings of the workshop sentiment analysis where AI meets psychology at the international joint conference for natural language processing (IJCNLP) (pp. 2–10). Chiang Mai, Thailand: Asian Federation of Natural Language Processing.
Bollen J., Mao H., Zeng X. J. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2, 1–8.
Carmines E. G., Zeller R. A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage.
Ceron A., Curini L., Iacus S. M. (2014). Every tweet counts? How sentiment analysis of social media can improve our knowledge of citizens’ political preferences with an application to Italy and France. New Media & Society, 16, 340–358.
Ceron A., Curini L., Iacus S. M. (2015). Using sentiment analysis to monitor electoral campaigns: Method matters—evidence from the United States and Italy. Social Science Computer Review, 33, 3–20.
Chaffee S. H., Metzger M. J. (2001). The end of mass communication? Mass Communication & Society, 4, 365–379.
Chakrabarti D., Punera K. (2011). Event summarization using tweets. In ICWSM 2011: Proceedings of the 5th international AAAI conference on weblogs and social media (pp. 66–73). Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
Choi H., Varian H. R. (2009). Predicting the present with google trends. Mountain View, CA: Google Inc. Retrieved from http://static.googleusercontent.com/media/www.google.com/de//googleblogs/pdfs/google_predicting_the_present.pdf
Contractor D., Faruquie T. A. (2013). Understanding election candidate approval ratings using social media data. In Schwabe D., Almeida V., Glaser H., Baeza-Yates R., Moon S. (Eds.), WWW 2013: Proceedings of the 22nd international conference on World Wide Web (pp. 189–190). Geneva, Switzerland: International World Wide Web Conferences Steering Committee.
Cronbach L. J., Meehl P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
Diaz F., Gamon M., Hofman J., Kıcıman E., Rothschild D. (2014). Online and social media data as a flawed continuous panel survey. Microsoft Research Working Paper. Retrieved from http://research.microsoft.com/en-us/UM/redmond/projects/flawedsurvey/flawedsurvey.pdf
DiGrazia J., McKelvey K., Bollen J., Rojas F. (2013). More tweets, more votes: Social media as a quantitative indicator of political behavior. PLoS One, 8, e79449.
Duggan M., Smith A. (2013). Social media update. Pew Internet & American Life Project. Retrieved from http://pewinternet.org/Reports/2013/Social-Media-Update.aspx
Fink C., Bos N., Perrone A., Liu E., Kopecky J. (2013). Twitter, public opinion, and the 2011 Nigerian Presidential election. In Bilof R. (Ed.), SocialCom 2013: The 5th IEEE international conference on social computing (pp. 311–320). Washington, DC: IEEE.
Franch F. (2013). (Wisdom of the crowds)2: 2010 UK election prediction with social media. Journal of Information Technology & Politics, 10, 57–71.
Gaurav M., Srivastava A., Kumar A., Miller S. (2013). Leveraging candidate popularity on Twitter to predict election outcome. In Grossman R. L., Uthurusamy R. (Eds.), SNAKDD 2013: Proceedings of the 7th workshop on social network mining and analysis (Article no. 7). New York, NY: ACM.
Gayo-Avello D. (2011). Don’t turn social media into another ‘Literary Digest’ poll. Communications of the ACM, 54, 121–128.
Gayo-Avello D. (2012). No, you cannot predict elections with Twitter. IEEE Internet Computing, 16, 91–94.
Gayo-Avello D. (2013). A meta-analysis of state-of-the-art electoral prediction from Twitter data. Social Science Computer Review, 31, 649–679.
Gerring J. (2012). Social science methodology (2nd ed.). New York, NY: Cambridge University Press.
Ginsberg J., Mohebbi M. H., Patel R. S., Brammer L., Smolinski M. S., Brilliant L. (2009). Detecting influenza epidemics using search engine query data. Nature, 457, 1012–1014.
González-Bailón S., Paltoglou G. (2015). Signals of public opinion in online communication: A comparison of methods and data sources. The ANNALS of the American Academy of Political and Social Science, 659, 95–107.
Hand D. J. (2004). Measurement theory and practice. London, UK: Wiley.
Hopkins D., King G. (2010). A method of automated nonparametric content analysis for social science. American Journal of Political Science, 54, 229–247.
Howison J., Wiggins A., Crowston K. (2011). Validity issues in the use of social network analysis with digital trace data. Journal of the Association for Information Systems, 12, 767–797.
Huberty M. (2015). Can we vote with our tweet? On the perennial difficulty of election forecasting with social media. International Journal of Forecasting, 31, 992–1007.
Hyndman R. J., Koehler A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22, 679–688.
Jensen M. J., Anstead N. (2013). Psephological investigations: Tweets, votes, and unknown unknowns in the republican nomination process. Policy & Internet, 5, 161–182.
Jungherr A. (2013). Tweets and votes, a special relationship: The 2009 federal election in Germany. In Weber I., Popesc A. M., Pennacchiotti M. (Eds.), PLEAD 2013: Proceedings of the 2nd workshop politics, elections and data (pp. 5–14). New York, NY: ACM.
Jungherr A. (2014). The logic of political coverage on Twitter: Temporal dynamics and content. Journal of Communication, 64, 239–259.
Jungherr A. (2015). Analyzing political communication with digital trace data: The role of Twitter messages in social science research. Heidelberg, Germany: Springer.
Jungherr A., Jürgens P., Schoen H. (2012). Why the pirate party won the German election of 2009 or the trouble with predictions: A response to Tumasjan, A., Sprenger, T.O., Sander, P.G. & Welpe, I.M. “Predicting elections with Twitter: what 140 characters reveal about political sentiment.”” Social Science Computer Review, 30, 229–234.
Jungherr A., Schoen H., Jürgens P. (2016). The mediation of politics through Twitter: An analysis of messages posted during the campaigns for the German federal elections 2009 and 2013. Journal of Computer-mediated Communication, 21, 50–68.
Kane M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–535.
Lazer D., Kennedy R., King G., Vespignani A. (2014). The parable of Google flu: Traps in big data analysis. Science, 343, 1203–1205.
Lazer D., Pentland A., Adamic L., Aral S., Barabási A. L., Brewer D.…Alstyne M. V. (2009). Computational social science. Science, 323, 721–723.
Lin Y. R., Margolin D., Keegan B., Lazer D. (2013). Voices of victory: A computational focus group framework for tracking opinion shift in real time. In Schwabe D., Almeida V., Glaser H., Baeza-Yates R., Moon S. (Eds.), WWW 2013: Proceedings of the 22nd international world wide web conference (pp. 737–748). Geneva, Switzerland: International World Wide Web Conferences Steering Committee.
MacKuen M. B., Erikson R. S., Stimson J. A. (1989). Macropartisanship. American Political Science Review, 83, 1125–1142.
Marchetti-Bowick M., Chambers N. (2012). Learning for microblogs with distant supervision: Political forecasting with Twitter. In Daelemans W. (Ed.), EACL ‘12: Proceedings of the 13th conference of the European chapter of the association for computational linguistics (pp. 603–612). Stroudsburg, PA: Association for Computational Linguistics.
Margolis M., Resnick D. (2000). Politics as usual: The Cyberspace “revolution.” Thousand Oaks, CA: Sage.
McKelvey K., DiGrazia J., Rojas F. (2014). Twitter publics: How online political communities signal electoral outcomes in the 2010 U.S. House election. Information, Communication & Society, 17, 436–450.
Messick S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5–11.
Metaxas P. T., Mustafaraj E., Gayo-Avello D. (2011). How (not) to predict elections. In Vinciarelli A., Pantic M., Bertino E., Zhan J. (Eds.), SocialCom 2011: The 3rd IEEE international conference on social computing (pp. 165–171). Washington, DC: IEEE.
Murphy J., Link M. W., Childs J. H., Tesfaye C. L., Dean E., Stern M.…Harwood P. (2014). Social media in public opinion research: Executive summary of the aapor task force on emerging technologies in public opinion research. Public Opinion Quarterly, 78, 788–794.
O’Connor B., Balasubramanyan R., Routledge B. R., Smith N. A. (2010). From tweets to polls: Linking text sentiment to public opinion time series. In Hearst M., Cohen W., Gosling S. (Eds.), ICWSM 2010: Proceedings of the 4th international AAAI conference on weblogs and social media. Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
Rainie L., Smith A., Schlozman K. L., Brady H., Verba S. (2012). Social media and political engagement. Pew Internet & American Life Project. Retrieved from http://pewinternet.org/Reports/2012/Political-engagement.aspx
R Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
Ruths D., Pfeffer J. (2014). Social media for large studies of behavior. Science, 346, 1063–1064.
Sakaki T., Okazaki M., Matsuo Y. (2010). Earthquake shakes twitter users: Real-time event detection by social sensors. In Rappa M., Jones P., Freire J., Chakrabarti S. (Eds.), WWW 2010: Proceedings of the 19th international conference on the World Wide Web (pp. 851–860). New York, NY: ACM.
Sartori G. (1970). Concept misformation in comparative politics. The American Political Science Review, 64, 1033–1053.
Schoen H., Gayo-Avello D., Metaxas P. T., Strohmaier M., Gloor P. (2013). The power of prediction with social media. Internet Research, 23, 528–543.
Skoric M., Poor N., Achananuparp P., Lim E. P., Jiang J. (2012). Tweets and votes: A study of the 2011 Singapore general election. In Sprague R. H. Jr (Ed.), HICSS 2012: Proceedings of the 45th Hawaii international conference on system science (pp. 2583–2591). Washington, DC: IEEE Computer Society.
Soler J. M., Cuartero F., Roblizo M. (2012). Twitter as a tool for predicting elections results. In Karray F., Polat F. (Eds.), ASONAM 2012: IEEE/ACM international conference on advances in social networks analysis and mining (pp. 1194–1200). Los Alamitos, CA: IEEE Press.
Thapen N. A., Ghanem M. M. (2013). Towards passive political opinion polling using Twitter. In Cocea M., Gabe M., Wiratunga N., Göker A. (Eds.), Proceedings of the BCS SGAI workshop on social media analysis 2013 (pp. 19–34). Aberdeen, UK: School of Computing Science and Digital Media, Robert Gordon University.
Tjong E., Sang K., Bos J. (2012). Predicting the 2011 Dutch Senate election results with Twitter. In Proceedings of the workshop on semantic analysis in social media (pp. 53–60). Stroudsburg, PA: Association for Computational Linguistics.
Tourangeau R., Rips L. J., Rasinski K. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
Tsakalidis A., Papadopoulos S., Cristea A. I., Kompatsiaris Y. (2015). Predicting elections for multiple countries using Twitter and polls. IEEE Intelligent Systems, 30, 10–17.
Tumasjan A., Sprenger T. O., Sandner P. G., Welpe I. M. (2010). Predicting elections with Twitter: What 140 characters reveal about political sentiment. In Hearst M., Cohen W., Gosling S. (Eds.), ICWSM 2010: Proceedings of the 4th international AAAI conference on weblogs and social media (pp. 178–185). Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
Vaccari C., Valeriani A., Barberá P., Bonneau R., Jost J. T., Nagler J., Tucker J. (2013). Social media and political communication: A survey of Twitter users during the 2013 Italian general election.” Rivista Italiana di Scienza, 43, 325–55.
Wang W., Rothschild D., Goel S., Gelman A. (2014). Forecasting elections with non-representative polls. International Journal of Forecasting. Retrieved from http://www.sciencedirect.com/science/article/pii/S0169207014000879
Whiteley P., Clarke H. D., Sanders D., Stewart M. C. (2013). Affluence, austerity and electoral change in Britain. Cambridge, England: Cambridge University Press.
Wickham H. (2009). ggplot2: Elegant graphics for data analysis. New York, NY: Springer.
All links were accessible on October 18, 2015.
Biographies
Andreas Jungherr is a research fellow at the chair for political psychology at the University of Mannheim, Germany. [email protected]
Harald Schoen holds the chair for political psychology at the University of Mannheim, Germany. [email protected]
Oliver Posegga is a research fellow at the chair for social network analysis at the University of Bamberg, Germany. [email protected]
Pascal Jürgens is a research associate at the Department of Mass Communication at the University of Mainz, Germany. [email protected]
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
- online_appendices.pdf
- 426.83 KB
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: February 15, 2016
Issue published: June 2017
Keywords
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in Social Science Computer Review.
View All Journal MetricsArticle usage*
Total views and downloads: 2442
*Article usage tracking started in December 2016
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 57 view articles Opens in new tab
Crossref: 65
- Can Social Media Engagement Predict Election Results? Bandwagon Effects of Tweets About US Senate Candidates
- Religionsanalyse und Theorieentwicklung
- From the Syrian to Ukrainian refugee crisis: Tracing the changes in the Italian Twitter discussions through network analysis
- The role of social media in enhancing adolescents’ digital life quality- survey study in the UAE
- Reporting after removal: the effects of journalist expulsion on foreign news coverage
- Can We Measure the Structural Dimension of Social Capital with Digital Footprint Data?—An Assessment of the Convergent Validity of an Indicator Extracted from Digital Footprint Data
- Stop trying to predict elections only with twitter – There are other data sources and technical issues to be improved
- Harnessing Government Digital Attention: Reducing Carbon Emissions Through the Pathways of Digitalization
- Topic modeling for short texts: comparative analysis of algorithms
- 2023 International Conference on Power, Electrical Engineering, Electronics and Control (PEEEC)
- View More
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
