Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)

Today, the unprecedented quantity of easily accessible data on social, political, and economic processes offers ground-breaking potential in guiding data-driven analysis in social and human sciences and in driving informed policy-making processes. The need for precise and high-quality information about a wide variety of events ranging from political violence, environmental catastrophes, and conflict, to international economic and health crises has rapidly escalated (Porta and Diani, 2015; Coleman et al. 2014). Governments, multilateral organizations, local and global NGOs, and social movements present an increasing demand for this data to prevent or resolve conflicts, provide relief for those that are afflicted, or improve the lives of and protect citizens in a variety of ways. For instance, Black Lives Matter protests[1] and conflict in Syria[2] events are only two examples where we must understand, analyze, and improve the real-life situations using such data.

Event extraction has long been a challenge for the natural language processing (NLP) community as it requires sophisticated methods in defining event ontologies, creating language resources, and developing algorithmic approaches (Pustojevsky et al. 2003; Boroş, 2018; Chen et al. 2021). Social and political scientists have been working to create socio-political event databases such as ACLED, EMBERS, GDELT, ICEWS, MMAD, PHOENIX, POLDEM, SPEED, TERRIER, and UCDP following similar steps for decades. These projects and the new ones increasingly rely on machine learning (ML) and NLP methods to deal better with the vast amount and variety of data in this domain (Hürriyetoğlu et al. 2020). Automation offers scholars not only the opportunity to improve existing practices, but also to vastly expand the scope of data that can be collected and studied, thus potentially opening up new research frontiers within the field of socio-political events, such as political violence & social movements. But automated approaches as well suffer from major issues like bias, generalizability, class imbalance, training data limitations, and ethical issues that have the potential to affect the results and their use drastically (Lau and Baldwin 2020; Bhatia et al. 2020; Chang et al. 2019). Moreover, the results of the automated systems for socio-political event information collection may not be comparable to each other or not of sufficient quality (Wang et al. 2016; Schrodt 2020).

Socio-political events are varied and nuanced. Both the political context and the local language used may affect whether and how they are reported. Therefore, all steps of information collection (event definition, language resources, and manual or algorithmic steps) may need to be constantly updated, leading to a series of challenging questions: Do events related to minority groups are represented well? Are new types of events covered? Are the event definitions and their operationalization comparable across systems? This workshop aims to seek answers to these questions, to inspire innovative technological and scientific solutions for tackling the aforementioned issues, and to quantify the quality of the automated event extraction systems. Moreover, the workshop will trigger a deeper understanding of the performance of the computational tools used and the usability of the resulting socio-political event datasets l.

We invite contributions from researchers in computer science, NLP, ML, AI, socio-political sciences, conflict analysis and forecasting, peace studies, as well as computational social science scholars involved in the collection and utilization of socio-political event data. Social and political scientists will be interested in reporting and discussing their approaches and observe what the state-of-the-art text processing systems can achieve for their domain. Computational scholars will have the opportunity to illustrate the capacity of their approaches in this domain and benefit from being challenged by real-world use cases. Academic workshops specific to tackling event information in general or for analyzing text in specific domains such as health, law, finance, and biomedical sciences have significantly accelerated progress in these topics and fields, respectively. However, there is not a comparable effort for handling socio-political events. We hope to fill this gap and contribute to social and political sciences in a similar spirit. We invite work on all aspects of automated coding of socio-political events from mono- or multi-lingual text sources. This includes (but is not limited to) the following topics

  • Extracting events in and beyond a sentence
  • Training data collection and annotation processes
  • Event coreference detection
  • Event-event relations, e.g., subevents, main events, causal relations
  • Event dataset evaluation in light of reliability and validity metrics
  • Defining, populating, and facilitating event schemas and ontologies
  • Automated tools and pipelines for event collection related tasks
  • Lexical, Syntactic, and pragmatic aspects of event information manifestation
  • Development and analysis of rule-based, ML, hybrid, and human-in-the-loop approaches for creating event datasets
  • COVID-19 related socio-political events
  • Applications of event databases
  • Online social movements
  • Bias and fairness of the sources and event datasets
  • Estimating what is missing in event datasets using internal and external information
  • Novel event detection
  • Release of new event datasets
  • Ethics, misinformation, privacy, and fairness concerns pertaining to event datasets
  • Copyright issues on event dataset creation, dissemination, and sharing
  • Qualities of the event information on various online and offline platforms

Shared tasks

We have prepared a cross-lingual (Spanish, Portuguese, and English), which was only in English in previous events we organized, a socio-political event information classification and extraction task. Moreover, we have prepared two additional data challenges about protests pertaining to BlackLivesMatter and COVID19[3]. These challenges will be about automatically replicating the spatiotemporal distribution of a manually curated events list. Common Crawl News will be exploited as a data source. Moreover, a Twitter dataset will be shared with the participants as well.

A Codalab page will be set up for each task. Participating teams will be required to submit a report, which will be peer reviewed, describing the methods and results they ran on the data. We will encourage submissions of the system papers on our available benchmarks (ProtestNews @ CLEF 2020 and AESPEN @ LREC 2020). We expect at least 100 subscriptions and up to 30 submissions for the shared tasks in total.

Organization Committee

Ali Hürriyetoğlu (Koc University, Turkey)

Hristo Tanev (Joint Research Centre (JRC), European Commission, Italy)

Vanni Zavarella (Joint Research Centre (JRC) of the European Commission, Italy)

Reyyan Yeniterzi (Sabancı University, Turkey)

Aline Villavicencio (University of Sheffield, the United Kingdom; and Institute of Informatics, Federal University of Rio Grande do Sul, Brazil)

Erdem Yörük (Koc University, Turkey),

Deniz Yuret (Koc University, Turkey)

Jakub Piskorski  (Polish Academy  of Sciences, Poland)

Gautam Kishore Shahi (University of Duisburg-Essen, Germany)

Keynotes

We invite leading scholars in computational linguistics and social and political sciences to deliver keynote speeches on collecting and exploiting socio-political event information. We can allocate some funds from the budget of our EU projects (ERC StG and ERC Twinning) for the registrations of one of the invited speakers (as registration of one speaker will be covered by the main conference budget), which can be up to around 500 Euros.

Social and political sciences: Prof. Eck has accepted our invitation as a keynote speaker. Kristine Eck is an Associate Professor at the Department of Peace and Conflict Research at Uppsala University, where she serves as the Director of the Uppsala Rotary Peace Center. Her research interests concern coercion and resistance, including human rights, police misconduct, state surveillance, and conflict data production. She served as the Director of the Uppsala Conflict Data Program (UCDP) 2017-2018 and has been a Visiting Researcher at Oxford University, Copenhagen University, the University of Notre Dame, and Kobe University. Dr. Eck’s research has been funded by the Swedish Research Council, the Swedish Foundation for Humanities and Social Sciences, and the Norwegian Foreign Ministry.

Computational linguistics: TBD

Program Committee

Tommaso Caselli (University of Groningen, the Netherlands),

Osman Mutlu (Koc University, Turkey),

Fırat Duruşan (Koc University, Turkey),

Ali Safaya (Koc University, Turkey),

Bharathi Raja Asoka Chakravarthi (Insight SFI Centre for Data Analytics, the United Kingdom),

Gautam Kishore Shahi (University of Duisburg-Essen, Germany),

Jakub Piskorski (Polish Academy of Sciences, Poland),

Matina Halkia (European Commission – Joint Research Centre, Italy),

Benjamin J. Radford (UNC Charlotte, the United States),

Mark Lee (University of Birmingham, the United Kingdom),

YiJyun Lin (University of Nevada, the United States),

Fredrik Olsson (RISE, Sweden),

Kristine Eck (Uppsala University, Sweden),

Nelleke Oostdijk (Radboud University, the Netherlands),

Francielle Vargas (University of São Paulo, Brazil),

Farhana Liza (University of Essex, the UK),

Nicoletta Calzolari (Institute for Computational Linguistics, Italy),

Milena Slavcheva (Bulgarian Academy of Sciences, Bulgaria),

Harish Tayyar Madabushi (University of Birmingham, tthe United Kingdom),

Ritesh Kumar (Dr. Bhimrao Ambedkar University, India).

References

Bhatia, S., Lau, J. H., & Baldwin, T. (2020). You are right. I am ALARMED–But by Climate Change Counter Movement. arXiv preprint arXiv:2004.14907.

Boroş, E. (2018). Neural Methods for Event Extraction. Ph.D. thesis, Université Paris-Saclay.

Chang, K. W., Prabhakaran, V., & Ordonez, V. (2019, November). Bias and fairness in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): Tutorial Abstracts.

Chen M., Zhang H., Ning Q., Li M., Ji H., Roth D. (2021). Event-centric Natural Language Understanding. Proc. The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI2021) Tutorial. URL: https://blender.cs.illinois.edu/paper/eventtutorial2021.pdf

Coleman, P. T., Deutsch, M., & Marcus, E. C. (Eds.). (2014). The handbook of conflict resolution: Theory and practice. John Wiley & Sons.

Della Porta, D., & Diani, M. (Eds.). (2015). The Oxford handbook of social movements. Oxford University Press.

Hürriyetoğlu, A., Zavarella, V., Tanev, H., Yörük, E., Safaya, A., & Mutlu, O. (2020, May). Automated Extraction of Socio-political Events from News (AESPEN): Workshop and Shared Task Report. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020 (pp. 1-6).

Lau, J. H., & Baldwin, T. (2020, July). Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 2908-2913).

Pustejovsky, J., Castano, J. M., Ingria, R., Sauri, R., Gaizauskas, R. J., Setzer, A., … & Radev, D. R. (2003). TimeML: Robust specification of event and temporal expressions in text. New directions in question answering, 3, 28-34.

Schrodt, P. A. (2020, May). Keynote Abstract: Current Open Questions for Operational Event Data. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020.

Wang, W., Kennedy, R., Lazer, D., & Ramakrishnan, N. (2016). Growing pains for global monitoring of societal events. Science, 353(6307), 1502-1503.

[1] http://protestmap.raceandpolicing.com, accessed on September 28, 2020.

[2] https://www.cartercenter.org/peace/conflict_resolution/syria-conflict-resolution.html, accessed on September 28, 2020.

[3] https://en.wikipedia.org/wiki/Protests_over_responses_to_the_COVID-19_pandemic, accessed on September 28, 2020.