Workshop description

Computational models of language are becoming omnipresent in research, both as objects of investigation and as tools for scientific inquiry. But what are their true capabilities? Do LLMs understand language, and, if so, do they do so like humans do? For what kinds of applications are they useful and reliable, and what kinds of research questions can they help us answer? This workshop brings together researchers interested in discussing current perspectives in natural language processing and its applications within the humanities, social sciences, and cognitive sciences. Over three sessions, each consisting of a keynote followed by two short talks by local early-career researchers, we will tackle some of these key questions, interleaving examples of applied research on LLMs with talks focusing on critically evaluating model capabilities. Keynotes will be given by three external speakers who are running cutting-edge research in different areas of NLP. Anyone who is doing work on or has an interest in this domain is welcome to register and join.

Programme

Keynote speakers

Dirk Hovy, Bocconi University
Title: Speaking Fluent Gibberish: Why Language Models Do Not Understand Language.
Abstract: In a world where AI can generate everything from translations to poetry and code, it's tempting to believe that these models truly understand us. Despite its linguistic prowess, today's generative AI still resembles a clever mimic rather than a genuine linguist. In this talk, we will draw on thought experiments like the Chinese Room and the octopus on a telegraph wire to look at the limitations of statistical models, their inability to grasp context and nuance, and the risks of overestimating their capabilities. We'll find out what this has to do with getting lost in translation, sounding like our own grandfathers, and encountering an incomprehensible Lion. Drawing on philosophy, linguistics, and NLP, we investigate what it truly means to 'understand' a language, emphasizing the theoretical and practical implications for future language technology.
Website: http://dirkhovy.com/

Maria Antoniak, Pioneer Centre for AI & University of Colorado Boulder
Title: From Stories to Sonnets: Data-Centered NLP for Creative Works
Abstract: In this talk, I'll share two recent studies that use natural language processing (NLP) techniques to model creative works like stories and poetry. In the first part of the talk, I'll discuss NLP approaches for story detection and analysis, focusing on how NLP methods can help us study storytelling at large scales and across diverse contexts. In the second part, I'll discuss the poetic capabilities of large language models (LLMs), focusing on audits of the vast pretraining datasets used to build these models. Both studies will highlight the challenges in creating open evaluation datasets for creative works and the importance of interdisciplinary collaboration between NLP and the humanities.
Website: https://maria-antoniak.github.io/

Sandro Pezzelle, University of Amsterdam
Title: Implicit and underspecified language as a communicative testbed for large language models
Abstract: The language we use in everyday communicative contexts exhibits a variety of phenomena—such as ambiguity, missing information, or semantic features expressed only indirectly—that make it often implicit or underspecified. Despite this, people are good at understanding and interpreting it. This is possible because we can exploit additional information from the linguistic or extralinguistic context and shared or prior knowledge. Given the ubiquity of these phenomena, NLP models must handle them appropriately to communicate effectively with users and avoid biased behavior, that can be potentially harmful. In this talk, I will present recent work from my group investigating how state-of-the-art transformer large language models (LLMs) handle these phenomena. In particular, I will focus on the understanding of sentences with atypical animacy (“a peanut fell in love”) and on the interpretation of sentences that are ambiguous (“Bob looked at Sam holding a yellow bag”) or where some information is missing or implicit (“don't spend too much”). I will show that, in some cases, LMs behave surprisingly similarly to speakers; in other cases, they fail quite spectacularly. I will argue that having access to multimodal information (e.g., from language and vision) should, in principle, give these models an advantage on these semantic phenomena—as long as we take a perspective aware of the communicative aspects of language use.
Bio: I study human-like natural language understanding and generation in text-only large language models (LLMs) and their multimodal versions combining language-and-vision  (VLMs). As such, my work combines methods and insights from Natural Language Processing, Computer Vision, and Cognitive Science. My current research interests span LLM and VLM evaluation and interpretability inspired by human cognition, how the learning of semantic and pragmatic abilities compares in humans and machines, and whether (and how) the cognitive mechanisms underlying human language communication can be used to develop better language models. I co-authored articles in top-tier conferences (ACL, EMNLP, EACL, NAACL, CoLM) and journals (TACL, Cognition, Cognitive Science). I am a member of the ELLIS society, a faculty member of the ELLIS Amsterdam Unit, and a board member of SigSem, the ACL special interest group in computational semantics. In 2024, I organized the UnImplicit workshop at EACL 2024 on understanding implicit and underspecified language.