Natural Language Processing (NLP) is a thriving field. With the ever increasing amount of available data and computing power, more and more advanced applications are being developed. However, most NLP approaches continue to rely on text data only. Modeling the grounding of linguistic meanings in the physical and behavioral context can play a key role in opening new avenues in NLP, as well as in generating insight in how we humans learn and understand our languages. Unfortunately collecting and processing suitable non-linguistic contextual data, for example through computer vision, still remains a formidable challenge. Virtual worlds in which people interact online provide us with an interesting opportunity to bypass this problem. In this talk I present a dataset that was collected through 'The Restaurant Game'. Thousands of people played the roles of the customer and the waitress in a virtual restaurant online. Their actions in this virtual environment provide context for their dialogue. Therefore this dataset gives us insight into natural language in context, while circumventing many of the complexities that using real world context would present. I demonstrate the potential of virtual world grounding by showing that meanings of words and phrases can be automatically learned from the dialogue and actions in this restaurant scenario, even with fairly simple methods and very basic initial assumptions.

About Computational Linguist Hilke Rechman