Blog CRC1646

The conferences of the International Society for Gesture Studies (ISGS) are one of the world’s leading meetings focused on multimodal language. It brings together researchers from diverse fields including gesture use, sign language linguistics, animal communication, language evolution, speech therapy, and machine learning. This year’s ISGS conference took place at Nijmegen, home of the Max Planck Institute for Psycholinguistics, a longstanding hub for multimodal language studies. It was thus an excellent venue for project C02 and C03 to present its ongoing work on multimodal creativity in speech-gesture production and furthermore an opportunity for members of C02 to meet with researchers with whom they plan to collaborate, including two members of the conference’s executive committee, Aslı Özyürek and Judith Holler.

The conference took place over three days, with most of the program devoted to talks in parallel sessions organized by topic, in addition to five keynote presentations and three poster sessions. Many of the talks were interpreted in real time from English to International Sign or vice versa, to facilitate participation by deaf researchers. One of the highlights of the conference was a keynote presentation by Professor Johanna Mesch, given in International Sign, about deafblind communication. She explained how deafblind sign language speaker use both sign language as well as nonconventional expressive gestures in tactile modalities, such as touching each other’s hands, to adapt to nonvisual communication.

Project C02 presented its research in two talks: Under the title “Co-speech Gestures Accompanying Linguistic Inventions”, Alon Fishman spoke in the session on semiotics. Afterwards, he received valuable suggestions on how to proceed with annotating and analysing of the collected multimodal data. Alon was also approached by colleagues who are currently conducting, or are planning to conduct in the future, studies on different research questions using similar methodologies. These discussions yielded a fruitful exchange of ideas and could potentially lead to collaborations, exploring how different experimental manipulations influence gesture use in multimodal interaction. The second talk was given by Lisa Gottschalk with the title “The Impact of Gestures on the Perception of Creative Linguistic Constructions by Humans and Virtual Agents” during the session focused on Machine Learning, AI, and Virtual Reality. She presented the contributions from the computer science component of the project, emphasizing the role of the perception of linguistic inventions in enhancing the modeling of co-speech gestures and their integration within AI-driven multimodal agents. Attendees demonstrated great interest in the work of C02, providing constructive suggestions for extending this analysis to languages beyond German. This exploration may yield insights into linguistic variations and speaker-agent alignment, potentially fostering collaborations with other researchers in the field. The third contribution was the poster presentation entitled “Patterns of multimodal creativity in people with and without aphasia: Development of a new coding scheme“, given by Lotta Heidemann. She discussed her coding scheme for patterns of multimodal creativity in people with and without aphasia which was newly developed during a pilot study for the project C03. In exchange with interested colleagues Lotta received precious suggestions on how to further develop the coding scheme and the field of gesture research in people with aphasia. This discussion with other gesture researchers resulted in a valuable exchange of ideas and may continue beyond the conference.

Overall, participation in ISGS10 allowed projects C02 and C03 to introduce the CRC to the wider gesture studies community, to foster relationships with researchers working on similar topics, and to deepen their understanding of the interplay between gestures and linguistic creativity, thereby contributing to the CRC's core goals of mapping the phenomenon of linguistic creativity and outlining the mechanisms involved.

Alon Fishman © ISGS 10

Lisa Gottschalk © Lotta Heidemann

Lotta Heidemann © Lisa Gottschalk

Projektteam C02 © Sascha Hermannski

Hintere Reihe v.l.n.r.: Alon Fishman, Stefan Kopp, Lisa Gottschalk & Joana Cholin

Vordere Reihe v.l.n.r.: Lisa Hudalla, Alice Schirrmeister

Projektteam C03 © Sascha Hermannski

Hintere Reihe v.l.n.r.: Sina Osthoff, Anna Köster, Laureen Wall, Lotta Heidemann & Mareike Hartmann

Vordere Reihe v.l.n.r.: Kerstin Richter, Carola de Beer, Esra Doksanbir & Johanna Firley 

Abwesend: Martina Hielscher-Fastabend