Blog CRC1646

The workshop “Multimodal creativity in speech and gesture production” addressed the question of how humans employ speech and gesture creatively in communicatively challenging situations, e.g., when conventionalized or commonly used communicative resources are insufficient. It featured presentations from projects C02, C03 and C04, as well as from invited guest speakers. It additionally included two targeted discussion sessions, one about operationalizing and measuring multimodal creativity, and one about cognitive and computational modeling of multimodal creativity.

The three talks by CRC projects all presented experimental results along with novel approaches to analyzing those results. A recurring theme throughout all three talks was the limited applicability of existing literature on creative gestures, and the challenge of extending the CRC’s working definition of linguistic creativity to the gestural channel. These issues were tackled directly in the discussion sessions, where the workshop participants split into groups to raise concrete problems, offer potential solutions, and share useful tools and techniques.

The talks by the invited guest speakers did not necessarily address linguistic creativity explicitly, but they were all nevertheless tremendously valuable and relevant to the workshop’s topics. James Trujillo (University of Amsterdam), Anna Palmann (University of Amsterdam) and Jiahao Yang (University of Bath) presented empirical investigations of multimodal communication in various communicatively challenging situations, e.g., in noisy environments, directly relating to the CRC’s research questions and hypotheses about the contextual conditions that support or require creative use of language. Angela Grimminger (Paderborn University) and Andy Lücking (Goethe University Frankfurt) presented empirical findings and formal representations, respectively, regarding relatively conventional multimodal productions, a necessary step to understanding deviation from such conventions, i.e., instances of genuine creativity.

Contribution from CRC-Members:

• Lisa Gottschalk, Alon Fishman, Joana Cholin & Stefan Kopp (C02): Creativity in speech and gesture and its perception in humans and AI
• Lotta Heidemann (C03): Coding creative gesture use by people with neurogenic language and communication disorders
• Luyao Wang (C04): Multimodal strategies for resolving lexical challenges: The role of creative gestures in four- and five-year-old children Program & Abstracts

Program & Abstracts

© SFB 1646