Researchers at Cluster of Excellence CITEC analyze brain waves

Just how cartoonish can a character in a movie or video game look so that the audience relates to him? Researchers at Bielefeld University’s Cluster of Excellence Cognitive Interaction Technology (CITEC) investigated this question, analyzing how the brain responds differently to photo-realistic and more stylized characters. Their work resulted in the following finding: cartoon characters can elicit the same emotions as human actors, but they don’t do not stay in the viewer’s memory as long. Along with their teams, psychology Professor Dr. Johanna Kißler and computer science Professor Dr. Mario Botsch presented their study on Thursday, 23 March 2017 in “Scientific Reports,” a journal published by the Nature Publishing Group.

“We asked ourselves: how realistic do cartoon characters have to be, and how stylized can virtual avatars be? When do we feel emotionally connected to these characters – and when do we not?” says computer scientist Eduard Zell, from the research group Computer Graphics and Geometry Processing, which is led by Professor Dr. Mario Botsch. In his dissertation, Zell deals with the design of virtual characters. In user studies, he analyzed how humans individually perceive these characters. In order to understand how the human brain fundamentally perceives such artificial characters, Zell and psychologist Dr. Sebastian Schindler, from the research group Affective Neuropsychology, developed the experiment for the study now being presented.

Professor Dr. Johanna Kißler’s group worked on the issue of how the human brain processes emotion. To do this, the team measured brain waves using electrodes. A cap equipped with electrodes recorded the electrical signals of the brainwaves on the subject’s scalp, and a computer then analyzed the signals. An electroencephalogram (EEG) device can measure when a person responds especially strongly to a stimulus, and afterwards the brain areal that generates the response can be calculated.

In the study, participants viewed 18 images, one after the other, of the same person. Three of these images were real photographs, portraying the person with happy, angry, and neutral facial expressions. In addition to this, for each expression (happiness, anger, and neutral), there were five variations of the image in which the person was depicted as an increasingly stylized cartoon figure.

Strongest Reaction to Extremely Cartoonish and Realistic Images
The participants viewed each respective image for one second. The EEG device recorded how strongly the brain responded to each individual image. Using the EEG data, Kißler’s team calculated which area of the brain the signals came from. A central finding here is that “the test subjects reacted very strongly to the extremes – both the real photographs and the images in which the character looked most like a cartoon character,” explains Johanna Kißler. “Humans are thus capable of creating a strong mental connection to cartoon characters.” EEG devices record a special amplitude generated in the back of the head. This amplitude occurs with a delay of 170 milliseconds and is thus called N170. “And it’s precisely this amplitude that we found to be particularly strong with the extremely cartoonish and realistic images,” explains Kißler.

“According to our measurements, the moderately stylized photos did not provoke a strong reaction – probably because the figures were perceived as unreal.”  This measurement explains why people experience an uncanny feeling when they see a figure that almost looks real. This phenomenon is called the “Uncanny Valley Hypothesis,” and under this theory, people notice instinctively for realistically animated characters that they are not actually real people. They immediately notice the smallest deviations that contribute to making the character unrealistic. “With cartoon characters, this effect does not occur because such characters are not meant to simulate real people in the first place. Accordingly, the brain generates its responses to them in an object processing area rather than the human face processing area that responds to the real faces”

Emotional Facial Expressions from Virtual Characters Also Impact Viewers
Regardless of the degree of realism depicted in the images, the brain reacts especially strongly when the person in question sees an emotional facial expression. “This could be a reason why people like watching cartoon movies. They identify with the characters and feel the same emotions as with real human actors in feature films,” says Kißler.

There is, however, a striking difference. “We quickly forget what the cartoon figure went through in a movie. The things experienced by real human actors can stay with us for days after we see a film.” This is because the brain processes the cartoon images and real pictures differently. “Our study demonstrates that the real photographs are processed in a particular region of the visual cortex that is responsible for perceiving people. It creates a mental connection to the characters and stores the faces in long-term memory,” explains the psychologist. “With the very stylized images, a part of the brain reacts that is responsible for perceiving objects. The brain, however, does not establish a sense of identification with such objects, and does not store them long term.”

The results of this study could be used in the design of avatars (virtual characters) and robots. “Accordingly, characters with a cartoonish, stereotypical appearance are particularly suitable for short-term interactions with humans, like having a robot, for instance, bring guests at a restaurant to their table,” says Professor Dr. Mario Botsch, one of the leaders of this study. “If a personal relationship is necessary, a humanoid character is more appropriate.”

The project KogniHome, which is coordinated by CITEC, makes use of such characters. Virtual agents that look like a member of the family are designed to provide assistance to senior citizens, for example, in their smart apartment. The research group Computer Graphics and Geometry Processing is developing these avatars. For the psychology study with Johanna Kißler, the computer scientists 3D-scanned  two people and turned them into computer-generated avatars.