uni.news

Computer scientists develop a method for monitoring by listening in factories, operating rooms, and postal logistics centres

When the alarm light starts blinking in the control room of a factory, the problem has already occurred. Computer scientists at the Cluster of Excellence Cognitive Interactive Technology (CITEC) at Bielefeld University and the University of Vienna have developed a method that allows control room staff to monitor several processes at the same time, which enables them to take preventative action. The trick: processes are coded with sounds. Workers hear, for example, whether there is enough material on the conveyor belt and can react before the supply is used up. 

Processes must be monitored in a wide range of industries – whether at a factory, in the operating room, at a postal logistics centre, or in space flight. “Up until now, monitoring processes has been a visually supported field of work,” explains CITEC computer scientist Thomas Hermann. “Computer screens or displays at control desks show whether everything is in order. With large quantities information, though, something can easily slip by unnoticed. Staff must maintain a high level of concentration to keep all processes in check,” says Hermann, who leads the “Ambient Intelligence” research group at CITEC. “With our new system, we use additional acoustic signals. Our method thus enables a kind of passive monitoring, that is, surveillance that can be accomplished alongside other tasks.”

Dr. Hermann developed the new system together with Tobias Hildebrandt and Professor Dr. Stefanie Rinderle-Ma from the University of Vienna. Using the example of a production plant, a simulation shows how the method works. Each station is given a different sound: the delivery is announced with the sound of chirping birds, bees buzzing are assigned to another station, and the sounds of branches rustling in the wind are heard at another.” Outgoing shipments are coded with the sound of dripping water. If everything is running normally, all four sounds are discretely in the background. “We chose these woodland sounds because they compose an acoustic ambience that is both pleasant to listen to and unobtrusive,” explains Hermann. A critical situation is then introduced at one of the stations –the finished product is beginning to back up at the outgoing shipping station – and the sound that belongs to this station becomes increasingly loud. A staff member can then react before a disruption occurs. In this case, the worker would determine that the products get loaded at an earlier stage, thus preventing an emergency stop on the shop floor. According to Dr. Hermann’s colleague Tobias Hildebrandt, who studies business information technology at the University of Vienna, the system is not only suitable for production facilities. “It could be introduced in almost every industry in which processes are centrally controlled or monitored – everywhere from hospitals to  traffic control desks for trains and buses,” says Hildebrandt.

“Monitoring processes by listening has several advantages,” explains Thomas Hermann. “Distraction is less of an issue in comparison to visual monitoring. Moreover, we use our ears to perceive everything that is going on around us. With our eyes, we must look precisely at the thing that is important to the current task. Generally speaking, the advantage of listening is that it happens all the time. We can close our eyelids, but there are no “earlids” that can be shut,” explains the computer scientist. Furthermore, according to Hermann, auditory stimuli are processed more quickly than visual stimuli. “The special feature of listening is that people are able to recognize the smallest changes in tones. For example, in a car, one can hear subtle changes in road conditions based on the sounds of driving on the street.”

The new system is called “SoProMon.” The name stands for “Sonification system for process monitoring as secondary task.” The research paper authored the three developers was awarded “Best Paper” in November 2014 at the IEFF International Conference on Cognitive Infocommunications (CogInfoCom) in Vietri sul Mare, Italy.

Thomas Hermann is an expert in sonification, the systematic representation of data using non-speech sound. In 2012, he worked with Berlin media artists to produce a software that perceptualizes the German “Twitterscape” in sounds. The software automatically assigns a sound to a topic. When a Twitter user writes a short message touching on the topic, the assigned sound can be heard. Dr. Hermann’s research group “Ambient Intelligence” develops intelligent environments, novel interactive objects, and attentional systems to support humans in everyday life. In addition to sonification, the researchers also focus on multimodal interaction, which is the principle that a device communicates with its user through several senses – from hearing to touch – and can also be controlled by the user via different sensory inputs.

Further information is available online at:
•    Video and Audio Demonstration of the SoProMon System: http://pub.uni-bielefeld.de/data/2695709
•    Website of the “Ambient Intelligence“ research group: www.cit-ec.de/ami
•    Bielefeld sonification researcher Dr. Thomas Hermann codes the German Twitterscape in sounds (Press release from 16 January 2012, in German): http://ekvv.uni-bielefeld.de/blog/pressemitteilungen/entry/bielefelder_sonifikationsforscher_dr_thomas_hermann

Original Publication:
Tobias Hildebrandt, Thomas Hermann, Stefanie Rinderle-Ma: A Sonification System for Process Monitoring as Secondary Task. Proceedings of the 5th IEEE Conference on Cognitive Infocommunication (in print), http://eprints.cs.univie.ac.at/4211/1/HildebrandtHermannRinderleMa2014-coginfocom_author.pdf

Kontakt:
Dr. Thomas Hermann, Bielefeld University
Cluster of Excellence Cognitive Information Technology (CITEC)
Telephone: +49 521 106- 12140
Email: thermann@techfak.uni-bielefeld.de

Tobias Hildebrandt, University of Vienna
Faculty of Informatics
Telephone: +43-1-4277-79141
Email: Tobias.Hildebrandt@univie.ac.at