New development from CITEC researchers at Bielefeld University

Instead of having to tediously search for the right part with one hand and flip through the pages of the instruction manual with the other hand, the smart glasses from the Adamaas system display what step comes next directly in the user’s field of vision. This works for both operating a coffee machine and assembling a bird house. Professor Dr. Thomas Schack, Dr. Kai Essig, and Dr. Matthias Schröder further developed the smart glasses at Bielefeld University’s Cluster of Excellence Cognitive Interaction Technology (CITEC). Using “computer vision”, the system can autonomously recognize objects and steps of an activity, and based on this, the glasses show appropriate step-by-step support directly in the display of the glasses. The Bundesministerium für Bildung und Forschung (BMBF, Federal Ministry of Education and Research) is providing the project with 1.2 million Euro in funding. A film about Adamaas can now be viewed online.

“When it comes to building a bird house, people with cognitive disabilities become quickly overwhelmed. With the smart glasses, they can build a bird house step by step, and are appropriately instructed based on their needs,” says Dr. Kai Essig, who is contributing to the development of the Adamaas system. “Using previous measurements taken of the users’ mental representation structures, the glasses can be personally customized for the subject’s needs.” Subjects can conveniently complete these measurements beforehand using a tablet, in which they assign different images to specific steps of an activity. If the person, for instance, has difficulties mounting the roof on the bird house, the system knows this and can accordingly provide more in-depth instructions for this step. In addition to this, the Adamaas system monitors heartrate and displays soothing tips in the display before the person even gets overwhelmed.

Previously, the Adamaas researchers had used this system, for example, to provide assistance in operating a modern coffee maker. The advantage here was that the coffee machine used (project partner Miele) immediately recognizes the current status and could relay this information immediately to the glasses. “The difficulty of building a bird house is that the individual parts and assembly steps have to be recognized directly in front of the glasses. Here, what is self-evident for humans poses significant challenges for a technical system,” says Dr. Matthias Schröder, who programmed the Adamaas system’s ‘computer vision’. “With the help of a camera, the glasses are not only  able to recognize where the object is located, but also which part the user has picked up with their hand or glanced at with their eyes.” The system can then give instructions based on the particular situation and display images to provide personally customized support with the steps of the activity. “With ‘computer vision,’ the Adamaas system can provide assistance for virtually any building process,” says Professor Dr. Thomas Schack, who heads the Adamaas project. “As such, subjects no longer have to fiddle around with the instruction manual and laboriously perform the steps of the activity with one hand: they can work with both hands and get the instructions for the steps one by one in the display. This is also exciting for industry.”

In addition to Prowerk and the v. Bodelschwingh Bethel Foundations’ Seniorenzentrum Breipohls Hof (Senior center Breipohls Hof), the family-owned company Hettich is also a partner on the Adamaas project. Operating instructions are often of little help when assembling a closet drawer powered by an electric motor. By displaying the steps of the assembly process step by step, the glasses would provide ideal support to people like apprentices or even workers. “It is conceivable, for example, that Hettich staff working in assembly could use the glasses to quickly learn how the drawers have to be put together,” says Schack. “The Adamaas system is also configured for Work 4.0 in that it provides support for assembly tasks using augmented reality.”

Currently, the researchers are conducting Adamaas studies with subjects from Hettich, Prowerk, and the Seniorenzentrum Breipohls Hof. In the future, studies with workers at Hettich are also planned, which will allow the researchers to even better address the individual needs of the users and explore new dimensions of research. In the future, the Adamaas system is planned to be able to work without an extra camera, using only the integrated camera in the glasses to provide support to the user.

Additional partners on the Adamaas project include eye tracking specialist SensoMotoric Instruments (SMI), who has developed and provided the hardware for the glasses. Velamed GmbH Medizintechnik und Biomechanische Konzepte (Velamed GmbH Medical Technology and Biomechanical Concepts) is developing a robust and efficient system to record and analyze vital signs. Furthermore, various faculties at Bielefeld University are also working together with external partners. In addition to Professor Dr. Helge Ritter (Neuroinformatics research group), Professor Dr. Thomas Schack (Neurocognition and Action - Biomechanics research group), Dr.-Ing. Sven Wachsmuth (Central Labs), and Dr. Kai Essig (Neurocognition and Action – Biomechanics research group) from CITEC, the Faculty of Psychology and Sports Science as well as the Faculty of Technology and the Network Bielefeld 2000plus are participating. The Faculty of Business Administration and Economics is also supporting Adamaas by developing a marketing and sales concept through its Institute for Technological Innovation, Market Development, and Entrepreneurship (iTIME).