For more than a decade the Neurocognition and Action (NCA) research group at Bielefeld University, Germany, has conducted research on analysing human memory structures and recently applied these findings to the next generation of user-adaptive cognitive assistance systems.
The NCA group employs researchers from a diverse range of fields such as cognitive assistance, cognitive psychology, sports and movement science, biology, neuroscience, computer science and engineering.
This interdisciplinary approach allowed them to establish software for assessing long-term memory structures and predicting human errors for a given task, for example using specific high-tech machines, building birdhouses from wood, or doing exercises at home.
The required data is gathered by a semi-automatised survey: Using a special app created by the NCA group, users are asked to judge whether they think that different pairs of actions belong together during the task execution.
Subsequently, the software calculates the person’s memory structure for each task. Based on this structure, researchers assess whether the person will know what to do in a given situation or not.
Job-related cognitive assistance and training
In many instances, people with cognitive disabilities face difficulties in organising work-related tasks autonomously. Some may fail to conduct all relevant tasks to prepare a coffee machine, while others may mix-up specific sequential tasks during their daily work. Based on our extensive previous research, we ascertain that mental representations play an important role in planning and executing complex actions and movements. In the joint project Adaptive Cognitive Training (ACT) between the NCA group and the von Bodelschwingh Foundation Bethel), one of the largest health care centres in Europe we investigated the memory structure of different work-related tasks by deploying the Structural Dimensional Analysis of Mental Representation (SDA-M) and found significant differences between the instructors and the handicapped trainees.
ACT uses this cognitive assessment to identify differences in the memory structure on an individual level. To support the learning process, we developed an adaptive training terminal to analyse the cognitive structure automatically and provide an adaptive feedback.
The terminal provides individualised cognitive assistance training advices and helps to identify unstructured working tasks. Depended on the trainee, the training can include self-instruction training or imagery training.
Furthermore, instructors use the terminal to address trainees in a more targeted way and intervene earlier. The regularly repeated cognitive diagnostic assessment makes changes and progress for the instructors and the trainees visible and enriches the learning progress. The terminal can easily be applied to different kind of tasks and complex movements, which makes it to a useful tool for accompany learning and working processes in rehabilitation, sport or other fields.
The personal trainer chair
In 2060, it is estimated that every third person (34%) will be at least 65 years old, and the number of 70-year-old persons will be twice as high as the number of new-born children in Germany. This demographic challenge requires the development of assistive technologies to support elderly people to live a self‐sufficient life in an age‐appropriate way. In the context of the Kognihome project, the Personal Trainer Chair (PTC) was developed to support and motivate daily exercise of all family members in a smart-home environment. It is designed to be a ubiquitous interaction system: by combining mental diagnostics, marker-free motion tracking and natural interaction, the system seamlessly integrates into the living room and provides a motivating and fun exercise tool, a relaxation mode and entertainment through gamiﬁed activities.
The PTC provides a closed-loop sensorimotor learning, interaction and training system. Neurocognitive diagnostic tools developed in our research group enable the SPC to identify user-speciﬁc problems in the execution of action sequences based on measured memory structures. The PTC will then adapt the training routines and provide adequate feedback. Monitoring the emotional and motivational status will be used to ensure long‐term motivation by proper goal setting.
A biofeedback based relaxation mode facilitates an integrated radar sensor to measure vital data and uses a virtual reality setting to help smart home inhabitants to relax after a long work day.
Smart glasses as a cognitive assistance device
The information about user’s memory structures and task knowledge has also been transferred to intelligent technical systems, so that they can adapt to individual expertise. In a current project at Bielefeld University’s Cognitive Interaction Technology Excellence Centre (CITEC), smart glasses are used to assist users in different activities. The so-called ‘ADAMAAS’ glasses use augmented reality and eye tracking technology to show helpful illustrations and virtual 3D animations overlaid with real world objects to provide assistance exactly at the time and place where the user needs it.
In order to reduce unwanted distraction from the task due to visual clutter, a cognitive diagnostics component predicts in which action steps this assistance is actually required.
NCA researchers are currently testing the glasses with handicapped people in an educational programme, with senior citizens, and in an industrial assembly setting.
While the current research focuses on a handful of specific activities to optimise the system’s accuracy and user experience, the potential future applications are theoretically almost unlimited.
The scientists believe that in a few years intelligent smart glasses like the ADAMAAS system from Bielefeld can be widely used as a cognitive assistance device during our daily life. Just like a universal remote control for hi-fi devices in the living room, intelligent smart glasses could easily be programmed to assist in new activities.
NCA Professor Thomas Schack expects such next-generation assistance systems to help people further develop and live a more satisfying life by enabling them to engage in new activities.