Hey human, how are you doing?


Machines are taking over more and more tasks. Ideally, they should also be capable to support the human in case of poor performance. To intervene appropriately, the machine should understand what is going on with the human. Fraunhofer scientists have developed a diagnostic tool that recognizes user states in real time and communicates them to the machine. The camera firmly focuses on the driver’s eyes. If they are closed for more than one second, an alarm is triggered. This technique prevents the dangerous micro-sleep at the wheel. «It is not always as easy for a machine to detect what state the human is in, as it is in this case», says Jessica Schwarz from the Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE in Wachtberg, just south of Bonn. For her doctoral thesis, the graduate psychologist examined the question of how to very precisely determine user states, what influence these may have on incorrect behavior and how automated systems can use this information. «For complex applications it is not sufficient to focus on only one impact factor», says the scientist. An increased heart rate, for example, does not automatically mean that a person is stressed. This can have various causes. Schwarz therefore examined what factors specifically impact human performance and created a holistic model that provides a detailed view on user states and their causes.In her model, she differentiates between six dimensions of user state that impact human performance: workload, motivation, situation awareness, attention, fatigue and the emotional state. She uses physiological and behavioral measures to detect these states. In addition, she combines these with external factors such as task, environmental factors, current level of automation and time of day, as well as individual factors—such as the user’s experience. «This allows us to assess the user’s state in more detail and also identify causes for critical states», Schwarz explains her procedure. The FKIE plans to complete the research project by the end of the year. The researchers are now on the look-out for industry partners. «The technology is very close to application. The know-how to develop specific products for individual use cases is already available», says Schwarz. Potential fields of application can be found in all highly automated tasks where critical user states can be a safety issue. For example, monotonous monitoring tasks in control rooms or training systems for pilots could be optimized by this technology. «Machines play an increasingly important role, but are also becoming more and more complex. This poses new challenges for the cooperation between human and machine. Adaptive systems that recognize different situations and adapt to them can solve known automation problems. A key aspect of this, however, is that not only the user understands the machine, but that the machine also understands the state of the human. We have now taken the first step towards this goal», Schwarz sums up.