iHMi

iHMi

The project is Financed by the Bulgarian National Science Fund.

Project Duration: 06.12.19 - 06.12.2022.


Investigation on intelligent human-machine interaction interfaces, capable to recognize high-risk emotional and cognitive conditions

At present, intelligent HMI do not yet have sensitivity to a person's cognitive and general physical condition, including levels of acute stress, attention level, cognitive load, or specific negative emotional states. Therefore, machines are deprived of the ability to assess the degree of risk of error due to human overload or fatigue. This greatly impedes effective human-robot collaboration and increases the risks of industrial accidents caused by the lack of awareness about the human aspects. All this motivates the need for in-depth research and the creation of new methods in the direction of machines to acquire at least partial sensitivity to the condition and behaviour of the human factor, and can prevent situations with a high risk of danger to human health and life. In this regard, the present project aims to provide support to the research activities of the team towards the development of new methods and approaches, and to assist in the development of intelligent human-machine interfaces by integrating awareness to the level of stress, cognitive and general physical state of humans.

The main goal of this project is to explore the opportunities of enhancing contemporary human-machine interfaces by the integration of new functionalities related to the automated recognition of acute stress, the degree of attention and cognitive load, as well as certain negative emotional states. In particular, these functionalities aim to identify combinations of factors, states, and behaviours that predetermine the increased risks and levels of danger in human-robot collaboration systems. An important component in these intelligent human-machine interfaces is the creation of effective, objective and non-intrusive methods and approaches for the automatic assessment of the degree of focus, cognitive load, levels of acute stress and associated negative emotional states.

In this regard, the project aims to provide theoretical and methodological support for the creation of intelligent human-machine interfaces, as well as the corresponding technological assistance. This will be implemented by creating resources and conceptual demonstrators of a new technological framework that will allow the automated assessment of the degree of attention, cognitive load, stress levels, and the detection of negative emotional states. In addition to the collaborative human-robot systems in the industry, on the long term, the results of this project may also support technology developers in various applied areas with social significance, including training, healthcare, high-risk professions etc.

The following fundamental scientific tasks are derived from the formulated objectives:

  • Scientific Task 1 (NZ-1): Testing the validity of the basic hypothesis: “The analysis of selected indicative physiological signals allows for an objective automated assessment of the momentary level of stress, cognitive load, degree of attention, and the automated recognition of certain negative emotional states, the level of emotional arousal, valence, and dominance."

The validation of this hypothesis requires the following major steps:

  • - NZ-1a - study and selection of physiological signals indicative of acute stress, cognitive load, degree of human focus, which can be measured and evaluated reliably;
  • - NZ-1b - creation of non-intrusive methods for objective measurement of stress levels, cognitive load, degree of human focus and detection of negative emotional states, and creation of appropriate methods and resources to allow demonstrators to be used for conceptual validation of methods;
  • - NZ-1c - comparison with a reference method for assessing stress levels, cognitive load, degree of human focus and detection of negative emotional states (questionnaires; chemical analysis of saliva, blood, etc.; expert evaluation by a psychologist).
  • The fundamental nature of the activities related to stages NZ-1a and NZ-1b define the main scientific task solved by the project (NZ-2), and the implementation of stage NZ-1b will be supported by profiled external experts.
  • Scientific Task 2 (NZ-2): Creating new architectures of adaptive human-machine interfaces that, through the integration of functionalities to automatically assess emotional and mental stress levels, cognitive load, attention span, etc., allow the implementation of adaptive communication strategies that improve the effectiveness of human-robot collaborations, increase safety, and protect human health.

For the overall implementation of NZ-1 and NZ-2, it is planned to carry out the following research tasks:

  • - NI-1: Definition of architecture, conceptual solution, methods and related hardware and software for synchronous recording of a large number of physiological signals and indicators. The focus is on creating methods and tools for recording physiological signals that accompany the onset of emotional states and / or correlated with the level of momentary stress, cognitive load, attention span, and more. 
  • - NI-2: Resource creation - a unique physiological signal database (BDFS), complete with informative qualitative and quantitative indicators that take into account the specificity of human-robot interfaces with industry applications, and a general plan contributing to the future development of technology in three broadly applied areas of major social importance (training, healthcare, high-risk professions),
  • - NI-3: Creating, researching, selecting and optimizing methods for efficiently extracting descriptors from biomedical signals and creating statistical models allowing differentiation of the level of cognitive load, degree of attention involvement, detection of negative emotional states and stress responses , (among which: skin surface temperature and conductivity, EEG, ECG, EMG signals).
  • - NI-4: Creating new architectures of intelligent human-machine interfaces that integrate functionalities that allow for automatic assessment of emotional and mental stress levels, cognitive load and attention span, and allow adaptation to a person's current state of accord with the set scenarios.
  • Successful implementation of these research tasks will create opportunities for the implementation of three applied research tasks (NP-1, NP-2, NP-3), aimed at creating the conceptual demonstrators needed for the experimental verification of the intended functionality. These tasks will support the verification of the hypothesis under the NZ-1 in specific areas of application:
  • - NP-1: creation of a demonstrator for the synchronous registration of a large set of heterogeneous physiological signals.
  • - NP-2: Creating a platform bringing together the models, hardware and software needed to develop adaptive human-machine interfaces and technologies to help automatically detect emotional states and determine levels of stress, cognitive load, and attention span.
  • - NP-3: Demonstrators of intelligent human-machine interfaces that implement the developed adaptive strategies.