FR3085221A1 - MULTIMEDIA SYSTEM COMPRISING HUMAN-MACHINE INTERACTION MATERIAL EQUIPMENT AND A COMPUTER - Google Patents

Abstract

The present invention relates to a multimedia method and system comprising hardware equipment for man-machine interaction and a computer executing a program controlling the unfolding of audiovisual sequences, characterized in that it further comprises a means of sound acquisition and / or associated visual, a camera for capturing facial images and a computer program for characterizing the emotional state by automatic analysis of a face identified on said facial images and for configuring the sequence of said audiovisual sequences.

Description

Multimedia system comprising hardware human-machine interaction equipment and a computer

Field of the invention

The present invention relates to the field of multimedia man-machine interaction systems, intended in particular for games, simulation, training or learning applications.

In the field of training and learning, such systems often implement a learning platform (sometimes called “digital learning environment” and in English “learning management system - LMS -”) which manages a process learning or educational path.

This kind of computer system offers a digital workspace comprising evaluation tests which are either subject to validation by the teacher or proposed as regulation activities in self-evaluation.

In the field of video games, a computer application manages the unfolding of the scenarios according to the actions of the player or players for shared games, according to predetermined rules and processes taking into account only the physical interactions of the player (s) with the commands delivering the game application input signals.

Recently, researchers have established the importance of taking into account socio-emotional skills that contribute to the individual and collective effectiveness of lessons. The range of qualities implied is very wide: empathy, respect for others, the ability to seek or lend help, the ability to adjust one's

-2emotions to situations or to discern those of others. The World Health Organization defines them as "a set of capacities allowing everyone to adopt an adaptable and positive behavior to respond effectively to the demands of everyday life".

The invention relates more precisely to the taking into account of emotional contexts in multimedia systems of human-computer interaction.

State of the art

The US patent application US20080254434 describing a learning management system server hosting a software interface exposing functions for processing learning activities associated with conforming learning objects presented in a participating client application having a first software interface adapter;

an intelligent tutorial application for selecting and providing learning objects for presentation to a participant via the participating client application, the intelligent tutorial application comprising a second software interface adapter for sending commands associated with the learning object compliant, the intelligent tutorial application further comprising a search and selection software module, a learning style analysis software module and a tutoring engine software module.

Another document, the international application W02008064431A1A, describes a method and a system for monitoring changes in emotional state in a subject based on facial expressions, based on the capture of first and then second facial image data. of the subject at a first then at a second instant, and the processing of the first and second facial image data to produce state change data

-3 emotional relating to the subject for the period between the capture of the first facial image data and the second facial image data. In some embodiments, a series of stimuli is applied to the subject and emotional state change data correlated with each stimulus is acquired. The method and the system can be used for modeling the relationships between cognitive and emotional responses. The subject's emotional state change models and analysis can be used in a variety of contexts such as decision-making in management, recruitment, evaluation, teaching, and in-depth data exploration.

Also known is the patent application US20140234816 describing an educational system based on a network for providing social and emotional education, comprising:

- a computer storage configured to store a plurality of skills for socio-emotional learning and education scenarios for the plurality of skills;

one or more servers configured to allow one of the training scenarios to be presented on a user device, in which the server or servers are configured to receive input from a child user in response to the presentation of one training scenarios;

wherein the computer storage is configured to store the history of a child user by viewing the education scenarios, the input of the child user during the presentation of the education scenarios, and a skills assessment of the child user in social emotional skills;

and an intelligence module configured to assess the skills of the child user in socio-emotional skills, in part based on the

-4contribution of the child user during the presentation of the education scenarios.

The patent application US20080091515 describes a first aid training method implementing the presentation of a plurality of activatable software agents and comprising a table of levels each associated with a specific behavior, and a main controller controlling the editing of a level parameter controlling the modification of the state of said software agents characterized in that it comprises the recording of a set level and intermittent steps of acquisition of information representative of the emotional state of said rescuer, the modification the level controlled by the controller is decreased if the measured level is higher than the set level, and increased if the measured level is lower than the set level.

Disadvantages of the prior art

The solutions of the prior art do not make it possible to dynamically take into account the emotional situation of the user (player, apprentice or person in training) in order to make the game or training process evolve in real time.

Solution provided by the invention

In order to remedy this drawback, the invention relates, in its most general sense, to a multimedia system comprising hardware equipment for human-machine interaction and a computer executing a program controlling the progress of audiovisual sequences, characterized in that it comprises in addition to an associated sound and / or visual acquisition means, a camera for capturing facial images and a computer program for characterizing the emotional state by automatic analysis

-5a face identified on said facial images and for the configuration of the sequence of said audiovisual sequences.

According to particular embodiments of the invention, the system also has one or more of the following characteristics:

- It also includes a hardware simulation equipment controlled by a computer according to the state of the setting of the sequence of said audiovisual sequences;

- it also includes a configurable interactive game program;

- said human machine interaction equipment consists of a virtual reality or augmented reality headset;

said a computer program for characterizing the emotional state includes means for analyzing the unmasked part of the identified face.

The invention also relates to a multimedia method for the control of a human machine interaction equipment comprising a step of controlling the unfolding of audiovisual sequences and steps of characterizing the emotional state by automatic analysis of a face identified on facial images captured by a camera.

Advantageously, the progress of the audiovisual sequences is determined in real time by the result of the characterization of the emotional state.

Preferably, the difficulty parameters are determined in real time by the result of the characterization of the emotional state.

According to a variant, the activation of digital aids is determined in real time by the result of the characterization of the emotional state.

-6 Detailed description of a nonlimiting example of the invention

The present invention will be better understood on reading the detailed description of a nonlimiting example of the invention which follows, referring to the accompanying drawings in which:

- Figure 1 shows a schematic view of a simulation system for training in cardiopulmonary resuscitation management;

- Figure 2 shows a schematic view of the emotion recognition process.

The invention is described in the following for a nonlimiting example of application to training in cardiopulmonary resuscitation gestures.

In the event of extra-hospital cardiac arrest, cardiopulmonary resuscitation performed by a witness to the event is crucial in terms of survival. Chest compressions and rapid defibrillation are the main determinants of survival after extrahospital cardiac arrest, and several data show that training for the general public improves survival at 30 days and 1 year.

Training the general public in basic resuscitation is effective in improving the number of people who volunteer to perform basic resuscitation in real situations, particularly in high-risk populations (for example, in areas where there is a high risk of cardiac arrest but a low rate of intervention by witnesses.

Training is generally provided with mannequins to put the trainees in a situation to practice Cardiopulmonary Resuscitation.

These mannequins present realistic anatomical landmarks allowing to train the trainees in the practice of ventilation and compression gestures simulated with a

-7 resistance corresponding to an adult or a child to practice and memorize good techniques. Such mannequins can be associated with virtual reality means which make it possible to immerse the trainees in a realistic situation and teach them the appropriate gestures in the face of cardiac arrest by being in a scene where a person is victim of a cardiac arrest.

The system comprises a mannequin (1) having the shape and dimensions of a human torso, a virtual reality mask (2), a camera (3) for acquiring the position of the mannequin and of the hands of the operator. in training and a computer (4) processing the images captured by the camera (3) and generating the video stream displayed by the virtual reality mask (2).

The mannequin (1) is essentially of known type.

It comprises a rigid upper plate (10) having the conformation of a human torso and a rigid lower plate (11). A compression spring (12) is installed between the upper plate (10) and the lower plate (11) to provide the torso with a resistance comparable to that of a human rib cage.

The mannequin also includes an inflatable bag (13) for simulating the resumption of breathing.

It also includes a head (14) having a mouth-shaped orifice (15) for carrying out insufflation simulations.

A sensor (16) connected to the upper plate (10) measures the force applied to the rib cage by means of a strain gauge (18) and provides information concerning the compression force exerted by the operator. , and also used to estimate the risk of a rating fracture.

The position of the hands can also be acquired via location system in reality

-8virtuelle (22, 23), for example the components marketed under the trade name of HTC Vive Tracker 2.0 communicating with a fixed beacon delivering information on the position and the three-dimensional orientation of each of the operator's handles.

Image processing

The computer (4) provides a video stream corresponding to a realistic scene viewed by the operator using a virtual reality headset (2) providing him with an immersive perception in an atmosphere simulating an intervention context.

The computer also calculates the overlay in the ambient scene of a digital mannequin, and recalculates in real time the mark of the synthetic images relative to the position of the real mannequin to replace the mannequin on the synthetic image. numeric in a position corresponding to the position of the dummy (1) relative to the operator.

The camera (3) captures the position of the hands and the computer (4) analyzes the image of the hands to search in a digital image library for an image of a couple of hands conforming to the position of the operator's hands. , and to control the orientation of the digital image of the selected hand torque with respect to the digital mannequin, to put it in correspondence with the position of the hands on the real mannequin (1).

Thus the operator is immersed in a totally immersive atmosphere, and not in an augmented reality atmosphere, and finds on the display screen the combination of synthetic images of the environment, of the synthetic image of the mannequin. digital or the image of a real victim repositioned by taking into account the position of the physical mannequin (1) and the position of the hands by extracting a synthetic image from a database of digital images, and

-9repositioned to correspond to the position of the operator's hands.

The helmet display screen (2) also displays technical indications, for example on the frequency and level of compressions, and on the conformity of the operator's actions. The position of this display area is not fixed relative to the frame of reference of the synthetic images of the environment, but is shifted according to the position of the operator's head and the direction of the head and 1 orientation / direction of the dummy. Thus, he can orient the head to take note of the technical information, or turn his head to move the display area of this information outside the area displayed.

Trainee facial expression acquisition camera

A camera (19) is integrated into the mannequin (1) to capture the image of the operator's face and provide images whose analysis makes it possible to determine his emotional state.

The camera (19) captures the image of the operator and his attitude and provides images which are subject to characterization processing according to a database of facial expressions and a characterization algorithm. as the algorithm described in the publication "Gil Levi and Tal Hassner. Recognition of emotions in nature via convolutional neural networks and mapped binary models. Proc. ACM International Conference on 1 ¹ Multimodal Interaction (ICMI), Seattle, 2015 ”.

This information is used to control the training scenario and the selection of the video sequences displayed on the mask (2).

-10Detection of emotions

The images captured by the camera (19) are subject to a first processing by a face detection module (100). This module is for example based on the method of Viola and Jones of object detection in a digital image, by a supervised learning process allowing to detect faces efficiently and in real time of objects in an image.

The module (100) is for example constituted by a computer application based on rectangular Haar classifiers and the integral representation of an input image, as described in the article by PA Viola, MJ Jones published in International Journal of Computer Vision, vol. 57, no. 2, pp. 137-154, 2004, incorporated by reference.

An example of face detection using a method based on Viola-Jones has been implemented in a free code library available at http; // www.mathworks.com/matlabcentral/fileexchange/19912.

The second step (200) concerns the extraction of facial features from the image area corresponding to the face. An example of treatment is described in the article “Srinivasan R, Golomb JD, Martinez AM. A Neural Basis of Facial Action Recognition in Humans. The Journal of Neuroscience. 2016; 36 (16): 4434-4442. doi: 10.1523 / JNEUROSCI.1704-15.2016. "

A classification processing is then carried out by an expression classification module (300) using a nearest neighbor classifier (NN) and the support vector machine (SVM), for example, as described in the article " Vishnubhotla, S., Support Vector Classification. 2005, ”or the article“ Dasarathy, B. V., Nearest Neighbor Standards (NN): NN Pattern Classification Techniques. 1991 ", incorporated by reference.

-11 A Euclidean-NN rule or the cosine-NN rule can also be used.

The facial region to be modeled is automatically annotated by a collection of landmarks. A shape vector is given by the concatenated coordinates of all landmarks and can be formally written as, s = (xl, x2,. .., xL, yl, y2, ..., yL) T, where L is the number of landmarks.

The shape model can be obtained by applying a principal component analysis (PCA) on all the aligned shapes.

After the creation of an appearance statistical model, an algorithm (400) is used to adapt the statistical model to a new image. This determines the best correspondence of the model to the image making it possible to find the parameters of the model which generate a synthetic image as close as possible to the target image.

This treatment makes it possible to associate the geometric characteristics affected during the expression of emotions.

As examples, when surprised, the eyes and mouth open wide, the latter resulting in an elongated chin. Sadness is expressed by frequent blinking of the eyes. When they are angry, the eyebrows tend to be drawn together.

Each image acquired by the camera (19) is classified in real time by comparing its parameters with a model corresponding to each expression. The models for each class are obtained by an average over a succession of images, or a median, over the vector of shape parameters for each expression.

The classifiers are of type expression 1 / expression 2, i.e. neutral / non-neutral, sad / nontrist, anger / non-anger, disgusted / not disgusted, fear /

-12no-fear, surprised / not-surprised, and happy / not-happy, and of type expression / non-expression.

The result of this processing defines a combination of parameters representative of the emotional state of the trainee 5 or of the player.

These parameters are used by the simulation or game program to control the sequencing of human-machine interactions in real time.

For example, in a simulation application, the neutral emotional parameters will control the selection of more difficult sequences. Emotional parameters of fear will control the selection of less anxiety-provoking sequences.

The goal is to put the trainee or the player in various contextual situations, adapted to improve his mastery of gestures in conditions close to real environments, taking into account his ability to learn under stress.

Claims (1)

Claims Multimedia system comprising hardware equipment for human-machine interaction and a computer executing a program controlling the unfolding of audiovisual sequences, characterized in that it further comprises an associated sound and / or visual acquisition means, a camera for capturing of facial images and a computer program for the characterization of the emotional state by automatic analysis of a face identified on said facial images and for the configuration of the sequence of said audiovisual sequences. Multimedia system according to claim 1 characterized in that it further comprises a hardware simulation equipment controlled by a computer as a function of the state of the setting of the sequence of said audiovisual sequences. Multimedia system according to claim 1 characterized in that it further comprises a configurable interactive game program. Multimedia system according to claim 1 characterized in that said hardware equipment for human-machine interaction consists of a virtual reality or augmented reality headset. Multimedia system according to the preceding claim, characterized in that said a computer program for the characterization of the emotional state comprises means for analyzing the unmasked part of the identified face. -14Multimedia method for the control of material equipment for human-machine interaction comprising a stage for controlling the progress of audiovisual sequences and stages for characterizing the emotional state by automatic analysis of a face identified on captured facial images by a camera. Multimedia method according to claim 6 characterized in that the progress of the audiovisual sequences is determined in real time by the result of the characterization of the emotional state. Multimedia method according to claim 6 characterized in that the difficulty parameters are determined in real time by the result of the characterization of the emotional state. Multimedia method according to claim 6 characterized in that the activation of digital aids is determined in real time by the result of the characterization of the emotional state