KR20200085696A - Method of processing video for determining emotion of a person - Google Patents

Abstract

The present invention relates to an emotion recognition method for processing an image to determine the emotional state of a person. The emotion recognition method for processing an image to determine the emotional state of a person comprises the following steps of: providing an image and a voice representing the appearance of a person, wherein the image includes a first image part, a second image part, and a third image part; processing the first image part to determine the emotional state of the person in the first image part, wherein the face and at least one hand of the person are shown and the at least one hand does not overlap any part of the face of the person in the first image part; and processing the second image part to determine the emotional state of the person, wherein the face and at least one hand of the person are shown and the at least one hand overlaps the face of the person in the second image part.

Description

Emotional recognition method that processes images to determine a person's emotional state{METHOD OF PROCESSING VIDEO FOR DETERMINING EMOTION OF A PERSON}

Embodiments of the present invention relate to a method for recognizing emotions for processing an image to determine a person's emotional state.

In the conventional technique, occlusion is recognized and treated as an error. Covering your mouth with your hands is an important piece of information that can help you determine the intensity of your emotional state. Recognition information may be insufficient due to occlusion problem with a static image.

In addition, when a subject speaks when recognizing emotion with a facial expression, an incorrect emotion recognition result is obtained. Emotion recognition through facial expression recognition is very important information, but when you speak, the mouth shape changes frequently, so you may have a mouth shape such as surprise, anger, or laughter.

As described above, in the prior art, there is almost no alternative to resolve the emotion when only the facial expression is recognized, and in the case of the multi-modal approach, the expression is mixed with the facial expression and the voice information to minimize the error to minimize the error. Doing. In this patent, it is possible to derive accurate emotion recognition results by tracking the face or mouth shape and determining whether it is currently speaking, and in the case of speaking, minimizing mouth shape information and expanding the weight of voice feature information.

Embodiments of the present invention is a multi-modal emotional recognition device, method, and storage medium that performs more accurate emotional recognition using temporal information together with hand movement and identification information, mouth shape information, voice information, and partial expression information. Want to provide

Emotion recognition method for processing an image to determine a person's emotional state according to an aspect of an embodiment of the present invention provides an image and an audio expressing a person's appearance, the image comprising a first image unit and the agent And a second image portion immediately following the first image portion and a third image portion immediately following the second image portion; In the first image section, the first image section is processed to determine the emotional state of the person, and in the first image section, the face of the person and at least one hand are shown, and the at least one hand is the view of the person. Characterized in that no part of the face overlaps; And the second imager processes the second imager to determine the emotional state of the person, wherein the second imager shows the person's face and at least one hand and the at least one hand is the person. It characterized in that it overlaps with the face of the, including, processing the first image, at least one of the first image portion to determine whether the at least one hand covers the face of the person Processing a frame of, finding a first face element of the person in the at least one frame of the first image portion, and in the state where the first face element is located, the at least one of the first image portion Acquiring first facial feature data of the first imaging unit based on the shape of the first facial element shown in the frame of and voice feature based on characteristics of the human voice in the first imaging unit Processing audio data of a first image unit to obtain a; and the emotion of the person with respect to the first image unit based on a plurality of data including first facial feature data and voice features of the first image unit. Including; determining the state, the processing of the second image portion, processing the at least one frame of the second image portion to determine whether the face of the person is covered by at least one hand In particular, it is determined whether or not the at least one hand covers the face of the person in the second image section, and finding the first face element of the person in at least one frame of the second image section. Step, in the state where the first face element is located, obtaining the first facial feature data of the first image portion based on the shape of the first face element shown in the at least one frame of the second image portion Step and audio of the second video unit in order to obtain speech feature data based on the human voice characteristics in the second video unit Processing data, the first facial feature data of the second image portion, the voice characteristic data of the second image portion, and additional data indicating a location where at least one hand covers a part of the face of the person. And determining the emotional state of the person with respect to the second image unit based on a plurality of data.

Further, the step of determining the emotional state of the person with respect to the second image unit based on the plurality of data may include when a part of the person's face is covered by at least one hand in the second image unit, The voice feature data of the second image portion may be weighted more than the first face feature data of the second image portion.

In addition, in the step of determining the emotional state of the person with respect to the second image unit based on the plurality of data, in the first image unit, any part of the person's face was not covered by at least one hand. When the part of the person's face is covered by at least one hand in the second image unit, the voice feature data of the second image portion may be weighted more than the voice feature data of the first image portion. .

Further, the processing of the first image portion may include: finding a second face element of the person in the at least one frame of the first image portion, and while the second face element is located, the first image And acquiring second facial feature data of the first image part based on the shape of the second face element shown in the at least one frame of the negative, in particular, the first face element of the first image part. Determining the emotional state of the person with respect to the first image portion based on a plurality of data including feature data, the second facial element feature data of the first image portion, and the voice feature data of the first image portion, The processing of the second image portion may include determining whether the second face element is covered by at least one hand, which finds the second face element of the person in the at least one frame of the second image portion. And, acquiring second face feature data of the second image part based on predetermined weights of the second face feature and the second face element obscured by the first image part. The at least one having the first facial element feature data of the second image portion, the second facial element feature data of the second image portion, the audio feature data of the second image portion, and a part of the face of the person The emotional state of the person with respect to the second image unit may be determined based on additional data indicating the position of the hand.

In addition, further comprising the step of processing the three image portion to determine the emotional state of the person with respect to the third image portion, wherein at least a part of the face of the person is not covered by my hand, the agent 3 The face of the person and one hand of the equator are shown on the image section, and the processing of the third image section may include the third step of determining whether the at least one hand covers the face of the person. Processing at least one frame of the image unit; Finding a first face element of the person in at least one frame of the third image unit; Acquiring first facial feature data of the third image portion based on a shape of the first facial element shown in the at least one frame of the third image portion while the first face element is located; Processing audio data of the first video unit to obtain audio characteristic data of the third video unit based on the human voice characteristics in the third video unit; And determining the emotional state of the person in the first image unit based on a plurality of data including the first facial feature data and the voice feature data in the third image unit.

When executed by a computer according to another aspect of the embodiment of the present invention, a computer-readable storage medium for storing predetermined instructions provides an image and audio representing a person's appearance, the image comprising a first image unit, And a second image portion immediately following the first image portion and a third image portion immediately following the second image portion; In the first image section, the first image section is processed to determine the emotional state of the person, and in the first image section, the face of the person and at least one hand are shown, and the at least one hand is the view of the person. Characterized in that no part of the face overlaps; And the second imager processes the second imager to determine the emotional state of the person, wherein the second imager shows the person's face and at least one hand and the at least one hand is the person. It characterized in that it overlaps with the face of the, including, processing the first image, at least one of the first image portion to determine whether the at least one hand covers the face of the person Processing a frame of, finding a first face element of the person in the at least one frame of the first image portion, and in the state where the first face element is located, the at least one of the first image portion Acquiring first facial feature data of the first imaging unit based on the shape of the first facial element shown in the frame of and voice feature based on characteristics of the human voice in the first imaging unit Processing audio data of a first image unit to obtain a; and the emotion of the person with respect to the first image unit based on a plurality of data including first facial feature data and voice features of the first image unit. Including; determining the state, the processing of the second image portion, processing the at least one frame of the second image portion to determine whether the face of the person is covered by at least one hand In particular, it is determined whether or not the at least one hand covers the face of the person in the second image section, and finding the first face element of the person in at least one frame of the second image section. Step, in the state where the first face element is located, obtaining the first facial feature data of the first image portion based on the shape of the first face element shown in the at least one frame of the second image portion Step and audio of the second video unit in order to obtain speech feature data based on the human voice characteristics in the second video unit Processing data, the first facial feature data of the second image portion, the voice characteristic data of the second image portion, and additional data indicating a location where at least one hand covers a part of the face of the person. A command for performing an emotion recognition method for processing an image is stored to determine the emotional state of a person, including determining the emotional state of the person for the second image unit based on a plurality of data.

According to the embodiment of the present invention as described above, the emotion recognition method for processing an image to determine the emotional state of a person can accurately grasp the emotional state in the case of conversation and masking the expression by an object such as a hand. have.

1 is a diagram schematically showing the configuration of a multi-modal emotional recognition device according to an embodiment of the present invention.
FIG. 2 is a diagram schematically showing the configuration of a data pre-processing unit among the multi-modal emotion recognition devices of FIG. 1.
FIG. 3 is a diagram schematically showing a configuration of a preliminary reasoning unit in the multi-modal emotion recognition device of FIG. 1.
FIG. 4 is a diagram schematically showing the configuration of the main reasoning unit among the multi-modal emotion recognition devices of FIG. 1.
5 is a flowchart illustrating a multi-modal emotion recognition method using the multi-modal emotion recognition apparatus of FIG. 1.
6 is a flowchart illustrating in detail the data preprocessing step of the multi-modal emotion recognition method of FIG. 5.
7 is a flowchart illustrating in detail the preliminary inference step of the multi-modal emotion recognition method of FIG. 5.
8 is a flowchart illustrating in detail the main reasoning step of the multi-modal emotion recognition method of FIG. 5.
9 is an exemplary diagram illustrating a face recognition process according to whether a situation changes in the multi-modal emotional recognition device of FIG. 1.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily practice.

The present invention can be implemented in many different forms and is not limited to the embodiments described herein. In the drawings, parts not related to the description are omitted in order to clearly describe the present invention, and the same reference numerals are attached to the same or similar elements throughout the specification. In addition, since the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to what is illustrated.

In the present invention, "~ on" means that it is located above or below the target member, and does not necessarily mean that it is located above the gravity direction. Also, in the specification, when a part “includes” a certain component, it means that the component may further include other components, rather than excluding other components, unless otherwise specified.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and the same or corresponding components will be given the same reference numerals when describing with reference to the drawings, and redundant description thereof will be omitted. .

The present invention derives a more accurate emotional recognition result using artificial intelligence considering facial expression, speech state, hand, and voice based on the subject's video and voice data.

1 is a diagram schematically showing the configuration of a multi-modal emotional recognition device according to an embodiment of the present invention.

Referring to FIG. 1, the multi-modal emotion recognition device 10 includes a data input unit 100, a data preprocessing unit 200, a preliminary reasoning unit 300, a main reasoning unit 400, and an output unit 500 can do.

The data input unit 100 may receive image data DV and audio data DS of the user.

The data input unit 100 may include an image input unit 110 that receives image data DV for user's emotional recognition and a voice input unit 120 that receives user's voice data DS.

In addition, the data pre-processing unit 200 generates one or more facial feature data DF ₁ from the voice pre-processing unit 220 that generates the voice feature data DF ₂ from the voice data DS, and the image data DV. An image pre-processing unit 210 may be included.

At this time, the facial feature data DF ₁ may include at least one of an image, location information, size information, face ratio information, and depth information, and the voice feature data DF ₂ may be accented or pitched. It may include information that can represent characteristics of the voice, such as information, speech intensity, and speech rate.

The image pre-processing unit 210 performs image pre-processing for extracting the facial feature data DF ₁ of the user from the image data DV.

The image pre-processing may convert image data DV for use in a learning model such as full or partial face recognition, noise removal, user face feature and image extraction, to an appropriate mode.

The voice pre-processing unit 220 performs voice pre-processing to extract the user's voice feature data DF ₂ from the voice data DS.

The voice pre-processing may convert voice data DS into an appropriate mode for using a learning model such as external noise removal, noise removal, and user voice feature extraction.

The preliminary reasoning unit 300 may generate context determination data P regarding whether a user's context changes according to temporal order based on the image data DV.

At this time, the situation determination data P is a recognition target region different from the tracking target region B which is a part of the entire image region of the dialogue judgment data P ₁ or whether the user is in a conversation state or the image data DV. A) may include overlapping determination data P ₂ as to whether or not it overlaps.

In detail, the preliminary inference unit 300 generates location inference data DM ₁ for inferring the location of the tracking target area B based on the image data DV, and facial feature data DF ₁ and Based on the location inference data DM ₁ , it is possible to generate overlap determination data P ₂ about whether the tracking target area B and the recognition target area A overlap.

Also, the preliminary reasoning unit 300 may generate conversation determination data P ₁ that determines whether the user is in a conversation state based on the facial feature data DF ₁ .

The main reasoning unit 400 generates at least one sub-feature map FM based on the voice feature data DF ₂ or the face feature data DF ₁ , and the sub feature map FM and situation determination data ( Based on P), the user's emotional state can be inferred.

The emotional state may include user's emotional state information such as happiness, anger, fear, disgust, sadness, and surprise.

The output unit 500 may output the result of the emotional state inferred by the main reasoning unit 400.

At this time, the output unit 500 can output in various forms using activation functions such as sigmoid functions, step functions, softmax functions, and rectified linear units (ReLUs). have.

FIG. 2 is a diagram schematically showing the configuration of a data pre-processing unit among the multi-modal emotion recognition devices of FIG. 1.

Referring to FIG. 2, the data preprocessing unit 200 may include an image preprocessing unit 210 and an audio preprocessing unit 220.

The image pre-processing unit 210 may include a face detector 211, an image pre-processing module 212, a landmark detection module 213, a position adjustment module 214, and a face element extraction module 215.

The face detector 211 may detect a recognition target area A, which is an area corresponding to a user's face, in the entire area of the image data DV.

The image pre-processing module 212 may correct the recognition target area A.

Specifically, the image pre-processing module 212 may perform brightness of an image, correction of a blur, and noise removal of image data DV.

The landmark detection module 213 may extract face element location information AL of the recognition target area A.

In detail, it is possible to perform face recognition by identifying location information of important elements of the face, such as the face, eyes, mouth, nose, and forehead, among the recognition target regions A.

The position adjustment module 214 may adjust the position based on the face element position information AL of the recognition target area A.

In detail, the position adjustment module 214 may align the image to fit horizontally or vertically based on the face element location information AL extracted from the landmark detection module 213.

The face element extraction module 215 is located in the recognition target area A and sets a sub recognition target area AA smaller than the recognition target area A, and facial feature data DF ₁ of the sub recognition target area AA ).

The sub-recognition target area AA may be a plurality of areas or one area in which at least one or more facial elements such as a face, eyes, mouth, nose, and forehead are determined.

For example, when the eye, nose, and mouth from which the facial element location information AL is extracted among the recognition target areas A, the facial element extraction module 215 is an eye recognition area that is a sub recognition target area AA. (A ₁ ), a nose recognition area A ₂ , and a mouth recognition area A ₃ may be set and at least one or more facial feature data DF ₁ may be generated for the set sub-recognition target area AA.

Also, the face element extraction module 215 may generate face feature data DF ₁ based on the recognition target area A when the sub-recognition target area AA is not set.

The voice pre-processing unit 220 may include a voice correction module 221 and a voice feature data extraction module 222.

The voice correction module 221 may correct the voice data DS.

In detail, the voice correction module 221 may perform various correction methods such as noise and external noise removal, volume control, and frequency correction included in the voice data DS to generate corrected voice data.

The voice feature data extraction module 222 may extract features of the voice data DS that have passed through the voice correction module 221 to generate voice feature data DF ₂ .

In detail, the voice feature data extraction module 222 may be used by one or more modules of voice data, frequency, and spectrum analysis modules such as Melc-frequency Cepstral Coefficients (MFCC), Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), Logbank, etc. The voice feature data DF ₂ can be generated.

At this time, the voice feature data extraction module 222 may use the corrected voice data or may use voice data DS.

3 is a diagram schematically showing a configuration of a preliminary reasoning unit in the multi-modal emotion recognition device of FIG. 1.

Referring to FIG. 3, the preliminary reasoning unit 300 may include a hand detection reasoning module 310, a dialogue state reasoning module 320, and a face overlap inspection module 330.

The dialogue state inference module 320 may use the first learning model LM ₁ and generate dialogue determination data P ₁ based on the facial feature data DF ₁ .

In detail, the dialogue state inference module 320 uses the first learning model LM ₁ that can determine whether the user is in a conversation state, using all or part of the facial feature data DF ₁ of the user. , It is possible to generate conversation judgment data P ₁ which is a conversation judgment.

The facial feature data DF ₁ includes mouth image data DV _{2 that} is image data DV for a portion corresponding to the user's mouth in the recognition target area A, and the first learning model LM ₁ By using, it is possible to generate conversation judgment data P ₁ about whether the user is in a conversation state from the input image data DV ₂ .

The first learning model (LM ₁ ) is an artificial object that can infer temporal or spatial features such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNNs), Deep Neural Networks (DNN), and Convolutional Neural Network (CNN). It may be at least one of intelligent model, machine learning, and deep learning methods.

The hand detection inference module 310 detects the hand image data DV ₁ for the tracking target region B from the image data DV, and uses the second learning model LM ₂ to generate the hand image data DV ₁₎ may generate a location inference data (DM ₁₎ is based on.

At this time, the second learning model LM ₂ may infer temporal features or spatial features such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNNs), Deep Neural Networks (DNN), and Convolutional Neural Network (CNN). It is at least one of artificial intelligence models, machine learning, and deep learning methods that can generate position inference data (DM ₁ ) for the hand.

In addition, the hand detection inference module 310 generates a location inference feature map FM ₁ for the location inference data DM ₁ , a sub feature map FM, situation determination data P, and a location inference feature. Based on the map FM ₁ , a user's emotional state may be inferred.

In this case, the location inference feature map FM ₁ may include meaningful information about the movement of the hand, such as feature information about the hand, that is, gesture information about the hand and location information about the hand.

The face overlap inspection module 330 determines whether the recognition target area A and the tracking target area B overlap based on the facial feature data DF ₁ and the location inference data DM ₁ , and determines whether or not the overlap occurs. Depending on the result, it is possible to generate the overlap determination data P ₂ .

Specifically, the overlap determination data P ₂ determines whether the recognition target area A and the tracking target area B overlap, and the corresponding facial feature data DF ₁ of the recognition target area A and voice One or more parameters that determine the importance and use of the feature data DF ₂ may be generated.

FIG. 4 is a diagram schematically showing the configuration of the main reasoning unit among the multi-modal emotion recognition devices of FIG. 1.

Referring to FIG. 4, the main reasoning unit 400 includes a plurality of sub-feature map generation units 410 (411, 412, 413, 414), a multi-modal feature map generation unit 420, and an emotion recognition reasoning unit 430. It may include.

The plurality of sub feature map generators 410 (411, 412, 413, and 414) use the third learning model LM ₃ to speech features based on the voice feature data DF ₂ and the face feature data DF ₁ . A plurality of sub feature maps FM for the data DF ₂ and the facial feature data DF ₁ may be generated.

In detail, the third learning model LM ₃ is at least one of an artificial intelligence model, a machine learning, and a deep learning method capable of inferring at least one spatial feature such as Deep Neural Networks (DNN), Convolutional Neural Network (CNN), and the like. One or more methods, and using the third learning model LM ₃ , generate a plurality of sub-feature maps FM having features of the voice feature data DF ₂ and the face feature data DF ₁ . Can.

The multi-modal feature map generator 420 may generate the multi-modal feature map M from a plurality of sub-feature maps FM with reference to the situation determination data P.

The situation determination data P has a predetermined situation determination value PV according to a user's situation, and the multi-modal feature map generation unit 420 includes at least one situation determination value among the plurality of sub-feature maps FM. (PV) may be applied to generate a multi-modal feature map (M).

In detail, the situation determination value PV may be a parameter indicating the importance and use of each sub feature map FM.

Through the operation of the situation determination data P and the sub feature map FM, a sub feature map FM to which the situation determination value PV of the situation determination data P is applied is generated, and a plurality of sub feature maps FM ) To create a multi-modal feature map (M).

For example, when the user's eyes are covered, the situation judgment value for the eye is output as 0, and 0 is output through the multiplication operation of the situation judgment value for the eye and the sub feature map FM for the eye. Thus, the main reasoning unit 400 may generate a multi-modal feature map M based on other sub feature maps except the sub feature map for the eye.

In addition, the location inference feature map FM ₁ is generated from the hand detection inference module 320, and the user's emotion is based on the sub feature map FM, the situation determination data P, and the location inference feature map FM ₁ . A multi-modal feature map (M) for inferring the state can be generated.

The multi-modal feature map (M) may be generated by merging at least one sub-feature map (FM) and location inference feature map (FM ₁ ) using Concat, Merge, and Deep Network.

The emotional recognition reasoning unit 430 may infer the emotional state based on the multi-modal feature map M using the fourth learning model LM ₄ .

At this time, the fourth learning model LM ₄ may be a temporal learning model, such as a cyclic neural network, such as a Long Short-Term Memory (LSTM), a Recurrent Neural Network (RNN), or a Gated Recurrent Unit (GRU). It may be at least one of artificial intelligence models, machine learning, and deep learning methods capable of inferring or analyzing spatial features.

5 is a flowchart illustrating a multi-modal emotion recognition method using the multi-modal emotion recognition apparatus of FIG. 1.

Referring to FIG. 5, a data input step (S100) of receiving a user's image data DV and audio data DS is performed.

Next, a data pre-processing step including a voice pre-processing step of generating voice feature data DF ₂ from the voice data DS and an image pre-processing step of generating one or more facial feature data DF ₁ from the image data DV. (S200) may be performed.

At this time, the data pre-processing step S200 may generate facial feature data DF ₁ and voice feature data DF ₂ for using the learning model.

The learning model may be artificial intelligence, machine learning and deep learning methods.

Then, based on the image data DV, a preliminary inference step S300 of generating context determination data P regarding whether the user's context changes according to temporal order may be performed.

At this time, the temporal order may be whether or not the conversation state, it may be data for identifying the characteristics of the movement of the body part.

In addition, the situation determination data P determines whether or not overlapping and a conversation state are obtained from the video data DV, and indicates the importance or use of one or more facial feature data DF ₁ or voice feature data DF ₂ . It may include parameters.

Also, feature information on a user's body part other than the one or more facial feature data DF ₁ generated in the data pre-processing step S200 may be extracted and generated.

Then, at least one sub-feature map FM is generated based on the voice feature data DF ₂ or the face feature data DF ₁ , and based on the sub-feature map FM and the situation determination data P The main reasoning step S400 of inferring the user's emotional state may be performed.

At this time, the sub-feature map (FM) including feature information extracted from the user and the situation determination data (P) including the parameter of importance or use of feature information are calculated, and the importance or use of the sub-feature map (FM) is calculated. Including information about whether or not, it is possible to infer the user's emotional state.

Then, a result derivation step S500 of outputting an inference result of the emotional state in the main reasoning step S400 is performed.

6 is a flowchart illustrating in detail the data preprocessing step of the multi-modal emotion recognition method of FIG. 5.

Referring to FIG. 6, the data pre-processing step (S200) includes an image pre-processing step (S210) and an audio pre-processing step (S220 ).

In the image pre-processing step S210, a face detection step is performed in which a recognition target image region and a recognition target region A are regions corresponding to a user's face in all regions of the image data DV.

Then, an image pre-processing step of correcting the recognition target area A is performed.

In detail, in the image pre-processing step, brightness of an image, correction of a blur, and noise removal of image data DV may be performed.

Then, a landmark detection step of extracting face element location information AL of the recognition target area A is performed.

In detail, face recognition may be performed by identifying location information of important elements of the face, such as the face, eyes, nose, mouth, and forehead, among the recognition target regions A.

Then, a position adjustment step of adjusting the position based on the face element position information AL of the recognition target area A may be performed.

In detail, the image may be aligned with horizontal or vertical based on the face element location information AL extracted from the landmark detection module 213.

Next, in the recognition target area A, a sub-recognition target area AA located in the recognition target area A and smaller than the recognition target area A is set based on the face element position information AL, and the sub-recognition A face element extraction step of generating face feature data DF ₁ of the target area AA may be performed.

In this case, the sub-recognition target area AA may be a plurality of areas or one area in which at least one or more facial elements such as the entire face, eyes, mouth, nose, and forehead are determined.

For example, when the eye, nose, and mouth from which the facial element location information AL is extracted among the recognition target areas A, the facial element extraction module 215 is an eye recognition area that is a sub recognition target area AA. (A ₁ ), a nose recognition area A ₂ , and a mouth recognition area A ₃ may be set and at least one or more facial feature data DF ₁ may be generated for the set sub-recognition target area AA.

In addition, in the face element extraction step, when the sub-recognition target area AA is not set, the facial feature data DF ₁ may be generated based on the recognition target area A.

The voice pre-processing step S220 includes a voice correction step and a voice feature data extraction step.

First, the voice correction step of correcting the voice data DS is performed.

Specifically, in the voice correction step, various correction methods such as noise and external noise removal, volume control, and frequency correction included in the voice data DS may be performed to generate corrected voice data.

The voice feature data extraction step of extracting features of the voice data DS that has undergone the voice correction step and generating voice feature data DF ₂ is performed.

Specifically, voice data such as Mel-frequency cepstral coefficients (MFCC), Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), Logbank, etc., generate voice characteristic data (DF ₂ ) of the user through one or more modules of a frequency and spectrum analysis module Can be.

At this time, the voice feature data extraction step may use the corrected voice data or the voice feature DS may be performed to generate voice feature data DF ₂ without performing the voice correction step.

In addition, this is an example, and at least some of the steps may be performed simultaneously with the steps before and after, or may be performed by changing the order.

7 is a flowchart illustrating in detail the preliminary inference step of the multi-modal emotion recognition method of FIG. 5.

A conversation state inference step S310 of using the first learning model LM ₁ and generating conversation determination data P ₁ based on the facial feature data DF ₁ may be performed.

In the conversation state inference step S310, the first learning model LM ₁ is used to detect whether there is a conversation state in the previous situation and to detect features and movements of the facial elements from the facial feature data DF ₁ . Can be.

In detail, using all or part of the facial feature data DF ₁ of the user, the conversation determination data P ₁ , which is a conversation determination, is determined using the first learning model LM ₁ whether the user is in conversation. Can be generated.

In this case, the facial feature data DF ₁ may include mouth image data DV ₂ for a portion of the recognition target area A corresponding to the user's mouth.

In addition, using the first learning model LM ₁ , it is possible to generate conversation determination data P ₁ about whether the user is in a conversation state from the input image data DV ₂ .

Then, from image data (DV) detects the hand image data (DV ₁₎ to the tracking target area (B), and the second using a learning model (LM ₂₎ based on the hand image data (DV ₁₎ location inferences A hand detection inference step S320 of generating data DM ₁ is performed.

At this time, temporal inference with the previous situation about the position of the hand may be possible using the second learning model LM ₂ . For example, it may be determined whether the hand is temporarily overlapping the face.

In addition, the hand detection inference step S320 generates a location inference feature map FM ₁ for the location inference data DM ₁ , a sub feature map FM, a situation determination data P, and a location inference feature. Based on the map FM ₁ , a user's emotional state may be inferred.

In detail, the location inference feature map FM ₁ may include meaningful information about the movement of the hand, such as information on a feature for recognizing a gesture for the hand and location for the hand.

Next, based on the facial feature data DF ₁ and the location inference data DM ₁ , it is determined whether the recognition target region A and the tracking target region B overlap, and the overlap determination data is determined according to the overlapping determination result. A face overlap inspection step (S330) of generating (P ₂ ) is performed.

Specifically, the overlap determination data P ₂ determines whether the recognition target area A and the tracking target area B overlap, and the corresponding facial feature data DF ₁ of the recognition target area A and voice It may include one or more parameters that determine the importance and use of the feature data DF ₂ .

8 is a flowchart illustrating in detail the main reasoning step of the multi-modal emotion recognition method of FIG. 5.

Referring to FIG. 8, the main reasoning step (S400) includes a plurality of sub-feature map generation steps (S410 ), a multi-modal feature map generation step (S420 ), and emotion recognition reasoning (S430 ).

First, the third learning model (LM ₃₎ the use of the voice characteristic data (DF ₂₎ and face characteristic data on the basis of (DF ₁₎ of the plurality of the voice characteristic data (DF ₂₎ and the face characteristic data (DF ₁₎ A plurality of sub feature map generation steps (S410) of generating a sub feature map (FM) are performed.

Next, the third learning model LM ₃ refers to the context determination data P, and generates a multi-modal feature map (S420) for generating a multi-modal feature map (M) from a plurality of sub feature maps (FM). Is performed.

At this time, the situation determination data P has a predetermined situation determination value PV according to the user's situation, and the multi-modal feature map generation step S420 is performed on at least one of the plurality of sub-feature maps FM. A multi-modal feature map M may be included by applying a situation determination value PV.

In addition, in the multi-modal feature map generation step (S420 ), the location inference feature map FM ₁ is generated from the hand detection inference module 320, the sub feature map FM, the situation determination data P, and the location inference feature. A multi-modal feature map M that infers the user's emotional state based on the map FM ₁ may be generated.

Then, using the fourth learning model LM ₄ , the emotion recognition inference step S430 of inferring the emotional state based on the multi-modal feature map M is performed.

At this time, the fourth learning model LM ₄ may be a temporal learning model, such as a cyclic neural network, such as a Long Short-Term Memory (LSTM), a Recurrent Neural Network (RNN), or a Gated Recurrent Unit (GRU). It may be at least one of artificial intelligence models, machine learning, and deep learning methods capable of inferring or analyzing spatial features.

9 is an exemplary diagram illustrating a face recognition process according to whether a situation changes in the multi-modal emotional recognition device of FIG. 1.

Referring to FIG. 9, ((A)), the user puts his hand on his face and the hand does not cover his mouth and nose.

The user's image data DV is input through the image input unit 110, and the user's voice data DS is input through the audio input unit 120.

Subsequently, the image preprocessing unit 210 generates facial feature data DF _{1 that} has been preprocessed, and also generates voice feature data DF _{2 that} has been preprocessed through the voice preprocessing unit 220, The image preprocessing unit 210 may recognize the eye recognition area A ₁ , the nose recognition area A ₂ , and the mouth recognition area A ₃ based on the recognizable user's facial element location information AL of the eyes, nose, and mouth. The recognition target area A is set, and the recognition target area A is transmitted to the preliminary reasoning unit 300.

Thereafter, the preliminary reasoning unit 300 generates hand image data DV ₁ for the tracking target region B ₁ detected from the image data DV.

At this time, the preliminary reasoning unit 300 generates position inference data DM ₁ that identifies the movement of the hand through the hand image data DV ₁ , and the tracking target area B based on the position inference data DM _{1 1} ) and the overlap determination data P ₂ are generated based on the determination whether the recognition target region A overlaps.

Here, the overlap determination data P ₂ may include parameters indicating the use of the eye recognition area A ₁ , the nose recognition area A ₂ , and the mouth recognition area A ₃ .

In addition, the conversation state inference module 310 determines whether the conversation state is through the mouth recognition area A ₃ based on the mouth image data DV ₂ and generates the conversation determination data P ₁ .

Subsequently, the sub-feature map generation unit 410 generates a plurality of sub-feature maps FM using the facial feature data DF ₁ corresponding to the eyes, nose, and mouth using the third learning model LM ₃ . .

Thereafter, the multi-modal feature map generation unit 420 generates a multi-modal feature map (M) by integrating a plurality of sub-feature maps (FM) and a location inference feature map (FM ₁ ) corresponding to the hand.

Thereafter, through the fourth learning model LM ₄ , emotion recognition may be inferred in consideration of the behavior of the previous user, and this may be expressed as a result of emotion recognition.

(Step B) Step B represents the continuous operation of Step A.

For example, step B may be assumed to be an image continuously taken after step A at a rate of 30 FPS.

As in step A, the user's image data DV is input through the image input unit 110 and the user's voice data DS is input through the audio input unit 120.

Thereafter, the voice pre-processing unit 220 generates voice pre-processed voice feature data DF ₂ , and the image pre-processor 210 generates face feature data DF ₁ and face element location information AL. Then, based on the face element location information AL, the recognition target area A including the eye recognition area A ₁ , the nose recognition area A ₂ , and the mouth recognition area A ₃ is set, and the recognition target The area A is transmitted to the preliminary reasoning unit 300.

At this time, the size of the recognition target area A may change according to a user's motion.

In step B, the size of the recognition target area A is changed according to the operation in comparison with the step A.

Thereafter, the preliminary reasoning unit 300 may generate position inference data DM ₁ based on the hand image data DV ₁ to track the hand movement from step A to step B.

The preliminary reasoning unit 300 generates superposition determination data P ₂ based on the determination of whether the tracking target region B ₂ and the recognition target region A overlap based on the location inference data DM ₁ .

In addition, the preliminary reasoning unit 300 determines whether or not the conversation state, and generates conversation judgment data P ₁ .

At this time, the preliminary reasoning unit 300 uses the first learning model LM ₁ to determine whether or not the conversation state is considered in consideration of whether the user who is the emotional recognition target continues in the previous situation including step (A). Can judge.

For example, if it is inferred that the user is not in a conversational state in step A, based on the result, the user's mouth shape temporarily changes based on the mouth recognition area A ₃ in step B. Even though similar to, the preliminary reasoning unit 300 may determine that the user is not in a conversation state by using the first learning model LM ₁ . That is, the preliminary reasoning unit 300 may perform inference on whether to determine the conversation state in the next scene, step B, based on the result of the conversation state determination in step A.

Thereafter, the main reasoning unit 400 generates a plurality of sub-feature maps FM using the received facial feature data DF ₁ and the voice feature data DF ₂ using the third learning model LM ₃ and , A multi-modal feature map (M) is generated by integrating a plurality of sub feature maps (FM) and a location inference feature map (FM ₁ ) corresponding to the hand.

Thereafter, the main reasoning unit 400 may infer emotion recognition in consideration of the user's behavior of the previous (step (A)) through the fourth learning model LM ₄ , and may express this as a result of emotion recognition.

(Step (C)) After step B, the user shows the behavior of covering his mouth with his hand.

The image preprocessing unit 210 sets the recognition target area A including the eye recognition area A ₁ based on the recognizable user's eye facial element location information AL, and sets the recognition target area A. It transmits to the preliminary reasoning unit 300.

Thereafter, the preliminary inference unit 300 generates hand image data DV ₁ for the tracking target area B ₃ detected from the image data DV. At this time, the position inference data DM ₁ that identifies the movement of the hand is generated through the hand image data DV ₁ , and the tracking target area B ₃ and the recognition target area (based on the position inference data DM ₁ ) Based on the determination of the overlap of A), the overlap determination data P ₂ is generated.

Here, the overlap determination data P ₂ may include a parameter indicating whether to use the facial feature data DF ₁ based on the eye recognition area A ₁ or the weight applied to the facial feature data DF ₁ .

In addition, the preliminary reasoning unit 300 is the area for the nose recognition area A ₂ or the mouth recognition area A ₃ and the user's hand position that were the recognition target area A in steps (A) and (B). By recognizing the overlap with the tracking target area B ₃ , a parameter indicating that it is excluded from the emotion recognition inference or has a low importance may be included in the overlap determination data P ₂ .

In addition, the preliminary reasoning unit 300 considers the situation in which the mouth image data DV ₂ corresponding to the mouth recognition area A ₃ is not recognized and the result of the determination of whether the user is in a previous conversation state, the voice feature data DF A value indicating whether to use ₂ ) may be included in the conversation determination data P ₁ .

Here, the determination result of the previous conversation state is inferred through the temporal learning model. In this case, the temporal learning model may be a temporal learning model such as a cyclic neural network such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNNs), or Gated Recurrent Unit (GRU).

Subsequently, the sub-feature map generation unit 410 generates a plurality of sub-feature maps FM using the facial feature data DF ₁ of the region corresponding to the eye using the third learning model LM ₃ .

Thereafter, the multi-modal feature map generation unit 420 generates a multi-modal feature map (M) by integrating a plurality of sub-feature maps (FM) and a location inference feature map (FM ₁ ) corresponding to the hand.

Thereafter, the emotion recognition reasoning unit 430 may infer emotion recognition in consideration of the behavior of the previous user through the fourth learning model LM ₄ , and may express this as a result of emotion recognition.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

The system or device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the systems, devices, and components described in the embodiments include, for example, processors, controllers, Arithmetic Logic Units (ALUs), digital signal processors (micro signal processors), microcomputers, and Field Programmable Arrays (FPAs). , A Programmable Logic Unit (PLU), microprocessor, or any other device capable of executing and responding to instructions (Instruction), may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configures the processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and constructed for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, magnetic media such as optical media such as CD-ROMs and DVDs, and floppy disks. Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes made by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

10: multi-modal emotion recognition device
100: data input unit 110: video input unit
120: voice input unit
200: data pre-processing unit 210: image pre-processing unit
211: face detection module 212: image pre-processing module
213: landmark detection module 214: position adjustment module
215: face element extraction module
220: voice pre-processing unit 221: voice correction module
222: voice feature data extraction module 300: preliminary reasoning unit
310: dialogue state inference module
320: hand detection reasoning module 330: face overlap inspection module
400: main reasoning
411: first sub-feature map generator 412: second sub-feature map generator
413: third sub-feature map generator 414: n-th sub-feature map generator
420: Multi-modal feature map generation unit 430: emotional recognition reasoning unit
500: output
S100: Data input step S200: Data pre-processing step
S210: image pre-processing step S220: audio pre-processing step
S300: Preliminary reasoning step S310: Dialogue state reasoning step
S320: Hand detection inference step S330: Face overlap inspection step
S400: Main reasoning step S410: Sub feature map generation step
S420: Multi-modal feature map generation step S430: Emotion recognition inference step
S500: Steps to derive results A ₁ : Eye recognition area
A ₂ : nose recognition area A ₃ : mouth recognition area
B _1, B _2, B ₃ : Area to be tracked

Claims (6)

In the emotional recognition method of processing an image to determine the emotional state of a person,
The image, which provides an image and audio expressing a human appearance, includes a first image portion, a second image portion immediately following the first image portion, and a third image portion immediately following the second image portion, step;
In the first image section, the first image section is processed to determine the emotional state of the person, and in the first image section, the face of the person and at least one hand are shown, and the at least one hand is the view of the person. Characterized in that no part of the face overlaps; And
The second image section processes the second image section to determine the emotional state of the person, and in the second image section, the face of the person and at least one hand are shown and the at least one hand of the person Characterized in that it overlaps with the face; includes,
The step of processing the first image unit,
Processing at least one frame of the first image unit to determine whether the at least one hand covers the face of the person;
Finding a first face element of the person in the at least one frame of the first image unit;
Acquiring first facial feature data of the first image portion based on a shape of the first facial element shown in the at least one frame of the first image portion while the first facial element is located;
Processing audio data of the first video unit to obtain a voice feature based on characteristics of the human voice in the first video unit; and,
And determining the emotional state of the person with respect to the first image unit based on a plurality of data including first facial feature data and voice features of the first image unit.
The processing of the second image unit may include:
In order to determine whether the face of the person is obscured by at least one hand, the at least one hand covers the face of the person in the second image section, in particular, processing the at least one frame of the second image section. The step of determining whether or not to cover, and
Finding the first face element of the person in at least one frame of the second image portion;
Acquiring first facial feature data of the first image portion based on a shape of the first facial element shown in the at least one frame of the second image portion while the first facial element is located;
Processing the audio data of the second video unit to obtain voice characteristic data based on the human voice characteristics in the second video unit;
Based on a plurality of data including the first facial feature data of the second image portion, the audio feature data of the second image portion, and additional data indicating a location where at least one hand covers a part of the face of the person. And determining a person's emotional state with respect to the second image unit.
According to claim 1,
Determining the emotional state of the person with respect to the second image unit based on the plurality of data,
When a part of the person's face is covered by at least one hand in the second image unit, a person who places more weight on the voice characteristic data of the second image unit than the first face feature data of the second image unit Emotion recognition method that processes an image to determine its emotional state.
According to claim 1,
Determining the emotional state of the person with respect to the second image unit based on the plurality of data,
In the first image portion, if any part of the face of the person is not covered by at least one hand, in the second image portion, when a part of the face of the person is covered by at least one hand, the first Emotion recognition method for processing an image to determine the emotional state of a person who places more weight on the audio feature data of the second image portion than the audio feature data of the image portion.
According to claim 1,
The step of processing the first image unit,
Finding a second face element of the person in the at least one frame of the first image portion,
And acquiring second facial feature data of the first image portion based on the shape of the second facial element shown in the at least one frame of the first image portion while the second facial element is located. Including,
In particular, the first image is based on a plurality of data including the first facial element feature data of the first image portion, the second facial element feature data of the first image portion, and audio feature data of the first image portion. Determine the person's emotional state of wealth,
The processing of the second image unit may include:
Determining whether the second face element is obscured by at least one hand, finding the second face element of the person in the at least one frame of the second image portion;
The method further includes acquiring second face feature data of the second image part based on a predetermined weight for the obscuration of the second face feature and the second face element of the first image part,
In particular, the at least the first facial element feature data of the second image portion, the second facial element feature data of the second image portion, the audio feature data of the second image portion, and the part of the face of the person Emotion recognition method for processing an image to determine the emotional state of the person, characterized in that for determining the emotional state of the person with respect to the second image unit based on the additional data indicating the position of one hand.
According to claim 1,
And processing the third image section to determine the emotional state of the person with respect to the third image section, and wherein at least no part of the person's face is covered by my hand, the third image On the wound, the person's face and the equator's one hand are visible,
The processing of the third image unit may include:
Processing at least one frame of the third image unit to determine whether the at least one hand covers the face of the person;
Finding a first face element of the person in at least one frame of the third image unit;
Acquiring first facial feature data of the third image portion based on a shape of the first facial element shown in the at least one frame of the third image portion while the first face element is located;
Processing audio data of the first video unit to obtain audio characteristic data of the third video unit based on the human voice characteristics in the third video unit; And
Determining an emotional state of the person in the first image unit based on a plurality of data including the first facial feature data and the audio feature data in the third image unit; Emotion recognition method to process video.
A computer readable storage medium storing instructions that, when executed by a computer, perform the method of claim 1.

Images