KR102564570B1 - System and method for analyzing multimodal emotion - Google Patents

Abstract

The present invention relates to a multimodal emotion analysis system and method, and the multimodal emotion analysis system according to the present invention includes a video input unit for receiving a user video; an image emotion analysis unit that analyzes emotion of the user's face image extracted from the user's video; a voice emotion analyzer for analyzing the emotion of the user's voice extracted from the user's video; a text emotion analyzer for analyzing the emotion of the text extracted from the user's video; and an ensemble emotion analysis unit that analyzes a combination of the image emotion characteristics calculated by the image emotion analysis unit, the voice emotion characteristics calculated by the voice emotion analysis unit, and the text emotion characteristics.
Accordingly, it is possible to accurately analyze emotions in consideration of emotions of face images, voices, and texts.

Description

Multimodal emotion analysis system and method {System and method for analyzing multimodal emotion}

The present invention relates to a multimodal emotion analysis system and method, and more particularly, to a multimodal emotion analysis system and method capable of accurately analyzing emotions in consideration of emotions of facial images, voices, and texts.

Longevity is a human wish, but disease, poverty, and loneliness are also aspects of longevity.

In order to solve these problems, the process of identifying the problem should be prioritized. Disease and poverty could be identified relatively easily through a quantitative method, but it was not easy to identify problems related to emotions such as loneliness in a quantitative way.

However, in recent years, many methods have been developed to understand human emotions and emotions through artificial intelligence.

Humans express emotions through facial expressions, actions, and words, and therefore, when artificial intelligence identifies human emotions, facial expressions, actions, and words are analyzed.

However, since a person explicitly expresses conflicting emotions in each of facial expressions, actions, and words, the accuracy of emotion identification may be reduced if only one means of expressing emotions is analyzed.

KR 10-2018-0057970 A

Accordingly, an object of the present invention is to solve such conventional problems, and to provide a multimodal emotion analysis system and method capable of accurately analyzing emotions in consideration of emotions of face images, voices, and texts.

The problem to be solved by the present invention is not limited to the above-mentioned problem, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

The above object, according to the present invention, a video input unit for receiving a user video input; an image emotion analysis unit that analyzes emotion of the user's face image extracted from the user's video; a voice emotion analyzer for analyzing the emotion of the user's voice extracted from the user's video; a text emotion analyzer for analyzing the emotion of the text extracted from the user's video; and an ensemble emotion analysis unit analyzing a combination of the image emotion characteristics calculated by the image emotion analysis unit, the voice emotion characteristics calculated by the voice emotion analysis unit, and the text emotion characteristics calculated by the text emotion analysis unit. It is achieved by a multimodal emotion analysis system comprising a.

The image emotion analysis unit includes a plurality of sub-image emotion analysis units to which different image analysis models are applied and an image emotion feature combining unit combining sub-image emotion characteristics calculated in each of the sub-image emotion analysis units, and the voice emotion analysis unit The unit includes a plurality of sub-voice emotion analysis units to which different voice analysis models are applied and a voice emotion feature combining unit for combining the sub-voice emotion characteristics calculated in each of the sub-voice emotion analysis units. A plurality of sub-text sentiment analysis units to which the text analysis model is applied and a text sentiment feature combining unit combining the sub-text sentiment characteristics calculated by each of the sub-text sentiment analysis units may be provided.

The image analysis model may be VGG, ResNet, and Inception.

The voice analysis model may be an SVM or a combination model of CNN and RNN.

In the multimodal emotion analysis system according to the present invention, the image emotion modeling unit stores the characteristics of each emotion of the image and provides the analysis standard in the image emotion analysis unit, and the voice emotion analysis unit stores the characteristics of each emotion of the voice. It may further include a voice emotion model unit providing analysis criteria of the text emotion model unit and a text emotion model unit storing characteristics for each emotion of the text and providing analysis criteria in the text emotion analysis unit.

The multimodal emotion analysis system according to the present invention may further include an ensemble emotion model unit for storing combination characteristics for each emotion and providing analysis criteria in the ensemble emotion analyzer.

The ensemble sentiment analyzer may analyze combination characteristics using LSTM.

According to another embodiment of the present invention, the video input step of receiving a user video input; Sentiment analysis step of analyzing each emotion of the user face image, user voice and text extracted from the user video; and an ensemble emotion analysis step of analyzing a combination of image emotion characteristics, voice emotion characteristics, and text emotion characteristics calculated in the emotion analysis step.

In the emotion analysis step, the user's face image is analyzed through a plurality of different analysis models to calculate a plurality of sub-image emotion characteristics, and the user's voice is analyzed through a plurality of different analysis models to obtain a plurality of sub-voice emotions. a sub-sentiment analysis step of calculating characteristics and calculating a plurality of sub-text sentiment characteristics by analyzing each of the texts through a plurality of different analysis models; and an emotion feature combining step of combining the sub image emotion feature, the sub voice emotion feature, and the text emotion feature calculated in the sub emotion analysis step for each user face image, user voice, and text.

According to the multimodal emotion analysis system according to the present invention, it is possible to accurately grasp the user's emotional state by considering the user's face, voice, and text together in the user's video.

In addition, since a combination feature combining image emotion characteristics, voice emotion characteristics, and text emotion characteristics is analyzed, the accuracy of the result of emotion analysis can be further increased.

After analyzing each of the emotions of the user's face image, voice, and text through a number of different analysis models, combining and analyzing the calculated sub-emotional features, it is possible to accurately analyze the emotion analysis results of each image emotion, voice emotion, and text. there is.

1 is a schematic configuration diagram of a multimodal emotion analysis system according to the present invention;
2 is an explanatory diagram of an image emotion model unit constituting a multimodal emotion analysis system according to the present invention;
3 is an explanatory diagram of a voice emotion model unit constituting a multimodal emotion analysis system according to the present invention;
4 is an explanatory diagram of a text emotion model unit constituting a multimodal emotion analysis system according to the present invention;
5 is an explanatory diagram of an ensemble emotion model unit constituting a multimodal emotion analysis system according to the present invention;
6 is an explanatory diagram of an image emotion analysis unit constituting a multimodal emotion analysis system according to the present invention;
7 is an explanatory diagram of a voice emotion analysis unit constituting a multimodal emotion analysis system according to the present invention;
8 is an explanatory diagram of a text sentiment analysis unit constituting a multimodal sentiment analysis system according to the present invention;
9 is an explanatory diagram of an ensemble emotion analysis unit constituting a multimodal emotion analysis system according to the present invention;
10 is a flowchart of a multimodal emotion analysis method according to the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings.

1 shows a schematic configuration diagram of a multimodal emotion analysis system 1 according to the present invention.

The multimodal emotion analysis system 1 according to the present invention includes a video input unit 10, an image emotion analyzer 20, a voice emotion analyzer 30, a text emotion analyzer 40, and an ensemble emotion analyzer 50. made including

The video input unit 10 receives a user video including the user's face image and voice from the user's mobile phone or computer. The user's face image, user's voice, and text are extracted from the user's video. Text can be extracted by applying a speech to text (STT) algorithm to a user's voice.

The image emotion analyzer 20 analyzes the emotion of the user's face image. The image emotion analysis unit 20 may analyze the emotion of the user's face image through characteristics such as eye blinking, mouth shape, and face length, for example.

The voice emotion analysis unit 30 analyzes the emotion of the user's voice. In the voice emotion analysis unit 30, for example, pitch, energy, entropy of energy, zero crossing rate, spectrogram, log-mel-spectrogram ( The emotion of the user's voice can be analyzed through acoustic characteristics such as log-mel-spectrogram) and frequency.

The text emotion analysis unit 40 analyzes the emotion of the user's text through the semantic characteristics of the text.

In the ensemble emotion analyzer 50, the image emotion characteristics calculated by the image emotion analyzer 20, the voice emotion characteristics calculated by the voice emotion analyzer 30, and the text emotion characteristics calculated by the text emotion analyzer 40 Analyze the combination characteristics of the combination.

The combination feature may be analyzed, for example, by averaging the probability of emotion calculated from the image emotion feature, the voice emotion feature, and the text emotion feature for each emotion. It can also be analyzed by calculation method, etc.

According to the multimodal emotion analysis system 1 according to the present invention, it is possible to accurately grasp the emotional state by considering the user's face, voice, and text together in the user's video.

In addition, since a combination of image emotion characteristics, voice emotion characteristics, and text emotion characteristics is analyzed, the accuracy of the emotion analysis result can be further improved.

The multimodal emotion analysis system 1 according to the present invention may include an image emotion model unit 60, a voice emotion model unit 70, and a text emotion model unit 80.

The image emotion model unit 60 may store characteristics for each emotion of the image and provide analysis criteria in the image emotion analyzer 20 . That is, the image emotion analysis unit 20 may derive an image emotion result by finding which emotion the feature of the image corresponds to in the image emotion model unit 60 .

As shown in FIG. 2 , the image sensibility model unit 60 may include an image data unit 61 , an image sensibility model learning unit 63 , and an image sensibility model storage unit 64 .

The image data unit 61 stores a plurality of images labeled with emotions. The image data may be obtained from a drama, movie, still image or moving picture file taken by a user, and includes information about what emotion the corresponding image data is related to.

The image emotion model learning unit 63 learns image data using an artificial intelligence algorithm. Accordingly, image data is classified according to emotion. The image sensitivity model learning unit 63 may learn image data using, for example, at least one artificial intelligence algorithm of VGG, ResNet, and Inception.

The image data may go through the image pre-processing unit 62 before being transmitted to the image sensitivity model learning unit 63 . The image pre-processing unit 62 may perform tasks such as extracting a still image from a video, extracting a face part from an image, and adjusting the brightness of an image.

The image sensitivity model derived from the image sensitivity model learning unit 63 is stored in the image sensitivity model storage unit 64 . The image emotion analysis unit 20 may analyze the emotion of the image by referring to the image emotion model stored in the image emotion model storage unit 64 and confirming which emotion image of the user's face image is similar to.

The voice emotion model unit 70 stores characteristics for each emotion of the voice and provides analysis criteria in the voice emotion analyzer 30 . That is, the voice emotion analysis unit 30 may derive a voice emotion result by finding which emotion the acoustic characteristics of the voice correspond to in the voice emotion model unit 70 .

As shown in FIG. 3, the voice emotion model unit 70 includes a voice data unit 71, a voice feature vector extractor 73, a voice emotion model learning unit 74, and a voice emotion model storage unit 75. It can be made including.

The voice data unit 71 stores a plurality of voice data labeled with emotions. The voice data may be obtained from a drama, movie, or a voice file recorded by a user, and includes information about what emotion the corresponding voice data is related to.

The speech feature vector extraction unit 73 extracts speech feature vectors of speech data. The voice feature vectors extracted by the voice feature vector extractor 73 may include pitch, energy, entropy of energy, zero crossing rate, spectrogram, log-mel-spectrogram, and frequency. The speech feature vector extractor 73 may extract speech feature vectors through MFCC (Mel Frequency Cpstral Coeffiecient), for example.

The voice data of the voice data unit 71 may be transmitted to the voice feature vector extractor 73 after passing through the voice pre-processor 72 .

The voice pre-processing unit 72 may include a pre-emphasis processing unit, a framing processing unit, and a hamming window processing unit. The pre-emphasis processor uses a pre-emphasis filter to emphasize high-frequency components of the audio signal, and the framing processor divides the audio signal into regular intervals. The Hamming window processing unit minimizes discontinuity at the boundary of each frame by multiplying each frame of the speech signal divided and formed in the framing processing unit by the Hamming window function.

The voice emotion model learning unit 74 learns the voice feature vector extracted by the voice feature vector extractor 73 using an artificial intelligence algorithm. Accordingly, voice feature vectors are classified according to emotion. The speech emotion model learning unit 74 may learn a speech feature vector through at least one of, for example, a support vector machine (SVM) model and a combination model of a convolution neural network (CNN) and a recurrent neural network (RNN). can

The voice emotion model derived from the voice emotion model learning unit 74 is stored in the voice emotion model storage unit 75 . The voice emotion analysis unit 30 may analyze the emotion of the user's voice by referring to the voice emotion model stored in the voice emotion model storage unit 75 and checking which emotion of the user's voice is similar to a voice feature vector.

In the text emotion model unit 80, characteristics for each emotion of the text are stored to provide analysis criteria in the text emotion analyzer 40. That is, the text emotion analysis unit 40 may derive a text emotion result by finding which emotion the text characteristic corresponds to in the text emotion model unit 80 .

More specifically, the text sentiment model unit 80 may include a text data unit 81 , a text sentiment model learning unit 82 , and a text sentiment model storage unit 83 . 4 shows a schematic configuration diagram of the text emotion model unit 80 .

The text data unit 81 stores text data labeled with emotions. Such text data may include data such as the NRC Sentimental Dictionary and the KNU Korean Sentimental Dictionary. For example, in the text data unit 81, vocabularies may be classified according to five emotions.

The text emotion model learning unit 82 learns text data for each emotion of the text data unit 81 through an artificial intelligence algorithm. Accordingly, the characteristics of the text are shunted by emotion. The text emotion model learning unit 82 may learn text through BERT or KoBERT, for example.

The multimodal emotion analysis system 1 according to the present invention may further include an ensemble emotion model unit 90 . 5 shows an explanatory diagram of the ensemble emotion model unit 90.

The ensemble emotion modeling unit 90 stores the combination characteristics for each emotion and provides analysis criteria in the ensemble emotion analysis unit 50. The image emotion model of the image emotion model unit 60 and the voice emotion model unit 70 It can be created by combining the voice emotion model of and the text emotion model of the text emotion model unit 80.

As shown in FIG. 6 , the image emotion analyzer 20 may include a plurality of sub-image emotion analyzers 21 and an image emotion feature combiner 22 .

Each of the sub-image emotion analysis units 21 analyzes the user's face image by applying different image analysis models.

As the image analysis model, for example, at least two of VGG, ResNet, and Inception may be used. In VGG, the number of parameters can be reduced by using a 3×3 filter, but it can exhibit superior performance compared to the case of using a 5×5 or 7×7 filter. ResNet can solve the vanishing gradient problem by adding a skip connection part. In addition, when using a convolutional neural network, Inception can improve analysis performance by applying all of them without selecting which size filter or pooling to apply.

The image sensibility feature combining unit 22 combines the sub-image sensibility characteristics calculated in each sub-image sensibility analysis unit 21 .

Since the image emotion feature created by combining several sub-image emotion features can have advantages of various image analysis models, the accuracy of image emotion analysis can be increased.

As shown in FIG. 7 , the voice emotion analysis unit 30 may include a sub voice emotion analysis unit 31 and a voice emotion feature combination unit 32 .

Each of the sub voice emotion analysis units 31 analyzes the user's voice by applying different voice analysis models.

As the voice analysis model, for example, SVM and a combined model of CNN and RNN may be used. In the case of using the SVM model, it is possible to analyze the user's voice to deep emotions. Using the combined model of CNN and RNN, almost no pre-processing of the user's voice is required.

The voice emotion feature combining unit 32 combines the sub voice emotion characteristics calculated in each of the sub voice emotion analysis units 31 .

Since the voice emotion feature created by combining several sub voice emotion features can have advantages of several voice analysis models, the accuracy of voice emotion analysis can be increased.

As shown in FIG. 8 , the text sentiment analysis unit 40 may include a sub-text sentiment analysis unit 41 and a text sentiment feature combination unit 42 .

Each of the sub-text sentiment analysis units 41 analyzes the sentiment of the text by applying different text sentiment analysis models.

For example, BERT and KoBERT may be used as a text sentiment analysis model. Because BERT contains a lot of information about the language, the text analysis result is accurate, and KoBERT overcomes the Korean performance limitation of BERT, so the text analysis result made in Korean is especially accurate.

For reference, KoBERT is based on BERT for embedding and CNN for classification.

The text sentiment feature combining unit 42 combines the sub-text sentiment characteristics calculated in each of the sub-text sentiment analysis units 41 .

Since the text sentiment features created by combining several sub-text sentiment features can have the advantages of various text analysis models, the accuracy of text sentiment analysis can be increased.

The ensemble emotion analysis unit 50 analyzes the combination characteristics using long-short term memory (LSTM).

LSTM is useful for processing long sequences of inputs because it can erase unnecessary memories and set things to remember by adding input gates, forget gates, and output gates to memory cells in the hidden layer.

Therefore, it is possible to improve the analysis performance of the user's face image, the user's voice, and the text extracted from the video and composed of several sections.

The ensemble emotion analysis unit 50 may include a result confirmation unit 51 and an adjustment unit 52 . FIG. 9 shows an explanatory diagram of the ensemble emotion analysis unit 50 .

The result checking unit 51 checks whether the analysis result of the combination feature, for example, a sentiment result having a superior probability is derived. And, if the emotional result having a superior probability is derived, the final result is output as it is, and if not, the adjustment unit 52 operates.

The adjustment unit 52 serves to derive the user's emotion when the emotion result having a superior probability is not derived. For example, the adjustment unit 52 analyzes the raw data of the user's video through a decision tree algorithm, then analyzes the degree of influence of each image, voice, and text on the analysis result of the raw data, and each emotion according to the degree of influence. Relative weights can be assigned to the analysis model. In addition, when the weight is applied, the emotion derived with the highest probability can be output as the final result of the emotion analysis. Separation criteria of the decision tree algorithm may include, for example, the size of voice frequencies and the length of syllables.

Hereinafter, a multimodal emotion analysis method according to the present invention will be described. While explaining the multimodal emotion analysis method according to the present invention, detailed descriptions of the parts mentioned in the description of the multimodal analysis system 1 may be omitted.

10 is a flowchart of a multimodal emotion analysis method according to the present invention.

The multimodal emotion analysis method according to the present invention includes a video input step (S10), a emotion analysis step (S20), and an ensemble emotion analysis step (S30).

In the video input step (S10), a user's video is input. A user's face image, user's voice, and text may be separately extracted from the user's video.

In the emotion analysis step (S20), the emotion of each of the user face image, user voice and text extracted from the user video is analyzed. In the user's face image, for example, emotion can be analyzed by detecting eye blinking, mouth shape and face length, etc., and in the user's voice, for example, voice pitch, energy, energy entropy, zero crossing rate, spectrogram, log -Sentiment can be analyzed through acoustic characteristics such as Mel-Spectrogram and frequency. In addition, sentiment analysis can be performed by grasping the meaning of the text.

In the ensemble emotion analysis step ( S30 ), a final emotion analysis result is derived by analyzing a combination of image emotion characteristics, voice emotion characteristics, and text emotion characteristics calculated in the emotion analysis step.

Since the combined features include all of the image sensibility features, voice sensibility features, and text sensibility features, a more accurate sentiment analysis result can be obtained by analyzing them.

The emotion analysis step (S20) may include a sub emotion analysis step and an emotion feature combination step (S22).

In the sub-sentiment analysis step (S21), a number of sub-image emotion characteristics are calculated by analyzing the user's face image through a number of different analysis models, and a number of sub-voice emotions are analyzed by analyzing the user's voice through a number of different analysis models. The feature is calculated, and the text is analyzed through a number of different analysis models to calculate a number of sub-text emotional characteristics.

In the emotion feature combining step S22, the sub image emotion characteristics, sub voice emotion features, and text emotion features calculated in the sub emotion analysis step are combined for each user face image, user voice, and text.

Emotional features created by combining each sub-emotional feature for each user's face image, user's voice, and text can have advantages of various analysis models, so that the accuracy of emotion analysis can be increased.

The multimodal emotion analysis system 1 and method according to the present invention may be implemented through one or more hardware or one or more software. Alternatively, it may be implemented through hardware and software.

The scope of the present invention is not limited to the above-described embodiments, but may be implemented in various forms of embodiments within the scope of the appended claims. Anyone with ordinary knowledge in the art to which the invention pertains without departing from the subject matter of the invention claimed in the claims is considered to be within the scope of the claims of the present invention to various extents that can be modified.

1: Multimodal Sentiment Analysis System
10: video input unit 21: sub image emotion analysis unit
22: image emotion feature combination unit 30: voice emotion analysis unit
31: sub voice emotion analysis unit 32: voice emotion feature combination unit
40: text sentiment analysis unit 41: subtext sentiment analysis unit
42: text emotion feature combination unit 50: ensemble emotion analysis unit
60: image emotion model unit 70: voice emotion model unit
80: text emotion model unit 90: ensemble emotion model unit

Claims (9)

a video input unit that receives a user video;
an image emotion analysis unit that analyzes emotion of the user's face image extracted from the user's video;
a voice emotion analyzer for analyzing the emotion of the user's voice extracted from the user's video;
a text emotion analyzer for analyzing the emotion of the text extracted from the user's video; and
an ensemble emotion analysis unit that analyzes a combination of the image emotion characteristics calculated by the image emotion analysis unit, the voice emotion characteristics calculated by the voice emotion analysis unit, and the text emotion characteristics calculated by the text emotion analysis unit; contains,
The image emotion analyzer includes a plurality of sub-image emotion analyzers to which different image analysis models are applied and an image emotion feature combination unit for combining sub-image emotion features calculated in each of the sub-image emotion analyzers;
The voice emotion analysis unit includes a plurality of sub voice emotion analysis units to which different voice analysis models are applied and a voice emotion feature combining unit that combines the sub voice emotion characteristics calculated by each of the sub voice emotion analysis units;
The text sentiment analysis unit includes a plurality of sub-text sentiment analysis units to which different text analysis models are applied and a text sentiment feature combining unit combining sub-text sentiment characteristics calculated in each of the sub-text sentiment analysis units;
The ensemble emotion analysis unit,
A result confirmation unit that checks whether a sentiment result having a superior probability has been derived from the analysis result of the combination feature, and outputs the sentiment result as a final result as it is when a sentiment result having a superior probability is derived; and
And an adjustment unit that outputs a final result when an emotional result having a superior probability is not derived from the result confirmation unit,
The adjustment unit,
After analyzing the raw data of the user video input from the video input unit through a decision tree algorithm, the degree of influence of each image, voice, and text on the analysis result of the raw data of the user video is analyzed, and according to the degree of influence, the image, A multimodal emotion analysis system, characterized in that it assigns weights to voice and text and outputs the emotion result derived with the highest probability when the weight is applied as the final result.
delete According to claim 1,
The image analysis model is a multimodal emotion analysis system, characterized in that VGG, ResNet, and Inception.
According to claim 1,
The speech analysis model is a multimodal emotion analysis system, characterized in that SVM, and a combined model of CNN and RNN.
According to claim 1,
An image emotion model unit for storing characteristics of each emotion of the image and providing an analysis criterion in the image emotion analyzer;
A voice emotion model unit for storing characteristics of each emotion of voice and providing analysis criteria in the voice emotion analyzer; and
The multimodal emotion analysis system further comprises a text emotion model unit for storing characteristics of each emotion of the text and providing an analysis criterion in the text emotion analyzer.
According to claim 1,
The multimodal emotion analysis system further comprising an ensemble emotion model unit storing combination characteristics for each emotion and providing an analysis criterion in the ensemble emotion analyzer.
According to claim 1,
The ensemble emotion analysis unit is a multimodal emotion analysis system, characterized in that for analyzing combination characteristics using LSTM.
delete delete

Images