KR20210027769A - Neural network based sentiment analysis and therapy system - Google Patents

Abstract

The present invention relates to a neural network-based emotion analysis method. The neural network-based emotion analysis method comprises the steps of: (a) calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user; (b) calculating a second emotion analysis result by applying a context analysis based on an emotion dictionary to the voice data; and (c) outputting an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as input to a neural network, thereby deriving the more accurate emotion analysis result.

Description

Neural network based emotion analysis and emotional therapy system {NEURAL NETWORK BASED SENTIMENT ANALYSIS AND THERAPY SYSTEM}

The present application relates to a neural network-based emotion analysis and emotional treatment system. In particular, the present application relates to a neural network-based emotion analysis device, method and system, and a neural network-based emotion analysis-based emotion (emotional) treatment device, method, and system.

Emotional engineering refers to the technical realization of non-verbal elements such as human mind and mood in engineering to be helpful in real life. Unlike the strategies of companies in the past that focused only on function and quality, in recent years, developing products by stimulating people's emotions (emotions) or utilizing them has become a central strategy.

Affective Computing is a field that researches and develops artificial intelligence related to designing systems and devices that can recognize, interpret, and process human emotions. It refers to an artificial intelligence-based system. This is a technology that can be used for interaction between humans and computers by recognizing the psychological reactions of people through physical or sensory stimulation.

With recent technological advances, machine-humans are in close relationship, and human characteristics are reason and sensibility, so it is essential to consider the emotional (emotional) aspect when humans and robots interact. Accordingly, many domestic and foreign companies are actively conducting research related to emotional computing, but it is still an early stage of commercialization and is a highly valuable field for research.

Conventionally, as technologies for analyzing user's emotions (sensibility), there are emotion analysis technologies using voice analysis and natural language processing.

However, conventionally known voice analysis-based emotion analysis techniques are difficult to recognize when a user speaks an emotional sentence with a neutral accent, and the emotion analysis results are derived differently due to individual differences because of the same sentence. There is a problem of poor accuracy.

The technology behind the present application is [Yelin Kim and Emily Mower Provost. 2015. Emotion Recognition During Speech Using Dynamics of Multiple Regions of the Face. ACM Trans. Multimedia Comput. Commun. Appl. 12, 1s, Article 25 (October 2015), 23 pages.].

The present application is to solve the problems of the prior art described above, and to provide a neural network-based emotion analysis device and method and a neural network-based emotion analysis-based emotion (emotional) treatment device and method capable of deriving more accurate emotion analysis results. The purpose.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the neural network-based emotion analysis method according to the first aspect of the present application includes (a) applying a neural network-based speech analysis to the voice data input from the user to obtain the first emotion analysis result. Calculating; (b) calculating a second emotion analysis result by applying a context analysis based on an emotion dictionary to the voice data; And (c) outputting an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as inputs of a neural network.

As a technical means for achieving the above technical task, the emotion treatment method based on neural network-based emotion analysis according to the second aspect of the present application is an integrated emotion analysis result provided by performing the neural network-based emotion analysis method according to the first aspect of the present application. Based on the emotion treatment method can be determined and provided.

As a technical means for achieving the above-described technical problem, the neural network-based emotion analysis apparatus according to the third aspect of the present application is a voice for calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user. Analysis module; A context analysis module for calculating a second emotion analysis result by applying a context analysis based on an emotion dictionary to the voice data; And an integrated module configured to output an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as inputs of a neural network.

As a technical means for achieving the above technical task, the emotion treatment apparatus based on a neural network based emotion analysis according to the fourth aspect of the present application performs a neural network based emotion analysis by the neural network based emotion analysis apparatus according to the third aspect of the present application. Thus, based on the result of integrated emotion analysis provided, an emotion treatment method can be determined and provided.

As a technical means for achieving the above technical task, the computer program according to the fifth aspect of the present application includes a neural network-based emotion analysis method according to the first aspect of the present application, and a neural network-based emotion analysis-based emotion analysis method according to the second aspect of the present application. It may be stored in a recording medium to execute the treatment method.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, the integrated emotion analysis result is output by considering the neural network-based voice analysis and the emotion dictionary-based context analysis together, so that the user's emotional state for the input voice data can be more accurately derived. There is an effect that can be.

According to the above-described problem solving means of the present application, by providing an emotion treatment method determined based on the result of integrated emotion analysis, it is possible to accurately grasp the emotional/emotional state of the user, and based on this, by inducing the user's current emotion to change more positively. , Can provide users with a better life.

According to the above-described problem solving means of the present application, by providing an emotion treatment method, if the user's emotion is a negative emotion or a neutral emotion, it is possible to induce the user's emotion to change into a positive emotion, and if the user's emotion is a positive emotion, the corresponding positive emotion This can be induced to be maintained or extended beyond that.

However, the effect obtainable in the present application is not limited to the above-described effects, and other effects may exist.

1 is a diagram showing a schematic configuration of a neural network-based emotion analysis system according to an embodiment of the present application.
2 is a diagram showing a schematic configuration of an apparatus for analyzing emotions based on a neural network according to an embodiment of the present application.
3 shows an example of an integrated emotion analysis result derived by a neural network-based emotion analysis device according to an embodiment of the present application.
4 shows another example of an integrated emotion analysis result derived by a neural network-based emotion analysis apparatus according to an embodiment of the present application.
5 is a diagram illustrating a schematic configuration of an emotion treatment system based on emotion analysis based on a neural network according to an embodiment of the present application.
6 is a diagram showing a schematic configuration of an emotion treatment apparatus based on emotion analysis based on a neural network according to an embodiment of the present application.
FIG. 7 is a diagram illustrating an example in which emotion treatment of a user is performed by an emotion treatment apparatus based on emotion analysis based on a neural network according to an exemplary embodiment of the present disclosure.
FIG. 8 is a diagram schematically illustrating a menu configuration in a mobile application linked with an emotion treatment device based on a neural network based emotion analysis according to an embodiment of the present application.
9 is a view for explaining an example of calculating a second emotion analysis result by a context analysis module in the emotion treatment apparatus based on emotion analysis based on a neural network according to an embodiment of the present application.
10A to 10C are diagrams for explaining an example of calculating a first emotion analysis result by a voice analysis module in the emotion treatment apparatus based on emotion analysis based on a neural network according to an exemplary embodiment of the present disclosure.
11 is a diagram for describing an example of calculating an integrated emotion analysis result by an integrated module in the emotion treatment apparatus based on emotion analysis based on a neural network according to an exemplary embodiment of the present application.
12 is a diagram showing an example of a screen display of an emotion analysis result to which a conventional natural language processing is applied to given voice data.
13 is a diagram illustrating an example of displaying an emotion analysis result on a screen through application of a context analysis by a context analysis module of a neural network-based emotion analysis-based emotion treatment apparatus according to an exemplary embodiment of the present application.
14 is a flowchart illustrating a method for analyzing emotions based on a neural network according to an embodiment of the present application.
15 is a flowchart illustrating an operation of an emotion treatment method based on a neural network based emotion analysis according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be "connected" with another part, it is not only the case that it is "directly connected", but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including the case.

Throughout the present specification, when a member is positioned "on", "upper", "upper", "under", "lower", and "lower" of another member, this means that a member is located on another member. This includes not only the case where they are in contact but also the case where another member exists between the two members.

In the entire specification of the present application, when a certain part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

1 is a diagram showing a schematic configuration of a neural network-based emotion analysis system 200 according to an embodiment of the present application. 2 is a diagram showing a schematic configuration of a neural network-based emotion analysis apparatus 100 according to an embodiment of the present application.

Hereinafter, for convenience of description, the neural network-based emotion analysis system 200 according to an embodiment of the present application is referred to as the present system 200, and the neural network-based emotion analysis apparatus 100 according to an embodiment of the present application is referred to as the present apparatus 100 Let's call it d.

1 and 2, the system 200 may include a user terminal 2 and the device 100.

The user terminal 2 may receive voice data d to be analyzed for emotion from the user 1. The user terminal 2 may transmit the received voice data d to the device 100.

The user terminal 2 may be, for example, a mobile and portable communication device such as a smart phone, a smart pad, a tablet PC, a notebook, and a wearable device. However, the user terminal 2 is not limited thereto, and the user terminal 2 includes a Personal Communication System (PCS), a Global System for Mobile communication (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), and a Personal Digital Assistant (PDA). ), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, Smartphone, SmartPad , A tablet PC, a notebook computer, a wearable device, a desktop PC, and the like.

The device 100 may analyze the user's emotions through analysis (speech analysis, context analysis) of the voice data d input from the user 1. The device 100 applies voice analysis and context analysis to the voice data (d) input from the user 1, and applies the first emotion analysis result (d1) and context analysis calculated through the application of the voice analysis. An integrated emotion analysis result (d3) in which the second emotion analysis result (d2) calculated through is integrated may be output as a result of the user's emotion analysis on the voice data (d). Here, the voice data d inputted by the apparatus 100 may be input through the user terminal 2 (in other words, transmitted from the user terminal).

The device 100 may be, for example, a server. However, the present invention is not limited thereto, and as another example, the device 100 may refer to a device provided in the user terminal 2 and, in particular, a mobile application 2a installed in the user terminal 2 itself.

An example of a case in which the device 100 is provided in the form of a server is as follows. The user terminal 2 may be installed with a mobile application 2a interworking with the device 100 that performs emotion analysis on the voice data d input from the user 1. The mobile application 2a in the user terminal 2 may transmit the voice data d input from the user 1 to the device 100. The device 100 applies the voice analysis and context analysis to the voice data (d) received from the mobile application (2a) to obtain the integrated emotion analysis result (d3) as a result of the user's emotion analysis on the voice data (d). Can be generated. Thereafter, the device 100 may provide the generated integrated emotion analysis result d3 to the mobile application 2a in the user terminal 2 (providing an output), and through this, on the screen of the user terminal 2 The integrated emotion analysis result d3 may be displayed (displayed). The user may receive and check the integrated emotion analysis result d3 output from the device 100 through the screen of the user terminal 2.

In this way, when the device 100 is a server (when it is provided in the form of a server), the device 100 and the user terminal 2 transmit and receive data through network communication (that is, voice data from the user terminal to the device). And transmission of the result of the integrated emotion analysis from the device to the user terminal) can be performed.

Network communication is, for example, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area). Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC (Near Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc. However, it is not limited thereto.

An example of a case in which the present device 100 (in particular, the emotion analysis method by the present device) is provided in the form of a mobile application 2a in the user terminal 2 is as follows. The device 100 receives voice data (d) from the user 1, and applies voice analysis and context analysis to the input voice data (d) as a result of the user's emotion analysis of the voice data (d). An integrated emotion analysis result (d3) may be generated. Thereafter, the device 100 may provide the generated integrated emotion analysis result d3 on the screen of the user terminal 2 (providing an output), through which integrated emotion analysis on the screen of the user terminal 2 The result d3 can be displayed (displayed). The user may receive and check the integrated emotion analysis result d3 output from the device 100 through the screen of the user terminal 2.

Hereinafter, for convenience of description, the device 100 will be illustrated as a server (a case provided in the form of a server), and a detailed description of the device 100 is as follows.

The device 100 may include a speech analysis module 10, a context analysis module 20, and an integration module 30.

The voice analysis module 10 may calculate the first emotion analysis result d1 by applying the voice analysis based on the neural network 12 to the voice data d input from the user 1. The speech analysis module 10 may transmit the calculated first emotion analysis result d1 to the integrated module 30.

The voice analysis module 10 may calculate the first emotion analysis result d1 through voice analysis based on the neural network 12 that receives the voice waveform feature extracted from the voice data d as an input.

Here, the voice waveform feature may be an MFCC feature value output by performing MFCC (Mel-Frequency Cepstral Coefficient) transformation on the voice data d. In addition, the first emotion analysis result calculated by the voice analysis module 10 is probabilistic for each of a plurality of emotions including happiness, neutrality, sadness, and anger. It may be a given value.

Specifically, the voice analysis module 10 may receive voice data d input through the mobile application 2a of the user terminal 2 from the user terminal 2.

For example, the user 1 may record a sentence representing the current emotion through the mobile application 2a of the user terminal 2. The voice file corresponding to the sentence representing the current emotion of the user 1 recorded through the mobile application 2a may be provided as an input of the voice analysis module 10 as voice data d to be voice analyzed. . In the present apparatus 100, as an example, the recorded file is considered as voice data (d), but is not limited thereto, and as another example, the voice data (d) may be voice data input in real time from a user. have.

The voice analysis module 10 may receive (deliver) a voice file having a .wav extension from the mobile application 2a as voice data d as a voice analysis target.

The speech analysis module 10 may apply Mel-Frequency Cepstral Coefficient (MFCC) as an example speech recognition algorithm to the input speech data d. The voice analysis module 10 can perform MFCC conversion 11 through the application of MFCC to the input voice data d, and output MFCC feature values as voice waveform features for voice data d. can do. That is, the speech analysis module 10 models the sound corresponding to the speech data (d) by performing MFCC conversion (11) on the input speech data (d), and converts the MFCC characteristic value (MFCC value) accordingly to the speech data. (d) can be output as a characteristic of the voice waveform.

The voice analysis module 10 may provide a voice waveform feature, which is an MFCC feature value output through the MFCC conversion 11, as an input of the neural network 12.

As the neural network 12, for example, Keras may be considered. Keras stands for an open source neural network library written in Python. Keras can run on MXNet, Deeplearning4j, TensorFlow, Microsoft Cognitive Toolkit or Theano. Keras is designed to enable rapid experimentation with deep neural networks and focuses on the possibility of extensibility with minimal modularity. Keras was developed as part of the research effort of the ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System) project.

Keras is a technology well known to those of ordinary skill in the art to which the present application belongs, so hereinafter, rather than a description of Keras itself, the Keras-based neural network 12 is used to analyze a neural network-based emotion according to an embodiment of the present application. An example applied to the device 100 will be mainly described.

In the apparatus 100, a convolutional neural network (CNN, convolutional neural network) may be considered as the neural network 12 as an example. However, the present invention is not limited thereto, and the neural network 12 considered in the device 100 is, for example, a recurrent neural network (RNN), a deep neural network, etc. Various neural networks to be developed can be applied. The neural network 12 may be expressed differently as a deep learning model or the like.

The voice analysis module 10 can provide the voice waveform feature, which is the MFCC feature value output through the MFCC conversion 11, as an input of the neural network 12, through which the neural network 12 performs voice analysis based on the neural network ( As an output of 12), the first emotion analysis result d1 can be calculated.

In this case, the voice analysis module 10 may provide not only voice waveform characteristics, but also model data obtained from a local storage 13 as an input of the neural network 12. Through this, the voice analysis module 10 may calculate the first emotion analysis result d1 through voice analysis based on the neural network 12.

Here, the model data is a learning value of the neural network determined by repeatedly learning the neural network in a neural network that is trained to output a first emotion analysis result corresponding to the input MFCC as an input of the neural network with MFCC feature values for a plurality of voice data. (Specifically, it may mean a weight value of a neural network determined through learning). That is, the model data may mean a weight value of the trained neural network. The local storage 13 may mean a database in which a plurality of such model data is stored.

The voice analysis module 10 passes the voice waveform feature (ie, MFCC feature value) of the voice data d and the model data obtained from the local storage 13 through the neural network 12, From 12), the first emotion analysis result d1 may be calculated (output).

The first emotion analysis result d1 may be a value probabilistically given to each of a plurality of emotions including happiness, neutrality, sadness, and anger. The first emotion analysis result d1, which is a value probabilistically assigned to each of the plurality of emotions, indicates the probability of the emotion possessed by the voice data d. In other words, the first emotion analysis result d1 represents a probability (probability value, probability score) for each of a plurality of emotions included in the voice data d.

As the first emotion analysis result d1 is a value probabilistically given to each of the plurality of emotions, the sum of values probabilistically given to each of the plurality of emotions may be 1.

Exemplarily, the speech analysis module 10 is a first emotion analysis result (d1) such as'Happy: 0.05, Neutral: 0.03, Sad: 0.7, Angry: 0.22' Can be calculated.

At this time, according to the first emotion analysis result d1, as the probability value (probability score) of the emotion corresponding to sadness among the four emotions has the largest value as 0.7, the speech analysis module 10 From the first emotion analysis result d1 calculated by the application of the neural network-based speech analysis, it can be determined that the emotion corresponding to'sadness' occupies the largest proportion in the input voice data d. In other words, the voice analysis module 10 may determine that the user's emotion with respect to the voice data d based on the neural network-based voice analysis is'sad' through the first emotion analysis result d1.

The device 100 is exemplarily illustrated as including happiness, neutrality, sadness, and anger as a plurality of four emotions, but is not limited thereto, and the device 100 includes a plurality of emotions or a plurality of emotions. The type of emotion can be changed and applied in more various ways.

The first emotion analysis result d1 calculated from the voice analysis module 10 may be input to the integrated module 30. The first emotion analysis result d1 may be used (used) to extract (generate) the integrated emotion analysis result d3 in the integrated module 30.

The context analysis module 20 applies the context analysis based on the emotion dictionary (sense word dictionary, emotion dictionary DB, emotion word DB) 23 to the voice data d input from the user 1, and results of the second emotion analysis. (d2) can be calculated.

The context analysis module 20 converts the voice data d into text, then extracts at least one basic word type (Lemma) from the converted text data, and converts the extracted at least one basic word type into an emotion dictionary DB ( The second emotion analysis result d2 may be calculated by matching with the word basic type included in 23).

Here, the emotion dictionary DB 23 may be constructed in a form in which emotion scores related to the strength of the positive and the strength of the negative are given to each of the plurality of basic types of words. The context analysis module 20 calculates a second emotion analysis result (d2) by considering the emotion score of each of the basic types of matching words that match the basic types of words included in the emotion dictionary DB 23 among the extracted basic types of at least one word. I can.

As a specific example, the context analysis module 20 may receive (deliver) a voice file having a .wav extension from the mobile application 2a as voice data d as a voice analysis target.

The context analysis module 20 may convert the input voice data (ie, .wav voice file) (d) into text (text data, Text Data) through STT (Speech To Text, 21). Thereafter, the context analysis module 20 may extract a word Lemma using the converted text data as an example using the Google Natural Language 22, which is Google's natural language processing technology. In other words, the context analysis module 20 may extract at least one word Lemma from the converted text by applying the Google Natural Language 22 to the converted text.

In the text, a basic word that becomes the root of a word that can be used by transforming the word is included, and such basic type words (basic type of word, basic type of word) can be called Lemma. For example,'write','writing','wrote', and'written' are all based on the basic form of'write'. Therefore, when at least one of'write','writing','wrote', and'written' is included in the converted text as an example, the context analysis module 20 is ), you can extract'write'.

In addition, the context analysis module 20 may extract not only a word basic type (Lemma) but also a dependency (Dependency) from the text data converted through the application of the Google Natural Language 22.

Here, the term “dependency” refers to the relationship of the dependency between word tokens (corpus within a sentence, word units). The dependence of the converted text can be extracted (identified) through a dependency tree (Dependency Tree, dependency tree, dependency tree) provided by Google Natural Language (22), for example. The description of the dependency tree is exemplarily described in “Universal Dependency Annotation for Multilingual Parsing”, Ryan McDonald, Joakim Nivre et al. 11, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 92-97, Sofia, Bulgaria, August 4-9 2013.] may be more easily understood, and a detailed description thereof will be omitted herein.

Dependency extracted from the context analysis module 20 may be used to reinforce the influence of emotion words (emotional words) and to determine the cause thereof.

After extracting at least one word basic type (Lemma) from the converted text (Text Data), the context analysis module 20 performs a matching of the extracted at least one word basic type with the word basic type included in the emotion dictionary DB 23 can do. At this time, the emotion dictionary DB 23 may be constructed and provided in a form in which emotion scores related to the strength of positive and the strength of negative are given to each of the plurality of basic types of words.

By matching the extracted at least one word basic type with the word basic type included in the emotion dictionary DB 23, the context analysis module 20 includes a word included in the emotion dictionary DB 23 among the extracted at least one basic word type. It is possible to obtain an emotion score for each of the basic types of matching words that match the basic type.

For example, the emotion dictionary DB 23 includes'Fun: +7.0','gift: +6.0','travel: +3.0','slipper: 0.0', Data such as'funeral: -6.0','fail: -5.0', etc. may be pre-stored.

Here,'Fun: +7.0' may mean that +7.0 is given as an emotion score related to the strength (strength) of a positive word for the basic form (Lemma) corresponding to'Fun'. . In addition,'funeral: -6.0' means that -6.0 is given as an emotional score related to the intensity (strength) of negativity for the basic form of the word corresponding to'funeral'. In addition,'slipper: 0.0' means that 0.0 is given as an emotional score related to the strength of negative (or positive) for the basic form of the word corresponding to'slipper'. The example of the emotion score for each basic type of words is only an example for helping the understanding of the present application, but is not limited thereto.

In this case, the emotion score of 0.0 may mean that the corresponding word is not a word indicating positive or negative, but a word indicating a neutral emotion. In addition, an emotional score of + means that the corresponding word is a word indicating affirmation, and this may mean that the strength (intensity, degree) of the positive is greater as the number increases. Conversely, an emotional score of-means that the corresponding word is a word representing negation, and this may mean that the greater the number, the greater the intensity (intensity, degree) of negation.

The context analysis module 20 may include an emotion score module 24 as an example. The emotion score module 24 may obtain an emotion score for each of the basic types of matching words that match the basic type of words included in the emotion dictionary DB 23 among the extracted basic types of at least one word. Thereafter, the emotion score module 24 may calculate the second emotion analysis result d2 in consideration of the emotion score of each of the acquired matching word basic types. In particular, the emotion score module 24 may add up the emotion scores of each of the basic matching words, and calculate a second emotion analysis result d2 based on the summed emotion score.

For example, suppose that'Fun' and'travel' are extracted as at least one word basic form from the converted text. In this case, the emotion score module 24 matches the extracted at least one word basic type with the word basic type included in the emotion dictionary DB 23, and thus the matching word basic type that matches the word basic type'Fun'. '+7.0' can be obtained as an emotion score of and'+3.0' can be obtained as an emotion score of a matching word basic type that matches'Travel', which is a basic word type. Thereafter, the emotion score module 24 may add up the emotion score of each of the acquired matching word basic types, and the summed emotion score of +10.0 (+7.0+3.0=+10.0) is calculated as a result of the second emotion analysis (d2). ) Can be calculated.

At this time, according to the second emotion analysis result d2, as the summed emotion score is +10.0, the context analysis module 20 is the second emotion analysis result calculated by application of the context analysis based on the emotion dictionary 23 From (d2), it can be determined that an emotion corresponding to negative rather than positive appears in the input voice data (d). In other words, the context analysis module 20 may determine that the user's emotion for the voice data d based on the context analysis based on the emotion dictionary is'positive emotion' through the second emotion analysis result d2. have.

The context analysis module 20 determines that the user's emotion is'positive emotion' when the summed emotion score is + value, and determines that the user's emotion is'negative emotion' if the summed emotion score is- If the score is a value of 0, it may be determined that the user's emotion is a'neutral emotion'.

In other words, the context analysis module 20 includes a word belonging to the emotion dictionary DB 23 (a matching word basic type) among words in a sentence (that is, at least one word basic type extracted from the converted text). , It is possible to obtain an emotion score (ie, an emotion score related to the strength of positive or negative) regarding the presence or absence of a positive/negative relationship with respect to a corresponding word (a basic type of matching word). Thereafter, the context analysis module 20 calculates the emotion score (ie, the summed emotion score) for the entire sentence (ie, the entire sentence corresponding to the voice data) based on the obtained emotion score, as a result of the second emotion analysis (d2). ) Can be calculated. At this time, the size of the summed emotion score represents the strength of the positive/negative emotions.

The second emotion analysis result d2 calculated from the context analysis module 20 may be input to the integration module 30. The second emotion analysis result d2 may be used (used) to extract (generate) the integrated emotion analysis result d3 in the integrated module 30.

The integrated module 30 outputs the integrated emotion analysis result d3 for the voice data d by using the first emotion analysis result d1 and the second emotion analysis result d2 as inputs of the neural network 31. I can. Here, as the neural network 31, as in the neural network 12 described above, for example, Keras may be considered.

That is, the integration module 30 acquires the first emotion analysis result d1 calculated by the speech analysis module 10 and the second emotion analysis result d2 calculated by the context analysis module 20 to obtain a neural network ( 31) can be provided. The integrated module 10 uses the first emotion analysis result (d1) and the second emotion analysis result (d2) as inputs of the neural network 31 through an analysis based on the neural network 31 to provide an output of the neural network 31. An integrated emotion analysis result d3 in which the first emotion analysis result d1 and the second emotion analysis result d2 are integrated may be calculated.

At this time, the integration module 30 provides the first/second emotion analysis results d1 and d2 as well as the model data obtained from the local storage 32 as an input of the neural network 31. I can. That is, when performing an analysis based on the neural network 31 for calculating the integrated emotion analysis result d3, the integrated module 30 is The previous model data can be loaded and used (utilized) from the storage 32. The integration module 30 may calculate an integrated emotion analysis result d3 through the neural network 31 based analysis.

Here, the model data uses a plurality of first emotion analysis results and a plurality of second emotion analysis results as inputs of the neural network, and outputs an integrated emotion analysis result corresponding to the input first/second emotion analysis results d1 and d2. In a neural network that is trained to be learned, it may mean a learning value of a neural network (specifically, a weight value of a neural network determined through learning) determined by repeatedly learning the neural network. That is, the model data may mean a weight value of the trained neural network. The local storage 32 may refer to a database in which a plurality of such model data is stored.

The integration module 30 passes the first/second emotion analysis results d1 and d2 and the model data obtained from the local storage 32 through the neural network 31, thereby passing the integrated emotion analysis result d3 from the neural network 31. ) Can be calculated (output).

Here, the result of the integrated emotion analysis is primarily derived in the form of a value that is probabilistically given to each of a plurality of emotions including happiness, neutrality, sadness, and anger. It may be a result of determining an emotion having a maximum probability among a plurality of emotions as a final emotion.

In other words, the integration module 30 is the first emotion analysis result (d1) received from the voice analysis module 10 (a value that is probabilistically given to each of a plurality of emotions including happiness, neutrality, sadness, and anger, As an example, the probability values of four emotions) and the second emotion analysis result (d2) received from the context analysis module 20 (the value of the summed emotion score for each emotion score of the basic matching word, the strength of the positive/negative and By inputting the value of the relevant emotion score) into the neural network 32 of the integration module 30, the final emotion of the user corresponding to the voice data d is output as the integrated emotion analysis result d3, which is an output of the neural network 32. Can provide.

Here, in order to provide the output of the integrated emotion analysis result (d3), the integrated module 30, similarly to the first emotion analysis result extracted from the voice analysis module 10, is a plurality of emotions. It is possible to derive a value (weight value) that is probabilistically assigned to each emotion (Happy, Neutral, Sad, Angry). In this case, the emotion corresponding to the maximum value among the first-derived probability values for the four emotions may be determined as the final emotion of the user 1 corresponding to the voice data d. That is, the integration module 30 determines the emotion having the maximum probability value among the probability values (weight values) for each of the four emotions derived first as the final emotion of the user 1 corresponding to the voice data (d). , The determined final emotion may be output as an integrated emotion analysis result (d3).

The device 100 analyzes the user's emotion for the voice data (d) by fusion of the neural network-based voice analysis and the emotion dictionary-based context analysis, thereby simply analyzing the emotion based on the voice analysis or the context analysis. Compared to the emotion analysis technology, the accuracy of the user's emotion analysis can be effectively improved.

3 shows an example of an integrated emotion analysis result d3 derived by the neural network-based emotion analysis apparatus 100 according to an embodiment of the present application.

Referring to FIG. 3, when there is no anti-linguistic characteristic (eg, intonation, tone, etc.) in the voice data d, the voice analysis module 10 analyzes the voice data d. It is impossible to analyze (identify) the user's emotion by applying it (it cannot accurately derive the user's emotion).

Therefore, it is difficult to accurately determine what kind of emotion the user is from the voice data (d) based on the voice analysis by the voice analysis module 10 because there is no anti-linguistic feature in the voice data (d), but the voice data ( In the case where a word (emotional word) expressing emotion is included in the utterance of d) (in other words, if there is no special feature part for intonation, etc. in the utterance, but the emotion word is included), the integrated module 30 is the first By considering not only the emotion analysis result (d1) but also the second emotion analysis result (d2) calculated through the application of the context analysis, an integrated emotion analysis result (d3) representing the user's final emotion for the voice data (d) is calculated. Can provide.

Exemplarily, in relation to a case in which an anti-verbal characteristic (actonation, tone, etc.) does not exist in the speech of the voice data d input from the user, "Why is it so selfish?" as the voice data d. Let's say that the utterance of is entered

The speech analysis module 10 may calculate a probability (probability value) for each of a plurality of the plurality of data as the first emotion analysis result d1 by applying the speech analysis to the speech data d. According to this, when a single processing of speech analysis is applied to the input speech data (d), as a result, the probability corresponding to happiness among the probabilities for a plurality of emotions (4 emotions) has the largest value. According to this, an emotion result corresponding to'Happy' can be calculated. That is, with a single application of voice analysis, it can be calculated that the user's emotion is'Happy'.

Meanwhile, as a result of applying the context analysis to the voice data d, the context analysis module 20 may calculate an emotion value corresponding to -3.0 (negative emotion) as the second emotion analysis result d2. That is, when a single process of context analysis is applied to the input voice data d, an emotion result corresponding to "-3.0 (negative emotion)" may be calculated as a result. In other words, with a single application of context analysis, it can be calculated that the user's emotion is'neutral'.

The integration module 30 is based on the neural network 31 by considering the second emotion analysis result (d2),'-3.0 (negative emotion)' and the first emotion analysis result (d1), the'probability for each of the plurality of emotions' together. By performing a fusion (integration) analysis of, it is possible to calculate the probability for each of the plurality of emotions as the result d3 of the integrated emotion analysis on the voice data (d). The integrated module 30 is a result of the integrated analysis of the first/second emotion analysis results d1 and d2, and the probability corresponding to sadness is the largest (high value) among the probabilities for the four emotions. As indicated, the user's final emotion for the voice data d may be calculated as'Sad'. That is, the integrated module 30 may calculate the user's final emotion as'Sad', since the emotion having the maximum probability among the plurality of emotions is'Sad' through the integrated analysis.

4 shows another example of an integrated emotion analysis result d3 derived by the neural network-based emotion analysis apparatus 100 according to an embodiment of the present application.

Referring to FIG. 4, when there is no word (emotion word) representing the user's emotion in the voice data d, the context analysis based on the emotion dictionary (emotion dictionary DB, 23) is applied to the voice data d. You cannot analyze (find) the user's emotions just by doing it.

Therefore, since there is no word representing emotion in the voice data (d), based on the context analysis by the context analysis module 20, it is determined whether the user's emotion is a positive emotion or a negative emotion from the voice data (d). In a difficult case (in other words, when the second emotion analysis result for the voice data is 0.0, which is calculated as a neutral emotion), the integration module 30 performs not only the second emotion analysis result d2, but also the second emotion analysis result d2. 1 In consideration of the emotion analysis result d1 together, an integrated emotion analysis result d3 representing the user's final emotion with respect to the voice data d may be calculated and provided.

For example, in relation to the case in which a word (emotional word) representing emotion does not exist in the speech of the voice data (d) input from the user, "I'm done. I don't" as the voice data (d). Let's say you have entered an utterance saying "want to play."

As a result of applying the context analysis to the voice data d, the context analysis module 20 may calculate an emotion value corresponding to 0.0 (neutral emotion) as the second emotion analysis result d2. That is, when a single process of context analysis is applied to the input voice data d, an emotion result corresponding to "0.0(-) (neutral emotion)" may be calculated as a result. In other words, with a single application of context analysis, it can be calculated that the user's emotion is'neutral'.

Meanwhile, the speech analysis module 10 may calculate a probability (probability value) for each of a plurality of the plurality of data as the first emotion analysis result d1 by applying the speech analysis to the speech data d. According to this, when a single processing of speech analysis is applied to the input speech data (d), as a result, the probability corresponding to anger is the largest value (0.00740) among the probabilities for a plurality of emotions (4 emotions). ), the emotional result corresponding to'Angry' can be calculated. That is, with a single application of voice analysis, it can be calculated that the user's emotion is'Angry'.

The integration module 30 considers '0.0 (neutral emotion)' as the second emotion analysis result (d2) and the'probability for each of the plurality of emotions' as the first emotion analysis result (d1). By performing the fusion (integrated) analysis, the probability for each of the plurality of emotions can be calculated as the result d3 of the integrated emotion analysis for the voice data d. The integrated module 30 is a result of the integrated analysis of the first/second emotion analysis results (d1, d2), and the probability corresponding to anger is the largest (high value) among the probabilities for the four emotions. As it appears, the user's final emotion for the voice data d can be calculated as'Angry'. That is, the integrated module 30 may calculate the user's final emotion as'anger' because the emotion having the maximum probability among the plurality of emotions is'anger' through the integrated analysis.

According to this, the present apparatus 100 combines two types of analysis (context analysis and speech analysis) techniques, compared to calculating the user's emotion by applying only one analysis of context analysis and speech analysis as a single process. By calculating the user's emotion in consideration, it is possible to effectively increase the accuracy of the calculation (estimation) of the user's emotion with respect to the voice data d.

That is, when the emotion analysis based on the emotion dictionary is difficult because the emotion word does not exist in the speech, the present apparatus 100 may improve the accuracy of emotion estimation by using the voice analysis together. In addition, the present apparatus 100 may increase the accuracy of emotion estimation by using the emotion dictionary-based context analysis together when there is no special part in the intonation of the spoken sentence, but the emotion word is included.

The integration module 30 may provide an integrated emotion analysis result d3 calculated based on the integrated analysis to the mobile application 2a of the user terminal 2 (providing an output). Through this, the integrated emotion analysis result d3 may be displayed (displayed) on the screen of the user terminal 2. The user may receive and check the integrated emotion analysis result d3 output from the device 100 through the screen of the user terminal 2.

5 is a diagram showing a schematic configuration of an emotion treatment system 200 ′ based on emotion analysis based on a neural network according to an embodiment of the present application. 6 is a diagram showing a schematic configuration of an emotion treatment apparatus 110 based on emotion analysis based on a neural network according to an embodiment of the present application.

Hereinafter, for convenience of description, the emotion treatment system 200 ′ based on a neural network based emotion analysis according to an exemplary embodiment of the present application is referred to as the present system 200 ′, and emotion treatment based on a neural network based emotion analysis according to an exemplary embodiment of the present application The device 110 will be referred to as the device 110. In addition, emotional therapy herein may be expressed differently as emotional therapy.

The system 200 ′ may be the same system as the system 200 (a neural network-based emotion analysis system according to an embodiment of the present disclosure) described above. In addition, the device 110 may be the same device as the device 100 described above (a neural network-based emotion analysis device according to an embodiment of the present disclosure). That is, the present system 200 ′ and the present device 110 differ only in that the configuration of the treatment module 40 is added compared to the present system 200 and the present device 100 described above. The configurations and their functions may be the same.

Accordingly, even if the contents are omitted below, the contents described with respect to the system 200 may be equally applied to the description of the system 200 ′. In addition, even if the contents are omitted below, the contents described with respect to the apparatus 100 may be equally applied to the description of the apparatus 110.

5 and 6, the system 200 ′ may include a user terminal 2 and the device 110. The device 110 may include a speech analysis module 10, a context analysis module 20, an integration module 30, and a treatment module 40. Here, since the description of the user terminal 2, the voice analysis module 10, the context analysis module 20, and the integration module 30 has been described in detail above, redundant descriptions will be omitted below.

The device 110 determines and provides an emotion treatment method (d4) based on the integrated emotion analysis result (d3) provided by performing a neural network-based emotion analysis by the device 100 (neural network-based emotion analysis device) described above. I can. A detailed description is as follows.

The integration module 30 calculates an integrated emotion analysis result d3 for the voice data d by using the first emotion analysis result d1 and the second emotion analysis result d2 as inputs of the neural network 31, The calculated integrated emotion analysis result d3 may be transmitted to the treatment module 40.

The treatment module 40 may receive and obtain the integrated emotion analysis result d3 calculated by the integration module 30 from the integration module 30.

The treatment module 40 may determine an emotion treatment method (d4) based on the obtained integrated emotion analysis result (d3) (treatment method determination, 41). That is, the treatment module 40 identifies the user's final emotion based on the integrated emotion analysis result (d3), and determines which emotion treatment method (treatment, d4) to provide to the user 1 according to the determined final emotion. I can.

Here, the type of emotional therapy (therapeutic method, d4) may include art therapy, music therapy, dance/movement therapy, etc., but is not limited thereto, and the device 110 can treat the user's emotion (sensibility). Various emotional treatment methods can be applied.

The emotion treatment method (d4) determined in the treatment module 40 shows a change in the emotion of the user 1 to a positive emotion when the user's emotion is shown as a negative emotion or a neutral emotion as a result of the integrated emotion analysis (d3). It can be provided in the form of inducing content. In addition, the emotion treatment method d4 may be provided in the form of content that induces positive maintenance and expansion of the user 1's emotion when the emotion of the user 1 is found to be a positive emotion as a result of the integrated emotion analysis.

Here, as an exemplary integrated emotion analysis result (d3), the fact that the user's emotions were negative emotions is among a plurality of emotions including Happy, Neutral, Sad, and Angry. This may mean a case in which the final emotion is determined as Sad or Angry. In addition, the fact that the user's emotion appears as a neutral emotion as a result of the integrated emotion analysis d3 may mean a case in which the final emotion among the plurality of emotions is determined to be neutral. In addition, the fact that the user's emotion appears as a positive emotion as a result of the integrated emotion analysis d3 may mean a case in which the final emotion among the plurality of emotions is determined as Happy.

Therefore, when the user's emotion is expressed as a negative emotion or a neutral emotion as a result of the integrated emotion analysis (d3) (that is, when the final emotion is determined to be one of neutral, sad, and anger), the user's current assessor An emotion treatment method d4 may be provided in the form of content that induces any one of neutrality, sadness, and anger to change into a positive emotion, a happy emotion.

In addition, the treatment module 40 maintains a positive emotion corresponding to the user's current emotion, happiness, when the user's emotion appears as a positive emotion as a result of the integrated emotion analysis (d3) (that is, when the final emotion is determined as happiness). Emotion therapy (d4) may be provided in the form of content that induces expansion or expansion.

As an example, the treatment module 40 may provide an art treatment method as a content type emotion treatment method that induces a change in a user's emotion into a positive emotion. In addition, the treatment module 40 may provide at least one of a movie, a music, and a book as an emotional treatment method in the form of content that induces the maintenance and expansion of positive emotions with respect to the user's emotions.

When the emotion treatment method (d4) is determined based on the obtained integrated emotion analysis result (d3) (the treatment method determination, 41), the treatment module 40 is determined so that the user can perform the emotion treatment based on the determined emotion treatment method. The emotion treatment method d4 may be provided in a form that enables the user to perform emotion treatment 42 and is provided as a mobile application 2a of the user terminal 2.

At this time, the treatment module 40 applies the treatment information obtained from the local storage 43 to the emotion treatment method determined based on the integrated emotion analysis result d3, so that the determined emotion treatment method d4 is applied to the emotion treatment method by the user ( It may be provided in a form that enables the performance 42 of the emotional therapy method). The treatment module 40 may provide the emotion treatment method d4 provided to enable the execution 42 of the emotion treatment method to the user terminal 2.

In the local storage 43, for each of the plurality of emotions, at least one treatment information corresponding to each emotion may be previously stored. Here, the treatment information may mean content information. For example, the treatment information may include information on a plurality of contents related to art pictures that can be provided when the final emotion is sadness, and information on a plurality of music related contents that can be provided when the final emotion is happiness.

For example, assume that an art therapy method is determined (41) as an emotional therapy method based on the integrated emotion analysis result (d3). At this time, the treatment module 40 enables treatment of the final emotion among a plurality of art drawings (for example, treatment of inducing a negative emotion, sadness, into a positive emotion, happiness, as treatment information from the local storage 43). You can acquire any one of the art paintings. Thereafter, the treatment module 40 prepares the determined emotion treatment method in a form that enables the user to perform the emotion treatment 42 based on the art picture corresponding to the treatment information obtained from the local storage 42, and the user terminal ( 2) can be provided.

As an example, the treatment module 40 may provide the art picture corresponding to the obtained treatment information in the form of a paint plate so that the user can see and follow it and provide it to the user terminal 2. As an example, according to the emotion treatment method d4 provided to the user terminal 2 by the treatment module 40, art corresponding to the emotion treatment method determined by the treatment module 40 in a region on the screen of the user terminal 2 A picture (i.e., an art picture obtained from local storage) is displayed, and in the remaining areas on the screen of the user terminal 2 except for the above one area, a paint plate provided so that the user can see and draw the corresponding art picture is displayed (provided ) Can be.

The screen of the user terminal 2 on which the integrated emotion analysis result (d3) and/or the emotion treatment method (d4) is displayed (provided) is, for example, the integrated module 30 and/or the treatment module ( Can be controlled by 40).

In other words, the integrated emotion analysis result (d3) and/or the emotion treatment method (d4) provided by the devices 110 and 110 may be displayed on the screen of the user terminal 2 through the mobile application 2a. have. To this end, the integrated module 30 and/or the treatment module 40 of the device 100 and 110 displays an integrated emotion analysis result (d3) and/or an emotion treatment method (d4) on the screen of the user terminal 2 The mobile application 2a can be controlled as much as possible.

In this way, the treatment module 40 provides the emotion treatment method (d4) determined in response to the integrated emotion analysis result (d3) to the user terminal 2, so that the user can use the emotion treatment method (d4) provided by the device 100 Emotional therapy can be performed based on.

FIG. 7 is a diagram for describing an example in which emotion treatment of a user is performed by the emotion treatment apparatus 110 based on emotion analysis based on a neural network according to an exemplary embodiment of the present disclosure. In particular, FIG. 7 is a diagram showing an example of a screen of the user terminal 2 displayed by the device 110 controlling the mobile application 2a to perform the user's emotion analysis and emotional treatment.

Referring to FIG. 7, for emotional treatment based on a neural network-based emotion analysis by the present apparatuses 100 and 110, according to the mobile application 2a, a'start' menu on the main screen of the user terminal 2 ,'List menu, etc. may be provided.

When a user input is made to the'Start' menu displayed on the screen of the user terminal 2, a voice data input button that enables input (recording) of the voice data d from the user 1 is displayed on the screen. Can be provided. The user may input a voice data input button (for example, a touch input, etc.) and then utter a voice for which emotion analysis is to be performed. The mobile application 2a of the user terminal 2 may provide voice data d corresponding to the utterance input from the user to the devices 100 and 110. The devices 100 and 110 may analyze a user's emotion (current emotion) (an emotion corresponding to the voice data) by applying voice analysis and context analysis to the input voice data d.

Thereafter, in response to the input voice data d, the devices 100 and 110 may calculate an integrated emotion analysis result d3 representing the user's final emotion as an analysis result. In addition, the apparatuses 100 and 110 may determine an emotion treatment method d4 based on the calculated integrated emotion analysis result d3 and provide the determined emotion treatment method d4 as the mobile application 2a.

For example, when the analyzed final emotion of the user is sad, the device 100 and 110 is an emotion treatment method d4 so that the negative emotion of the user (that is, the sad emotion) is induced to change into a positive emotion. Contents (for example, art pictures according to art therapy) can be provided to the mobile application 2a. Accordingly, an art picture for performing the art therapy method as the emotion therapy method d4 may be displayed (displayed) on the screen of the user terminal 2. In this case, the apparatuses 100 and 110 may provide not only an art drawing, but also a treatment execution mode through the mobile application 2a that enables the user 1 to perform emotional therapy using the corresponding art drawing. In this case, the treatment execution mode may mean, for example, a paint plate that is provided so that a user can see and follow a corresponding art picture. That is, the mobile application 2a may display a painting plate for art therapy on the screen of the user terminal 2 as a treatment mode for art therapy as an example.

According to this, the mobile application (2a) is a result of the user's emotion analysis on the input voice data (d), an emotional treatment method (d4) and an integrated emotion analysis result (d3) including information on an art picture or treatment mode. Can be displayed (provided) on the screen of the user terminal 2.

As another example, when the determined emotional therapy method d4 is a dance/motion therapy, a video or the like may be displayed on the screen of the user terminal 2 to allow the user to act according to a given movement as a therapy performance mode.

FIG. 8 is a diagram schematically illustrating a menu configuration in a mobile application 2a interlocked with the emotion treatment apparatus 110 based on emotion analysis based on a neural network according to an exemplary embodiment of the present disclosure. In FIG. 8, a case in which art therapy is considered as an emotional therapy method will be described.

Referring to FIG. 8, the mobile application 2a is a menu that can be displayed on the screen of the user terminal 2 and may include, for example, a recording menu and a history list menu.

The recording menu may include an art therapy example menu and an art therapy paint menu.

In the art therapy example menu, for example, when a user performs emotion therapy by the devices 100 and 110, information related to the emotion treatment method d4 that is provided from the devices 100 and 110 may be stored. That is, in the art therapy example menu, example pictures of art paintings for allowing the user to perform art therapy in response to the art therapy method may be stored. By considering (reference) example pictures of a plurality of art pictures stored in the art therapy example menu, the user himself can perform art therapy tasks such as drawing pictures. That is, an example of the display on the screen of the user terminal corresponding to the art therapy example menu may be the same as the example of the'pre-art therapy example picture example screen' shown in FIG. 7.

The art therapy paint menu provides a menu (i.e., a therapy execution mode) that can provide a screen that allows a user to draw a picture that actually performs an art therapy method determined according to the final emotion analyzed by the device (100, 110). It can mean a menu). That is, an example of the display on the screen of the user terminal corresponding to the art therapy paint may be the same as the example of the'user paint screen for art therapy' shown in FIG. 7. Based on the treatment mode provided through the art treatment paint menu, the user can select a color of a pen and perform an action such as drawing a picture according to the manual of the art treatment method, and art treatment is performed accordingly. I can.

The history list menu may include, for example, four menus including a date history menu, a mood history menu, a voice history menu, and a picture history menu. In this history list menu, history information (date, mood, voice) generated by the user performing emotional treatment based on data provided by the devices 100 and 110 (for example, integrated emotion analysis result or emotion treatment method) And picture related history information) may be stored. Here, the term “mood” may mean an emotion.

The history list menu has four menu functions (date, mood, voice, picture), and each function is the date and time when the user previously used the mobile application 2a linked to the devices 100 and 110. You can keep your input mood, voice, and picture.

The user can check the user's previous emotional state through the mood history menu. The user can check the mood at the time by opening the voice data (voice file) input (recorded) at the time of performing emotion analysis or emotional treatment through the voice history menu. Through the picture history menu, the user can check the picture, which is the treatment result (for example, the result of art therapy) for the emotion treatment method that was performed to induce a positive change in the mood at the time of performing emotion analysis or emotional treatment. .

The present apparatuses 100 and 110 combine the input voice data d with a neural network (for example, a CNN neural network)-based voice analysis and an emotion dictionary (sense word dictionary)-based context analysis to determine the voice data (d). User's emotion analysis can be performed. The devices 110 and 110 provide an emotion treatment method (emotional treatment technique) to the user 1 based on this fusion-based emotion analysis result (integrated emotion analysis result, d3), that is, the user can perform emotion treatment. You can do it.

The devices 100 and 110 may provide an art therapy method as an emotional therapy method. According to the emotion treatment method provided by the devices 100 and 110, it is possible to induce a user's negative emotion (Sad, Angry) or neutral emotion (Neutral) to change into a positive emotion. In addition, according to the emotion treatment method provided by the devices 110 and 110, it is possible to induce a user's positive emotion (Happy) to be maintained and expanded. That is, the emotion treatment method provided by the devices 100 and 110 may play a role of inducing a user's emotion to be changed more positively or to maintain and expand a positive emotion.

When the voice data d is input, the devices 100 and 110 may provide the input voice data d to the voice analysis module 10 and the context analysis module 20, respectively. The speech analysis module 10 calculates a probability value for each of the four emotions (Happy, Neutral, Sad, and Angry) as the first emotion analysis result (d1) through the processing of speech analysis on the speech data (d). can do. The context analysis module 20 performs a context analysis on the voice data (d), through which the emotion score related to the strength of the positive and the negative strength (i.e., the positive/negative strength-related emotion score, one positive/negative result) Value) may be calculated as the second emotion analysis result d2. Thereafter, the integrated module 30 may calculate an integrated emotion analysis result d3 including the user's final emotion by inputting the first and second emotion analysis results d1 and d2 as an input. In other words, based on the emotion data obtained from the voice analysis module 10 and the positive/negative result values obtained from the context analysis module 20, the integration module 30 combines the user's final emotion as an integrated emotion analysis result (d3). It can be calculated (extracted) as. The treatment module 40 determines the emotion treatment method d4 based on the calculated integrated emotion analysis result d3, and the mobile application 2a so that the determined emotion treatment method d4 is displayed on the screen of the user terminal 2 Emotional therapy (d4) determined as can be provided. Through this, the user 10 may perform emotional treatment based on the emotion treatment method d4 provided on the screen of the user terminal 2.

9 is a view for explaining an example of calculating a second emotion analysis result d2 by the context analysis module 20 in the emotion treatment apparatus 110 based on emotion analysis based on a neural network according to an exemplary embodiment of the present disclosure.

Referring to FIG. 9, as an example, assume that the input voice data d is'When you have meet so many kind and wonderful Stepmothers.'

The context analysis module 20 may convert the input voice data d into text through STT (Speech To Text, 21). Thereafter, the context analysis module 20 may extract a word Lemma with respect to the converted text using the Google Natural Language 22. For example, the context analysis module 20 may extract [I, have, met, so, many, kind, and, wonderful, Stepmothers] as at least one basic word type from the voice data d.

Thereafter, the context analysis module 20 performs matching with the word basic type included in the emotion dictionary DB (emotion word dictionary DB, 23) with respect to the extracted at least one basic type of word, thereby determining whether there is a matching word basic type. I can confirm. The context analysis module 20 may obtain an emotion score for each matching word basic type by performing matching between the extracted word basic type and the word basic type included in the emotion dictionary DB 23.

In other words, the context analysis module 20 extracts and extracts a word Lemma of a word representing emotion included in the converted text from the converted text through the application of the Google Natural Language 22. A pre-stored index can be obtained for a matching word basic type that matches the word basic type extracted from the emotion dictionary DB 23 through matching between the generated word basic type and the emotion dictionary DB 23. In this case, the context analysis module 20 may obtain an emotion score previously assigned to the basic type of matching word based on the obtained index. In this case, the emotion word DB 23 may be stored in the form of a lookup table in association with information corresponding to a basic word type, an index, and an emotion score.

For example, data such as'kind (index 3): +1.0' and'wonderful (index 9): +2.0' may be previously stored in the emotion dictionary DB 23. According to this, the context analysis module 20 uses'+3.0', which is the emotional score of kind, which is the basic type of matching word, and'+9.0', which is the emotional score of wonderful, which is the basic type of matching word, as an emotion score for each basic matching word type. It can be obtained from (23).

Thereafter, the context analysis module 20 may calculate the summed emotion score as the second emotion analysis result d2 by summing the emotion scores of matching words corresponding to each of the extracted at least one basic word type. For example, the summed emotion score (the sum of emotion scores, Sentiment Score), that is, the emotion score for the entire sentence corresponding to the voice data (d) is +3.0 (+1.0+2.0=+3.0) (positive emotion value). I can. According to this, when context analysis is performed on the voice data d, the user's emotion may be calculated as a'positive emotion'.

10A to 10C are diagrams for explaining an example of calculating the first emotion analysis result d1 by the voice analysis module 10 in the emotion treatment apparatus 110 based on emotion analysis based on a neural network according to an exemplary embodiment of the present disclosure to be.

In particular, FIG. 10A shows an example of a speech waveform characteristic (ie, MFCC characteristic value) extracted by applying the MFCC transformation 11 to speech data d. 10B shows an example of a learning curve of model data obtained from the local storage 13. 10C shows an example of the first emotion analysis result d1.

Referring to FIGS. 10A to 10C, as an example, assume that the input voice data d is'When you have meet so many kind and wonderful Stepmothers.'

The voice analysis module 10 may extract (detect, acquire) voice waveform features (ie, MFCC feature values) from voice data d by applying MFCC conversion 11 to voice data d. For example, the speech analysis module 10 may extract 26 speech waveform features from the speech data d. The speech analysis module 10 may provide the extracted speech waveform characteristics as an input of the neural network 12.

In addition, the speech analysis module 10 may obtain (load) model data from the local storage 13 and provide it as an input of the neural network 12. Here, the model data stored in the local storage 13 may mean a neural network model for speech classification that is learned to classify speech from MFCC feature values, or a weight value of a neural network determined through learning about a corresponding neural network model.

As an example, the model data may mean data related to a neural network model (eg, a weight value) obtained by learning MFCC feature values of 2000 Tess voice data. In FIG. 10B, a blue curve indicates accuracy, and model data applied to the devices 100 and 110 may be model data showing 98% learning accuracy, for example.

The speech analysis module 10 applies the characteristics of the speech waveform extracted from the speech data d and the model data obtained from the local storage 13 as inputs of the neural network 12, thereby performing a first analysis as a result of the neural network 12. The analysis result (d1) can be calculated.

In the first emotion analysis result d1 calculated by the voice analysis module 10, each of a plurality of emotions including happiness, neutrality, sadness, and anger is probabilistically It may contain a given value (probability value). In FIG. 10C, as an example, as a result of application of the speech analysis to the voice data (d), the first emotion analysis result d1 is shown so that the emotion corresponding to'Happy' among the four emotions has the largest value. It can be confirmed that it was calculated. According to this, when the voice analysis is performed on the voice data d, it may be calculated that the user's emotion is'Happy'.

The result of applying the speech analysis to the speech data d can be applied (adjusted) so that the sum becomes 1 using a Softmax function in the final step. According to this, the emotion analysis result (that is, the first emotion analysis result, d1) shown in FIG. 10C is a result of the application of the speech analysis to the voice data (d), which exemplarily refers to a value before the softmax function is applied. can do.

11 is a view for explaining an example of calculating an integrated emotion analysis result d3 by the integration module 30 in the emotion treatment apparatus 110 based on a neural network based emotion analysis according to an exemplary embodiment of the present disclosure.

Referring to FIG. 11, the integration module 30 provides the first emotion analysis result d1 and the second emotion analysis result d2 as inputs of the neural network 31 to perform fusion (integration) analysis, As a result of the integrated emotion analysis for (d) (d3), the probability for each of the plurality of emotions can be calculated.

The integrated module 30 is a result of the integrated analysis of the first and second emotion analysis results d1 and d2, and the probability corresponding to happiness among the four emotions is the highest (high value). As indicated, the user's final emotion for the voice data d can be calculated as'Happy'. That is, the integrated module 30 may calculate the user's final emotion as'Happy' because the emotion having the maximum probability among the plurality of emotions is'Happy' through the integrated analysis. The integrated module 30 may calculate an integrated emotion analysis result d3 as a result of an integrated analysis (integrated analysis performed with voice analysis and context analysis) in which'Happy' is determined as the final emotion of the user.

As an example, the integrated analysis emotion value shown in FIG. 11 is related to the integrated emotion analysis result d3 calculated by the integrated module 30, Happy, Neutral, Sad, and Anger ( It may mean a final result value for each of a plurality of emotions including (Angry). This is the result of application of the speech analysis to the speech data (d) calculated by the speech analysis module 10 whose sum is adjusted to 1 using the softmax function (i.e., the first emotion analysis result, d1) and context analysis. It may mean a final result value obtained by secondary processing the second emotion analysis result d2, which is the natural language analysis result calculated by the module 20, together. The sum of the final result values for each of the plurality of emotions may or may not be 1.

The treatment module 40 may determine an emotion treatment method d4 based on the integrated emotion analysis result d3 and provide it to the user terminal 2. For example, if the user's final emotion included in the integrated emotion analysis result (d3) is found to be a positive emotion as'Happy', the treatment module 40 is in the form of content that induces maintenance and expansion of the positive emotion. Emotional therapy (d4) can be provided. As an example, the treatment module 40 may provide a paint plate to enable emotional treatment for'drawing a happy figure' as an emotion treatment method d4.

The apparatuses 100 and 110 receive each of the emotion scores d1 and d2 through the text-based context analysis module 20 and the neural network-based speech analysis module 10 based on the voice data d of the user 1. After extraction, by integrating this in the integration module 30, the user's final emotion (final emotion) can be calculated (derived) as the analysis result d3.

The apparatuses 100 and 110 may provide an emotion treatment method d4 according to the result d3 of the integrated analysis. The present apparatuses 100 and 110 may provide an emotional treatment method d4 for art therapy or content recommendation (providing content such as movies, music, books, etc.) to the user terminal 2 according to the integrated analysis result d3. . Through this, the devices 100 and 110 may solve problems related to the user's stress and emotions through the provision of the emotion treatment method d4.

For example, when the user's final emotion (current final emotion) is'Sad', the devices 100 and 110 change the sad emotion, which is the final emotion, into a positive emotion (for example, a happy emotion). As an emotional therapy method (d4) that can be made, an art therapy method may be provided as an example.

The devices 100 and 110 may analyze a user's emotion by fusing voice analysis and context analysis. In the emotion analysis method using a single speech analysis, it is difficult to recognize a neutral intonation, and even for the same sentence, a problem may arise in that emotions are determined differently according to an individual's anti-linguistic characteristics.

Conventionally, several studies have been conducted in relation to the technology for analyzing emotions through voice, but there is a problem of showing relatively low accuracy, mostly with an accuracy of 60 to 70%. In addition, most of the conventionally known emotion analysis techniques only perform emotion analysis using data such as images, and there is no technology for performing emotion analysis by fusion of context analysis and voice analysis. In addition, conventional natural language processing-based emotion analysis techniques have a problem in that accurate emotion recognition cannot be achieved when input voice data does not contain words representing emotions (emotional words, emotional words).

Specifically, the conventional emotion analysis technology using voice analysis alone has the following problems. It is difficult to recognize the case where an emotional sentence is spoken with a neutral accent, and because the sentence is the same, the emotional analysis result may be determined (derived) differently due to individual differences.

One of the conventional techniques for recognizing human emotions is [Yelin Kim and Emily Mower Provost. 2015. Emotion Recognition During Speech Using Dynamics of Multiple Regions of the Face. ACM Trans. Multimedia Comput. Commun. Appl. 12, 1s, Article 25 (October 2015), 23 pages.] There is a literature.

The above-described conventional literature proposes an extension technology of the Deep Belief Network, and differs in that it does not use a neural network (deep learning) technology, unlike the technology proposed herein. In addition, the emotion analysis of the conventional literature is different in that it shows a low accuracy with an accuracy of 60 to 70%, and the emotion analysis is performed based on an image of a face.

Meanwhile, the conventional emotion analysis technology based on language processing has the following problems. Emotion recognition is difficult if the voice data obtained from the user does not contain a word (emotion word) representing emotion. In addition, even if the acquired voice data does not contain an appraisal word, an emotional sentence may appear in the acquired voice data due to inter-word dependency or voice characteristics. have.

12 is a diagram showing an example of a screen display of an emotion analysis result to which a conventional natural language processing is applied to given voice data. 13 illustrates an example of a screen display of an emotion analysis result through application of a context analysis by a context analysis module 20 of a neural network-based emotion analysis-based emotion treatment apparatus 110 according to an embodiment of the present application with respect to given voice data. It is a figure shown.

Referring to FIGS. 12 and 13, the emotion analysis result shown in FIG. 12 shows an example of applying Google's Google Natural Language (natural language processing) as a conventional natural language processing to voice data. On the other hand, the emotion analysis result shown in FIG. 13 is an emotion analysis technology that improves the conventional emotion analysis technology by using the Dependency Tree (dependency tree) of the conventional Google Natural Language (natural language processing) and the formula relation structure for voice data. It shows an example of the case where it is applied (that is, the case where the context analysis technology by this device is applied).

According to the description below, the improved emotion analysis technology applied to the present application (that is, the emotion analysis technology based on context analysis by the context analysis module of the device) simply performs emotion analysis using Google Natural Language, a conventional natural language processing technology. It can be seen that it shows better performance than that.

Specifically, FIGS. 12 and 13 are voice data that is subject to emotion analysis, and "I can't be happy sometimes not being with my family makes me feel so lonely (I can't be happy. I can't be with my family). It shows the result of sentiment analysis when the sentence "The facts sometimes make me too lonely" is considered.

Since the expressions "can't be happy" and "lonely" are included in an example of the given voice data, it may be preferable that the given voice data is determined to be an overall negative sentence. That is, according to an example of the given voice data, it may be desirable to determine that the user's emotion with respect to the given voice data corresponds to a negative emotion.

However, referring to FIG. 12, in the conventional Google Natural Language, an emotion score is given as 0.2 as a result of performing emotion analysis on given voice data, and accordingly, it is determined that the user's emotion on the voice data is a'neutral emotion'. can confirm. It can be seen that the accuracy of the emotion analysis result based on such conventional natural language processing is inferior.

On the other hand, referring to FIG. 13, in the context analysis module 20 of the present apparatuses 100 and 110, the emotion score is -2.5 as the result of performing the emotion analysis on the given voice data (ie, the second emotion analysis result, d2). And, accordingly, it can be confirmed that the user's emotion with respect to the voice data is determined to be a'negative emotion'. That is, the apparatuses 100 and 110 may derive a result that the sentence of the given voice data generally contains negative emotions through the application of the context analysis by the context analysis module 20.

It can be seen that the result of the sentiment analysis based on context analysis by the context analysis module 20 of the present devices 100 and 110 yields more accurate results compared to the conventional self-language processing (for example, Google Natural Language) technology. .

The present application may provide a neural network-based emotion analysis and emotion treatment system 200 and 200 ′. The present systems 200 and 200' can analyze the user's emotions for the input voice data by fusion of the voice analysis and the context analysis based on the emotion dictionary, and provide appropriate emotion treatment methods and contents to the user according to the analysis result. . These systems 200 and 200 ′ may be referred to as a Neural Network Based Sentiment Analysis and Therapy system (NBST).

The device (100, 110) and the system (200, 200') perform emotion analysis by fusion of voice analysis and context analysis based on emotion dictionary (emotional language dictionary) for the input voice data, and then based on the analysis result. As a result, it is possible to provide treatment techniques (emotional treatment) to users. As an emotion treatment method, an art treatment method may be considered as an example, and the negative emotion of the user may be positively changed through the provision of such emotion treatment method.

The apparatuses 100 and 110 may help a user to perform an emotional treatment corresponding to the determined emotional treatment method d4 by providing the emotion treatment method d4 determined by the treatment module 40 to the mobile application 2a.

In other words, the emotion treatment method d4 such as an art treatment method considered in the present systems 200 and 200 ′ may serve as a role of positively changing the negative emotion of the user 1. The apparatuses 100 and 110 may provide an emotion treatment method d4 for inducing a negative emotion or a neutral emotion to be changed into a positive emotion when the final emotion of the user is found to be a negative emotion or a neutral emotion as a result of the integrated analysis. In addition, if the integrated analysis result shows that the user's final emotion is positive emotion, the device 100 and 110 provides an emotion treatment method (d4) for contents such as movies, music, and books to the user, thereby providing the user's current emotion positive. Emotions can be maintained and/or expanded.

Conventionally, most mobile applications utilize text-based emotion analysis, and none of these mobile applications have an element that causes a user to change emotions. In other words, most conventional mobile applications that perform emotion analysis do not consider a technology for changing a user's emotion change at all. On the other hand, the system 200, 200' combines voice analysis and context analysis to analyze the user's emotions in a time-sequential manner to identify changes in the user's emotions, and provide emotional treatment methods such as art therapy according to the results. can do. The present system 200 and 200 ′ may change the user's emotion into a positive emotion when the current emotion of the user is expressed as a negative emotion or a neutral emotion through the provision of an emotion treatment method.

In other words, the system 200 and 200 ′ can grasp the user's emotion change by fusion of the voice analysis and the context analysis to identify the user's emotion for the input voice data and accumulate it and analyze it in a time series. Based on this result, the system 200 and 200 ′ may recommend (provide) an emotional treatment method (treatment technique) and content to the user.

The system 200 and 200 ′ may change into a positive emotion when the user currently shows a negative emotion or a neutral emotion as a result of the integrated emotion analysis. In addition, the system 200, 200' may recommend and provide contents such as movies, music, and books when the user currently shows positive emotions as a result of the integrated emotion analysis (that is, for users showing positive emotions). Through this, the user's current emotion, positive emotion, can be maintained or further expanded.

The systems 200 and 200' can be applied to Market Research, which occupies the largest portion of the global emotional (emotional) computing market. In addition, the systems 200 and 200 ′ can be directly used in the field of health care. In addition, the systems 200 and 200 ′ can be applied to the psychotherapy market, and emotional therapy can be provided through personal mobile applications.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly described.

14 is a flowchart illustrating a method for analyzing emotions based on a neural network according to an embodiment of the present application.

The neural network-based emotion analysis method illustrated in FIG. 14 may be performed by the apparatus 100 described above. Accordingly, even if the contents are omitted below, the contents described with respect to the apparatus 100 may be equally applied to the description of the neural network-based emotion analysis method.

Referring to FIG. 14, in step S11, the voice analysis module may calculate a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user.

At this time, in step S11, the voice analysis module may calculate the first emotion analysis result through a neural network-based voice analysis that inputs the voice waveform feature extracted from the voice data.

Here, the voice waveform feature may be an MFCC feature value output by performing MFCC (Mel-Frequency Cepstral Coefficient) transformation on voice data.

In addition, the first emotion analysis result may be a value probabilistically given to each of a plurality of emotions including happiness, neutrality, sadness, and anger.

Next, in step S12, the context analysis module may calculate a second emotion analysis result by applying the emotion dictionary-based context analysis to the voice data input from the user.

At this time, in step S12, the context analysis module converts the voice data into text, and then extracts at least one word basic type (Lemma) from the converted text, and converts the extracted at least one basic word type into the word basic type included in the emotion dictionary DB. By matching with, the second emotion analysis result may be calculated.

Here, the emotion dictionary DB may be constructed in a form in which emotion scores related to the strength of the positive and the strength of the negative are given to each of the plurality of basic types of words.

In step S12, the context analysis module may calculate the second emotion analysis result by considering the emotion score of each of the basic types of matching words that match the basic types of words included in the emotion dictionary DB among the extracted basic types of at least one word.

Next, in step S13, the integration module may output and provide an integrated emotion analysis result for voice data by using the first emotion analysis result calculated in step S11 and the second emotion analysis result calculated in step S12 as inputs of the neural network. have.

Here, the result of the integrated emotion analysis is derived primarily in the form of a value that is probabilistically given to each of a plurality of emotions including Happy, Neutral, Sad, and Angry. It may be a result of determining the emotion having the maximum probability among the emotions of the final emotion.

In the above description, steps S11 to S13 may be further divided into additional steps or may be combined into fewer steps, depending on the embodiment of the present application. In addition, some steps may be omitted as necessary, or the order between steps may be changed.

15 is a flowchart illustrating an operation of an emotion treatment method based on an emotion analysis based on a neural network according to an exemplary embodiment of the present application.

The emotion treatment method based on emotion analysis based on a neural network illustrated in FIG. 15 may be performed by the apparatus 110 described above. Therefore, even if omitted below, the description of the device 110 may be equally applied to a description of an emotion treatment method based on a neural network-based emotion analysis.

Each of steps S21 to S23 in FIG. 15 may be the same as each of steps S11 to S23 in FIG. 14 described above. Therefore, hereinafter, it will be briefly described in connection with steps S21 to S23.

Referring to FIG. 15, in step S21, the voice analysis module may calculate a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user. Next, in step S22, the context analysis module may calculate a second emotion analysis result by applying the emotion dictionary-based context analysis to the voice data input from the user. Next, in step S23, the integration module may output and provide an integrated emotion analysis result for voice data by using the first emotion analysis result calculated in step S21 and the second emotion analysis result calculated in step S22 as inputs of the neural network. have.

Next, in step S24, the treatment module may determine and provide an emotion treatment method based on the integrated emotion analysis result output in step S23.

Here, the emotion treatment method may be provided in the form of content that induces a change in the user's emotion to a positive emotion when the user's emotion is shown as a negative emotion or a neutral emotion as a result of the integrated emotion analysis.

In addition, the emotion treatment method may be provided in the form of content that induces positive maintenance and expansion of the user's emotion when it is found that the user's emotion is a positive emotion as a result of the integrated emotion analysis.

In the above description, steps S21 to S24 may be further divided into additional steps or may be combined into fewer steps, depending on the embodiment of the present application. In addition, some steps may be omitted as necessary, or the order between steps may be changed.

A neural network-based emotion analysis method according to an embodiment of the present application and a neural network-based emotion analysis-based emotion treatment method according to an embodiment of the present application are implemented in the form of program instructions that can be executed through various computer means, and are stored in a computer-readable medium. Can be recorded. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

In addition, the above-described neural network-based emotion analysis method and neural network-based emotion analysis method may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

200: Neural network based emotion analysis system
200': Emotional therapy system based on emotion analysis based on neural networks
100: neural network-based emotion analysis device
110: Emotional therapy device based on emotion analysis based on neural network
10: speech analysis module
20: context analysis module
30: integrated module
40: treatment module
2: user terminal

Claims (18)

As a neural network-based emotion analysis method,
(a) calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user;
(b) calculating a second emotion analysis result by applying a context analysis based on an emotion dictionary to the voice data; And
(c) outputting an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as inputs of a neural network,
Neural network-based emotion analysis method comprising a. The method of claim 1,
In the step (a), the first emotion analysis result is calculated through the neural network-based speech analysis in which the speech waveform feature extracted from the speech data is input as an input. The method of claim 2,
The voice waveform feature is an MFCC feature value output by performing MFCC (Mel-Frequency Cepstral Coefficient) transformation on the voice data. The method of claim 1,
The first emotion analysis result is a value probabilistically given to each of a plurality of emotions including happiness, neutrality, sadness, and anger, a neural network-based emotion analysis method . The method of claim 1,
In the step (b), the voice data is converted into text, and then at least one word basic type is extracted from the converted text, and the extracted at least one word basic type is converted to a word basic type included in the emotion dictionary DB. Matching to calculate the second emotion analysis result, neural network-based emotion analysis method. The method of claim 5,
The emotion dictionary DB is constructed in a form in which emotion scores related to the strength of positive and the strength of negative are given for each of a plurality of basic types of words,
In the step (b), the second emotion analysis result is calculated by considering the emotion score of each of the basic types of matching words that match the basic types of words included in the emotion dictionary DB among the extracted basic types of the at least one word, Neural network-based sentiment analysis method. The method of claim 1,
In the step (c), the result of the integrated emotion analysis is in the form of a value probabilistically given to each of a plurality of emotions including happiness, neutrality, sadness, and anger. The method for analyzing emotions based on a neural network is a result of determining an emotion having a maximum probability among the plurality of emotions as a final emotion. As an emotional treatment method based on a neural network-based emotion analysis,
A method for treating emotions based on a neural network based emotion analysis to determine and provide an emotion treatment method based on the result of the integrated emotion analysis provided by performing the neural network-based emotion analysis method according to claim 1. The method of claim 8,
The emotional treatment method,
When the integrated emotion analysis result indicates that the user's emotion is a negative emotion or a neutral emotion, it is provided in the form of content that induces a change to a positive emotion with respect to the user's emotion,
When the user's emotion is shown as a positive emotion as a result of the integrated emotion analysis, the emotion treatment method based on a neural network-based emotion analysis is provided in a form of content that induces positive maintenance and expansion of the user's emotion. As a neural network-based emotion analysis device,
A voice analysis module for calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user;
A context analysis module for calculating a second emotion analysis result by applying a context analysis based on an emotion dictionary to the voice data; And
An integrated module configured to output an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as inputs of a neural network,
Neural network-based emotion analysis device comprising a. The method of claim 10,
The voice analysis module is configured to calculate the first emotion analysis result through the neural network-based voice analysis that receives the voice waveform feature extracted from the voice data as an input. The method of claim 11,
The voice waveform feature is an MFCC feature value output by performing a Mel-Frequency Cepstral Coefficient (MFCC) transformation on the voice data. The method of claim 10,
The first emotion analysis result is a value probabilistically given to each of a plurality of emotions including Happy, Neutral, Sad, and Angry. . The method of claim 10,
The context analysis module converts the voice data to text, and then extracts at least one word basic type (Lemma) from the converted text, and matches the extracted at least one word basic type with the word basic type included in the emotion dictionary DB. And calculating the second emotion analysis result. The method of claim 14,
The emotion dictionary DB is constructed in a form in which emotion scores related to the strength of positive and the strength of negative are given for each of a plurality of basic types of words,
The context analysis module calculates the second emotion analysis result by considering the emotion score of each matching word basic type matched with the word basic type included in the emotion dictionary DB among the extracted at least one word basic type. Based emotion analysis device. The method of claim 10,
The integrated emotion analysis result is primarily derived in the form of a value given probabilistically for each of a plurality of emotions including Happy, Neutral, Sad, and Angry, and the plurality of The emotion analysis device based on a neural network is the result of determining as the final emotion the emotion having the maximum probability of emotion. As an emotion therapy device based on emotion analysis based on a neural network,
A neural network-based emotion analysis-based emotion treatment apparatus to determine and provide an emotion treatment method based on the integrated emotion analysis result provided by performing a neural network-based emotion analysis by the neural network-based emotion analysis device according to claim 10. The method of claim 17,
The emotional treatment method,
When the integrated emotion analysis result indicates that the user's emotion is a negative emotion or a neutral emotion, it is provided in the form of content that induces a change to a positive emotion with respect to the user's emotion,
When the user's emotion is found to be a positive emotion as a result of the integrated emotion analysis, the emotion treatment device based on a neural network based emotion analysis is provided in a form of content that induces positive maintenance and expansion of the user's emotion.

Images