KR102299455B1 - Method and apparatus for neural network based sentiment analysis and sentiment therapy apparatus based on the same - Google Patents

Abstract

To a neural network-based emotion analysis method, the neural network-based emotion analysis method includes the steps of: (a) calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user; (b) calculating a second emotion analysis result by applying context analysis based on emotion dictionary to the voice data; and (c) using the first emotion analysis result and the second emotion analysis result as inputs to a neural network, and outputting and providing an integrated emotion analysis result for the voice data.

Description

Neural network-based emotion analysis method and apparatus, and apparatus for emotion treatment based thereon

The present application relates to a neural network-based emotion analysis and emotion treatment system. In particular, the present application relates to a neural network-based emotion analysis apparatus, method, and system, and a neural network-based emotion analysis-based emotion (emotional) treatment apparatus, method, and system.

Emotional engineering refers to the technical realization of non-verbal elements such as the human mind and mood to be useful in real life by grafting them into engineering. Unlike the strategies of companies in the past, which focused only on function and quality, recently, developing products by stimulating people's emotions (emotions) or using them has become a central strategy.

Affective Computing is a field that researches and develops artificial intelligence related to designing systems and devices that can recognize, interpret, and process human emotions. AI-based systems. This is a technology that can be used for interaction between humans and computers by recognizing people's psychological responses through physical or sensory stimulation.

With the recent advancement of technology, there is a close relationship between machine and human, and human characteristics are rational and emotional, so it is essential to consider the emotional (emotional) aspect when humans and robots interact. Accordingly, many domestic and foreign companies are actively conducting research related to emotional computing, but it is still an early stage of commercialization, and it is a field worth researching.

Conventionally, as a technique for analyzing a user's emotion (emotion), there exist emotion analysis techniques using voice analysis, natural language processing, and the like.

However, in conventionally known voice analysis-based emotion analysis techniques, it is difficult to recognize when a user speaks an emotional sentence with a neutral accent, and the emotion analysis result is derived differently due to individual differences due to the same sentence. There is a problem that the accuracy of

The background technology of the present application is [Yelin Kim and Emily Mower Provost. 2015. Emotion Recognition During Speech Using Dynamics of Multiple Regions of the Face. ACM Trans. Multimedia Comput. Commun. Appl. 12, 1s, Article 25 (October 2015), 23 pages.]

The present application is to solve the problems of the prior art described above, and to provide a neural network-based emotion analysis apparatus and method that can derive more accurate emotion analysis results, and a neural network-based emotion analysis-based emotion (emotional) treatment apparatus and method The purpose.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the neural network-based emotion analysis method according to the first aspect of the present application, (a) applies a neural network-based voice analysis to voice data input from a user to obtain a first emotion analysis result calculating; (b) calculating a second emotion analysis result by applying context analysis based on emotion dictionary to the voice data; and (c) using the first emotion analysis result and the second emotion analysis result as inputs to a neural network, and outputting and providing an integrated emotion analysis result for the voice data.

As a technical means for achieving the above technical problem, the neural network-based emotion analysis-based emotion treatment method according to the second aspect of the present application is an integrated emotion analysis result provided by the present application performing the neural network-based emotion analysis method according to the first aspect of the present application Based on this, emotional therapy can be decided and provided.

As a technical means for achieving the above technical problem, the neural network-based emotion analysis apparatus according to the third aspect of the present application applies neural network-based speech analysis to voice data input from a user to produce a first emotion analysis result. analysis module; a context analysis module for calculating a second emotion analysis result by applying an emotion dictionary-based context analysis to the voice data; and an integrated module for outputting and providing an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as input to a neural network.

As a technical means for achieving the above technical problem, the neural network-based emotion analysis-based emotion treatment apparatus according to the fourth aspect of the present application performs neural network-based emotion analysis by the neural network-based emotion analysis apparatus according to the third aspect of the present application Thus, it is possible to determine and provide an emotion treatment method based on the provided integrated emotion analysis result.

As a technical means for achieving the above technical task, the computer program according to the fifth aspect of the present application, the neural network-based emotion analysis method according to the first aspect of the present application, and the neural network-based emotion analysis-based emotion according to the second aspect of the present application It may be stored in a recording medium in order to execute the treatment method.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, since the integrated emotion analysis result is output by considering the neural network-based voice analysis and the emotional dictionary-based context analysis together, it is possible to more accurately derive the user's emotional state for the input voice data. can have an effect.

According to the above-described problem solving means of the present application, by providing an emotion treatment method determined based on the integrated emotion analysis result, the user's emotional/emotional state can be accurately identified and based on this, the user's current emotion can be changed more positively. , can provide users with a better life.

According to the above-described problem solving means of the present application, by providing an emotion therapy method, if the user's emotion is a negative emotion or a neutral emotion, it is possible to induce the user's emotion to be changed into a positive emotion, and if the user's emotion is a positive emotion, the corresponding positive emotion may be maintained or extended beyond that.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a diagram showing a schematic configuration of a neural network-based emotion analysis system according to an embodiment of the present application.
2 is a diagram showing a schematic configuration of a neural network-based emotion analysis apparatus according to an embodiment of the present application.
3 shows an example of an integrated emotion analysis result derived by the neural network-based emotion analysis apparatus according to an embodiment of the present application.
4 shows another example of an integrated emotion analysis result derived by the neural network-based emotion analysis apparatus according to an embodiment of the present application.
5 is a diagram illustrating a schematic configuration of an emotion treatment system based on neural network-based emotion analysis according to an embodiment of the present application.
6 is a diagram illustrating a schematic configuration of an emotion treatment apparatus based on neural network-based emotion analysis according to an embodiment of the present application.
7 is a diagram for explaining an example in which a user's emotion treatment is performed by an emotion treatment apparatus based on a neural network-based emotion analysis according to an embodiment of the present application.
8 is a diagram schematically illustrating a menu configuration in a mobile application interworking with an emotion treatment apparatus based on neural network-based emotion analysis according to an embodiment of the present application.
FIG. 9 is a diagram for explaining an example of calculating a second emotion analysis result by a context analysis module in an emotion treatment apparatus based on a neural network-based emotion analysis according to an embodiment of the present application.
10A to 10C are diagrams for explaining an example of calculation of a first emotion analysis result by a voice analysis module in a neural network-based emotion analysis-based emotion treatment apparatus according to an embodiment of the present application.
11 is a view for explaining an example of calculation of an integrated emotion analysis result by an integrated module in an emotion treatment apparatus based on a neural network-based emotion analysis according to an embodiment of the present application.
12 is a view showing an example of a screen display of an emotion analysis result by applying a conventional natural language processing to a given voice data.
13 is a view showing an example of displaying a screen of an emotion analysis result through application of context analysis by a context analysis module of an emotion treatment apparatus based on neural network-based emotion analysis according to an embodiment of the present application for given voice data.
14 is an operation flowchart of a neural network-based emotion analysis method according to an embodiment of the present application.
15 is an operation flowchart of an emotion treatment method based on a neural network-based emotion analysis according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a diagram showing a schematic configuration of a neural network-based emotion analysis system 200 according to an embodiment of the present application. 2 is a diagram illustrating a schematic configuration of a neural network-based emotion analysis apparatus 100 according to an embodiment of the present application.

Hereinafter, for convenience of description, the neural network-based emotion analysis system 200 according to an embodiment of the present application is referred to as the present system 200 , and the neural network-based emotion analysis apparatus 100 according to an embodiment of the present application is referred to as the present device 100 . shall do

1 and 2 , the present system 200 may include a user terminal 2 and the present apparatus 100 .

The user terminal 2 may receive voice data d, which is an emotion analysis target, from the user 1 . The user terminal 2 may transmit the received voice data d to the device 100 .

The user terminal 2 may be, for example, a mobile and portable communication device such as a smart phone, a smart pad, a tablet PC, a notebook computer, and a wearable device. However, the present invention is not limited thereto, and the user terminal 2 includes a Personal Communication System (PCS), a Global System for Mobile communication (GSM), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), and a Personal Digital Assistant (PDA). ), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, Smartphone, SmartPad , tablet PCs, notebook computers, wearable devices, desktop PCs, and the like, may include all types of wired and wireless communication devices.

The apparatus 100 may analyze the user's emotion through analysis (voice analysis, context analysis) of the voice data d input from the user 1 . The device 100 applies voice analysis and context analysis to voice data d input from the user 1, and applies the first emotion analysis result d1 calculated through the application of voice analysis and context analysis The integrated emotion analysis result d3 in which the second emotion analysis result d2 calculated through . Here, the voice data d received by the device 100 may be input through the user terminal 2 (in other words, received from the user terminal).

The device 100 may be, for example, a server. However, the present invention is not limited thereto, and as another example, the device 100 is a device provided in the user terminal 2 , and in particular, may refer to the mobile application 2a itself installed in the user terminal 2 .

An example of a case in which the present device 100 is provided in the form of a server is as follows. The user terminal 2 may have a mobile application 2a interworking with the device 100 that performs emotion analysis on the voice data d input from the user 1 may be installed. The mobile application 2a in the user terminal 2 may transmit the voice data d received from the user 1 to the device 100 . The device 100 applies the voice analysis and context analysis to the voice data d received from the mobile application 2a, thereby generating the integrated emotion analysis result d3 as the user's emotion analysis result for the voice data d. can create Thereafter, the device 100 may provide (output) the generated integrated emotion analysis result d3 to the mobile application 2a in the user terminal 2 , and through this, on the screen of the user terminal 2 . The integrated emotion analysis result d3 may be displayed (displayed). The user may receive and check the integrated emotion analysis result d3 output from the device 100 through the screen of the user terminal 2 .

As such, when the device 100 is a server (provided in the form of a server), data transmission/reception between the device 100 and the user terminal 2 through network communication (that is, voice data from the user terminal to the device) transmission and transmission of the integrated emotion analysis result from the device to the user terminal) may be performed.

Network communication is, for example, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), and a Wireless Local Area (Wireless LAN) network. Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC (Near Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc. may be included. and is not limited thereto.

An example of a case in which the present device 100 (in particular, the emotion analysis method by the present device) is provided in the form of a mobile application 2a in the user terminal 2 is as follows. The apparatus 100 receives voice data d from the user 1 and applies voice analysis and context analysis to the input voice data d as a result of user's emotion analysis on the voice data d. An integrated emotion analysis result d3 may be generated. Thereafter, the device 100 may provide (output output) the generated integrated emotion analysis result d3 on the screen of the user terminal 2 , through which the integrated emotion analysis is performed on the screen of the user terminal 2 . The result d3 may be displayed (displayed). The user may receive and check the integrated emotion analysis result d3 output from the device 100 through the screen of the user terminal 2 .

Hereinafter, for convenience of description, the present device 100 will be exemplified as a server (provided in the form of a server), and a detailed description of the present device 100 is as follows.

The apparatus 100 may include a voice analysis module 10 , a context analysis module 20 , and an integration module 30 .

The voice analysis module 10 may calculate the first emotion analysis result d1 by applying the neural network 12-based voice analysis to the voice data d input from the user 1 . The voice analysis module 10 may transmit the calculated first emotion analysis result d1 to the integration module 30 .

The voice analysis module 10 may calculate the first emotion analysis result d1 through the neural network 12-based voice analysis to which the voice waveform features extracted from the voice data d are input.

Here, the voice waveform feature may be an MFCC feature value output by performing Mel-Frequency Cepstral Coefficient (MFCC) transformation on the voice data d. In addition, the first emotion analysis result calculated by the voice analysis module 10 is probabilistic for each of a plurality of emotions including happiness (Happy), neutral (Neutral), sadness (Sad) and anger (Angry). It may be a given value.

Specifically, the voice analysis module 10 may receive the voice data d input through the mobile application 2a of the user terminal 2 from the user terminal 2 .

For example, the user 1 may record a sentence representing the current emotion through the mobile application 2a of the user terminal 2 . The voice file corresponding to the sentence representing the current emotion of the user 1 recorded through the mobile application 2a may be provided as an input of the voice analysis module 10 as the voice data d to be subjected to voice analysis. . In the present device 100, as an example, a recorded file is considered as voice data d, but the present invention is not limited thereto, and as another example, voice data d may be voice data inputted in real time from a user. have.

The voice analysis module 10 may receive (transfer) a voice file having a .wav extension from the mobile application 2a as voice data d to be analyzed.

The voice analysis module 10 may apply a Mel-Frequency Cepstral Coefficient (MFCC) as an example of a voice recognition algorithm to the input voice data d. The voice analysis module 10 may perform MFCC conversion 11 through application of MFCC to the input voice data d, and output MFCC feature values as voice waveform features for the voice data d through this. can do. That is, the voice analysis module 10 performs MFCC transformation 11 on the input voice data d to model a sound corresponding to the voice data d, and converts the MFCC feature value (MFCC value) accordingly to the voice data. (d) can be output as the audio waveform feature.

The voice analysis module 10 may provide a voice waveform feature that is an MFCC feature value output through the MFCC transformation 11 as an input of the neural network 12 .

As the neural network 12 , Keras may be considered as an example. Keras stands for an open source neural network library written in Python. Keras can run on top of MXNet, Deeplearning4j, TensorFlow, Microsoft Cognitive Toolkit or Theano. Keras is designed to enable rapid experimentation with deep neural networks and is focused on minimally modular scalability. Keras was developed as part of the research effort of the Open-ended Neuro-Electronic Intelligent Robot Operating System (ONEIROS) project.

Since Keras is a technique well known to those of ordinary skill in the art to which this application belongs, hereinafter, rather than a description of Keras itself, the Keras-based neural network 12 is a neural network-based emotion analysis according to an embodiment of the present application. An example applied to the device 100 will be mainly described.

In the apparatus 100 , as the neural network 12 , a convolutional neural network (CNN, convolutional neural network) may be considered as an example. However, the present invention is not limited thereto, and examples of the neural network 12 considered in the present device 100 include a recurrent neural network (RNN), a deep neural network, etc. previously known or future Various neural networks to be developed can be applied. The neural network 12 may be expressed differently as a deep learning model or the like.

The voice analysis module 10 may provide a voice waveform feature, which is an MFCC feature value output through the MFCC transformation 11, as an input to the neural network 12, and through this, the neural network 12-based voice analysis is performed. As the output of 12), the first emotion analysis result d1 may be calculated.

In this case, the voice analysis module 10 may provide not only the voice waveform features but also the model data obtained from the local storage 13 as an input of the neural network 12 . Through this, the voice analysis module 10 may calculate the first emotion analysis result d1 through voice analysis based on the neural network 12 .

Here, the model data is a neural network that is trained to output a first emotion analysis result corresponding to the input MFCC by taking MFCC feature values for a plurality of voice data as input to the neural network. (specifically, a weight value of a neural network determined through learning). That is, the model data may mean a weight value of the learned neural network. The local storage 13 may mean a database in which a plurality of such model data are stored.

The voice analysis module 10 passes the voice waveform feature (ie, MFCC feature value) for the voice data d and the model data obtained from the local storage 13 through the neural network 12, so that the neural network ( 12), the first emotion analysis result d1 may be calculated (outputted).

The first emotion analysis result d1 may be a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sadness, and anger. The first emotion analysis result d1, which is a value probabilistically assigned to each of the plurality of emotions, represents the probability of the emotion of the voice data d. In other words, the first emotion analysis result d1 represents a probability (probability value, probability score) for each of a plurality of emotions included in the voice data d.

As the first emotion analysis result d1 is a value probabilistically assigned to each of the plurality of emotions, the sum of values probabilistically assigned to each of the plurality of emotions may be 1.

Illustratively, the voice analysis module 10 performs a first emotion analysis result (d1) such as 'Happy: 0.05, Neutral: 0.03, Sad: 0.7, Angry: 0.22' can be calculated.

At this time, according to the first emotion analysis result d1, as the probability value (probability score) of the emotion corresponding to Sad among the four emotions has the largest value as 0.7, the voice analysis module 10 is From the first emotion analysis result d1 calculated by applying the neural network-based voice analysis, it can be determined that the emotion corresponding to 'sadness' occupies the largest proportion in the input voice data d. In other words, the voice analysis module 10 may determine that the user's emotion for the voice data d based on the neural network-based voice analysis is 'sad' through the first emotion analysis result d1.

The device 100 exemplarily includes happiness, neutrality, sadness, and anger as four plurality of emotions, but is not limited thereto, and the device 100 includes a plurality of emotions or a plurality of emotions. The type of emotion, etc. may be applied by changing more diversely.

The first emotion analysis result d1 calculated from the voice analysis module 10 may be input to the integration module 30 . The first emotion analysis result d1 may be used (used) to extract (generate) the integrated emotion analysis result d3 in the integration module 30 .

The context analysis module 20 applies the context analysis based on the emotion dictionary (emotional word dictionary, emotion dictionary DB, emotion word DB) 23 to the voice data d input from the user 1 to obtain a second emotion analysis result (d2) can be calculated.

The context analysis module 20 converts the voice data (d) into text, then extracts at least one word primitive (Lemma) from the converted text (Text Data), and converts the extracted at least one word primitive into the emotion dictionary DB ( 23), the second emotion analysis result d2 may be calculated by matching with the basic form of the word.

Here, the emotion dictionary DB 23 may be constructed in a form in which emotion scores related to the strength of positive and negative strength are assigned to each of the plurality of basic types of words. The context analysis module 20 calculates the second emotion analysis result (d2) in consideration of the emotion scores of each of the matching word basic types matching the word basic types included in the emotion dictionary DB 23 among the extracted basic types of at least one word. can

As a specific example, the context analysis module 20 may receive (transfer) a voice file having a .wav extension from the mobile application 2a as voice data d to be analyzed.

The context analysis module 20 may convert the input voice data (ie, .wav voice file) (d) into text (text data, text data) through STT (Speech To Text, 21). Thereafter, the context analysis module 20 may extract a word primitive (Lemma) using the converted text data as an example, using Google Natural Language 22, which is Google's natural language processing technology. In other words, the context analysis module 20 may extract at least one word primitive (Lemma) from the converted text by applying the Google Natural Language 22 to the converted text.

In the text, basic words that are the roots of words that can be used by transforming words are included, and these basic words (basic form of word, basic form of word) can be called Lemma. For example, 'write', 'writing', 'wrote', and 'written' are all based on the basic form of 'write'. Therefore, when at least one of 'write', 'writing', 'wrote' and 'written' is included in the converted text as an example, the context analysis module 20 performs at least one word primitive (Lemma) from the converted text. ) as 'write'.

In addition, the context analysis module 20 may extract a word primitive (Lemma) as well as a dependency (Dependency) from the converted text (Text Data) through application of the Google Natural Language (22).

Here, the term “dependency” refers to a relationship with respect to dependency between word tokens (corpus in sentences, word units). The dependency on the converted text may be extracted (identified) through, for example, a dependency tree (dependency tree, dependency tree) provided by the Google Natural Language 22 . A description of the dependency tree is exemplarily described in "Universal Dependency Annotation for Multilingual Parsing", Ryan McDonald, Joakim Nivre et al., Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 92-97, Sofia, Bulgaria, August 4-9 2013.] can be more easily understood with reference to, and a detailed description will be omitted herein.

Dependency extracted from the context analysis module 20 may be utilized to strengthen the influence of the emotional word (emotional word) and to identify the cause.

After extracting at least one word primitive (Lemma) from the converted text (Text Data), the context analysis module 20 matches the extracted at least one word primitive type with the word primitive included in the emotion dictionary DB 23 can do. In this case, the emotion dictionary DB 23 may be constructed and provided in a form in which emotion scores related to the strength of positive and negative strength are given to each of the plurality of basic types of words.

By matching the extracted at least one word primitive and the word primitive included in the emotion dictionary DB 23 , the context analysis module 20 is a word included in the emotion dictionary DB 23 from among the at least one extracted basic word type. It is possible to obtain an emotion score for each basic type of a matching word matching the basic type.

For example, in the appraisal dictionary DB 23, 'Fun: +7.0', 'gift: +6.0', 'travel: +3.0', 'slipper: 0.0', Data such as 'funeral: -6.0' and 'fail: -5.0' may be pre-stored.

Here, 'Fun: +7.0' may mean that +7.0 is given as an emotional score related to the strength (strength) of the affirmation with respect to the basic form (Lemma) of the word corresponding to 'Fun'. . Also, 'funeral: -6.0' means that -6.0 is given as an emotional score related to the intensity (strength) of negation with respect to the basic form of the word corresponding to 'funeral'. In addition, 'slipper: 0.0' means that 0.0 is given as an emotional score related to the strength of negative (or positive) with respect to the basic form of the word corresponding to 'slipper'. The example of the emotional score for each basic word type is merely an example for helping understanding of the present application, and is not limited thereto.

In this case, the emotional score of 0.0 may mean that the corresponding word is not a word representing positive nor negative, but a word representing neutral emotion. In addition, an emotional score of + means that the corresponding word is an affirmative word, which may mean that the greater the number, the greater the positive strength (strength, degree). Conversely, an emotional score of - means that the corresponding word is a word representing negation, which may mean that the higher the number, the greater the intensity (strength, degree) of negation.

The context analysis module 20 may include, for example, an emotion score module 24 . The emotion score module 24 may obtain an emotion score of each of the matching word basic types matching the word basic types included in the emotion dictionary DB 23 among the extracted at least one basic word types. Thereafter, the emotion score module 24 may calculate the second emotion analysis result d2 in consideration of the emotion score of each of the obtained matching word basic types. In particular, the emotion score module 24 may, for example, sum up the emotion scores of each basic matching word, and calculate the second emotion analysis result d2 based on the summed emotion scores.

For example, it is assumed that 'Fun' and 'travel' are extracted as at least one word primitive form from the converted text. In this case, the emotion score module 24 matches the extracted at least one word basic type with the word primitive included in the emotion dictionary DB 23, thereby matching the word basic type 'Fun' and matching word basic type You can acquire '+7.0' as the emotional score of , and '+3.0' as the emotional score of the matching word basic type matching with the basic word 'Travel'. Thereafter, the emotional score module 24 may sum the emotional scores of each of the obtained matching word basic types, and use the summed emotional score of +10.0 (+7.0+3.0=+10.0) as the second emotion analysis result (d2). ) can be calculated as

At this time, according to the second emotion analysis result d2, as the summed emotion score is +10.0, the context analysis module 20 calculates the second emotion analysis result by applying the context analysis based on the emotion dictionary 23 From (d2), it can be determined that emotions corresponding to negative rather than positive appear in the input voice data (d). In other words, the context analysis module 20 may determine, through the second emotion analysis result d2, that the user's emotion for the voice data d based on the emotion dictionary-based context analysis is a 'positive emotion'. have.

The context analysis module 20 determines that the user's emotion is a 'positive emotion' when the summed emotional score is a + value, and determines that the user's emotion is a 'negative emotion' when the summed emotional score is - When the score is 0, it may be determined that the user's emotion is a 'neutral emotion'.

In other words, if there is a word (matching word basic type) belonging to the emotion dictionary DB 23 among words in the sentence (that is, at least one basic word type extracted from the converted text), the context analysis module 20 , it is possible to obtain an emotional score regarding the presence or absence of a positive/negative relationship for the corresponding word (ie, an emotional score related to the strength of positive or negative). Thereafter, the context analysis module 20 calculates the emotion score (ie, the summed emotion score) for the entire sentence (ie, the entire sentence corresponding to the voice data) based on the acquired emotion score as the second emotion analysis result (d2). ) can be calculated as In this case, the size of the summed emotional score indicates the strength of the emotion positive/negative.

The second emotion analysis result d2 calculated from the context analysis module 20 may be input to the integration module 30 . The second emotion analysis result d2 may be used (used) to extract (generate) the integrated emotion analysis result d3 in the integration module 30 .

The integration module 30 uses the first emotion analysis result d1 and the second emotion analysis result d2 as inputs to the neural network 31 to output the integrated emotion analysis result d3 for the voice data d. can Here, as the neural network 31 , as in the neural network 12 described above, Keras may be considered as an example.

That is, the integration module 30 acquires the first emotion analysis result (d1) calculated by the voice analysis module 10 and the second emotion analysis result (d2) calculated by the context analysis module 20 to obtain the neural network ( 31) can be provided as an input. The integration module 10 performs the first emotion analysis result (d1) and the second emotion analysis result (d2) as the output of the neural network 31 through the neural network 31-based analysis using the neural network 31 as inputs. An integrated emotion analysis result d3 in which the first emotion analysis result d1 and the second emotion analysis result d2 are integrated may be calculated.

At this time, the integration module 30 provides the first and second emotion analysis results d1 and d2 as well as the model data obtained from the local storage 32 as an input to the neural network 31 . can That is, the integration module 30 performs local analysis based on the neural network 31 for calculating the integrated emotion analysis result d3, depending on whether there is model data previously configured and stored in the local storage 32. Previous model data may be loaded and used (utilized) from the storage 32 . The integrated module 30 may calculate the integrated emotion analysis result d3 through the analysis based on the neural network 31 .

Here, the model data uses a plurality of first emotion analysis results and a plurality of second emotion analysis results as inputs to the neural network, and outputs an integrated emotion analysis result corresponding to the input first/second emotion analysis results d1 and d2. In a neural network that is learned to do so, it may mean a learning value of the neural network determined by repeatedly learning the neural network (specifically, a weight value of the neural network determined through learning). That is, the model data may mean a weight value of the learned neural network. The local storage 32 may mean a database in which a plurality of such model data are stored.

The integration module 30 passes the first/second emotion analysis results d1 and d2 and the model data obtained from the local storage 32 through the neural network 31, and the integrated emotion analysis result d3 from the neural network 31 ) can be calculated (output).

Here, the integrated emotion analysis result is primarily derived in the form of a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sad, and Angry, and the It may be a result of determining an emotion having a maximum probability among a plurality of emotions as the final emotion.

In other words, the integration module 30 is the first emotion analysis result d1 received from the voice analysis module 10 (values probabilistically assigned to each of a plurality of emotions including happiness, neutrality, sadness and anger, For example, the probability values of four emotions) and the second emotion analysis result (d2) received from the context analysis module 20 (the value of the summed emotion score for each emotion score of the matching word basic type, the strength of positive/negative By inputting the value of the related emotion score) into the neural network 32 of the integration module 30 , the final emotion of the user corresponding to the voice data d is output as the integrated emotion analysis result d3 that is the output of the neural network 32 . can provide

Here, the integrated module 30 is primarily configured to provide the output of the integrated emotion analysis result d3, similarly to the first emotion analysis result extracted from the voice analysis module 10, as a plurality of emotions. It is possible to derive a value (weight value) that is probabilistically assigned to each emotion (Happy, Neutral, Sad, Angry). In this case, the emotion corresponding to the maximum value among the probability values for the first derived four emotions may be determined as the final emotion of the user 1 corresponding to the voice data d. That is, the integration module 30 determines the emotion having the maximum probability value among the probability values (weight values) for each of the first derived four emotions as the final emotion of the user 1 corresponding to the voice data d, and , it is possible to output the determined final emotion as an integrated emotion analysis result d3.

The device 100 analyzes the user's emotions for the voice data (d) by fusing the neural network-based voice analysis and the emotion dictionary-based context analysis, thereby simply analyzing the emotions based on the voice analysis or based on the context analysis. It is possible to effectively increase the accuracy of the user's emotion analysis compared to the emotion analysis technique.

3 shows an example of an integrated emotion analysis result d3 derived by the neural network-based emotion analysis apparatus 100 according to an embodiment of the present application.

Referring to FIG. 3 , when anti-linguistic features (eg, intonation, tone, etc.) do not exist in the voice data d, the voice analysis by the voice analysis module 10 is performed on the voice data d. It is not possible to analyze (figure out) the user's emotions just by applying them (it is impossible to accurately derive the user's emotions).

Therefore, since there is no anti-linguistic feature in the voice data (d), it is difficult to accurately determine the emotion of the user from the voice data (d) based on the voice analysis by the voice analysis module 10, but the voice data ( When a word (emotional word) expressing emotion is included in the speech sentence of d) (in other words, when there is no special feature part for intonation, etc. in the speech sentence, but an emotion word is included), the integration module 30 is the first By considering not only the emotion analysis result (d1) but also the second emotion analysis result (d2) calculated through the application of context analysis, an integrated emotion analysis result (d3) representing the user's final emotion for the voice data (d) is calculated. can provide

Illustratively, in relation to a case in which anti-linguistic features (tongue, tone, etc.) do not exist in the utterance of the voice data d input from the user, as the voice data d, “Why is it so selfish?” Let's say that an utterance is entered.

The voice analysis module 10 may calculate a probability (probability value) for each of a plurality of as the first emotion analysis result d1 by applying the voice analysis to the voice data d. According to this, when a single processing of voice analysis is applied to the input voice data d, as a result, the probability corresponding to happiness among the probabilities for a plurality of emotions (four emotions) has the largest value. Accordingly, an emotional result corresponding to 'Happy' may be calculated. That is, the user's emotion can be calculated as 'Happy' with a single application of voice analysis.

Meanwhile, as a result of applying the context analysis to the voice data d, the context analysis module 20 may calculate an emotion value corresponding to -3.0 (negative emotion) as the second emotion analysis result d2. That is, when a single process of context analysis is applied to the input voice data d, an emotion result corresponding to "-3.0 (negative emotion)" may be calculated as a result. In other words, with a single application of context analysis, the user's emotion can be calculated as 'neutral'.

The integration module 30 is based on the neural network 31 by considering '-3.0 (negative emotion)', which is the second emotion analysis result (d2), and 'probability for each of a plurality of emotions', which is the first emotion analysis result (d1). By performing the fusion (integration) analysis of , it is possible to calculate a probability for each of a plurality of emotions as a result of the integrated emotion analysis d3 for the voice data d. As a result of the integrated analysis of the first and second emotion analysis results (d1 and d2), the integration module 30 sets the probability corresponding to sadness (Sad) to the highest value (highest value) among the probabilities for the four emotions. As it appears, the user's final emotion for the voice data d may be calculated as 'Sad'. That is, the integrated module 30 may calculate the final emotion of the user as 'Sad' because the emotion having the maximum probability among the plurality of emotions is 'Sad' through the integrated analysis.

4 shows another example of an integrated emotion analysis result d3 derived by the neural network-based emotion analysis apparatus 100 according to an embodiment of the present application.

Referring to FIG. 4 , if there is no word (emotional word) representing the user's emotion in the voice data d, context analysis based on the emotion dictionary (emotion dictionary DB, 23) is applied to the voice data d. It is not possible to analyze (figure out) the user's emotions just by doing it.

Therefore, there is no word expressing emotion in the voice data d, so based on the context analysis by the context analysis module 20, it is difficult to determine whether the user's emotion is a positive emotion or a negative emotion from the voice data d. In a difficult case (in other words, when the second emotion analysis result for the voice data is 0.0 and it is calculated to be a neutral emotion), the integration module 30 performs the second emotion analysis result d2 as well as the second emotion analysis result d2 calculated through the application of voice analysis. An integrated emotion analysis result d3 indicating the user's final emotion on the voice data d may be calculated and provided by considering the first emotion analysis result d1 together.

Illustratively, in relation to a case where a word (emotional word) representing an emotion does not exist in the utterance of the voice data d input from the user, as the voice data d, “I'm done. I don't” Let's say the utterance "want to play." is input.

As a result of applying the context analysis to the voice data d, the context analysis module 20 may calculate an emotion value corresponding to 0.0 (neutral emotion) as the second emotion analysis result d2. That is, when a single process of context analysis is applied to the input voice data d, an emotion result corresponding to "0.0(-) (neutral emotion)" may be calculated as a result. In other words, with a single application of context analysis, the user's emotion can be calculated as 'neutral'.

On the other hand, the voice analysis module 10 may calculate a probability (probability value) for each of a plurality of as the first emotion analysis result d1 by applying the voice analysis to the voice data d. According to this, when a single processing of voice analysis is applied to the input voice data d, the result is a value (0.00740) with the highest probability corresponding to Angry among the probabilities for a plurality of emotions (four emotions). ), an emotional result corresponding to 'Angry' can be calculated. That is, with a single application of voice analysis, it can be calculated that the user's emotion is 'Angry'.

The integration module 30 considers '0.0 (neutral emotion)', which is the second emotion analysis result (d2), and 'probability for each of a plurality of emotions', which is the first emotion analysis result (d1), together with the neural network 31-based By performing the fusion (integration) analysis, it is possible to calculate a probability for each of a plurality of emotions as an integrated emotion analysis result d3 for the voice data d. As a result of the integrated analysis of the first and second emotion analysis results (d1, d2), the integration module 30 sets the probability corresponding to Angry to the highest value (highest value) among the probabilities for the four emotions. As it appears, the user's final emotion for the voice data d may be calculated as 'Angry'. That is, through the integrated analysis, the integration module 30 may calculate the final emotion of the user as 'anger' since the emotion having the maximum probability among the plurality of emotions is 'anger'.

According to this, the device 100 combines two types of analysis (context analysis and voice analysis) techniques, in contrast to calculating the user's emotion by applying only one analysis of context analysis and voice analysis as a single process. By calculating the user's emotion in consideration of the user's emotion, the accuracy of calculating (estimating) the user's emotion with respect to the voice data d can be effectively increased.

That is, when the emotion analysis based on the emotion dictionary is difficult because the emotional word does not exist in the utterance, the apparatus 100 may increase the accuracy of emotion estimation by using the voice analysis together. In addition, the apparatus 100 may increase the accuracy of emotion estimation by using the sentiment dictionary-based context analysis together when an emotional word is included although there is no special part in the intonation of an utterance.

The integrated module 30 may provide (output) the integrated emotion analysis result d3 calculated based on the integrated analysis to the mobile application 2a of the user terminal 2 . Through this, the integrated emotion analysis result d3 may be displayed (displayed) on the screen of the user terminal 2 . The user may receive and check the integrated emotion analysis result d3 output from the device 100 through the screen of the user terminal 2 .

5 is a diagram showing a schematic configuration of a neural network-based emotion analysis-based emotion treatment system 200' according to an embodiment of the present application. 6 is a diagram illustrating a schematic configuration of an emotion treatment apparatus 110 based on neural network-based emotion analysis according to an embodiment of the present application.

Hereinafter, for convenience of explanation, the neural network-based emotion analysis-based emotion treatment system 200 ′ according to an embodiment of the present application is referred to as the present system 200 ′, and neural network-based emotion analysis-based emotion treatment system according to an embodiment of the present application The device 110 will be referred to as the present device 110 . In addition, emotion therapy herein may be expressed differently as emotional therapy.

The present system 200' may be the same system as the present system 200 (a neural network-based emotion analysis system according to an embodiment of the present application) described above. Also, the apparatus 110 may be the same apparatus as the apparatus 100 (a neural network-based emotion analysis apparatus according to an embodiment of the present application) described above. That is, the present system 200 ′ and the present apparatus 110 differ only in that the configuration of the treatment module 40 is added, as compared to the present system 200 and the present apparatus 100 described above. The configurations and their functions may be the same.

Accordingly, even if omitted below, the descriptions of the present system 200 may be equally applied to the description of the present system 200 ′. Also, even if omitted below, the description of the apparatus 100 may be equally applied to the description of the apparatus 110 .

5 and 6 , the system 200 ′ may include a user terminal 2 and the device 110 . The apparatus 110 may include a voice analysis module 10 , a context analysis module 20 , an integration module 30 , and a treatment module 40 . Here, since the description of the user terminal 2, the voice analysis module 10, the context analysis module 20, and the integration module 30 has been described in detail above, the overlapping description will be omitted below.

The device 110 determines and provides an emotion therapy method d4 based on the integrated emotion analysis result d3 provided by performing a neural network-based emotion analysis by the device 100 described above. can A detailed description is as follows.

The integration module 30 uses the first emotion analysis result d1 and the second emotion analysis result d2 as inputs to the neural network 31 to calculate the integrated emotion analysis result d3 for the voice data d, The calculated integrated emotion analysis result d3 may be transmitted to the treatment module 40 .

The treatment module 40 may receive and obtain the integrated emotion analysis result d3 calculated by the integration module 30 from the integration module 30 .

The treatment module 40 may determine the emotion treatment method d4 (treatment method determination, 41) based on the obtained integrated emotion analysis result d3. That is, the treatment module 40 determines the final emotion of the user based on the result of the integrated emotion analysis d3, and determines which emotion treatment method (treatment method, d4) to provide to the user 1 according to the identified final emotion. can

Here, the type of emotion therapy (therapeutic method, d4) may include art therapy, music therapy, dance/movement therapy, etc. A variety of emotional therapy methods can be applied.

The emotion therapy method d4 determined in the treatment module 40, when the user's emotion is shown to be a negative emotion or a neutral emotion as a result of the integrated emotion analysis d3, a change to a positive emotion for the emotion of the user 1 It may be provided in the form of inducing content. In addition, the emotion therapy method d4 may be provided in the form of content that induces positive maintenance and expansion of the user 1's emotion when the emotion of the user 1 is found to be a positive emotion as a result of the integrated emotion analysis.

Here, as an example of the integrated emotion analysis result (d3), the fact that the user's emotion is negative emotion is one of a plurality of emotions including happiness, neutrality, sadness, and anger. It may mean a case in which the final emotion is determined to be Sad or Angry. Also, as the result of the integrated emotion analysis d3 , that the user's emotion is a neutral emotion may mean a case in which the final emotion among a plurality of emotions is determined to be neutral. Also, as the result of the integrated emotion analysis d3 , that the user's emotion is a positive emotion may mean a case in which the final emotion among the plurality of emotions is determined to be Happy.

Therefore, when the user's emotion appears as a negative emotion or a neutral emotion as the result of the integrated emotion analysis d3 (that is, when the final emotion is determined as any one of neutral, sadness, and anger), the treatment module 40 is the user's current emotion The emotion therapy method d4 may be provided in the form of content that induces any one of neutrality, sadness, and anger to be changed into a positive emotion, that is, happiness.

In addition, the treatment module 40 maintains the positive emotion corresponding to the user's current emotion, the happiness emotion, when the user's emotion appears as a positive emotion as the result of the integrated emotion analysis d3 (that is, when the final emotion is determined as happiness). The emotion therapy method d4 may be provided in the form of content that induces or expands.

As an example, the treatment module 40 may provide an art therapy method as an emotion therapy method in the form of content that induces a change in a user's emotion into a positive emotion. In addition, the treatment module 40 may provide at least one content among movies, music, and books as an emotion therapy method in the form of content that induces maintenance and expansion of positive emotions with respect to the user's emotions.

The treatment module 40 determines the emotion treatment method d4 based on the obtained integrated emotion analysis result d3 (treatment method determination, 41) so that the user can perform emotion treatment based on the determined emotion treatment method. The emotion treatment method d4 may be provided in a form that enables the user to perform the emotion treatment 42 and provided as a mobile application 2a of the user terminal 2 .

At this time, the treatment module 40 applies the treatment information obtained from the local storage 43 to the emotion treatment method determined based on the integrated emotion analysis result d3, thereby converting the determined emotion treatment method d4 to the emotional treatment method by the user ( It may be provided in a form that enables the performance 42 of emotion therapy). The treatment module 40 may provide the emotion treatment method d4 prepared to enable the execution 42 of the emotion treatment method to the user terminal 2 .

At least one treatment information corresponding to each emotion for each emotion may be pre-stored in the local storage 43 . Here, the treatment information may mean content information. For example, the treatment information may include a plurality of pieces of art picture related content information that can be provided when the final emotion is sadness, and a plurality of pieces of music related content information that can be provided when the final emotion is happiness.

For example, it is assumed that an art therapy method is determined (41) as an emotion therapy method based on the integrated emotion analysis result (d3). At this time, the treatment module 40 enables treatment of the final emotion among the plurality of art pictures as treatment information from the local storage 43 (eg, a treatment that induces a change of a sadness emotion, which is a negative emotion, into a happiness emotion, which is a positive emotion). You can acquire any one art picture that you do. Thereafter, the treatment module 40 prepares the determined emotion treatment method in a form that enables the user to perform the emotion treatment 42 based on the art picture corresponding to the treatment information obtained from the local storage 42, so that the user terminal ( 2) can be provided.

As an example, the treatment module 40 may provide an art picture corresponding to the acquired treatment information in the form of a drawing board so that the user can see and draw along the same, and provide it to the user terminal 2 . As an example, according to the emotion therapy method d4 provided to the user terminal 2 by the treatment module 40 , art corresponding to the emotion therapy method determined by the treatment module 40 is displayed in an area on the screen of the user terminal 2 . A picture (that is, an art picture obtained from local storage) is displayed, and a painting board provided so that the user can see and draw the corresponding art picture is displayed on the remaining area except for the one area on the screen of the user terminal 2 (provided) ) can be

The screen of the user terminal 2 on which the integrated emotion analysis result d3 and/or the emotion therapy method d4 is displayed (provided) is, for example, the integrated module 30 and/or the treatment module ( 40) can be controlled.

In other words, the integrated emotion analysis result d3 and/or the emotion therapy method d4 provided by the devices 110 and 110 may be displayed on the screen of the user terminal 2 through the mobile application 2a. have. To this end, the integrated module 30 and/or the treatment module 40 of the apparatuses 100 and 110 displays the integrated emotion analysis result d3 and/or the emotion therapy method d4 on the screen of the user terminal 2 . It is possible to control the mobile application 2a as much as possible.

As such, the treatment module 40 provides the emotion treatment method d4 determined in response to the integrated emotion analysis result d3 to the user terminal 2 , so that the user can use the emotion treatment method d4 provided by the device 100 . Emotional therapy can be performed based on

7 is a diagram for explaining an example in which a user's emotion treatment is performed by the emotion treatment apparatus 110 based on neural network-based emotion analysis according to an embodiment of the present application. In particular, FIG. 7 is a diagram illustrating an example of a screen of the user terminal 2 displayed by the device 110 controlling the mobile application 2a in order to perform emotion analysis and emotion treatment of the user.

Referring to FIG. 7 , for emotion treatment based on neural network-based emotion analysis by the apparatuses 100 and 110 , according to the mobile application 2a, a 'start' menu is displayed on the main screen of the user terminal 2 . , 'List menu, etc. may be provided.

When a user input is made to the 'Start' menu displayed on the screen of the user terminal 2, a voice data input button that enables input (recording) of the voice data d from the user 1 is displayed on the screen. may be provided. The user may perform an input (eg, a touch input, etc.) to the voice data input button, and then utter a voice to perform emotion analysis. The mobile application 2a of the user terminal 2 may provide the voice data d corresponding to the utterance input by the user to the apparatuses 100 and 110 . The apparatuses 100 and 110 may analyze the user's emotion (current emotion) (emotion corresponding to the voice data) by applying voice analysis and context analysis to the input voice data d.

Thereafter, in response to the input voice data d, the apparatuses 100 and 110 may calculate an integrated emotion analysis result d3 representing the user's final emotion as an analysis result. Also, the apparatuses 100 and 110 may determine an emotion treatment method d4 based on the calculated integrated emotion analysis result d3 and provide the determined emotion treatment method d4 to the mobile application 2a.

As an example, when the analyzed final emotion of the user is sadness, the apparatuses 100 and 110 induce a change in the user's negative emotion (ie, sadness emotion) into a positive emotion as an emotion therapy method d4. content (eg, art pictures according to art therapy) may be provided as the mobile application 2a. Accordingly, on the screen of the user terminal 2, an art picture for performing the art therapy method as the emotion therapy method d4 may be displayed (displayed). In this case, the devices 100 and 110 may provide not only an art picture, but also a treatment execution mode that enables the user 1 to perform emotional treatment using the corresponding art picture as the mobile application 2a. In this case, the treatment execution mode may refer to, for example, a painting board provided so that the user can see and follow the corresponding art picture. That is, the mobile application 2a may display, for example, a painting board for art therapy on the screen of the user terminal 2 as a treatment execution mode for art therapy.

According to this, the mobile application 2a is a user's emotion analysis result for the input voice data d, and the emotion therapy method (d4) and the integrated emotion analysis result (d3) including information about an art picture or a treatment execution mode (d3) may be displayed (provided) on the screen of the user terminal 2 .

As another example, when the determined emotion therapy d4 is a dance/movement therapy, a video or the like may be displayed on the screen of the user terminal 2 to allow the user to act according to a given motion as a treatment execution mode.

FIG. 8 is a diagram schematically illustrating a menu configuration in the mobile application 2a that is linked to the neural network-based emotion analysis-based emotion treatment apparatus 110 according to an embodiment of the present application. In FIG. 8, as an example, a case in which art therapy is considered as an emotion therapy will be described.

Referring to FIG. 8 , the mobile application 2a is a menu that can be displayed on the screen of the user terminal 2 and may include, for example, a recording menu and a history list menu.

The recording menu may include an art therapy sample menu and an art therapy paint menu.

The art therapy example menu may store, for example, information related to the emotion therapy method d4 received from the devices 100 and 110 when the user performs emotion therapy by the devices 100 and 110 . That is, in the art therapy example menu, example pictures of art pictures for allowing the user to perform art therapy in response to art therapy may be stored. The user can perform art therapy work such as drawing a picture by the user by considering (reference) the example pictures of a plurality of art pictures stored in the art therapy example menu. That is, the display example on the screen of the user terminal corresponding to the art therapy example menu may be the same as the example of the 'art therapy example screen before art therapy' shown in FIG. 7 as an example.

The art therapy painting board menu is a menu that can provide a screen that allows the user to draw a picture that actually performs the art therapy method determined according to the final emotion analyzed by the devices 100 and 110 (that is, provides a treatment execution mode) menu) can mean That is, the display example on the screen of the user terminal corresponding to the art therapy painting board may be the same as the example of the 'user painting board screen for art therapy' shown in FIG. 7 as an example. Based on the treatment execution mode provided through the art therapy painting board menu, the user can select a pen color for example and perform actions such as drawing according to the art therapy manual, so that art therapy can be performed. can

The history list menu may include, for example, four menus including a date history menu, a mood history menu, a voice history menu, and a picture history menu. In this history list menu, history information (date, mood, voice and picture-related history information) may be stored. Here, the feeling may mean an emotion.

There are four menu functions (date, mood, voice, picture) in the history list menu, and each function is the date and time the user used the mobile application 2a linked with the devices 100 and 110 previously viewed. You can store the entered mood, voice and picture.

The user may check the user's previous emotional state through the mood history menu. The user can check the mood at the time by opening the voice data (voice file) input (recorded) at the time when emotion analysis or emotion treatment is performed through the voice history menu. The user can check the picture that is the treatment result (for example, the result of art therapy) for the emotion therapy method that was performed to induce a positive change in the mood at the time when the emotion analysis or emotion treatment was performed through the picture history menu. .

The apparatus 100, 110 fuses a neural network (eg, a CNN neural network)-based voice analysis and an emotion dictionary (emotional dictionary)-based context analysis for the input voice data (d) for voice data (d). The user's sentiment analysis can be performed. The devices 110 and 110 provide an emotion treatment method (emotion treatment technique) to the user 1 based on the fusion-based emotion analysis result (integrated emotion analysis result, d3), that is, the user can perform emotion treatment. can make it happen

The apparatuses 100 and 110 may provide art therapy as an example of emotional therapy. According to the emotion therapy provided by the devices 100 and 110 , the user's negative emotions (Sad, Angry) or neutral emotions (Neutral) can be induced to change into positive emotions. In addition, according to the emotion therapy provided by the devices 110 and 110 , it is possible to induce the user's positive emotion (Happy) to be maintained and expanded. That is, the emotion therapy provided by the apparatuses 100 and 110 may serve to induce the user's emotion to be more positively changed or to maintain and expand the user's emotion.

When the voice data d is input, the apparatuses 100 and 110 may provide the input voice data d to the voice analysis module 10 and the context analysis module 20 , respectively. The voice analysis module 10 calculates, for example, a probability value for each of the four emotions (Happy, Neutral, Sad, Angry) as a first emotion analysis result (d1) through the processing of voice analysis for the voice data (d) can do. The context analysis module 20 is an emotional score related to the strength of positive and negative strength through the processing of the context analysis for the voice data d (that is, the emotional score related to the strength of positive/negative, one positive/negative result) value) may be calculated as the second emotion analysis result d2. Thereafter, the integrated module 30 may calculate the integrated emotion analysis result d3 including the user's final emotion by receiving the first and second emotion analysis results d1 and d2 as inputs. In other words, the integrated module 30 integrates the final emotion of the user based on the emotion data obtained from the voice analysis module 10 and the positive/negative result values obtained from the context analysis module 20 as the integrated emotion analysis result (d3) It can be calculated (extracted) as The treatment module 40 determines the emotion treatment method d4 based on the calculated integrated emotion analysis result d3, and the determined emotion treatment method d4 is displayed on the screen of the user terminal 2 on the screen of the mobile application 2a. It is possible to provide the emotional treatment method (d4) determined by Through this, the user 10 may perform emotion treatment based on the emotion therapy method d4 provided on the screen of the user terminal 2 .

9 is a diagram for explaining an example of calculation of the second emotion analysis result d2 by the context analysis module 20 in the neural network-based emotion analysis-based emotion treatment apparatus 110 according to an embodiment of the present application.

Referring to FIG. 9 , as an example, it is assumed that the input voice data d is 'When you have meet so many kind and wonderful Stepmothers.'

The context analysis module 20 may convert the input voice data d into text through a Speech To Text (STT) 21 . Thereafter, the context analysis module 20 may extract a word primitive (Lemma) from the converted text using the Google Natural Language 22 . As an example, the context analysis module 20 may extract [I, have, met, so, many, kind, and, wonderful, Stepmothers] as at least one word basic form from the voice data d.

Then, the context analysis module 20 performs matching with the word primitive included in the emotion dictionary DB (emotion dictionary DB, 23) with respect to the extracted at least one word primitive, thereby determining whether a matching word primitive exists. can be checked The context analysis module 20 may obtain an emotion score for each of the matching word basic types by performing matching between the extracted basic word types and the basic word types included in the emotion dictionary DB 23 .

In other words, the context analysis module 20 extracts a word primitive (Lemma) of a word (emotion word) representing an emotion included in the converted text from the converted text through application of the Google Natural Language 22, and extracts Through matching between the basic form of the word and the emotion dictionary DB 23 , it is possible to obtain a pre-stored index with respect to the basic form of the matching word that matches the basic form of the word extracted from the emotion dictionary DB 23 . In this case, the context analysis module 20 may acquire a pre-given emotion score for the basic matching word based on the obtained index. In this case, the emotion word DB 23 may store information corresponding to a basic word type, an index, and an emotion score in connection with each other in a lookup table format.

For example, data such as 'kind (index 3): +1.0' and 'wonderful (index 9): +2.0' may be pre-stored in the emotion dictionary DB 23 . According to this, the context analysis module 20, as the emotion score of each matching word basic type, is an emotion dictionary DB of '+3.0', which is the emotion score of kind, which is the basic type of the matching word, and '+9.0', which is the emotion score of wonderful, which is the basic type of the matching word. It can be obtained from (23).

Thereafter, the context analysis module 20 may calculate the summed emotional score as the second emotion analysis result d2 by summing the emotional scores of the matching words corresponding to each of the extracted at least one basic word form. For example, the summed emotional score (the sum of the emotional scores, Sentiment Score), that is, the emotional score for the entire sentence corresponding to the voice data (d) is +3.0 (+1.0+2.0=+3.0) (positive emotion value). can According to this, when context analysis is performed on the voice data d, it can be calculated that the user's emotion is a 'positive emotion'.

10A to 10C are diagrams for explaining an example of calculation of the first emotion analysis result d1 by the voice analysis module 10 in the neural network-based emotion analysis-based emotion treatment apparatus 110 according to an embodiment of the present disclosure; am.

In particular, FIG. 10A shows an example of speech waveform features (ie, MFCC feature values) extracted by applying the MFCC transform 11 to speech data d. 10B shows an example of a learning curve of model data obtained from the local storage 13 . 10C shows an example of the first emotion analysis result d1.

10A to 10C , as an example, it is assumed that the input voice data d is 'When you have meet so many kind and wonderful Stepmothers.'

The voice analysis module 10 may extract (detect, acquire) voice waveform features (ie, MFCC feature values) from the voice data d by applying the MFCC transformation 11 to the voice data d. Exemplarily, the voice analysis module 10 may extract 26 voice waveform features from the voice data d. The voice analysis module 10 may provide the extracted voice waveform features as an input of the neural network 12 .

Also, the voice analysis module 10 may obtain (load) model data from the local storage 13 and provide it as an input to the neural network 12 . Here, the model data stored in the local storage 13 may mean a neural network model for voice classification that is learned to classify voices from MFCC feature values or a weight value of a neural network determined through learning of the neural network model.

For example, the model data may refer to neural network model-related data (eg, weight values) obtained by learning MFCC feature values of 2000 Tess speech data. In FIG. 10B , a blue curve indicates accuracy, and the model data applied to the apparatuses 100 and 110 may be, for example, model data showing a learning accuracy of 98%.

The voice analysis module 10 applies the voice waveform features extracted from the voice data d and the model data obtained from the local storage 13 to the input of the neural network 12, so that the first emotion as a result of the neural network 12 The analysis result d1 can be calculated.

In the first emotion analysis result d1 calculated by the voice analysis module 10, probabilistically for each of a plurality of emotions including happiness, neutral, sadness, and anger. A given value (probability value) may be included. As an example, in FIG. 10c , the first emotion analysis result d1 is shown so that, as a result of applying the voice analysis to the voice data d, the emotion corresponding to 'Happy' has the largest value among the four emotions. It can be confirmed that it has been calculated. Accordingly, when voice analysis is performed on the voice data d, it may be calculated that the user's emotion is 'Happy'.

The result of applying the voice analysis to the voice data d may be applied (adjusted) so that the sum becomes 1 by using a Softmax function in the final step. According to this, the emotion analysis result (that is, the first emotion analysis result, d1) shown in FIG. 10C is a result of applying the voice analysis to the voice data d, and exemplarily means a value before the softmax function is applied. can do.

11 is a diagram for explaining an example of calculation of the integrated emotion analysis result d3 by the integration module 30 in the neural network-based emotion analysis-based emotion treatment apparatus 110 according to an embodiment of the present application.

Referring to FIG. 11 , the integration module 30 provides the first emotion analysis result d1 and the second emotion analysis result d2 as inputs to the neural network 31 to perform fusion (integration) analysis, so that voice data As an integrated emotion analysis result (d3) for (d), a probability for each of a plurality of emotions may be calculated.

As a result of the integrated analysis of the first and second emotion analysis results (d1, d2), the integration module 30 sets the probability corresponding to happiness to the highest value (higher value) among the probabilities for the four emotions. As it appears, the user's final emotion for the voice data d may be calculated as 'Happy'. That is, the integration module 30 may calculate the final emotion of the user as 'Happy' because the emotion having the maximum probability among the plurality of emotions is 'Happy' through the integrated analysis. The integrated module 30 may calculate the integrated emotion analysis result d3 as the result of the integrated analysis (integrated analysis performed together with the voice analysis and the context analysis) in which 'Happy' is the final emotion of the user.

As an example, the integrated analysis emotion value shown in FIG. 11 is related to the integrated emotion analysis result (d3) calculated by the integration module 30, happiness (Happy), neutral (Neutral), sadness (Sad) and anger ( Angry) may mean a final result value for each of a plurality of emotions. This is the result of applying the voice analysis (ie, the first emotion analysis result, d1) to the voice data d calculated by the voice analysis module 10 whose sum is adjusted to 1 by using the softmax function, and the context analysis. It may mean a final result value obtained by secondary processing together with the second emotion analysis result d2, which is the natural language analysis result calculated by the module 20 . The sum of the final result values for each of the plurality of emotions may be 1 or may not be 1.

The treatment module 40 may determine the emotion treatment method d4 based on the integrated emotion analysis result d3 and provide it to the user terminal 2 . For example, when the final emotion of the user included in the integrated emotion analysis result d3 is shown to be a positive emotion as 'Happy', the treatment module 40 is converted into a content form that induces maintenance and expansion of the positive emotion. Emotional therapy (d4) can be provided. As an example, the treatment module 40 may provide a painting board to enable emotional treatment for 'drawing a happy figure' as the emotion therapy method d4.

The apparatuses 100 and 110 analyze the voice data d of the user 1 through the text-based context analysis module 20 and the neural network-based voice analysis module 10 to obtain respective emotional scores d1 and d2. After extraction, the final emotion (final emotion) of the user may be calculated (derived) as the analysis result d3 by integrating it in the integration module 30 .

The apparatuses 100 and 110 may provide the emotion therapy method d4 according to the result d3 of the integrated analysis. The devices 100 and 110 may provide an emotion therapy method d4 for art therapy or content recommendation (providing content such as movies, music, books, etc.) to the user terminal 2 according to the integrated analysis result d3. . Through this, the apparatuses 100 and 110 can solve problems related to the user's stress and emotions by providing the emotion treatment method d4.

For example, when the user's final emotion (the current final emotion) is 'Sad', the devices 100 and 110 change the final emotion, a sad emotion, into a positive emotion (for example, a happy emotion). As an emotional therapy method (d4) that can be performed, art therapy may be provided as an example.

The apparatuses 100 and 110 may analyze a user's emotions by combining voice analysis and context analysis. The emotion analysis method using voice analysis alone may have a problem in that it is difficult to recognize a neutral intonation, and even for the same sentence, emotions may be determined differently depending on an individual's anti-linguistic characteristics.

In the past, several studies have been conducted in relation to emotion analysis technology through voice, but most of them have a problem of showing a relatively low accuracy with an accuracy of 60 to 70%. In addition, most of the conventionally known emotion analysis techniques only perform emotion analysis using data such as images, and there is no technology for performing emotion analysis by fusion of context analysis and voice analysis. In addition, conventional natural language processing-based emotion analysis techniques have a problem in that accurate emotion recognition cannot be achieved when input voice data does not include words representing emotions (emotion words, emotion words).

Specifically, the conventional emotion analysis technique using voice analysis alone has the following problems. It is difficult to recognize when an emotional sentence is spoken with a neutral accent, and since it is the same sentence, the emotional analysis result may be determined (derived) differently due to individual differences.

One example of a conventional technique for recognizing human emotions is [Yelin Kim and Emily Mower Provost. 2015. Emotion Recognition During Speech Using Dynamics of Multiple Regions of the Face. ACM Trans. Multimedia Comput. Commun. Appl. 12, 1s, Article 25 (October 2015), 23 pages.] literature exists.

The above conventional literature proposes an extension technique of a Deep Belief Network, and is different from the technique proposed herein in that it does not use a neural network (Deep Learning) technique. In addition, the emotion analysis of the related art shows a low accuracy with an accuracy of 60 to 70%, and there is a difference in that emotion analysis is performed based on an image of a face.

On the other hand, the conventional emotion analysis technology based on self-language processing has the following problems. When the voice data obtained from the user does not include a word (emotional word) representing an emotion, it is difficult to recognize the emotion. In addition, even if emotional words are not included in the acquired voice data, emotional sentences may appear in the acquired voice data due to inter-word dependencies or voice characteristics, etc. have.

12 is a view showing an example of a screen display of an emotion analysis result by applying a conventional natural language processing to a given voice data. 13 is a screen display example of the emotion analysis result through the application of context analysis by the context analysis module 20 of the neural network-based emotion analysis-based emotion treatment apparatus 110 for the given voice data according to an embodiment of the present application. the drawing shown.

12 and 13 , the emotion analysis result shown in FIG. 12 shows an example in which Google's Google Natural Language is applied as a conventional natural language processing to voice data. On the other hand, the emotion analysis result shown in FIG. 13 is an emotion analysis technology that improves the conventional emotion analysis technology by using a Dependency Tree and a formula relation structure of the conventional Google Natural Language (Natural Language Processing) for voice data. An example of the applied case (ie, the case where the context analysis technique by the present device is applied) is shown.

As will be described later, the improved emotion analysis technology (that is, emotion analysis technology based on context analysis by the context analysis module of the device) applied to the present application simply performs emotion analysis using Google Natural Language, which is a conventional natural language processing technology. It can be seen that the performance is better than

Specifically, FIGS. 12 and 13 are voice data that are subject to emotional analysis, indicating that "I can't be happy sometimes not being with my family makes me feel so lonely" The fact sometimes makes me so lonely.)” It represents the emotion analysis result in the case where the sentence is considered.

Since expressions such as “can’t be happy” and “lonely” are included in an example of the given voice data, it may be desirable to determine that the given voice data is an overall negative sentence. That is, according to an example of the given voice data, it may be desirable to determine that the user's emotion for the given voice data corresponds to a negative emotion.

However, referring to FIG. 12, in the conventional Google Natural Language, an emotion score is given as 0.2 as a result of performing emotion analysis on the given voice data, and accordingly, it is determined that the user's emotion toward the voice data is 'neutral emotion'. can confirm. It can be seen that the conventional natural language processing-based emotion analysis result is not accurate.

On the other hand, referring to FIG. 13 , in the context analysis module 20 of the apparatuses 100 and 110 , the emotion score is -2.5 as the result of performing emotion analysis on the given voice data (ie, the second emotion analysis result, d2). , and accordingly, it can be confirmed that the user's emotion toward the voice data is determined to be a 'negative emotion'. That is, the apparatuses 100 and 110 may derive a result that the sentences of the given voice data generally include negative emotions through the application of the context analysis by the context analysis module 20 .

It can be confirmed that the emotion analysis result based on the context analysis by the context analysis module 20 of the present devices 100 and 110 derives more accurate results compared to the conventional self-language processing (eg, Google Natural Language) technology. .

The present application may provide neural network-based emotion analysis and emotion treatment systems 200 and 200 ′. The present systems 200 and 200' can analyze the user's emotions for the input voice data by fusion of voice analysis and context analysis based on emotion dictionary, and provide appropriate emotion therapy and content to the user according to the analysis result. . The present systems 200 and 200 ′ may be referred to as a Neural Network Based Sentiment Analysis and Therapy system (NBST).

The devices 100 and 110 and the systems 200 and 200 ′ perform emotion analysis by fusion of voice analysis and emotion dictionary (emotional dictionary)-based context analysis on the input voice data, and then based on the analysis results. can provide the user with a treatment technique (emotional therapy). Art therapy may be considered as an example of emotional therapy, and negative emotions of the user may be positively changed by providing such an emotional therapy.

The apparatuses 100 and 110 may provide the emotional treatment method d4 determined in the treatment module 40 to the mobile application 2a, thereby helping the user to perform an emotion treatment corresponding to the determined emotion treatment method d4.

In other words, the emotion therapy method d4, such as art therapy, which is considered in the present systems 200 and 200', may serve to change the negative emotion of the user 1 into a positive one. When it is found that the final emotion of the user is a negative emotion or a neutral emotion as a result of the integrated analysis, the apparatuses 100 and 110 may provide an emotion therapy method d4 for inducing a change in the negative emotion or the neutral emotion into a positive emotion. In addition, when the final emotion of the user is found to be a positive emotion as a result of the integrated analysis, the devices 100 and 110 provide the user with an emotion therapy method d4 for content such as movies, music, books, etc., so that the user's current emotion is positive. Emotions can be maintained and/or expanded.

Conventionally, most mobile applications utilize text-based emotion analysis, and none of these mobile applications has an element capable of causing a user to change their emotions. That is, most conventional mobile applications that perform emotion analysis do not consider a technology for changing a user's emotion at all. On the other hand, the present systems 200 and 200' analyze the user's emotions identified by convergence of voice analysis and context analysis to determine the user's emotional change, and provide emotional therapy such as art therapy according to the result. can do. The present systems 200 and 200' may change the user's emotion into a positive emotion when the user's current emotion is expressed as a negative emotion or a neutral emotion through the provision of emotion therapy.

In other words, the present systems 200 and 200' can recognize the user's emotional change by merging voice analysis and contextual analysis to identify the user's emotions with respect to the input voice data, accumulate them, and analyze them in a time-series manner. Based on this result, the present systems 200 and 200' can recommend (provide) an emotion therapy (treatment technique) and content to the user.

The present systems 200 and 200' may change into a positive emotion when the user currently shows a negative emotion or a neutral emotion as a result of the integrated emotion analysis. In addition, the present systems 200 and 200' can recommend and provide content such as movies, music, books, etc., when the user currently shows positive emotions as a result of the integrated emotion analysis (that is, for users who show positive emotions), This can help the user's current emotions, such as positive emotions, to be maintained or expanded.

The present systems 200 and 200' can be applied to Market Research, which occupies the largest portion of the global emotional (emotional) computing market. In addition, the present systems 200 and 200' may be directly utilized in the field of health care. In addition, the present systems 200 and 200' are applicable to the psychotherapy market, and it is possible to provide emotional treatment through a personal mobile application.

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

14 is an operation flowchart of a neural network-based emotion analysis method according to an embodiment of the present application.

The neural network-based emotion analysis method shown in FIG. 14 may be performed by the apparatus 100 described above. Therefore, even if omitted below, the description of the apparatus 100 may be equally applied to the description of the neural network-based emotion analysis method.

Referring to FIG. 14 , in step S11, the voice analysis module may calculate a first emotion analysis result by applying neural network-based voice analysis to voice data input from the user.

In this case, in step S11, the voice analysis module may calculate the first emotion analysis result through neural network-based voice analysis using the voice waveform features extracted from the voice data as an input.

Here, the voice waveform feature may be an MFCC feature value output by performing MFCC (Mel-Frequency Cepstral Coefficient) transformation on voice data.

Also, the first emotion analysis result may be a value probabilistically assigned to each of a plurality of emotions including happiness, neutrality, sadness, and anger.

Next, in step S12, the context analysis module may calculate the second emotion analysis result by applying the context analysis based on the emotion dictionary to the voice data input from the user.

At this time, in step S12, the context analysis module converts the voice data into text, then extracts at least one word primitive (Lemma) from the converted text, and extracts the extracted at least one word primitive type from the word primitive included in the emotion dictionary DB. It is possible to calculate a second emotion analysis result by matching with .

Here, the emotion dictionary DB may be constructed in a form in which emotion scores related to the strength of positive and negative strength are given to each of the plurality of basic types of words.

In step S12, the context analysis module may calculate the second emotion analysis result in consideration of the emotion scores of each of the matching word basic types matching the word basic types included in the emotion dictionary DB among the extracted at least one basic word types.

Next, in step S13, the integrated module may output the integrated emotion analysis result for voice data by using the first emotion analysis result calculated in step S11 and the second emotion analysis result calculated in step S12 as input to the neural network. have.

Here, the integrated emotion analysis result is primarily derived in the form of a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sad, and Angry, It may be a result of determining the emotion having the maximum probability among the emotions as the final emotion.

In the above description, steps S11 to S13 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

15 is an operation flowchart of an emotion treatment method based on a neural network-based emotion analysis according to an embodiment of the present application.

The emotion treatment method based on the neural network-based emotion analysis shown in FIG. 15 may be performed by the apparatus 110 described above. Therefore, even if omitted below, the description of the apparatus 110 may be equally applied to the description of the emotion treatment method based on the neural network-based emotion analysis.

Each of steps S21 to S23 in FIG. 15 may be the same as each of steps S11 to S23 in FIG. 14 described above. Therefore, hereinafter, it will be briefly described in relation to steps S21 to S23.

Referring to FIG. 15 , in step S21 , the voice analysis module may calculate a first emotion analysis result by applying neural network-based voice analysis to voice data input from the user. Next, in step S22, the context analysis module may calculate the second emotion analysis result by applying the context analysis based on the emotion dictionary to the voice data input from the user. Next, in step S23, the integrated module may output the integrated emotion analysis result for voice data by using the first emotion analysis result calculated in step S21 and the second emotion analysis result calculated in step S22 as input to the neural network. have.

Next, in step S24, the treatment module may determine and provide an emotion treatment method based on the integrated emotion analysis result output and provided in step S23.

Here, the emotion therapy may be provided in the form of content that induces a change to a positive emotion with respect to the user's emotion when the user's emotion is shown to be a negative emotion or a neutral emotion as a result of the integrated emotion analysis.

In addition, the emotion therapy method may be provided in the form of content that induces positive maintenance and expansion of the user's emotion when the user's emotion is found to be a positive emotion as a result of the integrated emotion analysis.

In the above description, steps S21 to S24 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The neural network-based emotion analysis method according to an embodiment of the present application and the neural network-based emotion analysis-based emotion treatment method according to an embodiment of the present application are implemented in the form of program instructions that can be performed through various computer means and stored in a computer-readable medium. can be recorded. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the above-described neural network-based emotion analysis method and neural network-based emotion analysis-based emotion treatment method may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

200: Neural network-based emotion analysis system
200': Neural network-based emotion analysis-based emotion treatment system
100: neural network-based emotion analysis device
110: Neural network-based emotion analysis-based emotion treatment device
10: voice analysis module
20: Context Analysis Module
30: integrated module
40: treatment module
2: User terminal

Claims (18)

A neural network-based emotion analysis method performed by a neural network-based emotion analysis device, comprising:
(a) calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user;
(b) calculating a second emotion analysis result by applying context analysis based on emotion dictionary to the voice data; and
(c) outputting an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as inputs to a neural network;
including,
The first emotion analysis result is a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sadness, and Angry,
The second emotion analysis result includes positive, negative, and neutral, but based on the emotional score corresponding to the neutral, the positive is a score form greater than the emotional score corresponding to the neutral, and the emotional score related to the strength of the positive emotion , and negation is an emotional score related to the intensity of negative emotion in the form of a score smaller than the emotional score corresponding to the neutral,
In the step (b), after converting the speech data into text, at least one word primitive (Lemma) and a dependency are extracted from the converted text, and the extracted at least one word primitive type is stored in an emotion dictionary DB. Calculating the second emotion analysis result by matching with the included word basic form, calculating the second emotion analysis result in consideration of the dependency,
In step (c), the neural network is
When the anti-linguistic feature does not exist in the voice data, the second emotion analysis result is higher than the first emotion analysis result because the first emotion analysis result is a value probabilistically assigned to each of the plurality of emotions. Constructed to derive the integrated emotion analysis results in consideration of further,
When an emotion score corresponding to neutral is given to the second emotion analysis result because an emotion word representing the user's emotion does not exist in the voice data, the maximum of the probability values for each of the plurality of emotions in the first emotion analysis result A neural network-based emotion analysis method that is constructed to calculate the integrated emotion analysis result with an emotion given a probability value. According to claim 1,
In the step (a), the first emotion analysis result is calculated through the neural network-based speech analysis using the speech waveform feature extracted from the speech data as an input. 3. The method of claim 2,
The voice waveform feature is an MFCC feature value output by performing Mel-Frequency Cepstral Coefficient (MFCC) transformation on the voice data, a neural network-based emotion analysis method. delete delete According to claim 1,
The emotion dictionary DB is constructed in a form in which emotion scores related to the strength of positive and negative strength are given to each of a plurality of basic types of words,
In the step (b), the second emotion analysis result is calculated in consideration of the emotion score of each of the matching word basic types matching the word basic types included in the emotion dictionary DB among the extracted basic types of the at least one word, A neural network-based emotion analysis method. According to claim 1,
In step (c), the result of the integrated emotion analysis is 1 in the form of a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sadness, and Angry. A neural network-based emotion analysis method, which is a result of determining an emotion, which is derived sequentially and has a maximum probability among the plurality of emotions, as the final emotion. delete delete A neural network-based emotion analysis device, comprising:
a voice analysis module for calculating a first emotion analysis result by applying a neural network-based voice analysis to voice data input from a user;
a context analysis module for calculating a second emotion analysis result by applying an emotion dictionary-based context analysis to the voice data; and
an integrated module for outputting and providing an integrated emotion analysis result for the voice data by using the first emotion analysis result and the second emotion analysis result as input to a neural network;
including,
The first emotion analysis result is a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sadness, and Angry,
The second emotion analysis result includes positive, negative, and neutral, but based on the emotional score corresponding to the neutral, the positive is a score form greater than the emotional score corresponding to the neutral, and the emotional score related to the strength of the positive emotion , and negation is an emotional score related to the intensity of negative emotion in the form of a score smaller than the emotional score corresponding to the neutral,
The context analysis module,
After converting the speech data into text, at least one word primitive type (Lemma) and a dependency are extracted from the converted text, and the extracted at least one word primitive type is matched with the word primitive type included in the emotion dictionary DB. Calculating the second emotion analysis result, characterized in that the second emotion analysis result is calculated in consideration of the dependency,
The neural network is
If the anti-verbal feature does not exist in the voice data, the second emotion analysis result is more important than the first emotion analysis result because the first emotion analysis result is a value probabilistically assigned to each of the plurality of emotions. Constructed to derive the integrated emotion analysis result in further consideration,
When an emotion score corresponding to neutral is given to the second emotion analysis result because the emotion word representing the user's emotion does not exist in the voice data, the maximum of the probability values for each of the plurality of emotions in the first emotion analysis result A neural network-based emotion analysis device that is constructed to calculate the integrated emotion analysis result with an emotion given a probability value. 11. The method of claim 10,
wherein the voice analysis module calculates the first emotion analysis result through the neural network-based voice analysis to which voice waveform features extracted from the voice data are input. 12. The method of claim 11,
The voice waveform feature is an MFCC feature value output by performing Mel-Frequency Cepstral Coefficient (MFCC) transformation on the voice data, a neural network-based emotion analysis apparatus. delete delete 11. The method of claim 10,
The emotion dictionary DB is constructed in a form in which emotion scores related to the strength of positive and negative strength are given to each of a plurality of basic types of words,
The context analysis module is configured to calculate the second emotion analysis result in consideration of the emotion score of each matching word primitive matching the word primitive included in the emotion dictionary DB among the extracted primitive types of the at least one word based sentiment analysis device. 11. The method of claim 10,
The integrated emotion analysis result is primarily derived in the form of a value probabilistically assigned to each of a plurality of emotions including happiness, neutral, sad, and Angry, and A neural network-based emotion analysis device that is the result of determining the emotion with the highest probability among the emotions as the final emotion. As an emotion treatment device based on neural network-based emotion analysis,
A neural network-based emotion analysis-based emotion treatment device that determines and provides an emotion treatment method based on the integrated emotion analysis result provided by performing a neural network-based emotion analysis by the neural network-based emotion analysis device according to claim 10 . 18. The method of claim 17,
The emotion therapy is
When it is shown that the user's emotion is a negative emotion or a neutral emotion as a result of the integrated emotion analysis, it is provided in the form of content that induces a change to a positive emotion with respect to the user's emotion,
When the user's emotion is found to be a positive emotion as the result of the integrated emotion analysis, the neural network-based emotion analysis-based emotion treatment apparatus will be provided in the form of content that induces positive maintenance and expansion of the user's emotion.

Images