KR20200025532A - An system for emotion recognition based voice data and method for applications thereof - Google Patents

Abstract

Disclosed are an emotion recognition system based on voice data and an application method thereof. The emotion recognition system based on voice data comprises: a user terminal providing a vital voice signal; an emotion analysis server generating a customized emotion value based on user voice information received from the user terminal to transmit the customized emotion value to the user terminal or an emotion value application server and an emotion adjustment hardware; the emotion adjustment hardware connected to the emotion analysis server through a wired/wireless communication network; and the emotion value application server. According to the present invention, a voice signal obtained from the emotion analysis server may be utilized in the user terminal, the emotion value application server, and the emotion adjustment hardware connected to the emotion analysis server through the wired/wireless communication network.

Description

An system for emotion recognition based voice data and method for applications

The present invention relates to a voice data-based emotion recognition system and an application method thereof, and more particularly, to collect big voice data of an individual to make big data, and deep learning by supervised learning using the big data. The present invention relates to a voice data-based emotion recognition system and an application method thereof that can derive personal emotion information and send it to a terminal, another server, or a separate emotion control hardware device.

With the development of mobile communication technology, the third generation mobile communication network (for example, W-CDMA network or HSDPA network) capable of transmitting large amounts of data at high speed is established, and it is possible to break away from the existing voice communication service. It is also possible to make a video call while looking directly at the face through the video. Accordingly, mobile carriers are developing and providing call services with various special functions in order to preempt subscribers of third generation mobile communication networks first.

These specially designed services not only add more fun and interest to the call by exchanging multimedia content such as emoticons, images, flashes and videos between the caller and the caller, but also further analyze the environment of the calling party through voice analysis. It is becoming more diverse, ranging from emotional call services that also show status and emotion. Speech recognition is the quantification by analyzing the vibrations of people's phonemes, syllables, words, etc., due to the different characteristics of human accent and pitch. It's a way of finding out.

However, the existing emotion analysis system using voice data does not consider the language feature information of the individual, so there is a problem in that accurate emotion analysis is limited.

That is, conventional emotion data analysis has a problem that uniform emotion analysis is performed without considering the voice characteristics of each individual.

Patent Document 0001) Korea Patent Publication 10-2010-0001928

The problem to be achieved by the technical idea of the present invention is to make a big data of the biometric voice signal of an individual obtained from a terminal such as a smart phone, and deep learning by using the supervised learning using a plurality of emotions during a specific cycle It is to provide a voice data-based emotion recognition system that can be inferred as one of them and utilized in the terminal, emotion value application server and emotion control hardware device.

Another object of the technical idea of the present invention is to make a big data of a biometric voice signal of an individual obtained from a terminal such as a smart phone, and use the same to conduct deep learning by using supervised learning. It is to provide an application method of a speech data-based emotion recognition system that can be inferred as one of emotions and utilized in a terminal, an emotion value application server, and an emotion control hardware device.

According to an embodiment of the present invention, a voice data-based emotion recognition system extracts user voice information including a pitch frequency, a volume decibel, and a speech rate from voice data input from a user, and transmits the user voice information to an emotion analysis server. Calculating a user terminal for receiving and displaying a user-defined emotion value based on the user voice information from the emotion analysis server, and an objective emotion value indicating an objective emotion based on the user voice information received from the user terminal, The emotion weight is calculated by comparing the user's voice information received from the stored user's voice accumulated information and calculating the user's individual emotion value, and generating a user-defined emotion value to which the emotion value is added to the objective emotion value and transmitting the emotion weight to the user terminal. An emotion server configured to receive a control command according to a customized emotion value calculated by the emotion analysis server, and to perform an operation that may affect the emotion control of the user; Equipped with a wired or wireless communication network that provides a wired or wireless communication by connecting the emotion value application server and the terminal, the emotion analysis server, the emotion control hardware device and the emotion value application server that can receive the customized emotion value as a reference material. It features.

According to an embodiment of the present invention, the emotion control hardware device, the Internet of Things (IoT) therein, and includes a configuration that can perform a command received from the emotion analysis server as the user's operating environment It may be any one selected from a process system consisting of an air conditioner, a heater, and a humidity control device exposed to or an artificial intelligence mounted therein and having an artificial intelligence capable of talking to the user.

In addition, according to an embodiment of the present invention, the emotion analysis server, an objective emotion value DB is assigned and stored for each voice information including voice frequency, voice decibels and voice speed, and user voice information received from the user terminal. Cumulatively stored user voice accumulated information DB, an objective emotion value analysis unit for extracting an objective emotion value matching the user voice information received from the user terminal from the objective emotion value DB, and the user voice accumulated information DB Emotion weights for extracting user voice average information including average voice frequency of the user, average voice decibels, and average voice speed, and calculating emotion weights by comparing the user voice information received from the user terminal with the user voice average information. To the calculation unit and the objective emotion value A user-customized emotion value providing unit may be provided to generate a user-customized emotion value obtained by adding the emotion weights to the user terminal.

According to an embodiment of the present invention, the user terminal includes a terminal input unit for receiving the voice data, a terminal display unit for displaying the user-specific emotion value, and a voice frequency and voice decibel from the voice data input from the user. And a user voice information providing unit for extracting user voice information including voice speed and transmitting the user voice information to the emotion analysis server, and a terminal controller for receiving a user-customized emotion value based on the user voice information from the emotion analysis server and recording it on the display unit. It is suitable to provide.

In addition, the user terminal further includes a background music DB, in which background music is allocated and stored for alleviating stress for each stress index, and the terminal controller further includes a stress index of a user-customized emotion value exceeding a preset threshold stress. In this case, it is preferable to extract and play the background music allocated to the stress index of the user from the background music DB, and to play the background music matching the received stress index by receiving the user's stress index in real time.

The user terminal may include a stress relaxation image DB in which a stress relaxation image capable of restoring stress is allocated and stored, and the terminal controller may include: when the stress index of the user-customized emotion value exceeds a preset threshold stress, The stress relief image allocated to each user's stress index is extracted from the stress relaxation image DB and displayed through a display unit, and the user's stress index is received in real time and displayed as a stress relaxation image matching the received stress index. Suitable.

According to another aspect of the present invention, there is provided a method of applying a speech data-based emotion recognition system, including providing speech information through a user in a terminal, and determining the speech information provided by the terminal in terms of volume decibel, pitch frequency, and speech. Analyzing the voice biosignal at the speed; transmitting the analyzed voice biosignal to the emotion analysis server; and collecting big data on the voice biosignal accumulated for each individual who provided the voice information from the emotion analysis server. Collecting, and performing deep learning by supervised learning using big data collected by operating an emotion analysis algorithm in the emotion analysis server, and volume decibels of the voice biosignal. , Based on the pitch frequency and voice speed, the user's emotions can be And classifying as one of information, it may be provided with the step of transmitting an emotion classified into one of the plurality of basic emotions in the emotion analysis server and the emotion value of the application server in conjunction with a wired or wireless communication network.

On the other hand, the method of deep learning by the supervised learning, the boundary value of the first emotion, the second emotion and the third emotion from the scatter diagram of the average value for each period of the volume decibels, and the scatter diagram of the average value for each period of the pitch frequency Determining the threshold value of the fourth emotion, the fifth emotion and the sixth emotion, and determining the threshold value of the seventh emotion, the eighth emotion and the ninth emotion in the scatter diagram of the average value for each period of the speech speed.

According to an embodiment of the present invention, the method of classifying the emotion of the user into one of a plurality of basic emotions for a predetermined period, obtains one emotion (P1) calculated based on the volume decibels, and based on the pitch frequency After obtaining another emotion (P2) calculated by, and another emotion (P3) calculated on the basis of the speech speed, one of the three emotions P1, P2, P3 obtained from the most out of the boundary value It is appropriate to choose emotions.

Preferably, before transmitting the emotion classified as one of the plurality of basic emotions to the emotion value application server, a control command corresponding to the selected emotion as one of the plurality of basic emotions is connected to the emotion analysis server through a wired / wireless communication network. The method may further include transmitting to the emotion control hardware device.

In this case, the emotion control hardware device is one selected from the group of devices that can affect the user's emotion consisting of the artificial intelligence speaker and the air conditioning device for controlling the temperature and humidity.

Therefore, according to the technical idea of the present invention, first, the user's voice information is collected in a terminal such as a smart phone and subdivided into other forms of biometric voice signals, for example, volume decibels, pitch frequencies, and voice speeds. By making big data, the big data can be analyzed by deep learning by supervised learning, and the user's emotions during a specific cycle can be inferred as a specific customized emotion value closest to the plurality of emotions.

Secondly, send the inferred personalized emotion value to the user terminal or an emotion control hardware device that the user is exposed to the operating environment, such as an artificial intelligence-equipped artificial speaker or an air conditioner with a temperature and humidity control function that can control emotion. This can be used actively.

Third, the personalized emotion value obtained by inferring from the emotion analysis server may be transmitted to an emotion value application server connected through a wired / wireless communication network, for example, to a server capable of collecting medical records of a user, and may be used as a reference for medical records.

1 is a block diagram for explaining the configuration of a voice data-based emotion recognition system according to an embodiment of the present invention.
2 is a flowchart illustrating a procedure for recognizing emotion based on voice data according to an embodiment of the present invention.
3 is a block diagram illustrating a configuration of an emotion analysis server according to an exemplary embodiment of the present invention.
4 is a diagram illustrating a stress correction value according to a job of a user according to an exemplary embodiment of the present invention.
Figure 5 is a diagram showing the fatigue correction value according to the weather in which the emotional analysis according to an embodiment of the present invention.
6 is a block diagram illustrating a configuration of a user terminal according to an embodiment of the present invention.
7 is a display screen of a user terminal displaying a stress relief image according to an embodiment of the present invention.
8 is a flowchart illustrating processes of emotion recognition based on voice data according to the first embodiment of the present invention.
9 is a flowchart for explaining a voice data based emotion recognition process and a method of applying the same according to a second embodiment of the present invention.
FIG. 10 is a conceptual diagram illustrating a voice data based emotion recognition process and a method of applying the same according to a second embodiment of the present invention.
FIG. 11 is a graph for explaining a method of calculating deep data and customized emotion values according to the big data and supervised learning of FIG. 9.
12 is a block diagram illustrating various types of customized emotion values of FIG. 9.

DETAILED DESCRIPTION In order to fully understand the constitution and effects of the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be embodied in various forms and various changes may be made. The embodiments introduced below are provided only to make the disclosure of the present invention complete, and to fully inform the scope of the invention to those skilled in the art.

Singular expressions include plural expressions unless the context clearly indicates otherwise. For example, the terms "comprises" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described on the specification, and one or more other features or numbers. It can be interpreted that steps, actions, components, parts or combinations thereof can be added.

Voice information in the present invention is meant to encompass the voice signal provided by the user to the terminal and the biological voice signal obtained by extracting it from the terminal. Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries are to be interpreted as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined in this application. .

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a voice data based emotion recognition system according to an embodiment of the present invention, Figure 2 is a flow chart for recognizing the voice data based emotion according to an embodiment of the present invention.

1 and 2, in the voice data-based emotion recognition system according to the present invention, a wired / wireless communication network 100, a user terminal 200, an emotion analysis server 300, an emotion value application server 400, and an emotion Adjustment hardware device 500 may be included.

The wired / wireless communication network 100 provides wired or wireless communication between the user terminal 200, the emotion analysis server 300, the emotion value application server 400, and the emotion control hardware device 500. When the wired / wireless communication network 100 is implemented as a wireless communication network, a wireless mobile communication network including a base transceiver station (BTS), a mobile switching center (MSC), and a home location register (HLR) is used. Data communication is possible. In addition, when the wired / wireless communication network 100 is implemented as a wired communication network, it may be implemented as a network communication network, and data communication may be performed according to an Internet protocol such as Transmission Control Protocol / Internet Protocol (TCP / IP).

The user terminal 200 is a terminal used by a user performing an emotion analysis. In the drawing, a smart phone is described as an example, but means for collecting voice information as well as a smartphone, for example, a desktop. PC (desktop PC), tablet PC (tablet PC), slate PC, notebook computer, digital broadcasting terminal, PDA (Personal Digital Assistants), Portable Multimedia Player (PMP), navigation, A digital camera, an MP3P (MPEG layer 3 Player), and the like may correspond. Of course, the terminal to which the present invention is applicable is not limited to the above-described type, and of course, may include all terminals capable of communicating with external devices.

The user terminal 200 extracts a voice biosignal, which is user voice information including voice frequency, voice decibels, and voice speed, from the voice data input from the user, and transmits the voice biosignal to the emotion analysis server 300. Performs a function of receiving and displaying a user-specific emotion value based on the emotion analysis server 300. In this case, a method of extracting voice data provided by a user into a biometric signal such as voice frequency, voice decibel, and voice speed may include a known application programming interface (API) already installed and operated in a user terminal. It is possible by driving.

Herein, the voice data input from the user may correspond to voices that can be input to the terminal, such as voices in voice / video calls, recorded voices, real-time input voices, and the like, and may correspond to voices in various forms and in various situations. have. In addition, the extracted user voice information may correspond to a frequency (Hertz) of the voice, a decibel of the voice, a speech speed, and the like. The user terminal 200 will be described in detail with reference to FIGS. 6 and 7 to be described later.

The emotion analysis server 300 has the same configuration as a conventional web server in hardware, and is implemented through various types of languages such as C, C ++, Java, Visual Basic, Visual C, etc. in software. Contains program modules that function. In addition, it can be implemented by using a web server program that is variously provided according to operating systems such as DOS, Windows, Linux, Unix, Macintosh, etc. in general server hardware. have.

The emotion analysis server 300 calculates an objective emotion value indicating an objective emotion determined based on preset reference voice information based on user voice information received from the user terminal 200. In addition, by comparing the user voice information received from the user terminal 200 and the cumulatively stored user voice cumulative information to calculate the emotion weight of the individual user emotion. In addition, a user-customized emotion value to which an emotion weight is added to the objective emotion value is generated and transmitted to the user terminal 200, the emotion value application server 400, and the emotion adjustment hardware device 500. An example of the method of generating the customized emotion value will be described once again with reference to FIGS. 11 and 12.

The emotion value application server 400 may be connected to the emotion analysis server 300 through a wired or wireless communication network 100. The emotion value application server 400 may receive a customized emotion value from the emotion analysis server 300. In this case, the emotion value application server may be a server that manages the information of the institution in which the personal medical record of the user of the terminal 200 is collected and stored.

The personal medical record may be a medical record indicating brain waves, pulse waves, blood pressure, and various body conditions. Therefore, the emotion value application server 400 analyzes the correlation by linking / comparing the personalized emotion value of the user of the terminal 200 provided from the emotion analysis server 300 with the personal medical record held therein to more effectively perform medical care. You can perform an action. Therefore, the customized value provided by the emotion analysis server 300 can be compared more efficiently than when it is not provided. Here, the emotion value application server 400 is not limited to a server that stores an individual's medical record, but may be applied to other types of servers as long as the field can be applied by referring to a personalized emotion value for each cycle.

The emotion control hardware device 500 is connected to the emotion analysis server 300 and the wired / wireless communication network 100 to become a means for more actively utilizing the customized emotion value obtained from the emotion analysis server 300. To this end, the emotion control hardware device 500 is configured to be capable of communicating with the Internet of Things (IoT) therein, and receives a control command based on a customized emotion value inferred from the emotion analysis server, for example, an operation command to expose the terminal user. It is appropriate to have an internal configuration that can perform actions that can affect the user's emotional control in the environment. For example, the emotion control hardware device 500 may be any one selected from a process system including an air conditioner, a heater, and a humidity control device that the user is exposed to the operating environment, or an artificial intelligence may be installed therein to communicate with the user. It can be an AI speaker that can.

Therefore, when the personalized emotion value of the terminal 200 user is excited, the temperature of an air conditioning device such as an air conditioner used by an individual may be lowered by a predetermined range, or, in the case of an artificial intelligence speaker, a conversation mode that may be dissipated may be specified. . In the same manner, when the user's personalized emotion value is in a tense state, the temperature of an air conditioning device such as a heater used by an individual is increased by a predetermined range, or in the case of an artificial intelligence speaker, a conversation mode that can bring emotions comfortably. Can be specified. Accordingly, the emotion value application server 400 and the emotion control hardware device 500 may be means for actively utilizing the customized emotion value calculated by the emotion analysis server. In addition, the emotion control hardware device 500 according to the present invention can be widely applied to other hardware devices that can control the emotions of the user of the terminal 200, without being limited to the temperature and humidity control air conditioning device and the artificial intelligence speaker. to be.

Hereinafter, the emotion analysis server 300 will be described with reference to FIGS. 3 to 5.

3 is a block diagram illustrating an emotion analysis server according to an embodiment of the present invention, FIG. 4 is a diagram showing a stress correction value according to a user's occupation according to an embodiment of the present invention, and FIG. 5 is an embodiment of the present invention. This figure shows the fatigue correction value according to the weather in which the emotional analysis is performed.

As shown in FIG. 3, the emotion analysis server 300 may include a server communication unit 350, an objective emotion value DB 320, a user speech accumulation information DB 340, an objective emotion value analysis unit 310, and an emotion weight. The calculator 330 may include a user-specific emotion value providing unit 360. In addition, the stress correction value calculator 370 and the fatigue correction value calculator 380 may be further included.

The server communication unit 350 supports hardware and software protocols for communicating with the user terminal 200, the emotion value application server 400, and the emotion adjustment hardware device 500. The server communication unit 350 may perform data communication according to, for example, an Internet protocol such as Transmission Control Protocol / Internet Protocol (TCP / IP).

The objective emotion value DB 320 is a database (DB; DataBase) in which objective emotion values are allocated and stored for each voice information including voice frequency, voice decibel, and voice speed of the voice. Here, the objective emotional value is an emotional value that is objectively determined through voice information, not individual feelings, such as stress index (excitement, tension, anxiety), fatigue index (fatigue, relaxing, lethargy), and emotional components (comfort, joy). , Satisfaction), and the like.

For reference, the database includes a hard disk drive, a solid state drive, a flash memory, a compact flash card, a secure digital card, and an SM card. A module capable of inputting / outputting information such as a smart media card, a multi-media card, a memory stick, or a memory stick may be provided inside the device, or may be provided in a separate device.

The user voice accumulation information DB 340 is a database in which user voice information received from the user terminal 200 is accumulated and stored. The user's voice frequency, voice decibel, and voice speed are accumulated and stored by date for each user ID.

The objective emotion value analyzer 310 analyzes the objective emotion value of the user by utilizing various voice analysis methods such as a known crystal fusion method. The decision fusion method extracts an objective emotion value matching the user voice information received from the user terminal 200 from the objective emotion value DB 320. For example, when the voice frequency is higher than the reference frequency, the emotion value is determined to be high stress, and when the voice decibel is lower than the reference decibel, the emotion value is determined to be high fatigue.

Furthermore, the objective emotion value analyzer 310 analyzes the pattern of information on the voice information regarding the emotion model, for example, using one or more statistical methods, and analyzes the emotion model by emotion and / or measurement target. Classify. In order to analyze the pattern of the user's voice information, a discriminant model, a logistic model, a neural network, a decision tree, etc. are applied, and the accuracy (eg, 70% to 78%) and / or The method with high reliability can finally be selected.

On the other hand, only the objective emotional value analysis described above does not perform accurate emotional analysis considering the characteristics of each user. For example, an accurate analysis may not be performed only by a single objective emotional value analysis for a user who normally has a high voice speed or a high frequency voice. Accordingly, the present invention calculates emotion weights based on the accumulated cumulative stored voice data of each user and further applies the emotion weights to objective emotion values to provide user-specific emotion values. The emotion weight may include a stress weight applied to the stress index and a fatigue weight applied to the fatigue index.

The emotion weight calculator 330 extracts user voice average information including a user's average voice frequency, an average voice decibel, and an average speech speed by using the user's voice accumulated information DB 340, and the user terminal 200. Emotion weights are calculated by comparing the user voice information received from the user with the user voice average information. For example, when the user's voice frequency is higher than the user's average voice frequency, the emotional weight of the stress index is determined to be '+1', and when the user's voice frequency is lower than the user's average voice frequency, the emotional weight of the stress index is determined. Is determined to be '-1'. Similarly, when the user's voice decibel is lower than the user's average decibel, the emotional weight of the fatigue index is determined to be '-1'.

The user-customized emotion value providing unit 360 generates a user-customized emotion value obtained by adding the emotion weight to the objective emotion value and transmits the generated emotion value to the user terminal 200. For example, when the objective emotional value is analyzed as a stress index of '+3' stage and the stress emotion weight is analyzed as '+1', the final stress index has a user-customized emotional value of '+4'.

Meanwhile, although an emotion weight is applied to a user-specific emotion value by using the accumulated voice information of the user, the user's job may be used for more accurate emotion value analysis. Jobs that deal with people, such as sales and sales, are more likely to be stressed out than minor ones.

To this end, the voice data-based emotion recognition system may further include a stress correction value calculator 370 that calculates a stress correction value according to a job of a user who is subjected to emotion analysis. The emotion weight calculator 330 may apply the user's emotion weight to a stress index that is corrected by applying a stress correction value of the user to a stress index calculated by comparing the user's voice information received from the user terminal 200 with the user's voice average information. Calculated as For example, as shown in FIG. 4, different stress correction values are assigned to each of the sales, sales, and office workers, and the stress correction value is additionally applied according to the job of the user to be analyzed.

Likewise, the weather of the day being analyzed can be used for a more accurate analysis of the fatigue index. This is because depressing weather such as rainy weather may have higher fatigue due to muscle pain than sunny weather.

To this end, the emotion recognition system includes a fatigue correction value calculator 380 that calculates a fatigue correction value according to the weather in which the emotion analysis is performed. The emotion weight calculation unit 330 may calculate a user's emotion weight by applying a fatigue index corrected by applying the user's fatigue correction value to the fatigue index calculated by comparing the user's voice information received from the user terminal 200 with the user's voice average information. It can be calculated as For example, as illustrated in FIG. 5, different fatigue correction values are allocated to each of the cloudy weather and the sunny weather, and the fatigue correction value is additionally applied according to the weather being analyzed.

6 is a block diagram illustrating a user terminal according to an exemplary embodiment of the present invention, and FIG. 7 is a view illustrating a display screen of a user terminal displaying a stress relief image according to an exemplary embodiment of the present invention.

As illustrated in FIG. 6, the user terminal 200 includes a terminal communication unit 210, a terminal input unit 220, a terminal display unit 230, a user voice information providing unit 240, and a terminal control unit 270. can do. In addition, the background music DB 250 and the stress relief image DB 260 may be included.

The terminal communication unit 210 is a module that performs a function of communicating through a mobile communication network. When performing a mobile communication such as 3G or 4G, an RF transmitter (not shown) which upconverts and amplifies a frequency of a wirelessly transmitted signal. And an RF receiver (not shown) for low noise amplifying the received wireless signal and down converting the frequency.

The terminal input unit 220 is a module that receives voice data, and may correspond to a microphone. When the user terminal 200 is a smartphone, when the user wants to analyze the voice, the user's voice and the other party's sound may be input through the smartphone's voice recognition sensor.

The terminal display unit 230 is a display module that displays a user-specific emotion value. The terminal input unit 220 and the terminal display unit 230 may be implemented as a single type of touch screen panel. The touch screen panel is a display window provided on the front of the terminal to display a work screen by providing a touch screen screen which can simultaneously perform input and display. A touch screen panel includes a Graphic User Interface (GUI) for communication with a user. Display.

The user voice information providing unit 240 extracts the user voice information including the biological voice signal, for example, the voice frequency, the voice decibel, and the voice speed from the voice data input from the user and transmits the user voice information to the emotion analysis server 300. Pitch statistics, voice decibels, and voice feature points of speech rate, crossing rate, and increasing rate can be extracted from the input voice data as user voice information. have.

The terminal controller 270 is implemented as a MCU (Main Control Unit) for controlling each function module, the module is installed emotion recognition app of the present invention. The emotion recognition app receives a user-specific emotion value based on user voice information from the emotion analysis server 300 and displays it on the display unit. For reference, the learner terminal 10 implemented as a smart phone (smart phone), etc., hundreds of various applications (applications) can be installed and added or deleted as desired by the user, so that the user can directly create a desired application. It is also possible to implement an interface suitable for oneself through various applications. Therefore, you can download emotion recognition apps from Google Market, Apple Store, etc. and install them on your smartphone.

The background music DB 250 is a database in which background music for alleviating stress for each stress index is allocated and stored. When the stress index of the user-customized emotion value exceeds a preset threshold stress, the terminal controller 270 extracts and plays back the background music allocated to the stress index of the user from the background music DB 250 and the stress index of the user. Can be played in real time as the background music matching the received stress index. Therefore, the background music that can alleviate the user's stress is updated and reproduced in real time through the terminal in a customized manner according to the stress index, thereby enabling efficient stress relief treatment. That is, when the stress index is increased, the background music corresponding thereto is updated and reproduced in real time, so that an effective stress relaxation treatment is performed.

In addition, the stress relaxation image DB 260 is a database in which a stress relaxation image that can alleviate stress is allocated and stored. When the stress index of the user-customized emotion value exceeds a preset threshold stress, the terminal controller 270 extracts a stress relaxation image allocated to each user's stress index from the stress relaxation image DB 260 and shows the result shown in FIG. 7. As shown through the display unit, the stress index of the user may be received in real time and displayed as a stress relief image matching the received stress index. Therefore, the stress relaxation image that can alleviate the user's stress is updated and displayed in real time through the terminal in accordance with the stress index, thereby enabling efficient stress relief treatment. For reference, the stress relief image may include not only the image form but also digital contents such as color colors, 3D shapes, augmented reality, and virtual reality.

8 is a flowchart illustrating voice data-based emotion recognition processes according to the first embodiment of the present invention.

In the speech data-based emotion recognition method of the present invention, as shown in FIG. 8, a user voice information provision process (S810), an objective emotion value calculation process (S820), an emotion weight calculation process (S830), and a user-specific emotion value provision are provided. In operation S860, the user-customized emotion value display process may be included (S870). In the process of providing user voice information (S810), the user terminal 200 extracts user voice information including voice frequency, voice decibels, and voice speed from voice data input from the user and transmits the user voice information to the emotion analysis server 300. It is a process.

In the objective emotion value calculation process (S820), the objective emotion value indicating the objective emotion determined by the emotion analysis server 300 based on the preset reference voice information based on the user voice information received from the user terminal 200. It is a process of calculating.

The emotion weight calculation process is a process of calculating the emotion weight of the individual individual emotionr by comparing the user voice information received from the user terminal 200 and the accumulated and stored user voice accumulated information. The user-customized emotion value providing process (S860) is a process in which the emotion analysis server 300 generates a user-customized emotion value to which the emotion weight is added to the objective emotion value and transmits it to the user terminal 200.

The user-customized emotion value display process (S870) is a process in which the user terminal 200 receives and displays a user-specific emotion value based on user voice information from the emotion analysis server 300.

Further, after the emotion weight calculation process (S830), it may further include a stress correction value calculation process (S840) for calculating the stress correction value according to the job of the user undergoing the emotion analysis. In the emotion weight calculation process (S830), the user's emotion weight is corrected by applying a stress correction value of the user to a stress index calculated by comparing the user's voice information received from the user terminal 200 with the user's voice average information. It can be calculated as Therefore, the stress correction value considering the user's occupation is applied to the stress emotion value, more accurate stress analysis can be made.

In addition, after the emotion weight calculation process (S830), may further include a fatigue correction value calculation process (S850) for calculating the fatigue correction value according to the weather in which the emotion analysis is performed. In the emotion weight calculation process (S830), the user's emotion weight is corrected by applying the user's fatigue correction value to the fatigue index calculated by comparing the user's voice information received from the user terminal 200 with the user's voice average information. Calculated as Therefore, the fatigue correction value considering the weather of the day on which the emotion analysis is performed is applied to the fatigue emotion value, so that more accurate fatigue analysis can be performed.

FIG. 9 is a flowchart illustrating a voice data based emotion recognition process and a method of applying the same according to a second embodiment of the present invention. FIG. 10 is a flowchart illustrating voice data based on a second embodiment of the present invention. This is a conceptual diagram to explain the emotion recognition process and how to apply it.

In the first exemplary embodiment of FIG. 8, the personalized emotion value calculated by the emotion analysis server is transmitted to the user terminal and displayed, the desktop screen is adjusted with the stress relaxation image of the user terminal, and the background music is adjusted. However, according to the second exemplary embodiment of the present invention, nine types of deep data are generated by using a personalized emotion value in a speech cumulative information DB and performing deep learning by supervised learning using the big data. A specific one of the plurality of emotions is derived as a customized emotion value. Thereafter, the derived customized emotion value is transmitted to the emotion value application server (400 of FIG. 1) and the emotion adjustment hardware device (500 of FIG. 1) to actively use it.

In detail, the user's terminal collects user voice information, for example, call voice data (S900). Subsequently, the terminal analyzes the data of the user voice information into a biometric signal (S910). Such biological voice signal data analysis is possible by operation of a publicly available application programming interface (API) installed and operated in a terminal. Subsequently, the analyzed biological voice signal is transmitted to the emotion analysis server through the wired / wireless communication network (S920). Thereafter, the emotion analysis server collects big data in the user's voice accumulation DB (S930).

Subsequently, the emotion analysis server proceeds to deep learning by supervised learning using the collected big data (S940). At this time, the emotion analysis algorithm may be operated during the supervised learning. In the field of artificial intelligence (AI) in the fourth industry, deep learning technology refers to the technology that allows computers to think and learn as if they were humans. To this end, deep learning technology serves to cluster or classify objects or data. Such deep learning technology is based on the prediction through classification, and there are two ways to classify and divide data in big data. One is supervised learning and the other is unsupervised learning. Supervised learning is a way of teaching computers first to classify and make predictions. The emotion analysis algorithm used for deep learning according to the present invention corresponds to supervised learning. That is, a method of classifying the biovoice signal range corresponding to each of the plurality of emotion types, for example, nine emotions shown in FIG. 12, and classifying them. Specifically, the boundary values of the first emotion, the second emotion, and the third emotion are determined from the scatter diagrams of the mean values for each period of the volume decibels, and the fourth emotions, the fifth emotions, and the sixth emotions are determined from the scatter diagrams of the mean values for each period of the pitch frequency. The threshold value is determined, and supervised learning is performed to determine the threshold values of the seventh emotion, the eighth emotion, and the ninth emotion on a scatter diagram of the average value of each period of the speech speed.

Next, the emotion analysis server calculates a personalized emotion value closest to the user from among a plurality of emotion types, for example, nine according to the speed, sound, and voice speed that constitute the biovoice signal. S950). Specifically, obtain one emotion (P1) calculated on the basis of the volume decibels, obtain another emotion (P2) calculated on the basis of the pitch frequency, another calculated on the basis of the speech speed After obtaining the emotion (P3) of, select one of the three emotions P1, P2, P3 that is out of the bounds of the maximum. This will be described in detail later with reference to FIG. 11.

Thereafter, the emotion analysis server transmits the calculated customized emotion value to the emotion value application server (S960) so as to utilize the determined emotion value for each cycle as a reference. The emotion value application server may be a server that stores a personal medical record of a user who provided voice information through a terminal. Therefore, the personalized emotion value for each period may be used as a reference as one of the personal medical records.

Finally, the emotion analysis server transmits an operation command corresponding to the calculated customized emotion value to an emotion control hardware device, for example, an air conditioning device capable of controlling temperature and humidity, or an artificial intelligence speaker with artificial intelligence. Operation (S970). Herein, the operation command refers to a command for operating an emotion control hardware device connected to the emotion analysis server through a wired or wireless communication network in an appropriate manner. For this purpose, the emotion control hardware device has an appropriate internal configuration for receiving and driving an operation command. Suitable.

Meanwhile, in FIG. 10, the emotion analysis API system may be an emotion analysis server, and the emotion analysis result may be one of a user terminal that receives a customized emotion value, an emotion value application server, and an emotion adjusting hardware device.

FIG. 11 is a graph illustrating a method of calculating the big data of FIG. 9, deep learning by using supervised learning, and customized emotion values, and FIG. 12 is a block diagram illustrating various types of customized emotion values of FIG. 9.

11 and 12, the X axis of the graph represents the magnitude of the negative biosignal, and the Y axis represents the frequency of occurrence of the biovoice signal collected from the big data. As is the case with most natural data, the distribution of biospeech signals by frequency has a Gaussian Distribution (D curve). For example, to conduct deep learning by supervised learning to classify the first emotion (helplessness of drawing), the second emotion (comfort of drawing), and the third emotion (exercise of drawing) with volume decibels, the boundary of these three emotions Value is set and trained by the emotion analysis server. For this purpose, the boundary values (N1 and N3 on the X axis of the drawing) are designated according to the volume of the decibels.

In this case, if the average value of the volume decibels for a predetermined period of the biological voice signals transmitted from the user terminal to the emotion analysis server is distributed in P1 of the figure, the customized emotion value may be inferred and calculated. Similarly, a Gaussian Distribution is obtained from the big data collected on the basis of the pitch frequency, the threshold is set, and P2, which is one emotion in the fourth to sixth emotions, is obtained. Then, a normal distribution (Gaussian Distribution) is obtained from the big data collected based on the sound velocity, which is the voice speed, and the threshold is set, and P3, which is one emotion in the seventh to ninth emotions, is obtained. Next, one of the three emotions P1, P2, and P3 obtained may be inferred and calculated as a customized emotion value corresponding to the bio-voice signal received from the user terminal. Such customized emotion values may be defined and applied to nine plural numbers as shown in FIG. 12. However, delimitation of the plurality of emotion types and the Gaussian Distribution of the biospeech signal may be optimized in various ways by psychological / emotional experts or statistical techniques.

The present invention is not limited to the above embodiments, and it is apparent that many modifications can be made by those skilled in the art within the technical spirit to which the present invention belongs.

Claims (13)

Extracting user voice information including pitch frequency, volume decibels, and voice speed from voice data input from the user and transmitting the voice information to the emotion analysis server, and receiving and displaying a user-defined emotion value based on the user voice information from the emotion analysis server User terminal;
An objective emotion value indicating an objective emotion is calculated based on the user voice information received from the user terminal, and the user individual emotion value is calculated by comparing the user voice information received from the user terminal with accumulated cumulative user voice information. Emotion analysis server;
An emotion adjustment hardware device capable of receiving a control command according to a customized emotion value calculated by the emotion analysis server and performing an operation that may affect the emotion adjustment of the user;
An emotion value application server that can receive the customized emotion value calculated by the emotion analysis server and use it as a reference material; And
And a wired / wireless communication network connecting the terminal, the emotion analysis server, the emotion control hardware device, and the emotion value application server to provide wired or wireless communication. The method of claim 1,
The emotion control hardware device,
The Internet of Things (LoT) is possible therein, and a voice data-based emotion recognition system comprising a configuration capable of executing a command received from the emotion analysis server. The method of claim 1,
The emotion control hardware device,
And a user selected from a process system including an air conditioner, a heater, and a humidity control device exposed to the operating environment. The method of claim 1,
The emotion control hardware device,
Voice data-based emotion recognition system, characterized in that the artificial intelligence is mounted inside the artificial speaker that can communicate with the user. The method of claim 1,
The emotion analysis server,
An objective emotion value DB, in which an objective emotion value is allocated and stored for each voice information including voice frequency, voice decibel, and voice speed;
A user voice accumulation information DB in which user voice information received from a user terminal is accumulated and stored;
An objective emotion value analysis unit for extracting an objective emotion value matching the user voice information received from the user terminal from the objective emotion value DB;
The user voice average information including the average voice frequency, the average voice decibel, and the average speech rate of the box is extracted using the user voice cumulative information DB, and the user voice information received from the user terminal and the user voice average information are extracted. An emotion weight calculator for comparing emotion weights; And
And a user-specific emotion value providing unit configured to generate a user-defined emotion value obtained by adding the emotion weight to the objective emotion value and transmit the generated emotion value to the user terminal. The method of claim 1,
The user terminal,
A terminal input unit configured to receive the voice data;
A terminal display unit displaying the user customized emotion value;
A user voice information providing unit configured to extract user voice information including voice frequency, voice decibels, and voice speed from the voice data input from the user and transmit the extracted user voice information to the emotion analysis server; And
And a terminal controller configured to receive a user-customized emotion value based on the user voice information from the emotion analysis server and record the same on the display unit. The method of claim 6,
The user terminal,
It further includes a background music DB, which is stored with music assigned to relieve tension by stress index.
The terminal control unit,
When the stress index of the user-customized emotion value exceeds a preset threshold stress, the background music allocated for each user's stress index is extracted from the background music DB and played, and the stress index received by receiving the user's stress index in real time. The voice data based emotion recognition system, characterized in that for playing by switching to a background music matching. The method of claim 1,
The user terminal,
Saved streams with assigned stress relief images that can relieve stress
Less relaxation image DB;
The terminal controller may be configured to preset a stress index of a user-customized emotion value.
When the exceeded threshold stress is exceeded, the stress relaxation image allocated for each user's stress index is extracted from the stress relaxation image DB and displayed through the display unit, and the stress corresponding to the stress index received by receiving the user's stress index in real time. Emotion recognition system based on voice data, characterized in that the display alternately as a relaxation image. Providing voice information through a user in a terminal;
Analyzing the voice information provided by the terminal as a voice biosignal consisting of a volume decibel, a pitch frequency, and a voice speed;
Transmitting the analyzed voice biosignal to an emotion analysis server;
Collecting big data on the voice biosignal accumulated by the individual providing the voice information in the emotion analysis server;
Performing deep learning by supervised learning using the big data collected by operating an emotion analysis algorithm in the emotion analysis server;
Classifying a user's emotion into one of a plurality of basic emotions for a predetermined period according to the volume decibel, pitch frequency, and voice speed of the voice biosignal; And
And transmitting the emotion classified into one of the plurality of basic emotions to an emotion value application server connected to the emotion analysis server through a wired / wireless communication network. The method of claim 9,
The method for deep learning by the supervised learning,
A boundary value of a first emotion, a second emotion, and a third emotion is determined from a scatter diagram of the mean value for each period of the volume decibels,
A boundary value of a fourth emotion, a fifth emotion, and a sixth emotion is determined from a scatter diagram of the mean value of each period of the pitch frequency;
And determining a boundary value of a seventh emotion, an eighth emotion, and a ninth emotion from a scatter diagram of the mean value for each period of the speech speed. The method of claim 10,
The method of classifying the user's emotion into one of a plurality of basic emotions for a predetermined period of time,
Obtaining one emotion (P1) calculated based on the volume decibels,
Obtaining another emotion P2 calculated based on the pitch frequency,
After obtaining another emotion P3 calculated based on the speech speed,
A method of applying a speech data-based emotion recognition system, characterized in that one of the three emotions P1, P2, and P3 obtained is selected from the largest boundary. The method of claim 9,
Before transmitting the emotion classified as one of the plurality of basic emotions to the emotion value application server,
And transmitting a control command corresponding to an emotion classified as one of the plurality of basic emotions to an emotion control hardware device connected to the emotion analysis server through a wired / wireless communication network. Way. The method of claim 12,
The emotion control hardware device,
Application method of a voice data-based emotion recognition system, characterized in that it is one selected from the group of devices that can affect the user's emotion consisting of an artificial speaker and an air conditioning device for controlling temperature and humidity.

Images