KR20180119913A - Apparatus and computer readable recorder medium stored program for recognizing emotion using biometric data - Google Patents

Abstract

The present invention relates to an emotion detection apparatus using biometric information and a recording medium which can be read by a computer recording a program to execute an emotion detection method. According to an embodiment of the present invention, the emotion detection apparatus comprises: a stimulus providing unit for providing stimulus information to a user; a biometric information receiving unit for receiving biometric information transmitted from a terminal of a user; an emotion model generating unit for learning the biometric information transmitted from the terminal of the user in response to the stimulus information to generate an artificial neural network model; an emotion model database storing the generated artificial neural network model; and an emotion detecting unit for extracting a feature point of the received biometric information and classifying the extracted feature point using the stored artificial neural network model to detect an emotion of the user. According to an embodiment of the present invention, the accuracy in emotion recognition using the biometric information of the user can be increased.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a biometric information detection apparatus and a computer-readable recording medium storing a program for executing the biometric information detection method.

The present invention relates to an emotion detecting apparatus using biometric information and a computer-readable recording medium storing a program for executing the emotion detecting method.

Emotional ICT (Information Communication Technologies) is a technology in which a user directly inputs desired emotions and uses the related products and services in the form of a camera or a sensor worn by a user in the form of using the user's facial expressions, speech, The system automatically collects and analyzes the body emotion information of the user, and the emotion of the user is changed according to the result of recognizing the emotion of the user.

Accordingly, in order to improve user satisfaction with products and services, it is necessary to analyze the user's body emotion information through physiological patterns optimized for each user. However, conventional emotion recognition requires a general physiological The accuracy of the sensitivity recognition and the user satisfaction may be lowered in terms of applying the pattern in common.

An object to be solved by one embodiment of the present invention is to provide a device for detecting emotion such as stress, fatigue and mood state based on a user's biological signal, and a computer readable recording medium recording a program for executing the detection method .

Embodiments according to the present invention can be used to accomplish other tasks not specifically mentioned other than the above-described tasks.

According to an aspect of the present invention, there is provided an apparatus for transmitting biometric information, the apparatus comprising: a stimulus providing unit for providing stimulus information to a user; a biometric information receiving unit for receiving biometric information transmitted from a user terminal; A sensory model database for storing the generated neural network model, and a feature point extracting unit for extracting the feature points of the received biometric information and extracting the extracted neural network models using the stored neural network model And a sentence detection unit which detects the sentiment of the user by classifying the extracted feature points.

Here, the bio-information receiving unit may receive at least one of the user's skin conductivity (EDA), pulse wave (PPG), or skin temperature (SKT).

Further, the emotion detecting unit may detect at least one of the user's stress index, fatigue index, or emotion state.

Also, the user's terminal may be a stretchable smart band.

According to an embodiment of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing an emotion detection method through an emotion detection apparatus, the computer readable recording medium having a function of providing stimulus information to a user, A function of generating an artificial neural network model of the user by learning biometric information corresponding to the stimulation information, a function of storing the generated artificial neural network model, extracting the feature points of the received biometric information, And a function of detecting the emotion of the user by classifying the extracted feature points by using the artificial neural network model of the artificial neural network model.

Here, the biometric information may include skin conductivity (EDA), pulse wave (PPG), and skin temperature (SKT).

In addition, the function of detecting the emotion of the user is to calculate the first stress index using the first artificial neural network model corresponding to the skin conductivity (EDA) and to calculate the second stress index using the second artificial neural network model corresponding to the pulse wave (PPG) 2 stress index, calculating a third stress index using a third artificial neural network model corresponding to skin temperature (SKT), and applying a weight to the first stress index, the second stress index, and the third stress index And calculating the total stress index of the user.

According to the embodiment of the present invention, the accuracy of emotion recognition using the user's biometric information can be improved. In addition, according to the emotion recognition result, a user-specific service can be provided to improve the satisfaction of the user.

1 shows a configuration of an emotion detecting apparatus according to an embodiment of the present invention.
Fig. 2 shows the emotion detection method using the emotion detecting apparatus of Fig.
FIG. 3 shows a method for extracting feature data and a method for detecting a stress step in FIG.
4 shows a method for calculating a stress level according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art to which the present invention pertains. The present invention may be embodied in many different forms and is therefore not limited to the embodiments described herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and the same reference numerals are used for the same or similar components throughout the specification. In the case of publicly known technologies, detailed description thereof will be omitted.

In this specification, when a part is referred to as "including " an element, it is to be understood that it may include other elements as well, without departing from the other elements unless specifically stated otherwise.

The terms " part, "" module," and the like, as used herein, refer to a unit that processes at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software.

1 shows a configuration of an emotion detecting apparatus according to an embodiment of the present invention.

The emotion detecting apparatus 100 of FIG. 1 detects the emotion of the user by analyzing the biometric information transmitted from the user terminal 200 and includes a body information receiving unit 110, a emotion learning unit 120, an emotion model database 130, A sensibility detection unit 140, and a sensibility output unit 150. [ The emotion detected through the emotion detecting apparatus 100 includes at least one of stress, fatigue, or emotion (joy, depression, sadness, anger).

The biometric information receiving unit 110 receives biometric information and identification information transmitted from the user terminal 200. At this time, the biometric information includes the user's skin conductivity information, pulse wave information, and skin temperature information, and the identification information includes a unique number of the user terminal 200 for user identification or user personal information.

The emotional learning unit 120 generates a sensory model as a result of providing stimulation information to a user and learning user's biometric information corresponding to the provided stimulation information and transmits the stimulus providing unit 121 and the sensibility model generating unit 122 . Here, the emotion model refers to an artificial neural network model.

The stimulus generator 121 outputs stimulus information for deriving emotion. Here, the stimulation information includes at least one of a calculation capable of deriving a user's emotion, a continuous performance task (CPT), or a medium. At this time, the media includes images, photographs, and texts that can lead to a user's joy, depression, sadness, or anger.

The stimulus provider 121 transmits the stimulus information to the user terminal 200. At this time, the user terminal 200 may receive the stimulation information transmitted from the stimulus providing unit 121 and provide stimulation information to the user.

The emotion model generating unit 122 analyzes the biometric information corresponding to the stimulation information provided through the stimulus providing unit 121 to generate a user's emotional model. At this time, the biometric information includes skin conductivity, pulse waves and skin temperature transmitted from the user terminal 200 in response to the stimulation information.

Specifically, the emotion model generation unit 122 extracts feature data corresponding to the user's skin conductance information, pulse wave information, and skin temperature information for each stimulus information, sets a feature data range for each emotion state of the user, Emotion Feature Base Set, EFBS). For example, the emotion feature base set can be generated based on emotion states (normal, stress, fatigue, emotion) and corresponding biometric information as shown in Table 1 below.

EDA PPG SKT EMOTION STEP F ₁ , ... , F _n1 F ₁ , ... , F _n2 F ₁ , ... , F _n3 NORMAL 00
STRESS
01 02 03
TIREDNESS
01 02 03
FELLING
HAPPY SAD ANGRY

The emotion model database 130 stores the emotion model generated by the emotion model generation unit 122. [ Here, the emotion model is stored together with the user identification information, and includes a plurality of emotion models corresponding to the user's biometric information. For example, a plurality of emotion models corresponding to skin conductivity, pulse wave and skin temperature, respectively, can be stored.

At this time, the emotion model stored in the emotion model database 130 is verified through learning of the biometric information transmitted from the user terminal 200 in response to the stimulation information provided through the stimulus providing unit 121, Can be updated accordingly.

The emotion detecting unit 140 searches the emotion model of the user in the emotion model database based on the identification information received from the biometric information receiving unit 110 and extracts the biometric information received from the biometric information receiving unit 110 using the sensed emotion model The emotion state score is calculated based on the extracted feature data to detect emotion of the user.

The emotion output unit 150 outputs the emotion detection result of the emotion detection unit 140. [ In addition, the emotion detection result may be transmitted to the user terminal 200 and output through the user terminal 200.

The user terminal 200 of FIG. 1 refers to a wearable terminal that collects biometric information of a user, and is connected to the emotion detecting apparatus 100 through a wireless communication network. In an embodiment of the present invention, the user terminal 200 is implemented as a stretchable smart band, but is not limited thereto and may be implemented in any form attached to or worn on a part of the user's body. For example, in the form of a smartphone, a watch, a pair of glasses, a hat, a garment, a shoe, or other accessory.

The user terminal 200 of FIG. 1 collects the user's biometric information in real time using a plurality of sensors and transmits the same to the emotion detecting apparatus 100. The user terminal 200 of the user terminal 200 transmits the stimulus information and the emotion detection result And outputs it.

The user terminal 200 of FIG. 1 includes an input / output unit 210, an electrodermal activity (EDA) collector 220, a photoplethysmography (PPG) collector 230, a skin temperature (SKT) collector 240, And a transmission unit 250.

The input / output unit 210 receives identification information and a control signal from the user (for example, a control signal for ON / OFF and initialization of the user terminal), outputs the stimulation information transmitted from the emotion detecting apparatus 100, Or the emotion detection result transmitted from the emotion detecting apparatus 100 is output.

The EDA collector 220 collects the skin conductivity of the user. For example, an EDA sensor measures and collects a user's electrodermal activity.

The PPG collecting unit 230 measures and collects the user's pulse wave using the PPG sensor.

The SKT collecting unit 240 measures and collects the user's skin temperature. Specifically, a temperature sensor including a negative temperature coefficient thermistor is used to measure the skin surface temperature of the user and collects the measured values.

The biometric information transmission unit 250 transmits the biometric information of the user collected through the user terminal 200 and the identification information inputted through the user terminal 200 to the emotion detecting apparatus 100. Specifically, the skin conduction information collected by the EDA collecting unit 220, the pulse wave information collected by the PPG collecting unit 230, and the skin temperature information collected by the SKT collecting unit 240 are transmitted. Also, the personal information of the user terminal 200 or user personal information inputted from the user through the input / output unit 210 is transmitted.

Although not shown in FIG. 1, the user terminal 200 may further include a sensor for detecting a contact failure or foreign matter between at least one of the EDA sensor, the PPG sensor, and the temperature sensor and the user's skin.

Although not shown in FIG. 1, the user terminal 200 includes a battery that supplies power to each configuration, and the battery may be implemented integrally with the user terminal 200 or in a form that can be combined with and detached from the user terminal 200 Can be implemented.

Fig. 2 shows the emotion detection method using the emotion detecting apparatus of Fig.

In FIG. 2, the emotion detection method of the first user will be described on the assumption that the emotion model of the first user is stored in the emotion model database 130 in advance.

First, biometric information and identification information of a first user transmitted from the user terminal 200 through the biometric information receiving unit 110 are received (S10).

Then, the emotion model of the first user is searched based on the identification information received in step S10 through the emotion detecting unit 140 (S20), and the feature data of the biometric information received in step S10 is extracted (S30).

In step S40, the emotion state index corresponding to the feature data extracted in step S30 is calculated using the emotion model retrieved in step S20 through the emotion detecting unit 140. [ At this time, stress level, fatigue level, and emotional state (joy, depression, sadness, anger, etc.) can be judged according to the emotional state index. For example, the stress stage may be one stage corresponding to a stress-free rest condition, two stages corresponding to a weak stress with an emotional state index of more than 0 and less than 2, three stages corresponding to an intermediate stress with an emotional state index of more than 2 and less than 5, And four stages corresponding to the strong stress that the emotional state index is more than 5 and 9 or less.

Then, the emotion output unit 150 outputs emotion detection result corresponding to the emotion state index in step S40.

Hereinafter, the method of calculating the stress index and the stress level in steps S30 to S40 of FIG. 2 will be described in detail with reference to FIGS. 3 and 4. FIG.

FIG. 3 shows a method for extracting feature data and a method for detecting a stress step in FIG.

Referring to FIG. 3, after the preprocessing process is performed on the skin conductivity (EDA), the pulse wave (PPG), and the skin temperature (SKT) data received in step S10 of FIG. 2, The stress index is calculated using the sensibility model that is searched.

Specifically, after removing noise of skin conductivity (EDA) data, pulse wave (PPG) data, and skin temperature (SKT) data using a butterworth filter, feature data corresponding to each data is extracted .

For example, first feature data is extracted from the skin conductance (EDA) data, and the extracted first feature data includes mean, standard deviation, differential mean, differential standard deviation, positive differential mean, and negative mean of the EDA signal can do.

For example, the second feature data is extracted from the pulse wave (PPG) data, and the extracted second feature data includes height average, standard deviation, mean and standard deviation of the PPI signal, PPI change size, Standard deviation may be included.

For example, the third feature data is extracted from the skin temperature (SKT) data, and the extracted third feature data includes a mean value, a standard deviation, a differential mean, a derivative standard deviation, and a reference (ref) Increase / decrease rate.

Thereafter, the EDA stress index is calculated by classifying the first feature data using the EDA artificial neural network, the PPG stress index is calculated by classifying the second feature data using the PPG artificial neural network, and the third feature data is classified using the SKT artificial neural network The feature data is classified to calculate the SKT stress index.

The composite stress index is then calculated based on the EDA Stress Index, the PPG Stress Index, and the SKT Stress Index.

4 shows a method for calculating a stress level according to an embodiment of the present invention.

4, a first stress level (EDA_Stress_Score) corresponding to skin conductivity (EDA) data, a second stress level (HRV_Stress_Score) corresponding to HRV (heat rate variability) data ) And a third stress level (SKT_Stress_Score) corresponding to the skin temperature (SKT) data, and calculates a total stress level (Stress_Score) by applying weights w ₁ , w ₂ and w _{3 for each of the} biological signals. At this time, the heart rate variability (HRV) includes the heart rate peak and the heart rate interval detected by the PPG measurement data.

The emotion detection method according to an embodiment of the present invention may be recorded in a recording medium that is basically installed in the emotion detection apparatus or implemented by a program directly installed by a user and can be read by a computer. Here, the computer may include a desktop, a notebook, a smart phone, a tablet PC, a personal digital assistant (PDA), and a mobile communication device. The recording medium may also include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical media storage, and the like.

According to the embodiment of the present invention, the emotion model corresponding to the user's bio signal can be learned and stored, and the sensibility of the user can be detected using the sensed model, thereby improving the accuracy of the emotion detection result and the satisfaction of the user. Further, it is possible to provide an alarm, a content, or a service based on a health satisfaction corresponding to a user's sensibility detection result.

While the present invention has been particularly shown and described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It belongs to the scope.

100: Emotion detection device
110: Biometric information receiver
120: Sensibility learning section
121: stimulant study
122: Emotion model generation unit
130: Emotion model database
140: Emotion detection unit
150: Emotion output unit
200: user terminal
210: Input /
220: EDA collector
230: PPG collecting unit
240: SKT collecting section
250: Biometric information transmission unit

Claims (7)

A stimulant for providing stimulus information to the user,
A biometric information receiving unit for receiving biometric information transmitted from the user terminal,
An emotion model generation unit that learns biometric information transmitted from the user terminal in response to the stimulation information and generates an artificial neural network model,
An emotion model database storing the generated artificial neural network model, and
Extracting feature points of the received biometric information, classifying the extracted feature points using the stored neural network model, and detecting the emotion of the user,
And the emotion detecting device. The method of claim 1,
Wherein the biometric information receiving unit receives at least one of data of the user's EDA, PPG, or skin temperature (SKT). The method of claim 1,
Wherein the emotion detecting unit detects at least one of the stress index, fatigue index, or emotion state of the user. The method of claim 1,
Wherein the terminal of the user is a stretchable smart band. A computer-readable recording medium storing a program for executing an emotion detection method through an emotion detecting apparatus,
A function of providing stimulus information to the user,
A function of receiving biometric information of the user,
A function of generating an artificial neural network model of the user by learning biometric information corresponding to the stimulation information,
A function of storing the generated artificial neural network model of the user, and
Extracting feature points of the received biometric information, classifying the extracted feature points using the user's neural network model, and detecting the emotion of the user
Readable recording medium having recorded thereon a program. The method of claim 5,
Wherein the bio-information includes a skin conductivity (EDA), a pulse wave (PPG), and a skin temperature (SKT). The method of claim 6,
The function of detecting the emotion of the user includes:
Calculating a first stress index using a first artificial neural network model corresponding to the skin conductivity (EDA), calculating a second stress index using a second artificial neural network model corresponding to the pulse wave (PPG) A function of calculating the third stress index using the third artificial neural network model corresponding to the SKT, and
Calculating a total stress index of the user by applying a weight to the first stress index, the second stress index and the third stress index;
Readable recording medium having recorded thereon a program.

Images