KR101307248B1 - Apparatus and Method for Decision of Emotional state through analysis of Bio information - Google Patents

Abstract

The present invention relates to an apparatus and method for analyzing emotional state through analysis of user's biometric information. The present invention relates to biological signals such as pulse wave (PPG), skin electrical resistance (GSR), skin temperature (SKT), and external temperature collected from a user's body. The purpose is to extract and calculate the bio-signals to analyze and determine the emotional state of the user.
An object of the present invention is an apparatus for analyzing emotional state through analysis of biometric information of a user, comprising: a biosensor unit including a pulse wave sensor, a skin temperature sensor, and a skin electrical resistance sensor that are in contact with a user's body to extract biometric information; An analog / digital conversion unit for converting each biometric information extracted from the biosensor unit into a digital signal; The pulse wave ratio (PPG), skin temperature (SKT) and skin electrical resistance (GSR) values of the biosensor unit inputted through the analog / digital conversion unit are input, and the ratio of average heart rate (HR) per minute (GSR / HR), Calculate autonomic balance (LF / HF) parameters, and use the corresponding standard indicators (T _SKT , T _SDNN , T _{LF / HF} , T _{GSR /} ) from each calculated parameter (PPG, SKT, GSR / HR, LF / HF). _HR ) and a main processor for determining the emotional state of the user based on the calculated standard indicator value; A memory for storing an emotional state determined by the main processor; And an LCD display for displaying the emotional state determined by the main processor.

Description

Apparatus and Method for Decision of Emotional state through analysis of Bio information}

The present invention relates to an emotional state analysis and determination system using human biometric information, and more particularly, to collect biological signals from a user, calculate necessary parameters from the collected biological signals, and analyze the calculated parameters. An apparatus and method for analyzing emotional state through analysis of biometric information of a user to determine a condition of the present invention.

Robot technology has recently been developed as a convergence technology in various fields such as mechanical, electronic, information, biological, and medical engineering. Robot technology that has accumulated such convergence technology is applied with cutting-edge technology such as sensor technology, control technology, artificial intelligence, user interface, IT technology, and cleaning, errand, security, cooking, and childcare as well as industrial robots in manufacturing industry. It is being activated as a home support service robot such as housekeeping aids such as assistants, leisure support robots such as education, entertainment, toys, games, and information support, caring, rehabilitation assistance for the elderly and the disabled, rehabilitation training, and remote medical support.

In addition, U-Healthcare service, which has been converged with the recent development of IT technology and medical information technology, utilizes various devices around the life, so it is not limited to one-off or treatment at the hospital, It is possible to monitor patients remotely anytime, anywhere in real life without limitation of time and space.

 Furthermore, the development of the convergence technology of ubiquitous healthcare and robot technology that can interact with robots can recognize emotions using information such as human voice, video, bio signals or gestures, and realize communication with robots. The development of a robot-based human emotional sympathy module is in progress.

A service apparatus and method for a robot that recognizes and responds to human emotion recognition in the prior art are disclosed in Korean Patent Publication No. 10-2010-0001928.

1 is a block diagram of a service apparatus based on emotion recognition according to the related art, and includes a data input unit 10 that receives voice and video data from a voice and video device, and recognizes a user's emotion from a received voice and video signal. An emotion recognition unit 20 to determine the service to be provided to the user based on the recognized user's emotion, and an output unit 40 to provide the service to the user based on the determined service. It consists of.

Here, the data input unit 10 receives audio and video data from the audio and video device. The data input unit 110 is composed of a voice and video input device such as a microphone or a camera.

The emotion recognition unit 20 recognizes a user's emotion from the received voice and image data. To this end, the emotion recognizer 20 includes a voice recognizer 22, an expression recognizer 24, and an emotion determiner 26.

The speech recognition unit 22 extracts features such as pitch statistic, loudness, number of sections, crossing rate, increasing rate, etc. from the speech signal. The speech recognizer 22 classifies emotion-specific patterns using Multi Layer Perception (MLP), which learns these extracted features by Backpropagation (BP) of artificial neural networks.

The facial expression recognition unit 24 performs image processing such as a skin tone accumulation algorithm on the input face image to extract a feature. In this process, the face region of the user moving from the background image should be detected. The facial expression recognition unit 24 measures the background at the time of capturing and acquires the detected background image according to the image difference from the input image. By subtracting the detected background image from the background image, only the newly input user's image is obtained. In this case, it should be noted that the input image may be changed by external light at the time of obtaining the image. This problem can be solved by re-shooting the image within a few seconds when the image is taken and receiving a continuous image to make the background image and the newly acquired image almost the same.

The emotion determination unit 26 determines the final user's emotion based on the recognition result obtained by the voice recognition unit 22 and / or the facial expression recognition unit 24. In determining emotion, a decision fusion method or a feature fusion method is used to determine a user's final emotional state.

The service determiner 30 determines a service to be provided to the user based on the recognized emotion of the user. First, the service determiner 30 first checks a service providing partner for confirming whether a person corresponds to a specific person in a database based on voice and video data.

If the service providing partner corresponds to a specific person in the database, the service determination unit 30 determines the type of service to be provided. In this case, the service determination unit 30 determines a service to be provided to the user according to each emotion classified by the emotion recognition unit 20. In contrast, if the service providing partner is not a specific person in the database, the service determination unit 30 determines not to provide the service.

The output unit 40 provides a service to a user based on the determined service type. For example, when the user's feelings are sad, the user speaks comforting words or actions, and when the user's feelings are joy, the user acts as a word of congratulations to make the user happy. That is, the output unit 40 provides various services to the user according to the emotional state of the user.

According to such a service apparatus based on emotion recognition according to the related art, it is possible to recognize a user's emotion through voice and video or complex data, and thereby provide a service corresponding to the user's emotion by the service robot and the service device. It becomes.

However, the service apparatus and method based on emotion recognition according to the prior art extracts only the voice and facial expression information of the user and recognizes and judges the emotion, and extracts and learns the voice feature and facial feature of the user from a database. The emotion is determined through the extracted user's face image and voice feature vectors. Since the face image and voice extract information through microphones and cameras, it is sensitive to noise depending on the surrounding environment, and thus cannot extract accurate data. There was this.

Therefore, the present invention is to collect the pulse wave (Photo Plethysmogram, abbreviated as "PPG"), skin electrical resistance (GSR) abbreviated as "GSR" collected from the user's body in order to improve the problems of the prior art Extraction of biosignals such as skin temperature (hereinafter abbreviated as "SKT") and external temperature, and calculating the biosignal to analyze and determine the emotional state of the user so as to analyze the biometric information of the user It is an object of the present invention to provide a condition analysis apparatus and method.

Another object of the present invention is an autonomic nerve balance (hereinafter abbreviated as "LF / HF") from the biometric information collected from the user (PPG, GSR, SKT), the envelope peak time interval of the PPG (hereinafter referred to as "RRI" After calculating the standard deviation (hereinafter abbreviated as "SDNN") calculated from the abbreviation) and the ratio of GSR and average heart rate per minute (hereinafter abbreviated as "GSR / HR"), each calculated value is calculated. The purpose is to recognize and judge users' emotions more accurately since it is converted into standard indicators and used as an emotion judgment element.

Emotional state analysis device through the analysis of the user's biometric information to achieve the object of the present invention comprises a biosensor unit including a pulse wave sensor, a skin temperature sensor and a skin electrical resistance sensor to contact the user's body to extract the biometric information; An analog / digital conversion unit for converting each biometric information extracted from the biosensor unit into a digital signal; The pulse wave ratio (PPG), skin temperature (SKT) and skin electrical resistance (GSR) values of the biosensor unit input through the analog / digital conversion unit are input to calculate the ratio of average heart rate (HR) per minute (GSR / HR). And calculating autonomic balance (LF / HF) parameters, and from the calculated parameters (PPG, SKT, GSR / HR, LF / HF), the corresponding standard indicators (T _SKT , T _SDNN , T _{LF / HF} , T). A main processor for calculating a _{GSR / HR} and determining an emotional state of the user based on the calculated standard indicator value; A memory for storing an emotional state determined by the main processor; And an LCD display unit which displays the emotional state determined by the main processor.
Here, in the calculated standard indicators (T _SKT , T _SDNN , T _{LF / HF} , T _{GSR / HR} ), the emotion type of human positive or negative is determined based on the standard indicators (T _SKT , T _SDNN ,), and the standard Determining the arousal level of tension or relaxation based on the values of the indicators (T _{LF / HF} , T _{GSR / HR} ) and determining the emotional state as "active, relaxed, anxious, relaxed" according to the emotion type and arousal level. It features.

delete

delete

Emotional state analysis process through the analysis of biological information to achieve the object of the present invention is to extract the user's biometric information (PPG, SKT, GSR) extracted by the biosensor unit including the pulse wave sensor, skin temperature sensor and skin electrical resistance sensor In the method of analyzing the current emotional state of the user through the user's emotion analysis process received from the main processor, the autonomic nerve balance diagram calculated from the skin temperature (SKT) and the pulse wave (PPG) signal extracted through the biosensor Calculate standard deviation (SDNN) calculated from (LF / HF), peak time interval (RRI) of pulse wave (PPG), and ratio of skin electrical resistance (GSR) and average heart rate (HR) per minute (GSR / HR) A parameter acquisition process calculated through the process; A standardization process of converting each parameter (LF / HF, SDNN, GSR / HR, SKT) value calculated in the parameter acquisition process into standard indicators (T _i , i = LF / HF, SDNN, GSR / HR, SKT); A coordinate value calculation process of calculating coordinate values (x, y) for analyzing an emotional state using the standard indicators (T _i ); An emotional state analysis process of comparing the calculated coordinate values (x, y) with the classified emotional state regions to determine a current emotional state of the user according to the position of the coordinate values; Emotion type (positive or negative), y-axis is characterized by the degree of arousal (tension or relaxation).

Here, the parameter acquisition process is a pre-process for the parameter acquisition process to calculate the envelope after removing the high- and low-pass components of the measured pulse wave (PPG), and to detect each peak time interval (RRI) of the envelope from the envelope step; A frequency sequence analysis step of removing noise from each peak time interval (RRI) detected in the preprocessing step and then calculating autonomic nerve balance (LF / HF) through frequency sequence analysis (FFT) (PSD); A time series analysis step of calculating a standard deviation (SDNN) and an average heart rate (HR) from the peak time interval (RRI) calculated in the frequency series analysis step; And a GSR / HR calculation step of calculating a ratio (GSR / HR) of the skin electrical resistance (GSR) and the average heart rate per minute (HR) calculated in the time series analysis step.

In addition, the standardization process, the standardization process is a step of calculating the standard index (T _{LF / HF} ) of the autonomic nerve balance (LF / HF) calculated in the frequency sequence analysis step of the parameter acquisition process; Calculating a standard index (T _SDNN ) of the standard deviation (SDNN) calculated in the time series analysis step of the parameter acquisition process; Calculating a standard index (T _{GSR / HR} ) of the GSR / HR calculated in the GSR / HR calculation step of the parameter acquisition process; And 4 steps of calculating the standard indicator (T _SKT ) of the skin temperature (SKT).

In addition, the emotional state analysis process, if the calculated coordinate value (x, y) is x _ref ≤ x <x _max , y _ref ≤ y <y _max , determine the emotional state of the user as the active state, the calculated If the coordinate value (x, y) is x _ref ≤ x <x _max , 0 <y <y _ref , the user's emotional state is determined as a comfort state, and the calculated coordinate value (x, y) is 0 <x < If x _ref , y _ref ≤ y <y _max , the user's emotional state is determined as anxiety state, and if the calculated coordinate value (x, y) is 0 <x <x _ref , 0 <y <y _ref , the user Determining the emotional state of the emotional state region, the reference coordinates (x _ref , y _ref ) of the x, y-axis region of the emotional state region is the sum (M + M) of the mean (M) of each parameter, the maximum value ( x _max , y _max ) is the sum of the maximum values of each parameter (2M + 2M).

Emotional reaction system and control method according to the analysis of the emotional state of the user according to the present invention by determining the emotional state of the user's activity, comfort, anxiety, carrying, etc. through the collection and operation of the biometric information and transmits it to the robot Since the robot is controlled to play music or dance to the music according to the emotional state of the user, there is an effect that the user and the robot can feel the feeling of sympathy.

In addition, the present invention is the autonomic neurological balance (LF / HF) calculated by using the biological signals such as PPG, GSR, SKT collected from the user, SDNN, GSR / HR, and SKT calculated from the RRI of the PPG user By using the emotion state to recognize and determine, there is an effect that can increase the accuracy of judging the emotion of the user.

In addition, since the noise generated by the user's movement during PPG information collection is removed, the envelope can be detected more accurately.

In addition, the present invention has an effect that can be alerted to the user using an alarm sound or vibration motor when the value of the risk level is determined by analyzing the bio-signal, such as PPG, GSR, SKT collected from the user.

1 is a block diagram of a service apparatus based on emotion recognition according to the prior art,
2 is a block diagram of an apparatus for analyzing emotional state through analysis of biometric information of a user according to an embodiment of the present invention;
3A and 3B are flowcharts illustrating an emotional state analysis process through analyzing biometric information of a user according to an embodiment of the present invention.
4 is a graph according to arousal and emotion types for analyzing and determining an emotional state for implementing the present invention,
FIG. 5 is a diagram illustrating an emotion classification graph classifying emotional states into regions according to coordinate values according to an exemplary embodiment of the present invention. FIG.

The apparatus and method for analyzing emotional state through analysis of user's biometric information according to an embodiment of the present invention will be described in more detail with reference to the accompanying drawings.

Figure 2 is a block diagram of the emotional state determination and analysis of the reaction system of the robot according to the analysis of the user's biometric information according to an embodiment of the present invention, wearing and calculating the biometric information in the unrestrained state worn on the body part of the user Emotion analysis device 100 for analyzing and determining the emotional state of the user, and emotional response device 200 to perform a set operation according to the emotional state information transmitted from the emotion analysis device 100.

Here, the emotion analysis apparatus 100 measures the pulse wave sensor 111 composed of an infrared light emitting unit and a light receiving unit, skin electroconductivity (GSR) to measure a user's optical volume pulse wave (PPG), and displays the voltage as a voltage value. Acceler to measure the user's movement change is composed of a biosensor unit consisting of a resistance sensor 112 and the skin temperature sensor 113 to measure the skin temperature (SKT) to display the temperature value, and a micro MEMS 3-axis accelerometer An accelerometer 114, an A / D converter 120 for converting analog biometric information measured from the sensors 111 to 114 into a digital signal, and the A / D converter 120 The main processor 130 for determining the emotional state of the user by calculating the bio-information input through the user, and transmits the emotional state information to the emotion reaction apparatus 200, the input data and the calculation result by the main processor 130 Memo to store processing data 141, an LCD display unit 142 for outputting the input / output information in the main processor 130 by the operation of the switching unit 144, and the emotion analysis apparatus 100 under the control of the main processor 130. LED display unit 143 indicating the communication state or power state of the, and when the measured PPG signal is more than the set reference value by the main processor 130 determines that the user is an emergency and generates an alarm sound or vibration to the user The buzzer 151 and the vibration motor 152 to be aware, and the Zigbee transmitter 153 for wirelessly transmitting the emotional state of the user determined by the main processor 130 to the emotion reaction device 200; And a power supply unit 101 for supplying power to the respective units.

The emotion reaction apparatus 200 includes a Zigbee receiver 201 that receives user emotion state information transmitted through the Zigbee transmitter 153 of the emotion analysis apparatus 100, and a user input from the Zigbee receiver 201. Emotion response controller 210 for driving the emotional response set according to the emotional state, and music playback unit 223 for extracting the music set by the control of the emotional response controller 210 from the sound source database 224 to play through the speaker When the emotional reaction device 200 is a robot, the control unit further includes an operation driver 221 and an operation database 222 for controlling the operation of the robot.

Referring to the operation of the present invention configured as described above in more detail with reference to Figures 2 to 5 as follows.

2 is a detailed block diagram of an emotion analysis device and an emotion reaction device according to an embodiment of the present invention.

First, the emotion analysis device 100 is to be worn on the user's body part, in the present invention is configured as a watch type to be worn on the wrist, pulse wave signal (PPG), skin electrical conductivity (GSR), skin temperature from the wrist (SKT) and the amount of change in movement, and according to the results of the analysis of the received sensor values, classify and determine the current emotional state of the user as active, relaxed, relaxed, anxiety state to the emotion reaction apparatus 200 Wireless transmission.

The pulse wave sensor 111 irradiates infrared rays to the radial artery located on the user's wrist, detects the pulse wave signal PPG which is a change amount of the BVP (Blood Volume Plus) by the reflected infrared rays, and outputs the pulse wave signal PPG to the main processor 130. .

In order to extract the correct RRI from the main processor 130 (PPG), the value changed according to the user's movement must be corrected. For this purpose, the accelerometer 114 composed of an ultra-small MEMS type 3-axis acceleration sensor is provided. By measuring the amount of change in the user's motion, and by correcting the pulse wave signal (PPG) changed due to the movement in the main processor 130 it is possible to extract a more accurate pulse wave signal.

The skin electrical resistance sensor 112 measures the electrical conductivity (GSR) of the skin according to the voltage value measured by the power applied to the electrode in contact with the wrist, and the measured value is amplified to a predetermined level (Voltage) It is represented by a value.

The skin temperature sensor 113 measures the skin temperature of the area in contact with the wrist, the skin temperature measurement value is displayed as an analog temperature value (° C).

The A / D converter 120 converts the analog values measured by the sensors 111 to 114 into digital signals and transmits them to the main processor 130.

The main processor 130 stores the input sensor values in the memory 141 and sets the four emotional states (active, relaxing, anxiety) of the user's current emotional state through a calculation process according to the emotional state analysis and determination algorithm. Or comfort).

On the other hand, the main processor 130 drives the buzzer 151 or the vibration motor 152 when the PPG obtained from the pulse wave sensor 111 is equal to or greater than the set threshold value, so that the user is aware of the alarm sound or vibration mode. In addition to the alarm box, the LCD display unit 142 displays the normal pulse and the current pulse value or graph to inform the degree of danger.

The LED display unit 143 displays a communication state with the emotion reaction device 200, and is configured with each of two color light emitting LEDs to check the battery or power state.

The LCD display unit 142 displays the sensor value input through the sensors 111 to 114 or the calculated emotional state of the user.

The Zigbee transmitter 153 transmits the emotion state information of the user determined by the main processor 130 to the emotion reaction apparatus 200 in a Zigbee wireless communication method.

The emotion reaction apparatus 200 receives the user emotion state information from the emotion analysis apparatus 100 through the ZigBee receiving unit 201 and transmits it to the emotion reaction control unit 210.

The emotional response control unit 210 performs an operation set according to the emotional state of the user. That is, by determining whether music is to be played according to whether the emotional state of the user is active, relaxed, anxious or relaxing, and driving the music player 223, the music player 223 is in an emotional state. Accordingly, the set music is extracted from the stored sound source 224 and the sound source data is classified and reproduced through the speaker.

On the other hand, when the emotion reaction device 200 is mounted on the robot, when active user emotion is transmitted from the emotion analysis device 100, the robot moves and drives the operation driver 221 of the robot together with the music reproduction. By allowing the user to perform a dance movement, the user can be kept active.

3A and 3B are detailed flowcharts of an emotional state analysis process for an emotional reaction of a user through a robot according to an embodiment of the present invention, and are extracted through an emotion analysis device equipped with a pulse wave sensor, a skin temperature sensor, and a skin electrical resistance sensor. An emotion analysis process of receiving the user's biometric information (PPG, SKT, GSR) from the main processor and analyzing the emotion state of the user, and performing a set operation according to the emotion state of the user calculated in the emotion analysis process It is an emotional reaction process.

The emotion analysis process is a process of obtaining PPG, GSR, SKT, motion measurement signal measured by each sensor (S101), autonomic nerve balance (LF / HF) calculated from each of the obtained PPG, PPG The standard deviation (SDNN) calculated from the RRI of RRI, the ratio of GSR and HR (GSR / HR), the parameter acquisition process (S111-S119) for acquiring SKT data, and the parameter acquisition process (S111-S119). Standardization process (S121) for converting the values of each parameter (LF / HF, SDNN, GSR / HR, SKT) into standard indicators (T _i ), and for each emotional state analysis using the standard indicators (T _i ) Comparing the coordinate value calculation process (S123) for calculating a coordinate value (x, y) with the calculated coordinate value (x, y) and the classified emotional state region, the current emotional state of the user according to the position of the coordinate value Determination is made of the emotional state determination process (S130).

The emotional reaction process is a process of extracting and playing music classified according to the emotional state by the emotional reaction device receiving the emotional state of the user determined in the emotional state determination process (S130) from a sound source database.

The parameter acquisition process (S111 ~ S119) is a pre-processing step of calculating the envelope after removing the noise of the high and low frequency components of the measured pulse wave (PPG), and detecting the time interval (RRI) of each peak of the envelope from the envelope (S111) (S113), and the noise is removed from each peak time interval (RRI) detected in the pre-processing step (S111) (S113), and then autonomic nerve balance (LF /) through frequency sequence analysis (FFT) (PSD) A time series analysis step of calculating a standard deviation SDNN and an average heart rate HR per minute from the frequency sequence analysis step S115 for calculating HF and the peak time interval RRI calculated in the frequency sequence analysis step S115. (S117), and the GSR / HR calculation step (S119) for calculating the ratio (GSR / HR) of the skin electrical resistance (GSR) and the average heart rate (HR) per minute calculated in the time series analysis step.

In addition, the emotional state determination process (S130) by comparing the calculated coordinate values (x, y) with a reference value, if x _ref ≤ x <x _max , y _ref ≤ y <y _max (S131) (S135), If the user's emotional state (S140) is determined as the active state (S141), and the calculated coordinate values (x, y) is x _ref ≤ x <x _max , 0 <y <y _ref (S131) (S135), If the user's emotional state is determined as the comfort state (S142), and the calculated coordinate values (x, y) are 0 <x <x _ref , y _ref ≤ y <y _max (S131) (S137), the user's emotion The state is determined to be an unstable state (S143), and if the calculated coordinate values (x, y) are 0 <x <x _ref , 0 <y <y _ref (S131) (S137), the user's emotional state is carried out. It is determined at S144.

Referring to the emotional state analysis process for the emotional reaction of the user through the robot according to an embodiment of the present invention as described above in detail as follows.

First, the watch-type emotion analysis device 100 worn by a user receives PPG, GSR, SKT, and a user's movement change amount from each of the sensors 111 to 114 provided therein, and converts the digital signal into the main signal. 130).

The process of analyzing and determining the emotional state of the user using the PPG, GSR, and SKT measurement values input from each sensor in the main processor 130 will now be described in detail.

The main processor 130 removes the noise included in the PPG through the noise removing process (S111), and then, through the pre-processing process, the high-frequency and low-frequency components through the low-pass filter and the high-pass filter. After removing, we generate the envelope waveform by the Square Root function.

In the noise removing process S111, when a user moves during the PPG measurement, noise due to the movement is generated. At this time, the accurate envelope cannot be obtained due to the generated noise, and the detected noise in the PPG containing the noise should be removed according to the amount of change in the user's movement.

In this case, the noise includes the DC component in the PPG measurement according to the user's movement. The DC value is compensated for the measured PPG signal to obtain the correct PPG.

The user's motion determines whether the user is stationary or in motion according to the signal obtained through the accelerometer 114, and removes noise of the PPG according to the change amount of the motion, thereby detecting more accurate envelopes.

The detected envelope calculates an RRI from the envelope using an envelope detection algorithm, wherein the time interval becomes a data point between R-peak and R-peak.

In this case, since the minimum number of RRIs used in the frequency analysis is 128, it is determined that if the number of RRIs is less than 128 (n <128), RRI data is continuously collected and 128 or more (n≥ 128), unnecessarily slow or very low frequency vibration signals are removed from the obtained RRI signal (HRV signal), and then autonomic nerve balance (LF / HF) is calculated through frequency sequence analysis.

That is, the HRV signal calculates a ratio of sympathetic activity (LF) and sympathetic activity (HF: High Frequency) through frequency sequence analysis (FFT, PSD) process (S115). The ratio of and sympathetic activity (HF) is autonomic balance (LF / HF), sympathetic activity (LF) means the area of the frequency domain between 0.05Hz and 0.15Hz, and parasympathetic activity (HF) at 0.15Hz It means the area of the frequency domain between 0.4Hz.

In addition, the frequency sequence analysis process S115 converts the frequency sequence to an HRV signal through a fast fourier transform (FFT) process, and calculates an absolute power value of the corresponding frequency from the HRV signal through a power spectral density (PSD).

On the other hand, the RRI value calculated through the preprocessing (S113) calculates the SDNN and heart rate (HR) through the time series analysis process (S117), and the HR per minute (HR) calculated through the time series analysis process (S117). The ratio (GSR / HR) of the GSR obtained from the skin electrical resistance sensor 112 is calculated.

Each of the calculated parameters LF / HF, SDNN, GSR, GSR / HR, and SKT should be interpreted as a statistical distribution of the same mean and standard deviation as one standard indicator through the standardization process (S121).

That is, as shown in Equation 1, each parameter (LF / HF, SDNN, GSR / HR, SKT) calculates each standard indicator (T _{LF / HF} , T _SDNN , T _{GSR / HR} , T _SKT ). do.

Here, P _i is the calculated LF / HF, SDNN, GSR / HR, SKT parameter values, m _i is the standard deviation of the mean of the standard group, M is the mean that is the basis of each parameter, SD Is the standard deviation based on each parameter, and sd _i is the standard deviation with respect to the mean (m _i ) of the standard group. In an embodiment of the present invention, the M value is set to 50 and may be changed as necessary.

In this way, each parameter value is converted into a standard index, and then a coordinate value for determining emotion is calculated.

Detailed description of the standardization process according to the standard index calculation method according to Equation 1 is published in Korean Patent Publication No. 10-0745972 (July 28, 2007) and the paper "Fingertip photoplethysmography and migraine" (Journal: Journal of the Neurological Sciences, Volume 216). , Issue 1, Pages 17-21, Publication Date: 15 December 2003), and detailed descriptions of the calculation method of the analysis index (Pi), the method of extracting the standard index (Ti), and the clinical significance are omitted.

4 is an emotional state area display graph for determining the emotional state according to the emotion type and arousal degree of a person for implementing the present invention, the emotion of the person (Valence, positive / negative) and arousal (Arousal, tension) / Relaxation) is divided into four areas including active, anxious, relaxed, and relaxed with each x, y axis. (S130)

In other words, the arousal and emotion types are divided into four areas: the active state area when tension and affirmation, the anxiety state area when tension and negative, and the relaxing state area when relaxation and negative. .

Here, the emotion of the active state area includes not only active but also similar emotional states such as joy, excitement and surprise (positive), and the anxiety state area includes emotional states similar to tension, stress and anger as well as anxiety. The relaxing state area includes not only relaxing but also emotional state similar to depression, depressed, etc., and the relaxing state area includes not only comfort but also emotional state similar to Deeply Relaxed, Blissful, etc. Include.

More specifically, the parameter GSR / HR calculated from each sensor value obtained in each of the sensors 111 to 114 is related to the arousal state of the person, and SDNN has a positive and negative emotion type ( Since the emotional state of the human being is determined by dividing the domain into active, anxious, relaxed, and relaxed states using the parameters SDNN and GSR / HR values as the x and y axes, respectively.

FIG. 5 is a view showing coordinates and an emotional region for determining an emotional state according to an embodiment of the present invention, wherein the LF / HF parameter affecting the aprousal and the emotional type for determining the emotional state is described. It is added to parameter GSR / HR of the awake (tension / relaxation) state and skin temperature (SKT) parameter affecting the emotion type (positive / negative) to the parameter SDNN of the positive / negative state x-axis. As a result, more accurate emotional state can be determined.

As such, when the emotional state of a person is judged only by the parameters GSR / HR and SDNN, not only the emotional state at the boundary of the coordinates that is difficult to determine can be judged as a clearer emotional state, but also the deepening of each emotional state. You can judge the degree.

That is, since a plurality of parameters affecting a person's emotional state are used as an emotional state judgment element, the emotional state of the user can be determined more accurately.

The emotional state judgment area for determining the emotional state of a person is as shown in Equation 2 below, where the x-axis is the sum of the parameters SDNN and standard indicators of SKT (T _SDNN). + T _SKT ), and the y-axis is set as the sum of the standard indicators of the parameters GSR / HR and LF / HF (T _{LF / HF} + T _{GSR / HR} ).

delete

이와 같이 설정된 좌표값(x,y)은 기준좌표값(x_ref, y_ref)에 의해 4개의 영역으로 나누어지며, 좌표값(x,y)이 "x_ref ≤ x < x_max, y_ref ≤ y < y_max"이면 제1영역, "x_ref ≤ x < x_max , 0 < y < y_ref"이면 제2영역, "0 < x < x_ref, y_ref ≤ y < y_max"이면 제3영역, "0 < x < x_ref, 0 < y < y_ref"이면 제4영역으로 분할한다.
상기 제1영역(①)은 활발(기쁨, 흥분, 놀람상태 등) 제2영역(②)은 평안(릴렉스 상태등), 제3영역(③)은 불안(긴장, 스트레스, 분노상태 등), 제4영역(④)은 나른(우울, 슬픔, 디프레스 상태 등)으로 감정상태를 설정한다.

The coordinate values (x, y) set as described above are divided into four regions by the reference coordinate values (x _ref , y _ref ), and the coordinate values (x, y) are defined as "x _ref ≤ x <x _max , y _ref ≤ the first region if y <y _max ", the second region if" x _ref ≤ x <x _max , 0 <y <y _ref ", and the third region if" 0 <x <x _ref , y _ref ≤ y <y _max " If the area, "0 <x <x _ref , 0 <y <y _ref ", it is divided into a fourth area.
The first area (①) is active (joy, excitement, surprise state, etc.) The second area (②) is peace (relax state, etc.), and the third area (③) is anxiety (tension, stress, anger state, etc.), The fourth area ④ sets the emotional state in a relaxing manner (depressed, sadness, depressed state, etc.).

Here, the reference coordinates (x _ref , y _ref ) of the x and y axis regions of the emotional state region are the sum (M + M) of the mean (M) of each parameter, and the maximum values (x _max , y _max ) are each The sum of the maximum values of the parameters (2M + 2M).
In an embodiment, the reference coordinate values (x _ref , y _ref ) are set to an average (M) of 50 for each parameter, and the reference coordinate value (x _ref ) on the x-axis is the sum of two standard indicators (T _SDNN + T _SKT ), so set it to 100, which is the average sum of parameters (M + M). Since the reference coordinate value (y _ref ) of the y-axis is also the sum of two standard indicators (T _{LF / HF} + T _{GSR / HR} ), it is set to 100, which is the average sum of the parameters (M + M). Therefore, the minimum value (x _min , y _min ) of each coordinate value (x, y) is set to '0,0' and the maximum value (x _max , y _max ) is set to '200,200'.

If the coordinate value (x, y) calculated by the user's biometric information acquisition process, the parameter and standard indicator calculation process, and the coordinate value calculation process is 100 ≦ x <200, 100 ≦ y <200, Therefore, it is determined that the emotional state of the user as the active state (S131, S135, S141) if the calculated coordinate values (x, y) is 100 ≤ x <200, 0 <y <100, it corresponds to the fourth region The user's emotional state is determined as a comfort state (S131, S135, S142) if the calculated coordinate values (x, y) is 0 <x <100, 100 ≤ y <200, the third area is the user's The emotional state is determined as an anxiety state (S131, S137, and S143). If the calculated coordinate values (x, y) are 0 <x <100, 0 <y <100, the fourth region corresponds to the fourth region. It is determined that the state is carried (S131, S137, S144).
Here, each coordinate value is an embodiment, and the maximum and minimum values of x and y may be variously modified according to the set value of the average M.

The emotional state of the user calculated through each process is wirelessly transmitted to the emotion reaction apparatus 200 through the Zigbee transmitter 153 to perform a reaction operation set by the emotion reaction controller 210.

The emotion reaction apparatus 200 may be various embodiments capable of responding to human emotions, such as a robot and a music playback apparatus. Here, the emotion reaction apparatus 200 may be applied to the robot as an embodiment of the emotion reaction apparatus 200. When mounted, when the emotion analysis apparatus 100 receives the user emotion state information transmitted through the Zigbee communication, the emotion reaction control unit 210 according to the user's emotion transmitted from the emotion analysis apparatus 100 according to the music playback unit ( 223 plays the set music, and when the emotional state of the user is active, plays the set music and drives the operation driver 221 to perform the set operation at the same time.

That is, the operation of the operation driver 221 extracts various types of dance motion data and algorithms stored in the motion database 222 to maintain the active state emotion of the user and drives the robot to perform the corresponding motion.

Here, as another embodiment of the emotion reaction device 200, not only a robot or a music playback device, but also various types of emotion reaction means that may affect the emotional state of the user is possible.

Emotional state analysis device and method through the analysis of the user's biometric information according to the present invention described above is not limited to the above embodiment, it can be variously changed according to the configuration of the emotional response device, active, comfortable user's emotion It is possible to be variously modified according to other emotional state in addition to, and anxiety state, any person having ordinary knowledge in the field to which the present invention belongs without departing from the gist of the present invention claimed in the claims below. The technical spirit is to the extent that it can be changed and implemented.

100: emotion analysis device 101: power supply
111: pulse wave sensor 112: skin electrical resistance sensor
113: skin temperature sensor 114: accelerometer
120: A / D conversion unit 130: main processor
141: memory 142: LCD display
143: LED display unit 144: switching unit
151: buzzer 152: vibration motor
153: Zigbee transmitter
200: Emotional response device 201: Zigbee receiving unit
210: emotional response control unit 221: operation driver
222: motion database 223: music player
224: sound source database

Claims (12)

A biosensor unit including a pulse wave sensor, a skin temperature sensor, and a skin electrical resistance sensor that are in contact with a user's body to extract biometric information;
An analog / digital conversion unit for converting each biometric information extracted from the biosensor unit into a digital signal; And
The pulse wave ratio (PPG), skin temperature (SKT) and skin electrical resistance (GSR) values of the biosensor unit input through the analog / digital conversion unit are input to calculate the ratio of average heart rate (HR) per minute (GSR / HR). And calculating autonomic balance (LF / HF) parameters, and from the calculated parameters (PPG, SKT, GSR / HR, LF / HF), the corresponding standard indicators (T _SKT , T _SDNN , T _{LF / HF} , T). A main processor for calculating a _{GSR / HR} and determining an emotional state of the user based on the calculated standard indicator value;
A memory for storing an emotional state determined by the main processor; And
LCD display for displaying the emotional state determined by the main processor;
The main processor determines the emotion type of a person's positive or negative based on the standard indicator (T _SKT , T _SDNN ,) value from the calculated standard indicators (T _SKT , T _SDNN , T _{LF / HF} , T _{GSR / HR} ). , By determining the arousal level of tension or relaxation by the standard indicator (T _{LF / HF} , T _{GSR / HR} ) value, and determines the emotional state as "active, relaxed, anxious, relaxed" according to the emotion type and arousal level Emotional state analysis device through the analysis of the user's biometric information, characterized in that.
delete The method of claim 1,
If the pulse wave measured from the pulse wave sensor by the main processor is more than the set reference value is determined as an emergency situation, the user characterized in that it further comprises a buzzer for generating an alarm sound to the user, and a vibration motor for alarm in the vibration mode Emotional state analysis device through the analysis of biometric information.
The method of claim 1,
In addition to the biosensor unit to correct the pulse wave transformed by the user's movement, it further comprises an accelerometer (Accelerometer) for measuring the user's movement change emotions through analysis of the user's biometric information Condition analyzer.
The method of claim 1,
And a wireless transmitter for wirelessly transmitting the emotional state determined by the main processor to an external system.
The user's current emotional state is received through the user's emotion analysis process by receiving the user's biometric information (PPG, SKT, GSR) extracted by the biosensor unit including pulse wave sensor, skin temperature sensor and skin electrical resistance sensor. In the method of judging,
The standard deviation calculated from the skin temperature SKT extracted through the biosensor unit, the autonomic nerve balance calculated from the pulse wave (PPG) signal, and the peak time interval (RRI) of the pulse wave (PPG). A parameter acquisition process of calculating a SDNN, a ratio of skin electrical resistance (GSR) and average heart rate (HR) per minute (GSR / HR) through a calculation process;
A standardization process of converting each parameter (LF / HF, SDNN, GSR / HR, SKT) value calculated in the parameter acquisition process into standard indicators (T _i , i = LF / HF, SDNN, GSR / HR, SKT);
A coordinate value calculation process of calculating coordinate values (x, y) for analyzing an emotional state using the standard indicators (T _i );
Emotional state determination process for determining the current emotional state of the user according to the calculated coordinate values (x, y);
The x-axis of the coordinate value is an emotion type (positive or negative), the y-axis is a degree of arousal (tension or relaxation), the emotional state analysis method through the user's biometric information analysis.
The method according to claim 6,
The parameter acquisition process may include a preprocessing step of calculating envelopes after removing high and low frequency noise of the extracted pulse wave (PPG), and detecting each peak time interval (RRI) of the envelopes from the envelopes;
A frequency sequence analysis step of removing noise from each peak time interval (RRI) detected in the preprocessing step and then calculating autonomic nerve balance (LF / HF) through frequency sequence analysis (FFT) (PSD);
A time series analysis step of calculating a standard deviation (SDNN) and an average heart rate (HR) from the peak time interval (RRI) calculated in the frequency series analysis step; And
And a GSR / HR calculation step of calculating a ratio (GSR / HR) of the skin electrical resistance (GSR) and the average heart rate (HR) calculated in the time series analysis step.
Each parameter calculated in each step is an emotional state analysis method through the user's biometric information, characterized in that for calculating the standard indicator through the standardization process.
The method according to claim 6 or 7,
The standardization process is a step of calculating a standard index (T _{LF / HF} ) of the autonomic nervous system balance (LF / HF) calculated in the frequency sequence analysis step of the parameter acquisition process;
Calculating a standard index (T _SDNN ) of the standard deviation (SDNN) calculated in the time series analysis step of the parameter acquisition process;
Calculating a standard index (T _{GSR / HR} ) of the GSR / HR calculated in the GSR / HR calculation step of the parameter acquisition process;
And a four step of calculating the standard indicator T _SKT of the skin temperature SKT.
The method of claim 8,
Standard indicators (T _i , i = LF / HF, SDNN, GSR / HR, SKT) calculated through the above steps are calculated by Equation 1 below. Way.
&Quot; (1) &quot;

Where P _i is the calculated LF / HF, SDNN, GSR / HR and SKT parameter values, m _i is the standard deviation of the mean of the standard group, M is the mean that is the basis of each parameter, and SD Is the standard deviation based on each parameter, and sd _i is the standard deviation of the mean (m _i ) of the standard group.
The method of claim 7, wherein
And a noise removing step of removing noise of the pulse wave (PPG) signal generated by the user's movement when the amount of change in the user's movement is detected from the accelerometer before the preprocessing step. Emotional state analysis method through the analysis of the user's biometric information.
The method according to claim 6,
The calculation of the coordinate value (x, y) in the coordinate value calculation process is the emotional state analysis method through the user's biometric information, characterized in that it is made by including the following equation (2).
&Quot; (2) &quot;

Here, T _SKT , T _SDNN , T _{LF / HF} and T _{GSR / HR} are standard indicators of SKT, SDNN, LF / HF, and GSR / HR.
The method according to claim 6,
In the emotional state determining process, if the coordinate values (x, y) calculated through the coordinate value calculation process are x _ref ≤ x <x _max and y _ref ≤ y <y _max , the emotional state of the user is determined as an active state. ,
If the calculated coordinate values (x, y) is x _ref ≤ x <x _max , 0 <y <y _ref , the user's emotional state is determined as a comfort state,
If the calculated coordinate values (x, y) is 0 <x <x _ref , y _ref ≤ y <y _max , the user's emotional state is determined as anxiety state,
If the calculated coordinate values (x, y) is 0 <x <x _ref , 0 <y <y _ref , the user's emotional state is determined as being separated,
The reference coordinates (x _ref , y _ref ) of the x and y axis regions of the emotional state region are the sum (M + M) of the mean (M) of each parameter, and the maximum value (x _max , y _max ) is the value of each parameter. Emotional state analysis method through the analysis of the user's biological information, characterized in that the sum of the maximum value (2M + 2M).

Images