KR102630803B1 - Emotion analysis result providing device and emotion analysis result providing system - Google Patents

Abstract

An emotional analysis result providing device according to an embodiment analyzes the client's emotions in real time based on at least one of first voice data and first video data extracted from video data received in real time from the client terminal and provides a plurality of first analysis results. A first analysis result acquisition unit that obtains; and
Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated over a predetermined time, the real-time received video data analyzes the client's emotions after the predetermined time and generates a plurality of second It may include a second analysis result acquisition unit that acquires the analysis results.

Description

Emotion analysis result providing device and emotion analysis result providing system {Emotion analysis result providing device and emotion analysis result providing system}

The present invention relates to an emotional analysis result providing device and an emotional analysis result providing system. More specifically, the present invention relates to a real-time client's emotions based on video data during psychological counseling, and also analyzes the client's emotions based on accumulated video data. It relates to an emotional analysis result providing device and an emotional analysis result providing system for analyzing a client's emotions from various angles and providing integrated results by analyzing them.

Psychological counseling is based on empathic understanding, unconditional positive respect, and sincerity in relationships with clients who are dealing with psychological problems, and is based on various theories of counseling psychology, including psychoanalysis, behaviorism, humanism, cognitivism, and formalism. , a treatment method that helps solve their problems by using content such as reality therapy, transactional analysis, and family therapy, and guides clients to explore human thoughts, emotions, behavior, and interpersonal relationships to understand and change their various problems. It means helping.

This type of psychological counseling was mainly done through the traditional counseling method that was conducted face-to-face in a given counseling room. Clients or patients with interpersonal problems or severe social anxiety needed face-to-face counseling and psychotherapy that required them to meet directly with the counselor. Psychological treatment may not be continued or may be discontinued due to burden or reluctance.

Therefore, as an alternative to face-to-face counseling, online psychotherapy programs that provide psychological counseling and treatment online are being proposed.

KR 2022-0005945 A

The present invention analyzes the client's emotions in real time based on video data when conducting an online psychotherapy program and also analyzes the client's emotions based on accumulated video data, thereby analyzing the client's emotions from various angles and providing integrated results. The purpose is to provide.

An emotional analysis result providing device according to an embodiment analyzes the client's emotions in real time based on at least one of the first voice data and the first video data extracted from video data received in real time from the client terminal, and produces a plurality of first analysis results. A first analysis result acquisition unit that obtains; and

Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated over a predetermined time, the real-time received video data analyzes the client's emotions after the predetermined time has elapsed and generates a plurality of second It may include a second analysis result acquisition unit that acquires the analysis results.

The plurality of first analysis results are:

It may include at least two of the following: probability values for each of a plurality of general emotional states, representative emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword organized by topic, voice parameters, gaze position information, and facial movement information. You can.

The results of the plurality of second analyzes are:

Probability values for each of a plurality of general emotional states, probability values for each of a plurality of substituted emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword organized by topic, voice parameters, gaze position information, and facial movement information. It may contain two or more.

The plurality of substituted emotional states may be a combination of emotional states defined as superordinate concepts of each of the plurality of general emotional states.

The second analysis result acquisition unit,

Probability values for each of a plurality of general emotional states may be obtained using a plurality of models based on ensemble learning based on the second image data, the second voice data, and text data converted from the second voice data.

The second analysis result acquisition unit,

Latent Dirichlet Allocation (LDA) topic modeling can be performed on the text data converted from the second voice data to obtain the frequency of each of the keywords configured for each topic.

The second analysis result acquisition unit,

Recognize the client's face based on the second image data, detect feature points of the recognized face, detect the pupil center position in a designated area of interest based on the feature points of the face, and detect the pupil center position of the detected pupil center position. The gaze position information, which is the degree of change over time, can be calculated based on the change in Euclidean distance.

An emotional analysis result providing system according to an embodiment includes an emotional analysis result providing system including a client terminal, a counselor terminal, and an emotional analysis result providing device,

The emotional analysis result providing device,

A first analysis result acquisition unit that acquires a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of first voice data and first video data extracted from video data received in real time from the client terminal;

Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated over a predetermined time, the real-time received video data analyzes the client's emotions after the predetermined time has elapsed and generates a plurality of second a second analysis result acquisition unit that acquires analysis results; and

A transmission unit configured to transmit information including all of the plurality of first analysis results and information including all of the plurality of second analysis results to the counselor terminal,

The counselor terminal is,

a first analysis result providing unit that provides information including all of the plurality of first analysis results received from the emotional analysis result providing device through one user interface screen; and

It may include a second analysis result providing unit that provides information including a second analysis result selected by the user among all the plurality of second analysis results received from the emotional analysis result providing device through the one user interface screen. You can.

The one user interface screen includes a plurality of menus representing the plurality of second analysis results,

The second analysis result providing unit,

Information including the second analysis result corresponding to the menu selected by the user among the plurality of menus may be provided through the single user interface screen.

According to the present invention, the first analysis result is obtained through real-time analysis through the first analysis result acquisition unit, but the second analysis result is obtained through analysis after a predetermined time has elapsed through the second analysis result acquisition unit, so that the first analysis result is acquired through real-time analysis with the same client. By allowing consultation details to be provided in real time through real-time analysis, and by allowing the entire consultation details to be provided through integrated analysis after the consultation is over, various types of analysis results can be provided for one consultation details. It becomes possible to manage more diverse and systematic consultation details.

In addition, the user interface screen when a consultation is in progress and the user interface screen that provides analysis results after the consultation is completed are implemented differently, making it easier to manage and confirm the counselor's consultation details. For example, when a consultation is in progress, real-time analysis results are provided while the overall analysis results are provided through a single user interface screen, making it quicker and easier for the counselor to check, analyze, and/or make decisions about real-time consultation details. When the consultation is completed, one (selected/individual) analysis result of the entire accumulated consultation history is provided through a screen in one user interface unit, allowing the user to check the entire consultation history in an integrated manner. do.

1 is a system diagram of a system 1 for providing emotion analysis results according to an embodiment.
2 and 3 are flowcharts explaining the operation of the device 20 for providing emotion analysis results according to an embodiment.
FIG. 4 illustrates a screen of the user interface unit 33 according to the operation of the first analysis result providing unit 31 of the counselor terminal 30.
5 to 11 illustrate screens of the user interface unit 33 according to the operation of the second analysis result providing unit 32 of the counselor terminal 30.
Figures 12 and 13 are diagrams referenced to explain the operation of the emotion analysis result providing device 20.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

For reference, in each flow chart of the present invention, each step is an example, and the present invention can be applied in the same/similar manner even when each order is changed and/or combined differently.

1 is a system diagram of a system 1 for providing emotion analysis results according to an embodiment.

The client terminal 10 is equipped with an application for conducting text/voice/video counseling and can be connected to the emotion analysis result providing device 20, through which text/voice/video counseling can be performed with the counselor terminal 30. there is. The client terminal 10 has a screen on which the counselor's appearance is displayed, a camera (not shown) that can capture the client's image and transmit it to the counselor terminal 30, and a microphone (not shown) that collects voice and allows a conversation to be exchanged with the counselor. (not shown) and a speaker (not shown) must be provided at least.

The client terminal 10 is a real-time emotional analysis result providing device (20) that provides video data including video data obtained by capturing the client's face using a camera (not shown) and the client's voice data collected using a microphone (not shown). ) can be transmitted.

Depending on the embodiment, the client terminal 10 may output the emotional analysis results received from the emotional analysis result providing device 2 through a screen of a user interface unit (not shown) and provide them to the client.

The counselor terminal 30 is equipped with an application for conducting text/voice/video counseling and can be connected to the emotion analysis result providing device 20, through which text/voice/video counseling can be performed with the client terminal 10. there is. The counselor terminal 30 is equipped with at least a screen on which the client's image is displayed, a camera that can capture the counselor's image and transmit it to the client terminal 10, and a microphone and speaker that collect voice and enable conversation with the client. It has to be.

The counselor terminal 30 is a real-time emotional analysis result providing device ( 20).

The counselor terminal 30 may output the emotional analysis results received from the emotional analysis result providing device 20 through the screen of the user interface unit 33 and provide them to the counselor or manager (user).

The emotion analysis result providing device 20 mediates the video consultation between the client terminal 10 and the counselor terminal 30, and transmits the result of the emotional analysis of the client according to the video consultation to the client terminal 10 and/or the counselor terminal 30. ) can be transmitted.

2 and 3 are flowcharts explaining the operation of the device 20 for providing emotion analysis results according to an embodiment.

Depending on the embodiment, the first analysis result acquisition unit 21a may receive real-time video data from the client terminal 10 (s11).

The video data may include first video data and first audio data in which the client's face is captured.

The first analysis result acquisition unit 21a can extract first audio data and first video data from video data (s12).

Depending on the embodiment, the first video data may be a single video frame received in real time, and the first audio data may be a single audio frame received in real time.

Depending on the embodiment, extracting the first image data and the first audio data from the video data may be implemented through various known algorithms.

The first analysis result acquisition unit 21a may acquire a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of the first voice data and the first image data (s13).

According to the embodiment, the first analysis result acquisition unit 21a is the same as the second analysis result acquisition unit 21b in order to obtain analysis results corresponding to the same category based on at least one of audio data and video data. A plurality of first analysis results can be obtained in real time through real-time analysis using similar analysis methods or various known algorithms. Accordingly, detailed description of the analysis to obtain a plurality of first analysis results will be omitted.

For example, the analysis method for the second analysis result acquisition unit 21b to obtain the probability value of each of a plurality of general emotional states includes the first analysis result acquisition unit 21a obtaining the probability value of each of a plurality of general emotional states. Can be used the same/similarly.

Depending on the embodiment, the plurality of first analysis results include probability values for each of a plurality of general emotional states, representative emotional states, heart rate and stress level, frequency for each keyword, frequency for each keyword organized by topic, voice parameter, and gaze position. It may include at least two of information and facial movement information.

Depending on the embodiment, the plurality of general emotional states may include at least two of joy, anger, contempt, surprise, fear, calmness, and sadness at the relevant time, but the present invention is not limited thereto.

Depending on the embodiment, the first analysis result acquisition unit 21a may acquire probability values for each of a plurality of general emotional states based on the first image data and the first audio data, which is provided by the second analysis result acquisition unit ( 21b) can be applied in the same/similar manner to obtaining probability values for each of a plurality of general emotional states based on the second image data and the second audio data.

Depending on the embodiment, the representative emotional state may be a final emotional state that can be represented by the client's emotion among a plurality of general emotional states at the relevant time. For example, the first analysis result acquisition unit 21a may determine the general emotional state corresponding to the highest probability value among the plurality of probability values for each general emotional state as the representative emotional state at the relevant time.

Depending on the embodiment, the heart rate may be obtained as an RPPG value, and the degree of stress may be obtained as a level such as high/medium/low.

Depending on the embodiment, the first analysis result acquisition unit 21a may acquire the heart rate and stress index based on the first image data. This may be applied in the same/similar manner to obtaining heart rate and stress index based on the second image data of the second analysis result acquisition unit 21b.

Depending on the embodiment, the frequency of each keyword may be defined as the frequency of each keyword uttered by the client among preset keywords. Depending on the embodiment, preset keywords may be changed and set by a user such as a real-time counselor or manager.

Depending on the embodiment, the first analysis result acquisition unit 21a may perform analysis on text data converted from the first voice data to obtain the frequency of each preset keyword.

This can be applied in the same/similar manner to the second analysis result acquisition unit 21b performing analysis on the text data converted from the second voice data to obtain the frequency of each preset keyword.

According to the embodiment, the first analysis result acquisition unit 21a performs Latent Dirichlet Allocation (LDA) topic modeling on the text data into which the first voice data has been converted to obtain information on each of the keywords configured for each topic. The frequency can be obtained, which is achieved by the second analysis result acquisition unit 21b performing Latent Dirichlet Allocation (LDA) topic modeling on the text data into which the second voice data has been converted, and keywords organized for each topic. It can be applied in the same/similar way to obtaining the frequency for each.

Depending on the embodiment, the first analysis result acquisition unit 21a may acquire voice parameters by analyzing based on the first voice data, which means that the second analysis result acquisition unit 21b may acquire voice parameters based on the second voice data. It can be applied in the same/similar way to obtaining voice parameters by analyzing .

Depending on the embodiment, voice parameters may include indicators indicating waveform, pitch, pause section, speech rate, and/or hesitation section.

Depending on the embodiment, the first analysis result acquisition unit 21a may acquire gaze position information and facial movement information by analyzing the first image data based on the first analysis result acquisition unit 21b. 2 It can be applied in the same/similar way to obtaining gaze position information and facial movement information by analyzing video data.

Depending on the embodiment, the first analysis result acquisition unit 21a generates a plurality of general emotional states in the form of a bar graph, generates representative emotional states by applying them to an icon shape and/or a radial graph, and generates frequencies for each preset keyword. is created by reflecting the size of the circle, and the frequency of each keyword organized by topic can be generated in table form. Depending on the embodiment, in the case of a preset frequency for each keyword, it can be generated by having it reflected in the size of the circle according to the level (e.g., level 1 to 5) corresponding to the frequency. For example, level 1 represents utterances made by the client once and is expressed as a relatively small circle, and level 5 represents utterances made by the client more than 5 times and is expressed as a relatively large circle. .

However, this is an example, and each of the plurality of first analysis results can be generated by applying various shapes, such as a bar graph, an icon, a radial graph, or a circular shape.

The transmission unit 21c may transmit information including all of the plurality of first analysis results to the counselor terminal 30 (s14).

Depending on the embodiment, the information including all of the plurality of first analysis results may include all of the plurality of first analysis results, video data received in real time from the client terminal 10, and the client's first analysis results separated from the video data received in real time. It may include content of text data converted from voice data and text data converted from the counselor's voice data.

FIG. 4 illustrates a screen of the user interface unit 33 according to the operation of the first analysis result providing unit 31 of the counselor terminal 30.

The first analysis result providing unit 31 of the counselor terminal 30 provides information including all of the plurality of first analysis results received from the emotional analysis result providing device 20 and counseling notes recorded by the counselor in real time into one user interface unit. It can be provided to users such as managers or counselors by printing it through the screen in (33).

Depending on the embodiment, the information including all of the plurality of first analysis results may include all of the plurality of first analysis results, video data in which the client was filmed, and text data converted from the client's voice data separated from the video data in which the client was filmed. It may include content corresponding to the text data converted from the counselor's voice data.

Depending on the embodiment, each content may be output in separate areas on the screen of the user interface unit 33.

For example, Figure 4 is an example of the screen of the user interface unit 33, in which a plurality of first analysis results (①), video data (②) received in real time from the client terminal 10, and the client's voice data are displayed. It may include converted text data and text data converted from the counselor's voice data (④), and consultation notes recorded by the counselor in real time (③).

(1-1) represents multiple general emotional states among the plurality of first analysis results (①), (1-2) represents representative emotional states, (1-3) represents heart rate and stress index, (1- 4) represents the frequency of each keyword, and (1-5) represents the frequency of each keyword organized by topic.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate accumulated video data in which video data received in real time is accumulated over a predetermined period of time (s21).

According to the embodiment, the second analysis result acquisition unit 21b collects the received video data in the section between the time when it started receiving video data from the client terminal 10 and the time when it finished receiving video data from the client terminal 10. It can be obtained as accumulated video data.

According to the embodiment, the second analysis result acquisition unit 21b may acquire video data within the section between the start of recording of video data received from the client terminal 10 and the end of recording as cumulative video data. there is.

Depending on the embodiment, commands for defining video data accumulated over the predetermined time as an analysis target are pre-programmed, and the second analysis result acquisition unit 21b can define the analysis target with reference to the commands. .

Depending on the embodiment, the predetermined time may correspond to a time period from the start of the consultation between the client and the counselor until the end.

Depending on the embodiment, the second analysis result acquisition unit 21b may extract second audio data and second video data from the accumulated video data (s22).

Depending on the embodiment, the first video data may be a single video frame received in real time, while the second video data may include a plurality of video frames accumulated over a predetermined time.

Depending on the embodiment, the first voice data may be a single voice frame received in real time, while the second voice data may include a plurality of voice frames accumulated over a predetermined time.

Depending on the embodiment, extracting the second video data and the second audio data from the accumulated video data may be implemented through various known algorithms.

The second analysis result acquisition unit 21b may acquire a plurality of second analysis results by analyzing the client's emotions after the predetermined time has elapsed, based on at least one of the second voice data and the second image data ( s23).

Depending on the embodiment, a command for analyzing immediately after a predetermined time has elapsed (or after a certain time has elapsed) or at a time set by the user may be programmed in advance, and the second analysis result acquisition unit 21b ) can perform analysis by referring to the corresponding command.

Depending on the embodiment, the plurality of second analysis results may include probability values for each of a plurality of general emotional states, probability values for each of a plurality of substituted emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword configured for each topic, and voice. It may include at least two of parameters, gaze position information, and facial movement information.

Depending on the embodiment, the second analysis result acquisition unit 21b may graph at least some of the plurality of second analysis results and transmit them.

Depending on the embodiment, the plurality of general emotional states may include at least two of joy, anger, contempt, surprise, fear, calmness, and sadness at the relevant time (or time), but the present invention is not limited thereto.

Depending on the embodiment, the probability value of each of a plurality of general emotional states may be obtained by averaging the probability values of the general emotional states acquired during a predetermined time period.

Depending on the embodiment, the second analysis result acquisition unit 21b may create and transmit a graph of the probability values of each of a plurality of general emotional states for each time period, as shown in FIG. 5 .

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire probability values for each of a plurality of general emotional states based on the second image data and the second audio data.

According to the embodiment, the second analysis result acquisition unit 21b uses a plurality of models based on ensemble learning based on the second image data, the second voice data, and the text data converted from the second voice data to generate a plurality of general The probability value of each emotional state can be obtained.

Specifically, the second analysis result acquisition unit 21b includes a plurality of video frames constituting the second video data, a plurality of audio frames constituting the second audio data, and a plurality of text data (STT) converted from the plurality of audio frames. You can obtain probability values for each of multiple general emotional states using multiple models based on ensemble learning.

According to the embodiment, the second analysis result acquisition unit 21b generates a first model learned based on the video frame for learning, generates a second model learned based on the audio frame for learning, and generates a second model learned based on text data for learning. A learned third model can be created, and the probability value of each of a plurality of general emotional states can be obtained by referring to the first weight of the first model, the second weight of the second model, and the third weight of the third model. .

Specifically, a first classification value is obtained through a first model based on at least one video frame among a plurality of video frames, and a second classification value is obtained through a second model based on at least one audio frame among the plurality of video frames. A value is obtained, a third classification value is obtained through a third model based on at least one text data among a plurality of text data, a first weight is applied to the first classification value, and the second classification value is applied to the second classification value. By applying a weight, and applying the third weight to the third classification value, the probability value of each of a plurality of general emotional states based on each of the video frame, the audio frame, and the text data can be obtained. Then, the corresponding process can be performed on the plurality of video frames, the plurality of audio frames, and the plurality of text data to obtain overall probability values for each of the plurality of general emotional states.

Depending on the embodiment, the first model may be a model based on MobileNet.

Depending on the embodiment, the second model may be a model based on SVM.

Depending on the embodiment, the third model may be a model based on Bert.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a first classification value representing the general emotional state of the client through a first model based on the client's image frame.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a first classification value representing the general emotional state of the client through the MobileNet model based on the video frame.

Depending on the embodiment, the first classification value may include a first preliminary probability value for each of a plurality of general emotional states.

The second analysis result acquisition unit 21b may obtain a second classification value representing the client's general emotional state through a second model based on the client's voice frame.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a second classification value representing the general emotional state of the client through a second model based on the SVM algorithm based on the voice frame.

Depending on the embodiment, the second classification value may include a second preliminary probability value for each of a plurality of general emotional states.

The second analysis result acquisition unit 21b may obtain a third classification value representing the general emotional state of the client through a third model based on text data.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain the third classification value through the BERT model based on text data.

Depending on the embodiment, the third classification value may include a third preliminary probability value for each of a plurality of general emotional states.

Specifically, referring to FIG. 13, the second analysis result acquisition unit 21b generates a plurality of first results obtained by applying a first weight to the first preliminary probability value of each of the plurality of general emotional states, respectively, and the plurality of general emotional states. A plurality of second results are obtained by applying a second weight to each of the second preliminary probability values of and a plurality of third results are obtained by applying a third weight to each of the third preliminary probability values of each of the plurality of general emotional states, and The probability value of each of a plurality of general emotional states can be obtained by adding up the first result, the second result, and the third result for each general emotional state.

For example, the first preliminary probability values of joy and anger are [0.7, 0.3], respectively, the second preliminary probability values of joy and anger are [0.8, 0.2], respectively, and the first preliminary probability values of joy are [0.7, 0.3], respectively. The third preliminary probability value and the third preliminary probability value of anger are respectively [0.2, 0.8], the weight (a) of the first model is 0.3, the weight (b) of the second model is 0.5, and the weight (c) of the third model If is 0.2, the summed probability value for joy is 0.7*0.3 + 0.8*0.5 + 0.2*0.2 = 0.65, and the summed probability value for anger is 0.3*0.3 + 0.2*0.5 + 0.8*0.2 = 0.35. You can.

Hereinafter, the learning process of each model will be described with reference to FIG. 13.

Depending on the embodiment, the plurality of substituted emotional states may be a combination of emotional states defined as higher-level concepts for each of the plurality of general emotional states. That is, each of the plurality of substituted emotional states may be defined so that one or more general emotional states defined as sub-concepts of the substituted emotional state are mapped. For example, if general emotional states can be expressed in 7 types, substitute emotional states can be expressed in 4 types (positive: joy, negative: anger/contempt/surprise/fear, neutral: calm, other: sadness). .

Depending on the embodiment, the probability value of each of the plurality of alternate emotional states may be obtained by adding the probability values of one or more general emotional states corresponding to each of the plurality of alternate emotional states. That is, the second analysis result acquisition unit 21b calculates the probability value of each of a plurality of general emotional states, then adds up the probability values of the general emotional states classified into the same group among each of the plurality of general emotional states, and calculates the probability values of the general emotional states classified into the same group as a replacement emotional state. It can be calculated as a probability value.

For example, at a specific time (or time), the probability value of joy is 20%, the probability value of anger is 10%, the probability value of contempt is 15%, the probability value of surprise is 5%, the probability value of fear is 20%, and the probability value of calmness is 20%. If the probability value of sadness is 10% and 20%, the probability value of positivity is 20%, the probability value of negativity is 50%, the probability value of neutrality is 10%, and the other probability values can be calculated as 20%.

Depending on the embodiment, the probability value of each of the plurality of alternate emotional states may be obtained by averaging the probability values of the alternate emotional states acquired during a predetermined time interval.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate and transmit a graph of the probability values of each of a plurality of alternative emotional states for each time period, as shown in FIG. 6 .

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire the heart rate and stress index based on the second image data.

Specifically, the second analysis result acquisition unit 21b calculates an rPPG (Remote photoplethysmography) signal using each image frame constituting the second image data, and extracts the heart rate using the calculated rPPG signal. can do.

According to the embodiment, a face is detected from an image frame (ROI) using the Haar cascade algorithm, and after noise removal and frequency conversion, the change in skin color is converted to RGB value channels, and then the average pixel in each channel is Changes in values are measured, and heart rate can be calculated using the values extracted from each channel.

In addition, the stress index can be calculated using the heart rate variability index based on the heart rate extracted from the calculated rPPG signal and the positive correlation between autonomic nervous system activation due to stress.

Depending on the embodiment, the heart rate variability index includes standard deviation of all RR interval (SDNN) value, square root of the mean squared differences of successive normal sinus intervals (RMSSD) value, SDNN/RMSSD ratio, normal to normal interval (NN), NN50 (the number of instances where the variation of consecutive NN intervals is greater than 50 ms), pNN50 (ratio of NN50 among all NN intervals), LF (low frequency) value, HF (high frequency) value, or LF/HF value, etc. It can be included.

For example, the wider the interval in the SDNN value, the greater the degree of change. In general, the greater the degree of change, the healthier it is. This can be used to calculate physical fatigue, and a high RMSSD value indicates that the heart is more stable. Using this, you can find out the activity of the parasympathetic nerve. The LF value reflects the activity of the sympathetic nervous system, and is influenced by mental stress and fatigue. In particular, it can evaluate the degree of acute stress, and is correlated with depression or anger. The HF value reflects the activity of the parasympathetic nervous system, and respiratory It is closely related to activity and is correlated with long-term stress, anxiety, or fear. The LF/HF value reflects the overall balance of the autonomic nervous system, and generally, in healthy people, the LF value is higher than the HF value while awake. Therefore, the stress index can be calculated by using the positive correlation between the heart rate variability index extracted from the rPPG signal and the activation of the autonomic nervous system due to stress.

Depending on the embodiment, the average value is calculated using each heart rate variability index obtained for 5 minutes after the consultation begins, and this is set as a reference value. The heart rate variability index after 5 minutes can be calculated and analyzed by comparing it with the reference value.

However, this is just an example, and heart rate, heart rate variability, and stress index can be calculated using various known methods.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate and transmit a graph of the heart rate and stress index for each time period as shown in FIG. 7 . Depending on the embodiment, the analysis result acquisition unit 21b may display the heart rate and stress index for each time period on one graph. Depending on the embodiment, the analysis result acquisition unit 21b provides heart rate and stress index, the measurement value itself, and/or classification information (normal/caution/borderline, etc.) corresponding to the measurement value (normal stress/moderate stress/severe stress). etc.) can be created and transmitted.

Depending on the embodiment, the second analysis result acquisition unit 21b may perform analysis on the text data converted from the second voice data to obtain the frequency of each preset keyword.

Specifically, the second analysis result acquisition unit 21b performs analysis on the entire text data (via STT) into which each voice frame constituting the second voice data has been converted to obtain the frequency of each keyword preset by the counselor. can do. This can be achieved through analysis that maps text data to preset keywords.

Depending on the embodiment, preset keywords may be implemented to be changed and set in real time by a counselor.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate and transmit a graph showing the frequency of each keyword, as shown in FIG. 8.

Depending on the embodiment, the second analysis result acquisition unit 21b may arrange the frequencies of each keyword in descending or ascending order and provide them.

Depending on the embodiment, the second analysis result acquisition unit 21b may provide the frequency of each keyword expressed in the size of a circle. For example, keywords uttered with a high frequency can be expressed as relatively large circles to create a graph.

Depending on the embodiment, the second analysis result acquisition unit 21b receives a set keyword from the counselor terminal 30, or the emotion analysis result providing device 20 receives user input through a separate user interface unit (not shown). You can directly enter the setting keyword through.

According to the embodiment, the second analysis result acquisition unit 21b performs Latent Dirichlet Allocation (LDA) topic modeling on the text data into which the second voice data has been converted to obtain information on each of the keywords configured for each topic. Frequencies can be obtained.

Specifically, the second analysis result acquisition unit 21b performs latent Dirichlet allocation (LDA) topic modeling on the entire text data (via STT) into which each voice frame constituting the second voice data has been converted. By performing this, the frequency count for each of the keywords configured for each topic can be obtained.

According to the embodiment, the second analysis result acquisition unit 21b preprocesses each of the plurality of documents constituting the entire text data, extracts at least one topic from the plurality of preprocessed documents, and configures topics for each topic. You can calculate the word frequency for each keyword.

Depending on the embodiment, the preprocessing process may include at least one of special character removal (or number removal), morpheme analysis, stop word removal, and synonym (or similar word) processing.

Depending on the embodiment, the second analysis result acquisition unit 21b may extract at least one topic using a plurality of pre-processed documents and match each of the plurality of pre-processed documents with each of the extracted at least one topic. there is. Specifically, keywords are extracted from the plurality of preprocessed documents, and frequency analysis of the extracted keywords is performed. Additionally, the second analysis result acquisition unit 21b may generate a term-document matrix for topic modeling. The second analysis result acquisition unit 21b can extract at least one topic (subject) through keywords constituting the document through the LDA (Latent Dirichlet Allocation) technique and classify each document by topic. At this time, one document may be included in multiple topics. In other words, a document and a topic can have a 1:N correspondence relationship (N is a natural number greater than or equal to 1). The LDA model (or LDA model) is a model that predicts what topic the document covers by analyzing the words in a given document based on the distribution of words for each topic calculated in advance. The standard for dividing topics is one of the cluster analysis techniques, which determines the number of K clusters through silhouette analysis and analyzes top keywords according to K topics.

Depending on the embodiment, the second analysis result acquisition unit 21b may graph the word frequencies for each keyword constituting the topic into a plurality of graphs, as shown in FIGS. 9A and 9B (enlarged graph of FIG. 9A). there is.

Depending on the embodiment, circles corresponding to the number of topics may be displayed on an arbitrary graph, and in particular, the graph may be graphed so that the size of the circles is proportional to the frequency of keywords corresponding to the topic. Additionally, the distance between circles can be calculated and graphed based on the similarity between topics. And, on another graph, it can be created to display keywords corresponding to the topic of each circle and frequency information for each keyword.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain voice parameters by analyzing the second voice data based on the second voice data.

According to the present invention, various known techniques can be applied to obtain each voice parameter for each voice frame using voice frames for the second voice data, and description thereof will be omitted.

Depending on the embodiment, voice parameters may include indicators indicating waveform, pitch, pause section, speech rate, and/or hesitation section.

Pitch can be determined based on the frequency and period of the voice.

A pause section can be defined as a section in which no waveform occurs or the waveform is interrupted.

Depending on the embodiment, if the pose section is longer than a predetermined time (e.g., > 2 seconds), the second analysis result acquisition unit (21b) determines this as a section in which the client hesitates and uses this to provide an indicator (e.g., hesitation) indicating the hesitation section. number of times, average time of hesitation) can be generated.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate a graph of voice parameters for each time period as shown in FIGS. 10A and 10B.

For example, Figure 10a shows the pitch change over time, and Figure 10b shows the pose change over time.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain gaze position information and facial movement information by analyzing the second image data based on the second image data.

According to the embodiment, the second analysis result acquisition unit 21b recognizes the client's face from each image frame constituting the second image data, detects feature points of the recognized face, and bases the feature points on the face. Gaze position information can be obtained by specifying a region of interest, detecting the position of the pupil center in the designated region of interest, and calculating the degree of change over time in the detected pupil center position based on the change in Euclidean distance. .

Recognizing the client's face from the video frame and detecting the characteristic points of the recognized face can be performed by applying a known algorithm.

According to the embodiment, the second analysis result acquisition unit 21b designates each region composed of feature points corresponding to the client's left eye and right eye, respectively, as a region of interest among the facial feature points, and each of the left eye region and the right eye region The position of the pupil center can be detected in the area.

Depending on the embodiment, the second analysis result acquisition unit 21b may designate a region of interest by comparing a pre-generated feature point template (including the region of interest) with the recognized facial feature points.

For example, referring to FIG. 12, the second analysis result acquisition unit 21b includes first end points based on the horizontal axis of the left eye area (and each of the right eye areas) and second end points based on the vertical axis. Relative coordinates (center coordinates between first end points, center coordinates between second end points) are calculated based on each coordinate, and the relative coordinates can be detected as the pupil center position.

Then, the Euclidean distance from the reference coordinates of the relative coordinates calculated according to the change in the client's gaze over time is calculated, and the average value of the Euclidean distances calculated from each of the left and right eye areas is used to obtain gaze position information. can be obtained.

Depending on the embodiment, the reference coordinates may be calculated as the average value of the Euclidean distance measured in real time for a predetermined period of time (e.g., about 1 minute after the consultation begins).

Depending on the embodiment, as shown in FIG. 11, the second analysis result acquisition unit 21b graphs the average value of the Euclidean distance (displacement per standard time, movement per second) according to time change, or Euclidean distance according to time change. The accumulated value of the average value of the dian distance (accumulated displacement per standard time, accumulated movement) can be created and transmitted as a graph.

According to the embodiment, the second analysis result acquisition unit 21b recognizes the client's face from each image frame constituting the second image data, detects the feature point of the recognized face, and determines the position of the detected feature point. Facial movement information can be obtained by calculating the degree of change over time based on the change in Euclidean distance.

Recognizing the client's face from the video frame and detecting the characteristic points of the recognized face can be performed by applying a known algorithm.

Depending on the embodiment, the second analysis result acquisition unit 21b may use the tip of the nose of the face to specifically detect the movement position of the face.

Depending on the embodiment, the second analysis result acquisition unit 21b may detect the tip of the nose as the target feature point by comparing a pre-generated feature point template (facial feature point including the nose tip) with the feature point of the recognized face.

Depending on the embodiment, the coordinate value of the tip of the nose may use an absolute coordinate value based on the entire image frame.

Depending on the embodiment, absolute coordinate values can be calculated by referring to the pixel units that make up the image frame.

In addition, facial movement location information can be obtained by calculating the Euclidean distance between the nose tip coordinates at the current time and the nose tip coordinates at the previous time, calculated according to changes in the client's facial movement over time.

Depending on the embodiment, as shown in FIG. 11, the second analysis result acquisition unit 21b graphs the average value of the Euclidean distance (displacement per standard time, movement per second) according to time change, or Euclidean distance according to time change. The accumulated value of the average value of the dian distance (accumulated displacement per standard time, accumulated movement) can be created and transmitted as a graph.

Depending on the embodiment, the second analysis result acquisition unit 21b may create and transmit graphs of gaze position information and facial movement information for each time period, as shown in FIG. 11 .

The transmission unit 21c may transmit information including all of the plurality of second analysis results to the counselor terminal 30 (s24).

Depending on the embodiment, the information including all of the plurality of second analysis results may include all of the plurality of second analysis results, video data received in real time from the client terminal 10, and voice data of the client separated from the video data received in real time. It may include content of the converted text data and the text data converted from the counselor's voice data.

5 to 11 illustrate screens of the user interface unit 33 according to the operation of the second analysis result providing unit 32 of the counselor terminal 30.

The second analysis result providing unit 32 of the counselor terminal 30 provides information including the second analysis result selected by the user among all the plurality of second analysis results received from the emotional analysis result providing device 20 and provides real-time information to the counselor. The recorded counseling notes can be provided to users such as managers or counselors by outputting them through the screen of one user interface unit 33.

Depending on the embodiment, the information including the second analysis result selected by the user among all of the plurality of second analysis results may include the second analysis result selected by the user among all of the plurality of second analysis results, and auxiliary information related to the selected second analysis result. As a result of the analysis, it may include video data in which the client was filmed, text data converted from the client's voice data separated from the video data in which the client was filmed, and content corresponding to the text data converted to the counselor's voice data.

Depending on the embodiment, the second analysis result providing unit 32 may receive a user input for outputting a second analysis result selected from among a plurality of second analysis results through the screen of the user interface unit 33.

Depending on the embodiment, the screen of the user interface unit 33 includes a plurality of menus (M) representing a plurality of second analysis results, and the second analysis result providing unit 32 provides a menu selected by the user. The corresponding second analysis result (①) can be output through the screen of the user interface unit 33 and provided to the user.

For example, Figure 5 illustrates a screen output by a user's selection input for a menu for outputting a plurality of general emotional states as second analysis results, and Figure 6 illustrates a plurality of substituted emotional states as second analysis results. FIG. 7 illustrates a screen output by a user's selection input for a menu for output, and FIG. 7 illustrates a screen output by a user's selection input on a menu for outputting heart rate and stress measurement values as a second analysis result. , Figure 8 illustrates a screen output by the user's selection input for a menu for outputting the frequency of each keyword as a second analysis result, and Figures 9a and 9b show the second analysis of the frequency of each of the keywords organized by topic. A screen output by a user's selection input for a menu for outputting a result is exemplified, and FIGS. 10A and 10B illustrate a screen output by a user's selection input for a menu for outputting a voice parameter as a second analysis result. 11 illustrates a screen output by a user's selection input for a menu for outputting gaze position information and/or facial movement information as a second analysis result.

Depending on the embodiment, each content may be output in separate areas on the screen of the user interface unit 33.

For example, as an example of the screen of the user interface unit 33 in FIGS. 5 to 11, the second analysis result (①), the auxiliary analysis result (②) related to the second analysis result, and video data captured by the patient (⑤), text data converted from the client's voice data, text data converted from the counselor's voice data (④), and counseling notes recorded by the counselor in real time (③).

In the screen of the user interface unit 33 of FIGS. 5 to 7 and FIGS. 10A, 10B, and 11, depending on the embodiment, the second analysis results for each time period may be provided in the form of a graph.

Specifically, the one user interface screen 33 is divided into a plurality of areas, and the second analysis result providing unit 32 provides the second analysis result selected by the user in any one of the plurality of areas. You can output a graph showing the analysis results by time period.

In the screen of the user interface unit 33 of FIGS. 5 to 11, depending on the embodiment, text data converted from the client's voice data and text data ④ converted from the counselor's voice data can be edited by the user or saved as a file. It may be provided in a form and provided to the user.

In the screen of the user interface unit 33 of FIGS. 5 to 11, depending on the embodiment, client information is listed and client information selected by the user can be searched.

Depending on the embodiment, in each area on the screen of each user interface unit 33 of FIGS. 5 to 11, only content corresponding to the second analysis result (①) and the auxiliary analysis result (②) related to the first analysis result is displayed. This is implemented differently, and the remaining content (③④⑤) can be implemented the same.

Referring to FIG. 5 , a graph representing the probability values of each of a plurality of general emotional states by time period may be provided in an area showing the second analysis result (①). Depending on the embodiment, the auxiliary analysis result (②) related to the second analysis result may include item information for each of a plurality of general emotional states, and depending on the embodiment, only the probability value for the general emotional state item selected by the user. It can also be set to be output on this graph.

Depending on the embodiment, among the probability values of a plurality of general emotional states appearing at the same time on the graph, the general emotional state with the largest probability value may be determined to be the representative emotional state at that time.

According to the present invention, through the second analysis result of FIG. 5, the user can overall check the change in the client's emotions by time zone (according to the passage of time) within the entire counseling section. At this time, the user can check the probability value of each of the plurality of general emotional states collectively, or individually the probability value of the general emotional state selected among them. Accordingly, the user can understand the context of the client's emotional changes as a whole and seek a direction for future counseling.

Referring to FIG. 6, a graph showing the probability value of each of a plurality of substituted emotional states by time period may be provided in the area showing the second analysis result (①). Depending on the embodiment, the auxiliary analysis result (②) related to the second analysis result may include frequency and/or probability information corresponding to each of positive, negative, neutral, and other, and tabular information thereof. Depending on the embodiment, it may be set to output only information corresponding to the replacement emotional state selected by the user.

According to the present invention, through the second analysis result of FIG. 6, the user can overall check the change in the client's emotions by time zone (according to the passage of time) within the entire counseling section. At this time, the user can collectively check the probability value of each of a plurality of substituted emotional states, or individually check the probability value of the selected substituted emotional state among them. Accordingly, the user can understand the context of the client's emotional changes as a whole and seek a direction for future counseling.

In particular, according to the present invention, through the second analysis result of FIG. 5, the user can confirm more detailed classified (categorized) emotional state information, while through the second analysis result of FIG. 6, the user can Since emotional state information classified (categorized) can be confirmed in a larger range, emotional state information can be confirmed in a variety of ways according to the user's needs and choices.

Referring to FIG. 7, a graph showing both heart rate and stress index for each time period may be provided in the area showing the second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include information such as average heart rate and average stress level. Depending on the embodiment, only information selected by the user among heart rate and stress level may be set to be output.

According to the present invention, through the second analysis result in FIG. 7, the user can overall check the client's stress level by time zone (according to the passage of time) within the entire counseling section. Accordingly, the user can seek a direction for future counseling by understanding the context, including the client's stress level or changes, as a whole.

Referring to FIG. 8, a graph showing the frequency of each keyword may be provided in an area showing the second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include keyword list information preset by the counselor.

According to the present invention, it is possible to check how many key keywords considered important by psychological counseling aspects and counselors appear among the words uttered by the client during counseling. For example, key words such as suicide and depression are keywords that clearly reveal the client's psychology and situation, so it is very important to check how many of these keywords appear. In addition, not only the set words pre-registered by the administrator, but also real-time words can be registered and checked by the counselor conducting the consultation. Words that you want to exclude from analysis can be set to be excluded, so that not only the set words but also the words you have set can be excluded. When you want to analyze all words uttered by a client, you can analyze the entire language.

Referring to FIG. 9A, the frequency of each keyword organized by topic may be graphed in a plurality of forms and provided in an area showing the second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include list information indicating the frequency and ranking of each keyword organized by topic.

According to the present invention, through the second analysis result of FIG. 9A, it is possible to confirm the topic area in which the client is mainly interested in the content of the counselor's counseling, and to examine the relationships with other topics together. There will be. Accordingly, the counselor can seek a direction for future counseling by gaining an overall understanding of the client's main areas of interest.

Referring to FIGS. 10A and 10B, voice parameters for each time period may be graphed and provided in an area showing the second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include information such as pitch average value, pose average value, average number of utterances per predetermined time, number of hesitations, and average hesitation value. Depending on the embodiment, only information selected by the user may be set to be output.

According to the present invention, the counselor can check when the client raises his voice, when he speaks quietly, when he speaks slowly, when he speaks quickly, etc. through a graph, and as a result, the counselor can monitor the client's emotions according to changes in the client's emotions and changes in the content of speech. By checking the changes in the voice waveform as a whole, it is possible to determine subtle changes in the client's emotions over time.

Referring to FIG. 11, gaze position information and facial movement information for each time period may be graphed and provided in an area showing the second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include an average gaze position change value, an accumulated average gaze position change value, an average facial movement change value, and an accumulated average facial movement change value. Depending on the embodiment, only information selected by the user may be set to be output.

According to the present invention, through the analysis results of FIG. 11, eye movements or facial movements can be evidence that the client's emotional state is revealed through the body, so through the results, the counselor confirms the client's unconscious behavior, and through this, It is possible to accurately determine the client's emotional state, and by finding the point in time when significant displacement occurred on the graph and checking the counseling content for that time period, the client's feelings can be identified more accurately.

According to the present invention, the first analysis result is obtained through real-time analysis through the first analysis result acquisition unit 21a, but the second analysis result is obtained through analysis after a predetermined time has elapsed through the second analysis result acquisition unit 21b. By doing so, real-time counseling details with the same client can be provided in real time through real-time analysis, and by allowing the entire counseling history to be provided through integrated analysis after the counseling is completed, various types of counseling details can be provided for one counseling history. By being able to receive analysis results, it becomes possible to manage more diverse and systematic consultation details.

In addition, the user interface screen when a consultation is in progress and the user interface screen that provides analysis results after the consultation is completed are implemented differently, making it easier to manage and confirm the counselor's consultation details. For example, when a consultation is in progress, real-time analysis results are provided and the entire analysis results are provided through a single screen of the user interface unit 33, so that the counselor can check, analyze, and/or make a judgment on the consultation details in real time. It can be done more quickly and easily, and when the consultation is completed, one (selected) analysis result of the entire accumulated consultation history is provided through the screen of one user interface unit 33, so that the entire consultation history can be viewed by the user. can be confirmed in an integrated manner.

FIG. 13 is a diagram referenced to explain a learning process for generating an ensemble model according to an embodiment.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate a plurality of models in advance and store them in the storage unit 22.

Depending on the embodiment, a plurality of models may be created as ensemble-based models that apply each of the learning video frames, learning audio frames, and learning text data to learning.

Depending on the embodiment, multiple models may be created through an ensemble of machine learning and deep learning.

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire the first model learned based on the training image frame and store it in the memory 22.

Depending on the embodiment, the first model may be a MobileNet model based on a CNN (Convolutional Neural Network) algorithm, but may include a Deep Neural Network (DNN), DeepConvolution Neural Network (DCNN), Recurrent Neural Network (RNN), or K-Nearest (KNN). It can also be created through learning based on algorithms such as Neighbor, SVM (Support Vector Machine), Random Forest, and Decision Tree. The MobileNet architecture adopts depthwise separable convolutions (a form of factorized convolutions) to minimize processing operations (i.e., floating point operations, multiplications and/or additions, etc.). Depthwise separable convolutions are a division of standard convolutions into depthwise convolutions and 1 x 1 convolutions (also called "pointwise convolutions"), with the aim of making processing faster by reducing or minimizing the number of operations required. factorizes it (e.g., spits out its functions). Depthwise convolution applies a single filter to each input channel. Pointwise convolution then applies a 1 x 1 convolution to combine the outputs of the depthwise convolution, filtering and combining functions/operations, rather than the single filtering and combining operations performed by standard convolutions. Separate into two stages. Accordingly, structures in the MobileNet architecture may include two convolutional layers, one depthwise convolutional layer, and one pointwise convolutional layer per structure to define or illustrate a “layer group.”

Depending on the embodiment, the second analysis result acquisition unit 21b may generate the first model by classifying the emotions of the training video frames through neural network learning through CNN.

The second analysis result acquisition unit 21b may acquire the second model learned based on the learning voice frame and store it in the storage unit 22.

Depending on the embodiment, the second model may be a model based on the Support Vector Machine (SVM) algorithm, but may be based on CNN, Deep Neural Network (DNN), DeepConvolution Neural Network (DCNN), Recurrent Neural Network (RNN), or KNN (K- It can also be generated by learning based on algorithms such as Nearest Neighbor, Random Forest, and Decision Tree. The SVM (Support Vector Machine) algorithm is used for classification and regression analysis, and a set of data belonging to one of two categories is given. When lost, the SVM algorithm creates a non-probabilistic binary linear classification model that determines which category the new data belongs to based on the given data set.

According to the embodiment, the second analysis result acquisition unit 21b extracts a voice feature vector based on frequency analysis of the voice frame for learning, learns the extracted voice feature vector through an SVM (Support Vector Machine) algorithm, and learns. In the process, a second model can be created by allowing voice feature vectors to be classified by emotion.

The second analysis result acquisition unit 21b may acquire the third model 22c learned based on the learning text data and store it in the storage unit 22.

Depending on the embodiment, the third model may be a BERT (Bidirectional Encoder Representations from Transformers) model, but may be a Deep Neural Network (DNN), DeepConvolution Neural Network (DCNN), CNN, Recurrent Neural Network (RNN), or K-Nearest (KNN). Neighbor), SVM (Support Vector Machine), Random Forest, and Decision Tree. BERT is an artificial intelligence model based on the Transformer architecture of the encoder-decoder structure, and is an artificial intelligence model based on the input It is characterized by stacking multiple transformer layers for deep representation and applying a masking process to the Masking Language Model, which is a token sequence. The BERT model shows high accuracy even with a small amount of data by going through a fine tuning process. It is an attention-based model that improves performance by drawing attention to specific vectors, and its performance does not deteriorate even as the sentence becomes longer, so it has the advantage of maintaining accuracy even in long sentences. There is.

According to the embodiment, the second analysis result acquisition unit 21b classifies the emotions for the learning text data through neural network learning to obtain a context-based embedding value based on the learning text data, thereby generating a third model. You can.

The embodiments described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination.

Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and usable by those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to carry out processing according to the invention and vice versa.

Aspects herein may take the form of a computer program product embodied entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or on one or more computer-readable media having computer-readable program code embodied thereon. .

The features, structures, effects, etc. described in the embodiments above are included in one embodiment of the present invention and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified and implemented in other embodiments by a person with ordinary knowledge in the field to which the embodiments belong. Therefore, contents related to such combinations and modifications should be construed as being included in the scope of the present invention.

In addition, although the above description has been made focusing on the examples, this is only an example and does not limit the present invention, and those skilled in the art will understand the above examples without departing from the essential characteristics of the present embodiment. You will be able to see that various modifications and applications are possible. For example, each component specifically shown in the examples can be modified and implemented. And these variations and differences in application should be construed as being included in the scope of the present invention as defined in the appended claims.

Claims (11)

A first analysis result acquisition unit that acquires a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of first voice data and first video data extracted from video data received in real time from the client terminal; and
Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated over a predetermined time, the real-time received video data analyzes the client's emotions after the predetermined time has elapsed and generates a plurality of second It includes a second analysis result acquisition unit that obtains the analysis results,
By integrating the real-time analysis by the first analysis result acquisition unit and the analysis after the predetermined time by the second analysis result acquisition unit, various types of analysis including the plurality of first analysis results and the plurality of second analysis results are obtained. Provides analysis results in the form of
The plurality of first analysis results are:
Contains at least two of the following: probability values for each of a plurality of general emotional states, representative emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword organized by topic, voice parameters, gaze position information, and facial movement information; ,
The results of the plurality of second analyzes are:
Probability values for each of a plurality of general emotional states, probability values for each of a plurality of substituted emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword organized by topic, voice parameters, gaze position information, and facial movement information. containing two or more,
A device that provides emotional analysis results.
delete delete According to clause 1,
The plurality of substituted emotional states are a combination of emotional states defined by a higher concept of each of the plurality of general emotional states,
A device that provides emotional analysis results.
According to clause 1,
The second analysis result acquisition unit,
Obtaining probability values for each of a plurality of general emotional states using a plurality of models based on ensemble learning based on the second image data, the second voice data, and text data converted from the second voice data,
A device that provides emotional analysis results.
According to clause 1,
The second analysis result acquisition unit,
Performing Latent Dirichlet Allocation (LDA) topic modeling on the text data into which the second voice data has been converted to obtain frequencies for each of the keywords configured for each topic.
A device that provides emotional analysis results.
According to clause 1,
The second analysis result acquisition unit,
Recognize the client's face based on the second image data, detect feature points of the recognized face, detect the pupil center position in a designated area of interest based on the feature points of the face, and detect the pupil center position of the detected pupil center position. Calculating the gaze position information, which is the degree of change over time, based on the change in Euclidean distance,
A device that provides emotional analysis results.
In an emotional analysis result providing system including a client terminal, a counselor terminal, and an emotional analysis result providing device,
The emotional analysis result providing device,
A first analysis result acquisition unit that acquires a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of the first voice data and the first video data extracted from the video data received in real time from the client terminal. ;
Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated over a predetermined time, the real-time received video data analyzes the client's emotions after the predetermined time has elapsed and generates a plurality of second a second analysis result acquisition unit that acquires analysis results; and
A transmission unit configured to transmit information including all of the plurality of first analysis results and information including all of the plurality of second analysis results to the counselor terminal,
The counselor terminal is,
a first analysis result providing unit that provides information including all of the plurality of first analysis results received from the emotional analysis result providing device through one user interface screen; and
A second analysis result providing unit that provides information including a second analysis result selected by the user among all the plurality of second analysis results received from the emotional analysis result providing device through the one user interface screen; and ,
Various forms including the plurality of first analysis results and the plurality of second analysis results are obtained by integrating the real-time analysis by the first analysis result acquisition unit and the analysis after the predetermined time has elapsed by the second analysis result acquisition unit. Provides analysis results of
The plurality of first analysis results are:
Contains at least two of the following: probability values for each of a plurality of general emotional states, representative emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword organized by topic, voice parameters, gaze position information, and facial movement information; ,
The results of the plurality of second analyzes are:
Probability values for each of a plurality of general emotional states, probability values for each of a plurality of substituted emotional states, heart rate and stress index, frequency for each keyword, frequency for each keyword organized by topic, voice parameters, gaze position information, and facial movement information. containing two or more,
Emotion analysis result provision system.
According to clause 8,
The one user interface screen includes a plurality of menus representing the plurality of second analysis results,
The second analysis result providing unit,
Providing information including the second analysis result corresponding to the menu selected by the user among the plurality of menus through the one user interface screen,
Emotion analysis result provision system.
delete delete