KR20230114195A - Emotion analysis result providing device and emotion analysis result providing system - Google Patents

Abstract

An emotion analysis result providing apparatus according to an embodiment analyzes a client's emotions in real time based on at least one of first audio data and first image data extracted from video data received in real time from a client terminal, and provides a plurality of first analysis results. A first analysis result obtaining unit for acquiring; and
Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated for a predetermined time, the client's emotion is analyzed after the predetermined time has elapsed, and the plurality of second video data received in real time is analyzed. It may include; a second analysis result acquisition unit for acquiring analysis results.

Description

Emotion analysis result providing device and emotion analysis result providing system

The present invention relates to an emotion analysis result providing device and an emotion analysis result providing system. The present invention relates to an emotion analysis result providing apparatus and an emotion analysis result providing system for analyzing a client's emotion from various angles and providing an integrated result by analyzing a client's emotion.

Psychological counseling is based on empathic understanding, unconditional positive respect, and sincerity in the relationship with the client who has experienced psychological problems by counselors with specialized knowledge. , reality therapy, exchange analysis, family therapy, etc., as a treatment method that helps them solve their problems. It guides clients to explore human thoughts, emotions, behaviors, and interpersonal relationships so that they can understand and change their various problems. say help

This kind of psychological counseling is mainly done in the traditional counseling method, which was conducted face-to-face in a given counseling room. Due to burden or reluctance, psychotherapy cannot be continued and is sometimes stopped.

Therefore, as an alternative to face-to-face counseling, an online psychotherapy program that provides psychological counseling and treatment online is being proposed.

KR 2022-0005945 A

The present invention analyzes the client's emotions in real time based on the video data during the online psychotherapy program, and also analyzes the client's emotions based on the accumulated video data, resulting in an integrated result of analyzing the client's emotions from various angles Its purpose is to provide

An emotion analysis result providing apparatus according to an embodiment analyzes a client's emotions in real time based on at least one of first audio data and first image data extracted from video data received in real time from a client terminal, and provides a plurality of first analysis results. A first analysis result obtaining unit for acquiring; and

Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated for a predetermined time, the client's emotion is analyzed after the predetermined time has elapsed, and the plurality of second video data received in real time is analyzed. It may include; a second analysis result acquisition unit for acquiring analysis results.

The plurality of first analysis results,

It may include at least two or more of a probability value of each of a plurality of general emotional states, a representative emotional state, a heart rate and a stress index, a frequency for each keyword, a frequency for each of keywords configured for each topic, a voice parameter, gaze position information, and facial movement information. can

The plurality of second analysis results,

At least one of a probability value of each of a plurality of general emotional states, a probability value of each of a plurality of substitutional emotional states, a heart rate and a stress index, a frequency of each keyword, a frequency of each of keywords configured for each topic, a voice parameter, gaze position information, and face movement information. may contain two or more.

The plurality of substitutional emotional states may be a combination of emotional states defined as a superordinate concept of each of the plurality of general emotional states.

The second analysis result acquisition unit,

Probability values of each of a plurality of general emotional states may be obtained using a plurality of models based on ensemble learning based on the second video data, the second audio data, and the text data obtained by converting the second audio data.

The second analysis result acquisition unit,

A frequency count for each of the keywords configured for each topic may be obtained by performing Latent Dirichlet Allocation (LDA) topic modeling on the text data obtained by converting the second voice data.

The second analysis result acquisition unit,

Based on the second image data, the client's face is recognized, a feature point of the recognized face is detected, a pupil center position is detected in a designated region of interest based on the face feature point, and a pupil center position is determined based on the detected pupil center position. The gaze position information, which is a degree of change over time, may be calculated based on a change in Euclidean distance.

An emotion analysis result providing system according to an embodiment is an emotion analysis result providing system including a client terminal, a counselor terminal, and an emotion analysis result providing device,

The emotion analysis result providing device,

a first analysis result obtaining unit configured to obtain a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of first audio data and first video data extracted from video data received from the client terminal in real time;

Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated for a predetermined time, the client's emotion is analyzed after the predetermined time has elapsed, and the plurality of second video data received in real time is analyzed. a second analysis result acquisition unit for acquiring analysis results; and

A transmission unit configured to transmit information including all of the plurality of first analysis results and information including all of the plurality of second analysis results to the counselor terminal;

The counselor terminal,

a first analysis result providing unit that provides information including all of the plurality of first analysis results received from the emotion analysis result providing device through a single user interface screen; and

a second analysis result providing unit that provides information including a second analysis result selected by a user among all of the plurality of second analysis results received from the emotion analysis result providing device through the one user interface screen; can

The one user interface screen includes a plurality of menus representing the plurality of second analysis results,

The second analysis result providing unit,

Information including the second analysis result corresponding to a menu selected by the user from among the plurality of menus may be provided through the single user interface screen.

According to the present invention, the first analysis result is acquired through real-time analysis through the first analysis result acquisition unit, but the second analysis result is acquired through analysis after a predetermined time has elapsed through the second analysis result acquisition unit, thereby real-time communication with the same client. While providing real-time counseling details through real-time analysis, it is possible to receive various types of analysis results for one consultation detail by enabling the entire consultation details to be provided through integrated analysis after the consultation is finished. It becomes possible to manage more diversified and systematized consultation details.

In addition, a user interface screen during counseling is implemented differently from a user interface screen that provides analysis results after counseling is completed, so that the counselor can more easily manage and check the counseling details. For example, when counseling is in progress, real-time analysis results are provided and all analysis results are provided through a single user interface screen, so that the counselor can check, analyze, and/or judge real-time consultation details more quickly and easily When the consultation is finished, one (selected/individual) analysis result for the entire accumulated consultation history is provided through the screen of one user interface unit, so that the user can check the entire consultation history in an integrated manner. do.

1 is a system diagram of an emotion analysis result providing system 1 according to an embodiment.
2 and 3 are flowcharts illustrating the operation of the emotion analysis result providing device 20 according to an embodiment.
4 illustrates the screen of the user interface unit 33 according to the operation of the first analysis result providing unit 31 of the counselor terminal 30 .
5 to 11 illustrate screens of the user interface unit 33 according to the operation of the second analysis result providing unit 32 of the counselor terminal 30 .
12 to 13 are diagrams referenced to describe the operation of the emotion analysis result providing device 20 .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

For reference, in each flowchart of the present invention, each step is an example, and the present invention can be equally/similarly applied even when each order is differently changed and/or combined.

1 is a system diagram of an emotion analysis result providing system 1 according to an embodiment.

The client terminal 10 may be equipped with an application for conducting text/voice/video counseling and may be connected to the emotion analysis result providing device 20, through which text/voice/video counseling may be conducted with the counselor terminal 30. there is. The client terminal 10 includes a screen on which the counselor's appearance is output, a camera (not shown) capable of photographing the client's appearance and transmitting the image to the counselor terminal 30, and a microphone (not shown) that collects voice and allows conversation with the counselor ( (not shown) and a speaker (not shown) should be provided at least.

The client terminal 10 is a device for providing real-time emotion analysis results, including video data obtained by photographing the client's face using a camera (not shown) and video data including the client's voice data collected using a microphone (not shown). ) can be transmitted.

Depending on the embodiment, the client terminal 10 may output the emotion analysis result received from the emotion analysis result providing device 2 through a screen of a user interface unit (not shown) and provide the result to the client.

The counselor terminal 30 may be equipped with an application for conducting text/voice/video counseling and may be connected to the emotion analysis result providing device 20, through which text/voice/video counseling may be conducted with the client terminal 10. there is. The counselor terminal 30 is provided with at least a screen on which the client's appearance is output, a camera capable of photographing the counselor's image and transmitting the image to the client terminal 10, and a microphone and speaker for collecting voice and communicating with the client. It should be.

The counselor terminal 30 is a device for providing real-time emotion analysis results, including image data obtained by photographing the counselor's counseling face using a camera (not shown) and video data including the counselor's voice data collected using a microphone (not shown) ( 20) can be sent.

The counselor terminal 30 may output the emotion analysis result received from the emotion analysis result providing device 20 through the screen of the user interface unit 33 and provide the result to the counselor or manager (user).

The emotion analysis result providing device 20 mediates video counseling between the client terminal 10 and the counselor terminal 30, and transmits the result of emotional analysis of the client according to the video consultation to the client terminal 10 and/or the counselor terminal 30. ) can be transmitted.

2 and 3 are flowcharts illustrating the operation of the emotion analysis result providing device 20 according to an embodiment.

Depending on the embodiment, the first analysis result acquisition unit 21a may receive real-time video data from the client terminal 10 (s11).

The video data may include first video data and first audio data in which the client's face is photographed.

The first analysis result acquisition unit 21a may extract first audio data and first image data from video data (s12).

Depending on the embodiment, the first video data may be a single video frame received in real time, and the first audio data may be a single audio frame received in real time.

Depending on the embodiment, extracting the first image data and the first audio data from the video data, respectively, may be implemented through various well-known algorithms.

The first analysis result acquisition unit 21a may obtain a plurality of first analysis results by analyzing the emotion of the client in real time based on at least one of the first voice data and the first image data (s13).

Depending on the embodiment, the first analysis result acquisition unit 21a is the same as the second analysis result acquisition unit 21b in order to obtain an analysis result corresponding to the same category based on at least one of audio data and video data. A plurality of first analysis results may be obtained in real time through real-time analysis using a similar analysis method or various well-known algorithms. Accordingly, a detailed description of the analysis for obtaining the plurality of first analysis results will be omitted.

For example, in the analysis method for the second analysis result acquisition unit 21b to obtain a probability value of each of a plurality of general emotional states, the first analysis result acquisition unit 21a obtains a probability value of each of a plurality of general emotional states. Can be used the same/similarly.

According to an embodiment, the plurality of first analysis results may include a probability value of each of a plurality of general emotional states, a representative emotional state, a heart rate and stress level, a frequency for each keyword, a frequency for each of keywords configured for each topic, a voice parameter, and a gaze position. information, and at least two of facial motion information.

According to embodiments, the plurality of general emotional states may include at least two or more of joy, anger, contempt, surprise, fear, tranquility, and sadness at a corresponding time, but the present invention is not limited thereto.

According to an embodiment, the first analysis result acquisition unit 21a may obtain a probability value of each of a plurality of general emotional states based on the first image data and the first audio data, which is obtained by the second analysis result acquisition unit ( 21b) may be applied in the same/similar manner to obtaining a probability value of each of a plurality of general emotional states based on the second image data and the second audio data.

Depending on the embodiment, the representative emotional state may be a final emotional state that can be represented by the emotion of the client among a plurality of general emotional states of the corresponding time. For example, the first analysis result acquisition unit 21a may determine a general emotional state corresponding to the highest probability value among a plurality of probability values of each general emotional state as a representative emotional state at the corresponding time.

Depending on the embodiment, the heart rate may be obtained as an RPGG value, and the degree of stress may be obtained as a high/medium/low level.

Depending on the embodiment, the first analysis result acquisition unit 21a may obtain a heart rate and a stress index based on the first image data. This may be applied identically/similarly to obtaining the heart rate and the stress index based on the second image data of the second analysis result acquisition unit 21b.

Depending on the embodiment, the frequency of each keyword may be defined as the frequency of each keyword uttered by the client among preset keywords. Depending on the embodiment, a preset keyword may be changed and set by a user such as a real-time counselor or manager.

Depending on the embodiment, the first analysis result acquisition unit 21a may obtain a frequency count for each keyword by performing analysis on the text data obtained by converting the first voice data.

This may be applied identically/similarly to the case where the second analysis result obtaining unit 21b acquires the frequency count for each preset keyword by performing analysis on the text data into which the second voice data is converted.

According to an embodiment, the first analysis result acquisition unit 21a performs Latent Dirichlet Allocation (LDA) topic modeling on the text data from which the first voice data is converted, so that each of the keywords configured for each topic The frequency count may be obtained, which is obtained by performing Latent Dirichlet Allocation (LDA) topic modeling on the text data in which the second voice data is converted by the second analysis result acquisition unit 21b to generate keywords configured for each topic. It can be applied in the same / similar manner to obtaining the frequency count for each.

Depending on the embodiment, the first analysis result acquisition unit 21a may obtain a voice parameter by analyzing the first voice data, which is determined by the second analysis result acquisition unit 21b based on the second voice data. It can be applied in the same / similar manner to acquiring voice parameters by analyzing with .

Depending on the embodiment, the voice parameter may include indicators representing a waveform, pitch, pause section, speech speed, and/or hesitation section.

Depending on the embodiment, the first analysis result acquisition unit 21a may obtain gaze position information and face movement information by analyzing the first image data based on which the second analysis result acquisition unit 21b obtains 2 It can be applied in the same/similar way as obtaining gaze position information and face motion information by analyzing based on image data.

According to an embodiment, the first analysis result acquisition unit 21a generates a plurality of general emotional states in the form of a bar graph, applies the representative emotional state to an icon form and/or a radial graph, and generates a frequency count for each keyword set in advance. is generated by reflecting the size of the circle, and the frequency count for each of the keywords organized by topic can be created in the form of a table. Depending on the embodiment, in the case of a preset frequency for each keyword, it may be generated by being reflected in the size of a circle according to a level corresponding to the frequency (eg, 1 to 5 levels). For example, level 1 indicates that the client uttered 1 time and is represented by a relatively small circle, and level 5 indicates that the client uttered 5 or more times and is represented by a relatively large circle. .

However, this is an example, and it can be generated by variously applying a bar graph form, an icon form, a radial graph form, a circle form, etc. to each of the plurality of first analysis results.

The transmission unit 21c may transmit information including all of the plurality of first analysis results to the counselor terminal 30 (s14).

According to the embodiment, the information including all of the plurality of first analysis results may include all of the plurality of first analysis results, video data received in real time from the client terminal 10, and the client's first video data separated from the video data received in real time. Contents of text data obtained by converting voice data and text data obtained by converting counselor's voice data may be included.

4 illustrates the screen of the user interface unit 33 according to the operation of the first analysis result providing unit 31 of the counselor terminal 30 .

The first analysis result providing unit 31 of the counselor terminal 30 transmits information including all of the plurality of first analysis results received from the emotion analysis result providing device 20 and counseling notes recorded by the counselor in real time into one user interface unit. By outputting through the screen of (33), it can be provided to users such as administrators or counselors.

According to an embodiment, the information including all of the plurality of first analysis results may include all of the plurality of first analysis results, video data in which the client is photographed, and text data obtained by converting the client's voice data separated from the video data in which the client is photographed. and content corresponding to the text data converted from the counselor's voice data.

Depending on the embodiment, each content may be output to a separate area on the screen of the user interface unit 33 .

For example, FIG. 4 is an example of the screen of the user interface unit 33, in which a plurality of first analysis results (①), video data received in real time from the client terminal 10 (②), and audio data of the client are displayed. It may include converted text data and text data (④) in which the counselor's voice data is converted, and counseling notes (③) recorded by the counselor in real time.

(1-1) represents a plurality of general emotional states among the plurality of first analysis results (①), (1-2) represents a representative emotional state, (1-3) represents heart rate and stress index, (1- 4) represents the frequency count for each keyword, and (1-5) represents the frequency count for each of the keywords organized by topic.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate accumulated video data obtained by accumulating video data received in real time for a predetermined time (s21).

Depending on the embodiment, the second analysis result acquisition unit 21b obtains received video data within a section between the start of receiving video data from the client terminal 10 and the end of receiving video data from the client terminal 10. It can be obtained as cumulative video data.

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire, as cumulative video data, video data within a section between the time when the recording of the video data received from the client terminal 10 starts and the time when the recording ends. there is.

Depending on the embodiment, commands for defining the video data accumulated for the predetermined time as an analysis target are programmed in advance, and the second analysis result acquisition unit 21b may refer to the corresponding commands to define the analysis target. .

Depending on the embodiment, the predetermined time may correspond to a time interval from the start of counseling between the client and the counselor to the end.

Depending on the embodiment, the second analysis result acquisition unit 21b may extract the second audio data and the second image data from the cumulative video data (S22).

According to embodiments, the first image data may be a single image frame received in real time, while the second image data may include a plurality of image frames accumulated over a predetermined period of time.

Depending on the embodiment, the first voice data may be a single voice frame received in real time, while the second voice data may include a plurality of voice frames accumulated over a predetermined period of time.

Depending on the embodiment, extracting the second image data and the second audio data from the cumulative video data may be implemented using various well-known algorithms.

The second analysis result obtaining unit 21b may obtain a plurality of second analysis results by analyzing the emotion of the client after the elapse of the predetermined time based on at least one of the second audio data and the second image data ( s23).

Depending on the embodiment, a command for analyzing immediately after a predetermined time has elapsed (or after a certain time has elapsed) or at a time set by the user may be programmed in advance, and the second analysis result acquisition unit 21b ) can perform analysis by referring to the corresponding command.

Depending on the embodiment, the plurality of second analysis results may be a probability value of each of a plurality of general emotional states, a probability value of each of a plurality of substitutional emotional states, a heart rate and a stress index, a frequency count for each keyword, a frequency count for each of keywords configured for each topic, and voice. It may include at least two or more of parameters, gaze position information, and face motion information.

Depending on the embodiment, the second analysis result acquisition unit 21b may transmit at least some of the plurality of second analysis results as a graph.

Depending on embodiments, the plurality of general emotional states may include at least two or more of joy, anger, contempt, surprise, fear, tranquility, and sadness at a corresponding time (or time), but the present invention is not limited thereto.

Depending on the embodiment, the probability value of each of the plurality of general emotional states may be obtained by averaging probability values of the general emotional states obtained during a predetermined time period.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate and transmit a probability value of each of a plurality of general emotional states by time period as a graph, as shown in FIG. 5 .

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a probability value of each of a plurality of general emotional states based on the second image data and the second audio data.

According to an embodiment, the second analysis result acquisition unit 21b uses a plurality of general models based on ensemble learning based on the second image data, the second audio data, and the text data in which the second audio data is converted. A probability value of each emotional state may be obtained.

Specifically, the second analysis result acquisition unit 21b includes a plurality of image frames constituting the second image data, a plurality of audio frames constituting the second audio data, and a plurality of text data (STT) obtained by converting the plurality of audio frames. Based), it is possible to obtain a probability value of each of a plurality of general emotional states by using a plurality of models based on ensemble learning.

According to an embodiment, the second analysis result acquisition unit 21b generates a first model learned based on the video frame for learning, generates a second model learned based on the audio frame for learning, and based on text data for learning. to generate a learned third model, and obtain a probability value of each of a plurality of general emotional states by referring to the first weight of the first model, the second weight of the second model, and the third weight of the third model. .

Specifically, a first classification value is obtained through a first model based on at least one image frame among a plurality of image frames, and a second classification value is obtained through a second model based on at least one audio frame among a plurality of image frames. value, obtains a third classification value through a third model based on at least one text data among a plurality of text data, applies a first weight to the first classification value, and applies the second classification value to the second classification value. A weight is applied, and by applying the third weight to the third classification value, a probability value of each of a plurality of general emotional states based on each of the video frame, the audio frame, and the text data may be obtained. In addition, a corresponding process may be performed on the plurality of video frames, the plurality of audio frames, and the plurality of text data to obtain a probability value of each of a plurality of general emotional states as a whole.

Depending on the embodiment, the first model may be a model based on MobileNet.

Depending on embodiments, the second model may be a model based on SVM.

Depending on embodiments, the third model may be a model based on Bert.

Depending on the embodiment, the second analysis result obtaining unit 21b may obtain a first classification value representing a general emotional state of the client through a first model based on the client's video frame.

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire a first classification value indicating a general emotional state of the client through a MobileNet model based on the video frame.

According to embodiments, the first classification value may include a first preliminary probability value of each of a plurality of general emotional states.

The second analysis result obtaining unit 21b may obtain a second classification value representing a general emotional state of the client through a second model based on the client's voice frame.

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire a second classification value representing the general emotional state of the client through a second model based on an SVM algorithm based on the voice frame.

Depending on embodiments, the second classification value may include a second preliminary probability value for each of a plurality of general emotional states.

The second analysis result acquisition unit 21b may obtain a third classification value representing a general emotional state of the client through a third model based on the text data.

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire the third classification value through the BERT model based on the text data.

According to embodiments, the third classification value may include a third preliminary probability value for each of a plurality of general emotional states.

Specifically, referring to FIG. 13 , the second analysis result obtaining unit 21b obtains a plurality of first results obtained by applying a first weight to a first preliminary probability value of each of a plurality of general emotional states, respectively, and each of the plurality of general emotional states. A plurality of second results obtained by applying a second weight to a second preliminary probability value of , respectively, and a plurality of third results obtained by applying a third weight to a third preliminary probability value of each of the plurality of general emotional states, respectively; A probability value of each of a plurality of general emotional states may be obtained by summing the first result, the second result, and the third result for each general emotional state.

For example, the first preliminary probability value of joy and the first preliminary probability value of anger are [0.7, 0.3], respectively, the second preliminary probability value of joy and the second preliminary probability value of anger are [0.8, 0.2], respectively, and the value of joy is [0.8, 0.2]. The third preliminary probability value and the third preliminary probability value of anger are [0.2, 0.8], respectively, the weight (a) of the first model is 0.3, the weight (b) of the second model is 0.5, and the weight (c) of the third model is 0.2, the summed probability for joy is 0.7*0.3 + 0.8*0.5 + 0.2*0.2 = 0.65, and the summed probability for anger is 0.3*0.3 + 0.2*0.5 + 0.8*0.2 = 0.35. can

Hereinafter, the learning process of each model will be described later with reference to FIG. 13 .

According to embodiments, the plurality of substitutional emotional states may be a combination of emotional states defined as superordinate concepts of each of the plurality of general emotional states. That is, each of the plurality of substitutional emotional states may be defined such that one or a plurality of general emotional states defined as sub-concepts of the substitutional emotional state are mapped. For example, if the general emotional state can be expressed in 7 ways, the substituted emotional state can be expressed in 4 types (positive: joy, negative: anger/contempt/surprise/fear, neutral: tranquility, other: sadness). .

Depending on the embodiment, the probability value of each of the plurality of substitute emotional states may be obtained by summing probability values of one or a plurality of normal emotional states corresponding to each of the plurality of substitute emotional states. That is, the second analysis result acquisition unit 21b calculates probability values of each of a plurality of general emotional states, and then sums the probability values of general emotional states classified into the same group among the plurality of general emotional states, and then sums the probability values of each of the plurality of general emotional states to form a replacement emotional state. It can be calculated as a probability value.

For example, at a specific time (or time), the probability of joy is 20%, the probability of anger is 10%, the probability of contempt is 15%, the probability of surprise is 5%, the probability of fear is 20%, and the probability of tranquility is 15%. When the probability value of sadness is 10% and the probability value is 20%, the probability value of positive is 20%, the probability of negative is 50%, the probability of neutral is 10%, and other probability values are 20%.

Depending on the embodiment, the probability value of each of the plurality of alternate emotional states may be obtained by averaging the probability values of the alternate emotional states obtained during a predetermined time interval.

Depending on the embodiment, the second analysis result acquisition unit 21b may graph, generate, and transmit probability values of each of a plurality of substitutional emotional states for each time period, as shown in FIG. 6 .

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a heart rate and a stress index based on the second image data.

Specifically, the second analysis result acquisition unit 21b calculates an rPPG (Remote photoplethysmography) signal using each image frame constituting the second image data, and extracts a heart rate using the calculated rPPG signal. can do.

According to the embodiment, a face is detected (ROI) from an image frame using a Haar cascade algorithm, and after noise removal and frequency conversion, skin color changes are converted into RGB value channels, and average pixels in each channel The change in the value is measured, and the heart rate can be calculated using the value extracted from each channel.

In addition, the stress index can be calculated using the static correlation between the heart rate variability index based on the heart rate extracted from the calculated rPPG signal and the activation of the autonomic nervous system by stress.

According to an embodiment, the heart rate variability index is SDNN (standard deviation of all RR interval) value, RMSSD (square root of the mean squared differences of successive normal sinus intervals) value, SDNN / RMSSD ratio, NN (normal to normal interval), NN50 (the number of cases where the variation of consecutive NN intervals is greater than 50 ms), pNN50 (the ratio of NN50 among all NN intervals), LF (low frequency) value, HF (high frequency) value, or LF/HF value can include

For example, the wider the interval in the SDNN value, the greater the variance. In general, the greater the variance, the healthier the body can be used to calculate the fatigue of the body, and the higher the RMSSD value, the higher the stability of the heart. By using it, the activity of the parasympathetic nerve can be known. The LF value reflects the activity of the sympathetic nervous system, which is affected by mental stress and fatigue, can evaluate the degree of acute stress, and is correlated with depression or anger. The HF value reflects the activity of the parasympathetic nervous system. It is closely related to activity and correlates with long-term stress, anxiety, or fear. The LF/HF value reflects the overall balance level of the autonomic nervous system, and in general, the LF value is higher than the HF value in a healthy person while awake. Therefore, the stress index can be calculated by using the static correlation between the heart rate variability index extracted from the rPPG signal and the activation of the autonomic nervous system by stress.

According to an embodiment, an average value may be calculated using each heart rate variability index acquired for 5 minutes after counseling starts, set as a reference value, and a heart rate variability index after 5 minutes may be calculated and analyzed while comparing with the reference value.

However, this is an example, and heart rate, heart rate variability, and stress index can be calculated by various known methods.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate and transmit graphs of heart rate and stress index for each time period as shown in FIG. 7 . Depending on the embodiment, the 2 analysis result acquisition unit 21b may display the heart rate and stress index for each time zone on a single graph. Depending on the embodiment, the second analysis result acquisition unit 21b measures the heart rate and the stress index, the measured value itself and/or the classification information (normal/attention/boundary, etc.) corresponding to the measured value (normal stress/moderate stress/severe stress). etc.) can be created and transmitted.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a frequency count for each keyword by performing analysis on the text data obtained by converting the second voice data.

Specifically, the second analysis result acquisition unit 21b acquires the frequency count for each keyword preset by the counselor by analyzing the entire text data (via STT) in which each voice frame constituting the second voice data is converted. can do. This may be achieved through analysis of mapping text data and preset keywords.

Depending on the embodiment, preset keywords may be implemented to be changed and set by a counselor in real time.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate and transmit a graph showing frequency counts for each keyword, as shown in FIG. 8 .

Depending on the embodiment, the second analysis result acquisition unit 21b may sort and provide frequency counts for each keyword in descending or ascending order.

Depending on the embodiment, the second analysis result acquisition unit 21b may provide the frequency count for each keyword represented by the size of a circle. For example, a graph may be created by expressing keywords that are uttered with a high frequency as a circle having a relatively large size.

Depending on the embodiment, the second analysis result acquisition unit 21b receives a set keyword from the counselor terminal 30, or the emotion analysis result providing device 20 receives a user input through a separate user interface unit (not shown). You can directly enter the set keyword through

Depending on the embodiment, the second analysis result acquisition unit 21b performs Latent Dirichlet Allocation (LDA) topic modeling on the text data from which the second voice data is converted, so that each of the keywords configured for each topic frequency can be obtained.

Specifically, the second analysis result acquisition unit 21b performs Latent Dirichlet Allocation (LDA) topic modeling for the entire text data (via STT) in which each voice frame constituting the second voice data is converted. It is possible to obtain a frequency count for each of the keywords configured for each topic.

According to an embodiment, the second analysis result acquisition unit 21b pre-processes each of a plurality of documents constituting the entire text data, extracts at least one topic from the plurality of pre-processed documents, and constructs a topic by topic. It is possible to calculate the word frequency for each of the keywords.

Depending on embodiments, the preprocessing process may include at least one of removing special characters (or removing numbers), morpheme analysis, removing stopwords, and processing synonyms (or synonyms).

Depending on the embodiment, the second analysis result acquisition unit 21b may extract at least one topic using a plurality of preprocessed documents and match each of the plurality of preprocessed documents with each of the extracted at least one topic. there is. Specifically, keywords are extracted from the plurality of preprocessed documents, and frequency analysis of the extracted keywords is performed. In addition, the second analysis result acquisition unit 21b may generate a term-document matrix for topic modeling. The second analysis result acquisition unit 21b may extract at least one topic (subject) through a keyword constituting the document through a Latent Dirichlet Allocation (LDA) technique, and classify each document by topic. In this case, one document may be included in a plurality of topics. That is, documents and topics may have a correspondence relationship of 1:N (N is a natural number greater than or equal to 1). The LDA model (or LDA model) is a model that predicts what topic the document is dealing with by analyzing the words of a given document based on the pre-calculated word distribution for each subject. The criterion for dividing topics is one of the cluster analysis techniques, which determines the number of K clusters through silhouette analysis and analyzes the top keywords according to K topics.

Depending on the embodiment, the second analysis result acquisition unit 21b may graph the frequency count of words for each of the keywords constituting the topic for each topic into a plurality of graphs as shown in FIGS. 9A and 9B (enlarged graph of FIG. 9A). there is.

Depending on the embodiment, circles corresponding to the number of topics may be marked on an arbitrary graph, and in particular, the graph may be made such that the size of the circles is proportional to the frequency of keywords corresponding to the topics. In addition, the distance between circles may be calculated and graphed based on the degree of similarity between topics. In addition, on another graph, keywords corresponding to topics of each circle and frequency information for each keyword may be displayed.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain a voice parameter by analyzing the second voice data.

According to the present invention, a variety of well-known techniques can be applied to obtain each voice parameter for each voice frame using voice frames for the second voice data, and a description thereof will be omitted.

Depending on the embodiment, the voice parameter may include indicators representing a waveform, pitch, pause section, speech speed, and/or hesitation section.

The pitch may be determined based on the frequency and period of the voice.

The pause period may be defined as a period in which a waveform does not occur or a waveform is discontinued.

Depending on the embodiment, the second analysis result acquisition unit 21b determines the pause period as a period in which the client hesitates when the pause period is longer than a predetermined time (eg > 2 seconds), and uses this to determine an index indicating the hesitation period (eg > hesitation). count, average hesitation time).

Depending on the embodiment, the second analysis result acquisition unit 21b may generate graphs of voice parameters for each time period as shown in FIGS. 10A and 10B.

For example, FIG. 10A shows a pitch change for each time period, and 10B shows a pose change for each time period.

Depending on the embodiment, the second analysis result acquisition unit 21b may obtain gaze position information and face movement information by analyzing based on the second image data.

According to an embodiment, the second analysis result acquisition unit 21b recognizes the client's face from each image frame constituting the second image data, detects the recognized face feature points, and based on the face feature points. Gaze position information may be obtained by designating a region of interest, detecting a pupil center position in the designated region of interest, and calculating a degree of change of the detected pupil center position over time based on a change in Euclidean distance. .

Recognizing the client's face from the image frame and detecting feature points of the recognized face may be performed by applying a known algorithm.

According to the embodiment, the second analysis result acquisition unit 21b designates each region composed of feature points corresponding to the client's left eye and right eye among the facial feature points as the region of interest, and each of the left eye region and the right eye region It is possible to detect the pupil center position in the region.

Depending on the embodiment, the second analysis result acquisition unit 21b may designate a region of interest by comparing the previously generated feature point template (including the region of interest) with the recognized facial feature points.

For example, referring to FIG. 12 , the second analysis result acquisition unit 21b obtains first both end points based on the horizontal axis and second both end points based on the vertical axis of the left eye region (and each right eye region). Relative coordinates (central coordinates between both first end points and center coordinates between both end points) may be calculated based on each coordinate, and the corresponding relative coordinates may be detected as the pupil center position.

In addition, the Euclidean distance between the relative coordinates calculated according to the change in the client's gaze over time and the reference coordinates is calculated, and the gaze position information is obtained using the average value of the Euclidean distances calculated from each of the left eye area and the right eye area. can be obtained.

Depending on the embodiment, the reference coordinates may be calculated as an average value of Euclidean distances measured in real time for a predetermined time (eg, about 1 minute after counseling starts).

Depending on the embodiment, as shown in FIG. 11, the second analysis result acquisition unit 21b graphs the average value of the Euclidean distance (displacement per reference time, movement per second) over time, or Euclidean distance over time. The accumulated value of the average value of the Dian distance (accumulated displacement per reference time, accumulated motion) can be graphed, prepared and transmitted.

According to an embodiment, the second analysis result acquisition unit 21b recognizes the client's face from each image frame constituting the second image data, detects a feature point of the recognized face, and determines the position of the detected feature point. Facial motion information may be obtained by calculating the degree of change over time based on the change in Euclidean distance.

Recognizing the client's face from the image frame and detecting feature points of the recognized face may be performed by applying a known algorithm.

Depending on the embodiment, the second analysis result acquisition unit 21b may use the tip of the nose of the face to detect the movement position of the face.

Depending on the embodiment, the second analysis result acquisition unit 21b may compare the previously generated feature point template (facial feature point including the nose tip) with the recognized facial feature point and detect the tip of the nose as the target feature point.

Depending on the embodiment, the coordinate values of the tip of the nose may use absolute coordinate values based on the entire image frame.

Depending on the embodiment, absolute coordinate values may be calculated with reference to a pixel unit constituting an image frame.

In addition, facial movement position information may be obtained by calculating a Euclidean distance between the coordinates of the tip of the nose at the current point in time and the coordinates of the tip of the nose at the previous point in time, which are calculated according to changes in the facial movement of the client over time.

Depending on the embodiment, as shown in FIG. 11, the second analysis result acquisition unit 21b graphs the average value of the Euclidean distance (displacement per reference time, movement per second) over time, or Euclidean distance over time. The accumulated value of the average value of the Dian distance (accumulated displacement per reference time, accumulated motion) can be graphed, prepared and transmitted.

Depending on the embodiment, the second analysis result acquisition unit 21b may create and transmit graphs of gaze position information and face movement information by time zone, respectively, as shown in FIG. 11 .

The transmission unit 21c may transmit information including all of the plurality of second analysis results to the counselor terminal 30 (s24).

According to the embodiment, information including all of the plurality of second analysis results may include all of the plurality of second analysis results, video data received in real time from the client terminal 10, and audio data of the client separated from the video data received in real time. may include the contents of the converted text data and the converted text data of the counselor's voice data.

5 to 11 illustrate screens of the user interface unit 33 according to the operation of the second analysis result providing unit 32 of the counselor terminal 30 .

The second analysis result providing unit 32 of the counselor terminal 30 provides information including the second analysis result selected by the user among the entirety of the plurality of second analysis results received from the emotion analysis result providing device 20 and the counselor provides information in real time. By outputting the recorded counseling notes through the screen of one user interface unit 33, it can be provided to users such as administrators or counselors.

Depending on the embodiment, the information including the second analysis result selected by the user from among the plurality of second analysis results may be the second analysis result selected by the user from among the plurality of second analysis results, and assistance related to the selected second analysis result. As a result of the analysis, video data of the client, text data obtained by converting the client's voice data separated from the video data of the client, and contents corresponding to text data obtained by converting the counselor's voice data may be included.

Depending on the embodiment, the second analysis result providing unit 32 may receive a user input for outputting a second analysis result selected from a plurality of second analysis results through the screen of the user interface unit 33 .

Depending on the embodiment, the screen of the user interface unit 33 includes a plurality of menus M indicating a plurality of second analysis results, and the second analysis result providing unit 32 provides menus selected by the user. The corresponding second analysis result (①) may be output through the screen of the user interface unit 33 and provided to the user.

For example, FIG. 5 illustrates a screen output as a user's selection input for a menu for outputting a plurality of general emotional states as the second analysis result, and FIG. 6 illustrates a plurality of substitute emotional states as the second analysis result. A screen output as a user's selection input for a menu for output is illustrated, and FIG. 7 illustrates a screen output as a user's selection input for a menu for outputting heart rate and stress measurement values as a second analysis result. 8 illustrates a screen output as a user's selection input for a menu for outputting the frequency count for each keyword as a second analysis result, and FIGS. A screen output as a user's selection input for a menu for output as a result is illustrated, and FIGS. 10A and 10B illustrate a screen output as a user's selection input for a menu for outputting a voice parameter as a second analysis result. 11 illustrates a screen output as a user's selection input for a menu for outputting gaze position information and/or face motion information as a second analysis result.

Depending on the embodiment, each content may be output to a separate area on the screen of the user interface unit 33 .

For example, as an example of the screen of the user interface unit 33 of FIGS. 5 to 11, the second analysis result (①), the auxiliary analysis result related to the second analysis result (②), and the video data of the client (⑤), text data obtained by converting the client's voice data, text data obtained by converting the counselor's voice data (④), and consultation notes recorded by the counselor in real time (③) may be included.

On the screen of the user interface unit 33 of FIGS. 5 to 7 and FIGS. 10A, 10B, and 11 , the second analysis result for each time period may be implemented in a graph form and provided according to an embodiment.

Specifically, the one user interface screen 33 is divided into a plurality of areas, and the second analysis result providing unit 32 is configured to select the second analysis result providing unit 32 in one of the plurality of areas by the user. A graph showing the analysis results by time period can be output.

On the screen of the user interface unit 33 of FIGS. 5 to 11, according to the embodiment, the text data obtained by converting the client's voice data and the text data (④) obtained by converting the counselor's voice data are edited by the user or converted into a file. It may be provided in a form and provided to the user.

On the screens of the user interface unit 33 of FIGS. 5 to 11 , depending on the embodiment, client information is listed and client information selected by the user can be searched.

Depending on the embodiment, in each region on the screen of each user interface unit 33 of FIGS. 5 to 11, only content corresponding to the second analysis result ① and the auxiliary analysis result related to the first analysis result ② This is implemented differently, and the remaining contents (③④⑤) can be implemented identically.

Referring to FIG. 5 , a graph representing probability values of each of a plurality of general emotional states for each time period may be provided in an area representing the second analysis result ①. According to an embodiment, the auxiliary analysis result (②) related to the second analysis result may include item information for each of a plurality of general emotional states, and according to an embodiment, only the probability value for the general emotional state item selected by the user It can also be set to be output on this graph.

According to an embodiment, among probability values of a plurality of general emotional states appearing on the graph at the same point in time, a general emotional state representing a value having the highest probability value may be determined to be a representative emotional state at that point in time.

According to the present invention, through the second analysis result of FIG. 5 , the user can check the client's emotional change for each time period (according to the lapse of time) within the entire counseling section as a whole. At this time, the user can collectively check the probability values of each of the plurality of general emotional states or individually check the probability values of the selected general emotional states. Accordingly, the user can seek the direction of counseling in the future by understanding the context of the emotional change of the client as a whole.

Referring to FIG. 6 , a graph representing probability values of each of a plurality of alternate emotional states for each time period may be provided in an area representing the second analysis result ①. Depending on the embodiment, the auxiliary analysis result (②) related to the second analysis result may include frequency and/or probability information corresponding to positive, negative, neutral, and the like, and tabular information thereof. Depending on the embodiment, it may be set so that only information corresponding to the alternate emotional state selected by the user is output.

According to the present invention, through the second analysis result of FIG. 6 , the user can check the client's emotional change for each time period (according to the passage of time) within the entire counseling section as a whole. At this time, the user can check the probability values of each of the plurality of alternate emotional states integrally or individually the probability values of the selected alternate emotional states. Accordingly, the user can seek the direction of counseling in the future by understanding the context of the emotional change of the client as a whole.

In particular, according to the present invention, through the second analysis result of FIG. 5, the user can check more detailed classified (categorized) emotional state information, whereas through the second analysis result of FIG. 6, the user It is possible to check the emotional state information classified (categorized) in a larger range, so that the emotional state information can be checked in various ways according to the user's needs and selection.

Referring to FIG. 7 , a graph showing the heart rate and stress index for each time period may be provided in an area showing the second analysis result ①.

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include information such as average heart rate and average stress level. Depending on the embodiment, it may be set to output only information selected by the user among heart rate and stress level.

According to the present invention, through the second analysis result of FIG. 7 , the user can check the stress level of the client as a whole for each time period (according to the passage of time) within the entire counseling section. Accordingly, the user can seek a direction for future counseling by fully understanding the context including the stress level or change of the client.

Referring to FIG. 8 , a graph representing the frequency of each keyword may be provided in an area representing the second analysis result (①).

Depending on the embodiment, the secondary analysis result related to the second analysis result may include keyword list information preset by the counselor.

According to the present invention, among words uttered by a client during counseling, it is possible to check how many key keywords, which are considered important by the counselor and aspects of psychological counseling, appear. For example, key words such as suicide and depression are keywords that clearly reveal the client's psychology and situation, so it is very important to check how many of these keywords appear. In addition, not only set words pre-registered by the manager, but also real-time words can be registered and confirmed by the counselor conducting the consultation, and words to be excluded from analysis are set to be excluded, and not only the set words, If you want to analyze all the words uttered by the client, you can analyze the entire language.

Referring to FIG. 9A , a frequency count for each of the keywords configured for each topic may be graphed in a plurality of forms and provided in an area representing the second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include list information indicating the frequency and rank of each of the keywords configured for each topic.

According to the present invention, through the second analysis result of FIG. 9a, it is possible to identify the subject field in which the client is mainly interested in the contents of the counseling by the counselor, and examine the relationship with other subjects together. there will be Accordingly, the counselor can seek the direction of future counseling by understanding the client's main field of interest as a whole.

Referring to FIGS. 10A and 10B , voice parameters for each time period may be graphed and provided to an area showing a second analysis result (①).

Depending on the embodiment, the auxiliary analysis result related to the second analysis result may include information such as an average pitch value, an average pose value, an average number of utterances per predetermined time, the number of hesitations, and an average number of hesitations. Depending on the embodiment, it may be set to output only information selected by the user among them.

According to the present invention, the counselor can check when the client raises his voice, when he speaks quietly, when he speaks slowly, when he speaks quickly, etc. through a graph. By checking the overall change of the voice waveform, it is possible to determine the client's minute emotional change by time period.

Referring to FIG. 11 , gaze position information and face motion information for each time period may be graphed and provided to an area representing the second analysis result ①.

Depending on embodiments, the auxiliary analysis result related to the second analysis result may include an average gaze position change value, an accumulated average gaze position change value, an average facial motion change value, and an accumulated average facial motion change value. Depending on the embodiment, it may be set to output only information selected by the user among them.

According to the present invention, through the analysis result of FIG. 11, since moving the gaze or face can be evidence that the emotional state of the client is revealed through the body, the counselor confirms the unconscious behavior of the client through the result. The client's emotional state can be accurately determined, and the client's emotion can be more accurately grasped while checking the counseling contents of the corresponding time period by finding the point in time when the displacement on the graph occurred greatly.

According to the present invention, the first analysis result is obtained through real-time analysis through the first analysis result acquisition unit 21a, but the second analysis result is acquired through analysis after a predetermined time has elapsed through the second analysis result acquisition unit 21b. By making it possible to receive real-time counseling details with the same client through real-time analysis, while providing the entire counseling details through integrated analysis after counseling is finished, various types of counseling details As analysis results can be provided, it becomes possible to manage more diversified and systematized counseling details.

In addition, a user interface screen during counseling is implemented differently from a user interface screen that provides analysis results after counseling is completed, so that the counselor can more easily manage and check the counseling details. For example, when counseling is in progress, the entire analysis result is provided through a screen of one user interface unit 33 while receiving the real-time analysis result, so that the counselor can check, analyze, and/or determine the real-time consultation details. It can be done more quickly and easily, and when the consultation is finished, one (selected) analysis result for the entire accumulated consultation history is provided through the screen of one user interface unit 33, so that the user can view the entire consultation history. can be checked collectively.

13 is a diagram referenced to describe a learning process for generating an ensemble model according to an embodiment.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate a plurality of models in advance and store them in the storage unit 22 .

According to an embodiment, a plurality of models may be generated as ensemble-based models in which each of a video frame for learning, an audio frame for learning, and text data for learning is applied to learning.

Depending on the embodiment, a plurality of models may be generated by ensemble of machine learning and deep learning.

Depending on the embodiment, the second analysis result acquisition unit 21b may acquire the first model learned based on the image frame for learning and store it in the memory 22 .

Depending on the embodiment, the first model may be a MobileNet model based on a Convolutional Neural Network (CNN) algorithm, but a Deep Neural Network (DNN), Deep Convolution Neural Network (DCNN), Recurrent Neural Network (RNN), K-Nearest (KNN) Neighbor), SVM (Support Vector Machine), Random Forest, and Decision Tree. The MobileNet architecture employs depthwise separable convolutions (a form of factored convolutions) to minimize processing operations (ie, floating point operations, multiplications and/or additions, etc.). Depthwise separable convolutions replace standard convolution with depthwise convolution and 1 x 1 convolution (also known as "pointwise convolution") in the view of speeding up processing by reducing or minimizing the number of operations required. (e.g., spit out its functions). Depthwise convolution applies a single filter to each input channel. The pointwise convolution then applies a 1 x 1 convolution to combine the outputs of the depthwise convolution, filtering and combining functions/operations, rather than a single filtering and combining operation performed by standard convolutions. separate into two stages. Accordingly, structures in the MobileNet architecture may include two convolutional layers, one depthwise convolutional layer and one pointwise convolutional layer per structure to define or instantiate a “group of layers”.

Depending on the embodiment, the second analysis result acquisition unit 21b may generate a first model by classifying emotions for the learning image frame through neural network learning through CNN.

The second analysis result acquisition unit 21b may obtain a second model learned based on the training voice frame and store it in the storage unit 22 .

Depending on the embodiment, the second model may be a support vector machine (SVM) algorithm-based model, but CNN, deep neural network (DNN), deep convolution neural network (DCNN), recurrent neural network (RNN), K-N (K- Nearest Neighbor), Random Forest, 　Decision　Tree, etc. Support Vector Machine (SVM) algorithm is used for classification and regression analysis, and a set of data belonging to either category is given. When lost, the SVM algorithm creates a non-probabilistic binary linear classification model based on a given data set to determine which category the new data belongs to.

According to an embodiment, the second analysis result acquisition unit 21b extracts a voice feature vector based on the frequency analysis of the learning voice frame, learns the extracted voice feature vector through a Support Vector Machine (SVM) algorithm, and In the process, the second model may be generated by classifying speech feature vectors by emotion.

The second analysis result acquisition unit 21b may acquire the third model 22c learned based on the text data for learning and store it in the storage unit 22 .

Depending on the embodiment, the third model may be a BERT (Bidirectional Encoder Representations from Transformers) model, but a Deep Neural Network (DNN), Deep Convolution Neural Network (DCNN), CNN, Recurrent Neural Network (RNN), K-Nearest (KNN) Neighbor), SVM (Support Vector Machine), Random Forest, Decision, Tree, etc. BERT is an artificial intelligence model based on the transformer architecture of encoder-decoder structure. It is characterized by stacking a plurality of transformer layers for in-depth representation and applying a masking process to a masking language model, which is a token sequence. The BERT model exhibits high accuracy even with a small amount of data by undergoing a fine-tuning process, and is an attention-based model that improves performance by drawing attention to a specific vector. there is

According to an embodiment, the second analysis result acquisition unit 21b generates a third model by classifying emotions for the text data for learning through neural network learning to acquire a context-based embedding value based on the text data for learning. can

The above-described embodiments may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules for executing processes according to the present invention and vice versa.

Aspects herein may take the form of entirely hardware, entirely software (including firmware, resident software, microcode, etc.) or a computer program product embodied entirely in one or more computer readable media having computer readable program code embodied thereon. .

Features, structures, effects, etc. described in the embodiments above are included in one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, the features, structures, and effects illustrated in each embodiment can be combined or modified with respect to other embodiments by those skilled in the art in the field to which the embodiments belong. Therefore, contents related to these combinations and variations should be construed as being included in the scope of the present invention.

In addition, although the above has been described with a focus on the embodiments, these are only examples and do not limit the present invention, and those skilled in the art to which the present invention belongs can exemplify the above to the extent that does not deviate from the essential characteristics of the present embodiment. It will be seen that various variations and applications that have not been made are possible. For example, each component specifically shown in the embodiment can be modified and implemented. And the differences related to these modifications and applications should be construed as being included in the scope of the present invention as defined in the appended claims.

Claims (11)

a first analysis result obtaining unit configured to obtain a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of first audio data and first video data extracted from video data received from the client terminal in real time; and
Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated for a predetermined time, the client's emotion is analyzed after the predetermined time has elapsed, and the plurality of second video data received in real time is analyzed. A second analysis result acquisition unit for acquiring analysis results; including,
A device that provides emotion analysis results.
According to claim 1,
The plurality of first analysis results,
At least two of a probability value of each of a plurality of general emotional states, a representative emotional state, a heart rate and a stress index, a frequency by keyword, a frequency for each of keywords configured by topic, a voice parameter, gaze position information, and facial movement information. ,
Sentiment analysis result providing device.
According to claim 1,
The plurality of second analysis results,
At least one of a probability value of each of a plurality of general emotional states, a probability value of each of a plurality of substitutional emotional states, a heart rate and a stress index, a frequency of each keyword, a frequency of each of keywords configured for each topic, a voice parameter, gaze position information, and face movement information. containing two or more
Sentiment analysis result providing device.
According to claim 3,
The plurality of substitutional emotional states are a combination of emotional states defined as a superordinate concept of each of the plurality of general emotional states,
Sentiment analysis result providing device.
According to claim 3,
The second analysis result acquisition unit,
Obtaining a probability value of each of a plurality of general emotional states using a plurality of models based on ensemble learning based on the second video data, the second audio data, and the text data in which the second audio data is converted,
Sentiment analysis result providing device.
According to claim 3,
The second analysis result acquisition unit,
Performing Latent Dirichlet Allocation (LDA) topic modeling on the text data from which the second voice data is converted to obtain a frequency count for each of the keywords configured for each topic.
Sentiment analysis result providing device.
According to claim 3,
The second analysis result acquisition unit,
Based on the second image data, the client's face is recognized, a feature point of the recognized face is detected, a pupil center position is detected in a designated region of interest based on the face feature point, and a pupil center position is determined based on the detected pupil center position. Calculating the gaze position information, which is a degree of change over time, based on a change in Euclidean distance,
Sentiment analysis result providing device.
In the emotion analysis result providing system including a client terminal, a counselor terminal, and an emotion analysis result providing device,
The emotion analysis result providing device,
a first analysis result obtaining unit configured to obtain a plurality of first analysis results by analyzing the client's emotions in real time based on at least one of first audio data and first video data extracted from video data received from the client terminal in real time;
Based on at least one of second voice data and second video data extracted from the accumulated video data accumulated for a predetermined time, the client's emotion is analyzed after the predetermined time has elapsed, and the plurality of second video data received in real time is analyzed. a second analysis result acquisition unit for acquiring analysis results; and
A transmitter configured to transmit information including all of the plurality of first analysis results and information including all of the plurality of second analysis results to the counselor terminal;
The counselor terminal,
a first analysis result providing unit that provides information including all of the plurality of first analysis results received from the emotion analysis result providing device through a single user interface screen; and
A second analysis result providing unit providing information including a second analysis result selected by a user among all of the plurality of second analysis results received from the emotion analysis result providing device through the one user interface screen; ,
Sentiment analysis result providing system.
According to claim 8,
The one user interface screen includes a plurality of menus representing the plurality of second analysis results,
The second analysis result providing unit,
Providing information including the second analysis result corresponding to a menu selected by a user from among the plurality of menus through the one user interface screen,
Sentiment analysis result providing system.
According to claim 8,
The plurality of first analysis results,
At least two of a probability value of each of a plurality of general emotional states, a representative emotional state, a heart rate and a stress index, a frequency by keyword, a frequency for each of keywords configured by topic, a voice parameter, gaze position information, and facial movement information. ,
Sentiment analysis result providing system.
According to claim 8,
The plurality of second analysis results,
At least one of a probability value of each of a plurality of general emotional states, a probability value of each of a plurality of substitutional emotional states, a heart rate and a stress index, a frequency of each keyword, a frequency of each of keywords configured for each topic, a voice parameter, gaze position information, and face movement information. containing two or more
Sentiment analysis result providing system.